* 4.8.8, bcache deadlock and hard lockup
@ 2016-11-18 16:46 Marc MERLIN
2016-11-18 18:49 ` Marc MERLIN
0 siblings, 1 reply; 20+ messages in thread
From: Marc MERLIN @ 2016-11-18 16:46 UTC (permalink / raw)
To: linux-bcache
I created a new array as bcache and tried to copy a bunch of data to it.
A few hours later, the kenrel died with
[36403.763610] BUG: workqueue lockup - pool cpus=5 node=0 flags=0x0 nice=0 stuck for 31s!
[36403.789660] BUG: workqueue lockup - pool cpus=7 node=0 flags=0x0 nice=0 stuck for 31s!
[36403.815675] Showing busy workqueues and worker pools:
[36403.832862] workqueue events: flags=0x0
[36403.846370] pwq 14: cpus=7 node=0 flags=0x0 nice=0 active=5/256
[36403.866827] pending: vmpressure_work_fn, drm_fb_helper_dirty_work, cache_reap, do_cache_clean [sunrpc], write_dirty [bcache]
[36403.905727] pwq 10: cpus=5 node=0 flags=0x0 nice=0 active=5/256
[36403.926137] pending: journal_write_work [bcache], write_dirty [bcache], cache_reap, btree_node_write_work [bcache], btree_node_write_work [bcache]
[36403.970745] pwq 8: cpus=4 node=0 flags=0x0 nice=0 active=1/256
[36403.990880] in-flight: 8141:write_dirty_finish [bcache]
[36404.009607] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256
[36404.029708] in-flight: 25326:write_dirty_finish [bcache]
[36404.048649] pending: cache_reap
[36404.061031] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=49/256
[36404.081293] in-flight: 8082:write_dirty_finish [bcache], 29134:write_dirty_finish [bcache], 29144:write_dirty_finish [bcache], 29316:write_dirty_finish [bcache], 1153:write_dirty_finish [bcache], 29150:write_dirty_finish [bcache], 29239:write_dirty_finish [bcache], 14064:write_dirty_finish [bcache], 15612:write_dirty_finish [bcache]
[36404.176733] pending: kernfs_notify_workfn, write_dirty_finish [bcache], write_dirty_finish [bcache], write_dirty_finish [bcache], write_dirty_finish [bcache], write_dirty_finish [bcache], write_dirty_finish [bcache], write_dirty_finish [bcache], write_dirty_finish [bcache], write_dirty_finish [bcache], write_dirty_finish [bcache], write_dirty_finish [bcache], write_dirty_finish [bcache], write_dirty_finish [bcache], write_dirty_finish [bcache], cache_reap, write_dirty [bcache], btree_node_write_work [bcache], update_writeback_rate [bcache], update_writeback_rate [bcache], write_dirty [bcache], write_dirty [bcache], write_dirty [bcache], write_dirty [bcache], write_dirty [bcache], write_dirty [bcache], write_dirty [bcache], write_dirty [bcache], write_dirty [bcache], write_dirty [bca
che], write_dirty [bcache], write_dirty [bcache], write_dirty [bcache], btree_node_write_work [bcache], write_dirty [bcache], write_dirty [bcache], write_dirty [bcache], write_dirty [bcache], write_dirty [bcache], write_dirty [bcache]
[36404.464686] workqueue events_unbound: flags=0x2
[36404.480072] pwq 16: cpus=0-7 flags=0x4 nice=0 active=1/512
[36404.498937] in-flight: 25187:call_usermodehelper_exec_work
[36404.518259] workqueue writeback: flags=0x4e
[36404.532531] workqueue vmstat: flags=0xc
[36404.545716] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256
[36404.565589] pending: vmstat_update
[36404.578593] workqueue bcache: flags=0x8
[36404.591714] pwq 14: cpus=7 node=0 flags=0x0 nice=0 active=256/256 MAYDAY
[36404.614102] in-flight: 26082:bch_data_insert_keys [bcache], 26036:bch_data_insert_keys [bcache], 26015:bch_data_insert_keys [bcache], 26076:bch_data_insert_keys [bcache], 26048:bch_data_insert_keys [bcache], 26063:bch_data_insert_keys [bcache], 26019:bch_data_insert_keys [bcache], 26033:bch_data_insert_keys [bcache], 15710:bch_data_insert_keys [bcache], 26090:bch_data_insert_keys [bcache], 15714:bch_data_insert_keys [bcache], 26011:bch_data_insert_keys [bcache], 15711:bch_data_insert_keys [bcache], 26087:bch_data_insert_keys [bcache], 26081:bch_data_insert_keys [bcache], 26031:bch_data_insert_keys [bcache], 26006:bch_data_insert_keys [bcache], 26089:bch_data_insert_keys [bcache], 26084:bch_data_insert_keys [bcache], 26073:bch_data_insert_keys [bcache], 26047:bch_data_insert_keys [bc
ache], 15706:bch_data_insert_keys [bcache], 26017:bch_data_insert_keys [bcache], 26043:bch_data_insert_keys [bcache], 26037:bch_data_insert_keys [bcache], 25339:bch_data_insert_keys [bcache], 15715:bch_data_insert_keys [bcache], 26088:bch_data_insert_keys [bcache], 26077:bch_data_insert_keys (many deleted)
[36405.826991] pending: bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache] (many deleted)
[36420.315213] CPU: 5 PID: 0 Comm: swapper/5 Tainted: G U 4.8.8-amd64-volpreempt-sysrq-20161108 #7
[36420.315214] Hardware name: System manufacturer System Product Name/P8H67-M PRO, BIOS 3904 04/27/2013
[36420.315214] task: ffff9b6f02f96100 task.stack: ffff9b6f02f98000
[36420.315214] RIP: 0010:[<ffffffff8857a128>] [<ffffffff8857a128>] cpuidle_enter_state+0x119/0x171
[36420.315215] RSP: 0018:ffff9b6f02f9bea0 EFLAGS: 00200246
[36420.315215] RAX: ffff9b6f1e3590c0 RBX: 0000000000000002 RCX: 000000000000001f
[36420.315215] RDX: 0000000000000000 RSI: 0000000000000005 RDI: 0000000000000000
[36420.315216] RBP: ffff9b6f02f9bed0 R08: 0000000000000f94 R09: 0000000000000f89
[36420.315216] R10: ffff9b6f02f9be50 R11: 071c71c71c71c71c R12: ffffd6657fd4b200
[36420.315216] R13: 0000211cbb6f6cf7 R14: 0000000000000002 R15: 0000211cbb3291c0
[36420.315217] FS: 0000000000000000(0000) GS:ffff9b6f1e340000(0000) knlGS:0000000000000000
[36420.315217] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[36420.315217] CR2: 00000000f5901024 CR3: 0000000306432000 CR4: 00000000001406e0
[36420.315218] Stack:
[36420.315218] 0000000000000ef6 ffffd6657fd4b200 ffffffff88ccc060 ffff9b6f02f9c000
[36420.315218] ffff9b6f02f98000 ffff9b6f02f9c000 ffff9b6f02f9bee0 ffffffff8857a1ac
[36420.315219] ffff9b6f02f9bf30 ffffffff8809238d ffff9b6f02f98000 0000000500000002
[36420.315219] Call Trace:
[36420.315219] [<ffffffff8857a1ac>] cpuidle_enter+0x17/0x19
[36420.315220] [<ffffffff8809238d>] cpu_startup_entry+0x210/0x28b
[36420.315220] [<ffffffff8803de22>] start_secondary+0x13e/0x140
[36420.315221] Code: 00 00 00 48 c7 c7 cd ae b2 88 c6 05 4b 8e 7a 00 01 e8 17 6c ae ff fa 66 0f 1f 44 00 00 31 ff e8 75 60 b4 ff fb 66 0f 1f 44 00 00 <4c> 89 e8 b9 e8 03 00 00 4c 29 f8 48 99 48 f7 f9 ba ff ff ff 7f
[36420.315221] Kernel panic - not syncing: Hard LOCKUP
[36420.315221] CPU: 5 PID: 0 Comm: swapper/5 Tainted: G U 4.8.8-amd64-volpreempt-sysrq-20161108 #7
[36420.315222] Hardware name: System manufacturer System Product Name/P8H67-M PRO, BIOS 3904 04/27/2013
[36420.315222] 0000000000000000 ffff9b6f1e345b00 ffffffff8836a05b ffff9b6f02f9bd00
[36420.315223] ffffffff88ab1d54 ffff9b6f1e345b80 ffffffff8812d1f0 ffff9b6f00000010
[36420.315223] ffff9b6f1e345b90 ffff9b6f1e345b28 ffffffff88b37748 ffffffff88ab1575
[36420.315223] Call Trace:
[36420.315223] <NMI> [<ffffffff8836a05b>] dump_stack+0x61/0x7d
[36420.315224] [<ffffffff8812d1f0>] panic+0xd8/0x21a
[36420.315224] [<ffffffff88060a9b>] nmi_panic+0x2c/0x37
[36420.315224] [<ffffffff880ea3c7>] watchdog_overflow_callback+0xc1/0xcc
[36420.315225]
A bit before that, I had an OOM event, but I'm not certain if it's related, especially as I had plenty of RAM left.
Fri Nov 18 03:55:01 PST 2016
total used free shared buffers cached
Mem: 24393028 22588352 1804676 0 96 20150652
-/+ buffers/cache: 2437604 21955424
Swap: 15616764 1412392 14204372
[33987.869396] Purging GPU memory, 0 pages freed, 774 pages still pinned.
[33987.891118] 96 and 0 pages still available in the bound and unbound GPU page lists.
[33987.916534] kthreadd invoked oom-killer: gfp_mask=0x27000c0(GFP_KERNEL_ACCOUNT|__GFP_NOTRACK), order=2, oom_score_adj=0
[33987.950979] kthreadd cpuset=/ mems_allowed=0
[33987.966007] CPU: 7 PID: 2 Comm: kthreadd Tainted: G U 4.8.8-amd64-volpreempt-sysrq-20161108 #7
[33987.997320] Hardware name: System manufacturer System Product Name/P8H67-M PRO, BIOS 3904 04/27/2013
[33988.027020] 0000000000000000 ffff9b6f02f23b98 ffffffff8836a05b ffff9b6f02f23d38
[33988.051780] ffff9b6e50380140 ffff9b6f02f23c00 ffffffff88195ffe 0000000000000000
[33988.076325] ffff9b6f02f23c78 0000000000000000 0000000000000206 ffff9b6f02f23bd8
[33988.100833] Call Trace:
[33988.110385] [<ffffffff8836a05b>] dump_stack+0x61/0x7d
[33988.127954] [<ffffffff88195ffe>] dump_header+0x5c/0x1d3
[33988.146034] [<ffffffff8869773b>] ? _raw_spin_unlock_irqrestore+0x14/0x16
[33988.167763] [<ffffffff8813204e>] oom_kill_process+0x83/0x31f
[33988.186343] [<ffffffff88068604>] ? has_ns_capability_noaudit+0x15/0x1c
[33988.207943] [<ffffffff88068622>] ? has_capability_noaudit+0x17/0x19
[33988.228879] [<ffffffff881326a0>] out_of_memory+0x367/0x382
[33988.247419] [<ffffffff88136434>] __alloc_pages_nodemask+0x9d7/0xa9f
[33988.268293] [<ffffffff8805ebae>] copy_process+0x18a/0x1725
[33988.286165] [<ffffffff88079f4d>] ? init_completion+0x24/0x24
[33988.304579] [<ffffffff8808849f>] ? set_next_entity+0x5b/0xbc
[33988.323293] [<ffffffff8808c6f6>] ? pick_next_task_fair+0x12c/0x348
[33988.343785] [<ffffffff880602b8>] _do_fork+0x81/0x223
[33988.360590] [<ffffffff88694d2d>] ? __schedule+0x3c3/0x574
[33988.378662] [<ffffffff88060483>] kernel_thread+0x29/0x2b
[33988.396581] [<ffffffff8807a712>] kthreadd+0x131/0x16b
[33988.413586] [<ffffffff88697a9f>] ret_from_fork+0x1f/0x40
[33988.431333] [<ffffffff8807a5e1>] ? kthread_create_on_cpu+0x5e/0x5e
[33988.451701] Mem-Info:
[33988.460014] active_anon:551103 inactive_anon:166870 isolated_anon:32
33988.460014] active_file:3638356 inactive_file:571701 isolated_file:0
[33988.460014] unevictable:1985 dirty:2406 writeback:570591 unstable:0
[33988.460014] slab_reclaimable:62971 slab_unreclaimable:124630
[33988.460014] mapped:571906 shmem:622799 pagetables:5534 bounce:0
[33988.460014] free:92854 free_pcp:812 free_cma:3110
[33988.575035] Node 0 active_anon:2204184kB inactive_anon:663764kB active_file:14553496kB inactive_file:2301244kB unevictable:7940kB isolated(anon):0kB isolated(file):128kB mapped:2287484kB dirty:9676kB writeback:2291708kB shmem:0kB shmem_thp: 0kB shmem_pmdmapped: 0kB anon_thp: 2491072kB writeback_tmp:0kB unstable:0kB pages_scanned:26435 all_unreclaimable? no
[33988.673542] Node 0 DMA free:15884kB min:168kB low:208kB high:248kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB writepending:0kB present:15976kB managed:15892kB mlocked:0kB slab_reclaimable:0kB slab_unreclaimable:8kB kernel_stack:0kB pagetables:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB
[33988.766919] lowmem_reserve[]: 0 3200 23767 23767 23767
[33988.783939] Node 0 DMA32 free:117664kB min:35424kB low:44280kB high:53136kB active_anon:64468kB inactive_anon:11928kB active_file:2253948kB inactive_file:348364kB unevictable:64kB writepending:349392kB present:3362068kB managed:3296500kB mlocked:64kB slab_reclaimable:57132kB slab_unreclaimable:33292kB kernel_stack:1552kB pagetables:2432kB bounce:0kB free_pcp:136kB local_pcp:0kB free_cma:0kB
[33988.892817] lowmem_reserve[]: 0 0 20567 20567 20567
[33988.909007] Node 0 Normal free:238580kB min:226544kB low:283180kB high:339816kB active_anon:2145316kB inactive_anon:639248kB active_file:12300184kB inactive_file:1948300kB unevictable:7876kB writepending:1962392kB present:21485568kB managed:21080636kB mlocked:7876kB slab_reclaimable:194656kB slab_unreclaimable:464164kB kernel_stack:22720kB pagetables:19672kB bounce:0kB free_pcp:944kB local_pcp:0kB free_cma:12440kB
[33989.024095] lowmem_reserve[]: 0 0 0 0 0
[33989.037169] Node 0 DMA: 1*4kB (U) 1*8kB (U) 0*16kB 0*32kB 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15884kB
[33989.080920] Node 0 DMA32: 17507*4kB (UME) 5928*8kB (UM) 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 117452kB
[33989.121604] Node 0 Normal: 35622*4kB (UMEHC) 10360*8kB (UMHC) 80*16kB (HC) 74*32kB (HC) 63*64kB (HC) 31*128kB (HC) 3*256kB (C) 2*512kB (HC) 0*1024kB 0*2048kB 0*4096kB = 238808kB
[33989.172906] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
[33989.199648] 4855832 total pagecache pages
[33989.213115] 22430 pages in swap cache
[33989.225565] Swap cache stats: add 4289109, delete 4266743, find 2271598/2744355
[33989.248937] Free swap = 14581328kB
[33989.260873] Total swap = 15616764kB
[33989.272779] 6215903 pages RAM
[33989.283138] 0 pages HighMem/MovableOnly
[33989.296160] 117646 pages reserved
[33989.307581] 4096 pages cma reserved
[33989.319493] 0 pages hwpoisoned
(...)
[33996.018149] Out of memory: Kill process 23119 (chrome) score 200 or sacrifice child
[33996.041502] Killed process 23119 (chrome) total-vm:181928kB, anon-rss:2376kB, file-rss:13824kB, shmem-rss:0kB
[33996.079371] oom_reaper: reaped process 23119 (chrome), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB
The BUG: workqueue lockup happened 40mn later...
Marc
--
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
Microsoft is to operating systems ....
.... what McDonalds is to gourmet cooking
Home page: http://marc.merlins.org/ | PGP 1024R/763BE901
^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: 4.8.8, bcache deadlock and hard lockup
2016-11-18 16:46 4.8.8, bcache deadlock and hard lockup Marc MERLIN
@ 2016-11-18 18:49 ` Marc MERLIN
2016-11-20 21:13 ` Coly Li
0 siblings, 1 reply; 20+ messages in thread
From: Marc MERLIN @ 2016-11-18 18:49 UTC (permalink / raw)
To: linux-bcache
On Fri, Nov 18, 2016 at 08:46:43AM -0800, Marc MERLIN wrote:
> I created a new array as bcache and tried to copy a bunch of data to it.
> A few hours later, the kenrel died with
> [36403.763610] BUG: workqueue lockup - pool cpus=5 node=0 flags=0x0 nice=0 stuck for 31s!
> [36403.789660] BUG: workqueue lockup - pool cpus=7 node=0 flags=0x0 nice=0 stuck for 31s!
> [36403.815675] Showing busy workqueues and worker pools:
Tried again, same bug, same crash, this time I didn't edit it to show
the high level of recursion on
in-flight: 21316:bch_data_insert_keys [bcache]
delayed: bch_data_insert_keys [bcache]
delayed: kcryptd_crypt
[12044.160812] BUG: workqueue lockup - pool cpus=1 node=0 flags=0x0 nice=0 stuck for 35s!
[12044.186217] Showing busy workqueues and worker pools:
[12044.203532] workqueue events: flags=0x0
[12044.216455] pwq 14: cpus=7 node=0 flags=0x0 nice=0 active=1/256
[12044.236998] in-flight: 26009:write_dirty_finish [bcache]
[12044.256155] pwq 10: cpus=5 node=0 flags=0x0 nice=0 active=2/256
[12044.276665] in-flight: 3368:write_dirty_finish [bcache]
[12044.295473] pending: cache_reap
[12044.308090] pwq 8: cpus=4 node=0 flags=0x0 nice=0 active=1/256
[12044.327592] pending: cache_reap
[12044.339434] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256
[12044.358914] in-flight: 12086:write_dirty_finish [bcache]
[12044.377223] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=6/256
[12044.396647] pending: vmpressure_work_fn, journal_write_work [bcache], mei_timer, vmstat_shepherd, cache_reap, btree_node_write_work [bcache]
[12044.438295] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=10/256
[12044.457969] in-flight: 21334:write_dirty_finish [bcache], 21314:write_dirty_finish [bcache]
[12044.485388] pending: write_dirty [bcache], kernfs_notify_workfn, usb_serial_port_work [usbserial], cache_reap, drm_fb_helper_dirty_work, write_dirty_finish [bcache], update_writeback_rate [bcache], usb_serial_port_work [usbserial]
[12044.550534] workqueue writeback: flags=0x4e
[12044.564304] pwq 16: cpus=0-7 flags=0x4 nice=0 active=4/256
[12044.582607] in-flight: 28522:wb_workfn wb_workfn, 16395:wb_workfn wb_workfn
[12044.605995] workqueue btrfs-delalloc: flags=0xe
[12044.620772] pwq 16: cpus=0-7 flags=0x4 nice=0 active=1/8
[12044.638508] in-flight: 12761:btrfs_delalloc_helper
[12044.655185] workqueue btrfs-endio-write: flags=0xe
[12044.670699] pwq 16: cpus=0-7 flags=0x4 nice=0 active=8/8
[12044.688370] in-flight: 16332:btrfs_endio_write_helper, 6306:btrfs_endio_write_helper, 8055:btrfs_endio_write_helper, 12765:btrfs_endio_write_helper, 28520:btrfs_endio_write_helper, 15209:btrfs_endio_write_helper, 28523:btrfs_endio_write_helper, 31016:btrfs_endio_write_helper
[12044.766023] workqueue btrfs-extent-refs: flags=0xe
[12044.782065] pwq 16: cpus=0-7 flags=0x4 nice=0 active=1/1
[12044.800798] in-flight: 2308:btrfs_extent_refs_helper
[12044.818317] delayed: btrfs_extent_refs_helper, btrfs_extent_refs_helper, btrfs_extent_refs_helper, btrfs_extent_refs_helper, btrfs_extent_refs_helper, btrfs_extent_refs_helper
[12044.869635] workqueue bcache: flags=0x8
[12044.882691] pwq 14: cpus=7 node=0 flags=0x0 nice=0 active=25/256 MAYDAY
[12044.904731] in-flight: 25983:bch_data_insert_keys [bcache], 12147:bch_data_insert_keys [bcache], 12149:bch_data_insert_keys [bcache], 26002:bch_data_insert_keys [bcache], 26008:bch_data_insert_keys [bcache], 25980:bch_data_insert_keys [bcache], 12146:bch_data_insert_keys [bcache], 25984:bch_data_insert_keys [bcache], 25982:bch_data_insert_keys [bcache], 26000:bch_data_insert_keys [bcache], 26003:bch_data_insert_keys [bcache], 25985:bch_data_insert_keys [bcache], 26006:bch_data_insert_keys [bcache], 25978:bch_data_insert_keys [bcache], 25979:bch_data_insert_keys [bcache], 26007:bch_data_insert_keys [bcache], 26014:bch_data_insert_keys [bcache], 25986:bch_data_insert_keys [bcache], 25987:bch_data_insert_keys [bcache], 2477(RESCUER):bch_data_insert_keys [bcache] bch_data_insert_keys [b
cache] bch_data_insert_keys [bcache], 25981:bch_data_insert_keys [bcache], 26005:bch_data_insert_keys [bcache], 26004:bch_data_insert_keys [bcache]
[12045.165297] pwq 10: cpus=5 node=0 flags=0x0 nice=0 active=168/256 MAYDAY
[12045.187565] in-flight: 23666:bch_data_insert_keys [bcache], 10647:bch_data_insert_keys [bcache], 23668:bch_data_insert_keys [bcache], 12503:bch_data_insert_keys [bcache], 12484:bch_data_insert_keys [bcache], 12486:bch_data_insert_keys [bcache], 12485:bch_data_insert_keys [bcache], 3398:bch_data_insert_keys [bcache], 23675:bch_data_insert_keys [bcache], 23670:bch_data_insert_keys [bcache], 12483:bch_data_insert_keys [bcache], 23673:bch_data_insert_keys [bcache], 23667:bch_data_insert_keys [bcache], 23664:bch_data_insert_keys [bcache], 12518:bch_data_insert_keys [bcache], 23671:bch_data_insert_keys [bcache], 10645:bch_data_insert_keys [bcache], 12491:bch_data_insert_keys [bcache], 23669:bch_data_insert_keys [bcache], 23674:bch_data_insert_keys [bcache], 3369:bch_data_insert_keys [bcac
he], 23663:bch_data_insert_keys [bcache], 23672:bch_data_insert_keys [bcache], 23665:bch_data_insert_keys [bcache]
[12045.439919] pending: bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache
], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys
[bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_ins
ert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_
data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcach
e], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache]
[12046.675915] pwq 8: cpus=4 node=0 flags=0x0 nice=0 active=11/256
[12046.695813] in-flight: 13421:bch_data_insert_keys [bcache], 13418:bch_data_insert_keys [bcache], 13416:bch_data_insert_keys [bcache], 13413:bch_data_insert_keys [bcache], 13420:bch_data_insert_keys [bcache], 13417:bch_data_insert_keys [bcache], 22449:bch_data_insert_keys [bcache], 13414:bch_data_insert_keys [bcache], 13415:bch_data_insert_keys [bcache], 13422:bch_data_insert_keys [bcache], 13419:bch_data_insert_keys [bcache]
[12046.817671] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=18/256 MAYDAY
[12046.839377] in-flight: 17775:bch_data_insert_keys [bcache], 25993:bch_data_insert_keys [bcache], 25992:bch_data_insert_keys [bcache], 26024:bch_data_insert_keys [bcache], 26022:bch_data_insert_keys [bcache], 26020:bch_data_insert_keys [bcache], 17774:bch_data_insert_keys [bcache], 26001:bch_data_insert_keys [bcache], 25990:bch_data_insert_keys [bcache], 25999:bch_data_insert_keys [bcache], 26031:bch_data_insert_keys [bcache], 26023:bch_data_insert_keys [bcache], 26032:bch_data_insert_keys [bcache], 25991:bch_data_insert_keys [bcache], 25994:bch_data_insert_keys [bcache], 26026:bch_data_insert_keys [bcache], 17772:bch_data_insert_keys [bcache], 26028:bch_data_insert_keys [bcache]
[12047.041497] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256
[12047.062667] in-flight: 12081:bch_data_insert_keys [bcache], 12084:bch_data_insert_keys [bcache], 12082:bch_data_insert_keys [bcache], 12085:bch_data_insert_keys [bcache]
[12047.114197] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=44/256 MAYDAY
[12047.136619] in-flight: 12627:bch_data_insert_keys [bcache], 12628:bch_data_insert_keys [bcache], 12626:bch_data_insert_keys [bcache], 12624:bch_data_insert_keys [bcache], 12625:bch_data_insert_keys [bcache], 12185:bch_data_insert_keys [bcache]
[12047.206924] pending: bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache
], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache]
[12047.538640] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=256/256 MAYDAY
[12047.560515] in-flight: 21316:bch_data_insert_keys [bcache], 21324:bch_data_insert_keys [bcache], 21317:bch_data_insert_keys [bcache], 21318:bch_data_insert_keys [bcache], 12395:bch_data_insert_keys [bcache], 21319:bch_data_insert_keys [bcache], 21331:bch_data_insert_keys [bcache], 21315:bch_data_insert_keys [bcache], 21320:bch_data_insert_keys [bcache], 21322:bch_data_insert_keys [bcache], 12387:bch_data_insert_keys [bcache], 21321:bch_data_insert_keys [bcache], 21333:bch_data_insert_keys [bcache], 12391:bch_data_insert_keys [bcache], 12388:bch_data_insert_keys [bcache], 21332:bch_data_insert_keys [bcache], 21328:bch_data_insert_keys [bcache], 21335:bch_data_insert_keys [bcache], 21323:bch_data_insert_keys [bcache], 12394:bch_data_insert_keys [bcache], 21325:bch_data_insert_keys [bc
ache], 12389:bch_data_insert_keys [bcache], 21330:bch_data_insert_keys [bcache], 12390:bch_data_insert_keys [bcache], 21326:bch_data_insert_keys [bcache], 21329:bch_data_insert_keys [bcache], 21327:bch_data_insert_keys [bcache], 21337:bch_data_insert_keys [bcache], 12393:bch_data_insert_keys [bcache], 12392:bch_data_insert_keys [bcache]
[12047.870547] pending: bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache
], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys
[bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_ins
ert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_
data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcach
e], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_key
s [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_in
sert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch
_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache]
[12049.838960] delayed: bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache
], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys
[bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_ins
ert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_
data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcach
e], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_key
s [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_in
sert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch
_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcac
he], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_ke
ys [bcache], bch_data_insert_keys [bcache]
[12052.109659] workqueue kcryptd: flags=0x2a
[12052.124689] pwq 16: cpus=0-7 flags=0x4 nice=0 active=8/8
[12052.143921] in-flight: 27452:kcryptd_crypt [dm_crypt], 2331:kcryptd_crypt [dm_crypt], 24384:kcryptd_crypt [dm_crypt], 11840:kcryptd_crypt [dm_crypt], 27434:kcryptd_crypt [dm_crypt], 27437:kcryptd_crypt [dm_crypt], 24101:kcryptd_crypt [dm_crypt], 31015:kcryptd_crypt [dm_crypt]
[12052.224929] delayed: kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_
crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_cr
ypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kc
ryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_cr
ypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_cryp
t [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcry
ptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_cryp
t], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt
[dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcrypt
d_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt]
, kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [d
m_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_
crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt],
kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_
crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_cr
ypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kc
ryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_cr
ypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_cryp
t [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcry
ptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_cryp
t], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt
[dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcrypt
d_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt]
, kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [d
m_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_
crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt],
kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_
crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_cr
ypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kc
ryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_cr
ypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_cryp
t [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcry
ptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_cryp
t], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt
[dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcrypt
d_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt]
, kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [d
m_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_
crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt],
kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_
crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_cr
ypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kc
ryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_cr
ypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_cryp
t [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcry
ptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_cryp
t], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt][12062.414098] NMI watchdog: Watchdog detected hard LOCKUP on cpu 7dModules linked in: veth ip6table_filter ip6_tables ebtable_nat ebtables ppdev lp xt_addrtype br_netfilter bridge stp llc tun autofs4 softdog binfmt_misc ftdi_sio nfsd auth_rpcgss nfs_acl nfs lockd grace fscache sunrpc ipt_REJECT nf_reject_ipv4 xt_conntrack xt_mark xt_nat xt_tcpudp nf_log_ipv4 nf_log_common xt_LOG iptable_mangle iptable_filter lm85 hwmon_vid pl23
03 dm_snapshot dm_bufio iptable_nat ip_tables nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_conntrack_ftp ipt_MASQUERADE nf_nat_masquerade_ipv4 nf_nat nf_conntrack x_tables sg st snd_pcm_oss snd_mixer_oss bcache kvm_intel kvm irqbypass snd_cmipci eeepc_wmi snd_mpu401_uart snd_hda_codec_realtek snd_hda_codec_generic snd_opl3_lib gameport snd_hda_intel snd_hda_codec snd_hda_core asus_wmi snd_rawmidi sparse_keymap snd_seq_device snd_hwdep i2c_i801 snd_pcm rc_ati_x10 snd_timer rfkill snd tpm_infineon i2c_smbus soundcore asix ati_remote input_leds pcspkr tpm_tis batte[12062.414098] CPU: 7 PID: 0 Comm: swapper/7 Tainted: G U 4.8.8-amd64-volpreempt-sysrq-20161108 #7
[12062.414099] Hardware name: System manufacturer System Product Name/P8H67-M PRO, BIOS 3904 04/27/2013
[12062.414099] task: ffff9ee0c2fa4180 task.stack: ffff9ee0c2fa8000
[12062.414100] RIP: 0010:[<ffffffffbb57a128>] [<ffffffffbb57a128>] cpuidle_enter_state+0x119/0x171
[12062.414100] RSP: 0000:ffff9ee0c2fabea0 EFLAGS: 00000246
[12062.414101] RAX: ffff9ee0de3d90c0 RBX: 0000000000000004 RCX: 000000000000001f
[12062.414101] RDX: 0000000000000000 RSI: 0000000000000007 RDI: 0000000000000000
[12062.414101] RBP: ffff9ee0c2fabed0 R08: 0000000000000f92 R09: 0000000000000f42
[12062.414102] R10: ffff9ee0c2fabe50 R11: 071c71c71c71c71c R12: ffffe047bfdcb200
[12062.414102] R13: 00000af626899577 R14: 0000000000000004 R15: 00000af6264cc557
[12062.414103] FS: 0000000000000000(0000) GS:ffff9ee0de3c0000(0000) knlGS:0000000000000000
[12062.414103] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[12062.414103] CR2: 000000000898b000 CR3: 000000045cc06000 CR4: 00000000001406e0
[12062.414104] Stack:
[12062.414104] 0000000000000f40 ffffe047bfdcb200 ffffffffbbccc060 ffff9ee0c2fac000
[12062.414105] ffff9ee0c2fa8000 ffff9ee0c2fac000 ffff9ee0c2fabee0 ffffffffbb57a1ac
[12062.414105] ffff9ee0c2fabf30 ffffffffbb09238d ffff9ee0c2fa8000 0000000700000004
[12062.414105] Call Trace:
[12062.414106] [<ffffffffbb57a1ac>] cpuidle_enter+0x17/0x19
[12062.414106] [<ffffffffbb09238d>] cpu_startup_entry+0x210/0x28b
[12062.414107] [<ffffffffbb03de22>] start_secondary+0x13e/0x140
[12062.414107] Code: 00 00 00 48 c7 c7 cd ae b2 bb c6 05 4b 8e 7a 00 01 e8 17 6c ae ff fa 66 0f 1f 44 00 00 31 ff e8 75 60 b4 ff fb 66 0f 1f 44 00 00 <4c> 89 e8 b9 e8 03 00 00 4c 29 f8 48 99 48 f7 f9 ba ff ff ff 7f
[12062.414108] Kernel panic - not syncing: Hard LOCKUP
[12062.414108] CPU: 7 PID: 0 Comm: swapper/7 Tainted: G U 4.8.8-amd64-volpreempt-sysrq-20161108 #7
[12062.414109] Hardware name: System manufacturer System Product Name/P8H67-M PRO, BIOS 3904 04/27/2013
[12062.414109] 0000000000000000 ffff9ee0de3c5b00 ffffffffbb36a05b ffff9ee0c2fabd00
[12062.414109] ffffffffbbab1d54 ffff9ee0de3c5b80 ffffffffbb12d1f0 ffff9ee000000010
[12062.414110] ffff9ee0de3c5b90 ffff9ee0de3c5b28 ffffffffbbb37748 ffffffffbbab1575
[12062.414110] Call Trace:
[12062.414111] <NMI> [<ffffffffbb36a05b>] dump_stack+0x61/0x7d
[12062.414111] [<ffffffffbb12d1f0>] panic+0xd8/0x21a
[12062.414111] [<ffffffffbb060a9b>] nmi_panic+0x2c/0x37
[12062.414112] [<ffffffffbb0ea3c7>] watchdog_overflow_callback+0xc1/0xcc
[12062.414112] [<ffffffffbb1240b1>] __perf_ev
--
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
Microsoft is to operating systems ....
.... what McDonalds is to gourmet cooking
Home page: http://marc.merlins.org/ | PGP 1024R/763BE901
^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: 4.8.8, bcache deadlock and hard lockup
2016-11-18 18:49 ` Marc MERLIN
@ 2016-11-20 21:13 ` Coly Li
2016-11-20 21:26 ` Marc MERLIN
` (2 more replies)
0 siblings, 3 replies; 20+ messages in thread
From: Coly Li @ 2016-11-20 21:13 UTC (permalink / raw)
To: Marc MERLIN, linux-bcache
Hi Marc,
Could you please give me an exact procedure that how the environment is
configured, and how the commands are executed ? I will try them on my
machine to reproduce it.
Thanks in advance.
Coly
在 2016/11/19 上午2:49, Marc MERLIN 写道:
> On Fri, Nov 18, 2016 at 08:46:43AM -0800, Marc MERLIN wrote:
>> I created a new array as bcache and tried to copy a bunch of data to it.
>> A few hours later, the kenrel died with
>> [36403.763610] BUG: workqueue lockup - pool cpus=5 node=0 flags=0x0 nice=0 stuck for 31s!
>> [36403.789660] BUG: workqueue lockup - pool cpus=7 node=0 flags=0x0 nice=0 stuck for 31s!
>> [36403.815675] Showing busy workqueues and worker pools:
>
> Tried again, same bug, same crash, this time I didn't edit it to show
> the high level of recursion on
> in-flight: 21316:bch_data_insert_keys [bcache]
> delayed: bch_data_insert_keys [bcache]
> delayed: kcryptd_crypt
>
> [12044.160812] BUG: workqueue lockup - pool cpus=1 node=0 flags=0x0 nice=0 stuck for 35s!
> [12044.186217] Showing busy workqueues and worker pools:
> [12044.203532] workqueue events: flags=0x0
> [12044.216455] pwq 14: cpus=7 node=0 flags=0x0 nice=0 active=1/256
> [12044.236998] in-flight: 26009:write_dirty_finish [bcache]
> [12044.256155] pwq 10: cpus=5 node=0 flags=0x0 nice=0 active=2/256
> [12044.276665] in-flight: 3368:write_dirty_finish [bcache]
> [12044.295473] pending: cache_reap
> [12044.308090] pwq 8: cpus=4 node=0 flags=0x0 nice=0 active=1/256
> [12044.327592] pending: cache_reap
> [12044.339434] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256
> [12044.358914] in-flight: 12086:write_dirty_finish [bcache]
> [12044.377223] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=6/256
> [12044.396647] pending: vmpressure_work_fn, journal_write_work [bcache], mei_timer, vmstat_shepherd, cache_reap, btree_node_write_work [bcache]
> [12044.438295] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=10/256
> [12044.457969] in-flight: 21334:write_dirty_finish [bcache], 21314:write_dirty_finish [bcache]
> [12044.485388] pending: write_dirty [bcache], kernfs_notify_workfn, usb_serial_port_work [usbserial], cache_reap, drm_fb_helper_dirty_work, write_dirty_finish [bcache], update_writeback_rate [bcache], usb_serial_port_work [usbserial]
> [12044.550534] workqueue writeback: flags=0x4e
> [12044.564304] pwq 16: cpus=0-7 flags=0x4 nice=0 active=4/256
> [12044.582607] in-flight: 28522:wb_workfn wb_workfn, 16395:wb_workfn wb_workfn
> [12044.605995] workqueue btrfs-delalloc: flags=0xe
> [12044.620772] pwq 16: cpus=0-7 flags=0x4 nice=0 active=1/8
> [12044.638508] in-flight: 12761:btrfs_delalloc_helper
> [12044.655185] workqueue btrfs-endio-write: flags=0xe
> [12044.670699] pwq 16: cpus=0-7 flags=0x4 nice=0 active=8/8
> [12044.688370] in-flight: 16332:btrfs_endio_write_helper, 6306:btrfs_endio_write_helper, 8055:btrfs_endio_write_helper, 12765:btrfs_endio_write_helper, 28520:btrfs_endio_write_helper, 15209:btrfs_endio_write_helper, 28523:btrfs_endio_write_helper, 31016:btrfs_endio_write_helper
> [12044.766023] workqueue btrfs-extent-refs: flags=0xe
> [12044.782065] pwq 16: cpus=0-7 flags=0x4 nice=0 active=1/1
> [12044.800798] in-flight: 2308:btrfs_extent_refs_helper
> [12044.818317] delayed: btrfs_extent_refs_helper, btrfs_extent_refs_helper, btrfs_extent_refs_helper, btrfs_extent_refs_helper, btrfs_extent_refs_helper, btrfs_extent_refs_helper
> [12044.869635] workqueue bcache: flags=0x8
> [12044.882691] pwq 14: cpus=7 node=0 flags=0x0 nice=0 active=25/256 MAYDAY
> [12044.904731] in-flight: 25983:bch_data_insert_keys [bcache], 12147:bch_data_insert_keys [bcache], 12149:bch_data_insert_keys [bcache], 26002:bch_data_insert_keys [bcache], 26008:bch_data_insert_keys [bcache], 25980:bch_data_insert_keys [bcache], 12146:bch_data_insert_keys [bcache], 25984:bch_data_insert_keys [bcache], 25982:bch_data_insert_keys [bcache], 26000:bch_data_insert_keys [bcache], 26003:bch_data_insert_keys [bcache], 25985:bch_data_insert_keys [bcache], 26006:bch_data_insert_keys [bcache], 25978:bch_data_insert_keys [bcache], 25979:bch_data_insert_keys [bcache], 26007:bch_data_insert_keys [bcache], 26014:bch_data_insert_keys [bcache], 25986:bch_data_insert_keys [bcache], 25987:bch_data_insert_keys [bcache], 2477(RESCUER):bch_data_insert_keys [bcache] bch_data_insert_keys [bcache] bch_data_insert_keys [bcache], 25981:bch_data_insert_keys [bcache], 26005:bch_data_insert_keys [bcache], 26004:bch_data_insert_keys [bcache]
> [12045.165297] pwq 10: cpus=5 node=0 flags=0x0 nice=0 active=168/256 MAYDAY
> [12045.187565] in-flight: 23666:bch_data_insert_keys [bcache], 10647:bch_data_insert_keys [bcache], 23668:bch_data_insert_keys [bcache], 12503:bch_data_insert_keys [bcache], 12484:bch_data_insert_keys [bcache], 12486:bch_data_insert_keys [bcache], 12485:bch_data_insert_keys [bcache], 3398:bch_data_insert_keys [bcache], 23675:bch_data_insert_keys [bcache], 23670:bch_data_insert_keys [bcache], 12483:bch_data_insert_keys [bcache], 23673:bch_data_insert_keys [bcache], 23667:bch_data_insert_keys [bcache], 23664:bch_data_insert_keys [bcache], 12518:bch_data_insert_keys [bcache], 23671:bch_data_insert_keys [bcache], 10645:bch_data_insert_keys [bcache], 12491:bch_data_insert_keys [bcache], 23669:bch_data_insert_keys [bcache], 23674:bch_data_insert_keys [bcache], 3369:bch_data_insert_keys [bcache], 23663:bch_data_insert_keys [bcache], 23672:bch_data_insert_keys [bcache], 23665:bch_data_insert_keys [bcache]
> [12045.439919] pending: bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache]
> [12046.675915] pwq 8: cpus=4 node=0 flags=0x0 nice=0 active=11/256
> [12046.695813] in-flight: 13421:bch_data_insert_keys [bcache], 13418:bch_data_insert_keys [bcache], 13416:bch_data_insert_keys [bcache], 13413:bch_data_insert_keys [bcache], 13420:bch_data_insert_keys [bcache], 13417:bch_data_insert_keys [bcache], 22449:bch_data_insert_keys [bcache], 13414:bch_data_insert_keys [bcache], 13415:bch_data_insert_keys [bcache], 13422:bch_data_insert_keys [bcache], 13419:bch_data_insert_keys [bcache]
> [12046.817671] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=18/256 MAYDAY
> [12046.839377] in-flight: 17775:bch_data_insert_keys [bcache], 25993:bch_data_insert_keys [bcache], 25992:bch_data_insert_keys [bcache], 26024:bch_data_insert_keys [bcache], 26022:bch_data_insert_keys [bcache], 26020:bch_data_insert_keys [bcache], 17774:bch_data_insert_keys [bcache], 26001:bch_data_insert_keys [bcache], 25990:bch_data_insert_keys [bcache], 25999:bch_data_insert_keys [bcache], 26031:bch_data_insert_keys [bcache], 26023:bch_data_insert_keys [bcache], 26032:bch_data_insert_keys [bcache], 25991:bch_data_insert_keys [bcache], 25994:bch_data_insert_keys [bcache], 26026:bch_data_insert_keys [bcache], 17772:bch_data_insert_keys [bcache], 26028:bch_data_insert_keys [bcache]
> [12047.041497] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256
> [12047.062667] in-flight: 12081:bch_data_insert_keys [bcache], 12084:bch_data_insert_keys [bcache], 12082:bch_data_insert_keys [bcache], 12085:bch_data_insert_keys [bcache]
> [12047.114197] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=44/256 MAYDAY
> [12047.136619] in-flight: 12627:bch_data_insert_keys [bcache], 12628:bch_data_insert_keys [bcache], 12626:bch_data_insert_keys [bcache], 12624:bch_data_insert_keys [bcache], 12625:bch_data_insert_keys [bcache], 12185:bch_data_insert_keys [bcache]
> [12047.206924] pending: bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache]
> [12047.538640] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=256/256 MAYDAY
> [12047.560515] in-flight: 21316:bch_data_insert_keys [bcache], 21324:bch_data_insert_keys [bcache], 21317:bch_data_insert_keys [bcache], 21318:bch_data_insert_keys [bcache], 12395:bch_data_insert_keys [bcache], 21319:bch_data_insert_keys [bcache], 21331:bch_data_insert_keys [bcache], 21315:bch_data_insert_keys [bcache], 21320:bch_data_insert_keys [bcache], 21322:bch_data_insert_keys [bcache], 12387:bch_data_insert_keys [bcache], 21321:bch_data_insert_keys [bcache], 21333:bch_data_insert_keys [bcache], 12391:bch_data_insert_keys [bcache], 12388:bch_data_insert_keys [bcache], 21332:bch_data_insert_keys [bcache], 21328:bch_data_insert_keys [bcache], 21335:bch_data_insert_keys [bcache], 21323:bch_data_insert_keys [bcache], 12394:bch_data_insert_keys [bcache], 21325:bch_data_insert_keys [bcache], 12389:bch_data_insert_keys [bcache], 21330:bch_data_insert_keys [bcache], 12390:bch_data_insert_keys [bcache], 21326:bch_data_insert_keys [bcache], 21329:bch_data_insert_keys [bcache], 21327:bch_data_insert_keys [bcache], 21337:bch_data_insert_keys [bcache], 12393:bch_data_insert_keys [bcache], 12392:bch_data_insert_keys [bcache]
> [12047.870547] pending: bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache]
> [12049.838960] delayed: bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache]
> [12052.109659] workqueue kcryptd: flags=0x2a
> [12052.124689] pwq 16: cpus=0-7 flags=0x4 nice=0 active=8/8
> [12052.143921] in-flight: 27452:kcryptd_crypt [dm_crypt], 2331:kcryptd_crypt [dm_crypt], 24384:kcryptd_crypt [dm_crypt], 11840:kcryptd_crypt [dm_crypt], 27434:kcryptd_crypt [dm_crypt], 27437:kcryptd_crypt [dm_crypt], 24101:kcryptd_crypt [dm_crypt], 31015:kcryptd_crypt [dm_crypt]
> [12052.224929] delayed: kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt][12062.414098] NMI watchdog: Watchdog detected hard LOCKUP on cpu 7dModules linked in: veth ip6table_filter ip6_tables ebtable_nat ebtables ppdev lp xt_addrtype br_netfilter bridge stp llc tun autofs4 softdog binfmt_misc ftdi_sio nfsd auth_rpcgss nfs_acl nfs lockd grace fscache sunrpc ipt_REJECT nf_reject_ipv4 xt_conntrack xt_mark xt_nat xt_tcpudp nf_log_ipv4 nf_log_common xt_LOG iptable_mangle iptable_filter lm85 hwmon_vid pl2303 dm_snapshot dm_bufio iptable_nat ip_tables nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_conntrack_ftp ipt_MASQUERADE nf_nat_masquerade_ipv4 nf_nat nf_conntrack x_tables sg st snd_pcm_oss snd_mixer_oss bcache kvm_intel kvm irqbypass snd_cmipci eeepc_wmi snd_mpu401_uart snd_hda_codec_realtek snd_hda_codec_generic snd_opl3_lib gameport snd_hda_intel snd_hda_codec snd_hda_core asus_wmi snd_rawmidi sparse_keymap snd_seq_device snd_hwdep i2c_i801 snd_pcm rc_ati_x10 snd_timer rfkill snd tpm_infineon i2c_smbus soundcore asix ati_remote input_leds pcspkr tpm_tis batte[12062.414098] CPU: 7 PID: 0 Comm: swapper/7 Tainted: G U 4.8.8-amd64-volpreempt-sysrq-20161108 #7
> [12062.414099] Hardware name: System manufacturer System Product Name/P8H67-M PRO, BIOS 3904 04/27/2013
> [12062.414099] task: ffff9ee0c2fa4180 task.stack: ffff9ee0c2fa8000
> [12062.414100] RIP: 0010:[<ffffffffbb57a128>] [<ffffffffbb57a128>] cpuidle_enter_state+0x119/0x171
> [12062.414100] RSP: 0000:ffff9ee0c2fabea0 EFLAGS: 00000246
> [12062.414101] RAX: ffff9ee0de3d90c0 RBX: 0000000000000004 RCX: 000000000000001f
> [12062.414101] RDX: 0000000000000000 RSI: 0000000000000007 RDI: 0000000000000000
> [12062.414101] RBP: ffff9ee0c2fabed0 R08: 0000000000000f92 R09: 0000000000000f42
> [12062.414102] R10: ffff9ee0c2fabe50 R11: 071c71c71c71c71c R12: ffffe047bfdcb200
> [12062.414102] R13: 00000af626899577 R14: 0000000000000004 R15: 00000af6264cc557
> [12062.414103] FS: 0000000000000000(0000) GS:ffff9ee0de3c0000(0000) knlGS:0000000000000000
> [12062.414103] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> [12062.414103] CR2: 000000000898b000 CR3: 000000045cc06000 CR4: 00000000001406e0
> [12062.414104] Stack:
> [12062.414104] 0000000000000f40 ffffe047bfdcb200 ffffffffbbccc060 ffff9ee0c2fac000
> [12062.414105] ffff9ee0c2fa8000 ffff9ee0c2fac000 ffff9ee0c2fabee0 ffffffffbb57a1ac
> [12062.414105] ffff9ee0c2fabf30 ffffffffbb09238d ffff9ee0c2fa8000 0000000700000004
> [12062.414105] Call Trace:
> [12062.414106] [<ffffffffbb57a1ac>] cpuidle_enter+0x17/0x19
> [12062.414106] [<ffffffffbb09238d>] cpu_startup_entry+0x210/0x28b
> [12062.414107] [<ffffffffbb03de22>] start_secondary+0x13e/0x140
> [12062.414107] Code: 00 00 00 48 c7 c7 cd ae b2 bb c6 05 4b 8e 7a 00 01 e8 17 6c ae ff fa 66 0f 1f 44 00 00 31 ff e8 75 60 b4 ff fb 66 0f 1f 44 00 00 <4c> 89 e8 b9 e8 03 00 00 4c 29 f8 48 99 48 f7 f9 ba ff ff ff 7f
> [12062.414108] Kernel panic - not syncing: Hard LOCKUP
> [12062.414108] CPU: 7 PID: 0 Comm: swapper/7 Tainted: G U 4.8.8-amd64-volpreempt-sysrq-20161108 #7
> [12062.414109] Hardware name: System manufacturer System Product Name/P8H67-M PRO, BIOS 3904 04/27/2013
> [12062.414109] 0000000000000000 ffff9ee0de3c5b00 ffffffffbb36a05b ffff9ee0c2fabd00
> [12062.414109] ffffffffbbab1d54 ffff9ee0de3c5b80 ffffffffbb12d1f0 ffff9ee000000010
> [12062.414110] ffff9ee0de3c5b90 ffff9ee0de3c5b28 ffffffffbbb37748 ffffffffbbab1575
> [12062.414110] Call Trace:
> [12062.414111] <NMI> [<ffffffffbb36a05b>] dump_stack+0x61/0x7d
> [12062.414111] [<ffffffffbb12d1f0>] panic+0xd8/0x21a
> [12062.414111] [<ffffffffbb060a9b>] nmi_panic+0x2c/0x37
> [12062.414112] [<ffffffffbb0ea3c7>] watchdog_overflow_callback+0xc1/0xcc
> [12062.414112] [<ffffffffbb1240b1>] __perf_ev
>
--
Coly Li
^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: 4.8.8, bcache deadlock and hard lockup
2016-11-20 21:13 ` Coly Li
@ 2016-11-20 21:26 ` Marc MERLIN
2016-11-21 0:04 ` Marc MERLIN
2016-11-29 20:59 ` Eric Wheeler
2 siblings, 0 replies; 20+ messages in thread
From: Marc MERLIN @ 2016-11-20 21:26 UTC (permalink / raw)
To: Coly Li; +Cc: linux-bcache
On Mon, Nov 21, 2016 at 05:13:41AM +0800, Coly Li wrote:
> Hi Marc,
>
> Could you please give me an exact procedure that how the environment is
> configured, and how the commands are executed ? I will try them on my
> machine to reproduce it.
Thanks for offering.
So I did more testing, and while bcache piled up and died in the end,
the problem seems to be a VM issue and bcache might not handle some
memory case quite right, or may simply do everything right and be in a
losing battle with a VM problem.
Current details on
https://plus.google.com/u/0/+MarcMERLIN/posts/A3FrLVo3kc6
Right now I'm testing 3.19rc5 to see if it gets around this VM problem,
but I'm fairly sure at this point that bcache is not the issue.
Will report back.
Marc
--
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
Microsoft is to operating systems ....
.... what McDonalds is to gourmet cooking
Home page: http://marc.merlins.org/ | PGP 1024R/763BE901
^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: 4.8.8, bcache deadlock and hard lockup
2016-11-20 21:13 ` Coly Li
2016-11-20 21:26 ` Marc MERLIN
@ 2016-11-21 0:04 ` Marc MERLIN
2016-11-29 20:59 ` Eric Wheeler
2 siblings, 0 replies; 20+ messages in thread
From: Marc MERLIN @ 2016-11-21 0:04 UTC (permalink / raw)
To: Coly Li; +Cc: linux-bcache
On Mon, Nov 21, 2016 at 05:13:41AM +0800, Coly Li wrote:
> Hi Marc,
>
> Could you please give me an exact procedure that how the environment is
> configured, and how the commands are executed ? I will try them on my
> machine to reproduce it.
Well, let's see:
source and destination look like this:
mdadm raid 5 \
ssd bcache ---- dmcrypt - btrfs
Then I do a 10TB btrfs send from dev1 to dev2
The exact commands, some I typed a long time ago. This is live data, not
a test environment I just spin up for testing.
That said, is that what you were asking about?
But as I said, right now I removed the caching device from dev2, so
bcache is still there but doing nothing, and I'm still getting kernel
OOM problems as per http://pastebin.com/2LUicF3k
I'm currently testing 4.19rc5 to see if the VM changes help my case.
Marc
--
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
Microsoft is to operating systems ....
.... what McDonalds is to gourmet cooking
Home page: http://marc.merlins.org/ | PGP 1024R/763BE901
^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: 4.8.8, bcache deadlock and hard lockup
2016-11-20 21:13 ` Coly Li
2016-11-20 21:26 ` Marc MERLIN
2016-11-21 0:04 ` Marc MERLIN
@ 2016-11-29 20:59 ` Eric Wheeler
2016-11-30 16:46 ` Marc MERLIN
2 siblings, 1 reply; 20+ messages in thread
From: Eric Wheeler @ 2016-11-29 20:59 UTC (permalink / raw)
To: Coly Li; +Cc: Marc MERLIN, linux-bcache
[-- Attachment #1: Type: TEXT/PLAIN, Size: 72313 bytes --]
On Mon, 27 Nov 2016, Coly Li wrote:
>
> Yes, too many work queues... I guess the locking might be caused by some
> very obscure reference of closure code. I cannot have any clue if I
> cannot find a stable procedure to reproduce this issue.
>
> Hmm, if there is a tool to clone all the meta data of the back end cache
> and whole cached device, there might be a method to replay the oops much
> easier.
>
> Eric, do you have any hint ?
Note that the backing device doesn't have any metadata, just a superblock.
You can easily dd that off onto some other volume without transferring the
data. By default, data starts at 8k, or whatever you used in `make-bcache
-w`.
For the caching device, there's not a metadata dump and restore tool that
I'm aware of---however I have had success dd'ing a cache and superblock
volume onto an lvthin volume so that I can snapshot the broken version
while testing fixes against a snapshot. This way I could revert to the
broken form if during testing I found a solution and could prove that it
was reproducible.
--
Eric Wheeler
On Mon, 21 Nov 2016, Coly Li wrote:
> Hi Marc,
>
> Could you please give me an exact procedure that how the environment is
> configured, and how the commands are executed ? I will try them on my
> machine to reproduce it.
>
> Thanks in advance.
>
> Coly
>
> 在 2016/11/19 上午2:49, Marc MERLIN 写道:
> > On Fri, Nov 18, 2016 at 08:46:43AM -0800, Marc MERLIN wrote:
> >> I created a new array as bcache and tried to copy a bunch of data to it.
> >> A few hours later, the kenrel died with
> >> [36403.763610] BUG: workqueue lockup - pool cpus=5 node=0 flags=0x0 nice=0 stuck for 31s!
> >> [36403.789660] BUG: workqueue lockup - pool cpus=7 node=0 flags=0x0 nice=0 stuck for 31s!
> >> [36403.815675] Showing busy workqueues and worker pools:
> >
> > Tried again, same bug, same crash, this time I didn't edit it to show
> > the high level of recursion on
> > in-flight: 21316:bch_data_insert_keys [bcache]
> > delayed: bch_data_insert_keys [bcache]
> > delayed: kcryptd_crypt
> >
> > [12044.160812] BUG: workqueue lockup - pool cpus=1 node=0 flags=0x0 nice=0 stuck for 35s!
> > [12044.186217] Showing busy workqueues and worker pools:
> > [12044.203532] workqueue events: flags=0x0
> > [12044.216455] pwq 14: cpus=7 node=0 flags=0x0 nice=0 active=1/256
> > [12044.236998] in-flight: 26009:write_dirty_finish [bcache]
> > [12044.256155] pwq 10: cpus=5 node=0 flags=0x0 nice=0 active=2/256
> > [12044.276665] in-flight: 3368:write_dirty_finish [bcache]
> > [12044.295473] pending: cache_reap
> > [12044.308090] pwq 8: cpus=4 node=0 flags=0x0 nice=0 active=1/256
> > [12044.327592] pending: cache_reap
> > [12044.339434] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256
> > [12044.358914] in-flight: 12086:write_dirty_finish [bcache]
> > [12044.377223] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=6/256
> > [12044.396647] pending: vmpressure_work_fn, journal_write_work [bcache], mei_timer, vmstat_shepherd, cache_reap, btree_node_write_work [bcache]
> > [12044.438295] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=10/256
> > [12044.457969] in-flight: 21334:write_dirty_finish [bcache], 21314:write_dirty_finish [bcache]
> > [12044.485388] pending: write_dirty [bcache], kernfs_notify_workfn, usb_serial_port_work [usbserial], cache_reap, drm_fb_helper_dirty_work, write_dirty_finish [bcache], update_writeback_rate [bcache], usb_serial_port_work [usbserial]
> > [12044.550534] workqueue writeback: flags=0x4e
> > [12044.564304] pwq 16: cpus=0-7 flags=0x4 nice=0 active=4/256
> > [12044.582607] in-flight: 28522:wb_workfn wb_workfn, 16395:wb_workfn wb_workfn
> > [12044.605995] workqueue btrfs-delalloc: flags=0xe
> > [12044.620772] pwq 16: cpus=0-7 flags=0x4 nice=0 active=1/8
> > [12044.638508] in-flight: 12761:btrfs_delalloc_helper
> > [12044.655185] workqueue btrfs-endio-write: flags=0xe
> > [12044.670699] pwq 16: cpus=0-7 flags=0x4 nice=0 active=8/8
> > [12044.688370] in-flight: 16332:btrfs_endio_write_helper, 6306:btrfs_endio_write_helper, 8055:btrfs_endio_write_helper, 12765:btrfs_endio_write_helper, 28520:btrfs_endio_write_helper, 15209:btrfs_endio_write_helper, 28523:btrfs_endio_write_helper, 31016:btrfs_endio_write_helper
> > [12044.766023] workqueue btrfs-extent-refs: flags=0xe
> > [12044.782065] pwq 16: cpus=0-7 flags=0x4 nice=0 active=1/1
> > [12044.800798] in-flight: 2308:btrfs_extent_refs_helper
> > [12044.818317] delayed: btrfs_extent_refs_helper, btrfs_extent_refs_helper, btrfs_extent_refs_helper, btrfs_extent_refs_helper, btrfs_extent_refs_helper, btrfs_extent_refs_helper
> > [12044.869635] workqueue bcache: flags=0x8
> > [12044.882691] pwq 14: cpus=7 node=0 flags=0x0 nice=0 active=25/256 MAYDAY
> > [12044.904731] in-flight: 25983:bch_data_insert_keys [bcache], 12147:bch_data_insert_keys [bcache], 12149:bch_data_insert_keys [bcache], 26002:bch_data_insert_keys [bcache], 26008:bch_data_insert_keys [bcache], 25980:bch_data_insert_keys [bcache], 12146:bch_data_insert_keys [bcache], 25984:bch_data_insert_keys [bcache], 25982:bch_data_insert_keys [bcache], 26000:bch_data_insert_keys [bcache], 26003:bch_data_insert_keys [bcache], 25985:bch_data_insert_keys [bcache], 26006:bch_data_insert_keys [bcache], 25978:bch_data_insert_keys [bcache], 25979:bch_data_insert_keys [bcache], 26007:bch_data_insert_keys [bcache], 26014:bch_data_insert_keys [bcache], 25986:bch_data_insert_keys [bcache], 25987:bch_data_insert_keys [bcache], 2477(RESCUER):bch_data_insert_keys [bcache] bch_data_insert_key
s [bcache] bch_data_insert_keys [bcache], 25981:bch_data_insert_keys [bcache], 26005:bch_data_insert_keys [bcache], 26004:bch_data_insert_keys [bcache]
> > [12045.165297] pwq 10: cpus=5 node=0 flags=0x0 nice=0 active=168/256 MAYDAY
> > [12045.187565] in-flight: 23666:bch_data_insert_keys [bcache], 10647:bch_data_insert_keys [bcache], 23668:bch_data_insert_keys [bcache], 12503:bch_data_insert_keys [bcache], 12484:bch_data_insert_keys [bcache], 12486:bch_data_insert_keys [bcache], 12485:bch_data_insert_keys [bcache], 3398:bch_data_insert_keys [bcache], 23675:bch_data_insert_keys [bcache], 23670:bch_data_insert_keys [bcache], 12483:bch_data_insert_keys [bcache], 23673:bch_data_insert_keys [bcache], 23667:bch_data_insert_keys [bcache], 23664:bch_data_insert_keys [bcache], 12518:bch_data_insert_keys [bcache], 23671:bch_data_insert_keys [bcache], 10645:bch_data_insert_keys [bcache], 12491:bch_data_insert_keys [bcache], 23669:bch_data_insert_keys [bcache], 23674:bch_data_insert_keys [bcache], 3369:bch_data_insert_keys [
bcache], 23663:bch_data_insert_keys [bcache], 23672:bch_data_insert_keys [bcache], 23665:bch_data_insert_keys [bcache]
> > [12045.439919] pending: bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bc
ache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache
], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys
[bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bca
che], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_ke
ys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [
bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert
_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_key
s [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache]
> > [12046.675915] pwq 8: cpus=4 node=0 flags=0x0 nice=0 active=11/256
> > [12046.695813] in-flight: 13421:bch_data_insert_keys [bcache], 13418:bch_data_insert_keys [bcache], 13416:bch_data_insert_keys [bcache], 13413:bch_data_insert_keys [bcache], 13420:bch_data_insert_keys [bcache], 13417:bch_data_insert_keys [bcache], 22449:bch_data_insert_keys [bcache], 13414:bch_data_insert_keys [bcache], 13415:bch_data_insert_keys [bcache], 13422:bch_data_insert_keys [bcache], 13419:bch_data_insert_keys [bcache]
> > [12046.817671] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=18/256 MAYDAY
> > [12046.839377] in-flight: 17775:bch_data_insert_keys [bcache], 25993:bch_data_insert_keys [bcache], 25992:bch_data_insert_keys [bcache], 26024:bch_data_insert_keys [bcache], 26022:bch_data_insert_keys [bcache], 26020:bch_data_insert_keys [bcache], 17774:bch_data_insert_keys [bcache], 26001:bch_data_insert_keys [bcache], 25990:bch_data_insert_keys [bcache], 25999:bch_data_insert_keys [bcache], 26031:bch_data_insert_keys [bcache], 26023:bch_data_insert_keys [bcache], 26032:bch_data_insert_keys [bcache], 25991:bch_data_insert_keys [bcache], 25994:bch_data_insert_keys [bcache], 26026:bch_data_insert_keys [bcache], 17772:bch_data_insert_keys [bcache], 26028:bch_data_insert_keys [bcache]
> > [12047.041497] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=4/256
> > [12047.062667] in-flight: 12081:bch_data_insert_keys [bcache], 12084:bch_data_insert_keys [bcache], 12082:bch_data_insert_keys [bcache], 12085:bch_data_insert_keys [bcache]
> > [12047.114197] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=44/256 MAYDAY
> > [12047.136619] in-flight: 12627:bch_data_insert_keys [bcache], 12628:bch_data_insert_keys [bcache], 12626:bch_data_insert_keys [bcache], 12624:bch_data_insert_keys [bcache], 12625:bch_data_insert_keys [bcache], 12185:bch_data_insert_keys [bcache]
> > [12047.206924] pending: bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bc
ache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache
], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache]
> > [12047.538640] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=256/256 MAYDAY
> > [12047.560515] in-flight: 21316:bch_data_insert_keys [bcache], 21324:bch_data_insert_keys [bcache], 21317:bch_data_insert_keys [bcache], 21318:bch_data_insert_keys [bcache], 12395:bch_data_insert_keys [bcache], 21319:bch_data_insert_keys [bcache], 21331:bch_data_insert_keys [bcache], 21315:bch_data_insert_keys [bcache], 21320:bch_data_insert_keys [bcache], 21322:bch_data_insert_keys [bcache], 12387:bch_data_insert_keys [bcache], 21321:bch_data_insert_keys [bcache], 21333:bch_data_insert_keys [bcache], 12391:bch_data_insert_keys [bcache], 12388:bch_data_insert_keys [bcache], 21332:bch_data_insert_keys [bcache], 21328:bch_data_insert_keys [bcache], 21335:bch_data_insert_keys [bcache], 21323:bch_data_insert_keys [bcache], 12394:bch_data_insert_keys [bcache], 21325:bch_data_insert_keys
[bcache], 12389:bch_data_insert_keys [bcache], 21330:bch_data_insert_keys [bcache], 12390:bch_data_insert_keys [bcache], 21326:bch_data_insert_keys [bcache], 21329:bch_data_insert_keys [bca
che], 21327:bch_data_insert_keys [bcache], 21337:bch_data_insert_keys [bcache], 12393:bch_data_insert_keys [bcache], 12392:bch_data_insert_keys [bcache]
> > [12047.870547] pending: bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bc
ache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache
], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys
[bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bca
che], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_ke
ys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [
bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert
_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_key
s [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_ins
ert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_
keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_
insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_inse
rt_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_da
ta_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_i
nsert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache]
> > [12049.838960] delayed: bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bc
ache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache
], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys
[bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bca
che], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_ke
ys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [
bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert
_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_key
s [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_ins
ert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_
keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_
insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_inse
rt_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_da
ta_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_i
nsert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch
_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_dat
a_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache], bch_data_insert_keys [bcache]
> > [12052.109659] workqueue kcryptd: flags=0x2a
> > [12052.124689] pwq 16: cpus=0-7 flags=0x4 nice=0 active=8/8
> > [12052.143921] in-flight: 27452:kcryptd_crypt [dm_crypt], 2331:kcryptd_crypt [dm_crypt], 24384:kcryptd_crypt [dm_crypt], 11840:kcryptd_crypt [dm_crypt], 27434:kcryptd_crypt [dm_crypt], 27437:kcryptd_crypt [dm_crypt], 24101:kcryptd_crypt [dm_crypt], 31015:kcryptd_crypt [dm_crypt]
> > [12052.224929] delayed: kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt
[dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_cryp
t], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [
dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt
], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [d
m_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt]
, kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm
_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt],
kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_
crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt],
kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_c
rypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], k
cryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_cr
ypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kc
ryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_cry
pt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcr
yptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_cryp
t], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcry
ptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt
], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryp
td_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt]
, kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcrypt
d_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt],
kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd
_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt],
kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_
crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], k
cryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_c
rypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kc
ryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_cr
ypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcr
yptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_cry
pt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcry
ptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_cryp
t [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryp
td_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt
[dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcrypt
d_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt
[dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd
_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [
dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_
crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [d
m_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_c
rypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm
_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_cr
ypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_
crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_cry
pt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_c
rypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_cryp
t [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_cr
ypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt
[dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_cry
pt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt
[dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_cryp
t], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [
dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt
], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [d
m_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt]
, kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm
_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt],
kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_
crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt],
kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_c
rypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], k
cryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_cr
ypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kc
ryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_cry
pt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcr
yptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_cryp
t], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcry
ptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt
], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryp
td_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt]
, kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcrypt
d_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt], kcryptd_crypt [dm_crypt][12062.414098] NMI watchdog: Watchdog detected hard LOCKUP on cpu 7dModules linked in: veth ip6table_filter ip6_tables ebtable_nat ebtables ppdev lp xt_addrtype br_netfilter bridge stp llc tun autofs4 softdog binfmt_misc ftdi_sio nfsd auth_rpcgss nfs_acl nfs lockd grace fscache sunrpc ipt_REJECT nf_reject_ipv4 x
t_conntrack xt_mark xt_nat xt_tcpudp nf_log_ipv4 nf_log_common xt_LOG iptable_mangle iptable_filter lm85 hwmon_vid pl2303 dm_snapshot dm_bufio iptable_nat ip_tables nf_conntrack_ipv4 nf_defr
ag_ipv4 nf_nat_ipv4 nf_conntrack_ftp ipt_MASQUERADE nf_nat_masquerade_ipv4 nf_nat nf_conntrack x_tables sg st snd_pcm_oss snd_mixer_oss bcache kvm_intel kvm irqbypass snd_cmipci eeepc_wmi snd_mpu401_uart snd_hda_codec_realtek snd_hda_codec_generic snd_opl3_lib gameport snd_hda_intel snd_hda_codec snd_hda_core asus_wmi snd_rawmidi sparse_keymap snd_seq_device snd_hwdep i2c_i801 snd_pcm rc_ati_x10 snd_timer rfkill snd tpm_infineon i2c_smbus soundcore asix ati_remote input_leds pcspkr tpm_tis batte[12062.414098] CPU: 7 PID: 0 Comm: swapper/7 Tainted: G U 4.8.8-amd64-volpreempt-sysrq-20161108 #7
> > [12062.414099] Hardware name: System manufacturer System Product Name/P8H67-M PRO, BIOS 3904 04/27/2013
> > [12062.414099] task: ffff9ee0c2fa4180 task.stack: ffff9ee0c2fa8000
> > [12062.414100] RIP: 0010:[<ffffffffbb57a128>] [<ffffffffbb57a128>] cpuidle_enter_state+0x119/0x171
> > [12062.414100] RSP: 0000:ffff9ee0c2fabea0 EFLAGS: 00000246
> > [12062.414101] RAX: ffff9ee0de3d90c0 RBX: 0000000000000004 RCX: 000000000000001f
> > [12062.414101] RDX: 0000000000000000 RSI: 0000000000000007 RDI: 0000000000000000
> > [12062.414101] RBP: ffff9ee0c2fabed0 R08: 0000000000000f92 R09: 0000000000000f42
> > [12062.414102] R10: ffff9ee0c2fabe50 R11: 071c71c71c71c71c R12: ffffe047bfdcb200
> > [12062.414102] R13: 00000af626899577 R14: 0000000000000004 R15: 00000af6264cc557
> > [12062.414103] FS: 0000000000000000(0000) GS:ffff9ee0de3c0000(0000) knlGS:0000000000000000
> > [12062.414103] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> > [12062.414103] CR2: 000000000898b000 CR3: 000000045cc06000 CR4: 00000000001406e0
> > [12062.414104] Stack:
> > [12062.414104] 0000000000000f40 ffffe047bfdcb200 ffffffffbbccc060 ffff9ee0c2fac000
> > [12062.414105] ffff9ee0c2fa8000 ffff9ee0c2fac000 ffff9ee0c2fabee0 ffffffffbb57a1ac
> > [12062.414105] ffff9ee0c2fabf30 ffffffffbb09238d ffff9ee0c2fa8000 0000000700000004
> > [12062.414105] Call Trace:
> > [12062.414106] [<ffffffffbb57a1ac>] cpuidle_enter+0x17/0x19
> > [12062.414106] [<ffffffffbb09238d>] cpu_startup_entry+0x210/0x28b
> > [12062.414107] [<ffffffffbb03de22>] start_secondary+0x13e/0x140
> > [12062.414107] Code: 00 00 00 48 c7 c7 cd ae b2 bb c6 05 4b 8e 7a 00 01 e8 17 6c ae ff fa 66 0f 1f 44 00 00 31 ff e8 75 60 b4 ff fb 66 0f 1f 44 00 00 <4c> 89 e8 b9 e8 03 00 00 4c 29 f8 48 99 48 f7 f9 ba ff ff ff 7f
> > [12062.414108] Kernel panic - not syncing: Hard LOCKUP
> > [12062.414108] CPU: 7 PID: 0 Comm: swapper/7 Tainted: G U 4.8.8-amd64-volpreempt-sysrq-20161108 #7
> > [12062.414109] Hardware name: System manufacturer System Product Name/P8H67-M PRO, BIOS 3904 04/27/2013
> > [12062.414109] 0000000000000000 ffff9ee0de3c5b00 ffffffffbb36a05b ffff9ee0c2fabd00
> > [12062.414109] ffffffffbbab1d54 ffff9ee0de3c5b80 ffffffffbb12d1f0 ffff9ee000000010
> > [12062.414110] ffff9ee0de3c5b90 ffff9ee0de3c5b28 ffffffffbbb37748 ffffffffbbab1575
> > [12062.414110] Call Trace:
> > [12062.414111] <NMI> [<ffffffffbb36a05b>] dump_stack+0x61/0x7d
> > [12062.414111] [<ffffffffbb12d1f0>] panic+0xd8/0x21a
> > [12062.414111] [<ffffffffbb060a9b>] nmi_panic+0x2c/0x37
> > [12062.414112] [<ffffffffbb0ea3c7>] watchdog_overflow_callback+0xc1/0xcc
> > [12062.414112] [<ffffffffbb1240b1>] __perf_ev
> >
>
>
> --
> Coly Li
> N嫥叉靣笡y???氊b瞂???千v豝???藓{.n???壏???{眒茰呰Ф\x17洝塄}???财爖???j:+v墾???珣赙zZ+€???zf＂穐殘???啳嗃i???鄗???畐ア???櫒璀??????撷^[f
^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: 4.8.8, bcache deadlock and hard lockup
2016-11-29 20:59 ` Eric Wheeler
@ 2016-11-30 16:46 ` Marc MERLIN
2016-11-30 17:16 ` Marc MERLIN
` (2 more replies)
0 siblings, 3 replies; 20+ messages in thread
From: Marc MERLIN @ 2016-11-30 16:46 UTC (permalink / raw)
To: Eric Wheeler; +Cc: Coly Li, linux-bcache, Btrfs BTRFS
+btrfs mailing list, see below why
On Tue, Nov 29, 2016 at 12:59:44PM -0800, Eric Wheeler wrote:
> On Mon, 27 Nov 2016, Coly Li wrote:
> >
> > Yes, too many work queues... I guess the locking might be caused by some
> > very obscure reference of closure code. I cannot have any clue if I
> > cannot find a stable procedure to reproduce this issue.
> >
> > Hmm, if there is a tool to clone all the meta data of the back end cache
> > and whole cached device, there might be a method to replay the oops much
> > easier.
> >
> > Eric, do you have any hint ?
>
> Note that the backing device doesn't have any metadata, just a superblock.
> You can easily dd that off onto some other volume without transferring the
> data. By default, data starts at 8k, or whatever you used in `make-bcache
> -w`.
Ok, Linus helped me find a workaround for this problem:
https://lkml.org/lkml/2016/11/29/667
namely:
echo 2 > /proc/sys/vm/dirty_ratio
echo 1 > /proc/sys/vm/dirty_background_ratio
(it's a 24GB system, so the defaults of 20 and 10 were creating too many
requests in th buffers)
Note that this is only a workaround, not a fix.
When I did this and re tried my big copy again, I still got 100+ kernel
work queues, but apparently the underlying swraid5 was able to unblock
and satisfy the write requests before too many accumulated and crashed
the kernel.
I'm not a kernel coder, but seems to me that bcache needs a way to
throttle incoming requests if there are too many so that it does not end
up in a state where things blow up due to too many piled up requests.
You should be able to reproduce this by taking 5 spinning rust drives,
put raid5 on top, dmcrypt, bcache and hopefully any filesystem (although
I used btrfs) and send lots of requests.
Actually to be honest, the problems have mostly been happening when I do
btrfs scrub and btrfs send/receive which both generate I/O from within
the kernel instead of user space.
So here, btrfs may be a contributor to the problem too, but while btrfs
still trashes my system if I remove the caching device on bcache (and
with the default dirty ratio values), it doesn't crash the kernel.
I'll start another separate thread with the btrfs folks on how much
pressure is put on the system, but on your side it would be good to help
ensure that bcache doesn't crash the system altogether if too many
requests are allowed to pile up.
Thanks,
Marc
--
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
Microsoft is to operating systems ....
.... what McDonalds is to gourmet cooking
Home page: http://marc.merlins.org/ | PGP 1024R/763BE901
^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: 4.8.8, bcache deadlock and hard lockup
2016-11-30 16:46 ` Marc MERLIN
@ 2016-11-30 17:16 ` Marc MERLIN
2016-11-30 17:18 ` btrfs flooding the I/O subsystem and hanging the machine, with bcache cache turned off Marc MERLIN
2016-11-30 23:57 ` 4.8.8, bcache deadlock and hard lockup Eric Wheeler
2 siblings, 0 replies; 20+ messages in thread
From: Marc MERLIN @ 2016-11-30 17:16 UTC (permalink / raw)
To: Eric Wheeler, Linus Torvalds; +Cc: Coly Li, linux-bcache, Btrfs BTRFS
On Wed, Nov 30, 2016 at 08:46:46AM -0800, Marc MERLIN wrote:
> +btrfs mailing list, see below why
>
> On Tue, Nov 29, 2016 at 12:59:44PM -0800, Eric Wheeler wrote:
> > On Mon, 27 Nov 2016, Coly Li wrote:
> > >
> > > Yes, too many work queues... I guess the locking might be caused by some
> > > very obscure reference of closure code. I cannot have any clue if I
> > > cannot find a stable procedure to reproduce this issue.
> > >
> > > Hmm, if there is a tool to clone all the meta data of the back end cache
> > > and whole cached device, there might be a method to replay the oops much
> > > easier.
> > >
> > > Eric, do you have any hint ?
> >
> > Note that the backing device doesn't have any metadata, just a superblock.
> > You can easily dd that off onto some other volume without transferring the
> > data. By default, data starts at 8k, or whatever you used in `make-bcache
> > -w`.
>
> Ok, Linus helped me find a workaround for this problem:
> https://lkml.org/lkml/2016/11/29/667
> namely:
> echo 2 > /proc/sys/vm/dirty_ratio
> echo 1 > /proc/sys/vm/dirty_background_ratio
> (it's a 24GB system, so the defaults of 20 and 10 were creating too many
> requests in th buffers)
>
> Note that this is only a workaround, not a fix.
Actually, I'm even more worried about the general bcache situation when
caching is enabled. In the message above, Linus wrote:
"One situation where I've seen something like this happen is
(a) lots and lots of dirty data queued up
(b) horribly slow storage
(c) filesystem that ends up serializing on writeback under certain
circumstances
The usual case for (b) in the modern world is big SSD's that have bad
worst-case behavior (ie they may do gbps speeds when doing well, and
then they come to a screeching halt when their buffers fill up and
they have to do rewrites, and their gbps throughput drops to mbps or
lower).
Generally you only find that kind of really nasty SSD in the USB stick
world these days."
Well, come to think of it, this is _exactly_ what bcache will create, by
design. It'll swallow up a lot of IO cached to the SSD, until the SSD
buffers fill up and then things will hang while bcache struggles to
write it all to slower spinning rust storage.
Looks to me like bcache and dirty_ratio need to be synced somehow, or
things will fall over reliably.
What do you think?
Thanks,
Marc
> When I did this and re tried my big copy again, I still got 100+ kernel
> work queues, but apparently the underlying swraid5 was able to unblock
> and satisfy the write requests before too many accumulated and crashed
> the kernel.
>
> I'm not a kernel coder, but seems to me that bcache needs a way to
> throttle incoming requests if there are too many so that it does not end
> up in a state where things blow up due to too many piled up requests.
>
> You should be able to reproduce this by taking 5 spinning rust drives,
> put raid5 on top, dmcrypt, bcache and hopefully any filesystem (although
> I used btrfs) and send lots of requests.
> Actually to be honest, the problems have mostly been happening when I do
> btrfs scrub and btrfs send/receive which both generate I/O from within
> the kernel instead of user space.
> So here, btrfs may be a contributor to the problem too, but while btrfs
> still trashes my system if I remove the caching device on bcache (and
> with the default dirty ratio values), it doesn't crash the kernel.
>
> I'll start another separate thread with the btrfs folks on how much
> pressure is put on the system, but on your side it would be good to help
> ensure that bcache doesn't crash the system altogether if too many
> requests are allowed to pile up.
>
> Thanks,
> Marc
> --
> "A mouse is a device used to point at the xterm you want to type in" - A.S.R.
> Microsoft is to operating systems ....
> .... what McDonalds is to gourmet cooking
> Home page: http://marc.merlins.org/ | PGP 1024R/763BE901
--
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
Microsoft is to operating systems ....
.... what McDonalds is to gourmet cooking
Home page: http://marc.merlins.org/ | PGP 1024R/763BE901
^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: btrfs flooding the I/O subsystem and hanging the machine, with bcache cache turned off
2016-11-30 16:46 ` Marc MERLIN
2016-11-30 17:16 ` Marc MERLIN
@ 2016-11-30 17:18 ` Marc MERLIN
2016-11-30 18:00 ` Austin S. Hemmelgarn
2016-11-30 23:57 ` 4.8.8, bcache deadlock and hard lockup Eric Wheeler
2 siblings, 1 reply; 20+ messages in thread
From: Marc MERLIN @ 2016-11-30 17:18 UTC (permalink / raw)
To: Btrfs BTRFS
On Wed, Nov 30, 2016 at 08:46:46AM -0800, Marc MERLIN wrote:
> +btrfs mailing list, see below why
>
> Ok, Linus helped me find a workaround for this problem:
> https://lkml.org/lkml/2016/11/29/667
> namely:
> echo 2 > /proc/sys/vm/dirty_ratio
> echo 1 > /proc/sys/vm/dirty_background_ratio
> (it's a 24GB system, so the defaults of 20 and 10 were creating too many
> requests in th buffers)
I'll remove the bcache list on this followup since I want to concentrate
here on the fact that btrfs does behave badly with the default
dirty_ratio values.
As a reminder, it's a btrfs send/receive copy between 2 swraid5 arrays
on spinning rust.
swraid5 < bcache < dmcrypt < btrfs
Copying with btrfs send/receive causes massive hangs on the system.
Please see this explanation from Linus on why the workaround was
suggested:
https://lkml.org/lkml/2016/11/29/667
The hangs that I'm getting with bcache cache turned off (i.e.
passthrough) are now very likely only due to btrfs and mess up anything
doing file IO that ends up timing out, break USB even as reads time out
in the middle of USB requests, interrupts lost, and so forth.
All of this mostly went away with Linus' suggestion:
echo 2 > /proc/sys/vm/dirty_ratio
echo 1 > /proc/sys/vm/dirty_background_ratio
But that's hiding the symptom which I think is that btrfs is piling up too many I/O
requests during btrfs send/receive and btrfs scrub (probably balance too) and not
looking at resulting impact to system health.
Is there a way to stop flodding the entire system with I/O and causing
so much strain on it?
(I realize that if there is a caching layer underneath that just takes
requests and says thank you without giving other clues that underneath
bad things are happening, it may be hard, but I'm asking anyway :)
[10338.968912] perf: interrupt took too long (3927 > 3917), lowering kernel.perf_event_max_sample_rate to 50750
[12971.047705] ftdi_sio ttyUSB15: usb_serial_generic_read_bulk_callback - urb stopped: -32
[17761.122238] usb 4-1.4: USB disconnect, device number 39
[17761.141063] usb 4-1.4: usbfs: USBDEVFS_CONTROL failed cmd hub-ctrl rqt 160 rq 6 len 1024 ret -108
[17761.263252] usb 4-1: reset SuperSpeed USB device number 2 using xhci_hcd
[17761.938575] usb 4-1.4: new SuperSpeed USB device number 40 using xhci_hcd
[24130.574425] hpet1: lost 2306 rtc interrupts
[24156.034950] hpet1: lost 1628 rtc interrupts
[24173.314738] hpet1: lost 1104 rtc interrupts
[24180.129950] hpet1: lost 436 rtc interrupts
[24257.557955] hpet1: lost 4954 rtc interrupts
[24267.522656] hpet1: lost 637 rtc interrupts
[28034.954435] INFO: task btrfs:5618 blocked for more than 120 seconds.
[28034.975471] Tainted: G U 4.8.10-amd64-preempt-sysrq-20161121vb3tj1 #12
[28035.000964] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[28035.025429] btrfs D ffff91154d33fc70 0 5618 5372 0x00000080
[28035.047717] ffff91154d33fc70 0000000000200246 ffff911842f880c0 ffff9115a4cf01c0
[28035.071020] ffff91154d33fc58 ffff91154d340000 ffff91165493bca0 ffff9115623773f0
[28035.094252] 0000000000001000 0000000000000001 ffff91154d33fc88 ffffffffb86cf1a6
[28035.117538] Call Trace:
[28035.125791] [<ffffffffb86cf1a6>] schedule+0x8b/0xa3
[28035.141550] [<ffffffffb82bd18e>] btrfs_start_ordered_extent+0xce/0x122
[28035.162457] [<ffffffffb809af6c>] ? wake_up_atomic_t+0x2c/0x2c
[28035.180891] [<ffffffffb82bd434>] btrfs_wait_ordered_range+0xa9/0x10d
[28035.201723] [<ffffffffb82aec04>] btrfs_truncate+0x40/0x24b
[28035.219269] [<ffffffffb82af437>] btrfs_setattr+0x1da/0x2d7
[28035.237032] [<ffffffffb81c7507>] notify_change+0x252/0x39c
[28035.254566] [<ffffffffb81ad35b>] do_truncate+0x81/0xb4
[28035.271057] [<ffffffffb81ad467>] vfs_truncate+0xd9/0xf9
[28035.287782] [<ffffffffb81ad4ea>] do_sys_truncate+0x63/0xa7
[28155.781987] INFO: task btrfs:5618 blocked for more than 120 seconds.
[28155.802229] Tainted: G U 4.8.10-amd64-preempt-sysrq-20161121vb3tj1 #12
[28155.827894] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[28155.852479] btrfs D ffff91154d33fc70 0 5618 5372 0x00000080
[28155.874761] ffff91154d33fc70 0000000000200246 ffff911842f880c0 ffff9115a4cf01c0
[28155.898059] ffff91154d33fc58 ffff91154d340000 ffff91165493bca0 ffff9115623773f0
[28155.921464] 0000000000001000 0000000000000001 ffff91154d33fc88 ffffffffb86cf1a6
[28155.944720] Call Trace:
[28155.953176] [<ffffffffb86cf1a6>] schedule+0x8b/0xa3
[28155.968945] [<ffffffffb82bd18e>] btrfs_start_ordered_extent+0xce/0x122
[28155.989811] [<ffffffffb809af6c>] ? wake_up_atomic_t+0x2c/0x2c
[28156.008195] [<ffffffffb82bd434>] btrfs_wait_ordered_range+0xa9/0x10d
[28156.028498] [<ffffffffb82aec04>] btrfs_truncate+0x40/0x24b
[28156.046081] [<ffffffffb82af437>] btrfs_setattr+0x1da/0x2d7
[28156.063621] [<ffffffffb81c7507>] notify_change+0x252/0x39c
[28156.081667] [<ffffffffb81ad35b>] do_truncate+0x81/0xb4
[28156.098732] [<ffffffffb81ad467>] vfs_truncate+0xd9/0xf9
[28156.115489] [<ffffffffb81ad4ea>] do_sys_truncate+0x63/0xa7
[28156.133389] [<ffffffffb81ad63a>] SyS_truncate+0xe/0x10
[28156.149831] [<ffffffffb80039ec>] do_syscall_64+0x61/0x72
[28156.167179] [<ffffffffb86d2265>] entry_SYSCALL64_slow_path+0x25/0x25
[28397.436986] INFO: task btrfs:5618 blocked for more than 120 seconds.
[28397.456798] Tainted: G U 4.8.10-amd64-preempt-sysrq-20161121vb3tj1 #12
[28397.482122] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[28397.506359] btrfs D ffff91154d33fc70 0 5618 5372 0x00000080
[28397.528494] ffff91154d33fc70 0000000000200246 ffff911842fa41c0 ffff9115a4cf01c0
[28397.551645] ffff91154d33fc58 ffff91154d340000 ffff91142ba20140 ffff9115ba3ad7e8
[28397.574829] 0000000000001000 0000000000000001 ffff91154d33fc88 ffffffffb86cf1a6
[28397.598015] Call Trace:
[28397.606120] [<ffffffffb86cf1a6>] schedule+0x8b/0xa3
[28397.621772] [<ffffffffb82bd18e>] btrfs_start_ordered_extent+0xce/0x122
[28397.642376] [<ffffffffb809af6c>] ? wake_up_atomic_t+0x2c/0x2c
[28397.660661] [<ffffffffb82bd434>] btrfs_wait_ordered_range+0xa9/0x10d
[28397.680801] [<ffffffffb82aec04>] btrfs_truncate+0x40/0x24b
[28397.698281] [<ffffffffb82af437>] btrfs_setattr+0x1da/0x2d7
[28397.715769] [<ffffffffb81c7507>] notify_change+0x252/0x39c
[28397.733279] [<ffffffffb81ad35b>] do_truncate+0x81/0xb4
[28397.749731] [<ffffffffb81ad467>] vfs_truncate+0xd9/0xf9
[28397.766379] [<ffffffffb81ad4ea>] do_sys_truncate+0x63/0xa7
[28397.783878] [<ffffffffb81ad63a>] SyS_truncate+0xe/0x10
[28397.800319] [<ffffffffb80039ec>] do_syscall_64+0x61/0x72
[28397.817286] [<ffffffffb86d2265>] entry_SYSCALL64_slow_path+0x25/0x25
[28518.264522] INFO: task btrfs:5618 blocked for more than 120 seconds.
[28518.285084] Tainted: G U 4.8.10-amd64-preempt-sysrq-20161121vb3tj1 #12
[28518.310600] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[28518.335590] btrfs D ffff91154d33fc70 0 5618 5372 0x00000080
[28518.357716] ffff91154d33fc70 0000000000200246 ffff911842fa41c0 ffff9115a4cf01c0
[28518.380902] ffff91154d33fc58 ffff91154d340000 ffff91142ba20140 ffff9115ba3ad7e8
[28518.404075] 0000000000001000 0000000000000001 ffff91154d33fc88 ffffffffb86cf1a6
[28518.427227] Call Trace:
[28518.435384] [<ffffffffb86cf1a6>] schedule+0x8b/0xa3
[28518.451151] [<ffffffffb82bd18e>] btrfs_start_ordered_extent+0xce/0x122
[28518.471821] [<ffffffffb809af6c>] ? wake_up_atomic_t+0x2c/0x2c
[28518.490186] [<ffffffffb82bd434>] btrfs_wait_ordered_range+0xa9/0x10d
[28518.510301] [<ffffffffb82aec04>] btrfs_truncate+0x40/0x24b
[28518.527818] [<ffffffffb82af437>] btrfs_setattr+0x1da/0x2d7
[28518.545320] [<ffffffffb81c7507>] notify_change+0x252/0x39c
[28518.562924] [<ffffffffb81ad35b>] do_truncate+0x81/0xb4
[28518.579434] [<ffffffffb81ad467>] vfs_truncate+0xd9/0xf9
[28518.596132] [<ffffffffb81ad4ea>] do_sys_truncate+0x63/0xa7
[28518.613687] [<ffffffffb81ad63a>] SyS_truncate+0xe/0x10
[28518.630115] [<ffffffffb80039ec>] do_syscall_64+0x61/0x72
[28518.647003] [<ffffffffb86d2265>] entry_SYSCALL64_slow_path+0x25/0x25
[28518.667036] INFO: task kworker/u16:34:27172 blocked for more than 120 seconds.
[28518.689602] Tainted: G U 4.8.10-amd64-preempt-sysrq-20161121vb3tj1 #12
[28518.715144] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[28518.739569] kworker/u16:34 D ffff9117ec52bc80 0 27172 2 0x00000080
[28518.761628] Workqueue: btrfs-endio-write btrfs_endio_write_helper
[28518.780669] ffff9117ec52bc80 0000000000200246 ffff911842f9e180 ffff91157c386000
[28518.803719] ffff9117ec52bc68 ffff9117ec52c000 ffff9115cbf779f0 ffff9115cbf77800
[28518.826783] 0000000000000000 ffff9116af353500 ffff9117ec52bc98 ffffffffb86cf1a6
[28518.849863] Call Trace:
[28518.858011] [<ffffffffb86cf1a6>] schedule+0x8b/0xa3
[28518.873624] [<ffffffffb82a364f>] wait_current_trans.isra.16+0xcc/0x107
[28518.894158] [<ffffffffb809af6c>] ? wake_up_atomic_t+0x2c/0x2c
[28518.912370] [<ffffffffb82a58e1>] start_transaction+0x2ba/0x418
[28518.930827] [<ffffffffb82a5a8a>] btrfs_join_transaction+0x17/0x19
[28518.950105] [<ffffffffb82ad3ec>] btrfs_finish_ordered_io+0x1e9/0x4db
[28518.970132] [<ffffffffb8088000>] ? preempt_latency_start+0x34/0x5d
[28518.989640] [<ffffffffb82ad946>] finish_ordered_fn+0x15/0x17
[28519.007579] [<ffffffffb82cfd68>] btrfs_scrubparity_helper+0x10e/0x258
[28519.027844] [<ffffffffb807b8c1>] ? pwq_activate_delayed_work+0x4d/0x5b
[28519.048420] [<ffffffffb82cff40>] btrfs_endio_write_helper+0xe/0x10
[28519.067907] [<ffffffffb807d068>] process_one_work+0x186/0x29d
[28519.086158] [<ffffffffb807d66d>] worker_thread+0x1ea/0x2ba
[28519.103608] [<ffffffffb807d483>] ? rescuer_thread+0x2d1/0x2d1
[28519.121783] [<ffffffffb8081df3>] kthread+0xb4/0xbc
[28519.137100] [<ffffffffb86d23df>] ret_from_fork+0x1f/0x40
[28519.153938] [<ffffffffb8081d3f>] ? init_completion+0x24/0x24
[28519.171842] INFO: task kworker/u16:8:8909 blocked for more than 120 seconds.
[28519.193688] Tainted: G U 4.8.10-amd64-preempt-sysrq-20161121vb3tj1 #12
[28519.218851] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[28519.243006] kworker/u16:8 D ffff9117b992fc80 0 8909 2 0x00000080
[28519.264933] Workqueue: btrfs-endio-write btrfs_endio_write_helper
[28519.284037] ffff9117b992fc80 0000000000200246 ffff911842f9e180 ffff91137ef54100
[28519.307073] ffff9117b992fc68 ffff9117b9930000 ffff9115cbf779f0 ffff9115cbf77800
[28519.330159] 0000000000000000 ffff911282f35e60 ffff9117b992fc98 ffffffffb86cf1a6
[28519.353218] Call Trace:
[28519.361192] [<ffffffffb86cf1a6>] schedule+0x8b/0xa3
[28519.376747] [<ffffffffb82a364f>] wait_current_trans.isra.16+0xcc/0x107
[28519.397211] [<ffffffffb809af6c>] ? wake_up_atomic_t+0x2c/0x2c
[28519.415378] [<ffffffffb82a58e1>] start_transaction+0x2ba/0x418
[28519.433752] [<ffffffffb82a5a8a>] btrfs_join_transaction+0x17/0x19
[28519.452929] [<ffffffffb82ad3ec>] btrfs_finish_ordered_io+0x1e9/0x4db
[28519.472945] [<ffffffffb8088000>] ? preempt_latency_start+0x34/0x5d
[28519.492434] [<ffffffffb82ad946>] finish_ordered_fn+0x15/0x17
[28519.510364] [<ffffffffb82cfd68>] btrfs_scrubparity_helper+0x10e/0x258
[28519.530582] [<ffffffffb807b8c1>] ? pwq_activate_delayed_work+0x4d/0x5b
[28519.551062] [<ffffffffb82cff40>] btrfs_endio_write_helper+0xe/0x10
[28519.570519] [<ffffffffb807d068>] process_one_work+0x186/0x29d
[28519.588711] [<ffffffffb807d66d>] worker_thread+0x1ea/0x2ba
[28519.606111] [<ffffffffb807d483>] ? rescuer_thread+0x2d1/0x2d1
[28519.624269] [<ffffffffb8081df3>] kthread+0xb4/0xbc
[28519.639539] [<ffffffffb86d23df>] ret_from_fork+0x1f/0x40
[28519.656380] [<ffffffffb8081d3f>] ? init_completion+0x24/0x24
--
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
Microsoft is to operating systems ....
.... what McDonalds is to gourmet cooking
Home page: http://marc.merlins.org/ | PGP 1024R/763BE901
^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: btrfs flooding the I/O subsystem and hanging the machine, with bcache cache turned off
2016-11-30 17:18 ` btrfs flooding the I/O subsystem and hanging the machine, with bcache cache turned off Marc MERLIN
@ 2016-11-30 18:00 ` Austin S. Hemmelgarn
2016-11-30 18:16 ` Marc MERLIN
0 siblings, 1 reply; 20+ messages in thread
From: Austin S. Hemmelgarn @ 2016-11-30 18:00 UTC (permalink / raw)
To: Marc MERLIN, Btrfs BTRFS
On 2016-11-30 12:18, Marc MERLIN wrote:
> On Wed, Nov 30, 2016 at 08:46:46AM -0800, Marc MERLIN wrote:
>> +btrfs mailing list, see below why
>>
>> Ok, Linus helped me find a workaround for this problem:
>> https://lkml.org/lkml/2016/11/29/667
>> namely:
>> echo 2 > /proc/sys/vm/dirty_ratio
>> echo 1 > /proc/sys/vm/dirty_background_ratio
>> (it's a 24GB system, so the defaults of 20 and 10 were creating too many
>> requests in th buffers)
>
> I'll remove the bcache list on this followup since I want to concentrate
> here on the fact that btrfs does behave badly with the default
> dirty_ratio values.
I will comment that on big systems, almost everything behaves badly with
the default dirty ratios, they're leftovers from when 1GB was a huge
amount of RAM. As usual though, BTRFS has pathological behavior
compared to other options.
> As a reminder, it's a btrfs send/receive copy between 2 swraid5 arrays
> on spinning rust.
> swraid5 < bcache < dmcrypt < btrfs
>
> Copying with btrfs send/receive causes massive hangs on the system.
> Please see this explanation from Linus on why the workaround was
> suggested:
> https://lkml.org/lkml/2016/11/29/667
And Linux' assessment is absolutely correct (at least, the general
assessment is, I have no idea about btrfs_start_shared_extent, but I'm
more than willing to bet he's correct that that's the culprit).
>
> The hangs that I'm getting with bcache cache turned off (i.e.
> passthrough) are now very likely only due to btrfs and mess up anything
> doing file IO that ends up timing out, break USB even as reads time out
> in the middle of USB requests, interrupts lost, and so forth.
>
> All of this mostly went away with Linus' suggestion:
> echo 2 > /proc/sys/vm/dirty_ratio
> echo 1 > /proc/sys/vm/dirty_background_ratio
>
> But that's hiding the symptom which I think is that btrfs is piling up too many I/O
> requests during btrfs send/receive and btrfs scrub (probably balance too) and not
> looking at resulting impact to system health.
I see pretty much identical behavior using any number of other storage
configurations on a USB 2.0 flash drive connected to a system with 16GB
of RAM with the default dirty ratios because it's trying to cache up to
3.2GB of data for writeback. While BTRFS is doing highly sub-optimal
things here, the ancient default writeback ratios are just as much a
culprit. I would suggest that get changed to 200MB or 20% of RAM,
whichever is smaller, which would give overall almost identical behavior
to x86-32, which in turn works reasonably well for most cases. I sadly
don't have the time, patience, or expertise to write up such a patch
myself though.
>
> Is there a way to stop flodding the entire system with I/O and causing
> so much strain on it?
> (I realize that if there is a caching layer underneath that just takes
> requests and says thank you without giving other clues that underneath
> bad things are happening, it may be hard, but I'm asking anyway :)
>
>
> [10338.968912] perf: interrupt took too long (3927 > 3917), lowering kernel.perf_event_max_sample_rate to 50750
>
> [12971.047705] ftdi_sio ttyUSB15: usb_serial_generic_read_bulk_callback - urb stopped: -32
>
> [17761.122238] usb 4-1.4: USB disconnect, device number 39
> [17761.141063] usb 4-1.4: usbfs: USBDEVFS_CONTROL failed cmd hub-ctrl rqt 160 rq 6 len 1024 ret -108
> [17761.263252] usb 4-1: reset SuperSpeed USB device number 2 using xhci_hcd
> [17761.938575] usb 4-1.4: new SuperSpeed USB device number 40 using xhci_hcd
>
> [24130.574425] hpet1: lost 2306 rtc interrupts
> [24156.034950] hpet1: lost 1628 rtc interrupts
> [24173.314738] hpet1: lost 1104 rtc interrupts
> [24180.129950] hpet1: lost 436 rtc interrupts
> [24257.557955] hpet1: lost 4954 rtc interrupts
> [24267.522656] hpet1: lost 637 rtc interrupts
>
> [28034.954435] INFO: task btrfs:5618 blocked for more than 120 seconds.
> [28034.975471] Tainted: G U 4.8.10-amd64-preempt-sysrq-20161121vb3tj1 #12
> [28035.000964] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> [28035.025429] btrfs D ffff91154d33fc70 0 5618 5372 0x00000080
> [28035.047717] ffff91154d33fc70 0000000000200246 ffff911842f880c0 ffff9115a4cf01c0
> [28035.071020] ffff91154d33fc58 ffff91154d340000 ffff91165493bca0 ffff9115623773f0
> [28035.094252] 0000000000001000 0000000000000001 ffff91154d33fc88 ffffffffb86cf1a6
> [28035.117538] Call Trace:
> [28035.125791] [<ffffffffb86cf1a6>] schedule+0x8b/0xa3
> [28035.141550] [<ffffffffb82bd18e>] btrfs_start_ordered_extent+0xce/0x122
> [28035.162457] [<ffffffffb809af6c>] ? wake_up_atomic_t+0x2c/0x2c
> [28035.180891] [<ffffffffb82bd434>] btrfs_wait_ordered_range+0xa9/0x10d
> [28035.201723] [<ffffffffb82aec04>] btrfs_truncate+0x40/0x24b
> [28035.219269] [<ffffffffb82af437>] btrfs_setattr+0x1da/0x2d7
> [28035.237032] [<ffffffffb81c7507>] notify_change+0x252/0x39c
> [28035.254566] [<ffffffffb81ad35b>] do_truncate+0x81/0xb4
> [28035.271057] [<ffffffffb81ad467>] vfs_truncate+0xd9/0xf9
> [28035.287782] [<ffffffffb81ad4ea>] do_sys_truncate+0x63/0xa7
>
> [28155.781987] INFO: task btrfs:5618 blocked for more than 120 seconds.
> [28155.802229] Tainted: G U 4.8.10-amd64-preempt-sysrq-20161121vb3tj1 #12
> [28155.827894] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> [28155.852479] btrfs D ffff91154d33fc70 0 5618 5372 0x00000080
> [28155.874761] ffff91154d33fc70 0000000000200246 ffff911842f880c0 ffff9115a4cf01c0
> [28155.898059] ffff91154d33fc58 ffff91154d340000 ffff91165493bca0 ffff9115623773f0
> [28155.921464] 0000000000001000 0000000000000001 ffff91154d33fc88 ffffffffb86cf1a6
> [28155.944720] Call Trace:
> [28155.953176] [<ffffffffb86cf1a6>] schedule+0x8b/0xa3
> [28155.968945] [<ffffffffb82bd18e>] btrfs_start_ordered_extent+0xce/0x122
> [28155.989811] [<ffffffffb809af6c>] ? wake_up_atomic_t+0x2c/0x2c
> [28156.008195] [<ffffffffb82bd434>] btrfs_wait_ordered_range+0xa9/0x10d
> [28156.028498] [<ffffffffb82aec04>] btrfs_truncate+0x40/0x24b
> [28156.046081] [<ffffffffb82af437>] btrfs_setattr+0x1da/0x2d7
> [28156.063621] [<ffffffffb81c7507>] notify_change+0x252/0x39c
> [28156.081667] [<ffffffffb81ad35b>] do_truncate+0x81/0xb4
> [28156.098732] [<ffffffffb81ad467>] vfs_truncate+0xd9/0xf9
> [28156.115489] [<ffffffffb81ad4ea>] do_sys_truncate+0x63/0xa7
> [28156.133389] [<ffffffffb81ad63a>] SyS_truncate+0xe/0x10
> [28156.149831] [<ffffffffb80039ec>] do_syscall_64+0x61/0x72
> [28156.167179] [<ffffffffb86d2265>] entry_SYSCALL64_slow_path+0x25/0x25
>
> [28397.436986] INFO: task btrfs:5618 blocked for more than 120 seconds.
> [28397.456798] Tainted: G U 4.8.10-amd64-preempt-sysrq-20161121vb3tj1 #12
> [28397.482122] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> [28397.506359] btrfs D ffff91154d33fc70 0 5618 5372 0x00000080
> [28397.528494] ffff91154d33fc70 0000000000200246 ffff911842fa41c0 ffff9115a4cf01c0
> [28397.551645] ffff91154d33fc58 ffff91154d340000 ffff91142ba20140 ffff9115ba3ad7e8
> [28397.574829] 0000000000001000 0000000000000001 ffff91154d33fc88 ffffffffb86cf1a6
> [28397.598015] Call Trace:
> [28397.606120] [<ffffffffb86cf1a6>] schedule+0x8b/0xa3
> [28397.621772] [<ffffffffb82bd18e>] btrfs_start_ordered_extent+0xce/0x122
> [28397.642376] [<ffffffffb809af6c>] ? wake_up_atomic_t+0x2c/0x2c
> [28397.660661] [<ffffffffb82bd434>] btrfs_wait_ordered_range+0xa9/0x10d
> [28397.680801] [<ffffffffb82aec04>] btrfs_truncate+0x40/0x24b
> [28397.698281] [<ffffffffb82af437>] btrfs_setattr+0x1da/0x2d7
> [28397.715769] [<ffffffffb81c7507>] notify_change+0x252/0x39c
> [28397.733279] [<ffffffffb81ad35b>] do_truncate+0x81/0xb4
> [28397.749731] [<ffffffffb81ad467>] vfs_truncate+0xd9/0xf9
> [28397.766379] [<ffffffffb81ad4ea>] do_sys_truncate+0x63/0xa7
> [28397.783878] [<ffffffffb81ad63a>] SyS_truncate+0xe/0x10
> [28397.800319] [<ffffffffb80039ec>] do_syscall_64+0x61/0x72
> [28397.817286] [<ffffffffb86d2265>] entry_SYSCALL64_slow_path+0x25/0x25
>
> [28518.264522] INFO: task btrfs:5618 blocked for more than 120 seconds.
> [28518.285084] Tainted: G U 4.8.10-amd64-preempt-sysrq-20161121vb3tj1 #12
> [28518.310600] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> [28518.335590] btrfs D ffff91154d33fc70 0 5618 5372 0x00000080
> [28518.357716] ffff91154d33fc70 0000000000200246 ffff911842fa41c0 ffff9115a4cf01c0
> [28518.380902] ffff91154d33fc58 ffff91154d340000 ffff91142ba20140 ffff9115ba3ad7e8
> [28518.404075] 0000000000001000 0000000000000001 ffff91154d33fc88 ffffffffb86cf1a6
> [28518.427227] Call Trace:
> [28518.435384] [<ffffffffb86cf1a6>] schedule+0x8b/0xa3
> [28518.451151] [<ffffffffb82bd18e>] btrfs_start_ordered_extent+0xce/0x122
> [28518.471821] [<ffffffffb809af6c>] ? wake_up_atomic_t+0x2c/0x2c
> [28518.490186] [<ffffffffb82bd434>] btrfs_wait_ordered_range+0xa9/0x10d
> [28518.510301] [<ffffffffb82aec04>] btrfs_truncate+0x40/0x24b
> [28518.527818] [<ffffffffb82af437>] btrfs_setattr+0x1da/0x2d7
> [28518.545320] [<ffffffffb81c7507>] notify_change+0x252/0x39c
> [28518.562924] [<ffffffffb81ad35b>] do_truncate+0x81/0xb4
> [28518.579434] [<ffffffffb81ad467>] vfs_truncate+0xd9/0xf9
> [28518.596132] [<ffffffffb81ad4ea>] do_sys_truncate+0x63/0xa7
> [28518.613687] [<ffffffffb81ad63a>] SyS_truncate+0xe/0x10
> [28518.630115] [<ffffffffb80039ec>] do_syscall_64+0x61/0x72
> [28518.647003] [<ffffffffb86d2265>] entry_SYSCALL64_slow_path+0x25/0x25
> [28518.667036] INFO: task kworker/u16:34:27172 blocked for more than 120 seconds.
> [28518.689602] Tainted: G U 4.8.10-amd64-preempt-sysrq-20161121vb3tj1 #12
> [28518.715144] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> [28518.739569] kworker/u16:34 D ffff9117ec52bc80 0 27172 2 0x00000080
> [28518.761628] Workqueue: btrfs-endio-write btrfs_endio_write_helper
> [28518.780669] ffff9117ec52bc80 0000000000200246 ffff911842f9e180 ffff91157c386000
> [28518.803719] ffff9117ec52bc68 ffff9117ec52c000 ffff9115cbf779f0 ffff9115cbf77800
> [28518.826783] 0000000000000000 ffff9116af353500 ffff9117ec52bc98 ffffffffb86cf1a6
> [28518.849863] Call Trace:
> [28518.858011] [<ffffffffb86cf1a6>] schedule+0x8b/0xa3
> [28518.873624] [<ffffffffb82a364f>] wait_current_trans.isra.16+0xcc/0x107
> [28518.894158] [<ffffffffb809af6c>] ? wake_up_atomic_t+0x2c/0x2c
> [28518.912370] [<ffffffffb82a58e1>] start_transaction+0x2ba/0x418
> [28518.930827] [<ffffffffb82a5a8a>] btrfs_join_transaction+0x17/0x19
> [28518.950105] [<ffffffffb82ad3ec>] btrfs_finish_ordered_io+0x1e9/0x4db
> [28518.970132] [<ffffffffb8088000>] ? preempt_latency_start+0x34/0x5d
> [28518.989640] [<ffffffffb82ad946>] finish_ordered_fn+0x15/0x17
> [28519.007579] [<ffffffffb82cfd68>] btrfs_scrubparity_helper+0x10e/0x258
> [28519.027844] [<ffffffffb807b8c1>] ? pwq_activate_delayed_work+0x4d/0x5b
> [28519.048420] [<ffffffffb82cff40>] btrfs_endio_write_helper+0xe/0x10
> [28519.067907] [<ffffffffb807d068>] process_one_work+0x186/0x29d
> [28519.086158] [<ffffffffb807d66d>] worker_thread+0x1ea/0x2ba
> [28519.103608] [<ffffffffb807d483>] ? rescuer_thread+0x2d1/0x2d1
> [28519.121783] [<ffffffffb8081df3>] kthread+0xb4/0xbc
> [28519.137100] [<ffffffffb86d23df>] ret_from_fork+0x1f/0x40
> [28519.153938] [<ffffffffb8081d3f>] ? init_completion+0x24/0x24
> [28519.171842] INFO: task kworker/u16:8:8909 blocked for more than 120 seconds.
> [28519.193688] Tainted: G U 4.8.10-amd64-preempt-sysrq-20161121vb3tj1 #12
> [28519.218851] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> [28519.243006] kworker/u16:8 D ffff9117b992fc80 0 8909 2 0x00000080
> [28519.264933] Workqueue: btrfs-endio-write btrfs_endio_write_helper
> [28519.284037] ffff9117b992fc80 0000000000200246 ffff911842f9e180 ffff91137ef54100
> [28519.307073] ffff9117b992fc68 ffff9117b9930000 ffff9115cbf779f0 ffff9115cbf77800
> [28519.330159] 0000000000000000 ffff911282f35e60 ffff9117b992fc98 ffffffffb86cf1a6
> [28519.353218] Call Trace:
> [28519.361192] [<ffffffffb86cf1a6>] schedule+0x8b/0xa3
> [28519.376747] [<ffffffffb82a364f>] wait_current_trans.isra.16+0xcc/0x107
> [28519.397211] [<ffffffffb809af6c>] ? wake_up_atomic_t+0x2c/0x2c
> [28519.415378] [<ffffffffb82a58e1>] start_transaction+0x2ba/0x418
> [28519.433752] [<ffffffffb82a5a8a>] btrfs_join_transaction+0x17/0x19
> [28519.452929] [<ffffffffb82ad3ec>] btrfs_finish_ordered_io+0x1e9/0x4db
> [28519.472945] [<ffffffffb8088000>] ? preempt_latency_start+0x34/0x5d
> [28519.492434] [<ffffffffb82ad946>] finish_ordered_fn+0x15/0x17
> [28519.510364] [<ffffffffb82cfd68>] btrfs_scrubparity_helper+0x10e/0x258
> [28519.530582] [<ffffffffb807b8c1>] ? pwq_activate_delayed_work+0x4d/0x5b
> [28519.551062] [<ffffffffb82cff40>] btrfs_endio_write_helper+0xe/0x10
> [28519.570519] [<ffffffffb807d068>] process_one_work+0x186/0x29d
> [28519.588711] [<ffffffffb807d66d>] worker_thread+0x1ea/0x2ba
> [28519.606111] [<ffffffffb807d483>] ? rescuer_thread+0x2d1/0x2d1
> [28519.624269] [<ffffffffb8081df3>] kthread+0xb4/0xbc
> [28519.639539] [<ffffffffb86d23df>] ret_from_fork+0x1f/0x40
> [28519.656380] [<ffffffffb8081d3f>] ? init_completion+0x24/0x24
>
>
^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: btrfs flooding the I/O subsystem and hanging the machine, with bcache cache turned off
2016-11-30 18:00 ` Austin S. Hemmelgarn
@ 2016-11-30 18:16 ` Marc MERLIN
0 siblings, 0 replies; 20+ messages in thread
From: Marc MERLIN @ 2016-11-30 18:16 UTC (permalink / raw)
To: Austin S. Hemmelgarn
Cc: Btrfs BTRFS, Michal Hocko, Vlastimil Babka, linux-mm,
Joonsoo Kim, torvalds
+folks from linux-mm thread for your suggestion
On Wed, Nov 30, 2016 at 01:00:45PM -0500, Austin S. Hemmelgarn wrote:
> > swraid5 < bcache < dmcrypt < btrfs
> >
> > Copying with btrfs send/receive causes massive hangs on the system.
> > Please see this explanation from Linus on why the workaround was
> > suggested:
> > https://lkml.org/lkml/2016/11/29/667
> And Linux' assessment is absolutely correct (at least, the general
> assessment is, I have no idea about btrfs_start_shared_extent, but I'm more
> than willing to bet he's correct that that's the culprit).
> > All of this mostly went away with Linus' suggestion:
> > echo 2 > /proc/sys/vm/dirty_ratio
> > echo 1 > /proc/sys/vm/dirty_background_ratio
> >
> > But that's hiding the symptom which I think is that btrfs is piling up too many I/O
> > requests during btrfs send/receive and btrfs scrub (probably balance too) and not
> > looking at resulting impact to system health.
> I see pretty much identical behavior using any number of other storage
> configurations on a USB 2.0 flash drive connected to a system with 16GB of
> RAM with the default dirty ratios because it's trying to cache up to 3.2GB
> of data for writeback. While BTRFS is doing highly sub-optimal things here,
> the ancient default writeback ratios are just as much a culprit. I would
> suggest that get changed to 200MB or 20% of RAM, whichever is smaller, which
> would give overall almost identical behavior to x86-32, which in turn works
> reasonably well for most cases. I sadly don't have the time, patience, or
> expertise to write up such a patch myself though.
Dear linux-mm folks, is that something you could consider (changing the
dirty_ratio defaults) given that it affects at least bcache and btrfs
(with or without bcache)?
By the way, on the 200MB max suggestion, when I had 2 and 1% (or 480MB
and 240MB on my 24GB system), this was enough to make btrfs behave
sanely, but only if I had bcache turned off.
With bcache enabled, those values were just enough so that bcache didn't
crash my system, but not enough that prevent undesirable behaviour
(things hanging, 100+ bcache kworkers piled up, and more). However, the
copy did succeed, despite the relative impact on the system, so it's
better than nothing :)
But the impact from bcache probably goes beyond what btrfs is
responsible for, so I have a separate thread on the bcache list:
http://marc.info/?l=linux-bcache&m=148052441423532&w=2
http://marc.info/?l=linux-bcache&m=148052620524162&w=2
On the plus side, btrfs did ok with 0 visible impact to my system with
those 480 and 240MB dirty ratio values.
Thanks for your reply, Austin.
Marc
--
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
Microsoft is to operating systems ....
.... what McDonalds is to gourmet cooking
Home page: http://marc.merlins.org/ | PGP 1024R/763BE901
^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: btrfs flooding the I/O subsystem and hanging the machine, with bcache cache turned off
@ 2016-11-30 18:16 ` Marc MERLIN
0 siblings, 0 replies; 20+ messages in thread
From: Marc MERLIN @ 2016-11-30 18:16 UTC (permalink / raw)
To: Austin S. Hemmelgarn
Cc: Btrfs BTRFS, Michal Hocko, Vlastimil Babka, linux-mm,
Joonsoo Kim, torvalds
+folks from linux-mm thread for your suggestion
On Wed, Nov 30, 2016 at 01:00:45PM -0500, Austin S. Hemmelgarn wrote:
> > swraid5 < bcache < dmcrypt < btrfs
> >
> > Copying with btrfs send/receive causes massive hangs on the system.
> > Please see this explanation from Linus on why the workaround was
> > suggested:
> > https://lkml.org/lkml/2016/11/29/667
> And Linux' assessment is absolutely correct (at least, the general
> assessment is, I have no idea about btrfs_start_shared_extent, but I'm more
> than willing to bet he's correct that that's the culprit).
> > All of this mostly went away with Linus' suggestion:
> > echo 2 > /proc/sys/vm/dirty_ratio
> > echo 1 > /proc/sys/vm/dirty_background_ratio
> >
> > But that's hiding the symptom which I think is that btrfs is piling up too many I/O
> > requests during btrfs send/receive and btrfs scrub (probably balance too) and not
> > looking at resulting impact to system health.
> I see pretty much identical behavior using any number of other storage
> configurations on a USB 2.0 flash drive connected to a system with 16GB of
> RAM with the default dirty ratios because it's trying to cache up to 3.2GB
> of data for writeback. While BTRFS is doing highly sub-optimal things here,
> the ancient default writeback ratios are just as much a culprit. I would
> suggest that get changed to 200MB or 20% of RAM, whichever is smaller, which
> would give overall almost identical behavior to x86-32, which in turn works
> reasonably well for most cases. I sadly don't have the time, patience, or
> expertise to write up such a patch myself though.
Dear linux-mm folks, is that something you could consider (changing the
dirty_ratio defaults) given that it affects at least bcache and btrfs
(with or without bcache)?
By the way, on the 200MB max suggestion, when I had 2 and 1% (or 480MB
and 240MB on my 24GB system), this was enough to make btrfs behave
sanely, but only if I had bcache turned off.
With bcache enabled, those values were just enough so that bcache didn't
crash my system, but not enough that prevent undesirable behaviour
(things hanging, 100+ bcache kworkers piled up, and more). However, the
copy did succeed, despite the relative impact on the system, so it's
better than nothing :)
But the impact from bcache probably goes beyond what btrfs is
responsible for, so I have a separate thread on the bcache list:
http://marc.info/?l=linux-bcache&m=148052441423532&w=2
http://marc.info/?l=linux-bcache&m=148052620524162&w=2
On the plus side, btrfs did ok with 0 visible impact to my system with
those 480 and 240MB dirty ratio values.
Thanks for your reply, Austin.
Marc
--
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
Microsoft is to operating systems ....
.... what McDonalds is to gourmet cooking
Home page: http://marc.merlins.org/ | PGP 1024R/763BE901
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: 4.8.8, bcache deadlock and hard lockup
2016-11-30 16:46 ` Marc MERLIN
2016-11-30 17:16 ` Marc MERLIN
2016-11-30 17:18 ` btrfs flooding the I/O subsystem and hanging the machine, with bcache cache turned off Marc MERLIN
@ 2016-11-30 23:57 ` Eric Wheeler
2016-12-01 0:09 ` Marc MERLIN
2016-12-01 0:48 ` Chris Murphy
2 siblings, 2 replies; 20+ messages in thread
From: Eric Wheeler @ 2016-11-30 23:57 UTC (permalink / raw)
To: Marc MERLIN; +Cc: Coly Li, linux-bcache, Btrfs BTRFS
On Wed, 30 Nov 2016, Marc MERLIN wrote:
> +btrfs mailing list, see below why
>
> On Tue, Nov 29, 2016 at 12:59:44PM -0800, Eric Wheeler wrote:
> > On Mon, 27 Nov 2016, Coly Li wrote:
> > >
> > > Yes, too many work queues... I guess the locking might be caused by some
> > > very obscure reference of closure code. I cannot have any clue if I
> > > cannot find a stable procedure to reproduce this issue.
> > >
> > > Hmm, if there is a tool to clone all the meta data of the back end cache
> > > and whole cached device, there might be a method to replay the oops much
> > > easier.
> > >
> > > Eric, do you have any hint ?
> >
> > Note that the backing device doesn't have any metadata, just a superblock.
> > You can easily dd that off onto some other volume without transferring the
> > data. By default, data starts at 8k, or whatever you used in `make-bcache
> > -w`.
>
> Ok, Linus helped me find a workaround for this problem:
> https://lkml.org/lkml/2016/11/29/667
> namely:
> echo 2 > /proc/sys/vm/dirty_ratio
> echo 1 > /proc/sys/vm/dirty_background_ratio
> (it's a 24GB system, so the defaults of 20 and 10 were creating too many
> requests in th buffers)
>
> Note that this is only a workaround, not a fix.
>
> When I did this and re tried my big copy again, I still got 100+ kernel
> work queues, but apparently the underlying swraid5 was able to unblock
> and satisfy the write requests before too many accumulated and crashed
> the kernel.
>
> I'm not a kernel coder, but seems to me that bcache needs a way to
> throttle incoming requests if there are too many so that it does not end
> up in a state where things blow up due to too many piled up requests.
>
> You should be able to reproduce this by taking 5 spinning rust drives,
> put raid5 on top, dmcrypt, bcache and hopefully any filesystem (although
> I used btrfs) and send lots of requests.
> Actually to be honest, the problems have mostly been happening when I do
> btrfs scrub and btrfs send/receive which both generate I/O from within
> the kernel instead of user space.
> So here, btrfs may be a contributor to the problem too, but while btrfs
> still trashes my system if I remove the caching device on bcache (and
> with the default dirty ratio values), it doesn't crash the kernel.
>
> I'll start another separate thread with the btrfs folks on how much
> pressure is put on the system, but on your side it would be good to help
> ensure that bcache doesn't crash the system altogether if too many
> requests are allowed to pile up.
Try BFQ. It is AWESOME and helps reduce the congestion problem with bulk
writes at the request queue on its way to the spinning disk or SSD:
http://algo.ing.unimo.it/people/paolo/disk_sched/
use the latest BFQ git here, merge it into v4.8.y:
https://github.com/linusw/linux-bfq/commits/bfq-v8
This doesn't completely fix the dirty_ration problem, but it is far better
than CFQ or deadline in my opinion (and experience).
-Eric
--
Eric Wheeler
>
> Thanks,
> Marc
> --
> "A mouse is a device used to point at the xterm you want to type in" - A.S.R.
> Microsoft is to operating systems ....
> .... what McDonalds is to gourmet cooking
> Home page: http://marc.merlins.org/ | PGP 1024R/763BE901
> --
> To unsubscribe from this list: send the line "unsubscribe linux-bcache" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
>
^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: 4.8.8, bcache deadlock and hard lockup
2016-11-30 23:57 ` 4.8.8, bcache deadlock and hard lockup Eric Wheeler
@ 2016-12-01 0:09 ` Marc MERLIN
2016-12-01 21:58 ` Eric Wheeler
2016-12-01 0:48 ` Chris Murphy
1 sibling, 1 reply; 20+ messages in thread
From: Marc MERLIN @ 2016-12-01 0:09 UTC (permalink / raw)
To: Eric Wheeler; +Cc: Coly Li, linux-bcache, Btrfs BTRFS
On Wed, Nov 30, 2016 at 03:57:28PM -0800, Eric Wheeler wrote:
> > I'll start another separate thread with the btrfs folks on how much
> > pressure is put on the system, but on your side it would be good to help
> > ensure that bcache doesn't crash the system altogether if too many
> > requests are allowed to pile up.
>
> Try BFQ. It is AWESOME and helps reduce the congestion problem with bulk
> writes at the request queue on its way to the spinning disk or SSD:
> http://algo.ing.unimo.it/people/paolo/disk_sched/
>
> use the latest BFQ git here, merge it into v4.8.y:
> https://github.com/linusw/linux-bfq/commits/bfq-v8
>
> This doesn't completely fix the dirty_ration problem, but it is far better
> than CFQ or deadline in my opinion (and experience).
That's good to know thanks.
But for my uninformed opinion, is there anything bcache can do to throttle
incoming requests if they are piling up, or they're coming from producers
upstream and bcache has no choice but try and process them as quickly as
possible without a way to block the sender if too many are coming?
Thanks,
Marc
--
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
Microsoft is to operating systems ....
.... what McDonalds is to gourmet cooking
Home page: http://marc.merlins.org/
^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: 4.8.8, bcache deadlock and hard lockup
2016-11-30 23:57 ` 4.8.8, bcache deadlock and hard lockup Eric Wheeler
2016-12-01 0:09 ` Marc MERLIN
@ 2016-12-01 0:48 ` Chris Murphy
2016-12-01 12:30 ` Austin S. Hemmelgarn
1 sibling, 1 reply; 20+ messages in thread
From: Chris Murphy @ 2016-12-01 0:48 UTC (permalink / raw)
To: Eric Wheeler; +Cc: Marc MERLIN, Coly Li, linux-bcache, Btrfs BTRFS
On Wed, Nov 30, 2016 at 4:57 PM, Eric Wheeler <bcache@lists.ewheeler.net> wrote:
> On Wed, 30 Nov 2016, Marc MERLIN wrote:
>> +btrfs mailing list, see below why
>>
>> On Tue, Nov 29, 2016 at 12:59:44PM -0800, Eric Wheeler wrote:
>> > On Mon, 27 Nov 2016, Coly Li wrote:
>> > >
>> > > Yes, too many work queues... I guess the locking might be caused by some
>> > > very obscure reference of closure code. I cannot have any clue if I
>> > > cannot find a stable procedure to reproduce this issue.
>> > >
>> > > Hmm, if there is a tool to clone all the meta data of the back end cache
>> > > and whole cached device, there might be a method to replay the oops much
>> > > easier.
>> > >
>> > > Eric, do you have any hint ?
>> >
>> > Note that the backing device doesn't have any metadata, just a superblock.
>> > You can easily dd that off onto some other volume without transferring the
>> > data. By default, data starts at 8k, or whatever you used in `make-bcache
>> > -w`.
>>
>> Ok, Linus helped me find a workaround for this problem:
>> https://lkml.org/lkml/2016/11/29/667
>> namely:
>> echo 2 > /proc/sys/vm/dirty_ratio
>> echo 1 > /proc/sys/vm/dirty_background_ratio
>> (it's a 24GB system, so the defaults of 20 and 10 were creating too many
>> requests in th buffers)
>>
>> Note that this is only a workaround, not a fix.
>>
>> When I did this and re tried my big copy again, I still got 100+ kernel
>> work queues, but apparently the underlying swraid5 was able to unblock
>> and satisfy the write requests before too many accumulated and crashed
>> the kernel.
>>
>> I'm not a kernel coder, but seems to me that bcache needs a way to
>> throttle incoming requests if there are too many so that it does not end
>> up in a state where things blow up due to too many piled up requests.
>>
>> You should be able to reproduce this by taking 5 spinning rust drives,
>> put raid5 on top, dmcrypt, bcache and hopefully any filesystem (although
>> I used btrfs) and send lots of requests.
>> Actually to be honest, the problems have mostly been happening when I do
>> btrfs scrub and btrfs send/receive which both generate I/O from within
>> the kernel instead of user space.
>> So here, btrfs may be a contributor to the problem too, but while btrfs
>> still trashes my system if I remove the caching device on bcache (and
>> with the default dirty ratio values), it doesn't crash the kernel.
>>
>> I'll start another separate thread with the btrfs folks on how much
>> pressure is put on the system, but on your side it would be good to help
>> ensure that bcache doesn't crash the system altogether if too many
>> requests are allowed to pile up.
>
>
> Try BFQ. It is AWESOME and helps reduce the congestion problem with bulk
> writes at the request queue on its way to the spinning disk or SSD:
> http://algo.ing.unimo.it/people/paolo/disk_sched/
>
> use the latest BFQ git here, merge it into v4.8.y:
> https://github.com/linusw/linux-bfq/commits/bfq-v8
>
> This doesn't completely fix the dirty_ration problem, but it is far better
> than CFQ or deadline in my opinion (and experience).
There are several threads over the past year with users having
problems no one else had previously reported, and they were using BFQ.
But there's no evidence whether BFQ was the cause, or exposing some
existing bug that another scheduler doesn't. Anyway, I'd say using an
out of tree scheduler means higher burden of testing and skepticism.
--
Chris Murphy
^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: 4.8.8, bcache deadlock and hard lockup
2016-12-01 0:48 ` Chris Murphy
@ 2016-12-01 12:30 ` Austin S. Hemmelgarn
0 siblings, 0 replies; 20+ messages in thread
From: Austin S. Hemmelgarn @ 2016-12-01 12:30 UTC (permalink / raw)
To: Chris Murphy, Eric Wheeler
Cc: Marc MERLIN, Coly Li, linux-bcache, Btrfs BTRFS
On 2016-11-30 19:48, Chris Murphy wrote:
> On Wed, Nov 30, 2016 at 4:57 PM, Eric Wheeler <bcache@lists.ewheeler.net> wrote:
>> On Wed, 30 Nov 2016, Marc MERLIN wrote:
>>> +btrfs mailing list, see below why
>>>
>>> On Tue, Nov 29, 2016 at 12:59:44PM -0800, Eric Wheeler wrote:
>>>> On Mon, 27 Nov 2016, Coly Li wrote:
>>>>>
>>>>> Yes, too many work queues... I guess the locking might be caused by some
>>>>> very obscure reference of closure code. I cannot have any clue if I
>>>>> cannot find a stable procedure to reproduce this issue.
>>>>>
>>>>> Hmm, if there is a tool to clone all the meta data of the back end cache
>>>>> and whole cached device, there might be a method to replay the oops much
>>>>> easier.
>>>>>
>>>>> Eric, do you have any hint ?
>>>>
>>>> Note that the backing device doesn't have any metadata, just a superblock.
>>>> You can easily dd that off onto some other volume without transferring the
>>>> data. By default, data starts at 8k, or whatever you used in `make-bcache
>>>> -w`.
>>>
>>> Ok, Linus helped me find a workaround for this problem:
>>> https://lkml.org/lkml/2016/11/29/667
>>> namely:
>>> echo 2 > /proc/sys/vm/dirty_ratio
>>> echo 1 > /proc/sys/vm/dirty_background_ratio
>>> (it's a 24GB system, so the defaults of 20 and 10 were creating too many
>>> requests in th buffers)
>>>
>>> Note that this is only a workaround, not a fix.
>>>
>>> When I did this and re tried my big copy again, I still got 100+ kernel
>>> work queues, but apparently the underlying swraid5 was able to unblock
>>> and satisfy the write requests before too many accumulated and crashed
>>> the kernel.
>>>
>>> I'm not a kernel coder, but seems to me that bcache needs a way to
>>> throttle incoming requests if there are too many so that it does not end
>>> up in a state where things blow up due to too many piled up requests.
>>>
>>> You should be able to reproduce this by taking 5 spinning rust drives,
>>> put raid5 on top, dmcrypt, bcache and hopefully any filesystem (although
>>> I used btrfs) and send lots of requests.
>>> Actually to be honest, the problems have mostly been happening when I do
>>> btrfs scrub and btrfs send/receive which both generate I/O from within
>>> the kernel instead of user space.
>>> So here, btrfs may be a contributor to the problem too, but while btrfs
>>> still trashes my system if I remove the caching device on bcache (and
>>> with the default dirty ratio values), it doesn't crash the kernel.
>>>
>>> I'll start another separate thread with the btrfs folks on how much
>>> pressure is put on the system, but on your side it would be good to help
>>> ensure that bcache doesn't crash the system altogether if too many
>>> requests are allowed to pile up.
>>
>>
>> Try BFQ. It is AWESOME and helps reduce the congestion problem with bulk
>> writes at the request queue on its way to the spinning disk or SSD:
>> http://algo.ing.unimo.it/people/paolo/disk_sched/
>>
>> use the latest BFQ git here, merge it into v4.8.y:
>> https://github.com/linusw/linux-bfq/commits/bfq-v8
>>
>> This doesn't completely fix the dirty_ration problem, but it is far better
>> than CFQ or deadline in my opinion (and experience).
>
> There are several threads over the past year with users having
> problems no one else had previously reported, and they were using BFQ.
> But there's no evidence whether BFQ was the cause, or exposing some
> existing bug that another scheduler doesn't. Anyway, I'd say using an
> out of tree scheduler means higher burden of testing and skepticism.
Normally I'd agree on this, but BFQ is a bit of a different situation
from usual because:
1. 90% of the reason that BFQ isn't in mainline is that the block
maintainers have declared the legacy (non blk-mq) code deprecated and
refuse to take anything new there despite having absolutely zero
scheduling in blk-mq.
2. It's been around for years with hundreds of thousands of users over
the years who have had no issues with it.
^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: btrfs flooding the I/O subsystem and hanging the machine, with bcache cache turned off
2016-11-30 18:16 ` Marc MERLIN
@ 2016-12-01 15:49 ` Michal Hocko
-1 siblings, 0 replies; 20+ messages in thread
From: Michal Hocko @ 2016-12-01 15:49 UTC (permalink / raw)
To: Marc MERLIN
Cc: Austin S. Hemmelgarn, Btrfs BTRFS, Vlastimil Babka, linux-mm,
Joonsoo Kim, torvalds
On Wed 30-11-16 10:16:53, Marc MERLIN wrote:
> +folks from linux-mm thread for your suggestion
>
> On Wed, Nov 30, 2016 at 01:00:45PM -0500, Austin S. Hemmelgarn wrote:
> > > swraid5 < bcache < dmcrypt < btrfs
> > >
> > > Copying with btrfs send/receive causes massive hangs on the system.
> > > Please see this explanation from Linus on why the workaround was
> > > suggested:
> > > https://lkml.org/lkml/2016/11/29/667
> > And Linux' assessment is absolutely correct (at least, the general
> > assessment is, I have no idea about btrfs_start_shared_extent, but I'm more
> > than willing to bet he's correct that that's the culprit).
>
> > > All of this mostly went away with Linus' suggestion:
> > > echo 2 > /proc/sys/vm/dirty_ratio
> > > echo 1 > /proc/sys/vm/dirty_background_ratio
> > >
> > > But that's hiding the symptom which I think is that btrfs is piling up too many I/O
> > > requests during btrfs send/receive and btrfs scrub (probably balance too) and not
> > > looking at resulting impact to system health.
>
> > I see pretty much identical behavior using any number of other storage
> > configurations on a USB 2.0 flash drive connected to a system with 16GB of
> > RAM with the default dirty ratios because it's trying to cache up to 3.2GB
> > of data for writeback. While BTRFS is doing highly sub-optimal things here,
> > the ancient default writeback ratios are just as much a culprit. I would
> > suggest that get changed to 200MB or 20% of RAM, whichever is smaller, which
> > would give overall almost identical behavior to x86-32, which in turn works
> > reasonably well for most cases. I sadly don't have the time, patience, or
> > expertise to write up such a patch myself though.
>
> Dear linux-mm folks, is that something you could consider (changing the
> dirty_ratio defaults) given that it affects at least bcache and btrfs
> (with or without bcache)?
As much as the dirty_*ratio defaults a major PITA this is not something
that would be _easy_ to change without high risks of regressions. This
topic has been discussed many times with many good ideas, nothing really
materialized from them though :/
To be honest I really do hate dirty_*ratio and have seen many issues on
very large machines and always suggested to use dirty_bytes instead but
a particular value has always been a challenge to get right. It has
always been very workload specific.
That being said this is something more for IO people than MM IMHO.
--
Michal Hocko
SUSE Labs
^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: btrfs flooding the I/O subsystem and hanging the machine, with bcache cache turned off
@ 2016-12-01 15:49 ` Michal Hocko
0 siblings, 0 replies; 20+ messages in thread
From: Michal Hocko @ 2016-12-01 15:49 UTC (permalink / raw)
To: Marc MERLIN
Cc: Austin S. Hemmelgarn, Btrfs BTRFS, Vlastimil Babka, linux-mm,
Joonsoo Kim, torvalds
On Wed 30-11-16 10:16:53, Marc MERLIN wrote:
> +folks from linux-mm thread for your suggestion
>
> On Wed, Nov 30, 2016 at 01:00:45PM -0500, Austin S. Hemmelgarn wrote:
> > > swraid5 < bcache < dmcrypt < btrfs
> > >
> > > Copying with btrfs send/receive causes massive hangs on the system.
> > > Please see this explanation from Linus on why the workaround was
> > > suggested:
> > > https://lkml.org/lkml/2016/11/29/667
> > And Linux' assessment is absolutely correct (at least, the general
> > assessment is, I have no idea about btrfs_start_shared_extent, but I'm more
> > than willing to bet he's correct that that's the culprit).
>
> > > All of this mostly went away with Linus' suggestion:
> > > echo 2 > /proc/sys/vm/dirty_ratio
> > > echo 1 > /proc/sys/vm/dirty_background_ratio
> > >
> > > But that's hiding the symptom which I think is that btrfs is piling up too many I/O
> > > requests during btrfs send/receive and btrfs scrub (probably balance too) and not
> > > looking at resulting impact to system health.
>
> > I see pretty much identical behavior using any number of other storage
> > configurations on a USB 2.0 flash drive connected to a system with 16GB of
> > RAM with the default dirty ratios because it's trying to cache up to 3.2GB
> > of data for writeback. While BTRFS is doing highly sub-optimal things here,
> > the ancient default writeback ratios are just as much a culprit. I would
> > suggest that get changed to 200MB or 20% of RAM, whichever is smaller, which
> > would give overall almost identical behavior to x86-32, which in turn works
> > reasonably well for most cases. I sadly don't have the time, patience, or
> > expertise to write up such a patch myself though.
>
> Dear linux-mm folks, is that something you could consider (changing the
> dirty_ratio defaults) given that it affects at least bcache and btrfs
> (with or without bcache)?
As much as the dirty_*ratio defaults a major PITA this is not something
that would be _easy_ to change without high risks of regressions. This
topic has been discussed many times with many good ideas, nothing really
materialized from them though :/
To be honest I really do hate dirty_*ratio and have seen many issues on
very large machines and always suggested to use dirty_bytes instead but
a particular value has always been a challenge to get right. It has
always been very workload specific.
That being said this is something more for IO people than MM IMHO.
--
Michal Hocko
SUSE Labs
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: btrfs flooding the I/O subsystem and hanging the machine, with bcache cache turned off
2016-12-01 15:49 ` Michal Hocko
(?)
@ 2016-12-01 21:21 ` Janos Toth F.
-1 siblings, 0 replies; 20+ messages in thread
From: Janos Toth F. @ 2016-12-01 21:21 UTC (permalink / raw)
To: Btrfs BTRFS
Is there any fundamental reason not to support huge writeback caches?
(I mean, besides working around bugs and/or questionably poor design
choices which no one wishes to fix.)
The obvious drawback is the increased risk of data loss upon hardware
failure or kernel panic but why couldn't the user be allowed to draw
the line between probability of data loss and potential performance
gains?
The last time I changed hardware, I put double the amount of RAM into
my little home server for the sole reason to use a relatively huge
cache, especially a huge writeback cache. Although I realized it soon
enough that writeback ratios like 20/45 will make the system unstable
(OOM reaping) even if ~90% of the memory is theoretically free = used
as some form of cache, read or write, depending on this ratio
parameter and I ended up below the default to get rid of The Reaper.
My plan was to try and decrease the fragmentation of files which are
created by dumping several parallel real-time video streams into
separate files (and also minimize the HDD head seeks due to that).
(The computer in question is on a UPS.)
On Thu, Dec 1, 2016 at 4:49 PM, Michal Hocko <mhocko@kernel.org> wrote:
> On Wed 30-11-16 10:16:53, Marc MERLIN wrote:
>> +folks from linux-mm thread for your suggestion
>>
>> On Wed, Nov 30, 2016 at 01:00:45PM -0500, Austin S. Hemmelgarn wrote:
>> > > swraid5 < bcache < dmcrypt < btrfs
>> > >
>> > > Copying with btrfs send/receive causes massive hangs on the system.
>> > > Please see this explanation from Linus on why the workaround was
>> > > suggested:
>> > > https://lkml.org/lkml/2016/11/29/667
>> > And Linux' assessment is absolutely correct (at least, the general
>> > assessment is, I have no idea about btrfs_start_shared_extent, but I'm more
>> > than willing to bet he's correct that that's the culprit).
>>
>> > > All of this mostly went away with Linus' suggestion:
>> > > echo 2 > /proc/sys/vm/dirty_ratio
>> > > echo 1 > /proc/sys/vm/dirty_background_ratio
>> > >
>> > > But that's hiding the symptom which I think is that btrfs is piling up too many I/O
>> > > requests during btrfs send/receive and btrfs scrub (probably balance too) and not
>> > > looking at resulting impact to system health.
>>
>> > I see pretty much identical behavior using any number of other storage
>> > configurations on a USB 2.0 flash drive connected to a system with 16GB of
>> > RAM with the default dirty ratios because it's trying to cache up to 3.2GB
>> > of data for writeback. While BTRFS is doing highly sub-optimal things here,
>> > the ancient default writeback ratios are just as much a culprit. I would
>> > suggest that get changed to 200MB or 20% of RAM, whichever is smaller, which
>> > would give overall almost identical behavior to x86-32, which in turn works
>> > reasonably well for most cases. I sadly don't have the time, patience, or
>> > expertise to write up such a patch myself though.
>>
>> Dear linux-mm folks, is that something you could consider (changing the
>> dirty_ratio defaults) given that it affects at least bcache and btrfs
>> (with or without bcache)?
>
> As much as the dirty_*ratio defaults a major PITA this is not something
> that would be _easy_ to change without high risks of regressions. This
> topic has been discussed many times with many good ideas, nothing really
> materialized from them though :/
>
> To be honest I really do hate dirty_*ratio and have seen many issues on
> very large machines and always suggested to use dirty_bytes instead but
> a particular value has always been a challenge to get right. It has
> always been very workload specific.
>
> That being said this is something more for IO people than MM IMHO.
>
> --
> Michal Hocko
> SUSE Labs
> --
> To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: 4.8.8, bcache deadlock and hard lockup
2016-12-01 0:09 ` Marc MERLIN
@ 2016-12-01 21:58 ` Eric Wheeler
0 siblings, 0 replies; 20+ messages in thread
From: Eric Wheeler @ 2016-12-01 21:58 UTC (permalink / raw)
To: Marc MERLIN; +Cc: Coly Li, linux-bcache, Btrfs BTRFS
On Wed, 30 Nov 2016, Marc MERLIN wrote:
> On Wed, Nov 30, 2016 at 03:57:28PM -0800, Eric Wheeler wrote:
> > > I'll start another separate thread with the btrfs folks on how much
> > > pressure is put on the system, but on your side it would be good to help
> > > ensure that bcache doesn't crash the system altogether if too many
> > > requests are allowed to pile up.
> >
> > Try BFQ. It is AWESOME and helps reduce the congestion problem with bulk
> > writes at the request queue on its way to the spinning disk or SSD:
> > http://algo.ing.unimo.it/people/paolo/disk_sched/
> >
> > use the latest BFQ git here, merge it into v4.8.y:
> > https://github.com/linusw/linux-bfq/commits/bfq-v8
> >
> > This doesn't completely fix the dirty_ration problem, but it is far better
> > than CFQ or deadline in my opinion (and experience).
>
> That's good to know thanks.
> But for my uninformed opinion, is there anything bcache can do to throttle
> incoming requests if they are piling up, or they're coming from producers
> upstream and bcache has no choice but try and process them as quickly as
> possible without a way to block the sender if too many are coming?
Not really. The congestion isn't in bcache, its at the disk queue beyond
bcache, but userspace processes are blocked by the (huge) pagecache dirty
writeback which happens before bcache gets it and must complete before
userspace may proceed:
fs -> pagecache -> bcache -> {ssd,disk}
The real issue is that the dirty page cache gets really big, flushes,
waits for downstream devices (bcache->ssd,disk) to finish, and then
returns to userspace. The only way to limit dirty cache are those options
that Linus mentioned.
BFQ can help for processes not tied to the flush because it may re-order
other process requests ahead of the big flush---so even though a big flush
is happening and that process is stalled, others might proceed without
delay.
See this thread, too:
https://groups.google.com/forum/#!msg/bfq-iosched/M2M_UhbC05A/hf6Ni9JbAQAJ
--
Eric Wheeler
>
> Thanks,
> Marc
> --
> "A mouse is a device used to point at the xterm you want to type in" - A.S.R.
> Microsoft is to operating systems ....
> .... what McDonalds is to gourmet cooking
> Home page: http://marc.merlins.org/
> --
> To unsubscribe from this list: send the line "unsubscribe linux-bcache" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
>
^ permalink raw reply [flat|nested] 20+ messages in thread
end of thread, other threads:[~2016-12-01 21:59 UTC | newest]
Thread overview: 20+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2016-11-18 16:46 4.8.8, bcache deadlock and hard lockup Marc MERLIN
2016-11-18 18:49 ` Marc MERLIN
2016-11-20 21:13 ` Coly Li
2016-11-20 21:26 ` Marc MERLIN
2016-11-21 0:04 ` Marc MERLIN
2016-11-29 20:59 ` Eric Wheeler
2016-11-30 16:46 ` Marc MERLIN
2016-11-30 17:16 ` Marc MERLIN
2016-11-30 17:18 ` btrfs flooding the I/O subsystem and hanging the machine, with bcache cache turned off Marc MERLIN
2016-11-30 18:00 ` Austin S. Hemmelgarn
2016-11-30 18:16 ` Marc MERLIN
2016-11-30 18:16 ` Marc MERLIN
2016-12-01 15:49 ` Michal Hocko
2016-12-01 15:49 ` Michal Hocko
2016-12-01 21:21 ` Janos Toth F.
2016-11-30 23:57 ` 4.8.8, bcache deadlock and hard lockup Eric Wheeler
2016-12-01 0:09 ` Marc MERLIN
2016-12-01 21:58 ` Eric Wheeler
2016-12-01 0:48 ` Chris Murphy
2016-12-01 12:30 ` Austin S. Hemmelgarn
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.