All of lore.kernel.org
 help / color / mirror / Atom feed
* [dm-crypt] Dm-integrity freeze
@ 2019-03-07 15:26 Victor Helmholtz
  2019-03-07 16:37 ` Milan Broz
  0 siblings, 1 reply; 10+ messages in thread
From: Victor Helmholtz @ 2019-03-07 15:26 UTC (permalink / raw)
  To: dm-crypt

[-- Attachment #1: Type: text/plain, Size: 2982 bytes --]

Hi

I have recently had a problem with my server: all writes to RAID drives were frozen and I had to force-reboot it. I looked at kernel logs and I think dm-integrity might be source of the problem. Could anyone help me understand meaning of following messages (this message was repeated 8 times, full kernel log is here: https://pastebin.com/DMgMtNJq <https://pastebin.com/DMgMtNJq> ):

Mar  1 23:48:21 unassigned-hostname kernel: [369732.498048] INFO: task kworker/0:112:26760 blocked for more than 120 seconds.
Mar  1 23:48:21 unassigned-hostname kernel: [369732.499784]       Not tainted 4.19.0-2-amd64 #1 Debian 4.19.16-1
Mar  1 23:48:21 unassigned-hostname kernel: [369732.501222] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Mar  1 23:48:21 unassigned-hostname kernel: [369732.503095] kworker/0:112   D    0 26760      2 0x80000080
Mar  1 23:48:21 unassigned-hostname kernel: [369732.504424] Workqueue: dm-integrity-metadata integrity_bio_wait [dm_integrity]
Mar  1 23:48:21 unassigned-hostname kernel: [369732.506149] Call Trace:
Mar  1 23:48:21 unassigned-hostname kernel: [369732.506782]  ? __schedule+0x2a2/0x870
Mar  1 23:48:21 unassigned-hostname kernel: [369732.507706]  schedule+0x28/0x80
Mar  1 23:48:21 unassigned-hostname kernel: [369732.508484]  io_schedule+0x12/0x40
Mar  1 23:48:21 unassigned-hostname kernel: [369732.509321]  wait_and_add_new_range+0x89/0xa0 [dm_integrity]
Mar  1 23:48:21 unassigned-hostname kernel: [369732.510709]  dm_integrity_map_continue+0xe71/0x1020 [dm_integrity]
Mar  1 23:48:21 unassigned-hostname kernel: [369732.512193]  ? __switch_to_asm+0x40/0x70
Mar  1 23:48:21 unassigned-hostname kernel: [369732.513149]  ? __switch_to_asm+0x34/0x70
Mar  1 23:48:21 unassigned-hostname kernel: [369732.514105]  ? __switch_to_asm+0x40/0x70
Mar  1 23:48:21 unassigned-hostname kernel: [369732.515088]  ? __switch_to_asm+0x40/0x70
Mar  1 23:48:21 unassigned-hostname kernel: [369732.516048]  ? __switch_to_asm+0x34/0x70
Mar  1 23:48:21 unassigned-hostname kernel: [369732.517000]  ? __switch_to_asm+0x40/0x70
Mar  1 23:48:21 unassigned-hostname kernel: [369732.517978]  ? __switch_to_asm+0x34/0x70
Mar  1 23:48:21 unassigned-hostname kernel: [369732.518968]  ? __switch_to_asm+0x34/0x70
Mar  1 23:48:21 unassigned-hostname kernel: [369732.519926]  ? __switch_to+0x8c/0x440
Mar  1 23:48:21 unassigned-hostname kernel: [369732.520822]  process_one_work+0x1a7/0x3a0
Mar  1 23:48:21 unassigned-hostname kernel: [369732.521798]  worker_thread+0x30/0x390
Mar  1 23:48:21 unassigned-hostname kernel: [369732.522725]  ? pwq_unbound_release_workfn+0xd0/0xd0
Mar  1 23:48:21 unassigned-hostname kernel: [369732.523901]  kthread+0x112/0x130
Mar  1 23:48:21 unassigned-hostname kernel: [369732.524691]  ? kthread_bind+0x30/0x30
Mar  1 23:48:21 unassigned-hostname kernel: [369732.525584]  ret_from_fork+0x35/0x40

Is this a bug in kernel or do I have some problems with hardware?

Regards
Victor

[-- Attachment #2: Type: text/html, Size: 4171 bytes --]

^ permalink raw reply	[flat|nested] 10+ messages in thread

* Re: [dm-crypt] Dm-integrity freeze
  2019-03-07 15:26 [dm-crypt] Dm-integrity freeze Victor Helmholtz
@ 2019-03-07 16:37 ` Milan Broz
  2019-05-21  8:43     ` [dm-crypt] " Hans van Kranenburg
  0 siblings, 1 reply; 10+ messages in thread
From: Milan Broz @ 2019-03-07 16:37 UTC (permalink / raw)
  To: Victor Helmholtz, dm-crypt; +Cc: device-mapper development, Mikulas Patocka

Hi,

from the full log it seems that it is stuck in tgtd (iSCSI).

Anyway, this is device-mapper, dm-devel is better list here.
(added to cc + Mikulas)

m.

On 3/7/19 4:26 PM, Victor Helmholtz wrote:
> Hi
> 
> I have recently had a problem with my server: all writes to RAID drives were frozen and I had to force-reboot it. I looked at kernel logs and I think dm-integrity might be source of the problem. Could anyone help me understand meaning of following messages (this message was repeated 8 times, full kernel log is here: https://pastebin.com/DMgMtNJq ):
> 
> Mar  1 23:48:21 unassigned-hostname kernel: [369732.498048] INFO: task kworker/0:112:26760 blocked for more than 120 seconds.
> Mar  1 23:48:21 unassigned-hostname kernel: [369732.499784]       Not tainted 4.19.0-2-amd64 #1 Debian 4.19.16-1
> Mar  1 23:48:21 unassigned-hostname kernel: [369732.501222] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> Mar  1 23:48:21 unassigned-hostname kernel: [369732.503095] kworker/0:112   D    0 26760      2 0x80000080
> Mar  1 23:48:21 unassigned-hostname kernel: [369732.504424] Workqueue: dm-integrity-metadata integrity_bio_wait [dm_integrity]
> Mar  1 23:48:21 unassigned-hostname kernel: [369732.506149] Call Trace:
> Mar  1 23:48:21 unassigned-hostname kernel: [369732.506782]  ? __schedule+0x2a2/0x870
> Mar  1 23:48:21 unassigned-hostname kernel: [369732.507706]  schedule+0x28/0x80
> Mar  1 23:48:21 unassigned-hostname kernel: [369732.508484]  io_schedule+0x12/0x40
> Mar  1 23:48:21 unassigned-hostname kernel: [369732.509321]  wait_and_add_new_range+0x89/0xa0 [dm_integrity]
> Mar  1 23:48:21 unassigned-hostname kernel: [369732.510709]  dm_integrity_map_continue+0xe71/0x1020 [dm_integrity]
> Mar  1 23:48:21 unassigned-hostname kernel: [369732.512193]  ? __switch_to_asm+0x40/0x70
> Mar  1 23:48:21 unassigned-hostname kernel: [369732.513149]  ? __switch_to_asm+0x34/0x70
> Mar  1 23:48:21 unassigned-hostname kernel: [369732.514105]  ? __switch_to_asm+0x40/0x70
> Mar  1 23:48:21 unassigned-hostname kernel: [369732.515088]  ? __switch_to_asm+0x40/0x70
> Mar  1 23:48:21 unassigned-hostname kernel: [369732.516048]  ? __switch_to_asm+0x34/0x70
> Mar  1 23:48:21 unassigned-hostname kernel: [369732.517000]  ? __switch_to_asm+0x40/0x70
> Mar  1 23:48:21 unassigned-hostname kernel: [369732.517978]  ? __switch_to_asm+0x34/0x70
> Mar  1 23:48:21 unassigned-hostname kernel: [369732.518968]  ? __switch_to_asm+0x34/0x70
> Mar  1 23:48:21 unassigned-hostname kernel: [369732.519926]  ? __switch_to+0x8c/0x440
> Mar  1 23:48:21 unassigned-hostname kernel: [369732.520822]  process_one_work+0x1a7/0x3a0
> Mar  1 23:48:21 unassigned-hostname kernel: [369732.521798]  worker_thread+0x30/0x390
> Mar  1 23:48:21 unassigned-hostname kernel: [369732.522725]  ? pwq_unbound_release_workfn+0xd0/0xd0
> Mar  1 23:48:21 unassigned-hostname kernel: [369732.523901]  kthread+0x112/0x130
> Mar  1 23:48:21 unassigned-hostname kernel: [369732.524691]  ? kthread_bind+0x30/0x30
> Mar  1 23:48:21 unassigned-hostname kernel: [369732.525584]  ret_from_fork+0x35/0x40
> 
> Is this a bug in kernel or do I have some problems with hardware?
> 
> Regards
> Victor
> 
> _______________________________________________
> dm-crypt mailing list
> dm-crypt@saout.de
> https://www.saout.de/mailman/listinfo/dm-crypt
> 

^ permalink raw reply	[flat|nested] 10+ messages in thread

* Re: [dm-crypt] [dm-devel]  Dm-integrity freeze
  2019-03-07 16:37 ` Milan Broz
@ 2019-05-21  8:43     ` Hans van Kranenburg
  0 siblings, 0 replies; 10+ messages in thread
From: Hans van Kranenburg @ 2019-05-21  8:43 UTC (permalink / raw)
  To: Milan Broz, Victor Helmholtz, dm-crypt
  Cc: device-mapper development, Mikulas Patocka

Hi,

I'm seeing the same lockup, also 4.19. This is mdadm RAID10 on top of 4x
a partition with only dm-integrity.

It just happened out of the blue, no heavy load or anything. All IO to
it is frozen now.

[777184.932426] INFO: task md127_raid10:507 blocked for more than 120
seconds.
[777184.932544]       Tainted: G        W         4.19.0-mx4-amd64 #1
Debian 4.19.28-2+mendix1
[777184.932666] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
disables this message.
[777184.932788] md127_raid10    D    0   507      2 0x80000000
[777184.932791] Call Trace:
[777184.932801]  ? __schedule+0x2a2/0x870
[777184.932804]  ? _raw_spin_unlock_irqrestore+0x14/0x20
[777184.932805]  schedule+0x28/0x80
[777184.932814]  md_super_wait+0x6e/0xa0 [md_mod]
[777184.932820]  ? finish_wait+0x80/0x80
[777184.932824]  write_page+0x172/0x320 [md_mod]
[777184.932826]  ? __schedule+0x2aa/0x870
[777184.932828]  ? _raw_spin_unlock_irqrestore+0x14/0x20
[777184.932832]  md_update_sb.part.64+0x382/0x8f0 [md_mod]
[777184.932837]  md_check_recovery+0x148/0x540 [md_mod]
[777184.932841]  raid10d+0x62/0x1440 [raid10]
[777184.932843]  ? __switch_to_asm+0x34/0x70
[777184.932845]  ? finish_task_switch+0x78/0x260
[777184.932847]  ? _raw_spin_lock_irqsave+0x15/0x40
[777184.932850]  ? lock_timer_base+0x67/0x80
[777184.932852]  ? _raw_spin_unlock_irqrestore+0x14/0x20
[777184.932854]  ? try_to_del_timer_sync+0x4d/0x80
[777184.932855]  ? del_timer_sync+0x35/0x40
[777184.932857]  ? schedule_timeout+0x173/0x390
[777184.932861]  ? md_rdev_init+0xb0/0xb0 [md_mod]
[777184.932864]  ? md_thread+0x94/0x150 [md_mod]
[777184.932866]  ? raid10_end_write_request+0x290/0x290 [raid10]
[777184.932869]  md_thread+0x94/0x150 [md_mod]
[777184.932872]  ? finish_wait+0x80/0x80
[777184.932873]  kthread+0x112/0x130
[777184.932875]  ? kthread_bind+0x30/0x30
[777184.932877]  ret_from_fork+0x35/0x40
[777184.932884] INFO: task 1.xvda-1:1657 blocked for more than 120 seconds.
[777184.932979]       Tainted: G        W         4.19.0-mx4-amd64 #1
Debian 4.19.28-2+mendix1
[777184.933101] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
disables this message.
[777184.933222] 1.xvda-1        D    0  1657      2 0x80000000
[777184.933223] Call Trace:
[777184.933225]  ? __schedule+0x2a2/0x870
[777184.933227]  ? _raw_spin_unlock_irqrestore+0x14/0x20
[777184.933228]  schedule+0x28/0x80
[777184.933232]  md_write_start+0x14b/0x220 [md_mod]
[777184.933234]  ? finish_wait+0x80/0x80
[777184.933236]  raid10_make_request+0x3d/0x130 [raid10]
[777184.933239]  ? finish_wait+0x80/0x80
[777184.933242]  md_handle_request+0x119/0x190 [md_mod]
[777184.933247]  md_make_request+0x78/0x160 [md_mod]
[777184.933251]  generic_make_request+0x1a4/0x410
[777184.933254]  ? bvec_alloc+0x51/0xe0
[777184.933256]  submit_bio+0x45/0x140
[777184.933258]  ? bio_add_page+0x48/0x60
[777184.933263]  dispatch_rw_block_io+0x68f/0x990 [xen_blkback]
[777184.933266]  ? branch_type+0x2e4/0x350
[777184.933269]  __do_block_io_op+0x2fc/0x5f0 [xen_blkback]
[777184.933271]  ? _raw_spin_unlock_irqrestore+0x14/0x20
[777184.933272]  ? try_to_del_timer_sync+0x4d/0x80
[777184.933275]  xen_blkif_schedule+0xce/0x620 [xen_blkback]
[777184.933277]  ? finish_wait+0x80/0x80
[777184.933279]  ? xen_blkif_be_int+0x30/0x30 [xen_blkback]
[777184.933281]  kthread+0x112/0x130
[777184.933282]  ? kthread_bind+0x30/0x30
[777184.933283]  ret_from_fork+0x35/0x40
[777184.933293] INFO: task 11.xvda-0:14864 blocked for more than 120
seconds.
[777184.933387]       Tainted: G        W         4.19.0-mx4-amd64 #1
Debian 4.19.28-2+mendix1
[777184.933509] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
disables this message.
[777184.933638] 11.xvda-0       D    0 14864      2 0x80000000
[777184.933640] Call Trace:
[777184.933642]  ? __schedule+0x2a2/0x870
[777184.933643]  ? _raw_spin_unlock_irqrestore+0x14/0x20
[777184.933644]  schedule+0x28/0x80
[777184.933648]  md_write_start+0x14b/0x220 [md_mod]
[777184.933650]  ? finish_wait+0x80/0x80
[777184.933652]  raid10_make_request+0x3d/0x130 [raid10]
[777184.933654]  ? finish_wait+0x80/0x80
[777184.933658]  md_handle_request+0x119/0x190 [md_mod]
[777184.933662]  md_make_request+0x78/0x160 [md_mod]
[777184.933664]  generic_make_request+0x1a4/0x410
[777184.933666]  submit_bio+0x45/0x140
[777184.933668]  ? bio_add_page+0x48/0x60
[777184.933670]  dispatch_rw_block_io+0x68f/0x990 [xen_blkback]
[777184.933672]  ? branch_type+0x2e7/0x350
[777184.933674]  __do_block_io_op+0x2fc/0x5f0 [xen_blkback]
[777184.933676]  ? _raw_spin_unlock_irqrestore+0x14/0x20
[777184.933678]  ? try_to_del_timer_sync+0x4d/0x80
[777184.933680]  xen_blkif_schedule+0xce/0x620 [xen_blkback]
[777184.933682]  ? __schedule+0x2aa/0x870
[777184.933684]  ? finish_wait+0x80/0x80
[777184.933686]  ? xen_blkif_be_int+0x30/0x30 [xen_blkback]
[777184.933687]  kthread+0x112/0x130
[777184.933688]  ? kthread_bind+0x30/0x30
[777184.933689]  ret_from_fork+0x35/0x40
[777184.933694] INFO: task btrfs-transacti:15468 blocked for more than
120 seconds.
[777184.933815]       Tainted: G        W         4.19.0-mx4-amd64 #1
Debian 4.19.28-2+mendix1
[777184.933937] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
disables this message.
[777184.934058] btrfs-transacti D    0 15468      2 0x80000000
[777184.934059] Call Trace:
[777184.934061]  ? __schedule+0x2a2/0x870
[777184.934062]  ? _raw_spin_unlock_irqrestore+0x14/0x20
[777184.934063]  schedule+0x28/0x80
[777184.934067]  md_write_start+0x14b/0x220 [md_mod]
[777184.934069]  ? finish_wait+0x80/0x80
[777184.934072]  raid10_make_request+0x3d/0x130 [raid10]
[777184.934074]  ? finish_wait+0x80/0x80
[777184.934077]  md_handle_request+0x119/0x190 [md_mod]
[777184.934081]  md_make_request+0x78/0x160 [md_mod]
[777184.934083]  generic_make_request+0x1a4/0x410
[777184.934085]  submit_bio+0x45/0x140
[777184.934086]  ? bio_clone_fast+0x2c/0x60
[777184.934119]  btrfs_map_bio+0x20a/0x350 [btrfs]
[777184.934136]  btree_submit_bio_hook+0x8d/0xc0 [btrfs]
[777184.934155]  submit_one_bio+0x5f/0x80 [btrfs]
[777184.934173]  flush_write_bio.isra.40+0x1d/0x30 [btrfs]
[777184.934191]  btree_write_cache_pages+0x243/0x360 [btrfs]
[777184.934210]  ? alloc_extent_state+0x1e/0xc0 [btrfs]
[777184.934226]  ? btrfs_buffer_uptodate+0x33/0x50 [btrfs]
[777184.934238]  ? read_block_for_search.isra.36+0xf2/0x340 [btrfs]
[777184.934241]  do_writepages+0x41/0xd0
[777184.934244]  ? __slab_alloc+0x27/0x30
[777184.934261]  ? merge_state.part.45+0x3f/0x130 [btrfs]
[777184.934262]  ? kmem_cache_alloc+0x15c/0x1c0
[777184.934279]  ? clear_state_bit+0xdb/0x180 [btrfs]
[777184.934281]  __filemap_fdatawrite_range+0xbe/0xf0
[777184.934298]  btrfs_write_marked_extents+0x68/0x150 [btrfs]
[777184.934314]  btrfs_write_and_wait_transaction.isra.21+0x4d/0xa0 [btrfs]
[777184.934331]  btrfs_commit_transaction+0x56c/0x870 [btrfs]
[777184.934347]  ? start_transaction+0x9d/0x3f0 [btrfs]
[777184.934362]  transaction_kthread+0x147/0x180 [btrfs]
[777184.934379]  ? btrfs_cleanup_transaction+0x530/0x530 [btrfs]
[777184.934380]  kthread+0x112/0x130
[777184.934381]  ? kthread_bind+0x30/0x30
[777184.934383]  ret_from_fork+0x35/0x40
[777184.934391] INFO: task kworker/1:3:16943 blocked for more than 120
seconds.
[777184.934487]       Tainted: G        W         4.19.0-mx4-amd64 #1
Debian 4.19.28-2+mendix1
[777184.934617] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
disables this message.
[777184.934747] kworker/1:3     D    0 16943      2 0x80000000
[777184.934753] Workqueue: dm-integrity-metadata integrity_bio_wait
[dm_integrity]
[777184.934754] Call Trace:
[777184.934756]  ? __schedule+0x2a2/0x870
[777184.934757]  schedule+0x28/0x80
[777184.934760]  io_schedule+0x12/0x40
[777184.934763]  wait_and_add_new_range+0x89/0xa0 [dm_integrity]
[777184.934766]  dm_integrity_map_continue+0xe71/0x1020 [dm_integrity]
[777184.934769]  ? xen_load_sp0+0x77/0x170
[777184.934771]  ? __switch_to+0x1e1/0x440
[777184.934773]  ? finish_task_switch+0x78/0x260
[777184.934775]  process_one_work+0x1a7/0x3a0
[777184.934777]  worker_thread+0x30/0x390
[777184.934779]  ? create_worker+0x1a0/0x1a0
[777184.934781]  kthread+0x112/0x130
[777184.934782]  ? kthread_bind+0x30/0x30
[777184.934783]  ret_from_fork+0x35/0x40
[777184.934787] INFO: task kworker/0:8:17282 blocked for more than 120
seconds.
[777184.934882]       Tainted: G        W         4.19.0-mx4-amd64 #1
Debian 4.19.28-2+mendix1
[777184.935005] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
disables this message.
[777184.935125] kworker/0:8     D    0 17282      2 0x80000000
[777184.935129] Workqueue: dm-integrity-metadata integrity_bio_wait
[dm_integrity]
[777184.935130] Call Trace:
[777184.935132]  ? __schedule+0x2a2/0x870
[777184.935133]  schedule+0x28/0x80
[777184.935135]  io_schedule+0x12/0x40
[777184.935137]  wait_and_add_new_range+0x89/0xa0 [dm_integrity]
[777184.935139]  dm_integrity_map_continue+0xe71/0x1020 [dm_integrity]
[777184.935141]  ? xen_load_sp0+0x77/0x170
[777184.935143]  ? __switch_to+0x1e1/0x440
[777184.935144]  ? finish_task_switch+0x78/0x260
[777184.935146]  process_one_work+0x1a7/0x3a0
[777184.935148]  worker_thread+0x30/0x390
[777184.935149]  ? create_worker+0x1a0/0x1a0
[777184.935150]  kthread+0x112/0x130
[777184.935151]  ? kthread_bind+0x30/0x30
[777184.935153]  ret_from_fork+0x35/0x40
[777184.935156] INFO: task kworker/u8:9:17284 blocked for more than 120
seconds.
[777184.935251]       Tainted: G        W         4.19.0-mx4-amd64 #1
Debian 4.19.28-2+mendix1
[777184.935373] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
disables this message.
[777184.935494] kworker/u8:9    D    0 17284      2 0x80000000
[777184.935515] Workqueue: btrfs-submit btrfs_submit_helper [btrfs]
[777184.935515] Call Trace:
[777184.935517]  ? __schedule+0x2a2/0x870
[777184.935519]  ? _raw_spin_unlock_irqrestore+0x14/0x20
[777184.935520]  schedule+0x28/0x80
[777184.935525]  md_write_start+0x14b/0x220 [md_mod]
[777184.935527]  ? finish_wait+0x80/0x80
[777184.935529]  raid10_make_request+0x3d/0x130 [raid10]
[777184.935531]  ? finish_wait+0x80/0x80
[777184.935535]  md_handle_request+0x119/0x190 [md_mod]
[777184.935539]  md_make_request+0x78/0x160 [md_mod]
[777184.935541]  generic_make_request+0x1a4/0x410
[777184.935544]  submit_bio+0x45/0x140
[777184.935562]  run_scheduled_bios+0x18e/0x440 [btrfs]
[777184.935581]  normal_work_helper+0xba/0x300 [btrfs]
[777184.935584]  process_one_work+0x1a7/0x3a0
[777184.935586]  worker_thread+0x30/0x390
[777184.935587]  ? create_worker+0x1a0/0x1a0
[777184.935588]  kthread+0x112/0x130
[777184.935589]  ? kthread_bind+0x30/0x30
[777184.935591]  ret_from_fork+0x35/0x40
[777184.935593] INFO: task kworker/1:13:17315 blocked for more than 120
seconds.
[777184.935713]       Tainted: G        W         4.19.0-mx4-amd64 #1
Debian 4.19.28-2+mendix1
[777184.935836] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
disables this message.
[777184.935957] kworker/1:13    D    0 17315      2 0x80000000
[777184.935960] Workqueue: dm-integrity-metadata integrity_bio_wait
[dm_integrity]
[777184.935961] Call Trace:
[777184.935963]  ? __schedule+0x2a2/0x870
[777184.935965]  schedule+0x28/0x80
[777184.935967]  io_schedule+0x12/0x40
[777184.935969]  wait_and_add_new_range+0x89/0xa0 [dm_integrity]
[777184.935971]  dm_integrity_map_continue+0xe71/0x1020 [dm_integrity]
[777184.935973]  ? xen_load_sp0+0x77/0x170
[777184.935975]  ? __switch_to+0x1e1/0x440
[777184.935976]  ? finish_task_switch+0x78/0x260
[777184.935978]  process_one_work+0x1a7/0x3a0
[777184.935980]  worker_thread+0x30/0x390
[777184.935981]  ? create_worker+0x1a0/0x1a0
[777184.935982]  kthread+0x112/0x130
[777184.935983]  ? kthread_bind+0x30/0x30
[777184.935985]  ret_from_fork+0x35/0x40
[777184.935988] INFO: task kworker/0:20:17330 blocked for more than 120
seconds.
[777184.936083]       Tainted: G        W         4.19.0-mx4-amd64 #1
Debian 4.19.28-2+mendix1
[777184.936205] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
disables this message.
[777184.936355] kworker/0:20    D    0 17330      2 0x80000000
[777184.936359] Workqueue: dm-integrity-metadata integrity_bio_wait
[dm_integrity]
[777184.936360] Call Trace:
[777184.936363]  ? __schedule+0x2a2/0x870
[777184.936364]  schedule+0x28/0x80
[777184.936366]  io_schedule+0x12/0x40
[777184.936368]  wait_and_add_new_range+0x89/0xa0 [dm_integrity]
[777184.936370]  dm_integrity_map_continue+0xe71/0x1020 [dm_integrity]
[777184.936373]  ? xen_load_sp0+0x77/0x170
[777184.936374]  ? __switch_to+0x1e1/0x440
[777184.936375]  ? finish_task_switch+0x78/0x260
[777184.936377]  process_one_work+0x1a7/0x3a0
[777184.936379]  worker_thread+0x30/0x390
[777184.936381]  ? create_worker+0x1a0/0x1a0
[777184.936382]  kthread+0x112/0x130
[777184.936383]  ? kthread_bind+0x30/0x30
[777184.936385]  ret_from_fork+0x35/0x40
[777184.936390] INFO: task kworker/1:23:17371 blocked for more than 120
seconds.
[777184.936500]       Tainted: G        W         4.19.0-mx4-amd64 #1
Debian 4.19.28-2+mendix1
[777184.936625] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
disables this message.
[777184.936762] kworker/1:23    D    0 17371      2 0x80000000
[777184.936775] Workqueue: dm-integrity-metadata integrity_bio_wait
[dm_integrity]
[777184.936786] Call Trace:
[777184.936791]  ? __schedule+0x2a2/0x870
[777184.936798]  schedule+0x28/0x80
[777184.936805]  io_schedule+0x12/0x40
[777184.936812]  wait_and_add_new_range+0x89/0xa0 [dm_integrity]
[777184.936819]  dm_integrity_map_continue+0xe71/0x1020 [dm_integrity]
[777184.936828]  ? xen_load_sp0+0x77/0x170
[777184.936834]  ? __switch_to+0x1e1/0x440
[777184.936842]  ? finish_task_switch+0x78/0x260
[777184.936849]  process_one_work+0x1a7/0x3a0
[777184.936856]  worker_thread+0x30/0x390
[777184.936862]  ? create_worker+0x1a0/0x1a0
[777184.936868]  kthread+0x112/0x130
[777184.936875]  ? kthread_bind+0x30/0x30
[777184.936880]  ret_from_fork+0x35/0x40

Hans

On 3/7/19 5:37 PM, Milan Broz wrote:
> Hi,
> 
> from the full log it seems that it is stuck in tgtd (iSCSI).
> 
> Anyway, this is device-mapper, dm-devel is better list here.
> (added to cc + Mikulas)
> 
> m.
> 
> On 3/7/19 4:26 PM, Victor Helmholtz wrote:
>> Hi
>>
>> I have recently had a problem with my server: all writes to RAID drives were frozen and I had to force-reboot it. I looked at kernel logs and I think dm-integrity might be source of the problem. Could anyone help me understand meaning of following messages (this message was repeated 8 times, full kernel log is here: https://pastebin.com/DMgMtNJq ):
>>
>> Mar  1 23:48:21 unassigned-hostname kernel: [369732.498048] INFO: task kworker/0:112:26760 blocked for more than 120 seconds.
>> Mar  1 23:48:21 unassigned-hostname kernel: [369732.499784]       Not tainted 4.19.0-2-amd64 #1 Debian 4.19.16-1
>> Mar  1 23:48:21 unassigned-hostname kernel: [369732.501222] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
>> Mar  1 23:48:21 unassigned-hostname kernel: [369732.503095] kworker/0:112   D    0 26760      2 0x80000080
>> Mar  1 23:48:21 unassigned-hostname kernel: [369732.504424] Workqueue: dm-integrity-metadata integrity_bio_wait [dm_integrity]
>> Mar  1 23:48:21 unassigned-hostname kernel: [369732.506149] Call Trace:
>> Mar  1 23:48:21 unassigned-hostname kernel: [369732.506782]  ? __schedule+0x2a2/0x870
>> Mar  1 23:48:21 unassigned-hostname kernel: [369732.507706]  schedule+0x28/0x80
>> Mar  1 23:48:21 unassigned-hostname kernel: [369732.508484]  io_schedule+0x12/0x40
>> Mar  1 23:48:21 unassigned-hostname kernel: [369732.509321]  wait_and_add_new_range+0x89/0xa0 [dm_integrity]
>> Mar  1 23:48:21 unassigned-hostname kernel: [369732.510709]  dm_integrity_map_continue+0xe71/0x1020 [dm_integrity]
>> Mar  1 23:48:21 unassigned-hostname kernel: [369732.512193]  ? __switch_to_asm+0x40/0x70
>> Mar  1 23:48:21 unassigned-hostname kernel: [369732.513149]  ? __switch_to_asm+0x34/0x70
>> Mar  1 23:48:21 unassigned-hostname kernel: [369732.514105]  ? __switch_to_asm+0x40/0x70
>> Mar  1 23:48:21 unassigned-hostname kernel: [369732.515088]  ? __switch_to_asm+0x40/0x70
>> Mar  1 23:48:21 unassigned-hostname kernel: [369732.516048]  ? __switch_to_asm+0x34/0x70
>> Mar  1 23:48:21 unassigned-hostname kernel: [369732.517000]  ? __switch_to_asm+0x40/0x70
>> Mar  1 23:48:21 unassigned-hostname kernel: [369732.517978]  ? __switch_to_asm+0x34/0x70
>> Mar  1 23:48:21 unassigned-hostname kernel: [369732.518968]  ? __switch_to_asm+0x34/0x70
>> Mar  1 23:48:21 unassigned-hostname kernel: [369732.519926]  ? __switch_to+0x8c/0x440
>> Mar  1 23:48:21 unassigned-hostname kernel: [369732.520822]  process_one_work+0x1a7/0x3a0
>> Mar  1 23:48:21 unassigned-hostname kernel: [369732.521798]  worker_thread+0x30/0x390
>> Mar  1 23:48:21 unassigned-hostname kernel: [369732.522725]  ? pwq_unbound_release_workfn+0xd0/0xd0
>> Mar  1 23:48:21 unassigned-hostname kernel: [369732.523901]  kthread+0x112/0x130
>> Mar  1 23:48:21 unassigned-hostname kernel: [369732.524691]  ? kthread_bind+0x30/0x30
>> Mar  1 23:48:21 unassigned-hostname kernel: [369732.525584]  ret_from_fork+0x35/0x40
>>
>> Is this a bug in kernel or do I have some problems with hardware?

^ permalink raw reply	[flat|nested] 10+ messages in thread

* Re: [dm-crypt] Dm-integrity freeze
@ 2019-05-21  8:43     ` Hans van Kranenburg
  0 siblings, 0 replies; 10+ messages in thread
From: Hans van Kranenburg @ 2019-05-21  8:43 UTC (permalink / raw)
  To: Milan Broz, Victor Helmholtz, dm-crypt
  Cc: device-mapper development, Mikulas Patocka

Hi,

I'm seeing the same lockup, also 4.19. This is mdadm RAID10 on top of 4x
a partition with only dm-integrity.

It just happened out of the blue, no heavy load or anything. All IO to
it is frozen now.

[777184.932426] INFO: task md127_raid10:507 blocked for more than 120
seconds.
[777184.932544]       Tainted: G        W         4.19.0-mx4-amd64 #1
Debian 4.19.28-2+mendix1
[777184.932666] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
disables this message.
[777184.932788] md127_raid10    D    0   507      2 0x80000000
[777184.932791] Call Trace:
[777184.932801]  ? __schedule+0x2a2/0x870
[777184.932804]  ? _raw_spin_unlock_irqrestore+0x14/0x20
[777184.932805]  schedule+0x28/0x80
[777184.932814]  md_super_wait+0x6e/0xa0 [md_mod]
[777184.932820]  ? finish_wait+0x80/0x80
[777184.932824]  write_page+0x172/0x320 [md_mod]
[777184.932826]  ? __schedule+0x2aa/0x870
[777184.932828]  ? _raw_spin_unlock_irqrestore+0x14/0x20
[777184.932832]  md_update_sb.part.64+0x382/0x8f0 [md_mod]
[777184.932837]  md_check_recovery+0x148/0x540 [md_mod]
[777184.932841]  raid10d+0x62/0x1440 [raid10]
[777184.932843]  ? __switch_to_asm+0x34/0x70
[777184.932845]  ? finish_task_switch+0x78/0x260
[777184.932847]  ? _raw_spin_lock_irqsave+0x15/0x40
[777184.932850]  ? lock_timer_base+0x67/0x80
[777184.932852]  ? _raw_spin_unlock_irqrestore+0x14/0x20
[777184.932854]  ? try_to_del_timer_sync+0x4d/0x80
[777184.932855]  ? del_timer_sync+0x35/0x40
[777184.932857]  ? schedule_timeout+0x173/0x390
[777184.932861]  ? md_rdev_init+0xb0/0xb0 [md_mod]
[777184.932864]  ? md_thread+0x94/0x150 [md_mod]
[777184.932866]  ? raid10_end_write_request+0x290/0x290 [raid10]
[777184.932869]  md_thread+0x94/0x150 [md_mod]
[777184.932872]  ? finish_wait+0x80/0x80
[777184.932873]  kthread+0x112/0x130
[777184.932875]  ? kthread_bind+0x30/0x30
[777184.932877]  ret_from_fork+0x35/0x40
[777184.932884] INFO: task 1.xvda-1:1657 blocked for more than 120 seconds.
[777184.932979]       Tainted: G        W         4.19.0-mx4-amd64 #1
Debian 4.19.28-2+mendix1
[777184.933101] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
disables this message.
[777184.933222] 1.xvda-1        D    0  1657      2 0x80000000
[777184.933223] Call Trace:
[777184.933225]  ? __schedule+0x2a2/0x870
[777184.933227]  ? _raw_spin_unlock_irqrestore+0x14/0x20
[777184.933228]  schedule+0x28/0x80
[777184.933232]  md_write_start+0x14b/0x220 [md_mod]
[777184.933234]  ? finish_wait+0x80/0x80
[777184.933236]  raid10_make_request+0x3d/0x130 [raid10]
[777184.933239]  ? finish_wait+0x80/0x80
[777184.933242]  md_handle_request+0x119/0x190 [md_mod]
[777184.933247]  md_make_request+0x78/0x160 [md_mod]
[777184.933251]  generic_make_request+0x1a4/0x410
[777184.933254]  ? bvec_alloc+0x51/0xe0
[777184.933256]  submit_bio+0x45/0x140
[777184.933258]  ? bio_add_page+0x48/0x60
[777184.933263]  dispatch_rw_block_io+0x68f/0x990 [xen_blkback]
[777184.933266]  ? branch_type+0x2e4/0x350
[777184.933269]  __do_block_io_op+0x2fc/0x5f0 [xen_blkback]
[777184.933271]  ? _raw_spin_unlock_irqrestore+0x14/0x20
[777184.933272]  ? try_to_del_timer_sync+0x4d/0x80
[777184.933275]  xen_blkif_schedule+0xce/0x620 [xen_blkback]
[777184.933277]  ? finish_wait+0x80/0x80
[777184.933279]  ? xen_blkif_be_int+0x30/0x30 [xen_blkback]
[777184.933281]  kthread+0x112/0x130
[777184.933282]  ? kthread_bind+0x30/0x30
[777184.933283]  ret_from_fork+0x35/0x40
[777184.933293] INFO: task 11.xvda-0:14864 blocked for more than 120
seconds.
[777184.933387]       Tainted: G        W         4.19.0-mx4-amd64 #1
Debian 4.19.28-2+mendix1
[777184.933509] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
disables this message.
[777184.933638] 11.xvda-0       D    0 14864      2 0x80000000
[777184.933640] Call Trace:
[777184.933642]  ? __schedule+0x2a2/0x870
[777184.933643]  ? _raw_spin_unlock_irqrestore+0x14/0x20
[777184.933644]  schedule+0x28/0x80
[777184.933648]  md_write_start+0x14b/0x220 [md_mod]
[777184.933650]  ? finish_wait+0x80/0x80
[777184.933652]  raid10_make_request+0x3d/0x130 [raid10]
[777184.933654]  ? finish_wait+0x80/0x80
[777184.933658]  md_handle_request+0x119/0x190 [md_mod]
[777184.933662]  md_make_request+0x78/0x160 [md_mod]
[777184.933664]  generic_make_request+0x1a4/0x410
[777184.933666]  submit_bio+0x45/0x140
[777184.933668]  ? bio_add_page+0x48/0x60
[777184.933670]  dispatch_rw_block_io+0x68f/0x990 [xen_blkback]
[777184.933672]  ? branch_type+0x2e7/0x350
[777184.933674]  __do_block_io_op+0x2fc/0x5f0 [xen_blkback]
[777184.933676]  ? _raw_spin_unlock_irqrestore+0x14/0x20
[777184.933678]  ? try_to_del_timer_sync+0x4d/0x80
[777184.933680]  xen_blkif_schedule+0xce/0x620 [xen_blkback]
[777184.933682]  ? __schedule+0x2aa/0x870
[777184.933684]  ? finish_wait+0x80/0x80
[777184.933686]  ? xen_blkif_be_int+0x30/0x30 [xen_blkback]
[777184.933687]  kthread+0x112/0x130
[777184.933688]  ? kthread_bind+0x30/0x30
[777184.933689]  ret_from_fork+0x35/0x40
[777184.933694] INFO: task btrfs-transacti:15468 blocked for more than
120 seconds.
[777184.933815]       Tainted: G        W         4.19.0-mx4-amd64 #1
Debian 4.19.28-2+mendix1
[777184.933937] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
disables this message.
[777184.934058] btrfs-transacti D    0 15468      2 0x80000000
[777184.934059] Call Trace:
[777184.934061]  ? __schedule+0x2a2/0x870
[777184.934062]  ? _raw_spin_unlock_irqrestore+0x14/0x20
[777184.934063]  schedule+0x28/0x80
[777184.934067]  md_write_start+0x14b/0x220 [md_mod]
[777184.934069]  ? finish_wait+0x80/0x80
[777184.934072]  raid10_make_request+0x3d/0x130 [raid10]
[777184.934074]  ? finish_wait+0x80/0x80
[777184.934077]  md_handle_request+0x119/0x190 [md_mod]
[777184.934081]  md_make_request+0x78/0x160 [md_mod]
[777184.934083]  generic_make_request+0x1a4/0x410
[777184.934085]  submit_bio+0x45/0x140
[777184.934086]  ? bio_clone_fast+0x2c/0x60
[777184.934119]  btrfs_map_bio+0x20a/0x350 [btrfs]
[777184.934136]  btree_submit_bio_hook+0x8d/0xc0 [btrfs]
[777184.934155]  submit_one_bio+0x5f/0x80 [btrfs]
[777184.934173]  flush_write_bio.isra.40+0x1d/0x30 [btrfs]
[777184.934191]  btree_write_cache_pages+0x243/0x360 [btrfs]
[777184.934210]  ? alloc_extent_state+0x1e/0xc0 [btrfs]
[777184.934226]  ? btrfs_buffer_uptodate+0x33/0x50 [btrfs]
[777184.934238]  ? read_block_for_search.isra.36+0xf2/0x340 [btrfs]
[777184.934241]  do_writepages+0x41/0xd0
[777184.934244]  ? __slab_alloc+0x27/0x30
[777184.934261]  ? merge_state.part.45+0x3f/0x130 [btrfs]
[777184.934262]  ? kmem_cache_alloc+0x15c/0x1c0
[777184.934279]  ? clear_state_bit+0xdb/0x180 [btrfs]
[777184.934281]  __filemap_fdatawrite_range+0xbe/0xf0
[777184.934298]  btrfs_write_marked_extents+0x68/0x150 [btrfs]
[777184.934314]  btrfs_write_and_wait_transaction.isra.21+0x4d/0xa0 [btrfs]
[777184.934331]  btrfs_commit_transaction+0x56c/0x870 [btrfs]
[777184.934347]  ? start_transaction+0x9d/0x3f0 [btrfs]
[777184.934362]  transaction_kthread+0x147/0x180 [btrfs]
[777184.934379]  ? btrfs_cleanup_transaction+0x530/0x530 [btrfs]
[777184.934380]  kthread+0x112/0x130
[777184.934381]  ? kthread_bind+0x30/0x30
[777184.934383]  ret_from_fork+0x35/0x40
[777184.934391] INFO: task kworker/1:3:16943 blocked for more than 120
seconds.
[777184.934487]       Tainted: G        W         4.19.0-mx4-amd64 #1
Debian 4.19.28-2+mendix1
[777184.934617] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
disables this message.
[777184.934747] kworker/1:3     D    0 16943      2 0x80000000
[777184.934753] Workqueue: dm-integrity-metadata integrity_bio_wait
[dm_integrity]
[777184.934754] Call Trace:
[777184.934756]  ? __schedule+0x2a2/0x870
[777184.934757]  schedule+0x28/0x80
[777184.934760]  io_schedule+0x12/0x40
[777184.934763]  wait_and_add_new_range+0x89/0xa0 [dm_integrity]
[777184.934766]  dm_integrity_map_continue+0xe71/0x1020 [dm_integrity]
[777184.934769]  ? xen_load_sp0+0x77/0x170
[777184.934771]  ? __switch_to+0x1e1/0x440
[777184.934773]  ? finish_task_switch+0x78/0x260
[777184.934775]  process_one_work+0x1a7/0x3a0
[777184.934777]  worker_thread+0x30/0x390
[777184.934779]  ? create_worker+0x1a0/0x1a0
[777184.934781]  kthread+0x112/0x130
[777184.934782]  ? kthread_bind+0x30/0x30
[777184.934783]  ret_from_fork+0x35/0x40
[777184.934787] INFO: task kworker/0:8:17282 blocked for more than 120
seconds.
[777184.934882]       Tainted: G        W         4.19.0-mx4-amd64 #1
Debian 4.19.28-2+mendix1
[777184.935005] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
disables this message.
[777184.935125] kworker/0:8     D    0 17282      2 0x80000000
[777184.935129] Workqueue: dm-integrity-metadata integrity_bio_wait
[dm_integrity]
[777184.935130] Call Trace:
[777184.935132]  ? __schedule+0x2a2/0x870
[777184.935133]  schedule+0x28/0x80
[777184.935135]  io_schedule+0x12/0x40
[777184.935137]  wait_and_add_new_range+0x89/0xa0 [dm_integrity]
[777184.935139]  dm_integrity_map_continue+0xe71/0x1020 [dm_integrity]
[777184.935141]  ? xen_load_sp0+0x77/0x170
[777184.935143]  ? __switch_to+0x1e1/0x440
[777184.935144]  ? finish_task_switch+0x78/0x260
[777184.935146]  process_one_work+0x1a7/0x3a0
[777184.935148]  worker_thread+0x30/0x390
[777184.935149]  ? create_worker+0x1a0/0x1a0
[777184.935150]  kthread+0x112/0x130
[777184.935151]  ? kthread_bind+0x30/0x30
[777184.935153]  ret_from_fork+0x35/0x40
[777184.935156] INFO: task kworker/u8:9:17284 blocked for more than 120
seconds.
[777184.935251]       Tainted: G        W         4.19.0-mx4-amd64 #1
Debian 4.19.28-2+mendix1
[777184.935373] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
disables this message.
[777184.935494] kworker/u8:9    D    0 17284      2 0x80000000
[777184.935515] Workqueue: btrfs-submit btrfs_submit_helper [btrfs]
[777184.935515] Call Trace:
[777184.935517]  ? __schedule+0x2a2/0x870
[777184.935519]  ? _raw_spin_unlock_irqrestore+0x14/0x20
[777184.935520]  schedule+0x28/0x80
[777184.935525]  md_write_start+0x14b/0x220 [md_mod]
[777184.935527]  ? finish_wait+0x80/0x80
[777184.935529]  raid10_make_request+0x3d/0x130 [raid10]
[777184.935531]  ? finish_wait+0x80/0x80
[777184.935535]  md_handle_request+0x119/0x190 [md_mod]
[777184.935539]  md_make_request+0x78/0x160 [md_mod]
[777184.935541]  generic_make_request+0x1a4/0x410
[777184.935544]  submit_bio+0x45/0x140
[777184.935562]  run_scheduled_bios+0x18e/0x440 [btrfs]
[777184.935581]  normal_work_helper+0xba/0x300 [btrfs]
[777184.935584]  process_one_work+0x1a7/0x3a0
[777184.935586]  worker_thread+0x30/0x390
[777184.935587]  ? create_worker+0x1a0/0x1a0
[777184.935588]  kthread+0x112/0x130
[777184.935589]  ? kthread_bind+0x30/0x30
[777184.935591]  ret_from_fork+0x35/0x40
[777184.935593] INFO: task kworker/1:13:17315 blocked for more than 120
seconds.
[777184.935713]       Tainted: G        W         4.19.0-mx4-amd64 #1
Debian 4.19.28-2+mendix1
[777184.935836] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
disables this message.
[777184.935957] kworker/1:13    D    0 17315      2 0x80000000
[777184.935960] Workqueue: dm-integrity-metadata integrity_bio_wait
[dm_integrity]
[777184.935961] Call Trace:
[777184.935963]  ? __schedule+0x2a2/0x870
[777184.935965]  schedule+0x28/0x80
[777184.935967]  io_schedule+0x12/0x40
[777184.935969]  wait_and_add_new_range+0x89/0xa0 [dm_integrity]
[777184.935971]  dm_integrity_map_continue+0xe71/0x1020 [dm_integrity]
[777184.935973]  ? xen_load_sp0+0x77/0x170
[777184.935975]  ? __switch_to+0x1e1/0x440
[777184.935976]  ? finish_task_switch+0x78/0x260
[777184.935978]  process_one_work+0x1a7/0x3a0
[777184.935980]  worker_thread+0x30/0x390
[777184.935981]  ? create_worker+0x1a0/0x1a0
[777184.935982]  kthread+0x112/0x130
[777184.935983]  ? kthread_bind+0x30/0x30
[777184.935985]  ret_from_fork+0x35/0x40
[777184.935988] INFO: task kworker/0:20:17330 blocked for more than 120
seconds.
[777184.936083]       Tainted: G        W         4.19.0-mx4-amd64 #1
Debian 4.19.28-2+mendix1
[777184.936205] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
disables this message.
[777184.936355] kworker/0:20    D    0 17330      2 0x80000000
[777184.936359] Workqueue: dm-integrity-metadata integrity_bio_wait
[dm_integrity]
[777184.936360] Call Trace:
[777184.936363]  ? __schedule+0x2a2/0x870
[777184.936364]  schedule+0x28/0x80
[777184.936366]  io_schedule+0x12/0x40
[777184.936368]  wait_and_add_new_range+0x89/0xa0 [dm_integrity]
[777184.936370]  dm_integrity_map_continue+0xe71/0x1020 [dm_integrity]
[777184.936373]  ? xen_load_sp0+0x77/0x170
[777184.936374]  ? __switch_to+0x1e1/0x440
[777184.936375]  ? finish_task_switch+0x78/0x260
[777184.936377]  process_one_work+0x1a7/0x3a0
[777184.936379]  worker_thread+0x30/0x390
[777184.936381]  ? create_worker+0x1a0/0x1a0
[777184.936382]  kthread+0x112/0x130
[777184.936383]  ? kthread_bind+0x30/0x30
[777184.936385]  ret_from_fork+0x35/0x40
[777184.936390] INFO: task kworker/1:23:17371 blocked for more than 120
seconds.
[777184.936500]       Tainted: G        W         4.19.0-mx4-amd64 #1
Debian 4.19.28-2+mendix1
[777184.936625] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
disables this message.
[777184.936762] kworker/1:23    D    0 17371      2 0x80000000
[777184.936775] Workqueue: dm-integrity-metadata integrity_bio_wait
[dm_integrity]
[777184.936786] Call Trace:
[777184.936791]  ? __schedule+0x2a2/0x870
[777184.936798]  schedule+0x28/0x80
[777184.936805]  io_schedule+0x12/0x40
[777184.936812]  wait_and_add_new_range+0x89/0xa0 [dm_integrity]
[777184.936819]  dm_integrity_map_continue+0xe71/0x1020 [dm_integrity]
[777184.936828]  ? xen_load_sp0+0x77/0x170
[777184.936834]  ? __switch_to+0x1e1/0x440
[777184.936842]  ? finish_task_switch+0x78/0x260
[777184.936849]  process_one_work+0x1a7/0x3a0
[777184.936856]  worker_thread+0x30/0x390
[777184.936862]  ? create_worker+0x1a0/0x1a0
[777184.936868]  kthread+0x112/0x130
[777184.936875]  ? kthread_bind+0x30/0x30
[777184.936880]  ret_from_fork+0x35/0x40

Hans

On 3/7/19 5:37 PM, Milan Broz wrote:
> Hi,
> 
> from the full log it seems that it is stuck in tgtd (iSCSI).
> 
> Anyway, this is device-mapper, dm-devel is better list here.
> (added to cc + Mikulas)
> 
> m.
> 
> On 3/7/19 4:26 PM, Victor Helmholtz wrote:
>> Hi
>>
>> I have recently had a problem with my server: all writes to RAID drives were frozen and I had to force-reboot it. I looked at kernel logs and I think dm-integrity might be source of the problem. Could anyone help me understand meaning of following messages (this message was repeated 8 times, full kernel log is here: https://pastebin.com/DMgMtNJq ):
>>
>> Mar  1 23:48:21 unassigned-hostname kernel: [369732.498048] INFO: task kworker/0:112:26760 blocked for more than 120 seconds.
>> Mar  1 23:48:21 unassigned-hostname kernel: [369732.499784]       Not tainted 4.19.0-2-amd64 #1 Debian 4.19.16-1
>> Mar  1 23:48:21 unassigned-hostname kernel: [369732.501222] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
>> Mar  1 23:48:21 unassigned-hostname kernel: [369732.503095] kworker/0:112   D    0 26760      2 0x80000080
>> Mar  1 23:48:21 unassigned-hostname kernel: [369732.504424] Workqueue: dm-integrity-metadata integrity_bio_wait [dm_integrity]
>> Mar  1 23:48:21 unassigned-hostname kernel: [369732.506149] Call Trace:
>> Mar  1 23:48:21 unassigned-hostname kernel: [369732.506782]  ? __schedule+0x2a2/0x870
>> Mar  1 23:48:21 unassigned-hostname kernel: [369732.507706]  schedule+0x28/0x80
>> Mar  1 23:48:21 unassigned-hostname kernel: [369732.508484]  io_schedule+0x12/0x40
>> Mar  1 23:48:21 unassigned-hostname kernel: [369732.509321]  wait_and_add_new_range+0x89/0xa0 [dm_integrity]
>> Mar  1 23:48:21 unassigned-hostname kernel: [369732.510709]  dm_integrity_map_continue+0xe71/0x1020 [dm_integrity]
>> Mar  1 23:48:21 unassigned-hostname kernel: [369732.512193]  ? __switch_to_asm+0x40/0x70
>> Mar  1 23:48:21 unassigned-hostname kernel: [369732.513149]  ? __switch_to_asm+0x34/0x70
>> Mar  1 23:48:21 unassigned-hostname kernel: [369732.514105]  ? __switch_to_asm+0x40/0x70
>> Mar  1 23:48:21 unassigned-hostname kernel: [369732.515088]  ? __switch_to_asm+0x40/0x70
>> Mar  1 23:48:21 unassigned-hostname kernel: [369732.516048]  ? __switch_to_asm+0x34/0x70
>> Mar  1 23:48:21 unassigned-hostname kernel: [369732.517000]  ? __switch_to_asm+0x40/0x70
>> Mar  1 23:48:21 unassigned-hostname kernel: [369732.517978]  ? __switch_to_asm+0x34/0x70
>> Mar  1 23:48:21 unassigned-hostname kernel: [369732.518968]  ? __switch_to_asm+0x34/0x70
>> Mar  1 23:48:21 unassigned-hostname kernel: [369732.519926]  ? __switch_to+0x8c/0x440
>> Mar  1 23:48:21 unassigned-hostname kernel: [369732.520822]  process_one_work+0x1a7/0x3a0
>> Mar  1 23:48:21 unassigned-hostname kernel: [369732.521798]  worker_thread+0x30/0x390
>> Mar  1 23:48:21 unassigned-hostname kernel: [369732.522725]  ? pwq_unbound_release_workfn+0xd0/0xd0
>> Mar  1 23:48:21 unassigned-hostname kernel: [369732.523901]  kthread+0x112/0x130
>> Mar  1 23:48:21 unassigned-hostname kernel: [369732.524691]  ? kthread_bind+0x30/0x30
>> Mar  1 23:48:21 unassigned-hostname kernel: [369732.525584]  ret_from_fork+0x35/0x40
>>
>> Is this a bug in kernel or do I have some problems with hardware?

^ permalink raw reply	[flat|nested] 10+ messages in thread

* Re: [dm-crypt] [dm-devel]  Dm-integrity freeze
  2019-05-21  8:43     ` [dm-crypt] " Hans van Kranenburg
@ 2019-05-21 20:33       ` Hans van Kranenburg
  -1 siblings, 0 replies; 10+ messages in thread
From: Hans van Kranenburg @ 2019-05-21 20:33 UTC (permalink / raw)
  To: Milan Broz, Victor Helmholtz, dm-crypt
  Cc: device-mapper development, Mikulas Patocka

Hi,

On 5/21/19 10:43 AM, Hans van Kranenburg wrote:
> Hi,
> 
> I'm seeing the same lockup, also 4.19. This is mdadm RAID10 on top of 4x
> a partition with only dm-integrity.
> 
> It just happened out of the blue, no heavy load or anything. All IO to
> it is frozen now.
> 
> [...]

There it is again... dmesg dump below. All cpus on 100% iowait.

It's triggered after a few minutes by running some Windows 2019 server
install (ugh, don't ask) in a Xen HVM domU, which writes into a raw
sparse file on a btrfs filesystem on LVM on mdadm RAID10 on 4x
dm-integrity (wheeee!!)...

This morning it was triggered a few minutes after starting an old
windows 2008 server image that I copied to this machine.

When running only other Linux vms, and when copying data onto
filesystems that live in LVM logical volumes I haven't seen this problem
yet, at all, in the last few weeks that this machine is running.

I noticed there's a "dm integrity: fix deadlock with overlapping I/O"
fix in a later 4.19. Is there any chance this is related? I have no
idea, but any hints or suggestions about what to try would be appreciated.

Right now, I still have the opportunity to stop the migration and keep
using the previous hardware for some days, having the possibility to
reproduce this a few times and try things, if you want.

If not, I likely have to go try remove the dm-integrity layer, but I'd
rather help fixing this issue.

Thanks,

[37218.238621] INFO: task md127_raid10:485 blocked for more than 120
seconds.
[37218.238713]       Not tainted 4.19.0-mx4-amd64 #1 Debian
4.19.28-2+mendix1
[37218.238782] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
disables this message.
[37218.238856] md127_raid10    D    0   485      2 0x80000000
[37218.238859] Call Trace:
[37218.238868]  ? __schedule+0x2a2/0x870
[37218.238871]  ? _raw_spin_unlock_irqrestore+0x14/0x20
[37218.238872]  schedule+0x28/0x80
[37218.238882]  md_super_wait+0x6e/0xa0 [md_mod]
[37218.238887]  ? finish_wait+0x80/0x80
[37218.238892]  write_page+0x172/0x320 [md_mod]
[37218.238894]  ? __schedule+0x2aa/0x870
[37218.238895]  ? _raw_spin_unlock_irqrestore+0x14/0x20
[37218.238900]  md_update_sb.part.64+0x382/0x8f0 [md_mod]
[37218.238904]  md_check_recovery+0x148/0x540 [md_mod]
[37218.238907]  raid10d+0x62/0x1440 [raid10]
[37218.238910]  ? __switch_to_asm+0x34/0x70
[37218.238912]  ? finish_task_switch+0x78/0x260
[37218.238914]  ? _raw_spin_lock_irqsave+0x15/0x40
[37218.238917]  ? lock_timer_base+0x67/0x80
[37218.238918]  ? _raw_spin_unlock_irqrestore+0x14/0x20
[37218.238920]  ? try_to_del_timer_sync+0x4d/0x80
[37218.238922]  ? del_timer_sync+0x35/0x40
[37218.238923]  ? schedule_timeout+0x173/0x390
[37218.238927]  ? md_rdev_init+0xb0/0xb0 [md_mod]
[37218.238930]  ? md_thread+0x94/0x150 [md_mod]
[37218.238932]  ? raid10_end_write_request+0x290/0x290 [raid10]
[37218.238936]  md_thread+0x94/0x150 [md_mod]
[37218.238938]  ? finish_wait+0x80/0x80
[37218.238940]  kthread+0x112/0x130
[37218.238941]  ? kthread_bind+0x30/0x30
[37218.238943]  ret_from_fork+0x35/0x40
[37218.238951] INFO: task btrfs-transacti:575 blocked for more than 120
seconds.
[37218.239023]       Not tainted 4.19.0-mx4-amd64 #1 Debian
4.19.28-2+mendix1
[37218.239092] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
disables this message.
[37218.239164] btrfs-transacti D    0   575      2 0x80000000
[37218.239166] Call Trace:
[37218.239168]  ? __schedule+0x2a2/0x870
[37218.239169]  ? _raw_spin_unlock_irqrestore+0x14/0x20
[37218.239171]  schedule+0x28/0x80
[37218.239175]  md_write_start+0x14b/0x220 [md_mod]
[37218.239177]  ? finish_wait+0x80/0x80
[37218.239179]  raid10_make_request+0x3d/0x130 [raid10]
[37218.239181]  ? finish_wait+0x80/0x80
[37218.239185]  md_handle_request+0x119/0x190 [md_mod]
[37218.239189]  md_make_request+0x78/0x160 [md_mod]
[37218.239192]  generic_make_request+0x1a4/0x410
[37218.239195]  submit_bio+0x45/0x140
[37218.239196]  ? bio_clone_fast+0x2c/0x60
[37218.239228]  btrfs_map_bio+0x20a/0x350 [btrfs]
[37218.239246]  btree_submit_bio_hook+0x8d/0xc0 [btrfs]
[37218.239264]  submit_one_bio+0x5f/0x80 [btrfs]
[37218.239283]  flush_write_bio.isra.40+0x1d/0x30 [btrfs]
[37218.239301]  btree_write_cache_pages+0x243/0x360 [btrfs]
[37218.239318]  ? btrfs_buffer_uptodate+0x33/0x50 [btrfs]
[37218.239331]  ? read_block_for_search.isra.36+0xf2/0x340 [btrfs]
[37218.239333]  do_writepages+0x41/0xd0
[37218.239351]  ? merge_state.part.45+0x3f/0x130 [btrfs]
[37218.239353]  ? kmem_cache_alloc+0x15c/0x1c0
[37218.239370]  ? clear_state_bit+0xdb/0x180 [btrfs]
[37218.239372]  __filemap_fdatawrite_range+0xbe/0xf0
[37218.239388]  btrfs_write_marked_extents+0x68/0x150 [btrfs]
[37218.239405]  btrfs_write_and_wait_transaction.isra.21+0x4d/0xa0 [btrfs]
[37218.239421]  btrfs_commit_transaction+0x56c/0x870 [btrfs]
[37218.239437]  ? start_transaction+0x9d/0x3f0 [btrfs]
[37218.239453]  transaction_kthread+0x147/0x180 [btrfs]
[37218.239468]  ? btrfs_cleanup_transaction+0x530/0x530 [btrfs]
[37218.239470]  kthread+0x112/0x130
[37218.239471]  ? kthread_bind+0x30/0x30
[37218.239473]  ret_from_fork+0x35/0x40
[37218.239481] INFO: task 1.xvda-1:1820 blocked for more than 120 seconds.
[37218.239551]       Not tainted 4.19.0-mx4-amd64 #1 Debian
4.19.28-2+mendix1
[37218.239620] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
disables this message.
[37218.239692] 1.xvda-1        D    0  1820      2 0x80000000
[37218.239694] Call Trace:
[37218.239696]  ? __schedule+0x2a2/0x870
[37218.239698]  ? _raw_spin_unlock_irqrestore+0x14/0x20
[37218.239699]  schedule+0x28/0x80
[37218.239703]  md_write_start+0x14b/0x220 [md_mod]
[37218.239706]  ? finish_wait+0x80/0x80
[37218.239708]  raid10_make_request+0x3d/0x130 [raid10]
[37218.239710]  ? finish_wait+0x80/0x80
[37218.239713]  md_handle_request+0x119/0x190 [md_mod]
[37218.239718]  md_make_request+0x78/0x160 [md_mod]
[37218.239720]  generic_make_request+0x1a4/0x410
[37218.239723]  ? bvec_alloc+0x51/0xe0
[37218.239725]  submit_bio+0x45/0x140
[37218.239727]  ? bio_add_page+0x48/0x60
[37218.239731]  dispatch_rw_block_io+0x68f/0x990 [xen_blkback]
[37218.239734]  ? branch_type+0x2e4/0x350
[37218.239737]  __do_block_io_op+0x2fc/0x5f0 [xen_blkback]
[37218.239739]  ? _raw_spin_unlock_irqrestore+0x14/0x20
[37218.239740]  ? try_to_del_timer_sync+0x4d/0x80
[37218.239743]  xen_blkif_schedule+0xce/0x620 [xen_blkback]
[37218.239745]  ? finish_wait+0x80/0x80
[37218.239748]  ? xen_blkif_be_int+0x30/0x30 [xen_blkback]
[37218.239749]  kthread+0x112/0x130
[37218.239750]  ? kthread_bind+0x30/0x30
[37218.239751]  ret_from_fork+0x35/0x40
[37218.239757] INFO: task 2.xvda-0:2125 blocked for more than 120 seconds.
[37218.239825]       Not tainted 4.19.0-mx4-amd64 #1 Debian
4.19.28-2+mendix1
[37218.239894] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
disables this message.
[37218.239967] 2.xvda-0        D    0  2125      2 0x80000000
[37218.239968] Call Trace:
[37218.239970]  ? __schedule+0x2a2/0x870
[37218.239971]  ? _raw_spin_unlock_irqrestore+0x14/0x20
[37218.239972]  schedule+0x28/0x80
[37218.239977]  md_write_start+0x14b/0x220 [md_mod]
[37218.239979]  ? finish_wait+0x80/0x80
[37218.239981]  raid10_make_request+0x3d/0x130 [raid10]
[37218.239983]  ? finish_wait+0x80/0x80
[37218.239986]  md_handle_request+0x119/0x190 [md_mod]
[37218.239990]  md_make_request+0x78/0x160 [md_mod]
[37218.239992]  generic_make_request+0x1a4/0x410
[37218.239994]  submit_bio+0x45/0x140
[37218.239996]  ? bio_add_page+0x48/0x60
[37218.239998]  dispatch_rw_block_io+0x68f/0x990 [xen_blkback]
[37218.240001]  ? branch_type+0x2e7/0x350
[37218.240003]  __do_block_io_op+0x2fc/0x5f0 [xen_blkback]
[37218.240005]  ? _raw_spin_unlock_irqrestore+0x14/0x20
[37218.240006]  ? try_to_del_timer_sync+0x4d/0x80
[37218.240009]  xen_blkif_schedule+0xce/0x620 [xen_blkback]
[37218.240010]  ? __schedule+0x2aa/0x870
[37218.240012]  ? finish_wait+0x80/0x80
[37218.240014]  ? xen_blkif_be_int+0x30/0x30 [xen_blkback]
[37218.240015]  kthread+0x112/0x130
[37218.240016]  ? kthread_bind+0x30/0x30
[37218.240018]  ret_from_fork+0x35/0x40
[37218.240021] INFO: task qemu-system-i38:6225 blocked for more than 120
seconds.
[37218.240092]       Not tainted 4.19.0-mx4-amd64 #1 Debian
4.19.28-2+mendix1
[37218.240161] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
disables this message.
[37218.240258] qemu-system-i38 D    0  6225      1 0x00000000
[37218.240260] Call Trace:
[37218.240262]  ? __schedule+0x2a2/0x870
[37218.240263]  schedule+0x28/0x80
[37218.240266]  io_schedule+0x12/0x40
[37218.240267]  wait_on_page_bit+0x114/0x150
[37218.240269]  ? page_cache_tree_insert+0xe0/0xe0
[37218.240288]  extent_write_cache_pages+0x33e/0x3e0 [btrfs]
[37218.240292]  ? timerqueue_del+0x1e/0x40
[37218.240293]  ? __remove_hrtimer+0x35/0x70
[37218.240311]  extent_writepages+0x50/0x80 [btrfs]
[37218.240314]  do_writepages+0x41/0xd0
[37218.240315]  ? __hrtimer_init+0xb0/0xb0
[37218.240317]  __filemap_fdatawrite_range+0xbe/0xf0
[37218.240334]  start_ordered_ops+0x4c/0x90 [btrfs]
[37218.240352]  btrfs_sync_file+0x9a/0x3d0 [btrfs]
[37218.240357]  do_fsync+0x38/0x70
[37218.240359]  __x64_sys_fdatasync+0x13/0x20
[37218.240361]  do_syscall_64+0x53/0x100
[37218.240364]  entry_SYSCALL_64_after_hwframe+0x44/0xa9
[37218.240366] RIP: 0033:0x7f2c4484f2e7
[37218.240373] Code: Bad RIP value.
[37218.240374] RSP: 002b:00007f2c40c24560 EFLAGS: 00000293 ORIG_RAX:
000000000000004b
[37218.240376] RAX: ffffffffffffffda RBX: 000000000000001c RCX:
00007f2c4484f2e7
[37218.240377] RDX: 0000000000000000 RSI: 0000000000000008 RDI:
000000000000001c
[37218.240378] RBP: 000055f297129ea0 R08: 0000000000000000 R09:
00000000ffffffff
[37218.240379] R10: 00007f2c40c24660 R11: 0000000000000293 R12:
000055f2964b06e0
[37218.240380] R13: 000055f2970ea078 R14: 000055f2974aa920 R15:
000055f297d92ab0
[37218.240383] INFO: task kworker/1:2:4661 blocked for more than 120
seconds.
[37218.240478]       Not tainted 4.19.0-mx4-amd64 #1 Debian
4.19.28-2+mendix1
[37218.240580] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
disables this message.
[37218.240701] kworker/1:2     D    0  4661      2 0x80000000
[37218.240708] Workqueue: dm-integrity-metadata integrity_bio_wait
[dm_integrity]
[37218.240709] Call Trace:
[37218.240711]  ? __schedule+0x2a2/0x870
[37218.240713]  schedule+0x28/0x80
[37218.240715]  io_schedule+0x12/0x40
[37218.240717]  wait_and_add_new_range+0x89/0xa0 [dm_integrity]
[37218.240719]  dm_integrity_map_continue+0xe71/0x1020 [dm_integrity]
[37218.240723]  ? xen_load_sp0+0x77/0x170
[37218.240725]  ? __switch_to+0x1e1/0x440
[37218.240727]  ? finish_task_switch+0x78/0x260
[37218.240730]  process_one_work+0x1a7/0x3a0
[37218.240732]  worker_thread+0x30/0x390
[37218.240734]  ? create_worker+0x1a0/0x1a0
[37218.240735]  kthread+0x112/0x130
[37218.240736]  ? kthread_bind+0x30/0x30
[37218.240738]  ret_from_fork+0x35/0x40
[37218.240741] INFO: task kworker/1:11:4887 blocked for more than 120
seconds.
[37218.240836]       Not tainted 4.19.0-mx4-amd64 #1 Debian
4.19.28-2+mendix1
[37218.240929] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
disables this message.
[37218.241050] kworker/1:11    D    0  4887      2 0x80000000
[37218.241053] Workqueue: dm-integrity-metadata integrity_bio_wait
[dm_integrity]
[37218.241054] Call Trace:
[37218.241056]  ? __schedule+0x2a2/0x870
[37218.241058]  schedule+0x28/0x80
[37218.241059]  io_schedule+0x12/0x40
[37218.241061]  wait_and_add_new_range+0x89/0xa0 [dm_integrity]
[37218.241063]  dm_integrity_map_continue+0xe71/0x1020 [dm_integrity]
[37218.241065]  ? xen_load_sp0+0x77/0x170
[37218.241067]  ? __switch_to+0x1e1/0x440
[37218.241068]  ? finish_task_switch+0x78/0x260
[37218.241070]  process_one_work+0x1a7/0x3a0
[37218.241072]  worker_thread+0x30/0x390
[37218.241073]  ? create_worker+0x1a0/0x1a0
[37218.241074]  kthread+0x112/0x130
[37218.241075]  ? kthread_bind+0x30/0x30
[37218.241077]  ret_from_fork+0x35/0x40
[37218.241082] INFO: task kworker/0:19:5061 blocked for more than 120
seconds.
[37218.241176]       Not tainted 4.19.0-mx4-amd64 #1 Debian
4.19.28-2+mendix1
[37218.241270] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
disables this message.
[37218.241391] kworker/0:19    D    0  5061      2 0x80000000
[37218.241394] Workqueue: dm-integrity-metadata integrity_bio_wait
[dm_integrity]
[37218.241395] Call Trace:
[37218.241397]  ? __schedule+0x2a2/0x870
[37218.241398]  schedule+0x28/0x80
[37218.241400]  io_schedule+0x12/0x40
[37218.241402]  wait_and_add_new_range+0x89/0xa0 [dm_integrity]
[37218.241404]  dm_integrity_map_continue+0xe71/0x1020 [dm_integrity]
[37218.241406]  ? xen_load_sp0+0x77/0x170
[37218.241407]  ? __switch_to+0x1e1/0x440
[37218.241408]  ? finish_task_switch+0x78/0x260
[37218.241410]  process_one_work+0x1a7/0x3a0
[37218.241412]  worker_thread+0x30/0x390
[37218.241414]  ? create_worker+0x1a0/0x1a0
[37218.241415]  kthread+0x112/0x130
[37218.241416]  ? kthread_bind+0x30/0x30
[37218.241417]  ret_from_fork+0x35/0x40
[37218.241420] INFO: task kworker/1:3:5090 blocked for more than 120
seconds.
[37218.241514]       Not tainted 4.19.0-mx4-amd64 #1 Debian
4.19.28-2+mendix1
[37218.241616] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
disables this message.
[37218.241737] kworker/1:3     D    0  5090      2 0x80000000
[37218.241740] Workqueue: dm-integrity-metadata integrity_bio_wait
[dm_integrity]
[37218.241740] Call Trace:
[37218.241742]  ? __schedule+0x2a2/0x870
[37218.241744]  schedule+0x28/0x80
[37218.241745]  io_schedule+0x12/0x40
[37218.241747]  wait_and_add_new_range+0x89/0xa0 [dm_integrity]
[37218.241749]  dm_integrity_map_continue+0xe71/0x1020 [dm_integrity]
[37218.241751]  ? xen_load_sp0+0x77/0x170
[37218.241752]  ? __switch_to+0x1e1/0x440
[37218.241753]  ? finish_task_switch+0x78/0x260
[37218.241755]  process_one_work+0x1a7/0x3a0
[37218.241757]  worker_thread+0x30/0x390
[37218.241758]  ? create_worker+0x1a0/0x1a0
[37218.241759]  kthread+0x112/0x130
[37218.241760]  ? kthread_bind+0x30/0x30
[37218.241762]  ret_from_fork+0x35/0x40
[37218.241764] INFO: task kworker/1:10:5264 blocked for more than 120
seconds.
[37218.241859]       Not tainted 4.19.0-mx4-amd64 #1 Debian
4.19.28-2+mendix1
[37218.241952] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
disables this message.
[37218.242073] kworker/1:10    D    0  5264      2 0x80000000
[37218.242076] Workqueue: dm-integrity-metadata integrity_bio_wait
[dm_integrity]
[37218.242077] Call Trace:
[37218.242079]  ? __schedule+0x2a2/0x870
[37218.242080]  schedule+0x28/0x80
[37218.242082]  io_schedule+0x12/0x40
[37218.242083]  wait_and_add_new_range+0x89/0xa0 [dm_integrity]
[37218.242085]  dm_integrity_map_continue+0xe71/0x1020 [dm_integrity]
[37218.242088]  ? xen_load_sp0+0x77/0x170
[37218.242089]  ? __switch_to+0x1e1/0x440
[37218.242090]  ? finish_task_switch+0x78/0x260
[37218.242092]  process_one_work+0x1a7/0x3a0
[37218.242094]  worker_thread+0x30/0x390
[37218.242095]  ? create_worker+0x1a0/0x1a0
[37218.242096]  kthread+0x112/0x130
[37218.242097]  ? kthread_bind+0x30/0x30
[37218.242099]  ret_from_fork+0x35/0x40

^ permalink raw reply	[flat|nested] 10+ messages in thread

* Re: [dm-crypt] Dm-integrity freeze
@ 2019-05-21 20:33       ` Hans van Kranenburg
  0 siblings, 0 replies; 10+ messages in thread
From: Hans van Kranenburg @ 2019-05-21 20:33 UTC (permalink / raw)
  To: Milan Broz, Victor Helmholtz, dm-crypt
  Cc: device-mapper development, Mikulas Patocka

Hi,

On 5/21/19 10:43 AM, Hans van Kranenburg wrote:
> Hi,
> 
> I'm seeing the same lockup, also 4.19. This is mdadm RAID10 on top of 4x
> a partition with only dm-integrity.
> 
> It just happened out of the blue, no heavy load or anything. All IO to
> it is frozen now.
> 
> [...]

There it is again... dmesg dump below. All cpus on 100% iowait.

It's triggered after a few minutes by running some Windows 2019 server
install (ugh, don't ask) in a Xen HVM domU, which writes into a raw
sparse file on a btrfs filesystem on LVM on mdadm RAID10 on 4x
dm-integrity (wheeee!!)...

This morning it was triggered a few minutes after starting an old
windows 2008 server image that I copied to this machine.

When running only other Linux vms, and when copying data onto
filesystems that live in LVM logical volumes I haven't seen this problem
yet, at all, in the last few weeks that this machine is running.

I noticed there's a "dm integrity: fix deadlock with overlapping I/O"
fix in a later 4.19. Is there any chance this is related? I have no
idea, but any hints or suggestions about what to try would be appreciated.

Right now, I still have the opportunity to stop the migration and keep
using the previous hardware for some days, having the possibility to
reproduce this a few times and try things, if you want.

If not, I likely have to go try remove the dm-integrity layer, but I'd
rather help fixing this issue.

Thanks,

[37218.238621] INFO: task md127_raid10:485 blocked for more than 120
seconds.
[37218.238713]       Not tainted 4.19.0-mx4-amd64 #1 Debian
4.19.28-2+mendix1
[37218.238782] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
disables this message.
[37218.238856] md127_raid10    D    0   485      2 0x80000000
[37218.238859] Call Trace:
[37218.238868]  ? __schedule+0x2a2/0x870
[37218.238871]  ? _raw_spin_unlock_irqrestore+0x14/0x20
[37218.238872]  schedule+0x28/0x80
[37218.238882]  md_super_wait+0x6e/0xa0 [md_mod]
[37218.238887]  ? finish_wait+0x80/0x80
[37218.238892]  write_page+0x172/0x320 [md_mod]
[37218.238894]  ? __schedule+0x2aa/0x870
[37218.238895]  ? _raw_spin_unlock_irqrestore+0x14/0x20
[37218.238900]  md_update_sb.part.64+0x382/0x8f0 [md_mod]
[37218.238904]  md_check_recovery+0x148/0x540 [md_mod]
[37218.238907]  raid10d+0x62/0x1440 [raid10]
[37218.238910]  ? __switch_to_asm+0x34/0x70
[37218.238912]  ? finish_task_switch+0x78/0x260
[37218.238914]  ? _raw_spin_lock_irqsave+0x15/0x40
[37218.238917]  ? lock_timer_base+0x67/0x80
[37218.238918]  ? _raw_spin_unlock_irqrestore+0x14/0x20
[37218.238920]  ? try_to_del_timer_sync+0x4d/0x80
[37218.238922]  ? del_timer_sync+0x35/0x40
[37218.238923]  ? schedule_timeout+0x173/0x390
[37218.238927]  ? md_rdev_init+0xb0/0xb0 [md_mod]
[37218.238930]  ? md_thread+0x94/0x150 [md_mod]
[37218.238932]  ? raid10_end_write_request+0x290/0x290 [raid10]
[37218.238936]  md_thread+0x94/0x150 [md_mod]
[37218.238938]  ? finish_wait+0x80/0x80
[37218.238940]  kthread+0x112/0x130
[37218.238941]  ? kthread_bind+0x30/0x30
[37218.238943]  ret_from_fork+0x35/0x40
[37218.238951] INFO: task btrfs-transacti:575 blocked for more than 120
seconds.
[37218.239023]       Not tainted 4.19.0-mx4-amd64 #1 Debian
4.19.28-2+mendix1
[37218.239092] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
disables this message.
[37218.239164] btrfs-transacti D    0   575      2 0x80000000
[37218.239166] Call Trace:
[37218.239168]  ? __schedule+0x2a2/0x870
[37218.239169]  ? _raw_spin_unlock_irqrestore+0x14/0x20
[37218.239171]  schedule+0x28/0x80
[37218.239175]  md_write_start+0x14b/0x220 [md_mod]
[37218.239177]  ? finish_wait+0x80/0x80
[37218.239179]  raid10_make_request+0x3d/0x130 [raid10]
[37218.239181]  ? finish_wait+0x80/0x80
[37218.239185]  md_handle_request+0x119/0x190 [md_mod]
[37218.239189]  md_make_request+0x78/0x160 [md_mod]
[37218.239192]  generic_make_request+0x1a4/0x410
[37218.239195]  submit_bio+0x45/0x140
[37218.239196]  ? bio_clone_fast+0x2c/0x60
[37218.239228]  btrfs_map_bio+0x20a/0x350 [btrfs]
[37218.239246]  btree_submit_bio_hook+0x8d/0xc0 [btrfs]
[37218.239264]  submit_one_bio+0x5f/0x80 [btrfs]
[37218.239283]  flush_write_bio.isra.40+0x1d/0x30 [btrfs]
[37218.239301]  btree_write_cache_pages+0x243/0x360 [btrfs]
[37218.239318]  ? btrfs_buffer_uptodate+0x33/0x50 [btrfs]
[37218.239331]  ? read_block_for_search.isra.36+0xf2/0x340 [btrfs]
[37218.239333]  do_writepages+0x41/0xd0
[37218.239351]  ? merge_state.part.45+0x3f/0x130 [btrfs]
[37218.239353]  ? kmem_cache_alloc+0x15c/0x1c0
[37218.239370]  ? clear_state_bit+0xdb/0x180 [btrfs]
[37218.239372]  __filemap_fdatawrite_range+0xbe/0xf0
[37218.239388]  btrfs_write_marked_extents+0x68/0x150 [btrfs]
[37218.239405]  btrfs_write_and_wait_transaction.isra.21+0x4d/0xa0 [btrfs]
[37218.239421]  btrfs_commit_transaction+0x56c/0x870 [btrfs]
[37218.239437]  ? start_transaction+0x9d/0x3f0 [btrfs]
[37218.239453]  transaction_kthread+0x147/0x180 [btrfs]
[37218.239468]  ? btrfs_cleanup_transaction+0x530/0x530 [btrfs]
[37218.239470]  kthread+0x112/0x130
[37218.239471]  ? kthread_bind+0x30/0x30
[37218.239473]  ret_from_fork+0x35/0x40
[37218.239481] INFO: task 1.xvda-1:1820 blocked for more than 120 seconds.
[37218.239551]       Not tainted 4.19.0-mx4-amd64 #1 Debian
4.19.28-2+mendix1
[37218.239620] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
disables this message.
[37218.239692] 1.xvda-1        D    0  1820      2 0x80000000
[37218.239694] Call Trace:
[37218.239696]  ? __schedule+0x2a2/0x870
[37218.239698]  ? _raw_spin_unlock_irqrestore+0x14/0x20
[37218.239699]  schedule+0x28/0x80
[37218.239703]  md_write_start+0x14b/0x220 [md_mod]
[37218.239706]  ? finish_wait+0x80/0x80
[37218.239708]  raid10_make_request+0x3d/0x130 [raid10]
[37218.239710]  ? finish_wait+0x80/0x80
[37218.239713]  md_handle_request+0x119/0x190 [md_mod]
[37218.239718]  md_make_request+0x78/0x160 [md_mod]
[37218.239720]  generic_make_request+0x1a4/0x410
[37218.239723]  ? bvec_alloc+0x51/0xe0
[37218.239725]  submit_bio+0x45/0x140
[37218.239727]  ? bio_add_page+0x48/0x60
[37218.239731]  dispatch_rw_block_io+0x68f/0x990 [xen_blkback]
[37218.239734]  ? branch_type+0x2e4/0x350
[37218.239737]  __do_block_io_op+0x2fc/0x5f0 [xen_blkback]
[37218.239739]  ? _raw_spin_unlock_irqrestore+0x14/0x20
[37218.239740]  ? try_to_del_timer_sync+0x4d/0x80
[37218.239743]  xen_blkif_schedule+0xce/0x620 [xen_blkback]
[37218.239745]  ? finish_wait+0x80/0x80
[37218.239748]  ? xen_blkif_be_int+0x30/0x30 [xen_blkback]
[37218.239749]  kthread+0x112/0x130
[37218.239750]  ? kthread_bind+0x30/0x30
[37218.239751]  ret_from_fork+0x35/0x40
[37218.239757] INFO: task 2.xvda-0:2125 blocked for more than 120 seconds.
[37218.239825]       Not tainted 4.19.0-mx4-amd64 #1 Debian
4.19.28-2+mendix1
[37218.239894] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
disables this message.
[37218.239967] 2.xvda-0        D    0  2125      2 0x80000000
[37218.239968] Call Trace:
[37218.239970]  ? __schedule+0x2a2/0x870
[37218.239971]  ? _raw_spin_unlock_irqrestore+0x14/0x20
[37218.239972]  schedule+0x28/0x80
[37218.239977]  md_write_start+0x14b/0x220 [md_mod]
[37218.239979]  ? finish_wait+0x80/0x80
[37218.239981]  raid10_make_request+0x3d/0x130 [raid10]
[37218.239983]  ? finish_wait+0x80/0x80
[37218.239986]  md_handle_request+0x119/0x190 [md_mod]
[37218.239990]  md_make_request+0x78/0x160 [md_mod]
[37218.239992]  generic_make_request+0x1a4/0x410
[37218.239994]  submit_bio+0x45/0x140
[37218.239996]  ? bio_add_page+0x48/0x60
[37218.239998]  dispatch_rw_block_io+0x68f/0x990 [xen_blkback]
[37218.240001]  ? branch_type+0x2e7/0x350
[37218.240003]  __do_block_io_op+0x2fc/0x5f0 [xen_blkback]
[37218.240005]  ? _raw_spin_unlock_irqrestore+0x14/0x20
[37218.240006]  ? try_to_del_timer_sync+0x4d/0x80
[37218.240009]  xen_blkif_schedule+0xce/0x620 [xen_blkback]
[37218.240010]  ? __schedule+0x2aa/0x870
[37218.240012]  ? finish_wait+0x80/0x80
[37218.240014]  ? xen_blkif_be_int+0x30/0x30 [xen_blkback]
[37218.240015]  kthread+0x112/0x130
[37218.240016]  ? kthread_bind+0x30/0x30
[37218.240018]  ret_from_fork+0x35/0x40
[37218.240021] INFO: task qemu-system-i38:6225 blocked for more than 120
seconds.
[37218.240092]       Not tainted 4.19.0-mx4-amd64 #1 Debian
4.19.28-2+mendix1
[37218.240161] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
disables this message.
[37218.240258] qemu-system-i38 D    0  6225      1 0x00000000
[37218.240260] Call Trace:
[37218.240262]  ? __schedule+0x2a2/0x870
[37218.240263]  schedule+0x28/0x80
[37218.240266]  io_schedule+0x12/0x40
[37218.240267]  wait_on_page_bit+0x114/0x150
[37218.240269]  ? page_cache_tree_insert+0xe0/0xe0
[37218.240288]  extent_write_cache_pages+0x33e/0x3e0 [btrfs]
[37218.240292]  ? timerqueue_del+0x1e/0x40
[37218.240293]  ? __remove_hrtimer+0x35/0x70
[37218.240311]  extent_writepages+0x50/0x80 [btrfs]
[37218.240314]  do_writepages+0x41/0xd0
[37218.240315]  ? __hrtimer_init+0xb0/0xb0
[37218.240317]  __filemap_fdatawrite_range+0xbe/0xf0
[37218.240334]  start_ordered_ops+0x4c/0x90 [btrfs]
[37218.240352]  btrfs_sync_file+0x9a/0x3d0 [btrfs]
[37218.240357]  do_fsync+0x38/0x70
[37218.240359]  __x64_sys_fdatasync+0x13/0x20
[37218.240361]  do_syscall_64+0x53/0x100
[37218.240364]  entry_SYSCALL_64_after_hwframe+0x44/0xa9
[37218.240366] RIP: 0033:0x7f2c4484f2e7
[37218.240373] Code: Bad RIP value.
[37218.240374] RSP: 002b:00007f2c40c24560 EFLAGS: 00000293 ORIG_RAX:
000000000000004b
[37218.240376] RAX: ffffffffffffffda RBX: 000000000000001c RCX:
00007f2c4484f2e7
[37218.240377] RDX: 0000000000000000 RSI: 0000000000000008 RDI:
000000000000001c
[37218.240378] RBP: 000055f297129ea0 R08: 0000000000000000 R09:
00000000ffffffff
[37218.240379] R10: 00007f2c40c24660 R11: 0000000000000293 R12:
000055f2964b06e0
[37218.240380] R13: 000055f2970ea078 R14: 000055f2974aa920 R15:
000055f297d92ab0
[37218.240383] INFO: task kworker/1:2:4661 blocked for more than 120
seconds.
[37218.240478]       Not tainted 4.19.0-mx4-amd64 #1 Debian
4.19.28-2+mendix1
[37218.240580] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
disables this message.
[37218.240701] kworker/1:2     D    0  4661      2 0x80000000
[37218.240708] Workqueue: dm-integrity-metadata integrity_bio_wait
[dm_integrity]
[37218.240709] Call Trace:
[37218.240711]  ? __schedule+0x2a2/0x870
[37218.240713]  schedule+0x28/0x80
[37218.240715]  io_schedule+0x12/0x40
[37218.240717]  wait_and_add_new_range+0x89/0xa0 [dm_integrity]
[37218.240719]  dm_integrity_map_continue+0xe71/0x1020 [dm_integrity]
[37218.240723]  ? xen_load_sp0+0x77/0x170
[37218.240725]  ? __switch_to+0x1e1/0x440
[37218.240727]  ? finish_task_switch+0x78/0x260
[37218.240730]  process_one_work+0x1a7/0x3a0
[37218.240732]  worker_thread+0x30/0x390
[37218.240734]  ? create_worker+0x1a0/0x1a0
[37218.240735]  kthread+0x112/0x130
[37218.240736]  ? kthread_bind+0x30/0x30
[37218.240738]  ret_from_fork+0x35/0x40
[37218.240741] INFO: task kworker/1:11:4887 blocked for more than 120
seconds.
[37218.240836]       Not tainted 4.19.0-mx4-amd64 #1 Debian
4.19.28-2+mendix1
[37218.240929] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
disables this message.
[37218.241050] kworker/1:11    D    0  4887      2 0x80000000
[37218.241053] Workqueue: dm-integrity-metadata integrity_bio_wait
[dm_integrity]
[37218.241054] Call Trace:
[37218.241056]  ? __schedule+0x2a2/0x870
[37218.241058]  schedule+0x28/0x80
[37218.241059]  io_schedule+0x12/0x40
[37218.241061]  wait_and_add_new_range+0x89/0xa0 [dm_integrity]
[37218.241063]  dm_integrity_map_continue+0xe71/0x1020 [dm_integrity]
[37218.241065]  ? xen_load_sp0+0x77/0x170
[37218.241067]  ? __switch_to+0x1e1/0x440
[37218.241068]  ? finish_task_switch+0x78/0x260
[37218.241070]  process_one_work+0x1a7/0x3a0
[37218.241072]  worker_thread+0x30/0x390
[37218.241073]  ? create_worker+0x1a0/0x1a0
[37218.241074]  kthread+0x112/0x130
[37218.241075]  ? kthread_bind+0x30/0x30
[37218.241077]  ret_from_fork+0x35/0x40
[37218.241082] INFO: task kworker/0:19:5061 blocked for more than 120
seconds.
[37218.241176]       Not tainted 4.19.0-mx4-amd64 #1 Debian
4.19.28-2+mendix1
[37218.241270] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
disables this message.
[37218.241391] kworker/0:19    D    0  5061      2 0x80000000
[37218.241394] Workqueue: dm-integrity-metadata integrity_bio_wait
[dm_integrity]
[37218.241395] Call Trace:
[37218.241397]  ? __schedule+0x2a2/0x870
[37218.241398]  schedule+0x28/0x80
[37218.241400]  io_schedule+0x12/0x40
[37218.241402]  wait_and_add_new_range+0x89/0xa0 [dm_integrity]
[37218.241404]  dm_integrity_map_continue+0xe71/0x1020 [dm_integrity]
[37218.241406]  ? xen_load_sp0+0x77/0x170
[37218.241407]  ? __switch_to+0x1e1/0x440
[37218.241408]  ? finish_task_switch+0x78/0x260
[37218.241410]  process_one_work+0x1a7/0x3a0
[37218.241412]  worker_thread+0x30/0x390
[37218.241414]  ? create_worker+0x1a0/0x1a0
[37218.241415]  kthread+0x112/0x130
[37218.241416]  ? kthread_bind+0x30/0x30
[37218.241417]  ret_from_fork+0x35/0x40
[37218.241420] INFO: task kworker/1:3:5090 blocked for more than 120
seconds.
[37218.241514]       Not tainted 4.19.0-mx4-amd64 #1 Debian
4.19.28-2+mendix1
[37218.241616] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
disables this message.
[37218.241737] kworker/1:3     D    0  5090      2 0x80000000
[37218.241740] Workqueue: dm-integrity-metadata integrity_bio_wait
[dm_integrity]
[37218.241740] Call Trace:
[37218.241742]  ? __schedule+0x2a2/0x870
[37218.241744]  schedule+0x28/0x80
[37218.241745]  io_schedule+0x12/0x40
[37218.241747]  wait_and_add_new_range+0x89/0xa0 [dm_integrity]
[37218.241749]  dm_integrity_map_continue+0xe71/0x1020 [dm_integrity]
[37218.241751]  ? xen_load_sp0+0x77/0x170
[37218.241752]  ? __switch_to+0x1e1/0x440
[37218.241753]  ? finish_task_switch+0x78/0x260
[37218.241755]  process_one_work+0x1a7/0x3a0
[37218.241757]  worker_thread+0x30/0x390
[37218.241758]  ? create_worker+0x1a0/0x1a0
[37218.241759]  kthread+0x112/0x130
[37218.241760]  ? kthread_bind+0x30/0x30
[37218.241762]  ret_from_fork+0x35/0x40
[37218.241764] INFO: task kworker/1:10:5264 blocked for more than 120
seconds.
[37218.241859]       Not tainted 4.19.0-mx4-amd64 #1 Debian
4.19.28-2+mendix1
[37218.241952] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
disables this message.
[37218.242073] kworker/1:10    D    0  5264      2 0x80000000
[37218.242076] Workqueue: dm-integrity-metadata integrity_bio_wait
[dm_integrity]
[37218.242077] Call Trace:
[37218.242079]  ? __schedule+0x2a2/0x870
[37218.242080]  schedule+0x28/0x80
[37218.242082]  io_schedule+0x12/0x40
[37218.242083]  wait_and_add_new_range+0x89/0xa0 [dm_integrity]
[37218.242085]  dm_integrity_map_continue+0xe71/0x1020 [dm_integrity]
[37218.242088]  ? xen_load_sp0+0x77/0x170
[37218.242089]  ? __switch_to+0x1e1/0x440
[37218.242090]  ? finish_task_switch+0x78/0x260
[37218.242092]  process_one_work+0x1a7/0x3a0
[37218.242094]  worker_thread+0x30/0x390
[37218.242095]  ? create_worker+0x1a0/0x1a0
[37218.242096]  kthread+0x112/0x130
[37218.242097]  ? kthread_bind+0x30/0x30
[37218.242099]  ret_from_fork+0x35/0x40

^ permalink raw reply	[flat|nested] 10+ messages in thread

* Re: [dm-crypt] Dm-integrity freeze
  2019-05-21 20:33       ` [dm-crypt] " Hans van Kranenburg
@ 2019-05-21 22:07         ` Mike Snitzer
  -1 siblings, 0 replies; 10+ messages in thread
From: Mike Snitzer @ 2019-05-21 22:07 UTC (permalink / raw)
  To: Hans van Kranenburg
  Cc: Milan Broz, Victor Helmholtz, dm-crypt,
	device-mapper development, Mikulas Patocka

On Tue, May 21 2019 at  4:33pm -0400,
Hans van Kranenburg <hans@knorrie.org> wrote:

> Hi,
> 
> On 5/21/19 10:43 AM, Hans van Kranenburg wrote:
> > Hi,
> > 
> > I'm seeing the same lockup, also 4.19. This is mdadm RAID10 on top of 4x
> > a partition with only dm-integrity.
> > 
> > It just happened out of the blue, no heavy load or anything. All IO to
> > it is frozen now.
> > 
> > [...]
> 
> There it is again... dmesg dump below. All cpus on 100% iowait.
> 
> It's triggered after a few minutes by running some Windows 2019 server
> install (ugh, don't ask) in a Xen HVM domU, which writes into a raw
> sparse file on a btrfs filesystem on LVM on mdadm RAID10 on 4x
> dm-integrity (wheeee!!)...
> 
> This morning it was triggered a few minutes after starting an old
> windows 2008 server image that I copied to this machine.
> 
> When running only other Linux vms, and when copying data onto
> filesystems that live in LVM logical volumes I haven't seen this problem
> yet, at all, in the last few weeks that this machine is running.
> 
> I noticed there's a "dm integrity: fix deadlock with overlapping I/O"
> fix in a later 4.19. Is there any chance this is related? I have no
> idea, but any hints or suggestions about what to try would be appreciated.

Yes, all your hung tasks are hung in wait_and_add_new_range().

Please use that later 4.19 or apply commit 4ed319c6ac08 ("dm integrity:
fix deadlock with overlapping I/O")

Mike

^ permalink raw reply	[flat|nested] 10+ messages in thread

* Re: Dm-integrity freeze
@ 2019-05-21 22:07         ` Mike Snitzer
  0 siblings, 0 replies; 10+ messages in thread
From: Mike Snitzer @ 2019-05-21 22:07 UTC (permalink / raw)
  To: Hans van Kranenburg
  Cc: dm-crypt, device-mapper development, Mikulas Patocka, Milan Broz,
	Victor Helmholtz

On Tue, May 21 2019 at  4:33pm -0400,
Hans van Kranenburg <hans@knorrie.org> wrote:

> Hi,
> 
> On 5/21/19 10:43 AM, Hans van Kranenburg wrote:
> > Hi,
> > 
> > I'm seeing the same lockup, also 4.19. This is mdadm RAID10 on top of 4x
> > a partition with only dm-integrity.
> > 
> > It just happened out of the blue, no heavy load or anything. All IO to
> > it is frozen now.
> > 
> > [...]
> 
> There it is again... dmesg dump below. All cpus on 100% iowait.
> 
> It's triggered after a few minutes by running some Windows 2019 server
> install (ugh, don't ask) in a Xen HVM domU, which writes into a raw
> sparse file on a btrfs filesystem on LVM on mdadm RAID10 on 4x
> dm-integrity (wheeee!!)...
> 
> This morning it was triggered a few minutes after starting an old
> windows 2008 server image that I copied to this machine.
> 
> When running only other Linux vms, and when copying data onto
> filesystems that live in LVM logical volumes I haven't seen this problem
> yet, at all, in the last few weeks that this machine is running.
> 
> I noticed there's a "dm integrity: fix deadlock with overlapping I/O"
> fix in a later 4.19. Is there any chance this is related? I have no
> idea, but any hints or suggestions about what to try would be appreciated.

Yes, all your hung tasks are hung in wait_and_add_new_range().

Please use that later 4.19 or apply commit 4ed319c6ac08 ("dm integrity:
fix deadlock with overlapping I/O")

Mike

^ permalink raw reply	[flat|nested] 10+ messages in thread

* Re: [dm-crypt] Dm-integrity freeze
  2019-05-21 22:07         ` Mike Snitzer
@ 2019-05-22  7:52           ` Hans van Kranenburg
  -1 siblings, 0 replies; 10+ messages in thread
From: Hans van Kranenburg @ 2019-05-22  7:52 UTC (permalink / raw)
  To: Mike Snitzer
  Cc: Milan Broz, Victor Helmholtz, dm-crypt,
	device-mapper development, Mikulas Patocka

On 5/22/19 12:07 AM, Mike Snitzer wrote:
> On Tue, May 21 2019 at  4:33pm -0400,
> Hans van Kranenburg <hans@knorrie.org> wrote:
> 
>> Hi,
>>
>> On 5/21/19 10:43 AM, Hans van Kranenburg wrote:
>>> Hi,
>>>
>>> I'm seeing the same lockup, also 4.19. This is mdadm RAID10 on top of 4x
>>> a partition with only dm-integrity.
>>>
>>> It just happened out of the blue, no heavy load or anything. All IO to
>>> it is frozen now.
>>>
>>> [...]
>>
>> There it is again... dmesg dump below. All cpus on 100% iowait.
>>
>> It's triggered after a few minutes by running some Windows 2019 server
>> install (ugh, don't ask) in a Xen HVM domU, which writes into a raw
>> sparse file on a btrfs filesystem on LVM on mdadm RAID10 on 4x
>> dm-integrity (wheeee!!)...
>>
>> This morning it was triggered a few minutes after starting an old
>> windows 2008 server image that I copied to this machine.
>>
>> When running only other Linux vms, and when copying data onto
>> filesystems that live in LVM logical volumes I haven't seen this problem
>> yet, at all, in the last few weeks that this machine is running.
>>
>> I noticed there's a "dm integrity: fix deadlock with overlapping I/O"
>> fix in a later 4.19. Is there any chance this is related? I have no
>> idea, but any hints or suggestions about what to try would be appreciated.
> 
> Yes, all your hung tasks are hung in wait_and_add_new_range().
> 
> Please use that later 4.19 or apply commit 4ed319c6ac08 ("dm integrity:
> fix deadlock with overlapping I/O")

Oh, silly me, that one is in 4.19.35 and I already have 4.9.37 based on
Debian in my repo, just rebooted the machine into it. Let's see what
happens today.

Thanks for confirming, sorry for the noise.

Hans

^ permalink raw reply	[flat|nested] 10+ messages in thread

* Re: Dm-integrity freeze
@ 2019-05-22  7:52           ` Hans van Kranenburg
  0 siblings, 0 replies; 10+ messages in thread
From: Hans van Kranenburg @ 2019-05-22  7:52 UTC (permalink / raw)
  To: Mike Snitzer
  Cc: dm-crypt, device-mapper development, Mikulas Patocka, Milan Broz,
	Victor Helmholtz

On 5/22/19 12:07 AM, Mike Snitzer wrote:
> On Tue, May 21 2019 at  4:33pm -0400,
> Hans van Kranenburg <hans@knorrie.org> wrote:
> 
>> Hi,
>>
>> On 5/21/19 10:43 AM, Hans van Kranenburg wrote:
>>> Hi,
>>>
>>> I'm seeing the same lockup, also 4.19. This is mdadm RAID10 on top of 4x
>>> a partition with only dm-integrity.
>>>
>>> It just happened out of the blue, no heavy load or anything. All IO to
>>> it is frozen now.
>>>
>>> [...]
>>
>> There it is again... dmesg dump below. All cpus on 100% iowait.
>>
>> It's triggered after a few minutes by running some Windows 2019 server
>> install (ugh, don't ask) in a Xen HVM domU, which writes into a raw
>> sparse file on a btrfs filesystem on LVM on mdadm RAID10 on 4x
>> dm-integrity (wheeee!!)...
>>
>> This morning it was triggered a few minutes after starting an old
>> windows 2008 server image that I copied to this machine.
>>
>> When running only other Linux vms, and when copying data onto
>> filesystems that live in LVM logical volumes I haven't seen this problem
>> yet, at all, in the last few weeks that this machine is running.
>>
>> I noticed there's a "dm integrity: fix deadlock with overlapping I/O"
>> fix in a later 4.19. Is there any chance this is related? I have no
>> idea, but any hints or suggestions about what to try would be appreciated.
> 
> Yes, all your hung tasks are hung in wait_and_add_new_range().
> 
> Please use that later 4.19 or apply commit 4ed319c6ac08 ("dm integrity:
> fix deadlock with overlapping I/O")

Oh, silly me, that one is in 4.19.35 and I already have 4.9.37 based on
Debian in my repo, just rebooted the machine into it. Let's see what
happens today.

Thanks for confirming, sorry for the noise.

Hans

^ permalink raw reply	[flat|nested] 10+ messages in thread

end of thread, other threads:[~2019-05-22  7:52 UTC | newest]

Thread overview: 10+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2019-03-07 15:26 [dm-crypt] Dm-integrity freeze Victor Helmholtz
2019-03-07 16:37 ` Milan Broz
2019-05-21  8:43   ` [dm-crypt] [dm-devel] " Hans van Kranenburg
2019-05-21  8:43     ` [dm-crypt] " Hans van Kranenburg
2019-05-21 20:33     ` [dm-crypt] [dm-devel] " Hans van Kranenburg
2019-05-21 20:33       ` [dm-crypt] " Hans van Kranenburg
2019-05-21 22:07       ` Mike Snitzer
2019-05-21 22:07         ` Mike Snitzer
2019-05-22  7:52         ` [dm-crypt] " Hans van Kranenburg
2019-05-22  7:52           ` Hans van Kranenburg

This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.