linux-block.vger.kernel.org archive mirror
 help / color / mirror / Atom feed
* blkdev_get deadlock
@ 2019-05-03 17:47 Evan Green
  2019-05-04  1:15 ` Bart Van Assche
  0 siblings, 1 reply; 3+ messages in thread
From: Evan Green @ 2019-05-03 17:47 UTC (permalink / raw)
  To: linux-block, LKML; +Cc: Jens Axboe, Bart Van Assche

Hey blockies,

I'm seeing a hung task in the kernel, and I wanted to share it in case
it's a known issue. I'm still trying to wrap my head around the stacks
myself. This is our Chrome OS 4.19 kernel, which is admittedly not
100% vanilla mainline master, but we try to keep it pretty close.

I can reproduce this reliably within our chrome OS installer, where
it's trying to dd from my system disk (NVMe) to a loop device backed
by a removable UFS card (4kb sectors) in a USB dongle.
-Evan

[  371.440691] INFO: task udevd:197 blocked for more than 120 seconds.
[  371.447807]       Not tainted 4.19.37 #8
[  371.452300] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
disables this message.
[  371.461261] udevd           D    0   197      1 0x00000000
[  371.467511] Call Trace:
[  371.470314]  ? __schedule+0x1308/0x1c2f
[  371.474650]  ? lock_acquire+0x377/0x4aa
[  371.478984]  ? is_mmconf_reserved+0x3bf/0x3bf
[  371.483893]  ? do_raw_spin_unlock+0x4f/0x24e
[  371.488702]  ? _raw_spin_unlock+0x7e/0xfa
[  371.493210]  ? _raw_spin_lock_bh+0x49/0x49
[  371.497816]  schedule+0x112/0x13c
[  371.501559]  schedule_preempt_disabled+0x17/0x22
[  371.506736]  __mutex_lock_common+0xe69/0x22b4
[  371.511634]  ? __blkdev_get+0x10f/0xdf2
[  371.515980]  ? mutex_lock_io_nested+0x5a/0x5a
[  371.520865]  ? cancel_work_sync+0xe/0xe
[  371.525172]  ? __mutex_unlock_slowpath+0x1b5/0x5fa
[  371.530575]  ? _raw_spin_unlock+0xfa/0xfa
[  371.535107]  ? mutex_unlock+0x10/0x10
[  371.539255]  mutex_lock_nested+0x20/0x26
[  371.543694]  __blkdev_get+0x10f/0xdf2
[  371.547800]  ? bd_acquire+0x206/0x23a
[  371.551912]  ? __lock_acquire+0x41fb/0x41fb
[  371.556667]  ? lock_acquire+0x377/0x4aa
[  371.560977]  ? blkdev_get+0x88e/0x88e
[  371.565094]  blkdev_get+0x114/0x88e
[  371.569201]  ? _raw_spin_unlock+0x7e/0xfa
[  371.573707]  ? _raw_spin_lock_bh+0x49/0x49
[  371.578311]  ? selinux_file_receive+0xdd/0xdd
[  371.583202]  ? do_raw_spin_unlock+0x4f/0x24e
[  371.587995]  ? bd_set_size+0xa1/0xa1
[  371.592019]  ? bd_acquire+0x206/0x23a
[  371.596125]  ? blkdev_open+0x11a/0x224
[  371.600339]  do_dentry_open+0x78b/0xe21
[  371.604652]  ? __devcgroup_check_permission+0x1dd/0x229
[  371.611026]  ? block_ioctl+0xee/0xee
[  371.615045]  ? finish_open+0xad/0xad
[  371.619067]  ? security_inode_permission+0x73/0xbb
[  371.624501]  path_openat+0x953/0x34a0
[  371.628625]  ? lock_acquire+0x4aa/0x4aa
[  371.632936]  ? __irqentry_text_end+0x1f9a26/0x1f9a26
[  371.638526]  ? do_filp_open+0x381/0x381
[  371.642831]  ? kasan_kmalloc+0x1aa/0x1d6
[  371.647227]  ? kmem_cache_alloc+0xeb/0x2b1
[  371.651822]  ? __alloc_fd+0x4b7/0x5ad
[  371.655968]  ? __lock_acquire+0x41fb/0x41fb
[  371.660668]  ? expand_files+0xe5/0x66a
[  371.664913]  ? do_raw_spin_lock+0xbd/0x1e9
[  371.669543]  do_filp_open+0x23a/0x381
[  371.673701]  ? vfs_tmpfile+0x1cd/0x1cd
[  371.677951]  ? __alloc_fd+0x4b7/0x5ad
[  371.682049]  ? exit_files+0x90/0x90
[  371.685974]  do_sys_open+0x2af/0x967
[  371.689993]  ? prepare_exit_to_usermode+0x2e0/0x2e0
[  371.695464]  ? file_open_root+0x49f/0x49f
[  371.700132]  ? trace_irq_disable_rcuidle+0x75/0x228
[  371.705611]  ? trace_irq_enable_rcuidle+0x75/0x228
[  371.710983]  ? trace_hardirqs_off+0x3f/0x3f
[  371.715683]  ? trace_hardirqs_on_thunk+0x1a/0x1c
[  371.720871]  ? trace_hardirqs_on+0x3f/0x3f
[  371.725504]  ? do_syscall_64+0x28/0x120
[  371.729817]  ? filp_close+0xd3/0x10a
[  371.733837]  do_syscall_64+0xcd/0x120
[  371.737950]  entry_SYSCALL_64_after_hwframe+0x49/0xbe
[  371.743613] RIP: 0033:0x7c577338adcf
[  371.747667] Code: Bad RIP value.
[  371.751287] RSP: 002b:00007fff83e955c0 EFLAGS: 00000246 ORIG_RAX:
0000000000000101
[  371.759776] RAX: ffffffffffffffda RBX: 00005c8073608370 RCX: 00007c577338adcf
[  371.767805] RDX: 00000000000a0800 RSI: 00005c807364e510 RDI: 00000000ffffff9c
[  371.775860] RBP: 00007fff83e95ba0 R08: 0000000000000004 R09: 00005c80720b5b00
[  371.783867] R10: 0000000000000000 R11: 0000000000000246 R12: 00005c807361d360
[  371.791860] R13: 00005c80736082a0 R14: 00007fff83e95660 R15: 00007fff83e95670
[  371.799938] INFO: lockdep is turned off.
[  371.804377]   task                        PC stack   pid father
[  371.811077] kworker/u16:3   D    0   119      2 0x80000000
[  371.817273] Workqueue: writeback wb_workfn (flush-8:0)
[  371.823079] Call Trace:
[  371.825827]  ? __schedule+0x1308/0x1c2f
[  371.830140]  ? try_to_wake_up+0xa20/0xf52
[  371.834817]  ? __lock_acquire+0x41fb/0x41fb
[  371.839518]  ? __sbitmap_get_word+0x96/0x16a
[  371.844328]  ? is_mmconf_reserved+0x3bf/0x3bf
[  371.849218]  ? blk_poll+0xc7/0xc7
[  371.852942]  ? sbitmap_get_shallow+0x11e/0x268
[  371.857926]  schedule+0x112/0x13c
[  371.861673]  io_schedule+0x19/0x69
[  371.865553]  blk_mq_get_tag+0x4ff/0xaee
[  371.869855]  ? __blk_mq_tag_idle+0x11c/0x11c
[  371.874646]  ? do_raw_spin_lock+0x84/0x1e9
[  371.879239]  ? lock_acquire+0x4aa/0x4aa
[  371.883540]  blk_mq_get_request+0x47d/0xfc1
[  371.888236]  ? blk_put_request+0xd4/0xd4
[  371.892641]  ? blk_mq_alloc_request+0x23a/0x23a
[  371.897782]  ? trace_hardirqs_on+0x3f/0x3f
[  371.902386]  ? print_irqtrace_events+0x223/0x223
[  371.907568]  ? update_cfs_rq_load_avg+0x31b/0x5ee
[  371.912850]  ? update_load_avg+0x203/0xe4c
[  371.917442]  ? __update_load_avg_se+0x8e1/0x8e1
[  371.922554]  ? update_stats_wait_end+0x88/0x644
[  371.927642]  ? ___slab_alloc+0x396/0x432
[  371.932047]  ? sched_group_set_shares+0x2f2/0x2f2
[  371.937336]  ? set_next_entity+0x191/0x46c
[  371.941950]  ? generic_make_request+0x410/0xb50
[  371.947027]  ? __lock_acquire+0x41fb/0x41fb
[  371.951725]  ? check_preempt_wakeup+0x6c9/0x6c9
[  371.956807]  ? prepare_lock_switch+0x94/0x119
[  371.961862]  ? __schedule+0x1308/0x1c2f
[  371.966193]  ? wait_on_page_bit_common+0x4ea/0x64b
[  371.971566]  ? generic_block_bmap+0x1d9/0x1d9
[  371.976471]  ? blk_flush_plug_list+0x496/0x912
[  371.981580]  ? submit_bh_wbc+0x4e5/0x5e9
[  371.986087]  ? is_mmconf_reserved+0x3bf/0x3bf
[  371.991028]  ? blk_poll+0xc7/0xc7
[  371.994901]  ? block_size_bits+0x1a/0x1a
[  371.999503]  ? _raw_spin_unlock_irq+0x83/0x100
[  372.004489]  ? schedule+0x112/0x13c
[  372.008438]  ? __block_write_full_page+0x65c/0x9b2
[  372.013808]  ? io_schedule+0x19/0x69
[  372.017824]  ? wait_on_page_bit_common+0x536/0x64b
[  372.023201]  ? mark_buffer_write_io_error+0xd7/0xd7
[  372.028666]  ? blkdev_direct_IO+0x9e/0x9e
[  372.033166]  ? wait_on_page_bit+0x37/0x37
[  372.037666]  ? trace_raw_output_file_check_and_advance_wb_err+0x1b4/0x1b4
[  372.045329]  ? write_cache_pages+0x9c8/0xf0f
[  372.050122]  ? generic_writepages+0x147/0x147
[  372.055035]  ? tag_pages_for_writeback+0x327/0x327
[  372.060413]  ? __update_load_avg_se+0x8e1/0x8e1
[  372.065496]  ? lock_acquire+0x4aa/0x4aa
[  372.069835]  ? __writeback_single_inode+0x727/0x1146
[  372.075398]  ? __lock_acquire+0x41fb/0x41fb
[  372.080113]  ? generic_writepages+0xeb/0x147
[  372.084907]  ? clear_page_dirty_for_io+0x43d/0x43d
[  372.090472]  ? update_cfs_rq_load_avg+0x31b/0x5ee
[  372.095758]  ? lock_acquire+0x4aa/0x4aa
[  372.100062]  ? do_raw_spin_unlock+0x4f/0x24e
[  372.104846]  ? lock_acquire+0x4aa/0x4aa
[  372.109154]  ? _raw_spin_unlock+0x7e/0xfa
[  372.113654]  ? _raw_spin_lock_bh+0x49/0x49
[  372.118244]  ? do_writepages+0xaf/0x12c
[  372.122553]  ? __writeback_single_inode+0x327/0x1146
[  372.128128]  ? redirty_tail+0x136/0x136
[  372.132429]  ? __lock_acquire+0x41fb/0x41fb
[  372.137123]  ? lock_acquire+0x377/0x4aa
[  372.141432]  ? writeback_sb_inodes+0xcf0/0xe5b
[  372.146430]  ? lock_downgrade+0x60a/0x60a
[  372.150959]  ? do_raw_spin_unlock+0x4f/0x24e
[  372.155749]  ? _raw_spin_unlock+0x7e/0xfa
[  372.160250]  ? _raw_spin_lock_bh+0x49/0x49
[  372.164849]  ? do_raw_spin_lock+0xbd/0x1e9
[  372.169446]  ? do_raw_spin_lock+0xbd/0x1e9
[  372.174035]  ? writeback_sb_inodes+0x4d9/0xe5b
[  372.179026]  ? queue_io+0x545/0x545
[  372.182934]  ? wb_over_bg_thresh+0x202/0x3c4
[  372.187714]  ? down_read_trylock+0x61/0xaa
[  372.192310]  ? trylock_super+0x1f/0xa6
[  372.196553]  ? __writeback_inodes_wb+0x110/0x1d1
[  372.201726]  ? wb_writeback+0x885/0xdcd
[  372.206033]  ? wb_io_lists_depopulated+0x10d/0x10d
[  372.211407]  ? wb_over_bg_thresh+0x202/0x3c4
[  372.216345]  ? balance_dirty_pages+0x16cc/0x16cc
[  372.221523]  ? trace_irq_enable_rcuidle+0x75/0x228
[  372.226897]  ? lock_downgrade+0x5b2/0x60a
[  372.231388]  ? __local_bh_enable_ip+0x10b/0x1b4
[  372.236472]  ? wb_workfn+0x810/0xeda
[  372.240519]  ? wb_workfn+0xb87/0xeda
[  372.244525]  ? update_cfs_rq_load_avg+0x31b/0x5ee
[  372.249797]  ? __inode_wait_for_writeback+0x294/0x294
[  372.255491]  ? trace_irq_enable_rcuidle+0x75/0x228
[  372.260860]  ? lock_acquire+0x377/0x4aa
[  372.265171]  ? trace_hardirqs_on+0x3f/0x3f
[  372.269817]  ? process_one_work+0x655/0x11b6
[  372.274609]  ? trace_irq_enable_rcuidle+0x75/0x228
[  372.279992]  ? lock_acquire+0x377/0x4aa
[  372.284302]  ? process_one_work+0x6a4/0x11b6
[  372.289144]  ? lock_downgrade+0x60a/0x60a
[  372.293643]  ? _raw_spin_unlock_irq+0x83/0x100
[  372.298667]  ? _raw_spin_unlock_irqrestore+0x12e/0x12e
[  372.304433]  ? read_word_at_a_time+0x12/0x18
[  372.309239]  ? strscpy+0x6c/0x256
[  372.312984]  ? process_one_work+0x90b/0x11b6
[  372.317774]  ? worker_detach_from_pool+0x1fa/0x1fa
[  372.323193]  ? is_mmconf_reserved+0x3bf/0x3bf
[  372.328122]  ? lock_downgrade+0x60a/0x60a
[  372.332626]  ? trace_irq_disable_rcuidle+0x75/0x228
[  372.338187]  ? lockdep_hardirqs_on+0x6d8/0x6d8
[  372.343235]  ? _raw_spin_unlock_irq+0x83/0x100
[  372.348366]  ? do_raw_spin_lock+0xbd/0x1e9
[  372.352967]  ? worker_thread+0xad5/0xdcc
[  372.357413]  ? pr_cont_work+0xe6/0xe6
[  372.361528]  ? kthread+0x34e/0x35e
[  372.365349]  ? pr_cont_work+0xe6/0xe6
[  372.369492]  ? kthread_blkcg+0xa2/0xa2
[  372.373697]  ? ret_from_fork+0x24/0x50
[  372.377922] udevd           D    0   197      1 0x00000000
[  372.384124] Call Trace:
[  372.386874]  ? __schedule+0x1308/0x1c2f
[  372.391194]  ? lock_acquire+0x377/0x4aa
[  372.395520]  ? is_mmconf_reserved+0x3bf/0x3bf
[  372.400410]  ? do_raw_spin_unlock+0x4f/0x24e
[  372.405217]  ? _raw_spin_unlock+0x7e/0xfa
[  372.409704]  ? _raw_spin_lock_bh+0x49/0x49
[  372.414300]  schedule+0x112/0x13c
[  372.418046]  schedule_preempt_disabled+0x17/0x22
[  372.423227]  __mutex_lock_common+0xe69/0x22b4
[  372.428124]  ? __blkdev_get+0x10f/0xdf2
[  372.432424]  ? mutex_lock_io_nested+0x5a/0x5a
[  372.437307]  ? cancel_work_sync+0xe/0xe
[  372.441611]  ? __mutex_unlock_slowpath+0x1b5/0x5fa
[  372.447017]  ? _raw_spin_unlock+0xfa/0xfa
[  372.451538]  ? mutex_unlock+0x10/0x10
[  372.455679]  mutex_lock_nested+0x20/0x26
[  372.460141]  __blkdev_get+0x10f/0xdf2
[  372.464253]  ? bd_acquire+0x206/0x23a
[  372.468363]  ? __lock_acquire+0x41fb/0x41fb
[  372.473107]  ? lock_acquire+0x377/0x4aa
[  372.477408]  ? blkdev_get+0x88e/0x88e
[  372.481678]  blkdev_get+0x114/0x88e
[  372.485596]  ? _raw_spin_unlock+0x7e/0xfa
[  372.490113]  ? _raw_spin_lock_bh+0x49/0x49
[  372.494708]  ? selinux_file_receive+0xdd/0xdd
[  372.499637]  ? do_raw_spin_unlock+0x4f/0x24e
[  372.504431]  ? bd_set_size+0xa1/0xa1
[  372.508481]  ? bd_acquire+0x206/0x23a
[  372.512596]  ? blkdev_open+0x11a/0x224
[  372.516834]  do_dentry_open+0x78b/0xe21
[  372.521171]  ? __devcgroup_check_permission+0x1dd/0x229
[  372.527042]  ? block_ioctl+0xee/0xee
[  372.531075]  ? finish_open+0xad/0xad
[  372.535090]  ? security_inode_permission+0x73/0xbb
[  372.540469]  path_openat+0x953/0x34a0
[  372.544601]  ? lock_acquire+0x4aa/0x4aa
[  372.548920]  ? __irqentry_text_end+0x1f9a26/0x1f9a26
[  372.554514]  ? do_filp_open+0x381/0x381
[  372.558834]  ? kasan_kmalloc+0x1aa/0x1d6
[  372.563248]  ? kmem_cache_alloc+0xeb/0x2b1
[  372.567862]  ? __alloc_fd+0x4b7/0x5ad
[  372.572030]  ? __lock_acquire+0x41fb/0x41fb
[  372.576726]  ? expand_files+0xe5/0x66a
[  372.580934]  ? do_raw_spin_lock+0xbd/0x1e9
[  372.585601]  do_filp_open+0x23a/0x381
[  372.589726]  ? vfs_tmpfile+0x1cd/0x1cd
[  372.593947]  ? __alloc_fd+0x4b7/0x5ad
[  372.598072]  ? exit_files+0x90/0x90
[  372.602003]  do_sys_open+0x2af/0x967
[  372.606024]  ? prepare_exit_to_usermode+0x2e0/0x2e0
[  372.611677]  ? file_open_root+0x49f/0x49f
[  372.616189]  ? trace_irq_disable_rcuidle+0x75/0x228
[  372.621669]  ? trace_irq_enable_rcuidle+0x75/0x228
[  372.627058]  ? trace_hardirqs_off+0x3f/0x3f
[  372.631754]  ? trace_hardirqs_on_thunk+0x1a/0x1c
[  372.636941]  ? trace_hardirqs_on+0x3f/0x3f
[  372.641549]  ? do_syscall_64+0x28/0x120
[  372.645855]  ? filp_close+0xd3/0x10a
[  372.649882]  do_syscall_64+0xcd/0x120
[  372.653997]  entry_SYSCALL_64_after_hwframe+0x49/0xbe
[  372.659674] RIP: 0033:0x7c577338adcf
[  372.663692] Code: Bad RIP value.
[  372.667323] RSP: 002b:00007fff83e955c0 EFLAGS: 00000246 ORIG_RAX:
0000000000000101
[  372.675825] RAX: ffffffffffffffda RBX: 00005c8073608370 RCX: 00007c577338adcf
[  372.683834] RDX: 00000000000a0800 RSI: 00005c807364e510 RDI: 00000000ffffff9c
[  372.691845] RBP: 00007fff83e95ba0 R08: 0000000000000004 R09: 00005c80720b5b00
[  372.699877] R10: 0000000000000000 R11: 0000000000000246 R12: 00005c807361d360
[  372.707891] R13: 00005c80736082a0 R14: 00007fff83e95660 R15: 00007fff83e95670
[  372.715995] dd              D    0  5250   4342 0x00000080
[  372.722293] Call Trace:
[  372.725416]  ? __schedule+0x1308/0x1c2f
[  372.729861]  ? unwind_next_frame+0xdee/0x16d2
[  372.735412]  ? is_mmconf_reserved+0x3bf/0x3bf
[  372.740326]  ? blk_poll+0xc7/0xc7
[  372.744364]  ? do_raw_spin_unlock+0x4f/0x24e
[  372.749176]  ? sbitmap_get_shallow+0x11e/0x268
[  372.754170]  schedule+0x112/0x13c
[  372.757931]  ? unwind_next_frame+0xdee/0x16d2
[  372.762833]  io_schedule+0x19/0x69
[  372.766682]  ? blk_mq_get_tag+0x4ff/0xaee
[  372.771179]  ? __blk_mq_tag_idle+0x11c/0x11c
[  372.775964]  ? do_raw_spin_lock+0x84/0x1e9
[  372.780578]  ? init_wait_entry+0xd6/0xd6
[  372.784976]  ? _raw_spin_unlock+0xdf/0xfa
[  372.789471]  ? blk_mq_get_request+0x47d/0xfc1
[  372.794363]  ? blk_mq_alloc_request+0x23a/0x23a
[  372.799446]  ? trace_hardirqs_on+0x3f/0x3f
[  372.804032]  ? blk_mq_make_request+0x620/0x15ff
[  372.809119]  ? blk_mq_requeue_work+0x5b8/0x5b8
[  372.814113]  ? trace_tlb_flush_rcuidle+0x79/0x280
[  372.819400]  ? print_irqtrace_events+0x223/0x223
[  372.824586]  ? ___slab_alloc+0x430/0x432
[  372.829008]  ? mempool_alloc_slab+0x15/0x17
[  372.833701]  ? blk_queue_enter+0x359/0x3a9
[  372.838305]  ? blk_alloc_queue_node+0x934/0x934
[  372.843399]  ? switch_mm_irqs_off+0x8a2/0x1445
[  372.848441]  ? generic_make_request+0x410/0xb50
[  372.853530]  ? blk_rq_bio_prep+0x233/0x233
[  372.858129]  ? print_irqtrace_events+0x223/0x223
[  372.863292]  ? submit_bio+0x21d/0x4b2
[  372.867404]  ? direct_make_request+0x165/0x165
[  372.872538]  ? bio_chain_endio+0xe1/0xe1
[  372.877136]  ? bio_add_page+0x7d/0xc5
[  372.881371]  ? submit_bh_wbc+0x4e5/0x5e9
[  372.885884]  ? block_size_bits+0x1a/0x1a
[  372.890393]  ? __block_write_full_page+0x65c/0x9b2
[  372.895958]  ? mark_buffer_write_io_error+0xd7/0xd7
[  372.901518]  ? blkdev_direct_IO+0x9e/0x9e
[  372.906144]  ? clean_bdev_aliases+0x676/0x676
[  372.911134]  ? wait_on_page_bit+0x37/0x37
[  372.915753]  ? block_write_full_page+0x11c/0x2c6
[  372.921073]  ? blkdev_direct_IO+0x9e/0x9e
[  372.925679]  ? trace_raw_output_file_check_and_advance_wb_err+0x1b4/0x1b4
[  372.933401]  ? clear_page_dirty_for_io+0x3aa/0x43d
[  372.938891]  ? write_cache_pages+0x9c8/0xf0f
[  372.943953]  ? generic_writepages+0x147/0x147
[  372.948884]  ? tag_pages_for_writeback+0x327/0x327
[  372.954265]  ? _raw_spin_unlock_irq+0x83/0x100
[  372.959246]  ? schedule_timeout+0x3b/0x20d
[  372.963877]  ? lock_downgrade+0x60a/0x60a
[  372.968384]  ? schedule_tail+0x267/0x267
[  372.972792]  ? __switch_to_asm+0x40/0x70
[  372.977222]  ? syscall_return_via_sysret+0xe/0x7e
[  372.982503]  ? generic_writepages+0xeb/0x147
[  372.987286]  ? clear_page_dirty_for_io+0x43d/0x43d
[  372.992688]  ? __mutex_lock_common+0x24e/0x22b4
[  372.997779]  ? do_writepages+0xaf/0x12c
[  373.002235]  ? __filemap_fdatawrite_range+0x237/0x2cf
[  373.007917]  ? blkdev_put+0x238/0x326
[  373.012051]  ? filemap_check_errors+0x19c/0x19c
[  373.017281]  ? preempt_schedule_common+0x24/0x56
[  373.022602]  ? preempt_schedule+0xc1/0xc8
[  373.027247]  ? trace_hardirqs_on+0x3f/0x3f
[  373.031864]  ? filemap_write_and_wait+0x5b/0xa7
[  373.036955]  ? __blkdev_put+0x1a4/0x54f
[  373.041255]  ? do_raw_spin_lock+0xbd/0x1e9
[  373.045883]  ? blkdev_get_by_dev+0x49/0x49
[  373.050482]  ? blkdev_put+0x238/0x326
[  373.054600]  ? blkdev_close+0x7b/0x9d
[  373.058738]  ? __fput+0x240/0x571
[  373.062471]  ? task_work_run+0x93/0x14e
[  373.066815]  ? prepare_exit_to_usermode+0x203/0x2e0
[  373.072318]  ? syscall_return_slowpath+0x12f/0x66e
[  373.077722]  ? prepare_exit_to_usermode+0x2e0/0x2e0
[  373.083222]  ? fput+0x10d/0x1d0
[  373.086776]  ? delayed_fput+0x68/0x68
[  373.090893]  ? trace_irq_enable_rcuidle+0x75/0x228
[  373.096270]  ? __put_unused_fd+0x44/0x139
[  373.100791]  ? filp_close+0xd3/0x10a
[  373.104813]  ? __x64_sys_close+0x69/0x8f
[  373.109213]  ? do_syscall_64+0xcd/0x120
[  373.113529]  ? entry_SYSCALL_64_after_hwframe+0x49/0xbe
[  373.119431] Binder:3211_1   D    0  8032   2347 0xa0020080
[  373.125593] Call Trace:
[  373.128542]  ? __schedule+0x1308/0x1c2f
[  373.132861]  ? is_mmconf_reserved+0x3bf/0x3bf
[  373.137770]  ? __might_sleep+0x4a/0xfc
[  373.141989]  schedule+0x112/0x13c
[  373.145742]  exit_mm+0x351/0x5e5
[  373.149406]  ? do_exit+0x229b/0x229b
[  373.153498]  ? do_exit+0xf36/0x229b
[  373.157418]  do_exit+0x5b2/0x229b
[  373.161182]  ? preempt_schedule_common+0x24/0x56
[  373.166359]  ? preempt_schedule+0xc1/0xc8
[  373.170872]  ? will_become_orphaned_pgrp+0x22f/0x22f
[  373.176460]  ? retint_kernel+0x2d/0x2d
[  373.180695]  ? ___preempt_schedule+0x16/0x18
[  373.185524]  ? _raw_spin_unlock_irq+0x83/0x100
[  373.190525]  ? _raw_spin_unlock_irq+0xe5/0x100
[  373.195510]  do_group_exit+0x225/0x260
[  373.199738]  get_signal+0x3f1/0x179e
[  373.203752]  ? lock_acquire+0x4aa/0x4aa
[  373.208067]  ? ptrace_do_notify+0x2c8/0x2c8
[  373.212766]  ? rcu_lock_release+0x4/0x1d
[  373.217203]  ? __lock_acquire+0x41fb/0x41fb
[  373.221907]  do_signal+0xd9/0x114a
[  373.225738]  ? audit_filter_inodes+0x365/0x3d5
[  373.230718]  ? signal_fault+0x1e5/0x1e5
[  373.235021]  ? rcu_lock_release+0x1d/0x1d
[  373.239532]  ? audit_take_context+0x3c4/0x50e
[  373.244430]  ? __audit_syscall_exit+0x554/0x6cb
[  373.249517]  ? trace_irq_enable_rcuidle+0x75/0x228
[  373.254904]  ? __audit_free+0x549/0x549
[  373.259468]  ? prepare_exit_to_usermode+0x20a/0x2e0
[  373.264952]  ? print_irqtrace_events+0x223/0x223
[  373.270145]  ? trace_irq_disable_rcuidle+0x75/0x228
[  373.275633]  ? trace_hardirqs_off+0x3f/0x3f
[  373.280340]  prepare_exit_to_usermode+0x230/0x2e0
[  373.285628]  syscall_return_slowpath+0x12f/0x66e
[  373.290816]  ? fput+0x74/0x1d0
[  373.294274]  ? prepare_exit_to_usermode+0x2e0/0x2e0
[  373.299757]  ? do_syscall_64+0x120/0x120
[  373.304178]  ? security_file_ioctl+0x62/0x95
[  373.308986]  ? __ia32_compat_sys_ioctl+0x182f/0x2112
[  373.314568]  ? do_int80_syscall_32+0x108/0x175
[  373.319586]  entry_INT80_compat+0x87/0xa0
[  373.324098] Binder:3211_2   D    0  8033   2347 0xa0020080
[  373.330252] Call Trace:
[  373.333002]  ? __schedule+0x1308/0x1c2f
[  373.337336]  ? _raw_spin_unlock_irqrestore+0xa5/0x12e
[  373.342994]  ? is_mmconf_reserved+0x3bf/0x3bf
[  373.347884]  ? _raw_spin_unlock_irqrestore+0xa5/0x12e
[  373.353576]  ? _raw_spin_unlock_irqrestore+0x113/0x12e
[  373.359351]  ? __wake_up_common+0x290/0x479
[  373.364054]  ? _raw_spin_unlock+0xfa/0xfa
[  373.368598]  schedule+0x112/0x13c
[  373.372324]  exit_mm+0x351/0x5e5
[  373.375952]  ? do_exit+0x229b/0x229b
[  373.379976]  ? do_exit+0xf36/0x229b
[  373.383902]  do_exit+0x5b2/0x229b
[  373.387644]  ? trace_irq_enable_rcuidle+0x75/0x228
[  373.393209]  ? will_become_orphaned_pgrp+0x22f/0x22f
[  373.398787]  ? lock_acquire+0x377/0x4aa
[  373.403100]  ? trace_hardirqs_on+0x3f/0x3f
[  373.407702]  ? _raw_spin_unlock_irq+0x83/0x100
[  373.412697]  ? print_irqtrace_events+0x223/0x223
[  373.417877]  ? _raw_spin_unlock_irq+0x83/0x100
[  373.422876]  do_group_exit+0x225/0x260
[  373.427088]  get_signal+0x3f1/0x179e
[  373.431113]  ? lock_acquire+0x4aa/0x4aa
[  373.435428]  ? ptrace_do_notify+0x2c8/0x2c8
[  373.440133]  ? rcu_lock_release+0x4/0x1d
[  373.444559]  ? __lock_acquire+0x41fb/0x41fb
[  373.449311]  do_signal+0xd9/0x114a
[  373.453149]  ? audit_filter_inodes+0x365/0x3d5
[  373.458142]  ? signal_fault+0x1e5/0x1e5
[  373.462454]  ? rcu_lock_release+0x1d/0x1d
[  373.466967]  ? audit_take_context+0x3c4/0x50e
[  373.471861]  ? __audit_syscall_exit+0x554/0x6cb
[  373.476957]  ? trace_irq_enable_rcuidle+0x75/0x228
[  373.482340]  ? __audit_free+0x549/0x549
[  373.486656]  ? prepare_exit_to_usermode+0x20a/0x2e0
[  373.492138]  ? print_irqtrace_events+0x223/0x223
[  373.497321]  ? trace_irq_disable_rcuidle+0x75/0x228
[  373.502801]  ? trace_hardirqs_off+0x3f/0x3f
[  373.507520]  prepare_exit_to_usermode+0x230/0x2e0
[  373.512803]  syscall_return_slowpath+0x12f/0x66e
[  373.517996]  ? fput+0x74/0x1d0
[  373.521674]  ? prepare_exit_to_usermode+0x2e0/0x2e0
[  373.527158]  ? do_syscall_64+0x120/0x120
[  373.531571]  ? security_file_ioctl+0x62/0x95
[  373.536372]  ? __ia32_compat_sys_ioctl+0x182f/0x2112
[  373.541948]  ? do_int80_syscall_32+0x108/0x175
[  373.546935]  entry_INT80_compat+0x87/0xa0
[  373.551485] DispSync        D    0  8034   2347 0xa0020080
[  373.557625] Call Trace:
[  373.560379]  ? __schedule+0x1308/0x1c2f
[  373.564691]  ? is_mmconf_reserved+0x3bf/0x3bf
[  373.569612]  ? __might_sleep+0x4a/0xfc
[  373.573825]  schedule+0x112/0x13c
[  373.577553]  exit_mm+0x351/0x5e5
[  373.581172]  ? do_exit+0x229b/0x229b
[  373.585186]  ? do_exit+0xf36/0x229b
[  373.589106]  do_exit+0x5b2/0x229b
[  373.592833]  ? trace_irq_enable_rcuidle+0x75/0x228
[  373.598202]  ? will_become_orphaned_pgrp+0x22f/0x22f
[  373.603774]  ? lock_acquire+0x377/0x4aa
[  373.608083]  ? trace_hardirqs_on+0x3f/0x3f
[  373.612670]  ? _raw_spin_unlock_irq+0x83/0x100
[  373.617656]  ? print_irqtrace_events+0x223/0x223
[  373.622847]  ? _raw_spin_unlock_irq+0x83/0x100
[  373.627890]  do_group_exit+0x225/0x260
[  373.632097]  get_signal+0x3f1/0x179e
[  373.636126]  ? lock_acquire+0x4aa/0x4aa
[  373.640447]  ? ptrace_do_notify+0x2c8/0x2c8
[  373.645143]  ? rcu_lock_release+0x4/0x1d
[  373.649724]  ? __lock_acquire+0x41fb/0x41fb
[  373.654438]  ? do_signal+0xd9/0x114a
[  373.658457]  do_signal+0xd9/0x114a
[  373.662308]  ? audit_filter_inodes+0x365/0x3d5
[  373.667292]  ? signal_fault+0x1e5/0x1e5
[  373.671599]  ? rcu_lock_release+0x1d/0x1d
[  373.676103]  ? audit_take_context+0x3c4/0x50e
[  373.680986]  ? __audit_syscall_exit+0x554/0x6cb
[  373.686074]  ? trace_irq_enable_rcuidle+0x75/0x228
[  373.691469]  ? __audit_free+0x549/0x549
[  373.695808]  ? prepare_exit_to_usermode+0x20a/0x2e0
[  373.701284]  ? print_irqtrace_events+0x223/0x223
[  373.706472]  ? trace_irq_disable_rcuidle+0x75/0x228
[  373.711941]  ? trace_hardirqs_off+0x3f/0x3f
[  373.716637]  prepare_exit_to_usermode+0x230/0x2e0
[  373.721919]  syscall_return_slowpath+0x12f/0x66e
[  373.727100]  ? prepare_exit_to_usermode+0x2e0/0x2e0
[  373.732574]  ? __ia32_compat_sys_futex+0x2d7/0x331
[  373.737940]  ? trace_hardirqs_on_thunk+0x1a/0x1c
[  373.743124]  ? __ia32_compat_sys_get_robust_list+0x26c/0x26c
[  373.749475]  ? do_int80_syscall_32+0x25/0x175
[  373.754380]  ? print_irqtrace_events+0x223/0x223
[  373.759566]  ? do_int80_syscall_32+0x108/0x175
[  373.764547]  entry_INT80_compat+0x87/0xa0
[  373.769053] appEventThread  D    0  8035   2347 0xa0020080
[  373.775229] Call Trace:
[  373.778127]  ? __schedule+0x1308/0x1c2f
[  373.782451]  ? is_mmconf_reserved+0x3bf/0x3bf
[  373.787361]  ? __might_sleep+0x4a/0xfc
[  373.791583]  schedule+0x112/0x13c
[  373.795356]  exit_mm+0x351/0x5e5
[  373.798984]  ? do_exit+0x229b/0x229b
[  373.803001]  ? do_exit+0xf36/0x229b
[  373.806901]  do_exit+0x5b2/0x229b
[  373.810629]  ? trace_irq_enable_rcuidle+0x75/0x228
[  373.816021]  ? will_become_orphaned_pgrp+0x22f/0x22f
[  373.821587]  ? lock_acquire+0x377/0x4aa
[  373.825909]  ? trace_hardirqs_on+0x3f/0x3f
[  373.830541]  ? _raw_spin_unlock_irq+0x83/0x100
[  373.835532]  ? print_irqtrace_events+0x223/0x223
[  373.840716]  ? _raw_spin_unlock_irq+0x83/0x100
[  373.845709]  do_group_exit+0x225/0x260
[  373.849921]  get_signal+0x3f1/0x179e
[  373.853938]  ? lock_acquire+0x4aa/0x4aa
[  373.858254]  ? ptrace_do_notify+0x2c8/0x2c8
[  373.862941]  ? rcu_lock_release+0x4/0x1d
[  373.867349]  ? __lock_acquire+0x41fb/0x41fb
[  373.872045]  ? do_signal+0xd9/0x114a
[  373.876059]  do_signal+0xd9/0x114a
[  373.879882]  ? audit_filter_inodes+0x365/0x3d5
[  373.884873]  ? signal_fault+0x1e5/0x1e5
[  373.889175]  ? rcu_lock_release+0x1d/0x1d
[  373.893679]  ? audit_take_context+0x3c4/0x50e
[  373.898570]  ? __audit_syscall_exit+0x554/0x6cb
[  373.903814]  ? trace_irq_enable_rcuidle+0x75/0x228
[  373.909191]  ? __audit_free+0x549/0x549
[  373.913516]  ? prepare_exit_to_usermode+0x20a/0x2e0
[  373.918991]  ? print_irqtrace_events+0x223/0x223
[  373.924173]  ? trace_irq_disable_rcuidle+0x75/0x228
[  373.929645]  ? trace_hardirqs_off+0x3f/0x3f
[  373.934355]  prepare_exit_to_usermode+0x230/0x2e0
[  373.939638]  syscall_return_slowpath+0x12f/0x66e
[  373.944815]  ? prepare_exit_to_usermode+0x2e0/0x2e0
[  373.950306]  ? __ia32_compat_sys_futex+0x2d7/0x331
[  373.955684]  ? trace_hardirqs_on_thunk+0x1a/0x1c
[  373.960860]  ? __ia32_compat_sys_get_robust_list+0x26c/0x26c
[  373.967220]  ? do_int80_syscall_32+0x25/0x175
[  373.972100]  ? print_irqtrace_events+0x223/0x223
[  373.977291]  ? do_int80_syscall_32+0x108/0x175
[  373.982281]  entry_INT80_compat+0x87/0xa0
[  373.986786] sfEventThread   D    0  8036   2347 0xa0020080
[  373.992942] Call Trace:
[  373.995695]  ? __schedule+0x1308/0x1c2f
[  374.000007]  ? is_mmconf_reserved+0x3bf/0x3bf
[  374.004898]  ? __might_sleep+0x4a/0xfc
[  374.009110]  schedule+0x112/0x13c
[  374.012830]  exit_mm+0x351/0x5e5
[  374.016484]  ? do_exit+0x229b/0x229b
[  374.020501]  ? do_exit+0xf36/0x229b
[  374.024418]  do_exit+0x5b2/0x229b
[  374.028148]  ? trace_irq_enable_rcuidle+0x75/0x228
[  374.033702]  ? will_become_orphaned_pgrp+0x22f/0x22f
[  374.039292]  ? lock_acquire+0x377/0x4aa
[  374.043617]  ? trace_hardirqs_on+0x3f/0x3f
[  374.048238]  ? _raw_spin_unlock_irq+0x83/0x100
[  374.053233]  ? print_irqtrace_events+0x223/0x223
[  374.058418]  ? _raw_spin_unlock_irq+0x83/0x100
[  374.063407]  do_group_exit+0x225/0x260
[  374.067612]  get_signal+0x3f1/0x179e
[  374.071619]  ? lock_acquire+0x4aa/0x4aa
[  374.075928]  ? ptrace_do_notify+0x2c8/0x2c8
[  374.080624]  ? rcu_lock_release+0x4/0x1d
[  374.085022]  ? __lock_acquire+0x41fb/0x41fb
[  374.089741]  ? do_signal+0xd9/0x114a
[  374.093789]  do_signal+0xd9/0x114a
[  374.097613]  ? audit_filter_inodes+0x365/0x3d5
[  374.102602]  ? signal_fault+0x1e5/0x1e5
[  374.106910]  ? rcu_lock_release+0x1d/0x1d
[  374.111415]  ? audit_take_context+0x3c4/0x50e
[  374.116307]  ? __audit_syscall_exit+0x554/0x6cb
[  374.121395]  ? trace_irq_enable_rcuidle+0x75/0x228
[  374.126773]  ? __audit_free+0x549/0x549
[  374.131079]  ? prepare_exit_to_usermode+0x20a/0x2e0
[  374.136557]  ? print_irqtrace_events+0x223/0x223
[  374.141730]  ? trace_irq_disable_rcuidle+0x75/0x228
[  374.147207]  ? trace_hardirqs_off+0x3f/0x3f
[  374.151896]  prepare_exit_to_usermode+0x230/0x2e0
[  374.157201]  syscall_return_slowpath+0x12f/0x66e
[  374.162589]  ? prepare_exit_to_usermode+0x2e0/0x2e0
[  374.168063]  ? __ia32_compat_sys_futex+0x2d7/0x331
[  374.173451]  ? trace_hardirqs_on_thunk+0x1a/0x1c
[  374.178652]  ? __ia32_compat_sys_get_robust_list+0x26c/0x26c
[  374.185001]  ? do_int80_syscall_32+0x25/0x175
[  374.189896]  ? print_irqtrace_events+0x223/0x223
[  374.195075]  ? do_int80_syscall_32+0x108/0x175
[  374.200065]  entry_INT80_compat+0x87/0xa0
[  374.204570] Binder:3211_3   D    0  8038   2347 0xa0020080
[  374.210721] Call Trace:
[  374.213479]  ? __schedule+0x1308/0x1c2f
[  374.217790]  ? is_mmconf_reserved+0x3bf/0x3bf
[  374.222685]  ? __might_sleep+0x4a/0xfc
[  374.226909]  schedule+0x112/0x13c
[  374.230635]  exit_mm+0x351/0x5e5
[  374.234246]  ? do_exit+0x229b/0x229b
[  374.238269]  ? do_exit+0xf36/0x229b
[  374.242225]  do_exit+0x5b2/0x229b
[  374.245951]  ? trace_irq_enable_rcuidle+0x75/0x228
[  374.251314]  ? will_become_orphaned_pgrp+0x22f/0x22f
[  374.256911]  ? lock_acquire+0x377/0x4aa
[  374.261214]  ? trace_hardirqs_on+0x3f/0x3f
[  374.265809]  ? _raw_spin_unlock_irq+0x83/0x100
[  374.270829]  ? print_irqtrace_events+0x223/0x223
[  374.276032]  ? _raw_spin_unlock_irq+0x83/0x100
[  374.281031]  do_group_exit+0x225/0x260
[  374.285264]  get_signal+0x3f1/0x179e
[  374.289282]  ? lock_acquire+0x4aa/0x4aa
[  374.293625]  ? ptrace_do_notify+0x2c8/0x2c8
[  374.298491]  ? rcu_lock_release+0x4/0x1d
[  374.302913]  ? __lock_acquire+0x41fb/0x41fb
[  374.307603]  do_signal+0xd9/0x114a
[  374.311437]  ? audit_filter_inodes+0x365/0x3d5
[  374.316462]  ? signal_fault+0x1e5/0x1e5
[  374.320771]  ? rcu_lock_release+0x1d/0x1d
[  374.325274]  ? audit_take_context+0x3c4/0x50e
[  374.330183]  ? __audit_syscall_exit+0x554/0x6cb
[  374.335263]  ? trace_irq_enable_rcuidle+0x75/0x228
[  374.340641]  ? __audit_free+0x549/0x549
[  374.344945]  ? prepare_exit_to_usermode+0x20a/0x2e0
[  374.350411]  ? print_irqtrace_events+0x223/0x223
[  374.355600]  ? trace_irq_disable_rcuidle+0x75/0x228
[  374.361109]  ? trace_hardirqs_off+0x3f/0x3f
[  374.365826]  prepare_exit_to_usermode+0x230/0x2e0
[  374.371113]  syscall_return_slowpath+0x12f/0x66e
[  374.376291]  ? fput+0x74/0x1d0
[  374.379722]  ? prepare_exit_to_usermode+0x2e0/0x2e0
[  374.385197]  ? do_syscall_64+0x120/0x120
[  374.389598]  ? security_file_ioctl+0x62/0x95
[  374.394409]  ? __ia32_compat_sys_ioctl+0x182f/0x2112
[  374.399985]  ? do_int80_syscall_32+0x108/0x175
[  374.404974]  entry_INT80_compat+0x87/0xa0
[  374.409479] crash_reporter  D    0  8043      2 0x00000080
[  374.415626] Call Trace:
[  374.418380]  ? __schedule+0x1308/0x1c2f
[  374.422698]  ? security_sb_umount+0x5a/0x89
[  374.427402]  ? is_mmconf_reserved+0x3bf/0x3bf
[  374.432479]  ? do_wait_for_common+0x381/0x5c2
[  374.437379]  ? trace_irq_enable_rcuidle+0x75/0x228
[  374.442759]  schedule+0x112/0x13c
[  374.446491]  schedule_timeout+0xa1/0x20d
[  374.450897]  ? console_conditional_schedule+0x29/0x29
[  374.456576]  ? _raw_spin_unlock_irq+0x83/0x100
[  374.461576]  ? _raw_spin_unlock_irqrestore+0x12e/0x12e
[  374.467370]  ? trace_irq_enable_rcuidle+0x75/0x228
[  374.472762]  ? avc_has_perm_noaudit+0x233/0x290
[  374.477863]  ? lock_downgrade+0x60a/0x60a
[  374.482385]  ? trace_hardirqs_on+0x3f/0x3f
[  374.487016]  ? __call_rcu+0x286/0x547
[  374.491142]  do_wait_for_common+0x391/0x5c2
[  374.495847]  ? console_conditional_schedule+0x29/0x29
[  374.501525]  ? wait_for_completion_killable_timeout+0x5f/0x5f
[  374.507992]  ? do_task_dead+0xb8/0xb8
[  374.512120]  ? invoke_rcu_core+0xdf/0xdf
[  374.516546]  ? umount_tree+0x7de/0xaad
[  374.520768]  ? do_raw_spin_lock+0xbd/0x1e9
[  374.525378]  wait_for_completion+0x4c/0x58
[  374.529997]  __wait_rcu_gp+0x29e/0x2ba
[  374.534220]  synchronize_rcu+0x161/0x1b4
[  374.538634]  ? cond_synchronize_rcu+0x24/0x24
[  374.543539]  ? sync_rcu_exp_handler+0x14a/0x14a
[  374.548637]  ? __lock_acquire+0x40fe/0x41fb
[  374.553345]  ? _raw_spin_unlock+0x7e/0xfa
[  374.557857]  ? rcu_read_lock_bh_held+0x1d8/0x1d8
[  374.563205]  namespace_unlock+0x11a/0x129
[  374.567710]  ? umount_tree+0xaad/0xaad
[  374.571936]  ? do_raw_spin_lock+0xbd/0x1e9
[  374.576540]  ? __x64_sys_umount+0x5a/0x63
[  374.581051]  ksys_umount+0xc68/0xcc6
[  374.585075]  ? kfree+0xe5/0x723
[  374.588612]  ? prepare_exit_to_usermode+0x2e0/0x2e0
[  374.594103]  ? namespace_unlock+0x129/0x129
[  374.598807]  ? trace_irq_enable_rcuidle+0x75/0x228
[  374.604216]  ? trace_hardirqs_off+0x3f/0x3f
[  374.608910]  ? trace_hardirqs_on_thunk+0x1a/0x1c
[  374.614122]  ? trace_hardirqs_on+0x3f/0x3f
[  374.618737]  ? do_syscall_64+0x28/0x120
[  374.623066]  ? print_irqtrace_events+0x223/0x223
[  374.628249]  __x64_sys_umount+0x5a/0x63
[  374.632550]  do_syscall_64+0xcd/0x120
[  374.636661]  entry_SYSCALL_64_after_hwframe+0x49/0xbe
[  374.642330] RIP: 0033:0x7a54d3c67e07
[  374.646341] Code: Bad RIP value.
[  374.649964] RSP: 002b:00007ffc2e494d38 EFLAGS: 00000286 ORIG_RAX:
00000000000000a6
[  374.658472] RAX: ffffffffffffffda RBX: 0000000000000003 RCX: 00007a54d3c67e07
[  374.666467] RDX: 000056b8fed1d1a0 RSI: 0000000000000002 RDI: 00007a54d434c661
[  374.674470] RBP: 00007ffc2e494dd0 R08: 0000000000000000 R09: 0000000000000000
[  374.682472] R10: 000000000000102f R11: 0000000000000286 R12: 0000000000000012
[  374.690485] R13: 00007ffc2e495a21 R14: 000056b8fed1cb50 R15: 000056b8fed1c7b0
[  374.698685] NMI backtrace for cpu 1
[  374.702724] CPU: 1 PID: 57 Comm: khungtaskd Not tainted 4.19.37 #8
[  374.719785] Call Trace:
[  374.722526]  dump_stack+0x122/0x1b5
[  374.726433]  ? arch_trigger_cpumask_backtrace+0x16/0x16
[  374.732297]  ? log_buf_vmcoreinfo_setup+0x131/0x131
[  374.737761]  ? swake_up_locked+0x75/0x139
[  374.742264]  ? show_regs_print_info+0x5/0x5
[  374.746958]  ? rcu_read_unlock_special+0x909/0xac8
[  374.752322]  nmi_cpu_backtrace+0xf7/0x197
[  374.756820]  ? nmi_trigger_cpumask_backtrace+0x2f2/0x2f2
[  374.762785]  ? rcu_lock_release+0x4/0x1d
[  374.767185]  ? arch_trigger_cpumask_backtrace+0x16/0x16
[  374.773043]  nmi_trigger_cpumask_backtrace+0x186/0x2f2
[  374.778804]  ? uevent_net_rcv_skb+0x3c1/0x3c1
[  374.783689]  ? __rcu_read_unlock+0xb7/0x141
[  374.788380]  ? show_state_filter+0x265/0x2e4
[  374.793162]  watchdog+0xeb0/0xec3
[  374.796884]  ? hungtask_pm_notify+0x25/0x25
[  374.801604]  ? _raw_spin_unlock+0xfa/0xfa
[  374.806093]  ? __kthread_parkme+0xc9/0x145
[  374.810690]  ? hungtask_pm_notify+0x25/0x25
[  374.815392]  kthread+0x34e/0x35e
[  374.819018]  ? hungtask_pm_notify+0x25/0x25
[  374.823711]  ? kthread_blkcg+0xa2/0xa2
[  374.827918]  ret_from_fork+0x24/0x50
[  374.832107] Sending NMI from CPU 1 to CPUs 0,2-7:
[  374.837425] NMI backtrace for cpu 0 skipped: idling at intel_idle+0x196/0x259
[  374.837427] NMI backtrace for cpu 2 skipped: idling at intel_idle+0x196/0x259
[  374.837497] NMI backtrace for cpu 4 skipped: idling at intel_idle+0x196/0x259
[  374.837499] NMI backtrace for cpu 5 skipped: idling at intel_idle+0x196/0x259
[  374.837501] NMI backtrace for cpu 6 skipped: idling at intel_idle+0x196/0x259
[  374.837503] NMI backtrace for cpu 7 skipped: idling at intel_idle+0x196/0x259
[  374.837523] NMI backtrace for cpu 3
[  374.837524] CPU: 3 PID: 116 Comm: kworker/u16:2 Not tainted 4.19.37 #8
[  374.837525] Workqueue: i915 __sleep_work
[  374.837527] RIP: 0010:qlist_move_cache+0x75/0x110
[  374.837544] Code: d0 4c 8d 58 08 48 8d 40 10 48 89 45 c8 49 bc 00
00 00 80 7f 77 00 00 eb 0b 4c 89 f7 eb 64 48 8b 7d d0 eb 73 48 89 d8
48 8b 1b <4c> 89 e1 48 be ff ff ff 7f ff ff ff ff 48 39 f0 76 08 48 8b
0c 25
[  374.837545] RSP: 0018:ffff88840701fb20 EFLAGS: 00000086
[  374.837546] RAX: ffff8883458b8388 RBX: ffff8883458baa08 RCX: 00000000000001c0
[  374.837549] RDX: ffff888408a3a140 RSI: 0000000000000000 RDI: ffff8883458ba848
[  374.837550] RBP: ffff88840701fb58 R08: ffffea0000000000 R09: ffffffff84e92a10
[  374.837552] R10: ffffffff84e92a10 R11: ffff88840701fb70 R12: 0000777f80000000
[  374.837554] R13: ffffffff84e92a08 R14: ffffffff84e92a00 R15: ffffffff84e92a08
[  374.837556] FS:  0000000000000000(0000) GS:ffff88840ecc0000(0000)
knlGS:0000000000000000
[  374.837558] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[  374.837560] CR2: 00000000def78ffc CR3: 0000000003c26003 CR4: 00000000003606e0
[  374.837562] Call Trace:
[  374.837564]  quarantine_remove_cache+0x74/0xfa
[  374.837566]  kmem_cache_shrink+0x1a/0x2e
[  374.837568]  __sleep_work+0xcc/0x1ab
[  374.837570]  process_one_work+0x90b/0x11b6
[  374.837572]  ? worker_detach_from_pool+0x1fa/0x1fa
[  374.837574]  ? is_mmconf_reserved+0x3bf/0x3bf
[  374.837577]  ? lock_downgrade+0x60a/0x60a
[  374.837578]  ? trace_irq_disable_rcuidle+0x75/0x228
[  374.837580]  ? lockdep_hardirqs_on+0x6d8/0x6d8
[  374.837582]  ? _raw_spin_unlock_irq+0x83/0x100
[  374.837584]  ? do_raw_spin_lock+0xbd/0x1e9
[  374.837586]  worker_thread+0xad5/0xdcc
[  374.837588]  ? pr_cont_work+0xe6/0xe6
[  374.837590]  kthread+0x34e/0x35e
[  374.837592]  ? pr_cont_work+0xe6/0xe6
[  374.837594]  ? kthread_blkcg+0xa2/0xa2
[  374.837596]  ret_from_fork+0x24/0x50
[  374.838582] Kernel panic - not syncing: hung_task: blocked tasks
[  375.093158] CPU: 1 PID: 57 Comm: khungtaskd Not tainted 4.19.37 #8
[  375.110218] Call Trace:
[  375.112968]  dump_stack+0x122/0x1b5
[  375.116881]  ? log_buf_vmcoreinfo_setup+0x131/0x131
[  375.122351]  ? show_regs_print_info+0x5/0x5
[  375.127044]  panic+0x1af/0x3c0
[  375.130470]  ? nmi_panic+0x6f/0x6f
[  375.134287]  ? __rcu_read_unlock+0xb7/0x141
[  375.138976]  ? show_state_filter+0x265/0x2e4
[  375.143766]  watchdog+0xec3/0xec3
[  375.147485]  ? hungtask_pm_notify+0x25/0x25
[  375.152169]  ? _raw_spin_unlock+0xfa/0xfa
[  375.156689]  ? __kthread_parkme+0xc9/0x145
[  375.161308]  ? hungtask_pm_notify+0x25/0x25
[  375.165994]  kthread+0x34e/0x35e
[  375.169616]  ? hungtask_pm_notify+0x25/0x25
[  375.174305]  ? kthread_blkcg+0xa2/0xa2
[  375.178499]  ret_from_fork+0x24/0x50
PANIC: hung_task: blocked tasks

^ permalink raw reply	[flat|nested] 3+ messages in thread

* Re: blkdev_get deadlock
  2019-05-03 17:47 blkdev_get deadlock Evan Green
@ 2019-05-04  1:15 ` Bart Van Assche
  2019-05-06 15:31   ` Evan Green
  0 siblings, 1 reply; 3+ messages in thread
From: Bart Van Assche @ 2019-05-04  1:15 UTC (permalink / raw)
  To: Evan Green, linux-block, LKML; +Cc: Jens Axboe

[-- Attachment #1: Type: text/plain, Size: 860 bytes --]

On 5/3/19 10:47 AM, Evan Green wrote:
> Hey blockies,
  ^^^^^^^^^^^^

That's the weirdest greeting I have encountered so far on the
linux-block mailing list.

> I'm seeing a hung task in the kernel, and I wanted to share it in case
> it's a known issue. I'm still trying to wrap my head around the stacks
> myself. This is our Chrome OS 4.19 kernel, which is admittedly not
> 100% vanilla mainline master, but we try to keep it pretty close.
> 
> I can reproduce this reliably within our chrome OS installer, where
> it's trying to dd from my system disk (NVMe) to a loop device backed
> by a removable UFS card (4kb sectors) in a USB dongle.

Although this is not the only possible cause such hangs are often caused
by a block driver or SCSI LLD not completing a request. A list of
pending requests can be obtained e.g. by running the attached script.

Bart.

[-- Attachment #2: list-pending-block-requests --]
[-- Type: text/plain, Size: 1501 bytes --]

#!/bin/bash

show_state() {
    local a dev=$1

    for a in device/state queue/scheduler; do
	[ -e "$dev/$a" ] && grep -aH . "$dev/$a"
    done
}

if [ -e /sys/kernel/debug/block ]; then
    devs=($(cd /sys/kernel/debug/block && echo ./*))
else
    devs=($(cd /sys/class/block && echo ./*))
fi

cd /sys/class/block || exit $?
for dev in "${devs[@]}"; do
    dev="${dev#./}"
    echo "$dev"
    pending=0
    if [ -e "$dev/mq" ]; then
	for f in "$dev"/mq/*/{pending,*/rq_list}; do
	    [ -e "$f" ] || continue
	    if { read -r line1 && read -r line2; } <"$f"; then
		echo "$f"
		echo "$line1 $line2" >/dev/null
		head -n 9 "$f"
		((pending++))
	    fi
	done
    fi
    (
	busy=0
	cd /sys/kernel/debug/block >&/dev/null &&
	    { grep -aH . $dev/requeue_list; true; } &&
	    for d in "$dev"/mq/hctx* "$dev"/hctx*; do
		[ ! -d "$d" ] && continue
		{ [ ! -e "$d/tags" ] ||
		      grep -q '^busy=0$' "$d/tags"; } &&
		    { [ ! -e "$d/sched_tags" ] ||
			  [ "$(<"$d/sched_tags")" = "" ] ||
			  grep -q '^busy=0$' "$d/sched_tags"; } && continue
		((busy++))
	        for f in "$d"/{active,busy,dispatch,flags,requeue_list,sched_tags,state,tags*,cpu*/rq_list,sched/*rqs}; do
		    [ -e "$f" ] && grep -aH . "$f"
		done
	    done
	exit $busy
    )
    pending=$((pending+$?))
    if [ "$pending" -gt 0 ]; then
	(
	    cd /sys/kernel/debug/block >&/dev/null &&
		if [ -e "$dev/mq/state" ]; then
		    grep -aH . "$dev/mq/state"
		else
		    grep -aH . "$dev/state"
		fi
	)
	show_state "$dev"
    fi
done

^ permalink raw reply	[flat|nested] 3+ messages in thread

* Re: blkdev_get deadlock
  2019-05-04  1:15 ` Bart Van Assche
@ 2019-05-06 15:31   ` Evan Green
  0 siblings, 0 replies; 3+ messages in thread
From: Evan Green @ 2019-05-06 15:31 UTC (permalink / raw)
  To: Bart Van Assche; +Cc: linux-block, LKML, Jens Axboe

On Fri, May 3, 2019 at 6:15 PM Bart Van Assche <bvanassche@acm.org> wrote:
>
> On 5/3/19 10:47 AM, Evan Green wrote:
> > Hey blockies,
>   ^^^^^^^^^^^^
>
> That's the weirdest greeting I have encountered so far on the
> linux-block mailing list.

Heh, achievement unlocked.

>
> > I'm seeing a hung task in the kernel, and I wanted to share it in case
> > it's a known issue. I'm still trying to wrap my head around the stacks
> > myself. This is our Chrome OS 4.19 kernel, which is admittedly not
> > 100% vanilla mainline master, but we try to keep it pretty close.
> >
> > I can reproduce this reliably within our chrome OS installer, where
> > it's trying to dd from my system disk (NVMe) to a loop device backed
> > by a removable UFS card (4kb sectors) in a USB dongle.
>
> Although this is not the only possible cause such hangs are often caused
> by a block driver or SCSI LLD not completing a request. A list of
> pending requests can be obtained e.g. by running the attached script.

Thanks for the script. I'll try a few different combinations of dd
involving the UFS card to see if I can at least remove the system disk
from the equation. Hopefully the system will still be responsive
enough to run the script if I keep it in the right place and maybe
pre-warm it up. I also might try an older kernel, since if it's a
misbehaving block device as you suggest then all kernel versions
should lock up.

>
> Bart.

^ permalink raw reply	[flat|nested] 3+ messages in thread

end of thread, other threads:[~2019-05-06 15:32 UTC | newest]

Thread overview: 3+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2019-05-03 17:47 blkdev_get deadlock Evan Green
2019-05-04  1:15 ` Bart Van Assche
2019-05-06 15:31   ` Evan Green

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).