Re: [next-20210827][ppc][multipathd] INFO: task hung in dm_table_add_target

[Abdul Haleem <abdhalee@linux.vnet.ibm.com>]

[-- Attachment #1: Type: text/plain, Size: 586 bytes --]


On 9/1/21 7:06 PM, Christoph Hellwig wrote:
> On Wed, Sep 01, 2021 at 04:47:26PM +0530, Abdul Haleem wrote:
>> Greeting's
>>
>> multiple task hung while adding the vfc disk back to the multipath on my
>> powerpc box running linux-next kernel
> Can you retry to reproduce this with lockdep enabled to see if there
> is anything interesting holding this lock?

LOCKDEP was earlier enabled by default

# cat .config | grep LOCKDEP
CONFIG_LOCKDEP_SUPPORT=y

BTW, Recreated again on 5.15.0-rc2 mainline kernel and attaching the logs

-- 
Regard's

Abdul Haleem
IBM Linux Technology Center


[-- Attachment #2: hungtasklogs.txt --]
[-- Type: text/plain, Size: 26095 bytes --]

device-mapper: multipath: 253:1: Reinstating path 8:16.
device-mapper: multipath: 253:0: Failing path 8:0.
device-mapper: multipath: 253:0: Failing path 8:32.
device-mapper: multipath: 253:0: Failing path 8:192.
device-mapper: multipath: 253:0: Failing path 8:208.
device-mapper: multipath: 253:0: Reinstating path 8:0.
device-mapper: multipath: 253:0: Reinstating path 8:32.
device-mapper: multipath: 253:0: Reinstating path 8:192.
device-mapper: multipath: 253:0: Reinstating path 8:208.
INFO: task multipathd:881519 blocked for more than 122 seconds.
      Not tainted 5.15.0-rc2+ #1
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
task:multipathd      state:D stack:    0 pid:881519 ppid:     1 flags:0x00040082
Call Trace:
[c000000096eff2b0] [c0000000ae18dd10] 0xc0000000ae18dd10 (unreliable)
[c000000096eff4a0] [c00000000001ea68] __switch_to+0x288/0x4a0
[c000000096eff500] [c000000000e07bfc] __schedule+0x30c/0x9f0
[c000000096eff5c0] [c000000000e08348] schedule+0x68/0x120
[c000000096eff5f0] [c000000000e08930] schedule_preempt_disabled+0x20/0x30
[c000000096eff610] [c000000000e0aedc] __mutex_lock.isra.11+0x36c/0x700
[c000000096eff6a0] [c000000000788e0c] bd_link_disk_holder+0x3c/0x280
[c000000096eff6f0] [c008000000fb5848] dm_get_table_device+0x1f0/0x2d0 [dm_mod]
[c000000096eff790] [c008000000fb9ce8] dm_get_device+0x130/0x2f0 [dm_mod]
[c000000096eff840] [c0080000011553b4] multipath_ctr+0x9cc/0x1000 [dm_multipath]
[c000000096eff9c0] [c008000000fba704] dm_table_add_target+0x1ac/0x420 [dm_mod]
[c000000096effa80] [c008000000fc0a04] table_load+0x16c/0x4c0 [dm_mod]
[c000000096effb30] [c008000000fc3734] ctl_ioctl+0x28c/0x7e0 [dm_mod]
[c000000096effd40] [c008000000fc3ca8] dm_ctl_ioctl+0x20/0x40 [dm_mod]
[c000000096effd60] [c000000000545db8] sys_ioctl+0xf8/0x150
[c000000096effdb0] [c000000000031074] system_call_exception+0x174/0x370
[c000000096effe10] [c00000000000c74c] system_call_common+0xec/0x250
--- interrupt: c00 at 0x7fffb86ac010
NIP:  00007fffb86ac010 LR: 00007fffb8a86924 CTR: 0000000000000000
REGS: c000000096effe80 TRAP: 0c00   Not tainted  (5.15.0-rc2+)
MSR:  800000000000d033 <SF,EE,PR,ME,IR,DR,RI,LE>  CR: 24042204  XER: 00000000
IRQMASK: 0 
GPR00: 0000000000000036 00007fffb7cec3a0 00007fffb8797300 0000000000000005 
GPR04: 00000000c138fd09 00007fffb0069c90 00007fffb8a8a118 00007fffb7cea298 
GPR08: 0000000000000005 0000000000000000 0000000000000000 0000000000000000 
GPR12: 0000000000000000 00007fffb7cf6300 00007fffb0069c90 00007fffb8a89e80 
GPR16: 00007fffb8a89e80 00007fffb8a89e80 00007fffb8ac3670 0000000000000000 
GPR20: 00007fffb8ac2040 00007fffb8a93460 00007fffb0069cc0 000001001f65ab80 
GPR24: 00007fffb8a89e80 00007fffb8a89e80 00007fffb8a89e80 0000000000000000 
GPR28: 00007fffb8a89e80 00007fffb8a89e80 0000000000000000 00007fffb8a89e80 
NIP [00007fffb86ac010] 0x7fffb86ac010
LR [00007fffb8a86924] 0x7fffb8a86924
--- interrupt: c00
INFO: task systemd-udevd:881738 blocked for more than 122 seconds.
      Not tainted 5.15.0-rc2+ #1
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
task:systemd-udevd   state:D stack:    0 pid:881738 ppid:   708 flags:0x00042482
Call Trace:
[c0000006b317b280] [c0000000007640a4] bio_associate_blkg+0x44/0xb0 (unreliable)
[c0000006b317b470] [c00000000001ea68] __switch_to+0x288/0x4a0
[c0000006b317b4d0] [c000000000e07bfc] __schedule+0x30c/0x9f0
[c0000006b317b590] [c000000000e08348] schedule+0x68/0x120
[c0000006b317b5c0] [c000000000e08adc] io_schedule+0x2c/0x50
[c0000006b317b5f0] [c0000000003ea624] __lock_page+0x1e4/0x430
[c0000006b317b6d0] [c000000000407fc8] truncate_inode_pages_range+0x338/0x8b0
[c0000006b317b850] [c000000000725714] kill_bdev.isra.14+0x44/0x60
[c0000006b317b880] [c0000000007261f4] blkdev_flush_mapping+0x54/0x260
[c0000006b317b960] [c000000000726488] blkdev_put_whole+0x88/0x90
[c0000006b317b9a0] [c00000000072714c] blkdev_put+0x1cc/0x280
[c0000006b317ba00] [c000000000727e9c] blkdev_close+0x3c/0x60
[c0000006b317ba30] [c000000000525694] __fput+0xc4/0x350
[c0000006b317ba80] [c000000000191128] task_work_run+0xf8/0x170
[c0000006b317bad0] [c000000000161c34] do_exit+0x4a4/0xd30
[c0000006b317bba0] [c000000000162594] do_group_exit+0x64/0xe0
[c0000006b317bbe0] [c000000000177fb8] get_signal+0x258/0xce0
[c0000006b317bcd0] [c0000000000219d4] do_notify_resume+0x114/0x480
[c0000006b317bd80] [c000000000030e40] interrupt_exit_user_prepare_main+0x1a0/0x260
[c0000006b317bde0] [c0000000000312e0] syscall_exit_prepare+0x70/0x150
[c0000006b317be10] [c00000000000c758] system_call_common+0xf8/0x250
--- interrupt: c00 at 0x7fff8c6c8674
NIP:  00007fff8c6c8674 LR: 00007fff8c757430 CTR: 0000000000000000
REGS: c0000006b317be80 TRAP: 0c00   Not tainted  (5.15.0-rc2+)
MSR:  800000000000d033 <SF,EE,PR,ME,IR,DR,RI,LE>  CR: 28002242  XER: 00000000
IRQMASK: 0 
GPR00: 0000000000000003 00007fffe0bff620 00007fff8c6f7f00 0000000000010000 
GPR04: 00007fff89980038 0000000000040000 00007fff8c6a0190 00007fff8c6a01a0 
GPR08: 00007fff8c6a0160 0000000000000000 0000000000000000 0000000000000000 
GPR12: 0000000000000000 00007fff8cc46480 0000000000000000 0000000000bab10c 
GPR16: 0cb1ba0000000000 00007fff8c79b700 0000000000000000 0000000000000002 
GPR20: 0000000000040000 000001002710e800 00000100270680f8 00007fff8c798d28 
GPR24: 00007fff8c7c03a4 00007fff8c7c03a4 00007fff89980028 000001002710e850 
GPR28: 0000000000040000 0000000c7ff80000 00007fff89980010 000001002710e800 
NIP [00007fff8c6c8674] 0x7fff8c6c8674
LR [00007fff8c757430] 0x7fff8c757430
--- interrupt: c00
INFO: task multipathd:881519 blocked for more than 245 seconds.
      Not tainted 5.15.0-rc2+ #1
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
task:multipathd      state:D stack:    0 pid:881519 ppid:     1 flags:0x00040082
Call Trace:
[c000000096eff2b0] [c0000000ae18dd10] 0xc0000000ae18dd10 (unreliable)
[c000000096eff4a0] [c00000000001ea68] __switch_to+0x288/0x4a0
[c000000096eff500] [c000000000e07bfc] __schedule+0x30c/0x9f0
[c000000096eff5c0] [c000000000e08348] schedule+0x68/0x120
[c000000096eff5f0] [c000000000e08930] schedule_preempt_disabled+0x20/0x30
[c000000096eff610] [c000000000e0aedc] __mutex_lock.isra.11+0x36c/0x700
[c000000096eff6a0] [c000000000788e0c] bd_link_disk_holder+0x3c/0x280
[c000000096eff6f0] [c008000000fb5848] dm_get_table_device+0x1f0/0x2d0 [dm_mod]
[c000000096eff790] [c008000000fb9ce8] dm_get_device+0x130/0x2f0 [dm_mod]
[c000000096eff840] [c0080000011553b4] multipath_ctr+0x9cc/0x1000 [dm_multipath]
[c000000096eff9c0] [c008000000fba704] dm_table_add_target+0x1ac/0x420 [dm_mod]
[c000000096effa80] [c008000000fc0a04] table_load+0x16c/0x4c0 [dm_mod]
[c000000096effb30] [c008000000fc3734] ctl_ioctl+0x28c/0x7e0 [dm_mod]
[c000000096effd40] [c008000000fc3ca8] dm_ctl_ioctl+0x20/0x40 [dm_mod]
[c000000096effd60] [c000000000545db8] sys_ioctl+0xf8/0x150
[c000000096effdb0] [c000000000031074] system_call_exception+0x174/0x370
[c000000096effe10] [c00000000000c74c] system_call_common+0xec/0x250
--- interrupt: c00 at 0x7fffb86ac010
NIP:  00007fffb86ac010 LR: 00007fffb8a86924 CTR: 0000000000000000
REGS: c000000096effe80 TRAP: 0c00   Not tainted  (5.15.0-rc2+)
MSR:  800000000000d033 <SF,EE,PR,ME,IR,DR,RI,LE>  CR: 24042204  XER: 00000000
IRQMASK: 0 
GPR00: 0000000000000036 00007fffb7cec3a0 00007fffb8797300 0000000000000005 
GPR04: 00000000c138fd09 00007fffb0069c90 00007fffb8a8a118 00007fffb7cea298 
GPR08: 0000000000000005 0000000000000000 0000000000000000 0000000000000000 
GPR12: 0000000000000000 00007fffb7cf6300 00007fffb0069c90 00007fffb8a89e80 
GPR16: 00007fffb8a89e80 00007fffb8a89e80 00007fffb8ac3670 0000000000000000 
GPR20: 00007fffb8ac2040 00007fffb8a93460 00007fffb0069cc0 000001001f65ab80 
GPR24: 00007fffb8a89e80 00007fffb8a89e80 00007fffb8a89e80 0000000000000000 
GPR28: 00007fffb8a89e80 00007fffb8a89e80 0000000000000000 00007fffb8a89e80 
NIP [00007fffb86ac010] 0x7fffb86ac010
LR [00007fffb8a86924] 0x7fffb8a86924
--- interrupt: c00
INFO: task systemd-udevd:881738 blocked for more than 245 seconds.
      Not tainted 5.15.0-rc2+ #1
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
task:systemd-udevd   state:D stack:    0 pid:881738 ppid:   708 flags:0x00042482
Call Trace:
[c0000006b317b280] [c0000000007640a4] bio_associate_blkg+0x44/0xb0 (unreliable)
[c0000006b317b470] [c00000000001ea68] __switch_to+0x288/0x4a0
[c0000006b317b4d0] [c000000000e07bfc] __schedule+0x30c/0x9f0
[c0000006b317b590] [c000000000e08348] schedule+0x68/0x120
[c0000006b317b5c0] [c000000000e08adc] io_schedule+0x2c/0x50
[c0000006b317b5f0] [c0000000003ea624] __lock_page+0x1e4/0x430
[c0000006b317b6d0] [c000000000407fc8] truncate_inode_pages_range+0x338/0x8b0
[c0000006b317b850] [c000000000725714] kill_bdev.isra.14+0x44/0x60
[c0000006b317b880] [c0000000007261f4] blkdev_flush_mapping+0x54/0x260
[c0000006b317b960] [c000000000726488] blkdev_put_whole+0x88/0x90
[c0000006b317b9a0] [c00000000072714c] blkdev_put+0x1cc/0x280
[c0000006b317ba00] [c000000000727e9c] blkdev_close+0x3c/0x60
[c0000006b317ba30] [c000000000525694] __fput+0xc4/0x350
[c0000006b317ba80] [c000000000191128] task_work_run+0xf8/0x170
[c0000006b317bad0] [c000000000161c34] do_exit+0x4a4/0xd30
[c0000006b317bba0] [c000000000162594] do_group_exit+0x64/0xe0
[c0000006b317bbe0] [c000000000177fb8] get_signal+0x258/0xce0
[c0000006b317bcd0] [c0000000000219d4] do_notify_resume+0x114/0x480
[c0000006b317bd80] [c000000000030e40] interrupt_exit_user_prepare_main+0x1a0/0x260
[c0000006b317bde0] [c0000000000312e0] syscall_exit_prepare+0x70/0x150
[c0000006b317be10] [c00000000000c758] system_call_common+0xf8/0x250
--- interrupt: c00 at 0x7fff8c6c8674
NIP:  00007fff8c6c8674 LR: 00007fff8c757430 CTR: 0000000000000000
REGS: c0000006b317be80 TRAP: 0c00   Not tainted  (5.15.0-rc2+)
MSR:  800000000000d033 <SF,EE,PR,ME,IR,DR,RI,LE>  CR: 28002242  XER: 00000000
IRQMASK: 0 
GPR00: 0000000000000003 00007fffe0bff620 00007fff8c6f7f00 0000000000010000 
GPR04: 00007fff89980038 0000000000040000 00007fff8c6a0190 00007fff8c6a01a0 
GPR08: 00007fff8c6a0160 0000000000000000 0000000000000000 0000000000000000 
GPR12: 0000000000000000 00007fff8cc46480 0000000000000000 0000000000bab10c 
GPR16: 0cb1ba0000000000 00007fff8c79b700 0000000000000000 0000000000000002 
GPR20: 0000000000040000 000001002710e800 00000100270680f8 00007fff8c798d28 
GPR24: 00007fff8c7c03a4 00007fff8c7c03a4 00007fff89980028 000001002710e850 
GPR28: 0000000000040000 0000000c7ff80000 00007fff89980010 000001002710e800 
NIP [00007fff8c6c8674] 0x7fff8c6c8674
LR [00007fff8c757430] 0x7fff8c757430
--- interrupt: c00
INFO: task multipathd:881519 blocked for more than 368 seconds.
      Not tainted 5.15.0-rc2+ #1
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
task:multipathd      state:D stack:    0 pid:881519 ppid:     1 flags:0x00040082
Call Trace:
[c000000096eff2b0] [c0000000ae18dd10] 0xc0000000ae18dd10 (unreliable)
[c000000096eff4a0] [c00000000001ea68] __switch_to+0x288/0x4a0
[c000000096eff500] [c000000000e07bfc] __schedule+0x30c/0x9f0
[c000000096eff5c0] [c000000000e08348] schedule+0x68/0x120
[c000000096eff5f0] [c000000000e08930] schedule_preempt_disabled+0x20/0x30
[c000000096eff610] [c000000000e0aedc] __mutex_lock.isra.11+0x36c/0x700
[c000000096eff6a0] [c000000000788e0c] bd_link_disk_holder+0x3c/0x280
[c000000096eff6f0] [c008000000fb5848] dm_get_table_device+0x1f0/0x2d0 [dm_mod]
[c000000096eff790] [c008000000fb9ce8] dm_get_device+0x130/0x2f0 [dm_mod]
[c000000096eff840] [c0080000011553b4] multipath_ctr+0x9cc/0x1000 [dm_multipath]
[c000000096eff9c0] [c008000000fba704] dm_table_add_target+0x1ac/0x420 [dm_mod]
[c000000096effa80] [c008000000fc0a04] table_load+0x16c/0x4c0 [dm_mod]
[c000000096effb30] [c008000000fc3734] ctl_ioctl+0x28c/0x7e0 [dm_mod]
[c000000096effd40] [c008000000fc3ca8] dm_ctl_ioctl+0x20/0x40 [dm_mod]
[c000000096effd60] [c000000000545db8] sys_ioctl+0xf8/0x150
[c000000096effdb0] [c000000000031074] system_call_exception+0x174/0x370
[c000000096effe10] [c00000000000c74c] system_call_common+0xec/0x250
--- interrupt: c00 at 0x7fffb86ac010
NIP:  00007fffb86ac010 LR: 00007fffb8a86924 CTR: 0000000000000000
REGS: c000000096effe80 TRAP: 0c00   Not tainted  (5.15.0-rc2+)
MSR:  800000000000d033 <SF,EE,PR,ME,IR,DR,RI,LE>  CR: 24042204  XER: 00000000
IRQMASK: 0 
GPR00: 0000000000000036 00007fffb7cec3a0 00007fffb8797300 0000000000000005 
GPR04: 00000000c138fd09 00007fffb0069c90 00007fffb8a8a118 00007fffb7cea298 
GPR08: 0000000000000005 0000000000000000 0000000000000000 0000000000000000 
GPR12: 0000000000000000 00007fffb7cf6300 00007fffb0069c90 00007fffb8a89e80 
GPR16: 00007fffb8a89e80 00007fffb8a89e80 00007fffb8ac3670 0000000000000000 
GPR20: 00007fffb8ac2040 00007fffb8a93460 00007fffb0069cc0 000001001f65ab80 
GPR24: 00007fffb8a89e80 00007fffb8a89e80 00007fffb8a89e80 0000000000000000 
GPR28: 00007fffb8a89e80 00007fffb8a89e80 0000000000000000 00007fffb8a89e80 
NIP [00007fffb86ac010] 0x7fffb86ac010
LR [00007fffb8a86924] 0x7fffb8a86924
--- interrupt: c00
INFO: task systemd-udevd:881738 blocked for more than 368 seconds.
      Not tainted 5.15.0-rc2+ #1
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
task:systemd-udevd   state:D stack:    0 pid:881738 ppid:   708 flags:0x00042482
Call Trace:
[c0000006b317b280] [c0000000007640a4] bio_associate_blkg+0x44/0xb0 (unreliable)
[c0000006b317b470] [c00000000001ea68] __switch_to+0x288/0x4a0
[c0000006b317b4d0] [c000000000e07bfc] __schedule+0x30c/0x9f0
[c0000006b317b590] [c000000000e08348] schedule+0x68/0x120
[c0000006b317b5c0] [c000000000e08adc] io_schedule+0x2c/0x50
[c0000006b317b5f0] [c0000000003ea624] __lock_page+0x1e4/0x430
[c0000006b317b6d0] [c000000000407fc8] truncate_inode_pages_range+0x338/0x8b0
[c0000006b317b850] [c000000000725714] kill_bdev.isra.14+0x44/0x60
[c0000006b317b880] [c0000000007261f4] blkdev_flush_mapping+0x54/0x260
[c0000006b317b960] [c000000000726488] blkdev_put_whole+0x88/0x90
[c0000006b317b9a0] [c00000000072714c] blkdev_put+0x1cc/0x280
[c0000006b317ba00] [c000000000727e9c] blkdev_close+0x3c/0x60
[c0000006b317ba30] [c000000000525694] __fput+0xc4/0x350
[c0000006b317ba80] [c000000000191128] task_work_run+0xf8/0x170
[c0000006b317bad0] [c000000000161c34] do_exit+0x4a4/0xd30
[c0000006b317bba0] [c000000000162594] do_group_exit+0x64/0xe0
[c0000006b317bbe0] [c000000000177fb8] get_signal+0x258/0xce0
[c0000006b317bcd0] [c0000000000219d4] do_notify_resume+0x114/0x480
[c0000006b317bd80] [c000000000030e40] interrupt_exit_user_prepare_main+0x1a0/0x260
[c0000006b317bde0] [c0000000000312e0] syscall_exit_prepare+0x70/0x150
[c0000006b317be10] [c00000000000c758] system_call_common+0xf8/0x250
--- interrupt: c00 at 0x7fff8c6c8674
NIP:  00007fff8c6c8674 LR: 00007fff8c757430 CTR: 0000000000000000
REGS: c0000006b317be80 TRAP: 0c00   Not tainted  (5.15.0-rc2+)
MSR:  800000000000d033 <SF,EE,PR,ME,IR,DR,RI,LE>  CR: 28002242  XER: 00000000
IRQMASK: 0 
GPR00: 0000000000000003 00007fffe0bff620 00007fff8c6f7f00 0000000000010000 
GPR04: 00007fff89980038 0000000000040000 00007fff8c6a0190 00007fff8c6a01a0 
GPR08: 00007fff8c6a0160 0000000000000000 0000000000000000 0000000000000000 
GPR12: 0000000000000000 00007fff8cc46480 0000000000000000 0000000000bab10c 
GPR16: 0cb1ba0000000000 00007fff8c79b700 0000000000000000 0000000000000002 
GPR20: 0000000000040000 000001002710e800 00000100270680f8 00007fff8c798d28 
GPR24: 00007fff8c7c03a4 00007fff8c7c03a4 00007fff89980028 000001002710e850 
GPR28: 0000000000040000 0000000c7ff80000 00007fff89980010 000001002710e800 
NIP [00007fff8c6c8674] 0x7fff8c6c8674
LR [00007fff8c757430] 0x7fff8c757430
--- interrupt: c00
INFO: task multipathd:881519 blocked for more than 491 seconds.
      Not tainted 5.15.0-rc2+ #1
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
task:multipathd      state:D stack:    0 pid:881519 ppid:     1 flags:0x00040082
Call Trace:
[c000000096eff2b0] [c0000000ae18dd10] 0xc0000000ae18dd10 (unreliable)
[c000000096eff4a0] [c00000000001ea68] __switch_to+0x288/0x4a0
[c000000096eff500] [c000000000e07bfc] __schedule+0x30c/0x9f0
[c000000096eff5c0] [c000000000e08348] schedule+0x68/0x120
[c000000096eff5f0] [c000000000e08930] schedule_preempt_disabled+0x20/0x30
[c000000096eff610] [c000000000e0aedc] __mutex_lock.isra.11+0x36c/0x700
[c000000096eff6a0] [c000000000788e0c] bd_link_disk_holder+0x3c/0x280
[c000000096eff6f0] [c008000000fb5848] dm_get_table_device+0x1f0/0x2d0 [dm_mod]
[c000000096eff790] [c008000000fb9ce8] dm_get_device+0x130/0x2f0 [dm_mod]
[c000000096eff840] [c0080000011553b4] multipath_ctr+0x9cc/0x1000 [dm_multipath]
[c000000096eff9c0] [c008000000fba704] dm_table_add_target+0x1ac/0x420 [dm_mod]
[c000000096effa80] [c008000000fc0a04] table_load+0x16c/0x4c0 [dm_mod]
[c000000096effb30] [c008000000fc3734] ctl_ioctl+0x28c/0x7e0 [dm_mod]
[c000000096effd40] [c008000000fc3ca8] dm_ctl_ioctl+0x20/0x40 [dm_mod]
[c000000096effd60] [c000000000545db8] sys_ioctl+0xf8/0x150
[c000000096effdb0] [c000000000031074] system_call_exception+0x174/0x370
[c000000096effe10] [c00000000000c74c] system_call_common+0xec/0x250
--- interrupt: c00 at 0x7fffb86ac010
NIP:  00007fffb86ac010 LR: 00007fffb8a86924 CTR: 0000000000000000
REGS: c000000096effe80 TRAP: 0c00   Not tainted  (5.15.0-rc2+)
MSR:  800000000000d033 <SF,EE,PR,ME,IR,DR,RI,LE>  CR: 24042204  XER: 00000000
IRQMASK: 0 
GPR00: 0000000000000036 00007fffb7cec3a0 00007fffb8797300 0000000000000005 
GPR04: 00000000c138fd09 00007fffb0069c90 00007fffb8a8a118 00007fffb7cea298 
GPR08: 0000000000000005 0000000000000000 0000000000000000 0000000000000000 
GPR12: 0000000000000000 00007fffb7cf6300 00007fffb0069c90 00007fffb8a89e80 
GPR16: 00007fffb8a89e80 00007fffb8a89e80 00007fffb8ac3670 0000000000000000 
GPR20: 00007fffb8ac2040 00007fffb8a93460 00007fffb0069cc0 000001001f65ab80 
GPR24: 00007fffb8a89e80 00007fffb8a89e80 00007fffb8a89e80 0000000000000000 
GPR28: 00007fffb8a89e80 00007fffb8a89e80 0000000000000000 00007fffb8a89e80 
NIP [00007fffb86ac010] 0x7fffb86ac010
LR [00007fffb8a86924] 0x7fffb8a86924
--- interrupt: c00
INFO: task systemd-udevd:881738 blocked for more than 491 seconds.
      Not tainted 5.15.0-rc2+ #1
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
task:systemd-udevd   state:D stack:    0 pid:881738 ppid:   708 flags:0x00042482
Call Trace:
[c0000006b317b280] [c0000000007640a4] bio_associate_blkg+0x44/0xb0 (unreliable)
[c0000006b317b470] [c00000000001ea68] __switch_to+0x288/0x4a0
[c0000006b317b4d0] [c000000000e07bfc] __schedule+0x30c/0x9f0
[c0000006b317b590] [c000000000e08348] schedule+0x68/0x120
[c0000006b317b5c0] [c000000000e08adc] io_schedule+0x2c/0x50
[c0000006b317b5f0] [c0000000003ea624] __lock_page+0x1e4/0x430
[c0000006b317b6d0] [c000000000407fc8] truncate_inode_pages_range+0x338/0x8b0
[c0000006b317b850] [c000000000725714] kill_bdev.isra.14+0x44/0x60
[c0000006b317b880] [c0000000007261f4] blkdev_flush_mapping+0x54/0x260
[c0000006b317b960] [c000000000726488] blkdev_put_whole+0x88/0x90
[c0000006b317b9a0] [c00000000072714c] blkdev_put+0x1cc/0x280
[c0000006b317ba00] [c000000000727e9c] blkdev_close+0x3c/0x60
[c0000006b317ba30] [c000000000525694] __fput+0xc4/0x350
[c0000006b317ba80] [c000000000191128] task_work_run+0xf8/0x170
[c0000006b317bad0] [c000000000161c34] do_exit+0x4a4/0xd30
[c0000006b317bba0] [c000000000162594] do_group_exit+0x64/0xe0
[c0000006b317bbe0] [c000000000177fb8] get_signal+0x258/0xce0
[c0000006b317bcd0] [c0000000000219d4] do_notify_resume+0x114/0x480
[c0000006b317bd80] [c000000000030e40] interrupt_exit_user_prepare_main+0x1a0/0x260
[c0000006b317bde0] [c0000000000312e0] syscall_exit_prepare+0x70/0x150
[c0000006b317be10] [c00000000000c758] system_call_common+0xf8/0x250
--- interrupt: c00 at 0x7fff8c6c8674
NIP:  00007fff8c6c8674 LR: 00007fff8c757430 CTR: 0000000000000000
REGS: c0000006b317be80 TRAP: 0c00   Not tainted  (5.15.0-rc2+)
MSR:  800000000000d033 <SF,EE,PR,ME,IR,DR,RI,LE>  CR: 28002242  XER: 00000000
IRQMASK: 0 
GPR00: 0000000000000003 00007fffe0bff620 00007fff8c6f7f00 0000000000010000 
GPR04: 00007fff89980038 0000000000040000 00007fff8c6a0190 00007fff8c6a01a0 
GPR08: 00007fff8c6a0160 0000000000000000 0000000000000000 0000000000000000 
GPR12: 0000000000000000 00007fff8cc46480 0000000000000000 0000000000bab10c 
GPR16: 0cb1ba0000000000 00007fff8c79b700 0000000000000000 0000000000000002 
GPR20: 0000000000040000 000001002710e800 00000100270680f8 00007fff8c798d28 
GPR24: 00007fff8c7c03a4 00007fff8c7c03a4 00007fff89980028 000001002710e850 
GPR28: 0000000000040000 0000000c7ff80000 00007fff89980010 000001002710e800 
NIP [00007fff8c6c8674] 0x7fff8c6c8674
LR [00007fff8c757430] 0x7fff8c757430
--- interrupt: c00
INFO: task multipathd:881519 blocked for more than 614 seconds.
      Not tainted 5.15.0-rc2+ #1
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
task:multipathd      state:D stack:    0 pid:881519 ppid:     1 flags:0x00040082
Call Trace:
[c000000096eff2b0] [c0000000ae18dd10] 0xc0000000ae18dd10 (unreliable)
[c000000096eff4a0] [c00000000001ea68] __switch_to+0x288/0x4a0
[c000000096eff500] [c000000000e07bfc] __schedule+0x30c/0x9f0
[c000000096eff5c0] [c000000000e08348] schedule+0x68/0x120
[c000000096eff5f0] [c000000000e08930] schedule_preempt_disabled+0x20/0x30
[c000000096eff610] [c000000000e0aedc] __mutex_lock.isra.11+0x36c/0x700
[c000000096eff6a0] [c000000000788e0c] bd_link_disk_holder+0x3c/0x280
[c000000096eff6f0] [c008000000fb5848] dm_get_table_device+0x1f0/0x2d0 [dm_mod]
[c000000096eff790] [c008000000fb9ce8] dm_get_device+0x130/0x2f0 [dm_mod]
[c000000096eff840] [c0080000011553b4] multipath_ctr+0x9cc/0x1000 [dm_multipath]
[c000000096eff9c0] [c008000000fba704] dm_table_add_target+0x1ac/0x420 [dm_mod]
[c000000096effa80] [c008000000fc0a04] table_load+0x16c/0x4c0 [dm_mod]
[c000000096effb30] [c008000000fc3734] ctl_ioctl+0x28c/0x7e0 [dm_mod]
[c000000096effd40] [c008000000fc3ca8] dm_ctl_ioctl+0x20/0x40 [dm_mod]
[c000000096effd60] [c000000000545db8] sys_ioctl+0xf8/0x150
[c000000096effdb0] [c000000000031074] system_call_exception+0x174/0x370
[c000000096effe10] [c00000000000c74c] system_call_common+0xec/0x250
--- interrupt: c00 at 0x7fffb86ac010
NIP:  00007fffb86ac010 LR: 00007fffb8a86924 CTR: 0000000000000000
REGS: c000000096effe80 TRAP: 0c00   Not tainted  (5.15.0-rc2+)
MSR:  800000000000d033 <SF,EE,PR,ME,IR,DR,RI,LE>  CR: 24042204  XER: 00000000
IRQMASK: 0 
GPR00: 0000000000000036 00007fffb7cec3a0 00007fffb8797300 0000000000000005 
GPR04: 00000000c138fd09 00007fffb0069c90 00007fffb8a8a118 00007fffb7cea298 
GPR08: 0000000000000005 0000000000000000 0000000000000000 0000000000000000 
GPR12: 0000000000000000 00007fffb7cf6300 00007fffb0069c90 00007fffb8a89e80 
GPR16: 00007fffb8a89e80 00007fffb8a89e80 00007fffb8ac3670 0000000000000000 
GPR20: 00007fffb8ac2040 00007fffb8a93460 00007fffb0069cc0 000001001f65ab80 
GPR24: 00007fffb8a89e80 00007fffb8a89e80 00007fffb8a89e80 0000000000000000 
GPR28: 00007fffb8a89e80 00007fffb8a89e80 0000000000000000 00007fffb8a89e80 
NIP [00007fffb86ac010] 0x7fffb86ac010
LR [00007fffb8a86924] 0x7fffb8a86924
--- interrupt: c00
INFO: task systemd-udevd:881738 blocked for more than 614 seconds.
      Not tainted 5.15.0-rc2+ #1
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
task:systemd-udevd   state:D stack:    0 pid:881738 ppid:   708 flags:0x00042482
Call Trace:
[c0000006b317b280] [c0000000007640a4] bio_associate_blkg+0x44/0xb0 (unreliable)
[c0000006b317b470] [c00000000001ea68] __switch_to+0x288/0x4a0
[c0000006b317b4d0] [c000000000e07bfc] __schedule+0x30c/0x9f0
[c0000006b317b590] [c000000000e08348] schedule+0x68/0x120
[c0000006b317b5c0] [c000000000e08adc] io_schedule+0x2c/0x50
[c0000006b317b5f0] [c0000000003ea624] __lock_page+0x1e4/0x430
[c0000006b317b6d0] [c000000000407fc8] truncate_inode_pages_range+0x338/0x8b0
[c0000006b317b850] [c000000000725714] kill_bdev.isra.14+0x44/0x60
[c0000006b317b880] [c0000000007261f4] blkdev_flush_mapping+0x54/0x260
[c0000006b317b960] [c000000000726488] blkdev_put_whole+0x88/0x90
[c0000006b317b9a0] [c00000000072714c] blkdev_put+0x1cc/0x280
[c0000006b317ba00] [c000000000727e9c] blkdev_close+0x3c/0x60
[c0000006b317ba30] [c000000000525694] __fput+0xc4/0x350
[c0000006b317ba80] [c000000000191128] task_work_run+0xf8/0x170
[c0000006b317bad0] [c000000000161c34] do_exit+0x4a4/0xd30
[c0000006b317bba0] [c000000000162594] do_group_exit+0x64/0xe0
[c0000006b317bbe0] [c000000000177fb8] get_signal+0x258/0xce0
[c0000006b317bcd0] [c0000000000219d4] do_notify_resume+0x114/0x480
[c0000006b317bd80] [c000000000030e40] interrupt_exit_user_prepare_main+0x1a0/0x260
[c0000006b317bde0] [c0000000000312e0] syscall_exit_prepare+0x70/0x150
[c0000006b317be10] [c00000000000c758] system_call_common+0xf8/0x250
--- interrupt: c00 at 0x7fff8c6c8674
NIP:  00007fff8c6c8674 LR: 00007fff8c757430 CTR: 0000000000000000
REGS: c0000006b317be80 TRAP: 0c00   Not tainted  (5.15.0-rc2+)
MSR:  800000000000d033 <SF,EE,PR,ME,IR,DR,RI,LE>  CR: 28002242  XER: 00000000
IRQMASK: 0 
GPR00: 0000000000000003 00007fffe0bff620 00007fff8c6f7f00 0000000000010000 
GPR04: 00007fff89980038 0000000000040000 00007fff8c6a0190 00007fff8c6a01a0 
GPR08: 00007fff8c6a0160 0000000000000000 0000000000000000 0000000000000000 
GPR12: 0000000000000000 00007fff8cc46480 0000000000000000 0000000000bab10c 
GPR16: 0cb1ba0000000000 00007fff8c79b700 0000000000000000 0000000000000002 
GPR20: 0000000000040000 000001002710e800 00000100270680f8 00007fff8c798d28 
GPR24: 00007fff8c7c03a4 00007fff8c7c03a4 00007fff89980028 000001002710e850 
GPR28: 0000000000040000 0000000c7ff80000 00007fff89980010 000001002710e800 
NIP [00007fff8c6c8674] 0x7fff8c6c8674
LR [00007fff8c757430] 0x7fff8c757430
--- interrupt: c00
EXT4-fs (dm-5): error count since last fsck: 3
EXT4-fs (dm-5): initial error at time 1632612344: ext4_map_blocks:593: inode 8: block 1081345
EXT4-fs (dm-5): last error at time 1632628929: ext4_map_blocks:593: inode 2: block 9251
 EXT4-fs error (device dm-5): ext4_map_blocks:593: inode #2: block 9251: comm scp: lblock 0 mapped to illegal pblock 9251 (length 1)
 EXT4-fs (dm-5): error count since last fsck: 4
 EXT4-fs (dm-5): initial error at time 1632612344: ext4_map_blocks:593: inode 8: block 1081345
 EXT4-fs (dm-5): last error at time 1632724166: ext4_map_blocks:593: inode 2: block 9251