* [next-20210827][ppc][multipathd] INFO: task hung in dm_table_add_target
@ 2021-09-01 11:17 ` Abdul Haleem
0 siblings, 0 replies; 12+ messages in thread
From: Abdul Haleem @ 2021-09-01 11:17 UTC (permalink / raw)
To: linux-kernel
Cc: linux-scsi, linux-next, hch, jack, axboe, linuxppc-dev, dm-devel,
sachinp, Brian King, dougmill
Greeting's
multiple task hung while adding the vfc disk back to the multipath on my
powerpc box running linux-next kernel
Test:
$ multipathd -k"remove path sde"
$ multipathd -k"add path sde"
After above command multipathd task hung with continuous call traces on
console, it requires reboot to stop call traces
systemd-udevd[180359]: Process '/sbin/mdadm -I /dev/dm-8' failed with
exit code 1.
multipathd[180272]: mpathf: load table [0 62914560 multipath 1
queue_if_no_path 1 alua 2 1 round-robin 0 2 1 8:64 1 65:0 1 round-robin
0 2 1 8:144 1 65:80 1]
systemd[1]: Started Device-Mapper Multipath Device Controller.
multipathd[180272]: mpatha: sdl - tur checker reports path is up
multipathd[180272]: 8:176: reinstated
multipathd[180272]: mpatha: remaining active paths: 2
multipathd[180272]: sde: remove path (operator)
multipathd[180272]: mpathf: load table [0 62914560 multipath 1
queue_if_no_path 1 alua 2 1 round-robin 0 1 1 65:0 1 round-robin 0 2 1
8:144 1 65:80 1]
multipathd[180272]: sde [8:64]: path removed from map mpathf
multipathd[180272]: sdq: remove path (operator)
multipathd[180272]: mpathf: load table [0 62914560 multipath 1
queue_if_no_path 1 alua 1 1 round-robin 0 2 1 8:144 1 65:80 1]
multipathd[180272]: sdq [65:0]: path removed from map mpathf
multipathd[180272]: sdj: remove path (operator)
multipathd[180272]: mpathf: load table [0 62914560 multipath 1
queue_if_no_path 1 alua 1 1 round-robin 0 1 1 65:80 1]
multipathd[180272]: sdj [8:144]: path removed from map mpathf
multipathd[180272]: sdv: remove path (operator)
multipathd[180272]: mpathf: map in use
multipathd[180272]: mpathf: can't flush
multipathd[180272]: mpathf: load table [0 62914560 multipath 1
queue_if_no_path 0 0 0]
multipathd[180272]: sdv [65:80]: path removed from map mpathf
systemd[1]: Starting system activity accounting tool...
systemd[1]: sysstat-collect.service: Succeeded.
systemd[1]: Started system activity accounting tool.
systemd-udevd[1156]: seq 5678 '/devices/virtual/block/dm-10' is taking a
long time
systemd-udevd[1156]: seq 5678 '/devices/virtual/block/dm-10' killed
multipathd[180272]: sde: add path (operator)
systemd[1]: Stopping Device-Mapper Multipath Device Controller...
systemd[1]: multipathd.service: State 'stop-sigterm' timed out. Killing.
systemd[1]: multipathd.service: Killing process 180272 (multipathd) with
signal SIGKILL.
systemd[1]: multipathd.service: Processes still around after SIGKILL.
Ignoring.
dbus-daemon[1920]: [system] Activating via systemd: service
name='net.reactivated.Fprint' unit='fprintd.service' requested by
':1.255' (uid=0 pid=95173 comm="/bin/login -p -- ")
systemd[1]: Starting Fingerprint Authentication Daemon...
dbus-daemon[1920]: [system] Successfully activated service
'net.reactivated.Fprint'
systemd[1]: Started Fingerprint Authentication Daemon.
systemd-logind[2032]: New session 6 of user root.
systemd[1]: Started Session 6 of user root.
kernel: INFO: task multipathd:180274 blocked for more than 122 seconds.
kernel: Not tainted 5.14.0-rc7-next-20210827-autotest #1
kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this
message.
kernel: task:multipathd state:D stack: 0 pid:180274 ppid: 1
flags:0x00040082
kernel: Call Trace:
kernel: [c00000002aaa72c0] [c00000002aaa73a0] 0xc00000002aaa73a0
(unreliable)
kernel: [c00000002aaa74b0] [c00000000001e638] __switch_to+0x278/0x490
kernel: [c00000002aaa7510] [c000000000c89cfc] __schedule+0x31c/0xa10
kernel: [c00000002aaa75d0] [c000000000c8a468] schedule+0x78/0x130
kernel: [c00000002aaa7600] [c000000000c8aa08]
schedule_preempt_disabled+0x18/0x30
kernel: [c00000002aaa7620] [c000000000c8cf0c]
__mutex_lock.isra.11+0x36c/0x700
kernel: [c00000002aaa76b0] [c00000000067efe4] bd_link_disk_holder+0x34/0x270
kernel: [c00000002aaa7700] [c0080000003e3b78]
dm_get_table_device+0x1e0/0x2c0 [dm_mod]
kernel: [c00000002aaa77a0] [c0080000003e7e48] dm_get_device+0x130/0x2f0
[dm_mod]
kernel: [c00000002aaa7850] [c008000000745234] multipath_ctr+0x9bc/0xff0
[dm_multipath]
kernel: [c00000002aaa79d0] [c0080000003e883c]
dm_table_add_target+0x1a4/0x420 [dm_mod]
kernel: [c00000002aaa7a90] [c0080000003ee874] table_load+0x15c/0x4a0
[dm_mod]
kernel: [c00000002aaa7b40] [c0080000003f1454] ctl_ioctl+0x27c/0x770 [dm_mod]
kernel: [c00000002aaa7d40] [c0080000003f1960] dm_ctl_ioctl+0x18/0x30
[dm_mod]
kernel: [c00000002aaa7d60] [c000000000481198] sys_ioctl+0xf8/0x150
kernel: [c00000002aaa7db0] [c00000000002ff28]
system_call_exception+0x158/0x2c0
kernel: [c00000002aaa7e10] [c00000000000c764] system_call_common+0xf4/0x258
kernel: --- interrupt: c00 at 0x7fffa06f4480
kernel: NIP: 00007fffa06f4480 LR: 00007fffa0ad6714 CTR: 0000000000000000
kernel: REGS: c00000002aaa7e80 TRAP: 0c00 Not tainted
(5.14.0-rc7-next-20210827-autotest)
kernel: MSR: 800000000000d033 <SF,EE,PR,ME,IR,DR,RI,LE> CR: 24042204
XER: 00000000
kernel: IRQMASK: 0 #012GPR00: 0000000000000036 00007fff9fd4c3a0
00007fffa07e7100 0000000000000005 #012GPR04: 00000000c138fd09
00007fff9806a0c0 00007fffa0ad9f18 00007fff9fd4a298 #012GPR08:
0000000000000005 0000000000000000 0000000000000000 0000000000000000
#012GPR12: 0000000000000000 00007fff9fd56300 00007fff9806a0c0
00007fffa0ad9c80 #012GPR16: 00007fffa0ad9c80 00007fffa0ad9c80
00007fffa0b13670 0000000000000000 #012GPR20: 00007fffa0ae3260
00007fffa0b12040 00007fff9806a0f0 00007fff9800eca0 #012GPR24:
00007fffa0ad9c80 00007fffa0ad9c80 00007fffa0ad9c80 0000000000000000
#012GPR28: 00007fffa0ad9c80 00007fffa0ad9c80 0000000000000000
00007fffa0ad9c80
kernel: NIP [00007fffa06f4480] 0x7fffa06f4480
kernel: LR [00007fffa0ad6714] 0x7fffa0ad6714
kernel: --- interrupt: c00
kernel: INFO: task systemd-udevd:180404 blocked for more than 122 seconds.
kernel: Not tainted 5.14.0-rc7-next-20210827-autotest #1
kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this
message.
kernel: task:systemd-udevd state:D stack: 0 pid:180404 ppid: 1156
flags:0x00042482
kernel: Call Trace:
Before test
------------
$multipath -ll
mpathf (360050768108001b3a8000000000000e8) dm-10 IBM,2145
size=30G features='1 queue_if_no_path' hwhandler='1 alua' wp=rw
|-+- policy='round-robin 0' prio=50 status=active
| |- 1:0:0:4 sde 8:64 active ready running
| `- 2:0:0:4 sdq 65:0 active ready running
`-+- policy='round-robin 0' prio=10 status=enabled
|- 1:0:1:4 sdj 8:144 active ready running
`- 2:0:1:4 sdv 65:80 active ready running
$
After test fail
------------
$ multipath -ll
mpathf (360050768108001b3a8000000000000e8) dm-10 ##,##
size=30G features='1 queue_if_no_path' hwhandler='0' wp=rw
$
--
Regard's
Abdul Haleem
IBM Linux Technology Center
^ permalink raw reply [flat|nested] 12+ messages in thread
* [next-20210827][ppc][multipathd] INFO: task hung in dm_table_add_target
@ 2021-09-01 11:17 ` Abdul Haleem
0 siblings, 0 replies; 12+ messages in thread
From: Abdul Haleem @ 2021-09-01 11:17 UTC (permalink / raw)
To: linux-kernel
Cc: axboe, sachinp, jack, linux-scsi, dm-devel, linux-next, dougmill,
Brian King, linuxppc-dev, hch
Greeting's
multiple task hung while adding the vfc disk back to the multipath on my
powerpc box running linux-next kernel
Test:
$ multipathd -k"remove path sde"
$ multipathd -k"add path sde"
After above command multipathd task hung with continuous call traces on
console, it requires reboot to stop call traces
systemd-udevd[180359]: Process '/sbin/mdadm -I /dev/dm-8' failed with
exit code 1.
multipathd[180272]: mpathf: load table [0 62914560 multipath 1
queue_if_no_path 1 alua 2 1 round-robin 0 2 1 8:64 1 65:0 1 round-robin
0 2 1 8:144 1 65:80 1]
systemd[1]: Started Device-Mapper Multipath Device Controller.
multipathd[180272]: mpatha: sdl - tur checker reports path is up
multipathd[180272]: 8:176: reinstated
multipathd[180272]: mpatha: remaining active paths: 2
multipathd[180272]: sde: remove path (operator)
multipathd[180272]: mpathf: load table [0 62914560 multipath 1
queue_if_no_path 1 alua 2 1 round-robin 0 1 1 65:0 1 round-robin 0 2 1
8:144 1 65:80 1]
multipathd[180272]: sde [8:64]: path removed from map mpathf
multipathd[180272]: sdq: remove path (operator)
multipathd[180272]: mpathf: load table [0 62914560 multipath 1
queue_if_no_path 1 alua 1 1 round-robin 0 2 1 8:144 1 65:80 1]
multipathd[180272]: sdq [65:0]: path removed from map mpathf
multipathd[180272]: sdj: remove path (operator)
multipathd[180272]: mpathf: load table [0 62914560 multipath 1
queue_if_no_path 1 alua 1 1 round-robin 0 1 1 65:80 1]
multipathd[180272]: sdj [8:144]: path removed from map mpathf
multipathd[180272]: sdv: remove path (operator)
multipathd[180272]: mpathf: map in use
multipathd[180272]: mpathf: can't flush
multipathd[180272]: mpathf: load table [0 62914560 multipath 1
queue_if_no_path 0 0 0]
multipathd[180272]: sdv [65:80]: path removed from map mpathf
systemd[1]: Starting system activity accounting tool...
systemd[1]: sysstat-collect.service: Succeeded.
systemd[1]: Started system activity accounting tool.
systemd-udevd[1156]: seq 5678 '/devices/virtual/block/dm-10' is taking a
long time
systemd-udevd[1156]: seq 5678 '/devices/virtual/block/dm-10' killed
multipathd[180272]: sde: add path (operator)
systemd[1]: Stopping Device-Mapper Multipath Device Controller...
systemd[1]: multipathd.service: State 'stop-sigterm' timed out. Killing.
systemd[1]: multipathd.service: Killing process 180272 (multipathd) with
signal SIGKILL.
systemd[1]: multipathd.service: Processes still around after SIGKILL.
Ignoring.
dbus-daemon[1920]: [system] Activating via systemd: service
name='net.reactivated.Fprint' unit='fprintd.service' requested by
':1.255' (uid=0 pid=95173 comm="/bin/login -p -- ")
systemd[1]: Starting Fingerprint Authentication Daemon...
dbus-daemon[1920]: [system] Successfully activated service
'net.reactivated.Fprint'
systemd[1]: Started Fingerprint Authentication Daemon.
systemd-logind[2032]: New session 6 of user root.
systemd[1]: Started Session 6 of user root.
kernel: INFO: task multipathd:180274 blocked for more than 122 seconds.
kernel: Not tainted 5.14.0-rc7-next-20210827-autotest #1
kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this
message.
kernel: task:multipathd state:D stack: 0 pid:180274 ppid: 1
flags:0x00040082
kernel: Call Trace:
kernel: [c00000002aaa72c0] [c00000002aaa73a0] 0xc00000002aaa73a0
(unreliable)
kernel: [c00000002aaa74b0] [c00000000001e638] __switch_to+0x278/0x490
kernel: [c00000002aaa7510] [c000000000c89cfc] __schedule+0x31c/0xa10
kernel: [c00000002aaa75d0] [c000000000c8a468] schedule+0x78/0x130
kernel: [c00000002aaa7600] [c000000000c8aa08]
schedule_preempt_disabled+0x18/0x30
kernel: [c00000002aaa7620] [c000000000c8cf0c]
__mutex_lock.isra.11+0x36c/0x700
kernel: [c00000002aaa76b0] [c00000000067efe4] bd_link_disk_holder+0x34/0x270
kernel: [c00000002aaa7700] [c0080000003e3b78]
dm_get_table_device+0x1e0/0x2c0 [dm_mod]
kernel: [c00000002aaa77a0] [c0080000003e7e48] dm_get_device+0x130/0x2f0
[dm_mod]
kernel: [c00000002aaa7850] [c008000000745234] multipath_ctr+0x9bc/0xff0
[dm_multipath]
kernel: [c00000002aaa79d0] [c0080000003e883c]
dm_table_add_target+0x1a4/0x420 [dm_mod]
kernel: [c00000002aaa7a90] [c0080000003ee874] table_load+0x15c/0x4a0
[dm_mod]
kernel: [c00000002aaa7b40] [c0080000003f1454] ctl_ioctl+0x27c/0x770 [dm_mod]
kernel: [c00000002aaa7d40] [c0080000003f1960] dm_ctl_ioctl+0x18/0x30
[dm_mod]
kernel: [c00000002aaa7d60] [c000000000481198] sys_ioctl+0xf8/0x150
kernel: [c00000002aaa7db0] [c00000000002ff28]
system_call_exception+0x158/0x2c0
kernel: [c00000002aaa7e10] [c00000000000c764] system_call_common+0xf4/0x258
kernel: --- interrupt: c00 at 0x7fffa06f4480
kernel: NIP: 00007fffa06f4480 LR: 00007fffa0ad6714 CTR: 0000000000000000
kernel: REGS: c00000002aaa7e80 TRAP: 0c00 Not tainted
(5.14.0-rc7-next-20210827-autotest)
kernel: MSR: 800000000000d033 <SF,EE,PR,ME,IR,DR,RI,LE> CR: 24042204
XER: 00000000
kernel: IRQMASK: 0 #012GPR00: 0000000000000036 00007fff9fd4c3a0
00007fffa07e7100 0000000000000005 #012GPR04: 00000000c138fd09
00007fff9806a0c0 00007fffa0ad9f18 00007fff9fd4a298 #012GPR08:
0000000000000005 0000000000000000 0000000000000000 0000000000000000
#012GPR12: 0000000000000000 00007fff9fd56300 00007fff9806a0c0
00007fffa0ad9c80 #012GPR16: 00007fffa0ad9c80 00007fffa0ad9c80
00007fffa0b13670 0000000000000000 #012GPR20: 00007fffa0ae3260
00007fffa0b12040 00007fff9806a0f0 00007fff9800eca0 #012GPR24:
00007fffa0ad9c80 00007fffa0ad9c80 00007fffa0ad9c80 0000000000000000
#012GPR28: 00007fffa0ad9c80 00007fffa0ad9c80 0000000000000000
00007fffa0ad9c80
kernel: NIP [00007fffa06f4480] 0x7fffa06f4480
kernel: LR [00007fffa0ad6714] 0x7fffa0ad6714
kernel: --- interrupt: c00
kernel: INFO: task systemd-udevd:180404 blocked for more than 122 seconds.
kernel: Not tainted 5.14.0-rc7-next-20210827-autotest #1
kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this
message.
kernel: task:systemd-udevd state:D stack: 0 pid:180404 ppid: 1156
flags:0x00042482
kernel: Call Trace:
Before test
------------
$multipath -ll
mpathf (360050768108001b3a8000000000000e8) dm-10 IBM,2145
size=30G features='1 queue_if_no_path' hwhandler='1 alua' wp=rw
|-+- policy='round-robin 0' prio=50 status=active
| |- 1:0:0:4 sde 8:64 active ready running
| `- 2:0:0:4 sdq 65:0 active ready running
`-+- policy='round-robin 0' prio=10 status=enabled
|- 1:0:1:4 sdj 8:144 active ready running
`- 2:0:1:4 sdv 65:80 active ready running
$
After test fail
------------
$ multipath -ll
mpathf (360050768108001b3a8000000000000e8) dm-10 ##,##
size=30G features='1 queue_if_no_path' hwhandler='0' wp=rw
$
--
Regard's
Abdul Haleem
IBM Linux Technology Center
^ permalink raw reply [flat|nested] 12+ messages in thread
* [dm-devel] [next-20210827][ppc][multipathd] INFO: task hung in dm_table_add_target
@ 2021-09-01 11:17 ` Abdul Haleem
0 siblings, 0 replies; 12+ messages in thread
From: Abdul Haleem @ 2021-09-01 11:17 UTC (permalink / raw)
To: linux-kernel
Cc: axboe, sachinp, jack, linux-scsi, dm-devel, linux-next, dougmill,
Brian King, linuxppc-dev, hch
Greeting's
multiple task hung while adding the vfc disk back to the multipath on my
powerpc box running linux-next kernel
Test:
$ multipathd -k"remove path sde"
$ multipathd -k"add path sde"
After above command multipathd task hung with continuous call traces on
console, it requires reboot to stop call traces
systemd-udevd[180359]: Process '/sbin/mdadm -I /dev/dm-8' failed with
exit code 1.
multipathd[180272]: mpathf: load table [0 62914560 multipath 1
queue_if_no_path 1 alua 2 1 round-robin 0 2 1 8:64 1 65:0 1 round-robin
0 2 1 8:144 1 65:80 1]
systemd[1]: Started Device-Mapper Multipath Device Controller.
multipathd[180272]: mpatha: sdl - tur checker reports path is up
multipathd[180272]: 8:176: reinstated
multipathd[180272]: mpatha: remaining active paths: 2
multipathd[180272]: sde: remove path (operator)
multipathd[180272]: mpathf: load table [0 62914560 multipath 1
queue_if_no_path 1 alua 2 1 round-robin 0 1 1 65:0 1 round-robin 0 2 1
8:144 1 65:80 1]
multipathd[180272]: sde [8:64]: path removed from map mpathf
multipathd[180272]: sdq: remove path (operator)
multipathd[180272]: mpathf: load table [0 62914560 multipath 1
queue_if_no_path 1 alua 1 1 round-robin 0 2 1 8:144 1 65:80 1]
multipathd[180272]: sdq [65:0]: path removed from map mpathf
multipathd[180272]: sdj: remove path (operator)
multipathd[180272]: mpathf: load table [0 62914560 multipath 1
queue_if_no_path 1 alua 1 1 round-robin 0 1 1 65:80 1]
multipathd[180272]: sdj [8:144]: path removed from map mpathf
multipathd[180272]: sdv: remove path (operator)
multipathd[180272]: mpathf: map in use
multipathd[180272]: mpathf: can't flush
multipathd[180272]: mpathf: load table [0 62914560 multipath 1
queue_if_no_path 0 0 0]
multipathd[180272]: sdv [65:80]: path removed from map mpathf
systemd[1]: Starting system activity accounting tool...
systemd[1]: sysstat-collect.service: Succeeded.
systemd[1]: Started system activity accounting tool.
systemd-udevd[1156]: seq 5678 '/devices/virtual/block/dm-10' is taking a
long time
systemd-udevd[1156]: seq 5678 '/devices/virtual/block/dm-10' killed
multipathd[180272]: sde: add path (operator)
systemd[1]: Stopping Device-Mapper Multipath Device Controller...
systemd[1]: multipathd.service: State 'stop-sigterm' timed out. Killing.
systemd[1]: multipathd.service: Killing process 180272 (multipathd) with
signal SIGKILL.
systemd[1]: multipathd.service: Processes still around after SIGKILL.
Ignoring.
dbus-daemon[1920]: [system] Activating via systemd: service
name='net.reactivated.Fprint' unit='fprintd.service' requested by
':1.255' (uid=0 pid=95173 comm="/bin/login -p -- ")
systemd[1]: Starting Fingerprint Authentication Daemon...
dbus-daemon[1920]: [system] Successfully activated service
'net.reactivated.Fprint'
systemd[1]: Started Fingerprint Authentication Daemon.
systemd-logind[2032]: New session 6 of user root.
systemd[1]: Started Session 6 of user root.
kernel: INFO: task multipathd:180274 blocked for more than 122 seconds.
kernel: Not tainted 5.14.0-rc7-next-20210827-autotest #1
kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this
message.
kernel: task:multipathd state:D stack: 0 pid:180274 ppid: 1
flags:0x00040082
kernel: Call Trace:
kernel: [c00000002aaa72c0] [c00000002aaa73a0] 0xc00000002aaa73a0
(unreliable)
kernel: [c00000002aaa74b0] [c00000000001e638] __switch_to+0x278/0x490
kernel: [c00000002aaa7510] [c000000000c89cfc] __schedule+0x31c/0xa10
kernel: [c00000002aaa75d0] [c000000000c8a468] schedule+0x78/0x130
kernel: [c00000002aaa7600] [c000000000c8aa08]
schedule_preempt_disabled+0x18/0x30
kernel: [c00000002aaa7620] [c000000000c8cf0c]
__mutex_lock.isra.11+0x36c/0x700
kernel: [c00000002aaa76b0] [c00000000067efe4] bd_link_disk_holder+0x34/0x270
kernel: [c00000002aaa7700] [c0080000003e3b78]
dm_get_table_device+0x1e0/0x2c0 [dm_mod]
kernel: [c00000002aaa77a0] [c0080000003e7e48] dm_get_device+0x130/0x2f0
[dm_mod]
kernel: [c00000002aaa7850] [c008000000745234] multipath_ctr+0x9bc/0xff0
[dm_multipath]
kernel: [c00000002aaa79d0] [c0080000003e883c]
dm_table_add_target+0x1a4/0x420 [dm_mod]
kernel: [c00000002aaa7a90] [c0080000003ee874] table_load+0x15c/0x4a0
[dm_mod]
kernel: [c00000002aaa7b40] [c0080000003f1454] ctl_ioctl+0x27c/0x770 [dm_mod]
kernel: [c00000002aaa7d40] [c0080000003f1960] dm_ctl_ioctl+0x18/0x30
[dm_mod]
kernel: [c00000002aaa7d60] [c000000000481198] sys_ioctl+0xf8/0x150
kernel: [c00000002aaa7db0] [c00000000002ff28]
system_call_exception+0x158/0x2c0
kernel: [c00000002aaa7e10] [c00000000000c764] system_call_common+0xf4/0x258
kernel: --- interrupt: c00 at 0x7fffa06f4480
kernel: NIP: 00007fffa06f4480 LR: 00007fffa0ad6714 CTR: 0000000000000000
kernel: REGS: c00000002aaa7e80 TRAP: 0c00 Not tainted
(5.14.0-rc7-next-20210827-autotest)
kernel: MSR: 800000000000d033 <SF,EE,PR,ME,IR,DR,RI,LE> CR: 24042204
XER: 00000000
kernel: IRQMASK: 0 #012GPR00: 0000000000000036 00007fff9fd4c3a0
00007fffa07e7100 0000000000000005 #012GPR04: 00000000c138fd09
00007fff9806a0c0 00007fffa0ad9f18 00007fff9fd4a298 #012GPR08:
0000000000000005 0000000000000000 0000000000000000 0000000000000000
#012GPR12: 0000000000000000 00007fff9fd56300 00007fff9806a0c0
00007fffa0ad9c80 #012GPR16: 00007fffa0ad9c80 00007fffa0ad9c80
00007fffa0b13670 0000000000000000 #012GPR20: 00007fffa0ae3260
00007fffa0b12040 00007fff9806a0f0 00007fff9800eca0 #012GPR24:
00007fffa0ad9c80 00007fffa0ad9c80 00007fffa0ad9c80 0000000000000000
#012GPR28: 00007fffa0ad9c80 00007fffa0ad9c80 0000000000000000
00007fffa0ad9c80
kernel: NIP [00007fffa06f4480] 0x7fffa06f4480
kernel: LR [00007fffa0ad6714] 0x7fffa0ad6714
kernel: --- interrupt: c00
kernel: INFO: task systemd-udevd:180404 blocked for more than 122 seconds.
kernel: Not tainted 5.14.0-rc7-next-20210827-autotest #1
kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this
message.
kernel: task:systemd-udevd state:D stack: 0 pid:180404 ppid: 1156
flags:0x00042482
kernel: Call Trace:
Before test
------------
$multipath -ll
mpathf (360050768108001b3a8000000000000e8) dm-10 IBM,2145
size=30G features='1 queue_if_no_path' hwhandler='1 alua' wp=rw
|-+- policy='round-robin 0' prio=50 status=active
| |- 1:0:0:4 sde 8:64 active ready running
| `- 2:0:0:4 sdq 65:0 active ready running
`-+- policy='round-robin 0' prio=10 status=enabled
|- 1:0:1:4 sdj 8:144 active ready running
`- 2:0:1:4 sdv 65:80 active ready running
$
After test fail
------------
$ multipath -ll
mpathf (360050768108001b3a8000000000000e8) dm-10 ##,##
size=30G features='1 queue_if_no_path' hwhandler='0' wp=rw
$
--
Regard's
Abdul Haleem
IBM Linux Technology Center
--
dm-devel mailing list
dm-devel@redhat.com
https://listman.redhat.com/mailman/listinfo/dm-devel
^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: [next-20210827][ppc][multipathd] INFO: task hung in dm_table_add_target
2021-09-01 11:17 ` Abdul Haleem
(?)
@ 2021-09-01 13:36 ` Christoph Hellwig
-1 siblings, 0 replies; 12+ messages in thread
From: Christoph Hellwig @ 2021-09-01 13:36 UTC (permalink / raw)
To: Abdul Haleem
Cc: linux-kernel, linux-scsi, linux-next, hch, jack, axboe,
linuxppc-dev, dm-devel, sachinp, Brian King, dougmill
On Wed, Sep 01, 2021 at 04:47:26PM +0530, Abdul Haleem wrote:
> Greeting's
>
> multiple task hung while adding the vfc disk back to the multipath on my
> powerpc box running linux-next kernel
Can you retry to reproduce this with lockdep enabled to see if there
is anything interesting holding this lock?
^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: [next-20210827][ppc][multipathd] INFO: task hung in dm_table_add_target
@ 2021-09-01 13:36 ` Christoph Hellwig
0 siblings, 0 replies; 12+ messages in thread
From: Christoph Hellwig @ 2021-09-01 13:36 UTC (permalink / raw)
To: Abdul Haleem
Cc: axboe, sachinp, jack, linux-scsi, linux-kernel, dm-devel,
linux-next, dougmill, Brian King, linuxppc-dev, hch
On Wed, Sep 01, 2021 at 04:47:26PM +0530, Abdul Haleem wrote:
> Greeting's
>
> multiple task hung while adding the vfc disk back to the multipath on my
> powerpc box running linux-next kernel
Can you retry to reproduce this with lockdep enabled to see if there
is anything interesting holding this lock?
^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: [dm-devel] [next-20210827][ppc][multipathd] INFO: task hung in dm_table_add_target
@ 2021-09-01 13:36 ` Christoph Hellwig
0 siblings, 0 replies; 12+ messages in thread
From: Christoph Hellwig @ 2021-09-01 13:36 UTC (permalink / raw)
To: Abdul Haleem
Cc: axboe, sachinp, jack, linux-scsi, linux-kernel, dm-devel,
linux-next, dougmill, Brian King, linuxppc-dev, hch
On Wed, Sep 01, 2021 at 04:47:26PM +0530, Abdul Haleem wrote:
> Greeting's
>
> multiple task hung while adding the vfc disk back to the multipath on my
> powerpc box running linux-next kernel
Can you retry to reproduce this with lockdep enabled to see if there
is anything interesting holding this lock?
--
dm-devel mailing list
dm-devel@redhat.com
https://listman.redhat.com/mailman/listinfo/dm-devel
^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: [next-20210827][ppc][multipathd] INFO: task hung in dm_table_add_target
2021-09-01 13:36 ` Christoph Hellwig
(?)
@ 2021-09-28 10:23 ` Abdul Haleem
-1 siblings, 0 replies; 12+ messages in thread
From: Abdul Haleem @ 2021-09-28 10:23 UTC (permalink / raw)
To: Christoph Hellwig
Cc: linux-kernel, linux-scsi, linux-next, jack, axboe, linuxppc-dev,
dm-devel, sachinp, Brian King, dougmill
[-- Attachment #1: Type: text/plain, Size: 586 bytes --]
On 9/1/21 7:06 PM, Christoph Hellwig wrote:
> On Wed, Sep 01, 2021 at 04:47:26PM +0530, Abdul Haleem wrote:
>> Greeting's
>>
>> multiple task hung while adding the vfc disk back to the multipath on my
>> powerpc box running linux-next kernel
> Can you retry to reproduce this with lockdep enabled to see if there
> is anything interesting holding this lock?
LOCKDEP was earlier enabled by default
# cat .config | grep LOCKDEP
CONFIG_LOCKDEP_SUPPORT=y
BTW, Recreated again on 5.15.0-rc2 mainline kernel and attaching the logs
--
Regard's
Abdul Haleem
IBM Linux Technology Center
[-- Attachment #2: hungtasklogs.txt --]
[-- Type: text/plain, Size: 26095 bytes --]
device-mapper: multipath: 253:1: Reinstating path 8:16.
device-mapper: multipath: 253:0: Failing path 8:0.
device-mapper: multipath: 253:0: Failing path 8:32.
device-mapper: multipath: 253:0: Failing path 8:192.
device-mapper: multipath: 253:0: Failing path 8:208.
device-mapper: multipath: 253:0: Reinstating path 8:0.
device-mapper: multipath: 253:0: Reinstating path 8:32.
device-mapper: multipath: 253:0: Reinstating path 8:192.
device-mapper: multipath: 253:0: Reinstating path 8:208.
INFO: task multipathd:881519 blocked for more than 122 seconds.
Not tainted 5.15.0-rc2+ #1
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
task:multipathd state:D stack: 0 pid:881519 ppid: 1 flags:0x00040082
Call Trace:
[c000000096eff2b0] [c0000000ae18dd10] 0xc0000000ae18dd10 (unreliable)
[c000000096eff4a0] [c00000000001ea68] __switch_to+0x288/0x4a0
[c000000096eff500] [c000000000e07bfc] __schedule+0x30c/0x9f0
[c000000096eff5c0] [c000000000e08348] schedule+0x68/0x120
[c000000096eff5f0] [c000000000e08930] schedule_preempt_disabled+0x20/0x30
[c000000096eff610] [c000000000e0aedc] __mutex_lock.isra.11+0x36c/0x700
[c000000096eff6a0] [c000000000788e0c] bd_link_disk_holder+0x3c/0x280
[c000000096eff6f0] [c008000000fb5848] dm_get_table_device+0x1f0/0x2d0 [dm_mod]
[c000000096eff790] [c008000000fb9ce8] dm_get_device+0x130/0x2f0 [dm_mod]
[c000000096eff840] [c0080000011553b4] multipath_ctr+0x9cc/0x1000 [dm_multipath]
[c000000096eff9c0] [c008000000fba704] dm_table_add_target+0x1ac/0x420 [dm_mod]
[c000000096effa80] [c008000000fc0a04] table_load+0x16c/0x4c0 [dm_mod]
[c000000096effb30] [c008000000fc3734] ctl_ioctl+0x28c/0x7e0 [dm_mod]
[c000000096effd40] [c008000000fc3ca8] dm_ctl_ioctl+0x20/0x40 [dm_mod]
[c000000096effd60] [c000000000545db8] sys_ioctl+0xf8/0x150
[c000000096effdb0] [c000000000031074] system_call_exception+0x174/0x370
[c000000096effe10] [c00000000000c74c] system_call_common+0xec/0x250
--- interrupt: c00 at 0x7fffb86ac010
NIP: 00007fffb86ac010 LR: 00007fffb8a86924 CTR: 0000000000000000
REGS: c000000096effe80 TRAP: 0c00 Not tainted (5.15.0-rc2+)
MSR: 800000000000d033 <SF,EE,PR,ME,IR,DR,RI,LE> CR: 24042204 XER: 00000000
IRQMASK: 0
GPR00: 0000000000000036 00007fffb7cec3a0 00007fffb8797300 0000000000000005
GPR04: 00000000c138fd09 00007fffb0069c90 00007fffb8a8a118 00007fffb7cea298
GPR08: 0000000000000005 0000000000000000 0000000000000000 0000000000000000
GPR12: 0000000000000000 00007fffb7cf6300 00007fffb0069c90 00007fffb8a89e80
GPR16: 00007fffb8a89e80 00007fffb8a89e80 00007fffb8ac3670 0000000000000000
GPR20: 00007fffb8ac2040 00007fffb8a93460 00007fffb0069cc0 000001001f65ab80
GPR24: 00007fffb8a89e80 00007fffb8a89e80 00007fffb8a89e80 0000000000000000
GPR28: 00007fffb8a89e80 00007fffb8a89e80 0000000000000000 00007fffb8a89e80
NIP [00007fffb86ac010] 0x7fffb86ac010
LR [00007fffb8a86924] 0x7fffb8a86924
--- interrupt: c00
INFO: task systemd-udevd:881738 blocked for more than 122 seconds.
Not tainted 5.15.0-rc2+ #1
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
task:systemd-udevd state:D stack: 0 pid:881738 ppid: 708 flags:0x00042482
Call Trace:
[c0000006b317b280] [c0000000007640a4] bio_associate_blkg+0x44/0xb0 (unreliable)
[c0000006b317b470] [c00000000001ea68] __switch_to+0x288/0x4a0
[c0000006b317b4d0] [c000000000e07bfc] __schedule+0x30c/0x9f0
[c0000006b317b590] [c000000000e08348] schedule+0x68/0x120
[c0000006b317b5c0] [c000000000e08adc] io_schedule+0x2c/0x50
[c0000006b317b5f0] [c0000000003ea624] __lock_page+0x1e4/0x430
[c0000006b317b6d0] [c000000000407fc8] truncate_inode_pages_range+0x338/0x8b0
[c0000006b317b850] [c000000000725714] kill_bdev.isra.14+0x44/0x60
[c0000006b317b880] [c0000000007261f4] blkdev_flush_mapping+0x54/0x260
[c0000006b317b960] [c000000000726488] blkdev_put_whole+0x88/0x90
[c0000006b317b9a0] [c00000000072714c] blkdev_put+0x1cc/0x280
[c0000006b317ba00] [c000000000727e9c] blkdev_close+0x3c/0x60
[c0000006b317ba30] [c000000000525694] __fput+0xc4/0x350
[c0000006b317ba80] [c000000000191128] task_work_run+0xf8/0x170
[c0000006b317bad0] [c000000000161c34] do_exit+0x4a4/0xd30
[c0000006b317bba0] [c000000000162594] do_group_exit+0x64/0xe0
[c0000006b317bbe0] [c000000000177fb8] get_signal+0x258/0xce0
[c0000006b317bcd0] [c0000000000219d4] do_notify_resume+0x114/0x480
[c0000006b317bd80] [c000000000030e40] interrupt_exit_user_prepare_main+0x1a0/0x260
[c0000006b317bde0] [c0000000000312e0] syscall_exit_prepare+0x70/0x150
[c0000006b317be10] [c00000000000c758] system_call_common+0xf8/0x250
--- interrupt: c00 at 0x7fff8c6c8674
NIP: 00007fff8c6c8674 LR: 00007fff8c757430 CTR: 0000000000000000
REGS: c0000006b317be80 TRAP: 0c00 Not tainted (5.15.0-rc2+)
MSR: 800000000000d033 <SF,EE,PR,ME,IR,DR,RI,LE> CR: 28002242 XER: 00000000
IRQMASK: 0
GPR00: 0000000000000003 00007fffe0bff620 00007fff8c6f7f00 0000000000010000
GPR04: 00007fff89980038 0000000000040000 00007fff8c6a0190 00007fff8c6a01a0
GPR08: 00007fff8c6a0160 0000000000000000 0000000000000000 0000000000000000
GPR12: 0000000000000000 00007fff8cc46480 0000000000000000 0000000000bab10c
GPR16: 0cb1ba0000000000 00007fff8c79b700 0000000000000000 0000000000000002
GPR20: 0000000000040000 000001002710e800 00000100270680f8 00007fff8c798d28
GPR24: 00007fff8c7c03a4 00007fff8c7c03a4 00007fff89980028 000001002710e850
GPR28: 0000000000040000 0000000c7ff80000 00007fff89980010 000001002710e800
NIP [00007fff8c6c8674] 0x7fff8c6c8674
LR [00007fff8c757430] 0x7fff8c757430
--- interrupt: c00
INFO: task multipathd:881519 blocked for more than 245 seconds.
Not tainted 5.15.0-rc2+ #1
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
task:multipathd state:D stack: 0 pid:881519 ppid: 1 flags:0x00040082
Call Trace:
[c000000096eff2b0] [c0000000ae18dd10] 0xc0000000ae18dd10 (unreliable)
[c000000096eff4a0] [c00000000001ea68] __switch_to+0x288/0x4a0
[c000000096eff500] [c000000000e07bfc] __schedule+0x30c/0x9f0
[c000000096eff5c0] [c000000000e08348] schedule+0x68/0x120
[c000000096eff5f0] [c000000000e08930] schedule_preempt_disabled+0x20/0x30
[c000000096eff610] [c000000000e0aedc] __mutex_lock.isra.11+0x36c/0x700
[c000000096eff6a0] [c000000000788e0c] bd_link_disk_holder+0x3c/0x280
[c000000096eff6f0] [c008000000fb5848] dm_get_table_device+0x1f0/0x2d0 [dm_mod]
[c000000096eff790] [c008000000fb9ce8] dm_get_device+0x130/0x2f0 [dm_mod]
[c000000096eff840] [c0080000011553b4] multipath_ctr+0x9cc/0x1000 [dm_multipath]
[c000000096eff9c0] [c008000000fba704] dm_table_add_target+0x1ac/0x420 [dm_mod]
[c000000096effa80] [c008000000fc0a04] table_load+0x16c/0x4c0 [dm_mod]
[c000000096effb30] [c008000000fc3734] ctl_ioctl+0x28c/0x7e0 [dm_mod]
[c000000096effd40] [c008000000fc3ca8] dm_ctl_ioctl+0x20/0x40 [dm_mod]
[c000000096effd60] [c000000000545db8] sys_ioctl+0xf8/0x150
[c000000096effdb0] [c000000000031074] system_call_exception+0x174/0x370
[c000000096effe10] [c00000000000c74c] system_call_common+0xec/0x250
--- interrupt: c00 at 0x7fffb86ac010
NIP: 00007fffb86ac010 LR: 00007fffb8a86924 CTR: 0000000000000000
REGS: c000000096effe80 TRAP: 0c00 Not tainted (5.15.0-rc2+)
MSR: 800000000000d033 <SF,EE,PR,ME,IR,DR,RI,LE> CR: 24042204 XER: 00000000
IRQMASK: 0
GPR00: 0000000000000036 00007fffb7cec3a0 00007fffb8797300 0000000000000005
GPR04: 00000000c138fd09 00007fffb0069c90 00007fffb8a8a118 00007fffb7cea298
GPR08: 0000000000000005 0000000000000000 0000000000000000 0000000000000000
GPR12: 0000000000000000 00007fffb7cf6300 00007fffb0069c90 00007fffb8a89e80
GPR16: 00007fffb8a89e80 00007fffb8a89e80 00007fffb8ac3670 0000000000000000
GPR20: 00007fffb8ac2040 00007fffb8a93460 00007fffb0069cc0 000001001f65ab80
GPR24: 00007fffb8a89e80 00007fffb8a89e80 00007fffb8a89e80 0000000000000000
GPR28: 00007fffb8a89e80 00007fffb8a89e80 0000000000000000 00007fffb8a89e80
NIP [00007fffb86ac010] 0x7fffb86ac010
LR [00007fffb8a86924] 0x7fffb8a86924
--- interrupt: c00
INFO: task systemd-udevd:881738 blocked for more than 245 seconds.
Not tainted 5.15.0-rc2+ #1
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
task:systemd-udevd state:D stack: 0 pid:881738 ppid: 708 flags:0x00042482
Call Trace:
[c0000006b317b280] [c0000000007640a4] bio_associate_blkg+0x44/0xb0 (unreliable)
[c0000006b317b470] [c00000000001ea68] __switch_to+0x288/0x4a0
[c0000006b317b4d0] [c000000000e07bfc] __schedule+0x30c/0x9f0
[c0000006b317b590] [c000000000e08348] schedule+0x68/0x120
[c0000006b317b5c0] [c000000000e08adc] io_schedule+0x2c/0x50
[c0000006b317b5f0] [c0000000003ea624] __lock_page+0x1e4/0x430
[c0000006b317b6d0] [c000000000407fc8] truncate_inode_pages_range+0x338/0x8b0
[c0000006b317b850] [c000000000725714] kill_bdev.isra.14+0x44/0x60
[c0000006b317b880] [c0000000007261f4] blkdev_flush_mapping+0x54/0x260
[c0000006b317b960] [c000000000726488] blkdev_put_whole+0x88/0x90
[c0000006b317b9a0] [c00000000072714c] blkdev_put+0x1cc/0x280
[c0000006b317ba00] [c000000000727e9c] blkdev_close+0x3c/0x60
[c0000006b317ba30] [c000000000525694] __fput+0xc4/0x350
[c0000006b317ba80] [c000000000191128] task_work_run+0xf8/0x170
[c0000006b317bad0] [c000000000161c34] do_exit+0x4a4/0xd30
[c0000006b317bba0] [c000000000162594] do_group_exit+0x64/0xe0
[c0000006b317bbe0] [c000000000177fb8] get_signal+0x258/0xce0
[c0000006b317bcd0] [c0000000000219d4] do_notify_resume+0x114/0x480
[c0000006b317bd80] [c000000000030e40] interrupt_exit_user_prepare_main+0x1a0/0x260
[c0000006b317bde0] [c0000000000312e0] syscall_exit_prepare+0x70/0x150
[c0000006b317be10] [c00000000000c758] system_call_common+0xf8/0x250
--- interrupt: c00 at 0x7fff8c6c8674
NIP: 00007fff8c6c8674 LR: 00007fff8c757430 CTR: 0000000000000000
REGS: c0000006b317be80 TRAP: 0c00 Not tainted (5.15.0-rc2+)
MSR: 800000000000d033 <SF,EE,PR,ME,IR,DR,RI,LE> CR: 28002242 XER: 00000000
IRQMASK: 0
GPR00: 0000000000000003 00007fffe0bff620 00007fff8c6f7f00 0000000000010000
GPR04: 00007fff89980038 0000000000040000 00007fff8c6a0190 00007fff8c6a01a0
GPR08: 00007fff8c6a0160 0000000000000000 0000000000000000 0000000000000000
GPR12: 0000000000000000 00007fff8cc46480 0000000000000000 0000000000bab10c
GPR16: 0cb1ba0000000000 00007fff8c79b700 0000000000000000 0000000000000002
GPR20: 0000000000040000 000001002710e800 00000100270680f8 00007fff8c798d28
GPR24: 00007fff8c7c03a4 00007fff8c7c03a4 00007fff89980028 000001002710e850
GPR28: 0000000000040000 0000000c7ff80000 00007fff89980010 000001002710e800
NIP [00007fff8c6c8674] 0x7fff8c6c8674
LR [00007fff8c757430] 0x7fff8c757430
--- interrupt: c00
INFO: task multipathd:881519 blocked for more than 368 seconds.
Not tainted 5.15.0-rc2+ #1
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
task:multipathd state:D stack: 0 pid:881519 ppid: 1 flags:0x00040082
Call Trace:
[c000000096eff2b0] [c0000000ae18dd10] 0xc0000000ae18dd10 (unreliable)
[c000000096eff4a0] [c00000000001ea68] __switch_to+0x288/0x4a0
[c000000096eff500] [c000000000e07bfc] __schedule+0x30c/0x9f0
[c000000096eff5c0] [c000000000e08348] schedule+0x68/0x120
[c000000096eff5f0] [c000000000e08930] schedule_preempt_disabled+0x20/0x30
[c000000096eff610] [c000000000e0aedc] __mutex_lock.isra.11+0x36c/0x700
[c000000096eff6a0] [c000000000788e0c] bd_link_disk_holder+0x3c/0x280
[c000000096eff6f0] [c008000000fb5848] dm_get_table_device+0x1f0/0x2d0 [dm_mod]
[c000000096eff790] [c008000000fb9ce8] dm_get_device+0x130/0x2f0 [dm_mod]
[c000000096eff840] [c0080000011553b4] multipath_ctr+0x9cc/0x1000 [dm_multipath]
[c000000096eff9c0] [c008000000fba704] dm_table_add_target+0x1ac/0x420 [dm_mod]
[c000000096effa80] [c008000000fc0a04] table_load+0x16c/0x4c0 [dm_mod]
[c000000096effb30] [c008000000fc3734] ctl_ioctl+0x28c/0x7e0 [dm_mod]
[c000000096effd40] [c008000000fc3ca8] dm_ctl_ioctl+0x20/0x40 [dm_mod]
[c000000096effd60] [c000000000545db8] sys_ioctl+0xf8/0x150
[c000000096effdb0] [c000000000031074] system_call_exception+0x174/0x370
[c000000096effe10] [c00000000000c74c] system_call_common+0xec/0x250
--- interrupt: c00 at 0x7fffb86ac010
NIP: 00007fffb86ac010 LR: 00007fffb8a86924 CTR: 0000000000000000
REGS: c000000096effe80 TRAP: 0c00 Not tainted (5.15.0-rc2+)
MSR: 800000000000d033 <SF,EE,PR,ME,IR,DR,RI,LE> CR: 24042204 XER: 00000000
IRQMASK: 0
GPR00: 0000000000000036 00007fffb7cec3a0 00007fffb8797300 0000000000000005
GPR04: 00000000c138fd09 00007fffb0069c90 00007fffb8a8a118 00007fffb7cea298
GPR08: 0000000000000005 0000000000000000 0000000000000000 0000000000000000
GPR12: 0000000000000000 00007fffb7cf6300 00007fffb0069c90 00007fffb8a89e80
GPR16: 00007fffb8a89e80 00007fffb8a89e80 00007fffb8ac3670 0000000000000000
GPR20: 00007fffb8ac2040 00007fffb8a93460 00007fffb0069cc0 000001001f65ab80
GPR24: 00007fffb8a89e80 00007fffb8a89e80 00007fffb8a89e80 0000000000000000
GPR28: 00007fffb8a89e80 00007fffb8a89e80 0000000000000000 00007fffb8a89e80
NIP [00007fffb86ac010] 0x7fffb86ac010
LR [00007fffb8a86924] 0x7fffb8a86924
--- interrupt: c00
INFO: task systemd-udevd:881738 blocked for more than 368 seconds.
Not tainted 5.15.0-rc2+ #1
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
task:systemd-udevd state:D stack: 0 pid:881738 ppid: 708 flags:0x00042482
Call Trace:
[c0000006b317b280] [c0000000007640a4] bio_associate_blkg+0x44/0xb0 (unreliable)
[c0000006b317b470] [c00000000001ea68] __switch_to+0x288/0x4a0
[c0000006b317b4d0] [c000000000e07bfc] __schedule+0x30c/0x9f0
[c0000006b317b590] [c000000000e08348] schedule+0x68/0x120
[c0000006b317b5c0] [c000000000e08adc] io_schedule+0x2c/0x50
[c0000006b317b5f0] [c0000000003ea624] __lock_page+0x1e4/0x430
[c0000006b317b6d0] [c000000000407fc8] truncate_inode_pages_range+0x338/0x8b0
[c0000006b317b850] [c000000000725714] kill_bdev.isra.14+0x44/0x60
[c0000006b317b880] [c0000000007261f4] blkdev_flush_mapping+0x54/0x260
[c0000006b317b960] [c000000000726488] blkdev_put_whole+0x88/0x90
[c0000006b317b9a0] [c00000000072714c] blkdev_put+0x1cc/0x280
[c0000006b317ba00] [c000000000727e9c] blkdev_close+0x3c/0x60
[c0000006b317ba30] [c000000000525694] __fput+0xc4/0x350
[c0000006b317ba80] [c000000000191128] task_work_run+0xf8/0x170
[c0000006b317bad0] [c000000000161c34] do_exit+0x4a4/0xd30
[c0000006b317bba0] [c000000000162594] do_group_exit+0x64/0xe0
[c0000006b317bbe0] [c000000000177fb8] get_signal+0x258/0xce0
[c0000006b317bcd0] [c0000000000219d4] do_notify_resume+0x114/0x480
[c0000006b317bd80] [c000000000030e40] interrupt_exit_user_prepare_main+0x1a0/0x260
[c0000006b317bde0] [c0000000000312e0] syscall_exit_prepare+0x70/0x150
[c0000006b317be10] [c00000000000c758] system_call_common+0xf8/0x250
--- interrupt: c00 at 0x7fff8c6c8674
NIP: 00007fff8c6c8674 LR: 00007fff8c757430 CTR: 0000000000000000
REGS: c0000006b317be80 TRAP: 0c00 Not tainted (5.15.0-rc2+)
MSR: 800000000000d033 <SF,EE,PR,ME,IR,DR,RI,LE> CR: 28002242 XER: 00000000
IRQMASK: 0
GPR00: 0000000000000003 00007fffe0bff620 00007fff8c6f7f00 0000000000010000
GPR04: 00007fff89980038 0000000000040000 00007fff8c6a0190 00007fff8c6a01a0
GPR08: 00007fff8c6a0160 0000000000000000 0000000000000000 0000000000000000
GPR12: 0000000000000000 00007fff8cc46480 0000000000000000 0000000000bab10c
GPR16: 0cb1ba0000000000 00007fff8c79b700 0000000000000000 0000000000000002
GPR20: 0000000000040000 000001002710e800 00000100270680f8 00007fff8c798d28
GPR24: 00007fff8c7c03a4 00007fff8c7c03a4 00007fff89980028 000001002710e850
GPR28: 0000000000040000 0000000c7ff80000 00007fff89980010 000001002710e800
NIP [00007fff8c6c8674] 0x7fff8c6c8674
LR [00007fff8c757430] 0x7fff8c757430
--- interrupt: c00
INFO: task multipathd:881519 blocked for more than 491 seconds.
Not tainted 5.15.0-rc2+ #1
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
task:multipathd state:D stack: 0 pid:881519 ppid: 1 flags:0x00040082
Call Trace:
[c000000096eff2b0] [c0000000ae18dd10] 0xc0000000ae18dd10 (unreliable)
[c000000096eff4a0] [c00000000001ea68] __switch_to+0x288/0x4a0
[c000000096eff500] [c000000000e07bfc] __schedule+0x30c/0x9f0
[c000000096eff5c0] [c000000000e08348] schedule+0x68/0x120
[c000000096eff5f0] [c000000000e08930] schedule_preempt_disabled+0x20/0x30
[c000000096eff610] [c000000000e0aedc] __mutex_lock.isra.11+0x36c/0x700
[c000000096eff6a0] [c000000000788e0c] bd_link_disk_holder+0x3c/0x280
[c000000096eff6f0] [c008000000fb5848] dm_get_table_device+0x1f0/0x2d0 [dm_mod]
[c000000096eff790] [c008000000fb9ce8] dm_get_device+0x130/0x2f0 [dm_mod]
[c000000096eff840] [c0080000011553b4] multipath_ctr+0x9cc/0x1000 [dm_multipath]
[c000000096eff9c0] [c008000000fba704] dm_table_add_target+0x1ac/0x420 [dm_mod]
[c000000096effa80] [c008000000fc0a04] table_load+0x16c/0x4c0 [dm_mod]
[c000000096effb30] [c008000000fc3734] ctl_ioctl+0x28c/0x7e0 [dm_mod]
[c000000096effd40] [c008000000fc3ca8] dm_ctl_ioctl+0x20/0x40 [dm_mod]
[c000000096effd60] [c000000000545db8] sys_ioctl+0xf8/0x150
[c000000096effdb0] [c000000000031074] system_call_exception+0x174/0x370
[c000000096effe10] [c00000000000c74c] system_call_common+0xec/0x250
--- interrupt: c00 at 0x7fffb86ac010
NIP: 00007fffb86ac010 LR: 00007fffb8a86924 CTR: 0000000000000000
REGS: c000000096effe80 TRAP: 0c00 Not tainted (5.15.0-rc2+)
MSR: 800000000000d033 <SF,EE,PR,ME,IR,DR,RI,LE> CR: 24042204 XER: 00000000
IRQMASK: 0
GPR00: 0000000000000036 00007fffb7cec3a0 00007fffb8797300 0000000000000005
GPR04: 00000000c138fd09 00007fffb0069c90 00007fffb8a8a118 00007fffb7cea298
GPR08: 0000000000000005 0000000000000000 0000000000000000 0000000000000000
GPR12: 0000000000000000 00007fffb7cf6300 00007fffb0069c90 00007fffb8a89e80
GPR16: 00007fffb8a89e80 00007fffb8a89e80 00007fffb8ac3670 0000000000000000
GPR20: 00007fffb8ac2040 00007fffb8a93460 00007fffb0069cc0 000001001f65ab80
GPR24: 00007fffb8a89e80 00007fffb8a89e80 00007fffb8a89e80 0000000000000000
GPR28: 00007fffb8a89e80 00007fffb8a89e80 0000000000000000 00007fffb8a89e80
NIP [00007fffb86ac010] 0x7fffb86ac010
LR [00007fffb8a86924] 0x7fffb8a86924
--- interrupt: c00
INFO: task systemd-udevd:881738 blocked for more than 491 seconds.
Not tainted 5.15.0-rc2+ #1
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
task:systemd-udevd state:D stack: 0 pid:881738 ppid: 708 flags:0x00042482
Call Trace:
[c0000006b317b280] [c0000000007640a4] bio_associate_blkg+0x44/0xb0 (unreliable)
[c0000006b317b470] [c00000000001ea68] __switch_to+0x288/0x4a0
[c0000006b317b4d0] [c000000000e07bfc] __schedule+0x30c/0x9f0
[c0000006b317b590] [c000000000e08348] schedule+0x68/0x120
[c0000006b317b5c0] [c000000000e08adc] io_schedule+0x2c/0x50
[c0000006b317b5f0] [c0000000003ea624] __lock_page+0x1e4/0x430
[c0000006b317b6d0] [c000000000407fc8] truncate_inode_pages_range+0x338/0x8b0
[c0000006b317b850] [c000000000725714] kill_bdev.isra.14+0x44/0x60
[c0000006b317b880] [c0000000007261f4] blkdev_flush_mapping+0x54/0x260
[c0000006b317b960] [c000000000726488] blkdev_put_whole+0x88/0x90
[c0000006b317b9a0] [c00000000072714c] blkdev_put+0x1cc/0x280
[c0000006b317ba00] [c000000000727e9c] blkdev_close+0x3c/0x60
[c0000006b317ba30] [c000000000525694] __fput+0xc4/0x350
[c0000006b317ba80] [c000000000191128] task_work_run+0xf8/0x170
[c0000006b317bad0] [c000000000161c34] do_exit+0x4a4/0xd30
[c0000006b317bba0] [c000000000162594] do_group_exit+0x64/0xe0
[c0000006b317bbe0] [c000000000177fb8] get_signal+0x258/0xce0
[c0000006b317bcd0] [c0000000000219d4] do_notify_resume+0x114/0x480
[c0000006b317bd80] [c000000000030e40] interrupt_exit_user_prepare_main+0x1a0/0x260
[c0000006b317bde0] [c0000000000312e0] syscall_exit_prepare+0x70/0x150
[c0000006b317be10] [c00000000000c758] system_call_common+0xf8/0x250
--- interrupt: c00 at 0x7fff8c6c8674
NIP: 00007fff8c6c8674 LR: 00007fff8c757430 CTR: 0000000000000000
REGS: c0000006b317be80 TRAP: 0c00 Not tainted (5.15.0-rc2+)
MSR: 800000000000d033 <SF,EE,PR,ME,IR,DR,RI,LE> CR: 28002242 XER: 00000000
IRQMASK: 0
GPR00: 0000000000000003 00007fffe0bff620 00007fff8c6f7f00 0000000000010000
GPR04: 00007fff89980038 0000000000040000 00007fff8c6a0190 00007fff8c6a01a0
GPR08: 00007fff8c6a0160 0000000000000000 0000000000000000 0000000000000000
GPR12: 0000000000000000 00007fff8cc46480 0000000000000000 0000000000bab10c
GPR16: 0cb1ba0000000000 00007fff8c79b700 0000000000000000 0000000000000002
GPR20: 0000000000040000 000001002710e800 00000100270680f8 00007fff8c798d28
GPR24: 00007fff8c7c03a4 00007fff8c7c03a4 00007fff89980028 000001002710e850
GPR28: 0000000000040000 0000000c7ff80000 00007fff89980010 000001002710e800
NIP [00007fff8c6c8674] 0x7fff8c6c8674
LR [00007fff8c757430] 0x7fff8c757430
--- interrupt: c00
INFO: task multipathd:881519 blocked for more than 614 seconds.
Not tainted 5.15.0-rc2+ #1
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
task:multipathd state:D stack: 0 pid:881519 ppid: 1 flags:0x00040082
Call Trace:
[c000000096eff2b0] [c0000000ae18dd10] 0xc0000000ae18dd10 (unreliable)
[c000000096eff4a0] [c00000000001ea68] __switch_to+0x288/0x4a0
[c000000096eff500] [c000000000e07bfc] __schedule+0x30c/0x9f0
[c000000096eff5c0] [c000000000e08348] schedule+0x68/0x120
[c000000096eff5f0] [c000000000e08930] schedule_preempt_disabled+0x20/0x30
[c000000096eff610] [c000000000e0aedc] __mutex_lock.isra.11+0x36c/0x700
[c000000096eff6a0] [c000000000788e0c] bd_link_disk_holder+0x3c/0x280
[c000000096eff6f0] [c008000000fb5848] dm_get_table_device+0x1f0/0x2d0 [dm_mod]
[c000000096eff790] [c008000000fb9ce8] dm_get_device+0x130/0x2f0 [dm_mod]
[c000000096eff840] [c0080000011553b4] multipath_ctr+0x9cc/0x1000 [dm_multipath]
[c000000096eff9c0] [c008000000fba704] dm_table_add_target+0x1ac/0x420 [dm_mod]
[c000000096effa80] [c008000000fc0a04] table_load+0x16c/0x4c0 [dm_mod]
[c000000096effb30] [c008000000fc3734] ctl_ioctl+0x28c/0x7e0 [dm_mod]
[c000000096effd40] [c008000000fc3ca8] dm_ctl_ioctl+0x20/0x40 [dm_mod]
[c000000096effd60] [c000000000545db8] sys_ioctl+0xf8/0x150
[c000000096effdb0] [c000000000031074] system_call_exception+0x174/0x370
[c000000096effe10] [c00000000000c74c] system_call_common+0xec/0x250
--- interrupt: c00 at 0x7fffb86ac010
NIP: 00007fffb86ac010 LR: 00007fffb8a86924 CTR: 0000000000000000
REGS: c000000096effe80 TRAP: 0c00 Not tainted (5.15.0-rc2+)
MSR: 800000000000d033 <SF,EE,PR,ME,IR,DR,RI,LE> CR: 24042204 XER: 00000000
IRQMASK: 0
GPR00: 0000000000000036 00007fffb7cec3a0 00007fffb8797300 0000000000000005
GPR04: 00000000c138fd09 00007fffb0069c90 00007fffb8a8a118 00007fffb7cea298
GPR08: 0000000000000005 0000000000000000 0000000000000000 0000000000000000
GPR12: 0000000000000000 00007fffb7cf6300 00007fffb0069c90 00007fffb8a89e80
GPR16: 00007fffb8a89e80 00007fffb8a89e80 00007fffb8ac3670 0000000000000000
GPR20: 00007fffb8ac2040 00007fffb8a93460 00007fffb0069cc0 000001001f65ab80
GPR24: 00007fffb8a89e80 00007fffb8a89e80 00007fffb8a89e80 0000000000000000
GPR28: 00007fffb8a89e80 00007fffb8a89e80 0000000000000000 00007fffb8a89e80
NIP [00007fffb86ac010] 0x7fffb86ac010
LR [00007fffb8a86924] 0x7fffb8a86924
--- interrupt: c00
INFO: task systemd-udevd:881738 blocked for more than 614 seconds.
Not tainted 5.15.0-rc2+ #1
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
task:systemd-udevd state:D stack: 0 pid:881738 ppid: 708 flags:0x00042482
Call Trace:
[c0000006b317b280] [c0000000007640a4] bio_associate_blkg+0x44/0xb0 (unreliable)
[c0000006b317b470] [c00000000001ea68] __switch_to+0x288/0x4a0
[c0000006b317b4d0] [c000000000e07bfc] __schedule+0x30c/0x9f0
[c0000006b317b590] [c000000000e08348] schedule+0x68/0x120
[c0000006b317b5c0] [c000000000e08adc] io_schedule+0x2c/0x50
[c0000006b317b5f0] [c0000000003ea624] __lock_page+0x1e4/0x430
[c0000006b317b6d0] [c000000000407fc8] truncate_inode_pages_range+0x338/0x8b0
[c0000006b317b850] [c000000000725714] kill_bdev.isra.14+0x44/0x60
[c0000006b317b880] [c0000000007261f4] blkdev_flush_mapping+0x54/0x260
[c0000006b317b960] [c000000000726488] blkdev_put_whole+0x88/0x90
[c0000006b317b9a0] [c00000000072714c] blkdev_put+0x1cc/0x280
[c0000006b317ba00] [c000000000727e9c] blkdev_close+0x3c/0x60
[c0000006b317ba30] [c000000000525694] __fput+0xc4/0x350
[c0000006b317ba80] [c000000000191128] task_work_run+0xf8/0x170
[c0000006b317bad0] [c000000000161c34] do_exit+0x4a4/0xd30
[c0000006b317bba0] [c000000000162594] do_group_exit+0x64/0xe0
[c0000006b317bbe0] [c000000000177fb8] get_signal+0x258/0xce0
[c0000006b317bcd0] [c0000000000219d4] do_notify_resume+0x114/0x480
[c0000006b317bd80] [c000000000030e40] interrupt_exit_user_prepare_main+0x1a0/0x260
[c0000006b317bde0] [c0000000000312e0] syscall_exit_prepare+0x70/0x150
[c0000006b317be10] [c00000000000c758] system_call_common+0xf8/0x250
--- interrupt: c00 at 0x7fff8c6c8674
NIP: 00007fff8c6c8674 LR: 00007fff8c757430 CTR: 0000000000000000
REGS: c0000006b317be80 TRAP: 0c00 Not tainted (5.15.0-rc2+)
MSR: 800000000000d033 <SF,EE,PR,ME,IR,DR,RI,LE> CR: 28002242 XER: 00000000
IRQMASK: 0
GPR00: 0000000000000003 00007fffe0bff620 00007fff8c6f7f00 0000000000010000
GPR04: 00007fff89980038 0000000000040000 00007fff8c6a0190 00007fff8c6a01a0
GPR08: 00007fff8c6a0160 0000000000000000 0000000000000000 0000000000000000
GPR12: 0000000000000000 00007fff8cc46480 0000000000000000 0000000000bab10c
GPR16: 0cb1ba0000000000 00007fff8c79b700 0000000000000000 0000000000000002
GPR20: 0000000000040000 000001002710e800 00000100270680f8 00007fff8c798d28
GPR24: 00007fff8c7c03a4 00007fff8c7c03a4 00007fff89980028 000001002710e850
GPR28: 0000000000040000 0000000c7ff80000 00007fff89980010 000001002710e800
NIP [00007fff8c6c8674] 0x7fff8c6c8674
LR [00007fff8c757430] 0x7fff8c757430
--- interrupt: c00
EXT4-fs (dm-5): error count since last fsck: 3
EXT4-fs (dm-5): initial error at time 1632612344: ext4_map_blocks:593: inode 8: block 1081345
EXT4-fs (dm-5): last error at time 1632628929: ext4_map_blocks:593: inode 2: block 9251
EXT4-fs error (device dm-5): ext4_map_blocks:593: inode #2: block 9251: comm scp: lblock 0 mapped to illegal pblock 9251 (length 1)
EXT4-fs (dm-5): error count since last fsck: 4
EXT4-fs (dm-5): initial error at time 1632612344: ext4_map_blocks:593: inode 8: block 1081345
EXT4-fs (dm-5): last error at time 1632724166: ext4_map_blocks:593: inode 2: block 9251
^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: [next-20210827][ppc][multipathd] INFO: task hung in dm_table_add_target
@ 2021-09-28 10:23 ` Abdul Haleem
0 siblings, 0 replies; 12+ messages in thread
From: Abdul Haleem @ 2021-09-28 10:23 UTC (permalink / raw)
To: Christoph Hellwig
Cc: axboe, sachinp, jack, linux-scsi, linux-kernel, dm-devel,
linux-next, dougmill, Brian King, linuxppc-dev
[-- Attachment #1: Type: text/plain, Size: 586 bytes --]
On 9/1/21 7:06 PM, Christoph Hellwig wrote:
> On Wed, Sep 01, 2021 at 04:47:26PM +0530, Abdul Haleem wrote:
>> Greeting's
>>
>> multiple task hung while adding the vfc disk back to the multipath on my
>> powerpc box running linux-next kernel
> Can you retry to reproduce this with lockdep enabled to see if there
> is anything interesting holding this lock?
LOCKDEP was earlier enabled by default
# cat .config | grep LOCKDEP
CONFIG_LOCKDEP_SUPPORT=y
BTW, Recreated again on 5.15.0-rc2 mainline kernel and attaching the logs
--
Regard's
Abdul Haleem
IBM Linux Technology Center
[-- Attachment #2: hungtasklogs.txt --]
[-- Type: text/plain, Size: 26095 bytes --]
device-mapper: multipath: 253:1: Reinstating path 8:16.
device-mapper: multipath: 253:0: Failing path 8:0.
device-mapper: multipath: 253:0: Failing path 8:32.
device-mapper: multipath: 253:0: Failing path 8:192.
device-mapper: multipath: 253:0: Failing path 8:208.
device-mapper: multipath: 253:0: Reinstating path 8:0.
device-mapper: multipath: 253:0: Reinstating path 8:32.
device-mapper: multipath: 253:0: Reinstating path 8:192.
device-mapper: multipath: 253:0: Reinstating path 8:208.
INFO: task multipathd:881519 blocked for more than 122 seconds.
Not tainted 5.15.0-rc2+ #1
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
task:multipathd state:D stack: 0 pid:881519 ppid: 1 flags:0x00040082
Call Trace:
[c000000096eff2b0] [c0000000ae18dd10] 0xc0000000ae18dd10 (unreliable)
[c000000096eff4a0] [c00000000001ea68] __switch_to+0x288/0x4a0
[c000000096eff500] [c000000000e07bfc] __schedule+0x30c/0x9f0
[c000000096eff5c0] [c000000000e08348] schedule+0x68/0x120
[c000000096eff5f0] [c000000000e08930] schedule_preempt_disabled+0x20/0x30
[c000000096eff610] [c000000000e0aedc] __mutex_lock.isra.11+0x36c/0x700
[c000000096eff6a0] [c000000000788e0c] bd_link_disk_holder+0x3c/0x280
[c000000096eff6f0] [c008000000fb5848] dm_get_table_device+0x1f0/0x2d0 [dm_mod]
[c000000096eff790] [c008000000fb9ce8] dm_get_device+0x130/0x2f0 [dm_mod]
[c000000096eff840] [c0080000011553b4] multipath_ctr+0x9cc/0x1000 [dm_multipath]
[c000000096eff9c0] [c008000000fba704] dm_table_add_target+0x1ac/0x420 [dm_mod]
[c000000096effa80] [c008000000fc0a04] table_load+0x16c/0x4c0 [dm_mod]
[c000000096effb30] [c008000000fc3734] ctl_ioctl+0x28c/0x7e0 [dm_mod]
[c000000096effd40] [c008000000fc3ca8] dm_ctl_ioctl+0x20/0x40 [dm_mod]
[c000000096effd60] [c000000000545db8] sys_ioctl+0xf8/0x150
[c000000096effdb0] [c000000000031074] system_call_exception+0x174/0x370
[c000000096effe10] [c00000000000c74c] system_call_common+0xec/0x250
--- interrupt: c00 at 0x7fffb86ac010
NIP: 00007fffb86ac010 LR: 00007fffb8a86924 CTR: 0000000000000000
REGS: c000000096effe80 TRAP: 0c00 Not tainted (5.15.0-rc2+)
MSR: 800000000000d033 <SF,EE,PR,ME,IR,DR,RI,LE> CR: 24042204 XER: 00000000
IRQMASK: 0
GPR00: 0000000000000036 00007fffb7cec3a0 00007fffb8797300 0000000000000005
GPR04: 00000000c138fd09 00007fffb0069c90 00007fffb8a8a118 00007fffb7cea298
GPR08: 0000000000000005 0000000000000000 0000000000000000 0000000000000000
GPR12: 0000000000000000 00007fffb7cf6300 00007fffb0069c90 00007fffb8a89e80
GPR16: 00007fffb8a89e80 00007fffb8a89e80 00007fffb8ac3670 0000000000000000
GPR20: 00007fffb8ac2040 00007fffb8a93460 00007fffb0069cc0 000001001f65ab80
GPR24: 00007fffb8a89e80 00007fffb8a89e80 00007fffb8a89e80 0000000000000000
GPR28: 00007fffb8a89e80 00007fffb8a89e80 0000000000000000 00007fffb8a89e80
NIP [00007fffb86ac010] 0x7fffb86ac010
LR [00007fffb8a86924] 0x7fffb8a86924
--- interrupt: c00
INFO: task systemd-udevd:881738 blocked for more than 122 seconds.
Not tainted 5.15.0-rc2+ #1
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
task:systemd-udevd state:D stack: 0 pid:881738 ppid: 708 flags:0x00042482
Call Trace:
[c0000006b317b280] [c0000000007640a4] bio_associate_blkg+0x44/0xb0 (unreliable)
[c0000006b317b470] [c00000000001ea68] __switch_to+0x288/0x4a0
[c0000006b317b4d0] [c000000000e07bfc] __schedule+0x30c/0x9f0
[c0000006b317b590] [c000000000e08348] schedule+0x68/0x120
[c0000006b317b5c0] [c000000000e08adc] io_schedule+0x2c/0x50
[c0000006b317b5f0] [c0000000003ea624] __lock_page+0x1e4/0x430
[c0000006b317b6d0] [c000000000407fc8] truncate_inode_pages_range+0x338/0x8b0
[c0000006b317b850] [c000000000725714] kill_bdev.isra.14+0x44/0x60
[c0000006b317b880] [c0000000007261f4] blkdev_flush_mapping+0x54/0x260
[c0000006b317b960] [c000000000726488] blkdev_put_whole+0x88/0x90
[c0000006b317b9a0] [c00000000072714c] blkdev_put+0x1cc/0x280
[c0000006b317ba00] [c000000000727e9c] blkdev_close+0x3c/0x60
[c0000006b317ba30] [c000000000525694] __fput+0xc4/0x350
[c0000006b317ba80] [c000000000191128] task_work_run+0xf8/0x170
[c0000006b317bad0] [c000000000161c34] do_exit+0x4a4/0xd30
[c0000006b317bba0] [c000000000162594] do_group_exit+0x64/0xe0
[c0000006b317bbe0] [c000000000177fb8] get_signal+0x258/0xce0
[c0000006b317bcd0] [c0000000000219d4] do_notify_resume+0x114/0x480
[c0000006b317bd80] [c000000000030e40] interrupt_exit_user_prepare_main+0x1a0/0x260
[c0000006b317bde0] [c0000000000312e0] syscall_exit_prepare+0x70/0x150
[c0000006b317be10] [c00000000000c758] system_call_common+0xf8/0x250
--- interrupt: c00 at 0x7fff8c6c8674
NIP: 00007fff8c6c8674 LR: 00007fff8c757430 CTR: 0000000000000000
REGS: c0000006b317be80 TRAP: 0c00 Not tainted (5.15.0-rc2+)
MSR: 800000000000d033 <SF,EE,PR,ME,IR,DR,RI,LE> CR: 28002242 XER: 00000000
IRQMASK: 0
GPR00: 0000000000000003 00007fffe0bff620 00007fff8c6f7f00 0000000000010000
GPR04: 00007fff89980038 0000000000040000 00007fff8c6a0190 00007fff8c6a01a0
GPR08: 00007fff8c6a0160 0000000000000000 0000000000000000 0000000000000000
GPR12: 0000000000000000 00007fff8cc46480 0000000000000000 0000000000bab10c
GPR16: 0cb1ba0000000000 00007fff8c79b700 0000000000000000 0000000000000002
GPR20: 0000000000040000 000001002710e800 00000100270680f8 00007fff8c798d28
GPR24: 00007fff8c7c03a4 00007fff8c7c03a4 00007fff89980028 000001002710e850
GPR28: 0000000000040000 0000000c7ff80000 00007fff89980010 000001002710e800
NIP [00007fff8c6c8674] 0x7fff8c6c8674
LR [00007fff8c757430] 0x7fff8c757430
--- interrupt: c00
INFO: task multipathd:881519 blocked for more than 245 seconds.
Not tainted 5.15.0-rc2+ #1
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
task:multipathd state:D stack: 0 pid:881519 ppid: 1 flags:0x00040082
Call Trace:
[c000000096eff2b0] [c0000000ae18dd10] 0xc0000000ae18dd10 (unreliable)
[c000000096eff4a0] [c00000000001ea68] __switch_to+0x288/0x4a0
[c000000096eff500] [c000000000e07bfc] __schedule+0x30c/0x9f0
[c000000096eff5c0] [c000000000e08348] schedule+0x68/0x120
[c000000096eff5f0] [c000000000e08930] schedule_preempt_disabled+0x20/0x30
[c000000096eff610] [c000000000e0aedc] __mutex_lock.isra.11+0x36c/0x700
[c000000096eff6a0] [c000000000788e0c] bd_link_disk_holder+0x3c/0x280
[c000000096eff6f0] [c008000000fb5848] dm_get_table_device+0x1f0/0x2d0 [dm_mod]
[c000000096eff790] [c008000000fb9ce8] dm_get_device+0x130/0x2f0 [dm_mod]
[c000000096eff840] [c0080000011553b4] multipath_ctr+0x9cc/0x1000 [dm_multipath]
[c000000096eff9c0] [c008000000fba704] dm_table_add_target+0x1ac/0x420 [dm_mod]
[c000000096effa80] [c008000000fc0a04] table_load+0x16c/0x4c0 [dm_mod]
[c000000096effb30] [c008000000fc3734] ctl_ioctl+0x28c/0x7e0 [dm_mod]
[c000000096effd40] [c008000000fc3ca8] dm_ctl_ioctl+0x20/0x40 [dm_mod]
[c000000096effd60] [c000000000545db8] sys_ioctl+0xf8/0x150
[c000000096effdb0] [c000000000031074] system_call_exception+0x174/0x370
[c000000096effe10] [c00000000000c74c] system_call_common+0xec/0x250
--- interrupt: c00 at 0x7fffb86ac010
NIP: 00007fffb86ac010 LR: 00007fffb8a86924 CTR: 0000000000000000
REGS: c000000096effe80 TRAP: 0c00 Not tainted (5.15.0-rc2+)
MSR: 800000000000d033 <SF,EE,PR,ME,IR,DR,RI,LE> CR: 24042204 XER: 00000000
IRQMASK: 0
GPR00: 0000000000000036 00007fffb7cec3a0 00007fffb8797300 0000000000000005
GPR04: 00000000c138fd09 00007fffb0069c90 00007fffb8a8a118 00007fffb7cea298
GPR08: 0000000000000005 0000000000000000 0000000000000000 0000000000000000
GPR12: 0000000000000000 00007fffb7cf6300 00007fffb0069c90 00007fffb8a89e80
GPR16: 00007fffb8a89e80 00007fffb8a89e80 00007fffb8ac3670 0000000000000000
GPR20: 00007fffb8ac2040 00007fffb8a93460 00007fffb0069cc0 000001001f65ab80
GPR24: 00007fffb8a89e80 00007fffb8a89e80 00007fffb8a89e80 0000000000000000
GPR28: 00007fffb8a89e80 00007fffb8a89e80 0000000000000000 00007fffb8a89e80
NIP [00007fffb86ac010] 0x7fffb86ac010
LR [00007fffb8a86924] 0x7fffb8a86924
--- interrupt: c00
INFO: task systemd-udevd:881738 blocked for more than 245 seconds.
Not tainted 5.15.0-rc2+ #1
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
task:systemd-udevd state:D stack: 0 pid:881738 ppid: 708 flags:0x00042482
Call Trace:
[c0000006b317b280] [c0000000007640a4] bio_associate_blkg+0x44/0xb0 (unreliable)
[c0000006b317b470] [c00000000001ea68] __switch_to+0x288/0x4a0
[c0000006b317b4d0] [c000000000e07bfc] __schedule+0x30c/0x9f0
[c0000006b317b590] [c000000000e08348] schedule+0x68/0x120
[c0000006b317b5c0] [c000000000e08adc] io_schedule+0x2c/0x50
[c0000006b317b5f0] [c0000000003ea624] __lock_page+0x1e4/0x430
[c0000006b317b6d0] [c000000000407fc8] truncate_inode_pages_range+0x338/0x8b0
[c0000006b317b850] [c000000000725714] kill_bdev.isra.14+0x44/0x60
[c0000006b317b880] [c0000000007261f4] blkdev_flush_mapping+0x54/0x260
[c0000006b317b960] [c000000000726488] blkdev_put_whole+0x88/0x90
[c0000006b317b9a0] [c00000000072714c] blkdev_put+0x1cc/0x280
[c0000006b317ba00] [c000000000727e9c] blkdev_close+0x3c/0x60
[c0000006b317ba30] [c000000000525694] __fput+0xc4/0x350
[c0000006b317ba80] [c000000000191128] task_work_run+0xf8/0x170
[c0000006b317bad0] [c000000000161c34] do_exit+0x4a4/0xd30
[c0000006b317bba0] [c000000000162594] do_group_exit+0x64/0xe0
[c0000006b317bbe0] [c000000000177fb8] get_signal+0x258/0xce0
[c0000006b317bcd0] [c0000000000219d4] do_notify_resume+0x114/0x480
[c0000006b317bd80] [c000000000030e40] interrupt_exit_user_prepare_main+0x1a0/0x260
[c0000006b317bde0] [c0000000000312e0] syscall_exit_prepare+0x70/0x150
[c0000006b317be10] [c00000000000c758] system_call_common+0xf8/0x250
--- interrupt: c00 at 0x7fff8c6c8674
NIP: 00007fff8c6c8674 LR: 00007fff8c757430 CTR: 0000000000000000
REGS: c0000006b317be80 TRAP: 0c00 Not tainted (5.15.0-rc2+)
MSR: 800000000000d033 <SF,EE,PR,ME,IR,DR,RI,LE> CR: 28002242 XER: 00000000
IRQMASK: 0
GPR00: 0000000000000003 00007fffe0bff620 00007fff8c6f7f00 0000000000010000
GPR04: 00007fff89980038 0000000000040000 00007fff8c6a0190 00007fff8c6a01a0
GPR08: 00007fff8c6a0160 0000000000000000 0000000000000000 0000000000000000
GPR12: 0000000000000000 00007fff8cc46480 0000000000000000 0000000000bab10c
GPR16: 0cb1ba0000000000 00007fff8c79b700 0000000000000000 0000000000000002
GPR20: 0000000000040000 000001002710e800 00000100270680f8 00007fff8c798d28
GPR24: 00007fff8c7c03a4 00007fff8c7c03a4 00007fff89980028 000001002710e850
GPR28: 0000000000040000 0000000c7ff80000 00007fff89980010 000001002710e800
NIP [00007fff8c6c8674] 0x7fff8c6c8674
LR [00007fff8c757430] 0x7fff8c757430
--- interrupt: c00
INFO: task multipathd:881519 blocked for more than 368 seconds.
Not tainted 5.15.0-rc2+ #1
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
task:multipathd state:D stack: 0 pid:881519 ppid: 1 flags:0x00040082
Call Trace:
[c000000096eff2b0] [c0000000ae18dd10] 0xc0000000ae18dd10 (unreliable)
[c000000096eff4a0] [c00000000001ea68] __switch_to+0x288/0x4a0
[c000000096eff500] [c000000000e07bfc] __schedule+0x30c/0x9f0
[c000000096eff5c0] [c000000000e08348] schedule+0x68/0x120
[c000000096eff5f0] [c000000000e08930] schedule_preempt_disabled+0x20/0x30
[c000000096eff610] [c000000000e0aedc] __mutex_lock.isra.11+0x36c/0x700
[c000000096eff6a0] [c000000000788e0c] bd_link_disk_holder+0x3c/0x280
[c000000096eff6f0] [c008000000fb5848] dm_get_table_device+0x1f0/0x2d0 [dm_mod]
[c000000096eff790] [c008000000fb9ce8] dm_get_device+0x130/0x2f0 [dm_mod]
[c000000096eff840] [c0080000011553b4] multipath_ctr+0x9cc/0x1000 [dm_multipath]
[c000000096eff9c0] [c008000000fba704] dm_table_add_target+0x1ac/0x420 [dm_mod]
[c000000096effa80] [c008000000fc0a04] table_load+0x16c/0x4c0 [dm_mod]
[c000000096effb30] [c008000000fc3734] ctl_ioctl+0x28c/0x7e0 [dm_mod]
[c000000096effd40] [c008000000fc3ca8] dm_ctl_ioctl+0x20/0x40 [dm_mod]
[c000000096effd60] [c000000000545db8] sys_ioctl+0xf8/0x150
[c000000096effdb0] [c000000000031074] system_call_exception+0x174/0x370
[c000000096effe10] [c00000000000c74c] system_call_common+0xec/0x250
--- interrupt: c00 at 0x7fffb86ac010
NIP: 00007fffb86ac010 LR: 00007fffb8a86924 CTR: 0000000000000000
REGS: c000000096effe80 TRAP: 0c00 Not tainted (5.15.0-rc2+)
MSR: 800000000000d033 <SF,EE,PR,ME,IR,DR,RI,LE> CR: 24042204 XER: 00000000
IRQMASK: 0
GPR00: 0000000000000036 00007fffb7cec3a0 00007fffb8797300 0000000000000005
GPR04: 00000000c138fd09 00007fffb0069c90 00007fffb8a8a118 00007fffb7cea298
GPR08: 0000000000000005 0000000000000000 0000000000000000 0000000000000000
GPR12: 0000000000000000 00007fffb7cf6300 00007fffb0069c90 00007fffb8a89e80
GPR16: 00007fffb8a89e80 00007fffb8a89e80 00007fffb8ac3670 0000000000000000
GPR20: 00007fffb8ac2040 00007fffb8a93460 00007fffb0069cc0 000001001f65ab80
GPR24: 00007fffb8a89e80 00007fffb8a89e80 00007fffb8a89e80 0000000000000000
GPR28: 00007fffb8a89e80 00007fffb8a89e80 0000000000000000 00007fffb8a89e80
NIP [00007fffb86ac010] 0x7fffb86ac010
LR [00007fffb8a86924] 0x7fffb8a86924
--- interrupt: c00
INFO: task systemd-udevd:881738 blocked for more than 368 seconds.
Not tainted 5.15.0-rc2+ #1
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
task:systemd-udevd state:D stack: 0 pid:881738 ppid: 708 flags:0x00042482
Call Trace:
[c0000006b317b280] [c0000000007640a4] bio_associate_blkg+0x44/0xb0 (unreliable)
[c0000006b317b470] [c00000000001ea68] __switch_to+0x288/0x4a0
[c0000006b317b4d0] [c000000000e07bfc] __schedule+0x30c/0x9f0
[c0000006b317b590] [c000000000e08348] schedule+0x68/0x120
[c0000006b317b5c0] [c000000000e08adc] io_schedule+0x2c/0x50
[c0000006b317b5f0] [c0000000003ea624] __lock_page+0x1e4/0x430
[c0000006b317b6d0] [c000000000407fc8] truncate_inode_pages_range+0x338/0x8b0
[c0000006b317b850] [c000000000725714] kill_bdev.isra.14+0x44/0x60
[c0000006b317b880] [c0000000007261f4] blkdev_flush_mapping+0x54/0x260
[c0000006b317b960] [c000000000726488] blkdev_put_whole+0x88/0x90
[c0000006b317b9a0] [c00000000072714c] blkdev_put+0x1cc/0x280
[c0000006b317ba00] [c000000000727e9c] blkdev_close+0x3c/0x60
[c0000006b317ba30] [c000000000525694] __fput+0xc4/0x350
[c0000006b317ba80] [c000000000191128] task_work_run+0xf8/0x170
[c0000006b317bad0] [c000000000161c34] do_exit+0x4a4/0xd30
[c0000006b317bba0] [c000000000162594] do_group_exit+0x64/0xe0
[c0000006b317bbe0] [c000000000177fb8] get_signal+0x258/0xce0
[c0000006b317bcd0] [c0000000000219d4] do_notify_resume+0x114/0x480
[c0000006b317bd80] [c000000000030e40] interrupt_exit_user_prepare_main+0x1a0/0x260
[c0000006b317bde0] [c0000000000312e0] syscall_exit_prepare+0x70/0x150
[c0000006b317be10] [c00000000000c758] system_call_common+0xf8/0x250
--- interrupt: c00 at 0x7fff8c6c8674
NIP: 00007fff8c6c8674 LR: 00007fff8c757430 CTR: 0000000000000000
REGS: c0000006b317be80 TRAP: 0c00 Not tainted (5.15.0-rc2+)
MSR: 800000000000d033 <SF,EE,PR,ME,IR,DR,RI,LE> CR: 28002242 XER: 00000000
IRQMASK: 0
GPR00: 0000000000000003 00007fffe0bff620 00007fff8c6f7f00 0000000000010000
GPR04: 00007fff89980038 0000000000040000 00007fff8c6a0190 00007fff8c6a01a0
GPR08: 00007fff8c6a0160 0000000000000000 0000000000000000 0000000000000000
GPR12: 0000000000000000 00007fff8cc46480 0000000000000000 0000000000bab10c
GPR16: 0cb1ba0000000000 00007fff8c79b700 0000000000000000 0000000000000002
GPR20: 0000000000040000 000001002710e800 00000100270680f8 00007fff8c798d28
GPR24: 00007fff8c7c03a4 00007fff8c7c03a4 00007fff89980028 000001002710e850
GPR28: 0000000000040000 0000000c7ff80000 00007fff89980010 000001002710e800
NIP [00007fff8c6c8674] 0x7fff8c6c8674
LR [00007fff8c757430] 0x7fff8c757430
--- interrupt: c00
INFO: task multipathd:881519 blocked for more than 491 seconds.
Not tainted 5.15.0-rc2+ #1
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
task:multipathd state:D stack: 0 pid:881519 ppid: 1 flags:0x00040082
Call Trace:
[c000000096eff2b0] [c0000000ae18dd10] 0xc0000000ae18dd10 (unreliable)
[c000000096eff4a0] [c00000000001ea68] __switch_to+0x288/0x4a0
[c000000096eff500] [c000000000e07bfc] __schedule+0x30c/0x9f0
[c000000096eff5c0] [c000000000e08348] schedule+0x68/0x120
[c000000096eff5f0] [c000000000e08930] schedule_preempt_disabled+0x20/0x30
[c000000096eff610] [c000000000e0aedc] __mutex_lock.isra.11+0x36c/0x700
[c000000096eff6a0] [c000000000788e0c] bd_link_disk_holder+0x3c/0x280
[c000000096eff6f0] [c008000000fb5848] dm_get_table_device+0x1f0/0x2d0 [dm_mod]
[c000000096eff790] [c008000000fb9ce8] dm_get_device+0x130/0x2f0 [dm_mod]
[c000000096eff840] [c0080000011553b4] multipath_ctr+0x9cc/0x1000 [dm_multipath]
[c000000096eff9c0] [c008000000fba704] dm_table_add_target+0x1ac/0x420 [dm_mod]
[c000000096effa80] [c008000000fc0a04] table_load+0x16c/0x4c0 [dm_mod]
[c000000096effb30] [c008000000fc3734] ctl_ioctl+0x28c/0x7e0 [dm_mod]
[c000000096effd40] [c008000000fc3ca8] dm_ctl_ioctl+0x20/0x40 [dm_mod]
[c000000096effd60] [c000000000545db8] sys_ioctl+0xf8/0x150
[c000000096effdb0] [c000000000031074] system_call_exception+0x174/0x370
[c000000096effe10] [c00000000000c74c] system_call_common+0xec/0x250
--- interrupt: c00 at 0x7fffb86ac010
NIP: 00007fffb86ac010 LR: 00007fffb8a86924 CTR: 0000000000000000
REGS: c000000096effe80 TRAP: 0c00 Not tainted (5.15.0-rc2+)
MSR: 800000000000d033 <SF,EE,PR,ME,IR,DR,RI,LE> CR: 24042204 XER: 00000000
IRQMASK: 0
GPR00: 0000000000000036 00007fffb7cec3a0 00007fffb8797300 0000000000000005
GPR04: 00000000c138fd09 00007fffb0069c90 00007fffb8a8a118 00007fffb7cea298
GPR08: 0000000000000005 0000000000000000 0000000000000000 0000000000000000
GPR12: 0000000000000000 00007fffb7cf6300 00007fffb0069c90 00007fffb8a89e80
GPR16: 00007fffb8a89e80 00007fffb8a89e80 00007fffb8ac3670 0000000000000000
GPR20: 00007fffb8ac2040 00007fffb8a93460 00007fffb0069cc0 000001001f65ab80
GPR24: 00007fffb8a89e80 00007fffb8a89e80 00007fffb8a89e80 0000000000000000
GPR28: 00007fffb8a89e80 00007fffb8a89e80 0000000000000000 00007fffb8a89e80
NIP [00007fffb86ac010] 0x7fffb86ac010
LR [00007fffb8a86924] 0x7fffb8a86924
--- interrupt: c00
INFO: task systemd-udevd:881738 blocked for more than 491 seconds.
Not tainted 5.15.0-rc2+ #1
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
task:systemd-udevd state:D stack: 0 pid:881738 ppid: 708 flags:0x00042482
Call Trace:
[c0000006b317b280] [c0000000007640a4] bio_associate_blkg+0x44/0xb0 (unreliable)
[c0000006b317b470] [c00000000001ea68] __switch_to+0x288/0x4a0
[c0000006b317b4d0] [c000000000e07bfc] __schedule+0x30c/0x9f0
[c0000006b317b590] [c000000000e08348] schedule+0x68/0x120
[c0000006b317b5c0] [c000000000e08adc] io_schedule+0x2c/0x50
[c0000006b317b5f0] [c0000000003ea624] __lock_page+0x1e4/0x430
[c0000006b317b6d0] [c000000000407fc8] truncate_inode_pages_range+0x338/0x8b0
[c0000006b317b850] [c000000000725714] kill_bdev.isra.14+0x44/0x60
[c0000006b317b880] [c0000000007261f4] blkdev_flush_mapping+0x54/0x260
[c0000006b317b960] [c000000000726488] blkdev_put_whole+0x88/0x90
[c0000006b317b9a0] [c00000000072714c] blkdev_put+0x1cc/0x280
[c0000006b317ba00] [c000000000727e9c] blkdev_close+0x3c/0x60
[c0000006b317ba30] [c000000000525694] __fput+0xc4/0x350
[c0000006b317ba80] [c000000000191128] task_work_run+0xf8/0x170
[c0000006b317bad0] [c000000000161c34] do_exit+0x4a4/0xd30
[c0000006b317bba0] [c000000000162594] do_group_exit+0x64/0xe0
[c0000006b317bbe0] [c000000000177fb8] get_signal+0x258/0xce0
[c0000006b317bcd0] [c0000000000219d4] do_notify_resume+0x114/0x480
[c0000006b317bd80] [c000000000030e40] interrupt_exit_user_prepare_main+0x1a0/0x260
[c0000006b317bde0] [c0000000000312e0] syscall_exit_prepare+0x70/0x150
[c0000006b317be10] [c00000000000c758] system_call_common+0xf8/0x250
--- interrupt: c00 at 0x7fff8c6c8674
NIP: 00007fff8c6c8674 LR: 00007fff8c757430 CTR: 0000000000000000
REGS: c0000006b317be80 TRAP: 0c00 Not tainted (5.15.0-rc2+)
MSR: 800000000000d033 <SF,EE,PR,ME,IR,DR,RI,LE> CR: 28002242 XER: 00000000
IRQMASK: 0
GPR00: 0000000000000003 00007fffe0bff620 00007fff8c6f7f00 0000000000010000
GPR04: 00007fff89980038 0000000000040000 00007fff8c6a0190 00007fff8c6a01a0
GPR08: 00007fff8c6a0160 0000000000000000 0000000000000000 0000000000000000
GPR12: 0000000000000000 00007fff8cc46480 0000000000000000 0000000000bab10c
GPR16: 0cb1ba0000000000 00007fff8c79b700 0000000000000000 0000000000000002
GPR20: 0000000000040000 000001002710e800 00000100270680f8 00007fff8c798d28
GPR24: 00007fff8c7c03a4 00007fff8c7c03a4 00007fff89980028 000001002710e850
GPR28: 0000000000040000 0000000c7ff80000 00007fff89980010 000001002710e800
NIP [00007fff8c6c8674] 0x7fff8c6c8674
LR [00007fff8c757430] 0x7fff8c757430
--- interrupt: c00
INFO: task multipathd:881519 blocked for more than 614 seconds.
Not tainted 5.15.0-rc2+ #1
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
task:multipathd state:D stack: 0 pid:881519 ppid: 1 flags:0x00040082
Call Trace:
[c000000096eff2b0] [c0000000ae18dd10] 0xc0000000ae18dd10 (unreliable)
[c000000096eff4a0] [c00000000001ea68] __switch_to+0x288/0x4a0
[c000000096eff500] [c000000000e07bfc] __schedule+0x30c/0x9f0
[c000000096eff5c0] [c000000000e08348] schedule+0x68/0x120
[c000000096eff5f0] [c000000000e08930] schedule_preempt_disabled+0x20/0x30
[c000000096eff610] [c000000000e0aedc] __mutex_lock.isra.11+0x36c/0x700
[c000000096eff6a0] [c000000000788e0c] bd_link_disk_holder+0x3c/0x280
[c000000096eff6f0] [c008000000fb5848] dm_get_table_device+0x1f0/0x2d0 [dm_mod]
[c000000096eff790] [c008000000fb9ce8] dm_get_device+0x130/0x2f0 [dm_mod]
[c000000096eff840] [c0080000011553b4] multipath_ctr+0x9cc/0x1000 [dm_multipath]
[c000000096eff9c0] [c008000000fba704] dm_table_add_target+0x1ac/0x420 [dm_mod]
[c000000096effa80] [c008000000fc0a04] table_load+0x16c/0x4c0 [dm_mod]
[c000000096effb30] [c008000000fc3734] ctl_ioctl+0x28c/0x7e0 [dm_mod]
[c000000096effd40] [c008000000fc3ca8] dm_ctl_ioctl+0x20/0x40 [dm_mod]
[c000000096effd60] [c000000000545db8] sys_ioctl+0xf8/0x150
[c000000096effdb0] [c000000000031074] system_call_exception+0x174/0x370
[c000000096effe10] [c00000000000c74c] system_call_common+0xec/0x250
--- interrupt: c00 at 0x7fffb86ac010
NIP: 00007fffb86ac010 LR: 00007fffb8a86924 CTR: 0000000000000000
REGS: c000000096effe80 TRAP: 0c00 Not tainted (5.15.0-rc2+)
MSR: 800000000000d033 <SF,EE,PR,ME,IR,DR,RI,LE> CR: 24042204 XER: 00000000
IRQMASK: 0
GPR00: 0000000000000036 00007fffb7cec3a0 00007fffb8797300 0000000000000005
GPR04: 00000000c138fd09 00007fffb0069c90 00007fffb8a8a118 00007fffb7cea298
GPR08: 0000000000000005 0000000000000000 0000000000000000 0000000000000000
GPR12: 0000000000000000 00007fffb7cf6300 00007fffb0069c90 00007fffb8a89e80
GPR16: 00007fffb8a89e80 00007fffb8a89e80 00007fffb8ac3670 0000000000000000
GPR20: 00007fffb8ac2040 00007fffb8a93460 00007fffb0069cc0 000001001f65ab80
GPR24: 00007fffb8a89e80 00007fffb8a89e80 00007fffb8a89e80 0000000000000000
GPR28: 00007fffb8a89e80 00007fffb8a89e80 0000000000000000 00007fffb8a89e80
NIP [00007fffb86ac010] 0x7fffb86ac010
LR [00007fffb8a86924] 0x7fffb8a86924
--- interrupt: c00
INFO: task systemd-udevd:881738 blocked for more than 614 seconds.
Not tainted 5.15.0-rc2+ #1
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
task:systemd-udevd state:D stack: 0 pid:881738 ppid: 708 flags:0x00042482
Call Trace:
[c0000006b317b280] [c0000000007640a4] bio_associate_blkg+0x44/0xb0 (unreliable)
[c0000006b317b470] [c00000000001ea68] __switch_to+0x288/0x4a0
[c0000006b317b4d0] [c000000000e07bfc] __schedule+0x30c/0x9f0
[c0000006b317b590] [c000000000e08348] schedule+0x68/0x120
[c0000006b317b5c0] [c000000000e08adc] io_schedule+0x2c/0x50
[c0000006b317b5f0] [c0000000003ea624] __lock_page+0x1e4/0x430
[c0000006b317b6d0] [c000000000407fc8] truncate_inode_pages_range+0x338/0x8b0
[c0000006b317b850] [c000000000725714] kill_bdev.isra.14+0x44/0x60
[c0000006b317b880] [c0000000007261f4] blkdev_flush_mapping+0x54/0x260
[c0000006b317b960] [c000000000726488] blkdev_put_whole+0x88/0x90
[c0000006b317b9a0] [c00000000072714c] blkdev_put+0x1cc/0x280
[c0000006b317ba00] [c000000000727e9c] blkdev_close+0x3c/0x60
[c0000006b317ba30] [c000000000525694] __fput+0xc4/0x350
[c0000006b317ba80] [c000000000191128] task_work_run+0xf8/0x170
[c0000006b317bad0] [c000000000161c34] do_exit+0x4a4/0xd30
[c0000006b317bba0] [c000000000162594] do_group_exit+0x64/0xe0
[c0000006b317bbe0] [c000000000177fb8] get_signal+0x258/0xce0
[c0000006b317bcd0] [c0000000000219d4] do_notify_resume+0x114/0x480
[c0000006b317bd80] [c000000000030e40] interrupt_exit_user_prepare_main+0x1a0/0x260
[c0000006b317bde0] [c0000000000312e0] syscall_exit_prepare+0x70/0x150
[c0000006b317be10] [c00000000000c758] system_call_common+0xf8/0x250
--- interrupt: c00 at 0x7fff8c6c8674
NIP: 00007fff8c6c8674 LR: 00007fff8c757430 CTR: 0000000000000000
REGS: c0000006b317be80 TRAP: 0c00 Not tainted (5.15.0-rc2+)
MSR: 800000000000d033 <SF,EE,PR,ME,IR,DR,RI,LE> CR: 28002242 XER: 00000000
IRQMASK: 0
GPR00: 0000000000000003 00007fffe0bff620 00007fff8c6f7f00 0000000000010000
GPR04: 00007fff89980038 0000000000040000 00007fff8c6a0190 00007fff8c6a01a0
GPR08: 00007fff8c6a0160 0000000000000000 0000000000000000 0000000000000000
GPR12: 0000000000000000 00007fff8cc46480 0000000000000000 0000000000bab10c
GPR16: 0cb1ba0000000000 00007fff8c79b700 0000000000000000 0000000000000002
GPR20: 0000000000040000 000001002710e800 00000100270680f8 00007fff8c798d28
GPR24: 00007fff8c7c03a4 00007fff8c7c03a4 00007fff89980028 000001002710e850
GPR28: 0000000000040000 0000000c7ff80000 00007fff89980010 000001002710e800
NIP [00007fff8c6c8674] 0x7fff8c6c8674
LR [00007fff8c757430] 0x7fff8c757430
--- interrupt: c00
EXT4-fs (dm-5): error count since last fsck: 3
EXT4-fs (dm-5): initial error at time 1632612344: ext4_map_blocks:593: inode 8: block 1081345
EXT4-fs (dm-5): last error at time 1632628929: ext4_map_blocks:593: inode 2: block 9251
EXT4-fs error (device dm-5): ext4_map_blocks:593: inode #2: block 9251: comm scp: lblock 0 mapped to illegal pblock 9251 (length 1)
EXT4-fs (dm-5): error count since last fsck: 4
EXT4-fs (dm-5): initial error at time 1632612344: ext4_map_blocks:593: inode 8: block 1081345
EXT4-fs (dm-5): last error at time 1632724166: ext4_map_blocks:593: inode 2: block 9251
^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: [dm-devel] [next-20210827][ppc][multipathd] INFO: task hung in dm_table_add_target
@ 2021-09-28 10:23 ` Abdul Haleem
0 siblings, 0 replies; 12+ messages in thread
From: Abdul Haleem @ 2021-09-28 10:23 UTC (permalink / raw)
To: Christoph Hellwig
Cc: axboe, sachinp, jack, linux-scsi, linux-kernel, dm-devel,
linux-next, dougmill, Brian King, linuxppc-dev
[-- Attachment #1: Type: text/plain, Size: 586 bytes --]
On 9/1/21 7:06 PM, Christoph Hellwig wrote:
> On Wed, Sep 01, 2021 at 04:47:26PM +0530, Abdul Haleem wrote:
>> Greeting's
>>
>> multiple task hung while adding the vfc disk back to the multipath on my
>> powerpc box running linux-next kernel
> Can you retry to reproduce this with lockdep enabled to see if there
> is anything interesting holding this lock?
LOCKDEP was earlier enabled by default
# cat .config | grep LOCKDEP
CONFIG_LOCKDEP_SUPPORT=y
BTW, Recreated again on 5.15.0-rc2 mainline kernel and attaching the logs
--
Regard's
Abdul Haleem
IBM Linux Technology Center
[-- Attachment #2: hungtasklogs.txt --]
[-- Type: text/plain, Size: 26095 bytes --]
device-mapper: multipath: 253:1: Reinstating path 8:16.
device-mapper: multipath: 253:0: Failing path 8:0.
device-mapper: multipath: 253:0: Failing path 8:32.
device-mapper: multipath: 253:0: Failing path 8:192.
device-mapper: multipath: 253:0: Failing path 8:208.
device-mapper: multipath: 253:0: Reinstating path 8:0.
device-mapper: multipath: 253:0: Reinstating path 8:32.
device-mapper: multipath: 253:0: Reinstating path 8:192.
device-mapper: multipath: 253:0: Reinstating path 8:208.
INFO: task multipathd:881519 blocked for more than 122 seconds.
Not tainted 5.15.0-rc2+ #1
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
task:multipathd state:D stack: 0 pid:881519 ppid: 1 flags:0x00040082
Call Trace:
[c000000096eff2b0] [c0000000ae18dd10] 0xc0000000ae18dd10 (unreliable)
[c000000096eff4a0] [c00000000001ea68] __switch_to+0x288/0x4a0
[c000000096eff500] [c000000000e07bfc] __schedule+0x30c/0x9f0
[c000000096eff5c0] [c000000000e08348] schedule+0x68/0x120
[c000000096eff5f0] [c000000000e08930] schedule_preempt_disabled+0x20/0x30
[c000000096eff610] [c000000000e0aedc] __mutex_lock.isra.11+0x36c/0x700
[c000000096eff6a0] [c000000000788e0c] bd_link_disk_holder+0x3c/0x280
[c000000096eff6f0] [c008000000fb5848] dm_get_table_device+0x1f0/0x2d0 [dm_mod]
[c000000096eff790] [c008000000fb9ce8] dm_get_device+0x130/0x2f0 [dm_mod]
[c000000096eff840] [c0080000011553b4] multipath_ctr+0x9cc/0x1000 [dm_multipath]
[c000000096eff9c0] [c008000000fba704] dm_table_add_target+0x1ac/0x420 [dm_mod]
[c000000096effa80] [c008000000fc0a04] table_load+0x16c/0x4c0 [dm_mod]
[c000000096effb30] [c008000000fc3734] ctl_ioctl+0x28c/0x7e0 [dm_mod]
[c000000096effd40] [c008000000fc3ca8] dm_ctl_ioctl+0x20/0x40 [dm_mod]
[c000000096effd60] [c000000000545db8] sys_ioctl+0xf8/0x150
[c000000096effdb0] [c000000000031074] system_call_exception+0x174/0x370
[c000000096effe10] [c00000000000c74c] system_call_common+0xec/0x250
--- interrupt: c00 at 0x7fffb86ac010
NIP: 00007fffb86ac010 LR: 00007fffb8a86924 CTR: 0000000000000000
REGS: c000000096effe80 TRAP: 0c00 Not tainted (5.15.0-rc2+)
MSR: 800000000000d033 <SF,EE,PR,ME,IR,DR,RI,LE> CR: 24042204 XER: 00000000
IRQMASK: 0
GPR00: 0000000000000036 00007fffb7cec3a0 00007fffb8797300 0000000000000005
GPR04: 00000000c138fd09 00007fffb0069c90 00007fffb8a8a118 00007fffb7cea298
GPR08: 0000000000000005 0000000000000000 0000000000000000 0000000000000000
GPR12: 0000000000000000 00007fffb7cf6300 00007fffb0069c90 00007fffb8a89e80
GPR16: 00007fffb8a89e80 00007fffb8a89e80 00007fffb8ac3670 0000000000000000
GPR20: 00007fffb8ac2040 00007fffb8a93460 00007fffb0069cc0 000001001f65ab80
GPR24: 00007fffb8a89e80 00007fffb8a89e80 00007fffb8a89e80 0000000000000000
GPR28: 00007fffb8a89e80 00007fffb8a89e80 0000000000000000 00007fffb8a89e80
NIP [00007fffb86ac010] 0x7fffb86ac010
LR [00007fffb8a86924] 0x7fffb8a86924
--- interrupt: c00
INFO: task systemd-udevd:881738 blocked for more than 122 seconds.
Not tainted 5.15.0-rc2+ #1
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
task:systemd-udevd state:D stack: 0 pid:881738 ppid: 708 flags:0x00042482
Call Trace:
[c0000006b317b280] [c0000000007640a4] bio_associate_blkg+0x44/0xb0 (unreliable)
[c0000006b317b470] [c00000000001ea68] __switch_to+0x288/0x4a0
[c0000006b317b4d0] [c000000000e07bfc] __schedule+0x30c/0x9f0
[c0000006b317b590] [c000000000e08348] schedule+0x68/0x120
[c0000006b317b5c0] [c000000000e08adc] io_schedule+0x2c/0x50
[c0000006b317b5f0] [c0000000003ea624] __lock_page+0x1e4/0x430
[c0000006b317b6d0] [c000000000407fc8] truncate_inode_pages_range+0x338/0x8b0
[c0000006b317b850] [c000000000725714] kill_bdev.isra.14+0x44/0x60
[c0000006b317b880] [c0000000007261f4] blkdev_flush_mapping+0x54/0x260
[c0000006b317b960] [c000000000726488] blkdev_put_whole+0x88/0x90
[c0000006b317b9a0] [c00000000072714c] blkdev_put+0x1cc/0x280
[c0000006b317ba00] [c000000000727e9c] blkdev_close+0x3c/0x60
[c0000006b317ba30] [c000000000525694] __fput+0xc4/0x350
[c0000006b317ba80] [c000000000191128] task_work_run+0xf8/0x170
[c0000006b317bad0] [c000000000161c34] do_exit+0x4a4/0xd30
[c0000006b317bba0] [c000000000162594] do_group_exit+0x64/0xe0
[c0000006b317bbe0] [c000000000177fb8] get_signal+0x258/0xce0
[c0000006b317bcd0] [c0000000000219d4] do_notify_resume+0x114/0x480
[c0000006b317bd80] [c000000000030e40] interrupt_exit_user_prepare_main+0x1a0/0x260
[c0000006b317bde0] [c0000000000312e0] syscall_exit_prepare+0x70/0x150
[c0000006b317be10] [c00000000000c758] system_call_common+0xf8/0x250
--- interrupt: c00 at 0x7fff8c6c8674
NIP: 00007fff8c6c8674 LR: 00007fff8c757430 CTR: 0000000000000000
REGS: c0000006b317be80 TRAP: 0c00 Not tainted (5.15.0-rc2+)
MSR: 800000000000d033 <SF,EE,PR,ME,IR,DR,RI,LE> CR: 28002242 XER: 00000000
IRQMASK: 0
GPR00: 0000000000000003 00007fffe0bff620 00007fff8c6f7f00 0000000000010000
GPR04: 00007fff89980038 0000000000040000 00007fff8c6a0190 00007fff8c6a01a0
GPR08: 00007fff8c6a0160 0000000000000000 0000000000000000 0000000000000000
GPR12: 0000000000000000 00007fff8cc46480 0000000000000000 0000000000bab10c
GPR16: 0cb1ba0000000000 00007fff8c79b700 0000000000000000 0000000000000002
GPR20: 0000000000040000 000001002710e800 00000100270680f8 00007fff8c798d28
GPR24: 00007fff8c7c03a4 00007fff8c7c03a4 00007fff89980028 000001002710e850
GPR28: 0000000000040000 0000000c7ff80000 00007fff89980010 000001002710e800
NIP [00007fff8c6c8674] 0x7fff8c6c8674
LR [00007fff8c757430] 0x7fff8c757430
--- interrupt: c00
INFO: task multipathd:881519 blocked for more than 245 seconds.
Not tainted 5.15.0-rc2+ #1
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
task:multipathd state:D stack: 0 pid:881519 ppid: 1 flags:0x00040082
Call Trace:
[c000000096eff2b0] [c0000000ae18dd10] 0xc0000000ae18dd10 (unreliable)
[c000000096eff4a0] [c00000000001ea68] __switch_to+0x288/0x4a0
[c000000096eff500] [c000000000e07bfc] __schedule+0x30c/0x9f0
[c000000096eff5c0] [c000000000e08348] schedule+0x68/0x120
[c000000096eff5f0] [c000000000e08930] schedule_preempt_disabled+0x20/0x30
[c000000096eff610] [c000000000e0aedc] __mutex_lock.isra.11+0x36c/0x700
[c000000096eff6a0] [c000000000788e0c] bd_link_disk_holder+0x3c/0x280
[c000000096eff6f0] [c008000000fb5848] dm_get_table_device+0x1f0/0x2d0 [dm_mod]
[c000000096eff790] [c008000000fb9ce8] dm_get_device+0x130/0x2f0 [dm_mod]
[c000000096eff840] [c0080000011553b4] multipath_ctr+0x9cc/0x1000 [dm_multipath]
[c000000096eff9c0] [c008000000fba704] dm_table_add_target+0x1ac/0x420 [dm_mod]
[c000000096effa80] [c008000000fc0a04] table_load+0x16c/0x4c0 [dm_mod]
[c000000096effb30] [c008000000fc3734] ctl_ioctl+0x28c/0x7e0 [dm_mod]
[c000000096effd40] [c008000000fc3ca8] dm_ctl_ioctl+0x20/0x40 [dm_mod]
[c000000096effd60] [c000000000545db8] sys_ioctl+0xf8/0x150
[c000000096effdb0] [c000000000031074] system_call_exception+0x174/0x370
[c000000096effe10] [c00000000000c74c] system_call_common+0xec/0x250
--- interrupt: c00 at 0x7fffb86ac010
NIP: 00007fffb86ac010 LR: 00007fffb8a86924 CTR: 0000000000000000
REGS: c000000096effe80 TRAP: 0c00 Not tainted (5.15.0-rc2+)
MSR: 800000000000d033 <SF,EE,PR,ME,IR,DR,RI,LE> CR: 24042204 XER: 00000000
IRQMASK: 0
GPR00: 0000000000000036 00007fffb7cec3a0 00007fffb8797300 0000000000000005
GPR04: 00000000c138fd09 00007fffb0069c90 00007fffb8a8a118 00007fffb7cea298
GPR08: 0000000000000005 0000000000000000 0000000000000000 0000000000000000
GPR12: 0000000000000000 00007fffb7cf6300 00007fffb0069c90 00007fffb8a89e80
GPR16: 00007fffb8a89e80 00007fffb8a89e80 00007fffb8ac3670 0000000000000000
GPR20: 00007fffb8ac2040 00007fffb8a93460 00007fffb0069cc0 000001001f65ab80
GPR24: 00007fffb8a89e80 00007fffb8a89e80 00007fffb8a89e80 0000000000000000
GPR28: 00007fffb8a89e80 00007fffb8a89e80 0000000000000000 00007fffb8a89e80
NIP [00007fffb86ac010] 0x7fffb86ac010
LR [00007fffb8a86924] 0x7fffb8a86924
--- interrupt: c00
INFO: task systemd-udevd:881738 blocked for more than 245 seconds.
Not tainted 5.15.0-rc2+ #1
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
task:systemd-udevd state:D stack: 0 pid:881738 ppid: 708 flags:0x00042482
Call Trace:
[c0000006b317b280] [c0000000007640a4] bio_associate_blkg+0x44/0xb0 (unreliable)
[c0000006b317b470] [c00000000001ea68] __switch_to+0x288/0x4a0
[c0000006b317b4d0] [c000000000e07bfc] __schedule+0x30c/0x9f0
[c0000006b317b590] [c000000000e08348] schedule+0x68/0x120
[c0000006b317b5c0] [c000000000e08adc] io_schedule+0x2c/0x50
[c0000006b317b5f0] [c0000000003ea624] __lock_page+0x1e4/0x430
[c0000006b317b6d0] [c000000000407fc8] truncate_inode_pages_range+0x338/0x8b0
[c0000006b317b850] [c000000000725714] kill_bdev.isra.14+0x44/0x60
[c0000006b317b880] [c0000000007261f4] blkdev_flush_mapping+0x54/0x260
[c0000006b317b960] [c000000000726488] blkdev_put_whole+0x88/0x90
[c0000006b317b9a0] [c00000000072714c] blkdev_put+0x1cc/0x280
[c0000006b317ba00] [c000000000727e9c] blkdev_close+0x3c/0x60
[c0000006b317ba30] [c000000000525694] __fput+0xc4/0x350
[c0000006b317ba80] [c000000000191128] task_work_run+0xf8/0x170
[c0000006b317bad0] [c000000000161c34] do_exit+0x4a4/0xd30
[c0000006b317bba0] [c000000000162594] do_group_exit+0x64/0xe0
[c0000006b317bbe0] [c000000000177fb8] get_signal+0x258/0xce0
[c0000006b317bcd0] [c0000000000219d4] do_notify_resume+0x114/0x480
[c0000006b317bd80] [c000000000030e40] interrupt_exit_user_prepare_main+0x1a0/0x260
[c0000006b317bde0] [c0000000000312e0] syscall_exit_prepare+0x70/0x150
[c0000006b317be10] [c00000000000c758] system_call_common+0xf8/0x250
--- interrupt: c00 at 0x7fff8c6c8674
NIP: 00007fff8c6c8674 LR: 00007fff8c757430 CTR: 0000000000000000
REGS: c0000006b317be80 TRAP: 0c00 Not tainted (5.15.0-rc2+)
MSR: 800000000000d033 <SF,EE,PR,ME,IR,DR,RI,LE> CR: 28002242 XER: 00000000
IRQMASK: 0
GPR00: 0000000000000003 00007fffe0bff620 00007fff8c6f7f00 0000000000010000
GPR04: 00007fff89980038 0000000000040000 00007fff8c6a0190 00007fff8c6a01a0
GPR08: 00007fff8c6a0160 0000000000000000 0000000000000000 0000000000000000
GPR12: 0000000000000000 00007fff8cc46480 0000000000000000 0000000000bab10c
GPR16: 0cb1ba0000000000 00007fff8c79b700 0000000000000000 0000000000000002
GPR20: 0000000000040000 000001002710e800 00000100270680f8 00007fff8c798d28
GPR24: 00007fff8c7c03a4 00007fff8c7c03a4 00007fff89980028 000001002710e850
GPR28: 0000000000040000 0000000c7ff80000 00007fff89980010 000001002710e800
NIP [00007fff8c6c8674] 0x7fff8c6c8674
LR [00007fff8c757430] 0x7fff8c757430
--- interrupt: c00
INFO: task multipathd:881519 blocked for more than 368 seconds.
Not tainted 5.15.0-rc2+ #1
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
task:multipathd state:D stack: 0 pid:881519 ppid: 1 flags:0x00040082
Call Trace:
[c000000096eff2b0] [c0000000ae18dd10] 0xc0000000ae18dd10 (unreliable)
[c000000096eff4a0] [c00000000001ea68] __switch_to+0x288/0x4a0
[c000000096eff500] [c000000000e07bfc] __schedule+0x30c/0x9f0
[c000000096eff5c0] [c000000000e08348] schedule+0x68/0x120
[c000000096eff5f0] [c000000000e08930] schedule_preempt_disabled+0x20/0x30
[c000000096eff610] [c000000000e0aedc] __mutex_lock.isra.11+0x36c/0x700
[c000000096eff6a0] [c000000000788e0c] bd_link_disk_holder+0x3c/0x280
[c000000096eff6f0] [c008000000fb5848] dm_get_table_device+0x1f0/0x2d0 [dm_mod]
[c000000096eff790] [c008000000fb9ce8] dm_get_device+0x130/0x2f0 [dm_mod]
[c000000096eff840] [c0080000011553b4] multipath_ctr+0x9cc/0x1000 [dm_multipath]
[c000000096eff9c0] [c008000000fba704] dm_table_add_target+0x1ac/0x420 [dm_mod]
[c000000096effa80] [c008000000fc0a04] table_load+0x16c/0x4c0 [dm_mod]
[c000000096effb30] [c008000000fc3734] ctl_ioctl+0x28c/0x7e0 [dm_mod]
[c000000096effd40] [c008000000fc3ca8] dm_ctl_ioctl+0x20/0x40 [dm_mod]
[c000000096effd60] [c000000000545db8] sys_ioctl+0xf8/0x150
[c000000096effdb0] [c000000000031074] system_call_exception+0x174/0x370
[c000000096effe10] [c00000000000c74c] system_call_common+0xec/0x250
--- interrupt: c00 at 0x7fffb86ac010
NIP: 00007fffb86ac010 LR: 00007fffb8a86924 CTR: 0000000000000000
REGS: c000000096effe80 TRAP: 0c00 Not tainted (5.15.0-rc2+)
MSR: 800000000000d033 <SF,EE,PR,ME,IR,DR,RI,LE> CR: 24042204 XER: 00000000
IRQMASK: 0
GPR00: 0000000000000036 00007fffb7cec3a0 00007fffb8797300 0000000000000005
GPR04: 00000000c138fd09 00007fffb0069c90 00007fffb8a8a118 00007fffb7cea298
GPR08: 0000000000000005 0000000000000000 0000000000000000 0000000000000000
GPR12: 0000000000000000 00007fffb7cf6300 00007fffb0069c90 00007fffb8a89e80
GPR16: 00007fffb8a89e80 00007fffb8a89e80 00007fffb8ac3670 0000000000000000
GPR20: 00007fffb8ac2040 00007fffb8a93460 00007fffb0069cc0 000001001f65ab80
GPR24: 00007fffb8a89e80 00007fffb8a89e80 00007fffb8a89e80 0000000000000000
GPR28: 00007fffb8a89e80 00007fffb8a89e80 0000000000000000 00007fffb8a89e80
NIP [00007fffb86ac010] 0x7fffb86ac010
LR [00007fffb8a86924] 0x7fffb8a86924
--- interrupt: c00
INFO: task systemd-udevd:881738 blocked for more than 368 seconds.
Not tainted 5.15.0-rc2+ #1
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
task:systemd-udevd state:D stack: 0 pid:881738 ppid: 708 flags:0x00042482
Call Trace:
[c0000006b317b280] [c0000000007640a4] bio_associate_blkg+0x44/0xb0 (unreliable)
[c0000006b317b470] [c00000000001ea68] __switch_to+0x288/0x4a0
[c0000006b317b4d0] [c000000000e07bfc] __schedule+0x30c/0x9f0
[c0000006b317b590] [c000000000e08348] schedule+0x68/0x120
[c0000006b317b5c0] [c000000000e08adc] io_schedule+0x2c/0x50
[c0000006b317b5f0] [c0000000003ea624] __lock_page+0x1e4/0x430
[c0000006b317b6d0] [c000000000407fc8] truncate_inode_pages_range+0x338/0x8b0
[c0000006b317b850] [c000000000725714] kill_bdev.isra.14+0x44/0x60
[c0000006b317b880] [c0000000007261f4] blkdev_flush_mapping+0x54/0x260
[c0000006b317b960] [c000000000726488] blkdev_put_whole+0x88/0x90
[c0000006b317b9a0] [c00000000072714c] blkdev_put+0x1cc/0x280
[c0000006b317ba00] [c000000000727e9c] blkdev_close+0x3c/0x60
[c0000006b317ba30] [c000000000525694] __fput+0xc4/0x350
[c0000006b317ba80] [c000000000191128] task_work_run+0xf8/0x170
[c0000006b317bad0] [c000000000161c34] do_exit+0x4a4/0xd30
[c0000006b317bba0] [c000000000162594] do_group_exit+0x64/0xe0
[c0000006b317bbe0] [c000000000177fb8] get_signal+0x258/0xce0
[c0000006b317bcd0] [c0000000000219d4] do_notify_resume+0x114/0x480
[c0000006b317bd80] [c000000000030e40] interrupt_exit_user_prepare_main+0x1a0/0x260
[c0000006b317bde0] [c0000000000312e0] syscall_exit_prepare+0x70/0x150
[c0000006b317be10] [c00000000000c758] system_call_common+0xf8/0x250
--- interrupt: c00 at 0x7fff8c6c8674
NIP: 00007fff8c6c8674 LR: 00007fff8c757430 CTR: 0000000000000000
REGS: c0000006b317be80 TRAP: 0c00 Not tainted (5.15.0-rc2+)
MSR: 800000000000d033 <SF,EE,PR,ME,IR,DR,RI,LE> CR: 28002242 XER: 00000000
IRQMASK: 0
GPR00: 0000000000000003 00007fffe0bff620 00007fff8c6f7f00 0000000000010000
GPR04: 00007fff89980038 0000000000040000 00007fff8c6a0190 00007fff8c6a01a0
GPR08: 00007fff8c6a0160 0000000000000000 0000000000000000 0000000000000000
GPR12: 0000000000000000 00007fff8cc46480 0000000000000000 0000000000bab10c
GPR16: 0cb1ba0000000000 00007fff8c79b700 0000000000000000 0000000000000002
GPR20: 0000000000040000 000001002710e800 00000100270680f8 00007fff8c798d28
GPR24: 00007fff8c7c03a4 00007fff8c7c03a4 00007fff89980028 000001002710e850
GPR28: 0000000000040000 0000000c7ff80000 00007fff89980010 000001002710e800
NIP [00007fff8c6c8674] 0x7fff8c6c8674
LR [00007fff8c757430] 0x7fff8c757430
--- interrupt: c00
INFO: task multipathd:881519 blocked for more than 491 seconds.
Not tainted 5.15.0-rc2+ #1
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
task:multipathd state:D stack: 0 pid:881519 ppid: 1 flags:0x00040082
Call Trace:
[c000000096eff2b0] [c0000000ae18dd10] 0xc0000000ae18dd10 (unreliable)
[c000000096eff4a0] [c00000000001ea68] __switch_to+0x288/0x4a0
[c000000096eff500] [c000000000e07bfc] __schedule+0x30c/0x9f0
[c000000096eff5c0] [c000000000e08348] schedule+0x68/0x120
[c000000096eff5f0] [c000000000e08930] schedule_preempt_disabled+0x20/0x30
[c000000096eff610] [c000000000e0aedc] __mutex_lock.isra.11+0x36c/0x700
[c000000096eff6a0] [c000000000788e0c] bd_link_disk_holder+0x3c/0x280
[c000000096eff6f0] [c008000000fb5848] dm_get_table_device+0x1f0/0x2d0 [dm_mod]
[c000000096eff790] [c008000000fb9ce8] dm_get_device+0x130/0x2f0 [dm_mod]
[c000000096eff840] [c0080000011553b4] multipath_ctr+0x9cc/0x1000 [dm_multipath]
[c000000096eff9c0] [c008000000fba704] dm_table_add_target+0x1ac/0x420 [dm_mod]
[c000000096effa80] [c008000000fc0a04] table_load+0x16c/0x4c0 [dm_mod]
[c000000096effb30] [c008000000fc3734] ctl_ioctl+0x28c/0x7e0 [dm_mod]
[c000000096effd40] [c008000000fc3ca8] dm_ctl_ioctl+0x20/0x40 [dm_mod]
[c000000096effd60] [c000000000545db8] sys_ioctl+0xf8/0x150
[c000000096effdb0] [c000000000031074] system_call_exception+0x174/0x370
[c000000096effe10] [c00000000000c74c] system_call_common+0xec/0x250
--- interrupt: c00 at 0x7fffb86ac010
NIP: 00007fffb86ac010 LR: 00007fffb8a86924 CTR: 0000000000000000
REGS: c000000096effe80 TRAP: 0c00 Not tainted (5.15.0-rc2+)
MSR: 800000000000d033 <SF,EE,PR,ME,IR,DR,RI,LE> CR: 24042204 XER: 00000000
IRQMASK: 0
GPR00: 0000000000000036 00007fffb7cec3a0 00007fffb8797300 0000000000000005
GPR04: 00000000c138fd09 00007fffb0069c90 00007fffb8a8a118 00007fffb7cea298
GPR08: 0000000000000005 0000000000000000 0000000000000000 0000000000000000
GPR12: 0000000000000000 00007fffb7cf6300 00007fffb0069c90 00007fffb8a89e80
GPR16: 00007fffb8a89e80 00007fffb8a89e80 00007fffb8ac3670 0000000000000000
GPR20: 00007fffb8ac2040 00007fffb8a93460 00007fffb0069cc0 000001001f65ab80
GPR24: 00007fffb8a89e80 00007fffb8a89e80 00007fffb8a89e80 0000000000000000
GPR28: 00007fffb8a89e80 00007fffb8a89e80 0000000000000000 00007fffb8a89e80
NIP [00007fffb86ac010] 0x7fffb86ac010
LR [00007fffb8a86924] 0x7fffb8a86924
--- interrupt: c00
INFO: task systemd-udevd:881738 blocked for more than 491 seconds.
Not tainted 5.15.0-rc2+ #1
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
task:systemd-udevd state:D stack: 0 pid:881738 ppid: 708 flags:0x00042482
Call Trace:
[c0000006b317b280] [c0000000007640a4] bio_associate_blkg+0x44/0xb0 (unreliable)
[c0000006b317b470] [c00000000001ea68] __switch_to+0x288/0x4a0
[c0000006b317b4d0] [c000000000e07bfc] __schedule+0x30c/0x9f0
[c0000006b317b590] [c000000000e08348] schedule+0x68/0x120
[c0000006b317b5c0] [c000000000e08adc] io_schedule+0x2c/0x50
[c0000006b317b5f0] [c0000000003ea624] __lock_page+0x1e4/0x430
[c0000006b317b6d0] [c000000000407fc8] truncate_inode_pages_range+0x338/0x8b0
[c0000006b317b850] [c000000000725714] kill_bdev.isra.14+0x44/0x60
[c0000006b317b880] [c0000000007261f4] blkdev_flush_mapping+0x54/0x260
[c0000006b317b960] [c000000000726488] blkdev_put_whole+0x88/0x90
[c0000006b317b9a0] [c00000000072714c] blkdev_put+0x1cc/0x280
[c0000006b317ba00] [c000000000727e9c] blkdev_close+0x3c/0x60
[c0000006b317ba30] [c000000000525694] __fput+0xc4/0x350
[c0000006b317ba80] [c000000000191128] task_work_run+0xf8/0x170
[c0000006b317bad0] [c000000000161c34] do_exit+0x4a4/0xd30
[c0000006b317bba0] [c000000000162594] do_group_exit+0x64/0xe0
[c0000006b317bbe0] [c000000000177fb8] get_signal+0x258/0xce0
[c0000006b317bcd0] [c0000000000219d4] do_notify_resume+0x114/0x480
[c0000006b317bd80] [c000000000030e40] interrupt_exit_user_prepare_main+0x1a0/0x260
[c0000006b317bde0] [c0000000000312e0] syscall_exit_prepare+0x70/0x150
[c0000006b317be10] [c00000000000c758] system_call_common+0xf8/0x250
--- interrupt: c00 at 0x7fff8c6c8674
NIP: 00007fff8c6c8674 LR: 00007fff8c757430 CTR: 0000000000000000
REGS: c0000006b317be80 TRAP: 0c00 Not tainted (5.15.0-rc2+)
MSR: 800000000000d033 <SF,EE,PR,ME,IR,DR,RI,LE> CR: 28002242 XER: 00000000
IRQMASK: 0
GPR00: 0000000000000003 00007fffe0bff620 00007fff8c6f7f00 0000000000010000
GPR04: 00007fff89980038 0000000000040000 00007fff8c6a0190 00007fff8c6a01a0
GPR08: 00007fff8c6a0160 0000000000000000 0000000000000000 0000000000000000
GPR12: 0000000000000000 00007fff8cc46480 0000000000000000 0000000000bab10c
GPR16: 0cb1ba0000000000 00007fff8c79b700 0000000000000000 0000000000000002
GPR20: 0000000000040000 000001002710e800 00000100270680f8 00007fff8c798d28
GPR24: 00007fff8c7c03a4 00007fff8c7c03a4 00007fff89980028 000001002710e850
GPR28: 0000000000040000 0000000c7ff80000 00007fff89980010 000001002710e800
NIP [00007fff8c6c8674] 0x7fff8c6c8674
LR [00007fff8c757430] 0x7fff8c757430
--- interrupt: c00
INFO: task multipathd:881519 blocked for more than 614 seconds.
Not tainted 5.15.0-rc2+ #1
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
task:multipathd state:D stack: 0 pid:881519 ppid: 1 flags:0x00040082
Call Trace:
[c000000096eff2b0] [c0000000ae18dd10] 0xc0000000ae18dd10 (unreliable)
[c000000096eff4a0] [c00000000001ea68] __switch_to+0x288/0x4a0
[c000000096eff500] [c000000000e07bfc] __schedule+0x30c/0x9f0
[c000000096eff5c0] [c000000000e08348] schedule+0x68/0x120
[c000000096eff5f0] [c000000000e08930] schedule_preempt_disabled+0x20/0x30
[c000000096eff610] [c000000000e0aedc] __mutex_lock.isra.11+0x36c/0x700
[c000000096eff6a0] [c000000000788e0c] bd_link_disk_holder+0x3c/0x280
[c000000096eff6f0] [c008000000fb5848] dm_get_table_device+0x1f0/0x2d0 [dm_mod]
[c000000096eff790] [c008000000fb9ce8] dm_get_device+0x130/0x2f0 [dm_mod]
[c000000096eff840] [c0080000011553b4] multipath_ctr+0x9cc/0x1000 [dm_multipath]
[c000000096eff9c0] [c008000000fba704] dm_table_add_target+0x1ac/0x420 [dm_mod]
[c000000096effa80] [c008000000fc0a04] table_load+0x16c/0x4c0 [dm_mod]
[c000000096effb30] [c008000000fc3734] ctl_ioctl+0x28c/0x7e0 [dm_mod]
[c000000096effd40] [c008000000fc3ca8] dm_ctl_ioctl+0x20/0x40 [dm_mod]
[c000000096effd60] [c000000000545db8] sys_ioctl+0xf8/0x150
[c000000096effdb0] [c000000000031074] system_call_exception+0x174/0x370
[c000000096effe10] [c00000000000c74c] system_call_common+0xec/0x250
--- interrupt: c00 at 0x7fffb86ac010
NIP: 00007fffb86ac010 LR: 00007fffb8a86924 CTR: 0000000000000000
REGS: c000000096effe80 TRAP: 0c00 Not tainted (5.15.0-rc2+)
MSR: 800000000000d033 <SF,EE,PR,ME,IR,DR,RI,LE> CR: 24042204 XER: 00000000
IRQMASK: 0
GPR00: 0000000000000036 00007fffb7cec3a0 00007fffb8797300 0000000000000005
GPR04: 00000000c138fd09 00007fffb0069c90 00007fffb8a8a118 00007fffb7cea298
GPR08: 0000000000000005 0000000000000000 0000000000000000 0000000000000000
GPR12: 0000000000000000 00007fffb7cf6300 00007fffb0069c90 00007fffb8a89e80
GPR16: 00007fffb8a89e80 00007fffb8a89e80 00007fffb8ac3670 0000000000000000
GPR20: 00007fffb8ac2040 00007fffb8a93460 00007fffb0069cc0 000001001f65ab80
GPR24: 00007fffb8a89e80 00007fffb8a89e80 00007fffb8a89e80 0000000000000000
GPR28: 00007fffb8a89e80 00007fffb8a89e80 0000000000000000 00007fffb8a89e80
NIP [00007fffb86ac010] 0x7fffb86ac010
LR [00007fffb8a86924] 0x7fffb8a86924
--- interrupt: c00
INFO: task systemd-udevd:881738 blocked for more than 614 seconds.
Not tainted 5.15.0-rc2+ #1
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
task:systemd-udevd state:D stack: 0 pid:881738 ppid: 708 flags:0x00042482
Call Trace:
[c0000006b317b280] [c0000000007640a4] bio_associate_blkg+0x44/0xb0 (unreliable)
[c0000006b317b470] [c00000000001ea68] __switch_to+0x288/0x4a0
[c0000006b317b4d0] [c000000000e07bfc] __schedule+0x30c/0x9f0
[c0000006b317b590] [c000000000e08348] schedule+0x68/0x120
[c0000006b317b5c0] [c000000000e08adc] io_schedule+0x2c/0x50
[c0000006b317b5f0] [c0000000003ea624] __lock_page+0x1e4/0x430
[c0000006b317b6d0] [c000000000407fc8] truncate_inode_pages_range+0x338/0x8b0
[c0000006b317b850] [c000000000725714] kill_bdev.isra.14+0x44/0x60
[c0000006b317b880] [c0000000007261f4] blkdev_flush_mapping+0x54/0x260
[c0000006b317b960] [c000000000726488] blkdev_put_whole+0x88/0x90
[c0000006b317b9a0] [c00000000072714c] blkdev_put+0x1cc/0x280
[c0000006b317ba00] [c000000000727e9c] blkdev_close+0x3c/0x60
[c0000006b317ba30] [c000000000525694] __fput+0xc4/0x350
[c0000006b317ba80] [c000000000191128] task_work_run+0xf8/0x170
[c0000006b317bad0] [c000000000161c34] do_exit+0x4a4/0xd30
[c0000006b317bba0] [c000000000162594] do_group_exit+0x64/0xe0
[c0000006b317bbe0] [c000000000177fb8] get_signal+0x258/0xce0
[c0000006b317bcd0] [c0000000000219d4] do_notify_resume+0x114/0x480
[c0000006b317bd80] [c000000000030e40] interrupt_exit_user_prepare_main+0x1a0/0x260
[c0000006b317bde0] [c0000000000312e0] syscall_exit_prepare+0x70/0x150
[c0000006b317be10] [c00000000000c758] system_call_common+0xf8/0x250
--- interrupt: c00 at 0x7fff8c6c8674
NIP: 00007fff8c6c8674 LR: 00007fff8c757430 CTR: 0000000000000000
REGS: c0000006b317be80 TRAP: 0c00 Not tainted (5.15.0-rc2+)
MSR: 800000000000d033 <SF,EE,PR,ME,IR,DR,RI,LE> CR: 28002242 XER: 00000000
IRQMASK: 0
GPR00: 0000000000000003 00007fffe0bff620 00007fff8c6f7f00 0000000000010000
GPR04: 00007fff89980038 0000000000040000 00007fff8c6a0190 00007fff8c6a01a0
GPR08: 00007fff8c6a0160 0000000000000000 0000000000000000 0000000000000000
GPR12: 0000000000000000 00007fff8cc46480 0000000000000000 0000000000bab10c
GPR16: 0cb1ba0000000000 00007fff8c79b700 0000000000000000 0000000000000002
GPR20: 0000000000040000 000001002710e800 00000100270680f8 00007fff8c798d28
GPR24: 00007fff8c7c03a4 00007fff8c7c03a4 00007fff89980028 000001002710e850
GPR28: 0000000000040000 0000000c7ff80000 00007fff89980010 000001002710e800
NIP [00007fff8c6c8674] 0x7fff8c6c8674
LR [00007fff8c757430] 0x7fff8c757430
--- interrupt: c00
EXT4-fs (dm-5): error count since last fsck: 3
EXT4-fs (dm-5): initial error at time 1632612344: ext4_map_blocks:593: inode 8: block 1081345
EXT4-fs (dm-5): last error at time 1632628929: ext4_map_blocks:593: inode 2: block 9251
EXT4-fs error (device dm-5): ext4_map_blocks:593: inode #2: block 9251: comm scp: lblock 0 mapped to illegal pblock 9251 (length 1)
EXT4-fs (dm-5): error count since last fsck: 4
EXT4-fs (dm-5): initial error at time 1632612344: ext4_map_blocks:593: inode 8: block 1081345
EXT4-fs (dm-5): last error at time 1632724166: ext4_map_blocks:593: inode 2: block 9251
[-- Attachment #3: Type: text/plain, Size: 97 bytes --]
--
dm-devel mailing list
dm-devel@redhat.com
https://listman.redhat.com/mailman/listinfo/dm-devel
^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: [next-20210827][ppc][multipathd] INFO: task hung in dm_table_add_target
2021-09-28 10:23 ` Abdul Haleem
(?)
@ 2021-09-29 6:52 ` Christoph Hellwig
-1 siblings, 0 replies; 12+ messages in thread
From: Christoph Hellwig @ 2021-09-29 6:52 UTC (permalink / raw)
To: Abdul Haleem
Cc: Christoph Hellwig, linux-kernel, linux-scsi, linux-next, jack,
axboe, linuxppc-dev, dm-devel, sachinp, Brian King, dougmill
On Tue, Sep 28, 2021 at 03:53:47PM +0530, Abdul Haleem wrote:
>
> On 9/1/21 7:06 PM, Christoph Hellwig wrote:
>> On Wed, Sep 01, 2021 at 04:47:26PM +0530, Abdul Haleem wrote:
>>> Greeting's
>>>
>>> multiple task hung while adding the vfc disk back to the multipath on my
>>> powerpc box running linux-next kernel
>> Can you retry to reproduce this with lockdep enabled to see if there
>> is anything interesting holding this lock?
>
> LOCKDEP was earlier enabled by default
>
> # cat .config | grep LOCKDEP
> CONFIG_LOCKDEP_SUPPORT=y
>
> BTW, Recreated again on 5.15.0-rc2 mainline kernel and attaching the logs
It seems the reinstate is blocking on the close which is blocking on
flushing dirty data. In other words it looks like the link blocking
looks like the symptom and not the cause.
^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: [next-20210827][ppc][multipathd] INFO: task hung in dm_table_add_target
@ 2021-09-29 6:52 ` Christoph Hellwig
0 siblings, 0 replies; 12+ messages in thread
From: Christoph Hellwig @ 2021-09-29 6:52 UTC (permalink / raw)
To: Abdul Haleem
Cc: axboe, sachinp, jack, linux-scsi, linux-kernel, dm-devel,
linux-next, dougmill, Brian King, linuxppc-dev,
Christoph Hellwig
On Tue, Sep 28, 2021 at 03:53:47PM +0530, Abdul Haleem wrote:
>
> On 9/1/21 7:06 PM, Christoph Hellwig wrote:
>> On Wed, Sep 01, 2021 at 04:47:26PM +0530, Abdul Haleem wrote:
>>> Greeting's
>>>
>>> multiple task hung while adding the vfc disk back to the multipath on my
>>> powerpc box running linux-next kernel
>> Can you retry to reproduce this with lockdep enabled to see if there
>> is anything interesting holding this lock?
>
> LOCKDEP was earlier enabled by default
>
> # cat .config | grep LOCKDEP
> CONFIG_LOCKDEP_SUPPORT=y
>
> BTW, Recreated again on 5.15.0-rc2 mainline kernel and attaching the logs
It seems the reinstate is blocking on the close which is blocking on
flushing dirty data. In other words it looks like the link blocking
looks like the symptom and not the cause.
^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: [dm-devel] [next-20210827][ppc][multipathd] INFO: task hung in dm_table_add_target
@ 2021-09-29 6:52 ` Christoph Hellwig
0 siblings, 0 replies; 12+ messages in thread
From: Christoph Hellwig @ 2021-09-29 6:52 UTC (permalink / raw)
To: Abdul Haleem
Cc: axboe, sachinp, jack, linux-scsi, linux-kernel, dm-devel,
linux-next, dougmill, Brian King, linuxppc-dev,
Christoph Hellwig
On Tue, Sep 28, 2021 at 03:53:47PM +0530, Abdul Haleem wrote:
>
> On 9/1/21 7:06 PM, Christoph Hellwig wrote:
>> On Wed, Sep 01, 2021 at 04:47:26PM +0530, Abdul Haleem wrote:
>>> Greeting's
>>>
>>> multiple task hung while adding the vfc disk back to the multipath on my
>>> powerpc box running linux-next kernel
>> Can you retry to reproduce this with lockdep enabled to see if there
>> is anything interesting holding this lock?
>
> LOCKDEP was earlier enabled by default
>
> # cat .config | grep LOCKDEP
> CONFIG_LOCKDEP_SUPPORT=y
>
> BTW, Recreated again on 5.15.0-rc2 mainline kernel and attaching the logs
It seems the reinstate is blocking on the close which is blocking on
flushing dirty data. In other words it looks like the link blocking
looks like the symptom and not the cause.
--
dm-devel mailing list
dm-devel@redhat.com
https://listman.redhat.com/mailman/listinfo/dm-devel
^ permalink raw reply [flat|nested] 12+ messages in thread
end of thread, other threads:[~2021-09-29 6:52 UTC | newest]
Thread overview: 12+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2021-09-01 11:17 [next-20210827][ppc][multipathd] INFO: task hung in dm_table_add_target Abdul Haleem
2021-09-01 11:17 ` [dm-devel] " Abdul Haleem
2021-09-01 11:17 ` Abdul Haleem
2021-09-01 13:36 ` Christoph Hellwig
2021-09-01 13:36 ` [dm-devel] " Christoph Hellwig
2021-09-01 13:36 ` Christoph Hellwig
2021-09-28 10:23 ` Abdul Haleem
2021-09-28 10:23 ` [dm-devel] " Abdul Haleem
2021-09-28 10:23 ` Abdul Haleem
2021-09-29 6:52 ` Christoph Hellwig
2021-09-29 6:52 ` [dm-devel] " Christoph Hellwig
2021-09-29 6:52 ` Christoph Hellwig
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.