* [linux] [5.19.0] task hung for indefinite time with call traces when rebooted with Kexec
@ 2022-08-05 6:54 Tasmiya Nalatwad
[not found] ` <6b5267ee-9372-3862-614a-298f7bd9ae60@I-love.SAKURA.ne.jp>
0 siblings, 1 reply; 3+ messages in thread
From: Tasmiya Nalatwad @ 2022-08-05 6:54 UTC (permalink / raw)
To: linux-kernel, linux-scsi, linux-block; +Cc: abdhalee, sachinp, mputtash
Greetings,
[linux] [5.19.0] task hung for indefinite time with call traces when
rebooted with Kexec, A restart is required to recover the machine.
--- Call Traces ---
Starting Reboot via kexec...
[ 909.964343] printk: systemd-shutdow: 38 output lines suppressed due
to ratelimiting
[ 909.967021] systemd-shutdown[1]: Syncing filesystems and block devices.
[ 909.969192] systemd-shutdown[1]: Sending SIGTERM to remaining
processes...
[ 910.003729] systemd-shutdown[1]: Sending SIGKILL to remaining
processes...
[ 910.007053] systemd-shutdown[1]: Unmounting file systems.
[ 910.007458] [228226]: Remounting '/' read-only in with options
'stripe=8'.
[ 910.090154] EXT4-fs (dm-5): re-mounted. Quota mode: none.
[ 910.096694] systemd-shutdown[1]: All filesystems unmounted.
[ 910.096697] systemd-shutdown[1]: Deactivating swaps.
[ 910.096714] systemd-shutdown[1]: All swaps deactivated.
[ 910.096716] systemd-shutdown[1]: Detaching loop devices.
[ 910.096784] systemd-shutdown[1]: All loop devices detached.
[ 910.096786] systemd-shutdown[1]: Detaching DM devices.
[ 910.422063] printk: shutdown: 18 output lines suppressed due to
ratelimiting
dracut Warning: Killing all remaining processes
[ 910.439917] dracut Warning: Killing all remaining processes
[ 911.783936] EXT4-fs (dm-5): unmounting filesystem.
[ 911.784387] dracut Warning: Unmounted /oldroot.
Aug 05 00:11:54 | /etc/multipath.conf line 3, duplicate keyword:
find_multipaths
[ 911.816135] dracut: Disassembling device-mapper devices
[ 911.936046] sd 2:2:0:0: [sdd] Synchronizing SCSI cache
[ 911.939345] sd 1:2:0:0: [sdc] Synchronizing SCSI cache
[ 911.944498] Removing IBM Power 842 compression device
[ 1104.673127] INFO: task kworker/3:1:221 blocked for more than 122 seconds.
[ 1104.673145] Not tainted 5.19.0-rc1-autotest #1
[ 1104.673148] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
disables this message.
[ 1104.673153] task:kworker/3:1 state:D stack: 0 pid: 221 ppid:
2 flags:0x00000800
[ 1104.673160] Workqueue: fc_wq_0 fc_rport_final_delete [scsi_transport_fc]
[ 1104.673170] Call Trace:
[ 1104.673173] [c0000000060eb860] [0000000000000004] 0x4 (unreliable)
[ 1104.673178] [c0000000060eba50] [c00000000001e378] __switch_to+0x288/0x4a0
[ 1104.673185] [c0000000060ebab0] [c000000000d23e84] __schedule+0x2c4/0x8c0
[ 1104.673190] [c0000000060ebb80] [c000000000d244e8] schedule+0x68/0x130
[ 1104.673196] [c0000000060ebbb0] [c0000000008d4574]
scsi_remove_target+0x314/0x390
[ 1104.673201] [c0000000060ebc40] [c008000000926460]
fc_rport_final_delete+0xc8/0x250 [scsi_transport_fc]
[ 1104.673208] [c0000000060ebc90] [c00000000017ef58]
process_one_work+0x288/0x560
[ 1104.673212] [c0000000060ebd30] [c00000000017f2a8]
worker_thread+0x78/0x620
[ 1104.673216] [c0000000060ebdc0] [c00000000018b9b4] kthread+0x124/0x130
[ 1104.673220] [c0000000060ebe10] [c00000000000ce54]
ret_from_kernel_thread+0x5c/0x64
[ 1104.673225] INFO: task kworker/3:0:227332 blocked for more than 122
seconds.
[ 1104.673228] Not tainted 5.19.0-rc1-autotest #1
[ 1104.673230] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
disables this message.
[ 1104.673233] task:kworker/3:0 state:D stack: 0 pid:227332 ppid:
2 flags:0x00000880
[ 1104.673237] Workqueue: fc_wq_0 fc_rport_final_delete [scsi_transport_fc]
[ 1104.673243] Call Trace:
[ 1104.673244] [c0000000726bb860] [c0000000001b9cb4]
enqueue_entity+0x184/0x4f0 (unreliable)
[ 1104.673250] [c0000000726bba50] [c00000000001e378] __switch_to+0x288/0x4a0
[ 1104.673254] [c0000000726bbab0] [c000000000d23e84] __schedule+0x2c4/0x8c0
[ 1104.673258] [c0000000726bbb80] [c000000000d244e8] schedule+0x68/0x130
[ 1104.673262] [c0000000726bbbb0] [c0000000008d4574]
scsi_remove_target+0x314/0x390
[ 1104.673265] [c0000000726bbc40] [c008000000926460]
fc_rport_final_delete+0xc8/0x250 [scsi_transport_fc]
[ 1104.673271] [c0000000726bbc90] [c00000000017ef58]
process_one_work+0x288/0x560
[ 1104.673275] [c0000000726bbd30] [c00000000017f2a8]
worker_thread+0x78/0x620
[ 1104.673279] [c0000000726bbdc0] [c00000000018b9b4] kthread+0x124/0x130
[ 1104.673283] [c0000000726bbe10] [c00000000000ce54]
ret_from_kernel_thread+0x5c/0x64
[ 1104.673288] INFO: task kexec:228289 blocked for more than 122 seconds.
[ 1104.673290] Not tainted 5.19.0-rc1-autotest #1
[ 1104.673292] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
disables this message.
[ 1104.673295] task:kexec state:D stack: 0 pid:228289 ppid:
1 flags:0x00040080
[ 1104.673299] Call Trace:
[ 1104.673301] [c000000069147510] [c00000000001e378] __switch_to+0x288/0x4a0
[ 1104.673305] [c000000069147570] [c000000000d23e84] __schedule+0x2c4/0x8c0
[ 1104.673309] [c000000069147640] [c000000000d244e8] schedule+0x68/0x130
[ 1104.673313] [c000000069147670] [c000000000d2e028]
schedule_timeout+0x348/0x3f0
[ 1104.673317] [c000000069147750] [c000000000d2554c]
wait_for_completion+0xcc/0x2b0
[ 1104.673321] [c0000000691477d0] [c00000000017cbe8]
flush_workqueue+0x158/0x520
[ 1104.673325] [c000000069147870] [c00000000017d068]
drain_workqueue+0xb8/0x240
[ 1104.673329] [c000000069147930] [c0000000001825e0]
destroy_workqueue+0x60/0x420
[ 1104.673333] [c0000000691479c0] [c0080000009291e4]
fc_remove_host+0x21c/0x280 [scsi_transport_fc]
[ 1104.673339] [c000000069147a30] [c0080000009b869c]
ibmvfc_remove+0xf4/0x3a0 [ibmvfc]
[ 1104.673345] [c000000069147ae0] [c00000000011114c]
vio_bus_remove+0x5c/0xd0
[ 1104.673349] [c000000069147b20] [c000000000866ba0]
device_shutdown+0x230/0x3c0
[ 1104.673353] [c000000069147bc0] [c000000000190c94]
kernel_restart_prepare+0x54/0x70
[ 1104.673357] [c000000069147be0] [c00000000026d6c4] kernel_kexec+0xa4/0xf0
[ 1104.673362] [c000000069147c50] [c0000000001912f4]
__do_sys_reboot+0x2b4/0x310
[ 1104.673366] [c000000069147db0] [c0000000000333fc]
system_call_exception+0x18c/0x390
[ 1104.673370] [c000000069147e10] [c00000000000c63c]
system_call_common+0xec/0x250
[ 1104.673375] --- interrupt: c00 at 0x7fff894f5ad4
[ 1104.673377] NIP: 00007fff894f5ad4 LR: 000000012c17469c CTR:
0000000000000000
[ 1104.673380] REGS: c000000069147e80 TRAP: 0c00 Not tainted
(5.19.0-rc1-autotest)
[ 1104.673383] MSR: 800000000000d033 <SF,EE,PR,ME,IR,DR,RI,LE> CR:
28022242 XER: 00000000
[ 1104.673392] IRQMASK: 0
[ 1104.673392] GPR00: 0000000000000058 00007fffd50e1f20 00007fff895e7100
fffffffffee1dead
[ 1104.673392] GPR04: 0000000028121969 0000000045584543 00007fffd50e1e85
0000000000000009
[ 1104.673392] GPR08: 0000000000000003 0000000000000000 0000000000000000
0000000000000000
[ 1104.673392] GPR12: 0000000000000000 00007fff8972b650 ffffffffffffffff
0000000000000002
[ 1104.673392] GPR16: 0000000000000000 00007fffd50e1fc8 0000000000000000
0000000000000000
[ 1104.673392] GPR20: 0000000000000001 0000000000000000 0000000000000002
0000000000000000
[ 1104.673392] GPR24: 0000000000000001 0000000000000001 0000000000000000
0000000000000000
[ 1104.673392] GPR28: 000000012c1922e8 000000012c1be7d0 00007fffd50e2578
0000000000000002
[ 1104.673429] NIP [00007fff894f5ad4] 0x7fff894f5ad4
[ 1104.673431] LR [000000012c17469c] 0x12c17469c
[ 1104.673433] --- interrupt: c00
[ 1227.553154] INFO: task kworker/3:1:221 blocked for more than 245 seconds.
[ 1227.553161] Not tainted 5.19.0-rc1-autotest #1
[ 1227.553163] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
disables this message.
[ 1227.553166] task:kworker/3:1 state:D stack: 0 pid: 221 ppid:
2 flags:0x00000800
[ 1227.553172] Workqueue: fc_wq_0 fc_rport_final_delete [scsi_transport_fc]
[ 1227.553180] Call Trace:
[ 1227.553182] [c0000000060eb860] [0000000000000004] 0x4 (unreliable)
[ 1227.553187] [c0000000060eba50] [c00000000001e378] __switch_to+0x288/0x4a0
[ 1227.553192] [c0000000060ebab0] [c000000000d23e84] __schedule+0x2c4/0x8c0
[ 1227.553197] [c0000000060ebb80] [c000000000d244e8] schedule+0x68/0x130
[ 1227.553202] [c0000000060ebbb0] [c0000000008d4574]
scsi_remove_target+0x314/0x390
[ 1227.553206] [c0000000060ebc40] [c008000000926460]
fc_rport_final_delete+0xc8/0x250 [scsi_transport_fc]
[ 1227.553213] [c0000000060ebc90] [c00000000017ef58]
process_one_work+0x288/0x560
[ 1227.553218] [c0000000060ebd30] [c00000000017f2a8]
worker_thread+0x78/0x620
[ 1227.553221] [c0000000060ebdc0] [c00000000018b9b4] kthread+0x124/0x130
[ 1227.553226] [c0000000060ebe10] [c00000000000ce54]
ret_from_kernel_thread+0x5c/0x64
[ 1227.553232] INFO: task kworker/3:0:227332 blocked for more than 245
seconds.
[ 1227.553235] Not tainted 5.19.0-rc1-autotest #1
[ 1227.553237] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
disables this message.
[ 1227.553240] task:kworker/3:0 state:D stack: 0 pid:227332 ppid:
2 flags:0x00000880
[ 1227.553245] Workqueue: fc_wq_0 fc_rport_final_delete [scsi_transport_fc]
[ 1227.553251] Call Trace:
[ 1227.553252] [c0000000726bb860] [c0000000001b9cb4]
enqueue_entity+0x184/0x4f0 (unreliable)
[ 1227.553257] [c0000000726bba50] [c00000000001e378] __switch_to+0x288/0x4a0
[ 1227.553261] [c0000000726bbab0] [c000000000d23e84] __schedule+0x2c4/0x8c0
[ 1227.553266] [c0000000726bbb80] [c000000000d244e8] schedule+0x68/0x130
[ 1227.553271] [c0000000726bbbb0] [c0000000008d4574]
scsi_remove_target+0x314/0x390
[ 1227.553276] [c0000000726bbc40] [c008000000926460]
fc_rport_final_delete+0xc8/0x250 [scsi_transport_fc]
[ 1227.553283] [c0000000726bbc90] [c00000000017ef58]
process_one_work+0x288/0x560
[ 1227.553286] [c0000000726bbd30] [c00000000017f2a8]
worker_thread+0x78/0x620
[ 1227.553290] [c0000000726bbdc0] [c00000000018b9b4] kthread+0x124/0x130
[ 1227.553294] [c0000000726bbe10] [c00000000000ce54]
ret_from_kernel_thread+0x5c/0x64
[ 1227.553299] INFO: task kexec:228289 blocked for more than 245 seconds.
[ 1227.553301] Not tainted 5.19.0-rc1-autotest #1
[ 1227.553304] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
disables this message.
[ 1227.553306] task:kexec state:D stack: 0 pid:228289 ppid:
1 flags:0x00040080
[ 1227.553310] Call Trace:
[ 1227.553311] [c000000069147510] [c00000000001e378] __switch_to+0x288/0x4a0
[ 1227.553315] [c000000069147570] [c000000000d23e84] __schedule+0x2c4/0x8c0
[ 1227.553320] [c000000069147640] [c000000000d244e8] schedule+0x68/0x130
[ 1227.553323] [c000000069147670] [c000000000d2e028]
schedule_timeout+0x348/0x3f0
[ 1227.553328] [c000000069147750] [c000000000d2554c]
wait_for_completion+0xcc/0x2b0
[ 1227.553332] [c0000000691477d0] [c00000000017cbe8]
flush_workqueue+0x158/0x520
[ 1227.553336] [c000000069147870] [c00000000017d068]
drain_workqueue+0xb8/0x240
[ 1227.553340] [c000000069147930] [c0000000001825e0]
destroy_workqueue+0x60/0x420
[ 1227.553344] [c0000000691479c0] [c0080000009291e4]
fc_remove_host+0x21c/0x280 [scsi_transport_fc]
[ 1227.553350] [c000000069147a30] [c0080000009b869c]
ibmvfc_remove+0xf4/0x3a0 [ibmvfc]
[ 1227.553355] [c000000069147ae0] [c00000000011114c]
vio_bus_remove+0x5c/0xd0
[ 1227.553359] [c000000069147b20] [c000000000866ba0]
device_shutdown+0x230/0x3c0
[ 1227.553364] [c000000069147bc0] [c000000000190c94]
kernel_restart_prepare+0x54/0x70
[ 1227.553368] [c000000069147be0] [c00000000026d6c4] kernel_kexec+0xa4/0xf0
[ 1227.553372] [c000000069147c50] [c0000000001912f4]
__do_sys_reboot+0x2b4/0x310
[ 1227.553376] [c000000069147db0] [c0000000000333fc]
system_call_exception+0x18c/0x390
[ 1227.553380] [c000000069147e10] [c00000000000c63c]
system_call_common+0xec/0x250
[ 1227.553385] --- interrupt: c00 at 0x7fff894f5ad4
[ 1227.553387] NIP: 00007fff894f5ad4 LR: 000000012c17469c CTR:
0000000000000000
[ 1227.553390] REGS: c000000069147e80 TRAP: 0c00 Not tainted
(5.19.0-rc1-autotest)
[ 1227.553393] MSR: 800000000000d033 <SF,EE,PR,ME,IR,DR,RI,LE> CR:
28022242 XER: 00000000
[ 1227.553401] IRQMASK: 0
[ 1227.553401] GPR00: 0000000000000058 00007fffd50e1f20 00007fff895e7100
fffffffffee1dead
[ 1227.553401] GPR04: 0000000028121969 0000000045584543 00007fffd50e1e85
0000000000000009
[ 1227.553401] GPR08: 0000000000000003 0000000000000000 0000000000000000
0000000000000000
[ 1227.553401] GPR12: 0000000000000000 00007fff8972b650 ffffffffffffffff
0000000000000002
[ 1227.553401] GPR16: 0000000000000000 00007fffd50e1fc8 0000000000000000
0000000000000000
[ 1227.553401] GPR20: 0000000000000001 0000000000000000 0000000000000002
0000000000000000
[ 1227.553401] GPR24: 0000000000000001 0000000000000001 0000000000000000
0000000000000000
[ 1227.553401] GPR28: 000000012c1922e8 000000012c1be7d0 00007fffd50e2578
0000000000000002
[ 1227.553437] NIP [00007fff894f5ad4] 0x7fff894f5ad4
[ 1227.553440] LR [000000012c17469c] 0x12c17469c
[ 1227.553442] --- interrupt: c00
[ 1350.433108] INFO: task kworker/3:1:221 blocked for more than 368 seconds.
[ 1350.433117] Not tainted 5.19.0-rc1-autotest #1
[ 1350.433119] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
disables this message.
[ 1350.433122] task:kworker/3:1 state:D stack: 0 pid: 221 ppid:
2 flags:0x00000800
[ 1350.433126] Workqueue: fc_wq_0 fc_rport_final_delete [scsi_transport_fc]
[ 1350.433134] Call Trace:
[ 1350.433135] [c0000000060eb860] [0000000000000004] 0x4 (unreliable)
[ 1350.433139] [c0000000060eba50] [c00000000001e378] __switch_to+0x288/0x4a0
[ 1350.433144] [c0000000060ebab0] [c000000000d23e84] __schedule+0x2c4/0x8c0
[ 1350.433148] [c0000000060ebb80] [c000000000d244e8] schedule+0x68/0x130
[ 1350.433152] [c0000000060ebbb0] [c0000000008d4574]
scsi_remove_target+0x314/0x390
[ 1350.433156] [c0000000060ebc40] [c008000000926460]
fc_rport_final_delete+0xc8/0x250 [scsi_transport_fc]
[ 1350.433162] [c0000000060ebc90] [c00000000017ef58]
process_one_work+0x288/0x560
[ 1350.433166] [c0000000060ebd30] [c00000000017f2a8]
worker_thread+0x78/0x620
[ 1350.433170] [c0000000060ebdc0] [c00000000018b9b4] kthread+0x124/0x130
[ 1350.433174] [c0000000060ebe10] [c00000000000ce54]
ret_from_kernel_thread+0x5c/0x64
[ 1350.433180] INFO: task kworker/3:0:227332 blocked for more than 368
seconds.
[ 1350.433183] Not tainted 5.19.0-rc1-autotest #1
[ 1350.433186] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
disables this message.
[ 1350.433188] task:kworker/3:0 state:D stack: 0 pid:227332 ppid:
2 flags:0x00000880
[ 1350.433192] Workqueue: fc_wq_0 fc_rport_final_delete [scsi_transport_fc]
[ 1350.433197] Call Trace:
[ 1350.433199] [c0000000726bb860] [c0000000001b9cb4]
enqueue_entity+0x184/0x4f0 (unreliable)
[ 1350.433204] [c0000000726bba50] [c00000000001e378] __switch_to+0x288/0x4a0
[ 1350.433208] [c0000000726bbab0] [c000000000d23e84] __schedule+0x2c4/0x8c0
[ 1350.433212] [c0000000726bbb80] [c000000000d244e8] schedule+0x68/0x130
[ 1350.433216] [c0000000726bbbb0] [c0000000008d4574]
scsi_remove_target+0x314/0x390
[ 1350.433220] [c0000000726bbc40] [c008000000926460]
fc_rport_final_delete+0xc8/0x250 [scsi_transport_fc]
[ 1350.433226] [c0000000726bbc90] [c00000000017ef58]
process_one_work+0x288/0x560
[ 1350.433230] [c0000000726bbd30] [c00000000017f2a8]
worker_thread+0x78/0x620
[ 1350.433233] [c0000000726bbdc0] [c00000000018b9b4] kthread+0x124/0x130
[ 1350.433237] [c0000000726bbe10] [c00000000000ce54]
ret_from_kernel_thread+0x5c/0x64
[ 1350.433243] INFO: task kexec:228289 blocked for more than 368 seconds.
[ 1350.433246] Not tainted 5.19.0-rc1-autotest #1
[ 1350.433248] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
disables this message.
[ 1350.433250] task:kexec state:D stack: 0 pid:228289 ppid:
1 flags:0x00040080
[ 1350.433254] Call Trace:
[ 1350.433256] [c000000069147510] [c00000000001e378] __switch_to+0x288/0x4a0
[ 1350.433260] [c000000069147570] [c000000000d23e84] __schedule+0x2c4/0x8c0
[ 1350.433264] [c000000069147640] [c000000000d244e8] schedule+0x68/0x130
[ 1350.433268] [c000000069147670] [c000000000d2e028]
schedule_timeout+0x348/0x3f0
[ 1350.433272] [c000000069147750] [c000000000d2554c]
wait_for_completion+0xcc/0x2b0
[ 1350.433277] [c0000000691477d0] [c00000000017cbe8]
flush_workqueue+0x158/0x520
[ 1350.433280] [c000000069147870] [c00000000017d068]
drain_workqueue+0xb8/0x240
[ 1350.433284] [c000000069147930] [c0000000001825e0]
destroy_workqueue+0x60/0x420
[ 1350.433288] [c0000000691479c0] [c0080000009291e4]
fc_remove_host+0x21c/0x280 [scsi_transport_fc]
[ 1350.433294] [c000000069147a30] [c0080000009b869c]
ibmvfc_remove+0xf4/0x3a0 [ibmvfc]
[ 1350.433300] [c000000069147ae0] [c00000000011114c]
vio_bus_remove+0x5c/0xd0
[ 1350.433304] [c000000069147b20] [c000000000866ba0]
device_shutdown+0x230/0x3c0
[ 1350.433308] [c000000069147bc0] [c000000000190c94]
kernel_restart_prepare+0x54/0x70
[ 1350.433312] [c000000069147be0] [c00000000026d6c4] kernel_kexec+0xa4/0xf0
[ 1350.433317] [c000000069147c50] [c0000000001912f4]
__do_sys_reboot+0x2b4/0x310
[ 1350.433321] [c000000069147db0] [c0000000000333fc]
system_call_exception+0x18c/0x390
[ 1350.433325] [c000000069147e10] [c00000000000c63c]
system_call_common+0xec/0x250
[ 1350.433330] --- interrupt: c00 at 0x7fff894f5ad4
[ 1350.433332] NIP: 00007fff894f5ad4 LR: 000000012c17469c CTR:
0000000000000000
[ 1350.433335] REGS: c000000069147e80 TRAP: 0c00 Not tainted
(5.19.0-rc1-autotest)
[ 1350.433338] MSR: 800000000000d033 <SF,EE,PR,ME,IR,DR,RI,LE> CR:
28022242 XER: 00000000
[ 1350.433347] IRQMASK: 0
[ 1350.433347] GPR00: 0000000000000058 00007fffd50e1f20 00007fff895e7100
fffffffffee1dead
[ 1350.433347] GPR04: 0000000028121969 0000000045584543 00007fffd50e1e85
0000000000000009
[ 1350.433347] GPR08: 0000000000000003 0000000000000000 0000000000000000
0000000000000000
[ 1350.433347] GPR12: 0000000000000000 00007fff8972b650 ffffffffffffffff
0000000000000002
[ 1350.433347] GPR16: 0000000000000000 00007fffd50e1fc8 0000000000000000
0000000000000000
[ 1350.433347] GPR20: 0000000000000001 0000000000000000 0000000000000002
0000000000000000
[ 1350.433347] GPR24: 0000000000000001 0000000000000001 0000000000000000
0000000000000000
[ 1350.433347] GPR28: 000000012c1922e8 000000012c1be7d0 00007fffd50e2578
0000000000000002
[ 1350.433383] NIP [00007fff894f5ad4] 0x7fff894f5ad4
[ 1350.433385] LR [000000012c17469c] 0x12c17469c
[ 1350.433388] --- interrupt: c00
[ 1473.313106] INFO: task kworker/3:1:221 blocked for more than 491 seconds.
[ 1473.313117] Not tainted 5.19.0-rc1-autotest #1
[ 1473.313120] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
disables this message.
[ 1473.313124] task:kworker/3:1 state:D stack: 0 pid: 221 ppid:
2 flags:0x00000800
[ 1473.313129] Workqueue: fc_wq_0 fc_rport_final_delete [scsi_transport_fc]
[ 1473.313138] Call Trace:
[ 1473.313140] [c0000000060eb860] [0000000000000004] 0x4 (unreliable)
[ 1473.313145] [c0000000060eba50] [c00000000001e378] __switch_to+0x288/0x4a0
[ 1473.313152] [c0000000060ebab0] [c000000000d23e84] __schedule+0x2c4/0x8c0
[ 1473.313158] [c0000000060ebb80] [c000000000d244e8] schedule+0x68/0x130
[ 1473.313162] [c0000000060ebbb0] [c0000000008d4574]
scsi_remove_target+0x314/0x390
[ 1473.313167] [c0000000060ebc40] [c008000000926460]
fc_rport_final_delete+0xc8/0x250 [scsi_transport_fc]
[ 1473.313173] [c0000000060ebc90] [c00000000017ef58]
process_one_work+0x288/0x560
[ 1473.313177] [c0000000060ebd30] [c00000000017f2a8]
worker_thread+0x78/0x620
[ 1473.313180] [c0000000060ebdc0] [c00000000018b9b4] kthread+0x124/0x130
[ 1473.313185] [c0000000060ebe10] [c00000000000ce54]
ret_from_kernel_thread+0x5c/0x64
--
Regards,
Tasmiya Nalatwad
IBM Linux Technology Center
^ permalink raw reply [flat|nested] 3+ messages in thread
* Re: [linux] [5.19.0] task hung for indefinite time with call traces when rebooted with Kexec
[not found] ` <6b5267ee-9372-3862-614a-298f7bd9ae60@I-love.SAKURA.ne.jp>
@ 2022-08-18 9:43 ` Tasmiya Nalatwad
2022-08-18 10:20 ` Tetsuo Handa
0 siblings, 1 reply; 3+ messages in thread
From: Tasmiya Nalatwad @ 2022-08-18 9:43 UTC (permalink / raw)
To: Tetsuo Handa, linux-block, linux-kernel; +Cc: abdhalee, mputtash, sachinp
Greetings,
Please find the location in source code from where i am seeing the call
traces generating
File : kernel/hung_task.c
/*
* Ok, the task did not get scheduled for more than 2 minutes,
* complain:
*/
if (sysctl_hung_task_warnings) {
if (sysctl_hung_task_warnings > 0)
sysctl_hung_task_warnings--;
pr_err("INFO: task %s:%d blocked for more than %ld
seconds.\n",
t->comm, t->pid, (jiffies - t->last_switch_time)
/ HZ);
pr_err(" %s %s %.*s\n",
print_tainted(), init_utsname()->release,
(int)strcspn(init_utsname()->version, " "),
init_utsname()->version);
pr_err("\"echo 0 >
/proc/sys/kernel/hung_task_timeout_secs\""
" disables this message.\n");
sched_show_task(t);
hung_task_show_lock = true;
if (sysctl_hung_task_all_cpu_backtrace)
hung_task_show_all_bt = true;
}
touch_nmi_watchdog();
}
On 8/6/22 15:30, Tetsuo Handa wrote:
> On 2022/08/05 15:54, Tasmiya Nalatwad wrote:
>> Greetings,
>>
>> [linux] [5.19.0] task hung for indefinite time with call traces when rebooted with Kexec, A restart is required to recover the machine.
>
> kexec is waiting for workqueues ("kworker/3:1" and "kworker/3:0") to complete.
> If this problem happens only when rebooting with kexec, something in kexec path
> might be preventing these workqueues from completing.
>
> Anyway, please repost with locations in source code like syzbot report does.
>
> [ 1104.673153] task:kworker/3:1 state:D stack: 0 pid: 221 ppid: 2 flags:0x00000800
> [ 1104.673160] Workqueue: fc_wq_0 fc_rport_final_delete [scsi_transport_fc]
> [ 1104.673170] Call Trace:
> [ 1104.673173] [c0000000060eb860] [0000000000000004] 0x4 (unreliable)
> [ 1104.673178] [c0000000060eba50] [c00000000001e378] __switch_to+0x288/0x4a0
> [ 1104.673185] [c0000000060ebab0] [c000000000d23e84] __schedule+0x2c4/0x8c0
> [ 1104.673190] [c0000000060ebb80] [c000000000d244e8] schedule+0x68/0x130
> [ 1104.673196] [c0000000060ebbb0] [c0000000008d4574] scsi_remove_target+0x314/0x390
>
> [ 1104.673233] task:kworker/3:0 state:D stack: 0 pid:227332 ppid: 2 flags:0x00000880
> [ 1104.673237] Workqueue: fc_wq_0 fc_rport_final_delete [scsi_transport_fc]
> [ 1104.673243] Call Trace:
> [ 1104.673244] [c0000000726bb860] [c0000000001b9cb4] enqueue_entity+0x184/0x4f0 (unreliable)
> [ 1104.673250] [c0000000726bba50] [c00000000001e378] __switch_to+0x288/0x4a0
> [ 1104.673254] [c0000000726bbab0] [c000000000d23e84] __schedule+0x2c4/0x8c0
> [ 1104.673258] [c0000000726bbb80] [c000000000d244e8] schedule+0x68/0x130
> [ 1104.673262] [c0000000726bbbb0] [c0000000008d4574] scsi_remove_target+0x314/0x390
> [
> [ 1104.673295] task:kexec state:D stack: 0 pid:228289 ppid: 1 flags:0x00040080
> [ 1104.673299] Call Trace:
> [ 1104.673301] [c000000069147510] [c00000000001e378] __switch_to+0x288/0x4a0
> [ 1104.673305] [c000000069147570] [c000000000d23e84] __schedule+0x2c4/0x8c0
> [ 1104.673309] [c000000069147640] [c000000000d244e8] schedule+0x68/0x130
> [ 1104.673313] [c000000069147670] [c000000000d2e028] schedule_timeout+0x348/0x3f0
> [ 1104.673317] [c000000069147750] [c000000000d2554c] wait_for_completion+0xcc/0x2b0
> [ 1104.673321] [c0000000691477d0] [c00000000017cbe8] flush_workqueue+0x158/0x520
> [ 1104.673325] [c000000069147870] [c00000000017d068] drain_workqueue+0xb8/0x240
> [ 1104.673329] [c000000069147930] [c0000000001825e0] destroy_workqueue+0x60/0x420
> [ 1104.673333] [c0000000691479c0] [c0080000009291e4] fc_remove_host+0x21c/0x280 [scsi_transport_fc]
>
>
--
Regards,
Tasmiya Nalatwad
IBM Linux Technology Center
^ permalink raw reply [flat|nested] 3+ messages in thread
* Re: [linux] [5.19.0] task hung for indefinite time with call traces when rebooted with Kexec
2022-08-18 9:43 ` Tasmiya Nalatwad
@ 2022-08-18 10:20 ` Tetsuo Handa
0 siblings, 0 replies; 3+ messages in thread
From: Tetsuo Handa @ 2022-08-18 10:20 UTC (permalink / raw)
To: Tasmiya Nalatwad; +Cc: abdhalee, mputtash, sachinp, linux-block, linux-kernel
Please check https://elinux.org/images/1/14/Linuxkerneldebugging.pdf for example.
e.g. ./scripts/faddr2line vmlinux scsi_remove_target+0x314/0x390
On 2022/08/18 18:43, Tasmiya Nalatwad wrote:
> Please find the location in source code from where i am seeing the call traces generating
>
> On 8/6/22 15:30, Tetsuo Handa wrote:
>> On 2022/08/05 15:54, Tasmiya Nalatwad wrote:
>>> Greetings,
>>>
>>> [linux] [5.19.0] task hung for indefinite time with call traces when rebooted with Kexec, A restart is required to recover the machine.
>>
>> kexec is waiting for workqueues ("kworker/3:1" and "kworker/3:0") to complete.
>> If this problem happens only when rebooting with kexec, something in kexec path
>> might be preventing these workqueues from completing.
>>
>> Anyway, please repost with locations in source code like syzbot report does.
^ permalink raw reply [flat|nested] 3+ messages in thread
end of thread, other threads:[~2022-08-18 10:20 UTC | newest]
Thread overview: 3+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2022-08-05 6:54 [linux] [5.19.0] task hung for indefinite time with call traces when rebooted with Kexec Tasmiya Nalatwad
[not found] ` <6b5267ee-9372-3862-614a-298f7bd9ae60@I-love.SAKURA.ne.jp>
2022-08-18 9:43 ` Tasmiya Nalatwad
2022-08-18 10:20 ` Tetsuo Handa
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.