* [Bug 214147] ISCSI broken in last release
2021-08-23 16:08 [Bug 214147] New: ISCSI broken in last release bugzilla-daemon
@ 2021-08-23 19:19 ` bugzilla-daemon
2021-09-01 23:48 ` [Bug 214147] New: " michael.christie
` (4 subsequent siblings)
5 siblings, 0 replies; 7+ messages in thread
From: bugzilla-daemon @ 2021-08-23 19:19 UTC (permalink / raw)
To: linux-scsi
https://bugzilla.kernel.org/show_bug.cgi?id=214147
--- Comment #1 from Badalian Slava (slavon.net@gmail.com) ---
5.13.4-1.el8.x86_64 also have this bug.
5.12 work ok
--
You may reply to this email to add a comment.
You are receiving this mail because:
You are the assignee for the bug.
^ permalink raw reply [flat|nested] 7+ messages in thread* Re: [Bug 214147] New: ISCSI broken in last release
2021-08-23 16:08 [Bug 214147] New: ISCSI broken in last release bugzilla-daemon
2021-08-23 19:19 ` [Bug 214147] " bugzilla-daemon
@ 2021-09-01 23:48 ` michael.christie
2021-09-01 23:48 ` [Bug 214147] " bugzilla-daemon
` (3 subsequent siblings)
5 siblings, 0 replies; 7+ messages in thread
From: michael.christie @ 2021-09-01 23:48 UTC (permalink / raw)
To: bugzilla-daemon, linux-scsi
On 8/23/21 6:08 AM, bugzilla-daemon@bugzilla.kernel.org wrote:
> https://bugzilla.kernel.org/show_bug.cgi?id=214147
>
> Bug ID: 214147
> Summary: ISCSI broken in last release
> Product: IO/Storage
> Version: 2.5
> Kernel Version: 5.13.12
> Hardware: All
> OS: Linux
> Tree: Mainline
> Status: NEW
> Severity: normal
> Priority: P1
> Component: SCSI
> Assignee: linux-scsi@vger.kernel.org
> Reporter: slavon.net@gmail.com
> Regression: Yes
>
> Created attachment 298441
> --> https://bugzilla.kernel.org/attachment.cgi?id=298441&action=edit
> dmesg log
>
> After some time iscsi go to broke and help only reboot
>
What are you doing when you hit the issue?
What does your target setup look like? What are you using for the
backing store?
Are you able to build your own kernels?
The only major changes between 5.12 and 5.13 is some target patches
to batch cmds. However, it looks like you start to hit a problem
earlier than when that code comes into play. We first see you hit
a data out timeout, so we don't even have all the data for the
cmd, so the target changes in 5.13 don't come into play yet.
[10931.107057] Unable to recover from DataOut timeout while in ERL=0, closing iSCSI connection for I_T Nexus iqn.1991-05.com.microsoft:vhost11.dev.obs.group,i,0x400001370002,iqn.2003-01.org.linux-iscsi.vm2.x8664:sn.b07943625401,t,0x01
However, we do see some cmds have made it to the core target layer
because we can see the target layer is waiting on cmds to complete
for part of the lun reset handling:
[19906.593285] INFO: task kworker/4:1:3770999 blocked for more than 122 seconds.
[19906.603670] Tainted: P O 5.13.12-1.el8.elrepo.x86_64 #1
[19906.613975] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[19906.624208] task:kworker/4:1 state:D stack: 0 pid:3770999 ppid: 2 flags:0x00004000
[19906.624212] Workqueue: events target_tmr_work [target_core_mod]
[19906.624247] Call Trace:
[19906.624249] __schedule+0x396/0x8a0
[19906.624252] schedule+0x3c/0xa0
[19906.624255] schedule_timeout+0x215/0x2b0
[19906.624258] ? kasprintf+0x4e/0x70
[19906.624261] wait_for_completion+0x9e/0x100
[19906.624264] target_put_cmd_and_wait+0x55/0x80 [target_core_mod]
[19906.624279] core_tmr_lun_reset+0x38b/0x660 [target_core_mod]
[19906.624294] target_tmr_work+0xb4/0x110 [target_core_mod]
[19906.624309] process_one_work+0x230/0x3d0
[19906.624312] worker_thread+0x2d/0x3e0
[19906.624314] ? process_one_work+0x3d0/0x3d0
[19906.624316] kthread+0x118/0x140
[19906.624318] ? set_kthread_struct+0x40/0x40
[19906.624320] ret_from_fork+0x1f/0x30
and we can see iscsi layer not able to relogin because of outstanding
cmds/tmfs.
I can send you a patch that reverts the core target patches. If we can
rule them out then it would help narrow things down.
Or, because it sounds like this is easy to reproduce we can turn on some
extra lio debugging.
^ permalink raw reply [flat|nested] 7+ messages in thread* [Bug 214147] ISCSI broken in last release
2021-08-23 16:08 [Bug 214147] New: ISCSI broken in last release bugzilla-daemon
2021-08-23 19:19 ` [Bug 214147] " bugzilla-daemon
2021-09-01 23:48 ` [Bug 214147] New: " michael.christie
@ 2021-09-01 23:48 ` bugzilla-daemon
2021-09-03 6:22 ` bugzilla-daemon
` (2 subsequent siblings)
5 siblings, 0 replies; 7+ messages in thread
From: bugzilla-daemon @ 2021-09-01 23:48 UTC (permalink / raw)
To: linux-scsi
https://bugzilla.kernel.org/show_bug.cgi?id=214147
--- Comment #2 from michael.christie@oracle.com ---
On 8/23/21 6:08 AM, bugzilla-daemon@bugzilla.kernel.org wrote:
> https://bugzilla.kernel.org/show_bug.cgi?id=214147
>
> Bug ID: 214147
> Summary: ISCSI broken in last release
> Product: IO/Storage
> Version: 2.5
> Kernel Version: 5.13.12
> Hardware: All
> OS: Linux
> Tree: Mainline
> Status: NEW
> Severity: normal
> Priority: P1
> Component: SCSI
> Assignee: linux-scsi@vger.kernel.org
> Reporter: slavon.net@gmail.com
> Regression: Yes
>
> Created attachment 298441
> --> https://bugzilla.kernel.org/attachment.cgi?id=298441&action=edit
> dmesg log
>
> After some time iscsi go to broke and help only reboot
>
What are you doing when you hit the issue?
What does your target setup look like? What are you using for the
backing store?
Are you able to build your own kernels?
The only major changes between 5.12 and 5.13 is some target patches
to batch cmds. However, it looks like you start to hit a problem
earlier than when that code comes into play. We first see you hit
a data out timeout, so we don't even have all the data for the
cmd, so the target changes in 5.13 don't come into play yet.
[10931.107057] Unable to recover from DataOut timeout while in ERL=0, closing
iSCSI connection for I_T Nexus
iqn.1991-05.com.microsoft:vhost11.dev.obs.group,i,0x400001370002,iqn.2003-01.org.linux-iscsi.vm2.x8664:sn.b07943625401,t,0x01
However, we do see some cmds have made it to the core target layer
because we can see the target layer is waiting on cmds to complete
for part of the lun reset handling:
[19906.593285] INFO: task kworker/4:1:3770999 blocked for more than 122
seconds.
[19906.603670] Tainted: P O 5.13.12-1.el8.elrepo.x86_64 #1
[19906.613975] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this
message.
[19906.624208] task:kworker/4:1 state:D stack: 0 pid:3770999 ppid: 2
flags:0x00004000
[19906.624212] Workqueue: events target_tmr_work [target_core_mod]
[19906.624247] Call Trace:
[19906.624249] __schedule+0x396/0x8a0
[19906.624252] schedule+0x3c/0xa0
[19906.624255] schedule_timeout+0x215/0x2b0
[19906.624258] ? kasprintf+0x4e/0x70
[19906.624261] wait_for_completion+0x9e/0x100
[19906.624264] target_put_cmd_and_wait+0x55/0x80 [target_core_mod]
[19906.624279] core_tmr_lun_reset+0x38b/0x660 [target_core_mod]
[19906.624294] target_tmr_work+0xb4/0x110 [target_core_mod]
[19906.624309] process_one_work+0x230/0x3d0
[19906.624312] worker_thread+0x2d/0x3e0
[19906.624314] ? process_one_work+0x3d0/0x3d0
[19906.624316] kthread+0x118/0x140
[19906.624318] ? set_kthread_struct+0x40/0x40
[19906.624320] ret_from_fork+0x1f/0x30
and we can see iscsi layer not able to relogin because of outstanding
cmds/tmfs.
I can send you a patch that reverts the core target patches. If we can
rule them out then it would help narrow things down.
Or, because it sounds like this is easy to reproduce we can turn on some
extra lio debugging.
--
You may reply to this email to add a comment.
You are receiving this mail because:
You are the assignee for the bug.
^ permalink raw reply [flat|nested] 7+ messages in thread* [Bug 214147] ISCSI broken in last release
2021-08-23 16:08 [Bug 214147] New: ISCSI broken in last release bugzilla-daemon
` (2 preceding siblings ...)
2021-09-01 23:48 ` [Bug 214147] " bugzilla-daemon
@ 2021-09-03 6:22 ` bugzilla-daemon
2021-09-16 5:06 ` bugzilla-daemon
2021-09-16 9:04 ` bugzilla-daemon
5 siblings, 0 replies; 7+ messages in thread
From: bugzilla-daemon @ 2021-09-03 6:22 UTC (permalink / raw)
To: linux-scsi
https://bugzilla.kernel.org/show_bug.cgi?id=214147
--- Comment #3 from Badalian Slava (slavon.net@gmail.com) ---
Comment on attachment 298441
--> https://bugzilla.kernel.org/attachment.cgi?id=298441
dmesg log
Ok. Send me patch, i can test it.
we use ZFS (last git(. Have 2 pools - DRAID from 6 HDD amd stripe for 10 SSD
850 EVI/
Windows Cluser connect ZFS blocked devices as Shared clustered disk on iSCSI in
10GB mellonox channel. (Noo RDMA support in windows side in iSCSI).
Linux connect in NFS over RDMA on infiniband 40 g network.
We wont recreate SSD pool and move all VMs to HDD. Thern try move next VM. In
Windows host start migrate HV disk from, one clustered disk to second.. 10-15
min and iSCSI broke.
5.12 reconnect after some time. 5.13 system is zrease. Linux can't reboot. Only
ipmi power reset.
SATA is LSI3008.
[root@vm2 ~]# lsscsi
[0:0:0:0] disk ATA Samsung SSD 850 3B6Q /dev/sdb
[0:0:1:0] disk ATA Samsung SSD 850 3B6Q /dev/sdc
[0:0:2:0] disk ATA Samsung SSD 850 3B6Q /dev/sdd
[0:0:3:0] disk ATA Samsung SSD 850 3B6Q /dev/sde
[0:0:4:0] disk ATA Samsung SSD 850 2B6Q /dev/sdf
[0:0:5:0] disk ATA Samsung SSD 850 2B6Q /dev/sdg
[0:0:6:0] disk ATA Samsung SSD 850 3B6Q /dev/sds
[0:0:7:0] disk ATA Samsung SSD 850 3B6Q /dev/sdt
[4:0:0:0] disk ATA KINGSTON SV300S3 BBF0 /dev/sda
[9:0:0:0] disk ATA Samsung SSD 850 3B6Q /dev/sdh
[9:0:1:0] disk ATA Samsung SSD 850 3B6Q /dev/sdi
[9:0:2:0] disk ATA ST2000DM001-1ER1 CC26 /dev/sdj
[9:0:3:0] disk ATA ST2000DM001-1ER1 CC26 /dev/sdk
[9:0:4:0] disk ATA INTEL SSDSC2CW24 400i /dev/sdl
[9:0:5:0] disk ATA ST2000DM001-1CH1 CC29 /dev/sdm
[9:0:6:0] disk ATA ST2000DM001-1CH1 CC29 /dev/sdn
[9:0:7:0] disk ATA WDC WD20EZRZ-00Z 0A80 /dev/sdo
[9:0:8:0] disk ATA INTEL SSDSC2CW24 400i /dev/sdp
[9:0:9:0] disk ATA ST3000DM001-1CH1 CC27 /dev/sdq
[9:0:10:0] disk ATA ST3000DM001-1CH1 CC27 /dev/sdr
[9:0:11:0] enclosu LSI SAS2X28 0e12 -
[N:0:1:1] disk Samsung SSD 950 PRO 256GB__1 /dev/nvme0n1
[N:1:1:1] disk Samsung SSD 950 PRO 256GB__1 /dev/nvme1n1
[root@vm2 ~]#
[root@vm2 ~]# zfs pool -^C
[root@vm2 ~]# zpool list -v
NAME SIZE ALLOC FREE CKPOINT
EXPANDSZ FRAG CAP DEDUP HEALTH ALTROOT
s 15.4T 11.5T 3.86T -
- 61% 74% 1.20x ONLINE -
sdf 3.62T 2.08T 1.54T -
- 40% 57.5% - ONLINE
sdg 3.62T 2.07T 1.56T -
- 41% 57.0% - ONLINE
ata-Samsung_SSD_850_EVO_1TB_S3NZNF0JC02715K 928G 812G 116G -
- 77% 87.5% - ONLINE
ata-Samsung_SSD_850_EVO_1TB_S3NZNF0JC02818K 928G 819G 109G -
- 77% 88.3% - ONLINE
ata-Samsung_SSD_850_EVO_1TB_S3NZNF0JC02714Y 928G 815G 113G -
- 77% 87.8% - ONLINE
ata-Samsung_SSD_850_EVO_1TB_S3NZNF0JC02708N 928G 827G 101G -
- 79% 89.2% - ONLINE
ata-Samsung_SSD_850_EVO_1TB_S3NZNF0JC02720L 928G 821G 107G -
- 72% 88.4% - ONLINE
ata-Samsung_SSD_850_EVO_1TB_S3NZNF0JC02717V 928G 834G 93.8G -
- 76% 89.9% - ONLINE
ata-Samsung_SSD_850_EVO_1TB_S3NZNF0JC02707F 928G 870G 58.1G -
- 84% 93.7% - ONLINE
ata-Samsung_SSD_850_EVO_1TB_S3NZNF0JC02719H 928G 907G 21.2G -
- 94% 97.7% - ONLINE
nvme1n1 238G 211G 27.1G -
- 87% 88.6% - ONLINE
nvme0n1 238G 210G 27.8G -
- 87% 88.3% - ONLINE
ata-INTEL_SSDSC2CW240A3_CVCV316506PA240FGN 222G 219G 2.55G -
- 87% 98.8% - ONLINE
ata-INTEL_SSDSC2CW240A3_CVCV316500UB240FGN 222G 220G 2.46G -
- 87% 98.9% - ONLINE
z 10.9T 4.22T 6.68T -
- 28% 38% 1.11x ONLINE -
draid2:4d:7c:1s 10.9T 4.22T 6.68T -
- 28% 38.7% - ONLINE
ata-ST2000DM001-1ER164_Z5606648 - - - -
- - - - ONLINE
ata-ST2000DM001-1ER164_Z4Z5X09J - - - -
- - - - ONLINE
ata-ST2000DM001-1CH164_Z340KWSB - - - -
- - - - ONLINE
ata-ST2000DM001-1CH164_Z1E668SP - - - -
- - - - ONLINE
ata-ST3000DM001-1CH166_Z1F3WA6S - - - -
- - - - ONLINE
ata-ST3000DM001-1CH166_Z1F3RQCR - - - -
- - - - ONLINE
ata-WDC_WD20EZRZ-00Z5HB0_WD-WCC4N2KHEF9K - - - -
- - - - ONLINE
spare - - - -
- - - - -
draid2-0-0 - - - -
- - - - AVAIL
--
You may reply to this email to add a comment.
You are receiving this mail because:
You are the assignee for the bug.
^ permalink raw reply [flat|nested] 7+ messages in thread* [Bug 214147] ISCSI broken in last release
2021-08-23 16:08 [Bug 214147] New: ISCSI broken in last release bugzilla-daemon
` (3 preceding siblings ...)
2021-09-03 6:22 ` bugzilla-daemon
@ 2021-09-16 5:06 ` bugzilla-daemon
2021-09-16 9:04 ` bugzilla-daemon
5 siblings, 0 replies; 7+ messages in thread
From: bugzilla-daemon @ 2021-09-16 5:06 UTC (permalink / raw)
To: linux-scsi
https://bugzilla.kernel.org/show_bug.cgi?id=214147
--- Comment #4 from Badalian Slava (slavon.net@gmail.com) ---
I get some info .
Test case
have 2 zfs pool. HDD is zdev (raid6) SDD is stype.
simple copy from SSD to HDD
Specific^
1. I use NFS with RDMA and 16 threads (for linux hosts)
2/ i use targetcli and nvmet for share devs to WinClusteredHosts
When se get deadlock i see in perf
amples: 1M of event 'cycles', Event count (approx.): 711199023386
Children Self Command Shared Object Symbol
- 30,57% 0,00% z_wr_iss [kernel.kallsyms] [k]
ret_from_fork
◆
ret_from_fork
▒
- kthread
▒
- 30,53% taskq_create
▒
- 16,50% zio_inject_init
▒
- 4,77% 0xffffffffc0c7285f
▒
- 4,73% raidz_rec_pr_coeff.isra.5
▒
- 3,86% secpolicy_vnode_setid_retain
▒
- 2,10% zthr_resume
▒
- 1,85% vdev_label_init
▒
- 1,03% secpolicy_vnode_setid_retain
▒
- 0,73% zthr_cancel
▒
0,72% vdev_rebuild_update_sync
▒
- 1,57% zthr_cancel
▒
- 1,55% vdev_rebuild_update_sync
▒
0,53% __mutex_lock.isra.13
▒
0,53% avl_remove
▒
0,71% abd_bio_map_off
▒
- 3,70% vdev_disk_open
▒
- 3,69% range_tree_remove_impl
▒
- 3,45% range_tree_add_impl
▒
- 2,66% mmp_thread
▒
1,24% metaslab_enable
▒
0,74% __mutex_lock.isra.13
▒
- 2,71% zvol_set_volsize
▒
- 2,51% zfs_aclset_common
▒
- 2,51% metaslab_load.part.29
▒
1,31% metaslab_class_evict_old
▒
0,96% metaslab_unload.part.28
▒
- 1,84% zvol_set_volmode_sync_cb
▒
- 1,83% zio_handle_device_injection_impl.part.15
▒
- 1,57% spl_taskq_fini
▒
- 1,55% _raw_spin_lock_irqsave
▒
native_queued_spin_lock_slowpath
▒
- 1,20% zvol_set_volmode_sync
▒
1,02% l2arc_read_done
▒
- 0,63% zio_inject_list_next
▒
- 0,62% zfs_mode_compute
▒
- 0,59% zfs_mode_compute
▒
0,53% abd_iterate_func2.part.16
▒
- 12,00% _raw_spin_lock_irqsave
▒
native_queued_spin_lock_slowpath
▒
- 0,93% schedule
▒
0,86% __sched_text_start
▒
- 30,57% 0,00% z_wr_iss [kernel.kallsyms] [k] kthread
▒
- 30,57% kthread
▒
- 30,53% taskq_create
▒
- 16,50% zio_inject_init
▒
- 4,77% 0xffffffffc0c7285f
▒
- 4,73% raidz_rec_pr_coeff.isra.5
and i see in
[root@vm2 ~]# cat /proc/locks
1: FLOCK ADVISORY WRITE 999321 00:18:3499 0 EOF
2: POSIX ADVISORY WRITE 949090 00:18:3395 0 EOF
3: POSIX ADVISORY WRITE 949090 00:18:3394 0 EOF
4: POSIX ADVISORY WRITE 948911 00:18:3322 0 EOF
5: POSIX ADVISORY WRITE 999305 00:18:3511 0 EOF
butt one.one/two in ACTIVE state., pid is nfs5 kernel p[process. deadlocked in
nfs!
--
You may reply to this email to add a comment.
You are receiving this mail because:
You are the assignee for the bug.
^ permalink raw reply [flat|nested] 7+ messages in thread* [Bug 214147] ISCSI broken in last release
2021-08-23 16:08 [Bug 214147] New: ISCSI broken in last release bugzilla-daemon
` (4 preceding siblings ...)
2021-09-16 5:06 ` bugzilla-daemon
@ 2021-09-16 9:04 ` bugzilla-daemon
5 siblings, 0 replies; 7+ messages in thread
From: bugzilla-daemon @ 2021-09-16 9:04 UTC (permalink / raw)
To: linux-scsi
https://bugzilla.kernel.org/show_bug.cgi?id=214147
--- Comment #5 from Badalian Slava (slavon.net@gmail.com) ---
WWith turned off ISCSI and NVMEt. only NFS active
.
```
[ 2164.273231] perf: interrupt took too long (3197 > 3148), lowering
kernel.perf_event_max_sample_rate to 62000
[ 3067.386864] perf: interrupt took too long (4023 > 3996), lowering
kernel.perf_event_max_sample_rate to 49000
[ 4508.876951] perf: interrupt took too long (5034 > 5028), lowering
kernel.perf_event_max_sample_rate to 39000
[ 7336.586251] perf: interrupt took too long (6305 > 6292), lowering
kernel.perf_event_max_sample_rate to 31000
[16343.647108] INFO: task txg_sync:897001 blocked for more than 122 seconds.
[16343.657122] Tainted: P O 5.13.12-1.el8.elrepo.x86_64 #1
[16343.667018] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this
message.
[16343.676854] task:txg_sync state:D stack: 0 pid:897001 ppid: 2
flags:0x00004004
[16343.686665] Call Trace:
[16343.696132] __schedule+0x396/0x8a0
[16343.705378] schedule+0x3c/0xa0
[16343.714360] schedule_timeout+0x197/0x2b0
[16343.723125] ? timer_update_keys+0x30/0x30
[16343.731752] io_schedule_timeout+0x19/0x40
[16343.740453] __cv_timedwait_common+0x12b/0x160 [spl]
[16343.749335] ? wait_woken+0x80/0x80
[16343.758275] __cv_timedwait_io+0x15/0x20 [spl]
[16343.767344] zio_wait+0x129/0x2b0 [zfs]
[16343.776599] dsl_pool_sync+0xd0/0x470 [zfs]
[16343.785629] ? spa_errlog_sync+0x25c/0x2b0 [zfs]
[16343.794490] ? __kmalloc_node+0x181/0x330
[16343.803219] spa_sync_iterate_to_convergence+0xf0/0x250 [zfs]
[16343.811995] spa_sync+0x389/0x690 [zfs]
[16343.820544] txg_sync_thread+0x270/0x2f0 [zfs]
[16343.829106] ? txg_dispatch_callbacks+0x120/0x120 [zfs]
[16343.837545] ? __thread_exit+0x20/0x20 [spl]
[16343.845678] thread_generic_wrapper+0x6c/0x80 [spl]
[16343.853833] kthread+0x118/0x140
[16343.861743] ? set_kthread_struct+0x40/0x40
[16343.869501] ret_from_fork+0x1f/0x30
[16589.391679] INFO: task txg_sync:897001 blocked for more than 122 seconds.
[16589.399411] Tainted: P O 5.13.12-1.el8.elrepo.x86_64 #1
[16589.407078] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this
message.
[16589.414718] task:txg_sync state:D stack: 0 pid:897001 ppid: 2
flags:0x00004004
[16589.422347] Call Trace:
[16589.429638] __schedule+0x396/0x8a0
[16589.436751] schedule+0x3c/0xa0
[16589.443608] schedule_timeout+0x197/0x2b0
[16589.450258] ? timer_update_keys+0x30/0x30
[16589.456706] io_schedule_timeout+0x19/0x40
[16589.462926] __cv_timedwait_common+0x12b/0x160 [spl]
[16589.469088] ? wait_woken+0x80/0x80
[16589.475108] __cv_timedwait_io+0x15/0x20 [spl]
[16589.481039] zio_wait+0x129/0x2b0 [zfs]
[16589.486879] dsl_pool_sync+0xd0/0x470 [zfs]
[16589.492585] ? spa_errlog_sync+0x25c/0x2b0 [zfs]
[16589.498376] ? __kmalloc_node+0x181/0x330
[16589.504065] spa_sync_iterate_to_convergence+0xf0/0x250 [zfs]
[16589.510015] spa_sync+0x389/0x690 [zfs]
[16589.515917] txg_sync_thread+0x270/0x2f0 [zfs]
[16589.521867] ? txg_dispatch_callbacks+0x120/0x120 [zfs]
[16589.527853] ? __thread_exit+0x20/0x20 [spl]
[16589.533715] thread_generic_wrapper+0x6c/0x80 [spl]
[16589.539629] kthread+0x118/0x140
[16589.545490] ? set_kthread_struct+0x40/0x40
[16589.551393] ret_from_fork+0x1f/0x30
[16712.262822] INFO: task txg_sync:897001 blocked for more than 245 seconds.
[16712.268937] Tainted: P O 5.13.12-1.el8.elrepo.x86_64 #1
[16712.275093] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this
message.
[16712.281379] task:txg_sync state:D stack: 0 pid:897001 ppid: 2
flags:0x00004004
[16712.287872] Call Trace:
[16712.294197] __schedule+0x396/0x8a0
[16712.300540] schedule+0x3c/0xa0
[16712.306795] schedule_timeout+0x197/0x2b0
[16712.313005] ? timer_update_keys+0x30/0x30
[16712.319211] io_schedule_timeout+0x19/0x40
[16712.325401] __cv_timedwait_common+0x12b/0x160 [spl]
[16712.331551] ? wait_woken+0x80/0x80
[16712.337545] __cv_timedwait_io+0x15/0x20 [spl]
[16712.343428] zio_wait+0x129/0x2b0 [zfs]
[16712.349237] dsl_pool_sync+0xd0/0x470 [zfs]
[16712.354918] ? spa_errlog_sync+0x25c/0x2b0 [zfs]
[16712.360658] ? __kmalloc_node+0x181/0x330
[16712.366315] spa_sync_iterate_to_convergence+0xf0/0x250 [zfs]
[16712.372219] spa_sync+0x389/0x690 [zfs]
[16712.378091] txg_sync_thread+0x270/0x2f0 [zfs]
[16712.384027] ? txg_dispatch_callbacks+0x120/0x120 [zfs]
[16712.389992] ? __thread_exit+0x20/0x20 [spl]
[16712.395841] thread_generic_wrapper+0x6c/0x80 [spl]
[16712.401736] kthread+0x118/0x140
[16712.407599] ? set_kthread_struct+0x40/0x40
[16712.413528] ret_from_fork+0x1f/0x30
[17080.879563] INFO: task txg_sync:897001 blocked for more than 122 seconds.
[17080.885652] Tainted: P O 5.13.12-1.el8.elrepo.x86_64 #1
[17080.891804] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this
message.
[17080.898104] task:txg_sync state:D stack: 0 pid:897001 ppid: 2
flags:0x00004004
[17080.904598] Call Trace:
[17080.910927] __schedule+0x396/0x8a0
[17080.917254] schedule+0x3c/0xa0
[17080.923501] schedule_timeout+0x197/0x2b0
[17080.929722] ? timer_update_keys+0x30/0x30
[17080.935962] io_schedule_timeout+0x19/0x40
[17080.942162] __cv_timedwait_common+0x12b/0x160 [spl]
[17080.948331] ? wait_woken+0x80/0x80
[17080.954333] __cv_timedwait_io+0x15/0x20 [spl]
[17080.960221] zio_wait+0x129/0x2b0 [zfs]
[17080.966039] dsl_pool_sync+0xd0/0x470 [zfs]
[17080.971729] ? spa_errlog_sync+0x25c/0x2b0 [zfs]
[17080.977490] ? __kmalloc_node+0x181/0x330
[17080.983150] spa_sync_iterate_to_convergence+0xf0/0x250 [zfs]
[17080.989056] spa_sync+0x389/0x690 [zfs]
[17080.994918] txg_sync_thread+0x270/0x2f0 [zfs]
[17081.000838] ? txg_dispatch_callbacks+0x120/0x120 [zfs]
[17081.006814] ? __thread_exit+0x20/0x20 [spl]
[17081.012668] thread_generic_wrapper+0x6c/0x80 [spl]
[17081.018579] kthread+0x118/0x140
[17081.024437] ? set_kthread_struct+0x40/0x40
[17081.030328] ret_from_fork+0x1f/0x30
[17449.496395] INFO: task txg_sync:897001 blocked for more than 122 seconds.
[17449.502486] Tainted: P O 5.13.12-1.el8.elrepo.x86_64 #1
[17449.508658] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this
message.
[17449.514971] task:txg_sync state:D stack: 0 pid:897001 ppid: 2
flags:0x00004004
[17449.521482] Call Trace:
[17449.527816] __schedule+0x396/0x8a0
[17449.534139] schedule+0x3c/0xa0
[17449.540394] schedule_timeout+0x197/0x2b0
[17449.546614] ? timer_update_keys+0x30/0x30
[17449.552862] io_schedule_timeout+0x19/0x40
[17449.559082] __cv_timedwait_common+0x12b/0x160 [spl]
[17449.565219] ? wait_woken+0x80/0x80
[17449.571213] __cv_timedwait_io+0x15/0x20 [spl]
[17449.577088] zio_wait+0x129/0x2b0 [zfs]
[17449.582897] dsl_pool_sync+0xd0/0x470 [zfs]
[17449.588566] ? spa_errlog_sync+0x25c/0x2b0 [zfs]
[17449.594304] ? __kmalloc_node+0x181/0x330
[17449.599966] spa_sync_iterate_to_convergence+0xf0/0x250 [zfs]
[17449.605877] spa_sync+0x389/0x690 [zfs]
[17449.611760] txg_sync_thread+0x270/0x2f0 [zfs]
[17449.617729] ? txg_dispatch_callbacks+0x120/0x120 [zfs]
[17449.623713] ? __thread_exit+0x20/0x20 [spl]
[17449.629578] thread_generic_wrapper+0x6c/0x80 [spl]
[17449.635512] kthread+0x118/0x140
[17449.641394] ? set_kthread_struct+0x40/0x40
[17449.647310] ret_from_fork+0x1f/0x30
[17940.988317] INFO: task txg_sync:897001 blocked for more than 122 seconds.
[17940.994463] Tainted: P O 5.13.12-1.el8.elrepo.x86_64 #1
[17941.000626] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this
message.
[17941.006941] task:txg_sync state:D stack: 0 pid:897001 ppid: 2
flags:0x00004004
[17941.013462] Call Trace:
[17941.019812] __schedule+0x396/0x8a0
[17941.026154] schedule+0x3c/0xa0
[17941.032430] schedule_timeout+0x197/0x2b0
[17941.038665] ? timer_update_keys+0x30/0x30
[17941.044893] io_schedule_timeout+0x19/0x40
[17941.051104] __cv_timedwait_common+0x12b/0x160 [spl]
[17941.057261] ? wait_woken+0x80/0x80
[17941.063275] __cv_timedwait_io+0x15/0x20 [spl]
[17941.069174] zio_wait+0x129/0x2b0 [zfs]
[17941.075005] dsl_pool_sync+0xd0/0x470 [zfs]
[17941.080690] ? spa_errlog_sync+0x25c/0x2b0 [zfs]
[17941.086450] ? __kmalloc_node+0x181/0x330
[17941.092137] spa_sync_iterate_to_convergence+0xf0/0x250 [zfs]
[17941.098102] spa_sync+0x389/0x690 [zfs]
[17941.103993] txg_sync_thread+0x270/0x2f0 [zfs]
[17941.109957] ? txg_dispatch_callbacks+0x120/0x120 [zfs]
[17941.115964] ? __thread_exit+0x20/0x20 [spl]
[17941.121841] thread_generic_wrapper+0x6c/0x80 [spl]
[17941.127779] kthread+0x118/0x140
[17941.133647] ? set_kthread_struct+0x40/0x40
[17941.139561] ret_from_fork+0x1f/0x30
[root@vm2 parameters]# cat /proc/locks
1: POSIX ADVISORY READ 999553 00:31:55 201 201
2: POSIX ADVISORY READ 999553 00:31:55 100 101
3: DELEG ACTIVE READ 999553 00:31:189 0 EOF
4: FLOCK ADVISORY WRITE 999321 00:18:3499 0 EOF
5: POSIX ADVISORY WRITE 949090 00:18:3395 0 EOF
6: POSIX ADVISORY WRITE 949090 00:18:3394 0 EOF
7: POSIX ADVISORY WRITE 948911 00:18:3322 0 EOF
8: DELEG ACTIVE READ 999553 00:31:55 0 EOF
9: POSIX ADVISORY WRITE 999305 00:18:3511 0 EOF
```
--
You may reply to this email to add a comment.
You are receiving this mail because:
You are the assignee for the bug.
^ permalink raw reply [flat|nested] 7+ messages in thread