* Re: nfsd hangs and nfsd_break_deleg_cb+0x170/0x190 warning
@ 2024-03-12 12:24 Rik Theys
2024-03-12 12:47 ` Jeff Layton
0 siblings, 1 reply; 26+ messages in thread
From: Rik Theys @ 2024-03-12 12:24 UTC (permalink / raw)
To: Jeff Layton, Linux Nfs
Hi Jeff,
On 3/12/24 12:22, Jeff Layton wrote:
> On Mon, 2024-03-11 at 19:43 +0100, Rik Theys wrote:
>> Since a few weeks our Rocky Linux 9 NFS server has periodically logged hung nfsd tasks. The initial effect was that some clients could no longer access the NFS server. This got worse and worse (probably as more nfsd threads got blocked) and we had to restart the server. Restarting the server also failed as the NFS server service could no longer be stopped.
>>
>>
>> The initial kernel we noticed this behavior on was kernel-5.14.0-362.18.1.el9_3.x86_64. Since then we've installed kernel-5.14.0-419.el9.x86_64 from CentOS Stream 9. The same issue happened again on this newer kernel version:
>>
>> [Mon Mar 11 14:10:08 2024] Not tainted 5.14.0-419.el9.x86_64 #1
>> [Mon Mar 11 14:10:08 2024] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
>> [Mon Mar 11 14:10:08 2024] task:nfsd state:D stack:0 pid:8865 ppid:2 flags:0x00004000
>> [Mon Mar 11 14:10:08 2024] Call Trace:
>> [Mon Mar 11 14:10:08 2024] <TASK>
>> [Mon Mar 11 14:10:08 2024] __schedule+0x21b/0x550
>> [Mon Mar 11 14:10:08 2024] schedule+0x2d/0x70
>> [Mon Mar 11 14:10:08 2024] schedule_timeout+0x11f/0x160
>> [Mon Mar 11 14:10:08 2024] ? select_idle_sibling+0x28/0x430
>> [Mon Mar 11 14:10:08 2024] ? wake_affine+0x62/0x1f0
>> [Mon Mar 11 14:10:08 2024] __wait_for_common+0x90/0x1d0
>> [Mon Mar 11 14:10:08 2024] ? __pfx_schedule_timeout+0x10/0x10
>> [Mon Mar 11 14:10:08 2024] __flush_workqueue+0x13a/0x3f0
>> [Mon Mar 11 14:10:08 2024] nfsd4_shutdown_callback+0x49/0x120 [nfsd]
>> [Mon Mar 11 14:10:08 2024] ? nfsd4_cld_remove+0x54/0x1d0 [nfsd]
>> [Mon Mar 11 14:10:08 2024] ? nfsd4_return_all_client_layouts+0xc4/0xf0 [nfsd]
>> [Mon Mar 11 14:10:08 2024] ? nfsd4_shutdown_copy+0x68/0xc0 [nfsd]
>> [Mon Mar 11 14:10:08 2024] __destroy_client+0x1f3/0x290 [nfsd]
>> [Mon Mar 11 14:10:08 2024] nfsd4_exchange_id+0x75f/0x770 [nfsd]
>> [Mon Mar 11 14:10:08 2024] ? nfsd4_decode_opaque+0x3a/0x90 [nfsd]
>> [Mon Mar 11 14:10:08 2024] nfsd4_proc_compound+0x44b/0x700 [nfsd]
>> [Mon Mar 11 14:10:08 2024] nfsd_dispatch+0x94/0x1c0 [nfsd]
>> [Mon Mar 11 14:10:08 2024] svc_process_common+0x2ec/0x660 [sunrpc]
>> [Mon Mar 11 14:10:08 2024] ? __pfx_nfsd_dispatch+0x10/0x10 [nfsd]
>> [Mon Mar 11 14:10:08 2024] ? __pfx_nfsd+0x10/0x10 [nfsd]
>> [Mon Mar 11 14:10:08 2024] svc_process+0x12d/0x170 [sunrpc]
>> [Mon Mar 11 14:10:08 2024] nfsd+0x84/0xb0 [nfsd]
>> [Mon Mar 11 14:10:08 2024] kthread+0xdd/0x100
>> [Mon Mar 11 14:10:08 2024] ? __pfx_kthread+0x10/0x10
>> [Mon Mar 11 14:10:08 2024] ret_from_fork+0x29/0x50
>> [Mon Mar 11 14:10:08 2024] </TASK>
>> [Mon Mar 11 14:10:08 2024] INFO: task nfsd:8866 blocked for more than 122 seconds.
>> [Mon Mar 11 14:10:08 2024] Not tainted 5.14.0-419.el9.x86_64 #1
>> [Mon Mar 11 14:10:08 2024] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
>> [Mon Mar 11 14:10:08 2024] task:nfsd state:D stack:0 pid:8866 ppid:2 flags:0x00004000
>> [Mon Mar 11 14:10:08 2024] Call Trace:
>> [Mon Mar 11 14:10:08 2024] <TASK>
>> [Mon Mar 11 14:10:08 2024] __schedule+0x21b/0x550
>> [Mon Mar 11 14:10:08 2024] schedule+0x2d/0x70
>> [Mon Mar 11 14:10:08 2024] schedule_timeout+0x11f/0x160
>> [Mon Mar 11 14:10:08 2024] ? select_idle_sibling+0x28/0x430
>> [Mon Mar 11 14:10:08 2024] ? tcp_recvmsg+0x196/0x210
>> [Mon Mar 11 14:10:08 2024] ? wake_affine+0x62/0x1f0
>> [Mon Mar 11 14:10:08 2024] __wait_for_common+0x90/0x1d0
>> [Mon Mar 11 14:10:08 2024] ? __pfx_schedule_timeout+0x10/0x10
>> [Mon Mar 11 14:10:08 2024] __flush_workqueue+0x13a/0x3f0
>> [Mon Mar 11 14:10:08 2024] nfsd4_destroy_session+0x1a4/0x240 [nfsd]
>> [Mon Mar 11 14:10:08 2024] nfsd4_proc_compound+0x44b/0x700 [nfsd]
>> [Mon Mar 11 14:10:08 2024] nfsd_dispatch+0x94/0x1c0 [nfsd]
>> [Mon Mar 11 14:10:08 2024] svc_process_common+0x2ec/0x660 [sunrpc]
>> [Mon Mar 11 14:10:08 2024] ? __pfx_nfsd_dispatch+0x10/0x10 [nfsd]
>> [Mon Mar 11 14:10:08 2024] ? __pfx_nfsd+0x10/0x10 [nfsd]
>> [Mon Mar 11 14:10:08 2024] svc_process+0x12d/0x170 [sunrpc]
>> [Mon Mar 11 14:10:08 2024] nfsd+0x84/0xb0 [nfsd]
>> [Mon Mar 11 14:10:08 2024] kthread+0xdd/0x100
>> [Mon Mar 11 14:10:08 2024] ? __pfx_kthread+0x10/0x10
>> [Mon Mar 11 14:10:08 2024] ret_from_fork+0x29/0x50
>> [Mon Mar 11 14:10:08 2024] </TASK>
>>
> The above threads are trying to flush the workqueue, so that probably
> means that they are stuck waiting on a workqueue job to finish.
>> The above is repeated a few times, and then this warning is also logged:
>>
>> [Mon Mar 11 14:12:04 2024] ------------[ cut here ]------------
>> [Mon Mar 11 14:12:04 2024] WARNING: CPU: 39 PID: 8844 at fs/nfsd/nfs4state.c:4919 nfsd_break_deleg_cb+0x170/0x190 [nfsd]
>> [Mon Mar 11 14:12:05 2024] Modules linked in: nfsv4 dns_resolver nfs fscache netfs rpcsec_gss_krb5 rpcrdma rdma_cm iw_cm ib_cm ib_core binfmt_misc bonding tls rfkill nft_counter nft_ct
>> nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_reject_inet nf_reject_ipv4 nf_reject_ipv6 nft_reject nf_tables nfnetlink vfat fat dm_thin_pool dm_persistent_data dm_bio_prison dm_bufio l
>> ibcrc32c dm_service_time dm_multipath intel_rapl_msr intel_rapl_common intel_uncore_frequency intel_uncore_frequency_common isst_if_common skx_edac nfit libnvdimm ipmi_ssif x86_pkg_temp
>> _thermal intel_powerclamp coretemp kvm_intel kvm irqbypass dcdbas rapl intel_cstate mgag200 i2c_algo_bit drm_shmem_helper dell_smbios drm_kms_helper dell_wmi_descriptor wmi_bmof intel_u
>> ncore syscopyarea pcspkr sysfillrect mei_me sysimgblt acpi_ipmi mei fb_sys_fops i2c_i801 ipmi_si intel_pch_thermal lpc_ich ipmi_devintf i2c_smbus ipmi_msghandler joydev acpi_power_meter
>> nfsd auth_rpcgss nfs_acl drm lockd grace fuse sunrpc ext4 mbcache jbd2 sd_mod sg lpfc
>> [Mon Mar 11 14:12:05 2024] nvmet_fc nvmet nvme_fc nvme_fabrics crct10dif_pclmul ahci libahci crc32_pclmul nvme_core crc32c_intel ixgbe megaraid_sas libata nvme_common ghash_clmulni_int
>> el t10_pi wdat_wdt scsi_transport_fc mdio wmi dca dm_mirror dm_region_hash dm_log dm_mod
>> [Mon Mar 11 14:12:05 2024] CPU: 39 PID: 8844 Comm: nfsd Not tainted 5.14.0-419.el9.x86_64 #1
>> [Mon Mar 11 14:12:05 2024] Hardware name: Dell Inc. PowerEdge R740/00WGD1, BIOS 2.20.1 09/13/2023
>> [Mon Mar 11 14:12:05 2024] RIP: 0010:nfsd_break_deleg_cb+0x170/0x190 [nfsd]
>> [Mon Mar 11 14:12:05 2024] Code: a6 95 c5 f3 e9 ff fe ff ff 48 89 df be 01 00 00 00 e8 34 b5 13 f4 48 8d bb 98 00 00 00 e8 c8 f9 00 00 84 c0 0f 85 2e ff ff ff <0f> 0b e9 27 ff ff ff be
>> 02 00 00 00 48 89 df e8 0c b5 13 f4 e9 01
>> [Mon Mar 11 14:12:05 2024] RSP: 0018:ffff9929e0bb7b80 EFLAGS: 00010246
>> [Mon Mar 11 14:12:05 2024] RAX: 0000000000000000 RBX: ffff8ada51930900 RCX: 0000000000000024
>> [Mon Mar 11 14:12:05 2024] RDX: ffff8ada519309c8 RSI: ffff8ad582933c00 RDI: 0000000000002000
>> [Mon Mar 11 14:12:05 2024] RBP: ffff8ad46bf21574 R08: ffff9929e0bb7b48 R09: 0000000000000000
>> [Mon Mar 11 14:12:05 2024] R10: ffff8aec859a2948 R11: 0000000000000000 R12: ffff8ad6f497c360
>> [Mon Mar 11 14:12:05 2024] R13: ffff8ad46bf21560 R14: ffff8ae5942e0b10 R15: ffff8ad6f497c360
>> [Mon Mar 11 14:12:05 2024] FS: 0000000000000000(0000) GS:ffff8b031fcc0000(0000) knlGS:0000000000000000
>> [Mon Mar 11 14:12:05 2024] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
>> [Mon Mar 11 14:12:05 2024] CR2: 00007fafe2060744 CR3: 00000018e58de006 CR4: 00000000007706e0
>> [Mon Mar 11 14:12:05 2024] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
>> [Mon Mar 11 14:12:05 2024] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
>> [Mon Mar 11 14:12:05 2024] PKRU: 55555554
>> [Mon Mar 11 14:12:05 2024] Call Trace:
>> [Mon Mar 11 14:12:05 2024] <TASK>
>> [Mon Mar 11 14:12:05 2024] ? show_trace_log_lvl+0x1c4/0x2df
>> [Mon Mar 11 14:12:05 2024] ? show_trace_log_lvl+0x1c4/0x2df
>> [Mon Mar 11 14:12:05 2024] ? __break_lease+0x16f/0x5f0
>> [Mon Mar 11 14:12:05 2024] ? nfsd_break_deleg_cb+0x170/0x190 [nfsd]
>> [Mon Mar 11 14:12:05 2024] ? __warn+0x81/0x110
>> [Mon Mar 11 14:12:05 2024] ? nfsd_break_deleg_cb+0x170/0x190 [nfsd]
>> [Mon Mar 11 14:12:05 2024] ? report_bug+0x10a/0x140
>> [Mon Mar 11 14:12:05 2024] ? handle_bug+0x3c/0x70
>> [Mon Mar 11 14:12:05 2024] ? exc_invalid_op+0x14/0x70
>> [Mon Mar 11 14:12:05 2024] ? asm_exc_invalid_op+0x16/0x20
>> [Mon Mar 11 14:12:05 2024] ? nfsd_break_deleg_cb+0x170/0x190 [nfsd]
>> [Mon Mar 11 14:12:05 2024] __break_lease+0x16f/0x5f0
>> [Mon Mar 11 14:12:05 2024] ? nfsd_file_lookup_locked+0x117/0x160 [nfsd]
>> [Mon Mar 11 14:12:05 2024] ? list_lru_del+0x101/0x150
>> [Mon Mar 11 14:12:05 2024] nfsd_file_do_acquire+0x790/0x830 [nfsd]
>> [Mon Mar 11 14:12:05 2024] nfs4_get_vfs_file+0x315/0x3a0 [nfsd]
>> [Mon Mar 11 14:12:05 2024] nfsd4_process_open2+0x430/0xa30 [nfsd]
>> [Mon Mar 11 14:12:05 2024] ? fh_verify+0x297/0x2f0 [nfsd]
>> [Mon Mar 11 14:12:05 2024] nfsd4_open+0x3ce/0x4b0 [nfsd]
>> [Mon Mar 11 14:12:05 2024] nfsd4_proc_compound+0x44b/0x700 [nfsd]
>> [Mon Mar 11 14:12:05 2024] nfsd_dispatch+0x94/0x1c0 [nfsd]
>> [Mon Mar 11 14:12:05 2024] svc_process_common+0x2ec/0x660 [sunrpc]
>> [Mon Mar 11 14:12:05 2024] ? __pfx_nfsd_dispatch+0x10/0x10 [nfsd]
>> [Mon Mar 11 14:12:05 2024] ? __pfx_nfsd+0x10/0x10 [nfsd]
>> [Mon Mar 11 14:12:05 2024] svc_process+0x12d/0x170 [sunrpc]
>> [Mon Mar 11 14:12:05 2024] nfsd+0x84/0xb0 [nfsd]
>> [Mon Mar 11 14:12:05 2024] kthread+0xdd/0x100
>> [Mon Mar 11 14:12:05 2024] ? __pfx_kthread+0x10/0x10
>> [Mon Mar 11 14:12:05 2024] ret_from_fork+0x29/0x50
>> [Mon Mar 11 14:12:05 2024] </TASK>
>> [Mon Mar 11 14:12:05 2024] ---[ end trace 7a039e17443dc651 ]---
> This is probably this WARN in nfsd_break_one_deleg:
>
> WARN_ON_ONCE(!nfsd4_run_cb(&dp->dl_recall));
>
> It means that a delegation break callback to the client couldn't be
> queued to the workqueue, and so it didn't run.
>
>> Could this be the same issue as described here:https://lore.kernel.org/linux-nfs/af0ec881-5ebf-4feb-98ae-3ed2a77f86f1@oracle.com/ ?
>>
> Yes, most likely the same problem.
If I read that thread correctly, this issue was introduced between
6.1.63 and 6.6.3? Is it possible the EL9 5.14.0-362.18.1.el9_3
backported these changes, or were we hitting some other bug with that
version? It seems the 6.1.x kernel is not affected? If so, that would be
the recommended kernel to run?
>
>
>> As described in that thread, I've tried to obtain the requested information.
>>
>>
>> Is it possible this is the issue that was fixed by the patches described here? https://lore.kernel.org/linux-nfs/2024022054-cause-suffering-eae8@gregkh/
>>
> Doubtful. Those are targeted toward a different set of issues.
>
> If you're willing, I do have some patches queued up for CentOS here that
> fix some backchannel problems that could be related. I'm mainly waiting
> on Chuck to send these to Linus and then we'll likely merge them into
> CentOS soon afterward:
>
> https://gitlab.com/redhat/centos-stream/src/kernel/centos-stream-9/-/merge_requests/3689
>
If you can send me a patch file, I can rebuild the C9S kernel with that
patch and run it. It can take a while for the bug to trigger as I
believe it seems to be very workload dependent (we were running very
stable for months and now hit this bug every other week).
So these patches are not yet upstream?
Regards,
Rik
--
Rik Theys
System Engineer
KU Leuven - Dept. Elektrotechniek (ESAT)
Kasteelpark Arenberg 10 bus 2440 - B-3001 Leuven-Heverlee
+32(0)16/32.11.07
----------------------------------------------------------------
<<Any errors in spelling, tact or fact are transmission errors>>
^ permalink raw reply [flat|nested] 26+ messages in thread
* Re: nfsd hangs and nfsd_break_deleg_cb+0x170/0x190 warning
2024-03-12 12:24 nfsd hangs and nfsd_break_deleg_cb+0x170/0x190 warning Rik Theys
@ 2024-03-12 12:47 ` Jeff Layton
2024-03-18 20:21 ` Rik Theys
0 siblings, 1 reply; 26+ messages in thread
From: Jeff Layton @ 2024-03-12 12:47 UTC (permalink / raw)
To: Rik Theys, Linux Nfs
On Tue, 2024-03-12 at 13:24 +0100, Rik Theys wrote:
> Hi Jeff,
>
> On 3/12/24 12:22, Jeff Layton wrote:
> > On Mon, 2024-03-11 at 19:43 +0100, Rik Theys wrote:
> > > Since a few weeks our Rocky Linux 9 NFS server has periodically logged hung nfsd tasks. The initial effect was that some clients could no longer access the NFS server. This got worse and worse (probably as more nfsd threads got blocked) and we had to restart the server. Restarting the server also failed as the NFS server service could no longer be stopped.
> > >
> > >
> > > The initial kernel we noticed this behavior on was kernel-5.14.0-362.18.1.el9_3.x86_64. Since then we've installed kernel-5.14.0-419.el9.x86_64 from CentOS Stream 9. The same issue happened again on this newer kernel version:
> > >
419 is fairly up to date with nfsd changes. There are some known bugs
around callbacks, and there is a draft MR in flight to fix it.
What kernel were you on prior to 5.14.0-362.18.1.el9_3.x86_64 ? If we
can bracket the changes around a particular version, then that might
help identify the problem.
> > > [Mon Mar 11 14:10:08 2024] Not tainted 5.14.0-419.el9.x86_64 #1
> > > [Mon Mar 11 14:10:08 2024] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> > > [Mon Mar 11 14:10:08 2024] task:nfsd state:D stack:0 pid:8865 ppid:2 flags:0x00004000
> > > [Mon Mar 11 14:10:08 2024] Call Trace:
> > > [Mon Mar 11 14:10:08 2024] <TASK>
> > > [Mon Mar 11 14:10:08 2024] __schedule+0x21b/0x550
> > > [Mon Mar 11 14:10:08 2024] schedule+0x2d/0x70
> > > [Mon Mar 11 14:10:08 2024] schedule_timeout+0x11f/0x160
> > > [Mon Mar 11 14:10:08 2024] ? select_idle_sibling+0x28/0x430
> > > [Mon Mar 11 14:10:08 2024] ? wake_affine+0x62/0x1f0
> > > [Mon Mar 11 14:10:08 2024] __wait_for_common+0x90/0x1d0
> > > [Mon Mar 11 14:10:08 2024] ? __pfx_schedule_timeout+0x10/0x10
> > > [Mon Mar 11 14:10:08 2024] __flush_workqueue+0x13a/0x3f0
> > > [Mon Mar 11 14:10:08 2024] nfsd4_shutdown_callback+0x49/0x120 [nfsd]
> > > [Mon Mar 11 14:10:08 2024] ? nfsd4_cld_remove+0x54/0x1d0 [nfsd]
> > > [Mon Mar 11 14:10:08 2024] ? nfsd4_return_all_client_layouts+0xc4/0xf0 [nfsd]
> > > [Mon Mar 11 14:10:08 2024] ? nfsd4_shutdown_copy+0x68/0xc0 [nfsd]
> > > [Mon Mar 11 14:10:08 2024] __destroy_client+0x1f3/0x290 [nfsd]
> > > [Mon Mar 11 14:10:08 2024] nfsd4_exchange_id+0x75f/0x770 [nfsd]
> > > [Mon Mar 11 14:10:08 2024] ? nfsd4_decode_opaque+0x3a/0x90 [nfsd]
> > > [Mon Mar 11 14:10:08 2024] nfsd4_proc_compound+0x44b/0x700 [nfsd]
> > > [Mon Mar 11 14:10:08 2024] nfsd_dispatch+0x94/0x1c0 [nfsd]
> > > [Mon Mar 11 14:10:08 2024] svc_process_common+0x2ec/0x660 [sunrpc]
> > > [Mon Mar 11 14:10:08 2024] ? __pfx_nfsd_dispatch+0x10/0x10 [nfsd]
> > > [Mon Mar 11 14:10:08 2024] ? __pfx_nfsd+0x10/0x10 [nfsd]
> > > [Mon Mar 11 14:10:08 2024] svc_process+0x12d/0x170 [sunrpc]
> > > [Mon Mar 11 14:10:08 2024] nfsd+0x84/0xb0 [nfsd]
> > > [Mon Mar 11 14:10:08 2024] kthread+0xdd/0x100
> > > [Mon Mar 11 14:10:08 2024] ? __pfx_kthread+0x10/0x10
> > > [Mon Mar 11 14:10:08 2024] ret_from_fork+0x29/0x50
> > > [Mon Mar 11 14:10:08 2024] </TASK>
> > > [Mon Mar 11 14:10:08 2024] INFO: task nfsd:8866 blocked for more than 122 seconds.
> > > [Mon Mar 11 14:10:08 2024] Not tainted 5.14.0-419.el9.x86_64 #1
> > > [Mon Mar 11 14:10:08 2024] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> > > [Mon Mar 11 14:10:08 2024] task:nfsd state:D stack:0 pid:8866 ppid:2 flags:0x00004000
> > > [Mon Mar 11 14:10:08 2024] Call Trace:
> > > [Mon Mar 11 14:10:08 2024] <TASK>
> > > [Mon Mar 11 14:10:08 2024] __schedule+0x21b/0x550
> > > [Mon Mar 11 14:10:08 2024] schedule+0x2d/0x70
> > > [Mon Mar 11 14:10:08 2024] schedule_timeout+0x11f/0x160
> > > [Mon Mar 11 14:10:08 2024] ? select_idle_sibling+0x28/0x430
> > > [Mon Mar 11 14:10:08 2024] ? tcp_recvmsg+0x196/0x210
> > > [Mon Mar 11 14:10:08 2024] ? wake_affine+0x62/0x1f0
> > > [Mon Mar 11 14:10:08 2024] __wait_for_common+0x90/0x1d0
> > > [Mon Mar 11 14:10:08 2024] ? __pfx_schedule_timeout+0x10/0x10
> > > [Mon Mar 11 14:10:08 2024] __flush_workqueue+0x13a/0x3f0
> > > [Mon Mar 11 14:10:08 2024] nfsd4_destroy_session+0x1a4/0x240 [nfsd]
> > > [Mon Mar 11 14:10:08 2024] nfsd4_proc_compound+0x44b/0x700 [nfsd]
> > > [Mon Mar 11 14:10:08 2024] nfsd_dispatch+0x94/0x1c0 [nfsd]
> > > [Mon Mar 11 14:10:08 2024] svc_process_common+0x2ec/0x660 [sunrpc]
> > > [Mon Mar 11 14:10:08 2024] ? __pfx_nfsd_dispatch+0x10/0x10 [nfsd]
> > > [Mon Mar 11 14:10:08 2024] ? __pfx_nfsd+0x10/0x10 [nfsd]
> > > [Mon Mar 11 14:10:08 2024] svc_process+0x12d/0x170 [sunrpc]
> > > [Mon Mar 11 14:10:08 2024] nfsd+0x84/0xb0 [nfsd]
> > > [Mon Mar 11 14:10:08 2024] kthread+0xdd/0x100
> > > [Mon Mar 11 14:10:08 2024] ? __pfx_kthread+0x10/0x10
> > > [Mon Mar 11 14:10:08 2024] ret_from_fork+0x29/0x50
> > > [Mon Mar 11 14:10:08 2024] </TASK>
> > >
> > The above threads are trying to flush the workqueue, so that probably
> > means that they are stuck waiting on a workqueue job to finish.
> > > The above is repeated a few times, and then this warning is also logged:
> > >
> > > [Mon Mar 11 14:12:04 2024] ------------[ cut here ]------------
> > > [Mon Mar 11 14:12:04 2024] WARNING: CPU: 39 PID: 8844 at fs/nfsd/nfs4state.c:4919 nfsd_break_deleg_cb+0x170/0x190 [nfsd]
> > > [Mon Mar 11 14:12:05 2024] Modules linked in: nfsv4 dns_resolver nfs fscache netfs rpcsec_gss_krb5 rpcrdma rdma_cm iw_cm ib_cm ib_core binfmt_misc bonding tls rfkill nft_counter nft_ct
> > > nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_reject_inet nf_reject_ipv4 nf_reject_ipv6 nft_reject nf_tables nfnetlink vfat fat dm_thin_pool dm_persistent_data dm_bio_prison dm_bufio l
> > > ibcrc32c dm_service_time dm_multipath intel_rapl_msr intel_rapl_common intel_uncore_frequency intel_uncore_frequency_common isst_if_common skx_edac nfit libnvdimm ipmi_ssif x86_pkg_temp
> > > _thermal intel_powerclamp coretemp kvm_intel kvm irqbypass dcdbas rapl intel_cstate mgag200 i2c_algo_bit drm_shmem_helper dell_smbios drm_kms_helper dell_wmi_descriptor wmi_bmof intel_u
> > > ncore syscopyarea pcspkr sysfillrect mei_me sysimgblt acpi_ipmi mei fb_sys_fops i2c_i801 ipmi_si intel_pch_thermal lpc_ich ipmi_devintf i2c_smbus ipmi_msghandler joydev acpi_power_meter
> > > nfsd auth_rpcgss nfs_acl drm lockd grace fuse sunrpc ext4 mbcache jbd2 sd_mod sg lpfc
> > > [Mon Mar 11 14:12:05 2024] nvmet_fc nvmet nvme_fc nvme_fabrics crct10dif_pclmul ahci libahci crc32_pclmul nvme_core crc32c_intel ixgbe megaraid_sas libata nvme_common ghash_clmulni_int
> > > el t10_pi wdat_wdt scsi_transport_fc mdio wmi dca dm_mirror dm_region_hash dm_log dm_mod
> > > [Mon Mar 11 14:12:05 2024] CPU: 39 PID: 8844 Comm: nfsd Not tainted 5.14.0-419.el9.x86_64 #1
> > > [Mon Mar 11 14:12:05 2024] Hardware name: Dell Inc. PowerEdge R740/00WGD1, BIOS 2.20.1 09/13/2023
> > > [Mon Mar 11 14:12:05 2024] RIP: 0010:nfsd_break_deleg_cb+0x170/0x190 [nfsd]
> > > [Mon Mar 11 14:12:05 2024] Code: a6 95 c5 f3 e9 ff fe ff ff 48 89 df be 01 00 00 00 e8 34 b5 13 f4 48 8d bb 98 00 00 00 e8 c8 f9 00 00 84 c0 0f 85 2e ff ff ff <0f> 0b e9 27 ff ff ff be
> > > 02 00 00 00 48 89 df e8 0c b5 13 f4 e9 01
> > > [Mon Mar 11 14:12:05 2024] RSP: 0018:ffff9929e0bb7b80 EFLAGS: 00010246
> > > [Mon Mar 11 14:12:05 2024] RAX: 0000000000000000 RBX: ffff8ada51930900 RCX: 0000000000000024
> > > [Mon Mar 11 14:12:05 2024] RDX: ffff8ada519309c8 RSI: ffff8ad582933c00 RDI: 0000000000002000
> > > [Mon Mar 11 14:12:05 2024] RBP: ffff8ad46bf21574 R08: ffff9929e0bb7b48 R09: 0000000000000000
> > > [Mon Mar 11 14:12:05 2024] R10: ffff8aec859a2948 R11: 0000000000000000 R12: ffff8ad6f497c360
> > > [Mon Mar 11 14:12:05 2024] R13: ffff8ad46bf21560 R14: ffff8ae5942e0b10 R15: ffff8ad6f497c360
> > > [Mon Mar 11 14:12:05 2024] FS: 0000000000000000(0000) GS:ffff8b031fcc0000(0000) knlGS:0000000000000000
> > > [Mon Mar 11 14:12:05 2024] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> > > [Mon Mar 11 14:12:05 2024] CR2: 00007fafe2060744 CR3: 00000018e58de006 CR4: 00000000007706e0
> > > [Mon Mar 11 14:12:05 2024] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
> > > [Mon Mar 11 14:12:05 2024] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
> > > [Mon Mar 11 14:12:05 2024] PKRU: 55555554
> > > [Mon Mar 11 14:12:05 2024] Call Trace:
> > > [Mon Mar 11 14:12:05 2024] <TASK>
> > > [Mon Mar 11 14:12:05 2024] ? show_trace_log_lvl+0x1c4/0x2df
> > > [Mon Mar 11 14:12:05 2024] ? show_trace_log_lvl+0x1c4/0x2df
> > > [Mon Mar 11 14:12:05 2024] ? __break_lease+0x16f/0x5f0
> > > [Mon Mar 11 14:12:05 2024] ? nfsd_break_deleg_cb+0x170/0x190 [nfsd]
> > > [Mon Mar 11 14:12:05 2024] ? __warn+0x81/0x110
> > > [Mon Mar 11 14:12:05 2024] ? nfsd_break_deleg_cb+0x170/0x190 [nfsd]
> > > [Mon Mar 11 14:12:05 2024] ? report_bug+0x10a/0x140
> > > [Mon Mar 11 14:12:05 2024] ? handle_bug+0x3c/0x70
> > > [Mon Mar 11 14:12:05 2024] ? exc_invalid_op+0x14/0x70
> > > [Mon Mar 11 14:12:05 2024] ? asm_exc_invalid_op+0x16/0x20
> > > [Mon Mar 11 14:12:05 2024] ? nfsd_break_deleg_cb+0x170/0x190 [nfsd]
> > > [Mon Mar 11 14:12:05 2024] __break_lease+0x16f/0x5f0
> > > [Mon Mar 11 14:12:05 2024] ? nfsd_file_lookup_locked+0x117/0x160 [nfsd]
> > > [Mon Mar 11 14:12:05 2024] ? list_lru_del+0x101/0x150
> > > [Mon Mar 11 14:12:05 2024] nfsd_file_do_acquire+0x790/0x830 [nfsd]
> > > [Mon Mar 11 14:12:05 2024] nfs4_get_vfs_file+0x315/0x3a0 [nfsd]
> > > [Mon Mar 11 14:12:05 2024] nfsd4_process_open2+0x430/0xa30 [nfsd]
> > > [Mon Mar 11 14:12:05 2024] ? fh_verify+0x297/0x2f0 [nfsd]
> > > [Mon Mar 11 14:12:05 2024] nfsd4_open+0x3ce/0x4b0 [nfsd]
> > > [Mon Mar 11 14:12:05 2024] nfsd4_proc_compound+0x44b/0x700 [nfsd]
> > > [Mon Mar 11 14:12:05 2024] nfsd_dispatch+0x94/0x1c0 [nfsd]
> > > [Mon Mar 11 14:12:05 2024] svc_process_common+0x2ec/0x660 [sunrpc]
> > > [Mon Mar 11 14:12:05 2024] ? __pfx_nfsd_dispatch+0x10/0x10 [nfsd]
> > > [Mon Mar 11 14:12:05 2024] ? __pfx_nfsd+0x10/0x10 [nfsd]
> > > [Mon Mar 11 14:12:05 2024] svc_process+0x12d/0x170 [sunrpc]
> > > [Mon Mar 11 14:12:05 2024] nfsd+0x84/0xb0 [nfsd]
> > > [Mon Mar 11 14:12:05 2024] kthread+0xdd/0x100
> > > [Mon Mar 11 14:12:05 2024] ? __pfx_kthread+0x10/0x10
> > > [Mon Mar 11 14:12:05 2024] ret_from_fork+0x29/0x50
> > > [Mon Mar 11 14:12:05 2024] </TASK>
> > > [Mon Mar 11 14:12:05 2024] ---[ end trace 7a039e17443dc651 ]---
> > This is probably this WARN in nfsd_break_one_deleg:
> >
> > WARN_ON_ONCE(!nfsd4_run_cb(&dp->dl_recall));
> >
> > It means that a delegation break callback to the client couldn't be
> > queued to the workqueue, and so it didn't run.
> >
> > > Could this be the same issue as described here:https://lore.kernel.org/linux-nfs/af0ec881-5ebf-4feb-98ae-3ed2a77f86f1@oracle.com/ ?
> > >
> > Yes, most likely the same problem.
> If I read that thread correctly, this issue was introduced between
> 6.1.63 and 6.6.3? Is it possible the EL9 5.14.0-362.18.1.el9_3
> backported these changes, or were we hitting some other bug with that
> version? It seems the 6.1.x kernel is not affected? If so, that would be
> the recommended kernel to run?
Anything is possible. We have to identify the problem first.
>
> >
> > > As described in that thread, I've tried to obtain the requested information.
> > >
> > >
> > > Is it possible this is the issue that was fixed by the patches described here? https://lore.kernel.org/linux-nfs/2024022054-cause-suffering-eae8@gregkh/
> > >
> > Doubtful. Those are targeted toward a different set of issues.
> >
> > If you're willing, I do have some patches queued up for CentOS here that
> > fix some backchannel problems that could be related. I'm mainly waiting
> > on Chuck to send these to Linus and then we'll likely merge them into
> > CentOS soon afterward:
> >
> > https://gitlab.com/redhat/centos-stream/src/kernel/centos-stream-9/-/merge_requests/3689
> >
> If you can send me a patch file, I can rebuild the C9S kernel with that
> patch and run it. It can take a while for the bug to trigger as I
> believe it seems to be very workload dependent (we were running very
> stable for months and now hit this bug every other week).
>
>
It's probably simpler to just pull down the build artifacts for that MR.
You have to drill down through the CI for it, but they are here:
https://s3.amazonaws.com/arr-cki-prod-trusted-artifacts/index.html?prefix=trusted-artifacts/1194300175/publish_x86_64/6278921877/artifacts/
There's even a repo file you can install on the box to pull them down.
--
Jeff Layton <jlayton@kernel.org>
^ permalink raw reply [flat|nested] 26+ messages in thread
* Re: nfsd hangs and nfsd_break_deleg_cb+0x170/0x190 warning
2024-03-12 12:47 ` Jeff Layton
@ 2024-03-18 20:21 ` Rik Theys
2024-03-18 21:15 ` Rik Theys
0 siblings, 1 reply; 26+ messages in thread
From: Rik Theys @ 2024-03-18 20:21 UTC (permalink / raw)
To: Jeff Layton, Linux Nfs
Hi Jeff,
On 3/12/24 13:47, Jeff Layton wrote:
> On Tue, 2024-03-12 at 13:24 +0100, Rik Theys wrote:
>> Hi Jeff,
>>
>> On 3/12/24 12:22, Jeff Layton wrote:
>>> On Mon, 2024-03-11 at 19:43 +0100, Rik Theys wrote:
>>>> Since a few weeks our Rocky Linux 9 NFS server has periodically logged hung nfsd tasks. The initial effect was that some clients could no longer access the NFS server. This got worse and worse (probably as more nfsd threads got blocked) and we had to restart the server. Restarting the server also failed as the NFS server service could no longer be stopped.
>>>>
>>>>
>>>> The initial kernel we noticed this behavior on was kernel-5.14.0-362.18.1.el9_3.x86_64. Since then we've installed kernel-5.14.0-419.el9.x86_64 from CentOS Stream 9. The same issue happened again on this newer kernel version:
>>>>
> 419 is fairly up to date with nfsd changes. There are some known bugs
> around callbacks, and there is a draft MR in flight to fix it.
>
> What kernel were you on prior to 5.14.0-362.18.1.el9_3.x86_64 ? If we
> can bracket the changes around a particular version, then that might
> help identify the problem.
>
>>>> [Mon Mar 11 14:10:08 2024] Not tainted 5.14.0-419.el9.x86_64 #1
>>>> [Mon Mar 11 14:10:08 2024] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
>>>> [Mon Mar 11 14:10:08 2024]task:nfsd state:D stack:0 pid:8865 ppid:2 flags:0x00004000
>>>> [Mon Mar 11 14:10:08 2024] Call Trace:
>>>> [Mon Mar 11 14:10:08 2024] <TASK>
>>>> [Mon Mar 11 14:10:08 2024] __schedule+0x21b/0x550
>>>> [Mon Mar 11 14:10:08 2024] schedule+0x2d/0x70
>>>> [Mon Mar 11 14:10:08 2024] schedule_timeout+0x11f/0x160
>>>> [Mon Mar 11 14:10:08 2024] ? select_idle_sibling+0x28/0x430
>>>> [Mon Mar 11 14:10:08 2024] ? wake_affine+0x62/0x1f0
>>>> [Mon Mar 11 14:10:08 2024] __wait_for_common+0x90/0x1d0
>>>> [Mon Mar 11 14:10:08 2024] ? __pfx_schedule_timeout+0x10/0x10
>>>> [Mon Mar 11 14:10:08 2024] __flush_workqueue+0x13a/0x3f0
>>>> [Mon Mar 11 14:10:08 2024] nfsd4_shutdown_callback+0x49/0x120 [nfsd]
>>>> [Mon Mar 11 14:10:08 2024] ? nfsd4_cld_remove+0x54/0x1d0 [nfsd]
>>>> [Mon Mar 11 14:10:08 2024] ? nfsd4_return_all_client_layouts+0xc4/0xf0 [nfsd]
>>>> [Mon Mar 11 14:10:08 2024] ? nfsd4_shutdown_copy+0x68/0xc0 [nfsd]
>>>> [Mon Mar 11 14:10:08 2024] __destroy_client+0x1f3/0x290 [nfsd]
>>>> [Mon Mar 11 14:10:08 2024] nfsd4_exchange_id+0x75f/0x770 [nfsd]
>>>> [Mon Mar 11 14:10:08 2024] ? nfsd4_decode_opaque+0x3a/0x90 [nfsd]
>>>> [Mon Mar 11 14:10:08 2024] nfsd4_proc_compound+0x44b/0x700 [nfsd]
>>>> [Mon Mar 11 14:10:08 2024] nfsd_dispatch+0x94/0x1c0 [nfsd]
>>>> [Mon Mar 11 14:10:08 2024] svc_process_common+0x2ec/0x660 [sunrpc]
>>>> [Mon Mar 11 14:10:08 2024] ? __pfx_nfsd_dispatch+0x10/0x10 [nfsd]
>>>> [Mon Mar 11 14:10:08 2024] ? __pfx_nfsd+0x10/0x10 [nfsd]
>>>> [Mon Mar 11 14:10:08 2024] svc_process+0x12d/0x170 [sunrpc]
>>>> [Mon Mar 11 14:10:08 2024] nfsd+0x84/0xb0 [nfsd]
>>>> [Mon Mar 11 14:10:08 2024] kthread+0xdd/0x100
>>>> [Mon Mar 11 14:10:08 2024] ? __pfx_kthread+0x10/0x10
>>>> [Mon Mar 11 14:10:08 2024] ret_from_fork+0x29/0x50
>>>> [Mon Mar 11 14:10:08 2024] </TASK>
>>>> [Mon Mar 11 14:10:08 2024] INFO: task nfsd:8866 blocked for more than 122 seconds.
>>>> [Mon Mar 11 14:10:08 2024] Not tainted 5.14.0-419.el9.x86_64 #1
>>>> [Mon Mar 11 14:10:08 2024] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
>>>> [Mon Mar 11 14:10:08 2024]task:nfsd state:D stack:0 pid:8866 ppid:2 flags:0x00004000
>>>> [Mon Mar 11 14:10:08 2024] Call Trace:
>>>> [Mon Mar 11 14:10:08 2024] <TASK>
>>>> [Mon Mar 11 14:10:08 2024] __schedule+0x21b/0x550
>>>> [Mon Mar 11 14:10:08 2024] schedule+0x2d/0x70
>>>> [Mon Mar 11 14:10:08 2024] schedule_timeout+0x11f/0x160
>>>> [Mon Mar 11 14:10:08 2024] ? select_idle_sibling+0x28/0x430
>>>> [Mon Mar 11 14:10:08 2024] ? tcp_recvmsg+0x196/0x210
>>>> [Mon Mar 11 14:10:08 2024] ? wake_affine+0x62/0x1f0
>>>> [Mon Mar 11 14:10:08 2024] __wait_for_common+0x90/0x1d0
>>>> [Mon Mar 11 14:10:08 2024] ? __pfx_schedule_timeout+0x10/0x10
>>>> [Mon Mar 11 14:10:08 2024] __flush_workqueue+0x13a/0x3f0
>>>> [Mon Mar 11 14:10:08 2024] nfsd4_destroy_session+0x1a4/0x240 [nfsd]
>>>> [Mon Mar 11 14:10:08 2024] nfsd4_proc_compound+0x44b/0x700 [nfsd]
>>>> [Mon Mar 11 14:10:08 2024] nfsd_dispatch+0x94/0x1c0 [nfsd]
>>>> [Mon Mar 11 14:10:08 2024] svc_process_common+0x2ec/0x660 [sunrpc]
>>>> [Mon Mar 11 14:10:08 2024] ? __pfx_nfsd_dispatch+0x10/0x10 [nfsd]
>>>> [Mon Mar 11 14:10:08 2024] ? __pfx_nfsd+0x10/0x10 [nfsd]
>>>> [Mon Mar 11 14:10:08 2024] svc_process+0x12d/0x170 [sunrpc]
>>>> [Mon Mar 11 14:10:08 2024] nfsd+0x84/0xb0 [nfsd]
>>>> [Mon Mar 11 14:10:08 2024] kthread+0xdd/0x100
>>>> [Mon Mar 11 14:10:08 2024] ? __pfx_kthread+0x10/0x10
>>>> [Mon Mar 11 14:10:08 2024] ret_from_fork+0x29/0x50
>>>> [Mon Mar 11 14:10:08 2024] </TASK>
>>>>
>>> The above threads are trying to flush the workqueue, so that probably
>>> means that they are stuck waiting on a workqueue job to finish.
>>>> The above is repeated a few times, and then this warning is also logged:
>>>>
>>>> [Mon Mar 11 14:12:04 2024] ------------[ cut here ]------------
>>>> [Mon Mar 11 14:12:04 2024] WARNING: CPU: 39 PID: 8844 at fs/nfsd/nfs4state.c:4919 nfsd_break_deleg_cb+0x170/0x190 [nfsd]
>>>> [Mon Mar 11 14:12:05 2024] Modules linked in: nfsv4 dns_resolver nfs fscache netfs rpcsec_gss_krb5 rpcrdma rdma_cm iw_cm ib_cm ib_core binfmt_misc bonding tls rfkill nft_counter nft_ct
>>>> nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_reject_inet nf_reject_ipv4 nf_reject_ipv6 nft_reject nf_tables nfnetlink vfat fat dm_thin_pool dm_persistent_data dm_bio_prison dm_bufio l
>>>> ibcrc32c dm_service_time dm_multipath intel_rapl_msr intel_rapl_common intel_uncore_frequency intel_uncore_frequency_common isst_if_common skx_edac nfit libnvdimm ipmi_ssif x86_pkg_temp
>>>> _thermal intel_powerclamp coretemp kvm_intel kvm irqbypass dcdbas rapl intel_cstate mgag200 i2c_algo_bit drm_shmem_helper dell_smbios drm_kms_helper dell_wmi_descriptor wmi_bmof intel_u
>>>> ncore syscopyarea pcspkr sysfillrect mei_me sysimgblt acpi_ipmi mei fb_sys_fops i2c_i801 ipmi_si intel_pch_thermal lpc_ich ipmi_devintf i2c_smbus ipmi_msghandler joydev acpi_power_meter
>>>> nfsd auth_rpcgss nfs_acl drm lockd grace fuse sunrpc ext4 mbcache jbd2 sd_mod sg lpfc
>>>> [Mon Mar 11 14:12:05 2024] nvmet_fc nvmet nvme_fc nvme_fabrics crct10dif_pclmul ahci libahci crc32_pclmul nvme_core crc32c_intel ixgbe megaraid_sas libata nvme_common ghash_clmulni_int
>>>> el t10_pi wdat_wdt scsi_transport_fc mdio wmi dca dm_mirror dm_region_hash dm_log dm_mod
>>>> [Mon Mar 11 14:12:05 2024] CPU: 39 PID: 8844 Comm: nfsd Not tainted 5.14.0-419.el9.x86_64 #1
>>>> [Mon Mar 11 14:12:05 2024] Hardware name: Dell Inc. PowerEdge R740/00WGD1, BIOS 2.20.1 09/13/2023
>>>> [Mon Mar 11 14:12:05 2024] RIP: 0010:nfsd_break_deleg_cb+0x170/0x190 [nfsd]
>>>> [Mon Mar 11 14:12:05 2024] Code: a6 95 c5 f3 e9 ff fe ff ff 48 89 df be 01 00 00 00 e8 34 b5 13 f4 48 8d bb 98 00 00 00 e8 c8 f9 00 00 84 c0 0f 85 2e ff ff ff <0f> 0b e9 27 ff ff ff be
>>>> 02 00 00 00 48 89 df e8 0c b5 13 f4 e9 01
>>>> [Mon Mar 11 14:12:05 2024] RSP: 0018:ffff9929e0bb7b80 EFLAGS: 00010246
>>>> [Mon Mar 11 14:12:05 2024] RAX: 0000000000000000 RBX: ffff8ada51930900 RCX: 0000000000000024
>>>> [Mon Mar 11 14:12:05 2024] RDX: ffff8ada519309c8 RSI: ffff8ad582933c00 RDI: 0000000000002000
>>>> [Mon Mar 11 14:12:05 2024] RBP: ffff8ad46bf21574 R08: ffff9929e0bb7b48 R09: 0000000000000000
>>>> [Mon Mar 11 14:12:05 2024] R10: ffff8aec859a2948 R11: 0000000000000000 R12: ffff8ad6f497c360
>>>> [Mon Mar 11 14:12:05 2024] R13: ffff8ad46bf21560 R14: ffff8ae5942e0b10 R15: ffff8ad6f497c360
>>>> [Mon Mar 11 14:12:05 2024] FS: 0000000000000000(0000) GS:ffff8b031fcc0000(0000) knlGS:0000000000000000
>>>> [Mon Mar 11 14:12:05 2024] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
>>>> [Mon Mar 11 14:12:05 2024] CR2: 00007fafe2060744 CR3: 00000018e58de006 CR4: 00000000007706e0
>>>> [Mon Mar 11 14:12:05 2024] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
>>>> [Mon Mar 11 14:12:05 2024] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
>>>> [Mon Mar 11 14:12:05 2024] PKRU: 55555554
>>>> [Mon Mar 11 14:12:05 2024] Call Trace:
>>>> [Mon Mar 11 14:12:05 2024] <TASK>
>>>> [Mon Mar 11 14:12:05 2024] ? show_trace_log_lvl+0x1c4/0x2df
>>>> [Mon Mar 11 14:12:05 2024] ? show_trace_log_lvl+0x1c4/0x2df
>>>> [Mon Mar 11 14:12:05 2024] ? __break_lease+0x16f/0x5f0
>>>> [Mon Mar 11 14:12:05 2024] ? nfsd_break_deleg_cb+0x170/0x190 [nfsd]
>>>> [Mon Mar 11 14:12:05 2024] ? __warn+0x81/0x110
>>>> [Mon Mar 11 14:12:05 2024] ? nfsd_break_deleg_cb+0x170/0x190 [nfsd]
>>>> [Mon Mar 11 14:12:05 2024] ? report_bug+0x10a/0x140
>>>> [Mon Mar 11 14:12:05 2024] ? handle_bug+0x3c/0x70
>>>> [Mon Mar 11 14:12:05 2024] ? exc_invalid_op+0x14/0x70
>>>> [Mon Mar 11 14:12:05 2024] ? asm_exc_invalid_op+0x16/0x20
>>>> [Mon Mar 11 14:12:05 2024] ? nfsd_break_deleg_cb+0x170/0x190 [nfsd]
>>>> [Mon Mar 11 14:12:05 2024] __break_lease+0x16f/0x5f0
>>>> [Mon Mar 11 14:12:05 2024] ? nfsd_file_lookup_locked+0x117/0x160 [nfsd]
>>>> [Mon Mar 11 14:12:05 2024] ? list_lru_del+0x101/0x150
>>>> [Mon Mar 11 14:12:05 2024] nfsd_file_do_acquire+0x790/0x830 [nfsd]
>>>> [Mon Mar 11 14:12:05 2024] nfs4_get_vfs_file+0x315/0x3a0 [nfsd]
>>>> [Mon Mar 11 14:12:05 2024] nfsd4_process_open2+0x430/0xa30 [nfsd]
>>>> [Mon Mar 11 14:12:05 2024] ? fh_verify+0x297/0x2f0 [nfsd]
>>>> [Mon Mar 11 14:12:05 2024] nfsd4_open+0x3ce/0x4b0 [nfsd]
>>>> [Mon Mar 11 14:12:05 2024] nfsd4_proc_compound+0x44b/0x700 [nfsd]
>>>> [Mon Mar 11 14:12:05 2024] nfsd_dispatch+0x94/0x1c0 [nfsd]
>>>> [Mon Mar 11 14:12:05 2024] svc_process_common+0x2ec/0x660 [sunrpc]
>>>> [Mon Mar 11 14:12:05 2024] ? __pfx_nfsd_dispatch+0x10/0x10 [nfsd]
>>>> [Mon Mar 11 14:12:05 2024] ? __pfx_nfsd+0x10/0x10 [nfsd]
>>>> [Mon Mar 11 14:12:05 2024] svc_process+0x12d/0x170 [sunrpc]
>>>> [Mon Mar 11 14:12:05 2024] nfsd+0x84/0xb0 [nfsd]
>>>> [Mon Mar 11 14:12:05 2024] kthread+0xdd/0x100
>>>> [Mon Mar 11 14:12:05 2024] ? __pfx_kthread+0x10/0x10
>>>> [Mon Mar 11 14:12:05 2024] ret_from_fork+0x29/0x50
>>>> [Mon Mar 11 14:12:05 2024] </TASK>
>>>> [Mon Mar 11 14:12:05 2024] ---[ end trace 7a039e17443dc651 ]---
>>> This is probably this WARN in nfsd_break_one_deleg:
>>>
>>> WARN_ON_ONCE(!nfsd4_run_cb(&dp->dl_recall));
>>>
>>> It means that a delegation break callback to the client couldn't be
>>> queued to the workqueue, and so it didn't run.
>>>
>>>> Could this be the same issue as described here:https://lore.kernel.org/linux-nfs/af0ec881-5ebf-4feb-98ae-3ed2a77f86f1@oracle.com/ ?
>>>>
>>> Yes, most likely the same problem.
>> If I read that thread correctly, this issue was introduced between
>> 6.1.63 and 6.6.3? Is it possible the EL9 5.14.0-362.18.1.el9_3
>> backported these changes, or were we hitting some other bug with that
>> version? It seems the 6.1.x kernel is not affected? If so, that would be
>> the recommended kernel to run?
> Anything is possible. We have to identify the problem first.
>>>> As described in that thread, I've tried to obtain the requested information.
>>>>
>>>>
>>>> Is it possible this is the issue that was fixed by the patches described here?https://lore.kernel.org/linux-nfs/2024022054-cause-suffering-eae8@gregkh/
>>>>
>>> Doubtful. Those are targeted toward a different set of issues.
>>>
>>> If you're willing, I do have some patches queued up for CentOS here that
>>> fix some backchannel problems that could be related. I'm mainly waiting
>>> on Chuck to send these to Linus and then we'll likely merge them into
>>> CentOS soon afterward:
>>>
>>> https://gitlab.com/redhat/centos-stream/src/kernel/centos-stream-9/-/merge_requests/3689
>>>
>> If you can send me a patch file, I can rebuild the C9S kernel with that
>> patch and run it. It can take a while for the bug to trigger as I
>> believe it seems to be very workload dependent (we were running very
>> stable for months and now hit this bug every other week).
>>
>>
> It's probably simpler to just pull down the build artifacts for that MR.
> You have to drill down through the CI for it, but they are here:
>
> https://s3.amazonaws.com/arr-cki-prod-trusted-artifacts/index.html?prefix=trusted-artifacts/1194300175/publish_x86_64/6278921877/artifacts/
>
> There's even a repo file you can install on the box to pull them down.
We installed this kernel on the server 3 days ago. Today, a user
informed us that their screen was black after logging in. Similar to
other occurrences of this issue, the mount command on the client was
hung. But in contrast to the other times, there were no messages in the
logs kernel logs on the server. Even restarting the client does not
resolve the issue.
Something still seems to be wrong on the server though. When I look at
the directories under /proc/fs/nfsd/clients, there's still a directory
for the specific client, even though it's no longer running:
# cat 155/info
clientid: 0xc8edb7f65f4a9ad
address: "10.87.31.152:819"
status: confirmed
seconds from last renew: 33163
name: "Linux NFSv4.2 bersalis.esat.kuleuven.be"
minor version: 2
Implementation domain: "kernel.org"
Implementation name: "Linux 5.14.0-362.18.1.el9_3.0.1.x86_64 #1 SMP
PREEMPT_DYNAMIC Sun Feb 11 13:49:23 UTC 2024 x86_64"
Implementation time: [0, 0]
callback state: DOWN
callback address: 10.87.31.152:0
The system seems to have identified that the client is no longer
reachable, but the client entry does not go away. When a mount was
hanging on the client, there would be two directories in clients for the
same client. Killing the mount command clears up the second entry.
Even after running conntrack -D on the server to remove the tcp
connection from the conntrack table, the entry doesn't go away and the
client still can not mount anything from the server.
A tcpdump on the client while a mount was running logged the following
messages over and over again:
request:
Frame 1: 378 bytes on wire (3024 bits), 378 bytes captured (3024 bits)
Ethernet II, Src: HP_19:7d:4b (e0:73:e7:19:7d:4b), Dst:
ArubaaHe_f9:8e:00 (88:3a:30:f9:8e:00)
Internet Protocol Version 4, Src: 10.87.31.152, Dst: 10.86.18.14
Transmission Control Protocol, Src Port: 932, Dst Port: 2049, Seq: 1,
Ack: 1, Len: 312
Remote Procedure Call, Type:Call XID:0x1d3220c4
Network File System
[Program Version: 4]
[V4 Procedure: COMPOUND (1)]
GSS Data, Ops(1): CREATE_SESSION
Length: 152
GSS Sequence Number: 76
Tag: <EMPTY>
minorversion: 2
Operations (count: 1): CREATE_SESSION
[Main Opcode: CREATE_SESSION (43)]
GSS Checksum:
00000028040404ffffffffff000000002c19055f1f8d442d594c13849628affc2797cbb2…
GSS Token Length: 40
GSS-API Generic Security Service Application Program Interface
krb5_blob:
040404ffffffffff000000002c19055f1f8d442d594c13849628affc2797cbb23fa080b0…
response:
Frame 2: 206 bytes on wire (1648 bits), 206 bytes captured (1648 bits)
Ethernet II, Src: ArubaaHe_f9:8e:00 (88:3a:30:f9:8e:00), Dst:
HP_19:7d:4b (e0:73:e7:19:7d:4b)
Internet Protocol Version 4, Src: 10.86.18.14, Dst: 10.87.31.152
Transmission Control Protocol, Src Port: 2049, Dst Port: 932, Seq: 1,
Ack: 313, Len: 140
Remote Procedure Call, Type:Reply XID:0x1d3220c4
Network File System
[Program Version: 4]
[V4 Procedure: COMPOUND (1)]
GSS Data, Ops(1): CREATE_SESSION(NFS4ERR_DELAY)
Length: 24
GSS Sequence Number: 76
Status: NFS4ERR_DELAY (10008)
Tag: <EMPTY>
Operations (count: 1)
[Main Opcode: CREATE_SESSION (43)]
GSS Checksum:
00000028040405ffffffffff000000000aa742d0798deaad1a8aa2d7c3a91bf4f6274222…
GSS Token Length: 40
GSS-API Generic Security Service Application Program Interface
krb5_blob:
040405ffffffffff000000000aa742d0798deaad1a8aa2d7c3a91bf4f627422226d74923…
I was hoping that giving the client a different IP address would resolve
the issue for this client, but it didn't. Even though the client had a
new IP address (hostname was kept the same), it failed to mount anything
from the server.
I created another dump of the workqueues and worker pools on the server:
[Mon Mar 18 14:59:33 2024] Showing busy workqueues and worker pools:
[Mon Mar 18 14:59:33 2024] workqueue events: flags=0x0
[Mon Mar 18 14:59:33 2024] pwq 54: cpus=27 node=1 flags=0x0 nice=0
active=1/256 refcnt=2
[Mon Mar 18 14:59:33 2024] pending: drm_fb_helper_damage_work
[drm_kms_helper]
[Mon Mar 18 14:59:33 2024] workqueue events_power_efficient: flags=0x80
[Mon Mar 18 14:59:33 2024] pwq 54: cpus=27 node=1 flags=0x0 nice=0
active=1/256 refcnt=2
[Mon Mar 18 14:59:33 2024] pending: fb_flashcursor
[Mon Mar 18 14:59:33 2024] workqueue mm_percpu_wq: flags=0x8
[Mon Mar 18 14:59:33 2024] pwq 54: cpus=27 node=1 flags=0x0 nice=0
active=1/256 refcnt=3
[Mon Mar 18 14:59:33 2024] pending: lru_add_drain_per_cpu BAR(362)
[Mon Mar 18 14:59:33 2024] workqueue kblockd: flags=0x18
[Mon Mar 18 14:59:33 2024] pwq 55: cpus=27 node=1 flags=0x0 nice=-20
active=1/256 refcnt=2
[Mon Mar 18 14:59:33 2024] pending: blk_mq_timeout_work
In contrast to last time, it doesn't show anything regarding nfs this time.
I also tried the suggestion from Dai Ngo (echo 3 >
/proc/sys/vm/drop_caches), but that didn't seem to make any difference.
We haven't restarted the server yet as it seems the impact seems to
affect fewer clients that before. Is there anything we can run on the
server to further debug this?
In the past, the issue seemed to deteriorate rapidly and resulted in
issues for almost all clients after about 20 minutes. This time the
impact seems to be less, but it's not gone.
How can we force the NFS server to forget about a specific client? I
haven't tried to restart the nfs service yet as I'm afraid it will fail
to stop as before.
Regards,
Rik
--
Rik Theys
System Engineer
KU Leuven - Dept. Elektrotechniek (ESAT)
Kasteelpark Arenberg 10 bus 2440 - B-3001 Leuven-Heverlee
+32(0)16/32.11.07
----------------------------------------------------------------
<<Any errors in spelling, tact or fact are transmission errors>>
^ permalink raw reply [flat|nested] 26+ messages in thread
* Re: nfsd hangs and nfsd_break_deleg_cb+0x170/0x190 warning
2024-03-18 20:21 ` Rik Theys
@ 2024-03-18 21:15 ` Rik Theys
2024-03-18 21:54 ` Jeff Layton
0 siblings, 1 reply; 26+ messages in thread
From: Rik Theys @ 2024-03-18 21:15 UTC (permalink / raw)
To: Jeff Layton, Linux Nfs
Hi,
On 3/18/24 21:21, Rik Theys wrote:
> Hi Jeff,
>
> On 3/12/24 13:47, Jeff Layton wrote:
>> On Tue, 2024-03-12 at 13:24 +0100, Rik Theys wrote:
>>> Hi Jeff,
>>>
>>> On 3/12/24 12:22, Jeff Layton wrote:
>>>> On Mon, 2024-03-11 at 19:43 +0100, Rik Theys wrote:
>>>>> Since a few weeks our Rocky Linux 9 NFS server has periodically
>>>>> logged hung nfsd tasks. The initial effect was that some clients
>>>>> could no longer access the NFS server. This got worse and worse
>>>>> (probably as more nfsd threads got blocked) and we had to restart
>>>>> the server. Restarting the server also failed as the NFS server
>>>>> service could no longer be stopped.
>>>>>
>>>>> The initial kernel we noticed this behavior on was
>>>>> kernel-5.14.0-362.18.1.el9_3.x86_64. Since then we've installed
>>>>> kernel-5.14.0-419.el9.x86_64 from CentOS Stream 9. The same issue
>>>>> happened again on this newer kernel version:
>> 419 is fairly up to date with nfsd changes. There are some known bugs
>> around callbacks, and there is a draft MR in flight to fix it.
>>
>> What kernel were you on prior to 5.14.0-362.18.1.el9_3.x86_64 ? If we
>> can bracket the changes around a particular version, then that might
>> help identify the problem.
>>
>>>>> [Mon Mar 11 14:10:08 2024] Not tainted 5.14.0-419.el9.x86_64 #1
>>>>> [Mon Mar 11 14:10:08 2024] "echo 0 >
>>>>> /proc/sys/kernel/hung_task_timeout_secs" disables this message.
>>>>> [Mon Mar 11 14:10:08 2024]task:nfsd state:D stack:0
>>>>> pid:8865 ppid:2 flags:0x00004000
>>>>> [Mon Mar 11 14:10:08 2024] Call Trace:
>>>>> [Mon Mar 11 14:10:08 2024] <TASK>
>>>>> [Mon Mar 11 14:10:08 2024] __schedule+0x21b/0x550
>>>>> [Mon Mar 11 14:10:08 2024] schedule+0x2d/0x70
>>>>> [Mon Mar 11 14:10:08 2024] schedule_timeout+0x11f/0x160
>>>>> [Mon Mar 11 14:10:08 2024] ? select_idle_sibling+0x28/0x430
>>>>> [Mon Mar 11 14:10:08 2024] ? wake_affine+0x62/0x1f0
>>>>> [Mon Mar 11 14:10:08 2024] __wait_for_common+0x90/0x1d0
>>>>> [Mon Mar 11 14:10:08 2024] ? __pfx_schedule_timeout+0x10/0x10
>>>>> [Mon Mar 11 14:10:08 2024] __flush_workqueue+0x13a/0x3f0
>>>>> [Mon Mar 11 14:10:08 2024] nfsd4_shutdown_callback+0x49/0x120
>>>>> [nfsd]
>>>>> [Mon Mar 11 14:10:08 2024] ? nfsd4_cld_remove+0x54/0x1d0 [nfsd]
>>>>> [Mon Mar 11 14:10:08 2024] ?
>>>>> nfsd4_return_all_client_layouts+0xc4/0xf0 [nfsd]
>>>>> [Mon Mar 11 14:10:08 2024] ? nfsd4_shutdown_copy+0x68/0xc0 [nfsd]
>>>>> [Mon Mar 11 14:10:08 2024] __destroy_client+0x1f3/0x290 [nfsd]
>>>>> [Mon Mar 11 14:10:08 2024] nfsd4_exchange_id+0x75f/0x770 [nfsd]
>>>>> [Mon Mar 11 14:10:08 2024] ? nfsd4_decode_opaque+0x3a/0x90 [nfsd]
>>>>> [Mon Mar 11 14:10:08 2024] nfsd4_proc_compound+0x44b/0x700 [nfsd]
>>>>> [Mon Mar 11 14:10:08 2024] nfsd_dispatch+0x94/0x1c0 [nfsd]
>>>>> [Mon Mar 11 14:10:08 2024] svc_process_common+0x2ec/0x660
>>>>> [sunrpc]
>>>>> [Mon Mar 11 14:10:08 2024] ? __pfx_nfsd_dispatch+0x10/0x10 [nfsd]
>>>>> [Mon Mar 11 14:10:08 2024] ? __pfx_nfsd+0x10/0x10 [nfsd]
>>>>> [Mon Mar 11 14:10:08 2024] svc_process+0x12d/0x170 [sunrpc]
>>>>> [Mon Mar 11 14:10:08 2024] nfsd+0x84/0xb0 [nfsd]
>>>>> [Mon Mar 11 14:10:08 2024] kthread+0xdd/0x100
>>>>> [Mon Mar 11 14:10:08 2024] ? __pfx_kthread+0x10/0x10
>>>>> [Mon Mar 11 14:10:08 2024] ret_from_fork+0x29/0x50
>>>>> [Mon Mar 11 14:10:08 2024] </TASK>
>>>>> [Mon Mar 11 14:10:08 2024] INFO: task nfsd:8866 blocked for
>>>>> more than 122 seconds.
>>>>> [Mon Mar 11 14:10:08 2024] Not tainted
>>>>> 5.14.0-419.el9.x86_64 #1
>>>>> [Mon Mar 11 14:10:08 2024] "echo 0 >
>>>>> /proc/sys/kernel/hung_task_timeout_secs" disables this message.
>>>>> [Mon Mar 11 14:10:08 2024]task:nfsd state:D stack:0
>>>>> pid:8866 ppid:2 flags:0x00004000
>>>>> [Mon Mar 11 14:10:08 2024] Call Trace:
>>>>> [Mon Mar 11 14:10:08 2024] <TASK>
>>>>> [Mon Mar 11 14:10:08 2024] __schedule+0x21b/0x550
>>>>> [Mon Mar 11 14:10:08 2024] schedule+0x2d/0x70
>>>>> [Mon Mar 11 14:10:08 2024] schedule_timeout+0x11f/0x160
>>>>> [Mon Mar 11 14:10:08 2024] ? select_idle_sibling+0x28/0x430
>>>>> [Mon Mar 11 14:10:08 2024] ? tcp_recvmsg+0x196/0x210
>>>>> [Mon Mar 11 14:10:08 2024] ? wake_affine+0x62/0x1f0
>>>>> [Mon Mar 11 14:10:08 2024] __wait_for_common+0x90/0x1d0
>>>>> [Mon Mar 11 14:10:08 2024] ? __pfx_schedule_timeout+0x10/0x10
>>>>> [Mon Mar 11 14:10:08 2024] __flush_workqueue+0x13a/0x3f0
>>>>> [Mon Mar 11 14:10:08 2024] nfsd4_destroy_session+0x1a4/0x240
>>>>> [nfsd]
>>>>> [Mon Mar 11 14:10:08 2024] nfsd4_proc_compound+0x44b/0x700 [nfsd]
>>>>> [Mon Mar 11 14:10:08 2024] nfsd_dispatch+0x94/0x1c0 [nfsd]
>>>>> [Mon Mar 11 14:10:08 2024] svc_process_common+0x2ec/0x660
>>>>> [sunrpc]
>>>>> [Mon Mar 11 14:10:08 2024] ? __pfx_nfsd_dispatch+0x10/0x10 [nfsd]
>>>>> [Mon Mar 11 14:10:08 2024] ? __pfx_nfsd+0x10/0x10 [nfsd]
>>>>> [Mon Mar 11 14:10:08 2024] svc_process+0x12d/0x170 [sunrpc]
>>>>> [Mon Mar 11 14:10:08 2024] nfsd+0x84/0xb0 [nfsd]
>>>>> [Mon Mar 11 14:10:08 2024] kthread+0xdd/0x100
>>>>> [Mon Mar 11 14:10:08 2024] ? __pfx_kthread+0x10/0x10
>>>>> [Mon Mar 11 14:10:08 2024] ret_from_fork+0x29/0x50
>>>>> [Mon Mar 11 14:10:08 2024] </TASK>
>>>>>
>>>> The above threads are trying to flush the workqueue, so that probably
>>>> means that they are stuck waiting on a workqueue job to finish.
>>>>> The above is repeated a few times, and then this warning is
>>>>> also logged:
>>>>> [Mon Mar 11 14:12:04 2024] ------------[ cut here ]------------
>>>>> [Mon Mar 11 14:12:04 2024] WARNING: CPU: 39 PID: 8844 at
>>>>> fs/nfsd/nfs4state.c:4919 nfsd_break_deleg_cb+0x170/0x190 [nfsd]
>>>>> [Mon Mar 11 14:12:05 2024] Modules linked in: nfsv4
>>>>> dns_resolver nfs fscache netfs rpcsec_gss_krb5 rpcrdma rdma_cm
>>>>> iw_cm ib_cm ib_core binfmt_misc bonding tls rfkill nft_counter nft_ct
>>>>> nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_reject_inet
>>>>> nf_reject_ipv4 nf_reject_ipv6 nft_reject nf_tables nfnetlink vfat
>>>>> fat dm_thin_pool dm_persistent_data dm_bio_prison dm_bufio l
>>>>> ibcrc32c dm_service_time dm_multipath intel_rapl_msr
>>>>> intel_rapl_common intel_uncore_frequency
>>>>> intel_uncore_frequency_common isst_if_common skx_edac nfit
>>>>> libnvdimm ipmi_ssif x86_pkg_temp
>>>>> _thermal intel_powerclamp coretemp kvm_intel kvm irqbypass
>>>>> dcdbas rapl intel_cstate mgag200 i2c_algo_bit drm_shmem_helper
>>>>> dell_smbios drm_kms_helper dell_wmi_descriptor wmi_bmof intel_u
>>>>> ncore syscopyarea pcspkr sysfillrect mei_me sysimgblt acpi_ipmi
>>>>> mei fb_sys_fops i2c_i801 ipmi_si intel_pch_thermal lpc_ich
>>>>> ipmi_devintf i2c_smbus ipmi_msghandler joydev acpi_power_meter
>>>>> nfsd auth_rpcgss nfs_acl drm lockd grace fuse sunrpc ext4
>>>>> mbcache jbd2 sd_mod sg lpfc
>>>>> [Mon Mar 11 14:12:05 2024] nvmet_fc nvmet nvme_fc nvme_fabrics
>>>>> crct10dif_pclmul ahci libahci crc32_pclmul nvme_core crc32c_intel
>>>>> ixgbe megaraid_sas libata nvme_common ghash_clmulni_int
>>>>> el t10_pi wdat_wdt scsi_transport_fc mdio wmi dca dm_mirror
>>>>> dm_region_hash dm_log dm_mod
>>>>> [Mon Mar 11 14:12:05 2024] CPU: 39 PID: 8844 Comm: nfsd Not
>>>>> tainted 5.14.0-419.el9.x86_64 #1
>>>>> [Mon Mar 11 14:12:05 2024] Hardware name: Dell Inc. PowerEdge
>>>>> R740/00WGD1, BIOS 2.20.1 09/13/2023
>>>>> [Mon Mar 11 14:12:05 2024] RIP:
>>>>> 0010:nfsd_break_deleg_cb+0x170/0x190 [nfsd]
>>>>> [Mon Mar 11 14:12:05 2024] Code: a6 95 c5 f3 e9 ff fe ff ff 48
>>>>> 89 df be 01 00 00 00 e8 34 b5 13 f4 48 8d bb 98 00 00 00 e8 c8 f9
>>>>> 00 00 84 c0 0f 85 2e ff ff ff <0f> 0b e9 27 ff ff ff be
>>>>> 02 00 00 00 48 89 df e8 0c b5 13 f4 e9 01
>>>>> [Mon Mar 11 14:12:05 2024] RSP: 0018:ffff9929e0bb7b80 EFLAGS:
>>>>> 00010246
>>>>> [Mon Mar 11 14:12:05 2024] RAX: 0000000000000000 RBX:
>>>>> ffff8ada51930900 RCX: 0000000000000024
>>>>> [Mon Mar 11 14:12:05 2024] RDX: ffff8ada519309c8 RSI:
>>>>> ffff8ad582933c00 RDI: 0000000000002000
>>>>> [Mon Mar 11 14:12:05 2024] RBP: ffff8ad46bf21574 R08:
>>>>> ffff9929e0bb7b48 R09: 0000000000000000
>>>>> [Mon Mar 11 14:12:05 2024] R10: ffff8aec859a2948 R11:
>>>>> 0000000000000000 R12: ffff8ad6f497c360
>>>>> [Mon Mar 11 14:12:05 2024] R13: ffff8ad46bf21560 R14:
>>>>> ffff8ae5942e0b10 R15: ffff8ad6f497c360
>>>>> [Mon Mar 11 14:12:05 2024] FS: 0000000000000000(0000)
>>>>> GS:ffff8b031fcc0000(0000) knlGS:0000000000000000
>>>>> [Mon Mar 11 14:12:05 2024] CS: 0010 DS: 0000 ES: 0000 CR0:
>>>>> 0000000080050033
>>>>> [Mon Mar 11 14:12:05 2024] CR2: 00007fafe2060744 CR3:
>>>>> 00000018e58de006 CR4: 00000000007706e0
>>>>> [Mon Mar 11 14:12:05 2024] DR0: 0000000000000000 DR1:
>>>>> 0000000000000000 DR2: 0000000000000000
>>>>> [Mon Mar 11 14:12:05 2024] DR3: 0000000000000000 DR6:
>>>>> 00000000fffe0ff0 DR7: 0000000000000400
>>>>> [Mon Mar 11 14:12:05 2024] PKRU: 55555554
>>>>> [Mon Mar 11 14:12:05 2024] Call Trace:
>>>>> [Mon Mar 11 14:12:05 2024] <TASK>
>>>>> [Mon Mar 11 14:12:05 2024] ? show_trace_log_lvl+0x1c4/0x2df
>>>>> [Mon Mar 11 14:12:05 2024] ? show_trace_log_lvl+0x1c4/0x2df
>>>>> [Mon Mar 11 14:12:05 2024] ? __break_lease+0x16f/0x5f0
>>>>> [Mon Mar 11 14:12:05 2024] ? nfsd_break_deleg_cb+0x170/0x190
>>>>> [nfsd]
>>>>> [Mon Mar 11 14:12:05 2024] ? __warn+0x81/0x110
>>>>> [Mon Mar 11 14:12:05 2024] ? nfsd_break_deleg_cb+0x170/0x190
>>>>> [nfsd]
>>>>> [Mon Mar 11 14:12:05 2024] ? report_bug+0x10a/0x140
>>>>> [Mon Mar 11 14:12:05 2024] ? handle_bug+0x3c/0x70
>>>>> [Mon Mar 11 14:12:05 2024] ? exc_invalid_op+0x14/0x70
>>>>> [Mon Mar 11 14:12:05 2024] ? asm_exc_invalid_op+0x16/0x20
>>>>> [Mon Mar 11 14:12:05 2024] ? nfsd_break_deleg_cb+0x170/0x190
>>>>> [nfsd]
>>>>> [Mon Mar 11 14:12:05 2024] __break_lease+0x16f/0x5f0
>>>>> [Mon Mar 11 14:12:05 2024] ?
>>>>> nfsd_file_lookup_locked+0x117/0x160 [nfsd]
>>>>> [Mon Mar 11 14:12:05 2024] ? list_lru_del+0x101/0x150
>>>>> [Mon Mar 11 14:12:05 2024] nfsd_file_do_acquire+0x790/0x830
>>>>> [nfsd]
>>>>> [Mon Mar 11 14:12:05 2024] nfs4_get_vfs_file+0x315/0x3a0 [nfsd]
>>>>> [Mon Mar 11 14:12:05 2024] nfsd4_process_open2+0x430/0xa30 [nfsd]
>>>>> [Mon Mar 11 14:12:05 2024] ? fh_verify+0x297/0x2f0 [nfsd]
>>>>> [Mon Mar 11 14:12:05 2024] nfsd4_open+0x3ce/0x4b0 [nfsd]
>>>>> [Mon Mar 11 14:12:05 2024] nfsd4_proc_compound+0x44b/0x700 [nfsd]
>>>>> [Mon Mar 11 14:12:05 2024] nfsd_dispatch+0x94/0x1c0 [nfsd]
>>>>> [Mon Mar 11 14:12:05 2024] svc_process_common+0x2ec/0x660
>>>>> [sunrpc]
>>>>> [Mon Mar 11 14:12:05 2024] ? __pfx_nfsd_dispatch+0x10/0x10 [nfsd]
>>>>> [Mon Mar 11 14:12:05 2024] ? __pfx_nfsd+0x10/0x10 [nfsd]
>>>>> [Mon Mar 11 14:12:05 2024] svc_process+0x12d/0x170 [sunrpc]
>>>>> [Mon Mar 11 14:12:05 2024] nfsd+0x84/0xb0 [nfsd]
>>>>> [Mon Mar 11 14:12:05 2024] kthread+0xdd/0x100
>>>>> [Mon Mar 11 14:12:05 2024] ? __pfx_kthread+0x10/0x10
>>>>> [Mon Mar 11 14:12:05 2024] ret_from_fork+0x29/0x50
>>>>> [Mon Mar 11 14:12:05 2024] </TASK>
>>>>> [Mon Mar 11 14:12:05 2024] ---[ end trace 7a039e17443dc651 ]---
>>>> This is probably this WARN in nfsd_break_one_deleg:
>>>>
>>>> WARN_ON_ONCE(!nfsd4_run_cb(&dp->dl_recall));
>>>>
>>>> It means that a delegation break callback to the client couldn't be
>>>> queued to the workqueue, and so it didn't run.
>>>>
>>>>> Could this be the same issue as described
>>>>> here:https://lore.kernel.org/linux-nfs/af0ec881-5ebf-4feb-98ae-3ed2a77f86f1@oracle.com/
>>>>> ?
>>>> Yes, most likely the same problem.
>>> If I read that thread correctly, this issue was introduced between
>>> 6.1.63 and 6.6.3? Is it possible the EL9 5.14.0-362.18.1.el9_3
>>> backported these changes, or were we hitting some other bug with that
>>> version? It seems the 6.1.x kernel is not affected? If so, that
>>> would be
>>> the recommended kernel to run?
>> Anything is possible. We have to identify the problem first.
>>>>> As described in that thread, I've tried to obtain the requested
>>>>> information.
>>>>>
>>>>> Is it possible this is the issue that was fixed by the patches
>>>>> described
>>>>> here?https://lore.kernel.org/linux-nfs/2024022054-cause-suffering-eae8@gregkh/
>>>>>
>>>> Doubtful. Those are targeted toward a different set of issues.
>>>>
>>>> If you're willing, I do have some patches queued up for CentOS here
>>>> that
>>>> fix some backchannel problems that could be related. I'm mainly
>>>> waiting
>>>> on Chuck to send these to Linus and then we'll likely merge them into
>>>> CentOS soon afterward:
>>>>
>>>> https://gitlab.com/redhat/centos-stream/src/kernel/centos-stream-9/-/merge_requests/3689
>>>>
>>>>
>>> If you can send me a patch file, I can rebuild the C9S kernel with that
>>> patch and run it. It can take a while for the bug to trigger as I
>>> believe it seems to be very workload dependent (we were running very
>>> stable for months and now hit this bug every other week).
>>>
>>>
>> It's probably simpler to just pull down the build artifacts for that MR.
>> You have to drill down through the CI for it, but they are here:
>>
>> https://s3.amazonaws.com/arr-cki-prod-trusted-artifacts/index.html?prefix=trusted-artifacts/1194300175/publish_x86_64/6278921877/artifacts/
>>
>>
>> There's even a repo file you can install on the box to pull them down.
>
> We installed this kernel on the server 3 days ago. Today, a user
> informed us that their screen was black after logging in. Similar to
> other occurrences of this issue, the mount command on the client was
> hung. But in contrast to the other times, there were no messages in
> the logs kernel logs on the server. Even restarting the client does
> not resolve the issue.
>
> Something still seems to be wrong on the server though. When I look at
> the directories under /proc/fs/nfsd/clients, there's still a directory
> for the specific client, even though it's no longer running:
>
> # cat 155/info
> clientid: 0xc8edb7f65f4a9ad
> address: "10.87.31.152:819"
> status: confirmed
> seconds from last renew: 33163
> name: "Linux NFSv4.2 bersalis.esat.kuleuven.be"
> minor version: 2
> Implementation domain: "kernel.org"
> Implementation name: "Linux 5.14.0-362.18.1.el9_3.0.1.x86_64 #1 SMP
> PREEMPT_DYNAMIC Sun Feb 11 13:49:23 UTC 2024 x86_64"
> Implementation time: [0, 0]
> callback state: DOWN
> callback address: 10.87.31.152:0
>
The nfsdclnts command for this client shows the following delegations:
# nfsdclnts -f 155/states -t all
Inode number | Type | Access | Deny | ip address | Filename
169346743 | open | r- | -- | 10.87.31.152:819 |
disconnected dentry
169346743 | deleg | r | | 10.87.31.152:819 |
disconnected dentry
169346746 | open | r- | -- | 10.87.31.152:819 |
disconnected dentry
169346746 | deleg | r | | 10.87.31.152:819 |
disconnected dentry
I see a lot of recent patches regarding directory delegations. Could
this be related to this?
Will a 5.14.0-362.18.1.el9_3.0.1 kernel try to use a directory delegation?
Regards,
Rik
>
> The system seems to have identified that the client is no longer
> reachable, but the client entry does not go away. When a mount was
> hanging on the client, there would be two directories in clients for
> the same client. Killing the mount command clears up the second entry.
>
> Even after running conntrack -D on the server to remove the tcp
> connection from the conntrack table, the entry doesn't go away and the
> client still can not mount anything from the server.
>
> A tcpdump on the client while a mount was running logged the following
> messages over and over again:
>
> request:
>
> Frame 1: 378 bytes on wire (3024 bits), 378 bytes captured (3024 bits)
> Ethernet II, Src: HP_19:7d:4b (e0:73:e7:19:7d:4b), Dst:
> ArubaaHe_f9:8e:00 (88:3a:30:f9:8e:00)
> Internet Protocol Version 4, Src: 10.87.31.152, Dst: 10.86.18.14
> Transmission Control Protocol, Src Port: 932, Dst Port: 2049, Seq: 1,
> Ack: 1, Len: 312
> Remote Procedure Call, Type:Call XID:0x1d3220c4
> Network File System
> [Program Version: 4]
> [V4 Procedure: COMPOUND (1)]
> GSS Data, Ops(1): CREATE_SESSION
> Length: 152
> GSS Sequence Number: 76
> Tag: <EMPTY>
> minorversion: 2
> Operations (count: 1): CREATE_SESSION
> [Main Opcode: CREATE_SESSION (43)]
> GSS Checksum:
> 00000028040404ffffffffff000000002c19055f1f8d442d594c13849628affc2797cbb2…
> GSS Token Length: 40
> GSS-API Generic Security Service Application Program Interface
> krb5_blob:
> 040404ffffffffff000000002c19055f1f8d442d594c13849628affc2797cbb23fa080b0…
>
> response:
>
> Frame 2: 206 bytes on wire (1648 bits), 206 bytes captured (1648 bits)
> Ethernet II, Src: ArubaaHe_f9:8e:00 (88:3a:30:f9:8e:00), Dst:
> HP_19:7d:4b (e0:73:e7:19:7d:4b)
> Internet Protocol Version 4, Src: 10.86.18.14, Dst: 10.87.31.152
> Transmission Control Protocol, Src Port: 2049, Dst Port: 932, Seq: 1,
> Ack: 313, Len: 140
> Remote Procedure Call, Type:Reply XID:0x1d3220c4
> Network File System
> [Program Version: 4]
> [V4 Procedure: COMPOUND (1)]
> GSS Data, Ops(1): CREATE_SESSION(NFS4ERR_DELAY)
> Length: 24
> GSS Sequence Number: 76
> Status: NFS4ERR_DELAY (10008)
> Tag: <EMPTY>
> Operations (count: 1)
> [Main Opcode: CREATE_SESSION (43)]
> GSS Checksum:
> 00000028040405ffffffffff000000000aa742d0798deaad1a8aa2d7c3a91bf4f6274222…
> GSS Token Length: 40
> GSS-API Generic Security Service Application Program Interface
> krb5_blob:
> 040405ffffffffff000000000aa742d0798deaad1a8aa2d7c3a91bf4f627422226d74923…
>
> I was hoping that giving the client a different IP address would
> resolve the issue for this client, but it didn't. Even though the
> client had a new IP address (hostname was kept the same), it failed to
> mount anything from the server.
>
> I created another dump of the workqueues and worker pools on the server:
>
> [Mon Mar 18 14:59:33 2024] Showing busy workqueues and worker pools:
> [Mon Mar 18 14:59:33 2024] workqueue events: flags=0x0
> [Mon Mar 18 14:59:33 2024] pwq 54: cpus=27 node=1 flags=0x0 nice=0
> active=1/256 refcnt=2
> [Mon Mar 18 14:59:33 2024] pending: drm_fb_helper_damage_work
> [drm_kms_helper]
> [Mon Mar 18 14:59:33 2024] workqueue events_power_efficient: flags=0x80
> [Mon Mar 18 14:59:33 2024] pwq 54: cpus=27 node=1 flags=0x0 nice=0
> active=1/256 refcnt=2
> [Mon Mar 18 14:59:33 2024] pending: fb_flashcursor
> [Mon Mar 18 14:59:33 2024] workqueue mm_percpu_wq: flags=0x8
> [Mon Mar 18 14:59:33 2024] pwq 54: cpus=27 node=1 flags=0x0 nice=0
> active=1/256 refcnt=3
> [Mon Mar 18 14:59:33 2024] pending: lru_add_drain_per_cpu BAR(362)
> [Mon Mar 18 14:59:33 2024] workqueue kblockd: flags=0x18
> [Mon Mar 18 14:59:33 2024] pwq 55: cpus=27 node=1 flags=0x0 nice=-20
> active=1/256 refcnt=2
> [Mon Mar 18 14:59:33 2024] pending: blk_mq_timeout_work
>
>
> In contrast to last time, it doesn't show anything regarding nfs this
> time.
>
> I also tried the suggestion from Dai Ngo (echo 3 >
> /proc/sys/vm/drop_caches), but that didn't seem to make any difference.
>
> We haven't restarted the server yet as it seems the impact seems to
> affect fewer clients that before. Is there anything we can run on the
> server to further debug this?
>
> In the past, the issue seemed to deteriorate rapidly and resulted in
> issues for almost all clients after about 20 minutes. This time the
> impact seems to be less, but it's not gone.
>
> How can we force the NFS server to forget about a specific client? I
> haven't tried to restart the nfs service yet as I'm afraid it will
> fail to stop as before.
>
>
> Regards,
>
> Rik
>
--
Rik Theys
System Engineer
KU Leuven - Dept. Elektrotechniek (ESAT)
Kasteelpark Arenberg 10 bus 2440 - B-3001 Leuven-Heverlee
+32(0)16/32.11.07
----------------------------------------------------------------
<<Any errors in spelling, tact or fact are transmission errors>>
^ permalink raw reply [flat|nested] 26+ messages in thread
* Re: nfsd hangs and nfsd_break_deleg_cb+0x170/0x190 warning
2024-03-18 21:15 ` Rik Theys
@ 2024-03-18 21:54 ` Jeff Layton
2024-03-19 7:58 ` Rik Theys
0 siblings, 1 reply; 26+ messages in thread
From: Jeff Layton @ 2024-03-18 21:54 UTC (permalink / raw)
To: Rik Theys, Linux Nfs
On Mon, 2024-03-18 at 22:15 +0100, Rik Theys wrote:
> Hi,
>
> On 3/18/24 21:21, Rik Theys wrote:
> > Hi Jeff,
> >
> > On 3/12/24 13:47, Jeff Layton wrote:
> > > On Tue, 2024-03-12 at 13:24 +0100, Rik Theys wrote:
> > > > Hi Jeff,
> > > >
> > > > On 3/12/24 12:22, Jeff Layton wrote:
> > > > > On Mon, 2024-03-11 at 19:43 +0100, Rik Theys wrote:
> > > > > > Since a few weeks our Rocky Linux 9 NFS server has periodically
> > > > > > logged hung nfsd tasks. The initial effect was that some clients
> > > > > > could no longer access the NFS server. This got worse and worse
> > > > > > (probably as more nfsd threads got blocked) and we had to restart
> > > > > > the server. Restarting the server also failed as the NFS server
> > > > > > service could no longer be stopped.
> > > > > >
> > > > > > The initial kernel we noticed this behavior on was
> > > > > > kernel-5.14.0-362.18.1.el9_3.x86_64. Since then we've installed
> > > > > > kernel-5.14.0-419.el9.x86_64 from CentOS Stream 9. The same issue
> > > > > > happened again on this newer kernel version:
> > > 419 is fairly up to date with nfsd changes. There are some known bugs
> > > around callbacks, and there is a draft MR in flight to fix it.
> > >
> > > What kernel were you on prior to 5.14.0-362.18.1.el9_3.x86_64 ? If we
> > > can bracket the changes around a particular version, then that might
> > > help identify the problem.
> > >
> > > > > > [Mon Mar 11 14:10:08 2024] Not tainted 5.14.0-419.el9.x86_64 #1
> > > > > > [Mon Mar 11 14:10:08 2024] "echo 0 >
> > > > > > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> > > > > > [Mon Mar 11 14:10:08 2024]task:nfsd state:D stack:0
> > > > > > pid:8865 ppid:2 flags:0x00004000
> > > > > > [Mon Mar 11 14:10:08 2024] Call Trace:
> > > > > > [Mon Mar 11 14:10:08 2024] <TASK>
> > > > > > [Mon Mar 11 14:10:08 2024] __schedule+0x21b/0x550
> > > > > > [Mon Mar 11 14:10:08 2024] schedule+0x2d/0x70
> > > > > > [Mon Mar 11 14:10:08 2024] schedule_timeout+0x11f/0x160
> > > > > > [Mon Mar 11 14:10:08 2024] ? select_idle_sibling+0x28/0x430
> > > > > > [Mon Mar 11 14:10:08 2024] ? wake_affine+0x62/0x1f0
> > > > > > [Mon Mar 11 14:10:08 2024] __wait_for_common+0x90/0x1d0
> > > > > > [Mon Mar 11 14:10:08 2024] ? __pfx_schedule_timeout+0x10/0x10
> > > > > > [Mon Mar 11 14:10:08 2024] __flush_workqueue+0x13a/0x3f0
> > > > > > [Mon Mar 11 14:10:08 2024] nfsd4_shutdown_callback+0x49/0x120
> > > > > > [nfsd]
> > > > > > [Mon Mar 11 14:10:08 2024] ? nfsd4_cld_remove+0x54/0x1d0 [nfsd]
> > > > > > [Mon Mar 11 14:10:08 2024] ?
> > > > > > nfsd4_return_all_client_layouts+0xc4/0xf0 [nfsd]
> > > > > > [Mon Mar 11 14:10:08 2024] ? nfsd4_shutdown_copy+0x68/0xc0 [nfsd]
> > > > > > [Mon Mar 11 14:10:08 2024] __destroy_client+0x1f3/0x290 [nfsd]
> > > > > > [Mon Mar 11 14:10:08 2024] nfsd4_exchange_id+0x75f/0x770 [nfsd]
> > > > > > [Mon Mar 11 14:10:08 2024] ? nfsd4_decode_opaque+0x3a/0x90 [nfsd]
> > > > > > [Mon Mar 11 14:10:08 2024] nfsd4_proc_compound+0x44b/0x700 [nfsd]
> > > > > > [Mon Mar 11 14:10:08 2024] nfsd_dispatch+0x94/0x1c0 [nfsd]
> > > > > > [Mon Mar 11 14:10:08 2024] svc_process_common+0x2ec/0x660
> > > > > > [sunrpc]
> > > > > > [Mon Mar 11 14:10:08 2024] ? __pfx_nfsd_dispatch+0x10/0x10 [nfsd]
> > > > > > [Mon Mar 11 14:10:08 2024] ? __pfx_nfsd+0x10/0x10 [nfsd]
> > > > > > [Mon Mar 11 14:10:08 2024] svc_process+0x12d/0x170 [sunrpc]
> > > > > > [Mon Mar 11 14:10:08 2024] nfsd+0x84/0xb0 [nfsd]
> > > > > > [Mon Mar 11 14:10:08 2024] kthread+0xdd/0x100
> > > > > > [Mon Mar 11 14:10:08 2024] ? __pfx_kthread+0x10/0x10
> > > > > > [Mon Mar 11 14:10:08 2024] ret_from_fork+0x29/0x50
> > > > > > [Mon Mar 11 14:10:08 2024] </TASK>
> > > > > > [Mon Mar 11 14:10:08 2024] INFO: task nfsd:8866 blocked for
> > > > > > more than 122 seconds.
> > > > > > [Mon Mar 11 14:10:08 2024] Not tainted
> > > > > > 5.14.0-419.el9.x86_64 #1
> > > > > > [Mon Mar 11 14:10:08 2024] "echo 0 >
> > > > > > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> > > > > > [Mon Mar 11 14:10:08 2024]task:nfsd state:D stack:0
> > > > > > pid:8866 ppid:2 flags:0x00004000
> > > > > > [Mon Mar 11 14:10:08 2024] Call Trace:
> > > > > > [Mon Mar 11 14:10:08 2024] <TASK>
> > > > > > [Mon Mar 11 14:10:08 2024] __schedule+0x21b/0x550
> > > > > > [Mon Mar 11 14:10:08 2024] schedule+0x2d/0x70
> > > > > > [Mon Mar 11 14:10:08 2024] schedule_timeout+0x11f/0x160
> > > > > > [Mon Mar 11 14:10:08 2024] ? select_idle_sibling+0x28/0x430
> > > > > > [Mon Mar 11 14:10:08 2024] ? tcp_recvmsg+0x196/0x210
> > > > > > [Mon Mar 11 14:10:08 2024] ? wake_affine+0x62/0x1f0
> > > > > > [Mon Mar 11 14:10:08 2024] __wait_for_common+0x90/0x1d0
> > > > > > [Mon Mar 11 14:10:08 2024] ? __pfx_schedule_timeout+0x10/0x10
> > > > > > [Mon Mar 11 14:10:08 2024] __flush_workqueue+0x13a/0x3f0
> > > > > > [Mon Mar 11 14:10:08 2024] nfsd4_destroy_session+0x1a4/0x240
> > > > > > [nfsd]
> > > > > > [Mon Mar 11 14:10:08 2024] nfsd4_proc_compound+0x44b/0x700 [nfsd]
> > > > > > [Mon Mar 11 14:10:08 2024] nfsd_dispatch+0x94/0x1c0 [nfsd]
> > > > > > [Mon Mar 11 14:10:08 2024] svc_process_common+0x2ec/0x660
> > > > > > [sunrpc]
> > > > > > [Mon Mar 11 14:10:08 2024] ? __pfx_nfsd_dispatch+0x10/0x10 [nfsd]
> > > > > > [Mon Mar 11 14:10:08 2024] ? __pfx_nfsd+0x10/0x10 [nfsd]
> > > > > > [Mon Mar 11 14:10:08 2024] svc_process+0x12d/0x170 [sunrpc]
> > > > > > [Mon Mar 11 14:10:08 2024] nfsd+0x84/0xb0 [nfsd]
> > > > > > [Mon Mar 11 14:10:08 2024] kthread+0xdd/0x100
> > > > > > [Mon Mar 11 14:10:08 2024] ? __pfx_kthread+0x10/0x10
> > > > > > [Mon Mar 11 14:10:08 2024] ret_from_fork+0x29/0x50
> > > > > > [Mon Mar 11 14:10:08 2024] </TASK>
> > > > > >
> > > > > The above threads are trying to flush the workqueue, so that probably
> > > > > means that they are stuck waiting on a workqueue job to finish.
> > > > > > The above is repeated a few times, and then this warning is
> > > > > > also logged:
> > > > > > [Mon Mar 11 14:12:04 2024] ------------[ cut here ]------------
> > > > > > [Mon Mar 11 14:12:04 2024] WARNING: CPU: 39 PID: 8844 at
> > > > > > fs/nfsd/nfs4state.c:4919 nfsd_break_deleg_cb+0x170/0x190 [nfsd]
> > > > > > [Mon Mar 11 14:12:05 2024] Modules linked in: nfsv4
> > > > > > dns_resolver nfs fscache netfs rpcsec_gss_krb5 rpcrdma rdma_cm
> > > > > > iw_cm ib_cm ib_core binfmt_misc bonding tls rfkill nft_counter nft_ct
> > > > > > nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_reject_inet
> > > > > > nf_reject_ipv4 nf_reject_ipv6 nft_reject nf_tables nfnetlink vfat
> > > > > > fat dm_thin_pool dm_persistent_data dm_bio_prison dm_bufio l
> > > > > > ibcrc32c dm_service_time dm_multipath intel_rapl_msr
> > > > > > intel_rapl_common intel_uncore_frequency
> > > > > > intel_uncore_frequency_common isst_if_common skx_edac nfit
> > > > > > libnvdimm ipmi_ssif x86_pkg_temp
> > > > > > _thermal intel_powerclamp coretemp kvm_intel kvm irqbypass
> > > > > > dcdbas rapl intel_cstate mgag200 i2c_algo_bit drm_shmem_helper
> > > > > > dell_smbios drm_kms_helper dell_wmi_descriptor wmi_bmof intel_u
> > > > > > ncore syscopyarea pcspkr sysfillrect mei_me sysimgblt acpi_ipmi
> > > > > > mei fb_sys_fops i2c_i801 ipmi_si intel_pch_thermal lpc_ich
> > > > > > ipmi_devintf i2c_smbus ipmi_msghandler joydev acpi_power_meter
> > > > > > nfsd auth_rpcgss nfs_acl drm lockd grace fuse sunrpc ext4
> > > > > > mbcache jbd2 sd_mod sg lpfc
> > > > > > [Mon Mar 11 14:12:05 2024] nvmet_fc nvmet nvme_fc nvme_fabrics
> > > > > > crct10dif_pclmul ahci libahci crc32_pclmul nvme_core crc32c_intel
> > > > > > ixgbe megaraid_sas libata nvme_common ghash_clmulni_int
> > > > > > el t10_pi wdat_wdt scsi_transport_fc mdio wmi dca dm_mirror
> > > > > > dm_region_hash dm_log dm_mod
> > > > > > [Mon Mar 11 14:12:05 2024] CPU: 39 PID: 8844 Comm: nfsd Not
> > > > > > tainted 5.14.0-419.el9.x86_64 #1
> > > > > > [Mon Mar 11 14:12:05 2024] Hardware name: Dell Inc. PowerEdge
> > > > > > R740/00WGD1, BIOS 2.20.1 09/13/2023
> > > > > > [Mon Mar 11 14:12:05 2024] RIP:
> > > > > > 0010:nfsd_break_deleg_cb+0x170/0x190 [nfsd]
> > > > > > [Mon Mar 11 14:12:05 2024] Code: a6 95 c5 f3 e9 ff fe ff ff 48
> > > > > > 89 df be 01 00 00 00 e8 34 b5 13 f4 48 8d bb 98 00 00 00 e8 c8 f9
> > > > > > 00 00 84 c0 0f 85 2e ff ff ff <0f> 0b e9 27 ff ff ff be
> > > > > > 02 00 00 00 48 89 df e8 0c b5 13 f4 e9 01
> > > > > > [Mon Mar 11 14:12:05 2024] RSP: 0018:ffff9929e0bb7b80 EFLAGS:
> > > > > > 00010246
> > > > > > [Mon Mar 11 14:12:05 2024] RAX: 0000000000000000 RBX:
> > > > > > ffff8ada51930900 RCX: 0000000000000024
> > > > > > [Mon Mar 11 14:12:05 2024] RDX: ffff8ada519309c8 RSI:
> > > > > > ffff8ad582933c00 RDI: 0000000000002000
> > > > > > [Mon Mar 11 14:12:05 2024] RBP: ffff8ad46bf21574 R08:
> > > > > > ffff9929e0bb7b48 R09: 0000000000000000
> > > > > > [Mon Mar 11 14:12:05 2024] R10: ffff8aec859a2948 R11:
> > > > > > 0000000000000000 R12: ffff8ad6f497c360
> > > > > > [Mon Mar 11 14:12:05 2024] R13: ffff8ad46bf21560 R14:
> > > > > > ffff8ae5942e0b10 R15: ffff8ad6f497c360
> > > > > > [Mon Mar 11 14:12:05 2024] FS: 0000000000000000(0000)
> > > > > > GS:ffff8b031fcc0000(0000) knlGS:0000000000000000
> > > > > > [Mon Mar 11 14:12:05 2024] CS: 0010 DS: 0000 ES: 0000 CR0:
> > > > > > 0000000080050033
> > > > > > [Mon Mar 11 14:12:05 2024] CR2: 00007fafe2060744 CR3:
> > > > > > 00000018e58de006 CR4: 00000000007706e0
> > > > > > [Mon Mar 11 14:12:05 2024] DR0: 0000000000000000 DR1:
> > > > > > 0000000000000000 DR2: 0000000000000000
> > > > > > [Mon Mar 11 14:12:05 2024] DR3: 0000000000000000 DR6:
> > > > > > 00000000fffe0ff0 DR7: 0000000000000400
> > > > > > [Mon Mar 11 14:12:05 2024] PKRU: 55555554
> > > > > > [Mon Mar 11 14:12:05 2024] Call Trace:
> > > > > > [Mon Mar 11 14:12:05 2024] <TASK>
> > > > > > [Mon Mar 11 14:12:05 2024] ? show_trace_log_lvl+0x1c4/0x2df
> > > > > > [Mon Mar 11 14:12:05 2024] ? show_trace_log_lvl+0x1c4/0x2df
> > > > > > [Mon Mar 11 14:12:05 2024] ? __break_lease+0x16f/0x5f0
> > > > > > [Mon Mar 11 14:12:05 2024] ? nfsd_break_deleg_cb+0x170/0x190
> > > > > > [nfsd]
> > > > > > [Mon Mar 11 14:12:05 2024] ? __warn+0x81/0x110
> > > > > > [Mon Mar 11 14:12:05 2024] ? nfsd_break_deleg_cb+0x170/0x190
> > > > > > [nfsd]
> > > > > > [Mon Mar 11 14:12:05 2024] ? report_bug+0x10a/0x140
> > > > > > [Mon Mar 11 14:12:05 2024] ? handle_bug+0x3c/0x70
> > > > > > [Mon Mar 11 14:12:05 2024] ? exc_invalid_op+0x14/0x70
> > > > > > [Mon Mar 11 14:12:05 2024] ? asm_exc_invalid_op+0x16/0x20
> > > > > > [Mon Mar 11 14:12:05 2024] ? nfsd_break_deleg_cb+0x170/0x190
> > > > > > [nfsd]
> > > > > > [Mon Mar 11 14:12:05 2024] __break_lease+0x16f/0x5f0
> > > > > > [Mon Mar 11 14:12:05 2024] ?
> > > > > > nfsd_file_lookup_locked+0x117/0x160 [nfsd]
> > > > > > [Mon Mar 11 14:12:05 2024] ? list_lru_del+0x101/0x150
> > > > > > [Mon Mar 11 14:12:05 2024] nfsd_file_do_acquire+0x790/0x830
> > > > > > [nfsd]
> > > > > > [Mon Mar 11 14:12:05 2024] nfs4_get_vfs_file+0x315/0x3a0 [nfsd]
> > > > > > [Mon Mar 11 14:12:05 2024] nfsd4_process_open2+0x430/0xa30 [nfsd]
> > > > > > [Mon Mar 11 14:12:05 2024] ? fh_verify+0x297/0x2f0 [nfsd]
> > > > > > [Mon Mar 11 14:12:05 2024] nfsd4_open+0x3ce/0x4b0 [nfsd]
> > > > > > [Mon Mar 11 14:12:05 2024] nfsd4_proc_compound+0x44b/0x700 [nfsd]
> > > > > > [Mon Mar 11 14:12:05 2024] nfsd_dispatch+0x94/0x1c0 [nfsd]
> > > > > > [Mon Mar 11 14:12:05 2024] svc_process_common+0x2ec/0x660
> > > > > > [sunrpc]
> > > > > > [Mon Mar 11 14:12:05 2024] ? __pfx_nfsd_dispatch+0x10/0x10 [nfsd]
> > > > > > [Mon Mar 11 14:12:05 2024] ? __pfx_nfsd+0x10/0x10 [nfsd]
> > > > > > [Mon Mar 11 14:12:05 2024] svc_process+0x12d/0x170 [sunrpc]
> > > > > > [Mon Mar 11 14:12:05 2024] nfsd+0x84/0xb0 [nfsd]
> > > > > > [Mon Mar 11 14:12:05 2024] kthread+0xdd/0x100
> > > > > > [Mon Mar 11 14:12:05 2024] ? __pfx_kthread+0x10/0x10
> > > > > > [Mon Mar 11 14:12:05 2024] ret_from_fork+0x29/0x50
> > > > > > [Mon Mar 11 14:12:05 2024] </TASK>
> > > > > > [Mon Mar 11 14:12:05 2024] ---[ end trace 7a039e17443dc651 ]---
> > > > > This is probably this WARN in nfsd_break_one_deleg:
> > > > >
> > > > > WARN_ON_ONCE(!nfsd4_run_cb(&dp->dl_recall));
> > > > >
> > > > > It means that a delegation break callback to the client couldn't be
> > > > > queued to the workqueue, and so it didn't run.
> > > > >
> > > > > > Could this be the same issue as described
> > > > > > here:https://lore.kernel.org/linux-nfs/af0ec881-5ebf-4feb-98ae-3ed2a77f86f1@oracle.com/
> > > > > > ?
> > > > > Yes, most likely the same problem.
> > > > If I read that thread correctly, this issue was introduced between
> > > > 6.1.63 and 6.6.3? Is it possible the EL9 5.14.0-362.18.1.el9_3
> > > > backported these changes, or were we hitting some other bug with that
> > > > version? It seems the 6.1.x kernel is not affected? If so, that
> > > > would be
> > > > the recommended kernel to run?
> > > Anything is possible. We have to identify the problem first.
> > > > > > As described in that thread, I've tried to obtain the requested
> > > > > > information.
> > > > > >
> > > > > > Is it possible this is the issue that was fixed by the patches
> > > > > > described
> > > > > > here?https://lore.kernel.org/linux-nfs/2024022054-cause-suffering-eae8@gregkh/
> > > > > >
> > > > > Doubtful. Those are targeted toward a different set of issues.
> > > > >
> > > > > If you're willing, I do have some patches queued up for CentOS here
> > > > > that
> > > > > fix some backchannel problems that could be related. I'm mainly
> > > > > waiting
> > > > > on Chuck to send these to Linus and then we'll likely merge them into
> > > > > CentOS soon afterward:
> > > > >
> > > > > https://gitlab.com/redhat/centos-stream/src/kernel/centos-stream-9/-/merge_requests/3689
> > > > >
> > > > >
> > > > If you can send me a patch file, I can rebuild the C9S kernel with that
> > > > patch and run it. It can take a while for the bug to trigger as I
> > > > believe it seems to be very workload dependent (we were running very
> > > > stable for months and now hit this bug every other week).
> > > >
> > > >
> > > It's probably simpler to just pull down the build artifacts for that MR.
> > > You have to drill down through the CI for it, but they are here:
> > >
> > > https://s3.amazonaws.com/arr-cki-prod-trusted-artifacts/index.html?prefix=trusted-artifacts/1194300175/publish_x86_64/6278921877/artifacts/
> > >
> > >
> > > There's even a repo file you can install on the box to pull them down.
> >
> > We installed this kernel on the server 3 days ago. Today, a user
> > informed us that their screen was black after logging in. Similar to
> > other occurrences of this issue, the mount command on the client was
> > hung. But in contrast to the other times, there were no messages in
> > the logs kernel logs on the server. Even restarting the client does
> > not resolve the issue.
Ok, so you rebooted the client and it's still unable to mount? That
sounds like a server problem if so.
Are both client and server running the same kernel?
> >
> > Something still seems to be wrong on the server though. When I look at
> > the directories under /proc/fs/nfsd/clients, there's still a directory
> > for the specific client, even though it's no longer running:
> >
> > # cat 155/info
> > clientid: 0xc8edb7f65f4a9ad
> > address: "10.87.31.152:819"
> > status: confirmed
> > seconds from last renew: 33163
> > name: "Linux NFSv4.2 bersalis.esat.kuleuven.be"
> > minor version: 2
> > Implementation domain: "kernel.org"
> > Implementation name: "Linux 5.14.0-362.18.1.el9_3.0.1.x86_64 #1 SMP
> > PREEMPT_DYNAMIC Sun Feb 11 13:49:23 UTC 2024 x86_64"
> > Implementation time: [0, 0]
> > callback state: DOWN
> > callback address: 10.87.31.152:0
> >
If you just shut down the client, the server won't immediately purge its
record. In fact, assuming you're running the same kernel on the server,
it won't purge the client record until there is a conflicting request
for its state.
> The nfsdclnts command for this client shows the following delegations:
>
> # nfsdclnts -f 155/states -t all
> Inode number | Type | Access | Deny | ip address | Filename
> 169346743 | open | r- | -- | 10.87.31.152:819 |
> disconnected dentry
> 169346743 | deleg | r | | 10.87.31.152:819 |
> disconnected dentry
> 169346746 | open | r- | -- | 10.87.31.152:819 |
> disconnected dentry
> 169346746 | deleg | r | | 10.87.31.152:819 |
> disconnected dentry
>
> I see a lot of recent patches regarding directory delegations. Could
> this be related to this?
>
> Will a 5.14.0-362.18.1.el9_3.0.1 kernel try to use a directory delegation?
>
>
No. Directory delegations are a new feature that's still under
development. They use some of the same machinery as file delegations,
but they wouldn't be a factor here.
>
> >
> > The system seems to have identified that the client is no longer
> > reachable, but the client entry does not go away. When a mount was
> > hanging on the client, there would be two directories in clients for
> > the same client. Killing the mount command clears up the second entry.
> >
> > Even after running conntrack -D on the server to remove the tcp
> > connection from the conntrack table, the entry doesn't go away and the
> > client still can not mount anything from the server.
> >
> > A tcpdump on the client while a mount was running logged the following
> > messages over and over again:
> >
> > request:
> >
> > Frame 1: 378 bytes on wire (3024 bits), 378 bytes captured (3024 bits)
> > Ethernet II, Src: HP_19:7d:4b (e0:73:e7:19:7d:4b), Dst:
> > ArubaaHe_f9:8e:00 (88:3a:30:f9:8e:00)
> > Internet Protocol Version 4, Src: 10.87.31.152, Dst: 10.86.18.14
> > Transmission Control Protocol, Src Port: 932, Dst Port: 2049, Seq: 1,
> > Ack: 1, Len: 312
> > Remote Procedure Call, Type:Call XID:0x1d3220c4
> > Network File System
> > [Program Version: 4]
> > [V4 Procedure: COMPOUND (1)]
> > GSS Data, Ops(1): CREATE_SESSION
> > Length: 152
> > GSS Sequence Number: 76
> > Tag: <EMPTY>
> > minorversion: 2
> > Operations (count: 1): CREATE_SESSION
> > [Main Opcode: CREATE_SESSION (43)]
> > GSS Checksum:
> > 00000028040404ffffffffff000000002c19055f1f8d442d594c13849628affc2797cbb2…
> > GSS Token Length: 40
> > GSS-API Generic Security Service Application Program Interface
> > krb5_blob:
> > 040404ffffffffff000000002c19055f1f8d442d594c13849628affc2797cbb23fa080b0…
> >
> > response:
> >
> > Frame 2: 206 bytes on wire (1648 bits), 206 bytes captured (1648 bits)
> > Ethernet II, Src: ArubaaHe_f9:8e:00 (88:3a:30:f9:8e:00), Dst:
> > HP_19:7d:4b (e0:73:e7:19:7d:4b)
> > Internet Protocol Version 4, Src: 10.86.18.14, Dst: 10.87.31.152
> > Transmission Control Protocol, Src Port: 2049, Dst Port: 932, Seq: 1,
> > Ack: 313, Len: 140
> > Remote Procedure Call, Type:Reply XID:0x1d3220c4
> > Network File System
> > [Program Version: 4]
> > [V4 Procedure: COMPOUND (1)]
> > GSS Data, Ops(1): CREATE_SESSION(NFS4ERR_DELAY)
> > Length: 24
> > GSS Sequence Number: 76
> > Status: NFS4ERR_DELAY (10008)
> > Tag: <EMPTY>
> > Operations (count: 1)
> > [Main Opcode: CREATE_SESSION (43)]
> > GSS Checksum:
> > 00000028040405ffffffffff000000000aa742d0798deaad1a8aa2d7c3a91bf4f6274222…
> > GSS Token Length: 40
> > GSS-API Generic Security Service Application Program Interface
> > krb5_blob:
> > 040405ffffffffff000000000aa742d0798deaad1a8aa2d7c3a91bf4f627422226d74923…
> >
> > I was hoping that giving the client a different IP address would
> > resolve the issue for this client, but it didn't. Even though the
> > client had a new IP address (hostname was kept the same), it failed to
> > mount anything from the server.
> >
Changing the IP address won't help. The client is probably using the
same long-form client id as before, so the server still identifies the
client even with the address change.
Unfortunately, the cause of an NFS4ERR_DELAY error is tough to guess.
The client is expected to back off and retry, so if the server keeps
returning that repeatedly, then a hung mount command is expected.
The question is why the server would keep returning DELAY. A lot of
different problems ranging from memory allocation issues to protocol
problems can result in that error. You may want to check the NFS server
and see if anything was logged there.
This is on a CREATE_SESSION call, so I wonder if the record held by the
(courteous) server is somehow blocking the attempt to reestablish the
session?
Do you have a way to reproduce this? Since this is a centos kernel, you
could follow the page here to open a bug:
https://wiki.centos.org/ReportBugs.html
> > I created another dump of the workqueues and worker pools on the server:
> >
> > [Mon Mar 18 14:59:33 2024] Showing busy workqueues and worker pools:
> > [Mon Mar 18 14:59:33 2024] workqueue events: flags=0x0
> > [Mon Mar 18 14:59:33 2024] pwq 54: cpus=27 node=1 flags=0x0 nice=0
> > active=1/256 refcnt=2
> > [Mon Mar 18 14:59:33 2024] pending: drm_fb_helper_damage_work
> > [drm_kms_helper]
> > [Mon Mar 18 14:59:33 2024] workqueue events_power_efficient: flags=0x80
> > [Mon Mar 18 14:59:33 2024] pwq 54: cpus=27 node=1 flags=0x0 nice=0
> > active=1/256 refcnt=2
> > [Mon Mar 18 14:59:33 2024] pending: fb_flashcursor
> > [Mon Mar 18 14:59:33 2024] workqueue mm_percpu_wq: flags=0x8
> > [Mon Mar 18 14:59:33 2024] pwq 54: cpus=27 node=1 flags=0x0 nice=0
> > active=1/256 refcnt=3
> > [Mon Mar 18 14:59:33 2024] pending: lru_add_drain_per_cpu BAR(362)
> > [Mon Mar 18 14:59:33 2024] workqueue kblockd: flags=0x18
> > [Mon Mar 18 14:59:33 2024] pwq 55: cpus=27 node=1 flags=0x0 nice=-20
> > active=1/256 refcnt=2
> > [Mon Mar 18 14:59:33 2024] pending: blk_mq_timeout_work
> >
> >
> > In contrast to last time, it doesn't show anything regarding nfs this
> > time.
> >
> > I also tried the suggestion from Dai Ngo (echo 3 >
> > /proc/sys/vm/drop_caches), but that didn't seem to make any difference.
> >
> > We haven't restarted the server yet as it seems the impact seems to
> > affect fewer clients that before. Is there anything we can run on the
> > server to further debug this?
> >
> > In the past, the issue seemed to deteriorate rapidly and resulted in
> > issues for almost all clients after about 20 minutes. This time the
> > impact seems to be less, but it's not gone.
> >
> > How can we force the NFS server to forget about a specific client? I
> > haven't tried to restart the nfs service yet as I'm afraid it will
> > fail to stop as before.
> >
Not with that kernel. There are some new administrative interfaces that
might allow that in the future, but they were just merged upstream and
aren't in that kernel.
--
Jeff Layton <jlayton@kernel.org>
^ permalink raw reply [flat|nested] 26+ messages in thread
* Re: nfsd hangs and nfsd_break_deleg_cb+0x170/0x190 warning
2024-03-18 21:54 ` Jeff Layton
@ 2024-03-19 7:58 ` Rik Theys
2024-03-19 10:39 ` Jeff Layton
2024-03-19 17:09 ` Dai Ngo
0 siblings, 2 replies; 26+ messages in thread
From: Rik Theys @ 2024-03-19 7:58 UTC (permalink / raw)
To: Jeff Layton, Linux Nfs
Hi,
On 3/18/24 22:54, Jeff Layton wrote:
> On Mon, 2024-03-18 at 22:15 +0100, Rik Theys wrote:
>> Hi,
>>
>> On 3/18/24 21:21, Rik Theys wrote:
>>> Hi Jeff,
>>>
>>> On 3/12/24 13:47, Jeff Layton wrote:
>>>> On Tue, 2024-03-12 at 13:24 +0100, Rik Theys wrote:
>>>>> Hi Jeff,
>>>>>
>>>>> On 3/12/24 12:22, Jeff Layton wrote:
>>>>>> On Mon, 2024-03-11 at 19:43 +0100, Rik Theys wrote:
>>>>>>> Since a few weeks our Rocky Linux 9 NFS server has periodically
>>>>>>> logged hung nfsd tasks. The initial effect was that some clients
>>>>>>> could no longer access the NFS server. This got worse and worse
>>>>>>> (probably as more nfsd threads got blocked) and we had to restart
>>>>>>> the server. Restarting the server also failed as the NFS server
>>>>>>> service could no longer be stopped.
>>>>>>>
>>>>>>> The initial kernel we noticed this behavior on was
>>>>>>> kernel-5.14.0-362.18.1.el9_3.x86_64. Since then we've installed
>>>>>>> kernel-5.14.0-419.el9.x86_64 from CentOS Stream 9. The same issue
>>>>>>> happened again on this newer kernel version:
>>>> 419 is fairly up to date with nfsd changes. There are some known bugs
>>>> around callbacks, and there is a draft MR in flight to fix it.
>>>>
>>>> What kernel were you on prior to 5.14.0-362.18.1.el9_3.x86_64 ? If we
>>>> can bracket the changes around a particular version, then that might
>>>> help identify the problem.
>>>>
>>>>>>> [Mon Mar 11 14:10:08 2024] Not tainted 5.14.0-419.el9.x86_64 #1
>>>>>>> [Mon Mar 11 14:10:08 2024] "echo 0 >
>>>>>>> /proc/sys/kernel/hung_task_timeout_secs" disables this message.
>>>>>>> [Mon Mar 11 14:10:08 2024]task:nfsd state:D stack:0
>>>>>>> pid:8865 ppid:2 flags:0x00004000
>>>>>>> [Mon Mar 11 14:10:08 2024] Call Trace:
>>>>>>> [Mon Mar 11 14:10:08 2024] <TASK>
>>>>>>> [Mon Mar 11 14:10:08 2024] __schedule+0x21b/0x550
>>>>>>> [Mon Mar 11 14:10:08 2024] schedule+0x2d/0x70
>>>>>>> [Mon Mar 11 14:10:08 2024] schedule_timeout+0x11f/0x160
>>>>>>> [Mon Mar 11 14:10:08 2024] ? select_idle_sibling+0x28/0x430
>>>>>>> [Mon Mar 11 14:10:08 2024] ? wake_affine+0x62/0x1f0
>>>>>>> [Mon Mar 11 14:10:08 2024] __wait_for_common+0x90/0x1d0
>>>>>>> [Mon Mar 11 14:10:08 2024] ? __pfx_schedule_timeout+0x10/0x10
>>>>>>> [Mon Mar 11 14:10:08 2024] __flush_workqueue+0x13a/0x3f0
>>>>>>> [Mon Mar 11 14:10:08 2024] nfsd4_shutdown_callback+0x49/0x120
>>>>>>> [nfsd]
>>>>>>> [Mon Mar 11 14:10:08 2024] ? nfsd4_cld_remove+0x54/0x1d0 [nfsd]
>>>>>>> [Mon Mar 11 14:10:08 2024] ?
>>>>>>> nfsd4_return_all_client_layouts+0xc4/0xf0 [nfsd]
>>>>>>> [Mon Mar 11 14:10:08 2024] ? nfsd4_shutdown_copy+0x68/0xc0 [nfsd]
>>>>>>> [Mon Mar 11 14:10:08 2024] __destroy_client+0x1f3/0x290 [nfsd]
>>>>>>> [Mon Mar 11 14:10:08 2024] nfsd4_exchange_id+0x75f/0x770 [nfsd]
>>>>>>> [Mon Mar 11 14:10:08 2024] ? nfsd4_decode_opaque+0x3a/0x90 [nfsd]
>>>>>>> [Mon Mar 11 14:10:08 2024] nfsd4_proc_compound+0x44b/0x700 [nfsd]
>>>>>>> [Mon Mar 11 14:10:08 2024] nfsd_dispatch+0x94/0x1c0 [nfsd]
>>>>>>> [Mon Mar 11 14:10:08 2024] svc_process_common+0x2ec/0x660
>>>>>>> [sunrpc]
>>>>>>> [Mon Mar 11 14:10:08 2024] ? __pfx_nfsd_dispatch+0x10/0x10 [nfsd]
>>>>>>> [Mon Mar 11 14:10:08 2024] ? __pfx_nfsd+0x10/0x10 [nfsd]
>>>>>>> [Mon Mar 11 14:10:08 2024] svc_process+0x12d/0x170 [sunrpc]
>>>>>>> [Mon Mar 11 14:10:08 2024] nfsd+0x84/0xb0 [nfsd]
>>>>>>> [Mon Mar 11 14:10:08 2024] kthread+0xdd/0x100
>>>>>>> [Mon Mar 11 14:10:08 2024] ? __pfx_kthread+0x10/0x10
>>>>>>> [Mon Mar 11 14:10:08 2024] ret_from_fork+0x29/0x50
>>>>>>> [Mon Mar 11 14:10:08 2024] </TASK>
>>>>>>> [Mon Mar 11 14:10:08 2024] INFO: task nfsd:8866 blocked for
>>>>>>> more than 122 seconds.
>>>>>>> [Mon Mar 11 14:10:08 2024] Not tainted
>>>>>>> 5.14.0-419.el9.x86_64 #1
>>>>>>> [Mon Mar 11 14:10:08 2024] "echo 0 >
>>>>>>> /proc/sys/kernel/hung_task_timeout_secs" disables this message.
>>>>>>> [Mon Mar 11 14:10:08 2024]task:nfsd state:D stack:0
>>>>>>> pid:8866 ppid:2 flags:0x00004000
>>>>>>> [Mon Mar 11 14:10:08 2024] Call Trace:
>>>>>>> [Mon Mar 11 14:10:08 2024] <TASK>
>>>>>>> [Mon Mar 11 14:10:08 2024] __schedule+0x21b/0x550
>>>>>>> [Mon Mar 11 14:10:08 2024] schedule+0x2d/0x70
>>>>>>> [Mon Mar 11 14:10:08 2024] schedule_timeout+0x11f/0x160
>>>>>>> [Mon Mar 11 14:10:08 2024] ? select_idle_sibling+0x28/0x430
>>>>>>> [Mon Mar 11 14:10:08 2024] ? tcp_recvmsg+0x196/0x210
>>>>>>> [Mon Mar 11 14:10:08 2024] ? wake_affine+0x62/0x1f0
>>>>>>> [Mon Mar 11 14:10:08 2024] __wait_for_common+0x90/0x1d0
>>>>>>> [Mon Mar 11 14:10:08 2024] ? __pfx_schedule_timeout+0x10/0x10
>>>>>>> [Mon Mar 11 14:10:08 2024] __flush_workqueue+0x13a/0x3f0
>>>>>>> [Mon Mar 11 14:10:08 2024] nfsd4_destroy_session+0x1a4/0x240
>>>>>>> [nfsd]
>>>>>>> [Mon Mar 11 14:10:08 2024] nfsd4_proc_compound+0x44b/0x700 [nfsd]
>>>>>>> [Mon Mar 11 14:10:08 2024] nfsd_dispatch+0x94/0x1c0 [nfsd]
>>>>>>> [Mon Mar 11 14:10:08 2024] svc_process_common+0x2ec/0x660
>>>>>>> [sunrpc]
>>>>>>> [Mon Mar 11 14:10:08 2024] ? __pfx_nfsd_dispatch+0x10/0x10 [nfsd]
>>>>>>> [Mon Mar 11 14:10:08 2024] ? __pfx_nfsd+0x10/0x10 [nfsd]
>>>>>>> [Mon Mar 11 14:10:08 2024] svc_process+0x12d/0x170 [sunrpc]
>>>>>>> [Mon Mar 11 14:10:08 2024] nfsd+0x84/0xb0 [nfsd]
>>>>>>> [Mon Mar 11 14:10:08 2024] kthread+0xdd/0x100
>>>>>>> [Mon Mar 11 14:10:08 2024] ? __pfx_kthread+0x10/0x10
>>>>>>> [Mon Mar 11 14:10:08 2024] ret_from_fork+0x29/0x50
>>>>>>> [Mon Mar 11 14:10:08 2024] </TASK>
>>>>>>>
>>>>>> The above threads are trying to flush the workqueue, so that probably
>>>>>> means that they are stuck waiting on a workqueue job to finish.
>>>>>>> The above is repeated a few times, and then this warning is
>>>>>>> also logged:
>>>>>>> [Mon Mar 11 14:12:04 2024] ------------[ cut here ]------------
>>>>>>> [Mon Mar 11 14:12:04 2024] WARNING: CPU: 39 PID: 8844 at
>>>>>>> fs/nfsd/nfs4state.c:4919 nfsd_break_deleg_cb+0x170/0x190 [nfsd]
>>>>>>> [Mon Mar 11 14:12:05 2024] Modules linked in: nfsv4
>>>>>>> dns_resolver nfs fscache netfs rpcsec_gss_krb5 rpcrdma rdma_cm
>>>>>>> iw_cm ib_cm ib_core binfmt_misc bonding tls rfkill nft_counter nft_ct
>>>>>>> nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_reject_inet
>>>>>>> nf_reject_ipv4 nf_reject_ipv6 nft_reject nf_tables nfnetlink vfat
>>>>>>> fat dm_thin_pool dm_persistent_data dm_bio_prison dm_bufio l
>>>>>>> ibcrc32c dm_service_time dm_multipath intel_rapl_msr
>>>>>>> intel_rapl_common intel_uncore_frequency
>>>>>>> intel_uncore_frequency_common isst_if_common skx_edac nfit
>>>>>>> libnvdimm ipmi_ssif x86_pkg_temp
>>>>>>> _thermal intel_powerclamp coretemp kvm_intel kvm irqbypass
>>>>>>> dcdbas rapl intel_cstate mgag200 i2c_algo_bit drm_shmem_helper
>>>>>>> dell_smbios drm_kms_helper dell_wmi_descriptor wmi_bmof intel_u
>>>>>>> ncore syscopyarea pcspkr sysfillrect mei_me sysimgblt acpi_ipmi
>>>>>>> mei fb_sys_fops i2c_i801 ipmi_si intel_pch_thermal lpc_ich
>>>>>>> ipmi_devintf i2c_smbus ipmi_msghandler joydev acpi_power_meter
>>>>>>> nfsd auth_rpcgss nfs_acl drm lockd grace fuse sunrpc ext4
>>>>>>> mbcache jbd2 sd_mod sg lpfc
>>>>>>> [Mon Mar 11 14:12:05 2024] nvmet_fc nvmet nvme_fc nvme_fabrics
>>>>>>> crct10dif_pclmul ahci libahci crc32_pclmul nvme_core crc32c_intel
>>>>>>> ixgbe megaraid_sas libata nvme_common ghash_clmulni_int
>>>>>>> el t10_pi wdat_wdt scsi_transport_fc mdio wmi dca dm_mirror
>>>>>>> dm_region_hash dm_log dm_mod
>>>>>>> [Mon Mar 11 14:12:05 2024] CPU: 39 PID: 8844 Comm: nfsd Not
>>>>>>> tainted 5.14.0-419.el9.x86_64 #1
>>>>>>> [Mon Mar 11 14:12:05 2024] Hardware name: Dell Inc. PowerEdge
>>>>>>> R740/00WGD1, BIOS 2.20.1 09/13/2023
>>>>>>> [Mon Mar 11 14:12:05 2024] RIP:
>>>>>>> 0010:nfsd_break_deleg_cb+0x170/0x190 [nfsd]
>>>>>>> [Mon Mar 11 14:12:05 2024] Code: a6 95 c5 f3 e9 ff fe ff ff 48
>>>>>>> 89 df be 01 00 00 00 e8 34 b5 13 f4 48 8d bb 98 00 00 00 e8 c8 f9
>>>>>>> 00 00 84 c0 0f 85 2e ff ff ff <0f> 0b e9 27 ff ff ff be
>>>>>>> 02 00 00 00 48 89 df e8 0c b5 13 f4 e9 01
>>>>>>> [Mon Mar 11 14:12:05 2024] RSP: 0018:ffff9929e0bb7b80 EFLAGS:
>>>>>>> 00010246
>>>>>>> [Mon Mar 11 14:12:05 2024] RAX: 0000000000000000 RBX:
>>>>>>> ffff8ada51930900 RCX: 0000000000000024
>>>>>>> [Mon Mar 11 14:12:05 2024] RDX: ffff8ada519309c8 RSI:
>>>>>>> ffff8ad582933c00 RDI: 0000000000002000
>>>>>>> [Mon Mar 11 14:12:05 2024] RBP: ffff8ad46bf21574 R08:
>>>>>>> ffff9929e0bb7b48 R09: 0000000000000000
>>>>>>> [Mon Mar 11 14:12:05 2024] R10: ffff8aec859a2948 R11:
>>>>>>> 0000000000000000 R12: ffff8ad6f497c360
>>>>>>> [Mon Mar 11 14:12:05 2024] R13: ffff8ad46bf21560 R14:
>>>>>>> ffff8ae5942e0b10 R15: ffff8ad6f497c360
>>>>>>> [Mon Mar 11 14:12:05 2024] FS: 0000000000000000(0000)
>>>>>>> GS:ffff8b031fcc0000(0000) knlGS:0000000000000000
>>>>>>> [Mon Mar 11 14:12:05 2024] CS: 0010 DS: 0000 ES: 0000 CR0:
>>>>>>> 0000000080050033
>>>>>>> [Mon Mar 11 14:12:05 2024] CR2: 00007fafe2060744 CR3:
>>>>>>> 00000018e58de006 CR4: 00000000007706e0
>>>>>>> [Mon Mar 11 14:12:05 2024] DR0: 0000000000000000 DR1:
>>>>>>> 0000000000000000 DR2: 0000000000000000
>>>>>>> [Mon Mar 11 14:12:05 2024] DR3: 0000000000000000 DR6:
>>>>>>> 00000000fffe0ff0 DR7: 0000000000000400
>>>>>>> [Mon Mar 11 14:12:05 2024] PKRU: 55555554
>>>>>>> [Mon Mar 11 14:12:05 2024] Call Trace:
>>>>>>> [Mon Mar 11 14:12:05 2024] <TASK>
>>>>>>> [Mon Mar 11 14:12:05 2024] ? show_trace_log_lvl+0x1c4/0x2df
>>>>>>> [Mon Mar 11 14:12:05 2024] ? show_trace_log_lvl+0x1c4/0x2df
>>>>>>> [Mon Mar 11 14:12:05 2024] ? __break_lease+0x16f/0x5f0
>>>>>>> [Mon Mar 11 14:12:05 2024] ? nfsd_break_deleg_cb+0x170/0x190
>>>>>>> [nfsd]
>>>>>>> [Mon Mar 11 14:12:05 2024] ? __warn+0x81/0x110
>>>>>>> [Mon Mar 11 14:12:05 2024] ? nfsd_break_deleg_cb+0x170/0x190
>>>>>>> [nfsd]
>>>>>>> [Mon Mar 11 14:12:05 2024] ? report_bug+0x10a/0x140
>>>>>>> [Mon Mar 11 14:12:05 2024] ? handle_bug+0x3c/0x70
>>>>>>> [Mon Mar 11 14:12:05 2024] ? exc_invalid_op+0x14/0x70
>>>>>>> [Mon Mar 11 14:12:05 2024] ? asm_exc_invalid_op+0x16/0x20
>>>>>>> [Mon Mar 11 14:12:05 2024] ? nfsd_break_deleg_cb+0x170/0x190
>>>>>>> [nfsd]
>>>>>>> [Mon Mar 11 14:12:05 2024] __break_lease+0x16f/0x5f0
>>>>>>> [Mon Mar 11 14:12:05 2024] ?
>>>>>>> nfsd_file_lookup_locked+0x117/0x160 [nfsd]
>>>>>>> [Mon Mar 11 14:12:05 2024] ? list_lru_del+0x101/0x150
>>>>>>> [Mon Mar 11 14:12:05 2024] nfsd_file_do_acquire+0x790/0x830
>>>>>>> [nfsd]
>>>>>>> [Mon Mar 11 14:12:05 2024] nfs4_get_vfs_file+0x315/0x3a0 [nfsd]
>>>>>>> [Mon Mar 11 14:12:05 2024] nfsd4_process_open2+0x430/0xa30 [nfsd]
>>>>>>> [Mon Mar 11 14:12:05 2024] ? fh_verify+0x297/0x2f0 [nfsd]
>>>>>>> [Mon Mar 11 14:12:05 2024] nfsd4_open+0x3ce/0x4b0 [nfsd]
>>>>>>> [Mon Mar 11 14:12:05 2024] nfsd4_proc_compound+0x44b/0x700 [nfsd]
>>>>>>> [Mon Mar 11 14:12:05 2024] nfsd_dispatch+0x94/0x1c0 [nfsd]
>>>>>>> [Mon Mar 11 14:12:05 2024] svc_process_common+0x2ec/0x660
>>>>>>> [sunrpc]
>>>>>>> [Mon Mar 11 14:12:05 2024] ? __pfx_nfsd_dispatch+0x10/0x10 [nfsd]
>>>>>>> [Mon Mar 11 14:12:05 2024] ? __pfx_nfsd+0x10/0x10 [nfsd]
>>>>>>> [Mon Mar 11 14:12:05 2024] svc_process+0x12d/0x170 [sunrpc]
>>>>>>> [Mon Mar 11 14:12:05 2024] nfsd+0x84/0xb0 [nfsd]
>>>>>>> [Mon Mar 11 14:12:05 2024] kthread+0xdd/0x100
>>>>>>> [Mon Mar 11 14:12:05 2024] ? __pfx_kthread+0x10/0x10
>>>>>>> [Mon Mar 11 14:12:05 2024] ret_from_fork+0x29/0x50
>>>>>>> [Mon Mar 11 14:12:05 2024] </TASK>
>>>>>>> [Mon Mar 11 14:12:05 2024] ---[ end trace 7a039e17443dc651 ]---
>>>>>> This is probably this WARN in nfsd_break_one_deleg:
>>>>>>
>>>>>> WARN_ON_ONCE(!nfsd4_run_cb(&dp->dl_recall));
>>>>>>
>>>>>> It means that a delegation break callback to the client couldn't be
>>>>>> queued to the workqueue, and so it didn't run.
>>>>>>
>>>>>>> Could this be the same issue as described
>>>>>>> here:https://lore.kernel.org/linux-nfs/af0ec881-5ebf-4feb-98ae-3ed2a77f86f1@oracle.com/
>>>>>>> ?
>>>>>> Yes, most likely the same problem.
>>>>> If I read that thread correctly, this issue was introduced between
>>>>> 6.1.63 and 6.6.3? Is it possible the EL9 5.14.0-362.18.1.el9_3
>>>>> backported these changes, or were we hitting some other bug with that
>>>>> version? It seems the 6.1.x kernel is not affected? If so, that
>>>>> would be
>>>>> the recommended kernel to run?
>>>> Anything is possible. We have to identify the problem first.
>>>>>>> As described in that thread, I've tried to obtain the requested
>>>>>>> information.
>>>>>>>
>>>>>>> Is it possible this is the issue that was fixed by the patches
>>>>>>> described
>>>>>>> here?https://lore.kernel.org/linux-nfs/2024022054-cause-suffering-eae8@gregkh/
>>>>>>>
>>>>>> Doubtful. Those are targeted toward a different set of issues.
>>>>>>
>>>>>> If you're willing, I do have some patches queued up for CentOS here
>>>>>> that
>>>>>> fix some backchannel problems that could be related. I'm mainly
>>>>>> waiting
>>>>>> on Chuck to send these to Linus and then we'll likely merge them into
>>>>>> CentOS soon afterward:
>>>>>>
>>>>>> https://gitlab.com/redhat/centos-stream/src/kernel/centos-stream-9/-/merge_requests/3689
>>>>>>
>>>>>>
>>>>> If you can send me a patch file, I can rebuild the C9S kernel with that
>>>>> patch and run it. It can take a while for the bug to trigger as I
>>>>> believe it seems to be very workload dependent (we were running very
>>>>> stable for months and now hit this bug every other week).
>>>>>
>>>>>
>>>> It's probably simpler to just pull down the build artifacts for that MR.
>>>> You have to drill down through the CI for it, but they are here:
>>>>
>>>> https://s3.amazonaws.com/arr-cki-prod-trusted-artifacts/index.html?prefix=trusted-artifacts/1194300175/publish_x86_64/6278921877/artifacts/
>>>>
>>>>
>>>> There's even a repo file you can install on the box to pull them down.
>>> We installed this kernel on the server 3 days ago. Today, a user
>>> informed us that their screen was black after logging in. Similar to
>>> other occurrences of this issue, the mount command on the client was
>>> hung. But in contrast to the other times, there were no messages in
>>> the logs kernel logs on the server. Even restarting the client does
>>> not resolve the issue.
>
> Ok, so you rebooted the client and it's still unable to mount? That
> sounds like a server problem if so.
>
> Are both client and server running the same kernel?
No, the server runs 5.14.0-427.3689_1194299994.el9 and the client
5.14.0-362.18.1.el9_3.
>
>>> Something still seems to be wrong on the server though. When I look at
>>> the directories under /proc/fs/nfsd/clients, there's still a directory
>>> for the specific client, even though it's no longer running:
>>>
>>> # cat 155/info
>>> clientid: 0xc8edb7f65f4a9ad
>>> address: "10.87.31.152:819"
>>> status: confirmed
>>> seconds from last renew: 33163
>>> name: "Linux NFSv4.2 bersalis.esat.kuleuven.be"
>>> minor version: 2
>>> Implementation domain: "kernel.org"
>>> Implementation name: "Linux 5.14.0-362.18.1.el9_3.0.1.x86_64 #1 SMP
>>> PREEMPT_DYNAMIC Sun Feb 11 13:49:23 UTC 2024 x86_64"
>>> Implementation time: [0, 0]
>>> callback state: DOWN
>>> callback address: 10.87.31.152:0
>>>
> If you just shut down the client, the server won't immediately purge its
> record. In fact, assuming you're running the same kernel on the server,
> it won't purge the client record until there is a conflicting request
> for its state.
Is there a way to force such a conflicting request (to get the client
record to purge)?
>
>
>> The nfsdclnts command for this client shows the following delegations:
>>
>> # nfsdclnts -f 155/states -t all
>> Inode number | Type | Access | Deny | ip address | Filename
>> 169346743 | open | r- | -- | 10.87.31.152:819 |
>> disconnected dentry
>> 169346743 | deleg | r | | 10.87.31.152:819 |
>> disconnected dentry
>> 169346746 | open | r- | -- | 10.87.31.152:819 |
>> disconnected dentry
>> 169346746 | deleg | r | | 10.87.31.152:819 |
>> disconnected dentry
>>
>> I see a lot of recent patches regarding directory delegations. Could
>> this be related to this?
>>
>> Will a 5.14.0-362.18.1.el9_3.0.1 kernel try to use a directory delegation?
>>
>>
> No. Directory delegations are a new feature that's still under
> development. They use some of the same machinery as file delegations,
> but they wouldn't be a factor here.
>
>>> The system seems to have identified that the client is no longer
>>> reachable, but the client entry does not go away. When a mount was
>>> hanging on the client, there would be two directories in clients for
>>> the same client. Killing the mount command clears up the second entry.
>>>
>>> Even after running conntrack -D on the server to remove the tcp
>>> connection from the conntrack table, the entry doesn't go away and the
>>> client still can not mount anything from the server.
>>>
>>> A tcpdump on the client while a mount was running logged the following
>>> messages over and over again:
>>>
>>> request:
>>>
>>> Frame 1: 378 bytes on wire (3024 bits), 378 bytes captured (3024 bits)
>>> Ethernet II, Src: HP_19:7d:4b (e0:73:e7:19:7d:4b), Dst:
>>> ArubaaHe_f9:8e:00 (88:3a:30:f9:8e:00)
>>> Internet Protocol Version 4, Src: 10.87.31.152, Dst: 10.86.18.14
>>> Transmission Control Protocol, Src Port: 932, Dst Port: 2049, Seq: 1,
>>> Ack: 1, Len: 312
>>> Remote Procedure Call, Type:Call XID:0x1d3220c4
>>> Network File System
>>> [Program Version: 4]
>>> [V4 Procedure: COMPOUND (1)]
>>> GSS Data, Ops(1): CREATE_SESSION
>>> Length: 152
>>> GSS Sequence Number: 76
>>> Tag: <EMPTY>
>>> minorversion: 2
>>> Operations (count: 1): CREATE_SESSION
>>> [Main Opcode: CREATE_SESSION (43)]
>>> GSS Checksum:
>>> 00000028040404ffffffffff000000002c19055f1f8d442d594c13849628affc2797cbb2…
>>> GSS Token Length: 40
>>> GSS-API Generic Security Service Application Program Interface
>>> krb5_blob:
>>> 040404ffffffffff000000002c19055f1f8d442d594c13849628affc2797cbb23fa080b0…
>>>
>>> response:
>>>
>>> Frame 2: 206 bytes on wire (1648 bits), 206 bytes captured (1648 bits)
>>> Ethernet II, Src: ArubaaHe_f9:8e:00 (88:3a:30:f9:8e:00), Dst:
>>> HP_19:7d:4b (e0:73:e7:19:7d:4b)
>>> Internet Protocol Version 4, Src: 10.86.18.14, Dst: 10.87.31.152
>>> Transmission Control Protocol, Src Port: 2049, Dst Port: 932, Seq: 1,
>>> Ack: 313, Len: 140
>>> Remote Procedure Call, Type:Reply XID:0x1d3220c4
>>> Network File System
>>> [Program Version: 4]
>>> [V4 Procedure: COMPOUND (1)]
>>> GSS Data, Ops(1): CREATE_SESSION(NFS4ERR_DELAY)
>>> Length: 24
>>> GSS Sequence Number: 76
>>> Status: NFS4ERR_DELAY (10008)
>>> Tag: <EMPTY>
>>> Operations (count: 1)
>>> [Main Opcode: CREATE_SESSION (43)]
>>> GSS Checksum:
>>> 00000028040405ffffffffff000000000aa742d0798deaad1a8aa2d7c3a91bf4f6274222…
>>> GSS Token Length: 40
>>> GSS-API Generic Security Service Application Program Interface
>>> krb5_blob:
>>> 040405ffffffffff000000000aa742d0798deaad1a8aa2d7c3a91bf4f627422226d74923…
>>>
>>> I was hoping that giving the client a different IP address would
>>> resolve the issue for this client, but it didn't. Even though the
>>> client had a new IP address (hostname was kept the same), it failed to
>>> mount anything from the server.
>>>
> Changing the IP address won't help. The client is probably using the
> same long-form client id as before, so the server still identifies the
> client even with the address change.
How is the client id determined? Will changing the hostname of the
client trigger a change of the client id?
>
> Unfortunately, the cause of an NFS4ERR_DELAY error is tough to guess.
> The client is expected to back off and retry, so if the server keeps
> returning that repeatedly, then a hung mount command is expected.
>
> The question is why the server would keep returning DELAY. A lot of
> different problems ranging from memory allocation issues to protocol
> problems can result in that error. You may want to check the NFS server
> and see if anything was logged there.
There are no messages in the system logs that indicate any sort of
memory issue. We also increased the min_kbytes_free sysctl to 2G on the
server before we restarted it with the newer kernel.
>
> This is on a CREATE_SESSION call, so I wonder if the record held by the
> (courteous) server is somehow blocking the attempt to reestablish the
> session?
>
> Do you have a way to reproduce this? Since this is a centos kernel, you
> could follow the page here to open a bug:
Unfortunately we haven't found a reliable way to reproduce it. But we do
seem to trigger it more and more lately.
Regards,
Rik
>
> https://wiki.centos.org/ReportBugs.html
>
>
>>> I created another dump of the workqueues and worker pools on the server:
>>>
>>> [Mon Mar 18 14:59:33 2024] Showing busy workqueues and worker pools:
>>> [Mon Mar 18 14:59:33 2024] workqueue events: flags=0x0
>>> [Mon Mar 18 14:59:33 2024] pwq 54: cpus=27 node=1 flags=0x0 nice=0
>>> active=1/256 refcnt=2
>>> [Mon Mar 18 14:59:33 2024] pending: drm_fb_helper_damage_work
>>> [drm_kms_helper]
>>> [Mon Mar 18 14:59:33 2024] workqueue events_power_efficient: flags=0x80
>>> [Mon Mar 18 14:59:33 2024] pwq 54: cpus=27 node=1 flags=0x0 nice=0
>>> active=1/256 refcnt=2
>>> [Mon Mar 18 14:59:33 2024] pending: fb_flashcursor
>>> [Mon Mar 18 14:59:33 2024] workqueue mm_percpu_wq: flags=0x8
>>> [Mon Mar 18 14:59:33 2024] pwq 54: cpus=27 node=1 flags=0x0 nice=0
>>> active=1/256 refcnt=3
>>> [Mon Mar 18 14:59:33 2024] pending: lru_add_drain_per_cpu BAR(362)
>>> [Mon Mar 18 14:59:33 2024] workqueue kblockd: flags=0x18
>>> [Mon Mar 18 14:59:33 2024] pwq 55: cpus=27 node=1 flags=0x0 nice=-20
>>> active=1/256 refcnt=2
>>> [Mon Mar 18 14:59:33 2024] pending: blk_mq_timeout_work
>>>
>>>
>>> In contrast to last time, it doesn't show anything regarding nfs this
>>> time.
>>>
>>> I also tried the suggestion from Dai Ngo (echo 3 >
>>> /proc/sys/vm/drop_caches), but that didn't seem to make any difference.
>>>
>>> We haven't restarted the server yet as it seems the impact seems to
>>> affect fewer clients that before. Is there anything we can run on the
>>> server to further debug this?
>>>
>>> In the past, the issue seemed to deteriorate rapidly and resulted in
>>> issues for almost all clients after about 20 minutes. This time the
>>> impact seems to be less, but it's not gone.
>>>
>>> How can we force the NFS server to forget about a specific client? I
>>> haven't tried to restart the nfs service yet as I'm afraid it will
>>> fail to stop as before.
>>>
> Not with that kernel. There are some new administrative interfaces that
> might allow that in the future, but they were just merged upstream and
> aren't in that kernel.
>
> --
> Jeff Layton <jlayton@kernel.org>
--
Rik Theys
System Engineer
KU Leuven - Dept. Elektrotechniek (ESAT)
Kasteelpark Arenberg 10 bus 2440 - B-3001 Leuven-Heverlee
+32(0)16/32.11.07
----------------------------------------------------------------
<<Any errors in spelling, tact or fact are transmission errors>>
^ permalink raw reply [flat|nested] 26+ messages in thread
* Re: nfsd hangs and nfsd_break_deleg_cb+0x170/0x190 warning
2024-03-19 7:58 ` Rik Theys
@ 2024-03-19 10:39 ` Jeff Layton
2024-03-19 10:58 ` Rik Theys
2024-03-19 17:09 ` Dai Ngo
1 sibling, 1 reply; 26+ messages in thread
From: Jeff Layton @ 2024-03-19 10:39 UTC (permalink / raw)
To: Rik Theys, Linux Nfs
On Tue, 2024-03-19 at 08:58 +0100, Rik Theys wrote:
> Hi,
>
> On 3/18/24 22:54, Jeff Layton wrote:
> > On Mon, 2024-03-18 at 22:15 +0100, Rik Theys wrote:
> > > Hi,
> > >
> > > On 3/18/24 21:21, Rik Theys wrote:
> > > > Hi Jeff,
> > > >
> > > > On 3/12/24 13:47, Jeff Layton wrote:
> > > > > On Tue, 2024-03-12 at 13:24 +0100, Rik Theys wrote:
> > > > > > Hi Jeff,
> > > > > >
> > > > > > On 3/12/24 12:22, Jeff Layton wrote:
> > > > > > > On Mon, 2024-03-11 at 19:43 +0100, Rik Theys wrote:
> > > > > > > > Since a few weeks our Rocky Linux 9 NFS server has periodically
> > > > > > > > logged hung nfsd tasks. The initial effect was that some clients
> > > > > > > > could no longer access the NFS server. This got worse and worse
> > > > > > > > (probably as more nfsd threads got blocked) and we had to restart
> > > > > > > > the server. Restarting the server also failed as the NFS server
> > > > > > > > service could no longer be stopped.
> > > > > > > >
> > > > > > > > The initial kernel we noticed this behavior on was
> > > > > > > > kernel-5.14.0-362.18.1.el9_3.x86_64. Since then we've installed
> > > > > > > > kernel-5.14.0-419.el9.x86_64 from CentOS Stream 9. The same issue
> > > > > > > > happened again on this newer kernel version:
> > > > > 419 is fairly up to date with nfsd changes. There are some known bugs
> > > > > around callbacks, and there is a draft MR in flight to fix it.
> > > > >
> > > > > What kernel were you on prior to 5.14.0-362.18.1.el9_3.x86_64 ? If we
> > > > > can bracket the changes around a particular version, then that might
> > > > > help identify the problem.
> > > > >
> > > > > > > > [Mon Mar 11 14:10:08 2024] Not tainted 5.14.0-419.el9.x86_64 #1
> > > > > > > > [Mon Mar 11 14:10:08 2024] "echo 0 >
> > > > > > > > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> > > > > > > > [Mon Mar 11 14:10:08 2024]task:nfsd state:D stack:0
> > > > > > > > pid:8865 ppid:2 flags:0x00004000
> > > > > > > > [Mon Mar 11 14:10:08 2024] Call Trace:
> > > > > > > > [Mon Mar 11 14:10:08 2024] <TASK>
> > > > > > > > [Mon Mar 11 14:10:08 2024] __schedule+0x21b/0x550
> > > > > > > > [Mon Mar 11 14:10:08 2024] schedule+0x2d/0x70
> > > > > > > > [Mon Mar 11 14:10:08 2024] schedule_timeout+0x11f/0x160
> > > > > > > > [Mon Mar 11 14:10:08 2024] ? select_idle_sibling+0x28/0x430
> > > > > > > > [Mon Mar 11 14:10:08 2024] ? wake_affine+0x62/0x1f0
> > > > > > > > [Mon Mar 11 14:10:08 2024] __wait_for_common+0x90/0x1d0
> > > > > > > > [Mon Mar 11 14:10:08 2024] ? __pfx_schedule_timeout+0x10/0x10
> > > > > > > > [Mon Mar 11 14:10:08 2024] __flush_workqueue+0x13a/0x3f0
> > > > > > > > [Mon Mar 11 14:10:08 2024] nfsd4_shutdown_callback+0x49/0x120
> > > > > > > > [nfsd]
> > > > > > > > [Mon Mar 11 14:10:08 2024] ? nfsd4_cld_remove+0x54/0x1d0 [nfsd]
> > > > > > > > [Mon Mar 11 14:10:08 2024] ?
> > > > > > > > nfsd4_return_all_client_layouts+0xc4/0xf0 [nfsd]
> > > > > > > > [Mon Mar 11 14:10:08 2024] ? nfsd4_shutdown_copy+0x68/0xc0 [nfsd]
> > > > > > > > [Mon Mar 11 14:10:08 2024] __destroy_client+0x1f3/0x290 [nfsd]
> > > > > > > > [Mon Mar 11 14:10:08 2024] nfsd4_exchange_id+0x75f/0x770 [nfsd]
> > > > > > > > [Mon Mar 11 14:10:08 2024] ? nfsd4_decode_opaque+0x3a/0x90 [nfsd]
> > > > > > > > [Mon Mar 11 14:10:08 2024] nfsd4_proc_compound+0x44b/0x700 [nfsd]
> > > > > > > > [Mon Mar 11 14:10:08 2024] nfsd_dispatch+0x94/0x1c0 [nfsd]
> > > > > > > > [Mon Mar 11 14:10:08 2024] svc_process_common+0x2ec/0x660
> > > > > > > > [sunrpc]
> > > > > > > > [Mon Mar 11 14:10:08 2024] ? __pfx_nfsd_dispatch+0x10/0x10 [nfsd]
> > > > > > > > [Mon Mar 11 14:10:08 2024] ? __pfx_nfsd+0x10/0x10 [nfsd]
> > > > > > > > [Mon Mar 11 14:10:08 2024] svc_process+0x12d/0x170 [sunrpc]
> > > > > > > > [Mon Mar 11 14:10:08 2024] nfsd+0x84/0xb0 [nfsd]
> > > > > > > > [Mon Mar 11 14:10:08 2024] kthread+0xdd/0x100
> > > > > > > > [Mon Mar 11 14:10:08 2024] ? __pfx_kthread+0x10/0x10
> > > > > > > > [Mon Mar 11 14:10:08 2024] ret_from_fork+0x29/0x50
> > > > > > > > [Mon Mar 11 14:10:08 2024] </TASK>
> > > > > > > > [Mon Mar 11 14:10:08 2024] INFO: task nfsd:8866 blocked for
> > > > > > > > more than 122 seconds.
> > > > > > > > [Mon Mar 11 14:10:08 2024] Not tainted
> > > > > > > > 5.14.0-419.el9.x86_64 #1
> > > > > > > > [Mon Mar 11 14:10:08 2024] "echo 0 >
> > > > > > > > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> > > > > > > > [Mon Mar 11 14:10:08 2024]task:nfsd state:D stack:0
> > > > > > > > pid:8866 ppid:2 flags:0x00004000
> > > > > > > > [Mon Mar 11 14:10:08 2024] Call Trace:
> > > > > > > > [Mon Mar 11 14:10:08 2024] <TASK>
> > > > > > > > [Mon Mar 11 14:10:08 2024] __schedule+0x21b/0x550
> > > > > > > > [Mon Mar 11 14:10:08 2024] schedule+0x2d/0x70
> > > > > > > > [Mon Mar 11 14:10:08 2024] schedule_timeout+0x11f/0x160
> > > > > > > > [Mon Mar 11 14:10:08 2024] ? select_idle_sibling+0x28/0x430
> > > > > > > > [Mon Mar 11 14:10:08 2024] ? tcp_recvmsg+0x196/0x210
> > > > > > > > [Mon Mar 11 14:10:08 2024] ? wake_affine+0x62/0x1f0
> > > > > > > > [Mon Mar 11 14:10:08 2024] __wait_for_common+0x90/0x1d0
> > > > > > > > [Mon Mar 11 14:10:08 2024] ? __pfx_schedule_timeout+0x10/0x10
> > > > > > > > [Mon Mar 11 14:10:08 2024] __flush_workqueue+0x13a/0x3f0
> > > > > > > > [Mon Mar 11 14:10:08 2024] nfsd4_destroy_session+0x1a4/0x240
> > > > > > > > [nfsd]
> > > > > > > > [Mon Mar 11 14:10:08 2024] nfsd4_proc_compound+0x44b/0x700 [nfsd]
> > > > > > > > [Mon Mar 11 14:10:08 2024] nfsd_dispatch+0x94/0x1c0 [nfsd]
> > > > > > > > [Mon Mar 11 14:10:08 2024] svc_process_common+0x2ec/0x660
> > > > > > > > [sunrpc]
> > > > > > > > [Mon Mar 11 14:10:08 2024] ? __pfx_nfsd_dispatch+0x10/0x10 [nfsd]
> > > > > > > > [Mon Mar 11 14:10:08 2024] ? __pfx_nfsd+0x10/0x10 [nfsd]
> > > > > > > > [Mon Mar 11 14:10:08 2024] svc_process+0x12d/0x170 [sunrpc]
> > > > > > > > [Mon Mar 11 14:10:08 2024] nfsd+0x84/0xb0 [nfsd]
> > > > > > > > [Mon Mar 11 14:10:08 2024] kthread+0xdd/0x100
> > > > > > > > [Mon Mar 11 14:10:08 2024] ? __pfx_kthread+0x10/0x10
> > > > > > > > [Mon Mar 11 14:10:08 2024] ret_from_fork+0x29/0x50
> > > > > > > > [Mon Mar 11 14:10:08 2024] </TASK>
> > > > > > > >
> > > > > > > The above threads are trying to flush the workqueue, so that probably
> > > > > > > means that they are stuck waiting on a workqueue job to finish.
> > > > > > > > The above is repeated a few times, and then this warning is
> > > > > > > > also logged:
> > > > > > > > [Mon Mar 11 14:12:04 2024] ------------[ cut here ]------------
> > > > > > > > [Mon Mar 11 14:12:04 2024] WARNING: CPU: 39 PID: 8844 at
> > > > > > > > fs/nfsd/nfs4state.c:4919 nfsd_break_deleg_cb+0x170/0x190 [nfsd]
> > > > > > > > [Mon Mar 11 14:12:05 2024] Modules linked in: nfsv4
> > > > > > > > dns_resolver nfs fscache netfs rpcsec_gss_krb5 rpcrdma rdma_cm
> > > > > > > > iw_cm ib_cm ib_core binfmt_misc bonding tls rfkill nft_counter nft_ct
> > > > > > > > nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_reject_inet
> > > > > > > > nf_reject_ipv4 nf_reject_ipv6 nft_reject nf_tables nfnetlink vfat
> > > > > > > > fat dm_thin_pool dm_persistent_data dm_bio_prison dm_bufio l
> > > > > > > > ibcrc32c dm_service_time dm_multipath intel_rapl_msr
> > > > > > > > intel_rapl_common intel_uncore_frequency
> > > > > > > > intel_uncore_frequency_common isst_if_common skx_edac nfit
> > > > > > > > libnvdimm ipmi_ssif x86_pkg_temp
> > > > > > > > _thermal intel_powerclamp coretemp kvm_intel kvm irqbypass
> > > > > > > > dcdbas rapl intel_cstate mgag200 i2c_algo_bit drm_shmem_helper
> > > > > > > > dell_smbios drm_kms_helper dell_wmi_descriptor wmi_bmof intel_u
> > > > > > > > ncore syscopyarea pcspkr sysfillrect mei_me sysimgblt acpi_ipmi
> > > > > > > > mei fb_sys_fops i2c_i801 ipmi_si intel_pch_thermal lpc_ich
> > > > > > > > ipmi_devintf i2c_smbus ipmi_msghandler joydev acpi_power_meter
> > > > > > > > nfsd auth_rpcgss nfs_acl drm lockd grace fuse sunrpc ext4
> > > > > > > > mbcache jbd2 sd_mod sg lpfc
> > > > > > > > [Mon Mar 11 14:12:05 2024] nvmet_fc nvmet nvme_fc nvme_fabrics
> > > > > > > > crct10dif_pclmul ahci libahci crc32_pclmul nvme_core crc32c_intel
> > > > > > > > ixgbe megaraid_sas libata nvme_common ghash_clmulni_int
> > > > > > > > el t10_pi wdat_wdt scsi_transport_fc mdio wmi dca dm_mirror
> > > > > > > > dm_region_hash dm_log dm_mod
> > > > > > > > [Mon Mar 11 14:12:05 2024] CPU: 39 PID: 8844 Comm: nfsd Not
> > > > > > > > tainted 5.14.0-419.el9.x86_64 #1
> > > > > > > > [Mon Mar 11 14:12:05 2024] Hardware name: Dell Inc. PowerEdge
> > > > > > > > R740/00WGD1, BIOS 2.20.1 09/13/2023
> > > > > > > > [Mon Mar 11 14:12:05 2024] RIP:
> > > > > > > > 0010:nfsd_break_deleg_cb+0x170/0x190 [nfsd]
> > > > > > > > [Mon Mar 11 14:12:05 2024] Code: a6 95 c5 f3 e9 ff fe ff ff 48
> > > > > > > > 89 df be 01 00 00 00 e8 34 b5 13 f4 48 8d bb 98 00 00 00 e8 c8 f9
> > > > > > > > 00 00 84 c0 0f 85 2e ff ff ff <0f> 0b e9 27 ff ff ff be
> > > > > > > > 02 00 00 00 48 89 df e8 0c b5 13 f4 e9 01
> > > > > > > > [Mon Mar 11 14:12:05 2024] RSP: 0018:ffff9929e0bb7b80 EFLAGS:
> > > > > > > > 00010246
> > > > > > > > [Mon Mar 11 14:12:05 2024] RAX: 0000000000000000 RBX:
> > > > > > > > ffff8ada51930900 RCX: 0000000000000024
> > > > > > > > [Mon Mar 11 14:12:05 2024] RDX: ffff8ada519309c8 RSI:
> > > > > > > > ffff8ad582933c00 RDI: 0000000000002000
> > > > > > > > [Mon Mar 11 14:12:05 2024] RBP: ffff8ad46bf21574 R08:
> > > > > > > > ffff9929e0bb7b48 R09: 0000000000000000
> > > > > > > > [Mon Mar 11 14:12:05 2024] R10: ffff8aec859a2948 R11:
> > > > > > > > 0000000000000000 R12: ffff8ad6f497c360
> > > > > > > > [Mon Mar 11 14:12:05 2024] R13: ffff8ad46bf21560 R14:
> > > > > > > > ffff8ae5942e0b10 R15: ffff8ad6f497c360
> > > > > > > > [Mon Mar 11 14:12:05 2024] FS: 0000000000000000(0000)
> > > > > > > > GS:ffff8b031fcc0000(0000) knlGS:0000000000000000
> > > > > > > > [Mon Mar 11 14:12:05 2024] CS: 0010 DS: 0000 ES: 0000 CR0:
> > > > > > > > 0000000080050033
> > > > > > > > [Mon Mar 11 14:12:05 2024] CR2: 00007fafe2060744 CR3:
> > > > > > > > 00000018e58de006 CR4: 00000000007706e0
> > > > > > > > [Mon Mar 11 14:12:05 2024] DR0: 0000000000000000 DR1:
> > > > > > > > 0000000000000000 DR2: 0000000000000000
> > > > > > > > [Mon Mar 11 14:12:05 2024] DR3: 0000000000000000 DR6:
> > > > > > > > 00000000fffe0ff0 DR7: 0000000000000400
> > > > > > > > [Mon Mar 11 14:12:05 2024] PKRU: 55555554
> > > > > > > > [Mon Mar 11 14:12:05 2024] Call Trace:
> > > > > > > > [Mon Mar 11 14:12:05 2024] <TASK>
> > > > > > > > [Mon Mar 11 14:12:05 2024] ? show_trace_log_lvl+0x1c4/0x2df
> > > > > > > > [Mon Mar 11 14:12:05 2024] ? show_trace_log_lvl+0x1c4/0x2df
> > > > > > > > [Mon Mar 11 14:12:05 2024] ? __break_lease+0x16f/0x5f0
> > > > > > > > [Mon Mar 11 14:12:05 2024] ? nfsd_break_deleg_cb+0x170/0x190
> > > > > > > > [nfsd]
> > > > > > > > [Mon Mar 11 14:12:05 2024] ? __warn+0x81/0x110
> > > > > > > > [Mon Mar 11 14:12:05 2024] ? nfsd_break_deleg_cb+0x170/0x190
> > > > > > > > [nfsd]
> > > > > > > > [Mon Mar 11 14:12:05 2024] ? report_bug+0x10a/0x140
> > > > > > > > [Mon Mar 11 14:12:05 2024] ? handle_bug+0x3c/0x70
> > > > > > > > [Mon Mar 11 14:12:05 2024] ? exc_invalid_op+0x14/0x70
> > > > > > > > [Mon Mar 11 14:12:05 2024] ? asm_exc_invalid_op+0x16/0x20
> > > > > > > > [Mon Mar 11 14:12:05 2024] ? nfsd_break_deleg_cb+0x170/0x190
> > > > > > > > [nfsd]
> > > > > > > > [Mon Mar 11 14:12:05 2024] __break_lease+0x16f/0x5f0
> > > > > > > > [Mon Mar 11 14:12:05 2024] ?
> > > > > > > > nfsd_file_lookup_locked+0x117/0x160 [nfsd]
> > > > > > > > [Mon Mar 11 14:12:05 2024] ? list_lru_del+0x101/0x150
> > > > > > > > [Mon Mar 11 14:12:05 2024] nfsd_file_do_acquire+0x790/0x830
> > > > > > > > [nfsd]
> > > > > > > > [Mon Mar 11 14:12:05 2024] nfs4_get_vfs_file+0x315/0x3a0 [nfsd]
> > > > > > > > [Mon Mar 11 14:12:05 2024] nfsd4_process_open2+0x430/0xa30 [nfsd]
> > > > > > > > [Mon Mar 11 14:12:05 2024] ? fh_verify+0x297/0x2f0 [nfsd]
> > > > > > > > [Mon Mar 11 14:12:05 2024] nfsd4_open+0x3ce/0x4b0 [nfsd]
> > > > > > > > [Mon Mar 11 14:12:05 2024] nfsd4_proc_compound+0x44b/0x700 [nfsd]
> > > > > > > > [Mon Mar 11 14:12:05 2024] nfsd_dispatch+0x94/0x1c0 [nfsd]
> > > > > > > > [Mon Mar 11 14:12:05 2024] svc_process_common+0x2ec/0x660
> > > > > > > > [sunrpc]
> > > > > > > > [Mon Mar 11 14:12:05 2024] ? __pfx_nfsd_dispatch+0x10/0x10 [nfsd]
> > > > > > > > [Mon Mar 11 14:12:05 2024] ? __pfx_nfsd+0x10/0x10 [nfsd]
> > > > > > > > [Mon Mar 11 14:12:05 2024] svc_process+0x12d/0x170 [sunrpc]
> > > > > > > > [Mon Mar 11 14:12:05 2024] nfsd+0x84/0xb0 [nfsd]
> > > > > > > > [Mon Mar 11 14:12:05 2024] kthread+0xdd/0x100
> > > > > > > > [Mon Mar 11 14:12:05 2024] ? __pfx_kthread+0x10/0x10
> > > > > > > > [Mon Mar 11 14:12:05 2024] ret_from_fork+0x29/0x50
> > > > > > > > [Mon Mar 11 14:12:05 2024] </TASK>
> > > > > > > > [Mon Mar 11 14:12:05 2024] ---[ end trace 7a039e17443dc651 ]---
> > > > > > > This is probably this WARN in nfsd_break_one_deleg:
> > > > > > >
> > > > > > > WARN_ON_ONCE(!nfsd4_run_cb(&dp->dl_recall));
> > > > > > >
> > > > > > > It means that a delegation break callback to the client couldn't be
> > > > > > > queued to the workqueue, and so it didn't run.
> > > > > > >
> > > > > > > > Could this be the same issue as described
> > > > > > > > here:https://lore.kernel.org/linux-nfs/af0ec881-5ebf-4feb-98ae-3ed2a77f86f1@oracle.com/
> > > > > > > > ?
> > > > > > > Yes, most likely the same problem.
> > > > > > If I read that thread correctly, this issue was introduced between
> > > > > > 6.1.63 and 6.6.3? Is it possible the EL9 5.14.0-362.18.1.el9_3
> > > > > > backported these changes, or were we hitting some other bug with that
> > > > > > version? It seems the 6.1.x kernel is not affected? If so, that
> > > > > > would be
> > > > > > the recommended kernel to run?
> > > > > Anything is possible. We have to identify the problem first.
> > > > > > > > As described in that thread, I've tried to obtain the requested
> > > > > > > > information.
> > > > > > > >
> > > > > > > > Is it possible this is the issue that was fixed by the patches
> > > > > > > > described
> > > > > > > > here?https://lore.kernel.org/linux-nfs/2024022054-cause-suffering-eae8@gregkh/
> > > > > > > >
> > > > > > > Doubtful. Those are targeted toward a different set of issues.
> > > > > > >
> > > > > > > If you're willing, I do have some patches queued up for CentOS here
> > > > > > > that
> > > > > > > fix some backchannel problems that could be related. I'm mainly
> > > > > > > waiting
> > > > > > > on Chuck to send these to Linus and then we'll likely merge them into
> > > > > > > CentOS soon afterward:
> > > > > > >
> > > > > > > https://gitlab.com/redhat/centos-stream/src/kernel/centos-stream-9/-/merge_requests/3689
> > > > > > >
> > > > > > >
> > > > > > If you can send me a patch file, I can rebuild the C9S kernel with that
> > > > > > patch and run it. It can take a while for the bug to trigger as I
> > > > > > believe it seems to be very workload dependent (we were running very
> > > > > > stable for months and now hit this bug every other week).
> > > > > >
> > > > > >
> > > > > It's probably simpler to just pull down the build artifacts for that MR.
> > > > > You have to drill down through the CI for it, but they are here:
> > > > >
> > > > > https://s3.amazonaws.com/arr-cki-prod-trusted-artifacts/index.html?prefix=trusted-artifacts/1194300175/publish_x86_64/6278921877/artifacts/
> > > > >
> > > > >
> > > > > There's even a repo file you can install on the box to pull them down.
> > > > We installed this kernel on the server 3 days ago. Today, a user
> > > > informed us that their screen was black after logging in. Similar to
> > > > other occurrences of this issue, the mount command on the client was
> > > > hung. But in contrast to the other times, there were no messages in
> > > > the logs kernel logs on the server. Even restarting the client does
> > > > not resolve the issue.
> >
> > Ok, so you rebooted the client and it's still unable to mount? That
> > sounds like a server problem if so.
> >
> > Are both client and server running the same kernel?
> No, the server runs 5.14.0-427.3689_1194299994.el9 and the client
> 5.14.0-362.18.1.el9_3.
Ok.
> >
> > > > Something still seems to be wrong on the server though. When I look at
> > > > the directories under /proc/fs/nfsd/clients, there's still a directory
> > > > for the specific client, even though it's no longer running:
> > > >
> > > > # cat 155/info
> > > > clientid: 0xc8edb7f65f4a9ad
> > > > address: "10.87.31.152:819"
> > > > status: confirmed
> > > > seconds from last renew: 33163
> > > > name: "Linux NFSv4.2 bersalis.esat.kuleuven.be"
> > > > minor version: 2
> > > > Implementation domain: "kernel.org"
> > > > Implementation name: "Linux 5.14.0-362.18.1.el9_3.0.1.x86_64 #1 SMP
> > > > PREEMPT_DYNAMIC Sun Feb 11 13:49:23 UTC 2024 x86_64"
> > > > Implementation time: [0, 0]
> > > > callback state: DOWN
> > > > callback address: 10.87.31.152:0
> > > >
> > If you just shut down the client, the server won't immediately purge its
> > record. In fact, assuming you're running the same kernel on the server,
> > it won't purge the client record until there is a conflicting request
> > for its state.
> Is there a way to force such a conflicting request (to get the client
> record to purge)?
From the server or a different client, you can try opening the inodes
that the stuck client is holding open. If you open them for write, that
may trigger the server to kick out the old client record.
The problem is that they are disconnected dentries, so finding them to
open via path may be difficult...
> >
> > > The nfsdclnts command for this client shows the following delegations:
> > >
> > > # nfsdclnts -f 155/states -t all
> > > Inode number | Type | Access | Deny | ip address | Filename
> > > 169346743 | open | r- | -- | 10.87.31.152:819 |
> > > disconnected dentry
> > > 169346743 | deleg | r | | 10.87.31.152:819 |
> > > disconnected dentry
> > > 169346746 | open | r- | -- | 10.87.31.152:819 |
> > > disconnected dentry
> > > 169346746 | deleg | r | | 10.87.31.152:819 |
> > > disconnected dentry
> > >
> > > I see a lot of recent patches regarding directory delegations. Could
> > > this be related to this?
> > >
> > > Will a 5.14.0-362.18.1.el9_3.0.1 kernel try to use a directory delegation?
> > >
> > >
> > No. Directory delegations are a new feature that's still under
> > development. They use some of the same machinery as file delegations,
> > but they wouldn't be a factor here.
> >
> > > > The system seems to have identified that the client is no longer
> > > > reachable, but the client entry does not go away. When a mount was
> > > > hanging on the client, there would be two directories in clients for
> > > > the same client. Killing the mount command clears up the second entry.
> > > >
> > > > Even after running conntrack -D on the server to remove the tcp
> > > > connection from the conntrack table, the entry doesn't go away and the
> > > > client still can not mount anything from the server.
> > > >
> > > > A tcpdump on the client while a mount was running logged the following
> > > > messages over and over again:
> > > >
> > > > request:
> > > >
> > > > Frame 1: 378 bytes on wire (3024 bits), 378 bytes captured (3024 bits)
> > > > Ethernet II, Src: HP_19:7d:4b (e0:73:e7:19:7d:4b), Dst:
> > > > ArubaaHe_f9:8e:00 (88:3a:30:f9:8e:00)
> > > > Internet Protocol Version 4, Src: 10.87.31.152, Dst: 10.86.18.14
> > > > Transmission Control Protocol, Src Port: 932, Dst Port: 2049, Seq: 1,
> > > > Ack: 1, Len: 312
> > > > Remote Procedure Call, Type:Call XID:0x1d3220c4
> > > > Network File System
> > > > [Program Version: 4]
> > > > [V4 Procedure: COMPOUND (1)]
> > > > GSS Data, Ops(1): CREATE_SESSION
> > > > Length: 152
> > > > GSS Sequence Number: 76
> > > > Tag: <EMPTY>
> > > > minorversion: 2
> > > > Operations (count: 1): CREATE_SESSION
> > > > [Main Opcode: CREATE_SESSION (43)]
> > > > GSS Checksum:
> > > > 00000028040404ffffffffff000000002c19055f1f8d442d594c13849628affc2797cbb2…
> > > > GSS Token Length: 40
> > > > GSS-API Generic Security Service Application Program Interface
> > > > krb5_blob:
> > > > 040404ffffffffff000000002c19055f1f8d442d594c13849628affc2797cbb23fa080b0…
> > > >
> > > > response:
> > > >
> > > > Frame 2: 206 bytes on wire (1648 bits), 206 bytes captured (1648 bits)
> > > > Ethernet II, Src: ArubaaHe_f9:8e:00 (88:3a:30:f9:8e:00), Dst:
> > > > HP_19:7d:4b (e0:73:e7:19:7d:4b)
> > > > Internet Protocol Version 4, Src: 10.86.18.14, Dst: 10.87.31.152
> > > > Transmission Control Protocol, Src Port: 2049, Dst Port: 932, Seq: 1,
> > > > Ack: 313, Len: 140
> > > > Remote Procedure Call, Type:Reply XID:0x1d3220c4
> > > > Network File System
> > > > [Program Version: 4]
> > > > [V4 Procedure: COMPOUND (1)]
> > > > GSS Data, Ops(1): CREATE_SESSION(NFS4ERR_DELAY)
> > > > Length: 24
> > > > GSS Sequence Number: 76
> > > > Status: NFS4ERR_DELAY (10008)
> > > > Tag: <EMPTY>
> > > > Operations (count: 1)
> > > > [Main Opcode: CREATE_SESSION (43)]
> > > > GSS Checksum:
> > > > 00000028040405ffffffffff000000000aa742d0798deaad1a8aa2d7c3a91bf4f6274222…
> > > > GSS Token Length: 40
> > > > GSS-API Generic Security Service Application Program Interface
> > > > krb5_blob:
> > > > 040405ffffffffff000000000aa742d0798deaad1a8aa2d7c3a91bf4f627422226d74923…
> > > >
> > > > I was hoping that giving the client a different IP address would
> > > > resolve the issue for this client, but it didn't. Even though the
> > > > client had a new IP address (hostname was kept the same), it failed to
> > > > mount anything from the server.
> > > >
> > Changing the IP address won't help. The client is probably using the
> > same long-form client id as before, so the server still identifies the
> > client even with the address change.
> How is the client id determined? Will changing the hostname of the
> client trigger a change of the client id?
In the client record you showed a bit above, there is a "name" field:
name: "Linux NFSv4.2 bersalis.esat.kuleuven.be"
That's the string the server uses to uniquely identify the client. So
yes, changing the hostname should change that string.
> >
> > Unfortunately, the cause of an NFS4ERR_DELAY error is tough to guess.
> > The client is expected to back off and retry, so if the server keeps
> > returning that repeatedly, then a hung mount command is expected.
> >
> > The question is why the server would keep returning DELAY. A lot of
> > different problems ranging from memory allocation issues to protocol
> > problems can result in that error. You may want to check the NFS server
> > and see if anything was logged there.
> There are no messages in the system logs that indicate any sort of
> memory issue. We also increased the min_kbytes_free sysctl to 2G on the
> server before we restarted it with the newer kernel.
Ok, I didn't expect to see anything like that, but it was a possibility.
> >
> > This is on a CREATE_SESSION call, so I wonder if the record held by the
> > (courteous) server is somehow blocking the attempt to reestablish the
> > session?
> >
> > Do you have a way to reproduce this? Since this is a centos kernel, you
> > could follow the page here to open a bug:
>
> Unfortunately we haven't found a reliable way to reproduce it. But we do
> seem to trigger it more and more lately.
>
>
Bummer, ok. Let us know if you figure out a way to reproduce it.
> > https://wiki.centos.org/ReportBugs.html
> >
> >
> > > > I created another dump of the workqueues and worker pools on the server:
> > > >
> > > > [Mon Mar 18 14:59:33 2024] Showing busy workqueues and worker pools:
> > > > [Mon Mar 18 14:59:33 2024] workqueue events: flags=0x0
> > > > [Mon Mar 18 14:59:33 2024] pwq 54: cpus=27 node=1 flags=0x0 nice=0
> > > > active=1/256 refcnt=2
> > > > [Mon Mar 18 14:59:33 2024] pending: drm_fb_helper_damage_work
> > > > [drm_kms_helper]
> > > > [Mon Mar 18 14:59:33 2024] workqueue events_power_efficient: flags=0x80
> > > > [Mon Mar 18 14:59:33 2024] pwq 54: cpus=27 node=1 flags=0x0 nice=0
> > > > active=1/256 refcnt=2
> > > > [Mon Mar 18 14:59:33 2024] pending: fb_flashcursor
> > > > [Mon Mar 18 14:59:33 2024] workqueue mm_percpu_wq: flags=0x8
> > > > [Mon Mar 18 14:59:33 2024] pwq 54: cpus=27 node=1 flags=0x0 nice=0
> > > > active=1/256 refcnt=3
> > > > [Mon Mar 18 14:59:33 2024] pending: lru_add_drain_per_cpu BAR(362)
> > > > [Mon Mar 18 14:59:33 2024] workqueue kblockd: flags=0x18
> > > > [Mon Mar 18 14:59:33 2024] pwq 55: cpus=27 node=1 flags=0x0 nice=-20
> > > > active=1/256 refcnt=2
> > > > [Mon Mar 18 14:59:33 2024] pending: blk_mq_timeout_work
> > > >
> > > >
> > > > In contrast to last time, it doesn't show anything regarding nfs this
> > > > time.
> > > >
> > > > I also tried the suggestion from Dai Ngo (echo 3 >
> > > > /proc/sys/vm/drop_caches), but that didn't seem to make any difference.
> > > >
> > > > We haven't restarted the server yet as it seems the impact seems to
> > > > affect fewer clients that before. Is there anything we can run on the
> > > > server to further debug this?
> > > >
> > > > In the past, the issue seemed to deteriorate rapidly and resulted in
> > > > issues for almost all clients after about 20 minutes. This time the
> > > > impact seems to be less, but it's not gone.
> > > >
> > > > How can we force the NFS server to forget about a specific client? I
> > > > haven't tried to restart the nfs service yet as I'm afraid it will
> > > > fail to stop as before.
> > > >
> > Not with that kernel. There are some new administrative interfaces that
> > might allow that in the future, but they were just merged upstream and
> > aren't in that kernel.
> >
> > --
> > Jeff Layton <jlayton@kernel.org>
>
--
Jeff Layton <jlayton@kernel.org>
^ permalink raw reply [flat|nested] 26+ messages in thread
* Re: nfsd hangs and nfsd_break_deleg_cb+0x170/0x190 warning
2024-03-19 10:39 ` Jeff Layton
@ 2024-03-19 10:58 ` Rik Theys
2024-03-19 11:36 ` Jeff Layton
0 siblings, 1 reply; 26+ messages in thread
From: Rik Theys @ 2024-03-19 10:58 UTC (permalink / raw)
To: Jeff Layton, Linux Nfs
Hi,
On 3/19/24 11:39, Jeff Layton wrote:
> On Tue, 2024-03-19 at 08:58 +0100, Rik Theys wrote:
>> Hi,
>>
>> On 3/18/24 22:54, Jeff Layton wrote:
>>> On Mon, 2024-03-18 at 22:15 +0100, Rik Theys wrote:
>>>> Hi,
>>>>
>>>> On 3/18/24 21:21, Rik Theys wrote:
>>>>> Hi Jeff,
>>>>>
>>>>> On 3/12/24 13:47, Jeff Layton wrote:
>>>>>> On Tue, 2024-03-12 at 13:24 +0100, Rik Theys wrote:
>>>>>>> Hi Jeff,
>>>>>>>
>>>>>>> On 3/12/24 12:22, Jeff Layton wrote:
>>>>>>>> On Mon, 2024-03-11 at 19:43 +0100, Rik Theys wrote:
>>>>>>>>> Since a few weeks our Rocky Linux 9 NFS server has periodically
>>>>>>>>> logged hung nfsd tasks. The initial effect was that some clients
>>>>>>>>> could no longer access the NFS server. This got worse and worse
>>>>>>>>> (probably as more nfsd threads got blocked) and we had to restart
>>>>>>>>> the server. Restarting the server also failed as the NFS server
>>>>>>>>> service could no longer be stopped.
>>>>>>>>>
>>>>>>>>> The initial kernel we noticed this behavior on was
>>>>>>>>> kernel-5.14.0-362.18.1.el9_3.x86_64. Since then we've installed
>>>>>>>>> kernel-5.14.0-419.el9.x86_64 from CentOS Stream 9. The same issue
>>>>>>>>> happened again on this newer kernel version:
>>>>>> 419 is fairly up to date with nfsd changes. There are some known bugs
>>>>>> around callbacks, and there is a draft MR in flight to fix it.
>>>>>>
>>>>>> What kernel were you on prior to 5.14.0-362.18.1.el9_3.x86_64 ? If we
>>>>>> can bracket the changes around a particular version, then that might
>>>>>> help identify the problem.
>>>>>>
>>>>>>>>> [Mon Mar 11 14:10:08 2024] Not tainted 5.14.0-419.el9.x86_64 #1
>>>>>>>>> [Mon Mar 11 14:10:08 2024] "echo 0 >
>>>>>>>>> /proc/sys/kernel/hung_task_timeout_secs" disables this message.
>>>>>>>>> [Mon Mar 11 14:10:08 2024]task:nfsd state:D stack:0
>>>>>>>>> pid:8865 ppid:2 flags:0x00004000
>>>>>>>>> [Mon Mar 11 14:10:08 2024] Call Trace:
>>>>>>>>> [Mon Mar 11 14:10:08 2024] <TASK>
>>>>>>>>> [Mon Mar 11 14:10:08 2024] __schedule+0x21b/0x550
>>>>>>>>> [Mon Mar 11 14:10:08 2024] schedule+0x2d/0x70
>>>>>>>>> [Mon Mar 11 14:10:08 2024] schedule_timeout+0x11f/0x160
>>>>>>>>> [Mon Mar 11 14:10:08 2024] ? select_idle_sibling+0x28/0x430
>>>>>>>>> [Mon Mar 11 14:10:08 2024] ? wake_affine+0x62/0x1f0
>>>>>>>>> [Mon Mar 11 14:10:08 2024] __wait_for_common+0x90/0x1d0
>>>>>>>>> [Mon Mar 11 14:10:08 2024] ? __pfx_schedule_timeout+0x10/0x10
>>>>>>>>> [Mon Mar 11 14:10:08 2024] __flush_workqueue+0x13a/0x3f0
>>>>>>>>> [Mon Mar 11 14:10:08 2024] nfsd4_shutdown_callback+0x49/0x120
>>>>>>>>> [nfsd]
>>>>>>>>> [Mon Mar 11 14:10:08 2024] ? nfsd4_cld_remove+0x54/0x1d0 [nfsd]
>>>>>>>>> [Mon Mar 11 14:10:08 2024] ?
>>>>>>>>> nfsd4_return_all_client_layouts+0xc4/0xf0 [nfsd]
>>>>>>>>> [Mon Mar 11 14:10:08 2024] ? nfsd4_shutdown_copy+0x68/0xc0 [nfsd]
>>>>>>>>> [Mon Mar 11 14:10:08 2024] __destroy_client+0x1f3/0x290 [nfsd]
>>>>>>>>> [Mon Mar 11 14:10:08 2024] nfsd4_exchange_id+0x75f/0x770 [nfsd]
>>>>>>>>> [Mon Mar 11 14:10:08 2024] ? nfsd4_decode_opaque+0x3a/0x90 [nfsd]
>>>>>>>>> [Mon Mar 11 14:10:08 2024] nfsd4_proc_compound+0x44b/0x700 [nfsd]
>>>>>>>>> [Mon Mar 11 14:10:08 2024] nfsd_dispatch+0x94/0x1c0 [nfsd]
>>>>>>>>> [Mon Mar 11 14:10:08 2024] svc_process_common+0x2ec/0x660
>>>>>>>>> [sunrpc]
>>>>>>>>> [Mon Mar 11 14:10:08 2024] ? __pfx_nfsd_dispatch+0x10/0x10 [nfsd]
>>>>>>>>> [Mon Mar 11 14:10:08 2024] ? __pfx_nfsd+0x10/0x10 [nfsd]
>>>>>>>>> [Mon Mar 11 14:10:08 2024] svc_process+0x12d/0x170 [sunrpc]
>>>>>>>>> [Mon Mar 11 14:10:08 2024] nfsd+0x84/0xb0 [nfsd]
>>>>>>>>> [Mon Mar 11 14:10:08 2024] kthread+0xdd/0x100
>>>>>>>>> [Mon Mar 11 14:10:08 2024] ? __pfx_kthread+0x10/0x10
>>>>>>>>> [Mon Mar 11 14:10:08 2024] ret_from_fork+0x29/0x50
>>>>>>>>> [Mon Mar 11 14:10:08 2024] </TASK>
>>>>>>>>> [Mon Mar 11 14:10:08 2024] INFO: task nfsd:8866 blocked for
>>>>>>>>> more than 122 seconds.
>>>>>>>>> [Mon Mar 11 14:10:08 2024] Not tainted
>>>>>>>>> 5.14.0-419.el9.x86_64 #1
>>>>>>>>> [Mon Mar 11 14:10:08 2024] "echo 0 >
>>>>>>>>> /proc/sys/kernel/hung_task_timeout_secs" disables this message.
>>>>>>>>> [Mon Mar 11 14:10:08 2024]task:nfsd state:D stack:0
>>>>>>>>> pid:8866 ppid:2 flags:0x00004000
>>>>>>>>> [Mon Mar 11 14:10:08 2024] Call Trace:
>>>>>>>>> [Mon Mar 11 14:10:08 2024] <TASK>
>>>>>>>>> [Mon Mar 11 14:10:08 2024] __schedule+0x21b/0x550
>>>>>>>>> [Mon Mar 11 14:10:08 2024] schedule+0x2d/0x70
>>>>>>>>> [Mon Mar 11 14:10:08 2024] schedule_timeout+0x11f/0x160
>>>>>>>>> [Mon Mar 11 14:10:08 2024] ? select_idle_sibling+0x28/0x430
>>>>>>>>> [Mon Mar 11 14:10:08 2024] ? tcp_recvmsg+0x196/0x210
>>>>>>>>> [Mon Mar 11 14:10:08 2024] ? wake_affine+0x62/0x1f0
>>>>>>>>> [Mon Mar 11 14:10:08 2024] __wait_for_common+0x90/0x1d0
>>>>>>>>> [Mon Mar 11 14:10:08 2024] ? __pfx_schedule_timeout+0x10/0x10
>>>>>>>>> [Mon Mar 11 14:10:08 2024] __flush_workqueue+0x13a/0x3f0
>>>>>>>>> [Mon Mar 11 14:10:08 2024] nfsd4_destroy_session+0x1a4/0x240
>>>>>>>>> [nfsd]
>>>>>>>>> [Mon Mar 11 14:10:08 2024] nfsd4_proc_compound+0x44b/0x700 [nfsd]
>>>>>>>>> [Mon Mar 11 14:10:08 2024] nfsd_dispatch+0x94/0x1c0 [nfsd]
>>>>>>>>> [Mon Mar 11 14:10:08 2024] svc_process_common+0x2ec/0x660
>>>>>>>>> [sunrpc]
>>>>>>>>> [Mon Mar 11 14:10:08 2024] ? __pfx_nfsd_dispatch+0x10/0x10 [nfsd]
>>>>>>>>> [Mon Mar 11 14:10:08 2024] ? __pfx_nfsd+0x10/0x10 [nfsd]
>>>>>>>>> [Mon Mar 11 14:10:08 2024] svc_process+0x12d/0x170 [sunrpc]
>>>>>>>>> [Mon Mar 11 14:10:08 2024] nfsd+0x84/0xb0 [nfsd]
>>>>>>>>> [Mon Mar 11 14:10:08 2024] kthread+0xdd/0x100
>>>>>>>>> [Mon Mar 11 14:10:08 2024] ? __pfx_kthread+0x10/0x10
>>>>>>>>> [Mon Mar 11 14:10:08 2024] ret_from_fork+0x29/0x50
>>>>>>>>> [Mon Mar 11 14:10:08 2024] </TASK>
>>>>>>>>>
>>>>>>>> The above threads are trying to flush the workqueue, so that probably
>>>>>>>> means that they are stuck waiting on a workqueue job to finish.
>>>>>>>>> The above is repeated a few times, and then this warning is
>>>>>>>>> also logged:
>>>>>>>>> [Mon Mar 11 14:12:04 2024] ------------[ cut here ]------------
>>>>>>>>> [Mon Mar 11 14:12:04 2024] WARNING: CPU: 39 PID: 8844 at
>>>>>>>>> fs/nfsd/nfs4state.c:4919 nfsd_break_deleg_cb+0x170/0x190 [nfsd]
>>>>>>>>> [Mon Mar 11 14:12:05 2024] Modules linked in: nfsv4
>>>>>>>>> dns_resolver nfs fscache netfs rpcsec_gss_krb5 rpcrdma rdma_cm
>>>>>>>>> iw_cm ib_cm ib_core binfmt_misc bonding tls rfkill nft_counter nft_ct
>>>>>>>>> nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_reject_inet
>>>>>>>>> nf_reject_ipv4 nf_reject_ipv6 nft_reject nf_tables nfnetlink vfat
>>>>>>>>> fat dm_thin_pool dm_persistent_data dm_bio_prison dm_bufio l
>>>>>>>>> ibcrc32c dm_service_time dm_multipath intel_rapl_msr
>>>>>>>>> intel_rapl_common intel_uncore_frequency
>>>>>>>>> intel_uncore_frequency_common isst_if_common skx_edac nfit
>>>>>>>>> libnvdimm ipmi_ssif x86_pkg_temp
>>>>>>>>> _thermal intel_powerclamp coretemp kvm_intel kvm irqbypass
>>>>>>>>> dcdbas rapl intel_cstate mgag200 i2c_algo_bit drm_shmem_helper
>>>>>>>>> dell_smbios drm_kms_helper dell_wmi_descriptor wmi_bmof intel_u
>>>>>>>>> ncore syscopyarea pcspkr sysfillrect mei_me sysimgblt acpi_ipmi
>>>>>>>>> mei fb_sys_fops i2c_i801 ipmi_si intel_pch_thermal lpc_ich
>>>>>>>>> ipmi_devintf i2c_smbus ipmi_msghandler joydev acpi_power_meter
>>>>>>>>> nfsd auth_rpcgss nfs_acl drm lockd grace fuse sunrpc ext4
>>>>>>>>> mbcache jbd2 sd_mod sg lpfc
>>>>>>>>> [Mon Mar 11 14:12:05 2024] nvmet_fc nvmet nvme_fc nvme_fabrics
>>>>>>>>> crct10dif_pclmul ahci libahci crc32_pclmul nvme_core crc32c_intel
>>>>>>>>> ixgbe megaraid_sas libata nvme_common ghash_clmulni_int
>>>>>>>>> el t10_pi wdat_wdt scsi_transport_fc mdio wmi dca dm_mirror
>>>>>>>>> dm_region_hash dm_log dm_mod
>>>>>>>>> [Mon Mar 11 14:12:05 2024] CPU: 39 PID: 8844 Comm: nfsd Not
>>>>>>>>> tainted 5.14.0-419.el9.x86_64 #1
>>>>>>>>> [Mon Mar 11 14:12:05 2024] Hardware name: Dell Inc. PowerEdge
>>>>>>>>> R740/00WGD1, BIOS 2.20.1 09/13/2023
>>>>>>>>> [Mon Mar 11 14:12:05 2024] RIP:
>>>>>>>>> 0010:nfsd_break_deleg_cb+0x170/0x190 [nfsd]
>>>>>>>>> [Mon Mar 11 14:12:05 2024] Code: a6 95 c5 f3 e9 ff fe ff ff 48
>>>>>>>>> 89 df be 01 00 00 00 e8 34 b5 13 f4 48 8d bb 98 00 00 00 e8 c8 f9
>>>>>>>>> 00 00 84 c0 0f 85 2e ff ff ff <0f> 0b e9 27 ff ff ff be
>>>>>>>>> 02 00 00 00 48 89 df e8 0c b5 13 f4 e9 01
>>>>>>>>> [Mon Mar 11 14:12:05 2024] RSP: 0018:ffff9929e0bb7b80 EFLAGS:
>>>>>>>>> 00010246
>>>>>>>>> [Mon Mar 11 14:12:05 2024] RAX: 0000000000000000 RBX:
>>>>>>>>> ffff8ada51930900 RCX: 0000000000000024
>>>>>>>>> [Mon Mar 11 14:12:05 2024] RDX: ffff8ada519309c8 RSI:
>>>>>>>>> ffff8ad582933c00 RDI: 0000000000002000
>>>>>>>>> [Mon Mar 11 14:12:05 2024] RBP: ffff8ad46bf21574 R08:
>>>>>>>>> ffff9929e0bb7b48 R09: 0000000000000000
>>>>>>>>> [Mon Mar 11 14:12:05 2024] R10: ffff8aec859a2948 R11:
>>>>>>>>> 0000000000000000 R12: ffff8ad6f497c360
>>>>>>>>> [Mon Mar 11 14:12:05 2024] R13: ffff8ad46bf21560 R14:
>>>>>>>>> ffff8ae5942e0b10 R15: ffff8ad6f497c360
>>>>>>>>> [Mon Mar 11 14:12:05 2024] FS: 0000000000000000(0000)
>>>>>>>>> GS:ffff8b031fcc0000(0000) knlGS:0000000000000000
>>>>>>>>> [Mon Mar 11 14:12:05 2024] CS: 0010 DS: 0000 ES: 0000 CR0:
>>>>>>>>> 0000000080050033
>>>>>>>>> [Mon Mar 11 14:12:05 2024] CR2: 00007fafe2060744 CR3:
>>>>>>>>> 00000018e58de006 CR4: 00000000007706e0
>>>>>>>>> [Mon Mar 11 14:12:05 2024] DR0: 0000000000000000 DR1:
>>>>>>>>> 0000000000000000 DR2: 0000000000000000
>>>>>>>>> [Mon Mar 11 14:12:05 2024] DR3: 0000000000000000 DR6:
>>>>>>>>> 00000000fffe0ff0 DR7: 0000000000000400
>>>>>>>>> [Mon Mar 11 14:12:05 2024] PKRU: 55555554
>>>>>>>>> [Mon Mar 11 14:12:05 2024] Call Trace:
>>>>>>>>> [Mon Mar 11 14:12:05 2024] <TASK>
>>>>>>>>> [Mon Mar 11 14:12:05 2024] ? show_trace_log_lvl+0x1c4/0x2df
>>>>>>>>> [Mon Mar 11 14:12:05 2024] ? show_trace_log_lvl+0x1c4/0x2df
>>>>>>>>> [Mon Mar 11 14:12:05 2024] ? __break_lease+0x16f/0x5f0
>>>>>>>>> [Mon Mar 11 14:12:05 2024] ? nfsd_break_deleg_cb+0x170/0x190
>>>>>>>>> [nfsd]
>>>>>>>>> [Mon Mar 11 14:12:05 2024] ? __warn+0x81/0x110
>>>>>>>>> [Mon Mar 11 14:12:05 2024] ? nfsd_break_deleg_cb+0x170/0x190
>>>>>>>>> [nfsd]
>>>>>>>>> [Mon Mar 11 14:12:05 2024] ? report_bug+0x10a/0x140
>>>>>>>>> [Mon Mar 11 14:12:05 2024] ? handle_bug+0x3c/0x70
>>>>>>>>> [Mon Mar 11 14:12:05 2024] ? exc_invalid_op+0x14/0x70
>>>>>>>>> [Mon Mar 11 14:12:05 2024] ? asm_exc_invalid_op+0x16/0x20
>>>>>>>>> [Mon Mar 11 14:12:05 2024] ? nfsd_break_deleg_cb+0x170/0x190
>>>>>>>>> [nfsd]
>>>>>>>>> [Mon Mar 11 14:12:05 2024] __break_lease+0x16f/0x5f0
>>>>>>>>> [Mon Mar 11 14:12:05 2024] ?
>>>>>>>>> nfsd_file_lookup_locked+0x117/0x160 [nfsd]
>>>>>>>>> [Mon Mar 11 14:12:05 2024] ? list_lru_del+0x101/0x150
>>>>>>>>> [Mon Mar 11 14:12:05 2024] nfsd_file_do_acquire+0x790/0x830
>>>>>>>>> [nfsd]
>>>>>>>>> [Mon Mar 11 14:12:05 2024] nfs4_get_vfs_file+0x315/0x3a0 [nfsd]
>>>>>>>>> [Mon Mar 11 14:12:05 2024] nfsd4_process_open2+0x430/0xa30 [nfsd]
>>>>>>>>> [Mon Mar 11 14:12:05 2024] ? fh_verify+0x297/0x2f0 [nfsd]
>>>>>>>>> [Mon Mar 11 14:12:05 2024] nfsd4_open+0x3ce/0x4b0 [nfsd]
>>>>>>>>> [Mon Mar 11 14:12:05 2024] nfsd4_proc_compound+0x44b/0x700 [nfsd]
>>>>>>>>> [Mon Mar 11 14:12:05 2024] nfsd_dispatch+0x94/0x1c0 [nfsd]
>>>>>>>>> [Mon Mar 11 14:12:05 2024] svc_process_common+0x2ec/0x660
>>>>>>>>> [sunrpc]
>>>>>>>>> [Mon Mar 11 14:12:05 2024] ? __pfx_nfsd_dispatch+0x10/0x10 [nfsd]
>>>>>>>>> [Mon Mar 11 14:12:05 2024] ? __pfx_nfsd+0x10/0x10 [nfsd]
>>>>>>>>> [Mon Mar 11 14:12:05 2024] svc_process+0x12d/0x170 [sunrpc]
>>>>>>>>> [Mon Mar 11 14:12:05 2024] nfsd+0x84/0xb0 [nfsd]
>>>>>>>>> [Mon Mar 11 14:12:05 2024] kthread+0xdd/0x100
>>>>>>>>> [Mon Mar 11 14:12:05 2024] ? __pfx_kthread+0x10/0x10
>>>>>>>>> [Mon Mar 11 14:12:05 2024] ret_from_fork+0x29/0x50
>>>>>>>>> [Mon Mar 11 14:12:05 2024] </TASK>
>>>>>>>>> [Mon Mar 11 14:12:05 2024] ---[ end trace 7a039e17443dc651 ]---
>>>>>>>> This is probably this WARN in nfsd_break_one_deleg:
>>>>>>>>
>>>>>>>> WARN_ON_ONCE(!nfsd4_run_cb(&dp->dl_recall));
>>>>>>>>
>>>>>>>> It means that a delegation break callback to the client couldn't be
>>>>>>>> queued to the workqueue, and so it didn't run.
>>>>>>>>
>>>>>>>>> Could this be the same issue as described
>>>>>>>>> here:https://lore.kernel.org/linux-nfs/af0ec881-5ebf-4feb-98ae-3ed2a77f86f1@oracle.com/
>>>>>>>>> ?
>>>>>>>> Yes, most likely the same problem.
>>>>>>> If I read that thread correctly, this issue was introduced between
>>>>>>> 6.1.63 and 6.6.3? Is it possible the EL9 5.14.0-362.18.1.el9_3
>>>>>>> backported these changes, or were we hitting some other bug with that
>>>>>>> version? It seems the 6.1.x kernel is not affected? If so, that
>>>>>>> would be
>>>>>>> the recommended kernel to run?
>>>>>> Anything is possible. We have to identify the problem first.
>>>>>>>>> As described in that thread, I've tried to obtain the requested
>>>>>>>>> information.
>>>>>>>>>
>>>>>>>>> Is it possible this is the issue that was fixed by the patches
>>>>>>>>> described
>>>>>>>>> here?https://lore.kernel.org/linux-nfs/2024022054-cause-suffering-eae8@gregkh/
>>>>>>>>>
>>>>>>>> Doubtful. Those are targeted toward a different set of issues.
>>>>>>>>
>>>>>>>> If you're willing, I do have some patches queued up for CentOS here
>>>>>>>> that
>>>>>>>> fix some backchannel problems that could be related. I'm mainly
>>>>>>>> waiting
>>>>>>>> on Chuck to send these to Linus and then we'll likely merge them into
>>>>>>>> CentOS soon afterward:
>>>>>>>>
>>>>>>>> https://gitlab.com/redhat/centos-stream/src/kernel/centos-stream-9/-/merge_requests/3689
>>>>>>>>
>>>>>>>>
>>>>>>> If you can send me a patch file, I can rebuild the C9S kernel with that
>>>>>>> patch and run it. It can take a while for the bug to trigger as I
>>>>>>> believe it seems to be very workload dependent (we were running very
>>>>>>> stable for months and now hit this bug every other week).
>>>>>>>
>>>>>>>
>>>>>> It's probably simpler to just pull down the build artifacts for that MR.
>>>>>> You have to drill down through the CI for it, but they are here:
>>>>>>
>>>>>> https://s3.amazonaws.com/arr-cki-prod-trusted-artifacts/index.html?prefix=trusted-artifacts/1194300175/publish_x86_64/6278921877/artifacts/
>>>>>>
>>>>>>
>>>>>> There's even a repo file you can install on the box to pull them down.
>>>>> We installed this kernel on the server 3 days ago. Today, a user
>>>>> informed us that their screen was black after logging in. Similar to
>>>>> other occurrences of this issue, the mount command on the client was
>>>>> hung. But in contrast to the other times, there were no messages in
>>>>> the logs kernel logs on the server. Even restarting the client does
>>>>> not resolve the issue.
>>> Ok, so you rebooted the client and it's still unable to mount? That
>>> sounds like a server problem if so.
>>>
>>> Are both client and server running the same kernel?
>> No, the server runs 5.14.0-427.3689_1194299994.el9 and the client
>> 5.14.0-362.18.1.el9_3.
> Ok.
>
>>>>> Something still seems to be wrong on the server though. When I look at
>>>>> the directories under /proc/fs/nfsd/clients, there's still a directory
>>>>> for the specific client, even though it's no longer running:
>>>>>
>>>>> # cat 155/info
>>>>> clientid: 0xc8edb7f65f4a9ad
>>>>> address: "10.87.31.152:819"
>>>>> status: confirmed
>>>>> seconds from last renew: 33163
>>>>> name: "Linux NFSv4.2 bersalis.esat.kuleuven.be"
>>>>> minor version: 2
>>>>> Implementation domain: "kernel.org"
>>>>> Implementation name: "Linux 5.14.0-362.18.1.el9_3.0.1.x86_64 #1 SMP
>>>>> PREEMPT_DYNAMIC Sun Feb 11 13:49:23 UTC 2024 x86_64"
>>>>> Implementation time: [0, 0]
>>>>> callback state: DOWN
>>>>> callback address: 10.87.31.152:0
>>>>>
>>> If you just shut down the client, the server won't immediately purge its
>>> record. In fact, assuming you're running the same kernel on the server,
>>> it won't purge the client record until there is a conflicting request
>>> for its state.
>> Is there a way to force such a conflicting request (to get the client
>> record to purge)?
> From the server or a different client, you can try opening the inodes
> that the stuck client is holding open. If you open them for write, that
> may trigger the server to kick out the old client record.
>
> The problem is that they are disconnected dentries, so finding them to
> open via path may be difficult...
I've located the file that matches one of these inodes. When I go to the
location of the file on another NFS client and touch the file, the touch
command
just hangs.
So I assume the server is trying to recall the delegation from the
dis-functional client?
When I run tcpdump on the dis-functional client, I see the
CREATE_SESSION / NFS4ERR_DELAY messages, but nothing to indicate the
server wants to revoke a delegation.
The entry for the dis-functional client is not removed on the server.
I there anything else I can do to provide more information about this
situation?
Regards,
Rik
>>>> The nfsdclnts command for this client shows the following delegations:
>>>>
>>>> # nfsdclnts -f 155/states -t all
>>>> Inode number | Type | Access | Deny | ip address | Filename
>>>> 169346743 | open | r- | -- | 10.87.31.152:819 |
>>>> disconnected dentry
>>>> 169346743 | deleg | r | | 10.87.31.152:819 |
>>>> disconnected dentry
>>>> 169346746 | open | r- | -- | 10.87.31.152:819 |
>>>> disconnected dentry
>>>> 169346746 | deleg | r | | 10.87.31.152:819 |
>>>> disconnected dentry
>>>>
>>>> I see a lot of recent patches regarding directory delegations. Could
>>>> this be related to this?
>>>>
>>>> Will a 5.14.0-362.18.1.el9_3.0.1 kernel try to use a directory delegation?
>>>>
>>>>
>>> No. Directory delegations are a new feature that's still under
>>> development. They use some of the same machinery as file delegations,
>>> but they wouldn't be a factor here.
>>>
>>>>> The system seems to have identified that the client is no longer
>>>>> reachable, but the client entry does not go away. When a mount was
>>>>> hanging on the client, there would be two directories in clients for
>>>>> the same client. Killing the mount command clears up the second entry.
>>>>>
>>>>> Even after running conntrack -D on the server to remove the tcp
>>>>> connection from the conntrack table, the entry doesn't go away and the
>>>>> client still can not mount anything from the server.
>>>>>
>>>>> A tcpdump on the client while a mount was running logged the following
>>>>> messages over and over again:
>>>>>
>>>>> request:
>>>>>
>>>>> Frame 1: 378 bytes on wire (3024 bits), 378 bytes captured (3024 bits)
>>>>> Ethernet II, Src: HP_19:7d:4b (e0:73:e7:19:7d:4b), Dst:
>>>>> ArubaaHe_f9:8e:00 (88:3a:30:f9:8e:00)
>>>>> Internet Protocol Version 4, Src: 10.87.31.152, Dst: 10.86.18.14
>>>>> Transmission Control Protocol, Src Port: 932, Dst Port: 2049, Seq: 1,
>>>>> Ack: 1, Len: 312
>>>>> Remote Procedure Call, Type:Call XID:0x1d3220c4
>>>>> Network File System
>>>>> [Program Version: 4]
>>>>> [V4 Procedure: COMPOUND (1)]
>>>>> GSS Data, Ops(1): CREATE_SESSION
>>>>> Length: 152
>>>>> GSS Sequence Number: 76
>>>>> Tag: <EMPTY>
>>>>> minorversion: 2
>>>>> Operations (count: 1): CREATE_SESSION
>>>>> [Main Opcode: CREATE_SESSION (43)]
>>>>> GSS Checksum:
>>>>> 00000028040404ffffffffff000000002c19055f1f8d442d594c13849628affc2797cbb2…
>>>>> GSS Token Length: 40
>>>>> GSS-API Generic Security Service Application Program Interface
>>>>> krb5_blob:
>>>>> 040404ffffffffff000000002c19055f1f8d442d594c13849628affc2797cbb23fa080b0…
>>>>>
>>>>> response:
>>>>>
>>>>> Frame 2: 206 bytes on wire (1648 bits), 206 bytes captured (1648 bits)
>>>>> Ethernet II, Src: ArubaaHe_f9:8e:00 (88:3a:30:f9:8e:00), Dst:
>>>>> HP_19:7d:4b (e0:73:e7:19:7d:4b)
>>>>> Internet Protocol Version 4, Src: 10.86.18.14, Dst: 10.87.31.152
>>>>> Transmission Control Protocol, Src Port: 2049, Dst Port: 932, Seq: 1,
>>>>> Ack: 313, Len: 140
>>>>> Remote Procedure Call, Type:Reply XID:0x1d3220c4
>>>>> Network File System
>>>>> [Program Version: 4]
>>>>> [V4 Procedure: COMPOUND (1)]
>>>>> GSS Data, Ops(1): CREATE_SESSION(NFS4ERR_DELAY)
>>>>> Length: 24
>>>>> GSS Sequence Number: 76
>>>>> Status: NFS4ERR_DELAY (10008)
>>>>> Tag: <EMPTY>
>>>>> Operations (count: 1)
>>>>> [Main Opcode: CREATE_SESSION (43)]
>>>>> GSS Checksum:
>>>>> 00000028040405ffffffffff000000000aa742d0798deaad1a8aa2d7c3a91bf4f6274222…
>>>>> GSS Token Length: 40
>>>>> GSS-API Generic Security Service Application Program Interface
>>>>> krb5_blob:
>>>>> 040405ffffffffff000000000aa742d0798deaad1a8aa2d7c3a91bf4f627422226d74923…
>>>>>
>>>>> I was hoping that giving the client a different IP address would
>>>>> resolve the issue for this client, but it didn't. Even though the
>>>>> client had a new IP address (hostname was kept the same), it failed to
>>>>> mount anything from the server.
>>>>>
>>> Changing the IP address won't help. The client is probably using the
>>> same long-form client id as before, so the server still identifies the
>>> client even with the address change.
>> How is the client id determined? Will changing the hostname of the
>> client trigger a change of the client id?
> In the client record you showed a bit above, there is a "name" field:
>
> name: "Linux NFSv4.2 bersalis.esat.kuleuven.be"
>
> That's the string the server uses to uniquely identify the client. So
> yes, changing the hostname should change that string.
>
>>> Unfortunately, the cause of an NFS4ERR_DELAY error is tough to guess.
>>> The client is expected to back off and retry, so if the server keeps
>>> returning that repeatedly, then a hung mount command is expected.
>>>
>>> The question is why the server would keep returning DELAY. A lot of
>>> different problems ranging from memory allocation issues to protocol
>>> problems can result in that error. You may want to check the NFS server
>>> and see if anything was logged there.
>> There are no messages in the system logs that indicate any sort of
>> memory issue. We also increased the min_kbytes_free sysctl to 2G on the
>> server before we restarted it with the newer kernel.
> Ok, I didn't expect to see anything like that, but it was a possibility.
>
>>> This is on a CREATE_SESSION call, so I wonder if the record held by the
>>> (courteous) server is somehow blocking the attempt to reestablish the
>>> session?
>>>
>>> Do you have a way to reproduce this? Since this is a centos kernel, you
>>> could follow the page here to open a bug:
>> Unfortunately we haven't found a reliable way to reproduce it. But we do
>> seem to trigger it more and more lately.
>>
>>
> Bummer, ok. Let us know if you figure out a way to reproduce it.
>
>>> https://wiki.centos.org/ReportBugs.html
>>>
>>>
>>>>> I created another dump of the workqueues and worker pools on the server:
>>>>>
>>>>> [Mon Mar 18 14:59:33 2024] Showing busy workqueues and worker pools:
>>>>> [Mon Mar 18 14:59:33 2024] workqueue events: flags=0x0
>>>>> [Mon Mar 18 14:59:33 2024] pwq 54: cpus=27 node=1 flags=0x0 nice=0
>>>>> active=1/256 refcnt=2
>>>>> [Mon Mar 18 14:59:33 2024] pending: drm_fb_helper_damage_work
>>>>> [drm_kms_helper]
>>>>> [Mon Mar 18 14:59:33 2024] workqueue events_power_efficient: flags=0x80
>>>>> [Mon Mar 18 14:59:33 2024] pwq 54: cpus=27 node=1 flags=0x0 nice=0
>>>>> active=1/256 refcnt=2
>>>>> [Mon Mar 18 14:59:33 2024] pending: fb_flashcursor
>>>>> [Mon Mar 18 14:59:33 2024] workqueue mm_percpu_wq: flags=0x8
>>>>> [Mon Mar 18 14:59:33 2024] pwq 54: cpus=27 node=1 flags=0x0 nice=0
>>>>> active=1/256 refcnt=3
>>>>> [Mon Mar 18 14:59:33 2024] pending: lru_add_drain_per_cpu BAR(362)
>>>>> [Mon Mar 18 14:59:33 2024] workqueue kblockd: flags=0x18
>>>>> [Mon Mar 18 14:59:33 2024] pwq 55: cpus=27 node=1 flags=0x0 nice=-20
>>>>> active=1/256 refcnt=2
>>>>> [Mon Mar 18 14:59:33 2024] pending: blk_mq_timeout_work
>>>>>
>>>>>
>>>>> In contrast to last time, it doesn't show anything regarding nfs this
>>>>> time.
>>>>>
>>>>> I also tried the suggestion from Dai Ngo (echo 3 >
>>>>> /proc/sys/vm/drop_caches), but that didn't seem to make any difference.
>>>>>
>>>>> We haven't restarted the server yet as it seems the impact seems to
>>>>> affect fewer clients that before. Is there anything we can run on the
>>>>> server to further debug this?
>>>>>
>>>>> In the past, the issue seemed to deteriorate rapidly and resulted in
>>>>> issues for almost all clients after about 20 minutes. This time the
>>>>> impact seems to be less, but it's not gone.
>>>>>
>>>>> How can we force the NFS server to forget about a specific client? I
>>>>> haven't tried to restart the nfs service yet as I'm afraid it will
>>>>> fail to stop as before.
>>>>>
>>> Not with that kernel. There are some new administrative interfaces that
>>> might allow that in the future, but they were just merged upstream and
>>> aren't in that kernel.
>>>
>>> --
>>> Jeff Layton <jlayton@kernel.org>
--
Rik Theys
System Engineer
KU Leuven - Dept. Elektrotechniek (ESAT)
Kasteelpark Arenberg 10 bus 2440 - B-3001 Leuven-Heverlee
+32(0)16/32.11.07
----------------------------------------------------------------
<<Any errors in spelling, tact or fact are transmission errors>>
^ permalink raw reply [flat|nested] 26+ messages in thread
* Re: nfsd hangs and nfsd_break_deleg_cb+0x170/0x190 warning
2024-03-19 10:58 ` Rik Theys
@ 2024-03-19 11:36 ` Jeff Layton
2024-03-20 13:03 ` Charles Hedrick
0 siblings, 1 reply; 26+ messages in thread
From: Jeff Layton @ 2024-03-19 11:36 UTC (permalink / raw)
To: Rik Theys, Linux Nfs
On Tue, 2024-03-19 at 11:58 +0100, Rik Theys wrote:
> Hi,
>
> On 3/19/24 11:39, Jeff Layton wrote:
> > On Tue, 2024-03-19 at 08:58 +0100, Rik Theys wrote:
> > > Hi,
> > >
> > > On 3/18/24 22:54, Jeff Layton wrote:
> > > > On Mon, 2024-03-18 at 22:15 +0100, Rik Theys wrote:
> > > > > Hi,
> > > > >
> > > > > On 3/18/24 21:21, Rik Theys wrote:
> > > > > > Hi Jeff,
> > > > > >
> > > > > > On 3/12/24 13:47, Jeff Layton wrote:
> > > > > > > On Tue, 2024-03-12 at 13:24 +0100, Rik Theys wrote:
> > > > > > > > Hi Jeff,
> > > > > > > >
> > > > > > > > On 3/12/24 12:22, Jeff Layton wrote:
> > > > > > > > > On Mon, 2024-03-11 at 19:43 +0100, Rik Theys wrote:
> > > > > > > > > > Since a few weeks our Rocky Linux 9 NFS server has periodically
> > > > > > > > > > logged hung nfsd tasks. The initial effect was that some clients
> > > > > > > > > > could no longer access the NFS server. This got worse and worse
> > > > > > > > > > (probably as more nfsd threads got blocked) and we had to restart
> > > > > > > > > > the server. Restarting the server also failed as the NFS server
> > > > > > > > > > service could no longer be stopped.
> > > > > > > > > >
> > > > > > > > > > The initial kernel we noticed this behavior on was
> > > > > > > > > > kernel-5.14.0-362.18.1.el9_3.x86_64. Since then we've installed
> > > > > > > > > > kernel-5.14.0-419.el9.x86_64 from CentOS Stream 9. The same issue
> > > > > > > > > > happened again on this newer kernel version:
> > > > > > > 419 is fairly up to date with nfsd changes. There are some known bugs
> > > > > > > around callbacks, and there is a draft MR in flight to fix it.
> > > > > > >
> > > > > > > What kernel were you on prior to 5.14.0-362.18.1.el9_3.x86_64 ? If we
> > > > > > > can bracket the changes around a particular version, then that might
> > > > > > > help identify the problem.
> > > > > > >
> > > > > > > > > > [Mon Mar 11 14:10:08 2024] Not tainted 5.14.0-419.el9.x86_64 #1
> > > > > > > > > > [Mon Mar 11 14:10:08 2024] "echo 0 >
> > > > > > > > > > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> > > > > > > > > > [Mon Mar 11 14:10:08 2024]task:nfsd state:D stack:0
> > > > > > > > > > pid:8865 ppid:2 flags:0x00004000
> > > > > > > > > > [Mon Mar 11 14:10:08 2024] Call Trace:
> > > > > > > > > > [Mon Mar 11 14:10:08 2024] <TASK>
> > > > > > > > > > [Mon Mar 11 14:10:08 2024] __schedule+0x21b/0x550
> > > > > > > > > > [Mon Mar 11 14:10:08 2024] schedule+0x2d/0x70
> > > > > > > > > > [Mon Mar 11 14:10:08 2024] schedule_timeout+0x11f/0x160
> > > > > > > > > > [Mon Mar 11 14:10:08 2024] ? select_idle_sibling+0x28/0x430
> > > > > > > > > > [Mon Mar 11 14:10:08 2024] ? wake_affine+0x62/0x1f0
> > > > > > > > > > [Mon Mar 11 14:10:08 2024] __wait_for_common+0x90/0x1d0
> > > > > > > > > > [Mon Mar 11 14:10:08 2024] ? __pfx_schedule_timeout+0x10/0x10
> > > > > > > > > > [Mon Mar 11 14:10:08 2024] __flush_workqueue+0x13a/0x3f0
> > > > > > > > > > [Mon Mar 11 14:10:08 2024] nfsd4_shutdown_callback+0x49/0x120
> > > > > > > > > > [nfsd]
> > > > > > > > > > [Mon Mar 11 14:10:08 2024] ? nfsd4_cld_remove+0x54/0x1d0 [nfsd]
> > > > > > > > > > [Mon Mar 11 14:10:08 2024] ?
> > > > > > > > > > nfsd4_return_all_client_layouts+0xc4/0xf0 [nfsd]
> > > > > > > > > > [Mon Mar 11 14:10:08 2024] ? nfsd4_shutdown_copy+0x68/0xc0 [nfsd]
> > > > > > > > > > [Mon Mar 11 14:10:08 2024] __destroy_client+0x1f3/0x290 [nfsd]
> > > > > > > > > > [Mon Mar 11 14:10:08 2024] nfsd4_exchange_id+0x75f/0x770 [nfsd]
> > > > > > > > > > [Mon Mar 11 14:10:08 2024] ? nfsd4_decode_opaque+0x3a/0x90 [nfsd]
> > > > > > > > > > [Mon Mar 11 14:10:08 2024] nfsd4_proc_compound+0x44b/0x700 [nfsd]
> > > > > > > > > > [Mon Mar 11 14:10:08 2024] nfsd_dispatch+0x94/0x1c0 [nfsd]
> > > > > > > > > > [Mon Mar 11 14:10:08 2024] svc_process_common+0x2ec/0x660
> > > > > > > > > > [sunrpc]
> > > > > > > > > > [Mon Mar 11 14:10:08 2024] ? __pfx_nfsd_dispatch+0x10/0x10 [nfsd]
> > > > > > > > > > [Mon Mar 11 14:10:08 2024] ? __pfx_nfsd+0x10/0x10 [nfsd]
> > > > > > > > > > [Mon Mar 11 14:10:08 2024] svc_process+0x12d/0x170 [sunrpc]
> > > > > > > > > > [Mon Mar 11 14:10:08 2024] nfsd+0x84/0xb0 [nfsd]
> > > > > > > > > > [Mon Mar 11 14:10:08 2024] kthread+0xdd/0x100
> > > > > > > > > > [Mon Mar 11 14:10:08 2024] ? __pfx_kthread+0x10/0x10
> > > > > > > > > > [Mon Mar 11 14:10:08 2024] ret_from_fork+0x29/0x50
> > > > > > > > > > [Mon Mar 11 14:10:08 2024] </TASK>
> > > > > > > > > > [Mon Mar 11 14:10:08 2024] INFO: task nfsd:8866 blocked for
> > > > > > > > > > more than 122 seconds.
> > > > > > > > > > [Mon Mar 11 14:10:08 2024] Not tainted
> > > > > > > > > > 5.14.0-419.el9.x86_64 #1
> > > > > > > > > > [Mon Mar 11 14:10:08 2024] "echo 0 >
> > > > > > > > > > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> > > > > > > > > > [Mon Mar 11 14:10:08 2024]task:nfsd state:D stack:0
> > > > > > > > > > pid:8866 ppid:2 flags:0x00004000
> > > > > > > > > > [Mon Mar 11 14:10:08 2024] Call Trace:
> > > > > > > > > > [Mon Mar 11 14:10:08 2024] <TASK>
> > > > > > > > > > [Mon Mar 11 14:10:08 2024] __schedule+0x21b/0x550
> > > > > > > > > > [Mon Mar 11 14:10:08 2024] schedule+0x2d/0x70
> > > > > > > > > > [Mon Mar 11 14:10:08 2024] schedule_timeout+0x11f/0x160
> > > > > > > > > > [Mon Mar 11 14:10:08 2024] ? select_idle_sibling+0x28/0x430
> > > > > > > > > > [Mon Mar 11 14:10:08 2024] ? tcp_recvmsg+0x196/0x210
> > > > > > > > > > [Mon Mar 11 14:10:08 2024] ? wake_affine+0x62/0x1f0
> > > > > > > > > > [Mon Mar 11 14:10:08 2024] __wait_for_common+0x90/0x1d0
> > > > > > > > > > [Mon Mar 11 14:10:08 2024] ? __pfx_schedule_timeout+0x10/0x10
> > > > > > > > > > [Mon Mar 11 14:10:08 2024] __flush_workqueue+0x13a/0x3f0
> > > > > > > > > > [Mon Mar 11 14:10:08 2024] nfsd4_destroy_session+0x1a4/0x240
> > > > > > > > > > [nfsd]
> > > > > > > > > > [Mon Mar 11 14:10:08 2024] nfsd4_proc_compound+0x44b/0x700 [nfsd]
> > > > > > > > > > [Mon Mar 11 14:10:08 2024] nfsd_dispatch+0x94/0x1c0 [nfsd]
> > > > > > > > > > [Mon Mar 11 14:10:08 2024] svc_process_common+0x2ec/0x660
> > > > > > > > > > [sunrpc]
> > > > > > > > > > [Mon Mar 11 14:10:08 2024] ? __pfx_nfsd_dispatch+0x10/0x10 [nfsd]
> > > > > > > > > > [Mon Mar 11 14:10:08 2024] ? __pfx_nfsd+0x10/0x10 [nfsd]
> > > > > > > > > > [Mon Mar 11 14:10:08 2024] svc_process+0x12d/0x170 [sunrpc]
> > > > > > > > > > [Mon Mar 11 14:10:08 2024] nfsd+0x84/0xb0 [nfsd]
> > > > > > > > > > [Mon Mar 11 14:10:08 2024] kthread+0xdd/0x100
> > > > > > > > > > [Mon Mar 11 14:10:08 2024] ? __pfx_kthread+0x10/0x10
> > > > > > > > > > [Mon Mar 11 14:10:08 2024] ret_from_fork+0x29/0x50
> > > > > > > > > > [Mon Mar 11 14:10:08 2024] </TASK>
> > > > > > > > > >
> > > > > > > > > The above threads are trying to flush the workqueue, so that probably
> > > > > > > > > means that they are stuck waiting on a workqueue job to finish.
> > > > > > > > > > The above is repeated a few times, and then this warning is
> > > > > > > > > > also logged:
> > > > > > > > > > [Mon Mar 11 14:12:04 2024] ------------[ cut here ]------------
> > > > > > > > > > [Mon Mar 11 14:12:04 2024] WARNING: CPU: 39 PID: 8844 at
> > > > > > > > > > fs/nfsd/nfs4state.c:4919 nfsd_break_deleg_cb+0x170/0x190 [nfsd]
> > > > > > > > > > [Mon Mar 11 14:12:05 2024] Modules linked in: nfsv4
> > > > > > > > > > dns_resolver nfs fscache netfs rpcsec_gss_krb5 rpcrdma rdma_cm
> > > > > > > > > > iw_cm ib_cm ib_core binfmt_misc bonding tls rfkill nft_counter nft_ct
> > > > > > > > > > nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_reject_inet
> > > > > > > > > > nf_reject_ipv4 nf_reject_ipv6 nft_reject nf_tables nfnetlink vfat
> > > > > > > > > > fat dm_thin_pool dm_persistent_data dm_bio_prison dm_bufio l
> > > > > > > > > > ibcrc32c dm_service_time dm_multipath intel_rapl_msr
> > > > > > > > > > intel_rapl_common intel_uncore_frequency
> > > > > > > > > > intel_uncore_frequency_common isst_if_common skx_edac nfit
> > > > > > > > > > libnvdimm ipmi_ssif x86_pkg_temp
> > > > > > > > > > _thermal intel_powerclamp coretemp kvm_intel kvm irqbypass
> > > > > > > > > > dcdbas rapl intel_cstate mgag200 i2c_algo_bit drm_shmem_helper
> > > > > > > > > > dell_smbios drm_kms_helper dell_wmi_descriptor wmi_bmof intel_u
> > > > > > > > > > ncore syscopyarea pcspkr sysfillrect mei_me sysimgblt acpi_ipmi
> > > > > > > > > > mei fb_sys_fops i2c_i801 ipmi_si intel_pch_thermal lpc_ich
> > > > > > > > > > ipmi_devintf i2c_smbus ipmi_msghandler joydev acpi_power_meter
> > > > > > > > > > nfsd auth_rpcgss nfs_acl drm lockd grace fuse sunrpc ext4
> > > > > > > > > > mbcache jbd2 sd_mod sg lpfc
> > > > > > > > > > [Mon Mar 11 14:12:05 2024] nvmet_fc nvmet nvme_fc nvme_fabrics
> > > > > > > > > > crct10dif_pclmul ahci libahci crc32_pclmul nvme_core crc32c_intel
> > > > > > > > > > ixgbe megaraid_sas libata nvme_common ghash_clmulni_int
> > > > > > > > > > el t10_pi wdat_wdt scsi_transport_fc mdio wmi dca dm_mirror
> > > > > > > > > > dm_region_hash dm_log dm_mod
> > > > > > > > > > [Mon Mar 11 14:12:05 2024] CPU: 39 PID: 8844 Comm: nfsd Not
> > > > > > > > > > tainted 5.14.0-419.el9.x86_64 #1
> > > > > > > > > > [Mon Mar 11 14:12:05 2024] Hardware name: Dell Inc. PowerEdge
> > > > > > > > > > R740/00WGD1, BIOS 2.20.1 09/13/2023
> > > > > > > > > > [Mon Mar 11 14:12:05 2024] RIP:
> > > > > > > > > > 0010:nfsd_break_deleg_cb+0x170/0x190 [nfsd]
> > > > > > > > > > [Mon Mar 11 14:12:05 2024] Code: a6 95 c5 f3 e9 ff fe ff ff 48
> > > > > > > > > > 89 df be 01 00 00 00 e8 34 b5 13 f4 48 8d bb 98 00 00 00 e8 c8 f9
> > > > > > > > > > 00 00 84 c0 0f 85 2e ff ff ff <0f> 0b e9 27 ff ff ff be
> > > > > > > > > > 02 00 00 00 48 89 df e8 0c b5 13 f4 e9 01
> > > > > > > > > > [Mon Mar 11 14:12:05 2024] RSP: 0018:ffff9929e0bb7b80 EFLAGS:
> > > > > > > > > > 00010246
> > > > > > > > > > [Mon Mar 11 14:12:05 2024] RAX: 0000000000000000 RBX:
> > > > > > > > > > ffff8ada51930900 RCX: 0000000000000024
> > > > > > > > > > [Mon Mar 11 14:12:05 2024] RDX: ffff8ada519309c8 RSI:
> > > > > > > > > > ffff8ad582933c00 RDI: 0000000000002000
> > > > > > > > > > [Mon Mar 11 14:12:05 2024] RBP: ffff8ad46bf21574 R08:
> > > > > > > > > > ffff9929e0bb7b48 R09: 0000000000000000
> > > > > > > > > > [Mon Mar 11 14:12:05 2024] R10: ffff8aec859a2948 R11:
> > > > > > > > > > 0000000000000000 R12: ffff8ad6f497c360
> > > > > > > > > > [Mon Mar 11 14:12:05 2024] R13: ffff8ad46bf21560 R14:
> > > > > > > > > > ffff8ae5942e0b10 R15: ffff8ad6f497c360
> > > > > > > > > > [Mon Mar 11 14:12:05 2024] FS: 0000000000000000(0000)
> > > > > > > > > > GS:ffff8b031fcc0000(0000) knlGS:0000000000000000
> > > > > > > > > > [Mon Mar 11 14:12:05 2024] CS: 0010 DS: 0000 ES: 0000 CR0:
> > > > > > > > > > 0000000080050033
> > > > > > > > > > [Mon Mar 11 14:12:05 2024] CR2: 00007fafe2060744 CR3:
> > > > > > > > > > 00000018e58de006 CR4: 00000000007706e0
> > > > > > > > > > [Mon Mar 11 14:12:05 2024] DR0: 0000000000000000 DR1:
> > > > > > > > > > 0000000000000000 DR2: 0000000000000000
> > > > > > > > > > [Mon Mar 11 14:12:05 2024] DR3: 0000000000000000 DR6:
> > > > > > > > > > 00000000fffe0ff0 DR7: 0000000000000400
> > > > > > > > > > [Mon Mar 11 14:12:05 2024] PKRU: 55555554
> > > > > > > > > > [Mon Mar 11 14:12:05 2024] Call Trace:
> > > > > > > > > > [Mon Mar 11 14:12:05 2024] <TASK>
> > > > > > > > > > [Mon Mar 11 14:12:05 2024] ? show_trace_log_lvl+0x1c4/0x2df
> > > > > > > > > > [Mon Mar 11 14:12:05 2024] ? show_trace_log_lvl+0x1c4/0x2df
> > > > > > > > > > [Mon Mar 11 14:12:05 2024] ? __break_lease+0x16f/0x5f0
> > > > > > > > > > [Mon Mar 11 14:12:05 2024] ? nfsd_break_deleg_cb+0x170/0x190
> > > > > > > > > > [nfsd]
> > > > > > > > > > [Mon Mar 11 14:12:05 2024] ? __warn+0x81/0x110
> > > > > > > > > > [Mon Mar 11 14:12:05 2024] ? nfsd_break_deleg_cb+0x170/0x190
> > > > > > > > > > [nfsd]
> > > > > > > > > > [Mon Mar 11 14:12:05 2024] ? report_bug+0x10a/0x140
> > > > > > > > > > [Mon Mar 11 14:12:05 2024] ? handle_bug+0x3c/0x70
> > > > > > > > > > [Mon Mar 11 14:12:05 2024] ? exc_invalid_op+0x14/0x70
> > > > > > > > > > [Mon Mar 11 14:12:05 2024] ? asm_exc_invalid_op+0x16/0x20
> > > > > > > > > > [Mon Mar 11 14:12:05 2024] ? nfsd_break_deleg_cb+0x170/0x190
> > > > > > > > > > [nfsd]
> > > > > > > > > > [Mon Mar 11 14:12:05 2024] __break_lease+0x16f/0x5f0
> > > > > > > > > > [Mon Mar 11 14:12:05 2024] ?
> > > > > > > > > > nfsd_file_lookup_locked+0x117/0x160 [nfsd]
> > > > > > > > > > [Mon Mar 11 14:12:05 2024] ? list_lru_del+0x101/0x150
> > > > > > > > > > [Mon Mar 11 14:12:05 2024] nfsd_file_do_acquire+0x790/0x830
> > > > > > > > > > [nfsd]
> > > > > > > > > > [Mon Mar 11 14:12:05 2024] nfs4_get_vfs_file+0x315/0x3a0 [nfsd]
> > > > > > > > > > [Mon Mar 11 14:12:05 2024] nfsd4_process_open2+0x430/0xa30 [nfsd]
> > > > > > > > > > [Mon Mar 11 14:12:05 2024] ? fh_verify+0x297/0x2f0 [nfsd]
> > > > > > > > > > [Mon Mar 11 14:12:05 2024] nfsd4_open+0x3ce/0x4b0 [nfsd]
> > > > > > > > > > [Mon Mar 11 14:12:05 2024] nfsd4_proc_compound+0x44b/0x700 [nfsd]
> > > > > > > > > > [Mon Mar 11 14:12:05 2024] nfsd_dispatch+0x94/0x1c0 [nfsd]
> > > > > > > > > > [Mon Mar 11 14:12:05 2024] svc_process_common+0x2ec/0x660
> > > > > > > > > > [sunrpc]
> > > > > > > > > > [Mon Mar 11 14:12:05 2024] ? __pfx_nfsd_dispatch+0x10/0x10 [nfsd]
> > > > > > > > > > [Mon Mar 11 14:12:05 2024] ? __pfx_nfsd+0x10/0x10 [nfsd]
> > > > > > > > > > [Mon Mar 11 14:12:05 2024] svc_process+0x12d/0x170 [sunrpc]
> > > > > > > > > > [Mon Mar 11 14:12:05 2024] nfsd+0x84/0xb0 [nfsd]
> > > > > > > > > > [Mon Mar 11 14:12:05 2024] kthread+0xdd/0x100
> > > > > > > > > > [Mon Mar 11 14:12:05 2024] ? __pfx_kthread+0x10/0x10
> > > > > > > > > > [Mon Mar 11 14:12:05 2024] ret_from_fork+0x29/0x50
> > > > > > > > > > [Mon Mar 11 14:12:05 2024] </TASK>
> > > > > > > > > > [Mon Mar 11 14:12:05 2024] ---[ end trace 7a039e17443dc651 ]---
> > > > > > > > > This is probably this WARN in nfsd_break_one_deleg:
> > > > > > > > >
> > > > > > > > > WARN_ON_ONCE(!nfsd4_run_cb(&dp->dl_recall));
> > > > > > > > >
> > > > > > > > > It means that a delegation break callback to the client couldn't be
> > > > > > > > > queued to the workqueue, and so it didn't run.
> > > > > > > > >
> > > > > > > > > > Could this be the same issue as described
> > > > > > > > > > here:https://lore.kernel.org/linux-nfs/af0ec881-5ebf-4feb-98ae-3ed2a77f86f1@oracle.com/
> > > > > > > > > > ?
> > > > > > > > > Yes, most likely the same problem.
> > > > > > > > If I read that thread correctly, this issue was introduced between
> > > > > > > > 6.1.63 and 6.6.3? Is it possible the EL9 5.14.0-362.18.1.el9_3
> > > > > > > > backported these changes, or were we hitting some other bug with that
> > > > > > > > version? It seems the 6.1.x kernel is not affected? If so, that
> > > > > > > > would be
> > > > > > > > the recommended kernel to run?
> > > > > > > Anything is possible. We have to identify the problem first.
> > > > > > > > > > As described in that thread, I've tried to obtain the requested
> > > > > > > > > > information.
> > > > > > > > > >
> > > > > > > > > > Is it possible this is the issue that was fixed by the patches
> > > > > > > > > > described
> > > > > > > > > > here?https://lore.kernel.org/linux-nfs/2024022054-cause-suffering-eae8@gregkh/
> > > > > > > > > >
> > > > > > > > > Doubtful. Those are targeted toward a different set of issues.
> > > > > > > > >
> > > > > > > > > If you're willing, I do have some patches queued up for CentOS here
> > > > > > > > > that
> > > > > > > > > fix some backchannel problems that could be related. I'm mainly
> > > > > > > > > waiting
> > > > > > > > > on Chuck to send these to Linus and then we'll likely merge them into
> > > > > > > > > CentOS soon afterward:
> > > > > > > > >
> > > > > > > > > https://gitlab.com/redhat/centos-stream/src/kernel/centos-stream-9/-/merge_requests/3689
> > > > > > > > >
> > > > > > > > >
> > > > > > > > If you can send me a patch file, I can rebuild the C9S kernel with that
> > > > > > > > patch and run it. It can take a while for the bug to trigger as I
> > > > > > > > believe it seems to be very workload dependent (we were running very
> > > > > > > > stable for months and now hit this bug every other week).
> > > > > > > >
> > > > > > > >
> > > > > > > It's probably simpler to just pull down the build artifacts for that MR.
> > > > > > > You have to drill down through the CI for it, but they are here:
> > > > > > >
> > > > > > > https://s3.amazonaws.com/arr-cki-prod-trusted-artifacts/index.html?prefix=trusted-artifacts/1194300175/publish_x86_64/6278921877/artifacts/
> > > > > > >
> > > > > > >
> > > > > > > There's even a repo file you can install on the box to pull them down.
> > > > > > We installed this kernel on the server 3 days ago. Today, a user
> > > > > > informed us that their screen was black after logging in. Similar to
> > > > > > other occurrences of this issue, the mount command on the client was
> > > > > > hung. But in contrast to the other times, there were no messages in
> > > > > > the logs kernel logs on the server. Even restarting the client does
> > > > > > not resolve the issue.
> > > > Ok, so you rebooted the client and it's still unable to mount? That
> > > > sounds like a server problem if so.
> > > >
> > > > Are both client and server running the same kernel?
> > > No, the server runs 5.14.0-427.3689_1194299994.el9 and the client
> > > 5.14.0-362.18.1.el9_3.
> > Ok.
> >
> > > > > > Something still seems to be wrong on the server though. When I look at
> > > > > > the directories under /proc/fs/nfsd/clients, there's still a directory
> > > > > > for the specific client, even though it's no longer running:
> > > > > >
> > > > > > # cat 155/info
> > > > > > clientid: 0xc8edb7f65f4a9ad
> > > > > > address: "10.87.31.152:819"
> > > > > > status: confirmed
> > > > > > seconds from last renew: 33163
> > > > > > name: "Linux NFSv4.2 bersalis.esat.kuleuven.be"
> > > > > > minor version: 2
> > > > > > Implementation domain: "kernel.org"
> > > > > > Implementation name: "Linux 5.14.0-362.18.1.el9_3.0.1.x86_64 #1 SMP
> > > > > > PREEMPT_DYNAMIC Sun Feb 11 13:49:23 UTC 2024 x86_64"
> > > > > > Implementation time: [0, 0]
> > > > > > callback state: DOWN
> > > > > > callback address: 10.87.31.152:0
> > > > > >
> > > > If you just shut down the client, the server won't immediately purge its
> > > > record. In fact, assuming you're running the same kernel on the server,
> > > > it won't purge the client record until there is a conflicting request
> > > > for its state.
> > > Is there a way to force such a conflicting request (to get the client
> > > record to purge)?
> > From the server or a different client, you can try opening the inodes
> > that the stuck client is holding open. If you open them for write, that
> > may trigger the server to kick out the old client record.
> >
> > The problem is that they are disconnected dentries, so finding them to
> > open via path may be difficult...
>
> I've located the file that matches one of these inodes. When I go to the
>
> location of the file on another NFS client and touch the file, the touch
> command
>
> just hangs.
>
> So I assume the server is trying to recall the delegation from the
> dis-functional client?
>
> When I run tcpdump on the dis-functional client, I see the
> CREATE_SESSION / NFS4ERR_DELAY messages, but nothing to indicate the
> server wants to revoke a delegation.
>
Right, according to the client record above, the callback channel is
DOWN, so the server can't communicate with the client. Given that there
are 33000s since the last lease renewal, the server should kick the
client out that is blocking other activity, but that doesn't seem to be
happening here.
> The entry for the dis-functional client is not removed on the server.
>
> I there anything else I can do to provide more information about this
> situation?
>
>
The main function that handles the CREATE_SESSION call is
nfsd4_create_session. It's somewhat complex and there are a number of
reasons that function could return NFS4ERR_DELAY (aka nfserr_jukebox in
the kernel code).
What I'd do at this point is turn up tracepoints and see whether they
shed any light on what's going wrong to make it continually delay your
CREATE_SESSION calls. There aren't a lot of tracepoints in that code,
however, so it may not show much.
In the absence of that, you can try to use bpftrace or something similar
to debug what's happening in that function.
>
>
> > > > > The nfsdclnts command for this client shows the following delegations:
> > > > >
> > > > > # nfsdclnts -f 155/states -t all
> > > > > Inode number | Type | Access | Deny | ip address | Filename
> > > > > 169346743 | open | r- | -- | 10.87.31.152:819 |
> > > > > disconnected dentry
> > > > > 169346743 | deleg | r | | 10.87.31.152:819 |
> > > > > disconnected dentry
> > > > > 169346746 | open | r- | -- | 10.87.31.152:819 |
> > > > > disconnected dentry
> > > > > 169346746 | deleg | r | | 10.87.31.152:819 |
> > > > > disconnected dentry
> > > > >
> > > > > I see a lot of recent patches regarding directory delegations. Could
> > > > > this be related to this?
> > > > >
> > > > > Will a 5.14.0-362.18.1.el9_3.0.1 kernel try to use a directory delegation?
> > > > >
> > > > >
> > > > No. Directory delegations are a new feature that's still under
> > > > development. They use some of the same machinery as file delegations,
> > > > but they wouldn't be a factor here.
> > > >
> > > > > > The system seems to have identified that the client is no longer
> > > > > > reachable, but the client entry does not go away. When a mount was
> > > > > > hanging on the client, there would be two directories in clients for
> > > > > > the same client. Killing the mount command clears up the second entry.
> > > > > >
> > > > > > Even after running conntrack -D on the server to remove the tcp
> > > > > > connection from the conntrack table, the entry doesn't go away and the
> > > > > > client still can not mount anything from the server.
> > > > > >
> > > > > > A tcpdump on the client while a mount was running logged the following
> > > > > > messages over and over again:
> > > > > >
> > > > > > request:
> > > > > >
> > > > > > Frame 1: 378 bytes on wire (3024 bits), 378 bytes captured (3024 bits)
> > > > > > Ethernet II, Src: HP_19:7d:4b (e0:73:e7:19:7d:4b), Dst:
> > > > > > ArubaaHe_f9:8e:00 (88:3a:30:f9:8e:00)
> > > > > > Internet Protocol Version 4, Src: 10.87.31.152, Dst: 10.86.18.14
> > > > > > Transmission Control Protocol, Src Port: 932, Dst Port: 2049, Seq: 1,
> > > > > > Ack: 1, Len: 312
> > > > > > Remote Procedure Call, Type:Call XID:0x1d3220c4
> > > > > > Network File System
> > > > > > [Program Version: 4]
> > > > > > [V4 Procedure: COMPOUND (1)]
> > > > > > GSS Data, Ops(1): CREATE_SESSION
> > > > > > Length: 152
> > > > > > GSS Sequence Number: 76
> > > > > > Tag: <EMPTY>
> > > > > > minorversion: 2
> > > > > > Operations (count: 1): CREATE_SESSION
> > > > > > [Main Opcode: CREATE_SESSION (43)]
> > > > > > GSS Checksum:
> > > > > > 00000028040404ffffffffff000000002c19055f1f8d442d594c13849628affc2797cbb2…
> > > > > > GSS Token Length: 40
> > > > > > GSS-API Generic Security Service Application Program Interface
> > > > > > krb5_blob:
> > > > > > 040404ffffffffff000000002c19055f1f8d442d594c13849628affc2797cbb23fa080b0…
> > > > > >
> > > > > > response:
> > > > > >
> > > > > > Frame 2: 206 bytes on wire (1648 bits), 206 bytes captured (1648 bits)
> > > > > > Ethernet II, Src: ArubaaHe_f9:8e:00 (88:3a:30:f9:8e:00), Dst:
> > > > > > HP_19:7d:4b (e0:73:e7:19:7d:4b)
> > > > > > Internet Protocol Version 4, Src: 10.86.18.14, Dst: 10.87.31.152
> > > > > > Transmission Control Protocol, Src Port: 2049, Dst Port: 932, Seq: 1,
> > > > > > Ack: 313, Len: 140
> > > > > > Remote Procedure Call, Type:Reply XID:0x1d3220c4
> > > > > > Network File System
> > > > > > [Program Version: 4]
> > > > > > [V4 Procedure: COMPOUND (1)]
> > > > > > GSS Data, Ops(1): CREATE_SESSION(NFS4ERR_DELAY)
> > > > > > Length: 24
> > > > > > GSS Sequence Number: 76
> > > > > > Status: NFS4ERR_DELAY (10008)
> > > > > > Tag: <EMPTY>
> > > > > > Operations (count: 1)
> > > > > > [Main Opcode: CREATE_SESSION (43)]
> > > > > > GSS Checksum:
> > > > > > 00000028040405ffffffffff000000000aa742d0798deaad1a8aa2d7c3a91bf4f6274222…
> > > > > > GSS Token Length: 40
> > > > > > GSS-API Generic Security Service Application Program Interface
> > > > > > krb5_blob:
> > > > > > 040405ffffffffff000000000aa742d0798deaad1a8aa2d7c3a91bf4f627422226d74923…
> > > > > >
> > > > > > I was hoping that giving the client a different IP address would
> > > > > > resolve the issue for this client, but it didn't. Even though the
> > > > > > client had a new IP address (hostname was kept the same), it failed to
> > > > > > mount anything from the server.
> > > > > >
> > > > Changing the IP address won't help. The client is probably using the
> > > > same long-form client id as before, so the server still identifies the
> > > > client even with the address change.
> > > How is the client id determined? Will changing the hostname of the
> > > client trigger a change of the client id?
> > In the client record you showed a bit above, there is a "name" field:
> >
> > name: "Linux NFSv4.2 bersalis.esat.kuleuven.be"
> >
> > That's the string the server uses to uniquely identify the client. So
> > yes, changing the hostname should change that string.
> >
> > > > Unfortunately, the cause of an NFS4ERR_DELAY error is tough to guess.
> > > > The client is expected to back off and retry, so if the server keeps
> > > > returning that repeatedly, then a hung mount command is expected.
> > > >
> > > > The question is why the server would keep returning DELAY. A lot of
> > > > different problems ranging from memory allocation issues to protocol
> > > > problems can result in that error. You may want to check the NFS server
> > > > and see if anything was logged there.
> > > There are no messages in the system logs that indicate any sort of
> > > memory issue. We also increased the min_kbytes_free sysctl to 2G on the
> > > server before we restarted it with the newer kernel.
> > Ok, I didn't expect to see anything like that, but it was a possibility.
> >
> > > > This is on a CREATE_SESSION call, so I wonder if the record held by the
> > > > (courteous) server is somehow blocking the attempt to reestablish the
> > > > session?
> > > >
> > > > Do you have a way to reproduce this? Since this is a centos kernel, you
> > > > could follow the page here to open a bug:
> > > Unfortunately we haven't found a reliable way to reproduce it. But we do
> > > seem to trigger it more and more lately.
> > >
> > >
> > Bummer, ok. Let us know if you figure out a way to reproduce it.
> >
> > > > https://wiki.centos.org/ReportBugs.html
> > > >
> > > >
> > > > > > I created another dump of the workqueues and worker pools on the server:
> > > > > >
> > > > > > [Mon Mar 18 14:59:33 2024] Showing busy workqueues and worker pools:
> > > > > > [Mon Mar 18 14:59:33 2024] workqueue events: flags=0x0
> > > > > > [Mon Mar 18 14:59:33 2024] pwq 54: cpus=27 node=1 flags=0x0 nice=0
> > > > > > active=1/256 refcnt=2
> > > > > > [Mon Mar 18 14:59:33 2024] pending: drm_fb_helper_damage_work
> > > > > > [drm_kms_helper]
> > > > > > [Mon Mar 18 14:59:33 2024] workqueue events_power_efficient: flags=0x80
> > > > > > [Mon Mar 18 14:59:33 2024] pwq 54: cpus=27 node=1 flags=0x0 nice=0
> > > > > > active=1/256 refcnt=2
> > > > > > [Mon Mar 18 14:59:33 2024] pending: fb_flashcursor
> > > > > > [Mon Mar 18 14:59:33 2024] workqueue mm_percpu_wq: flags=0x8
> > > > > > [Mon Mar 18 14:59:33 2024] pwq 54: cpus=27 node=1 flags=0x0 nice=0
> > > > > > active=1/256 refcnt=3
> > > > > > [Mon Mar 18 14:59:33 2024] pending: lru_add_drain_per_cpu BAR(362)
> > > > > > [Mon Mar 18 14:59:33 2024] workqueue kblockd: flags=0x18
> > > > > > [Mon Mar 18 14:59:33 2024] pwq 55: cpus=27 node=1 flags=0x0 nice=-20
> > > > > > active=1/256 refcnt=2
> > > > > > [Mon Mar 18 14:59:33 2024] pending: blk_mq_timeout_work
> > > > > >
> > > > > >
> > > > > > In contrast to last time, it doesn't show anything regarding nfs this
> > > > > > time.
> > > > > >
> > > > > > I also tried the suggestion from Dai Ngo (echo 3 >
> > > > > > /proc/sys/vm/drop_caches), but that didn't seem to make any difference.
> > > > > >
> > > > > > We haven't restarted the server yet as it seems the impact seems to
> > > > > > affect fewer clients that before. Is there anything we can run on the
> > > > > > server to further debug this?
> > > > > >
> > > > > > In the past, the issue seemed to deteriorate rapidly and resulted in
> > > > > > issues for almost all clients after about 20 minutes. This time the
> > > > > > impact seems to be less, but it's not gone.
> > > > > >
> > > > > > How can we force the NFS server to forget about a specific client? I
> > > > > > haven't tried to restart the nfs service yet as I'm afraid it will
> > > > > > fail to stop as before.
> > > > > >
> > > > Not with that kernel. There are some new administrative interfaces that
> > > > might allow that in the future, but they were just merged upstream and
> > > > aren't in that kernel.
> > > >
> > > > --
> > > > Jeff Layton <jlayton@kernel.org>
>
--
Jeff Layton <jlayton@kernel.org>
^ permalink raw reply [flat|nested] 26+ messages in thread
* Re: nfsd hangs and nfsd_break_deleg_cb+0x170/0x190 warning
2024-03-19 7:58 ` Rik Theys
2024-03-19 10:39 ` Jeff Layton
@ 2024-03-19 17:09 ` Dai Ngo
2024-03-19 19:41 ` Rik Theys
1 sibling, 1 reply; 26+ messages in thread
From: Dai Ngo @ 2024-03-19 17:09 UTC (permalink / raw)
To: Rik Theys, Jeff Layton, Linux Nfs
On 3/19/24 12:58 AM, Rik Theys wrote:
> Hi,
>
> On 3/18/24 22:54, Jeff Layton wrote:
>> On Mon, 2024-03-18 at 22:15 +0100, Rik Theys wrote:
>>> Hi,
>>>
>>> On 3/18/24 21:21, Rik Theys wrote:
>>>> Hi Jeff,
>>>>
>>>> On 3/12/24 13:47, Jeff Layton wrote:
>>>>> On Tue, 2024-03-12 at 13:24 +0100, Rik Theys wrote:
>>>>>> Hi Jeff,
>>>>>>
>>>>>> On 3/12/24 12:22, Jeff Layton wrote:
>>>>>>> On Mon, 2024-03-11 at 19:43 +0100, Rik Theys wrote:
>>>>>>>> Since a few weeks our Rocky Linux 9 NFS server has periodically
>>>>>>>> logged hung nfsd tasks. The initial effect was that some clients
>>>>>>>> could no longer access the NFS server. This got worse and worse
>>>>>>>> (probably as more nfsd threads got blocked) and we had to restart
>>>>>>>> the server. Restarting the server also failed as the NFS server
>>>>>>>> service could no longer be stopped.
>>>>>>>>
>>>>>>>> The initial kernel we noticed this behavior on was
>>>>>>>> kernel-5.14.0-362.18.1.el9_3.x86_64. Since then we've installed
>>>>>>>> kernel-5.14.0-419.el9.x86_64 from CentOS Stream 9. The same issue
>>>>>>>> happened again on this newer kernel version:
>>>>> 419 is fairly up to date with nfsd changes. There are some known bugs
>>>>> around callbacks, and there is a draft MR in flight to fix it.
>>>>>
>>>>> What kernel were you on prior to 5.14.0-362.18.1.el9_3.x86_64 ? If we
>>>>> can bracket the changes around a particular version, then that might
>>>>> help identify the problem.
>>>>>
>>>>>>>> [Mon Mar 11 14:10:08 2024] Not tainted
>>>>>>>> 5.14.0-419.el9.x86_64 #1
>>>>>>>> [Mon Mar 11 14:10:08 2024] "echo 0 >
>>>>>>>> /proc/sys/kernel/hung_task_timeout_secs" disables this message.
>>>>>>>> [Mon Mar 11 14:10:08 2024]task:nfsd state:D stack:0
>>>>>>>> pid:8865 ppid:2 flags:0x00004000
>>>>>>>> [Mon Mar 11 14:10:08 2024] Call Trace:
>>>>>>>> [Mon Mar 11 14:10:08 2024] <TASK>
>>>>>>>> [Mon Mar 11 14:10:08 2024] __schedule+0x21b/0x550
>>>>>>>> [Mon Mar 11 14:10:08 2024] schedule+0x2d/0x70
>>>>>>>> [Mon Mar 11 14:10:08 2024] schedule_timeout+0x11f/0x160
>>>>>>>> [Mon Mar 11 14:10:08 2024] ? select_idle_sibling+0x28/0x430
>>>>>>>> [Mon Mar 11 14:10:08 2024] ? wake_affine+0x62/0x1f0
>>>>>>>> [Mon Mar 11 14:10:08 2024] __wait_for_common+0x90/0x1d0
>>>>>>>> [Mon Mar 11 14:10:08 2024] ? __pfx_schedule_timeout+0x10/0x10
>>>>>>>> [Mon Mar 11 14:10:08 2024] __flush_workqueue+0x13a/0x3f0
>>>>>>>> [Mon Mar 11 14:10:08 2024] nfsd4_shutdown_callback+0x49/0x120
>>>>>>>> [nfsd]
>>>>>>>> [Mon Mar 11 14:10:08 2024] ? nfsd4_cld_remove+0x54/0x1d0
>>>>>>>> [nfsd]
>>>>>>>> [Mon Mar 11 14:10:08 2024] ?
>>>>>>>> nfsd4_return_all_client_layouts+0xc4/0xf0 [nfsd]
>>>>>>>> [Mon Mar 11 14:10:08 2024] ? nfsd4_shutdown_copy+0x68/0xc0
>>>>>>>> [nfsd]
>>>>>>>> [Mon Mar 11 14:10:08 2024] __destroy_client+0x1f3/0x290
>>>>>>>> [nfsd]
>>>>>>>> [Mon Mar 11 14:10:08 2024] nfsd4_exchange_id+0x75f/0x770
>>>>>>>> [nfsd]
>>>>>>>> [Mon Mar 11 14:10:08 2024] ? nfsd4_decode_opaque+0x3a/0x90
>>>>>>>> [nfsd]
>>>>>>>> [Mon Mar 11 14:10:08 2024] nfsd4_proc_compound+0x44b/0x700
>>>>>>>> [nfsd]
>>>>>>>> [Mon Mar 11 14:10:08 2024] nfsd_dispatch+0x94/0x1c0 [nfsd]
>>>>>>>> [Mon Mar 11 14:10:08 2024] svc_process_common+0x2ec/0x660
>>>>>>>> [sunrpc]
>>>>>>>> [Mon Mar 11 14:10:08 2024] ? __pfx_nfsd_dispatch+0x10/0x10
>>>>>>>> [nfsd]
>>>>>>>> [Mon Mar 11 14:10:08 2024] ? __pfx_nfsd+0x10/0x10 [nfsd]
>>>>>>>> [Mon Mar 11 14:10:08 2024] svc_process+0x12d/0x170 [sunrpc]
>>>>>>>> [Mon Mar 11 14:10:08 2024] nfsd+0x84/0xb0 [nfsd]
>>>>>>>> [Mon Mar 11 14:10:08 2024] kthread+0xdd/0x100
>>>>>>>> [Mon Mar 11 14:10:08 2024] ? __pfx_kthread+0x10/0x10
>>>>>>>> [Mon Mar 11 14:10:08 2024] ret_from_fork+0x29/0x50
>>>>>>>> [Mon Mar 11 14:10:08 2024] </TASK>
>>>>>>>> [Mon Mar 11 14:10:08 2024] INFO: task nfsd:8866 blocked for
>>>>>>>> more than 122 seconds.
>>>>>>>> [Mon Mar 11 14:10:08 2024] Not tainted
>>>>>>>> 5.14.0-419.el9.x86_64 #1
>>>>>>>> [Mon Mar 11 14:10:08 2024] "echo 0 >
>>>>>>>> /proc/sys/kernel/hung_task_timeout_secs" disables this message.
>>>>>>>> [Mon Mar 11 14:10:08 2024]task:nfsd state:D stack:0
>>>>>>>> pid:8866 ppid:2 flags:0x00004000
>>>>>>>> [Mon Mar 11 14:10:08 2024] Call Trace:
>>>>>>>> [Mon Mar 11 14:10:08 2024] <TASK>
>>>>>>>> [Mon Mar 11 14:10:08 2024] __schedule+0x21b/0x550
>>>>>>>> [Mon Mar 11 14:10:08 2024] schedule+0x2d/0x70
>>>>>>>> [Mon Mar 11 14:10:08 2024] schedule_timeout+0x11f/0x160
>>>>>>>> [Mon Mar 11 14:10:08 2024] ? select_idle_sibling+0x28/0x430
>>>>>>>> [Mon Mar 11 14:10:08 2024] ? tcp_recvmsg+0x196/0x210
>>>>>>>> [Mon Mar 11 14:10:08 2024] ? wake_affine+0x62/0x1f0
>>>>>>>> [Mon Mar 11 14:10:08 2024] __wait_for_common+0x90/0x1d0
>>>>>>>> [Mon Mar 11 14:10:08 2024] ? __pfx_schedule_timeout+0x10/0x10
>>>>>>>> [Mon Mar 11 14:10:08 2024] __flush_workqueue+0x13a/0x3f0
>>>>>>>> [Mon Mar 11 14:10:08 2024] nfsd4_destroy_session+0x1a4/0x240
>>>>>>>> [nfsd]
>>>>>>>> [Mon Mar 11 14:10:08 2024] nfsd4_proc_compound+0x44b/0x700
>>>>>>>> [nfsd]
>>>>>>>> [Mon Mar 11 14:10:08 2024] nfsd_dispatch+0x94/0x1c0 [nfsd]
>>>>>>>> [Mon Mar 11 14:10:08 2024] svc_process_common+0x2ec/0x660
>>>>>>>> [sunrpc]
>>>>>>>> [Mon Mar 11 14:10:08 2024] ? __pfx_nfsd_dispatch+0x10/0x10
>>>>>>>> [nfsd]
>>>>>>>> [Mon Mar 11 14:10:08 2024] ? __pfx_nfsd+0x10/0x10 [nfsd]
>>>>>>>> [Mon Mar 11 14:10:08 2024] svc_process+0x12d/0x170 [sunrpc]
>>>>>>>> [Mon Mar 11 14:10:08 2024] nfsd+0x84/0xb0 [nfsd]
>>>>>>>> [Mon Mar 11 14:10:08 2024] kthread+0xdd/0x100
>>>>>>>> [Mon Mar 11 14:10:08 2024] ? __pfx_kthread+0x10/0x10
>>>>>>>> [Mon Mar 11 14:10:08 2024] ret_from_fork+0x29/0x50
>>>>>>>> [Mon Mar 11 14:10:08 2024] </TASK>
>>>>>>>>
>>>>>>> The above threads are trying to flush the workqueue, so that
>>>>>>> probably
>>>>>>> means that they are stuck waiting on a workqueue job to finish.
>>>>>>>> The above is repeated a few times, and then this warning is
>>>>>>>> also logged:
>>>>>>>> [Mon Mar 11 14:12:04 2024] ------------[ cut here
>>>>>>>> ]------------
>>>>>>>> [Mon Mar 11 14:12:04 2024] WARNING: CPU: 39 PID: 8844 at
>>>>>>>> fs/nfsd/nfs4state.c:4919 nfsd_break_deleg_cb+0x170/0x190 [nfsd]
>>>>>>>> [Mon Mar 11 14:12:05 2024] Modules linked in: nfsv4
>>>>>>>> dns_resolver nfs fscache netfs rpcsec_gss_krb5 rpcrdma rdma_cm
>>>>>>>> iw_cm ib_cm ib_core binfmt_misc bonding tls rfkill nft_counter
>>>>>>>> nft_ct
>>>>>>>> nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_reject_inet
>>>>>>>> nf_reject_ipv4 nf_reject_ipv6 nft_reject nf_tables nfnetlink vfat
>>>>>>>> fat dm_thin_pool dm_persistent_data dm_bio_prison dm_bufio l
>>>>>>>> ibcrc32c dm_service_time dm_multipath intel_rapl_msr
>>>>>>>> intel_rapl_common intel_uncore_frequency
>>>>>>>> intel_uncore_frequency_common isst_if_common skx_edac nfit
>>>>>>>> libnvdimm ipmi_ssif x86_pkg_temp
>>>>>>>> _thermal intel_powerclamp coretemp kvm_intel kvm irqbypass
>>>>>>>> dcdbas rapl intel_cstate mgag200 i2c_algo_bit drm_shmem_helper
>>>>>>>> dell_smbios drm_kms_helper dell_wmi_descriptor wmi_bmof intel_u
>>>>>>>> ncore syscopyarea pcspkr sysfillrect mei_me sysimgblt
>>>>>>>> acpi_ipmi
>>>>>>>> mei fb_sys_fops i2c_i801 ipmi_si intel_pch_thermal lpc_ich
>>>>>>>> ipmi_devintf i2c_smbus ipmi_msghandler joydev acpi_power_meter
>>>>>>>> nfsd auth_rpcgss nfs_acl drm lockd grace fuse sunrpc ext4
>>>>>>>> mbcache jbd2 sd_mod sg lpfc
>>>>>>>> [Mon Mar 11 14:12:05 2024] nvmet_fc nvmet nvme_fc
>>>>>>>> nvme_fabrics
>>>>>>>> crct10dif_pclmul ahci libahci crc32_pclmul nvme_core crc32c_intel
>>>>>>>> ixgbe megaraid_sas libata nvme_common ghash_clmulni_int
>>>>>>>> el t10_pi wdat_wdt scsi_transport_fc mdio wmi dca dm_mirror
>>>>>>>> dm_region_hash dm_log dm_mod
>>>>>>>> [Mon Mar 11 14:12:05 2024] CPU: 39 PID: 8844 Comm: nfsd Not
>>>>>>>> tainted 5.14.0-419.el9.x86_64 #1
>>>>>>>> [Mon Mar 11 14:12:05 2024] Hardware name: Dell Inc. PowerEdge
>>>>>>>> R740/00WGD1, BIOS 2.20.1 09/13/2023
>>>>>>>> [Mon Mar 11 14:12:05 2024] RIP:
>>>>>>>> 0010:nfsd_break_deleg_cb+0x170/0x190 [nfsd]
>>>>>>>> [Mon Mar 11 14:12:05 2024] Code: a6 95 c5 f3 e9 ff fe ff ff 48
>>>>>>>> 89 df be 01 00 00 00 e8 34 b5 13 f4 48 8d bb 98 00 00 00 e8 c8 f9
>>>>>>>> 00 00 84 c0 0f 85 2e ff ff ff <0f> 0b e9 27 ff ff ff be
>>>>>>>> 02 00 00 00 48 89 df e8 0c b5 13 f4 e9 01
>>>>>>>> [Mon Mar 11 14:12:05 2024] RSP: 0018:ffff9929e0bb7b80 EFLAGS:
>>>>>>>> 00010246
>>>>>>>> [Mon Mar 11 14:12:05 2024] RAX: 0000000000000000 RBX:
>>>>>>>> ffff8ada51930900 RCX: 0000000000000024
>>>>>>>> [Mon Mar 11 14:12:05 2024] RDX: ffff8ada519309c8 RSI:
>>>>>>>> ffff8ad582933c00 RDI: 0000000000002000
>>>>>>>> [Mon Mar 11 14:12:05 2024] RBP: ffff8ad46bf21574 R08:
>>>>>>>> ffff9929e0bb7b48 R09: 0000000000000000
>>>>>>>> [Mon Mar 11 14:12:05 2024] R10: ffff8aec859a2948 R11:
>>>>>>>> 0000000000000000 R12: ffff8ad6f497c360
>>>>>>>> [Mon Mar 11 14:12:05 2024] R13: ffff8ad46bf21560 R14:
>>>>>>>> ffff8ae5942e0b10 R15: ffff8ad6f497c360
>>>>>>>> [Mon Mar 11 14:12:05 2024] FS: 0000000000000000(0000)
>>>>>>>> GS:ffff8b031fcc0000(0000) knlGS:0000000000000000
>>>>>>>> [Mon Mar 11 14:12:05 2024] CS: 0010 DS: 0000 ES: 0000 CR0:
>>>>>>>> 0000000080050033
>>>>>>>> [Mon Mar 11 14:12:05 2024] CR2: 00007fafe2060744 CR3:
>>>>>>>> 00000018e58de006 CR4: 00000000007706e0
>>>>>>>> [Mon Mar 11 14:12:05 2024] DR0: 0000000000000000 DR1:
>>>>>>>> 0000000000000000 DR2: 0000000000000000
>>>>>>>> [Mon Mar 11 14:12:05 2024] DR3: 0000000000000000 DR6:
>>>>>>>> 00000000fffe0ff0 DR7: 0000000000000400
>>>>>>>> [Mon Mar 11 14:12:05 2024] PKRU: 55555554
>>>>>>>> [Mon Mar 11 14:12:05 2024] Call Trace:
>>>>>>>> [Mon Mar 11 14:12:05 2024] <TASK>
>>>>>>>> [Mon Mar 11 14:12:05 2024] ? show_trace_log_lvl+0x1c4/0x2df
>>>>>>>> [Mon Mar 11 14:12:05 2024] ? show_trace_log_lvl+0x1c4/0x2df
>>>>>>>> [Mon Mar 11 14:12:05 2024] ? __break_lease+0x16f/0x5f0
>>>>>>>> [Mon Mar 11 14:12:05 2024] ? nfsd_break_deleg_cb+0x170/0x190
>>>>>>>> [nfsd]
>>>>>>>> [Mon Mar 11 14:12:05 2024] ? __warn+0x81/0x110
>>>>>>>> [Mon Mar 11 14:12:05 2024] ? nfsd_break_deleg_cb+0x170/0x190
>>>>>>>> [nfsd]
>>>>>>>> [Mon Mar 11 14:12:05 2024] ? report_bug+0x10a/0x140
>>>>>>>> [Mon Mar 11 14:12:05 2024] ? handle_bug+0x3c/0x70
>>>>>>>> [Mon Mar 11 14:12:05 2024] ? exc_invalid_op+0x14/0x70
>>>>>>>> [Mon Mar 11 14:12:05 2024] ? asm_exc_invalid_op+0x16/0x20
>>>>>>>> [Mon Mar 11 14:12:05 2024] ? nfsd_break_deleg_cb+0x170/0x190
>>>>>>>> [nfsd]
>>>>>>>> [Mon Mar 11 14:12:05 2024] __break_lease+0x16f/0x5f0
>>>>>>>> [Mon Mar 11 14:12:05 2024] ?
>>>>>>>> nfsd_file_lookup_locked+0x117/0x160 [nfsd]
>>>>>>>> [Mon Mar 11 14:12:05 2024] ? list_lru_del+0x101/0x150
>>>>>>>> [Mon Mar 11 14:12:05 2024] nfsd_file_do_acquire+0x790/0x830
>>>>>>>> [nfsd]
>>>>>>>> [Mon Mar 11 14:12:05 2024] nfs4_get_vfs_file+0x315/0x3a0
>>>>>>>> [nfsd]
>>>>>>>> [Mon Mar 11 14:12:05 2024] nfsd4_process_open2+0x430/0xa30
>>>>>>>> [nfsd]
>>>>>>>> [Mon Mar 11 14:12:05 2024] ? fh_verify+0x297/0x2f0 [nfsd]
>>>>>>>> [Mon Mar 11 14:12:05 2024] nfsd4_open+0x3ce/0x4b0 [nfsd]
>>>>>>>> [Mon Mar 11 14:12:05 2024] nfsd4_proc_compound+0x44b/0x700
>>>>>>>> [nfsd]
>>>>>>>> [Mon Mar 11 14:12:05 2024] nfsd_dispatch+0x94/0x1c0 [nfsd]
>>>>>>>> [Mon Mar 11 14:12:05 2024] svc_process_common+0x2ec/0x660
>>>>>>>> [sunrpc]
>>>>>>>> [Mon Mar 11 14:12:05 2024] ? __pfx_nfsd_dispatch+0x10/0x10
>>>>>>>> [nfsd]
>>>>>>>> [Mon Mar 11 14:12:05 2024] ? __pfx_nfsd+0x10/0x10 [nfsd]
>>>>>>>> [Mon Mar 11 14:12:05 2024] svc_process+0x12d/0x170 [sunrpc]
>>>>>>>> [Mon Mar 11 14:12:05 2024] nfsd+0x84/0xb0 [nfsd]
>>>>>>>> [Mon Mar 11 14:12:05 2024] kthread+0xdd/0x100
>>>>>>>> [Mon Mar 11 14:12:05 2024] ? __pfx_kthread+0x10/0x10
>>>>>>>> [Mon Mar 11 14:12:05 2024] ret_from_fork+0x29/0x50
>>>>>>>> [Mon Mar 11 14:12:05 2024] </TASK>
>>>>>>>> [Mon Mar 11 14:12:05 2024] ---[ end trace 7a039e17443dc651
>>>>>>>> ]---
>>>>>>> This is probably this WARN in nfsd_break_one_deleg:
>>>>>>>
>>>>>>> WARN_ON_ONCE(!nfsd4_run_cb(&dp->dl_recall));
>>>>>>>
>>>>>>> It means that a delegation break callback to the client couldn't be
>>>>>>> queued to the workqueue, and so it didn't run.
>>>>>>>
>>>>>>>> Could this be the same issue as described
>>>>>>>> here:https://urldefense.com/v3/__https://lore.kernel.org/linux-nfs/af0ec881-5ebf-4feb-98ae-3ed2a77f86f1@oracle.com/__;!!ACWV5N9M2RV99hQ!LV3yWeoSOhNAkRHkxFCH2tlm0iNFVD78mxnSLyP6lrX7yBVeA2TOJ4nv6oZsqLwP4kW56CMpDWhkjjwSkdBV9En7$
>>>>>>>> ?
>>>>>>> Yes, most likely the same problem.
>>>>>> If I read that thread correctly, this issue was introduced between
>>>>>> 6.1.63 and 6.6.3? Is it possible the EL9 5.14.0-362.18.1.el9_3
>>>>>> backported these changes, or were we hitting some other bug with
>>>>>> that
>>>>>> version? It seems the 6.1.x kernel is not affected? If so, that
>>>>>> would be
>>>>>> the recommended kernel to run?
>>>>> Anything is possible. We have to identify the problem first.
>>>>>>>> As described in that thread, I've tried to obtain the requested
>>>>>>>> information.
>>>>>>>>
>>>>>>>> Is it possible this is the issue that was fixed by the patches
>>>>>>>> described
>>>>>>>> here?https://urldefense.com/v3/__https://lore.kernel.org/linux-nfs/2024022054-cause-suffering-eae8@gregkh/__;!!ACWV5N9M2RV99hQ!LV3yWeoSOhNAkRHkxFCH2tlm0iNFVD78mxnSLyP6lrX7yBVeA2TOJ4nv6oZsqLwP4kW56CMpDWhkjjwSkedtUP09$
>>>>>>>>
>>>>>>> Doubtful. Those are targeted toward a different set of issues.
>>>>>>>
>>>>>>> If you're willing, I do have some patches queued up for CentOS here
>>>>>>> that
>>>>>>> fix some backchannel problems that could be related. I'm mainly
>>>>>>> waiting
>>>>>>> on Chuck to send these to Linus and then we'll likely merge them
>>>>>>> into
>>>>>>> CentOS soon afterward:
>>>>>>>
>>>>>>> https://urldefense.com/v3/__https://gitlab.com/redhat/centos-stream/src/kernel/centos-stream-9/-/merge_requests/3689__;!!ACWV5N9M2RV99hQ!LV3yWeoSOhNAkRHkxFCH2tlm0iNFVD78mxnSLyP6lrX7yBVeA2TOJ4nv6oZsqLwP4kW56CMpDWhkjjwSkdvDn8y7$
>>>>>>>
>>>>>>>
>>>>>> If you can send me a patch file, I can rebuild the C9S kernel
>>>>>> with that
>>>>>> patch and run it. It can take a while for the bug to trigger as I
>>>>>> believe it seems to be very workload dependent (we were running very
>>>>>> stable for months and now hit this bug every other week).
>>>>>>
>>>>>>
>>>>> It's probably simpler to just pull down the build artifacts for
>>>>> that MR.
>>>>> You have to drill down through the CI for it, but they are here:
>>>>>
>>>>> https://urldefense.com/v3/__https://s3.amazonaws.com/arr-cki-prod-trusted-artifacts/index.html?prefix=trusted-artifacts*1194300175*publish_x86_64*6278921877*artifacts*__;Ly8vLy8!!ACWV5N9M2RV99hQ!LV3yWeoSOhNAkRHkxFCH2tlm0iNFVD78mxnSLyP6lrX7yBVeA2TOJ4nv6oZsqLwP4kW56CMpDWhkjjwSkaP5eW8V$
>>>>>
>>>>>
>>>>> There's even a repo file you can install on the box to pull them
>>>>> down.
>>>> We installed this kernel on the server 3 days ago. Today, a user
>>>> informed us that their screen was black after logging in. Similar to
>>>> other occurrences of this issue, the mount command on the client was
>>>> hung. But in contrast to the other times, there were no messages in
>>>> the logs kernel logs on the server. Even restarting the client does
>>>> not resolve the issue.
>>
>> Ok, so you rebooted the client and it's still unable to mount? That
>> sounds like a server problem if so.
>>
>> Are both client and server running the same kernel?
> No, the server runs 5.14.0-427.3689_1194299994.el9 and the client
> 5.14.0-362.18.1.el9_3.
>>
>>>> Something still seems to be wrong on the server though. When I look at
>>>> the directories under /proc/fs/nfsd/clients, there's still a directory
>>>> for the specific client, even though it's no longer running:
>>>>
>>>> # cat 155/info
>>>> clientid: 0xc8edb7f65f4a9ad
>>>> address: "10.87.31.152:819"
>>>> status: confirmed
>>>> seconds from last renew: 33163
>>>> name: "Linux NFSv4.2 bersalis.esat.kuleuven.be"
>>>> minor version: 2
>>>> Implementation domain: "kernel.org"
>>>> Implementation name: "Linux 5.14.0-362.18.1.el9_3.0.1.x86_64 #1 SMP
>>>> PREEMPT_DYNAMIC Sun Feb 11 13:49:23 UTC 2024 x86_64"
>>>> Implementation time: [0, 0]
>>>> callback state: DOWN
>>>> callback address: 10.87.31.152:0
>>>>
>> If you just shut down the client, the server won't immediately purge its
>> record. In fact, assuming you're running the same kernel on the server,
>> it won't purge the client record until there is a conflicting request
>> for its state.
> Is there a way to force such a conflicting request (to get the client
> record to purge)?
Try:
# echo "expire" > /proc/fs/nfsd/clients/155/ctl
-Dai
>>
>>
>>> The nfsdclnts command for this client shows the following delegations:
>>>
>>> # nfsdclnts -f 155/states -t all
>>> Inode number | Type | Access | Deny | ip address | Filename
>>> 169346743 | open | r- | -- | 10.87.31.152:819 |
>>> disconnected dentry
>>> 169346743 | deleg | r | | 10.87.31.152:819 |
>>> disconnected dentry
>>> 169346746 | open | r- | -- | 10.87.31.152:819 |
>>> disconnected dentry
>>> 169346746 | deleg | r | | 10.87.31.152:819 |
>>> disconnected dentry
>>>
>>> I see a lot of recent patches regarding directory delegations. Could
>>> this be related to this?
>>>
>>> Will a 5.14.0-362.18.1.el9_3.0.1 kernel try to use a directory
>>> delegation?
>>>
>>>
>> No. Directory delegations are a new feature that's still under
>> development. They use some of the same machinery as file delegations,
>> but they wouldn't be a factor here.
>>
>>>> The system seems to have identified that the client is no longer
>>>> reachable, but the client entry does not go away. When a mount was
>>>> hanging on the client, there would be two directories in clients for
>>>> the same client. Killing the mount command clears up the second entry.
>>>>
>>>> Even after running conntrack -D on the server to remove the tcp
>>>> connection from the conntrack table, the entry doesn't go away and the
>>>> client still can not mount anything from the server.
>>>>
>>>> A tcpdump on the client while a mount was running logged the following
>>>> messages over and over again:
>>>>
>>>> request:
>>>>
>>>> Frame 1: 378 bytes on wire (3024 bits), 378 bytes captured (3024 bits)
>>>> Ethernet II, Src: HP_19:7d:4b (e0:73:e7:19:7d:4b), Dst:
>>>> ArubaaHe_f9:8e:00 (88:3a:30:f9:8e:00)
>>>> Internet Protocol Version 4, Src: 10.87.31.152, Dst: 10.86.18.14
>>>> Transmission Control Protocol, Src Port: 932, Dst Port: 2049, Seq: 1,
>>>> Ack: 1, Len: 312
>>>> Remote Procedure Call, Type:Call XID:0x1d3220c4
>>>> Network File System
>>>> [Program Version: 4]
>>>> [V4 Procedure: COMPOUND (1)]
>>>> GSS Data, Ops(1): CREATE_SESSION
>>>> Length: 152
>>>> GSS Sequence Number: 76
>>>> Tag: <EMPTY>
>>>> minorversion: 2
>>>> Operations (count: 1): CREATE_SESSION
>>>> [Main Opcode: CREATE_SESSION (43)]
>>>> GSS Checksum:
>>>> 00000028040404ffffffffff000000002c19055f1f8d442d594c13849628affc2797cbb2…
>>>>
>>>> GSS Token Length: 40
>>>> GSS-API Generic Security Service Application Program
>>>> Interface
>>>> krb5_blob:
>>>> 040404ffffffffff000000002c19055f1f8d442d594c13849628affc2797cbb23fa080b0…
>>>>
>>>>
>>>> response:
>>>>
>>>> Frame 2: 206 bytes on wire (1648 bits), 206 bytes captured (1648 bits)
>>>> Ethernet II, Src: ArubaaHe_f9:8e:00 (88:3a:30:f9:8e:00), Dst:
>>>> HP_19:7d:4b (e0:73:e7:19:7d:4b)
>>>> Internet Protocol Version 4, Src: 10.86.18.14, Dst: 10.87.31.152
>>>> Transmission Control Protocol, Src Port: 2049, Dst Port: 932, Seq: 1,
>>>> Ack: 313, Len: 140
>>>> Remote Procedure Call, Type:Reply XID:0x1d3220c4
>>>> Network File System
>>>> [Program Version: 4]
>>>> [V4 Procedure: COMPOUND (1)]
>>>> GSS Data, Ops(1): CREATE_SESSION(NFS4ERR_DELAY)
>>>> Length: 24
>>>> GSS Sequence Number: 76
>>>> Status: NFS4ERR_DELAY (10008)
>>>> Tag: <EMPTY>
>>>> Operations (count: 1)
>>>> [Main Opcode: CREATE_SESSION (43)]
>>>> GSS Checksum:
>>>> 00000028040405ffffffffff000000000aa742d0798deaad1a8aa2d7c3a91bf4f6274222…
>>>>
>>>> GSS Token Length: 40
>>>> GSS-API Generic Security Service Application Program
>>>> Interface
>>>> krb5_blob:
>>>> 040405ffffffffff000000000aa742d0798deaad1a8aa2d7c3a91bf4f627422226d74923…
>>>>
>>>>
>>>> I was hoping that giving the client a different IP address would
>>>> resolve the issue for this client, but it didn't. Even though the
>>>> client had a new IP address (hostname was kept the same), it failed to
>>>> mount anything from the server.
>>>>
>> Changing the IP address won't help. The client is probably using the
>> same long-form client id as before, so the server still identifies the
>> client even with the address change.
> How is the client id determined? Will changing the hostname of the
> client trigger a change of the client id?
>>
>> Unfortunately, the cause of an NFS4ERR_DELAY error is tough to guess.
>> The client is expected to back off and retry, so if the server keeps
>> returning that repeatedly, then a hung mount command is expected.
>>
>> The question is why the server would keep returning DELAY. A lot of
>> different problems ranging from memory allocation issues to protocol
>> problems can result in that error. You may want to check the NFS server
>> and see if anything was logged there.
> There are no messages in the system logs that indicate any sort of
> memory issue. We also increased the min_kbytes_free sysctl to 2G on
> the server before we restarted it with the newer kernel.
>>
>> This is on a CREATE_SESSION call, so I wonder if the record held by the
>> (courteous) server is somehow blocking the attempt to reestablish the
>> session?
>>
>> Do you have a way to reproduce this? Since this is a centos kernel, you
>> could follow the page here to open a bug:
>
> Unfortunately we haven't found a reliable way to reproduce it. But we
> do seem to trigger it more and more lately.
>
> Regards,
>
> Rik
>
>>
>> https://urldefense.com/v3/__https://wiki.centos.org/ReportBugs.html__;!!ACWV5N9M2RV99hQ!LV3yWeoSOhNAkRHkxFCH2tlm0iNFVD78mxnSLyP6lrX7yBVeA2TOJ4nv6oZsqLwP4kW56CMpDWhkjjwSkWIqsboq$
>>
>>
>>>> I created another dump of the workqueues and worker pools on the
>>>> server:
>>>>
>>>> [Mon Mar 18 14:59:33 2024] Showing busy workqueues and worker pools:
>>>> [Mon Mar 18 14:59:33 2024] workqueue events: flags=0x0
>>>> [Mon Mar 18 14:59:33 2024] pwq 54: cpus=27 node=1 flags=0x0 nice=0
>>>> active=1/256 refcnt=2
>>>> [Mon Mar 18 14:59:33 2024] pending: drm_fb_helper_damage_work
>>>> [drm_kms_helper]
>>>> [Mon Mar 18 14:59:33 2024] workqueue events_power_efficient:
>>>> flags=0x80
>>>> [Mon Mar 18 14:59:33 2024] pwq 54: cpus=27 node=1 flags=0x0 nice=0
>>>> active=1/256 refcnt=2
>>>> [Mon Mar 18 14:59:33 2024] pending: fb_flashcursor
>>>> [Mon Mar 18 14:59:33 2024] workqueue mm_percpu_wq: flags=0x8
>>>> [Mon Mar 18 14:59:33 2024] pwq 54: cpus=27 node=1 flags=0x0 nice=0
>>>> active=1/256 refcnt=3
>>>> [Mon Mar 18 14:59:33 2024] pending: lru_add_drain_per_cpu BAR(362)
>>>> [Mon Mar 18 14:59:33 2024] workqueue kblockd: flags=0x18
>>>> [Mon Mar 18 14:59:33 2024] pwq 55: cpus=27 node=1 flags=0x0 nice=-20
>>>> active=1/256 refcnt=2
>>>> [Mon Mar 18 14:59:33 2024] pending: blk_mq_timeout_work
>>>>
>>>>
>>>> In contrast to last time, it doesn't show anything regarding nfs this
>>>> time.
>>>>
>>>> I also tried the suggestion from Dai Ngo (echo 3 >
>>>> /proc/sys/vm/drop_caches), but that didn't seem to make any
>>>> difference.
>>>>
>>>> We haven't restarted the server yet as it seems the impact seems to
>>>> affect fewer clients that before. Is there anything we can run on the
>>>> server to further debug this?
>>>>
>>>> In the past, the issue seemed to deteriorate rapidly and resulted in
>>>> issues for almost all clients after about 20 minutes. This time the
>>>> impact seems to be less, but it's not gone.
>>>>
>>>> How can we force the NFS server to forget about a specific client? I
>>>> haven't tried to restart the nfs service yet as I'm afraid it will
>>>> fail to stop as before.
>>>>
>> Not with that kernel. There are some new administrative interfaces that
>> might allow that in the future, but they were just merged upstream and
>> aren't in that kernel.
>>
>> --
>> Jeff Layton <jlayton@kernel.org>
>
^ permalink raw reply [flat|nested] 26+ messages in thread
* Re: nfsd hangs and nfsd_break_deleg_cb+0x170/0x190 warning
2024-03-19 17:09 ` Dai Ngo
@ 2024-03-19 19:41 ` Rik Theys
2024-03-19 20:55 ` Jeff Layton
2024-03-19 21:42 ` Dai Ngo
0 siblings, 2 replies; 26+ messages in thread
From: Rik Theys @ 2024-03-19 19:41 UTC (permalink / raw)
To: Dai Ngo, Jeff Layton, Linux Nfs
Hi,
On 3/19/24 18:09, Dai Ngo wrote:
>
> On 3/19/24 12:58 AM, Rik Theys wrote:
>> Hi,
>>
>> On 3/18/24 22:54, Jeff Layton wrote:
>>> On Mon, 2024-03-18 at 22:15 +0100, Rik Theys wrote:
>>>> Hi,
>>>>
>>>> On 3/18/24 21:21, Rik Theys wrote:
>>>>> Hi Jeff,
>>>>>
>>>>> On 3/12/24 13:47, Jeff Layton wrote:
>>>>>> On Tue, 2024-03-12 at 13:24 +0100, Rik Theys wrote:
>>>>>>> Hi Jeff,
>>>>>>>
>>>>>>> On 3/12/24 12:22, Jeff Layton wrote:
>>>>>>>> On Mon, 2024-03-11 at 19:43 +0100, Rik Theys wrote:
>>>>>>>>> Since a few weeks our Rocky Linux 9 NFS server has periodically
>>>>>>>>> logged hung nfsd tasks. The initial effect was that some clients
>>>>>>>>> could no longer access the NFS server. This got worse and worse
>>>>>>>>> (probably as more nfsd threads got blocked) and we had to restart
>>>>>>>>> the server. Restarting the server also failed as the NFS server
>>>>>>>>> service could no longer be stopped.
>>>>>>>>>
>>>>>>>>> The initial kernel we noticed this behavior on was
>>>>>>>>> kernel-5.14.0-362.18.1.el9_3.x86_64. Since then we've installed
>>>>>>>>> kernel-5.14.0-419.el9.x86_64 from CentOS Stream 9. The same issue
>>>>>>>>> happened again on this newer kernel version:
>>>>>> 419 is fairly up to date with nfsd changes. There are some known
>>>>>> bugs
>>>>>> around callbacks, and there is a draft MR in flight to fix it.
>>>>>>
>>>>>> What kernel were you on prior to 5.14.0-362.18.1.el9_3.x86_64 ?
>>>>>> If we
>>>>>> can bracket the changes around a particular version, then that might
>>>>>> help identify the problem.
>>>>>>
>>>>>>>>> [Mon Mar 11 14:10:08 2024] Not tainted
>>>>>>>>> 5.14.0-419.el9.x86_64 #1
>>>>>>>>> [Mon Mar 11 14:10:08 2024] "echo 0 >
>>>>>>>>> /proc/sys/kernel/hung_task_timeout_secs" disables this message.
>>>>>>>>> [Mon Mar 11 14:10:08 2024]task:nfsd state:D
>>>>>>>>> stack:0
>>>>>>>>> pid:8865 ppid:2 flags:0x00004000
>>>>>>>>> [Mon Mar 11 14:10:08 2024] Call Trace:
>>>>>>>>> [Mon Mar 11 14:10:08 2024] <TASK>
>>>>>>>>> [Mon Mar 11 14:10:08 2024] __schedule+0x21b/0x550
>>>>>>>>> [Mon Mar 11 14:10:08 2024] schedule+0x2d/0x70
>>>>>>>>> [Mon Mar 11 14:10:08 2024] schedule_timeout+0x11f/0x160
>>>>>>>>> [Mon Mar 11 14:10:08 2024] ? select_idle_sibling+0x28/0x430
>>>>>>>>> [Mon Mar 11 14:10:08 2024] ? wake_affine+0x62/0x1f0
>>>>>>>>> [Mon Mar 11 14:10:08 2024] __wait_for_common+0x90/0x1d0
>>>>>>>>> [Mon Mar 11 14:10:08 2024] ?
>>>>>>>>> __pfx_schedule_timeout+0x10/0x10
>>>>>>>>> [Mon Mar 11 14:10:08 2024] __flush_workqueue+0x13a/0x3f0
>>>>>>>>> [Mon Mar 11 14:10:08 2024]
>>>>>>>>> nfsd4_shutdown_callback+0x49/0x120
>>>>>>>>> [nfsd]
>>>>>>>>> [Mon Mar 11 14:10:08 2024] ? nfsd4_cld_remove+0x54/0x1d0
>>>>>>>>> [nfsd]
>>>>>>>>> [Mon Mar 11 14:10:08 2024] ?
>>>>>>>>> nfsd4_return_all_client_layouts+0xc4/0xf0 [nfsd]
>>>>>>>>> [Mon Mar 11 14:10:08 2024] ?
>>>>>>>>> nfsd4_shutdown_copy+0x68/0xc0 [nfsd]
>>>>>>>>> [Mon Mar 11 14:10:08 2024] __destroy_client+0x1f3/0x290
>>>>>>>>> [nfsd]
>>>>>>>>> [Mon Mar 11 14:10:08 2024] nfsd4_exchange_id+0x75f/0x770
>>>>>>>>> [nfsd]
>>>>>>>>> [Mon Mar 11 14:10:08 2024] ?
>>>>>>>>> nfsd4_decode_opaque+0x3a/0x90 [nfsd]
>>>>>>>>> [Mon Mar 11 14:10:08 2024]
>>>>>>>>> nfsd4_proc_compound+0x44b/0x700 [nfsd]
>>>>>>>>> [Mon Mar 11 14:10:08 2024] nfsd_dispatch+0x94/0x1c0 [nfsd]
>>>>>>>>> [Mon Mar 11 14:10:08 2024] svc_process_common+0x2ec/0x660
>>>>>>>>> [sunrpc]
>>>>>>>>> [Mon Mar 11 14:10:08 2024] ?
>>>>>>>>> __pfx_nfsd_dispatch+0x10/0x10 [nfsd]
>>>>>>>>> [Mon Mar 11 14:10:08 2024] ? __pfx_nfsd+0x10/0x10 [nfsd]
>>>>>>>>> [Mon Mar 11 14:10:08 2024] svc_process+0x12d/0x170 [sunrpc]
>>>>>>>>> [Mon Mar 11 14:10:08 2024] nfsd+0x84/0xb0 [nfsd]
>>>>>>>>> [Mon Mar 11 14:10:08 2024] kthread+0xdd/0x100
>>>>>>>>> [Mon Mar 11 14:10:08 2024] ? __pfx_kthread+0x10/0x10
>>>>>>>>> [Mon Mar 11 14:10:08 2024] ret_from_fork+0x29/0x50
>>>>>>>>> [Mon Mar 11 14:10:08 2024] </TASK>
>>>>>>>>> [Mon Mar 11 14:10:08 2024] INFO: task nfsd:8866 blocked for
>>>>>>>>> more than 122 seconds.
>>>>>>>>> [Mon Mar 11 14:10:08 2024] Not tainted
>>>>>>>>> 5.14.0-419.el9.x86_64 #1
>>>>>>>>> [Mon Mar 11 14:10:08 2024] "echo 0 >
>>>>>>>>> /proc/sys/kernel/hung_task_timeout_secs" disables this message.
>>>>>>>>> [Mon Mar 11 14:10:08 2024]task:nfsd state:D
>>>>>>>>> stack:0
>>>>>>>>> pid:8866 ppid:2 flags:0x00004000
>>>>>>>>> [Mon Mar 11 14:10:08 2024] Call Trace:
>>>>>>>>> [Mon Mar 11 14:10:08 2024] <TASK>
>>>>>>>>> [Mon Mar 11 14:10:08 2024] __schedule+0x21b/0x550
>>>>>>>>> [Mon Mar 11 14:10:08 2024] schedule+0x2d/0x70
>>>>>>>>> [Mon Mar 11 14:10:08 2024] schedule_timeout+0x11f/0x160
>>>>>>>>> [Mon Mar 11 14:10:08 2024] ? select_idle_sibling+0x28/0x430
>>>>>>>>> [Mon Mar 11 14:10:08 2024] ? tcp_recvmsg+0x196/0x210
>>>>>>>>> [Mon Mar 11 14:10:08 2024] ? wake_affine+0x62/0x1f0
>>>>>>>>> [Mon Mar 11 14:10:08 2024] __wait_for_common+0x90/0x1d0
>>>>>>>>> [Mon Mar 11 14:10:08 2024] ?
>>>>>>>>> __pfx_schedule_timeout+0x10/0x10
>>>>>>>>> [Mon Mar 11 14:10:08 2024] __flush_workqueue+0x13a/0x3f0
>>>>>>>>> [Mon Mar 11 14:10:08 2024] nfsd4_destroy_session+0x1a4/0x240
>>>>>>>>> [nfsd]
>>>>>>>>> [Mon Mar 11 14:10:08 2024]
>>>>>>>>> nfsd4_proc_compound+0x44b/0x700 [nfsd]
>>>>>>>>> [Mon Mar 11 14:10:08 2024] nfsd_dispatch+0x94/0x1c0 [nfsd]
>>>>>>>>> [Mon Mar 11 14:10:08 2024] svc_process_common+0x2ec/0x660
>>>>>>>>> [sunrpc]
>>>>>>>>> [Mon Mar 11 14:10:08 2024] ?
>>>>>>>>> __pfx_nfsd_dispatch+0x10/0x10 [nfsd]
>>>>>>>>> [Mon Mar 11 14:10:08 2024] ? __pfx_nfsd+0x10/0x10 [nfsd]
>>>>>>>>> [Mon Mar 11 14:10:08 2024] svc_process+0x12d/0x170 [sunrpc]
>>>>>>>>> [Mon Mar 11 14:10:08 2024] nfsd+0x84/0xb0 [nfsd]
>>>>>>>>> [Mon Mar 11 14:10:08 2024] kthread+0xdd/0x100
>>>>>>>>> [Mon Mar 11 14:10:08 2024] ? __pfx_kthread+0x10/0x10
>>>>>>>>> [Mon Mar 11 14:10:08 2024] ret_from_fork+0x29/0x50
>>>>>>>>> [Mon Mar 11 14:10:08 2024] </TASK>
>>>>>>>>>
>>>>>>>> The above threads are trying to flush the workqueue, so that
>>>>>>>> probably
>>>>>>>> means that they are stuck waiting on a workqueue job to finish.
>>>>>>>>> The above is repeated a few times, and then this warning is
>>>>>>>>> also logged:
>>>>>>>>> [Mon Mar 11 14:12:04 2024] ------------[ cut here
>>>>>>>>> ]------------
>>>>>>>>> [Mon Mar 11 14:12:04 2024] WARNING: CPU: 39 PID: 8844 at
>>>>>>>>> fs/nfsd/nfs4state.c:4919 nfsd_break_deleg_cb+0x170/0x190 [nfsd]
>>>>>>>>> [Mon Mar 11 14:12:05 2024] Modules linked in: nfsv4
>>>>>>>>> dns_resolver nfs fscache netfs rpcsec_gss_krb5 rpcrdma rdma_cm
>>>>>>>>> iw_cm ib_cm ib_core binfmt_misc bonding tls rfkill nft_counter
>>>>>>>>> nft_ct
>>>>>>>>> nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_reject_inet
>>>>>>>>> nf_reject_ipv4 nf_reject_ipv6 nft_reject nf_tables nfnetlink vfat
>>>>>>>>> fat dm_thin_pool dm_persistent_data dm_bio_prison dm_bufio l
>>>>>>>>> ibcrc32c dm_service_time dm_multipath intel_rapl_msr
>>>>>>>>> intel_rapl_common intel_uncore_frequency
>>>>>>>>> intel_uncore_frequency_common isst_if_common skx_edac nfit
>>>>>>>>> libnvdimm ipmi_ssif x86_pkg_temp
>>>>>>>>> _thermal intel_powerclamp coretemp kvm_intel kvm irqbypass
>>>>>>>>> dcdbas rapl intel_cstate mgag200 i2c_algo_bit drm_shmem_helper
>>>>>>>>> dell_smbios drm_kms_helper dell_wmi_descriptor wmi_bmof intel_u
>>>>>>>>> ncore syscopyarea pcspkr sysfillrect mei_me sysimgblt
>>>>>>>>> acpi_ipmi
>>>>>>>>> mei fb_sys_fops i2c_i801 ipmi_si intel_pch_thermal lpc_ich
>>>>>>>>> ipmi_devintf i2c_smbus ipmi_msghandler joydev acpi_power_meter
>>>>>>>>> nfsd auth_rpcgss nfs_acl drm lockd grace fuse sunrpc ext4
>>>>>>>>> mbcache jbd2 sd_mod sg lpfc
>>>>>>>>> [Mon Mar 11 14:12:05 2024] nvmet_fc nvmet nvme_fc
>>>>>>>>> nvme_fabrics
>>>>>>>>> crct10dif_pclmul ahci libahci crc32_pclmul nvme_core crc32c_intel
>>>>>>>>> ixgbe megaraid_sas libata nvme_common ghash_clmulni_int
>>>>>>>>> el t10_pi wdat_wdt scsi_transport_fc mdio wmi dca dm_mirror
>>>>>>>>> dm_region_hash dm_log dm_mod
>>>>>>>>> [Mon Mar 11 14:12:05 2024] CPU: 39 PID: 8844 Comm: nfsd Not
>>>>>>>>> tainted 5.14.0-419.el9.x86_64 #1
>>>>>>>>> [Mon Mar 11 14:12:05 2024] Hardware name: Dell Inc. PowerEdge
>>>>>>>>> R740/00WGD1, BIOS 2.20.1 09/13/2023
>>>>>>>>> [Mon Mar 11 14:12:05 2024] RIP:
>>>>>>>>> 0010:nfsd_break_deleg_cb+0x170/0x190 [nfsd]
>>>>>>>>> [Mon Mar 11 14:12:05 2024] Code: a6 95 c5 f3 e9 ff fe ff
>>>>>>>>> ff 48
>>>>>>>>> 89 df be 01 00 00 00 e8 34 b5 13 f4 48 8d bb 98 00 00 00 e8 c8 f9
>>>>>>>>> 00 00 84 c0 0f 85 2e ff ff ff <0f> 0b e9 27 ff ff ff be
>>>>>>>>> 02 00 00 00 48 89 df e8 0c b5 13 f4 e9 01
>>>>>>>>> [Mon Mar 11 14:12:05 2024] RSP: 0018:ffff9929e0bb7b80 EFLAGS:
>>>>>>>>> 00010246
>>>>>>>>> [Mon Mar 11 14:12:05 2024] RAX: 0000000000000000 RBX:
>>>>>>>>> ffff8ada51930900 RCX: 0000000000000024
>>>>>>>>> [Mon Mar 11 14:12:05 2024] RDX: ffff8ada519309c8 RSI:
>>>>>>>>> ffff8ad582933c00 RDI: 0000000000002000
>>>>>>>>> [Mon Mar 11 14:12:05 2024] RBP: ffff8ad46bf21574 R08:
>>>>>>>>> ffff9929e0bb7b48 R09: 0000000000000000
>>>>>>>>> [Mon Mar 11 14:12:05 2024] R10: ffff8aec859a2948 R11:
>>>>>>>>> 0000000000000000 R12: ffff8ad6f497c360
>>>>>>>>> [Mon Mar 11 14:12:05 2024] R13: ffff8ad46bf21560 R14:
>>>>>>>>> ffff8ae5942e0b10 R15: ffff8ad6f497c360
>>>>>>>>> [Mon Mar 11 14:12:05 2024] FS: 0000000000000000(0000)
>>>>>>>>> GS:ffff8b031fcc0000(0000) knlGS:0000000000000000
>>>>>>>>> [Mon Mar 11 14:12:05 2024] CS: 0010 DS: 0000 ES: 0000 CR0:
>>>>>>>>> 0000000080050033
>>>>>>>>> [Mon Mar 11 14:12:05 2024] CR2: 00007fafe2060744 CR3:
>>>>>>>>> 00000018e58de006 CR4: 00000000007706e0
>>>>>>>>> [Mon Mar 11 14:12:05 2024] DR0: 0000000000000000 DR1:
>>>>>>>>> 0000000000000000 DR2: 0000000000000000
>>>>>>>>> [Mon Mar 11 14:12:05 2024] DR3: 0000000000000000 DR6:
>>>>>>>>> 00000000fffe0ff0 DR7: 0000000000000400
>>>>>>>>> [Mon Mar 11 14:12:05 2024] PKRU: 55555554
>>>>>>>>> [Mon Mar 11 14:12:05 2024] Call Trace:
>>>>>>>>> [Mon Mar 11 14:12:05 2024] <TASK>
>>>>>>>>> [Mon Mar 11 14:12:05 2024] ? show_trace_log_lvl+0x1c4/0x2df
>>>>>>>>> [Mon Mar 11 14:12:05 2024] ? show_trace_log_lvl+0x1c4/0x2df
>>>>>>>>> [Mon Mar 11 14:12:05 2024] ? __break_lease+0x16f/0x5f0
>>>>>>>>> [Mon Mar 11 14:12:05 2024] ? nfsd_break_deleg_cb+0x170/0x190
>>>>>>>>> [nfsd]
>>>>>>>>> [Mon Mar 11 14:12:05 2024] ? __warn+0x81/0x110
>>>>>>>>> [Mon Mar 11 14:12:05 2024] ? nfsd_break_deleg_cb+0x170/0x190
>>>>>>>>> [nfsd]
>>>>>>>>> [Mon Mar 11 14:12:05 2024] ? report_bug+0x10a/0x140
>>>>>>>>> [Mon Mar 11 14:12:05 2024] ? handle_bug+0x3c/0x70
>>>>>>>>> [Mon Mar 11 14:12:05 2024] ? exc_invalid_op+0x14/0x70
>>>>>>>>> [Mon Mar 11 14:12:05 2024] ? asm_exc_invalid_op+0x16/0x20
>>>>>>>>> [Mon Mar 11 14:12:05 2024] ? nfsd_break_deleg_cb+0x170/0x190
>>>>>>>>> [nfsd]
>>>>>>>>> [Mon Mar 11 14:12:05 2024] __break_lease+0x16f/0x5f0
>>>>>>>>> [Mon Mar 11 14:12:05 2024] ?
>>>>>>>>> nfsd_file_lookup_locked+0x117/0x160 [nfsd]
>>>>>>>>> [Mon Mar 11 14:12:05 2024] ? list_lru_del+0x101/0x150
>>>>>>>>> [Mon Mar 11 14:12:05 2024] nfsd_file_do_acquire+0x790/0x830
>>>>>>>>> [nfsd]
>>>>>>>>> [Mon Mar 11 14:12:05 2024] nfs4_get_vfs_file+0x315/0x3a0
>>>>>>>>> [nfsd]
>>>>>>>>> [Mon Mar 11 14:12:05 2024]
>>>>>>>>> nfsd4_process_open2+0x430/0xa30 [nfsd]
>>>>>>>>> [Mon Mar 11 14:12:05 2024] ? fh_verify+0x297/0x2f0 [nfsd]
>>>>>>>>> [Mon Mar 11 14:12:05 2024] nfsd4_open+0x3ce/0x4b0 [nfsd]
>>>>>>>>> [Mon Mar 11 14:12:05 2024]
>>>>>>>>> nfsd4_proc_compound+0x44b/0x700 [nfsd]
>>>>>>>>> [Mon Mar 11 14:12:05 2024] nfsd_dispatch+0x94/0x1c0 [nfsd]
>>>>>>>>> [Mon Mar 11 14:12:05 2024] svc_process_common+0x2ec/0x660
>>>>>>>>> [sunrpc]
>>>>>>>>> [Mon Mar 11 14:12:05 2024] ?
>>>>>>>>> __pfx_nfsd_dispatch+0x10/0x10 [nfsd]
>>>>>>>>> [Mon Mar 11 14:12:05 2024] ? __pfx_nfsd+0x10/0x10 [nfsd]
>>>>>>>>> [Mon Mar 11 14:12:05 2024] svc_process+0x12d/0x170 [sunrpc]
>>>>>>>>> [Mon Mar 11 14:12:05 2024] nfsd+0x84/0xb0 [nfsd]
>>>>>>>>> [Mon Mar 11 14:12:05 2024] kthread+0xdd/0x100
>>>>>>>>> [Mon Mar 11 14:12:05 2024] ? __pfx_kthread+0x10/0x10
>>>>>>>>> [Mon Mar 11 14:12:05 2024] ret_from_fork+0x29/0x50
>>>>>>>>> [Mon Mar 11 14:12:05 2024] </TASK>
>>>>>>>>> [Mon Mar 11 14:12:05 2024] ---[ end trace 7a039e17443dc651
>>>>>>>>> ]---
>>>>>>>> This is probably this WARN in nfsd_break_one_deleg:
>>>>>>>>
>>>>>>>> WARN_ON_ONCE(!nfsd4_run_cb(&dp->dl_recall));
>>>>>>>>
>>>>>>>> It means that a delegation break callback to the client
>>>>>>>> couldn't be
>>>>>>>> queued to the workqueue, and so it didn't run.
>>>>>>>>
>>>>>>>>> Could this be the same issue as described
>>>>>>>>> here:https://urldefense.com/v3/__https://lore.kernel.org/linux-nfs/af0ec881-5ebf-4feb-98ae-3ed2a77f86f1@oracle.com/__;!!ACWV5N9M2RV99hQ!LV3yWeoSOhNAkRHkxFCH2tlm0iNFVD78mxnSLyP6lrX7yBVeA2TOJ4nv6oZsqLwP4kW56CMpDWhkjjwSkdBV9En7$
>>>>>>>>> ?
>>>>>>>> Yes, most likely the same problem.
>>>>>>> If I read that thread correctly, this issue was introduced between
>>>>>>> 6.1.63 and 6.6.3? Is it possible the EL9 5.14.0-362.18.1.el9_3
>>>>>>> backported these changes, or were we hitting some other bug with
>>>>>>> that
>>>>>>> version? It seems the 6.1.x kernel is not affected? If so, that
>>>>>>> would be
>>>>>>> the recommended kernel to run?
>>>>>> Anything is possible. We have to identify the problem first.
>>>>>>>>> As described in that thread, I've tried to obtain the requested
>>>>>>>>> information.
>>>>>>>>>
>>>>>>>>> Is it possible this is the issue that was fixed by the patches
>>>>>>>>> described
>>>>>>>>> here?https://urldefense.com/v3/__https://lore.kernel.org/linux-nfs/2024022054-cause-suffering-eae8@gregkh/__;!!ACWV5N9M2RV99hQ!LV3yWeoSOhNAkRHkxFCH2tlm0iNFVD78mxnSLyP6lrX7yBVeA2TOJ4nv6oZsqLwP4kW56CMpDWhkjjwSkedtUP09$
>>>>>>>>>
>>>>>>>> Doubtful. Those are targeted toward a different set of issues.
>>>>>>>>
>>>>>>>> If you're willing, I do have some patches queued up for CentOS
>>>>>>>> here
>>>>>>>> that
>>>>>>>> fix some backchannel problems that could be related. I'm mainly
>>>>>>>> waiting
>>>>>>>> on Chuck to send these to Linus and then we'll likely merge
>>>>>>>> them into
>>>>>>>> CentOS soon afterward:
>>>>>>>>
>>>>>>>> https://urldefense.com/v3/__https://gitlab.com/redhat/centos-stream/src/kernel/centos-stream-9/-/merge_requests/3689__;!!ACWV5N9M2RV99hQ!LV3yWeoSOhNAkRHkxFCH2tlm0iNFVD78mxnSLyP6lrX7yBVeA2TOJ4nv6oZsqLwP4kW56CMpDWhkjjwSkdvDn8y7$
>>>>>>>>
>>>>>>>>
>>>>>>> If you can send me a patch file, I can rebuild the C9S kernel
>>>>>>> with that
>>>>>>> patch and run it. It can take a while for the bug to trigger as I
>>>>>>> believe it seems to be very workload dependent (we were running
>>>>>>> very
>>>>>>> stable for months and now hit this bug every other week).
>>>>>>>
>>>>>>>
>>>>>> It's probably simpler to just pull down the build artifacts for
>>>>>> that MR.
>>>>>> You have to drill down through the CI for it, but they are here:
>>>>>>
>>>>>> https://urldefense.com/v3/__https://s3.amazonaws.com/arr-cki-prod-trusted-artifacts/index.html?prefix=trusted-artifacts*1194300175*publish_x86_64*6278921877*artifacts*__;Ly8vLy8!!ACWV5N9M2RV99hQ!LV3yWeoSOhNAkRHkxFCH2tlm0iNFVD78mxnSLyP6lrX7yBVeA2TOJ4nv6oZsqLwP4kW56CMpDWhkjjwSkaP5eW8V$
>>>>>>
>>>>>>
>>>>>> There's even a repo file you can install on the box to pull them
>>>>>> down.
>>>>> We installed this kernel on the server 3 days ago. Today, a user
>>>>> informed us that their screen was black after logging in. Similar to
>>>>> other occurrences of this issue, the mount command on the client was
>>>>> hung. But in contrast to the other times, there were no messages in
>>>>> the logs kernel logs on the server. Even restarting the client does
>>>>> not resolve the issue.
>>>
>>> Ok, so you rebooted the client and it's still unable to mount? That
>>> sounds like a server problem if so.
>>>
>>> Are both client and server running the same kernel?
>> No, the server runs 5.14.0-427.3689_1194299994.el9 and the client
>> 5.14.0-362.18.1.el9_3.
>>>
>>>>> Something still seems to be wrong on the server though. When I
>>>>> look at
>>>>> the directories under /proc/fs/nfsd/clients, there's still a
>>>>> directory
>>>>> for the specific client, even though it's no longer running:
>>>>>
>>>>> # cat 155/info
>>>>> clientid: 0xc8edb7f65f4a9ad
>>>>> address: "10.87.31.152:819"
>>>>> status: confirmed
>>>>> seconds from last renew: 33163
>>>>> name: "Linux NFSv4.2 bersalis.esat.kuleuven.be"
>>>>> minor version: 2
>>>>> Implementation domain: "kernel.org"
>>>>> Implementation name: "Linux 5.14.0-362.18.1.el9_3.0.1.x86_64 #1 SMP
>>>>> PREEMPT_DYNAMIC Sun Feb 11 13:49:23 UTC 2024 x86_64"
>>>>> Implementation time: [0, 0]
>>>>> callback state: DOWN
>>>>> callback address: 10.87.31.152:0
>>>>>
>>> If you just shut down the client, the server won't immediately purge
>>> its
>>> record. In fact, assuming you're running the same kernel on the server,
>>> it won't purge the client record until there is a conflicting request
>>> for its state.
>> Is there a way to force such a conflicting request (to get the client
>> record to purge)?
>
> Try:
>
> # echo "expire" > /proc/fs/nfsd/clients/155/ctl
I've tried that. The command hangs and can not be interrupted with ctrl-c.
I've now also noticed in the dmesg output that the kernel issued the
following WARNING a few hours ago. It wasn't directly triggered by the
echo command above, but seems to have been triggered a few hours ago
(probably when another client started to have the same problem as more
clients are experiencing issues now).
[Tue Mar 19 14:53:44 2024] ------------[ cut here ]------------
[Tue Mar 19 14:53:44 2024] WARNING: CPU: 44 PID: 5843 at
fs/nfsd/nfs4state.c:4920 nfsd_break_deleg_cb+0x170/0x190 [nfsd]
[Tue Mar 19 14:53:44 2024] Modules linked in: nf_conntrack_netlink nfsv4
dns_resolver nfs fscache netfs binfmt_misc xsk_diag rpcsec_gss_krb5
rpcrdma rdma_cm iw_cm ib_cm ib_core bonding tls rfkill nft_counter
nft_ct nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_reject_inet
nf_reject_ipv4 nf_reject_ipv6 nft_reject nf_tables nfnetlink vfat fat
dm_thin_pool dm_persistent_data dm_bio_prison dm_bufio libcrc32c
dm_service_time dm_multipath intel_rapl_msr intel_rapl_common
intel_uncore_frequency intel_uncore_frequency_common isst_if_common
skx_edac nfit libnvdimm x86_pkg_temp_thermal intel_powerclamp coretemp
kvm_intel kvm dcdbas irqbypass ipmi_ssif rapl intel_cstate mgag200
i2c_algo_bit drm_shmem_helper drm_kms_helper dell_smbios syscopyarea
intel_uncore sysfillrect wmi_bmof dell_wmi_descriptor pcspkr sysimgblt
fb_sys_fops mei_me i2c_i801 mei intel_pch_thermal acpi_ipmi i2c_smbus
lpc_ich ipmi_si ipmi_devintf ipmi_msghandler joydev acpi_power_meter
nfsd nfs_acl lockd auth_rpcgss grace drm fuse sunrpc ext4
[Tue Mar 19 14:53:44 2024] mbcache jbd2 sd_mod sg lpfc nvmet_fc nvmet
nvme_fc nvme_fabrics crct10dif_pclmul crc32_pclmul nvme_core ixgbe
crc32c_intel ahci libahci nvme_common megaraid_sas t10_pi
ghash_clmulni_intel wdat_wdt libata scsi_transport_fc mdio dca wmi
dm_mirror dm_region_hash dm_log dm_mod
[Tue Mar 19 14:53:44 2024] CPU: 44 PID: 5843 Comm: nfsd Not tainted
5.14.0-427.3689_1194299994.el9.x86_64 #1
[Tue Mar 19 14:53:44 2024] Hardware name: Dell Inc. PowerEdge
R740/00WGD1, BIOS 2.20.1 09/13/2023
[Tue Mar 19 14:53:44 2024] RIP: 0010:nfsd_break_deleg_cb+0x170/0x190 [nfsd]
[Tue Mar 19 14:53:44 2024] Code: 76 76 cd de e9 ff fe ff ff 48 89 df be
01 00 00 00 e8 34 a1 1b df 48 8d bb 98 00 00 00 e8 a8 fe 00 00 84 c0 0f
85 2e ff ff ff <0f> 0b e9 27 ff ff ff be 02 00 00 00 48 89 df e8 0c a1
1b df e9 01
[Tue Mar 19 14:53:44 2024] RSP: 0018:ffffb2878f2cfc38 EFLAGS: 00010246
[Tue Mar 19 14:53:44 2024] RAX: 0000000000000000 RBX: ffff88d5171067b8
RCX: 0000000000000000
[Tue Mar 19 14:53:44 2024] RDX: ffff88d517106880 RSI: ffff88bdceec8600
RDI: 0000000000002000
[Tue Mar 19 14:53:44 2024] RBP: ffff88d68a38a284 R08: ffffb2878f2cfc00
R09: 0000000000000000
[Tue Mar 19 14:53:44 2024] R10: ffff88bf57dd7878 R11: 0000000000000000
R12: ffff88d5b79c4798
[Tue Mar 19 14:53:44 2024] R13: ffff88d68a38a270 R14: ffff88cab06ad0c8
R15: ffff88d5b79c4798
[Tue Mar 19 14:53:44 2024] FS: 0000000000000000(0000)
GS:ffff88d4a1180000(0000) knlGS:0000000000000000
[Tue Mar 19 14:53:44 2024] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[Tue Mar 19 14:53:44 2024] CR2: 00007fe46ef90000 CR3: 000000019d010004
CR4: 00000000007706e0
[Tue Mar 19 14:53:44 2024] DR0: 0000000000000000 DR1: 0000000000000000
DR2: 0000000000000000
[Tue Mar 19 14:53:44 2024] DR3: 0000000000000000 DR6: 00000000fffe0ff0
DR7: 0000000000000400
[Tue Mar 19 14:53:44 2024] PKRU: 55555554
[Tue Mar 19 14:53:44 2024] Call Trace:
[Tue Mar 19 14:53:44 2024] <TASK>
[Tue Mar 19 14:53:44 2024] ? show_trace_log_lvl+0x1c4/0x2df
[Tue Mar 19 14:53:44 2024] ? show_trace_log_lvl+0x1c4/0x2df
[Tue Mar 19 14:53:44 2024] ? __break_lease+0x16f/0x5f0
[Tue Mar 19 14:53:44 2024] ? nfsd_break_deleg_cb+0x170/0x190 [nfsd]
[Tue Mar 19 14:53:44 2024] ? __warn+0x81/0x110
[Tue Mar 19 14:53:44 2024] ? nfsd_break_deleg_cb+0x170/0x190 [nfsd]
[Tue Mar 19 14:53:44 2024] ? report_bug+0x10a/0x140
[Tue Mar 19 14:53:44 2024] ? handle_bug+0x3c/0x70
[Tue Mar 19 14:53:44 2024] ? exc_invalid_op+0x14/0x70
[Tue Mar 19 14:53:44 2024] ? asm_exc_invalid_op+0x16/0x20
[Tue Mar 19 14:53:44 2024] ? nfsd_break_deleg_cb+0x170/0x190 [nfsd]
[Tue Mar 19 14:53:44 2024] ? nfsd_break_deleg_cb+0x96/0x190 [nfsd]
[Tue Mar 19 14:53:44 2024] __break_lease+0x16f/0x5f0
[Tue Mar 19 14:53:44 2024] nfs4_get_vfs_file+0x164/0x3a0 [nfsd]
[Tue Mar 19 14:53:44 2024] nfsd4_process_open2+0x430/0xa30 [nfsd]
[Tue Mar 19 14:53:44 2024] ? fh_verify+0x297/0x2f0 [nfsd]
[Tue Mar 19 14:53:44 2024] nfsd4_open+0x3ce/0x4b0 [nfsd]
[Tue Mar 19 14:53:44 2024] nfsd4_proc_compound+0x44b/0x700 [nfsd]
[Tue Mar 19 14:53:44 2024] nfsd_dispatch+0x94/0x1c0 [nfsd]
[Tue Mar 19 14:53:44 2024] svc_process_common+0x2ec/0x660 [sunrpc]
[Tue Mar 19 14:53:44 2024] ? __pfx_nfsd_dispatch+0x10/0x10 [nfsd]
[Tue Mar 19 14:53:44 2024] ? __pfx_nfsd+0x10/0x10 [nfsd]
[Tue Mar 19 14:53:44 2024] svc_process+0x12d/0x170 [sunrpc]
[Tue Mar 19 14:53:44 2024] nfsd+0x84/0xb0 [nfsd]
[Tue Mar 19 14:53:44 2024] kthread+0xdd/0x100
[Tue Mar 19 14:53:44 2024] ? __pfx_kthread+0x10/0x10
[Tue Mar 19 14:53:44 2024] ret_from_fork+0x29/0x50
[Tue Mar 19 14:53:44 2024] </TASK>
[Tue Mar 19 14:53:44 2024] ---[ end trace ed0b2b3f135c637d ]---
It again seems to have been triggered in nfsd_break_deleg_cb?
I also had the following perf command running a tmux on the server:
perf trace -e nfsd:nfsd_cb_recall_any
This has spewed a lot of messages. I'm including a short list here:
...
33464866.721 kworker/u98:5/1591466 nfsd:nfsd_cb_recall_any(cl_boot:
1710533037, cl_id: 210688785, bmval0: 1, addr: 0x7f331bb116c8)
33464866.724 kworker/u98:5/1591466 nfsd:nfsd_cb_recall_any(cl_boot:
1710533037, cl_id: 210688827, bmval0: 1, addr: 0x7f331bb11738)
33464866.729 kworker/u98:5/1591466 nfsd:nfsd_cb_recall_any(cl_boot:
1710533037, cl_id: 210688767, bmval0: 1, addr: 0x7f331bb117a8)
33464866.732 kworker/u98:5/1591466 nfsd:nfsd_cb_recall_any(cl_boot:
1710533037, cl_id: 210718132, bmval0: 1, addr: 0x7f331bb11818)
33464866.737 kworker/u98:5/1591466 nfsd:nfsd_cb_recall_any(cl_boot:
1710533037, cl_id: 210688952, bmval0: 1, addr: 0x7f331bb11888)
33464866.741 kworker/u98:5/1591466 nfsd:nfsd_cb_recall_any(cl_boot:
1710533037, cl_id: 210702355, bmval0: 1, addr: 0x7f331bb118f8)
33868414.001 kthreadd/1597068 nfsd:nfsd_cb_recall_any(cl_boot:
1710533037, cl_id: 210688751, bmval0: 1, addr: 0x7f331be68620)
33868414.014 kthreadd/1597068 nfsd:nfsd_cb_recall_any(cl_boot:
1710533037, cl_id: 210718536, bmval0: 1, addr: 0x7f331be68690)
33868414.018 kthreadd/1597068 nfsd:nfsd_cb_recall_any(cl_boot:
1710533037, cl_id: 210719074, bmval0: 1, addr: 0x7f331be68700)
33868414.022 kthreadd/1597068 nfsd:nfsd_cb_recall_any(cl_boot:
1710533037, cl_id: 210688916, bmval0: 1, addr: 0x7f331be68770)
33868414.026 kthreadd/1597068 nfsd:nfsd_cb_recall_any(cl_boot:
1710533037, cl_id: 210688941, bmval0: 1, addr: 0x7f331be687e0)
...
33868414.924 kthreadd/1597068 nfsd:nfsd_cb_recall_any(cl_boot:
1710533037, cl_id: 210688744, bmval0: 1, addr: 0x7f331be6d7f0)
33868414.929 kthreadd/1597068 nfsd:nfsd_cb_recall_any(cl_boot:
1710533037, cl_id: 210717223, bmval0: 1, addr: 0x7f331be6d860)
33868414.934 kthreadd/1597068 nfsd:nfsd_cb_recall_any(cl_boot:
1710533037, cl_id: 210716137, bmval0: 1, addr: 0x7f331be6d8d0)
34021240.903 kworker/u98:5/1591466 nfsd:nfsd_cb_recall_any(cl_boot:
1710533037, cl_id: 210688941, bmval0: 1, addr: 0x7f331c207de8)
34021240.917 kworker/u98:5/1591466 nfsd:nfsd_cb_recall_any(cl_boot:
1710533037, cl_id: 210718750, bmval0: 1, addr: 0x7f331c207e58)
34021240.922 kworker/u98:5/1591466 nfsd:nfsd_cb_recall_any(cl_boot:
1710533037, cl_id: 210688955, bmval0: 1, addr: 0x7f331c207ec8)
34021240.925 kworker/u98:5/1591466 nfsd:nfsd_cb_recall_any(cl_boot:
1710533037, cl_id: 210688975, bmval0: 1, addr: 0x7f331c207f38)
...
I assume the cl_id is the client id? How can I map this to a client from
/proc/fs/nfsd/clients?
If I understand it correctly, the recall_any should be called when
either the system starts to experience memory pressure, or it reaches
the delegation limits? I doubt the system is actually running out of
memory here as there are no other indications. Shouldn't I get those
"page allocation failure" messages if it does? How can I check the
number of delegations/leases currently issued, what the current maximum
is and how to increase it?
Regarding the recall any call: from what I've read on kernelnewbies,
this feature was introduced in the 6.2 kernel? When I look at the tree
for 6.1.x, it was backported in 6.1.81? Is there a way to disable this
support somehow?
Regards,
Rik
>
> -Dai
>
>>>
>>>
>>>> The nfsdclnts command for this client shows the following delegations:
>>>>
>>>> # nfsdclnts -f 155/states -t all
>>>> Inode number | Type | Access | Deny | ip address | Filename
>>>> 169346743 | open | r- | -- | 10.87.31.152:819 |
>>>> disconnected dentry
>>>> 169346743 | deleg | r | | 10.87.31.152:819 |
>>>> disconnected dentry
>>>> 169346746 | open | r- | -- | 10.87.31.152:819 |
>>>> disconnected dentry
>>>> 169346746 | deleg | r | | 10.87.31.152:819 |
>>>> disconnected dentry
>>>>
>>>> I see a lot of recent patches regarding directory delegations. Could
>>>> this be related to this?
>>>>
>>>> Will a 5.14.0-362.18.1.el9_3.0.1 kernel try to use a directory
>>>> delegation?
>>>>
>>>>
>>> No. Directory delegations are a new feature that's still under
>>> development. They use some of the same machinery as file delegations,
>>> but they wouldn't be a factor here.
>>>
>>>>> The system seems to have identified that the client is no longer
>>>>> reachable, but the client entry does not go away. When a mount was
>>>>> hanging on the client, there would be two directories in clients for
>>>>> the same client. Killing the mount command clears up the second
>>>>> entry.
>>>>>
>>>>> Even after running conntrack -D on the server to remove the tcp
>>>>> connection from the conntrack table, the entry doesn't go away and
>>>>> the
>>>>> client still can not mount anything from the server.
>>>>>
>>>>> A tcpdump on the client while a mount was running logged the
>>>>> following
>>>>> messages over and over again:
>>>>>
>>>>> request:
>>>>>
>>>>> Frame 1: 378 bytes on wire (3024 bits), 378 bytes captured (3024
>>>>> bits)
>>>>> Ethernet II, Src: HP_19:7d:4b (e0:73:e7:19:7d:4b), Dst:
>>>>> ArubaaHe_f9:8e:00 (88:3a:30:f9:8e:00)
>>>>> Internet Protocol Version 4, Src: 10.87.31.152, Dst: 10.86.18.14
>>>>> Transmission Control Protocol, Src Port: 932, Dst Port: 2049, Seq: 1,
>>>>> Ack: 1, Len: 312
>>>>> Remote Procedure Call, Type:Call XID:0x1d3220c4
>>>>> Network File System
>>>>> [Program Version: 4]
>>>>> [V4 Procedure: COMPOUND (1)]
>>>>> GSS Data, Ops(1): CREATE_SESSION
>>>>> Length: 152
>>>>> GSS Sequence Number: 76
>>>>> Tag: <EMPTY>
>>>>> minorversion: 2
>>>>> Operations (count: 1): CREATE_SESSION
>>>>> [Main Opcode: CREATE_SESSION (43)]
>>>>> GSS Checksum:
>>>>> 00000028040404ffffffffff000000002c19055f1f8d442d594c13849628affc2797cbb2…
>>>>>
>>>>> GSS Token Length: 40
>>>>> GSS-API Generic Security Service Application Program
>>>>> Interface
>>>>> krb5_blob:
>>>>> 040404ffffffffff000000002c19055f1f8d442d594c13849628affc2797cbb23fa080b0…
>>>>>
>>>>>
>>>>> response:
>>>>>
>>>>> Frame 2: 206 bytes on wire (1648 bits), 206 bytes captured (1648
>>>>> bits)
>>>>> Ethernet II, Src: ArubaaHe_f9:8e:00 (88:3a:30:f9:8e:00), Dst:
>>>>> HP_19:7d:4b (e0:73:e7:19:7d:4b)
>>>>> Internet Protocol Version 4, Src: 10.86.18.14, Dst: 10.87.31.152
>>>>> Transmission Control Protocol, Src Port: 2049, Dst Port: 932, Seq: 1,
>>>>> Ack: 313, Len: 140
>>>>> Remote Procedure Call, Type:Reply XID:0x1d3220c4
>>>>> Network File System
>>>>> [Program Version: 4]
>>>>> [V4 Procedure: COMPOUND (1)]
>>>>> GSS Data, Ops(1): CREATE_SESSION(NFS4ERR_DELAY)
>>>>> Length: 24
>>>>> GSS Sequence Number: 76
>>>>> Status: NFS4ERR_DELAY (10008)
>>>>> Tag: <EMPTY>
>>>>> Operations (count: 1)
>>>>> [Main Opcode: CREATE_SESSION (43)]
>>>>> GSS Checksum:
>>>>> 00000028040405ffffffffff000000000aa742d0798deaad1a8aa2d7c3a91bf4f6274222…
>>>>>
>>>>> GSS Token Length: 40
>>>>> GSS-API Generic Security Service Application Program
>>>>> Interface
>>>>> krb5_blob:
>>>>> 040405ffffffffff000000000aa742d0798deaad1a8aa2d7c3a91bf4f627422226d74923…
>>>>>
>>>>>
>>>>> I was hoping that giving the client a different IP address would
>>>>> resolve the issue for this client, but it didn't. Even though the
>>>>> client had a new IP address (hostname was kept the same), it
>>>>> failed to
>>>>> mount anything from the server.
>>>>>
>>> Changing the IP address won't help. The client is probably using the
>>> same long-form client id as before, so the server still identifies the
>>> client even with the address change.
>> How is the client id determined? Will changing the hostname of the
>> client trigger a change of the client id?
>>>
>>> Unfortunately, the cause of an NFS4ERR_DELAY error is tough to guess.
>>> The client is expected to back off and retry, so if the server keeps
>>> returning that repeatedly, then a hung mount command is expected.
>>>
>>> The question is why the server would keep returning DELAY. A lot of
>>> different problems ranging from memory allocation issues to protocol
>>> problems can result in that error. You may want to check the NFS server
>>> and see if anything was logged there.
>> There are no messages in the system logs that indicate any sort of
>> memory issue. We also increased the min_kbytes_free sysctl to 2G on
>> the server before we restarted it with the newer kernel.
>>>
>>> This is on a CREATE_SESSION call, so I wonder if the record held by the
>>> (courteous) server is somehow blocking the attempt to reestablish the
>>> session?
>>>
>>> Do you have a way to reproduce this? Since this is a centos kernel, you
>>> could follow the page here to open a bug:
>>
>> Unfortunately we haven't found a reliable way to reproduce it. But we
>> do seem to trigger it more and more lately.
>>
>> Regards,
>>
>> Rik
>>
>>>
>>> https://urldefense.com/v3/__https://wiki.centos.org/ReportBugs.html__;!!ACWV5N9M2RV99hQ!LV3yWeoSOhNAkRHkxFCH2tlm0iNFVD78mxnSLyP6lrX7yBVeA2TOJ4nv6oZsqLwP4kW56CMpDWhkjjwSkWIqsboq$
>>>
>>>
>>>>> I created another dump of the workqueues and worker pools on the
>>>>> server:
>>>>>
>>>>> [Mon Mar 18 14:59:33 2024] Showing busy workqueues and worker pools:
>>>>> [Mon Mar 18 14:59:33 2024] workqueue events: flags=0x0
>>>>> [Mon Mar 18 14:59:33 2024] pwq 54: cpus=27 node=1 flags=0x0 nice=0
>>>>> active=1/256 refcnt=2
>>>>> [Mon Mar 18 14:59:33 2024] pending: drm_fb_helper_damage_work
>>>>> [drm_kms_helper]
>>>>> [Mon Mar 18 14:59:33 2024] workqueue events_power_efficient:
>>>>> flags=0x80
>>>>> [Mon Mar 18 14:59:33 2024] pwq 54: cpus=27 node=1 flags=0x0 nice=0
>>>>> active=1/256 refcnt=2
>>>>> [Mon Mar 18 14:59:33 2024] pending: fb_flashcursor
>>>>> [Mon Mar 18 14:59:33 2024] workqueue mm_percpu_wq: flags=0x8
>>>>> [Mon Mar 18 14:59:33 2024] pwq 54: cpus=27 node=1 flags=0x0 nice=0
>>>>> active=1/256 refcnt=3
>>>>> [Mon Mar 18 14:59:33 2024] pending: lru_add_drain_per_cpu
>>>>> BAR(362)
>>>>> [Mon Mar 18 14:59:33 2024] workqueue kblockd: flags=0x18
>>>>> [Mon Mar 18 14:59:33 2024] pwq 55: cpus=27 node=1 flags=0x0
>>>>> nice=-20
>>>>> active=1/256 refcnt=2
>>>>> [Mon Mar 18 14:59:33 2024] pending: blk_mq_timeout_work
>>>>>
>>>>>
>>>>> In contrast to last time, it doesn't show anything regarding nfs this
>>>>> time.
>>>>>
>>>>> I also tried the suggestion from Dai Ngo (echo 3 >
>>>>> /proc/sys/vm/drop_caches), but that didn't seem to make any
>>>>> difference.
>>>>>
>>>>> We haven't restarted the server yet as it seems the impact seems to
>>>>> affect fewer clients that before. Is there anything we can run on the
>>>>> server to further debug this?
>>>>>
>>>>> In the past, the issue seemed to deteriorate rapidly and resulted in
>>>>> issues for almost all clients after about 20 minutes. This time the
>>>>> impact seems to be less, but it's not gone.
>>>>>
>>>>> How can we force the NFS server to forget about a specific client? I
>>>>> haven't tried to restart the nfs service yet as I'm afraid it will
>>>>> fail to stop as before.
>>>>>
>>> Not with that kernel. There are some new administrative interfaces that
>>> might allow that in the future, but they were just merged upstream and
>>> aren't in that kernel.
>>>
>>> --
>>> Jeff Layton <jlayton@kernel.org>
>>
--
Rik Theys
System Engineer
KU Leuven - Dept. Elektrotechniek (ESAT)
Kasteelpark Arenberg 10 bus 2440 - B-3001 Leuven-Heverlee
+32(0)16/32.11.07
----------------------------------------------------------------
<<Any errors in spelling, tact or fact are transmission errors>>
^ permalink raw reply [flat|nested] 26+ messages in thread
* Re: nfsd hangs and nfsd_break_deleg_cb+0x170/0x190 warning
2024-03-19 19:41 ` Rik Theys
@ 2024-03-19 20:55 ` Jeff Layton
2024-03-19 21:42 ` Dai Ngo
1 sibling, 0 replies; 26+ messages in thread
From: Jeff Layton @ 2024-03-19 20:55 UTC (permalink / raw)
To: Rik Theys, Dai Ngo, Linux Nfs
On Tue, 2024-03-19 at 20:41 +0100, Rik Theys wrote:
> Hi,
>
> On 3/19/24 18:09, Dai Ngo wrote:
> >
> > On 3/19/24 12:58 AM, Rik Theys wrote:
> > > Hi,
> > >
> > > On 3/18/24 22:54, Jeff Layton wrote:
> > > > On Mon, 2024-03-18 at 22:15 +0100, Rik Theys wrote:
> > > > > Hi,
> > > > >
> > > > > On 3/18/24 21:21, Rik Theys wrote:
> > > > > > Hi Jeff,
> > > > > >
> > > > > > On 3/12/24 13:47, Jeff Layton wrote:
> > > > > > > On Tue, 2024-03-12 at 13:24 +0100, Rik Theys wrote:
> > > > > > > > Hi Jeff,
> > > > > > > >
> > > > > > > > On 3/12/24 12:22, Jeff Layton wrote:
> > > > > > > > > On Mon, 2024-03-11 at 19:43 +0100, Rik Theys wrote:
> > > > > > > > > > Since a few weeks our Rocky Linux 9 NFS server has periodically
> > > > > > > > > > logged hung nfsd tasks. The initial effect was that some clients
> > > > > > > > > > could no longer access the NFS server. This got worse and worse
> > > > > > > > > > (probably as more nfsd threads got blocked) and we had to restart
> > > > > > > > > > the server. Restarting the server also failed as the NFS server
> > > > > > > > > > service could no longer be stopped.
> > > > > > > > > >
> > > > > > > > > > The initial kernel we noticed this behavior on was
> > > > > > > > > > kernel-5.14.0-362.18.1.el9_3.x86_64. Since then we've installed
> > > > > > > > > > kernel-5.14.0-419.el9.x86_64 from CentOS Stream 9. The same issue
> > > > > > > > > > happened again on this newer kernel version:
> > > > > > > 419 is fairly up to date with nfsd changes. There are some known
> > > > > > > bugs
> > > > > > > around callbacks, and there is a draft MR in flight to fix it.
> > > > > > >
> > > > > > > What kernel were you on prior to 5.14.0-362.18.1.el9_3.x86_64 ?
> > > > > > > If we
> > > > > > > can bracket the changes around a particular version, then that might
> > > > > > > help identify the problem.
> > > > > > >
> > > > > > > > > > [Mon Mar 11 14:10:08 2024] Not tainted
> > > > > > > > > > 5.14.0-419.el9.x86_64 #1
> > > > > > > > > > [Mon Mar 11 14:10:08 2024] "echo 0 >
> > > > > > > > > > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> > > > > > > > > > [Mon Mar 11 14:10:08 2024]task:nfsd state:D
> > > > > > > > > > stack:0
> > > > > > > > > > pid:8865 ppid:2 flags:0x00004000
> > > > > > > > > > [Mon Mar 11 14:10:08 2024] Call Trace:
> > > > > > > > > > [Mon Mar 11 14:10:08 2024] <TASK>
> > > > > > > > > > [Mon Mar 11 14:10:08 2024] __schedule+0x21b/0x550
> > > > > > > > > > [Mon Mar 11 14:10:08 2024] schedule+0x2d/0x70
> > > > > > > > > > [Mon Mar 11 14:10:08 2024] schedule_timeout+0x11f/0x160
> > > > > > > > > > [Mon Mar 11 14:10:08 2024] ? select_idle_sibling+0x28/0x430
> > > > > > > > > > [Mon Mar 11 14:10:08 2024] ? wake_affine+0x62/0x1f0
> > > > > > > > > > [Mon Mar 11 14:10:08 2024] __wait_for_common+0x90/0x1d0
> > > > > > > > > > [Mon Mar 11 14:10:08 2024] ?
> > > > > > > > > > __pfx_schedule_timeout+0x10/0x10
> > > > > > > > > > [Mon Mar 11 14:10:08 2024] __flush_workqueue+0x13a/0x3f0
> > > > > > > > > > [Mon Mar 11 14:10:08 2024]
> > > > > > > > > > nfsd4_shutdown_callback+0x49/0x120
> > > > > > > > > > [nfsd]
> > > > > > > > > > [Mon Mar 11 14:10:08 2024] ? nfsd4_cld_remove+0x54/0x1d0
> > > > > > > > > > [nfsd]
> > > > > > > > > > [Mon Mar 11 14:10:08 2024] ?
> > > > > > > > > > nfsd4_return_all_client_layouts+0xc4/0xf0 [nfsd]
> > > > > > > > > > [Mon Mar 11 14:10:08 2024] ?
> > > > > > > > > > nfsd4_shutdown_copy+0x68/0xc0 [nfsd]
> > > > > > > > > > [Mon Mar 11 14:10:08 2024] __destroy_client+0x1f3/0x290
> > > > > > > > > > [nfsd]
> > > > > > > > > > [Mon Mar 11 14:10:08 2024] nfsd4_exchange_id+0x75f/0x770
> > > > > > > > > > [nfsd]
> > > > > > > > > > [Mon Mar 11 14:10:08 2024] ?
> > > > > > > > > > nfsd4_decode_opaque+0x3a/0x90 [nfsd]
> > > > > > > > > > [Mon Mar 11 14:10:08 2024]
> > > > > > > > > > nfsd4_proc_compound+0x44b/0x700 [nfsd]
> > > > > > > > > > [Mon Mar 11 14:10:08 2024] nfsd_dispatch+0x94/0x1c0 [nfsd]
> > > > > > > > > > [Mon Mar 11 14:10:08 2024] svc_process_common+0x2ec/0x660
> > > > > > > > > > [sunrpc]
> > > > > > > > > > [Mon Mar 11 14:10:08 2024] ?
> > > > > > > > > > __pfx_nfsd_dispatch+0x10/0x10 [nfsd]
> > > > > > > > > > [Mon Mar 11 14:10:08 2024] ? __pfx_nfsd+0x10/0x10 [nfsd]
> > > > > > > > > > [Mon Mar 11 14:10:08 2024] svc_process+0x12d/0x170 [sunrpc]
> > > > > > > > > > [Mon Mar 11 14:10:08 2024] nfsd+0x84/0xb0 [nfsd]
> > > > > > > > > > [Mon Mar 11 14:10:08 2024] kthread+0xdd/0x100
> > > > > > > > > > [Mon Mar 11 14:10:08 2024] ? __pfx_kthread+0x10/0x10
> > > > > > > > > > [Mon Mar 11 14:10:08 2024] ret_from_fork+0x29/0x50
> > > > > > > > > > [Mon Mar 11 14:10:08 2024] </TASK>
> > > > > > > > > > [Mon Mar 11 14:10:08 2024] INFO: task nfsd:8866 blocked for
> > > > > > > > > > more than 122 seconds.
> > > > > > > > > > [Mon Mar 11 14:10:08 2024] Not tainted
> > > > > > > > > > 5.14.0-419.el9.x86_64 #1
> > > > > > > > > > [Mon Mar 11 14:10:08 2024] "echo 0 >
> > > > > > > > > > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> > > > > > > > > > [Mon Mar 11 14:10:08 2024]task:nfsd state:D
> > > > > > > > > > stack:0
> > > > > > > > > > pid:8866 ppid:2 flags:0x00004000
> > > > > > > > > > [Mon Mar 11 14:10:08 2024] Call Trace:
> > > > > > > > > > [Mon Mar 11 14:10:08 2024] <TASK>
> > > > > > > > > > [Mon Mar 11 14:10:08 2024] __schedule+0x21b/0x550
> > > > > > > > > > [Mon Mar 11 14:10:08 2024] schedule+0x2d/0x70
> > > > > > > > > > [Mon Mar 11 14:10:08 2024] schedule_timeout+0x11f/0x160
> > > > > > > > > > [Mon Mar 11 14:10:08 2024] ? select_idle_sibling+0x28/0x430
> > > > > > > > > > [Mon Mar 11 14:10:08 2024] ? tcp_recvmsg+0x196/0x210
> > > > > > > > > > [Mon Mar 11 14:10:08 2024] ? wake_affine+0x62/0x1f0
> > > > > > > > > > [Mon Mar 11 14:10:08 2024] __wait_for_common+0x90/0x1d0
> > > > > > > > > > [Mon Mar 11 14:10:08 2024] ?
> > > > > > > > > > __pfx_schedule_timeout+0x10/0x10
> > > > > > > > > > [Mon Mar 11 14:10:08 2024] __flush_workqueue+0x13a/0x3f0
> > > > > > > > > > [Mon Mar 11 14:10:08 2024] nfsd4_destroy_session+0x1a4/0x240
> > > > > > > > > > [nfsd]
> > > > > > > > > > [Mon Mar 11 14:10:08 2024]
> > > > > > > > > > nfsd4_proc_compound+0x44b/0x700 [nfsd]
> > > > > > > > > > [Mon Mar 11 14:10:08 2024] nfsd_dispatch+0x94/0x1c0 [nfsd]
> > > > > > > > > > [Mon Mar 11 14:10:08 2024] svc_process_common+0x2ec/0x660
> > > > > > > > > > [sunrpc]
> > > > > > > > > > [Mon Mar 11 14:10:08 2024] ?
> > > > > > > > > > __pfx_nfsd_dispatch+0x10/0x10 [nfsd]
> > > > > > > > > > [Mon Mar 11 14:10:08 2024] ? __pfx_nfsd+0x10/0x10 [nfsd]
> > > > > > > > > > [Mon Mar 11 14:10:08 2024] svc_process+0x12d/0x170 [sunrpc]
> > > > > > > > > > [Mon Mar 11 14:10:08 2024] nfsd+0x84/0xb0 [nfsd]
> > > > > > > > > > [Mon Mar 11 14:10:08 2024] kthread+0xdd/0x100
> > > > > > > > > > [Mon Mar 11 14:10:08 2024] ? __pfx_kthread+0x10/0x10
> > > > > > > > > > [Mon Mar 11 14:10:08 2024] ret_from_fork+0x29/0x50
> > > > > > > > > > [Mon Mar 11 14:10:08 2024] </TASK>
> > > > > > > > > >
> > > > > > > > > The above threads are trying to flush the workqueue, so that
> > > > > > > > > probably
> > > > > > > > > means that they are stuck waiting on a workqueue job to finish.
> > > > > > > > > > The above is repeated a few times, and then this warning is
> > > > > > > > > > also logged:
> > > > > > > > > > [Mon Mar 11 14:12:04 2024] ------------[ cut here
> > > > > > > > > > ]------------
> > > > > > > > > > [Mon Mar 11 14:12:04 2024] WARNING: CPU: 39 PID: 8844 at
> > > > > > > > > > fs/nfsd/nfs4state.c:4919 nfsd_break_deleg_cb+0x170/0x190 [nfsd]
> > > > > > > > > > [Mon Mar 11 14:12:05 2024] Modules linked in: nfsv4
> > > > > > > > > > dns_resolver nfs fscache netfs rpcsec_gss_krb5 rpcrdma rdma_cm
> > > > > > > > > > iw_cm ib_cm ib_core binfmt_misc bonding tls rfkill nft_counter
> > > > > > > > > > nft_ct
> > > > > > > > > > nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_reject_inet
> > > > > > > > > > nf_reject_ipv4 nf_reject_ipv6 nft_reject nf_tables nfnetlink vfat
> > > > > > > > > > fat dm_thin_pool dm_persistent_data dm_bio_prison dm_bufio l
> > > > > > > > > > ibcrc32c dm_service_time dm_multipath intel_rapl_msr
> > > > > > > > > > intel_rapl_common intel_uncore_frequency
> > > > > > > > > > intel_uncore_frequency_common isst_if_common skx_edac nfit
> > > > > > > > > > libnvdimm ipmi_ssif x86_pkg_temp
> > > > > > > > > > _thermal intel_powerclamp coretemp kvm_intel kvm irqbypass
> > > > > > > > > > dcdbas rapl intel_cstate mgag200 i2c_algo_bit drm_shmem_helper
> > > > > > > > > > dell_smbios drm_kms_helper dell_wmi_descriptor wmi_bmof intel_u
> > > > > > > > > > ncore syscopyarea pcspkr sysfillrect mei_me sysimgblt
> > > > > > > > > > acpi_ipmi
> > > > > > > > > > mei fb_sys_fops i2c_i801 ipmi_si intel_pch_thermal lpc_ich
> > > > > > > > > > ipmi_devintf i2c_smbus ipmi_msghandler joydev acpi_power_meter
> > > > > > > > > > nfsd auth_rpcgss nfs_acl drm lockd grace fuse sunrpc ext4
> > > > > > > > > > mbcache jbd2 sd_mod sg lpfc
> > > > > > > > > > [Mon Mar 11 14:12:05 2024] nvmet_fc nvmet nvme_fc
> > > > > > > > > > nvme_fabrics
> > > > > > > > > > crct10dif_pclmul ahci libahci crc32_pclmul nvme_core crc32c_intel
> > > > > > > > > > ixgbe megaraid_sas libata nvme_common ghash_clmulni_int
> > > > > > > > > > el t10_pi wdat_wdt scsi_transport_fc mdio wmi dca dm_mirror
> > > > > > > > > > dm_region_hash dm_log dm_mod
> > > > > > > > > > [Mon Mar 11 14:12:05 2024] CPU: 39 PID: 8844 Comm: nfsd Not
> > > > > > > > > > tainted 5.14.0-419.el9.x86_64 #1
> > > > > > > > > > [Mon Mar 11 14:12:05 2024] Hardware name: Dell Inc. PowerEdge
> > > > > > > > > > R740/00WGD1, BIOS 2.20.1 09/13/2023
> > > > > > > > > > [Mon Mar 11 14:12:05 2024] RIP:
> > > > > > > > > > 0010:nfsd_break_deleg_cb+0x170/0x190 [nfsd]
> > > > > > > > > > [Mon Mar 11 14:12:05 2024] Code: a6 95 c5 f3 e9 ff fe ff
> > > > > > > > > > ff 48
> > > > > > > > > > 89 df be 01 00 00 00 e8 34 b5 13 f4 48 8d bb 98 00 00 00 e8 c8 f9
> > > > > > > > > > 00 00 84 c0 0f 85 2e ff ff ff <0f> 0b e9 27 ff ff ff be
> > > > > > > > > > 02 00 00 00 48 89 df e8 0c b5 13 f4 e9 01
> > > > > > > > > > [Mon Mar 11 14:12:05 2024] RSP: 0018:ffff9929e0bb7b80 EFLAGS:
> > > > > > > > > > 00010246
> > > > > > > > > > [Mon Mar 11 14:12:05 2024] RAX: 0000000000000000 RBX:
> > > > > > > > > > ffff8ada51930900 RCX: 0000000000000024
> > > > > > > > > > [Mon Mar 11 14:12:05 2024] RDX: ffff8ada519309c8 RSI:
> > > > > > > > > > ffff8ad582933c00 RDI: 0000000000002000
> > > > > > > > > > [Mon Mar 11 14:12:05 2024] RBP: ffff8ad46bf21574 R08:
> > > > > > > > > > ffff9929e0bb7b48 R09: 0000000000000000
> > > > > > > > > > [Mon Mar 11 14:12:05 2024] R10: ffff8aec859a2948 R11:
> > > > > > > > > > 0000000000000000 R12: ffff8ad6f497c360
> > > > > > > > > > [Mon Mar 11 14:12:05 2024] R13: ffff8ad46bf21560 R14:
> > > > > > > > > > ffff8ae5942e0b10 R15: ffff8ad6f497c360
> > > > > > > > > > [Mon Mar 11 14:12:05 2024] FS: 0000000000000000(0000)
> > > > > > > > > > GS:ffff8b031fcc0000(0000) knlGS:0000000000000000
> > > > > > > > > > [Mon Mar 11 14:12:05 2024] CS: 0010 DS: 0000 ES: 0000 CR0:
> > > > > > > > > > 0000000080050033
> > > > > > > > > > [Mon Mar 11 14:12:05 2024] CR2: 00007fafe2060744 CR3:
> > > > > > > > > > 00000018e58de006 CR4: 00000000007706e0
> > > > > > > > > > [Mon Mar 11 14:12:05 2024] DR0: 0000000000000000 DR1:
> > > > > > > > > > 0000000000000000 DR2: 0000000000000000
> > > > > > > > > > [Mon Mar 11 14:12:05 2024] DR3: 0000000000000000 DR6:
> > > > > > > > > > 00000000fffe0ff0 DR7: 0000000000000400
> > > > > > > > > > [Mon Mar 11 14:12:05 2024] PKRU: 55555554
> > > > > > > > > > [Mon Mar 11 14:12:05 2024] Call Trace:
> > > > > > > > > > [Mon Mar 11 14:12:05 2024] <TASK>
> > > > > > > > > > [Mon Mar 11 14:12:05 2024] ? show_trace_log_lvl+0x1c4/0x2df
> > > > > > > > > > [Mon Mar 11 14:12:05 2024] ? show_trace_log_lvl+0x1c4/0x2df
> > > > > > > > > > [Mon Mar 11 14:12:05 2024] ? __break_lease+0x16f/0x5f0
> > > > > > > > > > [Mon Mar 11 14:12:05 2024] ? nfsd_break_deleg_cb+0x170/0x190
> > > > > > > > > > [nfsd]
> > > > > > > > > > [Mon Mar 11 14:12:05 2024] ? __warn+0x81/0x110
> > > > > > > > > > [Mon Mar 11 14:12:05 2024] ? nfsd_break_deleg_cb+0x170/0x190
> > > > > > > > > > [nfsd]
> > > > > > > > > > [Mon Mar 11 14:12:05 2024] ? report_bug+0x10a/0x140
> > > > > > > > > > [Mon Mar 11 14:12:05 2024] ? handle_bug+0x3c/0x70
> > > > > > > > > > [Mon Mar 11 14:12:05 2024] ? exc_invalid_op+0x14/0x70
> > > > > > > > > > [Mon Mar 11 14:12:05 2024] ? asm_exc_invalid_op+0x16/0x20
> > > > > > > > > > [Mon Mar 11 14:12:05 2024] ? nfsd_break_deleg_cb+0x170/0x190
> > > > > > > > > > [nfsd]
> > > > > > > > > > [Mon Mar 11 14:12:05 2024] __break_lease+0x16f/0x5f0
> > > > > > > > > > [Mon Mar 11 14:12:05 2024] ?
> > > > > > > > > > nfsd_file_lookup_locked+0x117/0x160 [nfsd]
> > > > > > > > > > [Mon Mar 11 14:12:05 2024] ? list_lru_del+0x101/0x150
> > > > > > > > > > [Mon Mar 11 14:12:05 2024] nfsd_file_do_acquire+0x790/0x830
> > > > > > > > > > [nfsd]
> > > > > > > > > > [Mon Mar 11 14:12:05 2024] nfs4_get_vfs_file+0x315/0x3a0
> > > > > > > > > > [nfsd]
> > > > > > > > > > [Mon Mar 11 14:12:05 2024]
> > > > > > > > > > nfsd4_process_open2+0x430/0xa30 [nfsd]
> > > > > > > > > > [Mon Mar 11 14:12:05 2024] ? fh_verify+0x297/0x2f0 [nfsd]
> > > > > > > > > > [Mon Mar 11 14:12:05 2024] nfsd4_open+0x3ce/0x4b0 [nfsd]
> > > > > > > > > > [Mon Mar 11 14:12:05 2024]
> > > > > > > > > > nfsd4_proc_compound+0x44b/0x700 [nfsd]
> > > > > > > > > > [Mon Mar 11 14:12:05 2024] nfsd_dispatch+0x94/0x1c0 [nfsd]
> > > > > > > > > > [Mon Mar 11 14:12:05 2024] svc_process_common+0x2ec/0x660
> > > > > > > > > > [sunrpc]
> > > > > > > > > > [Mon Mar 11 14:12:05 2024] ?
> > > > > > > > > > __pfx_nfsd_dispatch+0x10/0x10 [nfsd]
> > > > > > > > > > [Mon Mar 11 14:12:05 2024] ? __pfx_nfsd+0x10/0x10 [nfsd]
> > > > > > > > > > [Mon Mar 11 14:12:05 2024] svc_process+0x12d/0x170 [sunrpc]
> > > > > > > > > > [Mon Mar 11 14:12:05 2024] nfsd+0x84/0xb0 [nfsd]
> > > > > > > > > > [Mon Mar 11 14:12:05 2024] kthread+0xdd/0x100
> > > > > > > > > > [Mon Mar 11 14:12:05 2024] ? __pfx_kthread+0x10/0x10
> > > > > > > > > > [Mon Mar 11 14:12:05 2024] ret_from_fork+0x29/0x50
> > > > > > > > > > [Mon Mar 11 14:12:05 2024] </TASK>
> > > > > > > > > > [Mon Mar 11 14:12:05 2024] ---[ end trace 7a039e17443dc651
> > > > > > > > > > ]---
> > > > > > > > > This is probably this WARN in nfsd_break_one_deleg:
> > > > > > > > >
> > > > > > > > > WARN_ON_ONCE(!nfsd4_run_cb(&dp->dl_recall));
> > > > > > > > >
> > > > > > > > > It means that a delegation break callback to the client
> > > > > > > > > couldn't be
> > > > > > > > > queued to the workqueue, and so it didn't run.
> > > > > > > > >
> > > > > > > > > > Could this be the same issue as described
> > > > > > > > > > here:https://urldefense.com/v3/__https://lore.kernel.org/linux-nfs/af0ec881-5ebf-4feb-98ae-3ed2a77f86f1@oracle.com/__;!!ACWV5N9M2RV99hQ!LV3yWeoSOhNAkRHkxFCH2tlm0iNFVD78mxnSLyP6lrX7yBVeA2TOJ4nv6oZsqLwP4kW56CMpDWhkjjwSkdBV9En7$
> > > > > > > > > > ?
> > > > > > > > > Yes, most likely the same problem.
> > > > > > > > If I read that thread correctly, this issue was introduced between
> > > > > > > > 6.1.63 and 6.6.3? Is it possible the EL9 5.14.0-362.18.1.el9_3
> > > > > > > > backported these changes, or were we hitting some other bug with
> > > > > > > > that
> > > > > > > > version? It seems the 6.1.x kernel is not affected? If so, that
> > > > > > > > would be
> > > > > > > > the recommended kernel to run?
> > > > > > > Anything is possible. We have to identify the problem first.
> > > > > > > > > > As described in that thread, I've tried to obtain the requested
> > > > > > > > > > information.
> > > > > > > > > >
> > > > > > > > > > Is it possible this is the issue that was fixed by the patches
> > > > > > > > > > described
> > > > > > > > > > here?https://urldefense.com/v3/__https://lore.kernel.org/linux-nfs/2024022054-cause-suffering-eae8@gregkh/__;!!ACWV5N9M2RV99hQ!LV3yWeoSOhNAkRHkxFCH2tlm0iNFVD78mxnSLyP6lrX7yBVeA2TOJ4nv6oZsqLwP4kW56CMpDWhkjjwSkedtUP09$
> > > > > > > > > >
> > > > > > > > > Doubtful. Those are targeted toward a different set of issues.
> > > > > > > > >
> > > > > > > > > If you're willing, I do have some patches queued up for CentOS
> > > > > > > > > here
> > > > > > > > > that
> > > > > > > > > fix some backchannel problems that could be related. I'm mainly
> > > > > > > > > waiting
> > > > > > > > > on Chuck to send these to Linus and then we'll likely merge
> > > > > > > > > them into
> > > > > > > > > CentOS soon afterward:
> > > > > > > > >
> > > > > > > > > https://urldefense.com/v3/__https://gitlab.com/redhat/centos-stream/src/kernel/centos-stream-9/-/merge_requests/3689__;!!ACWV5N9M2RV99hQ!LV3yWeoSOhNAkRHkxFCH2tlm0iNFVD78mxnSLyP6lrX7yBVeA2TOJ4nv6oZsqLwP4kW56CMpDWhkjjwSkdvDn8y7$
> > > > > > > > >
> > > > > > > > >
> > > > > > > > If you can send me a patch file, I can rebuild the C9S kernel
> > > > > > > > with that
> > > > > > > > patch and run it. It can take a while for the bug to trigger as I
> > > > > > > > believe it seems to be very workload dependent (we were running
> > > > > > > > very
> > > > > > > > stable for months and now hit this bug every other week).
> > > > > > > >
> > > > > > > >
> > > > > > > It's probably simpler to just pull down the build artifacts for
> > > > > > > that MR.
> > > > > > > You have to drill down through the CI for it, but they are here:
> > > > > > >
> > > > > > > https://urldefense.com/v3/__https://s3.amazonaws.com/arr-cki-prod-trusted-artifacts/index.html?prefix=trusted-artifacts*1194300175*publish_x86_64*6278921877*artifacts*__;Ly8vLy8!!ACWV5N9M2RV99hQ!LV3yWeoSOhNAkRHkxFCH2tlm0iNFVD78mxnSLyP6lrX7yBVeA2TOJ4nv6oZsqLwP4kW56CMpDWhkjjwSkaP5eW8V$
> > > > > > >
> > > > > > >
> > > > > > > There's even a repo file you can install on the box to pull them
> > > > > > > down.
> > > > > > We installed this kernel on the server 3 days ago. Today, a user
> > > > > > informed us that their screen was black after logging in. Similar to
> > > > > > other occurrences of this issue, the mount command on the client was
> > > > > > hung. But in contrast to the other times, there were no messages in
> > > > > > the logs kernel logs on the server. Even restarting the client does
> > > > > > not resolve the issue.
> > > >
> > > > Ok, so you rebooted the client and it's still unable to mount? That
> > > > sounds like a server problem if so.
> > > >
> > > > Are both client and server running the same kernel?
> > > No, the server runs 5.14.0-427.3689_1194299994.el9 and the client
> > > 5.14.0-362.18.1.el9_3.
> > > >
> > > > > > Something still seems to be wrong on the server though. When I
> > > > > > look at
> > > > > > the directories under /proc/fs/nfsd/clients, there's still a
> > > > > > directory
> > > > > > for the specific client, even though it's no longer running:
> > > > > >
> > > > > > # cat 155/info
> > > > > > clientid: 0xc8edb7f65f4a9ad
> > > > > > address: "10.87.31.152:819"
> > > > > > status: confirmed
> > > > > > seconds from last renew: 33163
> > > > > > name: "Linux NFSv4.2 bersalis.esat.kuleuven.be"
> > > > > > minor version: 2
> > > > > > Implementation domain: "kernel.org"
> > > > > > Implementation name: "Linux 5.14.0-362.18.1.el9_3.0.1.x86_64 #1 SMP
> > > > > > PREEMPT_DYNAMIC Sun Feb 11 13:49:23 UTC 2024 x86_64"
> > > > > > Implementation time: [0, 0]
> > > > > > callback state: DOWN
> > > > > > callback address: 10.87.31.152:0
> > > > > >
> > > > If you just shut down the client, the server won't immediately purge
> > > > its
> > > > record. In fact, assuming you're running the same kernel on the server,
> > > > it won't purge the client record until there is a conflicting request
> > > > for its state.
> > > Is there a way to force such a conflicting request (to get the client
> > > record to purge)?
> >
> > Try:
> >
> > # echo "expire" > /proc/fs/nfsd/clients/155/ctl
>
> I've tried that. The command hangs and can not be interrupted with ctrl-c.
>
I'd wager that's the wait_event() in force_expire_client. It seems like
that sleep should be killable.
> I've now also noticed in the dmesg output that the kernel issued the
> following WARNING a few hours ago. It wasn't directly triggered by the
> echo command above, but seems to have been triggered a few hours ago
> (probably when another client started to have the same problem as more
> clients are experiencing issues now).
>
> [Tue Mar 19 14:53:44 2024] ------------[ cut here ]------------
> [Tue Mar 19 14:53:44 2024] WARNING: CPU: 44 PID: 5843 at
> fs/nfsd/nfs4state.c:4920 nfsd_break_deleg_cb+0x170/0x190 [nfsd]
> [Tue Mar 19 14:53:44 2024] Modules linked in: nf_conntrack_netlink nfsv4
> dns_resolver nfs fscache netfs binfmt_misc xsk_diag rpcsec_gss_krb5
> rpcrdma rdma_cm iw_cm ib_cm ib_core bonding tls rfkill nft_counter
> nft_ct nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_reject_inet
> nf_reject_ipv4 nf_reject_ipv6 nft_reject nf_tables nfnetlink vfat fat
> dm_thin_pool dm_persistent_data dm_bio_prison dm_bufio libcrc32c
> dm_service_time dm_multipath intel_rapl_msr intel_rapl_common
> intel_uncore_frequency intel_uncore_frequency_common isst_if_common
> skx_edac nfit libnvdimm x86_pkg_temp_thermal intel_powerclamp coretemp
> kvm_intel kvm dcdbas irqbypass ipmi_ssif rapl intel_cstate mgag200
> i2c_algo_bit drm_shmem_helper drm_kms_helper dell_smbios syscopyarea
> intel_uncore sysfillrect wmi_bmof dell_wmi_descriptor pcspkr sysimgblt
> fb_sys_fops mei_me i2c_i801 mei intel_pch_thermal acpi_ipmi i2c_smbus
> lpc_ich ipmi_si ipmi_devintf ipmi_msghandler joydev acpi_power_meter
> nfsd nfs_acl lockd auth_rpcgss grace drm fuse sunrpc ext4
> [Tue Mar 19 14:53:44 2024] mbcache jbd2 sd_mod sg lpfc nvmet_fc nvmet
> nvme_fc nvme_fabrics crct10dif_pclmul crc32_pclmul nvme_core ixgbe
> crc32c_intel ahci libahci nvme_common megaraid_sas t10_pi
> ghash_clmulni_intel wdat_wdt libata scsi_transport_fc mdio dca wmi
> dm_mirror dm_region_hash dm_log dm_mod
> [Tue Mar 19 14:53:44 2024] CPU: 44 PID: 5843 Comm: nfsd Not tainted
> 5.14.0-427.3689_1194299994.el9.x86_64 #1
> [Tue Mar 19 14:53:44 2024] Hardware name: Dell Inc. PowerEdge
> R740/00WGD1, BIOS 2.20.1 09/13/2023
> [Tue Mar 19 14:53:44 2024] RIP: 0010:nfsd_break_deleg_cb+0x170/0x190 [nfsd]
> [Tue Mar 19 14:53:44 2024] Code: 76 76 cd de e9 ff fe ff ff 48 89 df be
> 01 00 00 00 e8 34 a1 1b df 48 8d bb 98 00 00 00 e8 a8 fe 00 00 84 c0 0f
> 85 2e ff ff ff <0f> 0b e9 27 ff ff ff be 02 00 00 00 48 89 df e8 0c a1
> 1b df e9 01
> [Tue Mar 19 14:53:44 2024] RSP: 0018:ffffb2878f2cfc38 EFLAGS: 00010246
> [Tue Mar 19 14:53:44 2024] RAX: 0000000000000000 RBX: ffff88d5171067b8
> RCX: 0000000000000000
> [Tue Mar 19 14:53:44 2024] RDX: ffff88d517106880 RSI: ffff88bdceec8600
> RDI: 0000000000002000
> [Tue Mar 19 14:53:44 2024] RBP: ffff88d68a38a284 R08: ffffb2878f2cfc00
> R09: 0000000000000000
> [Tue Mar 19 14:53:44 2024] R10: ffff88bf57dd7878 R11: 0000000000000000
> R12: ffff88d5b79c4798
> [Tue Mar 19 14:53:44 2024] R13: ffff88d68a38a270 R14: ffff88cab06ad0c8
> R15: ffff88d5b79c4798
> [Tue Mar 19 14:53:44 2024] FS: 0000000000000000(0000)
> GS:ffff88d4a1180000(0000) knlGS:0000000000000000
> [Tue Mar 19 14:53:44 2024] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> [Tue Mar 19 14:53:44 2024] CR2: 00007fe46ef90000 CR3: 000000019d010004
> CR4: 00000000007706e0
> [Tue Mar 19 14:53:44 2024] DR0: 0000000000000000 DR1: 0000000000000000
> DR2: 0000000000000000
> [Tue Mar 19 14:53:44 2024] DR3: 0000000000000000 DR6: 00000000fffe0ff0
> DR7: 0000000000000400
> [Tue Mar 19 14:53:44 2024] PKRU: 55555554
> [Tue Mar 19 14:53:44 2024] Call Trace:
> [Tue Mar 19 14:53:44 2024] <TASK>
> [Tue Mar 19 14:53:44 2024] ? show_trace_log_lvl+0x1c4/0x2df
> [Tue Mar 19 14:53:44 2024] ? show_trace_log_lvl+0x1c4/0x2df
> [Tue Mar 19 14:53:44 2024] ? __break_lease+0x16f/0x5f0
> [Tue Mar 19 14:53:44 2024] ? nfsd_break_deleg_cb+0x170/0x190 [nfsd]
> [Tue Mar 19 14:53:44 2024] ? __warn+0x81/0x110
> [Tue Mar 19 14:53:44 2024] ? nfsd_break_deleg_cb+0x170/0x190 [nfsd]
> [Tue Mar 19 14:53:44 2024] ? report_bug+0x10a/0x140
> [Tue Mar 19 14:53:44 2024] ? handle_bug+0x3c/0x70
> [Tue Mar 19 14:53:44 2024] ? exc_invalid_op+0x14/0x70
> [Tue Mar 19 14:53:44 2024] ? asm_exc_invalid_op+0x16/0x20
> [Tue Mar 19 14:53:44 2024] ? nfsd_break_deleg_cb+0x170/0x190 [nfsd]
> [Tue Mar 19 14:53:44 2024] ? nfsd_break_deleg_cb+0x96/0x190 [nfsd]
> [Tue Mar 19 14:53:44 2024] __break_lease+0x16f/0x5f0
> [Tue Mar 19 14:53:44 2024] nfs4_get_vfs_file+0x164/0x3a0 [nfsd]
> [Tue Mar 19 14:53:44 2024] nfsd4_process_open2+0x430/0xa30 [nfsd]
> [Tue Mar 19 14:53:44 2024] ? fh_verify+0x297/0x2f0 [nfsd]
> [Tue Mar 19 14:53:44 2024] nfsd4_open+0x3ce/0x4b0 [nfsd]
> [Tue Mar 19 14:53:44 2024] nfsd4_proc_compound+0x44b/0x700 [nfsd]
> [Tue Mar 19 14:53:44 2024] nfsd_dispatch+0x94/0x1c0 [nfsd]
> [Tue Mar 19 14:53:44 2024] svc_process_common+0x2ec/0x660 [sunrpc]
> [Tue Mar 19 14:53:44 2024] ? __pfx_nfsd_dispatch+0x10/0x10 [nfsd]
> [Tue Mar 19 14:53:44 2024] ? __pfx_nfsd+0x10/0x10 [nfsd]
> [Tue Mar 19 14:53:44 2024] svc_process+0x12d/0x170 [sunrpc]
> [Tue Mar 19 14:53:44 2024] nfsd+0x84/0xb0 [nfsd]
> [Tue Mar 19 14:53:44 2024] kthread+0xdd/0x100
> [Tue Mar 19 14:53:44 2024] ? __pfx_kthread+0x10/0x10
> [Tue Mar 19 14:53:44 2024] ret_from_fork+0x29/0x50
> [Tue Mar 19 14:53:44 2024] </TASK>
> [Tue Mar 19 14:53:44 2024] ---[ end trace ed0b2b3f135c637d ]---
>
> It again seems to have been triggered in nfsd_break_deleg_cb?
>
Same problem as before. It tried to submit the workqueue job, but it was
already queued, so the submission failed.
> I also had the following perf command running a tmux on the server:
>
> perf trace -e nfsd:nfsd_cb_recall_any
>
> This has spewed a lot of messages. I'm including a short list here:
>
> ...
>
> 33464866.721 kworker/u98:5/1591466 nfsd:nfsd_cb_recall_any(cl_boot:
> 1710533037, cl_id: 210688785, bmval0: 1, addr: 0x7f331bb116c8)
> 33464866.724 kworker/u98:5/1591466 nfsd:nfsd_cb_recall_any(cl_boot:
> 1710533037, cl_id: 210688827, bmval0: 1, addr: 0x7f331bb11738)
> 33464866.729 kworker/u98:5/1591466 nfsd:nfsd_cb_recall_any(cl_boot:
> 1710533037, cl_id: 210688767, bmval0: 1, addr: 0x7f331bb117a8)
> 33464866.732 kworker/u98:5/1591466 nfsd:nfsd_cb_recall_any(cl_boot:
> 1710533037, cl_id: 210718132, bmval0: 1, addr: 0x7f331bb11818)
> 33464866.737 kworker/u98:5/1591466 nfsd:nfsd_cb_recall_any(cl_boot:
> 1710533037, cl_id: 210688952, bmval0: 1, addr: 0x7f331bb11888)
> 33464866.741 kworker/u98:5/1591466 nfsd:nfsd_cb_recall_any(cl_boot:
> 1710533037, cl_id: 210702355, bmval0: 1, addr: 0x7f331bb118f8)
> 33868414.001 kthreadd/1597068 nfsd:nfsd_cb_recall_any(cl_boot:
> 1710533037, cl_id: 210688751, bmval0: 1, addr: 0x7f331be68620)
> 33868414.014 kthreadd/1597068 nfsd:nfsd_cb_recall_any(cl_boot:
> 1710533037, cl_id: 210718536, bmval0: 1, addr: 0x7f331be68690)
> 33868414.018 kthreadd/1597068 nfsd:nfsd_cb_recall_any(cl_boot:
> 1710533037, cl_id: 210719074, bmval0: 1, addr: 0x7f331be68700)
> 33868414.022 kthreadd/1597068 nfsd:nfsd_cb_recall_any(cl_boot:
> 1710533037, cl_id: 210688916, bmval0: 1, addr: 0x7f331be68770)
> 33868414.026 kthreadd/1597068 nfsd:nfsd_cb_recall_any(cl_boot:
> 1710533037, cl_id: 210688941, bmval0: 1, addr: 0x7f331be687e0)
> ...
>
> 33868414.924 kthreadd/1597068 nfsd:nfsd_cb_recall_any(cl_boot:
> 1710533037, cl_id: 210688744, bmval0: 1, addr: 0x7f331be6d7f0)
> 33868414.929 kthreadd/1597068 nfsd:nfsd_cb_recall_any(cl_boot:
> 1710533037, cl_id: 210717223, bmval0: 1, addr: 0x7f331be6d860)
> 33868414.934 kthreadd/1597068 nfsd:nfsd_cb_recall_any(cl_boot:
> 1710533037, cl_id: 210716137, bmval0: 1, addr: 0x7f331be6d8d0)
> 34021240.903 kworker/u98:5/1591466 nfsd:nfsd_cb_recall_any(cl_boot:
> 1710533037, cl_id: 210688941, bmval0: 1, addr: 0x7f331c207de8)
> 34021240.917 kworker/u98:5/1591466 nfsd:nfsd_cb_recall_any(cl_boot:
> 1710533037, cl_id: 210718750, bmval0: 1, addr: 0x7f331c207e58)
> 34021240.922 kworker/u98:5/1591466 nfsd:nfsd_cb_recall_any(cl_boot:
> 1710533037, cl_id: 210688955, bmval0: 1, addr: 0x7f331c207ec8)
> 34021240.925 kworker/u98:5/1591466 nfsd:nfsd_cb_recall_any(cl_boot:
> 1710533037, cl_id: 210688975, bmval0: 1, addr: 0x7f331c207f38)
> ...
>
> I assume the cl_id is the client id? How can I map this to a client from
> /proc/fs/nfsd/clients?
>
> If I understand it correctly, the recall_any should be called when
> either the system starts to experience memory pressure, or it reaches
> the delegation limits? I doubt the system is actually running out of
> memory here as there are no other indications. Shouldn't I get those
> "page allocation failure" messages if it does? How can I check the
> number of delegations/leases currently issued, what the current maximum
> is and how to increase it?
You probably log messages or anything for that. recall_any is hooked up
to a shrinker AFAIU, so we start sending those when the VM politely asks
us to release memory, not when it's an emergency.
The leases are (usually) shown in /proc/locks for most local
filesystems.
>
> Regarding the recall any call: from what I've read on kernelnewbies,
> this feature was introduced in the 6.2 kernel? When I look at the tree
> for 6.1.x, it was backported in 6.1.81? Is there a way to disable this
> support somehow?
>
Not currently. You can disable kernel leases altogether which will
disable delegations. You may want to start there if you're having
stability issues with them enabled.
That said, with a problem like this, it's easy to get lost in peripheral
details. I'm not clear on root cause of this problem yet.
You have a client that is unable to reestablish a session with the
server because the server is returning NFS4ERR_DELAY repeatedly. If
you're able, my suggestion would be to start by trying to determine the
cause of that problem, rather than guessing about different patches, or
turning off server functionality.
Note too that the kernel you're running on the server is a build
artifact from a merge request. Once you're ready to reboot, you may want
to update to the latest centos9 stream kernel, since those patches have
now been merged into it (along with some other NFS fixes).
> > > > > The nfsdclnts command for this client shows the following delegations:
> > > > >
> > > > > # nfsdclnts -f 155/states -t all
> > > > > Inode number | Type | Access | Deny | ip address | Filename
> > > > > 169346743 | open | r- | -- | 10.87.31.152:819 |
> > > > > disconnected dentry
> > > > > 169346743 | deleg | r | | 10.87.31.152:819 |
> > > > > disconnected dentry
> > > > > 169346746 | open | r- | -- | 10.87.31.152:819 |
> > > > > disconnected dentry
> > > > > 169346746 | deleg | r | | 10.87.31.152:819 |
> > > > > disconnected dentry
> > > > >
> > > > > I see a lot of recent patches regarding directory delegations. Could
> > > > > this be related to this?
> > > > >
> > > > > Will a 5.14.0-362.18.1.el9_3.0.1 kernel try to use a directory
> > > > > delegation?
> > > > >
> > > > >
> > > > No. Directory delegations are a new feature that's still under
> > > > development. They use some of the same machinery as file delegations,
> > > > but they wouldn't be a factor here.
> > > >
> > > > > > The system seems to have identified that the client is no longer
> > > > > > reachable, but the client entry does not go away. When a mount was
> > > > > > hanging on the client, there would be two directories in clients for
> > > > > > the same client. Killing the mount command clears up the second
> > > > > > entry.
> > > > > >
> > > > > > Even after running conntrack -D on the server to remove the tcp
> > > > > > connection from the conntrack table, the entry doesn't go away and
> > > > > > the
> > > > > > client still can not mount anything from the server.
> > > > > >
> > > > > > A tcpdump on the client while a mount was running logged the
> > > > > > following
> > > > > > messages over and over again:
> > > > > >
> > > > > > request:
> > > > > >
> > > > > > Frame 1: 378 bytes on wire (3024 bits), 378 bytes captured (3024
> > > > > > bits)
> > > > > > Ethernet II, Src: HP_19:7d:4b (e0:73:e7:19:7d:4b), Dst:
> > > > > > ArubaaHe_f9:8e:00 (88:3a:30:f9:8e:00)
> > > > > > Internet Protocol Version 4, Src: 10.87.31.152, Dst: 10.86.18.14
> > > > > > Transmission Control Protocol, Src Port: 932, Dst Port: 2049, Seq: 1,
> > > > > > Ack: 1, Len: 312
> > > > > > Remote Procedure Call, Type:Call XID:0x1d3220c4
> > > > > > Network File System
> > > > > > [Program Version: 4]
> > > > > > [V4 Procedure: COMPOUND (1)]
> > > > > > GSS Data, Ops(1): CREATE_SESSION
> > > > > > Length: 152
> > > > > > GSS Sequence Number: 76
> > > > > > Tag: <EMPTY>
> > > > > > minorversion: 2
> > > > > > Operations (count: 1): CREATE_SESSION
> > > > > > [Main Opcode: CREATE_SESSION (43)]
> > > > > > GSS Checksum:
> > > > > > 00000028040404ffffffffff000000002c19055f1f8d442d594c13849628affc2797cbb2…
> > > > > >
> > > > > > GSS Token Length: 40
> > > > > > GSS-API Generic Security Service Application Program
> > > > > > Interface
> > > > > > krb5_blob:
> > > > > > 040404ffffffffff000000002c19055f1f8d442d594c13849628affc2797cbb23fa080b0…
> > > > > >
> > > > > >
> > > > > > response:
> > > > > >
> > > > > > Frame 2: 206 bytes on wire (1648 bits), 206 bytes captured (1648
> > > > > > bits)
> > > > > > Ethernet II, Src: ArubaaHe_f9:8e:00 (88:3a:30:f9:8e:00), Dst:
> > > > > > HP_19:7d:4b (e0:73:e7:19:7d:4b)
> > > > > > Internet Protocol Version 4, Src: 10.86.18.14, Dst: 10.87.31.152
> > > > > > Transmission Control Protocol, Src Port: 2049, Dst Port: 932, Seq: 1,
> > > > > > Ack: 313, Len: 140
> > > > > > Remote Procedure Call, Type:Reply XID:0x1d3220c4
> > > > > > Network File System
> > > > > > [Program Version: 4]
> > > > > > [V4 Procedure: COMPOUND (1)]
> > > > > > GSS Data, Ops(1): CREATE_SESSION(NFS4ERR_DELAY)
> > > > > > Length: 24
> > > > > > GSS Sequence Number: 76
> > > > > > Status: NFS4ERR_DELAY (10008)
> > > > > > Tag: <EMPTY>
> > > > > > Operations (count: 1)
> > > > > > [Main Opcode: CREATE_SESSION (43)]
> > > > > > GSS Checksum:
> > > > > > 00000028040405ffffffffff000000000aa742d0798deaad1a8aa2d7c3a91bf4f6274222…
> > > > > >
> > > > > > GSS Token Length: 40
> > > > > > GSS-API Generic Security Service Application Program
> > > > > > Interface
> > > > > > krb5_blob:
> > > > > > 040405ffffffffff000000000aa742d0798deaad1a8aa2d7c3a91bf4f627422226d74923…
> > > > > >
> > > > > >
> > > > > > I was hoping that giving the client a different IP address would
> > > > > > resolve the issue for this client, but it didn't. Even though the
> > > > > > client had a new IP address (hostname was kept the same), it
> > > > > > failed to
> > > > > > mount anything from the server.
> > > > > >
> > > > Changing the IP address won't help. The client is probably using the
> > > > same long-form client id as before, so the server still identifies the
> > > > client even with the address change.
> > > How is the client id determined? Will changing the hostname of the
> > > client trigger a change of the client id?
> > > >
> > > > Unfortunately, the cause of an NFS4ERR_DELAY error is tough to guess.
> > > > The client is expected to back off and retry, so if the server keeps
> > > > returning that repeatedly, then a hung mount command is expected.
> > > >
> > > > The question is why the server would keep returning DELAY. A lot of
> > > > different problems ranging from memory allocation issues to protocol
> > > > problems can result in that error. You may want to check the NFS server
> > > > and see if anything was logged there.
> > > There are no messages in the system logs that indicate any sort of
> > > memory issue. We also increased the min_kbytes_free sysctl to 2G on
> > > the server before we restarted it with the newer kernel.
> > > >
> > > > This is on a CREATE_SESSION call, so I wonder if the record held by the
> > > > (courteous) server is somehow blocking the attempt to reestablish the
> > > > session?
> > > >
> > > > Do you have a way to reproduce this? Since this is a centos kernel, you
> > > > could follow the page here to open a bug:
> > >
> > > Unfortunately we haven't found a reliable way to reproduce it. But we
> > > do seem to trigger it more and more lately.
> > >
> > > Regards,
> > >
> > > Rik
> > >
> > > >
> > > > https://urldefense.com/v3/__https://wiki.centos.org/ReportBugs.html__;!!ACWV5N9M2RV99hQ!LV3yWeoSOhNAkRHkxFCH2tlm0iNFVD78mxnSLyP6lrX7yBVeA2TOJ4nv6oZsqLwP4kW56CMpDWhkjjwSkWIqsboq$
> > > >
> > > >
> > > > > > I created another dump of the workqueues and worker pools on the
> > > > > > server:
> > > > > >
> > > > > > [Mon Mar 18 14:59:33 2024] Showing busy workqueues and worker pools:
> > > > > > [Mon Mar 18 14:59:33 2024] workqueue events: flags=0x0
> > > > > > [Mon Mar 18 14:59:33 2024] pwq 54: cpus=27 node=1 flags=0x0 nice=0
> > > > > > active=1/256 refcnt=2
> > > > > > [Mon Mar 18 14:59:33 2024] pending: drm_fb_helper_damage_work
> > > > > > [drm_kms_helper]
> > > > > > [Mon Mar 18 14:59:33 2024] workqueue events_power_efficient:
> > > > > > flags=0x80
> > > > > > [Mon Mar 18 14:59:33 2024] pwq 54: cpus=27 node=1 flags=0x0 nice=0
> > > > > > active=1/256 refcnt=2
> > > > > > [Mon Mar 18 14:59:33 2024] pending: fb_flashcursor
> > > > > > [Mon Mar 18 14:59:33 2024] workqueue mm_percpu_wq: flags=0x8
> > > > > > [Mon Mar 18 14:59:33 2024] pwq 54: cpus=27 node=1 flags=0x0 nice=0
> > > > > > active=1/256 refcnt=3
> > > > > > [Mon Mar 18 14:59:33 2024] pending: lru_add_drain_per_cpu
> > > > > > BAR(362)
> > > > > > [Mon Mar 18 14:59:33 2024] workqueue kblockd: flags=0x18
> > > > > > [Mon Mar 18 14:59:33 2024] pwq 55: cpus=27 node=1 flags=0x0
> > > > > > nice=-20
> > > > > > active=1/256 refcnt=2
> > > > > > [Mon Mar 18 14:59:33 2024] pending: blk_mq_timeout_work
> > > > > >
> > > > > >
> > > > > > In contrast to last time, it doesn't show anything regarding nfs this
> > > > > > time.
> > > > > >
> > > > > > I also tried the suggestion from Dai Ngo (echo 3 >
> > > > > > /proc/sys/vm/drop_caches), but that didn't seem to make any
> > > > > > difference.
> > > > > >
> > > > > > We haven't restarted the server yet as it seems the impact seems to
> > > > > > affect fewer clients that before. Is there anything we can run on the
> > > > > > server to further debug this?
> > > > > >
> > > > > > In the past, the issue seemed to deteriorate rapidly and resulted in
> > > > > > issues for almost all clients after about 20 minutes. This time the
> > > > > > impact seems to be less, but it's not gone.
> > > > > >
> > > > > > How can we force the NFS server to forget about a specific client? I
> > > > > > haven't tried to restart the nfs service yet as I'm afraid it will
> > > > > > fail to stop as before.
> > > > > >
> > > > Not with that kernel. There are some new administrative interfaces that
> > > > might allow that in the future, but they were just merged upstream and
> > > > aren't in that kernel.
> > > >
> > > > --
> > > > Jeff Layton <jlayton@kernel.org>
> > >
--
Jeff Layton <jlayton@kernel.org>
^ permalink raw reply [flat|nested] 26+ messages in thread
* Re: nfsd hangs and nfsd_break_deleg_cb+0x170/0x190 warning
2024-03-19 19:41 ` Rik Theys
2024-03-19 20:55 ` Jeff Layton
@ 2024-03-19 21:42 ` Dai Ngo
2024-03-20 19:41 ` Rik Theys
1 sibling, 1 reply; 26+ messages in thread
From: Dai Ngo @ 2024-03-19 21:42 UTC (permalink / raw)
To: Rik Theys, Jeff Layton, Linux Nfs
On 3/19/24 12:41 PM, Rik Theys wrote:
> Hi,
>
> On 3/19/24 18:09, Dai Ngo wrote:
>>
>> On 3/19/24 12:58 AM, Rik Theys wrote:
>>> Hi,
>>>
>>> On 3/18/24 22:54, Jeff Layton wrote:
>>>> On Mon, 2024-03-18 at 22:15 +0100, Rik Theys wrote:
>>>>> Hi,
>>>>>
>>>>> On 3/18/24 21:21, Rik Theys wrote:
>>>>>> Hi Jeff,
>>>>>>
>>>>>> On 3/12/24 13:47, Jeff Layton wrote:
>>>>>>> On Tue, 2024-03-12 at 13:24 +0100, Rik Theys wrote:
>>>>>>>> Hi Jeff,
>>>>>>>>
>>>>>>>> On 3/12/24 12:22, Jeff Layton wrote:
>>>>>>>>> On Mon, 2024-03-11 at 19:43 +0100, Rik Theys wrote:
>>>>>>>>>> Since a few weeks our Rocky Linux 9 NFS server has periodically
>>>>>>>>>> logged hung nfsd tasks. The initial effect was that some clients
>>>>>>>>>> could no longer access the NFS server. This got worse and worse
>>>>>>>>>> (probably as more nfsd threads got blocked) and we had to
>>>>>>>>>> restart
>>>>>>>>>> the server. Restarting the server also failed as the NFS server
>>>>>>>>>> service could no longer be stopped.
>>>>>>>>>>
>>>>>>>>>> The initial kernel we noticed this behavior on was
>>>>>>>>>> kernel-5.14.0-362.18.1.el9_3.x86_64. Since then we've installed
>>>>>>>>>> kernel-5.14.0-419.el9.x86_64 from CentOS Stream 9. The same
>>>>>>>>>> issue
>>>>>>>>>> happened again on this newer kernel version:
>>>>>>> 419 is fairly up to date with nfsd changes. There are some known
>>>>>>> bugs
>>>>>>> around callbacks, and there is a draft MR in flight to fix it.
>>>>>>>
>>>>>>> What kernel were you on prior to 5.14.0-362.18.1.el9_3.x86_64 ?
>>>>>>> If we
>>>>>>> can bracket the changes around a particular version, then that
>>>>>>> might
>>>>>>> help identify the problem.
>>>>>>>
>>>>>>>>>> [Mon Mar 11 14:10:08 2024] Not tainted
>>>>>>>>>> 5.14.0-419.el9.x86_64 #1
>>>>>>>>>> [Mon Mar 11 14:10:08 2024] "echo 0 >
>>>>>>>>>> /proc/sys/kernel/hung_task_timeout_secs" disables this message.
>>>>>>>>>> [Mon Mar 11 14:10:08 2024]task:nfsd state:D
>>>>>>>>>> stack:0
>>>>>>>>>> pid:8865 ppid:2 flags:0x00004000
>>>>>>>>>> [Mon Mar 11 14:10:08 2024] Call Trace:
>>>>>>>>>> [Mon Mar 11 14:10:08 2024] <TASK>
>>>>>>>>>> [Mon Mar 11 14:10:08 2024] __schedule+0x21b/0x550
>>>>>>>>>> [Mon Mar 11 14:10:08 2024] schedule+0x2d/0x70
>>>>>>>>>> [Mon Mar 11 14:10:08 2024] schedule_timeout+0x11f/0x160
>>>>>>>>>> [Mon Mar 11 14:10:08 2024] ? select_idle_sibling+0x28/0x430
>>>>>>>>>> [Mon Mar 11 14:10:08 2024] ? wake_affine+0x62/0x1f0
>>>>>>>>>> [Mon Mar 11 14:10:08 2024] __wait_for_common+0x90/0x1d0
>>>>>>>>>> [Mon Mar 11 14:10:08 2024] ?
>>>>>>>>>> __pfx_schedule_timeout+0x10/0x10
>>>>>>>>>> [Mon Mar 11 14:10:08 2024] __flush_workqueue+0x13a/0x3f0
>>>>>>>>>> [Mon Mar 11 14:10:08 2024]
>>>>>>>>>> nfsd4_shutdown_callback+0x49/0x120
>>>>>>>>>> [nfsd]
>>>>>>>>>> [Mon Mar 11 14:10:08 2024] ? nfsd4_cld_remove+0x54/0x1d0
>>>>>>>>>> [nfsd]
>>>>>>>>>> [Mon Mar 11 14:10:08 2024] ?
>>>>>>>>>> nfsd4_return_all_client_layouts+0xc4/0xf0 [nfsd]
>>>>>>>>>> [Mon Mar 11 14:10:08 2024] ?
>>>>>>>>>> nfsd4_shutdown_copy+0x68/0xc0 [nfsd]
>>>>>>>>>> [Mon Mar 11 14:10:08 2024] __destroy_client+0x1f3/0x290
>>>>>>>>>> [nfsd]
>>>>>>>>>> [Mon Mar 11 14:10:08 2024] nfsd4_exchange_id+0x75f/0x770
>>>>>>>>>> [nfsd]
>>>>>>>>>> [Mon Mar 11 14:10:08 2024] ?
>>>>>>>>>> nfsd4_decode_opaque+0x3a/0x90 [nfsd]
>>>>>>>>>> [Mon Mar 11 14:10:08 2024]
>>>>>>>>>> nfsd4_proc_compound+0x44b/0x700 [nfsd]
>>>>>>>>>> [Mon Mar 11 14:10:08 2024] nfsd_dispatch+0x94/0x1c0 [nfsd]
>>>>>>>>>> [Mon Mar 11 14:10:08 2024] svc_process_common+0x2ec/0x660
>>>>>>>>>> [sunrpc]
>>>>>>>>>> [Mon Mar 11 14:10:08 2024] ?
>>>>>>>>>> __pfx_nfsd_dispatch+0x10/0x10 [nfsd]
>>>>>>>>>> [Mon Mar 11 14:10:08 2024] ? __pfx_nfsd+0x10/0x10 [nfsd]
>>>>>>>>>> [Mon Mar 11 14:10:08 2024] svc_process+0x12d/0x170 [sunrpc]
>>>>>>>>>> [Mon Mar 11 14:10:08 2024] nfsd+0x84/0xb0 [nfsd]
>>>>>>>>>> [Mon Mar 11 14:10:08 2024] kthread+0xdd/0x100
>>>>>>>>>> [Mon Mar 11 14:10:08 2024] ? __pfx_kthread+0x10/0x10
>>>>>>>>>> [Mon Mar 11 14:10:08 2024] ret_from_fork+0x29/0x50
>>>>>>>>>> [Mon Mar 11 14:10:08 2024] </TASK>
>>>>>>>>>> [Mon Mar 11 14:10:08 2024] INFO: task nfsd:8866 blocked for
>>>>>>>>>> more than 122 seconds.
>>>>>>>>>> [Mon Mar 11 14:10:08 2024] Not tainted
>>>>>>>>>> 5.14.0-419.el9.x86_64 #1
>>>>>>>>>> [Mon Mar 11 14:10:08 2024] "echo 0 >
>>>>>>>>>> /proc/sys/kernel/hung_task_timeout_secs" disables this message.
>>>>>>>>>> [Mon Mar 11 14:10:08 2024]task:nfsd state:D
>>>>>>>>>> stack:0
>>>>>>>>>> pid:8866 ppid:2 flags:0x00004000
>>>>>>>>>> [Mon Mar 11 14:10:08 2024] Call Trace:
>>>>>>>>>> [Mon Mar 11 14:10:08 2024] <TASK>
>>>>>>>>>> [Mon Mar 11 14:10:08 2024] __schedule+0x21b/0x550
>>>>>>>>>> [Mon Mar 11 14:10:08 2024] schedule+0x2d/0x70
>>>>>>>>>> [Mon Mar 11 14:10:08 2024] schedule_timeout+0x11f/0x160
>>>>>>>>>> [Mon Mar 11 14:10:08 2024] ? select_idle_sibling+0x28/0x430
>>>>>>>>>> [Mon Mar 11 14:10:08 2024] ? tcp_recvmsg+0x196/0x210
>>>>>>>>>> [Mon Mar 11 14:10:08 2024] ? wake_affine+0x62/0x1f0
>>>>>>>>>> [Mon Mar 11 14:10:08 2024] __wait_for_common+0x90/0x1d0
>>>>>>>>>> [Mon Mar 11 14:10:08 2024] ?
>>>>>>>>>> __pfx_schedule_timeout+0x10/0x10
>>>>>>>>>> [Mon Mar 11 14:10:08 2024] __flush_workqueue+0x13a/0x3f0
>>>>>>>>>> [Mon Mar 11 14:10:08 2024]
>>>>>>>>>> nfsd4_destroy_session+0x1a4/0x240
>>>>>>>>>> [nfsd]
>>>>>>>>>> [Mon Mar 11 14:10:08 2024]
>>>>>>>>>> nfsd4_proc_compound+0x44b/0x700 [nfsd]
>>>>>>>>>> [Mon Mar 11 14:10:08 2024] nfsd_dispatch+0x94/0x1c0 [nfsd]
>>>>>>>>>> [Mon Mar 11 14:10:08 2024] svc_process_common+0x2ec/0x660
>>>>>>>>>> [sunrpc]
>>>>>>>>>> [Mon Mar 11 14:10:08 2024] ?
>>>>>>>>>> __pfx_nfsd_dispatch+0x10/0x10 [nfsd]
>>>>>>>>>> [Mon Mar 11 14:10:08 2024] ? __pfx_nfsd+0x10/0x10 [nfsd]
>>>>>>>>>> [Mon Mar 11 14:10:08 2024] svc_process+0x12d/0x170 [sunrpc]
>>>>>>>>>> [Mon Mar 11 14:10:08 2024] nfsd+0x84/0xb0 [nfsd]
>>>>>>>>>> [Mon Mar 11 14:10:08 2024] kthread+0xdd/0x100
>>>>>>>>>> [Mon Mar 11 14:10:08 2024] ? __pfx_kthread+0x10/0x10
>>>>>>>>>> [Mon Mar 11 14:10:08 2024] ret_from_fork+0x29/0x50
>>>>>>>>>> [Mon Mar 11 14:10:08 2024] </TASK>
>>>>>>>>>>
>>>>>>>>> The above threads are trying to flush the workqueue, so that
>>>>>>>>> probably
>>>>>>>>> means that they are stuck waiting on a workqueue job to finish.
>>>>>>>>>> The above is repeated a few times, and then this warning is
>>>>>>>>>> also logged:
>>>>>>>>>> [Mon Mar 11 14:12:04 2024] ------------[ cut here
>>>>>>>>>> ]------------
>>>>>>>>>> [Mon Mar 11 14:12:04 2024] WARNING: CPU: 39 PID: 8844 at
>>>>>>>>>> fs/nfsd/nfs4state.c:4919 nfsd_break_deleg_cb+0x170/0x190 [nfsd]
>>>>>>>>>> [Mon Mar 11 14:12:05 2024] Modules linked in: nfsv4
>>>>>>>>>> dns_resolver nfs fscache netfs rpcsec_gss_krb5 rpcrdma rdma_cm
>>>>>>>>>> iw_cm ib_cm ib_core binfmt_misc bonding tls rfkill
>>>>>>>>>> nft_counter nft_ct
>>>>>>>>>> nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_reject_inet
>>>>>>>>>> nf_reject_ipv4 nf_reject_ipv6 nft_reject nf_tables nfnetlink
>>>>>>>>>> vfat
>>>>>>>>>> fat dm_thin_pool dm_persistent_data dm_bio_prison dm_bufio l
>>>>>>>>>> ibcrc32c dm_service_time dm_multipath intel_rapl_msr
>>>>>>>>>> intel_rapl_common intel_uncore_frequency
>>>>>>>>>> intel_uncore_frequency_common isst_if_common skx_edac nfit
>>>>>>>>>> libnvdimm ipmi_ssif x86_pkg_temp
>>>>>>>>>> _thermal intel_powerclamp coretemp kvm_intel kvm irqbypass
>>>>>>>>>> dcdbas rapl intel_cstate mgag200 i2c_algo_bit drm_shmem_helper
>>>>>>>>>> dell_smbios drm_kms_helper dell_wmi_descriptor wmi_bmof intel_u
>>>>>>>>>> ncore syscopyarea pcspkr sysfillrect mei_me sysimgblt
>>>>>>>>>> acpi_ipmi
>>>>>>>>>> mei fb_sys_fops i2c_i801 ipmi_si intel_pch_thermal lpc_ich
>>>>>>>>>> ipmi_devintf i2c_smbus ipmi_msghandler joydev acpi_power_meter
>>>>>>>>>> nfsd auth_rpcgss nfs_acl drm lockd grace fuse sunrpc ext4
>>>>>>>>>> mbcache jbd2 sd_mod sg lpfc
>>>>>>>>>> [Mon Mar 11 14:12:05 2024] nvmet_fc nvmet nvme_fc
>>>>>>>>>> nvme_fabrics
>>>>>>>>>> crct10dif_pclmul ahci libahci crc32_pclmul nvme_core
>>>>>>>>>> crc32c_intel
>>>>>>>>>> ixgbe megaraid_sas libata nvme_common ghash_clmulni_int
>>>>>>>>>> el t10_pi wdat_wdt scsi_transport_fc mdio wmi dca dm_mirror
>>>>>>>>>> dm_region_hash dm_log dm_mod
>>>>>>>>>> [Mon Mar 11 14:12:05 2024] CPU: 39 PID: 8844 Comm: nfsd Not
>>>>>>>>>> tainted 5.14.0-419.el9.x86_64 #1
>>>>>>>>>> [Mon Mar 11 14:12:05 2024] Hardware name: Dell Inc.
>>>>>>>>>> PowerEdge
>>>>>>>>>> R740/00WGD1, BIOS 2.20.1 09/13/2023
>>>>>>>>>> [Mon Mar 11 14:12:05 2024] RIP:
>>>>>>>>>> 0010:nfsd_break_deleg_cb+0x170/0x190 [nfsd]
>>>>>>>>>> [Mon Mar 11 14:12:05 2024] Code: a6 95 c5 f3 e9 ff fe ff
>>>>>>>>>> ff 48
>>>>>>>>>> 89 df be 01 00 00 00 e8 34 b5 13 f4 48 8d bb 98 00 00 00 e8
>>>>>>>>>> c8 f9
>>>>>>>>>> 00 00 84 c0 0f 85 2e ff ff ff <0f> 0b e9 27 ff ff ff be
>>>>>>>>>> 02 00 00 00 48 89 df e8 0c b5 13 f4 e9 01
>>>>>>>>>> [Mon Mar 11 14:12:05 2024] RSP: 0018:ffff9929e0bb7b80
>>>>>>>>>> EFLAGS:
>>>>>>>>>> 00010246
>>>>>>>>>> [Mon Mar 11 14:12:05 2024] RAX: 0000000000000000 RBX:
>>>>>>>>>> ffff8ada51930900 RCX: 0000000000000024
>>>>>>>>>> [Mon Mar 11 14:12:05 2024] RDX: ffff8ada519309c8 RSI:
>>>>>>>>>> ffff8ad582933c00 RDI: 0000000000002000
>>>>>>>>>> [Mon Mar 11 14:12:05 2024] RBP: ffff8ad46bf21574 R08:
>>>>>>>>>> ffff9929e0bb7b48 R09: 0000000000000000
>>>>>>>>>> [Mon Mar 11 14:12:05 2024] R10: ffff8aec859a2948 R11:
>>>>>>>>>> 0000000000000000 R12: ffff8ad6f497c360
>>>>>>>>>> [Mon Mar 11 14:12:05 2024] R13: ffff8ad46bf21560 R14:
>>>>>>>>>> ffff8ae5942e0b10 R15: ffff8ad6f497c360
>>>>>>>>>> [Mon Mar 11 14:12:05 2024] FS: 0000000000000000(0000)
>>>>>>>>>> GS:ffff8b031fcc0000(0000) knlGS:0000000000000000
>>>>>>>>>> [Mon Mar 11 14:12:05 2024] CS: 0010 DS: 0000 ES: 0000 CR0:
>>>>>>>>>> 0000000080050033
>>>>>>>>>> [Mon Mar 11 14:12:05 2024] CR2: 00007fafe2060744 CR3:
>>>>>>>>>> 00000018e58de006 CR4: 00000000007706e0
>>>>>>>>>> [Mon Mar 11 14:12:05 2024] DR0: 0000000000000000 DR1:
>>>>>>>>>> 0000000000000000 DR2: 0000000000000000
>>>>>>>>>> [Mon Mar 11 14:12:05 2024] DR3: 0000000000000000 DR6:
>>>>>>>>>> 00000000fffe0ff0 DR7: 0000000000000400
>>>>>>>>>> [Mon Mar 11 14:12:05 2024] PKRU: 55555554
>>>>>>>>>> [Mon Mar 11 14:12:05 2024] Call Trace:
>>>>>>>>>> [Mon Mar 11 14:12:05 2024] <TASK>
>>>>>>>>>> [Mon Mar 11 14:12:05 2024] ? show_trace_log_lvl+0x1c4/0x2df
>>>>>>>>>> [Mon Mar 11 14:12:05 2024] ? show_trace_log_lvl+0x1c4/0x2df
>>>>>>>>>> [Mon Mar 11 14:12:05 2024] ? __break_lease+0x16f/0x5f0
>>>>>>>>>> [Mon Mar 11 14:12:05 2024] ?
>>>>>>>>>> nfsd_break_deleg_cb+0x170/0x190
>>>>>>>>>> [nfsd]
>>>>>>>>>> [Mon Mar 11 14:12:05 2024] ? __warn+0x81/0x110
>>>>>>>>>> [Mon Mar 11 14:12:05 2024] ?
>>>>>>>>>> nfsd_break_deleg_cb+0x170/0x190
>>>>>>>>>> [nfsd]
>>>>>>>>>> [Mon Mar 11 14:12:05 2024] ? report_bug+0x10a/0x140
>>>>>>>>>> [Mon Mar 11 14:12:05 2024] ? handle_bug+0x3c/0x70
>>>>>>>>>> [Mon Mar 11 14:12:05 2024] ? exc_invalid_op+0x14/0x70
>>>>>>>>>> [Mon Mar 11 14:12:05 2024] ? asm_exc_invalid_op+0x16/0x20
>>>>>>>>>> [Mon Mar 11 14:12:05 2024] ?
>>>>>>>>>> nfsd_break_deleg_cb+0x170/0x190
>>>>>>>>>> [nfsd]
>>>>>>>>>> [Mon Mar 11 14:12:05 2024] __break_lease+0x16f/0x5f0
>>>>>>>>>> [Mon Mar 11 14:12:05 2024] ?
>>>>>>>>>> nfsd_file_lookup_locked+0x117/0x160 [nfsd]
>>>>>>>>>> [Mon Mar 11 14:12:05 2024] ? list_lru_del+0x101/0x150
>>>>>>>>>> [Mon Mar 11 14:12:05 2024] nfsd_file_do_acquire+0x790/0x830
>>>>>>>>>> [nfsd]
>>>>>>>>>> [Mon Mar 11 14:12:05 2024] nfs4_get_vfs_file+0x315/0x3a0
>>>>>>>>>> [nfsd]
>>>>>>>>>> [Mon Mar 11 14:12:05 2024]
>>>>>>>>>> nfsd4_process_open2+0x430/0xa30 [nfsd]
>>>>>>>>>> [Mon Mar 11 14:12:05 2024] ? fh_verify+0x297/0x2f0 [nfsd]
>>>>>>>>>> [Mon Mar 11 14:12:05 2024] nfsd4_open+0x3ce/0x4b0 [nfsd]
>>>>>>>>>> [Mon Mar 11 14:12:05 2024]
>>>>>>>>>> nfsd4_proc_compound+0x44b/0x700 [nfsd]
>>>>>>>>>> [Mon Mar 11 14:12:05 2024] nfsd_dispatch+0x94/0x1c0 [nfsd]
>>>>>>>>>> [Mon Mar 11 14:12:05 2024] svc_process_common+0x2ec/0x660
>>>>>>>>>> [sunrpc]
>>>>>>>>>> [Mon Mar 11 14:12:05 2024] ?
>>>>>>>>>> __pfx_nfsd_dispatch+0x10/0x10 [nfsd]
>>>>>>>>>> [Mon Mar 11 14:12:05 2024] ? __pfx_nfsd+0x10/0x10 [nfsd]
>>>>>>>>>> [Mon Mar 11 14:12:05 2024] svc_process+0x12d/0x170 [sunrpc]
>>>>>>>>>> [Mon Mar 11 14:12:05 2024] nfsd+0x84/0xb0 [nfsd]
>>>>>>>>>> [Mon Mar 11 14:12:05 2024] kthread+0xdd/0x100
>>>>>>>>>> [Mon Mar 11 14:12:05 2024] ? __pfx_kthread+0x10/0x10
>>>>>>>>>> [Mon Mar 11 14:12:05 2024] ret_from_fork+0x29/0x50
>>>>>>>>>> [Mon Mar 11 14:12:05 2024] </TASK>
>>>>>>>>>> [Mon Mar 11 14:12:05 2024] ---[ end trace
>>>>>>>>>> 7a039e17443dc651 ]---
>>>>>>>>> This is probably this WARN in nfsd_break_one_deleg:
>>>>>>>>>
>>>>>>>>> WARN_ON_ONCE(!nfsd4_run_cb(&dp->dl_recall));
>>>>>>>>>
>>>>>>>>> It means that a delegation break callback to the client
>>>>>>>>> couldn't be
>>>>>>>>> queued to the workqueue, and so it didn't run.
>>>>>>>>>
>>>>>>>>>> Could this be the same issue as described
>>>>>>>>>> here:https://urldefense.com/v3/__https://lore.kernel.org/linux-nfs/af0ec881-5ebf-4feb-98ae-3ed2a77f86f1@oracle.com/__;!!ACWV5N9M2RV99hQ!LV3yWeoSOhNAkRHkxFCH2tlm0iNFVD78mxnSLyP6lrX7yBVeA2TOJ4nv6oZsqLwP4kW56CMpDWhkjjwSkdBV9En7$
>>>>>>>>>> ?
>>>>>>>>> Yes, most likely the same problem.
>>>>>>>> If I read that thread correctly, this issue was introduced between
>>>>>>>> 6.1.63 and 6.6.3? Is it possible the EL9 5.14.0-362.18.1.el9_3
>>>>>>>> backported these changes, or were we hitting some other bug
>>>>>>>> with that
>>>>>>>> version? It seems the 6.1.x kernel is not affected? If so, that
>>>>>>>> would be
>>>>>>>> the recommended kernel to run?
>>>>>>> Anything is possible. We have to identify the problem first.
>>>>>>>>>> As described in that thread, I've tried to obtain the requested
>>>>>>>>>> information.
>>>>>>>>>>
>>>>>>>>>> Is it possible this is the issue that was fixed by the patches
>>>>>>>>>> described
>>>>>>>>>> here?https://urldefense.com/v3/__https://lore.kernel.org/linux-nfs/2024022054-cause-suffering-eae8@gregkh/__;!!ACWV5N9M2RV99hQ!LV3yWeoSOhNAkRHkxFCH2tlm0iNFVD78mxnSLyP6lrX7yBVeA2TOJ4nv6oZsqLwP4kW56CMpDWhkjjwSkedtUP09$
>>>>>>>>>>
>>>>>>>>> Doubtful. Those are targeted toward a different set of issues.
>>>>>>>>>
>>>>>>>>> If you're willing, I do have some patches queued up for CentOS
>>>>>>>>> here
>>>>>>>>> that
>>>>>>>>> fix some backchannel problems that could be related. I'm mainly
>>>>>>>>> waiting
>>>>>>>>> on Chuck to send these to Linus and then we'll likely merge
>>>>>>>>> them into
>>>>>>>>> CentOS soon afterward:
>>>>>>>>>
>>>>>>>>> https://urldefense.com/v3/__https://gitlab.com/redhat/centos-stream/src/kernel/centos-stream-9/-/merge_requests/3689__;!!ACWV5N9M2RV99hQ!LV3yWeoSOhNAkRHkxFCH2tlm0iNFVD78mxnSLyP6lrX7yBVeA2TOJ4nv6oZsqLwP4kW56CMpDWhkjjwSkdvDn8y7$
>>>>>>>>>
>>>>>>>>>
>>>>>>>> If you can send me a patch file, I can rebuild the C9S kernel
>>>>>>>> with that
>>>>>>>> patch and run it. It can take a while for the bug to trigger as I
>>>>>>>> believe it seems to be very workload dependent (we were running
>>>>>>>> very
>>>>>>>> stable for months and now hit this bug every other week).
>>>>>>>>
>>>>>>>>
>>>>>>> It's probably simpler to just pull down the build artifacts for
>>>>>>> that MR.
>>>>>>> You have to drill down through the CI for it, but they are here:
>>>>>>>
>>>>>>> https://urldefense.com/v3/__https://s3.amazonaws.com/arr-cki-prod-trusted-artifacts/index.html?prefix=trusted-artifacts*1194300175*publish_x86_64*6278921877*artifacts*__;Ly8vLy8!!ACWV5N9M2RV99hQ!LV3yWeoSOhNAkRHkxFCH2tlm0iNFVD78mxnSLyP6lrX7yBVeA2TOJ4nv6oZsqLwP4kW56CMpDWhkjjwSkaP5eW8V$
>>>>>>>
>>>>>>>
>>>>>>> There's even a repo file you can install on the box to pull them
>>>>>>> down.
>>>>>> We installed this kernel on the server 3 days ago. Today, a user
>>>>>> informed us that their screen was black after logging in. Similar to
>>>>>> other occurrences of this issue, the mount command on the client was
>>>>>> hung. But in contrast to the other times, there were no messages in
>>>>>> the logs kernel logs on the server. Even restarting the client does
>>>>>> not resolve the issue.
>>>>
>>>> Ok, so you rebooted the client and it's still unable to mount? That
>>>> sounds like a server problem if so.
>>>>
>>>> Are both client and server running the same kernel?
>>> No, the server runs 5.14.0-427.3689_1194299994.el9 and the client
>>> 5.14.0-362.18.1.el9_3.
>>>>
>>>>>> Something still seems to be wrong on the server though. When I
>>>>>> look at
>>>>>> the directories under /proc/fs/nfsd/clients, there's still a
>>>>>> directory
>>>>>> for the specific client, even though it's no longer running:
>>>>>>
>>>>>> # cat 155/info
>>>>>> clientid: 0xc8edb7f65f4a9ad
>>>>>> address: "10.87.31.152:819"
>>>>>> status: confirmed
>>>>>> seconds from last renew: 33163
>>>>>> name: "Linux NFSv4.2 bersalis.esat.kuleuven.be"
>>>>>> minor version: 2
>>>>>> Implementation domain: "kernel.org"
>>>>>> Implementation name: "Linux 5.14.0-362.18.1.el9_3.0.1.x86_64 #1 SMP
>>>>>> PREEMPT_DYNAMIC Sun Feb 11 13:49:23 UTC 2024 x86_64"
>>>>>> Implementation time: [0, 0]
>>>>>> callback state: DOWN
>>>>>> callback address: 10.87.31.152:0
>>>>>>
>>>> If you just shut down the client, the server won't immediately
>>>> purge its
>>>> record. In fact, assuming you're running the same kernel on the
>>>> server,
>>>> it won't purge the client record until there is a conflicting request
>>>> for its state.
>>> Is there a way to force such a conflicting request (to get the
>>> client record to purge)?
>>
>> Try:
>>
>> # echo "expire" > /proc/fs/nfsd/clients/155/ctl
>
> I've tried that. The command hangs and can not be interrupted with
> ctrl-c.
> I've now also noticed in the dmesg output that the kernel issued the
> following WARNING a few hours ago. It wasn't directly triggered by the
> echo command above, but seems to have been triggered a few hours ago
> (probably when another client started to have the same problem as more
> clients are experiencing issues now).
I think this warning message is harmless. However it indicates potential
problem with the workqueue which might be related to memory shortage.
What the output of 'cat /proc/meminfo' looks like?
Did you try 'echo 3 > /proc/sys/vm/drop_caches'?
>
> [Tue Mar 19 14:53:44 2024] ------------[ cut here ]------------
> [Tue Mar 19 14:53:44 2024] WARNING: CPU: 44 PID: 5843 at
> fs/nfsd/nfs4state.c:4920 nfsd_break_deleg_cb+0x170/0x190 [nfsd]
> [Tue Mar 19 14:53:44 2024] Modules linked in: nf_conntrack_netlink
> nfsv4 dns_resolver nfs fscache netfs binfmt_misc xsk_diag
> rpcsec_gss_krb5 rpcrdma rdma_cm iw_cm ib_cm ib_core bonding tls rfkill
> nft_counter nft_ct nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4
> nft_reject_inet nf_reject_ipv4 nf_reject_ipv6 nft_reject nf_tables
> nfnetlink vfat fat dm_thin_pool dm_persistent_data dm_bio_prison
> dm_bufio libcrc32c dm_service_time dm_multipath intel_rapl_msr
> intel_rapl_common intel_uncore_frequency intel_uncore_frequency_common
> isst_if_common skx_edac nfit libnvdimm x86_pkg_temp_thermal
> intel_powerclamp coretemp kvm_intel kvm dcdbas irqbypass ipmi_ssif
> rapl intel_cstate mgag200 i2c_algo_bit drm_shmem_helper drm_kms_helper
> dell_smbios syscopyarea intel_uncore sysfillrect wmi_bmof
> dell_wmi_descriptor pcspkr sysimgblt fb_sys_fops mei_me i2c_i801 mei
> intel_pch_thermal acpi_ipmi i2c_smbus lpc_ich ipmi_si ipmi_devintf
> ipmi_msghandler joydev acpi_power_meter nfsd nfs_acl lockd auth_rpcgss
> grace drm fuse sunrpc ext4
> [Tue Mar 19 14:53:44 2024] mbcache jbd2 sd_mod sg lpfc nvmet_fc nvmet
> nvme_fc nvme_fabrics crct10dif_pclmul crc32_pclmul nvme_core ixgbe
> crc32c_intel ahci libahci nvme_common megaraid_sas t10_pi
> ghash_clmulni_intel wdat_wdt libata scsi_transport_fc mdio dca wmi
> dm_mirror dm_region_hash dm_log dm_mod
> [Tue Mar 19 14:53:44 2024] CPU: 44 PID: 5843 Comm: nfsd Not tainted
> 5.14.0-427.3689_1194299994.el9.x86_64 #1
> [Tue Mar 19 14:53:44 2024] Hardware name: Dell Inc. PowerEdge
> R740/00WGD1, BIOS 2.20.1 09/13/2023
> [Tue Mar 19 14:53:44 2024] RIP: 0010:nfsd_break_deleg_cb+0x170/0x190
> [nfsd]
> [Tue Mar 19 14:53:44 2024] Code: 76 76 cd de e9 ff fe ff ff 48 89 df
> be 01 00 00 00 e8 34 a1 1b df 48 8d bb 98 00 00 00 e8 a8 fe 00 00 84
> c0 0f 85 2e ff ff ff <0f> 0b e9 27 ff ff ff be 02 00 00 00 48 89 df e8
> 0c a1 1b df e9 01
> [Tue Mar 19 14:53:44 2024] RSP: 0018:ffffb2878f2cfc38 EFLAGS: 00010246
> [Tue Mar 19 14:53:44 2024] RAX: 0000000000000000 RBX: ffff88d5171067b8
> RCX: 0000000000000000
> [Tue Mar 19 14:53:44 2024] RDX: ffff88d517106880 RSI: ffff88bdceec8600
> RDI: 0000000000002000
> [Tue Mar 19 14:53:44 2024] RBP: ffff88d68a38a284 R08: ffffb2878f2cfc00
> R09: 0000000000000000
> [Tue Mar 19 14:53:44 2024] R10: ffff88bf57dd7878 R11: 0000000000000000
> R12: ffff88d5b79c4798
> [Tue Mar 19 14:53:44 2024] R13: ffff88d68a38a270 R14: ffff88cab06ad0c8
> R15: ffff88d5b79c4798
> [Tue Mar 19 14:53:44 2024] FS: 0000000000000000(0000)
> GS:ffff88d4a1180000(0000) knlGS:0000000000000000
> [Tue Mar 19 14:53:44 2024] CS: 0010 DS: 0000 ES: 0000 CR0:
> 0000000080050033
> [Tue Mar 19 14:53:44 2024] CR2: 00007fe46ef90000 CR3: 000000019d010004
> CR4: 00000000007706e0
> [Tue Mar 19 14:53:44 2024] DR0: 0000000000000000 DR1: 0000000000000000
> DR2: 0000000000000000
> [Tue Mar 19 14:53:44 2024] DR3: 0000000000000000 DR6: 00000000fffe0ff0
> DR7: 0000000000000400
> [Tue Mar 19 14:53:44 2024] PKRU: 55555554
> [Tue Mar 19 14:53:44 2024] Call Trace:
> [Tue Mar 19 14:53:44 2024] <TASK>
> [Tue Mar 19 14:53:44 2024] ? show_trace_log_lvl+0x1c4/0x2df
> [Tue Mar 19 14:53:44 2024] ? show_trace_log_lvl+0x1c4/0x2df
> [Tue Mar 19 14:53:44 2024] ? __break_lease+0x16f/0x5f0
> [Tue Mar 19 14:53:44 2024] ? nfsd_break_deleg_cb+0x170/0x190 [nfsd]
> [Tue Mar 19 14:53:44 2024] ? __warn+0x81/0x110
> [Tue Mar 19 14:53:44 2024] ? nfsd_break_deleg_cb+0x170/0x190 [nfsd]
> [Tue Mar 19 14:53:44 2024] ? report_bug+0x10a/0x140
> [Tue Mar 19 14:53:44 2024] ? handle_bug+0x3c/0x70
> [Tue Mar 19 14:53:44 2024] ? exc_invalid_op+0x14/0x70
> [Tue Mar 19 14:53:44 2024] ? asm_exc_invalid_op+0x16/0x20
> [Tue Mar 19 14:53:44 2024] ? nfsd_break_deleg_cb+0x170/0x190 [nfsd]
> [Tue Mar 19 14:53:44 2024] ? nfsd_break_deleg_cb+0x96/0x190 [nfsd]
> [Tue Mar 19 14:53:44 2024] __break_lease+0x16f/0x5f0
> [Tue Mar 19 14:53:44 2024] nfs4_get_vfs_file+0x164/0x3a0 [nfsd]
> [Tue Mar 19 14:53:44 2024] nfsd4_process_open2+0x430/0xa30 [nfsd]
> [Tue Mar 19 14:53:44 2024] ? fh_verify+0x297/0x2f0 [nfsd]
> [Tue Mar 19 14:53:44 2024] nfsd4_open+0x3ce/0x4b0 [nfsd]
> [Tue Mar 19 14:53:44 2024] nfsd4_proc_compound+0x44b/0x700 [nfsd]
> [Tue Mar 19 14:53:44 2024] nfsd_dispatch+0x94/0x1c0 [nfsd]
> [Tue Mar 19 14:53:44 2024] svc_process_common+0x2ec/0x660 [sunrpc]
> [Tue Mar 19 14:53:44 2024] ? __pfx_nfsd_dispatch+0x10/0x10 [nfsd]
> [Tue Mar 19 14:53:44 2024] ? __pfx_nfsd+0x10/0x10 [nfsd]
> [Tue Mar 19 14:53:44 2024] svc_process+0x12d/0x170 [sunrpc]
> [Tue Mar 19 14:53:44 2024] nfsd+0x84/0xb0 [nfsd]
> [Tue Mar 19 14:53:44 2024] kthread+0xdd/0x100
> [Tue Mar 19 14:53:44 2024] ? __pfx_kthread+0x10/0x10
> [Tue Mar 19 14:53:44 2024] ret_from_fork+0x29/0x50
> [Tue Mar 19 14:53:44 2024] </TASK>
> [Tue Mar 19 14:53:44 2024] ---[ end trace ed0b2b3f135c637d ]---
>
> It again seems to have been triggered in nfsd_break_deleg_cb?
>
> I also had the following perf command running a tmux on the server:
>
> perf trace -e nfsd:nfsd_cb_recall_any
>
> This has spewed a lot of messages. I'm including a short list here:
>
> ...
>
> 33464866.721 kworker/u98:5/1591466 nfsd:nfsd_cb_recall_any(cl_boot:
> 1710533037, cl_id: 210688785, bmval0: 1, addr: 0x7f331bb116c8)
> 33464866.724 kworker/u98:5/1591466 nfsd:nfsd_cb_recall_any(cl_boot:
> 1710533037, cl_id: 210688827, bmval0: 1, addr: 0x7f331bb11738)
> 33464866.729 kworker/u98:5/1591466 nfsd:nfsd_cb_recall_any(cl_boot:
> 1710533037, cl_id: 210688767, bmval0: 1, addr: 0x7f331bb117a8)
> 33464866.732 kworker/u98:5/1591466 nfsd:nfsd_cb_recall_any(cl_boot:
> 1710533037, cl_id: 210718132, bmval0: 1, addr: 0x7f331bb11818)
> 33464866.737 kworker/u98:5/1591466 nfsd:nfsd_cb_recall_any(cl_boot:
> 1710533037, cl_id: 210688952, bmval0: 1, addr: 0x7f331bb11888)
> 33464866.741 kworker/u98:5/1591466 nfsd:nfsd_cb_recall_any(cl_boot:
> 1710533037, cl_id: 210702355, bmval0: 1, addr: 0x7f331bb118f8)
> 33868414.001 kthreadd/1597068 nfsd:nfsd_cb_recall_any(cl_boot:
> 1710533037, cl_id: 210688751, bmval0: 1, addr: 0x7f331be68620)
> 33868414.014 kthreadd/1597068 nfsd:nfsd_cb_recall_any(cl_boot:
> 1710533037, cl_id: 210718536, bmval0: 1, addr: 0x7f331be68690)
> 33868414.018 kthreadd/1597068 nfsd:nfsd_cb_recall_any(cl_boot:
> 1710533037, cl_id: 210719074, bmval0: 1, addr: 0x7f331be68700)
> 33868414.022 kthreadd/1597068 nfsd:nfsd_cb_recall_any(cl_boot:
> 1710533037, cl_id: 210688916, bmval0: 1, addr: 0x7f331be68770)
> 33868414.026 kthreadd/1597068 nfsd:nfsd_cb_recall_any(cl_boot:
> 1710533037, cl_id: 210688941, bmval0: 1, addr: 0x7f331be687e0)
> ...
>
> 33868414.924 kthreadd/1597068 nfsd:nfsd_cb_recall_any(cl_boot:
> 1710533037, cl_id: 210688744, bmval0: 1, addr: 0x7f331be6d7f0)
> 33868414.929 kthreadd/1597068 nfsd:nfsd_cb_recall_any(cl_boot:
> 1710533037, cl_id: 210717223, bmval0: 1, addr: 0x7f331be6d860)
> 33868414.934 kthreadd/1597068 nfsd:nfsd_cb_recall_any(cl_boot:
> 1710533037, cl_id: 210716137, bmval0: 1, addr: 0x7f331be6d8d0)
> 34021240.903 kworker/u98:5/1591466 nfsd:nfsd_cb_recall_any(cl_boot:
> 1710533037, cl_id: 210688941, bmval0: 1, addr: 0x7f331c207de8)
> 34021240.917 kworker/u98:5/1591466 nfsd:nfsd_cb_recall_any(cl_boot:
> 1710533037, cl_id: 210718750, bmval0: 1, addr: 0x7f331c207e58)
> 34021240.922 kworker/u98:5/1591466 nfsd:nfsd_cb_recall_any(cl_boot:
> 1710533037, cl_id: 210688955, bmval0: 1, addr: 0x7f331c207ec8)
> 34021240.925 kworker/u98:5/1591466 nfsd:nfsd_cb_recall_any(cl_boot:
> 1710533037, cl_id: 210688975, bmval0: 1, addr: 0x7f331c207f38)
> ...
>
> I assume the cl_id is the client id? How can I map this to a client
> from /proc/fs/nfsd/clients?
The hex value of 'clientid' printed from /proc/fs/nfsd/clients/XX/info
is a 64-bit value composed of:
typedef struct {
u32 cl_boot;
u32 cl_id;
} clientid_t
For example:
clientid: 0xc8edb7f65f4a9ad
cl_boot: 65f4a9add (1710533037)
cl_id: c8edb7f (21068895)
This should match a trace event with:
nfsd:nfsd_cb_recall_any(cl_boot: 1710533037, cl_id: 21068895, bmval0: XX, addr: 0xYYYYY)
>
> If I understand it correctly, the recall_any should be called when
> either the system starts to experience memory pressure,
yes.
> or it reaches the delegation limits?
No, this feature was added to nfsd very recently. I don't think your kernel has it.
> I doubt the system is actually running out of memory here as there are
> no other indications.
> Shouldn't I get those "page allocation failure" messages if it does?
> How can I check the number of delegations/leases currently issued,
> what the current maximum is and how to increase it?
Max delegations is 4 per 1MB of available memory. There is no
admin tool to adjust this value.
I do not recommend running a production system with delegation
disabled. But for this specific issue, it might help to temporarily
disable delegation to isolate problem areas.
-Dai
>
> Regarding the recall any call: from what I've read on kernelnewbies,
> this feature was introduced in the 6.2 kernel? When I look at the tree
> for 6.1.x, it was backported in 6.1.81? Is there a way to disable this
> support somehow?
>
> Regards,
>
> Rik
>
>
>>
>> -Dai
>>
>>>>
>>>>
>>>>> The nfsdclnts command for this client shows the following
>>>>> delegations:
>>>>>
>>>>> # nfsdclnts -f 155/states -t all
>>>>> Inode number | Type | Access | Deny | ip address | Filename
>>>>> 169346743 | open | r- | -- | 10.87.31.152:819 |
>>>>> disconnected dentry
>>>>> 169346743 | deleg | r | | 10.87.31.152:819 |
>>>>> disconnected dentry
>>>>> 169346746 | open | r- | -- | 10.87.31.152:819 |
>>>>> disconnected dentry
>>>>> 169346746 | deleg | r | | 10.87.31.152:819 |
>>>>> disconnected dentry
>>>>>
>>>>> I see a lot of recent patches regarding directory delegations. Could
>>>>> this be related to this?
>>>>>
>>>>> Will a 5.14.0-362.18.1.el9_3.0.1 kernel try to use a directory
>>>>> delegation?
>>>>>
>>>>>
>>>> No. Directory delegations are a new feature that's still under
>>>> development. They use some of the same machinery as file delegations,
>>>> but they wouldn't be a factor here.
>>>>
>>>>>> The system seems to have identified that the client is no longer
>>>>>> reachable, but the client entry does not go away. When a mount was
>>>>>> hanging on the client, there would be two directories in clients for
>>>>>> the same client. Killing the mount command clears up the second
>>>>>> entry.
>>>>>>
>>>>>> Even after running conntrack -D on the server to remove the tcp
>>>>>> connection from the conntrack table, the entry doesn't go away
>>>>>> and the
>>>>>> client still can not mount anything from the server.
>>>>>>
>>>>>> A tcpdump on the client while a mount was running logged the
>>>>>> following
>>>>>> messages over and over again:
>>>>>>
>>>>>> request:
>>>>>>
>>>>>> Frame 1: 378 bytes on wire (3024 bits), 378 bytes captured (3024
>>>>>> bits)
>>>>>> Ethernet II, Src: HP_19:7d:4b (e0:73:e7:19:7d:4b), Dst:
>>>>>> ArubaaHe_f9:8e:00 (88:3a:30:f9:8e:00)
>>>>>> Internet Protocol Version 4, Src: 10.87.31.152, Dst: 10.86.18.14
>>>>>> Transmission Control Protocol, Src Port: 932, Dst Port: 2049,
>>>>>> Seq: 1,
>>>>>> Ack: 1, Len: 312
>>>>>> Remote Procedure Call, Type:Call XID:0x1d3220c4
>>>>>> Network File System
>>>>>> [Program Version: 4]
>>>>>> [V4 Procedure: COMPOUND (1)]
>>>>>> GSS Data, Ops(1): CREATE_SESSION
>>>>>> Length: 152
>>>>>> GSS Sequence Number: 76
>>>>>> Tag: <EMPTY>
>>>>>> minorversion: 2
>>>>>> Operations (count: 1): CREATE_SESSION
>>>>>> [Main Opcode: CREATE_SESSION (43)]
>>>>>> GSS Checksum:
>>>>>> 00000028040404ffffffffff000000002c19055f1f8d442d594c13849628affc2797cbb2…
>>>>>>
>>>>>> GSS Token Length: 40
>>>>>> GSS-API Generic Security Service Application Program
>>>>>> Interface
>>>>>> krb5_blob:
>>>>>> 040404ffffffffff000000002c19055f1f8d442d594c13849628affc2797cbb23fa080b0…
>>>>>>
>>>>>>
>>>>>> response:
>>>>>>
>>>>>> Frame 2: 206 bytes on wire (1648 bits), 206 bytes captured (1648
>>>>>> bits)
>>>>>> Ethernet II, Src: ArubaaHe_f9:8e:00 (88:3a:30:f9:8e:00), Dst:
>>>>>> HP_19:7d:4b (e0:73:e7:19:7d:4b)
>>>>>> Internet Protocol Version 4, Src: 10.86.18.14, Dst: 10.87.31.152
>>>>>> Transmission Control Protocol, Src Port: 2049, Dst Port: 932,
>>>>>> Seq: 1,
>>>>>> Ack: 313, Len: 140
>>>>>> Remote Procedure Call, Type:Reply XID:0x1d3220c4
>>>>>> Network File System
>>>>>> [Program Version: 4]
>>>>>> [V4 Procedure: COMPOUND (1)]
>>>>>> GSS Data, Ops(1): CREATE_SESSION(NFS4ERR_DELAY)
>>>>>> Length: 24
>>>>>> GSS Sequence Number: 76
>>>>>> Status: NFS4ERR_DELAY (10008)
>>>>>> Tag: <EMPTY>
>>>>>> Operations (count: 1)
>>>>>> [Main Opcode: CREATE_SESSION (43)]
>>>>>> GSS Checksum:
>>>>>> 00000028040405ffffffffff000000000aa742d0798deaad1a8aa2d7c3a91bf4f6274222…
>>>>>>
>>>>>> GSS Token Length: 40
>>>>>> GSS-API Generic Security Service Application Program
>>>>>> Interface
>>>>>> krb5_blob:
>>>>>> 040405ffffffffff000000000aa742d0798deaad1a8aa2d7c3a91bf4f627422226d74923…
>>>>>>
>>>>>>
>>>>>> I was hoping that giving the client a different IP address would
>>>>>> resolve the issue for this client, but it didn't. Even though the
>>>>>> client had a new IP address (hostname was kept the same), it
>>>>>> failed to
>>>>>> mount anything from the server.
>>>>>>
>>>> Changing the IP address won't help. The client is probably using the
>>>> same long-form client id as before, so the server still identifies the
>>>> client even with the address change.
>>> How is the client id determined? Will changing the hostname of the
>>> client trigger a change of the client id?
>>>>
>>>> Unfortunately, the cause of an NFS4ERR_DELAY error is tough to guess.
>>>> The client is expected to back off and retry, so if the server keeps
>>>> returning that repeatedly, then a hung mount command is expected.
>>>>
>>>> The question is why the server would keep returning DELAY. A lot of
>>>> different problems ranging from memory allocation issues to protocol
>>>> problems can result in that error. You may want to check the NFS
>>>> server
>>>> and see if anything was logged there.
>>> There are no messages in the system logs that indicate any sort of
>>> memory issue. We also increased the min_kbytes_free sysctl to 2G on
>>> the server before we restarted it with the newer kernel.
>>>>
>>>> This is on a CREATE_SESSION call, so I wonder if the record held by
>>>> the
>>>> (courteous) server is somehow blocking the attempt to reestablish the
>>>> session?
>>>>
>>>> Do you have a way to reproduce this? Since this is a centos kernel,
>>>> you
>>>> could follow the page here to open a bug:
>>>
>>> Unfortunately we haven't found a reliable way to reproduce it. But
>>> we do seem to trigger it more and more lately.
>>>
>>> Regards,
>>>
>>> Rik
>>>
>>>>
>>>> https://urldefense.com/v3/__https://wiki.centos.org/ReportBugs.html__;!!ACWV5N9M2RV99hQ!LV3yWeoSOhNAkRHkxFCH2tlm0iNFVD78mxnSLyP6lrX7yBVeA2TOJ4nv6oZsqLwP4kW56CMpDWhkjjwSkWIqsboq$
>>>>
>>>>
>>>>>> I created another dump of the workqueues and worker pools on the
>>>>>> server:
>>>>>>
>>>>>> [Mon Mar 18 14:59:33 2024] Showing busy workqueues and worker pools:
>>>>>> [Mon Mar 18 14:59:33 2024] workqueue events: flags=0x0
>>>>>> [Mon Mar 18 14:59:33 2024] pwq 54: cpus=27 node=1 flags=0x0 nice=0
>>>>>> active=1/256 refcnt=2
>>>>>> [Mon Mar 18 14:59:33 2024] pending: drm_fb_helper_damage_work
>>>>>> [drm_kms_helper]
>>>>>> [Mon Mar 18 14:59:33 2024] workqueue events_power_efficient:
>>>>>> flags=0x80
>>>>>> [Mon Mar 18 14:59:33 2024] pwq 54: cpus=27 node=1 flags=0x0 nice=0
>>>>>> active=1/256 refcnt=2
>>>>>> [Mon Mar 18 14:59:33 2024] pending: fb_flashcursor
>>>>>> [Mon Mar 18 14:59:33 2024] workqueue mm_percpu_wq: flags=0x8
>>>>>> [Mon Mar 18 14:59:33 2024] pwq 54: cpus=27 node=1 flags=0x0 nice=0
>>>>>> active=1/256 refcnt=3
>>>>>> [Mon Mar 18 14:59:33 2024] pending: lru_add_drain_per_cpu
>>>>>> BAR(362)
>>>>>> [Mon Mar 18 14:59:33 2024] workqueue kblockd: flags=0x18
>>>>>> [Mon Mar 18 14:59:33 2024] pwq 55: cpus=27 node=1 flags=0x0
>>>>>> nice=-20
>>>>>> active=1/256 refcnt=2
>>>>>> [Mon Mar 18 14:59:33 2024] pending: blk_mq_timeout_work
>>>>>>
>>>>>>
>>>>>> In contrast to last time, it doesn't show anything regarding nfs
>>>>>> this
>>>>>> time.
>>>>>>
>>>>>> I also tried the suggestion from Dai Ngo (echo 3 >
>>>>>> /proc/sys/vm/drop_caches), but that didn't seem to make any
>>>>>> difference.
>>>>>>
>>>>>> We haven't restarted the server yet as it seems the impact seems to
>>>>>> affect fewer clients that before. Is there anything we can run on
>>>>>> the
>>>>>> server to further debug this?
>>>>>>
>>>>>> In the past, the issue seemed to deteriorate rapidly and resulted in
>>>>>> issues for almost all clients after about 20 minutes. This time the
>>>>>> impact seems to be less, but it's not gone.
>>>>>>
>>>>>> How can we force the NFS server to forget about a specific client? I
>>>>>> haven't tried to restart the nfs service yet as I'm afraid it will
>>>>>> fail to stop as before.
>>>>>>
>>>> Not with that kernel. There are some new administrative interfaces
>>>> that
>>>> might allow that in the future, but they were just merged upstream and
>>>> aren't in that kernel.
>>>>
>>>> --
>>>> Jeff Layton <jlayton@kernel.org>
>>>
^ permalink raw reply [flat|nested] 26+ messages in thread
* Re: nfsd hangs and nfsd_break_deleg_cb+0x170/0x190 warning
2024-03-19 11:36 ` Jeff Layton
@ 2024-03-20 13:03 ` Charles Hedrick
0 siblings, 0 replies; 26+ messages in thread
From: Charles Hedrick @ 2024-03-20 13:03 UTC (permalink / raw)
To: Jeff Layton; +Cc: Rik Theys, Linux Nfs
> On Mar 19, 2024, at 7:36 AM, Jeff Layton <jlayton@kernel.org> wrote:
>
> On Tue, 2024-03-19 at 11:58 +0100, Rik Theys wrote:
>> Hi,
>>
>> On 3/19/24 11:39, Jeff Layton wrote:
>>> On Tue, 2024-03-19 at 08:58 +0100, Rik Theys wrote:
>>>> Hi,
>>>>
>>>> On 3/18/24 22:54, Jeff Layton wrote:
>>>>> On Mon, 2024-03-18 at 22:15 +0100, Rik Theys wrote:
>>>>>> Hi,
>>>>>>
>>>>>> On 3/18/24 21:21, Rik Theys wrote:
>>>>>>> Hi Jeff,
>>>>>>>
>>>>>>> On 3/12/24 13:47, Jeff Layton wrote:
>>>>>>>> On Tue, 2024-03-12 at 13:24 +0100, Rik Theys wrote:
>>>>>>>>> Hi Jeff,
>>>>>>>>>
>>>>>>>>> On 3/12/24 12:22, Jeff Layton wrote:
>>>>>>>>>> On Mon, 2024-03-11 at 19:43 +0100, Rik Theys wrote:
>>>>>>>>>>> Since a few weeks our Rocky Linux 9 NFS server has periodically
>>>>>>>>>>> logged hung nfsd tasks. The initial effect was that some clients
>>>>>>>>>>> could no longer access the NFS server. This got worse and worse
>>>>>>>>>>> (probably as more nfsd threads got blocked) and we had to restart
>>>>>>>>>>> the server. Restarting the server also failed as the NFS server
>>>>>>>>>>> service could no longer be stopped.
>>>>>>>>>>>
>>>>>>>>>>> The initial kernel we noticed this behavior on was
>>>>>>>>>>> kernel-5.14.0-362.18.1.el9_3.x86_64. Since then we've installed
>>>>>>>>>>> kernel-5.14.0-419.el9.x86_64 from CentOS Stream 9. The same issue
>>>>>>>>>>> happened again on this newer kernel version:
>>>>>>>> 419 is fairly up to date with nfsd changes. There are some known bugs
>>>>>>>> around callbacks, and there is a draft MR in flight to fix it.
>>>>>>>>
>>>>>>>> What kernel were you on prior to 5.14.0-362.18.1.el9_3.x86_64 ? If we
>>>>>>>> can bracket the changes around a particular version, then that might
>>>>>>>> help identify the problem.
>>>>>>>>
>>>>>>>>>>> [Mon Mar 11 14:10:08 2024] Not tainted 5.14.0-419.el9.x86_64 #1
>>>>>>>>>>> [Mon Mar 11 14:10:08 2024] "echo 0 >
>>>>>>>>>>> /proc/sys/kernel/hung_task_timeout_secs" disables this message.
>>>>>>>>>>> [Mon Mar 11 14:10:08 2024]task:nfsd state:D stack:0
>>>>>>>>>>> pid:8865 ppid:2 flags:0x00004000
>>>>>>>>>>> [Mon Mar 11 14:10:08 2024] Call Trace:
>>>>>>>>>>> [Mon Mar 11 14:10:08 2024] <TASK>
>>>>>>>>>>> [Mon Mar 11 14:10:08 2024] __schedule+0x21b/0x550
>>>>>>>>>>> [Mon Mar 11 14:10:08 2024] schedule+0x2d/0x70
>>>>>>>>>>> [Mon Mar 11 14:10:08 2024] schedule_timeout+0x11f/0x160
>>>>>>>>>>> [Mon Mar 11 14:10:08 2024] ? select_idle_sibling+0x28/0x430
>>>>>>>>>>> [Mon Mar 11 14:10:08 2024] ? wake_affine+0x62/0x1f0
>>>>>>>>>>> [Mon Mar 11 14:10:08 2024] __wait_for_common+0x90/0x1d0
>>>>>>>>>>> [Mon Mar 11 14:10:08 2024] ? __pfx_schedule_timeout+0x10/0x10
>>>>>>>>>>> [Mon Mar 11 14:10:08 2024] __flush_workqueue+0x13a/0x3f0
>>>>>>>>>>> [Mon Mar 11 14:10:08 2024] nfsd4_shutdown_callback+0x49/0x120
>>>>>>>>>>> [nfsd]
>>>>>>>>>>> [Mon Mar 11 14:10:08 2024] ? nfsd4_cld_remove+0x54/0x1d0 [nfsd]
>>>>>>>>>>> [Mon Mar 11 14:10:08 2024] ?
>>>>>>>>>>> nfsd4_return_all_client_layouts+0xc4/0xf0 [nfsd]
>>>>>>>>>>> [Mon Mar 11 14:10:08 2024] ? nfsd4_shutdown_copy+0x68/0xc0 [nfsd]
>>>>>>>>>>> [Mon Mar 11 14:10:08 2024] __destroy_client+0x1f3/0x290 [nfsd]
>>>>>>>>>>> [Mon Mar 11 14:10:08 2024] nfsd4_exchange_id+0x75f/0x770 [nfsd]
>>>>>>>>>>> [Mon Mar 11 14:10:08 2024] ? nfsd4_decode_opaque+0x3a/0x90 [nfsd]
>>>>>>>>>>> [Mon Mar 11 14:10:08 2024] nfsd4_proc_compound+0x44b/0x700 [nfsd]
>>>>>>>>>>> [Mon Mar 11 14:10:08 2024] nfsd_dispatch+0x94/0x1c0 [nfsd]
>>>>>>>>>>> [Mon Mar 11 14:10:08 2024] svc_process_common+0x2ec/0x660
>>>>>>>>>>> [sunrpc]
>>>>>>>>>>> [Mon Mar 11 14:10:08 2024] ? __pfx_nfsd_dispatch+0x10/0x10 [nfsd]
>>>>>>>>>>> [Mon Mar 11 14:10:08 2024] ? __pfx_nfsd+0x10/0x10 [nfsd]
>>>>>>>>>>> [Mon Mar 11 14:10:08 2024] svc_process+0x12d/0x170 [sunrpc]
>>>>>>>>>>> [Mon Mar 11 14:10:08 2024] nfsd+0x84/0xb0 [nfsd]
>>>>>>>>>>> [Mon Mar 11 14:10:08 2024] kthread+0xdd/0x100
>>>>>>>>>>> [Mon Mar 11 14:10:08 2024] ? __pfx_kthread+0x10/0x10
>>>>>>>>>>> [Mon Mar 11 14:10:08 2024] ret_from_fork+0x29/0x50
>>>>>>>>>>> [Mon Mar 11 14:10:08 2024] </TASK>
>>>>>>>>>>> [Mon Mar 11 14:10:08 2024] INFO: task nfsd:8866 blocked for
>>>>>>>>>>> more than 122 seconds.
>>>>>>>>>>> [Mon Mar 11 14:10:08 2024] Not tainted
>>>>>>>>>>> 5.14.0-419.el9.x86_64 #1
>>>>>>>>>>> [Mon Mar 11 14:10:08 2024] "echo 0 >
>>>>>>>>>>> /proc/sys/kernel/hung_task_timeout_secs" disables this message.
>>>>>>>>>>> [Mon Mar 11 14:10:08 2024]task:nfsd state:D stack:0
>>>>>>>>>>> pid:8866 ppid:2 flags:0x00004000
>>>>>>>>>>> [Mon Mar 11 14:10:08 2024] Call Trace:
>>>>>>>>>>> [Mon Mar 11 14:10:08 2024] <TASK>
>>>>>>>>>>> [Mon Mar 11 14:10:08 2024] __schedule+0x21b/0x550
>>>>>>>>>>> [Mon Mar 11 14:10:08 2024] schedule+0x2d/0x70
>>>>>>>>>>> [Mon Mar 11 14:10:08 2024] schedule_timeout+0x11f/0x160
>>>>>>>>>>> [Mon Mar 11 14:10:08 2024] ? select_idle_sibling+0x28/0x430
>>>>>>>>>>> [Mon Mar 11 14:10:08 2024] ? tcp_recvmsg+0x196/0x210
>>>>>>>>>>> [Mon Mar 11 14:10:08 2024] ? wake_affine+0x62/0x1f0
>>>>>>>>>>> [Mon Mar 11 14:10:08 2024] __wait_for_common+0x90/0x1d0
>>>>>>>>>>> [Mon Mar 11 14:10:08 2024] ? __pfx_schedule_timeout+0x10/0x10
>>>>>>>>>>> [Mon Mar 11 14:10:08 2024] __flush_workqueue+0x13a/0x3f0
>>>>>>>>>>> [Mon Mar 11 14:10:08 2024] nfsd4_destroy_session+0x1a4/0x240
>>>>>>>>>>> [nfsd]
>>>>>>>>>>> [Mon Mar 11 14:10:08 2024] nfsd4_proc_compound+0x44b/0x700 [nfsd]
>>>>>>>>>>> [Mon Mar 11 14:10:08 2024] nfsd_dispatch+0x94/0x1c0 [nfsd]
>>>>>>>>>>> [Mon Mar 11 14:10:08 2024] svc_process_common+0x2ec/0x660
>>>>>>>>>>> [sunrpc]
>>>>>>>>>>> [Mon Mar 11 14:10:08 2024] ? __pfx_nfsd_dispatch+0x10/0x10 [nfsd]
>>>>>>>>>>> [Mon Mar 11 14:10:08 2024] ? __pfx_nfsd+0x10/0x10 [nfsd]
>>>>>>>>>>> [Mon Mar 11 14:10:08 2024] svc_process+0x12d/0x170 [sunrpc]
>>>>>>>>>>> [Mon Mar 11 14:10:08 2024] nfsd+0x84/0xb0 [nfsd]
>>>>>>>>>>> [Mon Mar 11 14:10:08 2024] kthread+0xdd/0x100
>>>>>>>>>>> [Mon Mar 11 14:10:08 2024] ? __pfx_kthread+0x10/0x10
>>>>>>>>>>> [Mon Mar 11 14:10:08 2024] ret_from_fork+0x29/0x50
>>>>>>>>>>> [Mon Mar 11 14:10:08 2024] </TASK>
>>>>>>>>>>>
>>>>>>>>>> The above threads are trying to flush the workqueue, so that probably
>>>>>>>>>> means that they are stuck waiting on a workqueue job to finish.
>>>>>>>>>>> The above is repeated a few times, and then this warning is
>>>>>>>>>>> also logged:
>>>>>>>>>>> [Mon Mar 11 14:12:04 2024] ------------[ cut here ]------------
>>>>>>>>>>> [Mon Mar 11 14:12:04 2024] WARNING: CPU: 39 PID: 8844 at
>>>>>>>>>>> fs/nfsd/nfs4state.c:4919 nfsd_break_deleg_cb+0x170/0x190 [nfsd]
>>>>>>>>>>> [Mon Mar 11 14:12:05 2024] Modules linked in: nfsv4
>>>>>>>>>>> dns_resolver nfs fscache netfs rpcsec_gss_krb5 rpcrdma rdma_cm
>>>>>>>>>>> iw_cm ib_cm ib_core binfmt_misc bonding tls rfkill nft_counter nft_ct
>>>>>>>>>>> nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_reject_inet
>>>>>>>>>>> nf_reject_ipv4 nf_reject_ipv6 nft_reject nf_tables nfnetlink vfat
>>>>>>>>>>> fat dm_thin_pool dm_persistent_data dm_bio_prison dm_bufio l
>>>>>>>>>>> ibcrc32c dm_service_time dm_multipath intel_rapl_msr
>>>>>>>>>>> intel_rapl_common intel_uncore_frequency
>>>>>>>>>>> intel_uncore_frequency_common isst_if_common skx_edac nfit
>>>>>>>>>>> libnvdimm ipmi_ssif x86_pkg_temp
>>>>>>>>>>> _thermal intel_powerclamp coretemp kvm_intel kvm irqbypass
>>>>>>>>>>> dcdbas rapl intel_cstate mgag200 i2c_algo_bit drm_shmem_helper
>>>>>>>>>>> dell_smbios drm_kms_helper dell_wmi_descriptor wmi_bmof intel_u
>>>>>>>>>>> ncore syscopyarea pcspkr sysfillrect mei_me sysimgblt acpi_ipmi
>>>>>>>>>>> mei fb_sys_fops i2c_i801 ipmi_si intel_pch_thermal lpc_ich
>>>>>>>>>>> ipmi_devintf i2c_smbus ipmi_msghandler joydev acpi_power_meter
>>>>>>>>>>> nfsd auth_rpcgss nfs_acl drm lockd grace fuse sunrpc ext4
>>>>>>>>>>> mbcache jbd2 sd_mod sg lpfc
>>>>>>>>>>> [Mon Mar 11 14:12:05 2024] nvmet_fc nvmet nvme_fc nvme_fabrics
>>>>>>>>>>> crct10dif_pclmul ahci libahci crc32_pclmul nvme_core crc32c_intel
>>>>>>>>>>> ixgbe megaraid_sas libata nvme_common ghash_clmulni_int
>>>>>>>>>>> el t10_pi wdat_wdt scsi_transport_fc mdio wmi dca dm_mirror
>>>>>>>>>>> dm_region_hash dm_log dm_mod
>>>>>>>>>>> [Mon Mar 11 14:12:05 2024] CPU: 39 PID: 8844 Comm: nfsd Not
>>>>>>>>>>> tainted 5.14.0-419.el9.x86_64 #1
>>>>>>>>>>> [Mon Mar 11 14:12:05 2024] Hardware name: Dell Inc. PowerEdge
>>>>>>>>>>> R740/00WGD1, BIOS 2.20.1 09/13/2023
>>>>>>>>>>> [Mon Mar 11 14:12:05 2024] RIP:
>>>>>>>>>>> 0010:nfsd_break_deleg_cb+0x170/0x190 [nfsd]
>>>>>>>>>>> [Mon Mar 11 14:12:05 2024] Code: a6 95 c5 f3 e9 ff fe ff ff 48
>>>>>>>>>>> 89 df be 01 00 00 00 e8 34 b5 13 f4 48 8d bb 98 00 00 00 e8 c8 f9
>>>>>>>>>>> 00 00 84 c0 0f 85 2e ff ff ff <0f> 0b e9 27 ff ff ff be
>>>>>>>>>>> 02 00 00 00 48 89 df e8 0c b5 13 f4 e9 01
>>>>>>>>>>> [Mon Mar 11 14:12:05 2024] RSP: 0018:ffff9929e0bb7b80 EFLAGS:
>>>>>>>>>>> 00010246
>>>>>>>>>>> [Mon Mar 11 14:12:05 2024] RAX: 0000000000000000 RBX:
>>>>>>>>>>> ffff8ada51930900 RCX: 0000000000000024
>>>>>>>>>>> [Mon Mar 11 14:12:05 2024] RDX: ffff8ada519309c8 RSI:
>>>>>>>>>>> ffff8ad582933c00 RDI: 0000000000002000
>>>>>>>>>>> [Mon Mar 11 14:12:05 2024] RBP: ffff8ad46bf21574 R08:
>>>>>>>>>>> ffff9929e0bb7b48 R09: 0000000000000000
>>>>>>>>>>> [Mon Mar 11 14:12:05 2024] R10: ffff8aec859a2948 R11:
>>>>>>>>>>> 0000000000000000 R12: ffff8ad6f497c360
>>>>>>>>>>> [Mon Mar 11 14:12:05 2024] R13: ffff8ad46bf21560 R14:
>>>>>>>>>>> ffff8ae5942e0b10 R15: ffff8ad6f497c360
>>>>>>>>>>> [Mon Mar 11 14:12:05 2024] FS: 0000000000000000(0000)
>>>>>>>>>>> GS:ffff8b031fcc0000(0000) knlGS:0000000000000000
>>>>>>>>>>> [Mon Mar 11 14:12:05 2024] CS: 0010 DS: 0000 ES: 0000 CR0:
>>>>>>>>>>> 0000000080050033
>>>>>>>>>>> [Mon Mar 11 14:12:05 2024] CR2: 00007fafe2060744 CR3:
>>>>>>>>>>> 00000018e58de006 CR4: 00000000007706e0
>>>>>>>>>>> [Mon Mar 11 14:12:05 2024] DR0: 0000000000000000 DR1:
>>>>>>>>>>> 0000000000000000 DR2: 0000000000000000
>>>>>>>>>>> [Mon Mar 11 14:12:05 2024] DR3: 0000000000000000 DR6:
>>>>>>>>>>> 00000000fffe0ff0 DR7: 0000000000000400
>>>>>>>>>>> [Mon Mar 11 14:12:05 2024] PKRU: 55555554
>>>>>>>>>>> [Mon Mar 11 14:12:05 2024] Call Trace:
>>>>>>>>>>> [Mon Mar 11 14:12:05 2024] <TASK>
>>>>>>>>>>> [Mon Mar 11 14:12:05 2024] ? show_trace_log_lvl+0x1c4/0x2df
>>>>>>>>>>> [Mon Mar 11 14:12:05 2024] ? show_trace_log_lvl+0x1c4/0x2df
>>>>>>>>>>> [Mon Mar 11 14:12:05 2024] ? __break_lease+0x16f/0x5f0
>>>>>>>>>>> [Mon Mar 11 14:12:05 2024] ? nfsd_break_deleg_cb+0x170/0x190
>>>>>>>>>>> [nfsd]
>>>>>>>>>>> [Mon Mar 11 14:12:05 2024] ? __warn+0x81/0x110
>>>>>>>>>>> [Mon Mar 11 14:12:05 2024] ? nfsd_break_deleg_cb+0x170/0x190
>>>>>>>>>>> [nfsd]
>>>>>>>>>>> [Mon Mar 11 14:12:05 2024] ? report_bug+0x10a/0x140
>>>>>>>>>>> [Mon Mar 11 14:12:05 2024] ? handle_bug+0x3c/0x70
>>>>>>>>>>> [Mon Mar 11 14:12:05 2024] ? exc_invalid_op+0x14/0x70
>>>>>>>>>>> [Mon Mar 11 14:12:05 2024] ? asm_exc_invalid_op+0x16/0x20
>>>>>>>>>>> [Mon Mar 11 14:12:05 2024] ? nfsd_break_deleg_cb+0x170/0x190
>>>>>>>>>>> [nfsd]
>>>>>>>>>>> [Mon Mar 11 14:12:05 2024] __break_lease+0x16f/0x5f0
>>>>>>>>>>> [Mon Mar 11 14:12:05 2024] ?
>>>>>>>>>>> nfsd_file_lookup_locked+0x117/0x160 [nfsd]
>>>>>>>>>>> [Mon Mar 11 14:12:05 2024] ? list_lru_del+0x101/0x150
>>>>>>>>>>> [Mon Mar 11 14:12:05 2024] nfsd_file_do_acquire+0x790/0x830
>>>>>>>>>>> [nfsd]
>>>>>>>>>>> [Mon Mar 11 14:12:05 2024] nfs4_get_vfs_file+0x315/0x3a0 [nfsd]
>>>>>>>>>>> [Mon Mar 11 14:12:05 2024] nfsd4_process_open2+0x430/0xa30 [nfsd]
>>>>>>>>>>> [Mon Mar 11 14:12:05 2024] ? fh_verify+0x297/0x2f0 [nfsd]
>>>>>>>>>>> [Mon Mar 11 14:12:05 2024] nfsd4_open+0x3ce/0x4b0 [nfsd]
>>>>>>>>>>> [Mon Mar 11 14:12:05 2024] nfsd4_proc_compound+0x44b/0x700 [nfsd]
>>>>>>>>>>> [Mon Mar 11 14:12:05 2024] nfsd_dispatch+0x94/0x1c0 [nfsd]
>>>>>>>>>>> [Mon Mar 11 14:12:05 2024] svc_process_common+0x2ec/0x660
>>>>>>>>>>> [sunrpc]
>>>>>>>>>>> [Mon Mar 11 14:12:05 2024] ? __pfx_nfsd_dispatch+0x10/0x10 [nfsd]
>>>>>>>>>>> [Mon Mar 11 14:12:05 2024] ? __pfx_nfsd+0x10/0x10 [nfsd]
>>>>>>>>>>> [Mon Mar 11 14:12:05 2024] svc_process+0x12d/0x170 [sunrpc]
>>>>>>>>>>> [Mon Mar 11 14:12:05 2024] nfsd+0x84/0xb0 [nfsd]
>>>>>>>>>>> [Mon Mar 11 14:12:05 2024] kthread+0xdd/0x100
>>>>>>>>>>> [Mon Mar 11 14:12:05 2024] ? __pfx_kthread+0x10/0x10
>>>>>>>>>>> [Mon Mar 11 14:12:05 2024] ret_from_fork+0x29/0x50
>>>>>>>>>>> [Mon Mar 11 14:12:05 2024] </TASK>
>>>>>>>>>>> [Mon Mar 11 14:12:05 2024] ---[ end trace 7a039e17443dc651 ]---
>>>>>>>>>> This is probably this WARN in nfsd_break_one_deleg:
>>>>>>>>>>
>>>>>>>>>> WARN_ON_ONCE(!nfsd4_run_cb(&dp->dl_recall));
>>>>>>>>>>
>>>>>>>>>> It means that a delegation break callback to the client couldn't be
>>>>>>>>>> queued to the workqueue, and so it didn't run.
>>>>>>>>>>
>>>>>>>>>>> Could this be the same issue as described
>>>>>>>>>>> here:https://lore.kernel.org/linux-nfs/af0ec881-5ebf-4feb-98ae-3ed2a77f86f1@oracle.com/
>>>>>>>>>>> ?
>>>>>>>>>> Yes, most likely the same problem.
>>>>>>>>> If I read that thread correctly, this issue was introduced between
>>>>>>>>> 6.1.63 and 6.6.3? Is it possible the EL9 5.14.0-362.18.1.el9_3
>>>>>>>>> backported these changes, or were we hitting some other bug with that
>>>>>>>>> version? It seems the 6.1.x kernel is not affected? If so, that
>>>>>>>>> would be
>>>>>>>>> the recommended kernel to run?
>>>>>>>> Anything is possible. We have to identify the problem first.
>>>>>>>>>>> As described in that thread, I've tried to obtain the requested
>>>>>>>>>>> information.
>>>>>>>>>>>
>>>>>>>>>>> Is it possible this is the issue that was fixed by the patches
>>>>>>>>>>> described
>>>>>>>>>>> here?https://lore.kernel.org/linux-nfs/2024022054-cause-suffering-eae8@gregkh/
>>>>>>>>>>>
>>>>>>>>>> Doubtful. Those are targeted toward a different set of issues.
>>>>>>>>>>
>>>>>>>>>> If you're willing, I do have some patches queued up for CentOS here
>>>>>>>>>> that
>>>>>>>>>> fix some backchannel problems that could be related. I'm mainly
>>>>>>>>>> waiting
>>>>>>>>>> on Chuck to send these to Linus and then we'll likely merge them into
>>>>>>>>>> CentOS soon afterward:
>>>>>>>>>>
>>>>>>>>>> https://gitlab.com/redhat/centos-stream/src/kernel/centos-stream-9/-/merge_requests/3689
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>> If you can send me a patch file, I can rebuild the C9S kernel with that
>>>>>>>>> patch and run it. It can take a while for the bug to trigger as I
>>>>>>>>> believe it seems to be very workload dependent (we were running very
>>>>>>>>> stable for months and now hit this bug every other week).
>>>>>>>>>
>>>>>>>>>
>>>>>>>> It's probably simpler to just pull down the build artifacts for that MR.
>>>>>>>> You have to drill down through the CI for it, but they are here:
>>>>>>>>
>>>>>>>> https://s3.amazonaws.com/arr-cki-prod-trusted-artifacts/index.html?prefix=trusted-artifacts/1194300175/publish_x86_64/6278921877/artifacts/
>>>>>>>>
>>>>>>>>
>>>>>>>> There's even a repo file you can install on the box to pull them down.
>>>>>>> We installed this kernel on the server 3 days ago. Today, a user
>>>>>>> informed us that their screen was black after logging in. Similar to
>>>>>>> other occurrences of this issue, the mount command on the client was
>>>>>>> hung. But in contrast to the other times, there were no messages in
>>>>>>> the logs kernel logs on the server. Even restarting the client does
>>>>>>> not resolve the issue.
>>>>> Ok, so you rebooted the client and it's still unable to mount? That
>>>>> sounds like a server problem if so.
>>>>>
>>>>> Are both client and server running the same kernel?
>>>> No, the server runs 5.14.0-427.3689_1194299994.el9 and the client
>>>> 5.14.0-362.18.1.el9_3.
>>> Ok.
>>>
>>>>>>> Something still seems to be wrong on the server though. When I look at
>>>>>>> the directories under /proc/fs/nfsd/clients, there's still a directory
>>>>>>> for the specific client, even though it's no longer running:
>>>>>>>
>>>>>>> # cat 155/info
>>>>>>> clientid: 0xc8edb7f65f4a9ad
>>>>>>> address: "10.87.31.152:819"
>>>>>>> status: confirmed
>>>>>>> seconds from last renew: 33163
>>>>>>> name: "Linux NFSv4.2 bersalis.esat.kuleuven.be"
>>>>>>> minor version: 2
>>>>>>> Implementation domain: "kernel.org"
>>>>>>> Implementation name: "Linux 5.14.0-362.18.1.el9_3.0.1.x86_64 #1 SMP
>>>>>>> PREEMPT_DYNAMIC Sun Feb 11 13:49:23 UTC 2024 x86_64"
>>>>>>> Implementation time: [0, 0]
>>>>>>> callback state: DOWN
>>>>>>> callback address: 10.87.31.152:0
>>>>>>>
>>>>> If you just shut down the client, the server won't immediately purge its
>>>>> record. In fact, assuming you're running the same kernel on the server,
>>>>> it won't purge the client record until there is a conflicting request
>>>>> for its state.
>>>> Is there a way to force such a conflicting request (to get the client
>>>> record to purge)?
>>> From the server or a different client, you can try opening the inodes
>>> that the stuck client is holding open. If you open them for write, that
>>> may trigger the server to kick out the old client record.
>>>
>>> The problem is that they are disconnected dentries, so finding them to
>>> open via path may be difficult...
>>
>> I've located the file that matches one of these inodes. When I go to the
>>
>> location of the file on another NFS client and touch the file, the touch
>> command
>>
>> just hangs.
>>
>> So I assume the server is trying to recall the delegation from the
>> dis-functional client?
>>
>> When I run tcpdump on the dis-functional client, I see the
>> CREATE_SESSION / NFS4ERR_DELAY messages, but nothing to indicate the
>> server wants to revoke a delegation.
>>
>
> Right, according to the client record above, the callback channel is
> DOWN, so the server can't communicate with the client. Given that there
> are 33000s since the last lease renewal, the server should kick the
> client out that is blocking other activity, but that doesn't seem to be
> happening here.
>
>
>> The entry for the dis-functional client is not removed on the server.
>>
>> I there anything else I can do to provide more information about this
>> situation?
>>
>>
>
> The main function that handles the CREATE_SESSION call is
> nfsd4_create_session. It's somewhat complex and there are a number of
> reasons that function could return NFS4ERR_DELAY (aka nfserr_jukebox in
> the kernel code).
>
> What I'd do at this point is turn up tracepoints and see whether they
> shed any light on what's going wrong to make it continually delay your
> CREATE_SESSION calls. There aren't a lot of tracepoints in that code,
> however, so it may not show much.
>
> In the absence of that, you can try to use bpftrace or something similar
> to debug what's happening in that function.
>
>>
>>
>>>>>> The nfsdclnts command for this client shows the following delegations:
>>>>>>
>>>>>> # nfsdclnts -f 155/states -t all
>>>>>> Inode number | Type | Access | Deny | ip address | Filename
>>>>>> 169346743 | open | r- | -- | 10.87.31.152:819 |
>>>>>> disconnected dentry
>>>>>> 169346743 | deleg | r | | 10.87.31.152:819 |
>>>>>> disconnected dentry
>>>>>> 169346746 | open | r- | -- | 10.87.31.152:819 |
>>>>>> disconnected dentry
>>>>>> 169346746 | deleg | r | | 10.87.31.152:819 |
>>>>>> disconnected dentry
>>>>>>
>>>>>> I see a lot of recent patches regarding directory delegations. Could
>>>>>> this be related to this?
>>>>>>
>>>>>> Will a 5.14.0-362.18.1.el9_3.0.1 kernel try to use a directory delegation?
>>>>>>
>>>>>>
>>>>> No. Directory delegations are a new feature that's still under
>>>>> development. They use some of the same machinery as file delegations,
>>>>> but they wouldn't be a factor here.
>>>>>
>>>>>>> The system seems to have identified that the client is no longer
>>>>>>> reachable, but the client entry does not go away. When a mount was
>>>>>>> hanging on the client, there would be two directories in clients for
>>>>>>> the same client. Killing the mount command clears up the second entry.
>>>>>>>
>>>>>>> Even after running conntrack -D on the server to remove the tcp
>>>>>>> connection from the conntrack table, the entry doesn't go away and the
>>>>>>> client still can not mount anything from the server.
>>>>>>>
>>>>>>> A tcpdump on the client while a mount was running logged the following
>>>>>>> messages over and over again:
>>>>>>>
>>>>>>> request:
>>>>>>>
>>>>>>> Frame 1: 378 bytes on wire (3024 bits), 378 bytes captured (3024 bits)
>>>>>>> Ethernet II, Src: HP_19:7d:4b (e0:73:e7:19:7d:4b), Dst:
>>>>>>> ArubaaHe_f9:8e:00 (88:3a:30:f9:8e:00)
>>>>>>> Internet Protocol Version 4, Src: 10.87.31.152, Dst: 10.86.18.14
>>>>>>> Transmission Control Protocol, Src Port: 932, Dst Port: 2049, Seq: 1,
>>>>>>> Ack: 1, Len: 312
>>>>>>> Remote Procedure Call, Type:Call XID:0x1d3220c4
>>>>>>> Network File System
>>>>>>> [Program Version: 4]
>>>>>>> [V4 Procedure: COMPOUND (1)]
>>>>>>> GSS Data, Ops(1): CREATE_SESSION
>>>>>>> Length: 152
>>>>>>> GSS Sequence Number: 76
>>>>>>> Tag: <EMPTY>
>>>>>>> minorversion: 2
>>>>>>> Operations (count: 1): CREATE_SESSION
>>>>>>> [Main Opcode: CREATE_SESSION (43)]
>>>>>>> GSS Checksum:
>>>>>>> 00000028040404ffffffffff000000002c19055f1f8d442d594c13849628affc2797cbb2…
>>>>>>> GSS Token Length: 40
>>>>>>> GSS-API Generic Security Service Application Program Interface
>>>>>>> krb5_blob:
>>>>>>> 040404ffffffffff000000002c19055f1f8d442d594c13849628affc2797cbb23fa080b0…
>>>>>>>
>>>>>>> response:
>>>>>>>
>>>>>>> Frame 2: 206 bytes on wire (1648 bits), 206 bytes captured (1648 bits)
>>>>>>> Ethernet II, Src: ArubaaHe_f9:8e:00 (88:3a:30:f9:8e:00), Dst:
>>>>>>> HP_19:7d:4b (e0:73:e7:19:7d:4b)
>>>>>>> Internet Protocol Version 4, Src: 10.86.18.14, Dst: 10.87.31.152
>>>>>>> Transmission Control Protocol, Src Port: 2049, Dst Port: 932, Seq: 1,
>>>>>>> Ack: 313, Len: 140
>>>>>>> Remote Procedure Call, Type:Reply XID:0x1d3220c4
>>>>>>> Network File System
>>>>>>> [Program Version: 4]
>>>>>>> [V4 Procedure: COMPOUND (1)]
>>>>>>> GSS Data, Ops(1): CREATE_SESSION(NFS4ERR_DELAY)
>>>>>>> Length: 24
>>>>>>> GSS Sequence Number: 76
>>>>>>> Status: NFS4ERR_DELAY (10008)
>>>>>>> Tag: <EMPTY>
>>>>>>> Operations (count: 1)
>>>>>>> [Main Opcode: CREATE_SESSION (43)]
>>>>>>> GSS Checksum:
>>>>>>> 00000028040405ffffffffff000000000aa742d0798deaad1a8aa2d7c3a91bf4f6274222…
>>>>>>> GSS Token Length: 40
>>>>>>> GSS-API Generic Security Service Application Program Interface
>>>>>>> krb5_blob:
>>>>>>> 040405ffffffffff000000000aa742d0798deaad1a8aa2d7c3a91bf4f627422226d74923…
>>>>>>>
>>>>>>> I was hoping that giving the client a different IP address would
>>>>>>> resolve the issue for this client, but it didn't. Even though the
>>>>>>> client had a new IP address (hostname was kept the same), it failed to
>>>>>>> mount anything from the server.
>>>>>>>
>>>>> Changing the IP address won't help. The client is probably using the
>>>>> same long-form client id as before, so the server still identifies the
>>>>> client even with the address change.
>>>> How is the client id determined? Will changing the hostname of the
>>>> client trigger a change of the client id?
>>> In the client record you showed a bit above, there is a "name" field:
>>>
>>> name: "Linux NFSv4.2 bersalis.esat.kuleuven.be"
>>>
>>> That's the string the server uses to uniquely identify the client. So
>>> yes, changing the hostname should change that string.
>>>
>>>>> Unfortunately, the cause of an NFS4ERR_DELAY error is tough to guess.
>>>>> The client is expected to back off and retry, so if the server keeps
>>>>> returning that repeatedly, then a hung mount command is expected.
>>>>>
>>>>> The question is why the server would keep returning DELAY. A lot of
>>>>> different problems ranging from memory allocation issues to protocol
>>>>> problems can result in that error. You may want to check the NFS server
>>>>> and see if anything was logged there.
>>>> There are no messages in the system logs that indicate any sort of
>>>> memory issue. We also increased the min_kbytes_free sysctl to 2G on the
>>>> server before we restarted it with the newer kernel.
>>> Ok, I didn't expect to see anything like that, but it was a possibility.
>>>
>>>>> This is on a CREATE_SESSION call, so I wonder if the record held by the
>>>>> (courteous) server is somehow blocking the attempt to reestablish the
>>>>> session?
>>>>>
>>>>> Do you have a way to reproduce this? Since this is a centos kernel, you
>>>>> could follow the page here to open a bug:
>>>> Unfortunately we haven't found a reliable way to reproduce it. But we do
>>>> seem to trigger it more and more lately.
>>>>
>>>>
>>> Bummer, ok. Let us know if you figure out a way to reproduce it.
>>>
>>>>> https://wiki.centos.org/ReportBugs.html
>>>>>
>>>>>
>>>>>>> I created another dump of the workqueues and worker pools on the server:
>>>>>>>
>>>>>>> [Mon Mar 18 14:59:33 2024] Showing busy workqueues and worker pools:
>>>>>>> [Mon Mar 18 14:59:33 2024] workqueue events: flags=0x0
>>>>>>> [Mon Mar 18 14:59:33 2024] pwq 54: cpus=27 node=1 flags=0x0 nice=0
>>>>>>> active=1/256 refcnt=2
>>>>>>> [Mon Mar 18 14:59:33 2024] pending: drm_fb_helper_damage_work
>>>>>>> [drm_kms_helper]
>>>>>>> [Mon Mar 18 14:59:33 2024] workqueue events_power_efficient: flags=0x80
>>>>>>> [Mon Mar 18 14:59:33 2024] pwq 54: cpus=27 node=1 flags=0x0 nice=0
>>>>>>> active=1/256 refcnt=2
>>>>>>> [Mon Mar 18 14:59:33 2024] pending: fb_flashcursor
>>>>>>> [Mon Mar 18 14:59:33 2024] workqueue mm_percpu_wq: flags=0x8
>>>>>>> [Mon Mar 18 14:59:33 2024] pwq 54: cpus=27 node=1 flags=0x0 nice=0
>>>>>>> active=1/256 refcnt=3
>>>>>>> [Mon Mar 18 14:59:33 2024] pending: lru_add_drain_per_cpu BAR(362)
>>>>>>> [Mon Mar 18 14:59:33 2024] workqueue kblockd: flags=0x18
>>>>>>> [Mon Mar 18 14:59:33 2024] pwq 55: cpus=27 node=1 flags=0x0 nice=-20
>>>>>>> active=1/256 refcnt=2
>>>>>>> [Mon Mar 18 14:59:33 2024] pending: blk_mq_timeout_work
>>>>>>>
>>>>>>>
>>>>>>> In contrast to last time, it doesn't show anything regarding nfs this
>>>>>>> time.
>>>>>>>
>>>>>>> I also tried the suggestion from Dai Ngo (echo 3 >
>>>>>>> /proc/sys/vm/drop_caches), but that didn't seem to make any difference.
>>>>>>>
>>>>>>> We haven't restarted the server yet as it seems the impact seems to
>>>>>>> affect fewer clients that before. Is there anything we can run on the
>>>>>>> server to further debug this?
>>>>>>>
>>>>>>> In the past, the issue seemed to deteriorate rapidly and resulted in
>>>>>>> issues for almost all clients after about 20 minutes. This time the
>>>>>>> impact seems to be less, but it's not gone.
>>>>>>>
>>>>>>> How can we force the NFS server to forget about a specific client? I
>>>>>>> haven't tried to restart the nfs service yet as I'm afraid it will
>>>>>>> fail to stop as before.
>>>>>>>
>>>>> Not with that kernel. There are some new administrative interfaces that
>>>>> might allow that in the future, but they were just merged upstream and
>>>>> aren't in that kernel.
>>>>>
>>>>> --
>>>>> Jeff Layton <jlayton@kernel.org>
>>
>
> --
> Jeff Layton <jlayton@kernel.org>
For years we’ve seen hangs with NFS4 on one particular server. The symptoms look precisely like this. Rebooting the client doesn’t help. Echo expire hangs. I haven’t been able to get a usable backtrace. What’s characteristic of this server is that delegations are off but there’s a high rate of locking and unlocking. Could that cause the same problem?
I’d be happy to collect data if you can suggest a way. Typical traces aren’t going to be useful unless they are very specific, because this is a very busy server, which we can’t take down for debugging.
>
^ permalink raw reply [flat|nested] 26+ messages in thread
* Re: nfsd hangs and nfsd_break_deleg_cb+0x170/0x190 warning
2024-03-19 21:42 ` Dai Ngo
@ 2024-03-20 19:41 ` Rik Theys
2024-03-21 20:48 ` Jeff Layton
0 siblings, 1 reply; 26+ messages in thread
From: Rik Theys @ 2024-03-20 19:41 UTC (permalink / raw)
To: Dai Ngo, Jeff Layton, Linux Nfs
Hi,
On 3/19/24 22:42, Dai Ngo wrote:
>
> On 3/19/24 12:41 PM, Rik Theys wrote:
>> Hi,
>>
>> On 3/19/24 18:09, Dai Ngo wrote:
>>>
>>> On 3/19/24 12:58 AM, Rik Theys wrote:
>>>> Hi,
>>>>
>>>> On 3/18/24 22:54, Jeff Layton wrote:
>>>>> On Mon, 2024-03-18 at 22:15 +0100, Rik Theys wrote:
>>>>>> Hi,
>>>>>>
>>>>>> On 3/18/24 21:21, Rik Theys wrote:
>>>>>>> Hi Jeff,
>>>>>>>
>>>>>>> On 3/12/24 13:47, Jeff Layton wrote:
>>>>>>>> On Tue, 2024-03-12 at 13:24 +0100, Rik Theys wrote:
>>>>>>>>> Hi Jeff,
>>>>>>>>>
>>>>>>>>> On 3/12/24 12:22, Jeff Layton wrote:
>>>>>>>>>> On Mon, 2024-03-11 at 19:43 +0100, Rik Theys wrote:
>>>>>>>>>>> Since a few weeks our Rocky Linux 9 NFS server has periodically
>>>>>>>>>>> logged hung nfsd tasks. The initial effect was that some
>>>>>>>>>>> clients
>>>>>>>>>>> could no longer access the NFS server. This got worse and worse
>>>>>>>>>>> (probably as more nfsd threads got blocked) and we had to
>>>>>>>>>>> restart
>>>>>>>>>>> the server. Restarting the server also failed as the NFS server
>>>>>>>>>>> service could no longer be stopped.
>>>>>>>>>>>
>>>>>>>>>>> The initial kernel we noticed this behavior on was
>>>>>>>>>>> kernel-5.14.0-362.18.1.el9_3.x86_64. Since then we've installed
>>>>>>>>>>> kernel-5.14.0-419.el9.x86_64 from CentOS Stream 9. The same
>>>>>>>>>>> issue
>>>>>>>>>>> happened again on this newer kernel version:
>>>>>>>> 419 is fairly up to date with nfsd changes. There are some
>>>>>>>> known bugs
>>>>>>>> around callbacks, and there is a draft MR in flight to fix it.
>>>>>>>>
>>>>>>>> What kernel were you on prior to 5.14.0-362.18.1.el9_3.x86_64 ?
>>>>>>>> If we
>>>>>>>> can bracket the changes around a particular version, then that
>>>>>>>> might
>>>>>>>> help identify the problem.
>>>>>>>>
>>>>>>>>>>> [Mon Mar 11 14:10:08 2024] Not tainted
>>>>>>>>>>> 5.14.0-419.el9.x86_64 #1
>>>>>>>>>>> [Mon Mar 11 14:10:08 2024] "echo 0 >
>>>>>>>>>>> /proc/sys/kernel/hung_task_timeout_secs" disables this message.
>>>>>>>>>>> [Mon Mar 11 14:10:08 2024]task:nfsd state:D
>>>>>>>>>>> stack:0
>>>>>>>>>>> pid:8865 ppid:2 flags:0x00004000
>>>>>>>>>>> [Mon Mar 11 14:10:08 2024] Call Trace:
>>>>>>>>>>> [Mon Mar 11 14:10:08 2024] <TASK>
>>>>>>>>>>> [Mon Mar 11 14:10:08 2024] __schedule+0x21b/0x550
>>>>>>>>>>> [Mon Mar 11 14:10:08 2024] schedule+0x2d/0x70
>>>>>>>>>>> [Mon Mar 11 14:10:08 2024] schedule_timeout+0x11f/0x160
>>>>>>>>>>> [Mon Mar 11 14:10:08 2024] ?
>>>>>>>>>>> select_idle_sibling+0x28/0x430
>>>>>>>>>>> [Mon Mar 11 14:10:08 2024] ? wake_affine+0x62/0x1f0
>>>>>>>>>>> [Mon Mar 11 14:10:08 2024] __wait_for_common+0x90/0x1d0
>>>>>>>>>>> [Mon Mar 11 14:10:08 2024] ?
>>>>>>>>>>> __pfx_schedule_timeout+0x10/0x10
>>>>>>>>>>> [Mon Mar 11 14:10:08 2024] __flush_workqueue+0x13a/0x3f0
>>>>>>>>>>> [Mon Mar 11 14:10:08 2024]
>>>>>>>>>>> nfsd4_shutdown_callback+0x49/0x120
>>>>>>>>>>> [nfsd]
>>>>>>>>>>> [Mon Mar 11 14:10:08 2024] ?
>>>>>>>>>>> nfsd4_cld_remove+0x54/0x1d0 [nfsd]
>>>>>>>>>>> [Mon Mar 11 14:10:08 2024] ?
>>>>>>>>>>> nfsd4_return_all_client_layouts+0xc4/0xf0 [nfsd]
>>>>>>>>>>> [Mon Mar 11 14:10:08 2024] ?
>>>>>>>>>>> nfsd4_shutdown_copy+0x68/0xc0 [nfsd]
>>>>>>>>>>> [Mon Mar 11 14:10:08 2024] __destroy_client+0x1f3/0x290
>>>>>>>>>>> [nfsd]
>>>>>>>>>>> [Mon Mar 11 14:10:08 2024]
>>>>>>>>>>> nfsd4_exchange_id+0x75f/0x770 [nfsd]
>>>>>>>>>>> [Mon Mar 11 14:10:08 2024] ?
>>>>>>>>>>> nfsd4_decode_opaque+0x3a/0x90 [nfsd]
>>>>>>>>>>> [Mon Mar 11 14:10:08 2024]
>>>>>>>>>>> nfsd4_proc_compound+0x44b/0x700 [nfsd]
>>>>>>>>>>> [Mon Mar 11 14:10:08 2024] nfsd_dispatch+0x94/0x1c0 [nfsd]
>>>>>>>>>>> [Mon Mar 11 14:10:08 2024] svc_process_common+0x2ec/0x660
>>>>>>>>>>> [sunrpc]
>>>>>>>>>>> [Mon Mar 11 14:10:08 2024] ?
>>>>>>>>>>> __pfx_nfsd_dispatch+0x10/0x10 [nfsd]
>>>>>>>>>>> [Mon Mar 11 14:10:08 2024] ? __pfx_nfsd+0x10/0x10 [nfsd]
>>>>>>>>>>> [Mon Mar 11 14:10:08 2024] svc_process+0x12d/0x170
>>>>>>>>>>> [sunrpc]
>>>>>>>>>>> [Mon Mar 11 14:10:08 2024] nfsd+0x84/0xb0 [nfsd]
>>>>>>>>>>> [Mon Mar 11 14:10:08 2024] kthread+0xdd/0x100
>>>>>>>>>>> [Mon Mar 11 14:10:08 2024] ? __pfx_kthread+0x10/0x10
>>>>>>>>>>> [Mon Mar 11 14:10:08 2024] ret_from_fork+0x29/0x50
>>>>>>>>>>> [Mon Mar 11 14:10:08 2024] </TASK>
>>>>>>>>>>> [Mon Mar 11 14:10:08 2024] INFO: task nfsd:8866 blocked for
>>>>>>>>>>> more than 122 seconds.
>>>>>>>>>>> [Mon Mar 11 14:10:08 2024] Not tainted
>>>>>>>>>>> 5.14.0-419.el9.x86_64 #1
>>>>>>>>>>> [Mon Mar 11 14:10:08 2024] "echo 0 >
>>>>>>>>>>> /proc/sys/kernel/hung_task_timeout_secs" disables this message.
>>>>>>>>>>> [Mon Mar 11 14:10:08 2024]task:nfsd state:D
>>>>>>>>>>> stack:0
>>>>>>>>>>> pid:8866 ppid:2 flags:0x00004000
>>>>>>>>>>> [Mon Mar 11 14:10:08 2024] Call Trace:
>>>>>>>>>>> [Mon Mar 11 14:10:08 2024] <TASK>
>>>>>>>>>>> [Mon Mar 11 14:10:08 2024] __schedule+0x21b/0x550
>>>>>>>>>>> [Mon Mar 11 14:10:08 2024] schedule+0x2d/0x70
>>>>>>>>>>> [Mon Mar 11 14:10:08 2024] schedule_timeout+0x11f/0x160
>>>>>>>>>>> [Mon Mar 11 14:10:08 2024] ?
>>>>>>>>>>> select_idle_sibling+0x28/0x430
>>>>>>>>>>> [Mon Mar 11 14:10:08 2024] ? tcp_recvmsg+0x196/0x210
>>>>>>>>>>> [Mon Mar 11 14:10:08 2024] ? wake_affine+0x62/0x1f0
>>>>>>>>>>> [Mon Mar 11 14:10:08 2024] __wait_for_common+0x90/0x1d0
>>>>>>>>>>> [Mon Mar 11 14:10:08 2024] ?
>>>>>>>>>>> __pfx_schedule_timeout+0x10/0x10
>>>>>>>>>>> [Mon Mar 11 14:10:08 2024] __flush_workqueue+0x13a/0x3f0
>>>>>>>>>>> [Mon Mar 11 14:10:08 2024]
>>>>>>>>>>> nfsd4_destroy_session+0x1a4/0x240
>>>>>>>>>>> [nfsd]
>>>>>>>>>>> [Mon Mar 11 14:10:08 2024]
>>>>>>>>>>> nfsd4_proc_compound+0x44b/0x700 [nfsd]
>>>>>>>>>>> [Mon Mar 11 14:10:08 2024] nfsd_dispatch+0x94/0x1c0 [nfsd]
>>>>>>>>>>> [Mon Mar 11 14:10:08 2024] svc_process_common+0x2ec/0x660
>>>>>>>>>>> [sunrpc]
>>>>>>>>>>> [Mon Mar 11 14:10:08 2024] ?
>>>>>>>>>>> __pfx_nfsd_dispatch+0x10/0x10 [nfsd]
>>>>>>>>>>> [Mon Mar 11 14:10:08 2024] ? __pfx_nfsd+0x10/0x10 [nfsd]
>>>>>>>>>>> [Mon Mar 11 14:10:08 2024] svc_process+0x12d/0x170
>>>>>>>>>>> [sunrpc]
>>>>>>>>>>> [Mon Mar 11 14:10:08 2024] nfsd+0x84/0xb0 [nfsd]
>>>>>>>>>>> [Mon Mar 11 14:10:08 2024] kthread+0xdd/0x100
>>>>>>>>>>> [Mon Mar 11 14:10:08 2024] ? __pfx_kthread+0x10/0x10
>>>>>>>>>>> [Mon Mar 11 14:10:08 2024] ret_from_fork+0x29/0x50
>>>>>>>>>>> [Mon Mar 11 14:10:08 2024] </TASK>
>>>>>>>>>>>
>>>>>>>>>> The above threads are trying to flush the workqueue, so that
>>>>>>>>>> probably
>>>>>>>>>> means that they are stuck waiting on a workqueue job to finish.
>>>>>>>>>>> The above is repeated a few times, and then this warning is
>>>>>>>>>>> also logged:
>>>>>>>>>>> [Mon Mar 11 14:12:04 2024] ------------[ cut here
>>>>>>>>>>> ]------------
>>>>>>>>>>> [Mon Mar 11 14:12:04 2024] WARNING: CPU: 39 PID: 8844 at
>>>>>>>>>>> fs/nfsd/nfs4state.c:4919 nfsd_break_deleg_cb+0x170/0x190 [nfsd]
>>>>>>>>>>> [Mon Mar 11 14:12:05 2024] Modules linked in: nfsv4
>>>>>>>>>>> dns_resolver nfs fscache netfs rpcsec_gss_krb5 rpcrdma rdma_cm
>>>>>>>>>>> iw_cm ib_cm ib_core binfmt_misc bonding tls rfkill
>>>>>>>>>>> nft_counter nft_ct
>>>>>>>>>>> nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_reject_inet
>>>>>>>>>>> nf_reject_ipv4 nf_reject_ipv6 nft_reject nf_tables nfnetlink
>>>>>>>>>>> vfat
>>>>>>>>>>> fat dm_thin_pool dm_persistent_data dm_bio_prison dm_bufio l
>>>>>>>>>>> ibcrc32c dm_service_time dm_multipath intel_rapl_msr
>>>>>>>>>>> intel_rapl_common intel_uncore_frequency
>>>>>>>>>>> intel_uncore_frequency_common isst_if_common skx_edac nfit
>>>>>>>>>>> libnvdimm ipmi_ssif x86_pkg_temp
>>>>>>>>>>> _thermal intel_powerclamp coretemp kvm_intel kvm irqbypass
>>>>>>>>>>> dcdbas rapl intel_cstate mgag200 i2c_algo_bit drm_shmem_helper
>>>>>>>>>>> dell_smbios drm_kms_helper dell_wmi_descriptor wmi_bmof intel_u
>>>>>>>>>>> ncore syscopyarea pcspkr sysfillrect mei_me sysimgblt
>>>>>>>>>>> acpi_ipmi
>>>>>>>>>>> mei fb_sys_fops i2c_i801 ipmi_si intel_pch_thermal lpc_ich
>>>>>>>>>>> ipmi_devintf i2c_smbus ipmi_msghandler joydev acpi_power_meter
>>>>>>>>>>> nfsd auth_rpcgss nfs_acl drm lockd grace fuse sunrpc ext4
>>>>>>>>>>> mbcache jbd2 sd_mod sg lpfc
>>>>>>>>>>> [Mon Mar 11 14:12:05 2024] nvmet_fc nvmet nvme_fc
>>>>>>>>>>> nvme_fabrics
>>>>>>>>>>> crct10dif_pclmul ahci libahci crc32_pclmul nvme_core
>>>>>>>>>>> crc32c_intel
>>>>>>>>>>> ixgbe megaraid_sas libata nvme_common ghash_clmulni_int
>>>>>>>>>>> el t10_pi wdat_wdt scsi_transport_fc mdio wmi dca dm_mirror
>>>>>>>>>>> dm_region_hash dm_log dm_mod
>>>>>>>>>>> [Mon Mar 11 14:12:05 2024] CPU: 39 PID: 8844 Comm: nfsd Not
>>>>>>>>>>> tainted 5.14.0-419.el9.x86_64 #1
>>>>>>>>>>> [Mon Mar 11 14:12:05 2024] Hardware name: Dell Inc.
>>>>>>>>>>> PowerEdge
>>>>>>>>>>> R740/00WGD1, BIOS 2.20.1 09/13/2023
>>>>>>>>>>> [Mon Mar 11 14:12:05 2024] RIP:
>>>>>>>>>>> 0010:nfsd_break_deleg_cb+0x170/0x190 [nfsd]
>>>>>>>>>>> [Mon Mar 11 14:12:05 2024] Code: a6 95 c5 f3 e9 ff fe ff
>>>>>>>>>>> ff 48
>>>>>>>>>>> 89 df be 01 00 00 00 e8 34 b5 13 f4 48 8d bb 98 00 00 00 e8
>>>>>>>>>>> c8 f9
>>>>>>>>>>> 00 00 84 c0 0f 85 2e ff ff ff <0f> 0b e9 27 ff ff ff be
>>>>>>>>>>> 02 00 00 00 48 89 df e8 0c b5 13 f4 e9 01
>>>>>>>>>>> [Mon Mar 11 14:12:05 2024] RSP: 0018:ffff9929e0bb7b80
>>>>>>>>>>> EFLAGS:
>>>>>>>>>>> 00010246
>>>>>>>>>>> [Mon Mar 11 14:12:05 2024] RAX: 0000000000000000 RBX:
>>>>>>>>>>> ffff8ada51930900 RCX: 0000000000000024
>>>>>>>>>>> [Mon Mar 11 14:12:05 2024] RDX: ffff8ada519309c8 RSI:
>>>>>>>>>>> ffff8ad582933c00 RDI: 0000000000002000
>>>>>>>>>>> [Mon Mar 11 14:12:05 2024] RBP: ffff8ad46bf21574 R08:
>>>>>>>>>>> ffff9929e0bb7b48 R09: 0000000000000000
>>>>>>>>>>> [Mon Mar 11 14:12:05 2024] R10: ffff8aec859a2948 R11:
>>>>>>>>>>> 0000000000000000 R12: ffff8ad6f497c360
>>>>>>>>>>> [Mon Mar 11 14:12:05 2024] R13: ffff8ad46bf21560 R14:
>>>>>>>>>>> ffff8ae5942e0b10 R15: ffff8ad6f497c360
>>>>>>>>>>> [Mon Mar 11 14:12:05 2024] FS: 0000000000000000(0000)
>>>>>>>>>>> GS:ffff8b031fcc0000(0000) knlGS:0000000000000000
>>>>>>>>>>> [Mon Mar 11 14:12:05 2024] CS: 0010 DS: 0000 ES: 0000 CR0:
>>>>>>>>>>> 0000000080050033
>>>>>>>>>>> [Mon Mar 11 14:12:05 2024] CR2: 00007fafe2060744 CR3:
>>>>>>>>>>> 00000018e58de006 CR4: 00000000007706e0
>>>>>>>>>>> [Mon Mar 11 14:12:05 2024] DR0: 0000000000000000 DR1:
>>>>>>>>>>> 0000000000000000 DR2: 0000000000000000
>>>>>>>>>>> [Mon Mar 11 14:12:05 2024] DR3: 0000000000000000 DR6:
>>>>>>>>>>> 00000000fffe0ff0 DR7: 0000000000000400
>>>>>>>>>>> [Mon Mar 11 14:12:05 2024] PKRU: 55555554
>>>>>>>>>>> [Mon Mar 11 14:12:05 2024] Call Trace:
>>>>>>>>>>> [Mon Mar 11 14:12:05 2024] <TASK>
>>>>>>>>>>> [Mon Mar 11 14:12:05 2024] ?
>>>>>>>>>>> show_trace_log_lvl+0x1c4/0x2df
>>>>>>>>>>> [Mon Mar 11 14:12:05 2024] ?
>>>>>>>>>>> show_trace_log_lvl+0x1c4/0x2df
>>>>>>>>>>> [Mon Mar 11 14:12:05 2024] ? __break_lease+0x16f/0x5f0
>>>>>>>>>>> [Mon Mar 11 14:12:05 2024] ?
>>>>>>>>>>> nfsd_break_deleg_cb+0x170/0x190
>>>>>>>>>>> [nfsd]
>>>>>>>>>>> [Mon Mar 11 14:12:05 2024] ? __warn+0x81/0x110
>>>>>>>>>>> [Mon Mar 11 14:12:05 2024] ?
>>>>>>>>>>> nfsd_break_deleg_cb+0x170/0x190
>>>>>>>>>>> [nfsd]
>>>>>>>>>>> [Mon Mar 11 14:12:05 2024] ? report_bug+0x10a/0x140
>>>>>>>>>>> [Mon Mar 11 14:12:05 2024] ? handle_bug+0x3c/0x70
>>>>>>>>>>> [Mon Mar 11 14:12:05 2024] ? exc_invalid_op+0x14/0x70
>>>>>>>>>>> [Mon Mar 11 14:12:05 2024] ? asm_exc_invalid_op+0x16/0x20
>>>>>>>>>>> [Mon Mar 11 14:12:05 2024] ?
>>>>>>>>>>> nfsd_break_deleg_cb+0x170/0x190
>>>>>>>>>>> [nfsd]
>>>>>>>>>>> [Mon Mar 11 14:12:05 2024] __break_lease+0x16f/0x5f0
>>>>>>>>>>> [Mon Mar 11 14:12:05 2024] ?
>>>>>>>>>>> nfsd_file_lookup_locked+0x117/0x160 [nfsd]
>>>>>>>>>>> [Mon Mar 11 14:12:05 2024] ? list_lru_del+0x101/0x150
>>>>>>>>>>> [Mon Mar 11 14:12:05 2024]
>>>>>>>>>>> nfsd_file_do_acquire+0x790/0x830
>>>>>>>>>>> [nfsd]
>>>>>>>>>>> [Mon Mar 11 14:12:05 2024]
>>>>>>>>>>> nfs4_get_vfs_file+0x315/0x3a0 [nfsd]
>>>>>>>>>>> [Mon Mar 11 14:12:05 2024]
>>>>>>>>>>> nfsd4_process_open2+0x430/0xa30 [nfsd]
>>>>>>>>>>> [Mon Mar 11 14:12:05 2024] ? fh_verify+0x297/0x2f0 [nfsd]
>>>>>>>>>>> [Mon Mar 11 14:12:05 2024] nfsd4_open+0x3ce/0x4b0 [nfsd]
>>>>>>>>>>> [Mon Mar 11 14:12:05 2024]
>>>>>>>>>>> nfsd4_proc_compound+0x44b/0x700 [nfsd]
>>>>>>>>>>> [Mon Mar 11 14:12:05 2024] nfsd_dispatch+0x94/0x1c0 [nfsd]
>>>>>>>>>>> [Mon Mar 11 14:12:05 2024] svc_process_common+0x2ec/0x660
>>>>>>>>>>> [sunrpc]
>>>>>>>>>>> [Mon Mar 11 14:12:05 2024] ?
>>>>>>>>>>> __pfx_nfsd_dispatch+0x10/0x10 [nfsd]
>>>>>>>>>>> [Mon Mar 11 14:12:05 2024] ? __pfx_nfsd+0x10/0x10 [nfsd]
>>>>>>>>>>> [Mon Mar 11 14:12:05 2024] svc_process+0x12d/0x170
>>>>>>>>>>> [sunrpc]
>>>>>>>>>>> [Mon Mar 11 14:12:05 2024] nfsd+0x84/0xb0 [nfsd]
>>>>>>>>>>> [Mon Mar 11 14:12:05 2024] kthread+0xdd/0x100
>>>>>>>>>>> [Mon Mar 11 14:12:05 2024] ? __pfx_kthread+0x10/0x10
>>>>>>>>>>> [Mon Mar 11 14:12:05 2024] ret_from_fork+0x29/0x50
>>>>>>>>>>> [Mon Mar 11 14:12:05 2024] </TASK>
>>>>>>>>>>> [Mon Mar 11 14:12:05 2024] ---[ end trace
>>>>>>>>>>> 7a039e17443dc651 ]---
>>>>>>>>>> This is probably this WARN in nfsd_break_one_deleg:
>>>>>>>>>>
>>>>>>>>>> WARN_ON_ONCE(!nfsd4_run_cb(&dp->dl_recall));
>>>>>>>>>>
>>>>>>>>>> It means that a delegation break callback to the client
>>>>>>>>>> couldn't be
>>>>>>>>>> queued to the workqueue, and so it didn't run.
>>>>>>>>>>
>>>>>>>>>>> Could this be the same issue as described
>>>>>>>>>>> here:https://urldefense.com/v3/__https://lore.kernel.org/linux-nfs/af0ec881-5ebf-4feb-98ae-3ed2a77f86f1@oracle.com/__;!!ACWV5N9M2RV99hQ!LV3yWeoSOhNAkRHkxFCH2tlm0iNFVD78mxnSLyP6lrX7yBVeA2TOJ4nv6oZsqLwP4kW56CMpDWhkjjwSkdBV9En7$
>>>>>>>>>>> ?
>>>>>>>>>> Yes, most likely the same problem.
>>>>>>>>> If I read that thread correctly, this issue was introduced
>>>>>>>>> between
>>>>>>>>> 6.1.63 and 6.6.3? Is it possible the EL9 5.14.0-362.18.1.el9_3
>>>>>>>>> backported these changes, or were we hitting some other bug
>>>>>>>>> with that
>>>>>>>>> version? It seems the 6.1.x kernel is not affected? If so, that
>>>>>>>>> would be
>>>>>>>>> the recommended kernel to run?
>>>>>>>> Anything is possible. We have to identify the problem first.
>>>>>>>>>>> As described in that thread, I've tried to obtain the requested
>>>>>>>>>>> information.
>>>>>>>>>>>
>>>>>>>>>>> Is it possible this is the issue that was fixed by the patches
>>>>>>>>>>> described
>>>>>>>>>>> here?https://urldefense.com/v3/__https://lore.kernel.org/linux-nfs/2024022054-cause-suffering-eae8@gregkh/__;!!ACWV5N9M2RV99hQ!LV3yWeoSOhNAkRHkxFCH2tlm0iNFVD78mxnSLyP6lrX7yBVeA2TOJ4nv6oZsqLwP4kW56CMpDWhkjjwSkedtUP09$
>>>>>>>>>>>
>>>>>>>>>> Doubtful. Those are targeted toward a different set of issues.
>>>>>>>>>>
>>>>>>>>>> If you're willing, I do have some patches queued up for
>>>>>>>>>> CentOS here
>>>>>>>>>> that
>>>>>>>>>> fix some backchannel problems that could be related. I'm mainly
>>>>>>>>>> waiting
>>>>>>>>>> on Chuck to send these to Linus and then we'll likely merge
>>>>>>>>>> them into
>>>>>>>>>> CentOS soon afterward:
>>>>>>>>>>
>>>>>>>>>> https://urldefense.com/v3/__https://gitlab.com/redhat/centos-stream/src/kernel/centos-stream-9/-/merge_requests/3689__;!!ACWV5N9M2RV99hQ!LV3yWeoSOhNAkRHkxFCH2tlm0iNFVD78mxnSLyP6lrX7yBVeA2TOJ4nv6oZsqLwP4kW56CMpDWhkjjwSkdvDn8y7$
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>> If you can send me a patch file, I can rebuild the C9S kernel
>>>>>>>>> with that
>>>>>>>>> patch and run it. It can take a while for the bug to trigger as I
>>>>>>>>> believe it seems to be very workload dependent (we were
>>>>>>>>> running very
>>>>>>>>> stable for months and now hit this bug every other week).
>>>>>>>>>
>>>>>>>>>
>>>>>>>> It's probably simpler to just pull down the build artifacts for
>>>>>>>> that MR.
>>>>>>>> You have to drill down through the CI for it, but they are here:
>>>>>>>>
>>>>>>>> https://urldefense.com/v3/__https://s3.amazonaws.com/arr-cki-prod-trusted-artifacts/index.html?prefix=trusted-artifacts*1194300175*publish_x86_64*6278921877*artifacts*__;Ly8vLy8!!ACWV5N9M2RV99hQ!LV3yWeoSOhNAkRHkxFCH2tlm0iNFVD78mxnSLyP6lrX7yBVeA2TOJ4nv6oZsqLwP4kW56CMpDWhkjjwSkaP5eW8V$
>>>>>>>>
>>>>>>>>
>>>>>>>> There's even a repo file you can install on the box to pull
>>>>>>>> them down.
>>>>>>> We installed this kernel on the server 3 days ago. Today, a user
>>>>>>> informed us that their screen was black after logging in.
>>>>>>> Similar to
>>>>>>> other occurrences of this issue, the mount command on the client
>>>>>>> was
>>>>>>> hung. But in contrast to the other times, there were no messages in
>>>>>>> the logs kernel logs on the server. Even restarting the client does
>>>>>>> not resolve the issue.
>>>>>
>>>>> Ok, so you rebooted the client and it's still unable to mount? That
>>>>> sounds like a server problem if so.
>>>>>
>>>>> Are both client and server running the same kernel?
>>>> No, the server runs 5.14.0-427.3689_1194299994.el9 and the client
>>>> 5.14.0-362.18.1.el9_3.
>>>>>
>>>>>>> Something still seems to be wrong on the server though. When I
>>>>>>> look at
>>>>>>> the directories under /proc/fs/nfsd/clients, there's still a
>>>>>>> directory
>>>>>>> for the specific client, even though it's no longer running:
>>>>>>>
>>>>>>> # cat 155/info
>>>>>>> clientid: 0xc8edb7f65f4a9ad
>>>>>>> address: "10.87.31.152:819"
>>>>>>> status: confirmed
>>>>>>> seconds from last renew: 33163
>>>>>>> name: "Linux NFSv4.2 bersalis.esat.kuleuven.be"
>>>>>>> minor version: 2
>>>>>>> Implementation domain: "kernel.org"
>>>>>>> Implementation name: "Linux 5.14.0-362.18.1.el9_3.0.1.x86_64 #1 SMP
>>>>>>> PREEMPT_DYNAMIC Sun Feb 11 13:49:23 UTC 2024 x86_64"
>>>>>>> Implementation time: [0, 0]
>>>>>>> callback state: DOWN
>>>>>>> callback address: 10.87.31.152:0
>>>>>>>
>>>>> If you just shut down the client, the server won't immediately
>>>>> purge its
>>>>> record. In fact, assuming you're running the same kernel on the
>>>>> server,
>>>>> it won't purge the client record until there is a conflicting request
>>>>> for its state.
>>>> Is there a way to force such a conflicting request (to get the
>>>> client record to purge)?
>>>
>>> Try:
>>>
>>> # echo "expire" > /proc/fs/nfsd/clients/155/ctl
>>
>> I've tried that. The command hangs and can not be interrupted with
>> ctrl-c.
>> I've now also noticed in the dmesg output that the kernel issued the
>> following WARNING a few hours ago. It wasn't directly triggered by
>> the echo command above, but seems to have been triggered a few hours
>> ago (probably when another client started to have the same problem as
>> more clients are experiencing issues now).
>
> I think this warning message is harmless. However it indicates potential
> problem with the workqueue which might be related to memory shortage.
>
> What the output of 'cat /proc/meminfo' looks like?
I doubt the current values are useful, but they are:
MemTotal: 196110860 kB
MemFree: 29357112 kB
MemAvailable: 179529420 kB
Buffers: 11996096 kB
Cached: 130589396 kB
SwapCached: 52 kB
Active: 1136988 kB
Inactive: 144192468 kB
Active(anon): 698564 kB
Inactive(anon): 2657256 kB
Active(file): 438424 kB
Inactive(file): 141535212 kB
Unevictable: 72140 kB
Mlocked: 69068 kB
SwapTotal: 67108860 kB
SwapFree: 67106276 kB
Zswap: 0 kB
Zswapped: 0 kB
Dirty: 80812 kB
Writeback: 0 kB
AnonPages: 2806592 kB
Mapped: 322700 kB
Shmem: 599308 kB
KReclaimable: 16977000 kB
Slab: 18898736 kB
SReclaimable: 16977000 kB
SUnreclaim: 1921736 kB
KernelStack: 18128 kB
PageTables: 31716 kB
SecPageTables: 0 kB
NFS_Unstable: 0 kB
Bounce: 0 kB
WritebackTmp: 0 kB
CommitLimit: 165164288 kB
Committed_AS: 5223940 kB
VmallocTotal: 34359738367 kB
VmallocUsed: 300064 kB
VmallocChunk: 0 kB
Percpu: 45888 kB
HardwareCorrupted: 0 kB
AnonHugePages: 2451456 kB
ShmemHugePages: 0 kB
ShmemPmdMapped: 0 kB
FileHugePages: 0 kB
FilePmdMapped: 0 kB
CmaTotal: 0 kB
CmaFree: 0 kB
Unaccepted: 0 kB
HugePages_Total: 0
HugePages_Free: 0
HugePages_Rsvd: 0
HugePages_Surp: 0
Hugepagesize: 2048 kB
Hugetlb: 0 kB
DirectMap4k: 1303552 kB
DirectMap2M: 28715008 kB
DirectMap1G: 171966464 kB
>
> Did you try 'echo 3 > /proc/sys/vm/drop_caches'?
Yes, I tried that when the first client hit the issue, but it didn't
result in any unlocking of the client.
>
>>
>> [Tue Mar 19 14:53:44 2024] ------------[ cut here ]------------
>> [Tue Mar 19 14:53:44 2024] WARNING: CPU: 44 PID: 5843 at
>> fs/nfsd/nfs4state.c:4920 nfsd_break_deleg_cb+0x170/0x190 [nfsd]
>> [Tue Mar 19 14:53:44 2024] Modules linked in: nf_conntrack_netlink
>> nfsv4 dns_resolver nfs fscache netfs binfmt_misc xsk_diag
>> rpcsec_gss_krb5 rpcrdma rdma_cm iw_cm ib_cm ib_core bonding tls
>> rfkill nft_counter nft_ct nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4
>> nft_reject_inet nf_reject_ipv4 nf_reject_ipv6 nft_reject nf_tables
>> nfnetlink vfat fat dm_thin_pool dm_persistent_data dm_bio_prison
>> dm_bufio libcrc32c dm_service_time dm_multipath intel_rapl_msr
>> intel_rapl_common intel_uncore_frequency
>> intel_uncore_frequency_common isst_if_common skx_edac nfit libnvdimm
>> x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm dcdbas
>> irqbypass ipmi_ssif rapl intel_cstate mgag200 i2c_algo_bit
>> drm_shmem_helper drm_kms_helper dell_smbios syscopyarea intel_uncore
>> sysfillrect wmi_bmof dell_wmi_descriptor pcspkr sysimgblt fb_sys_fops
>> mei_me i2c_i801 mei intel_pch_thermal acpi_ipmi i2c_smbus lpc_ich
>> ipmi_si ipmi_devintf ipmi_msghandler joydev acpi_power_meter nfsd
>> nfs_acl lockd auth_rpcgss grace drm fuse sunrpc ext4
>> [Tue Mar 19 14:53:44 2024] mbcache jbd2 sd_mod sg lpfc nvmet_fc
>> nvmet nvme_fc nvme_fabrics crct10dif_pclmul crc32_pclmul nvme_core
>> ixgbe crc32c_intel ahci libahci nvme_common megaraid_sas t10_pi
>> ghash_clmulni_intel wdat_wdt libata scsi_transport_fc mdio dca wmi
>> dm_mirror dm_region_hash dm_log dm_mod
>> [Tue Mar 19 14:53:44 2024] CPU: 44 PID: 5843 Comm: nfsd Not tainted
>> 5.14.0-427.3689_1194299994.el9.x86_64 #1
>> [Tue Mar 19 14:53:44 2024] Hardware name: Dell Inc. PowerEdge
>> R740/00WGD1, BIOS 2.20.1 09/13/2023
>> [Tue Mar 19 14:53:44 2024] RIP: 0010:nfsd_break_deleg_cb+0x170/0x190
>> [nfsd]
>> [Tue Mar 19 14:53:44 2024] Code: 76 76 cd de e9 ff fe ff ff 48 89 df
>> be 01 00 00 00 e8 34 a1 1b df 48 8d bb 98 00 00 00 e8 a8 fe 00 00 84
>> c0 0f 85 2e ff ff ff <0f> 0b e9 27 ff ff ff be 02 00 00 00 48 89 df
>> e8 0c a1 1b df e9 01
>> [Tue Mar 19 14:53:44 2024] RSP: 0018:ffffb2878f2cfc38 EFLAGS: 00010246
>> [Tue Mar 19 14:53:44 2024] RAX: 0000000000000000 RBX:
>> ffff88d5171067b8 RCX: 0000000000000000
>> [Tue Mar 19 14:53:44 2024] RDX: ffff88d517106880 RSI:
>> ffff88bdceec8600 RDI: 0000000000002000
>> [Tue Mar 19 14:53:44 2024] RBP: ffff88d68a38a284 R08:
>> ffffb2878f2cfc00 R09: 0000000000000000
>> [Tue Mar 19 14:53:44 2024] R10: ffff88bf57dd7878 R11:
>> 0000000000000000 R12: ffff88d5b79c4798
>> [Tue Mar 19 14:53:44 2024] R13: ffff88d68a38a270 R14:
>> ffff88cab06ad0c8 R15: ffff88d5b79c4798
>> [Tue Mar 19 14:53:44 2024] FS: 0000000000000000(0000)
>> GS:ffff88d4a1180000(0000) knlGS:0000000000000000
>> [Tue Mar 19 14:53:44 2024] CS: 0010 DS: 0000 ES: 0000 CR0:
>> 0000000080050033
>> [Tue Mar 19 14:53:44 2024] CR2: 00007fe46ef90000 CR3:
>> 000000019d010004 CR4: 00000000007706e0
>> [Tue Mar 19 14:53:44 2024] DR0: 0000000000000000 DR1:
>> 0000000000000000 DR2: 0000000000000000
>> [Tue Mar 19 14:53:44 2024] DR3: 0000000000000000 DR6:
>> 00000000fffe0ff0 DR7: 0000000000000400
>> [Tue Mar 19 14:53:44 2024] PKRU: 55555554
>> [Tue Mar 19 14:53:44 2024] Call Trace:
>> [Tue Mar 19 14:53:44 2024] <TASK>
>> [Tue Mar 19 14:53:44 2024] ? show_trace_log_lvl+0x1c4/0x2df
>> [Tue Mar 19 14:53:44 2024] ? show_trace_log_lvl+0x1c4/0x2df
>> [Tue Mar 19 14:53:44 2024] ? __break_lease+0x16f/0x5f0
>> [Tue Mar 19 14:53:44 2024] ? nfsd_break_deleg_cb+0x170/0x190 [nfsd]
>> [Tue Mar 19 14:53:44 2024] ? __warn+0x81/0x110
>> [Tue Mar 19 14:53:44 2024] ? nfsd_break_deleg_cb+0x170/0x190 [nfsd]
>> [Tue Mar 19 14:53:44 2024] ? report_bug+0x10a/0x140
>> [Tue Mar 19 14:53:44 2024] ? handle_bug+0x3c/0x70
>> [Tue Mar 19 14:53:44 2024] ? exc_invalid_op+0x14/0x70
>> [Tue Mar 19 14:53:44 2024] ? asm_exc_invalid_op+0x16/0x20
>> [Tue Mar 19 14:53:44 2024] ? nfsd_break_deleg_cb+0x170/0x190 [nfsd]
>> [Tue Mar 19 14:53:44 2024] ? nfsd_break_deleg_cb+0x96/0x190 [nfsd]
>> [Tue Mar 19 14:53:44 2024] __break_lease+0x16f/0x5f0
>> [Tue Mar 19 14:53:44 2024] nfs4_get_vfs_file+0x164/0x3a0 [nfsd]
>> [Tue Mar 19 14:53:44 2024] nfsd4_process_open2+0x430/0xa30 [nfsd]
>> [Tue Mar 19 14:53:44 2024] ? fh_verify+0x297/0x2f0 [nfsd]
>> [Tue Mar 19 14:53:44 2024] nfsd4_open+0x3ce/0x4b0 [nfsd]
>> [Tue Mar 19 14:53:44 2024] nfsd4_proc_compound+0x44b/0x700 [nfsd]
>> [Tue Mar 19 14:53:44 2024] nfsd_dispatch+0x94/0x1c0 [nfsd]
>> [Tue Mar 19 14:53:44 2024] svc_process_common+0x2ec/0x660 [sunrpc]
>> [Tue Mar 19 14:53:44 2024] ? __pfx_nfsd_dispatch+0x10/0x10 [nfsd]
>> [Tue Mar 19 14:53:44 2024] ? __pfx_nfsd+0x10/0x10 [nfsd]
>> [Tue Mar 19 14:53:44 2024] svc_process+0x12d/0x170 [sunrpc]
>> [Tue Mar 19 14:53:44 2024] nfsd+0x84/0xb0 [nfsd]
>> [Tue Mar 19 14:53:44 2024] kthread+0xdd/0x100
>> [Tue Mar 19 14:53:44 2024] ? __pfx_kthread+0x10/0x10
>> [Tue Mar 19 14:53:44 2024] ret_from_fork+0x29/0x50
>> [Tue Mar 19 14:53:44 2024] </TASK>
>> [Tue Mar 19 14:53:44 2024] ---[ end trace ed0b2b3f135c637d ]---
>>
>> It again seems to have been triggered in nfsd_break_deleg_cb?
>>
>> I also had the following perf command running a tmux on the server:
>>
>> perf trace -e nfsd:nfsd_cb_recall_any
>>
>> This has spewed a lot of messages. I'm including a short list here:
>>
>> ...
>>
>> 33464866.721 kworker/u98:5/1591466 nfsd:nfsd_cb_recall_any(cl_boot:
>> 1710533037, cl_id: 210688785, bmval0: 1, addr: 0x7f331bb116c8)
>> 33464866.724 kworker/u98:5/1591466 nfsd:nfsd_cb_recall_any(cl_boot:
>> 1710533037, cl_id: 210688827, bmval0: 1, addr: 0x7f331bb11738)
>> 33464866.729 kworker/u98:5/1591466 nfsd:nfsd_cb_recall_any(cl_boot:
>> 1710533037, cl_id: 210688767, bmval0: 1, addr: 0x7f331bb117a8)
>> 33464866.732 kworker/u98:5/1591466 nfsd:nfsd_cb_recall_any(cl_boot:
>> 1710533037, cl_id: 210718132, bmval0: 1, addr: 0x7f331bb11818)
>> 33464866.737 kworker/u98:5/1591466 nfsd:nfsd_cb_recall_any(cl_boot:
>> 1710533037, cl_id: 210688952, bmval0: 1, addr: 0x7f331bb11888)
>> 33464866.741 kworker/u98:5/1591466 nfsd:nfsd_cb_recall_any(cl_boot:
>> 1710533037, cl_id: 210702355, bmval0: 1, addr: 0x7f331bb118f8)
>> 33868414.001 kthreadd/1597068 nfsd:nfsd_cb_recall_any(cl_boot:
>> 1710533037, cl_id: 210688751, bmval0: 1, addr: 0x7f331be68620)
>> 33868414.014 kthreadd/1597068 nfsd:nfsd_cb_recall_any(cl_boot:
>> 1710533037, cl_id: 210718536, bmval0: 1, addr: 0x7f331be68690)
>> 33868414.018 kthreadd/1597068 nfsd:nfsd_cb_recall_any(cl_boot:
>> 1710533037, cl_id: 210719074, bmval0: 1, addr: 0x7f331be68700)
>> 33868414.022 kthreadd/1597068 nfsd:nfsd_cb_recall_any(cl_boot:
>> 1710533037, cl_id: 210688916, bmval0: 1, addr: 0x7f331be68770)
>> 33868414.026 kthreadd/1597068 nfsd:nfsd_cb_recall_any(cl_boot:
>> 1710533037, cl_id: 210688941, bmval0: 1, addr: 0x7f331be687e0)
>> ...
>>
>> 33868414.924 kthreadd/1597068 nfsd:nfsd_cb_recall_any(cl_boot:
>> 1710533037, cl_id: 210688744, bmval0: 1, addr: 0x7f331be6d7f0)
>> 33868414.929 kthreadd/1597068 nfsd:nfsd_cb_recall_any(cl_boot:
>> 1710533037, cl_id: 210717223, bmval0: 1, addr: 0x7f331be6d860)
>> 33868414.934 kthreadd/1597068 nfsd:nfsd_cb_recall_any(cl_boot:
>> 1710533037, cl_id: 210716137, bmval0: 1, addr: 0x7f331be6d8d0)
>> 34021240.903 kworker/u98:5/1591466 nfsd:nfsd_cb_recall_any(cl_boot:
>> 1710533037, cl_id: 210688941, bmval0: 1, addr: 0x7f331c207de8)
>> 34021240.917 kworker/u98:5/1591466 nfsd:nfsd_cb_recall_any(cl_boot:
>> 1710533037, cl_id: 210718750, bmval0: 1, addr: 0x7f331c207e58)
>> 34021240.922 kworker/u98:5/1591466 nfsd:nfsd_cb_recall_any(cl_boot:
>> 1710533037, cl_id: 210688955, bmval0: 1, addr: 0x7f331c207ec8)
>> 34021240.925 kworker/u98:5/1591466 nfsd:nfsd_cb_recall_any(cl_boot:
>> 1710533037, cl_id: 210688975, bmval0: 1, addr: 0x7f331c207f38)
>> ...
>>
>> I assume the cl_id is the client id? How can I map this to a client
>> from /proc/fs/nfsd/clients?
>
> The hex value of 'clientid' printed from /proc/fs/nfsd/clients/XX/info
> is a 64-bit value composed of:
>
> typedef struct {
> u32 cl_boot;
> u32 cl_id;
> } clientid_t
>
> For example:
>
> clientid: 0xc8edb7f65f4a9ad
>
> cl_boot: 65f4a9add (1710533037)
> cl_id: c8edb7f (21068895)
>
> This should match a trace event with:
>
> nfsd:nfsd_cb_recall_any(cl_boot: 1710533037, cl_id: 21068895, bmval0:
> XX, addr: 0xYYYYY)
>
>>
>> If I understand it correctly, the recall_any should be called when
>> either the system starts to experience memory pressure,
>
> yes.
It seems odd that the system gets in such a state that has such high
memory pressure. It doesn't run much else than NFS and Samba.
>
>> or it reaches the delegation limits?
>
> No, this feature was added to nfsd very recently. I don't think your
> kernel has it.
>
>> I doubt the system is actually running out of memory here as there
>> are no other indications.
>> Shouldn't I get those "page allocation failure" messages if it does?
>> How can I check the number of delegations/leases currently issued,
>> what the current maximum is and how to increase it?
>
> Max delegations is 4 per 1MB of available memory. There is no
> admin tool to adjust this value.
/proc/locks currently has about 130k DELEG lines, so that should be a
lot lower than the limit on a 192G ram server.
>
> I do not recommend running a production system with delegation
> disabled. But for this specific issue, it might help to temporarily
> disable delegation to isolate problem areas.
I'm going to reboot the system with the 6.1.82 kernel (kernel-lt from
elrepo). Maybe it has less new modern developments that may have
introduced this.
I've been able to reproduce the situation on an additional client now
that the issue happens on the server:
1. Log in on a client and mount the NFS share.
2. Open a file from the NFS share in vim so the client gets a read
delegation from the server
3. Verify on the server in /proc/fs/nfsd/clients/*/states that the
client has a delegation for the file
4. Forcefully reboot the client by running 'echo b > /proc/sysrq-trigger'
5. Watch the /proc/fs/nfsd/clients/*/info file on the server.
The "seconds from last renew" will go up and at some point the callback
state changes to "FAULT". Even when the lease delegation time (90s by
default?) is over, the
seconds from last renew keeps increasing. At some point the callback
state changes to "DOWN". When the client is up again and remounts the
share, the mount hangs on the client
and on the server I notice there's a second directory for this client in
the clients directory, even though the clientid is the same. The
callback state for this new client is "UNKNOWN" and the callback address
is "(einval)".
This is on a client running Fedora 39 with the 6.7.9 kernel.
I don't know yet if the same procedure can be used to trigger the
behavior after the server is rebooted. I'm going to try to reproduce
this on another system first.
I would expect the delegations to expire automatically after 90s, but
they remain in the states file of the "DOWN" client.
Regards,
Rik
>
> -Dai
>
>>
>> Regarding the recall any call: from what I've read on kernelnewbies,
>> this feature was introduced in the 6.2 kernel? When I look at the
>> tree for 6.1.x, it was backported in 6.1.81? Is there a way to
>> disable this support somehow?
>>
>> Regards,
>>
>> Rik
>>
>>
>>>
>>> -Dai
>>>
>>>>>
>>>>>
>>>>>> The nfsdclnts command for this client shows the following
>>>>>> delegations:
>>>>>>
>>>>>> # nfsdclnts -f 155/states -t all
>>>>>> Inode number | Type | Access | Deny | ip address | Filename
>>>>>> 169346743 | open | r- | -- | 10.87.31.152:819 |
>>>>>> disconnected dentry
>>>>>> 169346743 | deleg | r | | 10.87.31.152:819 |
>>>>>> disconnected dentry
>>>>>> 169346746 | open | r- | -- | 10.87.31.152:819 |
>>>>>> disconnected dentry
>>>>>> 169346746 | deleg | r | | 10.87.31.152:819 |
>>>>>> disconnected dentry
>>>>>>
>>>>>> I see a lot of recent patches regarding directory delegations. Could
>>>>>> this be related to this?
>>>>>>
>>>>>> Will a 5.14.0-362.18.1.el9_3.0.1 kernel try to use a directory
>>>>>> delegation?
>>>>>>
>>>>>>
>>>>> No. Directory delegations are a new feature that's still under
>>>>> development. They use some of the same machinery as file delegations,
>>>>> but they wouldn't be a factor here.
>>>>>
>>>>>>> The system seems to have identified that the client is no longer
>>>>>>> reachable, but the client entry does not go away. When a mount was
>>>>>>> hanging on the client, there would be two directories in clients
>>>>>>> for
>>>>>>> the same client. Killing the mount command clears up the second
>>>>>>> entry.
>>>>>>>
>>>>>>> Even after running conntrack -D on the server to remove the tcp
>>>>>>> connection from the conntrack table, the entry doesn't go away
>>>>>>> and the
>>>>>>> client still can not mount anything from the server.
>>>>>>>
>>>>>>> A tcpdump on the client while a mount was running logged the
>>>>>>> following
>>>>>>> messages over and over again:
>>>>>>>
>>>>>>> request:
>>>>>>>
>>>>>>> Frame 1: 378 bytes on wire (3024 bits), 378 bytes captured (3024
>>>>>>> bits)
>>>>>>> Ethernet II, Src: HP_19:7d:4b (e0:73:e7:19:7d:4b), Dst:
>>>>>>> ArubaaHe_f9:8e:00 (88:3a:30:f9:8e:00)
>>>>>>> Internet Protocol Version 4, Src: 10.87.31.152, Dst: 10.86.18.14
>>>>>>> Transmission Control Protocol, Src Port: 932, Dst Port: 2049,
>>>>>>> Seq: 1,
>>>>>>> Ack: 1, Len: 312
>>>>>>> Remote Procedure Call, Type:Call XID:0x1d3220c4
>>>>>>> Network File System
>>>>>>> [Program Version: 4]
>>>>>>> [V4 Procedure: COMPOUND (1)]
>>>>>>> GSS Data, Ops(1): CREATE_SESSION
>>>>>>> Length: 152
>>>>>>> GSS Sequence Number: 76
>>>>>>> Tag: <EMPTY>
>>>>>>> minorversion: 2
>>>>>>> Operations (count: 1): CREATE_SESSION
>>>>>>> [Main Opcode: CREATE_SESSION (43)]
>>>>>>> GSS Checksum:
>>>>>>> 00000028040404ffffffffff000000002c19055f1f8d442d594c13849628affc2797cbb2…
>>>>>>>
>>>>>>> GSS Token Length: 40
>>>>>>> GSS-API Generic Security Service Application Program
>>>>>>> Interface
>>>>>>> krb5_blob:
>>>>>>> 040404ffffffffff000000002c19055f1f8d442d594c13849628affc2797cbb23fa080b0…
>>>>>>>
>>>>>>>
>>>>>>> response:
>>>>>>>
>>>>>>> Frame 2: 206 bytes on wire (1648 bits), 206 bytes captured (1648
>>>>>>> bits)
>>>>>>> Ethernet II, Src: ArubaaHe_f9:8e:00 (88:3a:30:f9:8e:00), Dst:
>>>>>>> HP_19:7d:4b (e0:73:e7:19:7d:4b)
>>>>>>> Internet Protocol Version 4, Src: 10.86.18.14, Dst: 10.87.31.152
>>>>>>> Transmission Control Protocol, Src Port: 2049, Dst Port: 932,
>>>>>>> Seq: 1,
>>>>>>> Ack: 313, Len: 140
>>>>>>> Remote Procedure Call, Type:Reply XID:0x1d3220c4
>>>>>>> Network File System
>>>>>>> [Program Version: 4]
>>>>>>> [V4 Procedure: COMPOUND (1)]
>>>>>>> GSS Data, Ops(1): CREATE_SESSION(NFS4ERR_DELAY)
>>>>>>> Length: 24
>>>>>>> GSS Sequence Number: 76
>>>>>>> Status: NFS4ERR_DELAY (10008)
>>>>>>> Tag: <EMPTY>
>>>>>>> Operations (count: 1)
>>>>>>> [Main Opcode: CREATE_SESSION (43)]
>>>>>>> GSS Checksum:
>>>>>>> 00000028040405ffffffffff000000000aa742d0798deaad1a8aa2d7c3a91bf4f6274222…
>>>>>>>
>>>>>>> GSS Token Length: 40
>>>>>>> GSS-API Generic Security Service Application Program
>>>>>>> Interface
>>>>>>> krb5_blob:
>>>>>>> 040405ffffffffff000000000aa742d0798deaad1a8aa2d7c3a91bf4f627422226d74923…
>>>>>>>
>>>>>>>
>>>>>>> I was hoping that giving the client a different IP address would
>>>>>>> resolve the issue for this client, but it didn't. Even though the
>>>>>>> client had a new IP address (hostname was kept the same), it
>>>>>>> failed to
>>>>>>> mount anything from the server.
>>>>>>>
>>>>> Changing the IP address won't help. The client is probably using the
>>>>> same long-form client id as before, so the server still identifies
>>>>> the
>>>>> client even with the address change.
>>>> How is the client id determined? Will changing the hostname of the
>>>> client trigger a change of the client id?
>>>>>
>>>>> Unfortunately, the cause of an NFS4ERR_DELAY error is tough to guess.
>>>>> The client is expected to back off and retry, so if the server keeps
>>>>> returning that repeatedly, then a hung mount command is expected.
>>>>>
>>>>> The question is why the server would keep returning DELAY. A lot of
>>>>> different problems ranging from memory allocation issues to protocol
>>>>> problems can result in that error. You may want to check the NFS
>>>>> server
>>>>> and see if anything was logged there.
>>>> There are no messages in the system logs that indicate any sort of
>>>> memory issue. We also increased the min_kbytes_free sysctl to 2G on
>>>> the server before we restarted it with the newer kernel.
>>>>>
>>>>> This is on a CREATE_SESSION call, so I wonder if the record held
>>>>> by the
>>>>> (courteous) server is somehow blocking the attempt to reestablish the
>>>>> session?
>>>>>
>>>>> Do you have a way to reproduce this? Since this is a centos
>>>>> kernel, you
>>>>> could follow the page here to open a bug:
>>>>
>>>> Unfortunately we haven't found a reliable way to reproduce it. But
>>>> we do seem to trigger it more and more lately.
>>>>
>>>> Regards,
>>>>
>>>> Rik
>>>>
>>>>>
>>>>> https://urldefense.com/v3/__https://wiki.centos.org/ReportBugs.html__;!!ACWV5N9M2RV99hQ!LV3yWeoSOhNAkRHkxFCH2tlm0iNFVD78mxnSLyP6lrX7yBVeA2TOJ4nv6oZsqLwP4kW56CMpDWhkjjwSkWIqsboq$
>>>>>
>>>>>
>>>>>>> I created another dump of the workqueues and worker pools on the
>>>>>>> server:
>>>>>>>
>>>>>>> [Mon Mar 18 14:59:33 2024] Showing busy workqueues and worker
>>>>>>> pools:
>>>>>>> [Mon Mar 18 14:59:33 2024] workqueue events: flags=0x0
>>>>>>> [Mon Mar 18 14:59:33 2024] pwq 54: cpus=27 node=1 flags=0x0
>>>>>>> nice=0
>>>>>>> active=1/256 refcnt=2
>>>>>>> [Mon Mar 18 14:59:33 2024] pending: drm_fb_helper_damage_work
>>>>>>> [drm_kms_helper]
>>>>>>> [Mon Mar 18 14:59:33 2024] workqueue events_power_efficient:
>>>>>>> flags=0x80
>>>>>>> [Mon Mar 18 14:59:33 2024] pwq 54: cpus=27 node=1 flags=0x0
>>>>>>> nice=0
>>>>>>> active=1/256 refcnt=2
>>>>>>> [Mon Mar 18 14:59:33 2024] pending: fb_flashcursor
>>>>>>> [Mon Mar 18 14:59:33 2024] workqueue mm_percpu_wq: flags=0x8
>>>>>>> [Mon Mar 18 14:59:33 2024] pwq 54: cpus=27 node=1 flags=0x0
>>>>>>> nice=0
>>>>>>> active=1/256 refcnt=3
>>>>>>> [Mon Mar 18 14:59:33 2024] pending: lru_add_drain_per_cpu
>>>>>>> BAR(362)
>>>>>>> [Mon Mar 18 14:59:33 2024] workqueue kblockd: flags=0x18
>>>>>>> [Mon Mar 18 14:59:33 2024] pwq 55: cpus=27 node=1 flags=0x0
>>>>>>> nice=-20
>>>>>>> active=1/256 refcnt=2
>>>>>>> [Mon Mar 18 14:59:33 2024] pending: blk_mq_timeout_work
>>>>>>>
>>>>>>>
>>>>>>> In contrast to last time, it doesn't show anything regarding nfs
>>>>>>> this
>>>>>>> time.
>>>>>>>
>>>>>>> I also tried the suggestion from Dai Ngo (echo 3 >
>>>>>>> /proc/sys/vm/drop_caches), but that didn't seem to make any
>>>>>>> difference.
>>>>>>>
>>>>>>> We haven't restarted the server yet as it seems the impact seems to
>>>>>>> affect fewer clients that before. Is there anything we can run
>>>>>>> on the
>>>>>>> server to further debug this?
>>>>>>>
>>>>>>> In the past, the issue seemed to deteriorate rapidly and
>>>>>>> resulted in
>>>>>>> issues for almost all clients after about 20 minutes. This time the
>>>>>>> impact seems to be less, but it's not gone.
>>>>>>>
>>>>>>> How can we force the NFS server to forget about a specific
>>>>>>> client? I
>>>>>>> haven't tried to restart the nfs service yet as I'm afraid it will
>>>>>>> fail to stop as before.
>>>>>>>
>>>>> Not with that kernel. There are some new administrative interfaces
>>>>> that
>>>>> might allow that in the future, but they were just merged upstream
>>>>> and
>>>>> aren't in that kernel.
>>>>>
>>>>> --
>>>>> Jeff Layton <jlayton@kernel.org>
>>>>
--
Rik Theys
System Engineer
KU Leuven - Dept. Elektrotechniek (ESAT)
Kasteelpark Arenberg 10 bus 2440 - B-3001 Leuven-Heverlee
+32(0)16/32.11.07
----------------------------------------------------------------
<<Any errors in spelling, tact or fact are transmission errors>>
^ permalink raw reply [flat|nested] 26+ messages in thread
* Re: nfsd hangs and nfsd_break_deleg_cb+0x170/0x190 warning
2024-03-20 19:41 ` Rik Theys
@ 2024-03-21 20:48 ` Jeff Layton
2024-03-21 21:13 ` Rik Theys
0 siblings, 1 reply; 26+ messages in thread
From: Jeff Layton @ 2024-03-21 20:48 UTC (permalink / raw)
To: Rik Theys, Dai Ngo, Linux Nfs
On Wed, 2024-03-20 at 20:41 +0100, Rik Theys wrote:
> Hi,
>
> On 3/19/24 22:42, Dai Ngo wrote:
> >
> > On 3/19/24 12:41 PM, Rik Theys wrote:
> > > Hi,
> > >
> > > On 3/19/24 18:09, Dai Ngo wrote:
> > > >
> > > > On 3/19/24 12:58 AM, Rik Theys wrote:
> > > > > Hi,
> > > > >
> > > > > On 3/18/24 22:54, Jeff Layton wrote:
> > > > > > On Mon, 2024-03-18 at 22:15 +0100, Rik Theys wrote:
> > > > > > > Hi,
> > > > > > >
> > > > > > > On 3/18/24 21:21, Rik Theys wrote:
> > > > > > > > Hi Jeff,
> > > > > > > >
> > > > > > > > On 3/12/24 13:47, Jeff Layton wrote:
> > > > > > > > > On Tue, 2024-03-12 at 13:24 +0100, Rik Theys wrote:
> > > > > > > > > > Hi Jeff,
> > > > > > > > > >
> > > > > > > > > > On 3/12/24 12:22, Jeff Layton wrote:
> > > > > > > > > > > On Mon, 2024-03-11 at 19:43 +0100, Rik Theys wrote:
> > > > > > > > > > > > Since a few weeks our Rocky Linux 9 NFS server has periodically
> > > > > > > > > > > > logged hung nfsd tasks. The initial effect was that some
> > > > > > > > > > > > clients
> > > > > > > > > > > > could no longer access the NFS server. This got worse and worse
> > > > > > > > > > > > (probably as more nfsd threads got blocked) and we had to
> > > > > > > > > > > > restart
> > > > > > > > > > > > the server. Restarting the server also failed as the NFS server
> > > > > > > > > > > > service could no longer be stopped.
> > > > > > > > > > > >
> > > > > > > > > > > > The initial kernel we noticed this behavior on was
> > > > > > > > > > > > kernel-5.14.0-362.18.1.el9_3.x86_64. Since then we've installed
> > > > > > > > > > > > kernel-5.14.0-419.el9.x86_64 from CentOS Stream 9. The same
> > > > > > > > > > > > issue
> > > > > > > > > > > > happened again on this newer kernel version:
> > > > > > > > > 419 is fairly up to date with nfsd changes. There are some
> > > > > > > > > known bugs
> > > > > > > > > around callbacks, and there is a draft MR in flight to fix it.
> > > > > > > > >
> > > > > > > > > What kernel were you on prior to 5.14.0-362.18.1.el9_3.x86_64 ?
> > > > > > > > > If we
> > > > > > > > > can bracket the changes around a particular version, then that
> > > > > > > > > might
> > > > > > > > > help identify the problem.
> > > > > > > > >
> > > > > > > > > > > > [Mon Mar 11 14:10:08 2024] Not tainted
> > > > > > > > > > > > 5.14.0-419.el9.x86_64 #1
> > > > > > > > > > > > [Mon Mar 11 14:10:08 2024] "echo 0 >
> > > > > > > > > > > > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> > > > > > > > > > > > [Mon Mar 11 14:10:08 2024]task:nfsd state:D
> > > > > > > > > > > > stack:0
> > > > > > > > > > > > pid:8865 ppid:2 flags:0x00004000
> > > > > > > > > > > > [Mon Mar 11 14:10:08 2024] Call Trace:
> > > > > > > > > > > > [Mon Mar 11 14:10:08 2024] <TASK>
> > > > > > > > > > > > [Mon Mar 11 14:10:08 2024] __schedule+0x21b/0x550
> > > > > > > > > > > > [Mon Mar 11 14:10:08 2024] schedule+0x2d/0x70
> > > > > > > > > > > > [Mon Mar 11 14:10:08 2024] schedule_timeout+0x11f/0x160
> > > > > > > > > > > > [Mon Mar 11 14:10:08 2024] ?
> > > > > > > > > > > > select_idle_sibling+0x28/0x430
> > > > > > > > > > > > [Mon Mar 11 14:10:08 2024] ? wake_affine+0x62/0x1f0
> > > > > > > > > > > > [Mon Mar 11 14:10:08 2024] __wait_for_common+0x90/0x1d0
> > > > > > > > > > > > [Mon Mar 11 14:10:08 2024] ?
> > > > > > > > > > > > __pfx_schedule_timeout+0x10/0x10
> > > > > > > > > > > > [Mon Mar 11 14:10:08 2024] __flush_workqueue+0x13a/0x3f0
> > > > > > > > > > > > [Mon Mar 11 14:10:08 2024]
> > > > > > > > > > > > nfsd4_shutdown_callback+0x49/0x120
> > > > > > > > > > > > [nfsd]
> > > > > > > > > > > > [Mon Mar 11 14:10:08 2024] ?
> > > > > > > > > > > > nfsd4_cld_remove+0x54/0x1d0 [nfsd]
> > > > > > > > > > > > [Mon Mar 11 14:10:08 2024] ?
> > > > > > > > > > > > nfsd4_return_all_client_layouts+0xc4/0xf0 [nfsd]
> > > > > > > > > > > > [Mon Mar 11 14:10:08 2024] ?
> > > > > > > > > > > > nfsd4_shutdown_copy+0x68/0xc0 [nfsd]
> > > > > > > > > > > > [Mon Mar 11 14:10:08 2024] __destroy_client+0x1f3/0x290
> > > > > > > > > > > > [nfsd]
> > > > > > > > > > > > [Mon Mar 11 14:10:08 2024]
> > > > > > > > > > > > nfsd4_exchange_id+0x75f/0x770 [nfsd]
> > > > > > > > > > > > [Mon Mar 11 14:10:08 2024] ?
> > > > > > > > > > > > nfsd4_decode_opaque+0x3a/0x90 [nfsd]
> > > > > > > > > > > > [Mon Mar 11 14:10:08 2024]
> > > > > > > > > > > > nfsd4_proc_compound+0x44b/0x700 [nfsd]
> > > > > > > > > > > > [Mon Mar 11 14:10:08 2024] nfsd_dispatch+0x94/0x1c0 [nfsd]
> > > > > > > > > > > > [Mon Mar 11 14:10:08 2024] svc_process_common+0x2ec/0x660
> > > > > > > > > > > > [sunrpc]
> > > > > > > > > > > > [Mon Mar 11 14:10:08 2024] ?
> > > > > > > > > > > > __pfx_nfsd_dispatch+0x10/0x10 [nfsd]
> > > > > > > > > > > > [Mon Mar 11 14:10:08 2024] ? __pfx_nfsd+0x10/0x10 [nfsd]
> > > > > > > > > > > > [Mon Mar 11 14:10:08 2024] svc_process+0x12d/0x170
> > > > > > > > > > > > [sunrpc]
> > > > > > > > > > > > [Mon Mar 11 14:10:08 2024] nfsd+0x84/0xb0 [nfsd]
> > > > > > > > > > > > [Mon Mar 11 14:10:08 2024] kthread+0xdd/0x100
> > > > > > > > > > > > [Mon Mar 11 14:10:08 2024] ? __pfx_kthread+0x10/0x10
> > > > > > > > > > > > [Mon Mar 11 14:10:08 2024] ret_from_fork+0x29/0x50
> > > > > > > > > > > > [Mon Mar 11 14:10:08 2024] </TASK>
> > > > > > > > > > > > [Mon Mar 11 14:10:08 2024] INFO: task nfsd:8866 blocked for
> > > > > > > > > > > > more than 122 seconds.
> > > > > > > > > > > > [Mon Mar 11 14:10:08 2024] Not tainted
> > > > > > > > > > > > 5.14.0-419.el9.x86_64 #1
> > > > > > > > > > > > [Mon Mar 11 14:10:08 2024] "echo 0 >
> > > > > > > > > > > > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> > > > > > > > > > > > [Mon Mar 11 14:10:08 2024]task:nfsd state:D
> > > > > > > > > > > > stack:0
> > > > > > > > > > > > pid:8866 ppid:2 flags:0x00004000
> > > > > > > > > > > > [Mon Mar 11 14:10:08 2024] Call Trace:
> > > > > > > > > > > > [Mon Mar 11 14:10:08 2024] <TASK>
> > > > > > > > > > > > [Mon Mar 11 14:10:08 2024] __schedule+0x21b/0x550
> > > > > > > > > > > > [Mon Mar 11 14:10:08 2024] schedule+0x2d/0x70
> > > > > > > > > > > > [Mon Mar 11 14:10:08 2024] schedule_timeout+0x11f/0x160
> > > > > > > > > > > > [Mon Mar 11 14:10:08 2024] ?
> > > > > > > > > > > > select_idle_sibling+0x28/0x430
> > > > > > > > > > > > [Mon Mar 11 14:10:08 2024] ? tcp_recvmsg+0x196/0x210
> > > > > > > > > > > > [Mon Mar 11 14:10:08 2024] ? wake_affine+0x62/0x1f0
> > > > > > > > > > > > [Mon Mar 11 14:10:08 2024] __wait_for_common+0x90/0x1d0
> > > > > > > > > > > > [Mon Mar 11 14:10:08 2024] ?
> > > > > > > > > > > > __pfx_schedule_timeout+0x10/0x10
> > > > > > > > > > > > [Mon Mar 11 14:10:08 2024] __flush_workqueue+0x13a/0x3f0
> > > > > > > > > > > > [Mon Mar 11 14:10:08 2024]
> > > > > > > > > > > > nfsd4_destroy_session+0x1a4/0x240
> > > > > > > > > > > > [nfsd]
> > > > > > > > > > > > [Mon Mar 11 14:10:08 2024]
> > > > > > > > > > > > nfsd4_proc_compound+0x44b/0x700 [nfsd]
> > > > > > > > > > > > [Mon Mar 11 14:10:08 2024] nfsd_dispatch+0x94/0x1c0 [nfsd]
> > > > > > > > > > > > [Mon Mar 11 14:10:08 2024] svc_process_common+0x2ec/0x660
> > > > > > > > > > > > [sunrpc]
> > > > > > > > > > > > [Mon Mar 11 14:10:08 2024] ?
> > > > > > > > > > > > __pfx_nfsd_dispatch+0x10/0x10 [nfsd]
> > > > > > > > > > > > [Mon Mar 11 14:10:08 2024] ? __pfx_nfsd+0x10/0x10 [nfsd]
> > > > > > > > > > > > [Mon Mar 11 14:10:08 2024] svc_process+0x12d/0x170
> > > > > > > > > > > > [sunrpc]
> > > > > > > > > > > > [Mon Mar 11 14:10:08 2024] nfsd+0x84/0xb0 [nfsd]
> > > > > > > > > > > > [Mon Mar 11 14:10:08 2024] kthread+0xdd/0x100
> > > > > > > > > > > > [Mon Mar 11 14:10:08 2024] ? __pfx_kthread+0x10/0x10
> > > > > > > > > > > > [Mon Mar 11 14:10:08 2024] ret_from_fork+0x29/0x50
> > > > > > > > > > > > [Mon Mar 11 14:10:08 2024] </TASK>
> > > > > > > > > > > >
> > > > > > > > > > > The above threads are trying to flush the workqueue, so that
> > > > > > > > > > > probably
> > > > > > > > > > > means that they are stuck waiting on a workqueue job to finish.
> > > > > > > > > > > > The above is repeated a few times, and then this warning is
> > > > > > > > > > > > also logged:
> > > > > > > > > > > > [Mon Mar 11 14:12:04 2024] ------------[ cut here
> > > > > > > > > > > > ]------------
> > > > > > > > > > > > [Mon Mar 11 14:12:04 2024] WARNING: CPU: 39 PID: 8844 at
> > > > > > > > > > > > fs/nfsd/nfs4state.c:4919 nfsd_break_deleg_cb+0x170/0x190 [nfsd]
> > > > > > > > > > > > [Mon Mar 11 14:12:05 2024] Modules linked in: nfsv4
> > > > > > > > > > > > dns_resolver nfs fscache netfs rpcsec_gss_krb5 rpcrdma rdma_cm
> > > > > > > > > > > > iw_cm ib_cm ib_core binfmt_misc bonding tls rfkill
> > > > > > > > > > > > nft_counter nft_ct
> > > > > > > > > > > > nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_reject_inet
> > > > > > > > > > > > nf_reject_ipv4 nf_reject_ipv6 nft_reject nf_tables nfnetlink
> > > > > > > > > > > > vfat
> > > > > > > > > > > > fat dm_thin_pool dm_persistent_data dm_bio_prison dm_bufio l
> > > > > > > > > > > > ibcrc32c dm_service_time dm_multipath intel_rapl_msr
> > > > > > > > > > > > intel_rapl_common intel_uncore_frequency
> > > > > > > > > > > > intel_uncore_frequency_common isst_if_common skx_edac nfit
> > > > > > > > > > > > libnvdimm ipmi_ssif x86_pkg_temp
> > > > > > > > > > > > _thermal intel_powerclamp coretemp kvm_intel kvm irqbypass
> > > > > > > > > > > > dcdbas rapl intel_cstate mgag200 i2c_algo_bit drm_shmem_helper
> > > > > > > > > > > > dell_smbios drm_kms_helper dell_wmi_descriptor wmi_bmof intel_u
> > > > > > > > > > > > ncore syscopyarea pcspkr sysfillrect mei_me sysimgblt
> > > > > > > > > > > > acpi_ipmi
> > > > > > > > > > > > mei fb_sys_fops i2c_i801 ipmi_si intel_pch_thermal lpc_ich
> > > > > > > > > > > > ipmi_devintf i2c_smbus ipmi_msghandler joydev acpi_power_meter
> > > > > > > > > > > > nfsd auth_rpcgss nfs_acl drm lockd grace fuse sunrpc ext4
> > > > > > > > > > > > mbcache jbd2 sd_mod sg lpfc
> > > > > > > > > > > > [Mon Mar 11 14:12:05 2024] nvmet_fc nvmet nvme_fc
> > > > > > > > > > > > nvme_fabrics
> > > > > > > > > > > > crct10dif_pclmul ahci libahci crc32_pclmul nvme_core
> > > > > > > > > > > > crc32c_intel
> > > > > > > > > > > > ixgbe megaraid_sas libata nvme_common ghash_clmulni_int
> > > > > > > > > > > > el t10_pi wdat_wdt scsi_transport_fc mdio wmi dca dm_mirror
> > > > > > > > > > > > dm_region_hash dm_log dm_mod
> > > > > > > > > > > > [Mon Mar 11 14:12:05 2024] CPU: 39 PID: 8844 Comm: nfsd Not
> > > > > > > > > > > > tainted 5.14.0-419.el9.x86_64 #1
> > > > > > > > > > > > [Mon Mar 11 14:12:05 2024] Hardware name: Dell Inc.
> > > > > > > > > > > > PowerEdge
> > > > > > > > > > > > R740/00WGD1, BIOS 2.20.1 09/13/2023
> > > > > > > > > > > > [Mon Mar 11 14:12:05 2024] RIP:
> > > > > > > > > > > > 0010:nfsd_break_deleg_cb+0x170/0x190 [nfsd]
> > > > > > > > > > > > [Mon Mar 11 14:12:05 2024] Code: a6 95 c5 f3 e9 ff fe ff
> > > > > > > > > > > > ff 48
> > > > > > > > > > > > 89 df be 01 00 00 00 e8 34 b5 13 f4 48 8d bb 98 00 00 00 e8
> > > > > > > > > > > > c8 f9
> > > > > > > > > > > > 00 00 84 c0 0f 85 2e ff ff ff <0f> 0b e9 27 ff ff ff be
> > > > > > > > > > > > 02 00 00 00 48 89 df e8 0c b5 13 f4 e9 01
> > > > > > > > > > > > [Mon Mar 11 14:12:05 2024] RSP: 0018:ffff9929e0bb7b80
> > > > > > > > > > > > EFLAGS:
> > > > > > > > > > > > 00010246
> > > > > > > > > > > > [Mon Mar 11 14:12:05 2024] RAX: 0000000000000000 RBX:
> > > > > > > > > > > > ffff8ada51930900 RCX: 0000000000000024
> > > > > > > > > > > > [Mon Mar 11 14:12:05 2024] RDX: ffff8ada519309c8 RSI:
> > > > > > > > > > > > ffff8ad582933c00 RDI: 0000000000002000
> > > > > > > > > > > > [Mon Mar 11 14:12:05 2024] RBP: ffff8ad46bf21574 R08:
> > > > > > > > > > > > ffff9929e0bb7b48 R09: 0000000000000000
> > > > > > > > > > > > [Mon Mar 11 14:12:05 2024] R10: ffff8aec859a2948 R11:
> > > > > > > > > > > > 0000000000000000 R12: ffff8ad6f497c360
> > > > > > > > > > > > [Mon Mar 11 14:12:05 2024] R13: ffff8ad46bf21560 R14:
> > > > > > > > > > > > ffff8ae5942e0b10 R15: ffff8ad6f497c360
> > > > > > > > > > > > [Mon Mar 11 14:12:05 2024] FS: 0000000000000000(0000)
> > > > > > > > > > > > GS:ffff8b031fcc0000(0000) knlGS:0000000000000000
> > > > > > > > > > > > [Mon Mar 11 14:12:05 2024] CS: 0010 DS: 0000 ES: 0000 CR0:
> > > > > > > > > > > > 0000000080050033
> > > > > > > > > > > > [Mon Mar 11 14:12:05 2024] CR2: 00007fafe2060744 CR3:
> > > > > > > > > > > > 00000018e58de006 CR4: 00000000007706e0
> > > > > > > > > > > > [Mon Mar 11 14:12:05 2024] DR0: 0000000000000000 DR1:
> > > > > > > > > > > > 0000000000000000 DR2: 0000000000000000
> > > > > > > > > > > > [Mon Mar 11 14:12:05 2024] DR3: 0000000000000000 DR6:
> > > > > > > > > > > > 00000000fffe0ff0 DR7: 0000000000000400
> > > > > > > > > > > > [Mon Mar 11 14:12:05 2024] PKRU: 55555554
> > > > > > > > > > > > [Mon Mar 11 14:12:05 2024] Call Trace:
> > > > > > > > > > > > [Mon Mar 11 14:12:05 2024] <TASK>
> > > > > > > > > > > > [Mon Mar 11 14:12:05 2024] ?
> > > > > > > > > > > > show_trace_log_lvl+0x1c4/0x2df
> > > > > > > > > > > > [Mon Mar 11 14:12:05 2024] ?
> > > > > > > > > > > > show_trace_log_lvl+0x1c4/0x2df
> > > > > > > > > > > > [Mon Mar 11 14:12:05 2024] ? __break_lease+0x16f/0x5f0
> > > > > > > > > > > > [Mon Mar 11 14:12:05 2024] ?
> > > > > > > > > > > > nfsd_break_deleg_cb+0x170/0x190
> > > > > > > > > > > > [nfsd]
> > > > > > > > > > > > [Mon Mar 11 14:12:05 2024] ? __warn+0x81/0x110
> > > > > > > > > > > > [Mon Mar 11 14:12:05 2024] ?
> > > > > > > > > > > > nfsd_break_deleg_cb+0x170/0x190
> > > > > > > > > > > > [nfsd]
> > > > > > > > > > > > [Mon Mar 11 14:12:05 2024] ? report_bug+0x10a/0x140
> > > > > > > > > > > > [Mon Mar 11 14:12:05 2024] ? handle_bug+0x3c/0x70
> > > > > > > > > > > > [Mon Mar 11 14:12:05 2024] ? exc_invalid_op+0x14/0x70
> > > > > > > > > > > > [Mon Mar 11 14:12:05 2024] ? asm_exc_invalid_op+0x16/0x20
> > > > > > > > > > > > [Mon Mar 11 14:12:05 2024] ?
> > > > > > > > > > > > nfsd_break_deleg_cb+0x170/0x190
> > > > > > > > > > > > [nfsd]
> > > > > > > > > > > > [Mon Mar 11 14:12:05 2024] __break_lease+0x16f/0x5f0
> > > > > > > > > > > > [Mon Mar 11 14:12:05 2024] ?
> > > > > > > > > > > > nfsd_file_lookup_locked+0x117/0x160 [nfsd]
> > > > > > > > > > > > [Mon Mar 11 14:12:05 2024] ? list_lru_del+0x101/0x150
> > > > > > > > > > > > [Mon Mar 11 14:12:05 2024]
> > > > > > > > > > > > nfsd_file_do_acquire+0x790/0x830
> > > > > > > > > > > > [nfsd]
> > > > > > > > > > > > [Mon Mar 11 14:12:05 2024]
> > > > > > > > > > > > nfs4_get_vfs_file+0x315/0x3a0 [nfsd]
> > > > > > > > > > > > [Mon Mar 11 14:12:05 2024]
> > > > > > > > > > > > nfsd4_process_open2+0x430/0xa30 [nfsd]
> > > > > > > > > > > > [Mon Mar 11 14:12:05 2024] ? fh_verify+0x297/0x2f0 [nfsd]
> > > > > > > > > > > > [Mon Mar 11 14:12:05 2024] nfsd4_open+0x3ce/0x4b0 [nfsd]
> > > > > > > > > > > > [Mon Mar 11 14:12:05 2024]
> > > > > > > > > > > > nfsd4_proc_compound+0x44b/0x700 [nfsd]
> > > > > > > > > > > > [Mon Mar 11 14:12:05 2024] nfsd_dispatch+0x94/0x1c0 [nfsd]
> > > > > > > > > > > > [Mon Mar 11 14:12:05 2024] svc_process_common+0x2ec/0x660
> > > > > > > > > > > > [sunrpc]
> > > > > > > > > > > > [Mon Mar 11 14:12:05 2024] ?
> > > > > > > > > > > > __pfx_nfsd_dispatch+0x10/0x10 [nfsd]
> > > > > > > > > > > > [Mon Mar 11 14:12:05 2024] ? __pfx_nfsd+0x10/0x10 [nfsd]
> > > > > > > > > > > > [Mon Mar 11 14:12:05 2024] svc_process+0x12d/0x170
> > > > > > > > > > > > [sunrpc]
> > > > > > > > > > > > [Mon Mar 11 14:12:05 2024] nfsd+0x84/0xb0 [nfsd]
> > > > > > > > > > > > [Mon Mar 11 14:12:05 2024] kthread+0xdd/0x100
> > > > > > > > > > > > [Mon Mar 11 14:12:05 2024] ? __pfx_kthread+0x10/0x10
> > > > > > > > > > > > [Mon Mar 11 14:12:05 2024] ret_from_fork+0x29/0x50
> > > > > > > > > > > > [Mon Mar 11 14:12:05 2024] </TASK>
> > > > > > > > > > > > [Mon Mar 11 14:12:05 2024] ---[ end trace
> > > > > > > > > > > > 7a039e17443dc651 ]---
> > > > > > > > > > > This is probably this WARN in nfsd_break_one_deleg:
> > > > > > > > > > >
> > > > > > > > > > > WARN_ON_ONCE(!nfsd4_run_cb(&dp->dl_recall));
> > > > > > > > > > >
> > > > > > > > > > > It means that a delegation break callback to the client
> > > > > > > > > > > couldn't be
> > > > > > > > > > > queued to the workqueue, and so it didn't run.
> > > > > > > > > > >
> > > > > > > > > > > > Could this be the same issue as described
> > > > > > > > > > > > here:https://urldefense.com/v3/__https://lore.kernel.org/linux-nfs/af0ec881-5ebf-4feb-98ae-3ed2a77f86f1@oracle.com/__;!!ACWV5N9M2RV99hQ!LV3yWeoSOhNAkRHkxFCH2tlm0iNFVD78mxnSLyP6lrX7yBVeA2TOJ4nv6oZsqLwP4kW56CMpDWhkjjwSkdBV9En7$
> > > > > > > > > > > > ?
> > > > > > > > > > > Yes, most likely the same problem.
> > > > > > > > > > If I read that thread correctly, this issue was introduced
> > > > > > > > > > between
> > > > > > > > > > 6.1.63 and 6.6.3? Is it possible the EL9 5.14.0-362.18.1.el9_3
> > > > > > > > > > backported these changes, or were we hitting some other bug
> > > > > > > > > > with that
> > > > > > > > > > version? It seems the 6.1.x kernel is not affected? If so, that
> > > > > > > > > > would be
> > > > > > > > > > the recommended kernel to run?
> > > > > > > > > Anything is possible. We have to identify the problem first.
> > > > > > > > > > > > As described in that thread, I've tried to obtain the requested
> > > > > > > > > > > > information.
> > > > > > > > > > > >
> > > > > > > > > > > > Is it possible this is the issue that was fixed by the patches
> > > > > > > > > > > > described
> > > > > > > > > > > > here?https://urldefense.com/v3/__https://lore.kernel.org/linux-nfs/2024022054-cause-suffering-eae8@gregkh/__;!!ACWV5N9M2RV99hQ!LV3yWeoSOhNAkRHkxFCH2tlm0iNFVD78mxnSLyP6lrX7yBVeA2TOJ4nv6oZsqLwP4kW56CMpDWhkjjwSkedtUP09$
> > > > > > > > > > > >
> > > > > > > > > > > Doubtful. Those are targeted toward a different set of issues.
> > > > > > > > > > >
> > > > > > > > > > > If you're willing, I do have some patches queued up for
> > > > > > > > > > > CentOS here
> > > > > > > > > > > that
> > > > > > > > > > > fix some backchannel problems that could be related. I'm mainly
> > > > > > > > > > > waiting
> > > > > > > > > > > on Chuck to send these to Linus and then we'll likely merge
> > > > > > > > > > > them into
> > > > > > > > > > > CentOS soon afterward:
> > > > > > > > > > >
> > > > > > > > > > > https://urldefense.com/v3/__https://gitlab.com/redhat/centos-stream/src/kernel/centos-stream-9/-/merge_requests/3689__;!!ACWV5N9M2RV99hQ!LV3yWeoSOhNAkRHkxFCH2tlm0iNFVD78mxnSLyP6lrX7yBVeA2TOJ4nv6oZsqLwP4kW56CMpDWhkjjwSkdvDn8y7$
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > If you can send me a patch file, I can rebuild the C9S kernel
> > > > > > > > > > with that
> > > > > > > > > > patch and run it. It can take a while for the bug to trigger as I
> > > > > > > > > > believe it seems to be very workload dependent (we were
> > > > > > > > > > running very
> > > > > > > > > > stable for months and now hit this bug every other week).
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > It's probably simpler to just pull down the build artifacts for
> > > > > > > > > that MR.
> > > > > > > > > You have to drill down through the CI for it, but they are here:
> > > > > > > > >
> > > > > > > > > https://urldefense.com/v3/__https://s3.amazonaws.com/arr-cki-prod-trusted-artifacts/index.html?prefix=trusted-artifacts*1194300175*publish_x86_64*6278921877*artifacts*__;Ly8vLy8!!ACWV5N9M2RV99hQ!LV3yWeoSOhNAkRHkxFCH2tlm0iNFVD78mxnSLyP6lrX7yBVeA2TOJ4nv6oZsqLwP4kW56CMpDWhkjjwSkaP5eW8V$
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > There's even a repo file you can install on the box to pull
> > > > > > > > > them down.
> > > > > > > > We installed this kernel on the server 3 days ago. Today, a user
> > > > > > > > informed us that their screen was black after logging in.
> > > > > > > > Similar to
> > > > > > > > other occurrences of this issue, the mount command on the client
> > > > > > > > was
> > > > > > > > hung. But in contrast to the other times, there were no messages in
> > > > > > > > the logs kernel logs on the server. Even restarting the client does
> > > > > > > > not resolve the issue.
> > > > > >
> > > > > > Ok, so you rebooted the client and it's still unable to mount? That
> > > > > > sounds like a server problem if so.
> > > > > >
> > > > > > Are both client and server running the same kernel?
> > > > > No, the server runs 5.14.0-427.3689_1194299994.el9 and the client
> > > > > 5.14.0-362.18.1.el9_3.
> > > > > >
> > > > > > > > Something still seems to be wrong on the server though. When I
> > > > > > > > look at
> > > > > > > > the directories under /proc/fs/nfsd/clients, there's still a
> > > > > > > > directory
> > > > > > > > for the specific client, even though it's no longer running:
> > > > > > > >
> > > > > > > > # cat 155/info
> > > > > > > > clientid: 0xc8edb7f65f4a9ad
> > > > > > > > address: "10.87.31.152:819"
> > > > > > > > status: confirmed
> > > > > > > > seconds from last renew: 33163
> > > > > > > > name: "Linux NFSv4.2 bersalis.esat.kuleuven.be"
> > > > > > > > minor version: 2
> > > > > > > > Implementation domain: "kernel.org"
> > > > > > > > Implementation name: "Linux 5.14.0-362.18.1.el9_3.0.1.x86_64 #1 SMP
> > > > > > > > PREEMPT_DYNAMIC Sun Feb 11 13:49:23 UTC 2024 x86_64"
> > > > > > > > Implementation time: [0, 0]
> > > > > > > > callback state: DOWN
> > > > > > > > callback address: 10.87.31.152:0
> > > > > > > >
> > > > > > If you just shut down the client, the server won't immediately
> > > > > > purge its
> > > > > > record. In fact, assuming you're running the same kernel on the
> > > > > > server,
> > > > > > it won't purge the client record until there is a conflicting request
> > > > > > for its state.
> > > > > Is there a way to force such a conflicting request (to get the
> > > > > client record to purge)?
> > > >
> > > > Try:
> > > >
> > > > # echo "expire" > /proc/fs/nfsd/clients/155/ctl
> > >
> > > I've tried that. The command hangs and can not be interrupted with
> > > ctrl-c.
> > > I've now also noticed in the dmesg output that the kernel issued the
> > > following WARNING a few hours ago. It wasn't directly triggered by
> > > the echo command above, but seems to have been triggered a few hours
> > > ago (probably when another client started to have the same problem as
> > > more clients are experiencing issues now).
> >
> > I think this warning message is harmless. However it indicates potential
> > problem with the workqueue which might be related to memory shortage.
> >
> > What the output of 'cat /proc/meminfo' looks like?
>
> I doubt the current values are useful, but they are:
>
> MemTotal: 196110860 kB
> MemFree: 29357112 kB
> MemAvailable: 179529420 kB
> Buffers: 11996096 kB
> Cached: 130589396 kB
> SwapCached: 52 kB
> Active: 1136988 kB
> Inactive: 144192468 kB
> Active(anon): 698564 kB
> Inactive(anon): 2657256 kB
> Active(file): 438424 kB
> Inactive(file): 141535212 kB
> Unevictable: 72140 kB
> Mlocked: 69068 kB
> SwapTotal: 67108860 kB
> SwapFree: 67106276 kB
> Zswap: 0 kB
> Zswapped: 0 kB
> Dirty: 80812 kB
> Writeback: 0 kB
> AnonPages: 2806592 kB
> Mapped: 322700 kB
> Shmem: 599308 kB
> KReclaimable: 16977000 kB
> Slab: 18898736 kB
> SReclaimable: 16977000 kB
> SUnreclaim: 1921736 kB
> KernelStack: 18128 kB
> PageTables: 31716 kB
> SecPageTables: 0 kB
> NFS_Unstable: 0 kB
> Bounce: 0 kB
> WritebackTmp: 0 kB
> CommitLimit: 165164288 kB
> Committed_AS: 5223940 kB
> VmallocTotal: 34359738367 kB
> VmallocUsed: 300064 kB
> VmallocChunk: 0 kB
> Percpu: 45888 kB
> HardwareCorrupted: 0 kB
> AnonHugePages: 2451456 kB
> ShmemHugePages: 0 kB
> ShmemPmdMapped: 0 kB
> FileHugePages: 0 kB
> FilePmdMapped: 0 kB
> CmaTotal: 0 kB
> CmaFree: 0 kB
> Unaccepted: 0 kB
> HugePages_Total: 0
> HugePages_Free: 0
> HugePages_Rsvd: 0
> HugePages_Surp: 0
> Hugepagesize: 2048 kB
> Hugetlb: 0 kB
> DirectMap4k: 1303552 kB
> DirectMap2M: 28715008 kB
> DirectMap1G: 171966464 kB
>
>
> >
> > Did you try 'echo 3 > /proc/sys/vm/drop_caches'?
>
> Yes, I tried that when the first client hit the issue, but it didn't
> result in any unlocking of the client.
>
>
> >
> > >
> > > [Tue Mar 19 14:53:44 2024] ------------[ cut here ]------------
> > > [Tue Mar 19 14:53:44 2024] WARNING: CPU: 44 PID: 5843 at
> > > fs/nfsd/nfs4state.c:4920 nfsd_break_deleg_cb+0x170/0x190 [nfsd]
> > > [Tue Mar 19 14:53:44 2024] Modules linked in: nf_conntrack_netlink
> > > nfsv4 dns_resolver nfs fscache netfs binfmt_misc xsk_diag
> > > rpcsec_gss_krb5 rpcrdma rdma_cm iw_cm ib_cm ib_core bonding tls
> > > rfkill nft_counter nft_ct nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4
> > > nft_reject_inet nf_reject_ipv4 nf_reject_ipv6 nft_reject nf_tables
> > > nfnetlink vfat fat dm_thin_pool dm_persistent_data dm_bio_prison
> > > dm_bufio libcrc32c dm_service_time dm_multipath intel_rapl_msr
> > > intel_rapl_common intel_uncore_frequency
> > > intel_uncore_frequency_common isst_if_common skx_edac nfit libnvdimm
> > > x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm dcdbas
> > > irqbypass ipmi_ssif rapl intel_cstate mgag200 i2c_algo_bit
> > > drm_shmem_helper drm_kms_helper dell_smbios syscopyarea intel_uncore
> > > sysfillrect wmi_bmof dell_wmi_descriptor pcspkr sysimgblt fb_sys_fops
> > > mei_me i2c_i801 mei intel_pch_thermal acpi_ipmi i2c_smbus lpc_ich
> > > ipmi_si ipmi_devintf ipmi_msghandler joydev acpi_power_meter nfsd
> > > nfs_acl lockd auth_rpcgss grace drm fuse sunrpc ext4
> > > [Tue Mar 19 14:53:44 2024] mbcache jbd2 sd_mod sg lpfc nvmet_fc
> > > nvmet nvme_fc nvme_fabrics crct10dif_pclmul crc32_pclmul nvme_core
> > > ixgbe crc32c_intel ahci libahci nvme_common megaraid_sas t10_pi
> > > ghash_clmulni_intel wdat_wdt libata scsi_transport_fc mdio dca wmi
> > > dm_mirror dm_region_hash dm_log dm_mod
> > > [Tue Mar 19 14:53:44 2024] CPU: 44 PID: 5843 Comm: nfsd Not tainted
> > > 5.14.0-427.3689_1194299994.el9.x86_64 #1
> > > [Tue Mar 19 14:53:44 2024] Hardware name: Dell Inc. PowerEdge
> > > R740/00WGD1, BIOS 2.20.1 09/13/2023
> > > [Tue Mar 19 14:53:44 2024] RIP: 0010:nfsd_break_deleg_cb+0x170/0x190
> > > [nfsd]
> > > [Tue Mar 19 14:53:44 2024] Code: 76 76 cd de e9 ff fe ff ff 48 89 df
> > > be 01 00 00 00 e8 34 a1 1b df 48 8d bb 98 00 00 00 e8 a8 fe 00 00 84
> > > c0 0f 85 2e ff ff ff <0f> 0b e9 27 ff ff ff be 02 00 00 00 48 89 df
> > > e8 0c a1 1b df e9 01
> > > [Tue Mar 19 14:53:44 2024] RSP: 0018:ffffb2878f2cfc38 EFLAGS: 00010246
> > > [Tue Mar 19 14:53:44 2024] RAX: 0000000000000000 RBX:
> > > ffff88d5171067b8 RCX: 0000000000000000
> > > [Tue Mar 19 14:53:44 2024] RDX: ffff88d517106880 RSI:
> > > ffff88bdceec8600 RDI: 0000000000002000
> > > [Tue Mar 19 14:53:44 2024] RBP: ffff88d68a38a284 R08:
> > > ffffb2878f2cfc00 R09: 0000000000000000
> > > [Tue Mar 19 14:53:44 2024] R10: ffff88bf57dd7878 R11:
> > > 0000000000000000 R12: ffff88d5b79c4798
> > > [Tue Mar 19 14:53:44 2024] R13: ffff88d68a38a270 R14:
> > > ffff88cab06ad0c8 R15: ffff88d5b79c4798
> > > [Tue Mar 19 14:53:44 2024] FS: 0000000000000000(0000)
> > > GS:ffff88d4a1180000(0000) knlGS:0000000000000000
> > > [Tue Mar 19 14:53:44 2024] CS: 0010 DS: 0000 ES: 0000 CR0:
> > > 0000000080050033
> > > [Tue Mar 19 14:53:44 2024] CR2: 00007fe46ef90000 CR3:
> > > 000000019d010004 CR4: 00000000007706e0
> > > [Tue Mar 19 14:53:44 2024] DR0: 0000000000000000 DR1:
> > > 0000000000000000 DR2: 0000000000000000
> > > [Tue Mar 19 14:53:44 2024] DR3: 0000000000000000 DR6:
> > > 00000000fffe0ff0 DR7: 0000000000000400
> > > [Tue Mar 19 14:53:44 2024] PKRU: 55555554
> > > [Tue Mar 19 14:53:44 2024] Call Trace:
> > > [Tue Mar 19 14:53:44 2024] <TASK>
> > > [Tue Mar 19 14:53:44 2024] ? show_trace_log_lvl+0x1c4/0x2df
> > > [Tue Mar 19 14:53:44 2024] ? show_trace_log_lvl+0x1c4/0x2df
> > > [Tue Mar 19 14:53:44 2024] ? __break_lease+0x16f/0x5f0
> > > [Tue Mar 19 14:53:44 2024] ? nfsd_break_deleg_cb+0x170/0x190 [nfsd]
> > > [Tue Mar 19 14:53:44 2024] ? __warn+0x81/0x110
> > > [Tue Mar 19 14:53:44 2024] ? nfsd_break_deleg_cb+0x170/0x190 [nfsd]
> > > [Tue Mar 19 14:53:44 2024] ? report_bug+0x10a/0x140
> > > [Tue Mar 19 14:53:44 2024] ? handle_bug+0x3c/0x70
> > > [Tue Mar 19 14:53:44 2024] ? exc_invalid_op+0x14/0x70
> > > [Tue Mar 19 14:53:44 2024] ? asm_exc_invalid_op+0x16/0x20
> > > [Tue Mar 19 14:53:44 2024] ? nfsd_break_deleg_cb+0x170/0x190 [nfsd]
> > > [Tue Mar 19 14:53:44 2024] ? nfsd_break_deleg_cb+0x96/0x190 [nfsd]
> > > [Tue Mar 19 14:53:44 2024] __break_lease+0x16f/0x5f0
> > > [Tue Mar 19 14:53:44 2024] nfs4_get_vfs_file+0x164/0x3a0 [nfsd]
> > > [Tue Mar 19 14:53:44 2024] nfsd4_process_open2+0x430/0xa30 [nfsd]
> > > [Tue Mar 19 14:53:44 2024] ? fh_verify+0x297/0x2f0 [nfsd]
> > > [Tue Mar 19 14:53:44 2024] nfsd4_open+0x3ce/0x4b0 [nfsd]
> > > [Tue Mar 19 14:53:44 2024] nfsd4_proc_compound+0x44b/0x700 [nfsd]
> > > [Tue Mar 19 14:53:44 2024] nfsd_dispatch+0x94/0x1c0 [nfsd]
> > > [Tue Mar 19 14:53:44 2024] svc_process_common+0x2ec/0x660 [sunrpc]
> > > [Tue Mar 19 14:53:44 2024] ? __pfx_nfsd_dispatch+0x10/0x10 [nfsd]
> > > [Tue Mar 19 14:53:44 2024] ? __pfx_nfsd+0x10/0x10 [nfsd]
> > > [Tue Mar 19 14:53:44 2024] svc_process+0x12d/0x170 [sunrpc]
> > > [Tue Mar 19 14:53:44 2024] nfsd+0x84/0xb0 [nfsd]
> > > [Tue Mar 19 14:53:44 2024] kthread+0xdd/0x100
> > > [Tue Mar 19 14:53:44 2024] ? __pfx_kthread+0x10/0x10
> > > [Tue Mar 19 14:53:44 2024] ret_from_fork+0x29/0x50
> > > [Tue Mar 19 14:53:44 2024] </TASK>
> > > [Tue Mar 19 14:53:44 2024] ---[ end trace ed0b2b3f135c637d ]---
> > >
> > > It again seems to have been triggered in nfsd_break_deleg_cb?
> > >
> > > I also had the following perf command running a tmux on the server:
> > >
> > > perf trace -e nfsd:nfsd_cb_recall_any
> > >
> > > This has spewed a lot of messages. I'm including a short list here:
> > >
> > > ...
> > >
> > > 33464866.721 kworker/u98:5/1591466 nfsd:nfsd_cb_recall_any(cl_boot:
> > > 1710533037, cl_id: 210688785, bmval0: 1, addr: 0x7f331bb116c8)
> > > 33464866.724 kworker/u98:5/1591466 nfsd:nfsd_cb_recall_any(cl_boot:
> > > 1710533037, cl_id: 210688827, bmval0: 1, addr: 0x7f331bb11738)
> > > 33464866.729 kworker/u98:5/1591466 nfsd:nfsd_cb_recall_any(cl_boot:
> > > 1710533037, cl_id: 210688767, bmval0: 1, addr: 0x7f331bb117a8)
> > > 33464866.732 kworker/u98:5/1591466 nfsd:nfsd_cb_recall_any(cl_boot:
> > > 1710533037, cl_id: 210718132, bmval0: 1, addr: 0x7f331bb11818)
> > > 33464866.737 kworker/u98:5/1591466 nfsd:nfsd_cb_recall_any(cl_boot:
> > > 1710533037, cl_id: 210688952, bmval0: 1, addr: 0x7f331bb11888)
> > > 33464866.741 kworker/u98:5/1591466 nfsd:nfsd_cb_recall_any(cl_boot:
> > > 1710533037, cl_id: 210702355, bmval0: 1, addr: 0x7f331bb118f8)
> > > 33868414.001 kthreadd/1597068 nfsd:nfsd_cb_recall_any(cl_boot:
> > > 1710533037, cl_id: 210688751, bmval0: 1, addr: 0x7f331be68620)
> > > 33868414.014 kthreadd/1597068 nfsd:nfsd_cb_recall_any(cl_boot:
> > > 1710533037, cl_id: 210718536, bmval0: 1, addr: 0x7f331be68690)
> > > 33868414.018 kthreadd/1597068 nfsd:nfsd_cb_recall_any(cl_boot:
> > > 1710533037, cl_id: 210719074, bmval0: 1, addr: 0x7f331be68700)
> > > 33868414.022 kthreadd/1597068 nfsd:nfsd_cb_recall_any(cl_boot:
> > > 1710533037, cl_id: 210688916, bmval0: 1, addr: 0x7f331be68770)
> > > 33868414.026 kthreadd/1597068 nfsd:nfsd_cb_recall_any(cl_boot:
> > > 1710533037, cl_id: 210688941, bmval0: 1, addr: 0x7f331be687e0)
> > > ...
> > >
> > > 33868414.924 kthreadd/1597068 nfsd:nfsd_cb_recall_any(cl_boot:
> > > 1710533037, cl_id: 210688744, bmval0: 1, addr: 0x7f331be6d7f0)
> > > 33868414.929 kthreadd/1597068 nfsd:nfsd_cb_recall_any(cl_boot:
> > > 1710533037, cl_id: 210717223, bmval0: 1, addr: 0x7f331be6d860)
> > > 33868414.934 kthreadd/1597068 nfsd:nfsd_cb_recall_any(cl_boot:
> > > 1710533037, cl_id: 210716137, bmval0: 1, addr: 0x7f331be6d8d0)
> > > 34021240.903 kworker/u98:5/1591466 nfsd:nfsd_cb_recall_any(cl_boot:
> > > 1710533037, cl_id: 210688941, bmval0: 1, addr: 0x7f331c207de8)
> > > 34021240.917 kworker/u98:5/1591466 nfsd:nfsd_cb_recall_any(cl_boot:
> > > 1710533037, cl_id: 210718750, bmval0: 1, addr: 0x7f331c207e58)
> > > 34021240.922 kworker/u98:5/1591466 nfsd:nfsd_cb_recall_any(cl_boot:
> > > 1710533037, cl_id: 210688955, bmval0: 1, addr: 0x7f331c207ec8)
> > > 34021240.925 kworker/u98:5/1591466 nfsd:nfsd_cb_recall_any(cl_boot:
> > > 1710533037, cl_id: 210688975, bmval0: 1, addr: 0x7f331c207f38)
> > > ...
> > >
> > > I assume the cl_id is the client id? How can I map this to a client
> > > from /proc/fs/nfsd/clients?
> >
> > The hex value of 'clientid' printed from /proc/fs/nfsd/clients/XX/info
> > is a 64-bit value composed of:
> >
> > typedef struct {
> > u32 cl_boot;
> > u32 cl_id;
> > } clientid_t
> >
> > For example:
> >
> > clientid: 0xc8edb7f65f4a9ad
> >
> > cl_boot: 65f4a9add (1710533037)
> > cl_id: c8edb7f (21068895)
> >
> > This should match a trace event with:
> >
> > nfsd:nfsd_cb_recall_any(cl_boot: 1710533037, cl_id: 21068895, bmval0:
> > XX, addr: 0xYYYYY)
> >
> > >
> > > If I understand it correctly, the recall_any should be called when
> > > either the system starts to experience memory pressure,
> >
> > yes.
> It seems odd that the system gets in such a state that has such high
> memory pressure. It doesn't run much else than NFS and Samba.
> >
> > > or it reaches the delegation limits?
> >
> > No, this feature was added to nfsd very recently. I don't think your
> > kernel has it.
> >
> > > I doubt the system is actually running out of memory here as there
> > > are no other indications.
> > > Shouldn't I get those "page allocation failure" messages if it does?
> > > How can I check the number of delegations/leases currently issued,
> > > what the current maximum is and how to increase it?
> >
> > Max delegations is 4 per 1MB of available memory. There is no
> > admin tool to adjust this value.
> /proc/locks currently has about 130k DELEG lines, so that should be a
> lot lower than the limit on a 192G ram server.
>
>
> >
> > I do not recommend running a production system with delegation
> > disabled. But for this specific issue, it might help to temporarily
> > disable delegation to isolate problem areas.
>
>
> I'm going to reboot the system with the 6.1.82 kernel (kernel-lt from
> elrepo). Maybe it has less new modern developments that may have
> introduced this.
>
If v6.1-ish kernel turns out to not help, then you may want to give a
v6.7 or v6.8 kernel a try. It helps if we know whether this problem is
reproducible on in more up to date kernels.
> I've been able to reproduce the situation on an additional client now
> that the issue happens on the server:
>
> 1. Log in on a client and mount the NFS share.
> 2. Open a file from the NFS share in vim so the client gets a read
> delegation from the server
> 3. Verify on the server in /proc/fs/nfsd/clients/*/states that the
> client has a delegation for the file
> 4. Forcefully reboot the client by running 'echo b > /proc/sysrq-trigger'
> 5. Watch the /proc/fs/nfsd/clients/*/info file on the server.
>
> The "seconds from last renew" will go up and at some point the callback
> state changes to "FAULT". Even when the lease delegation time (90s by
> default?) is over, the
>
> seconds from last renew keeps increasing. At some point the callback
> state changes to "DOWN". When the client is up again and remounts the
> share, the mount hangs on the client
>
> and on the server I notice there's a second directory for this client in
> the clients directory, even though the clientid is the same. The
> callback state for this new client is "UNKNOWN" and the callback address
> is "(einval)".
>
> This is on a client running Fedora 39 with the 6.7.9 kernel.
>
I'm a little unclear...do the above steps work correctly when the server
isn't in this state? I assume the above steps are not sufficient to
cause a problem when the server is behaving normally?
>
> I don't know yet if the same procedure can be used to trigger the
> behavior after the server is rebooted. I'm going to try to reproduce
> this on another system first.
>
> I would expect the delegations to expire automatically after 90s, but
> they remain in the states file of the "DOWN" client.
>
That would have been true a year or so ago, but there were some recent
changes to make the server more "courteous" toward clients that lose
contact for a while. If there are no conflicting requests for the state
they hold then the server will hold onto the lease (basically)
indefinitely, until there is such a conflict.
The client _should_ be able to log in and it cancel the old client
record though. It sounds like that's not working properly for some
reason and it's interfering with the ability to do a CREATE_SESSION.
>
> >
> > -Dai
> >
> > >
> > > Regarding the recall any call: from what I've read on kernelnewbies,
> > > this feature was introduced in the 6.2 kernel? When I look at the
> > > tree for 6.1.x, it was backported in 6.1.81? Is there a way to
> > > disable this support somehow?
> > >
> > > Regards,
> > >
> > > Rik
> > >
> > >
> > > >
> > > > -Dai
> > > >
> > > > > >
> > > > > >
> > > > > > > The nfsdclnts command for this client shows the following
> > > > > > > delegations:
> > > > > > >
> > > > > > > # nfsdclnts -f 155/states -t all
> > > > > > > Inode number | Type | Access | Deny | ip address | Filename
> > > > > > > 169346743 | open | r- | -- | 10.87.31.152:819 |
> > > > > > > disconnected dentry
> > > > > > > 169346743 | deleg | r | | 10.87.31.152:819 |
> > > > > > > disconnected dentry
> > > > > > > 169346746 | open | r- | -- | 10.87.31.152:819 |
> > > > > > > disconnected dentry
> > > > > > > 169346746 | deleg | r | | 10.87.31.152:819 |
> > > > > > > disconnected dentry
> > > > > > >
> > > > > > > I see a lot of recent patches regarding directory delegations. Could
> > > > > > > this be related to this?
> > > > > > >
> > > > > > > Will a 5.14.0-362.18.1.el9_3.0.1 kernel try to use a directory
> > > > > > > delegation?
> > > > > > >
> > > > > > >
> > > > > > No. Directory delegations are a new feature that's still under
> > > > > > development. They use some of the same machinery as file delegations,
> > > > > > but they wouldn't be a factor here.
> > > > > >
> > > > > > > > The system seems to have identified that the client is no longer
> > > > > > > > reachable, but the client entry does not go away. When a mount was
> > > > > > > > hanging on the client, there would be two directories in clients
> > > > > > > > for
> > > > > > > > the same client. Killing the mount command clears up the second
> > > > > > > > entry.
> > > > > > > >
> > > > > > > > Even after running conntrack -D on the server to remove the tcp
> > > > > > > > connection from the conntrack table, the entry doesn't go away
> > > > > > > > and the
> > > > > > > > client still can not mount anything from the server.
> > > > > > > >
> > > > > > > > A tcpdump on the client while a mount was running logged the
> > > > > > > > following
> > > > > > > > messages over and over again:
> > > > > > > >
> > > > > > > > request:
> > > > > > > >
> > > > > > > > Frame 1: 378 bytes on wire (3024 bits), 378 bytes captured (3024
> > > > > > > > bits)
> > > > > > > > Ethernet II, Src: HP_19:7d:4b (e0:73:e7:19:7d:4b), Dst:
> > > > > > > > ArubaaHe_f9:8e:00 (88:3a:30:f9:8e:00)
> > > > > > > > Internet Protocol Version 4, Src: 10.87.31.152, Dst: 10.86.18.14
> > > > > > > > Transmission Control Protocol, Src Port: 932, Dst Port: 2049,
> > > > > > > > Seq: 1,
> > > > > > > > Ack: 1, Len: 312
> > > > > > > > Remote Procedure Call, Type:Call XID:0x1d3220c4
> > > > > > > > Network File System
> > > > > > > > [Program Version: 4]
> > > > > > > > [V4 Procedure: COMPOUND (1)]
> > > > > > > > GSS Data, Ops(1): CREATE_SESSION
> > > > > > > > Length: 152
> > > > > > > > GSS Sequence Number: 76
> > > > > > > > Tag: <EMPTY>
> > > > > > > > minorversion: 2
> > > > > > > > Operations (count: 1): CREATE_SESSION
> > > > > > > > [Main Opcode: CREATE_SESSION (43)]
> > > > > > > > GSS Checksum:
> > > > > > > > 00000028040404ffffffffff000000002c19055f1f8d442d594c13849628affc2797cbb2…
> > > > > > > >
> > > > > > > > GSS Token Length: 40
> > > > > > > > GSS-API Generic Security Service Application Program
> > > > > > > > Interface
> > > > > > > > krb5_blob:
> > > > > > > > 040404ffffffffff000000002c19055f1f8d442d594c13849628affc2797cbb23fa080b0…
> > > > > > > >
> > > > > > > >
> > > > > > > > response:
> > > > > > > >
> > > > > > > > Frame 2: 206 bytes on wire (1648 bits), 206 bytes captured (1648
> > > > > > > > bits)
> > > > > > > > Ethernet II, Src: ArubaaHe_f9:8e:00 (88:3a:30:f9:8e:00), Dst:
> > > > > > > > HP_19:7d:4b (e0:73:e7:19:7d:4b)
> > > > > > > > Internet Protocol Version 4, Src: 10.86.18.14, Dst: 10.87.31.152
> > > > > > > > Transmission Control Protocol, Src Port: 2049, Dst Port: 932,
> > > > > > > > Seq: 1,
> > > > > > > > Ack: 313, Len: 140
> > > > > > > > Remote Procedure Call, Type:Reply XID:0x1d3220c4
> > > > > > > > Network File System
> > > > > > > > [Program Version: 4]
> > > > > > > > [V4 Procedure: COMPOUND (1)]
> > > > > > > > GSS Data, Ops(1): CREATE_SESSION(NFS4ERR_DELAY)
> > > > > > > > Length: 24
> > > > > > > > GSS Sequence Number: 76
> > > > > > > > Status: NFS4ERR_DELAY (10008)
> > > > > > > > Tag: <EMPTY>
> > > > > > > > Operations (count: 1)
> > > > > > > > [Main Opcode: CREATE_SESSION (43)]
> > > > > > > > GSS Checksum:
> > > > > > > > 00000028040405ffffffffff000000000aa742d0798deaad1a8aa2d7c3a91bf4f6274222…
> > > > > > > >
> > > > > > > > GSS Token Length: 40
> > > > > > > > GSS-API Generic Security Service Application Program
> > > > > > > > Interface
> > > > > > > > krb5_blob:
> > > > > > > > 040405ffffffffff000000000aa742d0798deaad1a8aa2d7c3a91bf4f627422226d74923…
> > > > > > > >
> > > > > > > >
> > > > > > > > I was hoping that giving the client a different IP address would
> > > > > > > > resolve the issue for this client, but it didn't. Even though the
> > > > > > > > client had a new IP address (hostname was kept the same), it
> > > > > > > > failed to
> > > > > > > > mount anything from the server.
> > > > > > > >
> > > > > > Changing the IP address won't help. The client is probably using the
> > > > > > same long-form client id as before, so the server still identifies
> > > > > > the
> > > > > > client even with the address change.
> > > > > How is the client id determined? Will changing the hostname of the
> > > > > client trigger a change of the client id?
> > > > > >
> > > > > > Unfortunately, the cause of an NFS4ERR_DELAY error is tough to guess.
> > > > > > The client is expected to back off and retry, so if the server keeps
> > > > > > returning that repeatedly, then a hung mount command is expected.
> > > > > >
> > > > > > The question is why the server would keep returning DELAY. A lot of
> > > > > > different problems ranging from memory allocation issues to protocol
> > > > > > problems can result in that error. You may want to check the NFS
> > > > > > server
> > > > > > and see if anything was logged there.
> > > > > There are no messages in the system logs that indicate any sort of
> > > > > memory issue. We also increased the min_kbytes_free sysctl to 2G on
> > > > > the server before we restarted it with the newer kernel.
> > > > > >
> > > > > > This is on a CREATE_SESSION call, so I wonder if the record held
> > > > > > by the
> > > > > > (courteous) server is somehow blocking the attempt to reestablish the
> > > > > > session?
> > > > > >
> > > > > > Do you have a way to reproduce this? Since this is a centos
> > > > > > kernel, you
> > > > > > could follow the page here to open a bug:
> > > > >
> > > > > Unfortunately we haven't found a reliable way to reproduce it. But
> > > > > we do seem to trigger it more and more lately.
> > > > >
> > > > > Regards,
> > > > >
> > > > > Rik
> > > > >
> > > > > >
> > > > > > https://urldefense.com/v3/__https://wiki.centos.org/ReportBugs.html__;!!ACWV5N9M2RV99hQ!LV3yWeoSOhNAkRHkxFCH2tlm0iNFVD78mxnSLyP6lrX7yBVeA2TOJ4nv6oZsqLwP4kW56CMpDWhkjjwSkWIqsboq$
> > > > > >
> > > > > >
> > > > > > > > I created another dump of the workqueues and worker pools on the
> > > > > > > > server:
> > > > > > > >
> > > > > > > > [Mon Mar 18 14:59:33 2024] Showing busy workqueues and worker
> > > > > > > > pools:
> > > > > > > > [Mon Mar 18 14:59:33 2024] workqueue events: flags=0x0
> > > > > > > > [Mon Mar 18 14:59:33 2024] pwq 54: cpus=27 node=1 flags=0x0
> > > > > > > > nice=0
> > > > > > > > active=1/256 refcnt=2
> > > > > > > > [Mon Mar 18 14:59:33 2024] pending: drm_fb_helper_damage_work
> > > > > > > > [drm_kms_helper]
> > > > > > > > [Mon Mar 18 14:59:33 2024] workqueue events_power_efficient:
> > > > > > > > flags=0x80
> > > > > > > > [Mon Mar 18 14:59:33 2024] pwq 54: cpus=27 node=1 flags=0x0
> > > > > > > > nice=0
> > > > > > > > active=1/256 refcnt=2
> > > > > > > > [Mon Mar 18 14:59:33 2024] pending: fb_flashcursor
> > > > > > > > [Mon Mar 18 14:59:33 2024] workqueue mm_percpu_wq: flags=0x8
> > > > > > > > [Mon Mar 18 14:59:33 2024] pwq 54: cpus=27 node=1 flags=0x0
> > > > > > > > nice=0
> > > > > > > > active=1/256 refcnt=3
> > > > > > > > [Mon Mar 18 14:59:33 2024] pending: lru_add_drain_per_cpu
> > > > > > > > BAR(362)
> > > > > > > > [Mon Mar 18 14:59:33 2024] workqueue kblockd: flags=0x18
> > > > > > > > [Mon Mar 18 14:59:33 2024] pwq 55: cpus=27 node=1 flags=0x0
> > > > > > > > nice=-20
> > > > > > > > active=1/256 refcnt=2
> > > > > > > > [Mon Mar 18 14:59:33 2024] pending: blk_mq_timeout_work
> > > > > > > >
> > > > > > > >
> > > > > > > > In contrast to last time, it doesn't show anything regarding nfs
> > > > > > > > this
> > > > > > > > time.
> > > > > > > >
> > > > > > > > I also tried the suggestion from Dai Ngo (echo 3 >
> > > > > > > > /proc/sys/vm/drop_caches), but that didn't seem to make any
> > > > > > > > difference.
> > > > > > > >
> > > > > > > > We haven't restarted the server yet as it seems the impact seems to
> > > > > > > > affect fewer clients that before. Is there anything we can run
> > > > > > > > on the
> > > > > > > > server to further debug this?
> > > > > > > >
> > > > > > > > In the past, the issue seemed to deteriorate rapidly and
> > > > > > > > resulted in
> > > > > > > > issues for almost all clients after about 20 minutes. This time the
> > > > > > > > impact seems to be less, but it's not gone.
> > > > > > > >
> > > > > > > > How can we force the NFS server to forget about a specific
> > > > > > > > client? I
> > > > > > > > haven't tried to restart the nfs service yet as I'm afraid it will
> > > > > > > > fail to stop as before.
> > > > > > > >
> > > > > > Not with that kernel. There are some new administrative interfaces
> > > > > > that
> > > > > > might allow that in the future, but they were just merged upstream
> > > > > > and
> > > > > > aren't in that kernel.
> > > > > >
> > > > > > --
> > > > > > Jeff Layton <jlayton@kernel.org>
> > > > >
--
Jeff Layton <jlayton@kernel.org>
^ permalink raw reply [flat|nested] 26+ messages in thread
* Re: nfsd hangs and nfsd_break_deleg_cb+0x170/0x190 warning
2024-03-21 20:48 ` Jeff Layton
@ 2024-03-21 21:13 ` Rik Theys
0 siblings, 0 replies; 26+ messages in thread
From: Rik Theys @ 2024-03-21 21:13 UTC (permalink / raw)
To: Jeff Layton, Dai Ngo, Linux Nfs
Hi,
On 3/21/24 21:48, Jeff Layton wrote:
> On Wed, 2024-03-20 at 20:41 +0100, Rik Theys wrote:
>> Hi,
>>
>> On 3/19/24 22:42, Dai Ngo wrote:
>>> On 3/19/24 12:41 PM, Rik Theys wrote:
>>>> Hi,
>>>>
>>>> On 3/19/24 18:09, Dai Ngo wrote:
>>>>> On 3/19/24 12:58 AM, Rik Theys wrote:
>>>>>> Hi,
>>>>>>
>>>>>> On 3/18/24 22:54, Jeff Layton wrote:
>>>>>>> On Mon, 2024-03-18 at 22:15 +0100, Rik Theys wrote:
>>>>>>>> Hi,
>>>>>>>>
>>>>>>>> On 3/18/24 21:21, Rik Theys wrote:
>>>>>>>>> Hi Jeff,
>>>>>>>>>
>>>>>>>>> On 3/12/24 13:47, Jeff Layton wrote:
>>>>>>>>>> On Tue, 2024-03-12 at 13:24 +0100, Rik Theys wrote:
>>>>>>>>>>> Hi Jeff,
>>>>>>>>>>>
>>>>>>>>>>> On 3/12/24 12:22, Jeff Layton wrote:
>>>>>>>>>>>> On Mon, 2024-03-11 at 19:43 +0100, Rik Theys wrote:
>>>>>>>>>>>>> Since a few weeks our Rocky Linux 9 NFS server has periodically
>>>>>>>>>>>>> logged hung nfsd tasks. The initial effect was that some
>>>>>>>>>>>>> clients
>>>>>>>>>>>>> could no longer access the NFS server. This got worse and worse
>>>>>>>>>>>>> (probably as more nfsd threads got blocked) and we had to
>>>>>>>>>>>>> restart
>>>>>>>>>>>>> the server. Restarting the server also failed as the NFS server
>>>>>>>>>>>>> service could no longer be stopped.
>>>>>>>>>>>>>
>>>>>>>>>>>>> The initial kernel we noticed this behavior on was
>>>>>>>>>>>>> kernel-5.14.0-362.18.1.el9_3.x86_64. Since then we've installed
>>>>>>>>>>>>> kernel-5.14.0-419.el9.x86_64 from CentOS Stream 9. The same
>>>>>>>>>>>>> issue
>>>>>>>>>>>>> happened again on this newer kernel version:
>>>>>>>>>> 419 is fairly up to date with nfsd changes. There are some
>>>>>>>>>> known bugs
>>>>>>>>>> around callbacks, and there is a draft MR in flight to fix it.
>>>>>>>>>>
>>>>>>>>>> What kernel were you on prior to 5.14.0-362.18.1.el9_3.x86_64 ?
>>>>>>>>>> If we
>>>>>>>>>> can bracket the changes around a particular version, then that
>>>>>>>>>> might
>>>>>>>>>> help identify the problem.
>>>>>>>>>>
>>>>>>>>>>>>> [Mon Mar 11 14:10:08 2024] Not tainted
>>>>>>>>>>>>> 5.14.0-419.el9.x86_64 #1
>>>>>>>>>>>>> [Mon Mar 11 14:10:08 2024] "echo 0 >
>>>>>>>>>>>>> /proc/sys/kernel/hung_task_timeout_secs" disables this message.
>>>>>>>>>>>>> [Mon Mar 11 14:10:08 2024]task:nfsd state:D
>>>>>>>>>>>>> stack:0
>>>>>>>>>>>>> pid:8865 ppid:2 flags:0x00004000
>>>>>>>>>>>>> [Mon Mar 11 14:10:08 2024] Call Trace:
>>>>>>>>>>>>> [Mon Mar 11 14:10:08 2024] <TASK>
>>>>>>>>>>>>> [Mon Mar 11 14:10:08 2024] __schedule+0x21b/0x550
>>>>>>>>>>>>> [Mon Mar 11 14:10:08 2024] schedule+0x2d/0x70
>>>>>>>>>>>>> [Mon Mar 11 14:10:08 2024] schedule_timeout+0x11f/0x160
>>>>>>>>>>>>> [Mon Mar 11 14:10:08 2024] ?
>>>>>>>>>>>>> select_idle_sibling+0x28/0x430
>>>>>>>>>>>>> [Mon Mar 11 14:10:08 2024] ? wake_affine+0x62/0x1f0
>>>>>>>>>>>>> [Mon Mar 11 14:10:08 2024] __wait_for_common+0x90/0x1d0
>>>>>>>>>>>>> [Mon Mar 11 14:10:08 2024] ?
>>>>>>>>>>>>> __pfx_schedule_timeout+0x10/0x10
>>>>>>>>>>>>> [Mon Mar 11 14:10:08 2024] __flush_workqueue+0x13a/0x3f0
>>>>>>>>>>>>> [Mon Mar 11 14:10:08 2024]
>>>>>>>>>>>>> nfsd4_shutdown_callback+0x49/0x120
>>>>>>>>>>>>> [nfsd]
>>>>>>>>>>>>> [Mon Mar 11 14:10:08 2024] ?
>>>>>>>>>>>>> nfsd4_cld_remove+0x54/0x1d0 [nfsd]
>>>>>>>>>>>>> [Mon Mar 11 14:10:08 2024] ?
>>>>>>>>>>>>> nfsd4_return_all_client_layouts+0xc4/0xf0 [nfsd]
>>>>>>>>>>>>> [Mon Mar 11 14:10:08 2024] ?
>>>>>>>>>>>>> nfsd4_shutdown_copy+0x68/0xc0 [nfsd]
>>>>>>>>>>>>> [Mon Mar 11 14:10:08 2024] __destroy_client+0x1f3/0x290
>>>>>>>>>>>>> [nfsd]
>>>>>>>>>>>>> [Mon Mar 11 14:10:08 2024]
>>>>>>>>>>>>> nfsd4_exchange_id+0x75f/0x770 [nfsd]
>>>>>>>>>>>>> [Mon Mar 11 14:10:08 2024] ?
>>>>>>>>>>>>> nfsd4_decode_opaque+0x3a/0x90 [nfsd]
>>>>>>>>>>>>> [Mon Mar 11 14:10:08 2024]
>>>>>>>>>>>>> nfsd4_proc_compound+0x44b/0x700 [nfsd]
>>>>>>>>>>>>> [Mon Mar 11 14:10:08 2024] nfsd_dispatch+0x94/0x1c0 [nfsd]
>>>>>>>>>>>>> [Mon Mar 11 14:10:08 2024] svc_process_common+0x2ec/0x660
>>>>>>>>>>>>> [sunrpc]
>>>>>>>>>>>>> [Mon Mar 11 14:10:08 2024] ?
>>>>>>>>>>>>> __pfx_nfsd_dispatch+0x10/0x10 [nfsd]
>>>>>>>>>>>>> [Mon Mar 11 14:10:08 2024] ? __pfx_nfsd+0x10/0x10 [nfsd]
>>>>>>>>>>>>> [Mon Mar 11 14:10:08 2024] svc_process+0x12d/0x170
>>>>>>>>>>>>> [sunrpc]
>>>>>>>>>>>>> [Mon Mar 11 14:10:08 2024] nfsd+0x84/0xb0 [nfsd]
>>>>>>>>>>>>> [Mon Mar 11 14:10:08 2024] kthread+0xdd/0x100
>>>>>>>>>>>>> [Mon Mar 11 14:10:08 2024] ? __pfx_kthread+0x10/0x10
>>>>>>>>>>>>> [Mon Mar 11 14:10:08 2024] ret_from_fork+0x29/0x50
>>>>>>>>>>>>> [Mon Mar 11 14:10:08 2024] </TASK>
>>>>>>>>>>>>> [Mon Mar 11 14:10:08 2024] INFO: task nfsd:8866 blocked for
>>>>>>>>>>>>> more than 122 seconds.
>>>>>>>>>>>>> [Mon Mar 11 14:10:08 2024] Not tainted
>>>>>>>>>>>>> 5.14.0-419.el9.x86_64 #1
>>>>>>>>>>>>> [Mon Mar 11 14:10:08 2024] "echo 0 >
>>>>>>>>>>>>> /proc/sys/kernel/hung_task_timeout_secs" disables this message.
>>>>>>>>>>>>> [Mon Mar 11 14:10:08 2024]task:nfsd state:D
>>>>>>>>>>>>> stack:0
>>>>>>>>>>>>> pid:8866 ppid:2 flags:0x00004000
>>>>>>>>>>>>> [Mon Mar 11 14:10:08 2024] Call Trace:
>>>>>>>>>>>>> [Mon Mar 11 14:10:08 2024] <TASK>
>>>>>>>>>>>>> [Mon Mar 11 14:10:08 2024] __schedule+0x21b/0x550
>>>>>>>>>>>>> [Mon Mar 11 14:10:08 2024] schedule+0x2d/0x70
>>>>>>>>>>>>> [Mon Mar 11 14:10:08 2024] schedule_timeout+0x11f/0x160
>>>>>>>>>>>>> [Mon Mar 11 14:10:08 2024] ?
>>>>>>>>>>>>> select_idle_sibling+0x28/0x430
>>>>>>>>>>>>> [Mon Mar 11 14:10:08 2024] ? tcp_recvmsg+0x196/0x210
>>>>>>>>>>>>> [Mon Mar 11 14:10:08 2024] ? wake_affine+0x62/0x1f0
>>>>>>>>>>>>> [Mon Mar 11 14:10:08 2024] __wait_for_common+0x90/0x1d0
>>>>>>>>>>>>> [Mon Mar 11 14:10:08 2024] ?
>>>>>>>>>>>>> __pfx_schedule_timeout+0x10/0x10
>>>>>>>>>>>>> [Mon Mar 11 14:10:08 2024] __flush_workqueue+0x13a/0x3f0
>>>>>>>>>>>>> [Mon Mar 11 14:10:08 2024]
>>>>>>>>>>>>> nfsd4_destroy_session+0x1a4/0x240
>>>>>>>>>>>>> [nfsd]
>>>>>>>>>>>>> [Mon Mar 11 14:10:08 2024]
>>>>>>>>>>>>> nfsd4_proc_compound+0x44b/0x700 [nfsd]
>>>>>>>>>>>>> [Mon Mar 11 14:10:08 2024] nfsd_dispatch+0x94/0x1c0 [nfsd]
>>>>>>>>>>>>> [Mon Mar 11 14:10:08 2024] svc_process_common+0x2ec/0x660
>>>>>>>>>>>>> [sunrpc]
>>>>>>>>>>>>> [Mon Mar 11 14:10:08 2024] ?
>>>>>>>>>>>>> __pfx_nfsd_dispatch+0x10/0x10 [nfsd]
>>>>>>>>>>>>> [Mon Mar 11 14:10:08 2024] ? __pfx_nfsd+0x10/0x10 [nfsd]
>>>>>>>>>>>>> [Mon Mar 11 14:10:08 2024] svc_process+0x12d/0x170
>>>>>>>>>>>>> [sunrpc]
>>>>>>>>>>>>> [Mon Mar 11 14:10:08 2024] nfsd+0x84/0xb0 [nfsd]
>>>>>>>>>>>>> [Mon Mar 11 14:10:08 2024] kthread+0xdd/0x100
>>>>>>>>>>>>> [Mon Mar 11 14:10:08 2024] ? __pfx_kthread+0x10/0x10
>>>>>>>>>>>>> [Mon Mar 11 14:10:08 2024] ret_from_fork+0x29/0x50
>>>>>>>>>>>>> [Mon Mar 11 14:10:08 2024] </TASK>
>>>>>>>>>>>>>
>>>>>>>>>>>> The above threads are trying to flush the workqueue, so that
>>>>>>>>>>>> probably
>>>>>>>>>>>> means that they are stuck waiting on a workqueue job to finish.
>>>>>>>>>>>>> The above is repeated a few times, and then this warning is
>>>>>>>>>>>>> also logged:
>>>>>>>>>>>>> [Mon Mar 11 14:12:04 2024] ------------[ cut here
>>>>>>>>>>>>> ]------------
>>>>>>>>>>>>> [Mon Mar 11 14:12:04 2024] WARNING: CPU: 39 PID: 8844 at
>>>>>>>>>>>>> fs/nfsd/nfs4state.c:4919 nfsd_break_deleg_cb+0x170/0x190 [nfsd]
>>>>>>>>>>>>> [Mon Mar 11 14:12:05 2024] Modules linked in: nfsv4
>>>>>>>>>>>>> dns_resolver nfs fscache netfs rpcsec_gss_krb5 rpcrdma rdma_cm
>>>>>>>>>>>>> iw_cm ib_cm ib_core binfmt_misc bonding tls rfkill
>>>>>>>>>>>>> nft_counter nft_ct
>>>>>>>>>>>>> nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_reject_inet
>>>>>>>>>>>>> nf_reject_ipv4 nf_reject_ipv6 nft_reject nf_tables nfnetlink
>>>>>>>>>>>>> vfat
>>>>>>>>>>>>> fat dm_thin_pool dm_persistent_data dm_bio_prison dm_bufio l
>>>>>>>>>>>>> ibcrc32c dm_service_time dm_multipath intel_rapl_msr
>>>>>>>>>>>>> intel_rapl_common intel_uncore_frequency
>>>>>>>>>>>>> intel_uncore_frequency_common isst_if_common skx_edac nfit
>>>>>>>>>>>>> libnvdimm ipmi_ssif x86_pkg_temp
>>>>>>>>>>>>> _thermal intel_powerclamp coretemp kvm_intel kvm irqbypass
>>>>>>>>>>>>> dcdbas rapl intel_cstate mgag200 i2c_algo_bit drm_shmem_helper
>>>>>>>>>>>>> dell_smbios drm_kms_helper dell_wmi_descriptor wmi_bmof intel_u
>>>>>>>>>>>>> ncore syscopyarea pcspkr sysfillrect mei_me sysimgblt
>>>>>>>>>>>>> acpi_ipmi
>>>>>>>>>>>>> mei fb_sys_fops i2c_i801 ipmi_si intel_pch_thermal lpc_ich
>>>>>>>>>>>>> ipmi_devintf i2c_smbus ipmi_msghandler joydev acpi_power_meter
>>>>>>>>>>>>> nfsd auth_rpcgss nfs_acl drm lockd grace fuse sunrpc ext4
>>>>>>>>>>>>> mbcache jbd2 sd_mod sg lpfc
>>>>>>>>>>>>> [Mon Mar 11 14:12:05 2024] nvmet_fc nvmet nvme_fc
>>>>>>>>>>>>> nvme_fabrics
>>>>>>>>>>>>> crct10dif_pclmul ahci libahci crc32_pclmul nvme_core
>>>>>>>>>>>>> crc32c_intel
>>>>>>>>>>>>> ixgbe megaraid_sas libata nvme_common ghash_clmulni_int
>>>>>>>>>>>>> el t10_pi wdat_wdt scsi_transport_fc mdio wmi dca dm_mirror
>>>>>>>>>>>>> dm_region_hash dm_log dm_mod
>>>>>>>>>>>>> [Mon Mar 11 14:12:05 2024] CPU: 39 PID: 8844 Comm: nfsd Not
>>>>>>>>>>>>> tainted 5.14.0-419.el9.x86_64 #1
>>>>>>>>>>>>> [Mon Mar 11 14:12:05 2024] Hardware name: Dell Inc.
>>>>>>>>>>>>> PowerEdge
>>>>>>>>>>>>> R740/00WGD1, BIOS 2.20.1 09/13/2023
>>>>>>>>>>>>> [Mon Mar 11 14:12:05 2024] RIP:
>>>>>>>>>>>>> 0010:nfsd_break_deleg_cb+0x170/0x190 [nfsd]
>>>>>>>>>>>>> [Mon Mar 11 14:12:05 2024] Code: a6 95 c5 f3 e9 ff fe ff
>>>>>>>>>>>>> ff 48
>>>>>>>>>>>>> 89 df be 01 00 00 00 e8 34 b5 13 f4 48 8d bb 98 00 00 00 e8
>>>>>>>>>>>>> c8 f9
>>>>>>>>>>>>> 00 00 84 c0 0f 85 2e ff ff ff <0f> 0b e9 27 ff ff ff be
>>>>>>>>>>>>> 02 00 00 00 48 89 df e8 0c b5 13 f4 e9 01
>>>>>>>>>>>>> [Mon Mar 11 14:12:05 2024] RSP: 0018:ffff9929e0bb7b80
>>>>>>>>>>>>> EFLAGS:
>>>>>>>>>>>>> 00010246
>>>>>>>>>>>>> [Mon Mar 11 14:12:05 2024] RAX: 0000000000000000 RBX:
>>>>>>>>>>>>> ffff8ada51930900 RCX: 0000000000000024
>>>>>>>>>>>>> [Mon Mar 11 14:12:05 2024] RDX: ffff8ada519309c8 RSI:
>>>>>>>>>>>>> ffff8ad582933c00 RDI: 0000000000002000
>>>>>>>>>>>>> [Mon Mar 11 14:12:05 2024] RBP: ffff8ad46bf21574 R08:
>>>>>>>>>>>>> ffff9929e0bb7b48 R09: 0000000000000000
>>>>>>>>>>>>> [Mon Mar 11 14:12:05 2024] R10: ffff8aec859a2948 R11:
>>>>>>>>>>>>> 0000000000000000 R12: ffff8ad6f497c360
>>>>>>>>>>>>> [Mon Mar 11 14:12:05 2024] R13: ffff8ad46bf21560 R14:
>>>>>>>>>>>>> ffff8ae5942e0b10 R15: ffff8ad6f497c360
>>>>>>>>>>>>> [Mon Mar 11 14:12:05 2024] FS: 0000000000000000(0000)
>>>>>>>>>>>>> GS:ffff8b031fcc0000(0000) knlGS:0000000000000000
>>>>>>>>>>>>> [Mon Mar 11 14:12:05 2024] CS: 0010 DS: 0000 ES: 0000 CR0:
>>>>>>>>>>>>> 0000000080050033
>>>>>>>>>>>>> [Mon Mar 11 14:12:05 2024] CR2: 00007fafe2060744 CR3:
>>>>>>>>>>>>> 00000018e58de006 CR4: 00000000007706e0
>>>>>>>>>>>>> [Mon Mar 11 14:12:05 2024] DR0: 0000000000000000 DR1:
>>>>>>>>>>>>> 0000000000000000 DR2: 0000000000000000
>>>>>>>>>>>>> [Mon Mar 11 14:12:05 2024] DR3: 0000000000000000 DR6:
>>>>>>>>>>>>> 00000000fffe0ff0 DR7: 0000000000000400
>>>>>>>>>>>>> [Mon Mar 11 14:12:05 2024] PKRU: 55555554
>>>>>>>>>>>>> [Mon Mar 11 14:12:05 2024] Call Trace:
>>>>>>>>>>>>> [Mon Mar 11 14:12:05 2024] <TASK>
>>>>>>>>>>>>> [Mon Mar 11 14:12:05 2024] ?
>>>>>>>>>>>>> show_trace_log_lvl+0x1c4/0x2df
>>>>>>>>>>>>> [Mon Mar 11 14:12:05 2024] ?
>>>>>>>>>>>>> show_trace_log_lvl+0x1c4/0x2df
>>>>>>>>>>>>> [Mon Mar 11 14:12:05 2024] ? __break_lease+0x16f/0x5f0
>>>>>>>>>>>>> [Mon Mar 11 14:12:05 2024] ?
>>>>>>>>>>>>> nfsd_break_deleg_cb+0x170/0x190
>>>>>>>>>>>>> [nfsd]
>>>>>>>>>>>>> [Mon Mar 11 14:12:05 2024] ? __warn+0x81/0x110
>>>>>>>>>>>>> [Mon Mar 11 14:12:05 2024] ?
>>>>>>>>>>>>> nfsd_break_deleg_cb+0x170/0x190
>>>>>>>>>>>>> [nfsd]
>>>>>>>>>>>>> [Mon Mar 11 14:12:05 2024] ? report_bug+0x10a/0x140
>>>>>>>>>>>>> [Mon Mar 11 14:12:05 2024] ? handle_bug+0x3c/0x70
>>>>>>>>>>>>> [Mon Mar 11 14:12:05 2024] ? exc_invalid_op+0x14/0x70
>>>>>>>>>>>>> [Mon Mar 11 14:12:05 2024] ? asm_exc_invalid_op+0x16/0x20
>>>>>>>>>>>>> [Mon Mar 11 14:12:05 2024] ?
>>>>>>>>>>>>> nfsd_break_deleg_cb+0x170/0x190
>>>>>>>>>>>>> [nfsd]
>>>>>>>>>>>>> [Mon Mar 11 14:12:05 2024] __break_lease+0x16f/0x5f0
>>>>>>>>>>>>> [Mon Mar 11 14:12:05 2024] ?
>>>>>>>>>>>>> nfsd_file_lookup_locked+0x117/0x160 [nfsd]
>>>>>>>>>>>>> [Mon Mar 11 14:12:05 2024] ? list_lru_del+0x101/0x150
>>>>>>>>>>>>> [Mon Mar 11 14:12:05 2024]
>>>>>>>>>>>>> nfsd_file_do_acquire+0x790/0x830
>>>>>>>>>>>>> [nfsd]
>>>>>>>>>>>>> [Mon Mar 11 14:12:05 2024]
>>>>>>>>>>>>> nfs4_get_vfs_file+0x315/0x3a0 [nfsd]
>>>>>>>>>>>>> [Mon Mar 11 14:12:05 2024]
>>>>>>>>>>>>> nfsd4_process_open2+0x430/0xa30 [nfsd]
>>>>>>>>>>>>> [Mon Mar 11 14:12:05 2024] ? fh_verify+0x297/0x2f0 [nfsd]
>>>>>>>>>>>>> [Mon Mar 11 14:12:05 2024] nfsd4_open+0x3ce/0x4b0 [nfsd]
>>>>>>>>>>>>> [Mon Mar 11 14:12:05 2024]
>>>>>>>>>>>>> nfsd4_proc_compound+0x44b/0x700 [nfsd]
>>>>>>>>>>>>> [Mon Mar 11 14:12:05 2024] nfsd_dispatch+0x94/0x1c0 [nfsd]
>>>>>>>>>>>>> [Mon Mar 11 14:12:05 2024] svc_process_common+0x2ec/0x660
>>>>>>>>>>>>> [sunrpc]
>>>>>>>>>>>>> [Mon Mar 11 14:12:05 2024] ?
>>>>>>>>>>>>> __pfx_nfsd_dispatch+0x10/0x10 [nfsd]
>>>>>>>>>>>>> [Mon Mar 11 14:12:05 2024] ? __pfx_nfsd+0x10/0x10 [nfsd]
>>>>>>>>>>>>> [Mon Mar 11 14:12:05 2024] svc_process+0x12d/0x170
>>>>>>>>>>>>> [sunrpc]
>>>>>>>>>>>>> [Mon Mar 11 14:12:05 2024] nfsd+0x84/0xb0 [nfsd]
>>>>>>>>>>>>> [Mon Mar 11 14:12:05 2024] kthread+0xdd/0x100
>>>>>>>>>>>>> [Mon Mar 11 14:12:05 2024] ? __pfx_kthread+0x10/0x10
>>>>>>>>>>>>> [Mon Mar 11 14:12:05 2024] ret_from_fork+0x29/0x50
>>>>>>>>>>>>> [Mon Mar 11 14:12:05 2024] </TASK>
>>>>>>>>>>>>> [Mon Mar 11 14:12:05 2024] ---[ end trace
>>>>>>>>>>>>> 7a039e17443dc651 ]---
>>>>>>>>>>>> This is probably this WARN in nfsd_break_one_deleg:
>>>>>>>>>>>>
>>>>>>>>>>>> WARN_ON_ONCE(!nfsd4_run_cb(&dp->dl_recall));
>>>>>>>>>>>>
>>>>>>>>>>>> It means that a delegation break callback to the client
>>>>>>>>>>>> couldn't be
>>>>>>>>>>>> queued to the workqueue, and so it didn't run.
>>>>>>>>>>>>
>>>>>>>>>>>>> Could this be the same issue as described
>>>>>>>>>>>>> here:https://urldefense.com/v3/__https://lore.kernel.org/linux-nfs/af0ec881-5ebf-4feb-98ae-3ed2a77f86f1@oracle.com/__;!!ACWV5N9M2RV99hQ!LV3yWeoSOhNAkRHkxFCH2tlm0iNFVD78mxnSLyP6lrX7yBVeA2TOJ4nv6oZsqLwP4kW56CMpDWhkjjwSkdBV9En7$
>>>>>>>>>>>>> ?
>>>>>>>>>>>> Yes, most likely the same problem.
>>>>>>>>>>> If I read that thread correctly, this issue was introduced
>>>>>>>>>>> between
>>>>>>>>>>> 6.1.63 and 6.6.3? Is it possible the EL9 5.14.0-362.18.1.el9_3
>>>>>>>>>>> backported these changes, or were we hitting some other bug
>>>>>>>>>>> with that
>>>>>>>>>>> version? It seems the 6.1.x kernel is not affected? If so, that
>>>>>>>>>>> would be
>>>>>>>>>>> the recommended kernel to run?
>>>>>>>>>> Anything is possible. We have to identify the problem first.
>>>>>>>>>>>>> As described in that thread, I've tried to obtain the requested
>>>>>>>>>>>>> information.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Is it possible this is the issue that was fixed by the patches
>>>>>>>>>>>>> described
>>>>>>>>>>>>> here?https://urldefense.com/v3/__https://lore.kernel.org/linux-nfs/2024022054-cause-suffering-eae8@gregkh/__;!!ACWV5N9M2RV99hQ!LV3yWeoSOhNAkRHkxFCH2tlm0iNFVD78mxnSLyP6lrX7yBVeA2TOJ4nv6oZsqLwP4kW56CMpDWhkjjwSkedtUP09$
>>>>>>>>>>>>>
>>>>>>>>>>>> Doubtful. Those are targeted toward a different set of issues.
>>>>>>>>>>>>
>>>>>>>>>>>> If you're willing, I do have some patches queued up for
>>>>>>>>>>>> CentOS here
>>>>>>>>>>>> that
>>>>>>>>>>>> fix some backchannel problems that could be related. I'm mainly
>>>>>>>>>>>> waiting
>>>>>>>>>>>> on Chuck to send these to Linus and then we'll likely merge
>>>>>>>>>>>> them into
>>>>>>>>>>>> CentOS soon afterward:
>>>>>>>>>>>>
>>>>>>>>>>>> https://urldefense.com/v3/__https://gitlab.com/redhat/centos-stream/src/kernel/centos-stream-9/-/merge_requests/3689__;!!ACWV5N9M2RV99hQ!LV3yWeoSOhNAkRHkxFCH2tlm0iNFVD78mxnSLyP6lrX7yBVeA2TOJ4nv6oZsqLwP4kW56CMpDWhkjjwSkdvDn8y7$
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>> If you can send me a patch file, I can rebuild the C9S kernel
>>>>>>>>>>> with that
>>>>>>>>>>> patch and run it. It can take a while for the bug to trigger as I
>>>>>>>>>>> believe it seems to be very workload dependent (we were
>>>>>>>>>>> running very
>>>>>>>>>>> stable for months and now hit this bug every other week).
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>> It's probably simpler to just pull down the build artifacts for
>>>>>>>>>> that MR.
>>>>>>>>>> You have to drill down through the CI for it, but they are here:
>>>>>>>>>>
>>>>>>>>>> https://urldefense.com/v3/__https://s3.amazonaws.com/arr-cki-prod-trusted-artifacts/index.html?prefix=trusted-artifacts*1194300175*publish_x86_64*6278921877*artifacts*__;Ly8vLy8!!ACWV5N9M2RV99hQ!LV3yWeoSOhNAkRHkxFCH2tlm0iNFVD78mxnSLyP6lrX7yBVeA2TOJ4nv6oZsqLwP4kW56CMpDWhkjjwSkaP5eW8V$
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> There's even a repo file you can install on the box to pull
>>>>>>>>>> them down.
>>>>>>>>> We installed this kernel on the server 3 days ago. Today, a user
>>>>>>>>> informed us that their screen was black after logging in.
>>>>>>>>> Similar to
>>>>>>>>> other occurrences of this issue, the mount command on the client
>>>>>>>>> was
>>>>>>>>> hung. But in contrast to the other times, there were no messages in
>>>>>>>>> the logs kernel logs on the server. Even restarting the client does
>>>>>>>>> not resolve the issue.
>>>>>>> Ok, so you rebooted the client and it's still unable to mount? That
>>>>>>> sounds like a server problem if so.
>>>>>>>
>>>>>>> Are both client and server running the same kernel?
>>>>>> No, the server runs 5.14.0-427.3689_1194299994.el9 and the client
>>>>>> 5.14.0-362.18.1.el9_3.
>>>>>>>>> Something still seems to be wrong on the server though. When I
>>>>>>>>> look at
>>>>>>>>> the directories under /proc/fs/nfsd/clients, there's still a
>>>>>>>>> directory
>>>>>>>>> for the specific client, even though it's no longer running:
>>>>>>>>>
>>>>>>>>> # cat 155/info
>>>>>>>>> clientid: 0xc8edb7f65f4a9ad
>>>>>>>>> address: "10.87.31.152:819"
>>>>>>>>> status: confirmed
>>>>>>>>> seconds from last renew: 33163
>>>>>>>>> name: "Linux NFSv4.2 bersalis.esat.kuleuven.be"
>>>>>>>>> minor version: 2
>>>>>>>>> Implementation domain: "kernel.org"
>>>>>>>>> Implementation name: "Linux 5.14.0-362.18.1.el9_3.0.1.x86_64 #1 SMP
>>>>>>>>> PREEMPT_DYNAMIC Sun Feb 11 13:49:23 UTC 2024 x86_64"
>>>>>>>>> Implementation time: [0, 0]
>>>>>>>>> callback state: DOWN
>>>>>>>>> callback address: 10.87.31.152:0
>>>>>>>>>
>>>>>>> If you just shut down the client, the server won't immediately
>>>>>>> purge its
>>>>>>> record. In fact, assuming you're running the same kernel on the
>>>>>>> server,
>>>>>>> it won't purge the client record until there is a conflicting request
>>>>>>> for its state.
>>>>>> Is there a way to force such a conflicting request (to get the
>>>>>> client record to purge)?
>>>>> Try:
>>>>>
>>>>> # echo "expire" > /proc/fs/nfsd/clients/155/ctl
>>>> I've tried that. The command hangs and can not be interrupted with
>>>> ctrl-c.
>>>> I've now also noticed in the dmesg output that the kernel issued the
>>>> following WARNING a few hours ago. It wasn't directly triggered by
>>>> the echo command above, but seems to have been triggered a few hours
>>>> ago (probably when another client started to have the same problem as
>>>> more clients are experiencing issues now).
>>> I think this warning message is harmless. However it indicates potential
>>> problem with the workqueue which might be related to memory shortage.
>>>
>>> What the output of 'cat /proc/meminfo' looks like?
>> I doubt the current values are useful, but they are:
>>
>> MemTotal: 196110860 kB
>> MemFree: 29357112 kB
>> MemAvailable: 179529420 kB
>> Buffers: 11996096 kB
>> Cached: 130589396 kB
>> SwapCached: 52 kB
>> Active: 1136988 kB
>> Inactive: 144192468 kB
>> Active(anon): 698564 kB
>> Inactive(anon): 2657256 kB
>> Active(file): 438424 kB
>> Inactive(file): 141535212 kB
>> Unevictable: 72140 kB
>> Mlocked: 69068 kB
>> SwapTotal: 67108860 kB
>> SwapFree: 67106276 kB
>> Zswap: 0 kB
>> Zswapped: 0 kB
>> Dirty: 80812 kB
>> Writeback: 0 kB
>> AnonPages: 2806592 kB
>> Mapped: 322700 kB
>> Shmem: 599308 kB
>> KReclaimable: 16977000 kB
>> Slab: 18898736 kB
>> SReclaimable: 16977000 kB
>> SUnreclaim: 1921736 kB
>> KernelStack: 18128 kB
>> PageTables: 31716 kB
>> SecPageTables: 0 kB
>> NFS_Unstable: 0 kB
>> Bounce: 0 kB
>> WritebackTmp: 0 kB
>> CommitLimit: 165164288 kB
>> Committed_AS: 5223940 kB
>> VmallocTotal: 34359738367 kB
>> VmallocUsed: 300064 kB
>> VmallocChunk: 0 kB
>> Percpu: 45888 kB
>> HardwareCorrupted: 0 kB
>> AnonHugePages: 2451456 kB
>> ShmemHugePages: 0 kB
>> ShmemPmdMapped: 0 kB
>> FileHugePages: 0 kB
>> FilePmdMapped: 0 kB
>> CmaTotal: 0 kB
>> CmaFree: 0 kB
>> Unaccepted: 0 kB
>> HugePages_Total: 0
>> HugePages_Free: 0
>> HugePages_Rsvd: 0
>> HugePages_Surp: 0
>> Hugepagesize: 2048 kB
>> Hugetlb: 0 kB
>> DirectMap4k: 1303552 kB
>> DirectMap2M: 28715008 kB
>> DirectMap1G: 171966464 kB
>>
>>
>>> Did you try 'echo 3 > /proc/sys/vm/drop_caches'?
>> Yes, I tried that when the first client hit the issue, but it didn't
>> result in any unlocking of the client.
>>
>>
>>>> [Tue Mar 19 14:53:44 2024] ------------[ cut here ]------------
>>>> [Tue Mar 19 14:53:44 2024] WARNING: CPU: 44 PID: 5843 at
>>>> fs/nfsd/nfs4state.c:4920 nfsd_break_deleg_cb+0x170/0x190 [nfsd]
>>>> [Tue Mar 19 14:53:44 2024] Modules linked in: nf_conntrack_netlink
>>>> nfsv4 dns_resolver nfs fscache netfs binfmt_misc xsk_diag
>>>> rpcsec_gss_krb5 rpcrdma rdma_cm iw_cm ib_cm ib_core bonding tls
>>>> rfkill nft_counter nft_ct nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4
>>>> nft_reject_inet nf_reject_ipv4 nf_reject_ipv6 nft_reject nf_tables
>>>> nfnetlink vfat fat dm_thin_pool dm_persistent_data dm_bio_prison
>>>> dm_bufio libcrc32c dm_service_time dm_multipath intel_rapl_msr
>>>> intel_rapl_common intel_uncore_frequency
>>>> intel_uncore_frequency_common isst_if_common skx_edac nfit libnvdimm
>>>> x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm dcdbas
>>>> irqbypass ipmi_ssif rapl intel_cstate mgag200 i2c_algo_bit
>>>> drm_shmem_helper drm_kms_helper dell_smbios syscopyarea intel_uncore
>>>> sysfillrect wmi_bmof dell_wmi_descriptor pcspkr sysimgblt fb_sys_fops
>>>> mei_me i2c_i801 mei intel_pch_thermal acpi_ipmi i2c_smbus lpc_ich
>>>> ipmi_si ipmi_devintf ipmi_msghandler joydev acpi_power_meter nfsd
>>>> nfs_acl lockd auth_rpcgss grace drm fuse sunrpc ext4
>>>> [Tue Mar 19 14:53:44 2024] mbcache jbd2 sd_mod sg lpfc nvmet_fc
>>>> nvmet nvme_fc nvme_fabrics crct10dif_pclmul crc32_pclmul nvme_core
>>>> ixgbe crc32c_intel ahci libahci nvme_common megaraid_sas t10_pi
>>>> ghash_clmulni_intel wdat_wdt libata scsi_transport_fc mdio dca wmi
>>>> dm_mirror dm_region_hash dm_log dm_mod
>>>> [Tue Mar 19 14:53:44 2024] CPU: 44 PID: 5843 Comm: nfsd Not tainted
>>>> 5.14.0-427.3689_1194299994.el9.x86_64 #1
>>>> [Tue Mar 19 14:53:44 2024] Hardware name: Dell Inc. PowerEdge
>>>> R740/00WGD1, BIOS 2.20.1 09/13/2023
>>>> [Tue Mar 19 14:53:44 2024] RIP: 0010:nfsd_break_deleg_cb+0x170/0x190
>>>> [nfsd]
>>>> [Tue Mar 19 14:53:44 2024] Code: 76 76 cd de e9 ff fe ff ff 48 89 df
>>>> be 01 00 00 00 e8 34 a1 1b df 48 8d bb 98 00 00 00 e8 a8 fe 00 00 84
>>>> c0 0f 85 2e ff ff ff <0f> 0b e9 27 ff ff ff be 02 00 00 00 48 89 df
>>>> e8 0c a1 1b df e9 01
>>>> [Tue Mar 19 14:53:44 2024] RSP: 0018:ffffb2878f2cfc38 EFLAGS: 00010246
>>>> [Tue Mar 19 14:53:44 2024] RAX: 0000000000000000 RBX:
>>>> ffff88d5171067b8 RCX: 0000000000000000
>>>> [Tue Mar 19 14:53:44 2024] RDX: ffff88d517106880 RSI:
>>>> ffff88bdceec8600 RDI: 0000000000002000
>>>> [Tue Mar 19 14:53:44 2024] RBP: ffff88d68a38a284 R08:
>>>> ffffb2878f2cfc00 R09: 0000000000000000
>>>> [Tue Mar 19 14:53:44 2024] R10: ffff88bf57dd7878 R11:
>>>> 0000000000000000 R12: ffff88d5b79c4798
>>>> [Tue Mar 19 14:53:44 2024] R13: ffff88d68a38a270 R14:
>>>> ffff88cab06ad0c8 R15: ffff88d5b79c4798
>>>> [Tue Mar 19 14:53:44 2024] FS: 0000000000000000(0000)
>>>> GS:ffff88d4a1180000(0000) knlGS:0000000000000000
>>>> [Tue Mar 19 14:53:44 2024] CS: 0010 DS: 0000 ES: 0000 CR0:
>>>> 0000000080050033
>>>> [Tue Mar 19 14:53:44 2024] CR2: 00007fe46ef90000 CR3:
>>>> 000000019d010004 CR4: 00000000007706e0
>>>> [Tue Mar 19 14:53:44 2024] DR0: 0000000000000000 DR1:
>>>> 0000000000000000 DR2: 0000000000000000
>>>> [Tue Mar 19 14:53:44 2024] DR3: 0000000000000000 DR6:
>>>> 00000000fffe0ff0 DR7: 0000000000000400
>>>> [Tue Mar 19 14:53:44 2024] PKRU: 55555554
>>>> [Tue Mar 19 14:53:44 2024] Call Trace:
>>>> [Tue Mar 19 14:53:44 2024] <TASK>
>>>> [Tue Mar 19 14:53:44 2024] ? show_trace_log_lvl+0x1c4/0x2df
>>>> [Tue Mar 19 14:53:44 2024] ? show_trace_log_lvl+0x1c4/0x2df
>>>> [Tue Mar 19 14:53:44 2024] ? __break_lease+0x16f/0x5f0
>>>> [Tue Mar 19 14:53:44 2024] ? nfsd_break_deleg_cb+0x170/0x190 [nfsd]
>>>> [Tue Mar 19 14:53:44 2024] ? __warn+0x81/0x110
>>>> [Tue Mar 19 14:53:44 2024] ? nfsd_break_deleg_cb+0x170/0x190 [nfsd]
>>>> [Tue Mar 19 14:53:44 2024] ? report_bug+0x10a/0x140
>>>> [Tue Mar 19 14:53:44 2024] ? handle_bug+0x3c/0x70
>>>> [Tue Mar 19 14:53:44 2024] ? exc_invalid_op+0x14/0x70
>>>> [Tue Mar 19 14:53:44 2024] ? asm_exc_invalid_op+0x16/0x20
>>>> [Tue Mar 19 14:53:44 2024] ? nfsd_break_deleg_cb+0x170/0x190 [nfsd]
>>>> [Tue Mar 19 14:53:44 2024] ? nfsd_break_deleg_cb+0x96/0x190 [nfsd]
>>>> [Tue Mar 19 14:53:44 2024] __break_lease+0x16f/0x5f0
>>>> [Tue Mar 19 14:53:44 2024] nfs4_get_vfs_file+0x164/0x3a0 [nfsd]
>>>> [Tue Mar 19 14:53:44 2024] nfsd4_process_open2+0x430/0xa30 [nfsd]
>>>> [Tue Mar 19 14:53:44 2024] ? fh_verify+0x297/0x2f0 [nfsd]
>>>> [Tue Mar 19 14:53:44 2024] nfsd4_open+0x3ce/0x4b0 [nfsd]
>>>> [Tue Mar 19 14:53:44 2024] nfsd4_proc_compound+0x44b/0x700 [nfsd]
>>>> [Tue Mar 19 14:53:44 2024] nfsd_dispatch+0x94/0x1c0 [nfsd]
>>>> [Tue Mar 19 14:53:44 2024] svc_process_common+0x2ec/0x660 [sunrpc]
>>>> [Tue Mar 19 14:53:44 2024] ? __pfx_nfsd_dispatch+0x10/0x10 [nfsd]
>>>> [Tue Mar 19 14:53:44 2024] ? __pfx_nfsd+0x10/0x10 [nfsd]
>>>> [Tue Mar 19 14:53:44 2024] svc_process+0x12d/0x170 [sunrpc]
>>>> [Tue Mar 19 14:53:44 2024] nfsd+0x84/0xb0 [nfsd]
>>>> [Tue Mar 19 14:53:44 2024] kthread+0xdd/0x100
>>>> [Tue Mar 19 14:53:44 2024] ? __pfx_kthread+0x10/0x10
>>>> [Tue Mar 19 14:53:44 2024] ret_from_fork+0x29/0x50
>>>> [Tue Mar 19 14:53:44 2024] </TASK>
>>>> [Tue Mar 19 14:53:44 2024] ---[ end trace ed0b2b3f135c637d ]---
>>>>
>>>> It again seems to have been triggered in nfsd_break_deleg_cb?
>>>>
>>>> I also had the following perf command running a tmux on the server:
>>>>
>>>> perf trace -e nfsd:nfsd_cb_recall_any
>>>>
>>>> This has spewed a lot of messages. I'm including a short list here:
>>>>
>>>> ...
>>>>
>>>> 33464866.721 kworker/u98:5/1591466 nfsd:nfsd_cb_recall_any(cl_boot:
>>>> 1710533037, cl_id: 210688785, bmval0: 1, addr: 0x7f331bb116c8)
>>>> 33464866.724 kworker/u98:5/1591466 nfsd:nfsd_cb_recall_any(cl_boot:
>>>> 1710533037, cl_id: 210688827, bmval0: 1, addr: 0x7f331bb11738)
>>>> 33464866.729 kworker/u98:5/1591466 nfsd:nfsd_cb_recall_any(cl_boot:
>>>> 1710533037, cl_id: 210688767, bmval0: 1, addr: 0x7f331bb117a8)
>>>> 33464866.732 kworker/u98:5/1591466 nfsd:nfsd_cb_recall_any(cl_boot:
>>>> 1710533037, cl_id: 210718132, bmval0: 1, addr: 0x7f331bb11818)
>>>> 33464866.737 kworker/u98:5/1591466 nfsd:nfsd_cb_recall_any(cl_boot:
>>>> 1710533037, cl_id: 210688952, bmval0: 1, addr: 0x7f331bb11888)
>>>> 33464866.741 kworker/u98:5/1591466 nfsd:nfsd_cb_recall_any(cl_boot:
>>>> 1710533037, cl_id: 210702355, bmval0: 1, addr: 0x7f331bb118f8)
>>>> 33868414.001 kthreadd/1597068 nfsd:nfsd_cb_recall_any(cl_boot:
>>>> 1710533037, cl_id: 210688751, bmval0: 1, addr: 0x7f331be68620)
>>>> 33868414.014 kthreadd/1597068 nfsd:nfsd_cb_recall_any(cl_boot:
>>>> 1710533037, cl_id: 210718536, bmval0: 1, addr: 0x7f331be68690)
>>>> 33868414.018 kthreadd/1597068 nfsd:nfsd_cb_recall_any(cl_boot:
>>>> 1710533037, cl_id: 210719074, bmval0: 1, addr: 0x7f331be68700)
>>>> 33868414.022 kthreadd/1597068 nfsd:nfsd_cb_recall_any(cl_boot:
>>>> 1710533037, cl_id: 210688916, bmval0: 1, addr: 0x7f331be68770)
>>>> 33868414.026 kthreadd/1597068 nfsd:nfsd_cb_recall_any(cl_boot:
>>>> 1710533037, cl_id: 210688941, bmval0: 1, addr: 0x7f331be687e0)
>>>> ...
>>>>
>>>> 33868414.924 kthreadd/1597068 nfsd:nfsd_cb_recall_any(cl_boot:
>>>> 1710533037, cl_id: 210688744, bmval0: 1, addr: 0x7f331be6d7f0)
>>>> 33868414.929 kthreadd/1597068 nfsd:nfsd_cb_recall_any(cl_boot:
>>>> 1710533037, cl_id: 210717223, bmval0: 1, addr: 0x7f331be6d860)
>>>> 33868414.934 kthreadd/1597068 nfsd:nfsd_cb_recall_any(cl_boot:
>>>> 1710533037, cl_id: 210716137, bmval0: 1, addr: 0x7f331be6d8d0)
>>>> 34021240.903 kworker/u98:5/1591466 nfsd:nfsd_cb_recall_any(cl_boot:
>>>> 1710533037, cl_id: 210688941, bmval0: 1, addr: 0x7f331c207de8)
>>>> 34021240.917 kworker/u98:5/1591466 nfsd:nfsd_cb_recall_any(cl_boot:
>>>> 1710533037, cl_id: 210718750, bmval0: 1, addr: 0x7f331c207e58)
>>>> 34021240.922 kworker/u98:5/1591466 nfsd:nfsd_cb_recall_any(cl_boot:
>>>> 1710533037, cl_id: 210688955, bmval0: 1, addr: 0x7f331c207ec8)
>>>> 34021240.925 kworker/u98:5/1591466 nfsd:nfsd_cb_recall_any(cl_boot:
>>>> 1710533037, cl_id: 210688975, bmval0: 1, addr: 0x7f331c207f38)
>>>> ...
>>>>
>>>> I assume the cl_id is the client id? How can I map this to a client
>>>> from /proc/fs/nfsd/clients?
>>> The hex value of 'clientid' printed from /proc/fs/nfsd/clients/XX/info
>>> is a 64-bit value composed of:
>>>
>>> typedef struct {
>>> u32 cl_boot;
>>> u32 cl_id;
>>> } clientid_t
>>>
>>> For example:
>>>
>>> clientid: 0xc8edb7f65f4a9ad
>>>
>>> cl_boot: 65f4a9add (1710533037)
>>> cl_id: c8edb7f (21068895)
>>>
>>> This should match a trace event with:
>>>
>>> nfsd:nfsd_cb_recall_any(cl_boot: 1710533037, cl_id: 21068895, bmval0:
>>> XX, addr: 0xYYYYY)
>>>
>>>> If I understand it correctly, the recall_any should be called when
>>>> either the system starts to experience memory pressure,
>>> yes.
>> It seems odd that the system gets in such a state that has such high
>> memory pressure. It doesn't run much else than NFS and Samba.
>>>> or it reaches the delegation limits?
>>> No, this feature was added to nfsd very recently. I don't think your
>>> kernel has it.
>>>
>>>> I doubt the system is actually running out of memory here as there
>>>> are no other indications.
>>>> Shouldn't I get those "page allocation failure" messages if it does?
>>>> How can I check the number of delegations/leases currently issued,
>>>> what the current maximum is and how to increase it?
>>> Max delegations is 4 per 1MB of available memory. There is no
>>> admin tool to adjust this value.
>> /proc/locks currently has about 130k DELEG lines, so that should be a
>> lot lower than the limit on a 192G ram server.
>>
>>
>>> I do not recommend running a production system with delegation
>>> disabled. But for this specific issue, it might help to temporarily
>>> disable delegation to isolate problem areas.
>>
>> I'm going to reboot the system with the 6.1.82 kernel (kernel-lt from
>> elrepo). Maybe it has less new modern developments that may have
>> introduced this.
>>
> If v6.1-ish kernel turns out to not help, then you may want to give a
> v6.7 or v6.8 kernel a try. It helps if we know whether this problem is
> reproducible on in more up to date kernels.
Unfortunately the 6.1.82 kernel resulted in an issue with krb nfs
clients so I had to reboot the system again (see my other mail on the
linux-nfs list). It's now running the latest CentOS Stream 9 kernel (430).
I don't know how up to date it is on NFS patches. You mentioned there
were still additional nfs fixes between the 427 merge request version
you provided earlier and this one, but I failed to find any in the
changelog (which unfortunately seems to be truncated now).
I'm aware that there's a potential data corruption bug in the 430 version?
>
>> I've been able to reproduce the situation on an additional client now
>> that the issue happens on the server:
>>
>> 1. Log in on a client and mount the NFS share.
>> 2. Open a file from the NFS share in vim so the client gets a read
>> delegation from the server
>> 3. Verify on the server in /proc/fs/nfsd/clients/*/states that the
>> client has a delegation for the file
>> 4. Forcefully reboot the client by running 'echo b > /proc/sysrq-trigger'
>> 5. Watch the /proc/fs/nfsd/clients/*/info file on the server.
>>
>> The "seconds from last renew" will go up and at some point the callback
>> state changes to "FAULT". Even when the lease delegation time (90s by
>> default?) is over, the
>>
>> seconds from last renew keeps increasing. At some point the callback
>> state changes to "DOWN". When the client is up again and remounts the
>> share, the mount hangs on the client
>>
>> and on the server I notice there's a second directory for this client in
>> the clients directory, even though the clientid is the same. The
>> callback state for this new client is "UNKNOWN" and the callback address
>> is "(einval)".
>>
>> This is on a client running Fedora 39 with the 6.7.9 kernel.
>>
> I'm a little unclear...do the above steps work correctly when the server
> isn't in this state? I assume the above steps are not sufficient to
> cause a problem when the server is behaving normally?
These steps indeed don't work then the server is behaving normally. I'm
trying to reproduce the issue on a test system, but I'm unable to
trigger it there so far.
>
>> I don't know yet if the same procedure can be used to trigger the
>> behavior after the server is rebooted. I'm going to try to reproduce
>> this on another system first.
>>
>> I would expect the delegations to expire automatically after 90s, but
>> they remain in the states file of the "DOWN" client.
>>
> That would have been true a year or so ago, but there were some recent
> changes to make the server more "courteous" toward clients that lose
> contact for a while. If there are no conflicting requests for the state
> they hold then the server will hold onto the lease (basically)
> indefinitely, until there is such a conflict.
>
> The client _should_ be able to log in and it cancel the old client
> record though. It sounds like that's not working properly for some
> reason and it's interfering with the ability to do a CREATE_SESSION.
What happens if the server can't reach the original client at that point?
I've also noticed that the callback information seems to show a port
number for the callback channel. If I'm not mistaken NFS 4.2 also does
this over the regular 2049 port now?
Regards,
Rik
>
>>> -Dai
>>>
>>>> Regarding the recall any call: from what I've read on kernelnewbies,
>>>> this feature was introduced in the 6.2 kernel? When I look at the
>>>> tree for 6.1.x, it was backported in 6.1.81? Is there a way to
>>>> disable this support somehow?
>>>>
>>>> Regards,
>>>>
>>>> Rik
>>>>
>>>>
>>>>> -Dai
>>>>>
>>>>>>>
>>>>>>>> The nfsdclnts command for this client shows the following
>>>>>>>> delegations:
>>>>>>>>
>>>>>>>> # nfsdclnts -f 155/states -t all
>>>>>>>> Inode number | Type | Access | Deny | ip address | Filename
>>>>>>>> 169346743 | open | r- | -- | 10.87.31.152:819 |
>>>>>>>> disconnected dentry
>>>>>>>> 169346743 | deleg | r | | 10.87.31.152:819 |
>>>>>>>> disconnected dentry
>>>>>>>> 169346746 | open | r- | -- | 10.87.31.152:819 |
>>>>>>>> disconnected dentry
>>>>>>>> 169346746 | deleg | r | | 10.87.31.152:819 |
>>>>>>>> disconnected dentry
>>>>>>>>
>>>>>>>> I see a lot of recent patches regarding directory delegations. Could
>>>>>>>> this be related to this?
>>>>>>>>
>>>>>>>> Will a 5.14.0-362.18.1.el9_3.0.1 kernel try to use a directory
>>>>>>>> delegation?
>>>>>>>>
>>>>>>>>
>>>>>>> No. Directory delegations are a new feature that's still under
>>>>>>> development. They use some of the same machinery as file delegations,
>>>>>>> but they wouldn't be a factor here.
>>>>>>>
>>>>>>>>> The system seems to have identified that the client is no longer
>>>>>>>>> reachable, but the client entry does not go away. When a mount was
>>>>>>>>> hanging on the client, there would be two directories in clients
>>>>>>>>> for
>>>>>>>>> the same client. Killing the mount command clears up the second
>>>>>>>>> entry.
>>>>>>>>>
>>>>>>>>> Even after running conntrack -D on the server to remove the tcp
>>>>>>>>> connection from the conntrack table, the entry doesn't go away
>>>>>>>>> and the
>>>>>>>>> client still can not mount anything from the server.
>>>>>>>>>
>>>>>>>>> A tcpdump on the client while a mount was running logged the
>>>>>>>>> following
>>>>>>>>> messages over and over again:
>>>>>>>>>
>>>>>>>>> request:
>>>>>>>>>
>>>>>>>>> Frame 1: 378 bytes on wire (3024 bits), 378 bytes captured (3024
>>>>>>>>> bits)
>>>>>>>>> Ethernet II, Src: HP_19:7d:4b (e0:73:e7:19:7d:4b), Dst:
>>>>>>>>> ArubaaHe_f9:8e:00 (88:3a:30:f9:8e:00)
>>>>>>>>> Internet Protocol Version 4, Src: 10.87.31.152, Dst: 10.86.18.14
>>>>>>>>> Transmission Control Protocol, Src Port: 932, Dst Port: 2049,
>>>>>>>>> Seq: 1,
>>>>>>>>> Ack: 1, Len: 312
>>>>>>>>> Remote Procedure Call, Type:Call XID:0x1d3220c4
>>>>>>>>> Network File System
>>>>>>>>> [Program Version: 4]
>>>>>>>>> [V4 Procedure: COMPOUND (1)]
>>>>>>>>> GSS Data, Ops(1): CREATE_SESSION
>>>>>>>>> Length: 152
>>>>>>>>> GSS Sequence Number: 76
>>>>>>>>> Tag: <EMPTY>
>>>>>>>>> minorversion: 2
>>>>>>>>> Operations (count: 1): CREATE_SESSION
>>>>>>>>> [Main Opcode: CREATE_SESSION (43)]
>>>>>>>>> GSS Checksum:
>>>>>>>>> 00000028040404ffffffffff000000002c19055f1f8d442d594c13849628affc2797cbb2…
>>>>>>>>>
>>>>>>>>> GSS Token Length: 40
>>>>>>>>> GSS-API Generic Security Service Application Program
>>>>>>>>> Interface
>>>>>>>>> krb5_blob:
>>>>>>>>> 040404ffffffffff000000002c19055f1f8d442d594c13849628affc2797cbb23fa080b0…
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> response:
>>>>>>>>>
>>>>>>>>> Frame 2: 206 bytes on wire (1648 bits), 206 bytes captured (1648
>>>>>>>>> bits)
>>>>>>>>> Ethernet II, Src: ArubaaHe_f9:8e:00 (88:3a:30:f9:8e:00), Dst:
>>>>>>>>> HP_19:7d:4b (e0:73:e7:19:7d:4b)
>>>>>>>>> Internet Protocol Version 4, Src: 10.86.18.14, Dst: 10.87.31.152
>>>>>>>>> Transmission Control Protocol, Src Port: 2049, Dst Port: 932,
>>>>>>>>> Seq: 1,
>>>>>>>>> Ack: 313, Len: 140
>>>>>>>>> Remote Procedure Call, Type:Reply XID:0x1d3220c4
>>>>>>>>> Network File System
>>>>>>>>> [Program Version: 4]
>>>>>>>>> [V4 Procedure: COMPOUND (1)]
>>>>>>>>> GSS Data, Ops(1): CREATE_SESSION(NFS4ERR_DELAY)
>>>>>>>>> Length: 24
>>>>>>>>> GSS Sequence Number: 76
>>>>>>>>> Status: NFS4ERR_DELAY (10008)
>>>>>>>>> Tag: <EMPTY>
>>>>>>>>> Operations (count: 1)
>>>>>>>>> [Main Opcode: CREATE_SESSION (43)]
>>>>>>>>> GSS Checksum:
>>>>>>>>> 00000028040405ffffffffff000000000aa742d0798deaad1a8aa2d7c3a91bf4f6274222…
>>>>>>>>>
>>>>>>>>> GSS Token Length: 40
>>>>>>>>> GSS-API Generic Security Service Application Program
>>>>>>>>> Interface
>>>>>>>>> krb5_blob:
>>>>>>>>> 040405ffffffffff000000000aa742d0798deaad1a8aa2d7c3a91bf4f627422226d74923…
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> I was hoping that giving the client a different IP address would
>>>>>>>>> resolve the issue for this client, but it didn't. Even though the
>>>>>>>>> client had a new IP address (hostname was kept the same), it
>>>>>>>>> failed to
>>>>>>>>> mount anything from the server.
>>>>>>>>>
>>>>>>> Changing the IP address won't help. The client is probably using the
>>>>>>> same long-form client id as before, so the server still identifies
>>>>>>> the
>>>>>>> client even with the address change.
>>>>>> How is the client id determined? Will changing the hostname of the
>>>>>> client trigger a change of the client id?
>>>>>>> Unfortunately, the cause of an NFS4ERR_DELAY error is tough to guess.
>>>>>>> The client is expected to back off and retry, so if the server keeps
>>>>>>> returning that repeatedly, then a hung mount command is expected.
>>>>>>>
>>>>>>> The question is why the server would keep returning DELAY. A lot of
>>>>>>> different problems ranging from memory allocation issues to protocol
>>>>>>> problems can result in that error. You may want to check the NFS
>>>>>>> server
>>>>>>> and see if anything was logged there.
>>>>>> There are no messages in the system logs that indicate any sort of
>>>>>> memory issue. We also increased the min_kbytes_free sysctl to 2G on
>>>>>> the server before we restarted it with the newer kernel.
>>>>>>> This is on a CREATE_SESSION call, so I wonder if the record held
>>>>>>> by the
>>>>>>> (courteous) server is somehow blocking the attempt to reestablish the
>>>>>>> session?
>>>>>>>
>>>>>>> Do you have a way to reproduce this? Since this is a centos
>>>>>>> kernel, you
>>>>>>> could follow the page here to open a bug:
>>>>>> Unfortunately we haven't found a reliable way to reproduce it. But
>>>>>> we do seem to trigger it more and more lately.
>>>>>>
>>>>>> Regards,
>>>>>>
>>>>>> Rik
>>>>>>
>>>>>>> https://urldefense.com/v3/__https://wiki.centos.org/ReportBugs.html__;!!ACWV5N9M2RV99hQ!LV3yWeoSOhNAkRHkxFCH2tlm0iNFVD78mxnSLyP6lrX7yBVeA2TOJ4nv6oZsqLwP4kW56CMpDWhkjjwSkWIqsboq$
>>>>>>>
>>>>>>>
>>>>>>>>> I created another dump of the workqueues and worker pools on the
>>>>>>>>> server:
>>>>>>>>>
>>>>>>>>> [Mon Mar 18 14:59:33 2024] Showing busy workqueues and worker
>>>>>>>>> pools:
>>>>>>>>> [Mon Mar 18 14:59:33 2024] workqueue events: flags=0x0
>>>>>>>>> [Mon Mar 18 14:59:33 2024] pwq 54: cpus=27 node=1 flags=0x0
>>>>>>>>> nice=0
>>>>>>>>> active=1/256 refcnt=2
>>>>>>>>> [Mon Mar 18 14:59:33 2024] pending: drm_fb_helper_damage_work
>>>>>>>>> [drm_kms_helper]
>>>>>>>>> [Mon Mar 18 14:59:33 2024] workqueue events_power_efficient:
>>>>>>>>> flags=0x80
>>>>>>>>> [Mon Mar 18 14:59:33 2024] pwq 54: cpus=27 node=1 flags=0x0
>>>>>>>>> nice=0
>>>>>>>>> active=1/256 refcnt=2
>>>>>>>>> [Mon Mar 18 14:59:33 2024] pending: fb_flashcursor
>>>>>>>>> [Mon Mar 18 14:59:33 2024] workqueue mm_percpu_wq: flags=0x8
>>>>>>>>> [Mon Mar 18 14:59:33 2024] pwq 54: cpus=27 node=1 flags=0x0
>>>>>>>>> nice=0
>>>>>>>>> active=1/256 refcnt=3
>>>>>>>>> [Mon Mar 18 14:59:33 2024] pending: lru_add_drain_per_cpu
>>>>>>>>> BAR(362)
>>>>>>>>> [Mon Mar 18 14:59:33 2024] workqueue kblockd: flags=0x18
>>>>>>>>> [Mon Mar 18 14:59:33 2024] pwq 55: cpus=27 node=1 flags=0x0
>>>>>>>>> nice=-20
>>>>>>>>> active=1/256 refcnt=2
>>>>>>>>> [Mon Mar 18 14:59:33 2024] pending: blk_mq_timeout_work
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> In contrast to last time, it doesn't show anything regarding nfs
>>>>>>>>> this
>>>>>>>>> time.
>>>>>>>>>
>>>>>>>>> I also tried the suggestion from Dai Ngo (echo 3 >
>>>>>>>>> /proc/sys/vm/drop_caches), but that didn't seem to make any
>>>>>>>>> difference.
>>>>>>>>>
>>>>>>>>> We haven't restarted the server yet as it seems the impact seems to
>>>>>>>>> affect fewer clients that before. Is there anything we can run
>>>>>>>>> on the
>>>>>>>>> server to further debug this?
>>>>>>>>>
>>>>>>>>> In the past, the issue seemed to deteriorate rapidly and
>>>>>>>>> resulted in
>>>>>>>>> issues for almost all clients after about 20 minutes. This time the
>>>>>>>>> impact seems to be less, but it's not gone.
>>>>>>>>>
>>>>>>>>> How can we force the NFS server to forget about a specific
>>>>>>>>> client? I
>>>>>>>>> haven't tried to restart the nfs service yet as I'm afraid it will
>>>>>>>>> fail to stop as before.
>>>>>>>>>
>>>>>>> Not with that kernel. There are some new administrative interfaces
>>>>>>> that
>>>>>>> might allow that in the future, but they were just merged upstream
>>>>>>> and
>>>>>>> aren't in that kernel.
>>>>>>>
>>>>>>> --
>>>>>>> Jeff Layton <jlayton@kernel.org>
--
Rik Theys
System Engineer
KU Leuven - Dept. Elektrotechniek (ESAT)
Kasteelpark Arenberg 10 bus 2440 - B-3001 Leuven-Heverlee
+32(0)16/32.11.07
----------------------------------------------------------------
<<Any errors in spelling, tact or fact are transmission errors>>
^ permalink raw reply [flat|nested] 26+ messages in thread
* Re: nfsd hangs and nfsd_break_deleg_cb+0x170/0x190 warning
2024-03-13 19:50 ` Rik Theys
@ 2024-03-13 22:45 ` Dai Ngo
0 siblings, 0 replies; 26+ messages in thread
From: Dai Ngo @ 2024-03-13 22:45 UTC (permalink / raw)
To: Rik Theys, Jeff Layton, Linux Nfs
On 3/13/24 12:50 PM, Rik Theys wrote:
> Hi,
>
> On 3/13/24 19:44, Dai Ngo wrote:
>>
>> On 3/12/24 11:23 AM, Rik Theys wrote:
>>> Hi,
>>>
>>> On 3/12/24 17:43, Dai Ngo wrote:
>>>>
>>>> On 3/12/24 4:37 AM, Jeff Layton wrote:
>>>>> On Mon, 2024-03-11 at 19:43 +0100, Rik Theys wrote:
>>>>>>
>>>>>>
>>>>>>
>>>>>> Hi,
>>>>>>
>>>>>>
>>>>>>
>>>>>> Since a few weeks our Rocky Linux 9 NFS server has periodically
>>>>>> logged hung nfsd tasks. The initial effect was that some clients
>>>>>> could no longer access the NFS server. This got worse and worse
>>>>>> (probably as more nfsd threads got blocked) and we had to restart
>>>>>> the server. Restarting the server also failed as the NFS server
>>>>>> service could no longer be stopped.
>>>>>>
>>>>>>
>>>>>>
>>>>>> The initial kernel we noticed this behavior on was
>>>>>> kernel-5.14.0-362.18.1.el9_3.x86_64. Since then we've installed
>>>>>> kernel-5.14.0-419.el9.x86_64 from CentOS Stream 9. The same issue
>>>>>> happened again on this newer kernel version:
>>>>>>
>>>>>>
>>>>>>
>>>>>> [Mon Mar 11 14:10:08 2024] Not tainted
>>>>>> 5.14.0-419.el9.x86_64 #1
>>>>>> [Mon Mar 11 14:10:08 2024] "echo 0 >
>>>>>> /proc/sys/kernel/hung_task_timeout_secs" disables this message.
>>>>>> [Mon Mar 11 14:10:08 2024] task:nfsd state:D stack:0
>>>>>> pid:8865 ppid:2 flags:0x00004000
>>>>>> [Mon Mar 11 14:10:08 2024] Call Trace:
>>>>>> [Mon Mar 11 14:10:08 2024] <TASK>
>>>>>> [Mon Mar 11 14:10:08 2024] __schedule+0x21b/0x550
>>>>>> [Mon Mar 11 14:10:08 2024] schedule+0x2d/0x70
>>>>>> [Mon Mar 11 14:10:08 2024] schedule_timeout+0x11f/0x160
>>>>>> [Mon Mar 11 14:10:08 2024] ? select_idle_sibling+0x28/0x430
>>>>>> [Mon Mar 11 14:10:08 2024] ? wake_affine+0x62/0x1f0
>>>>>> [Mon Mar 11 14:10:08 2024] __wait_for_common+0x90/0x1d0
>>>>>> [Mon Mar 11 14:10:08 2024] ? __pfx_schedule_timeout+0x10/0x10
>>>>>> [Mon Mar 11 14:10:08 2024] __flush_workqueue+0x13a/0x3f0
>>>>>> [Mon Mar 11 14:10:08 2024] nfsd4_shutdown_callback+0x49/0x120
>>>>>> [nfsd]
>>>>>> [Mon Mar 11 14:10:08 2024] ? nfsd4_cld_remove+0x54/0x1d0 [nfsd]
>>>>>> [Mon Mar 11 14:10:08 2024] ?
>>>>>> nfsd4_return_all_client_layouts+0xc4/0xf0 [nfsd]
>>>>>> [Mon Mar 11 14:10:08 2024] ? nfsd4_shutdown_copy+0x68/0xc0 [nfsd]
>>>>>> [Mon Mar 11 14:10:08 2024] __destroy_client+0x1f3/0x290 [nfsd]
>>>>>> [Mon Mar 11 14:10:08 2024] nfsd4_exchange_id+0x75f/0x770 [nfsd]
>>>>>> [Mon Mar 11 14:10:08 2024] ? nfsd4_decode_opaque+0x3a/0x90 [nfsd]
>>>>>> [Mon Mar 11 14:10:08 2024] nfsd4_proc_compound+0x44b/0x700 [nfsd]
>>>>>> [Mon Mar 11 14:10:08 2024] nfsd_dispatch+0x94/0x1c0 [nfsd]
>>>>>> [Mon Mar 11 14:10:08 2024] svc_process_common+0x2ec/0x660
>>>>>> [sunrpc]
>>>>>> [Mon Mar 11 14:10:08 2024] ? __pfx_nfsd_dispatch+0x10/0x10 [nfsd]
>>>>>> [Mon Mar 11 14:10:08 2024] ? __pfx_nfsd+0x10/0x10 [nfsd]
>>>>>> [Mon Mar 11 14:10:08 2024] svc_process+0x12d/0x170 [sunrpc]
>>>>>> [Mon Mar 11 14:10:08 2024] nfsd+0x84/0xb0 [nfsd]
>>>>>> [Mon Mar 11 14:10:08 2024] kthread+0xdd/0x100
>>>>>> [Mon Mar 11 14:10:08 2024] ? __pfx_kthread+0x10/0x10
>>>>>> [Mon Mar 11 14:10:08 2024] ret_from_fork+0x29/0x50
>>>>>> [Mon Mar 11 14:10:08 2024] </TASK>
>>>>>> [Mon Mar 11 14:10:08 2024] INFO: task nfsd:8866 blocked for
>>>>>> more than 122 seconds.
>>>>>> [Mon Mar 11 14:10:08 2024] Not tainted
>>>>>> 5.14.0-419.el9.x86_64 #1
>>>>>> [Mon Mar 11 14:10:08 2024] "echo 0 >
>>>>>> /proc/sys/kernel/hung_task_timeout_secs" disables this message.
>>>>>> [Mon Mar 11 14:10:08 2024] task:nfsd state:D stack:0
>>>>>> pid:8866 ppid:2 flags:0x00004000
>>>>>> [Mon Mar 11 14:10:08 2024] Call Trace:
>>>>>> [Mon Mar 11 14:10:08 2024] <TASK>
>>>>>> [Mon Mar 11 14:10:08 2024] __schedule+0x21b/0x550
>>>>>> [Mon Mar 11 14:10:08 2024] schedule+0x2d/0x70
>>>>>> [Mon Mar 11 14:10:08 2024] schedule_timeout+0x11f/0x160
>>>>>> [Mon Mar 11 14:10:08 2024] ? select_idle_sibling+0x28/0x430
>>>>>> [Mon Mar 11 14:10:08 2024] ? tcp_recvmsg+0x196/0x210
>>>>>> [Mon Mar 11 14:10:08 2024] ? wake_affine+0x62/0x1f0
>>>>>> [Mon Mar 11 14:10:08 2024] __wait_for_common+0x90/0x1d0
>>>>>> [Mon Mar 11 14:10:08 2024] ? __pfx_schedule_timeout+0x10/0x10
>>>>>> [Mon Mar 11 14:10:08 2024] __flush_workqueue+0x13a/0x3f0
>>>>>> [Mon Mar 11 14:10:08 2024] nfsd4_destroy_session+0x1a4/0x240
>>>>>> [nfsd]
>>>>>> [Mon Mar 11 14:10:08 2024] nfsd4_proc_compound+0x44b/0x700 [nfsd]
>>>>>> [Mon Mar 11 14:10:08 2024] nfsd_dispatch+0x94/0x1c0 [nfsd]
>>>>>> [Mon Mar 11 14:10:08 2024] svc_process_common+0x2ec/0x660
>>>>>> [sunrpc]
>>>>>> [Mon Mar 11 14:10:08 2024] ? __pfx_nfsd_dispatch+0x10/0x10 [nfsd]
>>>>>> [Mon Mar 11 14:10:08 2024] ? __pfx_nfsd+0x10/0x10 [nfsd]
>>>>>> [Mon Mar 11 14:10:08 2024] svc_process+0x12d/0x170 [sunrpc]
>>>>>> [Mon Mar 11 14:10:08 2024] nfsd+0x84/0xb0 [nfsd]
>>>>>> [Mon Mar 11 14:10:08 2024] kthread+0xdd/0x100
>>>>>> [Mon Mar 11 14:10:08 2024] ? __pfx_kthread+0x10/0x10
>>>>>> [Mon Mar 11 14:10:08 2024] ret_from_fork+0x29/0x50
>>>>>> [Mon Mar 11 14:10:08 2024] </TASK>
>>>>>>
>>>>>>
>>>>>>
>>>>>> The above is repeated a few times, and then this warning is
>>>>>> also logged:
>>>>>>
>>>>>>
>>>>>>
>>>>>> [Mon Mar 11 14:12:04 2024] ------------[ cut here ]------------
>>>>>> [Mon Mar 11 14:12:04 2024] WARNING: CPU: 39 PID: 8844 at
>>>>>> fs/nfsd/nfs4state.c:4919 nfsd_break_deleg_cb+0x170/0x190 [nfsd]
>>>>>> [Mon Mar 11 14:12:05 2024] Modules linked in: nfsv4
>>>>>> dns_resolver nfs fscache netfs rpcsec_gss_krb5 rpcrdma rdma_cm
>>>>>> iw_cm ib_cm ib_core binfmt_misc bonding tls rfkill nft_counter
>>>>>> nft_ct
>>>>>> nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_reject_inet
>>>>>> nf_reject_ipv4 nf_reject_ipv6 nft_reject nf_tables nfnetlink vfat
>>>>>> fat dm_thin_pool dm_persistent_data dm_bio_prison dm_bufio l
>>>>>> ibcrc32c dm_service_time dm_multipath intel_rapl_msr
>>>>>> intel_rapl_common intel_uncore_frequency
>>>>>> intel_uncore_frequency_common isst_if_common skx_edac nfit
>>>>>> libnvdimm ipmi_ssif x86_pkg_temp
>>>>>> _thermal intel_powerclamp coretemp kvm_intel kvm irqbypass
>>>>>> dcdbas rapl intel_cstate mgag200 i2c_algo_bit drm_shmem_helper
>>>>>> dell_smbios drm_kms_helper dell_wmi_descriptor wmi_bmof intel_u
>>>>>> ncore syscopyarea pcspkr sysfillrect mei_me sysimgblt acpi_ipmi
>>>>>> mei fb_sys_fops i2c_i801 ipmi_si intel_pch_thermal lpc_ich
>>>>>> ipmi_devintf i2c_smbus ipmi_msghandler joydev acpi_power_meter
>>>>>> nfsd auth_rpcgss nfs_acl drm lockd grace fuse sunrpc ext4
>>>>>> mbcache jbd2 sd_mod sg lpfc
>>>>>> [Mon Mar 11 14:12:05 2024] nvmet_fc nvmet nvme_fc nvme_fabrics
>>>>>> crct10dif_pclmul ahci libahci crc32_pclmul nvme_core crc32c_intel
>>>>>> ixgbe megaraid_sas libata nvme_common ghash_clmulni_int
>>>>>> el t10_pi wdat_wdt scsi_transport_fc mdio wmi dca dm_mirror
>>>>>> dm_region_hash dm_log dm_mod
>>>>>> [Mon Mar 11 14:12:05 2024] CPU: 39 PID: 8844 Comm: nfsd Not
>>>>>> tainted 5.14.0-419.el9.x86_64 #1
>>>>>> [Mon Mar 11 14:12:05 2024] Hardware name: Dell Inc. PowerEdge
>>>>>> R740/00WGD1, BIOS 2.20.1 09/13/2023
>>>>>> [Mon Mar 11 14:12:05 2024] RIP:
>>>>>> 0010:nfsd_break_deleg_cb+0x170/0x190 [nfsd]
>>>>>> [Mon Mar 11 14:12:05 2024] Code: a6 95 c5 f3 e9 ff fe ff ff 48
>>>>>> 89 df be 01 00 00 00 e8 34 b5 13 f4 48 8d bb 98 00 00 00 e8 c8 f9
>>>>>> 00 00 84 c0 0f 85 2e ff ff ff <0f> 0b e9 27 ff ff ff be
>>>>>> 02 00 00 00 48 89 df e8 0c b5 13 f4 e9 01
>>>>>> [Mon Mar 11 14:12:05 2024] RSP: 0018:ffff9929e0bb7b80 EFLAGS:
>>>>>> 00010246
>>>>>> [Mon Mar 11 14:12:05 2024] RAX: 0000000000000000 RBX:
>>>>>> ffff8ada51930900 RCX: 0000000000000024
>>>>>> [Mon Mar 11 14:12:05 2024] RDX: ffff8ada519309c8 RSI:
>>>>>> ffff8ad582933c00 RDI: 0000000000002000
>>>>>> [Mon Mar 11 14:12:05 2024] RBP: ffff8ad46bf21574 R08:
>>>>>> ffff9929e0bb7b48 R09: 0000000000000000
>>>>>> [Mon Mar 11 14:12:05 2024] R10: ffff8aec859a2948 R11:
>>>>>> 0000000000000000 R12: ffff8ad6f497c360
>>>>>> [Mon Mar 11 14:12:05 2024] R13: ffff8ad46bf21560 R14:
>>>>>> ffff8ae5942e0b10 R15: ffff8ad6f497c360
>>>>>> [Mon Mar 11 14:12:05 2024] FS: 0000000000000000(0000)
>>>>>> GS:ffff8b031fcc0000(0000) knlGS:0000000000000000
>>>>>> [Mon Mar 11 14:12:05 2024] CS: 0010 DS: 0000 ES: 0000 CR0:
>>>>>> 0000000080050033
>>>>>> [Mon Mar 11 14:12:05 2024] CR2: 00007fafe2060744 CR3:
>>>>>> 00000018e58de006 CR4: 00000000007706e0
>>>>>> [Mon Mar 11 14:12:05 2024] DR0: 0000000000000000 DR1:
>>>>>> 0000000000000000 DR2: 0000000000000000
>>>>>> [Mon Mar 11 14:12:05 2024] DR3: 0000000000000000 DR6:
>>>>>> 00000000fffe0ff0 DR7: 0000000000000400
>>>>>> [Mon Mar 11 14:12:05 2024] PKRU: 55555554
>>>>>> [Mon Mar 11 14:12:05 2024] Call Trace:
>>>>>> [Mon Mar 11 14:12:05 2024] <TASK>
>>>>>> [Mon Mar 11 14:12:05 2024] ? show_trace_log_lvl+0x1c4/0x2df
>>>>>> [Mon Mar 11 14:12:05 2024] ? show_trace_log_lvl+0x1c4/0x2df
>>>>>> [Mon Mar 11 14:12:05 2024] ? __break_lease+0x16f/0x5f0
>>>>>> [Mon Mar 11 14:12:05 2024] ? nfsd_break_deleg_cb+0x170/0x190
>>>>>> [nfsd]
>>>>>> [Mon Mar 11 14:12:05 2024] ? __warn+0x81/0x110
>>>>>> [Mon Mar 11 14:12:05 2024] ? nfsd_break_deleg_cb+0x170/0x190
>>>>>> [nfsd]
>>>>>> [Mon Mar 11 14:12:05 2024] ? report_bug+0x10a/0x140
>>>>>> [Mon Mar 11 14:12:05 2024] ? handle_bug+0x3c/0x70
>>>>>> [Mon Mar 11 14:12:05 2024] ? exc_invalid_op+0x14/0x70
>>>>>> [Mon Mar 11 14:12:05 2024] ? asm_exc_invalid_op+0x16/0x20
>>>>>> [Mon Mar 11 14:12:05 2024] ? nfsd_break_deleg_cb+0x170/0x190
>>>>>> [nfsd]
>>>>>> [Mon Mar 11 14:12:05 2024] __break_lease+0x16f/0x5f0
>>>>>> [Mon Mar 11 14:12:05 2024] ?
>>>>>> nfsd_file_lookup_locked+0x117/0x160 [nfsd]
>>>>>> [Mon Mar 11 14:12:05 2024] ? list_lru_del+0x101/0x150
>>>>>> [Mon Mar 11 14:12:05 2024] nfsd_file_do_acquire+0x790/0x830
>>>>>> [nfsd]
>>>>>> [Mon Mar 11 14:12:05 2024] nfs4_get_vfs_file+0x315/0x3a0 [nfsd]
>>>>>> [Mon Mar 11 14:12:05 2024] nfsd4_process_open2+0x430/0xa30 [nfsd]
>>>>>> [Mon Mar 11 14:12:05 2024] ? fh_verify+0x297/0x2f0 [nfsd]
>>>>>> [Mon Mar 11 14:12:05 2024] nfsd4_open+0x3ce/0x4b0 [nfsd]
>>>>>> [Mon Mar 11 14:12:05 2024] nfsd4_proc_compound+0x44b/0x700 [nfsd]
>>>>>> [Mon Mar 11 14:12:05 2024] nfsd_dispatch+0x94/0x1c0 [nfsd]
>>>>>> [Mon Mar 11 14:12:05 2024] svc_process_common+0x2ec/0x660
>>>>>> [sunrpc]
>>>>>> [Mon Mar 11 14:12:05 2024] ? __pfx_nfsd_dispatch+0x10/0x10 [nfsd]
>>>>>> [Mon Mar 11 14:12:05 2024] ? __pfx_nfsd+0x10/0x10 [nfsd]
>>>>>> [Mon Mar 11 14:12:05 2024] svc_process+0x12d/0x170 [sunrpc]
>>>>>> [Mon Mar 11 14:12:05 2024] nfsd+0x84/0xb0 [nfsd]
>>>>>> [Mon Mar 11 14:12:05 2024] kthread+0xdd/0x100
>>>>>> [Mon Mar 11 14:12:05 2024] ? __pfx_kthread+0x10/0x10
>>>>>> [Mon Mar 11 14:12:05 2024] ret_from_fork+0x29/0x50
>>>>>> [Mon Mar 11 14:12:05 2024] </TASK>
>>>>>> [Mon Mar 11 14:12:05 2024] ---[ end trace 7a039e17443dc651 ]---
>>>>> [Mon Mar 11 14:29:16 2024] task:kworker/u96:3 state:D
>>>>> stack:0 pid:2451130 ppid:2 flags:0x00004000
>>>>> [Mon Mar 11 14:29:16 2024] Workqueue: nfsd4_callbacks
>>>>> nfsd4_run_cb_work [nfsd]
>>>>> [Mon Mar 11 14:29:16 2024] Call Trace:
>>>>> [Mon Mar 11 14:29:16 2024] <TASK>
>>>>> [Mon Mar 11 14:29:16 2024] __schedule+0x21b/0x550
>>>>> [Mon Mar 11 14:29:16 2024] schedule+0x2d/0x70
>>>>> [Mon Mar 11 14:29:16 2024] schedule_timeout+0x88/0x160
>>>>> [Mon Mar 11 14:29:16 2024] ? __pfx_process_timeout+0x10/0x10
>>>>> [Mon Mar 11 14:29:16 2024] rpc_shutdown_client+0xb3/0x150 [sunrpc]
>>>>> [Mon Mar 11 14:29:16 2024] ?
>>>>> __pfx_autoremove_wake_function+0x10/0x10
>>>>> [Mon Mar 11 14:29:16 2024] nfsd4_process_cb_update+0x3e/0x260 [nfsd]
>>>>> [Mon Mar 11 14:29:16 2024] ? sched_clock+0xc/0x30
>>>>> [Mon Mar 11 14:29:16 2024] ? raw_spin_rq_lock_nested+0x19/0x80
>>>>> [Mon Mar 11 14:29:16 2024] ? newidle_balance+0x26e/0x400
>>>>> [Mon Mar 11 14:29:16 2024] ? pick_next_task_fair+0x41/0x500
>>>>> [Mon Mar 11 14:29:16 2024] ? put_prev_task_fair+0x1e/0x40
>>>>> [Mon Mar 11 14:29:16 2024] ? pick_next_task+0x861/0x950
>>>>> [Mon Mar 11 14:29:16 2024] ? __update_idle_core+0x23/0xc0
>>>>> [Mon Mar 11 14:29:16 2024] ? __switch_to_asm+0x3a/0x80
>>>>> [Mon Mar 11 14:29:16 2024] ? finish_task_switch.isra.0+0x8c/0x2a0
>>>>> [Mon Mar 11 14:29:16 2024] nfsd4_run_cb_work+0x9f/0x150 [nfsd]
>>>>> [Mon Mar 11 14:29:16 2024] process_one_work+0x1e2/0x3b0
>>>>> [Mon Mar 11 14:29:16 2024] worker_thread+0x50/0x3a0
>>>>> [Mon Mar 11 14:29:16 2024] ? __pfx_worker_thread+0x10/0x10
>>>>> [Mon Mar 11 14:29:16 2024] kthread+0xdd/0x100
>>>>> [Mon Mar 11 14:29:16 2024] ? __pfx_kthread+0x10/0x10
>>>>> [Mon Mar 11 14:29:16 2024] ret_from_fork+0x29/0x50
>>>>> [Mon Mar 11 14:29:16 2024] </TASK>
>>>>>
>>>>> The above is the main task that I see in the cb workqueue. It's
>>>>> trying to call rpc_shutdown_client, which is waiting for this:
>>>>>
>>>>> wait_event_timeout(destroy_wait,
>>>>> list_empty(&clnt->cl_tasks), 1*HZ);
>>>>>
>>>>> ...so basically waiting for the cl_tasks list to go empty. It
>>>>> repeatedly
>>>>> does a rpc_killall_tasks though, so possibly trying to kill this
>>>>> task?
>>>>>
>>>>> 18423 2281 0 0x18 0x0 1354 nfsd4_cb_ops [nfsd]
>>>>> nfs4_cbv1 CB_RECALL_ANY a:call_start [sunrpc] q:delayq
>>>>
>>>> I wonder why this task is on delayq. Could it be related to memory
>>>> shortage issue, or connection related problems?
>>>> Output of /proc/meminfo on the nfs server at time of the problem
>>>> would shed some light.
>>>
>>> We don't have that anymore. I can check our monitoring host more
>>> closely for more fine grained stats tomorrow, but when I look at the
>>> sar statistics (see attachment) nothing special was going on memory
>>> or network wise.
>>
>> Thanks Rik for the info.
>>
>> At 2:10 PM sar statistics shows:
>> kbmemfree: 1014880
>> kbavail: 170836368
>> kbmemused: 2160028
>> %memused: 1.10
>> kbcached: 140151288
>>
>> Paging stats:
>> pgpgin/s pgpgout/s fault/s majflt/s pgfree/s
>> pgscank/s pgscand/s pgsteal/s %vmeff
>> 02:10:00 PM 2577.67 491251.09 2247.01 0.00 2415029.61
>> 75131.80 0.00 150276.28 200.02
>>
>> The kbmemfree is pretty low and the caches consume large amount of
>> memory.
>> The paging statistics also show lots of paging activities, 150276.28/s.
>>
> The workload at that time was a write-heavy workload. The writes were
> probably all going to memory until the buffers filled up (or the ext4
> commit interval?) and was then written out to disk. And then the
> process repeated itself as a large part of the writes were rewriting
> the same files over and over again.
>> In the previous rpc_tasks.txt, it shows a RPC task is on the delayq
>> waiting
>> to send the CB_RECALL_ANY. With this version of the kernel, the only
>> time
>> CB_RECALL_ANY is sent is when the system is under memory pressure and
>> the
>> nfsd shrinker task runs to free unused delegations.
>
> Would it help to increase /proc/sys/vm/min_free_kbytes in this case?
I'm not a VM subsystem expert but I've seen recommendation to set this
value to 1% of total memory.
>
>
>> Next time when this problem happens again, you can try to reclaim some
>> memory from the caches to see if it helps:
>>
>> # echo 3 > /proc/sys/vm/drop_caches
>
> Do you think this could help the system recover at that point?
There is no guarantee but it's worth a try.
-Dai
>
> Regards,
>
> Rik
>
>>
>> -Dai
>>
>>
>>
>>
>>>
>>> We start to get the cpu stall messages and the system load starts to
>>> rise (starts around 2:10 PM). At 3:00 PM we restart the server as
>>> our users can no longer work.
>>>
>>> Looking at the stats, the cpu's were ~idle. The only thing that may
>>> be related is that around 11:30 AM the write load (rx packets)
>>> starts to get a lot higher than the read load (tx packets). This
>>> goes on for hours (even after the server was restarted) and that
>>> workload was later identified. It was a workload that was constantly
>>> rewriting a statistics file.
>>>
>>> Regards,
>>>
>>> Rik
>>>
>>>
>>>>
>>>> Currently there is only 1 active task allowed for the nfsd callback
>>>> workqueue at a time. If for some reasons a callback task is stuck in
>>>> the workqueue it will block all other callback tasks which can effect
>>>> multiple clients.
>>>>
>>>> -Dai
>>>>
>>>>>
>>>>> Callbacks are soft RPC tasks though, so they should be easily
>>>>> killable.
>>>
^ permalink raw reply [flat|nested] 26+ messages in thread
* Re: nfsd hangs and nfsd_break_deleg_cb+0x170/0x190 warning
2024-03-13 18:44 ` Dai Ngo
@ 2024-03-13 19:50 ` Rik Theys
2024-03-13 22:45 ` Dai Ngo
0 siblings, 1 reply; 26+ messages in thread
From: Rik Theys @ 2024-03-13 19:50 UTC (permalink / raw)
To: Dai Ngo, Jeff Layton, Linux Nfs
Hi,
On 3/13/24 19:44, Dai Ngo wrote:
>
> On 3/12/24 11:23 AM, Rik Theys wrote:
>> Hi,
>>
>> On 3/12/24 17:43, Dai Ngo wrote:
>>>
>>> On 3/12/24 4:37 AM, Jeff Layton wrote:
>>>> On Mon, 2024-03-11 at 19:43 +0100, Rik Theys wrote:
>>>>>
>>>>>
>>>>>
>>>>> Hi,
>>>>>
>>>>>
>>>>>
>>>>> Since a few weeks our Rocky Linux 9 NFS server has periodically
>>>>> logged hung nfsd tasks. The initial effect was that some clients
>>>>> could no longer access the NFS server. This got worse and worse
>>>>> (probably as more nfsd threads got blocked) and we had to restart
>>>>> the server. Restarting the server also failed as the NFS server
>>>>> service could no longer be stopped.
>>>>>
>>>>>
>>>>>
>>>>> The initial kernel we noticed this behavior on was
>>>>> kernel-5.14.0-362.18.1.el9_3.x86_64. Since then we've installed
>>>>> kernel-5.14.0-419.el9.x86_64 from CentOS Stream 9. The same issue
>>>>> happened again on this newer kernel version:
>>>>>
>>>>>
>>>>>
>>>>> [Mon Mar 11 14:10:08 2024] Not tainted 5.14.0-419.el9.x86_64 #1
>>>>> [Mon Mar 11 14:10:08 2024] "echo 0 >
>>>>> /proc/sys/kernel/hung_task_timeout_secs" disables this message.
>>>>> [Mon Mar 11 14:10:08 2024] task:nfsd state:D stack:0
>>>>> pid:8865 ppid:2 flags:0x00004000
>>>>> [Mon Mar 11 14:10:08 2024] Call Trace:
>>>>> [Mon Mar 11 14:10:08 2024] <TASK>
>>>>> [Mon Mar 11 14:10:08 2024] __schedule+0x21b/0x550
>>>>> [Mon Mar 11 14:10:08 2024] schedule+0x2d/0x70
>>>>> [Mon Mar 11 14:10:08 2024] schedule_timeout+0x11f/0x160
>>>>> [Mon Mar 11 14:10:08 2024] ? select_idle_sibling+0x28/0x430
>>>>> [Mon Mar 11 14:10:08 2024] ? wake_affine+0x62/0x1f0
>>>>> [Mon Mar 11 14:10:08 2024] __wait_for_common+0x90/0x1d0
>>>>> [Mon Mar 11 14:10:08 2024] ? __pfx_schedule_timeout+0x10/0x10
>>>>> [Mon Mar 11 14:10:08 2024] __flush_workqueue+0x13a/0x3f0
>>>>> [Mon Mar 11 14:10:08 2024] nfsd4_shutdown_callback+0x49/0x120
>>>>> [nfsd]
>>>>> [Mon Mar 11 14:10:08 2024] ? nfsd4_cld_remove+0x54/0x1d0 [nfsd]
>>>>> [Mon Mar 11 14:10:08 2024] ?
>>>>> nfsd4_return_all_client_layouts+0xc4/0xf0 [nfsd]
>>>>> [Mon Mar 11 14:10:08 2024] ? nfsd4_shutdown_copy+0x68/0xc0 [nfsd]
>>>>> [Mon Mar 11 14:10:08 2024] __destroy_client+0x1f3/0x290 [nfsd]
>>>>> [Mon Mar 11 14:10:08 2024] nfsd4_exchange_id+0x75f/0x770 [nfsd]
>>>>> [Mon Mar 11 14:10:08 2024] ? nfsd4_decode_opaque+0x3a/0x90 [nfsd]
>>>>> [Mon Mar 11 14:10:08 2024] nfsd4_proc_compound+0x44b/0x700 [nfsd]
>>>>> [Mon Mar 11 14:10:08 2024] nfsd_dispatch+0x94/0x1c0 [nfsd]
>>>>> [Mon Mar 11 14:10:08 2024] svc_process_common+0x2ec/0x660 [sunrpc]
>>>>> [Mon Mar 11 14:10:08 2024] ? __pfx_nfsd_dispatch+0x10/0x10 [nfsd]
>>>>> [Mon Mar 11 14:10:08 2024] ? __pfx_nfsd+0x10/0x10 [nfsd]
>>>>> [Mon Mar 11 14:10:08 2024] svc_process+0x12d/0x170 [sunrpc]
>>>>> [Mon Mar 11 14:10:08 2024] nfsd+0x84/0xb0 [nfsd]
>>>>> [Mon Mar 11 14:10:08 2024] kthread+0xdd/0x100
>>>>> [Mon Mar 11 14:10:08 2024] ? __pfx_kthread+0x10/0x10
>>>>> [Mon Mar 11 14:10:08 2024] ret_from_fork+0x29/0x50
>>>>> [Mon Mar 11 14:10:08 2024] </TASK>
>>>>> [Mon Mar 11 14:10:08 2024] INFO: task nfsd:8866 blocked for more
>>>>> than 122 seconds.
>>>>> [Mon Mar 11 14:10:08 2024] Not tainted
>>>>> 5.14.0-419.el9.x86_64 #1
>>>>> [Mon Mar 11 14:10:08 2024] "echo 0 >
>>>>> /proc/sys/kernel/hung_task_timeout_secs" disables this message.
>>>>> [Mon Mar 11 14:10:08 2024] task:nfsd state:D stack:0
>>>>> pid:8866 ppid:2 flags:0x00004000
>>>>> [Mon Mar 11 14:10:08 2024] Call Trace:
>>>>> [Mon Mar 11 14:10:08 2024] <TASK>
>>>>> [Mon Mar 11 14:10:08 2024] __schedule+0x21b/0x550
>>>>> [Mon Mar 11 14:10:08 2024] schedule+0x2d/0x70
>>>>> [Mon Mar 11 14:10:08 2024] schedule_timeout+0x11f/0x160
>>>>> [Mon Mar 11 14:10:08 2024] ? select_idle_sibling+0x28/0x430
>>>>> [Mon Mar 11 14:10:08 2024] ? tcp_recvmsg+0x196/0x210
>>>>> [Mon Mar 11 14:10:08 2024] ? wake_affine+0x62/0x1f0
>>>>> [Mon Mar 11 14:10:08 2024] __wait_for_common+0x90/0x1d0
>>>>> [Mon Mar 11 14:10:08 2024] ? __pfx_schedule_timeout+0x10/0x10
>>>>> [Mon Mar 11 14:10:08 2024] __flush_workqueue+0x13a/0x3f0
>>>>> [Mon Mar 11 14:10:08 2024] nfsd4_destroy_session+0x1a4/0x240
>>>>> [nfsd]
>>>>> [Mon Mar 11 14:10:08 2024] nfsd4_proc_compound+0x44b/0x700 [nfsd]
>>>>> [Mon Mar 11 14:10:08 2024] nfsd_dispatch+0x94/0x1c0 [nfsd]
>>>>> [Mon Mar 11 14:10:08 2024] svc_process_common+0x2ec/0x660 [sunrpc]
>>>>> [Mon Mar 11 14:10:08 2024] ? __pfx_nfsd_dispatch+0x10/0x10 [nfsd]
>>>>> [Mon Mar 11 14:10:08 2024] ? __pfx_nfsd+0x10/0x10 [nfsd]
>>>>> [Mon Mar 11 14:10:08 2024] svc_process+0x12d/0x170 [sunrpc]
>>>>> [Mon Mar 11 14:10:08 2024] nfsd+0x84/0xb0 [nfsd]
>>>>> [Mon Mar 11 14:10:08 2024] kthread+0xdd/0x100
>>>>> [Mon Mar 11 14:10:08 2024] ? __pfx_kthread+0x10/0x10
>>>>> [Mon Mar 11 14:10:08 2024] ret_from_fork+0x29/0x50
>>>>> [Mon Mar 11 14:10:08 2024] </TASK>
>>>>>
>>>>>
>>>>>
>>>>> The above is repeated a few times, and then this warning is also
>>>>> logged:
>>>>>
>>>>>
>>>>>
>>>>> [Mon Mar 11 14:12:04 2024] ------------[ cut here ]------------
>>>>> [Mon Mar 11 14:12:04 2024] WARNING: CPU: 39 PID: 8844 at
>>>>> fs/nfsd/nfs4state.c:4919 nfsd_break_deleg_cb+0x170/0x190 [nfsd]
>>>>> [Mon Mar 11 14:12:05 2024] Modules linked in: nfsv4 dns_resolver
>>>>> nfs fscache netfs rpcsec_gss_krb5 rpcrdma rdma_cm iw_cm ib_cm
>>>>> ib_core binfmt_misc bonding tls rfkill nft_counter nft_ct
>>>>> nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_reject_inet
>>>>> nf_reject_ipv4 nf_reject_ipv6 nft_reject nf_tables nfnetlink vfat
>>>>> fat dm_thin_pool dm_persistent_data dm_bio_prison dm_bufio l
>>>>> ibcrc32c dm_service_time dm_multipath intel_rapl_msr
>>>>> intel_rapl_common intel_uncore_frequency
>>>>> intel_uncore_frequency_common isst_if_common skx_edac nfit
>>>>> libnvdimm ipmi_ssif x86_pkg_temp
>>>>> _thermal intel_powerclamp coretemp kvm_intel kvm irqbypass
>>>>> dcdbas rapl intel_cstate mgag200 i2c_algo_bit drm_shmem_helper
>>>>> dell_smbios drm_kms_helper dell_wmi_descriptor wmi_bmof intel_u
>>>>> ncore syscopyarea pcspkr sysfillrect mei_me sysimgblt acpi_ipmi
>>>>> mei fb_sys_fops i2c_i801 ipmi_si intel_pch_thermal lpc_ich
>>>>> ipmi_devintf i2c_smbus ipmi_msghandler joydev acpi_power_meter
>>>>> nfsd auth_rpcgss nfs_acl drm lockd grace fuse sunrpc ext4
>>>>> mbcache jbd2 sd_mod sg lpfc
>>>>> [Mon Mar 11 14:12:05 2024] nvmet_fc nvmet nvme_fc nvme_fabrics
>>>>> crct10dif_pclmul ahci libahci crc32_pclmul nvme_core crc32c_intel
>>>>> ixgbe megaraid_sas libata nvme_common ghash_clmulni_int
>>>>> el t10_pi wdat_wdt scsi_transport_fc mdio wmi dca dm_mirror
>>>>> dm_region_hash dm_log dm_mod
>>>>> [Mon Mar 11 14:12:05 2024] CPU: 39 PID: 8844 Comm: nfsd Not
>>>>> tainted 5.14.0-419.el9.x86_64 #1
>>>>> [Mon Mar 11 14:12:05 2024] Hardware name: Dell Inc. PowerEdge
>>>>> R740/00WGD1, BIOS 2.20.1 09/13/2023
>>>>> [Mon Mar 11 14:12:05 2024] RIP:
>>>>> 0010:nfsd_break_deleg_cb+0x170/0x190 [nfsd]
>>>>> [Mon Mar 11 14:12:05 2024] Code: a6 95 c5 f3 e9 ff fe ff ff 48
>>>>> 89 df be 01 00 00 00 e8 34 b5 13 f4 48 8d bb 98 00 00 00 e8 c8 f9
>>>>> 00 00 84 c0 0f 85 2e ff ff ff <0f> 0b e9 27 ff ff ff be
>>>>> 02 00 00 00 48 89 df e8 0c b5 13 f4 e9 01
>>>>> [Mon Mar 11 14:12:05 2024] RSP: 0018:ffff9929e0bb7b80 EFLAGS:
>>>>> 00010246
>>>>> [Mon Mar 11 14:12:05 2024] RAX: 0000000000000000 RBX:
>>>>> ffff8ada51930900 RCX: 0000000000000024
>>>>> [Mon Mar 11 14:12:05 2024] RDX: ffff8ada519309c8 RSI:
>>>>> ffff8ad582933c00 RDI: 0000000000002000
>>>>> [Mon Mar 11 14:12:05 2024] RBP: ffff8ad46bf21574 R08:
>>>>> ffff9929e0bb7b48 R09: 0000000000000000
>>>>> [Mon Mar 11 14:12:05 2024] R10: ffff8aec859a2948 R11:
>>>>> 0000000000000000 R12: ffff8ad6f497c360
>>>>> [Mon Mar 11 14:12:05 2024] R13: ffff8ad46bf21560 R14:
>>>>> ffff8ae5942e0b10 R15: ffff8ad6f497c360
>>>>> [Mon Mar 11 14:12:05 2024] FS: 0000000000000000(0000)
>>>>> GS:ffff8b031fcc0000(0000) knlGS:0000000000000000
>>>>> [Mon Mar 11 14:12:05 2024] CS: 0010 DS: 0000 ES: 0000 CR0:
>>>>> 0000000080050033
>>>>> [Mon Mar 11 14:12:05 2024] CR2: 00007fafe2060744 CR3:
>>>>> 00000018e58de006 CR4: 00000000007706e0
>>>>> [Mon Mar 11 14:12:05 2024] DR0: 0000000000000000 DR1:
>>>>> 0000000000000000 DR2: 0000000000000000
>>>>> [Mon Mar 11 14:12:05 2024] DR3: 0000000000000000 DR6:
>>>>> 00000000fffe0ff0 DR7: 0000000000000400
>>>>> [Mon Mar 11 14:12:05 2024] PKRU: 55555554
>>>>> [Mon Mar 11 14:12:05 2024] Call Trace:
>>>>> [Mon Mar 11 14:12:05 2024] <TASK>
>>>>> [Mon Mar 11 14:12:05 2024] ? show_trace_log_lvl+0x1c4/0x2df
>>>>> [Mon Mar 11 14:12:05 2024] ? show_trace_log_lvl+0x1c4/0x2df
>>>>> [Mon Mar 11 14:12:05 2024] ? __break_lease+0x16f/0x5f0
>>>>> [Mon Mar 11 14:12:05 2024] ? nfsd_break_deleg_cb+0x170/0x190
>>>>> [nfsd]
>>>>> [Mon Mar 11 14:12:05 2024] ? __warn+0x81/0x110
>>>>> [Mon Mar 11 14:12:05 2024] ? nfsd_break_deleg_cb+0x170/0x190
>>>>> [nfsd]
>>>>> [Mon Mar 11 14:12:05 2024] ? report_bug+0x10a/0x140
>>>>> [Mon Mar 11 14:12:05 2024] ? handle_bug+0x3c/0x70
>>>>> [Mon Mar 11 14:12:05 2024] ? exc_invalid_op+0x14/0x70
>>>>> [Mon Mar 11 14:12:05 2024] ? asm_exc_invalid_op+0x16/0x20
>>>>> [Mon Mar 11 14:12:05 2024] ? nfsd_break_deleg_cb+0x170/0x190
>>>>> [nfsd]
>>>>> [Mon Mar 11 14:12:05 2024] __break_lease+0x16f/0x5f0
>>>>> [Mon Mar 11 14:12:05 2024] ?
>>>>> nfsd_file_lookup_locked+0x117/0x160 [nfsd]
>>>>> [Mon Mar 11 14:12:05 2024] ? list_lru_del+0x101/0x150
>>>>> [Mon Mar 11 14:12:05 2024] nfsd_file_do_acquire+0x790/0x830 [nfsd]
>>>>> [Mon Mar 11 14:12:05 2024] nfs4_get_vfs_file+0x315/0x3a0 [nfsd]
>>>>> [Mon Mar 11 14:12:05 2024] nfsd4_process_open2+0x430/0xa30 [nfsd]
>>>>> [Mon Mar 11 14:12:05 2024] ? fh_verify+0x297/0x2f0 [nfsd]
>>>>> [Mon Mar 11 14:12:05 2024] nfsd4_open+0x3ce/0x4b0 [nfsd]
>>>>> [Mon Mar 11 14:12:05 2024] nfsd4_proc_compound+0x44b/0x700 [nfsd]
>>>>> [Mon Mar 11 14:12:05 2024] nfsd_dispatch+0x94/0x1c0 [nfsd]
>>>>> [Mon Mar 11 14:12:05 2024] svc_process_common+0x2ec/0x660 [sunrpc]
>>>>> [Mon Mar 11 14:12:05 2024] ? __pfx_nfsd_dispatch+0x10/0x10 [nfsd]
>>>>> [Mon Mar 11 14:12:05 2024] ? __pfx_nfsd+0x10/0x10 [nfsd]
>>>>> [Mon Mar 11 14:12:05 2024] svc_process+0x12d/0x170 [sunrpc]
>>>>> [Mon Mar 11 14:12:05 2024] nfsd+0x84/0xb0 [nfsd]
>>>>> [Mon Mar 11 14:12:05 2024] kthread+0xdd/0x100
>>>>> [Mon Mar 11 14:12:05 2024] ? __pfx_kthread+0x10/0x10
>>>>> [Mon Mar 11 14:12:05 2024] ret_from_fork+0x29/0x50
>>>>> [Mon Mar 11 14:12:05 2024] </TASK>
>>>>> [Mon Mar 11 14:12:05 2024] ---[ end trace 7a039e17443dc651 ]---
>>>> [Mon Mar 11 14:29:16 2024] task:kworker/u96:3 state:D stack:0
>>>> pid:2451130 ppid:2 flags:0x00004000
>>>> [Mon Mar 11 14:29:16 2024] Workqueue: nfsd4_callbacks
>>>> nfsd4_run_cb_work [nfsd]
>>>> [Mon Mar 11 14:29:16 2024] Call Trace:
>>>> [Mon Mar 11 14:29:16 2024] <TASK>
>>>> [Mon Mar 11 14:29:16 2024] __schedule+0x21b/0x550
>>>> [Mon Mar 11 14:29:16 2024] schedule+0x2d/0x70
>>>> [Mon Mar 11 14:29:16 2024] schedule_timeout+0x88/0x160
>>>> [Mon Mar 11 14:29:16 2024] ? __pfx_process_timeout+0x10/0x10
>>>> [Mon Mar 11 14:29:16 2024] rpc_shutdown_client+0xb3/0x150 [sunrpc]
>>>> [Mon Mar 11 14:29:16 2024] ? __pfx_autoremove_wake_function+0x10/0x10
>>>> [Mon Mar 11 14:29:16 2024] nfsd4_process_cb_update+0x3e/0x260 [nfsd]
>>>> [Mon Mar 11 14:29:16 2024] ? sched_clock+0xc/0x30
>>>> [Mon Mar 11 14:29:16 2024] ? raw_spin_rq_lock_nested+0x19/0x80
>>>> [Mon Mar 11 14:29:16 2024] ? newidle_balance+0x26e/0x400
>>>> [Mon Mar 11 14:29:16 2024] ? pick_next_task_fair+0x41/0x500
>>>> [Mon Mar 11 14:29:16 2024] ? put_prev_task_fair+0x1e/0x40
>>>> [Mon Mar 11 14:29:16 2024] ? pick_next_task+0x861/0x950
>>>> [Mon Mar 11 14:29:16 2024] ? __update_idle_core+0x23/0xc0
>>>> [Mon Mar 11 14:29:16 2024] ? __switch_to_asm+0x3a/0x80
>>>> [Mon Mar 11 14:29:16 2024] ? finish_task_switch.isra.0+0x8c/0x2a0
>>>> [Mon Mar 11 14:29:16 2024] nfsd4_run_cb_work+0x9f/0x150 [nfsd]
>>>> [Mon Mar 11 14:29:16 2024] process_one_work+0x1e2/0x3b0
>>>> [Mon Mar 11 14:29:16 2024] worker_thread+0x50/0x3a0
>>>> [Mon Mar 11 14:29:16 2024] ? __pfx_worker_thread+0x10/0x10
>>>> [Mon Mar 11 14:29:16 2024] kthread+0xdd/0x100
>>>> [Mon Mar 11 14:29:16 2024] ? __pfx_kthread+0x10/0x10
>>>> [Mon Mar 11 14:29:16 2024] ret_from_fork+0x29/0x50
>>>> [Mon Mar 11 14:29:16 2024] </TASK>
>>>>
>>>> The above is the main task that I see in the cb workqueue. It's
>>>> trying to call rpc_shutdown_client, which is waiting for this:
>>>>
>>>> wait_event_timeout(destroy_wait,
>>>> list_empty(&clnt->cl_tasks), 1*HZ);
>>>>
>>>> ...so basically waiting for the cl_tasks list to go empty. It
>>>> repeatedly
>>>> does a rpc_killall_tasks though, so possibly trying to kill this task?
>>>>
>>>> 18423 2281 0 0x18 0x0 1354 nfsd4_cb_ops [nfsd]
>>>> nfs4_cbv1 CB_RECALL_ANY a:call_start [sunrpc] q:delayq
>>>
>>> I wonder why this task is on delayq. Could it be related to memory
>>> shortage issue, or connection related problems?
>>> Output of /proc/meminfo on the nfs server at time of the problem
>>> would shed some light.
>>
>> We don't have that anymore. I can check our monitoring host more
>> closely for more fine grained stats tomorrow, but when I look at the
>> sar statistics (see attachment) nothing special was going on memory
>> or network wise.
>
> Thanks Rik for the info.
>
> At 2:10 PM sar statistics shows:
> kbmemfree: 1014880
> kbavail: 170836368
> kbmemused: 2160028
> %memused: 1.10
> kbcached: 140151288
>
> Paging stats:
> pgpgin/s pgpgout/s fault/s majflt/s pgfree/s
> pgscank/s pgscand/s pgsteal/s %vmeff
> 02:10:00 PM 2577.67 491251.09 2247.01 0.00 2415029.61
> 75131.80 0.00 150276.28 200.02
>
> The kbmemfree is pretty low and the caches consume large amount of
> memory.
> The paging statistics also show lots of paging activities, 150276.28/s.
>
The workload at that time was a write-heavy workload. The writes were
probably all going to memory until the buffers filled up (or the ext4
commit interval?) and was then written out to disk. And then the process
repeated itself as a large part of the writes were rewriting the same
files over and over again.
> In the previous rpc_tasks.txt, it shows a RPC task is on the delayq
> waiting
> to send the CB_RECALL_ANY. With this version of the kernel, the only time
> CB_RECALL_ANY is sent is when the system is under memory pressure and the
> nfsd shrinker task runs to free unused delegations.
Would it help to increase /proc/sys/vm/min_free_kbytes in this case?
> Next time when this problem happens again, you can try to reclaim some
> memory from the caches to see if it helps:
>
> # echo 3 > /proc/sys/vm/drop_caches
Do you think this could help the system recover at that point?
Regards,
Rik
>
> -Dai
>
>
>
>
>>
>> We start to get the cpu stall messages and the system load starts to
>> rise (starts around 2:10 PM). At 3:00 PM we restart the server as our
>> users can no longer work.
>>
>> Looking at the stats, the cpu's were ~idle. The only thing that may
>> be related is that around 11:30 AM the write load (rx packets) starts
>> to get a lot higher than the read load (tx packets). This goes on for
>> hours (even after the server was restarted) and that workload was
>> later identified. It was a workload that was constantly rewriting a
>> statistics file.
>>
>> Regards,
>>
>> Rik
>>
>>
>>>
>>> Currently there is only 1 active task allowed for the nfsd callback
>>> workqueue at a time. If for some reasons a callback task is stuck in
>>> the workqueue it will block all other callback tasks which can effect
>>> multiple clients.
>>>
>>> -Dai
>>>
>>>>
>>>> Callbacks are soft RPC tasks though, so they should be easily
>>>> killable.
>>
--
Rik Theys
System Engineer
KU Leuven - Dept. Elektrotechniek (ESAT)
Kasteelpark Arenberg 10 bus 2440 - B-3001 Leuven-Heverlee
+32(0)16/32.11.07
----------------------------------------------------------------
<<Any errors in spelling, tact or fact are transmission errors>>
^ permalink raw reply [flat|nested] 26+ messages in thread
* Re: nfsd hangs and nfsd_break_deleg_cb+0x170/0x190 warning
2024-03-12 18:23 ` Rik Theys
@ 2024-03-13 18:44 ` Dai Ngo
2024-03-13 19:50 ` Rik Theys
0 siblings, 1 reply; 26+ messages in thread
From: Dai Ngo @ 2024-03-13 18:44 UTC (permalink / raw)
To: Rik Theys, Jeff Layton, Linux Nfs
On 3/12/24 11:23 AM, Rik Theys wrote:
> Hi,
>
> On 3/12/24 17:43, Dai Ngo wrote:
>>
>> On 3/12/24 4:37 AM, Jeff Layton wrote:
>>> On Mon, 2024-03-11 at 19:43 +0100, Rik Theys wrote:
>>>>
>>>>
>>>>
>>>> Hi,
>>>>
>>>>
>>>>
>>>> Since a few weeks our Rocky Linux 9 NFS server has periodically
>>>> logged hung nfsd tasks. The initial effect was that some clients
>>>> could no longer access the NFS server. This got worse and worse
>>>> (probably as more nfsd threads got blocked) and we had to restart
>>>> the server. Restarting the server also failed as the NFS server
>>>> service could no longer be stopped.
>>>>
>>>>
>>>>
>>>> The initial kernel we noticed this behavior on was
>>>> kernel-5.14.0-362.18.1.el9_3.x86_64. Since then we've installed
>>>> kernel-5.14.0-419.el9.x86_64 from CentOS Stream 9. The same issue
>>>> happened again on this newer kernel version:
>>>>
>>>>
>>>>
>>>> [Mon Mar 11 14:10:08 2024] Not tainted 5.14.0-419.el9.x86_64 #1
>>>> [Mon Mar 11 14:10:08 2024] "echo 0 >
>>>> /proc/sys/kernel/hung_task_timeout_secs" disables this message.
>>>> [Mon Mar 11 14:10:08 2024] task:nfsd state:D stack:0
>>>> pid:8865 ppid:2 flags:0x00004000
>>>> [Mon Mar 11 14:10:08 2024] Call Trace:
>>>> [Mon Mar 11 14:10:08 2024] <TASK>
>>>> [Mon Mar 11 14:10:08 2024] __schedule+0x21b/0x550
>>>> [Mon Mar 11 14:10:08 2024] schedule+0x2d/0x70
>>>> [Mon Mar 11 14:10:08 2024] schedule_timeout+0x11f/0x160
>>>> [Mon Mar 11 14:10:08 2024] ? select_idle_sibling+0x28/0x430
>>>> [Mon Mar 11 14:10:08 2024] ? wake_affine+0x62/0x1f0
>>>> [Mon Mar 11 14:10:08 2024] __wait_for_common+0x90/0x1d0
>>>> [Mon Mar 11 14:10:08 2024] ? __pfx_schedule_timeout+0x10/0x10
>>>> [Mon Mar 11 14:10:08 2024] __flush_workqueue+0x13a/0x3f0
>>>> [Mon Mar 11 14:10:08 2024] nfsd4_shutdown_callback+0x49/0x120
>>>> [nfsd]
>>>> [Mon Mar 11 14:10:08 2024] ? nfsd4_cld_remove+0x54/0x1d0 [nfsd]
>>>> [Mon Mar 11 14:10:08 2024] ?
>>>> nfsd4_return_all_client_layouts+0xc4/0xf0 [nfsd]
>>>> [Mon Mar 11 14:10:08 2024] ? nfsd4_shutdown_copy+0x68/0xc0 [nfsd]
>>>> [Mon Mar 11 14:10:08 2024] __destroy_client+0x1f3/0x290 [nfsd]
>>>> [Mon Mar 11 14:10:08 2024] nfsd4_exchange_id+0x75f/0x770 [nfsd]
>>>> [Mon Mar 11 14:10:08 2024] ? nfsd4_decode_opaque+0x3a/0x90 [nfsd]
>>>> [Mon Mar 11 14:10:08 2024] nfsd4_proc_compound+0x44b/0x700 [nfsd]
>>>> [Mon Mar 11 14:10:08 2024] nfsd_dispatch+0x94/0x1c0 [nfsd]
>>>> [Mon Mar 11 14:10:08 2024] svc_process_common+0x2ec/0x660 [sunrpc]
>>>> [Mon Mar 11 14:10:08 2024] ? __pfx_nfsd_dispatch+0x10/0x10 [nfsd]
>>>> [Mon Mar 11 14:10:08 2024] ? __pfx_nfsd+0x10/0x10 [nfsd]
>>>> [Mon Mar 11 14:10:08 2024] svc_process+0x12d/0x170 [sunrpc]
>>>> [Mon Mar 11 14:10:08 2024] nfsd+0x84/0xb0 [nfsd]
>>>> [Mon Mar 11 14:10:08 2024] kthread+0xdd/0x100
>>>> [Mon Mar 11 14:10:08 2024] ? __pfx_kthread+0x10/0x10
>>>> [Mon Mar 11 14:10:08 2024] ret_from_fork+0x29/0x50
>>>> [Mon Mar 11 14:10:08 2024] </TASK>
>>>> [Mon Mar 11 14:10:08 2024] INFO: task nfsd:8866 blocked for more
>>>> than 122 seconds.
>>>> [Mon Mar 11 14:10:08 2024] Not tainted
>>>> 5.14.0-419.el9.x86_64 #1
>>>> [Mon Mar 11 14:10:08 2024] "echo 0 >
>>>> /proc/sys/kernel/hung_task_timeout_secs" disables this message.
>>>> [Mon Mar 11 14:10:08 2024] task:nfsd state:D stack:0
>>>> pid:8866 ppid:2 flags:0x00004000
>>>> [Mon Mar 11 14:10:08 2024] Call Trace:
>>>> [Mon Mar 11 14:10:08 2024] <TASK>
>>>> [Mon Mar 11 14:10:08 2024] __schedule+0x21b/0x550
>>>> [Mon Mar 11 14:10:08 2024] schedule+0x2d/0x70
>>>> [Mon Mar 11 14:10:08 2024] schedule_timeout+0x11f/0x160
>>>> [Mon Mar 11 14:10:08 2024] ? select_idle_sibling+0x28/0x430
>>>> [Mon Mar 11 14:10:08 2024] ? tcp_recvmsg+0x196/0x210
>>>> [Mon Mar 11 14:10:08 2024] ? wake_affine+0x62/0x1f0
>>>> [Mon Mar 11 14:10:08 2024] __wait_for_common+0x90/0x1d0
>>>> [Mon Mar 11 14:10:08 2024] ? __pfx_schedule_timeout+0x10/0x10
>>>> [Mon Mar 11 14:10:08 2024] __flush_workqueue+0x13a/0x3f0
>>>> [Mon Mar 11 14:10:08 2024] nfsd4_destroy_session+0x1a4/0x240 [nfsd]
>>>> [Mon Mar 11 14:10:08 2024] nfsd4_proc_compound+0x44b/0x700 [nfsd]
>>>> [Mon Mar 11 14:10:08 2024] nfsd_dispatch+0x94/0x1c0 [nfsd]
>>>> [Mon Mar 11 14:10:08 2024] svc_process_common+0x2ec/0x660 [sunrpc]
>>>> [Mon Mar 11 14:10:08 2024] ? __pfx_nfsd_dispatch+0x10/0x10 [nfsd]
>>>> [Mon Mar 11 14:10:08 2024] ? __pfx_nfsd+0x10/0x10 [nfsd]
>>>> [Mon Mar 11 14:10:08 2024] svc_process+0x12d/0x170 [sunrpc]
>>>> [Mon Mar 11 14:10:08 2024] nfsd+0x84/0xb0 [nfsd]
>>>> [Mon Mar 11 14:10:08 2024] kthread+0xdd/0x100
>>>> [Mon Mar 11 14:10:08 2024] ? __pfx_kthread+0x10/0x10
>>>> [Mon Mar 11 14:10:08 2024] ret_from_fork+0x29/0x50
>>>> [Mon Mar 11 14:10:08 2024] </TASK>
>>>>
>>>>
>>>>
>>>> The above is repeated a few times, and then this warning is also
>>>> logged:
>>>>
>>>>
>>>>
>>>> [Mon Mar 11 14:12:04 2024] ------------[ cut here ]------------
>>>> [Mon Mar 11 14:12:04 2024] WARNING: CPU: 39 PID: 8844 at
>>>> fs/nfsd/nfs4state.c:4919 nfsd_break_deleg_cb+0x170/0x190 [nfsd]
>>>> [Mon Mar 11 14:12:05 2024] Modules linked in: nfsv4 dns_resolver
>>>> nfs fscache netfs rpcsec_gss_krb5 rpcrdma rdma_cm iw_cm ib_cm
>>>> ib_core binfmt_misc bonding tls rfkill nft_counter nft_ct
>>>> nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_reject_inet
>>>> nf_reject_ipv4 nf_reject_ipv6 nft_reject nf_tables nfnetlink vfat
>>>> fat dm_thin_pool dm_persistent_data dm_bio_prison dm_bufio l
>>>> ibcrc32c dm_service_time dm_multipath intel_rapl_msr
>>>> intel_rapl_common intel_uncore_frequency
>>>> intel_uncore_frequency_common isst_if_common skx_edac nfit
>>>> libnvdimm ipmi_ssif x86_pkg_temp
>>>> _thermal intel_powerclamp coretemp kvm_intel kvm irqbypass dcdbas
>>>> rapl intel_cstate mgag200 i2c_algo_bit drm_shmem_helper dell_smbios
>>>> drm_kms_helper dell_wmi_descriptor wmi_bmof intel_u
>>>> ncore syscopyarea pcspkr sysfillrect mei_me sysimgblt acpi_ipmi
>>>> mei fb_sys_fops i2c_i801 ipmi_si intel_pch_thermal lpc_ich
>>>> ipmi_devintf i2c_smbus ipmi_msghandler joydev acpi_power_meter
>>>> nfsd auth_rpcgss nfs_acl drm lockd grace fuse sunrpc ext4 mbcache
>>>> jbd2 sd_mod sg lpfc
>>>> [Mon Mar 11 14:12:05 2024] nvmet_fc nvmet nvme_fc nvme_fabrics
>>>> crct10dif_pclmul ahci libahci crc32_pclmul nvme_core crc32c_intel
>>>> ixgbe megaraid_sas libata nvme_common ghash_clmulni_int
>>>> el t10_pi wdat_wdt scsi_transport_fc mdio wmi dca dm_mirror
>>>> dm_region_hash dm_log dm_mod
>>>> [Mon Mar 11 14:12:05 2024] CPU: 39 PID: 8844 Comm: nfsd Not
>>>> tainted 5.14.0-419.el9.x86_64 #1
>>>> [Mon Mar 11 14:12:05 2024] Hardware name: Dell Inc. PowerEdge
>>>> R740/00WGD1, BIOS 2.20.1 09/13/2023
>>>> [Mon Mar 11 14:12:05 2024] RIP:
>>>> 0010:nfsd_break_deleg_cb+0x170/0x190 [nfsd]
>>>> [Mon Mar 11 14:12:05 2024] Code: a6 95 c5 f3 e9 ff fe ff ff 48 89
>>>> df be 01 00 00 00 e8 34 b5 13 f4 48 8d bb 98 00 00 00 e8 c8 f9 00
>>>> 00 84 c0 0f 85 2e ff ff ff <0f> 0b e9 27 ff ff ff be
>>>> 02 00 00 00 48 89 df e8 0c b5 13 f4 e9 01
>>>> [Mon Mar 11 14:12:05 2024] RSP: 0018:ffff9929e0bb7b80 EFLAGS:
>>>> 00010246
>>>> [Mon Mar 11 14:12:05 2024] RAX: 0000000000000000 RBX:
>>>> ffff8ada51930900 RCX: 0000000000000024
>>>> [Mon Mar 11 14:12:05 2024] RDX: ffff8ada519309c8 RSI:
>>>> ffff8ad582933c00 RDI: 0000000000002000
>>>> [Mon Mar 11 14:12:05 2024] RBP: ffff8ad46bf21574 R08:
>>>> ffff9929e0bb7b48 R09: 0000000000000000
>>>> [Mon Mar 11 14:12:05 2024] R10: ffff8aec859a2948 R11:
>>>> 0000000000000000 R12: ffff8ad6f497c360
>>>> [Mon Mar 11 14:12:05 2024] R13: ffff8ad46bf21560 R14:
>>>> ffff8ae5942e0b10 R15: ffff8ad6f497c360
>>>> [Mon Mar 11 14:12:05 2024] FS: 0000000000000000(0000)
>>>> GS:ffff8b031fcc0000(0000) knlGS:0000000000000000
>>>> [Mon Mar 11 14:12:05 2024] CS: 0010 DS: 0000 ES: 0000 CR0:
>>>> 0000000080050033
>>>> [Mon Mar 11 14:12:05 2024] CR2: 00007fafe2060744 CR3:
>>>> 00000018e58de006 CR4: 00000000007706e0
>>>> [Mon Mar 11 14:12:05 2024] DR0: 0000000000000000 DR1:
>>>> 0000000000000000 DR2: 0000000000000000
>>>> [Mon Mar 11 14:12:05 2024] DR3: 0000000000000000 DR6:
>>>> 00000000fffe0ff0 DR7: 0000000000000400
>>>> [Mon Mar 11 14:12:05 2024] PKRU: 55555554
>>>> [Mon Mar 11 14:12:05 2024] Call Trace:
>>>> [Mon Mar 11 14:12:05 2024] <TASK>
>>>> [Mon Mar 11 14:12:05 2024] ? show_trace_log_lvl+0x1c4/0x2df
>>>> [Mon Mar 11 14:12:05 2024] ? show_trace_log_lvl+0x1c4/0x2df
>>>> [Mon Mar 11 14:12:05 2024] ? __break_lease+0x16f/0x5f0
>>>> [Mon Mar 11 14:12:05 2024] ? nfsd_break_deleg_cb+0x170/0x190 [nfsd]
>>>> [Mon Mar 11 14:12:05 2024] ? __warn+0x81/0x110
>>>> [Mon Mar 11 14:12:05 2024] ? nfsd_break_deleg_cb+0x170/0x190 [nfsd]
>>>> [Mon Mar 11 14:12:05 2024] ? report_bug+0x10a/0x140
>>>> [Mon Mar 11 14:12:05 2024] ? handle_bug+0x3c/0x70
>>>> [Mon Mar 11 14:12:05 2024] ? exc_invalid_op+0x14/0x70
>>>> [Mon Mar 11 14:12:05 2024] ? asm_exc_invalid_op+0x16/0x20
>>>> [Mon Mar 11 14:12:05 2024] ? nfsd_break_deleg_cb+0x170/0x190 [nfsd]
>>>> [Mon Mar 11 14:12:05 2024] __break_lease+0x16f/0x5f0
>>>> [Mon Mar 11 14:12:05 2024] ? nfsd_file_lookup_locked+0x117/0x160
>>>> [nfsd]
>>>> [Mon Mar 11 14:12:05 2024] ? list_lru_del+0x101/0x150
>>>> [Mon Mar 11 14:12:05 2024] nfsd_file_do_acquire+0x790/0x830 [nfsd]
>>>> [Mon Mar 11 14:12:05 2024] nfs4_get_vfs_file+0x315/0x3a0 [nfsd]
>>>> [Mon Mar 11 14:12:05 2024] nfsd4_process_open2+0x430/0xa30 [nfsd]
>>>> [Mon Mar 11 14:12:05 2024] ? fh_verify+0x297/0x2f0 [nfsd]
>>>> [Mon Mar 11 14:12:05 2024] nfsd4_open+0x3ce/0x4b0 [nfsd]
>>>> [Mon Mar 11 14:12:05 2024] nfsd4_proc_compound+0x44b/0x700 [nfsd]
>>>> [Mon Mar 11 14:12:05 2024] nfsd_dispatch+0x94/0x1c0 [nfsd]
>>>> [Mon Mar 11 14:12:05 2024] svc_process_common+0x2ec/0x660 [sunrpc]
>>>> [Mon Mar 11 14:12:05 2024] ? __pfx_nfsd_dispatch+0x10/0x10 [nfsd]
>>>> [Mon Mar 11 14:12:05 2024] ? __pfx_nfsd+0x10/0x10 [nfsd]
>>>> [Mon Mar 11 14:12:05 2024] svc_process+0x12d/0x170 [sunrpc]
>>>> [Mon Mar 11 14:12:05 2024] nfsd+0x84/0xb0 [nfsd]
>>>> [Mon Mar 11 14:12:05 2024] kthread+0xdd/0x100
>>>> [Mon Mar 11 14:12:05 2024] ? __pfx_kthread+0x10/0x10
>>>> [Mon Mar 11 14:12:05 2024] ret_from_fork+0x29/0x50
>>>> [Mon Mar 11 14:12:05 2024] </TASK>
>>>> [Mon Mar 11 14:12:05 2024] ---[ end trace 7a039e17443dc651 ]---
>>> [Mon Mar 11 14:29:16 2024] task:kworker/u96:3 state:D stack:0
>>> pid:2451130 ppid:2 flags:0x00004000
>>> [Mon Mar 11 14:29:16 2024] Workqueue: nfsd4_callbacks
>>> nfsd4_run_cb_work [nfsd]
>>> [Mon Mar 11 14:29:16 2024] Call Trace:
>>> [Mon Mar 11 14:29:16 2024] <TASK>
>>> [Mon Mar 11 14:29:16 2024] __schedule+0x21b/0x550
>>> [Mon Mar 11 14:29:16 2024] schedule+0x2d/0x70
>>> [Mon Mar 11 14:29:16 2024] schedule_timeout+0x88/0x160
>>> [Mon Mar 11 14:29:16 2024] ? __pfx_process_timeout+0x10/0x10
>>> [Mon Mar 11 14:29:16 2024] rpc_shutdown_client+0xb3/0x150 [sunrpc]
>>> [Mon Mar 11 14:29:16 2024] ? __pfx_autoremove_wake_function+0x10/0x10
>>> [Mon Mar 11 14:29:16 2024] nfsd4_process_cb_update+0x3e/0x260 [nfsd]
>>> [Mon Mar 11 14:29:16 2024] ? sched_clock+0xc/0x30
>>> [Mon Mar 11 14:29:16 2024] ? raw_spin_rq_lock_nested+0x19/0x80
>>> [Mon Mar 11 14:29:16 2024] ? newidle_balance+0x26e/0x400
>>> [Mon Mar 11 14:29:16 2024] ? pick_next_task_fair+0x41/0x500
>>> [Mon Mar 11 14:29:16 2024] ? put_prev_task_fair+0x1e/0x40
>>> [Mon Mar 11 14:29:16 2024] ? pick_next_task+0x861/0x950
>>> [Mon Mar 11 14:29:16 2024] ? __update_idle_core+0x23/0xc0
>>> [Mon Mar 11 14:29:16 2024] ? __switch_to_asm+0x3a/0x80
>>> [Mon Mar 11 14:29:16 2024] ? finish_task_switch.isra.0+0x8c/0x2a0
>>> [Mon Mar 11 14:29:16 2024] nfsd4_run_cb_work+0x9f/0x150 [nfsd]
>>> [Mon Mar 11 14:29:16 2024] process_one_work+0x1e2/0x3b0
>>> [Mon Mar 11 14:29:16 2024] worker_thread+0x50/0x3a0
>>> [Mon Mar 11 14:29:16 2024] ? __pfx_worker_thread+0x10/0x10
>>> [Mon Mar 11 14:29:16 2024] kthread+0xdd/0x100
>>> [Mon Mar 11 14:29:16 2024] ? __pfx_kthread+0x10/0x10
>>> [Mon Mar 11 14:29:16 2024] ret_from_fork+0x29/0x50
>>> [Mon Mar 11 14:29:16 2024] </TASK>
>>>
>>> The above is the main task that I see in the cb workqueue. It's
>>> trying to call rpc_shutdown_client, which is waiting for this:
>>>
>>> wait_event_timeout(destroy_wait,
>>> list_empty(&clnt->cl_tasks), 1*HZ);
>>>
>>> ...so basically waiting for the cl_tasks list to go empty. It
>>> repeatedly
>>> does a rpc_killall_tasks though, so possibly trying to kill this task?
>>>
>>> 18423 2281 0 0x18 0x0 1354 nfsd4_cb_ops [nfsd]
>>> nfs4_cbv1 CB_RECALL_ANY a:call_start [sunrpc] q:delayq
>>
>> I wonder why this task is on delayq. Could it be related to memory
>> shortage issue, or connection related problems?
>> Output of /proc/meminfo on the nfs server at time of the problem
>> would shed some light.
>
> We don't have that anymore. I can check our monitoring host more
> closely for more fine grained stats tomorrow, but when I look at the
> sar statistics (see attachment) nothing special was going on memory or
> network wise.
Thanks Rik for the info.
At 2:10 PM sar statistics shows:
kbmemfree: 1014880
kbavail: 170836368
kbmemused: 2160028
%memused: 1.10
kbcached: 140151288
Paging stats:
pgpgin/s pgpgout/s fault/s majflt/s pgfree/s pgscank/s pgscand/s pgsteal/s %vmeff
02:10:00 PM 2577.67 491251.09 2247.01 0.00 2415029.61 75131.80 0.00 150276.28 200.02
The kbmemfree is pretty low and the caches consume large amount of memory.
The paging statistics also show lots of paging activities, 150276.28/s.
In the previous rpc_tasks.txt, it shows a RPC task is on the delayq waiting
to send the CB_RECALL_ANY. With this version of the kernel, the only time
CB_RECALL_ANY is sent is when the system is under memory pressure and the
nfsd shrinker task runs to free unused delegations.
Next time when this problem happens again, you can try to reclaim some
memory from the caches to see if it helps:
# echo 3 > /proc/sys/vm/drop_caches
-Dai
>
> We start to get the cpu stall messages and the system load starts to
> rise (starts around 2:10 PM). At 3:00 PM we restart the server as our
> users can no longer work.
>
> Looking at the stats, the cpu's were ~idle. The only thing that may be
> related is that around 11:30 AM the write load (rx packets) starts to
> get a lot higher than the read load (tx packets). This goes on for
> hours (even after the server was restarted) and that workload was
> later identified. It was a workload that was constantly rewriting a
> statistics file.
>
> Regards,
>
> Rik
>
>
>>
>> Currently there is only 1 active task allowed for the nfsd callback
>> workqueue at a time. If for some reasons a callback task is stuck in
>> the workqueue it will block all other callback tasks which can effect
>> multiple clients.
>>
>> -Dai
>>
>>>
>>> Callbacks are soft RPC tasks though, so they should be easily killable.
>
^ permalink raw reply [flat|nested] 26+ messages in thread
* Re: nfsd hangs and nfsd_break_deleg_cb+0x170/0x190 warning
2024-03-12 16:43 ` Dai Ngo
@ 2024-03-12 18:23 ` Rik Theys
2024-03-13 18:44 ` Dai Ngo
0 siblings, 1 reply; 26+ messages in thread
From: Rik Theys @ 2024-03-12 18:23 UTC (permalink / raw)
To: Dai Ngo, Jeff Layton, Linux Nfs
[-- Attachment #1: Type: text/plain, Size: 14587 bytes --]
Hi,
On 3/12/24 17:43, Dai Ngo wrote:
>
> On 3/12/24 4:37 AM, Jeff Layton wrote:
>> On Mon, 2024-03-11 at 19:43 +0100, Rik Theys wrote:
>>>
>>>
>>>
>>> Hi,
>>>
>>>
>>>
>>> Since a few weeks our Rocky Linux 9 NFS server has periodically
>>> logged hung nfsd tasks. The initial effect was that some clients
>>> could no longer access the NFS server. This got worse and worse
>>> (probably as more nfsd threads got blocked) and we had to restart
>>> the server. Restarting the server also failed as the NFS server
>>> service could no longer be stopped.
>>>
>>>
>>>
>>> The initial kernel we noticed this behavior on was
>>> kernel-5.14.0-362.18.1.el9_3.x86_64. Since then we've installed
>>> kernel-5.14.0-419.el9.x86_64 from CentOS Stream 9. The same issue
>>> happened again on this newer kernel version:
>>>
>>>
>>>
>>> [Mon Mar 11 14:10:08 2024] Not tainted 5.14.0-419.el9.x86_64 #1
>>> [Mon Mar 11 14:10:08 2024] "echo 0 >
>>> /proc/sys/kernel/hung_task_timeout_secs" disables this message.
>>> [Mon Mar 11 14:10:08 2024] task:nfsd state:D stack:0
>>> pid:8865 ppid:2 flags:0x00004000
>>> [Mon Mar 11 14:10:08 2024] Call Trace:
>>> [Mon Mar 11 14:10:08 2024] <TASK>
>>> [Mon Mar 11 14:10:08 2024] __schedule+0x21b/0x550
>>> [Mon Mar 11 14:10:08 2024] schedule+0x2d/0x70
>>> [Mon Mar 11 14:10:08 2024] schedule_timeout+0x11f/0x160
>>> [Mon Mar 11 14:10:08 2024] ? select_idle_sibling+0x28/0x430
>>> [Mon Mar 11 14:10:08 2024] ? wake_affine+0x62/0x1f0
>>> [Mon Mar 11 14:10:08 2024] __wait_for_common+0x90/0x1d0
>>> [Mon Mar 11 14:10:08 2024] ? __pfx_schedule_timeout+0x10/0x10
>>> [Mon Mar 11 14:10:08 2024] __flush_workqueue+0x13a/0x3f0
>>> [Mon Mar 11 14:10:08 2024] nfsd4_shutdown_callback+0x49/0x120 [nfsd]
>>> [Mon Mar 11 14:10:08 2024] ? nfsd4_cld_remove+0x54/0x1d0 [nfsd]
>>> [Mon Mar 11 14:10:08 2024] ?
>>> nfsd4_return_all_client_layouts+0xc4/0xf0 [nfsd]
>>> [Mon Mar 11 14:10:08 2024] ? nfsd4_shutdown_copy+0x68/0xc0 [nfsd]
>>> [Mon Mar 11 14:10:08 2024] __destroy_client+0x1f3/0x290 [nfsd]
>>> [Mon Mar 11 14:10:08 2024] nfsd4_exchange_id+0x75f/0x770 [nfsd]
>>> [Mon Mar 11 14:10:08 2024] ? nfsd4_decode_opaque+0x3a/0x90 [nfsd]
>>> [Mon Mar 11 14:10:08 2024] nfsd4_proc_compound+0x44b/0x700 [nfsd]
>>> [Mon Mar 11 14:10:08 2024] nfsd_dispatch+0x94/0x1c0 [nfsd]
>>> [Mon Mar 11 14:10:08 2024] svc_process_common+0x2ec/0x660 [sunrpc]
>>> [Mon Mar 11 14:10:08 2024] ? __pfx_nfsd_dispatch+0x10/0x10 [nfsd]
>>> [Mon Mar 11 14:10:08 2024] ? __pfx_nfsd+0x10/0x10 [nfsd]
>>> [Mon Mar 11 14:10:08 2024] svc_process+0x12d/0x170 [sunrpc]
>>> [Mon Mar 11 14:10:08 2024] nfsd+0x84/0xb0 [nfsd]
>>> [Mon Mar 11 14:10:08 2024] kthread+0xdd/0x100
>>> [Mon Mar 11 14:10:08 2024] ? __pfx_kthread+0x10/0x10
>>> [Mon Mar 11 14:10:08 2024] ret_from_fork+0x29/0x50
>>> [Mon Mar 11 14:10:08 2024] </TASK>
>>> [Mon Mar 11 14:10:08 2024] INFO: task nfsd:8866 blocked for more
>>> than 122 seconds.
>>> [Mon Mar 11 14:10:08 2024] Not tainted 5.14.0-419.el9.x86_64 #1
>>> [Mon Mar 11 14:10:08 2024] "echo 0 >
>>> /proc/sys/kernel/hung_task_timeout_secs" disables this message.
>>> [Mon Mar 11 14:10:08 2024] task:nfsd state:D stack:0
>>> pid:8866 ppid:2 flags:0x00004000
>>> [Mon Mar 11 14:10:08 2024] Call Trace:
>>> [Mon Mar 11 14:10:08 2024] <TASK>
>>> [Mon Mar 11 14:10:08 2024] __schedule+0x21b/0x550
>>> [Mon Mar 11 14:10:08 2024] schedule+0x2d/0x70
>>> [Mon Mar 11 14:10:08 2024] schedule_timeout+0x11f/0x160
>>> [Mon Mar 11 14:10:08 2024] ? select_idle_sibling+0x28/0x430
>>> [Mon Mar 11 14:10:08 2024] ? tcp_recvmsg+0x196/0x210
>>> [Mon Mar 11 14:10:08 2024] ? wake_affine+0x62/0x1f0
>>> [Mon Mar 11 14:10:08 2024] __wait_for_common+0x90/0x1d0
>>> [Mon Mar 11 14:10:08 2024] ? __pfx_schedule_timeout+0x10/0x10
>>> [Mon Mar 11 14:10:08 2024] __flush_workqueue+0x13a/0x3f0
>>> [Mon Mar 11 14:10:08 2024] nfsd4_destroy_session+0x1a4/0x240 [nfsd]
>>> [Mon Mar 11 14:10:08 2024] nfsd4_proc_compound+0x44b/0x700 [nfsd]
>>> [Mon Mar 11 14:10:08 2024] nfsd_dispatch+0x94/0x1c0 [nfsd]
>>> [Mon Mar 11 14:10:08 2024] svc_process_common+0x2ec/0x660 [sunrpc]
>>> [Mon Mar 11 14:10:08 2024] ? __pfx_nfsd_dispatch+0x10/0x10 [nfsd]
>>> [Mon Mar 11 14:10:08 2024] ? __pfx_nfsd+0x10/0x10 [nfsd]
>>> [Mon Mar 11 14:10:08 2024] svc_process+0x12d/0x170 [sunrpc]
>>> [Mon Mar 11 14:10:08 2024] nfsd+0x84/0xb0 [nfsd]
>>> [Mon Mar 11 14:10:08 2024] kthread+0xdd/0x100
>>> [Mon Mar 11 14:10:08 2024] ? __pfx_kthread+0x10/0x10
>>> [Mon Mar 11 14:10:08 2024] ret_from_fork+0x29/0x50
>>> [Mon Mar 11 14:10:08 2024] </TASK>
>>>
>>>
>>>
>>> The above is repeated a few times, and then this warning is also
>>> logged:
>>>
>>>
>>>
>>> [Mon Mar 11 14:12:04 2024] ------------[ cut here ]------------
>>> [Mon Mar 11 14:12:04 2024] WARNING: CPU: 39 PID: 8844 at
>>> fs/nfsd/nfs4state.c:4919 nfsd_break_deleg_cb+0x170/0x190 [nfsd]
>>> [Mon Mar 11 14:12:05 2024] Modules linked in: nfsv4 dns_resolver
>>> nfs fscache netfs rpcsec_gss_krb5 rpcrdma rdma_cm iw_cm ib_cm
>>> ib_core binfmt_misc bonding tls rfkill nft_counter nft_ct
>>> nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_reject_inet
>>> nf_reject_ipv4 nf_reject_ipv6 nft_reject nf_tables nfnetlink vfat
>>> fat dm_thin_pool dm_persistent_data dm_bio_prison dm_bufio l
>>> ibcrc32c dm_service_time dm_multipath intel_rapl_msr
>>> intel_rapl_common intel_uncore_frequency
>>> intel_uncore_frequency_common isst_if_common skx_edac nfit libnvdimm
>>> ipmi_ssif x86_pkg_temp
>>> _thermal intel_powerclamp coretemp kvm_intel kvm irqbypass dcdbas
>>> rapl intel_cstate mgag200 i2c_algo_bit drm_shmem_helper dell_smbios
>>> drm_kms_helper dell_wmi_descriptor wmi_bmof intel_u
>>> ncore syscopyarea pcspkr sysfillrect mei_me sysimgblt acpi_ipmi
>>> mei fb_sys_fops i2c_i801 ipmi_si intel_pch_thermal lpc_ich
>>> ipmi_devintf i2c_smbus ipmi_msghandler joydev acpi_power_meter
>>> nfsd auth_rpcgss nfs_acl drm lockd grace fuse sunrpc ext4 mbcache
>>> jbd2 sd_mod sg lpfc
>>> [Mon Mar 11 14:12:05 2024] nvmet_fc nvmet nvme_fc nvme_fabrics
>>> crct10dif_pclmul ahci libahci crc32_pclmul nvme_core crc32c_intel
>>> ixgbe megaraid_sas libata nvme_common ghash_clmulni_int
>>> el t10_pi wdat_wdt scsi_transport_fc mdio wmi dca dm_mirror
>>> dm_region_hash dm_log dm_mod
>>> [Mon Mar 11 14:12:05 2024] CPU: 39 PID: 8844 Comm: nfsd Not
>>> tainted 5.14.0-419.el9.x86_64 #1
>>> [Mon Mar 11 14:12:05 2024] Hardware name: Dell Inc. PowerEdge
>>> R740/00WGD1, BIOS 2.20.1 09/13/2023
>>> [Mon Mar 11 14:12:05 2024] RIP:
>>> 0010:nfsd_break_deleg_cb+0x170/0x190 [nfsd]
>>> [Mon Mar 11 14:12:05 2024] Code: a6 95 c5 f3 e9 ff fe ff ff 48 89
>>> df be 01 00 00 00 e8 34 b5 13 f4 48 8d bb 98 00 00 00 e8 c8 f9 00 00
>>> 84 c0 0f 85 2e ff ff ff <0f> 0b e9 27 ff ff ff be
>>> 02 00 00 00 48 89 df e8 0c b5 13 f4 e9 01
>>> [Mon Mar 11 14:12:05 2024] RSP: 0018:ffff9929e0bb7b80 EFLAGS:
>>> 00010246
>>> [Mon Mar 11 14:12:05 2024] RAX: 0000000000000000 RBX:
>>> ffff8ada51930900 RCX: 0000000000000024
>>> [Mon Mar 11 14:12:05 2024] RDX: ffff8ada519309c8 RSI:
>>> ffff8ad582933c00 RDI: 0000000000002000
>>> [Mon Mar 11 14:12:05 2024] RBP: ffff8ad46bf21574 R08:
>>> ffff9929e0bb7b48 R09: 0000000000000000
>>> [Mon Mar 11 14:12:05 2024] R10: ffff8aec859a2948 R11:
>>> 0000000000000000 R12: ffff8ad6f497c360
>>> [Mon Mar 11 14:12:05 2024] R13: ffff8ad46bf21560 R14:
>>> ffff8ae5942e0b10 R15: ffff8ad6f497c360
>>> [Mon Mar 11 14:12:05 2024] FS: 0000000000000000(0000)
>>> GS:ffff8b031fcc0000(0000) knlGS:0000000000000000
>>> [Mon Mar 11 14:12:05 2024] CS: 0010 DS: 0000 ES: 0000 CR0:
>>> 0000000080050033
>>> [Mon Mar 11 14:12:05 2024] CR2: 00007fafe2060744 CR3:
>>> 00000018e58de006 CR4: 00000000007706e0
>>> [Mon Mar 11 14:12:05 2024] DR0: 0000000000000000 DR1:
>>> 0000000000000000 DR2: 0000000000000000
>>> [Mon Mar 11 14:12:05 2024] DR3: 0000000000000000 DR6:
>>> 00000000fffe0ff0 DR7: 0000000000000400
>>> [Mon Mar 11 14:12:05 2024] PKRU: 55555554
>>> [Mon Mar 11 14:12:05 2024] Call Trace:
>>> [Mon Mar 11 14:12:05 2024] <TASK>
>>> [Mon Mar 11 14:12:05 2024] ? show_trace_log_lvl+0x1c4/0x2df
>>> [Mon Mar 11 14:12:05 2024] ? show_trace_log_lvl+0x1c4/0x2df
>>> [Mon Mar 11 14:12:05 2024] ? __break_lease+0x16f/0x5f0
>>> [Mon Mar 11 14:12:05 2024] ? nfsd_break_deleg_cb+0x170/0x190 [nfsd]
>>> [Mon Mar 11 14:12:05 2024] ? __warn+0x81/0x110
>>> [Mon Mar 11 14:12:05 2024] ? nfsd_break_deleg_cb+0x170/0x190 [nfsd]
>>> [Mon Mar 11 14:12:05 2024] ? report_bug+0x10a/0x140
>>> [Mon Mar 11 14:12:05 2024] ? handle_bug+0x3c/0x70
>>> [Mon Mar 11 14:12:05 2024] ? exc_invalid_op+0x14/0x70
>>> [Mon Mar 11 14:12:05 2024] ? asm_exc_invalid_op+0x16/0x20
>>> [Mon Mar 11 14:12:05 2024] ? nfsd_break_deleg_cb+0x170/0x190 [nfsd]
>>> [Mon Mar 11 14:12:05 2024] __break_lease+0x16f/0x5f0
>>> [Mon Mar 11 14:12:05 2024] ? nfsd_file_lookup_locked+0x117/0x160
>>> [nfsd]
>>> [Mon Mar 11 14:12:05 2024] ? list_lru_del+0x101/0x150
>>> [Mon Mar 11 14:12:05 2024] nfsd_file_do_acquire+0x790/0x830 [nfsd]
>>> [Mon Mar 11 14:12:05 2024] nfs4_get_vfs_file+0x315/0x3a0 [nfsd]
>>> [Mon Mar 11 14:12:05 2024] nfsd4_process_open2+0x430/0xa30 [nfsd]
>>> [Mon Mar 11 14:12:05 2024] ? fh_verify+0x297/0x2f0 [nfsd]
>>> [Mon Mar 11 14:12:05 2024] nfsd4_open+0x3ce/0x4b0 [nfsd]
>>> [Mon Mar 11 14:12:05 2024] nfsd4_proc_compound+0x44b/0x700 [nfsd]
>>> [Mon Mar 11 14:12:05 2024] nfsd_dispatch+0x94/0x1c0 [nfsd]
>>> [Mon Mar 11 14:12:05 2024] svc_process_common+0x2ec/0x660 [sunrpc]
>>> [Mon Mar 11 14:12:05 2024] ? __pfx_nfsd_dispatch+0x10/0x10 [nfsd]
>>> [Mon Mar 11 14:12:05 2024] ? __pfx_nfsd+0x10/0x10 [nfsd]
>>> [Mon Mar 11 14:12:05 2024] svc_process+0x12d/0x170 [sunrpc]
>>> [Mon Mar 11 14:12:05 2024] nfsd+0x84/0xb0 [nfsd]
>>> [Mon Mar 11 14:12:05 2024] kthread+0xdd/0x100
>>> [Mon Mar 11 14:12:05 2024] ? __pfx_kthread+0x10/0x10
>>> [Mon Mar 11 14:12:05 2024] ret_from_fork+0x29/0x50
>>> [Mon Mar 11 14:12:05 2024] </TASK>
>>> [Mon Mar 11 14:12:05 2024] ---[ end trace 7a039e17443dc651 ]---
>> [Mon Mar 11 14:29:16 2024] task:kworker/u96:3 state:D stack:0
>> pid:2451130 ppid:2 flags:0x00004000
>> [Mon Mar 11 14:29:16 2024] Workqueue: nfsd4_callbacks
>> nfsd4_run_cb_work [nfsd]
>> [Mon Mar 11 14:29:16 2024] Call Trace:
>> [Mon Mar 11 14:29:16 2024] <TASK>
>> [Mon Mar 11 14:29:16 2024] __schedule+0x21b/0x550
>> [Mon Mar 11 14:29:16 2024] schedule+0x2d/0x70
>> [Mon Mar 11 14:29:16 2024] schedule_timeout+0x88/0x160
>> [Mon Mar 11 14:29:16 2024] ? __pfx_process_timeout+0x10/0x10
>> [Mon Mar 11 14:29:16 2024] rpc_shutdown_client+0xb3/0x150 [sunrpc]
>> [Mon Mar 11 14:29:16 2024] ? __pfx_autoremove_wake_function+0x10/0x10
>> [Mon Mar 11 14:29:16 2024] nfsd4_process_cb_update+0x3e/0x260 [nfsd]
>> [Mon Mar 11 14:29:16 2024] ? sched_clock+0xc/0x30
>> [Mon Mar 11 14:29:16 2024] ? raw_spin_rq_lock_nested+0x19/0x80
>> [Mon Mar 11 14:29:16 2024] ? newidle_balance+0x26e/0x400
>> [Mon Mar 11 14:29:16 2024] ? pick_next_task_fair+0x41/0x500
>> [Mon Mar 11 14:29:16 2024] ? put_prev_task_fair+0x1e/0x40
>> [Mon Mar 11 14:29:16 2024] ? pick_next_task+0x861/0x950
>> [Mon Mar 11 14:29:16 2024] ? __update_idle_core+0x23/0xc0
>> [Mon Mar 11 14:29:16 2024] ? __switch_to_asm+0x3a/0x80
>> [Mon Mar 11 14:29:16 2024] ? finish_task_switch.isra.0+0x8c/0x2a0
>> [Mon Mar 11 14:29:16 2024] nfsd4_run_cb_work+0x9f/0x150 [nfsd]
>> [Mon Mar 11 14:29:16 2024] process_one_work+0x1e2/0x3b0
>> [Mon Mar 11 14:29:16 2024] worker_thread+0x50/0x3a0
>> [Mon Mar 11 14:29:16 2024] ? __pfx_worker_thread+0x10/0x10
>> [Mon Mar 11 14:29:16 2024] kthread+0xdd/0x100
>> [Mon Mar 11 14:29:16 2024] ? __pfx_kthread+0x10/0x10
>> [Mon Mar 11 14:29:16 2024] ret_from_fork+0x29/0x50
>> [Mon Mar 11 14:29:16 2024] </TASK>
>>
>> The above is the main task that I see in the cb workqueue. It's
>> trying to call rpc_shutdown_client, which is waiting for this:
>>
>> wait_event_timeout(destroy_wait,
>> list_empty(&clnt->cl_tasks), 1*HZ);
>>
>> ...so basically waiting for the cl_tasks list to go empty. It repeatedly
>> does a rpc_killall_tasks though, so possibly trying to kill this task?
>>
>> 18423 2281 0 0x18 0x0 1354 nfsd4_cb_ops [nfsd]
>> nfs4_cbv1 CB_RECALL_ANY a:call_start [sunrpc] q:delayq
>
> I wonder why this task is on delayq. Could it be related to memory
> shortage issue, or connection related problems?
> Output of /proc/meminfo on the nfs server at time of the problem
> would shed some light.
We don't have that anymore. I can check our monitoring host more closely
for more fine grained stats tomorrow, but when I look at the sar
statistics (see attachment) nothing special was going on memory or
network wise.
We start to get the cpu stall messages and the system load starts to
rise (starts around 2:10 PM). At 3:00 PM we restart the server as our
users can no longer work.
Looking at the stats, the cpu's were ~idle. The only thing that may be
related is that around 11:30 AM the write load (rx packets) starts to
get a lot higher than the read load (tx packets). This goes on for hours
(even after the server was restarted) and that workload was later
identified. It was a workload that was constantly rewriting a statistics
file.
Regards,
Rik
>
> Currently there is only 1 active task allowed for the nfsd callback
> workqueue at a time. If for some reasons a callback task is stuck in
> the workqueue it will block all other callback tasks which can effect
> multiple clients.
>
> -Dai
>
>>
>> Callbacks are soft RPC tasks though, so they should be easily killable.
--
Rik Theys
System Engineer
KU Leuven - Dept. Elektrotechniek (ESAT)
Kasteelpark Arenberg 10 bus 2440 - B-3001 Leuven-Heverlee
+32(0)16/32.11.07
----------------------------------------------------------------
<<Any errors in spelling, tact or fact are transmission errors>>
[-- Attachment #2: sarstats.txt --]
[-- Type: text/plain, Size: 16817 bytes --]
11:20:00 AM runq-sz plist-sz ldavg-1 ldavg-5 ldavg-15 blocked
11:30:00 AM 2 1198 3.63 3.33 2.77 1
11:40:01 AM 4 1197 3.29 3.87 3.36 3
11:50:00 AM 0 1203 2.34 3.32 3.50 0
12:00:00 PM 3 1184 2.31 2.64 3.06 3
12:10:00 PM 4 1192 5.66 4.29 3.58 1
12:20:00 PM 1 1197 7.46 4.69 3.86 1
12:30:00 PM 2 1200 3.65 3.87 3.83 0
12:40:00 PM 4 1209 3.15 3.57 3.77 1
12:50:00 PM 0 1204 5.15 4.33 3.95 1
01:00:00 PM 6 1192 3.26 3.57 3.78 1
01:10:00 PM 3 1215 3.50 3.12 3.41 0
01:20:00 PM 3 1196 3.15 3.11 3.27 5
01:30:00 PM 4 1194 4.13 3.71 3.52 0
01:40:00 PM 1 1200 2.97 2.99 3.19 0
01:50:00 PM 1 1210 3.55 3.54 3.44 0
02:00:00 PM 2 1220 4.61 4.56 3.93 0
02:10:00 PM 1 1229 16.94 9.96 6.44 0
02:20:00 PM 1 1208 57.03 45.47 26.17 0
02:30:00 PM 3 1218 66.46 58.40 41.85 0
02:40:00 PM 2 1218 71.43 67.15 54.03 1
02:50:00 PM 4 1222 87.71 81.15 67.21 1
03:00:00 PM 5 1228 98.74 93.68 80.13 2
Average: 2 1120 6.00 5.46 4.53 0
03:11:33 PM LINUX RESTART (48 CPU)
11:20:00 AM CPU %user %nice %system %iowait %steal %idle
11:20:00 AM all 0.59 0.00 2.61 0.48 0.00 96.31
11:30:00 AM all 0.55 0.00 3.78 3.05 0.00 92.62
11:40:01 AM all 0.52 0.00 4.48 2.71 0.00 92.29
11:50:00 AM all 0.51 0.00 4.54 2.91 0.00 92.04
12:00:00 PM all 0.52 0.00 4.17 1.13 0.00 94.18
12:10:00 PM all 0.59 0.00 4.51 2.10 0.00 92.79
12:20:00 PM all 0.54 0.00 4.67 0.87 0.00 93.92
12:30:00 PM all 0.50 0.00 5.40 1.61 0.00 92.50
12:40:00 PM all 0.50 0.00 6.04 1.27 0.00 92.20
12:50:00 PM all 0.53 0.00 5.43 1.39 0.00 92.65
01:00:00 PM all 0.61 0.00 5.56 1.90 0.00 91.93
01:10:00 PM all 0.60 0.00 5.38 0.91 0.00 93.11
01:20:00 PM all 0.59 0.00 5.69 1.12 0.00 92.59
01:30:00 PM all 0.60 0.00 6.06 1.30 0.00 92.04
01:40:00 PM all 0.57 0.00 5.15 1.32 0.00 92.95
01:50:00 PM all 0.62 0.00 5.58 1.86 0.00 91.94
02:00:00 PM all 0.62 0.00 6.01 2.45 0.00 90.93
02:10:00 PM all 0.65 0.00 9.16 2.95 0.00 87.23
02:20:00 PM all 0.62 0.00 9.68 4.55 0.00 85.15
02:30:00 PM all 0.68 0.00 7.05 2.24 0.00 90.03
02:40:00 PM all 1.00 0.00 5.92 2.19 0.00 90.89
02:50:00 PM all 2.00 0.00 6.49 3.23 0.00 88.27
03:00:00 PM all 1.37 0.00 5.90 3.01 0.00 89.72
Average: all 0.32 0.00 2.84 0.81 0.00 96.03
03:11:33 PM LINUX RESTART (48 CPU)
11:20:00 AM pgpgin/s pgpgout/s fault/s majflt/s pgfree/s pgscank/s pgscand/s pgsteal/s %vmeff
11:20:00 AM 4020.11 8605.06 2157.62 0.00 191933.03 1217.98 0.00 2442.82 200.56
11:30:00 AM 55268.90 50540.80 2096.02 0.04 637897.57 17648.39 0.00 35296.47 200.00
11:40:01 AM 79943.11 116231.90 1859.00 0.00 760848.68 23008.54 0.00 46020.39 200.01
11:50:00 AM 22594.29 173959.91 1676.84 0.00 1084371.55 14182.57 0.00 28364.99 200.00
12:00:00 PM 2769.56 164286.91 1450.51 0.00 896969.12 1416.75 0.00 2836.86 200.24
12:10:00 PM 8822.01 155296.22 1431.25 0.00 931734.21 2687.33 0.00 5378.07 200.13
12:20:00 PM 2577.91 178625.24 1379.77 0.00 1180545.30 3497.12 0.00 6994.22 200.00
12:30:00 PM 5954.85 226744.21 1253.30 0.00 1918863.89 7147.24 0.00 14297.88 200.05
12:40:00 PM 2377.24 289116.52 1194.98 0.00 2580662.81 6756.25 0.00 13512.50 200.00
12:50:00 PM 1939.94 270595.12 1579.08 0.00 2189766.74 2785.89 0.00 5571.77 200.00
01:00:00 PM 3975.65 280647.85 1960.25 0.00 2421730.66 2904.24 0.00 5808.49 200.00
01:10:00 PM 3061.46 285614.89 2077.81 0.00 2256171.33 2655.80 0.00 5311.57 200.00
01:20:00 PM 1522.98 289014.26 1878.51 0.00 2514450.25 1213.70 0.00 2427.40 200.00
01:30:00 PM 4979.56 272158.71 1622.52 0.00 2252781.20 2183.79 0.00 4367.58 200.00
01:40:00 PM 4364.82 266270.12 1411.46 0.00 2013371.12 2164.64 0.00 4329.28 200.00
01:50:00 PM 9045.98 274397.77 1958.44 0.01 1970749.12 3996.12 0.00 7995.65 200.09
02:00:00 PM 6472.46 280631.95 1453.42 0.00 2098466.10 3921.65 0.00 7843.27 200.00
02:10:00 PM 2577.67 491251.09 2247.01 0.00 2415029.61 75131.80 0.00 150276.28 200.02
02:20:00 PM 8279.88 523464.79 2216.22 0.00 2554580.80 85277.68 0.00 170554.78 200.00
02:30:00 PM 4016.00 378426.50 2704.81 0.32 2155706.27 36195.48 0.00 72400.96 200.03
02:40:00 PM 10939.12 268965.25 2279.79 0.01 1920982.26 4331.20 0.00 8676.01 200.31
02:50:00 PM 35906.38 286698.69 1951.03 0.00 1994827.72 13385.01 0.00 26776.99 200.05
03:00:00 PM 30307.83 284038.19 1873.97 0.02 2041174.51 12174.66 0.00 24352.69 200.03
Average: 11214.03 84754.74 1052.37 0.01 641224.50 7068.27 0.06 14139.06 200.03
03:11:33 PM LINUX RESTART (48 CPU)
11:20:00 AM kbmemfree kbavail kbmemused %memused kbbuffers kbcached kbcommit %commit kbactive kbinact kbdirty kbanonpg kbslab kbkstack kbpgtbl kbvmused
11:20:00 AM 1865420 171187348 1998368 1.02 14285620 138954960 4464588 1.70 1157788 152702956 163464 672516 39006492 19600 46112 305960
11:30:00 AM 3687156 171152312 2037216 1.04 13766560 138071112 4532556 1.72 1209572 151275684 483556 700220 38548816 19904 48504 306200
11:40:01 AM 1766552 170336640 2143452 1.09 12970608 141160884 4478464 1.70 1171124 153592108 542440 683992 38069364 19888 47428 306200
11:50:00 AM 7281384 171091800 2096236 1.07 12565028 136191740 4527272 1.72 1185148 148217324 57744 697772 37976472 20000 48268 306216
12:00:00 PM 8276692 171026364 2151216 1.10 12754360 134945564 4521328 1.72 1175520 147155948 1041800 689424 37983028 19664 48056 305800
12:10:00 PM 2571948 171107708 2052636 1.05 12886184 140612248 4546576 1.73 1182792 152958724 475408 700172 37987844 19808 48144 306072
12:20:00 PM 4397388 171101628 2036452 1.04 12975352 138459220 4539876 1.72 1171936 150894616 796324 687384 38242448 19904 47496 306104
12:30:00 PM 7087352 170716108 2133384 1.09 13021692 135325824 4586040 1.74 1178288 147807308 1722952 693240 38542608 19968 48404 306216
12:40:00 PM 7524128 170954416 2165520 1.10 13097896 134570932 4613396 1.75 1189344 147128772 701080 704404 38752384 20144 49080 306312
12:50:00 PM 6051508 170910668 2199872 1.12 13263460 135803304 4612484 1.75 1186232 148528800 373732 705916 38792716 20000 49068 306104
01:00:00 PM 10992332 170726464 2138820 1.09 13288164 130804672 4624612 1.76 1177744 143554556 2956808 697308 38886872 19824 49076 306200
01:10:00 PM 3782976 170738340 2117004 1.08 13314832 137872584 4641200 1.76 1193912 150642700 404072 704116 39023464 20160 50028 306408
01:20:00 PM 6659444 170900236 2167088 1.11 13376480 134926060 4653100 1.77 1184556 147757720 388432 697876 38981788 19888 49888 306248
01:30:00 PM 13172768 170885616 2172216 1.11 13393784 128373088 4653708 1.77 1186008 141224736 4071760 700964 38999004 19840 49880 306120
01:40:00 PM 10682972 170893288 2144464 1.09 13517676 130723148 4701116 1.79 1193700 143698488 1415012 706440 39042600 19952 50844 306296
01:50:00 PM 6085404 170824968 2200596 1.12 13221836 135528552 4736532 1.80 1201632 148203920 2410612 712088 39074472 20176 51992 306344
02:00:00 PM 5555448 170810072 2201640 1.12 13658348 135470552 4758996 1.81 1223860 148582888 307464 731856 39224872 20192 53144 306456
02:10:00 PM 1014880 170836368 2160028 1.10 12728708 140151288 4823616 1.83 1227072 152329884 9645304 731748 40055956 20432 53516 306664
02:20:00 PM 2823068 170828288 2152108 1.10 13199316 137843512 4801392 1.82 1217876 150492036 8488084 721812 40092856 20064 53224 306408
02:30:00 PM 7005404 170678580 2261792 1.15 13358768 133429936 4893344 1.86 1271688 146226924 1625480 752752 40054960 20224 56408 306456
02:40:00 PM 4016676 170697044 2232824 1.14 13727020 136140404 4911972 1.87 1258276 149302784 167584 740892 39993936 20240 54472 306580
02:50:00 PM 9976392 170683428 2192384 1.12 13706924 130450508 4931124 1.87 1261640 143607304 1520456 760304 39784652 20304 55912 304648
03:00:00 PM 8165036 170637924 2235228 1.14 13969932 132128568 4956008 1.88 1249376 145549216 1503472 751636 39612096 20384 55216 304588
Average: 8003274 171354427 1997084 1.02 17978233 128153402 4009864 1.52 2948732 143744234 802698 618928 39978867 18658 35078 303511
03:11:33 PM LINUX RESTART (48 CPU)
12:00:01 AM IFACE rxpck/s txpck/s rxkB/s txkB/s rxcmp/s txcmp/s rxmcst/s %ifutil
10:20:01 AM bond0 19341.55 16613.52 14852.80 10063.03 0.00 0.00 7.59 0.61
10:30:00 AM bond0 22612.77 23802.93 17064.32 18021.10 0.00 0.00 7.54 0.74
10:40:00 AM bond0 36072.84 28650.68 36436.66 23838.56 0.00 0.00 7.54 1.49
10:50:00 AM bond0 30297.97 26476.98 28896.10 21851.80 0.00 0.00 7.59 1.18
11:00:00 AM bond0 24422.77 22650.70 17771.64 14655.68 0.00 0.00 7.57 0.73
11:10:00 AM bond0 32648.61 55389.48 27350.30 63200.39 0.00 0.00 7.61 2.59
11:20:00 AM bond0 20966.71 20631.12 15049.13 14239.36 0.00 0.00 7.59 0.62
11:30:00 AM bond0 59038.60 85522.45 62065.03 104016.82 0.00 0.00 7.54 4.26
11:40:01 AM bond0 109991.82 97051.54 129799.48 117008.01 0.00 0.00 7.54 5.32
11:50:00 AM bond0 142884.43 45306.77 188287.91 34672.37 0.00 0.00 7.60 7.71
12:00:00 PM bond0 135270.64 29202.31 180105.47 15043.09 0.00 0.00 7.54 7.38
12:10:00 PM bond0 128682.92 38263.76 167859.64 26517.80 0.00 0.00 7.65 6.88
12:20:00 PM bond0 147128.05 41190.26 195974.49 28419.00 0.00 0.00 7.59 8.03
12:30:00 PM bond0 183452.21 59540.64 248497.44 41698.82 0.00 0.00 7.54 10.18
12:40:00 PM bond0 223045.51 49631.56 308994.74 20892.16 0.00 0.00 7.56 12.66
12:50:00 PM bond0 211356.42 40242.54 292794.32 15753.38 0.00 0.00 7.55 11.99
01:00:00 PM bond0 221345.60 43010.18 307861.64 17748.64 0.00 0.00 7.55 12.61
01:10:00 PM bond0 223147.26 40941.92 311071.09 17092.65 0.00 0.00 7.58 12.74
01:20:00 PM bond0 226732.82 43452.27 315370.79 16082.25 0.00 0.00 7.61 12.92
01:30:00 PM bond0 222725.38 72765.59 303529.78 59891.91 0.00 0.00 7.52 12.43
01:40:00 PM bond0 206342.36 40977.06 286164.94 20165.04 0.00 0.00 7.54 11.72
01:50:00 PM bond0 218327.39 49942.46 300638.56 30757.69 0.00 0.00 7.60 12.31
02:00:00 PM bond0 218851.87 54196.47 299367.98 33044.98 0.00 0.00 7.57 12.26
02:10:00 PM bond0 378807.79 53458.22 538611.10 19944.21 0.00 0.00 7.58 22.06
02:20:00 PM bond0 387285.99 55460.02 555147.68 22975.23 0.00 0.00 7.58 22.74
02:30:00 PM bond0 274593.98 37885.29 390578.19 11658.62 0.00 0.00 7.56 16.00
02:40:00 PM bond0 202721.14 41980.65 282541.86 19462.85 0.00 0.00 7.56 11.57
02:50:00 PM bond0 218072.41 52718.98 301602.85 36963.31 0.00 0.00 7.59 12.35
03:00:00 PM bond0 214600.08 51698.15 299890.67 36181.66 0.00 0.00 7.58 12.28
Average: bond0 70892.56 27494.51 89394.02 19260.67 0.00 0.00 7.57 3.66
03:11:33 PM LINUX RESTART (48 CPU)
12:00:01 AM IFACE rxerr/s txerr/s coll/s rxdrop/s txdrop/s txcarr/s rxfram/s rxfifo/s txfifo/s
11:20:00 AM bond0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
11:30:00 AM bond0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
11:40:01 AM bond0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
11:50:00 AM bond0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
12:00:00 PM bond0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
12:10:00 PM bond0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
12:20:00 PM bond0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
12:30:00 PM bond0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
12:40:00 PM bond0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
12:50:00 PM bond0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
01:00:00 PM bond0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
01:10:00 PM bond0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
01:20:00 PM bond0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
01:30:00 PM bond0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
01:40:00 PM bond0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
01:50:00 PM bond0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
02:00:00 PM bond0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
02:10:00 PM bond0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
02:20:00 PM bond0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
02:30:00 PM bond0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
02:40:00 PM bond0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
02:50:00 PM bond0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
03:00:00 PM bond0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
Average: bond0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
03:11:33 PM LINUX RESTART (48 CPU)
^ permalink raw reply [flat|nested] 26+ messages in thread
* Re: nfsd hangs and nfsd_break_deleg_cb+0x170/0x190 warning
2024-03-12 11:37 ` Jeff Layton
@ 2024-03-12 16:43 ` Dai Ngo
2024-03-12 18:23 ` Rik Theys
0 siblings, 1 reply; 26+ messages in thread
From: Dai Ngo @ 2024-03-12 16:43 UTC (permalink / raw)
To: Jeff Layton, Rik Theys, Linux Nfs
On 3/12/24 4:37 AM, Jeff Layton wrote:
> On Mon, 2024-03-11 at 19:43 +0100, Rik Theys wrote:
>>
>>
>>
>>
>> Hi,
>>
>>
>>
>>
>> Since a few weeks our Rocky Linux 9 NFS server has periodically logged hung nfsd tasks. The initial effect was that some clients could no longer access the NFS server. This got worse and worse (probably as more nfsd threads got blocked) and we had to restart the server. Restarting the server also failed as the NFS server service could no longer be stopped.
>>
>>
>>
>>
>> The initial kernel we noticed this behavior on was kernel-5.14.0-362.18.1.el9_3.x86_64. Since then we've installed kernel-5.14.0-419.el9.x86_64 from CentOS Stream 9. The same issue happened again on this newer kernel version:
>>
>>
>>
>>
>> [Mon Mar 11 14:10:08 2024] Not tainted 5.14.0-419.el9.x86_64 #1
>> [Mon Mar 11 14:10:08 2024] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
>> [Mon Mar 11 14:10:08 2024] task:nfsd state:D stack:0 pid:8865 ppid:2 flags:0x00004000
>> [Mon Mar 11 14:10:08 2024] Call Trace:
>> [Mon Mar 11 14:10:08 2024] <TASK>
>> [Mon Mar 11 14:10:08 2024] __schedule+0x21b/0x550
>> [Mon Mar 11 14:10:08 2024] schedule+0x2d/0x70
>> [Mon Mar 11 14:10:08 2024] schedule_timeout+0x11f/0x160
>> [Mon Mar 11 14:10:08 2024] ? select_idle_sibling+0x28/0x430
>> [Mon Mar 11 14:10:08 2024] ? wake_affine+0x62/0x1f0
>> [Mon Mar 11 14:10:08 2024] __wait_for_common+0x90/0x1d0
>> [Mon Mar 11 14:10:08 2024] ? __pfx_schedule_timeout+0x10/0x10
>> [Mon Mar 11 14:10:08 2024] __flush_workqueue+0x13a/0x3f0
>> [Mon Mar 11 14:10:08 2024] nfsd4_shutdown_callback+0x49/0x120 [nfsd]
>> [Mon Mar 11 14:10:08 2024] ? nfsd4_cld_remove+0x54/0x1d0 [nfsd]
>> [Mon Mar 11 14:10:08 2024] ? nfsd4_return_all_client_layouts+0xc4/0xf0 [nfsd]
>> [Mon Mar 11 14:10:08 2024] ? nfsd4_shutdown_copy+0x68/0xc0 [nfsd]
>> [Mon Mar 11 14:10:08 2024] __destroy_client+0x1f3/0x290 [nfsd]
>> [Mon Mar 11 14:10:08 2024] nfsd4_exchange_id+0x75f/0x770 [nfsd]
>> [Mon Mar 11 14:10:08 2024] ? nfsd4_decode_opaque+0x3a/0x90 [nfsd]
>> [Mon Mar 11 14:10:08 2024] nfsd4_proc_compound+0x44b/0x700 [nfsd]
>> [Mon Mar 11 14:10:08 2024] nfsd_dispatch+0x94/0x1c0 [nfsd]
>> [Mon Mar 11 14:10:08 2024] svc_process_common+0x2ec/0x660 [sunrpc]
>> [Mon Mar 11 14:10:08 2024] ? __pfx_nfsd_dispatch+0x10/0x10 [nfsd]
>> [Mon Mar 11 14:10:08 2024] ? __pfx_nfsd+0x10/0x10 [nfsd]
>> [Mon Mar 11 14:10:08 2024] svc_process+0x12d/0x170 [sunrpc]
>> [Mon Mar 11 14:10:08 2024] nfsd+0x84/0xb0 [nfsd]
>> [Mon Mar 11 14:10:08 2024] kthread+0xdd/0x100
>> [Mon Mar 11 14:10:08 2024] ? __pfx_kthread+0x10/0x10
>> [Mon Mar 11 14:10:08 2024] ret_from_fork+0x29/0x50
>> [Mon Mar 11 14:10:08 2024] </TASK>
>> [Mon Mar 11 14:10:08 2024] INFO: task nfsd:8866 blocked for more than 122 seconds.
>> [Mon Mar 11 14:10:08 2024] Not tainted 5.14.0-419.el9.x86_64 #1
>> [Mon Mar 11 14:10:08 2024] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
>> [Mon Mar 11 14:10:08 2024] task:nfsd state:D stack:0 pid:8866 ppid:2 flags:0x00004000
>> [Mon Mar 11 14:10:08 2024] Call Trace:
>> [Mon Mar 11 14:10:08 2024] <TASK>
>> [Mon Mar 11 14:10:08 2024] __schedule+0x21b/0x550
>> [Mon Mar 11 14:10:08 2024] schedule+0x2d/0x70
>> [Mon Mar 11 14:10:08 2024] schedule_timeout+0x11f/0x160
>> [Mon Mar 11 14:10:08 2024] ? select_idle_sibling+0x28/0x430
>> [Mon Mar 11 14:10:08 2024] ? tcp_recvmsg+0x196/0x210
>> [Mon Mar 11 14:10:08 2024] ? wake_affine+0x62/0x1f0
>> [Mon Mar 11 14:10:08 2024] __wait_for_common+0x90/0x1d0
>> [Mon Mar 11 14:10:08 2024] ? __pfx_schedule_timeout+0x10/0x10
>> [Mon Mar 11 14:10:08 2024] __flush_workqueue+0x13a/0x3f0
>> [Mon Mar 11 14:10:08 2024] nfsd4_destroy_session+0x1a4/0x240 [nfsd]
>> [Mon Mar 11 14:10:08 2024] nfsd4_proc_compound+0x44b/0x700 [nfsd]
>> [Mon Mar 11 14:10:08 2024] nfsd_dispatch+0x94/0x1c0 [nfsd]
>> [Mon Mar 11 14:10:08 2024] svc_process_common+0x2ec/0x660 [sunrpc]
>> [Mon Mar 11 14:10:08 2024] ? __pfx_nfsd_dispatch+0x10/0x10 [nfsd]
>> [Mon Mar 11 14:10:08 2024] ? __pfx_nfsd+0x10/0x10 [nfsd]
>> [Mon Mar 11 14:10:08 2024] svc_process+0x12d/0x170 [sunrpc]
>> [Mon Mar 11 14:10:08 2024] nfsd+0x84/0xb0 [nfsd]
>> [Mon Mar 11 14:10:08 2024] kthread+0xdd/0x100
>> [Mon Mar 11 14:10:08 2024] ? __pfx_kthread+0x10/0x10
>> [Mon Mar 11 14:10:08 2024] ret_from_fork+0x29/0x50
>> [Mon Mar 11 14:10:08 2024] </TASK>
>>
>>
>>
>>
>> The above is repeated a few times, and then this warning is also logged:
>>
>>
>>
>>
>> [Mon Mar 11 14:12:04 2024] ------------[ cut here ]------------
>> [Mon Mar 11 14:12:04 2024] WARNING: CPU: 39 PID: 8844 at fs/nfsd/nfs4state.c:4919 nfsd_break_deleg_cb+0x170/0x190 [nfsd]
>> [Mon Mar 11 14:12:05 2024] Modules linked in: nfsv4 dns_resolver nfs fscache netfs rpcsec_gss_krb5 rpcrdma rdma_cm iw_cm ib_cm ib_core binfmt_misc bonding tls rfkill nft_counter nft_ct
>> nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_reject_inet nf_reject_ipv4 nf_reject_ipv6 nft_reject nf_tables nfnetlink vfat fat dm_thin_pool dm_persistent_data dm_bio_prison dm_bufio l
>> ibcrc32c dm_service_time dm_multipath intel_rapl_msr intel_rapl_common intel_uncore_frequency intel_uncore_frequency_common isst_if_common skx_edac nfit libnvdimm ipmi_ssif x86_pkg_temp
>> _thermal intel_powerclamp coretemp kvm_intel kvm irqbypass dcdbas rapl intel_cstate mgag200 i2c_algo_bit drm_shmem_helper dell_smbios drm_kms_helper dell_wmi_descriptor wmi_bmof intel_u
>> ncore syscopyarea pcspkr sysfillrect mei_me sysimgblt acpi_ipmi mei fb_sys_fops i2c_i801 ipmi_si intel_pch_thermal lpc_ich ipmi_devintf i2c_smbus ipmi_msghandler joydev acpi_power_meter
>> nfsd auth_rpcgss nfs_acl drm lockd grace fuse sunrpc ext4 mbcache jbd2 sd_mod sg lpfc
>> [Mon Mar 11 14:12:05 2024] nvmet_fc nvmet nvme_fc nvme_fabrics crct10dif_pclmul ahci libahci crc32_pclmul nvme_core crc32c_intel ixgbe megaraid_sas libata nvme_common ghash_clmulni_int
>> el t10_pi wdat_wdt scsi_transport_fc mdio wmi dca dm_mirror dm_region_hash dm_log dm_mod
>> [Mon Mar 11 14:12:05 2024] CPU: 39 PID: 8844 Comm: nfsd Not tainted 5.14.0-419.el9.x86_64 #1
>> [Mon Mar 11 14:12:05 2024] Hardware name: Dell Inc. PowerEdge R740/00WGD1, BIOS 2.20.1 09/13/2023
>> [Mon Mar 11 14:12:05 2024] RIP: 0010:nfsd_break_deleg_cb+0x170/0x190 [nfsd]
>> [Mon Mar 11 14:12:05 2024] Code: a6 95 c5 f3 e9 ff fe ff ff 48 89 df be 01 00 00 00 e8 34 b5 13 f4 48 8d bb 98 00 00 00 e8 c8 f9 00 00 84 c0 0f 85 2e ff ff ff <0f> 0b e9 27 ff ff ff be
>> 02 00 00 00 48 89 df e8 0c b5 13 f4 e9 01
>> [Mon Mar 11 14:12:05 2024] RSP: 0018:ffff9929e0bb7b80 EFLAGS: 00010246
>> [Mon Mar 11 14:12:05 2024] RAX: 0000000000000000 RBX: ffff8ada51930900 RCX: 0000000000000024
>> [Mon Mar 11 14:12:05 2024] RDX: ffff8ada519309c8 RSI: ffff8ad582933c00 RDI: 0000000000002000
>> [Mon Mar 11 14:12:05 2024] RBP: ffff8ad46bf21574 R08: ffff9929e0bb7b48 R09: 0000000000000000
>> [Mon Mar 11 14:12:05 2024] R10: ffff8aec859a2948 R11: 0000000000000000 R12: ffff8ad6f497c360
>> [Mon Mar 11 14:12:05 2024] R13: ffff8ad46bf21560 R14: ffff8ae5942e0b10 R15: ffff8ad6f497c360
>> [Mon Mar 11 14:12:05 2024] FS: 0000000000000000(0000) GS:ffff8b031fcc0000(0000) knlGS:0000000000000000
>> [Mon Mar 11 14:12:05 2024] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
>> [Mon Mar 11 14:12:05 2024] CR2: 00007fafe2060744 CR3: 00000018e58de006 CR4: 00000000007706e0
>> [Mon Mar 11 14:12:05 2024] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
>> [Mon Mar 11 14:12:05 2024] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
>> [Mon Mar 11 14:12:05 2024] PKRU: 55555554
>> [Mon Mar 11 14:12:05 2024] Call Trace:
>> [Mon Mar 11 14:12:05 2024] <TASK>
>> [Mon Mar 11 14:12:05 2024] ? show_trace_log_lvl+0x1c4/0x2df
>> [Mon Mar 11 14:12:05 2024] ? show_trace_log_lvl+0x1c4/0x2df
>> [Mon Mar 11 14:12:05 2024] ? __break_lease+0x16f/0x5f0
>> [Mon Mar 11 14:12:05 2024] ? nfsd_break_deleg_cb+0x170/0x190 [nfsd]
>> [Mon Mar 11 14:12:05 2024] ? __warn+0x81/0x110
>> [Mon Mar 11 14:12:05 2024] ? nfsd_break_deleg_cb+0x170/0x190 [nfsd]
>> [Mon Mar 11 14:12:05 2024] ? report_bug+0x10a/0x140
>> [Mon Mar 11 14:12:05 2024] ? handle_bug+0x3c/0x70
>> [Mon Mar 11 14:12:05 2024] ? exc_invalid_op+0x14/0x70
>> [Mon Mar 11 14:12:05 2024] ? asm_exc_invalid_op+0x16/0x20
>> [Mon Mar 11 14:12:05 2024] ? nfsd_break_deleg_cb+0x170/0x190 [nfsd]
>> [Mon Mar 11 14:12:05 2024] __break_lease+0x16f/0x5f0
>> [Mon Mar 11 14:12:05 2024] ? nfsd_file_lookup_locked+0x117/0x160 [nfsd]
>> [Mon Mar 11 14:12:05 2024] ? list_lru_del+0x101/0x150
>> [Mon Mar 11 14:12:05 2024] nfsd_file_do_acquire+0x790/0x830 [nfsd]
>> [Mon Mar 11 14:12:05 2024] nfs4_get_vfs_file+0x315/0x3a0 [nfsd]
>> [Mon Mar 11 14:12:05 2024] nfsd4_process_open2+0x430/0xa30 [nfsd]
>> [Mon Mar 11 14:12:05 2024] ? fh_verify+0x297/0x2f0 [nfsd]
>> [Mon Mar 11 14:12:05 2024] nfsd4_open+0x3ce/0x4b0 [nfsd]
>> [Mon Mar 11 14:12:05 2024] nfsd4_proc_compound+0x44b/0x700 [nfsd]
>> [Mon Mar 11 14:12:05 2024] nfsd_dispatch+0x94/0x1c0 [nfsd]
>> [Mon Mar 11 14:12:05 2024] svc_process_common+0x2ec/0x660 [sunrpc]
>> [Mon Mar 11 14:12:05 2024] ? __pfx_nfsd_dispatch+0x10/0x10 [nfsd]
>> [Mon Mar 11 14:12:05 2024] ? __pfx_nfsd+0x10/0x10 [nfsd]
>> [Mon Mar 11 14:12:05 2024] svc_process+0x12d/0x170 [sunrpc]
>> [Mon Mar 11 14:12:05 2024] nfsd+0x84/0xb0 [nfsd]
>> [Mon Mar 11 14:12:05 2024] kthread+0xdd/0x100
>> [Mon Mar 11 14:12:05 2024] ? __pfx_kthread+0x10/0x10
>> [Mon Mar 11 14:12:05 2024] ret_from_fork+0x29/0x50
>> [Mon Mar 11 14:12:05 2024] </TASK>
>> [Mon Mar 11 14:12:05 2024] ---[ end trace 7a039e17443dc651 ]---
>>
> [Mon Mar 11 14:29:16 2024] task:kworker/u96:3 state:D stack:0 pid:2451130 ppid:2 flags:0x00004000
> [Mon Mar 11 14:29:16 2024] Workqueue: nfsd4_callbacks nfsd4_run_cb_work [nfsd]
> [Mon Mar 11 14:29:16 2024] Call Trace:
> [Mon Mar 11 14:29:16 2024] <TASK>
> [Mon Mar 11 14:29:16 2024] __schedule+0x21b/0x550
> [Mon Mar 11 14:29:16 2024] schedule+0x2d/0x70
> [Mon Mar 11 14:29:16 2024] schedule_timeout+0x88/0x160
> [Mon Mar 11 14:29:16 2024] ? __pfx_process_timeout+0x10/0x10
> [Mon Mar 11 14:29:16 2024] rpc_shutdown_client+0xb3/0x150 [sunrpc]
> [Mon Mar 11 14:29:16 2024] ? __pfx_autoremove_wake_function+0x10/0x10
> [Mon Mar 11 14:29:16 2024] nfsd4_process_cb_update+0x3e/0x260 [nfsd]
> [Mon Mar 11 14:29:16 2024] ? sched_clock+0xc/0x30
> [Mon Mar 11 14:29:16 2024] ? raw_spin_rq_lock_nested+0x19/0x80
> [Mon Mar 11 14:29:16 2024] ? newidle_balance+0x26e/0x400
> [Mon Mar 11 14:29:16 2024] ? pick_next_task_fair+0x41/0x500
> [Mon Mar 11 14:29:16 2024] ? put_prev_task_fair+0x1e/0x40
> [Mon Mar 11 14:29:16 2024] ? pick_next_task+0x861/0x950
> [Mon Mar 11 14:29:16 2024] ? __update_idle_core+0x23/0xc0
> [Mon Mar 11 14:29:16 2024] ? __switch_to_asm+0x3a/0x80
> [Mon Mar 11 14:29:16 2024] ? finish_task_switch.isra.0+0x8c/0x2a0
> [Mon Mar 11 14:29:16 2024] nfsd4_run_cb_work+0x9f/0x150 [nfsd]
> [Mon Mar 11 14:29:16 2024] process_one_work+0x1e2/0x3b0
> [Mon Mar 11 14:29:16 2024] worker_thread+0x50/0x3a0
> [Mon Mar 11 14:29:16 2024] ? __pfx_worker_thread+0x10/0x10
> [Mon Mar 11 14:29:16 2024] kthread+0xdd/0x100
> [Mon Mar 11 14:29:16 2024] ? __pfx_kthread+0x10/0x10
> [Mon Mar 11 14:29:16 2024] ret_from_fork+0x29/0x50
> [Mon Mar 11 14:29:16 2024] </TASK>
>
> The above is the main task that I see in the cb workqueue. It's trying to call rpc_shutdown_client, which is waiting for this:
>
> wait_event_timeout(destroy_wait,
> list_empty(&clnt->cl_tasks), 1*HZ);
>
> ...so basically waiting for the cl_tasks list to go empty. It repeatedly
> does a rpc_killall_tasks though, so possibly trying to kill this task?
>
> 18423 2281 0 0x18 0x0 1354 nfsd4_cb_ops [nfsd] nfs4_cbv1 CB_RECALL_ANY a:call_start [sunrpc] q:delayq
I wonder why this task is on delayq. Could it be related to memory
shortage issue, or connection related problems?
Output of /proc/meminfo on the nfs server at time of the problem
would shed some light.
Currently there is only 1 active task allowed for the nfsd callback
workqueue at a time. If for some reasons a callback task is stuck in
the workqueue it will block all other callback tasks which can effect
multiple clients.
-Dai
>
> Callbacks are soft RPC tasks though, so they should be easily killable.
^ permalink raw reply [flat|nested] 26+ messages in thread
* Re: nfsd hangs and nfsd_break_deleg_cb+0x170/0x190 warning
@ 2024-03-12 13:30 Rik Theys
0 siblings, 0 replies; 26+ messages in thread
From: Rik Theys @ 2024-03-12 13:30 UTC (permalink / raw)
To: Jeff Layton, Linux Nfs
Hi,
On 3/12/24 13:47, Jeff Layton wrote:
> On Tue, 2024-03-12 at 13:24 +0100, Rik Theys wrote:
>> On 3/12/24 12:22, Jeff Layton wrote:
>>> On Mon, 2024-03-11 at 19:43 +0100, Rik Theys wrote:
>>>> Since a few weeks our Rocky Linux 9 NFS server has periodically logged hung nfsd tasks. The initial effect was that some clients could no longer access the NFS server. This got worse and worse (probably as more nfsd threads got blocked) and we had to restart the server. Restarting the server also failed as the NFS server service could no longer be stopped.
>>>>
>>>>
>>>> The initial kernel we noticed this behavior on was kernel-5.14.0-362.18.1.el9_3.x86_64. Since then we've installed kernel-5.14.0-419.el9.x86_64 from CentOS Stream 9. The same issue happened again on this newer kernel version:
>>>>
> 419 is fairly up to date with nfsd changes. There are some known bugs
> around callbacks, and there is a draft MR in flight to fix it.
>
> What kernel were you on prior to 5.14.0-362.18.1.el9_3.x86_64 ? If we
> can bracket the changes around a particular version, then that might
> help identify the problem.
The server on which we are experiencing this the most was upgraded from
CentOS 7 recently and only ran 5.14.0-362.18.1 (and now 419). Another
server on which we (less frequently) have this issue is running EL9 for
much longer (kernels 5.14.0-162.23.1, 5.14.0-284.11.1, 5.14.0-284.18.1,
5.14.0-284.30.1, 5.14.0-362.8.1), but we only started to experience the
issue on 5.14.0-362.18.1. It could be the bug was also present in older
versions and that we never triggered it there.
>
>>>> [Mon Mar 11 14:10:08 2024] Not tainted 5.14.0-419.el9.x86_64 #1
>>>> [Mon Mar 11 14:10:08 2024] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
>>>> [Mon Mar 11 14:10:08 2024] task:nfsd state:D stack:0 pid:8865 ppid:2 flags:0x00004000
>>>> [Mon Mar 11 14:10:08 2024] Call Trace:
>>>> [Mon Mar 11 14:10:08 2024] <TASK>
>>>> [Mon Mar 11 14:10:08 2024] __schedule+0x21b/0x550
>>>> [Mon Mar 11 14:10:08 2024] schedule+0x2d/0x70
>>>> [Mon Mar 11 14:10:08 2024] schedule_timeout+0x11f/0x160
>>>> [Mon Mar 11 14:10:08 2024] ? select_idle_sibling+0x28/0x430
>>>> [Mon Mar 11 14:10:08 2024] ? wake_affine+0x62/0x1f0
>>>> [Mon Mar 11 14:10:08 2024] __wait_for_common+0x90/0x1d0
>>>> [Mon Mar 11 14:10:08 2024] ? __pfx_schedule_timeout+0x10/0x10
>>>> [Mon Mar 11 14:10:08 2024] __flush_workqueue+0x13a/0x3f0
>>>> [Mon Mar 11 14:10:08 2024] nfsd4_shutdown_callback+0x49/0x120 [nfsd]
>>>> [Mon Mar 11 14:10:08 2024] ? nfsd4_cld_remove+0x54/0x1d0 [nfsd]
>>>> [Mon Mar 11 14:10:08 2024] ? nfsd4_return_all_client_layouts+0xc4/0xf0 [nfsd]
>>>> [Mon Mar 11 14:10:08 2024] ? nfsd4_shutdown_copy+0x68/0xc0 [nfsd]
>>>> [Mon Mar 11 14:10:08 2024] __destroy_client+0x1f3/0x290 [nfsd]
>>>> [Mon Mar 11 14:10:08 2024] nfsd4_exchange_id+0x75f/0x770 [nfsd]
>>>> [Mon Mar 11 14:10:08 2024] ? nfsd4_decode_opaque+0x3a/0x90 [nfsd]
>>>> [Mon Mar 11 14:10:08 2024] nfsd4_proc_compound+0x44b/0x700 [nfsd]
>>>> [Mon Mar 11 14:10:08 2024] nfsd_dispatch+0x94/0x1c0 [nfsd]
>>>> [Mon Mar 11 14:10:08 2024] svc_process_common+0x2ec/0x660 [sunrpc]
>>>> [Mon Mar 11 14:10:08 2024] ? __pfx_nfsd_dispatch+0x10/0x10 [nfsd]
>>>> [Mon Mar 11 14:10:08 2024] ? __pfx_nfsd+0x10/0x10 [nfsd]
>>>> [Mon Mar 11 14:10:08 2024] svc_process+0x12d/0x170 [sunrpc]
>>>> [Mon Mar 11 14:10:08 2024] nfsd+0x84/0xb0 [nfsd]
>>>> [Mon Mar 11 14:10:08 2024] kthread+0xdd/0x100
>>>> [Mon Mar 11 14:10:08 2024] ? __pfx_kthread+0x10/0x10
>>>> [Mon Mar 11 14:10:08 2024] ret_from_fork+0x29/0x50
>>>> [Mon Mar 11 14:10:08 2024] </TASK>
>>>> [Mon Mar 11 14:10:08 2024] INFO: task nfsd:8866 blocked for more than 122 seconds.
>>>> [Mon Mar 11 14:10:08 2024] Not tainted 5.14.0-419.el9.x86_64 #1
>>>> [Mon Mar 11 14:10:08 2024] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
>>>> [Mon Mar 11 14:10:08 2024] task:nfsd state:D stack:0 pid:8866 ppid:2 flags:0x00004000
>>>> [Mon Mar 11 14:10:08 2024] Call Trace:
>>>> [Mon Mar 11 14:10:08 2024] <TASK>
>>>> [Mon Mar 11 14:10:08 2024] __schedule+0x21b/0x550
>>>> [Mon Mar 11 14:10:08 2024] schedule+0x2d/0x70
>>>> [Mon Mar 11 14:10:08 2024] schedule_timeout+0x11f/0x160
>>>> [Mon Mar 11 14:10:08 2024] ? select_idle_sibling+0x28/0x430
>>>> [Mon Mar 11 14:10:08 2024] ? tcp_recvmsg+0x196/0x210
>>>> [Mon Mar 11 14:10:08 2024] ? wake_affine+0x62/0x1f0
>>>> [Mon Mar 11 14:10:08 2024] __wait_for_common+0x90/0x1d0
>>>> [Mon Mar 11 14:10:08 2024] ? __pfx_schedule_timeout+0x10/0x10
>>>> [Mon Mar 11 14:10:08 2024] __flush_workqueue+0x13a/0x3f0
>>>> [Mon Mar 11 14:10:08 2024] nfsd4_destroy_session+0x1a4/0x240 [nfsd]
>>>> [Mon Mar 11 14:10:08 2024] nfsd4_proc_compound+0x44b/0x700 [nfsd]
>>>> [Mon Mar 11 14:10:08 2024] nfsd_dispatch+0x94/0x1c0 [nfsd]
>>>> [Mon Mar 11 14:10:08 2024] svc_process_common+0x2ec/0x660 [sunrpc]
>>>> [Mon Mar 11 14:10:08 2024] ? __pfx_nfsd_dispatch+0x10/0x10 [nfsd]
>>>> [Mon Mar 11 14:10:08 2024] ? __pfx_nfsd+0x10/0x10 [nfsd]
>>>> [Mon Mar 11 14:10:08 2024] svc_process+0x12d/0x170 [sunrpc]
>>>> [Mon Mar 11 14:10:08 2024] nfsd+0x84/0xb0 [nfsd]
>>>> [Mon Mar 11 14:10:08 2024] kthread+0xdd/0x100
>>>> [Mon Mar 11 14:10:08 2024] ? __pfx_kthread+0x10/0x10
>>>> [Mon Mar 11 14:10:08 2024] ret_from_fork+0x29/0x50
>>>> [Mon Mar 11 14:10:08 2024] </TASK>
>>>>
>>> The above threads are trying to flush the workqueue, so that probably
>>> means that they are stuck waiting on a workqueue job to finish.
>>>> The above is repeated a few times, and then this warning is also logged:
>>>>
>>>> [Mon Mar 11 14:12:04 2024] ------------[ cut here ]------------
>>>> [Mon Mar 11 14:12:04 2024] WARNING: CPU: 39 PID: 8844 at fs/nfsd/nfs4state.c:4919 nfsd_break_deleg_cb+0x170/0x190 [nfsd]
>>>> [Mon Mar 11 14:12:05 2024] Modules linked in: nfsv4 dns_resolver nfs fscache netfs rpcsec_gss_krb5 rpcrdma rdma_cm iw_cm ib_cm ib_core binfmt_misc bonding tls rfkill nft_counter nft_ct
>>>> nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_reject_inet nf_reject_ipv4 nf_reject_ipv6 nft_reject nf_tables nfnetlink vfat fat dm_thin_pool dm_persistent_data dm_bio_prison dm_bufio l
>>>> ibcrc32c dm_service_time dm_multipath intel_rapl_msr intel_rapl_common intel_uncore_frequency intel_uncore_frequency_common isst_if_common skx_edac nfit libnvdimm ipmi_ssif x86_pkg_temp
>>>> _thermal intel_powerclamp coretemp kvm_intel kvm irqbypass dcdbas rapl intel_cstate mgag200 i2c_algo_bit drm_shmem_helper dell_smbios drm_kms_helper dell_wmi_descriptor wmi_bmof intel_u
>>>> ncore syscopyarea pcspkr sysfillrect mei_me sysimgblt acpi_ipmi mei fb_sys_fops i2c_i801 ipmi_si intel_pch_thermal lpc_ich ipmi_devintf i2c_smbus ipmi_msghandler joydev acpi_power_meter
>>>> nfsd auth_rpcgss nfs_acl drm lockd grace fuse sunrpc ext4 mbcache jbd2 sd_mod sg lpfc
>>>> [Mon Mar 11 14:12:05 2024] nvmet_fc nvmet nvme_fc nvme_fabrics crct10dif_pclmul ahci libahci crc32_pclmul nvme_core crc32c_intel ixgbe megaraid_sas libata nvme_common ghash_clmulni_int
>>>> el t10_pi wdat_wdt scsi_transport_fc mdio wmi dca dm_mirror dm_region_hash dm_log dm_mod
>>>> [Mon Mar 11 14:12:05 2024] CPU: 39 PID: 8844 Comm: nfsd Not tainted 5.14.0-419.el9.x86_64 #1
>>>> [Mon Mar 11 14:12:05 2024] Hardware name: Dell Inc. PowerEdge R740/00WGD1, BIOS 2.20.1 09/13/2023
>>>> [Mon Mar 11 14:12:05 2024] RIP: 0010:nfsd_break_deleg_cb+0x170/0x190 [nfsd]
>>>> [Mon Mar 11 14:12:05 2024] Code: a6 95 c5 f3 e9 ff fe ff ff 48 89 df be 01 00 00 00 e8 34 b5 13 f4 48 8d bb 98 00 00 00 e8 c8 f9 00 00 84 c0 0f 85 2e ff ff ff <0f> 0b e9 27 ff ff ff be
>>>> 02 00 00 00 48 89 df e8 0c b5 13 f4 e9 01
>>>> [Mon Mar 11 14:12:05 2024] RSP: 0018:ffff9929e0bb7b80 EFLAGS: 00010246
>>>> [Mon Mar 11 14:12:05 2024] RAX: 0000000000000000 RBX: ffff8ada51930900 RCX: 0000000000000024
>>>> [Mon Mar 11 14:12:05 2024] RDX: ffff8ada519309c8 RSI: ffff8ad582933c00 RDI: 0000000000002000
>>>> [Mon Mar 11 14:12:05 2024] RBP: ffff8ad46bf21574 R08: ffff9929e0bb7b48 R09: 0000000000000000
>>>> [Mon Mar 11 14:12:05 2024] R10: ffff8aec859a2948 R11: 0000000000000000 R12: ffff8ad6f497c360
>>>> [Mon Mar 11 14:12:05 2024] R13: ffff8ad46bf21560 R14: ffff8ae5942e0b10 R15: ffff8ad6f497c360
>>>> [Mon Mar 11 14:12:05 2024] FS: 0000000000000000(0000) GS:ffff8b031fcc0000(0000) knlGS:0000000000000000
>>>> [Mon Mar 11 14:12:05 2024] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
>>>> [Mon Mar 11 14:12:05 2024] CR2: 00007fafe2060744 CR3: 00000018e58de006 CR4: 00000000007706e0
>>>> [Mon Mar 11 14:12:05 2024] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
>>>> [Mon Mar 11 14:12:05 2024] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
>>>> [Mon Mar 11 14:12:05 2024] PKRU: 55555554
>>>> [Mon Mar 11 14:12:05 2024] Call Trace:
>>>> [Mon Mar 11 14:12:05 2024] <TASK>
>>>> [Mon Mar 11 14:12:05 2024] ? show_trace_log_lvl+0x1c4/0x2df
>>>> [Mon Mar 11 14:12:05 2024] ? show_trace_log_lvl+0x1c4/0x2df
>>>> [Mon Mar 11 14:12:05 2024] ? __break_lease+0x16f/0x5f0
>>>> [Mon Mar 11 14:12:05 2024] ? nfsd_break_deleg_cb+0x170/0x190 [nfsd]
>>>> [Mon Mar 11 14:12:05 2024] ? __warn+0x81/0x110
>>>> [Mon Mar 11 14:12:05 2024] ? nfsd_break_deleg_cb+0x170/0x190 [nfsd]
>>>> [Mon Mar 11 14:12:05 2024] ? report_bug+0x10a/0x140
>>>> [Mon Mar 11 14:12:05 2024] ? handle_bug+0x3c/0x70
>>>> [Mon Mar 11 14:12:05 2024] ? exc_invalid_op+0x14/0x70
>>>> [Mon Mar 11 14:12:05 2024] ? asm_exc_invalid_op+0x16/0x20
>>>> [Mon Mar 11 14:12:05 2024] ? nfsd_break_deleg_cb+0x170/0x190 [nfsd]
>>>> [Mon Mar 11 14:12:05 2024] __break_lease+0x16f/0x5f0
>>>> [Mon Mar 11 14:12:05 2024] ? nfsd_file_lookup_locked+0x117/0x160 [nfsd]
>>>> [Mon Mar 11 14:12:05 2024] ? list_lru_del+0x101/0x150
>>>> [Mon Mar 11 14:12:05 2024] nfsd_file_do_acquire+0x790/0x830 [nfsd]
>>>> [Mon Mar 11 14:12:05 2024] nfs4_get_vfs_file+0x315/0x3a0 [nfsd]
>>>> [Mon Mar 11 14:12:05 2024] nfsd4_process_open2+0x430/0xa30 [nfsd]
>>>> [Mon Mar 11 14:12:05 2024] ? fh_verify+0x297/0x2f0 [nfsd]
>>>> [Mon Mar 11 14:12:05 2024] nfsd4_open+0x3ce/0x4b0 [nfsd]
>>>> [Mon Mar 11 14:12:05 2024] nfsd4_proc_compound+0x44b/0x700 [nfsd]
>>>> [Mon Mar 11 14:12:05 2024] nfsd_dispatch+0x94/0x1c0 [nfsd]
>>>> [Mon Mar 11 14:12:05 2024] svc_process_common+0x2ec/0x660 [sunrpc]
>>>> [Mon Mar 11 14:12:05 2024] ? __pfx_nfsd_dispatch+0x10/0x10 [nfsd]
>>>> [Mon Mar 11 14:12:05 2024] ? __pfx_nfsd+0x10/0x10 [nfsd]
>>>> [Mon Mar 11 14:12:05 2024] svc_process+0x12d/0x170 [sunrpc]
>>>> [Mon Mar 11 14:12:05 2024] nfsd+0x84/0xb0 [nfsd]
>>>> [Mon Mar 11 14:12:05 2024] kthread+0xdd/0x100
>>>> [Mon Mar 11 14:12:05 2024] ? __pfx_kthread+0x10/0x10
>>>> [Mon Mar 11 14:12:05 2024] ret_from_fork+0x29/0x50
>>>> [Mon Mar 11 14:12:05 2024] </TASK>
>>>> [Mon Mar 11 14:12:05 2024] ---[ end trace 7a039e17443dc651 ]---
>>> This is probably this WARN in nfsd_break_one_deleg:
>>>
>>> WARN_ON_ONCE(!nfsd4_run_cb(&dp->dl_recall));
>>>
>>> It means that a delegation break callback to the client couldn't be
>>> queued to the workqueue, and so it didn't run.
>>>
>>>> Could this be the same issue as described here:https://lore.kernel.org/linux-nfs/af0ec881-5ebf-4feb-98ae-3ed2a77f86f1@oracle.com/ ?
>>>>
>>> Yes, most likely the same problem.
>> If I read that thread correctly, this issue was introduced between
>> 6.1.63 and 6.6.3? Is it possible the EL9 5.14.0-362.18.1.el9_3
>> backported these changes, or were we hitting some other bug with that
>> version? It seems the 6.1.x kernel is not affected? If so, that would be
>> the recommended kernel to run?
> Anything is possible. We have to identify the problem first.
>
>>>> As described in that thread, I've tried to obtain the requested information.
>>>>
>>>>
>>>> Is it possible this is the issue that was fixed by the patches described here? https://lore.kernel.org/linux-nfs/2024022054-cause-suffering-eae8@gregkh/
>>>>
>>> Doubtful. Those are targeted toward a different set of issues.
>>>
>>> If you're willing, I do have some patches queued up for CentOS here that
>>> fix some backchannel problems that could be related. I'm mainly waiting
>>> on Chuck to send these to Linus and then we'll likely merge them into
>>> CentOS soon afterward:
>>>
>>> https://gitlab.com/redhat/centos-stream/src/kernel/centos-stream-9/-/merge_requests/3689
>>>
>> If you can send me a patch file, I can rebuild the C9S kernel with that
>> patch and run it. It can take a while for the bug to trigger as I
>> believe it seems to be very workload dependent (we were running very
>> stable for months and now hit this bug every other week).
>>
>>
> It's probably simpler to just pull down the build artifacts for that MR.
> You have to drill down through the CI for it, but they are here:
>
> https://s3.amazonaws.com/arr-cki-prod-trusted-artifacts/index.html?prefix=trusted-artifacts/1194300175/publish_x86_64/6278921877/artifacts/
>
> There's even a repo file you can install on the box to pull them down.
Ok, I will try these instead.
Regards,
Rik
--
Rik Theys
System Engineer
KU Leuven - Dept. Elektrotechniek (ESAT)
Kasteelpark Arenberg 10 bus 2440 - B-3001 Leuven-Heverlee
+32(0)16/32.11.07
----------------------------------------------------------------
<<Any errors in spelling, tact or fact are transmission errors>>
^ permalink raw reply [flat|nested] 26+ messages in thread
* Re: nfsd hangs and nfsd_break_deleg_cb+0x170/0x190 warning
2024-03-11 18:43 Rik Theys
2024-03-12 11:22 ` Jeff Layton
@ 2024-03-12 11:37 ` Jeff Layton
2024-03-12 16:43 ` Dai Ngo
1 sibling, 1 reply; 26+ messages in thread
From: Jeff Layton @ 2024-03-12 11:37 UTC (permalink / raw)
To: Rik Theys, Linux Nfs
On Mon, 2024-03-11 at 19:43 +0100, Rik Theys wrote:
>
>
>
>
> Hi,
>
>
>
>
> Since a few weeks our Rocky Linux 9 NFS server has periodically logged hung nfsd tasks. The initial effect was that some clients could no longer access the NFS server. This got worse and worse (probably as more nfsd threads got blocked) and we had to restart the server. Restarting the server also failed as the NFS server service could no longer be stopped.
>
>
>
>
> The initial kernel we noticed this behavior on was kernel-5.14.0-362.18.1.el9_3.x86_64. Since then we've installed kernel-5.14.0-419.el9.x86_64 from CentOS Stream 9. The same issue happened again on this newer kernel version:
>
>
>
>
> [Mon Mar 11 14:10:08 2024] Not tainted 5.14.0-419.el9.x86_64 #1
> [Mon Mar 11 14:10:08 2024] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> [Mon Mar 11 14:10:08 2024] task:nfsd state:D stack:0 pid:8865 ppid:2 flags:0x00004000
> [Mon Mar 11 14:10:08 2024] Call Trace:
> [Mon Mar 11 14:10:08 2024] <TASK>
> [Mon Mar 11 14:10:08 2024] __schedule+0x21b/0x550
> [Mon Mar 11 14:10:08 2024] schedule+0x2d/0x70
> [Mon Mar 11 14:10:08 2024] schedule_timeout+0x11f/0x160
> [Mon Mar 11 14:10:08 2024] ? select_idle_sibling+0x28/0x430
> [Mon Mar 11 14:10:08 2024] ? wake_affine+0x62/0x1f0
> [Mon Mar 11 14:10:08 2024] __wait_for_common+0x90/0x1d0
> [Mon Mar 11 14:10:08 2024] ? __pfx_schedule_timeout+0x10/0x10
> [Mon Mar 11 14:10:08 2024] __flush_workqueue+0x13a/0x3f0
> [Mon Mar 11 14:10:08 2024] nfsd4_shutdown_callback+0x49/0x120 [nfsd]
> [Mon Mar 11 14:10:08 2024] ? nfsd4_cld_remove+0x54/0x1d0 [nfsd]
> [Mon Mar 11 14:10:08 2024] ? nfsd4_return_all_client_layouts+0xc4/0xf0 [nfsd]
> [Mon Mar 11 14:10:08 2024] ? nfsd4_shutdown_copy+0x68/0xc0 [nfsd]
> [Mon Mar 11 14:10:08 2024] __destroy_client+0x1f3/0x290 [nfsd]
> [Mon Mar 11 14:10:08 2024] nfsd4_exchange_id+0x75f/0x770 [nfsd]
> [Mon Mar 11 14:10:08 2024] ? nfsd4_decode_opaque+0x3a/0x90 [nfsd]
> [Mon Mar 11 14:10:08 2024] nfsd4_proc_compound+0x44b/0x700 [nfsd]
> [Mon Mar 11 14:10:08 2024] nfsd_dispatch+0x94/0x1c0 [nfsd]
> [Mon Mar 11 14:10:08 2024] svc_process_common+0x2ec/0x660 [sunrpc]
> [Mon Mar 11 14:10:08 2024] ? __pfx_nfsd_dispatch+0x10/0x10 [nfsd]
> [Mon Mar 11 14:10:08 2024] ? __pfx_nfsd+0x10/0x10 [nfsd]
> [Mon Mar 11 14:10:08 2024] svc_process+0x12d/0x170 [sunrpc]
> [Mon Mar 11 14:10:08 2024] nfsd+0x84/0xb0 [nfsd]
> [Mon Mar 11 14:10:08 2024] kthread+0xdd/0x100
> [Mon Mar 11 14:10:08 2024] ? __pfx_kthread+0x10/0x10
> [Mon Mar 11 14:10:08 2024] ret_from_fork+0x29/0x50
> [Mon Mar 11 14:10:08 2024] </TASK>
> [Mon Mar 11 14:10:08 2024] INFO: task nfsd:8866 blocked for more than 122 seconds.
> [Mon Mar 11 14:10:08 2024] Not tainted 5.14.0-419.el9.x86_64 #1
> [Mon Mar 11 14:10:08 2024] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> [Mon Mar 11 14:10:08 2024] task:nfsd state:D stack:0 pid:8866 ppid:2 flags:0x00004000
> [Mon Mar 11 14:10:08 2024] Call Trace:
> [Mon Mar 11 14:10:08 2024] <TASK>
> [Mon Mar 11 14:10:08 2024] __schedule+0x21b/0x550
> [Mon Mar 11 14:10:08 2024] schedule+0x2d/0x70
> [Mon Mar 11 14:10:08 2024] schedule_timeout+0x11f/0x160
> [Mon Mar 11 14:10:08 2024] ? select_idle_sibling+0x28/0x430
> [Mon Mar 11 14:10:08 2024] ? tcp_recvmsg+0x196/0x210
> [Mon Mar 11 14:10:08 2024] ? wake_affine+0x62/0x1f0
> [Mon Mar 11 14:10:08 2024] __wait_for_common+0x90/0x1d0
> [Mon Mar 11 14:10:08 2024] ? __pfx_schedule_timeout+0x10/0x10
> [Mon Mar 11 14:10:08 2024] __flush_workqueue+0x13a/0x3f0
> [Mon Mar 11 14:10:08 2024] nfsd4_destroy_session+0x1a4/0x240 [nfsd]
> [Mon Mar 11 14:10:08 2024] nfsd4_proc_compound+0x44b/0x700 [nfsd]
> [Mon Mar 11 14:10:08 2024] nfsd_dispatch+0x94/0x1c0 [nfsd]
> [Mon Mar 11 14:10:08 2024] svc_process_common+0x2ec/0x660 [sunrpc]
> [Mon Mar 11 14:10:08 2024] ? __pfx_nfsd_dispatch+0x10/0x10 [nfsd]
> [Mon Mar 11 14:10:08 2024] ? __pfx_nfsd+0x10/0x10 [nfsd]
> [Mon Mar 11 14:10:08 2024] svc_process+0x12d/0x170 [sunrpc]
> [Mon Mar 11 14:10:08 2024] nfsd+0x84/0xb0 [nfsd]
> [Mon Mar 11 14:10:08 2024] kthread+0xdd/0x100
> [Mon Mar 11 14:10:08 2024] ? __pfx_kthread+0x10/0x10
> [Mon Mar 11 14:10:08 2024] ret_from_fork+0x29/0x50
> [Mon Mar 11 14:10:08 2024] </TASK>
>
>
>
>
> The above is repeated a few times, and then this warning is also logged:
>
>
>
>
> [Mon Mar 11 14:12:04 2024] ------------[ cut here ]------------
> [Mon Mar 11 14:12:04 2024] WARNING: CPU: 39 PID: 8844 at fs/nfsd/nfs4state.c:4919 nfsd_break_deleg_cb+0x170/0x190 [nfsd]
> [Mon Mar 11 14:12:05 2024] Modules linked in: nfsv4 dns_resolver nfs fscache netfs rpcsec_gss_krb5 rpcrdma rdma_cm iw_cm ib_cm ib_core binfmt_misc bonding tls rfkill nft_counter nft_ct
> nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_reject_inet nf_reject_ipv4 nf_reject_ipv6 nft_reject nf_tables nfnetlink vfat fat dm_thin_pool dm_persistent_data dm_bio_prison dm_bufio l
> ibcrc32c dm_service_time dm_multipath intel_rapl_msr intel_rapl_common intel_uncore_frequency intel_uncore_frequency_common isst_if_common skx_edac nfit libnvdimm ipmi_ssif x86_pkg_temp
> _thermal intel_powerclamp coretemp kvm_intel kvm irqbypass dcdbas rapl intel_cstate mgag200 i2c_algo_bit drm_shmem_helper dell_smbios drm_kms_helper dell_wmi_descriptor wmi_bmof intel_u
> ncore syscopyarea pcspkr sysfillrect mei_me sysimgblt acpi_ipmi mei fb_sys_fops i2c_i801 ipmi_si intel_pch_thermal lpc_ich ipmi_devintf i2c_smbus ipmi_msghandler joydev acpi_power_meter
> nfsd auth_rpcgss nfs_acl drm lockd grace fuse sunrpc ext4 mbcache jbd2 sd_mod sg lpfc
> [Mon Mar 11 14:12:05 2024] nvmet_fc nvmet nvme_fc nvme_fabrics crct10dif_pclmul ahci libahci crc32_pclmul nvme_core crc32c_intel ixgbe megaraid_sas libata nvme_common ghash_clmulni_int
> el t10_pi wdat_wdt scsi_transport_fc mdio wmi dca dm_mirror dm_region_hash dm_log dm_mod
> [Mon Mar 11 14:12:05 2024] CPU: 39 PID: 8844 Comm: nfsd Not tainted 5.14.0-419.el9.x86_64 #1
> [Mon Mar 11 14:12:05 2024] Hardware name: Dell Inc. PowerEdge R740/00WGD1, BIOS 2.20.1 09/13/2023
> [Mon Mar 11 14:12:05 2024] RIP: 0010:nfsd_break_deleg_cb+0x170/0x190 [nfsd]
> [Mon Mar 11 14:12:05 2024] Code: a6 95 c5 f3 e9 ff fe ff ff 48 89 df be 01 00 00 00 e8 34 b5 13 f4 48 8d bb 98 00 00 00 e8 c8 f9 00 00 84 c0 0f 85 2e ff ff ff <0f> 0b e9 27 ff ff ff be
> 02 00 00 00 48 89 df e8 0c b5 13 f4 e9 01
> [Mon Mar 11 14:12:05 2024] RSP: 0018:ffff9929e0bb7b80 EFLAGS: 00010246
> [Mon Mar 11 14:12:05 2024] RAX: 0000000000000000 RBX: ffff8ada51930900 RCX: 0000000000000024
> [Mon Mar 11 14:12:05 2024] RDX: ffff8ada519309c8 RSI: ffff8ad582933c00 RDI: 0000000000002000
> [Mon Mar 11 14:12:05 2024] RBP: ffff8ad46bf21574 R08: ffff9929e0bb7b48 R09: 0000000000000000
> [Mon Mar 11 14:12:05 2024] R10: ffff8aec859a2948 R11: 0000000000000000 R12: ffff8ad6f497c360
> [Mon Mar 11 14:12:05 2024] R13: ffff8ad46bf21560 R14: ffff8ae5942e0b10 R15: ffff8ad6f497c360
> [Mon Mar 11 14:12:05 2024] FS: 0000000000000000(0000) GS:ffff8b031fcc0000(0000) knlGS:0000000000000000
> [Mon Mar 11 14:12:05 2024] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> [Mon Mar 11 14:12:05 2024] CR2: 00007fafe2060744 CR3: 00000018e58de006 CR4: 00000000007706e0
> [Mon Mar 11 14:12:05 2024] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
> [Mon Mar 11 14:12:05 2024] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
> [Mon Mar 11 14:12:05 2024] PKRU: 55555554
> [Mon Mar 11 14:12:05 2024] Call Trace:
> [Mon Mar 11 14:12:05 2024] <TASK>
> [Mon Mar 11 14:12:05 2024] ? show_trace_log_lvl+0x1c4/0x2df
> [Mon Mar 11 14:12:05 2024] ? show_trace_log_lvl+0x1c4/0x2df
> [Mon Mar 11 14:12:05 2024] ? __break_lease+0x16f/0x5f0
> [Mon Mar 11 14:12:05 2024] ? nfsd_break_deleg_cb+0x170/0x190 [nfsd]
> [Mon Mar 11 14:12:05 2024] ? __warn+0x81/0x110
> [Mon Mar 11 14:12:05 2024] ? nfsd_break_deleg_cb+0x170/0x190 [nfsd]
> [Mon Mar 11 14:12:05 2024] ? report_bug+0x10a/0x140
> [Mon Mar 11 14:12:05 2024] ? handle_bug+0x3c/0x70
> [Mon Mar 11 14:12:05 2024] ? exc_invalid_op+0x14/0x70
> [Mon Mar 11 14:12:05 2024] ? asm_exc_invalid_op+0x16/0x20
> [Mon Mar 11 14:12:05 2024] ? nfsd_break_deleg_cb+0x170/0x190 [nfsd]
> [Mon Mar 11 14:12:05 2024] __break_lease+0x16f/0x5f0
> [Mon Mar 11 14:12:05 2024] ? nfsd_file_lookup_locked+0x117/0x160 [nfsd]
> [Mon Mar 11 14:12:05 2024] ? list_lru_del+0x101/0x150
> [Mon Mar 11 14:12:05 2024] nfsd_file_do_acquire+0x790/0x830 [nfsd]
> [Mon Mar 11 14:12:05 2024] nfs4_get_vfs_file+0x315/0x3a0 [nfsd]
> [Mon Mar 11 14:12:05 2024] nfsd4_process_open2+0x430/0xa30 [nfsd]
> [Mon Mar 11 14:12:05 2024] ? fh_verify+0x297/0x2f0 [nfsd]
> [Mon Mar 11 14:12:05 2024] nfsd4_open+0x3ce/0x4b0 [nfsd]
> [Mon Mar 11 14:12:05 2024] nfsd4_proc_compound+0x44b/0x700 [nfsd]
> [Mon Mar 11 14:12:05 2024] nfsd_dispatch+0x94/0x1c0 [nfsd]
> [Mon Mar 11 14:12:05 2024] svc_process_common+0x2ec/0x660 [sunrpc]
> [Mon Mar 11 14:12:05 2024] ? __pfx_nfsd_dispatch+0x10/0x10 [nfsd]
> [Mon Mar 11 14:12:05 2024] ? __pfx_nfsd+0x10/0x10 [nfsd]
> [Mon Mar 11 14:12:05 2024] svc_process+0x12d/0x170 [sunrpc]
> [Mon Mar 11 14:12:05 2024] nfsd+0x84/0xb0 [nfsd]
> [Mon Mar 11 14:12:05 2024] kthread+0xdd/0x100
> [Mon Mar 11 14:12:05 2024] ? __pfx_kthread+0x10/0x10
> [Mon Mar 11 14:12:05 2024] ret_from_fork+0x29/0x50
> [Mon Mar 11 14:12:05 2024] </TASK>
> [Mon Mar 11 14:12:05 2024] ---[ end trace 7a039e17443dc651 ]---
>
[Mon Mar 11 14:29:16 2024] task:kworker/u96:3 state:D stack:0 pid:2451130 ppid:2 flags:0x00004000
[Mon Mar 11 14:29:16 2024] Workqueue: nfsd4_callbacks nfsd4_run_cb_work [nfsd]
[Mon Mar 11 14:29:16 2024] Call Trace:
[Mon Mar 11 14:29:16 2024] <TASK>
[Mon Mar 11 14:29:16 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:16 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:16 2024] schedule_timeout+0x88/0x160
[Mon Mar 11 14:29:16 2024] ? __pfx_process_timeout+0x10/0x10
[Mon Mar 11 14:29:16 2024] rpc_shutdown_client+0xb3/0x150 [sunrpc]
[Mon Mar 11 14:29:16 2024] ? __pfx_autoremove_wake_function+0x10/0x10
[Mon Mar 11 14:29:16 2024] nfsd4_process_cb_update+0x3e/0x260 [nfsd]
[Mon Mar 11 14:29:16 2024] ? sched_clock+0xc/0x30
[Mon Mar 11 14:29:16 2024] ? raw_spin_rq_lock_nested+0x19/0x80
[Mon Mar 11 14:29:16 2024] ? newidle_balance+0x26e/0x400
[Mon Mar 11 14:29:16 2024] ? pick_next_task_fair+0x41/0x500
[Mon Mar 11 14:29:16 2024] ? put_prev_task_fair+0x1e/0x40
[Mon Mar 11 14:29:16 2024] ? pick_next_task+0x861/0x950
[Mon Mar 11 14:29:16 2024] ? __update_idle_core+0x23/0xc0
[Mon Mar 11 14:29:16 2024] ? __switch_to_asm+0x3a/0x80
[Mon Mar 11 14:29:16 2024] ? finish_task_switch.isra.0+0x8c/0x2a0
[Mon Mar 11 14:29:16 2024] nfsd4_run_cb_work+0x9f/0x150 [nfsd]
[Mon Mar 11 14:29:16 2024] process_one_work+0x1e2/0x3b0
[Mon Mar 11 14:29:16 2024] worker_thread+0x50/0x3a0
[Mon Mar 11 14:29:16 2024] ? __pfx_worker_thread+0x10/0x10
[Mon Mar 11 14:29:16 2024] kthread+0xdd/0x100
[Mon Mar 11 14:29:16 2024] ? __pfx_kthread+0x10/0x10
[Mon Mar 11 14:29:16 2024] ret_from_fork+0x29/0x50
[Mon Mar 11 14:29:16 2024] </TASK>
The above is the main task that I see in the cb workqueue. It's trying to call rpc_shutdown_client, which is waiting for this:
wait_event_timeout(destroy_wait,
list_empty(&clnt->cl_tasks), 1*HZ);
...so basically waiting for the cl_tasks list to go empty. It repeatedly
does a rpc_killall_tasks though, so possibly trying to kill this task?
18423 2281 0 0x18 0x0 1354 nfsd4_cb_ops [nfsd] nfs4_cbv1 CB_RECALL_ANY a:call_start [sunrpc] q:delayq
Callbacks are soft RPC tasks though, so they should be easily killable.
--
Jeff Layton <jlayton@kernel.org>
^ permalink raw reply [flat|nested] 26+ messages in thread
* Re: nfsd hangs and nfsd_break_deleg_cb+0x170/0x190 warning
2024-03-11 18:43 Rik Theys
@ 2024-03-12 11:22 ` Jeff Layton
2024-03-12 11:37 ` Jeff Layton
1 sibling, 0 replies; 26+ messages in thread
From: Jeff Layton @ 2024-03-12 11:22 UTC (permalink / raw)
To: Rik Theys, Linux Nfs
On Mon, 2024-03-11 at 19:43 +0100, Rik Theys wrote:
>
>
>
>
> Hi,
>
>
>
>
> Since a few weeks our Rocky Linux 9 NFS server has periodically logged hung nfsd tasks. The initial effect was that some clients could no longer access the NFS server. This got worse and worse (probably as more nfsd threads got blocked) and we had to restart the server. Restarting the server also failed as the NFS server service could no longer be stopped.
>
>
>
>
> The initial kernel we noticed this behavior on was kernel-5.14.0-362.18.1.el9_3.x86_64. Since then we've installed kernel-5.14.0-419.el9.x86_64 from CentOS Stream 9. The same issue happened again on this newer kernel version:
>
>
>
>
> [Mon Mar 11 14:10:08 2024] Not tainted 5.14.0-419.el9.x86_64 #1
> [Mon Mar 11 14:10:08 2024] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> [Mon Mar 11 14:10:08 2024] task:nfsd state:D stack:0 pid:8865 ppid:2 flags:0x00004000
> [Mon Mar 11 14:10:08 2024] Call Trace:
> [Mon Mar 11 14:10:08 2024] <TASK>
> [Mon Mar 11 14:10:08 2024] __schedule+0x21b/0x550
> [Mon Mar 11 14:10:08 2024] schedule+0x2d/0x70
> [Mon Mar 11 14:10:08 2024] schedule_timeout+0x11f/0x160
> [Mon Mar 11 14:10:08 2024] ? select_idle_sibling+0x28/0x430
> [Mon Mar 11 14:10:08 2024] ? wake_affine+0x62/0x1f0
> [Mon Mar 11 14:10:08 2024] __wait_for_common+0x90/0x1d0
> [Mon Mar 11 14:10:08 2024] ? __pfx_schedule_timeout+0x10/0x10
> [Mon Mar 11 14:10:08 2024] __flush_workqueue+0x13a/0x3f0
> [Mon Mar 11 14:10:08 2024] nfsd4_shutdown_callback+0x49/0x120 [nfsd]
> [Mon Mar 11 14:10:08 2024] ? nfsd4_cld_remove+0x54/0x1d0 [nfsd]
> [Mon Mar 11 14:10:08 2024] ? nfsd4_return_all_client_layouts+0xc4/0xf0 [nfsd]
> [Mon Mar 11 14:10:08 2024] ? nfsd4_shutdown_copy+0x68/0xc0 [nfsd]
> [Mon Mar 11 14:10:08 2024] __destroy_client+0x1f3/0x290 [nfsd]
> [Mon Mar 11 14:10:08 2024] nfsd4_exchange_id+0x75f/0x770 [nfsd]
> [Mon Mar 11 14:10:08 2024] ? nfsd4_decode_opaque+0x3a/0x90 [nfsd]
> [Mon Mar 11 14:10:08 2024] nfsd4_proc_compound+0x44b/0x700 [nfsd]
> [Mon Mar 11 14:10:08 2024] nfsd_dispatch+0x94/0x1c0 [nfsd]
> [Mon Mar 11 14:10:08 2024] svc_process_common+0x2ec/0x660 [sunrpc]
> [Mon Mar 11 14:10:08 2024] ? __pfx_nfsd_dispatch+0x10/0x10 [nfsd]
> [Mon Mar 11 14:10:08 2024] ? __pfx_nfsd+0x10/0x10 [nfsd]
> [Mon Mar 11 14:10:08 2024] svc_process+0x12d/0x170 [sunrpc]
> [Mon Mar 11 14:10:08 2024] nfsd+0x84/0xb0 [nfsd]
> [Mon Mar 11 14:10:08 2024] kthread+0xdd/0x100
> [Mon Mar 11 14:10:08 2024] ? __pfx_kthread+0x10/0x10
> [Mon Mar 11 14:10:08 2024] ret_from_fork+0x29/0x50
> [Mon Mar 11 14:10:08 2024] </TASK>
> [Mon Mar 11 14:10:08 2024] INFO: task nfsd:8866 blocked for more than 122 seconds.
> [Mon Mar 11 14:10:08 2024] Not tainted 5.14.0-419.el9.x86_64 #1
> [Mon Mar 11 14:10:08 2024] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> [Mon Mar 11 14:10:08 2024] task:nfsd state:D stack:0 pid:8866 ppid:2 flags:0x00004000
> [Mon Mar 11 14:10:08 2024] Call Trace:
> [Mon Mar 11 14:10:08 2024] <TASK>
> [Mon Mar 11 14:10:08 2024] __schedule+0x21b/0x550
> [Mon Mar 11 14:10:08 2024] schedule+0x2d/0x70
> [Mon Mar 11 14:10:08 2024] schedule_timeout+0x11f/0x160
> [Mon Mar 11 14:10:08 2024] ? select_idle_sibling+0x28/0x430
> [Mon Mar 11 14:10:08 2024] ? tcp_recvmsg+0x196/0x210
> [Mon Mar 11 14:10:08 2024] ? wake_affine+0x62/0x1f0
> [Mon Mar 11 14:10:08 2024] __wait_for_common+0x90/0x1d0
> [Mon Mar 11 14:10:08 2024] ? __pfx_schedule_timeout+0x10/0x10
> [Mon Mar 11 14:10:08 2024] __flush_workqueue+0x13a/0x3f0
> [Mon Mar 11 14:10:08 2024] nfsd4_destroy_session+0x1a4/0x240 [nfsd]
> [Mon Mar 11 14:10:08 2024] nfsd4_proc_compound+0x44b/0x700 [nfsd]
> [Mon Mar 11 14:10:08 2024] nfsd_dispatch+0x94/0x1c0 [nfsd]
> [Mon Mar 11 14:10:08 2024] svc_process_common+0x2ec/0x660 [sunrpc]
> [Mon Mar 11 14:10:08 2024] ? __pfx_nfsd_dispatch+0x10/0x10 [nfsd]
> [Mon Mar 11 14:10:08 2024] ? __pfx_nfsd+0x10/0x10 [nfsd]
> [Mon Mar 11 14:10:08 2024] svc_process+0x12d/0x170 [sunrpc]
> [Mon Mar 11 14:10:08 2024] nfsd+0x84/0xb0 [nfsd]
> [Mon Mar 11 14:10:08 2024] kthread+0xdd/0x100
> [Mon Mar 11 14:10:08 2024] ? __pfx_kthread+0x10/0x10
> [Mon Mar 11 14:10:08 2024] ret_from_fork+0x29/0x50
> [Mon Mar 11 14:10:08 2024] </TASK>
>
>
>
>
>
>
>
>
>
> Hi,
>
>
>
>
>
>
>
>
> Since a few weeks our Rocky Linux 9 NFS server has periodically logged hung nfsd tasks. The initial effect was that some clients could no longer access the NFS server. This got worse and worse (probably as more nfsd threads got blocked) and we had to restart the server. Restarting the server also failed as the NFS server service could no longer be stopped.
>
>
>
>
>
>
>
>
> The initial kernel we noticed this behavior on was kernel-5.14.0-362.18.1.el9_3.x86_64. Since then we've installed kernel-5.14.0-419.el9.x86_64 from CentOS Stream 9. The same issue happened again on this newer kernel version:
>
>
>
>
>
>
>
>
> [Mon Mar 11 14:10:08 2024] Not tainted 5.14.0-419.el9.x86_64 #1
> [Mon Mar 11 14:10:08 2024] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> [Mon Mar 11 14:10:08 2024] task:nfsd state:D stack:0 pid:8865 ppid:2 flags:0x00004000
> [Mon Mar 11 14:10:08 2024] Call Trace:
> [Mon Mar 11 14:10:08 2024] <TASK>
> [Mon Mar 11 14:10:08 2024] __schedule+0x21b/0x550
> [Mon Mar 11 14:10:08 2024] schedule+0x2d/0x70
> [Mon Mar 11 14:10:08 2024] schedule_timeout+0x11f/0x160
> [Mon Mar 11 14:10:08 2024] ? select_idle_sibling+0x28/0x430
> [Mon Mar 11 14:10:08 2024] ? wake_affine+0x62/0x1f0
> [Mon Mar 11 14:10:08 2024] __wait_for_common+0x90/0x1d0
> [Mon Mar 11 14:10:08 2024] ? __pfx_schedule_timeout+0x10/0x10
> [Mon Mar 11 14:10:08 2024] __flush_workqueue+0x13a/0x3f0
> [Mon Mar 11 14:10:08 2024] nfsd4_shutdown_callback+0x49/0x120 [nfsd]
> [Mon Mar 11 14:10:08 2024] ? nfsd4_cld_remove+0x54/0x1d0 [nfsd]
> [Mon Mar 11 14:10:08 2024] ? nfsd4_return_all_client_layouts+0xc4/0xf0 [nfsd]
> [Mon Mar 11 14:10:08 2024] ? nfsd4_shutdown_copy+0x68/0xc0 [nfsd]
> [Mon Mar 11 14:10:08 2024] __destroy_client+0x1f3/0x290 [nfsd]
> [Mon Mar 11 14:10:08 2024] nfsd4_exchange_id+0x75f/0x770 [nfsd]
> [Mon Mar 11 14:10:08 2024] ? nfsd4_decode_opaque+0x3a/0x90 [nfsd]
> [Mon Mar 11 14:10:08 2024] nfsd4_proc_compound+0x44b/0x700 [nfsd]
> [Mon Mar 11 14:10:08 2024] nfsd_dispatch+0x94/0x1c0 [nfsd]
> [Mon Mar 11 14:10:08 2024] svc_process_common+0x2ec/0x660 [sunrpc]
> [Mon Mar 11 14:10:08 2024] ? __pfx_nfsd_dispatch+0x10/0x10 [nfsd]
> [Mon Mar 11 14:10:08 2024] ? __pfx_nfsd+0x10/0x10 [nfsd]
> [Mon Mar 11 14:10:08 2024] svc_process+0x12d/0x170 [sunrpc]
> [Mon Mar 11 14:10:08 2024] nfsd+0x84/0xb0 [nfsd]
> [Mon Mar 11 14:10:08 2024] kthread+0xdd/0x100
> [Mon Mar 11 14:10:08 2024] ? __pfx_kthread+0x10/0x10
> [Mon Mar 11 14:10:08 2024] ret_from_fork+0x29/0x50
> [Mon Mar 11 14:10:08 2024] </TASK>
> [Mon Mar 11 14:10:08 2024] INFO: task nfsd:8866 blocked for more than 122 seconds.
> [Mon Mar 11 14:10:08 2024] Not tainted 5.14.0-419.el9.x86_64 #1
> [Mon Mar 11 14:10:08 2024] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> [Mon Mar 11 14:10:08 2024] task:nfsd state:D stack:0 pid:8866 ppid:2 flags:0x00004000
> [Mon Mar 11 14:10:08 2024] Call Trace:
> [Mon Mar 11 14:10:08 2024] <TASK>
> [Mon Mar 11 14:10:08 2024] __schedule+0x21b/0x550
> [Mon Mar 11 14:10:08 2024] schedule+0x2d/0x70
> [Mon Mar 11 14:10:08 2024] schedule_timeout+0x11f/0x160
> [Mon Mar 11 14:10:08 2024] ? select_idle_sibling+0x28/0x430
> [Mon Mar 11 14:10:08 2024] ? tcp_recvmsg+0x196/0x210
> [Mon Mar 11 14:10:08 2024] ? wake_affine+0x62/0x1f0
> [Mon Mar 11 14:10:08 2024] __wait_for_common+0x90/0x1d0
> [Mon Mar 11 14:10:08 2024] ? __pfx_schedule_timeout+0x10/0x10
> [Mon Mar 11 14:10:08 2024] __flush_workqueue+0x13a/0x3f0
> [Mon Mar 11 14:10:08 2024] nfsd4_destroy_session+0x1a4/0x240 [nfsd]
> [Mon Mar 11 14:10:08 2024] nfsd4_proc_compound+0x44b/0x700 [nfsd]
> [Mon Mar 11 14:10:08 2024] nfsd_dispatch+0x94/0x1c0 [nfsd]
> [Mon Mar 11 14:10:08 2024] svc_process_common+0x2ec/0x660 [sunrpc]
> [Mon Mar 11 14:10:08 2024] ? __pfx_nfsd_dispatch+0x10/0x10 [nfsd]
> [Mon Mar 11 14:10:08 2024] ? __pfx_nfsd+0x10/0x10 [nfsd]
> [Mon Mar 11 14:10:08 2024] svc_process+0x12d/0x170 [sunrpc]
> [Mon Mar 11 14:10:08 2024] nfsd+0x84/0xb0 [nfsd]
> [Mon Mar 11 14:10:08 2024] kthread+0xdd/0x100
> [Mon Mar 11 14:10:08 2024] ? __pfx_kthread+0x10/0x10
> [Mon Mar 11 14:10:08 2024] ret_from_fork+0x29/0x50
> [Mon Mar 11 14:10:08 2024] </TASK>
>
The above threads are trying to flush the workqueue, so that probably
means that they are stuck waiting on a workqueue job to finish.
>
> The above is repeated a few times, and then this warning is also logged:
>
> [Mon Mar 11 14:12:04 2024] ------------[ cut here ]------------
> [Mon Mar 11 14:12:04 2024] WARNING: CPU: 39 PID: 8844 at fs/nfsd/nfs4state.c:4919 nfsd_break_deleg_cb+0x170/0x190 [nfsd]
> [Mon Mar 11 14:12:05 2024] Modules linked in: nfsv4 dns_resolver nfs fscache netfs rpcsec_gss_krb5 rpcrdma rdma_cm iw_cm ib_cm ib_core binfmt_misc bonding tls rfkill nft_counter nft_ct
> nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_reject_inet nf_reject_ipv4 nf_reject_ipv6 nft_reject nf_tables nfnetlink vfat fat dm_thin_pool dm_persistent_data dm_bio_prison dm_bufio l
> ibcrc32c dm_service_time dm_multipath intel_rapl_msr intel_rapl_common intel_uncore_frequency intel_uncore_frequency_common isst_if_common skx_edac nfit libnvdimm ipmi_ssif x86_pkg_temp
> _thermal intel_powerclamp coretemp kvm_intel kvm irqbypass dcdbas rapl intel_cstate mgag200 i2c_algo_bit drm_shmem_helper dell_smbios drm_kms_helper dell_wmi_descriptor wmi_bmof intel_u
> ncore syscopyarea pcspkr sysfillrect mei_me sysimgblt acpi_ipmi mei fb_sys_fops i2c_i801 ipmi_si intel_pch_thermal lpc_ich ipmi_devintf i2c_smbus ipmi_msghandler joydev acpi_power_meter
> nfsd auth_rpcgss nfs_acl drm lockd grace fuse sunrpc ext4 mbcache jbd2 sd_mod sg lpfc
> [Mon Mar 11 14:12:05 2024] nvmet_fc nvmet nvme_fc nvme_fabrics crct10dif_pclmul ahci libahci crc32_pclmul nvme_core crc32c_intel ixgbe megaraid_sas libata nvme_common ghash_clmulni_int
> el t10_pi wdat_wdt scsi_transport_fc mdio wmi dca dm_mirror dm_region_hash dm_log dm_mod
> [Mon Mar 11 14:12:05 2024] CPU: 39 PID: 8844 Comm: nfsd Not tainted 5.14.0-419.el9.x86_64 #1
> [Mon Mar 11 14:12:05 2024] Hardware name: Dell Inc. PowerEdge R740/00WGD1, BIOS 2.20.1 09/13/2023
> [Mon Mar 11 14:12:05 2024] RIP: 0010:nfsd_break_deleg_cb+0x170/0x190 [nfsd]
> [Mon Mar 11 14:12:05 2024] Code: a6 95 c5 f3 e9 ff fe ff ff 48 89 df be 01 00 00 00 e8 34 b5 13 f4 48 8d bb 98 00 00 00 e8 c8 f9 00 00 84 c0 0f 85 2e ff ff ff <0f> 0b e9 27 ff ff ff be
> 02 00 00 00 48 89 df e8 0c b5 13 f4 e9 01
> [Mon Mar 11 14:12:05 2024] RSP: 0018:ffff9929e0bb7b80 EFLAGS: 00010246
> [Mon Mar 11 14:12:05 2024] RAX: 0000000000000000 RBX: ffff8ada51930900 RCX: 0000000000000024
> [Mon Mar 11 14:12:05 2024] RDX: ffff8ada519309c8 RSI: ffff8ad582933c00 RDI: 0000000000002000
> [Mon Mar 11 14:12:05 2024] RBP: ffff8ad46bf21574 R08: ffff9929e0bb7b48 R09: 0000000000000000
> [Mon Mar 11 14:12:05 2024] R10: ffff8aec859a2948 R11: 0000000000000000 R12: ffff8ad6f497c360
> [Mon Mar 11 14:12:05 2024] R13: ffff8ad46bf21560 R14: ffff8ae5942e0b10 R15: ffff8ad6f497c360
> [Mon Mar 11 14:12:05 2024] FS: 0000000000000000(0000) GS:ffff8b031fcc0000(0000) knlGS:0000000000000000
> [Mon Mar 11 14:12:05 2024] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> [Mon Mar 11 14:12:05 2024] CR2: 00007fafe2060744 CR3: 00000018e58de006 CR4: 00000000007706e0
> [Mon Mar 11 14:12:05 2024] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
> [Mon Mar 11 14:12:05 2024] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
> [Mon Mar 11 14:12:05 2024] PKRU: 55555554
> [Mon Mar 11 14:12:05 2024] Call Trace:
> [Mon Mar 11 14:12:05 2024] <TASK>
> [Mon Mar 11 14:12:05 2024] ? show_trace_log_lvl+0x1c4/0x2df
> [Mon Mar 11 14:12:05 2024] ? show_trace_log_lvl+0x1c4/0x2df
> [Mon Mar 11 14:12:05 2024] ? __break_lease+0x16f/0x5f0
> [Mon Mar 11 14:12:05 2024] ? nfsd_break_deleg_cb+0x170/0x190 [nfsd]
> [Mon Mar 11 14:12:05 2024] ? __warn+0x81/0x110
> [Mon Mar 11 14:12:05 2024] ? nfsd_break_deleg_cb+0x170/0x190 [nfsd]
> [Mon Mar 11 14:12:05 2024] ? report_bug+0x10a/0x140
> [Mon Mar 11 14:12:05 2024] ? handle_bug+0x3c/0x70
> [Mon Mar 11 14:12:05 2024] ? exc_invalid_op+0x14/0x70
> [Mon Mar 11 14:12:05 2024] ? asm_exc_invalid_op+0x16/0x20
> [Mon Mar 11 14:12:05 2024] ? nfsd_break_deleg_cb+0x170/0x190 [nfsd]
> [Mon Mar 11 14:12:05 2024] __break_lease+0x16f/0x5f0
> [Mon Mar 11 14:12:05 2024] ? nfsd_file_lookup_locked+0x117/0x160 [nfsd]
> [Mon Mar 11 14:12:05 2024] ? list_lru_del+0x101/0x150
> [Mon Mar 11 14:12:05 2024] nfsd_file_do_acquire+0x790/0x830 [nfsd]
> [Mon Mar 11 14:12:05 2024] nfs4_get_vfs_file+0x315/0x3a0 [nfsd]
> [Mon Mar 11 14:12:05 2024] nfsd4_process_open2+0x430/0xa30 [nfsd]
> [Mon Mar 11 14:12:05 2024] ? fh_verify+0x297/0x2f0 [nfsd]
> [Mon Mar 11 14:12:05 2024] nfsd4_open+0x3ce/0x4b0 [nfsd]
> [Mon Mar 11 14:12:05 2024] nfsd4_proc_compound+0x44b/0x700 [nfsd]
> [Mon Mar 11 14:12:05 2024] nfsd_dispatch+0x94/0x1c0 [nfsd]
> [Mon Mar 11 14:12:05 2024] svc_process_common+0x2ec/0x660 [sunrpc]
> [Mon Mar 11 14:12:05 2024] ? __pfx_nfsd_dispatch+0x10/0x10 [nfsd]
> [Mon Mar 11 14:12:05 2024] ? __pfx_nfsd+0x10/0x10 [nfsd]
> [Mon Mar 11 14:12:05 2024] svc_process+0x12d/0x170 [sunrpc]
> [Mon Mar 11 14:12:05 2024] nfsd+0x84/0xb0 [nfsd]
> [Mon Mar 11 14:12:05 2024] kthread+0xdd/0x100
> [Mon Mar 11 14:12:05 2024] ? __pfx_kthread+0x10/0x10
> [Mon Mar 11 14:12:05 2024] ret_from_fork+0x29/0x50
> [Mon Mar 11 14:12:05 2024] </TASK>
> [Mon Mar 11 14:12:05 2024] ---[ end trace 7a039e17443dc651 ]---
This is probably this WARN in nfsd_break_one_deleg:
WARN_ON_ONCE(!nfsd4_run_cb(&dp->dl_recall));
It means that a delegation break callback to the client couldn't be
queued to the workqueue, and so it didn't run.
>
> Could this be the same issue as described here:https://lore.kernel.org/linux-nfs/af0ec881-5ebf-4feb-98ae-3ed2a77f86f1@oracle.com/ ?
>
Yes, most likely the same problem.
> As described in that thread, I've tried to obtain the requested information.
>
>
> Is it possible this is the issue that was fixed by the patches described here? https://lore.kernel.org/linux-nfs/2024022054-cause-suffering-eae8@gregkh/
>
Doubtful. Those are targeted toward a different set of issues.
If you're willing, I do have some patches queued up for CentOS here that
fix some backchannel problems that could be related. I'm mainly waiting
on Chuck to send these to Linus and then we'll likely merge them into
CentOS soon afterward:
https://gitlab.com/redhat/centos-stream/src/kernel/centos-stream-9/-/merge_requests/3689
--
Jeff Layton <jlayton@poochiereds.net>
^ permalink raw reply [flat|nested] 26+ messages in thread
* nfsd hangs and nfsd_break_deleg_cb+0x170/0x190 warning
@ 2024-03-11 18:43 Rik Theys
2024-03-12 11:22 ` Jeff Layton
2024-03-12 11:37 ` Jeff Layton
0 siblings, 2 replies; 26+ messages in thread
From: Rik Theys @ 2024-03-11 18:43 UTC (permalink / raw)
To: Linux Nfs
[-- Attachment #1.1: Type: text/plain, Size: 10254 bytes --]
Hi,
Since a few weeks our Rocky Linux 9 NFS server has periodically logged
hung nfsd tasks. The initial effect was that some clients could no
longer access the NFS server. This got worse and worse (probably as more
nfsd threads got blocked) and we had to restart the server. Restarting
the server also failed as the NFS server service could no longer be stopped.
The initial kernel we noticed this behavior on was
kernel-5.14.0-362.18.1.el9_3.x86_64. Since then we've installed
kernel-5.14.0-419.el9.x86_64 from CentOS Stream 9. The same issue
happened again on this newer kernel version:
[Mon Mar 11 14:10:08 2024] Not tainted 5.14.0-419.el9.x86_64 #1
[Mon Mar 11 14:10:08 2024] "echo 0 >
/proc/sys/kernel/hung_task_timeout_secs" disables this message.
[Mon Mar 11 14:10:08 2024] task:nfsd state:D stack:0
pid:8865 ppid:2 flags:0x00004000
[Mon Mar 11 14:10:08 2024] Call Trace:
[Mon Mar 11 14:10:08 2024] <TASK>
[Mon Mar 11 14:10:08 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:10:08 2024] schedule+0x2d/0x70
[Mon Mar 11 14:10:08 2024] schedule_timeout+0x11f/0x160
[Mon Mar 11 14:10:08 2024] ? select_idle_sibling+0x28/0x430
[Mon Mar 11 14:10:08 2024] ? wake_affine+0x62/0x1f0
[Mon Mar 11 14:10:08 2024] __wait_for_common+0x90/0x1d0
[Mon Mar 11 14:10:08 2024] ? __pfx_schedule_timeout+0x10/0x10
[Mon Mar 11 14:10:08 2024] __flush_workqueue+0x13a/0x3f0
[Mon Mar 11 14:10:08 2024] nfsd4_shutdown_callback+0x49/0x120 [nfsd]
[Mon Mar 11 14:10:08 2024] ? nfsd4_cld_remove+0x54/0x1d0 [nfsd]
[Mon Mar 11 14:10:08 2024] ? nfsd4_return_all_client_layouts+0xc4/0xf0
[nfsd]
[Mon Mar 11 14:10:08 2024] ? nfsd4_shutdown_copy+0x68/0xc0 [nfsd]
[Mon Mar 11 14:10:08 2024] __destroy_client+0x1f3/0x290 [nfsd]
[Mon Mar 11 14:10:08 2024] nfsd4_exchange_id+0x75f/0x770 [nfsd]
[Mon Mar 11 14:10:08 2024] ? nfsd4_decode_opaque+0x3a/0x90 [nfsd]
[Mon Mar 11 14:10:08 2024] nfsd4_proc_compound+0x44b/0x700 [nfsd]
[Mon Mar 11 14:10:08 2024] nfsd_dispatch+0x94/0x1c0 [nfsd]
[Mon Mar 11 14:10:08 2024] svc_process_common+0x2ec/0x660 [sunrpc]
[Mon Mar 11 14:10:08 2024] ? __pfx_nfsd_dispatch+0x10/0x10 [nfsd]
[Mon Mar 11 14:10:08 2024] ? __pfx_nfsd+0x10/0x10 [nfsd]
[Mon Mar 11 14:10:08 2024] svc_process+0x12d/0x170 [sunrpc]
[Mon Mar 11 14:10:08 2024] nfsd+0x84/0xb0 [nfsd]
[Mon Mar 11 14:10:08 2024] kthread+0xdd/0x100
[Mon Mar 11 14:10:08 2024] ? __pfx_kthread+0x10/0x10
[Mon Mar 11 14:10:08 2024] ret_from_fork+0x29/0x50
[Mon Mar 11 14:10:08 2024] </TASK>
[Mon Mar 11 14:10:08 2024] INFO: task nfsd:8866 blocked for more than
122 seconds.
[Mon Mar 11 14:10:08 2024] Not tainted 5.14.0-419.el9.x86_64 #1
[Mon Mar 11 14:10:08 2024] "echo 0 >
/proc/sys/kernel/hung_task_timeout_secs" disables this message.
[Mon Mar 11 14:10:08 2024] task:nfsd state:D stack:0
pid:8866 ppid:2 flags:0x00004000
[Mon Mar 11 14:10:08 2024] Call Trace:
[Mon Mar 11 14:10:08 2024] <TASK>
[Mon Mar 11 14:10:08 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:10:08 2024] schedule+0x2d/0x70
[Mon Mar 11 14:10:08 2024] schedule_timeout+0x11f/0x160
[Mon Mar 11 14:10:08 2024] ? select_idle_sibling+0x28/0x430
[Mon Mar 11 14:10:08 2024] ? tcp_recvmsg+0x196/0x210
[Mon Mar 11 14:10:08 2024] ? wake_affine+0x62/0x1f0
[Mon Mar 11 14:10:08 2024] __wait_for_common+0x90/0x1d0
[Mon Mar 11 14:10:08 2024] ? __pfx_schedule_timeout+0x10/0x10
[Mon Mar 11 14:10:08 2024] __flush_workqueue+0x13a/0x3f0
[Mon Mar 11 14:10:08 2024] nfsd4_destroy_session+0x1a4/0x240 [nfsd]
[Mon Mar 11 14:10:08 2024] nfsd4_proc_compound+0x44b/0x700 [nfsd]
[Mon Mar 11 14:10:08 2024] nfsd_dispatch+0x94/0x1c0 [nfsd]
[Mon Mar 11 14:10:08 2024] svc_process_common+0x2ec/0x660 [sunrpc]
[Mon Mar 11 14:10:08 2024] ? __pfx_nfsd_dispatch+0x10/0x10 [nfsd]
[Mon Mar 11 14:10:08 2024] ? __pfx_nfsd+0x10/0x10 [nfsd]
[Mon Mar 11 14:10:08 2024] svc_process+0x12d/0x170 [sunrpc]
[Mon Mar 11 14:10:08 2024] nfsd+0x84/0xb0 [nfsd]
[Mon Mar 11 14:10:08 2024] kthread+0xdd/0x100
[Mon Mar 11 14:10:08 2024] ? __pfx_kthread+0x10/0x10
[Mon Mar 11 14:10:08 2024] ret_from_fork+0x29/0x50
[Mon Mar 11 14:10:08 2024] </TASK>
The above is repeated a few times, and then this warning is also logged:
[Mon Mar 11 14:12:04 2024] ------------[ cut here ]------------
[Mon Mar 11 14:12:04 2024] WARNING: CPU: 39 PID: 8844 at
fs/nfsd/nfs4state.c:4919 nfsd_break_deleg_cb+0x170/0x190 [nfsd]
[Mon Mar 11 14:12:05 2024] Modules linked in: nfsv4 dns_resolver nfs
fscache netfs rpcsec_gss_krb5 rpcrdma rdma_cm iw_cm ib_cm ib_core
binfmt_misc bonding tls rfkill nft_counter nft_ct
nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_reject_inet
nf_reject_ipv4 nf_reject_ipv6 nft_reject nf_tables nfnetlink vfat fat
dm_thin_pool dm_persistent_data dm_bio_prison dm_bufio l
ibcrc32c dm_service_time dm_multipath intel_rapl_msr intel_rapl_common
intel_uncore_frequency intel_uncore_frequency_common isst_if_common
skx_edac nfit libnvdimm ipmi_ssif x86_pkg_temp
_thermal intel_powerclamp coretemp kvm_intel kvm irqbypass dcdbas rapl
intel_cstate mgag200 i2c_algo_bit drm_shmem_helper dell_smbios
drm_kms_helper dell_wmi_descriptor wmi_bmof intel_u
ncore syscopyarea pcspkr sysfillrect mei_me sysimgblt acpi_ipmi mei
fb_sys_fops i2c_i801 ipmi_si intel_pch_thermal lpc_ich ipmi_devintf
i2c_smbus ipmi_msghandler joydev acpi_power_meter
nfsd auth_rpcgss nfs_acl drm lockd grace fuse sunrpc ext4 mbcache jbd2
sd_mod sg lpfc
[Mon Mar 11 14:12:05 2024] nvmet_fc nvmet nvme_fc nvme_fabrics
crct10dif_pclmul ahci libahci crc32_pclmul nvme_core crc32c_intel ixgbe
megaraid_sas libata nvme_common ghash_clmulni_int
el t10_pi wdat_wdt scsi_transport_fc mdio wmi dca dm_mirror
dm_region_hash dm_log dm_mod
[Mon Mar 11 14:12:05 2024] CPU: 39 PID: 8844 Comm: nfsd Not tainted
5.14.0-419.el9.x86_64 #1
[Mon Mar 11 14:12:05 2024] Hardware name: Dell Inc. PowerEdge
R740/00WGD1, BIOS 2.20.1 09/13/2023
[Mon Mar 11 14:12:05 2024] RIP: 0010:nfsd_break_deleg_cb+0x170/0x190 [nfsd]
[Mon Mar 11 14:12:05 2024] Code: a6 95 c5 f3 e9 ff fe ff ff 48 89 df be
01 00 00 00 e8 34 b5 13 f4 48 8d bb 98 00 00 00 e8 c8 f9 00 00 84 c0 0f
85 2e ff ff ff <0f> 0b e9 27 ff ff ff be
02 00 00 00 48 89 df e8 0c b5 13 f4 e9 01
[Mon Mar 11 14:12:05 2024] RSP: 0018:ffff9929e0bb7b80 EFLAGS: 00010246
[Mon Mar 11 14:12:05 2024] RAX: 0000000000000000 RBX: ffff8ada51930900
RCX: 0000000000000024
[Mon Mar 11 14:12:05 2024] RDX: ffff8ada519309c8 RSI: ffff8ad582933c00
RDI: 0000000000002000
[Mon Mar 11 14:12:05 2024] RBP: ffff8ad46bf21574 R08: ffff9929e0bb7b48
R09: 0000000000000000
[Mon Mar 11 14:12:05 2024] R10: ffff8aec859a2948 R11: 0000000000000000
R12: ffff8ad6f497c360
[Mon Mar 11 14:12:05 2024] R13: ffff8ad46bf21560 R14: ffff8ae5942e0b10
R15: ffff8ad6f497c360
[Mon Mar 11 14:12:05 2024] FS: 0000000000000000(0000)
GS:ffff8b031fcc0000(0000) knlGS:0000000000000000
[Mon Mar 11 14:12:05 2024] CS: 0010 DS: 0000 ES: 0000 CR0:
0000000080050033
[Mon Mar 11 14:12:05 2024] CR2: 00007fafe2060744 CR3: 00000018e58de006
CR4: 00000000007706e0
[Mon Mar 11 14:12:05 2024] DR0: 0000000000000000 DR1: 0000000000000000
DR2: 0000000000000000
[Mon Mar 11 14:12:05 2024] DR3: 0000000000000000 DR6: 00000000fffe0ff0
DR7: 0000000000000400
[Mon Mar 11 14:12:05 2024] PKRU: 55555554
[Mon Mar 11 14:12:05 2024] Call Trace:
[Mon Mar 11 14:12:05 2024] <TASK>
[Mon Mar 11 14:12:05 2024] ? show_trace_log_lvl+0x1c4/0x2df
[Mon Mar 11 14:12:05 2024] ? show_trace_log_lvl+0x1c4/0x2df
[Mon Mar 11 14:12:05 2024] ? __break_lease+0x16f/0x5f0
[Mon Mar 11 14:12:05 2024] ? nfsd_break_deleg_cb+0x170/0x190 [nfsd]
[Mon Mar 11 14:12:05 2024] ? __warn+0x81/0x110
[Mon Mar 11 14:12:05 2024] ? nfsd_break_deleg_cb+0x170/0x190 [nfsd]
[Mon Mar 11 14:12:05 2024] ? report_bug+0x10a/0x140
[Mon Mar 11 14:12:05 2024] ? handle_bug+0x3c/0x70
[Mon Mar 11 14:12:05 2024] ? exc_invalid_op+0x14/0x70
[Mon Mar 11 14:12:05 2024] ? asm_exc_invalid_op+0x16/0x20
[Mon Mar 11 14:12:05 2024] ? nfsd_break_deleg_cb+0x170/0x190 [nfsd]
[Mon Mar 11 14:12:05 2024] __break_lease+0x16f/0x5f0
[Mon Mar 11 14:12:05 2024] ? nfsd_file_lookup_locked+0x117/0x160 [nfsd]
[Mon Mar 11 14:12:05 2024] ? list_lru_del+0x101/0x150
[Mon Mar 11 14:12:05 2024] nfsd_file_do_acquire+0x790/0x830 [nfsd]
[Mon Mar 11 14:12:05 2024] nfs4_get_vfs_file+0x315/0x3a0 [nfsd]
[Mon Mar 11 14:12:05 2024] nfsd4_process_open2+0x430/0xa30 [nfsd]
[Mon Mar 11 14:12:05 2024] ? fh_verify+0x297/0x2f0 [nfsd]
[Mon Mar 11 14:12:05 2024] nfsd4_open+0x3ce/0x4b0 [nfsd]
[Mon Mar 11 14:12:05 2024] nfsd4_proc_compound+0x44b/0x700 [nfsd]
[Mon Mar 11 14:12:05 2024] nfsd_dispatch+0x94/0x1c0 [nfsd]
[Mon Mar 11 14:12:05 2024] svc_process_common+0x2ec/0x660 [sunrpc]
[Mon Mar 11 14:12:05 2024] ? __pfx_nfsd_dispatch+0x10/0x10 [nfsd]
[Mon Mar 11 14:12:05 2024] ? __pfx_nfsd+0x10/0x10 [nfsd]
[Mon Mar 11 14:12:05 2024] svc_process+0x12d/0x170 [sunrpc]
[Mon Mar 11 14:12:05 2024] nfsd+0x84/0xb0 [nfsd]
[Mon Mar 11 14:12:05 2024] kthread+0xdd/0x100
[Mon Mar 11 14:12:05 2024] ? __pfx_kthread+0x10/0x10
[Mon Mar 11 14:12:05 2024] ret_from_fork+0x29/0x50
[Mon Mar 11 14:12:05 2024] </TASK>
[Mon Mar 11 14:12:05 2024] ---[ end trace 7a039e17443dc651 ]---
Could this be the same issue as described here:
https://lore.kernel.org/linux-nfs/af0ec881-5ebf-4feb-98ae-3ed2a77f86f1@oracle.com/
?
As described in that thread, I've tried to obtain the requested information.
The attached workqueue_info.txt file contains the dmesg output after
running 'echo t > /proc/sysrq-trigger'. It's possibly truncated :-(.
I'm also attaching rpc_tasks.txt run on the server, and the
nfs_threads.txt file run on one of the clients that fails to mount the
server when the issue occurs.
Is it possible this is the issue that was fixed by the patches described
here?
https://lore.kernel.org/linux-nfs/2024022054-cause-suffering-eae8@gregkh/
Regards,
Rik
--
Rik Theys
System Engineer
KU Leuven - Dept. Elektrotechniek (ESAT)
Kasteelpark Arenberg 10 bus 2440 - B-3001 Leuven-Heverlee
+32(0)16/32.11.07
----------------------------------------------------------------
<<Any errors in spelling, tact or fact are transmission errors>>
[-- Attachment #1.2: Type: text/html, Size: 14668 bytes --]
[-- Attachment #2: workqueue_info.txt --]
[-- Type: text/plain, Size: 1280351 bytes --]
[Mon Mar 11 14:29:12 2024] task:ext4-rsv-conver state:I stack:0 pid:4327 ppid:2 flags:0x00004000
[Mon Mar 11 14:29:12 2024] Call Trace:
[Mon Mar 11 14:29:12 2024] <TASK>
[Mon Mar 11 14:29:12 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:12 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:12 2024] rescuer_thread+0x2ca/0x390
[Mon Mar 11 14:29:12 2024] ? __pfx_rescuer_thread+0x10/0x10
[Mon Mar 11 14:29:12 2024] kthread+0xdd/0x100
[Mon Mar 11 14:29:12 2024] ? __pfx_kthread+0x10/0x10
[Mon Mar 11 14:29:12 2024] ret_from_fork+0x29/0x50
[Mon Mar 11 14:29:12 2024] </TASK>
[Mon Mar 11 14:29:12 2024] task:jbd2/dm-88-8 state:S stack:0 pid:4330 ppid:2 flags:0x00004000
[Mon Mar 11 14:29:12 2024] Call Trace:
[Mon Mar 11 14:29:12 2024] <TASK>
[Mon Mar 11 14:29:12 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:12 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:12 2024] kjournald2+0x221/0x280 [jbd2]
[Mon Mar 11 14:29:12 2024] ? __pfx_autoremove_wake_function+0x10/0x10
[Mon Mar 11 14:29:12 2024] ? __pfx_kjournald2+0x10/0x10 [jbd2]
[Mon Mar 11 14:29:12 2024] kthread+0xdd/0x100
[Mon Mar 11 14:29:12 2024] ? __pfx_kthread+0x10/0x10
[Mon Mar 11 14:29:12 2024] ret_from_fork+0x29/0x50
[Mon Mar 11 14:29:12 2024] </TASK>
[Mon Mar 11 14:29:12 2024] task:ext4-rsv-conver state:I stack:0 pid:4331 ppid:2 flags:0x00004000
[Mon Mar 11 14:29:12 2024] Call Trace:
[Mon Mar 11 14:29:12 2024] <TASK>
[Mon Mar 11 14:29:12 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:12 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:12 2024] rescuer_thread+0x2ca/0x390
[Mon Mar 11 14:29:12 2024] ? __pfx_rescuer_thread+0x10/0x10
[Mon Mar 11 14:29:12 2024] kthread+0xdd/0x100
[Mon Mar 11 14:29:12 2024] ? __pfx_kthread+0x10/0x10
[Mon Mar 11 14:29:12 2024] ret_from_fork+0x29/0x50
[Mon Mar 11 14:29:12 2024] </TASK>
[Mon Mar 11 14:29:12 2024] task:sssd_be state:S stack:0 pid:4348 ppid:4316 flags:0x00000002
[Mon Mar 11 14:29:12 2024] Call Trace:
[Mon Mar 11 14:29:12 2024] <TASK>
[Mon Mar 11 14:29:12 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:12 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:12 2024] schedule_hrtimeout_range_clock+0x95/0x120
[Mon Mar 11 14:29:12 2024] ? __pfx_hrtimer_wakeup+0x10/0x10
[Mon Mar 11 14:29:12 2024] ep_poll+0x348/0x3b0
[Mon Mar 11 14:29:12 2024] ? __pfx_autoremove_wake_function+0x10/0x10
[Mon Mar 11 14:29:12 2024] do_epoll_wait+0xb1/0xd0
[Mon Mar 11 14:29:12 2024] __x64_sys_epoll_wait+0x60/0x100
[Mon Mar 11 14:29:12 2024] do_syscall_64+0x59/0x90
[Mon Mar 11 14:29:12 2024] ? syscall_exit_work+0x103/0x130
[Mon Mar 11 14:29:12 2024] ? syscall_exit_to_user_mode+0x22/0x40
[Mon Mar 11 14:29:12 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:12 2024] ? syscall_exit_work+0x103/0x130
[Mon Mar 11 14:29:12 2024] ? syscall_exit_to_user_mode+0x22/0x40
[Mon Mar 11 14:29:12 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:12 2024] ? sysvec_apic_timer_interrupt+0x3c/0x90
[Mon Mar 11 14:29:12 2024] entry_SYSCALL_64_after_hwframe+0x72/0xdc
[Mon Mar 11 14:29:12 2024] RIP: 0033:0x7f410254e80a
[Mon Mar 11 14:29:12 2024] RSP: 002b:00007fffca4884f8 EFLAGS: 00000246 ORIG_RAX: 00000000000000e8
[Mon Mar 11 14:29:12 2024] RAX: ffffffffffffffda RBX: 000055a3030565f0 RCX: 00007f410254e80a
[Mon Mar 11 14:29:12 2024] RDX: 0000000000000001 RSI: 00007fffca48852c RDI: 0000000000000003
[Mon Mar 11 14:29:12 2024] RBP: 000055a303056460 R08: 0000000000078fa6 R09: 0000000000000000
[Mon Mar 11 14:29:12 2024] R10: 0000000000001190 R11: 0000000000000246 R12: 000055a303056740
[Mon Mar 11 14:29:12 2024] R13: 0000000000001190 R14: 000055a302bbf2f8 R15: 00007f410294b000
[Mon Mar 11 14:29:12 2024] </TASK>
[Mon Mar 11 14:29:12 2024] task:NetworkManager state:S stack:0 pid:4349 ppid:1 flags:0x00000002
[Mon Mar 11 14:29:12 2024] Call Trace:
[Mon Mar 11 14:29:12 2024] <TASK>
[Mon Mar 11 14:29:12 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:12 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:12 2024] schedule_hrtimeout_range_clock+0x10f/0x120
[Mon Mar 11 14:29:12 2024] ? __ep_eventpoll_poll.isra.0+0x143/0x170
[Mon Mar 11 14:29:12 2024] do_poll.constprop.0+0x248/0x390
[Mon Mar 11 14:29:12 2024] ? poll_freewait+0x45/0xa0
[Mon Mar 11 14:29:12 2024] do_sys_poll+0x1c8/0x250
[Mon Mar 11 14:29:12 2024] ? __skb_try_recv_datagram+0xb4/0x190
[Mon Mar 11 14:29:12 2024] ? __skb_recv_datagram+0x85/0xc0
[Mon Mar 11 14:29:12 2024] ? __pfx_pollwake+0x10/0x10
[Mon Mar 11 14:29:12 2024] ? __pfx_pollwake+0x10/0x10
[Mon Mar 11 14:29:12 2024] ? __pfx_pollwake+0x10/0x10
[Mon Mar 11 14:29:12 2024] ? __pfx_pollwake+0x10/0x10
[Mon Mar 11 14:29:12 2024] ? __pfx_pollwake+0x10/0x10
[Mon Mar 11 14:29:12 2024] ? __pfx_pollwake+0x10/0x10
[Mon Mar 11 14:29:12 2024] ? __pfx_pollwake+0x10/0x10
[Mon Mar 11 14:29:12 2024] ? __pfx_pollwake+0x10/0x10
[Mon Mar 11 14:29:12 2024] ? audit_reset_context.part.0.constprop.0+0x273/0x2e0
[Mon Mar 11 14:29:12 2024] ? auditd_test_task+0x3c/0x50
[Mon Mar 11 14:29:12 2024] __x64_sys_poll+0x39/0x140
[Mon Mar 11 14:29:12 2024] ? syscall_trace_enter.constprop.0+0x126/0x1a0
[Mon Mar 11 14:29:12 2024] do_syscall_64+0x59/0x90
[Mon Mar 11 14:29:12 2024] ? audit_reset_context.part.0.constprop.0+0x273/0x2e0
[Mon Mar 11 14:29:12 2024] ? syscall_exit_work+0x103/0x130
[Mon Mar 11 14:29:12 2024] ? syscall_exit_to_user_mode+0x22/0x40
[Mon Mar 11 14:29:12 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:12 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:12 2024] entry_SYSCALL_64_after_hwframe+0x72/0xdc
[Mon Mar 11 14:29:12 2024] RIP: 0033:0x7f8b675426ff
[Mon Mar 11 14:29:12 2024] RSP: 002b:00007fff32a22790 EFLAGS: 00000293 ORIG_RAX: 0000000000000007
[Mon Mar 11 14:29:12 2024] RAX: ffffffffffffffda RBX: 00007f8b677c7071 RCX: 00007f8b675426ff
[Mon Mar 11 14:29:12 2024] RDX: 00000000ffffffff RSI: 0000000000000008 RDI: 00005561c946f2d0
[Mon Mar 11 14:29:12 2024] RBP: 00005561c946f2d0 R08: 0000000000000000 R09: 00007fff32a22620
[Mon Mar 11 14:29:12 2024] R10: 00007fff32b28080 R11: 0000000000000293 R12: 0000000000000008
[Mon Mar 11 14:29:12 2024] R13: 0000000000000008 R14: 00007fff32a22800 R15: 00005561c939a670
[Mon Mar 11 14:29:12 2024] </TASK>
[Mon Mar 11 14:29:12 2024] task:gmain state:S stack:0 pid:4355 ppid:1 flags:0x00000002
[Mon Mar 11 14:29:12 2024] Call Trace:
[Mon Mar 11 14:29:12 2024] <TASK>
[Mon Mar 11 14:29:12 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:12 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:12 2024] schedule_hrtimeout_range_clock+0x95/0x120
[Mon Mar 11 14:29:12 2024] ? __pfx_hrtimer_wakeup+0x10/0x10
[Mon Mar 11 14:29:12 2024] do_poll.constprop.0+0x248/0x390
[Mon Mar 11 14:29:12 2024] do_sys_poll+0x1c8/0x250
[Mon Mar 11 14:29:12 2024] ? avc_has_perm_noaudit+0x94/0x110
[Mon Mar 11 14:29:12 2024] ? avc_has_perm_noaudit+0x94/0x110
[Mon Mar 11 14:29:12 2024] ? __pfx_pollwake+0x10/0x10
[Mon Mar 11 14:29:12 2024] ? __pfx_pollwake+0x10/0x10
[Mon Mar 11 14:29:12 2024] ? path_lookupat+0xa2/0x1c0
[Mon Mar 11 14:29:12 2024] ? filename_lookup+0xcf/0x1d0
[Mon Mar 11 14:29:12 2024] ? audit_filter_rules.constprop.0+0x2c5/0xd30
[Mon Mar 11 14:29:12 2024] ? __audit_filter_op+0xa5/0xf0
[Mon Mar 11 14:29:12 2024] ? __audit_filter_op+0xa5/0xf0
[Mon Mar 11 14:29:12 2024] ? mntput_no_expire+0x4a/0x250
[Mon Mar 11 14:29:12 2024] ? audit_reset_context.part.0.constprop.0+0x273/0x2e0
[Mon Mar 11 14:29:12 2024] ? ktime_get_ts64+0x49/0xf0
[Mon Mar 11 14:29:12 2024] __x64_sys_poll+0xa6/0x140
[Mon Mar 11 14:29:12 2024] do_syscall_64+0x59/0x90
[Mon Mar 11 14:29:12 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:12 2024] ? switch_fpu_return+0x4c/0xd0
[Mon Mar 11 14:29:12 2024] ? exit_to_user_mode_prepare+0xec/0x100
[Mon Mar 11 14:29:12 2024] ? syscall_exit_to_user_mode+0x22/0x40
[Mon Mar 11 14:29:12 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:12 2024] ? syscall_exit_to_user_mode+0x22/0x40
[Mon Mar 11 14:29:12 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:12 2024] entry_SYSCALL_64_after_hwframe+0x72/0xdc
[Mon Mar 11 14:29:12 2024] RIP: 0033:0x7f8b675426ff
[Mon Mar 11 14:29:12 2024] RSP: 002b:00007f8b661a7f80 EFLAGS: 00000293 ORIG_RAX: 0000000000000007
[Mon Mar 11 14:29:12 2024] RAX: ffffffffffffffda RBX: 00007f8b677c7071 RCX: 00007f8b675426ff
[Mon Mar 11 14:29:12 2024] RDX: 0000000000000f9d RSI: 0000000000000002 RDI: 00005561c939a460
[Mon Mar 11 14:29:12 2024] RBP: 00005561c939a460 R08: 0000000000000000 R09: 0000000000000000
[Mon Mar 11 14:29:12 2024] R10: 00007fff32b28080 R11: 0000000000000293 R12: 0000000000000002
[Mon Mar 11 14:29:12 2024] R13: 0000000000000002 R14: 00007f8b661a7ff0 R15: 00005561c939a7a0
[Mon Mar 11 14:29:12 2024] </TASK>
[Mon Mar 11 14:29:12 2024] task:gdbus state:S stack:0 pid:4359 ppid:1 flags:0x00000002
[Mon Mar 11 14:29:12 2024] Call Trace:
[Mon Mar 11 14:29:12 2024] <TASK>
[Mon Mar 11 14:29:12 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:12 2024] ? __wake_up_common+0x75/0xa0
[Mon Mar 11 14:29:12 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:12 2024] schedule_hrtimeout_range_clock+0x10f/0x120
[Mon Mar 11 14:29:12 2024] ? add_wait_queue+0x65/0xa0
[Mon Mar 11 14:29:12 2024] ? unix_poll+0xf4/0x100
[Mon Mar 11 14:29:12 2024] ? sock_poll+0x4c/0xe0
[Mon Mar 11 14:29:12 2024] do_poll.constprop.0+0x248/0x390
[Mon Mar 11 14:29:12 2024] do_sys_poll+0x1c8/0x250
[Mon Mar 11 14:29:12 2024] ? unix_stream_recvmsg+0x92/0xa0
[Mon Mar 11 14:29:12 2024] ? __pfx_unix_stream_read_actor+0x10/0x10
[Mon Mar 11 14:29:12 2024] ? select_idle_sibling+0x28/0x430
[Mon Mar 11 14:29:12 2024] ? __pfx_pollwake+0x10/0x10
[Mon Mar 11 14:29:12 2024] ? __pfx_pollwake+0x10/0x10
[Mon Mar 11 14:29:12 2024] ? __pfx_pollwake+0x10/0x10
[Mon Mar 11 14:29:12 2024] ? _copy_to_iter+0x7e/0x630
[Mon Mar 11 14:29:12 2024] ? eventfd_read+0xe2/0x2b0
[Mon Mar 11 14:29:12 2024] ? __audit_filter_op+0xa5/0xf0
[Mon Mar 11 14:29:12 2024] ? auditd_test_task+0x3c/0x50
[Mon Mar 11 14:29:12 2024] ? __audit_syscall_entry+0xef/0x140
[Mon Mar 11 14:29:12 2024] __x64_sys_poll+0x39/0x140
[Mon Mar 11 14:29:12 2024] ? syscall_trace_enter.constprop.0+0x126/0x1a0
[Mon Mar 11 14:29:12 2024] do_syscall_64+0x59/0x90
[Mon Mar 11 14:29:12 2024] ? __irq_exit_rcu+0x46/0xc0
[Mon Mar 11 14:29:12 2024] ? sysvec_apic_timer_interrupt+0x3c/0x90
[Mon Mar 11 14:29:12 2024] entry_SYSCALL_64_after_hwframe+0x72/0xdc
[Mon Mar 11 14:29:12 2024] RIP: 0033:0x7f8b675426ff
[Mon Mar 11 14:29:12 2024] RSP: 002b:00007f8b659a6f80 EFLAGS: 00000293 ORIG_RAX: 0000000000000007
[Mon Mar 11 14:29:12 2024] RAX: ffffffffffffffda RBX: 00007f8b677c7071 RCX: 00007f8b675426ff
[Mon Mar 11 14:29:12 2024] RDX: 00000000ffffffff RSI: 0000000000000003 RDI: 00007f8b58008c10
[Mon Mar 11 14:29:12 2024] RBP: 00007f8b58008c10 R08: 0000000000000000 R09: 00007f8b659a6e10
[Mon Mar 11 14:29:12 2024] R10: 00007fff32b28080 R11: 0000000000000293 R12: 0000000000000003
[Mon Mar 11 14:29:12 2024] R13: 0000000000000003 R14: 00007f8b659a6ff0 R15: 00005561c93b04b0
[Mon Mar 11 14:29:12 2024] </TASK>
[Mon Mar 11 14:29:12 2024] task:jbd2/dm-83-8 state:S stack:0 pid:4350 ppid:2 flags:0x00004000
[Mon Mar 11 14:29:12 2024] Call Trace:
[Mon Mar 11 14:29:12 2024] <TASK>
[Mon Mar 11 14:29:12 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:12 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:12 2024] kjournald2+0x221/0x280 [jbd2]
[Mon Mar 11 14:29:12 2024] ? __pfx_autoremove_wake_function+0x10/0x10
[Mon Mar 11 14:29:12 2024] ? __pfx_kjournald2+0x10/0x10 [jbd2]
[Mon Mar 11 14:29:12 2024] kthread+0xdd/0x100
[Mon Mar 11 14:29:12 2024] ? __pfx_kthread+0x10/0x10
[Mon Mar 11 14:29:12 2024] ret_from_fork+0x29/0x50
[Mon Mar 11 14:29:12 2024] </TASK>
[Mon Mar 11 14:29:12 2024] task:ext4-rsv-conver state:I stack:0 pid:4351 ppid:2 flags:0x00004000
[Mon Mar 11 14:29:12 2024] Call Trace:
[Mon Mar 11 14:29:12 2024] <TASK>
[Mon Mar 11 14:29:12 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:12 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:12 2024] rescuer_thread+0x2ca/0x390
[Mon Mar 11 14:29:12 2024] ? __pfx_rescuer_thread+0x10/0x10
[Mon Mar 11 14:29:12 2024] kthread+0xdd/0x100
[Mon Mar 11 14:29:12 2024] ? __pfx_kthread+0x10/0x10
[Mon Mar 11 14:29:12 2024] ret_from_fork+0x29/0x50
[Mon Mar 11 14:29:12 2024] </TASK>
[Mon Mar 11 14:29:12 2024] task:jbd2/dm-77-8 state:S stack:0 pid:4353 ppid:2 flags:0x00004000
[Mon Mar 11 14:29:12 2024] Call Trace:
[Mon Mar 11 14:29:12 2024] <TASK>
[Mon Mar 11 14:29:12 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:12 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:12 2024] kjournald2+0x221/0x280 [jbd2]
[Mon Mar 11 14:29:12 2024] ? __pfx_autoremove_wake_function+0x10/0x10
[Mon Mar 11 14:29:12 2024] ? __pfx_kjournald2+0x10/0x10 [jbd2]
[Mon Mar 11 14:29:12 2024] kthread+0xdd/0x100
[Mon Mar 11 14:29:12 2024] ? __pfx_kthread+0x10/0x10
[Mon Mar 11 14:29:12 2024] ret_from_fork+0x29/0x50
[Mon Mar 11 14:29:12 2024] </TASK>
[Mon Mar 11 14:29:12 2024] task:ext4-rsv-conver state:I stack:0 pid:4354 ppid:2 flags:0x00004000
[Mon Mar 11 14:29:12 2024] Call Trace:
[Mon Mar 11 14:29:12 2024] <TASK>
[Mon Mar 11 14:29:12 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:12 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:12 2024] rescuer_thread+0x2ca/0x390
[Mon Mar 11 14:29:12 2024] ? __pfx_rescuer_thread+0x10/0x10
[Mon Mar 11 14:29:12 2024] kthread+0xdd/0x100
[Mon Mar 11 14:29:12 2024] ? __pfx_kthread+0x10/0x10
[Mon Mar 11 14:29:12 2024] ret_from_fork+0x29/0x50
[Mon Mar 11 14:29:12 2024] </TASK>
[Mon Mar 11 14:29:12 2024] task:sssd_nss state:S stack:0 pid:4356 ppid:4316 flags:0x00000002
[Mon Mar 11 14:29:12 2024] Call Trace:
[Mon Mar 11 14:29:12 2024] <TASK>
[Mon Mar 11 14:29:12 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:12 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:12 2024] schedule_hrtimeout_range_clock+0x95/0x120
[Mon Mar 11 14:29:12 2024] ? __pfx_hrtimer_wakeup+0x10/0x10
[Mon Mar 11 14:29:12 2024] ep_poll+0x348/0x3b0
[Mon Mar 11 14:29:12 2024] ? __pfx_autoremove_wake_function+0x10/0x10
[Mon Mar 11 14:29:12 2024] do_epoll_wait+0xb1/0xd0
[Mon Mar 11 14:29:12 2024] __x64_sys_epoll_wait+0x60/0x100
[Mon Mar 11 14:29:12 2024] do_syscall_64+0x59/0x90
[Mon Mar 11 14:29:12 2024] ? audit_reset_context.part.0.constprop.0+0x273/0x2e0
[Mon Mar 11 14:29:12 2024] ? syscall_exit_work+0x103/0x130
[Mon Mar 11 14:29:12 2024] ? syscall_exit_to_user_mode+0x22/0x40
[Mon Mar 11 14:29:12 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:12 2024] ? sysvec_apic_timer_interrupt+0x3c/0x90
[Mon Mar 11 14:29:12 2024] entry_SYSCALL_64_after_hwframe+0x72/0xdc
[Mon Mar 11 14:29:12 2024] RIP: 0033:0x7f05d854e80a
[Mon Mar 11 14:29:12 2024] RSP: 002b:00007ffc788f8588 EFLAGS: 00000246 ORIG_RAX: 00000000000000e8
[Mon Mar 11 14:29:12 2024] RAX: ffffffffffffffda RBX: 000055bd5961e430 RCX: 00007f05d854e80a
[Mon Mar 11 14:29:12 2024] RDX: 0000000000000001 RSI: 00007ffc788f85bc RDI: 0000000000000003
[Mon Mar 11 14:29:12 2024] RBP: 000055bd5961e2a0 R08: 00000000000048ac R09: 0000000000000000
[Mon Mar 11 14:29:12 2024] R10: 0000000000000fb3 R11: 0000000000000246 R12: 000055bd5961e580
[Mon Mar 11 14:29:12 2024] R13: 0000000000000fb3 R14: 00007ffc788f87e0 R15: 000055bd59632e10
[Mon Mar 11 14:29:12 2024] </TASK>
[Mon Mar 11 14:29:12 2024] task:sssd_pam state:S stack:0 pid:4357 ppid:4316 flags:0x00000002
[Mon Mar 11 14:29:12 2024] Call Trace:
[Mon Mar 11 14:29:12 2024] <TASK>
[Mon Mar 11 14:29:12 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:12 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:12 2024] schedule_hrtimeout_range_clock+0x95/0x120
[Mon Mar 11 14:29:12 2024] ? __pfx_hrtimer_wakeup+0x10/0x10
[Mon Mar 11 14:29:12 2024] ep_poll+0x348/0x3b0
[Mon Mar 11 14:29:12 2024] ? __pfx_autoremove_wake_function+0x10/0x10
[Mon Mar 11 14:29:12 2024] do_epoll_wait+0xb1/0xd0
[Mon Mar 11 14:29:12 2024] __x64_sys_epoll_wait+0x60/0x100
[Mon Mar 11 14:29:12 2024] do_syscall_64+0x59/0x90
[Mon Mar 11 14:29:12 2024] ? exit_to_user_mode_prepare+0xec/0x100
[Mon Mar 11 14:29:12 2024] ? syscall_exit_to_user_mode+0x22/0x40
[Mon Mar 11 14:29:12 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:12 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:12 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:12 2024] ? sysvec_apic_timer_interrupt+0x3c/0x90
[Mon Mar 11 14:29:12 2024] entry_SYSCALL_64_after_hwframe+0x72/0xdc
[Mon Mar 11 14:29:12 2024] RIP: 0033:0x7fc84134e80a
[Mon Mar 11 14:29:12 2024] RSP: 002b:00007ffc7c7f4b48 EFLAGS: 00000246 ORIG_RAX: 00000000000000e8
[Mon Mar 11 14:29:12 2024] RAX: ffffffffffffffda RBX: 000055857c8fa430 RCX: 00007fc84134e80a
[Mon Mar 11 14:29:12 2024] RDX: 0000000000000001 RSI: 00007ffc7c7f4b7c RDI: 0000000000000005
[Mon Mar 11 14:29:12 2024] RBP: 000055857c8fa2a0 R08: 00000000000f4239 R09: 0000000000000000
[Mon Mar 11 14:29:12 2024] R10: 0000000000002710 R11: 0000000000000246 R12: 000055857c8fa580
[Mon Mar 11 14:29:12 2024] R13: 0000000000002710 R14: 000055857c90c100 R15: 000055857b0180e3
[Mon Mar 11 14:29:12 2024] </TASK>
[Mon Mar 11 14:29:12 2024] task:sssd_autofs state:S stack:0 pid:4358 ppid:4316 flags:0x00000002
[Mon Mar 11 14:29:12 2024] Call Trace:
[Mon Mar 11 14:29:12 2024] <TASK>
[Mon Mar 11 14:29:12 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:12 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:12 2024] schedule_hrtimeout_range_clock+0x95/0x120
[Mon Mar 11 14:29:12 2024] ? __pfx_hrtimer_wakeup+0x10/0x10
[Mon Mar 11 14:29:12 2024] ep_poll+0x348/0x3b0
[Mon Mar 11 14:29:12 2024] ? __pfx_autoremove_wake_function+0x10/0x10
[Mon Mar 11 14:29:12 2024] do_epoll_wait+0xb1/0xd0
[Mon Mar 11 14:29:12 2024] __x64_sys_epoll_wait+0x60/0x100
[Mon Mar 11 14:29:12 2024] do_syscall_64+0x59/0x90
[Mon Mar 11 14:29:12 2024] ? exit_to_user_mode_loop+0xd0/0x130
[Mon Mar 11 14:29:12 2024] ? exit_to_user_mode_prepare+0xb6/0x100
[Mon Mar 11 14:29:12 2024] ? syscall_exit_to_user_mode+0x22/0x40
[Mon Mar 11 14:29:12 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:12 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:12 2024] ? sysvec_apic_timer_interrupt+0x3c/0x90
[Mon Mar 11 14:29:12 2024] entry_SYSCALL_64_after_hwframe+0x72/0xdc
[Mon Mar 11 14:29:12 2024] RIP: 0033:0x7f7d58b4e80a
[Mon Mar 11 14:29:12 2024] RSP: 002b:00007ffcb26c3c28 EFLAGS: 00000246 ORIG_RAX: 00000000000000e8
[Mon Mar 11 14:29:12 2024] RAX: ffffffffffffffda RBX: 0000557364468430 RCX: 00007f7d58b4e80a
[Mon Mar 11 14:29:12 2024] RDX: 0000000000000001 RSI: 00007ffcb26c3c5c RDI: 0000000000000003
[Mon Mar 11 14:29:12 2024] RBP: 00005573644682a0 R08: 00000000000f423b R09: 0000000000000000
[Mon Mar 11 14:29:12 2024] R10: 0000000000002710 R11: 0000000000000246 R12: 0000557364468580
[Mon Mar 11 14:29:12 2024] R13: 0000000000002710 R14: 00007ffcb26c3de0 R15: 000055736447ca60
[Mon Mar 11 14:29:12 2024] </TASK>
[Mon Mar 11 14:29:12 2024] task:jbd2/dm-25-8 state:S stack:0 pid:4360 ppid:2 flags:0x00004000
[Mon Mar 11 14:29:12 2024] Call Trace:
[Mon Mar 11 14:29:12 2024] <TASK>
[Mon Mar 11 14:29:12 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:12 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:12 2024] kjournald2+0x221/0x280 [jbd2]
[Mon Mar 11 14:29:12 2024] ? __pfx_autoremove_wake_function+0x10/0x10
[Mon Mar 11 14:29:12 2024] ? __pfx_kjournald2+0x10/0x10 [jbd2]
[Mon Mar 11 14:29:12 2024] kthread+0xdd/0x100
[Mon Mar 11 14:29:12 2024] ? __pfx_kthread+0x10/0x10
[Mon Mar 11 14:29:12 2024] ret_from_fork+0x29/0x50
[Mon Mar 11 14:29:12 2024] </TASK>
[Mon Mar 11 14:29:12 2024] task:ext4-rsv-conver state:I stack:0 pid:4361 ppid:2 flags:0x00004000
[Mon Mar 11 14:29:12 2024] Call Trace:
[Mon Mar 11 14:29:12 2024] <TASK>
[Mon Mar 11 14:29:12 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:12 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:12 2024] rescuer_thread+0x2ca/0x390
[Mon Mar 11 14:29:12 2024] ? __pfx_rescuer_thread+0x10/0x10
[Mon Mar 11 14:29:12 2024] kthread+0xdd/0x100
[Mon Mar 11 14:29:12 2024] ? __pfx_kthread+0x10/0x10
[Mon Mar 11 14:29:12 2024] ret_from_fork+0x29/0x50
[Mon Mar 11 14:29:12 2024] </TASK>
[Mon Mar 11 14:29:12 2024] task:systemd-logind state:S stack:0 pid:4364 ppid:1 flags:0x00000002
[Mon Mar 11 14:29:12 2024] Call Trace:
[Mon Mar 11 14:29:12 2024] <TASK>
[Mon Mar 11 14:29:12 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:12 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:12 2024] schedule_hrtimeout_range_clock+0x10f/0x120
[Mon Mar 11 14:29:12 2024] ? ep_send_events+0x272/0x2c0
[Mon Mar 11 14:29:12 2024] ep_poll+0x348/0x3b0
[Mon Mar 11 14:29:12 2024] ? __pfx_autoremove_wake_function+0x10/0x10
[Mon Mar 11 14:29:12 2024] do_epoll_wait+0xb1/0xd0
[Mon Mar 11 14:29:12 2024] __x64_sys_epoll_wait+0x60/0x100
[Mon Mar 11 14:29:12 2024] ? syscall_trace_enter.constprop.0+0x126/0x1a0
[Mon Mar 11 14:29:12 2024] do_syscall_64+0x59/0x90
[Mon Mar 11 14:29:12 2024] ? syscall_exit_work+0x103/0x130
[Mon Mar 11 14:29:12 2024] ? syscall_exit_to_user_mode+0x22/0x40
[Mon Mar 11 14:29:12 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:12 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:12 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:12 2024] ? sysvec_apic_timer_interrupt+0x3c/0x90
[Mon Mar 11 14:29:12 2024] entry_SYSCALL_64_after_hwframe+0x72/0xdc
[Mon Mar 11 14:29:12 2024] RIP: 0033:0x7f0f6c34e84e
[Mon Mar 11 14:29:12 2024] RSP: 002b:00007fff450edf80 EFLAGS: 00000293 ORIG_RAX: 00000000000000e8
[Mon Mar 11 14:29:12 2024] RAX: ffffffffffffffda RBX: 0000556409c24a00 RCX: 00007f0f6c34e84e
[Mon Mar 11 14:29:12 2024] RDX: 0000000000000022 RSI: 0000556409c33d10 RDI: 0000000000000004
[Mon Mar 11 14:29:12 2024] RBP: 0000556409c24b90 R08: 0000000000000000 R09: 1e84d9f954ce397b
[Mon Mar 11 14:29:12 2024] R10: 00000000ffffffff R11: 0000000000000293 R12: 00000000000000b4
[Mon Mar 11 14:29:12 2024] R13: 0000556409c24a00 R14: 0000000000000022 R15: 0000000000000012
[Mon Mar 11 14:29:12 2024] </TASK>
[Mon Mar 11 14:29:12 2024] task:tls-strp state:I stack:0 pid:4391 ppid:2 flags:0x00004000
[Mon Mar 11 14:29:12 2024] Call Trace:
[Mon Mar 11 14:29:12 2024] <TASK>
[Mon Mar 11 14:29:12 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:12 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:12 2024] rescuer_thread+0x2ca/0x390
[Mon Mar 11 14:29:12 2024] ? __pfx_rescuer_thread+0x10/0x10
[Mon Mar 11 14:29:12 2024] kthread+0xdd/0x100
[Mon Mar 11 14:29:12 2024] ? __pfx_kthread+0x10/0x10
[Mon Mar 11 14:29:12 2024] ret_from_fork+0x29/0x50
[Mon Mar 11 14:29:12 2024] </TASK>
[Mon Mar 11 14:29:12 2024] task:bond0 state:I stack:0 pid:4392 ppid:2 flags:0x00004000
[Mon Mar 11 14:29:12 2024] Call Trace:
[Mon Mar 11 14:29:12 2024] <TASK>
[Mon Mar 11 14:29:12 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:12 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:12 2024] rescuer_thread+0x2ca/0x390
[Mon Mar 11 14:29:12 2024] ? __pfx_rescuer_thread+0x10/0x10
[Mon Mar 11 14:29:12 2024] kthread+0xdd/0x100
[Mon Mar 11 14:29:12 2024] ? __pfx_kthread+0x10/0x10
[Mon Mar 11 14:29:12 2024] ret_from_fork+0x29/0x50
[Mon Mar 11 14:29:12 2024] </TASK>
[Mon Mar 11 14:29:12 2024] task:cupsd state:S stack:0 pid:4399 ppid:1 flags:0x00000002
[Mon Mar 11 14:29:12 2024] Call Trace:
[Mon Mar 11 14:29:12 2024] <TASK>
[Mon Mar 11 14:29:12 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:12 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:12 2024] schedule_hrtimeout_range_clock+0x95/0x120
[Mon Mar 11 14:29:12 2024] ? __pfx_hrtimer_wakeup+0x10/0x10
[Mon Mar 11 14:29:12 2024] ep_poll+0x348/0x3b0
[Mon Mar 11 14:29:12 2024] ? __pfx_autoremove_wake_function+0x10/0x10
[Mon Mar 11 14:29:12 2024] do_epoll_wait+0xb1/0xd0
[Mon Mar 11 14:29:12 2024] __x64_sys_epoll_wait+0x60/0x100
[Mon Mar 11 14:29:12 2024] do_syscall_64+0x59/0x90
[Mon Mar 11 14:29:12 2024] ? audit_reset_context.part.0.constprop.0+0x273/0x2e0
[Mon Mar 11 14:29:12 2024] ? syscall_exit_work+0x103/0x130
[Mon Mar 11 14:29:12 2024] ? syscall_exit_to_user_mode+0x22/0x40
[Mon Mar 11 14:29:12 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:12 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:12 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:12 2024] ? sysvec_apic_timer_interrupt+0x3c/0x90
[Mon Mar 11 14:29:12 2024] entry_SYSCALL_64_after_hwframe+0x72/0xdc
[Mon Mar 11 14:29:12 2024] RIP: 0033:0x7f739954e80a
[Mon Mar 11 14:29:12 2024] RSP: 002b:00007ffd5b08db38 EFLAGS: 00000246 ORIG_RAX: 00000000000000e8
[Mon Mar 11 14:29:12 2024] RAX: ffffffffffffffda RBX: 000055ffd032cae6 RCX: 00007f739954e80a
[Mon Mar 11 14:29:12 2024] RDX: 0000000000080000 RSI: 00007f7397f46010 RDI: 0000000000000004
[Mon Mar 11 14:29:12 2024] RBP: 0000000000000000 R08: 000055ffd032cae6 R09: 0000000000000000
[Mon Mar 11 14:29:12 2024] R10: 00000000000dbba0 R11: 0000000000000246 R12: 0000000000000384
[Mon Mar 11 14:29:12 2024] R13: 0000000000000000 R14: 0000000065ef04e6 R15: 0000000000000383
[Mon Mar 11 14:29:12 2024] </TASK>
[Mon Mar 11 14:29:12 2024] task:uwsgi state:S stack:0 pid:4405 ppid:1 flags:0x00000002
[Mon Mar 11 14:29:12 2024] Call Trace:
[Mon Mar 11 14:29:12 2024] <TASK>
[Mon Mar 11 14:29:12 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:12 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:12 2024] schedule_hrtimeout_range_clock+0x95/0x120
[Mon Mar 11 14:29:12 2024] ? __pfx_hrtimer_wakeup+0x10/0x10
[Mon Mar 11 14:29:12 2024] ep_poll+0x348/0x3b0
[Mon Mar 11 14:29:12 2024] ? __pfx_autoremove_wake_function+0x10/0x10
[Mon Mar 11 14:29:12 2024] do_epoll_wait+0xb1/0xd0
[Mon Mar 11 14:29:12 2024] __x64_sys_epoll_wait+0x60/0x100
[Mon Mar 11 14:29:12 2024] do_syscall_64+0x59/0x90
[Mon Mar 11 14:29:12 2024] ? syscall_exit_work+0x103/0x130
[Mon Mar 11 14:29:12 2024] ? syscall_exit_to_user_mode+0x22/0x40
[Mon Mar 11 14:29:12 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:12 2024] ? syscall_exit_work+0x103/0x130
[Mon Mar 11 14:29:12 2024] ? syscall_exit_to_user_mode+0x22/0x40
[Mon Mar 11 14:29:12 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:12 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:12 2024] entry_SYSCALL_64_after_hwframe+0x72/0xdc
[Mon Mar 11 14:29:12 2024] RIP: 0033:0x7fb64274e80a
[Mon Mar 11 14:29:12 2024] RSP: 002b:00007fff2b12bbc8 EFLAGS: 00000246 ORIG_RAX: 00000000000000e8
[Mon Mar 11 14:29:12 2024] RAX: ffffffffffffffda RBX: 00007fff2b12bc2c RCX: 00007fb64274e80a
[Mon Mar 11 14:29:12 2024] RDX: 0000000000000001 RSI: 00007fff2b12bbd4 RDI: 0000000000000012
[Mon Mar 11 14:29:12 2024] RBP: 0000000000ad0040 R08: 0000000000000000 R09: 0000000000000000
[Mon Mar 11 14:29:12 2024] R10: 00000000000003e8 R11: 0000000000000246 R12: 0000000000000000
[Mon Mar 11 14:29:12 2024] R13: 00000000004e32b8 R14: 0000000000000000 R15: 000000000048ed5a
[Mon Mar 11 14:29:12 2024] </TASK>
[Mon Mar 11 14:29:12 2024] task:sshd state:S stack:0 pid:4412 ppid:1 flags:0x00000002
[Mon Mar 11 14:29:12 2024] Call Trace:
[Mon Mar 11 14:29:12 2024] <TASK>
[Mon Mar 11 14:29:12 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:12 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:12 2024] schedule_hrtimeout_range_clock+0x10f/0x120
[Mon Mar 11 14:29:12 2024] ? add_wait_queue+0x65/0xa0
[Mon Mar 11 14:29:12 2024] ? tcp_poll+0x1ce/0x370
[Mon Mar 11 14:29:12 2024] ? sock_poll+0x4c/0xe0
[Mon Mar 11 14:29:12 2024] do_select+0x69e/0x7c0
[Mon Mar 11 14:29:12 2024] ? __pfx_pollwake+0x10/0x10
[Mon Mar 11 14:29:12 2024] ? __pfx_pollwake+0x10/0x10
[Mon Mar 11 14:29:12 2024] ? __pfx_pollwake+0x10/0x10
[Mon Mar 11 14:29:12 2024] ? __mod_memcg_state+0x63/0xb0
[Mon Mar 11 14:29:12 2024] ? memcg_account_kmem+0x1e/0x60
[Mon Mar 11 14:29:12 2024] ? rmqueue_pcplist+0xda/0x210
[Mon Mar 11 14:29:12 2024] ? do_wp_page+0x381/0x540
[Mon Mar 11 14:29:12 2024] ? xas_load+0x9/0xa0
[Mon Mar 11 14:29:12 2024] ? xa_load+0x70/0xb0
[Mon Mar 11 14:29:12 2024] ? rcu_nocb_try_bypass+0x5e/0x460
[Mon Mar 11 14:29:12 2024] ? rcu_nocb_try_bypass+0x5e/0x460
[Mon Mar 11 14:29:12 2024] ? __pfx_inode_free_by_rcu+0x10/0x10
[Mon Mar 11 14:29:12 2024] ? fsnotify_grab_connector+0x49/0x80
[Mon Mar 11 14:29:12 2024] ? __pfx___d_free+0x10/0x10
[Mon Mar 11 14:29:12 2024] ? __call_rcu_common.constprop.0+0x117/0x2b0
[Mon Mar 11 14:29:12 2024] ? rcu_nocb_try_bypass+0x5e/0x460
[Mon Mar 11 14:29:12 2024] ? d_walk+0x1be/0x2a0
[Mon Mar 11 14:29:12 2024] ? rcu_nocb_try_bypass+0x5e/0x460
[Mon Mar 11 14:29:12 2024] ? __pfx_select_collect+0x10/0x10
[Mon Mar 11 14:29:12 2024] ? rcu_nocb_try_bypass+0x5e/0x460
[Mon Mar 11 14:29:12 2024] ? __pfx_inode_free_by_rcu+0x10/0x10
[Mon Mar 11 14:29:12 2024] core_sys_select+0x1a0/0x3b0
[Mon Mar 11 14:29:12 2024] ? free_unref_page_commit+0x7e/0x170
[Mon Mar 11 14:29:12 2024] ? rseq_get_rseq_cs+0x1d/0x240
[Mon Mar 11 14:29:12 2024] ? _copy_from_user+0x27/0x60
[Mon Mar 11 14:29:12 2024] do_pselect.constprop.0+0xca/0x170
[Mon Mar 11 14:29:12 2024] __x64_sys_pselect6+0x39/0x70
[Mon Mar 11 14:29:12 2024] do_syscall_64+0x59/0x90
[Mon Mar 11 14:29:12 2024] ? syscall_exit_to_user_mode+0x22/0x40
[Mon Mar 11 14:29:12 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:12 2024] ? syscall_exit_work+0x103/0x130
[Mon Mar 11 14:29:12 2024] ? syscall_exit_to_user_mode+0x22/0x40
[Mon Mar 11 14:29:12 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:12 2024] ? syscall_exit_to_user_mode+0x22/0x40
[Mon Mar 11 14:29:12 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:12 2024] ? exc_page_fault+0x62/0x150
[Mon Mar 11 14:29:12 2024] entry_SYSCALL_64_after_hwframe+0x72/0xdc
[Mon Mar 11 14:29:12 2024] RIP: 0033:0x7f3a07144f64
[Mon Mar 11 14:29:12 2024] RSP: 002b:00007ffdbce1f6b0 EFLAGS: 00000246 ORIG_RAX: 000000000000010e
[Mon Mar 11 14:29:12 2024] RAX: ffffffffffffffda RBX: 0000000000000010 RCX: 00007f3a07144f64
[Mon Mar 11 14:29:12 2024] RDX: 0000000000000000 RSI: 0000564a951d0e30 RDI: 0000000000000008
[Mon Mar 11 14:29:12 2024] RBP: 0000564a951d0e30 R08: 0000000000000000 R09: 00007ffdbce1f6f0
[Mon Mar 11 14:29:12 2024] R10: 0000000000000000 R11: 0000000000000246 R12: 0000564a936a6181
[Mon Mar 11 14:29:12 2024] R13: 00000000000002d8 R14: 0000000000000000 R15: 0000564a936da0b0
[Mon Mar 11 14:29:12 2024] </TASK>
[Mon Mar 11 14:29:12 2024] task:tuned state:S stack:0 pid:4413 ppid:1 flags:0x00000002
[Mon Mar 11 14:29:12 2024] Call Trace:
[Mon Mar 11 14:29:12 2024] <TASK>
[Mon Mar 11 14:29:12 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:12 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:12 2024] futex_wait_queue+0x70/0xd0
[Mon Mar 11 14:29:12 2024] futex_wait+0x175/0x260
[Mon Mar 11 14:29:12 2024] ? __pfx_hrtimer_wakeup+0x10/0x10
[Mon Mar 11 14:29:12 2024] do_futex+0x12d/0x1d0
[Mon Mar 11 14:29:12 2024] __x64_sys_futex+0x73/0x1d0
[Mon Mar 11 14:29:12 2024] do_syscall_64+0x59/0x90
[Mon Mar 11 14:29:12 2024] ? irqtime_account_irq+0x3c/0xb0
[Mon Mar 11 14:29:12 2024] ? __irq_exit_rcu+0x46/0xc0
[Mon Mar 11 14:29:12 2024] ? sysvec_apic_timer_interrupt+0x3c/0x90
[Mon Mar 11 14:29:12 2024] entry_SYSCALL_64_after_hwframe+0x72/0xdc
[Mon Mar 11 14:29:12 2024] RIP: 0033:0x7f211669c39a
[Mon Mar 11 14:29:12 2024] RSP: 002b:00007fff397bb840 EFLAGS: 00000246 ORIG_RAX: 00000000000000ca
[Mon Mar 11 14:29:12 2024] RAX: ffffffffffffffda RBX: 000055ee4b14a7d0 RCX: 00007f211669c39a
[Mon Mar 11 14:29:12 2024] RDX: 0000000000000000 RSI: 0000000000000089 RDI: 000055ee4b14a7d0
[Mon Mar 11 14:29:12 2024] RBP: 0000000000000000 R08: 0000000000000000 R09: 00000000ffffffff
[Mon Mar 11 14:29:12 2024] R10: 00007fff397bb900 R11: 0000000000000246 R12: 0000000000000000
[Mon Mar 11 14:29:12 2024] R13: 0000000000000001 R14: 00007fff397bb880 R15: fffffffeffffffff
[Mon Mar 11 14:29:12 2024] </TASK>
[Mon Mar 11 14:29:12 2024] task:tuned state:S stack:0 pid:4666 ppid:1 flags:0x00000002
[Mon Mar 11 14:29:12 2024] Call Trace:
[Mon Mar 11 14:29:12 2024] <TASK>
[Mon Mar 11 14:29:12 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:12 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:12 2024] futex_wait_queue+0x70/0xd0
[Mon Mar 11 14:29:12 2024] futex_wait+0x175/0x260
[Mon Mar 11 14:29:12 2024] ? __pfx_hrtimer_wakeup+0x10/0x10
[Mon Mar 11 14:29:12 2024] do_futex+0x12d/0x1d0
[Mon Mar 11 14:29:12 2024] __x64_sys_futex+0x73/0x1d0
[Mon Mar 11 14:29:12 2024] do_syscall_64+0x59/0x90
[Mon Mar 11 14:29:12 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:12 2024] ? syscall_exit_to_user_mode+0x22/0x40
[Mon Mar 11 14:29:12 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:12 2024] ? switch_fpu_return+0x4c/0xd0
[Mon Mar 11 14:29:12 2024] ? exit_to_user_mode_prepare+0xec/0x100
[Mon Mar 11 14:29:12 2024] ? syscall_exit_to_user_mode+0x22/0x40
[Mon Mar 11 14:29:12 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:12 2024] ? __irq_exit_rcu+0x46/0xc0
[Mon Mar 11 14:29:12 2024] ? sysvec_apic_timer_interrupt+0x3c/0x90
[Mon Mar 11 14:29:12 2024] entry_SYSCALL_64_after_hwframe+0x72/0xdc
[Mon Mar 11 14:29:12 2024] RIP: 0033:0x7f211669c39a
[Mon Mar 11 14:29:12 2024] RSP: 002b:00007f2115076f30 EFLAGS: 00000246 ORIG_RAX: 00000000000000ca
[Mon Mar 11 14:29:12 2024] RAX: ffffffffffffffda RBX: 00007f21100bfe30 RCX: 00007f211669c39a
[Mon Mar 11 14:29:12 2024] RDX: 0000000000000000 RSI: 0000000000000089 RDI: 00007f21100bfe30
[Mon Mar 11 14:29:12 2024] RBP: 0000000000000000 R08: 0000000000000000 R09: 00000000ffffffff
[Mon Mar 11 14:29:12 2024] R10: 00007f2115076ff0 R11: 0000000000000246 R12: 0000000000000000
[Mon Mar 11 14:29:12 2024] R13: 0000000000000001 R14: 00007f2115076f70 R15: fffffffeffffffff
[Mon Mar 11 14:29:12 2024] </TASK>
[Mon Mar 11 14:29:12 2024] task:tuned state:S stack:0 pid:5104 ppid:1 flags:0x00000002
[Mon Mar 11 14:29:12 2024] Call Trace:
[Mon Mar 11 14:29:12 2024] <TASK>
[Mon Mar 11 14:29:12 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:12 2024] ? finish_task_switch.isra.0+0x8c/0x2a0
[Mon Mar 11 14:29:12 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:12 2024] schedule_hrtimeout_range_clock+0x10f/0x120
[Mon Mar 11 14:29:12 2024] ? add_wait_queue+0x65/0xa0
[Mon Mar 11 14:29:12 2024] ? add_wait_queue+0x65/0xa0
[Mon Mar 11 14:29:12 2024] do_poll.constprop.0+0x248/0x390
[Mon Mar 11 14:29:12 2024] do_sys_poll+0x1c8/0x250
[Mon Mar 11 14:29:12 2024] ? avc_has_perm_noaudit+0x94/0x110
[Mon Mar 11 14:29:12 2024] ? avc_has_perm_noaudit+0x94/0x110
[Mon Mar 11 14:29:12 2024] ? xattr_find_entry+0x3e/0x140 [ext4]
[Mon Mar 11 14:29:12 2024] ? mutex_lock+0xe/0x30
[Mon Mar 11 14:29:12 2024] ? kernfs_vfs_xattr_get+0x42/0x70
[Mon Mar 11 14:29:12 2024] ? avc_has_perm_noaudit+0x94/0x110
[Mon Mar 11 14:29:12 2024] ? get_vfs_caps_from_disk+0x70/0x210
[Mon Mar 11 14:29:12 2024] ? __pfx_pollwake+0x10/0x10
[Mon Mar 11 14:29:12 2024] ? __pfx_pollwake+0x10/0x10
[Mon Mar 11 14:29:12 2024] ? terminate_walk+0x61/0xf0
[Mon Mar 11 14:29:12 2024] ? path_lookupat+0xa2/0x1c0
[Mon Mar 11 14:29:12 2024] ? filename_lookup+0xcf/0x1d0
[Mon Mar 11 14:29:12 2024] ? update_load_avg+0x7e/0x740
[Mon Mar 11 14:29:12 2024] ? pick_next_task_rt+0xbd/0x190
[Mon Mar 11 14:29:12 2024] ? pick_next_task+0x4d4/0x950
[Mon Mar 11 14:29:12 2024] ? rseq_get_rseq_cs+0x1d/0x240
[Mon Mar 11 14:29:12 2024] ? finish_task_switch.isra.0+0x8c/0x2a0
[Mon Mar 11 14:29:12 2024] ? rseq_ip_fixup+0x6e/0x1a0
[Mon Mar 11 14:29:12 2024] ? auditd_test_task+0x3c/0x50
[Mon Mar 11 14:29:12 2024] __x64_sys_poll+0x39/0x140
[Mon Mar 11 14:29:12 2024] ? syscall_trace_enter.constprop.0+0x126/0x1a0
[Mon Mar 11 14:29:12 2024] do_syscall_64+0x59/0x90
[Mon Mar 11 14:29:12 2024] ? exit_to_user_mode_prepare+0xec/0x100
[Mon Mar 11 14:29:12 2024] ? syscall_exit_to_user_mode+0x22/0x40
[Mon Mar 11 14:29:12 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:12 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:12 2024] ? syscall_exit_work+0x103/0x130
[Mon Mar 11 14:29:12 2024] ? syscall_exit_to_user_mode+0x22/0x40
[Mon Mar 11 14:29:12 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:12 2024] ? syscall_exit_to_user_mode+0x22/0x40
[Mon Mar 11 14:29:12 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:12 2024] ? syscall_exit_to_user_mode+0x22/0x40
[Mon Mar 11 14:29:12 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:12 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:12 2024] ? sysvec_apic_timer_interrupt+0x3c/0x90
[Mon Mar 11 14:29:12 2024] entry_SYSCALL_64_after_hwframe+0x72/0xdc
[Mon Mar 11 14:29:12 2024] RIP: 0033:0x7f21167426ff
[Mon Mar 11 14:29:12 2024] RSP: 002b:00007f21148360c0 EFLAGS: 00000293 ORIG_RAX: 0000000000000007
[Mon Mar 11 14:29:12 2024] RAX: ffffffffffffffda RBX: ffffffffffffffff RCX: 00007f21167426ff
[Mon Mar 11 14:29:12 2024] RDX: 00000000ffffffff RSI: 0000000000000002 RDI: 00007f21151afa30
[Mon Mar 11 14:29:12 2024] RBP: 00007f21148509e8 R08: 0000000000000000 R09: 0000000000000000
[Mon Mar 11 14:29:12 2024] R10: 0000000000000014 R11: 0000000000000293 R12: 00007f21148375c0
[Mon Mar 11 14:29:12 2024] R13: 00007f2110013d40 R14: 00007f211509bd80 R15: 00007f2114846c00
[Mon Mar 11 14:29:12 2024] </TASK>
[Mon Mar 11 14:29:12 2024] task:tuned state:S stack:0 pid:5263 ppid:1 flags:0x00000002
[Mon Mar 11 14:29:12 2024] Call Trace:
[Mon Mar 11 14:29:12 2024] <TASK>
[Mon Mar 11 14:29:12 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:12 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:12 2024] schedule_hrtimeout_range_clock+0x10f/0x120
[Mon Mar 11 14:29:12 2024] ? add_wait_queue+0x65/0xa0
[Mon Mar 11 14:29:12 2024] ? add_wait_queue+0x65/0xa0
[Mon Mar 11 14:29:12 2024] do_poll.constprop.0+0x248/0x390
[Mon Mar 11 14:29:12 2024] do_sys_poll+0x1c8/0x250
[Mon Mar 11 14:29:12 2024] ? ____sys_sendmsg+0x31c/0x340
[Mon Mar 11 14:29:12 2024] ? import_iovec+0x17/0x20
[Mon Mar 11 14:29:12 2024] ? copy_msghdr_from_user+0x6d/0xa0
[Mon Mar 11 14:29:12 2024] ? __pfx_pollwake+0x10/0x10
[Mon Mar 11 14:29:12 2024] ? __pfx_pollwake+0x10/0x10
[Mon Mar 11 14:29:12 2024] ? __pfx_pollwake+0x10/0x10
[Mon Mar 11 14:29:12 2024] ? eventfd_read+0xe2/0x2b0
[Mon Mar 11 14:29:12 2024] ? __audit_filter_op+0xa5/0xf0
[Mon Mar 11 14:29:12 2024] ? audit_reset_context.part.0.constprop.0+0x273/0x2e0
[Mon Mar 11 14:29:12 2024] ? syscall_exit_work+0x103/0x130
[Mon Mar 11 14:29:12 2024] ? auditd_test_task+0x3c/0x50
[Mon Mar 11 14:29:12 2024] __x64_sys_poll+0x39/0x140
[Mon Mar 11 14:29:12 2024] ? syscall_trace_enter.constprop.0+0x126/0x1a0
[Mon Mar 11 14:29:12 2024] do_syscall_64+0x59/0x90
[Mon Mar 11 14:29:12 2024] ? syscall_exit_to_user_mode+0x22/0x40
[Mon Mar 11 14:29:12 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:12 2024] entry_SYSCALL_64_after_hwframe+0x72/0xdc
[Mon Mar 11 14:29:12 2024] RIP: 0033:0x7f21167426ff
[Mon Mar 11 14:29:12 2024] RSP: 002b:00007f210fffe110 EFLAGS: 00000293 ORIG_RAX: 0000000000000007
[Mon Mar 11 14:29:12 2024] RAX: ffffffffffffffda RBX: 00007f21158da071 RCX: 00007f21167426ff
[Mon Mar 11 14:29:12 2024] RDX: 00000000ffffffff RSI: 0000000000000002 RDI: 00007f2104002f90
[Mon Mar 11 14:29:12 2024] RBP: 00007f2104002f90 R08: 0000000000000000 R09: 00007f210fffdfa0
[Mon Mar 11 14:29:12 2024] R10: 00007fff397de080 R11: 0000000000000293 R12: 0000000000000002
[Mon Mar 11 14:29:12 2024] R13: 0000000000000002 R14: 00007f210fffe180 R15: 000055ee4b144ba0
[Mon Mar 11 14:29:12 2024] </TASK>
[Mon Mar 11 14:29:12 2024] task:winbindd state:S stack:0 pid:4415 ppid:1 flags:0x00000002
[Mon Mar 11 14:29:12 2024] Call Trace:
[Mon Mar 11 14:29:12 2024] <TASK>
[Mon Mar 11 14:29:12 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:12 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:12 2024] schedule_hrtimeout_range_clock+0x95/0x120
[Mon Mar 11 14:29:12 2024] ? __pfx_hrtimer_wakeup+0x10/0x10
[Mon Mar 11 14:29:12 2024] ep_poll+0x348/0x3b0
[Mon Mar 11 14:29:12 2024] ? __pfx_autoremove_wake_function+0x10/0x10
[Mon Mar 11 14:29:12 2024] do_epoll_wait+0xb1/0xd0
[Mon Mar 11 14:29:12 2024] __x64_sys_epoll_wait+0x60/0x100
[Mon Mar 11 14:29:12 2024] do_syscall_64+0x59/0x90
[Mon Mar 11 14:29:12 2024] ? syscall_exit_work+0x103/0x130
[Mon Mar 11 14:29:12 2024] ? syscall_exit_to_user_mode+0x22/0x40
[Mon Mar 11 14:29:12 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:12 2024] ? sysvec_apic_timer_interrupt+0x3c/0x90
[Mon Mar 11 14:29:12 2024] entry_SYSCALL_64_after_hwframe+0x72/0xdc
[Mon Mar 11 14:29:12 2024] RIP: 0033:0x7f26a134e80a
[Mon Mar 11 14:29:12 2024] RSP: 002b:00007ffe9e3345d8 EFLAGS: 00000246 ORIG_RAX: 00000000000000e8
[Mon Mar 11 14:29:12 2024] RAX: ffffffffffffffda RBX: 00005586337521b0 RCX: 00007f26a134e80a
[Mon Mar 11 14:29:12 2024] RDX: 0000000000000001 RSI: 00007ffe9e33460c RDI: 0000000000000005
[Mon Mar 11 14:29:12 2024] RBP: 00005586337422e0 R08: 00000000000dbed4 R09: 000000000000007d
[Mon Mar 11 14:29:12 2024] R10: 000000000000076d R11: 0000000000000246 R12: 0000558633752240
[Mon Mar 11 14:29:12 2024] R13: 000000000000076d R14: 0000558632a15344 R15: 0000558633761ef0
[Mon Mar 11 14:29:12 2024] </TASK>
[Mon Mar 11 14:29:12 2024] task:fail2ban-server state:S stack:0 pid:4416 ppid:1 flags:0x00000002
[Mon Mar 11 14:29:12 2024] Call Trace:
[Mon Mar 11 14:29:12 2024] <TASK>
[Mon Mar 11 14:29:12 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:12 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:12 2024] schedule_hrtimeout_range_clock+0x95/0x120
[Mon Mar 11 14:29:12 2024] ? __pfx_hrtimer_wakeup+0x10/0x10
[Mon Mar 11 14:29:12 2024] do_select+0x69e/0x7c0
[Mon Mar 11 14:29:12 2024] ? __pfx_pollwake+0x10/0x10
[Mon Mar 11 14:29:12 2024] ? node_is_toptier+0x3e/0x60
[Mon Mar 11 14:29:12 2024] ? sysvec_apic_timer_interrupt+0xb/0x90
[Mon Mar 11 14:29:12 2024] ? asm_sysvec_apic_timer_interrupt+0x16/0x20
[Mon Mar 11 14:29:12 2024] ? __check_object_size.part.0+0x35/0xd0
[Mon Mar 11 14:29:12 2024] ? __check_object_size.part.0+0x35/0xd0
[Mon Mar 11 14:29:12 2024] ? __check_object_size.part.0+0x35/0xd0
[Mon Mar 11 14:29:12 2024] ? core_sys_select+0x1f7/0x3b0
[Mon Mar 11 14:29:12 2024] core_sys_select+0x1a0/0x3b0
[Mon Mar 11 14:29:12 2024] ? rseq_get_rseq_cs+0x1d/0x240
[Mon Mar 11 14:29:12 2024] ? rseq_ip_fixup+0x6e/0x1a0
[Mon Mar 11 14:29:12 2024] ? __audit_filter_op+0xa5/0xf0
[Mon Mar 11 14:29:12 2024] ? _copy_from_user+0x27/0x60
[Mon Mar 11 14:29:12 2024] do_pselect.constprop.0+0xca/0x170
[Mon Mar 11 14:29:12 2024] __x64_sys_pselect6+0x48/0x70
[Mon Mar 11 14:29:12 2024] do_syscall_64+0x59/0x90
[Mon Mar 11 14:29:12 2024] ? syscall_exit_to_user_mode+0x22/0x40
[Mon Mar 11 14:29:12 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:12 2024] ? sysvec_apic_timer_interrupt+0x3c/0x90
[Mon Mar 11 14:29:12 2024] entry_SYSCALL_64_after_hwframe+0x72/0xdc
[Mon Mar 11 14:29:12 2024] RIP: 0033:0x7f3bf4344e5d
[Mon Mar 11 14:29:12 2024] RSP: 002b:00007ffcf503fcf0 EFLAGS: 00000246 ORIG_RAX: 000000000000010e
[Mon Mar 11 14:29:12 2024] RAX: ffffffffffffffda RBX: 00007ffcf503fd90 RCX: 00007f3bf4344e5d
[Mon Mar 11 14:29:12 2024] RDX: 00007ffcf503fe20 RSI: 00007ffcf503fda0 RDI: 0000000000000005
[Mon Mar 11 14:29:12 2024] RBP: 00007ffcf503fda0 R08: 00007ffcf503fd00 R09: 0000000000000000
[Mon Mar 11 14:29:12 2024] R10: 00007ffcf503fea0 R11: 0000000000000246 R12: 0000000000000005
[Mon Mar 11 14:29:12 2024] R13: 00007ffcf503fd00 R14: 0000000000000000 R15: 00007ffcf503fe20
[Mon Mar 11 14:29:12 2024] </TASK>
[Mon Mar 11 14:29:12 2024] task:f2b/observer state:S stack:0 pid:4437 ppid:1 flags:0x00000002
[Mon Mar 11 14:29:12 2024] Call Trace:
[Mon Mar 11 14:29:12 2024] <TASK>
[Mon Mar 11 14:29:12 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:12 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:12 2024] futex_wait_queue+0x70/0xd0
[Mon Mar 11 14:29:12 2024] futex_wait+0x175/0x260
[Mon Mar 11 14:29:12 2024] ? __pfx_hrtimer_wakeup+0x10/0x10
[Mon Mar 11 14:29:12 2024] do_futex+0x12d/0x1d0
[Mon Mar 11 14:29:12 2024] __x64_sys_futex+0x73/0x1d0
[Mon Mar 11 14:29:12 2024] do_syscall_64+0x59/0x90
[Mon Mar 11 14:29:12 2024] ? __irq_exit_rcu+0x46/0xc0
[Mon Mar 11 14:29:12 2024] ? sysvec_apic_timer_interrupt+0x3c/0x90
[Mon Mar 11 14:29:12 2024] entry_SYSCALL_64_after_hwframe+0x72/0xdc
[Mon Mar 11 14:29:12 2024] RIP: 0033:0x7f3bf429c39a
[Mon Mar 11 14:29:12 2024] RSP: 002b:00007f3bf2d44cd0 EFLAGS: 00000246 ORIG_RAX: 00000000000000ca
[Mon Mar 11 14:29:12 2024] RAX: ffffffffffffffda RBX: 00007f3be40014b0 RCX: 00007f3bf429c39a
[Mon Mar 11 14:29:12 2024] RDX: 0000000000000000 RSI: 0000000000000089 RDI: 00007f3be40014b0
[Mon Mar 11 14:29:12 2024] RBP: 0000000000000000 R08: 0000000000000000 R09: 00000000ffffffff
[Mon Mar 11 14:29:12 2024] R10: 00007f3bf2d44d90 R11: 0000000000000246 R12: 0000000000000000
[Mon Mar 11 14:29:12 2024] R13: 0000000000000001 R14: 00007f3bf2d44d10 R15: fffffffeffffffff
[Mon Mar 11 14:29:12 2024] </TASK>
[Mon Mar 11 14:29:12 2024] task:f2b/f.samba state:S stack:0 pid:4439 ppid:1 flags:0x00000002
[Mon Mar 11 14:29:12 2024] Call Trace:
[Mon Mar 11 14:29:12 2024] <TASK>
[Mon Mar 11 14:29:12 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:12 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:12 2024] schedule_hrtimeout_range_clock+0x95/0x120
[Mon Mar 11 14:29:12 2024] ? __pfx_hrtimer_wakeup+0x10/0x10
[Mon Mar 11 14:29:12 2024] do_select+0x69e/0x7c0
[Mon Mar 11 14:29:12 2024] ? __pfx_hrtimer_wakeup+0x10/0x10
[Mon Mar 11 14:29:12 2024] ? do_select+0x719/0x7c0
[Mon Mar 11 14:29:12 2024] ? __find_get_block+0x1fb/0x370
[Mon Mar 11 14:29:12 2024] ? __ext4_get_inode_loc+0x112/0x4a0 [ext4]
[Mon Mar 11 14:29:12 2024] ? check_xattrs+0xe9/0x360 [ext4]
[Mon Mar 11 14:29:12 2024] ? xattr_find_entry+0x3e/0x140 [ext4]
[Mon Mar 11 14:29:12 2024] ? ext4_xattr_ibody_get+0x16e/0x1b0 [ext4]
[Mon Mar 11 14:29:12 2024] ? ext4_xattr_get+0x87/0xd0 [ext4]
[Mon Mar 11 14:29:12 2024] ? __vfs_getxattr+0x50/0x70
[Mon Mar 11 14:29:12 2024] ? get_vfs_caps_from_disk+0x70/0x210
[Mon Mar 11 14:29:12 2024] ? __legitimize_path+0x27/0x60
[Mon Mar 11 14:29:12 2024] ? audit_copy_inode+0x99/0xd0
[Mon Mar 11 14:29:12 2024] core_sys_select+0x1a0/0x3b0
[Mon Mar 11 14:29:12 2024] ? selinux_inode_getattr+0x99/0xc0
[Mon Mar 11 14:29:12 2024] ? _copy_to_user+0x1a/0x30
[Mon Mar 11 14:29:12 2024] ? cp_new_stat+0x150/0x180
[Mon Mar 11 14:29:12 2024] ? _copy_from_user+0x27/0x60
[Mon Mar 11 14:29:12 2024] do_pselect.constprop.0+0xca/0x170
[Mon Mar 11 14:29:12 2024] __x64_sys_pselect6+0x48/0x70
[Mon Mar 11 14:29:12 2024] do_syscall_64+0x59/0x90
[Mon Mar 11 14:29:12 2024] ? __rseq_handle_notify_resume+0x26/0xb0
[Mon Mar 11 14:29:12 2024] ? exit_to_user_mode_loop+0xd0/0x130
[Mon Mar 11 14:29:12 2024] ? exit_to_user_mode_prepare+0xec/0x100
[Mon Mar 11 14:29:12 2024] ? syscall_exit_to_user_mode+0x22/0x40
[Mon Mar 11 14:29:12 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:12 2024] ? clockevents_program_event+0x93/0x100
[Mon Mar 11 14:29:12 2024] ? hrtimer_interrupt+0x126/0x210
[Mon Mar 11 14:29:12 2024] ? sched_clock+0xc/0x30
[Mon Mar 11 14:29:12 2024] ? sched_clock_cpu+0x9/0xc0
[Mon Mar 11 14:29:12 2024] ? irqtime_account_irq+0x3c/0xb0
[Mon Mar 11 14:29:12 2024] ? __irq_exit_rcu+0x46/0xc0
[Mon Mar 11 14:29:12 2024] ? sysvec_apic_timer_interrupt+0x3c/0x90
[Mon Mar 11 14:29:12 2024] entry_SYSCALL_64_after_hwframe+0x72/0xdc
[Mon Mar 11 14:29:12 2024] RIP: 0033:0x7f3bf4344e5d
[Mon Mar 11 14:29:12 2024] RSP: 002b:00007f3bf1d02f40 EFLAGS: 00000246 ORIG_RAX: 000000000000010e
[Mon Mar 11 14:29:12 2024] RAX: ffffffffffffffda RBX: 00007f3bf1d02fc0 RCX: 00007f3bf4344e5d
[Mon Mar 11 14:29:12 2024] RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000000
[Mon Mar 11 14:29:12 2024] RBP: 0000000000000000 R08: 00007f3bf1d02f50 R09: 0000000000000000
[Mon Mar 11 14:29:12 2024] R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000000000
[Mon Mar 11 14:29:12 2024] R13: 00007f3bf1d02f50 R14: 0000000000000000 R15: 0000000000000000
[Mon Mar 11 14:29:12 2024] </TASK>
[Mon Mar 11 14:29:12 2024] task:f2b/a.samba state:S stack:0 pid:4440 ppid:1 flags:0x00000002
[Mon Mar 11 14:29:12 2024] Call Trace:
[Mon Mar 11 14:29:12 2024] <TASK>
[Mon Mar 11 14:29:12 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:12 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:12 2024] schedule_hrtimeout_range_clock+0x95/0x120
[Mon Mar 11 14:29:12 2024] ? __pfx_hrtimer_wakeup+0x10/0x10
[Mon Mar 11 14:29:12 2024] do_select+0x69e/0x7c0
[Mon Mar 11 14:29:12 2024] ? do_select+0x719/0x7c0
[Mon Mar 11 14:29:12 2024] ? task_numa_placement+0x3b7/0x4b0
[Mon Mar 11 14:29:12 2024] ? task_scan_max+0x12c/0x180
[Mon Mar 11 14:29:12 2024] ? sysvec_apic_timer_interrupt+0xb/0x90
[Mon Mar 11 14:29:12 2024] ? sysvec_apic_timer_interrupt+0xb/0x90
[Mon Mar 11 14:29:12 2024] ? asm_sysvec_apic_timer_interrupt+0x16/0x20
[Mon Mar 11 14:29:12 2024] ? update_sg_lb_stats+0x7e/0x450
[Mon Mar 11 14:29:12 2024] core_sys_select+0x1a0/0x3b0
[Mon Mar 11 14:29:12 2024] ? core_sys_select+0x1a0/0x3b0
[Mon Mar 11 14:29:12 2024] ? rseq_get_rseq_cs+0x1d/0x240
[Mon Mar 11 14:29:12 2024] ? rseq_ip_fixup+0x6e/0x1a0
[Mon Mar 11 14:29:12 2024] ? _copy_from_user+0x27/0x60
[Mon Mar 11 14:29:12 2024] do_pselect.constprop.0+0xca/0x170
[Mon Mar 11 14:29:12 2024] __x64_sys_pselect6+0x48/0x70
[Mon Mar 11 14:29:12 2024] do_syscall_64+0x59/0x90
[Mon Mar 11 14:29:12 2024] ? syscall_exit_to_user_mode+0x22/0x40
[Mon Mar 11 14:29:12 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:12 2024] ? switch_fpu_return+0x4c/0xd0
[Mon Mar 11 14:29:12 2024] ? exit_to_user_mode_prepare+0xec/0x100
[Mon Mar 11 14:29:12 2024] ? syscall_exit_to_user_mode+0x22/0x40
[Mon Mar 11 14:29:12 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:12 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:12 2024] ? sysvec_apic_timer_interrupt+0x3c/0x90
[Mon Mar 11 14:29:12 2024] entry_SYSCALL_64_after_hwframe+0x72/0xdc
[Mon Mar 11 14:29:12 2024] RIP: 0033:0x7f3bf4344e5d
[Mon Mar 11 14:29:12 2024] RSP: 002b:00007f3bf1501f40 EFLAGS: 00000246 ORIG_RAX: 000000000000010e
[Mon Mar 11 14:29:12 2024] RAX: ffffffffffffffda RBX: 00007f3bf1501fc0 RCX: 00007f3bf4344e5d
[Mon Mar 11 14:29:12 2024] RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000000
[Mon Mar 11 14:29:12 2024] RBP: 0000000000000000 R08: 00007f3bf1501f50 R09: 0000000000000000
[Mon Mar 11 14:29:12 2024] R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000000000
[Mon Mar 11 14:29:12 2024] R13: 00007f3bf1501f50 R14: 0000000000000000 R15: 0000000000000000
[Mon Mar 11 14:29:12 2024] </TASK>
[Mon Mar 11 14:29:12 2024] task:f2b/observer state:S stack:0 pid:2470448 ppid:1 flags:0x00000002
[Mon Mar 11 14:29:12 2024] Call Trace:
[Mon Mar 11 14:29:12 2024] <TASK>
[Mon Mar 11 14:29:12 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:12 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:12 2024] futex_wait_queue+0x70/0xd0
[Mon Mar 11 14:29:12 2024] futex_wait+0x175/0x260
[Mon Mar 11 14:29:12 2024] ? sched_clock+0xc/0x30
[Mon Mar 11 14:29:12 2024] ? __pfx_hrtimer_wakeup+0x10/0x10
[Mon Mar 11 14:29:12 2024] do_futex+0x12d/0x1d0
[Mon Mar 11 14:29:12 2024] __x64_sys_futex+0x73/0x1d0
[Mon Mar 11 14:29:12 2024] do_syscall_64+0x59/0x90
[Mon Mar 11 14:29:12 2024] ? audit_reset_context.part.0.constprop.0+0x273/0x2e0
[Mon Mar 11 14:29:12 2024] ? syscall_exit_work+0x103/0x130
[Mon Mar 11 14:29:12 2024] ? syscall_exit_to_user_mode+0x22/0x40
[Mon Mar 11 14:29:12 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:12 2024] entry_SYSCALL_64_after_hwframe+0x72/0xdc
[Mon Mar 11 14:29:12 2024] RIP: 0033:0x7f3bf429c39a
[Mon Mar 11 14:29:12 2024] RSP: 002b:00007f3bf2544290 EFLAGS: 00000246 ORIG_RAX: 00000000000000ca
[Mon Mar 11 14:29:12 2024] RAX: ffffffffffffffda RBX: 00007f3be8001120 RCX: 00007f3bf429c39a
[Mon Mar 11 14:29:12 2024] RDX: 0000000000000000 RSI: 0000000000000089 RDI: 00007f3be8001120
[Mon Mar 11 14:29:12 2024] RBP: 0000000000000000 R08: 0000000000000000 R09: 00000000ffffffff
[Mon Mar 11 14:29:12 2024] R10: 00007f3bf2544350 R11: 0000000000000246 R12: 0000000000000000
[Mon Mar 11 14:29:12 2024] R13: 0000000000000001 R14: 00007f3bf25442d0 R15: fffffffeffffffff
[Mon Mar 11 14:29:12 2024] </TASK>
[Mon Mar 11 14:29:12 2024] task:gssproxy state:S stack:0 pid:4420 ppid:1 flags:0x00000002
[Mon Mar 11 14:29:12 2024] Call Trace:
[Mon Mar 11 14:29:12 2024] <TASK>
[Mon Mar 11 14:29:12 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:12 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:12 2024] schedule_hrtimeout_range_clock+0x95/0x120
[Mon Mar 11 14:29:12 2024] ? __pfx_hrtimer_wakeup+0x10/0x10
[Mon Mar 11 14:29:12 2024] ep_poll+0x348/0x3b0
[Mon Mar 11 14:29:12 2024] ? __pfx_autoremove_wake_function+0x10/0x10
[Mon Mar 11 14:29:12 2024] do_epoll_wait+0xb1/0xd0
[Mon Mar 11 14:29:12 2024] __x64_sys_epoll_wait+0x60/0x100
[Mon Mar 11 14:29:12 2024] do_syscall_64+0x59/0x90
[Mon Mar 11 14:29:12 2024] ? syscall_exit_work+0x103/0x130
[Mon Mar 11 14:29:12 2024] ? syscall_exit_to_user_mode+0x22/0x40
[Mon Mar 11 14:29:12 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:12 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:12 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:12 2024] ? sysvec_apic_timer_interrupt+0x3c/0x90
[Mon Mar 11 14:29:12 2024] entry_SYSCALL_64_after_hwframe+0x72/0xdc
[Mon Mar 11 14:29:12 2024] RIP: 0033:0x7f5dde14e84e
[Mon Mar 11 14:29:12 2024] RSP: 002b:00007ffd94499d10 EFLAGS: 00000293 ORIG_RAX: 00000000000000e8
[Mon Mar 11 14:29:12 2024] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f5dde14e84e
[Mon Mar 11 14:29:12 2024] RDX: 0000000000000040 RSI: 00005584ab5fa210 RDI: 0000000000000005
[Mon Mar 11 14:29:12 2024] RBP: 00007f5dde246060 R08: 0000000000000000 R09: 0000000000000000
[Mon Mar 11 14:29:12 2024] R10: 000000000000e95f R11: 0000000000000293 R12: 00007f5dde246060
[Mon Mar 11 14:29:12 2024] R13: 00000000fffffffe R14: 000000000000000a R15: 0000000000000000
[Mon Mar 11 14:29:12 2024] </TASK>
[Mon Mar 11 14:29:12 2024] task:gssproxy state:S stack:0 pid:4425 ppid:1 flags:0x00000002
[Mon Mar 11 14:29:12 2024] Call Trace:
[Mon Mar 11 14:29:12 2024] <TASK>
[Mon Mar 11 14:29:12 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:12 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:12 2024] futex_wait_queue+0x70/0xd0
[Mon Mar 11 14:29:12 2024] futex_wait+0x175/0x260
[Mon Mar 11 14:29:12 2024] do_futex+0x12d/0x1d0
[Mon Mar 11 14:29:12 2024] __x64_sys_futex+0x73/0x1d0
[Mon Mar 11 14:29:12 2024] do_syscall_64+0x59/0x90
[Mon Mar 11 14:29:12 2024] ? switch_fpu_return+0x4c/0xd0
[Mon Mar 11 14:29:12 2024] ? exit_to_user_mode_prepare+0xec/0x100
[Mon Mar 11 14:29:12 2024] entry_SYSCALL_64_after_hwframe+0x72/0xdc
[Mon Mar 11 14:29:12 2024] RIP: 0033:0x7f5dde09c39a
[Mon Mar 11 14:29:12 2024] RSP: 002b:00007f5ddd5fe770 EFLAGS: 00000246 ORIG_RAX: 00000000000000ca
[Mon Mar 11 14:29:12 2024] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f5dde09c39a
[Mon Mar 11 14:29:12 2024] RDX: 0000000000000000 RSI: 0000000000000189 RDI: 00005584ab5fb4a8
[Mon Mar 11 14:29:12 2024] RBP: 0000000000000000 R08: 0000000000000000 R09: 00000000ffffffff
[Mon Mar 11 14:29:12 2024] R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000000000
[Mon Mar 11 14:29:12 2024] R13: 00005584ab5fb4a8 R14: 0000000000000000 R15: 0000000000000000
[Mon Mar 11 14:29:12 2024] </TASK>
[Mon Mar 11 14:29:12 2024] task:gssproxy state:S stack:0 pid:4426 ppid:1 flags:0x00000002
[Mon Mar 11 14:29:12 2024] Call Trace:
[Mon Mar 11 14:29:12 2024] <TASK>
[Mon Mar 11 14:29:12 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:12 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:12 2024] futex_wait_queue+0x70/0xd0
[Mon Mar 11 14:29:12 2024] futex_wait+0x175/0x260
[Mon Mar 11 14:29:12 2024] do_futex+0x12d/0x1d0
[Mon Mar 11 14:29:12 2024] __x64_sys_futex+0x73/0x1d0
[Mon Mar 11 14:29:12 2024] do_syscall_64+0x59/0x90
[Mon Mar 11 14:29:12 2024] ? exit_to_user_mode_prepare+0xec/0x100
[Mon Mar 11 14:29:12 2024] entry_SYSCALL_64_after_hwframe+0x72/0xdc
[Mon Mar 11 14:29:12 2024] RIP: 0033:0x7f5dde09c39a
[Mon Mar 11 14:29:12 2024] RSP: 002b:00007f5ddcdfd770 EFLAGS: 00000246 ORIG_RAX: 00000000000000ca
[Mon Mar 11 14:29:12 2024] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f5dde09c39a
[Mon Mar 11 14:29:12 2024] RDX: 0000000000000000 RSI: 0000000000000189 RDI: 00005584ab5fb698
[Mon Mar 11 14:29:12 2024] RBP: 0000000000000000 R08: 0000000000000000 R09: 00000000ffffffff
[Mon Mar 11 14:29:12 2024] R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000000000
[Mon Mar 11 14:29:12 2024] R13: 00005584ab5fb698 R14: 0000000000000000 R15: 0000000000000000
[Mon Mar 11 14:29:12 2024] </TASK>
[Mon Mar 11 14:29:12 2024] task:gssproxy state:S stack:0 pid:4427 ppid:1 flags:0x00000002
[Mon Mar 11 14:29:12 2024] Call Trace:
[Mon Mar 11 14:29:12 2024] <TASK>
[Mon Mar 11 14:29:12 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:12 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:12 2024] futex_wait_queue+0x70/0xd0
[Mon Mar 11 14:29:12 2024] futex_wait+0x175/0x260
[Mon Mar 11 14:29:12 2024] ? rseq_get_rseq_cs+0x1d/0x240
[Mon Mar 11 14:29:12 2024] ? rseq_ip_fixup+0x6e/0x1a0
[Mon Mar 11 14:29:12 2024] do_futex+0x12d/0x1d0
[Mon Mar 11 14:29:12 2024] __x64_sys_futex+0x73/0x1d0
[Mon Mar 11 14:29:12 2024] do_syscall_64+0x59/0x90
[Mon Mar 11 14:29:12 2024] ? audit_reset_context.part.0.constprop.0+0x273/0x2e0
[Mon Mar 11 14:29:12 2024] ? syscall_exit_work+0x103/0x130
[Mon Mar 11 14:29:12 2024] ? syscall_exit_to_user_mode+0x22/0x40
[Mon Mar 11 14:29:12 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:12 2024] ? exc_page_fault+0x62/0x150
[Mon Mar 11 14:29:12 2024] entry_SYSCALL_64_after_hwframe+0x72/0xdc
[Mon Mar 11 14:29:12 2024] RIP: 0033:0x7f5dde09c39a
[Mon Mar 11 14:29:12 2024] RSP: 002b:00007f5ddc5fc770 EFLAGS: 00000246 ORIG_RAX: 00000000000000ca
[Mon Mar 11 14:29:12 2024] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f5dde09c39a
[Mon Mar 11 14:29:12 2024] RDX: 0000000000000000 RSI: 0000000000000189 RDI: 00005584ab5fb888
[Mon Mar 11 14:29:12 2024] RBP: 0000000000000000 R08: 0000000000000000 R09: 00000000ffffffff
[Mon Mar 11 14:29:12 2024] R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000000000
[Mon Mar 11 14:29:12 2024] R13: 00005584ab5fb888 R14: 0000000000000000 R15: 0000000000000000
[Mon Mar 11 14:29:12 2024] </TASK>
[Mon Mar 11 14:29:12 2024] task:gssproxy state:S stack:0 pid:4428 ppid:1 flags:0x00000002
[Mon Mar 11 14:29:12 2024] Call Trace:
[Mon Mar 11 14:29:12 2024] <TASK>
[Mon Mar 11 14:29:12 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:12 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:12 2024] futex_wait_queue+0x70/0xd0
[Mon Mar 11 14:29:12 2024] futex_wait+0x175/0x260
[Mon Mar 11 14:29:12 2024] do_futex+0x12d/0x1d0
[Mon Mar 11 14:29:12 2024] __x64_sys_futex+0x73/0x1d0
[Mon Mar 11 14:29:12 2024] do_syscall_64+0x59/0x90
[Mon Mar 11 14:29:12 2024] ? exc_page_fault+0x62/0x150
[Mon Mar 11 14:29:12 2024] entry_SYSCALL_64_after_hwframe+0x72/0xdc
[Mon Mar 11 14:29:12 2024] RIP: 0033:0x7f5dde09c39a
[Mon Mar 11 14:29:12 2024] RSP: 002b:00007f5ddbdfb770 EFLAGS: 00000246 ORIG_RAX: 00000000000000ca
[Mon Mar 11 14:29:12 2024] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f5dde09c39a
[Mon Mar 11 14:29:12 2024] RDX: 0000000000000000 RSI: 0000000000000189 RDI: 00005584ab5fba78
[Mon Mar 11 14:29:12 2024] RBP: 0000000000000000 R08: 0000000000000000 R09: 00000000ffffffff
[Mon Mar 11 14:29:12 2024] R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000000000
[Mon Mar 11 14:29:12 2024] R13: 00005584ab5fba78 R14: 0000000000000000 R15: 0000000000000000
[Mon Mar 11 14:29:12 2024] </TASK>
[Mon Mar 11 14:29:12 2024] task:gssproxy state:S stack:0 pid:4429 ppid:1 flags:0x00000002
[Mon Mar 11 14:29:12 2024] Call Trace:
[Mon Mar 11 14:29:12 2024] <TASK>
[Mon Mar 11 14:29:12 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:12 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:12 2024] futex_wait_queue+0x70/0xd0
[Mon Mar 11 14:29:12 2024] futex_wait+0x175/0x260
[Mon Mar 11 14:29:12 2024] ? file_update_time+0xb4/0xd0
[Mon Mar 11 14:29:12 2024] ? pipe_write+0x122/0x650
[Mon Mar 11 14:29:12 2024] do_futex+0x12d/0x1d0
[Mon Mar 11 14:29:12 2024] __x64_sys_futex+0x73/0x1d0
[Mon Mar 11 14:29:12 2024] do_syscall_64+0x59/0x90
[Mon Mar 11 14:29:12 2024] ? __irq_exit_rcu+0x46/0xc0
[Mon Mar 11 14:29:12 2024] ? sysvec_apic_timer_interrupt+0x3c/0x90
[Mon Mar 11 14:29:12 2024] entry_SYSCALL_64_after_hwframe+0x72/0xdc
[Mon Mar 11 14:29:12 2024] RIP: 0033:0x7f5dde09c39a
[Mon Mar 11 14:29:12 2024] RSP: 002b:00007f5ddb5fa770 EFLAGS: 00000246 ORIG_RAX: 00000000000000ca
[Mon Mar 11 14:29:12 2024] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f5dde09c39a
[Mon Mar 11 14:29:12 2024] RDX: 0000000000000000 RSI: 0000000000000189 RDI: 00005584ab5fbc6c
[Mon Mar 11 14:29:12 2024] RBP: 0000000000000000 R08: 0000000000000000 R09: 00000000ffffffff
[Mon Mar 11 14:29:12 2024] R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000000000
[Mon Mar 11 14:29:12 2024] R13: 00005584ab5fbc6c R14: 0000000000000001 R15: 0000000000000000
[Mon Mar 11 14:29:12 2024] </TASK>
[Mon Mar 11 14:29:12 2024] task:rpc.gssd state:S stack:0 pid:4431 ppid:1 flags:0x00000002
[Mon Mar 11 14:29:12 2024] Call Trace:
[Mon Mar 11 14:29:12 2024] <TASK>
[Mon Mar 11 14:29:12 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:12 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:12 2024] schedule_hrtimeout_range_clock+0x10f/0x120
[Mon Mar 11 14:29:12 2024] ? ep_send_events+0x272/0x2c0
[Mon Mar 11 14:29:12 2024] ep_poll+0x348/0x3b0
[Mon Mar 11 14:29:12 2024] ? __pfx_autoremove_wake_function+0x10/0x10
[Mon Mar 11 14:29:12 2024] do_epoll_wait+0xb1/0xd0
[Mon Mar 11 14:29:12 2024] __x64_sys_epoll_wait+0x60/0x100
[Mon Mar 11 14:29:12 2024] ? syscall_trace_enter.constprop.0+0x126/0x1a0
[Mon Mar 11 14:29:12 2024] do_syscall_64+0x59/0x90
[Mon Mar 11 14:29:12 2024] ? syscall_exit_work+0x103/0x130
[Mon Mar 11 14:29:12 2024] ? syscall_exit_to_user_mode+0x22/0x40
[Mon Mar 11 14:29:12 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:12 2024] ? syscall_exit_to_user_mode+0x22/0x40
[Mon Mar 11 14:29:12 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:12 2024] ? syscall_exit_work+0x103/0x130
[Mon Mar 11 14:29:12 2024] ? syscall_exit_to_user_mode+0x22/0x40
[Mon Mar 11 14:29:12 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:12 2024] ? syscall_exit_to_user_mode+0x22/0x40
[Mon Mar 11 14:29:12 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:12 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:12 2024] ? sysvec_apic_timer_interrupt+0x3c/0x90
[Mon Mar 11 14:29:12 2024] entry_SYSCALL_64_after_hwframe+0x72/0xdc
[Mon Mar 11 14:29:12 2024] RIP: 0033:0x7efc96f4e84e
[Mon Mar 11 14:29:12 2024] RSP: 002b:00007ffcc24423d0 EFLAGS: 00000293 ORIG_RAX: 00000000000000e8
[Mon Mar 11 14:29:12 2024] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007efc96f4e84e
[Mon Mar 11 14:29:12 2024] RDX: 0000000000000020 RSI: 000055cc3a428990 RDI: 0000000000000005
[Mon Mar 11 14:29:12 2024] RBP: 000055cc3a4286f0 R08: 0000000000000000 R09: 00007efc96fb14e0
[Mon Mar 11 14:29:12 2024] R10: 00000000ffffffff R11: 0000000000000293 R12: 000055cc3a428990
[Mon Mar 11 14:29:12 2024] R13: 000055cc3a423730 R14: 000055cc3a423730 R15: 00000000ffffffff
[Mon Mar 11 14:29:12 2024] </TASK>
[Mon Mar 11 14:29:12 2024] task:rpc.gssd state:S stack:0 pid:4432 ppid:1 flags:0x00000002
[Mon Mar 11 14:29:12 2024] Call Trace:
[Mon Mar 11 14:29:12 2024] <TASK>
[Mon Mar 11 14:29:12 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:12 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:12 2024] do_nanosleep+0x67/0x190
[Mon Mar 11 14:29:12 2024] ? rseq_ip_fixup+0x6e/0x1a0
[Mon Mar 11 14:29:12 2024] hrtimer_nanosleep+0xbe/0x1a0
[Mon Mar 11 14:29:12 2024] ? __pfx_hrtimer_wakeup+0x10/0x10
[Mon Mar 11 14:29:12 2024] common_nsleep+0x40/0x50
[Mon Mar 11 14:29:12 2024] __x64_sys_clock_nanosleep+0xbc/0x130
[Mon Mar 11 14:29:12 2024] do_syscall_64+0x59/0x90
[Mon Mar 11 14:29:12 2024] ? syscall_exit_to_user_mode+0x22/0x40
[Mon Mar 11 14:29:12 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:12 2024] ? switch_fpu_return+0x4c/0xd0
[Mon Mar 11 14:29:12 2024] ? exit_to_user_mode_prepare+0xec/0x100
[Mon Mar 11 14:29:12 2024] ? syscall_exit_to_user_mode+0x22/0x40
[Mon Mar 11 14:29:12 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:12 2024] ? sysvec_apic_timer_interrupt+0x3c/0x90
[Mon Mar 11 14:29:12 2024] entry_SYSCALL_64_after_hwframe+0x72/0xdc
[Mon Mar 11 14:29:12 2024] RIP: 0033:0x7efc96f13975
[Mon Mar 11 14:29:12 2024] RSP: 002b:00007efc967fec60 EFLAGS: 00000293 ORIG_RAX: 00000000000000e6
[Mon Mar 11 14:29:12 2024] RAX: ffffffffffffffda RBX: ffffffffffffff60 RCX: 00007efc96f13975
[Mon Mar 11 14:29:12 2024] RDX: 00007efc967feca0 RSI: 0000000000000000 RDI: 0000000000000000
[Mon Mar 11 14:29:12 2024] RBP: 0000000000000000 R08: 0000000000000000 R09: 0000000000000000
[Mon Mar 11 14:29:12 2024] R10: 00007efc967feca0 R11: 0000000000000293 R12: 000055cc39724af0
[Mon Mar 11 14:29:12 2024] R13: 000055cc3971d328 R14: 000055cc39724b00 R15: 0000000000000000
[Mon Mar 11 14:29:12 2024] </TASK>
[Mon Mar 11 14:29:12 2024] task:polkitd state:S stack:0 pid:4667 ppid:1 flags:0x00000002
[Mon Mar 11 14:29:12 2024] Call Trace:
[Mon Mar 11 14:29:12 2024] <TASK>
[Mon Mar 11 14:29:12 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:12 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:12 2024] schedule_hrtimeout_range_clock+0x10f/0x120
[Mon Mar 11 14:29:12 2024] ? __fget_light+0x9f/0x130
[Mon Mar 11 14:29:12 2024] ? add_wait_queue+0x65/0xa0
[Mon Mar 11 14:29:12 2024] do_poll.constprop.0+0x248/0x390
[Mon Mar 11 14:29:12 2024] do_sys_poll+0x1c8/0x250
[Mon Mar 11 14:29:12 2024] ? do_sys_poll+0x1d8/0x250
[Mon Mar 11 14:29:12 2024] ? __pfx_pollwake+0x10/0x10
[Mon Mar 11 14:29:12 2024] ? __pfx_pollwake+0x10/0x10
[Mon Mar 11 14:29:12 2024] ? try_to_wake_up+0x3e2/0x5d0
[Mon Mar 11 14:29:12 2024] ? _copy_to_iter+0x7e/0x630
[Mon Mar 11 14:29:12 2024] ? pollwake+0x74/0xa0
[Mon Mar 11 14:29:12 2024] ? __pfx_default_wake_function+0x10/0x10
[Mon Mar 11 14:29:12 2024] ? __wake_up_common+0x75/0xa0
[Mon Mar 11 14:29:12 2024] ? eventfd_read+0xe2/0x2b0
[Mon Mar 11 14:29:12 2024] ? __audit_filter_op+0xa5/0xf0
[Mon Mar 11 14:29:12 2024] ? auditd_test_task+0x3c/0x50
[Mon Mar 11 14:29:12 2024] ? __audit_syscall_entry+0xef/0x140
[Mon Mar 11 14:29:12 2024] __x64_sys_poll+0x39/0x140
[Mon Mar 11 14:29:12 2024] ? syscall_trace_enter.constprop.0+0x126/0x1a0
[Mon Mar 11 14:29:12 2024] do_syscall_64+0x59/0x90
[Mon Mar 11 14:29:12 2024] ? syscall_exit_work+0x103/0x130
[Mon Mar 11 14:29:12 2024] ? syscall_exit_to_user_mode+0x22/0x40
[Mon Mar 11 14:29:12 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:12 2024] ? syscall_exit_work+0x103/0x130
[Mon Mar 11 14:29:12 2024] ? syscall_exit_to_user_mode+0x22/0x40
[Mon Mar 11 14:29:12 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:12 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:12 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:12 2024] ? sysvec_apic_timer_interrupt+0x3c/0x90
[Mon Mar 11 14:29:12 2024] entry_SYSCALL_64_after_hwframe+0x72/0xdc
[Mon Mar 11 14:29:12 2024] RIP: 0033:0x7ff7713426ff
[Mon Mar 11 14:29:12 2024] RSP: 002b:00007ffc44dec9a0 EFLAGS: 00000293 ORIG_RAX: 0000000000000007
[Mon Mar 11 14:29:12 2024] RAX: ffffffffffffffda RBX: 00007ff773775071 RCX: 00007ff7713426ff
[Mon Mar 11 14:29:12 2024] RDX: 00000000ffffffff RSI: 0000000000000002 RDI: 000055597da4b720
[Mon Mar 11 14:29:12 2024] RBP: 000055597da4b720 R08: 0000000000000000 R09: 00007ffc44dec830
[Mon Mar 11 14:29:12 2024] R10: 00007ffc44df5080 R11: 0000000000000293 R12: 0000000000000002
[Mon Mar 11 14:29:12 2024] R13: 0000000000000002 R14: 00007ffc44deca10 R15: 000055597d9517d0
[Mon Mar 11 14:29:12 2024] </TASK>
[Mon Mar 11 14:29:12 2024] task:gmain state:S stack:0 pid:5162 ppid:1 flags:0x00000002
[Mon Mar 11 14:29:12 2024] Call Trace:
[Mon Mar 11 14:29:12 2024] <TASK>
[Mon Mar 11 14:29:12 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:12 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:12 2024] schedule_hrtimeout_range_clock+0x10f/0x120
[Mon Mar 11 14:29:12 2024] ? add_wait_queue+0x65/0xa0
[Mon Mar 11 14:29:12 2024] do_poll.constprop.0+0x248/0x390
[Mon Mar 11 14:29:12 2024] do_sys_poll+0x1c8/0x250
[Mon Mar 11 14:29:12 2024] ? __pfx_pollwake+0x10/0x10
[Mon Mar 11 14:29:12 2024] ? __pfx_pollwake+0x10/0x10
[Mon Mar 11 14:29:12 2024] ? __pfx_pollwake+0x10/0x10
[Mon Mar 11 14:29:12 2024] ? _copy_to_iter+0x7e/0x630
[Mon Mar 11 14:29:12 2024] ? eventfd_read+0xe2/0x2b0
[Mon Mar 11 14:29:12 2024] ? auditd_test_task+0x3c/0x50
[Mon Mar 11 14:29:12 2024] ? __audit_syscall_entry+0xef/0x140
[Mon Mar 11 14:29:12 2024] __x64_sys_poll+0x39/0x140
[Mon Mar 11 14:29:12 2024] ? syscall_trace_enter.constprop.0+0x126/0x1a0
[Mon Mar 11 14:29:12 2024] do_syscall_64+0x59/0x90
[Mon Mar 11 14:29:12 2024] ? audit_reset_context.part.0.constprop.0+0x273/0x2e0
[Mon Mar 11 14:29:12 2024] ? syscall_exit_work+0x103/0x130
[Mon Mar 11 14:29:12 2024] ? syscall_exit_to_user_mode+0x22/0x40
[Mon Mar 11 14:29:12 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:12 2024] ? exit_to_user_mode_prepare+0xec/0x100
[Mon Mar 11 14:29:12 2024] entry_SYSCALL_64_after_hwframe+0x72/0xdc
[Mon Mar 11 14:29:12 2024] RIP: 0033:0x7ff7713426ff
[Mon Mar 11 14:29:12 2024] RSP: 002b:00007ff76fdfeb40 EFLAGS: 00000293 ORIG_RAX: 0000000000000007
[Mon Mar 11 14:29:12 2024] RAX: ffffffffffffffda RBX: 00007ff773775071 RCX: 00007ff7713426ff
[Mon Mar 11 14:29:12 2024] RDX: 00000000ffffffff RSI: 0000000000000002 RDI: 00007ff768000b80
[Mon Mar 11 14:29:12 2024] RBP: 00007ff768000b80 R08: 0000000000000000 R09: 0000000000000000
[Mon Mar 11 14:29:12 2024] R10: 00007ffc44df5080 R11: 0000000000000293 R12: 0000000000000002
[Mon Mar 11 14:29:12 2024] R13: 0000000000000002 R14: 00007ff76fdfebb0 R15: 000055597d950410
[Mon Mar 11 14:29:12 2024] </TASK>
[Mon Mar 11 14:29:12 2024] task:gdbus state:S stack:0 pid:5165 ppid:1 flags:0x00000002
[Mon Mar 11 14:29:12 2024] Call Trace:
[Mon Mar 11 14:29:12 2024] <TASK>
[Mon Mar 11 14:29:12 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:12 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:12 2024] schedule_hrtimeout_range_clock+0x10f/0x120
[Mon Mar 11 14:29:12 2024] ? add_wait_queue+0x65/0xa0
[Mon Mar 11 14:29:12 2024] ? add_wait_queue+0x65/0xa0
[Mon Mar 11 14:29:12 2024] do_poll.constprop.0+0x248/0x390
[Mon Mar 11 14:29:12 2024] do_sys_poll+0x1c8/0x250
[Mon Mar 11 14:29:12 2024] ? ep_poll_callback+0x25d/0x2a0
[Mon Mar 11 14:29:12 2024] ? __wake_up_common+0x75/0xa0
[Mon Mar 11 14:29:12 2024] ? __wake_up_sync_key+0x39/0x50
[Mon Mar 11 14:29:12 2024] ? sock_def_readable+0x3e/0xc0
[Mon Mar 11 14:29:12 2024] ? unix_stream_sendmsg+0x476/0x4b0
[Mon Mar 11 14:29:12 2024] ? __pfx_pollwake+0x10/0x10
[Mon Mar 11 14:29:12 2024] ? __pfx_pollwake+0x10/0x10
[Mon Mar 11 14:29:12 2024] ? ___sys_sendmsg+0x95/0xd0
[Mon Mar 11 14:29:12 2024] ? eventfd_read+0xe2/0x2b0
[Mon Mar 11 14:29:12 2024] ? __audit_filter_op+0xa5/0xf0
[Mon Mar 11 14:29:12 2024] ? auditd_test_task+0x3c/0x50
[Mon Mar 11 14:29:12 2024] __x64_sys_poll+0x39/0x140
[Mon Mar 11 14:29:12 2024] ? syscall_trace_enter.constprop.0+0x126/0x1a0
[Mon Mar 11 14:29:12 2024] do_syscall_64+0x59/0x90
[Mon Mar 11 14:29:12 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:12 2024] ? syscall_exit_work+0x103/0x130
[Mon Mar 11 14:29:12 2024] ? syscall_exit_to_user_mode+0x22/0x40
[Mon Mar 11 14:29:12 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:12 2024] ? sysvec_apic_timer_interrupt+0x3c/0x90
[Mon Mar 11 14:29:12 2024] entry_SYSCALL_64_after_hwframe+0x72/0xdc
[Mon Mar 11 14:29:12 2024] RIP: 0033:0x7ff7713426ff
[Mon Mar 11 14:29:12 2024] RSP: 002b:00007ff76f5fdb40 EFLAGS: 00000293 ORIG_RAX: 0000000000000007
[Mon Mar 11 14:29:12 2024] RAX: ffffffffffffffda RBX: 00007ff773775071 RCX: 00007ff7713426ff
[Mon Mar 11 14:29:12 2024] RDX: 00000000ffffffff RSI: 0000000000000002 RDI: 000055597d966df0
[Mon Mar 11 14:29:12 2024] RBP: 000055597d966df0 R08: 0000000000000000 R09: 00007ff76f5fd9d0
[Mon Mar 11 14:29:12 2024] R10: 00007ffc44df5080 R11: 0000000000000293 R12: 0000000000000002
[Mon Mar 11 14:29:12 2024] R13: 0000000000000002 R14: 00007ff76f5fdbb0 R15: 000055597d964e20
[Mon Mar 11 14:29:12 2024] </TASK>
[Mon Mar 11 14:29:12 2024] task:JS Helper state:S stack:0 pid:5175 ppid:1 flags:0x00000002
[Mon Mar 11 14:29:12 2024] Call Trace:
[Mon Mar 11 14:29:12 2024] <TASK>
[Mon Mar 11 14:29:12 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:12 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:12 2024] futex_wait_queue+0x70/0xd0
[Mon Mar 11 14:29:12 2024] futex_wait+0x175/0x260
[Mon Mar 11 14:29:12 2024] ? sched_clock+0xc/0x30
[Mon Mar 11 14:29:12 2024] ? __smp_call_single_queue+0x93/0x120
[Mon Mar 11 14:29:12 2024] ? ttwu_queue_wakelist+0xf2/0x110
[Mon Mar 11 14:29:12 2024] do_futex+0x12d/0x1d0
[Mon Mar 11 14:29:12 2024] __x64_sys_futex+0x73/0x1d0
[Mon Mar 11 14:29:12 2024] do_syscall_64+0x59/0x90
[Mon Mar 11 14:29:12 2024] ? syscall_exit_work+0x103/0x130
[Mon Mar 11 14:29:12 2024] ? syscall_exit_to_user_mode+0x22/0x40
[Mon Mar 11 14:29:12 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:12 2024] ? syscall_exit_work+0x103/0x130
[Mon Mar 11 14:29:12 2024] ? syscall_exit_to_user_mode+0x22/0x40
[Mon Mar 11 14:29:12 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:12 2024] ? exc_page_fault+0x62/0x150
[Mon Mar 11 14:29:12 2024] entry_SYSCALL_64_after_hwframe+0x72/0xdc
[Mon Mar 11 14:29:12 2024] RIP: 0033:0x7ff77129c39a
[Mon Mar 11 14:29:12 2024] RSP: 002b:00007ff76edfca40 EFLAGS: 00000246 ORIG_RAX: 00000000000000ca
[Mon Mar 11 14:29:12 2024] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff77129c39a
[Mon Mar 11 14:29:12 2024] RDX: 0000000000000000 RSI: 0000000000000189 RDI: 000055597d93b868
[Mon Mar 11 14:29:12 2024] RBP: 0000000000000000 R08: 0000000000000000 R09: 00000000ffffffff
[Mon Mar 11 14:29:12 2024] R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000000000
[Mon Mar 11 14:29:12 2024] R13: 000055597d93b868 R14: 0000000000000000 R15: 0000000000000000
[Mon Mar 11 14:29:12 2024] </TASK>
[Mon Mar 11 14:29:12 2024] task:JS Helper state:S stack:0 pid:5176 ppid:1 flags:0x00000002
[Mon Mar 11 14:29:12 2024] Call Trace:
[Mon Mar 11 14:29:12 2024] <TASK>
[Mon Mar 11 14:29:12 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:12 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:12 2024] futex_wait_queue+0x70/0xd0
[Mon Mar 11 14:29:12 2024] futex_wait+0x175/0x260
[Mon Mar 11 14:29:12 2024] ? futex_wait+0x17f/0x260
[Mon Mar 11 14:29:12 2024] do_futex+0x12d/0x1d0
[Mon Mar 11 14:29:12 2024] __x64_sys_futex+0x73/0x1d0
[Mon Mar 11 14:29:12 2024] do_syscall_64+0x59/0x90
[Mon Mar 11 14:29:12 2024] ? syscall_exit_to_user_mode+0x22/0x40
[Mon Mar 11 14:29:12 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:12 2024] ? flush_tlb_func+0x1ba/0x1f0
[Mon Mar 11 14:29:12 2024] ? sched_clock+0xc/0x30
[Mon Mar 11 14:29:12 2024] ? sched_clock_cpu+0x9/0xc0
[Mon Mar 11 14:29:12 2024] ? irqtime_account_irq+0x3c/0xb0
[Mon Mar 11 14:29:12 2024] ? __irq_exit_rcu+0x46/0xc0
[Mon Mar 11 14:29:12 2024] ? sysvec_call_function_single+0x3c/0x90
[Mon Mar 11 14:29:12 2024] entry_SYSCALL_64_after_hwframe+0x72/0xdc
[Mon Mar 11 14:29:12 2024] RIP: 0033:0x7ff77129c39a
[Mon Mar 11 14:29:12 2024] RSP: 002b:00007ff76ebfda40 EFLAGS: 00000246 ORIG_RAX: 00000000000000ca
[Mon Mar 11 14:29:12 2024] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff77129c39a
[Mon Mar 11 14:29:12 2024] RDX: 0000000000000000 RSI: 0000000000000189 RDI: 000055597d93b868
[Mon Mar 11 14:29:12 2024] RBP: 0000000000000000 R08: 0000000000000000 R09: 00000000ffffffff
[Mon Mar 11 14:29:12 2024] R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000000000
[Mon Mar 11 14:29:12 2024] R13: 000055597d93b868 R14: 0000000000000000 R15: 0000000000000000
[Mon Mar 11 14:29:12 2024] </TASK>
[Mon Mar 11 14:29:12 2024] task:JS Helper state:S stack:0 pid:5177 ppid:1 flags:0x00000002
[Mon Mar 11 14:29:12 2024] Call Trace:
[Mon Mar 11 14:29:12 2024] <TASK>
[Mon Mar 11 14:29:12 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:12 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:12 2024] futex_wait_queue+0x70/0xd0
[Mon Mar 11 14:29:12 2024] futex_wait+0x175/0x260
[Mon Mar 11 14:29:12 2024] ? futex_wake+0x151/0x180
[Mon Mar 11 14:29:12 2024] do_futex+0x12d/0x1d0
[Mon Mar 11 14:29:12 2024] __x64_sys_futex+0x73/0x1d0
[Mon Mar 11 14:29:12 2024] do_syscall_64+0x59/0x90
[Mon Mar 11 14:29:12 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:12 2024] ? syscall_exit_to_user_mode+0x22/0x40
[Mon Mar 11 14:29:12 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:12 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:12 2024] ? syscall_exit_to_user_mode+0x22/0x40
[Mon Mar 11 14:29:12 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:12 2024] ? syscall_exit_to_user_mode+0x22/0x40
[Mon Mar 11 14:29:12 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:12 2024] ? exc_page_fault+0x62/0x150
[Mon Mar 11 14:29:12 2024] entry_SYSCALL_64_after_hwframe+0x72/0xdc
[Mon Mar 11 14:29:12 2024] RIP: 0033:0x7ff77129c39a
[Mon Mar 11 14:29:12 2024] RSP: 002b:00007ff76e9fea40 EFLAGS: 00000246 ORIG_RAX: 00000000000000ca
[Mon Mar 11 14:29:12 2024] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff77129c39a
[Mon Mar 11 14:29:12 2024] RDX: 0000000000000000 RSI: 0000000000000189 RDI: 000055597d93b868
[Mon Mar 11 14:29:12 2024] RBP: 0000000000000000 R08: 0000000000000000 R09: 00000000ffffffff
[Mon Mar 11 14:29:12 2024] R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000000000
[Mon Mar 11 14:29:12 2024] R13: 000055597d93b868 R14: 0000000000000000 R15: 0000000000000000
[Mon Mar 11 14:29:12 2024] </TASK>
[Mon Mar 11 14:29:12 2024] task:JS Helper state:S stack:0 pid:5178 ppid:1 flags:0x00000002
[Mon Mar 11 14:29:12 2024] Call Trace:
[Mon Mar 11 14:29:12 2024] <TASK>
[Mon Mar 11 14:29:12 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:12 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:12 2024] futex_wait_queue+0x70/0xd0
[Mon Mar 11 14:29:12 2024] futex_wait+0x175/0x260
[Mon Mar 11 14:29:12 2024] ? ttwu_queue_wakelist+0xf2/0x110
[Mon Mar 11 14:29:12 2024] ? try_to_wake_up+0x3e2/0x5d0
[Mon Mar 11 14:29:12 2024] do_futex+0x12d/0x1d0
[Mon Mar 11 14:29:12 2024] __x64_sys_futex+0x73/0x1d0
[Mon Mar 11 14:29:12 2024] do_syscall_64+0x59/0x90
[Mon Mar 11 14:29:12 2024] ? syscall_exit_to_user_mode+0x22/0x40
[Mon Mar 11 14:29:12 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:12 2024] ? syscall_exit_to_user_mode+0x22/0x40
[Mon Mar 11 14:29:12 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:12 2024] ? exc_page_fault+0x62/0x150
[Mon Mar 11 14:29:12 2024] entry_SYSCALL_64_after_hwframe+0x72/0xdc
[Mon Mar 11 14:29:12 2024] RIP: 0033:0x7ff77129c39a
[Mon Mar 11 14:29:12 2024] RSP: 002b:00007ff76e7ffa40 EFLAGS: 00000246 ORIG_RAX: 00000000000000ca
[Mon Mar 11 14:29:12 2024] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff77129c39a
[Mon Mar 11 14:29:12 2024] RDX: 0000000000000000 RSI: 0000000000000189 RDI: 000055597d93b868
[Mon Mar 11 14:29:12 2024] RBP: 0000000000000000 R08: 0000000000000000 R09: 00000000ffffffff
[Mon Mar 11 14:29:12 2024] R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000000000
[Mon Mar 11 14:29:12 2024] R13: 000055597d93b868 R14: 0000000000000000 R15: 0000000000000000
[Mon Mar 11 14:29:12 2024] </TASK>
[Mon Mar 11 14:29:12 2024] task:JS Helper state:S stack:0 pid:5179 ppid:1 flags:0x00000002
[Mon Mar 11 14:29:12 2024] Call Trace:
[Mon Mar 11 14:29:12 2024] <TASK>
[Mon Mar 11 14:29:12 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:12 2024] ? free_unref_page_commit+0x7e/0x170
[Mon Mar 11 14:29:12 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:12 2024] futex_wait_queue+0x70/0xd0
[Mon Mar 11 14:29:12 2024] futex_wait+0x175/0x260
[Mon Mar 11 14:29:12 2024] ? ttwu_queue_wakelist+0xf2/0x110
[Mon Mar 11 14:29:12 2024] ? try_to_wake_up+0x3e2/0x5d0
[Mon Mar 11 14:29:12 2024] ? audit_filter_rules.constprop.0+0x148/0xd30
[Mon Mar 11 14:29:12 2024] do_futex+0x12d/0x1d0
[Mon Mar 11 14:29:12 2024] __x64_sys_futex+0x73/0x1d0
[Mon Mar 11 14:29:12 2024] do_syscall_64+0x59/0x90
[Mon Mar 11 14:29:12 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:12 2024] ? syscall_exit_to_user_mode+0x22/0x40
[Mon Mar 11 14:29:12 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:12 2024] entry_SYSCALL_64_after_hwframe+0x72/0xdc
[Mon Mar 11 14:29:12 2024] RIP: 0033:0x7ff77129c39a
[Mon Mar 11 14:29:12 2024] RSP: 002b:00007ff76e600a40 EFLAGS: 00000246 ORIG_RAX: 00000000000000ca
[Mon Mar 11 14:29:12 2024] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff77129c39a
[Mon Mar 11 14:29:12 2024] RDX: 0000000000000000 RSI: 0000000000000189 RDI: 000055597d93b868
[Mon Mar 11 14:29:12 2024] RBP: 0000000000000000 R08: 0000000000000000 R09: 00000000ffffffff
[Mon Mar 11 14:29:12 2024] R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000000000
[Mon Mar 11 14:29:12 2024] R13: 000055597d93b868 R14: 0000000000000000 R15: 0000000000000000
[Mon Mar 11 14:29:12 2024] </TASK>
[Mon Mar 11 14:29:12 2024] task:JS Helper state:S stack:0 pid:5180 ppid:1 flags:0x00000002
[Mon Mar 11 14:29:12 2024] Call Trace:
[Mon Mar 11 14:29:12 2024] <TASK>
[Mon Mar 11 14:29:12 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:12 2024] ? wake_affine+0x62/0x1f0
[Mon Mar 11 14:29:12 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:12 2024] futex_wait_queue+0x70/0xd0
[Mon Mar 11 14:29:12 2024] futex_wait+0x175/0x260
[Mon Mar 11 14:29:12 2024] ? ttwu_queue_wakelist+0xf2/0x110
[Mon Mar 11 14:29:12 2024] ? try_to_wake_up+0x3e2/0x5d0
[Mon Mar 11 14:29:12 2024] do_futex+0x12d/0x1d0
[Mon Mar 11 14:29:12 2024] __x64_sys_futex+0x73/0x1d0
[Mon Mar 11 14:29:12 2024] do_syscall_64+0x59/0x90
[Mon Mar 11 14:29:12 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:12 2024] ? syscall_exit_work+0x103/0x130
[Mon Mar 11 14:29:12 2024] ? syscall_exit_to_user_mode+0x22/0x40
[Mon Mar 11 14:29:12 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:12 2024] ? syscall_exit_work+0x103/0x130
[Mon Mar 11 14:29:12 2024] ? syscall_exit_to_user_mode+0x22/0x40
[Mon Mar 11 14:29:12 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:12 2024] entry_SYSCALL_64_after_hwframe+0x72/0xdc
[Mon Mar 11 14:29:12 2024] RIP: 0033:0x7ff77129c39a
[Mon Mar 11 14:29:12 2024] RSP: 002b:00007ff76e401a40 EFLAGS: 00000246 ORIG_RAX: 00000000000000ca
[Mon Mar 11 14:29:12 2024] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff77129c39a
[Mon Mar 11 14:29:12 2024] RDX: 0000000000000000 RSI: 0000000000000189 RDI: 000055597d93b868
[Mon Mar 11 14:29:12 2024] RBP: 0000000000000000 R08: 0000000000000000 R09: 00000000ffffffff
[Mon Mar 11 14:29:12 2024] R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000000000
[Mon Mar 11 14:29:12 2024] R13: 000055597d93b868 R14: 0000000000000000 R15: 0000000000000000
[Mon Mar 11 14:29:12 2024] </TASK>
[Mon Mar 11 14:29:12 2024] task:JS Helper state:S stack:0 pid:5181 ppid:1 flags:0x00000002
[Mon Mar 11 14:29:12 2024] Call Trace:
[Mon Mar 11 14:29:12 2024] <TASK>
[Mon Mar 11 14:29:12 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:12 2024] ? __smp_call_single_queue+0x93/0x120
[Mon Mar 11 14:29:12 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:12 2024] futex_wait_queue+0x70/0xd0
[Mon Mar 11 14:29:12 2024] futex_wait+0x175/0x260
[Mon Mar 11 14:29:12 2024] ? futex_wake+0x151/0x180
[Mon Mar 11 14:29:12 2024] do_futex+0x12d/0x1d0
[Mon Mar 11 14:29:12 2024] __x64_sys_futex+0x73/0x1d0
[Mon Mar 11 14:29:12 2024] do_syscall_64+0x59/0x90
[Mon Mar 11 14:29:12 2024] entry_SYSCALL_64_after_hwframe+0x72/0xdc
[Mon Mar 11 14:29:12 2024] RIP: 0033:0x7ff77129c39a
[Mon Mar 11 14:29:12 2024] RSP: 002b:00007ff76e202a40 EFLAGS: 00000246 ORIG_RAX: 00000000000000ca
[Mon Mar 11 14:29:12 2024] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff77129c39a
[Mon Mar 11 14:29:12 2024] RDX: 0000000000000000 RSI: 0000000000000189 RDI: 000055597d93b868
[Mon Mar 11 14:29:12 2024] RBP: 0000000000000000 R08: 0000000000000000 R09: 00000000ffffffff
[Mon Mar 11 14:29:12 2024] R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000000000
[Mon Mar 11 14:29:12 2024] R13: 000055597d93b868 R14: 0000000000000000 R15: 0000000000000000
[Mon Mar 11 14:29:12 2024] </TASK>
[Mon Mar 11 14:29:12 2024] task:JS Helper state:S stack:0 pid:5182 ppid:1 flags:0x00000002
[Mon Mar 11 14:29:12 2024] Call Trace:
[Mon Mar 11 14:29:12 2024] <TASK>
[Mon Mar 11 14:29:12 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:12 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:12 2024] futex_wait_queue+0x70/0xd0
[Mon Mar 11 14:29:12 2024] futex_wait+0x175/0x260
[Mon Mar 11 14:29:12 2024] ? __mod_memcg_lruvec_state+0x84/0xd0
[Mon Mar 11 14:29:12 2024] ? sched_clock+0xc/0x30
[Mon Mar 11 14:29:12 2024] ? __smp_call_single_queue+0x93/0x120
[Mon Mar 11 14:29:12 2024] ? ttwu_queue_wakelist+0xf2/0x110
[Mon Mar 11 14:29:12 2024] do_futex+0x12d/0x1d0
[Mon Mar 11 14:29:12 2024] __x64_sys_futex+0x73/0x1d0
[Mon Mar 11 14:29:12 2024] do_syscall_64+0x59/0x90
[Mon Mar 11 14:29:12 2024] ? syscall_exit_work+0x103/0x130
[Mon Mar 11 14:29:12 2024] ? syscall_exit_to_user_mode+0x22/0x40
[Mon Mar 11 14:29:12 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:12 2024] ? __irq_exit_rcu+0x46/0xc0
[Mon Mar 11 14:29:12 2024] ? sysvec_call_function+0x3c/0x90
[Mon Mar 11 14:29:12 2024] entry_SYSCALL_64_after_hwframe+0x72/0xdc
[Mon Mar 11 14:29:12 2024] RIP: 0033:0x7ff77129c39a
[Mon Mar 11 14:29:12 2024] RSP: 002b:00007ff76e003a40 EFLAGS: 00000246 ORIG_RAX: 00000000000000ca
[Mon Mar 11 14:29:12 2024] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff77129c39a
[Mon Mar 11 14:29:12 2024] RDX: 0000000000000000 RSI: 0000000000000189 RDI: 000055597d93b868
[Mon Mar 11 14:29:12 2024] RBP: 0000000000000000 R08: 0000000000000000 R09: 00000000ffffffff
[Mon Mar 11 14:29:12 2024] R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000000000
[Mon Mar 11 14:29:12 2024] R13: 000055597d93b868 R14: 0000000000000000 R15: 0000000000000000
[Mon Mar 11 14:29:12 2024] </TASK>
[Mon Mar 11 14:29:12 2024] task:polkitd state:S stack:0 pid:5197 ppid:1 flags:0x00000002
[Mon Mar 11 14:29:12 2024] Call Trace:
[Mon Mar 11 14:29:12 2024] <TASK>
[Mon Mar 11 14:29:12 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:12 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:12 2024] schedule_hrtimeout_range_clock+0x10f/0x120
[Mon Mar 11 14:29:12 2024] ? do_poll.constprop.0+0x298/0x390
[Mon Mar 11 14:29:12 2024] ? add_wait_queue+0x65/0xa0
[Mon Mar 11 14:29:12 2024] do_poll.constprop.0+0x248/0x390
[Mon Mar 11 14:29:12 2024] do_sys_poll+0x1c8/0x250
[Mon Mar 11 14:29:12 2024] ? do_sys_poll+0x1d8/0x250
[Mon Mar 11 14:29:12 2024] ? rmqueue_pcplist+0xda/0x210
[Mon Mar 11 14:29:12 2024] ? __pfx_pollwake+0x10/0x10
[Mon Mar 11 14:29:12 2024] ? __pfx_pollwake+0x10/0x10
[Mon Mar 11 14:29:12 2024] ? __pfx_pollwake+0x10/0x10
[Mon Mar 11 14:29:12 2024] ? _copy_to_iter+0x7e/0x630
[Mon Mar 11 14:29:12 2024] ? _copy_to_iter+0x7e/0x630
[Mon Mar 11 14:29:12 2024] ? _copy_to_iter+0x7e/0x630
[Mon Mar 11 14:29:12 2024] ? rseq_get_rseq_cs+0x1d/0x240
[Mon Mar 11 14:29:12 2024] ? _copy_to_iter+0x7e/0x630
[Mon Mar 11 14:29:12 2024] ? rseq_ip_fixup+0x6e/0x1a0
[Mon Mar 11 14:29:12 2024] ? __audit_filter_op+0xa5/0xf0
[Mon Mar 11 14:29:12 2024] ? eventfd_read+0xe2/0x2b0
[Mon Mar 11 14:29:12 2024] ? auditd_test_task+0x3c/0x50
[Mon Mar 11 14:29:12 2024] ? __audit_syscall_entry+0xef/0x140
[Mon Mar 11 14:29:12 2024] __x64_sys_poll+0x39/0x140
[Mon Mar 11 14:29:12 2024] ? syscall_trace_enter.constprop.0+0x126/0x1a0
[Mon Mar 11 14:29:12 2024] do_syscall_64+0x59/0x90
[Mon Mar 11 14:29:12 2024] ? audit_reset_context.part.0.constprop.0+0x273/0x2e0
[Mon Mar 11 14:29:12 2024] ? syscall_exit_work+0x103/0x130
[Mon Mar 11 14:29:12 2024] ? syscall_exit_to_user_mode+0x22/0x40
[Mon Mar 11 14:29:12 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:12 2024] ? exc_page_fault+0x62/0x150
[Mon Mar 11 14:29:12 2024] entry_SYSCALL_64_after_hwframe+0x72/0xdc
[Mon Mar 11 14:29:12 2024] RIP: 0033:0x7ff7713426ff
[Mon Mar 11 14:29:12 2024] RSP: 002b:00007ff76de04b30 EFLAGS: 00000293 ORIG_RAX: 0000000000000007
[Mon Mar 11 14:29:12 2024] RAX: ffffffffffffffda RBX: 00007ff773775071 RCX: 00007ff7713426ff
[Mon Mar 11 14:29:12 2024] RDX: 00000000ffffffff RSI: 0000000000000001 RDI: 000055597db471e0
[Mon Mar 11 14:29:12 2024] RBP: 000055597db471e0 R08: 0000000000000000 R09: 0000000000000000
[Mon Mar 11 14:29:12 2024] R10: 00007ffc44df5080 R11: 0000000000000293 R12: 0000000000000001
[Mon Mar 11 14:29:12 2024] R13: 0000000000000001 R14: 00007ff76de04ba0 R15: 000055597dac05e0
[Mon Mar 11 14:29:12 2024] </TASK>
[Mon Mar 11 14:29:12 2024] task:wb[KWAK] state:S stack:0 pid:4668 ppid:4415 flags:0x00000002
[Mon Mar 11 14:29:12 2024] Call Trace:
[Mon Mar 11 14:29:12 2024] <TASK>
[Mon Mar 11 14:29:12 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:12 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:12 2024] schedule_hrtimeout_range_clock+0x95/0x120
[Mon Mar 11 14:29:12 2024] ? __pfx_hrtimer_wakeup+0x10/0x10
[Mon Mar 11 14:29:12 2024] ep_poll+0x348/0x3b0
[Mon Mar 11 14:29:12 2024] ? __pfx_autoremove_wake_function+0x10/0x10
[Mon Mar 11 14:29:12 2024] do_epoll_wait+0xb1/0xd0
[Mon Mar 11 14:29:12 2024] __x64_sys_epoll_wait+0x60/0x100
[Mon Mar 11 14:29:12 2024] do_syscall_64+0x59/0x90
[Mon Mar 11 14:29:12 2024] ? syscall_exit_to_user_mode+0x22/0x40
[Mon Mar 11 14:29:12 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:12 2024] ? syscall_exit_to_user_mode+0x22/0x40
[Mon Mar 11 14:29:12 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:12 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:12 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:12 2024] ? sysvec_apic_timer_interrupt+0x3c/0x90
[Mon Mar 11 14:29:12 2024] entry_SYSCALL_64_after_hwframe+0x72/0xdc
[Mon Mar 11 14:29:12 2024] RIP: 0033:0x7f26a134e80a
[Mon Mar 11 14:29:12 2024] RSP: 002b:00007ffe9e332b58 EFLAGS: 00000246 ORIG_RAX: 00000000000000e8
[Mon Mar 11 14:29:12 2024] RAX: ffffffffffffffda RBX: 00005586337521b0 RCX: 00007f26a134e80a
[Mon Mar 11 14:29:12 2024] RDX: 0000000000000001 RSI: 00007ffe9e332b8c RDI: 0000000000000005
[Mon Mar 11 14:29:12 2024] RBP: 00005586337422e0 R08: 000055863376e4b0 R09: 00007ffe9e332300
[Mon Mar 11 14:29:12 2024] R10: 0000000000007530 R11: 0000000000000246 R12: 000055863375b3a0
[Mon Mar 11 14:29:12 2024] R13: 0000000000007530 R14: 0000558632a302f8 R15: 0000558632a4c020
[Mon Mar 11 14:29:12 2024] </TASK>
[Mon Mar 11 14:29:12 2024] task:collectd state:S stack:0 pid:4728 ppid:1 flags:0x00000002
[Mon Mar 11 14:29:12 2024] Call Trace:
[Mon Mar 11 14:29:12 2024] <TASK>
[Mon Mar 11 14:29:12 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:12 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:12 2024] do_nanosleep+0x67/0x190
[Mon Mar 11 14:29:12 2024] ? switch_fpu_return+0x4c/0xd0
[Mon Mar 11 14:29:12 2024] hrtimer_nanosleep+0xbe/0x1a0
[Mon Mar 11 14:29:12 2024] ? __pfx_hrtimer_wakeup+0x10/0x10
[Mon Mar 11 14:29:12 2024] common_nsleep+0x40/0x50
[Mon Mar 11 14:29:12 2024] __x64_sys_clock_nanosleep+0xbc/0x130
[Mon Mar 11 14:29:12 2024] do_syscall_64+0x59/0x90
[Mon Mar 11 14:29:12 2024] ? syscall_exit_to_user_mode+0x22/0x40
[Mon Mar 11 14:29:12 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:12 2024] entry_SYSCALL_64_after_hwframe+0x72/0xdc
[Mon Mar 11 14:29:12 2024] RIP: 0033:0x7f3353913975
[Mon Mar 11 14:29:12 2024] RSP: 002b:00007fff67a02380 EFLAGS: 00000293 ORIG_RAX: 00000000000000e6
[Mon Mar 11 14:29:12 2024] RAX: ffffffffffffffda RBX: 00007fff67a023c0 RCX: 00007f3353913975
[Mon Mar 11 14:29:12 2024] RDX: 00007fff67a023c0 RSI: 0000000000000000 RDI: 0000000000000000
[Mon Mar 11 14:29:12 2024] RBP: 197bc1d4d90f9051 R08: 0000000000000000 R09: 00007fff67a020b8
[Mon Mar 11 14:29:12 2024] R10: 00007fff67a023c0 R11: 0000000000000293 R12: 0000000280000000
[Mon Mar 11 14:29:12 2024] R13: 0000560fc6c574c0 R14: 197bc1d25914941f R15: 0000000000000003
[Mon Mar 11 14:29:12 2024] </TASK>
[Mon Mar 11 14:29:12 2024] task:writer#0 state:S stack:0 pid:5327 ppid:1 flags:0x00000002
[Mon Mar 11 14:29:12 2024] Call Trace:
[Mon Mar 11 14:29:12 2024] <TASK>
[Mon Mar 11 14:29:12 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:12 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:12 2024] futex_wait_queue+0x70/0xd0
[Mon Mar 11 14:29:12 2024] futex_wait+0x175/0x260
[Mon Mar 11 14:29:12 2024] ? futex_wake+0x151/0x180
[Mon Mar 11 14:29:12 2024] do_futex+0x12d/0x1d0
[Mon Mar 11 14:29:12 2024] __x64_sys_futex+0x73/0x1d0
[Mon Mar 11 14:29:12 2024] do_syscall_64+0x59/0x90
[Mon Mar 11 14:29:12 2024] ? syscall_exit_work+0x103/0x130
[Mon Mar 11 14:29:12 2024] ? syscall_exit_to_user_mode+0x22/0x40
[Mon Mar 11 14:29:12 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:12 2024] ? syscall_exit_to_user_mode+0x22/0x40
[Mon Mar 11 14:29:12 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:12 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:12 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:12 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:12 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:12 2024] ? sysvec_apic_timer_interrupt+0x3c/0x90
[Mon Mar 11 14:29:12 2024] entry_SYSCALL_64_after_hwframe+0x72/0xdc
[Mon Mar 11 14:29:12 2024] RIP: 0033:0x7f335389c39a
[Mon Mar 11 14:29:12 2024] RSP: 002b:00007f33536c5cd0 EFLAGS: 00000246 ORIG_RAX: 00000000000000ca
[Mon Mar 11 14:29:12 2024] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f335389c39a
[Mon Mar 11 14:29:12 2024] RDX: 0000000000000000 RSI: 0000000000000189 RDI: 0000560fc6c6bc68
[Mon Mar 11 14:29:12 2024] RBP: 0000000000000000 R08: 0000000000000000 R09: 00000000ffffffff
[Mon Mar 11 14:29:12 2024] R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000000000
[Mon Mar 11 14:29:12 2024] R13: 0000560fc6c6bc68 R14: 0000000000000000 R15: 0000000000000000
[Mon Mar 11 14:29:12 2024] </TASK>
[Mon Mar 11 14:29:12 2024] task:writer#1 state:S stack:0 pid:5329 ppid:1 flags:0x00000002
[Mon Mar 11 14:29:12 2024] Call Trace:
[Mon Mar 11 14:29:12 2024] <TASK>
[Mon Mar 11 14:29:12 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:12 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:12 2024] futex_wait_queue+0x70/0xd0
[Mon Mar 11 14:29:12 2024] futex_wait+0x175/0x260
[Mon Mar 11 14:29:12 2024] ? select_idle_sibling+0x28/0x430
[Mon Mar 11 14:29:12 2024] ? wake_affine+0x62/0x1f0
[Mon Mar 11 14:29:12 2024] do_futex+0x12d/0x1d0
[Mon Mar 11 14:29:12 2024] __x64_sys_futex+0x73/0x1d0
[Mon Mar 11 14:29:12 2024] do_syscall_64+0x59/0x90
[Mon Mar 11 14:29:12 2024] ? __audit_filter_op+0xa5/0xf0
[Mon Mar 11 14:29:12 2024] ? audit_reset_context.part.0.constprop.0+0x273/0x2e0
[Mon Mar 11 14:29:12 2024] ? syscall_exit_work+0x103/0x130
[Mon Mar 11 14:29:12 2024] ? syscall_exit_to_user_mode+0x22/0x40
[Mon Mar 11 14:29:12 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:12 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:12 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:12 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:12 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:12 2024] ? sysvec_apic_timer_interrupt+0x3c/0x90
[Mon Mar 11 14:29:12 2024] entry_SYSCALL_64_after_hwframe+0x72/0xdc
[Mon Mar 11 14:29:12 2024] RIP: 0033:0x7f335389c39a
[Mon Mar 11 14:29:12 2024] RSP: 002b:00007f3352ec4cd0 EFLAGS: 00000246 ORIG_RAX: 00000000000000ca
[Mon Mar 11 14:29:12 2024] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f335389c39a
[Mon Mar 11 14:29:12 2024] RDX: 0000000000000000 RSI: 0000000000000189 RDI: 0000560fc6c6bc68
[Mon Mar 11 14:29:12 2024] RBP: 0000000000000000 R08: 0000000000000000 R09: 00000000ffffffff
[Mon Mar 11 14:29:12 2024] R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000000000
[Mon Mar 11 14:29:12 2024] R13: 0000560fc6c6bc68 R14: 0000000000000000 R15: 0000000000000000
[Mon Mar 11 14:29:12 2024] </TASK>
[Mon Mar 11 14:29:12 2024] task:writer#2 state:S stack:0 pid:5330 ppid:1 flags:0x00000002
[Mon Mar 11 14:29:12 2024] Call Trace:
[Mon Mar 11 14:29:12 2024] <TASK>
[Mon Mar 11 14:29:12 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:12 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:12 2024] futex_wait_queue+0x70/0xd0
[Mon Mar 11 14:29:12 2024] futex_wait+0x175/0x260
[Mon Mar 11 14:29:12 2024] ? futex_wait_queue+0x82/0xd0
[Mon Mar 11 14:29:12 2024] ? futex_wait+0x17f/0x260
[Mon Mar 11 14:29:12 2024] do_futex+0x12d/0x1d0
[Mon Mar 11 14:29:12 2024] __x64_sys_futex+0x73/0x1d0
[Mon Mar 11 14:29:12 2024] do_syscall_64+0x59/0x90
[Mon Mar 11 14:29:12 2024] ? __audit_filter_op+0xa5/0xf0
[Mon Mar 11 14:29:12 2024] ? audit_reset_context.part.0.constprop.0+0x273/0x2e0
[Mon Mar 11 14:29:12 2024] ? syscall_exit_work+0x103/0x130
[Mon Mar 11 14:29:12 2024] ? syscall_exit_to_user_mode+0x22/0x40
[Mon Mar 11 14:29:12 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:12 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:12 2024] ? sysvec_apic_timer_interrupt+0x3c/0x90
[Mon Mar 11 14:29:12 2024] entry_SYSCALL_64_after_hwframe+0x72/0xdc
[Mon Mar 11 14:29:12 2024] RIP: 0033:0x7f335389c39a
[Mon Mar 11 14:29:12 2024] RSP: 002b:00007f33526c3cd0 EFLAGS: 00000246 ORIG_RAX: 00000000000000ca
[Mon Mar 11 14:29:12 2024] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f335389c39a
[Mon Mar 11 14:29:12 2024] RDX: 0000000000000000 RSI: 0000000000000189 RDI: 0000560fc6c6bc68
[Mon Mar 11 14:29:12 2024] RBP: 0000000000000000 R08: 0000000000000000 R09: 00000000ffffffff
[Mon Mar 11 14:29:12 2024] R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000000000
[Mon Mar 11 14:29:12 2024] R13: 0000560fc6c6bc68 R14: 0000000000000000 R15: 0000000000000000
[Mon Mar 11 14:29:12 2024] </TASK>
[Mon Mar 11 14:29:12 2024] task:writer#3 state:S stack:0 pid:5331 ppid:1 flags:0x00000002
[Mon Mar 11 14:29:12 2024] Call Trace:
[Mon Mar 11 14:29:12 2024] <TASK>
[Mon Mar 11 14:29:12 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:12 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:12 2024] futex_wait_queue+0x70/0xd0
[Mon Mar 11 14:29:12 2024] futex_wait+0x175/0x260
[Mon Mar 11 14:29:12 2024] ? futex_wake+0x151/0x180
[Mon Mar 11 14:29:12 2024] do_futex+0x12d/0x1d0
[Mon Mar 11 14:29:12 2024] __x64_sys_futex+0x73/0x1d0
[Mon Mar 11 14:29:12 2024] do_syscall_64+0x59/0x90
[Mon Mar 11 14:29:12 2024] ? audit_reset_context.part.0.constprop.0+0x273/0x2e0
[Mon Mar 11 14:29:12 2024] ? syscall_exit_work+0x103/0x130
[Mon Mar 11 14:29:12 2024] ? syscall_exit_to_user_mode+0x22/0x40
[Mon Mar 11 14:29:12 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:12 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:12 2024] ? sysvec_apic_timer_interrupt+0x3c/0x90
[Mon Mar 11 14:29:12 2024] entry_SYSCALL_64_after_hwframe+0x72/0xdc
[Mon Mar 11 14:29:12 2024] RIP: 0033:0x7f335389c39a
[Mon Mar 11 14:29:12 2024] RSP: 002b:00007f3351ec2cd0 EFLAGS: 00000246 ORIG_RAX: 00000000000000ca
[Mon Mar 11 14:29:12 2024] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f335389c39a
[Mon Mar 11 14:29:12 2024] RDX: 0000000000000000 RSI: 0000000000000189 RDI: 0000560fc6c6bc68
[Mon Mar 11 14:29:12 2024] RBP: 0000000000000000 R08: 0000000000000000 R09: 00000000ffffffff
[Mon Mar 11 14:29:12 2024] R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000000000
[Mon Mar 11 14:29:12 2024] R13: 0000560fc6c6bc68 R14: 0000000000000000 R15: 0000000000000000
[Mon Mar 11 14:29:12 2024] </TASK>
[Mon Mar 11 14:29:12 2024] task:writer#4 state:S stack:0 pid:5332 ppid:1 flags:0x00000002
[Mon Mar 11 14:29:12 2024] Call Trace:
[Mon Mar 11 14:29:12 2024] <TASK>
[Mon Mar 11 14:29:12 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:12 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:12 2024] futex_wait_queue+0x70/0xd0
[Mon Mar 11 14:29:12 2024] futex_wait+0x175/0x260
[Mon Mar 11 14:29:12 2024] ? try_to_wake_up+0x3e2/0x5d0
[Mon Mar 11 14:29:12 2024] do_futex+0x12d/0x1d0
[Mon Mar 11 14:29:12 2024] __x64_sys_futex+0x73/0x1d0
[Mon Mar 11 14:29:12 2024] do_syscall_64+0x59/0x90
[Mon Mar 11 14:29:12 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:12 2024] ? __audit_filter_op+0xa5/0xf0
[Mon Mar 11 14:29:12 2024] ? __rseq_handle_notify_resume+0x26/0xb0
[Mon Mar 11 14:29:12 2024] ? exit_to_user_mode_loop+0xd0/0x130
[Mon Mar 11 14:29:12 2024] ? exit_to_user_mode_prepare+0xec/0x100
[Mon Mar 11 14:29:12 2024] ? syscall_exit_to_user_mode+0x22/0x40
[Mon Mar 11 14:29:12 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:12 2024] ? exit_to_user_mode_prepare+0xec/0x100
[Mon Mar 11 14:29:12 2024] ? syscall_exit_to_user_mode+0x22/0x40
[Mon Mar 11 14:29:12 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:12 2024] ? sysvec_apic_timer_interrupt+0x3c/0x90
[Mon Mar 11 14:29:12 2024] entry_SYSCALL_64_after_hwframe+0x72/0xdc
[Mon Mar 11 14:29:12 2024] RIP: 0033:0x7f335389c39a
[Mon Mar 11 14:29:12 2024] RSP: 002b:00007f33516c1cd0 EFLAGS: 00000246 ORIG_RAX: 00000000000000ca
[Mon Mar 11 14:29:12 2024] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f335389c39a
[Mon Mar 11 14:29:12 2024] RDX: 0000000000000000 RSI: 0000000000000189 RDI: 0000560fc6c6bc68
[Mon Mar 11 14:29:12 2024] RBP: 0000000000000000 R08: 0000000000000000 R09: 00000000ffffffff
[Mon Mar 11 14:29:12 2024] R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000000000
[Mon Mar 11 14:29:12 2024] R13: 0000560fc6c6bc68 R14: 0000000000000000 R15: 0000000000000000
[Mon Mar 11 14:29:12 2024] </TASK>
[Mon Mar 11 14:29:12 2024] task:reader#0 state:S stack:0 pid:5333 ppid:1 flags:0x00000002
[Mon Mar 11 14:29:12 2024] Call Trace:
[Mon Mar 11 14:29:12 2024] <TASK>
[Mon Mar 11 14:29:12 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:12 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:12 2024] futex_wait_queue+0x70/0xd0
[Mon Mar 11 14:29:12 2024] futex_wait+0x175/0x260
[Mon Mar 11 14:29:12 2024] ? __pfx_hrtimer_wakeup+0x10/0x10
[Mon Mar 11 14:29:12 2024] do_futex+0x12d/0x1d0
[Mon Mar 11 14:29:12 2024] __x64_sys_futex+0x73/0x1d0
[Mon Mar 11 14:29:12 2024] do_syscall_64+0x59/0x90
[Mon Mar 11 14:29:12 2024] ? irqtime_account_irq+0x3c/0xb0
[Mon Mar 11 14:29:12 2024] ? __irq_exit_rcu+0x46/0xc0
[Mon Mar 11 14:29:12 2024] ? sysvec_apic_timer_interrupt+0x3c/0x90
[Mon Mar 11 14:29:12 2024] entry_SYSCALL_64_after_hwframe+0x72/0xdc
[Mon Mar 11 14:29:12 2024] RIP: 0033:0x7f335389c39a
[Mon Mar 11 14:29:12 2024] RSP: 002b:00007f3350ec0ca0 EFLAGS: 00000246 ORIG_RAX: 00000000000000ca
[Mon Mar 11 14:29:12 2024] RAX: ffffffffffffffda RBX: 00007f3350ec0dd0 RCX: 00007f335389c39a
[Mon Mar 11 14:29:12 2024] RDX: 0000000000000000 RSI: 0000000000000189 RDI: 0000560fc6c6bce8
[Mon Mar 11 14:29:12 2024] RBP: 0000000000000000 R08: 0000000000000000 R09: 00000000ffffffff
[Mon Mar 11 14:29:12 2024] R10: 00007f3350ec0dd0 R11: 0000000000000246 R12: 0000000000000000
[Mon Mar 11 14:29:12 2024] R13: 0000560fc6c6bce8 R14: 0000000000000000 R15: 0000560fc6c6bd00
[Mon Mar 11 14:29:12 2024] </TASK>
[Mon Mar 11 14:29:12 2024] task:reader#1 state:S stack:0 pid:5334 ppid:1 flags:0x00000002
[Mon Mar 11 14:29:12 2024] Call Trace:
[Mon Mar 11 14:29:12 2024] <TASK>
[Mon Mar 11 14:29:12 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:12 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:12 2024] futex_wait_queue+0x70/0xd0
[Mon Mar 11 14:29:12 2024] futex_wait+0x175/0x260
[Mon Mar 11 14:29:12 2024] ? __pfx_hrtimer_wakeup+0x10/0x10
[Mon Mar 11 14:29:12 2024] do_futex+0x12d/0x1d0
[Mon Mar 11 14:29:12 2024] __x64_sys_futex+0x73/0x1d0
[Mon Mar 11 14:29:12 2024] do_syscall_64+0x59/0x90
[Mon Mar 11 14:29:12 2024] ? exit_to_user_mode_prepare+0xb6/0x100
[Mon Mar 11 14:29:12 2024] ? syscall_exit_to_user_mode+0x22/0x40
[Mon Mar 11 14:29:12 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:12 2024] ? __irq_exit_rcu+0x46/0xc0
[Mon Mar 11 14:29:12 2024] ? sysvec_apic_timer_interrupt+0x3c/0x90
[Mon Mar 11 14:29:12 2024] entry_SYSCALL_64_after_hwframe+0x72/0xdc
[Mon Mar 11 14:29:12 2024] RIP: 0033:0x7f335389c39a
[Mon Mar 11 14:29:12 2024] RSP: 002b:00007f33506bfca0 EFLAGS: 00000246 ORIG_RAX: 00000000000000ca
[Mon Mar 11 14:29:12 2024] RAX: ffffffffffffffda RBX: 00007f33506bfdd0 RCX: 00007f335389c39a
[Mon Mar 11 14:29:12 2024] RDX: 0000000000000000 RSI: 0000000000000189 RDI: 0000560fc6c6bce8
[Mon Mar 11 14:29:12 2024] RBP: 0000000000000000 R08: 0000000000000000 R09: 00000000ffffffff
[Mon Mar 11 14:29:12 2024] R10: 00007f33506bfdd0 R11: 0000000000000246 R12: 0000000000000000
[Mon Mar 11 14:29:12 2024] R13: 0000560fc6c6bce8 R14: 0000000000000000 R15: 0000560fc6c6bd00
[Mon Mar 11 14:29:12 2024] </TASK>
[Mon Mar 11 14:29:12 2024] task:reader#2 state:S stack:0 pid:5335 ppid:1 flags:0x00000002
[Mon Mar 11 14:29:12 2024] Call Trace:
[Mon Mar 11 14:29:12 2024] <TASK>
[Mon Mar 11 14:29:12 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:12 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:12 2024] futex_wait_queue+0x70/0xd0
[Mon Mar 11 14:29:12 2024] futex_wait+0x175/0x260
[Mon Mar 11 14:29:12 2024] ? __pfx_hrtimer_wakeup+0x10/0x10
[Mon Mar 11 14:29:12 2024] do_futex+0x12d/0x1d0
[Mon Mar 11 14:29:12 2024] __x64_sys_futex+0x73/0x1d0
[Mon Mar 11 14:29:12 2024] do_syscall_64+0x59/0x90
[Mon Mar 11 14:29:12 2024] ? syscall_exit_work+0x103/0x130
[Mon Mar 11 14:29:12 2024] ? syscall_exit_to_user_mode+0x22/0x40
[Mon Mar 11 14:29:12 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:12 2024] ? sysvec_apic_timer_interrupt+0x3c/0x90
[Mon Mar 11 14:29:12 2024] entry_SYSCALL_64_after_hwframe+0x72/0xdc
[Mon Mar 11 14:29:12 2024] RIP: 0033:0x7f335389c39a
[Mon Mar 11 14:29:12 2024] RSP: 002b:00007f334febeca0 EFLAGS: 00000246 ORIG_RAX: 00000000000000ca
[Mon Mar 11 14:29:12 2024] RAX: ffffffffffffffda RBX: 00007f334febedd0 RCX: 00007f335389c39a
[Mon Mar 11 14:29:12 2024] RDX: 0000000000000000 RSI: 0000000000000189 RDI: 0000560fc6c6bce8
[Mon Mar 11 14:29:12 2024] RBP: 0000000000000000 R08: 0000000000000000 R09: 00000000ffffffff
[Mon Mar 11 14:29:12 2024] R10: 00007f334febedd0 R11: 0000000000000246 R12: 0000000000000000
[Mon Mar 11 14:29:12 2024] R13: 0000560fc6c6bce8 R14: 0000000000000000 R15: 0000560fc6c6bd00
[Mon Mar 11 14:29:12 2024] </TASK>
[Mon Mar 11 14:29:12 2024] task:reader#3 state:S stack:0 pid:5336 ppid:1 flags:0x00000002
[Mon Mar 11 14:29:12 2024] Call Trace:
[Mon Mar 11 14:29:12 2024] <TASK>
[Mon Mar 11 14:29:12 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:12 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:12 2024] futex_wait_queue+0x70/0xd0
[Mon Mar 11 14:29:12 2024] futex_wait+0x175/0x260
[Mon Mar 11 14:29:12 2024] ? __pfx_hrtimer_wakeup+0x10/0x10
[Mon Mar 11 14:29:12 2024] do_futex+0x12d/0x1d0
[Mon Mar 11 14:29:12 2024] __x64_sys_futex+0x73/0x1d0
[Mon Mar 11 14:29:12 2024] do_syscall_64+0x59/0x90
[Mon Mar 11 14:29:12 2024] ? syscall_exit_to_user_mode+0x22/0x40
[Mon Mar 11 14:29:12 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:12 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:12 2024] ? syscall_exit_to_user_mode+0x22/0x40
[Mon Mar 11 14:29:12 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:12 2024] ? syscall_exit_to_user_mode+0x22/0x40
[Mon Mar 11 14:29:12 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:12 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:12 2024] ? syscall_exit_to_user_mode+0x22/0x40
[Mon Mar 11 14:29:12 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:12 2024] ? __irq_exit_rcu+0x46/0xc0
[Mon Mar 11 14:29:12 2024] ? sysvec_apic_timer_interrupt+0x3c/0x90
[Mon Mar 11 14:29:12 2024] entry_SYSCALL_64_after_hwframe+0x72/0xdc
[Mon Mar 11 14:29:12 2024] RIP: 0033:0x7f335389c39a
[Mon Mar 11 14:29:12 2024] RSP: 002b:00007f334f6bdca0 EFLAGS: 00000246 ORIG_RAX: 00000000000000ca
[Mon Mar 11 14:29:12 2024] RAX: ffffffffffffffda RBX: 00007f334f6bddd0 RCX: 00007f335389c39a
[Mon Mar 11 14:29:12 2024] RDX: 0000000000000000 RSI: 0000000000000189 RDI: 0000560fc6c6bce8
[Mon Mar 11 14:29:12 2024] RBP: 0000000000000000 R08: 0000000000000000 R09: 00000000ffffffff
[Mon Mar 11 14:29:12 2024] R10: 00007f334f6bddd0 R11: 0000000000000246 R12: 0000000000000000
[Mon Mar 11 14:29:12 2024] R13: 0000560fc6c6bce8 R14: 0000000000000000 R15: 0000560fc6c6bd00
[Mon Mar 11 14:29:12 2024] </TASK>
[Mon Mar 11 14:29:12 2024] task:reader#4 state:S stack:0 pid:5337 ppid:1 flags:0x00000002
[Mon Mar 11 14:29:12 2024] Call Trace:
[Mon Mar 11 14:29:12 2024] <TASK>
[Mon Mar 11 14:29:12 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:12 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:12 2024] futex_wait_queue+0x70/0xd0
[Mon Mar 11 14:29:12 2024] futex_wait+0x175/0x260
[Mon Mar 11 14:29:12 2024] ? __pfx_hrtimer_wakeup+0x10/0x10
[Mon Mar 11 14:29:12 2024] do_futex+0x12d/0x1d0
[Mon Mar 11 14:29:12 2024] __x64_sys_futex+0x73/0x1d0
[Mon Mar 11 14:29:12 2024] do_syscall_64+0x59/0x90
[Mon Mar 11 14:29:12 2024] ? syscall_exit_work+0x103/0x130
[Mon Mar 11 14:29:12 2024] ? syscall_exit_to_user_mode+0x22/0x40
[Mon Mar 11 14:29:12 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:12 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:12 2024] ? sysvec_apic_timer_interrupt+0x3c/0x90
[Mon Mar 11 14:29:12 2024] entry_SYSCALL_64_after_hwframe+0x72/0xdc
[Mon Mar 11 14:29:12 2024] RIP: 0033:0x7f335389c39a
[Mon Mar 11 14:29:12 2024] RSP: 002b:00007f334eebcca0 EFLAGS: 00000246 ORIG_RAX: 00000000000000ca
[Mon Mar 11 14:29:12 2024] RAX: ffffffffffffffda RBX: 00007f334eebcdd0 RCX: 00007f335389c39a
[Mon Mar 11 14:29:12 2024] RDX: 0000000000000000 RSI: 0000000000000189 RDI: 0000560fc6c6bce8
[Mon Mar 11 14:29:12 2024] RBP: 0000000000000000 R08: 0000000000000000 R09: 00000000ffffffff
[Mon Mar 11 14:29:12 2024] R10: 00007f334eebcdd0 R11: 0000000000000246 R12: 0000000000000000
[Mon Mar 11 14:29:12 2024] R13: 0000560fc6c6bce8 R14: 0000000000000000 R15: 0000560fc6c6bd00
[Mon Mar 11 14:29:12 2024] </TASK>
[Mon Mar 11 14:29:12 2024] task:nrpe state:S stack:0 pid:4740 ppid:1 flags:0x00000002
[Mon Mar 11 14:29:12 2024] Call Trace:
[Mon Mar 11 14:29:12 2024] <TASK>
[Mon Mar 11 14:29:12 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:12 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:12 2024] schedule_hrtimeout_range_clock+0x10f/0x120
[Mon Mar 11 14:29:12 2024] ? add_wait_queue+0x65/0xa0
[Mon Mar 11 14:29:12 2024] ? tcp_poll+0x1ce/0x370
[Mon Mar 11 14:29:12 2024] ? sock_poll+0x4c/0xe0
[Mon Mar 11 14:29:12 2024] do_select+0x69e/0x7c0
[Mon Mar 11 14:29:12 2024] ? __pfx_pollwake+0x10/0x10
[Mon Mar 11 14:29:12 2024] ? __pfx_pollwake+0x10/0x10
[Mon Mar 11 14:29:12 2024] ? rmqueue_pcplist+0xda/0x210
[Mon Mar 11 14:29:12 2024] ? rmqueue+0x7d3/0xd40
[Mon Mar 11 14:29:12 2024] ? do_wp_page+0x381/0x540
[Mon Mar 11 14:29:12 2024] ? __handle_mm_fault+0x32b/0x670
[Mon Mar 11 14:29:12 2024] core_sys_select+0x1a0/0x3b0
[Mon Mar 11 14:29:12 2024] ? exc_page_fault+0x62/0x150
[Mon Mar 11 14:29:12 2024] ? asm_exc_page_fault+0x22/0x30
[Mon Mar 11 14:29:12 2024] ? do_wp_page+0x381/0x540
[Mon Mar 11 14:29:12 2024] ? __handle_mm_fault+0x32b/0x670
[Mon Mar 11 14:29:12 2024] do_pselect.constprop.0+0xca/0x170
[Mon Mar 11 14:29:12 2024] __x64_sys_pselect6+0x48/0x70
[Mon Mar 11 14:29:12 2024] do_syscall_64+0x59/0x90
[Mon Mar 11 14:29:12 2024] ? handle_mm_fault+0xcd/0x290
[Mon Mar 11 14:29:12 2024] ? do_user_addr_fault+0x1d6/0x6a0
[Mon Mar 11 14:29:12 2024] ? syscall_exit_to_user_mode+0x22/0x40
[Mon Mar 11 14:29:12 2024] ? exc_page_fault+0x62/0x150
[Mon Mar 11 14:29:12 2024] entry_SYSCALL_64_after_hwframe+0x72/0xdc
[Mon Mar 11 14:29:12 2024] RIP: 0033:0x7fb70e344dc9
[Mon Mar 11 14:29:12 2024] RSP: 002b:00007ffcc3211280 EFLAGS: 00000246 ORIG_RAX: 000000000000010e
[Mon Mar 11 14:29:12 2024] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007fb70e344dc9
[Mon Mar 11 14:29:12 2024] RDX: 0000000000000000 RSI: 0000562bfde0a920 RDI: 0000000000000006
[Mon Mar 11 14:29:12 2024] RBP: 0000562bfde0a920 R08: 0000000000000000 R09: 0000000000000000
[Mon Mar 11 14:29:12 2024] R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000000006
[Mon Mar 11 14:29:12 2024] R13: 0000000000000000 R14: 0000562bfcae0da0 R15: 0000000000000000
[Mon Mar 11 14:29:12 2024] </TASK>
[Mon Mar 11 14:29:12 2024] task:wb[ESAT] state:S stack:0 pid:4744 ppid:4415 flags:0x00000002
[Mon Mar 11 14:29:12 2024] Call Trace:
[Mon Mar 11 14:29:12 2024] <TASK>
[Mon Mar 11 14:29:12 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:12 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:12 2024] schedule_hrtimeout_range_clock+0x95/0x120
[Mon Mar 11 14:29:12 2024] ? __pfx_hrtimer_wakeup+0x10/0x10
[Mon Mar 11 14:29:12 2024] ep_poll+0x348/0x3b0
[Mon Mar 11 14:29:12 2024] ? __pfx_autoremove_wake_function+0x10/0x10
[Mon Mar 11 14:29:12 2024] do_epoll_wait+0xb1/0xd0
[Mon Mar 11 14:29:12 2024] __x64_sys_epoll_wait+0x60/0x100
[Mon Mar 11 14:29:12 2024] do_syscall_64+0x59/0x90
[Mon Mar 11 14:29:12 2024] ? audit_reset_context.part.0.constprop.0+0x273/0x2e0
[Mon Mar 11 14:29:12 2024] ? syscall_exit_work+0x103/0x130
[Mon Mar 11 14:29:12 2024] ? syscall_exit_to_user_mode+0x22/0x40
[Mon Mar 11 14:29:12 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:12 2024] ? syscall_exit_work+0x103/0x130
[Mon Mar 11 14:29:12 2024] ? syscall_exit_to_user_mode+0x22/0x40
[Mon Mar 11 14:29:12 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:12 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:12 2024] entry_SYSCALL_64_after_hwframe+0x72/0xdc
[Mon Mar 11 14:29:12 2024] RIP: 0033:0x7f26a134e80a
[Mon Mar 11 14:29:12 2024] RSP: 002b:00007ffe9e332b58 EFLAGS: 00000246 ORIG_RAX: 00000000000000e8
[Mon Mar 11 14:29:12 2024] RAX: ffffffffffffffda RBX: 00005586337521b0 RCX: 00007f26a134e80a
[Mon Mar 11 14:29:12 2024] RDX: 0000000000000001 RSI: 00007ffe9e332b8c RDI: 0000000000000005
[Mon Mar 11 14:29:12 2024] RBP: 00005586337422e0 R08: 0000558633785180 R09: 0000000000004a9c
[Mon Mar 11 14:29:12 2024] R10: 0000000000007530 R11: 0000000000000246 R12: 000055863375b3a0
[Mon Mar 11 14:29:12 2024] R13: 0000000000007530 R14: 0000558632a302f8 R15: 0000558632a4c020
[Mon Mar 11 14:29:12 2024] </TASK>
[Mon Mar 11 14:29:12 2024] task:rsyslogd state:S stack:0 pid:4752 ppid:1 flags:0x00000002
[Mon Mar 11 14:29:12 2024] Call Trace:
[Mon Mar 11 14:29:12 2024] <TASK>
[Mon Mar 11 14:29:12 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:12 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:12 2024] schedule_hrtimeout_range_clock+0x95/0x120
[Mon Mar 11 14:29:12 2024] ? __pfx_hrtimer_wakeup+0x10/0x10
[Mon Mar 11 14:29:12 2024] do_select+0x69e/0x7c0
[Mon Mar 11 14:29:12 2024] ? __pfx_hrtimer_wakeup+0x10/0x10
[Mon Mar 11 14:29:12 2024] ? do_select+0x719/0x7c0
[Mon Mar 11 14:29:12 2024] ? do_select+0x719/0x7c0
[Mon Mar 11 14:29:12 2024] ? task_scan_max+0x12c/0x180
[Mon Mar 11 14:29:12 2024] ? update_task_scan_period+0xce/0x180
[Mon Mar 11 14:29:12 2024] ? node_is_toptier+0x3e/0x60
[Mon Mar 11 14:29:12 2024] ? task_numa_fault+0x6d/0x340
[Mon Mar 11 14:29:12 2024] ? do_numa_page+0x29a/0x460
[Mon Mar 11 14:29:12 2024] ? __handle_mm_fault+0x32b/0x670
[Mon Mar 11 14:29:12 2024] ? task_numa_fault+0x6d/0x340
[Mon Mar 11 14:29:12 2024] core_sys_select+0x1a0/0x3b0
[Mon Mar 11 14:29:12 2024] ? _copy_from_user+0x27/0x60
[Mon Mar 11 14:29:12 2024] ? _copy_from_user+0x27/0x60
[Mon Mar 11 14:29:12 2024] do_pselect.constprop.0+0xca/0x170
[Mon Mar 11 14:29:12 2024] __x64_sys_pselect6+0x39/0x70
[Mon Mar 11 14:29:12 2024] do_syscall_64+0x59/0x90
[Mon Mar 11 14:29:12 2024] ? audit_reset_context.part.0.constprop.0+0x273/0x2e0
[Mon Mar 11 14:29:12 2024] ? syscall_exit_work+0x103/0x130
[Mon Mar 11 14:29:12 2024] ? syscall_exit_to_user_mode+0x22/0x40
[Mon Mar 11 14:29:12 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:12 2024] ? exc_page_fault+0x62/0x150
[Mon Mar 11 14:29:12 2024] entry_SYSCALL_64_after_hwframe+0x72/0xdc
[Mon Mar 11 14:29:12 2024] RIP: 0033:0x7f0682b44fd2
[Mon Mar 11 14:29:12 2024] RSP: 002b:00007ffde7a1ca20 EFLAGS: 00000293 ORIG_RAX: 000000000000010e
[Mon Mar 11 14:29:12 2024] RAX: ffffffffffffffda RBX: 00007ffde7a1cac0 RCX: 00007f0682b44fd2
[Mon Mar 11 14:29:12 2024] RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000000
[Mon Mar 11 14:29:12 2024] RBP: 00007ffde7a1cad0 R08: 00007ffde7a1ca50 R09: 00007ffde7a1ca60
[Mon Mar 11 14:29:12 2024] R10: 0000000000000000 R11: 0000000000000293 R12: 0000000000000000
[Mon Mar 11 14:29:12 2024] R13: 000055c79f7f83a0 R14: 0000000000000000 R15: 0000000000000000
[Mon Mar 11 14:29:12 2024] </TASK>
[Mon Mar 11 14:29:12 2024] task:in:imjournal state:D stack:0 pid:4837 ppid:1 flags:0x00000002
[Mon Mar 11 14:29:12 2024] Call Trace:
[Mon Mar 11 14:29:12 2024] <TASK>
[Mon Mar 11 14:29:12 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:12 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:12 2024] schedule_preempt_disabled+0x11/0x20
[Mon Mar 11 14:29:12 2024] rwsem_down_write_slowpath+0x23d/0x500
[Mon Mar 11 14:29:12 2024] down_write_killable+0x58/0x80
[Mon Mar 11 14:29:12 2024] do_mprotect_pkey+0xc5/0x400
[Mon Mar 11 14:29:12 2024] ? __seccomp_filter+0x45/0x480
[Mon Mar 11 14:29:12 2024] ? auditd_test_task+0x3c/0x50
[Mon Mar 11 14:29:12 2024] ? __audit_syscall_entry+0xef/0x140
[Mon Mar 11 14:29:12 2024] __x64_sys_mprotect+0x1b/0x30
[Mon Mar 11 14:29:12 2024] do_syscall_64+0x59/0x90
[Mon Mar 11 14:29:12 2024] ? switch_fpu_return+0x4c/0xd0
[Mon Mar 11 14:29:12 2024] ? exit_to_user_mode_prepare+0xec/0x100
[Mon Mar 11 14:29:12 2024] ? syscall_exit_to_user_mode+0x22/0x40
[Mon Mar 11 14:29:12 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:12 2024] ? exc_page_fault+0x62/0x150
[Mon Mar 11 14:29:12 2024] entry_SYSCALL_64_after_hwframe+0x72/0xdc
[Mon Mar 11 14:29:12 2024] RIP: 0033:0x7f0682a3eebb
[Mon Mar 11 14:29:12 2024] RSP: 002b:00007f0681c41898 EFLAGS: 00000206 ORIG_RAX: 000000000000000a
[Mon Mar 11 14:29:12 2024] RAX: ffffffffffffffda RBX: 00007f0674000020 RCX: 00007f0682a3eebb
[Mon Mar 11 14:29:12 2024] RDX: 0000000000000003 RSI: 0000000000001000 RDI: 00007f0674cb9000
[Mon Mar 11 14:29:12 2024] RBP: 0000000000001010 R08: 0000000000cb9000 R09: 0000000000cba000
[Mon Mar 11 14:29:12 2024] R10: 0000000000001030 R11: 0000000000000206 R12: 00000000000003e0
[Mon Mar 11 14:29:12 2024] R13: 0000000000001000 R14: 00007f0674cb8c20 R15: fffffffffffff000
[Mon Mar 11 14:29:12 2024] </TASK>
[Mon Mar 11 14:29:12 2024] task:in:imfile state:S stack:0 pid:4838 ppid:1 flags:0x00000002
[Mon Mar 11 14:29:12 2024] Call Trace:
[Mon Mar 11 14:29:12 2024] <TASK>
[Mon Mar 11 14:29:12 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:12 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:12 2024] schedule_timeout+0x11f/0x160
[Mon Mar 11 14:29:12 2024] ? add_wait_queue+0x65/0xa0
[Mon Mar 11 14:29:12 2024] wait_woken+0x50/0x60
[Mon Mar 11 14:29:12 2024] inotify_read+0x1e8/0x280
[Mon Mar 11 14:29:12 2024] ? __pfx_woken_wake_function+0x10/0x10
[Mon Mar 11 14:29:12 2024] vfs_read+0xa4/0x330
[Mon Mar 11 14:29:12 2024] ? __seccomp_filter+0x45/0x480
[Mon Mar 11 14:29:12 2024] ? auditd_test_task+0x3c/0x50
[Mon Mar 11 14:29:12 2024] ? __fget_light+0x9f/0x130
[Mon Mar 11 14:29:12 2024] ksys_read+0x5f/0xe0
[Mon Mar 11 14:29:12 2024] do_syscall_64+0x59/0x90
[Mon Mar 11 14:29:12 2024] ? mntput_no_expire+0x4a/0x250
[Mon Mar 11 14:29:12 2024] ? audit_reset_context.part.0.constprop.0+0x273/0x2e0
[Mon Mar 11 14:29:12 2024] ? syscall_exit_work+0x103/0x130
[Mon Mar 11 14:29:12 2024] ? syscall_exit_to_user_mode+0x22/0x40
[Mon Mar 11 14:29:12 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:12 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:12 2024] ? syscall_exit_to_user_mode+0x22/0x40
[Mon Mar 11 14:29:12 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:12 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:12 2024] ? __irq_exit_rcu+0x46/0xc0
[Mon Mar 11 14:29:12 2024] ? sysvec_apic_timer_interrupt+0x3c/0x90
[Mon Mar 11 14:29:12 2024] entry_SYSCALL_64_after_hwframe+0x72/0xdc
[Mon Mar 11 14:29:12 2024] RIP: 0033:0x7f0682b3e8bc
[Mon Mar 11 14:29:12 2024] RSP: 002b:00007f067bffcb90 EFLAGS: 00000246 ORIG_RAX: 0000000000000000
[Mon Mar 11 14:29:12 2024] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f0682b3e8bc
[Mon Mar 11 14:29:12 2024] RDX: 0000000000002000 RSI: 00007f067bffcc00 RDI: 0000000000000005
[Mon Mar 11 14:29:12 2024] RBP: 00007f066c000f20 R08: 0000000000000000 R09: 0000000000000000
[Mon Mar 11 14:29:12 2024] R10: 0000000000000000 R11: 0000000000000246 R12: 00007f067bffcc00
[Mon Mar 11 14:29:12 2024] R13: 000055c79f7f844c R14: 0000000000000000 R15: 00007f0682918025
[Mon Mar 11 14:29:12 2024] </TASK>
[Mon Mar 11 14:29:12 2024] task:rs:main Q:Reg state:D stack:0 pid:4840 ppid:1 flags:0x00004002
[Mon Mar 11 14:29:12 2024] Call Trace:
[Mon Mar 11 14:29:12 2024] <TASK>
[Mon Mar 11 14:29:12 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:12 2024] ? pick_next_task+0x4d4/0x950
[Mon Mar 11 14:29:12 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:12 2024] schedule_timeout+0x11f/0x160
[Mon Mar 11 14:29:12 2024] __wait_for_common+0x90/0x1d0
[Mon Mar 11 14:29:12 2024] ? __pfx_schedule_timeout+0x10/0x10
[Mon Mar 11 14:29:12 2024] stop_two_cpus+0x177/0x1b0
[Mon Mar 11 14:29:12 2024] ? __pfx_migrate_swap_stop+0x10/0x10
[Mon Mar 11 14:29:12 2024] ? __pfx_multi_cpu_stop+0x10/0x10
[Mon Mar 11 14:29:12 2024] ? migrate_swap+0xb4/0x110
[Mon Mar 11 14:29:12 2024] ? __pfx_multi_cpu_stop+0x10/0x10
[Mon Mar 11 14:29:12 2024] ? migrate_swap+0xb4/0x110
[Mon Mar 11 14:29:12 2024] migrate_swap+0xb4/0x110
[Mon Mar 11 14:29:12 2024] task_numa_migrate.isra.0+0x65a/0x950
[Mon Mar 11 14:29:12 2024] ? set_next_task_fair+0x2d/0xd0
[Mon Mar 11 14:29:12 2024] task_numa_fault+0x28d/0x340
[Mon Mar 11 14:29:12 2024] do_numa_page+0x29a/0x460
[Mon Mar 11 14:29:12 2024] __handle_mm_fault+0x32b/0x670
[Mon Mar 11 14:29:12 2024] handle_mm_fault+0xcd/0x290
[Mon Mar 11 14:29:12 2024] do_user_addr_fault+0x1b4/0x6a0
[Mon Mar 11 14:29:12 2024] exc_page_fault+0x62/0x150
[Mon Mar 11 14:29:12 2024] asm_exc_page_fault+0x22/0x30
[Mon Mar 11 14:29:12 2024] RIP: 0010:__get_user_8+0xd/0x20
[Mon Mar 11 14:29:12 2024] Code: ca c3 cc cc cc cc 0f 1f 80 00 00 00 00 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 48 89 c2 48 c1 fa 3f 48 09 d0 0f 01 cb <48> 8b 10 31 c0 0f 01 ca c3 cc cc cc cc 66 0f 1f 44 00 00 90 90 90
[Mon Mar 11 14:29:12 2024] RSP: 0018:ffff9929e08dfe10 EFLAGS: 00050206
[Mon Mar 11 14:29:12 2024] RAX: 00007f0681842fe8 RBX: ffff8aeb9d648000 RCX: 0000000000000000
[Mon Mar 11 14:29:12 2024] RDX: 0000000000000000 RSI: ffff9929e08dfe40 RDI: ffff8aeb9d648000
[Mon Mar 11 14:29:12 2024] RBP: ffff9929e08dfeb0 R08: ffff9929e08dfe94 R09: ffff8ad4d10c9540
[Mon Mar 11 14:29:12 2024] R10: 000000000000003e R11: 0000000000000000 R12: 0000000000000002
[Mon Mar 11 14:29:12 2024] R13: 00007f0682a9c39a R14: ffff9929e08dff58 R15: 0000000000000000
[Mon Mar 11 14:29:12 2024] rseq_get_rseq_cs+0x1d/0x240
[Mon Mar 11 14:29:12 2024] rseq_ip_fixup+0x46/0x1a0
[Mon Mar 11 14:29:12 2024] ? __audit_filter_op+0xa5/0xf0
[Mon Mar 11 14:29:12 2024] __rseq_handle_notify_resume+0x26/0xb0
[Mon Mar 11 14:29:12 2024] exit_to_user_mode_loop+0xd0/0x130
[Mon Mar 11 14:29:12 2024] exit_to_user_mode_prepare+0xb6/0x100
[Mon Mar 11 14:29:12 2024] syscall_exit_to_user_mode+0x12/0x40
[Mon Mar 11 14:29:12 2024] do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:12 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:12 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:12 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:12 2024] ? sysvec_apic_timer_interrupt+0x3c/0x90
[Mon Mar 11 14:29:12 2024] entry_SYSCALL_64_after_hwframe+0x72/0xdc
[Mon Mar 11 14:29:12 2024] RIP: 0033:0x7f0682a9c39a
[Mon Mar 11 14:29:12 2024] RSP: 002b:00007f0681841b00 EFLAGS: 00000246 ORIG_RAX: 00000000000000ca
[Mon Mar 11 14:29:12 2024] RAX: 0000000000000000 RBX: 0000000000000000 RCX: 00007f0682a9c39a
[Mon Mar 11 14:29:12 2024] RDX: 0000000000000000 RSI: 0000000000000189 RDI: 000055c79fd7bd20
[Mon Mar 11 14:29:12 2024] RBP: 0000000000000001 R08: 0000000000000000 R09: 00000000ffffffff
[Mon Mar 11 14:29:12 2024] R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000000000
[Mon Mar 11 14:29:12 2024] R13: 000055c79fd7bd20 R14: 0000000000000000 R15: 0000000000000000
[Mon Mar 11 14:29:12 2024] </TASK>
[Mon Mar 11 14:29:12 2024] task:rs:action-0-bui state:S stack:0 pid:4841 ppid:1 flags:0x00000002
[Mon Mar 11 14:29:12 2024] Call Trace:
[Mon Mar 11 14:29:12 2024] <TASK>
[Mon Mar 11 14:29:12 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:12 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:12 2024] futex_wait_queue+0x70/0xd0
[Mon Mar 11 14:29:12 2024] futex_wait+0x175/0x260
[Mon Mar 11 14:29:12 2024] ? release_sock+0x40/0x90
[Mon Mar 11 14:29:12 2024] ? tcp_sendmsg+0x33/0x40
[Mon Mar 11 14:29:12 2024] do_futex+0x12d/0x1d0
[Mon Mar 11 14:29:12 2024] __x64_sys_futex+0x73/0x1d0
[Mon Mar 11 14:29:12 2024] do_syscall_64+0x59/0x90
[Mon Mar 11 14:29:12 2024] ? syscall_exit_to_user_mode+0x22/0x40
[Mon Mar 11 14:29:12 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:12 2024] ? syscall_exit_to_user_mode+0x22/0x40
[Mon Mar 11 14:29:12 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:12 2024] ? syscall_exit_to_user_mode+0x22/0x40
[Mon Mar 11 14:29:12 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:12 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:12 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:12 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:12 2024] ? sysvec_apic_timer_interrupt+0x3c/0x90
[Mon Mar 11 14:29:12 2024] entry_SYSCALL_64_after_hwframe+0x72/0xdc
[Mon Mar 11 14:29:12 2024] RIP: 0033:0x7f0682a9c39a
[Mon Mar 11 14:29:12 2024] RSP: 002b:00007f0681040b00 EFLAGS: 00000246 ORIG_RAX: 00000000000000ca
[Mon Mar 11 14:29:12 2024] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f0682a9c39a
[Mon Mar 11 14:29:12 2024] RDX: 0000000000000000 RSI: 0000000000000189 RDI: 000055c79fd68b74
[Mon Mar 11 14:29:12 2024] RBP: 0000000000000001 R08: 0000000000000000 R09: 00000000ffffffff
[Mon Mar 11 14:29:12 2024] R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000000000
[Mon Mar 11 14:29:12 2024] R13: 000055c79fd68b74 R14: 0000000000000001 R15: 0000000000000000
[Mon Mar 11 14:29:12 2024] </TASK>
[Mon Mar 11 14:29:12 2024] task:rs:action-1-bui state:S stack:0 pid:4867 ppid:1 flags:0x00000002
[Mon Mar 11 14:29:12 2024] Call Trace:
[Mon Mar 11 14:29:12 2024] <TASK>
[Mon Mar 11 14:29:12 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:12 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:12 2024] futex_wait_queue+0x70/0xd0
[Mon Mar 11 14:29:12 2024] futex_wait+0x175/0x260
[Mon Mar 11 14:29:12 2024] ? __tcp_push_pending_frames+0x32/0xf0
[Mon Mar 11 14:29:12 2024] ? tcp_sendmsg_locked+0xaae/0xc20
[Mon Mar 11 14:29:12 2024] do_futex+0x12d/0x1d0
[Mon Mar 11 14:29:12 2024] __x64_sys_futex+0x73/0x1d0
[Mon Mar 11 14:29:12 2024] do_syscall_64+0x59/0x90
[Mon Mar 11 14:29:12 2024] ? __audit_filter_op+0xa5/0xf0
[Mon Mar 11 14:29:12 2024] ? audit_reset_context.part.0.constprop.0+0x273/0x2e0
[Mon Mar 11 14:29:12 2024] ? syscall_exit_work+0x103/0x130
[Mon Mar 11 14:29:12 2024] ? syscall_exit_to_user_mode+0x22/0x40
[Mon Mar 11 14:29:12 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:12 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:12 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:12 2024] ? sysvec_apic_timer_interrupt+0x3c/0x90
[Mon Mar 11 14:29:12 2024] entry_SYSCALL_64_after_hwframe+0x72/0xdc
[Mon Mar 11 14:29:12 2024] RIP: 0033:0x7f0682a9c39a
[Mon Mar 11 14:29:12 2024] RSP: 002b:00007f067b3fcb00 EFLAGS: 00000246 ORIG_RAX: 00000000000000ca
[Mon Mar 11 14:29:12 2024] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f0682a9c39a
[Mon Mar 11 14:29:12 2024] RDX: 0000000000000000 RSI: 0000000000000189 RDI: 000055c79fd723c0
[Mon Mar 11 14:29:12 2024] RBP: 0000000000000001 R08: 0000000000000000 R09: 00000000ffffffff
[Mon Mar 11 14:29:12 2024] R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000000000
[Mon Mar 11 14:29:12 2024] R13: 000055c79fd723c0 R14: 0000000000000000 R15: 0000000000000000
[Mon Mar 11 14:29:12 2024] </TASK>
[Mon Mar 11 14:29:12 2024] task:smbd state:S stack:0 pid:4757 ppid:1 flags:0x00000002
[Mon Mar 11 14:29:12 2024] Call Trace:
[Mon Mar 11 14:29:12 2024] <TASK>
[Mon Mar 11 14:29:12 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:12 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:12 2024] schedule_hrtimeout_range_clock+0x95/0x120
[Mon Mar 11 14:29:12 2024] ? __pfx_hrtimer_wakeup+0x10/0x10
[Mon Mar 11 14:29:12 2024] ep_poll+0x348/0x3b0
[Mon Mar 11 14:29:12 2024] ? __pfx_autoremove_wake_function+0x10/0x10
[Mon Mar 11 14:29:12 2024] do_epoll_wait+0xb1/0xd0
[Mon Mar 11 14:29:12 2024] __x64_sys_epoll_wait+0x60/0x100
[Mon Mar 11 14:29:12 2024] do_syscall_64+0x59/0x90
[Mon Mar 11 14:29:12 2024] ? do_user_addr_fault+0x1d6/0x6a0
[Mon Mar 11 14:29:12 2024] ? exc_page_fault+0x62/0x150
[Mon Mar 11 14:29:12 2024] entry_SYSCALL_64_after_hwframe+0x72/0xdc
[Mon Mar 11 14:29:12 2024] RIP: 0033:0x7fedb7d4e80a
[Mon Mar 11 14:29:12 2024] RSP: 002b:00007ffc284a9da8 EFLAGS: 00000246 ORIG_RAX: 00000000000000e8
[Mon Mar 11 14:29:12 2024] RAX: ffffffffffffffda RBX: 000055f87dd397c0 RCX: 00007fedb7d4e80a
[Mon Mar 11 14:29:12 2024] RDX: 0000000000000001 RSI: 00007ffc284a9ddc RDI: 0000000000000005
[Mon Mar 11 14:29:12 2024] RBP: 000055f87dd2aa20 R08: 000000000006012f R09: 00007ffc284a9790
[Mon Mar 11 14:29:12 2024] R10: 00000000000bf80a R11: 0000000000000246 R12: 000055f87dd39850
[Mon Mar 11 14:29:12 2024] R13: 00000000000bf80a R14: 000055f87dd37590 R15: 00000000000001bd
[Mon Mar 11 14:29:12 2024] </TASK>
[Mon Mar 11 14:29:12 2024] task:smbd-notifyd state:S stack:0 pid:4819 ppid:4757 flags:0x00000002
[Mon Mar 11 14:29:12 2024] Call Trace:
[Mon Mar 11 14:29:12 2024] <TASK>
[Mon Mar 11 14:29:12 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:12 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:12 2024] schedule_hrtimeout_range_clock+0x95/0x120
[Mon Mar 11 14:29:12 2024] ? __pfx_hrtimer_wakeup+0x10/0x10
[Mon Mar 11 14:29:12 2024] ep_poll+0x348/0x3b0
[Mon Mar 11 14:29:12 2024] ? __pfx_autoremove_wake_function+0x10/0x10
[Mon Mar 11 14:29:12 2024] do_epoll_wait+0xb1/0xd0
[Mon Mar 11 14:29:12 2024] __x64_sys_epoll_wait+0x60/0x100
[Mon Mar 11 14:29:12 2024] do_syscall_64+0x59/0x90
[Mon Mar 11 14:29:12 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:12 2024] ? syscall_exit_to_user_mode+0x22/0x40
[Mon Mar 11 14:29:12 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:12 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:12 2024] ? common_interrupt+0x43/0xa0
[Mon Mar 11 14:29:12 2024] entry_SYSCALL_64_after_hwframe+0x72/0xdc
[Mon Mar 11 14:29:12 2024] RIP: 0033:0x7fedb7d4e80a
[Mon Mar 11 14:29:12 2024] RSP: 002b:00007ffc284a9d58 EFLAGS: 00000246 ORIG_RAX: 00000000000000e8
[Mon Mar 11 14:29:12 2024] RAX: ffffffffffffffda RBX: 000055f87dd397c0 RCX: 00007fedb7d4e80a
[Mon Mar 11 14:29:12 2024] RDX: 0000000000000001 RSI: 00007ffc284a9d8c RDI: 0000000000000005
[Mon Mar 11 14:29:12 2024] RBP: 000055f87dd2aa20 R08: 000000000006974d R09: 000055f87dda1da0
[Mon Mar 11 14:29:12 2024] R10: 00000000000001b0 R11: 0000000000000246 R12: 000055f87dd39850
[Mon Mar 11 14:29:12 2024] R13: 00000000000001b0 R14: 000055f87dd375d8 R15: 000055f87bfcdda2
[Mon Mar 11 14:29:12 2024] </TASK>
[Mon Mar 11 14:29:12 2024] task:smbd-cleanupd state:S stack:0 pid:4821 ppid:4757 flags:0x00000002
[Mon Mar 11 14:29:12 2024] Call Trace:
[Mon Mar 11 14:29:12 2024] <TASK>
[Mon Mar 11 14:29:12 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:12 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:12 2024] schedule_hrtimeout_range_clock+0x95/0x120
[Mon Mar 11 14:29:12 2024] ? __pfx_hrtimer_wakeup+0x10/0x10
[Mon Mar 11 14:29:12 2024] ep_poll+0x348/0x3b0
[Mon Mar 11 14:29:12 2024] ? __pfx_autoremove_wake_function+0x10/0x10
[Mon Mar 11 14:29:12 2024] do_epoll_wait+0xb1/0xd0
[Mon Mar 11 14:29:12 2024] __x64_sys_epoll_wait+0x60/0x100
[Mon Mar 11 14:29:12 2024] do_syscall_64+0x59/0x90
[Mon Mar 11 14:29:12 2024] ? syscall_exit_to_user_mode+0x22/0x40
[Mon Mar 11 14:29:12 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:12 2024] ? syscall_exit_work+0x103/0x130
[Mon Mar 11 14:29:12 2024] ? syscall_exit_to_user_mode+0x22/0x40
[Mon Mar 11 14:29:12 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:12 2024] ? syscall_exit_to_user_mode+0x22/0x40
[Mon Mar 11 14:29:12 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:12 2024] entry_SYSCALL_64_after_hwframe+0x72/0xdc
[Mon Mar 11 14:29:12 2024] RIP: 0033:0x7fedb7d4e80a
[Mon Mar 11 14:29:12 2024] RSP: 002b:00007ffc284a9d28 EFLAGS: 00000246 ORIG_RAX: 00000000000000e8
[Mon Mar 11 14:29:12 2024] RAX: ffffffffffffffda RBX: 000055f87dd397c0 RCX: 00007fedb7d4e80a
[Mon Mar 11 14:29:12 2024] RDX: 0000000000000001 RSI: 00007ffc284a9d5c RDI: 0000000000000005
[Mon Mar 11 14:29:12 2024] RBP: 000055f87dd2aa20 R08: 000055f87dd39b90 R09: 00007ffc284a90f0
[Mon Mar 11 14:29:12 2024] R10: 0000000000007530 R11: 0000000000000246 R12: 000055f87dd39850
[Mon Mar 11 14:29:12 2024] R13: 0000000000007530 R14: 0000000000000000 R15: 0000000000001295
[Mon Mar 11 14:29:12 2024] </TASK>
[Mon Mar 11 14:29:12 2024] task:master state:S stack:0 pid:4982 ppid:1 flags:0x00000002
[Mon Mar 11 14:29:12 2024] Call Trace:
[Mon Mar 11 14:29:12 2024] <TASK>
[Mon Mar 11 14:29:12 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:12 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:12 2024] schedule_hrtimeout_range_clock+0x95/0x120
[Mon Mar 11 14:29:12 2024] ? __pfx_hrtimer_wakeup+0x10/0x10
[Mon Mar 11 14:29:12 2024] ep_poll+0x348/0x3b0
[Mon Mar 11 14:29:12 2024] ? __pfx_autoremove_wake_function+0x10/0x10
[Mon Mar 11 14:29:12 2024] do_epoll_wait+0xb1/0xd0
[Mon Mar 11 14:29:12 2024] __x64_sys_epoll_wait+0x60/0x100
[Mon Mar 11 14:29:12 2024] do_syscall_64+0x59/0x90
[Mon Mar 11 14:29:12 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:12 2024] ? syscall_exit_to_user_mode+0x22/0x40
[Mon Mar 11 14:29:12 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:12 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:12 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:12 2024] ? sysvec_apic_timer_interrupt+0x3c/0x90
[Mon Mar 11 14:29:12 2024] entry_SYSCALL_64_after_hwframe+0x72/0xdc
[Mon Mar 11 14:29:12 2024] RIP: 0033:0x7f6750b4e80a
[Mon Mar 11 14:29:12 2024] RSP: 002b:00007ffd4c37d4b8 EFLAGS: 00000246 ORIG_RAX: 00000000000000e8
[Mon Mar 11 14:29:12 2024] RAX: ffffffffffffffda RBX: 00007ffd4c37d4d0 RCX: 00007f6750b4e80a
[Mon Mar 11 14:29:12 2024] RDX: 0000000000000064 RSI: 00007ffd4c37d4d0 RDI: 0000000000000010
[Mon Mar 11 14:29:12 2024] RBP: 00007f6750e237d0 R08: 00005654ff244850 R09: 000000000000001b
[Mon Mar 11 14:29:12 2024] R10: 0000000000004268 R11: 0000000000000246 R12: 00007f6750e23794
[Mon Mar 11 14:29:12 2024] R13: 00000000000001f4 R14: 00005654fdb16821 R15: 00005654fdb167cc
[Mon Mar 11 14:29:12 2024] </TASK>
[Mon Mar 11 14:29:12 2024] task:qmgr state:S stack:0 pid:5126 ppid:4982 flags:0x00000002
[Mon Mar 11 14:29:12 2024] Call Trace:
[Mon Mar 11 14:29:12 2024] <TASK>
[Mon Mar 11 14:29:12 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:12 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:12 2024] schedule_hrtimeout_range_clock+0x95/0x120
[Mon Mar 11 14:29:12 2024] ? __pfx_hrtimer_wakeup+0x10/0x10
[Mon Mar 11 14:29:12 2024] ep_poll+0x348/0x3b0
[Mon Mar 11 14:29:12 2024] ? __pfx_autoremove_wake_function+0x10/0x10
[Mon Mar 11 14:29:12 2024] do_epoll_wait+0xb1/0xd0
[Mon Mar 11 14:29:12 2024] __x64_sys_epoll_wait+0x60/0x100
[Mon Mar 11 14:29:12 2024] do_syscall_64+0x59/0x90
[Mon Mar 11 14:29:12 2024] ? syscall_exit_work+0x103/0x130
[Mon Mar 11 14:29:12 2024] ? syscall_exit_to_user_mode+0x22/0x40
[Mon Mar 11 14:29:12 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:12 2024] ? syscall_exit_to_user_mode+0x22/0x40
[Mon Mar 11 14:29:12 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:12 2024] ? sysvec_apic_timer_interrupt+0x3c/0x90
[Mon Mar 11 14:29:12 2024] entry_SYSCALL_64_after_hwframe+0x72/0xdc
[Mon Mar 11 14:29:12 2024] RIP: 0033:0x7f638e14e80a
[Mon Mar 11 14:29:12 2024] RSP: 002b:00007fff8af04018 EFLAGS: 00000246 ORIG_RAX: 00000000000000e8
[Mon Mar 11 14:29:12 2024] RAX: ffffffffffffffda RBX: 00007fff8af04030 RCX: 00007f638e14e80a
[Mon Mar 11 14:29:12 2024] RDX: 0000000000000064 RSI: 00007fff8af04030 RDI: 000000000000000a
[Mon Mar 11 14:29:12 2024] RBP: 00007f638e2677d0 R08: 00005582908f7350 R09: 0000000000000078
[Mon Mar 11 14:29:12 2024] R10: 00000000000493e0 R11: 0000000000000246 R12: 00007f638e267794
[Mon Mar 11 14:29:12 2024] R13: 00000000ffffffff R14: 00005582900937b0 R15: 00007f638e267794
[Mon Mar 11 14:29:12 2024] </TASK>
[Mon Mar 11 14:29:12 2024] task:rpc.mountd state:S stack:0 pid:5127 ppid:1 flags:0x00004002
[Mon Mar 11 14:29:12 2024] Call Trace:
[Mon Mar 11 14:29:12 2024] <TASK>
[Mon Mar 11 14:29:12 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:12 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:12 2024] schedule_hrtimeout_range_clock+0x10f/0x120
[Mon Mar 11 14:29:12 2024] ? add_wait_queue+0x65/0xa0
[Mon Mar 11 14:29:12 2024] ? udp_poll+0x18/0xb0
[Mon Mar 11 14:29:12 2024] ? cache_poll+0x72/0xa0 [sunrpc]
[Mon Mar 11 14:29:12 2024] do_select+0x69e/0x7c0
[Mon Mar 11 14:29:12 2024] ? sock_poll+0x4c/0xe0
[Mon Mar 11 14:29:12 2024] ? __pfx_pollwake+0x10/0x10
[Mon Mar 11 14:29:12 2024] ? __pfx_pollwake+0x10/0x10
[Mon Mar 11 14:29:12 2024] ? __pfx_pollwake+0x10/0x10
[Mon Mar 11 14:29:12 2024] ? __pfx_pollwake+0x10/0x10
[Mon Mar 11 14:29:12 2024] ? __pfx_pollwake+0x10/0x10
[Mon Mar 11 14:29:12 2024] ? __pfx_pollwake+0x10/0x10
[Mon Mar 11 14:29:12 2024] ? __pfx_pollwake+0x10/0x10
[Mon Mar 11 14:29:12 2024] ? __pfx_pollwake+0x10/0x10
[Mon Mar 11 14:29:12 2024] ? add_ptr_to_bulk_krc_lock+0x5b/0x200
[Mon Mar 11 14:29:12 2024] core_sys_select+0x1a0/0x3b0
[Mon Mar 11 14:29:12 2024] ? cache_flush+0x2a/0x40 [sunrpc]
[Mon Mar 11 14:29:12 2024] ? ip_map_parse+0x1e2/0x200 [sunrpc]
[Mon Mar 11 14:29:12 2024] ? cache_write_procfs+0x52/0xb0 [sunrpc]
[Mon Mar 11 14:29:12 2024] do_pselect.constprop.0+0xca/0x170
[Mon Mar 11 14:29:12 2024] __x64_sys_pselect6+0x48/0x70
[Mon Mar 11 14:29:12 2024] do_syscall_64+0x59/0x90
[Mon Mar 11 14:29:12 2024] ? syscall_exit_work+0x103/0x130
[Mon Mar 11 14:29:12 2024] ? syscall_exit_to_user_mode+0x22/0x40
[Mon Mar 11 14:29:12 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:12 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:12 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:12 2024] ? sysvec_apic_timer_interrupt+0x3c/0x90
[Mon Mar 11 14:29:12 2024] entry_SYSCALL_64_after_hwframe+0x72/0xdc
[Mon Mar 11 14:29:12 2024] RIP: 0033:0x7f619d344dc9
[Mon Mar 11 14:29:12 2024] RSP: 002b:00007ffdb13093d0 EFLAGS: 00000246 ORIG_RAX: 000000000000010e
[Mon Mar 11 14:29:12 2024] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f619d344dc9
[Mon Mar 11 14:29:12 2024] RDX: 0000000000000000 RSI: 00007ffdb1309520 RDI: 0000000000000400
[Mon Mar 11 14:29:12 2024] RBP: 00007ffdb1309520 R08: 0000000000000000 R09: 0000000000000000
[Mon Mar 11 14:29:12 2024] R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000000400
[Mon Mar 11 14:29:12 2024] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000000
[Mon Mar 11 14:29:12 2024] </TASK>
[Mon Mar 11 14:29:12 2024] task:uwsgi state:S stack:0 pid:5236 ppid:4405 flags:0x00000002
[Mon Mar 11 14:29:12 2024] Call Trace:
[Mon Mar 11 14:29:12 2024] <TASK>
[Mon Mar 11 14:29:12 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:12 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:12 2024] schedule_hrtimeout_range_clock+0x10f/0x120
[Mon Mar 11 14:29:12 2024] ? ep_send_events+0x272/0x2c0
[Mon Mar 11 14:29:12 2024] ep_poll+0x348/0x3b0
[Mon Mar 11 14:29:12 2024] ? __pfx_autoremove_wake_function+0x10/0x10
[Mon Mar 11 14:29:12 2024] do_epoll_wait+0xb1/0xd0
[Mon Mar 11 14:29:12 2024] __x64_sys_epoll_wait+0x60/0x100
[Mon Mar 11 14:29:12 2024] ? syscall_trace_enter.constprop.0+0x126/0x1a0
[Mon Mar 11 14:29:12 2024] do_syscall_64+0x59/0x90
[Mon Mar 11 14:29:12 2024] ? exc_page_fault+0x62/0x150
[Mon Mar 11 14:29:12 2024] entry_SYSCALL_64_after_hwframe+0x72/0xdc
[Mon Mar 11 14:29:12 2024] RIP: 0033:0x7fb64274e80a
[Mon Mar 11 14:29:12 2024] RSP: 002b:00007fff2b12bb08 EFLAGS: 00000246 ORIG_RAX: 00000000000000e8
[Mon Mar 11 14:29:12 2024] RAX: ffffffffffffffda RBX: 00007fff2b12bb58 RCX: 00007fb64274e80a
[Mon Mar 11 14:29:12 2024] RDX: 0000000000000001 RSI: 00007fff2b12bb14 RDI: 0000000000000007
[Mon Mar 11 14:29:12 2024] RBP: 00007fb642563078 R08: 0000000000000001 R09: 0000000000000000
[Mon Mar 11 14:29:12 2024] R10: 00000000ffffffff R11: 0000000000000246 R12: 00000000004e0ee0
[Mon Mar 11 14:29:12 2024] R13: 0000000000000007 R14: 00007fb642563000 R15: 00007fff2b12bb9c
[Mon Mar 11 14:29:12 2024] </TASK>
[Mon Mar 11 14:29:12 2024] task:uwsgi state:S stack:0 pid:5237 ppid:4405 flags:0x00000002
[Mon Mar 11 14:29:12 2024] Call Trace:
[Mon Mar 11 14:29:12 2024] <TASK>
[Mon Mar 11 14:29:12 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:12 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:12 2024] schedule_hrtimeout_range_clock+0x10f/0x120
[Mon Mar 11 14:29:12 2024] ? ep_send_events+0x272/0x2c0
[Mon Mar 11 14:29:12 2024] ep_poll+0x348/0x3b0
[Mon Mar 11 14:29:12 2024] ? __pfx_autoremove_wake_function+0x10/0x10
[Mon Mar 11 14:29:12 2024] do_epoll_wait+0xb1/0xd0
[Mon Mar 11 14:29:12 2024] __x64_sys_epoll_wait+0x60/0x100
[Mon Mar 11 14:29:12 2024] ? syscall_trace_enter.constprop.0+0x126/0x1a0
[Mon Mar 11 14:29:12 2024] do_syscall_64+0x59/0x90
[Mon Mar 11 14:29:12 2024] ? syscall_exit_to_user_mode+0x22/0x40
[Mon Mar 11 14:29:12 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:12 2024] ? syscall_exit_to_user_mode+0x22/0x40
[Mon Mar 11 14:29:12 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:12 2024] ? switch_fpu_return+0x4c/0xd0
[Mon Mar 11 14:29:12 2024] ? exit_to_user_mode_prepare+0xec/0x100
[Mon Mar 11 14:29:12 2024] ? syscall_exit_to_user_mode+0x22/0x40
[Mon Mar 11 14:29:12 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:12 2024] ? __irq_exit_rcu+0x46/0xc0
[Mon Mar 11 14:29:12 2024] ? sysvec_apic_timer_interrupt+0x3c/0x90
[Mon Mar 11 14:29:12 2024] entry_SYSCALL_64_after_hwframe+0x72/0xdc
[Mon Mar 11 14:29:12 2024] RIP: 0033:0x7fb64274e80a
[Mon Mar 11 14:29:12 2024] RSP: 002b:00007fff2b12bb08 EFLAGS: 00000246 ORIG_RAX: 00000000000000e8
[Mon Mar 11 14:29:12 2024] RAX: ffffffffffffffda RBX: 00007fff2b12bb58 RCX: 00007fb64274e80a
[Mon Mar 11 14:29:12 2024] RDX: 0000000000000001 RSI: 00007fff2b12bb14 RDI: 0000000000000007
[Mon Mar 11 14:29:12 2024] RBP: 00007fb64210e078 R08: 0000000000000002 R09: 0000000000000000
[Mon Mar 11 14:29:12 2024] R10: 00000000ffffffff R11: 0000000000000246 R12: 00000000004e0ee0
[Mon Mar 11 14:29:12 2024] R13: 0000000000000007 R14: 00007fb64210e000 R15: 00007fff2b12bb9c
[Mon Mar 11 14:29:12 2024] </TASK>
[Mon Mar 11 14:29:12 2024] task:uwsgi state:S stack:0 pid:5238 ppid:4405 flags:0x00000002
[Mon Mar 11 14:29:12 2024] Call Trace:
[Mon Mar 11 14:29:12 2024] <TASK>
[Mon Mar 11 14:29:12 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:12 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:12 2024] schedule_hrtimeout_range_clock+0x10f/0x120
[Mon Mar 11 14:29:12 2024] ? ep_send_events+0x272/0x2c0
[Mon Mar 11 14:29:12 2024] ep_poll+0x348/0x3b0
[Mon Mar 11 14:29:12 2024] ? __pfx_autoremove_wake_function+0x10/0x10
[Mon Mar 11 14:29:12 2024] do_epoll_wait+0xb1/0xd0
[Mon Mar 11 14:29:12 2024] __x64_sys_epoll_wait+0x60/0x100
[Mon Mar 11 14:29:12 2024] do_syscall_64+0x59/0x90
[Mon Mar 11 14:29:12 2024] ? __audit_filter_op+0xa5/0xf0
[Mon Mar 11 14:29:12 2024] ? audit_reset_context.part.0.constprop.0+0x273/0x2e0
[Mon Mar 11 14:29:12 2024] ? syscall_exit_work+0x103/0x130
[Mon Mar 11 14:29:12 2024] ? syscall_exit_to_user_mode+0x22/0x40
[Mon Mar 11 14:29:12 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:12 2024] ? exc_page_fault+0x62/0x150
[Mon Mar 11 14:29:12 2024] entry_SYSCALL_64_after_hwframe+0x72/0xdc
[Mon Mar 11 14:29:12 2024] RIP: 0033:0x7fb64274e80a
[Mon Mar 11 14:29:12 2024] RSP: 002b:00007fff2b12bb08 EFLAGS: 00000246 ORIG_RAX: 00000000000000e8
[Mon Mar 11 14:29:12 2024] RAX: ffffffffffffffda RBX: 00007fff2b12bb58 RCX: 00007fb64274e80a
[Mon Mar 11 14:29:12 2024] RDX: 0000000000000001 RSI: 00007fff2b12bb14 RDI: 0000000000000007
[Mon Mar 11 14:29:12 2024] RBP: 00007fb64135c078 R08: 0000000000000003 R09: 0000000000000000
[Mon Mar 11 14:29:12 2024] R10: 00000000ffffffff R11: 0000000000000246 R12: 00000000004e0ee0
[Mon Mar 11 14:29:12 2024] R13: 0000000000000007 R14: 00007fb64135c000 R15: 00007fff2b12bb9c
[Mon Mar 11 14:29:12 2024] </TASK>
[Mon Mar 11 14:29:12 2024] task:uwsgi state:S stack:0 pid:5239 ppid:4405 flags:0x00000002
[Mon Mar 11 14:29:12 2024] Call Trace:
[Mon Mar 11 14:29:12 2024] <TASK>
[Mon Mar 11 14:29:12 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:12 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:12 2024] schedule_hrtimeout_range_clock+0x10f/0x120
[Mon Mar 11 14:29:12 2024] ? rcu_nocb_try_bypass+0x5e/0x460
[Mon Mar 11 14:29:12 2024] ? add_ptr_to_bulk_krc_lock+0x164/0x200
[Mon Mar 11 14:29:12 2024] ? __update_load_avg_cfs_rq+0x289/0x2f0
[Mon Mar 11 14:29:12 2024] ? __pfx_inode_free_by_rcu+0x10/0x10
[Mon Mar 11 14:29:12 2024] ep_poll+0x348/0x3b0
[Mon Mar 11 14:29:12 2024] ? __pfx_autoremove_wake_function+0x10/0x10
[Mon Mar 11 14:29:12 2024] do_epoll_wait+0xb1/0xd0
[Mon Mar 11 14:29:12 2024] __x64_sys_epoll_wait+0x60/0x100
[Mon Mar 11 14:29:12 2024] ? syscall_trace_enter.constprop.0+0x126/0x1a0
[Mon Mar 11 14:29:12 2024] do_syscall_64+0x59/0x90
[Mon Mar 11 14:29:12 2024] ? sched_clock+0xc/0x30
[Mon Mar 11 14:29:12 2024] ? sched_clock_cpu+0x9/0xc0
[Mon Mar 11 14:29:12 2024] ? irqtime_account_irq+0x3c/0xb0
[Mon Mar 11 14:29:12 2024] ? __do_softirq+0x16a/0x2ac
[Mon Mar 11 14:29:12 2024] ? __irq_exit_rcu+0x46/0xc0
[Mon Mar 11 14:29:12 2024] ? sysvec_apic_timer_interrupt+0x3c/0x90
[Mon Mar 11 14:29:12 2024] entry_SYSCALL_64_after_hwframe+0x72/0xdc
[Mon Mar 11 14:29:12 2024] RIP: 0033:0x7fb64274e80a
[Mon Mar 11 14:29:12 2024] RSP: 002b:00007fff2b12ba88 EFLAGS: 00000246 ORIG_RAX: 00000000000000e8
[Mon Mar 11 14:29:12 2024] RAX: ffffffffffffffda RBX: 00007fb642f08f20 RCX: 00007fb64274e80a
[Mon Mar 11 14:29:12 2024] RDX: 0000000000000040 RSI: 0000000000a44080 RDI: 0000000000000006
[Mon Mar 11 14:29:12 2024] RBP: 00000000004e0ee0 R08: 00000000ffffffff R09: 0000000000000000
[Mon Mar 11 14:29:12 2024] R10: 00000000ffffffff R11: 0000000000000246 R12: 0000000000adcc60
[Mon Mar 11 14:29:12 2024] R13: 0000000000a44080 R14: 0000000000a44080 R15: 0000000000a4408c
[Mon Mar 11 14:29:12 2024] </TASK>
[Mon Mar 11 14:29:12 2024] task:nsrexecd state:S stack:0 pid:5339 ppid:1 flags:0x00000002
[Mon Mar 11 14:29:12 2024] Call Trace:
[Mon Mar 11 14:29:12 2024] <TASK>
[Mon Mar 11 14:29:12 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:12 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:12 2024] schedule_hrtimeout_range_clock+0x95/0x120
[Mon Mar 11 14:29:12 2024] ? __pfx_hrtimer_wakeup+0x10/0x10
[Mon Mar 11 14:29:12 2024] do_poll.constprop.0+0x248/0x390
[Mon Mar 11 14:29:12 2024] do_sys_poll+0x1c8/0x250
[Mon Mar 11 14:29:12 2024] ? ext4_xattr_ibody_get+0x16e/0x1b0 [ext4]
[Mon Mar 11 14:29:12 2024] ? xas_load+0x9/0xa0
[Mon Mar 11 14:29:12 2024] ? __filemap_get_folio+0x26e/0x330
[Mon Mar 11 14:29:12 2024] ? pagecache_get_page+0x3a/0x70
[Mon Mar 11 14:29:12 2024] ? __pfx_pollwake+0x10/0x10
[Mon Mar 11 14:29:12 2024] ? ext4_htree_store_dirent+0x36/0x100 [ext4]
[Mon Mar 11 14:29:12 2024] ? htree_dirblock_to_tree+0x27e/0x330 [ext4]
[Mon Mar 11 14:29:12 2024] ? ext4_htree_fill_tree+0xca/0x3b0 [ext4]
[Mon Mar 11 14:29:12 2024] ? rseq_get_rseq_cs+0x1d/0x240
[Mon Mar 11 14:29:12 2024] ? rcu_nocb_try_bypass+0x5e/0x460
[Mon Mar 11 14:29:12 2024] ? rseq_ip_fixup+0x6e/0x1a0
[Mon Mar 11 14:29:12 2024] ? ktime_get_ts64+0x49/0xf0
[Mon Mar 11 14:29:12 2024] __x64_sys_poll+0xa6/0x140
[Mon Mar 11 14:29:12 2024] do_syscall_64+0x59/0x90
[Mon Mar 11 14:29:12 2024] ? audit_reset_context.part.0.constprop.0+0x273/0x2e0
[Mon Mar 11 14:29:12 2024] ? syscall_exit_work+0x103/0x130
[Mon Mar 11 14:29:12 2024] ? syscall_exit_to_user_mode+0x22/0x40
[Mon Mar 11 14:29:12 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:12 2024] ? syscall_exit_work+0x103/0x130
[Mon Mar 11 14:29:12 2024] ? syscall_exit_to_user_mode+0x22/0x40
[Mon Mar 11 14:29:12 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:12 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:12 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:12 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:12 2024] ? sysvec_apic_timer_interrupt+0x3c/0x90
[Mon Mar 11 14:29:12 2024] entry_SYSCALL_64_after_hwframe+0x72/0xdc
[Mon Mar 11 14:29:12 2024] RIP: 0033:0x7fef6b3426ff
[Mon Mar 11 14:29:12 2024] RSP: 002b:00007ffcfb738320 EFLAGS: 00000293 ORIG_RAX: 0000000000000007
[Mon Mar 11 14:29:12 2024] RAX: ffffffffffffffda RBX: 0000000000001b58 RCX: 00007fef6b3426ff
[Mon Mar 11 14:29:12 2024] RDX: 0000000000001b58 RSI: 0000000000000001 RDI: 0000000001f08210
[Mon Mar 11 14:29:12 2024] RBP: 0000000000000000 R08: 0000000000000000 R09: 00007ffcfb729df0
[Mon Mar 11 14:29:12 2024] R10: 0000000000000000 R11: 0000000000000293 R12: 0000000001ef3cd0
[Mon Mar 11 14:29:12 2024] R13: 0000000065ef0748 R14: 0000000065ef0748 R15: 00007ffcfb73d4e8
[Mon Mar 11 14:29:12 2024] </TASK>
[Mon Mar 11 14:29:12 2024] task:nsrexecd state:S stack:0 pid:5342 ppid:1 flags:0x00000002
[Mon Mar 11 14:29:12 2024] Call Trace:
[Mon Mar 11 14:29:12 2024] <TASK>
[Mon Mar 11 14:29:12 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:12 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:12 2024] schedule_hrtimeout_range_clock+0x95/0x120
[Mon Mar 11 14:29:12 2024] ? __pfx_hrtimer_wakeup+0x10/0x10
[Mon Mar 11 14:29:12 2024] do_poll.constprop.0+0x248/0x390
[Mon Mar 11 14:29:12 2024] do_sys_poll+0x1c8/0x250
[Mon Mar 11 14:29:12 2024] ? do_sys_poll+0x1d8/0x250
[Mon Mar 11 14:29:12 2024] ? __pfx_pollwake+0x10/0x10
[Mon Mar 11 14:29:12 2024] ? __pfx_pollwake+0x10/0x10
[Mon Mar 11 14:29:12 2024] ? __pfx_pollwake+0x10/0x10
[Mon Mar 11 14:29:12 2024] ? __pfx_pollwake+0x10/0x10
[Mon Mar 11 14:29:12 2024] ? __pfx_pollwake+0x10/0x10
[Mon Mar 11 14:29:12 2024] ? rseq_get_rseq_cs+0x1d/0x240
[Mon Mar 11 14:29:12 2024] ? rseq_ip_fixup+0x6e/0x1a0
[Mon Mar 11 14:29:12 2024] ? __audit_filter_op+0xa5/0xf0
[Mon Mar 11 14:29:12 2024] ? ktime_get_ts64+0x49/0xf0
[Mon Mar 11 14:29:12 2024] __x64_sys_poll+0xa6/0x140
[Mon Mar 11 14:29:12 2024] do_syscall_64+0x59/0x90
[Mon Mar 11 14:29:12 2024] ? audit_reset_context.part.0.constprop.0+0x273/0x2e0
[Mon Mar 11 14:29:12 2024] ? syscall_exit_work+0x103/0x130
[Mon Mar 11 14:29:12 2024] ? syscall_exit_to_user_mode+0x22/0x40
[Mon Mar 11 14:29:12 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:12 2024] ? syscall_exit_to_user_mode+0x22/0x40
[Mon Mar 11 14:29:12 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:12 2024] entry_SYSCALL_64_after_hwframe+0x72/0xdc
[Mon Mar 11 14:29:12 2024] RIP: 0033:0x7fef6b3426ff
[Mon Mar 11 14:29:12 2024] RSP: 002b:00007fef691fec10 EFLAGS: 00000293 ORIG_RAX: 0000000000000007
[Mon Mar 11 14:29:12 2024] RAX: ffffffffffffffda RBX: 00007fef6401d940 RCX: 00007fef6b3426ff
[Mon Mar 11 14:29:12 2024] RDX: 000000000000ea60 RSI: 0000000000000003 RDI: 00007fef64013720
[Mon Mar 11 14:29:12 2024] RBP: 0000000000000003 R08: 0000000000000000 R09: 0000000000000000
[Mon Mar 11 14:29:12 2024] R10: 0000000000000080 R11: 0000000000000293 R12: 00007fef6401d960
[Mon Mar 11 14:29:12 2024] R13: 0000000000000000 R14: 00007fef64000b80 R15: 00007fef691fede0
[Mon Mar 11 14:29:12 2024] </TASK>
[Mon Mar 11 14:29:12 2024] task:nsrexecd state:S stack:0 pid:5343 ppid:1 flags:0x00000002
[Mon Mar 11 14:29:12 2024] Call Trace:
[Mon Mar 11 14:29:12 2024] <TASK>
[Mon Mar 11 14:29:12 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:12 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:12 2024] schedule_hrtimeout_range_clock+0x95/0x120
[Mon Mar 11 14:29:12 2024] ? __pfx_hrtimer_wakeup+0x10/0x10
[Mon Mar 11 14:29:12 2024] do_poll.constprop.0+0x248/0x390
[Mon Mar 11 14:29:12 2024] do_sys_poll+0x1c8/0x250
[Mon Mar 11 14:29:12 2024] ? __kmem_cache_alloc_node+0x1c7/0x2d0
[Mon Mar 11 14:29:12 2024] ? ext4_htree_store_dirent+0x36/0x100 [ext4]
[Mon Mar 11 14:29:12 2024] ? ext4_htree_store_dirent+0x36/0x100 [ext4]
[Mon Mar 11 14:29:12 2024] ? __pfx_pollwake+0x10/0x10
[Mon Mar 11 14:29:12 2024] ? ext4_htree_fill_tree+0xca/0x3b0 [ext4]
[Mon Mar 11 14:29:12 2024] ? __kmem_cache_alloc_node+0x1c7/0x2d0
[Mon Mar 11 14:29:12 2024] ? verify_dirent_name+0x1c/0x40
[Mon Mar 11 14:29:12 2024] ? filldir64+0x3b/0x190
[Mon Mar 11 14:29:12 2024] ? current_time+0x2b/0xf0
[Mon Mar 11 14:29:12 2024] ? rseq_get_rseq_cs+0x1d/0x240
[Mon Mar 11 14:29:12 2024] ? rcu_nocb_try_bypass+0x5e/0x460
[Mon Mar 11 14:29:12 2024] ? rseq_ip_fixup+0x6e/0x1a0
[Mon Mar 11 14:29:12 2024] ? __rseq_handle_notify_resume+0x26/0xb0
[Mon Mar 11 14:29:12 2024] ? ktime_get_ts64+0x49/0xf0
[Mon Mar 11 14:29:12 2024] __x64_sys_poll+0xa6/0x140
[Mon Mar 11 14:29:12 2024] do_syscall_64+0x59/0x90
[Mon Mar 11 14:29:12 2024] ? syscall_exit_work+0x103/0x130
[Mon Mar 11 14:29:12 2024] ? syscall_exit_to_user_mode+0x22/0x40
[Mon Mar 11 14:29:12 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:12 2024] ? sysvec_call_function+0x3c/0x90
[Mon Mar 11 14:29:12 2024] entry_SYSCALL_64_after_hwframe+0x72/0xdc
[Mon Mar 11 14:29:12 2024] RIP: 0033:0x7fef6b3426ff
[Mon Mar 11 14:29:12 2024] RSP: 002b:00007fef63ffec90 EFLAGS: 00000293 ORIG_RAX: 0000000000000007
[Mon Mar 11 14:29:12 2024] RAX: ffffffffffffffda RBX: 00007fef63ffed80 RCX: 00007fef6b3426ff
[Mon Mar 11 14:29:12 2024] RDX: 0000000000007530 RSI: 0000000000000001 RDI: 00007fef5c014860
[Mon Mar 11 14:29:12 2024] RBP: 0000000000ba0cd8 R08: 0000000000000000 R09: 0000000000000000
[Mon Mar 11 14:29:12 2024] R10: 0000000001000000 R11: 0000000000000293 R12: 00007fef5c000b80
[Mon Mar 11 14:29:12 2024] R13: 00007fef5c00c710 R14: 00007fef6b29f530 R15: 0000000000000000
[Mon Mar 11 14:29:12 2024] </TASK>
[Mon Mar 11 14:29:12 2024] task:nsrexecd state:S stack:0 pid:5345 ppid:1 flags:0x00000002
[Mon Mar 11 14:29:12 2024] Call Trace:
[Mon Mar 11 14:29:12 2024] <TASK>
[Mon Mar 11 14:29:12 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:12 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:12 2024] schedule_hrtimeout_range_clock+0x95/0x120
[Mon Mar 11 14:29:12 2024] ? __pfx_hrtimer_wakeup+0x10/0x10
[Mon Mar 11 14:29:12 2024] do_poll.constprop.0+0x248/0x390
[Mon Mar 11 14:29:12 2024] do_sys_poll+0x1c8/0x250
[Mon Mar 11 14:29:12 2024] ? __pfx_pollwake+0x10/0x10
[Mon Mar 11 14:29:12 2024] ? __pfx_pollwake+0x10/0x10
[Mon Mar 11 14:29:12 2024] ? __pfx_pollwake+0x10/0x10
[Mon Mar 11 14:29:12 2024] ? exc_page_fault+0x62/0x150
[Mon Mar 11 14:29:12 2024] ? node_is_toptier+0x3e/0x60
[Mon Mar 11 14:29:12 2024] ? __audit_filter_op+0xa5/0xf0
[Mon Mar 11 14:29:12 2024] ? audit_reset_context.part.0.constprop.0+0x273/0x2e0
[Mon Mar 11 14:29:12 2024] ? syscall_exit_work+0x103/0x130
[Mon Mar 11 14:29:12 2024] ? syscall_exit_to_user_mode+0x22/0x40
[Mon Mar 11 14:29:12 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:12 2024] ? ktime_get_ts64+0x49/0xf0
[Mon Mar 11 14:29:12 2024] __x64_sys_poll+0xa6/0x140
[Mon Mar 11 14:29:12 2024] do_syscall_64+0x59/0x90
[Mon Mar 11 14:29:12 2024] ? exit_to_user_mode_prepare+0xec/0x100
[Mon Mar 11 14:29:12 2024] ? syscall_exit_to_user_mode+0x22/0x40
[Mon Mar 11 14:29:12 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:12 2024] ? syscall_exit_to_user_mode+0x22/0x40
[Mon Mar 11 14:29:12 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:12 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:12 2024] entry_SYSCALL_64_after_hwframe+0x72/0xdc
[Mon Mar 11 14:29:12 2024] RIP: 0033:0x7fef6b3426ff
[Mon Mar 11 14:29:12 2024] RSP: 002b:00007fef61ffcca0 EFLAGS: 00000293 ORIG_RAX: 0000000000000007
[Mon Mar 11 14:29:12 2024] RAX: ffffffffffffffda RBX: 00007fef58003010 RCX: 00007fef6b3426ff
[Mon Mar 11 14:29:12 2024] RDX: 00000000000105b8 RSI: 0000000000000001 RDI: 00007fef58016cf0
[Mon Mar 11 14:29:12 2024] RBP: 0000000000000000 R08: 0000000000000000 R09: 0000000001ebc708
[Mon Mar 11 14:29:12 2024] R10: 0000000000000000 R11: 0000000000000293 R12: 000000000000000d
[Mon Mar 11 14:29:12 2024] R13: 0000000000ba0ce8 R14: 00007fef5800eba0 R15: 0000000000000000
[Mon Mar 11 14:29:12 2024] </TASK>
[Mon Mar 11 14:29:12 2024] task:nsrexecd state:S stack:0 pid:5346 ppid:1 flags:0x00000002
[Mon Mar 11 14:29:12 2024] Call Trace:
[Mon Mar 11 14:29:12 2024] <TASK>
[Mon Mar 11 14:29:12 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:12 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:12 2024] futex_wait_queue+0x70/0xd0
[Mon Mar 11 14:29:12 2024] futex_wait+0x175/0x260
[Mon Mar 11 14:29:12 2024] ? __pfx_hrtimer_wakeup+0x10/0x10
[Mon Mar 11 14:29:12 2024] do_futex+0x12d/0x1d0
[Mon Mar 11 14:29:12 2024] __x64_sys_futex+0x73/0x1d0
[Mon Mar 11 14:29:12 2024] do_syscall_64+0x59/0x90
[Mon Mar 11 14:29:12 2024] ? syscall_exit_work+0x103/0x130
[Mon Mar 11 14:29:12 2024] ? syscall_exit_to_user_mode+0x22/0x40
[Mon Mar 11 14:29:12 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:12 2024] ? audit_reset_context.part.0.constprop.0+0x273/0x2e0
[Mon Mar 11 14:29:12 2024] ? syscall_exit_work+0x103/0x130
[Mon Mar 11 14:29:12 2024] ? syscall_exit_to_user_mode+0x22/0x40
[Mon Mar 11 14:29:12 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:12 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:12 2024] entry_SYSCALL_64_after_hwframe+0x72/0xdc
[Mon Mar 11 14:29:12 2024] RIP: 0033:0x7fef6b29c39a
[Mon Mar 11 14:29:12 2024] RSP: 002b:00007fef53ffeca0 EFLAGS: 00000246 ORIG_RAX: 00000000000000ca
[Mon Mar 11 14:29:12 2024] RAX: ffffffffffffffda RBX: 00007fef4c009180 RCX: 00007fef6b29c39a
[Mon Mar 11 14:29:12 2024] RDX: 0000000000000000 RSI: 0000000000000189 RDI: 00007fef4c009180
[Mon Mar 11 14:29:12 2024] RBP: 0000000000000000 R08: 0000000000000000 R09: 00000000ffffffff
[Mon Mar 11 14:29:12 2024] R10: 00007fef53ffed40 R11: 0000000000000246 R12: 0000000000000000
[Mon Mar 11 14:29:12 2024] R13: 00007fef53ffece0 R14: fffffffeffffffff R15: 0000000000000000
[Mon Mar 11 14:29:12 2024] </TASK>
[Mon Mar 11 14:29:12 2024] task:nsrexecd state:S stack:0 pid:5347 ppid:1 flags:0x00000002
[Mon Mar 11 14:29:12 2024] Call Trace:
[Mon Mar 11 14:29:12 2024] <TASK>
[Mon Mar 11 14:29:12 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:12 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:12 2024] schedule_hrtimeout_range_clock+0x95/0x120
[Mon Mar 11 14:29:12 2024] ? __pfx_hrtimer_wakeup+0x10/0x10
[Mon Mar 11 14:29:12 2024] do_poll.constprop.0+0x248/0x390
[Mon Mar 11 14:29:12 2024] do_sys_poll+0x1c8/0x250
[Mon Mar 11 14:29:12 2024] ? do_sys_poll+0x1d8/0x250
[Mon Mar 11 14:29:12 2024] ? __pfx_pollwake+0x10/0x10
[Mon Mar 11 14:29:12 2024] ? __pfx_pollwake+0x10/0x10
[Mon Mar 11 14:29:12 2024] ? __pfx_pollwake+0x10/0x10
[Mon Mar 11 14:29:12 2024] ? __pfx_pollwake+0x10/0x10
[Mon Mar 11 14:29:12 2024] ? __pfx_pollwake+0x10/0x10
[Mon Mar 11 14:29:12 2024] ? common_interrupt+0xf/0xa0
[Mon Mar 11 14:29:12 2024] ? asm_common_interrupt+0x22/0x40
[Mon Mar 11 14:29:12 2024] ? common_interrupt+0xf/0xa0
[Mon Mar 11 14:29:12 2024] ? rseq_get_rseq_cs+0x1d/0x240
[Mon Mar 11 14:29:12 2024] ? rseq_get_rseq_cs+0x1d/0x240
[Mon Mar 11 14:29:12 2024] ? rseq_ip_fixup+0x6e/0x1a0
[Mon Mar 11 14:29:12 2024] ? ktime_get_ts64+0x49/0xf0
[Mon Mar 11 14:29:12 2024] __x64_sys_poll+0xa6/0x140
[Mon Mar 11 14:29:12 2024] do_syscall_64+0x59/0x90
[Mon Mar 11 14:29:12 2024] ? syscall_exit_to_user_mode+0x22/0x40
[Mon Mar 11 14:29:12 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:12 2024] ? exit_to_user_mode_prepare+0xec/0x100
[Mon Mar 11 14:29:12 2024] ? syscall_exit_to_user_mode+0x22/0x40
[Mon Mar 11 14:29:12 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:12 2024] ? syscall_exit_to_user_mode+0x22/0x40
[Mon Mar 11 14:29:12 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:12 2024] ? syscall_exit_to_user_mode+0x22/0x40
[Mon Mar 11 14:29:12 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:12 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:12 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:12 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:12 2024] entry_SYSCALL_64_after_hwframe+0x72/0xdc
[Mon Mar 11 14:29:12 2024] RIP: 0033:0x7fef6b3426ff
[Mon Mar 11 14:29:12 2024] RSP: 002b:00007fef52ffdcc0 EFLAGS: 00000293 ORIG_RAX: 0000000000000007
[Mon Mar 11 14:29:12 2024] RAX: ffffffffffffffda RBX: 00007fef44014d60 RCX: 00007fef6b3426ff
[Mon Mar 11 14:29:12 2024] RDX: 0000000000001388 RSI: 0000000000000001 RDI: 00007fef4401cf30
[Mon Mar 11 14:29:12 2024] RBP: 0000000001ef85b0 R08: 0000000000000000 R09: 00007fef52ffd750
[Mon Mar 11 14:29:12 2024] R10: 0000000000000000 R11: 0000000000000293 R12: 00007ffcfb743d10
[Mon Mar 11 14:29:12 2024] R13: 0000000000000002 R14: 00007fef6b29f530 R15: 0000000000000000
[Mon Mar 11 14:29:12 2024] </TASK>
[Mon Mar 11 14:29:12 2024] task:nsrexecd state:S stack:0 pid:5349 ppid:1 flags:0x00000002
[Mon Mar 11 14:29:12 2024] Call Trace:
[Mon Mar 11 14:29:12 2024] <TASK>
[Mon Mar 11 14:29:12 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:12 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:12 2024] do_nanosleep+0x67/0x190
[Mon Mar 11 14:29:12 2024] hrtimer_nanosleep+0xbe/0x1a0
[Mon Mar 11 14:29:12 2024] ? __pfx_hrtimer_wakeup+0x10/0x10
[Mon Mar 11 14:29:12 2024] common_nsleep+0x40/0x50
[Mon Mar 11 14:29:12 2024] __x64_sys_clock_nanosleep+0xbc/0x130
[Mon Mar 11 14:29:12 2024] do_syscall_64+0x59/0x90
[Mon Mar 11 14:29:12 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:12 2024] ? audit_reset_context.part.0.constprop.0+0x273/0x2e0
[Mon Mar 11 14:29:12 2024] ? syscall_exit_work+0x103/0x130
[Mon Mar 11 14:29:12 2024] ? syscall_exit_to_user_mode+0x22/0x40
[Mon Mar 11 14:29:12 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:12 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:12 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:12 2024] entry_SYSCALL_64_after_hwframe+0x72/0xdc
[Mon Mar 11 14:29:12 2024] RIP: 0033:0x7fef6b313975
[Mon Mar 11 14:29:12 2024] RSP: 002b:00007fef4bff8bc0 EFLAGS: 00000293 ORIG_RAX: 00000000000000e6
[Mon Mar 11 14:29:12 2024] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007fef6b313975
[Mon Mar 11 14:29:12 2024] RDX: 00007fef4bff8c10 RSI: 0000000000000000 RDI: 0000000000000000
[Mon Mar 11 14:29:12 2024] RBP: 00007fef4bff8c50 R08: 0000000000000000 R09: 0000000000000000
[Mon Mar 11 14:29:12 2024] R10: 00007fef4bff8c00 R11: 0000000000000293 R12: 0000000001e7b7c0
[Mon Mar 11 14:29:12 2024] R13: 00007fef4bffec50 R14: 0000000000b8a714 R15: 0000000000b8a744
[Mon Mar 11 14:29:12 2024] </TASK>
[Mon Mar 11 14:29:12 2024] task:wb-idmap state:S stack:0 pid:5362 ppid:4415 flags:0x00000002
[Mon Mar 11 14:29:12 2024] Call Trace:
[Mon Mar 11 14:29:12 2024] <TASK>
[Mon Mar 11 14:29:12 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:12 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:12 2024] schedule_hrtimeout_range_clock+0x95/0x120
[Mon Mar 11 14:29:12 2024] ? __pfx_hrtimer_wakeup+0x10/0x10
[Mon Mar 11 14:29:12 2024] ep_poll+0x348/0x3b0
[Mon Mar 11 14:29:12 2024] ? __pfx_autoremove_wake_function+0x10/0x10
[Mon Mar 11 14:29:12 2024] do_epoll_wait+0xb1/0xd0
[Mon Mar 11 14:29:12 2024] __x64_sys_epoll_wait+0x60/0x100
[Mon Mar 11 14:29:12 2024] do_syscall_64+0x59/0x90
[Mon Mar 11 14:29:12 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:12 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:12 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:12 2024] ? sysvec_apic_timer_interrupt+0x3c/0x90
[Mon Mar 11 14:29:12 2024] entry_SYSCALL_64_after_hwframe+0x72/0xdc
[Mon Mar 11 14:29:12 2024] RIP: 0033:0x7f26a134e80a
[Mon Mar 11 14:29:12 2024] RSP: 002b:00007ffe9e332b58 EFLAGS: 00000246 ORIG_RAX: 00000000000000e8
[Mon Mar 11 14:29:12 2024] RAX: ffffffffffffffda RBX: 00005586337521b0 RCX: 00007f26a134e80a
[Mon Mar 11 14:29:12 2024] RDX: 0000000000000001 RSI: 00007ffe9e332b8c RDI: 0000000000000005
[Mon Mar 11 14:29:12 2024] RBP: 00005586337422e0 R08: 000055863376b800 R09: 00000000000069df
[Mon Mar 11 14:29:12 2024] R10: 0000000000007530 R11: 0000000000000246 R12: 000055863375b3a0
[Mon Mar 11 14:29:12 2024] R13: 0000000000007530 R14: 0000558632a302f8 R15: 0000558632a4c020
[Mon Mar 11 14:29:12 2024] </TASK>
[Mon Mar 11 14:29:12 2024] task:systemd state:S stack:0 pid:5558 ppid:1 flags:0x00000002
[Mon Mar 11 14:29:12 2024] Call Trace:
[Mon Mar 11 14:29:12 2024] <TASK>
[Mon Mar 11 14:29:12 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:12 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:12 2024] schedule_hrtimeout_range_clock+0x10f/0x120
[Mon Mar 11 14:29:12 2024] ? ep_send_events+0x272/0x2c0
[Mon Mar 11 14:29:12 2024] ep_poll+0x348/0x3b0
[Mon Mar 11 14:29:12 2024] ? __pfx_autoremove_wake_function+0x10/0x10
[Mon Mar 11 14:29:12 2024] do_epoll_wait+0xb1/0xd0
[Mon Mar 11 14:29:12 2024] __x64_sys_epoll_wait+0x60/0x100
[Mon Mar 11 14:29:12 2024] ? syscall_trace_enter.constprop.0+0x126/0x1a0
[Mon Mar 11 14:29:12 2024] do_syscall_64+0x59/0x90
[Mon Mar 11 14:29:12 2024] ? syscall_exit_work+0x103/0x130
[Mon Mar 11 14:29:12 2024] ? syscall_exit_to_user_mode+0x22/0x40
[Mon Mar 11 14:29:12 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:12 2024] ? syscall_exit_to_user_mode+0x22/0x40
[Mon Mar 11 14:29:12 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:12 2024] ? __do_softirq+0x16a/0x2ac
[Mon Mar 11 14:29:12 2024] ? __irq_exit_rcu+0x46/0xc0
[Mon Mar 11 14:29:12 2024] ? common_interrupt+0x43/0xa0
[Mon Mar 11 14:29:12 2024] entry_SYSCALL_64_after_hwframe+0x72/0xdc
[Mon Mar 11 14:29:12 2024] RIP: 0033:0x7fc22df4e80a
[Mon Mar 11 14:29:12 2024] RSP: 002b:00007ffe065f5858 EFLAGS: 00000246 ORIG_RAX: 00000000000000e8
[Mon Mar 11 14:29:12 2024] RAX: ffffffffffffffda RBX: 000056476cd91670 RCX: 00007fc22df4e80a
[Mon Mar 11 14:29:12 2024] RDX: 0000000000000020 RSI: 000056476cff0240 RDI: 0000000000000004
[Mon Mar 11 14:29:12 2024] RBP: 000056476cd91800 R08: 0000000000000020 R09: b36493637b16ad2e
[Mon Mar 11 14:29:12 2024] R10: 00000000ffffffff R11: 0000000000000246 R12: 00000000000000b4
[Mon Mar 11 14:29:12 2024] R13: 000056476cd91670 R14: 0000000000000020 R15: 0000000000000012
[Mon Mar 11 14:29:12 2024] </TASK>
[Mon Mar 11 14:29:12 2024] task:(sd-pam) state:S stack:0 pid:5572 ppid:5558 flags:0x00000002
[Mon Mar 11 14:29:12 2024] Call Trace:
[Mon Mar 11 14:29:12 2024] <TASK>
[Mon Mar 11 14:29:12 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:12 2024] ? __smp_call_single_queue+0x93/0x120
[Mon Mar 11 14:29:12 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:12 2024] schedule_hrtimeout_range_clock+0x10f/0x120
[Mon Mar 11 14:29:12 2024] ? pollwake+0x74/0xa0
[Mon Mar 11 14:29:12 2024] ? dequeue_signal+0x68/0x200
[Mon Mar 11 14:29:12 2024] do_sigtimedwait+0x16f/0x210
[Mon Mar 11 14:29:12 2024] __x64_sys_rt_sigtimedwait+0x6e/0xe0
[Mon Mar 11 14:29:12 2024] ? syscall_trace_enter.constprop.0+0x126/0x1a0
[Mon Mar 11 14:29:12 2024] do_syscall_64+0x59/0x90
[Mon Mar 11 14:29:12 2024] ? audit_reset_context.part.0.constprop.0+0x273/0x2e0
[Mon Mar 11 14:29:12 2024] ? syscall_exit_work+0x103/0x130
[Mon Mar 11 14:29:12 2024] ? syscall_exit_to_user_mode+0x22/0x40
[Mon Mar 11 14:29:12 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:12 2024] ? syscall_exit_to_user_mode+0x22/0x40
[Mon Mar 11 14:29:12 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:12 2024] ? exc_page_fault+0x62/0x150
[Mon Mar 11 14:29:12 2024] entry_SYSCALL_64_after_hwframe+0x72/0xdc
[Mon Mar 11 14:29:12 2024] RIP: 0033:0x7f97a2a55a68
[Mon Mar 11 14:29:12 2024] RSP: 002b:00007ffca488be10 EFLAGS: 00000246 ORIG_RAX: 0000000000000080
[Mon Mar 11 14:29:12 2024] RAX: ffffffffffffffda RBX: 00007ffca488be40 RCX: 00007f97a2a55a68
[Mon Mar 11 14:29:12 2024] RDX: 0000000000000000 RSI: 00007ffca488be40 RDI: 00007ffca488bfd0
[Mon Mar 11 14:29:12 2024] RBP: 00007ffca488be40 R08: 0000000000000001 R09: 0000000000000000
[Mon Mar 11 14:29:12 2024] R10: 0000000000000008 R11: 0000000000000246 R12: 00007ffca488bf28
[Mon Mar 11 14:29:12 2024] R13: 00007ffca488bfd0 R14: 0000000000000000 R15: 00007ffca488bf30
[Mon Mar 11 14:29:12 2024] </TASK>
[Mon Mar 11 14:29:12 2024] task:kdmflush/253:90 state:I stack:0 pid:6393 ppid:2 flags:0x00004000
[Mon Mar 11 14:29:12 2024] Call Trace:
[Mon Mar 11 14:29:12 2024] <TASK>
[Mon Mar 11 14:29:12 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:12 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:12 2024] rescuer_thread+0x2ca/0x390
[Mon Mar 11 14:29:12 2024] ? __pfx_rescuer_thread+0x10/0x10
[Mon Mar 11 14:29:12 2024] kthread+0xdd/0x100
[Mon Mar 11 14:29:12 2024] ? __pfx_kthread+0x10/0x10
[Mon Mar 11 14:29:12 2024] ret_from_fork+0x29/0x50
[Mon Mar 11 14:29:12 2024] </TASK>
[Mon Mar 11 14:29:12 2024] task:kcopyd state:I stack:0 pid:6394 ppid:2 flags:0x00004000
[Mon Mar 11 14:29:12 2024] Call Trace:
[Mon Mar 11 14:29:12 2024] <TASK>
[Mon Mar 11 14:29:12 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:12 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:12 2024] rescuer_thread+0x2ca/0x390
[Mon Mar 11 14:29:12 2024] ? __pfx_rescuer_thread+0x10/0x10
[Mon Mar 11 14:29:12 2024] kthread+0xdd/0x100
[Mon Mar 11 14:29:12 2024] ? __pfx_kthread+0x10/0x10
[Mon Mar 11 14:29:12 2024] ret_from_fork+0x29/0x50
[Mon Mar 11 14:29:12 2024] </TASK>
[Mon Mar 11 14:29:12 2024] task:dm-thin state:I stack:0 pid:6395 ppid:2 flags:0x00004000
[Mon Mar 11 14:29:12 2024] Call Trace:
[Mon Mar 11 14:29:12 2024] <TASK>
[Mon Mar 11 14:29:12 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:12 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:12 2024] rescuer_thread+0x2ca/0x390
[Mon Mar 11 14:29:12 2024] ? __pfx_rescuer_thread+0x10/0x10
[Mon Mar 11 14:29:12 2024] kthread+0xdd/0x100
[Mon Mar 11 14:29:12 2024] ? __pfx_kthread+0x10/0x10
[Mon Mar 11 14:29:12 2024] ret_from_fork+0x29/0x50
[Mon Mar 11 14:29:12 2024] </TASK>
[Mon Mar 11 14:29:12 2024] task:kdmflush/253:91 state:I stack:0 pid:6396 ppid:2 flags:0x00004000
[Mon Mar 11 14:29:12 2024] Call Trace:
[Mon Mar 11 14:29:12 2024] <TASK>
[Mon Mar 11 14:29:12 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:12 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:12 2024] rescuer_thread+0x2ca/0x390
[Mon Mar 11 14:29:12 2024] ? __pfx_rescuer_thread+0x10/0x10
[Mon Mar 11 14:29:12 2024] kthread+0xdd/0x100
[Mon Mar 11 14:29:12 2024] ? __pfx_kthread+0x10/0x10
[Mon Mar 11 14:29:12 2024] ret_from_fork+0x29/0x50
[Mon Mar 11 14:29:12 2024] </TASK>
[Mon Mar 11 14:29:12 2024] task:dmeventd state:S stack:0 pid:6398 ppid:1 flags:0x00000002
[Mon Mar 11 14:29:12 2024] Call Trace:
[Mon Mar 11 14:29:12 2024] <TASK>
[Mon Mar 11 14:29:12 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:12 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:12 2024] schedule_hrtimeout_range_clock+0x95/0x120
[Mon Mar 11 14:29:12 2024] ? __pfx_hrtimer_wakeup+0x10/0x10
[Mon Mar 11 14:29:12 2024] do_select+0x69e/0x7c0
[Mon Mar 11 14:29:12 2024] ? __pfx_pollwake+0x10/0x10
[Mon Mar 11 14:29:12 2024] ? __pfx_pollwake+0x10/0x10
[Mon Mar 11 14:29:12 2024] ? __pfx_pollwake+0x10/0x10
[Mon Mar 11 14:29:12 2024] ? __pfx_pollwake+0x10/0x10
[Mon Mar 11 14:29:12 2024] ? sysvec_apic_timer_interrupt+0xb/0x90
[Mon Mar 11 14:29:12 2024] ? asm_sysvec_apic_timer_interrupt+0x16/0x20
[Mon Mar 11 14:29:12 2024] ? sysvec_apic_timer_interrupt+0xb/0x90
[Mon Mar 11 14:29:12 2024] ? asm_sysvec_apic_timer_interrupt+0x16/0x20
[Mon Mar 11 14:29:12 2024] ? sysvec_apic_timer_interrupt+0xb/0x90
[Mon Mar 11 14:29:12 2024] ? asm_sysvec_apic_timer_interrupt+0x16/0x20
[Mon Mar 11 14:29:12 2024] ? update_sg_lb_stats+0x7e/0x450
[Mon Mar 11 14:29:12 2024] ? sysvec_apic_timer_interrupt+0xb/0x90
[Mon Mar 11 14:29:12 2024] ? sysvec_apic_timer_interrupt+0xb/0x90
[Mon Mar 11 14:29:12 2024] ? asm_sysvec_apic_timer_interrupt+0x16/0x20
[Mon Mar 11 14:29:12 2024] ? __check_object_size.part.0+0x35/0xd0
[Mon Mar 11 14:29:12 2024] ? core_sys_select+0x32f/0x3b0
[Mon Mar 11 14:29:12 2024] core_sys_select+0x1a0/0x3b0
[Mon Mar 11 14:29:12 2024] ? rseq_get_rseq_cs+0x1d/0x240
[Mon Mar 11 14:29:12 2024] ? rseq_ip_fixup+0x6e/0x1a0
[Mon Mar 11 14:29:12 2024] ? __audit_filter_op+0xa5/0xf0
[Mon Mar 11 14:29:12 2024] ? _copy_from_user+0x27/0x60
[Mon Mar 11 14:29:12 2024] do_pselect.constprop.0+0xca/0x170
[Mon Mar 11 14:29:12 2024] __x64_sys_pselect6+0x48/0x70
[Mon Mar 11 14:29:12 2024] do_syscall_64+0x59/0x90
[Mon Mar 11 14:29:12 2024] ? syscall_exit_to_user_mode+0x22/0x40
[Mon Mar 11 14:29:12 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:12 2024] ? switch_fpu_return+0x4c/0xd0
[Mon Mar 11 14:29:12 2024] ? exit_to_user_mode_prepare+0xec/0x100
[Mon Mar 11 14:29:12 2024] ? syscall_exit_to_user_mode+0x22/0x40
[Mon Mar 11 14:29:12 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:12 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:12 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:12 2024] ? sysvec_apic_timer_interrupt+0x3c/0x90
[Mon Mar 11 14:29:12 2024] entry_SYSCALL_64_after_hwframe+0x72/0xdc
[Mon Mar 11 14:29:12 2024] RIP: 0033:0x7fc940f44e5d
[Mon Mar 11 14:29:12 2024] RSP: 002b:00007ffcdf586250 EFLAGS: 00000246 ORIG_RAX: 000000000000010e
[Mon Mar 11 14:29:12 2024] RAX: ffffffffffffffda RBX: 00007ffcdf5862f0 RCX: 00007fc940f44e5d
[Mon Mar 11 14:29:12 2024] RDX: 0000000000000000 RSI: 00007ffcdf586300 RDI: 0000000000000005
[Mon Mar 11 14:29:12 2024] RBP: 00007ffcdf586300 R08: 00007ffcdf586260 R09: 0000000000000000
[Mon Mar 11 14:29:12 2024] R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000000005
[Mon Mar 11 14:29:12 2024] R13: 00007ffcdf586260 R14: 0000000000000000 R15: 0000000000000000
[Mon Mar 11 14:29:12 2024] </TASK>
[Mon Mar 11 14:29:12 2024] task:dmeventd state:S stack:0 pid:6410 ppid:1 flags:0x00000002
[Mon Mar 11 14:29:12 2024] Call Trace:
[Mon Mar 11 14:29:12 2024] <TASK>
[Mon Mar 11 14:29:12 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:12 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:12 2024] dm_wait_event+0x6d/0xa0 [dm_mod]
[Mon Mar 11 14:29:12 2024] ? __pfx_autoremove_wake_function+0x10/0x10
[Mon Mar 11 14:29:12 2024] dev_wait+0x67/0x100 [dm_mod]
[Mon Mar 11 14:29:12 2024] ctl_ioctl+0x19f/0x290 [dm_mod]
[Mon Mar 11 14:29:12 2024] dm_ctl_ioctl+0xa/0x20 [dm_mod]
[Mon Mar 11 14:29:12 2024] __x64_sys_ioctl+0x87/0xc0
[Mon Mar 11 14:29:12 2024] do_syscall_64+0x59/0x90
[Mon Mar 11 14:29:12 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:12 2024] ? syscall_exit_work+0x103/0x130
[Mon Mar 11 14:29:12 2024] ? syscall_exit_to_user_mode+0x22/0x40
[Mon Mar 11 14:29:12 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:12 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:12 2024] ? __irq_exit_rcu+0x46/0xc0
[Mon Mar 11 14:29:12 2024] ? sysvec_apic_timer_interrupt+0x3c/0x90
[Mon Mar 11 14:29:12 2024] entry_SYSCALL_64_after_hwframe+0x72/0xdc
[Mon Mar 11 14:29:12 2024] RIP: 0033:0x7fc940e3ec6b
[Mon Mar 11 14:29:12 2024] RSP: 002b:00007fc940327998 EFLAGS: 00000206 ORIG_RAX: 0000000000000010
[Mon Mar 11 14:29:12 2024] RAX: ffffffffffffffda RBX: 000055fb2dfdf460 RCX: 00007fc940e3ec6b
[Mon Mar 11 14:29:12 2024] RDX: 00007fc93c254fb0 RSI: 00000000c138fd08 RDI: 0000000000000007
[Mon Mar 11 14:29:12 2024] RBP: 00007fc94120a276 R08: 0000000000000004 R09: 00007fc94120d010
[Mon Mar 11 14:29:12 2024] R10: 0000000000000007 R11: 0000000000000206 R12: 00007fc93c254fb0
[Mon Mar 11 14:29:12 2024] R13: 00007fc93c255060 R14: 00007fc94120a276 R15: 000055fb2f482bc0
[Mon Mar 11 14:29:12 2024] </TASK>
[Mon Mar 11 14:29:12 2024] task:dmeventd state:S stack:0 pid:6411 ppid:1 flags:0x00000002
[Mon Mar 11 14:29:12 2024] Call Trace:
[Mon Mar 11 14:29:12 2024] <TASK>
[Mon Mar 11 14:29:12 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:12 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:12 2024] futex_wait_queue+0x70/0xd0
[Mon Mar 11 14:29:12 2024] futex_wait+0x175/0x260
[Mon Mar 11 14:29:12 2024] ? __pfx_hrtimer_wakeup+0x10/0x10
[Mon Mar 11 14:29:12 2024] do_futex+0x12d/0x1d0
[Mon Mar 11 14:29:12 2024] __x64_sys_futex+0x73/0x1d0
[Mon Mar 11 14:29:12 2024] do_syscall_64+0x59/0x90
[Mon Mar 11 14:29:12 2024] ? syscall_exit_to_user_mode+0x22/0x40
[Mon Mar 11 14:29:12 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:12 2024] ? syscall_exit_to_user_mode+0x22/0x40
[Mon Mar 11 14:29:12 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:12 2024] ? audit_reset_context.part.0.constprop.0+0x273/0x2e0
[Mon Mar 11 14:29:12 2024] ? syscall_exit_work+0x103/0x130
[Mon Mar 11 14:29:12 2024] ? syscall_exit_to_user_mode+0x22/0x40
[Mon Mar 11 14:29:12 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:12 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:12 2024] ? syscall_exit_to_user_mode+0x22/0x40
[Mon Mar 11 14:29:12 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:12 2024] ? syscall_exit_to_user_mode+0x22/0x40
[Mon Mar 11 14:29:12 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:12 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:12 2024] ? sysvec_apic_timer_interrupt+0x3c/0x90
[Mon Mar 11 14:29:12 2024] entry_SYSCALL_64_after_hwframe+0x72/0xdc
[Mon Mar 11 14:29:12 2024] RIP: 0033:0x7fc940e9c39a
[Mon Mar 11 14:29:12 2024] RSP: 002b:00007fc9402dabd0 EFLAGS: 00000246 ORIG_RAX: 00000000000000ca
[Mon Mar 11 14:29:12 2024] RAX: ffffffffffffffda RBX: 00007fc9402dad00 RCX: 00007fc940e9c39a
[Mon Mar 11 14:29:12 2024] RDX: 0000000000000000 RSI: 0000000000000189 RDI: 000055fb2dfe60ec
[Mon Mar 11 14:29:12 2024] RBP: 0000000000000000 R08: 0000000000000000 R09: 00000000ffffffff
[Mon Mar 11 14:29:12 2024] R10: 00007fc9402dad00 R11: 0000000000000246 R12: 0000000000000000
[Mon Mar 11 14:29:12 2024] R13: 000055fb2dfe60ec R14: 0000000000000000 R15: 000055fb2dfe6100
[Mon Mar 11 14:29:12 2024] </TASK>
[Mon Mar 11 14:29:12 2024] task:dmeventd state:S stack:0 pid:8698 ppid:1 flags:0x00000002
[Mon Mar 11 14:29:12 2024] Call Trace:
[Mon Mar 11 14:29:12 2024] <TASK>
[Mon Mar 11 14:29:12 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:12 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:12 2024] dm_wait_event+0x6d/0xa0 [dm_mod]
[Mon Mar 11 14:29:12 2024] ? __pfx_autoremove_wake_function+0x10/0x10
[Mon Mar 11 14:29:12 2024] dev_wait+0x67/0x100 [dm_mod]
[Mon Mar 11 14:29:12 2024] ctl_ioctl+0x19f/0x290 [dm_mod]
[Mon Mar 11 14:29:12 2024] dm_ctl_ioctl+0xa/0x20 [dm_mod]
[Mon Mar 11 14:29:12 2024] __x64_sys_ioctl+0x87/0xc0
[Mon Mar 11 14:29:12 2024] do_syscall_64+0x59/0x90
[Mon Mar 11 14:29:12 2024] ? syscall_exit_work+0x103/0x130
[Mon Mar 11 14:29:12 2024] ? syscall_exit_to_user_mode+0x22/0x40
[Mon Mar 11 14:29:12 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:12 2024] ? sysvec_apic_timer_interrupt+0x3c/0x90
[Mon Mar 11 14:29:12 2024] entry_SYSCALL_64_after_hwframe+0x72/0xdc
[Mon Mar 11 14:29:12 2024] RIP: 0033:0x7fc940e3ec6b
[Mon Mar 11 14:29:12 2024] RSP: 002b:00007fc9401ff998 EFLAGS: 00000206 ORIG_RAX: 0000000000000010
[Mon Mar 11 14:29:12 2024] RAX: ffffffffffffffda RBX: 000055fb2dfdf460 RCX: 00007fc940e3ec6b
[Mon Mar 11 14:29:12 2024] RDX: 00007fc934004cd0 RSI: 00000000c138fd08 RDI: 0000000000000007
[Mon Mar 11 14:29:12 2024] RBP: 00007fc94120a276 R08: 0000000000000004 R09: 00007fc94120d010
[Mon Mar 11 14:29:12 2024] R10: 0000000000000007 R11: 0000000000000206 R12: 00007fc934004cd0
[Mon Mar 11 14:29:12 2024] R13: 00007fc934004d80 R14: 00007fc94120a276 R15: 000055fb2f482ca0
[Mon Mar 11 14:29:12 2024] </TASK>
[Mon Mar 11 14:29:12 2024] task:kdmflush/253:92 state:I stack:0 pid:6412 ppid:2 flags:0x00004000
[Mon Mar 11 14:29:12 2024] Call Trace:
[Mon Mar 11 14:29:12 2024] <TASK>
[Mon Mar 11 14:29:12 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:12 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:12 2024] rescuer_thread+0x2ca/0x390
[Mon Mar 11 14:29:12 2024] ? __pfx_rescuer_thread+0x10/0x10
[Mon Mar 11 14:29:12 2024] kthread+0xdd/0x100
[Mon Mar 11 14:29:12 2024] ? __pfx_kthread+0x10/0x10
[Mon Mar 11 14:29:12 2024] ret_from_fork+0x29/0x50
[Mon Mar 11 14:29:12 2024] </TASK>
[Mon Mar 11 14:29:12 2024] task:kdmflush/253:93 state:I stack:0 pid:6413 ppid:2 flags:0x00004000
[Mon Mar 11 14:29:12 2024] Call Trace:
[Mon Mar 11 14:29:12 2024] <TASK>
[Mon Mar 11 14:29:12 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:12 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:12 2024] rescuer_thread+0x2ca/0x390
[Mon Mar 11 14:29:12 2024] ? __pfx_rescuer_thread+0x10/0x10
[Mon Mar 11 14:29:12 2024] kthread+0xdd/0x100
[Mon Mar 11 14:29:12 2024] ? __pfx_kthread+0x10/0x10
[Mon Mar 11 14:29:12 2024] ret_from_fork+0x29/0x50
[Mon Mar 11 14:29:12 2024] </TASK>
[Mon Mar 11 14:29:12 2024] task:kdmflush/253:94 state:I stack:0 pid:6417 ppid:2 flags:0x00004000
[Mon Mar 11 14:29:12 2024] Call Trace:
[Mon Mar 11 14:29:12 2024] <TASK>
[Mon Mar 11 14:29:12 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:12 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:12 2024] rescuer_thread+0x2ca/0x390
[Mon Mar 11 14:29:12 2024] ? __pfx_rescuer_thread+0x10/0x10
[Mon Mar 11 14:29:12 2024] kthread+0xdd/0x100
[Mon Mar 11 14:29:12 2024] ? __pfx_kthread+0x10/0x10
[Mon Mar 11 14:29:12 2024] ret_from_fork+0x29/0x50
[Mon Mar 11 14:29:12 2024] </TASK>
[Mon Mar 11 14:29:12 2024] task:kdmflush/253:95 state:I stack:0 pid:6422 ppid:2 flags:0x00004000
[Mon Mar 11 14:29:12 2024] Call Trace:
[Mon Mar 11 14:29:12 2024] <TASK>
[Mon Mar 11 14:29:12 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:12 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:12 2024] rescuer_thread+0x2ca/0x390
[Mon Mar 11 14:29:12 2024] ? __pfx_rescuer_thread+0x10/0x10
[Mon Mar 11 14:29:12 2024] kthread+0xdd/0x100
[Mon Mar 11 14:29:12 2024] ? __pfx_kthread+0x10/0x10
[Mon Mar 11 14:29:12 2024] ret_from_fork+0x29/0x50
[Mon Mar 11 14:29:12 2024] </TASK>
[Mon Mar 11 14:29:12 2024] task:kdmflush/253:96 state:I stack:0 pid:6427 ppid:2 flags:0x00004000
[Mon Mar 11 14:29:12 2024] Call Trace:
[Mon Mar 11 14:29:12 2024] <TASK>
[Mon Mar 11 14:29:12 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:12 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:12 2024] rescuer_thread+0x2ca/0x390
[Mon Mar 11 14:29:12 2024] ? __pfx_rescuer_thread+0x10/0x10
[Mon Mar 11 14:29:12 2024] kthread+0xdd/0x100
[Mon Mar 11 14:29:12 2024] ? __pfx_kthread+0x10/0x10
[Mon Mar 11 14:29:12 2024] ret_from_fork+0x29/0x50
[Mon Mar 11 14:29:12 2024] </TASK>
[Mon Mar 11 14:29:12 2024] task:kdmflush/253:97 state:I stack:0 pid:6433 ppid:2 flags:0x00004000
[Mon Mar 11 14:29:12 2024] Call Trace:
[Mon Mar 11 14:29:12 2024] <TASK>
[Mon Mar 11 14:29:12 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:12 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:12 2024] rescuer_thread+0x2ca/0x390
[Mon Mar 11 14:29:12 2024] ? __pfx_rescuer_thread+0x10/0x10
[Mon Mar 11 14:29:12 2024] kthread+0xdd/0x100
[Mon Mar 11 14:29:12 2024] ? __pfx_kthread+0x10/0x10
[Mon Mar 11 14:29:12 2024] ret_from_fork+0x29/0x50
[Mon Mar 11 14:29:12 2024] </TASK>
[Mon Mar 11 14:29:12 2024] task:kdmflush/253:98 state:I stack:0 pid:6437 ppid:2 flags:0x00004000
[Mon Mar 11 14:29:12 2024] Call Trace:
[Mon Mar 11 14:29:12 2024] <TASK>
[Mon Mar 11 14:29:12 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:12 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:12 2024] rescuer_thread+0x2ca/0x390
[Mon Mar 11 14:29:12 2024] ? __pfx_rescuer_thread+0x10/0x10
[Mon Mar 11 14:29:12 2024] kthread+0xdd/0x100
[Mon Mar 11 14:29:12 2024] ? __pfx_kthread+0x10/0x10
[Mon Mar 11 14:29:12 2024] ret_from_fork+0x29/0x50
[Mon Mar 11 14:29:12 2024] </TASK>
[Mon Mar 11 14:29:12 2024] task:jbd2/dm-93-8 state:S stack:0 pid:6467 ppid:2 flags:0x00004000
[Mon Mar 11 14:29:12 2024] Call Trace:
[Mon Mar 11 14:29:12 2024] <TASK>
[Mon Mar 11 14:29:12 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:12 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:12 2024] kjournald2+0x221/0x280 [jbd2]
[Mon Mar 11 14:29:12 2024] ? __pfx_autoremove_wake_function+0x10/0x10
[Mon Mar 11 14:29:12 2024] ? __pfx_kjournald2+0x10/0x10 [jbd2]
[Mon Mar 11 14:29:12 2024] kthread+0xdd/0x100
[Mon Mar 11 14:29:12 2024] ? __pfx_kthread+0x10/0x10
[Mon Mar 11 14:29:12 2024] ret_from_fork+0x29/0x50
[Mon Mar 11 14:29:12 2024] </TASK>
[Mon Mar 11 14:29:12 2024] task:ext4-rsv-conver state:I stack:0 pid:6468 ppid:2 flags:0x00004000
[Mon Mar 11 14:29:12 2024] Call Trace:
[Mon Mar 11 14:29:12 2024] <TASK>
[Mon Mar 11 14:29:12 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:12 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:12 2024] rescuer_thread+0x2ca/0x390
[Mon Mar 11 14:29:12 2024] ? __pfx_rescuer_thread+0x10/0x10
[Mon Mar 11 14:29:12 2024] kthread+0xdd/0x100
[Mon Mar 11 14:29:12 2024] ? __pfx_kthread+0x10/0x10
[Mon Mar 11 14:29:12 2024] ret_from_fork+0x29/0x50
[Mon Mar 11 14:29:12 2024] </TASK>
[Mon Mar 11 14:29:12 2024] task:jbd2/dm-94-8 state:S stack:0 pid:6473 ppid:2 flags:0x00004000
[Mon Mar 11 14:29:12 2024] Call Trace:
[Mon Mar 11 14:29:12 2024] <TASK>
[Mon Mar 11 14:29:12 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:12 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:12 2024] kjournald2+0x221/0x280 [jbd2]
[Mon Mar 11 14:29:12 2024] ? __pfx_autoremove_wake_function+0x10/0x10
[Mon Mar 11 14:29:12 2024] ? __pfx_kjournald2+0x10/0x10 [jbd2]
[Mon Mar 11 14:29:12 2024] kthread+0xdd/0x100
[Mon Mar 11 14:29:12 2024] ? __pfx_kthread+0x10/0x10
[Mon Mar 11 14:29:12 2024] ret_from_fork+0x29/0x50
[Mon Mar 11 14:29:12 2024] </TASK>
[Mon Mar 11 14:29:12 2024] task:ext4-rsv-conver state:I stack:0 pid:6474 ppid:2 flags:0x00004000
[Mon Mar 11 14:29:12 2024] Call Trace:
[Mon Mar 11 14:29:12 2024] <TASK>
[Mon Mar 11 14:29:12 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:12 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:12 2024] rescuer_thread+0x2ca/0x390
[Mon Mar 11 14:29:12 2024] ? __pfx_rescuer_thread+0x10/0x10
[Mon Mar 11 14:29:12 2024] kthread+0xdd/0x100
[Mon Mar 11 14:29:12 2024] ? __pfx_kthread+0x10/0x10
[Mon Mar 11 14:29:12 2024] ret_from_fork+0x29/0x50
[Mon Mar 11 14:29:12 2024] </TASK>
[Mon Mar 11 14:29:12 2024] task:jbd2/dm-92-8 state:S stack:0 pid:6475 ppid:2 flags:0x00004000
[Mon Mar 11 14:29:12 2024] Call Trace:
[Mon Mar 11 14:29:12 2024] <TASK>
[Mon Mar 11 14:29:12 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:12 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:12 2024] kjournald2+0x221/0x280 [jbd2]
[Mon Mar 11 14:29:12 2024] ? __pfx_autoremove_wake_function+0x10/0x10
[Mon Mar 11 14:29:12 2024] ? __pfx_kjournald2+0x10/0x10 [jbd2]
[Mon Mar 11 14:29:12 2024] kthread+0xdd/0x100
[Mon Mar 11 14:29:12 2024] ? __pfx_kthread+0x10/0x10
[Mon Mar 11 14:29:12 2024] ret_from_fork+0x29/0x50
[Mon Mar 11 14:29:12 2024] </TASK>
[Mon Mar 11 14:29:12 2024] task:ext4-rsv-conver state:I stack:0 pid:6476 ppid:2 flags:0x00004000
[Mon Mar 11 14:29:12 2024] Call Trace:
[Mon Mar 11 14:29:12 2024] <TASK>
[Mon Mar 11 14:29:12 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:12 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:12 2024] rescuer_thread+0x2ca/0x390
[Mon Mar 11 14:29:12 2024] ? __pfx_rescuer_thread+0x10/0x10
[Mon Mar 11 14:29:12 2024] kthread+0xdd/0x100
[Mon Mar 11 14:29:12 2024] ? __pfx_kthread+0x10/0x10
[Mon Mar 11 14:29:12 2024] ret_from_fork+0x29/0x50
[Mon Mar 11 14:29:12 2024] </TASK>
[Mon Mar 11 14:29:12 2024] task:jbd2/dm-97-8 state:S stack:0 pid:6477 ppid:2 flags:0x00004000
[Mon Mar 11 14:29:12 2024] Call Trace:
[Mon Mar 11 14:29:12 2024] <TASK>
[Mon Mar 11 14:29:12 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:12 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:12 2024] kjournald2+0x221/0x280 [jbd2]
[Mon Mar 11 14:29:12 2024] ? __pfx_autoremove_wake_function+0x10/0x10
[Mon Mar 11 14:29:12 2024] ? __pfx_kjournald2+0x10/0x10 [jbd2]
[Mon Mar 11 14:29:12 2024] kthread+0xdd/0x100
[Mon Mar 11 14:29:12 2024] ? __pfx_kthread+0x10/0x10
[Mon Mar 11 14:29:12 2024] ret_from_fork+0x29/0x50
[Mon Mar 11 14:29:12 2024] </TASK>
[Mon Mar 11 14:29:12 2024] task:ext4-rsv-conver state:I stack:0 pid:6478 ppid:2 flags:0x00004000
[Mon Mar 11 14:29:12 2024] Call Trace:
[Mon Mar 11 14:29:12 2024] <TASK>
[Mon Mar 11 14:29:12 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:12 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:12 2024] rescuer_thread+0x2ca/0x390
[Mon Mar 11 14:29:12 2024] ? __pfx_rescuer_thread+0x10/0x10
[Mon Mar 11 14:29:12 2024] kthread+0xdd/0x100
[Mon Mar 11 14:29:12 2024] ? __pfx_kthread+0x10/0x10
[Mon Mar 11 14:29:12 2024] ret_from_fork+0x29/0x50
[Mon Mar 11 14:29:12 2024] </TASK>
[Mon Mar 11 14:29:12 2024] task:jbd2/dm-98-8 state:S stack:0 pid:6487 ppid:2 flags:0x00004000
[Mon Mar 11 14:29:12 2024] Call Trace:
[Mon Mar 11 14:29:12 2024] <TASK>
[Mon Mar 11 14:29:12 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:12 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:12 2024] kjournald2+0x221/0x280 [jbd2]
[Mon Mar 11 14:29:12 2024] ? __pfx_autoremove_wake_function+0x10/0x10
[Mon Mar 11 14:29:12 2024] ? __pfx_kjournald2+0x10/0x10 [jbd2]
[Mon Mar 11 14:29:12 2024] kthread+0xdd/0x100
[Mon Mar 11 14:29:12 2024] ? __pfx_kthread+0x10/0x10
[Mon Mar 11 14:29:12 2024] ret_from_fork+0x29/0x50
[Mon Mar 11 14:29:12 2024] </TASK>
[Mon Mar 11 14:29:12 2024] task:jbd2/dm-95-8 state:S stack:0 pid:6488 ppid:2 flags:0x00004000
[Mon Mar 11 14:29:12 2024] Call Trace:
[Mon Mar 11 14:29:12 2024] <TASK>
[Mon Mar 11 14:29:12 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:12 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:12 2024] kjournald2+0x221/0x280 [jbd2]
[Mon Mar 11 14:29:12 2024] ? __pfx_autoremove_wake_function+0x10/0x10
[Mon Mar 11 14:29:12 2024] ? __pfx_kjournald2+0x10/0x10 [jbd2]
[Mon Mar 11 14:29:12 2024] kthread+0xdd/0x100
[Mon Mar 11 14:29:12 2024] ? __pfx_kthread+0x10/0x10
[Mon Mar 11 14:29:12 2024] ret_from_fork+0x29/0x50
[Mon Mar 11 14:29:12 2024] </TASK>
[Mon Mar 11 14:29:12 2024] task:ext4-rsv-conver state:I stack:0 pid:6489 ppid:2 flags:0x00004000
[Mon Mar 11 14:29:12 2024] Call Trace:
[Mon Mar 11 14:29:12 2024] <TASK>
[Mon Mar 11 14:29:12 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:12 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:12 2024] rescuer_thread+0x2ca/0x390
[Mon Mar 11 14:29:12 2024] ? __pfx_rescuer_thread+0x10/0x10
[Mon Mar 11 14:29:12 2024] kthread+0xdd/0x100
[Mon Mar 11 14:29:12 2024] ? __pfx_kthread+0x10/0x10
[Mon Mar 11 14:29:12 2024] ret_from_fork+0x29/0x50
[Mon Mar 11 14:29:12 2024] </TASK>
[Mon Mar 11 14:29:12 2024] task:ext4-rsv-conver state:I stack:0 pid:6490 ppid:2 flags:0x00004000
[Mon Mar 11 14:29:12 2024] Call Trace:
[Mon Mar 11 14:29:12 2024] <TASK>
[Mon Mar 11 14:29:12 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:12 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:12 2024] rescuer_thread+0x2ca/0x390
[Mon Mar 11 14:29:12 2024] ? __pfx_rescuer_thread+0x10/0x10
[Mon Mar 11 14:29:12 2024] kthread+0xdd/0x100
[Mon Mar 11 14:29:12 2024] ? __pfx_kthread+0x10/0x10
[Mon Mar 11 14:29:12 2024] ret_from_fork+0x29/0x50
[Mon Mar 11 14:29:12 2024] </TASK>
[Mon Mar 11 14:29:12 2024] task:jbd2/dm-96-8 state:S stack:0 pid:6491 ppid:2 flags:0x00004000
[Mon Mar 11 14:29:12 2024] Call Trace:
[Mon Mar 11 14:29:12 2024] <TASK>
[Mon Mar 11 14:29:12 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:12 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:12 2024] kjournald2+0x221/0x280 [jbd2]
[Mon Mar 11 14:29:12 2024] ? __pfx_autoremove_wake_function+0x10/0x10
[Mon Mar 11 14:29:12 2024] ? __pfx_kjournald2+0x10/0x10 [jbd2]
[Mon Mar 11 14:29:12 2024] kthread+0xdd/0x100
[Mon Mar 11 14:29:12 2024] ? __pfx_kthread+0x10/0x10
[Mon Mar 11 14:29:12 2024] ret_from_fork+0x29/0x50
[Mon Mar 11 14:29:12 2024] </TASK>
[Mon Mar 11 14:29:12 2024] task:ext4-rsv-conver state:I stack:0 pid:6492 ppid:2 flags:0x00004000
[Mon Mar 11 14:29:12 2024] Call Trace:
[Mon Mar 11 14:29:12 2024] <TASK>
[Mon Mar 11 14:29:12 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:12 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:12 2024] rescuer_thread+0x2ca/0x390
[Mon Mar 11 14:29:12 2024] ? __pfx_rescuer_thread+0x10/0x10
[Mon Mar 11 14:29:12 2024] kthread+0xdd/0x100
[Mon Mar 11 14:29:12 2024] ? __pfx_kthread+0x10/0x10
[Mon Mar 11 14:29:12 2024] ret_from_fork+0x29/0x50
[Mon Mar 11 14:29:12 2024] </TASK>
[Mon Mar 11 14:29:12 2024] task:upsclient state:S stack:0 pid:7154 ppid:1 flags:0x00000002
[Mon Mar 11 14:29:12 2024] Call Trace:
[Mon Mar 11 14:29:12 2024] <TASK>
[Mon Mar 11 14:29:12 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:12 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:12 2024] schedule_hrtimeout_range_clock+0x95/0x120
[Mon Mar 11 14:29:12 2024] ? __pfx_hrtimer_wakeup+0x10/0x10
[Mon Mar 11 14:29:12 2024] do_select+0x69e/0x7c0
[Mon Mar 11 14:29:12 2024] ? schedule+0x2d/0x70
[Mon Mar 11 14:29:12 2024] ? __pfx_pollwake+0x10/0x10
[Mon Mar 11 14:29:12 2024] ? __pfx_pollwake+0x10/0x10
[Mon Mar 11 14:29:12 2024] ? check_heap_object+0x34/0x150
[Mon Mar 11 14:29:12 2024] ? _copy_to_iter+0x1d4/0x630
[Mon Mar 11 14:29:12 2024] ? check_heap_object+0x34/0x150
[Mon Mar 11 14:29:12 2024] ? __check_object_size.part.0+0x47/0xd0
[Mon Mar 11 14:29:12 2024] ? __skb_datagram_iter+0x79/0x2e0
[Mon Mar 11 14:29:12 2024] ? __pfx_simple_copy_to_iter+0x10/0x10
[Mon Mar 11 14:29:12 2024] ? __tcp_cleanup_rbuf+0xa0/0xc0
[Mon Mar 11 14:29:12 2024] ? tcp_recvmsg_locked+0x272/0x910
[Mon Mar 11 14:29:12 2024] core_sys_select+0x1a0/0x3b0
[Mon Mar 11 14:29:12 2024] ? core_sys_select+0x1f7/0x3b0
[Mon Mar 11 14:29:12 2024] ? _copy_from_user+0x27/0x60
[Mon Mar 11 14:29:12 2024] do_pselect.constprop.0+0xca/0x170
[Mon Mar 11 14:29:12 2024] __x64_sys_pselect6+0x48/0x70
[Mon Mar 11 14:29:12 2024] do_syscall_64+0x59/0x90
[Mon Mar 11 14:29:12 2024] ? switch_fpu_return+0x4c/0xd0
[Mon Mar 11 14:29:12 2024] ? exit_to_user_mode_prepare+0xec/0x100
[Mon Mar 11 14:29:12 2024] ? syscall_exit_to_user_mode+0x22/0x40
[Mon Mar 11 14:29:12 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:12 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:12 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:12 2024] ? sysvec_apic_timer_interrupt+0x3c/0x90
[Mon Mar 11 14:29:12 2024] entry_SYSCALL_64_after_hwframe+0x72/0xdc
[Mon Mar 11 14:29:12 2024] RIP: 0033:0x7f977ff44e5d
[Mon Mar 11 14:29:12 2024] RSP: 002b:00007ffd00510e60 EFLAGS: 00000246 ORIG_RAX: 000000000000010e
[Mon Mar 11 14:29:12 2024] RAX: ffffffffffffffda RBX: 00007ffd00510f00 RCX: 00007f977ff44e5d
[Mon Mar 11 14:29:12 2024] RDX: 00007ffd00510f90 RSI: 00007ffd00510f10 RDI: 0000000000000005
[Mon Mar 11 14:29:12 2024] RBP: 00007ffd00510f10 R08: 00007ffd00510e70 R09: 0000000000000000
[Mon Mar 11 14:29:12 2024] R10: 00007ffd00511010 R11: 0000000000000246 R12: 0000000000000005
[Mon Mar 11 14:29:12 2024] R13: 00007ffd00510e70 R14: 0000000000000000 R15: 00007ffd00510f90
[Mon Mar 11 14:29:12 2024] </TASK>
[Mon Mar 11 14:29:12 2024] task:upsclient state:S stack:0 pid:7160 ppid:1 flags:0x00000002
[Mon Mar 11 14:29:12 2024] Call Trace:
[Mon Mar 11 14:29:12 2024] <TASK>
[Mon Mar 11 14:29:12 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:12 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:12 2024] futex_wait_queue+0x70/0xd0
[Mon Mar 11 14:29:12 2024] futex_wait+0x175/0x260
[Mon Mar 11 14:29:12 2024] ? sched_clock+0xc/0x30
[Mon Mar 11 14:29:12 2024] ? __smp_call_single_queue+0x93/0x120
[Mon Mar 11 14:29:12 2024] ? ttwu_queue_wakelist+0xf2/0x110
[Mon Mar 11 14:29:12 2024] do_futex+0x12d/0x1d0
[Mon Mar 11 14:29:12 2024] __x64_sys_futex+0x73/0x1d0
[Mon Mar 11 14:29:12 2024] do_syscall_64+0x59/0x90
[Mon Mar 11 14:29:12 2024] ? syscall_exit_work+0x103/0x130
[Mon Mar 11 14:29:12 2024] ? syscall_exit_to_user_mode+0x22/0x40
[Mon Mar 11 14:29:12 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:12 2024] ? exc_page_fault+0x62/0x150
[Mon Mar 11 14:29:12 2024] entry_SYSCALL_64_after_hwframe+0x72/0xdc
[Mon Mar 11 14:29:12 2024] RIP: 0033:0x7f977fe9c39a
[Mon Mar 11 14:29:12 2024] RSP: 002b:00007f977ef4d2a0 EFLAGS: 00000246 ORIG_RAX: 00000000000000ca
[Mon Mar 11 14:29:12 2024] RAX: ffffffffffffffda RBX: 00007f9778000d80 RCX: 00007f977fe9c39a
[Mon Mar 11 14:29:12 2024] RDX: 0000000000000000 RSI: 0000000000000189 RDI: 00007f9778000d80
[Mon Mar 11 14:29:12 2024] RBP: 0000000000000000 R08: 0000000000000000 R09: 00000000ffffffff
[Mon Mar 11 14:29:12 2024] R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000000000
[Mon Mar 11 14:29:12 2024] R13: fffffffeffffffff R14: 00007f97800ce501 R15: 0000000000000000
[Mon Mar 11 14:29:12 2024] </TASK>
[Mon Mar 11 14:29:12 2024] task:upsclient state:S stack:0 pid:7162 ppid:1 flags:0x00000002
[Mon Mar 11 14:29:12 2024] Call Trace:
[Mon Mar 11 14:29:12 2024] <TASK>
[Mon Mar 11 14:29:12 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:12 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:12 2024] futex_wait_queue+0x70/0xd0
[Mon Mar 11 14:29:12 2024] futex_wait+0x175/0x260
[Mon Mar 11 14:29:12 2024] ? select_idle_sibling+0x28/0x430
[Mon Mar 11 14:29:12 2024] ? wake_affine+0x62/0x1f0
[Mon Mar 11 14:29:12 2024] ? sched_clock+0xc/0x30
[Mon Mar 11 14:29:12 2024] ? __smp_call_single_queue+0x93/0x120
[Mon Mar 11 14:29:12 2024] ? ttwu_queue_wakelist+0xf2/0x110
[Mon Mar 11 14:29:12 2024] do_futex+0x12d/0x1d0
[Mon Mar 11 14:29:12 2024] __x64_sys_futex+0x73/0x1d0
[Mon Mar 11 14:29:12 2024] do_syscall_64+0x59/0x90
[Mon Mar 11 14:29:12 2024] ? audit_reset_context.part.0.constprop.0+0x273/0x2e0
[Mon Mar 11 14:29:12 2024] ? syscall_exit_work+0x103/0x130
[Mon Mar 11 14:29:12 2024] ? syscall_exit_to_user_mode+0x22/0x40
[Mon Mar 11 14:29:12 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:12 2024] entry_SYSCALL_64_after_hwframe+0x72/0xdc
[Mon Mar 11 14:29:12 2024] RIP: 0033:0x7f977fe9c39a
[Mon Mar 11 14:29:12 2024] RSP: 002b:00007f977e74c2a0 EFLAGS: 00000246 ORIG_RAX: 00000000000000ca
[Mon Mar 11 14:29:12 2024] RAX: ffffffffffffffda RBX: 00007f9770000d80 RCX: 00007f977fe9c39a
[Mon Mar 11 14:29:12 2024] RDX: 0000000000000000 RSI: 0000000000000189 RDI: 00007f9770000d80
[Mon Mar 11 14:29:12 2024] RBP: 0000000000000000 R08: 0000000000000000 R09: 00000000ffffffff
[Mon Mar 11 14:29:12 2024] R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000000000
[Mon Mar 11 14:29:12 2024] R13: fffffffeffffffff R14: 00007f97800ce501 R15: 0000000000000000
[Mon Mar 11 14:29:12 2024] </TASK>
[Mon Mar 11 14:29:12 2024] task:kdmflush/253:99 state:I stack:0 pid:8694 ppid:2 flags:0x00004000
[Mon Mar 11 14:29:12 2024] Call Trace:
[Mon Mar 11 14:29:12 2024] <TASK>
[Mon Mar 11 14:29:12 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:12 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:12 2024] rescuer_thread+0x2ca/0x390
[Mon Mar 11 14:29:12 2024] ? __pfx_rescuer_thread+0x10/0x10
[Mon Mar 11 14:29:12 2024] kthread+0xdd/0x100
[Mon Mar 11 14:29:12 2024] ? __pfx_kthread+0x10/0x10
[Mon Mar 11 14:29:12 2024] ret_from_fork+0x29/0x50
[Mon Mar 11 14:29:12 2024] </TASK>
[Mon Mar 11 14:29:12 2024] task:kcopyd state:I stack:0 pid:8695 ppid:2 flags:0x00004000
[Mon Mar 11 14:29:12 2024] Call Trace:
[Mon Mar 11 14:29:12 2024] <TASK>
[Mon Mar 11 14:29:12 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:12 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:12 2024] rescuer_thread+0x2ca/0x390
[Mon Mar 11 14:29:12 2024] ? __pfx_rescuer_thread+0x10/0x10
[Mon Mar 11 14:29:12 2024] kthread+0xdd/0x100
[Mon Mar 11 14:29:12 2024] ? __pfx_kthread+0x10/0x10
[Mon Mar 11 14:29:12 2024] ret_from_fork+0x29/0x50
[Mon Mar 11 14:29:12 2024] </TASK>
[Mon Mar 11 14:29:12 2024] task:dm-thin state:I stack:0 pid:8696 ppid:2 flags:0x00004000
[Mon Mar 11 14:29:12 2024] Call Trace:
[Mon Mar 11 14:29:12 2024] <TASK>
[Mon Mar 11 14:29:12 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:12 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:12 2024] rescuer_thread+0x2ca/0x390
[Mon Mar 11 14:29:12 2024] ? __pfx_rescuer_thread+0x10/0x10
[Mon Mar 11 14:29:12 2024] kthread+0xdd/0x100
[Mon Mar 11 14:29:12 2024] ? __pfx_kthread+0x10/0x10
[Mon Mar 11 14:29:12 2024] ret_from_fork+0x29/0x50
[Mon Mar 11 14:29:12 2024] </TASK>
[Mon Mar 11 14:29:12 2024] task:kdmflush/253:10 state:I stack:0 pid:8697 ppid:2 flags:0x00004000
[Mon Mar 11 14:29:12 2024] Call Trace:
[Mon Mar 11 14:29:12 2024] <TASK>
[Mon Mar 11 14:29:12 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:12 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:12 2024] rescuer_thread+0x2ca/0x390
[Mon Mar 11 14:29:12 2024] ? __pfx_rescuer_thread+0x10/0x10
[Mon Mar 11 14:29:12 2024] kthread+0xdd/0x100
[Mon Mar 11 14:29:12 2024] ? __pfx_kthread+0x10/0x10
[Mon Mar 11 14:29:12 2024] ret_from_fork+0x29/0x50
[Mon Mar 11 14:29:12 2024] </TASK>
[Mon Mar 11 14:29:12 2024] task:kdmflush/253:10 state:I stack:0 pid:8699 ppid:2 flags:0x00004000
[Mon Mar 11 14:29:12 2024] Call Trace:
[Mon Mar 11 14:29:12 2024] <TASK>
[Mon Mar 11 14:29:12 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:12 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:12 2024] rescuer_thread+0x2ca/0x390
[Mon Mar 11 14:29:12 2024] ? __pfx_rescuer_thread+0x10/0x10
[Mon Mar 11 14:29:12 2024] kthread+0xdd/0x100
[Mon Mar 11 14:29:12 2024] ? __pfx_kthread+0x10/0x10
[Mon Mar 11 14:29:12 2024] ret_from_fork+0x29/0x50
[Mon Mar 11 14:29:12 2024] </TASK>
[Mon Mar 11 14:29:12 2024] task:kdmflush/253:10 state:I stack:0 pid:8710 ppid:2 flags:0x00004000
[Mon Mar 11 14:29:12 2024] Call Trace:
[Mon Mar 11 14:29:12 2024] <TASK>
[Mon Mar 11 14:29:12 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:12 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:12 2024] rescuer_thread+0x2ca/0x390
[Mon Mar 11 14:29:12 2024] ? __pfx_rescuer_thread+0x10/0x10
[Mon Mar 11 14:29:12 2024] kthread+0xdd/0x100
[Mon Mar 11 14:29:12 2024] ? __pfx_kthread+0x10/0x10
[Mon Mar 11 14:29:12 2024] ret_from_fork+0x29/0x50
[Mon Mar 11 14:29:12 2024] </TASK>
[Mon Mar 11 14:29:12 2024] task:kdmflush/253:10 state:I stack:0 pid:8719 ppid:2 flags:0x00004000
[Mon Mar 11 14:29:12 2024] Call Trace:
[Mon Mar 11 14:29:12 2024] <TASK>
[Mon Mar 11 14:29:12 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:12 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:12 2024] rescuer_thread+0x2ca/0x390
[Mon Mar 11 14:29:12 2024] ? __pfx_rescuer_thread+0x10/0x10
[Mon Mar 11 14:29:12 2024] kthread+0xdd/0x100
[Mon Mar 11 14:29:12 2024] ? __pfx_kthread+0x10/0x10
[Mon Mar 11 14:29:12 2024] ret_from_fork+0x29/0x50
[Mon Mar 11 14:29:12 2024] </TASK>
[Mon Mar 11 14:29:12 2024] task:jbd2/dm-102-8 state:S stack:0 pid:8732 ppid:2 flags:0x00004000
[Mon Mar 11 14:29:12 2024] Call Trace:
[Mon Mar 11 14:29:12 2024] <TASK>
[Mon Mar 11 14:29:12 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:12 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:12 2024] kjournald2+0x221/0x280 [jbd2]
[Mon Mar 11 14:29:12 2024] ? __pfx_autoremove_wake_function+0x10/0x10
[Mon Mar 11 14:29:12 2024] ? __pfx_kjournald2+0x10/0x10 [jbd2]
[Mon Mar 11 14:29:12 2024] kthread+0xdd/0x100
[Mon Mar 11 14:29:12 2024] ? __pfx_kthread+0x10/0x10
[Mon Mar 11 14:29:12 2024] ret_from_fork+0x29/0x50
[Mon Mar 11 14:29:12 2024] </TASK>
[Mon Mar 11 14:29:12 2024] task:ext4-rsv-conver state:I stack:0 pid:8733 ppid:2 flags:0x00004000
[Mon Mar 11 14:29:12 2024] Call Trace:
[Mon Mar 11 14:29:12 2024] <TASK>
[Mon Mar 11 14:29:12 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:12 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:12 2024] rescuer_thread+0x2ca/0x390
[Mon Mar 11 14:29:12 2024] ? __pfx_rescuer_thread+0x10/0x10
[Mon Mar 11 14:29:12 2024] kthread+0xdd/0x100
[Mon Mar 11 14:29:12 2024] ? __pfx_kthread+0x10/0x10
[Mon Mar 11 14:29:12 2024] ret_from_fork+0x29/0x50
[Mon Mar 11 14:29:12 2024] </TASK>
[Mon Mar 11 14:29:12 2024] task:jbd2/dm-101-8 state:S stack:0 pid:8746 ppid:2 flags:0x00004000
[Mon Mar 11 14:29:12 2024] Call Trace:
[Mon Mar 11 14:29:12 2024] <TASK>
[Mon Mar 11 14:29:12 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:12 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:12 2024] kjournald2+0x221/0x280 [jbd2]
[Mon Mar 11 14:29:12 2024] ? __pfx_autoremove_wake_function+0x10/0x10
[Mon Mar 11 14:29:12 2024] ? __pfx_kjournald2+0x10/0x10 [jbd2]
[Mon Mar 11 14:29:12 2024] kthread+0xdd/0x100
[Mon Mar 11 14:29:12 2024] ? __pfx_kthread+0x10/0x10
[Mon Mar 11 14:29:12 2024] ret_from_fork+0x29/0x50
[Mon Mar 11 14:29:12 2024] </TASK>
[Mon Mar 11 14:29:12 2024] task:ext4-rsv-conver state:I stack:0 pid:8747 ppid:2 flags:0x00004000
[Mon Mar 11 14:29:12 2024] Call Trace:
[Mon Mar 11 14:29:12 2024] <TASK>
[Mon Mar 11 14:29:12 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:12 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:12 2024] rescuer_thread+0x2ca/0x390
[Mon Mar 11 14:29:12 2024] ? __pfx_rescuer_thread+0x10/0x10
[Mon Mar 11 14:29:12 2024] kthread+0xdd/0x100
[Mon Mar 11 14:29:12 2024] ? __pfx_kthread+0x10/0x10
[Mon Mar 11 14:29:12 2024] ret_from_fork+0x29/0x50
[Mon Mar 11 14:29:12 2024] </TASK>
[Mon Mar 11 14:29:12 2024] task:ib-comp-wq state:I stack:0 pid:8762 ppid:2 flags:0x00004000
[Mon Mar 11 14:29:12 2024] Call Trace:
[Mon Mar 11 14:29:12 2024] <TASK>
[Mon Mar 11 14:29:12 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:12 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:12 2024] rescuer_thread+0x2ca/0x390
[Mon Mar 11 14:29:12 2024] ? __pfx_rescuer_thread+0x10/0x10
[Mon Mar 11 14:29:12 2024] kthread+0xdd/0x100
[Mon Mar 11 14:29:12 2024] ? __pfx_kthread+0x10/0x10
[Mon Mar 11 14:29:12 2024] ret_from_fork+0x29/0x50
[Mon Mar 11 14:29:12 2024] </TASK>
[Mon Mar 11 14:29:12 2024] task:ib-comp-unb-wq state:I stack:0 pid:8763 ppid:2 flags:0x00004000
[Mon Mar 11 14:29:12 2024] Call Trace:
[Mon Mar 11 14:29:12 2024] <TASK>
[Mon Mar 11 14:29:12 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:12 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:12 2024] rescuer_thread+0x2ca/0x390
[Mon Mar 11 14:29:12 2024] ? __pfx_rescuer_thread+0x10/0x10
[Mon Mar 11 14:29:12 2024] kthread+0xdd/0x100
[Mon Mar 11 14:29:12 2024] ? __pfx_kthread+0x10/0x10
[Mon Mar 11 14:29:12 2024] ret_from_fork+0x29/0x50
[Mon Mar 11 14:29:12 2024] </TASK>
[Mon Mar 11 14:29:12 2024] task:ib_mcast state:I stack:0 pid:8764 ppid:2 flags:0x00004000
[Mon Mar 11 14:29:12 2024] Call Trace:
[Mon Mar 11 14:29:12 2024] <TASK>
[Mon Mar 11 14:29:12 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:12 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:12 2024] rescuer_thread+0x2ca/0x390
[Mon Mar 11 14:29:12 2024] ? __pfx_rescuer_thread+0x10/0x10
[Mon Mar 11 14:29:12 2024] kthread+0xdd/0x100
[Mon Mar 11 14:29:12 2024] ? __pfx_kthread+0x10/0x10
[Mon Mar 11 14:29:12 2024] ret_from_fork+0x29/0x50
[Mon Mar 11 14:29:12 2024] </TASK>
[Mon Mar 11 14:29:12 2024] task:ib_nl_sa_wq state:I stack:0 pid:8765 ppid:2 flags:0x00004000
[Mon Mar 11 14:29:12 2024] Call Trace:
[Mon Mar 11 14:29:12 2024] <TASK>
[Mon Mar 11 14:29:12 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:12 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:12 2024] rescuer_thread+0x2ca/0x390
[Mon Mar 11 14:29:12 2024] ? __pfx_rescuer_thread+0x10/0x10
[Mon Mar 11 14:29:12 2024] kthread+0xdd/0x100
[Mon Mar 11 14:29:12 2024] ? __pfx_kthread+0x10/0x10
[Mon Mar 11 14:29:12 2024] ret_from_fork+0x29/0x50
[Mon Mar 11 14:29:12 2024] </TASK>
[Mon Mar 11 14:29:12 2024] task:rdma_cm state:I stack:0 pid:8770 ppid:2 flags:0x00004000
[Mon Mar 11 14:29:12 2024] Call Trace:
[Mon Mar 11 14:29:12 2024] <TASK>
[Mon Mar 11 14:29:12 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:12 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:12 2024] rescuer_thread+0x2ca/0x390
[Mon Mar 11 14:29:12 2024] ? __pfx_rescuer_thread+0x10/0x10
[Mon Mar 11 14:29:12 2024] kthread+0xdd/0x100
[Mon Mar 11 14:29:12 2024] ? __pfx_kthread+0x10/0x10
[Mon Mar 11 14:29:12 2024] ret_from_fork+0x29/0x50
[Mon Mar 11 14:29:12 2024] </TASK>
[Mon Mar 11 14:29:12 2024] task:nfsd state:I stack:0 pid:8782 ppid:2 flags:0x00004000
[Mon Mar 11 14:29:12 2024] Call Trace:
[Mon Mar 11 14:29:12 2024] <TASK>
[Mon Mar 11 14:29:12 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:12 2024] ? __pfx_nfsd+0x10/0x10 [nfsd]
[Mon Mar 11 14:29:12 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:12 2024] svc_get_next_xprt+0x10e/0x190 [sunrpc]
[Mon Mar 11 14:29:12 2024] svc_recv+0x57/0x150 [sunrpc]
[Mon Mar 11 14:29:12 2024] nfsd+0x84/0xb0 [nfsd]
[Mon Mar 11 14:29:12 2024] kthread+0xdd/0x100
[Mon Mar 11 14:29:12 2024] ? __pfx_kthread+0x10/0x10
[Mon Mar 11 14:29:12 2024] ret_from_fork+0x29/0x50
[Mon Mar 11 14:29:12 2024] </TASK>
[Mon Mar 11 14:29:12 2024] task:nfsd state:I stack:0 pid:8783 ppid:2 flags:0x00004000
[Mon Mar 11 14:29:12 2024] Call Trace:
[Mon Mar 11 14:29:12 2024] <TASK>
[Mon Mar 11 14:29:12 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:12 2024] ? __pfx_nfsd+0x10/0x10 [nfsd]
[Mon Mar 11 14:29:12 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:12 2024] svc_get_next_xprt+0x10e/0x190 [sunrpc]
[Mon Mar 11 14:29:12 2024] svc_recv+0x57/0x150 [sunrpc]
[Mon Mar 11 14:29:12 2024] nfsd+0x84/0xb0 [nfsd]
[Mon Mar 11 14:29:12 2024] kthread+0xdd/0x100
[Mon Mar 11 14:29:12 2024] ? __pfx_kthread+0x10/0x10
[Mon Mar 11 14:29:12 2024] ret_from_fork+0x29/0x50
[Mon Mar 11 14:29:12 2024] </TASK>
[Mon Mar 11 14:29:12 2024] task:nfsd state:I stack:0 pid:8784 ppid:2 flags:0x00004000
[Mon Mar 11 14:29:12 2024] Call Trace:
[Mon Mar 11 14:29:12 2024] <TASK>
[Mon Mar 11 14:29:12 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:12 2024] ? __pfx_nfsd+0x10/0x10 [nfsd]
[Mon Mar 11 14:29:12 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:12 2024] svc_get_next_xprt+0x10e/0x190 [sunrpc]
[Mon Mar 11 14:29:12 2024] svc_recv+0x57/0x150 [sunrpc]
[Mon Mar 11 14:29:12 2024] nfsd+0x84/0xb0 [nfsd]
[Mon Mar 11 14:29:12 2024] kthread+0xdd/0x100
[Mon Mar 11 14:29:12 2024] ? __pfx_kthread+0x10/0x10
[Mon Mar 11 14:29:12 2024] ret_from_fork+0x29/0x50
[Mon Mar 11 14:29:12 2024] </TASK>
[Mon Mar 11 14:29:12 2024] task:nfsd state:I stack:0 pid:8785 ppid:2 flags:0x00004000
[Mon Mar 11 14:29:12 2024] Call Trace:
[Mon Mar 11 14:29:12 2024] <TASK>
[Mon Mar 11 14:29:12 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:12 2024] ? __pfx_nfsd+0x10/0x10 [nfsd]
[Mon Mar 11 14:29:12 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:12 2024] svc_get_next_xprt+0x10e/0x190 [sunrpc]
[Mon Mar 11 14:29:12 2024] svc_recv+0x57/0x150 [sunrpc]
[Mon Mar 11 14:29:12 2024] nfsd+0x84/0xb0 [nfsd]
[Mon Mar 11 14:29:12 2024] kthread+0xdd/0x100
[Mon Mar 11 14:29:12 2024] ? __pfx_kthread+0x10/0x10
[Mon Mar 11 14:29:12 2024] ret_from_fork+0x29/0x50
[Mon Mar 11 14:29:12 2024] </TASK>
[Mon Mar 11 14:29:12 2024] task:nfsd state:I stack:0 pid:8786 ppid:2 flags:0x00004000
[Mon Mar 11 14:29:12 2024] Call Trace:
[Mon Mar 11 14:29:12 2024] <TASK>
[Mon Mar 11 14:29:12 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:12 2024] ? __pfx_nfsd+0x10/0x10 [nfsd]
[Mon Mar 11 14:29:12 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:12 2024] svc_get_next_xprt+0x10e/0x190 [sunrpc]
[Mon Mar 11 14:29:12 2024] svc_recv+0x57/0x150 [sunrpc]
[Mon Mar 11 14:29:12 2024] nfsd+0x84/0xb0 [nfsd]
[Mon Mar 11 14:29:12 2024] kthread+0xdd/0x100
[Mon Mar 11 14:29:12 2024] ? __pfx_kthread+0x10/0x10
[Mon Mar 11 14:29:12 2024] ret_from_fork+0x29/0x50
[Mon Mar 11 14:29:12 2024] </TASK>
[Mon Mar 11 14:29:12 2024] task:nfsd state:I stack:0 pid:8787 ppid:2 flags:0x00004000
[Mon Mar 11 14:29:12 2024] Call Trace:
[Mon Mar 11 14:29:12 2024] <TASK>
[Mon Mar 11 14:29:12 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:12 2024] ? __pfx_nfsd+0x10/0x10 [nfsd]
[Mon Mar 11 14:29:12 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:12 2024] svc_get_next_xprt+0x10e/0x190 [sunrpc]
[Mon Mar 11 14:29:12 2024] svc_recv+0x57/0x150 [sunrpc]
[Mon Mar 11 14:29:12 2024] nfsd+0x84/0xb0 [nfsd]
[Mon Mar 11 14:29:12 2024] kthread+0xdd/0x100
[Mon Mar 11 14:29:12 2024] ? __pfx_kthread+0x10/0x10
[Mon Mar 11 14:29:12 2024] ret_from_fork+0x29/0x50
[Mon Mar 11 14:29:12 2024] </TASK>
[Mon Mar 11 14:29:12 2024] task:nfsd state:I stack:0 pid:8788 ppid:2 flags:0x00004000
[Mon Mar 11 14:29:12 2024] Call Trace:
[Mon Mar 11 14:29:12 2024] <TASK>
[Mon Mar 11 14:29:12 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:12 2024] ? __pfx_nfsd+0x10/0x10 [nfsd]
[Mon Mar 11 14:29:12 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:12 2024] svc_get_next_xprt+0x10e/0x190 [sunrpc]
[Mon Mar 11 14:29:12 2024] svc_recv+0x57/0x150 [sunrpc]
[Mon Mar 11 14:29:12 2024] nfsd+0x84/0xb0 [nfsd]
[Mon Mar 11 14:29:12 2024] kthread+0xdd/0x100
[Mon Mar 11 14:29:12 2024] ? __pfx_kthread+0x10/0x10
[Mon Mar 11 14:29:12 2024] ret_from_fork+0x29/0x50
[Mon Mar 11 14:29:12 2024] </TASK>
[Mon Mar 11 14:29:12 2024] task:nfsd state:I stack:0 pid:8789 ppid:2 flags:0x00004000
[Mon Mar 11 14:29:12 2024] Call Trace:
[Mon Mar 11 14:29:12 2024] <TASK>
[Mon Mar 11 14:29:12 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:12 2024] ? __pfx_nfsd+0x10/0x10 [nfsd]
[Mon Mar 11 14:29:12 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:12 2024] svc_get_next_xprt+0x10e/0x190 [sunrpc]
[Mon Mar 11 14:29:12 2024] svc_recv+0x57/0x150 [sunrpc]
[Mon Mar 11 14:29:12 2024] nfsd+0x84/0xb0 [nfsd]
[Mon Mar 11 14:29:12 2024] kthread+0xdd/0x100
[Mon Mar 11 14:29:12 2024] ? __pfx_kthread+0x10/0x10
[Mon Mar 11 14:29:12 2024] ret_from_fork+0x29/0x50
[Mon Mar 11 14:29:12 2024] </TASK>
[Mon Mar 11 14:29:12 2024] task:nfsd state:I stack:0 pid:8790 ppid:2 flags:0x00004000
[Mon Mar 11 14:29:12 2024] Call Trace:
[Mon Mar 11 14:29:12 2024] <TASK>
[Mon Mar 11 14:29:12 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:12 2024] ? __pfx_nfsd+0x10/0x10 [nfsd]
[Mon Mar 11 14:29:12 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:12 2024] svc_get_next_xprt+0x10e/0x190 [sunrpc]
[Mon Mar 11 14:29:12 2024] svc_recv+0x57/0x150 [sunrpc]
[Mon Mar 11 14:29:12 2024] nfsd+0x84/0xb0 [nfsd]
[Mon Mar 11 14:29:12 2024] kthread+0xdd/0x100
[Mon Mar 11 14:29:12 2024] ? __pfx_kthread+0x10/0x10
[Mon Mar 11 14:29:12 2024] ret_from_fork+0x29/0x50
[Mon Mar 11 14:29:12 2024] </TASK>
[Mon Mar 11 14:29:12 2024] task:nfsd state:I stack:0 pid:8791 ppid:2 flags:0x00004000
[Mon Mar 11 14:29:12 2024] Call Trace:
[Mon Mar 11 14:29:12 2024] <TASK>
[Mon Mar 11 14:29:12 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:12 2024] ? __pfx_nfsd+0x10/0x10 [nfsd]
[Mon Mar 11 14:29:12 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:12 2024] svc_get_next_xprt+0x10e/0x190 [sunrpc]
[Mon Mar 11 14:29:12 2024] svc_recv+0x57/0x150 [sunrpc]
[Mon Mar 11 14:29:12 2024] nfsd+0x84/0xb0 [nfsd]
[Mon Mar 11 14:29:12 2024] kthread+0xdd/0x100
[Mon Mar 11 14:29:12 2024] ? __pfx_kthread+0x10/0x10
[Mon Mar 11 14:29:12 2024] ret_from_fork+0x29/0x50
[Mon Mar 11 14:29:12 2024] </TASK>
[Mon Mar 11 14:29:12 2024] task:nfsd state:I stack:0 pid:8792 ppid:2 flags:0x00004000
[Mon Mar 11 14:29:12 2024] Call Trace:
[Mon Mar 11 14:29:12 2024] <TASK>
[Mon Mar 11 14:29:12 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:12 2024] ? __pfx_nfsd+0x10/0x10 [nfsd]
[Mon Mar 11 14:29:12 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:12 2024] svc_get_next_xprt+0x10e/0x190 [sunrpc]
[Mon Mar 11 14:29:12 2024] svc_recv+0x57/0x150 [sunrpc]
[Mon Mar 11 14:29:12 2024] nfsd+0x84/0xb0 [nfsd]
[Mon Mar 11 14:29:12 2024] kthread+0xdd/0x100
[Mon Mar 11 14:29:12 2024] ? __pfx_kthread+0x10/0x10
[Mon Mar 11 14:29:12 2024] ret_from_fork+0x29/0x50
[Mon Mar 11 14:29:12 2024] </TASK>
[Mon Mar 11 14:29:12 2024] task:nfsd state:I stack:0 pid:8793 ppid:2 flags:0x00004000
[Mon Mar 11 14:29:12 2024] Call Trace:
[Mon Mar 11 14:29:12 2024] <TASK>
[Mon Mar 11 14:29:12 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:12 2024] ? __pfx_nfsd+0x10/0x10 [nfsd]
[Mon Mar 11 14:29:12 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:12 2024] svc_get_next_xprt+0x10e/0x190 [sunrpc]
[Mon Mar 11 14:29:12 2024] svc_recv+0x57/0x150 [sunrpc]
[Mon Mar 11 14:29:12 2024] nfsd+0x84/0xb0 [nfsd]
[Mon Mar 11 14:29:12 2024] kthread+0xdd/0x100
[Mon Mar 11 14:29:12 2024] ? __pfx_kthread+0x10/0x10
[Mon Mar 11 14:29:12 2024] ret_from_fork+0x29/0x50
[Mon Mar 11 14:29:12 2024] </TASK>
[Mon Mar 11 14:29:12 2024] task:nfsd state:I stack:0 pid:8794 ppid:2 flags:0x00004000
[Mon Mar 11 14:29:12 2024] Call Trace:
[Mon Mar 11 14:29:12 2024] <TASK>
[Mon Mar 11 14:29:12 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:12 2024] ? __pfx_nfsd+0x10/0x10 [nfsd]
[Mon Mar 11 14:29:12 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:12 2024] svc_get_next_xprt+0x10e/0x190 [sunrpc]
[Mon Mar 11 14:29:12 2024] svc_recv+0x57/0x150 [sunrpc]
[Mon Mar 11 14:29:12 2024] nfsd+0x84/0xb0 [nfsd]
[Mon Mar 11 14:29:12 2024] kthread+0xdd/0x100
[Mon Mar 11 14:29:12 2024] ? __pfx_kthread+0x10/0x10
[Mon Mar 11 14:29:12 2024] ret_from_fork+0x29/0x50
[Mon Mar 11 14:29:12 2024] </TASK>
[Mon Mar 11 14:29:12 2024] task:nfsd state:I stack:0 pid:8795 ppid:2 flags:0x00004000
[Mon Mar 11 14:29:12 2024] Call Trace:
[Mon Mar 11 14:29:12 2024] <TASK>
[Mon Mar 11 14:29:12 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:12 2024] ? __pfx_nfsd+0x10/0x10 [nfsd]
[Mon Mar 11 14:29:12 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:12 2024] svc_get_next_xprt+0x10e/0x190 [sunrpc]
[Mon Mar 11 14:29:12 2024] svc_recv+0x57/0x150 [sunrpc]
[Mon Mar 11 14:29:12 2024] nfsd+0x84/0xb0 [nfsd]
[Mon Mar 11 14:29:12 2024] kthread+0xdd/0x100
[Mon Mar 11 14:29:12 2024] ? __pfx_kthread+0x10/0x10
[Mon Mar 11 14:29:12 2024] ret_from_fork+0x29/0x50
[Mon Mar 11 14:29:12 2024] </TASK>
[Mon Mar 11 14:29:12 2024] task:nfsd state:I stack:0 pid:8796 ppid:2 flags:0x00004000
[Mon Mar 11 14:29:12 2024] Call Trace:
[Mon Mar 11 14:29:12 2024] <TASK>
[Mon Mar 11 14:29:12 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:12 2024] ? __pfx_nfsd+0x10/0x10 [nfsd]
[Mon Mar 11 14:29:12 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:12 2024] svc_get_next_xprt+0x10e/0x190 [sunrpc]
[Mon Mar 11 14:29:12 2024] svc_recv+0x57/0x150 [sunrpc]
[Mon Mar 11 14:29:12 2024] nfsd+0x84/0xb0 [nfsd]
[Mon Mar 11 14:29:12 2024] kthread+0xdd/0x100
[Mon Mar 11 14:29:12 2024] ? __pfx_kthread+0x10/0x10
[Mon Mar 11 14:29:12 2024] ret_from_fork+0x29/0x50
[Mon Mar 11 14:29:12 2024] </TASK>
[Mon Mar 11 14:29:12 2024] task:nfsd state:I stack:0 pid:8797 ppid:2 flags:0x00004000
[Mon Mar 11 14:29:12 2024] Call Trace:
[Mon Mar 11 14:29:12 2024] <TASK>
[Mon Mar 11 14:29:12 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:12 2024] ? __pfx_nfsd+0x10/0x10 [nfsd]
[Mon Mar 11 14:29:12 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:12 2024] svc_get_next_xprt+0x10e/0x190 [sunrpc]
[Mon Mar 11 14:29:12 2024] svc_recv+0x57/0x150 [sunrpc]
[Mon Mar 11 14:29:12 2024] nfsd+0x84/0xb0 [nfsd]
[Mon Mar 11 14:29:12 2024] kthread+0xdd/0x100
[Mon Mar 11 14:29:12 2024] ? __pfx_kthread+0x10/0x10
[Mon Mar 11 14:29:12 2024] ret_from_fork+0x29/0x50
[Mon Mar 11 14:29:12 2024] </TASK>
[Mon Mar 11 14:29:12 2024] task:nfsd state:I stack:0 pid:8798 ppid:2 flags:0x00004000
[Mon Mar 11 14:29:12 2024] Call Trace:
[Mon Mar 11 14:29:12 2024] <TASK>
[Mon Mar 11 14:29:12 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:12 2024] ? __pfx_nfsd+0x10/0x10 [nfsd]
[Mon Mar 11 14:29:12 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:12 2024] svc_get_next_xprt+0x10e/0x190 [sunrpc]
[Mon Mar 11 14:29:12 2024] svc_recv+0x57/0x150 [sunrpc]
[Mon Mar 11 14:29:12 2024] nfsd+0x84/0xb0 [nfsd]
[Mon Mar 11 14:29:12 2024] kthread+0xdd/0x100
[Mon Mar 11 14:29:12 2024] ? __pfx_kthread+0x10/0x10
[Mon Mar 11 14:29:12 2024] ret_from_fork+0x29/0x50
[Mon Mar 11 14:29:12 2024] </TASK>
[Mon Mar 11 14:29:12 2024] task:nfsd state:I stack:0 pid:8799 ppid:2 flags:0x00004000
[Mon Mar 11 14:29:12 2024] Call Trace:
[Mon Mar 11 14:29:12 2024] <TASK>
[Mon Mar 11 14:29:12 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:12 2024] ? __pfx_nfsd+0x10/0x10 [nfsd]
[Mon Mar 11 14:29:12 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:12 2024] svc_get_next_xprt+0x10e/0x190 [sunrpc]
[Mon Mar 11 14:29:12 2024] svc_recv+0x57/0x150 [sunrpc]
[Mon Mar 11 14:29:12 2024] nfsd+0x84/0xb0 [nfsd]
[Mon Mar 11 14:29:12 2024] kthread+0xdd/0x100
[Mon Mar 11 14:29:12 2024] ? __pfx_kthread+0x10/0x10
[Mon Mar 11 14:29:12 2024] ret_from_fork+0x29/0x50
[Mon Mar 11 14:29:12 2024] </TASK>
[Mon Mar 11 14:29:12 2024] task:nfsd state:I stack:0 pid:8800 ppid:2 flags:0x00004000
[Mon Mar 11 14:29:12 2024] Call Trace:
[Mon Mar 11 14:29:12 2024] <TASK>
[Mon Mar 11 14:29:12 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:12 2024] ? __pfx_nfsd+0x10/0x10 [nfsd]
[Mon Mar 11 14:29:12 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:12 2024] svc_get_next_xprt+0x10e/0x190 [sunrpc]
[Mon Mar 11 14:29:12 2024] svc_recv+0x57/0x150 [sunrpc]
[Mon Mar 11 14:29:12 2024] nfsd+0x84/0xb0 [nfsd]
[Mon Mar 11 14:29:12 2024] kthread+0xdd/0x100
[Mon Mar 11 14:29:12 2024] ? __pfx_kthread+0x10/0x10
[Mon Mar 11 14:29:12 2024] ret_from_fork+0x29/0x50
[Mon Mar 11 14:29:12 2024] </TASK>
[Mon Mar 11 14:29:12 2024] task:nfsd state:I stack:0 pid:8801 ppid:2 flags:0x00004000
[Mon Mar 11 14:29:12 2024] Call Trace:
[Mon Mar 11 14:29:12 2024] <TASK>
[Mon Mar 11 14:29:12 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:12 2024] ? __pfx_nfsd+0x10/0x10 [nfsd]
[Mon Mar 11 14:29:12 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:12 2024] svc_get_next_xprt+0x10e/0x190 [sunrpc]
[Mon Mar 11 14:29:12 2024] svc_recv+0x57/0x150 [sunrpc]
[Mon Mar 11 14:29:12 2024] nfsd+0x84/0xb0 [nfsd]
[Mon Mar 11 14:29:12 2024] kthread+0xdd/0x100
[Mon Mar 11 14:29:12 2024] ? __pfx_kthread+0x10/0x10
[Mon Mar 11 14:29:12 2024] ret_from_fork+0x29/0x50
[Mon Mar 11 14:29:12 2024] </TASK>
[Mon Mar 11 14:29:12 2024] task:nfsd state:I stack:0 pid:8802 ppid:2 flags:0x00004000
[Mon Mar 11 14:29:12 2024] Call Trace:
[Mon Mar 11 14:29:12 2024] <TASK>
[Mon Mar 11 14:29:12 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:12 2024] ? __pfx_nfsd+0x10/0x10 [nfsd]
[Mon Mar 11 14:29:12 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:12 2024] svc_get_next_xprt+0x10e/0x190 [sunrpc]
[Mon Mar 11 14:29:12 2024] svc_recv+0x57/0x150 [sunrpc]
[Mon Mar 11 14:29:12 2024] nfsd+0x84/0xb0 [nfsd]
[Mon Mar 11 14:29:12 2024] kthread+0xdd/0x100
[Mon Mar 11 14:29:12 2024] ? __pfx_kthread+0x10/0x10
[Mon Mar 11 14:29:12 2024] ret_from_fork+0x29/0x50
[Mon Mar 11 14:29:12 2024] </TASK>
[Mon Mar 11 14:29:12 2024] task:nfsd state:I stack:0 pid:8803 ppid:2 flags:0x00004000
[Mon Mar 11 14:29:12 2024] Call Trace:
[Mon Mar 11 14:29:12 2024] <TASK>
[Mon Mar 11 14:29:12 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:12 2024] ? __pfx_nfsd+0x10/0x10 [nfsd]
[Mon Mar 11 14:29:12 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:12 2024] svc_get_next_xprt+0x10e/0x190 [sunrpc]
[Mon Mar 11 14:29:12 2024] svc_recv+0x57/0x150 [sunrpc]
[Mon Mar 11 14:29:12 2024] nfsd+0x84/0xb0 [nfsd]
[Mon Mar 11 14:29:12 2024] kthread+0xdd/0x100
[Mon Mar 11 14:29:12 2024] ? __pfx_kthread+0x10/0x10
[Mon Mar 11 14:29:12 2024] ret_from_fork+0x29/0x50
[Mon Mar 11 14:29:12 2024] </TASK>
[Mon Mar 11 14:29:12 2024] task:nfsd state:I stack:0 pid:8804 ppid:2 flags:0x00004000
[Mon Mar 11 14:29:12 2024] Call Trace:
[Mon Mar 11 14:29:12 2024] <TASK>
[Mon Mar 11 14:29:12 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:12 2024] ? __pfx_nfsd+0x10/0x10 [nfsd]
[Mon Mar 11 14:29:12 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:12 2024] svc_get_next_xprt+0x10e/0x190 [sunrpc]
[Mon Mar 11 14:29:12 2024] svc_recv+0x57/0x150 [sunrpc]
[Mon Mar 11 14:29:12 2024] nfsd+0x84/0xb0 [nfsd]
[Mon Mar 11 14:29:12 2024] kthread+0xdd/0x100
[Mon Mar 11 14:29:12 2024] ? __pfx_kthread+0x10/0x10
[Mon Mar 11 14:29:12 2024] ret_from_fork+0x29/0x50
[Mon Mar 11 14:29:12 2024] </TASK>
[Mon Mar 11 14:29:12 2024] task:nfsd state:I stack:0 pid:8805 ppid:2 flags:0x00004000
[Mon Mar 11 14:29:12 2024] Call Trace:
[Mon Mar 11 14:29:12 2024] <TASK>
[Mon Mar 11 14:29:12 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:12 2024] ? __pfx_nfsd+0x10/0x10 [nfsd]
[Mon Mar 11 14:29:12 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:12 2024] svc_get_next_xprt+0x10e/0x190 [sunrpc]
[Mon Mar 11 14:29:12 2024] svc_recv+0x57/0x150 [sunrpc]
[Mon Mar 11 14:29:12 2024] nfsd+0x84/0xb0 [nfsd]
[Mon Mar 11 14:29:12 2024] kthread+0xdd/0x100
[Mon Mar 11 14:29:12 2024] ? __pfx_kthread+0x10/0x10
[Mon Mar 11 14:29:12 2024] ret_from_fork+0x29/0x50
[Mon Mar 11 14:29:12 2024] </TASK>
[Mon Mar 11 14:29:12 2024] task:nfsd state:I stack:0 pid:8806 ppid:2 flags:0x00004000
[Mon Mar 11 14:29:12 2024] Call Trace:
[Mon Mar 11 14:29:12 2024] <TASK>
[Mon Mar 11 14:29:12 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:12 2024] ? __pfx_nfsd+0x10/0x10 [nfsd]
[Mon Mar 11 14:29:12 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:12 2024] svc_get_next_xprt+0x10e/0x190 [sunrpc]
[Mon Mar 11 14:29:12 2024] svc_recv+0x57/0x150 [sunrpc]
[Mon Mar 11 14:29:12 2024] nfsd+0x84/0xb0 [nfsd]
[Mon Mar 11 14:29:12 2024] kthread+0xdd/0x100
[Mon Mar 11 14:29:12 2024] ? __pfx_kthread+0x10/0x10
[Mon Mar 11 14:29:12 2024] ret_from_fork+0x29/0x50
[Mon Mar 11 14:29:12 2024] </TASK>
[Mon Mar 11 14:29:12 2024] task:nfsd state:I stack:0 pid:8807 ppid:2 flags:0x00004000
[Mon Mar 11 14:29:12 2024] Call Trace:
[Mon Mar 11 14:29:12 2024] <TASK>
[Mon Mar 11 14:29:12 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:12 2024] ? __pfx_nfsd+0x10/0x10 [nfsd]
[Mon Mar 11 14:29:12 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:12 2024] svc_get_next_xprt+0x10e/0x190 [sunrpc]
[Mon Mar 11 14:29:12 2024] svc_recv+0x57/0x150 [sunrpc]
[Mon Mar 11 14:29:12 2024] nfsd+0x84/0xb0 [nfsd]
[Mon Mar 11 14:29:12 2024] kthread+0xdd/0x100
[Mon Mar 11 14:29:12 2024] ? __pfx_kthread+0x10/0x10
[Mon Mar 11 14:29:12 2024] ret_from_fork+0x29/0x50
[Mon Mar 11 14:29:12 2024] </TASK>
[Mon Mar 11 14:29:12 2024] task:nfsd state:I stack:0 pid:8808 ppid:2 flags:0x00004000
[Mon Mar 11 14:29:12 2024] Call Trace:
[Mon Mar 11 14:29:12 2024] <TASK>
[Mon Mar 11 14:29:12 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:12 2024] ? __pfx_nfsd+0x10/0x10 [nfsd]
[Mon Mar 11 14:29:12 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:12 2024] svc_get_next_xprt+0x10e/0x190 [sunrpc]
[Mon Mar 11 14:29:12 2024] svc_recv+0x57/0x150 [sunrpc]
[Mon Mar 11 14:29:12 2024] nfsd+0x84/0xb0 [nfsd]
[Mon Mar 11 14:29:12 2024] kthread+0xdd/0x100
[Mon Mar 11 14:29:12 2024] ? __pfx_kthread+0x10/0x10
[Mon Mar 11 14:29:12 2024] ret_from_fork+0x29/0x50
[Mon Mar 11 14:29:12 2024] </TASK>
[Mon Mar 11 14:29:12 2024] task:nfsd state:I stack:0 pid:8809 ppid:2 flags:0x00004000
[Mon Mar 11 14:29:12 2024] Call Trace:
[Mon Mar 11 14:29:12 2024] <TASK>
[Mon Mar 11 14:29:12 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:12 2024] ? __pfx_nfsd+0x10/0x10 [nfsd]
[Mon Mar 11 14:29:12 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:12 2024] svc_get_next_xprt+0x10e/0x190 [sunrpc]
[Mon Mar 11 14:29:12 2024] svc_recv+0x57/0x150 [sunrpc]
[Mon Mar 11 14:29:12 2024] nfsd+0x84/0xb0 [nfsd]
[Mon Mar 11 14:29:12 2024] kthread+0xdd/0x100
[Mon Mar 11 14:29:12 2024] ? __pfx_kthread+0x10/0x10
[Mon Mar 11 14:29:12 2024] ret_from_fork+0x29/0x50
[Mon Mar 11 14:29:12 2024] </TASK>
[Mon Mar 11 14:29:12 2024] task:nfsd state:I stack:0 pid:8810 ppid:2 flags:0x00004000
[Mon Mar 11 14:29:12 2024] Call Trace:
[Mon Mar 11 14:29:12 2024] <TASK>
[Mon Mar 11 14:29:12 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:12 2024] ? __pfx_nfsd+0x10/0x10 [nfsd]
[Mon Mar 11 14:29:12 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:12 2024] svc_get_next_xprt+0x10e/0x190 [sunrpc]
[Mon Mar 11 14:29:12 2024] svc_recv+0x57/0x150 [sunrpc]
[Mon Mar 11 14:29:12 2024] nfsd+0x84/0xb0 [nfsd]
[Mon Mar 11 14:29:12 2024] kthread+0xdd/0x100
[Mon Mar 11 14:29:12 2024] ? __pfx_kthread+0x10/0x10
[Mon Mar 11 14:29:12 2024] ret_from_fork+0x29/0x50
[Mon Mar 11 14:29:12 2024] </TASK>
[Mon Mar 11 14:29:12 2024] task:nfsd state:I stack:0 pid:8811 ppid:2 flags:0x00004000
[Mon Mar 11 14:29:12 2024] Call Trace:
[Mon Mar 11 14:29:12 2024] <TASK>
[Mon Mar 11 14:29:12 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:12 2024] ? __pfx_nfsd+0x10/0x10 [nfsd]
[Mon Mar 11 14:29:12 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:12 2024] svc_get_next_xprt+0x10e/0x190 [sunrpc]
[Mon Mar 11 14:29:12 2024] svc_recv+0x57/0x150 [sunrpc]
[Mon Mar 11 14:29:12 2024] nfsd+0x84/0xb0 [nfsd]
[Mon Mar 11 14:29:12 2024] kthread+0xdd/0x100
[Mon Mar 11 14:29:12 2024] ? __pfx_kthread+0x10/0x10
[Mon Mar 11 14:29:12 2024] ret_from_fork+0x29/0x50
[Mon Mar 11 14:29:12 2024] </TASK>
[Mon Mar 11 14:29:12 2024] task:nfsd state:I stack:0 pid:8812 ppid:2 flags:0x00004000
[Mon Mar 11 14:29:12 2024] Call Trace:
[Mon Mar 11 14:29:12 2024] <TASK>
[Mon Mar 11 14:29:12 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:12 2024] ? __pfx_nfsd+0x10/0x10 [nfsd]
[Mon Mar 11 14:29:12 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:12 2024] svc_get_next_xprt+0x10e/0x190 [sunrpc]
[Mon Mar 11 14:29:12 2024] ? __pfx_nfsd+0x10/0x10 [nfsd]
[Mon Mar 11 14:29:12 2024] svc_recv+0x57/0x150 [sunrpc]
[Mon Mar 11 14:29:12 2024] nfsd+0x84/0xb0 [nfsd]
[Mon Mar 11 14:29:12 2024] kthread+0xdd/0x100
[Mon Mar 11 14:29:12 2024] ? __pfx_kthread+0x10/0x10
[Mon Mar 11 14:29:12 2024] ret_from_fork+0x29/0x50
[Mon Mar 11 14:29:12 2024] </TASK>
[Mon Mar 11 14:29:12 2024] task:nfsd state:I stack:0 pid:8813 ppid:2 flags:0x00004000
[Mon Mar 11 14:29:12 2024] Call Trace:
[Mon Mar 11 14:29:12 2024] <TASK>
[Mon Mar 11 14:29:12 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:12 2024] ? __pfx_nfsd+0x10/0x10 [nfsd]
[Mon Mar 11 14:29:12 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:12 2024] svc_get_next_xprt+0x10e/0x190 [sunrpc]
[Mon Mar 11 14:29:12 2024] svc_recv+0x57/0x150 [sunrpc]
[Mon Mar 11 14:29:12 2024] nfsd+0x84/0xb0 [nfsd]
[Mon Mar 11 14:29:12 2024] kthread+0xdd/0x100
[Mon Mar 11 14:29:12 2024] ? __pfx_kthread+0x10/0x10
[Mon Mar 11 14:29:12 2024] ret_from_fork+0x29/0x50
[Mon Mar 11 14:29:12 2024] </TASK>
[Mon Mar 11 14:29:12 2024] task:nfsd state:I stack:0 pid:8814 ppid:2 flags:0x00004000
[Mon Mar 11 14:29:12 2024] Call Trace:
[Mon Mar 11 14:29:12 2024] <TASK>
[Mon Mar 11 14:29:12 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:12 2024] ? __pfx_nfsd+0x10/0x10 [nfsd]
[Mon Mar 11 14:29:12 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:12 2024] svc_get_next_xprt+0x10e/0x190 [sunrpc]
[Mon Mar 11 14:29:12 2024] svc_recv+0x57/0x150 [sunrpc]
[Mon Mar 11 14:29:12 2024] nfsd+0x84/0xb0 [nfsd]
[Mon Mar 11 14:29:12 2024] kthread+0xdd/0x100
[Mon Mar 11 14:29:12 2024] ? __pfx_kthread+0x10/0x10
[Mon Mar 11 14:29:12 2024] ret_from_fork+0x29/0x50
[Mon Mar 11 14:29:12 2024] </TASK>
[Mon Mar 11 14:29:12 2024] task:nfsd state:I stack:0 pid:8815 ppid:2 flags:0x00004000
[Mon Mar 11 14:29:12 2024] Call Trace:
[Mon Mar 11 14:29:12 2024] <TASK>
[Mon Mar 11 14:29:12 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:12 2024] ? __pfx_nfsd+0x10/0x10 [nfsd]
[Mon Mar 11 14:29:12 2024] ? __pfx_nfsd+0x10/0x10 [nfsd]
[Mon Mar 11 14:29:12 2024] ? __pfx_nfsd+0x10/0x10 [nfsd]
[Mon Mar 11 14:29:12 2024] ? nfsd+0x84/0xb0 [nfsd]
[Mon Mar 11 14:29:12 2024] ? kthread+0xdd/0x100
[Mon Mar 11 14:29:12 2024] ? __pfx_kthread+0x10/0x10
[Mon Mar 11 14:29:12 2024] ? ret_from_fork+0x29/0x50
[Mon Mar 11 14:29:12 2024] </TASK>
[Mon Mar 11 14:29:12 2024] task:nfsd state:D stack:0 pid:8816 ppid:2 flags:0x00004000
[Mon Mar 11 14:29:12 2024] Call Trace:
[Mon Mar 11 14:29:12 2024] <TASK>
[Mon Mar 11 14:29:12 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:12 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:12 2024] io_schedule+0x42/0x70
[Mon Mar 11 14:29:12 2024] bit_wait_io+0xd/0x60
[Mon Mar 11 14:29:12 2024] __wait_on_bit+0x48/0x150
[Mon Mar 11 14:29:12 2024] ? __pfx_bit_wait_io+0x10/0x10
[Mon Mar 11 14:29:12 2024] out_of_line_wait_on_bit+0x92/0xb0
[Mon Mar 11 14:29:12 2024] ? __pfx_wake_bit_function+0x10/0x10
[Mon Mar 11 14:29:12 2024] ext4_read_bh+0x84/0x90 [ext4]
[Mon Mar 11 14:29:12 2024] ext4_bread+0x4d/0x70 [ext4]
[Mon Mar 11 14:29:12 2024] __ext4_read_dirblock.part.0+0x2f/0x310 [ext4]
[Mon Mar 11 14:29:12 2024] ext4_dx_find_entry+0x74/0x210 [ext4]
[Mon Mar 11 14:29:12 2024] __ext4_find_entry+0x3a6/0x440 [ext4]
[Mon Mar 11 14:29:12 2024] ? d_alloc+0x83/0xa0
[Mon Mar 11 14:29:12 2024] ext4_lookup.part.0+0x58/0x1c0 [ext4]
[Mon Mar 11 14:29:12 2024] __lookup_slow+0x81/0x130
[Mon Mar 11 14:29:12 2024] lookup_one_len_unlocked+0x90/0xb0
[Mon Mar 11 14:29:12 2024] ? security_inode_permission+0x2d/0x50
[Mon Mar 11 14:29:12 2024] nfsd_lookup_dentry+0x95/0x2a0 [nfsd]
[Mon Mar 11 14:29:12 2024] nfsd_lookup+0x8b/0x150 [nfsd]
[Mon Mar 11 14:29:12 2024] ? nfsd4_encode_operation+0xa3/0x2b0 [nfsd]
[Mon Mar 11 14:29:12 2024] nfsd4_proc_compound+0x44b/0x700 [nfsd]
[Mon Mar 11 14:29:12 2024] nfsd_dispatch+0x94/0x1c0 [nfsd]
[Mon Mar 11 14:29:12 2024] svc_process_common+0x2ec/0x660 [sunrpc]
[Mon Mar 11 14:29:12 2024] ? __pfx_nfsd_dispatch+0x10/0x10 [nfsd]
[Mon Mar 11 14:29:12 2024] ? __pfx_nfsd+0x10/0x10 [nfsd]
[Mon Mar 11 14:29:12 2024] svc_process+0x12d/0x170 [sunrpc]
[Mon Mar 11 14:29:12 2024] nfsd+0x84/0xb0 [nfsd]
[Mon Mar 11 14:29:12 2024] kthread+0xdd/0x100
[Mon Mar 11 14:29:12 2024] ? __pfx_kthread+0x10/0x10
[Mon Mar 11 14:29:12 2024] ret_from_fork+0x29/0x50
[Mon Mar 11 14:29:12 2024] </TASK>
[Mon Mar 11 14:29:12 2024] task:nfsd state:D stack:0 pid:8817 ppid:2 flags:0x00004000
[Mon Mar 11 14:29:12 2024] Call Trace:
[Mon Mar 11 14:29:12 2024] <TASK>
[Mon Mar 11 14:29:12 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:12 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:12 2024] io_schedule+0x42/0x70
[Mon Mar 11 14:29:12 2024] folio_wait_bit+0xe9/0x200
[Mon Mar 11 14:29:12 2024] ? xas_load+0x9/0xa0
[Mon Mar 11 14:29:12 2024] ? __pfx_wake_page_function+0x10/0x10
[Mon Mar 11 14:29:12 2024] folio_wait_writeback+0x28/0x80
[Mon Mar 11 14:29:12 2024] truncate_inode_partial_folio+0x5a/0x130
[Mon Mar 11 14:29:12 2024] truncate_inode_pages_range+0x1ba/0x440
[Mon Mar 11 14:29:12 2024] ? __ext4_handle_dirty_metadata+0x58/0x180 [ext4]
[Mon Mar 11 14:29:12 2024] ? ext4_do_update_inode.isra.0+0x17a/0x3f0 [ext4]
[Mon Mar 11 14:29:12 2024] ? __wake_up+0x40/0x60
[Mon Mar 11 14:29:12 2024] ? kmem_cache_free+0x15/0x360
[Mon Mar 11 14:29:12 2024] ? jbd2_journal_stop+0x152/0x2e0 [jbd2]
[Mon Mar 11 14:29:12 2024] ? down_read+0xe/0xa0
[Mon Mar 11 14:29:12 2024] ? unmap_mapping_range+0x7e/0x140
[Mon Mar 11 14:29:12 2024] truncate_pagecache+0x44/0x60
[Mon Mar 11 14:29:12 2024] ext4_setattr+0x3df/0x980 [ext4]
[Mon Mar 11 14:29:12 2024] notify_change+0x3c5/0x550
[Mon Mar 11 14:29:12 2024] ? nfsd_setuser+0x116/0x270 [nfsd]
[Mon Mar 11 14:29:12 2024] ? __nfsd_setattr+0x5b/0xe0 [nfsd]
[Mon Mar 11 14:29:12 2024] __nfsd_setattr+0x5b/0xe0 [nfsd]
[Mon Mar 11 14:29:12 2024] nfsd_setattr+0x20f/0x480 [nfsd]
[Mon Mar 11 14:29:12 2024] nfsd4_setattr+0x16f/0x250 [nfsd]
[Mon Mar 11 14:29:12 2024] nfsd4_proc_compound+0x44b/0x700 [nfsd]
[Mon Mar 11 14:29:12 2024] nfsd_dispatch+0x94/0x1c0 [nfsd]
[Mon Mar 11 14:29:12 2024] svc_process_common+0x2ec/0x660 [sunrpc]
[Mon Mar 11 14:29:12 2024] ? __pfx_nfsd_dispatch+0x10/0x10 [nfsd]
[Mon Mar 11 14:29:12 2024] ? __pfx_nfsd+0x10/0x10 [nfsd]
[Mon Mar 11 14:29:12 2024] svc_process+0x12d/0x170 [sunrpc]
[Mon Mar 11 14:29:12 2024] nfsd+0x84/0xb0 [nfsd]
[Mon Mar 11 14:29:12 2024] kthread+0xdd/0x100
[Mon Mar 11 14:29:12 2024] ? __pfx_kthread+0x10/0x10
[Mon Mar 11 14:29:12 2024] ret_from_fork+0x29/0x50
[Mon Mar 11 14:29:12 2024] </TASK>
[Mon Mar 11 14:29:12 2024] task:nfsd state:D stack:0 pid:8818 ppid:2 flags:0x00004000
[Mon Mar 11 14:29:12 2024] Call Trace:
[Mon Mar 11 14:29:12 2024] <TASK>
[Mon Mar 11 14:29:12 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:12 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:12 2024] schedule_timeout+0x11f/0x160
[Mon Mar 11 14:29:12 2024] ? select_idle_sibling+0x344/0x430
[Mon Mar 11 14:29:12 2024] ? sched_clock+0xc/0x30
[Mon Mar 11 14:29:12 2024] __wait_for_common+0x90/0x1d0
[Mon Mar 11 14:29:12 2024] ? __pfx_schedule_timeout+0x10/0x10
[Mon Mar 11 14:29:12 2024] __flush_workqueue+0x13a/0x3f0
[Mon Mar 11 14:29:12 2024] nfsd4_shutdown_callback+0x49/0x120 [nfsd]
[Mon Mar 11 14:29:12 2024] ? nfsd4_cld_remove+0x54/0x1d0 [nfsd]
[Mon Mar 11 14:29:12 2024] ? nfsd4_return_all_client_layouts+0xc4/0xf0 [nfsd]
[Mon Mar 11 14:29:12 2024] ? nfsd4_shutdown_copy+0x68/0xc0 [nfsd]
[Mon Mar 11 14:29:12 2024] __destroy_client+0x1f3/0x290 [nfsd]
[Mon Mar 11 14:29:12 2024] nfsd4_exchange_id+0x378/0x770 [nfsd]
[Mon Mar 11 14:29:12 2024] ? nfsd4_decode_opaque+0x3a/0x90 [nfsd]
[Mon Mar 11 14:29:12 2024] nfsd4_proc_compound+0x44b/0x700 [nfsd]
[Mon Mar 11 14:29:12 2024] nfsd_dispatch+0x94/0x1c0 [nfsd]
[Mon Mar 11 14:29:12 2024] svc_process_common+0x2ec/0x660 [sunrpc]
[Mon Mar 11 14:29:12 2024] ? __pfx_nfsd_dispatch+0x10/0x10 [nfsd]
[Mon Mar 11 14:29:12 2024] ? __pfx_nfsd+0x10/0x10 [nfsd]
[Mon Mar 11 14:29:12 2024] svc_process+0x12d/0x170 [sunrpc]
[Mon Mar 11 14:29:12 2024] nfsd+0x84/0xb0 [nfsd]
[Mon Mar 11 14:29:12 2024] kthread+0xdd/0x100
[Mon Mar 11 14:29:12 2024] ? __pfx_kthread+0x10/0x10
[Mon Mar 11 14:29:12 2024] ret_from_fork+0x29/0x50
[Mon Mar 11 14:29:12 2024] </TASK>
[Mon Mar 11 14:29:12 2024] task:nfsd state:D stack:0 pid:8819 ppid:2 flags:0x00004000
[Mon Mar 11 14:29:12 2024] Call Trace:
[Mon Mar 11 14:29:12 2024] <TASK>
[Mon Mar 11 14:29:12 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:12 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:12 2024] io_schedule+0x42/0x70
[Mon Mar 11 14:29:12 2024] folio_wait_bit+0xe9/0x200
[Mon Mar 11 14:29:12 2024] ? xas_load+0x9/0xa0
[Mon Mar 11 14:29:12 2024] ? __pfx_wake_page_function+0x10/0x10
[Mon Mar 11 14:29:12 2024] folio_wait_writeback+0x28/0x80
[Mon Mar 11 14:29:12 2024] truncate_inode_partial_folio+0x5a/0x130
[Mon Mar 11 14:29:12 2024] truncate_inode_pages_range+0x1ba/0x440
[Mon Mar 11 14:29:12 2024] ? __ext4_handle_dirty_metadata+0x58/0x180 [ext4]
[Mon Mar 11 14:29:12 2024] ? ext4_do_update_inode.isra.0+0x17a/0x3f0 [ext4]
[Mon Mar 11 14:29:12 2024] ? __wake_up+0x40/0x60
[Mon Mar 11 14:29:12 2024] ? kmem_cache_free+0x15/0x360
[Mon Mar 11 14:29:12 2024] ? jbd2_journal_stop+0x152/0x2e0 [jbd2]
[Mon Mar 11 14:29:12 2024] ? down_read+0xe/0xa0
[Mon Mar 11 14:29:12 2024] ? unmap_mapping_range+0x7e/0x140
[Mon Mar 11 14:29:12 2024] truncate_pagecache+0x44/0x60
[Mon Mar 11 14:29:12 2024] ext4_setattr+0x3df/0x980 [ext4]
[Mon Mar 11 14:29:12 2024] notify_change+0x3c5/0x550
[Mon Mar 11 14:29:12 2024] ? nfsd_setuser+0x116/0x270 [nfsd]
[Mon Mar 11 14:29:12 2024] ? __nfsd_setattr+0x5b/0xe0 [nfsd]
[Mon Mar 11 14:29:12 2024] __nfsd_setattr+0x5b/0xe0 [nfsd]
[Mon Mar 11 14:29:12 2024] nfsd_setattr+0x20f/0x480 [nfsd]
[Mon Mar 11 14:29:12 2024] nfsd4_setattr+0x16f/0x250 [nfsd]
[Mon Mar 11 14:29:12 2024] nfsd4_proc_compound+0x44b/0x700 [nfsd]
[Mon Mar 11 14:29:12 2024] nfsd_dispatch+0x94/0x1c0 [nfsd]
[Mon Mar 11 14:29:12 2024] svc_process_common+0x2ec/0x660 [sunrpc]
[Mon Mar 11 14:29:12 2024] ? __pfx_nfsd_dispatch+0x10/0x10 [nfsd]
[Mon Mar 11 14:29:12 2024] ? __pfx_nfsd+0x10/0x10 [nfsd]
[Mon Mar 11 14:29:12 2024] svc_process+0x12d/0x170 [sunrpc]
[Mon Mar 11 14:29:12 2024] nfsd+0x84/0xb0 [nfsd]
[Mon Mar 11 14:29:12 2024] kthread+0xdd/0x100
[Mon Mar 11 14:29:12 2024] ? __pfx_kthread+0x10/0x10
[Mon Mar 11 14:29:12 2024] ret_from_fork+0x29/0x50
[Mon Mar 11 14:29:12 2024] </TASK>
[Mon Mar 11 14:29:12 2024] task:nfsd state:D stack:0 pid:8820 ppid:2 flags:0x00004000
[Mon Mar 11 14:29:12 2024] Call Trace:
[Mon Mar 11 14:29:12 2024] <TASK>
[Mon Mar 11 14:29:12 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:12 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:12 2024] schedule_timeout+0x11f/0x160
[Mon Mar 11 14:29:12 2024] ? select_idle_sibling+0x28/0x430
[Mon Mar 11 14:29:12 2024] __wait_for_common+0x90/0x1d0
[Mon Mar 11 14:29:12 2024] ? __pfx_schedule_timeout+0x10/0x10
[Mon Mar 11 14:29:12 2024] __flush_workqueue+0x13a/0x3f0
[Mon Mar 11 14:29:12 2024] nfsd4_shutdown_callback+0x49/0x120 [nfsd]
[Mon Mar 11 14:29:12 2024] ? nfsd4_cld_remove+0x54/0x1d0 [nfsd]
[Mon Mar 11 14:29:12 2024] ? nfsd4_return_all_client_layouts+0xc4/0xf0 [nfsd]
[Mon Mar 11 14:29:12 2024] ? nfsd4_shutdown_copy+0x68/0xc0 [nfsd]
[Mon Mar 11 14:29:12 2024] __destroy_client+0x1f3/0x290 [nfsd]
[Mon Mar 11 14:29:12 2024] nfsd4_exchange_id+0x378/0x770 [nfsd]
[Mon Mar 11 14:29:12 2024] ? nfsd4_decode_opaque+0x3a/0x90 [nfsd]
[Mon Mar 11 14:29:12 2024] nfsd4_proc_compound+0x44b/0x700 [nfsd]
[Mon Mar 11 14:29:12 2024] nfsd_dispatch+0x94/0x1c0 [nfsd]
[Mon Mar 11 14:29:12 2024] svc_process_common+0x2ec/0x660 [sunrpc]
[Mon Mar 11 14:29:12 2024] ? __pfx_nfsd_dispatch+0x10/0x10 [nfsd]
[Mon Mar 11 14:29:12 2024] ? __pfx_nfsd+0x10/0x10 [nfsd]
[Mon Mar 11 14:29:12 2024] svc_process+0x12d/0x170 [sunrpc]
[Mon Mar 11 14:29:12 2024] nfsd+0x84/0xb0 [nfsd]
[Mon Mar 11 14:29:12 2024] kthread+0xdd/0x100
[Mon Mar 11 14:29:12 2024] ? __pfx_kthread+0x10/0x10
[Mon Mar 11 14:29:12 2024] ret_from_fork+0x29/0x50
[Mon Mar 11 14:29:12 2024] </TASK>
[Mon Mar 11 14:29:12 2024] task:nfsd state:D stack:0 pid:8821 ppid:2 flags:0x00004000
[Mon Mar 11 14:29:12 2024] Call Trace:
[Mon Mar 11 14:29:12 2024] <TASK>
[Mon Mar 11 14:29:12 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:12 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:12 2024] schedule_timeout+0x11f/0x160
[Mon Mar 11 14:29:12 2024] ? select_idle_sibling+0x28/0x430
[Mon Mar 11 14:29:12 2024] ? tcp_recvmsg+0x196/0x210
[Mon Mar 11 14:29:12 2024] ? wake_affine+0x62/0x1f0
[Mon Mar 11 14:29:12 2024] __wait_for_common+0x90/0x1d0
[Mon Mar 11 14:29:12 2024] ? __pfx_schedule_timeout+0x10/0x10
[Mon Mar 11 14:29:12 2024] __flush_workqueue+0x13a/0x3f0
[Mon Mar 11 14:29:12 2024] nfsd4_destroy_session+0x1a4/0x240 [nfsd]
[Mon Mar 11 14:29:12 2024] nfsd4_proc_compound+0x44b/0x700 [nfsd]
[Mon Mar 11 14:29:12 2024] nfsd_dispatch+0x94/0x1c0 [nfsd]
[Mon Mar 11 14:29:12 2024] svc_process_common+0x2ec/0x660 [sunrpc]
[Mon Mar 11 14:29:12 2024] ? __pfx_nfsd_dispatch+0x10/0x10 [nfsd]
[Mon Mar 11 14:29:12 2024] ? __pfx_nfsd+0x10/0x10 [nfsd]
[Mon Mar 11 14:29:12 2024] svc_process+0x12d/0x170 [sunrpc]
[Mon Mar 11 14:29:12 2024] nfsd+0x84/0xb0 [nfsd]
[Mon Mar 11 14:29:12 2024] kthread+0xdd/0x100
[Mon Mar 11 14:29:12 2024] ? __pfx_kthread+0x10/0x10
[Mon Mar 11 14:29:12 2024] ret_from_fork+0x29/0x50
[Mon Mar 11 14:29:12 2024] </TASK>
[Mon Mar 11 14:29:12 2024] task:nfsd state:D stack:0 pid:8822 ppid:2 flags:0x00004000
[Mon Mar 11 14:29:12 2024] Call Trace:
[Mon Mar 11 14:29:12 2024] <TASK>
[Mon Mar 11 14:29:12 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:12 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:12 2024] schedule_timeout+0x11f/0x160
[Mon Mar 11 14:29:12 2024] ? select_idle_sibling+0x28/0x430
[Mon Mar 11 14:29:12 2024] ? tcp_recvmsg+0x196/0x210
[Mon Mar 11 14:29:12 2024] ? wake_affine+0x62/0x1f0
[Mon Mar 11 14:29:12 2024] __wait_for_common+0x90/0x1d0
[Mon Mar 11 14:29:12 2024] ? __pfx_schedule_timeout+0x10/0x10
[Mon Mar 11 14:29:12 2024] __flush_workqueue+0x13a/0x3f0
[Mon Mar 11 14:29:12 2024] nfsd4_destroy_session+0x1a4/0x240 [nfsd]
[Mon Mar 11 14:29:12 2024] nfsd4_proc_compound+0x44b/0x700 [nfsd]
[Mon Mar 11 14:29:12 2024] nfsd_dispatch+0x94/0x1c0 [nfsd]
[Mon Mar 11 14:29:12 2024] svc_process_common+0x2ec/0x660 [sunrpc]
[Mon Mar 11 14:29:12 2024] ? __pfx_nfsd_dispatch+0x10/0x10 [nfsd]
[Mon Mar 11 14:29:12 2024] ? __pfx_nfsd+0x10/0x10 [nfsd]
[Mon Mar 11 14:29:12 2024] svc_process+0x12d/0x170 [sunrpc]
[Mon Mar 11 14:29:12 2024] nfsd+0x84/0xb0 [nfsd]
[Mon Mar 11 14:29:12 2024] kthread+0xdd/0x100
[Mon Mar 11 14:29:12 2024] ? __pfx_kthread+0x10/0x10
[Mon Mar 11 14:29:12 2024] ret_from_fork+0x29/0x50
[Mon Mar 11 14:29:12 2024] </TASK>
[Mon Mar 11 14:29:12 2024] task:nfsd state:R running task stack:0 pid:8823 ppid:2 flags:0x00004000
[Mon Mar 11 14:29:12 2024] Call Trace:
[Mon Mar 11 14:29:12 2024] <TASK>
[Mon Mar 11 14:29:12 2024] ? free_unref_page+0x118/0x1e0
[Mon Mar 11 14:29:12 2024] ? __folio_put+0x27/0x60
[Mon Mar 11 14:29:12 2024] ? svc_tcp_recvfrom+0xed/0x4a0 [sunrpc]
[Mon Mar 11 14:29:12 2024] ? __pfx_nfsd+0x10/0x10 [nfsd]
[Mon Mar 11 14:29:12 2024] ? __pfx_nfsd+0x10/0x10 [nfsd]
[Mon Mar 11 14:29:12 2024] ? __pfx_nfsd+0x10/0x10 [nfsd]
[Mon Mar 11 14:29:12 2024] ? svc_recv+0x57/0x150 [sunrpc]
[Mon Mar 11 14:29:12 2024] ? nfsd+0x84/0xb0 [nfsd]
[Mon Mar 11 14:29:12 2024] ? kthread+0xdd/0x100
[Mon Mar 11 14:29:12 2024] ? __pfx_kthread+0x10/0x10
[Mon Mar 11 14:29:12 2024] ? ret_from_fork+0x29/0x50
[Mon Mar 11 14:29:12 2024] </TASK>
[Mon Mar 11 14:29:12 2024] task:nfsd state:D stack:0 pid:8824 ppid:2 flags:0x00004000
[Mon Mar 11 14:29:12 2024] Call Trace:
[Mon Mar 11 14:29:12 2024] <TASK>
[Mon Mar 11 14:29:12 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:12 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:12 2024] io_schedule+0x42/0x70
[Mon Mar 11 14:29:12 2024] filemap_update_page+0x3b0/0x500
[Mon Mar 11 14:29:12 2024] ? __pfx_wake_page_function+0x10/0x10
[Mon Mar 11 14:29:12 2024] filemap_get_pages+0x210/0x350
[Mon Mar 11 14:29:12 2024] ? update_sg_lb_stats+0x7e/0x450
[Mon Mar 11 14:29:12 2024] filemap_read+0xb9/0x310
[Mon Mar 11 14:29:12 2024] ? __fsnotify_parent+0xff/0x300
[Mon Mar 11 14:29:12 2024] ? __fsnotify_parent+0x10f/0x300
[Mon Mar 11 14:29:12 2024] generic_file_splice_read+0xd7/0x1b0
[Mon Mar 11 14:29:12 2024] splice_direct_to_actor+0xb0/0x210
[Mon Mar 11 14:29:12 2024] ? selinux_inode_permission+0x10e/0x1d0
[Mon Mar 11 14:29:12 2024] ? __pfx_nfsd_direct_splice_actor+0x10/0x10 [nfsd]
[Mon Mar 11 14:29:12 2024] nfsd_splice_read+0x67/0x100 [nfsd]
[Mon Mar 11 14:29:12 2024] nfsd4_encode_splice_read+0x58/0x100 [nfsd]
[Mon Mar 11 14:29:12 2024] nfsd4_encode_read+0x107/0x160 [nfsd]
[Mon Mar 11 14:29:12 2024] nfsd4_encode_operation+0xa3/0x2b0 [nfsd]
[Mon Mar 11 14:29:12 2024] nfsd4_proc_compound+0x1d0/0x700 [nfsd]
[Mon Mar 11 14:29:12 2024] nfsd_dispatch+0x94/0x1c0 [nfsd]
[Mon Mar 11 14:29:12 2024] svc_process_common+0x2ec/0x660 [sunrpc]
[Mon Mar 11 14:29:12 2024] ? __pfx_nfsd_dispatch+0x10/0x10 [nfsd]
[Mon Mar 11 14:29:12 2024] ? __pfx_nfsd+0x10/0x10 [nfsd]
[Mon Mar 11 14:29:12 2024] svc_process+0x12d/0x170 [sunrpc]
[Mon Mar 11 14:29:12 2024] nfsd+0x84/0xb0 [nfsd]
[Mon Mar 11 14:29:12 2024] kthread+0xdd/0x100
[Mon Mar 11 14:29:12 2024] ? __pfx_kthread+0x10/0x10
[Mon Mar 11 14:29:12 2024] ret_from_fork+0x29/0x50
[Mon Mar 11 14:29:12 2024] </TASK>
[Mon Mar 11 14:29:12 2024] task:nfsd state:D stack:0 pid:8825 ppid:2 flags:0x00004000
[Mon Mar 11 14:29:12 2024] Call Trace:
[Mon Mar 11 14:29:12 2024] <TASK>
[Mon Mar 11 14:29:12 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:12 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:12 2024] schedule_timeout+0x11f/0x160
[Mon Mar 11 14:29:12 2024] ? select_idle_sibling+0x28/0x430
[Mon Mar 11 14:29:12 2024] ? wake_affine+0x62/0x1f0
[Mon Mar 11 14:29:12 2024] __wait_for_common+0x90/0x1d0
[Mon Mar 11 14:29:12 2024] ? __pfx_schedule_timeout+0x10/0x10
[Mon Mar 11 14:29:12 2024] __flush_workqueue+0x13a/0x3f0
[Mon Mar 11 14:29:12 2024] nfsd4_shutdown_callback+0x49/0x120 [nfsd]
[Mon Mar 11 14:29:12 2024] ? nfsd4_cld_remove+0x54/0x1d0 [nfsd]
[Mon Mar 11 14:29:12 2024] ? nfsd4_return_all_client_layouts+0xc4/0xf0 [nfsd]
[Mon Mar 11 14:29:12 2024] ? nfsd4_shutdown_copy+0x68/0xc0 [nfsd]
[Mon Mar 11 14:29:12 2024] __destroy_client+0x1f3/0x290 [nfsd]
[Mon Mar 11 14:29:12 2024] nfsd4_exchange_id+0x378/0x770 [nfsd]
[Mon Mar 11 14:29:12 2024] ? nfsd4_decode_opaque+0x3a/0x90 [nfsd]
[Mon Mar 11 14:29:12 2024] nfsd4_proc_compound+0x44b/0x700 [nfsd]
[Mon Mar 11 14:29:12 2024] nfsd_dispatch+0x94/0x1c0 [nfsd]
[Mon Mar 11 14:29:12 2024] svc_process_common+0x2ec/0x660 [sunrpc]
[Mon Mar 11 14:29:12 2024] ? __pfx_nfsd_dispatch+0x10/0x10 [nfsd]
[Mon Mar 11 14:29:12 2024] ? __pfx_nfsd+0x10/0x10 [nfsd]
[Mon Mar 11 14:29:12 2024] svc_process+0x12d/0x170 [sunrpc]
[Mon Mar 11 14:29:12 2024] nfsd+0x84/0xb0 [nfsd]
[Mon Mar 11 14:29:12 2024] kthread+0xdd/0x100
[Mon Mar 11 14:29:12 2024] ? __pfx_kthread+0x10/0x10
[Mon Mar 11 14:29:12 2024] ret_from_fork+0x29/0x50
[Mon Mar 11 14:29:12 2024] </TASK>
[Mon Mar 11 14:29:12 2024] task:nfsd state:D stack:0 pid:8826 ppid:2 flags:0x00004000
[Mon Mar 11 14:29:12 2024] Call Trace:
[Mon Mar 11 14:29:12 2024] <TASK>
[Mon Mar 11 14:29:12 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:12 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:12 2024] schedule_timeout+0x11f/0x160
[Mon Mar 11 14:29:12 2024] ? select_idle_sibling+0x28/0x430
[Mon Mar 11 14:29:12 2024] ? wake_affine+0x62/0x1f0
[Mon Mar 11 14:29:12 2024] __wait_for_common+0x90/0x1d0
[Mon Mar 11 14:29:12 2024] ? __pfx_schedule_timeout+0x10/0x10
[Mon Mar 11 14:29:12 2024] __flush_workqueue+0x13a/0x3f0
[Mon Mar 11 14:29:12 2024] nfsd4_shutdown_callback+0x49/0x120 [nfsd]
[Mon Mar 11 14:29:12 2024] ? nfsd4_cld_remove+0x54/0x1d0 [nfsd]
[Mon Mar 11 14:29:12 2024] ? nfsd4_return_all_client_layouts+0xc4/0xf0 [nfsd]
[Mon Mar 11 14:29:12 2024] ? nfsd4_shutdown_copy+0x68/0xc0 [nfsd]
[Mon Mar 11 14:29:12 2024] __destroy_client+0x1f3/0x290 [nfsd]
[Mon Mar 11 14:29:12 2024] nfsd4_exchange_id+0x75f/0x770 [nfsd]
[Mon Mar 11 14:29:12 2024] ? nfsd4_decode_opaque+0x3a/0x90 [nfsd]
[Mon Mar 11 14:29:12 2024] nfsd4_proc_compound+0x44b/0x700 [nfsd]
[Mon Mar 11 14:29:12 2024] nfsd_dispatch+0x94/0x1c0 [nfsd]
[Mon Mar 11 14:29:12 2024] svc_process_common+0x2ec/0x660 [sunrpc]
[Mon Mar 11 14:29:12 2024] ? __pfx_nfsd_dispatch+0x10/0x10 [nfsd]
[Mon Mar 11 14:29:12 2024] ? __pfx_nfsd+0x10/0x10 [nfsd]
[Mon Mar 11 14:29:12 2024] svc_process+0x12d/0x170 [sunrpc]
[Mon Mar 11 14:29:12 2024] nfsd+0x84/0xb0 [nfsd]
[Mon Mar 11 14:29:12 2024] kthread+0xdd/0x100
[Mon Mar 11 14:29:12 2024] ? __pfx_kthread+0x10/0x10
[Mon Mar 11 14:29:12 2024] ret_from_fork+0x29/0x50
[Mon Mar 11 14:29:12 2024] </TASK>
[Mon Mar 11 14:29:12 2024] task:nfsd state:D stack:0 pid:8827 ppid:2 flags:0x00004000
[Mon Mar 11 14:29:12 2024] Call Trace:
[Mon Mar 11 14:29:12 2024] <TASK>
[Mon Mar 11 14:29:12 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:12 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:12 2024] schedule_timeout+0x11f/0x160
[Mon Mar 11 14:29:12 2024] ? select_idle_sibling+0x28/0x430
[Mon Mar 11 14:29:12 2024] ? wake_affine+0x62/0x1f0
[Mon Mar 11 14:29:12 2024] __wait_for_common+0x90/0x1d0
[Mon Mar 11 14:29:12 2024] ? __pfx_schedule_timeout+0x10/0x10
[Mon Mar 11 14:29:12 2024] __flush_workqueue+0x13a/0x3f0
[Mon Mar 11 14:29:12 2024] nfsd4_shutdown_callback+0x49/0x120 [nfsd]
[Mon Mar 11 14:29:12 2024] ? nfsd4_cld_remove+0x54/0x1d0 [nfsd]
[Mon Mar 11 14:29:12 2024] ? nfsd4_return_all_client_layouts+0xc4/0xf0 [nfsd]
[Mon Mar 11 14:29:12 2024] ? nfsd4_shutdown_copy+0x68/0xc0 [nfsd]
[Mon Mar 11 14:29:12 2024] __destroy_client+0x1f3/0x290 [nfsd]
[Mon Mar 11 14:29:12 2024] nfsd4_exchange_id+0x75f/0x770 [nfsd]
[Mon Mar 11 14:29:12 2024] ? nfsd4_decode_opaque+0x3a/0x90 [nfsd]
[Mon Mar 11 14:29:12 2024] nfsd4_proc_compound+0x44b/0x700 [nfsd]
[Mon Mar 11 14:29:12 2024] nfsd_dispatch+0x94/0x1c0 [nfsd]
[Mon Mar 11 14:29:12 2024] svc_process_common+0x2ec/0x660 [sunrpc]
[Mon Mar 11 14:29:12 2024] ? __pfx_nfsd_dispatch+0x10/0x10 [nfsd]
[Mon Mar 11 14:29:12 2024] ? __pfx_nfsd+0x10/0x10 [nfsd]
[Mon Mar 11 14:29:12 2024] svc_process+0x12d/0x170 [sunrpc]
[Mon Mar 11 14:29:12 2024] nfsd+0x84/0xb0 [nfsd]
[Mon Mar 11 14:29:12 2024] kthread+0xdd/0x100
[Mon Mar 11 14:29:12 2024] ? __pfx_kthread+0x10/0x10
[Mon Mar 11 14:29:12 2024] ret_from_fork+0x29/0x50
[Mon Mar 11 14:29:12 2024] </TASK>
[Mon Mar 11 14:29:12 2024] task:nfsd state:D stack:0 pid:8828 ppid:2 flags:0x00004000
[Mon Mar 11 14:29:12 2024] Call Trace:
[Mon Mar 11 14:29:12 2024] <TASK>
[Mon Mar 11 14:29:12 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:12 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:12 2024] schedule_timeout+0x11f/0x160
[Mon Mar 11 14:29:12 2024] ? select_idle_sibling+0x28/0x430
[Mon Mar 11 14:29:12 2024] ? wake_affine+0x62/0x1f0
[Mon Mar 11 14:29:12 2024] __wait_for_common+0x90/0x1d0
[Mon Mar 11 14:29:12 2024] ? __pfx_schedule_timeout+0x10/0x10
[Mon Mar 11 14:29:12 2024] __flush_workqueue+0x13a/0x3f0
[Mon Mar 11 14:29:12 2024] nfsd4_shutdown_callback+0x49/0x120 [nfsd]
[Mon Mar 11 14:29:12 2024] ? nfsd4_cld_remove+0x54/0x1d0 [nfsd]
[Mon Mar 11 14:29:12 2024] ? nfsd4_return_all_client_layouts+0xc4/0xf0 [nfsd]
[Mon Mar 11 14:29:12 2024] ? nfsd4_shutdown_copy+0x68/0xc0 [nfsd]
[Mon Mar 11 14:29:12 2024] __destroy_client+0x1f3/0x290 [nfsd]
[Mon Mar 11 14:29:12 2024] nfsd4_exchange_id+0x75f/0x770 [nfsd]
[Mon Mar 11 14:29:12 2024] ? nfsd4_decode_opaque+0x3a/0x90 [nfsd]
[Mon Mar 11 14:29:12 2024] nfsd4_proc_compound+0x44b/0x700 [nfsd]
[Mon Mar 11 14:29:12 2024] nfsd_dispatch+0x94/0x1c0 [nfsd]
[Mon Mar 11 14:29:12 2024] svc_process_common+0x2ec/0x660 [sunrpc]
[Mon Mar 11 14:29:12 2024] ? __pfx_nfsd_dispatch+0x10/0x10 [nfsd]
[Mon Mar 11 14:29:12 2024] ? __pfx_nfsd+0x10/0x10 [nfsd]
[Mon Mar 11 14:29:12 2024] svc_process+0x12d/0x170 [sunrpc]
[Mon Mar 11 14:29:12 2024] nfsd+0x84/0xb0 [nfsd]
[Mon Mar 11 14:29:12 2024] kthread+0xdd/0x100
[Mon Mar 11 14:29:12 2024] ? __pfx_kthread+0x10/0x10
[Mon Mar 11 14:29:12 2024] ret_from_fork+0x29/0x50
[Mon Mar 11 14:29:12 2024] </TASK>
[Mon Mar 11 14:29:12 2024] task:nfsd state:D stack:0 pid:8829 ppid:2 flags:0x00004000
[Mon Mar 11 14:29:12 2024] Call Trace:
[Mon Mar 11 14:29:12 2024] <TASK>
[Mon Mar 11 14:29:12 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:12 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:12 2024] schedule_timeout+0x11f/0x160
[Mon Mar 11 14:29:12 2024] ? select_idle_sibling+0x28/0x430
[Mon Mar 11 14:29:12 2024] ? tcp_recvmsg+0x196/0x210
[Mon Mar 11 14:29:12 2024] ? wake_affine+0x62/0x1f0
[Mon Mar 11 14:29:12 2024] __wait_for_common+0x90/0x1d0
[Mon Mar 11 14:29:12 2024] ? __pfx_schedule_timeout+0x10/0x10
[Mon Mar 11 14:29:12 2024] __flush_workqueue+0x13a/0x3f0
[Mon Mar 11 14:29:12 2024] nfsd4_destroy_session+0x1a4/0x240 [nfsd]
[Mon Mar 11 14:29:12 2024] nfsd4_proc_compound+0x44b/0x700 [nfsd]
[Mon Mar 11 14:29:12 2024] nfsd_dispatch+0x94/0x1c0 [nfsd]
[Mon Mar 11 14:29:12 2024] svc_process_common+0x2ec/0x660 [sunrpc]
[Mon Mar 11 14:29:12 2024] ? __pfx_nfsd_dispatch+0x10/0x10 [nfsd]
[Mon Mar 11 14:29:12 2024] ? __pfx_nfsd+0x10/0x10 [nfsd]
[Mon Mar 11 14:29:12 2024] svc_process+0x12d/0x170 [sunrpc]
[Mon Mar 11 14:29:12 2024] nfsd+0x84/0xb0 [nfsd]
[Mon Mar 11 14:29:12 2024] kthread+0xdd/0x100
[Mon Mar 11 14:29:12 2024] ? __pfx_kthread+0x10/0x10
[Mon Mar 11 14:29:12 2024] ret_from_fork+0x29/0x50
[Mon Mar 11 14:29:12 2024] </TASK>
[Mon Mar 11 14:29:12 2024] task:nfsd state:D stack:0 pid:8830 ppid:2 flags:0x00004000
[Mon Mar 11 14:29:12 2024] Call Trace:
[Mon Mar 11 14:29:12 2024] <TASK>
[Mon Mar 11 14:29:12 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:12 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:12 2024] schedule_timeout+0x11f/0x160
[Mon Mar 11 14:29:12 2024] ? select_idle_sibling+0x28/0x430
[Mon Mar 11 14:29:12 2024] ? wake_affine+0x62/0x1f0
[Mon Mar 11 14:29:12 2024] __wait_for_common+0x90/0x1d0
[Mon Mar 11 14:29:12 2024] ? __pfx_schedule_timeout+0x10/0x10
[Mon Mar 11 14:29:12 2024] __flush_workqueue+0x13a/0x3f0
[Mon Mar 11 14:29:12 2024] nfsd4_shutdown_callback+0x49/0x120 [nfsd]
[Mon Mar 11 14:29:12 2024] ? nfsd4_cld_remove+0x54/0x1d0 [nfsd]
[Mon Mar 11 14:29:12 2024] ? nfsd4_return_all_client_layouts+0xc4/0xf0 [nfsd]
[Mon Mar 11 14:29:12 2024] ? nfsd4_shutdown_copy+0x68/0xc0 [nfsd]
[Mon Mar 11 14:29:12 2024] __destroy_client+0x1f3/0x290 [nfsd]
[Mon Mar 11 14:29:12 2024] nfsd4_exchange_id+0x75f/0x770 [nfsd]
[Mon Mar 11 14:29:12 2024] ? nfsd4_decode_opaque+0x3a/0x90 [nfsd]
[Mon Mar 11 14:29:12 2024] nfsd4_proc_compound+0x44b/0x700 [nfsd]
[Mon Mar 11 14:29:12 2024] nfsd_dispatch+0x94/0x1c0 [nfsd]
[Mon Mar 11 14:29:12 2024] svc_process_common+0x2ec/0x660 [sunrpc]
[Mon Mar 11 14:29:12 2024] ? __pfx_nfsd_dispatch+0x10/0x10 [nfsd]
[Mon Mar 11 14:29:12 2024] ? __pfx_nfsd+0x10/0x10 [nfsd]
[Mon Mar 11 14:29:12 2024] svc_process+0x12d/0x170 [sunrpc]
[Mon Mar 11 14:29:12 2024] nfsd+0x84/0xb0 [nfsd]
[Mon Mar 11 14:29:12 2024] kthread+0xdd/0x100
[Mon Mar 11 14:29:12 2024] ? __pfx_kthread+0x10/0x10
[Mon Mar 11 14:29:12 2024] ret_from_fork+0x29/0x50
[Mon Mar 11 14:29:12 2024] </TASK>
[Mon Mar 11 14:29:12 2024] task:nfsd state:D stack:0 pid:8831 ppid:2 flags:0x00004000
[Mon Mar 11 14:29:12 2024] Call Trace:
[Mon Mar 11 14:29:12 2024] <TASK>
[Mon Mar 11 14:29:12 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:12 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:12 2024] schedule_timeout+0x11f/0x160
[Mon Mar 11 14:29:12 2024] ? select_idle_sibling+0x28/0x430
[Mon Mar 11 14:29:12 2024] ? wake_affine+0x62/0x1f0
[Mon Mar 11 14:29:12 2024] __wait_for_common+0x90/0x1d0
[Mon Mar 11 14:29:12 2024] ? __pfx_schedule_timeout+0x10/0x10
[Mon Mar 11 14:29:12 2024] __flush_workqueue+0x13a/0x3f0
[Mon Mar 11 14:29:12 2024] nfsd4_shutdown_callback+0x49/0x120 [nfsd]
[Mon Mar 11 14:29:12 2024] ? nfsd4_cld_remove+0x54/0x1d0 [nfsd]
[Mon Mar 11 14:29:12 2024] ? nfsd4_return_all_client_layouts+0xc4/0xf0 [nfsd]
[Mon Mar 11 14:29:12 2024] ? nfsd4_shutdown_copy+0x68/0xc0 [nfsd]
[Mon Mar 11 14:29:12 2024] __destroy_client+0x1f3/0x290 [nfsd]
[Mon Mar 11 14:29:12 2024] nfsd4_exchange_id+0x75f/0x770 [nfsd]
[Mon Mar 11 14:29:12 2024] ? nfsd4_decode_opaque+0x3a/0x90 [nfsd]
[Mon Mar 11 14:29:12 2024] nfsd4_proc_compound+0x44b/0x700 [nfsd]
[Mon Mar 11 14:29:12 2024] nfsd_dispatch+0x94/0x1c0 [nfsd]
[Mon Mar 11 14:29:12 2024] svc_process_common+0x2ec/0x660 [sunrpc]
[Mon Mar 11 14:29:12 2024] ? __pfx_nfsd_dispatch+0x10/0x10 [nfsd]
[Mon Mar 11 14:29:12 2024] ? __pfx_nfsd+0x10/0x10 [nfsd]
[Mon Mar 11 14:29:12 2024] svc_process+0x12d/0x170 [sunrpc]
[Mon Mar 11 14:29:12 2024] nfsd+0x84/0xb0 [nfsd]
[Mon Mar 11 14:29:12 2024] kthread+0xdd/0x100
[Mon Mar 11 14:29:12 2024] ? __pfx_kthread+0x10/0x10
[Mon Mar 11 14:29:12 2024] ret_from_fork+0x29/0x50
[Mon Mar 11 14:29:12 2024] </TASK>
[Mon Mar 11 14:29:12 2024] task:nfsd state:D stack:0 pid:8832 ppid:2 flags:0x00004000
[Mon Mar 11 14:29:12 2024] Call Trace:
[Mon Mar 11 14:29:12 2024] <TASK>
[Mon Mar 11 14:29:12 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:12 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:12 2024] schedule_timeout+0x11f/0x160
[Mon Mar 11 14:29:12 2024] ? select_idle_sibling+0x28/0x430
[Mon Mar 11 14:29:12 2024] ? wake_affine+0x62/0x1f0
[Mon Mar 11 14:29:12 2024] __wait_for_common+0x90/0x1d0
[Mon Mar 11 14:29:12 2024] ? __pfx_schedule_timeout+0x10/0x10
[Mon Mar 11 14:29:12 2024] __flush_workqueue+0x13a/0x3f0
[Mon Mar 11 14:29:12 2024] nfsd4_shutdown_callback+0x49/0x120 [nfsd]
[Mon Mar 11 14:29:12 2024] ? nfsd4_cld_remove+0x54/0x1d0 [nfsd]
[Mon Mar 11 14:29:12 2024] ? nfsd4_return_all_client_layouts+0xc4/0xf0 [nfsd]
[Mon Mar 11 14:29:12 2024] ? nfsd4_shutdown_copy+0x68/0xc0 [nfsd]
[Mon Mar 11 14:29:12 2024] __destroy_client+0x1f3/0x290 [nfsd]
[Mon Mar 11 14:29:12 2024] nfsd4_exchange_id+0x378/0x770 [nfsd]
[Mon Mar 11 14:29:12 2024] ? nfsd4_decode_opaque+0x3a/0x90 [nfsd]
[Mon Mar 11 14:29:12 2024] nfsd4_proc_compound+0x44b/0x700 [nfsd]
[Mon Mar 11 14:29:12 2024] nfsd_dispatch+0x94/0x1c0 [nfsd]
[Mon Mar 11 14:29:12 2024] svc_process_common+0x2ec/0x660 [sunrpc]
[Mon Mar 11 14:29:12 2024] ? __pfx_nfsd_dispatch+0x10/0x10 [nfsd]
[Mon Mar 11 14:29:12 2024] ? __pfx_nfsd+0x10/0x10 [nfsd]
[Mon Mar 11 14:29:12 2024] svc_process+0x12d/0x170 [sunrpc]
[Mon Mar 11 14:29:12 2024] nfsd+0x84/0xb0 [nfsd]
[Mon Mar 11 14:29:12 2024] kthread+0xdd/0x100
[Mon Mar 11 14:29:12 2024] ? __pfx_kthread+0x10/0x10
[Mon Mar 11 14:29:12 2024] ret_from_fork+0x29/0x50
[Mon Mar 11 14:29:12 2024] </TASK>
[Mon Mar 11 14:29:12 2024] task:nfsd state:D stack:0 pid:8833 ppid:2 flags:0x00004000
[Mon Mar 11 14:29:12 2024] Call Trace:
[Mon Mar 11 14:29:12 2024] <TASK>
[Mon Mar 11 14:29:12 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:12 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:12 2024] schedule_timeout+0x11f/0x160
[Mon Mar 11 14:29:12 2024] ? select_idle_sibling+0x28/0x430
[Mon Mar 11 14:29:12 2024] __wait_for_common+0x90/0x1d0
[Mon Mar 11 14:29:12 2024] ? __pfx_schedule_timeout+0x10/0x10
[Mon Mar 11 14:29:12 2024] __flush_workqueue+0x13a/0x3f0
[Mon Mar 11 14:29:12 2024] nfsd4_shutdown_callback+0x49/0x120 [nfsd]
[Mon Mar 11 14:29:12 2024] ? nfsd4_cld_remove+0x54/0x1d0 [nfsd]
[Mon Mar 11 14:29:12 2024] ? nfsd4_return_all_client_layouts+0xc4/0xf0 [nfsd]
[Mon Mar 11 14:29:12 2024] ? nfsd4_shutdown_copy+0x68/0xc0 [nfsd]
[Mon Mar 11 14:29:12 2024] __destroy_client+0x1f3/0x290 [nfsd]
[Mon Mar 11 14:29:12 2024] nfsd4_exchange_id+0x75f/0x770 [nfsd]
[Mon Mar 11 14:29:12 2024] ? percpu_counter_add_batch+0x67/0x70
[Mon Mar 11 14:29:12 2024] nfsd4_proc_compound+0x44b/0x700 [nfsd]
[Mon Mar 11 14:29:12 2024] nfsd_dispatch+0x94/0x1c0 [nfsd]
[Mon Mar 11 14:29:12 2024] svc_process_common+0x2ec/0x660 [sunrpc]
[Mon Mar 11 14:29:12 2024] ? __pfx_nfsd_dispatch+0x10/0x10 [nfsd]
[Mon Mar 11 14:29:12 2024] ? __pfx_nfsd+0x10/0x10 [nfsd]
[Mon Mar 11 14:29:12 2024] svc_process+0x12d/0x170 [sunrpc]
[Mon Mar 11 14:29:12 2024] nfsd+0x84/0xb0 [nfsd]
[Mon Mar 11 14:29:12 2024] kthread+0xdd/0x100
[Mon Mar 11 14:29:12 2024] ? __pfx_kthread+0x10/0x10
[Mon Mar 11 14:29:12 2024] ret_from_fork+0x29/0x50
[Mon Mar 11 14:29:12 2024] </TASK>
[Mon Mar 11 14:29:12 2024] task:nfsd state:D stack:0 pid:8834 ppid:2 flags:0x00004000
[Mon Mar 11 14:29:12 2024] Call Trace:
[Mon Mar 11 14:29:12 2024] <TASK>
[Mon Mar 11 14:29:12 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:12 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:12 2024] schedule_timeout+0x11f/0x160
[Mon Mar 11 14:29:12 2024] ? select_idle_sibling+0x28/0x430
[Mon Mar 11 14:29:12 2024] __wait_for_common+0x90/0x1d0
[Mon Mar 11 14:29:12 2024] ? __pfx_schedule_timeout+0x10/0x10
[Mon Mar 11 14:29:12 2024] __flush_workqueue+0x13a/0x3f0
[Mon Mar 11 14:29:12 2024] nfsd4_shutdown_callback+0x49/0x120 [nfsd]
[Mon Mar 11 14:29:12 2024] ? nfsd4_cld_remove+0x54/0x1d0 [nfsd]
[Mon Mar 11 14:29:12 2024] ? nfsd4_return_all_client_layouts+0xc4/0xf0 [nfsd]
[Mon Mar 11 14:29:12 2024] ? nfsd4_shutdown_copy+0x68/0xc0 [nfsd]
[Mon Mar 11 14:29:12 2024] __destroy_client+0x1f3/0x290 [nfsd]
[Mon Mar 11 14:29:12 2024] nfsd4_exchange_id+0x75f/0x770 [nfsd]
[Mon Mar 11 14:29:12 2024] ? nfsd4_decode_opaque+0x3a/0x90 [nfsd]
[Mon Mar 11 14:29:12 2024] nfsd4_proc_compound+0x44b/0x700 [nfsd]
[Mon Mar 11 14:29:12 2024] nfsd_dispatch+0x94/0x1c0 [nfsd]
[Mon Mar 11 14:29:12 2024] svc_process_common+0x2ec/0x660 [sunrpc]
[Mon Mar 11 14:29:12 2024] ? __pfx_nfsd_dispatch+0x10/0x10 [nfsd]
[Mon Mar 11 14:29:12 2024] ? __pfx_nfsd+0x10/0x10 [nfsd]
[Mon Mar 11 14:29:12 2024] svc_process+0x12d/0x170 [sunrpc]
[Mon Mar 11 14:29:12 2024] nfsd+0x84/0xb0 [nfsd]
[Mon Mar 11 14:29:12 2024] kthread+0xdd/0x100
[Mon Mar 11 14:29:12 2024] ? __pfx_kthread+0x10/0x10
[Mon Mar 11 14:29:12 2024] ret_from_fork+0x29/0x50
[Mon Mar 11 14:29:12 2024] </TASK>
[Mon Mar 11 14:29:12 2024] task:nfsd state:D stack:0 pid:8835 ppid:2 flags:0x00004000
[Mon Mar 11 14:29:12 2024] Call Trace:
[Mon Mar 11 14:29:12 2024] <TASK>
[Mon Mar 11 14:29:12 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:12 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:12 2024] schedule_timeout+0x11f/0x160
[Mon Mar 11 14:29:12 2024] ? select_idle_sibling+0x28/0x430
[Mon Mar 11 14:29:12 2024] ? wake_affine+0x62/0x1f0
[Mon Mar 11 14:29:12 2024] __wait_for_common+0x90/0x1d0
[Mon Mar 11 14:29:12 2024] ? __pfx_schedule_timeout+0x10/0x10
[Mon Mar 11 14:29:12 2024] __flush_workqueue+0x13a/0x3f0
[Mon Mar 11 14:29:12 2024] nfsd4_shutdown_callback+0x49/0x120 [nfsd]
[Mon Mar 11 14:29:12 2024] ? nfsd4_cld_remove+0x54/0x1d0 [nfsd]
[Mon Mar 11 14:29:12 2024] ? nfsd4_return_all_client_layouts+0xc4/0xf0 [nfsd]
[Mon Mar 11 14:29:12 2024] ? nfsd4_shutdown_copy+0x68/0xc0 [nfsd]
[Mon Mar 11 14:29:12 2024] __destroy_client+0x1f3/0x290 [nfsd]
[Mon Mar 11 14:29:12 2024] nfsd4_exchange_id+0x75f/0x770 [nfsd]
[Mon Mar 11 14:29:12 2024] ? nfsd4_decode_opaque+0x3a/0x90 [nfsd]
[Mon Mar 11 14:29:12 2024] nfsd4_proc_compound+0x44b/0x700 [nfsd]
[Mon Mar 11 14:29:12 2024] nfsd_dispatch+0x94/0x1c0 [nfsd]
[Mon Mar 11 14:29:12 2024] svc_process_common+0x2ec/0x660 [sunrpc]
[Mon Mar 11 14:29:12 2024] ? __pfx_nfsd_dispatch+0x10/0x10 [nfsd]
[Mon Mar 11 14:29:12 2024] ? __pfx_nfsd+0x10/0x10 [nfsd]
[Mon Mar 11 14:29:12 2024] svc_process+0x12d/0x170 [sunrpc]
[Mon Mar 11 14:29:12 2024] nfsd+0x84/0xb0 [nfsd]
[Mon Mar 11 14:29:12 2024] kthread+0xdd/0x100
[Mon Mar 11 14:29:12 2024] ? __pfx_kthread+0x10/0x10
[Mon Mar 11 14:29:12 2024] ret_from_fork+0x29/0x50
[Mon Mar 11 14:29:12 2024] </TASK>
[Mon Mar 11 14:29:12 2024] task:nfsd state:D stack:0 pid:8836 ppid:2 flags:0x00004000
[Mon Mar 11 14:29:12 2024] Call Trace:
[Mon Mar 11 14:29:12 2024] <TASK>
[Mon Mar 11 14:29:12 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:12 2024] ? __skb_datagram_iter+0x79/0x2e0
[Mon Mar 11 14:29:12 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:12 2024] schedule_timeout+0x11f/0x160
[Mon Mar 11 14:29:12 2024] ? select_idle_sibling+0x28/0x430
[Mon Mar 11 14:29:12 2024] ? tcp_recvmsg_locked+0x272/0x910
[Mon Mar 11 14:29:12 2024] ? wake_affine+0x62/0x1f0
[Mon Mar 11 14:29:12 2024] __wait_for_common+0x90/0x1d0
[Mon Mar 11 14:29:12 2024] ? __pfx_schedule_timeout+0x10/0x10
[Mon Mar 11 14:29:12 2024] __flush_workqueue+0x13a/0x3f0
[Mon Mar 11 14:29:12 2024] nfsd4_shutdown_callback+0x49/0x120 [nfsd]
[Mon Mar 11 14:29:12 2024] ? wake_affine+0x62/0x1f0
[Mon Mar 11 14:29:12 2024] ? nfsd4_cld_remove+0x54/0x1d0 [nfsd]
[Mon Mar 11 14:29:12 2024] ? nfsd4_return_all_client_layouts+0xc4/0xf0 [nfsd]
[Mon Mar 11 14:29:12 2024] ? nfsd4_shutdown_copy+0x68/0xc0 [nfsd]
[Mon Mar 11 14:29:12 2024] __destroy_client+0x1f3/0x290 [nfsd]
[Mon Mar 11 14:29:12 2024] nfsd4_destroy_clientid+0xe2/0x1c0 [nfsd]
[Mon Mar 11 14:29:12 2024] nfsd4_proc_compound+0x44b/0x700 [nfsd]
[Mon Mar 11 14:29:12 2024] nfsd_dispatch+0x94/0x1c0 [nfsd]
[Mon Mar 11 14:29:12 2024] svc_process_common+0x2ec/0x660 [sunrpc]
[Mon Mar 11 14:29:12 2024] ? __pfx_nfsd_dispatch+0x10/0x10 [nfsd]
[Mon Mar 11 14:29:12 2024] ? __pfx_nfsd+0x10/0x10 [nfsd]
[Mon Mar 11 14:29:12 2024] svc_process+0x12d/0x170 [sunrpc]
[Mon Mar 11 14:29:12 2024] nfsd+0x84/0xb0 [nfsd]
[Mon Mar 11 14:29:12 2024] kthread+0xdd/0x100
[Mon Mar 11 14:29:12 2024] ? __pfx_kthread+0x10/0x10
[Mon Mar 11 14:29:12 2024] ret_from_fork+0x29/0x50
[Mon Mar 11 14:29:12 2024] </TASK>
[Mon Mar 11 14:29:12 2024] task:nfsd state:D stack:0 pid:8837 ppid:2 flags:0x00004000
[Mon Mar 11 14:29:12 2024] Call Trace:
[Mon Mar 11 14:29:12 2024] <TASK>
[Mon Mar 11 14:29:12 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:12 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:12 2024] schedule_timeout+0x11f/0x160
[Mon Mar 11 14:29:12 2024] ? select_idle_sibling+0x28/0x430
[Mon Mar 11 14:29:12 2024] ? tcp_recvmsg+0x196/0x210
[Mon Mar 11 14:29:12 2024] __wait_for_common+0x90/0x1d0
[Mon Mar 11 14:29:12 2024] ? __pfx_schedule_timeout+0x10/0x10
[Mon Mar 11 14:29:12 2024] __flush_workqueue+0x13a/0x3f0
[Mon Mar 11 14:29:12 2024] nfsd4_destroy_session+0x1a4/0x240 [nfsd]
[Mon Mar 11 14:29:12 2024] nfsd4_proc_compound+0x44b/0x700 [nfsd]
[Mon Mar 11 14:29:12 2024] nfsd_dispatch+0x94/0x1c0 [nfsd]
[Mon Mar 11 14:29:12 2024] svc_process_common+0x2ec/0x660 [sunrpc]
[Mon Mar 11 14:29:12 2024] ? __pfx_nfsd_dispatch+0x10/0x10 [nfsd]
[Mon Mar 11 14:29:12 2024] ? __pfx_nfsd+0x10/0x10 [nfsd]
[Mon Mar 11 14:29:12 2024] svc_process+0x12d/0x170 [sunrpc]
[Mon Mar 11 14:29:12 2024] nfsd+0x84/0xb0 [nfsd]
[Mon Mar 11 14:29:12 2024] kthread+0xdd/0x100
[Mon Mar 11 14:29:12 2024] ? __pfx_kthread+0x10/0x10
[Mon Mar 11 14:29:12 2024] ret_from_fork+0x29/0x50
[Mon Mar 11 14:29:12 2024] </TASK>
[Mon Mar 11 14:29:12 2024] task:nfsd state:D stack:0 pid:8838 ppid:2 flags:0x00004000
[Mon Mar 11 14:29:12 2024] Call Trace:
[Mon Mar 11 14:29:12 2024] <TASK>
[Mon Mar 11 14:29:12 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:12 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:12 2024] schedule_timeout+0x11f/0x160
[Mon Mar 11 14:29:12 2024] ? select_idle_sibling+0x28/0x430
[Mon Mar 11 14:29:12 2024] ? tcp_recvmsg+0x196/0x210
[Mon Mar 11 14:29:12 2024] ? wake_affine+0x62/0x1f0
[Mon Mar 11 14:29:12 2024] __wait_for_common+0x90/0x1d0
[Mon Mar 11 14:29:12 2024] ? __pfx_schedule_timeout+0x10/0x10
[Mon Mar 11 14:29:12 2024] __flush_workqueue+0x13a/0x3f0
[Mon Mar 11 14:29:12 2024] nfsd4_destroy_session+0x1a4/0x240 [nfsd]
[Mon Mar 11 14:29:12 2024] nfsd4_proc_compound+0x44b/0x700 [nfsd]
[Mon Mar 11 14:29:12 2024] nfsd_dispatch+0x94/0x1c0 [nfsd]
[Mon Mar 11 14:29:12 2024] svc_process_common+0x2ec/0x660 [sunrpc]
[Mon Mar 11 14:29:12 2024] ? __pfx_nfsd_dispatch+0x10/0x10 [nfsd]
[Mon Mar 11 14:29:12 2024] ? __pfx_nfsd+0x10/0x10 [nfsd]
[Mon Mar 11 14:29:12 2024] svc_process+0x12d/0x170 [sunrpc]
[Mon Mar 11 14:29:12 2024] nfsd+0x84/0xb0 [nfsd]
[Mon Mar 11 14:29:12 2024] kthread+0xdd/0x100
[Mon Mar 11 14:29:12 2024] ? __pfx_kthread+0x10/0x10
[Mon Mar 11 14:29:12 2024] ret_from_fork+0x29/0x50
[Mon Mar 11 14:29:12 2024] </TASK>
[Mon Mar 11 14:29:12 2024] task:nfsd state:D stack:0 pid:8839 ppid:2 flags:0x00004000
[Mon Mar 11 14:29:12 2024] Call Trace:
[Mon Mar 11 14:29:12 2024] <TASK>
[Mon Mar 11 14:29:12 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:12 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:12 2024] schedule_timeout+0x11f/0x160
[Mon Mar 11 14:29:12 2024] ? select_idle_sibling+0x28/0x430
[Mon Mar 11 14:29:12 2024] ? wake_affine+0x62/0x1f0
[Mon Mar 11 14:29:12 2024] __wait_for_common+0x90/0x1d0
[Mon Mar 11 14:29:12 2024] ? __pfx_schedule_timeout+0x10/0x10
[Mon Mar 11 14:29:12 2024] __flush_workqueue+0x13a/0x3f0
[Mon Mar 11 14:29:12 2024] nfsd4_shutdown_callback+0x49/0x120 [nfsd]
[Mon Mar 11 14:29:12 2024] ? nfsd4_cld_remove+0x54/0x1d0 [nfsd]
[Mon Mar 11 14:29:12 2024] ? nfsd4_return_all_client_layouts+0xc4/0xf0 [nfsd]
[Mon Mar 11 14:29:12 2024] ? nfsd4_shutdown_copy+0x68/0xc0 [nfsd]
[Mon Mar 11 14:29:12 2024] __destroy_client+0x1f3/0x290 [nfsd]
[Mon Mar 11 14:29:12 2024] nfsd4_exchange_id+0x75f/0x770 [nfsd]
[Mon Mar 11 14:29:12 2024] ? nfsd4_decode_opaque+0x3a/0x90 [nfsd]
[Mon Mar 11 14:29:12 2024] nfsd4_proc_compound+0x44b/0x700 [nfsd]
[Mon Mar 11 14:29:12 2024] nfsd_dispatch+0x94/0x1c0 [nfsd]
[Mon Mar 11 14:29:12 2024] svc_process_common+0x2ec/0x660 [sunrpc]
[Mon Mar 11 14:29:12 2024] ? __pfx_nfsd_dispatch+0x10/0x10 [nfsd]
[Mon Mar 11 14:29:12 2024] ? __pfx_nfsd+0x10/0x10 [nfsd]
[Mon Mar 11 14:29:12 2024] svc_process+0x12d/0x170 [sunrpc]
[Mon Mar 11 14:29:12 2024] nfsd+0x84/0xb0 [nfsd]
[Mon Mar 11 14:29:12 2024] kthread+0xdd/0x100
[Mon Mar 11 14:29:12 2024] ? __pfx_kthread+0x10/0x10
[Mon Mar 11 14:29:12 2024] ret_from_fork+0x29/0x50
[Mon Mar 11 14:29:12 2024] </TASK>
[Mon Mar 11 14:29:12 2024] task:nfsd state:D stack:0 pid:8840 ppid:2 flags:0x00004000
[Mon Mar 11 14:29:12 2024] Call Trace:
[Mon Mar 11 14:29:12 2024] <TASK>
[Mon Mar 11 14:29:12 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:12 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:12 2024] schedule_timeout+0x11f/0x160
[Mon Mar 11 14:29:12 2024] ? select_idle_sibling+0x28/0x430
[Mon Mar 11 14:29:12 2024] ? tcp_recvmsg+0x196/0x210
[Mon Mar 11 14:29:12 2024] __wait_for_common+0x90/0x1d0
[Mon Mar 11 14:29:12 2024] ? __pfx_schedule_timeout+0x10/0x10
[Mon Mar 11 14:29:12 2024] __flush_workqueue+0x13a/0x3f0
[Mon Mar 11 14:29:12 2024] nfsd4_destroy_session+0x1a4/0x240 [nfsd]
[Mon Mar 11 14:29:12 2024] nfsd4_proc_compound+0x44b/0x700 [nfsd]
[Mon Mar 11 14:29:12 2024] nfsd_dispatch+0x94/0x1c0 [nfsd]
[Mon Mar 11 14:29:12 2024] svc_process_common+0x2ec/0x660 [sunrpc]
[Mon Mar 11 14:29:12 2024] ? __pfx_nfsd_dispatch+0x10/0x10 [nfsd]
[Mon Mar 11 14:29:12 2024] ? __pfx_nfsd+0x10/0x10 [nfsd]
[Mon Mar 11 14:29:12 2024] svc_process+0x12d/0x170 [sunrpc]
[Mon Mar 11 14:29:12 2024] nfsd+0x84/0xb0 [nfsd]
[Mon Mar 11 14:29:12 2024] kthread+0xdd/0x100
[Mon Mar 11 14:29:12 2024] ? __pfx_kthread+0x10/0x10
[Mon Mar 11 14:29:12 2024] ret_from_fork+0x29/0x50
[Mon Mar 11 14:29:12 2024] </TASK>
[Mon Mar 11 14:29:12 2024] task:nfsd state:D stack:0 pid:8841 ppid:2 flags:0x00004000
[Mon Mar 11 14:29:12 2024] Call Trace:
[Mon Mar 11 14:29:12 2024] <TASK>
[Mon Mar 11 14:29:12 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:12 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:12 2024] schedule_timeout+0x11f/0x160
[Mon Mar 11 14:29:12 2024] ? select_idle_sibling+0x28/0x430
[Mon Mar 11 14:29:12 2024] ? tcp_recvmsg+0x196/0x210
[Mon Mar 11 14:29:12 2024] ? wake_affine+0x62/0x1f0
[Mon Mar 11 14:29:12 2024] __wait_for_common+0x90/0x1d0
[Mon Mar 11 14:29:12 2024] ? __pfx_schedule_timeout+0x10/0x10
[Mon Mar 11 14:29:12 2024] __flush_workqueue+0x13a/0x3f0
[Mon Mar 11 14:29:12 2024] nfsd4_destroy_session+0x1a4/0x240 [nfsd]
[Mon Mar 11 14:29:12 2024] nfsd4_proc_compound+0x44b/0x700 [nfsd]
[Mon Mar 11 14:29:12 2024] nfsd_dispatch+0x94/0x1c0 [nfsd]
[Mon Mar 11 14:29:12 2024] svc_process_common+0x2ec/0x660 [sunrpc]
[Mon Mar 11 14:29:12 2024] ? __pfx_nfsd_dispatch+0x10/0x10 [nfsd]
[Mon Mar 11 14:29:12 2024] ? __pfx_nfsd+0x10/0x10 [nfsd]
[Mon Mar 11 14:29:12 2024] svc_process+0x12d/0x170 [sunrpc]
[Mon Mar 11 14:29:12 2024] nfsd+0x84/0xb0 [nfsd]
[Mon Mar 11 14:29:12 2024] kthread+0xdd/0x100
[Mon Mar 11 14:29:12 2024] ? __pfx_kthread+0x10/0x10
[Mon Mar 11 14:29:12 2024] ret_from_fork+0x29/0x50
[Mon Mar 11 14:29:12 2024] </TASK>
[Mon Mar 11 14:29:12 2024] task:nfsd state:D stack:0 pid:8842 ppid:2 flags:0x00004000
[Mon Mar 11 14:29:12 2024] Call Trace:
[Mon Mar 11 14:29:12 2024] <TASK>
[Mon Mar 11 14:29:12 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:12 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:12 2024] schedule_timeout+0x11f/0x160
[Mon Mar 11 14:29:12 2024] ? select_idle_sibling+0x28/0x430
[Mon Mar 11 14:29:12 2024] ? tcp_recvmsg+0x196/0x210
[Mon Mar 11 14:29:12 2024] ? wake_affine+0x62/0x1f0
[Mon Mar 11 14:29:12 2024] __wait_for_common+0x90/0x1d0
[Mon Mar 11 14:29:12 2024] ? __pfx_schedule_timeout+0x10/0x10
[Mon Mar 11 14:29:12 2024] __flush_workqueue+0x13a/0x3f0
[Mon Mar 11 14:29:12 2024] nfsd4_destroy_session+0x1a4/0x240 [nfsd]
[Mon Mar 11 14:29:12 2024] nfsd4_proc_compound+0x44b/0x700 [nfsd]
[Mon Mar 11 14:29:12 2024] nfsd_dispatch+0x94/0x1c0 [nfsd]
[Mon Mar 11 14:29:12 2024] svc_process_common+0x2ec/0x660 [sunrpc]
[Mon Mar 11 14:29:12 2024] ? __pfx_nfsd_dispatch+0x10/0x10 [nfsd]
[Mon Mar 11 14:29:12 2024] ? __pfx_nfsd+0x10/0x10 [nfsd]
[Mon Mar 11 14:29:12 2024] svc_process+0x12d/0x170 [sunrpc]
[Mon Mar 11 14:29:12 2024] nfsd+0x84/0xb0 [nfsd]
[Mon Mar 11 14:29:12 2024] kthread+0xdd/0x100
[Mon Mar 11 14:29:12 2024] ? __pfx_kthread+0x10/0x10
[Mon Mar 11 14:29:12 2024] ret_from_fork+0x29/0x50
[Mon Mar 11 14:29:12 2024] </TASK>
[Mon Mar 11 14:29:12 2024] task:nfsd state:D stack:0 pid:8843 ppid:2 flags:0x00004000
[Mon Mar 11 14:29:12 2024] Call Trace:
[Mon Mar 11 14:29:12 2024] <TASK>
[Mon Mar 11 14:29:12 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:12 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:12 2024] schedule_timeout+0x11f/0x160
[Mon Mar 11 14:29:12 2024] ? select_idle_sibling+0x28/0x430
[Mon Mar 11 14:29:12 2024] ? wake_affine+0x62/0x1f0
[Mon Mar 11 14:29:12 2024] ? fsnotify+0x2e1/0x3a0
[Mon Mar 11 14:29:12 2024] __wait_for_common+0x90/0x1d0
[Mon Mar 11 14:29:12 2024] ? __pfx_schedule_timeout+0x10/0x10
[Mon Mar 11 14:29:12 2024] __flush_workqueue+0x13a/0x3f0
[Mon Mar 11 14:29:12 2024] nfsd4_create_session+0x8c1/0xb30 [nfsd]
[Mon Mar 11 14:29:12 2024] ? nfsd4_proc_compound+0x44b/0x700 [nfsd]
[Mon Mar 11 14:29:12 2024] nfsd4_proc_compound+0x44b/0x700 [nfsd]
[Mon Mar 11 14:29:12 2024] nfsd_dispatch+0x94/0x1c0 [nfsd]
[Mon Mar 11 14:29:12 2024] svc_process_common+0x2ec/0x660 [sunrpc]
[Mon Mar 11 14:29:12 2024] ? __pfx_nfsd_dispatch+0x10/0x10 [nfsd]
[Mon Mar 11 14:29:12 2024] ? __pfx_nfsd+0x10/0x10 [nfsd]
[Mon Mar 11 14:29:12 2024] svc_process+0x12d/0x170 [sunrpc]
[Mon Mar 11 14:29:12 2024] nfsd+0x84/0xb0 [nfsd]
[Mon Mar 11 14:29:12 2024] kthread+0xdd/0x100
[Mon Mar 11 14:29:12 2024] ? __pfx_kthread+0x10/0x10
[Mon Mar 11 14:29:12 2024] ret_from_fork+0x29/0x50
[Mon Mar 11 14:29:12 2024] </TASK>
[Mon Mar 11 14:29:12 2024] task:nfsd state:D stack:0 pid:8844 ppid:2 flags:0x00004000
[Mon Mar 11 14:29:12 2024] Call Trace:
[Mon Mar 11 14:29:12 2024] <TASK>
[Mon Mar 11 14:29:12 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:12 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:12 2024] schedule_timeout+0x11f/0x160
[Mon Mar 11 14:29:12 2024] ? select_idle_sibling+0x28/0x430
[Mon Mar 11 14:29:12 2024] ? tcp_recvmsg+0x196/0x210
[Mon Mar 11 14:29:12 2024] ? wake_affine+0x62/0x1f0
[Mon Mar 11 14:29:12 2024] __wait_for_common+0x90/0x1d0
[Mon Mar 11 14:29:12 2024] ? __pfx_schedule_timeout+0x10/0x10
[Mon Mar 11 14:29:12 2024] __flush_workqueue+0x13a/0x3f0
[Mon Mar 11 14:29:12 2024] nfsd4_destroy_session+0x1a4/0x240 [nfsd]
[Mon Mar 11 14:29:12 2024] nfsd4_proc_compound+0x44b/0x700 [nfsd]
[Mon Mar 11 14:29:12 2024] nfsd_dispatch+0x94/0x1c0 [nfsd]
[Mon Mar 11 14:29:12 2024] svc_process_common+0x2ec/0x660 [sunrpc]
[Mon Mar 11 14:29:12 2024] ? __pfx_nfsd_dispatch+0x10/0x10 [nfsd]
[Mon Mar 11 14:29:12 2024] ? __pfx_nfsd+0x10/0x10 [nfsd]
[Mon Mar 11 14:29:12 2024] svc_process+0x12d/0x170 [sunrpc]
[Mon Mar 11 14:29:12 2024] nfsd+0x84/0xb0 [nfsd]
[Mon Mar 11 14:29:12 2024] kthread+0xdd/0x100
[Mon Mar 11 14:29:12 2024] ? __pfx_kthread+0x10/0x10
[Mon Mar 11 14:29:12 2024] ret_from_fork+0x29/0x50
[Mon Mar 11 14:29:12 2024] </TASK>
[Mon Mar 11 14:29:12 2024] task:nfsd state:D stack:0 pid:8845 ppid:2 flags:0x00004000
[Mon Mar 11 14:29:12 2024] Call Trace:
[Mon Mar 11 14:29:12 2024] <TASK>
[Mon Mar 11 14:29:12 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:12 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:12 2024] schedule_timeout+0x11f/0x160
[Mon Mar 11 14:29:12 2024] ? select_idle_sibling+0x28/0x430
[Mon Mar 11 14:29:12 2024] ? wake_affine+0x62/0x1f0
[Mon Mar 11 14:29:12 2024] __wait_for_common+0x90/0x1d0
[Mon Mar 11 14:29:12 2024] ? __pfx_schedule_timeout+0x10/0x10
[Mon Mar 11 14:29:12 2024] __flush_workqueue+0x13a/0x3f0
[Mon Mar 11 14:29:12 2024] nfsd4_shutdown_callback+0x49/0x120 [nfsd]
[Mon Mar 11 14:29:12 2024] ? nfsd4_cld_remove+0x54/0x1d0 [nfsd]
[Mon Mar 11 14:29:12 2024] ? nfsd4_return_all_client_layouts+0xc4/0xf0 [nfsd]
[Mon Mar 11 14:29:12 2024] ? nfsd4_shutdown_copy+0x68/0xc0 [nfsd]
[Mon Mar 11 14:29:12 2024] __destroy_client+0x1f3/0x290 [nfsd]
[Mon Mar 11 14:29:12 2024] nfsd4_exchange_id+0x75f/0x770 [nfsd]
[Mon Mar 11 14:29:12 2024] ? nfsd4_decode_opaque+0x3a/0x90 [nfsd]
[Mon Mar 11 14:29:12 2024] nfsd4_proc_compound+0x44b/0x700 [nfsd]
[Mon Mar 11 14:29:12 2024] nfsd_dispatch+0x94/0x1c0 [nfsd]
[Mon Mar 11 14:29:12 2024] svc_process_common+0x2ec/0x660 [sunrpc]
[Mon Mar 11 14:29:12 2024] ? __pfx_nfsd_dispatch+0x10/0x10 [nfsd]
[Mon Mar 11 14:29:12 2024] ? __pfx_nfsd+0x10/0x10 [nfsd]
[Mon Mar 11 14:29:12 2024] svc_process+0x12d/0x170 [sunrpc]
[Mon Mar 11 14:29:12 2024] nfsd+0x84/0xb0 [nfsd]
[Mon Mar 11 14:29:12 2024] kthread+0xdd/0x100
[Mon Mar 11 14:29:12 2024] ? __pfx_kthread+0x10/0x10
[Mon Mar 11 14:29:12 2024] ret_from_fork+0x29/0x50
[Mon Mar 11 14:29:12 2024] </TASK>
[Mon Mar 11 14:29:12 2024] task:nfsd state:D stack:0 pid:8846 ppid:2 flags:0x00004000
[Mon Mar 11 14:29:12 2024] Call Trace:
[Mon Mar 11 14:29:12 2024] <TASK>
[Mon Mar 11 14:29:12 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:12 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:12 2024] schedule_timeout+0x11f/0x160
[Mon Mar 11 14:29:12 2024] ? select_idle_sibling+0x28/0x430
[Mon Mar 11 14:29:12 2024] ? wake_affine+0x62/0x1f0
[Mon Mar 11 14:29:12 2024] __wait_for_common+0x90/0x1d0
[Mon Mar 11 14:29:12 2024] ? __pfx_schedule_timeout+0x10/0x10
[Mon Mar 11 14:29:12 2024] __flush_workqueue+0x13a/0x3f0
[Mon Mar 11 14:29:12 2024] nfsd4_shutdown_callback+0x49/0x120 [nfsd]
[Mon Mar 11 14:29:12 2024] ? nfsd4_cld_remove+0x54/0x1d0 [nfsd]
[Mon Mar 11 14:29:12 2024] ? nfsd4_return_all_client_layouts+0xc4/0xf0 [nfsd]
[Mon Mar 11 14:29:12 2024] ? nfsd4_shutdown_copy+0x68/0xc0 [nfsd]
[Mon Mar 11 14:29:12 2024] __destroy_client+0x1f3/0x290 [nfsd]
[Mon Mar 11 14:29:12 2024] nfsd4_exchange_id+0x75f/0x770 [nfsd]
[Mon Mar 11 14:29:12 2024] ? nfsd4_decode_opaque+0x3a/0x90 [nfsd]
[Mon Mar 11 14:29:12 2024] nfsd4_proc_compound+0x44b/0x700 [nfsd]
[Mon Mar 11 14:29:12 2024] nfsd_dispatch+0x94/0x1c0 [nfsd]
[Mon Mar 11 14:29:12 2024] svc_process_common+0x2ec/0x660 [sunrpc]
[Mon Mar 11 14:29:12 2024] ? __pfx_nfsd_dispatch+0x10/0x10 [nfsd]
[Mon Mar 11 14:29:12 2024] ? __pfx_nfsd+0x10/0x10 [nfsd]
[Mon Mar 11 14:29:12 2024] svc_process+0x12d/0x170 [sunrpc]
[Mon Mar 11 14:29:12 2024] nfsd+0x84/0xb0 [nfsd]
[Mon Mar 11 14:29:12 2024] kthread+0xdd/0x100
[Mon Mar 11 14:29:12 2024] ? __pfx_kthread+0x10/0x10
[Mon Mar 11 14:29:12 2024] ret_from_fork+0x29/0x50
[Mon Mar 11 14:29:12 2024] </TASK>
[Mon Mar 11 14:29:12 2024] task:nfsd state:D stack:0 pid:8847 ppid:2 flags:0x00004000
[Mon Mar 11 14:29:12 2024] Call Trace:
[Mon Mar 11 14:29:12 2024] <TASK>
[Mon Mar 11 14:29:12 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:12 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:12 2024] schedule_timeout+0x11f/0x160
[Mon Mar 11 14:29:12 2024] ? select_idle_sibling+0x28/0x430
[Mon Mar 11 14:29:12 2024] ? tcp_recvmsg+0x196/0x210
[Mon Mar 11 14:29:12 2024] ? wake_affine+0x62/0x1f0
[Mon Mar 11 14:29:12 2024] __wait_for_common+0x90/0x1d0
[Mon Mar 11 14:29:12 2024] ? __pfx_schedule_timeout+0x10/0x10
[Mon Mar 11 14:29:12 2024] __flush_workqueue+0x13a/0x3f0
[Mon Mar 11 14:29:12 2024] nfsd4_destroy_session+0x1a4/0x240 [nfsd]
[Mon Mar 11 14:29:12 2024] nfsd4_proc_compound+0x44b/0x700 [nfsd]
[Mon Mar 11 14:29:12 2024] nfsd_dispatch+0x94/0x1c0 [nfsd]
[Mon Mar 11 14:29:12 2024] svc_process_common+0x2ec/0x660 [sunrpc]
[Mon Mar 11 14:29:12 2024] ? __pfx_nfsd_dispatch+0x10/0x10 [nfsd]
[Mon Mar 11 14:29:12 2024] ? __pfx_nfsd+0x10/0x10 [nfsd]
[Mon Mar 11 14:29:12 2024] svc_process+0x12d/0x170 [sunrpc]
[Mon Mar 11 14:29:12 2024] nfsd+0x84/0xb0 [nfsd]
[Mon Mar 11 14:29:12 2024] kthread+0xdd/0x100
[Mon Mar 11 14:29:12 2024] ? __pfx_kthread+0x10/0x10
[Mon Mar 11 14:29:12 2024] ret_from_fork+0x29/0x50
[Mon Mar 11 14:29:12 2024] </TASK>
[Mon Mar 11 14:29:12 2024] task:nfsd state:D stack:0 pid:8848 ppid:2 flags:0x00004000
[Mon Mar 11 14:29:12 2024] Call Trace:
[Mon Mar 11 14:29:12 2024] <TASK>
[Mon Mar 11 14:29:12 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:12 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:12 2024] schedule_timeout+0x11f/0x160
[Mon Mar 11 14:29:12 2024] ? select_idle_sibling+0x28/0x430
[Mon Mar 11 14:29:12 2024] ? tcp_recvmsg+0x196/0x210
[Mon Mar 11 14:29:12 2024] ? wake_affine+0x62/0x1f0
[Mon Mar 11 14:29:12 2024] __wait_for_common+0x90/0x1d0
[Mon Mar 11 14:29:12 2024] ? __pfx_schedule_timeout+0x10/0x10
[Mon Mar 11 14:29:12 2024] __flush_workqueue+0x13a/0x3f0
[Mon Mar 11 14:29:12 2024] nfsd4_destroy_session+0x1a4/0x240 [nfsd]
[Mon Mar 11 14:29:12 2024] nfsd4_proc_compound+0x44b/0x700 [nfsd]
[Mon Mar 11 14:29:12 2024] nfsd_dispatch+0x94/0x1c0 [nfsd]
[Mon Mar 11 14:29:12 2024] svc_process_common+0x2ec/0x660 [sunrpc]
[Mon Mar 11 14:29:12 2024] ? __pfx_nfsd_dispatch+0x10/0x10 [nfsd]
[Mon Mar 11 14:29:12 2024] ? __pfx_nfsd+0x10/0x10 [nfsd]
[Mon Mar 11 14:29:12 2024] svc_process+0x12d/0x170 [sunrpc]
[Mon Mar 11 14:29:12 2024] nfsd+0x84/0xb0 [nfsd]
[Mon Mar 11 14:29:12 2024] kthread+0xdd/0x100
[Mon Mar 11 14:29:12 2024] ? __pfx_kthread+0x10/0x10
[Mon Mar 11 14:29:12 2024] ret_from_fork+0x29/0x50
[Mon Mar 11 14:29:12 2024] </TASK>
[Mon Mar 11 14:29:12 2024] task:nfsd state:D stack:0 pid:8849 ppid:2 flags:0x00004000
[Mon Mar 11 14:29:12 2024] Call Trace:
[Mon Mar 11 14:29:12 2024] <TASK>
[Mon Mar 11 14:29:12 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:12 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:12 2024] schedule_timeout+0x11f/0x160
[Mon Mar 11 14:29:12 2024] ? select_idle_sibling+0x28/0x430
[Mon Mar 11 14:29:12 2024] ? wake_affine+0x62/0x1f0
[Mon Mar 11 14:29:12 2024] __wait_for_common+0x90/0x1d0
[Mon Mar 11 14:29:12 2024] ? __pfx_schedule_timeout+0x10/0x10
[Mon Mar 11 14:29:12 2024] __flush_workqueue+0x13a/0x3f0
[Mon Mar 11 14:29:12 2024] nfsd4_shutdown_callback+0x49/0x120 [nfsd]
[Mon Mar 11 14:29:12 2024] ? nfsd4_cld_remove+0x54/0x1d0 [nfsd]
[Mon Mar 11 14:29:12 2024] ? nfsd4_return_all_client_layouts+0xc4/0xf0 [nfsd]
[Mon Mar 11 14:29:12 2024] ? nfsd4_shutdown_copy+0x68/0xc0 [nfsd]
[Mon Mar 11 14:29:12 2024] __destroy_client+0x1f3/0x290 [nfsd]
[Mon Mar 11 14:29:12 2024] nfsd4_exchange_id+0x75f/0x770 [nfsd]
[Mon Mar 11 14:29:12 2024] ? nfsd4_decode_opaque+0x3a/0x90 [nfsd]
[Mon Mar 11 14:29:12 2024] nfsd4_proc_compound+0x44b/0x700 [nfsd]
[Mon Mar 11 14:29:12 2024] nfsd_dispatch+0x94/0x1c0 [nfsd]
[Mon Mar 11 14:29:12 2024] svc_process_common+0x2ec/0x660 [sunrpc]
[Mon Mar 11 14:29:12 2024] ? __pfx_nfsd_dispatch+0x10/0x10 [nfsd]
[Mon Mar 11 14:29:12 2024] ? __pfx_nfsd+0x10/0x10 [nfsd]
[Mon Mar 11 14:29:12 2024] svc_process+0x12d/0x170 [sunrpc]
[Mon Mar 11 14:29:12 2024] nfsd+0x84/0xb0 [nfsd]
[Mon Mar 11 14:29:12 2024] kthread+0xdd/0x100
[Mon Mar 11 14:29:12 2024] ? __pfx_kthread+0x10/0x10
[Mon Mar 11 14:29:12 2024] ret_from_fork+0x29/0x50
[Mon Mar 11 14:29:12 2024] </TASK>
[Mon Mar 11 14:29:12 2024] task:nfsd state:D stack:0 pid:8850 ppid:2 flags:0x00004000
[Mon Mar 11 14:29:12 2024] Call Trace:
[Mon Mar 11 14:29:12 2024] <TASK>
[Mon Mar 11 14:29:12 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:12 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:12 2024] schedule_timeout+0x11f/0x160
[Mon Mar 11 14:29:12 2024] ? select_idle_sibling+0x28/0x430
[Mon Mar 11 14:29:12 2024] ? tcp_recvmsg+0x196/0x210
[Mon Mar 11 14:29:12 2024] ? wake_affine+0x62/0x1f0
[Mon Mar 11 14:29:12 2024] __wait_for_common+0x90/0x1d0
[Mon Mar 11 14:29:12 2024] ? __pfx_schedule_timeout+0x10/0x10
[Mon Mar 11 14:29:12 2024] __flush_workqueue+0x13a/0x3f0
[Mon Mar 11 14:29:12 2024] nfsd4_destroy_session+0x1a4/0x240 [nfsd]
[Mon Mar 11 14:29:12 2024] nfsd4_proc_compound+0x44b/0x700 [nfsd]
[Mon Mar 11 14:29:12 2024] nfsd_dispatch+0x94/0x1c0 [nfsd]
[Mon Mar 11 14:29:12 2024] svc_process_common+0x2ec/0x660 [sunrpc]
[Mon Mar 11 14:29:12 2024] ? __pfx_nfsd_dispatch+0x10/0x10 [nfsd]
[Mon Mar 11 14:29:12 2024] ? __pfx_nfsd+0x10/0x10 [nfsd]
[Mon Mar 11 14:29:12 2024] svc_process+0x12d/0x170 [sunrpc]
[Mon Mar 11 14:29:12 2024] nfsd+0x84/0xb0 [nfsd]
[Mon Mar 11 14:29:12 2024] kthread+0xdd/0x100
[Mon Mar 11 14:29:12 2024] ? __pfx_kthread+0x10/0x10
[Mon Mar 11 14:29:12 2024] ret_from_fork+0x29/0x50
[Mon Mar 11 14:29:12 2024] </TASK>
[Mon Mar 11 14:29:12 2024] task:nfsd state:D stack:0 pid:8851 ppid:2 flags:0x00004000
[Mon Mar 11 14:29:12 2024] Call Trace:
[Mon Mar 11 14:29:12 2024] <TASK>
[Mon Mar 11 14:29:12 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:12 2024] ? allocate_slab+0x24e/0x490
[Mon Mar 11 14:29:12 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:12 2024] schedule_timeout+0x11f/0x160
[Mon Mar 11 14:29:12 2024] ? select_idle_sibling+0x28/0x430
[Mon Mar 11 14:29:12 2024] ? wake_affine+0x62/0x1f0
[Mon Mar 11 14:29:12 2024] __wait_for_common+0x90/0x1d0
[Mon Mar 11 14:29:12 2024] ? __pfx_schedule_timeout+0x10/0x10
[Mon Mar 11 14:29:12 2024] __flush_workqueue+0x13a/0x3f0
[Mon Mar 11 14:29:12 2024] nfsd4_create_session+0x8c1/0xb30 [nfsd]
[Mon Mar 11 14:29:12 2024] ? nfsd4_proc_compound+0x44b/0x700 [nfsd]
[Mon Mar 11 14:29:12 2024] nfsd4_proc_compound+0x44b/0x700 [nfsd]
[Mon Mar 11 14:29:12 2024] nfsd_dispatch+0x94/0x1c0 [nfsd]
[Mon Mar 11 14:29:12 2024] svc_process_common+0x2ec/0x660 [sunrpc]
[Mon Mar 11 14:29:12 2024] ? __pfx_nfsd_dispatch+0x10/0x10 [nfsd]
[Mon Mar 11 14:29:12 2024] ? __pfx_nfsd+0x10/0x10 [nfsd]
[Mon Mar 11 14:29:12 2024] svc_process+0x12d/0x170 [sunrpc]
[Mon Mar 11 14:29:12 2024] nfsd+0x84/0xb0 [nfsd]
[Mon Mar 11 14:29:12 2024] kthread+0xdd/0x100
[Mon Mar 11 14:29:12 2024] ? __pfx_kthread+0x10/0x10
[Mon Mar 11 14:29:12 2024] ret_from_fork+0x29/0x50
[Mon Mar 11 14:29:12 2024] </TASK>
[Mon Mar 11 14:29:12 2024] task:nfsd state:D stack:0 pid:8852 ppid:2 flags:0x00004000
[Mon Mar 11 14:29:12 2024] Call Trace:
[Mon Mar 11 14:29:12 2024] <TASK>
[Mon Mar 11 14:29:12 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:12 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:12 2024] schedule_timeout+0x11f/0x160
[Mon Mar 11 14:29:12 2024] ? select_idle_sibling+0x28/0x430
[Mon Mar 11 14:29:12 2024] ? wake_affine+0x62/0x1f0
[Mon Mar 11 14:29:12 2024] __wait_for_common+0x90/0x1d0
[Mon Mar 11 14:29:12 2024] ? __pfx_schedule_timeout+0x10/0x10
[Mon Mar 11 14:29:12 2024] __flush_workqueue+0x13a/0x3f0
[Mon Mar 11 14:29:12 2024] nfsd4_shutdown_callback+0x49/0x120 [nfsd]
[Mon Mar 11 14:29:12 2024] ? nfsd4_cld_remove+0x54/0x1d0 [nfsd]
[Mon Mar 11 14:29:12 2024] ? nfsd4_return_all_client_layouts+0xc4/0xf0 [nfsd]
[Mon Mar 11 14:29:12 2024] ? nfsd4_shutdown_copy+0x68/0xc0 [nfsd]
[Mon Mar 11 14:29:12 2024] __destroy_client+0x1f3/0x290 [nfsd]
[Mon Mar 11 14:29:12 2024] nfsd4_exchange_id+0x75f/0x770 [nfsd]
[Mon Mar 11 14:29:12 2024] ? nfsd4_decode_opaque+0x3a/0x90 [nfsd]
[Mon Mar 11 14:29:12 2024] nfsd4_proc_compound+0x44b/0x700 [nfsd]
[Mon Mar 11 14:29:12 2024] nfsd_dispatch+0x94/0x1c0 [nfsd]
[Mon Mar 11 14:29:12 2024] svc_process_common+0x2ec/0x660 [sunrpc]
[Mon Mar 11 14:29:12 2024] ? __pfx_nfsd_dispatch+0x10/0x10 [nfsd]
[Mon Mar 11 14:29:12 2024] ? __pfx_nfsd+0x10/0x10 [nfsd]
[Mon Mar 11 14:29:12 2024] svc_process+0x12d/0x170 [sunrpc]
[Mon Mar 11 14:29:12 2024] nfsd+0x84/0xb0 [nfsd]
[Mon Mar 11 14:29:12 2024] kthread+0xdd/0x100
[Mon Mar 11 14:29:12 2024] ? __pfx_kthread+0x10/0x10
[Mon Mar 11 14:29:12 2024] ret_from_fork+0x29/0x50
[Mon Mar 11 14:29:12 2024] </TASK>
[Mon Mar 11 14:29:12 2024] task:nfsd state:D stack:0 pid:8853 ppid:2 flags:0x00004000
[Mon Mar 11 14:29:12 2024] Call Trace:
[Mon Mar 11 14:29:12 2024] <TASK>
[Mon Mar 11 14:29:12 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:12 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:12 2024] schedule_timeout+0x11f/0x160
[Mon Mar 11 14:29:12 2024] ? select_idle_sibling+0x28/0x430
[Mon Mar 11 14:29:12 2024] ? tcp_recvmsg+0x196/0x210
[Mon Mar 11 14:29:12 2024] __wait_for_common+0x90/0x1d0
[Mon Mar 11 14:29:12 2024] ? __pfx_schedule_timeout+0x10/0x10
[Mon Mar 11 14:29:12 2024] __flush_workqueue+0x13a/0x3f0
[Mon Mar 11 14:29:12 2024] nfsd4_destroy_session+0x1a4/0x240 [nfsd]
[Mon Mar 11 14:29:12 2024] nfsd4_proc_compound+0x44b/0x700 [nfsd]
[Mon Mar 11 14:29:12 2024] nfsd_dispatch+0x94/0x1c0 [nfsd]
[Mon Mar 11 14:29:12 2024] svc_process_common+0x2ec/0x660 [sunrpc]
[Mon Mar 11 14:29:12 2024] ? __pfx_nfsd_dispatch+0x10/0x10 [nfsd]
[Mon Mar 11 14:29:12 2024] ? __pfx_nfsd+0x10/0x10 [nfsd]
[Mon Mar 11 14:29:12 2024] svc_process+0x12d/0x170 [sunrpc]
[Mon Mar 11 14:29:12 2024] nfsd+0x84/0xb0 [nfsd]
[Mon Mar 11 14:29:12 2024] kthread+0xdd/0x100
[Mon Mar 11 14:29:12 2024] ? __pfx_kthread+0x10/0x10
[Mon Mar 11 14:29:12 2024] ret_from_fork+0x29/0x50
[Mon Mar 11 14:29:12 2024] </TASK>
[Mon Mar 11 14:29:12 2024] task:nfsd state:D stack:0 pid:8854 ppid:2 flags:0x00004000
[Mon Mar 11 14:29:12 2024] Call Trace:
[Mon Mar 11 14:29:12 2024] <TASK>
[Mon Mar 11 14:29:12 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:12 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:12 2024] schedule_timeout+0x11f/0x160
[Mon Mar 11 14:29:12 2024] ? select_idle_sibling+0x28/0x430
[Mon Mar 11 14:29:12 2024] ? tcp_recvmsg+0x196/0x210
[Mon Mar 11 14:29:12 2024] ? wake_affine+0x62/0x1f0
[Mon Mar 11 14:29:12 2024] __wait_for_common+0x90/0x1d0
[Mon Mar 11 14:29:12 2024] ? __pfx_schedule_timeout+0x10/0x10
[Mon Mar 11 14:29:12 2024] __flush_workqueue+0x13a/0x3f0
[Mon Mar 11 14:29:12 2024] nfsd4_destroy_session+0x1a4/0x240 [nfsd]
[Mon Mar 11 14:29:12 2024] nfsd4_proc_compound+0x44b/0x700 [nfsd]
[Mon Mar 11 14:29:12 2024] nfsd_dispatch+0x94/0x1c0 [nfsd]
[Mon Mar 11 14:29:12 2024] svc_process_common+0x2ec/0x660 [sunrpc]
[Mon Mar 11 14:29:12 2024] ? __pfx_nfsd_dispatch+0x10/0x10 [nfsd]
[Mon Mar 11 14:29:12 2024] ? __pfx_nfsd+0x10/0x10 [nfsd]
[Mon Mar 11 14:29:12 2024] svc_process+0x12d/0x170 [sunrpc]
[Mon Mar 11 14:29:12 2024] nfsd+0x84/0xb0 [nfsd]
[Mon Mar 11 14:29:12 2024] kthread+0xdd/0x100
[Mon Mar 11 14:29:12 2024] ? __pfx_kthread+0x10/0x10
[Mon Mar 11 14:29:12 2024] ret_from_fork+0x29/0x50
[Mon Mar 11 14:29:12 2024] </TASK>
[Mon Mar 11 14:29:12 2024] task:nfsd state:D stack:0 pid:8855 ppid:2 flags:0x00004000
[Mon Mar 11 14:29:12 2024] Call Trace:
[Mon Mar 11 14:29:12 2024] <TASK>
[Mon Mar 11 14:29:12 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:12 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:12 2024] schedule_timeout+0x11f/0x160
[Mon Mar 11 14:29:12 2024] ? select_idle_sibling+0x28/0x430
[Mon Mar 11 14:29:12 2024] ? tcp_recvmsg+0x196/0x210
[Mon Mar 11 14:29:12 2024] ? wake_affine+0x62/0x1f0
[Mon Mar 11 14:29:12 2024] __wait_for_common+0x90/0x1d0
[Mon Mar 11 14:29:12 2024] ? __pfx_schedule_timeout+0x10/0x10
[Mon Mar 11 14:29:12 2024] __flush_workqueue+0x13a/0x3f0
[Mon Mar 11 14:29:12 2024] nfsd4_destroy_session+0x1a4/0x240 [nfsd]
[Mon Mar 11 14:29:12 2024] nfsd4_proc_compound+0x44b/0x700 [nfsd]
[Mon Mar 11 14:29:12 2024] nfsd_dispatch+0x94/0x1c0 [nfsd]
[Mon Mar 11 14:29:12 2024] svc_process_common+0x2ec/0x660 [sunrpc]
[Mon Mar 11 14:29:12 2024] ? __pfx_nfsd_dispatch+0x10/0x10 [nfsd]
[Mon Mar 11 14:29:12 2024] ? __pfx_nfsd+0x10/0x10 [nfsd]
[Mon Mar 11 14:29:12 2024] svc_process+0x12d/0x170 [sunrpc]
[Mon Mar 11 14:29:12 2024] nfsd+0x84/0xb0 [nfsd]
[Mon Mar 11 14:29:12 2024] kthread+0xdd/0x100
[Mon Mar 11 14:29:12 2024] ? __pfx_kthread+0x10/0x10
[Mon Mar 11 14:29:12 2024] ret_from_fork+0x29/0x50
[Mon Mar 11 14:29:12 2024] </TASK>
[Mon Mar 11 14:29:12 2024] task:nfsd state:D stack:0 pid:8856 ppid:2 flags:0x00004000
[Mon Mar 11 14:29:12 2024] Call Trace:
[Mon Mar 11 14:29:12 2024] <TASK>
[Mon Mar 11 14:29:12 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:12 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:12 2024] schedule_timeout+0x11f/0x160
[Mon Mar 11 14:29:12 2024] ? select_idle_sibling+0x28/0x430
[Mon Mar 11 14:29:12 2024] ? tcp_recvmsg+0x196/0x210
[Mon Mar 11 14:29:12 2024] ? wake_affine+0x62/0x1f0
[Mon Mar 11 14:29:12 2024] __wait_for_common+0x90/0x1d0
[Mon Mar 11 14:29:12 2024] ? __pfx_schedule_timeout+0x10/0x10
[Mon Mar 11 14:29:12 2024] __flush_workqueue+0x13a/0x3f0
[Mon Mar 11 14:29:12 2024] nfsd4_destroy_session+0x1a4/0x240 [nfsd]
[Mon Mar 11 14:29:12 2024] nfsd4_proc_compound+0x44b/0x700 [nfsd]
[Mon Mar 11 14:29:12 2024] nfsd_dispatch+0x94/0x1c0 [nfsd]
[Mon Mar 11 14:29:12 2024] svc_process_common+0x2ec/0x660 [sunrpc]
[Mon Mar 11 14:29:12 2024] ? __pfx_nfsd_dispatch+0x10/0x10 [nfsd]
[Mon Mar 11 14:29:12 2024] ? __pfx_nfsd+0x10/0x10 [nfsd]
[Mon Mar 11 14:29:12 2024] svc_process+0x12d/0x170 [sunrpc]
[Mon Mar 11 14:29:12 2024] nfsd+0x84/0xb0 [nfsd]
[Mon Mar 11 14:29:12 2024] kthread+0xdd/0x100
[Mon Mar 11 14:29:12 2024] ? __pfx_kthread+0x10/0x10
[Mon Mar 11 14:29:12 2024] ret_from_fork+0x29/0x50
[Mon Mar 11 14:29:12 2024] </TASK>
[Mon Mar 11 14:29:12 2024] task:nfsd state:D stack:0 pid:8857 ppid:2 flags:0x00004000
[Mon Mar 11 14:29:12 2024] Call Trace:
[Mon Mar 11 14:29:12 2024] <TASK>
[Mon Mar 11 14:29:12 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:12 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:12 2024] schedule_timeout+0x11f/0x160
[Mon Mar 11 14:29:12 2024] ? select_idle_sibling+0x28/0x430
[Mon Mar 11 14:29:12 2024] ? tcp_recvmsg+0x196/0x210
[Mon Mar 11 14:29:12 2024] ? wake_affine+0x62/0x1f0
[Mon Mar 11 14:29:12 2024] __wait_for_common+0x90/0x1d0
[Mon Mar 11 14:29:12 2024] ? __pfx_schedule_timeout+0x10/0x10
[Mon Mar 11 14:29:12 2024] __flush_workqueue+0x13a/0x3f0
[Mon Mar 11 14:29:12 2024] nfsd4_destroy_session+0x1a4/0x240 [nfsd]
[Mon Mar 11 14:29:12 2024] nfsd4_proc_compound+0x44b/0x700 [nfsd]
[Mon Mar 11 14:29:12 2024] nfsd_dispatch+0x94/0x1c0 [nfsd]
[Mon Mar 11 14:29:12 2024] svc_process_common+0x2ec/0x660 [sunrpc]
[Mon Mar 11 14:29:12 2024] ? __pfx_nfsd_dispatch+0x10/0x10 [nfsd]
[Mon Mar 11 14:29:12 2024] ? __pfx_nfsd+0x10/0x10 [nfsd]
[Mon Mar 11 14:29:12 2024] svc_process+0x12d/0x170 [sunrpc]
[Mon Mar 11 14:29:12 2024] nfsd+0x84/0xb0 [nfsd]
[Mon Mar 11 14:29:12 2024] kthread+0xdd/0x100
[Mon Mar 11 14:29:12 2024] ? __pfx_kthread+0x10/0x10
[Mon Mar 11 14:29:12 2024] ret_from_fork+0x29/0x50
[Mon Mar 11 14:29:12 2024] </TASK>
[Mon Mar 11 14:29:12 2024] task:nfsd state:D stack:0 pid:8858 ppid:2 flags:0x00004000
[Mon Mar 11 14:29:12 2024] Call Trace:
[Mon Mar 11 14:29:12 2024] <TASK>
[Mon Mar 11 14:29:12 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:12 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:12 2024] schedule_timeout+0x11f/0x160
[Mon Mar 11 14:29:12 2024] ? select_idle_sibling+0x28/0x430
[Mon Mar 11 14:29:12 2024] ? tcp_recvmsg+0x196/0x210
[Mon Mar 11 14:29:12 2024] ? wake_affine+0x62/0x1f0
[Mon Mar 11 14:29:12 2024] __wait_for_common+0x90/0x1d0
[Mon Mar 11 14:29:12 2024] ? __pfx_schedule_timeout+0x10/0x10
[Mon Mar 11 14:29:12 2024] __flush_workqueue+0x13a/0x3f0
[Mon Mar 11 14:29:12 2024] nfsd4_destroy_session+0x1a4/0x240 [nfsd]
[Mon Mar 11 14:29:12 2024] nfsd4_proc_compound+0x44b/0x700 [nfsd]
[Mon Mar 11 14:29:12 2024] nfsd_dispatch+0x94/0x1c0 [nfsd]
[Mon Mar 11 14:29:12 2024] svc_process_common+0x2ec/0x660 [sunrpc]
[Mon Mar 11 14:29:12 2024] ? __pfx_nfsd_dispatch+0x10/0x10 [nfsd]
[Mon Mar 11 14:29:12 2024] ? __pfx_nfsd+0x10/0x10 [nfsd]
[Mon Mar 11 14:29:12 2024] svc_process+0x12d/0x170 [sunrpc]
[Mon Mar 11 14:29:12 2024] nfsd+0x84/0xb0 [nfsd]
[Mon Mar 11 14:29:12 2024] kthread+0xdd/0x100
[Mon Mar 11 14:29:12 2024] ? __pfx_kthread+0x10/0x10
[Mon Mar 11 14:29:12 2024] ret_from_fork+0x29/0x50
[Mon Mar 11 14:29:12 2024] </TASK>
[Mon Mar 11 14:29:12 2024] task:nfsd state:D stack:0 pid:8859 ppid:2 flags:0x00004000
[Mon Mar 11 14:29:12 2024] Call Trace:
[Mon Mar 11 14:29:12 2024] <TASK>
[Mon Mar 11 14:29:12 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:12 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:12 2024] schedule_timeout+0x11f/0x160
[Mon Mar 11 14:29:12 2024] ? select_idle_sibling+0x28/0x430
[Mon Mar 11 14:29:12 2024] ? tcp_recvmsg+0x196/0x210
[Mon Mar 11 14:29:12 2024] __wait_for_common+0x90/0x1d0
[Mon Mar 11 14:29:12 2024] ? __pfx_schedule_timeout+0x10/0x10
[Mon Mar 11 14:29:12 2024] __flush_workqueue+0x13a/0x3f0
[Mon Mar 11 14:29:12 2024] nfsd4_destroy_session+0x1a4/0x240 [nfsd]
[Mon Mar 11 14:29:12 2024] nfsd4_proc_compound+0x44b/0x700 [nfsd]
[Mon Mar 11 14:29:12 2024] nfsd_dispatch+0x94/0x1c0 [nfsd]
[Mon Mar 11 14:29:12 2024] svc_process_common+0x2ec/0x660 [sunrpc]
[Mon Mar 11 14:29:12 2024] ? __pfx_nfsd_dispatch+0x10/0x10 [nfsd]
[Mon Mar 11 14:29:12 2024] ? __pfx_nfsd+0x10/0x10 [nfsd]
[Mon Mar 11 14:29:12 2024] svc_process+0x12d/0x170 [sunrpc]
[Mon Mar 11 14:29:12 2024] nfsd+0x84/0xb0 [nfsd]
[Mon Mar 11 14:29:12 2024] kthread+0xdd/0x100
[Mon Mar 11 14:29:12 2024] ? __pfx_kthread+0x10/0x10
[Mon Mar 11 14:29:12 2024] ret_from_fork+0x29/0x50
[Mon Mar 11 14:29:12 2024] </TASK>
[Mon Mar 11 14:29:12 2024] task:nfsd state:D stack:0 pid:8860 ppid:2 flags:0x00004000
[Mon Mar 11 14:29:12 2024] Call Trace:
[Mon Mar 11 14:29:12 2024] <TASK>
[Mon Mar 11 14:29:12 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:12 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:12 2024] schedule_timeout+0x11f/0x160
[Mon Mar 11 14:29:12 2024] ? select_idle_sibling+0x28/0x430
[Mon Mar 11 14:29:12 2024] ? tcp_recvmsg+0x196/0x210
[Mon Mar 11 14:29:12 2024] ? wake_affine+0x62/0x1f0
[Mon Mar 11 14:29:12 2024] __wait_for_common+0x90/0x1d0
[Mon Mar 11 14:29:12 2024] ? __pfx_schedule_timeout+0x10/0x10
[Mon Mar 11 14:29:12 2024] __flush_workqueue+0x13a/0x3f0
[Mon Mar 11 14:29:12 2024] nfsd4_destroy_session+0x1a4/0x240 [nfsd]
[Mon Mar 11 14:29:12 2024] nfsd4_proc_compound+0x44b/0x700 [nfsd]
[Mon Mar 11 14:29:12 2024] nfsd_dispatch+0x94/0x1c0 [nfsd]
[Mon Mar 11 14:29:12 2024] svc_process_common+0x2ec/0x660 [sunrpc]
[Mon Mar 11 14:29:12 2024] ? __pfx_nfsd_dispatch+0x10/0x10 [nfsd]
[Mon Mar 11 14:29:12 2024] ? __pfx_nfsd+0x10/0x10 [nfsd]
[Mon Mar 11 14:29:12 2024] svc_process+0x12d/0x170 [sunrpc]
[Mon Mar 11 14:29:12 2024] nfsd+0x84/0xb0 [nfsd]
[Mon Mar 11 14:29:12 2024] kthread+0xdd/0x100
[Mon Mar 11 14:29:12 2024] ? __pfx_kthread+0x10/0x10
[Mon Mar 11 14:29:12 2024] ret_from_fork+0x29/0x50
[Mon Mar 11 14:29:12 2024] </TASK>
[Mon Mar 11 14:29:12 2024] task:nfsd state:D stack:0 pid:8861 ppid:2 flags:0x00004000
[Mon Mar 11 14:29:12 2024] Call Trace:
[Mon Mar 11 14:29:12 2024] <TASK>
[Mon Mar 11 14:29:12 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:12 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:12 2024] schedule_timeout+0x11f/0x160
[Mon Mar 11 14:29:12 2024] ? select_idle_sibling+0x28/0x430
[Mon Mar 11 14:29:12 2024] ? tcp_recvmsg+0x196/0x210
[Mon Mar 11 14:29:12 2024] ? wake_affine+0x62/0x1f0
[Mon Mar 11 14:29:12 2024] __wait_for_common+0x90/0x1d0
[Mon Mar 11 14:29:12 2024] ? __pfx_schedule_timeout+0x10/0x10
[Mon Mar 11 14:29:12 2024] __flush_workqueue+0x13a/0x3f0
[Mon Mar 11 14:29:12 2024] nfsd4_destroy_session+0x1a4/0x240 [nfsd]
[Mon Mar 11 14:29:12 2024] nfsd4_proc_compound+0x44b/0x700 [nfsd]
[Mon Mar 11 14:29:12 2024] nfsd_dispatch+0x94/0x1c0 [nfsd]
[Mon Mar 11 14:29:12 2024] svc_process_common+0x2ec/0x660 [sunrpc]
[Mon Mar 11 14:29:12 2024] ? __pfx_nfsd_dispatch+0x10/0x10 [nfsd]
[Mon Mar 11 14:29:12 2024] ? __pfx_nfsd+0x10/0x10 [nfsd]
[Mon Mar 11 14:29:12 2024] svc_process+0x12d/0x170 [sunrpc]
[Mon Mar 11 14:29:12 2024] nfsd+0x84/0xb0 [nfsd]
[Mon Mar 11 14:29:12 2024] kthread+0xdd/0x100
[Mon Mar 11 14:29:12 2024] ? __pfx_kthread+0x10/0x10
[Mon Mar 11 14:29:12 2024] ret_from_fork+0x29/0x50
[Mon Mar 11 14:29:12 2024] </TASK>
[Mon Mar 11 14:29:12 2024] task:nfsd state:D stack:0 pid:8862 ppid:2 flags:0x00004000
[Mon Mar 11 14:29:12 2024] Call Trace:
[Mon Mar 11 14:29:12 2024] <TASK>
[Mon Mar 11 14:29:12 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:12 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:12 2024] schedule_timeout+0x11f/0x160
[Mon Mar 11 14:29:12 2024] ? select_idle_sibling+0x28/0x430
[Mon Mar 11 14:29:12 2024] __wait_for_common+0x90/0x1d0
[Mon Mar 11 14:29:12 2024] ? __pfx_schedule_timeout+0x10/0x10
[Mon Mar 11 14:29:12 2024] __flush_workqueue+0x13a/0x3f0
[Mon Mar 11 14:29:12 2024] nfsd4_shutdown_callback+0x49/0x120 [nfsd]
[Mon Mar 11 14:29:12 2024] ? nfsd4_cld_remove+0x54/0x1d0 [nfsd]
[Mon Mar 11 14:29:12 2024] ? nfsd4_return_all_client_layouts+0xc4/0xf0 [nfsd]
[Mon Mar 11 14:29:12 2024] ? nfsd4_shutdown_copy+0x68/0xc0 [nfsd]
[Mon Mar 11 14:29:12 2024] __destroy_client+0x1f3/0x290 [nfsd]
[Mon Mar 11 14:29:12 2024] nfsd4_exchange_id+0x75f/0x770 [nfsd]
[Mon Mar 11 14:29:12 2024] ? percpu_counter_add_batch+0x67/0x70
[Mon Mar 11 14:29:12 2024] nfsd4_proc_compound+0x44b/0x700 [nfsd]
[Mon Mar 11 14:29:12 2024] nfsd_dispatch+0x94/0x1c0 [nfsd]
[Mon Mar 11 14:29:12 2024] svc_process_common+0x2ec/0x660 [sunrpc]
[Mon Mar 11 14:29:12 2024] ? __pfx_nfsd_dispatch+0x10/0x10 [nfsd]
[Mon Mar 11 14:29:12 2024] ? __pfx_nfsd+0x10/0x10 [nfsd]
[Mon Mar 11 14:29:12 2024] svc_process+0x12d/0x170 [sunrpc]
[Mon Mar 11 14:29:12 2024] nfsd+0x84/0xb0 [nfsd]
[Mon Mar 11 14:29:12 2024] kthread+0xdd/0x100
[Mon Mar 11 14:29:12 2024] ? __pfx_kthread+0x10/0x10
[Mon Mar 11 14:29:12 2024] ret_from_fork+0x29/0x50
[Mon Mar 11 14:29:12 2024] </TASK>
[Mon Mar 11 14:29:12 2024] task:nfsd state:D stack:0 pid:8863 ppid:2 flags:0x00004000
[Mon Mar 11 14:29:12 2024] Call Trace:
[Mon Mar 11 14:29:12 2024] <TASK>
[Mon Mar 11 14:29:12 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:12 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:12 2024] schedule_timeout+0x11f/0x160
[Mon Mar 11 14:29:12 2024] ? select_idle_sibling+0x28/0x430
[Mon Mar 11 14:29:12 2024] ? wake_affine+0x62/0x1f0
[Mon Mar 11 14:29:12 2024] __wait_for_common+0x90/0x1d0
[Mon Mar 11 14:29:12 2024] ? __pfx_schedule_timeout+0x10/0x10
[Mon Mar 11 14:29:12 2024] __flush_workqueue+0x13a/0x3f0
[Mon Mar 11 14:29:12 2024] nfsd4_shutdown_callback+0x49/0x120 [nfsd]
[Mon Mar 11 14:29:12 2024] ? nfsd4_cld_remove+0x54/0x1d0 [nfsd]
[Mon Mar 11 14:29:12 2024] ? nfsd4_return_all_client_layouts+0xc4/0xf0 [nfsd]
[Mon Mar 11 14:29:12 2024] ? nfsd4_shutdown_copy+0x68/0xc0 [nfsd]
[Mon Mar 11 14:29:12 2024] __destroy_client+0x1f3/0x290 [nfsd]
[Mon Mar 11 14:29:12 2024] nfsd4_exchange_id+0x75f/0x770 [nfsd]
[Mon Mar 11 14:29:12 2024] nfsd4_proc_compound+0x44b/0x700 [nfsd]
[Mon Mar 11 14:29:12 2024] nfsd_dispatch+0x94/0x1c0 [nfsd]
[Mon Mar 11 14:29:12 2024] svc_process_common+0x2ec/0x660 [sunrpc]
[Mon Mar 11 14:29:12 2024] ? __pfx_nfsd_dispatch+0x10/0x10 [nfsd]
[Mon Mar 11 14:29:12 2024] ? __pfx_nfsd+0x10/0x10 [nfsd]
[Mon Mar 11 14:29:12 2024] svc_process+0x12d/0x170 [sunrpc]
[Mon Mar 11 14:29:12 2024] nfsd+0x84/0xb0 [nfsd]
[Mon Mar 11 14:29:12 2024] kthread+0xdd/0x100
[Mon Mar 11 14:29:12 2024] ? __pfx_kthread+0x10/0x10
[Mon Mar 11 14:29:12 2024] ret_from_fork+0x29/0x50
[Mon Mar 11 14:29:12 2024] </TASK>
[Mon Mar 11 14:29:12 2024] task:nfsd state:D stack:0 pid:8864 ppid:2 flags:0x00004000
[Mon Mar 11 14:29:12 2024] Call Trace:
[Mon Mar 11 14:29:12 2024] <TASK>
[Mon Mar 11 14:29:12 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:12 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:12 2024] schedule_timeout+0x11f/0x160
[Mon Mar 11 14:29:12 2024] ? select_idle_sibling+0x28/0x430
[Mon Mar 11 14:29:12 2024] ? tcp_recvmsg+0x196/0x210
[Mon Mar 11 14:29:12 2024] ? wake_affine+0x62/0x1f0
[Mon Mar 11 14:29:12 2024] __wait_for_common+0x90/0x1d0
[Mon Mar 11 14:29:12 2024] ? __pfx_schedule_timeout+0x10/0x10
[Mon Mar 11 14:29:12 2024] __flush_workqueue+0x13a/0x3f0
[Mon Mar 11 14:29:12 2024] nfsd4_destroy_session+0x1a4/0x240 [nfsd]
[Mon Mar 11 14:29:12 2024] nfsd4_proc_compound+0x44b/0x700 [nfsd]
[Mon Mar 11 14:29:12 2024] nfsd_dispatch+0x94/0x1c0 [nfsd]
[Mon Mar 11 14:29:12 2024] svc_process_common+0x2ec/0x660 [sunrpc]
[Mon Mar 11 14:29:12 2024] ? __pfx_nfsd_dispatch+0x10/0x10 [nfsd]
[Mon Mar 11 14:29:12 2024] ? __pfx_nfsd+0x10/0x10 [nfsd]
[Mon Mar 11 14:29:12 2024] svc_process+0x12d/0x170 [sunrpc]
[Mon Mar 11 14:29:12 2024] nfsd+0x84/0xb0 [nfsd]
[Mon Mar 11 14:29:12 2024] kthread+0xdd/0x100
[Mon Mar 11 14:29:12 2024] ? __pfx_kthread+0x10/0x10
[Mon Mar 11 14:29:12 2024] ret_from_fork+0x29/0x50
[Mon Mar 11 14:29:12 2024] </TASK>
[Mon Mar 11 14:29:12 2024] task:nfsd state:D stack:0 pid:8865 ppid:2 flags:0x00004000
[Mon Mar 11 14:29:12 2024] Call Trace:
[Mon Mar 11 14:29:12 2024] <TASK>
[Mon Mar 11 14:29:12 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:12 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:12 2024] schedule_timeout+0x11f/0x160
[Mon Mar 11 14:29:12 2024] ? select_idle_sibling+0x28/0x430
[Mon Mar 11 14:29:12 2024] ? wake_affine+0x62/0x1f0
[Mon Mar 11 14:29:12 2024] __wait_for_common+0x90/0x1d0
[Mon Mar 11 14:29:12 2024] ? __pfx_schedule_timeout+0x10/0x10
[Mon Mar 11 14:29:12 2024] __flush_workqueue+0x13a/0x3f0
[Mon Mar 11 14:29:12 2024] nfsd4_shutdown_callback+0x49/0x120 [nfsd]
[Mon Mar 11 14:29:12 2024] ? nfsd4_cld_remove+0x54/0x1d0 [nfsd]
[Mon Mar 11 14:29:12 2024] ? nfsd4_return_all_client_layouts+0xc4/0xf0 [nfsd]
[Mon Mar 11 14:29:12 2024] ? nfsd4_shutdown_copy+0x68/0xc0 [nfsd]
[Mon Mar 11 14:29:12 2024] __destroy_client+0x1f3/0x290 [nfsd]
[Mon Mar 11 14:29:12 2024] nfsd4_exchange_id+0x75f/0x770 [nfsd]
[Mon Mar 11 14:29:12 2024] ? nfsd4_decode_opaque+0x3a/0x90 [nfsd]
[Mon Mar 11 14:29:12 2024] nfsd4_proc_compound+0x44b/0x700 [nfsd]
[Mon Mar 11 14:29:12 2024] nfsd_dispatch+0x94/0x1c0 [nfsd]
[Mon Mar 11 14:29:12 2024] svc_process_common+0x2ec/0x660 [sunrpc]
[Mon Mar 11 14:29:12 2024] ? __pfx_nfsd_dispatch+0x10/0x10 [nfsd]
[Mon Mar 11 14:29:12 2024] ? __pfx_nfsd+0x10/0x10 [nfsd]
[Mon Mar 11 14:29:12 2024] svc_process+0x12d/0x170 [sunrpc]
[Mon Mar 11 14:29:12 2024] nfsd+0x84/0xb0 [nfsd]
[Mon Mar 11 14:29:12 2024] kthread+0xdd/0x100
[Mon Mar 11 14:29:12 2024] ? __pfx_kthread+0x10/0x10
[Mon Mar 11 14:29:12 2024] ret_from_fork+0x29/0x50
[Mon Mar 11 14:29:12 2024] </TASK>
[Mon Mar 11 14:29:12 2024] task:nfsd state:D stack:0 pid:8866 ppid:2 flags:0x00004000
[Mon Mar 11 14:29:12 2024] Call Trace:
[Mon Mar 11 14:29:12 2024] <TASK>
[Mon Mar 11 14:29:12 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:12 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:12 2024] schedule_timeout+0x11f/0x160
[Mon Mar 11 14:29:12 2024] ? select_idle_sibling+0x28/0x430
[Mon Mar 11 14:29:12 2024] ? tcp_recvmsg+0x196/0x210
[Mon Mar 11 14:29:12 2024] ? wake_affine+0x62/0x1f0
[Mon Mar 11 14:29:12 2024] __wait_for_common+0x90/0x1d0
[Mon Mar 11 14:29:12 2024] ? __pfx_schedule_timeout+0x10/0x10
[Mon Mar 11 14:29:12 2024] __flush_workqueue+0x13a/0x3f0
[Mon Mar 11 14:29:12 2024] nfsd4_destroy_session+0x1a4/0x240 [nfsd]
[Mon Mar 11 14:29:12 2024] nfsd4_proc_compound+0x44b/0x700 [nfsd]
[Mon Mar 11 14:29:12 2024] nfsd_dispatch+0x94/0x1c0 [nfsd]
[Mon Mar 11 14:29:12 2024] svc_process_common+0x2ec/0x660 [sunrpc]
[Mon Mar 11 14:29:12 2024] ? __pfx_nfsd_dispatch+0x10/0x10 [nfsd]
[Mon Mar 11 14:29:12 2024] ? __pfx_nfsd+0x10/0x10 [nfsd]
[Mon Mar 11 14:29:12 2024] svc_process+0x12d/0x170 [sunrpc]
[Mon Mar 11 14:29:12 2024] nfsd+0x84/0xb0 [nfsd]
[Mon Mar 11 14:29:12 2024] kthread+0xdd/0x100
[Mon Mar 11 14:29:12 2024] ? __pfx_kthread+0x10/0x10
[Mon Mar 11 14:29:12 2024] ret_from_fork+0x29/0x50
[Mon Mar 11 14:29:12 2024] </TASK>
[Mon Mar 11 14:29:12 2024] task:nfsd state:D stack:0 pid:8867 ppid:2 flags:0x00004000
[Mon Mar 11 14:29:12 2024] Call Trace:
[Mon Mar 11 14:29:12 2024] <TASK>
[Mon Mar 11 14:29:12 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:12 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:12 2024] schedule_timeout+0x11f/0x160
[Mon Mar 11 14:29:12 2024] ? sysvec_apic_timer_interrupt+0xb/0x90
[Mon Mar 11 14:29:12 2024] ? asm_sysvec_apic_timer_interrupt+0x16/0x20
[Mon Mar 11 14:29:12 2024] __wait_for_common+0x90/0x1d0
[Mon Mar 11 14:29:12 2024] ? __pfx_schedule_timeout+0x10/0x10
[Mon Mar 11 14:29:12 2024] __flush_workqueue+0x13a/0x3f0
[Mon Mar 11 14:29:12 2024] nfsd4_destroy_session+0x1a4/0x240 [nfsd]
[Mon Mar 11 14:29:12 2024] nfsd4_proc_compound+0x44b/0x700 [nfsd]
[Mon Mar 11 14:29:12 2024] nfsd_dispatch+0x94/0x1c0 [nfsd]
[Mon Mar 11 14:29:12 2024] svc_process_common+0x2ec/0x660 [sunrpc]
[Mon Mar 11 14:29:12 2024] ? __pfx_nfsd_dispatch+0x10/0x10 [nfsd]
[Mon Mar 11 14:29:12 2024] ? __pfx_nfsd+0x10/0x10 [nfsd]
[Mon Mar 11 14:29:12 2024] svc_process+0x12d/0x170 [sunrpc]
[Mon Mar 11 14:29:12 2024] nfsd+0x84/0xb0 [nfsd]
[Mon Mar 11 14:29:12 2024] kthread+0xdd/0x100
[Mon Mar 11 14:29:12 2024] ? __pfx_kthread+0x10/0x10
[Mon Mar 11 14:29:12 2024] ret_from_fork+0x29/0x50
[Mon Mar 11 14:29:12 2024] </TASK>
[Mon Mar 11 14:29:12 2024] task:nfsd state:D stack:0 pid:8868 ppid:2 flags:0x00004000
[Mon Mar 11 14:29:12 2024] Call Trace:
[Mon Mar 11 14:29:12 2024] <TASK>
[Mon Mar 11 14:29:12 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:12 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:12 2024] schedule_timeout+0x11f/0x160
[Mon Mar 11 14:29:12 2024] ? select_idle_sibling+0x28/0x430
[Mon Mar 11 14:29:12 2024] ? tcp_recvmsg+0x196/0x210
[Mon Mar 11 14:29:12 2024] ? wake_affine+0x62/0x1f0
[Mon Mar 11 14:29:12 2024] __wait_for_common+0x90/0x1d0
[Mon Mar 11 14:29:12 2024] ? __pfx_schedule_timeout+0x10/0x10
[Mon Mar 11 14:29:12 2024] __flush_workqueue+0x13a/0x3f0
[Mon Mar 11 14:29:12 2024] nfsd4_destroy_session+0x1a4/0x240 [nfsd]
[Mon Mar 11 14:29:12 2024] nfsd4_proc_compound+0x44b/0x700 [nfsd]
[Mon Mar 11 14:29:12 2024] nfsd_dispatch+0x94/0x1c0 [nfsd]
[Mon Mar 11 14:29:12 2024] svc_process_common+0x2ec/0x660 [sunrpc]
[Mon Mar 11 14:29:12 2024] ? __pfx_nfsd_dispatch+0x10/0x10 [nfsd]
[Mon Mar 11 14:29:12 2024] ? __pfx_nfsd+0x10/0x10 [nfsd]
[Mon Mar 11 14:29:12 2024] svc_process+0x12d/0x170 [sunrpc]
[Mon Mar 11 14:29:12 2024] nfsd+0x84/0xb0 [nfsd]
[Mon Mar 11 14:29:12 2024] kthread+0xdd/0x100
[Mon Mar 11 14:29:12 2024] ? __pfx_kthread+0x10/0x10
[Mon Mar 11 14:29:12 2024] ret_from_fork+0x29/0x50
[Mon Mar 11 14:29:12 2024] </TASK>
[Mon Mar 11 14:29:12 2024] task:nfsd state:D stack:0 pid:8869 ppid:2 flags:0x00004000
[Mon Mar 11 14:29:12 2024] Call Trace:
[Mon Mar 11 14:29:12 2024] <TASK>
[Mon Mar 11 14:29:12 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:12 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:12 2024] schedule_timeout+0x11f/0x160
[Mon Mar 11 14:29:12 2024] ? select_idle_sibling+0x28/0x430
[Mon Mar 11 14:29:12 2024] ? tcp_recvmsg+0x196/0x210
[Mon Mar 11 14:29:12 2024] __wait_for_common+0x90/0x1d0
[Mon Mar 11 14:29:12 2024] ? __pfx_schedule_timeout+0x10/0x10
[Mon Mar 11 14:29:12 2024] __flush_workqueue+0x13a/0x3f0
[Mon Mar 11 14:29:12 2024] nfsd4_destroy_session+0x1a4/0x240 [nfsd]
[Mon Mar 11 14:29:12 2024] nfsd4_proc_compound+0x44b/0x700 [nfsd]
[Mon Mar 11 14:29:12 2024] nfsd_dispatch+0x94/0x1c0 [nfsd]
[Mon Mar 11 14:29:12 2024] svc_process_common+0x2ec/0x660 [sunrpc]
[Mon Mar 11 14:29:12 2024] ? __pfx_nfsd_dispatch+0x10/0x10 [nfsd]
[Mon Mar 11 14:29:12 2024] ? __pfx_nfsd+0x10/0x10 [nfsd]
[Mon Mar 11 14:29:12 2024] svc_process+0x12d/0x170 [sunrpc]
[Mon Mar 11 14:29:12 2024] nfsd+0x84/0xb0 [nfsd]
[Mon Mar 11 14:29:12 2024] kthread+0xdd/0x100
[Mon Mar 11 14:29:12 2024] ? __pfx_kthread+0x10/0x10
[Mon Mar 11 14:29:12 2024] ret_from_fork+0x29/0x50
[Mon Mar 11 14:29:12 2024] </TASK>
[Mon Mar 11 14:29:12 2024] task:nfsd state:D stack:0 pid:8870 ppid:2 flags:0x00004000
[Mon Mar 11 14:29:12 2024] Call Trace:
[Mon Mar 11 14:29:12 2024] <TASK>
[Mon Mar 11 14:29:12 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:12 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:12 2024] schedule_timeout+0x11f/0x160
[Mon Mar 11 14:29:12 2024] ? select_idle_sibling+0x28/0x430
[Mon Mar 11 14:29:12 2024] ? wake_affine+0x62/0x1f0
[Mon Mar 11 14:29:12 2024] __wait_for_common+0x90/0x1d0
[Mon Mar 11 14:29:12 2024] ? __pfx_schedule_timeout+0x10/0x10
[Mon Mar 11 14:29:12 2024] __flush_workqueue+0x13a/0x3f0
[Mon Mar 11 14:29:12 2024] nfsd4_shutdown_callback+0x49/0x120 [nfsd]
[Mon Mar 11 14:29:12 2024] ? nfsd4_cld_remove+0x54/0x1d0 [nfsd]
[Mon Mar 11 14:29:12 2024] ? nfsd4_return_all_client_layouts+0xc4/0xf0 [nfsd]
[Mon Mar 11 14:29:12 2024] ? nfsd4_shutdown_copy+0x68/0xc0 [nfsd]
[Mon Mar 11 14:29:12 2024] __destroy_client+0x1f3/0x290 [nfsd]
[Mon Mar 11 14:29:12 2024] nfsd4_exchange_id+0x378/0x770 [nfsd]
[Mon Mar 11 14:29:12 2024] ? nfsd4_decode_opaque+0x3a/0x90 [nfsd]
[Mon Mar 11 14:29:12 2024] nfsd4_proc_compound+0x44b/0x700 [nfsd]
[Mon Mar 11 14:29:12 2024] nfsd_dispatch+0x94/0x1c0 [nfsd]
[Mon Mar 11 14:29:12 2024] svc_process_common+0x2ec/0x660 [sunrpc]
[Mon Mar 11 14:29:12 2024] ? __pfx_nfsd_dispatch+0x10/0x10 [nfsd]
[Mon Mar 11 14:29:12 2024] ? __pfx_nfsd+0x10/0x10 [nfsd]
[Mon Mar 11 14:29:12 2024] svc_process+0x12d/0x170 [sunrpc]
[Mon Mar 11 14:29:12 2024] nfsd+0x84/0xb0 [nfsd]
[Mon Mar 11 14:29:12 2024] kthread+0xdd/0x100
[Mon Mar 11 14:29:12 2024] ? __pfx_kthread+0x10/0x10
[Mon Mar 11 14:29:12 2024] ret_from_fork+0x29/0x50
[Mon Mar 11 14:29:12 2024] </TASK>
[Mon Mar 11 14:29:12 2024] task:nfsd state:D stack:0 pid:8871 ppid:2 flags:0x00004000
[Mon Mar 11 14:29:12 2024] Call Trace:
[Mon Mar 11 14:29:12 2024] <TASK>
[Mon Mar 11 14:29:12 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:12 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:12 2024] schedule_timeout+0x11f/0x160
[Mon Mar 11 14:29:12 2024] ? select_idle_sibling+0x28/0x430
[Mon Mar 11 14:29:12 2024] ? wake_affine+0x62/0x1f0
[Mon Mar 11 14:29:12 2024] ? fsnotify+0x2e1/0x3a0
[Mon Mar 11 14:29:12 2024] __wait_for_common+0x90/0x1d0
[Mon Mar 11 14:29:12 2024] ? __pfx_schedule_timeout+0x10/0x10
[Mon Mar 11 14:29:12 2024] __flush_workqueue+0x13a/0x3f0
[Mon Mar 11 14:29:12 2024] nfsd4_create_session+0x8c1/0xb30 [nfsd]
[Mon Mar 11 14:29:12 2024] ? nfsd4_proc_compound+0x44b/0x700 [nfsd]
[Mon Mar 11 14:29:12 2024] nfsd4_proc_compound+0x44b/0x700 [nfsd]
[Mon Mar 11 14:29:12 2024] nfsd_dispatch+0x94/0x1c0 [nfsd]
[Mon Mar 11 14:29:12 2024] svc_process_common+0x2ec/0x660 [sunrpc]
[Mon Mar 11 14:29:12 2024] ? __pfx_nfsd_dispatch+0x10/0x10 [nfsd]
[Mon Mar 11 14:29:12 2024] ? __pfx_nfsd+0x10/0x10 [nfsd]
[Mon Mar 11 14:29:12 2024] svc_process+0x12d/0x170 [sunrpc]
[Mon Mar 11 14:29:12 2024] nfsd+0x84/0xb0 [nfsd]
[Mon Mar 11 14:29:12 2024] kthread+0xdd/0x100
[Mon Mar 11 14:29:12 2024] ? __pfx_kthread+0x10/0x10
[Mon Mar 11 14:29:12 2024] ret_from_fork+0x29/0x50
[Mon Mar 11 14:29:12 2024] </TASK>
[Mon Mar 11 14:29:12 2024] task:nfsd state:D stack:0 pid:8872 ppid:2 flags:0x00004000
[Mon Mar 11 14:29:12 2024] Call Trace:
[Mon Mar 11 14:29:12 2024] <TASK>
[Mon Mar 11 14:29:12 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:12 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:12 2024] schedule_timeout+0x11f/0x160
[Mon Mar 11 14:29:12 2024] ? select_idle_sibling+0x28/0x430
[Mon Mar 11 14:29:12 2024] ? tcp_recvmsg+0x196/0x210
[Mon Mar 11 14:29:12 2024] ? wake_affine+0x62/0x1f0
[Mon Mar 11 14:29:12 2024] __wait_for_common+0x90/0x1d0
[Mon Mar 11 14:29:12 2024] ? __pfx_schedule_timeout+0x10/0x10
[Mon Mar 11 14:29:12 2024] __flush_workqueue+0x13a/0x3f0
[Mon Mar 11 14:29:12 2024] nfsd4_destroy_session+0x1a4/0x240 [nfsd]
[Mon Mar 11 14:29:12 2024] nfsd4_proc_compound+0x44b/0x700 [nfsd]
[Mon Mar 11 14:29:12 2024] nfsd_dispatch+0x94/0x1c0 [nfsd]
[Mon Mar 11 14:29:12 2024] svc_process_common+0x2ec/0x660 [sunrpc]
[Mon Mar 11 14:29:12 2024] ? __pfx_nfsd_dispatch+0x10/0x10 [nfsd]
[Mon Mar 11 14:29:12 2024] ? __pfx_nfsd+0x10/0x10 [nfsd]
[Mon Mar 11 14:29:12 2024] svc_process+0x12d/0x170 [sunrpc]
[Mon Mar 11 14:29:12 2024] nfsd+0x84/0xb0 [nfsd]
[Mon Mar 11 14:29:12 2024] kthread+0xdd/0x100
[Mon Mar 11 14:29:12 2024] ? __pfx_kthread+0x10/0x10
[Mon Mar 11 14:29:12 2024] ret_from_fork+0x29/0x50
[Mon Mar 11 14:29:12 2024] </TASK>
[Mon Mar 11 14:29:12 2024] task:nfsd state:D stack:0 pid:8873 ppid:2 flags:0x00004000
[Mon Mar 11 14:29:12 2024] Call Trace:
[Mon Mar 11 14:29:12 2024] <TASK>
[Mon Mar 11 14:29:12 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:12 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:12 2024] schedule_timeout+0x11f/0x160
[Mon Mar 11 14:29:12 2024] ? select_idle_sibling+0x28/0x430
[Mon Mar 11 14:29:12 2024] __wait_for_common+0x90/0x1d0
[Mon Mar 11 14:29:12 2024] ? __pfx_schedule_timeout+0x10/0x10
[Mon Mar 11 14:29:12 2024] __flush_workqueue+0x13a/0x3f0
[Mon Mar 11 14:29:12 2024] nfsd4_shutdown_callback+0x49/0x120 [nfsd]
[Mon Mar 11 14:29:12 2024] ? nfsd4_cld_remove+0x54/0x1d0 [nfsd]
[Mon Mar 11 14:29:12 2024] ? nfsd4_return_all_client_layouts+0xc4/0xf0 [nfsd]
[Mon Mar 11 14:29:12 2024] ? nfsd4_shutdown_copy+0x68/0xc0 [nfsd]
[Mon Mar 11 14:29:12 2024] __destroy_client+0x1f3/0x290 [nfsd]
[Mon Mar 11 14:29:12 2024] nfsd4_exchange_id+0x75f/0x770 [nfsd]
[Mon Mar 11 14:29:12 2024] ? nfsd4_decode_opaque+0x3a/0x90 [nfsd]
[Mon Mar 11 14:29:12 2024] nfsd4_proc_compound+0x44b/0x700 [nfsd]
[Mon Mar 11 14:29:12 2024] nfsd_dispatch+0x94/0x1c0 [nfsd]
[Mon Mar 11 14:29:12 2024] svc_process_common+0x2ec/0x660 [sunrpc]
[Mon Mar 11 14:29:12 2024] ? __pfx_nfsd_dispatch+0x10/0x10 [nfsd]
[Mon Mar 11 14:29:12 2024] ? __pfx_nfsd+0x10/0x10 [nfsd]
[Mon Mar 11 14:29:12 2024] svc_process+0x12d/0x170 [sunrpc]
[Mon Mar 11 14:29:12 2024] nfsd+0x84/0xb0 [nfsd]
[Mon Mar 11 14:29:12 2024] kthread+0xdd/0x100
[Mon Mar 11 14:29:12 2024] ? __pfx_kthread+0x10/0x10
[Mon Mar 11 14:29:12 2024] ret_from_fork+0x29/0x50
[Mon Mar 11 14:29:12 2024] </TASK>
[Mon Mar 11 14:29:12 2024] task:nfsd state:D stack:0 pid:8874 ppid:2 flags:0x00004000
[Mon Mar 11 14:29:12 2024] Call Trace:
[Mon Mar 11 14:29:12 2024] <TASK>
[Mon Mar 11 14:29:12 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:12 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:12 2024] schedule_timeout+0x11f/0x160
[Mon Mar 11 14:29:12 2024] ? select_idle_sibling+0x28/0x430
[Mon Mar 11 14:29:12 2024] ? tcp_recvmsg+0x196/0x210
[Mon Mar 11 14:29:12 2024] __wait_for_common+0x90/0x1d0
[Mon Mar 11 14:29:12 2024] ? __pfx_schedule_timeout+0x10/0x10
[Mon Mar 11 14:29:12 2024] __flush_workqueue+0x13a/0x3f0
[Mon Mar 11 14:29:12 2024] nfsd4_destroy_session+0x1a4/0x240 [nfsd]
[Mon Mar 11 14:29:12 2024] nfsd4_proc_compound+0x44b/0x700 [nfsd]
[Mon Mar 11 14:29:12 2024] nfsd_dispatch+0x94/0x1c0 [nfsd]
[Mon Mar 11 14:29:12 2024] svc_process_common+0x2ec/0x660 [sunrpc]
[Mon Mar 11 14:29:12 2024] ? __pfx_nfsd_dispatch+0x10/0x10 [nfsd]
[Mon Mar 11 14:29:12 2024] ? __pfx_nfsd+0x10/0x10 [nfsd]
[Mon Mar 11 14:29:12 2024] svc_process+0x12d/0x170 [sunrpc]
[Mon Mar 11 14:29:12 2024] nfsd+0x84/0xb0 [nfsd]
[Mon Mar 11 14:29:12 2024] kthread+0xdd/0x100
[Mon Mar 11 14:29:12 2024] ? __pfx_kthread+0x10/0x10
[Mon Mar 11 14:29:12 2024] ret_from_fork+0x29/0x50
[Mon Mar 11 14:29:12 2024] </TASK>
[Mon Mar 11 14:29:12 2024] task:nfsd state:D stack:0 pid:8875 ppid:2 flags:0x00004000
[Mon Mar 11 14:29:12 2024] Call Trace:
[Mon Mar 11 14:29:12 2024] <TASK>
[Mon Mar 11 14:29:12 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:12 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:12 2024] schedule_timeout+0x11f/0x160
[Mon Mar 11 14:29:12 2024] ? select_idle_sibling+0x101/0x430
[Mon Mar 11 14:29:12 2024] ? remove_entity_load_avg+0x2e/0x70
[Mon Mar 11 14:29:12 2024] __wait_for_common+0x90/0x1d0
[Mon Mar 11 14:29:12 2024] ? __pfx_schedule_timeout+0x10/0x10
[Mon Mar 11 14:29:12 2024] __flush_workqueue+0x13a/0x3f0
[Mon Mar 11 14:29:12 2024] nfsd4_shutdown_callback+0x49/0x120 [nfsd]
[Mon Mar 11 14:29:12 2024] ? nfsd4_cld_remove+0x54/0x1d0 [nfsd]
[Mon Mar 11 14:29:12 2024] ? nfsd4_return_all_client_layouts+0xc4/0xf0 [nfsd]
[Mon Mar 11 14:29:12 2024] ? nfsd4_shutdown_copy+0x68/0xc0 [nfsd]
[Mon Mar 11 14:29:12 2024] __destroy_client+0x1f3/0x290 [nfsd]
[Mon Mar 11 14:29:12 2024] nfsd4_exchange_id+0x75f/0x770 [nfsd]
[Mon Mar 11 14:29:12 2024] ? nfsd4_decode_opaque+0x3a/0x90 [nfsd]
[Mon Mar 11 14:29:12 2024] nfsd4_proc_compound+0x44b/0x700 [nfsd]
[Mon Mar 11 14:29:12 2024] nfsd_dispatch+0x94/0x1c0 [nfsd]
[Mon Mar 11 14:29:12 2024] svc_process_common+0x2ec/0x660 [sunrpc]
[Mon Mar 11 14:29:12 2024] ? __pfx_nfsd_dispatch+0x10/0x10 [nfsd]
[Mon Mar 11 14:29:12 2024] ? __pfx_nfsd+0x10/0x10 [nfsd]
[Mon Mar 11 14:29:12 2024] svc_process+0x12d/0x170 [sunrpc]
[Mon Mar 11 14:29:12 2024] nfsd+0x84/0xb0 [nfsd]
[Mon Mar 11 14:29:12 2024] kthread+0xdd/0x100
[Mon Mar 11 14:29:12 2024] ? __pfx_kthread+0x10/0x10
[Mon Mar 11 14:29:12 2024] ret_from_fork+0x29/0x50
[Mon Mar 11 14:29:12 2024] </TASK>
[Mon Mar 11 14:29:12 2024] task:nfsd state:D stack:0 pid:8876 ppid:2 flags:0x00004000
[Mon Mar 11 14:29:12 2024] Call Trace:
[Mon Mar 11 14:29:12 2024] <TASK>
[Mon Mar 11 14:29:12 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:12 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:12 2024] schedule_timeout+0x11f/0x160
[Mon Mar 11 14:29:12 2024] ? select_idle_sibling+0x28/0x430
[Mon Mar 11 14:29:12 2024] ? wake_affine+0x62/0x1f0
[Mon Mar 11 14:29:12 2024] __wait_for_common+0x90/0x1d0
[Mon Mar 11 14:29:12 2024] ? __pfx_schedule_timeout+0x10/0x10
[Mon Mar 11 14:29:12 2024] __flush_workqueue+0x13a/0x3f0
[Mon Mar 11 14:29:12 2024] nfsd4_shutdown_callback+0x49/0x120 [nfsd]
[Mon Mar 11 14:29:12 2024] ? nfsd4_cld_remove+0x54/0x1d0 [nfsd]
[Mon Mar 11 14:29:12 2024] ? nfsd4_return_all_client_layouts+0xc4/0xf0 [nfsd]
[Mon Mar 11 14:29:12 2024] ? nfsd4_shutdown_copy+0x68/0xc0 [nfsd]
[Mon Mar 11 14:29:12 2024] __destroy_client+0x1f3/0x290 [nfsd]
[Mon Mar 11 14:29:12 2024] nfsd4_exchange_id+0x75f/0x770 [nfsd]
[Mon Mar 11 14:29:12 2024] ? nfsd4_decode_opaque+0x3a/0x90 [nfsd]
[Mon Mar 11 14:29:12 2024] nfsd4_proc_compound+0x44b/0x700 [nfsd]
[Mon Mar 11 14:29:12 2024] nfsd_dispatch+0x94/0x1c0 [nfsd]
[Mon Mar 11 14:29:12 2024] svc_process_common+0x2ec/0x660 [sunrpc]
[Mon Mar 11 14:29:12 2024] ? __pfx_nfsd_dispatch+0x10/0x10 [nfsd]
[Mon Mar 11 14:29:12 2024] ? __pfx_nfsd+0x10/0x10 [nfsd]
[Mon Mar 11 14:29:12 2024] svc_process+0x12d/0x170 [sunrpc]
[Mon Mar 11 14:29:12 2024] nfsd+0x84/0xb0 [nfsd]
[Mon Mar 11 14:29:12 2024] kthread+0xdd/0x100
[Mon Mar 11 14:29:12 2024] ? __pfx_kthread+0x10/0x10
[Mon Mar 11 14:29:12 2024] ret_from_fork+0x29/0x50
[Mon Mar 11 14:29:12 2024] </TASK>
[Mon Mar 11 14:29:12 2024] task:nfsd state:D stack:0 pid:8877 ppid:2 flags:0x00004000
[Mon Mar 11 14:29:12 2024] Call Trace:
[Mon Mar 11 14:29:12 2024] <TASK>
[Mon Mar 11 14:29:12 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:12 2024] ? crypto_sha1_update+0x56/0x1c0
[Mon Mar 11 14:29:12 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:12 2024] schedule_timeout+0x11f/0x160
[Mon Mar 11 14:29:12 2024] ? select_idle_sibling+0x28/0x430
[Mon Mar 11 14:29:12 2024] ? wake_affine+0x62/0x1f0
[Mon Mar 11 14:29:12 2024] __wait_for_common+0x90/0x1d0
[Mon Mar 11 14:29:12 2024] ? __pfx_schedule_timeout+0x10/0x10
[Mon Mar 11 14:29:12 2024] __flush_workqueue+0x13a/0x3f0
[Mon Mar 11 14:29:12 2024] nfsd4_destroy_session+0x1a4/0x240 [nfsd]
[Mon Mar 11 14:29:12 2024] nfsd4_proc_compound+0x44b/0x700 [nfsd]
[Mon Mar 11 14:29:12 2024] nfsd_dispatch+0x94/0x1c0 [nfsd]
[Mon Mar 11 14:29:12 2024] svc_process_common+0x2ec/0x660 [sunrpc]
[Mon Mar 11 14:29:12 2024] ? __pfx_nfsd_dispatch+0x10/0x10 [nfsd]
[Mon Mar 11 14:29:12 2024] ? __pfx_nfsd+0x10/0x10 [nfsd]
[Mon Mar 11 14:29:12 2024] svc_process+0x12d/0x170 [sunrpc]
[Mon Mar 11 14:29:12 2024] nfsd+0x84/0xb0 [nfsd]
[Mon Mar 11 14:29:12 2024] kthread+0xdd/0x100
[Mon Mar 11 14:29:12 2024] ? __pfx_kthread+0x10/0x10
[Mon Mar 11 14:29:12 2024] ret_from_fork+0x29/0x50
[Mon Mar 11 14:29:12 2024] </TASK>
[Mon Mar 11 14:29:12 2024] task:cups-browsed state:S stack:0 pid:21421 ppid:1 flags:0x00000002
[Mon Mar 11 14:29:12 2024] Call Trace:
[Mon Mar 11 14:29:12 2024] <TASK>
[Mon Mar 11 14:29:12 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:12 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:12 2024] schedule_hrtimeout_range_clock+0x95/0x120
[Mon Mar 11 14:29:12 2024] ? __pfx_hrtimer_wakeup+0x10/0x10
[Mon Mar 11 14:29:12 2024] do_poll.constprop.0+0x248/0x390
[Mon Mar 11 14:29:12 2024] do_sys_poll+0x1c8/0x250
[Mon Mar 11 14:29:12 2024] ? ip_finish_output2+0x1a5/0x440
[Mon Mar 11 14:29:12 2024] ? __ip_queue_xmit+0x184/0x430
[Mon Mar 11 14:29:12 2024] ? tcp_update_skb_after_send+0x69/0xd0
[Mon Mar 11 14:29:12 2024] ? lock_timer_base+0x61/0x80
[Mon Mar 11 14:29:12 2024] ? __pfx_pollwake+0x10/0x10
[Mon Mar 11 14:29:12 2024] ? __pfx_pollwake+0x10/0x10
[Mon Mar 11 14:29:12 2024] ? tcp_write_xmit+0x6f5/0xaa0
[Mon Mar 11 14:29:12 2024] ? rcu_nocb_try_bypass+0x5e/0x460
[Mon Mar 11 14:29:12 2024] ? mem_cgroup_charge_skmem+0xbb/0xf0
[Mon Mar 11 14:29:12 2024] ? __mod_memcg_lruvec_state+0x84/0xd0
[Mon Mar 11 14:29:12 2024] ? mod_objcg_state+0x1fe/0x300
[Mon Mar 11 14:29:12 2024] ? rcu_nocb_try_bypass+0x5e/0x460
[Mon Mar 11 14:29:12 2024] ? rseq_get_rseq_cs+0x1d/0x240
[Mon Mar 11 14:29:12 2024] ? rseq_ip_fixup+0x6e/0x1a0
[Mon Mar 11 14:29:12 2024] ? __call_rcu_common.constprop.0+0x117/0x2b0
[Mon Mar 11 14:29:12 2024] ? __rseq_handle_notify_resume+0x26/0xb0
[Mon Mar 11 14:29:12 2024] ? exit_to_user_mode_loop+0xd0/0x130
[Mon Mar 11 14:29:12 2024] ? exit_to_user_mode_prepare+0xb6/0x100
[Mon Mar 11 14:29:12 2024] ? syscall_exit_to_user_mode+0x22/0x40
[Mon Mar 11 14:29:12 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:12 2024] ? ktime_get_ts64+0x49/0xf0
[Mon Mar 11 14:29:12 2024] __x64_sys_poll+0xa6/0x140
[Mon Mar 11 14:29:12 2024] do_syscall_64+0x59/0x90
[Mon Mar 11 14:29:12 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:12 2024] ? syscall_exit_to_user_mode+0x22/0x40
[Mon Mar 11 14:29:12 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:12 2024] ? __irq_exit_rcu+0x46/0xc0
[Mon Mar 11 14:29:12 2024] ? sysvec_call_function_single+0x3c/0x90
[Mon Mar 11 14:29:12 2024] entry_SYSCALL_64_after_hwframe+0x72/0xdc
[Mon Mar 11 14:29:12 2024] RIP: 0033:0x7f4034b426ff
[Mon Mar 11 14:29:12 2024] RSP: 002b:00007ffc78a741f0 EFLAGS: 00000293 ORIG_RAX: 0000000000000007
[Mon Mar 11 14:29:12 2024] RAX: ffffffffffffffda RBX: 00007f4034e53071 RCX: 00007f4034b426ff
[Mon Mar 11 14:29:12 2024] RDX: 00000000001b76c5 RSI: 0000000000000002 RDI: 00007f402400c650
[Mon Mar 11 14:29:12 2024] RBP: 00007f402400c650 R08: 0000000000000000 R09: 00007ffc78a74080
[Mon Mar 11 14:29:12 2024] R10: 00007ffc78abe080 R11: 0000000000000293 R12: 0000000000000002
[Mon Mar 11 14:29:12 2024] R13: 0000000000000002 R14: 00007ffc78a74260 R15: 000056181391da70
[Mon Mar 11 14:29:12 2024] </TASK>
[Mon Mar 11 14:29:12 2024] task:gmain state:S stack:0 pid:21425 ppid:1 flags:0x00000002
[Mon Mar 11 14:29:12 2024] Call Trace:
[Mon Mar 11 14:29:12 2024] <TASK>
[Mon Mar 11 14:29:12 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:12 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:12 2024] schedule_hrtimeout_range_clock+0x10f/0x120
[Mon Mar 11 14:29:12 2024] ? add_wait_queue+0x65/0xa0
[Mon Mar 11 14:29:12 2024] do_poll.constprop.0+0x248/0x390
[Mon Mar 11 14:29:12 2024] do_sys_poll+0x1c8/0x250
[Mon Mar 11 14:29:12 2024] ? rmqueue_pcplist+0xda/0x210
[Mon Mar 11 14:29:12 2024] ? rmqueue+0x7d3/0xd40
[Mon Mar 11 14:29:12 2024] ? rmqueue+0x7d3/0xd40
[Mon Mar 11 14:29:12 2024] ? __pfx_pollwake+0x10/0x10
[Mon Mar 11 14:29:12 2024] ? __pfx_pollwake+0x10/0x10
[Mon Mar 11 14:29:12 2024] ? get_page_from_freelist+0x387/0x530
[Mon Mar 11 14:29:12 2024] ? sysvec_call_function_single+0xb/0x90
[Mon Mar 11 14:29:12 2024] ? asm_sysvec_call_function_single+0x16/0x20
[Mon Mar 11 14:29:12 2024] ? __mod_memcg_lruvec_state+0x84/0xd0
[Mon Mar 11 14:29:12 2024] ? _copy_to_iter+0x7e/0x630
[Mon Mar 11 14:29:12 2024] ? __audit_filter_op+0xa5/0xf0
[Mon Mar 11 14:29:12 2024] ? audit_reset_context.part.0.constprop.0+0x273/0x2e0
[Mon Mar 11 14:29:12 2024] ? auditd_test_task+0x3c/0x50
[Mon Mar 11 14:29:12 2024] __x64_sys_poll+0x39/0x140
[Mon Mar 11 14:29:12 2024] ? syscall_trace_enter.constprop.0+0x126/0x1a0
[Mon Mar 11 14:29:12 2024] do_syscall_64+0x59/0x90
[Mon Mar 11 14:29:12 2024] ? audit_reset_context.part.0.constprop.0+0x273/0x2e0
[Mon Mar 11 14:29:12 2024] ? syscall_exit_work+0x103/0x130
[Mon Mar 11 14:29:12 2024] ? syscall_exit_to_user_mode+0x22/0x40
[Mon Mar 11 14:29:12 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:12 2024] ? exc_page_fault+0x62/0x150
[Mon Mar 11 14:29:12 2024] entry_SYSCALL_64_after_hwframe+0x72/0xdc
[Mon Mar 11 14:29:12 2024] RIP: 0033:0x7f4034b426ff
[Mon Mar 11 14:29:12 2024] RSP: 002b:00007f403364b980 EFLAGS: 00000293 ORIG_RAX: 0000000000000007
[Mon Mar 11 14:29:12 2024] RAX: ffffffffffffffda RBX: 00007f4034e53071 RCX: 00007f4034b426ff
[Mon Mar 11 14:29:12 2024] RDX: 00000000ffffffff RSI: 0000000000000001 RDI: 00005618138df790
[Mon Mar 11 14:29:12 2024] RBP: 00005618138df790 R08: 0000000000000000 R09: 0000000000000000
[Mon Mar 11 14:29:12 2024] R10: 00007ffc78abe080 R11: 0000000000000293 R12: 0000000000000001
[Mon Mar 11 14:29:12 2024] R13: 0000000000000001 R14: 00007f403364b9f0 R15: 0000561813913e70
[Mon Mar 11 14:29:12 2024] </TASK>
[Mon Mar 11 14:29:12 2024] task:gdbus state:S stack:0 pid:21426 ppid:1 flags:0x00000002
[Mon Mar 11 14:29:12 2024] Call Trace:
[Mon Mar 11 14:29:12 2024] <TASK>
[Mon Mar 11 14:29:12 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:12 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:12 2024] schedule_hrtimeout_range_clock+0x10f/0x120
[Mon Mar 11 14:29:12 2024] ? add_wait_queue+0x65/0xa0
[Mon Mar 11 14:29:12 2024] ? add_wait_queue+0x65/0xa0
[Mon Mar 11 14:29:12 2024] do_poll.constprop.0+0x248/0x390
[Mon Mar 11 14:29:12 2024] ? add_wait_queue+0x65/0xa0
[Mon Mar 11 14:29:12 2024] do_sys_poll+0x1c8/0x250
[Mon Mar 11 14:29:12 2024] ? remove_wait_queue+0x20/0x60
[Mon Mar 11 14:29:12 2024] ? poll_freewait+0x45/0xa0
[Mon Mar 11 14:29:12 2024] ? do_sys_poll+0x1d8/0x250
[Mon Mar 11 14:29:12 2024] ? __pfx_pollwake+0x10/0x10
[Mon Mar 11 14:29:12 2024] ? __pfx_pollwake+0x10/0x10
[Mon Mar 11 14:29:12 2024] ? sock_recvmsg+0x99/0xa0
[Mon Mar 11 14:29:12 2024] ? __check_object_size.part.0+0x35/0xd0
[Mon Mar 11 14:29:12 2024] ? __pfx_pollwake+0x10/0x10
[Mon Mar 11 14:29:12 2024] ? _copy_to_iter+0x7e/0x630
[Mon Mar 11 14:29:12 2024] ? __pfx_pollwake+0x10/0x10
[Mon Mar 11 14:29:12 2024] ? eventfd_read+0xe2/0x2b0
[Mon Mar 11 14:29:12 2024] ? auditd_test_task+0x3c/0x50
[Mon Mar 11 14:29:12 2024] ? __audit_syscall_entry+0xef/0x140
[Mon Mar 11 14:29:12 2024] __x64_sys_poll+0x39/0x140
[Mon Mar 11 14:29:12 2024] ? syscall_trace_enter.constprop.0+0x126/0x1a0
[Mon Mar 11 14:29:12 2024] do_syscall_64+0x59/0x90
[Mon Mar 11 14:29:12 2024] ? syscall_exit_work+0x103/0x130
[Mon Mar 11 14:29:12 2024] ? syscall_exit_to_user_mode+0x22/0x40
[Mon Mar 11 14:29:12 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:12 2024] ? audit_reset_context.part.0.constprop.0+0x273/0x2e0
[Mon Mar 11 14:29:12 2024] ? syscall_exit_work+0x103/0x130
[Mon Mar 11 14:29:12 2024] ? syscall_exit_to_user_mode+0x22/0x40
[Mon Mar 11 14:29:12 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:12 2024] ? __irq_exit_rcu+0x46/0xc0
[Mon Mar 11 14:29:12 2024] ? sysvec_apic_timer_interrupt+0x3c/0x90
[Mon Mar 11 14:29:12 2024] entry_SYSCALL_64_after_hwframe+0x72/0xdc
[Mon Mar 11 14:29:12 2024] RIP: 0033:0x7f4034b426ff
[Mon Mar 11 14:29:12 2024] RSP: 002b:00007f4032e4a980 EFLAGS: 00000293 ORIG_RAX: 0000000000000007
[Mon Mar 11 14:29:12 2024] RAX: ffffffffffffffda RBX: 00007f4034e53071 RCX: 00007f4034b426ff
[Mon Mar 11 14:29:12 2024] RDX: 00000000ffffffff RSI: 0000000000000002 RDI: 00005618138e90d0
[Mon Mar 11 14:29:12 2024] RBP: 00005618138e90d0 R08: 0000000000000000 R09: 00007f4032e4a810
[Mon Mar 11 14:29:12 2024] R10: 00007ffc78abe080 R11: 0000000000000293 R12: 0000000000000002
[Mon Mar 11 14:29:12 2024] R13: 0000000000000002 R14: 00007f4032e4a9f0 R15: 0000561813925c70
[Mon Mar 11 14:29:12 2024] </TASK>
[Mon Mar 11 14:29:12 2024] task:dbus state:S stack:0 pid:21786 ppid:4399 flags:0x00000002
[Mon Mar 11 14:29:12 2024] Call Trace:
[Mon Mar 11 14:29:12 2024] <TASK>
[Mon Mar 11 14:29:12 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:12 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:12 2024] pipe_read+0x38b/0x4c0
[Mon Mar 11 14:29:12 2024] ? __pfx_autoremove_wake_function+0x10/0x10
[Mon Mar 11 14:29:12 2024] vfs_read+0x2f9/0x330
[Mon Mar 11 14:29:12 2024] ksys_read+0xab/0xe0
[Mon Mar 11 14:29:12 2024] ? syscall_trace_enter.constprop.0+0x126/0x1a0
[Mon Mar 11 14:29:12 2024] do_syscall_64+0x59/0x90
[Mon Mar 11 14:29:12 2024] ? audit_reset_context.part.0.constprop.0+0x273/0x2e0
[Mon Mar 11 14:29:12 2024] ? syscall_exit_work+0x103/0x130
[Mon Mar 11 14:29:12 2024] ? syscall_exit_to_user_mode+0x22/0x40
[Mon Mar 11 14:29:12 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:12 2024] ? sysvec_apic_timer_interrupt+0x3c/0x90
[Mon Mar 11 14:29:12 2024] entry_SYSCALL_64_after_hwframe+0x72/0xdc
[Mon Mar 11 14:29:12 2024] RIP: 0033:0x7fc04133e882
[Mon Mar 11 14:29:12 2024] RSP: 002b:00007fff2c40bcf8 EFLAGS: 00000246 ORIG_RAX: 0000000000000000
[Mon Mar 11 14:29:12 2024] RAX: ffffffffffffffda RBX: 00005646b2641790 RCX: 00007fc04133e882
[Mon Mar 11 14:29:12 2024] RDX: 0000000000000008 RSI: 00005646b2636d71 RDI: 0000000000000000
[Mon Mar 11 14:29:12 2024] RBP: 00007fff2c40c19c R08: 00005646b2634620 R09: 0000000000000000
[Mon Mar 11 14:29:12 2024] R10: 0000000000000003 R11: 0000000000000246 R12: 00007fc041525ad0
[Mon Mar 11 14:29:12 2024] R13: 00005646b263f4f0 R14: 00005646b2536143 R15: 00005646b2636d71
[Mon Mar 11 14:29:12 2024] </TASK>
[Mon Mar 11 14:29:12 2024] task:colord state:S stack:0 pid:21791 ppid:1 flags:0x00000002
[Mon Mar 11 14:29:12 2024] Call Trace:
[Mon Mar 11 14:29:12 2024] <TASK>
[Mon Mar 11 14:29:12 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:12 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:12 2024] schedule_hrtimeout_range_clock+0x10f/0x120
[Mon Mar 11 14:29:12 2024] ? add_wait_queue+0x65/0xa0
[Mon Mar 11 14:29:12 2024] ? datagram_poll+0xdb/0x110
[Mon Mar 11 14:29:12 2024] ? sock_poll+0x4c/0xe0
[Mon Mar 11 14:29:12 2024] do_poll.constprop.0+0x248/0x390
[Mon Mar 11 14:29:12 2024] do_sys_poll+0x1c8/0x250
[Mon Mar 11 14:29:12 2024] ? do_sys_poll+0x1d8/0x250
[Mon Mar 11 14:29:12 2024] ? rmqueue_pcplist+0xda/0x210
[Mon Mar 11 14:29:12 2024] ? rmqueue+0x7d3/0xd40
[Mon Mar 11 14:29:12 2024] ? __pfx_pollwake+0x10/0x10
[Mon Mar 11 14:29:12 2024] ? __pfx_pollwake+0x10/0x10
[Mon Mar 11 14:29:12 2024] ? __pfx_pollwake+0x10/0x10
[Mon Mar 11 14:29:12 2024] ? __pfx_pollwake+0x10/0x10
[Mon Mar 11 14:29:12 2024] ? futex_wake+0x151/0x180
[Mon Mar 11 14:29:12 2024] ? __audit_filter_op+0xa5/0xf0
[Mon Mar 11 14:29:12 2024] ? auditd_test_task+0x3c/0x50
[Mon Mar 11 14:29:12 2024] __x64_sys_poll+0x39/0x140
[Mon Mar 11 14:29:12 2024] ? syscall_trace_enter.constprop.0+0x126/0x1a0
[Mon Mar 11 14:29:12 2024] do_syscall_64+0x59/0x90
[Mon Mar 11 14:29:12 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:12 2024] ? handle_mm_fault+0xcd/0x290
[Mon Mar 11 14:29:12 2024] ? do_user_addr_fault+0x1d6/0x6a0
[Mon Mar 11 14:29:12 2024] ? sched_clock_cpu+0x9/0xc0
[Mon Mar 11 14:29:12 2024] ? exc_page_fault+0x62/0x150
[Mon Mar 11 14:29:12 2024] entry_SYSCALL_64_after_hwframe+0x72/0xdc
[Mon Mar 11 14:29:12 2024] RIP: 0033:0x7f41a23426ff
[Mon Mar 11 14:29:12 2024] RSP: 002b:00007fff02d31390 EFLAGS: 00000293 ORIG_RAX: 0000000000000007
[Mon Mar 11 14:29:15 2024] RAX: ffffffffffffffda RBX: 00007f41a2830071 RCX: 00007f41a23426ff
[Mon Mar 11 14:29:15 2024] RDX: 00000000ffffffff RSI: 0000000000000004 RDI: 000055e840bce740
[Mon Mar 11 14:29:15 2024] RBP: 000055e840bce740 R08: 0000000000000000 R09: 00007fff02d31220
[Mon Mar 11 14:29:15 2024] R10: 00007fff02d83080 R11: 0000000000000293 R12: 0000000000000004
[Mon Mar 11 14:29:15 2024] R13: 0000000000000004 R14: 00007fff02d31400 R15: 000055e840b13180
[Mon Mar 11 14:29:15 2024] </TASK>
[Mon Mar 11 14:29:15 2024] task:pool-spawner state:S stack:0 pid:21804 ppid:1 flags:0x00000002
[Mon Mar 11 14:29:15 2024] Call Trace:
[Mon Mar 11 14:29:15 2024] <TASK>
[Mon Mar 11 14:29:15 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:15 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:15 2024] futex_wait_queue+0x70/0xd0
[Mon Mar 11 14:29:15 2024] futex_wait+0x175/0x260
[Mon Mar 11 14:29:15 2024] ? select_idle_sibling+0x28/0x430
[Mon Mar 11 14:29:15 2024] ? wake_affine+0x62/0x1f0
[Mon Mar 11 14:29:15 2024] ? enqueue_task+0x47/0x110
[Mon Mar 11 14:29:15 2024] ? sched_clock+0xc/0x30
[Mon Mar 11 14:29:15 2024] do_futex+0x12d/0x1d0
[Mon Mar 11 14:29:15 2024] __x64_sys_futex+0x73/0x1d0
[Mon Mar 11 14:29:15 2024] do_syscall_64+0x59/0x90
[Mon Mar 11 14:29:15 2024] ? audit_reset_context.part.0.constprop.0+0x273/0x2e0
[Mon Mar 11 14:29:15 2024] ? syscall_exit_work+0x103/0x130
[Mon Mar 11 14:29:15 2024] ? syscall_exit_to_user_mode+0x22/0x40
[Mon Mar 11 14:29:15 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:15 2024] ? exc_page_fault+0x62/0x150
[Mon Mar 11 14:29:15 2024] entry_SYSCALL_64_after_hwframe+0x72/0xdc
[Mon Mar 11 14:29:15 2024] RIP: 0033:0x7f41a223ee5d
[Mon Mar 11 14:29:15 2024] RSP: 002b:00007f41a19febf8 EFLAGS: 00000246 ORIG_RAX: 00000000000000ca
[Mon Mar 11 14:29:15 2024] RAX: ffffffffffffffda RBX: 0000000000000002 RCX: 00007f41a223ee5d
[Mon Mar 11 14:29:15 2024] RDX: 0000000000000002 RSI: 0000000000000080 RDI: 000055e840ba9500
[Mon Mar 11 14:29:15 2024] RBP: 000055e840ba94f0 R08: 00007f41a19feae0 R09: 0000000000000001
[Mon Mar 11 14:29:15 2024] R10: 0000000000000000 R11: 0000000000000246 R12: 000055e840ba9500
[Mon Mar 11 14:29:15 2024] R13: 000055e840ba9508 R14: 000055e840ba94f8 R15: 00007f41a2841065
[Mon Mar 11 14:29:15 2024] </TASK>
[Mon Mar 11 14:29:15 2024] task:gmain state:S stack:0 pid:21805 ppid:1 flags:0x00000002
[Mon Mar 11 14:29:15 2024] Call Trace:
[Mon Mar 11 14:29:15 2024] <TASK>
[Mon Mar 11 14:29:15 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:15 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:15 2024] schedule_hrtimeout_range_clock+0x10f/0x120
[Mon Mar 11 14:29:15 2024] ? add_wait_queue+0x65/0xa0
[Mon Mar 11 14:29:15 2024] ? add_wait_queue+0x65/0xa0
[Mon Mar 11 14:29:15 2024] do_poll.constprop.0+0x248/0x390
[Mon Mar 11 14:29:15 2024] do_sys_poll+0x1c8/0x250
[Mon Mar 11 14:29:15 2024] ? rmqueue_pcplist+0xda/0x210
[Mon Mar 11 14:29:15 2024] ? __pfx_pollwake+0x10/0x10
[Mon Mar 11 14:29:15 2024] ? __pfx_pollwake+0x10/0x10
[Mon Mar 11 14:29:15 2024] ? get_page_from_freelist+0x387/0x530
[Mon Mar 11 14:29:15 2024] ? __mod_memcg_state+0x63/0xb0
[Mon Mar 11 14:29:15 2024] ? _copy_to_iter+0x7e/0x630
[Mon Mar 11 14:29:15 2024] ? _copy_to_iter+0x7e/0x630
[Mon Mar 11 14:29:15 2024] ? eventfd_read+0xe2/0x2b0
[Mon Mar 11 14:29:15 2024] ? __audit_filter_op+0xa5/0xf0
[Mon Mar 11 14:29:15 2024] ? auditd_test_task+0x3c/0x50
[Mon Mar 11 14:29:15 2024] ? __audit_syscall_entry+0xef/0x140
[Mon Mar 11 14:29:15 2024] __x64_sys_poll+0x39/0x140
[Mon Mar 11 14:29:15 2024] ? syscall_trace_enter.constprop.0+0x126/0x1a0
[Mon Mar 11 14:29:15 2024] do_syscall_64+0x59/0x90
[Mon Mar 11 14:29:15 2024] ? syscall_exit_to_user_mode+0x22/0x40
[Mon Mar 11 14:29:15 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:15 2024] ? syscall_exit_work+0x103/0x130
[Mon Mar 11 14:29:15 2024] ? syscall_exit_to_user_mode+0x22/0x40
[Mon Mar 11 14:29:15 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:15 2024] ? exc_page_fault+0x62/0x150
[Mon Mar 11 14:29:15 2024] entry_SYSCALL_64_after_hwframe+0x72/0xdc
[Mon Mar 11 14:29:15 2024] RIP: 0033:0x7f41a23426ff
[Mon Mar 11 14:29:15 2024] RSP: 002b:00007f41991fdbc0 EFLAGS: 00000293 ORIG_RAX: 0000000000000007
[Mon Mar 11 14:29:15 2024] RAX: ffffffffffffffda RBX: 00007f41a2830071 RCX: 00007f41a23426ff
[Mon Mar 11 14:29:15 2024] RDX: 00000000ffffffff RSI: 0000000000000002 RDI: 000055e840bb8740
[Mon Mar 11 14:29:15 2024] RBP: 000055e840bb8740 R08: 0000000000000000 R09: 0000000000000000
[Mon Mar 11 14:29:15 2024] R10: 00007fff02d83080 R11: 0000000000000293 R12: 0000000000000002
[Mon Mar 11 14:29:15 2024] R13: 0000000000000002 R14: 00007f41991fdc30 R15: 000055e840bb8620
[Mon Mar 11 14:29:15 2024] </TASK>
[Mon Mar 11 14:29:15 2024] task:gdbus state:S stack:0 pid:21808 ppid:1 flags:0x00000002
[Mon Mar 11 14:29:15 2024] Call Trace:
[Mon Mar 11 14:29:15 2024] <TASK>
[Mon Mar 11 14:29:15 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:15 2024] ? mod_objcg_state+0x1fe/0x300
[Mon Mar 11 14:29:15 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:15 2024] schedule_hrtimeout_range_clock+0x10f/0x120
[Mon Mar 11 14:29:15 2024] ? add_wait_queue+0x65/0xa0
[Mon Mar 11 14:29:15 2024] ? unix_poll+0xf4/0x100
[Mon Mar 11 14:29:15 2024] ? sock_poll+0x4c/0xe0
[Mon Mar 11 14:29:15 2024] do_poll.constprop.0+0x248/0x390
[Mon Mar 11 14:29:15 2024] do_sys_poll+0x1c8/0x250
[Mon Mar 11 14:29:15 2024] ? __alloc_skb+0xea/0x1d0
[Mon Mar 11 14:29:15 2024] ? _copy_from_iter+0x144/0x590
[Mon Mar 11 14:29:15 2024] ? ep_poll_callback+0x25d/0x2a0
[Mon Mar 11 14:29:15 2024] ? __wake_up_common+0x75/0xa0
[Mon Mar 11 14:29:15 2024] ? __pfx_pollwake+0x10/0x10
[Mon Mar 11 14:29:15 2024] ? __pfx_pollwake+0x10/0x10
[Mon Mar 11 14:29:15 2024] ? ____sys_sendmsg+0x31c/0x340
[Mon Mar 11 14:29:15 2024] ? import_iovec+0x17/0x20
[Mon Mar 11 14:29:15 2024] ? copy_msghdr_from_user+0x6d/0xa0
[Mon Mar 11 14:29:15 2024] ? ___sys_sendmsg+0x95/0xd0
[Mon Mar 11 14:29:15 2024] ? __audit_filter_op+0xa5/0xf0
[Mon Mar 11 14:29:15 2024] ? auditd_test_task+0x3c/0x50
[Mon Mar 11 14:29:15 2024] __x64_sys_poll+0x39/0x140
[Mon Mar 11 14:29:15 2024] ? syscall_trace_enter.constprop.0+0x126/0x1a0
[Mon Mar 11 14:29:15 2024] do_syscall_64+0x59/0x90
[Mon Mar 11 14:29:15 2024] ? audit_reset_context.part.0.constprop.0+0x273/0x2e0
[Mon Mar 11 14:29:15 2024] ? syscall_exit_work+0x103/0x130
[Mon Mar 11 14:29:15 2024] ? syscall_exit_to_user_mode+0x22/0x40
[Mon Mar 11 14:29:15 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:15 2024] ? syscall_exit_to_user_mode+0x22/0x40
[Mon Mar 11 14:29:15 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:15 2024] ? syscall_exit_to_user_mode+0x22/0x40
[Mon Mar 11 14:29:15 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:15 2024] ? __irq_exit_rcu+0x46/0xc0
[Mon Mar 11 14:29:15 2024] ? sysvec_apic_timer_interrupt+0x3c/0x90
[Mon Mar 11 14:29:15 2024] entry_SYSCALL_64_after_hwframe+0x72/0xdc
[Mon Mar 11 14:29:15 2024] RIP: 0033:0x7f41a23426ff
[Mon Mar 11 14:29:15 2024] RSP: 002b:00007f41a09fcbc0 EFLAGS: 00000293 ORIG_RAX: 0000000000000007
[Mon Mar 11 14:29:15 2024] RAX: ffffffffffffffda RBX: 00007f41a2830071 RCX: 00007f41a23426ff
[Mon Mar 11 14:29:15 2024] RDX: 00000000ffffffff RSI: 0000000000000002 RDI: 00007f418c0107d0
[Mon Mar 11 14:29:15 2024] RBP: 00007f418c0107d0 R08: 0000000000000000 R09: 00007f41a09fca50
[Mon Mar 11 14:29:15 2024] R10: 00007fff02d83080 R11: 0000000000000293 R12: 0000000000000002
[Mon Mar 11 14:29:15 2024] R13: 0000000000000002 R14: 00007f41a09fcc30 R15: 00007f418c00d9c0
[Mon Mar 11 14:29:15 2024] </TASK>
[Mon Mar 11 14:29:15 2024] task:smbd[10.91.30.5 state:S stack:0 pid:86031 ppid:4757 flags:0x00000002
[Mon Mar 11 14:29:15 2024] Call Trace:
[Mon Mar 11 14:29:15 2024] <TASK>
[Mon Mar 11 14:29:15 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:15 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:15 2024] schedule_hrtimeout_range_clock+0x95/0x120
[Mon Mar 11 14:29:15 2024] ? __pfx_hrtimer_wakeup+0x10/0x10
[Mon Mar 11 14:29:15 2024] ep_poll+0x348/0x3b0
[Mon Mar 11 14:29:15 2024] ? __pfx_autoremove_wake_function+0x10/0x10
[Mon Mar 11 14:29:15 2024] do_epoll_wait+0xb1/0xd0
[Mon Mar 11 14:29:15 2024] __x64_sys_epoll_wait+0x60/0x100
[Mon Mar 11 14:29:15 2024] do_syscall_64+0x59/0x90
[Mon Mar 11 14:29:15 2024] ? __audit_filter_op+0xa5/0xf0
[Mon Mar 11 14:29:15 2024] ? audit_reset_context.part.0.constprop.0+0x273/0x2e0
[Mon Mar 11 14:29:15 2024] ? syscall_exit_work+0x103/0x130
[Mon Mar 11 14:29:15 2024] ? syscall_exit_to_user_mode+0x22/0x40
[Mon Mar 11 14:29:15 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:15 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:15 2024] ? exc_page_fault+0x62/0x150
[Mon Mar 11 14:29:15 2024] entry_SYSCALL_64_after_hwframe+0x72/0xdc
[Mon Mar 11 14:29:15 2024] RIP: 0033:0x7fedb7d4e84e
[Mon Mar 11 14:29:15 2024] RSP: 002b:00007ffc284a9ab0 EFLAGS: 00000293 ORIG_RAX: 00000000000000e8
[Mon Mar 11 14:29:15 2024] RAX: ffffffffffffffda RBX: 000055f87dd397c0 RCX: 00007fedb7d4e84e
[Mon Mar 11 14:29:15 2024] RDX: 0000000000000001 RSI: 00007ffc284a9b0c RDI: 0000000000000005
[Mon Mar 11 14:29:15 2024] RBP: 000055f87dd2aa20 R08: 0000000000000000 R09: 0000000000000000
[Mon Mar 11 14:29:15 2024] R10: 000000000000bbef R11: 0000000000000293 R12: 000055f87dd58980
[Mon Mar 11 14:29:15 2024] R13: 000000000000bbef R14: 0000000000000027 R15: 000055f87dd5b760
[Mon Mar 11 14:29:15 2024] </TASK>
[Mon Mar 11 14:29:15 2024] task:automount state:S stack:0 pid:88092 ppid:1 flags:0x00000002
[Mon Mar 11 14:29:15 2024] Call Trace:
[Mon Mar 11 14:29:15 2024] <TASK>
[Mon Mar 11 14:29:15 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:15 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:15 2024] schedule_hrtimeout_range_clock+0x10f/0x120
[Mon Mar 11 14:29:15 2024] ? rseq_ip_fixup+0x6e/0x1a0
[Mon Mar 11 14:29:15 2024] ? dequeue_signal+0x68/0x200
[Mon Mar 11 14:29:15 2024] do_sigtimedwait+0x16f/0x210
[Mon Mar 11 14:29:15 2024] __x64_sys_rt_sigtimedwait+0x6e/0xe0
[Mon Mar 11 14:29:15 2024] ? syscall_trace_enter.constprop.0+0x126/0x1a0
[Mon Mar 11 14:29:15 2024] do_syscall_64+0x59/0x90
[Mon Mar 11 14:29:15 2024] ? do_user_addr_fault+0x1d6/0x6a0
[Mon Mar 11 14:29:15 2024] ? syscall_exit_to_user_mode+0x22/0x40
[Mon Mar 11 14:29:15 2024] ? exc_page_fault+0x62/0x150
[Mon Mar 11 14:29:15 2024] entry_SYSCALL_64_after_hwframe+0x72/0xdc
[Mon Mar 11 14:29:15 2024] RIP: 0033:0x7f66af255aca
[Mon Mar 11 14:29:15 2024] RSP: 002b:00007ffd9217b0c0 EFLAGS: 00000293 ORIG_RAX: 0000000000000080
[Mon Mar 11 14:29:15 2024] RAX: ffffffffffffffda RBX: 00007ffd9217b0f0 RCX: 00007f66af255aca
[Mon Mar 11 14:29:15 2024] RDX: 0000000000000000 RSI: 00007ffd9217b0f0 RDI: 00007ffd9217b230
[Mon Mar 11 14:29:15 2024] RBP: 00007ffd9217b0f0 R08: 0000000000000000 R09: 0000000000000000
[Mon Mar 11 14:29:15 2024] R10: 0000000000000008 R11: 0000000000000293 R12: 00007ffd9217b1f8
[Mon Mar 11 14:29:15 2024] R13: 000055b9efc16220 R14: 00007f66af825110 R15: 00007f66af825b10
[Mon Mar 11 14:29:15 2024] </TASK>
[Mon Mar 11 14:29:15 2024] task:automount state:S stack:0 pid:88101 ppid:1 flags:0x00000002
[Mon Mar 11 14:29:15 2024] Call Trace:
[Mon Mar 11 14:29:15 2024] <TASK>
[Mon Mar 11 14:29:15 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:15 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:15 2024] schedule_hrtimeout_range_clock+0x10f/0x120
[Mon Mar 11 14:29:15 2024] ? add_wait_queue+0x65/0xa0
[Mon Mar 11 14:29:15 2024] do_poll.constprop.0+0x248/0x390
[Mon Mar 11 14:29:15 2024] do_sys_poll+0x1c8/0x250
[Mon Mar 11 14:29:15 2024] ? context_to_sid+0x95/0x110
[Mon Mar 11 14:29:15 2024] ? sidtab_context_to_sid+0x37/0x430
[Mon Mar 11 14:29:15 2024] ? mls_level_isvalid+0x3e/0x70
[Mon Mar 11 14:29:15 2024] ? __pfx_pollwake+0x10/0x10
[Mon Mar 11 14:29:15 2024] ? __pfx_pollwake+0x10/0x10
[Mon Mar 11 14:29:15 2024] ? __wake_up+0x40/0x60
[Mon Mar 11 14:29:15 2024] ? ima_file_check+0x53/0x80
[Mon Mar 11 14:29:15 2024] ? mntput_no_expire+0x4a/0x250
[Mon Mar 11 14:29:15 2024] ? terminate_walk+0x61/0xf0
[Mon Mar 11 14:29:15 2024] ? path_openat+0xc1/0x280
[Mon Mar 11 14:29:15 2024] ? do_filp_open+0xb2/0x160
[Mon Mar 11 14:29:15 2024] ? _copy_from_user+0x27/0x60
[Mon Mar 11 14:29:15 2024] __x64_sys_ppoll+0xbc/0x150
[Mon Mar 11 14:29:15 2024] do_syscall_64+0x59/0x90
[Mon Mar 11 14:29:15 2024] ? syscall_exit_to_user_mode+0x22/0x40
[Mon Mar 11 14:29:15 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:15 2024] ? syscall_exit_work+0x103/0x130
[Mon Mar 11 14:29:15 2024] ? syscall_exit_to_user_mode+0x22/0x40
[Mon Mar 11 14:29:15 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:15 2024] ? syscall_exit_to_user_mode+0x22/0x40
[Mon Mar 11 14:29:15 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:15 2024] ? exc_page_fault+0x62/0x150
[Mon Mar 11 14:29:15 2024] entry_SYSCALL_64_after_hwframe+0x72/0xdc
[Mon Mar 11 14:29:15 2024] RIP: 0033:0x7f66af3427fe
[Mon Mar 11 14:29:15 2024] RSP: 002b:00007f66ae90fa10 EFLAGS: 00000293 ORIG_RAX: 000000000000010f
[Mon Mar 11 14:29:15 2024] RAX: ffffffffffffffda RBX: 00007f66ae910528 RCX: 00007f66af3427fe
[Mon Mar 11 14:29:15 2024] RDX: 0000000000000000 RSI: 0000000000000001 RDI: 00007f66ae90fa68
[Mon Mar 11 14:29:15 2024] RBP: 00007f66ae90fa70 R08: 0000000000000008 R09: 0000000000000000
[Mon Mar 11 14:29:15 2024] R10: 00007f66ae90fa70 R11: 0000000000000293 R12: 00007f66ae90fa68
[Mon Mar 11 14:29:15 2024] R13: 0000000000000000 R14: 000055b9efc09f20 R15: 0000000000000000
[Mon Mar 11 14:29:15 2024] </TASK>
[Mon Mar 11 14:29:15 2024] task:automount state:S stack:0 pid:88102 ppid:1 flags:0x00000002
[Mon Mar 11 14:29:15 2024] Call Trace:
[Mon Mar 11 14:29:15 2024] <TASK>
[Mon Mar 11 14:29:15 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:15 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:15 2024] futex_wait_queue+0x70/0xd0
[Mon Mar 11 14:29:15 2024] futex_wait+0x175/0x260
[Mon Mar 11 14:29:15 2024] ? sched_clock+0xc/0x30
[Mon Mar 11 14:29:15 2024] ? __pfx_hrtimer_wakeup+0x10/0x10
[Mon Mar 11 14:29:15 2024] do_futex+0x12d/0x1d0
[Mon Mar 11 14:29:15 2024] __x64_sys_futex+0x73/0x1d0
[Mon Mar 11 14:29:15 2024] do_syscall_64+0x59/0x90
[Mon Mar 11 14:29:15 2024] ? syscall_exit_to_user_mode+0x22/0x40
[Mon Mar 11 14:29:15 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:15 2024] ? sysvec_apic_timer_interrupt+0x3c/0x90
[Mon Mar 11 14:29:15 2024] entry_SYSCALL_64_after_hwframe+0x72/0xdc
[Mon Mar 11 14:29:15 2024] RIP: 0033:0x7f66af29c39a
[Mon Mar 11 14:29:15 2024] RSP: 002b:00007f66ae10eb60 EFLAGS: 00000246 ORIG_RAX: 00000000000000ca
[Mon Mar 11 14:29:15 2024] RAX: ffffffffffffffda RBX: 00007f66ae10ec70 RCX: 00007f66af29c39a
[Mon Mar 11 14:29:15 2024] RDX: 0000000000000000 RSI: 0000000000000089 RDI: 00007f66af587c6c
[Mon Mar 11 14:29:15 2024] RBP: 0000000000000000 R08: 0000000000000000 R09: 00000000ffffffff
[Mon Mar 11 14:29:15 2024] R10: 00007f66ae10ec70 R11: 0000000000000246 R12: 0000000000000000
[Mon Mar 11 14:29:15 2024] R13: 00007f66af587c6c R14: 0000000000000000 R15: 00007f66af587c00
[Mon Mar 11 14:29:15 2024] </TASK>
[Mon Mar 11 14:29:15 2024] task:automount state:S stack:0 pid:88103 ppid:1 flags:0x00000002
[Mon Mar 11 14:29:15 2024] Call Trace:
[Mon Mar 11 14:29:15 2024] <TASK>
[Mon Mar 11 14:29:15 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:15 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:15 2024] futex_wait_queue+0x70/0xd0
[Mon Mar 11 14:29:15 2024] futex_wait+0x175/0x260
[Mon Mar 11 14:29:15 2024] ? enqueue_task_fair+0x88/0x3d0
[Mon Mar 11 14:29:15 2024] ? rseq_get_rseq_cs+0x1d/0x240
[Mon Mar 11 14:29:15 2024] do_futex+0x12d/0x1d0
[Mon Mar 11 14:29:15 2024] __x64_sys_futex+0x73/0x1d0
[Mon Mar 11 14:29:15 2024] do_syscall_64+0x59/0x90
[Mon Mar 11 14:29:15 2024] ? audit_reset_context.part.0.constprop.0+0x273/0x2e0
[Mon Mar 11 14:29:15 2024] ? syscall_exit_work+0x103/0x130
[Mon Mar 11 14:29:15 2024] ? syscall_exit_to_user_mode+0x22/0x40
[Mon Mar 11 14:29:15 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:15 2024] ? syscall_exit_work+0x103/0x130
[Mon Mar 11 14:29:15 2024] ? syscall_exit_to_user_mode+0x22/0x40
[Mon Mar 11 14:29:15 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:15 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:15 2024] ? sysvec_apic_timer_interrupt+0x3c/0x90
[Mon Mar 11 14:29:15 2024] entry_SYSCALL_64_after_hwframe+0x72/0xdc
[Mon Mar 11 14:29:15 2024] RIP: 0033:0x7f66af29c39a
[Mon Mar 11 14:29:15 2024] RSP: 002b:00007f66ad90dba0 EFLAGS: 00000246 ORIG_RAX: 00000000000000ca
[Mon Mar 11 14:29:15 2024] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f66af29c39a
[Mon Mar 11 14:29:15 2024] RDX: 0000000000000000 RSI: 0000000000000189 RDI: 000055b9efc171c8
[Mon Mar 11 14:29:15 2024] RBP: 0000000000000000 R08: 0000000000000000 R09: 00000000ffffffff
[Mon Mar 11 14:29:15 2024] R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000000000
[Mon Mar 11 14:29:15 2024] R13: 000055b9efc171c8 R14: 0000000000000000 R15: 0000000000000000
[Mon Mar 11 14:29:15 2024] </TASK>
[Mon Mar 11 14:29:15 2024] task:automount state:S stack:0 pid:88118 ppid:1 flags:0x00000002
[Mon Mar 11 14:29:15 2024] Call Trace:
[Mon Mar 11 14:29:15 2024] <TASK>
[Mon Mar 11 14:29:15 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:15 2024] ? __update_idle_core+0x23/0xc0
[Mon Mar 11 14:29:15 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:15 2024] schedule_hrtimeout_range_clock+0x10f/0x120
[Mon Mar 11 14:29:15 2024] ? add_wait_queue+0x65/0xa0
[Mon Mar 11 14:29:15 2024] do_poll.constprop.0+0x248/0x390
[Mon Mar 11 14:29:15 2024] do_sys_poll+0x1c8/0x250
[Mon Mar 11 14:29:15 2024] ? do_sys_poll+0x1d8/0x250
[Mon Mar 11 14:29:15 2024] ? update_sg_wakeup_stats+0x78/0x3b0
[Mon Mar 11 14:29:15 2024] ? newidle_balance+0x2e5/0x400
[Mon Mar 11 14:29:15 2024] ? update_load_avg+0x7e/0x740
[Mon Mar 11 14:29:15 2024] ? __pfx_pollwake+0x10/0x10
[Mon Mar 11 14:29:15 2024] ? __pfx_pollwake+0x10/0x10
[Mon Mar 11 14:29:15 2024] ? select_idle_sibling+0x28/0x430
[Mon Mar 11 14:29:15 2024] ? wake_affine+0x62/0x1f0
[Mon Mar 11 14:29:15 2024] ? sched_clock+0xc/0x30
[Mon Mar 11 14:29:15 2024] ? __smp_call_single_queue+0x93/0x120
[Mon Mar 11 14:29:15 2024] ? ttwu_queue_wakelist+0xf2/0x110
[Mon Mar 11 14:29:15 2024] ? try_to_wake_up+0x3e2/0x5d0
[Mon Mar 11 14:29:15 2024] ? autoremove_wake_function+0x30/0x60
[Mon Mar 11 14:29:15 2024] ? __wake_up_common+0x75/0xa0
[Mon Mar 11 14:29:15 2024] ? __wake_up+0x40/0x60
[Mon Mar 11 14:29:15 2024] __x64_sys_ppoll+0xbc/0x150
[Mon Mar 11 14:29:15 2024] do_syscall_64+0x59/0x90
[Mon Mar 11 14:29:15 2024] ? syscall_exit_work+0x103/0x130
[Mon Mar 11 14:29:15 2024] ? syscall_exit_to_user_mode+0x22/0x40
[Mon Mar 11 14:29:15 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:15 2024] ? syscall_exit_to_user_mode+0x22/0x40
[Mon Mar 11 14:29:15 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:15 2024] entry_SYSCALL_64_after_hwframe+0x72/0xdc
[Mon Mar 11 14:29:15 2024] RIP: 0033:0x7f66af3427fe
[Mon Mar 11 14:29:15 2024] RSP: 002b:00007f66ad10c8a0 EFLAGS: 00000293 ORIG_RAX: 000000000000010f
[Mon Mar 11 14:29:15 2024] RAX: ffffffffffffffda RBX: 00007f66ad10d528 RCX: 00007f66af3427fe
[Mon Mar 11 14:29:15 2024] RDX: 0000000000000000 RSI: 0000000000000001 RDI: 00007f66ad10c8f8
[Mon Mar 11 14:29:15 2024] RBP: 00007f66ad10c900 R08: 0000000000000008 R09: 0000000000000000
[Mon Mar 11 14:29:15 2024] R10: 00007f66ad10c900 R11: 0000000000000293 R12: 00007ffd9217b000
[Mon Mar 11 14:29:15 2024] R13: 0000000000000000 R14: 000055b9f0c671c0 R15: 0000000000000000
[Mon Mar 11 14:29:15 2024] </TASK>
[Mon Mar 11 14:29:15 2024] task:automount state:S stack:0 pid:88121 ppid:1 flags:0x00000002
[Mon Mar 11 14:29:15 2024] Call Trace:
[Mon Mar 11 14:29:15 2024] <TASK>
[Mon Mar 11 14:29:15 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:15 2024] ? do_user_addr_fault+0x1d6/0x6a0
[Mon Mar 11 14:29:15 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:15 2024] schedule_hrtimeout_range_clock+0x10f/0x120
[Mon Mar 11 14:29:15 2024] ? asm_exc_page_fault+0x22/0x30
[Mon Mar 11 14:29:15 2024] ? add_wait_queue+0x65/0xa0
[Mon Mar 11 14:29:15 2024] do_poll.constprop.0+0x248/0x390
[Mon Mar 11 14:29:15 2024] do_sys_poll+0x1c8/0x250
[Mon Mar 11 14:29:15 2024] ? find_idlest_group+0x2b2/0x530
[Mon Mar 11 14:29:15 2024] ? newidle_balance+0x2e5/0x400
[Mon Mar 11 14:29:15 2024] ? update_load_avg+0x7e/0x740
[Mon Mar 11 14:29:15 2024] ? __pfx_pollwake+0x10/0x10
[Mon Mar 11 14:29:15 2024] ? finish_task_switch.isra.0+0x8c/0x2a0
[Mon Mar 11 14:29:15 2024] ? __schedule+0x223/0x550
[Mon Mar 11 14:29:15 2024] ? timerqueue_del+0x2a/0x50
[Mon Mar 11 14:29:15 2024] ? __remove_hrtimer+0x39/0x90
[Mon Mar 11 14:29:15 2024] ? hrtimer_try_to_cancel.part.0+0x50/0xf0
[Mon Mar 11 14:29:15 2024] ? hrtimer_cancel+0x1d/0x40
[Mon Mar 11 14:29:15 2024] ? futex_wait+0x23e/0x260
[Mon Mar 11 14:29:15 2024] ? __pfx_hrtimer_wakeup+0x10/0x10
[Mon Mar 11 14:29:15 2024] ? __audit_filter_op+0xa5/0xf0
[Mon Mar 11 14:29:15 2024] ? audit_reset_context.part.0.constprop.0+0x273/0x2e0
[Mon Mar 11 14:29:15 2024] __x64_sys_ppoll+0xbc/0x150
[Mon Mar 11 14:29:15 2024] do_syscall_64+0x59/0x90
[Mon Mar 11 14:29:15 2024] ? exc_page_fault+0x62/0x150
[Mon Mar 11 14:29:15 2024] entry_SYSCALL_64_after_hwframe+0x72/0xdc
[Mon Mar 11 14:29:15 2024] RIP: 0033:0x7f66af3427fe
[Mon Mar 11 14:29:15 2024] RSP: 002b:00007f66ac9018a0 EFLAGS: 00000293 ORIG_RAX: 000000000000010f
[Mon Mar 11 14:29:15 2024] RAX: ffffffffffffffda RBX: 00007f66ac902528 RCX: 00007f66af3427fe
[Mon Mar 11 14:29:15 2024] RDX: 0000000000000000 RSI: 0000000000000001 RDI: 00007f66ac9018f8
[Mon Mar 11 14:29:15 2024] RBP: 00007f66ac901900 R08: 0000000000000008 R09: 0000000000000000
[Mon Mar 11 14:29:15 2024] R10: 00007f66ac901900 R11: 0000000000000293 R12: 00007ffd9217b000
[Mon Mar 11 14:29:15 2024] R13: 0000000000000000 R14: 000055b9f0c713a0 R15: 0000000000000000
[Mon Mar 11 14:29:15 2024] </TASK>
[Mon Mar 11 14:29:15 2024] task:automount state:S stack:0 pid:88122 ppid:1 flags:0x00000002
[Mon Mar 11 14:29:15 2024] Call Trace:
[Mon Mar 11 14:29:15 2024] <TASK>
[Mon Mar 11 14:29:15 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:15 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:15 2024] schedule_hrtimeout_range_clock+0x10f/0x120
[Mon Mar 11 14:29:15 2024] ? add_wait_queue+0x65/0xa0
[Mon Mar 11 14:29:15 2024] do_poll.constprop.0+0x248/0x390
[Mon Mar 11 14:29:15 2024] do_sys_poll+0x1c8/0x250
[Mon Mar 11 14:29:15 2024] ? ttwu_queue_wakelist+0xf2/0x110
[Mon Mar 11 14:29:15 2024] ? try_to_wake_up+0x3e2/0x5d0
[Mon Mar 11 14:29:15 2024] ? __alloc_skb+0x8e/0x1d0
[Mon Mar 11 14:29:15 2024] ? autoremove_wake_function+0x30/0x60
[Mon Mar 11 14:29:15 2024] ? __wake_up_common+0x75/0xa0
[Mon Mar 11 14:29:15 2024] ? __pfx_pollwake+0x10/0x10
[Mon Mar 11 14:29:15 2024] ? __wake_up_common+0x75/0xa0
[Mon Mar 11 14:29:15 2024] ? select_idle_sibling+0x28/0x430
[Mon Mar 11 14:29:15 2024] ? wake_affine+0x62/0x1f0
[Mon Mar 11 14:29:15 2024] ? sched_clock+0xc/0x30
[Mon Mar 11 14:29:15 2024] ? __smp_call_single_queue+0x93/0x120
[Mon Mar 11 14:29:15 2024] ? ttwu_queue_wakelist+0xf2/0x110
[Mon Mar 11 14:29:15 2024] ? try_to_wake_up+0x3e2/0x5d0
[Mon Mar 11 14:29:15 2024] ? __audit_filter_op+0xa5/0xf0
[Mon Mar 11 14:29:15 2024] ? audit_reset_context.part.0.constprop.0+0x273/0x2e0
[Mon Mar 11 14:29:15 2024] __x64_sys_ppoll+0xbc/0x150
[Mon Mar 11 14:29:15 2024] do_syscall_64+0x59/0x90
[Mon Mar 11 14:29:15 2024] ? syscall_exit_to_user_mode+0x22/0x40
[Mon Mar 11 14:29:15 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:15 2024] ? exc_page_fault+0x62/0x150
[Mon Mar 11 14:29:15 2024] entry_SYSCALL_64_after_hwframe+0x72/0xdc
[Mon Mar 11 14:29:15 2024] RIP: 0033:0x7f66af3427fe
[Mon Mar 11 14:29:15 2024] RSP: 002b:00007f66a7ffe8a0 EFLAGS: 00000293 ORIG_RAX: 000000000000010f
[Mon Mar 11 14:29:15 2024] RAX: ffffffffffffffda RBX: 00007f66a7fff528 RCX: 00007f66af3427fe
[Mon Mar 11 14:29:15 2024] RDX: 0000000000000000 RSI: 0000000000000001 RDI: 00007f66a7ffe8f8
[Mon Mar 11 14:29:15 2024] RBP: 00007f66a7ffe900 R08: 0000000000000008 R09: 0000000000000000
[Mon Mar 11 14:29:15 2024] R10: 00007f66a7ffe900 R11: 0000000000000293 R12: 00007ffd9217b000
[Mon Mar 11 14:29:15 2024] R13: 0000000000000000 R14: 000055b9f0c72290 R15: 0000000000000000
[Mon Mar 11 14:29:15 2024] </TASK>
[Mon Mar 11 14:29:15 2024] task:automount state:S stack:0 pid:88123 ppid:1 flags:0x00000002
[Mon Mar 11 14:29:15 2024] Call Trace:
[Mon Mar 11 14:29:15 2024] <TASK>
[Mon Mar 11 14:29:15 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:15 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:15 2024] schedule_hrtimeout_range_clock+0x10f/0x120
[Mon Mar 11 14:29:15 2024] ? add_wait_queue+0x65/0xa0
[Mon Mar 11 14:29:15 2024] do_poll.constprop.0+0x248/0x390
[Mon Mar 11 14:29:15 2024] do_sys_poll+0x1c8/0x250
[Mon Mar 11 14:29:15 2024] ? __kmem_cache_alloc_node+0x1c7/0x2d0
[Mon Mar 11 14:29:15 2024] ? __kmem_cache_alloc_node+0x1c7/0x2d0
[Mon Mar 11 14:29:15 2024] ? select_idle_sibling+0x28/0x430
[Mon Mar 11 14:29:15 2024] ? wake_affine+0x62/0x1f0
[Mon Mar 11 14:29:15 2024] ? sched_clock+0xc/0x30
[Mon Mar 11 14:29:15 2024] ? __smp_call_single_queue+0x93/0x120
[Mon Mar 11 14:29:15 2024] ? ttwu_queue_wakelist+0xf2/0x110
[Mon Mar 11 14:29:15 2024] ? __pfx_pollwake+0x10/0x10
[Mon Mar 11 14:29:15 2024] ? autoremove_wake_function+0x30/0x60
[Mon Mar 11 14:29:15 2024] ? __wake_up_common+0x75/0xa0
[Mon Mar 11 14:29:15 2024] ? __wake_up+0x40/0x60
[Mon Mar 11 14:29:15 2024] ? select_idle_sibling+0x28/0x430
[Mon Mar 11 14:29:15 2024] ? wake_affine+0x62/0x1f0
[Mon Mar 11 14:29:15 2024] ? ep_poll_callback+0x25d/0x2a0
[Mon Mar 11 14:29:15 2024] ? sched_clock+0xc/0x30
[Mon Mar 11 14:29:15 2024] ? __smp_call_single_queue+0x93/0x120
[Mon Mar 11 14:29:15 2024] ? ttwu_queue_wakelist+0xf2/0x110
[Mon Mar 11 14:29:15 2024] ? try_to_wake_up+0x3e2/0x5d0
[Mon Mar 11 14:29:15 2024] ? ttwu_queue_wakelist+0xf2/0x110
[Mon Mar 11 14:29:15 2024] ? futex_wake+0x151/0x180
[Mon Mar 11 14:29:15 2024] __x64_sys_ppoll+0xbc/0x150
[Mon Mar 11 14:29:15 2024] do_syscall_64+0x59/0x90
[Mon Mar 11 14:29:15 2024] ? syscall_exit_work+0x103/0x130
[Mon Mar 11 14:29:15 2024] ? syscall_exit_to_user_mode+0x22/0x40
[Mon Mar 11 14:29:15 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:15 2024] ? syscall_exit_work+0x103/0x130
[Mon Mar 11 14:29:15 2024] ? syscall_exit_to_user_mode+0x22/0x40
[Mon Mar 11 14:29:15 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:15 2024] entry_SYSCALL_64_after_hwframe+0x72/0xdc
[Mon Mar 11 14:29:15 2024] RIP: 0033:0x7f66af3427fe
[Mon Mar 11 14:29:15 2024] RSP: 002b:00007f66a77fd8a0 EFLAGS: 00000293 ORIG_RAX: 000000000000010f
[Mon Mar 11 14:29:15 2024] RAX: ffffffffffffffda RBX: 00007f66a77fe528 RCX: 00007f66af3427fe
[Mon Mar 11 14:29:15 2024] RDX: 0000000000000000 RSI: 0000000000000001 RDI: 00007f66a77fd8f8
[Mon Mar 11 14:29:15 2024] RBP: 00007f66a77fd900 R08: 0000000000000008 R09: 0000000000000000
[Mon Mar 11 14:29:15 2024] R10: 00007f66a77fd900 R11: 0000000000000293 R12: 00007ffd9217b000
[Mon Mar 11 14:29:15 2024] R13: 0000000000000000 R14: 000055b9f0c72620 R15: 0000000000000000
[Mon Mar 11 14:29:15 2024] </TASK>
[Mon Mar 11 14:29:15 2024] task:automount state:S stack:0 pid:88124 ppid:1 flags:0x00000002
[Mon Mar 11 14:29:15 2024] Call Trace:
[Mon Mar 11 14:29:15 2024] <TASK>
[Mon Mar 11 14:29:15 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:15 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:15 2024] schedule_hrtimeout_range_clock+0x10f/0x120
[Mon Mar 11 14:29:15 2024] ? add_wait_queue+0x65/0xa0
[Mon Mar 11 14:29:15 2024] do_poll.constprop.0+0x248/0x390
[Mon Mar 11 14:29:15 2024] do_sys_poll+0x1c8/0x250
[Mon Mar 11 14:29:15 2024] ? update_sg_wakeup_stats+0x78/0x3b0
[Mon Mar 11 14:29:15 2024] ? update_load_avg+0x7e/0x740
[Mon Mar 11 14:29:15 2024] ? set_next_entity+0xda/0x150
[Mon Mar 11 14:29:15 2024] ? pick_next_task_fair+0x1dc/0x500
[Mon Mar 11 14:29:15 2024] ? __pfx_pollwake+0x10/0x10
[Mon Mar 11 14:29:15 2024] ? __schedule+0x223/0x550
[Mon Mar 11 14:29:15 2024] ? timerqueue_del+0x2a/0x50
[Mon Mar 11 14:29:15 2024] ? __remove_hrtimer+0x39/0x90
[Mon Mar 11 14:29:15 2024] ? hrtimer_try_to_cancel.part.0+0x50/0xf0
[Mon Mar 11 14:29:15 2024] ? hrtimer_cancel+0x1d/0x40
[Mon Mar 11 14:29:15 2024] ? futex_wait+0x23e/0x260
[Mon Mar 11 14:29:15 2024] ? __pfx_hrtimer_wakeup+0x10/0x10
[Mon Mar 11 14:29:15 2024] ? rseq_get_rseq_cs+0x1d/0x240
[Mon Mar 11 14:29:15 2024] ? __audit_filter_op+0xa5/0xf0
[Mon Mar 11 14:29:15 2024] ? audit_reset_context.part.0.constprop.0+0x273/0x2e0
[Mon Mar 11 14:29:15 2024] ? _copy_from_user+0x27/0x60
[Mon Mar 11 14:29:15 2024] __x64_sys_ppoll+0xbc/0x150
[Mon Mar 11 14:29:15 2024] do_syscall_64+0x59/0x90
[Mon Mar 11 14:29:15 2024] ? syscall_exit_to_user_mode+0x22/0x40
[Mon Mar 11 14:29:15 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:15 2024] ? syscall_exit_work+0x103/0x130
[Mon Mar 11 14:29:15 2024] ? syscall_exit_to_user_mode+0x22/0x40
[Mon Mar 11 14:29:15 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:15 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:15 2024] ? exc_page_fault+0x62/0x150
[Mon Mar 11 14:29:15 2024] entry_SYSCALL_64_after_hwframe+0x72/0xdc
[Mon Mar 11 14:29:15 2024] RIP: 0033:0x7f66af3427fe
[Mon Mar 11 14:29:15 2024] RSP: 002b:00007f66a6ffc8a0 EFLAGS: 00000293 ORIG_RAX: 000000000000010f
[Mon Mar 11 14:29:15 2024] RAX: ffffffffffffffda RBX: 00007f66a6ffd528 RCX: 00007f66af3427fe
[Mon Mar 11 14:29:15 2024] RDX: 0000000000000000 RSI: 0000000000000001 RDI: 00007f66a6ffc8f8
[Mon Mar 11 14:29:15 2024] RBP: 00007f66a6ffc900 R08: 0000000000000008 R09: 0000000000000000
[Mon Mar 11 14:29:15 2024] R10: 00007f66a6ffc900 R11: 0000000000000293 R12: 00007ffd9217b000
[Mon Mar 11 14:29:15 2024] R13: 0000000000000000 R14: 000055b9f0c9f920 R15: 0000000000000000
[Mon Mar 11 14:29:15 2024] </TASK>
[Mon Mar 11 14:29:15 2024] task:automount state:S stack:0 pid:88125 ppid:1 flags:0x00000002
[Mon Mar 11 14:29:15 2024] Call Trace:
[Mon Mar 11 14:29:15 2024] <TASK>
[Mon Mar 11 14:29:15 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:15 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:15 2024] schedule_hrtimeout_range_clock+0x10f/0x120
[Mon Mar 11 14:29:15 2024] ? __ext4_handle_dirty_metadata+0x58/0x180 [ext4]
[Mon Mar 11 14:29:15 2024] ? add_wait_queue+0x65/0xa0
[Mon Mar 11 14:29:15 2024] do_poll.constprop.0+0x248/0x390
[Mon Mar 11 14:29:15 2024] do_sys_poll+0x1c8/0x250
[Mon Mar 11 14:29:15 2024] ? try_to_wake_up+0x3e2/0x5d0
[Mon Mar 11 14:29:15 2024] ? __alloc_skb+0x8e/0x1d0
[Mon Mar 11 14:29:15 2024] ? autoremove_wake_function+0x30/0x60
[Mon Mar 11 14:29:15 2024] ? __wake_up_common+0x75/0xa0
[Mon Mar 11 14:29:15 2024] ? __wake_up+0x40/0x60
[Mon Mar 11 14:29:15 2024] ? __pfx_pollwake+0x10/0x10
[Mon Mar 11 14:29:15 2024] ? select_idle_sibling+0x28/0x430
[Mon Mar 11 14:29:15 2024] ? __wake_up_sync_key+0x39/0x50
[Mon Mar 11 14:29:15 2024] ? wake_affine+0x62/0x1f0
[Mon Mar 11 14:29:15 2024] ? sched_clock+0xc/0x30
[Mon Mar 11 14:29:15 2024] ? __smp_call_single_queue+0x93/0x120
[Mon Mar 11 14:29:15 2024] ? ttwu_queue_wakelist+0xf2/0x110
[Mon Mar 11 14:29:15 2024] ? try_to_wake_up+0x3e2/0x5d0
[Mon Mar 11 14:29:15 2024] ? ttwu_queue_wakelist+0xf2/0x110
[Mon Mar 11 14:29:15 2024] ? futex_wake+0x151/0x180
[Mon Mar 11 14:29:15 2024] ? __audit_filter_op+0xa5/0xf0
[Mon Mar 11 14:29:15 2024] ? __audit_filter_op+0xa5/0xf0
[Mon Mar 11 14:29:15 2024] ? _copy_from_user+0x27/0x60
[Mon Mar 11 14:29:15 2024] __x64_sys_ppoll+0xbc/0x150
[Mon Mar 11 14:29:15 2024] do_syscall_64+0x59/0x90
[Mon Mar 11 14:29:15 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:15 2024] ? exc_page_fault+0x62/0x150
[Mon Mar 11 14:29:15 2024] entry_SYSCALL_64_after_hwframe+0x72/0xdc
[Mon Mar 11 14:29:15 2024] RIP: 0033:0x7f66af3427fe
[Mon Mar 11 14:29:15 2024] RSP: 002b:00007f66a67fb8a0 EFLAGS: 00000293 ORIG_RAX: 000000000000010f
[Mon Mar 11 14:29:15 2024] RAX: ffffffffffffffda RBX: 00007f66a67fc528 RCX: 00007f66af3427fe
[Mon Mar 11 14:29:15 2024] RDX: 0000000000000000 RSI: 0000000000000001 RDI: 00007f66a67fb8f8
[Mon Mar 11 14:29:15 2024] RBP: 00007f66a67fb900 R08: 0000000000000008 R09: 0000000000000000
[Mon Mar 11 14:29:15 2024] R10: 00007f66a67fb900 R11: 0000000000000293 R12: 00007ffd9217b000
[Mon Mar 11 14:29:15 2024] R13: 0000000000000000 R14: 000055b9f0ca5cb0 R15: 0000000000000000
[Mon Mar 11 14:29:15 2024] </TASK>
[Mon Mar 11 14:29:15 2024] task:automount state:S stack:0 pid:88126 ppid:1 flags:0x00000002
[Mon Mar 11 14:29:15 2024] Call Trace:
[Mon Mar 11 14:29:15 2024] <TASK>
[Mon Mar 11 14:29:15 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:15 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:15 2024] schedule_hrtimeout_range_clock+0x10f/0x120
[Mon Mar 11 14:29:15 2024] ? handle_mm_fault+0xcd/0x290
[Mon Mar 11 14:29:15 2024] ? add_wait_queue+0x65/0xa0
[Mon Mar 11 14:29:15 2024] do_poll.constprop.0+0x248/0x390
[Mon Mar 11 14:29:15 2024] do_sys_poll+0x1c8/0x250
[Mon Mar 11 14:29:15 2024] ? chacha_block_generic+0x6f/0xb0
[Mon Mar 11 14:29:15 2024] ? update_sg_wakeup_stats+0x78/0x3b0
[Mon Mar 11 14:29:15 2024] ? __pfx_pollwake+0x10/0x10
[Mon Mar 11 14:29:15 2024] ? pick_next_task_fair+0x41/0x500
[Mon Mar 11 14:29:15 2024] ? put_prev_task_fair+0x1e/0x40
[Mon Mar 11 14:29:15 2024] ? pick_next_task+0x861/0x950
[Mon Mar 11 14:29:15 2024] ? __update_idle_core+0x23/0xc0
[Mon Mar 11 14:29:15 2024] ? __switch_to_asm+0x3a/0x80
[Mon Mar 11 14:29:15 2024] ? finish_task_switch.isra.0+0x8c/0x2a0
[Mon Mar 11 14:29:15 2024] ? __schedule+0x223/0x550
[Mon Mar 11 14:29:15 2024] ? timerqueue_del+0x2a/0x50
[Mon Mar 11 14:29:15 2024] ? __remove_hrtimer+0x39/0x90
[Mon Mar 11 14:29:15 2024] ? hrtimer_try_to_cancel.part.0+0x50/0xf0
[Mon Mar 11 14:29:15 2024] ? hrtimer_cancel+0x1d/0x40
[Mon Mar 11 14:29:15 2024] ? futex_wait+0x23e/0x260
[Mon Mar 11 14:29:15 2024] ? __pfx_hrtimer_wakeup+0x10/0x10
[Mon Mar 11 14:29:15 2024] __x64_sys_ppoll+0xbc/0x150
[Mon Mar 11 14:29:15 2024] do_syscall_64+0x59/0x90
[Mon Mar 11 14:29:15 2024] ? audit_reset_context.part.0.constprop.0+0x273/0x2e0
[Mon Mar 11 14:29:15 2024] ? syscall_exit_work+0x103/0x130
[Mon Mar 11 14:29:15 2024] ? syscall_exit_to_user_mode+0x22/0x40
[Mon Mar 11 14:29:15 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:15 2024] ? syscall_exit_to_user_mode+0x22/0x40
[Mon Mar 11 14:29:15 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:15 2024] ? exc_page_fault+0x62/0x150
[Mon Mar 11 14:29:15 2024] entry_SYSCALL_64_after_hwframe+0x72/0xdc
[Mon Mar 11 14:29:15 2024] RIP: 0033:0x7f66af3427fe
[Mon Mar 11 14:29:15 2024] RSP: 002b:00007f66a5ffa8a0 EFLAGS: 00000293 ORIG_RAX: 000000000000010f
[Mon Mar 11 14:29:15 2024] RAX: ffffffffffffffda RBX: 00007f66a5ffb528 RCX: 00007f66af3427fe
[Mon Mar 11 14:29:15 2024] RDX: 0000000000000000 RSI: 0000000000000001 RDI: 00007f66a5ffa8f8
[Mon Mar 11 14:29:15 2024] RBP: 00007f66a5ffa900 R08: 0000000000000008 R09: 0000000000000000
[Mon Mar 11 14:29:15 2024] R10: 00007f66a5ffa900 R11: 0000000000000293 R12: 00007ffd9217b000
[Mon Mar 11 14:29:15 2024] R13: 0000000000000000 R14: 000055b9f0cac040 R15: 0000000000000000
[Mon Mar 11 14:29:15 2024] </TASK>
[Mon Mar 11 14:29:15 2024] task:automount state:S stack:0 pid:88127 ppid:1 flags:0x00000002
[Mon Mar 11 14:29:15 2024] Call Trace:
[Mon Mar 11 14:29:15 2024] <TASK>
[Mon Mar 11 14:29:15 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:15 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:15 2024] schedule_hrtimeout_range_clock+0x10f/0x120
[Mon Mar 11 14:29:15 2024] ? add_wait_queue+0x65/0xa0
[Mon Mar 11 14:29:15 2024] do_poll.constprop.0+0x248/0x390
[Mon Mar 11 14:29:15 2024] do_sys_poll+0x1c8/0x250
[Mon Mar 11 14:29:15 2024] ? autoremove_wake_function+0x30/0x60
[Mon Mar 11 14:29:15 2024] ? __wake_up_common+0x75/0xa0
[Mon Mar 11 14:29:15 2024] ? __wake_up+0x40/0x60
[Mon Mar 11 14:29:15 2024] ? ep_poll_callback+0x25d/0x2a0
[Mon Mar 11 14:29:15 2024] ? __pfx_pollwake+0x10/0x10
[Mon Mar 11 14:29:15 2024] ? sched_clock+0xc/0x30
[Mon Mar 11 14:29:15 2024] ? __smp_call_single_queue+0x93/0x120
[Mon Mar 11 14:29:15 2024] ? ttwu_queue_wakelist+0xf2/0x110
[Mon Mar 11 14:29:15 2024] ? try_to_wake_up+0x3e2/0x5d0
[Mon Mar 11 14:29:15 2024] ? __smp_call_single_queue+0x93/0x120
[Mon Mar 11 14:29:15 2024] ? futex_wake+0x151/0x180
[Mon Mar 11 14:29:15 2024] ? __audit_filter_op+0xa5/0xf0
[Mon Mar 11 14:29:15 2024] ? audit_reset_context.part.0.constprop.0+0x273/0x2e0
[Mon Mar 11 14:29:15 2024] ? __audit_filter_op+0xa5/0xf0
[Mon Mar 11 14:29:15 2024] ? _copy_from_user+0x27/0x60
[Mon Mar 11 14:29:15 2024] __x64_sys_ppoll+0xbc/0x150
[Mon Mar 11 14:29:15 2024] do_syscall_64+0x59/0x90
[Mon Mar 11 14:29:15 2024] ? exc_page_fault+0x62/0x150
[Mon Mar 11 14:29:15 2024] entry_SYSCALL_64_after_hwframe+0x72/0xdc
[Mon Mar 11 14:29:15 2024] RIP: 0033:0x7f66af3427fe
[Mon Mar 11 14:29:15 2024] RSP: 002b:00007f66a57f98a0 EFLAGS: 00000293 ORIG_RAX: 000000000000010f
[Mon Mar 11 14:29:15 2024] RAX: ffffffffffffffda RBX: 00007f66a57fa528 RCX: 00007f66af3427fe
[Mon Mar 11 14:29:15 2024] RDX: 0000000000000000 RSI: 0000000000000001 RDI: 00007f66a57f98f8
[Mon Mar 11 14:29:15 2024] RBP: 00007f66a57f9900 R08: 0000000000000008 R09: 0000000000000000
[Mon Mar 11 14:29:15 2024] R10: 00007f66a57f9900 R11: 0000000000000293 R12: 00007ffd9217b000
[Mon Mar 11 14:29:15 2024] R13: 0000000000000000 R14: 000055b9f0cb23d0 R15: 0000000000000000
[Mon Mar 11 14:29:15 2024] </TASK>
[Mon Mar 11 14:29:15 2024] task:automount state:S stack:0 pid:88128 ppid:1 flags:0x00000002
[Mon Mar 11 14:29:15 2024] Call Trace:
[Mon Mar 11 14:29:15 2024] <TASK>
[Mon Mar 11 14:29:15 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:15 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:15 2024] schedule_hrtimeout_range_clock+0x10f/0x120
[Mon Mar 11 14:29:15 2024] ? add_wait_queue+0x65/0xa0
[Mon Mar 11 14:29:15 2024] do_poll.constprop.0+0x248/0x390
[Mon Mar 11 14:29:15 2024] do_sys_poll+0x1c8/0x250
[Mon Mar 11 14:29:15 2024] ? __ext4_handle_dirty_metadata+0x58/0x180 [ext4]
[Mon Mar 11 14:29:15 2024] ? select_idle_sibling+0x28/0x430
[Mon Mar 11 14:29:15 2024] ? wake_affine+0x62/0x1f0
[Mon Mar 11 14:29:15 2024] ? sched_clock+0xc/0x30
[Mon Mar 11 14:29:15 2024] ? __smp_call_single_queue+0x93/0x120
[Mon Mar 11 14:29:15 2024] ? ttwu_queue_wakelist+0xf2/0x110
[Mon Mar 11 14:29:15 2024] ? try_to_wake_up+0x3e2/0x5d0
[Mon Mar 11 14:29:15 2024] ? __alloc_skb+0x8e/0x1d0
[Mon Mar 11 14:29:15 2024] ? __pfx_pollwake+0x10/0x10
[Mon Mar 11 14:29:15 2024] ? __wake_up+0x40/0x60
[Mon Mar 11 14:29:15 2024] ? ep_poll_callback+0x25d/0x2a0
[Mon Mar 11 14:29:15 2024] ? __wake_up_common+0x75/0xa0
[Mon Mar 11 14:29:15 2024] ? select_idle_sibling+0x28/0x430
[Mon Mar 11 14:29:15 2024] ? wake_affine+0x62/0x1f0
[Mon Mar 11 14:29:15 2024] ? sched_clock+0xc/0x30
[Mon Mar 11 14:29:15 2024] ? __smp_call_single_queue+0x93/0x120
[Mon Mar 11 14:29:15 2024] ? ttwu_queue_wakelist+0xf2/0x110
[Mon Mar 11 14:29:15 2024] ? select_idle_sibling+0x28/0x430
[Mon Mar 11 14:29:15 2024] ? try_to_wake_up+0x3e2/0x5d0
[Mon Mar 11 14:29:15 2024] ? wake_affine+0x62/0x1f0
[Mon Mar 11 14:29:15 2024] ? sched_clock+0xc/0x30
[Mon Mar 11 14:29:15 2024] ? __smp_call_single_queue+0x93/0x120
[Mon Mar 11 14:29:15 2024] ? ttwu_queue_wakelist+0xf2/0x110
[Mon Mar 11 14:29:15 2024] ? try_to_wake_up+0x3e2/0x5d0
[Mon Mar 11 14:29:15 2024] __x64_sys_ppoll+0xbc/0x150
[Mon Mar 11 14:29:15 2024] do_syscall_64+0x59/0x90
[Mon Mar 11 14:29:15 2024] ? syscall_exit_work+0x103/0x130
[Mon Mar 11 14:29:15 2024] ? syscall_exit_to_user_mode+0x22/0x40
[Mon Mar 11 14:29:15 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:15 2024] ? syscall_exit_work+0x103/0x130
[Mon Mar 11 14:29:15 2024] ? syscall_exit_to_user_mode+0x22/0x40
[Mon Mar 11 14:29:15 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:15 2024] ? do_user_addr_fault+0x1d6/0x6a0
[Mon Mar 11 14:29:15 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:15 2024] ? exc_page_fault+0x62/0x150
[Mon Mar 11 14:29:15 2024] entry_SYSCALL_64_after_hwframe+0x72/0xdc
[Mon Mar 11 14:29:15 2024] RIP: 0033:0x7f66af3427fe
[Mon Mar 11 14:29:15 2024] RSP: 002b:00007f66a4ff88a0 EFLAGS: 00000293 ORIG_RAX: 000000000000010f
[Mon Mar 11 14:29:15 2024] RAX: ffffffffffffffda RBX: 00007f66a4ff9528 RCX: 00007f66af3427fe
[Mon Mar 11 14:29:15 2024] RDX: 0000000000000000 RSI: 0000000000000001 RDI: 00007f66a4ff88f8
[Mon Mar 11 14:29:15 2024] RBP: 00007f66a4ff8900 R08: 0000000000000008 R09: 0000000000000000
[Mon Mar 11 14:29:15 2024] R10: 00007f66a4ff8900 R11: 0000000000000293 R12: 00007ffd9217b000
[Mon Mar 11 14:29:15 2024] R13: 0000000000000000 R14: 000055b9f0cb8760 R15: 0000000000000000
[Mon Mar 11 14:29:15 2024] </TASK>
[Mon Mar 11 14:29:15 2024] task:automount state:S stack:0 pid:88129 ppid:1 flags:0x00000002
[Mon Mar 11 14:29:15 2024] Call Trace:
[Mon Mar 11 14:29:15 2024] <TASK>
[Mon Mar 11 14:29:15 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:15 2024] ? __find_get_block+0x1fb/0x370
[Mon Mar 11 14:29:15 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:15 2024] schedule_hrtimeout_range_clock+0x10f/0x120
[Mon Mar 11 14:29:15 2024] ? rmqueue+0x7d3/0xd40
[Mon Mar 11 14:29:15 2024] ? add_wait_queue+0x65/0xa0
[Mon Mar 11 14:29:15 2024] do_poll.constprop.0+0x248/0x390
[Mon Mar 11 14:29:15 2024] do_sys_poll+0x1c8/0x250
[Mon Mar 11 14:29:15 2024] ? sched_clock+0xc/0x30
[Mon Mar 11 14:29:15 2024] ? __smp_call_single_queue+0x93/0x120
[Mon Mar 11 14:29:15 2024] ? ttwu_queue_wakelist+0xf2/0x110
[Mon Mar 11 14:29:15 2024] ? try_to_wake_up+0x3e2/0x5d0
[Mon Mar 11 14:29:15 2024] ? __alloc_skb+0x8e/0x1d0
[Mon Mar 11 14:29:15 2024] ? autoremove_wake_function+0x30/0x60
[Mon Mar 11 14:29:15 2024] ? __wake_up_common+0x75/0xa0
[Mon Mar 11 14:29:15 2024] ? __pfx_pollwake+0x10/0x10
[Mon Mar 11 14:29:15 2024] ? __wake_up_common+0x75/0xa0
[Mon Mar 11 14:29:15 2024] ? select_idle_sibling+0x28/0x430
[Mon Mar 11 14:29:15 2024] ? wake_affine+0x62/0x1f0
[Mon Mar 11 14:29:15 2024] ? sched_clock+0xc/0x30
[Mon Mar 11 14:29:15 2024] ? __smp_call_single_queue+0x93/0x120
[Mon Mar 11 14:29:15 2024] ? ttwu_queue_wakelist+0xf2/0x110
[Mon Mar 11 14:29:15 2024] ? try_to_wake_up+0x3e2/0x5d0
[Mon Mar 11 14:29:15 2024] ? select_idle_sibling+0x28/0x430
[Mon Mar 11 14:29:15 2024] ? wake_affine+0x62/0x1f0
[Mon Mar 11 14:29:15 2024] ? sched_clock+0xc/0x30
[Mon Mar 11 14:29:15 2024] ? __smp_call_single_queue+0x93/0x120
[Mon Mar 11 14:29:15 2024] ? ttwu_queue_wakelist+0xf2/0x110
[Mon Mar 11 14:29:15 2024] ? _copy_from_user+0x27/0x60
[Mon Mar 11 14:29:15 2024] __x64_sys_ppoll+0xbc/0x150
[Mon Mar 11 14:29:15 2024] do_syscall_64+0x59/0x90
[Mon Mar 11 14:29:15 2024] ? syscall_exit_to_user_mode+0x22/0x40
[Mon Mar 11 14:29:15 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:15 2024] ? audit_reset_context.part.0.constprop.0+0x273/0x2e0
[Mon Mar 11 14:29:15 2024] ? syscall_exit_work+0x103/0x130
[Mon Mar 11 14:29:15 2024] ? syscall_exit_to_user_mode+0x22/0x40
[Mon Mar 11 14:29:15 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:15 2024] ? exc_page_fault+0x62/0x150
[Mon Mar 11 14:29:15 2024] entry_SYSCALL_64_after_hwframe+0x72/0xdc
[Mon Mar 11 14:29:15 2024] RIP: 0033:0x7f66af3427fe
[Mon Mar 11 14:29:15 2024] RSP: 002b:00007f6683ffe8a0 EFLAGS: 00000293 ORIG_RAX: 000000000000010f
[Mon Mar 11 14:29:15 2024] RAX: ffffffffffffffda RBX: 00007f6683fff528 RCX: 00007f66af3427fe
[Mon Mar 11 14:29:15 2024] RDX: 0000000000000000 RSI: 0000000000000001 RDI: 00007f6683ffe8f8
[Mon Mar 11 14:29:15 2024] RBP: 00007f6683ffe900 R08: 0000000000000008 R09: 0000000000000000
[Mon Mar 11 14:29:15 2024] R10: 00007f6683ffe900 R11: 0000000000000293 R12: 00007ffd9217b000
[Mon Mar 11 14:29:15 2024] R13: 0000000000000000 R14: 000055b9f0cbeaf0 R15: 0000000000000000
[Mon Mar 11 14:29:15 2024] </TASK>
[Mon Mar 11 14:29:15 2024] task:automount state:S stack:0 pid:88130 ppid:1 flags:0x00000002
[Mon Mar 11 14:29:15 2024] Call Trace:
[Mon Mar 11 14:29:15 2024] <TASK>
[Mon Mar 11 14:29:15 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:15 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:15 2024] schedule_hrtimeout_range_clock+0x10f/0x120
[Mon Mar 11 14:29:15 2024] ? add_wait_queue+0x65/0xa0
[Mon Mar 11 14:29:15 2024] do_poll.constprop.0+0x248/0x390
[Mon Mar 11 14:29:15 2024] do_sys_poll+0x1c8/0x250
[Mon Mar 11 14:29:15 2024] ? do_sys_poll+0x21a/0x250
[Mon Mar 11 14:29:15 2024] ? chacha_block_generic+0x6f/0xb0
[Mon Mar 11 14:29:15 2024] ? __pfx_pollwake+0x10/0x10
[Mon Mar 11 14:29:15 2024] ? newidle_balance+0x2e5/0x400
[Mon Mar 11 14:29:15 2024] ? update_load_avg+0x7e/0x740
[Mon Mar 11 14:29:15 2024] ? pick_next_task_fair+0x41/0x500
[Mon Mar 11 14:29:15 2024] ? put_prev_task_fair+0x1e/0x40
[Mon Mar 11 14:29:15 2024] ? pick_next_task+0x861/0x950
[Mon Mar 11 14:29:15 2024] ? __update_idle_core+0x23/0xc0
[Mon Mar 11 14:29:15 2024] ? __switch_to_asm+0x3a/0x80
[Mon Mar 11 14:29:15 2024] ? finish_task_switch.isra.0+0x8c/0x2a0
[Mon Mar 11 14:29:15 2024] ? __schedule+0x223/0x550
[Mon Mar 11 14:29:15 2024] ? timerqueue_del+0x2a/0x50
[Mon Mar 11 14:29:15 2024] ? __remove_hrtimer+0x39/0x90
[Mon Mar 11 14:29:15 2024] ? hrtimer_try_to_cancel.part.0+0x50/0xf0
[Mon Mar 11 14:29:15 2024] ? futex_wait+0x23e/0x260
[Mon Mar 11 14:29:15 2024] __x64_sys_ppoll+0xbc/0x150
[Mon Mar 11 14:29:15 2024] do_syscall_64+0x59/0x90
[Mon Mar 11 14:29:15 2024] ? audit_reset_context.part.0.constprop.0+0x273/0x2e0
[Mon Mar 11 14:29:15 2024] ? syscall_exit_work+0x103/0x130
[Mon Mar 11 14:29:15 2024] ? syscall_exit_to_user_mode+0x22/0x40
[Mon Mar 11 14:29:15 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:15 2024] ? exit_to_user_mode_prepare+0xec/0x100
[Mon Mar 11 14:29:15 2024] ? syscall_exit_to_user_mode+0x22/0x40
[Mon Mar 11 14:29:15 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:15 2024] ? syscall_exit_to_user_mode+0x22/0x40
[Mon Mar 11 14:29:15 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:15 2024] ? exc_page_fault+0x62/0x150
[Mon Mar 11 14:29:15 2024] entry_SYSCALL_64_after_hwframe+0x72/0xdc
[Mon Mar 11 14:29:15 2024] RIP: 0033:0x7f66af3427fe
[Mon Mar 11 14:29:15 2024] RSP: 002b:00007f66837fd8a0 EFLAGS: 00000293 ORIG_RAX: 000000000000010f
[Mon Mar 11 14:29:15 2024] RAX: ffffffffffffffda RBX: 00007f66837fe528 RCX: 00007f66af3427fe
[Mon Mar 11 14:29:15 2024] RDX: 0000000000000000 RSI: 0000000000000001 RDI: 00007f66837fd8f8
[Mon Mar 11 14:29:15 2024] RBP: 00007f66837fd900 R08: 0000000000000008 R09: 0000000000000000
[Mon Mar 11 14:29:15 2024] R10: 00007f66837fd900 R11: 0000000000000293 R12: 00007ffd9217b000
[Mon Mar 11 14:29:15 2024] R13: 0000000000000000 R14: 000055b9f0cc4e80 R15: 0000000000000000
[Mon Mar 11 14:29:15 2024] </TASK>
[Mon Mar 11 14:29:15 2024] task:automount state:S stack:0 pid:88131 ppid:1 flags:0x00000002
[Mon Mar 11 14:29:15 2024] Call Trace:
[Mon Mar 11 14:29:15 2024] <TASK>
[Mon Mar 11 14:29:15 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:15 2024] ? xas_load+0x9/0xa0
[Mon Mar 11 14:29:15 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:15 2024] schedule_hrtimeout_range_clock+0x10f/0x120
[Mon Mar 11 14:29:15 2024] ? add_wait_queue+0x65/0xa0
[Mon Mar 11 14:29:15 2024] do_poll.constprop.0+0x248/0x390
[Mon Mar 11 14:29:15 2024] do_sys_poll+0x1c8/0x250
[Mon Mar 11 14:29:15 2024] ? select_idle_sibling+0x28/0x430
[Mon Mar 11 14:29:15 2024] ? wake_affine+0x62/0x1f0
[Mon Mar 11 14:29:15 2024] ? sched_clock+0xc/0x30
[Mon Mar 11 14:29:15 2024] ? __smp_call_single_queue+0x93/0x120
[Mon Mar 11 14:29:15 2024] ? ttwu_queue_wakelist+0xf2/0x110
[Mon Mar 11 14:29:15 2024] ? try_to_wake_up+0x3e2/0x5d0
[Mon Mar 11 14:29:15 2024] ? __pfx_pollwake+0x10/0x10
[Mon Mar 11 14:29:15 2024] ? __wake_up_common+0x75/0xa0
[Mon Mar 11 14:29:15 2024] ? select_idle_sibling+0x28/0x430
[Mon Mar 11 14:29:15 2024] ? __wake_up+0x40/0x60
[Mon Mar 11 14:29:15 2024] ? wake_affine+0x62/0x1f0
[Mon Mar 11 14:29:15 2024] ? sched_clock+0xc/0x30
[Mon Mar 11 14:29:15 2024] ? __smp_call_single_queue+0x93/0x120
[Mon Mar 11 14:29:15 2024] ? ttwu_queue_wakelist+0xf2/0x110
[Mon Mar 11 14:29:15 2024] ? try_to_wake_up+0x3e2/0x5d0
[Mon Mar 11 14:29:15 2024] ? sched_clock+0xc/0x30
[Mon Mar 11 14:29:15 2024] ? __smp_call_single_queue+0x93/0x120
[Mon Mar 11 14:29:15 2024] ? futex_wake+0x151/0x180
[Mon Mar 11 14:29:15 2024] ? __audit_filter_op+0xa5/0xf0
[Mon Mar 11 14:29:15 2024] ? audit_reset_context.part.0.constprop.0+0x273/0x2e0
[Mon Mar 11 14:29:15 2024] __x64_sys_ppoll+0xbc/0x150
[Mon Mar 11 14:29:15 2024] do_syscall_64+0x59/0x90
[Mon Mar 11 14:29:15 2024] ? syscall_exit_work+0x103/0x130
[Mon Mar 11 14:29:15 2024] ? syscall_exit_to_user_mode+0x22/0x40
[Mon Mar 11 14:29:15 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:15 2024] entry_SYSCALL_64_after_hwframe+0x72/0xdc
[Mon Mar 11 14:29:15 2024] RIP: 0033:0x7f66af3427fe
[Mon Mar 11 14:29:15 2024] RSP: 002b:00007f6682ffc8a0 EFLAGS: 00000293 ORIG_RAX: 000000000000010f
[Mon Mar 11 14:29:15 2024] RAX: ffffffffffffffda RBX: 00007f6682ffd528 RCX: 00007f66af3427fe
[Mon Mar 11 14:29:15 2024] RDX: 0000000000000000 RSI: 0000000000000001 RDI: 00007f6682ffc8f8
[Mon Mar 11 14:29:15 2024] RBP: 00007f6682ffc900 R08: 0000000000000008 R09: 0000000000000000
[Mon Mar 11 14:29:15 2024] R10: 00007f6682ffc900 R11: 0000000000000293 R12: 00007ffd9217b000
[Mon Mar 11 14:29:15 2024] R13: 0000000000000000 R14: 000055b9f0ccb210 R15: 0000000000000000
[Mon Mar 11 14:29:15 2024] </TASK>
[Mon Mar 11 14:29:15 2024] task:automount state:S stack:0 pid:88132 ppid:1 flags:0x00000002
[Mon Mar 11 14:29:15 2024] Call Trace:
[Mon Mar 11 14:29:15 2024] <TASK>
[Mon Mar 11 14:29:15 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:15 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:15 2024] schedule_hrtimeout_range_clock+0x10f/0x120
[Mon Mar 11 14:29:15 2024] ? add_wait_queue+0x65/0xa0
[Mon Mar 11 14:29:15 2024] do_poll.constprop.0+0x248/0x390
[Mon Mar 11 14:29:15 2024] do_sys_poll+0x1c8/0x250
[Mon Mar 11 14:29:15 2024] ? __kmem_cache_alloc_node+0x1c7/0x2d0
[Mon Mar 11 14:29:15 2024] ? select_idle_sibling+0x28/0x430
[Mon Mar 11 14:29:15 2024] ? wake_affine+0x62/0x1f0
[Mon Mar 11 14:29:15 2024] ? sched_clock+0xc/0x30
[Mon Mar 11 14:29:15 2024] ? __smp_call_single_queue+0x93/0x120
[Mon Mar 11 14:29:15 2024] ? ttwu_queue_wakelist+0xf2/0x110
[Mon Mar 11 14:29:15 2024] ? try_to_wake_up+0x3e2/0x5d0
[Mon Mar 11 14:29:15 2024] ? __pfx_pollwake+0x10/0x10
[Mon Mar 11 14:29:15 2024] ? __wake_up_common+0x75/0xa0
[Mon Mar 11 14:29:15 2024] ? __wake_up+0x40/0x60
[Mon Mar 11 14:29:15 2024] ? ep_poll_callback+0x25d/0x2a0
[Mon Mar 11 14:29:15 2024] ? __wake_up_common+0x75/0xa0
[Mon Mar 11 14:29:15 2024] ? __wake_up_sync_key+0x39/0x50
[Mon Mar 11 14:29:15 2024] ? sock_def_readable+0x3e/0xc0
[Mon Mar 11 14:29:15 2024] ? select_idle_sibling+0x28/0x430
[Mon Mar 11 14:29:15 2024] ? unix_dgram_sendmsg+0x5d3/0x9c0
[Mon Mar 11 14:29:15 2024] ? select_idle_sibling+0x28/0x430
[Mon Mar 11 14:29:15 2024] ? wake_affine+0x62/0x1f0
[Mon Mar 11 14:29:15 2024] ? sched_clock+0xc/0x30
[Mon Mar 11 14:29:15 2024] ? __smp_call_single_queue+0x93/0x120
[Mon Mar 11 14:29:15 2024] ? ttwu_queue_wakelist+0xf2/0x110
[Mon Mar 11 14:29:15 2024] ? try_to_wake_up+0x3e2/0x5d0
[Mon Mar 11 14:29:15 2024] ? try_to_wake_up+0x3e2/0x5d0
[Mon Mar 11 14:29:15 2024] ? _copy_from_user+0x27/0x60
[Mon Mar 11 14:29:15 2024] __x64_sys_ppoll+0xbc/0x150
[Mon Mar 11 14:29:15 2024] do_syscall_64+0x59/0x90
[Mon Mar 11 14:29:15 2024] ? audit_reset_context.part.0.constprop.0+0x273/0x2e0
[Mon Mar 11 14:29:15 2024] ? syscall_exit_work+0x103/0x130
[Mon Mar 11 14:29:15 2024] ? syscall_exit_to_user_mode+0x22/0x40
[Mon Mar 11 14:29:15 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:15 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:15 2024] ? syscall_exit_to_user_mode+0x22/0x40
[Mon Mar 11 14:29:15 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:15 2024] ? exc_page_fault+0x62/0x150
[Mon Mar 11 14:29:15 2024] entry_SYSCALL_64_after_hwframe+0x72/0xdc
[Mon Mar 11 14:29:15 2024] RIP: 0033:0x7f66af3427fe
[Mon Mar 11 14:29:15 2024] RSP: 002b:00007f66827fb8a0 EFLAGS: 00000293 ORIG_RAX: 000000000000010f
[Mon Mar 11 14:29:15 2024] RAX: ffffffffffffffda RBX: 00007f66827fc528 RCX: 00007f66af3427fe
[Mon Mar 11 14:29:15 2024] RDX: 0000000000000000 RSI: 0000000000000001 RDI: 00007f66827fb8f8
[Mon Mar 11 14:29:15 2024] RBP: 00007f66827fb900 R08: 0000000000000008 R09: 0000000000000000
[Mon Mar 11 14:29:15 2024] R10: 00007f66827fb900 R11: 0000000000000293 R12: 00007ffd9217b000
[Mon Mar 11 14:29:15 2024] R13: 0000000000000000 R14: 000055b9f0cd15a0 R15: 0000000000000000
[Mon Mar 11 14:29:15 2024] </TASK>
[Mon Mar 11 14:29:15 2024] task:automount state:S stack:0 pid:88133 ppid:1 flags:0x00000002
[Mon Mar 11 14:29:15 2024] Call Trace:
[Mon Mar 11 14:29:15 2024] <TASK>
[Mon Mar 11 14:29:15 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:15 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:15 2024] schedule_hrtimeout_range_clock+0x10f/0x120
[Mon Mar 11 14:29:15 2024] ? rmqueue_pcplist+0xda/0x210
[Mon Mar 11 14:29:15 2024] ? add_wait_queue+0x65/0xa0
[Mon Mar 11 14:29:15 2024] do_poll.constprop.0+0x248/0x390
[Mon Mar 11 14:29:15 2024] do_sys_poll+0x1c8/0x250
[Mon Mar 11 14:29:15 2024] ? try_to_wake_up+0x3e2/0x5d0
[Mon Mar 11 14:29:15 2024] ? autoremove_wake_function+0x30/0x60
[Mon Mar 11 14:29:15 2024] ? __wake_up_common+0x75/0xa0
[Mon Mar 11 14:29:15 2024] ? __wake_up+0x40/0x60
[Mon Mar 11 14:29:15 2024] ? ep_poll_callback+0x25d/0x2a0
[Mon Mar 11 14:29:15 2024] ? __pfx_pollwake+0x10/0x10
[Mon Mar 11 14:29:15 2024] ? sched_clock+0xc/0x30
[Mon Mar 11 14:29:15 2024] ? __smp_call_single_queue+0x93/0x120
[Mon Mar 11 14:29:15 2024] ? select_idle_sibling+0x28/0x430
[Mon Mar 11 14:29:15 2024] ? ttwu_queue_wakelist+0xf2/0x110
[Mon Mar 11 14:29:15 2024] ? wake_affine+0x62/0x1f0
[Mon Mar 11 14:29:15 2024] ? sched_clock+0xc/0x30
[Mon Mar 11 14:29:15 2024] ? __smp_call_single_queue+0x93/0x120
[Mon Mar 11 14:29:15 2024] ? ttwu_queue_wakelist+0xf2/0x110
[Mon Mar 11 14:29:15 2024] ? try_to_wake_up+0x3e2/0x5d0
[Mon Mar 11 14:29:15 2024] ? futex_wake+0x151/0x180
[Mon Mar 11 14:29:15 2024] ? __audit_filter_op+0xa5/0xf0
[Mon Mar 11 14:29:15 2024] ? _copy_from_user+0x27/0x60
[Mon Mar 11 14:29:15 2024] __x64_sys_ppoll+0xbc/0x150
[Mon Mar 11 14:29:15 2024] do_syscall_64+0x59/0x90
[Mon Mar 11 14:29:15 2024] ? exc_page_fault+0x62/0x150
[Mon Mar 11 14:29:15 2024] entry_SYSCALL_64_after_hwframe+0x72/0xdc
[Mon Mar 11 14:29:15 2024] RIP: 0033:0x7f66af3427fe
[Mon Mar 11 14:29:15 2024] RSP: 002b:00007f6681ffa8a0 EFLAGS: 00000293 ORIG_RAX: 000000000000010f
[Mon Mar 11 14:29:15 2024] RAX: ffffffffffffffda RBX: 00007f6681ffb528 RCX: 00007f66af3427fe
[Mon Mar 11 14:29:15 2024] RDX: 0000000000000000 RSI: 0000000000000001 RDI: 00007f6681ffa8f8
[Mon Mar 11 14:29:15 2024] RBP: 00007f6681ffa900 R08: 0000000000000008 R09: 0000000000000000
[Mon Mar 11 14:29:15 2024] R10: 00007f6681ffa900 R11: 0000000000000293 R12: 00007ffd9217b000
[Mon Mar 11 14:29:15 2024] R13: 0000000000000000 R14: 000055b9f0cd7930 R15: 0000000000000000
[Mon Mar 11 14:29:15 2024] </TASK>
[Mon Mar 11 14:29:15 2024] task:automount state:S stack:0 pid:88134 ppid:1 flags:0x00000002
[Mon Mar 11 14:29:15 2024] Call Trace:
[Mon Mar 11 14:29:15 2024] <TASK>
[Mon Mar 11 14:29:15 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:15 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:15 2024] schedule_hrtimeout_range_clock+0x10f/0x120
[Mon Mar 11 14:29:15 2024] ? __getblk_gfp+0x28/0xd0
[Mon Mar 11 14:29:15 2024] ? xas_load+0x9/0xa0
[Mon Mar 11 14:29:15 2024] ? __find_get_block+0x1fb/0x370
[Mon Mar 11 14:29:15 2024] ? add_wait_queue+0x65/0xa0
[Mon Mar 11 14:29:15 2024] do_poll.constprop.0+0x248/0x390
[Mon Mar 11 14:29:15 2024] ? __ext4_handle_dirty_metadata+0x58/0x180 [ext4]
[Mon Mar 11 14:29:15 2024] do_sys_poll+0x1c8/0x250
[Mon Mar 11 14:29:15 2024] ? select_idle_sibling+0x28/0x430
[Mon Mar 11 14:29:15 2024] ? wake_affine+0x62/0x1f0
[Mon Mar 11 14:29:15 2024] ? sched_clock+0xc/0x30
[Mon Mar 11 14:29:15 2024] ? __smp_call_single_queue+0x93/0x120
[Mon Mar 11 14:29:15 2024] ? ttwu_queue_wakelist+0xf2/0x110
[Mon Mar 11 14:29:15 2024] ? try_to_wake_up+0x3e2/0x5d0
[Mon Mar 11 14:29:15 2024] ? __alloc_skb+0x8e/0x1d0
[Mon Mar 11 14:29:15 2024] ? __pfx_pollwake+0x10/0x10
[Mon Mar 11 14:29:15 2024] ? __wake_up+0x40/0x60
[Mon Mar 11 14:29:15 2024] ? ep_poll_callback+0x25d/0x2a0
[Mon Mar 11 14:29:15 2024] ? select_idle_sibling+0x28/0x430
[Mon Mar 11 14:29:15 2024] ? wake_affine+0x62/0x1f0
[Mon Mar 11 14:29:15 2024] ? sched_clock+0xc/0x30
[Mon Mar 11 14:29:15 2024] ? __smp_call_single_queue+0x93/0x120
[Mon Mar 11 14:29:15 2024] ? ttwu_queue_wakelist+0xf2/0x110
[Mon Mar 11 14:29:15 2024] ? try_to_wake_up+0x3e2/0x5d0
[Mon Mar 11 14:29:15 2024] ? sched_clock+0xc/0x30
[Mon Mar 11 14:29:15 2024] ? futex_wake+0x151/0x180
[Mon Mar 11 14:29:15 2024] ? _copy_from_user+0x27/0x60
[Mon Mar 11 14:29:15 2024] __x64_sys_ppoll+0xbc/0x150
[Mon Mar 11 14:29:15 2024] do_syscall_64+0x59/0x90
[Mon Mar 11 14:29:15 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:15 2024] ? syscall_exit_to_user_mode+0x22/0x40
[Mon Mar 11 14:29:15 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:15 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:15 2024] ? exc_page_fault+0x62/0x150
[Mon Mar 11 14:29:15 2024] entry_SYSCALL_64_after_hwframe+0x72/0xdc
[Mon Mar 11 14:29:15 2024] RIP: 0033:0x7f66af3427fe
[Mon Mar 11 14:29:15 2024] RSP: 002b:00007f66817f98a0 EFLAGS: 00000293 ORIG_RAX: 000000000000010f
[Mon Mar 11 14:29:15 2024] RAX: ffffffffffffffda RBX: 00007f66817fa528 RCX: 00007f66af3427fe
[Mon Mar 11 14:29:15 2024] RDX: 0000000000000000 RSI: 0000000000000001 RDI: 00007f66817f98f8
[Mon Mar 11 14:29:15 2024] RBP: 00007f66817f9900 R08: 0000000000000008 R09: 0000000000000000
[Mon Mar 11 14:29:15 2024] R10: 00007f66817f9900 R11: 0000000000000293 R12: 00007ffd9217b000
[Mon Mar 11 14:29:15 2024] R13: 0000000000000000 R14: 000055b9f0cddcc0 R15: 0000000000000000
[Mon Mar 11 14:29:15 2024] </TASK>
[Mon Mar 11 14:29:15 2024] task:automount state:S stack:0 pid:88135 ppid:1 flags:0x00000002
[Mon Mar 11 14:29:15 2024] Call Trace:
[Mon Mar 11 14:29:15 2024] <TASK>
[Mon Mar 11 14:29:15 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:15 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:15 2024] schedule_hrtimeout_range_clock+0x10f/0x120
[Mon Mar 11 14:29:15 2024] ? add_wait_queue+0x65/0xa0
[Mon Mar 11 14:29:15 2024] do_poll.constprop.0+0x248/0x390
[Mon Mar 11 14:29:15 2024] do_sys_poll+0x1c8/0x250
[Mon Mar 11 14:29:15 2024] ? kmem_cache_alloc+0x17d/0x340
[Mon Mar 11 14:29:15 2024] ? update_sg_wakeup_stats+0x78/0x3b0
[Mon Mar 11 14:29:15 2024] ? find_idlest_group+0x2b2/0x530
[Mon Mar 11 14:29:15 2024] ? newidle_balance+0x2e5/0x400
[Mon Mar 11 14:29:15 2024] ? update_load_avg+0x7e/0x740
[Mon Mar 11 14:29:15 2024] ? __pfx_pollwake+0x10/0x10
[Mon Mar 11 14:29:15 2024] ? put_prev_task_fair+0x1e/0x40
[Mon Mar 11 14:29:15 2024] ? pick_next_task+0x861/0x950
[Mon Mar 11 14:29:15 2024] ? __update_idle_core+0x23/0xc0
[Mon Mar 11 14:29:15 2024] ? __switch_to_asm+0x3a/0x80
[Mon Mar 11 14:29:15 2024] ? finish_task_switch.isra.0+0x8c/0x2a0
[Mon Mar 11 14:29:15 2024] ? __schedule+0x223/0x550
[Mon Mar 11 14:29:15 2024] ? timerqueue_del+0x2a/0x50
[Mon Mar 11 14:29:15 2024] ? __remove_hrtimer+0x39/0x90
[Mon Mar 11 14:29:15 2024] ? hrtimer_try_to_cancel.part.0+0x50/0xf0
[Mon Mar 11 14:29:15 2024] ? hrtimer_cancel+0x1d/0x40
[Mon Mar 11 14:29:15 2024] ? futex_wait+0x23e/0x260
[Mon Mar 11 14:29:15 2024] ? __audit_filter_op+0xa5/0xf0
[Mon Mar 11 14:29:15 2024] ? _copy_from_user+0x27/0x60
[Mon Mar 11 14:29:15 2024] __x64_sys_ppoll+0xbc/0x150
[Mon Mar 11 14:29:15 2024] do_syscall_64+0x59/0x90
[Mon Mar 11 14:29:15 2024] ? audit_reset_context.part.0.constprop.0+0x273/0x2e0
[Mon Mar 11 14:29:15 2024] ? syscall_exit_work+0x103/0x130
[Mon Mar 11 14:29:15 2024] ? syscall_exit_to_user_mode+0x22/0x40
[Mon Mar 11 14:29:15 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:15 2024] ? exc_page_fault+0x62/0x150
[Mon Mar 11 14:29:15 2024] entry_SYSCALL_64_after_hwframe+0x72/0xdc
[Mon Mar 11 14:29:15 2024] RIP: 0033:0x7f66af3427fe
[Mon Mar 11 14:29:15 2024] RSP: 002b:00007f6680ff88a0 EFLAGS: 00000293 ORIG_RAX: 000000000000010f
[Mon Mar 11 14:29:15 2024] RAX: ffffffffffffffda RBX: 00007f6680ff9528 RCX: 00007f66af3427fe
[Mon Mar 11 14:29:15 2024] RDX: 0000000000000000 RSI: 0000000000000001 RDI: 00007f6680ff88f8
[Mon Mar 11 14:29:15 2024] RBP: 00007f6680ff8900 R08: 0000000000000008 R09: 0000000000000000
[Mon Mar 11 14:29:15 2024] R10: 00007f6680ff8900 R11: 0000000000000293 R12: 00007ffd9217b000
[Mon Mar 11 14:29:15 2024] R13: 0000000000000000 R14: 000055b9f0ce4050 R15: 0000000000000000
[Mon Mar 11 14:29:15 2024] </TASK>
[Mon Mar 11 14:29:15 2024] task:automount state:S stack:0 pid:88136 ppid:1 flags:0x00000002
[Mon Mar 11 14:29:15 2024] Call Trace:
[Mon Mar 11 14:29:15 2024] <TASK>
[Mon Mar 11 14:29:15 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:15 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:15 2024] schedule_hrtimeout_range_clock+0x10f/0x120
[Mon Mar 11 14:29:15 2024] ? add_wait_queue+0x65/0xa0
[Mon Mar 11 14:29:15 2024] do_poll.constprop.0+0x248/0x390
[Mon Mar 11 14:29:15 2024] do_sys_poll+0x1c8/0x250
[Mon Mar 11 14:29:15 2024] ? ttwu_queue_wakelist+0xf2/0x110
[Mon Mar 11 14:29:15 2024] ? try_to_wake_up+0x3e2/0x5d0
[Mon Mar 11 14:29:15 2024] ? __alloc_skb+0x8e/0x1d0
[Mon Mar 11 14:29:15 2024] ? autoremove_wake_function+0x30/0x60
[Mon Mar 11 14:29:15 2024] ? __wake_up_common+0x75/0xa0
[Mon Mar 11 14:29:15 2024] ? __pfx_pollwake+0x10/0x10
[Mon Mar 11 14:29:15 2024] ? __wake_up_common+0x75/0xa0
[Mon Mar 11 14:29:15 2024] ? __wake_up_sync_key+0x39/0x50
[Mon Mar 11 14:29:15 2024] ? select_idle_sibling+0x28/0x430
[Mon Mar 11 14:29:15 2024] ? sock_def_readable+0x3e/0xc0
[Mon Mar 11 14:29:15 2024] ? wake_affine+0x62/0x1f0
[Mon Mar 11 14:29:15 2024] ? sched_clock+0xc/0x30
[Mon Mar 11 14:29:15 2024] ? __smp_call_single_queue+0x93/0x120
[Mon Mar 11 14:29:15 2024] ? ttwu_queue_wakelist+0xf2/0x110
[Mon Mar 11 14:29:15 2024] ? try_to_wake_up+0x3e2/0x5d0
[Mon Mar 11 14:29:15 2024] ? __audit_filter_op+0xa5/0xf0
[Mon Mar 11 14:29:15 2024] __x64_sys_ppoll+0xbc/0x150
[Mon Mar 11 14:29:15 2024] do_syscall_64+0x59/0x90
[Mon Mar 11 14:29:15 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:15 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:15 2024] ? do_user_addr_fault+0x1d6/0x6a0
[Mon Mar 11 14:29:15 2024] ? exc_page_fault+0x62/0x150
[Mon Mar 11 14:29:15 2024] entry_SYSCALL_64_after_hwframe+0x72/0xdc
[Mon Mar 11 14:29:15 2024] RIP: 0033:0x7f66af3427fe
[Mon Mar 11 14:29:15 2024] RSP: 002b:00007f6663ffe8a0 EFLAGS: 00000293 ORIG_RAX: 000000000000010f
[Mon Mar 11 14:29:15 2024] RAX: ffffffffffffffda RBX: 00007f6663fff528 RCX: 00007f66af3427fe
[Mon Mar 11 14:29:15 2024] RDX: 0000000000000000 RSI: 0000000000000001 RDI: 00007f6663ffe8f8
[Mon Mar 11 14:29:15 2024] RBP: 00007f6663ffe900 R08: 0000000000000008 R09: 0000000000000000
[Mon Mar 11 14:29:15 2024] R10: 00007f6663ffe900 R11: 0000000000000293 R12: 00007ffd9217b000
[Mon Mar 11 14:29:15 2024] R13: 0000000000000000 R14: 000055b9f0cea3e0 R15: 0000000000000000
[Mon Mar 11 14:29:15 2024] </TASK>
[Mon Mar 11 14:29:15 2024] task:automount state:S stack:0 pid:88137 ppid:1 flags:0x00000002
[Mon Mar 11 14:29:15 2024] Call Trace:
[Mon Mar 11 14:29:15 2024] <TASK>
[Mon Mar 11 14:29:15 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:15 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:15 2024] schedule_hrtimeout_range_clock+0x10f/0x120
[Mon Mar 11 14:29:15 2024] ? add_wait_queue+0x65/0xa0
[Mon Mar 11 14:29:15 2024] do_poll.constprop.0+0x248/0x390
[Mon Mar 11 14:29:15 2024] do_sys_poll+0x1c8/0x250
[Mon Mar 11 14:29:15 2024] ? do_sys_poll+0x1d8/0x250
[Mon Mar 11 14:29:15 2024] ? do_wp_page+0x381/0x540
[Mon Mar 11 14:29:15 2024] ? newidle_balance+0x2e5/0x400
[Mon Mar 11 14:29:15 2024] ? update_load_avg+0x7e/0x740
[Mon Mar 11 14:29:15 2024] ? __pfx_pollwake+0x10/0x10
[Mon Mar 11 14:29:15 2024] ? __pfx_pollwake+0x10/0x10
[Mon Mar 11 14:29:15 2024] ? __schedule+0x223/0x550
[Mon Mar 11 14:29:15 2024] ? timerqueue_del+0x2a/0x50
[Mon Mar 11 14:29:15 2024] ? __remove_hrtimer+0x39/0x90
[Mon Mar 11 14:29:15 2024] ? hrtimer_try_to_cancel.part.0+0x50/0xf0
[Mon Mar 11 14:29:15 2024] ? _copy_to_iter+0x7e/0x630
[Mon Mar 11 14:29:15 2024] ? hrtimer_cancel+0x1d/0x40
[Mon Mar 11 14:29:15 2024] ? futex_wait+0x23e/0x260
[Mon Mar 11 14:29:15 2024] ? select_idle_sibling+0x28/0x430
[Mon Mar 11 14:29:15 2024] ? current_time+0x2b/0xf0
[Mon Mar 11 14:29:15 2024] ? wake_affine+0x62/0x1f0
[Mon Mar 11 14:29:15 2024] ? sched_clock+0xc/0x30
[Mon Mar 11 14:29:15 2024] ? __smp_call_single_queue+0x93/0x120
[Mon Mar 11 14:29:15 2024] ? ttwu_queue_wakelist+0xf2/0x110
[Mon Mar 11 14:29:15 2024] ? try_to_wake_up+0x3e2/0x5d0
[Mon Mar 11 14:29:15 2024] ? _copy_from_user+0x27/0x60
[Mon Mar 11 14:29:15 2024] __x64_sys_ppoll+0xbc/0x150
[Mon Mar 11 14:29:15 2024] do_syscall_64+0x59/0x90
[Mon Mar 11 14:29:15 2024] ? audit_reset_context.part.0.constprop.0+0x273/0x2e0
[Mon Mar 11 14:29:15 2024] ? syscall_exit_work+0x103/0x130
[Mon Mar 11 14:29:15 2024] ? syscall_exit_to_user_mode+0x22/0x40
[Mon Mar 11 14:29:15 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:15 2024] ? exc_page_fault+0x62/0x150
[Mon Mar 11 14:29:15 2024] entry_SYSCALL_64_after_hwframe+0x72/0xdc
[Mon Mar 11 14:29:15 2024] RIP: 0033:0x7f66af3427fe
[Mon Mar 11 14:29:15 2024] RSP: 002b:00007f66637fd8a0 EFLAGS: 00000293 ORIG_RAX: 000000000000010f
[Mon Mar 11 14:29:15 2024] RAX: ffffffffffffffda RBX: 00007f66637fe528 RCX: 00007f66af3427fe
[Mon Mar 11 14:29:15 2024] RDX: 0000000000000000 RSI: 0000000000000001 RDI: 00007f66637fd8f8
[Mon Mar 11 14:29:15 2024] RBP: 00007f66637fd900 R08: 0000000000000008 R09: 0000000000000000
[Mon Mar 11 14:29:15 2024] R10: 00007f66637fd900 R11: 0000000000000293 R12: 00007ffd9217b000
[Mon Mar 11 14:29:15 2024] R13: 0000000000000000 R14: 000055b9f0cf0790 R15: 0000000000000000
[Mon Mar 11 14:29:15 2024] </TASK>
[Mon Mar 11 14:29:15 2024] task:automount state:S stack:0 pid:88138 ppid:1 flags:0x00000002
[Mon Mar 11 14:29:15 2024] Call Trace:
[Mon Mar 11 14:29:15 2024] <TASK>
[Mon Mar 11 14:29:15 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:15 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:15 2024] schedule_hrtimeout_range_clock+0x10f/0x120
[Mon Mar 11 14:29:15 2024] ? add_wait_queue+0x65/0xa0
[Mon Mar 11 14:29:15 2024] ? add_wait_queue+0x65/0xa0
[Mon Mar 11 14:29:15 2024] do_poll.constprop.0+0x248/0x390
[Mon Mar 11 14:29:15 2024] do_sys_poll+0x1c8/0x250
[Mon Mar 11 14:29:15 2024] ? __pfx_pollwake+0x10/0x10
[Mon Mar 11 14:29:15 2024] ? sched_clock+0xc/0x30
[Mon Mar 11 14:29:15 2024] ? __smp_call_single_queue+0x93/0x120
[Mon Mar 11 14:29:15 2024] ? ttwu_queue_wakelist+0xf2/0x110
[Mon Mar 11 14:29:15 2024] ? try_to_wake_up+0x3e2/0x5d0
[Mon Mar 11 14:29:15 2024] ? current_time+0x2b/0xf0
[Mon Mar 11 14:29:15 2024] ? select_idle_sibling+0x28/0x430
[Mon Mar 11 14:29:15 2024] ? _copy_to_iter+0x7e/0x630
[Mon Mar 11 14:29:15 2024] ? wake_affine+0x62/0x1f0
[Mon Mar 11 14:29:15 2024] ? sched_clock+0xc/0x30
[Mon Mar 11 14:29:15 2024] ? __smp_call_single_queue+0x93/0x120
[Mon Mar 11 14:29:15 2024] ? try_to_wake_up+0x3e2/0x5d0
[Mon Mar 11 14:29:15 2024] ? __audit_filter_op+0xa5/0xf0
[Mon Mar 11 14:29:15 2024] ? audit_reset_context.part.0.constprop.0+0x273/0x2e0
[Mon Mar 11 14:29:15 2024] ? _copy_from_user+0x27/0x60
[Mon Mar 11 14:29:15 2024] __x64_sys_ppoll+0xbc/0x150
[Mon Mar 11 14:29:15 2024] do_syscall_64+0x59/0x90
[Mon Mar 11 14:29:15 2024] ? syscall_exit_work+0x103/0x130
[Mon Mar 11 14:29:15 2024] ? syscall_exit_to_user_mode+0x22/0x40
[Mon Mar 11 14:29:15 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:15 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:15 2024] entry_SYSCALL_64_after_hwframe+0x72/0xdc
[Mon Mar 11 14:29:15 2024] RIP: 0033:0x7f66af3427fe
[Mon Mar 11 14:29:15 2024] RSP: 002b:00007f6662ffc8a0 EFLAGS: 00000293 ORIG_RAX: 000000000000010f
[Mon Mar 11 14:29:15 2024] RAX: ffffffffffffffda RBX: 00007f6662ffd528 RCX: 00007f66af3427fe
[Mon Mar 11 14:29:15 2024] RDX: 0000000000000000 RSI: 0000000000000001 RDI: 00007f6662ffc8f8
[Mon Mar 11 14:29:15 2024] RBP: 00007f6662ffc900 R08: 0000000000000008 R09: 0000000000000000
[Mon Mar 11 14:29:15 2024] R10: 00007f6662ffc900 R11: 0000000000000293 R12: 00007ffd9217b000
[Mon Mar 11 14:29:15 2024] R13: 0000000000000000 R14: 000055b9f0cf6b00 R15: 0000000000000000
[Mon Mar 11 14:29:15 2024] </TASK>
[Mon Mar 11 14:29:15 2024] task:automount state:S stack:0 pid:88139 ppid:1 flags:0x00000002
[Mon Mar 11 14:29:15 2024] Call Trace:
[Mon Mar 11 14:29:15 2024] <TASK>
[Mon Mar 11 14:29:15 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:15 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:15 2024] schedule_hrtimeout_range_clock+0x10f/0x120
[Mon Mar 11 14:29:15 2024] ? add_wait_queue+0x65/0xa0
[Mon Mar 11 14:29:15 2024] do_poll.constprop.0+0x248/0x390
[Mon Mar 11 14:29:15 2024] do_sys_poll+0x1c8/0x250
[Mon Mar 11 14:29:15 2024] ? __mod_memcg_lruvec_state+0x84/0xd0
[Mon Mar 11 14:29:15 2024] ? mod_objcg_state+0x1fe/0x300
[Mon Mar 11 14:29:15 2024] ? select_idle_sibling+0x28/0x430
[Mon Mar 11 14:29:15 2024] ? wake_affine+0x62/0x1f0
[Mon Mar 11 14:29:15 2024] ? sched_clock+0xc/0x30
[Mon Mar 11 14:29:15 2024] ? __smp_call_single_queue+0x93/0x120
[Mon Mar 11 14:29:15 2024] ? ttwu_queue_wakelist+0xf2/0x110
[Mon Mar 11 14:29:15 2024] ? try_to_wake_up+0x3e2/0x5d0
[Mon Mar 11 14:29:15 2024] ? __pfx_pollwake+0x10/0x10
[Mon Mar 11 14:29:15 2024] ? __wake_up+0x40/0x60
[Mon Mar 11 14:29:15 2024] ? ep_poll_callback+0x25d/0x2a0
[Mon Mar 11 14:29:15 2024] ? select_idle_sibling+0x28/0x430
[Mon Mar 11 14:29:15 2024] ? wake_affine+0x62/0x1f0
[Mon Mar 11 14:29:15 2024] ? sched_clock+0xc/0x30
[Mon Mar 11 14:29:15 2024] ? select_idle_sibling+0x28/0x430
[Mon Mar 11 14:29:15 2024] ? wake_affine+0x62/0x1f0
[Mon Mar 11 14:29:15 2024] ? sched_clock+0xc/0x30
[Mon Mar 11 14:29:15 2024] ? __smp_call_single_queue+0x93/0x120
[Mon Mar 11 14:29:15 2024] ? ttwu_queue_wakelist+0xf2/0x110
[Mon Mar 11 14:29:15 2024] ? try_to_wake_up+0x3e2/0x5d0
[Mon Mar 11 14:29:15 2024] ? _copy_from_user+0x27/0x60
[Mon Mar 11 14:29:15 2024] __x64_sys_ppoll+0xbc/0x150
[Mon Mar 11 14:29:15 2024] do_syscall_64+0x59/0x90
[Mon Mar 11 14:29:15 2024] ? syscall_exit_to_user_mode+0x22/0x40
[Mon Mar 11 14:29:15 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:15 2024] ? syscall_exit_work+0x103/0x130
[Mon Mar 11 14:29:15 2024] ? syscall_exit_to_user_mode+0x22/0x40
[Mon Mar 11 14:29:15 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:15 2024] ? do_user_addr_fault+0x1d6/0x6a0
[Mon Mar 11 14:29:15 2024] ? exc_page_fault+0x62/0x150
[Mon Mar 11 14:29:15 2024] entry_SYSCALL_64_after_hwframe+0x72/0xdc
[Mon Mar 11 14:29:15 2024] RIP: 0033:0x7f66af3427fe
[Mon Mar 11 14:29:15 2024] RSP: 002b:00007f66627fb8a0 EFLAGS: 00000293 ORIG_RAX: 000000000000010f
[Mon Mar 11 14:29:15 2024] RAX: ffffffffffffffda RBX: 00007f66627fc528 RCX: 00007f66af3427fe
[Mon Mar 11 14:29:15 2024] RDX: 0000000000000000 RSI: 0000000000000001 RDI: 00007f66627fb8f8
[Mon Mar 11 14:29:15 2024] RBP: 00007f66627fb900 R08: 0000000000000008 R09: 0000000000000000
[Mon Mar 11 14:29:15 2024] R10: 00007f66627fb900 R11: 0000000000000293 R12: 00007ffd9217b000
[Mon Mar 11 14:29:15 2024] R13: 0000000000000000 R14: 000055b9f0cfce90 R15: 0000000000000000
[Mon Mar 11 14:29:15 2024] </TASK>
[Mon Mar 11 14:29:15 2024] task:automount state:S stack:0 pid:88140 ppid:1 flags:0x00000002
[Mon Mar 11 14:29:15 2024] Call Trace:
[Mon Mar 11 14:29:15 2024] <TASK>
[Mon Mar 11 14:29:15 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:15 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:15 2024] schedule_hrtimeout_range_clock+0x10f/0x120
[Mon Mar 11 14:29:15 2024] ? __find_get_block+0x1fb/0x370
[Mon Mar 11 14:29:15 2024] ? add_wait_queue+0x65/0xa0
[Mon Mar 11 14:29:15 2024] do_poll.constprop.0+0x248/0x390
[Mon Mar 11 14:29:15 2024] do_sys_poll+0x1c8/0x250
[Mon Mar 11 14:29:15 2024] ? select_idle_sibling+0x28/0x430
[Mon Mar 11 14:29:15 2024] ? sched_clock+0xc/0x30
[Mon Mar 11 14:29:15 2024] ? __smp_call_single_queue+0x93/0x120
[Mon Mar 11 14:29:15 2024] ? ttwu_queue_wakelist+0xf2/0x110
[Mon Mar 11 14:29:15 2024] ? try_to_wake_up+0x3e2/0x5d0
[Mon Mar 11 14:29:15 2024] ? __alloc_skb+0x8e/0x1d0
[Mon Mar 11 14:29:15 2024] ? autoremove_wake_function+0x30/0x60
[Mon Mar 11 14:29:15 2024] ? __wake_up_common+0x75/0xa0
[Mon Mar 11 14:29:15 2024] ? __pfx_pollwake+0x10/0x10
[Mon Mar 11 14:29:15 2024] ? __wake_up_common+0x75/0xa0
[Mon Mar 11 14:29:15 2024] ? __wake_up_sync_key+0x39/0x50
[Mon Mar 11 14:29:15 2024] ? sock_def_readable+0x3e/0xc0
[Mon Mar 11 14:29:15 2024] ? select_idle_sibling+0x28/0x430
[Mon Mar 11 14:29:15 2024] ? select_idle_sibling+0x28/0x430
[Mon Mar 11 14:29:15 2024] ? wake_affine+0x62/0x1f0
[Mon Mar 11 14:29:15 2024] ? wake_affine+0x62/0x1f0
[Mon Mar 11 14:29:15 2024] ? sched_clock+0xc/0x30
[Mon Mar 11 14:29:15 2024] ? __smp_call_single_queue+0x93/0x120
[Mon Mar 11 14:29:15 2024] ? ttwu_queue_wakelist+0xf2/0x110
[Mon Mar 11 14:29:15 2024] ? try_to_wake_up+0x3e2/0x5d0
[Mon Mar 11 14:29:15 2024] ? futex_wake+0x151/0x180
[Mon Mar 11 14:29:15 2024] ? _copy_from_user+0x27/0x60
[Mon Mar 11 14:29:15 2024] __x64_sys_ppoll+0xbc/0x150
[Mon Mar 11 14:29:15 2024] do_syscall_64+0x59/0x90
[Mon Mar 11 14:29:15 2024] ? syscall_exit_to_user_mode+0x22/0x40
[Mon Mar 11 14:29:15 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:15 2024] ? syscall_exit_to_user_mode+0x22/0x40
[Mon Mar 11 14:29:15 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:15 2024] ? do_user_addr_fault+0x1d6/0x6a0
[Mon Mar 11 14:29:15 2024] ? exc_page_fault+0x62/0x150
[Mon Mar 11 14:29:15 2024] entry_SYSCALL_64_after_hwframe+0x72/0xdc
[Mon Mar 11 14:29:15 2024] RIP: 0033:0x7f66af3427fe
[Mon Mar 11 14:29:15 2024] RSP: 002b:00007f6661ffa8a0 EFLAGS: 00000293 ORIG_RAX: 000000000000010f
[Mon Mar 11 14:29:15 2024] RAX: ffffffffffffffda RBX: 00007f6661ffb528 RCX: 00007f66af3427fe
[Mon Mar 11 14:29:15 2024] RDX: 0000000000000000 RSI: 0000000000000001 RDI: 00007f6661ffa8f8
[Mon Mar 11 14:29:15 2024] RBP: 00007f6661ffa900 R08: 0000000000000008 R09: 0000000000000000
[Mon Mar 11 14:29:15 2024] R10: 00007f6661ffa900 R11: 0000000000000293 R12: 00007ffd9217b000
[Mon Mar 11 14:29:15 2024] R13: 0000000000000000 R14: 000055b9f0d03220 R15: 0000000000000000
[Mon Mar 11 14:29:15 2024] </TASK>
[Mon Mar 11 14:29:15 2024] task:automount state:S stack:0 pid:88141 ppid:1 flags:0x00000002
[Mon Mar 11 14:29:15 2024] Call Trace:
[Mon Mar 11 14:29:15 2024] <TASK>
[Mon Mar 11 14:29:15 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:15 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:15 2024] schedule_hrtimeout_range_clock+0x10f/0x120
[Mon Mar 11 14:29:15 2024] ? asm_exc_page_fault+0x22/0x30
[Mon Mar 11 14:29:15 2024] ? add_wait_queue+0x65/0xa0
[Mon Mar 11 14:29:15 2024] do_poll.constprop.0+0x248/0x390
[Mon Mar 11 14:29:15 2024] do_sys_poll+0x1c8/0x250
[Mon Mar 11 14:29:15 2024] ? update_sg_wakeup_stats+0x78/0x3b0
[Mon Mar 11 14:29:15 2024] ? newidle_balance+0x2e5/0x400
[Mon Mar 11 14:29:15 2024] ? update_load_avg+0x7e/0x740
[Mon Mar 11 14:29:15 2024] ? pick_next_task_fair+0x41/0x500
[Mon Mar 11 14:29:15 2024] ? put_prev_task_fair+0x1e/0x40
[Mon Mar 11 14:29:15 2024] ? pick_next_task+0x861/0x950
[Mon Mar 11 14:29:15 2024] ? __pfx_pollwake+0x10/0x10
[Mon Mar 11 14:29:15 2024] ? timerqueue_del+0x2a/0x50
[Mon Mar 11 14:29:15 2024] ? __remove_hrtimer+0x39/0x90
[Mon Mar 11 14:29:15 2024] ? hrtimer_try_to_cancel.part.0+0x50/0xf0
[Mon Mar 11 14:29:15 2024] ? hrtimer_cancel+0x1d/0x40
[Mon Mar 11 14:29:15 2024] ? futex_wait+0x23e/0x260
[Mon Mar 11 14:29:15 2024] ? __pfx_hrtimer_wakeup+0x10/0x10
[Mon Mar 11 14:29:15 2024] ? futex_wake+0x7c/0x180
[Mon Mar 11 14:29:15 2024] ? __audit_filter_op+0xa5/0xf0
[Mon Mar 11 14:29:15 2024] ? __audit_filter_op+0xa5/0xf0
[Mon Mar 11 14:29:15 2024] ? _copy_from_user+0x27/0x60
[Mon Mar 11 14:29:15 2024] __x64_sys_ppoll+0xbc/0x150
[Mon Mar 11 14:29:15 2024] do_syscall_64+0x59/0x90
[Mon Mar 11 14:29:15 2024] entry_SYSCALL_64_after_hwframe+0x72/0xdc
[Mon Mar 11 14:29:15 2024] RIP: 0033:0x7f66af3427fe
[Mon Mar 11 14:29:15 2024] RSP: 002b:00007f66617f98a0 EFLAGS: 00000293 ORIG_RAX: 000000000000010f
[Mon Mar 11 14:29:15 2024] RAX: ffffffffffffffda RBX: 00007f66617fa528 RCX: 00007f66af3427fe
[Mon Mar 11 14:29:15 2024] RDX: 0000000000000000 RSI: 0000000000000001 RDI: 00007f66617f98f8
[Mon Mar 11 14:29:15 2024] RBP: 00007f66617f9900 R08: 0000000000000008 R09: 0000000000000000
[Mon Mar 11 14:29:15 2024] R10: 00007f66617f9900 R11: 0000000000000293 R12: 00007ffd9217b000
[Mon Mar 11 14:29:15 2024] R13: 0000000000000000 R14: 000055b9f0d095b0 R15: 0000000000000000
[Mon Mar 11 14:29:15 2024] </TASK>
[Mon Mar 11 14:29:15 2024] task:automount state:S stack:0 pid:88142 ppid:1 flags:0x00000002
[Mon Mar 11 14:29:15 2024] Call Trace:
[Mon Mar 11 14:29:15 2024] <TASK>
[Mon Mar 11 14:29:15 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:15 2024] ? ext4_inode_csum+0x199/0x210 [ext4]
[Mon Mar 11 14:29:15 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:15 2024] schedule_hrtimeout_range_clock+0x10f/0x120
[Mon Mar 11 14:29:15 2024] ? add_wait_queue+0x65/0xa0
[Mon Mar 11 14:29:15 2024] do_poll.constprop.0+0x248/0x390
[Mon Mar 11 14:29:15 2024] do_sys_poll+0x1c8/0x250
[Mon Mar 11 14:29:15 2024] ? wake_affine+0x62/0x1f0
[Mon Mar 11 14:29:15 2024] ? __smp_call_single_queue+0x93/0x120
[Mon Mar 11 14:29:15 2024] ? ttwu_queue_wakelist+0xf2/0x110
[Mon Mar 11 14:29:15 2024] ? try_to_wake_up+0x3e2/0x5d0
[Mon Mar 11 14:29:15 2024] ? __alloc_skb+0x8e/0x1d0
[Mon Mar 11 14:29:15 2024] ? autoremove_wake_function+0x30/0x60
[Mon Mar 11 14:29:15 2024] ? __wake_up_common+0x75/0xa0
[Mon Mar 11 14:29:15 2024] ? __pfx_pollwake+0x10/0x10
[Mon Mar 11 14:29:15 2024] ? __wake_up_common+0x75/0xa0
[Mon Mar 11 14:29:15 2024] ? __wake_up_sync_key+0x39/0x50
[Mon Mar 11 14:29:15 2024] ? select_idle_sibling+0x28/0x430
[Mon Mar 11 14:29:15 2024] ? wake_affine+0x62/0x1f0
[Mon Mar 11 14:29:15 2024] ? sched_clock+0xc/0x30
[Mon Mar 11 14:29:15 2024] ? __smp_call_single_queue+0x93/0x120
[Mon Mar 11 14:29:15 2024] ? ttwu_queue_wakelist+0xf2/0x110
[Mon Mar 11 14:29:15 2024] ? try_to_wake_up+0x3e2/0x5d0
[Mon Mar 11 14:29:15 2024] ? _copy_from_user+0x27/0x60
[Mon Mar 11 14:29:15 2024] __x64_sys_ppoll+0xbc/0x150
[Mon Mar 11 14:29:15 2024] do_syscall_64+0x59/0x90
[Mon Mar 11 14:29:15 2024] ? syscall_exit_work+0x103/0x130
[Mon Mar 11 14:29:15 2024] ? syscall_exit_to_user_mode+0x22/0x40
[Mon Mar 11 14:29:15 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:15 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:15 2024] ? exc_page_fault+0x62/0x150
[Mon Mar 11 14:29:15 2024] entry_SYSCALL_64_after_hwframe+0x72/0xdc
[Mon Mar 11 14:29:15 2024] RIP: 0033:0x7f66af3427fe
[Mon Mar 11 14:29:15 2024] RSP: 002b:00007f6660ff88a0 EFLAGS: 00000293 ORIG_RAX: 000000000000010f
[Mon Mar 11 14:29:15 2024] RAX: ffffffffffffffda RBX: 00007f6660ff9528 RCX: 00007f66af3427fe
[Mon Mar 11 14:29:15 2024] RDX: 0000000000000000 RSI: 0000000000000001 RDI: 00007f6660ff88f8
[Mon Mar 11 14:29:15 2024] RBP: 00007f6660ff8900 R08: 0000000000000008 R09: 0000000000000000
[Mon Mar 11 14:29:15 2024] R10: 00007f6660ff8900 R11: 0000000000000293 R12: 00007ffd9217b000
[Mon Mar 11 14:29:15 2024] R13: 0000000000000000 R14: 000055b9f0d0f940 R15: 0000000000000000
[Mon Mar 11 14:29:15 2024] </TASK>
[Mon Mar 11 14:29:15 2024] task:automount state:S stack:0 pid:88143 ppid:1 flags:0x00000002
[Mon Mar 11 14:29:15 2024] Call Trace:
[Mon Mar 11 14:29:15 2024] <TASK>
[Mon Mar 11 14:29:15 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:15 2024] ? finish_task_switch.isra.0+0x8c/0x2a0
[Mon Mar 11 14:29:15 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:15 2024] schedule_hrtimeout_range_clock+0x10f/0x120
[Mon Mar 11 14:29:15 2024] ? add_wait_queue+0x65/0xa0
[Mon Mar 11 14:29:15 2024] do_poll.constprop.0+0x248/0x390
[Mon Mar 11 14:29:15 2024] do_sys_poll+0x1c8/0x250
[Mon Mar 11 14:29:15 2024] ? do_sys_poll+0x1d8/0x250
[Mon Mar 11 14:29:15 2024] ? kmem_cache_alloc+0x17d/0x340
[Mon Mar 11 14:29:15 2024] ? update_sg_wakeup_stats+0x78/0x3b0
[Mon Mar 11 14:29:15 2024] ? __pfx_pollwake+0x10/0x10
[Mon Mar 11 14:29:15 2024] ? newidle_balance+0x2e5/0x400
[Mon Mar 11 14:29:15 2024] ? update_load_avg+0x7e/0x740
[Mon Mar 11 14:29:15 2024] ? pick_next_task_fair+0x41/0x500
[Mon Mar 11 14:29:15 2024] ? put_prev_task_fair+0x1e/0x40
[Mon Mar 11 14:29:15 2024] ? pick_next_task+0x861/0x950
[Mon Mar 11 14:29:15 2024] ? __update_idle_core+0x23/0xc0
[Mon Mar 11 14:29:15 2024] ? __switch_to_asm+0x3a/0x80
[Mon Mar 11 14:29:15 2024] ? finish_task_switch.isra.0+0x8c/0x2a0
[Mon Mar 11 14:29:15 2024] ? __schedule+0x223/0x550
[Mon Mar 11 14:29:15 2024] ? timerqueue_del+0x2a/0x50
[Mon Mar 11 14:29:15 2024] ? __remove_hrtimer+0x39/0x90
[Mon Mar 11 14:29:15 2024] ? hrtimer_try_to_cancel.part.0+0x50/0xf0
[Mon Mar 11 14:29:15 2024] ? futex_wait+0x23e/0x260
[Mon Mar 11 14:29:15 2024] ? futex_wake+0x7c/0x180
[Mon Mar 11 14:29:15 2024] __x64_sys_ppoll+0xbc/0x150
[Mon Mar 11 14:29:15 2024] do_syscall_64+0x59/0x90
[Mon Mar 11 14:29:15 2024] ? syscall_exit_work+0x103/0x130
[Mon Mar 11 14:29:15 2024] ? syscall_exit_to_user_mode+0x22/0x40
[Mon Mar 11 14:29:15 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:15 2024] ? exit_to_user_mode_prepare+0xec/0x100
[Mon Mar 11 14:29:15 2024] ? syscall_exit_to_user_mode+0x22/0x40
[Mon Mar 11 14:29:15 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:15 2024] ? syscall_exit_to_user_mode+0x22/0x40
[Mon Mar 11 14:29:15 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:15 2024] ? exc_page_fault+0x62/0x150
[Mon Mar 11 14:29:15 2024] entry_SYSCALL_64_after_hwframe+0x72/0xdc
[Mon Mar 11 14:29:15 2024] RIP: 0033:0x7f66af3427fe
[Mon Mar 11 14:29:15 2024] RSP: 002b:00007f6643ffe8a0 EFLAGS: 00000293 ORIG_RAX: 000000000000010f
[Mon Mar 11 14:29:15 2024] RAX: ffffffffffffffda RBX: 00007f6643fff528 RCX: 00007f66af3427fe
[Mon Mar 11 14:29:15 2024] RDX: 0000000000000000 RSI: 0000000000000001 RDI: 00007f6643ffe8f8
[Mon Mar 11 14:29:15 2024] RBP: 00007f6643ffe900 R08: 0000000000000008 R09: 0000000000000000
[Mon Mar 11 14:29:15 2024] R10: 00007f6643ffe900 R11: 0000000000000293 R12: 00007ffd9217b000
[Mon Mar 11 14:29:15 2024] R13: 0000000000000000 R14: 000055b9f0d15cd0 R15: 0000000000000000
[Mon Mar 11 14:29:15 2024] </TASK>
[Mon Mar 11 14:29:15 2024] task:automount state:S stack:0 pid:88144 ppid:1 flags:0x00000002
[Mon Mar 11 14:29:15 2024] Call Trace:
[Mon Mar 11 14:29:15 2024] <TASK>
[Mon Mar 11 14:29:15 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:15 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:15 2024] schedule_hrtimeout_range_clock+0x10f/0x120
[Mon Mar 11 14:29:15 2024] ? add_wait_queue+0x65/0xa0
[Mon Mar 11 14:29:15 2024] do_poll.constprop.0+0x248/0x390
[Mon Mar 11 14:29:15 2024] do_sys_poll+0x1c8/0x250
[Mon Mar 11 14:29:15 2024] ? ttwu_queue_wakelist+0xf2/0x110
[Mon Mar 11 14:29:15 2024] ? try_to_wake_up+0x3e2/0x5d0
[Mon Mar 11 14:29:15 2024] ? __alloc_skb+0x8e/0x1d0
[Mon Mar 11 14:29:15 2024] ? autoremove_wake_function+0x30/0x60
[Mon Mar 11 14:29:15 2024] ? __wake_up_common+0x75/0xa0
[Mon Mar 11 14:29:15 2024] ? __pfx_pollwake+0x10/0x10
[Mon Mar 11 14:29:15 2024] ? __wake_up_common+0x75/0xa0
[Mon Mar 11 14:29:15 2024] ? select_idle_sibling+0x28/0x430
[Mon Mar 11 14:29:15 2024] ? wake_affine+0x62/0x1f0
[Mon Mar 11 14:29:15 2024] ? sched_clock+0xc/0x30
[Mon Mar 11 14:29:15 2024] ? __smp_call_single_queue+0x93/0x120
[Mon Mar 11 14:29:15 2024] ? ttwu_queue_wakelist+0xf2/0x110
[Mon Mar 11 14:29:15 2024] ? try_to_wake_up+0x3e2/0x5d0
[Mon Mar 11 14:29:15 2024] ? __audit_filter_op+0xa5/0xf0
[Mon Mar 11 14:29:15 2024] ? audit_reset_context.part.0.constprop.0+0x273/0x2e0
[Mon Mar 11 14:29:15 2024] ? _copy_from_user+0x27/0x60
[Mon Mar 11 14:29:15 2024] __x64_sys_ppoll+0xbc/0x150
[Mon Mar 11 14:29:15 2024] do_syscall_64+0x59/0x90
[Mon Mar 11 14:29:15 2024] ? syscall_exit_work+0x103/0x130
[Mon Mar 11 14:29:15 2024] ? syscall_exit_to_user_mode+0x22/0x40
[Mon Mar 11 14:29:15 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:15 2024] ? exc_page_fault+0x62/0x150
[Mon Mar 11 14:29:15 2024] entry_SYSCALL_64_after_hwframe+0x72/0xdc
[Mon Mar 11 14:29:15 2024] RIP: 0033:0x7f66af3427fe
[Mon Mar 11 14:29:15 2024] RSP: 002b:00007f66437fd8a0 EFLAGS: 00000293 ORIG_RAX: 000000000000010f
[Mon Mar 11 14:29:15 2024] RAX: ffffffffffffffda RBX: 00007f66437fe528 RCX: 00007f66af3427fe
[Mon Mar 11 14:29:15 2024] RDX: 0000000000000000 RSI: 0000000000000001 RDI: 00007f66437fd8f8
[Mon Mar 11 14:29:15 2024] RBP: 00007f66437fd900 R08: 0000000000000008 R09: 0000000000000000
[Mon Mar 11 14:29:15 2024] R10: 00007f66437fd900 R11: 0000000000000293 R12: 00007ffd9217b000
[Mon Mar 11 14:29:15 2024] R13: 0000000000000000 R14: 000055b9f0d1c060 R15: 0000000000000000
[Mon Mar 11 14:29:15 2024] </TASK>
[Mon Mar 11 14:29:15 2024] task:automount state:S stack:0 pid:88145 ppid:1 flags:0x00000002
[Mon Mar 11 14:29:15 2024] Call Trace:
[Mon Mar 11 14:29:15 2024] <TASK>
[Mon Mar 11 14:29:15 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:15 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:15 2024] schedule_hrtimeout_range_clock+0x10f/0x120
[Mon Mar 11 14:29:15 2024] ? add_wait_queue+0x65/0xa0
[Mon Mar 11 14:29:15 2024] do_poll.constprop.0+0x248/0x390
[Mon Mar 11 14:29:15 2024] do_sys_poll+0x1c8/0x250
[Mon Mar 11 14:29:15 2024] ? wake_affine+0x62/0x1f0
[Mon Mar 11 14:29:15 2024] ? sched_clock+0xc/0x30
[Mon Mar 11 14:29:15 2024] ? __smp_call_single_queue+0x93/0x120
[Mon Mar 11 14:29:15 2024] ? ttwu_queue_wakelist+0xf2/0x110
[Mon Mar 11 14:29:15 2024] ? try_to_wake_up+0x3e2/0x5d0
[Mon Mar 11 14:29:15 2024] ? __alloc_skb+0x8e/0x1d0
[Mon Mar 11 14:29:15 2024] ? __pfx_pollwake+0x10/0x10
[Mon Mar 11 14:29:15 2024] ? ep_poll_callback+0x25d/0x2a0
[Mon Mar 11 14:29:15 2024] ? select_idle_sibling+0x28/0x430
[Mon Mar 11 14:29:15 2024] ? wake_affine+0x62/0x1f0
[Mon Mar 11 14:29:15 2024] ? sched_clock+0xc/0x30
[Mon Mar 11 14:29:15 2024] ? __smp_call_single_queue+0x93/0x120
[Mon Mar 11 14:29:15 2024] ? ttwu_queue_wakelist+0xf2/0x110
[Mon Mar 11 14:29:15 2024] ? try_to_wake_up+0x3e2/0x5d0
[Mon Mar 11 14:29:15 2024] ? unix_dgram_sendmsg+0x5d3/0x9c0
[Mon Mar 11 14:29:15 2024] ? futex_wake+0x151/0x180
[Mon Mar 11 14:29:15 2024] ? __audit_filter_op+0xa5/0xf0
[Mon Mar 11 14:29:15 2024] ? audit_reset_context.part.0.constprop.0+0x273/0x2e0
[Mon Mar 11 14:29:15 2024] ? _copy_from_user+0x27/0x60
[Mon Mar 11 14:29:15 2024] __x64_sys_ppoll+0xbc/0x150
[Mon Mar 11 14:29:15 2024] do_syscall_64+0x59/0x90
[Mon Mar 11 14:29:15 2024] ? syscall_exit_work+0x103/0x130
[Mon Mar 11 14:29:15 2024] ? syscall_exit_to_user_mode+0x22/0x40
[Mon Mar 11 14:29:15 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:15 2024] ? syscall_exit_work+0x103/0x130
[Mon Mar 11 14:29:15 2024] ? syscall_exit_to_user_mode+0x22/0x40
[Mon Mar 11 14:29:15 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:15 2024] ? exc_page_fault+0x62/0x150
[Mon Mar 11 14:29:15 2024] entry_SYSCALL_64_after_hwframe+0x72/0xdc
[Mon Mar 11 14:29:15 2024] RIP: 0033:0x7f66af3427fe
[Mon Mar 11 14:29:15 2024] RSP: 002b:00007f6642ffc8a0 EFLAGS: 00000293 ORIG_RAX: 000000000000010f
[Mon Mar 11 14:29:15 2024] RAX: ffffffffffffffda RBX: 00007f6642ffd528 RCX: 00007f66af3427fe
[Mon Mar 11 14:29:15 2024] RDX: 0000000000000000 RSI: 0000000000000001 RDI: 00007f6642ffc8f8
[Mon Mar 11 14:29:15 2024] RBP: 00007f6642ffc900 R08: 0000000000000008 R09: 0000000000000000
[Mon Mar 11 14:29:15 2024] R10: 00007f6642ffc900 R11: 0000000000000293 R12: 00007ffd9217b000
[Mon Mar 11 14:29:15 2024] R13: 0000000000000000 R14: 000055b9f0d223f0 R15: 0000000000000000
[Mon Mar 11 14:29:15 2024] </TASK>
[Mon Mar 11 14:29:15 2024] task:automount state:S stack:0 pid:88146 ppid:1 flags:0x00000002
[Mon Mar 11 14:29:15 2024] Call Trace:
[Mon Mar 11 14:29:15 2024] <TASK>
[Mon Mar 11 14:29:15 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:15 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:15 2024] schedule_hrtimeout_range_clock+0x10f/0x120
[Mon Mar 11 14:29:15 2024] ? add_wait_queue+0x65/0xa0
[Mon Mar 11 14:29:15 2024] do_poll.constprop.0+0x248/0x390
[Mon Mar 11 14:29:15 2024] ? memcg_alloc_slab_cgroups+0x39/0xa0
[Mon Mar 11 14:29:15 2024] do_sys_poll+0x1c8/0x250
[Mon Mar 11 14:29:15 2024] ? avc_has_perm_noaudit+0x94/0x110
[Mon Mar 11 14:29:15 2024] ? wake_affine+0x62/0x1f0
[Mon Mar 11 14:29:15 2024] ? sched_clock+0xc/0x30
[Mon Mar 11 14:29:15 2024] ? __smp_call_single_queue+0x93/0x120
[Mon Mar 11 14:29:15 2024] ? ttwu_queue_wakelist+0xf2/0x110
[Mon Mar 11 14:29:15 2024] ? try_to_wake_up+0x3e2/0x5d0
[Mon Mar 11 14:29:15 2024] ? __alloc_skb+0x8e/0x1d0
[Mon Mar 11 14:29:15 2024] ? __pfx_pollwake+0x10/0x10
[Mon Mar 11 14:29:15 2024] ? ep_poll_callback+0x25d/0x2a0
[Mon Mar 11 14:29:15 2024] ? select_idle_sibling+0x28/0x430
[Mon Mar 11 14:29:15 2024] ? wake_affine+0x62/0x1f0
[Mon Mar 11 14:29:15 2024] ? select_idle_sibling+0x28/0x430
[Mon Mar 11 14:29:15 2024] ? wake_affine+0x62/0x1f0
[Mon Mar 11 14:29:15 2024] ? __smp_call_single_queue+0x93/0x120
[Mon Mar 11 14:29:15 2024] ? sched_clock+0xc/0x30
[Mon Mar 11 14:29:15 2024] ? __smp_call_single_queue+0x93/0x120
[Mon Mar 11 14:29:15 2024] ? ttwu_queue_wakelist+0xf2/0x110
[Mon Mar 11 14:29:15 2024] ? __audit_filter_op+0xa5/0xf0
[Mon Mar 11 14:29:15 2024] ? audit_reset_context.part.0.constprop.0+0x273/0x2e0
[Mon Mar 11 14:29:15 2024] __x64_sys_ppoll+0xbc/0x150
[Mon Mar 11 14:29:15 2024] do_syscall_64+0x59/0x90
[Mon Mar 11 14:29:15 2024] ? syscall_exit_to_user_mode+0x22/0x40
[Mon Mar 11 14:29:15 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:15 2024] ? exc_page_fault+0x62/0x150
[Mon Mar 11 14:29:15 2024] entry_SYSCALL_64_after_hwframe+0x72/0xdc
[Mon Mar 11 14:29:15 2024] RIP: 0033:0x7f66af3427fe
[Mon Mar 11 14:29:15 2024] RSP: 002b:00007f66427fb8a0 EFLAGS: 00000293 ORIG_RAX: 000000000000010f
[Mon Mar 11 14:29:15 2024] RAX: ffffffffffffffda RBX: 00007f66427fc528 RCX: 00007f66af3427fe
[Mon Mar 11 14:29:15 2024] RDX: 0000000000000000 RSI: 0000000000000001 RDI: 00007f66427fb8f8
[Mon Mar 11 14:29:15 2024] RBP: 00007f66427fb900 R08: 0000000000000008 R09: 0000000000000000
[Mon Mar 11 14:29:15 2024] R10: 00007f66427fb900 R11: 0000000000000293 R12: 00007ffd9217b000
[Mon Mar 11 14:29:15 2024] R13: 0000000000000000 R14: 000055b9f0d28780 R15: 0000000000000000
[Mon Mar 11 14:29:15 2024] </TASK>
[Mon Mar 11 14:29:15 2024] task:automount state:S stack:0 pid:88147 ppid:1 flags:0x00000002
[Mon Mar 11 14:29:15 2024] Call Trace:
[Mon Mar 11 14:29:15 2024] <TASK>
[Mon Mar 11 14:29:15 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:15 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:15 2024] schedule_hrtimeout_range_clock+0x10f/0x120
[Mon Mar 11 14:29:15 2024] ? add_wait_queue+0x65/0xa0
[Mon Mar 11 14:29:15 2024] do_poll.constprop.0+0x248/0x390
[Mon Mar 11 14:29:15 2024] do_sys_poll+0x1c8/0x250
[Mon Mar 11 14:29:15 2024] ? newidle_balance+0x2e5/0x400
[Mon Mar 11 14:29:15 2024] ? update_load_avg+0x7e/0x740
[Mon Mar 11 14:29:15 2024] ? pick_next_task_fair+0x41/0x500
[Mon Mar 11 14:29:15 2024] ? put_prev_task_fair+0x1e/0x40
[Mon Mar 11 14:29:15 2024] ? pick_next_task+0x861/0x950
[Mon Mar 11 14:29:15 2024] ? __update_idle_core+0x23/0xc0
[Mon Mar 11 14:29:15 2024] ? __pfx_pollwake+0x10/0x10
[Mon Mar 11 14:29:15 2024] ? __remove_hrtimer+0x39/0x90
[Mon Mar 11 14:29:15 2024] ? hrtimer_try_to_cancel.part.0+0x50/0xf0
[Mon Mar 11 14:29:15 2024] ? hrtimer_cancel+0x1d/0x40
[Mon Mar 11 14:29:15 2024] ? futex_wait+0x23e/0x260
[Mon Mar 11 14:29:15 2024] ? __pfx_hrtimer_wakeup+0x10/0x10
[Mon Mar 11 14:29:15 2024] ? __audit_filter_op+0xa5/0xf0
[Mon Mar 11 14:29:15 2024] ? audit_reset_context.part.0.constprop.0+0x273/0x2e0
[Mon Mar 11 14:29:15 2024] ? _copy_from_user+0x27/0x60
[Mon Mar 11 14:29:15 2024] __x64_sys_ppoll+0xbc/0x150
[Mon Mar 11 14:29:15 2024] do_syscall_64+0x59/0x90
[Mon Mar 11 14:29:15 2024] ? exc_page_fault+0x62/0x150
[Mon Mar 11 14:29:15 2024] entry_SYSCALL_64_after_hwframe+0x72/0xdc
[Mon Mar 11 14:29:15 2024] RIP: 0033:0x7f66af3427fe
[Mon Mar 11 14:29:15 2024] RSP: 002b:00007f6641ffa8a0 EFLAGS: 00000293 ORIG_RAX: 000000000000010f
[Mon Mar 11 14:29:15 2024] RAX: ffffffffffffffda RBX: 00007f6641ffb528 RCX: 00007f66af3427fe
[Mon Mar 11 14:29:15 2024] RDX: 0000000000000000 RSI: 0000000000000001 RDI: 00007f6641ffa8f8
[Mon Mar 11 14:29:15 2024] RBP: 00007f6641ffa900 R08: 0000000000000008 R09: 0000000000000000
[Mon Mar 11 14:29:15 2024] R10: 00007f6641ffa900 R11: 0000000000000293 R12: 00007ffd9217b000
[Mon Mar 11 14:29:15 2024] R13: 0000000000000000 R14: 000055b9f0d2eb10 R15: 0000000000000000
[Mon Mar 11 14:29:15 2024] </TASK>
[Mon Mar 11 14:29:15 2024] task:automount state:S stack:0 pid:88148 ppid:1 flags:0x00000002
[Mon Mar 11 14:29:15 2024] Call Trace:
[Mon Mar 11 14:29:15 2024] <TASK>
[Mon Mar 11 14:29:15 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:15 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:15 2024] schedule_hrtimeout_range_clock+0x10f/0x120
[Mon Mar 11 14:29:15 2024] ? add_wait_queue+0x65/0xa0
[Mon Mar 11 14:29:15 2024] do_poll.constprop.0+0x248/0x390
[Mon Mar 11 14:29:15 2024] do_sys_poll+0x1c8/0x250
[Mon Mar 11 14:29:15 2024] ? sched_clock+0xc/0x30
[Mon Mar 11 14:29:15 2024] ? __smp_call_single_queue+0x93/0x120
[Mon Mar 11 14:29:15 2024] ? ttwu_queue_wakelist+0xf2/0x110
[Mon Mar 11 14:29:15 2024] ? try_to_wake_up+0x3e2/0x5d0
[Mon Mar 11 14:29:15 2024] ? __alloc_skb+0x8e/0x1d0
[Mon Mar 11 14:29:15 2024] ? autoremove_wake_function+0x30/0x60
[Mon Mar 11 14:29:15 2024] ? __wake_up_common+0x75/0xa0
[Mon Mar 11 14:29:15 2024] ? __pfx_pollwake+0x10/0x10
[Mon Mar 11 14:29:15 2024] ? __wake_up_common+0x75/0xa0
[Mon Mar 11 14:29:15 2024] ? select_idle_sibling+0x28/0x430
[Mon Mar 11 14:29:15 2024] ? __wake_up_sync_key+0x39/0x50
[Mon Mar 11 14:29:15 2024] ? wake_affine+0x62/0x1f0
[Mon Mar 11 14:29:15 2024] ? select_idle_sibling+0x28/0x430
[Mon Mar 11 14:29:15 2024] ? sched_clock+0xc/0x30
[Mon Mar 11 14:29:15 2024] ? wake_affine+0x62/0x1f0
[Mon Mar 11 14:29:15 2024] ? sched_clock+0xc/0x30
[Mon Mar 11 14:29:15 2024] ? __smp_call_single_queue+0x93/0x120
[Mon Mar 11 14:29:15 2024] ? ttwu_queue_wakelist+0xf2/0x110
[Mon Mar 11 14:29:15 2024] ? __audit_filter_op+0xa5/0xf0
[Mon Mar 11 14:29:15 2024] ? audit_reset_context.part.0.constprop.0+0x273/0x2e0
[Mon Mar 11 14:29:15 2024] __x64_sys_ppoll+0xbc/0x150
[Mon Mar 11 14:29:15 2024] do_syscall_64+0x59/0x90
[Mon Mar 11 14:29:15 2024] ? exc_page_fault+0x62/0x150
[Mon Mar 11 14:29:15 2024] entry_SYSCALL_64_after_hwframe+0x72/0xdc
[Mon Mar 11 14:29:15 2024] RIP: 0033:0x7f66af3427fe
[Mon Mar 11 14:29:15 2024] RSP: 002b:00007f66417f98a0 EFLAGS: 00000293 ORIG_RAX: 000000000000010f
[Mon Mar 11 14:29:15 2024] RAX: ffffffffffffffda RBX: 00007f66417fa528 RCX: 00007f66af3427fe
[Mon Mar 11 14:29:15 2024] RDX: 0000000000000000 RSI: 0000000000000001 RDI: 00007f66417f98f8
[Mon Mar 11 14:29:15 2024] RBP: 00007f66417f9900 R08: 0000000000000008 R09: 0000000000000000
[Mon Mar 11 14:29:15 2024] R10: 00007f66417f9900 R11: 0000000000000293 R12: 00007ffd9217b000
[Mon Mar 11 14:29:15 2024] R13: 0000000000000000 R14: 000055b9f0d34ea0 R15: 0000000000000000
[Mon Mar 11 14:29:15 2024] </TASK>
[Mon Mar 11 14:29:15 2024] task:automount state:S stack:0 pid:88149 ppid:1 flags:0x00000002
[Mon Mar 11 14:29:15 2024] Call Trace:
[Mon Mar 11 14:29:15 2024] <TASK>
[Mon Mar 11 14:29:15 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:15 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:15 2024] schedule_hrtimeout_range_clock+0x10f/0x120
[Mon Mar 11 14:29:15 2024] ? xas_load+0x9/0xa0
[Mon Mar 11 14:29:15 2024] ? xa_load+0x70/0xb0
[Mon Mar 11 14:29:15 2024] ? add_wait_queue+0x65/0xa0
[Mon Mar 11 14:29:15 2024] do_poll.constprop.0+0x248/0x390
[Mon Mar 11 14:29:15 2024] do_sys_poll+0x1c8/0x250
[Mon Mar 11 14:29:15 2024] ? ttwu_queue_wakelist+0xf2/0x110
[Mon Mar 11 14:29:15 2024] ? try_to_wake_up+0x3e2/0x5d0
[Mon Mar 11 14:29:15 2024] ? __alloc_skb+0x8e/0x1d0
[Mon Mar 11 14:29:15 2024] ? autoremove_wake_function+0x30/0x60
[Mon Mar 11 14:29:15 2024] ? __wake_up_common+0x75/0xa0
[Mon Mar 11 14:29:15 2024] ? __pfx_pollwake+0x10/0x10
[Mon Mar 11 14:29:15 2024] ? select_idle_sibling+0x28/0x430
[Mon Mar 11 14:29:15 2024] ? wake_affine+0x62/0x1f0
[Mon Mar 11 14:29:15 2024] ? sched_clock+0xc/0x30
[Mon Mar 11 14:29:15 2024] ? __smp_call_single_queue+0x93/0x120
[Mon Mar 11 14:29:15 2024] ? ttwu_queue_wakelist+0xf2/0x110
[Mon Mar 11 14:29:15 2024] ? try_to_wake_up+0x3e2/0x5d0
[Mon Mar 11 14:29:15 2024] ? ttwu_queue_wakelist+0xf2/0x110
[Mon Mar 11 14:29:15 2024] ? futex_wake+0x151/0x180
[Mon Mar 11 14:29:15 2024] ? __audit_filter_op+0xa5/0xf0
[Mon Mar 11 14:29:15 2024] ? _copy_from_user+0x27/0x60
[Mon Mar 11 14:29:15 2024] __x64_sys_ppoll+0xbc/0x150
[Mon Mar 11 14:29:15 2024] do_syscall_64+0x59/0x90
[Mon Mar 11 14:29:15 2024] ? syscall_exit_work+0x103/0x130
[Mon Mar 11 14:29:15 2024] ? syscall_exit_to_user_mode+0x22/0x40
[Mon Mar 11 14:29:15 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:15 2024] ? exc_page_fault+0x62/0x150
[Mon Mar 11 14:29:15 2024] entry_SYSCALL_64_after_hwframe+0x72/0xdc
[Mon Mar 11 14:29:15 2024] RIP: 0033:0x7f66af3427fe
[Mon Mar 11 14:29:15 2024] RSP: 002b:00007f6640ff88a0 EFLAGS: 00000293 ORIG_RAX: 000000000000010f
[Mon Mar 11 14:29:15 2024] RAX: ffffffffffffffda RBX: 00007f6640ff9528 RCX: 00007f66af3427fe
[Mon Mar 11 14:29:15 2024] RDX: 0000000000000000 RSI: 0000000000000001 RDI: 00007f6640ff88f8
[Mon Mar 11 14:29:15 2024] RBP: 00007f6640ff8900 R08: 0000000000000008 R09: 0000000000000000
[Mon Mar 11 14:29:15 2024] R10: 00007f6640ff8900 R11: 0000000000000293 R12: 00007ffd9217b000
[Mon Mar 11 14:29:15 2024] R13: 0000000000000000 R14: 000055b9f0d3b230 R15: 0000000000000000
[Mon Mar 11 14:29:15 2024] </TASK>
[Mon Mar 11 14:29:15 2024] task:automount state:S stack:0 pid:88150 ppid:1 flags:0x00000002
[Mon Mar 11 14:29:15 2024] Call Trace:
[Mon Mar 11 14:29:15 2024] <TASK>
[Mon Mar 11 14:29:15 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:15 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:15 2024] schedule_hrtimeout_range_clock+0x10f/0x120
[Mon Mar 11 14:29:15 2024] ? handle_mm_fault+0xcd/0x290
[Mon Mar 11 14:29:15 2024] ? add_wait_queue+0x65/0xa0
[Mon Mar 11 14:29:15 2024] do_poll.constprop.0+0x248/0x390
[Mon Mar 11 14:29:15 2024] do_sys_poll+0x1c8/0x250
[Mon Mar 11 14:29:15 2024] ? update_sg_wakeup_stats+0x78/0x3b0
[Mon Mar 11 14:29:15 2024] ? find_idlest_group+0x2b2/0x530
[Mon Mar 11 14:29:15 2024] ? newidle_balance+0x2e5/0x400
[Mon Mar 11 14:29:15 2024] ? update_load_avg+0x7e/0x740
[Mon Mar 11 14:29:15 2024] ? __pfx_pollwake+0x10/0x10
[Mon Mar 11 14:29:15 2024] ? __schedule+0x223/0x550
[Mon Mar 11 14:29:15 2024] ? timerqueue_del+0x2a/0x50
[Mon Mar 11 14:29:15 2024] ? __remove_hrtimer+0x39/0x90
[Mon Mar 11 14:29:15 2024] ? hrtimer_try_to_cancel.part.0+0x50/0xf0
[Mon Mar 11 14:29:15 2024] ? hrtimer_cancel+0x1d/0x40
[Mon Mar 11 14:29:15 2024] ? futex_wait+0x23e/0x260
[Mon Mar 11 14:29:15 2024] ? __pfx_hrtimer_wakeup+0x10/0x10
[Mon Mar 11 14:29:15 2024] ? futex_wake+0x7c/0x180
[Mon Mar 11 14:29:15 2024] ? __audit_filter_op+0xa5/0xf0
[Mon Mar 11 14:29:15 2024] __x64_sys_ppoll+0xbc/0x150
[Mon Mar 11 14:29:15 2024] do_syscall_64+0x59/0x90
[Mon Mar 11 14:29:15 2024] ? syscall_exit_work+0x103/0x130
[Mon Mar 11 14:29:15 2024] ? syscall_exit_to_user_mode+0x22/0x40
[Mon Mar 11 14:29:15 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:15 2024] entry_SYSCALL_64_after_hwframe+0x72/0xdc
[Mon Mar 11 14:29:15 2024] RIP: 0033:0x7f66af3427fe
[Mon Mar 11 14:29:15 2024] RSP: 002b:00007f6623ffe8a0 EFLAGS: 00000293 ORIG_RAX: 000000000000010f
[Mon Mar 11 14:29:15 2024] RAX: ffffffffffffffda RBX: 00007f6623fff528 RCX: 00007f66af3427fe
[Mon Mar 11 14:29:15 2024] RDX: 0000000000000000 RSI: 0000000000000001 RDI: 00007f6623ffe8f8
[Mon Mar 11 14:29:15 2024] RBP: 00007f6623ffe900 R08: 0000000000000008 R09: 0000000000000000
[Mon Mar 11 14:29:15 2024] R10: 00007f6623ffe900 R11: 0000000000000293 R12: 00007ffd9217b000
[Mon Mar 11 14:29:15 2024] R13: 0000000000000000 R14: 000055b9f0d415c0 R15: 0000000000000000
[Mon Mar 11 14:29:15 2024] </TASK>
[Mon Mar 11 14:29:15 2024] task:automount state:S stack:0 pid:88151 ppid:1 flags:0x00000002
[Mon Mar 11 14:29:15 2024] Call Trace:
[Mon Mar 11 14:29:15 2024] <TASK>
[Mon Mar 11 14:29:15 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:15 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:15 2024] schedule_hrtimeout_range_clock+0x10f/0x120
[Mon Mar 11 14:29:15 2024] ? add_wait_queue+0x65/0xa0
[Mon Mar 11 14:29:15 2024] do_poll.constprop.0+0x248/0x390
[Mon Mar 11 14:29:15 2024] do_sys_poll+0x1c8/0x250
[Mon Mar 11 14:29:15 2024] ? __kmem_cache_alloc_node+0x1c7/0x2d0
[Mon Mar 11 14:29:15 2024] ? memcg_alloc_slab_cgroups+0x39/0xa0
[Mon Mar 11 14:29:15 2024] ? select_idle_sibling+0x28/0x430
[Mon Mar 11 14:29:15 2024] ? wake_affine+0x62/0x1f0
[Mon Mar 11 14:29:15 2024] ? __pfx_pollwake+0x10/0x10
[Mon Mar 11 14:29:15 2024] ? __alloc_skb+0x8e/0x1d0
[Mon Mar 11 14:29:15 2024] ? autoremove_wake_function+0x30/0x60
[Mon Mar 11 14:29:15 2024] ? __wake_up_common+0x75/0xa0
[Mon Mar 11 14:29:15 2024] ? __wake_up+0x40/0x60
[Mon Mar 11 14:29:15 2024] ? select_idle_sibling+0x28/0x430
[Mon Mar 11 14:29:15 2024] ? wake_affine+0x62/0x1f0
[Mon Mar 11 14:29:15 2024] ? sched_clock+0xc/0x30
[Mon Mar 11 14:29:15 2024] ? __smp_call_single_queue+0x93/0x120
[Mon Mar 11 14:29:15 2024] ? select_idle_sibling+0x28/0x430
[Mon Mar 11 14:29:15 2024] ? ttwu_queue_wakelist+0xf2/0x110
[Mon Mar 11 14:29:15 2024] ? wake_affine+0x62/0x1f0
[Mon Mar 11 14:29:15 2024] ? sched_clock+0xc/0x30
[Mon Mar 11 14:29:15 2024] ? __smp_call_single_queue+0x93/0x120
[Mon Mar 11 14:29:15 2024] ? ttwu_queue_wakelist+0xf2/0x110
[Mon Mar 11 14:29:15 2024] ? try_to_wake_up+0x3e2/0x5d0
[Mon Mar 11 14:29:15 2024] __x64_sys_ppoll+0xbc/0x150
[Mon Mar 11 14:29:15 2024] do_syscall_64+0x59/0x90
[Mon Mar 11 14:29:15 2024] ? audit_reset_context.part.0.constprop.0+0x273/0x2e0
[Mon Mar 11 14:29:15 2024] ? syscall_exit_work+0x103/0x130
[Mon Mar 11 14:29:15 2024] ? syscall_exit_to_user_mode+0x22/0x40
[Mon Mar 11 14:29:15 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:15 2024] ? syscall_exit_to_user_mode+0x22/0x40
[Mon Mar 11 14:29:15 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:15 2024] ? exc_page_fault+0x62/0x150
[Mon Mar 11 14:29:15 2024] entry_SYSCALL_64_after_hwframe+0x72/0xdc
[Mon Mar 11 14:29:15 2024] RIP: 0033:0x7f66af3427fe
[Mon Mar 11 14:29:15 2024] RSP: 002b:00007f66237fd8a0 EFLAGS: 00000293 ORIG_RAX: 000000000000010f
[Mon Mar 11 14:29:15 2024] RAX: ffffffffffffffda RBX: 00007f66237fe528 RCX: 00007f66af3427fe
[Mon Mar 11 14:29:15 2024] RDX: 0000000000000000 RSI: 0000000000000001 RDI: 00007f66237fd8f8
[Mon Mar 11 14:29:15 2024] RBP: 00007f66237fd900 R08: 0000000000000008 R09: 0000000000000000
[Mon Mar 11 14:29:15 2024] R10: 00007f66237fd900 R11: 0000000000000293 R12: 00007ffd9217b000
[Mon Mar 11 14:29:15 2024] R13: 0000000000000000 R14: 000055b9f0d47950 R15: 0000000000000000
[Mon Mar 11 14:29:15 2024] </TASK>
[Mon Mar 11 14:29:15 2024] task:automount state:S stack:0 pid:88152 ppid:1 flags:0x00000002
[Mon Mar 11 14:29:15 2024] Call Trace:
[Mon Mar 11 14:29:15 2024] <TASK>
[Mon Mar 11 14:29:15 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:15 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:15 2024] schedule_hrtimeout_range_clock+0x10f/0x120
[Mon Mar 11 14:29:15 2024] ? add_wait_queue+0x65/0xa0
[Mon Mar 11 14:29:15 2024] do_poll.constprop.0+0x248/0x390
[Mon Mar 11 14:29:15 2024] do_sys_poll+0x1c8/0x250
[Mon Mar 11 14:29:15 2024] ? ttwu_queue_wakelist+0xf2/0x110
[Mon Mar 11 14:29:15 2024] ? try_to_wake_up+0x3e2/0x5d0
[Mon Mar 11 14:29:15 2024] ? __alloc_skb+0x8e/0x1d0
[Mon Mar 11 14:29:15 2024] ? autoremove_wake_function+0x30/0x60
[Mon Mar 11 14:29:15 2024] ? __wake_up_common+0x75/0xa0
[Mon Mar 11 14:29:15 2024] ? __pfx_pollwake+0x10/0x10
[Mon Mar 11 14:29:15 2024] ? __wake_up_common+0x75/0xa0
[Mon Mar 11 14:29:15 2024] ? select_idle_sibling+0x28/0x430
[Mon Mar 11 14:29:15 2024] ? wake_affine+0x62/0x1f0
[Mon Mar 11 14:29:15 2024] ? sched_clock+0xc/0x30
[Mon Mar 11 14:29:15 2024] ? __smp_call_single_queue+0x93/0x120
[Mon Mar 11 14:29:15 2024] ? ttwu_queue_wakelist+0xf2/0x110
[Mon Mar 11 14:29:15 2024] ? try_to_wake_up+0x3e2/0x5d0
[Mon Mar 11 14:29:15 2024] ? __audit_filter_op+0xa5/0xf0
[Mon Mar 11 14:29:15 2024] ? audit_reset_context.part.0.constprop.0+0x273/0x2e0
[Mon Mar 11 14:29:15 2024] ? _copy_from_user+0x27/0x60
[Mon Mar 11 14:29:15 2024] __x64_sys_ppoll+0xbc/0x150
[Mon Mar 11 14:29:15 2024] do_syscall_64+0x59/0x90
[Mon Mar 11 14:29:15 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:15 2024] ? syscall_exit_to_user_mode+0x22/0x40
[Mon Mar 11 14:29:15 2024] ? exc_page_fault+0x62/0x150
[Mon Mar 11 14:29:15 2024] entry_SYSCALL_64_after_hwframe+0x72/0xdc
[Mon Mar 11 14:29:15 2024] RIP: 0033:0x7f66af3427fe
[Mon Mar 11 14:29:15 2024] RSP: 002b:00007f6622ffc8a0 EFLAGS: 00000293 ORIG_RAX: 000000000000010f
[Mon Mar 11 14:29:15 2024] RAX: ffffffffffffffda RBX: 00007f6622ffd528 RCX: 00007f66af3427fe
[Mon Mar 11 14:29:15 2024] RDX: 0000000000000000 RSI: 0000000000000001 RDI: 00007f6622ffc8f8
[Mon Mar 11 14:29:15 2024] RBP: 00007f6622ffc900 R08: 0000000000000008 R09: 0000000000000000
[Mon Mar 11 14:29:15 2024] R10: 00007f6622ffc900 R11: 0000000000000293 R12: 00007ffd9217b000
[Mon Mar 11 14:29:15 2024] R13: 0000000000000000 R14: 000055b9f0d4dce0 R15: 0000000000000000
[Mon Mar 11 14:29:15 2024] </TASK>
[Mon Mar 11 14:29:15 2024] task:automount state:S stack:0 pid:88153 ppid:1 flags:0x00000002
[Mon Mar 11 14:29:15 2024] Call Trace:
[Mon Mar 11 14:29:15 2024] <TASK>
[Mon Mar 11 14:29:15 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:15 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:15 2024] schedule_hrtimeout_range_clock+0x10f/0x120
[Mon Mar 11 14:29:15 2024] ? add_wait_queue+0x65/0xa0
[Mon Mar 11 14:29:15 2024] do_poll.constprop.0+0x248/0x390
[Mon Mar 11 14:29:15 2024] do_sys_poll+0x1c8/0x250
[Mon Mar 11 14:29:15 2024] ? select_idle_sibling+0x28/0x430
[Mon Mar 11 14:29:15 2024] ? wake_affine+0x62/0x1f0
[Mon Mar 11 14:29:15 2024] ? sched_clock+0xc/0x30
[Mon Mar 11 14:29:15 2024] ? __smp_call_single_queue+0x93/0x120
[Mon Mar 11 14:29:15 2024] ? ttwu_queue_wakelist+0xf2/0x110
[Mon Mar 11 14:29:15 2024] ? try_to_wake_up+0x3e2/0x5d0
[Mon Mar 11 14:29:15 2024] ? __alloc_skb+0x8e/0x1d0
[Mon Mar 11 14:29:15 2024] ? __pfx_pollwake+0x10/0x10
[Mon Mar 11 14:29:15 2024] ? __wake_up+0x40/0x60
[Mon Mar 11 14:29:15 2024] ? ep_poll_callback+0x25d/0x2a0
[Mon Mar 11 14:29:15 2024] ? select_idle_sibling+0x28/0x430
[Mon Mar 11 14:29:15 2024] ? wake_affine+0x62/0x1f0
[Mon Mar 11 14:29:15 2024] ? select_idle_sibling+0x28/0x430
[Mon Mar 11 14:29:15 2024] ? sched_clock+0xc/0x30
[Mon Mar 11 14:29:15 2024] ? __smp_call_single_queue+0x93/0x120
[Mon Mar 11 14:29:15 2024] ? wake_affine+0x62/0x1f0
[Mon Mar 11 14:29:15 2024] ? sched_clock+0xc/0x30
[Mon Mar 11 14:29:15 2024] ? __smp_call_single_queue+0x93/0x120
[Mon Mar 11 14:29:15 2024] ? ttwu_queue_wakelist+0xf2/0x110
[Mon Mar 11 14:29:15 2024] ? try_to_wake_up+0x3e2/0x5d0
[Mon Mar 11 14:29:15 2024] ? futex_wake+0x151/0x180
[Mon Mar 11 14:29:15 2024] ? _copy_from_user+0x27/0x60
[Mon Mar 11 14:29:15 2024] __x64_sys_ppoll+0xbc/0x150
[Mon Mar 11 14:29:15 2024] do_syscall_64+0x59/0x90
[Mon Mar 11 14:29:15 2024] ? audit_reset_context.part.0.constprop.0+0x273/0x2e0
[Mon Mar 11 14:29:15 2024] ? syscall_exit_work+0x103/0x130
[Mon Mar 11 14:29:15 2024] ? syscall_exit_to_user_mode+0x22/0x40
[Mon Mar 11 14:29:15 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:15 2024] ? exc_page_fault+0x62/0x150
[Mon Mar 11 14:29:15 2024] entry_SYSCALL_64_after_hwframe+0x72/0xdc
[Mon Mar 11 14:29:15 2024] RIP: 0033:0x7f66af3427fe
[Mon Mar 11 14:29:15 2024] RSP: 002b:00007f66227fb8a0 EFLAGS: 00000293 ORIG_RAX: 000000000000010f
[Mon Mar 11 14:29:15 2024] RAX: ffffffffffffffda RBX: 00007f66227fc528 RCX: 00007f66af3427fe
[Mon Mar 11 14:29:15 2024] RDX: 0000000000000000 RSI: 0000000000000001 RDI: 00007f66227fb8f8
[Mon Mar 11 14:29:15 2024] RBP: 00007f66227fb900 R08: 0000000000000008 R09: 0000000000000000
[Mon Mar 11 14:29:15 2024] R10: 00007f66227fb900 R11: 0000000000000293 R12: 00007ffd9217b000
[Mon Mar 11 14:29:15 2024] R13: 0000000000000000 R14: 000055b9f0d54070 R15: 0000000000000000
[Mon Mar 11 14:29:15 2024] </TASK>
[Mon Mar 11 14:29:15 2024] task:automount state:S stack:0 pid:88154 ppid:1 flags:0x00000002
[Mon Mar 11 14:29:15 2024] Call Trace:
[Mon Mar 11 14:29:15 2024] <TASK>
[Mon Mar 11 14:29:15 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:15 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:15 2024] schedule_hrtimeout_range_clock+0x10f/0x120
[Mon Mar 11 14:29:15 2024] ? xas_load+0x9/0xa0
[Mon Mar 11 14:29:15 2024] ? __find_get_block+0x1fb/0x370
[Mon Mar 11 14:29:15 2024] ? add_wait_queue+0x65/0xa0
[Mon Mar 11 14:29:15 2024] do_poll.constprop.0+0x248/0x390
[Mon Mar 11 14:29:15 2024] do_sys_poll+0x1c8/0x250
[Mon Mar 11 14:29:15 2024] ? sched_clock+0xc/0x30
[Mon Mar 11 14:29:15 2024] ? __smp_call_single_queue+0x93/0x120
[Mon Mar 11 14:29:15 2024] ? ttwu_queue_wakelist+0xf2/0x110
[Mon Mar 11 14:29:15 2024] ? try_to_wake_up+0x3e2/0x5d0
[Mon Mar 11 14:29:15 2024] ? __alloc_skb+0x8e/0x1d0
[Mon Mar 11 14:29:15 2024] ? autoremove_wake_function+0x30/0x60
[Mon Mar 11 14:29:15 2024] ? __pfx_pollwake+0x10/0x10
[Mon Mar 11 14:29:15 2024] ? ep_poll_callback+0x25d/0x2a0
[Mon Mar 11 14:29:15 2024] ? select_idle_sibling+0x28/0x430
[Mon Mar 11 14:29:15 2024] ? __wake_up_common+0x75/0xa0
[Mon Mar 11 14:29:15 2024] ? wake_affine+0x62/0x1f0
[Mon Mar 11 14:29:15 2024] ? sched_clock+0xc/0x30
[Mon Mar 11 14:29:15 2024] ? __smp_call_single_queue+0x93/0x120
[Mon Mar 11 14:29:15 2024] ? ttwu_queue_wakelist+0xf2/0x110
[Mon Mar 11 14:29:15 2024] ? try_to_wake_up+0x3e2/0x5d0
[Mon Mar 11 14:29:15 2024] ? __audit_filter_op+0xa5/0xf0
[Mon Mar 11 14:29:15 2024] ? audit_reset_context.part.0.constprop.0+0x273/0x2e0
[Mon Mar 11 14:29:15 2024] ? _copy_from_user+0x27/0x60
[Mon Mar 11 14:29:15 2024] __x64_sys_ppoll+0xbc/0x150
[Mon Mar 11 14:29:15 2024] do_syscall_64+0x59/0x90
[Mon Mar 11 14:29:15 2024] ? do_user_addr_fault+0x1d6/0x6a0
[Mon Mar 11 14:29:15 2024] ? syscall_exit_work+0x103/0x130
[Mon Mar 11 14:29:15 2024] ? exc_page_fault+0x62/0x150
[Mon Mar 11 14:29:15 2024] entry_SYSCALL_64_after_hwframe+0x72/0xdc
[Mon Mar 11 14:29:15 2024] RIP: 0033:0x7f66af3427fe
[Mon Mar 11 14:29:15 2024] RSP: 002b:00007f6621ffa8a0 EFLAGS: 00000293 ORIG_RAX: 000000000000010f
[Mon Mar 11 14:29:15 2024] RAX: ffffffffffffffda RBX: 00007f6621ffb528 RCX: 00007f66af3427fe
[Mon Mar 11 14:29:15 2024] RDX: 0000000000000000 RSI: 0000000000000001 RDI: 00007f6621ffa8f8
[Mon Mar 11 14:29:15 2024] RBP: 00007f6621ffa900 R08: 0000000000000008 R09: 0000000000000000
[Mon Mar 11 14:29:15 2024] R10: 00007f6621ffa900 R11: 0000000000000293 R12: 00007ffd9217b000
[Mon Mar 11 14:29:15 2024] R13: 0000000000000000 R14: 000055b9f0d5a400 R15: 0000000000000000
[Mon Mar 11 14:29:15 2024] </TASK>
[Mon Mar 11 14:29:15 2024] task:automount state:S stack:0 pid:88155 ppid:1 flags:0x00000002
[Mon Mar 11 14:29:15 2024] Call Trace:
[Mon Mar 11 14:29:15 2024] <TASK>
[Mon Mar 11 14:29:15 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:15 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:15 2024] schedule_hrtimeout_range_clock+0x10f/0x120
[Mon Mar 11 14:29:15 2024] ? __find_get_block+0x1fb/0x370
[Mon Mar 11 14:29:15 2024] ? add_wait_queue+0x65/0xa0
[Mon Mar 11 14:29:15 2024] do_poll.constprop.0+0x248/0x390
[Mon Mar 11 14:29:15 2024] do_sys_poll+0x1c8/0x250
[Mon Mar 11 14:29:15 2024] ? __mod_memcg_lruvec_state+0x84/0xd0
[Mon Mar 11 14:29:15 2024] ? mod_objcg_state+0x1fe/0x300
[Mon Mar 11 14:29:15 2024] ? select_idle_sibling+0x28/0x430
[Mon Mar 11 14:29:15 2024] ? wake_affine+0x62/0x1f0
[Mon Mar 11 14:29:15 2024] ? __pfx_pollwake+0x10/0x10
[Mon Mar 11 14:29:15 2024] ? __alloc_skb+0x8e/0x1d0
[Mon Mar 11 14:29:15 2024] ? autoremove_wake_function+0x30/0x60
[Mon Mar 11 14:29:15 2024] ? __wake_up_common+0x75/0xa0
[Mon Mar 11 14:29:15 2024] ? __wake_up+0x40/0x60
[Mon Mar 11 14:29:15 2024] ? ep_poll_callback+0x25d/0x2a0
[Mon Mar 11 14:29:15 2024] ? __wake_up_common+0x75/0xa0
[Mon Mar 11 14:29:15 2024] ? select_idle_sibling+0x28/0x430
[Mon Mar 11 14:29:15 2024] ? wake_affine+0x62/0x1f0
[Mon Mar 11 14:29:15 2024] ? sched_clock+0xc/0x30
[Mon Mar 11 14:29:15 2024] ? __smp_call_single_queue+0x93/0x120
[Mon Mar 11 14:29:15 2024] ? ttwu_queue_wakelist+0xf2/0x110
[Mon Mar 11 14:29:15 2024] ? try_to_wake_up+0x3e2/0x5d0
[Mon Mar 11 14:29:15 2024] __x64_sys_ppoll+0xbc/0x150
[Mon Mar 11 14:29:15 2024] do_syscall_64+0x59/0x90
[Mon Mar 11 14:29:15 2024] ? __audit_filter_op+0xa5/0xf0
[Mon Mar 11 14:29:15 2024] ? audit_reset_context.part.0.constprop.0+0x273/0x2e0
[Mon Mar 11 14:29:15 2024] ? syscall_exit_work+0x103/0x130
[Mon Mar 11 14:29:15 2024] ? syscall_exit_to_user_mode+0x22/0x40
[Mon Mar 11 14:29:15 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:15 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:15 2024] entry_SYSCALL_64_after_hwframe+0x72/0xdc
[Mon Mar 11 14:29:15 2024] RIP: 0033:0x7f66af3427fe
[Mon Mar 11 14:29:15 2024] RSP: 002b:00007f66217f98a0 EFLAGS: 00000293 ORIG_RAX: 000000000000010f
[Mon Mar 11 14:29:15 2024] RAX: ffffffffffffffda RBX: 00007f66217fa528 RCX: 00007f66af3427fe
[Mon Mar 11 14:29:15 2024] RDX: 0000000000000000 RSI: 0000000000000001 RDI: 00007f66217f98f8
[Mon Mar 11 14:29:15 2024] RBP: 00007f66217f9900 R08: 0000000000000008 R09: 0000000000000000
[Mon Mar 11 14:29:15 2024] R10: 00007f66217f9900 R11: 0000000000000293 R12: 00007ffd9217b000
[Mon Mar 11 14:29:15 2024] R13: 0000000000000000 R14: 000055b9f0d60790 R15: 0000000000000000
[Mon Mar 11 14:29:15 2024] </TASK>
[Mon Mar 11 14:29:15 2024] task:automount state:S stack:0 pid:88156 ppid:1 flags:0x00000002
[Mon Mar 11 14:29:15 2024] Call Trace:
[Mon Mar 11 14:29:15 2024] <TASK>
[Mon Mar 11 14:29:15 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:15 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:15 2024] schedule_hrtimeout_range_clock+0x10f/0x120
[Mon Mar 11 14:29:15 2024] ? add_wait_queue+0x65/0xa0
[Mon Mar 11 14:29:15 2024] do_poll.constprop.0+0x248/0x390
[Mon Mar 11 14:29:15 2024] do_sys_poll+0x1c8/0x250
[Mon Mar 11 14:29:15 2024] ? chacha_block_generic+0x6f/0xb0
[Mon Mar 11 14:29:15 2024] ? update_sg_wakeup_stats+0x78/0x3b0
[Mon Mar 11 14:29:15 2024] ? __pfx_pollwake+0x10/0x10
[Mon Mar 11 14:29:15 2024] ? pick_next_task_fair+0x41/0x500
[Mon Mar 11 14:29:15 2024] ? put_prev_task_fair+0x1e/0x40
[Mon Mar 11 14:29:15 2024] ? pick_next_task+0x861/0x950
[Mon Mar 11 14:29:15 2024] ? __update_idle_core+0x23/0xc0
[Mon Mar 11 14:29:15 2024] ? __switch_to_asm+0x3a/0x80
[Mon Mar 11 14:29:15 2024] ? finish_task_switch.isra.0+0x8c/0x2a0
[Mon Mar 11 14:29:15 2024] ? __schedule+0x223/0x550
[Mon Mar 11 14:29:15 2024] ? timerqueue_del+0x2a/0x50
[Mon Mar 11 14:29:15 2024] ? __remove_hrtimer+0x39/0x90
[Mon Mar 11 14:29:15 2024] ? hrtimer_cancel+0x1d/0x40
[Mon Mar 11 14:29:15 2024] ? __audit_filter_op+0xa5/0xf0
[Mon Mar 11 14:29:15 2024] ? audit_reset_context.part.0.constprop.0+0x273/0x2e0
[Mon Mar 11 14:29:15 2024] ? _copy_from_user+0x27/0x60
[Mon Mar 11 14:29:15 2024] __x64_sys_ppoll+0xbc/0x150
[Mon Mar 11 14:29:15 2024] do_syscall_64+0x59/0x90
[Mon Mar 11 14:29:15 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:15 2024] ? exit_to_user_mode_prepare+0xec/0x100
[Mon Mar 11 14:29:15 2024] ? syscall_exit_to_user_mode+0x22/0x40
[Mon Mar 11 14:29:15 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:15 2024] ? exc_page_fault+0x62/0x150
[Mon Mar 11 14:29:15 2024] entry_SYSCALL_64_after_hwframe+0x72/0xdc
[Mon Mar 11 14:29:15 2024] RIP: 0033:0x7f66af3427fe
[Mon Mar 11 14:29:15 2024] RSP: 002b:00007f6620ff88a0 EFLAGS: 00000293 ORIG_RAX: 000000000000010f
[Mon Mar 11 14:29:15 2024] RAX: ffffffffffffffda RBX: 00007f6620ff9528 RCX: 00007f66af3427fe
[Mon Mar 11 14:29:15 2024] RDX: 0000000000000000 RSI: 0000000000000001 RDI: 00007f6620ff88f8
[Mon Mar 11 14:29:15 2024] RBP: 00007f6620ff8900 R08: 0000000000000008 R09: 0000000000000000
[Mon Mar 11 14:29:15 2024] R10: 00007f6620ff8900 R11: 0000000000000293 R12: 00007ffd9217b000
[Mon Mar 11 14:29:15 2024] R13: 0000000000000000 R14: 000055b9f0d66b20 R15: 0000000000000000
[Mon Mar 11 14:29:15 2024] </TASK>
[Mon Mar 11 14:29:15 2024] task:automount state:S stack:0 pid:88157 ppid:1 flags:0x00000002
[Mon Mar 11 14:29:15 2024] Call Trace:
[Mon Mar 11 14:29:15 2024] <TASK>
[Mon Mar 11 14:29:15 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:15 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:15 2024] schedule_hrtimeout_range_clock+0x10f/0x120
[Mon Mar 11 14:29:15 2024] ? add_wait_queue+0x65/0xa0
[Mon Mar 11 14:29:15 2024] do_poll.constprop.0+0x248/0x390
[Mon Mar 11 14:29:15 2024] do_sys_poll+0x1c8/0x250
[Mon Mar 11 14:29:15 2024] ? sched_clock+0xc/0x30
[Mon Mar 11 14:29:15 2024] ? __smp_call_single_queue+0x93/0x120
[Mon Mar 11 14:29:15 2024] ? ttwu_queue_wakelist+0xf2/0x110
[Mon Mar 11 14:29:15 2024] ? try_to_wake_up+0x3e2/0x5d0
[Mon Mar 11 14:29:15 2024] ? __alloc_skb+0x8e/0x1d0
[Mon Mar 11 14:29:15 2024] ? autoremove_wake_function+0x30/0x60
[Mon Mar 11 14:29:15 2024] ? page_counter_uncharge+0x35/0x80
[Mon Mar 11 14:29:15 2024] ? __pfx_pollwake+0x10/0x10
[Mon Mar 11 14:29:15 2024] ? put_prev_task_fair+0x1e/0x40
[Mon Mar 11 14:29:15 2024] ? pick_next_task+0x861/0x950
[Mon Mar 11 14:29:15 2024] ? __update_idle_core+0x23/0xc0
[Mon Mar 11 14:29:15 2024] ? __switch_to_asm+0x3a/0x80
[Mon Mar 11 14:29:15 2024] ? finish_task_switch.isra.0+0x8c/0x2a0
[Mon Mar 11 14:29:15 2024] ? __schedule+0x223/0x550
[Mon Mar 11 14:29:15 2024] ? timerqueue_del+0x2a/0x50
[Mon Mar 11 14:29:15 2024] ? __remove_hrtimer+0x39/0x90
[Mon Mar 11 14:29:15 2024] ? hrtimer_try_to_cancel.part.0+0x50/0xf0
[Mon Mar 11 14:29:15 2024] ? hrtimer_cancel+0x1d/0x40
[Mon Mar 11 14:29:15 2024] ? futex_wait+0x23e/0x260
[Mon Mar 11 14:29:15 2024] ? futex_wake+0x7c/0x180
[Mon Mar 11 14:29:15 2024] ? __audit_filter_op+0xa5/0xf0
[Mon Mar 11 14:29:15 2024] __x64_sys_ppoll+0xbc/0x150
[Mon Mar 11 14:29:15 2024] do_syscall_64+0x59/0x90
[Mon Mar 11 14:29:15 2024] ? syscall_exit_to_user_mode+0x22/0x40
[Mon Mar 11 14:29:15 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:15 2024] ? syscall_exit_work+0x103/0x130
[Mon Mar 11 14:29:15 2024] ? syscall_exit_to_user_mode+0x22/0x40
[Mon Mar 11 14:29:15 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:15 2024] ? exc_page_fault+0x62/0x150
[Mon Mar 11 14:29:15 2024] entry_SYSCALL_64_after_hwframe+0x72/0xdc
[Mon Mar 11 14:29:15 2024] RIP: 0033:0x7f66af3427fe
[Mon Mar 11 14:29:15 2024] RSP: 002b:00007f6603ffe8a0 EFLAGS: 00000293 ORIG_RAX: 000000000000010f
[Mon Mar 11 14:29:15 2024] RAX: ffffffffffffffda RBX: 00007f6603fff528 RCX: 00007f66af3427fe
[Mon Mar 11 14:29:15 2024] RDX: 0000000000000000 RSI: 0000000000000001 RDI: 00007f6603ffe8f8
[Mon Mar 11 14:29:15 2024] RBP: 00007f6603ffe900 R08: 0000000000000008 R09: 0000000000000000
[Mon Mar 11 14:29:15 2024] R10: 00007f6603ffe900 R11: 0000000000000293 R12: 00007ffd9217b000
[Mon Mar 11 14:29:15 2024] R13: 0000000000000000 R14: 000055b9f0d6ceb0 R15: 0000000000000000
[Mon Mar 11 14:29:15 2024] </TASK>
[Mon Mar 11 14:29:15 2024] task:automount state:S stack:0 pid:88158 ppid:1 flags:0x00000002
[Mon Mar 11 14:29:15 2024] Call Trace:
[Mon Mar 11 14:29:15 2024] <TASK>
[Mon Mar 11 14:29:15 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:15 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:15 2024] schedule_hrtimeout_range_clock+0x10f/0x120
[Mon Mar 11 14:29:15 2024] ? add_wait_queue+0x65/0xa0
[Mon Mar 11 14:29:15 2024] do_poll.constprop.0+0x248/0x390
[Mon Mar 11 14:29:15 2024] do_sys_poll+0x1c8/0x250
[Mon Mar 11 14:29:15 2024] ? try_to_wake_up+0x3e2/0x5d0
[Mon Mar 11 14:29:15 2024] ? __alloc_skb+0x8e/0x1d0
[Mon Mar 11 14:29:15 2024] ? autoremove_wake_function+0x30/0x60
[Mon Mar 11 14:29:15 2024] ? __wake_up_common+0x75/0xa0
[Mon Mar 11 14:29:15 2024] ? __wake_up+0x40/0x60
[Mon Mar 11 14:29:15 2024] ? __pfx_pollwake+0x10/0x10
[Mon Mar 11 14:29:15 2024] ? __wake_up_common+0x75/0xa0
[Mon Mar 11 14:29:15 2024] ? wake_affine+0x62/0x1f0
[Mon Mar 11 14:29:15 2024] ? sched_clock+0xc/0x30
[Mon Mar 11 14:29:15 2024] ? __smp_call_single_queue+0x93/0x120
[Mon Mar 11 14:29:15 2024] ? ttwu_queue_wakelist+0xf2/0x110
[Mon Mar 11 14:29:15 2024] ? try_to_wake_up+0x3e2/0x5d0
[Mon Mar 11 14:29:15 2024] ? __audit_filter_op+0xa5/0xf0
[Mon Mar 11 14:29:15 2024] ? audit_reset_context.part.0.constprop.0+0x273/0x2e0
[Mon Mar 11 14:29:15 2024] ? syscall_exit_work+0x103/0x130
[Mon Mar 11 14:29:15 2024] ? syscall_exit_to_user_mode+0x22/0x40
[Mon Mar 11 14:29:15 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:15 2024] ? _copy_from_user+0x27/0x60
[Mon Mar 11 14:29:15 2024] __x64_sys_ppoll+0xbc/0x150
[Mon Mar 11 14:29:15 2024] do_syscall_64+0x59/0x90
[Mon Mar 11 14:29:15 2024] ? exc_page_fault+0x62/0x150
[Mon Mar 11 14:29:15 2024] entry_SYSCALL_64_after_hwframe+0x72/0xdc
[Mon Mar 11 14:29:15 2024] RIP: 0033:0x7f66af3427fe
[Mon Mar 11 14:29:15 2024] RSP: 002b:00007f66037fd8a0 EFLAGS: 00000293 ORIG_RAX: 000000000000010f
[Mon Mar 11 14:29:15 2024] RAX: ffffffffffffffda RBX: 00007f66037fe528 RCX: 00007f66af3427fe
[Mon Mar 11 14:29:15 2024] RDX: 0000000000000000 RSI: 0000000000000001 RDI: 00007f66037fd8f8
[Mon Mar 11 14:29:15 2024] RBP: 00007f66037fd900 R08: 0000000000000008 R09: 0000000000000000
[Mon Mar 11 14:29:15 2024] R10: 00007f66037fd900 R11: 0000000000000293 R12: 00007ffd9217b000
[Mon Mar 11 14:29:15 2024] R13: 0000000000000000 R14: 000055b9f0d73240 R15: 0000000000000000
[Mon Mar 11 14:29:15 2024] </TASK>
[Mon Mar 11 14:29:15 2024] task:automount state:S stack:0 pid:88159 ppid:1 flags:0x00000002
[Mon Mar 11 14:29:15 2024] Call Trace:
[Mon Mar 11 14:29:15 2024] <TASK>
[Mon Mar 11 14:29:15 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:15 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:15 2024] schedule_hrtimeout_range_clock+0x10f/0x120
[Mon Mar 11 14:29:15 2024] ? add_wait_queue+0x65/0xa0
[Mon Mar 11 14:29:15 2024] do_poll.constprop.0+0x248/0x390
[Mon Mar 11 14:29:15 2024] do_sys_poll+0x1c8/0x250
[Mon Mar 11 14:29:15 2024] ? mod_objcg_state+0x1fe/0x300
[Mon Mar 11 14:29:15 2024] ? wake_affine+0x62/0x1f0
[Mon Mar 11 14:29:15 2024] ? sched_clock+0xc/0x30
[Mon Mar 11 14:29:15 2024] ? __smp_call_single_queue+0x93/0x120
[Mon Mar 11 14:29:15 2024] ? ttwu_queue_wakelist+0xf2/0x110
[Mon Mar 11 14:29:15 2024] ? try_to_wake_up+0x3e2/0x5d0
[Mon Mar 11 14:29:15 2024] ? __alloc_skb+0x8e/0x1d0
[Mon Mar 11 14:29:15 2024] ? __pfx_pollwake+0x10/0x10
[Mon Mar 11 14:29:15 2024] ? wake_affine+0x62/0x1f0
[Mon Mar 11 14:29:15 2024] ? sched_clock+0xc/0x30
[Mon Mar 11 14:29:15 2024] ? __smp_call_single_queue+0x93/0x120
[Mon Mar 11 14:29:15 2024] ? select_idle_sibling+0x28/0x430
[Mon Mar 11 14:29:15 2024] ? ttwu_queue_wakelist+0xf2/0x110
[Mon Mar 11 14:29:15 2024] ? wake_affine+0x62/0x1f0
[Mon Mar 11 14:29:15 2024] ? sched_clock+0xc/0x30
[Mon Mar 11 14:29:15 2024] ? __smp_call_single_queue+0x93/0x120
[Mon Mar 11 14:29:15 2024] ? ttwu_queue_wakelist+0xf2/0x110
[Mon Mar 11 14:29:15 2024] ? __audit_filter_op+0xa5/0xf0
[Mon Mar 11 14:29:15 2024] ? audit_reset_context.part.0.constprop.0+0x273/0x2e0
[Mon Mar 11 14:29:15 2024] ? syscall_exit_work+0x103/0x130
[Mon Mar 11 14:29:15 2024] ? syscall_exit_to_user_mode+0x22/0x40
[Mon Mar 11 14:29:15 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:15 2024] ? audit_reset_context.part.0.constprop.0+0x273/0x2e0
[Mon Mar 11 14:29:15 2024] ? _copy_from_user+0x27/0x60
[Mon Mar 11 14:29:15 2024] __x64_sys_ppoll+0xbc/0x150
[Mon Mar 11 14:29:15 2024] do_syscall_64+0x59/0x90
[Mon Mar 11 14:29:15 2024] ? syscall_exit_to_user_mode+0x22/0x40
[Mon Mar 11 14:29:15 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:15 2024] ? exc_page_fault+0x62/0x150
[Mon Mar 11 14:29:15 2024] entry_SYSCALL_64_after_hwframe+0x72/0xdc
[Mon Mar 11 14:29:15 2024] RIP: 0033:0x7f66af3427fe
[Mon Mar 11 14:29:15 2024] RSP: 002b:00007f6602ffc8a0 EFLAGS: 00000293 ORIG_RAX: 000000000000010f
[Mon Mar 11 14:29:15 2024] RAX: ffffffffffffffda RBX: 00007f6602ffd528 RCX: 00007f66af3427fe
[Mon Mar 11 14:29:15 2024] RDX: 0000000000000000 RSI: 0000000000000001 RDI: 00007f6602ffc8f8
[Mon Mar 11 14:29:15 2024] RBP: 00007f6602ffc900 R08: 0000000000000008 R09: 0000000000000000
[Mon Mar 11 14:29:15 2024] R10: 00007f6602ffc900 R11: 0000000000000293 R12: 00007ffd9217b000
[Mon Mar 11 14:29:15 2024] R13: 0000000000000000 R14: 000055b9f0d795d0 R15: 0000000000000000
[Mon Mar 11 14:29:15 2024] </TASK>
[Mon Mar 11 14:29:15 2024] task:automount state:S stack:0 pid:88160 ppid:1 flags:0x00000002
[Mon Mar 11 14:29:15 2024] Call Trace:
[Mon Mar 11 14:29:15 2024] <TASK>
[Mon Mar 11 14:29:15 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:15 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:15 2024] schedule_hrtimeout_range_clock+0x10f/0x120
[Mon Mar 11 14:29:15 2024] ? xas_alloc+0x4b/0xd0
[Mon Mar 11 14:29:15 2024] ? add_wait_queue+0x65/0xa0
[Mon Mar 11 14:29:15 2024] do_poll.constprop.0+0x248/0x390
[Mon Mar 11 14:29:15 2024] do_sys_poll+0x1c8/0x250
[Mon Mar 11 14:29:15 2024] ? wake_affine+0x62/0x1f0
[Mon Mar 11 14:29:15 2024] ? __smp_call_single_queue+0x93/0x120
[Mon Mar 11 14:29:15 2024] ? ttwu_queue_wakelist+0xf2/0x110
[Mon Mar 11 14:29:15 2024] ? try_to_wake_up+0x3e2/0x5d0
[Mon Mar 11 14:29:15 2024] ? __alloc_skb+0x8e/0x1d0
[Mon Mar 11 14:29:15 2024] ? autoremove_wake_function+0x30/0x60
[Mon Mar 11 14:29:15 2024] ? __wake_up_common+0x75/0xa0
[Mon Mar 11 14:29:15 2024] ? __pfx_pollwake+0x10/0x10
[Mon Mar 11 14:29:15 2024] ? select_idle_sibling+0x28/0x430
[Mon Mar 11 14:29:15 2024] ? __wake_up_common+0x75/0xa0
[Mon Mar 11 14:29:15 2024] ? wake_affine+0x62/0x1f0
[Mon Mar 11 14:29:15 2024] ? select_idle_sibling+0x28/0x430
[Mon Mar 11 14:29:15 2024] ? wake_affine+0x62/0x1f0
[Mon Mar 11 14:29:15 2024] ? sched_clock+0xc/0x30
[Mon Mar 11 14:29:15 2024] ? sched_clock+0xc/0x30
[Mon Mar 11 14:29:15 2024] ? __smp_call_single_queue+0x93/0x120
[Mon Mar 11 14:29:15 2024] ? ttwu_queue_wakelist+0xf2/0x110
[Mon Mar 11 14:29:15 2024] ? __audit_filter_op+0xa5/0xf0
[Mon Mar 11 14:29:15 2024] ? audit_reset_context.part.0.constprop.0+0x273/0x2e0
[Mon Mar 11 14:29:15 2024] ? syscall_exit_work+0x103/0x130
[Mon Mar 11 14:29:15 2024] ? syscall_exit_to_user_mode+0x22/0x40
[Mon Mar 11 14:29:15 2024] __x64_sys_ppoll+0xbc/0x150
[Mon Mar 11 14:29:15 2024] do_syscall_64+0x59/0x90
[Mon Mar 11 14:29:15 2024] ? exc_page_fault+0x62/0x150
[Mon Mar 11 14:29:15 2024] entry_SYSCALL_64_after_hwframe+0x72/0xdc
[Mon Mar 11 14:29:15 2024] RIP: 0033:0x7f66af3427fe
[Mon Mar 11 14:29:15 2024] RSP: 002b:00007f66027fb8a0 EFLAGS: 00000293 ORIG_RAX: 000000000000010f
[Mon Mar 11 14:29:15 2024] RAX: ffffffffffffffda RBX: 00007f66027fc528 RCX: 00007f66af3427fe
[Mon Mar 11 14:29:15 2024] RDX: 0000000000000000 RSI: 0000000000000001 RDI: 00007f66027fb8f8
[Mon Mar 11 14:29:15 2024] RBP: 00007f66027fb900 R08: 0000000000000008 R09: 0000000000000000
[Mon Mar 11 14:29:15 2024] R10: 00007f66027fb900 R11: 0000000000000293 R12: 00007ffd9217b000
[Mon Mar 11 14:29:15 2024] R13: 0000000000000000 R14: 000055b9f0d7f980 R15: 0000000000000000
[Mon Mar 11 14:29:15 2024] </TASK>
[Mon Mar 11 14:29:15 2024] task:automount state:S stack:0 pid:88161 ppid:1 flags:0x00000002
[Mon Mar 11 14:29:15 2024] Call Trace:
[Mon Mar 11 14:29:15 2024] <TASK>
[Mon Mar 11 14:29:15 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:15 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:15 2024] schedule_hrtimeout_range_clock+0x10f/0x120
[Mon Mar 11 14:29:15 2024] ? asm_exc_page_fault+0x22/0x30
[Mon Mar 11 14:29:15 2024] ? add_wait_queue+0x65/0xa0
[Mon Mar 11 14:29:15 2024] do_poll.constprop.0+0x248/0x390
[Mon Mar 11 14:29:15 2024] do_sys_poll+0x1c8/0x250
[Mon Mar 11 14:29:15 2024] ? kmem_cache_alloc+0x17d/0x340
[Mon Mar 11 14:29:15 2024] ? update_sg_wakeup_stats+0x78/0x3b0
[Mon Mar 11 14:29:15 2024] ? newidle_balance+0x2e5/0x400
[Mon Mar 11 14:29:15 2024] ? update_load_avg+0x7e/0x740
[Mon Mar 11 14:29:15 2024] ? __pfx_pollwake+0x10/0x10
[Mon Mar 11 14:29:15 2024] ? pick_next_task+0x861/0x950
[Mon Mar 11 14:29:15 2024] ? __update_idle_core+0x23/0xc0
[Mon Mar 11 14:29:15 2024] ? __switch_to_asm+0x3a/0x80
[Mon Mar 11 14:29:15 2024] ? finish_task_switch.isra.0+0x8c/0x2a0
[Mon Mar 11 14:29:15 2024] ? __schedule+0x223/0x550
[Mon Mar 11 14:29:15 2024] ? timerqueue_del+0x2a/0x50
[Mon Mar 11 14:29:15 2024] ? __remove_hrtimer+0x39/0x90
[Mon Mar 11 14:29:15 2024] ? hrtimer_try_to_cancel.part.0+0x50/0xf0
[Mon Mar 11 14:29:15 2024] ? hrtimer_cancel+0x1d/0x40
[Mon Mar 11 14:29:15 2024] ? futex_wait+0x23e/0x260
[Mon Mar 11 14:29:15 2024] ? __pfx_hrtimer_wakeup+0x10/0x10
[Mon Mar 11 14:29:15 2024] ? rseq_get_rseq_cs+0x1d/0x240
[Mon Mar 11 14:29:15 2024] ? rseq_ip_fixup+0x6e/0x1a0
[Mon Mar 11 14:29:15 2024] ? __audit_filter_op+0xa5/0xf0
[Mon Mar 11 14:29:15 2024] __x64_sys_ppoll+0xbc/0x150
[Mon Mar 11 14:29:15 2024] do_syscall_64+0x59/0x90
[Mon Mar 11 14:29:15 2024] ? audit_reset_context.part.0.constprop.0+0x273/0x2e0
[Mon Mar 11 14:29:15 2024] ? syscall_exit_work+0x103/0x130
[Mon Mar 11 14:29:15 2024] ? syscall_exit_to_user_mode+0x22/0x40
[Mon Mar 11 14:29:15 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:15 2024] ? syscall_exit_to_user_mode+0x22/0x40
[Mon Mar 11 14:29:15 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:15 2024] entry_SYSCALL_64_after_hwframe+0x72/0xdc
[Mon Mar 11 14:29:15 2024] RIP: 0033:0x7f66af3427fe
[Mon Mar 11 14:29:15 2024] RSP: 002b:00007f6601ffa8a0 EFLAGS: 00000293 ORIG_RAX: 000000000000010f
[Mon Mar 11 14:29:15 2024] RAX: ffffffffffffffda RBX: 00007f6601ffb528 RCX: 00007f66af3427fe
[Mon Mar 11 14:29:15 2024] RDX: 0000000000000000 RSI: 0000000000000001 RDI: 00007f6601ffa8f8
[Mon Mar 11 14:29:15 2024] RBP: 00007f6601ffa900 R08: 0000000000000008 R09: 0000000000000000
[Mon Mar 11 14:29:15 2024] R10: 00007f6601ffa900 R11: 0000000000000293 R12: 00007ffd9217b000
[Mon Mar 11 14:29:15 2024] R13: 0000000000000000 R14: 000055b9f0d85cf0 R15: 0000000000000000
[Mon Mar 11 14:29:15 2024] </TASK>
[Mon Mar 11 14:29:15 2024] task:automount state:S stack:0 pid:88162 ppid:1 flags:0x00000002
[Mon Mar 11 14:29:15 2024] Call Trace:
[Mon Mar 11 14:29:15 2024] <TASK>
[Mon Mar 11 14:29:15 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:15 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:15 2024] schedule_hrtimeout_range_clock+0x10f/0x120
[Mon Mar 11 14:29:15 2024] ? __find_get_block+0x1fb/0x370
[Mon Mar 11 14:29:15 2024] ? add_wait_queue+0x65/0xa0
[Mon Mar 11 14:29:15 2024] do_poll.constprop.0+0x248/0x390
[Mon Mar 11 14:29:15 2024] do_sys_poll+0x1c8/0x250
[Mon Mar 11 14:29:15 2024] ? __mod_memcg_lruvec_state+0x84/0xd0
[Mon Mar 11 14:29:15 2024] ? mod_objcg_state+0x195/0x300
[Mon Mar 11 14:29:15 2024] ? kmem_cache_alloc_lru+0x12f/0x2b0
[Mon Mar 11 14:29:15 2024] ? select_idle_sibling+0x28/0x430
[Mon Mar 11 14:29:15 2024] ? wake_affine+0x62/0x1f0
[Mon Mar 11 14:29:15 2024] ? sched_clock+0xc/0x30
[Mon Mar 11 14:29:15 2024] ? __smp_call_single_queue+0x93/0x120
[Mon Mar 11 14:29:15 2024] ? ttwu_queue_wakelist+0xf2/0x110
[Mon Mar 11 14:29:15 2024] ? __pfx_pollwake+0x10/0x10
[Mon Mar 11 14:29:15 2024] ? __wake_up_common+0x75/0xa0
[Mon Mar 11 14:29:15 2024] ? __wake_up+0x40/0x60
[Mon Mar 11 14:29:15 2024] ? ep_poll_callback+0x25d/0x2a0
[Mon Mar 11 14:29:15 2024] ? select_idle_sibling+0x28/0x430
[Mon Mar 11 14:29:15 2024] ? wake_affine+0x62/0x1f0
[Mon Mar 11 14:29:15 2024] ? sched_clock+0xc/0x30
[Mon Mar 11 14:29:15 2024] ? __smp_call_single_queue+0x93/0x120
[Mon Mar 11 14:29:15 2024] ? select_idle_sibling+0x28/0x430
[Mon Mar 11 14:29:15 2024] ? ttwu_queue_wakelist+0xf2/0x110
[Mon Mar 11 14:29:15 2024] ? wake_affine+0x62/0x1f0
[Mon Mar 11 14:29:15 2024] ? sched_clock+0xc/0x30
[Mon Mar 11 14:29:15 2024] ? __smp_call_single_queue+0x93/0x120
[Mon Mar 11 14:29:15 2024] ? ttwu_queue_wakelist+0xf2/0x110
[Mon Mar 11 14:29:15 2024] ? __audit_filter_op+0xa5/0xf0
[Mon Mar 11 14:29:15 2024] __x64_sys_ppoll+0xbc/0x150
[Mon Mar 11 14:29:15 2024] do_syscall_64+0x59/0x90
[Mon Mar 11 14:29:15 2024] ? audit_reset_context.part.0.constprop.0+0x273/0x2e0
[Mon Mar 11 14:29:15 2024] ? syscall_exit_work+0x103/0x130
[Mon Mar 11 14:29:15 2024] ? syscall_exit_to_user_mode+0x22/0x40
[Mon Mar 11 14:29:15 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:15 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:15 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:15 2024] ? exc_page_fault+0x62/0x150
[Mon Mar 11 14:29:15 2024] entry_SYSCALL_64_after_hwframe+0x72/0xdc
[Mon Mar 11 14:29:15 2024] RIP: 0033:0x7f66af3427fe
[Mon Mar 11 14:29:15 2024] RSP: 002b:00007f66017f98a0 EFLAGS: 00000293 ORIG_RAX: 000000000000010f
[Mon Mar 11 14:29:15 2024] RAX: ffffffffffffffda RBX: 00007f66017fa528 RCX: 00007f66af3427fe
[Mon Mar 11 14:29:15 2024] RDX: 0000000000000000 RSI: 0000000000000001 RDI: 00007f66017f98f8
[Mon Mar 11 14:29:15 2024] RBP: 00007f66017f9900 R08: 0000000000000008 R09: 0000000000000000
[Mon Mar 11 14:29:15 2024] R10: 00007f66017f9900 R11: 0000000000000293 R12: 00007ffd9217b000
[Mon Mar 11 14:29:15 2024] R13: 0000000000000000 R14: 000055b9f0d8c080 R15: 0000000000000000
[Mon Mar 11 14:29:15 2024] </TASK>
[Mon Mar 11 14:29:15 2024] task:automount state:S stack:0 pid:88163 ppid:1 flags:0x00000002
[Mon Mar 11 14:29:15 2024] Call Trace:
[Mon Mar 11 14:29:15 2024] <TASK>
[Mon Mar 11 14:29:15 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:15 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:15 2024] schedule_hrtimeout_range_clock+0x10f/0x120
[Mon Mar 11 14:29:15 2024] ? rmqueue+0x7d3/0xd40
[Mon Mar 11 14:29:15 2024] ? __ext4_handle_dirty_metadata+0x58/0x180 [ext4]
[Mon Mar 11 14:29:15 2024] ? add_wait_queue+0x65/0xa0
[Mon Mar 11 14:29:15 2024] do_poll.constprop.0+0x248/0x390
[Mon Mar 11 14:29:15 2024] do_sys_poll+0x1c8/0x250
[Mon Mar 11 14:29:15 2024] ? ttwu_queue_wakelist+0xf2/0x110
[Mon Mar 11 14:29:15 2024] ? try_to_wake_up+0x3e2/0x5d0
[Mon Mar 11 14:29:15 2024] ? __alloc_skb+0x8e/0x1d0
[Mon Mar 11 14:29:15 2024] ? autoremove_wake_function+0x30/0x60
[Mon Mar 11 14:29:15 2024] ? __wake_up_common+0x75/0xa0
[Mon Mar 11 14:29:15 2024] ? __pfx_pollwake+0x10/0x10
[Mon Mar 11 14:29:15 2024] ? __wake_up_common+0x75/0xa0
[Mon Mar 11 14:29:15 2024] ? select_idle_sibling+0x28/0x430
[Mon Mar 11 14:29:15 2024] ? wake_affine+0x62/0x1f0
[Mon Mar 11 14:29:15 2024] ? sched_clock+0xc/0x30
[Mon Mar 11 14:29:15 2024] ? __smp_call_single_queue+0x93/0x120
[Mon Mar 11 14:29:15 2024] ? ttwu_queue_wakelist+0xf2/0x110
[Mon Mar 11 14:29:15 2024] ? try_to_wake_up+0x3e2/0x5d0
[Mon Mar 11 14:29:15 2024] ? select_idle_sibling+0x28/0x430
[Mon Mar 11 14:29:15 2024] ? wake_affine+0x62/0x1f0
[Mon Mar 11 14:29:15 2024] ? sched_clock+0xc/0x30
[Mon Mar 11 14:29:15 2024] ? __smp_call_single_queue+0x93/0x120
[Mon Mar 11 14:29:15 2024] ? ttwu_queue_wakelist+0xf2/0x110
[Mon Mar 11 14:29:15 2024] ? try_to_wake_up+0x3e2/0x5d0
[Mon Mar 11 14:29:15 2024] __x64_sys_ppoll+0xbc/0x150
[Mon Mar 11 14:29:15 2024] do_syscall_64+0x59/0x90
[Mon Mar 11 14:29:15 2024] ? audit_reset_context.part.0.constprop.0+0x273/0x2e0
[Mon Mar 11 14:29:15 2024] ? syscall_exit_work+0x103/0x130
[Mon Mar 11 14:29:15 2024] ? syscall_exit_to_user_mode+0x22/0x40
[Mon Mar 11 14:29:15 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:15 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:15 2024] ? exc_page_fault+0x62/0x150
[Mon Mar 11 14:29:15 2024] entry_SYSCALL_64_after_hwframe+0x72/0xdc
[Mon Mar 11 14:29:15 2024] RIP: 0033:0x7f66af3427fe
[Mon Mar 11 14:29:15 2024] RSP: 002b:00007f6600ff88a0 EFLAGS: 00000293 ORIG_RAX: 000000000000010f
[Mon Mar 11 14:29:15 2024] RAX: ffffffffffffffda RBX: 00007f6600ff9528 RCX: 00007f66af3427fe
[Mon Mar 11 14:29:15 2024] RDX: 0000000000000000 RSI: 0000000000000001 RDI: 00007f6600ff88f8
[Mon Mar 11 14:29:15 2024] RBP: 00007f6600ff8900 R08: 0000000000000008 R09: 0000000000000000
[Mon Mar 11 14:29:15 2024] R10: 00007f6600ff8900 R11: 0000000000000293 R12: 00007ffd9217b000
[Mon Mar 11 14:29:15 2024] R13: 0000000000000000 R14: 000055b9f0d92410 R15: 0000000000000000
[Mon Mar 11 14:29:15 2024] </TASK>
[Mon Mar 11 14:29:15 2024] task:automount state:S stack:0 pid:88164 ppid:1 flags:0x00000002
[Mon Mar 11 14:29:15 2024] Call Trace:
[Mon Mar 11 14:29:15 2024] <TASK>
[Mon Mar 11 14:29:15 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:15 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:15 2024] schedule_hrtimeout_range_clock+0x10f/0x120
[Mon Mar 11 14:29:15 2024] ? add_wait_queue+0x65/0xa0
[Mon Mar 11 14:29:15 2024] do_poll.constprop.0+0x248/0x390
[Mon Mar 11 14:29:15 2024] do_sys_poll+0x1c8/0x250
[Mon Mar 11 14:29:15 2024] ? __smp_call_single_queue+0x93/0x120
[Mon Mar 11 14:29:15 2024] ? try_to_wake_up+0x3e2/0x5d0
[Mon Mar 11 14:29:15 2024] ? __alloc_skb+0x8e/0x1d0
[Mon Mar 11 14:29:15 2024] ? autoremove_wake_function+0x30/0x60
[Mon Mar 11 14:29:15 2024] ? __wake_up_common+0x75/0xa0
[Mon Mar 11 14:29:15 2024] ? __wake_up+0x40/0x60
[Mon Mar 11 14:29:15 2024] ? __pfx_pollwake+0x10/0x10
[Mon Mar 11 14:29:15 2024] ? __wake_up_sync_key+0x39/0x50
[Mon Mar 11 14:29:15 2024] ? select_idle_sibling+0x28/0x430
[Mon Mar 11 14:29:15 2024] ? sock_def_readable+0x3e/0xc0
[Mon Mar 11 14:29:15 2024] ? wake_affine+0x62/0x1f0
[Mon Mar 11 14:29:15 2024] ? unix_dgram_sendmsg+0x5d3/0x9c0
[Mon Mar 11 14:29:15 2024] ? sched_clock+0xc/0x30
[Mon Mar 11 14:29:15 2024] ? __smp_call_single_queue+0x93/0x120
[Mon Mar 11 14:29:15 2024] ? ttwu_queue_wakelist+0xf2/0x110
[Mon Mar 11 14:29:15 2024] ? select_idle_sibling+0x28/0x430
[Mon Mar 11 14:29:15 2024] ? try_to_wake_up+0x3e2/0x5d0
[Mon Mar 11 14:29:15 2024] ? wake_affine+0x62/0x1f0
[Mon Mar 11 14:29:15 2024] ? sched_clock+0xc/0x30
[Mon Mar 11 14:29:15 2024] ? __smp_call_single_queue+0x93/0x120
[Mon Mar 11 14:29:15 2024] ? ttwu_queue_wakelist+0xf2/0x110
[Mon Mar 11 14:29:15 2024] ? try_to_wake_up+0x3e2/0x5d0
[Mon Mar 11 14:29:15 2024] __x64_sys_ppoll+0xbc/0x150
[Mon Mar 11 14:29:15 2024] do_syscall_64+0x59/0x90
[Mon Mar 11 14:29:15 2024] ? syscall_exit_work+0x103/0x130
[Mon Mar 11 14:29:15 2024] ? syscall_exit_to_user_mode+0x22/0x40
[Mon Mar 11 14:29:15 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:15 2024] ? exc_page_fault+0x62/0x150
[Mon Mar 11 14:29:15 2024] entry_SYSCALL_64_after_hwframe+0x72/0xdc
[Mon Mar 11 14:29:15 2024] RIP: 0033:0x7f66af3427fe
[Mon Mar 11 14:29:15 2024] RSP: 002b:00007f65e3ffe8a0 EFLAGS: 00000293 ORIG_RAX: 000000000000010f
[Mon Mar 11 14:29:15 2024] RAX: ffffffffffffffda RBX: 00007f65e3fff528 RCX: 00007f66af3427fe
[Mon Mar 11 14:29:15 2024] RDX: 0000000000000000 RSI: 0000000000000001 RDI: 00007f65e3ffe8f8
[Mon Mar 11 14:29:15 2024] RBP: 00007f65e3ffe900 R08: 0000000000000008 R09: 0000000000000000
[Mon Mar 11 14:29:15 2024] R10: 00007f65e3ffe900 R11: 0000000000000293 R12: 00007ffd9217b000
[Mon Mar 11 14:29:15 2024] R13: 0000000000000000 R14: 000055b9f0d987a0 R15: 0000000000000000
[Mon Mar 11 14:29:15 2024] </TASK>
[Mon Mar 11 14:29:15 2024] task:atd state:S stack:0 pid:88110 ppid:1 flags:0x00000002
[Mon Mar 11 14:29:15 2024] Call Trace:
[Mon Mar 11 14:29:15 2024] <TASK>
[Mon Mar 11 14:29:15 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:15 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:15 2024] __do_sys_pause+0x30/0x60
[Mon Mar 11 14:29:15 2024] do_syscall_64+0x59/0x90
[Mon Mar 11 14:29:15 2024] ? audit_reset_context.part.0.constprop.0+0x273/0x2e0
[Mon Mar 11 14:29:15 2024] ? syscall_exit_work+0x103/0x130
[Mon Mar 11 14:29:15 2024] ? syscall_exit_to_user_mode+0x22/0x40
[Mon Mar 11 14:29:15 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:15 2024] ? syscall_exit_to_user_mode+0x22/0x40
[Mon Mar 11 14:29:15 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:15 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:15 2024] ? sysvec_apic_timer_interrupt+0x3c/0x90
[Mon Mar 11 14:29:15 2024] entry_SYSCALL_64_after_hwframe+0x72/0xdc
[Mon Mar 11 14:29:15 2024] RIP: 0033:0x7f17b21184a7
[Mon Mar 11 14:29:15 2024] RSP: 002b:00007ffeef973918 EFLAGS: 00000246 ORIG_RAX: 0000000000000022
[Mon Mar 11 14:29:15 2024] RAX: ffffffffffffffda RBX: 000055ae3b68c0a0 RCX: 00007f17b21184a7
[Mon Mar 11 14:29:15 2024] RDX: 000055ae3b68c0a0 RSI: 0000000000000001 RDI: 0000000000000000
[Mon Mar 11 14:29:15 2024] RBP: 000055ae3b68937c R08: 0000000000000000 R09: 0000000000000078
[Mon Mar 11 14:29:15 2024] R10: 0000000000000000 R11: 0000000000000246 R12: 00007ffeef973930
[Mon Mar 11 14:29:15 2024] R13: 0000000000000003 R14: 000055ae3bf1e7b0 R15: 000055ae3b68c0cc
[Mon Mar 11 14:29:15 2024] </TASK>
[Mon Mar 11 14:29:15 2024] task:agetty state:S stack:0 pid:88117 ppid:1 flags:0x00000002
[Mon Mar 11 14:29:15 2024] Call Trace:
[Mon Mar 11 14:29:15 2024] <TASK>
[Mon Mar 11 14:29:15 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:15 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:15 2024] schedule_timeout+0x11f/0x160
[Mon Mar 11 14:29:15 2024] ? up+0x12/0x60
[Mon Mar 11 14:29:15 2024] ? console_unlock+0xc8/0x320
[Mon Mar 11 14:29:15 2024] ? add_wait_queue+0x65/0xa0
[Mon Mar 11 14:29:15 2024] wait_woken+0x50/0x60
[Mon Mar 11 14:29:15 2024] n_tty_read+0x512/0x660
[Mon Mar 11 14:29:15 2024] ? __pfx_woken_wake_function+0x10/0x10
[Mon Mar 11 14:29:15 2024] tty_read+0x12c/0x220
[Mon Mar 11 14:29:15 2024] ? file_has_perm+0xc6/0xd0
[Mon Mar 11 14:29:15 2024] vfs_read+0x1e6/0x330
[Mon Mar 11 14:29:15 2024] ksys_read+0x5f/0xe0
[Mon Mar 11 14:29:15 2024] do_syscall_64+0x59/0x90
[Mon Mar 11 14:29:15 2024] ? syscall_exit_work+0x103/0x130
[Mon Mar 11 14:29:15 2024] ? syscall_exit_to_user_mode+0x22/0x40
[Mon Mar 11 14:29:15 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:15 2024] entry_SYSCALL_64_after_hwframe+0x72/0xdc
[Mon Mar 11 14:29:15 2024] RIP: 0033:0x7fb8efd3e882
[Mon Mar 11 14:29:15 2024] RSP: 002b:00007ffec6777f08 EFLAGS: 00000246 ORIG_RAX: 0000000000000000
[Mon Mar 11 14:29:15 2024] RAX: ffffffffffffffda RBX: 00007fb8eff5f6c0 RCX: 00007fb8efd3e882
[Mon Mar 11 14:29:15 2024] RDX: 0000000000000001 RSI: 00007ffec6777f7f RDI: 0000000000000000
[Mon Mar 11 14:29:15 2024] RBP: 000000000000006c R08: 0000000000000000 R09: 0000000000000000
[Mon Mar 11 14:29:15 2024] R10: 0000000000000000 R11: 0000000000000246 R12: 00007ffec6777f7f
[Mon Mar 11 14:29:15 2024] R13: 00007ffec6777fb0 R14: 0000000000000015 R15: 0000000000000000
[Mon Mar 11 14:29:15 2024] </TASK>
[Mon Mar 11 14:29:15 2024] task:crond state:S stack:0 pid:88166 ppid:1 flags:0x00000002
[Mon Mar 11 14:29:15 2024] Call Trace:
[Mon Mar 11 14:29:15 2024] <TASK>
[Mon Mar 11 14:29:15 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:15 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:15 2024] do_nanosleep+0x67/0x190
[Mon Mar 11 14:29:15 2024] hrtimer_nanosleep+0xbe/0x1a0
[Mon Mar 11 14:29:15 2024] ? __pfx_hrtimer_wakeup+0x10/0x10
[Mon Mar 11 14:29:15 2024] common_nsleep+0x40/0x50
[Mon Mar 11 14:29:15 2024] __x64_sys_clock_nanosleep+0xbc/0x130
[Mon Mar 11 14:29:15 2024] do_syscall_64+0x59/0x90
[Mon Mar 11 14:29:15 2024] ? audit_reset_context.part.0.constprop.0+0x273/0x2e0
[Mon Mar 11 14:29:15 2024] ? syscall_exit_work+0x103/0x130
[Mon Mar 11 14:29:15 2024] ? syscall_exit_to_user_mode+0x22/0x40
[Mon Mar 11 14:29:15 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:15 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:15 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:15 2024] ? syscall_exit_work+0x103/0x130
[Mon Mar 11 14:29:15 2024] ? syscall_exit_to_user_mode+0x22/0x40
[Mon Mar 11 14:29:15 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:15 2024] entry_SYSCALL_64_after_hwframe+0x72/0xdc
[Mon Mar 11 14:29:15 2024] RIP: 0033:0x7f546971393a
[Mon Mar 11 14:29:15 2024] RSP: 002b:00007ffcf602bda8 EFLAGS: 00000246 ORIG_RAX: 00000000000000e6
[Mon Mar 11 14:29:15 2024] RAX: ffffffffffffffda RBX: fffffffffffffe98 RCX: 00007f546971393a
[Mon Mar 11 14:29:15 2024] RDX: 00007ffcf602bdc0 RSI: 0000000000000000 RDI: 0000000000000000
[Mon Mar 11 14:29:15 2024] RBP: 000000000000000a R08: 0000000000000000 R09: 0000000065ef1568
[Mon Mar 11 14:29:15 2024] R10: 00007ffcf602bdc0 R11: 0000000000000246 R12: 000000000000003c
[Mon Mar 11 14:29:15 2024] R13: 0000000001b2eb06 R14: 000000000000003b R15: 000055e5b1d47a58
[Mon Mar 11 14:29:15 2024] </TASK>
[Mon Mar 11 14:29:15 2024] task:tmux: server state:S stack:0 pid:91743 ppid:1 flags:0x00000002
[Mon Mar 11 14:29:15 2024] Call Trace:
[Mon Mar 11 14:29:15 2024] <TASK>
[Mon Mar 11 14:29:15 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:15 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:15 2024] schedule_hrtimeout_range_clock+0x95/0x120
[Mon Mar 11 14:29:15 2024] ? __pfx_hrtimer_wakeup+0x10/0x10
[Mon Mar 11 14:29:15 2024] do_poll.constprop.0+0x248/0x390
[Mon Mar 11 14:29:15 2024] do_sys_poll+0x1c8/0x250
[Mon Mar 11 14:29:15 2024] ? zap_pmd_range.isra.0+0x10f/0x230
[Mon Mar 11 14:29:15 2024] ? zap_pte_range+0x3ec/0xac0
[Mon Mar 11 14:29:15 2024] ? unmap_page_range+0x2b9/0x4c0
[Mon Mar 11 14:29:15 2024] ? __pfx_pollwake+0x10/0x10
[Mon Mar 11 14:29:15 2024] ? __pfx_pollwake+0x10/0x10
[Mon Mar 11 14:29:15 2024] ? __pfx_pollwake+0x10/0x10
[Mon Mar 11 14:29:15 2024] ? __pfx_pollwake+0x10/0x10
[Mon Mar 11 14:29:15 2024] ? __blk_flush_plug+0xf1/0x150
[Mon Mar 11 14:29:15 2024] ? __audit_filter_op+0xa5/0xf0
[Mon Mar 11 14:29:15 2024] ? audit_reset_context.part.0.constprop.0+0x273/0x2e0
[Mon Mar 11 14:29:15 2024] ? syscall_exit_work+0x103/0x130
[Mon Mar 11 14:29:15 2024] ? syscall_exit_to_user_mode+0x22/0x40
[Mon Mar 11 14:29:15 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:15 2024] ? __audit_filter_op+0xa5/0xf0
[Mon Mar 11 14:29:15 2024] ? ktime_get_ts64+0x49/0xf0
[Mon Mar 11 14:29:15 2024] __x64_sys_poll+0xa6/0x140
[Mon Mar 11 14:29:15 2024] do_syscall_64+0x59/0x90
[Mon Mar 11 14:29:15 2024] ? syscall_exit_to_user_mode+0x22/0x40
[Mon Mar 11 14:29:15 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:15 2024] ? sysvec_apic_timer_interrupt+0x3c/0x90
[Mon Mar 11 14:29:15 2024] entry_SYSCALL_64_after_hwframe+0x72/0xdc
[Mon Mar 11 14:29:15 2024] RIP: 0033:0x7feaa0f426c7
[Mon Mar 11 14:29:15 2024] RSP: 002b:00007ffc39a31b08 EFLAGS: 00000246 ORIG_RAX: 0000000000000007
[Mon Mar 11 14:29:15 2024] RAX: ffffffffffffffda RBX: 00005631b1e68c20 RCX: 00007feaa0f426c7
[Mon Mar 11 14:29:15 2024] RDX: 000000000036ee80 RSI: 0000000000000003 RDI: 00005631b1e68a30
[Mon Mar 11 14:29:15 2024] RBP: 0000000000000003 R08: 00007ffc39a31b80 R09: 00007feaa1234d20
[Mon Mar 11 14:29:15 2024] R10: 00007ffc39be9080 R11: 0000000000000246 R12: 00005631b1e68070
[Mon Mar 11 14:29:15 2024] R13: 00005631b1e68a30 R14: 000000000036ee80 R15: 0000000000000003
[Mon Mar 11 14:29:15 2024] </TASK>
[Mon Mar 11 14:29:15 2024] task:bash state:S stack:0 pid:91744 ppid:91743 flags:0x00000002
[Mon Mar 11 14:29:15 2024] Call Trace:
[Mon Mar 11 14:29:15 2024] <TASK>
[Mon Mar 11 14:29:15 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:15 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:15 2024] do_wait+0x15b/0x2f0
[Mon Mar 11 14:29:15 2024] kernel_wait4+0xa6/0x140
[Mon Mar 11 14:29:15 2024] ? __pfx_child_wait_callback+0x10/0x10
[Mon Mar 11 14:29:15 2024] do_syscall_64+0x59/0x90
[Mon Mar 11 14:29:15 2024] ? __audit_filter_op+0xa5/0xf0
[Mon Mar 11 14:29:15 2024] ? audit_reset_context.part.0.constprop.0+0x273/0x2e0
[Mon Mar 11 14:29:15 2024] ? syscall_exit_work+0x103/0x130
[Mon Mar 11 14:29:15 2024] ? syscall_exit_to_user_mode+0x22/0x40
[Mon Mar 11 14:29:15 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:15 2024] ? exc_page_fault+0x62/0x150
[Mon Mar 11 14:29:15 2024] entry_SYSCALL_64_after_hwframe+0x72/0xdc
[Mon Mar 11 14:29:15 2024] RIP: 0033:0x7fafb8b182ca
[Mon Mar 11 14:29:15 2024] RSP: 002b:00007ffcaac0e548 EFLAGS: 00000246 ORIG_RAX: 000000000000003d
[Mon Mar 11 14:29:15 2024] RAX: ffffffffffffffda RBX: 0000000000016676 RCX: 00007fafb8b182ca
[Mon Mar 11 14:29:15 2024] RDX: 000000000000000a RSI: 00007ffcaac0e570 RDI: 00000000ffffffff
[Mon Mar 11 14:29:15 2024] RBP: 0000000000000001 R08: 0000000000000001 R09: 0000000000000000
[Mon Mar 11 14:29:15 2024] R10: 0000000000000000 R11: 0000000000000246 R12: 000000000000000a
[Mon Mar 11 14:29:15 2024] R13: 00007ffcaac0e5d0 R14: 0000000000000000 R15: 0000000000000000
[Mon Mar 11 14:29:15 2024] </TASK>
[Mon Mar 11 14:29:15 2024] task:nfsclients.sh state:S stack:0 pid:91766 ppid:91744 flags:0x00000002
[Mon Mar 11 14:29:15 2024] Call Trace:
[Mon Mar 11 14:29:15 2024] <TASK>
[Mon Mar 11 14:29:15 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:15 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:15 2024] do_wait+0x15b/0x2f0
[Mon Mar 11 14:29:15 2024] kernel_wait4+0xa6/0x140
[Mon Mar 11 14:29:15 2024] ? __pfx_child_wait_callback+0x10/0x10
[Mon Mar 11 14:29:15 2024] do_syscall_64+0x59/0x90
[Mon Mar 11 14:29:15 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:15 2024] ? syscall_exit_work+0x103/0x130
[Mon Mar 11 14:29:15 2024] ? syscall_exit_to_user_mode+0x22/0x40
[Mon Mar 11 14:29:15 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:15 2024] ? syscall_exit_to_user_mode+0x22/0x40
[Mon Mar 11 14:29:15 2024] ? exc_page_fault+0x62/0x150
[Mon Mar 11 14:29:15 2024] entry_SYSCALL_64_after_hwframe+0x72/0xdc
[Mon Mar 11 14:29:15 2024] RIP: 0033:0x7f7e9d9182ca
[Mon Mar 11 14:29:15 2024] RSP: 002b:00007ffc688a3c08 EFLAGS: 00000246 ORIG_RAX: 000000000000003d
[Mon Mar 11 14:29:15 2024] RAX: ffffffffffffffda RBX: 000000000025d049 RCX: 00007f7e9d9182ca
[Mon Mar 11 14:29:15 2024] RDX: 0000000000000000 RSI: 00007ffc688a3c30 RDI: 00000000ffffffff
[Mon Mar 11 14:29:15 2024] RBP: 0000000000000001 R08: 0000000000000001 R09: 0000000000000000
[Mon Mar 11 14:29:15 2024] R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000000000
[Mon Mar 11 14:29:15 2024] R13: 00007ffc688a3c90 R14: 0000000000000000 R15: 0000000000000000
[Mon Mar 11 14:29:15 2024] </TASK>
[Mon Mar 11 14:29:15 2024] task:smbd[10.86.26.7 state:S stack:0 pid:93091 ppid:4757 flags:0x00000002
[Mon Mar 11 14:29:15 2024] Call Trace:
[Mon Mar 11 14:29:15 2024] <TASK>
[Mon Mar 11 14:29:15 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:15 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:15 2024] schedule_hrtimeout_range_clock+0x95/0x120
[Mon Mar 11 14:29:15 2024] ? __pfx_hrtimer_wakeup+0x10/0x10
[Mon Mar 11 14:29:15 2024] ep_poll+0x348/0x3b0
[Mon Mar 11 14:29:15 2024] ? __pfx_autoremove_wake_function+0x10/0x10
[Mon Mar 11 14:29:15 2024] do_epoll_wait+0xb1/0xd0
[Mon Mar 11 14:29:15 2024] __x64_sys_epoll_wait+0x60/0x100
[Mon Mar 11 14:29:15 2024] do_syscall_64+0x59/0x90
[Mon Mar 11 14:29:15 2024] ? audit_reset_context.part.0.constprop.0+0x273/0x2e0
[Mon Mar 11 14:29:15 2024] ? syscall_exit_work+0x103/0x130
[Mon Mar 11 14:29:15 2024] ? syscall_exit_to_user_mode+0x22/0x40
[Mon Mar 11 14:29:15 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:15 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:15 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:15 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:15 2024] ? __irq_exit_rcu+0x46/0xc0
[Mon Mar 11 14:29:15 2024] ? common_interrupt+0x43/0xa0
[Mon Mar 11 14:29:15 2024] entry_SYSCALL_64_after_hwframe+0x72/0xdc
[Mon Mar 11 14:29:15 2024] RIP: 0033:0x7fedb7d4e84e
[Mon Mar 11 14:29:15 2024] RSP: 002b:00007ffc284a9ab0 EFLAGS: 00000293 ORIG_RAX: 00000000000000e8
[Mon Mar 11 14:29:15 2024] RAX: ffffffffffffffda RBX: 000055f87dd397c0 RCX: 00007fedb7d4e84e
[Mon Mar 11 14:29:15 2024] RDX: 0000000000000001 RSI: 00007ffc284a9b0c RDI: 0000000000000005
[Mon Mar 11 14:29:15 2024] RBP: 000055f87dd2aa20 R08: 0000000000000000 R09: 0000000000000000
[Mon Mar 11 14:29:15 2024] R10: 000000000000ea60 R11: 0000000000000293 R12: 000055f87dd39970
[Mon Mar 11 14:29:15 2024] R13: 000000000000ea60 R14: 0000000000000026 R15: 000055f87dd5b760
[Mon Mar 11 14:29:15 2024] </TASK>
[Mon Mar 11 14:29:15 2024] task:tlsmgr state:S stack:0 pid:158515 ppid:4982 flags:0x00000002
[Mon Mar 11 14:29:15 2024] Call Trace:
[Mon Mar 11 14:29:15 2024] <TASK>
[Mon Mar 11 14:29:15 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:15 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:15 2024] schedule_hrtimeout_range_clock+0x95/0x120
[Mon Mar 11 14:29:15 2024] ? __pfx_hrtimer_wakeup+0x10/0x10
[Mon Mar 11 14:29:15 2024] ep_poll+0x348/0x3b0
[Mon Mar 11 14:29:15 2024] ? __pfx_autoremove_wake_function+0x10/0x10
[Mon Mar 11 14:29:15 2024] do_epoll_wait+0xb1/0xd0
[Mon Mar 11 14:29:15 2024] __x64_sys_epoll_wait+0x60/0x100
[Mon Mar 11 14:29:15 2024] do_syscall_64+0x59/0x90
[Mon Mar 11 14:29:15 2024] ? syscall_exit_to_user_mode+0x22/0x40
[Mon Mar 11 14:29:15 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:15 2024] ? sysvec_apic_timer_interrupt+0x3c/0x90
[Mon Mar 11 14:29:15 2024] entry_SYSCALL_64_after_hwframe+0x72/0xdc
[Mon Mar 11 14:29:15 2024] RIP: 0033:0x7f5a7794e80a
[Mon Mar 11 14:29:15 2024] RSP: 002b:00007ffe2b1ad418 EFLAGS: 00000246 ORIG_RAX: 00000000000000e8
[Mon Mar 11 14:29:15 2024] RAX: ffffffffffffffda RBX: 00007ffe2b1ad430 RCX: 00007f5a7794e80a
[Mon Mar 11 14:29:15 2024] RDX: 0000000000000064 RSI: 00007ffe2b1ad430 RDI: 000000000000000c
[Mon Mar 11 14:29:15 2024] RBP: 00007f5a781797d0 R08: 000055b209bc0750 R09: 0000000000000000
[Mon Mar 11 14:29:15 2024] R10: 000000000017ed00 R11: 0000000000000246 R12: 00007f5a78179794
[Mon Mar 11 14:29:15 2024] R13: 00000000ffffffff R14: 000055b2085fcbd0 R15: 00007f5a78179794
[Mon Mar 11 14:29:15 2024] </TASK>
[Mon Mar 11 14:29:15 2024] task:nfsiod state:I stack:0 pid:179473 ppid:2 flags:0x00004000
[Mon Mar 11 14:29:15 2024] Call Trace:
[Mon Mar 11 14:29:15 2024] <TASK>
[Mon Mar 11 14:29:15 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:15 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:15 2024] rescuer_thread+0x2ca/0x390
[Mon Mar 11 14:29:15 2024] ? __pfx_rescuer_thread+0x10/0x10
[Mon Mar 11 14:29:15 2024] kthread+0xdd/0x100
[Mon Mar 11 14:29:15 2024] ? __pfx_kthread+0x10/0x10
[Mon Mar 11 14:29:15 2024] ret_from_fork+0x29/0x50
[Mon Mar 11 14:29:15 2024] </TASK>
[Mon Mar 11 14:29:15 2024] task:sssd_kcm state:S stack:0 pid:179479 ppid:1 flags:0x00000002
[Mon Mar 11 14:29:15 2024] Call Trace:
[Mon Mar 11 14:29:15 2024] <TASK>
[Mon Mar 11 14:29:15 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:15 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:15 2024] schedule_hrtimeout_range_clock+0x95/0x120
[Mon Mar 11 14:29:15 2024] ? __pfx_hrtimer_wakeup+0x10/0x10
[Mon Mar 11 14:29:15 2024] ep_poll+0x348/0x3b0
[Mon Mar 11 14:29:15 2024] ? __pfx_autoremove_wake_function+0x10/0x10
[Mon Mar 11 14:29:15 2024] do_epoll_wait+0xb1/0xd0
[Mon Mar 11 14:29:15 2024] __x64_sys_epoll_wait+0x60/0x100
[Mon Mar 11 14:29:15 2024] do_syscall_64+0x59/0x90
[Mon Mar 11 14:29:15 2024] ? syscall_exit_work+0x103/0x130
[Mon Mar 11 14:29:15 2024] ? syscall_exit_to_user_mode+0x22/0x40
[Mon Mar 11 14:29:15 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:15 2024] ? sysvec_apic_timer_interrupt+0x3c/0x90
[Mon Mar 11 14:29:15 2024] entry_SYSCALL_64_after_hwframe+0x72/0xdc
[Mon Mar 11 14:29:15 2024] RIP: 0033:0x7f811714e80a
[Mon Mar 11 14:29:15 2024] RSP: 002b:00007fff32ced548 EFLAGS: 00000246 ORIG_RAX: 00000000000000e8
[Mon Mar 11 14:29:15 2024] RAX: ffffffffffffffda RBX: 00005599d32a6430 RCX: 00007f811714e80a
[Mon Mar 11 14:29:15 2024] RDX: 0000000000000001 RSI: 00007fff32ced57c RDI: 0000000000000005
[Mon Mar 11 14:29:15 2024] RBP: 00005599d32a62a0 R08: 00000000000f423b R09: 0000000000000000
[Mon Mar 11 14:29:15 2024] R10: 0000000000002710 R11: 0000000000000246 R12: 00005599d32a6580
[Mon Mar 11 14:29:15 2024] R13: 0000000000002710 R14: 00005599d32b6bd0 R15: 0000000000000001
[Mon Mar 11 14:29:15 2024] </TASK>
[Mon Mar 11 14:29:15 2024] task:smbd[10.91.30.5 state:S stack:0 pid:316974 ppid:4757 flags:0x00000002
[Mon Mar 11 14:29:15 2024] Call Trace:
[Mon Mar 11 14:29:15 2024] <TASK>
[Mon Mar 11 14:29:15 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:15 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:15 2024] schedule_hrtimeout_range_clock+0x95/0x120
[Mon Mar 11 14:29:15 2024] ? __pfx_hrtimer_wakeup+0x10/0x10
[Mon Mar 11 14:29:15 2024] ep_poll+0x348/0x3b0
[Mon Mar 11 14:29:15 2024] ? __pfx_autoremove_wake_function+0x10/0x10
[Mon Mar 11 14:29:15 2024] do_epoll_wait+0xb1/0xd0
[Mon Mar 11 14:29:15 2024] __x64_sys_epoll_wait+0x60/0x100
[Mon Mar 11 14:29:15 2024] do_syscall_64+0x59/0x90
[Mon Mar 11 14:29:15 2024] ? syscall_exit_work+0x103/0x130
[Mon Mar 11 14:29:15 2024] ? syscall_exit_to_user_mode+0x22/0x40
[Mon Mar 11 14:29:15 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:15 2024] ? syscall_exit_to_user_mode+0x22/0x40
[Mon Mar 11 14:29:15 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:15 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:15 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:15 2024] ? exc_page_fault+0x62/0x150
[Mon Mar 11 14:29:15 2024] entry_SYSCALL_64_after_hwframe+0x72/0xdc
[Mon Mar 11 14:29:15 2024] RIP: 0033:0x7fedb7d4e84e
[Mon Mar 11 14:29:15 2024] RSP: 002b:00007ffc284a9ab0 EFLAGS: 00000293 ORIG_RAX: 00000000000000e8
[Mon Mar 11 14:29:15 2024] RAX: ffffffffffffffda RBX: 000055f87dd397c0 RCX: 00007fedb7d4e84e
[Mon Mar 11 14:29:15 2024] RDX: 0000000000000001 RSI: 00007ffc284a9b0c RDI: 0000000000000005
[Mon Mar 11 14:29:15 2024] RBP: 000055f87dd2aa20 R08: 0000000000000000 R09: 0000000000000000
[Mon Mar 11 14:29:15 2024] R10: 000000000000e1d4 R11: 0000000000000293 R12: 000055f87dd61190
[Mon Mar 11 14:29:15 2024] R13: 000000000000e1d4 R14: 0000000000000026 R15: 000055f87dd5b760
[Mon Mar 11 14:29:15 2024] </TASK>
[Mon Mar 11 14:29:15 2024] task:smbd[10.91.30.5 state:S stack:0 pid:317495 ppid:4757 flags:0x00000002
[Mon Mar 11 14:29:15 2024] Call Trace:
[Mon Mar 11 14:29:15 2024] <TASK>
[Mon Mar 11 14:29:15 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:15 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:15 2024] schedule_hrtimeout_range_clock+0x95/0x120
[Mon Mar 11 14:29:15 2024] ? __pfx_hrtimer_wakeup+0x10/0x10
[Mon Mar 11 14:29:15 2024] ep_poll+0x348/0x3b0
[Mon Mar 11 14:29:15 2024] ? __pfx_autoremove_wake_function+0x10/0x10
[Mon Mar 11 14:29:15 2024] do_epoll_wait+0xb1/0xd0
[Mon Mar 11 14:29:15 2024] __x64_sys_epoll_wait+0x60/0x100
[Mon Mar 11 14:29:15 2024] do_syscall_64+0x59/0x90
[Mon Mar 11 14:29:15 2024] ? __audit_filter_op+0xa5/0xf0
[Mon Mar 11 14:29:15 2024] ? mntput_no_expire+0x4a/0x250
[Mon Mar 11 14:29:15 2024] ? audit_reset_context.part.0.constprop.0+0x273/0x2e0
[Mon Mar 11 14:29:15 2024] ? syscall_exit_work+0x103/0x130
[Mon Mar 11 14:29:15 2024] ? syscall_exit_to_user_mode+0x22/0x40
[Mon Mar 11 14:29:15 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:15 2024] entry_SYSCALL_64_after_hwframe+0x72/0xdc
[Mon Mar 11 14:29:15 2024] RIP: 0033:0x7fedb7d4e84e
[Mon Mar 11 14:29:15 2024] RSP: 002b:00007ffc284a9ab0 EFLAGS: 00000293 ORIG_RAX: 00000000000000e8
[Mon Mar 11 14:29:15 2024] RAX: ffffffffffffffda RBX: 000055f87dd397c0 RCX: 00007fedb7d4e84e
[Mon Mar 11 14:29:15 2024] RDX: 0000000000000001 RSI: 00007ffc284a9b0c RDI: 0000000000000005
[Mon Mar 11 14:29:15 2024] RBP: 000055f87dd2aa20 R08: 0000000000000000 R09: 0000000000000000
[Mon Mar 11 14:29:15 2024] R10: 0000000000005f8c R11: 0000000000000293 R12: 000055f87dd61190
[Mon Mar 11 14:29:15 2024] R13: 0000000000005f8c R14: 0000000000000026 R15: 000055f87dd5b760
[Mon Mar 11 14:29:15 2024] </TASK>
[Mon Mar 11 14:29:15 2024] task:smbd[10.86.26.7 state:S stack:0 pid:348298 ppid:4757 flags:0x00000002
[Mon Mar 11 14:29:15 2024] Call Trace:
[Mon Mar 11 14:29:15 2024] <TASK>
[Mon Mar 11 14:29:15 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:15 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:15 2024] schedule_hrtimeout_range_clock+0x95/0x120
[Mon Mar 11 14:29:15 2024] ? __pfx_hrtimer_wakeup+0x10/0x10
[Mon Mar 11 14:29:15 2024] ep_poll+0x348/0x3b0
[Mon Mar 11 14:29:15 2024] ? __pfx_autoremove_wake_function+0x10/0x10
[Mon Mar 11 14:29:15 2024] do_epoll_wait+0xb1/0xd0
[Mon Mar 11 14:29:15 2024] __x64_sys_epoll_wait+0x60/0x100
[Mon Mar 11 14:29:15 2024] do_syscall_64+0x59/0x90
[Mon Mar 11 14:29:15 2024] ? exc_page_fault+0x62/0x150
[Mon Mar 11 14:29:15 2024] entry_SYSCALL_64_after_hwframe+0x72/0xdc
[Mon Mar 11 14:29:15 2024] RIP: 0033:0x7fedb7d4e80a
[Mon Mar 11 14:29:15 2024] RSP: 002b:00007ffc284a9ad8 EFLAGS: 00000246 ORIG_RAX: 00000000000000e8
[Mon Mar 11 14:29:15 2024] RAX: ffffffffffffffda RBX: 000055f87dd397c0 RCX: 00007fedb7d4e80a
[Mon Mar 11 14:29:15 2024] RDX: 0000000000000001 RSI: 00007ffc284a9b0c RDI: 0000000000000005
[Mon Mar 11 14:29:15 2024] RBP: 000055f87dd2aa20 R08: 00000000000f4072 R09: 0000000000000000
[Mon Mar 11 14:29:15 2024] R10: 000000000000ea60 R11: 0000000000000246 R12: 000055f87dd61190
[Mon Mar 11 14:29:15 2024] R13: 000000000000ea60 R14: 0000000000000026 R15: 000055f87dd5b760
[Mon Mar 11 14:29:15 2024] </TASK>
[Mon Mar 11 14:29:15 2024] task:smbd[10.87.27.1 state:S stack:0 pid:789171 ppid:4757 flags:0x00000002
[Mon Mar 11 14:29:15 2024] Call Trace:
[Mon Mar 11 14:29:15 2024] <TASK>
[Mon Mar 11 14:29:15 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:15 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:15 2024] schedule_hrtimeout_range_clock+0x95/0x120
[Mon Mar 11 14:29:15 2024] ? __pfx_hrtimer_wakeup+0x10/0x10
[Mon Mar 11 14:29:15 2024] ep_poll+0x348/0x3b0
[Mon Mar 11 14:29:15 2024] ? __pfx_autoremove_wake_function+0x10/0x10
[Mon Mar 11 14:29:15 2024] do_epoll_wait+0xb1/0xd0
[Mon Mar 11 14:29:15 2024] __x64_sys_epoll_wait+0x60/0x100
[Mon Mar 11 14:29:15 2024] do_syscall_64+0x59/0x90
[Mon Mar 11 14:29:15 2024] ? audit_reset_context.part.0.constprop.0+0x273/0x2e0
[Mon Mar 11 14:29:15 2024] ? syscall_exit_work+0x103/0x130
[Mon Mar 11 14:29:15 2024] ? syscall_exit_to_user_mode+0x22/0x40
[Mon Mar 11 14:29:15 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:15 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:15 2024] entry_SYSCALL_64_after_hwframe+0x72/0xdc
[Mon Mar 11 14:29:15 2024] RIP: 0033:0x7fedb7d4e80a
[Mon Mar 11 14:29:15 2024] RSP: 002b:00007ffc284a9ad8 EFLAGS: 00000246 ORIG_RAX: 00000000000000e8
[Mon Mar 11 14:29:15 2024] RAX: ffffffffffffffda RBX: 000055f87dd397c0 RCX: 00007fedb7d4e80a
[Mon Mar 11 14:29:15 2024] RDX: 0000000000000001 RSI: 00007ffc284a9b0c RDI: 0000000000000005
[Mon Mar 11 14:29:15 2024] RBP: 000055f87dd2aa20 R08: 00000000000f407c R09: 0000000000000000
[Mon Mar 11 14:29:15 2024] R10: 000000000000ea60 R11: 0000000000000246 R12: 000055f87dd61190
[Mon Mar 11 14:29:15 2024] R13: 000000000000ea60 R14: 0000000000000026 R15: 000055f87dd5b760
[Mon Mar 11 14:29:15 2024] </TASK>
[Mon Mar 11 14:29:15 2024] task:smbd[10.88.18.1 state:S stack:0 pid:3000371 ppid:4757 flags:0x00000002
[Mon Mar 11 14:29:15 2024] Call Trace:
[Mon Mar 11 14:29:15 2024] <TASK>
[Mon Mar 11 14:29:15 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:15 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:15 2024] schedule_hrtimeout_range_clock+0x95/0x120
[Mon Mar 11 14:29:15 2024] ? __pfx_hrtimer_wakeup+0x10/0x10
[Mon Mar 11 14:29:15 2024] ep_poll+0x348/0x3b0
[Mon Mar 11 14:29:15 2024] ? __pfx_autoremove_wake_function+0x10/0x10
[Mon Mar 11 14:29:15 2024] do_epoll_wait+0xb1/0xd0
[Mon Mar 11 14:29:15 2024] __x64_sys_epoll_wait+0x60/0x100
[Mon Mar 11 14:29:15 2024] do_syscall_64+0x59/0x90
[Mon Mar 11 14:29:15 2024] ? syscall_exit_to_user_mode+0x22/0x40
[Mon Mar 11 14:29:15 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:15 2024] entry_SYSCALL_64_after_hwframe+0x72/0xdc
[Mon Mar 11 14:29:15 2024] RIP: 0033:0x7fedb7d4e84e
[Mon Mar 11 14:29:15 2024] RSP: 002b:00007ffc284a9ab0 EFLAGS: 00000293 ORIG_RAX: 00000000000000e8
[Mon Mar 11 14:29:15 2024] RAX: ffffffffffffffda RBX: 000055f87dd397c0 RCX: 00007fedb7d4e84e
[Mon Mar 11 14:29:15 2024] RDX: 0000000000000001 RSI: 00007ffc284a9b0c RDI: 0000000000000005
[Mon Mar 11 14:29:15 2024] RBP: 000055f87dd2aa20 R08: 0000000000000000 R09: 0000000000000000
[Mon Mar 11 14:29:15 2024] R10: 000000000000ea60 R11: 0000000000000293 R12: 000055f87dd61190
[Mon Mar 11 14:29:15 2024] R13: 000000000000ea60 R14: 0000000000000026 R15: 000055f87dd5b760
[Mon Mar 11 14:29:15 2024] </TASK>
[Mon Mar 11 14:29:15 2024] task:smbd[10.91.30.5 state:S stack:0 pid:3294561 ppid:4757 flags:0x00000002
[Mon Mar 11 14:29:15 2024] Call Trace:
[Mon Mar 11 14:29:15 2024] <TASK>
[Mon Mar 11 14:29:15 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:15 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:15 2024] schedule_hrtimeout_range_clock+0x95/0x120
[Mon Mar 11 14:29:15 2024] ? __pfx_hrtimer_wakeup+0x10/0x10
[Mon Mar 11 14:29:15 2024] ep_poll+0x348/0x3b0
[Mon Mar 11 14:29:15 2024] ? __pfx_autoremove_wake_function+0x10/0x10
[Mon Mar 11 14:29:15 2024] do_epoll_wait+0xb1/0xd0
[Mon Mar 11 14:29:15 2024] __x64_sys_epoll_wait+0x60/0x100
[Mon Mar 11 14:29:15 2024] do_syscall_64+0x59/0x90
[Mon Mar 11 14:29:15 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:15 2024] ? syscall_exit_to_user_mode+0x22/0x40
[Mon Mar 11 14:29:15 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:15 2024] ? common_interrupt+0x43/0xa0
[Mon Mar 11 14:29:15 2024] entry_SYSCALL_64_after_hwframe+0x72/0xdc
[Mon Mar 11 14:29:15 2024] RIP: 0033:0x7fedb7d4e84e
[Mon Mar 11 14:29:15 2024] RSP: 002b:00007ffc284a9ab0 EFLAGS: 00000293 ORIG_RAX: 00000000000000e8
[Mon Mar 11 14:29:15 2024] RAX: ffffffffffffffda RBX: 000055f87dd397c0 RCX: 00007fedb7d4e84e
[Mon Mar 11 14:29:15 2024] RDX: 0000000000000001 RSI: 00007ffc284a9b0c RDI: 0000000000000005
[Mon Mar 11 14:29:15 2024] RBP: 000055f87dd2aa20 R08: 0000000000000000 R09: 0000000000000000
[Mon Mar 11 14:29:15 2024] R10: 0000000000007018 R11: 0000000000000293 R12: 000055f87dd61190
[Mon Mar 11 14:29:15 2024] R13: 0000000000007018 R14: 0000000000000026 R15: 000055f87dd5b760
[Mon Mar 11 14:29:15 2024] </TASK>
[Mon Mar 11 14:29:15 2024] task:smbd[10.87.20.2 state:S stack:0 pid:3797717 ppid:4757 flags:0x00000002
[Mon Mar 11 14:29:15 2024] Call Trace:
[Mon Mar 11 14:29:15 2024] <TASK>
[Mon Mar 11 14:29:15 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:15 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:15 2024] schedule_hrtimeout_range_clock+0x95/0x120
[Mon Mar 11 14:29:15 2024] ? __pfx_hrtimer_wakeup+0x10/0x10
[Mon Mar 11 14:29:15 2024] ep_poll+0x348/0x3b0
[Mon Mar 11 14:29:15 2024] ? __pfx_autoremove_wake_function+0x10/0x10
[Mon Mar 11 14:29:15 2024] do_epoll_wait+0xb1/0xd0
[Mon Mar 11 14:29:15 2024] __x64_sys_epoll_wait+0x60/0x100
[Mon Mar 11 14:29:15 2024] do_syscall_64+0x59/0x90
[Mon Mar 11 14:29:15 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:15 2024] ? syscall_exit_work+0x103/0x130
[Mon Mar 11 14:29:15 2024] ? syscall_exit_to_user_mode+0x22/0x40
[Mon Mar 11 14:29:15 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:15 2024] ? syscall_exit_work+0x103/0x130
[Mon Mar 11 14:29:15 2024] ? syscall_exit_to_user_mode+0x22/0x40
[Mon Mar 11 14:29:15 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:15 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:15 2024] ? exc_page_fault+0x62/0x150
[Mon Mar 11 14:29:15 2024] entry_SYSCALL_64_after_hwframe+0x72/0xdc
[Mon Mar 11 14:29:15 2024] RIP: 0033:0x7fedb7d4e80a
[Mon Mar 11 14:29:15 2024] RSP: 002b:00007ffc284a9ad8 EFLAGS: 00000246 ORIG_RAX: 00000000000000e8
[Mon Mar 11 14:29:15 2024] RAX: ffffffffffffffda RBX: 000055f87dd397c0 RCX: 00007fedb7d4e80a
[Mon Mar 11 14:29:15 2024] RDX: 0000000000000001 RSI: 00007ffc284a9b0c RDI: 0000000000000005
[Mon Mar 11 14:29:15 2024] RBP: 000055f87dd2aa20 R08: 00000000000f41dd R09: 0000000000000000
[Mon Mar 11 14:29:15 2024] R10: 000000000000ea60 R11: 0000000000000246 R12: 000055f87dd58980
[Mon Mar 11 14:29:15 2024] R13: 000000000000ea60 R14: 0000000000000027 R15: 000055f87dd5b760
[Mon Mar 11 14:29:15 2024] </TASK>
[Mon Mar 11 14:29:15 2024] task:smbd[10.87.30.2 state:S stack:0 pid:185208 ppid:4757 flags:0x00000002
[Mon Mar 11 14:29:15 2024] Call Trace:
[Mon Mar 11 14:29:15 2024] <TASK>
[Mon Mar 11 14:29:15 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:15 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:15 2024] schedule_hrtimeout_range_clock+0x95/0x120
[Mon Mar 11 14:29:15 2024] ? __pfx_hrtimer_wakeup+0x10/0x10
[Mon Mar 11 14:29:15 2024] ep_poll+0x348/0x3b0
[Mon Mar 11 14:29:15 2024] ? __pfx_autoremove_wake_function+0x10/0x10
[Mon Mar 11 14:29:15 2024] do_epoll_wait+0xb1/0xd0
[Mon Mar 11 14:29:15 2024] __x64_sys_epoll_wait+0x60/0x100
[Mon Mar 11 14:29:15 2024] do_syscall_64+0x59/0x90
[Mon Mar 11 14:29:15 2024] ? audit_reset_context.part.0.constprop.0+0x273/0x2e0
[Mon Mar 11 14:29:15 2024] ? syscall_exit_work+0x103/0x130
[Mon Mar 11 14:29:15 2024] ? syscall_exit_to_user_mode+0x22/0x40
[Mon Mar 11 14:29:15 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:15 2024] ? syscall_exit_work+0x103/0x130
[Mon Mar 11 14:29:15 2024] ? syscall_exit_to_user_mode+0x22/0x40
[Mon Mar 11 14:29:15 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:15 2024] ? __irq_exit_rcu+0x46/0xc0
[Mon Mar 11 14:29:15 2024] ? sysvec_apic_timer_interrupt+0x3c/0x90
[Mon Mar 11 14:29:15 2024] entry_SYSCALL_64_after_hwframe+0x72/0xdc
[Mon Mar 11 14:29:15 2024] RIP: 0033:0x7fedb7d4e84e
[Mon Mar 11 14:29:15 2024] RSP: 002b:00007ffc284a9ab0 EFLAGS: 00000293 ORIG_RAX: 00000000000000e8
[Mon Mar 11 14:29:15 2024] RAX: ffffffffffffffda RBX: 000055f87dd397c0 RCX: 00007fedb7d4e84e
[Mon Mar 11 14:29:15 2024] RDX: 0000000000000001 RSI: 00007ffc284a9b0c RDI: 0000000000000005
[Mon Mar 11 14:29:15 2024] RBP: 000055f87dd2aa20 R08: 0000000000000000 R09: 0000000000000000
[Mon Mar 11 14:29:15 2024] R10: 00000000000067a2 R11: 0000000000000293 R12: 000055f87dd61190
[Mon Mar 11 14:29:15 2024] R13: 00000000000067a2 R14: 0000000000000026 R15: 000055f87dd5b760
[Mon Mar 11 14:29:15 2024] </TASK>
[Mon Mar 11 14:29:15 2024] task:smbd[10.91.30.5 state:S stack:0 pid:662850 ppid:4757 flags:0x00000002
[Mon Mar 11 14:29:15 2024] Call Trace:
[Mon Mar 11 14:29:15 2024] <TASK>
[Mon Mar 11 14:29:15 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:15 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:15 2024] schedule_hrtimeout_range_clock+0x95/0x120
[Mon Mar 11 14:29:15 2024] ? __pfx_hrtimer_wakeup+0x10/0x10
[Mon Mar 11 14:29:15 2024] ep_poll+0x348/0x3b0
[Mon Mar 11 14:29:15 2024] ? __pfx_autoremove_wake_function+0x10/0x10
[Mon Mar 11 14:29:15 2024] do_epoll_wait+0xb1/0xd0
[Mon Mar 11 14:29:15 2024] __x64_sys_epoll_wait+0x60/0x100
[Mon Mar 11 14:29:15 2024] do_syscall_64+0x59/0x90
[Mon Mar 11 14:29:15 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:15 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:15 2024] ? exit_to_user_mode_prepare+0xb6/0x100
[Mon Mar 11 14:29:15 2024] ? syscall_exit_to_user_mode+0x22/0x40
[Mon Mar 11 14:29:15 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:15 2024] ? syscall_exit_work+0x103/0x130
[Mon Mar 11 14:29:15 2024] ? syscall_exit_to_user_mode+0x22/0x40
[Mon Mar 11 14:29:15 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:15 2024] ? syscall_exit_to_user_mode+0x22/0x40
[Mon Mar 11 14:29:15 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:15 2024] entry_SYSCALL_64_after_hwframe+0x72/0xdc
[Mon Mar 11 14:29:15 2024] RIP: 0033:0x7fedb7d4e84e
[Mon Mar 11 14:29:15 2024] RSP: 002b:00007ffc284a9ab0 EFLAGS: 00000293 ORIG_RAX: 00000000000000e8
[Mon Mar 11 14:29:15 2024] RAX: ffffffffffffffda RBX: 000055f87dd397c0 RCX: 00007fedb7d4e84e
[Mon Mar 11 14:29:15 2024] RDX: 0000000000000001 RSI: 00007ffc284a9b0c RDI: 0000000000000005
[Mon Mar 11 14:29:15 2024] RBP: 000055f87dd2aa20 R08: 0000000000000000 R09: 0000000000000000
[Mon Mar 11 14:29:15 2024] R10: 000000000000ea60 R11: 0000000000000293 R12: 000055f87dd61190
[Mon Mar 11 14:29:15 2024] R13: 000000000000ea60 R14: 0000000000000026 R15: 000055f87dd5b760
[Mon Mar 11 14:29:15 2024] </TASK>
[Mon Mar 11 14:29:15 2024] task:smbd[134.58.56. state:S stack:0 pid:960684 ppid:4757 flags:0x00000002
[Mon Mar 11 14:29:15 2024] Call Trace:
[Mon Mar 11 14:29:15 2024] <TASK>
[Mon Mar 11 14:29:15 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:15 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:15 2024] schedule_hrtimeout_range_clock+0x95/0x120
[Mon Mar 11 14:29:15 2024] ? __pfx_hrtimer_wakeup+0x10/0x10
[Mon Mar 11 14:29:15 2024] ep_poll+0x348/0x3b0
[Mon Mar 11 14:29:15 2024] ? __pfx_autoremove_wake_function+0x10/0x10
[Mon Mar 11 14:29:15 2024] do_epoll_wait+0xb1/0xd0
[Mon Mar 11 14:29:15 2024] __x64_sys_epoll_wait+0x60/0x100
[Mon Mar 11 14:29:15 2024] do_syscall_64+0x59/0x90
[Mon Mar 11 14:29:15 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:15 2024] entry_SYSCALL_64_after_hwframe+0x72/0xdc
[Mon Mar 11 14:29:15 2024] RIP: 0033:0x7fedb7d4e80a
[Mon Mar 11 14:29:15 2024] RSP: 002b:00007ffc284a9ad8 EFLAGS: 00000246 ORIG_RAX: 00000000000000e8
[Mon Mar 11 14:29:15 2024] RAX: ffffffffffffffda RBX: 000055f87dd397c0 RCX: 00007fedb7d4e80a
[Mon Mar 11 14:29:15 2024] RDX: 0000000000000001 RSI: 00007ffc284a9b0c RDI: 0000000000000005
[Mon Mar 11 14:29:15 2024] RBP: 000055f87dd2aa20 R08: 00000000000f41d8 R09: 0000000000000000
[Mon Mar 11 14:29:15 2024] R10: 000000000000ea60 R11: 0000000000000246 R12: 000055f87dd61190
[Mon Mar 11 14:29:15 2024] R13: 000000000000ea60 R14: 0000000000000026 R15: 000055f87dd5b760
[Mon Mar 11 14:29:15 2024] </TASK>
[Mon Mar 11 14:29:15 2024] task:smbd[10.87.19.5 state:S stack:0 pid:978979 ppid:4757 flags:0x00000002
[Mon Mar 11 14:29:15 2024] Call Trace:
[Mon Mar 11 14:29:15 2024] <TASK>
[Mon Mar 11 14:29:15 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:15 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:15 2024] schedule_hrtimeout_range_clock+0x95/0x120
[Mon Mar 11 14:29:15 2024] ? __pfx_hrtimer_wakeup+0x10/0x10
[Mon Mar 11 14:29:15 2024] ep_poll+0x348/0x3b0
[Mon Mar 11 14:29:15 2024] ? __pfx_autoremove_wake_function+0x10/0x10
[Mon Mar 11 14:29:15 2024] do_epoll_wait+0xb1/0xd0
[Mon Mar 11 14:29:15 2024] __x64_sys_epoll_wait+0x60/0x100
[Mon Mar 11 14:29:15 2024] do_syscall_64+0x59/0x90
[Mon Mar 11 14:29:15 2024] ? audit_reset_context.part.0.constprop.0+0x273/0x2e0
[Mon Mar 11 14:29:15 2024] ? syscall_exit_work+0x103/0x130
[Mon Mar 11 14:29:15 2024] ? syscall_exit_to_user_mode+0x22/0x40
[Mon Mar 11 14:29:15 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:15 2024] ? exit_to_user_mode_prepare+0xec/0x100
[Mon Mar 11 14:29:15 2024] ? syscall_exit_to_user_mode+0x22/0x40
[Mon Mar 11 14:29:15 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:15 2024] ? sysvec_apic_timer_interrupt+0x3c/0x90
[Mon Mar 11 14:29:15 2024] entry_SYSCALL_64_after_hwframe+0x72/0xdc
[Mon Mar 11 14:29:15 2024] RIP: 0033:0x7fedb7d4e80a
[Mon Mar 11 14:29:15 2024] RSP: 002b:00007ffc284a9ad8 EFLAGS: 00000246 ORIG_RAX: 00000000000000e8
[Mon Mar 11 14:29:15 2024] RAX: ffffffffffffffda RBX: 000055f87dd397c0 RCX: 00007fedb7d4e80a
[Mon Mar 11 14:29:15 2024] RDX: 0000000000000001 RSI: 00007ffc284a9b0c RDI: 0000000000000005
[Mon Mar 11 14:29:15 2024] RBP: 000055f87dd2aa20 R08: 00000000000f41d2 R09: 0000000000000000
[Mon Mar 11 14:29:15 2024] R10: 000000000000ea60 R11: 0000000000000246 R12: 000055f87dd58980
[Mon Mar 11 14:29:15 2024] R13: 000000000000ea60 R14: 0000000000000027 R15: 000055f87dd5b760
[Mon Mar 11 14:29:15 2024] </TASK>
[Mon Mar 11 14:29:15 2024] task:smbd[10.87.27.1 state:S stack:0 pid:1045161 ppid:4757 flags:0x00000002
[Mon Mar 11 14:29:15 2024] Call Trace:
[Mon Mar 11 14:29:15 2024] <TASK>
[Mon Mar 11 14:29:15 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:15 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:15 2024] schedule_hrtimeout_range_clock+0x95/0x120
[Mon Mar 11 14:29:15 2024] ? __pfx_hrtimer_wakeup+0x10/0x10
[Mon Mar 11 14:29:15 2024] ep_poll+0x348/0x3b0
[Mon Mar 11 14:29:15 2024] ? __pfx_autoremove_wake_function+0x10/0x10
[Mon Mar 11 14:29:15 2024] do_epoll_wait+0xb1/0xd0
[Mon Mar 11 14:29:15 2024] __x64_sys_epoll_wait+0x60/0x100
[Mon Mar 11 14:29:15 2024] do_syscall_64+0x59/0x90
[Mon Mar 11 14:29:15 2024] ? __irq_exit_rcu+0x46/0xc0
[Mon Mar 11 14:29:15 2024] ? common_interrupt+0x43/0xa0
[Mon Mar 11 14:29:15 2024] entry_SYSCALL_64_after_hwframe+0x72/0xdc
[Mon Mar 11 14:29:15 2024] RIP: 0033:0x7fedb7d4e80a
[Mon Mar 11 14:29:15 2024] RSP: 002b:00007ffc284a9ad8 EFLAGS: 00000246 ORIG_RAX: 00000000000000e8
[Mon Mar 11 14:29:15 2024] RAX: ffffffffffffffda RBX: 000055f87dd397c0 RCX: 00007fedb7d4e80a
[Mon Mar 11 14:29:15 2024] RDX: 0000000000000001 RSI: 00007ffc284a9b0c RDI: 0000000000000005
[Mon Mar 11 14:29:15 2024] RBP: 000055f87dd2aa20 R08: 00000000000f41df R09: 0000000000000000
[Mon Mar 11 14:29:15 2024] R10: 000000000000ea60 R11: 0000000000000246 R12: 000055f87dd61190
[Mon Mar 11 14:29:15 2024] R13: 000000000000ea60 R14: 0000000000000026 R15: 000055f87dd5b760
[Mon Mar 11 14:29:15 2024] </TASK>
[Mon Mar 11 14:29:15 2024] task:smbd[10.87.28.1 state:S stack:0 pid:1049691 ppid:4757 flags:0x00000002
[Mon Mar 11 14:29:15 2024] Call Trace:
[Mon Mar 11 14:29:15 2024] <TASK>
[Mon Mar 11 14:29:15 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:15 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:15 2024] schedule_hrtimeout_range_clock+0x95/0x120
[Mon Mar 11 14:29:15 2024] ? __pfx_hrtimer_wakeup+0x10/0x10
[Mon Mar 11 14:29:15 2024] ep_poll+0x348/0x3b0
[Mon Mar 11 14:29:15 2024] ? __pfx_autoremove_wake_function+0x10/0x10
[Mon Mar 11 14:29:15 2024] do_epoll_wait+0xb1/0xd0
[Mon Mar 11 14:29:15 2024] __x64_sys_epoll_wait+0x60/0x100
[Mon Mar 11 14:29:15 2024] do_syscall_64+0x59/0x90
[Mon Mar 11 14:29:15 2024] ? syscall_exit_to_user_mode+0x22/0x40
[Mon Mar 11 14:29:15 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:15 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:15 2024] entry_SYSCALL_64_after_hwframe+0x72/0xdc
[Mon Mar 11 14:29:15 2024] RIP: 0033:0x7fedb7d4e84e
[Mon Mar 11 14:29:15 2024] RSP: 002b:00007ffc284a9ab0 EFLAGS: 00000293 ORIG_RAX: 00000000000000e8
[Mon Mar 11 14:29:15 2024] RAX: ffffffffffffffda RBX: 000055f87dd397c0 RCX: 00007fedb7d4e84e
[Mon Mar 11 14:29:15 2024] RDX: 0000000000000001 RSI: 00007ffc284a9b0c RDI: 0000000000000005
[Mon Mar 11 14:29:15 2024] RBP: 000055f87dd2aa20 R08: 0000000000000000 R09: 0000000000000000
[Mon Mar 11 14:29:15 2024] R10: 0000000000009906 R11: 0000000000000293 R12: 000055f87dd61190
[Mon Mar 11 14:29:15 2024] R13: 0000000000009906 R14: 0000000000000026 R15: 000055f87dd5b760
[Mon Mar 11 14:29:15 2024] </TASK>
[Mon Mar 11 14:29:15 2024] task:smbd[10.87.31.1 state:S stack:0 pid:1059267 ppid:4757 flags:0x00000002
[Mon Mar 11 14:29:15 2024] Call Trace:
[Mon Mar 11 14:29:15 2024] <TASK>
[Mon Mar 11 14:29:15 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:15 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:15 2024] schedule_hrtimeout_range_clock+0x95/0x120
[Mon Mar 11 14:29:15 2024] ? __pfx_hrtimer_wakeup+0x10/0x10
[Mon Mar 11 14:29:15 2024] ep_poll+0x348/0x3b0
[Mon Mar 11 14:29:15 2024] ? __pfx_autoremove_wake_function+0x10/0x10
[Mon Mar 11 14:29:15 2024] do_epoll_wait+0xb1/0xd0
[Mon Mar 11 14:29:15 2024] __x64_sys_epoll_wait+0x60/0x100
[Mon Mar 11 14:29:15 2024] do_syscall_64+0x59/0x90
[Mon Mar 11 14:29:15 2024] ? syscall_exit_to_user_mode+0x22/0x40
[Mon Mar 11 14:29:15 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:15 2024] ? exc_page_fault+0x62/0x150
[Mon Mar 11 14:29:15 2024] entry_SYSCALL_64_after_hwframe+0x72/0xdc
[Mon Mar 11 14:29:15 2024] RIP: 0033:0x7fedb7d4e80a
[Mon Mar 11 14:29:15 2024] RSP: 002b:00007ffc284a9ad8 EFLAGS: 00000246 ORIG_RAX: 00000000000000e8
[Mon Mar 11 14:29:15 2024] RAX: ffffffffffffffda RBX: 000055f87dd397c0 RCX: 00007fedb7d4e80a
[Mon Mar 11 14:29:15 2024] RDX: 0000000000000001 RSI: 00007ffc284a9b0c RDI: 0000000000000005
[Mon Mar 11 14:29:15 2024] RBP: 000055f87dd2aa20 R08: 000000000000f29b R09: 0000000000000000
[Mon Mar 11 14:29:15 2024] R10: 0000000000004a77 R11: 0000000000000246 R12: 000055f87dd61190
[Mon Mar 11 14:29:15 2024] R13: 0000000000004a77 R14: 0000000000000026 R15: 000055f87dd5b760
[Mon Mar 11 14:29:15 2024] </TASK>
[Mon Mar 11 14:29:15 2024] task:smbd[10.91.30.5 state:S stack:0 pid:1292148 ppid:4757 flags:0x00000002
[Mon Mar 11 14:29:15 2024] Call Trace:
[Mon Mar 11 14:29:15 2024] <TASK>
[Mon Mar 11 14:29:15 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:15 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:15 2024] schedule_hrtimeout_range_clock+0x95/0x120
[Mon Mar 11 14:29:15 2024] ? __pfx_hrtimer_wakeup+0x10/0x10
[Mon Mar 11 14:29:15 2024] ep_poll+0x348/0x3b0
[Mon Mar 11 14:29:15 2024] ? __pfx_autoremove_wake_function+0x10/0x10
[Mon Mar 11 14:29:15 2024] do_epoll_wait+0xb1/0xd0
[Mon Mar 11 14:29:15 2024] __x64_sys_epoll_wait+0x60/0x100
[Mon Mar 11 14:29:15 2024] do_syscall_64+0x59/0x90
[Mon Mar 11 14:29:15 2024] ? __rseq_handle_notify_resume+0x26/0xb0
[Mon Mar 11 14:29:15 2024] ? exit_to_user_mode_loop+0xd0/0x130
[Mon Mar 11 14:29:15 2024] ? exit_to_user_mode_prepare+0xb6/0x100
[Mon Mar 11 14:29:15 2024] ? syscall_exit_to_user_mode+0x22/0x40
[Mon Mar 11 14:29:15 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:15 2024] ? syscall_exit_to_user_mode+0x22/0x40
[Mon Mar 11 14:29:15 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:15 2024] ? __irq_exit_rcu+0x46/0xc0
[Mon Mar 11 14:29:15 2024] ? sysvec_apic_timer_interrupt+0x3c/0x90
[Mon Mar 11 14:29:15 2024] entry_SYSCALL_64_after_hwframe+0x72/0xdc
[Mon Mar 11 14:29:15 2024] RIP: 0033:0x7fedb7d4e84e
[Mon Mar 11 14:29:15 2024] RSP: 002b:00007ffc284a9ab0 EFLAGS: 00000293 ORIG_RAX: 00000000000000e8
[Mon Mar 11 14:29:15 2024] RAX: ffffffffffffffda RBX: 000055f87dd397c0 RCX: 00007fedb7d4e84e
[Mon Mar 11 14:29:15 2024] RDX: 0000000000000001 RSI: 00007ffc284a9b0c RDI: 0000000000000005
[Mon Mar 11 14:29:15 2024] RBP: 000055f87dd2aa20 R08: 0000000000000000 R09: 0000000000000000
[Mon Mar 11 14:29:15 2024] R10: 000000000000e17e R11: 0000000000000293 R12: 000055f87dd61190
[Mon Mar 11 14:29:15 2024] R13: 000000000000e17e R14: 0000000000000026 R15: 000055f87dd5b760
[Mon Mar 11 14:29:15 2024] </TASK>
[Mon Mar 11 14:29:15 2024] task:smbd[10.87.27.4 state:S stack:0 pid:1367201 ppid:4757 flags:0x00000002
[Mon Mar 11 14:29:15 2024] Call Trace:
[Mon Mar 11 14:29:15 2024] <TASK>
[Mon Mar 11 14:29:15 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:15 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:15 2024] schedule_hrtimeout_range_clock+0x95/0x120
[Mon Mar 11 14:29:15 2024] ? __pfx_hrtimer_wakeup+0x10/0x10
[Mon Mar 11 14:29:15 2024] ep_poll+0x348/0x3b0
[Mon Mar 11 14:29:15 2024] ? __pfx_autoremove_wake_function+0x10/0x10
[Mon Mar 11 14:29:15 2024] do_epoll_wait+0xb1/0xd0
[Mon Mar 11 14:29:15 2024] __x64_sys_epoll_wait+0x60/0x100
[Mon Mar 11 14:29:15 2024] do_syscall_64+0x59/0x90
[Mon Mar 11 14:29:15 2024] ? audit_reset_context.part.0.constprop.0+0x273/0x2e0
[Mon Mar 11 14:29:15 2024] ? syscall_exit_work+0x103/0x130
[Mon Mar 11 14:29:15 2024] ? syscall_exit_to_user_mode+0x22/0x40
[Mon Mar 11 14:29:15 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:15 2024] ? exc_page_fault+0x62/0x150
[Mon Mar 11 14:29:15 2024] entry_SYSCALL_64_after_hwframe+0x72/0xdc
[Mon Mar 11 14:29:15 2024] RIP: 0033:0x7fedb7d4e80a
[Mon Mar 11 14:29:15 2024] RSP: 002b:00007ffc284a9ad8 EFLAGS: 00000246 ORIG_RAX: 00000000000000e8
[Mon Mar 11 14:29:15 2024] RAX: ffffffffffffffda RBX: 000055f87dd397c0 RCX: 00007fedb7d4e80a
[Mon Mar 11 14:29:15 2024] RDX: 0000000000000001 RSI: 00007ffc284a9b0c RDI: 0000000000000005
[Mon Mar 11 14:29:15 2024] RBP: 000055f87dd2aa20 R08: 00000000000f4064 R09: 0000000000000000
[Mon Mar 11 14:29:15 2024] R10: 000000000000ea60 R11: 0000000000000246 R12: 000055f87dd61190
[Mon Mar 11 14:29:15 2024] R13: 000000000000ea60 R14: 0000000000000026 R15: 000055f87dd5b760
[Mon Mar 11 14:29:15 2024] </TASK>
[Mon Mar 11 14:29:15 2024] task:smbd[10.91.30.5 state:S stack:0 pid:1834355 ppid:4757 flags:0x00000002
[Mon Mar 11 14:29:15 2024] Call Trace:
[Mon Mar 11 14:29:15 2024] <TASK>
[Mon Mar 11 14:29:15 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:15 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:15 2024] schedule_hrtimeout_range_clock+0x95/0x120
[Mon Mar 11 14:29:15 2024] ? __pfx_hrtimer_wakeup+0x10/0x10
[Mon Mar 11 14:29:15 2024] ep_poll+0x348/0x3b0
[Mon Mar 11 14:29:15 2024] ? __pfx_autoremove_wake_function+0x10/0x10
[Mon Mar 11 14:29:15 2024] do_epoll_wait+0xb1/0xd0
[Mon Mar 11 14:29:15 2024] __x64_sys_epoll_wait+0x60/0x100
[Mon Mar 11 14:29:15 2024] do_syscall_64+0x59/0x90
[Mon Mar 11 14:29:15 2024] ? __rseq_handle_notify_resume+0x26/0xb0
[Mon Mar 11 14:29:15 2024] ? exit_to_user_mode_loop+0xd0/0x130
[Mon Mar 11 14:29:15 2024] ? exit_to_user_mode_prepare+0xb6/0x100
[Mon Mar 11 14:29:15 2024] ? syscall_exit_to_user_mode+0x22/0x40
[Mon Mar 11 14:29:15 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:15 2024] ? syscall_exit_to_user_mode+0x22/0x40
[Mon Mar 11 14:29:15 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:15 2024] ? sysvec_apic_timer_interrupt+0x3c/0x90
[Mon Mar 11 14:29:15 2024] entry_SYSCALL_64_after_hwframe+0x72/0xdc
[Mon Mar 11 14:29:15 2024] RIP: 0033:0x7fedb7d4e84e
[Mon Mar 11 14:29:15 2024] RSP: 002b:00007ffc284a9ab0 EFLAGS: 00000293 ORIG_RAX: 00000000000000e8
[Mon Mar 11 14:29:15 2024] RAX: ffffffffffffffda RBX: 000055f87dd397c0 RCX: 00007fedb7d4e84e
[Mon Mar 11 14:29:15 2024] RDX: 0000000000000001 RSI: 00007ffc284a9b0c RDI: 0000000000000005
[Mon Mar 11 14:29:15 2024] RBP: 000055f87dd2aa20 R08: 0000000000000000 R09: 0000000000000000
[Mon Mar 11 14:29:15 2024] R10: 000000000000580b R11: 0000000000000293 R12: 000055f87dd58980
[Mon Mar 11 14:29:15 2024] R13: 000000000000580b R14: 0000000000000027 R15: 000055f87dd5b760
[Mon Mar 11 14:29:15 2024] </TASK>
[Mon Mar 11 14:29:15 2024] task:kworker/46:1 state:I stack:0 pid:2050654 ppid:2 flags:0x00004000
[Mon Mar 11 14:29:15 2024] Workqueue: 0x0 (mm_percpu_wq)
[Mon Mar 11 14:29:15 2024] Call Trace:
[Mon Mar 11 14:29:15 2024] <TASK>
[Mon Mar 11 14:29:15 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:15 2024] ? __pfx_worker_thread+0x10/0x10
[Mon Mar 11 14:29:15 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:15 2024] worker_thread+0xbb/0x3a0
[Mon Mar 11 14:29:15 2024] ? __pfx_worker_thread+0x10/0x10
[Mon Mar 11 14:29:15 2024] kthread+0xdd/0x100
[Mon Mar 11 14:29:15 2024] ? __pfx_kthread+0x10/0x10
[Mon Mar 11 14:29:15 2024] ret_from_fork+0x29/0x50
[Mon Mar 11 14:29:15 2024] </TASK>
[Mon Mar 11 14:29:15 2024] task:kworker/18:1 state:I stack:0 pid:2072443 ppid:2 flags:0x00004000
[Mon Mar 11 14:29:15 2024] Workqueue: 0x0 (mm_percpu_wq)
[Mon Mar 11 14:29:15 2024] Call Trace:
[Mon Mar 11 14:29:15 2024] <TASK>
[Mon Mar 11 14:29:15 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:15 2024] ? __pfx_worker_thread+0x10/0x10
[Mon Mar 11 14:29:15 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:15 2024] worker_thread+0xbb/0x3a0
[Mon Mar 11 14:29:15 2024] ? __pfx_worker_thread+0x10/0x10
[Mon Mar 11 14:29:15 2024] kthread+0xdd/0x100
[Mon Mar 11 14:29:15 2024] ? __pfx_kthread+0x10/0x10
[Mon Mar 11 14:29:15 2024] ret_from_fork+0x29/0x50
[Mon Mar 11 14:29:15 2024] </TASK>
[Mon Mar 11 14:29:15 2024] task:kworker/34:0 state:I stack:0 pid:2078485 ppid:2 flags:0x00004000
[Mon Mar 11 14:29:15 2024] Workqueue: 0x0 (mm_percpu_wq)
[Mon Mar 11 14:29:15 2024] Call Trace:
[Mon Mar 11 14:29:15 2024] <TASK>
[Mon Mar 11 14:29:15 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:15 2024] ? __pfx_worker_thread+0x10/0x10
[Mon Mar 11 14:29:15 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:15 2024] worker_thread+0xbb/0x3a0
[Mon Mar 11 14:29:15 2024] ? __pfx_worker_thread+0x10/0x10
[Mon Mar 11 14:29:15 2024] kthread+0xdd/0x100
[Mon Mar 11 14:29:15 2024] ? __pfx_kthread+0x10/0x10
[Mon Mar 11 14:29:15 2024] ret_from_fork+0x29/0x50
[Mon Mar 11 14:29:15 2024] </TASK>
[Mon Mar 11 14:29:15 2024] task:kworker/27:2 state:I stack:0 pid:2144617 ppid:2 flags:0x00004000
[Mon Mar 11 14:29:15 2024] Workqueue: 0x0 (mm_percpu_wq)
[Mon Mar 11 14:29:15 2024] Call Trace:
[Mon Mar 11 14:29:15 2024] <TASK>
[Mon Mar 11 14:29:15 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:15 2024] ? __pfx_worker_thread+0x10/0x10
[Mon Mar 11 14:29:15 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:15 2024] worker_thread+0xbb/0x3a0
[Mon Mar 11 14:29:15 2024] ? __pfx_worker_thread+0x10/0x10
[Mon Mar 11 14:29:15 2024] kthread+0xdd/0x100
[Mon Mar 11 14:29:15 2024] ? __pfx_kthread+0x10/0x10
[Mon Mar 11 14:29:15 2024] ret_from_fork+0x29/0x50
[Mon Mar 11 14:29:15 2024] </TASK>
[Mon Mar 11 14:29:15 2024] task:kworker/40:0 state:I stack:0 pid:2193065 ppid:2 flags:0x00004000
[Mon Mar 11 14:29:15 2024] Workqueue: 0x0 (mm_percpu_wq)
[Mon Mar 11 14:29:15 2024] Call Trace:
[Mon Mar 11 14:29:15 2024] <TASK>
[Mon Mar 11 14:29:15 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:15 2024] ? __pfx_worker_thread+0x10/0x10
[Mon Mar 11 14:29:15 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:15 2024] worker_thread+0xbb/0x3a0
[Mon Mar 11 14:29:15 2024] ? __pfx_worker_thread+0x10/0x10
[Mon Mar 11 14:29:15 2024] kthread+0xdd/0x100
[Mon Mar 11 14:29:15 2024] ? __pfx_kthread+0x10/0x10
[Mon Mar 11 14:29:15 2024] ret_from_fork+0x29/0x50
[Mon Mar 11 14:29:15 2024] </TASK>
[Mon Mar 11 14:29:15 2024] task:smbd[10.87.27.3 state:S stack:0 pid:2193786 ppid:4757 flags:0x00000002
[Mon Mar 11 14:29:15 2024] Call Trace:
[Mon Mar 11 14:29:15 2024] <TASK>
[Mon Mar 11 14:29:15 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:15 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:15 2024] schedule_hrtimeout_range_clock+0x95/0x120
[Mon Mar 11 14:29:15 2024] ? __pfx_hrtimer_wakeup+0x10/0x10
[Mon Mar 11 14:29:15 2024] ep_poll+0x348/0x3b0
[Mon Mar 11 14:29:15 2024] ? __pfx_autoremove_wake_function+0x10/0x10
[Mon Mar 11 14:29:15 2024] do_epoll_wait+0xb1/0xd0
[Mon Mar 11 14:29:15 2024] __x64_sys_epoll_wait+0x60/0x100
[Mon Mar 11 14:29:15 2024] do_syscall_64+0x59/0x90
[Mon Mar 11 14:29:15 2024] ? exit_to_user_mode_prepare+0xec/0x100
[Mon Mar 11 14:29:15 2024] ? syscall_exit_to_user_mode+0x22/0x40
[Mon Mar 11 14:29:15 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:15 2024] ? switch_fpu_return+0x4c/0xd0
[Mon Mar 11 14:29:15 2024] ? exit_to_user_mode_prepare+0xec/0x100
[Mon Mar 11 14:29:15 2024] ? syscall_exit_to_user_mode+0x22/0x40
[Mon Mar 11 14:29:15 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:15 2024] ? __irq_exit_rcu+0x46/0xc0
[Mon Mar 11 14:29:15 2024] ? sysvec_apic_timer_interrupt+0x3c/0x90
[Mon Mar 11 14:29:15 2024] entry_SYSCALL_64_after_hwframe+0x72/0xdc
[Mon Mar 11 14:29:15 2024] RIP: 0033:0x7fedb7d4e80a
[Mon Mar 11 14:29:15 2024] RSP: 002b:00007ffc284a9ad8 EFLAGS: 00000246 ORIG_RAX: 00000000000000e8
[Mon Mar 11 14:29:15 2024] RAX: ffffffffffffffda RBX: 000055f87dd397c0 RCX: 00007fedb7d4e80a
[Mon Mar 11 14:29:15 2024] RDX: 0000000000000001 RSI: 00007ffc284a9b0c RDI: 0000000000000005
[Mon Mar 11 14:29:15 2024] RBP: 000055f87dd2aa20 R08: 00000000000e8e4a R09: 0000000000000000
[Mon Mar 11 14:29:15 2024] R10: 000000000000ceda R11: 0000000000000246 R12: 000055f87dd58980
[Mon Mar 11 14:29:15 2024] R13: 000000000000ceda R14: 0000000000000027 R15: 000055f87dd5b760
[Mon Mar 11 14:29:15 2024] </TASK>
[Mon Mar 11 14:29:15 2024] task:smbd[10.87.27.3 state:S stack:0 pid:2193788 ppid:4757 flags:0x00000002
[Mon Mar 11 14:29:15 2024] Call Trace:
[Mon Mar 11 14:29:15 2024] <TASK>
[Mon Mar 11 14:29:15 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:15 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:15 2024] schedule_hrtimeout_range_clock+0x95/0x120
[Mon Mar 11 14:29:15 2024] ? __pfx_hrtimer_wakeup+0x10/0x10
[Mon Mar 11 14:29:15 2024] ep_poll+0x348/0x3b0
[Mon Mar 11 14:29:15 2024] ? __pfx_autoremove_wake_function+0x10/0x10
[Mon Mar 11 14:29:15 2024] do_epoll_wait+0xb1/0xd0
[Mon Mar 11 14:29:15 2024] __x64_sys_epoll_wait+0x60/0x100
[Mon Mar 11 14:29:15 2024] do_syscall_64+0x59/0x90
[Mon Mar 11 14:29:15 2024] ? syscall_exit_to_user_mode+0x22/0x40
[Mon Mar 11 14:29:15 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:15 2024] ? syscall_exit_to_user_mode+0x22/0x40
[Mon Mar 11 14:29:15 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:15 2024] ? syscall_exit_to_user_mode+0x22/0x40
[Mon Mar 11 14:29:15 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:15 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:15 2024] entry_SYSCALL_64_after_hwframe+0x72/0xdc
[Mon Mar 11 14:29:15 2024] RIP: 0033:0x7fedb7d4e80a
[Mon Mar 11 14:29:15 2024] RSP: 002b:00007ffc284a9ad8 EFLAGS: 00000246 ORIG_RAX: 00000000000000e8
[Mon Mar 11 14:29:15 2024] RAX: ffffffffffffffda RBX: 000055f87dd397c0 RCX: 00007fedb7d4e80a
[Mon Mar 11 14:29:15 2024] RDX: 0000000000000001 RSI: 00007ffc284a9b0c RDI: 0000000000000005
[Mon Mar 11 14:29:15 2024] RBP: 000055f87dd2aa20 R08: 000000000000658b R09: 0000000000000000
[Mon Mar 11 14:29:15 2024] R10: 000000000000d30a R11: 0000000000000246 R12: 000055f87dd58980
[Mon Mar 11 14:29:15 2024] R13: 000000000000d30a R14: 0000000000000027 R15: 000055f87dd5b760
[Mon Mar 11 14:29:15 2024] </TASK>
[Mon Mar 11 14:29:15 2024] task:smbd[10.40.238. state:S stack:0 pid:2211016 ppid:4757 flags:0x00000002
[Mon Mar 11 14:29:15 2024] Call Trace:
[Mon Mar 11 14:29:15 2024] <TASK>
[Mon Mar 11 14:29:15 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:15 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:15 2024] schedule_hrtimeout_range_clock+0x95/0x120
[Mon Mar 11 14:29:15 2024] ? __pfx_hrtimer_wakeup+0x10/0x10
[Mon Mar 11 14:29:15 2024] ep_poll+0x348/0x3b0
[Mon Mar 11 14:29:15 2024] ? __pfx_autoremove_wake_function+0x10/0x10
[Mon Mar 11 14:29:15 2024] do_epoll_wait+0xb1/0xd0
[Mon Mar 11 14:29:15 2024] __x64_sys_epoll_wait+0x60/0x100
[Mon Mar 11 14:29:15 2024] do_syscall_64+0x59/0x90
[Mon Mar 11 14:29:15 2024] ? syscall_exit_work+0x103/0x130
[Mon Mar 11 14:29:15 2024] ? syscall_exit_to_user_mode+0x22/0x40
[Mon Mar 11 14:29:15 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:15 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:15 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:15 2024] ? exit_to_user_mode_loop+0xd0/0x130
[Mon Mar 11 14:29:15 2024] ? exit_to_user_mode_prepare+0xb6/0x100
[Mon Mar 11 14:29:15 2024] ? syscall_exit_to_user_mode+0x22/0x40
[Mon Mar 11 14:29:15 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:15 2024] ? exc_page_fault+0x62/0x150
[Mon Mar 11 14:29:15 2024] entry_SYSCALL_64_after_hwframe+0x72/0xdc
[Mon Mar 11 14:29:15 2024] RIP: 0033:0x7fedb7d4e84e
[Mon Mar 11 14:29:15 2024] RSP: 002b:00007ffc284a9ab0 EFLAGS: 00000293 ORIG_RAX: 00000000000000e8
[Mon Mar 11 14:29:15 2024] RAX: ffffffffffffffda RBX: 000055f87dd397c0 RCX: 00007fedb7d4e84e
[Mon Mar 11 14:29:15 2024] RDX: 0000000000000001 RSI: 00007ffc284a9b0c RDI: 0000000000000005
[Mon Mar 11 14:29:15 2024] RBP: 000055f87dd2aa20 R08: 0000000000000000 R09: 0000000000000000
[Mon Mar 11 14:29:15 2024] R10: 000000000000ea60 R11: 0000000000000293 R12: 000055f87dd58980
[Mon Mar 11 14:29:15 2024] R13: 000000000000ea60 R14: 0000000000000027 R15: 000055f87dd5b760
[Mon Mar 11 14:29:15 2024] </TASK>
[Mon Mar 11 14:29:15 2024] task:kworker/5:2 state:I stack:0 pid:2242731 ppid:2 flags:0x00004000
[Mon Mar 11 14:29:15 2024] Workqueue: 0x0 (mm_percpu_wq)
[Mon Mar 11 14:29:15 2024] Call Trace:
[Mon Mar 11 14:29:15 2024] <TASK>
[Mon Mar 11 14:29:15 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:15 2024] ? __pfx_worker_thread+0x10/0x10
[Mon Mar 11 14:29:15 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:15 2024] worker_thread+0xbb/0x3a0
[Mon Mar 11 14:29:15 2024] ? __pfx_worker_thread+0x10/0x10
[Mon Mar 11 14:29:15 2024] kthread+0xdd/0x100
[Mon Mar 11 14:29:15 2024] ? __pfx_kthread+0x10/0x10
[Mon Mar 11 14:29:15 2024] ret_from_fork+0x29/0x50
[Mon Mar 11 14:29:15 2024] </TASK>
[Mon Mar 11 14:29:15 2024] task:kworker/39:2 state:I stack:0 pid:2251290 ppid:2 flags:0x00004000
[Mon Mar 11 14:29:15 2024] Workqueue: 0x0 (mm_percpu_wq)
[Mon Mar 11 14:29:15 2024] Call Trace:
[Mon Mar 11 14:29:15 2024] <TASK>
[Mon Mar 11 14:29:15 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:15 2024] ? __pfx_worker_thread+0x10/0x10
[Mon Mar 11 14:29:15 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:15 2024] worker_thread+0xbb/0x3a0
[Mon Mar 11 14:29:15 2024] ? __pfx_worker_thread+0x10/0x10
[Mon Mar 11 14:29:15 2024] kthread+0xdd/0x100
[Mon Mar 11 14:29:15 2024] ? __pfx_kthread+0x10/0x10
[Mon Mar 11 14:29:15 2024] ret_from_fork+0x29/0x50
[Mon Mar 11 14:29:15 2024] </TASK>
[Mon Mar 11 14:29:15 2024] task:smbd[10.87.29.1 state:S stack:0 pid:2279961 ppid:4757 flags:0x00000002
[Mon Mar 11 14:29:15 2024] Call Trace:
[Mon Mar 11 14:29:15 2024] <TASK>
[Mon Mar 11 14:29:15 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:15 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:15 2024] schedule_hrtimeout_range_clock+0x95/0x120
[Mon Mar 11 14:29:15 2024] ? __pfx_hrtimer_wakeup+0x10/0x10
[Mon Mar 11 14:29:15 2024] ep_poll+0x348/0x3b0
[Mon Mar 11 14:29:15 2024] ? __pfx_autoremove_wake_function+0x10/0x10
[Mon Mar 11 14:29:15 2024] do_epoll_wait+0xb1/0xd0
[Mon Mar 11 14:29:15 2024] __x64_sys_epoll_wait+0x60/0x100
[Mon Mar 11 14:29:15 2024] do_syscall_64+0x59/0x90
[Mon Mar 11 14:29:15 2024] ? syscall_exit_work+0x103/0x130
[Mon Mar 11 14:29:15 2024] ? syscall_exit_to_user_mode+0x22/0x40
[Mon Mar 11 14:29:15 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:15 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:15 2024] entry_SYSCALL_64_after_hwframe+0x72/0xdc
[Mon Mar 11 14:29:15 2024] RIP: 0033:0x7fedb7d4e80a
[Mon Mar 11 14:29:15 2024] RSP: 002b:00007ffc284a9ad8 EFLAGS: 00000246 ORIG_RAX: 00000000000000e8
[Mon Mar 11 14:29:15 2024] RAX: ffffffffffffffda RBX: 000055f87dd397c0 RCX: 00007fedb7d4e80a
[Mon Mar 11 14:29:15 2024] RDX: 0000000000000001 RSI: 00007ffc284a9b0c RDI: 0000000000000005
[Mon Mar 11 14:29:15 2024] RBP: 000055f87dd2aa20 R08: 00000000000f420c R09: 0000000000000000
[Mon Mar 11 14:29:15 2024] R10: 000000000000ea60 R11: 0000000000000246 R12: 000055f87dd58980
[Mon Mar 11 14:29:15 2024] R13: 000000000000ea60 R14: 0000000000000027 R15: 000055f87dd5b760
[Mon Mar 11 14:29:15 2024] </TASK>
[Mon Mar 11 14:29:15 2024] task:smbd[10.87.29.9 state:S stack:0 pid:2279969 ppid:4757 flags:0x00000002
[Mon Mar 11 14:29:15 2024] Call Trace:
[Mon Mar 11 14:29:15 2024] <TASK>
[Mon Mar 11 14:29:15 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:15 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:15 2024] schedule_hrtimeout_range_clock+0x95/0x120
[Mon Mar 11 14:29:15 2024] ? __pfx_hrtimer_wakeup+0x10/0x10
[Mon Mar 11 14:29:15 2024] ep_poll+0x348/0x3b0
[Mon Mar 11 14:29:15 2024] ? __pfx_autoremove_wake_function+0x10/0x10
[Mon Mar 11 14:29:15 2024] do_epoll_wait+0xb1/0xd0
[Mon Mar 11 14:29:15 2024] __x64_sys_epoll_wait+0x60/0x100
[Mon Mar 11 14:29:15 2024] do_syscall_64+0x59/0x90
[Mon Mar 11 14:29:15 2024] ? __audit_filter_op+0xa5/0xf0
[Mon Mar 11 14:29:15 2024] ? mntput_no_expire+0x4a/0x250
[Mon Mar 11 14:29:15 2024] ? audit_reset_context.part.0.constprop.0+0x273/0x2e0
[Mon Mar 11 14:29:15 2024] ? syscall_exit_work+0x103/0x130
[Mon Mar 11 14:29:15 2024] ? syscall_exit_to_user_mode+0x22/0x40
[Mon Mar 11 14:29:15 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:15 2024] entry_SYSCALL_64_after_hwframe+0x72/0xdc
[Mon Mar 11 14:29:15 2024] RIP: 0033:0x7fedb7d4e80a
[Mon Mar 11 14:29:15 2024] RSP: 002b:00007ffc284a9ad8 EFLAGS: 00000246 ORIG_RAX: 00000000000000e8
[Mon Mar 11 14:29:16 2024] RAX: ffffffffffffffda RBX: 000055f87dd397c0 RCX: 00007fedb7d4e80a
[Mon Mar 11 14:29:16 2024] RDX: 0000000000000001 RSI: 00007ffc284a9b0c RDI: 0000000000000005
[Mon Mar 11 14:29:16 2024] RBP: 000055f87dd2aa20 R08: 00000000000f41df R09: 0000000000000077
[Mon Mar 11 14:29:16 2024] R10: 000000000000ea60 R11: 0000000000000246 R12: 000055f87dd58980
[Mon Mar 11 14:29:16 2024] R13: 000000000000ea60 R14: 0000000000000027 R15: 000055f87dd5b760
[Mon Mar 11 14:29:16 2024] </TASK>
[Mon Mar 11 14:29:16 2024] task:smbd[10.87.29.9 state:S stack:0 pid:2279987 ppid:4757 flags:0x00000002
[Mon Mar 11 14:29:16 2024] Call Trace:
[Mon Mar 11 14:29:16 2024] <TASK>
[Mon Mar 11 14:29:16 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:16 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:16 2024] schedule_hrtimeout_range_clock+0x95/0x120
[Mon Mar 11 14:29:16 2024] ? __pfx_hrtimer_wakeup+0x10/0x10
[Mon Mar 11 14:29:16 2024] ep_poll+0x348/0x3b0
[Mon Mar 11 14:29:16 2024] ? __pfx_autoremove_wake_function+0x10/0x10
[Mon Mar 11 14:29:16 2024] do_epoll_wait+0xb1/0xd0
[Mon Mar 11 14:29:16 2024] __x64_sys_epoll_wait+0x60/0x100
[Mon Mar 11 14:29:16 2024] do_syscall_64+0x59/0x90
[Mon Mar 11 14:29:16 2024] ? __audit_filter_op+0xa5/0xf0
[Mon Mar 11 14:29:16 2024] ? mntput_no_expire+0x4a/0x250
[Mon Mar 11 14:29:16 2024] ? audit_reset_context.part.0.constprop.0+0x273/0x2e0
[Mon Mar 11 14:29:16 2024] ? syscall_exit_work+0x103/0x130
[Mon Mar 11 14:29:16 2024] ? syscall_exit_to_user_mode+0x22/0x40
[Mon Mar 11 14:29:16 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:16 2024] ? syscall_exit_to_user_mode+0x22/0x40
[Mon Mar 11 14:29:16 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:16 2024] entry_SYSCALL_64_after_hwframe+0x72/0xdc
[Mon Mar 11 14:29:16 2024] RIP: 0033:0x7fedb7d4e80a
[Mon Mar 11 14:29:16 2024] RSP: 002b:00007ffc284a9ad8 EFLAGS: 00000246 ORIG_RAX: 00000000000000e8
[Mon Mar 11 14:29:16 2024] RAX: ffffffffffffffda RBX: 000055f87dd397c0 RCX: 00007fedb7d4e80a
[Mon Mar 11 14:29:16 2024] RDX: 0000000000000001 RSI: 00007ffc284a9b0c RDI: 0000000000000005
[Mon Mar 11 14:29:16 2024] RBP: 000055f87dd2aa20 R08: 00000000000f41dd R09: 0000000000000000
[Mon Mar 11 14:29:16 2024] R10: 000000000000ea60 R11: 0000000000000246 R12: 000055f87dd61190
[Mon Mar 11 14:29:16 2024] R13: 000000000000ea60 R14: 0000000000000026 R15: 000055f87dd5b760
[Mon Mar 11 14:29:16 2024] </TASK>
[Mon Mar 11 14:29:16 2024] task:smbd[10.87.27.5 state:S stack:0 pid:2292584 ppid:4757 flags:0x00000002
[Mon Mar 11 14:29:16 2024] Call Trace:
[Mon Mar 11 14:29:16 2024] <TASK>
[Mon Mar 11 14:29:16 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:16 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:16 2024] schedule_hrtimeout_range_clock+0x95/0x120
[Mon Mar 11 14:29:16 2024] ? __pfx_hrtimer_wakeup+0x10/0x10
[Mon Mar 11 14:29:16 2024] ep_poll+0x348/0x3b0
[Mon Mar 11 14:29:16 2024] ? __pfx_autoremove_wake_function+0x10/0x10
[Mon Mar 11 14:29:16 2024] do_epoll_wait+0xb1/0xd0
[Mon Mar 11 14:29:16 2024] __x64_sys_epoll_wait+0x60/0x100
[Mon Mar 11 14:29:16 2024] do_syscall_64+0x59/0x90
[Mon Mar 11 14:29:16 2024] ? audit_reset_context.part.0.constprop.0+0x273/0x2e0
[Mon Mar 11 14:29:16 2024] ? syscall_exit_work+0x103/0x130
[Mon Mar 11 14:29:16 2024] ? syscall_exit_to_user_mode+0x22/0x40
[Mon Mar 11 14:29:16 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:16 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:16 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:16 2024] entry_SYSCALL_64_after_hwframe+0x72/0xdc
[Mon Mar 11 14:29:16 2024] RIP: 0033:0x7fedb7d4e84e
[Mon Mar 11 14:29:16 2024] RSP: 002b:00007ffc284a9ab0 EFLAGS: 00000293 ORIG_RAX: 00000000000000e8
[Mon Mar 11 14:29:16 2024] RAX: ffffffffffffffda RBX: 000055f87dd397c0 RCX: 00007fedb7d4e84e
[Mon Mar 11 14:29:16 2024] RDX: 0000000000000001 RSI: 00007ffc284a9b0c RDI: 0000000000000005
[Mon Mar 11 14:29:16 2024] RBP: 000055f87dd2aa20 R08: 0000000000000000 R09: 0000000000000000
[Mon Mar 11 14:29:16 2024] R10: 000000000000ea60 R11: 0000000000000293 R12: 000055f87dd61190
[Mon Mar 11 14:29:16 2024] R13: 000000000000ea60 R14: 0000000000000026 R15: 000055f87dd5b760
[Mon Mar 11 14:29:16 2024] </TASK>
[Mon Mar 11 14:29:16 2024] task:smbd[10.87.29.1 state:S stack:0 pid:2297566 ppid:4757 flags:0x00000002
[Mon Mar 11 14:29:16 2024] Call Trace:
[Mon Mar 11 14:29:16 2024] <TASK>
[Mon Mar 11 14:29:16 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:16 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:16 2024] schedule_hrtimeout_range_clock+0x95/0x120
[Mon Mar 11 14:29:16 2024] ? __pfx_hrtimer_wakeup+0x10/0x10
[Mon Mar 11 14:29:16 2024] ep_poll+0x348/0x3b0
[Mon Mar 11 14:29:16 2024] ? __pfx_autoremove_wake_function+0x10/0x10
[Mon Mar 11 14:29:16 2024] do_epoll_wait+0xb1/0xd0
[Mon Mar 11 14:29:16 2024] __x64_sys_epoll_wait+0x60/0x100
[Mon Mar 11 14:29:16 2024] do_syscall_64+0x59/0x90
[Mon Mar 11 14:29:16 2024] ? syscall_exit_work+0x103/0x130
[Mon Mar 11 14:29:16 2024] ? syscall_exit_to_user_mode+0x22/0x40
[Mon Mar 11 14:29:16 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:16 2024] ? syscall_exit_work+0x103/0x130
[Mon Mar 11 14:29:16 2024] ? syscall_exit_to_user_mode+0x22/0x40
[Mon Mar 11 14:29:16 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:16 2024] ? exc_page_fault+0x62/0x150
[Mon Mar 11 14:29:16 2024] entry_SYSCALL_64_after_hwframe+0x72/0xdc
[Mon Mar 11 14:29:16 2024] RIP: 0033:0x7fedb7d4e80a
[Mon Mar 11 14:29:16 2024] RSP: 002b:00007ffc284a9ad8 EFLAGS: 00000246 ORIG_RAX: 00000000000000e8
[Mon Mar 11 14:29:16 2024] RAX: ffffffffffffffda RBX: 000055f87dd397c0 RCX: 00007fedb7d4e80a
[Mon Mar 11 14:29:16 2024] RDX: 0000000000000001 RSI: 00007ffc284a9b0c RDI: 0000000000000005
[Mon Mar 11 14:29:16 2024] RBP: 000055f87dd2aa20 R08: 0000000000049811 R09: 0000000000000000
[Mon Mar 11 14:29:16 2024] R10: 0000000000002456 R11: 0000000000000246 R12: 000055f87dd61190
[Mon Mar 11 14:29:16 2024] R13: 0000000000002456 R14: 0000000000000026 R15: 000055f87dd5b760
[Mon Mar 11 14:29:16 2024] </TASK>
[Mon Mar 11 14:29:16 2024] task:smbd[10.87.29.1 state:S stack:0 pid:2304037 ppid:4757 flags:0x00000002
[Mon Mar 11 14:29:16 2024] Call Trace:
[Mon Mar 11 14:29:16 2024] <TASK>
[Mon Mar 11 14:29:16 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:16 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:16 2024] schedule_hrtimeout_range_clock+0x95/0x120
[Mon Mar 11 14:29:16 2024] ? __pfx_hrtimer_wakeup+0x10/0x10
[Mon Mar 11 14:29:16 2024] ep_poll+0x348/0x3b0
[Mon Mar 11 14:29:16 2024] ? __pfx_autoremove_wake_function+0x10/0x10
[Mon Mar 11 14:29:16 2024] do_epoll_wait+0xb1/0xd0
[Mon Mar 11 14:29:16 2024] __x64_sys_epoll_wait+0x60/0x100
[Mon Mar 11 14:29:16 2024] do_syscall_64+0x59/0x90
[Mon Mar 11 14:29:16 2024] ? syscall_exit_to_user_mode+0x22/0x40
[Mon Mar 11 14:29:16 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:16 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:16 2024] ? exc_page_fault+0x62/0x150
[Mon Mar 11 14:29:16 2024] entry_SYSCALL_64_after_hwframe+0x72/0xdc
[Mon Mar 11 14:29:16 2024] RIP: 0033:0x7fedb7d4e80a
[Mon Mar 11 14:29:16 2024] RSP: 002b:00007ffc284a9ad8 EFLAGS: 00000246 ORIG_RAX: 00000000000000e8
[Mon Mar 11 14:29:16 2024] RAX: ffffffffffffffda RBX: 000055f87dd397c0 RCX: 00007fedb7d4e80a
[Mon Mar 11 14:29:16 2024] RDX: 0000000000000001 RSI: 00007ffc284a9b0c RDI: 0000000000000005
[Mon Mar 11 14:29:16 2024] RBP: 000055f87dd2aa20 R08: 00000000000f4064 R09: 0000000000000000
[Mon Mar 11 14:29:16 2024] R10: 000000000000ea60 R11: 0000000000000246 R12: 000055f87dd61190
[Mon Mar 11 14:29:16 2024] R13: 000000000000ea60 R14: 0000000000000026 R15: 000055f87dd5b760
[Mon Mar 11 14:29:16 2024] </TASK>
[Mon Mar 11 14:29:16 2024] task:smbd[134.58.56. state:S stack:0 pid:2310377 ppid:4757 flags:0x00000002
[Mon Mar 11 14:29:16 2024] Call Trace:
[Mon Mar 11 14:29:16 2024] <TASK>
[Mon Mar 11 14:29:16 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:16 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:16 2024] schedule_hrtimeout_range_clock+0x95/0x120
[Mon Mar 11 14:29:16 2024] ? __pfx_hrtimer_wakeup+0x10/0x10
[Mon Mar 11 14:29:16 2024] ep_poll+0x348/0x3b0
[Mon Mar 11 14:29:16 2024] ? __pfx_autoremove_wake_function+0x10/0x10
[Mon Mar 11 14:29:16 2024] do_epoll_wait+0xb1/0xd0
[Mon Mar 11 14:29:16 2024] __x64_sys_epoll_wait+0x60/0x100
[Mon Mar 11 14:29:16 2024] do_syscall_64+0x59/0x90
[Mon Mar 11 14:29:16 2024] ? syscall_exit_to_user_mode+0x22/0x40
[Mon Mar 11 14:29:16 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:16 2024] ? syscall_exit_to_user_mode+0x22/0x40
[Mon Mar 11 14:29:16 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:16 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:16 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:16 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:16 2024] entry_SYSCALL_64_after_hwframe+0x72/0xdc
[Mon Mar 11 14:29:16 2024] RIP: 0033:0x7fedb7d4e84e
[Mon Mar 11 14:29:16 2024] RSP: 002b:00007ffc284a9ab0 EFLAGS: 00000293 ORIG_RAX: 00000000000000e8
[Mon Mar 11 14:29:16 2024] RAX: ffffffffffffffda RBX: 000055f87dd397c0 RCX: 00007fedb7d4e84e
[Mon Mar 11 14:29:16 2024] RDX: 0000000000000001 RSI: 00007ffc284a9b0c RDI: 0000000000000005
[Mon Mar 11 14:29:16 2024] RBP: 000055f87dd2aa20 R08: 0000000000000000 R09: 0000000000000000
[Mon Mar 11 14:29:16 2024] R10: 000000000000ea60 R11: 0000000000000293 R12: 000055f87dd61190
[Mon Mar 11 14:29:16 2024] R13: 000000000000ea60 R14: 0000000000000026 R15: 000055f87dd5b760
[Mon Mar 11 14:29:16 2024] </TASK>
[Mon Mar 11 14:29:16 2024] task:smbd[10.87.27.3 state:S stack:0 pid:2318500 ppid:4757 flags:0x00000002
[Mon Mar 11 14:29:16 2024] Call Trace:
[Mon Mar 11 14:29:16 2024] <TASK>
[Mon Mar 11 14:29:16 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:16 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:16 2024] schedule_hrtimeout_range_clock+0x95/0x120
[Mon Mar 11 14:29:16 2024] ? __pfx_hrtimer_wakeup+0x10/0x10
[Mon Mar 11 14:29:16 2024] ep_poll+0x348/0x3b0
[Mon Mar 11 14:29:16 2024] ? __pfx_autoremove_wake_function+0x10/0x10
[Mon Mar 11 14:29:16 2024] do_epoll_wait+0xb1/0xd0
[Mon Mar 11 14:29:16 2024] __x64_sys_epoll_wait+0x60/0x100
[Mon Mar 11 14:29:16 2024] do_syscall_64+0x59/0x90
[Mon Mar 11 14:29:16 2024] ? syscall_exit_work+0x103/0x130
[Mon Mar 11 14:29:16 2024] ? syscall_exit_to_user_mode+0x22/0x40
[Mon Mar 11 14:29:16 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:16 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:16 2024] entry_SYSCALL_64_after_hwframe+0x72/0xdc
[Mon Mar 11 14:29:16 2024] RIP: 0033:0x7fedb7d4e80a
[Mon Mar 11 14:29:16 2024] RSP: 002b:00007ffc284a9ad8 EFLAGS: 00000246 ORIG_RAX: 00000000000000e8
[Mon Mar 11 14:29:16 2024] RAX: ffffffffffffffda RBX: 000055f87dd397c0 RCX: 00007fedb7d4e80a
[Mon Mar 11 14:29:16 2024] RDX: 0000000000000001 RSI: 00007ffc284a9b0c RDI: 0000000000000005
[Mon Mar 11 14:29:16 2024] RBP: 000055f87dd2aa20 R08: 00000000000f41c2 R09: 0000000000000000
[Mon Mar 11 14:29:16 2024] R10: 000000000000ea60 R11: 0000000000000246 R12: 000055f87dd58980
[Mon Mar 11 14:29:16 2024] R13: 000000000000ea60 R14: 0000000000000027 R15: 000055f87dd5b760
[Mon Mar 11 14:29:16 2024] </TASK>
[Mon Mar 11 14:29:16 2024] task:smbd[10.87.28.1 state:S stack:0 pid:2322257 ppid:4757 flags:0x00000002
[Mon Mar 11 14:29:16 2024] Call Trace:
[Mon Mar 11 14:29:16 2024] <TASK>
[Mon Mar 11 14:29:16 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:16 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:16 2024] schedule_hrtimeout_range_clock+0x95/0x120
[Mon Mar 11 14:29:16 2024] ? __pfx_hrtimer_wakeup+0x10/0x10
[Mon Mar 11 14:29:16 2024] ep_poll+0x348/0x3b0
[Mon Mar 11 14:29:16 2024] ? __pfx_autoremove_wake_function+0x10/0x10
[Mon Mar 11 14:29:16 2024] do_epoll_wait+0xb1/0xd0
[Mon Mar 11 14:29:16 2024] __x64_sys_epoll_wait+0x60/0x100
[Mon Mar 11 14:29:16 2024] do_syscall_64+0x59/0x90
[Mon Mar 11 14:29:16 2024] ? syscall_exit_to_user_mode+0x22/0x40
[Mon Mar 11 14:29:16 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:16 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:16 2024] ? exit_to_user_mode_prepare+0xb6/0x100
[Mon Mar 11 14:29:16 2024] ? syscall_exit_to_user_mode+0x22/0x40
[Mon Mar 11 14:29:16 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:16 2024] ? syscall_exit_to_user_mode+0x22/0x40
[Mon Mar 11 14:29:16 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:16 2024] ? syscall_exit_to_user_mode+0x22/0x40
[Mon Mar 11 14:29:16 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:16 2024] ? exc_page_fault+0x62/0x150
[Mon Mar 11 14:29:16 2024] entry_SYSCALL_64_after_hwframe+0x72/0xdc
[Mon Mar 11 14:29:16 2024] RIP: 0033:0x7fedb7d4e84e
[Mon Mar 11 14:29:16 2024] RSP: 002b:00007ffc284a9ab0 EFLAGS: 00000293 ORIG_RAX: 00000000000000e8
[Mon Mar 11 14:29:16 2024] RAX: ffffffffffffffda RBX: 000055f87dd397c0 RCX: 00007fedb7d4e84e
[Mon Mar 11 14:29:16 2024] RDX: 0000000000000001 RSI: 00007ffc284a9b0c RDI: 0000000000000005
[Mon Mar 11 14:29:16 2024] RBP: 000055f87dd2aa20 R08: 0000000000000000 R09: 0000000000000000
[Mon Mar 11 14:29:16 2024] R10: 000000000000ea60 R11: 0000000000000293 R12: 000055f87dd61190
[Mon Mar 11 14:29:16 2024] R13: 000000000000ea60 R14: 0000000000000026 R15: 000055f87dd5b760
[Mon Mar 11 14:29:16 2024] </TASK>
[Mon Mar 11 14:29:16 2024] task:smbd[10.87.28.1 state:S stack:0 pid:2322491 ppid:4757 flags:0x00000002
[Mon Mar 11 14:29:16 2024] Call Trace:
[Mon Mar 11 14:29:16 2024] <TASK>
[Mon Mar 11 14:29:16 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:16 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:16 2024] schedule_hrtimeout_range_clock+0x95/0x120
[Mon Mar 11 14:29:16 2024] ? __pfx_hrtimer_wakeup+0x10/0x10
[Mon Mar 11 14:29:16 2024] ep_poll+0x348/0x3b0
[Mon Mar 11 14:29:16 2024] ? __pfx_autoremove_wake_function+0x10/0x10
[Mon Mar 11 14:29:16 2024] do_epoll_wait+0xb1/0xd0
[Mon Mar 11 14:29:16 2024] __x64_sys_epoll_wait+0x60/0x100
[Mon Mar 11 14:29:16 2024] do_syscall_64+0x59/0x90
[Mon Mar 11 14:29:16 2024] ? audit_reset_context.part.0.constprop.0+0x273/0x2e0
[Mon Mar 11 14:29:16 2024] ? syscall_exit_work+0x103/0x130
[Mon Mar 11 14:29:16 2024] ? syscall_exit_to_user_mode+0x22/0x40
[Mon Mar 11 14:29:16 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:16 2024] ? common_interrupt+0x43/0xa0
[Mon Mar 11 14:29:16 2024] entry_SYSCALL_64_after_hwframe+0x72/0xdc
[Mon Mar 11 14:29:16 2024] RIP: 0033:0x7fedb7d4e84e
[Mon Mar 11 14:29:16 2024] RSP: 002b:00007ffc284a9ab0 EFLAGS: 00000293 ORIG_RAX: 00000000000000e8
[Mon Mar 11 14:29:16 2024] RAX: ffffffffffffffda RBX: 000055f87dd397c0 RCX: 00007fedb7d4e84e
[Mon Mar 11 14:29:16 2024] RDX: 0000000000000001 RSI: 00007ffc284a9b0c RDI: 0000000000000005
[Mon Mar 11 14:29:16 2024] RBP: 000055f87dd2aa20 R08: 0000000000000000 R09: 0000000000000000
[Mon Mar 11 14:29:16 2024] R10: 000000000000ea60 R11: 0000000000000293 R12: 000055f87dd61190
[Mon Mar 11 14:29:16 2024] R13: 000000000000ea60 R14: 0000000000000026 R15: 000055f87dd5b760
[Mon Mar 11 14:29:16 2024] </TASK>
[Mon Mar 11 14:29:16 2024] task:smbd[10.33.88.4 state:S stack:0 pid:2322831 ppid:4757 flags:0x00000002
[Mon Mar 11 14:29:16 2024] Call Trace:
[Mon Mar 11 14:29:16 2024] <TASK>
[Mon Mar 11 14:29:16 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:16 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:16 2024] schedule_hrtimeout_range_clock+0x95/0x120
[Mon Mar 11 14:29:16 2024] ? __pfx_hrtimer_wakeup+0x10/0x10
[Mon Mar 11 14:29:16 2024] ep_poll+0x348/0x3b0
[Mon Mar 11 14:29:16 2024] ? __pfx_autoremove_wake_function+0x10/0x10
[Mon Mar 11 14:29:16 2024] do_epoll_wait+0xb1/0xd0
[Mon Mar 11 14:29:16 2024] __x64_sys_epoll_wait+0x60/0x100
[Mon Mar 11 14:29:16 2024] do_syscall_64+0x59/0x90
[Mon Mar 11 14:29:16 2024] ? audit_reset_context.part.0.constprop.0+0x273/0x2e0
[Mon Mar 11 14:29:16 2024] ? syscall_exit_work+0x103/0x130
[Mon Mar 11 14:29:16 2024] ? syscall_exit_to_user_mode+0x22/0x40
[Mon Mar 11 14:29:16 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:16 2024] ? exit_to_user_mode_prepare+0xec/0x100
[Mon Mar 11 14:29:16 2024] ? sysvec_apic_timer_interrupt+0x3c/0x90
[Mon Mar 11 14:29:16 2024] entry_SYSCALL_64_after_hwframe+0x72/0xdc
[Mon Mar 11 14:29:16 2024] RIP: 0033:0x7fedb7d4e84e
[Mon Mar 11 14:29:16 2024] RSP: 002b:00007ffc284a9ab0 EFLAGS: 00000293 ORIG_RAX: 00000000000000e8
[Mon Mar 11 14:29:16 2024] RAX: ffffffffffffffda RBX: 000055f87dd397c0 RCX: 00007fedb7d4e84e
[Mon Mar 11 14:29:16 2024] RDX: 0000000000000001 RSI: 00007ffc284a9b0c RDI: 0000000000000005
[Mon Mar 11 14:29:16 2024] RBP: 000055f87dd2aa20 R08: 0000000000000000 R09: 0000000000000000
[Mon Mar 11 14:29:16 2024] R10: 0000000000006d32 R11: 0000000000000293 R12: 000055f87dd61190
[Mon Mar 11 14:29:16 2024] R13: 0000000000006d32 R14: 0000000000000026 R15: 000055f87dd5b760
[Mon Mar 11 14:29:16 2024] </TASK>
[Mon Mar 11 14:29:16 2024] task:smbd[10.87.28.1 state:S stack:0 pid:2323430 ppid:4757 flags:0x00000002
[Mon Mar 11 14:29:16 2024] Call Trace:
[Mon Mar 11 14:29:16 2024] <TASK>
[Mon Mar 11 14:29:16 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:16 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:16 2024] schedule_hrtimeout_range_clock+0x95/0x120
[Mon Mar 11 14:29:16 2024] ? __pfx_hrtimer_wakeup+0x10/0x10
[Mon Mar 11 14:29:16 2024] ep_poll+0x348/0x3b0
[Mon Mar 11 14:29:16 2024] ? __pfx_autoremove_wake_function+0x10/0x10
[Mon Mar 11 14:29:16 2024] do_epoll_wait+0xb1/0xd0
[Mon Mar 11 14:29:16 2024] __x64_sys_epoll_wait+0x60/0x100
[Mon Mar 11 14:29:16 2024] do_syscall_64+0x59/0x90
[Mon Mar 11 14:29:16 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:16 2024] ? audit_reset_context.part.0.constprop.0+0x273/0x2e0
[Mon Mar 11 14:29:16 2024] ? syscall_exit_work+0x103/0x130
[Mon Mar 11 14:29:16 2024] ? syscall_exit_to_user_mode+0x22/0x40
[Mon Mar 11 14:29:16 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:16 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:16 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:16 2024] entry_SYSCALL_64_after_hwframe+0x72/0xdc
[Mon Mar 11 14:29:16 2024] RIP: 0033:0x7fedb7d4e80a
[Mon Mar 11 14:29:16 2024] RSP: 002b:00007ffc284a9ad8 EFLAGS: 00000246 ORIG_RAX: 00000000000000e8
[Mon Mar 11 14:29:16 2024] RAX: ffffffffffffffda RBX: 000055f87dd397c0 RCX: 00007fedb7d4e80a
[Mon Mar 11 14:29:16 2024] RDX: 0000000000000001 RSI: 00007ffc284a9b0c RDI: 0000000000000005
[Mon Mar 11 14:29:16 2024] RBP: 000055f87dd2aa20 R08: 00000000000f41da R09: 0000000000000000
[Mon Mar 11 14:29:16 2024] R10: 000000000000ea60 R11: 0000000000000246 R12: 000055f87dd61190
[Mon Mar 11 14:29:16 2024] R13: 000000000000ea60 R14: 0000000000000026 R15: 000055f87dd5b760
[Mon Mar 11 14:29:16 2024] </TASK>
[Mon Mar 11 14:29:16 2024] task:smbd[10.87.28.1 state:S stack:0 pid:2323597 ppid:4757 flags:0x00000002
[Mon Mar 11 14:29:16 2024] Call Trace:
[Mon Mar 11 14:29:16 2024] <TASK>
[Mon Mar 11 14:29:16 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:16 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:16 2024] schedule_hrtimeout_range_clock+0x95/0x120
[Mon Mar 11 14:29:16 2024] ? __pfx_hrtimer_wakeup+0x10/0x10
[Mon Mar 11 14:29:16 2024] ep_poll+0x348/0x3b0
[Mon Mar 11 14:29:16 2024] ? __pfx_autoremove_wake_function+0x10/0x10
[Mon Mar 11 14:29:16 2024] do_epoll_wait+0xb1/0xd0
[Mon Mar 11 14:29:16 2024] __x64_sys_epoll_wait+0x60/0x100
[Mon Mar 11 14:29:16 2024] do_syscall_64+0x59/0x90
[Mon Mar 11 14:29:16 2024] ? syscall_exit_to_user_mode+0x22/0x40
[Mon Mar 11 14:29:16 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:16 2024] ? syscall_exit_to_user_mode+0x22/0x40
[Mon Mar 11 14:29:16 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:16 2024] ? syscall_exit_to_user_mode+0x22/0x40
[Mon Mar 11 14:29:16 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:16 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:16 2024] entry_SYSCALL_64_after_hwframe+0x72/0xdc
[Mon Mar 11 14:29:16 2024] RIP: 0033:0x7fedb7d4e80a
[Mon Mar 11 14:29:16 2024] RSP: 002b:00007ffc284a9ad8 EFLAGS: 00000246 ORIG_RAX: 00000000000000e8
[Mon Mar 11 14:29:16 2024] RAX: ffffffffffffffda RBX: 000055f87dd397c0 RCX: 00007fedb7d4e80a
[Mon Mar 11 14:29:16 2024] RDX: 0000000000000001 RSI: 00007ffc284a9b0c RDI: 0000000000000005
[Mon Mar 11 14:29:16 2024] RBP: 000055f87dd2aa20 R08: 00000000000f41dd R09: 0000000000000000
[Mon Mar 11 14:29:16 2024] R10: 000000000000ea60 R11: 0000000000000246 R12: 000055f87dd61190
[Mon Mar 11 14:29:16 2024] R13: 000000000000ea60 R14: 0000000000000026 R15: 000055f87dd5b760
[Mon Mar 11 14:29:16 2024] </TASK>
[Mon Mar 11 14:29:16 2024] task:smbd[10.87.19.1 state:S stack:0 pid:2324043 ppid:4757 flags:0x00000002
[Mon Mar 11 14:29:16 2024] Call Trace:
[Mon Mar 11 14:29:16 2024] <TASK>
[Mon Mar 11 14:29:16 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:16 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:16 2024] schedule_hrtimeout_range_clock+0x95/0x120
[Mon Mar 11 14:29:16 2024] ? __pfx_hrtimer_wakeup+0x10/0x10
[Mon Mar 11 14:29:16 2024] ep_poll+0x348/0x3b0
[Mon Mar 11 14:29:16 2024] ? __pfx_autoremove_wake_function+0x10/0x10
[Mon Mar 11 14:29:16 2024] do_epoll_wait+0xb1/0xd0
[Mon Mar 11 14:29:16 2024] __x64_sys_epoll_wait+0x60/0x100
[Mon Mar 11 14:29:16 2024] do_syscall_64+0x59/0x90
[Mon Mar 11 14:29:16 2024] ? syscall_exit_work+0x103/0x130
[Mon Mar 11 14:29:16 2024] ? syscall_exit_to_user_mode+0x22/0x40
[Mon Mar 11 14:29:16 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:16 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:16 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:16 2024] entry_SYSCALL_64_after_hwframe+0x72/0xdc
[Mon Mar 11 14:29:16 2024] RIP: 0033:0x7fedb7d4e80a
[Mon Mar 11 14:29:16 2024] RSP: 002b:00007ffc284a9ad8 EFLAGS: 00000246 ORIG_RAX: 00000000000000e8
[Mon Mar 11 14:29:16 2024] RAX: ffffffffffffffda RBX: 000055f87dd397c0 RCX: 00007fedb7d4e80a
[Mon Mar 11 14:29:16 2024] RDX: 0000000000000001 RSI: 00007ffc284a9b0c RDI: 0000000000000005
[Mon Mar 11 14:29:16 2024] RBP: 000055f87dd2aa20 R08: 00000000000aa95a R09: 0000000000000000
[Mon Mar 11 14:29:16 2024] R10: 000000000000ba53 R11: 0000000000000246 R12: 000055f87dd61190
[Mon Mar 11 14:29:16 2024] R13: 000000000000ba53 R14: 0000000000000026 R15: 000055f87dd5b760
[Mon Mar 11 14:29:16 2024] </TASK>
[Mon Mar 11 14:29:16 2024] task:smbd[10.87.19.1 state:S stack:0 pid:2324056 ppid:4757 flags:0x00000002
[Mon Mar 11 14:29:16 2024] Call Trace:
[Mon Mar 11 14:29:16 2024] <TASK>
[Mon Mar 11 14:29:16 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:16 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:16 2024] schedule_hrtimeout_range_clock+0x95/0x120
[Mon Mar 11 14:29:16 2024] ? __pfx_hrtimer_wakeup+0x10/0x10
[Mon Mar 11 14:29:16 2024] ep_poll+0x348/0x3b0
[Mon Mar 11 14:29:16 2024] ? __pfx_autoremove_wake_function+0x10/0x10
[Mon Mar 11 14:29:16 2024] do_epoll_wait+0xb1/0xd0
[Mon Mar 11 14:29:16 2024] __x64_sys_epoll_wait+0x60/0x100
[Mon Mar 11 14:29:16 2024] do_syscall_64+0x59/0x90
[Mon Mar 11 14:29:16 2024] ? __audit_filter_op+0xa5/0xf0
[Mon Mar 11 14:29:16 2024] ? mntput_no_expire+0x4a/0x250
[Mon Mar 11 14:29:16 2024] ? audit_reset_context.part.0.constprop.0+0x273/0x2e0
[Mon Mar 11 14:29:16 2024] ? syscall_exit_work+0x103/0x130
[Mon Mar 11 14:29:16 2024] ? syscall_exit_to_user_mode+0x22/0x40
[Mon Mar 11 14:29:16 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:16 2024] ? syscall_exit_to_user_mode+0x22/0x40
[Mon Mar 11 14:29:16 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:16 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:16 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:16 2024] entry_SYSCALL_64_after_hwframe+0x72/0xdc
[Mon Mar 11 14:29:16 2024] RIP: 0033:0x7fedb7d4e80a
[Mon Mar 11 14:29:16 2024] RSP: 002b:00007ffc284a9ad8 EFLAGS: 00000246 ORIG_RAX: 00000000000000e8
[Mon Mar 11 14:29:16 2024] RAX: ffffffffffffffda RBX: 000055f87dd397c0 RCX: 00007fedb7d4e80a
[Mon Mar 11 14:29:16 2024] RDX: 0000000000000001 RSI: 00007ffc284a9b0c RDI: 0000000000000005
[Mon Mar 11 14:29:16 2024] RBP: 000055f87dd2aa20 R08: 00000000000f4072 R09: 0000000000000000
[Mon Mar 11 14:29:16 2024] R10: 000000000000ea60 R11: 0000000000000246 R12: 000055f87dd61190
[Mon Mar 11 14:29:16 2024] R13: 000000000000ea60 R14: 0000000000000026 R15: 000055f87dd5b760
[Mon Mar 11 14:29:16 2024] </TASK>
[Mon Mar 11 14:29:16 2024] task:smbd[10.87.28.8 state:S stack:0 pid:2325987 ppid:4757 flags:0x00000002
[Mon Mar 11 14:29:16 2024] Call Trace:
[Mon Mar 11 14:29:16 2024] <TASK>
[Mon Mar 11 14:29:16 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:16 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:16 2024] schedule_hrtimeout_range_clock+0x95/0x120
[Mon Mar 11 14:29:16 2024] ? __pfx_hrtimer_wakeup+0x10/0x10
[Mon Mar 11 14:29:16 2024] ep_poll+0x348/0x3b0
[Mon Mar 11 14:29:16 2024] ? __pfx_autoremove_wake_function+0x10/0x10
[Mon Mar 11 14:29:16 2024] do_epoll_wait+0xb1/0xd0
[Mon Mar 11 14:29:16 2024] __x64_sys_epoll_wait+0x60/0x100
[Mon Mar 11 14:29:16 2024] do_syscall_64+0x59/0x90
[Mon Mar 11 14:29:16 2024] ? syscall_exit_to_user_mode+0x22/0x40
[Mon Mar 11 14:29:16 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:16 2024] ? syscall_exit_to_user_mode+0x22/0x40
[Mon Mar 11 14:29:16 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:16 2024] ? sysvec_apic_timer_interrupt+0x3c/0x90
[Mon Mar 11 14:29:16 2024] entry_SYSCALL_64_after_hwframe+0x72/0xdc
[Mon Mar 11 14:29:16 2024] RIP: 0033:0x7fedb7d4e80a
[Mon Mar 11 14:29:16 2024] RSP: 002b:00007ffc284a9ad8 EFLAGS: 00000246 ORIG_RAX: 00000000000000e8
[Mon Mar 11 14:29:16 2024] RAX: ffffffffffffffda RBX: 000055f87dd397c0 RCX: 00007fedb7d4e80a
[Mon Mar 11 14:29:16 2024] RDX: 0000000000000001 RSI: 00007ffc284a9b0c RDI: 0000000000000005
[Mon Mar 11 14:29:16 2024] RBP: 000055f87dd2aa20 R08: 000000000002d516 R09: 0000000000000000
[Mon Mar 11 14:29:16 2024] R10: 0000000000002f9a R11: 0000000000000246 R12: 000055f87dd61190
[Mon Mar 11 14:29:16 2024] R13: 0000000000002f9a R14: 0000000000000026 R15: 000055f87dd5b760
[Mon Mar 11 14:29:16 2024] </TASK>
[Mon Mar 11 14:29:16 2024] task:smbd[10.87.28.8 state:S stack:0 pid:2326005 ppid:4757 flags:0x00000002
[Mon Mar 11 14:29:16 2024] Call Trace:
[Mon Mar 11 14:29:16 2024] <TASK>
[Mon Mar 11 14:29:16 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:16 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:16 2024] schedule_hrtimeout_range_clock+0x95/0x120
[Mon Mar 11 14:29:16 2024] ? __pfx_hrtimer_wakeup+0x10/0x10
[Mon Mar 11 14:29:16 2024] ep_poll+0x348/0x3b0
[Mon Mar 11 14:29:16 2024] ? __pfx_autoremove_wake_function+0x10/0x10
[Mon Mar 11 14:29:16 2024] do_epoll_wait+0xb1/0xd0
[Mon Mar 11 14:29:16 2024] __x64_sys_epoll_wait+0x60/0x100
[Mon Mar 11 14:29:16 2024] do_syscall_64+0x59/0x90
[Mon Mar 11 14:29:16 2024] ? syscall_exit_work+0x103/0x130
[Mon Mar 11 14:29:16 2024] ? syscall_exit_to_user_mode+0x22/0x40
[Mon Mar 11 14:29:16 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:16 2024] ? sysvec_apic_timer_interrupt+0x3c/0x90
[Mon Mar 11 14:29:16 2024] entry_SYSCALL_64_after_hwframe+0x72/0xdc
[Mon Mar 11 14:29:16 2024] RIP: 0033:0x7fedb7d4e84e
[Mon Mar 11 14:29:16 2024] RSP: 002b:00007ffc284a9ab0 EFLAGS: 00000293 ORIG_RAX: 00000000000000e8
[Mon Mar 11 14:29:16 2024] RAX: ffffffffffffffda RBX: 000055f87dd397c0 RCX: 00007fedb7d4e84e
[Mon Mar 11 14:29:16 2024] RDX: 0000000000000001 RSI: 00007ffc284a9b0c RDI: 0000000000000005
[Mon Mar 11 14:29:16 2024] RBP: 000055f87dd2aa20 R08: 0000000000000000 R09: 0000000000000000
[Mon Mar 11 14:29:16 2024] R10: 0000000000003864 R11: 0000000000000293 R12: 000055f87dd61190
[Mon Mar 11 14:29:16 2024] R13: 0000000000003864 R14: 0000000000000026 R15: 000055f87dd5b760
[Mon Mar 11 14:29:16 2024] </TASK>
[Mon Mar 11 14:29:16 2024] task:smbd[10.87.18.2 state:S stack:0 pid:2329756 ppid:4757 flags:0x00000002
[Mon Mar 11 14:29:16 2024] Call Trace:
[Mon Mar 11 14:29:16 2024] <TASK>
[Mon Mar 11 14:29:16 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:16 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:16 2024] schedule_hrtimeout_range_clock+0x95/0x120
[Mon Mar 11 14:29:16 2024] ? __pfx_hrtimer_wakeup+0x10/0x10
[Mon Mar 11 14:29:16 2024] ep_poll+0x348/0x3b0
[Mon Mar 11 14:29:16 2024] ? __pfx_autoremove_wake_function+0x10/0x10
[Mon Mar 11 14:29:16 2024] do_epoll_wait+0xb1/0xd0
[Mon Mar 11 14:29:16 2024] __x64_sys_epoll_wait+0x60/0x100
[Mon Mar 11 14:29:16 2024] do_syscall_64+0x59/0x90
[Mon Mar 11 14:29:16 2024] ? syscall_exit_to_user_mode+0x22/0x40
[Mon Mar 11 14:29:16 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:16 2024] entry_SYSCALL_64_after_hwframe+0x72/0xdc
[Mon Mar 11 14:29:16 2024] RIP: 0033:0x7fedb7d4e80a
[Mon Mar 11 14:29:16 2024] RSP: 002b:00007ffc284a9ad8 EFLAGS: 00000246 ORIG_RAX: 00000000000000e8
[Mon Mar 11 14:29:16 2024] RAX: ffffffffffffffda RBX: 000055f87dd397c0 RCX: 00007fedb7d4e80a
[Mon Mar 11 14:29:16 2024] RDX: 0000000000000001 RSI: 00007ffc284a9b0c RDI: 0000000000000005
[Mon Mar 11 14:29:16 2024] RBP: 000055f87dd2aa20 R08: 00000000000f41be R09: 0000000000000000
[Mon Mar 11 14:29:16 2024] R10: 000000000000ea60 R11: 0000000000000246 R12: 000055f87dd61190
[Mon Mar 11 14:29:16 2024] R13: 000000000000ea60 R14: 0000000000000026 R15: 000055f87dd5b760
[Mon Mar 11 14:29:16 2024] </TASK>
[Mon Mar 11 14:29:16 2024] task:kworker/6:1 state:I stack:0 pid:2330800 ppid:2 flags:0x00004000
[Mon Mar 11 14:29:16 2024] Workqueue: 0x0 (mm_percpu_wq)
[Mon Mar 11 14:29:16 2024] Call Trace:
[Mon Mar 11 14:29:16 2024] <TASK>
[Mon Mar 11 14:29:16 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:16 2024] ? __pfx_worker_thread+0x10/0x10
[Mon Mar 11 14:29:16 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:16 2024] worker_thread+0xbb/0x3a0
[Mon Mar 11 14:29:16 2024] ? __pfx_worker_thread+0x10/0x10
[Mon Mar 11 14:29:16 2024] kthread+0xdd/0x100
[Mon Mar 11 14:29:16 2024] ? __pfx_kthread+0x10/0x10
[Mon Mar 11 14:29:16 2024] ret_from_fork+0x29/0x50
[Mon Mar 11 14:29:16 2024] </TASK>
[Mon Mar 11 14:29:16 2024] task:smbd[10.87.29.1 state:S stack:0 pid:2330921 ppid:4757 flags:0x00000002
[Mon Mar 11 14:29:16 2024] Call Trace:
[Mon Mar 11 14:29:16 2024] <TASK>
[Mon Mar 11 14:29:16 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:16 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:16 2024] schedule_hrtimeout_range_clock+0x95/0x120
[Mon Mar 11 14:29:16 2024] ? __pfx_hrtimer_wakeup+0x10/0x10
[Mon Mar 11 14:29:16 2024] ep_poll+0x348/0x3b0
[Mon Mar 11 14:29:16 2024] ? __pfx_autoremove_wake_function+0x10/0x10
[Mon Mar 11 14:29:16 2024] do_epoll_wait+0xb1/0xd0
[Mon Mar 11 14:29:16 2024] __x64_sys_epoll_wait+0x60/0x100
[Mon Mar 11 14:29:16 2024] do_syscall_64+0x59/0x90
[Mon Mar 11 14:29:16 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:16 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:16 2024] entry_SYSCALL_64_after_hwframe+0x72/0xdc
[Mon Mar 11 14:29:16 2024] RIP: 0033:0x7fedb7d4e84e
[Mon Mar 11 14:29:16 2024] RSP: 002b:00007ffc284a9ab0 EFLAGS: 00000293 ORIG_RAX: 00000000000000e8
[Mon Mar 11 14:29:16 2024] RAX: ffffffffffffffda RBX: 000055f87dd397c0 RCX: 00007fedb7d4e84e
[Mon Mar 11 14:29:16 2024] RDX: 0000000000000001 RSI: 00007ffc284a9b0c RDI: 0000000000000005
[Mon Mar 11 14:29:16 2024] RBP: 000055f87dd2aa20 R08: 0000000000000000 R09: 0000000000000000
[Mon Mar 11 14:29:16 2024] R10: 000000000000ea60 R11: 0000000000000293 R12: 000055f87dd61190
[Mon Mar 11 14:29:16 2024] R13: 000000000000ea60 R14: 0000000000000026 R15: 000055f87dd5b760
[Mon Mar 11 14:29:16 2024] </TASK>
[Mon Mar 11 14:29:16 2024] task:smbd[10.87.29.5 state:S stack:0 pid:2331098 ppid:4757 flags:0x00000002
[Mon Mar 11 14:29:16 2024] Call Trace:
[Mon Mar 11 14:29:16 2024] <TASK>
[Mon Mar 11 14:29:16 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:16 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:16 2024] schedule_hrtimeout_range_clock+0x95/0x120
[Mon Mar 11 14:29:16 2024] ? __pfx_hrtimer_wakeup+0x10/0x10
[Mon Mar 11 14:29:16 2024] ep_poll+0x348/0x3b0
[Mon Mar 11 14:29:16 2024] ? __pfx_autoremove_wake_function+0x10/0x10
[Mon Mar 11 14:29:16 2024] do_epoll_wait+0xb1/0xd0
[Mon Mar 11 14:29:16 2024] __x64_sys_epoll_wait+0x60/0x100
[Mon Mar 11 14:29:16 2024] do_syscall_64+0x59/0x90
[Mon Mar 11 14:29:16 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:16 2024] ? sysvec_call_function+0x3c/0x90
[Mon Mar 11 14:29:16 2024] entry_SYSCALL_64_after_hwframe+0x72/0xdc
[Mon Mar 11 14:29:16 2024] RIP: 0033:0x7fedb7d4e80a
[Mon Mar 11 14:29:16 2024] RSP: 002b:00007ffc284a9ad8 EFLAGS: 00000246 ORIG_RAX: 00000000000000e8
[Mon Mar 11 14:29:16 2024] RAX: ffffffffffffffda RBX: 000055f87dd397c0 RCX: 00007fedb7d4e80a
[Mon Mar 11 14:29:16 2024] RDX: 0000000000000001 RSI: 00007ffc284a9b0c RDI: 0000000000000005
[Mon Mar 11 14:29:16 2024] RBP: 000055f87dd2aa20 R08: 00000000000f41bc R09: 0000000000000000
[Mon Mar 11 14:29:16 2024] R10: 000000000000ea60 R11: 0000000000000246 R12: 000055f87dd61190
[Mon Mar 11 14:29:16 2024] R13: 000000000000ea60 R14: 0000000000000026 R15: 000055f87dd5b760
[Mon Mar 11 14:29:16 2024] </TASK>
[Mon Mar 11 14:29:16 2024] task:smbd[10.87.30.4 state:S stack:0 pid:2334575 ppid:4757 flags:0x00000002
[Mon Mar 11 14:29:16 2024] Call Trace:
[Mon Mar 11 14:29:16 2024] <TASK>
[Mon Mar 11 14:29:16 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:16 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:16 2024] schedule_hrtimeout_range_clock+0x95/0x120
[Mon Mar 11 14:29:16 2024] ? __pfx_hrtimer_wakeup+0x10/0x10
[Mon Mar 11 14:29:16 2024] ep_poll+0x348/0x3b0
[Mon Mar 11 14:29:16 2024] ? __pfx_autoremove_wake_function+0x10/0x10
[Mon Mar 11 14:29:16 2024] do_epoll_wait+0xb1/0xd0
[Mon Mar 11 14:29:16 2024] __x64_sys_epoll_wait+0x60/0x100
[Mon Mar 11 14:29:16 2024] do_syscall_64+0x59/0x90
[Mon Mar 11 14:29:16 2024] ? audit_reset_context.part.0.constprop.0+0x273/0x2e0
[Mon Mar 11 14:29:16 2024] ? syscall_exit_work+0x103/0x130
[Mon Mar 11 14:29:16 2024] ? syscall_exit_to_user_mode+0x22/0x40
[Mon Mar 11 14:29:16 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:16 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:16 2024] ? exit_to_user_mode_prepare+0xec/0x100
[Mon Mar 11 14:29:16 2024] ? syscall_exit_to_user_mode+0x22/0x40
[Mon Mar 11 14:29:16 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:16 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:16 2024] entry_SYSCALL_64_after_hwframe+0x72/0xdc
[Mon Mar 11 14:29:16 2024] RIP: 0033:0x7fedb7d4e80a
[Mon Mar 11 14:29:16 2024] RSP: 002b:00007ffc284a9ad8 EFLAGS: 00000246 ORIG_RAX: 00000000000000e8
[Mon Mar 11 14:29:16 2024] RAX: ffffffffffffffda RBX: 000055f87dd397c0 RCX: 00007fedb7d4e80a
[Mon Mar 11 14:29:16 2024] RDX: 0000000000000001 RSI: 00007ffc284a9b0c RDI: 0000000000000005
[Mon Mar 11 14:29:16 2024] RBP: 000055f87dd2aa20 R08: 00000000000f41e2 R09: 0000000000000000
[Mon Mar 11 14:29:16 2024] R10: 000000000000ea60 R11: 0000000000000246 R12: 000055f87dd58980
[Mon Mar 11 14:29:16 2024] R13: 000000000000ea60 R14: 0000000000000027 R15: 000055f87dd5b760
[Mon Mar 11 14:29:16 2024] </TASK>
[Mon Mar 11 14:29:16 2024] task:smbd[10.87.18.2 state:S stack:0 pid:2338596 ppid:4757 flags:0x00000002
[Mon Mar 11 14:29:16 2024] Call Trace:
[Mon Mar 11 14:29:16 2024] <TASK>
[Mon Mar 11 14:29:16 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:16 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:16 2024] schedule_hrtimeout_range_clock+0x95/0x120
[Mon Mar 11 14:29:16 2024] ? __pfx_hrtimer_wakeup+0x10/0x10
[Mon Mar 11 14:29:16 2024] ep_poll+0x348/0x3b0
[Mon Mar 11 14:29:16 2024] ? __pfx_autoremove_wake_function+0x10/0x10
[Mon Mar 11 14:29:16 2024] do_epoll_wait+0xb1/0xd0
[Mon Mar 11 14:29:16 2024] __x64_sys_epoll_wait+0x60/0x100
[Mon Mar 11 14:29:16 2024] do_syscall_64+0x59/0x90
[Mon Mar 11 14:29:16 2024] ? mntput_no_expire+0x4a/0x250
[Mon Mar 11 14:29:16 2024] ? audit_reset_context.part.0.constprop.0+0x273/0x2e0
[Mon Mar 11 14:29:16 2024] ? syscall_exit_work+0x103/0x130
[Mon Mar 11 14:29:16 2024] ? syscall_exit_to_user_mode+0x22/0x40
[Mon Mar 11 14:29:16 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:16 2024] ? syscall_exit_work+0x103/0x130
[Mon Mar 11 14:29:16 2024] ? syscall_exit_to_user_mode+0x22/0x40
[Mon Mar 11 14:29:16 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:16 2024] ? exc_page_fault+0x62/0x150
[Mon Mar 11 14:29:16 2024] entry_SYSCALL_64_after_hwframe+0x72/0xdc
[Mon Mar 11 14:29:16 2024] RIP: 0033:0x7fedb7d4e80a
[Mon Mar 11 14:29:16 2024] RSP: 002b:00007ffc284a9ad8 EFLAGS: 00000246 ORIG_RAX: 00000000000000e8
[Mon Mar 11 14:29:16 2024] RAX: ffffffffffffffda RBX: 000055f87dd397c0 RCX: 00007fedb7d4e80a
[Mon Mar 11 14:29:16 2024] RDX: 0000000000000001 RSI: 00007ffc284a9b0c RDI: 0000000000000005
[Mon Mar 11 14:29:16 2024] RBP: 000055f87dd2aa20 R08: 00000000000f4121 R09: 0000000000000000
[Mon Mar 11 14:29:16 2024] R10: 000000000000ea60 R11: 0000000000000246 R12: 000055f87dd61190
[Mon Mar 11 14:29:16 2024] R13: 000000000000ea60 R14: 0000000000000026 R15: 000055f87dd5b760
[Mon Mar 11 14:29:16 2024] </TASK>
[Mon Mar 11 14:29:16 2024] task:smbd[10.40.228. state:S stack:0 pid:2338708 ppid:4757 flags:0x00000002
[Mon Mar 11 14:29:16 2024] Call Trace:
[Mon Mar 11 14:29:16 2024] <TASK>
[Mon Mar 11 14:29:16 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:16 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:16 2024] schedule_hrtimeout_range_clock+0x95/0x120
[Mon Mar 11 14:29:16 2024] ? __pfx_hrtimer_wakeup+0x10/0x10
[Mon Mar 11 14:29:16 2024] ep_poll+0x348/0x3b0
[Mon Mar 11 14:29:16 2024] ? __pfx_autoremove_wake_function+0x10/0x10
[Mon Mar 11 14:29:16 2024] do_epoll_wait+0xb1/0xd0
[Mon Mar 11 14:29:16 2024] __x64_sys_epoll_wait+0x60/0x100
[Mon Mar 11 14:29:16 2024] do_syscall_64+0x59/0x90
[Mon Mar 11 14:29:16 2024] ? audit_reset_context.part.0.constprop.0+0x273/0x2e0
[Mon Mar 11 14:29:16 2024] ? syscall_exit_work+0x103/0x130
[Mon Mar 11 14:29:16 2024] ? syscall_exit_to_user_mode+0x22/0x40
[Mon Mar 11 14:29:16 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:16 2024] ? syscall_exit_to_user_mode+0x22/0x40
[Mon Mar 11 14:29:16 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:16 2024] entry_SYSCALL_64_after_hwframe+0x72/0xdc
[Mon Mar 11 14:29:16 2024] RIP: 0033:0x7fedb7d4e84e
[Mon Mar 11 14:29:16 2024] RSP: 002b:00007ffc284a9ab0 EFLAGS: 00000293 ORIG_RAX: 00000000000000e8
[Mon Mar 11 14:29:16 2024] RAX: ffffffffffffffda RBX: 000055f87dd397c0 RCX: 00007fedb7d4e84e
[Mon Mar 11 14:29:16 2024] RDX: 0000000000000001 RSI: 00007ffc284a9b0c RDI: 0000000000000005
[Mon Mar 11 14:29:16 2024] RBP: 000055f87dd2aa20 R08: 0000000000000000 R09: 0000000000000000
[Mon Mar 11 14:29:16 2024] R10: 00000000000003e7 R11: 0000000000000293 R12: 000055f87dd61190
[Mon Mar 11 14:29:16 2024] R13: 00000000000003e7 R14: 0000000000000026 R15: 000055f87dd5b760
[Mon Mar 11 14:29:16 2024] </TASK>
[Mon Mar 11 14:29:16 2024] task:smbd[10.33.22.1 state:S stack:0 pid:2339568 ppid:4757 flags:0x00000002
[Mon Mar 11 14:29:16 2024] Call Trace:
[Mon Mar 11 14:29:16 2024] <TASK>
[Mon Mar 11 14:29:16 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:16 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:16 2024] schedule_hrtimeout_range_clock+0x95/0x120
[Mon Mar 11 14:29:16 2024] ? __pfx_hrtimer_wakeup+0x10/0x10
[Mon Mar 11 14:29:16 2024] ep_poll+0x348/0x3b0
[Mon Mar 11 14:29:16 2024] ? __pfx_autoremove_wake_function+0x10/0x10
[Mon Mar 11 14:29:16 2024] do_epoll_wait+0xb1/0xd0
[Mon Mar 11 14:29:16 2024] __x64_sys_epoll_wait+0x60/0x100
[Mon Mar 11 14:29:16 2024] do_syscall_64+0x59/0x90
[Mon Mar 11 14:29:16 2024] ? syscall_exit_to_user_mode+0x22/0x40
[Mon Mar 11 14:29:16 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:16 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:16 2024] ? exc_page_fault+0x62/0x150
[Mon Mar 11 14:29:16 2024] entry_SYSCALL_64_after_hwframe+0x72/0xdc
[Mon Mar 11 14:29:16 2024] RIP: 0033:0x7fedb7d4e84e
[Mon Mar 11 14:29:16 2024] RSP: 002b:00007ffc284a9ab0 EFLAGS: 00000293 ORIG_RAX: 00000000000000e8
[Mon Mar 11 14:29:16 2024] RAX: ffffffffffffffda RBX: 000055f87dd397c0 RCX: 00007fedb7d4e84e
[Mon Mar 11 14:29:16 2024] RDX: 0000000000000001 RSI: 00007ffc284a9b0c RDI: 0000000000000005
[Mon Mar 11 14:29:16 2024] RBP: 000055f87dd2aa20 R08: 0000000000000000 R09: 0000000000000000
[Mon Mar 11 14:29:16 2024] R10: 000000000000ea60 R11: 0000000000000293 R12: 000055f87dd61190
[Mon Mar 11 14:29:16 2024] R13: 000000000000ea60 R14: 0000000000000026 R15: 000055f87dd5b760
[Mon Mar 11 14:29:16 2024] </TASK>
[Mon Mar 11 14:29:16 2024] task:smbd[10.87.27.1 state:S stack:0 pid:2339621 ppid:4757 flags:0x00000002
[Mon Mar 11 14:29:16 2024] Call Trace:
[Mon Mar 11 14:29:16 2024] <TASK>
[Mon Mar 11 14:29:16 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:16 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:16 2024] schedule_hrtimeout_range_clock+0x95/0x120
[Mon Mar 11 14:29:16 2024] ? __pfx_hrtimer_wakeup+0x10/0x10
[Mon Mar 11 14:29:16 2024] ep_poll+0x348/0x3b0
[Mon Mar 11 14:29:16 2024] ? __pfx_autoremove_wake_function+0x10/0x10
[Mon Mar 11 14:29:16 2024] do_epoll_wait+0xb1/0xd0
[Mon Mar 11 14:29:16 2024] __x64_sys_epoll_wait+0x60/0x100
[Mon Mar 11 14:29:16 2024] do_syscall_64+0x59/0x90
[Mon Mar 11 14:29:16 2024] ? syscall_exit_to_user_mode+0x22/0x40
[Mon Mar 11 14:29:16 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:16 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:16 2024] ? syscall_exit_to_user_mode+0x22/0x40
[Mon Mar 11 14:29:16 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:16 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:16 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:16 2024] ? exc_page_fault+0x62/0x150
[Mon Mar 11 14:29:16 2024] entry_SYSCALL_64_after_hwframe+0x72/0xdc
[Mon Mar 11 14:29:16 2024] RIP: 0033:0x7fedb7d4e80a
[Mon Mar 11 14:29:16 2024] RSP: 002b:00007ffc284a9ad8 EFLAGS: 00000246 ORIG_RAX: 00000000000000e8
[Mon Mar 11 14:29:16 2024] RAX: ffffffffffffffda RBX: 000055f87dd397c0 RCX: 00007fedb7d4e80a
[Mon Mar 11 14:29:16 2024] RDX: 0000000000000001 RSI: 00007ffc284a9b0c RDI: 0000000000000005
[Mon Mar 11 14:29:16 2024] RBP: 000055f87dd2aa20 R08: 00000000000f41d5 R09: 0000000000000000
[Mon Mar 11 14:29:16 2024] R10: 000000000000ea60 R11: 0000000000000246 R12: 000055f87dd61190
[Mon Mar 11 14:29:16 2024] R13: 000000000000ea60 R14: 0000000000000026 R15: 000055f87dd5b760
[Mon Mar 11 14:29:16 2024] </TASK>
[Mon Mar 11 14:29:16 2024] task:samba-dcerpcd state:S stack:0 pid:2339807 ppid:1 flags:0x00000002
[Mon Mar 11 14:29:16 2024] Call Trace:
[Mon Mar 11 14:29:16 2024] <TASK>
[Mon Mar 11 14:29:16 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:16 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:16 2024] schedule_hrtimeout_range_clock+0x95/0x120
[Mon Mar 11 14:29:16 2024] ? __pfx_hrtimer_wakeup+0x10/0x10
[Mon Mar 11 14:29:16 2024] ep_poll+0x348/0x3b0
[Mon Mar 11 14:29:16 2024] ? __pfx_autoremove_wake_function+0x10/0x10
[Mon Mar 11 14:29:16 2024] do_epoll_wait+0xb1/0xd0
[Mon Mar 11 14:29:16 2024] __x64_sys_epoll_wait+0x60/0x100
[Mon Mar 11 14:29:16 2024] do_syscall_64+0x59/0x90
[Mon Mar 11 14:29:16 2024] ? syscall_exit_work+0x103/0x130
[Mon Mar 11 14:29:16 2024] ? syscall_exit_to_user_mode+0x22/0x40
[Mon Mar 11 14:29:16 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:16 2024] ? __rseq_handle_notify_resume+0x26/0xb0
[Mon Mar 11 14:29:16 2024] ? exit_to_user_mode_loop+0xd0/0x130
[Mon Mar 11 14:29:16 2024] ? exit_to_user_mode_prepare+0xec/0x100
[Mon Mar 11 14:29:16 2024] ? syscall_exit_to_user_mode+0x22/0x40
[Mon Mar 11 14:29:16 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:16 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:16 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:16 2024] ? sysvec_apic_timer_interrupt+0x3c/0x90
[Mon Mar 11 14:29:16 2024] entry_SYSCALL_64_after_hwframe+0x72/0xdc
[Mon Mar 11 14:29:16 2024] RIP: 0033:0x7fe4e334e80a
[Mon Mar 11 14:29:16 2024] RSP: 002b:00007ffc108b62d8 EFLAGS: 00000246 ORIG_RAX: 00000000000000e8
[Mon Mar 11 14:29:16 2024] RAX: ffffffffffffffda RBX: 000056040c137190 RCX: 00007fe4e334e80a
[Mon Mar 11 14:29:16 2024] RDX: 0000000000000001 RSI: 00007ffc108b630c RDI: 0000000000000003
[Mon Mar 11 14:29:16 2024] RBP: 000056040c128640 R08: 00000000000e4eb8 R09: 000056040c1452d0
[Mon Mar 11 14:29:16 2024] R10: 00000000000003aa R11: 0000000000000246 R12: 000056040c137220
[Mon Mar 11 14:29:16 2024] R13: 00000000000003aa R14: 000056040c1423f0 R15: 000056040c10b950
[Mon Mar 11 14:29:16 2024] </TASK>
[Mon Mar 11 14:29:16 2024] task:rpcd_spoolss state:S stack:0 pid:2339816 ppid:2339807 flags:0x00000002
[Mon Mar 11 14:29:16 2024] Call Trace:
[Mon Mar 11 14:29:16 2024] <TASK>
[Mon Mar 11 14:29:16 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:16 2024] ? unix_stream_read_generic+0x28b/0x6c0
[Mon Mar 11 14:29:16 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:16 2024] schedule_hrtimeout_range_clock+0x10f/0x120
[Mon Mar 11 14:29:16 2024] ? unix_stream_recvmsg+0x92/0xa0
[Mon Mar 11 14:29:16 2024] ? __pfx_unix_stream_read_actor+0x10/0x10
[Mon Mar 11 14:29:16 2024] ep_poll+0x348/0x3b0
[Mon Mar 11 14:29:16 2024] ? __pfx_autoremove_wake_function+0x10/0x10
[Mon Mar 11 14:29:16 2024] do_epoll_wait+0xb1/0xd0
[Mon Mar 11 14:29:16 2024] __x64_sys_epoll_wait+0x60/0x100
[Mon Mar 11 14:29:16 2024] ? syscall_trace_enter.constprop.0+0x126/0x1a0
[Mon Mar 11 14:29:16 2024] do_syscall_64+0x59/0x90
[Mon Mar 11 14:29:16 2024] ? syscall_exit_work+0x103/0x130
[Mon Mar 11 14:29:16 2024] ? syscall_exit_to_user_mode+0x22/0x40
[Mon Mar 11 14:29:16 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:16 2024] ? audit_reset_context.part.0.constprop.0+0x273/0x2e0
[Mon Mar 11 14:29:16 2024] ? syscall_exit_work+0x103/0x130
[Mon Mar 11 14:29:16 2024] ? syscall_exit_to_user_mode+0x22/0x40
[Mon Mar 11 14:29:16 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:16 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:16 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:16 2024] ? sysvec_apic_timer_interrupt+0x3c/0x90
[Mon Mar 11 14:29:16 2024] entry_SYSCALL_64_after_hwframe+0x72/0xdc
[Mon Mar 11 14:29:16 2024] RIP: 0033:0x7f6df214e80a
[Mon Mar 11 14:29:16 2024] RSP: 002b:00007ffe24c77e08 EFLAGS: 00000246 ORIG_RAX: 00000000000000e8
[Mon Mar 11 14:29:16 2024] RAX: ffffffffffffffda RBX: 00005565e4644850 RCX: 00007f6df214e80a
[Mon Mar 11 14:29:16 2024] RDX: 0000000000000008 RSI: 00005565e4585680 RDI: 0000000000000019
[Mon Mar 11 14:29:16 2024] RBP: 7fffffffffffffff R08: 00005565e4585680 R09: 0000000000000000
[Mon Mar 11 14:29:16 2024] R10: 00000000ffffffff R11: 0000000000000246 R12: 0000000000000028
[Mon Mar 11 14:29:16 2024] R13: 0000000000000008 R14: 0000000000000004 R15: 00005565e46449e0
[Mon Mar 11 14:29:16 2024] </TASK>
[Mon Mar 11 14:29:16 2024] task:samba-bgqd state:S stack:0 pid:2339818 ppid:2339816 flags:0x00000002
[Mon Mar 11 14:29:16 2024] Call Trace:
[Mon Mar 11 14:29:16 2024] <TASK>
[Mon Mar 11 14:29:16 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:16 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:16 2024] schedule_hrtimeout_range_clock+0x95/0x120
[Mon Mar 11 14:29:16 2024] ? __pfx_hrtimer_wakeup+0x10/0x10
[Mon Mar 11 14:29:16 2024] ep_poll+0x348/0x3b0
[Mon Mar 11 14:29:16 2024] ? __pfx_autoremove_wake_function+0x10/0x10
[Mon Mar 11 14:29:16 2024] do_epoll_wait+0xb1/0xd0
[Mon Mar 11 14:29:16 2024] __x64_sys_epoll_wait+0x60/0x100
[Mon Mar 11 14:29:16 2024] do_syscall_64+0x59/0x90
[Mon Mar 11 14:29:16 2024] ? syscall_exit_to_user_mode+0x22/0x40
[Mon Mar 11 14:29:16 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:16 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:16 2024] ? syscall_exit_to_user_mode+0x22/0x40
[Mon Mar 11 14:29:16 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:16 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:16 2024] ? common_interrupt+0x43/0xa0
[Mon Mar 11 14:29:16 2024] entry_SYSCALL_64_after_hwframe+0x72/0xdc
[Mon Mar 11 14:29:16 2024] RIP: 0033:0x7f6aff94e80a
[Mon Mar 11 14:29:16 2024] RSP: 002b:00007fff4f039db8 EFLAGS: 00000246 ORIG_RAX: 00000000000000e8
[Mon Mar 11 14:29:16 2024] RAX: ffffffffffffffda RBX: 00005588fecf7900 RCX: 00007f6aff94e80a
[Mon Mar 11 14:29:16 2024] RDX: 0000000000000001 RSI: 00007fff4f039dec RDI: 0000000000000004
[Mon Mar 11 14:29:16 2024] RBP: 00005588fece1710 R08: 00000000000768d2 R09: 00007fff4f039550
[Mon Mar 11 14:29:16 2024] R10: 00000000000a8936 R11: 0000000000000246 R12: 00005588fecf7990
[Mon Mar 11 14:29:16 2024] R13: 00000000000a8936 R14: 00005588fed05cf0 R15: 0000000000000000
[Mon Mar 11 14:29:16 2024] </TASK>
[Mon Mar 11 14:29:16 2024] task:rpcd_spoolss state:S stack:0 pid:2339825 ppid:2339807 flags:0x00000002
[Mon Mar 11 14:29:16 2024] Call Trace:
[Mon Mar 11 14:29:16 2024] <TASK>
[Mon Mar 11 14:29:16 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:16 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:16 2024] schedule_hrtimeout_range_clock+0x10f/0x120
[Mon Mar 11 14:29:16 2024] ? __sys_recvfrom+0xa8/0x120
[Mon Mar 11 14:29:16 2024] ep_poll+0x348/0x3b0
[Mon Mar 11 14:29:16 2024] ? __pfx_autoremove_wake_function+0x10/0x10
[Mon Mar 11 14:29:16 2024] do_epoll_wait+0xb1/0xd0
[Mon Mar 11 14:29:16 2024] __x64_sys_epoll_wait+0x60/0x100
[Mon Mar 11 14:29:16 2024] ? syscall_trace_enter.constprop.0+0x126/0x1a0
[Mon Mar 11 14:29:16 2024] do_syscall_64+0x59/0x90
[Mon Mar 11 14:29:16 2024] ? syscall_exit_to_user_mode+0x22/0x40
[Mon Mar 11 14:29:16 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:16 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:16 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:16 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:16 2024] ? __irq_exit_rcu+0x46/0xc0
[Mon Mar 11 14:29:16 2024] ? sysvec_apic_timer_interrupt+0x3c/0x90
[Mon Mar 11 14:29:16 2024] entry_SYSCALL_64_after_hwframe+0x72/0xdc
[Mon Mar 11 14:29:16 2024] RIP: 0033:0x7fb696b4e80a
[Mon Mar 11 14:29:16 2024] RSP: 002b:00007fff0751fc48 EFLAGS: 00000246 ORIG_RAX: 00000000000000e8
[Mon Mar 11 14:29:16 2024] RAX: ffffffffffffffda RBX: 00005619fa7f5c60 RCX: 00007fb696b4e80a
[Mon Mar 11 14:29:16 2024] RDX: 0000000000000008 RSI: 00005619fa8465b0 RDI: 0000000000000019
[Mon Mar 11 14:29:16 2024] RBP: 7fffffffffffffff R08: 00005619fa8465b0 R09: 0000000000000000
[Mon Mar 11 14:29:16 2024] R10: 00000000ffffffff R11: 0000000000000246 R12: 0000000000000028
[Mon Mar 11 14:29:16 2024] R13: 0000000000000008 R14: 0000000000000004 R15: 00005619fa7f5df0
[Mon Mar 11 14:29:16 2024] </TASK>
[Mon Mar 11 14:29:16 2024] task:smbd[10.87.28.1 state:S stack:0 pid:2340410 ppid:4757 flags:0x00000002
[Mon Mar 11 14:29:16 2024] Call Trace:
[Mon Mar 11 14:29:16 2024] <TASK>
[Mon Mar 11 14:29:16 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:16 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:16 2024] schedule_hrtimeout_range_clock+0x95/0x120
[Mon Mar 11 14:29:16 2024] ? __pfx_hrtimer_wakeup+0x10/0x10
[Mon Mar 11 14:29:16 2024] ep_poll+0x348/0x3b0
[Mon Mar 11 14:29:16 2024] ? __pfx_autoremove_wake_function+0x10/0x10
[Mon Mar 11 14:29:16 2024] do_epoll_wait+0xb1/0xd0
[Mon Mar 11 14:29:16 2024] __x64_sys_epoll_wait+0x60/0x100
[Mon Mar 11 14:29:16 2024] do_syscall_64+0x59/0x90
[Mon Mar 11 14:29:16 2024] ? common_interrupt+0x43/0xa0
[Mon Mar 11 14:29:16 2024] entry_SYSCALL_64_after_hwframe+0x72/0xdc
[Mon Mar 11 14:29:16 2024] RIP: 0033:0x7fedb7d4e80a
[Mon Mar 11 14:29:16 2024] RSP: 002b:00007ffc284a9ad8 EFLAGS: 00000246 ORIG_RAX: 00000000000000e8
[Mon Mar 11 14:29:16 2024] RAX: ffffffffffffffda RBX: 000055f87dd397c0 RCX: 00007fedb7d4e80a
[Mon Mar 11 14:29:16 2024] RDX: 0000000000000001 RSI: 00007ffc284a9b0c RDI: 0000000000000005
[Mon Mar 11 14:29:16 2024] RBP: 000055f87dd2aa20 R08: 00000000000f4017 R09: 0000000000000000
[Mon Mar 11 14:29:16 2024] R10: 000000000000ea60 R11: 0000000000000246 R12: 000055f87dd61190
[Mon Mar 11 14:29:16 2024] R13: 000000000000ea60 R14: 0000000000000026 R15: 000055f87dd5b760
[Mon Mar 11 14:29:16 2024] </TASK>
[Mon Mar 11 14:29:16 2024] task:smbd[10.40.230. state:S stack:0 pid:2340528 ppid:4757 flags:0x00000002
[Mon Mar 11 14:29:16 2024] Call Trace:
[Mon Mar 11 14:29:16 2024] <TASK>
[Mon Mar 11 14:29:16 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:16 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:16 2024] schedule_hrtimeout_range_clock+0x95/0x120
[Mon Mar 11 14:29:16 2024] ? __pfx_hrtimer_wakeup+0x10/0x10
[Mon Mar 11 14:29:16 2024] ep_poll+0x348/0x3b0
[Mon Mar 11 14:29:16 2024] ? __pfx_autoremove_wake_function+0x10/0x10
[Mon Mar 11 14:29:16 2024] do_epoll_wait+0xb1/0xd0
[Mon Mar 11 14:29:16 2024] __x64_sys_epoll_wait+0x60/0x100
[Mon Mar 11 14:29:16 2024] do_syscall_64+0x59/0x90
[Mon Mar 11 14:29:16 2024] ? common_interrupt+0x43/0xa0
[Mon Mar 11 14:29:16 2024] entry_SYSCALL_64_after_hwframe+0x72/0xdc
[Mon Mar 11 14:29:16 2024] RIP: 0033:0x7fedb7d4e84e
[Mon Mar 11 14:29:16 2024] RSP: 002b:00007ffc284a9ab0 EFLAGS: 00000293 ORIG_RAX: 00000000000000e8
[Mon Mar 11 14:29:16 2024] RAX: ffffffffffffffda RBX: 000055f87dd397c0 RCX: 00007fedb7d4e84e
[Mon Mar 11 14:29:16 2024] RDX: 0000000000000001 RSI: 00007ffc284a9b0c RDI: 0000000000000005
[Mon Mar 11 14:29:16 2024] RBP: 000055f87dd2aa20 R08: 0000000000000000 R09: 0000000000000000
[Mon Mar 11 14:29:16 2024] R10: 000000000000ce30 R11: 0000000000000293 R12: 000055f87dd61190
[Mon Mar 11 14:29:16 2024] R13: 000000000000ce30 R14: 0000000000000026 R15: 000055f87dd5b760
[Mon Mar 11 14:29:16 2024] </TASK>
[Mon Mar 11 14:29:16 2024] task:smbd[10.87.30.3 state:S stack:0 pid:2341031 ppid:4757 flags:0x00000002
[Mon Mar 11 14:29:16 2024] Call Trace:
[Mon Mar 11 14:29:16 2024] <TASK>
[Mon Mar 11 14:29:16 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:16 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:16 2024] schedule_hrtimeout_range_clock+0x95/0x120
[Mon Mar 11 14:29:16 2024] ? __pfx_hrtimer_wakeup+0x10/0x10
[Mon Mar 11 14:29:16 2024] ep_poll+0x348/0x3b0
[Mon Mar 11 14:29:16 2024] ? __pfx_autoremove_wake_function+0x10/0x10
[Mon Mar 11 14:29:16 2024] do_epoll_wait+0xb1/0xd0
[Mon Mar 11 14:29:16 2024] __x64_sys_epoll_wait+0x60/0x100
[Mon Mar 11 14:29:16 2024] do_syscall_64+0x59/0x90
[Mon Mar 11 14:29:16 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:16 2024] ? __do_softirq+0x16a/0x2ac
[Mon Mar 11 14:29:16 2024] ? __irq_exit_rcu+0x46/0xc0
[Mon Mar 11 14:29:16 2024] ? common_interrupt+0x43/0xa0
[Mon Mar 11 14:29:16 2024] entry_SYSCALL_64_after_hwframe+0x72/0xdc
[Mon Mar 11 14:29:16 2024] RIP: 0033:0x7fedb7d4e80a
[Mon Mar 11 14:29:16 2024] RSP: 002b:00007ffc284a9ad8 EFLAGS: 00000246 ORIG_RAX: 00000000000000e8
[Mon Mar 11 14:29:16 2024] RAX: ffffffffffffffda RBX: 000055f87dd397c0 RCX: 00007fedb7d4e80a
[Mon Mar 11 14:29:16 2024] RDX: 0000000000000001 RSI: 00007ffc284a9b0c RDI: 0000000000000005
[Mon Mar 11 14:29:16 2024] RBP: 000055f87dd2aa20 R08: 00000000000f41e0 R09: 0000000000000000
[Mon Mar 11 14:29:16 2024] R10: 000000000000ea60 R11: 0000000000000246 R12: 000055f87dd61190
[Mon Mar 11 14:29:16 2024] R13: 000000000000ea60 R14: 0000000000000026 R15: 000055f87dd5b760
[Mon Mar 11 14:29:16 2024] </TASK>
[Mon Mar 11 14:29:16 2024] task:smbd[10.87.29.1 state:S stack:0 pid:2341353 ppid:4757 flags:0x00000002
[Mon Mar 11 14:29:16 2024] Call Trace:
[Mon Mar 11 14:29:16 2024] <TASK>
[Mon Mar 11 14:29:16 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:16 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:16 2024] schedule_hrtimeout_range_clock+0x95/0x120
[Mon Mar 11 14:29:16 2024] ? __pfx_hrtimer_wakeup+0x10/0x10
[Mon Mar 11 14:29:16 2024] ep_poll+0x348/0x3b0
[Mon Mar 11 14:29:16 2024] ? __pfx_autoremove_wake_function+0x10/0x10
[Mon Mar 11 14:29:16 2024] do_epoll_wait+0xb1/0xd0
[Mon Mar 11 14:29:16 2024] __x64_sys_epoll_wait+0x60/0x100
[Mon Mar 11 14:29:16 2024] do_syscall_64+0x59/0x90
[Mon Mar 11 14:29:16 2024] ? syscall_exit_to_user_mode+0x22/0x40
[Mon Mar 11 14:29:16 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:16 2024] ? audit_reset_context.part.0.constprop.0+0x273/0x2e0
[Mon Mar 11 14:29:16 2024] ? syscall_exit_work+0x103/0x130
[Mon Mar 11 14:29:16 2024] ? syscall_exit_to_user_mode+0x22/0x40
[Mon Mar 11 14:29:16 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:16 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:16 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:16 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:16 2024] entry_SYSCALL_64_after_hwframe+0x72/0xdc
[Mon Mar 11 14:29:16 2024] RIP: 0033:0x7fedb7d4e80a
[Mon Mar 11 14:29:16 2024] RSP: 002b:00007ffc284a9ad8 EFLAGS: 00000246 ORIG_RAX: 00000000000000e8
[Mon Mar 11 14:29:16 2024] RAX: ffffffffffffffda RBX: 000055f87dd397c0 RCX: 00007fedb7d4e80a
[Mon Mar 11 14:29:16 2024] RDX: 0000000000000001 RSI: 00007ffc284a9b0c RDI: 0000000000000005
[Mon Mar 11 14:29:16 2024] RBP: 000055f87dd2aa20 R08: 00000000000f4155 R09: 0000000000000000
[Mon Mar 11 14:29:16 2024] R10: 000000000000ea60 R11: 0000000000000246 R12: 000055f87dd61190
[Mon Mar 11 14:29:16 2024] R13: 000000000000ea60 R14: 0000000000000026 R15: 000055f87dd5b760
[Mon Mar 11 14:29:16 2024] </TASK>
[Mon Mar 11 14:29:16 2024] task:smbd[10.87.18.2 state:S stack:0 pid:2341452 ppid:4757 flags:0x00000002
[Mon Mar 11 14:29:16 2024] Call Trace:
[Mon Mar 11 14:29:16 2024] <TASK>
[Mon Mar 11 14:29:16 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:16 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:16 2024] schedule_hrtimeout_range_clock+0x95/0x120
[Mon Mar 11 14:29:16 2024] ? __pfx_hrtimer_wakeup+0x10/0x10
[Mon Mar 11 14:29:16 2024] ep_poll+0x348/0x3b0
[Mon Mar 11 14:29:16 2024] ? __pfx_autoremove_wake_function+0x10/0x10
[Mon Mar 11 14:29:16 2024] do_epoll_wait+0xb1/0xd0
[Mon Mar 11 14:29:16 2024] __x64_sys_epoll_wait+0x60/0x100
[Mon Mar 11 14:29:16 2024] do_syscall_64+0x59/0x90
[Mon Mar 11 14:29:16 2024] ? syscall_exit_work+0x103/0x130
[Mon Mar 11 14:29:16 2024] ? syscall_exit_to_user_mode+0x22/0x40
[Mon Mar 11 14:29:16 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:16 2024] ? syscall_exit_work+0x103/0x130
[Mon Mar 11 14:29:16 2024] ? syscall_exit_to_user_mode+0x22/0x40
[Mon Mar 11 14:29:16 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:16 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:16 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:16 2024] entry_SYSCALL_64_after_hwframe+0x72/0xdc
[Mon Mar 11 14:29:16 2024] RIP: 0033:0x7fedb7d4e80a
[Mon Mar 11 14:29:16 2024] RSP: 002b:00007ffc284a9ad8 EFLAGS: 00000246 ORIG_RAX: 00000000000000e8
[Mon Mar 11 14:29:16 2024] RAX: ffffffffffffffda RBX: 000055f87dd397c0 RCX: 00007fedb7d4e80a
[Mon Mar 11 14:29:16 2024] RDX: 0000000000000001 RSI: 00007ffc284a9b0c RDI: 0000000000000005
[Mon Mar 11 14:29:16 2024] RBP: 000055f87dd2aa20 R08: 00000000000f41d7 R09: 0000000000000000
[Mon Mar 11 14:29:16 2024] R10: 000000000000ea60 R11: 0000000000000246 R12: 000055f87dd61190
[Mon Mar 11 14:29:16 2024] R13: 000000000000ea60 R14: 0000000000000026 R15: 000055f87dd5b760
[Mon Mar 11 14:29:16 2024] </TASK>
[Mon Mar 11 14:29:16 2024] task:smbd[10.40.157. state:S stack:0 pid:2341830 ppid:4757 flags:0x00000002
[Mon Mar 11 14:29:16 2024] Call Trace:
[Mon Mar 11 14:29:16 2024] <TASK>
[Mon Mar 11 14:29:16 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:16 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:16 2024] schedule_hrtimeout_range_clock+0x95/0x120
[Mon Mar 11 14:29:16 2024] ? __pfx_hrtimer_wakeup+0x10/0x10
[Mon Mar 11 14:29:16 2024] ep_poll+0x348/0x3b0
[Mon Mar 11 14:29:16 2024] ? __pfx_autoremove_wake_function+0x10/0x10
[Mon Mar 11 14:29:16 2024] do_epoll_wait+0xb1/0xd0
[Mon Mar 11 14:29:16 2024] __x64_sys_epoll_wait+0x60/0x100
[Mon Mar 11 14:29:16 2024] do_syscall_64+0x59/0x90
[Mon Mar 11 14:29:16 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:16 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:16 2024] ? exc_page_fault+0x62/0x150
[Mon Mar 11 14:29:16 2024] entry_SYSCALL_64_after_hwframe+0x72/0xdc
[Mon Mar 11 14:29:16 2024] RIP: 0033:0x7fedb7d4e84e
[Mon Mar 11 14:29:16 2024] RSP: 002b:00007ffc284a9ab0 EFLAGS: 00000293 ORIG_RAX: 00000000000000e8
[Mon Mar 11 14:29:16 2024] RAX: ffffffffffffffda RBX: 000055f87dd397c0 RCX: 00007fedb7d4e84e
[Mon Mar 11 14:29:16 2024] RDX: 0000000000000001 RSI: 00007ffc284a9b0c RDI: 0000000000000005
[Mon Mar 11 14:29:16 2024] RBP: 000055f87dd2aa20 R08: 0000000000000000 R09: 0000000000000000
[Mon Mar 11 14:29:16 2024] R10: 0000000000009086 R11: 0000000000000293 R12: 000055f87dd58980
[Mon Mar 11 14:29:16 2024] R13: 0000000000009086 R14: 0000000000000027 R15: 000055f87dd5b760
[Mon Mar 11 14:29:16 2024] </TASK>
[Mon Mar 11 14:29:16 2024] task:smbd[10.40.142. state:S stack:0 pid:2342339 ppid:4757 flags:0x00000002
[Mon Mar 11 14:29:16 2024] Call Trace:
[Mon Mar 11 14:29:16 2024] <TASK>
[Mon Mar 11 14:29:16 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:16 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:16 2024] schedule_hrtimeout_range_clock+0x95/0x120
[Mon Mar 11 14:29:16 2024] ? __pfx_hrtimer_wakeup+0x10/0x10
[Mon Mar 11 14:29:16 2024] ep_poll+0x348/0x3b0
[Mon Mar 11 14:29:16 2024] ? __pfx_autoremove_wake_function+0x10/0x10
[Mon Mar 11 14:29:16 2024] do_epoll_wait+0xb1/0xd0
[Mon Mar 11 14:29:16 2024] __x64_sys_epoll_wait+0x60/0x100
[Mon Mar 11 14:29:16 2024] do_syscall_64+0x59/0x90
[Mon Mar 11 14:29:16 2024] ? syscall_exit_work+0x103/0x130
[Mon Mar 11 14:29:16 2024] ? syscall_exit_to_user_mode+0x22/0x40
[Mon Mar 11 14:29:16 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:16 2024] entry_SYSCALL_64_after_hwframe+0x72/0xdc
[Mon Mar 11 14:29:16 2024] RIP: 0033:0x7fedb7d4e84e
[Mon Mar 11 14:29:16 2024] RSP: 002b:00007ffc284a9ab0 EFLAGS: 00000293 ORIG_RAX: 00000000000000e8
[Mon Mar 11 14:29:16 2024] RAX: ffffffffffffffda RBX: 000055f87dd397c0 RCX: 00007fedb7d4e84e
[Mon Mar 11 14:29:16 2024] RDX: 0000000000000001 RSI: 00007ffc284a9b0c RDI: 0000000000000005
[Mon Mar 11 14:29:16 2024] RBP: 000055f87dd2aa20 R08: 0000000000000000 R09: 0000000000000000
[Mon Mar 11 14:29:16 2024] R10: 000000000000ea60 R11: 0000000000000293 R12: 000055f87dd58980
[Mon Mar 11 14:29:16 2024] R13: 000000000000ea60 R14: 0000000000000027 R15: 000055f87dd5b760
[Mon Mar 11 14:29:16 2024] </TASK>
[Mon Mar 11 14:29:16 2024] task:smbd[10.87.18.3 state:S stack:0 pid:2342764 ppid:4757 flags:0x00000002
[Mon Mar 11 14:29:16 2024] Call Trace:
[Mon Mar 11 14:29:16 2024] <TASK>
[Mon Mar 11 14:29:16 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:16 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:16 2024] schedule_hrtimeout_range_clock+0x95/0x120
[Mon Mar 11 14:29:16 2024] ? __pfx_hrtimer_wakeup+0x10/0x10
[Mon Mar 11 14:29:16 2024] ep_poll+0x348/0x3b0
[Mon Mar 11 14:29:16 2024] ? __pfx_autoremove_wake_function+0x10/0x10
[Mon Mar 11 14:29:16 2024] do_epoll_wait+0xb1/0xd0
[Mon Mar 11 14:29:16 2024] __x64_sys_epoll_wait+0x60/0x100
[Mon Mar 11 14:29:16 2024] do_syscall_64+0x59/0x90
[Mon Mar 11 14:29:16 2024] ? syscall_exit_work+0x103/0x130
[Mon Mar 11 14:29:16 2024] ? syscall_exit_to_user_mode+0x22/0x40
[Mon Mar 11 14:29:16 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:16 2024] ? syscall_exit_work+0x103/0x130
[Mon Mar 11 14:29:16 2024] ? syscall_exit_to_user_mode+0x22/0x40
[Mon Mar 11 14:29:16 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:16 2024] ? __irq_exit_rcu+0x46/0xc0
[Mon Mar 11 14:29:16 2024] ? sysvec_apic_timer_interrupt+0x3c/0x90
[Mon Mar 11 14:29:16 2024] entry_SYSCALL_64_after_hwframe+0x72/0xdc
[Mon Mar 11 14:29:16 2024] RIP: 0033:0x7fedb7d4e84e
[Mon Mar 11 14:29:16 2024] RSP: 002b:00007ffc284a9ab0 EFLAGS: 00000293 ORIG_RAX: 00000000000000e8
[Mon Mar 11 14:29:16 2024] RAX: ffffffffffffffda RBX: 000055f87dd397c0 RCX: 00007fedb7d4e84e
[Mon Mar 11 14:29:16 2024] RDX: 0000000000000001 RSI: 00007ffc284a9b0c RDI: 0000000000000005
[Mon Mar 11 14:29:16 2024] RBP: 000055f87dd2aa20 R08: 0000000000000000 R09: 0000000000000000
[Mon Mar 11 14:29:16 2024] R10: 000000000000ea60 R11: 0000000000000293 R12: 000055f87dd61190
[Mon Mar 11 14:29:16 2024] R13: 000000000000ea60 R14: 0000000000000026 R15: 000055f87dd5b760
[Mon Mar 11 14:29:16 2024] </TASK>
[Mon Mar 11 14:29:16 2024] task:smbd[10.87.27.1 state:S stack:0 pid:2343759 ppid:4757 flags:0x00000002
[Mon Mar 11 14:29:16 2024] Call Trace:
[Mon Mar 11 14:29:16 2024] <TASK>
[Mon Mar 11 14:29:16 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:16 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:16 2024] schedule_hrtimeout_range_clock+0x95/0x120
[Mon Mar 11 14:29:16 2024] ? __pfx_hrtimer_wakeup+0x10/0x10
[Mon Mar 11 14:29:16 2024] ep_poll+0x348/0x3b0
[Mon Mar 11 14:29:16 2024] ? __pfx_autoremove_wake_function+0x10/0x10
[Mon Mar 11 14:29:16 2024] do_epoll_wait+0xb1/0xd0
[Mon Mar 11 14:29:16 2024] __x64_sys_epoll_wait+0x60/0x100
[Mon Mar 11 14:29:16 2024] do_syscall_64+0x59/0x90
[Mon Mar 11 14:29:16 2024] ? syscall_exit_work+0x103/0x130
[Mon Mar 11 14:29:16 2024] ? syscall_exit_to_user_mode+0x22/0x40
[Mon Mar 11 14:29:16 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:16 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:16 2024] ? exc_page_fault+0x62/0x150
[Mon Mar 11 14:29:16 2024] entry_SYSCALL_64_after_hwframe+0x72/0xdc
[Mon Mar 11 14:29:16 2024] RIP: 0033:0x7fedb7d4e84e
[Mon Mar 11 14:29:16 2024] RSP: 002b:00007ffc284a9ab0 EFLAGS: 00000293 ORIG_RAX: 00000000000000e8
[Mon Mar 11 14:29:16 2024] RAX: ffffffffffffffda RBX: 000055f87dd397c0 RCX: 00007fedb7d4e84e
[Mon Mar 11 14:29:16 2024] RDX: 0000000000000001 RSI: 00007ffc284a9b0c RDI: 0000000000000005
[Mon Mar 11 14:29:16 2024] RBP: 000055f87dd2aa20 R08: 0000000000000000 R09: 0000000000000000
[Mon Mar 11 14:29:16 2024] R10: 000000000000e91b R11: 0000000000000293 R12: 000055f87dd61190
[Mon Mar 11 14:29:16 2024] R13: 000000000000e91b R14: 0000000000000026 R15: 000055f87dd5b760
[Mon Mar 11 14:29:16 2024] </TASK>
[Mon Mar 11 14:29:16 2024] task:smbd[10.87.27.1 state:S stack:0 pid:2343951 ppid:4757 flags:0x00000002
[Mon Mar 11 14:29:16 2024] Call Trace:
[Mon Mar 11 14:29:16 2024] <TASK>
[Mon Mar 11 14:29:16 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:16 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:16 2024] schedule_hrtimeout_range_clock+0x95/0x120
[Mon Mar 11 14:29:16 2024] ? __pfx_hrtimer_wakeup+0x10/0x10
[Mon Mar 11 14:29:16 2024] ep_poll+0x348/0x3b0
[Mon Mar 11 14:29:16 2024] ? __pfx_autoremove_wake_function+0x10/0x10
[Mon Mar 11 14:29:16 2024] do_epoll_wait+0xb1/0xd0
[Mon Mar 11 14:29:16 2024] __x64_sys_epoll_wait+0x60/0x100
[Mon Mar 11 14:29:16 2024] do_syscall_64+0x59/0x90
[Mon Mar 11 14:29:16 2024] ? syscall_exit_to_user_mode+0x22/0x40
[Mon Mar 11 14:29:16 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:16 2024] ? exc_page_fault+0x62/0x150
[Mon Mar 11 14:29:16 2024] entry_SYSCALL_64_after_hwframe+0x72/0xdc
[Mon Mar 11 14:29:16 2024] RIP: 0033:0x7fedb7d4e84e
[Mon Mar 11 14:29:16 2024] RSP: 002b:00007ffc284a9ab0 EFLAGS: 00000293 ORIG_RAX: 00000000000000e8
[Mon Mar 11 14:29:16 2024] RAX: ffffffffffffffda RBX: 000055f87dd397c0 RCX: 00007fedb7d4e84e
[Mon Mar 11 14:29:16 2024] RDX: 0000000000000001 RSI: 00007ffc284a9b0c RDI: 0000000000000005
[Mon Mar 11 14:29:16 2024] RBP: 000055f87dd2aa20 R08: 0000000000000000 R09: 0000000000000000
[Mon Mar 11 14:29:16 2024] R10: 000000000000ea60 R11: 0000000000000293 R12: 000055f87dd61190
[Mon Mar 11 14:29:16 2024] R13: 000000000000ea60 R14: 0000000000000026 R15: 000055f87dd5b760
[Mon Mar 11 14:29:16 2024] </TASK>
[Mon Mar 11 14:29:16 2024] task:kworker/35:0 state:I stack:0 pid:2344315 ppid:2 flags:0x00004000
[Mon Mar 11 14:29:16 2024] Workqueue: 0x0 (mm_percpu_wq)
[Mon Mar 11 14:29:16 2024] Call Trace:
[Mon Mar 11 14:29:16 2024] <TASK>
[Mon Mar 11 14:29:16 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:16 2024] ? __pfx_worker_thread+0x10/0x10
[Mon Mar 11 14:29:16 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:16 2024] worker_thread+0xbb/0x3a0
[Mon Mar 11 14:29:16 2024] ? __pfx_worker_thread+0x10/0x10
[Mon Mar 11 14:29:16 2024] kthread+0xdd/0x100
[Mon Mar 11 14:29:16 2024] ? __pfx_kthread+0x10/0x10
[Mon Mar 11 14:29:16 2024] ret_from_fork+0x29/0x50
[Mon Mar 11 14:29:16 2024] </TASK>
[Mon Mar 11 14:29:16 2024] task:kworker/25:1 state:I stack:0 pid:2344356 ppid:2 flags:0x00004000
[Mon Mar 11 14:29:16 2024] Workqueue: 0x0 (mm_percpu_wq)
[Mon Mar 11 14:29:16 2024] Call Trace:
[Mon Mar 11 14:29:16 2024] <TASK>
[Mon Mar 11 14:29:16 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:16 2024] ? __pfx_worker_thread+0x10/0x10
[Mon Mar 11 14:29:16 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:16 2024] worker_thread+0xbb/0x3a0
[Mon Mar 11 14:29:16 2024] ? __pfx_worker_thread+0x10/0x10
[Mon Mar 11 14:29:16 2024] kthread+0xdd/0x100
[Mon Mar 11 14:29:16 2024] ? __pfx_kthread+0x10/0x10
[Mon Mar 11 14:29:16 2024] ret_from_fork+0x29/0x50
[Mon Mar 11 14:29:16 2024] </TASK>
[Mon Mar 11 14:29:16 2024] task:kworker/32:0 state:I stack:0 pid:2344368 ppid:2 flags:0x00004000
[Mon Mar 11 14:29:16 2024] Workqueue: 0x0 (events_power_efficient)
[Mon Mar 11 14:29:16 2024] Call Trace:
[Mon Mar 11 14:29:16 2024] <TASK>
[Mon Mar 11 14:29:16 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:16 2024] ? neigh_managed_work+0x9a/0xb0
[Mon Mar 11 14:29:16 2024] ? __pfx_worker_thread+0x10/0x10
[Mon Mar 11 14:29:16 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:16 2024] worker_thread+0xbb/0x3a0
[Mon Mar 11 14:29:16 2024] ? __pfx_worker_thread+0x10/0x10
[Mon Mar 11 14:29:16 2024] kthread+0xdd/0x100
[Mon Mar 11 14:29:16 2024] ? __pfx_kthread+0x10/0x10
[Mon Mar 11 14:29:16 2024] ret_from_fork+0x29/0x50
[Mon Mar 11 14:29:16 2024] </TASK>
[Mon Mar 11 14:29:16 2024] task:smbd[10.87.27.1 state:S stack:0 pid:2344554 ppid:4757 flags:0x00000002
[Mon Mar 11 14:29:16 2024] Call Trace:
[Mon Mar 11 14:29:16 2024] <TASK>
[Mon Mar 11 14:29:16 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:16 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:16 2024] schedule_hrtimeout_range_clock+0x95/0x120
[Mon Mar 11 14:29:16 2024] ? __pfx_hrtimer_wakeup+0x10/0x10
[Mon Mar 11 14:29:16 2024] ep_poll+0x348/0x3b0
[Mon Mar 11 14:29:16 2024] ? __pfx_autoremove_wake_function+0x10/0x10
[Mon Mar 11 14:29:16 2024] do_epoll_wait+0xb1/0xd0
[Mon Mar 11 14:29:16 2024] __x64_sys_epoll_wait+0x60/0x100
[Mon Mar 11 14:29:16 2024] do_syscall_64+0x59/0x90
[Mon Mar 11 14:29:16 2024] ? syscall_exit_work+0x103/0x130
[Mon Mar 11 14:29:16 2024] ? syscall_exit_to_user_mode+0x22/0x40
[Mon Mar 11 14:29:16 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:16 2024] entry_SYSCALL_64_after_hwframe+0x72/0xdc
[Mon Mar 11 14:29:16 2024] RIP: 0033:0x7fedb7d4e84e
[Mon Mar 11 14:29:16 2024] RSP: 002b:00007ffc284a9ab0 EFLAGS: 00000293 ORIG_RAX: 00000000000000e8
[Mon Mar 11 14:29:16 2024] RAX: ffffffffffffffda RBX: 000055f87dd397c0 RCX: 00007fedb7d4e84e
[Mon Mar 11 14:29:16 2024] RDX: 0000000000000001 RSI: 00007ffc284a9b0c RDI: 0000000000000005
[Mon Mar 11 14:29:16 2024] RBP: 000055f87dd2aa20 R08: 0000000000000000 R09: 0000000000000000
[Mon Mar 11 14:29:16 2024] R10: 000000000000ea60 R11: 0000000000000293 R12: 000055f87dd61190
[Mon Mar 11 14:29:16 2024] R13: 000000000000ea60 R14: 0000000000000026 R15: 000055f87dd5b760
[Mon Mar 11 14:29:16 2024] </TASK>
[Mon Mar 11 14:29:16 2024] task:smbd[10.87.27.1 state:S stack:0 pid:2344564 ppid:4757 flags:0x00000002
[Mon Mar 11 14:29:16 2024] Call Trace:
[Mon Mar 11 14:29:16 2024] <TASK>
[Mon Mar 11 14:29:16 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:16 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:16 2024] schedule_hrtimeout_range_clock+0x95/0x120
[Mon Mar 11 14:29:16 2024] ? __pfx_hrtimer_wakeup+0x10/0x10
[Mon Mar 11 14:29:16 2024] ep_poll+0x348/0x3b0
[Mon Mar 11 14:29:16 2024] ? __pfx_autoremove_wake_function+0x10/0x10
[Mon Mar 11 14:29:16 2024] do_epoll_wait+0xb1/0xd0
[Mon Mar 11 14:29:16 2024] __x64_sys_epoll_wait+0x60/0x100
[Mon Mar 11 14:29:16 2024] do_syscall_64+0x59/0x90
[Mon Mar 11 14:29:16 2024] ? syscall_exit_work+0x103/0x130
[Mon Mar 11 14:29:16 2024] ? syscall_exit_to_user_mode+0x22/0x40
[Mon Mar 11 14:29:16 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:16 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:16 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:16 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:16 2024] entry_SYSCALL_64_after_hwframe+0x72/0xdc
[Mon Mar 11 14:29:16 2024] RIP: 0033:0x7fedb7d4e80a
[Mon Mar 11 14:29:16 2024] RSP: 002b:00007ffc284a9ad8 EFLAGS: 00000246 ORIG_RAX: 00000000000000e8
[Mon Mar 11 14:29:16 2024] RAX: ffffffffffffffda RBX: 000055f87dd397c0 RCX: 00007fedb7d4e80a
[Mon Mar 11 14:29:16 2024] RDX: 0000000000000001 RSI: 00007ffc284a9b0c RDI: 0000000000000005
[Mon Mar 11 14:29:16 2024] RBP: 000055f87dd2aa20 R08: 00000000000f41dd R09: 0000000000000000
[Mon Mar 11 14:29:16 2024] R10: 000000000000ea60 R11: 0000000000000246 R12: 000055f87dd61190
[Mon Mar 11 14:29:16 2024] R13: 000000000000ea60 R14: 0000000000000026 R15: 000055f87dd5b760
[Mon Mar 11 14:29:16 2024] </TASK>
[Mon Mar 11 14:29:16 2024] task:smbd[10.87.29.9 state:S stack:0 pid:2346134 ppid:4757 flags:0x00000002
[Mon Mar 11 14:29:16 2024] Call Trace:
[Mon Mar 11 14:29:16 2024] <TASK>
[Mon Mar 11 14:29:16 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:16 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:16 2024] schedule_hrtimeout_range_clock+0x95/0x120
[Mon Mar 11 14:29:16 2024] ? __pfx_hrtimer_wakeup+0x10/0x10
[Mon Mar 11 14:29:16 2024] ep_poll+0x348/0x3b0
[Mon Mar 11 14:29:16 2024] ? __pfx_autoremove_wake_function+0x10/0x10
[Mon Mar 11 14:29:16 2024] do_epoll_wait+0xb1/0xd0
[Mon Mar 11 14:29:16 2024] __x64_sys_epoll_wait+0x60/0x100
[Mon Mar 11 14:29:16 2024] do_syscall_64+0x59/0x90
[Mon Mar 11 14:29:16 2024] ? audit_reset_context.part.0.constprop.0+0x273/0x2e0
[Mon Mar 11 14:29:16 2024] ? syscall_exit_work+0x103/0x130
[Mon Mar 11 14:29:16 2024] ? syscall_exit_to_user_mode+0x22/0x40
[Mon Mar 11 14:29:16 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:16 2024] ? syscall_exit_to_user_mode+0x22/0x40
[Mon Mar 11 14:29:16 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:16 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:16 2024] ? syscall_exit_to_user_mode+0x22/0x40
[Mon Mar 11 14:29:16 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:16 2024] ? __irq_exit_rcu+0x46/0xc0
[Mon Mar 11 14:29:16 2024] ? sysvec_apic_timer_interrupt+0x3c/0x90
[Mon Mar 11 14:29:16 2024] entry_SYSCALL_64_after_hwframe+0x72/0xdc
[Mon Mar 11 14:29:16 2024] RIP: 0033:0x7fedb7d4e84e
[Mon Mar 11 14:29:16 2024] RSP: 002b:00007ffc284a9ab0 EFLAGS: 00000293 ORIG_RAX: 00000000000000e8
[Mon Mar 11 14:29:16 2024] RAX: ffffffffffffffda RBX: 000055f87dd397c0 RCX: 00007fedb7d4e84e
[Mon Mar 11 14:29:16 2024] RDX: 0000000000000001 RSI: 00007ffc284a9b0c RDI: 0000000000000005
[Mon Mar 11 14:29:16 2024] RBP: 000055f87dd2aa20 R08: 0000000000000000 R09: 0000000000000000
[Mon Mar 11 14:29:16 2024] R10: 000000000000ea60 R11: 0000000000000293 R12: 000055f87dd61190
[Mon Mar 11 14:29:16 2024] R13: 000000000000ea60 R14: 0000000000000026 R15: 000055f87dd5b760
[Mon Mar 11 14:29:16 2024] </TASK>
[Mon Mar 11 14:29:16 2024] task:smbd[10.87.28.5 state:S stack:0 pid:2346299 ppid:4757 flags:0x00000002
[Mon Mar 11 14:29:16 2024] Call Trace:
[Mon Mar 11 14:29:16 2024] <TASK>
[Mon Mar 11 14:29:16 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:16 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:16 2024] schedule_hrtimeout_range_clock+0x95/0x120
[Mon Mar 11 14:29:16 2024] ? __pfx_hrtimer_wakeup+0x10/0x10
[Mon Mar 11 14:29:16 2024] ep_poll+0x348/0x3b0
[Mon Mar 11 14:29:16 2024] ? __pfx_autoremove_wake_function+0x10/0x10
[Mon Mar 11 14:29:16 2024] do_epoll_wait+0xb1/0xd0
[Mon Mar 11 14:29:16 2024] __x64_sys_epoll_wait+0x60/0x100
[Mon Mar 11 14:29:16 2024] do_syscall_64+0x59/0x90
[Mon Mar 11 14:29:16 2024] ? syscall_exit_to_user_mode+0x22/0x40
[Mon Mar 11 14:29:16 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:16 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:16 2024] ? exc_page_fault+0x62/0x150
[Mon Mar 11 14:29:16 2024] entry_SYSCALL_64_after_hwframe+0x72/0xdc
[Mon Mar 11 14:29:16 2024] RIP: 0033:0x7fedb7d4e80a
[Mon Mar 11 14:29:16 2024] RSP: 002b:00007ffc284a9ad8 EFLAGS: 00000246 ORIG_RAX: 00000000000000e8
[Mon Mar 11 14:29:16 2024] RAX: ffffffffffffffda RBX: 000055f87dd397c0 RCX: 00007fedb7d4e80a
[Mon Mar 11 14:29:16 2024] RDX: 0000000000000001 RSI: 00007ffc284a9b0c RDI: 0000000000000005
[Mon Mar 11 14:29:16 2024] RBP: 000055f87dd2aa20 R08: 00000000000f41d4 R09: 0000000000000000
[Mon Mar 11 14:29:16 2024] R10: 000000000000ea60 R11: 0000000000000246 R12: 000055f87dd61190
[Mon Mar 11 14:29:16 2024] R13: 000000000000ea60 R14: 0000000000000026 R15: 000055f87dd5b760
[Mon Mar 11 14:29:16 2024] </TASK>
[Mon Mar 11 14:29:16 2024] task:smbd[10.87.29.8 state:S stack:0 pid:2346897 ppid:4757 flags:0x00000002
[Mon Mar 11 14:29:16 2024] Call Trace:
[Mon Mar 11 14:29:16 2024] <TASK>
[Mon Mar 11 14:29:16 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:16 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:16 2024] schedule_hrtimeout_range_clock+0x95/0x120
[Mon Mar 11 14:29:16 2024] ? __pfx_hrtimer_wakeup+0x10/0x10
[Mon Mar 11 14:29:16 2024] ep_poll+0x348/0x3b0
[Mon Mar 11 14:29:16 2024] ? __pfx_autoremove_wake_function+0x10/0x10
[Mon Mar 11 14:29:16 2024] do_epoll_wait+0xb1/0xd0
[Mon Mar 11 14:29:16 2024] __x64_sys_epoll_wait+0x60/0x100
[Mon Mar 11 14:29:16 2024] do_syscall_64+0x59/0x90
[Mon Mar 11 14:29:16 2024] ? syscall_exit_work+0x103/0x130
[Mon Mar 11 14:29:16 2024] ? syscall_exit_to_user_mode+0x22/0x40
[Mon Mar 11 14:29:16 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:16 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:16 2024] entry_SYSCALL_64_after_hwframe+0x72/0xdc
[Mon Mar 11 14:29:16 2024] RIP: 0033:0x7fedb7d4e80a
[Mon Mar 11 14:29:16 2024] RSP: 002b:00007ffc284a9ad8 EFLAGS: 00000246 ORIG_RAX: 00000000000000e8
[Mon Mar 11 14:29:16 2024] RAX: ffffffffffffffda RBX: 000055f87dd397c0 RCX: 00007fedb7d4e80a
[Mon Mar 11 14:29:16 2024] RDX: 0000000000000001 RSI: 00007ffc284a9b0c RDI: 0000000000000005
[Mon Mar 11 14:29:16 2024] RBP: 000055f87dd2aa20 R08: 00000000000f41cc R09: 0000000000000000
[Mon Mar 11 14:29:16 2024] R10: 000000000000ea60 R11: 0000000000000246 R12: 000055f87dd61190
[Mon Mar 11 14:29:16 2024] R13: 000000000000ea60 R14: 0000000000000026 R15: 000055f87dd5b760
[Mon Mar 11 14:29:16 2024] </TASK>
[Mon Mar 11 14:29:16 2024] task:smbd[10.87.18.2 state:S stack:0 pid:2347077 ppid:4757 flags:0x00000002
[Mon Mar 11 14:29:16 2024] Call Trace:
[Mon Mar 11 14:29:16 2024] <TASK>
[Mon Mar 11 14:29:16 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:16 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:16 2024] schedule_hrtimeout_range_clock+0x95/0x120
[Mon Mar 11 14:29:16 2024] ? __pfx_hrtimer_wakeup+0x10/0x10
[Mon Mar 11 14:29:16 2024] ep_poll+0x348/0x3b0
[Mon Mar 11 14:29:16 2024] ? __pfx_autoremove_wake_function+0x10/0x10
[Mon Mar 11 14:29:16 2024] do_epoll_wait+0xb1/0xd0
[Mon Mar 11 14:29:16 2024] __x64_sys_epoll_wait+0x60/0x100
[Mon Mar 11 14:29:16 2024] do_syscall_64+0x59/0x90
[Mon Mar 11 14:29:16 2024] ? audit_reset_context.part.0.constprop.0+0x273/0x2e0
[Mon Mar 11 14:29:16 2024] ? syscall_exit_work+0x103/0x130
[Mon Mar 11 14:29:16 2024] ? syscall_exit_to_user_mode+0x22/0x40
[Mon Mar 11 14:29:16 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:16 2024] ? syscall_exit_to_user_mode+0x22/0x40
[Mon Mar 11 14:29:16 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:16 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:16 2024] ? syscall_exit_to_user_mode+0x22/0x40
[Mon Mar 11 14:29:16 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:16 2024] ? __irq_exit_rcu+0x46/0xc0
[Mon Mar 11 14:29:16 2024] ? common_interrupt+0x43/0xa0
[Mon Mar 11 14:29:16 2024] entry_SYSCALL_64_after_hwframe+0x72/0xdc
[Mon Mar 11 14:29:16 2024] RIP: 0033:0x7fedb7d4e84e
[Mon Mar 11 14:29:16 2024] RSP: 002b:00007ffc284a9ab0 EFLAGS: 00000293 ORIG_RAX: 00000000000000e8
[Mon Mar 11 14:29:16 2024] RAX: ffffffffffffffda RBX: 000055f87dd397c0 RCX: 00007fedb7d4e84e
[Mon Mar 11 14:29:16 2024] RDX: 0000000000000001 RSI: 00007ffc284a9b0c RDI: 0000000000000005
[Mon Mar 11 14:29:16 2024] RBP: 000055f87dd2aa20 R08: 0000000000000000 R09: 0000000000000000
[Mon Mar 11 14:29:16 2024] R10: 000000000000ea60 R11: 0000000000000293 R12: 000055f87dd61190
[Mon Mar 11 14:29:16 2024] R13: 000000000000ea60 R14: 0000000000000026 R15: 000055f87dd5b760
[Mon Mar 11 14:29:16 2024] </TASK>
[Mon Mar 11 14:29:16 2024] task:smbd[10.87.18.2 state:S stack:0 pid:2347080 ppid:4757 flags:0x00000002
[Mon Mar 11 14:29:16 2024] Call Trace:
[Mon Mar 11 14:29:16 2024] <TASK>
[Mon Mar 11 14:29:16 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:16 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:16 2024] schedule_hrtimeout_range_clock+0x95/0x120
[Mon Mar 11 14:29:16 2024] ? __pfx_hrtimer_wakeup+0x10/0x10
[Mon Mar 11 14:29:16 2024] ep_poll+0x348/0x3b0
[Mon Mar 11 14:29:16 2024] ? __pfx_autoremove_wake_function+0x10/0x10
[Mon Mar 11 14:29:16 2024] do_epoll_wait+0xb1/0xd0
[Mon Mar 11 14:29:16 2024] __x64_sys_epoll_wait+0x60/0x100
[Mon Mar 11 14:29:16 2024] do_syscall_64+0x59/0x90
[Mon Mar 11 14:29:16 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:16 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:16 2024] ? exit_to_user_mode_prepare+0xec/0x100
[Mon Mar 11 14:29:16 2024] ? syscall_exit_to_user_mode+0x22/0x40
[Mon Mar 11 14:29:16 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:16 2024] entry_SYSCALL_64_after_hwframe+0x72/0xdc
[Mon Mar 11 14:29:16 2024] RIP: 0033:0x7fedb7d4e84e
[Mon Mar 11 14:29:16 2024] RSP: 002b:00007ffc284a9ab0 EFLAGS: 00000293 ORIG_RAX: 00000000000000e8
[Mon Mar 11 14:29:16 2024] RAX: ffffffffffffffda RBX: 000055f87dd397c0 RCX: 00007fedb7d4e84e
[Mon Mar 11 14:29:16 2024] RDX: 0000000000000001 RSI: 00007ffc284a9b0c RDI: 0000000000000005
[Mon Mar 11 14:29:16 2024] RBP: 000055f87dd2aa20 R08: 0000000000000000 R09: 0000000000000000
[Mon Mar 11 14:29:16 2024] R10: 000000000000ea60 R11: 0000000000000293 R12: 000055f87dd61190
[Mon Mar 11 14:29:16 2024] R13: 000000000000ea60 R14: 0000000000000026 R15: 000055f87dd5b760
[Mon Mar 11 14:29:16 2024] </TASK>
[Mon Mar 11 14:29:16 2024] task:smbd[10.87.18.2 state:S stack:0 pid:2347082 ppid:4757 flags:0x00000002
[Mon Mar 11 14:29:16 2024] Call Trace:
[Mon Mar 11 14:29:16 2024] <TASK>
[Mon Mar 11 14:29:16 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:16 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:16 2024] schedule_hrtimeout_range_clock+0x95/0x120
[Mon Mar 11 14:29:16 2024] ? __pfx_hrtimer_wakeup+0x10/0x10
[Mon Mar 11 14:29:16 2024] ep_poll+0x348/0x3b0
[Mon Mar 11 14:29:16 2024] ? __pfx_autoremove_wake_function+0x10/0x10
[Mon Mar 11 14:29:16 2024] do_epoll_wait+0xb1/0xd0
[Mon Mar 11 14:29:16 2024] __x64_sys_epoll_wait+0x60/0x100
[Mon Mar 11 14:29:16 2024] do_syscall_64+0x59/0x90
[Mon Mar 11 14:29:16 2024] ? syscall_exit_work+0x103/0x130
[Mon Mar 11 14:29:16 2024] ? syscall_exit_to_user_mode+0x22/0x40
[Mon Mar 11 14:29:16 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:16 2024] entry_SYSCALL_64_after_hwframe+0x72/0xdc
[Mon Mar 11 14:29:16 2024] RIP: 0033:0x7fedb7d4e80a
[Mon Mar 11 14:29:16 2024] RSP: 002b:00007ffc284a9ad8 EFLAGS: 00000246 ORIG_RAX: 00000000000000e8
[Mon Mar 11 14:29:16 2024] RAX: ffffffffffffffda RBX: 000055f87dd397c0 RCX: 00007fedb7d4e80a
[Mon Mar 11 14:29:16 2024] RDX: 0000000000000001 RSI: 00007ffc284a9b0c RDI: 0000000000000005
[Mon Mar 11 14:29:16 2024] RBP: 000055f87dd2aa20 R08: 00000000000f41eb R09: 0000000000000000
[Mon Mar 11 14:29:16 2024] R10: 000000000000ea60 R11: 0000000000000246 R12: 000055f87dd61190
[Mon Mar 11 14:29:16 2024] R13: 000000000000ea60 R14: 0000000000000026 R15: 000055f87dd5b760
[Mon Mar 11 14:29:16 2024] </TASK>
[Mon Mar 11 14:29:16 2024] task:smbd[10.33.88.9 state:S stack:0 pid:2347580 ppid:4757 flags:0x00000002
[Mon Mar 11 14:29:16 2024] Call Trace:
[Mon Mar 11 14:29:16 2024] <TASK>
[Mon Mar 11 14:29:16 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:16 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:16 2024] schedule_hrtimeout_range_clock+0x95/0x120
[Mon Mar 11 14:29:16 2024] ? __pfx_hrtimer_wakeup+0x10/0x10
[Mon Mar 11 14:29:16 2024] ep_poll+0x348/0x3b0
[Mon Mar 11 14:29:16 2024] ? __pfx_autoremove_wake_function+0x10/0x10
[Mon Mar 11 14:29:16 2024] do_epoll_wait+0xb1/0xd0
[Mon Mar 11 14:29:16 2024] __x64_sys_epoll_wait+0x60/0x100
[Mon Mar 11 14:29:16 2024] do_syscall_64+0x59/0x90
[Mon Mar 11 14:29:16 2024] ? audit_reset_context.part.0.constprop.0+0x273/0x2e0
[Mon Mar 11 14:29:16 2024] ? syscall_exit_work+0x103/0x130
[Mon Mar 11 14:29:16 2024] ? syscall_exit_to_user_mode+0x22/0x40
[Mon Mar 11 14:29:16 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:16 2024] ? exc_page_fault+0x62/0x150
[Mon Mar 11 14:29:16 2024] entry_SYSCALL_64_after_hwframe+0x72/0xdc
[Mon Mar 11 14:29:16 2024] RIP: 0033:0x7fedb7d4e80a
[Mon Mar 11 14:29:16 2024] RSP: 002b:00007ffc284a9ad8 EFLAGS: 00000246 ORIG_RAX: 00000000000000e8
[Mon Mar 11 14:29:16 2024] RAX: ffffffffffffffda RBX: 000055f87dd397c0 RCX: 00007fedb7d4e80a
[Mon Mar 11 14:29:16 2024] RDX: 0000000000000001 RSI: 00007ffc284a9b0c RDI: 0000000000000005
[Mon Mar 11 14:29:16 2024] RBP: 000055f87dd2aa20 R08: 00000000000f4096 R09: 0000000000000000
[Mon Mar 11 14:29:16 2024] R10: 000000000000ea60 R11: 0000000000000246 R12: 000055f87dd61190
[Mon Mar 11 14:29:16 2024] R13: 000000000000ea60 R14: 0000000000000026 R15: 000055f87dd5b760
[Mon Mar 11 14:29:16 2024] </TASK>
[Mon Mar 11 14:29:16 2024] task:smbd[10.87.27.1 state:S stack:0 pid:2347638 ppid:4757 flags:0x00000002
[Mon Mar 11 14:29:16 2024] Call Trace:
[Mon Mar 11 14:29:16 2024] <TASK>
[Mon Mar 11 14:29:16 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:16 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:16 2024] schedule_hrtimeout_range_clock+0x95/0x120
[Mon Mar 11 14:29:16 2024] ? __pfx_hrtimer_wakeup+0x10/0x10
[Mon Mar 11 14:29:16 2024] ep_poll+0x348/0x3b0
[Mon Mar 11 14:29:16 2024] ? __pfx_autoremove_wake_function+0x10/0x10
[Mon Mar 11 14:29:16 2024] do_epoll_wait+0xb1/0xd0
[Mon Mar 11 14:29:16 2024] __x64_sys_epoll_wait+0x60/0x100
[Mon Mar 11 14:29:16 2024] do_syscall_64+0x59/0x90
[Mon Mar 11 14:29:16 2024] ? syscall_exit_work+0x103/0x130
[Mon Mar 11 14:29:16 2024] ? syscall_exit_to_user_mode+0x22/0x40
[Mon Mar 11 14:29:16 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:16 2024] ? exit_to_user_mode_prepare+0xb6/0x100
[Mon Mar 11 14:29:16 2024] ? syscall_exit_to_user_mode+0x22/0x40
[Mon Mar 11 14:29:16 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:16 2024] entry_SYSCALL_64_after_hwframe+0x72/0xdc
[Mon Mar 11 14:29:16 2024] RIP: 0033:0x7fedb7d4e84e
[Mon Mar 11 14:29:16 2024] RSP: 002b:00007ffc284a9ab0 EFLAGS: 00000293 ORIG_RAX: 00000000000000e8
[Mon Mar 11 14:29:16 2024] RAX: ffffffffffffffda RBX: 000055f87dd397c0 RCX: 00007fedb7d4e84e
[Mon Mar 11 14:29:16 2024] RDX: 0000000000000001 RSI: 00007ffc284a9b0c RDI: 0000000000000005
[Mon Mar 11 14:29:16 2024] RBP: 000055f87dd2aa20 R08: 0000000000000000 R09: 0000000000000000
[Mon Mar 11 14:29:16 2024] R10: 000000000000b25e R11: 0000000000000293 R12: 000055f87dd61190
[Mon Mar 11 14:29:16 2024] R13: 000000000000b25e R14: 0000000000000026 R15: 000055f87dd5b760
[Mon Mar 11 14:29:16 2024] </TASK>
[Mon Mar 11 14:29:16 2024] task:rpcd_winreg state:S stack:0 pid:2347737 ppid:2339807 flags:0x00000002
[Mon Mar 11 14:29:16 2024] Call Trace:
[Mon Mar 11 14:29:16 2024] <TASK>
[Mon Mar 11 14:29:16 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:16 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:16 2024] schedule_hrtimeout_range_clock+0x95/0x120
[Mon Mar 11 14:29:16 2024] ? __pfx_hrtimer_wakeup+0x10/0x10
[Mon Mar 11 14:29:16 2024] ep_poll+0x348/0x3b0
[Mon Mar 11 14:29:16 2024] ? __pfx_autoremove_wake_function+0x10/0x10
[Mon Mar 11 14:29:16 2024] do_epoll_wait+0xb1/0xd0
[Mon Mar 11 14:29:16 2024] __x64_sys_epoll_wait+0x60/0x100
[Mon Mar 11 14:29:16 2024] do_syscall_64+0x59/0x90
[Mon Mar 11 14:29:16 2024] ? exit_to_user_mode_prepare+0xb6/0x100
[Mon Mar 11 14:29:16 2024] ? syscall_exit_to_user_mode+0x22/0x40
[Mon Mar 11 14:29:16 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:16 2024] ? syscall_exit_to_user_mode+0x22/0x40
[Mon Mar 11 14:29:16 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:16 2024] entry_SYSCALL_64_after_hwframe+0x72/0xdc
[Mon Mar 11 14:29:16 2024] RIP: 0033:0x7fafead4e80a
[Mon Mar 11 14:29:16 2024] RSP: 002b:00007ffeb54e23c8 EFLAGS: 00000246 ORIG_RAX: 00000000000000e8
[Mon Mar 11 14:29:16 2024] RAX: ffffffffffffffda RBX: 0000563370d7e770 RCX: 00007fafead4e80a
[Mon Mar 11 14:29:16 2024] RDX: 0000000000000001 RSI: 00007ffeb54e23fc RDI: 0000000000000003
[Mon Mar 11 14:29:16 2024] RBP: 0000563370d6f2e0 R08: 0000563370dc2220 R09: 000000000000e0de
[Mon Mar 11 14:29:16 2024] R10: 0000000000007530 R11: 0000000000000246 R12: 0000563370d7e800
[Mon Mar 11 14:29:16 2024] R13: 0000000000007530 R14: 00007fafeb771db0 R15: 0000000000000000
[Mon Mar 11 14:29:16 2024] </TASK>
[Mon Mar 11 14:29:16 2024] task:smbd[10.87.27.1 state:S stack:0 pid:2347868 ppid:4757 flags:0x00000002
[Mon Mar 11 14:29:16 2024] Call Trace:
[Mon Mar 11 14:29:16 2024] <TASK>
[Mon Mar 11 14:29:16 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:16 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:16 2024] schedule_hrtimeout_range_clock+0x95/0x120
[Mon Mar 11 14:29:16 2024] ? __pfx_hrtimer_wakeup+0x10/0x10
[Mon Mar 11 14:29:16 2024] ep_poll+0x348/0x3b0
[Mon Mar 11 14:29:16 2024] ? __pfx_autoremove_wake_function+0x10/0x10
[Mon Mar 11 14:29:16 2024] do_epoll_wait+0xb1/0xd0
[Mon Mar 11 14:29:16 2024] __x64_sys_epoll_wait+0x60/0x100
[Mon Mar 11 14:29:16 2024] do_syscall_64+0x59/0x90
[Mon Mar 11 14:29:16 2024] ? __rseq_handle_notify_resume+0x26/0xb0
[Mon Mar 11 14:29:16 2024] ? exit_to_user_mode_loop+0xd0/0x130
[Mon Mar 11 14:29:16 2024] ? exit_to_user_mode_prepare+0xec/0x100
[Mon Mar 11 14:29:16 2024] ? syscall_exit_to_user_mode+0x22/0x40
[Mon Mar 11 14:29:16 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:16 2024] ? syscall_exit_work+0x103/0x130
[Mon Mar 11 14:29:16 2024] ? syscall_exit_to_user_mode+0x22/0x40
[Mon Mar 11 14:29:16 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:16 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:16 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:16 2024] entry_SYSCALL_64_after_hwframe+0x72/0xdc
[Mon Mar 11 14:29:16 2024] RIP: 0033:0x7fedb7d4e84e
[Mon Mar 11 14:29:16 2024] RSP: 002b:00007ffc284a9ab0 EFLAGS: 00000293 ORIG_RAX: 00000000000000e8
[Mon Mar 11 14:29:16 2024] RAX: ffffffffffffffda RBX: 000055f87dd397c0 RCX: 00007fedb7d4e84e
[Mon Mar 11 14:29:16 2024] RDX: 0000000000000001 RSI: 00007ffc284a9b0c RDI: 0000000000000005
[Mon Mar 11 14:29:16 2024] RBP: 000055f87dd2aa20 R08: 0000000000000000 R09: 0000000000000000
[Mon Mar 11 14:29:16 2024] R10: 0000000000001e79 R11: 0000000000000293 R12: 000055f87dd61190
[Mon Mar 11 14:29:16 2024] R13: 0000000000001e79 R14: 0000000000000026 R15: 000055f87dd5b760
[Mon Mar 11 14:29:16 2024] </TASK>
[Mon Mar 11 14:29:16 2024] task:rpcd_winreg state:S stack:0 pid:2347933 ppid:2339807 flags:0x00000002
[Mon Mar 11 14:29:16 2024] Call Trace:
[Mon Mar 11 14:29:16 2024] <TASK>
[Mon Mar 11 14:29:16 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:16 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:16 2024] schedule_hrtimeout_range_clock+0x95/0x120
[Mon Mar 11 14:29:16 2024] ? __pfx_hrtimer_wakeup+0x10/0x10
[Mon Mar 11 14:29:16 2024] ep_poll+0x348/0x3b0
[Mon Mar 11 14:29:16 2024] ? __pfx_autoremove_wake_function+0x10/0x10
[Mon Mar 11 14:29:16 2024] do_epoll_wait+0xb1/0xd0
[Mon Mar 11 14:29:16 2024] __x64_sys_epoll_wait+0x60/0x100
[Mon Mar 11 14:29:16 2024] do_syscall_64+0x59/0x90
[Mon Mar 11 14:29:16 2024] ? exit_to_user_mode_prepare+0xec/0x100
[Mon Mar 11 14:29:16 2024] ? syscall_exit_to_user_mode+0x22/0x40
[Mon Mar 11 14:29:16 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:16 2024] ? exit_to_user_mode_prepare+0xb6/0x100
[Mon Mar 11 14:29:16 2024] ? syscall_exit_to_user_mode+0x22/0x40
[Mon Mar 11 14:29:16 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:16 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:16 2024] ? sysvec_apic_timer_interrupt+0x3c/0x90
[Mon Mar 11 14:29:16 2024] entry_SYSCALL_64_after_hwframe+0x72/0xdc
[Mon Mar 11 14:29:16 2024] RIP: 0033:0x7f00b774e80a
[Mon Mar 11 14:29:16 2024] RSP: 002b:00007fff41efac18 EFLAGS: 00000246 ORIG_RAX: 00000000000000e8
[Mon Mar 11 14:29:16 2024] RAX: ffffffffffffffda RBX: 000055faa0996770 RCX: 00007f00b774e80a
[Mon Mar 11 14:29:16 2024] RDX: 0000000000000001 RSI: 00007fff41efac4c RDI: 0000000000000003
[Mon Mar 11 14:29:16 2024] RBP: 000055faa09872e0 R08: 000055faa09dc950 R09: 0000000000009dc2
[Mon Mar 11 14:29:16 2024] R10: 0000000000007530 R11: 0000000000000246 R12: 000055faa0996800
[Mon Mar 11 14:29:16 2024] R13: 0000000000007530 R14: 00007f00b8133db0 R15: 0000000000000000
[Mon Mar 11 14:29:16 2024] </TASK>
[Mon Mar 11 14:29:16 2024] task:smbd[10.33.22.1 state:S stack:0 pid:2349497 ppid:4757 flags:0x00000002
[Mon Mar 11 14:29:16 2024] Call Trace:
[Mon Mar 11 14:29:16 2024] <TASK>
[Mon Mar 11 14:29:16 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:16 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:16 2024] schedule_hrtimeout_range_clock+0x95/0x120
[Mon Mar 11 14:29:16 2024] ? __pfx_hrtimer_wakeup+0x10/0x10
[Mon Mar 11 14:29:16 2024] ep_poll+0x348/0x3b0
[Mon Mar 11 14:29:16 2024] ? __pfx_autoremove_wake_function+0x10/0x10
[Mon Mar 11 14:29:16 2024] do_epoll_wait+0xb1/0xd0
[Mon Mar 11 14:29:16 2024] __x64_sys_epoll_wait+0x60/0x100
[Mon Mar 11 14:29:16 2024] do_syscall_64+0x59/0x90
[Mon Mar 11 14:29:16 2024] ? audit_reset_context.part.0.constprop.0+0x273/0x2e0
[Mon Mar 11 14:29:16 2024] ? syscall_exit_work+0x103/0x130
[Mon Mar 11 14:29:16 2024] ? syscall_exit_to_user_mode+0x22/0x40
[Mon Mar 11 14:29:16 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:16 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:16 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:16 2024] entry_SYSCALL_64_after_hwframe+0x72/0xdc
[Mon Mar 11 14:29:16 2024] RIP: 0033:0x7fedb7d4e80a
[Mon Mar 11 14:29:16 2024] RSP: 002b:00007ffc284a9ad8 EFLAGS: 00000246 ORIG_RAX: 00000000000000e8
[Mon Mar 11 14:29:16 2024] RAX: ffffffffffffffda RBX: 000055f87dd397c0 RCX: 00007fedb7d4e80a
[Mon Mar 11 14:29:16 2024] RDX: 0000000000000001 RSI: 00007ffc284a9b0c RDI: 0000000000000005
[Mon Mar 11 14:29:16 2024] RBP: 000055f87dd2aa20 R08: 000000000000c5d2 R09: 0000000000000000
[Mon Mar 11 14:29:16 2024] R10: 000000000000c383 R11: 0000000000000246 R12: 000055f87dd61190
[Mon Mar 11 14:29:16 2024] R13: 000000000000c383 R14: 0000000000000026 R15: 000055f87dd5b760
[Mon Mar 11 14:29:16 2024] </TASK>
[Mon Mar 11 14:29:16 2024] task:smbd[10.40.232. state:S stack:0 pid:2351731 ppid:4757 flags:0x00000002
[Mon Mar 11 14:29:16 2024] Call Trace:
[Mon Mar 11 14:29:16 2024] <TASK>
[Mon Mar 11 14:29:16 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:16 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:16 2024] schedule_hrtimeout_range_clock+0x95/0x120
[Mon Mar 11 14:29:16 2024] ? __pfx_hrtimer_wakeup+0x10/0x10
[Mon Mar 11 14:29:16 2024] ep_poll+0x348/0x3b0
[Mon Mar 11 14:29:16 2024] ? __pfx_autoremove_wake_function+0x10/0x10
[Mon Mar 11 14:29:16 2024] do_epoll_wait+0xb1/0xd0
[Mon Mar 11 14:29:16 2024] __x64_sys_epoll_wait+0x60/0x100
[Mon Mar 11 14:29:16 2024] do_syscall_64+0x59/0x90
[Mon Mar 11 14:29:16 2024] ? handle_mm_fault+0xcd/0x290
[Mon Mar 11 14:29:16 2024] ? do_user_addr_fault+0x1d6/0x6a0
[Mon Mar 11 14:29:16 2024] ? exc_page_fault+0x62/0x150
[Mon Mar 11 14:29:16 2024] entry_SYSCALL_64_after_hwframe+0x72/0xdc
[Mon Mar 11 14:29:16 2024] RIP: 0033:0x7fedb7d4e84e
[Mon Mar 11 14:29:16 2024] RSP: 002b:00007ffc284a9ab0 EFLAGS: 00000293 ORIG_RAX: 00000000000000e8
[Mon Mar 11 14:29:16 2024] RAX: ffffffffffffffda RBX: 000055f87dd397c0 RCX: 00007fedb7d4e84e
[Mon Mar 11 14:29:16 2024] RDX: 0000000000000001 RSI: 00007ffc284a9b0c RDI: 0000000000000005
[Mon Mar 11 14:29:16 2024] RBP: 000055f87dd2aa20 R08: 0000000000000000 R09: 0000000000000000
[Mon Mar 11 14:29:16 2024] R10: 000000000000ad58 R11: 0000000000000293 R12: 000055f87dd58980
[Mon Mar 11 14:29:16 2024] R13: 000000000000ad58 R14: 0000000000000027 R15: 000055f87dd5b760
[Mon Mar 11 14:29:16 2024] </TASK>
[Mon Mar 11 14:29:16 2024] task:smbd[10.87.31.1 state:S stack:0 pid:2352076 ppid:4757 flags:0x00000002
[Mon Mar 11 14:29:16 2024] Call Trace:
[Mon Mar 11 14:29:16 2024] <TASK>
[Mon Mar 11 14:29:16 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:16 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:16 2024] schedule_hrtimeout_range_clock+0x95/0x120
[Mon Mar 11 14:29:16 2024] ? __pfx_hrtimer_wakeup+0x10/0x10
[Mon Mar 11 14:29:16 2024] ep_poll+0x348/0x3b0
[Mon Mar 11 14:29:16 2024] ? __pfx_autoremove_wake_function+0x10/0x10
[Mon Mar 11 14:29:16 2024] do_epoll_wait+0xb1/0xd0
[Mon Mar 11 14:29:16 2024] __x64_sys_epoll_wait+0x60/0x100
[Mon Mar 11 14:29:16 2024] do_syscall_64+0x59/0x90
[Mon Mar 11 14:29:16 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:16 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:16 2024] ? syscall_exit_to_user_mode+0x22/0x40
[Mon Mar 11 14:29:16 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:16 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:16 2024] ? syscall_exit_to_user_mode+0x22/0x40
[Mon Mar 11 14:29:16 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:16 2024] ? __irq_exit_rcu+0x46/0xc0
[Mon Mar 11 14:29:16 2024] ? sysvec_apic_timer_interrupt+0x3c/0x90
[Mon Mar 11 14:29:16 2024] entry_SYSCALL_64_after_hwframe+0x72/0xdc
[Mon Mar 11 14:29:16 2024] RIP: 0033:0x7fedb7d4e80a
[Mon Mar 11 14:29:16 2024] RSP: 002b:00007ffc284a9ad8 EFLAGS: 00000246 ORIG_RAX: 00000000000000e8
[Mon Mar 11 14:29:16 2024] RAX: ffffffffffffffda RBX: 000055f87dd397c0 RCX: 00007fedb7d4e80a
[Mon Mar 11 14:29:16 2024] RDX: 0000000000000001 RSI: 00007ffc284a9b0c RDI: 0000000000000005
[Mon Mar 11 14:29:16 2024] RBP: 000055f87dd2aa20 R08: 00000000000f41d2 R09: 0000000000000000
[Mon Mar 11 14:29:16 2024] R10: 000000000000ea60 R11: 0000000000000246 R12: 000055f87dd61190
[Mon Mar 11 14:29:16 2024] R13: 000000000000ea60 R14: 0000000000000026 R15: 000055f87dd5b760
[Mon Mar 11 14:29:16 2024] </TASK>
[Mon Mar 11 14:29:16 2024] task:smbd[10.40.157. state:S stack:0 pid:2352279 ppid:4757 flags:0x00000002
[Mon Mar 11 14:29:16 2024] Call Trace:
[Mon Mar 11 14:29:16 2024] <TASK>
[Mon Mar 11 14:29:16 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:16 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:16 2024] schedule_hrtimeout_range_clock+0x95/0x120
[Mon Mar 11 14:29:16 2024] ? __pfx_hrtimer_wakeup+0x10/0x10
[Mon Mar 11 14:29:16 2024] ep_poll+0x348/0x3b0
[Mon Mar 11 14:29:16 2024] ? __pfx_autoremove_wake_function+0x10/0x10
[Mon Mar 11 14:29:16 2024] do_epoll_wait+0xb1/0xd0
[Mon Mar 11 14:29:16 2024] __x64_sys_epoll_wait+0x60/0x100
[Mon Mar 11 14:29:16 2024] do_syscall_64+0x59/0x90
[Mon Mar 11 14:29:16 2024] ? audit_reset_context.part.0.constprop.0+0x273/0x2e0
[Mon Mar 11 14:29:16 2024] ? syscall_exit_work+0x103/0x130
[Mon Mar 11 14:29:16 2024] ? syscall_exit_to_user_mode+0x22/0x40
[Mon Mar 11 14:29:16 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:16 2024] entry_SYSCALL_64_after_hwframe+0x72/0xdc
[Mon Mar 11 14:29:16 2024] RIP: 0033:0x7fedb7d4e80a
[Mon Mar 11 14:29:16 2024] RSP: 002b:00007ffc284a9ad8 EFLAGS: 00000246 ORIG_RAX: 00000000000000e8
[Mon Mar 11 14:29:16 2024] RAX: ffffffffffffffda RBX: 000055f87dd397c0 RCX: 00007fedb7d4e80a
[Mon Mar 11 14:29:16 2024] RDX: 0000000000000001 RSI: 00007ffc284a9b0c RDI: 0000000000000005
[Mon Mar 11 14:29:16 2024] RBP: 000055f87dd2aa20 R08: 00000000000f41d2 R09: 0000000000000000
[Mon Mar 11 14:29:16 2024] R10: 000000000000ea60 R11: 0000000000000246 R12: 000055f87dd61190
[Mon Mar 11 14:29:16 2024] R13: 000000000000ea60 R14: 0000000000000026 R15: 000055f87dd5b760
[Mon Mar 11 14:29:16 2024] </TASK>
[Mon Mar 11 14:29:16 2024] task:smbd[10.87.29.8 state:S stack:0 pid:2352500 ppid:4757 flags:0x00000002
[Mon Mar 11 14:29:16 2024] Call Trace:
[Mon Mar 11 14:29:16 2024] <TASK>
[Mon Mar 11 14:29:16 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:16 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:16 2024] schedule_hrtimeout_range_clock+0x95/0x120
[Mon Mar 11 14:29:16 2024] ? __pfx_hrtimer_wakeup+0x10/0x10
[Mon Mar 11 14:29:16 2024] ep_poll+0x348/0x3b0
[Mon Mar 11 14:29:16 2024] ? __pfx_autoremove_wake_function+0x10/0x10
[Mon Mar 11 14:29:16 2024] do_epoll_wait+0xb1/0xd0
[Mon Mar 11 14:29:16 2024] __x64_sys_epoll_wait+0x60/0x100
[Mon Mar 11 14:29:16 2024] do_syscall_64+0x59/0x90
[Mon Mar 11 14:29:16 2024] ? exit_to_user_mode_prepare+0xb6/0x100
[Mon Mar 11 14:29:16 2024] ? syscall_exit_to_user_mode+0x22/0x40
[Mon Mar 11 14:29:16 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:16 2024] ? exc_page_fault+0x62/0x150
[Mon Mar 11 14:29:16 2024] entry_SYSCALL_64_after_hwframe+0x72/0xdc
[Mon Mar 11 14:29:16 2024] RIP: 0033:0x7fedb7d4e84e
[Mon Mar 11 14:29:16 2024] RSP: 002b:00007ffc284a9ab0 EFLAGS: 00000293 ORIG_RAX: 00000000000000e8
[Mon Mar 11 14:29:16 2024] RAX: ffffffffffffffda RBX: 000055f87dd397c0 RCX: 00007fedb7d4e84e
[Mon Mar 11 14:29:16 2024] RDX: 0000000000000001 RSI: 00007ffc284a9b0c RDI: 0000000000000005
[Mon Mar 11 14:29:16 2024] RBP: 000055f87dd2aa20 R08: 0000000000000000 R09: 0000000000000000
[Mon Mar 11 14:29:16 2024] R10: 000000000000e222 R11: 0000000000000293 R12: 000055f87dd58980
[Mon Mar 11 14:29:16 2024] R13: 000000000000e222 R14: 0000000000000027 R15: 000055f87dd5b760
[Mon Mar 11 14:29:16 2024] </TASK>
[Mon Mar 11 14:29:16 2024] task:smbd[10.87.27.7 state:S stack:0 pid:2352778 ppid:4757 flags:0x00000002
[Mon Mar 11 14:29:16 2024] Call Trace:
[Mon Mar 11 14:29:16 2024] <TASK>
[Mon Mar 11 14:29:16 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:16 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:16 2024] schedule_hrtimeout_range_clock+0x95/0x120
[Mon Mar 11 14:29:16 2024] ? __pfx_hrtimer_wakeup+0x10/0x10
[Mon Mar 11 14:29:16 2024] ep_poll+0x348/0x3b0
[Mon Mar 11 14:29:16 2024] ? __pfx_autoremove_wake_function+0x10/0x10
[Mon Mar 11 14:29:16 2024] do_epoll_wait+0xb1/0xd0
[Mon Mar 11 14:29:16 2024] __x64_sys_epoll_wait+0x60/0x100
[Mon Mar 11 14:29:16 2024] do_syscall_64+0x59/0x90
[Mon Mar 11 14:29:16 2024] entry_SYSCALL_64_after_hwframe+0x72/0xdc
[Mon Mar 11 14:29:16 2024] RIP: 0033:0x7fedb7d4e80a
[Mon Mar 11 14:29:16 2024] RSP: 002b:00007ffc284a9ad8 EFLAGS: 00000246 ORIG_RAX: 00000000000000e8
[Mon Mar 11 14:29:16 2024] RAX: ffffffffffffffda RBX: 000055f87dd397c0 RCX: 00007fedb7d4e80a
[Mon Mar 11 14:29:16 2024] RDX: 0000000000000001 RSI: 00007ffc284a9b0c RDI: 0000000000000005
[Mon Mar 11 14:29:16 2024] RBP: 000055f87dd2aa20 R08: 00000000000f4039 R09: 0000000000000000
[Mon Mar 11 14:29:16 2024] R10: 000000000000ea60 R11: 0000000000000246 R12: 000055f87dd61190
[Mon Mar 11 14:29:16 2024] R13: 000000000000ea60 R14: 0000000000000026 R15: 000055f87dd5b760
[Mon Mar 11 14:29:16 2024] </TASK>
[Mon Mar 11 14:29:16 2024] task:smbd[10.87.29.8 state:S stack:0 pid:2353738 ppid:4757 flags:0x00000002
[Mon Mar 11 14:29:16 2024] Call Trace:
[Mon Mar 11 14:29:16 2024] <TASK>
[Mon Mar 11 14:29:16 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:16 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:16 2024] schedule_hrtimeout_range_clock+0x95/0x120
[Mon Mar 11 14:29:16 2024] ? __pfx_hrtimer_wakeup+0x10/0x10
[Mon Mar 11 14:29:16 2024] ep_poll+0x348/0x3b0
[Mon Mar 11 14:29:16 2024] ? __pfx_autoremove_wake_function+0x10/0x10
[Mon Mar 11 14:29:16 2024] do_epoll_wait+0xb1/0xd0
[Mon Mar 11 14:29:16 2024] __x64_sys_epoll_wait+0x60/0x100
[Mon Mar 11 14:29:16 2024] do_syscall_64+0x59/0x90
[Mon Mar 11 14:29:16 2024] ? exc_page_fault+0x62/0x150
[Mon Mar 11 14:29:16 2024] entry_SYSCALL_64_after_hwframe+0x72/0xdc
[Mon Mar 11 14:29:16 2024] RIP: 0033:0x7fedb7d4e84e
[Mon Mar 11 14:29:16 2024] RSP: 002b:00007ffc284a9ab0 EFLAGS: 00000293 ORIG_RAX: 00000000000000e8
[Mon Mar 11 14:29:16 2024] RAX: ffffffffffffffda RBX: 000055f87dd397c0 RCX: 00007fedb7d4e84e
[Mon Mar 11 14:29:16 2024] RDX: 0000000000000001 RSI: 00007ffc284a9b0c RDI: 0000000000000005
[Mon Mar 11 14:29:16 2024] RBP: 000055f87dd2aa20 R08: 0000000000000000 R09: 0000000000000000
[Mon Mar 11 14:29:16 2024] R10: 000000000000ea60 R11: 0000000000000293 R12: 000055f87dd61190
[Mon Mar 11 14:29:16 2024] R13: 000000000000ea60 R14: 0000000000000026 R15: 000055f87dd5b760
[Mon Mar 11 14:29:16 2024] </TASK>
[Mon Mar 11 14:29:16 2024] task:smbd[10.40.229. state:S stack:0 pid:2355747 ppid:4757 flags:0x00000002
[Mon Mar 11 14:29:16 2024] Call Trace:
[Mon Mar 11 14:29:16 2024] <TASK>
[Mon Mar 11 14:29:16 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:16 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:16 2024] schedule_hrtimeout_range_clock+0x95/0x120
[Mon Mar 11 14:29:16 2024] ? __pfx_hrtimer_wakeup+0x10/0x10
[Mon Mar 11 14:29:16 2024] ep_poll+0x348/0x3b0
[Mon Mar 11 14:29:16 2024] ? __pfx_autoremove_wake_function+0x10/0x10
[Mon Mar 11 14:29:16 2024] do_epoll_wait+0xb1/0xd0
[Mon Mar 11 14:29:16 2024] __x64_sys_epoll_wait+0x60/0x100
[Mon Mar 11 14:29:16 2024] do_syscall_64+0x59/0x90
[Mon Mar 11 14:29:16 2024] ? syscall_exit_work+0x103/0x130
[Mon Mar 11 14:29:16 2024] ? syscall_exit_to_user_mode+0x22/0x40
[Mon Mar 11 14:29:16 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:16 2024] ? syscall_exit_to_user_mode+0x22/0x40
[Mon Mar 11 14:29:16 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:16 2024] ? syscall_exit_to_user_mode+0x22/0x40
[Mon Mar 11 14:29:16 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:16 2024] entry_SYSCALL_64_after_hwframe+0x72/0xdc
[Mon Mar 11 14:29:16 2024] RIP: 0033:0x7fedb7d4e84e
[Mon Mar 11 14:29:16 2024] RSP: 002b:00007ffc284a9ab0 EFLAGS: 00000293 ORIG_RAX: 00000000000000e8
[Mon Mar 11 14:29:16 2024] RAX: ffffffffffffffda RBX: 000055f87dd397c0 RCX: 00007fedb7d4e84e
[Mon Mar 11 14:29:16 2024] RDX: 0000000000000001 RSI: 00007ffc284a9b0c RDI: 0000000000000005
[Mon Mar 11 14:29:16 2024] RBP: 000055f87dd2aa20 R08: 0000000000000000 R09: 0000000000000000
[Mon Mar 11 14:29:16 2024] R10: 000000000000ea60 R11: 0000000000000293 R12: 000055f87dd61190
[Mon Mar 11 14:29:16 2024] R13: 000000000000ea60 R14: 0000000000000026 R15: 000055f87dd5b760
[Mon Mar 11 14:29:16 2024] </TASK>
[Mon Mar 11 14:29:16 2024] task:smbd[10.40.140. state:S stack:0 pid:2356004 ppid:4757 flags:0x00000002
[Mon Mar 11 14:29:16 2024] Call Trace:
[Mon Mar 11 14:29:16 2024] <TASK>
[Mon Mar 11 14:29:16 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:16 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:16 2024] schedule_hrtimeout_range_clock+0x95/0x120
[Mon Mar 11 14:29:16 2024] ? __pfx_hrtimer_wakeup+0x10/0x10
[Mon Mar 11 14:29:16 2024] ep_poll+0x348/0x3b0
[Mon Mar 11 14:29:16 2024] ? __pfx_autoremove_wake_function+0x10/0x10
[Mon Mar 11 14:29:16 2024] do_epoll_wait+0xb1/0xd0
[Mon Mar 11 14:29:16 2024] __x64_sys_epoll_wait+0x60/0x100
[Mon Mar 11 14:29:16 2024] do_syscall_64+0x59/0x90
[Mon Mar 11 14:29:16 2024] ? syscall_exit_to_user_mode+0x22/0x40
[Mon Mar 11 14:29:16 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:16 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:16 2024] ? __irq_exit_rcu+0x46/0xc0
[Mon Mar 11 14:29:16 2024] ? common_interrupt+0x43/0xa0
[Mon Mar 11 14:29:16 2024] entry_SYSCALL_64_after_hwframe+0x72/0xdc
[Mon Mar 11 14:29:16 2024] RIP: 0033:0x7fedb7d4e84e
[Mon Mar 11 14:29:16 2024] RSP: 002b:00007ffc284a9ab0 EFLAGS: 00000293 ORIG_RAX: 00000000000000e8
[Mon Mar 11 14:29:16 2024] RAX: ffffffffffffffda RBX: 000055f87dd397c0 RCX: 00007fedb7d4e84e
[Mon Mar 11 14:29:16 2024] RDX: 0000000000000001 RSI: 00007ffc284a9b0c RDI: 0000000000000005
[Mon Mar 11 14:29:16 2024] RBP: 000055f87dd2aa20 R08: 0000000000000000 R09: 0000000000000000
[Mon Mar 11 14:29:16 2024] R10: 000000000000ea60 R11: 0000000000000293 R12: 000055f87dd61190
[Mon Mar 11 14:29:16 2024] R13: 000000000000ea60 R14: 0000000000000026 R15: 000055f87dd5b760
[Mon Mar 11 14:29:16 2024] </TASK>
[Mon Mar 11 14:29:16 2024] task:smbd[10.87.29.2 state:S stack:0 pid:2362257 ppid:4757 flags:0x00000002
[Mon Mar 11 14:29:16 2024] Call Trace:
[Mon Mar 11 14:29:16 2024] <TASK>
[Mon Mar 11 14:29:16 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:16 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:16 2024] schedule_hrtimeout_range_clock+0x95/0x120
[Mon Mar 11 14:29:16 2024] ? __pfx_hrtimer_wakeup+0x10/0x10
[Mon Mar 11 14:29:16 2024] ep_poll+0x348/0x3b0
[Mon Mar 11 14:29:16 2024] ? __pfx_autoremove_wake_function+0x10/0x10
[Mon Mar 11 14:29:16 2024] do_epoll_wait+0xb1/0xd0
[Mon Mar 11 14:29:16 2024] __x64_sys_epoll_wait+0x60/0x100
[Mon Mar 11 14:29:16 2024] do_syscall_64+0x59/0x90
[Mon Mar 11 14:29:16 2024] ? audit_reset_context.part.0.constprop.0+0x273/0x2e0
[Mon Mar 11 14:29:16 2024] ? syscall_exit_work+0x103/0x130
[Mon Mar 11 14:29:16 2024] ? syscall_exit_to_user_mode+0x22/0x40
[Mon Mar 11 14:29:16 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:16 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:16 2024] entry_SYSCALL_64_after_hwframe+0x72/0xdc
[Mon Mar 11 14:29:16 2024] RIP: 0033:0x7fedb7d4e80a
[Mon Mar 11 14:29:16 2024] RSP: 002b:00007ffc284a9ad8 EFLAGS: 00000246 ORIG_RAX: 00000000000000e8
[Mon Mar 11 14:29:16 2024] RAX: ffffffffffffffda RBX: 000055f87dd397c0 RCX: 00007fedb7d4e80a
[Mon Mar 11 14:29:16 2024] RDX: 0000000000000001 RSI: 00007ffc284a9b0c RDI: 0000000000000005
[Mon Mar 11 14:29:16 2024] RBP: 000055f87dd2aa20 R08: 00000000000f4062 R09: 0000000000000000
[Mon Mar 11 14:29:16 2024] R10: 000000000000ea60 R11: 0000000000000246 R12: 000055f87dd61190
[Mon Mar 11 14:29:16 2024] R13: 000000000000ea60 R14: 0000000000000026 R15: 000055f87dd5b760
[Mon Mar 11 14:29:16 2024] </TASK>
[Mon Mar 11 14:29:16 2024] task:smbd[10.87.30.8 state:S stack:0 pid:2363822 ppid:4757 flags:0x00000002
[Mon Mar 11 14:29:16 2024] Call Trace:
[Mon Mar 11 14:29:16 2024] <TASK>
[Mon Mar 11 14:29:16 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:16 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:16 2024] schedule_hrtimeout_range_clock+0x95/0x120
[Mon Mar 11 14:29:16 2024] ? __pfx_hrtimer_wakeup+0x10/0x10
[Mon Mar 11 14:29:16 2024] ep_poll+0x348/0x3b0
[Mon Mar 11 14:29:16 2024] ? __pfx_autoremove_wake_function+0x10/0x10
[Mon Mar 11 14:29:16 2024] do_epoll_wait+0xb1/0xd0
[Mon Mar 11 14:29:16 2024] __x64_sys_epoll_wait+0x60/0x100
[Mon Mar 11 14:29:16 2024] do_syscall_64+0x59/0x90
[Mon Mar 11 14:29:16 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:16 2024] ? syscall_exit_work+0x103/0x130
[Mon Mar 11 14:29:16 2024] ? syscall_exit_to_user_mode+0x22/0x40
[Mon Mar 11 14:29:16 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:16 2024] ? syscall_exit_work+0x103/0x130
[Mon Mar 11 14:29:16 2024] ? syscall_exit_to_user_mode+0x22/0x40
[Mon Mar 11 14:29:16 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:16 2024] ? syscall_exit_work+0x103/0x130
[Mon Mar 11 14:29:16 2024] ? syscall_exit_to_user_mode+0x22/0x40
[Mon Mar 11 14:29:16 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:16 2024] ? exc_page_fault+0x62/0x150
[Mon Mar 11 14:29:16 2024] entry_SYSCALL_64_after_hwframe+0x72/0xdc
[Mon Mar 11 14:29:16 2024] RIP: 0033:0x7fedb7d4e84e
[Mon Mar 11 14:29:16 2024] RSP: 002b:00007ffc284a9ab0 EFLAGS: 00000293 ORIG_RAX: 00000000000000e8
[Mon Mar 11 14:29:16 2024] RAX: ffffffffffffffda RBX: 000055f87dd397c0 RCX: 00007fedb7d4e84e
[Mon Mar 11 14:29:16 2024] RDX: 0000000000000001 RSI: 00007ffc284a9b0c RDI: 0000000000000005
[Mon Mar 11 14:29:16 2024] RBP: 000055f87dd2aa20 R08: 0000000000000000 R09: 0000000000000000
[Mon Mar 11 14:29:16 2024] R10: 000000000000ea60 R11: 0000000000000293 R12: 000055f87dd61190
[Mon Mar 11 14:29:16 2024] R13: 000000000000ea60 R14: 0000000000000026 R15: 000055f87dd5b760
[Mon Mar 11 14:29:16 2024] </TASK>
[Mon Mar 11 14:29:16 2024] task:smbd[10.87.29.9 state:S stack:0 pid:2364172 ppid:4757 flags:0x00000002
[Mon Mar 11 14:29:16 2024] Call Trace:
[Mon Mar 11 14:29:16 2024] <TASK>
[Mon Mar 11 14:29:16 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:16 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:16 2024] schedule_hrtimeout_range_clock+0x95/0x120
[Mon Mar 11 14:29:16 2024] ? __pfx_hrtimer_wakeup+0x10/0x10
[Mon Mar 11 14:29:16 2024] ep_poll+0x348/0x3b0
[Mon Mar 11 14:29:16 2024] ? __pfx_autoremove_wake_function+0x10/0x10
[Mon Mar 11 14:29:16 2024] do_epoll_wait+0xb1/0xd0
[Mon Mar 11 14:29:16 2024] __x64_sys_epoll_wait+0x60/0x100
[Mon Mar 11 14:29:16 2024] do_syscall_64+0x59/0x90
[Mon Mar 11 14:29:16 2024] ? __audit_filter_op+0xa5/0xf0
[Mon Mar 11 14:29:16 2024] ? mntput_no_expire+0x4a/0x250
[Mon Mar 11 14:29:16 2024] ? audit_reset_context.part.0.constprop.0+0x273/0x2e0
[Mon Mar 11 14:29:16 2024] ? syscall_exit_work+0x103/0x130
[Mon Mar 11 14:29:16 2024] ? syscall_exit_to_user_mode+0x22/0x40
[Mon Mar 11 14:29:16 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:16 2024] entry_SYSCALL_64_after_hwframe+0x72/0xdc
[Mon Mar 11 14:29:16 2024] RIP: 0033:0x7fedb7d4e84e
[Mon Mar 11 14:29:16 2024] RSP: 002b:00007ffc284a9ab0 EFLAGS: 00000293 ORIG_RAX: 00000000000000e8
[Mon Mar 11 14:29:16 2024] RAX: ffffffffffffffda RBX: 000055f87dd397c0 RCX: 00007fedb7d4e84e
[Mon Mar 11 14:29:16 2024] RDX: 0000000000000001 RSI: 00007ffc284a9b0c RDI: 0000000000000005
[Mon Mar 11 14:29:16 2024] RBP: 000055f87dd2aa20 R08: 0000000000000000 R09: 0000000000000000
[Mon Mar 11 14:29:16 2024] R10: 000000000000ea60 R11: 0000000000000293 R12: 000055f87dd61190
[Mon Mar 11 14:29:16 2024] R13: 000000000000ea60 R14: 0000000000000026 R15: 000055f87dd5b760
[Mon Mar 11 14:29:16 2024] </TASK>
[Mon Mar 11 14:29:16 2024] task:smbd[10.87.29.9 state:S stack:0 pid:2364174 ppid:4757 flags:0x00000002
[Mon Mar 11 14:29:16 2024] Call Trace:
[Mon Mar 11 14:29:16 2024] <TASK>
[Mon Mar 11 14:29:16 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:16 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:16 2024] schedule_hrtimeout_range_clock+0x95/0x120
[Mon Mar 11 14:29:16 2024] ? __pfx_hrtimer_wakeup+0x10/0x10
[Mon Mar 11 14:29:16 2024] ep_poll+0x348/0x3b0
[Mon Mar 11 14:29:16 2024] ? __pfx_autoremove_wake_function+0x10/0x10
[Mon Mar 11 14:29:16 2024] do_epoll_wait+0xb1/0xd0
[Mon Mar 11 14:29:16 2024] __x64_sys_epoll_wait+0x60/0x100
[Mon Mar 11 14:29:16 2024] do_syscall_64+0x59/0x90
[Mon Mar 11 14:29:16 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:16 2024] ? syscall_exit_to_user_mode+0x22/0x40
[Mon Mar 11 14:29:16 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:16 2024] ? exit_to_user_mode_prepare+0xb6/0x100
[Mon Mar 11 14:29:16 2024] ? syscall_exit_to_user_mode+0x22/0x40
[Mon Mar 11 14:29:16 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:16 2024] entry_SYSCALL_64_after_hwframe+0x72/0xdc
[Mon Mar 11 14:29:16 2024] RIP: 0033:0x7fedb7d4e80a
[Mon Mar 11 14:29:16 2024] RSP: 002b:00007ffc284a9ad8 EFLAGS: 00000246 ORIG_RAX: 00000000000000e8
[Mon Mar 11 14:29:16 2024] RAX: ffffffffffffffda RBX: 000055f87dd397c0 RCX: 00007fedb7d4e80a
[Mon Mar 11 14:29:16 2024] RDX: 0000000000000001 RSI: 00007ffc284a9b0c RDI: 0000000000000005
[Mon Mar 11 14:29:16 2024] RBP: 000055f87dd2aa20 R08: 00000000000f41c2 R09: 0000000000000000
[Mon Mar 11 14:29:16 2024] R10: 000000000000ea60 R11: 0000000000000246 R12: 000055f87dd61190
[Mon Mar 11 14:29:16 2024] R13: 000000000000ea60 R14: 0000000000000026 R15: 000055f87dd5b760
[Mon Mar 11 14:29:16 2024] </TASK>
[Mon Mar 11 14:29:16 2024] task:kworker/36:1 state:I stack:0 pid:2364257 ppid:2 flags:0x00004000
[Mon Mar 11 14:29:16 2024] Workqueue: 0x0 (mm_percpu_wq)
[Mon Mar 11 14:29:16 2024] Call Trace:
[Mon Mar 11 14:29:16 2024] <TASK>
[Mon Mar 11 14:29:16 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:16 2024] ? __pfx_worker_thread+0x10/0x10
[Mon Mar 11 14:29:16 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:16 2024] worker_thread+0xbb/0x3a0
[Mon Mar 11 14:29:16 2024] ? __pfx_worker_thread+0x10/0x10
[Mon Mar 11 14:29:16 2024] kthread+0xdd/0x100
[Mon Mar 11 14:29:16 2024] ? __pfx_kthread+0x10/0x10
[Mon Mar 11 14:29:16 2024] ret_from_fork+0x29/0x50
[Mon Mar 11 14:29:16 2024] </TASK>
[Mon Mar 11 14:29:16 2024] task:kworker/38:2 state:I stack:0 pid:2364258 ppid:2 flags:0x00004000
[Mon Mar 11 14:29:16 2024] Workqueue: 0x0 (mm_percpu_wq)
[Mon Mar 11 14:29:16 2024] Call Trace:
[Mon Mar 11 14:29:16 2024] <TASK>
[Mon Mar 11 14:29:16 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:16 2024] ? __pfx_worker_thread+0x10/0x10
[Mon Mar 11 14:29:16 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:16 2024] worker_thread+0xbb/0x3a0
[Mon Mar 11 14:29:16 2024] ? __pfx_worker_thread+0x10/0x10
[Mon Mar 11 14:29:16 2024] kthread+0xdd/0x100
[Mon Mar 11 14:29:16 2024] ? __pfx_kthread+0x10/0x10
[Mon Mar 11 14:29:16 2024] ret_from_fork+0x29/0x50
[Mon Mar 11 14:29:16 2024] </TASK>
[Mon Mar 11 14:29:16 2024] task:kworker/30:0 state:D stack:0 pid:2364267 ppid:2 flags:0x00004000
[Mon Mar 11 14:29:16 2024] Workqueue: events delayed_fput
[Mon Mar 11 14:29:16 2024] Call Trace:
[Mon Mar 11 14:29:16 2024] <TASK>
[Mon Mar 11 14:29:16 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:16 2024] ? __pfx_wbt_cleanup_cb+0x10/0x10
[Mon Mar 11 14:29:16 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:16 2024] io_schedule+0x42/0x70
[Mon Mar 11 14:29:16 2024] rq_qos_wait+0xbb/0x130
[Mon Mar 11 14:29:16 2024] ? __pfx_rq_qos_wake_function+0x10/0x10
[Mon Mar 11 14:29:16 2024] ? __pfx_wbt_inflight_cb+0x10/0x10
[Mon Mar 11 14:29:16 2024] wbt_wait+0x9c/0x100
[Mon Mar 11 14:29:16 2024] __rq_qos_throttle+0x20/0x40
[Mon Mar 11 14:29:16 2024] blk_mq_submit_bio+0x183/0x580
[Mon Mar 11 14:29:16 2024] __submit_bio_noacct+0x7e/0x1e0
[Mon Mar 11 14:29:16 2024] ext4_bio_write_page+0x198/0x490 [ext4]
[Mon Mar 11 14:29:16 2024] ? folio_clear_dirty_for_io+0x13d/0x1b0
[Mon Mar 11 14:29:16 2024] mpage_submit_page+0x5a/0x70 [ext4]
[Mon Mar 11 14:29:16 2024] mpage_map_and_submit_buffers+0x146/0x240 [ext4]
[Mon Mar 11 14:29:16 2024] mpage_map_and_submit_extent+0x56/0x300 [ext4]
[Mon Mar 11 14:29:16 2024] ext4_do_writepages+0x5f0/0x780 [ext4]
[Mon Mar 11 14:29:16 2024] ext4_writepages+0xac/0x150 [ext4]
[Mon Mar 11 14:29:16 2024] do_writepages+0xcc/0x1d0
[Mon Mar 11 14:29:16 2024] ? sched_clock+0xc/0x30
[Mon Mar 11 14:29:16 2024] ? raw_spin_rq_lock_nested+0x19/0x80
[Mon Mar 11 14:29:16 2024] filemap_fdatawrite_wbc+0x66/0x90
[Mon Mar 11 14:29:16 2024] __filemap_fdatawrite_range+0x54/0x80
[Mon Mar 11 14:29:16 2024] ext4_release_file+0x70/0xb0 [ext4]
[Mon Mar 11 14:29:16 2024] __fput+0x91/0x250
[Mon Mar 11 14:29:16 2024] delayed_fput+0x1f/0x30
[Mon Mar 11 14:29:16 2024] process_one_work+0x1e2/0x3b0
[Mon Mar 11 14:29:16 2024] ? __pfx_worker_thread+0x10/0x10
[Mon Mar 11 14:29:16 2024] worker_thread+0x50/0x3a0
[Mon Mar 11 14:29:16 2024] ? __pfx_worker_thread+0x10/0x10
[Mon Mar 11 14:29:16 2024] kthread+0xdd/0x100
[Mon Mar 11 14:29:16 2024] ? __pfx_kthread+0x10/0x10
[Mon Mar 11 14:29:16 2024] ret_from_fork+0x29/0x50
[Mon Mar 11 14:29:16 2024] </TASK>
[Mon Mar 11 14:29:16 2024] task:smbd[10.87.30.4 state:S stack:0 pid:2364892 ppid:4757 flags:0x00000002
[Mon Mar 11 14:29:16 2024] Call Trace:
[Mon Mar 11 14:29:16 2024] <TASK>
[Mon Mar 11 14:29:16 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:16 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:16 2024] schedule_hrtimeout_range_clock+0x95/0x120
[Mon Mar 11 14:29:16 2024] ? __pfx_hrtimer_wakeup+0x10/0x10
[Mon Mar 11 14:29:16 2024] ep_poll+0x348/0x3b0
[Mon Mar 11 14:29:16 2024] ? __pfx_autoremove_wake_function+0x10/0x10
[Mon Mar 11 14:29:16 2024] do_epoll_wait+0xb1/0xd0
[Mon Mar 11 14:29:16 2024] __x64_sys_epoll_wait+0x60/0x100
[Mon Mar 11 14:29:16 2024] do_syscall_64+0x59/0x90
[Mon Mar 11 14:29:16 2024] ? syscall_exit_work+0x103/0x130
[Mon Mar 11 14:29:16 2024] ? syscall_exit_to_user_mode+0x22/0x40
[Mon Mar 11 14:29:16 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:16 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:16 2024] entry_SYSCALL_64_after_hwframe+0x72/0xdc
[Mon Mar 11 14:29:16 2024] RIP: 0033:0x7fedb7d4e80a
[Mon Mar 11 14:29:16 2024] RSP: 002b:00007ffc284a9ad8 EFLAGS: 00000246 ORIG_RAX: 00000000000000e8
[Mon Mar 11 14:29:16 2024] RAX: ffffffffffffffda RBX: 000055f87dd397c0 RCX: 00007fedb7d4e80a
[Mon Mar 11 14:29:16 2024] RDX: 0000000000000001 RSI: 00007ffc284a9b0c RDI: 0000000000000005
[Mon Mar 11 14:29:16 2024] RBP: 000055f87dd2aa20 R08: 00000000000f41bc R09: 0000000000000000
[Mon Mar 11 14:29:16 2024] R10: 000000000000ea60 R11: 0000000000000246 R12: 000055f87dd61190
[Mon Mar 11 14:29:16 2024] R13: 000000000000ea60 R14: 0000000000000026 R15: 000055f87dd5b760
[Mon Mar 11 14:29:16 2024] </TASK>
[Mon Mar 11 14:29:16 2024] task:smbd[10.87.30.4 state:S stack:0 pid:2364895 ppid:4757 flags:0x00000002
[Mon Mar 11 14:29:16 2024] Call Trace:
[Mon Mar 11 14:29:16 2024] <TASK>
[Mon Mar 11 14:29:16 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:16 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:16 2024] schedule_hrtimeout_range_clock+0x95/0x120
[Mon Mar 11 14:29:16 2024] ? __pfx_hrtimer_wakeup+0x10/0x10
[Mon Mar 11 14:29:16 2024] ep_poll+0x348/0x3b0
[Mon Mar 11 14:29:16 2024] ? __pfx_autoremove_wake_function+0x10/0x10
[Mon Mar 11 14:29:16 2024] do_epoll_wait+0xb1/0xd0
[Mon Mar 11 14:29:16 2024] __x64_sys_epoll_wait+0x60/0x100
[Mon Mar 11 14:29:16 2024] do_syscall_64+0x59/0x90
[Mon Mar 11 14:29:16 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:16 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:16 2024] entry_SYSCALL_64_after_hwframe+0x72/0xdc
[Mon Mar 11 14:29:16 2024] RIP: 0033:0x7fedb7d4e80a
[Mon Mar 11 14:29:16 2024] RSP: 002b:00007ffc284a9ad8 EFLAGS: 00000246 ORIG_RAX: 00000000000000e8
[Mon Mar 11 14:29:16 2024] RAX: ffffffffffffffda RBX: 000055f87dd397c0 RCX: 00007fedb7d4e80a
[Mon Mar 11 14:29:16 2024] RDX: 0000000000000001 RSI: 00007ffc284a9b0c RDI: 0000000000000005
[Mon Mar 11 14:29:16 2024] RBP: 000055f87dd2aa20 R08: 00000000000f41fe R09: 0000000000000000
[Mon Mar 11 14:29:16 2024] R10: 000000000000ea60 R11: 0000000000000246 R12: 000055f87dd61190
[Mon Mar 11 14:29:16 2024] R13: 000000000000ea60 R14: 0000000000000026 R15: 000055f87dd5b760
[Mon Mar 11 14:29:16 2024] </TASK>
[Mon Mar 11 14:29:16 2024] task:smbd[10.40.131. state:S stack:0 pid:2365796 ppid:4757 flags:0x00000002
[Mon Mar 11 14:29:16 2024] Call Trace:
[Mon Mar 11 14:29:16 2024] <TASK>
[Mon Mar 11 14:29:16 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:16 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:16 2024] schedule_hrtimeout_range_clock+0x95/0x120
[Mon Mar 11 14:29:16 2024] ? __pfx_hrtimer_wakeup+0x10/0x10
[Mon Mar 11 14:29:16 2024] ep_poll+0x348/0x3b0
[Mon Mar 11 14:29:16 2024] ? __pfx_autoremove_wake_function+0x10/0x10
[Mon Mar 11 14:29:16 2024] do_epoll_wait+0xb1/0xd0
[Mon Mar 11 14:29:16 2024] __x64_sys_epoll_wait+0x60/0x100
[Mon Mar 11 14:29:16 2024] do_syscall_64+0x59/0x90
[Mon Mar 11 14:29:16 2024] ? audit_reset_context.part.0.constprop.0+0x273/0x2e0
[Mon Mar 11 14:29:16 2024] ? syscall_exit_work+0x103/0x130
[Mon Mar 11 14:29:16 2024] ? syscall_exit_to_user_mode+0x22/0x40
[Mon Mar 11 14:29:16 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:16 2024] entry_SYSCALL_64_after_hwframe+0x72/0xdc
[Mon Mar 11 14:29:16 2024] RIP: 0033:0x7fedb7d4e80a
[Mon Mar 11 14:29:16 2024] RSP: 002b:00007ffc284a9ad8 EFLAGS: 00000246 ORIG_RAX: 00000000000000e8
[Mon Mar 11 14:29:16 2024] RAX: ffffffffffffffda RBX: 000055f87dd397c0 RCX: 00007fedb7d4e80a
[Mon Mar 11 14:29:16 2024] RDX: 0000000000000001 RSI: 00007ffc284a9b0c RDI: 0000000000000005
[Mon Mar 11 14:29:16 2024] RBP: 000055f87dd2aa20 R08: 00000000000f406c R09: 0000000000000000
[Mon Mar 11 14:29:16 2024] R10: 000000000000ea60 R11: 0000000000000246 R12: 000055f87dd61190
[Mon Mar 11 14:29:16 2024] R13: 000000000000ea60 R14: 0000000000000026 R15: 000055f87dd5b760
[Mon Mar 11 14:29:16 2024] </TASK>
[Mon Mar 11 14:29:16 2024] task:smbd[10.86.26.7 state:S stack:0 pid:2366328 ppid:4757 flags:0x00000002
[Mon Mar 11 14:29:16 2024] Call Trace:
[Mon Mar 11 14:29:16 2024] <TASK>
[Mon Mar 11 14:29:16 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:16 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:16 2024] schedule_hrtimeout_range_clock+0x95/0x120
[Mon Mar 11 14:29:16 2024] ? __pfx_hrtimer_wakeup+0x10/0x10
[Mon Mar 11 14:29:16 2024] ep_poll+0x348/0x3b0
[Mon Mar 11 14:29:16 2024] ? __pfx_autoremove_wake_function+0x10/0x10
[Mon Mar 11 14:29:16 2024] do_epoll_wait+0xb1/0xd0
[Mon Mar 11 14:29:16 2024] __x64_sys_epoll_wait+0x60/0x100
[Mon Mar 11 14:29:16 2024] do_syscall_64+0x59/0x90
[Mon Mar 11 14:29:16 2024] ? audit_reset_context.part.0.constprop.0+0x273/0x2e0
[Mon Mar 11 14:29:16 2024] ? syscall_exit_work+0x103/0x130
[Mon Mar 11 14:29:16 2024] ? syscall_exit_to_user_mode+0x22/0x40
[Mon Mar 11 14:29:16 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:16 2024] ? syscall_exit_work+0x103/0x130
[Mon Mar 11 14:29:16 2024] ? syscall_exit_to_user_mode+0x22/0x40
[Mon Mar 11 14:29:16 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:16 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:16 2024] ? sysvec_call_function_single+0x3c/0x90
[Mon Mar 11 14:29:16 2024] entry_SYSCALL_64_after_hwframe+0x72/0xdc
[Mon Mar 11 14:29:16 2024] RIP: 0033:0x7fedb7d4e80a
[Mon Mar 11 14:29:16 2024] RSP: 002b:00007ffc284a9ad8 EFLAGS: 00000246 ORIG_RAX: 00000000000000e8
[Mon Mar 11 14:29:16 2024] RAX: ffffffffffffffda RBX: 000055f87dd397c0 RCX: 00007fedb7d4e80a
[Mon Mar 11 14:29:16 2024] RDX: 0000000000000001 RSI: 00007ffc284a9b0c RDI: 0000000000000005
[Mon Mar 11 14:29:16 2024] RBP: 000055f87dd2aa20 R08: 00000000000f41d9 R09: 0000000000000000
[Mon Mar 11 14:29:16 2024] R10: 000000000000ea60 R11: 0000000000000246 R12: 000055f87dd58980
[Mon Mar 11 14:29:16 2024] R13: 000000000000ea60 R14: 0000000000000027 R15: 000055f87dd5b760
[Mon Mar 11 14:29:16 2024] </TASK>
[Mon Mar 11 14:29:16 2024] task:smbd[10.87.28.1 state:S stack:0 pid:2367684 ppid:4757 flags:0x00000002
[Mon Mar 11 14:29:16 2024] Call Trace:
[Mon Mar 11 14:29:16 2024] <TASK>
[Mon Mar 11 14:29:16 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:16 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:16 2024] schedule_hrtimeout_range_clock+0x95/0x120
[Mon Mar 11 14:29:16 2024] ? __pfx_hrtimer_wakeup+0x10/0x10
[Mon Mar 11 14:29:16 2024] ep_poll+0x348/0x3b0
[Mon Mar 11 14:29:16 2024] ? __pfx_autoremove_wake_function+0x10/0x10
[Mon Mar 11 14:29:16 2024] do_epoll_wait+0xb1/0xd0
[Mon Mar 11 14:29:16 2024] __x64_sys_epoll_wait+0x60/0x100
[Mon Mar 11 14:29:16 2024] do_syscall_64+0x59/0x90
[Mon Mar 11 14:29:16 2024] ? syscall_exit_work+0x103/0x130
[Mon Mar 11 14:29:16 2024] ? syscall_exit_to_user_mode+0x22/0x40
[Mon Mar 11 14:29:16 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:16 2024] ? syscall_exit_to_user_mode+0x22/0x40
[Mon Mar 11 14:29:16 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:16 2024] ? syscall_exit_work+0x103/0x130
[Mon Mar 11 14:29:16 2024] ? syscall_exit_to_user_mode+0x22/0x40
[Mon Mar 11 14:29:16 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:16 2024] ? syscall_exit_to_user_mode+0x22/0x40
[Mon Mar 11 14:29:16 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:16 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:16 2024] entry_SYSCALL_64_after_hwframe+0x72/0xdc
[Mon Mar 11 14:29:16 2024] RIP: 0033:0x7fedb7d4e80a
[Mon Mar 11 14:29:16 2024] RSP: 002b:00007ffc284a9ad8 EFLAGS: 00000246 ORIG_RAX: 00000000000000e8
[Mon Mar 11 14:29:16 2024] RAX: ffffffffffffffda RBX: 000055f87dd397c0 RCX: 00007fedb7d4e80a
[Mon Mar 11 14:29:16 2024] RDX: 0000000000000001 RSI: 00007ffc284a9b0c RDI: 0000000000000005
[Mon Mar 11 14:29:16 2024] RBP: 000055f87dd2aa20 R08: 00000000000f41d6 R09: 0000000000000000
[Mon Mar 11 14:29:16 2024] R10: 000000000000ea60 R11: 0000000000000246 R12: 000055f87dd61190
[Mon Mar 11 14:29:16 2024] R13: 000000000000ea60 R14: 0000000000000026 R15: 000055f87dd5b760
[Mon Mar 11 14:29:16 2024] </TASK>
[Mon Mar 11 14:29:16 2024] task:smbd[10.46.198. state:S stack:0 pid:2368564 ppid:4757 flags:0x00000002
[Mon Mar 11 14:29:16 2024] Call Trace:
[Mon Mar 11 14:29:16 2024] <TASK>
[Mon Mar 11 14:29:16 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:16 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:16 2024] schedule_hrtimeout_range_clock+0x95/0x120
[Mon Mar 11 14:29:16 2024] ? __pfx_hrtimer_wakeup+0x10/0x10
[Mon Mar 11 14:29:16 2024] ep_poll+0x348/0x3b0
[Mon Mar 11 14:29:16 2024] ? __pfx_autoremove_wake_function+0x10/0x10
[Mon Mar 11 14:29:16 2024] do_epoll_wait+0xb1/0xd0
[Mon Mar 11 14:29:16 2024] __x64_sys_epoll_wait+0x60/0x100
[Mon Mar 11 14:29:16 2024] do_syscall_64+0x59/0x90
[Mon Mar 11 14:29:16 2024] ? audit_reset_context.part.0.constprop.0+0x273/0x2e0
[Mon Mar 11 14:29:16 2024] ? syscall_exit_work+0x103/0x130
[Mon Mar 11 14:29:16 2024] ? syscall_exit_to_user_mode+0x22/0x40
[Mon Mar 11 14:29:16 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:16 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:16 2024] ? sysvec_apic_timer_interrupt+0x3c/0x90
[Mon Mar 11 14:29:16 2024] entry_SYSCALL_64_after_hwframe+0x72/0xdc
[Mon Mar 11 14:29:16 2024] RIP: 0033:0x7fedb7d4e80a
[Mon Mar 11 14:29:16 2024] RSP: 002b:00007ffc284a9ad8 EFLAGS: 00000246 ORIG_RAX: 00000000000000e8
[Mon Mar 11 14:29:16 2024] RAX: ffffffffffffffda RBX: 000055f87dd397c0 RCX: 00007fedb7d4e80a
[Mon Mar 11 14:29:16 2024] RDX: 0000000000000001 RSI: 00007ffc284a9b0c RDI: 0000000000000005
[Mon Mar 11 14:29:16 2024] RBP: 000055f87dd2aa20 R08: 00000000000f41d0 R09: 0000000000000000
[Mon Mar 11 14:29:16 2024] R10: 000000000000ea60 R11: 0000000000000246 R12: 000055f87dd61190
[Mon Mar 11 14:29:16 2024] R13: 000000000000ea60 R14: 0000000000000026 R15: 000055f87dd5b760
[Mon Mar 11 14:29:16 2024] </TASK>
[Mon Mar 11 14:29:16 2024] task:smbd[10.87.27.2 state:S stack:0 pid:2370451 ppid:4757 flags:0x00000002
[Mon Mar 11 14:29:16 2024] Call Trace:
[Mon Mar 11 14:29:16 2024] <TASK>
[Mon Mar 11 14:29:16 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:16 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:16 2024] schedule_hrtimeout_range_clock+0x95/0x120
[Mon Mar 11 14:29:16 2024] ? __pfx_hrtimer_wakeup+0x10/0x10
[Mon Mar 11 14:29:16 2024] ep_poll+0x348/0x3b0
[Mon Mar 11 14:29:16 2024] ? __pfx_autoremove_wake_function+0x10/0x10
[Mon Mar 11 14:29:16 2024] do_epoll_wait+0xb1/0xd0
[Mon Mar 11 14:29:16 2024] __x64_sys_epoll_wait+0x60/0x100
[Mon Mar 11 14:29:16 2024] do_syscall_64+0x59/0x90
[Mon Mar 11 14:29:16 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:16 2024] ? syscall_exit_work+0x103/0x130
[Mon Mar 11 14:29:16 2024] ? syscall_exit_to_user_mode+0x22/0x40
[Mon Mar 11 14:29:16 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:16 2024] ? exit_to_user_mode_prepare+0xb6/0x100
[Mon Mar 11 14:29:16 2024] ? syscall_exit_to_user_mode+0x22/0x40
[Mon Mar 11 14:29:16 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:16 2024] ? exc_page_fault+0x62/0x150
[Mon Mar 11 14:29:16 2024] entry_SYSCALL_64_after_hwframe+0x72/0xdc
[Mon Mar 11 14:29:16 2024] RIP: 0033:0x7fedb7d4e80a
[Mon Mar 11 14:29:16 2024] RSP: 002b:00007ffc284a9ad8 EFLAGS: 00000246 ORIG_RAX: 00000000000000e8
[Mon Mar 11 14:29:16 2024] RAX: ffffffffffffffda RBX: 000055f87dd397c0 RCX: 00007fedb7d4e80a
[Mon Mar 11 14:29:16 2024] RDX: 0000000000000001 RSI: 00007ffc284a9b0c RDI: 0000000000000005
[Mon Mar 11 14:29:16 2024] RBP: 000055f87dd2aa20 R08: 00000000000f4072 R09: 0000000000000000
[Mon Mar 11 14:29:16 2024] R10: 000000000000ea60 R11: 0000000000000246 R12: 000055f87dd61190
[Mon Mar 11 14:29:16 2024] R13: 000000000000ea60 R14: 0000000000000026 R15: 000055f87dd5b760
[Mon Mar 11 14:29:16 2024] </TASK>
[Mon Mar 11 14:29:16 2024] task:smbd[10.87.29.6 state:S stack:0 pid:2371619 ppid:4757 flags:0x00000002
[Mon Mar 11 14:29:16 2024] Call Trace:
[Mon Mar 11 14:29:16 2024] <TASK>
[Mon Mar 11 14:29:16 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:16 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:16 2024] schedule_hrtimeout_range_clock+0x95/0x120
[Mon Mar 11 14:29:16 2024] ? __pfx_hrtimer_wakeup+0x10/0x10
[Mon Mar 11 14:29:16 2024] ep_poll+0x348/0x3b0
[Mon Mar 11 14:29:16 2024] ? __pfx_autoremove_wake_function+0x10/0x10
[Mon Mar 11 14:29:16 2024] do_epoll_wait+0xb1/0xd0
[Mon Mar 11 14:29:16 2024] __x64_sys_epoll_wait+0x60/0x100
[Mon Mar 11 14:29:16 2024] do_syscall_64+0x59/0x90
[Mon Mar 11 14:29:16 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:16 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:16 2024] ? syscall_exit_to_user_mode+0x22/0x40
[Mon Mar 11 14:29:16 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:16 2024] ? syscall_exit_to_user_mode+0x22/0x40
[Mon Mar 11 14:29:16 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:16 2024] ? sysvec_apic_timer_interrupt+0x3c/0x90
[Mon Mar 11 14:29:16 2024] entry_SYSCALL_64_after_hwframe+0x72/0xdc
[Mon Mar 11 14:29:16 2024] RIP: 0033:0x7fedb7d4e84e
[Mon Mar 11 14:29:16 2024] RSP: 002b:00007ffc284a9ab0 EFLAGS: 00000293 ORIG_RAX: 00000000000000e8
[Mon Mar 11 14:29:16 2024] RAX: ffffffffffffffda RBX: 000055f87dd397c0 RCX: 00007fedb7d4e84e
[Mon Mar 11 14:29:16 2024] RDX: 0000000000000001 RSI: 00007ffc284a9b0c RDI: 0000000000000005
[Mon Mar 11 14:29:16 2024] RBP: 000055f87dd2aa20 R08: 0000000000000000 R09: 0000000000000000
[Mon Mar 11 14:29:16 2024] R10: 00000000000043b3 R11: 0000000000000293 R12: 000055f87dd61190
[Mon Mar 11 14:29:16 2024] R13: 00000000000043b3 R14: 0000000000000026 R15: 000055f87dd5b760
[Mon Mar 11 14:29:16 2024] </TASK>
[Mon Mar 11 14:29:16 2024] task:sshd state:S stack:0 pid:2371675 ppid:4412 flags:0x00000002
[Mon Mar 11 14:29:16 2024] Call Trace:
[Mon Mar 11 14:29:16 2024] <TASK>
[Mon Mar 11 14:29:16 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:16 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:16 2024] schedule_hrtimeout_range_clock+0x10f/0x120
[Mon Mar 11 14:29:16 2024] ? add_wait_queue+0x65/0xa0
[Mon Mar 11 14:29:16 2024] ? unix_poll+0xf4/0x100
[Mon Mar 11 14:29:16 2024] ? sock_poll+0x4c/0xe0
[Mon Mar 11 14:29:16 2024] do_poll.constprop.0+0x248/0x390
[Mon Mar 11 14:29:16 2024] do_sys_poll+0x1c8/0x250
[Mon Mar 11 14:29:16 2024] ? __kmem_cache_alloc_node+0x1c7/0x2d0
[Mon Mar 11 14:29:16 2024] ? __alloc_skb+0x8e/0x1d0
[Mon Mar 11 14:29:16 2024] ? _copy_to_iter+0x1d4/0x630
[Mon Mar 11 14:29:16 2024] ? __check_object_size.part.0+0x47/0xd0
[Mon Mar 11 14:29:16 2024] ? __pfx_pollwake+0x10/0x10
[Mon Mar 11 14:29:16 2024] ? __skb_datagram_iter+0x79/0x2e0
[Mon Mar 11 14:29:16 2024] ? __pfx_simple_copy_to_iter+0x10/0x10
[Mon Mar 11 14:29:16 2024] ? __kmem_cache_alloc_node+0x1c7/0x2d0
[Mon Mar 11 14:29:16 2024] ? kmalloc_trace+0x25/0xa0
[Mon Mar 11 14:29:16 2024] ? rcu_nocb_try_bypass+0x5e/0x460
[Mon Mar 11 14:29:16 2024] ? __check_object_size.part.0+0x35/0xd0
[Mon Mar 11 14:29:16 2024] ? rcu_nocb_try_bypass+0x5e/0x460
[Mon Mar 11 14:29:16 2024] ? __pfx_deferred_put_nlk_sk+0x10/0x10
[Mon Mar 11 14:29:16 2024] ? rcu_nocb_try_bypass+0x5e/0x460
[Mon Mar 11 14:29:16 2024] ? __pfx_inode_free_by_rcu+0x10/0x10
[Mon Mar 11 14:29:16 2024] ? rseq_get_rseq_cs+0x1d/0x240
[Mon Mar 11 14:29:16 2024] ? __dentry_kill+0x13a/0x180
[Mon Mar 11 14:29:16 2024] ? rseq_ip_fixup+0x6e/0x1a0
[Mon Mar 11 14:29:16 2024] ? auditd_test_task+0x3c/0x50
[Mon Mar 11 14:29:16 2024] ? __audit_syscall_entry+0xef/0x140
[Mon Mar 11 14:29:16 2024] __x64_sys_poll+0x39/0x140
[Mon Mar 11 14:29:16 2024] ? syscall_trace_enter.constprop.0+0x126/0x1a0
[Mon Mar 11 14:29:16 2024] do_syscall_64+0x59/0x90
[Mon Mar 11 14:29:16 2024] ? syscall_exit_to_user_mode+0x22/0x40
[Mon Mar 11 14:29:16 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:16 2024] ? irqtime_account_irq+0x3c/0xb0
[Mon Mar 11 14:29:16 2024] ? __irq_exit_rcu+0x46/0xc0
[Mon Mar 11 14:29:16 2024] ? sysvec_apic_timer_interrupt+0x3c/0x90
[Mon Mar 11 14:29:16 2024] entry_SYSCALL_64_after_hwframe+0x72/0xdc
[Mon Mar 11 14:29:16 2024] RIP: 0033:0x7fdd8e1426c7
[Mon Mar 11 14:29:16 2024] RSP: 002b:00007ffc4bb859f8 EFLAGS: 00000246 ORIG_RAX: 0000000000000007
[Mon Mar 11 14:29:16 2024] RAX: ffffffffffffffda RBX: 00007ffc4bb85a10 RCX: 00007fdd8e1426c7
[Mon Mar 11 14:29:16 2024] RDX: 00000000ffffffff RSI: 0000000000000001 RDI: 00007ffc4bb85a10
[Mon Mar 11 14:29:16 2024] RBP: 000055bad15460c0 R08: 0000000000000001 R09: 00007ffc4bb7ed28
[Mon Mar 11 14:29:16 2024] R10: 0000000000000040 R11: 0000000000000246 R12: 000055bad15460c0
[Mon Mar 11 14:29:16 2024] R13: 000055bad20ed680 R14: 000055bad20ec910 R15: 00000000ffffffff
[Mon Mar 11 14:29:16 2024] </TASK>
[Mon Mar 11 14:29:16 2024] task:sshd state:S stack:0 pid:2371682 ppid:2371675 flags:0x00000002
[Mon Mar 11 14:29:16 2024] Call Trace:
[Mon Mar 11 14:29:16 2024] <TASK>
[Mon Mar 11 14:29:16 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:16 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:16 2024] schedule_hrtimeout_range_clock+0x10f/0x120
[Mon Mar 11 14:29:16 2024] ? n_tty_poll+0x1cb/0x1f0
[Mon Mar 11 14:29:16 2024] ? tty_poll+0x71/0xa0
[Mon Mar 11 14:29:16 2024] do_select+0x69e/0x7c0
[Mon Mar 11 14:29:16 2024] ? __bond_start_xmit+0x93/0x430 [bonding]
[Mon Mar 11 14:29:16 2024] ? __pfx_pollwake+0x10/0x10
[Mon Mar 11 14:29:16 2024] ? __pfx_pollwake+0x10/0x10
[Mon Mar 11 14:29:16 2024] ? __pfx_pollwake+0x10/0x10
[Mon Mar 11 14:29:16 2024] ? __mod_timer+0x286/0x3d0
[Mon Mar 11 14:29:16 2024] ? sk_reset_timer+0x14/0x60
[Mon Mar 11 14:29:16 2024] ? tcp_schedule_loss_probe.part.0+0x12c/0x1a0
[Mon Mar 11 14:29:16 2024] ? tcp_write_xmit+0x6f5/0xaa0
[Mon Mar 11 14:29:16 2024] ? __tcp_push_pending_frames+0x32/0xf0
[Mon Mar 11 14:29:16 2024] ? tcp_sendmsg_locked+0xaae/0xc20
[Mon Mar 11 14:29:16 2024] core_sys_select+0x1a0/0x3b0
[Mon Mar 11 14:29:16 2024] ? __audit_filter_op+0xa5/0xf0
[Mon Mar 11 14:29:16 2024] ? _copy_from_user+0x27/0x60
[Mon Mar 11 14:29:16 2024] do_pselect.constprop.0+0xca/0x170
[Mon Mar 11 14:29:16 2024] __x64_sys_pselect6+0x39/0x70
[Mon Mar 11 14:29:16 2024] do_syscall_64+0x59/0x90
[Mon Mar 11 14:29:16 2024] ? __irq_exit_rcu+0x46/0xc0
[Mon Mar 11 14:29:16 2024] ? common_interrupt+0x43/0xa0
[Mon Mar 11 14:29:16 2024] entry_SYSCALL_64_after_hwframe+0x72/0xdc
[Mon Mar 11 14:29:16 2024] RIP: 0033:0x7fdd8e144f64
[Mon Mar 11 14:29:16 2024] RSP: 002b:00007ffc4bb856b0 EFLAGS: 00000246 ORIG_RAX: 000000000000010e
[Mon Mar 11 14:29:16 2024] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007fdd8e144f64
[Mon Mar 11 14:29:16 2024] RDX: 000055bad2111b80 RSI: 000055bad213f450 RDI: 000000000000000f
[Mon Mar 11 14:29:16 2024] RBP: 000055bad213f450 R08: 0000000000000000 R09: 00007ffc4bb856f0
[Mon Mar 11 14:29:16 2024] R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000000008
[Mon Mar 11 14:29:16 2024] R13: 000055bad20ef430 R14: 000055bad2111b80 R15: 000055bad20ed680
[Mon Mar 11 14:29:16 2024] </TASK>
[Mon Mar 11 14:29:16 2024] task:bash state:S stack:0 pid:2371683 ppid:2371682 flags:0x00000002
[Mon Mar 11 14:29:16 2024] Call Trace:
[Mon Mar 11 14:29:16 2024] <TASK>
[Mon Mar 11 14:29:16 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:16 2024] ? schedule_hrtimeout_range_clock+0x10f/0x120
[Mon Mar 11 14:29:16 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:16 2024] schedule_hrtimeout_range_clock+0x10f/0x120
[Mon Mar 11 14:29:16 2024] ? n_tty_poll+0x1cb/0x1f0
[Mon Mar 11 14:29:16 2024] ? tty_poll+0x71/0xa0
[Mon Mar 11 14:29:16 2024] do_select+0x69e/0x7c0
[Mon Mar 11 14:29:16 2024] ? __pfx_pollwake+0x10/0x10
[Mon Mar 11 14:29:16 2024] ? __pfx_pollwake+0x10/0x10
[Mon Mar 11 14:29:16 2024] ? select_idle_sibling+0x28/0x430
[Mon Mar 11 14:29:16 2024] ? wake_affine+0x62/0x1f0
[Mon Mar 11 14:29:16 2024] ? sched_clock+0xc/0x30
[Mon Mar 11 14:29:16 2024] ? __smp_call_single_queue+0x93/0x120
[Mon Mar 11 14:29:16 2024] ? select_idle_sibling+0x28/0x430
[Mon Mar 11 14:29:16 2024] ? ttwu_queue_wakelist+0xf2/0x110
[Mon Mar 11 14:29:16 2024] ? wake_affine+0x62/0x1f0
[Mon Mar 11 14:29:16 2024] ? sched_clock+0xc/0x30
[Mon Mar 11 14:29:16 2024] ? __smp_call_single_queue+0x93/0x120
[Mon Mar 11 14:29:16 2024] ? select_idle_sibling+0x28/0x430
[Mon Mar 11 14:29:16 2024] ? select_idle_sibling+0x28/0x430
[Mon Mar 11 14:29:16 2024] ? ttwu_queue_wakelist+0xf2/0x110
[Mon Mar 11 14:29:16 2024] ? wake_affine+0x62/0x1f0
[Mon Mar 11 14:29:16 2024] ? wake_affine+0x62/0x1f0
[Mon Mar 11 14:29:16 2024] ? sched_clock+0xc/0x30
[Mon Mar 11 14:29:16 2024] ? __smp_call_single_queue+0x93/0x120
[Mon Mar 11 14:29:16 2024] ? ttwu_queue_wakelist+0xf2/0x110
[Mon Mar 11 14:29:16 2024] ? try_to_wake_up+0x3e2/0x5d0
[Mon Mar 11 14:29:16 2024] core_sys_select+0x1a0/0x3b0
[Mon Mar 11 14:29:16 2024] ? n_tty_write+0x267/0x3c0
[Mon Mar 11 14:29:16 2024] ? __wake_up+0x40/0x60
[Mon Mar 11 14:29:16 2024] ? do_tty_write+0x1a4/0x250
[Mon Mar 11 14:29:16 2024] ? __pfx_n_tty_write+0x10/0x10
[Mon Mar 11 14:29:16 2024] ? file_has_perm+0xc6/0xd0
[Mon Mar 11 14:29:16 2024] ? _copy_from_user+0x27/0x60
[Mon Mar 11 14:29:16 2024] do_pselect.constprop.0+0xca/0x170
[Mon Mar 11 14:29:16 2024] __x64_sys_pselect6+0x39/0x70
[Mon Mar 11 14:29:16 2024] do_syscall_64+0x59/0x90
[Mon Mar 11 14:29:16 2024] ? syscall_exit_work+0x103/0x130
[Mon Mar 11 14:29:16 2024] ? syscall_exit_to_user_mode+0x22/0x40
[Mon Mar 11 14:29:16 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:16 2024] ? syscall_exit_work+0x103/0x130
[Mon Mar 11 14:29:16 2024] ? syscall_exit_to_user_mode+0x22/0x40
[Mon Mar 11 14:29:16 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:16 2024] ? sysvec_apic_timer_interrupt+0x3c/0x90
[Mon Mar 11 14:29:16 2024] entry_SYSCALL_64_after_hwframe+0x72/0xdc
[Mon Mar 11 14:29:16 2024] RIP: 0033:0x7ff463344f64
[Mon Mar 11 14:29:16 2024] RSP: 002b:00007ffdf9661b90 EFLAGS: 00000246 ORIG_RAX: 000000000000010e
[Mon Mar 11 14:29:16 2024] RAX: ffffffffffffffda RBX: 00007ff4633faaa0 RCX: 00007ff463344f64
[Mon Mar 11 14:29:16 2024] RDX: 0000000000000000 RSI: 00007ffdf9661c20 RDI: 0000000000000001
[Mon Mar 11 14:29:16 2024] RBP: 00007ffdf9661c20 R08: 0000000000000000 R09: 00007ffdf9661bd0
[Mon Mar 11 14:29:16 2024] R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000000000
[Mon Mar 11 14:29:16 2024] R13: 0000000000000000 R14: 0000000000000000 R15: 000055745bd3f580
[Mon Mar 11 14:29:16 2024] </TASK>
[Mon Mar 11 14:29:16 2024] task:smbd[10.87.27.8 state:S stack:0 pid:2372510 ppid:4757 flags:0x00000002
[Mon Mar 11 14:29:16 2024] Call Trace:
[Mon Mar 11 14:29:16 2024] <TASK>
[Mon Mar 11 14:29:16 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:16 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:16 2024] schedule_hrtimeout_range_clock+0x95/0x120
[Mon Mar 11 14:29:16 2024] ? __pfx_hrtimer_wakeup+0x10/0x10
[Mon Mar 11 14:29:16 2024] ep_poll+0x348/0x3b0
[Mon Mar 11 14:29:16 2024] ? __pfx_autoremove_wake_function+0x10/0x10
[Mon Mar 11 14:29:16 2024] do_epoll_wait+0xb1/0xd0
[Mon Mar 11 14:29:16 2024] __x64_sys_epoll_wait+0x60/0x100
[Mon Mar 11 14:29:16 2024] do_syscall_64+0x59/0x90
[Mon Mar 11 14:29:16 2024] ? syscall_exit_work+0x103/0x130
[Mon Mar 11 14:29:16 2024] ? syscall_exit_to_user_mode+0x22/0x40
[Mon Mar 11 14:29:16 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:16 2024] ? syscall_exit_to_user_mode+0x22/0x40
[Mon Mar 11 14:29:16 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:16 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:16 2024] entry_SYSCALL_64_after_hwframe+0x72/0xdc
[Mon Mar 11 14:29:16 2024] RIP: 0033:0x7fedb7d4e80a
[Mon Mar 11 14:29:16 2024] RSP: 002b:00007ffc284a9ad8 EFLAGS: 00000246 ORIG_RAX: 00000000000000e8
[Mon Mar 11 14:29:16 2024] RAX: ffffffffffffffda RBX: 000055f87dd397c0 RCX: 00007fedb7d4e80a
[Mon Mar 11 14:29:16 2024] RDX: 0000000000000001 RSI: 00007ffc284a9b0c RDI: 0000000000000005
[Mon Mar 11 14:29:16 2024] RBP: 000055f87dd2aa20 R08: 00000000000f4073 R09: 0000000000000000
[Mon Mar 11 14:29:16 2024] R10: 000000000000ea60 R11: 0000000000000246 R12: 000055f87dd61190
[Mon Mar 11 14:29:16 2024] R13: 000000000000ea60 R14: 0000000000000026 R15: 000055f87dd5b760
[Mon Mar 11 14:29:16 2024] </TASK>
[Mon Mar 11 14:29:16 2024] task:kworker/14:1 state:I stack:0 pid:2374246 ppid:2 flags:0x00004000
[Mon Mar 11 14:29:16 2024] Workqueue: 0x0 (rcu_gp)
[Mon Mar 11 14:29:16 2024] Call Trace:
[Mon Mar 11 14:29:16 2024] <TASK>
[Mon Mar 11 14:29:16 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:16 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:16 2024] worker_thread+0xbb/0x3a0
[Mon Mar 11 14:29:16 2024] ? __pfx_worker_thread+0x10/0x10
[Mon Mar 11 14:29:16 2024] kthread+0xdd/0x100
[Mon Mar 11 14:29:16 2024] ? __pfx_kthread+0x10/0x10
[Mon Mar 11 14:29:16 2024] ret_from_fork+0x29/0x50
[Mon Mar 11 14:29:16 2024] </TASK>
[Mon Mar 11 14:29:16 2024] task:smbd[10.87.30.6 state:S stack:0 pid:2376116 ppid:4757 flags:0x00000002
[Mon Mar 11 14:29:16 2024] Call Trace:
[Mon Mar 11 14:29:16 2024] <TASK>
[Mon Mar 11 14:29:16 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:16 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:16 2024] schedule_hrtimeout_range_clock+0x95/0x120
[Mon Mar 11 14:29:16 2024] ? __pfx_hrtimer_wakeup+0x10/0x10
[Mon Mar 11 14:29:16 2024] ep_poll+0x348/0x3b0
[Mon Mar 11 14:29:16 2024] ? __pfx_autoremove_wake_function+0x10/0x10
[Mon Mar 11 14:29:16 2024] do_epoll_wait+0xb1/0xd0
[Mon Mar 11 14:29:16 2024] __x64_sys_epoll_wait+0x60/0x100
[Mon Mar 11 14:29:16 2024] do_syscall_64+0x59/0x90
[Mon Mar 11 14:29:16 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:16 2024] ? exc_page_fault+0x62/0x150
[Mon Mar 11 14:29:16 2024] entry_SYSCALL_64_after_hwframe+0x72/0xdc
[Mon Mar 11 14:29:16 2024] RIP: 0033:0x7fedb7d4e84e
[Mon Mar 11 14:29:16 2024] RSP: 002b:00007ffc284a9ab0 EFLAGS: 00000293 ORIG_RAX: 00000000000000e8
[Mon Mar 11 14:29:16 2024] RAX: ffffffffffffffda RBX: 000055f87dd397c0 RCX: 00007fedb7d4e84e
[Mon Mar 11 14:29:16 2024] RDX: 0000000000000001 RSI: 00007ffc284a9b0c RDI: 0000000000000005
[Mon Mar 11 14:29:16 2024] RBP: 000055f87dd2aa20 R08: 0000000000000000 R09: 0000000000000000
[Mon Mar 11 14:29:16 2024] R10: 000000000000a242 R11: 0000000000000293 R12: 000055f87dd61190
[Mon Mar 11 14:29:16 2024] R13: 000000000000a242 R14: 0000000000000026 R15: 000055f87dd5b760
[Mon Mar 11 14:29:16 2024] </TASK>
[Mon Mar 11 14:29:16 2024] task:smbd[10.87.27.1 state:S stack:0 pid:2376851 ppid:4757 flags:0x00000002
[Mon Mar 11 14:29:16 2024] Call Trace:
[Mon Mar 11 14:29:16 2024] <TASK>
[Mon Mar 11 14:29:16 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:16 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:16 2024] schedule_hrtimeout_range_clock+0x95/0x120
[Mon Mar 11 14:29:16 2024] ? __pfx_hrtimer_wakeup+0x10/0x10
[Mon Mar 11 14:29:16 2024] ep_poll+0x348/0x3b0
[Mon Mar 11 14:29:16 2024] ? __pfx_autoremove_wake_function+0x10/0x10
[Mon Mar 11 14:29:16 2024] do_epoll_wait+0xb1/0xd0
[Mon Mar 11 14:29:16 2024] __x64_sys_epoll_wait+0x60/0x100
[Mon Mar 11 14:29:16 2024] do_syscall_64+0x59/0x90
[Mon Mar 11 14:29:16 2024] ? audit_reset_context.part.0.constprop.0+0x273/0x2e0
[Mon Mar 11 14:29:16 2024] ? syscall_exit_work+0x103/0x130
[Mon Mar 11 14:29:16 2024] ? syscall_exit_to_user_mode+0x22/0x40
[Mon Mar 11 14:29:16 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:16 2024] entry_SYSCALL_64_after_hwframe+0x72/0xdc
[Mon Mar 11 14:29:16 2024] RIP: 0033:0x7fedb7d4e80a
[Mon Mar 11 14:29:16 2024] RSP: 002b:00007ffc284a9ad8 EFLAGS: 00000246 ORIG_RAX: 00000000000000e8
[Mon Mar 11 14:29:16 2024] RAX: ffffffffffffffda RBX: 000055f87dd397c0 RCX: 00007fedb7d4e80a
[Mon Mar 11 14:29:16 2024] RDX: 0000000000000001 RSI: 00007ffc284a9b0c RDI: 0000000000000005
[Mon Mar 11 14:29:16 2024] RBP: 000055f87dd2aa20 R08: 00000000000f4078 R09: 0000000000000000
[Mon Mar 11 14:29:16 2024] R10: 000000000000ea60 R11: 0000000000000246 R12: 000055f87dd61190
[Mon Mar 11 14:29:16 2024] R13: 000000000000ea60 R14: 0000000000000026 R15: 000055f87dd5b760
[Mon Mar 11 14:29:16 2024] </TASK>
[Mon Mar 11 14:29:16 2024] task:smbd[10.40.147. state:S stack:0 pid:2377087 ppid:4757 flags:0x00000002
[Mon Mar 11 14:29:16 2024] Call Trace:
[Mon Mar 11 14:29:16 2024] <TASK>
[Mon Mar 11 14:29:16 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:16 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:16 2024] schedule_hrtimeout_range_clock+0x95/0x120
[Mon Mar 11 14:29:16 2024] ? __pfx_hrtimer_wakeup+0x10/0x10
[Mon Mar 11 14:29:16 2024] ep_poll+0x348/0x3b0
[Mon Mar 11 14:29:16 2024] ? __pfx_autoremove_wake_function+0x10/0x10
[Mon Mar 11 14:29:16 2024] do_epoll_wait+0xb1/0xd0
[Mon Mar 11 14:29:16 2024] __x64_sys_epoll_wait+0x60/0x100
[Mon Mar 11 14:29:16 2024] do_syscall_64+0x59/0x90
[Mon Mar 11 14:29:16 2024] ? audit_reset_context.part.0.constprop.0+0x273/0x2e0
[Mon Mar 11 14:29:16 2024] ? syscall_exit_work+0x103/0x130
[Mon Mar 11 14:29:16 2024] ? syscall_exit_to_user_mode+0x22/0x40
[Mon Mar 11 14:29:16 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:16 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:16 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:16 2024] entry_SYSCALL_64_after_hwframe+0x72/0xdc
[Mon Mar 11 14:29:16 2024] RIP: 0033:0x7fedb7d4e84e
[Mon Mar 11 14:29:16 2024] RSP: 002b:00007ffc284a9ab0 EFLAGS: 00000293 ORIG_RAX: 00000000000000e8
[Mon Mar 11 14:29:16 2024] RAX: ffffffffffffffda RBX: 000055f87dd397c0 RCX: 00007fedb7d4e84e
[Mon Mar 11 14:29:16 2024] RDX: 0000000000000001 RSI: 00007ffc284a9b0c RDI: 0000000000000005
[Mon Mar 11 14:29:16 2024] RBP: 000055f87dd2aa20 R08: 0000000000000000 R09: 0000000000000000
[Mon Mar 11 14:29:16 2024] R10: 000000000000ea60 R11: 0000000000000293 R12: 000055f87dd58980
[Mon Mar 11 14:29:16 2024] R13: 000000000000ea60 R14: 0000000000000027 R15: 000055f87dd5b760
[Mon Mar 11 14:29:16 2024] </TASK>
[Mon Mar 11 14:29:16 2024] task:smbd[10.87.29.1 state:S stack:0 pid:2377277 ppid:4757 flags:0x00000002
[Mon Mar 11 14:29:16 2024] Call Trace:
[Mon Mar 11 14:29:16 2024] <TASK>
[Mon Mar 11 14:29:16 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:16 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:16 2024] schedule_hrtimeout_range_clock+0x95/0x120
[Mon Mar 11 14:29:16 2024] ? __pfx_hrtimer_wakeup+0x10/0x10
[Mon Mar 11 14:29:16 2024] ep_poll+0x348/0x3b0
[Mon Mar 11 14:29:16 2024] ? __pfx_autoremove_wake_function+0x10/0x10
[Mon Mar 11 14:29:16 2024] do_epoll_wait+0xb1/0xd0
[Mon Mar 11 14:29:16 2024] __x64_sys_epoll_wait+0x60/0x100
[Mon Mar 11 14:29:16 2024] do_syscall_64+0x59/0x90
[Mon Mar 11 14:29:16 2024] ? syscall_exit_work+0x103/0x130
[Mon Mar 11 14:29:16 2024] ? syscall_exit_to_user_mode+0x22/0x40
[Mon Mar 11 14:29:16 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:16 2024] ? syscall_exit_to_user_mode+0x22/0x40
[Mon Mar 11 14:29:16 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:16 2024] entry_SYSCALL_64_after_hwframe+0x72/0xdc
[Mon Mar 11 14:29:16 2024] RIP: 0033:0x7fedb7d4e80a
[Mon Mar 11 14:29:16 2024] RSP: 002b:00007ffc284a9ad8 EFLAGS: 00000246 ORIG_RAX: 00000000000000e8
[Mon Mar 11 14:29:16 2024] RAX: ffffffffffffffda RBX: 000055f87dd397c0 RCX: 00007fedb7d4e80a
[Mon Mar 11 14:29:16 2024] RDX: 0000000000000001 RSI: 00007ffc284a9b0c RDI: 0000000000000005
[Mon Mar 11 14:29:16 2024] RBP: 000055f87dd2aa20 R08: 00000000000f4209 R09: 0000000000000000
[Mon Mar 11 14:29:16 2024] R10: 000000000000ea60 R11: 0000000000000246 R12: 000055f87dd61190
[Mon Mar 11 14:29:16 2024] R13: 000000000000ea60 R14: 0000000000000026 R15: 000055f87dd5b760
[Mon Mar 11 14:29:16 2024] </TASK>
[Mon Mar 11 14:29:16 2024] task:smbd[10.33.88.1 state:S stack:0 pid:2377288 ppid:4757 flags:0x00000002
[Mon Mar 11 14:29:16 2024] Call Trace:
[Mon Mar 11 14:29:16 2024] <TASK>
[Mon Mar 11 14:29:16 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:16 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:16 2024] schedule_hrtimeout_range_clock+0x95/0x120
[Mon Mar 11 14:29:16 2024] ? __pfx_hrtimer_wakeup+0x10/0x10
[Mon Mar 11 14:29:16 2024] ep_poll+0x348/0x3b0
[Mon Mar 11 14:29:16 2024] ? __pfx_autoremove_wake_function+0x10/0x10
[Mon Mar 11 14:29:16 2024] do_epoll_wait+0xb1/0xd0
[Mon Mar 11 14:29:16 2024] __x64_sys_epoll_wait+0x60/0x100
[Mon Mar 11 14:29:16 2024] do_syscall_64+0x59/0x90
[Mon Mar 11 14:29:16 2024] ? audit_reset_context.part.0.constprop.0+0x273/0x2e0
[Mon Mar 11 14:29:16 2024] ? syscall_exit_work+0x103/0x130
[Mon Mar 11 14:29:16 2024] ? syscall_exit_to_user_mode+0x22/0x40
[Mon Mar 11 14:29:16 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:16 2024] ? syscall_exit_work+0x103/0x130
[Mon Mar 11 14:29:16 2024] ? syscall_exit_to_user_mode+0x22/0x40
[Mon Mar 11 14:29:16 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:16 2024] entry_SYSCALL_64_after_hwframe+0x72/0xdc
[Mon Mar 11 14:29:16 2024] RIP: 0033:0x7fedb7d4e80a
[Mon Mar 11 14:29:16 2024] RSP: 002b:00007ffc284a9ad8 EFLAGS: 00000246 ORIG_RAX: 00000000000000e8
[Mon Mar 11 14:29:16 2024] RAX: ffffffffffffffda RBX: 000055f87dd397c0 RCX: 00007fedb7d4e80a
[Mon Mar 11 14:29:16 2024] RDX: 0000000000000001 RSI: 00007ffc284a9b0c RDI: 0000000000000005
[Mon Mar 11 14:29:16 2024] RBP: 000055f87dd2aa20 R08: 00000000000f41e2 R09: 0000000000000077
[Mon Mar 11 14:29:16 2024] R10: 000000000000ea60 R11: 0000000000000246 R12: 000055f87dd61190
[Mon Mar 11 14:29:16 2024] R13: 000000000000ea60 R14: 0000000000000026 R15: 000055f87dd5b760
[Mon Mar 11 14:29:16 2024] </TASK>
[Mon Mar 11 14:29:16 2024] task:smbd[10.40.138. state:S stack:0 pid:2377602 ppid:4757 flags:0x00000002
[Mon Mar 11 14:29:16 2024] Call Trace:
[Mon Mar 11 14:29:16 2024] <TASK>
[Mon Mar 11 14:29:16 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:16 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:16 2024] schedule_hrtimeout_range_clock+0x95/0x120
[Mon Mar 11 14:29:16 2024] ? __pfx_hrtimer_wakeup+0x10/0x10
[Mon Mar 11 14:29:16 2024] ep_poll+0x348/0x3b0
[Mon Mar 11 14:29:16 2024] ? __pfx_autoremove_wake_function+0x10/0x10
[Mon Mar 11 14:29:16 2024] do_epoll_wait+0xb1/0xd0
[Mon Mar 11 14:29:16 2024] __x64_sys_epoll_wait+0x60/0x100
[Mon Mar 11 14:29:16 2024] do_syscall_64+0x59/0x90
[Mon Mar 11 14:29:16 2024] ? syscall_exit_work+0x103/0x130
[Mon Mar 11 14:29:16 2024] ? syscall_exit_to_user_mode+0x22/0x40
[Mon Mar 11 14:29:16 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:16 2024] ? syscall_exit_work+0x103/0x130
[Mon Mar 11 14:29:16 2024] ? syscall_exit_to_user_mode+0x22/0x40
[Mon Mar 11 14:29:16 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:16 2024] entry_SYSCALL_64_after_hwframe+0x72/0xdc
[Mon Mar 11 14:29:16 2024] RIP: 0033:0x7fedb7d4e80a
[Mon Mar 11 14:29:16 2024] RSP: 002b:00007ffc284a9ad8 EFLAGS: 00000246 ORIG_RAX: 00000000000000e8
[Mon Mar 11 14:29:16 2024] RAX: ffffffffffffffda RBX: 000055f87dd397c0 RCX: 00007fedb7d4e80a
[Mon Mar 11 14:29:16 2024] RDX: 0000000000000001 RSI: 00007ffc284a9b0c RDI: 0000000000000005
[Mon Mar 11 14:29:16 2024] RBP: 000055f87dd2aa20 R08: 00000000000f41d6 R09: 0000000000000000
[Mon Mar 11 14:29:16 2024] R10: 000000000000ea60 R11: 0000000000000246 R12: 000055f87dd61190
[Mon Mar 11 14:29:16 2024] R13: 000000000000ea60 R14: 0000000000000026 R15: 000055f87dd5b760
[Mon Mar 11 14:29:16 2024] </TASK>
[Mon Mar 11 14:29:16 2024] task:smbd[10.40.140. state:S stack:0 pid:2379578 ppid:4757 flags:0x00000002
[Mon Mar 11 14:29:16 2024] Call Trace:
[Mon Mar 11 14:29:16 2024] <TASK>
[Mon Mar 11 14:29:16 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:16 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:16 2024] schedule_hrtimeout_range_clock+0x10f/0x120
[Mon Mar 11 14:29:16 2024] ? unix_stream_read_generic+0x28b/0x6c0
[Mon Mar 11 14:29:16 2024] ep_poll+0x348/0x3b0
[Mon Mar 11 14:29:16 2024] ? __pfx_autoremove_wake_function+0x10/0x10
[Mon Mar 11 14:29:16 2024] do_epoll_wait+0xb1/0xd0
[Mon Mar 11 14:29:16 2024] __x64_sys_epoll_wait+0x60/0x100
[Mon Mar 11 14:29:16 2024] ? syscall_trace_enter.constprop.0+0x126/0x1a0
[Mon Mar 11 14:29:16 2024] do_syscall_64+0x59/0x90
[Mon Mar 11 14:29:16 2024] ? __audit_filter_op+0xa5/0xf0
[Mon Mar 11 14:29:16 2024] ? audit_reset_context.part.0.constprop.0+0x273/0x2e0
[Mon Mar 11 14:29:16 2024] ? syscall_exit_work+0x103/0x130
[Mon Mar 11 14:29:16 2024] ? syscall_exit_to_user_mode+0x22/0x40
[Mon Mar 11 14:29:16 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:16 2024] entry_SYSCALL_64_after_hwframe+0x72/0xdc
[Mon Mar 11 14:29:16 2024] RIP: 0033:0x7fedb7d4e84e
[Mon Mar 11 14:29:16 2024] RSP: 002b:00007ffc284a8d30 EFLAGS: 00000293 ORIG_RAX: 00000000000000e8
[Mon Mar 11 14:29:16 2024] RAX: ffffffffffffffda RBX: 000055f87ddfa890 RCX: 00007fedb7d4e84e
[Mon Mar 11 14:29:16 2024] RDX: 0000000000000008 RSI: 000055f87de08650 RDI: 0000000000000022
[Mon Mar 11 14:29:16 2024] RBP: 7fffffffffffffff R08: 0000000000000000 R09: 0000000000000000
[Mon Mar 11 14:29:16 2024] R10: 00000000ffffffff R11: 0000000000000293 R12: 0000000000000028
[Mon Mar 11 14:29:16 2024] R13: 0000000000000008 R14: 0000000000000004 R15: 000055f87ddfaa20
[Mon Mar 11 14:29:16 2024] </TASK>
[Mon Mar 11 14:29:16 2024] task:smbd[10.40.140. state:S stack:0 pid:2381553 ppid:4757 flags:0x00000002
[Mon Mar 11 14:29:16 2024] Call Trace:
[Mon Mar 11 14:29:16 2024] <TASK>
[Mon Mar 11 14:29:16 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:16 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:16 2024] schedule_hrtimeout_range_clock+0x95/0x120
[Mon Mar 11 14:29:16 2024] ? __pfx_hrtimer_wakeup+0x10/0x10
[Mon Mar 11 14:29:16 2024] ep_poll+0x348/0x3b0
[Mon Mar 11 14:29:16 2024] ? __pfx_autoremove_wake_function+0x10/0x10
[Mon Mar 11 14:29:16 2024] do_epoll_wait+0xb1/0xd0
[Mon Mar 11 14:29:16 2024] __x64_sys_epoll_wait+0x60/0x100
[Mon Mar 11 14:29:16 2024] do_syscall_64+0x59/0x90
[Mon Mar 11 14:29:16 2024] entry_SYSCALL_64_after_hwframe+0x72/0xdc
[Mon Mar 11 14:29:16 2024] RIP: 0033:0x7fedb7d4e80a
[Mon Mar 11 14:29:16 2024] RSP: 002b:00007ffc284a9ad8 EFLAGS: 00000246 ORIG_RAX: 00000000000000e8
[Mon Mar 11 14:29:16 2024] RAX: ffffffffffffffda RBX: 000055f87dd397c0 RCX: 00007fedb7d4e80a
[Mon Mar 11 14:29:16 2024] RDX: 0000000000000001 RSI: 00007ffc284a9b0c RDI: 0000000000000005
[Mon Mar 11 14:29:16 2024] RBP: 000055f87dd2aa20 R08: 00000000000f402f R09: 0000000000000000
[Mon Mar 11 14:29:16 2024] R10: 000000000000ea60 R11: 0000000000000246 R12: 000055f87dd58980
[Mon Mar 11 14:29:16 2024] R13: 000000000000ea60 R14: 0000000000000027 R15: 000055f87dd5b760
[Mon Mar 11 14:29:16 2024] </TASK>
[Mon Mar 11 14:29:16 2024] task:smbd[10.40.142. state:S stack:0 pid:2381864 ppid:4757 flags:0x00000002
[Mon Mar 11 14:29:16 2024] Call Trace:
[Mon Mar 11 14:29:16 2024] <TASK>
[Mon Mar 11 14:29:16 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:16 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:16 2024] schedule_hrtimeout_range_clock+0x95/0x120
[Mon Mar 11 14:29:16 2024] ? __pfx_hrtimer_wakeup+0x10/0x10
[Mon Mar 11 14:29:16 2024] ep_poll+0x348/0x3b0
[Mon Mar 11 14:29:16 2024] ? __pfx_autoremove_wake_function+0x10/0x10
[Mon Mar 11 14:29:16 2024] do_epoll_wait+0xb1/0xd0
[Mon Mar 11 14:29:16 2024] __x64_sys_epoll_wait+0x60/0x100
[Mon Mar 11 14:29:16 2024] do_syscall_64+0x59/0x90
[Mon Mar 11 14:29:16 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:16 2024] ? exc_page_fault+0x62/0x150
[Mon Mar 11 14:29:16 2024] entry_SYSCALL_64_after_hwframe+0x72/0xdc
[Mon Mar 11 14:29:16 2024] RIP: 0033:0x7fedb7d4e80a
[Mon Mar 11 14:29:16 2024] RSP: 002b:00007ffc284a9ad8 EFLAGS: 00000246 ORIG_RAX: 00000000000000e8
[Mon Mar 11 14:29:16 2024] RAX: ffffffffffffffda RBX: 000055f87dd397c0 RCX: 00007fedb7d4e80a
[Mon Mar 11 14:29:16 2024] RDX: 0000000000000001 RSI: 00007ffc284a9b0c RDI: 0000000000000005
[Mon Mar 11 14:29:16 2024] RBP: 000055f87dd2aa20 R08: 00000000000f41d8 R09: 0000000000000000
[Mon Mar 11 14:29:16 2024] R10: 000000000000ea60 R11: 0000000000000246 R12: 000055f87dd61190
[Mon Mar 11 14:29:16 2024] R13: 000000000000ea60 R14: 0000000000000026 R15: 000055f87dd5b760
[Mon Mar 11 14:29:16 2024] </TASK>
[Mon Mar 11 14:29:16 2024] task:smbd[10.40.133. state:S stack:0 pid:2384413 ppid:4757 flags:0x00000002
[Mon Mar 11 14:29:16 2024] Call Trace:
[Mon Mar 11 14:29:16 2024] <TASK>
[Mon Mar 11 14:29:16 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:16 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:16 2024] schedule_hrtimeout_range_clock+0x95/0x120
[Mon Mar 11 14:29:16 2024] ? __pfx_hrtimer_wakeup+0x10/0x10
[Mon Mar 11 14:29:16 2024] ep_poll+0x348/0x3b0
[Mon Mar 11 14:29:16 2024] ? __pfx_autoremove_wake_function+0x10/0x10
[Mon Mar 11 14:29:16 2024] do_epoll_wait+0xb1/0xd0
[Mon Mar 11 14:29:16 2024] __x64_sys_epoll_wait+0x60/0x100
[Mon Mar 11 14:29:16 2024] do_syscall_64+0x59/0x90
[Mon Mar 11 14:29:16 2024] ? syscall_exit_work+0x103/0x130
[Mon Mar 11 14:29:16 2024] ? syscall_exit_to_user_mode+0x22/0x40
[Mon Mar 11 14:29:16 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:16 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:16 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:16 2024] ? syscall_exit_to_user_mode+0x22/0x40
[Mon Mar 11 14:29:16 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:16 2024] ? exc_page_fault+0x62/0x150
[Mon Mar 11 14:29:16 2024] entry_SYSCALL_64_after_hwframe+0x72/0xdc
[Mon Mar 11 14:29:16 2024] RIP: 0033:0x7fedb7d4e80a
[Mon Mar 11 14:29:16 2024] RSP: 002b:00007ffc284a9ad8 EFLAGS: 00000246 ORIG_RAX: 00000000000000e8
[Mon Mar 11 14:29:16 2024] RAX: ffffffffffffffda RBX: 000055f87dd397c0 RCX: 00007fedb7d4e80a
[Mon Mar 11 14:29:16 2024] RDX: 0000000000000001 RSI: 00007ffc284a9b0c RDI: 0000000000000005
[Mon Mar 11 14:29:16 2024] RBP: 000055f87dd2aa20 R08: 00000000000f41d4 R09: 0000000000000000
[Mon Mar 11 14:29:16 2024] R10: 000000000000ea60 R11: 0000000000000246 R12: 000055f87dd58980
[Mon Mar 11 14:29:16 2024] R13: 000000000000ea60 R14: 0000000000000027 R15: 000055f87dd5b760
[Mon Mar 11 14:29:16 2024] </TASK>
[Mon Mar 11 14:29:16 2024] task:kworker/45:0 state:I stack:0 pid:2385965 ppid:2 flags:0x00004000
[Mon Mar 11 14:29:16 2024] Workqueue: 0x0 (mm_percpu_wq)
[Mon Mar 11 14:29:16 2024] Call Trace:
[Mon Mar 11 14:29:16 2024] <TASK>
[Mon Mar 11 14:29:16 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:16 2024] ? __pfx_worker_thread+0x10/0x10
[Mon Mar 11 14:29:16 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:16 2024] worker_thread+0xbb/0x3a0
[Mon Mar 11 14:29:16 2024] ? __pfx_worker_thread+0x10/0x10
[Mon Mar 11 14:29:16 2024] kthread+0xdd/0x100
[Mon Mar 11 14:29:16 2024] ? __pfx_kthread+0x10/0x10
[Mon Mar 11 14:29:16 2024] ret_from_fork+0x29/0x50
[Mon Mar 11 14:29:16 2024] </TASK>
[Mon Mar 11 14:29:16 2024] task:NFSv4 callback state:I stack:0 pid:2387784 ppid:2 flags:0x00004000
[Mon Mar 11 14:29:16 2024] Call Trace:
[Mon Mar 11 14:29:16 2024] <TASK>
[Mon Mar 11 14:29:16 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:16 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:16 2024] nfs41_callback_svc+0x186/0x190 [nfsv4]
[Mon Mar 11 14:29:16 2024] ? __pfx_autoremove_wake_function+0x10/0x10
[Mon Mar 11 14:29:16 2024] ? __pfx_nfs41_callback_svc+0x10/0x10 [nfsv4]
[Mon Mar 11 14:29:16 2024] kthread+0xdd/0x100
[Mon Mar 11 14:29:16 2024] ? __pfx_kthread+0x10/0x10
[Mon Mar 11 14:29:16 2024] ret_from_fork+0x29/0x50
[Mon Mar 11 14:29:16 2024] </TASK>
[Mon Mar 11 14:29:16 2024] task:smbd[10.87.29.5 state:S stack:0 pid:2388610 ppid:4757 flags:0x00000002
[Mon Mar 11 14:29:16 2024] Call Trace:
[Mon Mar 11 14:29:16 2024] <TASK>
[Mon Mar 11 14:29:16 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:16 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:16 2024] schedule_hrtimeout_range_clock+0x95/0x120
[Mon Mar 11 14:29:16 2024] ? __pfx_hrtimer_wakeup+0x10/0x10
[Mon Mar 11 14:29:16 2024] ep_poll+0x348/0x3b0
[Mon Mar 11 14:29:16 2024] ? __pfx_autoremove_wake_function+0x10/0x10
[Mon Mar 11 14:29:16 2024] do_epoll_wait+0xb1/0xd0
[Mon Mar 11 14:29:16 2024] __x64_sys_epoll_wait+0x60/0x100
[Mon Mar 11 14:29:16 2024] do_syscall_64+0x59/0x90
[Mon Mar 11 14:29:16 2024] ? syscall_exit_to_user_mode+0x22/0x40
[Mon Mar 11 14:29:16 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:16 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:16 2024] ? exc_page_fault+0x62/0x150
[Mon Mar 11 14:29:16 2024] entry_SYSCALL_64_after_hwframe+0x72/0xdc
[Mon Mar 11 14:29:16 2024] RIP: 0033:0x7fedb7d4e84e
[Mon Mar 11 14:29:16 2024] RSP: 002b:00007ffc284a9ab0 EFLAGS: 00000293 ORIG_RAX: 00000000000000e8
[Mon Mar 11 14:29:16 2024] RAX: ffffffffffffffda RBX: 000055f87dd397c0 RCX: 00007fedb7d4e84e
[Mon Mar 11 14:29:16 2024] RDX: 0000000000000001 RSI: 00007ffc284a9b0c RDI: 0000000000000005
[Mon Mar 11 14:29:16 2024] RBP: 000055f87dd2aa20 R08: 0000000000000000 R09: 0000000000000000
[Mon Mar 11 14:29:16 2024] R10: 000000000000ea60 R11: 0000000000000293 R12: 000055f87dd61190
[Mon Mar 11 14:29:16 2024] R13: 000000000000ea60 R14: 0000000000000026 R15: 000055f87dd5b760
[Mon Mar 11 14:29:16 2024] </TASK>
[Mon Mar 11 14:29:16 2024] task:kworker/7:0 state:I stack:0 pid:2391347 ppid:2 flags:0x00004000
[Mon Mar 11 14:29:16 2024] Workqueue: 0x0 (mm_percpu_wq)
[Mon Mar 11 14:29:16 2024] Call Trace:
[Mon Mar 11 14:29:16 2024] <TASK>
[Mon Mar 11 14:29:16 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:16 2024] ? __pfx_worker_thread+0x10/0x10
[Mon Mar 11 14:29:16 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:16 2024] worker_thread+0xbb/0x3a0
[Mon Mar 11 14:29:16 2024] ? __pfx_worker_thread+0x10/0x10
[Mon Mar 11 14:29:16 2024] kthread+0xdd/0x100
[Mon Mar 11 14:29:16 2024] ? __pfx_kthread+0x10/0x10
[Mon Mar 11 14:29:16 2024] ret_from_fork+0x29/0x50
[Mon Mar 11 14:29:16 2024] </TASK>
[Mon Mar 11 14:29:16 2024] task:kworker/17:1 state:I stack:0 pid:2391348 ppid:2 flags:0x00004000
[Mon Mar 11 14:29:16 2024] Workqueue: 0x0 (mm_percpu_wq)
[Mon Mar 11 14:29:16 2024] Call Trace:
[Mon Mar 11 14:29:16 2024] <TASK>
[Mon Mar 11 14:29:16 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:16 2024] ? __pfx_worker_thread+0x10/0x10
[Mon Mar 11 14:29:16 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:16 2024] worker_thread+0xbb/0x3a0
[Mon Mar 11 14:29:16 2024] ? __pfx_worker_thread+0x10/0x10
[Mon Mar 11 14:29:16 2024] kthread+0xdd/0x100
[Mon Mar 11 14:29:16 2024] ? __pfx_kthread+0x10/0x10
[Mon Mar 11 14:29:16 2024] ret_from_fork+0x29/0x50
[Mon Mar 11 14:29:16 2024] </TASK>
[Mon Mar 11 14:29:16 2024] task:kworker/13:0 state:I stack:0 pid:2391582 ppid:2 flags:0x00004000
[Mon Mar 11 14:29:16 2024] Workqueue: 0x0 (mm_percpu_wq)
[Mon Mar 11 14:29:16 2024] Call Trace:
[Mon Mar 11 14:29:16 2024] <TASK>
[Mon Mar 11 14:29:16 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:16 2024] ? __pfx_worker_thread+0x10/0x10
[Mon Mar 11 14:29:16 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:16 2024] worker_thread+0xbb/0x3a0
[Mon Mar 11 14:29:16 2024] ? __pfx_worker_thread+0x10/0x10
[Mon Mar 11 14:29:16 2024] kthread+0xdd/0x100
[Mon Mar 11 14:29:16 2024] ? __pfx_kthread+0x10/0x10
[Mon Mar 11 14:29:16 2024] ret_from_fork+0x29/0x50
[Mon Mar 11 14:29:16 2024] </TASK>
[Mon Mar 11 14:29:16 2024] task:smbd[10.87.30.2 state:S stack:0 pid:2393076 ppid:4757 flags:0x00000002
[Mon Mar 11 14:29:16 2024] Call Trace:
[Mon Mar 11 14:29:16 2024] <TASK>
[Mon Mar 11 14:29:16 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:16 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:16 2024] schedule_hrtimeout_range_clock+0x95/0x120
[Mon Mar 11 14:29:16 2024] ? __pfx_hrtimer_wakeup+0x10/0x10
[Mon Mar 11 14:29:16 2024] ep_poll+0x348/0x3b0
[Mon Mar 11 14:29:16 2024] ? __pfx_autoremove_wake_function+0x10/0x10
[Mon Mar 11 14:29:16 2024] do_epoll_wait+0xb1/0xd0
[Mon Mar 11 14:29:16 2024] __x64_sys_epoll_wait+0x60/0x100
[Mon Mar 11 14:29:16 2024] do_syscall_64+0x59/0x90
[Mon Mar 11 14:29:16 2024] ? audit_reset_context.part.0.constprop.0+0x273/0x2e0
[Mon Mar 11 14:29:16 2024] ? syscall_exit_work+0x103/0x130
[Mon Mar 11 14:29:16 2024] ? syscall_exit_to_user_mode+0x22/0x40
[Mon Mar 11 14:29:16 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:16 2024] entry_SYSCALL_64_after_hwframe+0x72/0xdc
[Mon Mar 11 14:29:16 2024] RIP: 0033:0x7fedb7d4e80a
[Mon Mar 11 14:29:16 2024] RSP: 002b:00007ffc284a9ad8 EFLAGS: 00000246 ORIG_RAX: 00000000000000e8
[Mon Mar 11 14:29:16 2024] RAX: ffffffffffffffda RBX: 000055f87dd397c0 RCX: 00007fedb7d4e80a
[Mon Mar 11 14:29:16 2024] RDX: 0000000000000001 RSI: 00007ffc284a9b0c RDI: 0000000000000005
[Mon Mar 11 14:29:16 2024] RBP: 000055f87dd2aa20 R08: 00000000000f41d2 R09: 0000000000000000
[Mon Mar 11 14:29:16 2024] R10: 000000000000ea60 R11: 0000000000000246 R12: 000055f87dd61190
[Mon Mar 11 14:29:16 2024] R13: 000000000000ea60 R14: 0000000000000026 R15: 000055f87dd5b760
[Mon Mar 11 14:29:16 2024] </TASK>
[Mon Mar 11 14:29:16 2024] task:smbd[10.87.29.9 state:S stack:0 pid:2395603 ppid:4757 flags:0x00000002
[Mon Mar 11 14:29:16 2024] Call Trace:
[Mon Mar 11 14:29:16 2024] <TASK>
[Mon Mar 11 14:29:16 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:16 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:16 2024] schedule_hrtimeout_range_clock+0x95/0x120
[Mon Mar 11 14:29:16 2024] ? __pfx_hrtimer_wakeup+0x10/0x10
[Mon Mar 11 14:29:16 2024] ep_poll+0x348/0x3b0
[Mon Mar 11 14:29:16 2024] ? __pfx_autoremove_wake_function+0x10/0x10
[Mon Mar 11 14:29:16 2024] do_epoll_wait+0xb1/0xd0
[Mon Mar 11 14:29:16 2024] __x64_sys_epoll_wait+0x60/0x100
[Mon Mar 11 14:29:16 2024] do_syscall_64+0x59/0x90
[Mon Mar 11 14:29:16 2024] ? syscall_exit_to_user_mode+0x22/0x40
[Mon Mar 11 14:29:16 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:16 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:16 2024] ? syscall_exit_to_user_mode+0x22/0x40
[Mon Mar 11 14:29:16 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:16 2024] entry_SYSCALL_64_after_hwframe+0x72/0xdc
[Mon Mar 11 14:29:16 2024] RIP: 0033:0x7fedb7d4e80a
[Mon Mar 11 14:29:16 2024] RSP: 002b:00007ffc284a9ad8 EFLAGS: 00000246 ORIG_RAX: 00000000000000e8
[Mon Mar 11 14:29:16 2024] RAX: ffffffffffffffda RBX: 000055f87dd397c0 RCX: 00007fedb7d4e80a
[Mon Mar 11 14:29:16 2024] RDX: 0000000000000001 RSI: 00007ffc284a9b0c RDI: 0000000000000005
[Mon Mar 11 14:29:16 2024] RBP: 000055f87dd2aa20 R08: 00000000000f41f2 R09: 0000000000000000
[Mon Mar 11 14:29:16 2024] R10: 000000000000ea60 R11: 0000000000000246 R12: 000055f87dd61190
[Mon Mar 11 14:29:16 2024] R13: 000000000000ea60 R14: 0000000000000026 R15: 000055f87dd5b760
[Mon Mar 11 14:29:16 2024] </TASK>
[Mon Mar 11 14:29:16 2024] task:smbd[10.40.143. state:S stack:0 pid:2396242 ppid:4757 flags:0x00000002
[Mon Mar 11 14:29:16 2024] Call Trace:
[Mon Mar 11 14:29:16 2024] <TASK>
[Mon Mar 11 14:29:16 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:16 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:16 2024] schedule_hrtimeout_range_clock+0x95/0x120
[Mon Mar 11 14:29:16 2024] ? __pfx_hrtimer_wakeup+0x10/0x10
[Mon Mar 11 14:29:16 2024] ep_poll+0x348/0x3b0
[Mon Mar 11 14:29:16 2024] ? __pfx_autoremove_wake_function+0x10/0x10
[Mon Mar 11 14:29:16 2024] do_epoll_wait+0xb1/0xd0
[Mon Mar 11 14:29:16 2024] __x64_sys_epoll_wait+0x60/0x100
[Mon Mar 11 14:29:16 2024] do_syscall_64+0x59/0x90
[Mon Mar 11 14:29:16 2024] ? mntput_no_expire+0x4a/0x250
[Mon Mar 11 14:29:16 2024] ? audit_reset_context.part.0.constprop.0+0x273/0x2e0
[Mon Mar 11 14:29:16 2024] ? syscall_exit_work+0x103/0x130
[Mon Mar 11 14:29:16 2024] ? syscall_exit_to_user_mode+0x22/0x40
[Mon Mar 11 14:29:16 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:16 2024] entry_SYSCALL_64_after_hwframe+0x72/0xdc
[Mon Mar 11 14:29:16 2024] RIP: 0033:0x7fedb7d4e80a
[Mon Mar 11 14:29:16 2024] RSP: 002b:00007ffc284a9ad8 EFLAGS: 00000246 ORIG_RAX: 00000000000000e8
[Mon Mar 11 14:29:16 2024] RAX: ffffffffffffffda RBX: 000055f87dd397c0 RCX: 00007fedb7d4e80a
[Mon Mar 11 14:29:16 2024] RDX: 0000000000000001 RSI: 00007ffc284a9b0c RDI: 0000000000000005
[Mon Mar 11 14:29:16 2024] RBP: 000055f87dd2aa20 R08: 00000000000f4110 R09: 0000000000000000
[Mon Mar 11 14:29:16 2024] R10: 000000000000ea60 R11: 0000000000000246 R12: 000055f87dd61190
[Mon Mar 11 14:29:16 2024] R13: 000000000000ea60 R14: 0000000000000026 R15: 000055f87dd5b760
[Mon Mar 11 14:29:16 2024] </TASK>
[Mon Mar 11 14:29:16 2024] task:smbd[10.87.31.1 state:S stack:0 pid:2396394 ppid:4757 flags:0x00000002
[Mon Mar 11 14:29:16 2024] Call Trace:
[Mon Mar 11 14:29:16 2024] <TASK>
[Mon Mar 11 14:29:16 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:16 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:16 2024] schedule_hrtimeout_range_clock+0x95/0x120
[Mon Mar 11 14:29:16 2024] ? __pfx_hrtimer_wakeup+0x10/0x10
[Mon Mar 11 14:29:16 2024] ep_poll+0x348/0x3b0
[Mon Mar 11 14:29:16 2024] ? __pfx_autoremove_wake_function+0x10/0x10
[Mon Mar 11 14:29:16 2024] do_epoll_wait+0xb1/0xd0
[Mon Mar 11 14:29:16 2024] __x64_sys_epoll_wait+0x60/0x100
[Mon Mar 11 14:29:16 2024] do_syscall_64+0x59/0x90
[Mon Mar 11 14:29:16 2024] ? common_interrupt+0x43/0xa0
[Mon Mar 11 14:29:16 2024] entry_SYSCALL_64_after_hwframe+0x72/0xdc
[Mon Mar 11 14:29:16 2024] RIP: 0033:0x7fedb7d4e80a
[Mon Mar 11 14:29:16 2024] RSP: 002b:00007ffc284a9ad8 EFLAGS: 00000246 ORIG_RAX: 00000000000000e8
[Mon Mar 11 14:29:16 2024] RAX: ffffffffffffffda RBX: 000055f87dd397c0 RCX: 00007fedb7d4e80a
[Mon Mar 11 14:29:16 2024] RDX: 0000000000000001 RSI: 00007ffc284a9b0c RDI: 0000000000000005
[Mon Mar 11 14:29:16 2024] RBP: 000055f87dd2aa20 R08: 00000000000f41bc R09: 0000000000000000
[Mon Mar 11 14:29:16 2024] R10: 000000000000ea60 R11: 0000000000000246 R12: 000055f87dd61190
[Mon Mar 11 14:29:16 2024] R13: 000000000000ea60 R14: 0000000000000026 R15: 000055f87dd5b760
[Mon Mar 11 14:29:16 2024] </TASK>
[Mon Mar 11 14:29:16 2024] task:smbd[10.87.29.8 state:S stack:0 pid:2397415 ppid:4757 flags:0x00000002
[Mon Mar 11 14:29:16 2024] Call Trace:
[Mon Mar 11 14:29:16 2024] <TASK>
[Mon Mar 11 14:29:16 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:16 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:16 2024] schedule_hrtimeout_range_clock+0x95/0x120
[Mon Mar 11 14:29:16 2024] ? __pfx_hrtimer_wakeup+0x10/0x10
[Mon Mar 11 14:29:16 2024] ep_poll+0x348/0x3b0
[Mon Mar 11 14:29:16 2024] ? __pfx_autoremove_wake_function+0x10/0x10
[Mon Mar 11 14:29:16 2024] do_epoll_wait+0xb1/0xd0
[Mon Mar 11 14:29:16 2024] __x64_sys_epoll_wait+0x60/0x100
[Mon Mar 11 14:29:16 2024] do_syscall_64+0x59/0x90
[Mon Mar 11 14:29:16 2024] ? irqtime_account_irq+0x3c/0xb0
[Mon Mar 11 14:29:16 2024] ? __do_softirq+0x16a/0x2ac
[Mon Mar 11 14:29:16 2024] ? __irq_exit_rcu+0x46/0xc0
[Mon Mar 11 14:29:16 2024] ? common_interrupt+0x43/0xa0
[Mon Mar 11 14:29:16 2024] entry_SYSCALL_64_after_hwframe+0x72/0xdc
[Mon Mar 11 14:29:16 2024] RIP: 0033:0x7fedb7d4e80a
[Mon Mar 11 14:29:16 2024] RSP: 002b:00007ffc284a9ad8 EFLAGS: 00000246 ORIG_RAX: 00000000000000e8
[Mon Mar 11 14:29:16 2024] RAX: ffffffffffffffda RBX: 000055f87dd397c0 RCX: 00007fedb7d4e80a
[Mon Mar 11 14:29:16 2024] RDX: 0000000000000001 RSI: 00007ffc284a9b0c RDI: 0000000000000005
[Mon Mar 11 14:29:16 2024] RBP: 000055f87dd2aa20 R08: 0000000000020aef R09: 0000000000000000
[Mon Mar 11 14:29:16 2024] R10: 000000000000cba6 R11: 0000000000000246 R12: 000055f87dd61190
[Mon Mar 11 14:29:16 2024] R13: 000000000000cba6 R14: 0000000000000026 R15: 000055f87dd5b760
[Mon Mar 11 14:29:16 2024] </TASK>
[Mon Mar 11 14:29:16 2024] task:kworker/16:2 state:I stack:0 pid:2397744 ppid:2 flags:0x00004000
[Mon Mar 11 14:29:16 2024] Workqueue: 0x0 (mm_percpu_wq)
[Mon Mar 11 14:29:16 2024] Call Trace:
[Mon Mar 11 14:29:16 2024] <TASK>
[Mon Mar 11 14:29:16 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:16 2024] ? __pfx_worker_thread+0x10/0x10
[Mon Mar 11 14:29:16 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:16 2024] worker_thread+0xbb/0x3a0
[Mon Mar 11 14:29:16 2024] ? __pfx_worker_thread+0x10/0x10
[Mon Mar 11 14:29:16 2024] kthread+0xdd/0x100
[Mon Mar 11 14:29:16 2024] ? __pfx_kthread+0x10/0x10
[Mon Mar 11 14:29:16 2024] ret_from_fork+0x29/0x50
[Mon Mar 11 14:29:16 2024] </TASK>
[Mon Mar 11 14:29:16 2024] task:smbd[10.87.28.1 state:S stack:0 pid:2398554 ppid:4757 flags:0x00000002
[Mon Mar 11 14:29:16 2024] Call Trace:
[Mon Mar 11 14:29:16 2024] <TASK>
[Mon Mar 11 14:29:16 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:16 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:16 2024] schedule_hrtimeout_range_clock+0x95/0x120
[Mon Mar 11 14:29:16 2024] ? __pfx_hrtimer_wakeup+0x10/0x10
[Mon Mar 11 14:29:16 2024] ep_poll+0x348/0x3b0
[Mon Mar 11 14:29:16 2024] ? __pfx_autoremove_wake_function+0x10/0x10
[Mon Mar 11 14:29:16 2024] do_epoll_wait+0xb1/0xd0
[Mon Mar 11 14:29:16 2024] __x64_sys_epoll_wait+0x60/0x100
[Mon Mar 11 14:29:16 2024] do_syscall_64+0x59/0x90
[Mon Mar 11 14:29:16 2024] ? __do_softirq+0x16a/0x2ac
[Mon Mar 11 14:29:16 2024] ? __irq_exit_rcu+0x46/0xc0
[Mon Mar 11 14:29:16 2024] ? common_interrupt+0x43/0xa0
[Mon Mar 11 14:29:16 2024] entry_SYSCALL_64_after_hwframe+0x72/0xdc
[Mon Mar 11 14:29:16 2024] RIP: 0033:0x7fedb7d4e84e
[Mon Mar 11 14:29:16 2024] RSP: 002b:00007ffc284a9ab0 EFLAGS: 00000293 ORIG_RAX: 00000000000000e8
[Mon Mar 11 14:29:16 2024] RAX: ffffffffffffffda RBX: 000055f87dd397c0 RCX: 00007fedb7d4e84e
[Mon Mar 11 14:29:16 2024] RDX: 0000000000000001 RSI: 00007ffc284a9b0c RDI: 0000000000000005
[Mon Mar 11 14:29:16 2024] RBP: 000055f87dd2aa20 R08: 0000000000000000 R09: 0000000000000000
[Mon Mar 11 14:29:16 2024] R10: 000000000000ea60 R11: 0000000000000293 R12: 000055f87dd58980
[Mon Mar 11 14:29:16 2024] R13: 000000000000ea60 R14: 0000000000000027 R15: 000055f87dd5b760
[Mon Mar 11 14:29:16 2024] </TASK>
[Mon Mar 11 14:29:16 2024] task:smbd[10.40.144. state:S stack:0 pid:2399030 ppid:4757 flags:0x00000002
[Mon Mar 11 14:29:16 2024] Call Trace:
[Mon Mar 11 14:29:16 2024] <TASK>
[Mon Mar 11 14:29:16 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:16 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:16 2024] schedule_hrtimeout_range_clock+0x95/0x120
[Mon Mar 11 14:29:16 2024] ? __pfx_hrtimer_wakeup+0x10/0x10
[Mon Mar 11 14:29:16 2024] ep_poll+0x348/0x3b0
[Mon Mar 11 14:29:16 2024] ? __pfx_autoremove_wake_function+0x10/0x10
[Mon Mar 11 14:29:16 2024] do_epoll_wait+0xb1/0xd0
[Mon Mar 11 14:29:16 2024] __x64_sys_epoll_wait+0x60/0x100
[Mon Mar 11 14:29:16 2024] do_syscall_64+0x59/0x90
[Mon Mar 11 14:29:16 2024] ? syscall_exit_to_user_mode+0x22/0x40
[Mon Mar 11 14:29:16 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:16 2024] ? syscall_exit_to_user_mode+0x22/0x40
[Mon Mar 11 14:29:16 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:16 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:16 2024] entry_SYSCALL_64_after_hwframe+0x72/0xdc
[Mon Mar 11 14:29:16 2024] RIP: 0033:0x7fedb7d4e80a
[Mon Mar 11 14:29:16 2024] RSP: 002b:00007ffc284a9ad8 EFLAGS: 00000246 ORIG_RAX: 00000000000000e8
[Mon Mar 11 14:29:16 2024] RAX: ffffffffffffffda RBX: 000055f87dd397c0 RCX: 00007fedb7d4e80a
[Mon Mar 11 14:29:16 2024] RDX: 0000000000000001 RSI: 00007ffc284a9b0c RDI: 0000000000000005
[Mon Mar 11 14:29:16 2024] RBP: 000055f87dd2aa20 R08: 00000000000f41dc R09: 0000000000000000
[Mon Mar 11 14:29:16 2024] R10: 000000000000ea60 R11: 0000000000000246 R12: 000055f87dd58980
[Mon Mar 11 14:29:16 2024] R13: 000000000000ea60 R14: 0000000000000027 R15: 000055f87dd5b760
[Mon Mar 11 14:29:16 2024] </TASK>
[Mon Mar 11 14:29:16 2024] task:kworker/1:1 state:I stack:0 pid:2399485 ppid:2 flags:0x00004000
[Mon Mar 11 14:29:16 2024] Workqueue: 0x0 (events_power_efficient)
[Mon Mar 11 14:29:16 2024] Call Trace:
[Mon Mar 11 14:29:16 2024] <TASK>
[Mon Mar 11 14:29:16 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:16 2024] ? neigh_managed_work+0x9a/0xb0
[Mon Mar 11 14:29:16 2024] ? __pfx_worker_thread+0x10/0x10
[Mon Mar 11 14:29:16 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:16 2024] worker_thread+0xbb/0x3a0
[Mon Mar 11 14:29:16 2024] ? __pfx_worker_thread+0x10/0x10
[Mon Mar 11 14:29:16 2024] kthread+0xdd/0x100
[Mon Mar 11 14:29:16 2024] ? __pfx_kthread+0x10/0x10
[Mon Mar 11 14:29:16 2024] ret_from_fork+0x29/0x50
[Mon Mar 11 14:29:16 2024] </TASK>
[Mon Mar 11 14:29:16 2024] task:kworker/3:2 state:I stack:0 pid:2399492 ppid:2 flags:0x00004000
[Mon Mar 11 14:29:16 2024] Workqueue: 0x0 (mm_percpu_wq)
[Mon Mar 11 14:29:16 2024] Call Trace:
[Mon Mar 11 14:29:16 2024] <TASK>
[Mon Mar 11 14:29:16 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:16 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:16 2024] worker_thread+0xbb/0x3a0
[Mon Mar 11 14:29:16 2024] ? __pfx_worker_thread+0x10/0x10
[Mon Mar 11 14:29:16 2024] kthread+0xdd/0x100
[Mon Mar 11 14:29:16 2024] ? __pfx_kthread+0x10/0x10
[Mon Mar 11 14:29:16 2024] ret_from_fork+0x29/0x50
[Mon Mar 11 14:29:16 2024] </TASK>
[Mon Mar 11 14:29:16 2024] task:smbd[10.87.27.1 state:S stack:0 pid:2400150 ppid:4757 flags:0x00000002
[Mon Mar 11 14:29:16 2024] Call Trace:
[Mon Mar 11 14:29:16 2024] <TASK>
[Mon Mar 11 14:29:16 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:16 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:16 2024] schedule_hrtimeout_range_clock+0x95/0x120
[Mon Mar 11 14:29:16 2024] ? __pfx_hrtimer_wakeup+0x10/0x10
[Mon Mar 11 14:29:16 2024] ep_poll+0x348/0x3b0
[Mon Mar 11 14:29:16 2024] ? __pfx_autoremove_wake_function+0x10/0x10
[Mon Mar 11 14:29:16 2024] do_epoll_wait+0xb1/0xd0
[Mon Mar 11 14:29:16 2024] __x64_sys_epoll_wait+0x60/0x100
[Mon Mar 11 14:29:16 2024] do_syscall_64+0x59/0x90
[Mon Mar 11 14:29:16 2024] ? audit_reset_context.part.0.constprop.0+0x273/0x2e0
[Mon Mar 11 14:29:16 2024] ? syscall_exit_work+0x103/0x130
[Mon Mar 11 14:29:16 2024] ? syscall_exit_to_user_mode+0x22/0x40
[Mon Mar 11 14:29:16 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:16 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:16 2024] ? syscall_exit_work+0x103/0x130
[Mon Mar 11 14:29:16 2024] ? syscall_exit_to_user_mode+0x22/0x40
[Mon Mar 11 14:29:16 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:16 2024] entry_SYSCALL_64_after_hwframe+0x72/0xdc
[Mon Mar 11 14:29:16 2024] RIP: 0033:0x7fedb7d4e80a
[Mon Mar 11 14:29:16 2024] RSP: 002b:00007ffc284a9ad8 EFLAGS: 00000246 ORIG_RAX: 00000000000000e8
[Mon Mar 11 14:29:16 2024] RAX: ffffffffffffffda RBX: 000055f87dd397c0 RCX: 00007fedb7d4e80a
[Mon Mar 11 14:29:16 2024] RDX: 0000000000000001 RSI: 00007ffc284a9b0c RDI: 0000000000000005
[Mon Mar 11 14:29:16 2024] RBP: 000055f87dd2aa20 R08: 00000000000f41df R09: 0000000000000000
[Mon Mar 11 14:29:16 2024] R10: 000000000000ea60 R11: 0000000000000246 R12: 000055f87dd61190
[Mon Mar 11 14:29:16 2024] R13: 000000000000ea60 R14: 0000000000000026 R15: 000055f87dd5b760
[Mon Mar 11 14:29:16 2024] </TASK>
[Mon Mar 11 14:29:16 2024] task:smbd[10.40.137. state:S stack:0 pid:2400165 ppid:4757 flags:0x00000002
[Mon Mar 11 14:29:16 2024] Call Trace:
[Mon Mar 11 14:29:16 2024] <TASK>
[Mon Mar 11 14:29:16 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:16 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:16 2024] schedule_hrtimeout_range_clock+0x95/0x120
[Mon Mar 11 14:29:16 2024] ? __pfx_hrtimer_wakeup+0x10/0x10
[Mon Mar 11 14:29:16 2024] ep_poll+0x348/0x3b0
[Mon Mar 11 14:29:16 2024] ? __pfx_autoremove_wake_function+0x10/0x10
[Mon Mar 11 14:29:16 2024] do_epoll_wait+0xb1/0xd0
[Mon Mar 11 14:29:16 2024] __x64_sys_epoll_wait+0x60/0x100
[Mon Mar 11 14:29:16 2024] do_syscall_64+0x59/0x90
[Mon Mar 11 14:29:16 2024] ? __audit_filter_op+0xa5/0xf0
[Mon Mar 11 14:29:16 2024] ? audit_reset_context.part.0.constprop.0+0x273/0x2e0
[Mon Mar 11 14:29:16 2024] ? syscall_exit_work+0x103/0x130
[Mon Mar 11 14:29:16 2024] ? syscall_exit_to_user_mode+0x22/0x40
[Mon Mar 11 14:29:16 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:16 2024] entry_SYSCALL_64_after_hwframe+0x72/0xdc
[Mon Mar 11 14:29:16 2024] RIP: 0033:0x7fedb7d4e84e
[Mon Mar 11 14:29:16 2024] RSP: 002b:00007ffc284a9ab0 EFLAGS: 00000293 ORIG_RAX: 00000000000000e8
[Mon Mar 11 14:29:16 2024] RAX: ffffffffffffffda RBX: 000055f87dd397c0 RCX: 00007fedb7d4e84e
[Mon Mar 11 14:29:16 2024] RDX: 0000000000000001 RSI: 00007ffc284a9b0c RDI: 0000000000000005
[Mon Mar 11 14:29:16 2024] RBP: 000055f87dd2aa20 R08: 0000000000000000 R09: 0000000000000000
[Mon Mar 11 14:29:16 2024] R10: 000000000000ea60 R11: 0000000000000293 R12: 000055f87dd61190
[Mon Mar 11 14:29:16 2024] R13: 000000000000ea60 R14: 0000000000000026 R15: 000055f87dd5b760
[Mon Mar 11 14:29:16 2024] </TASK>
[Mon Mar 11 14:29:16 2024] task:smbd[10.87.27.6 state:S stack:0 pid:2400787 ppid:4757 flags:0x00000002
[Mon Mar 11 14:29:16 2024] Call Trace:
[Mon Mar 11 14:29:16 2024] <TASK>
[Mon Mar 11 14:29:16 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:16 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:16 2024] schedule_hrtimeout_range_clock+0x95/0x120
[Mon Mar 11 14:29:16 2024] ? __pfx_hrtimer_wakeup+0x10/0x10
[Mon Mar 11 14:29:16 2024] ep_poll+0x348/0x3b0
[Mon Mar 11 14:29:16 2024] ? __pfx_autoremove_wake_function+0x10/0x10
[Mon Mar 11 14:29:16 2024] do_epoll_wait+0xb1/0xd0
[Mon Mar 11 14:29:16 2024] __x64_sys_epoll_wait+0x60/0x100
[Mon Mar 11 14:29:16 2024] do_syscall_64+0x59/0x90
[Mon Mar 11 14:29:16 2024] ? exit_to_user_mode_prepare+0xb6/0x100
[Mon Mar 11 14:29:16 2024] ? syscall_exit_to_user_mode+0x22/0x40
[Mon Mar 11 14:29:16 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:16 2024] ? exc_page_fault+0x62/0x150
[Mon Mar 11 14:29:16 2024] entry_SYSCALL_64_after_hwframe+0x72/0xdc
[Mon Mar 11 14:29:16 2024] RIP: 0033:0x7fedb7d4e80a
[Mon Mar 11 14:29:16 2024] RSP: 002b:00007ffc284a9ad8 EFLAGS: 00000246 ORIG_RAX: 00000000000000e8
[Mon Mar 11 14:29:16 2024] RAX: ffffffffffffffda RBX: 000055f87dd397c0 RCX: 00007fedb7d4e80a
[Mon Mar 11 14:29:16 2024] RDX: 0000000000000001 RSI: 00007ffc284a9b0c RDI: 0000000000000005
[Mon Mar 11 14:29:16 2024] RBP: 000055f87dd2aa20 R08: 00000000000f4047 R09: 0000000000000077
[Mon Mar 11 14:29:16 2024] R10: 000000000000ea60 R11: 0000000000000246 R12: 000055f87dd61190
[Mon Mar 11 14:29:16 2024] R13: 000000000000ea60 R14: 0000000000000026 R15: 000055f87dd5b760
[Mon Mar 11 14:29:16 2024] </TASK>
[Mon Mar 11 14:29:16 2024] task:kworker/26:2 state:I stack:0 pid:2401639 ppid:2 flags:0x00004000
[Mon Mar 11 14:29:16 2024] Workqueue: 0x0 (mm_percpu_wq)
[Mon Mar 11 14:29:16 2024] Call Trace:
[Mon Mar 11 14:29:16 2024] <TASK>
[Mon Mar 11 14:29:16 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:16 2024] ? __pfx_worker_thread+0x10/0x10
[Mon Mar 11 14:29:16 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:16 2024] worker_thread+0xbb/0x3a0
[Mon Mar 11 14:29:16 2024] ? __pfx_worker_thread+0x10/0x10
[Mon Mar 11 14:29:16 2024] kthread+0xdd/0x100
[Mon Mar 11 14:29:16 2024] ? __pfx_kthread+0x10/0x10
[Mon Mar 11 14:29:16 2024] ret_from_fork+0x29/0x50
[Mon Mar 11 14:29:16 2024] </TASK>
[Mon Mar 11 14:29:16 2024] task:kworker/10:0 state:I stack:0 pid:2401926 ppid:2 flags:0x00004000
[Mon Mar 11 14:29:16 2024] Workqueue: 0x0 (events)
[Mon Mar 11 14:29:16 2024] Call Trace:
[Mon Mar 11 14:29:16 2024] <TASK>
[Mon Mar 11 14:29:16 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:16 2024] ? __pfx_worker_thread+0x10/0x10
[Mon Mar 11 14:29:16 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:16 2024] worker_thread+0xbb/0x3a0
[Mon Mar 11 14:29:16 2024] ? __pfx_worker_thread+0x10/0x10
[Mon Mar 11 14:29:16 2024] kthread+0xdd/0x100
[Mon Mar 11 14:29:16 2024] ? __pfx_kthread+0x10/0x10
[Mon Mar 11 14:29:16 2024] ret_from_fork+0x29/0x50
[Mon Mar 11 14:29:16 2024] </TASK>
[Mon Mar 11 14:29:16 2024] task:kworker/19:1 state:I stack:0 pid:2402658 ppid:2 flags:0x00004000
[Mon Mar 11 14:29:16 2024] Workqueue: 0x0 (events)
[Mon Mar 11 14:29:16 2024] Call Trace:
[Mon Mar 11 14:29:16 2024] <TASK>
[Mon Mar 11 14:29:16 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:16 2024] ? __pfx_worker_thread+0x10/0x10
[Mon Mar 11 14:29:16 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:16 2024] worker_thread+0xbb/0x3a0
[Mon Mar 11 14:29:16 2024] ? __pfx_worker_thread+0x10/0x10
[Mon Mar 11 14:29:16 2024] kthread+0xdd/0x100
[Mon Mar 11 14:29:16 2024] ? __pfx_kthread+0x10/0x10
[Mon Mar 11 14:29:16 2024] ret_from_fork+0x29/0x50
[Mon Mar 11 14:29:16 2024] </TASK>
[Mon Mar 11 14:29:16 2024] task:smbd[10.40.135. state:S stack:0 pid:2403402 ppid:4757 flags:0x00000002
[Mon Mar 11 14:29:16 2024] Call Trace:
[Mon Mar 11 14:29:16 2024] <TASK>
[Mon Mar 11 14:29:16 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:16 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:16 2024] schedule_hrtimeout_range_clock+0x95/0x120
[Mon Mar 11 14:29:16 2024] ? __pfx_hrtimer_wakeup+0x10/0x10
[Mon Mar 11 14:29:16 2024] ep_poll+0x348/0x3b0
[Mon Mar 11 14:29:16 2024] ? __pfx_autoremove_wake_function+0x10/0x10
[Mon Mar 11 14:29:16 2024] do_epoll_wait+0xb1/0xd0
[Mon Mar 11 14:29:16 2024] __x64_sys_epoll_wait+0x60/0x100
[Mon Mar 11 14:29:16 2024] do_syscall_64+0x59/0x90
[Mon Mar 11 14:29:16 2024] ? __irq_exit_rcu+0x46/0xc0
[Mon Mar 11 14:29:16 2024] ? common_interrupt+0x43/0xa0
[Mon Mar 11 14:29:16 2024] entry_SYSCALL_64_after_hwframe+0x72/0xdc
[Mon Mar 11 14:29:16 2024] RIP: 0033:0x7fedb7d4e80a
[Mon Mar 11 14:29:16 2024] RSP: 002b:00007ffc284a9ad8 EFLAGS: 00000246 ORIG_RAX: 00000000000000e8
[Mon Mar 11 14:29:16 2024] RAX: ffffffffffffffda RBX: 000055f87dd397c0 RCX: 00007fedb7d4e80a
[Mon Mar 11 14:29:16 2024] RDX: 0000000000000001 RSI: 00007ffc284a9b0c RDI: 0000000000000005
[Mon Mar 11 14:29:16 2024] RBP: 000055f87dd2aa20 R08: 00000000000a4475 R09: 0000000000000000
[Mon Mar 11 14:29:16 2024] R10: 0000000000008f41 R11: 0000000000000246 R12: 000055f87dd61190
[Mon Mar 11 14:29:16 2024] R13: 0000000000008f41 R14: 0000000000000026 R15: 000055f87dd5b760
[Mon Mar 11 14:29:16 2024] </TASK>
[Mon Mar 11 14:29:16 2024] task:smbd[10.40.234. state:S stack:0 pid:2404568 ppid:4757 flags:0x00000002
[Mon Mar 11 14:29:16 2024] Call Trace:
[Mon Mar 11 14:29:16 2024] <TASK>
[Mon Mar 11 14:29:16 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:16 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:16 2024] schedule_hrtimeout_range_clock+0x95/0x120
[Mon Mar 11 14:29:16 2024] ? __pfx_hrtimer_wakeup+0x10/0x10
[Mon Mar 11 14:29:16 2024] ep_poll+0x348/0x3b0
[Mon Mar 11 14:29:16 2024] ? __pfx_autoremove_wake_function+0x10/0x10
[Mon Mar 11 14:29:16 2024] do_epoll_wait+0xb1/0xd0
[Mon Mar 11 14:29:16 2024] __x64_sys_epoll_wait+0x60/0x100
[Mon Mar 11 14:29:16 2024] do_syscall_64+0x59/0x90
[Mon Mar 11 14:29:16 2024] ? syscall_exit_to_user_mode+0x22/0x40
[Mon Mar 11 14:29:16 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:16 2024] ? syscall_exit_to_user_mode+0x22/0x40
[Mon Mar 11 14:29:16 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:16 2024] ? syscall_exit_to_user_mode+0x22/0x40
[Mon Mar 11 14:29:16 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:16 2024] ? syscall_exit_to_user_mode+0x22/0x40
[Mon Mar 11 14:29:16 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:16 2024] entry_SYSCALL_64_after_hwframe+0x72/0xdc
[Mon Mar 11 14:29:16 2024] RIP: 0033:0x7fedb7d4e80a
[Mon Mar 11 14:29:16 2024] RSP: 002b:00007ffc284a9ad8 EFLAGS: 00000246 ORIG_RAX: 00000000000000e8
[Mon Mar 11 14:29:16 2024] RAX: ffffffffffffffda RBX: 000055f87dd397c0 RCX: 00007fedb7d4e80a
[Mon Mar 11 14:29:16 2024] RDX: 0000000000000001 RSI: 00007ffc284a9b0c RDI: 0000000000000005
[Mon Mar 11 14:29:16 2024] RBP: 000055f87dd2aa20 R08: 00000000000f41df R09: 0000000000000000
[Mon Mar 11 14:29:16 2024] R10: 000000000000ea60 R11: 0000000000000246 R12: 000055f87dd61190
[Mon Mar 11 14:29:16 2024] R13: 000000000000ea60 R14: 0000000000000026 R15: 000055f87dd5b760
[Mon Mar 11 14:29:16 2024] </TASK>
[Mon Mar 11 14:29:16 2024] task:smbd[10.88.16.7 state:S stack:0 pid:2407419 ppid:4757 flags:0x00000002
[Mon Mar 11 14:29:16 2024] Call Trace:
[Mon Mar 11 14:29:16 2024] <TASK>
[Mon Mar 11 14:29:16 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:16 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:16 2024] schedule_hrtimeout_range_clock+0x95/0x120
[Mon Mar 11 14:29:16 2024] ? __pfx_hrtimer_wakeup+0x10/0x10
[Mon Mar 11 14:29:16 2024] ep_poll+0x348/0x3b0
[Mon Mar 11 14:29:16 2024] ? __pfx_autoremove_wake_function+0x10/0x10
[Mon Mar 11 14:29:16 2024] do_epoll_wait+0xb1/0xd0
[Mon Mar 11 14:29:16 2024] __x64_sys_epoll_wait+0x60/0x100
[Mon Mar 11 14:29:16 2024] do_syscall_64+0x59/0x90
[Mon Mar 11 14:29:16 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:16 2024] ? syscall_exit_to_user_mode+0x22/0x40
[Mon Mar 11 14:29:16 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:16 2024] ? syscall_exit_work+0x103/0x130
[Mon Mar 11 14:29:16 2024] ? syscall_exit_to_user_mode+0x22/0x40
[Mon Mar 11 14:29:16 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:16 2024] ? audit_reset_context.part.0.constprop.0+0x273/0x2e0
[Mon Mar 11 14:29:16 2024] ? syscall_exit_work+0x103/0x130
[Mon Mar 11 14:29:16 2024] ? syscall_exit_to_user_mode+0x22/0x40
[Mon Mar 11 14:29:16 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:16 2024] entry_SYSCALL_64_after_hwframe+0x72/0xdc
[Mon Mar 11 14:29:16 2024] RIP: 0033:0x7fedb7d4e84e
[Mon Mar 11 14:29:16 2024] RSP: 002b:00007ffc284a9ab0 EFLAGS: 00000293 ORIG_RAX: 00000000000000e8
[Mon Mar 11 14:29:16 2024] RAX: ffffffffffffffda RBX: 000055f87dd397c0 RCX: 00007fedb7d4e84e
[Mon Mar 11 14:29:16 2024] RDX: 0000000000000001 RSI: 00007ffc284a9b0c RDI: 0000000000000005
[Mon Mar 11 14:29:16 2024] RBP: 000055f87dd2aa20 R08: 0000000000000000 R09: 0000000000000000
[Mon Mar 11 14:29:16 2024] R10: 000000000000ea60 R11: 0000000000000293 R12: 000055f87dd61190
[Mon Mar 11 14:29:16 2024] R13: 000000000000ea60 R14: 0000000000000026 R15: 000055f87dd5b760
[Mon Mar 11 14:29:16 2024] </TASK>
[Mon Mar 11 14:29:16 2024] task:kworker/41:1 state:I stack:0 pid:2408233 ppid:2 flags:0x00004000
[Mon Mar 11 14:29:16 2024] Workqueue: 0x0 (mm_percpu_wq)
[Mon Mar 11 14:29:16 2024] Call Trace:
[Mon Mar 11 14:29:16 2024] <TASK>
[Mon Mar 11 14:29:16 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:16 2024] ? __pfx_worker_thread+0x10/0x10
[Mon Mar 11 14:29:16 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:16 2024] worker_thread+0xbb/0x3a0
[Mon Mar 11 14:29:16 2024] ? __pfx_worker_thread+0x10/0x10
[Mon Mar 11 14:29:16 2024] kthread+0xdd/0x100
[Mon Mar 11 14:29:16 2024] ? __pfx_kthread+0x10/0x10
[Mon Mar 11 14:29:16 2024] ret_from_fork+0x29/0x50
[Mon Mar 11 14:29:16 2024] </TASK>
[Mon Mar 11 14:29:16 2024] task:kworker/28:0 state:I stack:0 pid:2409584 ppid:2 flags:0x00004000
[Mon Mar 11 14:29:16 2024] Workqueue: 0x0 (events)
[Mon Mar 11 14:29:16 2024] Call Trace:
[Mon Mar 11 14:29:16 2024] <TASK>
[Mon Mar 11 14:29:16 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:16 2024] ? __pfx_worker_thread+0x10/0x10
[Mon Mar 11 14:29:16 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:16 2024] worker_thread+0xbb/0x3a0
[Mon Mar 11 14:29:16 2024] ? __pfx_worker_thread+0x10/0x10
[Mon Mar 11 14:29:16 2024] kthread+0xdd/0x100
[Mon Mar 11 14:29:16 2024] ? __pfx_kthread+0x10/0x10
[Mon Mar 11 14:29:16 2024] ret_from_fork+0x29/0x50
[Mon Mar 11 14:29:16 2024] </TASK>
[Mon Mar 11 14:29:16 2024] task:kworker/37:1 state:I stack:0 pid:2411099 ppid:2 flags:0x00004000
[Mon Mar 11 14:29:16 2024] Workqueue: 0x0 (mm_percpu_wq)
[Mon Mar 11 14:29:16 2024] Call Trace:
[Mon Mar 11 14:29:16 2024] <TASK>
[Mon Mar 11 14:29:16 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:16 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:16 2024] worker_thread+0xbb/0x3a0
[Mon Mar 11 14:29:16 2024] ? __pfx_worker_thread+0x10/0x10
[Mon Mar 11 14:29:16 2024] kthread+0xdd/0x100
[Mon Mar 11 14:29:16 2024] ? __pfx_kthread+0x10/0x10
[Mon Mar 11 14:29:16 2024] ret_from_fork+0x29/0x50
[Mon Mar 11 14:29:16 2024] </TASK>
[Mon Mar 11 14:29:16 2024] task:smbd[10.39.226. state:S stack:0 pid:2411168 ppid:4757 flags:0x00000002
[Mon Mar 11 14:29:16 2024] Call Trace:
[Mon Mar 11 14:29:16 2024] <TASK>
[Mon Mar 11 14:29:16 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:16 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:16 2024] schedule_hrtimeout_range_clock+0x95/0x120
[Mon Mar 11 14:29:16 2024] ? __pfx_hrtimer_wakeup+0x10/0x10
[Mon Mar 11 14:29:16 2024] ep_poll+0x348/0x3b0
[Mon Mar 11 14:29:16 2024] ? __pfx_autoremove_wake_function+0x10/0x10
[Mon Mar 11 14:29:16 2024] do_epoll_wait+0xb1/0xd0
[Mon Mar 11 14:29:16 2024] __x64_sys_epoll_wait+0x60/0x100
[Mon Mar 11 14:29:16 2024] do_syscall_64+0x59/0x90
[Mon Mar 11 14:29:16 2024] ? syscall_exit_work+0x103/0x130
[Mon Mar 11 14:29:16 2024] ? syscall_exit_to_user_mode+0x22/0x40
[Mon Mar 11 14:29:16 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:16 2024] ? exc_page_fault+0x62/0x150
[Mon Mar 11 14:29:16 2024] entry_SYSCALL_64_after_hwframe+0x72/0xdc
[Mon Mar 11 14:29:16 2024] RIP: 0033:0x7fedb7d4e84e
[Mon Mar 11 14:29:16 2024] RSP: 002b:00007ffc284a9ab0 EFLAGS: 00000293 ORIG_RAX: 00000000000000e8
[Mon Mar 11 14:29:16 2024] RAX: ffffffffffffffda RBX: 000055f87dd397c0 RCX: 00007fedb7d4e84e
[Mon Mar 11 14:29:16 2024] RDX: 0000000000000001 RSI: 00007ffc284a9b0c RDI: 0000000000000005
[Mon Mar 11 14:29:16 2024] RBP: 000055f87dd2aa20 R08: 0000000000000000 R09: 0000000000000000
[Mon Mar 11 14:29:16 2024] R10: 000000000000d3a9 R11: 0000000000000293 R12: 000055f87dd61190
[Mon Mar 11 14:29:16 2024] R13: 000000000000d3a9 R14: 0000000000000026 R15: 000055f87dd5b760
[Mon Mar 11 14:29:16 2024] </TASK>
[Mon Mar 11 14:29:16 2024] task:smbd[10.87.27.4 state:S stack:0 pid:2412672 ppid:4757 flags:0x00000002
[Mon Mar 11 14:29:16 2024] Call Trace:
[Mon Mar 11 14:29:16 2024] <TASK>
[Mon Mar 11 14:29:16 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:16 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:16 2024] schedule_hrtimeout_range_clock+0x95/0x120
[Mon Mar 11 14:29:16 2024] ? __pfx_hrtimer_wakeup+0x10/0x10
[Mon Mar 11 14:29:16 2024] ep_poll+0x348/0x3b0
[Mon Mar 11 14:29:16 2024] ? __pfx_autoremove_wake_function+0x10/0x10
[Mon Mar 11 14:29:16 2024] do_epoll_wait+0xb1/0xd0
[Mon Mar 11 14:29:16 2024] __x64_sys_epoll_wait+0x60/0x100
[Mon Mar 11 14:29:16 2024] do_syscall_64+0x59/0x90
[Mon Mar 11 14:29:16 2024] ? syscall_exit_to_user_mode+0x22/0x40
[Mon Mar 11 14:29:16 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:16 2024] entry_SYSCALL_64_after_hwframe+0x72/0xdc
[Mon Mar 11 14:29:16 2024] RIP: 0033:0x7fedb7d4e80a
[Mon Mar 11 14:29:16 2024] RSP: 002b:00007ffc284a9ad8 EFLAGS: 00000246 ORIG_RAX: 00000000000000e8
[Mon Mar 11 14:29:16 2024] RAX: ffffffffffffffda RBX: 000055f87dd397c0 RCX: 00007fedb7d4e80a
[Mon Mar 11 14:29:16 2024] RDX: 0000000000000001 RSI: 00007ffc284a9b0c RDI: 0000000000000005
[Mon Mar 11 14:29:16 2024] RBP: 000055f87dd2aa20 R08: 00000000000f3f59 R09: 0000000000000000
[Mon Mar 11 14:29:16 2024] R10: 000000000000ea60 R11: 0000000000000246 R12: 000055f87dd61190
[Mon Mar 11 14:29:16 2024] R13: 000000000000ea60 R14: 0000000000000026 R15: 000055f87dd5b760
[Mon Mar 11 14:29:16 2024] </TASK>
[Mon Mar 11 14:29:16 2024] task:kworker/43:2 state:I stack:0 pid:2414694 ppid:2 flags:0x00004000
[Mon Mar 11 14:29:16 2024] Workqueue: 0x0 (mm_percpu_wq)
[Mon Mar 11 14:29:16 2024] Call Trace:
[Mon Mar 11 14:29:16 2024] <TASK>
[Mon Mar 11 14:29:16 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:16 2024] ? __pfx_worker_thread+0x10/0x10
[Mon Mar 11 14:29:16 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:16 2024] worker_thread+0xbb/0x3a0
[Mon Mar 11 14:29:16 2024] ? __pfx_worker_thread+0x10/0x10
[Mon Mar 11 14:29:16 2024] kthread+0xdd/0x100
[Mon Mar 11 14:29:16 2024] ? __pfx_kthread+0x10/0x10
[Mon Mar 11 14:29:16 2024] ret_from_fork+0x29/0x50
[Mon Mar 11 14:29:16 2024] </TASK>
[Mon Mar 11 14:29:16 2024] task:smbd[10.87.29.9 state:S stack:0 pid:2415574 ppid:4757 flags:0x00000002
[Mon Mar 11 14:29:16 2024] Call Trace:
[Mon Mar 11 14:29:16 2024] <TASK>
[Mon Mar 11 14:29:16 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:16 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:16 2024] schedule_hrtimeout_range_clock+0x95/0x120
[Mon Mar 11 14:29:16 2024] ? __pfx_hrtimer_wakeup+0x10/0x10
[Mon Mar 11 14:29:16 2024] ep_poll+0x348/0x3b0
[Mon Mar 11 14:29:16 2024] ? __pfx_autoremove_wake_function+0x10/0x10
[Mon Mar 11 14:29:16 2024] do_epoll_wait+0xb1/0xd0
[Mon Mar 11 14:29:16 2024] __x64_sys_epoll_wait+0x60/0x100
[Mon Mar 11 14:29:16 2024] do_syscall_64+0x59/0x90
[Mon Mar 11 14:29:16 2024] ? audit_reset_context.part.0.constprop.0+0x273/0x2e0
[Mon Mar 11 14:29:16 2024] ? syscall_exit_work+0x103/0x130
[Mon Mar 11 14:29:16 2024] ? syscall_exit_to_user_mode+0x22/0x40
[Mon Mar 11 14:29:16 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:16 2024] ? __irq_exit_rcu+0x46/0xc0
[Mon Mar 11 14:29:16 2024] ? common_interrupt+0x43/0xa0
[Mon Mar 11 14:29:16 2024] entry_SYSCALL_64_after_hwframe+0x72/0xdc
[Mon Mar 11 14:29:16 2024] RIP: 0033:0x7fedb7d4e84e
[Mon Mar 11 14:29:16 2024] RSP: 002b:00007ffc284a9ab0 EFLAGS: 00000293 ORIG_RAX: 00000000000000e8
[Mon Mar 11 14:29:16 2024] RAX: ffffffffffffffda RBX: 000055f87dd397c0 RCX: 00007fedb7d4e84e
[Mon Mar 11 14:29:16 2024] RDX: 0000000000000001 RSI: 00007ffc284a9b0c RDI: 0000000000000005
[Mon Mar 11 14:29:16 2024] RBP: 000055f87dd2aa20 R08: 0000000000000000 R09: 0000000000000000
[Mon Mar 11 14:29:16 2024] R10: 000000000000ea60 R11: 0000000000000293 R12: 000055f87dd61190
[Mon Mar 11 14:29:16 2024] R13: 000000000000ea60 R14: 0000000000000026 R15: 000055f87dd5b760
[Mon Mar 11 14:29:16 2024] </TASK>
[Mon Mar 11 14:29:16 2024] task:kworker/20:2 state:I stack:0 pid:2415685 ppid:2 flags:0x00004000
[Mon Mar 11 14:29:16 2024] Workqueue: 0x0 (mm_percpu_wq)
[Mon Mar 11 14:29:16 2024] Call Trace:
[Mon Mar 11 14:29:16 2024] <TASK>
[Mon Mar 11 14:29:16 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:16 2024] ? __pfx_worker_thread+0x10/0x10
[Mon Mar 11 14:29:16 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:16 2024] worker_thread+0xbb/0x3a0
[Mon Mar 11 14:29:16 2024] ? __pfx_worker_thread+0x10/0x10
[Mon Mar 11 14:29:16 2024] kthread+0xdd/0x100
[Mon Mar 11 14:29:16 2024] ? __pfx_kthread+0x10/0x10
[Mon Mar 11 14:29:16 2024] ret_from_fork+0x29/0x50
[Mon Mar 11 14:29:16 2024] </TASK>
[Mon Mar 11 14:29:16 2024] task:kworker/9:1 state:I stack:0 pid:2418267 ppid:2 flags:0x00004000
[Mon Mar 11 14:29:16 2024] Workqueue: 0x0 (mm_percpu_wq)
[Mon Mar 11 14:29:16 2024] Call Trace:
[Mon Mar 11 14:29:16 2024] <TASK>
[Mon Mar 11 14:29:16 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:16 2024] ? __pfx_worker_thread+0x10/0x10
[Mon Mar 11 14:29:16 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:16 2024] worker_thread+0xbb/0x3a0
[Mon Mar 11 14:29:16 2024] ? __pfx_worker_thread+0x10/0x10
[Mon Mar 11 14:29:16 2024] kthread+0xdd/0x100
[Mon Mar 11 14:29:16 2024] ? __pfx_kthread+0x10/0x10
[Mon Mar 11 14:29:16 2024] ret_from_fork+0x29/0x50
[Mon Mar 11 14:29:16 2024] </TASK>
[Mon Mar 11 14:29:16 2024] task:smbd[10.87.28.6 state:S stack:0 pid:2419992 ppid:4757 flags:0x00000002
[Mon Mar 11 14:29:16 2024] Call Trace:
[Mon Mar 11 14:29:16 2024] <TASK>
[Mon Mar 11 14:29:16 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:16 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:16 2024] schedule_hrtimeout_range_clock+0x95/0x120
[Mon Mar 11 14:29:16 2024] ? __pfx_hrtimer_wakeup+0x10/0x10
[Mon Mar 11 14:29:16 2024] ep_poll+0x348/0x3b0
[Mon Mar 11 14:29:16 2024] ? __pfx_autoremove_wake_function+0x10/0x10
[Mon Mar 11 14:29:16 2024] do_epoll_wait+0xb1/0xd0
[Mon Mar 11 14:29:16 2024] __x64_sys_epoll_wait+0x60/0x100
[Mon Mar 11 14:29:16 2024] do_syscall_64+0x59/0x90
[Mon Mar 11 14:29:16 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:16 2024] ? exc_page_fault+0x62/0x150
[Mon Mar 11 14:29:16 2024] entry_SYSCALL_64_after_hwframe+0x72/0xdc
[Mon Mar 11 14:29:16 2024] RIP: 0033:0x7fedb7d4e84e
[Mon Mar 11 14:29:16 2024] RSP: 002b:00007ffc284a9ab0 EFLAGS: 00000293 ORIG_RAX: 00000000000000e8
[Mon Mar 11 14:29:16 2024] RAX: ffffffffffffffda RBX: 000055f87dd397c0 RCX: 00007fedb7d4e84e
[Mon Mar 11 14:29:16 2024] RDX: 0000000000000001 RSI: 00007ffc284a9b0c RDI: 0000000000000005
[Mon Mar 11 14:29:16 2024] RBP: 000055f87dd2aa20 R08: 0000000000000000 R09: 0000000000000000
[Mon Mar 11 14:29:16 2024] R10: 000000000000ea60 R11: 0000000000000293 R12: 000055f87dd58980
[Mon Mar 11 14:29:16 2024] R13: 000000000000ea60 R14: 0000000000000027 R15: 000055f87dd5b760
[Mon Mar 11 14:29:16 2024] </TASK>
[Mon Mar 11 14:29:16 2024] task:smbd[10.87.28.1 state:S stack:0 pid:2419997 ppid:4757 flags:0x00000002
[Mon Mar 11 14:29:16 2024] Call Trace:
[Mon Mar 11 14:29:16 2024] <TASK>
[Mon Mar 11 14:29:16 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:16 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:16 2024] schedule_hrtimeout_range_clock+0x95/0x120
[Mon Mar 11 14:29:16 2024] ? __pfx_hrtimer_wakeup+0x10/0x10
[Mon Mar 11 14:29:16 2024] ep_poll+0x348/0x3b0
[Mon Mar 11 14:29:16 2024] ? __pfx_autoremove_wake_function+0x10/0x10
[Mon Mar 11 14:29:16 2024] do_epoll_wait+0xb1/0xd0
[Mon Mar 11 14:29:16 2024] __x64_sys_epoll_wait+0x60/0x100
[Mon Mar 11 14:29:16 2024] do_syscall_64+0x59/0x90
[Mon Mar 11 14:29:16 2024] ? syscall_exit_work+0x103/0x130
[Mon Mar 11 14:29:16 2024] ? syscall_exit_to_user_mode+0x22/0x40
[Mon Mar 11 14:29:16 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:16 2024] ? __irq_exit_rcu+0x46/0xc0
[Mon Mar 11 14:29:16 2024] ? sysvec_apic_timer_interrupt+0x3c/0x90
[Mon Mar 11 14:29:16 2024] entry_SYSCALL_64_after_hwframe+0x72/0xdc
[Mon Mar 11 14:29:16 2024] RIP: 0033:0x7fedb7d4e80a
[Mon Mar 11 14:29:16 2024] RSP: 002b:00007ffc284a9ad8 EFLAGS: 00000246 ORIG_RAX: 00000000000000e8
[Mon Mar 11 14:29:16 2024] RAX: ffffffffffffffda RBX: 000055f87dd397c0 RCX: 00007fedb7d4e80a
[Mon Mar 11 14:29:16 2024] RDX: 0000000000000001 RSI: 00007ffc284a9b0c RDI: 0000000000000005
[Mon Mar 11 14:29:16 2024] RBP: 000055f87dd2aa20 R08: 00000000000f41cf R09: 0000000000000000
[Mon Mar 11 14:29:16 2024] R10: 000000000000ea60 R11: 0000000000000246 R12: 000055f87dd58980
[Mon Mar 11 14:29:16 2024] R13: 000000000000ea60 R14: 0000000000000027 R15: 000055f87dd5b760
[Mon Mar 11 14:29:16 2024] </TASK>
[Mon Mar 11 14:29:16 2024] task:smbd[10.40.153. state:S stack:0 pid:2423410 ppid:4757 flags:0x00000002
[Mon Mar 11 14:29:16 2024] Call Trace:
[Mon Mar 11 14:29:16 2024] <TASK>
[Mon Mar 11 14:29:16 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:16 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:16 2024] schedule_hrtimeout_range_clock+0x95/0x120
[Mon Mar 11 14:29:16 2024] ? __pfx_hrtimer_wakeup+0x10/0x10
[Mon Mar 11 14:29:16 2024] ep_poll+0x348/0x3b0
[Mon Mar 11 14:29:16 2024] ? __pfx_autoremove_wake_function+0x10/0x10
[Mon Mar 11 14:29:16 2024] do_epoll_wait+0xb1/0xd0
[Mon Mar 11 14:29:16 2024] __x64_sys_epoll_wait+0x60/0x100
[Mon Mar 11 14:29:16 2024] do_syscall_64+0x59/0x90
[Mon Mar 11 14:29:16 2024] ? syscall_exit_work+0x103/0x130
[Mon Mar 11 14:29:16 2024] ? syscall_exit_to_user_mode+0x22/0x40
[Mon Mar 11 14:29:16 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:16 2024] ? syscall_exit_to_user_mode+0x22/0x40
[Mon Mar 11 14:29:16 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:16 2024] entry_SYSCALL_64_after_hwframe+0x72/0xdc
[Mon Mar 11 14:29:16 2024] RIP: 0033:0x7fedb7d4e84e
[Mon Mar 11 14:29:16 2024] RSP: 002b:00007ffc284a9ab0 EFLAGS: 00000293 ORIG_RAX: 00000000000000e8
[Mon Mar 11 14:29:16 2024] RAX: ffffffffffffffda RBX: 000055f87dd397c0 RCX: 00007fedb7d4e84e
[Mon Mar 11 14:29:16 2024] RDX: 0000000000000001 RSI: 00007ffc284a9b0c RDI: 0000000000000005
[Mon Mar 11 14:29:16 2024] RBP: 000055f87dd2aa20 R08: 0000000000000000 R09: 0000000000000000
[Mon Mar 11 14:29:16 2024] R10: 000000000000ea60 R11: 0000000000000293 R12: 000055f87dd61190
[Mon Mar 11 14:29:16 2024] R13: 000000000000ea60 R14: 0000000000000026 R15: 000055f87dd5b760
[Mon Mar 11 14:29:16 2024] </TASK>
[Mon Mar 11 14:29:16 2024] task:kworker/2:2 state:I stack:0 pid:2424238 ppid:2 flags:0x00004000
[Mon Mar 11 14:29:16 2024] Workqueue: 0x0 (mm_percpu_wq)
[Mon Mar 11 14:29:16 2024] Call Trace:
[Mon Mar 11 14:29:16 2024] <TASK>
[Mon Mar 11 14:29:16 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:16 2024] ? __pfx_worker_thread+0x10/0x10
[Mon Mar 11 14:29:16 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:16 2024] worker_thread+0xbb/0x3a0
[Mon Mar 11 14:29:16 2024] ? __pfx_worker_thread+0x10/0x10
[Mon Mar 11 14:29:16 2024] kthread+0xdd/0x100
[Mon Mar 11 14:29:16 2024] ? __pfx_kthread+0x10/0x10
[Mon Mar 11 14:29:16 2024] ret_from_fork+0x29/0x50
[Mon Mar 11 14:29:16 2024] </TASK>
[Mon Mar 11 14:29:16 2024] task:smbd[10.87.18.7 state:S stack:0 pid:2425212 ppid:4757 flags:0x00000002
[Mon Mar 11 14:29:16 2024] Call Trace:
[Mon Mar 11 14:29:16 2024] <TASK>
[Mon Mar 11 14:29:16 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:16 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:16 2024] schedule_hrtimeout_range_clock+0x95/0x120
[Mon Mar 11 14:29:16 2024] ? __pfx_hrtimer_wakeup+0x10/0x10
[Mon Mar 11 14:29:16 2024] ep_poll+0x348/0x3b0
[Mon Mar 11 14:29:16 2024] ? __pfx_autoremove_wake_function+0x10/0x10
[Mon Mar 11 14:29:16 2024] do_epoll_wait+0xb1/0xd0
[Mon Mar 11 14:29:16 2024] __x64_sys_epoll_wait+0x60/0x100
[Mon Mar 11 14:29:16 2024] do_syscall_64+0x59/0x90
[Mon Mar 11 14:29:16 2024] ? audit_reset_context.part.0.constprop.0+0x273/0x2e0
[Mon Mar 11 14:29:16 2024] ? syscall_exit_work+0x103/0x130
[Mon Mar 11 14:29:16 2024] ? syscall_exit_to_user_mode+0x22/0x40
[Mon Mar 11 14:29:16 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:16 2024] ? syscall_exit_work+0x103/0x130
[Mon Mar 11 14:29:16 2024] ? syscall_exit_to_user_mode+0x22/0x40
[Mon Mar 11 14:29:16 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:16 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:16 2024] entry_SYSCALL_64_after_hwframe+0x72/0xdc
[Mon Mar 11 14:29:16 2024] RIP: 0033:0x7fedb7d4e80a
[Mon Mar 11 14:29:16 2024] RSP: 002b:00007ffc284a9ad8 EFLAGS: 00000246 ORIG_RAX: 00000000000000e8
[Mon Mar 11 14:29:16 2024] RAX: ffffffffffffffda RBX: 000055f87dd397c0 RCX: 00007fedb7d4e80a
[Mon Mar 11 14:29:16 2024] RDX: 0000000000000001 RSI: 00007ffc284a9b0c RDI: 0000000000000005
[Mon Mar 11 14:29:16 2024] RBP: 000055f87dd2aa20 R08: 00000000000f41ce R09: 0000000000000000
[Mon Mar 11 14:29:16 2024] R10: 000000000000ea60 R11: 0000000000000246 R12: 000055f87dd61190
[Mon Mar 11 14:29:16 2024] R13: 000000000000ea60 R14: 0000000000000026 R15: 000055f87dd5b760
[Mon Mar 11 14:29:16 2024] </TASK>
[Mon Mar 11 14:29:16 2024] task:kworker/21:0 state:I stack:0 pid:2425747 ppid:2 flags:0x00004000
[Mon Mar 11 14:29:16 2024] Workqueue: 0x0 (mm_percpu_wq)
[Mon Mar 11 14:29:16 2024] Call Trace:
[Mon Mar 11 14:29:16 2024] <TASK>
[Mon Mar 11 14:29:16 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:16 2024] ? __pfx_worker_thread+0x10/0x10
[Mon Mar 11 14:29:16 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:16 2024] worker_thread+0xbb/0x3a0
[Mon Mar 11 14:29:16 2024] ? __pfx_worker_thread+0x10/0x10
[Mon Mar 11 14:29:16 2024] kthread+0xdd/0x100
[Mon Mar 11 14:29:16 2024] ? __pfx_kthread+0x10/0x10
[Mon Mar 11 14:29:16 2024] ret_from_fork+0x29/0x50
[Mon Mar 11 14:29:16 2024] </TASK>
[Mon Mar 11 14:29:16 2024] task:kworker/42:0 state:I stack:0 pid:2427884 ppid:2 flags:0x00004000
[Mon Mar 11 14:29:16 2024] Workqueue: 0x0 (mm_percpu_wq)
[Mon Mar 11 14:29:16 2024] Call Trace:
[Mon Mar 11 14:29:16 2024] <TASK>
[Mon Mar 11 14:29:16 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:16 2024] ? __pfx_worker_thread+0x10/0x10
[Mon Mar 11 14:29:16 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:16 2024] worker_thread+0xbb/0x3a0
[Mon Mar 11 14:29:16 2024] ? __pfx_worker_thread+0x10/0x10
[Mon Mar 11 14:29:16 2024] kthread+0xdd/0x100
[Mon Mar 11 14:29:16 2024] ? __pfx_kthread+0x10/0x10
[Mon Mar 11 14:29:16 2024] ret_from_fork+0x29/0x50
[Mon Mar 11 14:29:16 2024] </TASK>
[Mon Mar 11 14:29:16 2024] task:smbd[10.87.18.7 state:S stack:0 pid:2428733 ppid:4757 flags:0x00000002
[Mon Mar 11 14:29:16 2024] Call Trace:
[Mon Mar 11 14:29:16 2024] <TASK>
[Mon Mar 11 14:29:16 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:16 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:16 2024] schedule_hrtimeout_range_clock+0x95/0x120
[Mon Mar 11 14:29:16 2024] ? __pfx_hrtimer_wakeup+0x10/0x10
[Mon Mar 11 14:29:16 2024] ep_poll+0x348/0x3b0
[Mon Mar 11 14:29:16 2024] ? __pfx_autoremove_wake_function+0x10/0x10
[Mon Mar 11 14:29:16 2024] do_epoll_wait+0xb1/0xd0
[Mon Mar 11 14:29:16 2024] __x64_sys_epoll_wait+0x60/0x100
[Mon Mar 11 14:29:16 2024] do_syscall_64+0x59/0x90
[Mon Mar 11 14:29:16 2024] ? syscall_exit_work+0x103/0x130
[Mon Mar 11 14:29:16 2024] ? syscall_exit_to_user_mode+0x22/0x40
[Mon Mar 11 14:29:16 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:16 2024] ? syscall_exit_to_user_mode+0x22/0x40
[Mon Mar 11 14:29:16 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:16 2024] ? syscall_exit_to_user_mode+0x22/0x40
[Mon Mar 11 14:29:16 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:16 2024] ? exc_page_fault+0x62/0x150
[Mon Mar 11 14:29:16 2024] entry_SYSCALL_64_after_hwframe+0x72/0xdc
[Mon Mar 11 14:29:16 2024] RIP: 0033:0x7fedb7d4e84e
[Mon Mar 11 14:29:16 2024] RSP: 002b:00007ffc284a9ab0 EFLAGS: 00000293 ORIG_RAX: 00000000000000e8
[Mon Mar 11 14:29:16 2024] RAX: ffffffffffffffda RBX: 000055f87dd397c0 RCX: 00007fedb7d4e84e
[Mon Mar 11 14:29:16 2024] RDX: 0000000000000001 RSI: 00007ffc284a9b0c RDI: 0000000000000005
[Mon Mar 11 14:29:16 2024] RBP: 000055f87dd2aa20 R08: 0000000000000000 R09: 0000000000000000
[Mon Mar 11 14:29:16 2024] R10: 000000000000ea60 R11: 0000000000000293 R12: 000055f87dd61190
[Mon Mar 11 14:29:16 2024] R13: 000000000000ea60 R14: 0000000000000026 R15: 000055f87dd5b760
[Mon Mar 11 14:29:16 2024] </TASK>
[Mon Mar 11 14:29:16 2024] task:kworker/8:2 state:I stack:0 pid:2431582 ppid:2 flags:0x00004000
[Mon Mar 11 14:29:16 2024] Workqueue: 0x0 (mm_percpu_wq)
[Mon Mar 11 14:29:16 2024] Call Trace:
[Mon Mar 11 14:29:16 2024] <TASK>
[Mon Mar 11 14:29:16 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:16 2024] ? __pfx_worker_thread+0x10/0x10
[Mon Mar 11 14:29:16 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:16 2024] worker_thread+0xbb/0x3a0
[Mon Mar 11 14:29:16 2024] ? __pfx_worker_thread+0x10/0x10
[Mon Mar 11 14:29:16 2024] kthread+0xdd/0x100
[Mon Mar 11 14:29:16 2024] ? __pfx_kthread+0x10/0x10
[Mon Mar 11 14:29:16 2024] ret_from_fork+0x29/0x50
[Mon Mar 11 14:29:16 2024] </TASK>
[Mon Mar 11 14:29:16 2024] task:smbd[10.87.18.9 state:S stack:0 pid:2432065 ppid:4757 flags:0x00000002
[Mon Mar 11 14:29:16 2024] Call Trace:
[Mon Mar 11 14:29:16 2024] <TASK>
[Mon Mar 11 14:29:16 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:16 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:16 2024] schedule_hrtimeout_range_clock+0x95/0x120
[Mon Mar 11 14:29:16 2024] ? __pfx_hrtimer_wakeup+0x10/0x10
[Mon Mar 11 14:29:16 2024] ep_poll+0x348/0x3b0
[Mon Mar 11 14:29:16 2024] ? __pfx_autoremove_wake_function+0x10/0x10
[Mon Mar 11 14:29:16 2024] do_epoll_wait+0xb1/0xd0
[Mon Mar 11 14:29:16 2024] __x64_sys_epoll_wait+0x60/0x100
[Mon Mar 11 14:29:16 2024] do_syscall_64+0x59/0x90
[Mon Mar 11 14:29:16 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:16 2024] ? syscall_exit_to_user_mode+0x22/0x40
[Mon Mar 11 14:29:16 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:16 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:16 2024] ? common_interrupt+0x43/0xa0
[Mon Mar 11 14:29:16 2024] entry_SYSCALL_64_after_hwframe+0x72/0xdc
[Mon Mar 11 14:29:16 2024] RIP: 0033:0x7fedb7d4e80a
[Mon Mar 11 14:29:16 2024] RSP: 002b:00007ffc284a9ad8 EFLAGS: 00000246 ORIG_RAX: 00000000000000e8
[Mon Mar 11 14:29:16 2024] RAX: ffffffffffffffda RBX: 000055f87dd397c0 RCX: 00007fedb7d4e80a
[Mon Mar 11 14:29:16 2024] RDX: 0000000000000001 RSI: 00007ffc284a9b0c RDI: 0000000000000005
[Mon Mar 11 14:29:16 2024] RBP: 000055f87dd2aa20 R08: 00000000000f41da R09: 0000000000000000
[Mon Mar 11 14:29:16 2024] R10: 000000000000ea60 R11: 0000000000000246 R12: 000055f87dd61190
[Mon Mar 11 14:29:16 2024] R13: 000000000000ea60 R14: 0000000000000026 R15: 000055f87dd5b760
[Mon Mar 11 14:29:16 2024] </TASK>
[Mon Mar 11 14:29:16 2024] task:smbd[10.87.18.9 state:S stack:0 pid:2432071 ppid:4757 flags:0x00000002
[Mon Mar 11 14:29:16 2024] Call Trace:
[Mon Mar 11 14:29:16 2024] <TASK>
[Mon Mar 11 14:29:16 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:16 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:16 2024] schedule_hrtimeout_range_clock+0x95/0x120
[Mon Mar 11 14:29:16 2024] ? __pfx_hrtimer_wakeup+0x10/0x10
[Mon Mar 11 14:29:16 2024] ep_poll+0x348/0x3b0
[Mon Mar 11 14:29:16 2024] ? __pfx_autoremove_wake_function+0x10/0x10
[Mon Mar 11 14:29:16 2024] do_epoll_wait+0xb1/0xd0
[Mon Mar 11 14:29:16 2024] __x64_sys_epoll_wait+0x60/0x100
[Mon Mar 11 14:29:16 2024] do_syscall_64+0x59/0x90
[Mon Mar 11 14:29:16 2024] ? syscall_exit_work+0x103/0x130
[Mon Mar 11 14:29:16 2024] ? syscall_exit_to_user_mode+0x22/0x40
[Mon Mar 11 14:29:16 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:16 2024] ? __do_softirq+0x16a/0x2ac
[Mon Mar 11 14:29:16 2024] ? __irq_exit_rcu+0x46/0xc0
[Mon Mar 11 14:29:16 2024] ? sysvec_apic_timer_interrupt+0x3c/0x90
[Mon Mar 11 14:29:16 2024] entry_SYSCALL_64_after_hwframe+0x72/0xdc
[Mon Mar 11 14:29:16 2024] RIP: 0033:0x7fedb7d4e80a
[Mon Mar 11 14:29:16 2024] RSP: 002b:00007ffc284a9ad8 EFLAGS: 00000246 ORIG_RAX: 00000000000000e8
[Mon Mar 11 14:29:16 2024] RAX: ffffffffffffffda RBX: 000055f87dd397c0 RCX: 00007fedb7d4e80a
[Mon Mar 11 14:29:16 2024] RDX: 0000000000000001 RSI: 00007ffc284a9b0c RDI: 0000000000000005
[Mon Mar 11 14:29:16 2024] RBP: 000055f87dd2aa20 R08: 00000000000f41c1 R09: 0000000000000000
[Mon Mar 11 14:29:16 2024] R10: 000000000000ea60 R11: 0000000000000246 R12: 000055f87dd61190
[Mon Mar 11 14:29:16 2024] R13: 000000000000ea60 R14: 0000000000000026 R15: 000055f87dd5b760
[Mon Mar 11 14:29:16 2024] </TASK>
[Mon Mar 11 14:29:16 2024] task:smbd[10.87.31.1 state:S stack:0 pid:2432547 ppid:4757 flags:0x00000002
[Mon Mar 11 14:29:16 2024] Call Trace:
[Mon Mar 11 14:29:16 2024] <TASK>
[Mon Mar 11 14:29:16 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:16 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:16 2024] schedule_hrtimeout_range_clock+0x95/0x120
[Mon Mar 11 14:29:16 2024] ? __pfx_hrtimer_wakeup+0x10/0x10
[Mon Mar 11 14:29:16 2024] ep_poll+0x348/0x3b0
[Mon Mar 11 14:29:16 2024] ? __pfx_autoremove_wake_function+0x10/0x10
[Mon Mar 11 14:29:16 2024] do_epoll_wait+0xb1/0xd0
[Mon Mar 11 14:29:16 2024] __x64_sys_epoll_wait+0x60/0x100
[Mon Mar 11 14:29:16 2024] do_syscall_64+0x59/0x90
[Mon Mar 11 14:29:16 2024] ? exc_page_fault+0x62/0x150
[Mon Mar 11 14:29:16 2024] entry_SYSCALL_64_after_hwframe+0x72/0xdc
[Mon Mar 11 14:29:16 2024] RIP: 0033:0x7fedb7d4e80a
[Mon Mar 11 14:29:16 2024] RSP: 002b:00007ffc284a9ad8 EFLAGS: 00000246 ORIG_RAX: 00000000000000e8
[Mon Mar 11 14:29:16 2024] RAX: ffffffffffffffda RBX: 000055f87dd397c0 RCX: 00007fedb7d4e80a
[Mon Mar 11 14:29:16 2024] RDX: 0000000000000001 RSI: 00007ffc284a9b0c RDI: 0000000000000005
[Mon Mar 11 14:29:16 2024] RBP: 000055f87dd2aa20 R08: 00000000000f4074 R09: 0000000000000000
[Mon Mar 11 14:29:16 2024] R10: 000000000000ea60 R11: 0000000000000246 R12: 000055f87dd61190
[Mon Mar 11 14:29:16 2024] R13: 000000000000ea60 R14: 0000000000000026 R15: 000055f87dd5b760
[Mon Mar 11 14:29:16 2024] </TASK>
[Mon Mar 11 14:29:16 2024] task:kworker/15:1 state:I stack:0 pid:2432551 ppid:2 flags:0x00004000
[Mon Mar 11 14:29:16 2024] Workqueue: 0x0 (mm_percpu_wq)
[Mon Mar 11 14:29:16 2024] Call Trace:
[Mon Mar 11 14:29:16 2024] <TASK>
[Mon Mar 11 14:29:16 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:16 2024] ? __pfx_worker_thread+0x10/0x10
[Mon Mar 11 14:29:16 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:16 2024] worker_thread+0xbb/0x3a0
[Mon Mar 11 14:29:16 2024] ? __pfx_worker_thread+0x10/0x10
[Mon Mar 11 14:29:16 2024] kthread+0xdd/0x100
[Mon Mar 11 14:29:16 2024] ? __pfx_kthread+0x10/0x10
[Mon Mar 11 14:29:16 2024] ret_from_fork+0x29/0x50
[Mon Mar 11 14:29:16 2024] </TASK>
[Mon Mar 11 14:29:16 2024] task:kworker/0:0 state:I stack:0 pid:2433691 ppid:2 flags:0x00004000
[Mon Mar 11 14:29:16 2024] Workqueue: 0x0 (lpfc_wq)
[Mon Mar 11 14:29:16 2024] Call Trace:
[Mon Mar 11 14:29:16 2024] <TASK>
[Mon Mar 11 14:29:16 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:16 2024] ? __pfx_worker_thread+0x10/0x10
[Mon Mar 11 14:29:16 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:16 2024] worker_thread+0xbb/0x3a0
[Mon Mar 11 14:29:16 2024] ? __pfx_worker_thread+0x10/0x10
[Mon Mar 11 14:29:16 2024] kthread+0xdd/0x100
[Mon Mar 11 14:29:16 2024] ? __pfx_kthread+0x10/0x10
[Mon Mar 11 14:29:16 2024] ret_from_fork+0x29/0x50
[Mon Mar 11 14:29:16 2024] </TASK>
[Mon Mar 11 14:29:16 2024] task:smbd[10.87.31.4 state:S stack:0 pid:2433848 ppid:4757 flags:0x00000002
[Mon Mar 11 14:29:16 2024] Call Trace:
[Mon Mar 11 14:29:16 2024] <TASK>
[Mon Mar 11 14:29:16 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:16 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:16 2024] schedule_hrtimeout_range_clock+0x95/0x120
[Mon Mar 11 14:29:16 2024] ? __pfx_hrtimer_wakeup+0x10/0x10
[Mon Mar 11 14:29:16 2024] ep_poll+0x348/0x3b0
[Mon Mar 11 14:29:16 2024] ? __pfx_autoremove_wake_function+0x10/0x10
[Mon Mar 11 14:29:16 2024] do_epoll_wait+0xb1/0xd0
[Mon Mar 11 14:29:16 2024] __x64_sys_epoll_wait+0x60/0x100
[Mon Mar 11 14:29:16 2024] do_syscall_64+0x59/0x90
[Mon Mar 11 14:29:16 2024] ? syscall_exit_work+0x103/0x130
[Mon Mar 11 14:29:16 2024] ? syscall_exit_to_user_mode+0x22/0x40
[Mon Mar 11 14:29:16 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:16 2024] ? exit_to_user_mode_prepare+0xec/0x100
[Mon Mar 11 14:29:16 2024] ? syscall_exit_to_user_mode+0x22/0x40
[Mon Mar 11 14:29:16 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:16 2024] entry_SYSCALL_64_after_hwframe+0x72/0xdc
[Mon Mar 11 14:29:16 2024] RIP: 0033:0x7fedb7d4e80a
[Mon Mar 11 14:29:16 2024] RSP: 002b:00007ffc284a9ad8 EFLAGS: 00000246 ORIG_RAX: 00000000000000e8
[Mon Mar 11 14:29:16 2024] RAX: ffffffffffffffda RBX: 000055f87dd397c0 RCX: 00007fedb7d4e80a
[Mon Mar 11 14:29:16 2024] RDX: 0000000000000001 RSI: 00007ffc284a9b0c RDI: 0000000000000005
[Mon Mar 11 14:29:16 2024] RBP: 000055f87dd2aa20 R08: 00000000000f41dc R09: 0000000000000077
[Mon Mar 11 14:29:16 2024] R10: 000000000000ea60 R11: 0000000000000246 R12: 000055f87dd61190
[Mon Mar 11 14:29:16 2024] R13: 000000000000ea60 R14: 0000000000000026 R15: 000055f87dd5b760
[Mon Mar 11 14:29:16 2024] </TASK>
[Mon Mar 11 14:29:16 2024] task:smbd[10.40.231. state:S stack:0 pid:2433966 ppid:4757 flags:0x00000002
[Mon Mar 11 14:29:16 2024] Call Trace:
[Mon Mar 11 14:29:16 2024] <TASK>
[Mon Mar 11 14:29:16 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:16 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:16 2024] schedule_hrtimeout_range_clock+0x95/0x120
[Mon Mar 11 14:29:16 2024] ? __pfx_hrtimer_wakeup+0x10/0x10
[Mon Mar 11 14:29:16 2024] ep_poll+0x348/0x3b0
[Mon Mar 11 14:29:16 2024] ? __pfx_autoremove_wake_function+0x10/0x10
[Mon Mar 11 14:29:16 2024] do_epoll_wait+0xb1/0xd0
[Mon Mar 11 14:29:16 2024] __x64_sys_epoll_wait+0x60/0x100
[Mon Mar 11 14:29:16 2024] do_syscall_64+0x59/0x90
[Mon Mar 11 14:29:16 2024] ? audit_reset_context.part.0.constprop.0+0x273/0x2e0
[Mon Mar 11 14:29:16 2024] ? syscall_exit_work+0x103/0x130
[Mon Mar 11 14:29:16 2024] ? syscall_exit_to_user_mode+0x22/0x40
[Mon Mar 11 14:29:16 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:16 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:16 2024] entry_SYSCALL_64_after_hwframe+0x72/0xdc
[Mon Mar 11 14:29:16 2024] RIP: 0033:0x7fedb7d4e84e
[Mon Mar 11 14:29:16 2024] RSP: 002b:00007ffc284a9ab0 EFLAGS: 00000293 ORIG_RAX: 00000000000000e8
[Mon Mar 11 14:29:16 2024] RAX: ffffffffffffffda RBX: 000055f87dd397c0 RCX: 00007fedb7d4e84e
[Mon Mar 11 14:29:16 2024] RDX: 0000000000000001 RSI: 00007ffc284a9b0c RDI: 0000000000000005
[Mon Mar 11 14:29:16 2024] RBP: 000055f87dd2aa20 R08: 0000000000000000 R09: 0000000000000000
[Mon Mar 11 14:29:16 2024] R10: 000000000000ea60 R11: 0000000000000293 R12: 000055f87dd58980
[Mon Mar 11 14:29:16 2024] R13: 000000000000ea60 R14: 0000000000000027 R15: 000055f87dd5b760
[Mon Mar 11 14:29:16 2024] </TASK>
[Mon Mar 11 14:29:16 2024] task:smbd[10.87.19.4 state:S stack:0 pid:2434016 ppid:4757 flags:0x00000002
[Mon Mar 11 14:29:16 2024] Call Trace:
[Mon Mar 11 14:29:16 2024] <TASK>
[Mon Mar 11 14:29:16 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:16 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:16 2024] schedule_hrtimeout_range_clock+0x95/0x120
[Mon Mar 11 14:29:16 2024] ? __pfx_hrtimer_wakeup+0x10/0x10
[Mon Mar 11 14:29:16 2024] ep_poll+0x348/0x3b0
[Mon Mar 11 14:29:16 2024] ? __pfx_autoremove_wake_function+0x10/0x10
[Mon Mar 11 14:29:16 2024] do_epoll_wait+0xb1/0xd0
[Mon Mar 11 14:29:16 2024] __x64_sys_epoll_wait+0x60/0x100
[Mon Mar 11 14:29:16 2024] do_syscall_64+0x59/0x90
[Mon Mar 11 14:29:16 2024] ? syscall_exit_to_user_mode+0x22/0x40
[Mon Mar 11 14:29:16 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:16 2024] ? syscall_exit_work+0x103/0x130
[Mon Mar 11 14:29:16 2024] ? syscall_exit_to_user_mode+0x22/0x40
[Mon Mar 11 14:29:16 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:16 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:16 2024] ? exc_page_fault+0x62/0x150
[Mon Mar 11 14:29:16 2024] entry_SYSCALL_64_after_hwframe+0x72/0xdc
[Mon Mar 11 14:29:16 2024] RIP: 0033:0x7fedb7d4e80a
[Mon Mar 11 14:29:16 2024] RSP: 002b:00007ffc284a9ad8 EFLAGS: 00000246 ORIG_RAX: 00000000000000e8
[Mon Mar 11 14:29:16 2024] RAX: ffffffffffffffda RBX: 000055f87dd397c0 RCX: 00007fedb7d4e80a
[Mon Mar 11 14:29:16 2024] RDX: 0000000000000001 RSI: 00007ffc284a9b0c RDI: 0000000000000005
[Mon Mar 11 14:29:16 2024] RBP: 000055f87dd2aa20 R08: 00000000000af6ab R09: 0000000000000000
[Mon Mar 11 14:29:16 2024] R10: 0000000000008f6f R11: 0000000000000246 R12: 000055f87dd61190
[Mon Mar 11 14:29:16 2024] R13: 0000000000008f6f R14: 0000000000000026 R15: 000055f87dd5b760
[Mon Mar 11 14:29:16 2024] </TASK>
[Mon Mar 11 14:29:16 2024] task:kworker/33:1 state:I stack:0 pid:2434371 ppid:2 flags:0x00004000
[Mon Mar 11 14:29:16 2024] Workqueue: 0x0 (mm_percpu_wq)
[Mon Mar 11 14:29:16 2024] Call Trace:
[Mon Mar 11 14:29:16 2024] <TASK>
[Mon Mar 11 14:29:16 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:16 2024] ? __pfx_worker_thread+0x10/0x10
[Mon Mar 11 14:29:16 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:16 2024] worker_thread+0xbb/0x3a0
[Mon Mar 11 14:29:16 2024] ? __pfx_worker_thread+0x10/0x10
[Mon Mar 11 14:29:16 2024] kthread+0xdd/0x100
[Mon Mar 11 14:29:16 2024] ? __pfx_kthread+0x10/0x10
[Mon Mar 11 14:29:16 2024] ret_from_fork+0x29/0x50
[Mon Mar 11 14:29:16 2024] </TASK>
[Mon Mar 11 14:29:16 2024] task:kworker/31:2 state:I stack:0 pid:2436168 ppid:2 flags:0x00004000
[Mon Mar 11 14:29:16 2024] Workqueue: 0x0 (mm_percpu_wq)
[Mon Mar 11 14:29:16 2024] Call Trace:
[Mon Mar 11 14:29:16 2024] <TASK>
[Mon Mar 11 14:29:16 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:16 2024] ? __pfx_worker_thread+0x10/0x10
[Mon Mar 11 14:29:16 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:16 2024] worker_thread+0xbb/0x3a0
[Mon Mar 11 14:29:16 2024] ? __pfx_worker_thread+0x10/0x10
[Mon Mar 11 14:29:16 2024] kthread+0xdd/0x100
[Mon Mar 11 14:29:16 2024] ? __pfx_kthread+0x10/0x10
[Mon Mar 11 14:29:16 2024] ret_from_fork+0x29/0x50
[Mon Mar 11 14:29:16 2024] </TASK>
[Mon Mar 11 14:29:16 2024] task:kworker/29:0 state:I stack:0 pid:2436183 ppid:2 flags:0x00004000
[Mon Mar 11 14:29:16 2024] Workqueue: 0x0 (mm_percpu_wq)
[Mon Mar 11 14:29:16 2024] Call Trace:
[Mon Mar 11 14:29:16 2024] <TASK>
[Mon Mar 11 14:29:16 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:16 2024] ? __pfx_worker_thread+0x10/0x10
[Mon Mar 11 14:29:16 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:16 2024] worker_thread+0xbb/0x3a0
[Mon Mar 11 14:29:16 2024] ? __pfx_worker_thread+0x10/0x10
[Mon Mar 11 14:29:16 2024] kthread+0xdd/0x100
[Mon Mar 11 14:29:16 2024] ? __pfx_kthread+0x10/0x10
[Mon Mar 11 14:29:16 2024] ret_from_fork+0x29/0x50
[Mon Mar 11 14:29:16 2024] </TASK>
[Mon Mar 11 14:29:16 2024] task:smbd[10.87.29.1 state:S stack:0 pid:2436431 ppid:4757 flags:0x00000002
[Mon Mar 11 14:29:16 2024] Call Trace:
[Mon Mar 11 14:29:16 2024] <TASK>
[Mon Mar 11 14:29:16 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:16 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:16 2024] schedule_hrtimeout_range_clock+0x95/0x120
[Mon Mar 11 14:29:16 2024] ? __pfx_hrtimer_wakeup+0x10/0x10
[Mon Mar 11 14:29:16 2024] ep_poll+0x348/0x3b0
[Mon Mar 11 14:29:16 2024] ? __pfx_autoremove_wake_function+0x10/0x10
[Mon Mar 11 14:29:16 2024] do_epoll_wait+0xb1/0xd0
[Mon Mar 11 14:29:16 2024] __x64_sys_epoll_wait+0x60/0x100
[Mon Mar 11 14:29:16 2024] do_syscall_64+0x59/0x90
[Mon Mar 11 14:29:16 2024] ? audit_reset_context.part.0.constprop.0+0x273/0x2e0
[Mon Mar 11 14:29:16 2024] ? syscall_exit_work+0x103/0x130
[Mon Mar 11 14:29:16 2024] ? syscall_exit_to_user_mode+0x22/0x40
[Mon Mar 11 14:29:16 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:16 2024] ? common_interrupt+0x43/0xa0
[Mon Mar 11 14:29:16 2024] entry_SYSCALL_64_after_hwframe+0x72/0xdc
[Mon Mar 11 14:29:16 2024] RIP: 0033:0x7fedb7d4e80a
[Mon Mar 11 14:29:16 2024] RSP: 002b:00007ffc284a9ad8 EFLAGS: 00000246 ORIG_RAX: 00000000000000e8
[Mon Mar 11 14:29:16 2024] RAX: ffffffffffffffda RBX: 000055f87dd397c0 RCX: 00007fedb7d4e80a
[Mon Mar 11 14:29:16 2024] RDX: 0000000000000001 RSI: 00007ffc284a9b0c RDI: 0000000000000005
[Mon Mar 11 14:29:16 2024] RBP: 000055f87dd2aa20 R08: 00000000000f41df R09: 0000000000000000
[Mon Mar 11 14:29:16 2024] R10: 000000000000ea60 R11: 0000000000000246 R12: 000055f87dd61190
[Mon Mar 11 14:29:16 2024] R13: 000000000000ea60 R14: 0000000000000026 R15: 000055f87dd5b760
[Mon Mar 11 14:29:16 2024] </TASK>
[Mon Mar 11 14:29:16 2024] task:kworker/11:1 state:R running task stack:0 pid:2440048 ppid:2 flags:0x00004000
[Mon Mar 11 14:29:16 2024] Workqueue: 0x0 (mm_percpu_wq)
[Mon Mar 11 14:29:16 2024] Call Trace:
[Mon Mar 11 14:29:16 2024] <TASK>
[Mon Mar 11 14:29:16 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:16 2024] ? __pfx_worker_thread+0x10/0x10
[Mon Mar 11 14:29:16 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:16 2024] worker_thread+0xbb/0x3a0
[Mon Mar 11 14:29:16 2024] ? __pfx_worker_thread+0x10/0x10
[Mon Mar 11 14:29:16 2024] kthread+0xdd/0x100
[Mon Mar 11 14:29:16 2024] ? __pfx_kthread+0x10/0x10
[Mon Mar 11 14:29:16 2024] ret_from_fork+0x29/0x50
[Mon Mar 11 14:29:16 2024] </TASK>
[Mon Mar 11 14:29:16 2024] task:smbd[10.87.29.1 state:S stack:0 pid:2440119 ppid:4757 flags:0x00000002
[Mon Mar 11 14:29:16 2024] Call Trace:
[Mon Mar 11 14:29:16 2024] <TASK>
[Mon Mar 11 14:29:16 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:16 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:16 2024] schedule_hrtimeout_range_clock+0x95/0x120
[Mon Mar 11 14:29:16 2024] ? __pfx_hrtimer_wakeup+0x10/0x10
[Mon Mar 11 14:29:16 2024] ep_poll+0x348/0x3b0
[Mon Mar 11 14:29:16 2024] ? __pfx_autoremove_wake_function+0x10/0x10
[Mon Mar 11 14:29:16 2024] do_epoll_wait+0xb1/0xd0
[Mon Mar 11 14:29:16 2024] __x64_sys_epoll_wait+0x60/0x100
[Mon Mar 11 14:29:16 2024] do_syscall_64+0x59/0x90
[Mon Mar 11 14:29:16 2024] ? __audit_filter_op+0xa5/0xf0
[Mon Mar 11 14:29:16 2024] ? mntput_no_expire+0x4a/0x250
[Mon Mar 11 14:29:16 2024] ? audit_reset_context.part.0.constprop.0+0x273/0x2e0
[Mon Mar 11 14:29:16 2024] ? syscall_exit_work+0x103/0x130
[Mon Mar 11 14:29:16 2024] ? syscall_exit_to_user_mode+0x22/0x40
[Mon Mar 11 14:29:16 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:16 2024] ? syscall_exit_to_user_mode+0x22/0x40
[Mon Mar 11 14:29:16 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:16 2024] entry_SYSCALL_64_after_hwframe+0x72/0xdc
[Mon Mar 11 14:29:16 2024] RIP: 0033:0x7fedb7d4e80a
[Mon Mar 11 14:29:16 2024] RSP: 002b:00007ffc284a9ad8 EFLAGS: 00000246 ORIG_RAX: 00000000000000e8
[Mon Mar 11 14:29:16 2024] RAX: ffffffffffffffda RBX: 000055f87dd397c0 RCX: 00007fedb7d4e80a
[Mon Mar 11 14:29:16 2024] RDX: 0000000000000001 RSI: 00007ffc284a9b0c RDI: 0000000000000005
[Mon Mar 11 14:29:16 2024] RBP: 000055f87dd2aa20 R08: 00000000000f41d2 R09: 0000000000000077
[Mon Mar 11 14:29:16 2024] R10: 000000000000ea60 R11: 0000000000000246 R12: 000055f87dd58980
[Mon Mar 11 14:29:16 2024] R13: 000000000000ea60 R14: 0000000000000027 R15: 000055f87dd5b760
[Mon Mar 11 14:29:16 2024] </TASK>
[Mon Mar 11 14:29:16 2024] task:smbd[10.87.27.2 state:S stack:0 pid:2440138 ppid:4757 flags:0x00000002
[Mon Mar 11 14:29:16 2024] Call Trace:
[Mon Mar 11 14:29:16 2024] <TASK>
[Mon Mar 11 14:29:16 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:16 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:16 2024] schedule_hrtimeout_range_clock+0x95/0x120
[Mon Mar 11 14:29:16 2024] ? __pfx_hrtimer_wakeup+0x10/0x10
[Mon Mar 11 14:29:16 2024] ep_poll+0x348/0x3b0
[Mon Mar 11 14:29:16 2024] ? __pfx_autoremove_wake_function+0x10/0x10
[Mon Mar 11 14:29:16 2024] do_epoll_wait+0xb1/0xd0
[Mon Mar 11 14:29:16 2024] __x64_sys_epoll_wait+0x60/0x100
[Mon Mar 11 14:29:16 2024] do_syscall_64+0x59/0x90
[Mon Mar 11 14:29:16 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:16 2024] ? exc_page_fault+0x62/0x150
[Mon Mar 11 14:29:16 2024] entry_SYSCALL_64_after_hwframe+0x72/0xdc
[Mon Mar 11 14:29:16 2024] RIP: 0033:0x7fedb7d4e80a
[Mon Mar 11 14:29:16 2024] RSP: 002b:00007ffc284a9ad8 EFLAGS: 00000246 ORIG_RAX: 00000000000000e8
[Mon Mar 11 14:29:16 2024] RAX: ffffffffffffffda RBX: 000055f87dd397c0 RCX: 00007fedb7d4e80a
[Mon Mar 11 14:29:16 2024] RDX: 0000000000000001 RSI: 00007ffc284a9b0c RDI: 0000000000000005
[Mon Mar 11 14:29:16 2024] RBP: 000055f87dd2aa20 R08: 00000000000f41d4 R09: 0000000000000000
[Mon Mar 11 14:29:16 2024] R10: 000000000000ea60 R11: 0000000000000246 R12: 000055f87dd61190
[Mon Mar 11 14:29:16 2024] R13: 000000000000ea60 R14: 0000000000000026 R15: 000055f87dd5b760
[Mon Mar 11 14:29:16 2024] </TASK>
[Mon Mar 11 14:29:16 2024] task:smbd[10.46.163. state:S stack:0 pid:2441678 ppid:4757 flags:0x00000002
[Mon Mar 11 14:29:16 2024] Call Trace:
[Mon Mar 11 14:29:16 2024] <TASK>
[Mon Mar 11 14:29:16 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:16 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:16 2024] schedule_hrtimeout_range_clock+0x95/0x120
[Mon Mar 11 14:29:16 2024] ? __pfx_hrtimer_wakeup+0x10/0x10
[Mon Mar 11 14:29:16 2024] ep_poll+0x348/0x3b0
[Mon Mar 11 14:29:16 2024] ? __pfx_autoremove_wake_function+0x10/0x10
[Mon Mar 11 14:29:16 2024] do_epoll_wait+0xb1/0xd0
[Mon Mar 11 14:29:16 2024] __x64_sys_epoll_wait+0x60/0x100
[Mon Mar 11 14:29:16 2024] do_syscall_64+0x59/0x90
[Mon Mar 11 14:29:16 2024] ? __do_softirq+0x16a/0x2ac
[Mon Mar 11 14:29:16 2024] ? __irq_exit_rcu+0x46/0xc0
[Mon Mar 11 14:29:16 2024] ? common_interrupt+0x43/0xa0
[Mon Mar 11 14:29:16 2024] entry_SYSCALL_64_after_hwframe+0x72/0xdc
[Mon Mar 11 14:29:16 2024] RIP: 0033:0x7fedb7d4e80a
[Mon Mar 11 14:29:16 2024] RSP: 002b:00007ffc284a9ad8 EFLAGS: 00000246 ORIG_RAX: 00000000000000e8
[Mon Mar 11 14:29:16 2024] RAX: ffffffffffffffda RBX: 000055f87dd397c0 RCX: 00007fedb7d4e80a
[Mon Mar 11 14:29:16 2024] RDX: 0000000000000001 RSI: 00007ffc284a9b0c RDI: 0000000000000005
[Mon Mar 11 14:29:16 2024] RBP: 000055f87dd2aa20 R08: 00000000000f3fe4 R09: 0000000000000000
[Mon Mar 11 14:29:16 2024] R10: 000000000000ea60 R11: 0000000000000246 R12: 000055f87dd61190
[Mon Mar 11 14:29:16 2024] R13: 000000000000ea60 R14: 0000000000000026 R15: 000055f87dd5b760
[Mon Mar 11 14:29:16 2024] </TASK>
[Mon Mar 11 14:29:16 2024] task:smbd[10.87.28.1 state:S stack:0 pid:2441978 ppid:4757 flags:0x00000002
[Mon Mar 11 14:29:16 2024] Call Trace:
[Mon Mar 11 14:29:16 2024] <TASK>
[Mon Mar 11 14:29:16 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:16 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:16 2024] schedule_hrtimeout_range_clock+0x95/0x120
[Mon Mar 11 14:29:16 2024] ? __pfx_hrtimer_wakeup+0x10/0x10
[Mon Mar 11 14:29:16 2024] ep_poll+0x348/0x3b0
[Mon Mar 11 14:29:16 2024] ? __pfx_autoremove_wake_function+0x10/0x10
[Mon Mar 11 14:29:16 2024] do_epoll_wait+0xb1/0xd0
[Mon Mar 11 14:29:16 2024] __x64_sys_epoll_wait+0x60/0x100
[Mon Mar 11 14:29:16 2024] do_syscall_64+0x59/0x90
[Mon Mar 11 14:29:16 2024] ? audit_reset_context.part.0.constprop.0+0x273/0x2e0
[Mon Mar 11 14:29:16 2024] ? syscall_exit_work+0x103/0x130
[Mon Mar 11 14:29:16 2024] ? syscall_exit_to_user_mode+0x22/0x40
[Mon Mar 11 14:29:16 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:16 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:16 2024] ? syscall_exit_to_user_mode+0x22/0x40
[Mon Mar 11 14:29:16 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:16 2024] entry_SYSCALL_64_after_hwframe+0x72/0xdc
[Mon Mar 11 14:29:16 2024] RIP: 0033:0x7fedb7d4e84e
[Mon Mar 11 14:29:16 2024] RSP: 002b:00007ffc284a9ab0 EFLAGS: 00000293 ORIG_RAX: 00000000000000e8
[Mon Mar 11 14:29:16 2024] RAX: ffffffffffffffda RBX: 000055f87dd397c0 RCX: 00007fedb7d4e84e
[Mon Mar 11 14:29:16 2024] RDX: 0000000000000001 RSI: 00007ffc284a9b0c RDI: 0000000000000005
[Mon Mar 11 14:29:16 2024] RBP: 000055f87dd2aa20 R08: 0000000000000000 R09: 0000000000000000
[Mon Mar 11 14:29:16 2024] R10: 000000000000ea60 R11: 0000000000000293 R12: 000055f87dd61190
[Mon Mar 11 14:29:16 2024] R13: 000000000000ea60 R14: 0000000000000026 R15: 000055f87dd5b760
[Mon Mar 11 14:29:16 2024] </TASK>
[Mon Mar 11 14:29:16 2024] task:kworker/u96:4 state:I stack:0 pid:2442472 ppid:2 flags:0x00004000
[Mon Mar 11 14:29:16 2024] Workqueue: 0x0 (dm-thin)
[Mon Mar 11 14:29:16 2024] Call Trace:
[Mon Mar 11 14:29:16 2024] <TASK>
[Mon Mar 11 14:29:16 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:16 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:16 2024] worker_thread+0xbb/0x3a0
[Mon Mar 11 14:29:16 2024] ? __pfx_worker_thread+0x10/0x10
[Mon Mar 11 14:29:16 2024] kthread+0xdd/0x100
[Mon Mar 11 14:29:16 2024] ? __pfx_kthread+0x10/0x10
[Mon Mar 11 14:29:16 2024] ret_from_fork+0x29/0x50
[Mon Mar 11 14:29:16 2024] </TASK>
[Mon Mar 11 14:29:16 2024] task:kworker/12:0 state:I stack:0 pid:2442877 ppid:2 flags:0x00004000
[Mon Mar 11 14:29:16 2024] Workqueue: 0x0 (kdmflush/253:33)
[Mon Mar 11 14:29:16 2024] Call Trace:
[Mon Mar 11 14:29:16 2024] <TASK>
[Mon Mar 11 14:29:16 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:16 2024] ? __pfx_worker_thread+0x10/0x10
[Mon Mar 11 14:29:16 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:16 2024] worker_thread+0xbb/0x3a0
[Mon Mar 11 14:29:16 2024] ? __pfx_worker_thread+0x10/0x10
[Mon Mar 11 14:29:16 2024] kthread+0xdd/0x100
[Mon Mar 11 14:29:16 2024] ? __pfx_kthread+0x10/0x10
[Mon Mar 11 14:29:16 2024] ret_from_fork+0x29/0x50
[Mon Mar 11 14:29:16 2024] </TASK>
[Mon Mar 11 14:29:16 2024] task:smbd[10.33.88.4 state:S stack:0 pid:2443657 ppid:4757 flags:0x00000002
[Mon Mar 11 14:29:16 2024] Call Trace:
[Mon Mar 11 14:29:16 2024] <TASK>
[Mon Mar 11 14:29:16 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:16 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:16 2024] schedule_hrtimeout_range_clock+0x95/0x120
[Mon Mar 11 14:29:16 2024] ? __pfx_hrtimer_wakeup+0x10/0x10
[Mon Mar 11 14:29:16 2024] ep_poll+0x348/0x3b0
[Mon Mar 11 14:29:16 2024] ? __pfx_autoremove_wake_function+0x10/0x10
[Mon Mar 11 14:29:16 2024] do_epoll_wait+0xb1/0xd0
[Mon Mar 11 14:29:16 2024] __x64_sys_epoll_wait+0x60/0x100
[Mon Mar 11 14:29:16 2024] do_syscall_64+0x59/0x90
[Mon Mar 11 14:29:16 2024] entry_SYSCALL_64_after_hwframe+0x72/0xdc
[Mon Mar 11 14:29:16 2024] RIP: 0033:0x7fedb7d4e80a
[Mon Mar 11 14:29:16 2024] RSP: 002b:00007ffc284a9ad8 EFLAGS: 00000246 ORIG_RAX: 00000000000000e8
[Mon Mar 11 14:29:16 2024] RAX: ffffffffffffffda RBX: 000055f87dd397c0 RCX: 00007fedb7d4e80a
[Mon Mar 11 14:29:16 2024] RDX: 0000000000000001 RSI: 00007ffc284a9b0c RDI: 0000000000000005
[Mon Mar 11 14:29:16 2024] RBP: 000055f87dd2aa20 R08: 00000000000f41cf R09: 0000000000000000
[Mon Mar 11 14:29:16 2024] R10: 000000000000ea60 R11: 0000000000000246 R12: 000055f87dd58980
[Mon Mar 11 14:29:16 2024] R13: 000000000000ea60 R14: 0000000000000027 R15: 000055f87dd5b760
[Mon Mar 11 14:29:16 2024] </TASK>
[Mon Mar 11 14:29:16 2024] task:smbd[10.40.138. state:S stack:0 pid:2445114 ppid:4757 flags:0x00000002
[Mon Mar 11 14:29:16 2024] Call Trace:
[Mon Mar 11 14:29:16 2024] <TASK>
[Mon Mar 11 14:29:16 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:16 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:16 2024] schedule_hrtimeout_range_clock+0x95/0x120
[Mon Mar 11 14:29:16 2024] ? __pfx_hrtimer_wakeup+0x10/0x10
[Mon Mar 11 14:29:16 2024] ep_poll+0x348/0x3b0
[Mon Mar 11 14:29:16 2024] ? __pfx_autoremove_wake_function+0x10/0x10
[Mon Mar 11 14:29:16 2024] do_epoll_wait+0xb1/0xd0
[Mon Mar 11 14:29:16 2024] __x64_sys_epoll_wait+0x60/0x100
[Mon Mar 11 14:29:16 2024] do_syscall_64+0x59/0x90
[Mon Mar 11 14:29:16 2024] ? syscall_exit_work+0x103/0x130
[Mon Mar 11 14:29:16 2024] ? syscall_exit_to_user_mode+0x22/0x40
[Mon Mar 11 14:29:16 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:16 2024] ? syscall_exit_to_user_mode+0x22/0x40
[Mon Mar 11 14:29:16 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:16 2024] ? exc_page_fault+0x62/0x150
[Mon Mar 11 14:29:16 2024] entry_SYSCALL_64_after_hwframe+0x72/0xdc
[Mon Mar 11 14:29:16 2024] RIP: 0033:0x7fedb7d4e80a
[Mon Mar 11 14:29:16 2024] RSP: 002b:00007ffc284a9ad8 EFLAGS: 00000246 ORIG_RAX: 00000000000000e8
[Mon Mar 11 14:29:16 2024] RAX: ffffffffffffffda RBX: 000055f87dd397c0 RCX: 00007fedb7d4e80a
[Mon Mar 11 14:29:16 2024] RDX: 0000000000000001 RSI: 00007ffc284a9b0c RDI: 0000000000000005
[Mon Mar 11 14:29:16 2024] RBP: 000055f87dd2aa20 R08: 00000000000f41d2 R09: 0000000000000000
[Mon Mar 11 14:29:16 2024] R10: 000000000000ea60 R11: 0000000000000246 R12: 000055f87dd61190
[Mon Mar 11 14:29:16 2024] R13: 000000000000ea60 R14: 0000000000000026 R15: 000055f87dd5b760
[Mon Mar 11 14:29:16 2024] </TASK>
[Mon Mar 11 14:29:16 2024] task:pickup state:S stack:0 pid:2449277 ppid:4982 flags:0x00000002
[Mon Mar 11 14:29:16 2024] Call Trace:
[Mon Mar 11 14:29:16 2024] <TASK>
[Mon Mar 11 14:29:16 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:16 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:16 2024] schedule_hrtimeout_range_clock+0x95/0x120
[Mon Mar 11 14:29:16 2024] ? __pfx_hrtimer_wakeup+0x10/0x10
[Mon Mar 11 14:29:16 2024] ep_poll+0x348/0x3b0
[Mon Mar 11 14:29:16 2024] ? __pfx_autoremove_wake_function+0x10/0x10
[Mon Mar 11 14:29:16 2024] do_epoll_wait+0xb1/0xd0
[Mon Mar 11 14:29:16 2024] __x64_sys_epoll_wait+0x60/0x100
[Mon Mar 11 14:29:16 2024] do_syscall_64+0x59/0x90
[Mon Mar 11 14:29:16 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:16 2024] ? sysvec_apic_timer_interrupt+0x3c/0x90
[Mon Mar 11 14:29:16 2024] entry_SYSCALL_64_after_hwframe+0x72/0xdc
[Mon Mar 11 14:29:16 2024] RIP: 0033:0x7f32d434e80a
[Mon Mar 11 14:29:16 2024] RSP: 002b:00007ffff46c1188 EFLAGS: 00000246 ORIG_RAX: 00000000000000e8
[Mon Mar 11 14:29:16 2024] RAX: ffffffffffffffda RBX: 00007ffff46c11a0 RCX: 00007f32d434e80a
[Mon Mar 11 14:29:16 2024] RDX: 0000000000000064 RSI: 00007ffff46c11a0 RDI: 000000000000000a
[Mon Mar 11 14:29:16 2024] RBP: 00007f32d45a57d0 R08: 0000563eee563940 R09: 0000000000000078
[Mon Mar 11 14:29:16 2024] R10: 00000000000186a0 R11: 0000000000000246 R12: 00007f32d45a5794
[Mon Mar 11 14:29:16 2024] R13: 00000000ffffffff R14: 0000563eec931080 R15: 00007f32d45a5794
[Mon Mar 11 14:29:16 2024] </TASK>
[Mon Mar 11 14:29:16 2024] task:kworker/u96:3 state:D stack:0 pid:2451130 ppid:2 flags:0x00004000
[Mon Mar 11 14:29:16 2024] Workqueue: nfsd4_callbacks nfsd4_run_cb_work [nfsd]
[Mon Mar 11 14:29:16 2024] Call Trace:
[Mon Mar 11 14:29:16 2024] <TASK>
[Mon Mar 11 14:29:16 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:16 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:16 2024] schedule_timeout+0x88/0x160
[Mon Mar 11 14:29:16 2024] ? __pfx_process_timeout+0x10/0x10
[Mon Mar 11 14:29:16 2024] rpc_shutdown_client+0xb3/0x150 [sunrpc]
[Mon Mar 11 14:29:16 2024] ? __pfx_autoremove_wake_function+0x10/0x10
[Mon Mar 11 14:29:16 2024] nfsd4_process_cb_update+0x3e/0x260 [nfsd]
[Mon Mar 11 14:29:16 2024] ? sched_clock+0xc/0x30
[Mon Mar 11 14:29:16 2024] ? raw_spin_rq_lock_nested+0x19/0x80
[Mon Mar 11 14:29:16 2024] ? newidle_balance+0x26e/0x400
[Mon Mar 11 14:29:16 2024] ? pick_next_task_fair+0x41/0x500
[Mon Mar 11 14:29:16 2024] ? put_prev_task_fair+0x1e/0x40
[Mon Mar 11 14:29:16 2024] ? pick_next_task+0x861/0x950
[Mon Mar 11 14:29:16 2024] ? __update_idle_core+0x23/0xc0
[Mon Mar 11 14:29:16 2024] ? __switch_to_asm+0x3a/0x80
[Mon Mar 11 14:29:16 2024] ? finish_task_switch.isra.0+0x8c/0x2a0
[Mon Mar 11 14:29:16 2024] nfsd4_run_cb_work+0x9f/0x150 [nfsd]
[Mon Mar 11 14:29:16 2024] process_one_work+0x1e2/0x3b0
[Mon Mar 11 14:29:16 2024] worker_thread+0x50/0x3a0
[Mon Mar 11 14:29:16 2024] ? __pfx_worker_thread+0x10/0x10
[Mon Mar 11 14:29:16 2024] kthread+0xdd/0x100
[Mon Mar 11 14:29:16 2024] ? __pfx_kthread+0x10/0x10
[Mon Mar 11 14:29:16 2024] ret_from_fork+0x29/0x50
[Mon Mar 11 14:29:16 2024] </TASK>
[Mon Mar 11 14:29:16 2024] task:kworker/u97:1 state:I stack:0 pid:2451949 ppid:2 flags:0x00004000
[Mon Mar 11 14:29:16 2024] Workqueue: 0x0 (events_unbound)
[Mon Mar 11 14:29:16 2024] Call Trace:
[Mon Mar 11 14:29:16 2024] <TASK>
[Mon Mar 11 14:29:16 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:16 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:16 2024] worker_thread+0xbb/0x3a0
[Mon Mar 11 14:29:16 2024] ? __pfx_worker_thread+0x10/0x10
[Mon Mar 11 14:29:16 2024] kthread+0xdd/0x100
[Mon Mar 11 14:29:16 2024] ? __pfx_kthread+0x10/0x10
[Mon Mar 11 14:29:16 2024] ret_from_fork+0x29/0x50
[Mon Mar 11 14:29:16 2024] </TASK>
[Mon Mar 11 14:29:16 2024] task:kworker/4:2 state:I stack:0 pid:2452213 ppid:2 flags:0x00004000
[Mon Mar 11 14:29:16 2024] Workqueue: 0x0 (mm_percpu_wq)
[Mon Mar 11 14:29:16 2024] Call Trace:
[Mon Mar 11 14:29:16 2024] <TASK>
[Mon Mar 11 14:29:16 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:16 2024] ? __pfx_worker_thread+0x10/0x10
[Mon Mar 11 14:29:16 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:16 2024] worker_thread+0xbb/0x3a0
[Mon Mar 11 14:29:16 2024] ? __pfx_worker_thread+0x10/0x10
[Mon Mar 11 14:29:16 2024] kthread+0xdd/0x100
[Mon Mar 11 14:29:16 2024] ? __pfx_kthread+0x10/0x10
[Mon Mar 11 14:29:16 2024] ret_from_fork+0x29/0x50
[Mon Mar 11 14:29:16 2024] </TASK>
[Mon Mar 11 14:29:16 2024] task:smbd[10.87.28.1 state:S stack:0 pid:2452452 ppid:4757 flags:0x00000002
[Mon Mar 11 14:29:16 2024] Call Trace:
[Mon Mar 11 14:29:16 2024] <TASK>
[Mon Mar 11 14:29:16 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:16 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:16 2024] schedule_hrtimeout_range_clock+0x95/0x120
[Mon Mar 11 14:29:16 2024] ? __pfx_hrtimer_wakeup+0x10/0x10
[Mon Mar 11 14:29:16 2024] ep_poll+0x348/0x3b0
[Mon Mar 11 14:29:16 2024] ? __pfx_autoremove_wake_function+0x10/0x10
[Mon Mar 11 14:29:16 2024] do_epoll_wait+0xb1/0xd0
[Mon Mar 11 14:29:16 2024] __x64_sys_epoll_wait+0x60/0x100
[Mon Mar 11 14:29:16 2024] do_syscall_64+0x59/0x90
[Mon Mar 11 14:29:16 2024] ? syscall_exit_work+0x103/0x130
[Mon Mar 11 14:29:16 2024] ? syscall_exit_to_user_mode+0x22/0x40
[Mon Mar 11 14:29:16 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:16 2024] ? syscall_exit_work+0x103/0x130
[Mon Mar 11 14:29:16 2024] ? syscall_exit_to_user_mode+0x22/0x40
[Mon Mar 11 14:29:16 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:16 2024] entry_SYSCALL_64_after_hwframe+0x72/0xdc
[Mon Mar 11 14:29:16 2024] RIP: 0033:0x7fedb7d4e80a
[Mon Mar 11 14:29:16 2024] RSP: 002b:00007ffc284a9ad8 EFLAGS: 00000246 ORIG_RAX: 00000000000000e8
[Mon Mar 11 14:29:16 2024] RAX: ffffffffffffffda RBX: 000055f87dd397c0 RCX: 00007fedb7d4e80a
[Mon Mar 11 14:29:16 2024] RDX: 0000000000000001 RSI: 00007ffc284a9b0c RDI: 0000000000000005
[Mon Mar 11 14:29:16 2024] RBP: 000055f87dd2aa20 R08: 00000000000f41da R09: 0000000000000000
[Mon Mar 11 14:29:16 2024] R10: 000000000000ea60 R11: 0000000000000246 R12: 000055f87dd61190
[Mon Mar 11 14:29:16 2024] R13: 000000000000ea60 R14: 0000000000000026 R15: 000055f87dd5b760
[Mon Mar 11 14:29:16 2024] </TASK>
[Mon Mar 11 14:29:16 2024] task:kworker/47:2 state:I stack:0 pid:2452570 ppid:2 flags:0x00004000
[Mon Mar 11 14:29:16 2024] Workqueue: 0x0 (mm_percpu_wq)
[Mon Mar 11 14:29:16 2024] Call Trace:
[Mon Mar 11 14:29:16 2024] <TASK>
[Mon Mar 11 14:29:16 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:16 2024] ? __pfx_worker_thread+0x10/0x10
[Mon Mar 11 14:29:16 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:16 2024] worker_thread+0xbb/0x3a0
[Mon Mar 11 14:29:16 2024] ? __pfx_worker_thread+0x10/0x10
[Mon Mar 11 14:29:16 2024] kthread+0xdd/0x100
[Mon Mar 11 14:29:16 2024] ? __pfx_kthread+0x10/0x10
[Mon Mar 11 14:29:16 2024] ret_from_fork+0x29/0x50
[Mon Mar 11 14:29:16 2024] </TASK>
[Mon Mar 11 14:29:16 2024] task:smbd[10.87.30.8 state:S stack:0 pid:2453051 ppid:4757 flags:0x00000002
[Mon Mar 11 14:29:16 2024] Call Trace:
[Mon Mar 11 14:29:16 2024] <TASK>
[Mon Mar 11 14:29:16 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:16 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:16 2024] schedule_hrtimeout_range_clock+0x95/0x120
[Mon Mar 11 14:29:16 2024] ? __pfx_hrtimer_wakeup+0x10/0x10
[Mon Mar 11 14:29:16 2024] ep_poll+0x348/0x3b0
[Mon Mar 11 14:29:16 2024] ? __pfx_autoremove_wake_function+0x10/0x10
[Mon Mar 11 14:29:16 2024] do_epoll_wait+0xb1/0xd0
[Mon Mar 11 14:29:16 2024] __x64_sys_epoll_wait+0x60/0x100
[Mon Mar 11 14:29:16 2024] do_syscall_64+0x59/0x90
[Mon Mar 11 14:29:16 2024] ? audit_reset_context.part.0.constprop.0+0x273/0x2e0
[Mon Mar 11 14:29:16 2024] ? syscall_exit_work+0x103/0x130
[Mon Mar 11 14:29:16 2024] ? syscall_exit_to_user_mode+0x22/0x40
[Mon Mar 11 14:29:16 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:16 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:16 2024] entry_SYSCALL_64_after_hwframe+0x72/0xdc
[Mon Mar 11 14:29:16 2024] RIP: 0033:0x7fedb7d4e84e
[Mon Mar 11 14:29:16 2024] RSP: 002b:00007ffc284a9ab0 EFLAGS: 00000293 ORIG_RAX: 00000000000000e8
[Mon Mar 11 14:29:16 2024] RAX: ffffffffffffffda RBX: 000055f87dd397c0 RCX: 00007fedb7d4e84e
[Mon Mar 11 14:29:16 2024] RDX: 0000000000000001 RSI: 00007ffc284a9b0c RDI: 0000000000000005
[Mon Mar 11 14:29:16 2024] RBP: 000055f87dd2aa20 R08: 0000000000000000 R09: 0000000000000000
[Mon Mar 11 14:29:16 2024] R10: 000000000000ea60 R11: 0000000000000293 R12: 000055f87dd61190
[Mon Mar 11 14:29:16 2024] R13: 000000000000ea60 R14: 0000000000000026 R15: 000055f87dd5b760
[Mon Mar 11 14:29:16 2024] </TASK>
[Mon Mar 11 14:29:16 2024] task:kworker/24:2 state:I stack:0 pid:2453510 ppid:2 flags:0x00004000
[Mon Mar 11 14:29:16 2024] Workqueue: 0x0 (mm_percpu_wq)
[Mon Mar 11 14:29:16 2024] Call Trace:
[Mon Mar 11 14:29:16 2024] <TASK>
[Mon Mar 11 14:29:16 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:16 2024] ? __pfx_worker_thread+0x10/0x10
[Mon Mar 11 14:29:16 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:16 2024] worker_thread+0xbb/0x3a0
[Mon Mar 11 14:29:16 2024] ? __pfx_worker_thread+0x10/0x10
[Mon Mar 11 14:29:16 2024] kthread+0xdd/0x100
[Mon Mar 11 14:29:16 2024] ? __pfx_kthread+0x10/0x10
[Mon Mar 11 14:29:16 2024] ret_from_fork+0x29/0x50
[Mon Mar 11 14:29:16 2024] </TASK>
[Mon Mar 11 14:29:16 2024] task:smbd[10.40.140. state:S stack:0 pid:2453653 ppid:4757 flags:0x00000002
[Mon Mar 11 14:29:16 2024] Call Trace:
[Mon Mar 11 14:29:16 2024] <TASK>
[Mon Mar 11 14:29:16 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:16 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:16 2024] schedule_hrtimeout_range_clock+0x95/0x120
[Mon Mar 11 14:29:16 2024] ? __pfx_hrtimer_wakeup+0x10/0x10
[Mon Mar 11 14:29:16 2024] ep_poll+0x348/0x3b0
[Mon Mar 11 14:29:16 2024] ? __pfx_autoremove_wake_function+0x10/0x10
[Mon Mar 11 14:29:16 2024] do_epoll_wait+0xb1/0xd0
[Mon Mar 11 14:29:16 2024] __x64_sys_epoll_wait+0x60/0x100
[Mon Mar 11 14:29:16 2024] do_syscall_64+0x59/0x90
[Mon Mar 11 14:29:16 2024] ? syscall_exit_to_user_mode+0x22/0x40
[Mon Mar 11 14:29:16 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:16 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:16 2024] entry_SYSCALL_64_after_hwframe+0x72/0xdc
[Mon Mar 11 14:29:16 2024] RIP: 0033:0x7fedb7d4e80a
[Mon Mar 11 14:29:16 2024] RSP: 002b:00007ffc284a9ad8 EFLAGS: 00000246 ORIG_RAX: 00000000000000e8
[Mon Mar 11 14:29:16 2024] RAX: ffffffffffffffda RBX: 000055f87dd397c0 RCX: 00007fedb7d4e80a
[Mon Mar 11 14:29:16 2024] RDX: 0000000000000001 RSI: 00007ffc284a9b0c RDI: 0000000000000005
[Mon Mar 11 14:29:16 2024] RBP: 000055f87dd2aa20 R08: 00000000000f41d4 R09: 0000000000000000
[Mon Mar 11 14:29:16 2024] R10: 000000000000ea60 R11: 0000000000000246 R12: 000055f87dd58980
[Mon Mar 11 14:29:16 2024] R13: 000000000000ea60 R14: 0000000000000027 R15: 000055f87dd5b760
[Mon Mar 11 14:29:16 2024] </TASK>
[Mon Mar 11 14:29:16 2024] task:smbd[10.40.233. state:D stack:0 pid:2455152 ppid:4757 flags:0x00004002
[Mon Mar 11 14:29:16 2024] Call Trace:
[Mon Mar 11 14:29:16 2024] <TASK>
[Mon Mar 11 14:29:16 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:16 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:16 2024] io_schedule+0x42/0x70
[Mon Mar 11 14:29:16 2024] bit_wait_io+0xd/0x60
[Mon Mar 11 14:29:16 2024] __wait_on_bit+0x48/0x150
[Mon Mar 11 14:29:16 2024] ? __pfx_bit_wait_io+0x10/0x10
[Mon Mar 11 14:29:16 2024] out_of_line_wait_on_bit+0x92/0xb0
[Mon Mar 11 14:29:16 2024] ? __pfx_wake_bit_function+0x10/0x10
[Mon Mar 11 14:29:16 2024] ext4_read_bh+0x84/0x90 [ext4]
[Mon Mar 11 14:29:16 2024] __ext4_sb_bread_gfp.isra.0+0x5f/0x80 [ext4]
[Mon Mar 11 14:29:16 2024] ext4_xattr_block_get+0x5f/0x1d0 [ext4]
[Mon Mar 11 14:29:16 2024] ext4_xattr_get+0xb1/0xd0 [ext4]
[Mon Mar 11 14:29:16 2024] __vfs_getxattr+0x50/0x70
[Mon Mar 11 14:29:16 2024] get_vfs_caps_from_disk+0x70/0x210
[Mon Mar 11 14:29:16 2024] ? walk_component+0x70/0x1d0
[Mon Mar 11 14:29:16 2024] audit_copy_inode+0x99/0xd0
[Mon Mar 11 14:29:16 2024] filename_lookup+0x17b/0x1d0
[Mon Mar 11 14:29:16 2024] ? __check_object_size.part.0+0x47/0xd0
[Mon Mar 11 14:29:16 2024] ? _copy_to_user+0x1a/0x30
[Mon Mar 11 14:29:16 2024] ? do_getxattr+0xd3/0x160
[Mon Mar 11 14:29:16 2024] ? path_get+0x11/0x30
[Mon Mar 11 14:29:16 2024] vfs_statx+0x8d/0x170
[Mon Mar 11 14:29:16 2024] vfs_fstatat+0x54/0x70
[Mon Mar 11 14:29:16 2024] __do_sys_newfstatat+0x26/0x60
[Mon Mar 11 14:29:16 2024] ? auditd_test_task+0x3c/0x50
[Mon Mar 11 14:29:16 2024] ? __audit_syscall_entry+0xef/0x140
[Mon Mar 11 14:29:16 2024] ? syscall_trace_enter.constprop.0+0x126/0x1a0
[Mon Mar 11 14:29:16 2024] do_syscall_64+0x59/0x90
[Mon Mar 11 14:29:16 2024] ? __rseq_handle_notify_resume+0x26/0xb0
[Mon Mar 11 14:29:16 2024] ? exit_to_user_mode_loop+0xd0/0x130
[Mon Mar 11 14:29:16 2024] ? exit_to_user_mode_prepare+0xb6/0x100
[Mon Mar 11 14:29:16 2024] ? syscall_exit_to_user_mode+0x22/0x40
[Mon Mar 11 14:29:16 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:16 2024] ? exc_page_fault+0x62/0x150
[Mon Mar 11 14:29:16 2024] entry_SYSCALL_64_after_hwframe+0x72/0xdc
[Mon Mar 11 14:29:16 2024] RIP: 0033:0x7fedb7d3debe
[Mon Mar 11 14:29:16 2024] RSP: 002b:00007ffc284a9048 EFLAGS: 00000246 ORIG_RAX: 0000000000000106
[Mon Mar 11 14:29:16 2024] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007fedb7d3debe
[Mon Mar 11 14:29:16 2024] RDX: 00007ffc284a9050 RSI: 000055f87dd97493 RDI: 0000000000000023
[Mon Mar 11 14:29:16 2024] RBP: 00007ffc284a9150 R08: 0000000000000000 R09: 0000000000000001
[Mon Mar 11 14:29:16 2024] R10: 0000000000000100 R11: 0000000000000246 R12: 00007ffc284a9150
[Mon Mar 11 14:29:16 2024] R13: 00007ffc284a9050 R14: 00007ffc284a9390 R15: 000055f87dd97480
[Mon Mar 11 14:29:16 2024] </TASK>
[Mon Mar 11 14:29:16 2024] task:smbd[10.40.146. state:S stack:0 pid:2456493 ppid:4757 flags:0x00000002
[Mon Mar 11 14:29:16 2024] Call Trace:
[Mon Mar 11 14:29:16 2024] <TASK>
[Mon Mar 11 14:29:16 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:16 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:16 2024] schedule_hrtimeout_range_clock+0x95/0x120
[Mon Mar 11 14:29:16 2024] ? __pfx_hrtimer_wakeup+0x10/0x10
[Mon Mar 11 14:29:16 2024] ep_poll+0x348/0x3b0
[Mon Mar 11 14:29:16 2024] ? __pfx_autoremove_wake_function+0x10/0x10
[Mon Mar 11 14:29:16 2024] do_epoll_wait+0xb1/0xd0
[Mon Mar 11 14:29:16 2024] __x64_sys_epoll_wait+0x60/0x100
[Mon Mar 11 14:29:16 2024] do_syscall_64+0x59/0x90
[Mon Mar 11 14:29:16 2024] ? audit_reset_context.part.0.constprop.0+0x273/0x2e0
[Mon Mar 11 14:29:16 2024] ? syscall_exit_work+0x103/0x130
[Mon Mar 11 14:29:16 2024] ? syscall_exit_to_user_mode+0x22/0x40
[Mon Mar 11 14:29:16 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:16 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:16 2024] ? exc_page_fault+0x62/0x150
[Mon Mar 11 14:29:16 2024] entry_SYSCALL_64_after_hwframe+0x72/0xdc
[Mon Mar 11 14:29:16 2024] RIP: 0033:0x7fedb7d4e80a
[Mon Mar 11 14:29:16 2024] RSP: 002b:00007ffc284a9ad8 EFLAGS: 00000246 ORIG_RAX: 00000000000000e8
[Mon Mar 11 14:29:16 2024] RAX: ffffffffffffffda RBX: 000055f87dd397c0 RCX: 00007fedb7d4e80a
[Mon Mar 11 14:29:16 2024] RDX: 0000000000000001 RSI: 00007ffc284a9b0c RDI: 0000000000000005
[Mon Mar 11 14:29:16 2024] RBP: 000055f87dd2aa20 R08: 00000000000f4107 R09: 0000000000000000
[Mon Mar 11 14:29:16 2024] R10: 000000000000ea60 R11: 0000000000000246 R12: 000055f87dd58980
[Mon Mar 11 14:29:16 2024] R13: 000000000000ea60 R14: 0000000000000027 R15: 000055f87dd5b760
[Mon Mar 11 14:29:16 2024] </TASK>
[Mon Mar 11 14:29:16 2024] task:smbd[10.87.28.1 state:S stack:0 pid:2457400 ppid:4757 flags:0x00000002
[Mon Mar 11 14:29:16 2024] Call Trace:
[Mon Mar 11 14:29:16 2024] <TASK>
[Mon Mar 11 14:29:16 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:16 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:16 2024] schedule_hrtimeout_range_clock+0x95/0x120
[Mon Mar 11 14:29:16 2024] ? __pfx_hrtimer_wakeup+0x10/0x10
[Mon Mar 11 14:29:16 2024] ep_poll+0x348/0x3b0
[Mon Mar 11 14:29:16 2024] ? __pfx_autoremove_wake_function+0x10/0x10
[Mon Mar 11 14:29:16 2024] do_epoll_wait+0xb1/0xd0
[Mon Mar 11 14:29:16 2024] __x64_sys_epoll_wait+0x60/0x100
[Mon Mar 11 14:29:16 2024] do_syscall_64+0x59/0x90
[Mon Mar 11 14:29:16 2024] ? audit_reset_context.part.0.constprop.0+0x273/0x2e0
[Mon Mar 11 14:29:16 2024] ? syscall_exit_work+0x103/0x130
[Mon Mar 11 14:29:16 2024] ? syscall_exit_to_user_mode+0x22/0x40
[Mon Mar 11 14:29:16 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:16 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:16 2024] ? exc_page_fault+0x62/0x150
[Mon Mar 11 14:29:16 2024] entry_SYSCALL_64_after_hwframe+0x72/0xdc
[Mon Mar 11 14:29:16 2024] RIP: 0033:0x7fedb7d4e80a
[Mon Mar 11 14:29:16 2024] RSP: 002b:00007ffc284a9ad8 EFLAGS: 00000246 ORIG_RAX: 00000000000000e8
[Mon Mar 11 14:29:16 2024] RAX: ffffffffffffffda RBX: 000055f87dd397c0 RCX: 00007fedb7d4e80a
[Mon Mar 11 14:29:16 2024] RDX: 0000000000000001 RSI: 00007ffc284a9b0c RDI: 0000000000000005
[Mon Mar 11 14:29:16 2024] RBP: 000055f87dd2aa20 R08: 00000000000f41c4 R09: 0000000000000000
[Mon Mar 11 14:29:16 2024] R10: 000000000000ea60 R11: 0000000000000246 R12: 000055f87dd58980
[Mon Mar 11 14:29:16 2024] R13: 000000000000ea60 R14: 0000000000000027 R15: 000055f87dd5b760
[Mon Mar 11 14:29:16 2024] </TASK>
[Mon Mar 11 14:29:16 2024] task:smbd[10.87.28.1 state:S stack:0 pid:2457742 ppid:4757 flags:0x00000002
[Mon Mar 11 14:29:16 2024] Call Trace:
[Mon Mar 11 14:29:16 2024] <TASK>
[Mon Mar 11 14:29:16 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:16 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:16 2024] schedule_hrtimeout_range_clock+0x95/0x120
[Mon Mar 11 14:29:16 2024] ? __pfx_hrtimer_wakeup+0x10/0x10
[Mon Mar 11 14:29:16 2024] ep_poll+0x348/0x3b0
[Mon Mar 11 14:29:16 2024] ? __pfx_autoremove_wake_function+0x10/0x10
[Mon Mar 11 14:29:16 2024] do_epoll_wait+0xb1/0xd0
[Mon Mar 11 14:29:16 2024] __x64_sys_epoll_wait+0x60/0x100
[Mon Mar 11 14:29:16 2024] do_syscall_64+0x59/0x90
[Mon Mar 11 14:29:16 2024] ? syscall_exit_work+0x103/0x130
[Mon Mar 11 14:29:16 2024] ? syscall_exit_to_user_mode+0x22/0x40
[Mon Mar 11 14:29:16 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:16 2024] entry_SYSCALL_64_after_hwframe+0x72/0xdc
[Mon Mar 11 14:29:16 2024] RIP: 0033:0x7fedb7d4e80a
[Mon Mar 11 14:29:16 2024] RSP: 002b:00007ffc284a9ad8 EFLAGS: 00000246 ORIG_RAX: 00000000000000e8
[Mon Mar 11 14:29:16 2024] RAX: ffffffffffffffda RBX: 000055f87dd397c0 RCX: 00007fedb7d4e80a
[Mon Mar 11 14:29:16 2024] RDX: 0000000000000001 RSI: 00007ffc284a9b0c RDI: 0000000000000005
[Mon Mar 11 14:29:16 2024] RBP: 000055f87dd2aa20 R08: 00000000000f4067 R09: 0000000000000000
[Mon Mar 11 14:29:16 2024] R10: 000000000000ea60 R11: 0000000000000246 R12: 000055f87dd61190
[Mon Mar 11 14:29:16 2024] R13: 000000000000ea60 R14: 0000000000000026 R15: 000055f87dd5b760
[Mon Mar 11 14:29:16 2024] </TASK>
[Mon Mar 11 14:29:16 2024] task:smbd[10.87.29.9 state:S stack:0 pid:2458024 ppid:4757 flags:0x00000002
[Mon Mar 11 14:29:16 2024] Call Trace:
[Mon Mar 11 14:29:16 2024] <TASK>
[Mon Mar 11 14:29:16 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:16 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:16 2024] schedule_hrtimeout_range_clock+0x95/0x120
[Mon Mar 11 14:29:16 2024] ? __pfx_hrtimer_wakeup+0x10/0x10
[Mon Mar 11 14:29:16 2024] ep_poll+0x348/0x3b0
[Mon Mar 11 14:29:16 2024] ? __pfx_autoremove_wake_function+0x10/0x10
[Mon Mar 11 14:29:16 2024] do_epoll_wait+0xb1/0xd0
[Mon Mar 11 14:29:16 2024] __x64_sys_epoll_wait+0x60/0x100
[Mon Mar 11 14:29:16 2024] do_syscall_64+0x59/0x90
[Mon Mar 11 14:29:16 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:16 2024] ? syscall_exit_to_user_mode+0x22/0x40
[Mon Mar 11 14:29:16 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:16 2024] ? syscall_exit_work+0x103/0x130
[Mon Mar 11 14:29:16 2024] ? syscall_exit_to_user_mode+0x22/0x40
[Mon Mar 11 14:29:16 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:16 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:16 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:16 2024] entry_SYSCALL_64_after_hwframe+0x72/0xdc
[Mon Mar 11 14:29:16 2024] RIP: 0033:0x7fedb7d4e80a
[Mon Mar 11 14:29:16 2024] RSP: 002b:00007ffc284a9ad8 EFLAGS: 00000246 ORIG_RAX: 00000000000000e8
[Mon Mar 11 14:29:16 2024] RAX: ffffffffffffffda RBX: 000055f87dd397c0 RCX: 00007fedb7d4e80a
[Mon Mar 11 14:29:16 2024] RDX: 0000000000000001 RSI: 00007ffc284a9b0c RDI: 0000000000000005
[Mon Mar 11 14:29:16 2024] RBP: 000055f87dd2aa20 R08: 00000000000b8d4d R09: 0000000000000000
[Mon Mar 11 14:29:16 2024] R10: 000000000000e586 R11: 0000000000000246 R12: 000055f87dd61190
[Mon Mar 11 14:29:16 2024] R13: 000000000000e586 R14: 0000000000000026 R15: 000055f87dd5b760
[Mon Mar 11 14:29:16 2024] </TASK>
[Mon Mar 11 14:29:16 2024] task:kworker/22:2 state:I stack:0 pid:2458065 ppid:2 flags:0x00004000
[Mon Mar 11 14:29:16 2024] Workqueue: 0x0 (mm_percpu_wq)
[Mon Mar 11 14:29:16 2024] Call Trace:
[Mon Mar 11 14:29:16 2024] <TASK>
[Mon Mar 11 14:29:16 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:16 2024] ? __pfx_worker_thread+0x10/0x10
[Mon Mar 11 14:29:16 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:16 2024] worker_thread+0xbb/0x3a0
[Mon Mar 11 14:29:16 2024] ? __pfx_worker_thread+0x10/0x10
[Mon Mar 11 14:29:16 2024] kthread+0xdd/0x100
[Mon Mar 11 14:29:16 2024] ? __pfx_kthread+0x10/0x10
[Mon Mar 11 14:29:16 2024] ret_from_fork+0x29/0x50
[Mon Mar 11 14:29:16 2024] </TASK>
[Mon Mar 11 14:29:16 2024] task:smbd[10.87.29.8 state:S stack:0 pid:2459178 ppid:4757 flags:0x00000002
[Mon Mar 11 14:29:16 2024] Call Trace:
[Mon Mar 11 14:29:16 2024] <TASK>
[Mon Mar 11 14:29:16 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:16 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:16 2024] schedule_hrtimeout_range_clock+0x95/0x120
[Mon Mar 11 14:29:16 2024] ? __pfx_hrtimer_wakeup+0x10/0x10
[Mon Mar 11 14:29:16 2024] ep_poll+0x348/0x3b0
[Mon Mar 11 14:29:16 2024] ? __pfx_autoremove_wake_function+0x10/0x10
[Mon Mar 11 14:29:16 2024] do_epoll_wait+0xb1/0xd0
[Mon Mar 11 14:29:16 2024] __x64_sys_epoll_wait+0x60/0x100
[Mon Mar 11 14:29:16 2024] do_syscall_64+0x59/0x90
[Mon Mar 11 14:29:16 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:16 2024] entry_SYSCALL_64_after_hwframe+0x72/0xdc
[Mon Mar 11 14:29:16 2024] RIP: 0033:0x7fedb7d4e84e
[Mon Mar 11 14:29:16 2024] RSP: 002b:00007ffc284a9ab0 EFLAGS: 00000293 ORIG_RAX: 00000000000000e8
[Mon Mar 11 14:29:16 2024] RAX: ffffffffffffffda RBX: 000055f87dd397c0 RCX: 00007fedb7d4e84e
[Mon Mar 11 14:29:16 2024] RDX: 0000000000000001 RSI: 00007ffc284a9b0c RDI: 0000000000000005
[Mon Mar 11 14:29:16 2024] RBP: 000055f87dd2aa20 R08: 0000000000000000 R09: 0000000000000000
[Mon Mar 11 14:29:16 2024] R10: 000000000000ea60 R11: 0000000000000293 R12: 000055f87dd61190
[Mon Mar 11 14:29:16 2024] R13: 000000000000ea60 R14: 0000000000000026 R15: 000055f87dd5b760
[Mon Mar 11 14:29:16 2024] </TASK>
[Mon Mar 11 14:29:16 2024] task:smbd[10.40.136. state:S stack:0 pid:2459316 ppid:4757 flags:0x00000002
[Mon Mar 11 14:29:16 2024] Call Trace:
[Mon Mar 11 14:29:16 2024] <TASK>
[Mon Mar 11 14:29:16 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:16 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:16 2024] schedule_hrtimeout_range_clock+0x95/0x120
[Mon Mar 11 14:29:16 2024] ? __pfx_hrtimer_wakeup+0x10/0x10
[Mon Mar 11 14:29:16 2024] ep_poll+0x348/0x3b0
[Mon Mar 11 14:29:16 2024] ? __pfx_autoremove_wake_function+0x10/0x10
[Mon Mar 11 14:29:16 2024] do_epoll_wait+0xb1/0xd0
[Mon Mar 11 14:29:16 2024] __x64_sys_epoll_wait+0x60/0x100
[Mon Mar 11 14:29:16 2024] do_syscall_64+0x59/0x90
[Mon Mar 11 14:29:16 2024] ? __rseq_handle_notify_resume+0x26/0xb0
[Mon Mar 11 14:29:16 2024] ? exit_to_user_mode_loop+0xd0/0x130
[Mon Mar 11 14:29:16 2024] ? exit_to_user_mode_prepare+0xec/0x100
[Mon Mar 11 14:29:16 2024] ? syscall_exit_to_user_mode+0x22/0x40
[Mon Mar 11 14:29:16 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:16 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:16 2024] ? syscall_exit_to_user_mode+0x22/0x40
[Mon Mar 11 14:29:16 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:16 2024] entry_SYSCALL_64_after_hwframe+0x72/0xdc
[Mon Mar 11 14:29:16 2024] RIP: 0033:0x7fedb7d4e80a
[Mon Mar 11 14:29:16 2024] RSP: 002b:00007ffc284a9ad8 EFLAGS: 00000246 ORIG_RAX: 00000000000000e8
[Mon Mar 11 14:29:16 2024] RAX: ffffffffffffffda RBX: 000055f87dd397c0 RCX: 00007fedb7d4e80a
[Mon Mar 11 14:29:16 2024] RDX: 0000000000000001 RSI: 00007ffc284a9b0c RDI: 0000000000000005
[Mon Mar 11 14:29:16 2024] RBP: 000055f87dd2aa20 R08: 00000000000f41ad R09: 0000000000000000
[Mon Mar 11 14:29:16 2024] R10: 000000000000ea60 R11: 0000000000000246 R12: 000055f87dd61190
[Mon Mar 11 14:29:16 2024] R13: 000000000000ea60 R14: 0000000000000026 R15: 000055f87dd5b760
[Mon Mar 11 14:29:16 2024] </TASK>
[Mon Mar 11 14:29:16 2024] task:smbd[10.87.28.1 state:S stack:0 pid:2461553 ppid:4757 flags:0x00000002
[Mon Mar 11 14:29:16 2024] Call Trace:
[Mon Mar 11 14:29:16 2024] <TASK>
[Mon Mar 11 14:29:16 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:16 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:16 2024] schedule_hrtimeout_range_clock+0x95/0x120
[Mon Mar 11 14:29:16 2024] ? __pfx_hrtimer_wakeup+0x10/0x10
[Mon Mar 11 14:29:16 2024] ep_poll+0x348/0x3b0
[Mon Mar 11 14:29:16 2024] ? __pfx_autoremove_wake_function+0x10/0x10
[Mon Mar 11 14:29:16 2024] do_epoll_wait+0xb1/0xd0
[Mon Mar 11 14:29:16 2024] __x64_sys_epoll_wait+0x60/0x100
[Mon Mar 11 14:29:16 2024] do_syscall_64+0x59/0x90
[Mon Mar 11 14:29:16 2024] ? syscall_exit_to_user_mode+0x22/0x40
[Mon Mar 11 14:29:16 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:16 2024] ? syscall_exit_to_user_mode+0x22/0x40
[Mon Mar 11 14:29:16 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:16 2024] ? syscall_exit_work+0x103/0x130
[Mon Mar 11 14:29:16 2024] ? syscall_exit_to_user_mode+0x22/0x40
[Mon Mar 11 14:29:16 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:16 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:16 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:16 2024] ? __irq_exit_rcu+0x46/0xc0
[Mon Mar 11 14:29:16 2024] ? common_interrupt+0x43/0xa0
[Mon Mar 11 14:29:16 2024] entry_SYSCALL_64_after_hwframe+0x72/0xdc
[Mon Mar 11 14:29:16 2024] RIP: 0033:0x7fedb7d4e84e
[Mon Mar 11 14:29:16 2024] RSP: 002b:00007ffc284a9ab0 EFLAGS: 00000293 ORIG_RAX: 00000000000000e8
[Mon Mar 11 14:29:16 2024] RAX: ffffffffffffffda RBX: 000055f87dd397c0 RCX: 00007fedb7d4e84e
[Mon Mar 11 14:29:16 2024] RDX: 0000000000000001 RSI: 00007ffc284a9b0c RDI: 0000000000000005
[Mon Mar 11 14:29:16 2024] RBP: 000055f87dd2aa20 R08: 0000000000000000 R09: 0000000000000000
[Mon Mar 11 14:29:16 2024] R10: 0000000000006126 R11: 0000000000000293 R12: 000055f87dd61190
[Mon Mar 11 14:29:16 2024] R13: 0000000000006126 R14: 0000000000000026 R15: 000055f87dd5b760
[Mon Mar 11 14:29:16 2024] </TASK>
[Mon Mar 11 14:29:16 2024] task:kworker/u98:3 state:I stack:0 pid:2461839 ppid:2 flags:0x00004000
[Mon Mar 11 14:29:16 2024] Workqueue: 0x0 (flush-253:79)
[Mon Mar 11 14:29:16 2024] Call Trace:
[Mon Mar 11 14:29:16 2024] <TASK>
[Mon Mar 11 14:29:16 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:16 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:16 2024] worker_thread+0xbb/0x3a0
[Mon Mar 11 14:29:16 2024] ? __pfx_worker_thread+0x10/0x10
[Mon Mar 11 14:29:16 2024] kthread+0xdd/0x100
[Mon Mar 11 14:29:16 2024] ? __pfx_kthread+0x10/0x10
[Mon Mar 11 14:29:16 2024] ret_from_fork+0x29/0x50
[Mon Mar 11 14:29:16 2024] </TASK>
[Mon Mar 11 14:29:16 2024] task:smbd[10.87.20.1 state:S stack:0 pid:2462719 ppid:4757 flags:0x00000002
[Mon Mar 11 14:29:16 2024] Call Trace:
[Mon Mar 11 14:29:16 2024] <TASK>
[Mon Mar 11 14:29:16 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:16 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:16 2024] schedule_hrtimeout_range_clock+0x95/0x120
[Mon Mar 11 14:29:16 2024] ? __pfx_hrtimer_wakeup+0x10/0x10
[Mon Mar 11 14:29:16 2024] ep_poll+0x348/0x3b0
[Mon Mar 11 14:29:16 2024] ? __pfx_autoremove_wake_function+0x10/0x10
[Mon Mar 11 14:29:16 2024] do_epoll_wait+0xb1/0xd0
[Mon Mar 11 14:29:16 2024] __x64_sys_epoll_wait+0x60/0x100
[Mon Mar 11 14:29:16 2024] do_syscall_64+0x59/0x90
[Mon Mar 11 14:29:16 2024] ? audit_reset_context.part.0.constprop.0+0x273/0x2e0
[Mon Mar 11 14:29:16 2024] ? syscall_exit_work+0x103/0x130
[Mon Mar 11 14:29:16 2024] ? syscall_exit_to_user_mode+0x22/0x40
[Mon Mar 11 14:29:16 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:16 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:16 2024] entry_SYSCALL_64_after_hwframe+0x72/0xdc
[Mon Mar 11 14:29:16 2024] RIP: 0033:0x7fedb7d4e84e
[Mon Mar 11 14:29:16 2024] RSP: 002b:00007ffc284a9ab0 EFLAGS: 00000293 ORIG_RAX: 00000000000000e8
[Mon Mar 11 14:29:16 2024] RAX: ffffffffffffffda RBX: 000055f87dd397c0 RCX: 00007fedb7d4e84e
[Mon Mar 11 14:29:16 2024] RDX: 0000000000000001 RSI: 00007ffc284a9b0c RDI: 0000000000000005
[Mon Mar 11 14:29:16 2024] RBP: 000055f87dd2aa20 R08: 0000000000000000 R09: 0000000000000000
[Mon Mar 11 14:29:16 2024] R10: 000000000000ea60 R11: 0000000000000293 R12: 000055f87dd61190
[Mon Mar 11 14:29:16 2024] R13: 000000000000ea60 R14: 0000000000000026 R15: 000055f87dd5b760
[Mon Mar 11 14:29:16 2024] </TASK>
[Mon Mar 11 14:29:16 2024] task:kworker/u97:3 state:I stack:0 pid:2462779 ppid:2 flags:0x00004000
[Mon Mar 11 14:29:16 2024] Workqueue: 0x0 (flush-253:77)
[Mon Mar 11 14:29:16 2024] Call Trace:
[Mon Mar 11 14:29:16 2024] <TASK>
[Mon Mar 11 14:29:16 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:16 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:16 2024] worker_thread+0xbb/0x3a0
[Mon Mar 11 14:29:16 2024] ? __pfx_worker_thread+0x10/0x10
[Mon Mar 11 14:29:16 2024] kthread+0xdd/0x100
[Mon Mar 11 14:29:16 2024] ? __pfx_kthread+0x10/0x10
[Mon Mar 11 14:29:16 2024] ret_from_fork+0x29/0x50
[Mon Mar 11 14:29:16 2024] </TASK>
[Mon Mar 11 14:29:16 2024] task:kworker/u98:7 state:I stack:0 pid:2463319 ppid:2 flags:0x00004000
[Mon Mar 11 14:29:16 2024] Workqueue: 0x0 (events_unbound)
[Mon Mar 11 14:29:16 2024] Call Trace:
[Mon Mar 11 14:29:16 2024] <TASK>
[Mon Mar 11 14:29:16 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:16 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:16 2024] worker_thread+0xbb/0x3a0
[Mon Mar 11 14:29:16 2024] ? __pfx_worker_thread+0x10/0x10
[Mon Mar 11 14:29:16 2024] kthread+0xdd/0x100
[Mon Mar 11 14:29:16 2024] ? __pfx_kthread+0x10/0x10
[Mon Mar 11 14:29:16 2024] ret_from_fork+0x29/0x50
[Mon Mar 11 14:29:16 2024] </TASK>
[Mon Mar 11 14:29:16 2024] task:kworker/u98:10 state:I stack:0 pid:2463433 ppid:2 flags:0x00004000
[Mon Mar 11 14:29:16 2024] Workqueue: 0x0 (writeback)
[Mon Mar 11 14:29:16 2024] Call Trace:
[Mon Mar 11 14:29:16 2024] <TASK>
[Mon Mar 11 14:29:16 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:16 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:16 2024] worker_thread+0xbb/0x3a0
[Mon Mar 11 14:29:16 2024] ? __pfx_worker_thread+0x10/0x10
[Mon Mar 11 14:29:16 2024] kthread+0xdd/0x100
[Mon Mar 11 14:29:16 2024] ? __pfx_kthread+0x10/0x10
[Mon Mar 11 14:29:16 2024] ret_from_fork+0x29/0x50
[Mon Mar 11 14:29:16 2024] </TASK>
[Mon Mar 11 14:29:16 2024] task:smbd[10.33.88.9 state:S stack:0 pid:2464105 ppid:4757 flags:0x00000002
[Mon Mar 11 14:29:16 2024] Call Trace:
[Mon Mar 11 14:29:16 2024] <TASK>
[Mon Mar 11 14:29:16 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:16 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:16 2024] schedule_hrtimeout_range_clock+0x95/0x120
[Mon Mar 11 14:29:16 2024] ? __pfx_hrtimer_wakeup+0x10/0x10
[Mon Mar 11 14:29:16 2024] ep_poll+0x348/0x3b0
[Mon Mar 11 14:29:16 2024] ? __pfx_autoremove_wake_function+0x10/0x10
[Mon Mar 11 14:29:16 2024] do_epoll_wait+0xb1/0xd0
[Mon Mar 11 14:29:16 2024] __x64_sys_epoll_wait+0x60/0x100
[Mon Mar 11 14:29:16 2024] do_syscall_64+0x59/0x90
[Mon Mar 11 14:29:16 2024] ? __irq_exit_rcu+0x46/0xc0
[Mon Mar 11 14:29:16 2024] ? common_interrupt+0x43/0xa0
[Mon Mar 11 14:29:16 2024] entry_SYSCALL_64_after_hwframe+0x72/0xdc
[Mon Mar 11 14:29:16 2024] RIP: 0033:0x7fedb7d4e84e
[Mon Mar 11 14:29:16 2024] RSP: 002b:00007ffc284a9ab0 EFLAGS: 00000293 ORIG_RAX: 00000000000000e8
[Mon Mar 11 14:29:16 2024] RAX: ffffffffffffffda RBX: 000055f87dd397c0 RCX: 00007fedb7d4e84e
[Mon Mar 11 14:29:16 2024] RDX: 0000000000000001 RSI: 00007ffc284a9b0c RDI: 0000000000000005
[Mon Mar 11 14:29:16 2024] RBP: 000055f87dd2aa20 R08: 0000000000000000 R09: 0000000000000000
[Mon Mar 11 14:29:16 2024] R10: 000000000000926f R11: 0000000000000293 R12: 000055f87dd61190
[Mon Mar 11 14:29:16 2024] R13: 000000000000926f R14: 0000000000000026 R15: 000055f87dd5b760
[Mon Mar 11 14:29:16 2024] </TASK>
[Mon Mar 11 14:29:16 2024] task:kworker/u98:12 state:I stack:0 pid:2464235 ppid:2 flags:0x00004000
[Mon Mar 11 14:29:16 2024] Workqueue: 0x0 (flush-253:0)
[Mon Mar 11 14:29:16 2024] Call Trace:
[Mon Mar 11 14:29:16 2024] <TASK>
[Mon Mar 11 14:29:16 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:16 2024] ? __pfx_worker_thread+0x10/0x10
[Mon Mar 11 14:29:16 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:16 2024] worker_thread+0xbb/0x3a0
[Mon Mar 11 14:29:16 2024] ? __pfx_worker_thread+0x10/0x10
[Mon Mar 11 14:29:16 2024] kthread+0xdd/0x100
[Mon Mar 11 14:29:16 2024] ? __pfx_kthread+0x10/0x10
[Mon Mar 11 14:29:16 2024] ret_from_fork+0x29/0x50
[Mon Mar 11 14:29:16 2024] </TASK>
[Mon Mar 11 14:29:16 2024] task:smbd[10.46.159. state:S stack:0 pid:2465617 ppid:4757 flags:0x00000002
[Mon Mar 11 14:29:16 2024] Call Trace:
[Mon Mar 11 14:29:16 2024] <TASK>
[Mon Mar 11 14:29:16 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:16 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:16 2024] schedule_hrtimeout_range_clock+0x95/0x120
[Mon Mar 11 14:29:16 2024] ? __pfx_hrtimer_wakeup+0x10/0x10
[Mon Mar 11 14:29:16 2024] ep_poll+0x348/0x3b0
[Mon Mar 11 14:29:16 2024] ? __pfx_autoremove_wake_function+0x10/0x10
[Mon Mar 11 14:29:16 2024] do_epoll_wait+0xb1/0xd0
[Mon Mar 11 14:29:16 2024] __x64_sys_epoll_wait+0x60/0x100
[Mon Mar 11 14:29:16 2024] do_syscall_64+0x59/0x90
[Mon Mar 11 14:29:16 2024] ? audit_reset_context.part.0.constprop.0+0x273/0x2e0
[Mon Mar 11 14:29:16 2024] ? syscall_exit_work+0x103/0x130
[Mon Mar 11 14:29:16 2024] ? syscall_exit_to_user_mode+0x22/0x40
[Mon Mar 11 14:29:16 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:16 2024] ? syscall_exit_to_user_mode+0x22/0x40
[Mon Mar 11 14:29:16 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:16 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:16 2024] ? sysvec_apic_timer_interrupt+0x3c/0x90
[Mon Mar 11 14:29:16 2024] entry_SYSCALL_64_after_hwframe+0x72/0xdc
[Mon Mar 11 14:29:16 2024] RIP: 0033:0x7fedb7d4e80a
[Mon Mar 11 14:29:16 2024] RSP: 002b:00007ffc284a9ad8 EFLAGS: 00000246 ORIG_RAX: 00000000000000e8
[Mon Mar 11 14:29:16 2024] RAX: ffffffffffffffda RBX: 000055f87dd397c0 RCX: 00007fedb7d4e80a
[Mon Mar 11 14:29:16 2024] RDX: 0000000000000001 RSI: 00007ffc284a9b0c RDI: 0000000000000005
[Mon Mar 11 14:29:16 2024] RBP: 000055f87dd2aa20 R08: 00000000000f41c4 R09: 0000000000000000
[Mon Mar 11 14:29:16 2024] R10: 000000000000ea60 R11: 0000000000000246 R12: 000055f87dd61190
[Mon Mar 11 14:29:16 2024] R13: 000000000000ea60 R14: 0000000000000026 R15: 000055f87dd5b760
[Mon Mar 11 14:29:16 2024] </TASK>
[Mon Mar 11 14:29:16 2024] task:kworker/u96:1 state:I stack:0 pid:2466032 ppid:2 flags:0x00004000
[Mon Mar 11 14:29:16 2024] Workqueue: 0x0 (ext4-rsv-conversion)
[Mon Mar 11 14:29:16 2024] Call Trace:
[Mon Mar 11 14:29:16 2024] <TASK>
[Mon Mar 11 14:29:16 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:16 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:16 2024] worker_thread+0xbb/0x3a0
[Mon Mar 11 14:29:16 2024] ? __pfx_worker_thread+0x10/0x10
[Mon Mar 11 14:29:16 2024] kthread+0xdd/0x100
[Mon Mar 11 14:29:16 2024] ? __pfx_kthread+0x10/0x10
[Mon Mar 11 14:29:16 2024] ret_from_fork+0x29/0x50
[Mon Mar 11 14:29:16 2024] </TASK>
[Mon Mar 11 14:29:16 2024] task:kworker/u96:5 state:I stack:0 pid:2466033 ppid:2 flags:0x00004000
[Mon Mar 11 14:29:16 2024] Workqueue: 0x0 (bond0)
[Mon Mar 11 14:29:16 2024] Call Trace:
[Mon Mar 11 14:29:16 2024] <TASK>
[Mon Mar 11 14:29:16 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:16 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:16 2024] worker_thread+0xbb/0x3a0
[Mon Mar 11 14:29:16 2024] ? __pfx_worker_thread+0x10/0x10
[Mon Mar 11 14:29:16 2024] kthread+0xdd/0x100
[Mon Mar 11 14:29:16 2024] ? __pfx_kthread+0x10/0x10
[Mon Mar 11 14:29:16 2024] ret_from_fork+0x29/0x50
[Mon Mar 11 14:29:16 2024] </TASK>
[Mon Mar 11 14:29:16 2024] task:smbd[10.87.30.8 state:S stack:0 pid:2468009 ppid:4757 flags:0x00000002
[Mon Mar 11 14:29:16 2024] Call Trace:
[Mon Mar 11 14:29:16 2024] <TASK>
[Mon Mar 11 14:29:16 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:16 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:16 2024] schedule_hrtimeout_range_clock+0x95/0x120
[Mon Mar 11 14:29:16 2024] ? __pfx_hrtimer_wakeup+0x10/0x10
[Mon Mar 11 14:29:16 2024] ep_poll+0x348/0x3b0
[Mon Mar 11 14:29:16 2024] ? __pfx_autoremove_wake_function+0x10/0x10
[Mon Mar 11 14:29:16 2024] do_epoll_wait+0xb1/0xd0
[Mon Mar 11 14:29:16 2024] __x64_sys_epoll_wait+0x60/0x100
[Mon Mar 11 14:29:16 2024] do_syscall_64+0x59/0x90
[Mon Mar 11 14:29:16 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:16 2024] ? audit_reset_context.part.0.constprop.0+0x273/0x2e0
[Mon Mar 11 14:29:16 2024] ? syscall_exit_work+0x103/0x130
[Mon Mar 11 14:29:16 2024] ? syscall_exit_to_user_mode+0x22/0x40
[Mon Mar 11 14:29:16 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:16 2024] entry_SYSCALL_64_after_hwframe+0x72/0xdc
[Mon Mar 11 14:29:16 2024] RIP: 0033:0x7fedb7d4e84e
[Mon Mar 11 14:29:16 2024] RSP: 002b:00007ffc284a9ab0 EFLAGS: 00000293 ORIG_RAX: 00000000000000e8
[Mon Mar 11 14:29:16 2024] RAX: ffffffffffffffda RBX: 000055f87dd397c0 RCX: 00007fedb7d4e84e
[Mon Mar 11 14:29:16 2024] RDX: 0000000000000001 RSI: 00007ffc284a9b0c RDI: 0000000000000005
[Mon Mar 11 14:29:16 2024] RBP: 000055f87dd2aa20 R08: 0000000000000000 R09: 0000000000000000
[Mon Mar 11 14:29:16 2024] R10: 000000000000ea60 R11: 0000000000000293 R12: 000055f87dd61190
[Mon Mar 11 14:29:16 2024] R13: 000000000000ea60 R14: 0000000000000026 R15: 000055f87dd5b760
[Mon Mar 11 14:29:16 2024] </TASK>
[Mon Mar 11 14:29:16 2024] task:kworker/u98:17 state:D stack:0 pid:2468089 ppid:2 flags:0x00004000
[Mon Mar 11 14:29:16 2024] Workqueue: nfsd4 laundromat_main [nfsd]
[Mon Mar 11 14:29:16 2024] Call Trace:
[Mon Mar 11 14:29:16 2024] <TASK>
[Mon Mar 11 14:29:16 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:16 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:16 2024] schedule_timeout+0x11f/0x160
[Mon Mar 11 14:29:16 2024] ? select_idle_sibling+0x28/0x430
[Mon Mar 11 14:29:16 2024] ? wake_affine+0x62/0x1f0
[Mon Mar 11 14:29:16 2024] __wait_for_common+0x90/0x1d0
[Mon Mar 11 14:29:16 2024] ? __pfx_schedule_timeout+0x10/0x10
[Mon Mar 11 14:29:16 2024] __flush_workqueue+0x13a/0x3f0
[Mon Mar 11 14:29:16 2024] nfsd4_shutdown_callback+0x49/0x120 [nfsd]
[Mon Mar 11 14:29:16 2024] ? nfsd4_return_all_client_layouts+0xc4/0xf0 [nfsd]
[Mon Mar 11 14:29:16 2024] ? nfsd4_shutdown_copy+0x68/0xc0 [nfsd]
[Mon Mar 11 14:29:16 2024] __destroy_client+0x1f3/0x290 [nfsd]
[Mon Mar 11 14:29:16 2024] nfs4_process_client_reaplist+0xa1/0x110 [nfsd]
[Mon Mar 11 14:29:16 2024] nfs4_laundromat+0x126/0x6e0 [nfsd]
[Mon Mar 11 14:29:16 2024] laundromat_main+0x16/0x40 [nfsd]
[Mon Mar 11 14:29:16 2024] process_one_work+0x1e2/0x3b0
[Mon Mar 11 14:29:16 2024] ? __pfx_worker_thread+0x10/0x10
[Mon Mar 11 14:29:16 2024] worker_thread+0x50/0x3a0
[Mon Mar 11 14:29:16 2024] ? __pfx_worker_thread+0x10/0x10
[Mon Mar 11 14:29:16 2024] kthread+0xdd/0x100
[Mon Mar 11 14:29:16 2024] ? __pfx_kthread+0x10/0x10
[Mon Mar 11 14:29:16 2024] ret_from_fork+0x29/0x50
[Mon Mar 11 14:29:16 2024] </TASK>
[Mon Mar 11 14:29:16 2024] task:kworker/23:0 state:I stack:0 pid:2469472 ppid:2 flags:0x00004000
[Mon Mar 11 14:29:16 2024] Workqueue: 0x0 (mm_percpu_wq)
[Mon Mar 11 14:29:16 2024] Call Trace:
[Mon Mar 11 14:29:16 2024] <TASK>
[Mon Mar 11 14:29:16 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:16 2024] ? __pfx_worker_thread+0x10/0x10
[Mon Mar 11 14:29:16 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:16 2024] worker_thread+0xbb/0x3a0
[Mon Mar 11 14:29:16 2024] ? __pfx_worker_thread+0x10/0x10
[Mon Mar 11 14:29:16 2024] kthread+0xdd/0x100
[Mon Mar 11 14:29:16 2024] ? __pfx_kthread+0x10/0x10
[Mon Mar 11 14:29:16 2024] ret_from_fork+0x29/0x50
[Mon Mar 11 14:29:16 2024] </TASK>
[Mon Mar 11 14:29:16 2024] task:smbd[10.87.19.5 state:S stack:0 pid:2470233 ppid:4757 flags:0x00000002
[Mon Mar 11 14:29:16 2024] Call Trace:
[Mon Mar 11 14:29:16 2024] <TASK>
[Mon Mar 11 14:29:16 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:16 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:16 2024] schedule_hrtimeout_range_clock+0x95/0x120
[Mon Mar 11 14:29:16 2024] ? __pfx_hrtimer_wakeup+0x10/0x10
[Mon Mar 11 14:29:16 2024] ep_poll+0x348/0x3b0
[Mon Mar 11 14:29:16 2024] ? __pfx_autoremove_wake_function+0x10/0x10
[Mon Mar 11 14:29:16 2024] do_epoll_wait+0xb1/0xd0
[Mon Mar 11 14:29:16 2024] __x64_sys_epoll_wait+0x60/0x100
[Mon Mar 11 14:29:16 2024] do_syscall_64+0x59/0x90
[Mon Mar 11 14:29:16 2024] ? exc_page_fault+0x62/0x150
[Mon Mar 11 14:29:16 2024] entry_SYSCALL_64_after_hwframe+0x72/0xdc
[Mon Mar 11 14:29:16 2024] RIP: 0033:0x7fedb7d4e84e
[Mon Mar 11 14:29:16 2024] RSP: 002b:00007ffc284a9ab0 EFLAGS: 00000293 ORIG_RAX: 00000000000000e8
[Mon Mar 11 14:29:16 2024] RAX: ffffffffffffffda RBX: 000055f87dd397c0 RCX: 00007fedb7d4e84e
[Mon Mar 11 14:29:16 2024] RDX: 0000000000000001 RSI: 00007ffc284a9b0c RDI: 0000000000000005
[Mon Mar 11 14:29:16 2024] RBP: 000055f87dd2aa20 R08: 0000000000000000 R09: 0000000000000000
[Mon Mar 11 14:29:16 2024] R10: 000000000000d7b4 R11: 0000000000000293 R12: 000055f87dd58980
[Mon Mar 11 14:29:16 2024] R13: 000000000000d7b4 R14: 0000000000000027 R15: 000055f87dd5b760
[Mon Mar 11 14:29:16 2024] </TASK>
[Mon Mar 11 14:29:16 2024] task:kworker/u97:2 state:I stack:0 pid:2470473 ppid:2 flags:0x00004000
[Mon Mar 11 14:29:16 2024] Workqueue: 0x0 (events_unbound)
[Mon Mar 11 14:29:16 2024] Call Trace:
[Mon Mar 11 14:29:16 2024] <TASK>
[Mon Mar 11 14:29:16 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:16 2024] ? __pfx_worker_thread+0x10/0x10
[Mon Mar 11 14:29:16 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:16 2024] worker_thread+0xbb/0x3a0
[Mon Mar 11 14:29:16 2024] ? __pfx_worker_thread+0x10/0x10
[Mon Mar 11 14:29:16 2024] kthread+0xdd/0x100
[Mon Mar 11 14:29:16 2024] ? __pfx_kthread+0x10/0x10
[Mon Mar 11 14:29:16 2024] ret_from_fork+0x29/0x50
[Mon Mar 11 14:29:16 2024] </TASK>
[Mon Mar 11 14:29:16 2024] task:kworker/32:1 state:I stack:0 pid:2470837 ppid:2 flags:0x00004000
[Mon Mar 11 14:29:16 2024] Workqueue: 0x0 (mm_percpu_wq)
[Mon Mar 11 14:29:16 2024] Call Trace:
[Mon Mar 11 14:29:16 2024] <TASK>
[Mon Mar 11 14:29:16 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:16 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:16 2024] worker_thread+0xbb/0x3a0
[Mon Mar 11 14:29:16 2024] ? __pfx_worker_thread+0x10/0x10
[Mon Mar 11 14:29:16 2024] kthread+0xdd/0x100
[Mon Mar 11 14:29:16 2024] ? __pfx_kthread+0x10/0x10
[Mon Mar 11 14:29:16 2024] ret_from_fork+0x29/0x50
[Mon Mar 11 14:29:16 2024] </TASK>
[Mon Mar 11 14:29:16 2024] task:kworker/u96:2 state:I stack:0 pid:2470993 ppid:2 flags:0x00004000
[Mon Mar 11 14:29:16 2024] Workqueue: 0x0 (bond0)
[Mon Mar 11 14:29:16 2024] Call Trace:
[Mon Mar 11 14:29:16 2024] <TASK>
[Mon Mar 11 14:29:16 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:16 2024] ? __pfx_worker_thread+0x10/0x10
[Mon Mar 11 14:29:16 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:16 2024] worker_thread+0xbb/0x3a0
[Mon Mar 11 14:29:16 2024] ? __pfx_worker_thread+0x10/0x10
[Mon Mar 11 14:29:16 2024] kthread+0xdd/0x100
[Mon Mar 11 14:29:16 2024] ? __pfx_kthread+0x10/0x10
[Mon Mar 11 14:29:16 2024] ret_from_fork+0x29/0x50
[Mon Mar 11 14:29:16 2024] </TASK>
[Mon Mar 11 14:29:16 2024] task:kworker/15:2 state:I stack:0 pid:2471252 ppid:2 flags:0x00004000
[Mon Mar 11 14:29:16 2024] Call Trace:
[Mon Mar 11 14:29:16 2024] <TASK>
[Mon Mar 11 14:29:16 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:16 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:16 2024] worker_thread+0xbb/0x3a0
[Mon Mar 11 14:29:16 2024] ? __pfx_worker_thread+0x10/0x10
[Mon Mar 11 14:29:16 2024] kthread+0xdd/0x100
[Mon Mar 11 14:29:16 2024] ? __pfx_kthread+0x10/0x10
[Mon Mar 11 14:29:16 2024] ret_from_fork+0x29/0x50
[Mon Mar 11 14:29:16 2024] </TASK>
[Mon Mar 11 14:29:16 2024] task:kworker/43:1 state:I stack:0 pid:2471690 ppid:2 flags:0x00004000
[Mon Mar 11 14:29:16 2024] Workqueue: 0x0 (events)
[Mon Mar 11 14:29:16 2024] Call Trace:
[Mon Mar 11 14:29:16 2024] <TASK>
[Mon Mar 11 14:29:16 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:16 2024] ? __pfx_worker_thread+0x10/0x10
[Mon Mar 11 14:29:16 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:16 2024] worker_thread+0xbb/0x3a0
[Mon Mar 11 14:29:16 2024] ? __pfx_worker_thread+0x10/0x10
[Mon Mar 11 14:29:16 2024] kthread+0xdd/0x100
[Mon Mar 11 14:29:16 2024] ? __pfx_kthread+0x10/0x10
[Mon Mar 11 14:29:16 2024] ret_from_fork+0x29/0x50
[Mon Mar 11 14:29:16 2024] </TASK>
[Mon Mar 11 14:29:16 2024] task:kworker/u98:0 state:I stack:0 pid:2471807 ppid:2 flags:0x00004000
[Mon Mar 11 14:29:16 2024] Workqueue: 0x0 (flush-253:0)
[Mon Mar 11 14:29:16 2024] Call Trace:
[Mon Mar 11 14:29:16 2024] <TASK>
[Mon Mar 11 14:29:16 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:16 2024] ? __pfx_worker_thread+0x10/0x10
[Mon Mar 11 14:29:16 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:16 2024] worker_thread+0xbb/0x3a0
[Mon Mar 11 14:29:16 2024] ? __pfx_worker_thread+0x10/0x10
[Mon Mar 11 14:29:16 2024] kthread+0xdd/0x100
[Mon Mar 11 14:29:16 2024] ? __pfx_kthread+0x10/0x10
[Mon Mar 11 14:29:16 2024] ret_from_fork+0x29/0x50
[Mon Mar 11 14:29:16 2024] </TASK>
[Mon Mar 11 14:29:16 2024] task:kworker/39:0 state:I stack:0 pid:2471904 ppid:2 flags:0x00004000
[Mon Mar 11 14:29:16 2024] Workqueue: 0x0 (events)
[Mon Mar 11 14:29:16 2024] Call Trace:
[Mon Mar 11 14:29:16 2024] <TASK>
[Mon Mar 11 14:29:16 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:16 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:16 2024] worker_thread+0xbb/0x3a0
[Mon Mar 11 14:29:16 2024] ? __pfx_worker_thread+0x10/0x10
[Mon Mar 11 14:29:16 2024] kthread+0xdd/0x100
[Mon Mar 11 14:29:16 2024] ? __pfx_kthread+0x10/0x10
[Mon Mar 11 14:29:16 2024] ret_from_fork+0x29/0x50
[Mon Mar 11 14:29:16 2024] </TASK>
[Mon Mar 11 14:29:16 2024] task:kworker/44:2 state:I stack:0 pid:2472062 ppid:2 flags:0x00004000
[Mon Mar 11 14:29:16 2024] Workqueue: 0x0 (mm_percpu_wq)
[Mon Mar 11 14:29:16 2024] Call Trace:
[Mon Mar 11 14:29:16 2024] <TASK>
[Mon Mar 11 14:29:16 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:16 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:16 2024] worker_thread+0xbb/0x3a0
[Mon Mar 11 14:29:16 2024] ? __pfx_worker_thread+0x10/0x10
[Mon Mar 11 14:29:16 2024] kthread+0xdd/0x100
[Mon Mar 11 14:29:16 2024] ? __pfx_kthread+0x10/0x10
[Mon Mar 11 14:29:16 2024] ret_from_fork+0x29/0x50
[Mon Mar 11 14:29:16 2024] </TASK>
[Mon Mar 11 14:29:16 2024] task:smbd[10.87.27.1 state:S stack:0 pid:2472086 ppid:4757 flags:0x00000002
[Mon Mar 11 14:29:16 2024] Call Trace:
[Mon Mar 11 14:29:16 2024] <TASK>
[Mon Mar 11 14:29:16 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:16 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:16 2024] schedule_hrtimeout_range_clock+0x95/0x120
[Mon Mar 11 14:29:16 2024] ? __pfx_hrtimer_wakeup+0x10/0x10
[Mon Mar 11 14:29:16 2024] ep_poll+0x348/0x3b0
[Mon Mar 11 14:29:16 2024] ? __pfx_autoremove_wake_function+0x10/0x10
[Mon Mar 11 14:29:16 2024] do_epoll_wait+0xb1/0xd0
[Mon Mar 11 14:29:16 2024] __x64_sys_epoll_wait+0x60/0x100
[Mon Mar 11 14:29:16 2024] do_syscall_64+0x59/0x90
[Mon Mar 11 14:29:16 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:16 2024] ? exc_page_fault+0x62/0x150
[Mon Mar 11 14:29:16 2024] entry_SYSCALL_64_after_hwframe+0x72/0xdc
[Mon Mar 11 14:29:16 2024] RIP: 0033:0x7fedb7d4e80a
[Mon Mar 11 14:29:16 2024] RSP: 002b:00007ffc284a9ad8 EFLAGS: 00000246 ORIG_RAX: 00000000000000e8
[Mon Mar 11 14:29:16 2024] RAX: ffffffffffffffda RBX: 000055f87dd397c0 RCX: 00007fedb7d4e80a
[Mon Mar 11 14:29:16 2024] RDX: 0000000000000001 RSI: 00007ffc284a9b0c RDI: 0000000000000005
[Mon Mar 11 14:29:16 2024] RBP: 000055f87dd2aa20 R08: 00000000000f4037 R09: 0000000000000000
[Mon Mar 11 14:29:16 2024] R10: 000000000000ea60 R11: 0000000000000246 R12: 000055f87dd58980
[Mon Mar 11 14:29:16 2024] R13: 000000000000ea60 R14: 0000000000000027 R15: 000055f87dd5b760
[Mon Mar 11 14:29:16 2024] </TASK>
[Mon Mar 11 14:29:16 2024] task:kworker/46:2 state:I stack:0 pid:2472151 ppid:2 flags:0x00004000
[Mon Mar 11 14:29:16 2024] Call Trace:
[Mon Mar 11 14:29:16 2024] <TASK>
[Mon Mar 11 14:29:16 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:16 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:16 2024] worker_thread+0xbb/0x3a0
[Mon Mar 11 14:29:16 2024] ? __pfx_worker_thread+0x10/0x10
[Mon Mar 11 14:29:16 2024] kthread+0xdd/0x100
[Mon Mar 11 14:29:16 2024] ? __pfx_kthread+0x10/0x10
[Mon Mar 11 14:29:16 2024] ret_from_fork+0x29/0x50
[Mon Mar 11 14:29:16 2024] </TASK>
[Mon Mar 11 14:29:16 2024] task:kworker/36:0 state:I stack:0 pid:2472450 ppid:2 flags:0x00004000
[Mon Mar 11 14:29:16 2024] Workqueue: 0x0 (mm_percpu_wq)
[Mon Mar 11 14:29:16 2024] Call Trace:
[Mon Mar 11 14:29:16 2024] <TASK>
[Mon Mar 11 14:29:16 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:16 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:16 2024] worker_thread+0xbb/0x3a0
[Mon Mar 11 14:29:16 2024] ? __pfx_worker_thread+0x10/0x10
[Mon Mar 11 14:29:16 2024] kthread+0xdd/0x100
[Mon Mar 11 14:29:16 2024] ? __pfx_kthread+0x10/0x10
[Mon Mar 11 14:29:16 2024] ret_from_fork+0x29/0x50
[Mon Mar 11 14:29:16 2024] </TASK>
[Mon Mar 11 14:29:16 2024] task:smbd[10.40.140. state:S stack:0 pid:2472699 ppid:4757 flags:0x00000002
[Mon Mar 11 14:29:16 2024] Call Trace:
[Mon Mar 11 14:29:16 2024] <TASK>
[Mon Mar 11 14:29:16 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:16 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:16 2024] schedule_hrtimeout_range_clock+0x95/0x120
[Mon Mar 11 14:29:16 2024] ? __pfx_hrtimer_wakeup+0x10/0x10
[Mon Mar 11 14:29:16 2024] ep_poll+0x348/0x3b0
[Mon Mar 11 14:29:16 2024] ? __pfx_autoremove_wake_function+0x10/0x10
[Mon Mar 11 14:29:16 2024] do_epoll_wait+0xb1/0xd0
[Mon Mar 11 14:29:16 2024] __x64_sys_epoll_wait+0x60/0x100
[Mon Mar 11 14:29:16 2024] do_syscall_64+0x59/0x90
[Mon Mar 11 14:29:16 2024] ? syscall_exit_work+0x103/0x130
[Mon Mar 11 14:29:16 2024] ? syscall_exit_to_user_mode+0x22/0x40
[Mon Mar 11 14:29:16 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:16 2024] ? common_interrupt+0x43/0xa0
[Mon Mar 11 14:29:16 2024] entry_SYSCALL_64_after_hwframe+0x72/0xdc
[Mon Mar 11 14:29:16 2024] RIP: 0033:0x7fedb7d4e80a
[Mon Mar 11 14:29:16 2024] RSP: 002b:00007ffc284a9ad8 EFLAGS: 00000246 ORIG_RAX: 00000000000000e8
[Mon Mar 11 14:29:16 2024] RAX: ffffffffffffffda RBX: 000055f87dd397c0 RCX: 00007fedb7d4e80a
[Mon Mar 11 14:29:16 2024] RDX: 0000000000000001 RSI: 00007ffc284a9b0c RDI: 0000000000000005
[Mon Mar 11 14:29:16 2024] RBP: 000055f87dd2aa20 R08: 00000000000f41d9 R09: 0000000000000000
[Mon Mar 11 14:29:16 2024] R10: 000000000000ea60 R11: 0000000000000246 R12: 000055f87dd61190
[Mon Mar 11 14:29:16 2024] R13: 000000000000ea60 R14: 0000000000000026 R15: 000055f87dd5b760
[Mon Mar 11 14:29:16 2024] </TASK>
[Mon Mar 11 14:29:16 2024] task:kworker/u96:6 state:I stack:0 pid:2472989 ppid:2 flags:0x00004000
[Mon Mar 11 14:29:16 2024] Workqueue: 0x0 (dm-thin)
[Mon Mar 11 14:29:16 2024] Call Trace:
[Mon Mar 11 14:29:16 2024] <TASK>
[Mon Mar 11 14:29:16 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:16 2024] ? __pfx_worker_thread+0x10/0x10
[Mon Mar 11 14:29:16 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:16 2024] worker_thread+0xbb/0x3a0
[Mon Mar 11 14:29:16 2024] ? __pfx_worker_thread+0x10/0x10
[Mon Mar 11 14:29:16 2024] kthread+0xdd/0x100
[Mon Mar 11 14:29:16 2024] ? __pfx_kthread+0x10/0x10
[Mon Mar 11 14:29:16 2024] ret_from_fork+0x29/0x50
[Mon Mar 11 14:29:16 2024] </TASK>
[Mon Mar 11 14:29:16 2024] task:kworker/12:2 state:I stack:0 pid:2473001 ppid:2 flags:0x00004000
[Mon Mar 11 14:29:16 2024] Workqueue: 0x0 (lpfc_wq)
[Mon Mar 11 14:29:16 2024] Call Trace:
[Mon Mar 11 14:29:16 2024] <TASK>
[Mon Mar 11 14:29:16 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:16 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:16 2024] worker_thread+0xbb/0x3a0
[Mon Mar 11 14:29:16 2024] ? __pfx_worker_thread+0x10/0x10
[Mon Mar 11 14:29:16 2024] kthread+0xdd/0x100
[Mon Mar 11 14:29:16 2024] ? __pfx_kthread+0x10/0x10
[Mon Mar 11 14:29:16 2024] ret_from_fork+0x29/0x50
[Mon Mar 11 14:29:16 2024] </TASK>
[Mon Mar 11 14:29:16 2024] task:kworker/u97:0 state:I stack:0 pid:2473005 ppid:2 flags:0x00004000
[Mon Mar 11 14:29:16 2024] Workqueue: 0x0 (rpciod)
[Mon Mar 11 14:29:16 2024] Call Trace:
[Mon Mar 11 14:29:16 2024] <TASK>
[Mon Mar 11 14:29:16 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:16 2024] ? __pfx_worker_thread+0x10/0x10
[Mon Mar 11 14:29:16 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:16 2024] worker_thread+0xbb/0x3a0
[Mon Mar 11 14:29:16 2024] ? __pfx_worker_thread+0x10/0x10
[Mon Mar 11 14:29:16 2024] kthread+0xdd/0x100
[Mon Mar 11 14:29:16 2024] ? __pfx_kthread+0x10/0x10
[Mon Mar 11 14:29:16 2024] ret_from_fork+0x29/0x50
[Mon Mar 11 14:29:16 2024] </TASK>
[Mon Mar 11 14:29:16 2024] task:kworker/2:1 state:I stack:0 pid:2473316 ppid:2 flags:0x00004000
[Mon Mar 11 14:29:16 2024] Workqueue: 0x0 (rcu_par_gp)
[Mon Mar 11 14:29:16 2024] Call Trace:
[Mon Mar 11 14:29:16 2024] <TASK>
[Mon Mar 11 14:29:16 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:16 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:16 2024] worker_thread+0xbb/0x3a0
[Mon Mar 11 14:29:16 2024] ? __pfx_worker_thread+0x10/0x10
[Mon Mar 11 14:29:16 2024] kthread+0xdd/0x100
[Mon Mar 11 14:29:16 2024] ? __pfx_kthread+0x10/0x10
[Mon Mar 11 14:29:16 2024] ret_from_fork+0x29/0x50
[Mon Mar 11 14:29:16 2024] </TASK>
[Mon Mar 11 14:29:16 2024] task:kworker/45:2 state:I stack:0 pid:2473358 ppid:2 flags:0x00004000
[Mon Mar 11 14:29:16 2024] Call Trace:
[Mon Mar 11 14:29:16 2024] <TASK>
[Mon Mar 11 14:29:16 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:16 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:16 2024] worker_thread+0xbb/0x3a0
[Mon Mar 11 14:29:16 2024] ? __pfx_worker_thread+0x10/0x10
[Mon Mar 11 14:29:16 2024] kthread+0xdd/0x100
[Mon Mar 11 14:29:16 2024] ? __pfx_kthread+0x10/0x10
[Mon Mar 11 14:29:16 2024] ret_from_fork+0x29/0x50
[Mon Mar 11 14:29:16 2024] </TASK>
[Mon Mar 11 14:29:16 2024] task:kworker/25:2 state:I stack:0 pid:2473359 ppid:2 flags:0x00004000
[Mon Mar 11 14:29:16 2024] Call Trace:
[Mon Mar 11 14:29:16 2024] <TASK>
[Mon Mar 11 14:29:16 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:16 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:16 2024] worker_thread+0xbb/0x3a0
[Mon Mar 11 14:29:16 2024] ? __pfx_worker_thread+0x10/0x10
[Mon Mar 11 14:29:16 2024] kthread+0xdd/0x100
[Mon Mar 11 14:29:16 2024] ? __pfx_kthread+0x10/0x10
[Mon Mar 11 14:29:16 2024] ret_from_fork+0x29/0x50
[Mon Mar 11 14:29:16 2024] </TASK>
[Mon Mar 11 14:29:16 2024] task:kworker/30:1 state:I stack:0 pid:2473379 ppid:2 flags:0x00004000
[Mon Mar 11 14:29:16 2024] Call Trace:
[Mon Mar 11 14:29:16 2024] <TASK>
[Mon Mar 11 14:29:16 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:16 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:16 2024] worker_thread+0xbb/0x3a0
[Mon Mar 11 14:29:16 2024] ? __pfx_worker_thread+0x10/0x10
[Mon Mar 11 14:29:16 2024] kthread+0xdd/0x100
[Mon Mar 11 14:29:16 2024] ? __pfx_kthread+0x10/0x10
[Mon Mar 11 14:29:16 2024] ret_from_fork+0x29/0x50
[Mon Mar 11 14:29:16 2024] </TASK>
[Mon Mar 11 14:29:16 2024] task:kworker/u98:1 state:I stack:0 pid:2473403 ppid:2 flags:0x00004000
[Mon Mar 11 14:29:16 2024] Workqueue: 0x0 (flush-253:79)
[Mon Mar 11 14:29:16 2024] Call Trace:
[Mon Mar 11 14:29:16 2024] <TASK>
[Mon Mar 11 14:29:16 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:16 2024] ? __pfx_worker_thread+0x10/0x10
[Mon Mar 11 14:29:16 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:16 2024] worker_thread+0xbb/0x3a0
[Mon Mar 11 14:29:16 2024] ? __pfx_worker_thread+0x10/0x10
[Mon Mar 11 14:29:16 2024] kthread+0xdd/0x100
[Mon Mar 11 14:29:16 2024] ? __pfx_kthread+0x10/0x10
[Mon Mar 11 14:29:16 2024] ret_from_fork+0x29/0x50
[Mon Mar 11 14:29:16 2024] </TASK>
[Mon Mar 11 14:29:16 2024] task:kworker/27:0 state:I stack:0 pid:2473569 ppid:2 flags:0x00004000
[Mon Mar 11 14:29:16 2024] Call Trace:
[Mon Mar 11 14:29:16 2024] <TASK>
[Mon Mar 11 14:29:16 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:16 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:16 2024] worker_thread+0xbb/0x3a0
[Mon Mar 11 14:29:16 2024] ? __pfx_worker_thread+0x10/0x10
[Mon Mar 11 14:29:16 2024] kthread+0xdd/0x100
[Mon Mar 11 14:29:16 2024] ? __pfx_kthread+0x10/0x10
[Mon Mar 11 14:29:16 2024] ret_from_fork+0x29/0x50
[Mon Mar 11 14:29:16 2024] </TASK>
[Mon Mar 11 14:29:16 2024] task:smbd[10.87.29.1 state:S stack:0 pid:2473572 ppid:4757 flags:0x00000002
[Mon Mar 11 14:29:16 2024] Call Trace:
[Mon Mar 11 14:29:16 2024] <TASK>
[Mon Mar 11 14:29:16 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:16 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:16 2024] schedule_hrtimeout_range_clock+0x95/0x120
[Mon Mar 11 14:29:16 2024] ? __pfx_hrtimer_wakeup+0x10/0x10
[Mon Mar 11 14:29:16 2024] ep_poll+0x348/0x3b0
[Mon Mar 11 14:29:16 2024] ? __pfx_autoremove_wake_function+0x10/0x10
[Mon Mar 11 14:29:16 2024] do_epoll_wait+0xb1/0xd0
[Mon Mar 11 14:29:16 2024] __x64_sys_epoll_wait+0x60/0x100
[Mon Mar 11 14:29:16 2024] do_syscall_64+0x59/0x90
[Mon Mar 11 14:29:16 2024] ? exc_page_fault+0x62/0x150
[Mon Mar 11 14:29:16 2024] entry_SYSCALL_64_after_hwframe+0x72/0xdc
[Mon Mar 11 14:29:16 2024] RIP: 0033:0x7fedb7d4e80a
[Mon Mar 11 14:29:16 2024] RSP: 002b:00007ffc284a9ad8 EFLAGS: 00000246 ORIG_RAX: 00000000000000e8
[Mon Mar 11 14:29:16 2024] RAX: ffffffffffffffda RBX: 000055f87dd397c0 RCX: 00007fedb7d4e80a
[Mon Mar 11 14:29:16 2024] RDX: 0000000000000001 RSI: 00007ffc284a9b0c RDI: 0000000000000005
[Mon Mar 11 14:29:16 2024] RBP: 000055f87dd2aa20 R08: 00000000000f4107 R09: 0000000000000000
[Mon Mar 11 14:29:16 2024] R10: 000000000000ea60 R11: 0000000000000246 R12: 000055f87dd61190
[Mon Mar 11 14:29:16 2024] R13: 000000000000ea60 R14: 0000000000000026 R15: 000055f87dd5b760
[Mon Mar 11 14:29:16 2024] </TASK>
[Mon Mar 11 14:29:16 2024] task:kworker/4:0 state:I stack:0 pid:2473634 ppid:2 flags:0x00004000
[Mon Mar 11 14:29:16 2024] Call Trace:
[Mon Mar 11 14:29:16 2024] <TASK>
[Mon Mar 11 14:29:16 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:16 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:16 2024] worker_thread+0xbb/0x3a0
[Mon Mar 11 14:29:16 2024] ? __pfx_worker_thread+0x10/0x10
[Mon Mar 11 14:29:16 2024] kthread+0xdd/0x100
[Mon Mar 11 14:29:16 2024] ? __pfx_kthread+0x10/0x10
[Mon Mar 11 14:29:16 2024] ret_from_fork+0x29/0x50
[Mon Mar 11 14:29:16 2024] </TASK>
[Mon Mar 11 14:29:16 2024] task:kworker/15:0 state:I stack:0 pid:2473641 ppid:2 flags:0x00004000
[Mon Mar 11 14:29:16 2024] Workqueue: 0x0 (rcu_par_gp)
[Mon Mar 11 14:29:16 2024] Call Trace:
[Mon Mar 11 14:29:16 2024] <TASK>
[Mon Mar 11 14:29:16 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:16 2024] ? __rcu_report_exp_rnp+0x77/0xc0
[Mon Mar 11 14:29:16 2024] ? __pfx_worker_thread+0x10/0x10
[Mon Mar 11 14:29:16 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:16 2024] worker_thread+0xbb/0x3a0
[Mon Mar 11 14:29:16 2024] ? __pfx_worker_thread+0x10/0x10
[Mon Mar 11 14:29:16 2024] kthread+0xdd/0x100
[Mon Mar 11 14:29:16 2024] ? __pfx_kthread+0x10/0x10
[Mon Mar 11 14:29:16 2024] ret_from_fork+0x29/0x50
[Mon Mar 11 14:29:16 2024] </TASK>
[Mon Mar 11 14:29:16 2024] task:kworker/22:1 state:I stack:0 pid:2474143 ppid:2 flags:0x00004000
[Mon Mar 11 14:29:16 2024] Workqueue: 0x0 (mm_percpu_wq)
[Mon Mar 11 14:29:16 2024] Call Trace:
[Mon Mar 11 14:29:16 2024] <TASK>
[Mon Mar 11 14:29:16 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:16 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:16 2024] worker_thread+0xbb/0x3a0
[Mon Mar 11 14:29:16 2024] ? __pfx_worker_thread+0x10/0x10
[Mon Mar 11 14:29:16 2024] kthread+0xdd/0x100
[Mon Mar 11 14:29:16 2024] ? __pfx_kthread+0x10/0x10
[Mon Mar 11 14:29:16 2024] ret_from_fork+0x29/0x50
[Mon Mar 11 14:29:16 2024] </TASK>
[Mon Mar 11 14:29:16 2024] task:kworker/42:2 state:I stack:0 pid:2474171 ppid:2 flags:0x00004000
[Mon Mar 11 14:29:16 2024] Call Trace:
[Mon Mar 11 14:29:16 2024] <TASK>
[Mon Mar 11 14:29:16 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:16 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:16 2024] worker_thread+0xbb/0x3a0
[Mon Mar 11 14:29:16 2024] ? __pfx_worker_thread+0x10/0x10
[Mon Mar 11 14:29:16 2024] kthread+0xdd/0x100
[Mon Mar 11 14:29:16 2024] ? __pfx_kthread+0x10/0x10
[Mon Mar 11 14:29:16 2024] ret_from_fork+0x29/0x50
[Mon Mar 11 14:29:16 2024] </TASK>
[Mon Mar 11 14:29:16 2024] task:kworker/18:2 state:I stack:0 pid:2474270 ppid:2 flags:0x00004000
[Mon Mar 11 14:29:16 2024] Call Trace:
[Mon Mar 11 14:29:16 2024] <TASK>
[Mon Mar 11 14:29:16 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:16 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:16 2024] worker_thread+0xbb/0x3a0
[Mon Mar 11 14:29:16 2024] ? __pfx_worker_thread+0x10/0x10
[Mon Mar 11 14:29:16 2024] kthread+0xdd/0x100
[Mon Mar 11 14:29:16 2024] ? __pfx_kthread+0x10/0x10
[Mon Mar 11 14:29:16 2024] ret_from_fork+0x29/0x50
[Mon Mar 11 14:29:16 2024] </TASK>
[Mon Mar 11 14:29:16 2024] task:kworker/8:1 state:I stack:0 pid:2474342 ppid:2 flags:0x00004000
[Mon Mar 11 14:29:16 2024] Workqueue: 0x0 (mm_percpu_wq)
[Mon Mar 11 14:29:16 2024] Call Trace:
[Mon Mar 11 14:29:16 2024] <TASK>
[Mon Mar 11 14:29:16 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:16 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:16 2024] worker_thread+0xbb/0x3a0
[Mon Mar 11 14:29:16 2024] ? __pfx_worker_thread+0x10/0x10
[Mon Mar 11 14:29:16 2024] kthread+0xdd/0x100
[Mon Mar 11 14:29:16 2024] ? __pfx_kthread+0x10/0x10
[Mon Mar 11 14:29:16 2024] ret_from_fork+0x29/0x50
[Mon Mar 11 14:29:16 2024] </TASK>
[Mon Mar 11 14:29:16 2024] task:kworker/47:1 state:I stack:0 pid:2474397 ppid:2 flags:0x00004000
[Mon Mar 11 14:29:16 2024] Workqueue: 0x0 (mm_percpu_wq)
[Mon Mar 11 14:29:16 2024] Call Trace:
[Mon Mar 11 14:29:16 2024] <TASK>
[Mon Mar 11 14:29:16 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:16 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:16 2024] worker_thread+0xbb/0x3a0
[Mon Mar 11 14:29:16 2024] ? __pfx_worker_thread+0x10/0x10
[Mon Mar 11 14:29:16 2024] kthread+0xdd/0x100
[Mon Mar 11 14:29:16 2024] ? __pfx_kthread+0x10/0x10
[Mon Mar 11 14:29:16 2024] ret_from_fork+0x29/0x50
[Mon Mar 11 14:29:16 2024] </TASK>
[Mon Mar 11 14:29:16 2024] task:kworker/26:1 state:I stack:0 pid:2474460 ppid:2 flags:0x00004000
[Mon Mar 11 14:29:16 2024] Workqueue: 0x0 (rcu_par_gp)
[Mon Mar 11 14:29:16 2024] Call Trace:
[Mon Mar 11 14:29:16 2024] <TASK>
[Mon Mar 11 14:29:16 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:16 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:16 2024] worker_thread+0xbb/0x3a0
[Mon Mar 11 14:29:16 2024] ? __pfx_worker_thread+0x10/0x10
[Mon Mar 11 14:29:16 2024] kthread+0xdd/0x100
[Mon Mar 11 14:29:16 2024] ? __pfx_kthread+0x10/0x10
[Mon Mar 11 14:29:16 2024] ret_from_fork+0x29/0x50
[Mon Mar 11 14:29:16 2024] </TASK>
[Mon Mar 11 14:29:16 2024] task:kworker/44:0 state:I stack:0 pid:2474523 ppid:2 flags:0x00004000
[Mon Mar 11 14:29:16 2024] Workqueue: 0x0 (mm_percpu_wq)
[Mon Mar 11 14:29:16 2024] Call Trace:
[Mon Mar 11 14:29:16 2024] <TASK>
[Mon Mar 11 14:29:16 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:16 2024] ? __pfx_worker_thread+0x10/0x10
[Mon Mar 11 14:29:16 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:16 2024] worker_thread+0xbb/0x3a0
[Mon Mar 11 14:29:16 2024] ? __pfx_worker_thread+0x10/0x10
[Mon Mar 11 14:29:16 2024] kthread+0xdd/0x100
[Mon Mar 11 14:29:16 2024] ? __pfx_kthread+0x10/0x10
[Mon Mar 11 14:29:16 2024] ret_from_fork+0x29/0x50
[Mon Mar 11 14:29:16 2024] </TASK>
[Mon Mar 11 14:29:16 2024] task:kworker/23:1 state:I stack:0 pid:2474524 ppid:2 flags:0x00004000
[Mon Mar 11 14:29:16 2024] Call Trace:
[Mon Mar 11 14:29:16 2024] <TASK>
[Mon Mar 11 14:29:16 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:16 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:16 2024] worker_thread+0xbb/0x3a0
[Mon Mar 11 14:29:16 2024] ? __pfx_worker_thread+0x10/0x10
[Mon Mar 11 14:29:16 2024] kthread+0xdd/0x100
[Mon Mar 11 14:29:16 2024] ? __pfx_kthread+0x10/0x10
[Mon Mar 11 14:29:16 2024] ret_from_fork+0x29/0x50
[Mon Mar 11 14:29:16 2024] </TASK>
[Mon Mar 11 14:29:16 2024] task:kworker/28:2 state:I stack:0 pid:2474525 ppid:2 flags:0x00004000
[Mon Mar 11 14:29:16 2024] Call Trace:
[Mon Mar 11 14:29:16 2024] <TASK>
[Mon Mar 11 14:29:16 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:16 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:16 2024] worker_thread+0xbb/0x3a0
[Mon Mar 11 14:29:16 2024] ? __pfx_worker_thread+0x10/0x10
[Mon Mar 11 14:29:16 2024] kthread+0xdd/0x100
[Mon Mar 11 14:29:16 2024] ? __pfx_kthread+0x10/0x10
[Mon Mar 11 14:29:16 2024] ret_from_fork+0x29/0x50
[Mon Mar 11 14:29:16 2024] </TASK>
[Mon Mar 11 14:29:16 2024] task:kworker/37:0 state:I stack:0 pid:2474527 ppid:2 flags:0x00004000
[Mon Mar 11 14:29:16 2024] Call Trace:
[Mon Mar 11 14:29:16 2024] <TASK>
[Mon Mar 11 14:29:16 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:16 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:16 2024] worker_thread+0xbb/0x3a0
[Mon Mar 11 14:29:16 2024] ? __pfx_worker_thread+0x10/0x10
[Mon Mar 11 14:29:16 2024] kthread+0xdd/0x100
[Mon Mar 11 14:29:16 2024] ? __pfx_kthread+0x10/0x10
[Mon Mar 11 14:29:16 2024] ret_from_fork+0x29/0x50
[Mon Mar 11 14:29:16 2024] </TASK>
[Mon Mar 11 14:29:16 2024] task:kworker/11:0 state:I stack:0 pid:2474533 ppid:2 flags:0x00004000
[Mon Mar 11 14:29:16 2024] Call Trace:
[Mon Mar 11 14:29:16 2024] <TASK>
[Mon Mar 11 14:29:16 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:16 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:16 2024] worker_thread+0xbb/0x3a0
[Mon Mar 11 14:29:16 2024] ? __pfx_worker_thread+0x10/0x10
[Mon Mar 11 14:29:16 2024] kthread+0xdd/0x100
[Mon Mar 11 14:29:16 2024] ? __pfx_kthread+0x10/0x10
[Mon Mar 11 14:29:16 2024] ret_from_fork+0x29/0x50
[Mon Mar 11 14:29:16 2024] </TASK>
[Mon Mar 11 14:29:16 2024] task:kworker/31:0 state:I stack:0 pid:2474534 ppid:2 flags:0x00004000
[Mon Mar 11 14:29:16 2024] Workqueue: 0x0 (rcu_par_gp)
[Mon Mar 11 14:29:16 2024] Call Trace:
[Mon Mar 11 14:29:16 2024] <TASK>
[Mon Mar 11 14:29:16 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:16 2024] ? __pfx_worker_thread+0x10/0x10
[Mon Mar 11 14:29:16 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:16 2024] worker_thread+0xbb/0x3a0
[Mon Mar 11 14:29:16 2024] ? __pfx_worker_thread+0x10/0x10
[Mon Mar 11 14:29:16 2024] kthread+0xdd/0x100
[Mon Mar 11 14:29:16 2024] ? __pfx_kthread+0x10/0x10
[Mon Mar 11 14:29:16 2024] ret_from_fork+0x29/0x50
[Mon Mar 11 14:29:16 2024] </TASK>
[Mon Mar 11 14:29:16 2024] task:kworker/33:2 state:I stack:0 pid:2474557 ppid:2 flags:0x00004000
[Mon Mar 11 14:29:16 2024] Call Trace:
[Mon Mar 11 14:29:16 2024] <TASK>
[Mon Mar 11 14:29:16 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:16 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:16 2024] worker_thread+0xbb/0x3a0
[Mon Mar 11 14:29:16 2024] ? __pfx_worker_thread+0x10/0x10
[Mon Mar 11 14:29:16 2024] kthread+0xdd/0x100
[Mon Mar 11 14:29:16 2024] ? __pfx_kthread+0x10/0x10
[Mon Mar 11 14:29:16 2024] ret_from_fork+0x29/0x50
[Mon Mar 11 14:29:16 2024] </TASK>
[Mon Mar 11 14:29:16 2024] task:kworker/9:2 state:I stack:0 pid:2474561 ppid:2 flags:0x00004000
[Mon Mar 11 14:29:16 2024] Workqueue: 0x0 (mm_percpu_wq)
[Mon Mar 11 14:29:16 2024] Call Trace:
[Mon Mar 11 14:29:16 2024] <TASK>
[Mon Mar 11 14:29:16 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:16 2024] ? __pfx_worker_thread+0x10/0x10
[Mon Mar 11 14:29:16 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:16 2024] worker_thread+0xbb/0x3a0
[Mon Mar 11 14:29:16 2024] ? __pfx_worker_thread+0x10/0x10
[Mon Mar 11 14:29:16 2024] kthread+0xdd/0x100
[Mon Mar 11 14:29:16 2024] ? __pfx_kthread+0x10/0x10
[Mon Mar 11 14:29:16 2024] ret_from_fork+0x29/0x50
[Mon Mar 11 14:29:16 2024] </TASK>
[Mon Mar 11 14:29:16 2024] task:sshd state:S stack:0 pid:2474602 ppid:4412 flags:0x00000002
[Mon Mar 11 14:29:16 2024] Call Trace:
[Mon Mar 11 14:29:16 2024] <TASK>
[Mon Mar 11 14:29:16 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:16 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:16 2024] schedule_hrtimeout_range_clock+0x10f/0x120
[Mon Mar 11 14:29:16 2024] ? add_wait_queue+0x65/0xa0
[Mon Mar 11 14:29:16 2024] ? unix_poll+0xf4/0x100
[Mon Mar 11 14:29:16 2024] ? sock_poll+0x4c/0xe0
[Mon Mar 11 14:29:16 2024] do_poll.constprop.0+0x248/0x390
[Mon Mar 11 14:29:16 2024] ? remove_wait_queue+0x20/0x60
[Mon Mar 11 14:29:16 2024] do_sys_poll+0x1c8/0x250
[Mon Mar 11 14:29:16 2024] ? skb_queue_tail+0x1b/0x50
[Mon Mar 11 14:29:16 2024] ? sock_def_readable+0x10/0xc0
[Mon Mar 11 14:29:16 2024] ? _copy_to_iter+0x1d4/0x630
[Mon Mar 11 14:29:16 2024] ? _copy_to_iter+0x1d4/0x630
[Mon Mar 11 14:29:16 2024] ? __pfx_pollwake+0x10/0x10
[Mon Mar 11 14:29:16 2024] ? __skb_datagram_iter+0x79/0x2e0
[Mon Mar 11 14:29:16 2024] ? __pfx_simple_copy_to_iter+0x10/0x10
[Mon Mar 11 14:29:16 2024] ? rcu_nocb_try_bypass+0x5e/0x460
[Mon Mar 11 14:29:16 2024] ? __kmem_cache_alloc_node+0x1c7/0x2d0
[Mon Mar 11 14:29:16 2024] ? __audit_sockaddr+0x5f/0x80
[Mon Mar 11 14:29:16 2024] ? rcu_nocb_try_bypass+0x5e/0x460
[Mon Mar 11 14:29:16 2024] ? __pfx_deferred_put_nlk_sk+0x10/0x10
[Mon Mar 11 14:29:16 2024] ? rcu_nocb_try_bypass+0x5e/0x460
[Mon Mar 11 14:29:16 2024] ? __pfx_inode_free_by_rcu+0x10/0x10
[Mon Mar 11 14:29:16 2024] ? rseq_get_rseq_cs+0x1d/0x240
[Mon Mar 11 14:29:16 2024] ? rcu_nocb_try_bypass+0x5e/0x460
[Mon Mar 11 14:29:16 2024] ? rseq_ip_fixup+0x6e/0x1a0
[Mon Mar 11 14:29:16 2024] ? __rseq_handle_notify_resume+0x26/0xb0
[Mon Mar 11 14:29:16 2024] ? exit_to_user_mode_loop+0xd0/0x130
[Mon Mar 11 14:29:16 2024] ? exit_to_user_mode_prepare+0xb6/0x100
[Mon Mar 11 14:29:16 2024] ? auditd_test_task+0x3c/0x50
[Mon Mar 11 14:29:16 2024] __x64_sys_poll+0x39/0x140
[Mon Mar 11 14:29:16 2024] ? syscall_trace_enter.constprop.0+0x126/0x1a0
[Mon Mar 11 14:29:16 2024] do_syscall_64+0x59/0x90
[Mon Mar 11 14:29:16 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:16 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:16 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:16 2024] entry_SYSCALL_64_after_hwframe+0x72/0xdc
[Mon Mar 11 14:29:16 2024] RIP: 0033:0x7fb3535426c7
[Mon Mar 11 14:29:16 2024] RSP: 002b:00007ffce5622d88 EFLAGS: 00000246 ORIG_RAX: 0000000000000007
[Mon Mar 11 14:29:16 2024] RAX: ffffffffffffffda RBX: 00007ffce5622da0 RCX: 00007fb3535426c7
[Mon Mar 11 14:29:16 2024] RDX: 00000000ffffffff RSI: 0000000000000001 RDI: 00007ffce5622da0
[Mon Mar 11 14:29:16 2024] RBP: 000055c449fb00c0 R08: 0000000000000001 R09: 00007ffce561c0b8
[Mon Mar 11 14:29:16 2024] R10: 0000000000000040 R11: 0000000000000246 R12: 000055c449fb00c0
[Mon Mar 11 14:29:16 2024] R13: 000055c44a8fd2b0 R14: 000055c44a8e6650 R15: 00000000ffffffff
[Mon Mar 11 14:29:16 2024] </TASK>
[Mon Mar 11 14:29:16 2024] task:sshd state:S stack:0 pid:2474605 ppid:2474602 flags:0x00000002
[Mon Mar 11 14:29:16 2024] Call Trace:
[Mon Mar 11 14:29:16 2024] <TASK>
[Mon Mar 11 14:29:16 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:16 2024] ? dequeue_skb+0x7d/0x500
[Mon Mar 11 14:29:16 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:16 2024] schedule_hrtimeout_range_clock+0x10f/0x120
[Mon Mar 11 14:29:16 2024] ? n_tty_poll+0x1cb/0x1f0
[Mon Mar 11 14:29:16 2024] ? tty_poll+0x71/0xa0
[Mon Mar 11 14:29:16 2024] do_select+0x69e/0x7c0
[Mon Mar 11 14:29:16 2024] ? __pfx_pollwake+0x10/0x10
[Mon Mar 11 14:29:16 2024] ? __pfx_pollwake+0x10/0x10
[Mon Mar 11 14:29:16 2024] ? __pfx_pollwake+0x10/0x10
[Mon Mar 11 14:29:16 2024] ? __mod_timer+0x286/0x3d0
[Mon Mar 11 14:29:16 2024] ? sk_reset_timer+0x14/0x60
[Mon Mar 11 14:29:16 2024] ? tcp_schedule_loss_probe.part.0+0x12c/0x1a0
[Mon Mar 11 14:29:16 2024] ? tcp_write_xmit+0x6f5/0xaa0
[Mon Mar 11 14:29:16 2024] ? __tcp_push_pending_frames+0x32/0xf0
[Mon Mar 11 14:29:16 2024] ? tcp_sendmsg_locked+0xaae/0xc20
[Mon Mar 11 14:29:16 2024] core_sys_select+0x1a0/0x3b0
[Mon Mar 11 14:29:16 2024] ? __audit_filter_op+0xa5/0xf0
[Mon Mar 11 14:29:16 2024] ? audit_reset_context.part.0.constprop.0+0x273/0x2e0
[Mon Mar 11 14:29:16 2024] ? _copy_from_user+0x27/0x60
[Mon Mar 11 14:29:16 2024] do_pselect.constprop.0+0xca/0x170
[Mon Mar 11 14:29:16 2024] __x64_sys_pselect6+0x39/0x70
[Mon Mar 11 14:29:16 2024] do_syscall_64+0x59/0x90
[Mon Mar 11 14:29:16 2024] ? syscall_exit_to_user_mode+0x22/0x40
[Mon Mar 11 14:29:16 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:16 2024] ? syscall_exit_to_user_mode+0x22/0x40
[Mon Mar 11 14:29:16 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:16 2024] ? sysvec_apic_timer_interrupt+0x3c/0x90
[Mon Mar 11 14:29:16 2024] entry_SYSCALL_64_after_hwframe+0x72/0xdc
[Mon Mar 11 14:29:16 2024] RIP: 0033:0x7fb353544f64
[Mon Mar 11 14:29:16 2024] RSP: 002b:00007ffce5622a40 EFLAGS: 00000246 ORIG_RAX: 000000000000010e
[Mon Mar 11 14:29:16 2024] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007fb353544f64
[Mon Mar 11 14:29:16 2024] RDX: 000055c44a929570 RSI: 000055c44a94b810 RDI: 000000000000000f
[Mon Mar 11 14:29:16 2024] RBP: 000055c44a94b810 R08: 0000000000000000 R09: 00007ffce5622a80
[Mon Mar 11 14:29:16 2024] R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000000008
[Mon Mar 11 14:29:16 2024] R13: 000055c44a8fefe0 R14: 000055c44a929570 R15: 000055c44a8fd2b0
[Mon Mar 11 14:29:16 2024] </TASK>
[Mon Mar 11 14:29:16 2024] task:bash state:S stack:0 pid:2474608 ppid:2474605 flags:0x00000002
[Mon Mar 11 14:29:16 2024] Call Trace:
[Mon Mar 11 14:29:16 2024] <TASK>
[Mon Mar 11 14:29:16 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:16 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:16 2024] do_wait+0x15b/0x2f0
[Mon Mar 11 14:29:16 2024] kernel_wait4+0xa6/0x140
[Mon Mar 11 14:29:16 2024] ? __pfx_child_wait_callback+0x10/0x10
[Mon Mar 11 14:29:16 2024] do_syscall_64+0x59/0x90
[Mon Mar 11 14:29:16 2024] ? audit_reset_context.part.0.constprop.0+0x273/0x2e0
[Mon Mar 11 14:29:16 2024] ? syscall_exit_work+0x103/0x130
[Mon Mar 11 14:29:16 2024] ? syscall_exit_to_user_mode+0x22/0x40
[Mon Mar 11 14:29:16 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:16 2024] ? syscall_exit_work+0x103/0x130
[Mon Mar 11 14:29:16 2024] ? syscall_exit_to_user_mode+0x22/0x40
[Mon Mar 11 14:29:16 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:16 2024] ? exc_page_fault+0x62/0x150
[Mon Mar 11 14:29:16 2024] entry_SYSCALL_64_after_hwframe+0x72/0xdc
[Mon Mar 11 14:29:16 2024] RIP: 0033:0x7f8d3c3182ca
[Mon Mar 11 14:29:16 2024] RSP: 002b:00007ffcf6dd9a98 EFLAGS: 00000246 ORIG_RAX: 000000000000003d
[Mon Mar 11 14:29:16 2024] RAX: ffffffffffffffda RBX: 000000000025c49f RCX: 00007f8d3c3182ca
[Mon Mar 11 14:29:16 2024] RDX: 000000000000000a RSI: 00007ffcf6dd9ac0 RDI: 00000000ffffffff
[Mon Mar 11 14:29:16 2024] RBP: 0000000000000001 R08: 0000000000000001 R09: 0000000000000000
[Mon Mar 11 14:29:16 2024] R10: 0000000000000000 R11: 0000000000000246 R12: 000000000000000a
[Mon Mar 11 14:29:16 2024] R13: 00007ffcf6dd9b20 R14: 0000000000000000 R15: 0000000000000000
[Mon Mar 11 14:29:16 2024] </TASK>
[Mon Mar 11 14:29:16 2024] task:kworker/14:2 state:R running task stack:0 pid:2474614 ppid:2 flags:0x00004000
[Mon Mar 11 14:29:16 2024] Workqueue: 0x0 (events)
[Mon Mar 11 14:29:16 2024] Call Trace:
[Mon Mar 11 14:29:16 2024] <TASK>
[Mon Mar 11 14:29:16 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:16 2024] ? __pfx_worker_thread+0x10/0x10
[Mon Mar 11 14:29:16 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:16 2024] worker_thread+0xbb/0x3a0
[Mon Mar 11 14:29:16 2024] ? __pfx_worker_thread+0x10/0x10
[Mon Mar 11 14:29:16 2024] kthread+0xdd/0x100
[Mon Mar 11 14:29:16 2024] ? __pfx_kthread+0x10/0x10
[Mon Mar 11 14:29:16 2024] ret_from_fork+0x29/0x50
[Mon Mar 11 14:29:16 2024] </TASK>
[Mon Mar 11 14:29:16 2024] task:kworker/20:0 state:I stack:0 pid:2474728 ppid:2 flags:0x00004000
[Mon Mar 11 14:29:16 2024] Workqueue: 0x0 (events)
[Mon Mar 11 14:29:16 2024] Call Trace:
[Mon Mar 11 14:29:16 2024] <TASK>
[Mon Mar 11 14:29:16 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:16 2024] ? __pfx_worker_thread+0x10/0x10
[Mon Mar 11 14:29:16 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:16 2024] worker_thread+0xbb/0x3a0
[Mon Mar 11 14:29:16 2024] ? __pfx_worker_thread+0x10/0x10
[Mon Mar 11 14:29:16 2024] kthread+0xdd/0x100
[Mon Mar 11 14:29:16 2024] ? __pfx_kthread+0x10/0x10
[Mon Mar 11 14:29:16 2024] ret_from_fork+0x29/0x50
[Mon Mar 11 14:29:16 2024] </TASK>
[Mon Mar 11 14:29:16 2024] task:kworker/21:2 state:I stack:0 pid:2474756 ppid:2 flags:0x00004000
[Mon Mar 11 14:29:16 2024] Workqueue: 0x0 (rcu_par_gp)
[Mon Mar 11 14:29:16 2024] Call Trace:
[Mon Mar 11 14:29:16 2024] <TASK>
[Mon Mar 11 14:29:16 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:16 2024] ? __rcu_report_exp_rnp+0x77/0xc0
[Mon Mar 11 14:29:16 2024] ? __pfx_worker_thread+0x10/0x10
[Mon Mar 11 14:29:16 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:16 2024] worker_thread+0xbb/0x3a0
[Mon Mar 11 14:29:16 2024] ? __pfx_worker_thread+0x10/0x10
[Mon Mar 11 14:29:16 2024] kthread+0xdd/0x100
[Mon Mar 11 14:29:16 2024] ? __pfx_kthread+0x10/0x10
[Mon Mar 11 14:29:16 2024] ret_from_fork+0x29/0x50
[Mon Mar 11 14:29:16 2024] </TASK>
[Mon Mar 11 14:29:16 2024] task:kworker/17:0 state:I stack:0 pid:2474774 ppid:2 flags:0x00004000
[Mon Mar 11 14:29:16 2024] Call Trace:
[Mon Mar 11 14:29:16 2024] <TASK>
[Mon Mar 11 14:29:16 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:16 2024] ? __pfx_worker_thread+0x10/0x10
[Mon Mar 11 14:29:16 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:16 2024] worker_thread+0xbb/0x3a0
[Mon Mar 11 14:29:16 2024] ? __pfx_worker_thread+0x10/0x10
[Mon Mar 11 14:29:16 2024] kthread+0xdd/0x100
[Mon Mar 11 14:29:16 2024] ? __pfx_kthread+0x10/0x10
[Mon Mar 11 14:29:16 2024] ret_from_fork+0x29/0x50
[Mon Mar 11 14:29:16 2024] </TASK>
[Mon Mar 11 14:29:16 2024] task:kworker/7:1 state:I stack:0 pid:2474775 ppid:2 flags:0x00004000
[Mon Mar 11 14:29:16 2024] Workqueue: 0x0 (lpfc_wq)
[Mon Mar 11 14:29:16 2024] Call Trace:
[Mon Mar 11 14:29:16 2024] <TASK>
[Mon Mar 11 14:29:16 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:16 2024] ? __pfx_worker_thread+0x10/0x10
[Mon Mar 11 14:29:16 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:16 2024] worker_thread+0xbb/0x3a0
[Mon Mar 11 14:29:16 2024] ? __pfx_worker_thread+0x10/0x10
[Mon Mar 11 14:29:16 2024] kthread+0xdd/0x100
[Mon Mar 11 14:29:16 2024] ? __pfx_kthread+0x10/0x10
[Mon Mar 11 14:29:16 2024] ret_from_fork+0x29/0x50
[Mon Mar 11 14:29:16 2024] </TASK>
[Mon Mar 11 14:29:16 2024] task:kworker/29:2 state:I stack:0 pid:2474780 ppid:2 flags:0x00004000
[Mon Mar 11 14:29:16 2024] Call Trace:
[Mon Mar 11 14:29:16 2024] <TASK>
[Mon Mar 11 14:29:16 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:16 2024] ? __pfx_worker_thread+0x10/0x10
[Mon Mar 11 14:29:16 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:16 2024] worker_thread+0xbb/0x3a0
[Mon Mar 11 14:29:16 2024] ? __pfx_worker_thread+0x10/0x10
[Mon Mar 11 14:29:16 2024] kthread+0xdd/0x100
[Mon Mar 11 14:29:16 2024] ? __pfx_kthread+0x10/0x10
[Mon Mar 11 14:29:16 2024] ret_from_fork+0x29/0x50
[Mon Mar 11 14:29:16 2024] </TASK>
[Mon Mar 11 14:29:16 2024] task:kworker/3:1 state:I stack:0 pid:2474822 ppid:2 flags:0x00004000
[Mon Mar 11 14:29:16 2024] Workqueue: 0x0 (mm_percpu_wq)
[Mon Mar 11 14:29:16 2024] Call Trace:
[Mon Mar 11 14:29:16 2024] <TASK>
[Mon Mar 11 14:29:16 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:16 2024] ? __pfx_worker_thread+0x10/0x10
[Mon Mar 11 14:29:16 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:16 2024] worker_thread+0xbb/0x3a0
[Mon Mar 11 14:29:16 2024] ? __pfx_worker_thread+0x10/0x10
[Mon Mar 11 14:29:16 2024] kthread+0xdd/0x100
[Mon Mar 11 14:29:16 2024] ? __pfx_kthread+0x10/0x10
[Mon Mar 11 14:29:16 2024] ret_from_fork+0x29/0x50
[Mon Mar 11 14:29:16 2024] </TASK>
[Mon Mar 11 14:29:16 2024] task:kworker/24:1 state:I stack:0 pid:2474838 ppid:2 flags:0x00004000
[Mon Mar 11 14:29:16 2024] Call Trace:
[Mon Mar 11 14:29:16 2024] <TASK>
[Mon Mar 11 14:29:16 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:16 2024] ? __pfx_worker_thread+0x10/0x10
[Mon Mar 11 14:29:16 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:16 2024] worker_thread+0xbb/0x3a0
[Mon Mar 11 14:29:16 2024] ? __pfx_worker_thread+0x10/0x10
[Mon Mar 11 14:29:16 2024] kthread+0xdd/0x100
[Mon Mar 11 14:29:16 2024] ? __pfx_kthread+0x10/0x10
[Mon Mar 11 14:29:16 2024] ret_from_fork+0x29/0x50
[Mon Mar 11 14:29:16 2024] </TASK>
[Mon Mar 11 14:29:16 2024] task:kworker/1:2 state:I stack:0 pid:2474995 ppid:2 flags:0x00004000
[Mon Mar 11 14:29:16 2024] Workqueue: 0x0 (mm_percpu_wq)
[Mon Mar 11 14:29:16 2024] Call Trace:
[Mon Mar 11 14:29:16 2024] <TASK>
[Mon Mar 11 14:29:16 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:16 2024] ? __pfx_worker_thread+0x10/0x10
[Mon Mar 11 14:29:16 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:16 2024] worker_thread+0xbb/0x3a0
[Mon Mar 11 14:29:16 2024] ? __pfx_worker_thread+0x10/0x10
[Mon Mar 11 14:29:16 2024] kthread+0xdd/0x100
[Mon Mar 11 14:29:16 2024] ? __pfx_kthread+0x10/0x10
[Mon Mar 11 14:29:16 2024] ret_from_fork+0x29/0x50
[Mon Mar 11 14:29:16 2024] </TASK>
[Mon Mar 11 14:29:16 2024] task:kworker/5:0 state:I stack:0 pid:2475005 ppid:2 flags:0x00004000
[Mon Mar 11 14:29:16 2024] Call Trace:
[Mon Mar 11 14:29:16 2024] <TASK>
[Mon Mar 11 14:29:16 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:16 2024] ? __pfx_worker_thread+0x10/0x10
[Mon Mar 11 14:29:16 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:16 2024] worker_thread+0xbb/0x3a0
[Mon Mar 11 14:29:16 2024] ? __pfx_worker_thread+0x10/0x10
[Mon Mar 11 14:29:16 2024] kthread+0xdd/0x100
[Mon Mar 11 14:29:16 2024] ? __pfx_kthread+0x10/0x10
[Mon Mar 11 14:29:16 2024] ret_from_fork+0x29/0x50
[Mon Mar 11 14:29:16 2024] </TASK>
[Mon Mar 11 14:29:16 2024] task:journalctl state:S stack:0 pid:2475167 ppid:2474608 flags:0x00000002
[Mon Mar 11 14:29:16 2024] Call Trace:
[Mon Mar 11 14:29:16 2024] <TASK>
[Mon Mar 11 14:29:16 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:16 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:16 2024] schedule_hrtimeout_range_clock+0x10f/0x120
[Mon Mar 11 14:29:16 2024] ? n_tty_poll+0x1cb/0x1f0
[Mon Mar 11 14:29:16 2024] ? tty_poll+0x71/0xa0
[Mon Mar 11 14:29:16 2024] do_poll.constprop.0+0x248/0x390
[Mon Mar 11 14:29:16 2024] ? select_idle_sibling+0x28/0x430
[Mon Mar 11 14:29:16 2024] do_sys_poll+0x1c8/0x250
[Mon Mar 11 14:29:16 2024] ? sched_clock+0xc/0x30
[Mon Mar 11 14:29:16 2024] ? __smp_call_single_queue+0x93/0x120
[Mon Mar 11 14:29:16 2024] ? select_idle_sibling+0x28/0x430
[Mon Mar 11 14:29:16 2024] ? ttwu_queue_wakelist+0xf2/0x110
[Mon Mar 11 14:29:16 2024] ? wake_affine+0x62/0x1f0
[Mon Mar 11 14:29:16 2024] ? sched_clock+0xc/0x30
[Mon Mar 11 14:29:16 2024] ? __smp_call_single_queue+0x93/0x120
[Mon Mar 11 14:29:16 2024] ? ttwu_queue_wakelist+0xf2/0x110
[Mon Mar 11 14:29:16 2024] ? try_to_wake_up+0x3e2/0x5d0
[Mon Mar 11 14:29:16 2024] ? __pfx_pollwake+0x10/0x10
[Mon Mar 11 14:29:16 2024] ? __pfx_pollwake+0x10/0x10
[Mon Mar 11 14:29:16 2024] ? __pfx_pollwake+0x10/0x10
[Mon Mar 11 14:29:16 2024] ? n_tty_write+0x267/0x3c0
[Mon Mar 11 14:29:16 2024] ? __wake_up+0x40/0x60
[Mon Mar 11 14:29:16 2024] ? do_tty_write+0x1a4/0x250
[Mon Mar 11 14:29:16 2024] ? __pfx_n_tty_write+0x10/0x10
[Mon Mar 11 14:29:16 2024] ? file_has_perm+0xc6/0xd0
[Mon Mar 11 14:29:16 2024] ? file_tty_write.constprop.0+0x98/0xc0
[Mon Mar 11 14:29:16 2024] ? __audit_filter_op+0xa5/0xf0
[Mon Mar 11 14:29:16 2024] __x64_sys_ppoll+0xbc/0x150
[Mon Mar 11 14:29:16 2024] do_syscall_64+0x59/0x90
[Mon Mar 11 14:29:16 2024] ? sysvec_apic_timer_interrupt+0x3c/0x90
[Mon Mar 11 14:29:16 2024] entry_SYSCALL_64_after_hwframe+0x72/0xdc
[Mon Mar 11 14:29:16 2024] RIP: 0033:0x7f027814279f
[Mon Mar 11 14:29:16 2024] RSP: 002b:00007fffd52e26b0 EFLAGS: 00000246 ORIG_RAX: 000000000000010f
[Mon Mar 11 14:29:16 2024] RAX: ffffffffffffffda RBX: 0000000000000002 RCX: 00007f027814279f
[Mon Mar 11 14:29:16 2024] RDX: 0000000000000000 RSI: 0000000000000002 RDI: 00007fffd52e2910
[Mon Mar 11 14:29:16 2024] RBP: 00007fffd52e2910 R08: 0000000000000008 R09: 00007fffd52e2670
[Mon Mar 11 14:29:16 2024] R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000003eee
[Mon Mar 11 14:29:16 2024] R13: 0000000000000000 R14: 00007fffd52e27d0 R15: 000055ae95ac4ad0
[Mon Mar 11 14:29:16 2024] </TASK>
[Mon Mar 11 14:29:16 2024] task:kworker/41:0 state:I stack:0 pid:2475205 ppid:2 flags:0x00004000
[Mon Mar 11 14:29:16 2024] Call Trace:
[Mon Mar 11 14:29:16 2024] <TASK>
[Mon Mar 11 14:29:16 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:16 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:16 2024] worker_thread+0xbb/0x3a0
[Mon Mar 11 14:29:16 2024] ? __pfx_worker_thread+0x10/0x10
[Mon Mar 11 14:29:16 2024] kthread+0xdd/0x100
[Mon Mar 11 14:29:16 2024] ? __pfx_kthread+0x10/0x10
[Mon Mar 11 14:29:16 2024] ret_from_fork+0x29/0x50
[Mon Mar 11 14:29:16 2024] </TASK>
[Mon Mar 11 14:29:16 2024] task:kworker/16:1 state:I stack:0 pid:2475250 ppid:2 flags:0x00004000
[Mon Mar 11 14:29:16 2024] Call Trace:
[Mon Mar 11 14:29:16 2024] <TASK>
[Mon Mar 11 14:29:16 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:16 2024] ? __pfx_worker_thread+0x10/0x10
[Mon Mar 11 14:29:16 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:16 2024] worker_thread+0xbb/0x3a0
[Mon Mar 11 14:29:16 2024] ? __pfx_worker_thread+0x10/0x10
[Mon Mar 11 14:29:16 2024] kthread+0xdd/0x100
[Mon Mar 11 14:29:16 2024] ? __pfx_kthread+0x10/0x10
[Mon Mar 11 14:29:16 2024] ret_from_fork+0x29/0x50
[Mon Mar 11 14:29:16 2024] </TASK>
[Mon Mar 11 14:29:16 2024] task:kworker/u97:4 state:I stack:0 pid:2475262 ppid:2 flags:0x00004000
[Mon Mar 11 14:29:16 2024] Workqueue: 0x0 (flush-253:77)
[Mon Mar 11 14:29:16 2024] Call Trace:
[Mon Mar 11 14:29:16 2024] <TASK>
[Mon Mar 11 14:29:16 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:16 2024] ? __pfx_worker_thread+0x10/0x10
[Mon Mar 11 14:29:16 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:16 2024] worker_thread+0xbb/0x3a0
[Mon Mar 11 14:29:16 2024] ? __pfx_worker_thread+0x10/0x10
[Mon Mar 11 14:29:16 2024] kthread+0xdd/0x100
[Mon Mar 11 14:29:16 2024] ? __pfx_kthread+0x10/0x10
[Mon Mar 11 14:29:16 2024] ret_from_fork+0x29/0x50
[Mon Mar 11 14:29:16 2024] </TASK>
[Mon Mar 11 14:29:16 2024] task:kworker/19:0 state:I stack:0 pid:2475271 ppid:2 flags:0x00004000
[Mon Mar 11 14:29:16 2024] Workqueue: 0x0 (mm_percpu_wq)
[Mon Mar 11 14:29:16 2024] Call Trace:
[Mon Mar 11 14:29:16 2024] <TASK>
[Mon Mar 11 14:29:16 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:16 2024] ? __pfx_worker_thread+0x10/0x10
[Mon Mar 11 14:29:16 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:16 2024] worker_thread+0xbb/0x3a0
[Mon Mar 11 14:29:16 2024] ? __pfx_worker_thread+0x10/0x10
[Mon Mar 11 14:29:16 2024] kthread+0xdd/0x100
[Mon Mar 11 14:29:16 2024] ? __pfx_kthread+0x10/0x10
[Mon Mar 11 14:29:16 2024] ret_from_fork+0x29/0x50
[Mon Mar 11 14:29:16 2024] </TASK>
[Mon Mar 11 14:29:16 2024] task:kworker/0:2 state:I stack:0 pid:2475316 ppid:2 flags:0x00004000
[Mon Mar 11 14:29:16 2024] Workqueue: 0x0 (rcu_gp)
[Mon Mar 11 14:29:16 2024] Call Trace:
[Mon Mar 11 14:29:16 2024] <TASK>
[Mon Mar 11 14:29:16 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:16 2024] ? __pfx_worker_thread+0x10/0x10
[Mon Mar 11 14:29:16 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:16 2024] worker_thread+0xbb/0x3a0
[Mon Mar 11 14:29:16 2024] ? __pfx_worker_thread+0x10/0x10
[Mon Mar 11 14:29:16 2024] kthread+0xdd/0x100
[Mon Mar 11 14:29:16 2024] ? __pfx_kthread+0x10/0x10
[Mon Mar 11 14:29:16 2024] ret_from_fork+0x29/0x50
[Mon Mar 11 14:29:16 2024] </TASK>
[Mon Mar 11 14:29:16 2024] task:kworker/34:2 state:I stack:0 pid:2475317 ppid:2 flags:0x00004000
[Mon Mar 11 14:29:16 2024] Workqueue: 0x0 (mm_percpu_wq)
[Mon Mar 11 14:29:16 2024] Call Trace:
[Mon Mar 11 14:29:16 2024] <TASK>
[Mon Mar 11 14:29:16 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:16 2024] ? __pfx_worker_thread+0x10/0x10
[Mon Mar 11 14:29:16 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:16 2024] worker_thread+0xbb/0x3a0
[Mon Mar 11 14:29:16 2024] ? __pfx_worker_thread+0x10/0x10
[Mon Mar 11 14:29:16 2024] kthread+0xdd/0x100
[Mon Mar 11 14:29:16 2024] ? __pfx_kthread+0x10/0x10
[Mon Mar 11 14:29:16 2024] ret_from_fork+0x29/0x50
[Mon Mar 11 14:29:16 2024] </TASK>
[Mon Mar 11 14:29:16 2024] task:kworker/10:2 state:I stack:0 pid:2475318 ppid:2 flags:0x00004000
[Mon Mar 11 14:29:16 2024] Call Trace:
[Mon Mar 11 14:29:16 2024] <TASK>
[Mon Mar 11 14:29:16 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:16 2024] ? __pfx_worker_thread+0x10/0x10
[Mon Mar 11 14:29:16 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:16 2024] worker_thread+0xbb/0x3a0
[Mon Mar 11 14:29:16 2024] ? __pfx_worker_thread+0x10/0x10
[Mon Mar 11 14:29:16 2024] kthread+0xdd/0x100
[Mon Mar 11 14:29:16 2024] ? __pfx_kthread+0x10/0x10
[Mon Mar 11 14:29:16 2024] ret_from_fork+0x29/0x50
[Mon Mar 11 14:29:16 2024] </TASK>
[Mon Mar 11 14:29:16 2024] task:kworker/40:1 state:I stack:0 pid:2475325 ppid:2 flags:0x00004000
[Mon Mar 11 14:29:16 2024] Workqueue: 0x0 (rcu_par_gp)
[Mon Mar 11 14:29:16 2024] Call Trace:
[Mon Mar 11 14:29:16 2024] <TASK>
[Mon Mar 11 14:29:16 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:16 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:16 2024] worker_thread+0xbb/0x3a0
[Mon Mar 11 14:29:16 2024] ? __pfx_worker_thread+0x10/0x10
[Mon Mar 11 14:29:16 2024] kthread+0xdd/0x100
[Mon Mar 11 14:29:16 2024] ? __pfx_kthread+0x10/0x10
[Mon Mar 11 14:29:16 2024] ret_from_fork+0x29/0x50
[Mon Mar 11 14:29:16 2024] </TASK>
[Mon Mar 11 14:29:16 2024] task:kworker/6:0 state:I stack:0 pid:2475365 ppid:2 flags:0x00004000
[Mon Mar 11 14:29:16 2024] Workqueue: 0x0 (mm_percpu_wq)
[Mon Mar 11 14:29:16 2024] Call Trace:
[Mon Mar 11 14:29:16 2024] <TASK>
[Mon Mar 11 14:29:16 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:16 2024] ? __pfx_worker_thread+0x10/0x10
[Mon Mar 11 14:29:16 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:16 2024] worker_thread+0xbb/0x3a0
[Mon Mar 11 14:29:16 2024] ? __pfx_worker_thread+0x10/0x10
[Mon Mar 11 14:29:16 2024] kthread+0xdd/0x100
[Mon Mar 11 14:29:16 2024] ? __pfx_kthread+0x10/0x10
[Mon Mar 11 14:29:16 2024] ret_from_fork+0x29/0x50
[Mon Mar 11 14:29:16 2024] </TASK>
[Mon Mar 11 14:29:16 2024] task:kworker/46:0 state:I stack:0 pid:2475460 ppid:2 flags:0x00004000
[Mon Mar 11 14:29:16 2024] Workqueue: 0x0 (mm_percpu_wq)
[Mon Mar 11 14:29:16 2024] Call Trace:
[Mon Mar 11 14:29:16 2024] <TASK>
[Mon Mar 11 14:29:16 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:16 2024] ? __pfx_worker_thread+0x10/0x10
[Mon Mar 11 14:29:16 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:16 2024] worker_thread+0xbb/0x3a0
[Mon Mar 11 14:29:16 2024] ? __pfx_worker_thread+0x10/0x10
[Mon Mar 11 14:29:16 2024] kthread+0xdd/0x100
[Mon Mar 11 14:29:16 2024] ? __pfx_kthread+0x10/0x10
[Mon Mar 11 14:29:16 2024] ret_from_fork+0x29/0x50
[Mon Mar 11 14:29:16 2024] </TASK>
[Mon Mar 11 14:29:16 2024] task:kworker/35:1 state:I stack:0 pid:2475663 ppid:2 flags:0x00004000
[Mon Mar 11 14:29:16 2024] Workqueue: 0x0 (mm_percpu_wq)
[Mon Mar 11 14:29:16 2024] Call Trace:
[Mon Mar 11 14:29:16 2024] <TASK>
[Mon Mar 11 14:29:16 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:16 2024] ? __pfx_worker_thread+0x10/0x10
[Mon Mar 11 14:29:16 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:16 2024] worker_thread+0xbb/0x3a0
[Mon Mar 11 14:29:16 2024] ? __pfx_worker_thread+0x10/0x10
[Mon Mar 11 14:29:16 2024] kthread+0xdd/0x100
[Mon Mar 11 14:29:16 2024] ? __pfx_kthread+0x10/0x10
[Mon Mar 11 14:29:16 2024] ret_from_fork+0x29/0x50
[Mon Mar 11 14:29:16 2024] </TASK>
[Mon Mar 11 14:29:16 2024] task:kworker/38:1 state:I stack:0 pid:2475692 ppid:2 flags:0x00004000
[Mon Mar 11 14:29:16 2024] Call Trace:
[Mon Mar 11 14:29:16 2024] <TASK>
[Mon Mar 11 14:29:16 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:16 2024] ? __pfx_worker_thread+0x10/0x10
[Mon Mar 11 14:29:16 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:16 2024] worker_thread+0xbb/0x3a0
[Mon Mar 11 14:29:16 2024] ? __pfx_worker_thread+0x10/0x10
[Mon Mar 11 14:29:16 2024] kthread+0xdd/0x100
[Mon Mar 11 14:29:16 2024] ? __pfx_kthread+0x10/0x10
[Mon Mar 11 14:29:16 2024] ret_from_fork+0x29/0x50
[Mon Mar 11 14:29:16 2024] </TASK>
[Mon Mar 11 14:29:16 2024] task:kworker/13:2 state:I stack:0 pid:2475727 ppid:2 flags:0x00004000
[Mon Mar 11 14:29:16 2024] Workqueue: 0x0 (mm_percpu_wq)
[Mon Mar 11 14:29:16 2024] Call Trace:
[Mon Mar 11 14:29:16 2024] <TASK>
[Mon Mar 11 14:29:16 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:16 2024] ? __pfx_worker_thread+0x10/0x10
[Mon Mar 11 14:29:16 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:16 2024] worker_thread+0xbb/0x3a0
[Mon Mar 11 14:29:16 2024] ? __pfx_worker_thread+0x10/0x10
[Mon Mar 11 14:29:16 2024] kthread+0xdd/0x100
[Mon Mar 11 14:29:16 2024] ? __pfx_kthread+0x10/0x10
[Mon Mar 11 14:29:16 2024] ret_from_fork+0x29/0x50
[Mon Mar 11 14:29:16 2024] </TASK>
[Mon Mar 11 14:29:16 2024] task:kworker/32:2 state:I stack:0 pid:2475859 ppid:2 flags:0x00004000
[Mon Mar 11 14:29:16 2024] Workqueue: 0x0 (rcu_gp)
[Mon Mar 11 14:29:16 2024] Call Trace:
[Mon Mar 11 14:29:16 2024] <TASK>
[Mon Mar 11 14:29:16 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:16 2024] ? __pfx_worker_thread+0x10/0x10
[Mon Mar 11 14:29:16 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:16 2024] worker_thread+0xbb/0x3a0
[Mon Mar 11 14:29:16 2024] ? __pfx_worker_thread+0x10/0x10
[Mon Mar 11 14:29:16 2024] kthread+0xdd/0x100
[Mon Mar 11 14:29:16 2024] ? __pfx_kthread+0x10/0x10
[Mon Mar 11 14:29:16 2024] ret_from_fork+0x29/0x50
[Mon Mar 11 14:29:16 2024] </TASK>
[Mon Mar 11 14:29:16 2024] task:kworker/30:2 state:I stack:0 pid:2475882 ppid:2 flags:0x00004000
[Mon Mar 11 14:29:16 2024] Workqueue: 0x0 (mm_percpu_wq)
[Mon Mar 11 14:29:16 2024] Call Trace:
[Mon Mar 11 14:29:16 2024] <TASK>
[Mon Mar 11 14:29:16 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:16 2024] ? __pfx_worker_thread+0x10/0x10
[Mon Mar 11 14:29:16 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:16 2024] worker_thread+0xbb/0x3a0
[Mon Mar 11 14:29:16 2024] ? __pfx_worker_thread+0x10/0x10
[Mon Mar 11 14:29:16 2024] kthread+0xdd/0x100
[Mon Mar 11 14:29:16 2024] ? __pfx_kthread+0x10/0x10
[Mon Mar 11 14:29:16 2024] ret_from_fork+0x29/0x50
[Mon Mar 11 14:29:16 2024] </TASK>
[Mon Mar 11 14:29:16 2024] task:kworker/45:1 state:I stack:0 pid:2476033 ppid:2 flags:0x00004000
[Mon Mar 11 14:29:16 2024] Workqueue: 0x0 (mm_percpu_wq)
[Mon Mar 11 14:29:16 2024] Call Trace:
[Mon Mar 11 14:29:16 2024] <TASK>
[Mon Mar 11 14:29:16 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:16 2024] ? __pfx_worker_thread+0x10/0x10
[Mon Mar 11 14:29:16 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:16 2024] worker_thread+0xbb/0x3a0
[Mon Mar 11 14:29:16 2024] ? __pfx_worker_thread+0x10/0x10
[Mon Mar 11 14:29:16 2024] kthread+0xdd/0x100
[Mon Mar 11 14:29:16 2024] ? __pfx_kthread+0x10/0x10
[Mon Mar 11 14:29:16 2024] ret_from_fork+0x29/0x50
[Mon Mar 11 14:29:16 2024] </TASK>
[Mon Mar 11 14:29:16 2024] task:kworker/36:2 state:I stack:0 pid:2476104 ppid:2 flags:0x00004000
[Mon Mar 11 14:29:16 2024] Call Trace:
[Mon Mar 11 14:29:16 2024] <TASK>
[Mon Mar 11 14:29:16 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:16 2024] ? __pfx_worker_thread+0x10/0x10
[Mon Mar 11 14:29:16 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:16 2024] worker_thread+0xbb/0x3a0
[Mon Mar 11 14:29:16 2024] ? __pfx_worker_thread+0x10/0x10
[Mon Mar 11 14:29:16 2024] kthread+0xdd/0x100
[Mon Mar 11 14:29:16 2024] ? __pfx_kthread+0x10/0x10
[Mon Mar 11 14:29:16 2024] ret_from_fork+0x29/0x50
[Mon Mar 11 14:29:16 2024] </TASK>
[Mon Mar 11 14:29:16 2024] task:kworker/27:1 state:I stack:0 pid:2476181 ppid:2 flags:0x00004000
[Mon Mar 11 14:29:16 2024] Workqueue: 0x0 (mm_percpu_wq)
[Mon Mar 11 14:29:16 2024] Call Trace:
[Mon Mar 11 14:29:16 2024] <TASK>
[Mon Mar 11 14:29:16 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:16 2024] ? __pfx_worker_thread+0x10/0x10
[Mon Mar 11 14:29:16 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:16 2024] worker_thread+0xbb/0x3a0
[Mon Mar 11 14:29:16 2024] ? __pfx_worker_thread+0x10/0x10
[Mon Mar 11 14:29:16 2024] kthread+0xdd/0x100
[Mon Mar 11 14:29:16 2024] ? __pfx_kthread+0x10/0x10
[Mon Mar 11 14:29:16 2024] ret_from_fork+0x29/0x50
[Mon Mar 11 14:29:16 2024] </TASK>
[Mon Mar 11 14:29:16 2024] task:kworker/12:1 state:I stack:0 pid:2476314 ppid:2 flags:0x00004000
[Mon Mar 11 14:29:16 2024] Workqueue: 0x0 (rcu_gp)
[Mon Mar 11 14:29:16 2024] Call Trace:
[Mon Mar 11 14:29:16 2024] <TASK>
[Mon Mar 11 14:29:16 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:16 2024] ? __pfx_worker_thread+0x10/0x10
[Mon Mar 11 14:29:16 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:16 2024] worker_thread+0xbb/0x3a0
[Mon Mar 11 14:29:16 2024] ? __pfx_worker_thread+0x10/0x10
[Mon Mar 11 14:29:16 2024] kthread+0xdd/0x100
[Mon Mar 11 14:29:16 2024] ? __pfx_kthread+0x10/0x10
[Mon Mar 11 14:29:16 2024] ret_from_fork+0x29/0x50
[Mon Mar 11 14:29:16 2024] </TASK>
[Mon Mar 11 14:29:16 2024] task:kworker/25:0 state:I stack:0 pid:2476365 ppid:2 flags:0x00004000
[Mon Mar 11 14:29:16 2024] Call Trace:
[Mon Mar 11 14:29:16 2024] <TASK>
[Mon Mar 11 14:29:16 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:16 2024] ? __pfx_worker_thread+0x10/0x10
[Mon Mar 11 14:29:16 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:16 2024] worker_thread+0xbb/0x3a0
[Mon Mar 11 14:29:16 2024] ? __pfx_worker_thread+0x10/0x10
[Mon Mar 11 14:29:16 2024] kthread+0xdd/0x100
[Mon Mar 11 14:29:16 2024] ? __pfx_kthread+0x10/0x10
[Mon Mar 11 14:29:16 2024] ret_from_fork+0x29/0x50
[Mon Mar 11 14:29:16 2024] </TASK>
[Mon Mar 11 14:29:16 2024] task:sshd state:S stack:0 pid:2476431 ppid:4412 flags:0x00000002
[Mon Mar 11 14:29:16 2024] Call Trace:
[Mon Mar 11 14:29:16 2024] <TASK>
[Mon Mar 11 14:29:16 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:16 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:16 2024] schedule_hrtimeout_range_clock+0x10f/0x120
[Mon Mar 11 14:29:16 2024] ? add_wait_queue+0x65/0xa0
[Mon Mar 11 14:29:16 2024] ? unix_poll+0xf4/0x100
[Mon Mar 11 14:29:16 2024] ? sock_poll+0x4c/0xe0
[Mon Mar 11 14:29:16 2024] do_poll.constprop.0+0x248/0x390
[Mon Mar 11 14:29:16 2024] do_sys_poll+0x1c8/0x250
[Mon Mar 11 14:29:16 2024] ? skb_queue_tail+0x1b/0x50
[Mon Mar 11 14:29:16 2024] ? _copy_to_iter+0x1d4/0x630
[Mon Mar 11 14:29:16 2024] ? __check_object_size.part.0+0x47/0xd0
[Mon Mar 11 14:29:16 2024] ? __pfx_pollwake+0x10/0x10
[Mon Mar 11 14:29:16 2024] ? __audit_sockaddr+0x5f/0x80
[Mon Mar 11 14:29:16 2024] ? kmalloc_trace+0x25/0xa0
[Mon Mar 11 14:29:16 2024] ? __check_object_size.part.0+0x35/0xd0
[Mon Mar 11 14:29:16 2024] ? _copy_to_user+0x1a/0x30
[Mon Mar 11 14:29:16 2024] ? move_addr_to_user+0x4b/0xe0
[Mon Mar 11 14:29:16 2024] ? rcu_nocb_try_bypass+0x5e/0x460
[Mon Mar 11 14:29:16 2024] ? rcu_nocb_try_bypass+0x5e/0x460
[Mon Mar 11 14:29:16 2024] ? __pfx_deferred_put_nlk_sk+0x10/0x10
[Mon Mar 11 14:29:16 2024] ? rcu_nocb_try_bypass+0x5e/0x460
[Mon Mar 11 14:29:16 2024] ? __pfx_inode_free_by_rcu+0x10/0x10
[Mon Mar 11 14:29:16 2024] ? rseq_get_rseq_cs+0x1d/0x240
[Mon Mar 11 14:29:16 2024] ? __dentry_kill+0x13a/0x180
[Mon Mar 11 14:29:16 2024] ? rseq_ip_fixup+0x6e/0x1a0
[Mon Mar 11 14:29:16 2024] ? __call_rcu_common.constprop.0+0x117/0x2b0
[Mon Mar 11 14:29:16 2024] ? auditd_test_task+0x3c/0x50
[Mon Mar 11 14:29:16 2024] __x64_sys_poll+0x39/0x140
[Mon Mar 11 14:29:16 2024] ? syscall_trace_enter.constprop.0+0x126/0x1a0
[Mon Mar 11 14:29:16 2024] do_syscall_64+0x59/0x90
[Mon Mar 11 14:29:16 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:16 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:16 2024] ? exc_page_fault+0x62/0x150
[Mon Mar 11 14:29:16 2024] entry_SYSCALL_64_after_hwframe+0x72/0xdc
[Mon Mar 11 14:29:16 2024] RIP: 0033:0x7f62089426c7
[Mon Mar 11 14:29:16 2024] RSP: 002b:00007ffc2b4f3f88 EFLAGS: 00000246 ORIG_RAX: 0000000000000007
[Mon Mar 11 14:29:16 2024] RAX: ffffffffffffffda RBX: 00007ffc2b4f3fa0 RCX: 00007f62089426c7
[Mon Mar 11 14:29:16 2024] RDX: 00000000ffffffff RSI: 0000000000000001 RDI: 00007ffc2b4f3fa0
[Mon Mar 11 14:29:16 2024] RBP: 000055abfba6f0c0 R08: 0000000000000001 R09: 00007ffc2b4ed2b8
[Mon Mar 11 14:29:16 2024] R10: 0000000000000040 R11: 0000000000000246 R12: 000055abfba6f0c0
[Mon Mar 11 14:29:16 2024] R13: 000055abfbe944d0 R14: 000055abfbe8aad0 R15: 00000000ffffffff
[Mon Mar 11 14:29:16 2024] </TASK>
[Mon Mar 11 14:29:16 2024] task:sshd state:S stack:0 pid:2476434 ppid:2476431 flags:0x00000002
[Mon Mar 11 14:29:16 2024] Call Trace:
[Mon Mar 11 14:29:16 2024] <TASK>
[Mon Mar 11 14:29:16 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:16 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:16 2024] schedule_hrtimeout_range_clock+0x10f/0x120
[Mon Mar 11 14:29:16 2024] ? n_tty_poll+0x1cb/0x1f0
[Mon Mar 11 14:29:16 2024] ? tty_poll+0x71/0xa0
[Mon Mar 11 14:29:16 2024] do_select+0x69e/0x7c0
[Mon Mar 11 14:29:16 2024] ? __pfx_pollwake+0x10/0x10
[Mon Mar 11 14:29:16 2024] ? __pfx_pollwake+0x10/0x10
[Mon Mar 11 14:29:16 2024] ? __pfx_pollwake+0x10/0x10
[Mon Mar 11 14:29:16 2024] ? drain_stock+0x5e/0x90
[Mon Mar 11 14:29:16 2024] ? refill_stock+0x7e/0x90
[Mon Mar 11 14:29:16 2024] ? cgroup_rstat_updated+0x42/0xd0
[Mon Mar 11 14:29:16 2024] ? __mod_memcg_lruvec_state+0x84/0xd0
[Mon Mar 11 14:29:16 2024] ? drain_obj_stock+0xb6/0x2a0
[Mon Mar 11 14:29:16 2024] ? __mod_memcg_state+0x63/0xb0
[Mon Mar 11 14:29:16 2024] ? memcg_slab_post_alloc_hook+0x181/0x250
[Mon Mar 11 14:29:16 2024] ? try_to_wake_up+0x64/0x5d0
[Mon Mar 11 14:29:16 2024] ? kmem_cache_alloc+0x17d/0x340
[Mon Mar 11 14:29:16 2024] ? complete_signal+0x107/0x300
[Mon Mar 11 14:29:16 2024] core_sys_select+0x1a0/0x3b0
[Mon Mar 11 14:29:16 2024] ? _copy_from_user+0x27/0x60
[Mon Mar 11 14:29:16 2024] do_pselect.constprop.0+0xca/0x170
[Mon Mar 11 14:29:16 2024] __x64_sys_pselect6+0x39/0x70
[Mon Mar 11 14:29:16 2024] do_syscall_64+0x59/0x90
[Mon Mar 11 14:29:16 2024] ? syscall_exit_to_user_mode+0x22/0x40
[Mon Mar 11 14:29:16 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:16 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:16 2024] ? __irq_exit_rcu+0x46/0xc0
[Mon Mar 11 14:29:16 2024] ? sysvec_apic_timer_interrupt+0x3c/0x90
[Mon Mar 11 14:29:16 2024] entry_SYSCALL_64_after_hwframe+0x72/0xdc
[Mon Mar 11 14:29:16 2024] RIP: 0033:0x7f6208944f64
[Mon Mar 11 14:29:16 2024] RSP: 002b:00007ffc2b4f3c40 EFLAGS: 00000246 ORIG_RAX: 000000000000010e
[Mon Mar 11 14:29:16 2024] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f6208944f64
[Mon Mar 11 14:29:16 2024] RDX: 000055abfbeb2c00 RSI: 000055abfbee1840 RDI: 000000000000000f
[Mon Mar 11 14:29:16 2024] RBP: 000055abfbee1840 R08: 0000000000000000 R09: 00007ffc2b4f3c80
[Mon Mar 11 14:29:16 2024] R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000000008
[Mon Mar 11 14:29:16 2024] R13: 000055abfbe962a0 R14: 000055abfbeb2c00 R15: 000055abfbe944d0
[Mon Mar 11 14:29:16 2024] </TASK>
[Mon Mar 11 14:29:16 2024] task:bash state:R running task stack:0 pid:2476436 ppid:2476434 flags:0x0000400e
[Mon Mar 11 14:29:16 2024] Call Trace:
[Mon Mar 11 14:29:16 2024] <TASK>
[Mon Mar 11 14:29:16 2024] sched_show_task.cold+0xca/0xe4
[Mon Mar 11 14:29:16 2024] show_state_filter+0x7f/0xe0
[Mon Mar 11 14:29:16 2024] sysrq_handle_showstate+0xc/0x20
[Mon Mar 11 14:29:16 2024] __handle_sysrq.cold+0x40/0x11f
[Mon Mar 11 14:29:16 2024] write_sysrq_trigger+0x24/0x40
[Mon Mar 11 14:29:16 2024] proc_reg_write+0x53/0xa0
[Mon Mar 11 14:29:16 2024] vfs_write+0xe4/0x410
[Mon Mar 11 14:29:16 2024] ? __audit_filter_op+0xa5/0xf0
[Mon Mar 11 14:29:16 2024] ? auditd_test_task+0x3c/0x50
[Mon Mar 11 14:29:16 2024] ksys_write+0x5f/0xe0
[Mon Mar 11 14:29:16 2024] do_syscall_64+0x59/0x90
[Mon Mar 11 14:29:16 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:16 2024] ? syscall_exit_work+0x103/0x130
[Mon Mar 11 14:29:16 2024] ? syscall_exit_to_user_mode+0x22/0x40
[Mon Mar 11 14:29:16 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:16 2024] ? __irq_exit_rcu+0x46/0xc0
[Mon Mar 11 14:29:16 2024] ? common_interrupt+0x43/0xa0
[Mon Mar 11 14:29:16 2024] entry_SYSCALL_64_after_hwframe+0x72/0xdc
[Mon Mar 11 14:29:16 2024] RIP: 0033:0x7f2ac893e927
[Mon Mar 11 14:29:16 2024] Code: 0b 00 f7 d8 64 89 02 48 c7 c0 ff ff ff ff eb b7 0f 1f 00 f3 0f 1e fa 64 8b 04 25 18 00 00 00 85 c0 75 10 b8 01 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 51 c3 48 83 ec 28 48 89 54 24 18 48 89 74 24
[Mon Mar 11 14:29:16 2024] RSP: 002b:00007fff07fc0038 EFLAGS: 00000246 ORIG_RAX: 0000000000000001
[Mon Mar 11 14:29:16 2024] RAX: ffffffffffffffda RBX: 0000000000000002 RCX: 00007f2ac893e927
[Mon Mar 11 14:29:16 2024] RDX: 0000000000000002 RSI: 000055fdc2dc7490 RDI: 0000000000000001
[Mon Mar 11 14:29:16 2024] RBP: 000055fdc2dc7490 R08: 0000000000000000 R09: 00007f2ac89b14e0
[Mon Mar 11 14:29:16 2024] R10: 00007f2ac89b13e0 R11: 0000000000000246 R12: 0000000000000002
[Mon Mar 11 14:29:16 2024] R13: 00007f2ac89fb780 R14: 0000000000000002 R15: 00007f2ac89f69e0
[Mon Mar 11 14:29:16 2024] </TASK>
[Mon Mar 11 14:29:16 2024] task:kworker/u98:2 state:D stack:0 pid:2476491 ppid:2 flags:0x00004000
[Mon Mar 11 14:29:16 2024] Workqueue: writeback wb_workfn (flush-253:96)
[Mon Mar 11 14:29:16 2024] Call Trace:
[Mon Mar 11 14:29:16 2024] <TASK>
[Mon Mar 11 14:29:16 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:16 2024] ? __pfx_wbt_cleanup_cb+0x10/0x10
[Mon Mar 11 14:29:16 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:16 2024] io_schedule+0x42/0x70
[Mon Mar 11 14:29:16 2024] rq_qos_wait+0xbb/0x130
[Mon Mar 11 14:29:16 2024] ? __pfx_rq_qos_wake_function+0x10/0x10
[Mon Mar 11 14:29:16 2024] ? __pfx_wbt_inflight_cb+0x10/0x10
[Mon Mar 11 14:29:16 2024] wbt_wait+0x9c/0x100
[Mon Mar 11 14:29:16 2024] __rq_qos_throttle+0x20/0x40
[Mon Mar 11 14:29:16 2024] blk_mq_submit_bio+0x183/0x580
[Mon Mar 11 14:29:16 2024] __submit_bio_noacct+0x7e/0x1e0
[Mon Mar 11 14:29:16 2024] submit_bh_wbc+0x115/0x140
[Mon Mar 11 14:29:16 2024] __block_write_full_page+0x217/0x500
[Mon Mar 11 14:29:16 2024] ? __pfx_end_buffer_async_write+0x10/0x10
[Mon Mar 11 14:29:16 2024] ? __pfx_blkdev_get_block+0x10/0x10
[Mon Mar 11 14:29:16 2024] __writepage+0x17/0x70
[Mon Mar 11 14:29:16 2024] write_cache_pages+0x179/0x4c0
[Mon Mar 11 14:29:16 2024] ? __pfx___writepage+0x10/0x10
[Mon Mar 11 14:29:16 2024] do_writepages+0x129/0x1d0
[Mon Mar 11 14:29:16 2024] ? find_busiest_group+0x11d/0x240
[Mon Mar 11 14:29:16 2024] __writeback_single_inode+0x41/0x270
[Mon Mar 11 14:29:16 2024] writeback_sb_inodes+0x209/0x4a0
[Mon Mar 11 14:29:16 2024] __writeback_inodes_wb+0x4c/0xe0
[Mon Mar 11 14:29:16 2024] wb_writeback+0x1d7/0x2d0
[Mon Mar 11 14:29:16 2024] wb_do_writeback+0x1d1/0x2b0
[Mon Mar 11 14:29:16 2024] wb_workfn+0x5e/0x290
[Mon Mar 11 14:29:16 2024] ? try_to_wake_up+0x3e2/0x5d0
[Mon Mar 11 14:29:16 2024] process_one_work+0x1e2/0x3b0
[Mon Mar 11 14:29:16 2024] ? __pfx_worker_thread+0x10/0x10
[Mon Mar 11 14:29:16 2024] worker_thread+0x50/0x3a0
[Mon Mar 11 14:29:16 2024] ? __pfx_worker_thread+0x10/0x10
[Mon Mar 11 14:29:16 2024] kthread+0xdd/0x100
[Mon Mar 11 14:29:16 2024] ? __pfx_kthread+0x10/0x10
[Mon Mar 11 14:29:16 2024] ret_from_fork+0x29/0x50
[Mon Mar 11 14:29:16 2024] </TASK>
[Mon Mar 11 14:29:16 2024] task:smbd[10.46.231. state:S stack:0 pid:2476746 ppid:4757 flags:0x00000002
[Mon Mar 11 14:29:16 2024] Call Trace:
[Mon Mar 11 14:29:16 2024] <TASK>
[Mon Mar 11 14:29:16 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:16 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:16 2024] schedule_hrtimeout_range_clock+0x95/0x120
[Mon Mar 11 14:29:16 2024] ? __pfx_hrtimer_wakeup+0x10/0x10
[Mon Mar 11 14:29:16 2024] ep_poll+0x348/0x3b0
[Mon Mar 11 14:29:16 2024] ? __pfx_autoremove_wake_function+0x10/0x10
[Mon Mar 11 14:29:16 2024] do_epoll_wait+0xb1/0xd0
[Mon Mar 11 14:29:16 2024] __x64_sys_epoll_wait+0x60/0x100
[Mon Mar 11 14:29:16 2024] do_syscall_64+0x59/0x90
[Mon Mar 11 14:29:16 2024] ? syscall_exit_to_user_mode+0x22/0x40
[Mon Mar 11 14:29:16 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:16 2024] entry_SYSCALL_64_after_hwframe+0x72/0xdc
[Mon Mar 11 14:29:16 2024] RIP: 0033:0x7fedb7d4e80a
[Mon Mar 11 14:29:16 2024] RSP: 002b:00007ffc284a9ad8 EFLAGS: 00000246 ORIG_RAX: 00000000000000e8
[Mon Mar 11 14:29:16 2024] RAX: ffffffffffffffda RBX: 000055f87dd397c0 RCX: 00007fedb7d4e80a
[Mon Mar 11 14:29:16 2024] RDX: 0000000000000001 RSI: 00007ffc284a9b0c RDI: 0000000000000005
[Mon Mar 11 14:29:16 2024] RBP: 000055f87dd2aa20 R08: 00000000000f41da R09: 0000000000000000
[Mon Mar 11 14:29:16 2024] R10: 000000000000ea60 R11: 0000000000000246 R12: 000055f87dd61190
[Mon Mar 11 14:29:16 2024] R13: 000000000000ea60 R14: 0000000000000026 R15: 000055f87dd5b760
[Mon Mar 11 14:29:16 2024] </TASK>
[Mon Mar 11 14:29:16 2024] task:kworker/22:0 state:I stack:0 pid:2476753 ppid:2 flags:0x00004000
[Mon Mar 11 14:29:16 2024] Call Trace:
[Mon Mar 11 14:29:16 2024] <TASK>
[Mon Mar 11 14:29:16 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:16 2024] ? __pfx_worker_thread+0x10/0x10
[Mon Mar 11 14:29:16 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:16 2024] worker_thread+0xbb/0x3a0
[Mon Mar 11 14:29:16 2024] ? __pfx_worker_thread+0x10/0x10
[Mon Mar 11 14:29:16 2024] kthread+0xdd/0x100
[Mon Mar 11 14:29:16 2024] ? __pfx_kthread+0x10/0x10
[Mon Mar 11 14:29:16 2024] ret_from_fork+0x29/0x50
[Mon Mar 11 14:29:16 2024] </TASK>
[Mon Mar 11 14:29:16 2024] task:kworker/47:0 state:I stack:0 pid:2476852 ppid:2 flags:0x00004000
[Mon Mar 11 14:29:16 2024] Call Trace:
[Mon Mar 11 14:29:16 2024] <TASK>
[Mon Mar 11 14:29:16 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:16 2024] ? __pfx_worker_thread+0x10/0x10
[Mon Mar 11 14:29:16 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:16 2024] worker_thread+0xbb/0x3a0
[Mon Mar 11 14:29:16 2024] ? __pfx_worker_thread+0x10/0x10
[Mon Mar 11 14:29:16 2024] kthread+0xdd/0x100
[Mon Mar 11 14:29:16 2024] ? __pfx_kthread+0x10/0x10
[Mon Mar 11 14:29:16 2024] ret_from_fork+0x29/0x50
[Mon Mar 11 14:29:16 2024] </TASK>
[Mon Mar 11 14:29:16 2024] task:kworker/44:1 state:I stack:0 pid:2476853 ppid:2 flags:0x00004000
[Mon Mar 11 14:29:16 2024] Workqueue: 0x0 (mm_percpu_wq)
[Mon Mar 11 14:29:16 2024] Call Trace:
[Mon Mar 11 14:29:16 2024] <TASK>
[Mon Mar 11 14:29:16 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:16 2024] ? __pfx_worker_thread+0x10/0x10
[Mon Mar 11 14:29:16 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:16 2024] worker_thread+0xbb/0x3a0
[Mon Mar 11 14:29:16 2024] ? __pfx_worker_thread+0x10/0x10
[Mon Mar 11 14:29:16 2024] kthread+0xdd/0x100
[Mon Mar 11 14:29:16 2024] ? __pfx_kthread+0x10/0x10
[Mon Mar 11 14:29:16 2024] ret_from_fork+0x29/0x50
[Mon Mar 11 14:29:16 2024] </TASK>
[Mon Mar 11 14:29:16 2024] task:kworker/37:2 state:I stack:0 pid:2476855 ppid:2 flags:0x00004000
[Mon Mar 11 14:29:16 2024] Workqueue: 0x0 (mm_percpu_wq)
[Mon Mar 11 14:29:16 2024] Call Trace:
[Mon Mar 11 14:29:16 2024] <TASK>
[Mon Mar 11 14:29:16 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:16 2024] ? __pfx_worker_thread+0x10/0x10
[Mon Mar 11 14:29:16 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:16 2024] worker_thread+0xbb/0x3a0
[Mon Mar 11 14:29:16 2024] ? __pfx_worker_thread+0x10/0x10
[Mon Mar 11 14:29:16 2024] kthread+0xdd/0x100
[Mon Mar 11 14:29:16 2024] ? __pfx_kthread+0x10/0x10
[Mon Mar 11 14:29:16 2024] ret_from_fork+0x29/0x50
[Mon Mar 11 14:29:16 2024] </TASK>
[Mon Mar 11 14:29:16 2024] task:rpcd_classic state:S stack:0 pid:2477044 ppid:2339807 flags:0x00000002
[Mon Mar 11 14:29:16 2024] Call Trace:
[Mon Mar 11 14:29:16 2024] <TASK>
[Mon Mar 11 14:29:16 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:16 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:16 2024] schedule_hrtimeout_range_clock+0x95/0x120
[Mon Mar 11 14:29:16 2024] ? __pfx_hrtimer_wakeup+0x10/0x10
[Mon Mar 11 14:29:16 2024] ep_poll+0x348/0x3b0
[Mon Mar 11 14:29:16 2024] ? __pfx_autoremove_wake_function+0x10/0x10
[Mon Mar 11 14:29:16 2024] do_epoll_wait+0xb1/0xd0
[Mon Mar 11 14:29:16 2024] __x64_sys_epoll_wait+0x60/0x100
[Mon Mar 11 14:29:16 2024] do_syscall_64+0x59/0x90
[Mon Mar 11 14:29:16 2024] ? ktime_get+0x35/0xa0
[Mon Mar 11 14:29:16 2024] ? clockevents_program_event+0x93/0x100
[Mon Mar 11 14:29:16 2024] ? hrtimer_interrupt+0x126/0x210
[Mon Mar 11 14:29:16 2024] ? sched_clock+0xc/0x30
[Mon Mar 11 14:29:16 2024] ? sched_clock_cpu+0x9/0xc0
[Mon Mar 11 14:29:16 2024] ? irqtime_account_irq+0x3c/0xb0
[Mon Mar 11 14:29:16 2024] ? __irq_exit_rcu+0x46/0xc0
[Mon Mar 11 14:29:16 2024] ? sysvec_apic_timer_interrupt+0x3c/0x90
[Mon Mar 11 14:29:16 2024] entry_SYSCALL_64_after_hwframe+0x72/0xdc
[Mon Mar 11 14:29:16 2024] RIP: 0033:0x7f4ea274e80a
[Mon Mar 11 14:29:16 2024] RSP: 002b:00007ffdaf1b6a28 EFLAGS: 00000246 ORIG_RAX: 00000000000000e8
[Mon Mar 11 14:29:16 2024] RAX: ffffffffffffffda RBX: 0000562818753770 RCX: 00007f4ea274e80a
[Mon Mar 11 14:29:16 2024] RDX: 0000000000000001 RSI: 00007ffdaf1b6a5c RDI: 0000000000000003
[Mon Mar 11 14:29:16 2024] RBP: 00005628187442e0 R08: 00000000000f4200 R09: 0000000000006cd4
[Mon Mar 11 14:29:16 2024] R10: 00000000000003e8 R11: 0000000000000246 R12: 0000562818753800
[Mon Mar 11 14:29:16 2024] R13: 00000000000003e8 R14: 00007f4ea3c6fdb0 R15: 0000000000000000
[Mon Mar 11 14:29:16 2024] </TASK>
[Mon Mar 11 14:29:16 2024] task:kworker/8:0 state:I stack:0 pid:2477090 ppid:2 flags:0x00004000
[Mon Mar 11 14:29:16 2024] Call Trace:
[Mon Mar 11 14:29:16 2024] <TASK>
[Mon Mar 11 14:29:16 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:16 2024] ? __pfx_worker_thread+0x10/0x10
[Mon Mar 11 14:29:16 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:16 2024] worker_thread+0xbb/0x3a0
[Mon Mar 11 14:29:16 2024] ? __pfx_worker_thread+0x10/0x10
[Mon Mar 11 14:29:16 2024] kthread+0xdd/0x100
[Mon Mar 11 14:29:16 2024] ? __pfx_kthread+0x10/0x10
[Mon Mar 11 14:29:16 2024] ret_from_fork+0x29/0x50
[Mon Mar 11 14:29:16 2024] </TASK>
[Mon Mar 11 14:29:16 2024] task:kworker/4:1 state:I stack:0 pid:2477133 ppid:2 flags:0x00004000
[Mon Mar 11 14:29:16 2024] Workqueue: 0x0 (rcu_gp)
[Mon Mar 11 14:29:16 2024] Call Trace:
[Mon Mar 11 14:29:16 2024] <TASK>
[Mon Mar 11 14:29:16 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:16 2024] ? __pfx_worker_thread+0x10/0x10
[Mon Mar 11 14:29:16 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:16 2024] worker_thread+0xbb/0x3a0
[Mon Mar 11 14:29:16 2024] ? __pfx_worker_thread+0x10/0x10
[Mon Mar 11 14:29:16 2024] kthread+0xdd/0x100
[Mon Mar 11 14:29:16 2024] ? __pfx_kthread+0x10/0x10
[Mon Mar 11 14:29:16 2024] ret_from_fork+0x29/0x50
[Mon Mar 11 14:29:16 2024] </TASK>
[Mon Mar 11 14:29:16 2024] task:kworker/2:0 state:I stack:0 pid:2477161 ppid:2 flags:0x00004000
[Mon Mar 11 14:29:16 2024] Call Trace:
[Mon Mar 11 14:29:16 2024] <TASK>
[Mon Mar 11 14:29:16 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:16 2024] ? __pfx_worker_thread+0x10/0x10
[Mon Mar 11 14:29:16 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:16 2024] worker_thread+0xbb/0x3a0
[Mon Mar 11 14:29:16 2024] ? __pfx_worker_thread+0x10/0x10
[Mon Mar 11 14:29:16 2024] kthread+0xdd/0x100
[Mon Mar 11 14:29:16 2024] ? __pfx_kthread+0x10/0x10
[Mon Mar 11 14:29:16 2024] ret_from_fork+0x29/0x50
[Mon Mar 11 14:29:16 2024] </TASK>
[Mon Mar 11 14:29:16 2024] task:kworker/23:2 state:I stack:0 pid:2477164 ppid:2 flags:0x00004000
[Mon Mar 11 14:29:16 2024] Call Trace:
[Mon Mar 11 14:29:16 2024] <TASK>
[Mon Mar 11 14:29:16 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:16 2024] ? __pfx_worker_thread+0x10/0x10
[Mon Mar 11 14:29:16 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:16 2024] worker_thread+0xbb/0x3a0
[Mon Mar 11 14:29:16 2024] ? __pfx_worker_thread+0x10/0x10
[Mon Mar 11 14:29:16 2024] kthread+0xdd/0x100
[Mon Mar 11 14:29:16 2024] ? __pfx_kthread+0x10/0x10
[Mon Mar 11 14:29:16 2024] ret_from_fork+0x29/0x50
[Mon Mar 11 14:29:16 2024] </TASK>
[Mon Mar 11 14:29:16 2024] task:kworker/18:0 state:I stack:0 pid:2477203 ppid:2 flags:0x00004000
[Mon Mar 11 14:29:16 2024] Call Trace:
[Mon Mar 11 14:29:16 2024] <TASK>
[Mon Mar 11 14:29:16 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:16 2024] ? __pfx_worker_thread+0x10/0x10
[Mon Mar 11 14:29:16 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:16 2024] worker_thread+0xbb/0x3a0
[Mon Mar 11 14:29:16 2024] ? __pfx_worker_thread+0x10/0x10
[Mon Mar 11 14:29:16 2024] kthread+0xdd/0x100
[Mon Mar 11 14:29:16 2024] ? __pfx_kthread+0x10/0x10
[Mon Mar 11 14:29:16 2024] ret_from_fork+0x29/0x50
[Mon Mar 11 14:29:16 2024] </TASK>
[Mon Mar 11 14:29:16 2024] task:kworker/11:2 state:I stack:0 pid:2477378 ppid:2 flags:0x00004000
[Mon Mar 11 14:29:16 2024] Call Trace:
[Mon Mar 11 14:29:16 2024] <TASK>
[Mon Mar 11 14:29:16 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:16 2024] ? __pfx_worker_thread+0x10/0x10
[Mon Mar 11 14:29:16 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:16 2024] worker_thread+0xbb/0x3a0
[Mon Mar 11 14:29:16 2024] ? __pfx_worker_thread+0x10/0x10
[Mon Mar 11 14:29:16 2024] kthread+0xdd/0x100
[Mon Mar 11 14:29:16 2024] ? __pfx_kthread+0x10/0x10
[Mon Mar 11 14:29:16 2024] ret_from_fork+0x29/0x50
[Mon Mar 11 14:29:16 2024] </TASK>
[Mon Mar 11 14:29:16 2024] task:rpcd_spoolss state:S stack:0 pid:2477388 ppid:2339807 flags:0x00000002
[Mon Mar 11 14:29:16 2024] Call Trace:
[Mon Mar 11 14:29:16 2024] <TASK>
[Mon Mar 11 14:29:16 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:16 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:16 2024] schedule_hrtimeout_range_clock+0x95/0x120
[Mon Mar 11 14:29:16 2024] ? __pfx_hrtimer_wakeup+0x10/0x10
[Mon Mar 11 14:29:16 2024] ep_poll+0x348/0x3b0
[Mon Mar 11 14:29:16 2024] ? __pfx_autoremove_wake_function+0x10/0x10
[Mon Mar 11 14:29:16 2024] do_epoll_wait+0xb1/0xd0
[Mon Mar 11 14:29:16 2024] __x64_sys_epoll_wait+0x60/0x100
[Mon Mar 11 14:29:16 2024] do_syscall_64+0x59/0x90
[Mon Mar 11 14:29:16 2024] ? exit_to_user_mode_loop+0xd0/0x130
[Mon Mar 11 14:29:16 2024] ? exit_to_user_mode_prepare+0xb6/0x100
[Mon Mar 11 14:29:16 2024] ? syscall_exit_to_user_mode+0x22/0x40
[Mon Mar 11 14:29:16 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:16 2024] ? exit_to_user_mode_prepare+0xb6/0x100
[Mon Mar 11 14:29:16 2024] ? syscall_exit_to_user_mode+0x22/0x40
[Mon Mar 11 14:29:16 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:16 2024] ? syscall_exit_to_user_mode+0x22/0x40
[Mon Mar 11 14:29:16 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:16 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:16 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:16 2024] ? sysvec_apic_timer_interrupt+0x3c/0x90
[Mon Mar 11 14:29:16 2024] entry_SYSCALL_64_after_hwframe+0x72/0xdc
[Mon Mar 11 14:29:16 2024] RIP: 0033:0x7f558274e80a
[Mon Mar 11 14:29:16 2024] RSP: 002b:00007ffcbcc91568 EFLAGS: 00000246 ORIG_RAX: 00000000000000e8
[Mon Mar 11 14:29:16 2024] RAX: ffffffffffffffda RBX: 000056172c69c770 RCX: 00007f558274e80a
[Mon Mar 11 14:29:16 2024] RDX: 0000000000000001 RSI: 00007ffcbcc9159c RDI: 0000000000000003
[Mon Mar 11 14:29:16 2024] RBP: 000056172c68d2e0 R08: 000056172c754ef0 R09: 0000000000006eec
[Mon Mar 11 14:29:16 2024] R10: 0000000000007530 R11: 0000000000000246 R12: 000056172c69c800
[Mon Mar 11 14:29:16 2024] R13: 0000000000007530 R14: 00007f5583850db0 R15: 0000000000000000
[Mon Mar 11 14:29:16 2024] </TASK>
[Mon Mar 11 14:29:16 2024] task:rpcd_spoolss state:S stack:0 pid:2477392 ppid:2339807 flags:0x00000002
[Mon Mar 11 14:29:16 2024] Call Trace:
[Mon Mar 11 14:29:16 2024] <TASK>
[Mon Mar 11 14:29:16 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:16 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:16 2024] schedule_hrtimeout_range_clock+0x95/0x120
[Mon Mar 11 14:29:16 2024] ? __pfx_hrtimer_wakeup+0x10/0x10
[Mon Mar 11 14:29:16 2024] ep_poll+0x348/0x3b0
[Mon Mar 11 14:29:16 2024] ? __pfx_autoremove_wake_function+0x10/0x10
[Mon Mar 11 14:29:16 2024] do_epoll_wait+0xb1/0xd0
[Mon Mar 11 14:29:16 2024] __x64_sys_epoll_wait+0x60/0x100
[Mon Mar 11 14:29:16 2024] do_syscall_64+0x59/0x90
[Mon Mar 11 14:29:16 2024] ? exc_page_fault+0x62/0x150
[Mon Mar 11 14:29:16 2024] entry_SYSCALL_64_after_hwframe+0x72/0xdc
[Mon Mar 11 14:29:16 2024] RIP: 0033:0x7fc5d954e80a
[Mon Mar 11 14:29:16 2024] RSP: 002b:00007ffc70429658 EFLAGS: 00000246 ORIG_RAX: 00000000000000e8
[Mon Mar 11 14:29:16 2024] RAX: ffffffffffffffda RBX: 00005635431b0770 RCX: 00007fc5d954e80a
[Mon Mar 11 14:29:16 2024] RDX: 0000000000000001 RSI: 00007ffc7042968c RDI: 0000000000000003
[Mon Mar 11 14:29:16 2024] RBP: 00005635431a12e0 R08: 00005635431b11d0 R09: 0000000000000000
[Mon Mar 11 14:29:16 2024] R10: 0000000000007530 R11: 0000000000000246 R12: 00005635431b0800
[Mon Mar 11 14:29:16 2024] R13: 0000000000007530 R14: 00007fc5da59adb0 R15: 0000000000000000
[Mon Mar 11 14:29:16 2024] </TASK>
[Mon Mar 11 14:29:16 2024] task:rpcd_spoolss state:S stack:0 pid:2477396 ppid:2339807 flags:0x00000002
[Mon Mar 11 14:29:16 2024] Call Trace:
[Mon Mar 11 14:29:16 2024] <TASK>
[Mon Mar 11 14:29:16 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:16 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:16 2024] schedule_hrtimeout_range_clock+0x95/0x120
[Mon Mar 11 14:29:16 2024] ? __pfx_hrtimer_wakeup+0x10/0x10
[Mon Mar 11 14:29:16 2024] ep_poll+0x348/0x3b0
[Mon Mar 11 14:29:16 2024] ? __pfx_autoremove_wake_function+0x10/0x10
[Mon Mar 11 14:29:16 2024] do_epoll_wait+0xb1/0xd0
[Mon Mar 11 14:29:16 2024] __x64_sys_epoll_wait+0x60/0x100
[Mon Mar 11 14:29:16 2024] do_syscall_64+0x59/0x90
[Mon Mar 11 14:29:16 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:16 2024] ? audit_reset_context.part.0.constprop.0+0x273/0x2e0
[Mon Mar 11 14:29:16 2024] ? syscall_exit_work+0x103/0x130
[Mon Mar 11 14:29:16 2024] ? syscall_exit_to_user_mode+0x22/0x40
[Mon Mar 11 14:29:16 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:16 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:16 2024] entry_SYSCALL_64_after_hwframe+0x72/0xdc
[Mon Mar 11 14:29:16 2024] RIP: 0033:0x7efc1514e80a
[Mon Mar 11 14:29:16 2024] RSP: 002b:00007ffee9480748 EFLAGS: 00000246 ORIG_RAX: 00000000000000e8
[Mon Mar 11 14:29:16 2024] RAX: ffffffffffffffda RBX: 0000561499450770 RCX: 00007efc1514e80a
[Mon Mar 11 14:29:16 2024] RDX: 0000000000000001 RSI: 00007ffee948077c RDI: 0000000000000003
[Mon Mar 11 14:29:16 2024] RBP: 00005614994412e0 R08: 00005614994511d0 R09: 0000000000000000
[Mon Mar 11 14:29:16 2024] R10: 0000000000007530 R11: 0000000000000246 R12: 0000561499450800
[Mon Mar 11 14:29:16 2024] R13: 0000000000007530 R14: 00007efc16237db0 R15: 0000000000000000
[Mon Mar 11 14:29:16 2024] </TASK>
[Mon Mar 11 14:29:16 2024] task:rpcd_spoolss state:S stack:0 pid:2477400 ppid:2339807 flags:0x00000002
[Mon Mar 11 14:29:16 2024] Call Trace:
[Mon Mar 11 14:29:16 2024] <TASK>
[Mon Mar 11 14:29:16 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:16 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:16 2024] schedule_hrtimeout_range_clock+0x95/0x120
[Mon Mar 11 14:29:16 2024] ? __pfx_hrtimer_wakeup+0x10/0x10
[Mon Mar 11 14:29:16 2024] ep_poll+0x348/0x3b0
[Mon Mar 11 14:29:16 2024] ? __pfx_autoremove_wake_function+0x10/0x10
[Mon Mar 11 14:29:16 2024] do_epoll_wait+0xb1/0xd0
[Mon Mar 11 14:29:16 2024] __x64_sys_epoll_wait+0x60/0x100
[Mon Mar 11 14:29:16 2024] do_syscall_64+0x59/0x90
[Mon Mar 11 14:29:16 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:16 2024] entry_SYSCALL_64_after_hwframe+0x72/0xdc
[Mon Mar 11 14:29:16 2024] RIP: 0033:0x7f37e8d4e80a
[Mon Mar 11 14:29:16 2024] RSP: 002b:00007ffcc5cf46a8 EFLAGS: 00000246 ORIG_RAX: 00000000000000e8
[Mon Mar 11 14:29:16 2024] RAX: ffffffffffffffda RBX: 000055f45dcdf770 RCX: 00007f37e8d4e80a
[Mon Mar 11 14:29:16 2024] RDX: 0000000000000001 RSI: 00007ffcc5cf46dc RDI: 0000000000000003
[Mon Mar 11 14:29:16 2024] RBP: 000055f45dcd02e0 R08: 000055f45dce01d0 R09: 0000000000000000
[Mon Mar 11 14:29:16 2024] R10: 0000000000007530 R11: 0000000000000246 R12: 000055f45dcdf800
[Mon Mar 11 14:29:16 2024] R13: 0000000000007530 R14: 00007f37e9d1fdb0 R15: 0000000000000000
[Mon Mar 11 14:29:16 2024] </TASK>
[Mon Mar 11 14:29:16 2024] task:rpcd_winreg state:S stack:0 pid:2477414 ppid:2339807 flags:0x00000002
[Mon Mar 11 14:29:16 2024] Call Trace:
[Mon Mar 11 14:29:16 2024] <TASK>
[Mon Mar 11 14:29:16 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:16 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:16 2024] schedule_hrtimeout_range_clock+0x95/0x120
[Mon Mar 11 14:29:16 2024] ? __pfx_hrtimer_wakeup+0x10/0x10
[Mon Mar 11 14:29:16 2024] ep_poll+0x348/0x3b0
[Mon Mar 11 14:29:16 2024] ? __pfx_autoremove_wake_function+0x10/0x10
[Mon Mar 11 14:29:16 2024] do_epoll_wait+0xb1/0xd0
[Mon Mar 11 14:29:16 2024] __x64_sys_epoll_wait+0x60/0x100
[Mon Mar 11 14:29:16 2024] do_syscall_64+0x59/0x90
[Mon Mar 11 14:29:16 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:16 2024] entry_SYSCALL_64_after_hwframe+0x72/0xdc
[Mon Mar 11 14:29:16 2024] RIP: 0033:0x7efff154e80a
[Mon Mar 11 14:29:16 2024] RSP: 002b:00007fff50d7d268 EFLAGS: 00000246 ORIG_RAX: 00000000000000e8
[Mon Mar 11 14:29:16 2024] RAX: ffffffffffffffda RBX: 000055b21beab770 RCX: 00007efff154e80a
[Mon Mar 11 14:29:16 2024] RDX: 0000000000000001 RSI: 00007fff50d7d29c RDI: 0000000000000003
[Mon Mar 11 14:29:16 2024] RBP: 000055b21be9c2e0 R08: 000055b21bee97a0 R09: 000000000000145b
[Mon Mar 11 14:29:16 2024] R10: 0000000000007530 R11: 0000000000000246 R12: 000055b21beab800
[Mon Mar 11 14:29:16 2024] R13: 0000000000007530 R14: 00007efff2015db0 R15: 0000000000000000
[Mon Mar 11 14:29:16 2024] </TASK>
[Mon Mar 11 14:29:16 2024] task:kworker/26:0 state:I stack:0 pid:2477469 ppid:2 flags:0x00004000
[Mon Mar 11 14:29:16 2024] Workqueue: 0x0 (mm_percpu_wq)
[Mon Mar 11 14:29:16 2024] Call Trace:
[Mon Mar 11 14:29:16 2024] <TASK>
[Mon Mar 11 14:29:16 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:16 2024] ? __pfx_worker_thread+0x10/0x10
[Mon Mar 11 14:29:16 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:16 2024] worker_thread+0xbb/0x3a0
[Mon Mar 11 14:29:16 2024] ? __pfx_worker_thread+0x10/0x10
[Mon Mar 11 14:29:16 2024] kthread+0xdd/0x100
[Mon Mar 11 14:29:16 2024] ? __pfx_kthread+0x10/0x10
[Mon Mar 11 14:29:16 2024] ret_from_fork+0x29/0x50
[Mon Mar 11 14:29:16 2024] </TASK>
[Mon Mar 11 14:29:16 2024] task:kworker/28:1 state:I stack:0 pid:2477534 ppid:2 flags:0x00004000
[Mon Mar 11 14:29:16 2024] Workqueue: 0x0 (events)
[Mon Mar 11 14:29:16 2024] Call Trace:
[Mon Mar 11 14:29:16 2024] <TASK>
[Mon Mar 11 14:29:16 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:16 2024] ? __pfx_worker_thread+0x10/0x10
[Mon Mar 11 14:29:16 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:16 2024] worker_thread+0xbb/0x3a0
[Mon Mar 11 14:29:16 2024] ? __pfx_worker_thread+0x10/0x10
[Mon Mar 11 14:29:16 2024] kthread+0xdd/0x100
[Mon Mar 11 14:29:16 2024] ? __pfx_kthread+0x10/0x10
[Mon Mar 11 14:29:16 2024] ret_from_fork+0x29/0x50
[Mon Mar 11 14:29:16 2024] </TASK>
[Mon Mar 11 14:29:16 2024] task:kworker/42:1 state:I stack:0 pid:2477540 ppid:2 flags:0x00004000
[Mon Mar 11 14:29:16 2024] Workqueue: 0x0 (mm_percpu_wq)
[Mon Mar 11 14:29:16 2024] Call Trace:
[Mon Mar 11 14:29:16 2024] <TASK>
[Mon Mar 11 14:29:16 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:16 2024] ? __pfx_worker_thread+0x10/0x10
[Mon Mar 11 14:29:16 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:16 2024] worker_thread+0xbb/0x3a0
[Mon Mar 11 14:29:16 2024] ? __pfx_worker_thread+0x10/0x10
[Mon Mar 11 14:29:16 2024] kthread+0xdd/0x100
[Mon Mar 11 14:29:16 2024] ? __pfx_kthread+0x10/0x10
[Mon Mar 11 14:29:16 2024] ret_from_fork+0x29/0x50
[Mon Mar 11 14:29:16 2024] </TASK>
[Mon Mar 11 14:29:16 2024] task:kworker/14:0 state:I stack:0 pid:2477542 ppid:2 flags:0x00004000
[Mon Mar 11 14:29:16 2024] Call Trace:
[Mon Mar 11 14:29:16 2024] <TASK>
[Mon Mar 11 14:29:16 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:16 2024] ? __pfx_worker_thread+0x10/0x10
[Mon Mar 11 14:29:16 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:16 2024] worker_thread+0xbb/0x3a0
[Mon Mar 11 14:29:16 2024] ? __pfx_worker_thread+0x10/0x10
[Mon Mar 11 14:29:16 2024] kthread+0xdd/0x100
[Mon Mar 11 14:29:16 2024] ? __pfx_kthread+0x10/0x10
[Mon Mar 11 14:29:16 2024] ret_from_fork+0x29/0x50
[Mon Mar 11 14:29:16 2024] </TASK>
[Mon Mar 11 14:29:16 2024] task:kworker/33:0 state:I stack:0 pid:2477675 ppid:2 flags:0x00004000
[Mon Mar 11 14:29:16 2024] Call Trace:
[Mon Mar 11 14:29:16 2024] <TASK>
[Mon Mar 11 14:29:16 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:16 2024] ? __pfx_worker_thread+0x10/0x10
[Mon Mar 11 14:29:16 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:16 2024] worker_thread+0xbb/0x3a0
[Mon Mar 11 14:29:16 2024] ? __pfx_worker_thread+0x10/0x10
[Mon Mar 11 14:29:16 2024] kthread+0xdd/0x100
[Mon Mar 11 14:29:16 2024] ? __pfx_kthread+0x10/0x10
[Mon Mar 11 14:29:16 2024] ret_from_fork+0x29/0x50
[Mon Mar 11 14:29:16 2024] </TASK>
[Mon Mar 11 14:29:16 2024] task:kworker/3:0 state:I stack:0 pid:2477683 ppid:2 flags:0x00004000
[Mon Mar 11 14:29:16 2024] Workqueue: 0x0 (lpfc_wq)
[Mon Mar 11 14:29:16 2024] Call Trace:
[Mon Mar 11 14:29:16 2024] <TASK>
[Mon Mar 11 14:29:16 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:16 2024] ? __pfx_worker_thread+0x10/0x10
[Mon Mar 11 14:29:16 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:16 2024] worker_thread+0xbb/0x3a0
[Mon Mar 11 14:29:16 2024] ? __pfx_worker_thread+0x10/0x10
[Mon Mar 11 14:29:16 2024] kthread+0xdd/0x100
[Mon Mar 11 14:29:16 2024] ? __pfx_kthread+0x10/0x10
[Mon Mar 11 14:29:16 2024] ret_from_fork+0x29/0x50
[Mon Mar 11 14:29:16 2024] </TASK>
[Mon Mar 11 14:29:16 2024] task:kworker/u97:5 state:I stack:0 pid:2477730 ppid:2 flags:0x00004000
[Mon Mar 11 14:29:16 2024] Workqueue: 0x0 (writeback)
[Mon Mar 11 14:29:16 2024] Call Trace:
[Mon Mar 11 14:29:16 2024] <TASK>
[Mon Mar 11 14:29:16 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:16 2024] ? __pfx_worker_thread+0x10/0x10
[Mon Mar 11 14:29:16 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:16 2024] worker_thread+0xbb/0x3a0
[Mon Mar 11 14:29:16 2024] ? __pfx_worker_thread+0x10/0x10
[Mon Mar 11 14:29:16 2024] kthread+0xdd/0x100
[Mon Mar 11 14:29:16 2024] ? __pfx_kthread+0x10/0x10
[Mon Mar 11 14:29:16 2024] ret_from_fork+0x29/0x50
[Mon Mar 11 14:29:16 2024] </TASK>
[Mon Mar 11 14:29:16 2024] task:kworker/41:2 state:I stack:0 pid:2477809 ppid:2 flags:0x00004000
[Mon Mar 11 14:29:16 2024] Call Trace:
[Mon Mar 11 14:29:16 2024] <TASK>
[Mon Mar 11 14:29:16 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:16 2024] ? __pfx_worker_thread+0x10/0x10
[Mon Mar 11 14:29:16 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:16 2024] worker_thread+0xbb/0x3a0
[Mon Mar 11 14:29:16 2024] ? __pfx_worker_thread+0x10/0x10
[Mon Mar 11 14:29:16 2024] kthread+0xdd/0x100
[Mon Mar 11 14:29:16 2024] ? __pfx_kthread+0x10/0x10
[Mon Mar 11 14:29:16 2024] ret_from_fork+0x29/0x50
[Mon Mar 11 14:29:16 2024] </TASK>
[Mon Mar 11 14:29:16 2024] task:sleep state:S stack:0 pid:2478153 ppid:91766 flags:0x00000002
[Mon Mar 11 14:29:16 2024] Call Trace:
[Mon Mar 11 14:29:16 2024] <TASK>
[Mon Mar 11 14:29:16 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:16 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:16 2024] do_nanosleep+0x67/0x190
[Mon Mar 11 14:29:16 2024] hrtimer_nanosleep+0xbe/0x1a0
[Mon Mar 11 14:29:16 2024] ? __pfx_hrtimer_wakeup+0x10/0x10
[Mon Mar 11 14:29:16 2024] common_nsleep+0x40/0x50
[Mon Mar 11 14:29:16 2024] __x64_sys_clock_nanosleep+0xbc/0x130
[Mon Mar 11 14:29:16 2024] do_syscall_64+0x59/0x90
[Mon Mar 11 14:29:16 2024] ? handle_mm_fault+0xcd/0x290
[Mon Mar 11 14:29:16 2024] ? do_user_addr_fault+0x1d6/0x6a0
[Mon Mar 11 14:29:16 2024] ? exc_page_fault+0x62/0x150
[Mon Mar 11 14:29:16 2024] entry_SYSCALL_64_after_hwframe+0x72/0xdc
[Mon Mar 11 14:29:16 2024] RIP: 0033:0x7f4c8f71393a
[Mon Mar 11 14:29:16 2024] RSP: 002b:00007ffea37e54f8 EFLAGS: 00000246 ORIG_RAX: 00000000000000e6
[Mon Mar 11 14:29:16 2024] RAX: ffffffffffffffda RBX: 00007f4c8f8926c0 RCX: 00007f4c8f71393a
[Mon Mar 11 14:29:16 2024] RDX: 00007ffea37e5550 RSI: 0000000000000000 RDI: 0000000000000000
[Mon Mar 11 14:29:16 2024] RBP: 000000000000003c R08: 0000000000000000 R09: 0000000000000000
[Mon Mar 11 14:29:16 2024] R10: 00007ffea37e5540 R11: 0000000000000246 R12: 00007ffea37e5540
[Mon Mar 11 14:29:16 2024] R13: 00007ffea37e5550 R14: 00007ffea37e56c8 R15: 000055722ebd3040
[Mon Mar 11 14:29:16 2024] </TASK>
[Mon Mar 11 14:29:16 2024] task:smbd[10.40.128. state:S stack:0 pid:2478178 ppid:4757 flags:0x00000002
[Mon Mar 11 14:29:16 2024] Call Trace:
[Mon Mar 11 14:29:16 2024] <TASK>
[Mon Mar 11 14:29:16 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:16 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:16 2024] schedule_hrtimeout_range_clock+0x95/0x120
[Mon Mar 11 14:29:16 2024] ? __pfx_hrtimer_wakeup+0x10/0x10
[Mon Mar 11 14:29:16 2024] ep_poll+0x348/0x3b0
[Mon Mar 11 14:29:16 2024] ? __pfx_autoremove_wake_function+0x10/0x10
[Mon Mar 11 14:29:16 2024] do_epoll_wait+0xb1/0xd0
[Mon Mar 11 14:29:16 2024] __x64_sys_epoll_wait+0x60/0x100
[Mon Mar 11 14:29:16 2024] do_syscall_64+0x59/0x90
[Mon Mar 11 14:29:16 2024] ? common_interrupt+0x43/0xa0
[Mon Mar 11 14:29:16 2024] entry_SYSCALL_64_after_hwframe+0x72/0xdc
[Mon Mar 11 14:29:16 2024] RIP: 0033:0x7fedb7d4e84e
[Mon Mar 11 14:29:16 2024] RSP: 002b:00007ffc284a9ab0 EFLAGS: 00000293 ORIG_RAX: 00000000000000e8
[Mon Mar 11 14:29:16 2024] RAX: ffffffffffffffda RBX: 000055f87dd397c0 RCX: 00007fedb7d4e84e
[Mon Mar 11 14:29:16 2024] RDX: 0000000000000001 RSI: 00007ffc284a9b0c RDI: 0000000000000005
[Mon Mar 11 14:29:16 2024] RBP: 000055f87dd2aa20 R08: 0000000000000000 R09: 0000000000000000
[Mon Mar 11 14:29:16 2024] R10: 000000000000cada R11: 0000000000000293 R12: 000055f87dd58980
[Mon Mar 11 14:29:16 2024] R13: 000000000000cada R14: 0000000000000027 R15: 000055f87dd5b760
[Mon Mar 11 14:29:16 2024] </TASK>
[Mon Mar 11 14:29:16 2024] task:kworker/39:1 state:I stack:0 pid:2478206 ppid:2 flags:0x00004000
[Mon Mar 11 14:29:16 2024] Call Trace:
[Mon Mar 11 14:29:16 2024] <TASK>
[Mon Mar 11 14:29:16 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:16 2024] ? __pfx_worker_thread+0x10/0x10
[Mon Mar 11 14:29:16 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:16 2024] worker_thread+0xbb/0x3a0
[Mon Mar 11 14:29:16 2024] ? __pfx_worker_thread+0x10/0x10
[Mon Mar 11 14:29:16 2024] kthread+0xdd/0x100
[Mon Mar 11 14:29:16 2024] ? __pfx_kthread+0x10/0x10
[Mon Mar 11 14:29:16 2024] ret_from_fork+0x29/0x50
[Mon Mar 11 14:29:16 2024] </TASK>
[Mon Mar 11 14:29:16 2024] task:kworker/40:2 state:I stack:0 pid:2478207 ppid:2 flags:0x00004000
[Mon Mar 11 14:29:16 2024] Workqueue: 0x0 (events)
[Mon Mar 11 14:29:16 2024] Call Trace:
[Mon Mar 11 14:29:16 2024] <TASK>
[Mon Mar 11 14:29:16 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:16 2024] ? __pfx_worker_thread+0x10/0x10
[Mon Mar 11 14:29:16 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:16 2024] worker_thread+0xbb/0x3a0
[Mon Mar 11 14:29:16 2024] ? __pfx_worker_thread+0x10/0x10
[Mon Mar 11 14:29:16 2024] kthread+0xdd/0x100
[Mon Mar 11 14:29:16 2024] ? __pfx_kthread+0x10/0x10
[Mon Mar 11 14:29:16 2024] ret_from_fork+0x29/0x50
[Mon Mar 11 14:29:16 2024] </TASK>
[Mon Mar 11 14:29:16 2024] task:rpcd_spoolss state:S stack:0 pid:2478220 ppid:2339807 flags:0x00000002
[Mon Mar 11 14:29:16 2024] Call Trace:
[Mon Mar 11 14:29:16 2024] <TASK>
[Mon Mar 11 14:29:16 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:16 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:16 2024] schedule_hrtimeout_range_clock+0x95/0x120
[Mon Mar 11 14:29:16 2024] ? __pfx_hrtimer_wakeup+0x10/0x10
[Mon Mar 11 14:29:16 2024] ep_poll+0x348/0x3b0
[Mon Mar 11 14:29:16 2024] ? __pfx_autoremove_wake_function+0x10/0x10
[Mon Mar 11 14:29:16 2024] do_epoll_wait+0xb1/0xd0
[Mon Mar 11 14:29:16 2024] __x64_sys_epoll_wait+0x60/0x100
[Mon Mar 11 14:29:16 2024] do_syscall_64+0x59/0x90
[Mon Mar 11 14:29:16 2024] ? common_interrupt+0x43/0xa0
[Mon Mar 11 14:29:16 2024] entry_SYSCALL_64_after_hwframe+0x72/0xdc
[Mon Mar 11 14:29:16 2024] RIP: 0033:0x7f4471b4e80a
[Mon Mar 11 14:29:16 2024] RSP: 002b:00007ffe40c13e78 EFLAGS: 00000246 ORIG_RAX: 00000000000000e8
[Mon Mar 11 14:29:16 2024] RAX: ffffffffffffffda RBX: 000055e6e8026770 RCX: 00007f4471b4e80a
[Mon Mar 11 14:29:16 2024] RDX: 0000000000000001 RSI: 00007ffe40c13eac RDI: 0000000000000003
[Mon Mar 11 14:29:16 2024] RBP: 000055e6e80172e0 R08: 000055e6e8083a50 R09: 000000000000119c
[Mon Mar 11 14:29:16 2024] R10: 0000000000007530 R11: 0000000000000246 R12: 000055e6e8026800
[Mon Mar 11 14:29:16 2024] R13: 0000000000007530 R14: 00007f4472ce0db0 R15: 0000000000000000
[Mon Mar 11 14:29:16 2024] </TASK>
[Mon Mar 11 14:29:16 2024] task:rpcd_spoolss state:S stack:0 pid:2478222 ppid:2339807 flags:0x00000002
[Mon Mar 11 14:29:16 2024] Call Trace:
[Mon Mar 11 14:29:16 2024] <TASK>
[Mon Mar 11 14:29:16 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:16 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:16 2024] schedule_hrtimeout_range_clock+0x95/0x120
[Mon Mar 11 14:29:16 2024] ? __pfx_hrtimer_wakeup+0x10/0x10
[Mon Mar 11 14:29:16 2024] ep_poll+0x348/0x3b0
[Mon Mar 11 14:29:16 2024] ? __pfx_autoremove_wake_function+0x10/0x10
[Mon Mar 11 14:29:16 2024] do_epoll_wait+0xb1/0xd0
[Mon Mar 11 14:29:16 2024] __x64_sys_epoll_wait+0x60/0x100
[Mon Mar 11 14:29:16 2024] do_syscall_64+0x59/0x90
[Mon Mar 11 14:29:16 2024] ? syscall_exit_to_user_mode+0x22/0x40
[Mon Mar 11 14:29:16 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:16 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:16 2024] ? exc_page_fault+0x62/0x150
[Mon Mar 11 14:29:16 2024] entry_SYSCALL_64_after_hwframe+0x72/0xdc
[Mon Mar 11 14:29:16 2024] RIP: 0033:0x7fd76154e80a
[Mon Mar 11 14:29:16 2024] RSP: 002b:00007ffcedfd9bc8 EFLAGS: 00000246 ORIG_RAX: 00000000000000e8
[Mon Mar 11 14:29:16 2024] RAX: ffffffffffffffda RBX: 0000556208bf4770 RCX: 00007fd76154e80a
[Mon Mar 11 14:29:16 2024] RDX: 0000000000000001 RSI: 00007ffcedfd9bfc RDI: 0000000000000003
[Mon Mar 11 14:29:16 2024] RBP: 0000556208be52e0 R08: 0000556208bf51d0 R09: 00007ffcedfd97c0
[Mon Mar 11 14:29:16 2024] R10: 0000000000007530 R11: 0000000000000246 R12: 0000556208bf4800
[Mon Mar 11 14:29:16 2024] R13: 0000000000007530 R14: 00007fd76267adb0 R15: 0000000000000000
[Mon Mar 11 14:29:16 2024] </TASK>
[Mon Mar 11 14:29:16 2024] task:rpcd_spoolss state:S stack:0 pid:2478223 ppid:2339807 flags:0x00000002
[Mon Mar 11 14:29:16 2024] Call Trace:
[Mon Mar 11 14:29:16 2024] <TASK>
[Mon Mar 11 14:29:16 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:16 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:16 2024] schedule_hrtimeout_range_clock+0x95/0x120
[Mon Mar 11 14:29:16 2024] ? __pfx_hrtimer_wakeup+0x10/0x10
[Mon Mar 11 14:29:16 2024] ep_poll+0x348/0x3b0
[Mon Mar 11 14:29:16 2024] ? __pfx_autoremove_wake_function+0x10/0x10
[Mon Mar 11 14:29:16 2024] do_epoll_wait+0xb1/0xd0
[Mon Mar 11 14:29:16 2024] __x64_sys_epoll_wait+0x60/0x100
[Mon Mar 11 14:29:16 2024] do_syscall_64+0x59/0x90
[Mon Mar 11 14:29:16 2024] entry_SYSCALL_64_after_hwframe+0x72/0xdc
[Mon Mar 11 14:29:16 2024] RIP: 0033:0x7f827294e80a
[Mon Mar 11 14:29:16 2024] RSP: 002b:00007ffdf1f6dce8 EFLAGS: 00000246 ORIG_RAX: 00000000000000e8
[Mon Mar 11 14:29:16 2024] RAX: ffffffffffffffda RBX: 000056414398a770 RCX: 00007f827294e80a
[Mon Mar 11 14:29:16 2024] RDX: 0000000000000001 RSI: 00007ffdf1f6dd1c RDI: 0000000000000003
[Mon Mar 11 14:29:16 2024] RBP: 000056414397b2e0 R08: 000056414398b1d0 R09: 00007ffdf1f6d8e0
[Mon Mar 11 14:29:16 2024] R10: 0000000000007530 R11: 0000000000000246 R12: 000056414398a800
[Mon Mar 11 14:29:16 2024] R13: 0000000000007530 R14: 00007f8273a41db0 R15: 0000000000000000
[Mon Mar 11 14:29:16 2024] </TASK>
[Mon Mar 11 14:29:16 2024] task:rpcd_spoolss state:S stack:0 pid:2478224 ppid:2339807 flags:0x00000002
[Mon Mar 11 14:29:16 2024] Call Trace:
[Mon Mar 11 14:29:16 2024] <TASK>
[Mon Mar 11 14:29:16 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:16 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:16 2024] schedule_hrtimeout_range_clock+0x95/0x120
[Mon Mar 11 14:29:16 2024] ? __pfx_hrtimer_wakeup+0x10/0x10
[Mon Mar 11 14:29:16 2024] ep_poll+0x348/0x3b0
[Mon Mar 11 14:29:16 2024] ? __pfx_autoremove_wake_function+0x10/0x10
[Mon Mar 11 14:29:16 2024] do_epoll_wait+0xb1/0xd0
[Mon Mar 11 14:29:16 2024] __x64_sys_epoll_wait+0x60/0x100
[Mon Mar 11 14:29:16 2024] do_syscall_64+0x59/0x90
[Mon Mar 11 14:29:16 2024] ? exit_to_user_mode_prepare+0xec/0x100
[Mon Mar 11 14:29:16 2024] ? syscall_exit_to_user_mode+0x22/0x40
[Mon Mar 11 14:29:16 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:16 2024] ? syscall_exit_to_user_mode+0x22/0x40
[Mon Mar 11 14:29:16 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:16 2024] entry_SYSCALL_64_after_hwframe+0x72/0xdc
[Mon Mar 11 14:29:16 2024] RIP: 0033:0x7fd74f14e80a
[Mon Mar 11 14:29:16 2024] RSP: 002b:00007ffc29aada38 EFLAGS: 00000246 ORIG_RAX: 00000000000000e8
[Mon Mar 11 14:29:16 2024] RAX: ffffffffffffffda RBX: 000055658895b770 RCX: 00007fd74f14e80a
[Mon Mar 11 14:29:16 2024] RDX: 0000000000000001 RSI: 00007ffc29aada6c RDI: 0000000000000003
[Mon Mar 11 14:29:16 2024] RBP: 000055658894c2e0 R08: 000055658895c1d0 R09: 00007ffc29aad630
[Mon Mar 11 14:29:16 2024] R10: 0000000000007530 R11: 0000000000000246 R12: 000055658895b800
[Mon Mar 11 14:29:16 2024] R13: 0000000000007530 R14: 00007fd750266db0 R15: 0000000000000000
[Mon Mar 11 14:29:16 2024] </TASK>
[Mon Mar 11 14:29:16 2024] task:nrpe state:S stack:0 pid:2478244 ppid:1 flags:0x00000002
[Mon Mar 11 14:29:16 2024] Call Trace:
[Mon Mar 11 14:29:16 2024] <TASK>
[Mon Mar 11 14:29:16 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:16 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:16 2024] do_wait+0x15b/0x2f0
[Mon Mar 11 14:29:16 2024] kernel_wait4+0xa6/0x140
[Mon Mar 11 14:29:16 2024] ? __pfx_child_wait_callback+0x10/0x10
[Mon Mar 11 14:29:16 2024] do_syscall_64+0x59/0x90
[Mon Mar 11 14:29:16 2024] ? __audit_filter_op+0xa5/0xf0
[Mon Mar 11 14:29:16 2024] ? audit_reset_context.part.0.constprop.0+0x273/0x2e0
[Mon Mar 11 14:29:16 2024] ? syscall_exit_work+0x103/0x130
[Mon Mar 11 14:29:16 2024] ? syscall_exit_to_user_mode+0x22/0x40
[Mon Mar 11 14:29:16 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:16 2024] ? exc_page_fault+0x62/0x150
[Mon Mar 11 14:29:16 2024] entry_SYSCALL_64_after_hwframe+0x72/0xdc
[Mon Mar 11 14:29:16 2024] RIP: 0033:0x7fb70e3182ca
[Mon Mar 11 14:29:16 2024] RSP: 002b:00007ffcc320ed78 EFLAGS: 00000246 ORIG_RAX: 000000000000003d
[Mon Mar 11 14:29:16 2024] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007fb70e3182ca
[Mon Mar 11 14:29:16 2024] RDX: 0000000000000000 RSI: 00007ffcc320edc4 RDI: 000000000025d0a5
[Mon Mar 11 14:29:16 2024] RBP: 0000562bfde0c5e0 R08: 000000000025d0a5 R09: 000000000025d0a5
[Mon Mar 11 14:29:16 2024] R10: 0000000000000000 R11: 0000000000000246 R12: 00007ffcc320edd0
[Mon Mar 11 14:29:16 2024] R13: 0000000000000000 R14: 0000000000000006 R15: 0000562bfde1bac0
[Mon Mar 11 14:29:16 2024] </TASK>
[Mon Mar 11 14:29:16 2024] task:nrpe state:S stack:0 pid:2478245 ppid:2478244 flags:0x00000002
[Mon Mar 11 14:29:16 2024] Call Trace:
[Mon Mar 11 14:29:16 2024] <TASK>
[Mon Mar 11 14:29:16 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:16 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:16 2024] pipe_read+0x38b/0x4c0
[Mon Mar 11 14:29:16 2024] ? __pfx_autoremove_wake_function+0x10/0x10
[Mon Mar 11 14:29:16 2024] vfs_read+0x2f9/0x330
[Mon Mar 11 14:29:16 2024] ksys_read+0xab/0xe0
[Mon Mar 11 14:29:16 2024] ? syscall_trace_enter.constprop.0+0x126/0x1a0
[Mon Mar 11 14:29:16 2024] do_syscall_64+0x59/0x90
[Mon Mar 11 14:29:16 2024] ? syscall_exit_to_user_mode+0x22/0x40
[Mon Mar 11 14:29:16 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:16 2024] ? do_user_addr_fault+0x1d6/0x6a0
[Mon Mar 11 14:29:16 2024] ? exc_page_fault+0x62/0x150
[Mon Mar 11 14:29:16 2024] entry_SYSCALL_64_after_hwframe+0x72/0xdc
[Mon Mar 11 14:29:16 2024] RIP: 0033:0x7fb70e33e882
[Mon Mar 11 14:29:16 2024] RSP: 002b:00007ffcc320eca8 EFLAGS: 00000246 ORIG_RAX: 0000000000000000
[Mon Mar 11 14:29:16 2024] RAX: ffffffffffffffda RBX: 0000562bfde21250 RCX: 00007fb70e33e882
[Mon Mar 11 14:29:16 2024] RDX: 0000000000001000 RSI: 0000562bfde1bee0 RDI: 0000000000000004
[Mon Mar 11 14:29:16 2024] RBP: 00007fb70e3f6c20 R08: 0000000000000004 R09: 0000000000000000
[Mon Mar 11 14:29:16 2024] R10: 0000000000000200 R11: 0000000000000246 R12: 00000000000007ff
[Mon Mar 11 14:29:16 2024] R13: 0000000000000d68 R14: 00007fb70e3f69e0 R15: 0000000000000d68
[Mon Mar 11 14:29:16 2024] </TASK>
[Mon Mar 11 14:29:16 2024] task:check_fan_statu state:S stack:0 pid:2478246 ppid:2478245 flags:0x00000002
[Mon Mar 11 14:29:16 2024] Call Trace:
[Mon Mar 11 14:29:16 2024] <TASK>
[Mon Mar 11 14:29:16 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:16 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:16 2024] pipe_read+0x38b/0x4c0
[Mon Mar 11 14:29:16 2024] ? __pfx_autoremove_wake_function+0x10/0x10
[Mon Mar 11 14:29:16 2024] vfs_read+0x2f9/0x330
[Mon Mar 11 14:29:16 2024] ksys_read+0xab/0xe0
[Mon Mar 11 14:29:16 2024] ? syscall_trace_enter.constprop.0+0x126/0x1a0
[Mon Mar 11 14:29:16 2024] do_syscall_64+0x59/0x90
[Mon Mar 11 14:29:16 2024] ? do_user_addr_fault+0x1d6/0x6a0
[Mon Mar 11 14:29:16 2024] ? exc_page_fault+0x62/0x150
[Mon Mar 11 14:29:16 2024] entry_SYSCALL_64_after_hwframe+0x72/0xdc
[Mon Mar 11 14:29:16 2024] RIP: 0033:0x7f2e4753e882
[Mon Mar 11 14:29:16 2024] RSP: 002b:00007ffead72a018 EFLAGS: 00000246 ORIG_RAX: 0000000000000000
[Mon Mar 11 14:29:16 2024] RAX: ffffffffffffffda RBX: 0000000000002000 RCX: 00007f2e4753e882
[Mon Mar 11 14:29:16 2024] RDX: 0000000000002000 RSI: 000055887f375a60 RDI: 0000000000000003
[Mon Mar 11 14:29:16 2024] RBP: 000055887f375a60 R08: 0000000000000000 R09: 0000000000000000
[Mon Mar 11 14:29:16 2024] R10: 0000000000000100 R11: 0000000000000246 R12: 0000000000000000
[Mon Mar 11 14:29:16 2024] R13: 000055887f2992a0 R14: 0000000000000003 R15: 000055887f3a5b00
[Mon Mar 11 14:29:16 2024] </TASK>
[Mon Mar 11 14:29:16 2024] task:sudo state:S stack:0 pid:2478247 ppid:2478246 flags:0x00000002
[Mon Mar 11 14:29:16 2024] Call Trace:
[Mon Mar 11 14:29:16 2024] <TASK>
[Mon Mar 11 14:29:16 2024] __schedule+0x21b/0x550
[Mon Mar 11 14:29:16 2024] schedule+0x2d/0x70
[Mon Mar 11 14:29:16 2024] schedule_hrtimeout_range_clock+0x95/0x120
[Mon Mar 11 14:29:16 2024] ? __pfx_hrtimer_wakeup+0x10/0x10
[Mon Mar 11 14:29:16 2024] do_poll.constprop.0+0x248/0x390
[Mon Mar 11 14:29:16 2024] do_sys_poll+0x1c8/0x250
[Mon Mar 11 14:29:16 2024] ? __mod_memcg_lruvec_state+0x84/0xd0
[Mon Mar 11 14:29:16 2024] ? mod_objcg_state+0x1fe/0x300
[Mon Mar 11 14:29:16 2024] ? scm_recv.constprop.0+0x43/0x1a0
[Mon Mar 11 14:29:16 2024] ? scm_recv.constprop.0+0x43/0x1a0
[Mon Mar 11 14:29:16 2024] ? avc_has_perm+0x8f/0x1b0
[Mon Mar 11 14:29:16 2024] ? __pfx_pollwake+0x10/0x10
[Mon Mar 11 14:29:16 2024] ? unix_stream_recvmsg+0x92/0xa0
[Mon Mar 11 14:29:16 2024] ? __pfx_unix_stream_read_actor+0x10/0x10
[Mon Mar 11 14:29:16 2024] ? sock_recvmsg+0x99/0xa0
[Mon Mar 11 14:29:16 2024] ? __check_object_size.part.0+0x35/0xd0
[Mon Mar 11 14:29:16 2024] ? ____sys_recvmsg+0x87/0x1b0
[Mon Mar 11 14:29:16 2024] ? __import_iovec+0x46/0x150
[Mon Mar 11 14:29:16 2024] ? import_iovec+0x17/0x20
[Mon Mar 11 14:29:16 2024] ? copy_msghdr_from_user+0x6d/0xa0
[Mon Mar 11 14:29:16 2024] ? ___sys_recvmsg+0x88/0xd0
[Mon Mar 11 14:29:16 2024] ? __audit_filter_op+0xa5/0xf0
[Mon Mar 11 14:29:16 2024] ? _copy_from_user+0x27/0x60
[Mon Mar 11 14:29:16 2024] __x64_sys_ppoll+0xbc/0x150
[Mon Mar 11 14:29:16 2024] do_syscall_64+0x59/0x90
[Mon Mar 11 14:29:16 2024] ? syscall_exit_work+0x103/0x130
[Mon Mar 11 14:29:16 2024] ? syscall_exit_to_user_mode+0x22/0x40
[Mon Mar 11 14:29:16 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:16 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:16 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:16 2024] ? syscall_exit_to_user_mode+0x22/0x40
[Mon Mar 11 14:29:16 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:16 2024] ? do_syscall_64+0x69/0x90
[Mon Mar 11 14:29:16 2024] entry_SYSCALL_64_after_hwframe+0x72/0xdc
[Mon Mar 11 14:29:16 2024] RIP: 0033:0x7fb58794279f
[Mon Mar 11 14:29:16 2024] RSP: 002b:00007ffc79971dc0 EFLAGS: 00000246 ORIG_RAX: 000000000000010f
[Mon Mar 11 14:29:16 2024] RAX: ffffffffffffffda RBX: 00007ffc79971e60 RCX: 00007fb58794279f
[Mon Mar 11 14:29:16 2024] RDX: 00007ffc79971de0 RSI: 0000000000000001 RDI: 00007ffc79971e60
[Mon Mar 11 14:29:16 2024] RBP: 0000000000000001 R08: 0000000000000008 R09: 00007ffc79970d78
[Mon Mar 11 14:29:16 2024] R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000000001
[Mon Mar 11 14:29:16 2024] R13: 00007ffc79971de0 R14: 0000012aabc0581a R15: 0000000000000001
[Mon Mar 11 14:29:16 2024] </TASK>
[Mon Mar 11 14:29:16 2024] Sched Debug Version: v0.11, 5.14.0-419.el9.x86_64 #1
[Mon Mar 11 14:29:16 2024] ktime : 1282664683.579695
[Mon Mar 11 14:29:16 2024] sched_clk : 1282630518.614694
[Mon Mar 11 14:29:16 2024] cpu_clk : 1282630436.154729
[Mon Mar 11 14:29:16 2024] jiffies : 5577331856
[Mon Mar 11 14:29:16 2024] sched_clock_stable() : 1
[Mon Mar 11 14:29:16 2024] sysctl_sched
[Mon Mar 11 14:29:16 2024] .sysctl_sched_latency : 24.000000
[Mon Mar 11 14:29:16 2024] .sysctl_sched_min_granularity : 3.000000
[Mon Mar 11 14:29:16 2024] .sysctl_sched_idle_min_granularity : 0.750000
[Mon Mar 11 14:29:16 2024] .sysctl_sched_wakeup_granularity : 4.000000
[Mon Mar 11 14:29:16 2024] .sysctl_sched_child_runs_first : 0
[Mon Mar 11 14:29:16 2024] .sysctl_sched_features : 125720123
[Mon Mar 11 14:29:16 2024] .sysctl_sched_tunable_scaling : 1 (logarithmic)
[Mon Mar 11 14:29:16 2024] cpu#0, 2600.000 MHz
[Mon Mar 11 14:29:16 2024] .nr_running : 0
[Mon Mar 11 14:29:16 2024] .nr_switches : 2296394021
[Mon Mar 11 14:29:16 2024] .nr_uninterruptible : 51887
[Mon Mar 11 14:29:16 2024] .next_balance : 5577.331860
[Mon Mar 11 14:29:16 2024] .curr->pid : 0
[Mon Mar 11 14:29:16 2024] .clock : 1282630435.244990
[Mon Mar 11 14:29:16 2024] .clock_task : 1253286234.451401
[Mon Mar 11 14:29:16 2024] .avg_idle : 1000000
[Mon Mar 11 14:29:16 2024] .max_idle_balance_cost : 500000
[Mon Mar 11 14:29:16 2024] rt_rq[0]:
[Mon Mar 11 14:29:16 2024] .rt_nr_running : 0
[Mon Mar 11 14:29:16 2024] .rt_nr_migratory : 0
[Mon Mar 11 14:29:16 2024] .rt_throttled : 0
[Mon Mar 11 14:29:16 2024] .rt_time : 0.000000
[Mon Mar 11 14:29:16 2024] .rt_runtime : 950.000000
[Mon Mar 11 14:29:16 2024] dl_rq[0]:
[Mon Mar 11 14:29:16 2024] .dl_nr_running : 0
[Mon Mar 11 14:29:16 2024] .dl_nr_migratory : 0
[Mon Mar 11 14:29:16 2024] .dl_bw->bw : 996147
[Mon Mar 11 14:29:16 2024] .dl_bw->total_bw : 0
[Mon Mar 11 14:29:16 2024] runnable tasks:
[Mon Mar 11 14:29:16 2024] S task PID tree-key switches prio wait-time sum-exec sum-sleep
[Mon Mar 11 14:29:16 2024] -------------------------------------------------------------------------------------------------------------
[Mon Mar 11 14:29:16 2024] I rcu_gp 3 7.021156 2 100 0.000000 0.003112 0.000000 0.000000 0 0 /
[Mon Mar 11 14:29:16 2024] I rcu_par_gp 4 8.521748 2 100 0.000000 0.001816 0.000000 0.000000 0 0 /
[Mon Mar 11 14:29:16 2024] I slub_flushwq 5 10.522486 2 100 0.000000 0.001692 0.000000 0.000000 0 0 /
[Mon Mar 11 14:29:16 2024] I netns 6 12.523243 2 100 0.000000 0.001795 0.000000 0.000000 0 0 /
[Mon Mar 11 14:29:16 2024] I kworker/0:0H 8 3203.959207 4 100 0.000000 0.042887 0.000000 0.000000 0 0 /
[Mon Mar 11 14:29:16 2024] I mm_percpu_wq 12 21.277050 2 100 0.000000 0.001855 0.000000 0.000000 0 0 /
[Mon Mar 11 14:29:16 2024] I rcu_tasks_kthre 13 3162.976921 9 120 0.000000 0.949708 0.000000 0.000000 0 0 /
[Mon Mar 11 14:29:16 2024] I rcu_tasks_rude_ 14 25.279343 2 120 0.000000 0.002068 0.000000 0.000000 0 0 /
[Mon Mar 11 14:29:16 2024] S ksoftirqd/0 16 20417584.604598 31070353 120 0.000000 364032.682502 0.000000 0.000000 0 0 /
[Mon Mar 11 14:29:16 2024] S migration/0 19 0.000000 373527 0 0.000000 738.700624 0.000000 0.000000 0 0 /
[Mon Mar 11 14:29:16 2024] S idle_inject/0 20 0.000000 3 49 0.000000 0.004950 0.000000 0.000000 0 0 /
[Mon Mar 11 14:29:16 2024] S cpuhp/0 22 22468.084833 25 120 0.000000 2.588689 0.000000 0.000000 0 0 /
[Mon Mar 11 14:29:16 2024] S khungtaskd 361 20416782.155692 10440 120 0.000000 2653.563724 0.000000 0.000000 0 0 /
[Mon Mar 11 14:29:16 2024] I kworker/0:1H 800 20417584.597643 2972197 100 0.000000 61296.732473 0.000000 0.000000 0 0 /
[Mon Mar 11 14:29:16 2024] S irq/88-lpfc:0 935 0.000000 6383586 49 0.000000 127926.322832 0.000000 0.000000 0 0 /
[Mon Mar 11 14:29:16 2024] S irq/171-lpfc:0 990 0.000000 6396763 49 0.000000 128916.085338 0.000000 0.000000 0 0 /
[Mon Mar 11 14:29:16 2024] I kdmflush/253:0 2697 8171.305813 2 100 0.000000 0.025829 0.000000 0.000000 0 0 /
[Mon Mar 11 14:29:16 2024] S cupsd 4399 7.881855 4138 120 0.000000 2607.818928 0.000000 0.000000 0 0 /autogroup-87
[Mon Mar 11 14:29:16 2024] I kdmflush/253:91 6396 26185.289233 2 100 0.000000 0.023215 0.000000 0.000000 0 0 /
[Mon Mar 11 14:29:16 2024] I kdmflush/253:95 6422 26241.860538 2 100 0.000000 0.022122 0.000000 0.000000 0 0 /
[Mon Mar 11 14:29:16 2024] S jbd2/dm-94-8 6473 20361729.729941 574 120 0.000000 66.244708 0.000000 0.000000 0 0 /
[Mon Mar 11 14:29:16 2024] I ext4-rsv-conver 6492 26311.227438 2 100 0.000000 0.021092 0.000000 0.000000 0 0 /
[Mon Mar 11 14:29:16 2024] D nfsd 8850 20388822.869354 8289195 120 0.000000 357900.690801 0.000000 0.000000 0 0 /
[Mon Mar 11 14:29:16 2024] S automount 88129 11.464030 1 120 0.000000 0.512613 0.000000 0.000000 0 0 /autogroup-222
[Mon Mar 11 14:29:16 2024] S automount 88152 37.342455 1 120 0.000000 0.975275 0.000000 0.000000 0 0 /autogroup-222
[Mon Mar 11 14:29:16 2024] S automount 88163 50.261558 1 120 0.000000 0.919110 0.000000 0.000000 0 0 /autogroup-222
[Mon Mar 11 14:29:16 2024] S smbd[10.87.27.1 2343759 2214650.205580 236494 120 0.000000 95542.192550 0.000000 0.000000 0 0 /autogroup-113
[Mon Mar 11 14:29:16 2024] S smbd[10.87.29.8 2397415 2214625.547948 12317 120 0.000000 13032.835012 0.000000 0.000000 0 0 /autogroup-113
[Mon Mar 11 14:29:16 2024] I kworker/0:0 2433691 20417637.659863 4257088 120 0.000000 27949.242499 0.000000 0.000000 0 0 /
[Mon Mar 11 14:29:16 2024] I kworker/0:2 2475316 20414092.023162 32 120 0.000000 0.224977 0.000000 0.000000 0 0 /
[Mon Mar 11 14:29:16 2024] S check_fan_statu 2478246 276534.262092 10 120 0.000000 24.088894 0.000000 0.000000 0 0 /autogroup-109
[Mon Mar 11 14:29:16 2024] cpu#1, 2600.000 MHz
[Mon Mar 11 14:29:16 2024] .nr_running : 1
[Mon Mar 11 14:29:16 2024] .nr_switches : 807444859
[Mon Mar 11 14:29:16 2024] .nr_uninterruptible : 23435
[Mon Mar 11 14:29:16 2024] .next_balance : 5577.331904
[Mon Mar 11 14:29:16 2024] .curr->pid : 17
[Mon Mar 11 14:29:16 2024] .clock : 1282630436.239387
[Mon Mar 11 14:29:16 2024] .clock_task : 1279670360.339756
[Mon Mar 11 14:29:16 2024] .avg_idle : 904862
[Mon Mar 11 14:29:16 2024] .max_idle_balance_cost : 500000
[Mon Mar 11 14:29:16 2024] cfs_rq[1]:/autogroup-113
[Mon Mar 11 14:29:16 2024] .exec_clock : 0.000000
[Mon Mar 11 14:29:16 2024] .MIN_vruntime : 0.000001
[Mon Mar 11 14:29:16 2024] .min_vruntime : 8115658.175504
[Mon Mar 11 14:29:16 2024] .max_vruntime : 0.000001
[Mon Mar 11 14:29:16 2024] .spread : 0.000000
[Mon Mar 11 14:29:16 2024] .spread0 : -12301989.984208
[Mon Mar 11 14:29:16 2024] .nr_spread_over : 0
[Mon Mar 11 14:29:16 2024] .nr_running : 0
[Mon Mar 11 14:29:16 2024] .h_nr_running : 0
[Mon Mar 11 14:29:16 2024] .idle_nr_running : 0
[Mon Mar 11 14:29:16 2024] .idle_h_nr_running : 0
[Mon Mar 11 14:29:16 2024] .load : 0
[Mon Mar 11 14:29:16 2024] .load_avg : 0
[Mon Mar 11 14:29:16 2024] .runnable_avg : 0
[Mon Mar 11 14:29:16 2024] .util_avg : 0
[Mon Mar 11 14:29:16 2024] .util_est_enqueued : 0
[Mon Mar 11 14:29:16 2024] .removed.load_avg : 0
[Mon Mar 11 14:29:16 2024] .removed.util_avg : 0
[Mon Mar 11 14:29:16 2024] .removed.runnable_avg : 0
[Mon Mar 11 14:29:16 2024] .tg_load_avg_contrib : 0
[Mon Mar 11 14:29:16 2024] .tg_load_avg : 73
[Mon Mar 11 14:29:16 2024] .throttled : 0
[Mon Mar 11 14:29:16 2024] .throttle_count : 0
[Mon Mar 11 14:29:16 2024] .se->exec_start : 1279670338.088628
[Mon Mar 11 14:29:16 2024] .se->vruntime : 26186993.311434
[Mon Mar 11 14:29:16 2024] .se->sum_exec_runtime : 10639176.007842
[Mon Mar 11 14:29:16 2024] .se->load.weight : 914143
[Mon Mar 11 14:29:16 2024] .se->avg.load_avg : 0
[Mon Mar 11 14:29:16 2024] .se->avg.util_avg : 0
[Mon Mar 11 14:29:16 2024] .se->avg.runnable_avg : 0
[Mon Mar 11 14:29:16 2024] cfs_rq[1]:/
[Mon Mar 11 14:29:16 2024] .exec_clock : 0.000000
[Mon Mar 11 14:29:16 2024] .MIN_vruntime : 0.000001
[Mon Mar 11 14:29:16 2024] .min_vruntime : 26187002.773003
[Mon Mar 11 14:29:16 2024] .max_vruntime : 0.000001
[Mon Mar 11 14:29:16 2024] .spread : 0.000000
[Mon Mar 11 14:29:16 2024] .spread0 : 5769354.613291
[Mon Mar 11 14:29:16 2024] .nr_spread_over : 0
[Mon Mar 11 14:29:16 2024] .nr_running : 1
[Mon Mar 11 14:29:16 2024] .h_nr_running : 1
[Mon Mar 11 14:29:16 2024] .idle_nr_running : 0
[Mon Mar 11 14:29:16 2024] .idle_h_nr_running : 0
[Mon Mar 11 14:29:16 2024] .load : 1048576
[Mon Mar 11 14:29:16 2024] .load_avg : 94
[Mon Mar 11 14:29:16 2024] .runnable_avg : 93
[Mon Mar 11 14:29:16 2024] .util_avg : 93
[Mon Mar 11 14:29:16 2024] .util_est_enqueued : 9
[Mon Mar 11 14:29:16 2024] .removed.load_avg : 0
[Mon Mar 11 14:29:16 2024] .removed.util_avg : 0
[Mon Mar 11 14:29:16 2024] .removed.runnable_avg : 0
[Mon Mar 11 14:29:16 2024] .tg_load_avg_contrib : 0
[Mon Mar 11 14:29:16 2024] .tg_load_avg : 0
[Mon Mar 11 14:29:16 2024] .throttled : 0
[Mon Mar 11 14:29:16 2024] .throttle_count : 0
[Mon Mar 11 14:29:16 2024] rt_rq[1]:
[Mon Mar 11 14:29:16 2024] .rt_nr_running : 0
[Mon Mar 11 14:29:16 2024] .rt_nr_migratory : 0
[Mon Mar 11 14:29:16 2024] .rt_throttled : 0
[Mon Mar 11 14:29:16 2024] .rt_time : 0.173428
[Mon Mar 11 14:29:16 2024] .rt_runtime : 950.000000
[Mon Mar 11 14:29:16 2024] dl_rq[1]:
[Mon Mar 11 14:29:16 2024] .dl_nr_running : 0
[Mon Mar 11 14:29:16 2024] .dl_nr_migratory : 0
[Mon Mar 11 14:29:16 2024] .dl_bw->bw : 996147
[Mon Mar 11 14:29:16 2024] .dl_bw->total_bw : 0
[Mon Mar 11 14:29:16 2024] runnable tasks:
[Mon Mar 11 14:29:16 2024] S task PID tree-key switches prio wait-time sum-exec sum-sleep
[Mon Mar 11 14:29:16 2024] -------------------------------------------------------------------------------------------------------------
[Mon Mar 11 14:29:16 2024] >R pr/tty0 17 26187002.773003 6172000 120 0.000000 55552.090725 0.000000 0.000000 1 0 /
[Mon Mar 11 14:29:16 2024] S cpuhp/1 23 26928.561263 26 120 0.000000 1.297977 0.000000 0.000000 1 0 /
[Mon Mar 11 14:29:16 2024] S idle_inject/1 24 -2.996642 3 49 0.000000 0.009771 0.000000 0.000000 1 0 /
[Mon Mar 11 14:29:16 2024] S migration/1 25 0.000000 323819 0 0.000000 1720.423424 0.000000 0.000000 1 0 /
[Mon Mar 11 14:29:16 2024] S ksoftirqd/1 26 26186873.342690 11067261 120 0.000000 108044.304272 0.000000 0.000000 1 0 /
[Mon Mar 11 14:29:16 2024] I kworker/1:0H 28 402.193425 4 100 0.000000 0.026642 0.000000 0.000000 1 0 /
[Mon Mar 11 14:29:16 2024] I kworker/1:1H 378 26186873.129661 3241923 100 0.000000 71103.741959 0.000000 0.000000 1 0 /
[Mon Mar 11 14:29:16 2024] I kworker/u101:0 463 443.724114 2 100 0.000000 0.064463 0.000000 0.000000 1 0 /
[Mon Mar 11 14:29:16 2024] I nvmet-zbd-wq 894 1693.887859 2 100 0.000000 0.008278 0.000000 0.000000 1 0 /
[Mon Mar 11 14:29:16 2024] I nvmet-buffered- 895 1705.893374 2 100 0.000000 0.006049 0.000000 0.000000 1 0 /
[Mon Mar 11 14:29:16 2024] I nvmet-wq 896 1717.898662 2 100 0.000000 0.005741 0.000000 0.000000 1 0 /
[Mon Mar 11 14:29:16 2024] S scsi_eh_5 906 1745.293639 25 120 0.000000 3.381642 0.000000 0.000000 1 0 /
[Mon Mar 11 14:29:16 2024] I scsi_tmf_6 909 1753.919270 2 100 0.000000 0.007559 0.000000 0.000000 1 0 /
[Mon Mar 11 14:29:16 2024] S irq/89-lpfc:1 936 0.000000 5293866 49 0.000000 126916.418104 0.000000 0.000000 1 0 /
[Mon Mar 11 14:29:16 2024] I lpfc_wq 988 1765.985777 2 100 0.000000 0.069727 0.000000 0.000000 1 0 /
[Mon Mar 11 14:29:16 2024] S irq/172-lpfc:1 991 0.000000 5300318 49 0.000000 127800.522092 0.000000 0.000000 1 0 /
[Mon Mar 11 14:29:16 2024] I kdmflush/253:24 3594 27759.349464 2 100 0.000000 0.016555 0.000000 0.000000 1 0 /
[Mon Mar 11 14:29:16 2024] I kdmflush/253:28 3623 27822.825312 2 100 0.000000 0.016761 0.000000 0.000000 1 0 /
[Mon Mar 11 14:29:16 2024] I kdmflush/253:70 3986 28214.746330 2 100 0.000000 0.008648 0.000000 0.000000 1 0 /
[Mon Mar 11 14:29:16 2024] I ext4-rsv-conver 4247 28865.751109 2 100 0.000000 0.010033 0.000000 0.000000 1 0 /
[Mon Mar 11 14:29:16 2024] S dbus-broker 4303 4980.347132 3735806 120 0.000000 177080.383037 0.000000 0.000000 1 0 /autogroup-65
[Mon Mar 11 14:29:16 2024] S rpc.rquotad 4319 3714.291187 4249 120 0.000000 25209.231292 0.000000 0.000000 1 0 /autogroup-80
[Mon Mar 11 14:29:16 2024] S sssd_be 4348 281228.994060 1313607 120 0.000000 930669.978920 0.000000 0.000000 1 0 /autogroup-78
[Mon Mar 11 14:29:16 2024] S f2b/observer 4437 19412.555733 22153 120 0.000000 2173.269196 0.000000 0.000000 1 4437 /autogroup-96
[Mon Mar 11 14:29:16 2024] S rpc.gssd 4431 1027.778699 132195 120 0.000000 16979.021952 0.000000 0.000000 1 0 /autogroup-100
[Mon Mar 11 14:29:16 2024] S JS Helper 5178 11.302628 12 120 0.000000 0.351211 0.000000 0.000000 1 0 /autogroup-104
[Mon Mar 11 14:29:16 2024] S wb[ESAT] 4744 532837.348257 5600132 120 0.000000 2375724.443295 0.000000 0.000000 1 0 /autogroup-95
[Mon Mar 11 14:29:16 2024] I ext4-rsv-conver 6474 31232.973357 2 100 0.000000 0.065043 0.000000 0.000000 1 0 /
[Mon Mar 11 14:29:16 2024] I ib_nl_sa_wq 8765 35034.846633 2 100 0.000000 0.066047 0.000000 0.000000 1 0 /
[Mon Mar 11 14:29:16 2024] S automount 88118 5326.058588 115 120 0.000000 10.781388 0.000000 0.000000 1 0 /autogroup-222
[Mon Mar 11 14:29:16 2024] S automount 88124 5811.144173 98 120 0.000000 12.312219 0.000000 0.000000 1 0 /autogroup-222
[Mon Mar 11 14:29:16 2024] S automount 88135 5485.804293 39 120 0.000000 5.965224 0.000000 0.000000 1 0 /autogroup-222
[Mon Mar 11 14:29:16 2024] S automount 88156 5030.712050 15 120 0.000000 2.524423 0.000000 0.000000 1 0 /autogroup-222
[Mon Mar 11 14:29:16 2024] S atd 88110 48.570911 367 120 0.000000 108.183500 0.000000 0.000000 1 0 /autogroup-224
[Mon Mar 11 14:29:16 2024] S smbd[10.91.30.5 316974 8115278.994138 1283604 120 0.000000 1605373.289340 0.000000 0.000000 1 0 /autogroup-113
[Mon Mar 11 14:29:16 2024] S smbd[10.87.27.1 2347868 8115546.966585 3940904 120 0.000000 569321.239319 0.000000 0.000000 1 0 /autogroup-113
[Mon Mar 11 14:29:16 2024] S smbd[10.87.29.2 2362257 8115255.817322 789 120 0.000000 492.355096 0.000000 0.000000 1 0 /autogroup-113
[Mon Mar 11 14:29:16 2024] S smbd[10.87.28.1 2367684 8114281.525028 402 120 0.000000 135.984713 0.000000 0.000000 1 0 /autogroup-113
[Mon Mar 11 14:29:16 2024] I kworker/1:1 2399485 26186949.892506 152162 120 0.000000 63911.939232 0.000000 0.000000 1 0 /
[Mon Mar 11 14:29:16 2024] I kworker/1:2 2474995 26170899.951685 4 120 0.000000 0.010487 0.000000 0.000000 1 0 /
[Mon Mar 11 14:29:16 2024] cpu#2, 2600.000 MHz
[Mon Mar 11 14:29:16 2024] .nr_running : 0
[Mon Mar 11 14:29:16 2024] .nr_switches : 3090233980
[Mon Mar 11 14:29:16 2024] .nr_uninterruptible : -204094
[Mon Mar 11 14:29:16 2024] .next_balance : 5577.331861
[Mon Mar 11 14:29:16 2024] .curr->pid : 0
[Mon Mar 11 14:29:16 2024] .clock : 1282630436.244759
[Mon Mar 11 14:29:16 2024] .clock_task : 1256281662.020018
[Mon Mar 11 14:29:16 2024] .avg_idle : 1000000
[Mon Mar 11 14:29:16 2024] .max_idle_balance_cost : 500000
[Mon Mar 11 14:29:16 2024] rt_rq[2]:
[Mon Mar 11 14:29:16 2024] .rt_nr_running : 0
[Mon Mar 11 14:29:16 2024] .rt_nr_migratory : 0
[Mon Mar 11 14:29:16 2024] .rt_throttled : 0
[Mon Mar 11 14:29:16 2024] .rt_time : 0.000000
[Mon Mar 11 14:29:16 2024] .rt_runtime : 950.000000
[Mon Mar 11 14:29:16 2024] dl_rq[2]:
[Mon Mar 11 14:29:16 2024] .dl_nr_running : 0
[Mon Mar 11 14:29:16 2024] .dl_nr_migratory : 0
[Mon Mar 11 14:29:16 2024] .dl_bw->bw : 996147
[Mon Mar 11 14:29:16 2024] .dl_bw->total_bw : 0
[Mon Mar 11 14:29:16 2024] runnable tasks:
[Mon Mar 11 14:29:16 2024] S task PID tree-key switches prio wait-time sum-exec sum-sleep
[Mon Mar 11 14:29:16 2024] -------------------------------------------------------------------------------------------------------------
[Mon Mar 11 14:29:16 2024] I rcu_tasks_trace 15 191.591316 7 120 0.000000 1.228390 0.000000 0.000000 0 0 /
[Mon Mar 11 14:29:16 2024] S cpuhp/2 30 25785.486246 25 120 0.000000 1.377357 0.000000 0.000000 0 0 /
[Mon Mar 11 14:29:16 2024] S idle_inject/2 31 -6.000000 3 49 0.000000 0.000000 0.000000 0.000000 0 0 /
[Mon Mar 11 14:29:16 2024] S migration/2 32 76.554899 374546 0 0.000000 1716.502442 0.000000 0.000000 0 0 /
[Mon Mar 11 14:29:16 2024] S ksoftirqd/2 33 41439826.273721 30996709 120 0.000000 347325.255191 0.000000 0.000000 0 0 /
[Mon Mar 11 14:29:16 2024] I kworker/2:0H 35 308.812723 4 100 0.000000 0.030917 0.000000 0.000000 0 0 /
[Mon Mar 11 14:29:16 2024] I kworker/u100:0 462 128.393642 2 100 0.000000 0.061361 0.000000 0.000000 0 0 /
[Mon Mar 11 14:29:16 2024] I kworker/2:1H 571 41439798.996811 4193050 100 0.000000 103229.132660 0.000000 0.000000 0 0 /
[Mon Mar 11 14:29:16 2024] S scsi_eh_8 916 1921.950948 25 120 0.000000 2.154584 0.000000 0.000000 0 0 /
[Mon Mar 11 14:29:16 2024] S irq/90-lpfc:2 937 0.000000 12174502 49 0.000000 237076.318537 0.000000 0.000000 0 0 /
[Mon Mar 11 14:29:16 2024] S irq/173-lpfc:2 992 0.000000 12201226 49 0.000000 237719.656129 0.000000 0.000000 0 0 /
[Mon Mar 11 14:29:16 2024] S scsi_eh_16 1014 2254.017550 2 120 0.000000 0.008601 0.000000 0.000000 0 0 /
[Mon Mar 11 14:29:16 2024] I ext4-rsv-conver 2717 3374.585719 2 100 0.000000 0.006945 0.000000 0.000000 0 0 /
[Mon Mar 11 14:29:16 2024] I kdmflush/253:2 3217 22117.128323 2 100 0.000000 0.007700 0.000000 0.000000 0 0 /
[Mon Mar 11 14:29:16 2024] I kdmflush/253:3 3218 22128.897705 2 100 0.000000 0.008865 0.000000 0.000000 0 0 /
[Mon Mar 11 14:29:16 2024] I ext4-rsv-conver 4253 27593.089036 2 100 0.000000 0.061857 0.000000 0.000000 0 0 /
[Mon Mar 11 14:29:16 2024] I ext4-rsv-conver 4331 27648.261936 2 100 0.000000 0.066374 0.000000 0.000000 0 0 /
[Mon Mar 11 14:29:16 2024] S gmain 4355 3281.528039 320919 120 0.000000 27062.449859 0.000000 0.000000 0 4355 /autogroup-82
[Mon Mar 11 14:29:16 2024] D rs:main Q:Reg 4840 2062.828757 2229489 120 0.000000 93711.695350 0.000000 0.000000 0 4837 /autogroup-112
[Mon Mar 11 14:29:16 2024] S smbd 4757 2727645.254704 1012106 120 0.000000 1020301.486781 0.000000 0.000000 0 0 /autogroup-113
[Mon Mar 11 14:29:16 2024] D nfsd 8795 41439836.435463 85261 120 0.000000 9703.829192 0.000000 0.000000 0 0 /
[Mon Mar 11 14:29:16 2024] D nfsd 8802 41439827.346838 110528 120 0.000000 11529.146586 0.000000 0.000000 0 0 /
[Mon Mar 11 14:29:16 2024] D nfsd 8808 41439834.679721 143362 120 0.000000 14661.554591 0.000000 0.000000 0 0 /
[Mon Mar 11 14:29:16 2024] S automount 88158 11.907868 1 120 0.000000 0.956451 0.000000 0.000000 0 0 /autogroup-222
[Mon Mar 11 14:29:16 2024] S smbd[10.87.30.4 2334575 2727645.282735 527 120 0.000000 168.578035 0.000000 0.000000 0 0 /autogroup-113
[Mon Mar 11 14:29:16 2024] S smbd[10.40.137. 2400165 2727554.305010 74673 120 0.000000 12342.589900 0.000000 0.000000 0 0 /autogroup-113
[Mon Mar 11 14:29:16 2024] I kworker/2:2 2424238 41439826.357798 643767 120 0.000000 13462.215571 0.000000 0.000000 0 0 /
[Mon Mar 11 14:29:16 2024] I kworker/2:1 2473316 41432871.673072 9 120 0.000000 0.074297 0.000000 0.000000 0 0 /
[Mon Mar 11 14:29:16 2024] I kworker/2:0 2477161 41434261.267573 3 120 0.000000 0.063449 0.000000 0.000000 0 0 /
[Mon Mar 11 14:29:16 2024] I kworker/u97:5 2477730 41438381.243921 9797 120 0.000000 114.740676 0.000000 0.000000 0 0 /
[Mon Mar 11 14:29:16 2024] cpu#3, 2600.000 MHz
[Mon Mar 11 14:29:16 2024] .nr_running : 0
[Mon Mar 11 14:29:16 2024] .nr_switches : 1174028061
[Mon Mar 11 14:29:16 2024] .nr_uninterruptible : -5162
[Mon Mar 11 14:29:16 2024] .next_balance : 5577.331859
[Mon Mar 11 14:29:16 2024] .curr->pid : 0
[Mon Mar 11 14:29:16 2024] .clock : 1282630436.898544
[Mon Mar 11 14:29:16 2024] .clock_task : 1280169077.244563
[Mon Mar 11 14:29:16 2024] .avg_idle : 734944
[Mon Mar 11 14:29:16 2024] .max_idle_balance_cost : 500000
[Mon Mar 11 14:29:16 2024] cfs_rq[3]:/autogroup-113
[Mon Mar 11 14:29:16 2024] .exec_clock : 0.000000
[Mon Mar 11 14:29:16 2024] .MIN_vruntime : 0.000001
[Mon Mar 11 14:29:16 2024] .min_vruntime : 8959496.244470
[Mon Mar 11 14:29:16 2024] .max_vruntime : 0.000001
[Mon Mar 11 14:29:16 2024] .spread : 0.000000
[Mon Mar 11 14:29:16 2024] .spread0 : -11458151.915242
[Mon Mar 11 14:29:16 2024] .nr_spread_over : 0
[Mon Mar 11 14:29:16 2024] .nr_running : 0
[Mon Mar 11 14:29:16 2024] .h_nr_running : 0
[Mon Mar 11 14:29:16 2024] .idle_nr_running : 0
[Mon Mar 11 14:29:16 2024] .idle_h_nr_running : 0
[Mon Mar 11 14:29:16 2024] .load : 0
[Mon Mar 11 14:29:16 2024] .load_avg : 68
[Mon Mar 11 14:29:16 2024] .runnable_avg : 68
[Mon Mar 11 14:29:16 2024] .util_avg : 68
[Mon Mar 11 14:29:16 2024] .util_est_enqueued : 0
[Mon Mar 11 14:29:16 2024] .removed.load_avg : 0
[Mon Mar 11 14:29:16 2024] .removed.util_avg : 0
[Mon Mar 11 14:29:16 2024] .removed.runnable_avg : 0
[Mon Mar 11 14:29:16 2024] .tg_load_avg_contrib : 68
[Mon Mar 11 14:29:16 2024] .tg_load_avg : 73
[Mon Mar 11 14:29:16 2024] .throttled : 0
[Mon Mar 11 14:29:16 2024] .throttle_count : 0
[Mon Mar 11 14:29:16 2024] .se->exec_start : 1280169077.242552
[Mon Mar 11 14:29:16 2024] .se->vruntime : 31376681.441971
[Mon Mar 11 14:29:16 2024] .se->sum_exec_runtime : 10844857.316671
[Mon Mar 11 14:29:16 2024] .se->load.weight : 976755
[Mon Mar 11 14:29:16 2024] .se->avg.load_avg : 63
[Mon Mar 11 14:29:16 2024] .se->avg.util_avg : 67
[Mon Mar 11 14:29:16 2024] .se->avg.runnable_avg : 67
[Mon Mar 11 14:29:16 2024] cfs_rq[3]:/
[Mon Mar 11 14:29:16 2024] .exec_clock : 0.000000
[Mon Mar 11 14:29:16 2024] .MIN_vruntime : 0.000001
[Mon Mar 11 14:29:16 2024] .min_vruntime : 31376681.441971
[Mon Mar 11 14:29:16 2024] .max_vruntime : 0.000001
[Mon Mar 11 14:29:16 2024] .spread : 0.000000
[Mon Mar 11 14:29:16 2024] .spread0 : 10959033.282259
[Mon Mar 11 14:29:16 2024] .nr_spread_over : 0
[Mon Mar 11 14:29:16 2024] .nr_running : 0
[Mon Mar 11 14:29:16 2024] .h_nr_running : 0
[Mon Mar 11 14:29:16 2024] .idle_nr_running : 0
[Mon Mar 11 14:29:16 2024] .idle_h_nr_running : 0
[Mon Mar 11 14:29:16 2024] .load : 0
[Mon Mar 11 14:29:16 2024] .load_avg : 64
[Mon Mar 11 14:29:16 2024] .runnable_avg : 68
[Mon Mar 11 14:29:16 2024] .util_avg : 68
[Mon Mar 11 14:29:16 2024] .util_est_enqueued : 0
[Mon Mar 11 14:29:16 2024] .removed.load_avg : 0
[Mon Mar 11 14:29:16 2024] .removed.util_avg : 0
[Mon Mar 11 14:29:16 2024] .removed.runnable_avg : 0
[Mon Mar 11 14:29:16 2024] .tg_load_avg_contrib : 0
[Mon Mar 11 14:29:16 2024] .tg_load_avg : 0
[Mon Mar 11 14:29:16 2024] .throttled : 0
[Mon Mar 11 14:29:16 2024] .throttle_count : 0
[Mon Mar 11 14:29:16 2024] rt_rq[3]:
[Mon Mar 11 14:29:16 2024] .rt_nr_running : 0
[Mon Mar 11 14:29:16 2024] .rt_nr_migratory : 0
[Mon Mar 11 14:29:16 2024] .rt_throttled : 0
[Mon Mar 11 14:29:16 2024] .rt_time : 0.000000
[Mon Mar 11 14:29:16 2024] .rt_runtime : 950.000000
[Mon Mar 11 14:29:16 2024] dl_rq[3]:
[Mon Mar 11 14:29:16 2024] .dl_nr_running : 0
[Mon Mar 11 14:29:16 2024] .dl_nr_migratory : 0
[Mon Mar 11 14:29:16 2024] .dl_bw->bw : 996147
[Mon Mar 11 14:29:16 2024] .dl_bw->total_bw : 0
[Mon Mar 11 14:29:16 2024] runnable tasks:
[Mon Mar 11 14:29:16 2024] S task PID tree-key switches prio wait-time sum-exec sum-sleep
[Mon Mar 11 14:29:16 2024] -------------------------------------------------------------------------------------------------------------
[Mon Mar 11 14:29:16 2024] S cpuhp/3 36 23563.131278 25 120 0.000000 0.877111 0.000000 0.000000 1 0 /
[Mon Mar 11 14:29:16 2024] S idle_inject/3 37 -6.000000 3 49 0.000000 0.000000 0.000000 0.000000 1 0 /
[Mon Mar 11 14:29:16 2024] S migration/3 38 108.554913 330113 0 0.000000 1818.512264 0.000000 0.000000 1 0 /
[Mon Mar 11 14:29:16 2024] S ksoftirqd/3 39 31376644.361624 12352259 120 0.000000 128122.475375 0.000000 0.000000 1 0 /
[Mon Mar 11 14:29:16 2024] I kworker/3:0H 41 222.855626 4 100 0.000000 0.049993 0.000000 0.000000 1 0 /
[Mon Mar 11 14:29:16 2024] I kworker/3:1H 774 31376644.275358 3358888 100 0.000000 73020.212897 0.000000 0.000000 1 0 /
[Mon Mar 11 14:29:16 2024] S irq/91-lpfc:3 938 0.000000 5727349 49 0.000000 129632.349893 0.000000 0.000000 1 0 /
[Mon Mar 11 14:29:16 2024] S irq/174-lpfc:3 993 0.000000 5737101 49 0.000000 130917.486645 0.000000 0.000000 1 0 /
[Mon Mar 11 14:29:16 2024] I kdmflush/253:23 3587 24829.743353 2 100 0.000000 0.010353 0.000000 0.000000 1 0 /
[Mon Mar 11 14:29:16 2024] I kdmflush/253:74 4102 25515.587727 2 100 0.000000 0.018098 0.000000 0.000000 1 0 /
[Mon Mar 11 14:29:16 2024] I kdmflush/253:85 4161 25602.653500 2 100 0.000000 0.018730 0.000000 0.000000 1 0 /
[Mon Mar 11 14:29:16 2024] I ext4-rsv-conver 4309 25836.120153 2 100 0.000000 0.013754 0.000000 0.000000 1 0 /
[Mon Mar 11 14:29:16 2024] S gdbus 4359 1957.218446 48401 120 0.000000 16123.263267 0.000000 0.000000 1 4359 /autogroup-82
[Mon Mar 11 14:29:16 2024] S tuned 5104 345.552364 51 120 0.000000 126.060503 0.000000 0.000000 1 0 /autogroup-93
[Mon Mar 11 14:29:16 2024] S winbindd 4415 445384.357935 9168157 120 0.000000 2562461.686949 0.000000 0.000000 1 0 /autogroup-95
[Mon Mar 11 14:29:16 2024] I kcopyd 6394 27488.051576 2 100 0.000000 0.065770 0.000000 0.000000 1 0 /
[Mon Mar 11 14:29:16 2024] I kdmflush/253:96 6427 27515.609358 2 100 0.000000 0.020021 0.000000 0.000000 1 0 /
[Mon Mar 11 14:29:16 2024] I ext4-rsv-conver 6476 27537.464132 2 100 0.000000 0.064131 0.000000 0.000000 1 0 /
[Mon Mar 11 14:29:16 2024] S pool-spawner 21804 14.045188 3 120 0.000000 0.579274 0.000000 0.000000 1 0 /autogroup-204
[Mon Mar 11 14:29:16 2024] S automount 88161 3162.684439 20 120 0.000000 3.225959 0.000000 0.000000 1 0 /autogroup-222
[Mon Mar 11 14:29:16 2024] S tmux: server 91743 1.658587 792 120 0.000000 126.457696 0.000000 0.000000 1 0 /autogroup-291
[Mon Mar 11 14:29:16 2024] S smbd[10.91.30.5 317495 8959491.175142 1305728 120 0.000000 1683541.041424 0.000000 0.000000 1 0 /autogroup-113
[Mon Mar 11 14:29:16 2024] S smbd[10.91.30.5 1292148 8959481.947024 2301055 120 0.000000 2057467.722051 0.000000 0.000000 1 0 /autogroup-113
[Mon Mar 11 14:29:16 2024] S smbd[10.40.238. 2211016 8958702.249213 1819 120 0.000000 848.317512 0.000000 0.000000 1 0 /autogroup-113
[Mon Mar 11 14:29:16 2024] S smbd[10.87.29.9 2279969 8959034.574789 941 120 0.000000 274.050833 0.000000 0.000000 1 0 /autogroup-113
[Mon Mar 11 14:29:16 2024] S smbd[10.87.28.1 2322257 8958677.490533 230593 120 0.000000 53481.407463 0.000000 0.000000 1 0 /autogroup-113
[Mon Mar 11 14:29:16 2024] S smbd[10.87.30.6 2376116 8959496.244470 1041340 120 0.000000 314530.985059 0.000000 0.000000 1 0 /autogroup-113
[Mon Mar 11 14:29:16 2024] S smbd[10.40.140. 2379578 8959491.055968 231804 120 0.000000 109159.750924 0.000000 0.000000 1 0 /autogroup-113
[Mon Mar 11 14:29:16 2024] I NFSv4 callback 2387784 30627143.826700 2 120 0.000000 0.065421 0.000000 0.000000 1 0 /
[Mon Mar 11 14:29:16 2024] S smbd[10.87.30.2 2393076 8958677.147265 306 120 0.000000 102.086484 0.000000 0.000000 1 0 /autogroup-113
[Mon Mar 11 14:29:16 2024] I kworker/3:2 2399492 31372271.815654 124996 120 0.000000 12495.976990 0.000000 0.000000 1 0 /
[Mon Mar 11 14:29:16 2024] S smbd[10.87.27.6 2400787 8958817.704219 334 120 0.000000 117.720586 0.000000 0.000000 1 0 /autogroup-113
[Mon Mar 11 14:29:16 2024] S sshd 2474602 39.992787 29 120 0.000000 68.415199 0.000000 0.000000 1 0 /autogroup-23273
[Mon Mar 11 14:29:16 2024] I kworker/3:1 2474822 31376661.430807 8593 120 0.000000 1394.753629 0.000000 0.000000 1 0 /
[Mon Mar 11 14:29:16 2024] I kworker/3:0 2477683 31372867.190878 228 120 0.000000 2.310259 0.000000 0.000000 1 0 /
[Mon Mar 11 14:29:16 2024] cpu#4, 2600.000 MHz
[Mon Mar 11 14:29:16 2024] .nr_running : 0
[Mon Mar 11 14:29:16 2024] .nr_switches : 3120418900
[Mon Mar 11 14:29:16 2024] .nr_uninterruptible : -148839
[Mon Mar 11 14:29:16 2024] .next_balance : 5577.331860
[Mon Mar 11 14:29:16 2024] .curr->pid : 0
[Mon Mar 11 14:29:16 2024] .clock : 1282630437.242753
[Mon Mar 11 14:29:16 2024] .clock_task : 1255462453.137023
[Mon Mar 11 14:29:16 2024] .avg_idle : 778545
[Mon Mar 11 14:29:16 2024] .max_idle_balance_cost : 500000
[Mon Mar 11 14:29:16 2024] rt_rq[4]:
[Mon Mar 11 14:29:16 2024] .rt_nr_running : 0
[Mon Mar 11 14:29:16 2024] .rt_nr_migratory : 0
[Mon Mar 11 14:29:16 2024] .rt_throttled : 0
[Mon Mar 11 14:29:16 2024] .rt_time : 0.668072
[Mon Mar 11 14:29:16 2024] .rt_runtime : 950.000000
[Mon Mar 11 14:29:16 2024] dl_rq[4]:
[Mon Mar 11 14:29:16 2024] .dl_nr_running : 0
[Mon Mar 11 14:29:16 2024] .dl_nr_migratory : 0
[Mon Mar 11 14:29:16 2024] .dl_bw->bw : 996147
[Mon Mar 11 14:29:16 2024] .dl_bw->total_bw : 0
[Mon Mar 11 14:29:16 2024] runnable tasks:
[Mon Mar 11 14:29:16 2024] S task PID tree-key switches prio wait-time sum-exec sum-sleep
[Mon Mar 11 14:29:16 2024] -------------------------------------------------------------------------------------------------------------
[Mon Mar 11 14:29:16 2024] S cpuhp/4 42 26980.275535 25 120 0.000000 1.425394 0.000000 0.000000 0 0 /
[Mon Mar 11 14:29:16 2024] S idle_inject/4 43 -9.000000 3 49 0.000000 0.000000 0.000000 0.000000 0 0 /
[Mon Mar 11 14:29:16 2024] S migration/4 44 148.554901 385185 0 0.000000 1940.748008 0.000000 0.000000 0 0 /
[Mon Mar 11 14:29:16 2024] S ksoftirqd/4 45 41862412.801888 30271470 120 0.000000 341652.049409 0.000000 0.000000 0 0 /
[Mon Mar 11 14:29:16 2024] I kworker/4:0H 47 277.551054 4 100 0.000000 0.033847 0.000000 0.000000 0 0 /
[Mon Mar 11 14:29:16 2024] S irq/24-pciehp 398 0.000000 3 49 0.000000 0.026089 0.000000 0.000000 0 0 /
[Mon Mar 11 14:29:16 2024] I kworker/4:1H 610 41862382.170150 4156540 100 0.000000 102020.392773 0.000000 0.000000 0 0 /
[Mon Mar 11 14:29:16 2024] I scsi_tmf_9 919 2752.901188 2 100 0.000000 0.088164 0.000000 0.000000 0 0 /
[Mon Mar 11 14:29:16 2024] S irq/92-lpfc:4 939 0.000000 11772394 49 0.000000 229099.392793 0.000000 0.000000 0 0 /
[Mon Mar 11 14:29:16 2024] S irq/175-lpfc:4 994 0.000000 11800966 49 0.000000 229372.196113 0.000000 0.000000 0 0 /
[Mon Mar 11 14:29:16 2024] I scsi_tmf_16 1015 2775.172950 2 100 0.000000 0.017693 0.000000 0.000000 0 0 /
[Mon Mar 11 14:29:16 2024] S multipathd 3356 0.000000 5399061 0 0.000000 1183666.226787 0.000000 0.000000 0 0 /autogroup-45
[Mon Mar 11 14:29:16 2024] I kdmflush/253:45 3779 28119.086445 2 100 0.000000 0.010071 0.000000 0.000000 0 0 /
[Mon Mar 11 14:29:16 2024] I kdmflush/253:80 4154 28188.682925 2 100 0.000000 0.079162 0.000000 0.000000 0 0 /
[Mon Mar 11 14:29:16 2024] I kdmflush/253:98 6437 29853.097809 2 100 0.000000 0.011114 0.000000 0.000000 0 0 /
[Mon Mar 11 14:29:16 2024] I kcopyd 8695 30939.783039 2 100 0.000000 0.067660 0.000000 0.000000 0 0 /
[Mon Mar 11 14:29:16 2024] I dm-thin 8696 30951.795800 2 100 0.000000 0.014825 0.000000 0.000000 0 0 /
[Mon Mar 11 14:29:16 2024] D nfsd 8791 41862425.319621 81235 120 0.000000 9172.129488 0.000000 0.000000 0 0 /
[Mon Mar 11 14:29:16 2024] D nfsd 8843 41796142.132805 4948485 120 0.000000 144691.340628 0.000000 0.000000 0 0 /
[Mon Mar 11 14:29:16 2024] S automount 88123 11.878998 1 120 0.000000 0.927581 0.000000 0.000000 0 0 /autogroup-222
[Mon Mar 11 14:29:16 2024] S automount 88136 24.711510 1 120 0.000000 0.832519 0.000000 0.000000 0 0 /autogroup-222
[Mon Mar 11 14:29:16 2024] S smbd[10.87.18.2 2347080 2709130.511401 12987 120 0.000000 3746.791301 0.000000 0.000000 0 0 /autogroup-113
[Mon Mar 11 14:29:16 2024] S smbd[10.40.232. 2351731 2709229.186605 14796 120 0.000000 7543.722573 0.000000 0.000000 0 0 /autogroup-113
[Mon Mar 11 14:29:16 2024] S smbd[10.40.135. 2403402 2708971.809793 287 120 0.000000 97.036061 0.000000 0.000000 0 0 /autogroup-113
[Mon Mar 11 14:29:16 2024] S smbd[10.46.163. 2441678 2708911.686753 911 120 0.000000 156.069362 0.000000 0.000000 0 0 /autogroup-113
[Mon Mar 11 14:29:16 2024] I kworker/4:2 2452213 41862413.326802 237244 120 0.000000 5050.575391 0.000000 0.000000 0 0 /
[Mon Mar 11 14:29:16 2024] I kworker/u97:2 2470473 41862412.807654 186924 120 0.000000 10523.843178 0.000000 0.000000 0 0 /
[Mon Mar 11 14:29:16 2024] I kworker/4:0 2473634 41857072.777179 7 120 0.000000 0.033662 0.000000 0.000000 0 0 /
[Mon Mar 11 14:29:16 2024] I kworker/4:1 2477133 41860198.475875 35 120 0.000000 0.283880 0.000000 0.000000 0 0 /
[Mon Mar 11 14:29:16 2024] S rpcd_spoolss 2477396 31910.264427 62 120 0.000000 90.942701 0.000000 0.000000 0 0 /autogroup-22820
[Mon Mar 11 14:29:16 2024] S nrpe 2478244 208179.039870 3 120 0.000000 15.346552 0.000000 0.000000 0 0 /autogroup-109
[Mon Mar 11 14:29:16 2024] cpu#5, 2600.000 MHz
[Mon Mar 11 14:29:16 2024] .nr_running : 0
[Mon Mar 11 14:29:16 2024] .nr_switches : 1139975259
[Mon Mar 11 14:29:16 2024] .nr_uninterruptible : -4251
[Mon Mar 11 14:29:16 2024] .next_balance : 5577.331859
[Mon Mar 11 14:29:16 2024] .curr->pid : 0
[Mon Mar 11 14:29:16 2024] .clock : 1282630437.242683
[Mon Mar 11 14:29:16 2024] .clock_task : 1280534890.217122
[Mon Mar 11 14:29:16 2024] .avg_idle : 1000000
[Mon Mar 11 14:29:16 2024] .max_idle_balance_cost : 500000
[Mon Mar 11 14:29:16 2024] rt_rq[5]:
[Mon Mar 11 14:29:16 2024] .rt_nr_running : 0
[Mon Mar 11 14:29:16 2024] .rt_nr_migratory : 0
[Mon Mar 11 14:29:16 2024] .rt_throttled : 0
[Mon Mar 11 14:29:16 2024] .rt_time : 0.000000
[Mon Mar 11 14:29:16 2024] .rt_runtime : 950.000000
[Mon Mar 11 14:29:16 2024] dl_rq[5]:
[Mon Mar 11 14:29:16 2024] .dl_nr_running : 0
[Mon Mar 11 14:29:16 2024] .dl_nr_migratory : 0
[Mon Mar 11 14:29:16 2024] .dl_bw->bw : 996147
[Mon Mar 11 14:29:16 2024] .dl_bw->total_bw : 0
[Mon Mar 11 14:29:16 2024] runnable tasks:
[Mon Mar 11 14:29:16 2024] S task PID tree-key switches prio wait-time sum-exec sum-sleep
[Mon Mar 11 14:29:16 2024] -------------------------------------------------------------------------------------------------------------
[Mon Mar 11 14:29:16 2024] S cpuhp/5 48 23840.805171 25 120 0.000000 1.280069 0.000000 0.000000 1 0 /
[Mon Mar 11 14:29:16 2024] S idle_inject/5 49 -8.994369 3 49 0.000000 0.012037 0.000000 0.000000 1 0 /
[Mon Mar 11 14:29:16 2024] S migration/5 50 199.554905 329972 0 0.000000 1864.897757 0.000000 0.000000 1 0 /
[Mon Mar 11 14:29:16 2024] S ksoftirqd/5 51 26058068.522290 12057924 120 0.000000 124246.931181 0.000000 0.000000 1 0 /
[Mon Mar 11 14:29:16 2024] I kworker/5:0H 53 486.560136 4 100 0.000000 0.048330 0.000000 0.000000 1 0 /
[Mon Mar 11 14:29:16 2024] S kswapd1 381 26052246.390816 16339445 120 0.000000 2287795.277913 0.000000 0.000000 1 0 /
[Mon Mar 11 14:29:16 2024] I kworker/5:1H 615 26058122.471557 3249837 100 0.000000 71370.026925 0.000000 0.000000 1 0 /
[Mon Mar 11 14:29:16 2024] S irq/93-lpfc:5 940 0.000000 5610497 49 0.000000 125280.602707 0.000000 0.000000 1 0 /
[Mon Mar 11 14:29:16 2024] S irq/176-lpfc:5 995 0.000000 5616253 49 0.000000 125663.929586 0.000000 0.000000 1 0 /
[Mon Mar 11 14:29:16 2024] I kdmflush/253:52 3838 25119.079266 2 100 0.000000 0.008433 0.000000 0.000000 1 0 /
[Mon Mar 11 14:29:16 2024] S irqbalance 4311 71484.448529 152715 120 0.000000 1573417.962703 0.000000 0.000000 1 0 /autogroup-75
[Mon Mar 11 14:29:16 2024] S collectd 4728 369511.632364 128435 120 0.000000 44485.078293 0.000000 0.000000 1 4728 /autogroup-105
[Mon Mar 11 14:29:16 2024] S reader#0 5333 369523.382421 2793417 120 0.000000 724497.962908 0.000000 0.000000 1 5327 /autogroup-105
[Mon Mar 11 14:29:16 2024] I ext4-rsv-conver 6478 27816.058401 2 100 0.000000 0.062162 0.000000 0.000000 1 0 /
[Mon Mar 11 14:29:16 2024] I rdma_cm 8770 29806.725348 2 100 0.000000 0.063150 0.000000 0.000000 1 0 /
[Mon Mar 11 14:29:16 2024] D nfsd 8876 25969186.179753 11159235552 120 0.000000 289576859.332050 0.000000 0.000000 1 0 /
[Mon Mar 11 14:29:16 2024] S crond 88166 6986.579535 27986 120 0.000000 22611.125469 0.000000 0.000000 1 0 /autogroup-225
[Mon Mar 11 14:29:16 2024] I kworker/5:2 2242731 26058122.483201 197886 120 0.000000 9667.622454 0.000000 0.000000 1 0 /
[Mon Mar 11 14:29:16 2024] S smbd[10.87.18.2 2329756 4881602.958776 1265 120 0.000000 546.789685 0.000000 0.000000 1 0 /autogroup-113
[Mon Mar 11 14:29:16 2024] S smbd[10.87.29.5 2388610 4881139.695164 975 120 0.000000 416.662184 0.000000 0.000000 1 0 /autogroup-113
[Mon Mar 11 14:29:16 2024] S smbd[10.87.29.1 2473572 4881674.998961 48 120 0.000000 42.769145 0.000000 0.000000 1 0 /autogroup-113
[Mon Mar 11 14:29:16 2024] I kworker/5:0 2475005 26043571.627354 4 120 0.000000 0.068560 0.000000 0.000000 1 0 /
[Mon Mar 11 14:29:16 2024] S sshd 2476431 82.921151 28 120 0.000000 85.269690 0.000000 0.000000 1 0 /autogroup-23279
[Mon Mar 11 14:29:16 2024] cpu#6, 2600.000 MHz
[Mon Mar 11 14:29:16 2024] .nr_running : 0
[Mon Mar 11 14:29:16 2024] .nr_switches : 3062242901
[Mon Mar 11 14:29:16 2024] .nr_uninterruptible : -106944
[Mon Mar 11 14:29:16 2024] .next_balance : 5577.331860
[Mon Mar 11 14:29:16 2024] .curr->pid : 0
[Mon Mar 11 14:29:16 2024] .clock : 1282630437.244431
[Mon Mar 11 14:29:16 2024] .clock_task : 1258947745.276602
[Mon Mar 11 14:29:16 2024] .avg_idle : 1000000
[Mon Mar 11 14:29:16 2024] .max_idle_balance_cost : 500000
[Mon Mar 11 14:29:16 2024] cfs_rq[6]:/
[Mon Mar 11 14:29:16 2024] .exec_clock : 0.000000
[Mon Mar 11 14:29:16 2024] .MIN_vruntime : 0.000001
[Mon Mar 11 14:29:16 2024] .min_vruntime : 41504835.555867
[Mon Mar 11 14:29:16 2024] .max_vruntime : 0.000001
[Mon Mar 11 14:29:16 2024] .spread : 0.000000
[Mon Mar 11 14:29:16 2024] .spread0 : 21087187.396155
[Mon Mar 11 14:29:16 2024] .nr_spread_over : 0
[Mon Mar 11 14:29:16 2024] .nr_running : 0
[Mon Mar 11 14:29:16 2024] .h_nr_running : 0
[Mon Mar 11 14:29:16 2024] .idle_nr_running : 0
[Mon Mar 11 14:29:16 2024] .idle_h_nr_running : 0
[Mon Mar 11 14:29:16 2024] .load : 0
[Mon Mar 11 14:29:16 2024] .load_avg : 0
[Mon Mar 11 14:29:16 2024] .runnable_avg : 0
[Mon Mar 11 14:29:16 2024] .util_avg : 0
[Mon Mar 11 14:29:16 2024] .util_est_enqueued : 0
[Mon Mar 11 14:29:16 2024] .removed.load_avg : 0
[Mon Mar 11 14:29:16 2024] .removed.util_avg : 0
[Mon Mar 11 14:29:16 2024] .removed.runnable_avg : 0
[Mon Mar 11 14:29:16 2024] .tg_load_avg_contrib : 0
[Mon Mar 11 14:29:16 2024] .tg_load_avg : 0
[Mon Mar 11 14:29:16 2024] .throttled : 0
[Mon Mar 11 14:29:16 2024] .throttle_count : 0
[Mon Mar 11 14:29:16 2024] rt_rq[6]:
[Mon Mar 11 14:29:16 2024] .rt_nr_running : 0
[Mon Mar 11 14:29:16 2024] .rt_nr_migratory : 0
[Mon Mar 11 14:29:16 2024] .rt_throttled : 0
[Mon Mar 11 14:29:16 2024] .rt_time : 0.000000
[Mon Mar 11 14:29:16 2024] .rt_runtime : 950.000000
[Mon Mar 11 14:29:16 2024] dl_rq[6]:
[Mon Mar 11 14:29:16 2024] .dl_nr_running : 0
[Mon Mar 11 14:29:16 2024] .dl_nr_migratory : 0
[Mon Mar 11 14:29:16 2024] .dl_bw->bw : 996147
[Mon Mar 11 14:29:16 2024] .dl_bw->total_bw : 0
[Mon Mar 11 14:29:16 2024] runnable tasks:
[Mon Mar 11 14:29:16 2024] S task PID tree-key switches prio wait-time sum-exec sum-sleep
[Mon Mar 11 14:29:16 2024] -------------------------------------------------------------------------------------------------------------
[Mon Mar 11 14:29:16 2024] S cpuhp/6 54 23519.575129 25 120 0.000000 1.241424 0.000000 0.000000 0 0 /
[Mon Mar 11 14:29:16 2024] S idle_inject/6 55 -9.000000 3 49 0.000000 0.000000 0.000000 0.000000 0 0 /
[Mon Mar 11 14:29:16 2024] S migration/6 56 250.554887 374824 0 0.000000 1810.588735 0.000000 0.000000 0 0 /
[Mon Mar 11 14:29:16 2024] S ksoftirqd/6 57 41504823.786919 32888344 120 0.000000 355399.772702 0.000000 0.000000 0 0 /
[Mon Mar 11 14:29:16 2024] I kworker/6:0H 59 150.090689 4 100 0.000000 0.073018 0.000000 0.000000 0 0 /
[Mon Mar 11 14:29:16 2024] I mld 405 62.449205 2 100 0.000000 0.066417 0.000000 0.000000 0 0 /
[Mon Mar 11 14:29:16 2024] I kworker/6:1H 613 41504823.556371 4039032 100 0.000000 97076.317301 0.000000 0.000000 0 0 /
[Mon Mar 11 14:29:16 2024] I scsi_tmf_1 898 2354.323831 2 100 0.000000 0.012448 0.000000 0.000000 0 0 /
[Mon Mar 11 14:29:16 2024] S irq/94-lpfc:6 941 0.000000 13430824 49 0.000000 257079.391528 0.000000 0.000000 0 0 /
[Mon Mar 11 14:29:16 2024] S irq/177-lpfc:6 996 0.000000 13450622 49 0.000000 257802.504851 0.000000 0.000000 0 0 /
[Mon Mar 11 14:29:16 2024] I scsi_wq_16 1016 2602.303152 2 100 0.000000 0.062782 0.000000 0.000000 0 0 /
[Mon Mar 11 14:29:16 2024] I kdmflush/253:4 3425 24453.777229 2 100 0.000000 0.062827 0.000000 0.000000 0 0 /
[Mon Mar 11 14:29:16 2024] I kdmflush/253:83 4158 24702.121807 2 100 0.000000 0.019086 0.000000 0.000000 0 0 /
[Mon Mar 11 14:29:16 2024] S jbd2/dm-87-8 4324 41357138.761618 199311 120 0.000000 9861.838368 0.000000 0.000000 0 0 /
[Mon Mar 11 14:29:16 2024] S smbd-cleanupd 4821 2349962.387306 460709 120 0.000000 69235.366569 0.000000 0.000000 0 0 /autogroup-113
[Mon Mar 11 14:29:16 2024] D nfsd 8801 41504830.774732 105987 120 0.000000 11312.552665 0.000000 0.000000 0 0 /
[Mon Mar 11 14:29:16 2024] D nfsd 8827 41498207.481865 4224456 120 0.000000 210282.411687 0.000000 0.000000 0 0 /
[Mon Mar 11 14:29:16 2024] D nfsd 8873 41410770.973687 2421393893 120 0.000000 57569030.172983 0.000000 0.000000 0 0 /
[Mon Mar 11 14:29:16 2024] S automount 88122 11.902441 1 120 0.000000 0.951024 0.000000 0.000000 0 0 /autogroup-222
[Mon Mar 11 14:29:16 2024] S automount 88133 24.817010 1 120 0.000000 0.914576 0.000000 0.000000 0 0 /autogroup-222
[Mon Mar 11 14:29:16 2024] S automount 88144 37.757844 1 120 0.000000 0.940841 0.000000 0.000000 0 0 /autogroup-222
[Mon Mar 11 14:29:16 2024] I kworker/6:1 2330800 41504823.765693 1115575 120 0.000000 24590.300719 0.000000 0.000000 0 0 /
[Mon Mar 11 14:29:16 2024] S smbd[10.40.140. 2381553 2349898.617444 908 120 0.000000 462.246333 0.000000 0.000000 0 0 /autogroup-113
[Mon Mar 11 14:29:16 2024] S smbd[10.87.29.1 2436431 2349991.737764 801 120 0.000000 495.571531 0.000000 0.000000 0 0 /autogroup-113
[Mon Mar 11 14:29:16 2024] I kworker/u97:4 2475262 41504824.588632 95756 120 0.000000 1368.427000 0.000000 0.000000 0 0 /
[Mon Mar 11 14:29:16 2024] I kworker/6:0 2475365 41500429.775511 55 120 0.000000 0.301423 0.000000 0.000000 0 0 /
[Mon Mar 11 14:29:16 2024] cpu#7, 2600.000 MHz
[Mon Mar 11 14:29:16 2024] .nr_running : 0
[Mon Mar 11 14:29:16 2024] .nr_switches : 1112296375
[Mon Mar 11 14:29:16 2024] .nr_uninterruptible : -2901
[Mon Mar 11 14:29:16 2024] .next_balance : 5577.331861
[Mon Mar 11 14:29:16 2024] .curr->pid : 0
[Mon Mar 11 14:29:16 2024] .clock : 1282630437.242950
[Mon Mar 11 14:29:16 2024] .clock_task : 1280675585.100171
[Mon Mar 11 14:29:16 2024] .avg_idle : 1000000
[Mon Mar 11 14:29:16 2024] .max_idle_balance_cost : 500000
[Mon Mar 11 14:29:16 2024] cfs_rq[7]:/autogroup-96
[Mon Mar 11 14:29:16 2024] .exec_clock : 0.000000
[Mon Mar 11 14:29:16 2024] .MIN_vruntime : 0.000001
[Mon Mar 11 14:29:16 2024] .min_vruntime : 21539.185882
[Mon Mar 11 14:29:16 2024] .max_vruntime : 0.000001
[Mon Mar 11 14:29:16 2024] .spread : 0.000000
[Mon Mar 11 14:29:16 2024] .spread0 : -20396108.973830
[Mon Mar 11 14:29:16 2024] .nr_spread_over : 0
[Mon Mar 11 14:29:16 2024] .nr_running : 0
[Mon Mar 11 14:29:16 2024] .h_nr_running : 0
[Mon Mar 11 14:29:16 2024] .idle_nr_running : 0
[Mon Mar 11 14:29:16 2024] .idle_h_nr_running : 0
[Mon Mar 11 14:29:16 2024] .load : 0
[Mon Mar 11 14:29:16 2024] .load_avg : 0
[Mon Mar 11 14:29:16 2024] .runnable_avg : 0
[Mon Mar 11 14:29:16 2024] .util_avg : 0
[Mon Mar 11 14:29:16 2024] .util_est_enqueued : 0
[Mon Mar 11 14:29:16 2024] .removed.load_avg : 0
[Mon Mar 11 14:29:16 2024] .removed.util_avg : 0
[Mon Mar 11 14:29:16 2024] .removed.runnable_avg : 0
[Mon Mar 11 14:29:16 2024] .tg_load_avg_contrib : 0
[Mon Mar 11 14:29:16 2024] .tg_load_avg : 0
[Mon Mar 11 14:29:16 2024] .throttled : 0
[Mon Mar 11 14:29:16 2024] .throttle_count : 0
[Mon Mar 11 14:29:16 2024] .se->exec_start : 1280675389.308501
[Mon Mar 11 14:29:16 2024] .se->vruntime : 24021038.076396
[Mon Mar 11 14:29:16 2024] .se->sum_exec_runtime : 22219.907153
[Mon Mar 11 14:29:16 2024] .se->load.weight : 2
[Mon Mar 11 14:29:16 2024] .se->avg.load_avg : 0
[Mon Mar 11 14:29:16 2024] .se->avg.util_avg : 0
[Mon Mar 11 14:29:16 2024] .se->avg.runnable_avg : 0
[Mon Mar 11 14:29:16 2024] cfs_rq[7]:/
[Mon Mar 11 14:29:16 2024] .exec_clock : 0.000000
[Mon Mar 11 14:29:16 2024] .MIN_vruntime : 0.000001
[Mon Mar 11 14:29:16 2024] .min_vruntime : 24021049.637632
[Mon Mar 11 14:29:16 2024] .max_vruntime : 0.000001
[Mon Mar 11 14:29:16 2024] .spread : 0.000000
[Mon Mar 11 14:29:16 2024] .spread0 : 3603401.477920
[Mon Mar 11 14:29:16 2024] .nr_spread_over : 0
[Mon Mar 11 14:29:16 2024] .nr_running : 0
[Mon Mar 11 14:29:16 2024] .h_nr_running : 0
[Mon Mar 11 14:29:16 2024] .idle_nr_running : 0
[Mon Mar 11 14:29:16 2024] .idle_h_nr_running : 0
[Mon Mar 11 14:29:16 2024] .load : 0
[Mon Mar 11 14:29:16 2024] .load_avg : 0
[Mon Mar 11 14:29:16 2024] .runnable_avg : 0
[Mon Mar 11 14:29:16 2024] .util_avg : 0
[Mon Mar 11 14:29:16 2024] .util_est_enqueued : 0
[Mon Mar 11 14:29:16 2024] .removed.load_avg : 0
[Mon Mar 11 14:29:16 2024] .removed.util_avg : 0
[Mon Mar 11 14:29:16 2024] .removed.runnable_avg : 0
[Mon Mar 11 14:29:16 2024] .tg_load_avg_contrib : 0
[Mon Mar 11 14:29:16 2024] .tg_load_avg : 0
[Mon Mar 11 14:29:16 2024] .throttled : 0
[Mon Mar 11 14:29:16 2024] .throttle_count : 0
[Mon Mar 11 14:29:16 2024] rt_rq[7]:
[Mon Mar 11 14:29:16 2024] .rt_nr_running : 0
[Mon Mar 11 14:29:16 2024] .rt_nr_migratory : 0
[Mon Mar 11 14:29:16 2024] .rt_throttled : 0
[Mon Mar 11 14:29:16 2024] .rt_time : 0.000000
[Mon Mar 11 14:29:16 2024] .rt_runtime : 950.000000
[Mon Mar 11 14:29:16 2024] dl_rq[7]:
[Mon Mar 11 14:29:16 2024] .dl_nr_running : 0
[Mon Mar 11 14:29:16 2024] .dl_nr_migratory : 0
[Mon Mar 11 14:29:16 2024] .dl_bw->bw : 996147
[Mon Mar 11 14:29:16 2024] .dl_bw->total_bw : 0
[Mon Mar 11 14:29:16 2024] runnable tasks:
[Mon Mar 11 14:29:16 2024] S task PID tree-key switches prio wait-time sum-exec sum-sleep
[Mon Mar 11 14:29:16 2024] -------------------------------------------------------------------------------------------------------------
[Mon Mar 11 14:29:16 2024] S cpuhp/7 60 25600.518470 25 120 0.000000 0.759082 0.000000 0.000000 1 0 /
[Mon Mar 11 14:29:16 2024] S idle_inject/7 61 -9.000000 3 49 0.000000 0.000000 0.000000 0.000000 1 0 /
[Mon Mar 11 14:29:16 2024] S migration/7 62 301.554888 329845 0 0.000000 1858.327994 0.000000 0.000000 1 0 /
[Mon Mar 11 14:29:16 2024] S ksoftirqd/7 63 24021037.658927 11704933 120 0.000000 120270.586416 0.000000 0.000000 1 0 /
[Mon Mar 11 14:29:16 2024] I kworker/7:0H 65 73.578695 4 100 0.000000 0.046980 0.000000 0.000000 1 0 /
[Mon Mar 11 14:29:16 2024] S khugepaged 367 24020232.060326 340935 139 0.000000 76892.395465 0.000000 0.000000 1 0 /
[Mon Mar 11 14:29:16 2024] I kworker/7:1H 619 24020664.170955 3126434 100 0.000000 69349.082053 0.000000 0.000000 1 0 /
[Mon Mar 11 14:29:16 2024] I scsi_tmf_0 870 1175.527247 2 100 0.000000 0.006271 0.000000 0.000000 1 0 /
[Mon Mar 11 14:29:16 2024] S irq/95-lpfc:7 942 0.000000 5447227 49 0.000000 122896.733492 0.000000 0.000000 1 0 /
[Mon Mar 11 14:29:16 2024] S irq/178-lpfc:7 997 0.000000 5457646 49 0.000000 123405.516396 0.000000 0.000000 1 0 /
[Mon Mar 11 14:29:16 2024] I kdmflush/253:10 3461 28161.873680 2 100 0.000000 0.012247 0.000000 0.000000 1 0 /
[Mon Mar 11 14:29:16 2024] I kdmflush/253:34 3679 28651.780026 2 100 0.000000 0.013257 0.000000 0.000000 1 0 /
[Mon Mar 11 14:29:16 2024] I kdmflush/253:63 3935 29301.128057 2 100 0.000000 0.012104 0.000000 0.000000 1 0 /
[Mon Mar 11 14:29:16 2024] I kdmflush/253:65 3946 29313.138543 2 100 0.000000 0.011999 0.000000 0.000000 1 0 /
[Mon Mar 11 14:29:16 2024] I kdmflush/253:67 3969 29459.933995 2 100 0.000000 0.010088 0.000000 0.000000 1 0 /
[Mon Mar 11 14:29:16 2024] S systemd-logind 4364 7093.815773 776962 120 0.000000 217926.070558 0.000000 0.000000 1 4364 /autogroup-84
[Mon Mar 11 14:29:16 2024] S f2b/a.samba 4440 21539.185882 2563857 120 0.000000 198600.281732 0.000000 0.000000 1 4440 /autogroup-96
[Mon Mar 11 14:29:16 2024] I kdmflush/253:10 8710 33174.450924 2 100 0.000000 0.011034 0.000000 0.000000 1 0 /
[Mon Mar 11 14:29:16 2024] D nfsd 8862 23953455.845450 35363239 120 0.000000 1419385.003862 0.000000 0.000000 1 0 /
[Mon Mar 11 14:29:16 2024] S automount 88141 3786.846992 651 120 0.000000 64.560537 0.000000 0.000000 1 0 /autogroup-222
[Mon Mar 11 14:29:16 2024] S automount 88147 3787.796525 77 120 0.000000 10.927760 0.000000 0.000000 1 0 /autogroup-222
[Mon Mar 11 14:29:16 2024] S smbd[10.87.28.1 2323430 3210966.245174 555 120 0.000000 167.225211 0.000000 0.000000 1 0 /autogroup-113
[Mon Mar 11 14:29:16 2024] S smbd[10.87.18.2 2338596 3211057.972836 1259 120 0.000000 277.296081 0.000000 0.000000 1 0 /autogroup-113
[Mon Mar 11 14:29:16 2024] S smbd[10.87.29.9 2364174 3211041.499312 1136 120 0.000000 403.241845 0.000000 0.000000 1 0 /autogroup-113
[Mon Mar 11 14:29:16 2024] I kworker/7:0 2391347 24021037.655783 137663 120 0.000000 9623.996685 0.000000 0.000000 1 0 /
[Mon Mar 11 14:29:16 2024] I kworker/u96:4 2442472 24021037.799546 304814 120 0.000000 63891.940913 0.000000 0.000000 1 0 /
[Mon Mar 11 14:29:16 2024] I kworker/7:1 2474775 24000060.421482 209 120 0.000000 3.712917 0.000000 0.000000 1 0 /
[Mon Mar 11 14:29:16 2024] cpu#8, 2600.000 MHz
[Mon Mar 11 14:29:16 2024] .nr_running : 0
[Mon Mar 11 14:29:16 2024] .nr_switches : 3080471161
[Mon Mar 11 14:29:16 2024] .nr_uninterruptible : -30317
[Mon Mar 11 14:29:16 2024] .next_balance : 5577.331860
[Mon Mar 11 14:29:16 2024] .curr->pid : 0
[Mon Mar 11 14:29:16 2024] .clock : 1282630438.242448
[Mon Mar 11 14:29:16 2024] .clock_task : 1256901903.258196
[Mon Mar 11 14:29:16 2024] .avg_idle : 1000000
[Mon Mar 11 14:29:16 2024] .max_idle_balance_cost : 500000
[Mon Mar 11 14:29:16 2024] rt_rq[8]:
[Mon Mar 11 14:29:16 2024] .rt_nr_running : 0
[Mon Mar 11 14:29:19 2024] .rt_nr_migratory : 0
[Mon Mar 11 14:29:19 2024] .rt_throttled : 0
[Mon Mar 11 14:29:19 2024] .rt_time : 0.000000
[Mon Mar 11 14:29:19 2024] .rt_runtime : 950.000000
[Mon Mar 11 14:29:19 2024] dl_rq[8]:
[Mon Mar 11 14:29:19 2024] .dl_nr_running : 0
[Mon Mar 11 14:29:19 2024] .dl_nr_migratory : 0
[Mon Mar 11 14:29:19 2024] .dl_bw->bw : 996147
[Mon Mar 11 14:29:19 2024] .dl_bw->total_bw : 0
[Mon Mar 11 14:29:19 2024] runnable tasks:
[Mon Mar 11 14:29:19 2024] S task PID tree-key switches prio wait-time sum-exec sum-sleep
[Mon Mar 11 14:29:19 2024] -------------------------------------------------------------------------------------------------------------
[Mon Mar 11 14:29:19 2024] S cpuhp/8 66 22813.348156 25 120 0.000000 1.336489 0.000000 0.000000 0 0 /
[Mon Mar 11 14:29:19 2024] S idle_inject/8 67 -12.000000 3 49 0.000000 0.000000 0.000000 0.000000 0 0 /
[Mon Mar 11 14:29:19 2024] S migration/8 68 357.554869 387485 0 0.000000 1970.780585 0.000000 0.000000 0 0 /
[Mon Mar 11 14:29:19 2024] S ksoftirqd/8 69 41993741.123056 30769426 120 0.000000 346907.840181 0.000000 0.000000 0 0 /
[Mon Mar 11 14:29:19 2024] I kworker/8:0H 71 36.241745 4 100 0.000000 0.020097 0.000000 0.000000 0 0 /
[Mon Mar 11 14:29:19 2024] I kworker/8:1H 799 41993740.947888 4052722 100 0.000000 95313.659806 0.000000 0.000000 0 0 /
[Mon Mar 11 14:29:19 2024] S irq/96-lpfc:8 943 0.000000 11960101 49 0.000000 228860.096783 0.000000 0.000000 0 0 /
[Mon Mar 11 14:29:19 2024] S irq/179-lpfc:8 998 0.000000 11993726 49 0.000000 230112.493484 0.000000 0.000000 0 0 /
[Mon Mar 11 14:29:19 2024] I kdmflush/253:29 3632 24087.446820 2 100 0.000000 0.014320 0.000000 0.000000 0 0 /
[Mon Mar 11 14:29:19 2024] I ext4-rsv-conver 4131 24288.073500 2 100 0.000000 0.069462 0.000000 0.000000 0 0 /
[Mon Mar 11 14:29:19 2024] I kdmflush/253:84 4159 24300.086548 2 100 0.000000 0.016590 0.000000 0.000000 0 0 /
[Mon Mar 11 14:29:19 2024] S jbd2/dm-82-8 4242 41972109.154059 3931 120 0.000000 178.158919 0.000000 0.000000 0 0 /
[Mon Mar 11 14:29:19 2024] S jbd2/dm-84-8 4244 41993741.112736 1597998 120 0.000000 212185.515871 0.000000 0.000000 0 0 /
[Mon Mar 11 14:29:19 2024] I ext4-rsv-conver 4278 24611.945622 2 100 0.000000 0.017580 0.000000 0.000000 0 0 /
[Mon Mar 11 14:29:19 2024] S jbd2/dm-25-8 4360 41993741.699910 1066605 120 0.000000 293716.195293 0.000000 0.000000 0 0 /
[Mon Mar 11 14:29:19 2024] D nfsd 8817 41993649.655561 767920 120 0.000000 49042.709876 0.000000 0.000000 0 0 /
[Mon Mar 11 14:29:19 2024] D nfsd 8864 41907025.586779 54018370 120 0.000000 2034308.717942 0.000000 0.000000 0 0 /
[Mon Mar 11 14:29:19 2024] D nfsd 8874 41899093.774416 4328278877 120 0.000000 101222700.873244 0.000000 0.000000 0 0 /
[Mon Mar 11 14:29:19 2024] S cups-browsed 21421 1279.981811 47834 120 0.000000 27899.941482 0.000000 0.000000 0 21421 /autogroup-154
[Mon Mar 11 14:29:19 2024] S automount 88103 171406.346032 677612 120 0.000000 38289.352546 0.000000 0.000000 0 0 /autogroup-222
[Mon Mar 11 14:29:19 2024] S automount 88138 169291.699417 661 120 0.000000 44.329067 0.000000 0.000000 0 0 /autogroup-222
[Mon Mar 11 14:29:19 2024] S smbd[10.86.26.7 93091 2320327.111029 43106 120 0.000000 12089.788779 0.000000 0.000000 0 0 /autogroup-113
[Mon Mar 11 14:29:19 2024] S smbd[10.87.28.8 2326005 2320338.779057 1552 120 0.000000 341.377713 0.000000 0.000000 0 0 /autogroup-113
[Mon Mar 11 14:29:19 2024] S smbd[10.33.22.1 2339568 2320268.034009 107927 120 0.000000 40172.081903 0.000000 0.000000 0 0 /autogroup-113
[Mon Mar 11 14:29:19 2024] S smbd[10.40.157. 2352279 2320235.131961 11030 120 0.000000 2004.617139 0.000000 0.000000 0 0 /autogroup-113
[Mon Mar 11 14:29:19 2024] S smbd[10.40.144. 2399030 2320327.093264 305 120 0.000000 98.948081 0.000000 0.000000 0 0 /autogroup-113
[Mon Mar 11 14:29:19 2024] S smbd[10.87.29.9 2415574 2320267.683136 520 120 0.000000 168.165490 0.000000 0.000000 0 0 /autogroup-113
[Mon Mar 11 14:29:19 2024] I kworker/8:2 2431582 41993741.442363 342847 120 0.000000 12146.515928 0.000000 0.000000 0 0 /
[Mon Mar 11 14:29:19 2024] S smbd[10.87.30.8 2453051 2320268.057994 44409 120 0.000000 8253.224522 0.000000 0.000000 0 0 /autogroup-113
[Mon Mar 11 14:29:19 2024] S smbd[10.40.140. 2472699 2320267.676972 42 120 0.000000 37.048968 0.000000 0.000000 0 0 /autogroup-113
[Mon Mar 11 14:29:19 2024] I kworker/8:1 2474342 41986420.118242 9 120 0.000000 0.128683 0.000000 0.000000 0 0 /
[Mon Mar 11 14:29:19 2024] S smbd[10.46.231. 2476746 2320327.103551 32 120 0.000000 29.047644 0.000000 0.000000 0 0 /autogroup-113
[Mon Mar 11 14:29:19 2024] I kworker/8:0 2477090 41986432.114724 3 120 0.000000 0.062005 0.000000 0.000000 0 0 /
[Mon Mar 11 14:29:19 2024] cpu#9, 2600.000 MHz
[Mon Mar 11 14:29:19 2024] .nr_running : 0
[Mon Mar 11 14:29:19 2024] .nr_switches : 1090744483
[Mon Mar 11 14:29:19 2024] .nr_uninterruptible : -3629
[Mon Mar 11 14:29:19 2024] .next_balance : 5577.334873
[Mon Mar 11 14:29:19 2024] .curr->pid : 0
[Mon Mar 11 14:29:19 2024] .clock : 1282633448.162660
[Mon Mar 11 14:29:19 2024] .clock_task : 1280766998.957531
[Mon Mar 11 14:29:19 2024] .avg_idle : 1000000
[Mon Mar 11 14:29:19 2024] .max_idle_balance_cost : 500000
[Mon Mar 11 14:29:19 2024] cfs_rq[9]:/autogroup-96
[Mon Mar 11 14:29:19 2024] .exec_clock : 0.000000
[Mon Mar 11 14:29:19 2024] .MIN_vruntime : 0.000001
[Mon Mar 11 14:29:19 2024] .min_vruntime : 21979.774007
[Mon Mar 11 14:29:19 2024] .max_vruntime : 0.000001
[Mon Mar 11 14:29:19 2024] .spread : 0.000000
[Mon Mar 11 14:29:19 2024] .spread0 : -20395686.734108
[Mon Mar 11 14:29:19 2024] .nr_spread_over : 0
[Mon Mar 11 14:29:19 2024] .nr_running : 0
[Mon Mar 11 14:29:19 2024] .h_nr_running : 0
[Mon Mar 11 14:29:19 2024] .idle_nr_running : 0
[Mon Mar 11 14:29:19 2024] .idle_h_nr_running : 0
[Mon Mar 11 14:29:19 2024] .load : 0
[Mon Mar 11 14:29:19 2024] .load_avg : 0
[Mon Mar 11 14:29:19 2024] .runnable_avg : 0
[Mon Mar 11 14:29:19 2024] .util_avg : 0
[Mon Mar 11 14:29:19 2024] .util_est_enqueued : 0
[Mon Mar 11 14:29:19 2024] .removed.load_avg : 0
[Mon Mar 11 14:29:19 2024] .removed.util_avg : 0
[Mon Mar 11 14:29:19 2024] .removed.runnable_avg : 0
[Mon Mar 11 14:29:19 2024] .tg_load_avg_contrib : 0
[Mon Mar 11 14:29:19 2024] .tg_load_avg : 0
[Mon Mar 11 14:29:19 2024] .throttled : 0
[Mon Mar 11 14:29:19 2024] .throttle_count : 0
[Mon Mar 11 14:29:19 2024] .se->exec_start : 1280766963.128372
[Mon Mar 11 14:29:19 2024] .se->vruntime : 22917043.656286
[Mon Mar 11 14:29:19 2024] .se->sum_exec_runtime : 23075.337183
[Mon Mar 11 14:29:19 2024] .se->load.weight : 2
[Mon Mar 11 14:29:19 2024] .se->avg.load_avg : 0
[Mon Mar 11 14:29:19 2024] .se->avg.util_avg : 0
[Mon Mar 11 14:29:19 2024] .se->avg.runnable_avg : 0
[Mon Mar 11 14:29:19 2024] cfs_rq[9]:/
[Mon Mar 11 14:29:19 2024] .exec_clock : 0.000000
[Mon Mar 11 14:29:19 2024] .MIN_vruntime : 0.000001
[Mon Mar 11 14:29:19 2024] .min_vruntime : 22917055.569646
[Mon Mar 11 14:29:19 2024] .max_vruntime : 0.000001
[Mon Mar 11 14:29:19 2024] .spread : 0.000000
[Mon Mar 11 14:29:19 2024] .spread0 : 2499389.061531
[Mon Mar 11 14:29:19 2024] .nr_spread_over : 0
[Mon Mar 11 14:29:19 2024] .nr_running : 0
[Mon Mar 11 14:29:19 2024] .h_nr_running : 0
[Mon Mar 11 14:29:19 2024] .idle_nr_running : 0
[Mon Mar 11 14:29:19 2024] .idle_h_nr_running : 0
[Mon Mar 11 14:29:19 2024] .load : 0
[Mon Mar 11 14:29:19 2024] .load_avg : 0
[Mon Mar 11 14:29:19 2024] .runnable_avg : 0
[Mon Mar 11 14:29:19 2024] .util_avg : 0
[Mon Mar 11 14:29:19 2024] .util_est_enqueued : 0
[Mon Mar 11 14:29:19 2024] .removed.load_avg : 0
[Mon Mar 11 14:29:19 2024] .removed.util_avg : 0
[Mon Mar 11 14:29:19 2024] .removed.runnable_avg : 0
[Mon Mar 11 14:29:19 2024] .tg_load_avg_contrib : 0
[Mon Mar 11 14:29:19 2024] .tg_load_avg : 0
[Mon Mar 11 14:29:19 2024] .throttled : 0
[Mon Mar 11 14:29:19 2024] .throttle_count : 0
[Mon Mar 11 14:29:19 2024] rt_rq[9]:
[Mon Mar 11 14:29:19 2024] .rt_nr_running : 0
[Mon Mar 11 14:29:19 2024] .rt_nr_migratory : 0
[Mon Mar 11 14:29:19 2024] .rt_throttled : 0
[Mon Mar 11 14:29:19 2024] .rt_time : 0.000000
[Mon Mar 11 14:29:19 2024] .rt_runtime : 950.000000
[Mon Mar 11 14:29:19 2024] dl_rq[9]:
[Mon Mar 11 14:29:19 2024] .dl_nr_running : 0
[Mon Mar 11 14:29:19 2024] .dl_nr_migratory : 0
[Mon Mar 11 14:29:19 2024] .dl_bw->bw : 996147
[Mon Mar 11 14:29:19 2024] .dl_bw->total_bw : 0
[Mon Mar 11 14:29:19 2024] runnable tasks:
[Mon Mar 11 14:29:19 2024] S task PID tree-key switches prio wait-time sum-exec sum-sleep
[Mon Mar 11 14:29:19 2024] -------------------------------------------------------------------------------------------------------------
[Mon Mar 11 14:29:19 2024] S cpuhp/9 72 29047.055904 25 120 0.000000 0.706423 0.000000 0.000000 1 0 /
[Mon Mar 11 14:29:19 2024] S idle_inject/9 73 -12.000000 3 49 0.000000 0.000000 0.000000 0.000000 1 0 /
[Mon Mar 11 14:29:19 2024] S migration/9 74 425.554870 328399 0 0.000000 1905.062763 0.000000 0.000000 1 0 /
[Mon Mar 11 14:29:19 2024] S ksoftirqd/9 75 22917042.504240 11796005 120 0.000000 118900.184131 0.000000 0.000000 1 0 /
[Mon Mar 11 14:29:19 2024] I kworker/9:0H 77 17.594484 4 100 0.000000 0.031421 0.000000 0.000000 1 0 /
[Mon Mar 11 14:29:19 2024] I kworker/9:1H 798 22917036.281674 3075585 100 0.000000 67882.014782 0.000000 0.000000 1 0 /
[Mon Mar 11 14:29:19 2024] I ata_sff 867 571.058372 2 100 0.000000 0.006700 0.000000 0.000000 1 0 /
[Mon Mar 11 14:29:19 2024] S irq/97-lpfc:9 944 0.000000 5536116 49 0.000000 127272.511024 0.000000 0.000000 1 0 /
[Mon Mar 11 14:29:19 2024] S scsi_eh_15 959 1878.947891 2 120 0.000000 0.009502 0.000000 0.000000 1 0 /
[Mon Mar 11 14:29:19 2024] I scsi_tmf_15 960 1890.957250 2 100 0.000000 0.012156 0.000000 0.000000 1 0 /
[Mon Mar 11 14:29:19 2024] S irq/180-lpfc:9 999 0.000000 5553780 49 0.000000 128033.745593 0.000000 0.000000 1 0 /
[Mon Mar 11 14:29:19 2024] I kdmflush/253:17 3532 30033.921325 2 100 0.000000 0.009476 0.000000 0.000000 1 0 /
[Mon Mar 11 14:29:19 2024] I kdmflush/253:39 3726 30381.573887 2 100 0.000000 0.013348 0.000000 0.000000 1 0 /
[Mon Mar 11 14:29:19 2024] I kdmflush/253:50 3826 30487.698585 2 100 0.000000 0.014397 0.000000 0.000000 1 0 /
[Mon Mar 11 14:29:19 2024] I kdmflush/253:58 3884 30653.471853 2 100 0.000000 0.012816 0.000000 0.000000 1 0 /
[Mon Mar 11 14:29:19 2024] I kdmflush/253:66 3952 30939.385028 2 100 0.000000 0.007816 0.000000 0.000000 1 0 /
[Mon Mar 11 14:29:19 2024] I kdmflush/253:71 3997 31061.969971 2 100 0.000000 0.008695 0.000000 0.000000 1 0 /
[Mon Mar 11 14:29:19 2024] S sssd_nss 4356 221629.645707 38248803 120 0.000000 4923177.836521 0.000000 0.000000 1 4356 /autogroup-78
[Mon Mar 11 14:29:19 2024] S fail2ban-server 4416 21979.774007 641345 120 0.000000 45596.034563 0.000000 0.000000 1 4416 /autogroup-96
[Mon Mar 11 14:29:19 2024] S writer#0 5327 408668.739206 55283017 120 0.000000 1457346.615143 0.000000 0.000000 1 5327 /autogroup-105
[Mon Mar 11 14:29:19 2024] D in:imfile 4838 26394.652682 264187 120 0.000000 34718.483624 0.000000 0.000000 1 4838 /autogroup-112
[Mon Mar 11 14:29:19 2024] S dmeventd 6398 3330.084132 1281468 120 0.000000 40871.730928 0.000000 0.000000 1 0 /autogroup-125
[Mon Mar 11 14:29:19 2024] I ext4-rsv-conver 6490 32974.312305 2 100 0.000000 0.064333 0.000000 0.000000 1 0 /
[Mon Mar 11 14:29:19 2024] D nfsd 8877 22820276.982805 11701419907 120 0.000000 429441809.897510 0.000000 0.000000 1 0 /
[Mon Mar 11 14:29:19 2024] S automount 88157 1642.947080 3 120 0.000000 1.122149 0.000000 0.000000 1 0 /autogroup-222
[Mon Mar 11 14:29:19 2024] S bash 91744 28.774970 30 120 0.000000 37.281288 0.000000 0.000000 1 0 /autogroup-292
[Mon Mar 11 14:29:19 2024] S smbd[10.91.30.5 1834355 2237471.759777 617868 120 0.000000 425214.117932 0.000000 0.000000 1 0 /autogroup-113
[Mon Mar 11 14:29:19 2024] S smbd[10.87.29.6 2371619 2237299.651627 11689 120 0.000000 5215.294441 0.000000 0.000000 1 0 /autogroup-113
[Mon Mar 11 14:29:19 2024] I kworker/9:1 2418267 22917043.584166 112155 120 0.000000 4847.929284 0.000000 0.000000 1 0 /
[Mon Mar 11 14:29:19 2024] I kworker/9:2 2474561 22906344.871472 10 120 0.000000 0.029112 0.000000 0.000000 1 0 /
[Mon Mar 11 14:29:19 2024] cpu#10, 2600.000 MHz
[Mon Mar 11 14:29:19 2024] .nr_running : 0
[Mon Mar 11 14:29:19 2024] .nr_switches : 3077610603
[Mon Mar 11 14:29:19 2024] .nr_uninterruptible : 9252
[Mon Mar 11 14:29:19 2024] .next_balance : 5577.334873
[Mon Mar 11 14:29:19 2024] .curr->pid : 0
[Mon Mar 11 14:29:19 2024] .clock : 1282633448.162887
[Mon Mar 11 14:29:19 2024] .clock_task : 1257146076.810369
[Mon Mar 11 14:29:19 2024] .avg_idle : 1000000
[Mon Mar 11 14:29:19 2024] .max_idle_balance_cost : 500000
[Mon Mar 11 14:29:19 2024] cfs_rq[10]:/
[Mon Mar 11 14:29:19 2024] .exec_clock : 0.000000
[Mon Mar 11 14:29:19 2024] .MIN_vruntime : 0.000001
[Mon Mar 11 14:29:19 2024] .min_vruntime : 42255559.680698
[Mon Mar 11 14:29:19 2024] .max_vruntime : 0.000001
[Mon Mar 11 14:29:19 2024] .spread : 0.000000
[Mon Mar 11 14:29:19 2024] .spread0 : 21837893.172583
[Mon Mar 11 14:29:19 2024] .nr_spread_over : 0
[Mon Mar 11 14:29:19 2024] .nr_running : 0
[Mon Mar 11 14:29:19 2024] .h_nr_running : 0
[Mon Mar 11 14:29:19 2024] .idle_nr_running : 0
[Mon Mar 11 14:29:19 2024] .idle_h_nr_running : 0
[Mon Mar 11 14:29:19 2024] .load : 0
[Mon Mar 11 14:29:19 2024] .load_avg : 0
[Mon Mar 11 14:29:19 2024] .runnable_avg : 0
[Mon Mar 11 14:29:19 2024] .util_avg : 0
[Mon Mar 11 14:29:19 2024] .util_est_enqueued : 0
[Mon Mar 11 14:29:19 2024] .removed.load_avg : 0
[Mon Mar 11 14:29:19 2024] .removed.util_avg : 0
[Mon Mar 11 14:29:19 2024] .removed.runnable_avg : 0
[Mon Mar 11 14:29:19 2024] .tg_load_avg_contrib : 0
[Mon Mar 11 14:29:19 2024] .tg_load_avg : 0
[Mon Mar 11 14:29:19 2024] .throttled : 0
[Mon Mar 11 14:29:19 2024] .throttle_count : 0
[Mon Mar 11 14:29:19 2024] rt_rq[10]:
[Mon Mar 11 14:29:19 2024] .rt_nr_running : 0
[Mon Mar 11 14:29:19 2024] .rt_nr_migratory : 0
[Mon Mar 11 14:29:19 2024] .rt_throttled : 0
[Mon Mar 11 14:29:19 2024] .rt_time : 0.000000
[Mon Mar 11 14:29:19 2024] .rt_runtime : 950.000000
[Mon Mar 11 14:29:19 2024] dl_rq[10]:
[Mon Mar 11 14:29:19 2024] .dl_nr_running : 0
[Mon Mar 11 14:29:19 2024] .dl_nr_migratory : 0
[Mon Mar 11 14:29:19 2024] .dl_bw->bw : 996147
[Mon Mar 11 14:29:19 2024] .dl_bw->total_bw : 0
[Mon Mar 11 14:29:19 2024] runnable tasks:
[Mon Mar 11 14:29:19 2024] S task PID tree-key switches prio wait-time sum-exec sum-sleep
[Mon Mar 11 14:29:19 2024] -------------------------------------------------------------------------------------------------------------
[Mon Mar 11 14:29:19 2024] S cpuhp/10 78 24108.683097 25 120 0.000000 1.152375 0.000000 0.000000 0 0 /
[Mon Mar 11 14:29:19 2024] S idle_inject/10 79 -11.995539 3 49 0.000000 0.009791 0.000000 0.000000 0 0 /
[Mon Mar 11 14:29:19 2024] S migration/10 80 493.554845 385461 0 0.000000 1959.101367 0.000000 0.000000 0 0 /
[Mon Mar 11 14:29:19 2024] S ksoftirqd/10 81 42255547.942246 29617116 120 0.000000 343513.297688 0.000000 0.000000 0 0 /
[Mon Mar 11 14:29:19 2024] I kworker/10:0H 83 262.155786 4 100 0.000000 0.042404 0.000000 0.000000 0 0 /
[Mon Mar 11 14:29:19 2024] S ksmd 366 61.731636 2 125 0.000000 0.000000 0.000000 0.000000 0 0 /
[Mon Mar 11 14:29:19 2024] I kstrp 417 143.241105 2 100 0.000000 0.059071 0.000000 0.000000 0 0 /
[Mon Mar 11 14:29:19 2024] I kworker/10:1H 631 42255547.580610 3803773 100 0.000000 88277.991540 0.000000 0.000000 0 0 /
[Mon Mar 11 14:29:19 2024] S irq/98-lpfc:10 945 0.000000 11076341 49 0.000000 214020.198408 0.000000 0.000000 0 0 /
[Mon Mar 11 14:29:19 2024] S irq/181-lpfc:10 1000 0.000000 11099261 49 0.000000 215041.190507 0.000000 0.000000 0 0 /
[Mon Mar 11 14:29:19 2024] I kmpath_handlerd 3347 25133.557815 2 100 0.000000 0.069959 0.000000 0.000000 0 0 /
[Mon Mar 11 14:29:19 2024] S chronyd 4323 1275.762076 51060 120 0.000000 10054.334465 0.000000 0.000000 0 0 /autogroup-81
[Mon Mar 11 14:29:19 2024] I ext4-rsv-conver 4354 27144.495240 2 100 0.000000 0.049604 0.000000 0.000000 0 0 /
[Mon Mar 11 14:29:19 2024] I ext4-rsv-conver 6468 28116.080474 2 100 0.000000 0.013171 0.000000 0.000000 0 0 /
[Mon Mar 11 14:29:19 2024] D nfsd 8856 42173632.065713 13632169 120 0.000000 603724.437899 0.000000 0.000000 0 0 /
[Mon Mar 11 14:29:19 2024] S automount 88148 54.631573 1 120 0.000000 0.856774 0.000000 0.000000 0 0 /autogroup-222
[Mon Mar 11 14:29:19 2024] S automount 88160 67.523148 5 120 0.000000 0.891582 0.000000 0.000000 0 0 /autogroup-222
[Mon Mar 11 14:29:19 2024] S smbd[10.87.27.5 2292584 2190088.310647 163982 120 0.000000 6384.805423 0.000000 0.000000 0 0 /autogroup-113
[Mon Mar 11 14:29:19 2024] S smbd[10.40.140. 2356004 2189870.240777 12790 120 0.000000 3567.702978 0.000000 0.000000 0 0 /autogroup-113
[Mon Mar 11 14:29:19 2024] I kworker/10:0 2401926 42255547.630933 525694 120 0.000000 17289.892005 0.000000 0.000000 0 0 /
[Mon Mar 11 14:29:19 2024] I kworker/u96:1 2466032 42245270.741957 72826 120 0.000000 23712.342502 0.000000 0.000000 0 0 /
[Mon Mar 11 14:29:19 2024] S smbd[10.87.30.8 2468009 2189815.902585 12594 120 0.000000 4359.969999 0.000000 0.000000 0 0 /autogroup-113
[Mon Mar 11 14:29:19 2024] I kworker/u97:4 2475262 42255550.330981 95797 120 0.000000 1370.146091 0.000000 0.000000 0 0 /
[Mon Mar 11 14:29:19 2024] I kworker/10:2 2475318 42249467.038340 5 120 0.000000 0.085163 0.000000 0.000000 0 0 /
[Mon Mar 11 14:29:19 2024] cpu#11, 2600.000 MHz
[Mon Mar 11 14:29:19 2024] .nr_running : 3
[Mon Mar 11 14:29:19 2024] .nr_switches : 983131110
[Mon Mar 11 14:29:19 2024] .nr_uninterruptible : 3744
[Mon Mar 11 14:29:19 2024] .next_balance : 5577.334927
[Mon Mar 11 14:29:19 2024] .curr->pid : 86
[Mon Mar 11 14:29:19 2024] .clock : 1282633449.082238
[Mon Mar 11 14:29:19 2024] .clock_task : 1280919089.410077
[Mon Mar 11 14:29:19 2024] .avg_idle : 102186
[Mon Mar 11 14:29:19 2024] .max_idle_balance_cost : 500000
[Mon Mar 11 14:29:19 2024] cfs_rq[11]:/
[Mon Mar 11 14:29:19 2024] .exec_clock : 0.000000
[Mon Mar 11 14:29:19 2024] .MIN_vruntime : 20562562.119539
[Mon Mar 11 14:29:19 2024] .min_vruntime : 20562574.119539
[Mon Mar 11 14:29:19 2024] .max_vruntime : 20562562.119539
[Mon Mar 11 14:29:19 2024] .spread : 0.000000
[Mon Mar 11 14:29:19 2024] .spread0 : 144907.611424
[Mon Mar 11 14:29:19 2024] .nr_spread_over : 0
[Mon Mar 11 14:29:19 2024] .nr_running : 1
[Mon Mar 11 14:29:19 2024] .h_nr_running : 1
[Mon Mar 11 14:29:19 2024] .idle_nr_running : 0
[Mon Mar 11 14:29:19 2024] .idle_h_nr_running : 0
[Mon Mar 11 14:29:19 2024] .load : 1048576
[Mon Mar 11 14:29:19 2024] .load_avg : 1024
[Mon Mar 11 14:29:19 2024] .runnable_avg : 1024
[Mon Mar 11 14:29:19 2024] .util_avg : 0
[Mon Mar 11 14:29:19 2024] .util_est_enqueued : 9
[Mon Mar 11 14:29:19 2024] .removed.load_avg : 0
[Mon Mar 11 14:29:19 2024] .removed.util_avg : 0
[Mon Mar 11 14:29:19 2024] .removed.runnable_avg : 0
[Mon Mar 11 14:29:19 2024] .tg_load_avg_contrib : 0
[Mon Mar 11 14:29:19 2024] .tg_load_avg : 0
[Mon Mar 11 14:29:19 2024] .throttled : 0
[Mon Mar 11 14:29:19 2024] .throttle_count : 0
[Mon Mar 11 14:29:19 2024] rt_rq[11]:
[Mon Mar 11 14:29:19 2024] .rt_nr_running : 1
[Mon Mar 11 14:29:19 2024] .rt_nr_migratory : 0
[Mon Mar 11 14:29:19 2024] .rt_throttled : 0
[Mon Mar 11 14:29:19 2024] .rt_time : 0.000000
[Mon Mar 11 14:29:19 2024] .rt_runtime : 950.000000
[Mon Mar 11 14:29:19 2024] dl_rq[11]:
[Mon Mar 11 14:29:19 2024] .dl_nr_running : 0
[Mon Mar 11 14:29:19 2024] .dl_nr_migratory : 0
[Mon Mar 11 14:29:19 2024] .dl_bw->bw : 996147
[Mon Mar 11 14:29:19 2024] .dl_bw->total_bw : 0
[Mon Mar 11 14:29:19 2024] runnable tasks:
[Mon Mar 11 14:29:19 2024] S task PID tree-key switches prio wait-time sum-exec sum-sleep
[Mon Mar 11 14:29:19 2024] -------------------------------------------------------------------------------------------------------------
[Mon Mar 11 14:29:19 2024] S kthreadd 2 20562543.414729 107941 120 0.000000 15887.833398 0.000000 0.000000 1 0 /
[Mon Mar 11 14:29:19 2024] S cpuhp/11 84 22207.434922 25 120 0.000000 1.022700 0.000000 0.000000 1 0 /
[Mon Mar 11 14:29:19 2024] S idle_inject/11 85 -12.000000 3 49 0.000000 0.000000 0.000000 0.000000 1 0 /
[Mon Mar 11 14:29:19 2024] >R migration/11 86 561.554834 327264 0 0.000000 1883.315363 0.000000 0.000000 1 0 /
[Mon Mar 11 14:29:19 2024] S ksoftirqd/11 87 20562550.833718 11080392 120 0.000000 109299.051192 0.000000 0.000000 1 0 /
[Mon Mar 11 14:29:19 2024] I kworker/11:0H 89 29.689160 4 100 0.000000 0.033328 0.000000 0.000000 1 0 /
[Mon Mar 11 14:29:19 2024] I kworker/11:1H 802 20562542.792120 2925885 100 0.000000 64456.062898 0.000000 0.000000 1 0 /
[Mon Mar 11 14:29:19 2024] S irq/99-lpfc:11 946 0.000000 5284428 49 0.000000 120758.038103 0.000000 0.000000 1 0 /
[Mon Mar 11 14:29:19 2024] R irq/182-lpfc:11 1001 0.000000 5293562 49 0.000000 121415.771605 0.000000 0.000000 1 0 /
[Mon Mar 11 14:29:19 2024] I kdmflush/253:12 3481 22915.468222 2 100 0.000000 0.016748 0.000000 0.000000 1 0 /
[Mon Mar 11 14:29:19 2024] I kdmflush/253:37 3706 23441.812951 2 100 0.000000 0.015555 0.000000 0.000000 1 0 /
[Mon Mar 11 14:29:19 2024] I kdmflush/253:49 3817 23486.558578 2 100 0.000000 0.011377 0.000000 0.000000 1 0 /
[Mon Mar 11 14:29:19 2024] I kdmflush/253:57 3883 23537.168495 2 100 0.000000 0.012880 0.000000 0.000000 1 0 /
[Mon Mar 11 14:29:19 2024] S jbd2/dm-89-8 4218 20424617.823835 653 120 0.000000 27.585155 0.000000 0.000000 1 0 /
[Mon Mar 11 14:29:19 2024] S audit_prune_tre 4291 24531.883373 2 120 0.000000 0.014097 0.000000 0.000000 1 0 /
[Mon Mar 11 14:29:19 2024] I kdmflush/253:93 6413 27512.589667 2 100 0.000000 0.063343 0.000000 0.000000 1 0 /
[Mon Mar 11 14:29:19 2024] S upsclient 7162 11.197316 1 120 0.000000 0.245900 0.000000 0.000000 1 0 /autogroup-126
[Mon Mar 11 14:29:19 2024] D nfsd 8849 20507918.442387 10267739 120 0.000000 469683.450533 0.000000 0.000000 1 0 /
[Mon Mar 11 14:29:19 2024] S automount 88137 3593.618790 1456 120 0.000000 119.295303 0.000000 0.000000 1 0 /autogroup-222
[Mon Mar 11 14:29:19 2024] S smbd[134.58.56. 2310377 1686860.839989 609 120 0.000000 183.039133 0.000000 0.000000 1 0 /autogroup-113
[Mon Mar 11 14:29:19 2024] S smbd[10.40.157. 2341830 1686939.465307 4773 120 0.000000 1892.961002 0.000000 0.000000 1 0 /autogroup-113
[Mon Mar 11 14:29:19 2024] S smbd[10.40.143. 2396242 1686944.454449 297 120 0.000000 107.117138 0.000000 0.000000 1 0 /autogroup-113
[Mon Mar 11 14:29:19 2024] R kworker/11:1 2440048 20562562.119539 67677 120 0.000000 2908.935182 0.000000 0.000000 1 0 /
[Mon Mar 11 14:29:19 2024] I kworker/11:0 2474533 20556132.404902 7 120 0.000000 0.031680 0.000000 0.000000 1 0 /
[Mon Mar 11 14:29:19 2024] I kworker/11:2 2477378 20556144.397274 2 120 0.000000 0.009717 0.000000 0.000000 1 0 /
[Mon Mar 11 14:29:19 2024] cpu#12, 2600.000 MHz
[Mon Mar 11 14:29:19 2024] .nr_running : 0
[Mon Mar 11 14:29:19 2024] .nr_switches : 3338589998
[Mon Mar 11 14:29:19 2024] .nr_uninterruptible : 11188
[Mon Mar 11 14:29:19 2024] .next_balance : 5577.334873
[Mon Mar 11 14:29:19 2024] .curr->pid : 0
[Mon Mar 11 14:29:19 2024] .clock : 1282633449.161841
[Mon Mar 11 14:29:19 2024] .clock_task : 1256035048.123324
[Mon Mar 11 14:29:19 2024] .avg_idle : 1000000
[Mon Mar 11 14:29:19 2024] .max_idle_balance_cost : 500000
[Mon Mar 11 14:29:19 2024] cfs_rq[12]:/
[Mon Mar 11 14:29:19 2024] .exec_clock : 0.000000
[Mon Mar 11 14:29:19 2024] .MIN_vruntime : 0.000001
[Mon Mar 11 14:29:19 2024] .min_vruntime : 46210263.511947
[Mon Mar 11 14:29:19 2024] .max_vruntime : 0.000001
[Mon Mar 11 14:29:19 2024] .spread : 0.000000
[Mon Mar 11 14:29:19 2024] .spread0 : 25792597.003832
[Mon Mar 11 14:29:19 2024] .nr_spread_over : 0
[Mon Mar 11 14:29:19 2024] .nr_running : 0
[Mon Mar 11 14:29:19 2024] .h_nr_running : 0
[Mon Mar 11 14:29:19 2024] .idle_nr_running : 0
[Mon Mar 11 14:29:19 2024] .idle_h_nr_running : 0
[Mon Mar 11 14:29:19 2024] .load : 0
[Mon Mar 11 14:29:19 2024] .load_avg : 0
[Mon Mar 11 14:29:19 2024] .runnable_avg : 0
[Mon Mar 11 14:29:19 2024] .util_avg : 0
[Mon Mar 11 14:29:19 2024] .util_est_enqueued : 0
[Mon Mar 11 14:29:19 2024] .removed.load_avg : 0
[Mon Mar 11 14:29:19 2024] .removed.util_avg : 0
[Mon Mar 11 14:29:19 2024] .removed.runnable_avg : 0
[Mon Mar 11 14:29:19 2024] .tg_load_avg_contrib : 0
[Mon Mar 11 14:29:19 2024] .tg_load_avg : 0
[Mon Mar 11 14:29:19 2024] .throttled : 0
[Mon Mar 11 14:29:19 2024] .throttle_count : 0
[Mon Mar 11 14:29:19 2024] rt_rq[12]:
[Mon Mar 11 14:29:19 2024] .rt_nr_running : 0
[Mon Mar 11 14:29:19 2024] .rt_nr_migratory : 0
[Mon Mar 11 14:29:19 2024] .rt_throttled : 0
[Mon Mar 11 14:29:19 2024] .rt_time : 0.000000
[Mon Mar 11 14:29:19 2024] .rt_runtime : 950.000000
[Mon Mar 11 14:29:19 2024] dl_rq[12]:
[Mon Mar 11 14:29:19 2024] .dl_nr_running : 0
[Mon Mar 11 14:29:19 2024] .dl_nr_migratory : 0
[Mon Mar 11 14:29:19 2024] .dl_bw->bw : 996147
[Mon Mar 11 14:29:19 2024] .dl_bw->total_bw : 0
[Mon Mar 11 14:29:19 2024] runnable tasks:
[Mon Mar 11 14:29:19 2024] S task PID tree-key switches prio wait-time sum-exec sum-sleep
[Mon Mar 11 14:29:19 2024] -------------------------------------------------------------------------------------------------------------
[Mon Mar 11 14:29:19 2024] S cpuhp/12 90 25731.680618 25 120 0.000000 0.983012 0.000000 0.000000 0 0 /
[Mon Mar 11 14:29:19 2024] S idle_inject/12 91 -12.000000 3 49 0.000000 0.000000 0.000000 0.000000 0 0 /
[Mon Mar 11 14:29:19 2024] S migration/12 92 629.555545 374149 0 0.000000 1826.965259 0.000000 0.000000 0 0 /
[Mon Mar 11 14:29:19 2024] S ksoftirqd/12 93 46210251.580863 29657772 120 0.000000 358635.023831 0.000000 0.000000 0 0 /
[Mon Mar 11 14:29:19 2024] I kworker/12:0H 95 1.915455 4 100 0.000000 0.051643 0.000000 0.000000 0 0 /
[Mon Mar 11 14:29:19 2024] I md 373 25.354222 2 100 0.000000 0.000000 0.000000 0.000000 0 0 /
[Mon Mar 11 14:29:19 2024] S kswapd0 380 46208966.307891 40235627 120 0.000000 6355971.113823 0.000000 0.000000 0 0 /
[Mon Mar 11 14:29:19 2024] I kworker/12:1H 747 46210251.512380 3697677 100 0.000000 83453.417349 0.000000 0.000000 0 0 /
[Mon Mar 11 14:29:19 2024] S irq/100-lpfc:12 947 0.000000 10590023 49 0.000000 203345.694616 0.000000 0.000000 0 0 /
[Mon Mar 11 14:29:19 2024] S irq/183-lpfc:12 1002 0.000000 10613815 49 0.000000 204134.348373 0.000000 0.000000 0 0 /
[Mon Mar 11 14:29:19 2024] S upsclient 7160 11.218298 1 120 0.000000 0.266883 0.000000 0.000000 0 0 /autogroup-126
[Mon Mar 11 14:29:19 2024] D nfsd 8788 46210252.315939 77325 120 0.000000 8906.084762 0.000000 0.000000 0 0 /
[Mon Mar 11 14:29:19 2024] D nfsd 8805 46210255.242512 119022 120 0.000000 13064.632830 0.000000 0.000000 0 0 /
[Mon Mar 11 14:29:19 2024] S automount 88102 169945.320216 207379 120 0.000000 11302.113336 0.000000 0.000000 0 0 /autogroup-222
[Mon Mar 11 14:29:19 2024] S automount 88132 11.893369 2 120 0.000000 0.941952 0.000000 0.000000 0 0 /autogroup-222
[Mon Mar 11 14:29:19 2024] I nfsiod 179473 542438.112080 2 100 0.000000 0.010287 0.000000 0.000000 0 0 /
[Mon Mar 11 14:29:19 2024] S smbd[10.87.29.1 2304037 2066706.191575 669 120 0.000000 189.555346 0.000000 0.000000 0 0 /autogroup-113
[Mon Mar 11 14:29:19 2024] I kworker/12:0 2442877 46210251.571222 190481 120 0.000000 10075.320554 0.000000 0.000000 0 0 /
[Mon Mar 11 14:29:19 2024] D kworker/u96:3 2451130 46210251.545939 129668 120 0.000000 29904.483577 0.000000 0.000000 0 0 /
[Mon Mar 11 14:29:19 2024] I kworker/12:2 2473001 46197648.762509 587 120 0.000000 8.733456 0.000000 0.000000 0 0 /
[Mon Mar 11 14:29:19 2024] I kworker/12:1 2476314 46208506.263384 27 120 0.000000 0.226203 0.000000 0.000000 0 0 /
[Mon Mar 11 14:29:19 2024] S rpcd_spoolss 2477400 22505.277956 58 120 0.000000 76.820899 0.000000 0.000000 0 0 /autogroup-22820
[Mon Mar 11 14:29:19 2024] S rpcd_winreg 2477414 22506.245189 124 120 0.000000 72.379947 0.000000 0.000000 0 0 /autogroup-22820
[Mon Mar 11 14:29:19 2024] cpu#13, 2600.000 MHz
[Mon Mar 11 14:29:19 2024] .nr_running : 0
[Mon Mar 11 14:29:19 2024] .nr_switches : 1036523359
[Mon Mar 11 14:29:19 2024] .nr_uninterruptible : -5000
[Mon Mar 11 14:29:19 2024] .next_balance : 5577.334873
[Mon Mar 11 14:29:19 2024] .curr->pid : 0
[Mon Mar 11 14:29:19 2024] .clock : 1282633449.161350
[Mon Mar 11 14:29:19 2024] .clock_task : 1280928849.652503
[Mon Mar 11 14:29:19 2024] .avg_idle : 1000000
[Mon Mar 11 14:29:19 2024] .max_idle_balance_cost : 500000
[Mon Mar 11 14:29:19 2024] cfs_rq[13]:/autogroup-90
[Mon Mar 11 14:29:19 2024] .exec_clock : 0.000000
[Mon Mar 11 14:29:19 2024] .MIN_vruntime : 0.000001
[Mon Mar 11 14:29:19 2024] .min_vruntime : 5454.450532
[Mon Mar 11 14:29:19 2024] .max_vruntime : 0.000001
[Mon Mar 11 14:29:19 2024] .spread : 0.000000
[Mon Mar 11 14:29:19 2024] .spread0 : -20412212.057583
[Mon Mar 11 14:29:19 2024] .nr_spread_over : 0
[Mon Mar 11 14:29:19 2024] .nr_running : 0
[Mon Mar 11 14:29:19 2024] .h_nr_running : 0
[Mon Mar 11 14:29:19 2024] .idle_nr_running : 0
[Mon Mar 11 14:29:19 2024] .idle_h_nr_running : 0
[Mon Mar 11 14:29:19 2024] .load : 0
[Mon Mar 11 14:29:19 2024] .load_avg : 0
[Mon Mar 11 14:29:19 2024] .runnable_avg : 0
[Mon Mar 11 14:29:19 2024] .util_avg : 0
[Mon Mar 11 14:29:19 2024] .util_est_enqueued : 0
[Mon Mar 11 14:29:19 2024] .removed.load_avg : 0
[Mon Mar 11 14:29:19 2024] .removed.util_avg : 0
[Mon Mar 11 14:29:19 2024] .removed.runnable_avg : 0
[Mon Mar 11 14:29:19 2024] .tg_load_avg_contrib : 0
[Mon Mar 11 14:29:19 2024] .tg_load_avg : 0
[Mon Mar 11 14:29:19 2024] .throttled : 0
[Mon Mar 11 14:29:19 2024] .throttle_count : 0
[Mon Mar 11 14:29:19 2024] .se->exec_start : 1280928745.080027
[Mon Mar 11 14:29:19 2024] .se->vruntime : 20987299.723570
[Mon Mar 11 14:29:19 2024] .se->sum_exec_runtime : 5296.369357
[Mon Mar 11 14:29:19 2024] .se->load.weight : 2
[Mon Mar 11 14:29:19 2024] .se->avg.load_avg : 0
[Mon Mar 11 14:29:19 2024] .se->avg.util_avg : 0
[Mon Mar 11 14:29:19 2024] .se->avg.runnable_avg : 0
[Mon Mar 11 14:29:19 2024] cfs_rq[13]:/
[Mon Mar 11 14:29:19 2024] .exec_clock : 0.000000
[Mon Mar 11 14:29:19 2024] .MIN_vruntime : 0.000001
[Mon Mar 11 14:29:19 2024] .min_vruntime : 20987311.625055
[Mon Mar 11 14:29:19 2024] .max_vruntime : 0.000001
[Mon Mar 11 14:29:19 2024] .spread : 0.000000
[Mon Mar 11 14:29:19 2024] .spread0 : 569645.116940
[Mon Mar 11 14:29:19 2024] .nr_spread_over : 0
[Mon Mar 11 14:29:19 2024] .nr_running : 0
[Mon Mar 11 14:29:19 2024] .h_nr_running : 0
[Mon Mar 11 14:29:19 2024] .idle_nr_running : 0
[Mon Mar 11 14:29:19 2024] .idle_h_nr_running : 0
[Mon Mar 11 14:29:19 2024] .load : 0
[Mon Mar 11 14:29:19 2024] .load_avg : 0
[Mon Mar 11 14:29:19 2024] .runnable_avg : 0
[Mon Mar 11 14:29:19 2024] .util_avg : 0
[Mon Mar 11 14:29:19 2024] .util_est_enqueued : 0
[Mon Mar 11 14:29:19 2024] .removed.load_avg : 0
[Mon Mar 11 14:29:19 2024] .removed.util_avg : 0
[Mon Mar 11 14:29:19 2024] .removed.runnable_avg : 0
[Mon Mar 11 14:29:19 2024] .tg_load_avg_contrib : 0
[Mon Mar 11 14:29:19 2024] .tg_load_avg : 0
[Mon Mar 11 14:29:19 2024] .throttled : 0
[Mon Mar 11 14:29:19 2024] .throttle_count : 0
[Mon Mar 11 14:29:19 2024] rt_rq[13]:
[Mon Mar 11 14:29:19 2024] .rt_nr_running : 0
[Mon Mar 11 14:29:19 2024] .rt_nr_migratory : 0
[Mon Mar 11 14:29:19 2024] .rt_throttled : 0
[Mon Mar 11 14:29:19 2024] .rt_time : 0.000000
[Mon Mar 11 14:29:19 2024] .rt_runtime : 950.000000
[Mon Mar 11 14:29:19 2024] dl_rq[13]:
[Mon Mar 11 14:29:19 2024] .dl_nr_running : 0
[Mon Mar 11 14:29:19 2024] .dl_nr_migratory : 0
[Mon Mar 11 14:29:19 2024] .dl_bw->bw : 996147
[Mon Mar 11 14:29:19 2024] .dl_bw->total_bw : 0
[Mon Mar 11 14:29:19 2024] runnable tasks:
[Mon Mar 11 14:29:19 2024] S task PID tree-key switches prio wait-time sum-exec sum-sleep
[Mon Mar 11 14:29:19 2024] -------------------------------------------------------------------------------------------------------------
[Mon Mar 11 14:29:19 2024] S cpuhp/13 96 26515.315734 25 120 0.000000 0.688334 0.000000 0.000000 1 0 /
[Mon Mar 11 14:29:19 2024] S idle_inject/13 97 -12.000000 3 49 0.000000 0.000000 0.000000 0.000000 1 0 /
[Mon Mar 11 14:29:19 2024] S migration/13 98 697.555545 326310 0 0.000000 1881.477268 0.000000 0.000000 1 0 /
[Mon Mar 11 14:29:19 2024] S ksoftirqd/13 99 20987289.672704 11269408 120 0.000000 112124.135999 0.000000 0.000000 1 0 /
[Mon Mar 11 14:29:19 2024] I kworker/13:0H 101 90.426995 4 100 0.000000 0.026951 0.000000 0.000000 1 0 /
[Mon Mar 11 14:29:19 2024] S kdevtmpfs 354 516771.568484 710 120 0.000000 18.397420 0.000000 0.000000 1 0 /
[Mon Mar 11 14:29:19 2024] S kcompactd1 365 20986812.257170 16436853 120 0.000000 16724122.224337 0.000000 0.000000 1 0 /
[Mon Mar 11 14:29:19 2024] I kworker/13:1H 834 20987176.604530 2997375 100 0.000000 65602.061173 0.000000 0.000000 1 0 /
[Mon Mar 11 14:29:19 2024] S irq/101-lpfc:13 948 0.000000 5357741 49 0.000000 119902.659859 0.000000 0.000000 1 0 /
[Mon Mar 11 14:29:19 2024] S irq/184-lpfc:13 1003 0.000000 5364473 49 0.000000 120366.136674 0.000000 0.000000 1 0 /
[Mon Mar 11 14:29:19 2024] I kdmflush/253:35 3685 27334.473742 2 100 0.000000 0.008046 0.000000 0.000000 1 0 /
[Mon Mar 11 14:29:19 2024] I kdmflush/253:55 3864 27991.061987 2 100 0.000000 0.006029 0.000000 0.000000 1 0 /
[Mon Mar 11 14:29:19 2024] I ext4-rsv-conver 4272 28712.301627 2 100 0.000000 0.016582 0.000000 0.000000 1 0 /
[Mon Mar 11 14:29:19 2024] S uwsgi 4405 5454.450532 1281793 120 0.000000 84916.178768 0.000000 0.000000 1 0 /autogroup-90
[Mon Mar 11 14:29:19 2024] S wb[KWAK] 4668 197262.716916 44244 120 0.000000 2305.033634 0.000000 0.000000 1 0 /autogroup-95
[Mon Mar 11 14:29:19 2024] S reader#1 5334 425719.681523 2839491 120 0.000000 733106.796630 0.000000 0.000000 1 5327 /autogroup-105
[Mon Mar 11 14:29:19 2024] S rpc.mountd 5127 99005.393330 29154322 120 0.000000 2787512.543888 0.000000 0.000000 1 0 /autogroup-116
[Mon Mar 11 14:29:19 2024] S nsrexecd 5343 9856.559984 56447 120 0.000000 11562.303259 0.000000 0.000000 1 5343 /autogroup-117
[Mon Mar 11 14:29:19 2024] D nfsd 8842 20919524.034021 3343927 120 0.000000 102718.189630 0.000000 0.000000 1 0 /
[Mon Mar 11 14:29:19 2024] D nfsd 8854 20923697.512266 11372684 120 0.000000 513772.396025 0.000000 0.000000 1 0 /
[Mon Mar 11 14:29:19 2024] S samba-dcerpcd 2339807 283330.761782 3017914 120 0.000000 473688.121149 0.000000 0.000000 1 0 /autogroup-22820
[Mon Mar 11 14:29:19 2024] S smbd[10.46.198. 2368564 1470206.698971 428 120 0.000000 144.529590 0.000000 0.000000 1 0 /autogroup-113
[Mon Mar 11 14:29:19 2024] I kworker/13:0 2391582 20987299.648567 149490 120 0.000000 10192.771420 0.000000 0.000000 1 0 /
[Mon Mar 11 14:29:19 2024] S smbd[10.40.138. 2445114 1470255.148564 124 120 0.000000 60.487331 0.000000 0.000000 1 0 /autogroup-113
[Mon Mar 11 14:29:19 2024] I kworker/13:2 2475727 20979233.554358 5 120 0.000000 0.068236 0.000000 0.000000 1 0 /
[Mon Mar 11 14:29:19 2024] S sudo 2478247 231296.670850 7 120 0.000000 27.938848 0.000000 0.000000 1 0 /autogroup-109
[Mon Mar 11 14:29:19 2024] cpu#14, 2600.000 MHz
[Mon Mar 11 14:29:19 2024] .nr_running : 5
[Mon Mar 11 14:29:19 2024] .nr_switches : 2948085043
[Mon Mar 11 14:29:19 2024] .nr_uninterruptible : 29866
[Mon Mar 11 14:29:19 2024] .next_balance : 5577.334904
[Mon Mar 11 14:29:19 2024] .curr->pid : 2476436
[Mon Mar 11 14:29:19 2024] .clock : 1282633449.160128
[Mon Mar 11 14:29:19 2024] .clock_task : 1259194347.476126
[Mon Mar 11 14:29:19 2024] .avg_idle : 837639
[Mon Mar 11 14:29:19 2024] .max_idle_balance_cost : 500000
[Mon Mar 11 14:29:19 2024] cfs_rq[14]:/autogroup-23280
[Mon Mar 11 14:29:19 2024] .exec_clock : 0.000000
[Mon Mar 11 14:29:19 2024] .MIN_vruntime : 0.000001
[Mon Mar 11 14:29:19 2024] .min_vruntime : 9788.922277
[Mon Mar 11 14:29:19 2024] .max_vruntime : 0.000001
[Mon Mar 11 14:29:19 2024] .spread : 0.000000
[Mon Mar 11 14:29:19 2024] .spread0 : -20407877.585838
[Mon Mar 11 14:29:19 2024] .nr_spread_over : 0
[Mon Mar 11 14:29:19 2024] .nr_running : 1
[Mon Mar 11 14:29:19 2024] .h_nr_running : 1
[Mon Mar 11 14:29:19 2024] .idle_nr_running : 0
[Mon Mar 11 14:29:19 2024] .idle_h_nr_running : 0
[Mon Mar 11 14:29:19 2024] .load : 1048576
[Mon Mar 11 14:29:19 2024] .load_avg : 1024
[Mon Mar 11 14:29:19 2024] .runnable_avg : 1024
[Mon Mar 11 14:29:19 2024] .util_avg : 1024
[Mon Mar 11 14:29:19 2024] .util_est_enqueued : 0
[Mon Mar 11 14:29:19 2024] .removed.load_avg : 0
[Mon Mar 11 14:29:19 2024] .removed.util_avg : 0
[Mon Mar 11 14:29:19 2024] .removed.runnable_avg : 0
[Mon Mar 11 14:29:19 2024] .tg_load_avg_contrib : 1024
[Mon Mar 11 14:29:19 2024] .tg_load_avg : 1024
[Mon Mar 11 14:29:19 2024] .throttled : 0
[Mon Mar 11 14:29:19 2024] .throttle_count : 0
[Mon Mar 11 14:29:19 2024] .se->exec_start : 1259194347.474548
[Mon Mar 11 14:29:19 2024] .se->vruntime : 41124696.732550
[Mon Mar 11 14:29:19 2024] .se->sum_exec_runtime : 9777.970860
[Mon Mar 11 14:29:19 2024] .se->load.weight : 1048576
[Mon Mar 11 14:29:19 2024] .se->avg.load_avg : 1024
[Mon Mar 11 14:29:19 2024] .se->avg.util_avg : 1024
[Mon Mar 11 14:29:19 2024] .se->avg.runnable_avg : 1024
[Mon Mar 11 14:29:19 2024] cfs_rq[14]:/
[Mon Mar 11 14:29:19 2024] .exec_clock : 0.000000
[Mon Mar 11 14:29:19 2024] .MIN_vruntime : 41114924.866591
[Mon Mar 11 14:29:19 2024] .min_vruntime : 41114936.866591
[Mon Mar 11 14:29:19 2024] .max_vruntime : 41114924.879998
[Mon Mar 11 14:29:19 2024] .spread : 0.013407
[Mon Mar 11 14:29:19 2024] .spread0 : 20697270.358476
[Mon Mar 11 14:29:19 2024] .nr_spread_over : 0
[Mon Mar 11 14:29:19 2024] .nr_running : 4
[Mon Mar 11 14:29:19 2024] .h_nr_running : 4
[Mon Mar 11 14:29:19 2024] .idle_nr_running : 0
[Mon Mar 11 14:29:19 2024] .idle_h_nr_running : 0
[Mon Mar 11 14:29:19 2024] .load : 94036992
[Mon Mar 11 14:29:19 2024] .load_avg : 91190
[Mon Mar 11 14:29:19 2024] .runnable_avg : 4098
[Mon Mar 11 14:29:19 2024] .util_avg : 1024
[Mon Mar 11 14:29:19 2024] .util_est_enqueued : 34
[Mon Mar 11 14:29:19 2024] .removed.load_avg : 0
[Mon Mar 11 14:29:19 2024] .removed.util_avg : 0
[Mon Mar 11 14:29:19 2024] .removed.runnable_avg : 0
[Mon Mar 11 14:29:19 2024] .tg_load_avg_contrib : 0
[Mon Mar 11 14:29:19 2024] .tg_load_avg : 0
[Mon Mar 11 14:29:19 2024] .throttled : 0
[Mon Mar 11 14:29:19 2024] .throttle_count : 0
[Mon Mar 11 14:29:19 2024] rt_rq[14]:
[Mon Mar 11 14:29:19 2024] .rt_nr_running : 0
[Mon Mar 11 14:29:19 2024] .rt_nr_migratory : 0
[Mon Mar 11 14:29:19 2024] .rt_throttled : 0
[Mon Mar 11 14:29:19 2024] .rt_time : 0.000000
[Mon Mar 11 14:29:19 2024] .rt_runtime : 950.000000
[Mon Mar 11 14:29:19 2024] dl_rq[14]:
[Mon Mar 11 14:29:19 2024] .dl_nr_running : 0
[Mon Mar 11 14:29:19 2024] .dl_nr_migratory : 0
[Mon Mar 11 14:29:19 2024] .dl_bw->bw : 996147
[Mon Mar 11 14:29:19 2024] .dl_bw->total_bw : 0
[Mon Mar 11 14:29:19 2024] runnable tasks:
[Mon Mar 11 14:29:19 2024] S task PID tree-key switches prio wait-time sum-exec sum-sleep
[Mon Mar 11 14:29:19 2024] -------------------------------------------------------------------------------------------------------------
[Mon Mar 11 14:29:19 2024] S cpuhp/14 102 22719.181663 25 120 0.000000 1.249714 0.000000 0.000000 0 0 /
[Mon Mar 11 14:29:19 2024] S idle_inject/14 103 -12.000000 3 49 0.000000 0.000000 0.000000 0.000000 0 0 /
[Mon Mar 11 14:29:19 2024] R migration/14 104 765.555520 376699 0 0.000000 1916.601274 0.000000 0.000000 0 0 /
[Mon Mar 11 14:29:19 2024] R ksoftirqd/14 105 41114924.866591 25737578 120 0.000000 309956.742814 0.000000 0.000000 0 0 /
[Mon Mar 11 14:29:19 2024] I kworker/14:0H 107 56.763093 4 100 0.000000 0.136773 0.000000 0.000000 0 0 /
[Mon Mar 11 14:29:19 2024] R kworker/14:1H 752 41114924.866591 3470597 100 0.000000 78490.546961 0.000000 0.000000 0 0 /
[Mon Mar 11 14:29:19 2024] S irq/102-lpfc:14 949 0.000000 9526323 49 0.000000 186895.913108 0.000000 0.000000 0 0 /
[Mon Mar 11 14:29:19 2024] S irq/185-lpfc:14 1004 0.000000 9549881 49 0.000000 187545.281753 0.000000 0.000000 0 0 /
[Mon Mar 11 14:29:19 2024] I kdmflush/253:30 3648 24153.462119 2 100 0.000000 0.012569 0.000000 0.000000 0 0 /
[Mon Mar 11 14:29:19 2024] S nsrexecd 5349 5354.579903 1307 120 0.000000 378.281657 0.000000 0.000000 0 5349 /autogroup-117
[Mon Mar 11 14:29:19 2024] D nfsd 8830 41098507.541959 3085393 120 0.000000 165356.212779 0.000000 0.000000 0 0 /
[Mon Mar 11 14:29:19 2024] D nfsd 8866 41029179.219205 90894108 120 0.000000 3155404.032331 0.000000 0.000000 0 0 /
[Mon Mar 11 14:29:19 2024] S automount 88128 11.889018 1 120 0.000000 0.937601 0.000000 0.000000 0 0 /autogroup-222
[Mon Mar 11 14:29:19 2024] S automount 88142 24.776843 1 120 0.000000 0.887832 0.000000 0.000000 0 0 /autogroup-222
[Mon Mar 11 14:29:19 2024] S automount 88153 37.663462 1 120 0.000000 0.886626 0.000000 0.000000 0 0 /autogroup-222
[Mon Mar 11 14:29:19 2024] S automount 88162 50.497713 1 120 0.000000 0.834258 0.000000 0.000000 0 0 /autogroup-222
[Mon Mar 11 14:29:19 2024] S smbd[10.87.18.3 2342764 2392344.215358 15522 120 0.000000 3813.478027 0.000000 0.000000 0 0 /autogroup-113
[Mon Mar 11 14:29:19 2024] S smbd[10.87.18.2 2347082 2392345.221014 681 120 0.000000 211.301623 0.000000 0.000000 0 0 /autogroup-113
[Mon Mar 11 14:29:19 2024] I kworker/14:1 2374246 41111361.615456 465506 120 0.000000 16576.849713 0.000000 0.000000 0 0 /
[Mon Mar 11 14:29:19 2024] R kworker/14:2 2474614 41114924.879998 18507 120 0.000000 545.498992 0.000000 0.000000 0 0 /
[Mon Mar 11 14:29:19 2024] >R bash 2476436 9788.922277 119 120 0.000000 9851.190462 0.000000 0.000000 0 0 /autogroup-23280
[Mon Mar 11 14:29:19 2024] I kworker/14:0 2477542 41113314.286995 3 120 0.000000 0.066917 0.000000 0.000000 0 0 /
[Mon Mar 11 14:29:19 2024] cpu#15, 2600.000 MHz
[Mon Mar 11 14:29:19 2024] .nr_running : 0
[Mon Mar 11 14:29:19 2024] .nr_switches : 1005229434
[Mon Mar 11 14:29:19 2024] .nr_uninterruptible : -3188
[Mon Mar 11 14:29:19 2024] .next_balance : 5577.334872
[Mon Mar 11 14:29:19 2024] .curr->pid : 0
[Mon Mar 11 14:29:19 2024] .clock : 1282633449.160962
[Mon Mar 11 14:29:19 2024] .clock_task : 1280982883.172370
[Mon Mar 11 14:29:19 2024] .avg_idle : 1000000
[Mon Mar 11 14:29:19 2024] .max_idle_balance_cost : 500000
[Mon Mar 11 14:29:19 2024] rt_rq[15]:
[Mon Mar 11 14:29:19 2024] .rt_nr_running : 0
[Mon Mar 11 14:29:19 2024] .rt_nr_migratory : 0
[Mon Mar 11 14:29:19 2024] .rt_throttled : 0
[Mon Mar 11 14:29:19 2024] .rt_time : 0.000000
[Mon Mar 11 14:29:19 2024] .rt_runtime : 950.000000
[Mon Mar 11 14:29:19 2024] dl_rq[15]:
[Mon Mar 11 14:29:19 2024] .dl_nr_running : 0
[Mon Mar 11 14:29:19 2024] .dl_nr_migratory : 0
[Mon Mar 11 14:29:19 2024] .dl_bw->bw : 996147
[Mon Mar 11 14:29:19 2024] .dl_bw->total_bw : 0
[Mon Mar 11 14:29:19 2024] runnable tasks:
[Mon Mar 11 14:29:19 2024] S task PID tree-key switches prio wait-time sum-exec sum-sleep
[Mon Mar 11 14:29:19 2024] -------------------------------------------------------------------------------------------------------------
[Mon Mar 11 14:29:19 2024] S cpuhp/15 108 22016.621560 25 120 0.000000 0.986302 0.000000 0.000000 1 0 /
[Mon Mar 11 14:29:19 2024] S idle_inject/15 109 -11.994113 3 49 0.000000 0.012580 0.000000 0.000000 1 0 /
[Mon Mar 11 14:29:19 2024] S migration/15 110 833.555516 325620 0 0.000000 1875.252549 0.000000 0.000000 1 0 /
[Mon Mar 11 14:29:19 2024] S ksoftirqd/15 111 20127088.769605 11165480 120 0.000000 111030.425195 0.000000 0.000000 1 0 /
[Mon Mar 11 14:29:19 2024] I kworker/15:0H 113 379.039483 4 100 0.000000 0.028546 0.000000 0.000000 1 0 /
[Mon Mar 11 14:29:19 2024] I kworker/15:1H 860 20126968.675867 2914917 100 0.000000 64936.831180 0.000000 0.000000 1 0 /
[Mon Mar 11 14:29:19 2024] S irq/103-lpfc:15 950 0.000000 5321549 49 0.000000 119777.169310 0.000000 0.000000 1 0 /
[Mon Mar 11 14:29:19 2024] S irq/186-lpfc:15 1005 0.000000 5333301 49 0.000000 120350.773208 0.000000 0.000000 1 0 /
[Mon Mar 11 14:29:19 2024] I kdmflush/253:9 3457 22713.167289 2 100 0.000000 0.017612 0.000000 0.000000 1 0 /
[Mon Mar 11 14:29:19 2024] I kdmflush/253:32 3660 22994.875960 2 100 0.000000 0.009589 0.000000 0.000000 1 0 /
[Mon Mar 11 14:29:19 2024] I kdmflush/253:64 3940 23521.177608 2 100 0.000000 0.010222 0.000000 0.000000 1 0 /
[Mon Mar 11 14:29:19 2024] I kdmflush/253:69 3975 23623.818868 2 100 0.000000 0.010667 0.000000 0.000000 1 0 /
[Mon Mar 11 14:29:19 2024] I kdmflush/253:75 4118 23677.152489 2 100 0.000000 0.018529 0.000000 0.000000 1 0 /
[Mon Mar 11 14:29:19 2024] I kdmflush/253:76 4132 23763.797476 2 100 0.000000 0.022192 0.000000 0.000000 1 0 /
[Mon Mar 11 14:29:19 2024] I kdmflush/253:78 4135 23775.812473 2 100 0.000000 0.017592 0.000000 0.000000 1 0 /
[Mon Mar 11 14:29:19 2024] S jbd2/dm-80-8 4259 20127091.336665 1264823 120 0.000000 191045.514599 0.000000 0.000000 1 0 /
[Mon Mar 11 14:29:19 2024] I tls-strp 4391 24323.090938 2 100 0.000000 0.024125 0.000000 0.000000 1 0 /
[Mon Mar 11 14:29:19 2024] S reader#2 5335 424932.858954 2847979 120 0.000000 726093.279887 0.000000 0.000000 1 5327 /autogroup-105
[Mon Mar 11 14:29:19 2024] S reader#4 5337 424944.736826 561134 120 0.000000 598804.485160 0.000000 0.000000 1 5327 /autogroup-105
[Mon Mar 11 14:29:19 2024] S rs:action-0-bui 4841 43053.521425 1927079 120 0.000000 118600.890131 0.000000 0.000000 1 4837 /autogroup-112
[Mon Mar 11 14:29:19 2024] S nsrexecd 5345 8951.175593 19130 120 0.000000 687.157688 0.000000 0.000000 1 0 /autogroup-117
[Mon Mar 11 14:29:19 2024] I kdmflush/253:90 6393 26005.424303 2 100 0.000000 0.016044 0.000000 0.000000 1 0 /
[Mon Mar 11 14:29:19 2024] D nfsd 8868 20056073.178896 183235980 120 0.000000 5701430.764254 0.000000 0.000000 1 0 /
[Mon Mar 11 14:29:19 2024] S automount 88150 2611.413623 75 120 0.000000 10.976391 0.000000 0.000000 1 0 /autogroup-222
[Mon Mar 11 14:29:19 2024] S rpcd_spoolss 2339816 252403.714872 10457808 120 0.000000 2805930.098163 0.000000 0.000000 1 0 /autogroup-22820
[Mon Mar 11 14:29:19 2024] S smbd[10.87.30.8 2363822 1249553.896512 371552 120 0.000000 81201.860000 0.000000 0.000000 1 0 /autogroup-113
[Mon Mar 11 14:29:19 2024] S smbd[10.87.27.4 2412672 1249554.528361 224 120 0.000000 77.994581 0.000000 0.000000 1 0 /autogroup-113
[Mon Mar 11 14:29:19 2024] I kworker/15:1 2432551 20127090.624557 85626 120 0.000000 5562.173355 0.000000 0.000000 1 0 /
[Mon Mar 11 14:29:19 2024] I kworker/u96:2 2470993 20124091.050429 59898 120 0.000000 18123.536914 0.000000 0.000000 1 0 /
[Mon Mar 11 14:29:19 2024] I kworker/15:2 2471252 20091034.410305 5 120 0.000000 0.034046 0.000000 0.000000 1 0 /
[Mon Mar 11 14:29:19 2024] I kworker/15:0 2473641 20115589.531683 17 120 0.000000 0.114230 0.000000 0.000000 1 0 /
[Mon Mar 11 14:29:19 2024] cpu#16, 2600.000 MHz
[Mon Mar 11 14:29:19 2024] .nr_running : 0
[Mon Mar 11 14:29:19 2024] .nr_switches : 2705289591
[Mon Mar 11 14:29:19 2024] .nr_uninterruptible : 28417
[Mon Mar 11 14:29:19 2024] .next_balance : 5577.334873
[Mon Mar 11 14:29:19 2024] .curr->pid : 0
[Mon Mar 11 14:29:19 2024] .clock : 1282633450.161643
[Mon Mar 11 14:29:19 2024] .clock_task : 1262596047.551255
[Mon Mar 11 14:29:19 2024] .avg_idle : 1000000
[Mon Mar 11 14:29:19 2024] .max_idle_balance_cost : 500000
[Mon Mar 11 14:29:19 2024] cfs_rq[16]:/autogroup-113
[Mon Mar 11 14:29:19 2024] .exec_clock : 0.000000
[Mon Mar 11 14:29:19 2024] .MIN_vruntime : 0.000001
[Mon Mar 11 14:29:19 2024] .min_vruntime : 2197473.318280
[Mon Mar 11 14:29:19 2024] .max_vruntime : 0.000001
[Mon Mar 11 14:29:19 2024] .spread : 0.000000
[Mon Mar 11 14:29:19 2024] .spread0 : -18220193.189835
[Mon Mar 11 14:29:19 2024] .nr_spread_over : 0
[Mon Mar 11 14:29:19 2024] .nr_running : 0
[Mon Mar 11 14:29:19 2024] .h_nr_running : 0
[Mon Mar 11 14:29:19 2024] .idle_nr_running : 0
[Mon Mar 11 14:29:19 2024] .idle_h_nr_running : 0
[Mon Mar 11 14:29:19 2024] .load : 0
[Mon Mar 11 14:29:19 2024] .load_avg : 0
[Mon Mar 11 14:29:19 2024] .runnable_avg : 0
[Mon Mar 11 14:29:19 2024] .util_avg : 0
[Mon Mar 11 14:29:19 2024] .util_est_enqueued : 0
[Mon Mar 11 14:29:19 2024] .removed.load_avg : 0
[Mon Mar 11 14:29:19 2024] .removed.util_avg : 0
[Mon Mar 11 14:29:19 2024] .removed.runnable_avg : 0
[Mon Mar 11 14:29:19 2024] .tg_load_avg_contrib : 0
[Mon Mar 11 14:29:19 2024] .tg_load_avg : 102
[Mon Mar 11 14:29:19 2024] .throttled : 0
[Mon Mar 11 14:29:19 2024] .throttle_count : 0
[Mon Mar 11 14:29:19 2024] .se->exec_start : 1262595914.330907
[Mon Mar 11 14:29:19 2024] .se->vruntime : 38224184.868966
[Mon Mar 11 14:29:19 2024] .se->sum_exec_runtime : 1951453.474657
[Mon Mar 11 14:29:19 2024] .se->load.weight : 2
[Mon Mar 11 14:29:19 2024] .se->avg.load_avg : 0
[Mon Mar 11 14:29:19 2024] .se->avg.util_avg : 0
[Mon Mar 11 14:29:19 2024] .se->avg.runnable_avg : 0
[Mon Mar 11 14:29:19 2024] cfs_rq[16]:/
[Mon Mar 11 14:29:19 2024] .exec_clock : 0.000000
[Mon Mar 11 14:29:19 2024] .MIN_vruntime : 0.000001
[Mon Mar 11 14:29:19 2024] .min_vruntime : 38224184.868966
[Mon Mar 11 14:29:19 2024] .max_vruntime : 0.000001
[Mon Mar 11 14:29:19 2024] .spread : 0.000000
[Mon Mar 11 14:29:19 2024] .spread0 : 17806518.360851
[Mon Mar 11 14:29:19 2024] .nr_spread_over : 0
[Mon Mar 11 14:29:19 2024] .nr_running : 0
[Mon Mar 11 14:29:19 2024] .h_nr_running : 0
[Mon Mar 11 14:29:19 2024] .idle_nr_running : 0
[Mon Mar 11 14:29:19 2024] .idle_h_nr_running : 0
[Mon Mar 11 14:29:19 2024] .load : 0
[Mon Mar 11 14:29:19 2024] .load_avg : 0
[Mon Mar 11 14:29:19 2024] .runnable_avg : 0
[Mon Mar 11 14:29:19 2024] .util_avg : 0
[Mon Mar 11 14:29:19 2024] .util_est_enqueued : 0
[Mon Mar 11 14:29:19 2024] .removed.load_avg : 0
[Mon Mar 11 14:29:19 2024] .removed.util_avg : 0
[Mon Mar 11 14:29:19 2024] .removed.runnable_avg : 0
[Mon Mar 11 14:29:19 2024] .tg_load_avg_contrib : 0
[Mon Mar 11 14:29:19 2024] .tg_load_avg : 0
[Mon Mar 11 14:29:19 2024] .throttled : 0
[Mon Mar 11 14:29:19 2024] .throttle_count : 0
[Mon Mar 11 14:29:19 2024] rt_rq[16]:
[Mon Mar 11 14:29:19 2024] .rt_nr_running : 0
[Mon Mar 11 14:29:19 2024] .rt_nr_migratory : 0
[Mon Mar 11 14:29:19 2024] .rt_throttled : 0
[Mon Mar 11 14:29:19 2024] .rt_time : 0.000000
[Mon Mar 11 14:29:19 2024] .rt_runtime : 950.000000
[Mon Mar 11 14:29:19 2024] dl_rq[16]:
[Mon Mar 11 14:29:19 2024] .dl_nr_running : 0
[Mon Mar 11 14:29:19 2024] .dl_nr_migratory : 0
[Mon Mar 11 14:29:19 2024] .dl_bw->bw : 996147
[Mon Mar 11 14:29:19 2024] .dl_bw->total_bw : 0
[Mon Mar 11 14:29:19 2024] runnable tasks:
[Mon Mar 11 14:29:19 2024] S task PID tree-key switches prio wait-time sum-exec sum-sleep
[Mon Mar 11 14:29:19 2024] -------------------------------------------------------------------------------------------------------------
[Mon Mar 11 14:29:19 2024] S cpuhp/16 114 24295.744054 25 120 0.000000 1.032223 0.000000 0.000000 0 0 /
[Mon Mar 11 14:29:19 2024] S idle_inject/16 115 -12.000000 3 49 0.000000 0.000000 0.000000 0.000000 0 0 /
[Mon Mar 11 14:29:19 2024] S migration/16 116 901.555492 365900 0 0.000000 1917.531004 0.000000 0.000000 0 0 /
[Mon Mar 11 14:29:19 2024] S ksoftirqd/16 117 38224167.386205 24915738 120 0.000000 293710.902055 0.000000 0.000000 0 0 /
[Mon Mar 11 14:29:19 2024] I kworker/16:0H 119 172.856368 4 100 0.000000 0.045496 0.000000 0.000000 0 0 /
[Mon Mar 11 14:29:19 2024] I kworker/16:1H 762 38224167.268598 3333979 100 0.000000 73903.836713 0.000000 0.000000 0 0 /
[Mon Mar 11 14:29:19 2024] S irq/112-lpfc:16 951 0.000000 9859731 49 0.000000 192667.887444 0.000000 0.000000 0 0 /
[Mon Mar 11 14:29:19 2024] S irq/187-lpfc:16 1006 0.000000 9874120 49 0.000000 192984.835533 0.000000 0.000000 0 0 /
[Mon Mar 11 14:29:19 2024] I kdmflush/253:82 4157 25916.349937 2 100 0.000000 0.066049 0.000000 0.000000 0 0 /
[Mon Mar 11 14:29:19 2024] I ext4-rsv-conver 4208 26413.245510 2 100 0.000000 0.061676 0.000000 0.000000 0 0 /
[Mon Mar 11 14:29:19 2024] S jbd2/dm-86-8 4246 38223138.250801 154711 120 0.000000 11717.183685 0.000000 0.000000 0 0 /
[Mon Mar 11 14:29:19 2024] S JS Helper 5182 23.472142 5 120 0.000000 0.486658 0.000000 0.000000 0 0 /autogroup-104
[Mon Mar 11 14:29:19 2024] D nfsd 8807 38224175.746992 133432 120 0.000000 13706.050891 0.000000 0.000000 0 0 /
[Mon Mar 11 14:29:19 2024] S automount 88125 11.862926 1 120 0.000000 0.911509 0.000000 0.000000 0 0 /autogroup-222
[Mon Mar 11 14:29:19 2024] S automount 88151 24.752428 1 120 0.000000 0.889509 0.000000 0.000000 0 0 /autogroup-222
[Mon Mar 11 14:29:19 2024] S smbd[10.91.30.5 3294561 2197473.318280 1271825 120 0.000000 1577525.997307 0.000000 0.000000 0 0 /autogroup-113
[Mon Mar 11 14:29:19 2024] S smbd[10.91.30.5 662850 2197042.215530 6803 120 0.000000 1846.429649 0.000000 0.000000 0 0 /autogroup-113
[Mon Mar 11 14:29:19 2024] S smbd[10.40.142. 2342339 2197099.276461 2288 120 0.000000 815.417760 0.000000 0.000000 0 0 /autogroup-113
[Mon Mar 11 14:29:19 2024] S smbd[10.40.229. 2355747 2197407.517671 91686 120 0.000000 4474.950891 0.000000 0.000000 0 0 /autogroup-113
[Mon Mar 11 14:29:19 2024] S smbd[10.40.133. 2384413 2197355.953824 360 120 0.000000 117.548543 0.000000 0.000000 0 0 /autogroup-113
[Mon Mar 11 14:29:19 2024] I kworker/16:2 2397744 38224172.768503 358373 120 0.000000 14281.570565 0.000000 0.000000 0 0 /
[Mon Mar 11 14:29:19 2024] S smbd[10.40.153. 2423410 2197204.000123 4598 120 0.000000 2057.641245 0.000000 0.000000 0 0 /autogroup-113
[Mon Mar 11 14:29:19 2024] S smbd[10.87.28.1 2452452 2197461.427192 124 120 0.000000 58.217814 0.000000 0.000000 0 0 /autogroup-113
[Mon Mar 11 14:29:19 2024] I kworker/u97:0 2473005 38224167.279710 75915 120 0.000000 2203.181420 0.000000 0.000000 0 0 /
[Mon Mar 11 14:29:19 2024] I kworker/16:1 2475250 38218369.616277 7 120 0.000000 0.073999 0.000000 0.000000 0 0 /
[Mon Mar 11 14:29:19 2024] S sshd 2476434 11.303906 522 120 0.000000 85.795163 0.000000 0.000000 0 0 /autogroup-23279
[Mon Mar 11 14:29:19 2024] S nrpe 2478245 206330.570850 2 120 0.000000 0.829470 0.000000 0.000000 0 0 /autogroup-109
[Mon Mar 11 14:29:19 2024] cpu#17, 2600.000 MHz
[Mon Mar 11 14:29:19 2024] .nr_running : 0
[Mon Mar 11 14:29:19 2024] .nr_switches : 964910989
[Mon Mar 11 14:29:19 2024] .nr_uninterruptible : -2169
[Mon Mar 11 14:29:19 2024] .next_balance : 5577.334873
[Mon Mar 11 14:29:19 2024] .curr->pid : 0
[Mon Mar 11 14:29:19 2024] .clock : 1282633450.160629
[Mon Mar 11 14:29:19 2024] .clock_task : 1281032327.768478
[Mon Mar 11 14:29:19 2024] .avg_idle : 554673
[Mon Mar 11 14:29:19 2024] .max_idle_balance_cost : 500000
[Mon Mar 11 14:29:19 2024] rt_rq[17]:
[Mon Mar 11 14:29:19 2024] .rt_nr_running : 0
[Mon Mar 11 14:29:19 2024] .rt_nr_migratory : 0
[Mon Mar 11 14:29:19 2024] .rt_throttled : 0
[Mon Mar 11 14:29:19 2024] .rt_time : 0.178580
[Mon Mar 11 14:29:19 2024] .rt_runtime : 950.000000
[Mon Mar 11 14:29:19 2024] dl_rq[17]:
[Mon Mar 11 14:29:19 2024] .dl_nr_running : 0
[Mon Mar 11 14:29:19 2024] .dl_nr_migratory : 0
[Mon Mar 11 14:29:19 2024] .dl_bw->bw : 996147
[Mon Mar 11 14:29:19 2024] .dl_bw->total_bw : 0
[Mon Mar 11 14:29:19 2024] runnable tasks:
[Mon Mar 11 14:29:19 2024] S task PID tree-key switches prio wait-time sum-exec sum-sleep
[Mon Mar 11 14:29:19 2024] -------------------------------------------------------------------------------------------------------------
[Mon Mar 11 14:29:19 2024] S cpuhp/17 120 25941.651935 25 120 0.000000 1.166897 0.000000 0.000000 1 0 /
[Mon Mar 11 14:29:19 2024] S idle_inject/17 121 -12.000000 3 49 0.000000 0.000000 0.000000 0.000000 1 0 /
[Mon Mar 11 14:29:19 2024] S migration/17 122 969.555489 324874 0 0.000000 1888.896263 0.000000 0.000000 1 0 /
[Mon Mar 11 14:29:19 2024] S ksoftirqd/17 123 19431575.263295 11514961 120 0.000000 111385.611222 0.000000 0.000000 1 0 /
[Mon Mar 11 14:29:19 2024] I kworker/17:0H 125 24.660770 4 100 0.000000 0.014031 0.000000 0.000000 1 0 /
[Mon Mar 11 14:29:19 2024] I kworker/17:1H 856 19431575.224780 3489009 100 0.000000 78027.237958 0.000000 0.000000 1 0 /
[Mon Mar 11 14:29:19 2024] S irq/113-lpfc:17 952 0.000000 5591148 49 0.000000 130003.261092 0.000000 0.000000 1 0 /
[Mon Mar 11 14:29:19 2024] S irq/188-lpfc:17 1007 0.000000 5602516 49 0.000000 130871.522555 0.000000 0.000000 1 0 /
[Mon Mar 11 14:29:19 2024] S multipathd 3353 0.000000 87 0 0.000000 4.046831 0.000000 0.000000 1 0 /autogroup-45
[Mon Mar 11 14:29:19 2024] S multipathd 3357 0.000000 13 0 0.000000 8.041793 0.000000 0.000000 1 0 /autogroup-45
[Mon Mar 11 14:29:19 2024] I kdmflush/253:7 3445 26574.657879 2 100 0.000000 0.019338 0.000000 0.000000 1 0 /
[Mon Mar 11 14:29:19 2024] I kdmflush/253:8 3453 26574.664939 2 100 0.000000 0.020543 0.000000 0.000000 1 0 /
[Mon Mar 11 14:29:19 2024] I kdmflush/253:15 3509 26746.732035 2 100 0.000000 0.008958 0.000000 0.000000 1 0 /
[Mon Mar 11 14:29:19 2024] I kdmflush/253:31 3657 27037.815929 2 100 0.000000 0.017258 0.000000 0.000000 1 0 /
[Mon Mar 11 14:29:19 2024] I kdmflush/253:36 3700 27097.935838 2 100 0.000000 0.010510 0.000000 0.000000 1 0 /
[Mon Mar 11 14:29:19 2024] I kdmflush/253:38 3714 27097.939852 2 100 0.000000 0.013735 0.000000 0.000000 1 0 /
[Mon Mar 11 14:29:19 2024] I kdmflush/253:41 3738 27111.960020 2 100 0.000000 0.017637 0.000000 0.000000 1 0 /
[Mon Mar 11 14:29:19 2024] I kdmflush/253:42 3752 27162.219368 2 100 0.000000 0.028321 0.000000 0.000000 1 0 /
[Mon Mar 11 14:29:19 2024] I kdmflush/253:43 3762 27175.281779 2 100 0.000000 0.012739 0.000000 0.000000 1 0 /
[Mon Mar 11 14:29:19 2024] I kdmflush/253:62 3928 27472.029779 2 100 0.000000 0.012568 0.000000 0.000000 1 0 /
[Mon Mar 11 14:29:19 2024] I kdmflush/253:68 3973 27472.430219 2 100 0.000000 0.016183 0.000000 0.000000 1 0 /
[Mon Mar 11 14:29:19 2024] I kdmflush/253:77 4134 27585.867067 2 100 0.000000 0.018980 0.000000 0.000000 1 0 /
[Mon Mar 11 14:29:19 2024] I kdmflush/253:88 4169 27597.882802 2 100 0.000000 0.019246 0.000000 0.000000 1 0 /
[Mon Mar 11 14:29:19 2024] I ext4-rsv-conver 4220 27714.742733 2 100 0.000000 0.018409 0.000000 0.000000 1 0 /
[Mon Mar 11 14:29:19 2024] I ext4-rsv-conver 4221 27726.755093 2 100 0.000000 0.015219 0.000000 0.000000 1 0 /
[Mon Mar 11 14:29:19 2024] S tuned 5263 325.642727 8 120 0.000000 3.884221 0.000000 0.000000 1 0 /autogroup-93
[Mon Mar 11 14:29:19 2024] S rpc.gssd 4432 1369.486182 42756 120 0.000000 1184.327586 0.000000 0.000000 1 0 /autogroup-100
[Mon Mar 11 14:29:19 2024] S JS Helper 5176 11.190193 9 120 0.000000 0.238776 0.000000 0.000000 1 0 /autogroup-104
[Mon Mar 11 14:29:19 2024] I kdmflush/253:94 6417 30182.914014 2 100 0.000000 0.019072 0.000000 0.000000 1 0 /
[Mon Mar 11 14:29:19 2024] I kdmflush/253:10 8697 31492.309009 2 100 0.000000 0.065396 0.000000 0.000000 1 0 /
[Mon Mar 11 14:29:19 2024] I ext4-rsv-conver 8747 31504.372007 2 100 0.000000 0.066252 0.000000 0.000000 1 0 /
[Mon Mar 11 14:29:19 2024] I ib-comp-unb-wq 8763 31516.433269 2 100 0.000000 0.064634 0.000000 0.000000 1 0 /
[Mon Mar 11 14:29:19 2024] D nfsd 8796 19431574.200343 89864 120 0.000000 10105.801789 0.000000 0.000000 1 0 /
[Mon Mar 11 14:29:19 2024] S smbd[10.87.19.1 2324056 1158357.407324 525 120 0.000000 145.674088 0.000000 0.000000 1 0 /autogroup-113
[Mon Mar 11 14:29:19 2024] S rpcd_spoolss 2339825 247435.964896 1517035 120 0.000000 396329.321166 0.000000 0.000000 1 0 /autogroup-22820
[Mon Mar 11 14:29:19 2024] S smbd[10.33.88.9 2347580 1158392.953252 461 120 0.000000 152.838733 0.000000 0.000000 1 0 /autogroup-113
[Mon Mar 11 14:29:19 2024] S smbd[10.87.31.1 2352076 1158392.712303 1764 120 0.000000 903.633187 0.000000 0.000000 1 0 /autogroup-113
[Mon Mar 11 14:29:19 2024] I kworker/17:1 2391348 19431575.274720 165612 120 0.000000 9291.833357 0.000000 0.000000 1 0 /
[Mon Mar 11 14:29:19 2024] S smbd[10.88.16.7 2407419 1158392.652293 3145 120 0.000000 2538.127809 0.000000 0.000000 1 0 /autogroup-113
[Mon Mar 11 14:29:19 2024] S smbd[10.87.18.9 2432071 1158392.710280 174 120 0.000000 62.798772 0.000000 0.000000 1 0 /autogroup-113
[Mon Mar 11 14:29:19 2024] S smbd[10.46.159. 2465617 1158393.007578 64 120 0.000000 36.620453 0.000000 0.000000 1 0 /autogroup-113
[Mon Mar 11 14:29:19 2024] I kworker/17:0 2474774 19410071.831729 3 120 0.000000 0.063514 0.000000 0.000000 1 0 /
[Mon Mar 11 14:29:19 2024] cpu#18, 2600.000 MHz
[Mon Mar 11 14:29:19 2024] .nr_running : 0
[Mon Mar 11 14:29:19 2024] .nr_switches : 2434048599
[Mon Mar 11 14:29:19 2024] .nr_uninterruptible : 37600
[Mon Mar 11 14:29:19 2024] .next_balance : 5577.334874
[Mon Mar 11 14:29:19 2024] .curr->pid : 0
[Mon Mar 11 14:29:19 2024] .clock : 1282633450.161181
[Mon Mar 11 14:29:19 2024] .clock_task : 1265217668.233471
[Mon Mar 11 14:29:19 2024] .avg_idle : 1000000
[Mon Mar 11 14:29:19 2024] .max_idle_balance_cost : 500000
[Mon Mar 11 14:29:19 2024] rt_rq[18]:
[Mon Mar 11 14:29:19 2024] .rt_nr_running : 0
[Mon Mar 11 14:29:19 2024] .rt_nr_migratory : 0
[Mon Mar 11 14:29:19 2024] .rt_throttled : 0
[Mon Mar 11 14:29:19 2024] .rt_time : 0.000000
[Mon Mar 11 14:29:19 2024] .rt_runtime : 950.000000
[Mon Mar 11 14:29:19 2024] dl_rq[18]:
[Mon Mar 11 14:29:19 2024] .dl_nr_running : 0
[Mon Mar 11 14:29:19 2024] .dl_nr_migratory : 0
[Mon Mar 11 14:29:19 2024] .dl_bw->bw : 996147
[Mon Mar 11 14:29:19 2024] .dl_bw->total_bw : 0
[Mon Mar 11 14:29:19 2024] runnable tasks:
[Mon Mar 11 14:29:19 2024] S task PID tree-key switches prio wait-time sum-exec sum-sleep
[Mon Mar 11 14:29:19 2024] -------------------------------------------------------------------------------------------------------------
[Mon Mar 11 14:29:19 2024] S cpuhp/18 126 28898.836945 25 120 0.000000 0.871054 0.000000 0.000000 0 0 /
[Mon Mar 11 14:29:19 2024] S idle_inject/18 127 -11.995276 3 49 0.000000 0.270707 0.000000 0.000000 0 0 /
[Mon Mar 11 14:29:19 2024] S migration/18 128 1037.816154 365466 0 0.000000 1891.855065 0.000000 0.000000 0 0 /
[Mon Mar 11 14:29:19 2024] S ksoftirqd/18 129 34240895.372719 22823678 120 0.000000 263730.895545 0.000000 0.000000 0 0 /
[Mon Mar 11 14:29:19 2024] I kworker/18:0H 131 47.497810 4 100 0.000000 0.063433 0.000000 0.000000 0 0 /
[Mon Mar 11 14:29:19 2024] I inet_frag_wq 355 59.367178 2 100 0.000000 0.000000 0.000000 0.000000 0 0 /
[Mon Mar 11 14:29:19 2024] I kworker/18:1H 754 34240765.437724 3174907 100 0.000000 70296.065673 0.000000 0.000000 0 0 /
[Mon Mar 11 14:29:19 2024] S irq/114-lpfc:18 953 0.000000 9362298 49 0.000000 183411.599762 0.000000 0.000000 0 0 /
[Mon Mar 11 14:29:19 2024] S irq/189-lpfc:18 1008 0.000000 9377696 49 0.000000 184275.156559 0.000000 0.000000 0 0 /
[Mon Mar 11 14:29:19 2024] I kdmflush/253:14 3502 29470.742691 2 100 0.000000 0.014193 0.000000 0.000000 0 0 /
[Mon Mar 11 14:29:19 2024] I kdmflush/253:53 3839 30342.884213 2 100 0.000000 0.007693 0.000000 0.000000 0 0 /
[Mon Mar 11 14:29:19 2024] I kdmflush/253:81 4155 30954.360271 2 100 0.000000 0.066985 0.000000 0.000000 0 0 /
[Mon Mar 11 14:29:19 2024] I ext4-rsv-conver 4361 31615.105410 2 100 0.000000 0.064344 0.000000 0.000000 0 0 /
[Mon Mar 11 14:29:19 2024] S gssproxy 4425 11.057469 1 120 0.000000 0.106054 0.000000 0.000000 0 0 /autogroup-98
[Mon Mar 11 14:29:19 2024] S rsyslogd 4752 1226.531223 2223 120 0.000000 192.539333 0.000000 0.000000 0 4752 /autogroup-112
[Mon Mar 11 14:29:19 2024] S jbd2/dm-92-8 6475 34106951.884440 18557 120 0.000000 2405.892759 0.000000 0.000000 0 0 /
[Mon Mar 11 14:29:19 2024] D nfsd 8834 34230189.304574 4819229 120 0.000000 262121.772772 0.000000 0.000000 0 0 /
[Mon Mar 11 14:29:19 2024] S automount 88126 166384.444556 11 120 0.000000 1.574640 0.000000 0.000000 0 0 /autogroup-222
[Mon Mar 11 14:29:19 2024] S sssd_kcm 179479 1875.901308 255137 120 0.000000 50715.484538 0.000000 0.000000 0 0 /autogroup-707
[Mon Mar 11 14:29:19 2024] S smbd[10.87.27.1 789171 2219101.590427 21804 120 0.000000 5621.762420 0.000000 0.000000 0 0 /autogroup-113
[Mon Mar 11 14:29:19 2024] S smbd[10.87.27.4 1367201 2219146.691139 4113 120 0.000000 1085.094801 0.000000 0.000000 0 0 /autogroup-113
[Mon Mar 11 14:29:19 2024] I kworker/18:1 2072443 34240921.800259 1049137 120 0.000000 31341.343265 0.000000 0.000000 0 0 /
[Mon Mar 11 14:29:19 2024] S smbd[10.40.228. 2338708 2219299.761504 27951 120 0.000000 10241.970906 0.000000 0.000000 0 0 /autogroup-113
[Mon Mar 11 14:29:19 2024] S smbd[10.33.88.1 2377288 2219251.750338 358 120 0.000000 114.269577 0.000000 0.000000 0 0 /autogroup-113
[Mon Mar 11 14:29:19 2024] I kworker/18:2 2474270 34236501.832518 8 120 0.000000 0.092269 0.000000 0.000000 0 0 /
[Mon Mar 11 14:29:19 2024] I kworker/18:0 2477203 34236513.814739 2 120 0.000000 0.062411 0.000000 0.000000 0 0 /
[Mon Mar 11 14:29:19 2024] cpu#19, 2600.000 MHz
[Mon Mar 11 14:29:19 2024] .nr_running : 0
[Mon Mar 11 14:29:19 2024] .nr_switches : 993035978
[Mon Mar 11 14:29:19 2024] .nr_uninterruptible : -2982
[Mon Mar 11 14:29:19 2024] .next_balance : 5577.334872
[Mon Mar 11 14:29:19 2024] .curr->pid : 0
[Mon Mar 11 14:29:19 2024] .clock : 1282633450.161098
[Mon Mar 11 14:29:19 2024] .clock_task : 1281059918.581047
[Mon Mar 11 14:29:19 2024] .avg_idle : 1000000
[Mon Mar 11 14:29:19 2024] .max_idle_balance_cost : 500000
[Mon Mar 11 14:29:19 2024] rt_rq[19]:
[Mon Mar 11 14:29:19 2024] .rt_nr_running : 0
[Mon Mar 11 14:29:19 2024] .rt_nr_migratory : 0
[Mon Mar 11 14:29:19 2024] .rt_throttled : 0
[Mon Mar 11 14:29:19 2024] .rt_time : 0.000000
[Mon Mar 11 14:29:19 2024] .rt_runtime : 950.000000
[Mon Mar 11 14:29:19 2024] dl_rq[19]:
[Mon Mar 11 14:29:19 2024] .dl_nr_running : 0
[Mon Mar 11 14:29:19 2024] .dl_nr_migratory : 0
[Mon Mar 11 14:29:19 2024] .dl_bw->bw : 996147
[Mon Mar 11 14:29:19 2024] .dl_bw->total_bw : 0
[Mon Mar 11 14:29:19 2024] runnable tasks:
[Mon Mar 11 14:29:19 2024] S task PID tree-key switches prio wait-time sum-exec sum-sleep
[Mon Mar 11 14:29:19 2024] -------------------------------------------------------------------------------------------------------------
[Mon Mar 11 14:29:19 2024] S cpuhp/19 132 21715.636766 25 120 0.000000 0.845631 0.000000 0.000000 1 0 /
[Mon Mar 11 14:29:19 2024] S idle_inject/19 133 -12.000000 3 49 0.000000 0.000000 0.000000 0.000000 1 0 /
[Mon Mar 11 14:29:19 2024] S migration/19 134 1105.816150 324822 0 0.000000 1896.862061 0.000000 0.000000 1 0 /
[Mon Mar 11 14:29:19 2024] S ksoftirqd/19 135 19607795.729144 11829573 120 0.000000 114680.925698 0.000000 0.000000 1 0 /
[Mon Mar 11 14:29:19 2024] I kworker/19:0H 137 102.031486 4 100 0.000000 0.018501 0.000000 0.000000 1 0 /
[Mon Mar 11 14:29:19 2024] I kworker/19:1H 813 19606020.360141 3491399 100 0.000000 78246.847787 0.000000 0.000000 1 0 /
[Mon Mar 11 14:29:19 2024] S lpfc_worker_0 934 19608286.899265 284753 100 0.000000 5576.249969 0.000000 0.000000 1 0 /
[Mon Mar 11 14:29:19 2024] S irq/115-lpfc:19 954 0.000000 5710627 49 0.000000 131958.422164 0.000000 0.000000 1 0 /
[Mon Mar 11 14:29:19 2024] S irq/190-lpfc:19 1009 0.000000 5727911 49 0.000000 133043.353601 0.000000 0.000000 1 0 /
[Mon Mar 11 14:29:19 2024] I kdmflush/253:47 3799 23287.908181 2 100 0.000000 0.007126 0.000000 0.000000 1 0 /
[Mon Mar 11 14:29:19 2024] I kdmflush/253:60 3907 23530.099872 2 100 0.000000 0.015141 0.000000 0.000000 1 0 /
[Mon Mar 11 14:29:19 2024] S jbd2/dm-76-8 4252 19605506.735404 141298 120 0.000000 7487.841466 0.000000 0.000000 1 0 /
[Mon Mar 11 14:29:19 2024] S nfsdcld 4302 2653.021662 641126 120 0.000000 117340.045585 0.000000 0.000000 1 0 /autogroup-69
[Mon Mar 11 14:29:19 2024] S gmain 4320 11.139325 1 120 0.000000 0.187910 0.000000 0.000000 1 0 /autogroup-75
[Mon Mar 11 14:29:19 2024] S polkitd 4667 3411.197544 317560 120 0.000000 38594.323126 0.000000 0.000000 1 4667 /autogroup-104
[Mon Mar 11 14:29:19 2024] S dmeventd 6411 2393.343824 128300 120 0.000000 13541.927602 0.000000 0.000000 1 0 /autogroup-125
[Mon Mar 11 14:29:19 2024] S jbd2/dm-102-8 8732 19486067.157920 2021 120 0.000000 868.781011 0.000000 0.000000 1 0 /
[Mon Mar 11 14:29:19 2024] S smbd[10.87.29.1 2279961 1011902.994770 828 120 0.000000 235.659148 0.000000 0.000000 1 0 /autogroup-113
[Mon Mar 11 14:29:19 2024] S rpcd_winreg 2347737 225923.629574 7485002 120 0.000000 1825461.507361 0.000000 0.000000 1 0 /autogroup-22820
[Mon Mar 11 14:29:19 2024] S smbd[10.87.27.1 2400150 1011903.054564 265 120 0.000000 87.155161 0.000000 0.000000 1 0 /autogroup-113
[Mon Mar 11 14:29:19 2024] I kworker/19:1 2402658 19608286.911586 139868 120 0.000000 9668.678759 0.000000 0.000000 1 0 /
[Mon Mar 11 14:29:19 2024] I kworker/19:0 2475271 19595419.771327 5 120 0.000000 0.026001 0.000000 0.000000 1 0 /
[Mon Mar 11 14:29:19 2024] cpu#20, 2600.000 MHz
[Mon Mar 11 14:29:19 2024] .nr_running : 0
[Mon Mar 11 14:29:19 2024] .nr_switches : 2852629227
[Mon Mar 11 14:29:19 2024] .nr_uninterruptible : 25671
[Mon Mar 11 14:29:19 2024] .next_balance : 5577.334873
[Mon Mar 11 14:29:19 2024] .curr->pid : 0
[Mon Mar 11 14:29:19 2024] .clock : 1282633450.161536
[Mon Mar 11 14:29:19 2024] .clock_task : 1257094457.580888
[Mon Mar 11 14:29:19 2024] .avg_idle : 1000000
[Mon Mar 11 14:29:19 2024] .max_idle_balance_cost : 500000
[Mon Mar 11 14:29:19 2024] rt_rq[20]:
[Mon Mar 11 14:29:19 2024] .rt_nr_running : 0
[Mon Mar 11 14:29:19 2024] .rt_nr_migratory : 0
[Mon Mar 11 14:29:19 2024] .rt_throttled : 0
[Mon Mar 11 14:29:19 2024] .rt_time : 0.000000
[Mon Mar 11 14:29:19 2024] .rt_runtime : 950.000000
[Mon Mar 11 14:29:19 2024] dl_rq[20]:
[Mon Mar 11 14:29:19 2024] .dl_nr_running : 0
[Mon Mar 11 14:29:19 2024] .dl_nr_migratory : 0
[Mon Mar 11 14:29:19 2024] .dl_bw->bw : 996147
[Mon Mar 11 14:29:19 2024] .dl_bw->total_bw : 0
[Mon Mar 11 14:29:19 2024] runnable tasks:
[Mon Mar 11 14:29:19 2024] S task PID tree-key switches prio wait-time sum-exec sum-sleep
[Mon Mar 11 14:29:19 2024] -------------------------------------------------------------------------------------------------------------
[Mon Mar 11 14:29:19 2024] S cpuhp/20 138 25632.867284 25 120 0.000000 1.425623 0.000000 0.000000 0 0 /
[Mon Mar 11 14:29:19 2024] S idle_inject/20 139 -12.000000 3 49 0.000000 0.000000 0.000000 0.000000 0 0 /
[Mon Mar 11 14:29:19 2024] S migration/20 140 1173.816857 390003 0 0.000000 2032.939139 0.000000 0.000000 0 0 /
[Mon Mar 11 14:29:19 2024] S ksoftirqd/20 141 40444902.519184 26626240 120 0.000000 318014.453433 0.000000 0.000000 0 0 /
[Mon Mar 11 14:29:19 2024] I kworker/20:0H 143 23.820108 4 100 0.000000 0.022756 0.000000 0.000000 0 0 /
[Mon Mar 11 14:29:19 2024] I kworker/20:1H 781 40444797.388260 3229792 100 0.000000 69745.633514 0.000000 0.000000 0 0 /
[Mon Mar 11 14:29:19 2024] S scsi_eh_4 904 1735.685815 25 120 0.000000 6.093795 0.000000 0.000000 0 0 /
[Mon Mar 11 14:29:19 2024] I scsi_tmf_11 923 1712.093830 2 100 0.000000 0.381054 0.000000 0.000000 0 0 /
[Mon Mar 11 14:29:19 2024] S irq/116-lpfc:20 955 0.000000 9318267 49 0.000000 181639.257968 0.000000 0.000000 0 0 /
[Mon Mar 11 14:29:19 2024] S irq/191-lpfc:20 1010 0.000000 9325964 49 0.000000 182449.194570 0.000000 0.000000 0 0 /
[Mon Mar 11 14:29:19 2024] I ext4-rsv-conver 4351 27971.424278 2 100 0.000000 0.065866 0.000000 0.000000 0 0 /
[Mon Mar 11 14:29:19 2024] S upsclient 7154 6115.836221 1303786 120 0.000000 138592.310149 0.000000 0.000000 0 0 /autogroup-126
[Mon Mar 11 14:29:19 2024] D nfsd 8804 40444905.590664 118475 120 0.000000 12210.491394 0.000000 0.000000 0 0 /
[Mon Mar 11 14:29:19 2024] D nfsd 8841 40377251.850810 2216516 120 0.000000 79303.374470 0.000000 0.000000 0 0 /
[Mon Mar 11 14:29:19 2024] S automount 88092 10.614597 98 120 0.000000 30.665238 0.000000 0.000000 0 0 /autogroup-222
[Mon Mar 11 14:29:19 2024] S agetty 88117 0.819970 18 120 0.000000 60.743827 0.000000 0.000000 0 0 /autogroup-228
[Mon Mar 11 14:29:19 2024] S smbd[10.87.29.8 2352500 2360986.791597 211258 120 0.000000 41698.277658 0.000000 0.000000 0 0 /autogroup-113
[Mon Mar 11 14:29:19 2024] I kworker/20:2 2415685 40444902.893030 238245 120 0.000000 15568.767948 0.000000 0.000000 0 0 /
[Mon Mar 11 14:29:19 2024] S smbd[10.87.28.1 2461553 2361021.694063 1189 120 0.000000 418.895454 0.000000 0.000000 0 0 /autogroup-113
[Mon Mar 11 14:29:19 2024] I kworker/20:0 2474728 40438956.586822 12 120 0.000000 0.057860 0.000000 0.000000 0 0 /
[Mon Mar 11 14:29:19 2024] cpu#21, 2600.000 MHz
[Mon Mar 11 14:29:19 2024] .nr_running : 0
[Mon Mar 11 14:29:19 2024] .nr_switches : 973798643
[Mon Mar 11 14:29:19 2024] .nr_uninterruptible : -1736
[Mon Mar 11 14:29:19 2024] .next_balance : 5577.334876
[Mon Mar 11 14:29:19 2024] .curr->pid : 0
[Mon Mar 11 14:29:19 2024] .clock : 1282633451.162417
[Mon Mar 11 14:29:19 2024] .clock_task : 1281088670.122286
[Mon Mar 11 14:29:19 2024] .avg_idle : 1000000
[Mon Mar 11 14:29:19 2024] .max_idle_balance_cost : 500000
[Mon Mar 11 14:29:19 2024] rt_rq[21]:
[Mon Mar 11 14:29:19 2024] .rt_nr_running : 0
[Mon Mar 11 14:29:19 2024] .rt_nr_migratory : 0
[Mon Mar 11 14:29:19 2024] .rt_throttled : 0
[Mon Mar 11 14:29:19 2024] .rt_time : 0.062506
[Mon Mar 11 14:29:19 2024] .rt_runtime : 950.000000
[Mon Mar 11 14:29:19 2024] dl_rq[21]:
[Mon Mar 11 14:29:19 2024] .dl_nr_running : 0
[Mon Mar 11 14:29:19 2024] .dl_nr_migratory : 0
[Mon Mar 11 14:29:19 2024] .dl_bw->bw : 996147
[Mon Mar 11 14:29:19 2024] .dl_bw->total_bw : 0
[Mon Mar 11 14:29:19 2024] runnable tasks:
[Mon Mar 11 14:29:19 2024] S task PID tree-key switches prio wait-time sum-exec sum-sleep
[Mon Mar 11 14:29:19 2024] -------------------------------------------------------------------------------------------------------------
[Mon Mar 11 14:29:19 2024] D systemd 1 144507.589599 31252056 120 0.000000 3898869.479139 0.000000 0.000000 1 1 /autogroup-1
[Mon Mar 11 14:29:19 2024] S cpuhp/21 144 25698.411482 25 120 0.000000 0.694450 0.000000 0.000000 1 0 /
[Mon Mar 11 14:29:19 2024] S idle_inject/21 145 -12.000000 3 49 0.000000 0.000000 0.000000 0.000000 1 0 /
[Mon Mar 11 14:29:19 2024] S migration/21 146 1241.816861 324561 0 0.000000 1924.545200 0.000000 0.000000 1 0 /
[Mon Mar 11 14:29:19 2024] S ksoftirqd/21 147 19220270.131493 11478007 120 0.000000 112369.656912 0.000000 0.000000 1 0 /
[Mon Mar 11 14:29:19 2024] I kworker/21:0H 149 92.900758 4 100 0.000000 0.028353 0.000000 0.000000 1 0 /
[Mon Mar 11 14:29:19 2024] I kworker/21:1H 829 19220269.864386 3237280 100 0.000000 71690.939907 0.000000 0.000000 1 0 /
[Mon Mar 11 14:29:19 2024] S irq/117-lpfc:21 956 0.000000 5541480 49 0.000000 128874.523612 0.000000 0.000000 1 0 /
[Mon Mar 11 14:29:19 2024] S irq/192-lpfc:21 1011 0.000000 5553172 49 0.000000 129798.137598 0.000000 0.000000 1 0 /
[Mon Mar 11 14:29:19 2024] I kdmflush/253:5 3431 26358.748805 2 100 0.000000 0.011092 0.000000 0.000000 1 0 /
[Mon Mar 11 14:29:19 2024] I kdmflush/253:6 3437 26370.763969 2 100 0.000000 0.017879 0.000000 0.000000 1 0 /
[Mon Mar 11 14:29:19 2024] I kdmflush/253:11 3476 26382.778699 2 100 0.000000 0.017673 0.000000 0.000000 1 0 /
[Mon Mar 11 14:29:19 2024] I kdmflush/253:13 3499 26394.790383 2 100 0.000000 0.014494 0.000000 0.000000 1 0 /
[Mon Mar 11 14:29:19 2024] I kdmflush/253:22 3574 26419.509153 2 100 0.000000 0.013064 0.000000 0.000000 1 0 /
[Mon Mar 11 14:29:19 2024] S sssd_pam 4357 158564.306573 260101 120 0.000000 15215.612361 0.000000 0.000000 1 0 /autogroup-78
[Mon Mar 11 14:29:19 2024] S sshd 4412 2764.722592 25472 120 0.000000 15730.520853 0.000000 0.000000 1 0 /autogroup-92
[Mon Mar 11 14:29:19 2024] S wb-idmap 5362 105007.202701 1179982 120 0.000000 316610.932947 0.000000 0.000000 1 0 /autogroup-95
[Mon Mar 11 14:29:19 2024] I kdmflush/253:92 6412 28925.135872 2 100 0.000000 0.063711 0.000000 0.000000 1 0 /
[Mon Mar 11 14:29:19 2024] S smbd[10.87.19.1 2324043 993397.331435 9307 120 0.000000 3831.526501 0.000000 0.000000 1 0 /autogroup-113
[Mon Mar 11 14:29:19 2024] S smbd[10.87.18.2 2341452 993352.281372 696 120 0.000000 234.585944 0.000000 0.000000 1 0 /autogroup-113
[Mon Mar 11 14:29:19 2024] S smbd[10.87.27.7 2352778 993352.670934 457 120 0.000000 143.696125 0.000000 0.000000 1 0 /autogroup-113
[Mon Mar 11 14:29:19 2024] S smbd[10.87.29.1 2377277 993445.463755 463 120 0.000000 177.910535 0.000000 0.000000 1 0 /autogroup-113
[Mon Mar 11 14:29:19 2024] I kworker/21:0 2425747 19220269.892242 104836 120 0.000000 5100.510376 0.000000 0.000000 1 0 /
[Mon Mar 11 14:29:19 2024] S smbd[10.87.31.1 2432547 993352.640353 169 120 0.000000 70.783218 0.000000 0.000000 1 0 /autogroup-113
[Mon Mar 11 14:29:19 2024] D kworker/u98:17 2468089 19149447.567297 28079 120 0.000000 558.428481 0.000000 0.000000 1 0 /
[Mon Mar 11 14:29:19 2024] I kworker/u98:0 2471807 19208677.733933 99309 120 0.000000 4540.946140 0.000000 0.000000 1 0 /
[Mon Mar 11 14:29:19 2024] I kworker/21:2 2474756 19208593.509733 11 120 0.000000 0.141297 0.000000 0.000000 1 0 /
[Mon Mar 11 14:29:19 2024] cpu#22, 2600.000 MHz
[Mon Mar 11 14:29:19 2024] .nr_running : 0
[Mon Mar 11 14:29:19 2024] .nr_switches : 2828003316
[Mon Mar 11 14:29:19 2024] .nr_uninterruptible : 31570
[Mon Mar 11 14:29:19 2024] .next_balance : 5577.334876
[Mon Mar 11 14:29:19 2024] .curr->pid : 0
[Mon Mar 11 14:29:19 2024] .clock : 1282633451.163209
[Mon Mar 11 14:29:19 2024] .clock_task : 1258829139.958911
[Mon Mar 11 14:29:19 2024] .avg_idle : 1000000
[Mon Mar 11 14:29:19 2024] .max_idle_balance_cost : 500000
[Mon Mar 11 14:29:19 2024] cfs_rq[22]:/autogroup-113
[Mon Mar 11 14:29:19 2024] .exec_clock : 0.000000
[Mon Mar 11 14:29:19 2024] .MIN_vruntime : 0.000001
[Mon Mar 11 14:29:19 2024] .min_vruntime : 2563344.661088
[Mon Mar 11 14:29:19 2024] .max_vruntime : 0.000001
[Mon Mar 11 14:29:19 2024] .spread : 0.000000
[Mon Mar 11 14:29:19 2024] .spread0 : -17854321.847027
[Mon Mar 11 14:29:19 2024] .nr_spread_over : 0
[Mon Mar 11 14:29:19 2024] .nr_running : 0
[Mon Mar 11 14:29:19 2024] .h_nr_running : 0
[Mon Mar 11 14:29:19 2024] .idle_nr_running : 0
[Mon Mar 11 14:29:19 2024] .idle_h_nr_running : 0
[Mon Mar 11 14:29:19 2024] .load : 0
[Mon Mar 11 14:29:19 2024] .load_avg : 6
[Mon Mar 11 14:29:19 2024] .runnable_avg : 6
[Mon Mar 11 14:29:19 2024] .util_avg : 6
[Mon Mar 11 14:29:19 2024] .util_est_enqueued : 0
[Mon Mar 11 14:29:19 2024] .removed.load_avg : 0
[Mon Mar 11 14:29:19 2024] .removed.util_avg : 0
[Mon Mar 11 14:29:19 2024] .removed.runnable_avg : 0
[Mon Mar 11 14:29:19 2024] .tg_load_avg_contrib : 6
[Mon Mar 11 14:29:19 2024] .tg_load_avg : 102
[Mon Mar 11 14:29:19 2024] .throttled : 0
[Mon Mar 11 14:29:19 2024] .throttle_count : 0
[Mon Mar 11 14:29:19 2024] .se->exec_start : 1258829130.225422
[Mon Mar 11 14:29:19 2024] .se->vruntime : 40243007.765829
[Mon Mar 11 14:29:19 2024] .se->sum_exec_runtime : 2317918.909840
[Mon Mar 11 14:29:19 2024] .se->load.weight : 51909
[Mon Mar 11 14:29:19 2024] .se->avg.load_avg : 0
[Mon Mar 11 14:29:19 2024] .se->avg.util_avg : 6
[Mon Mar 11 14:29:19 2024] .se->avg.runnable_avg : 6
[Mon Mar 11 14:29:19 2024] cfs_rq[22]:/
[Mon Mar 11 14:29:19 2024] .exec_clock : 0.000000
[Mon Mar 11 14:29:19 2024] .MIN_vruntime : 0.000001
[Mon Mar 11 14:29:19 2024] .min_vruntime : 40243007.765829
[Mon Mar 11 14:29:19 2024] .max_vruntime : 0.000001
[Mon Mar 11 14:29:19 2024] .spread : 0.000000
[Mon Mar 11 14:29:19 2024] .spread0 : 19825341.257714
[Mon Mar 11 14:29:19 2024] .nr_spread_over : 0
[Mon Mar 11 14:29:19 2024] .nr_running : 0
[Mon Mar 11 14:29:19 2024] .h_nr_running : 0
[Mon Mar 11 14:29:19 2024] .idle_nr_running : 0
[Mon Mar 11 14:29:19 2024] .idle_h_nr_running : 0
[Mon Mar 11 14:29:19 2024] .load : 0
[Mon Mar 11 14:29:19 2024] .load_avg : 8
[Mon Mar 11 14:29:19 2024] .runnable_avg : 6
[Mon Mar 11 14:29:19 2024] .util_avg : 6
[Mon Mar 11 14:29:19 2024] .util_est_enqueued : 0
[Mon Mar 11 14:29:19 2024] .removed.load_avg : 0
[Mon Mar 11 14:29:19 2024] .removed.util_avg : 0
[Mon Mar 11 14:29:19 2024] .removed.runnable_avg : 0
[Mon Mar 11 14:29:19 2024] .tg_load_avg_contrib : 0
[Mon Mar 11 14:29:19 2024] .tg_load_avg : 0
[Mon Mar 11 14:29:19 2024] .throttled : 0
[Mon Mar 11 14:29:19 2024] .throttle_count : 0
[Mon Mar 11 14:29:19 2024] rt_rq[22]:
[Mon Mar 11 14:29:19 2024] .rt_nr_running : 0
[Mon Mar 11 14:29:19 2024] .rt_nr_migratory : 0
[Mon Mar 11 14:29:19 2024] .rt_throttled : 0
[Mon Mar 11 14:29:19 2024] .rt_time : 0.244233
[Mon Mar 11 14:29:19 2024] .rt_runtime : 950.000000
[Mon Mar 11 14:29:19 2024] dl_rq[22]:
[Mon Mar 11 14:29:19 2024] .dl_nr_running : 0
[Mon Mar 11 14:29:19 2024] .dl_nr_migratory : 0
[Mon Mar 11 14:29:19 2024] .dl_bw->bw : 996147
[Mon Mar 11 14:29:19 2024] .dl_bw->total_bw : 0
[Mon Mar 11 14:29:19 2024] runnable tasks:
[Mon Mar 11 14:29:19 2024] S task PID tree-key switches prio wait-time sum-exec sum-sleep
[Mon Mar 11 14:29:19 2024] -------------------------------------------------------------------------------------------------------------
[Mon Mar 11 14:29:19 2024] S cpuhp/22 150 21402.380149 25 120 0.000000 1.221941 0.000000 0.000000 0 0 /
[Mon Mar 11 14:29:19 2024] S idle_inject/22 151 -11.995205 3 49 0.000000 0.011427 0.000000 0.000000 0 0 /
[Mon Mar 11 14:29:19 2024] S migration/22 152 0.000000 383587 0 0.000000 1967.072244 0.000000 0.000000 0 0 /
[Mon Mar 11 14:29:19 2024] S ksoftirqd/22 153 40242994.971666 25427905 120 0.000000 306278.206932 0.000000 0.000000 0 0 /
[Mon Mar 11 14:29:19 2024] I kworker/22:0H 155 26.115073 4 100 0.000000 0.030861 0.000000 0.000000 0 0 /
[Mon Mar 11 14:29:19 2024] I kworker/22:1H 766 40242994.961953 3057485 100 0.000000 64809.087501 0.000000 0.000000 0 0 /
[Mon Mar 11 14:29:19 2024] S scsi_eh_1 897 2174.245846 25 120 0.000000 5.054257 0.000000 0.000000 0 0 /
[Mon Mar 11 14:29:19 2024] S irq/118-lpfc:22 957 0.000000 9010652 49 0.000000 175390.608767 0.000000 0.000000 0 0 /
[Mon Mar 11 14:29:19 2024] S irq/193-lpfc:22 1012 0.000000 9019551 49 0.000000 176127.281438 0.000000 0.000000 0 0 /
[Mon Mar 11 14:29:19 2024] S jbd2/dm-78-8 4219 40212107.891589 5126 120 0.000000 179.238307 0.000000 0.000000 0 0 /
[Mon Mar 11 14:29:19 2024] D nfsd 8784 40242950.005258 69603 120 0.000000 8387.456362 0.000000 0.000000 0 0 /
[Mon Mar 11 14:29:19 2024] D nfsd 8860 40144536.158368 24059818 120 0.000000 1013552.127209 0.000000 0.000000 0 0 /
[Mon Mar 11 14:29:19 2024] S automount 88131 17.460556 1 120 0.000000 0.898839 0.000000 0.000000 0 0 /autogroup-222
[Mon Mar 11 14:29:19 2024] S automount 88145 30.372599 1 120 0.000000 0.912050 0.000000 0.000000 0 0 /autogroup-222
[Mon Mar 11 14:29:19 2024] S smbd[10.87.29.9 2279987 2563247.133670 1378 120 0.000000 586.551502 0.000000 0.000000 0 0 /autogroup-113
[Mon Mar 11 14:29:19 2024] S smbd[10.87.27.1 2376851 2563199.166218 386 120 0.000000 101.924994 0.000000 0.000000 0 0 /autogroup-113
[Mon Mar 11 14:29:19 2024] S smbd[10.40.233. 2455152 2563344.661088 256894 120 0.000000 51451.562581 0.000000 0.000000 0 0 /autogroup-113
[Mon Mar 11 14:29:19 2024] I kworker/22:2 2458065 40242982.937937 81387 120 0.000000 5647.071224 0.000000 0.000000 0 0 /
[Mon Mar 11 14:29:19 2024] I kworker/22:1 2474143 40233379.342697 23 120 0.000000 0.081104 0.000000 0.000000 0 0 /
[Mon Mar 11 14:29:19 2024] S bash 2474608 1.140863 174 120 0.000000 88.433570 0.000000 0.000000 0 0 /autogroup-23274
[Mon Mar 11 14:29:19 2024] I kworker/22:0 2476753 40241331.317160 5 120 0.000000 0.069463 0.000000 0.000000 0 0 /
[Mon Mar 11 14:29:19 2024] S sleep 2478153 85500.673951 1 120 0.000000 1.066047 0.000000 0.000000 0 0 /autogroup-292
[Mon Mar 11 14:29:19 2024] cpu#23, 2600.000 MHz
[Mon Mar 11 14:29:19 2024] .nr_running : 0
[Mon Mar 11 14:29:19 2024] .nr_switches : 952435926
[Mon Mar 11 14:29:19 2024] .nr_uninterruptible : 785
[Mon Mar 11 14:29:19 2024] .next_balance : 5577.334875
[Mon Mar 11 14:29:19 2024] .curr->pid : 0
[Mon Mar 11 14:29:19 2024] .clock : 1282633451.160900
[Mon Mar 11 14:29:19 2024] .clock_task : 1281142140.549718
[Mon Mar 11 14:29:19 2024] .avg_idle : 1000000
[Mon Mar 11 14:29:19 2024] .max_idle_balance_cost : 500000
[Mon Mar 11 14:29:19 2024] rt_rq[23]:
[Mon Mar 11 14:29:19 2024] .rt_nr_running : 0
[Mon Mar 11 14:29:19 2024] .rt_nr_migratory : 0
[Mon Mar 11 14:29:19 2024] .rt_throttled : 0
[Mon Mar 11 14:29:19 2024] .rt_time : 0.000000
[Mon Mar 11 14:29:19 2024] .rt_runtime : 950.000000
[Mon Mar 11 14:29:19 2024] dl_rq[23]:
[Mon Mar 11 14:29:19 2024] .dl_nr_running : 0
[Mon Mar 11 14:29:19 2024] .dl_nr_migratory : 0
[Mon Mar 11 14:29:19 2024] .dl_bw->bw : 996147
[Mon Mar 11 14:29:19 2024] .dl_bw->total_bw : 0
[Mon Mar 11 14:29:19 2024] runnable tasks:
[Mon Mar 11 14:29:19 2024] S task PID tree-key switches prio wait-time sum-exec sum-sleep
[Mon Mar 11 14:29:19 2024] -------------------------------------------------------------------------------------------------------------
[Mon Mar 11 14:29:19 2024] S cpuhp/23 156 25666.725145 25 120 0.000000 1.041153 0.000000 0.000000 1 0 /
[Mon Mar 11 14:29:19 2024] S idle_inject/23 157 -12.000000 3 49 0.000000 0.000000 0.000000 0.000000 1 0 /
[Mon Mar 11 14:29:19 2024] S migration/23 158 1377.822149 323858 0 0.000000 1902.997882 0.000000 0.000000 1 0 /
[Mon Mar 11 14:29:19 2024] S ksoftirqd/23 159 18608895.385798 12287619 120 0.000000 115302.907533 0.000000 0.000000 1 0 /
[Mon Mar 11 14:29:19 2024] I kworker/23:0H 161 112.099119 4 100 0.000000 0.027491 0.000000 0.000000 1 0 /
[Mon Mar 11 14:29:19 2024] I kworker/23:1H 858 18608895.379447 3903217 100 0.000000 87479.525904 0.000000 0.000000 1 0 /
[Mon Mar 11 14:29:19 2024] S scsi_eh_0 869 361.861815 2 120 0.000000 0.005336 0.000000 0.000000 1 0 /
[Mon Mar 11 14:29:19 2024] S irq/119-lpfc:23 958 0.000000 6013015 49 0.000000 132947.985111 0.000000 0.000000 1 0 /
[Mon Mar 11 14:29:19 2024] S irq/194-lpfc:23 1013 0.000000 6028473 49 0.000000 133837.073157 0.000000 0.000000 1 0 /
[Mon Mar 11 14:29:19 2024] S multipathd 3352 24.687224 19 120 0.000000 1.447322 0.000000 0.000000 1 0 /autogroup-45
[Mon Mar 11 14:29:19 2024] I kdmflush/253:16 3520 27086.146499 2 100 0.000000 0.015280 0.000000 0.000000 1 0 /
[Mon Mar 11 14:29:19 2024] I kdmflush/253:19 3544 27160.529421 2 100 0.000000 0.016152 0.000000 0.000000 1 0 /
[Mon Mar 11 14:29:19 2024] I kdmflush/253:20 3554 27172.541157 2 100 0.000000 0.013776 0.000000 0.000000 1 0 /
[Mon Mar 11 14:29:19 2024] I kdmflush/253:33 3667 27186.703773 2 100 0.000000 0.009435 0.000000 0.000000 1 0 /
[Mon Mar 11 14:29:19 2024] I kdmflush/253:73 4086 28148.232554 2 100 0.000000 0.016868 0.000000 0.000000 1 0 /
[Mon Mar 11 14:29:19 2024] S auditd 4254 944.938064 251145 116 0.000000 42840.017228 0.000000 0.000000 1 0 /autogroup-62
[Mon Mar 11 14:29:19 2024] I bond0 4392 28372.006056 2 100 0.000000 0.023458 0.000000 0.000000 1 0 /
[Mon Mar 11 14:29:19 2024] S JS Helper 5175 11.162914 9 120 0.000000 0.211497 0.000000 0.000000 1 0 /autogroup-104
[Mon Mar 11 14:29:19 2024] S jbd2/dm-97-8 6477 18495169.132305 2604 120 0.000000 124.399513 0.000000 0.000000 1 0 /
[Mon Mar 11 14:29:19 2024] I ext4-rsv-conver 6489 29630.607598 2 100 0.000000 0.063328 0.000000 0.000000 1 0 /
[Mon Mar 11 14:29:19 2024] I ib_mcast 8764 29960.631165 2 100 0.000000 0.071093 0.000000 0.000000 1 0 /
[Mon Mar 11 14:29:19 2024] D nfsd 8806 18609016.993217 129194 120 0.000000 13400.081893 0.000000 0.000000 1 0 /
[Mon Mar 11 14:29:19 2024] D nfsd 8832 18597529.564306 5346074 120 0.000000 274914.850237 0.000000 0.000000 1 0 /
[Mon Mar 11 14:29:19 2024] S smbd[10.87.29.9 2346134 919880.063625 3814 120 0.000000 1177.145176 0.000000 0.000000 1 0 /autogroup-113
[Mon Mar 11 14:29:19 2024] I kworker/23:0 2469472 18609154.536183 21772 120 0.000000 1320.907831 0.000000 0.000000 1 0 /
[Mon Mar 11 14:29:19 2024] I kworker/23:1 2474524 18598648.636634 9 120 0.000000 0.054034 0.000000 0.000000 1 0 /
[Mon Mar 11 14:29:19 2024] S sshd 2474605 344.020612 7161 120 0.000000 458.950673 0.000000 0.000000 1 0 /autogroup-23273
[Mon Mar 11 14:29:19 2024] I kworker/23:2 2477164 18598660.621122 2 120 0.000000 0.010650 0.000000 0.000000 1 0 /
[Mon Mar 11 14:29:19 2024] cpu#24, 2600.000 MHz
[Mon Mar 11 14:29:19 2024] .nr_running : 0
[Mon Mar 11 14:29:19 2024] .nr_switches : 1945978603
[Mon Mar 11 14:29:19 2024] .nr_uninterruptible : 7002
[Mon Mar 11 14:29:19 2024] .next_balance : 5577.334875
[Mon Mar 11 14:29:19 2024] .curr->pid : 0
[Mon Mar 11 14:29:19 2024] .clock : 1282633451.161485
[Mon Mar 11 14:29:19 2024] .clock_task : 1260960834.948137
[Mon Mar 11 14:29:19 2024] .avg_idle : 1000000
[Mon Mar 11 14:29:19 2024] .max_idle_balance_cost : 500000
[Mon Mar 11 14:29:19 2024] cfs_rq[24]:/
[Mon Mar 11 14:29:19 2024] .exec_clock : 0.000000
[Mon Mar 11 14:29:19 2024] .MIN_vruntime : 0.000001
[Mon Mar 11 14:29:19 2024] .min_vruntime : 31682885.436619
[Mon Mar 11 14:29:19 2024] .max_vruntime : 0.000001
[Mon Mar 11 14:29:19 2024] .spread : 0.000000
[Mon Mar 11 14:29:19 2024] .spread0 : 11265218.928504
[Mon Mar 11 14:29:19 2024] .nr_spread_over : 0
[Mon Mar 11 14:29:19 2024] .nr_running : 0
[Mon Mar 11 14:29:19 2024] .h_nr_running : 0
[Mon Mar 11 14:29:19 2024] .idle_nr_running : 0
[Mon Mar 11 14:29:19 2024] .idle_h_nr_running : 0
[Mon Mar 11 14:29:19 2024] .load : 0
[Mon Mar 11 14:29:19 2024] .load_avg : 0
[Mon Mar 11 14:29:19 2024] .runnable_avg : 0
[Mon Mar 11 14:29:19 2024] .util_avg : 0
[Mon Mar 11 14:29:19 2024] .util_est_enqueued : 0
[Mon Mar 11 14:29:19 2024] .removed.load_avg : 0
[Mon Mar 11 14:29:19 2024] .removed.util_avg : 0
[Mon Mar 11 14:29:19 2024] .removed.runnable_avg : 0
[Mon Mar 11 14:29:19 2024] .tg_load_avg_contrib : 0
[Mon Mar 11 14:29:19 2024] .tg_load_avg : 0
[Mon Mar 11 14:29:19 2024] .throttled : 0
[Mon Mar 11 14:29:19 2024] .throttle_count : 0
[Mon Mar 11 14:29:19 2024] rt_rq[24]:
[Mon Mar 11 14:29:19 2024] .rt_nr_running : 0
[Mon Mar 11 14:29:19 2024] .rt_nr_migratory : 0
[Mon Mar 11 14:29:19 2024] .rt_throttled : 0
[Mon Mar 11 14:29:19 2024] .rt_time : 0.000000
[Mon Mar 11 14:29:19 2024] .rt_runtime : 950.000000
[Mon Mar 11 14:29:19 2024] dl_rq[24]:
[Mon Mar 11 14:29:19 2024] .dl_nr_running : 0
[Mon Mar 11 14:29:19 2024] .dl_nr_migratory : 0
[Mon Mar 11 14:29:19 2024] .dl_bw->bw : 996147
[Mon Mar 11 14:29:19 2024] .dl_bw->total_bw : 0
[Mon Mar 11 14:29:19 2024] runnable tasks:
[Mon Mar 11 14:29:19 2024] S task PID tree-key switches prio wait-time sum-exec sum-sleep
[Mon Mar 11 14:29:19 2024] -------------------------------------------------------------------------------------------------------------
[Mon Mar 11 14:29:19 2024] S cpuhp/24 162 23393.664740 25 120 0.000000 1.542294 0.000000 0.000000 0 0 /
[Mon Mar 11 14:29:19 2024] S idle_inject/24 163 -12.000000 3 49 0.000000 0.003321 0.000000 0.000000 0 0 /
[Mon Mar 11 14:29:19 2024] S migration/24 164 1445.822873 376915 0 0.000000 2100.214786 0.000000 0.000000 0 0 /
[Mon Mar 11 14:29:19 2024] S ksoftirqd/24 165 31682504.570533 6107745 120 0.000000 129659.113170 0.000000 0.000000 0 0 /
[Mon Mar 11 14:29:19 2024] I kworker/24:0H 167 1118.798300 4 100 0.000000 0.047430 0.000000 0.000000 0 0 /
[Mon Mar 11 14:29:19 2024] S kcompactd0 364 31682198.784436 39248453 120 0.000000 43301648.709535 0.000000 0.000000 0 0 /
[Mon Mar 11 14:29:19 2024] I kblockd 370 53.253502 2 100 0.000000 0.001368 0.000000 0.000000 0 0 /
[Mon Mar 11 14:29:19 2024] I kworker/24:1H 768 31682872.632778 1213017 100 0.000000 35678.223987 0.000000 0.000000 0 0 /
[Mon Mar 11 14:29:19 2024] I scsi_tmf_2 901 2713.897643 2 100 0.000000 0.012790 0.000000 0.000000 0 0 /
[Mon Mar 11 14:29:19 2024] S scsi_eh_10 920 2722.756590 27 120 0.000000 4.679658 0.000000 0.000000 0 0 /
[Mon Mar 11 14:29:19 2024] S rpcbind 4250 279.561892 56341 120 0.000000 9059.901200 0.000000 0.000000 0 0 /autogroup-61
[Mon Mar 11 14:29:19 2024] S nsrexecd 5346 3774.551353 643778 120 0.000000 30115.929347 0.000000 0.000000 0 5346 /autogroup-117
[Mon Mar 11 14:29:19 2024] S jbd2/dm-93-8 6467 31523220.237211 518 120 0.000000 27.247637 0.000000 0.000000 0 0 /
[Mon Mar 11 14:29:19 2024] D nfsd 8833 31604688.052593 686363 120 0.000000 47271.239267 0.000000 0.000000 0 0 /
[Mon Mar 11 14:29:19 2024] S smbd[10.87.30.3 2341031 2240546.214685 564 120 0.000000 181.558215 0.000000 0.000000 0 0 /autogroup-113
[Mon Mar 11 14:29:19 2024] S smbd[10.87.27.1 2344554 2240544.812087 781 120 0.000000 200.331122 0.000000 0.000000 0 0 /autogroup-113
[Mon Mar 11 14:29:19 2024] S smbd[10.87.18.9 2432065 2240546.215642 178 120 0.000000 75.915833 0.000000 0.000000 0 0 /autogroup-113
[Mon Mar 11 14:29:19 2024] S smbd[10.87.31.4 2433848 2240505.044033 154 120 0.000000 60.890954 0.000000 0.000000 0 0 /autogroup-113
[Mon Mar 11 14:29:19 2024] I kworker/24:2 2453510 31682873.369693 32225 120 0.000000 2143.587188 0.000000 0.000000 0 0 /
[Mon Mar 11 14:29:19 2024] I kworker/24:1 2474838 31672624.576239 5 120 0.000000 0.068126 0.000000 0.000000 0 0 /
[Mon Mar 11 14:29:19 2024] S rpcd_spoolss 2478224 21971.978621 50 120 0.000000 81.756013 0.000000 0.000000 0 0 /autogroup-22820
[Mon Mar 11 14:29:19 2024] cpu#25, 2600.000 MHz
[Mon Mar 11 14:29:19 2024] .nr_running : 0
[Mon Mar 11 14:29:19 2024] .nr_switches : 934302642
[Mon Mar 11 14:29:19 2024] .nr_uninterruptible : -22436
[Mon Mar 11 14:29:19 2024] .next_balance : 5577.334873
[Mon Mar 11 14:29:19 2024] .curr->pid : 0
[Mon Mar 11 14:29:19 2024] .clock : 1282633451.162025
[Mon Mar 11 14:29:19 2024] .clock_task : 1281056889.770396
[Mon Mar 11 14:29:19 2024] .avg_idle : 1000000
[Mon Mar 11 14:29:19 2024] .max_idle_balance_cost : 500000
[Mon Mar 11 14:29:19 2024] rt_rq[25]:
[Mon Mar 11 14:29:19 2024] .rt_nr_running : 0
[Mon Mar 11 14:29:19 2024] .rt_nr_migratory : 0
[Mon Mar 11 14:29:19 2024] .rt_throttled : 0
[Mon Mar 11 14:29:19 2024] .rt_time : 0.000000
[Mon Mar 11 14:29:19 2024] .rt_runtime : 950.000000
[Mon Mar 11 14:29:19 2024] dl_rq[25]:
[Mon Mar 11 14:29:19 2024] .dl_nr_running : 0
[Mon Mar 11 14:29:19 2024] .dl_nr_migratory : 0
[Mon Mar 11 14:29:19 2024] .dl_bw->bw : 996147
[Mon Mar 11 14:29:19 2024] .dl_bw->total_bw : 0
[Mon Mar 11 14:29:19 2024] runnable tasks:
[Mon Mar 11 14:29:19 2024] S task PID tree-key switches prio wait-time sum-exec sum-sleep
[Mon Mar 11 14:29:19 2024] -------------------------------------------------------------------------------------------------------------
[Mon Mar 11 14:29:19 2024] S cpuhp/25 168 24885.855059 25 120 0.000000 0.768376 0.000000 0.000000 1 0 /
[Mon Mar 11 14:29:19 2024] S idle_inject/25 169 -12.000000 3 49 0.000000 0.000000 0.000000 0.000000 1 0 /
[Mon Mar 11 14:29:19 2024] S migration/25 170 1513.822841 323311 0 0.000000 1902.890051 0.000000 0.000000 1 0 /
[Mon Mar 11 14:29:19 2024] S ksoftirqd/25 171 20632998.120255 1885551 120 0.000000 58547.705586 0.000000 0.000000 1 0 /
[Mon Mar 11 14:29:19 2024] I kworker/25:0H 173 1587.428642 4 100 0.000000 0.045985 0.000000 0.000000 1 0 /
[Mon Mar 11 14:29:19 2024] I scsi_tmf_8 917 1538.904813 2 100 0.000000 0.012754 0.000000 0.000000 1 0 /
[Mon Mar 11 14:29:19 2024] I scsi_tmf_10 921 1550.914652 2 100 0.000000 0.011461 0.000000 0.000000 1 0 /
[Mon Mar 11 14:29:19 2024] I kworker/25:1H 962 20632471.806627 996076 100 0.000000 35301.941463 0.000000 0.000000 1 0 /
[Mon Mar 11 14:29:19 2024] I dm_bufio_cache 3811 25802.615710 2 100 0.000000 0.006567 0.000000 0.000000 1 0 /
[Mon Mar 11 14:29:19 2024] I ext4-rsv-conver 4245 26119.690907 2 100 0.000000 0.015519 0.000000 0.000000 1 0 /
[Mon Mar 11 14:29:19 2024] S rpc.idmapd 4301 2950.453399 948878 120 0.000000 123631.196827 0.000000 0.000000 1 0 /autogroup-68
[Mon Mar 11 14:29:19 2024] S rs:action-1-bui 4867 35046.877416 263332 120 0.000000 21148.632049 0.000000 0.000000 1 4838 /autogroup-112
[Mon Mar 11 14:29:19 2024] S nsrexecd 5342 3799.218789 25728 120 0.000000 1553.951151 0.000000 0.000000 1 5342 /autogroup-117
[Mon Mar 11 14:29:19 2024] S dmeventd 8698 2319.804457 151145 120 0.000000 21099.248873 0.000000 0.000000 1 8698 /autogroup-125
[Mon Mar 11 14:29:19 2024] D nfsd 8799 20633199.012571 95725 120 0.000000 10695.949003 0.000000 0.000000 1 0 /
[Mon Mar 11 14:29:19 2024] S gdbus 21808 67.110399 2245 120 0.000000 253.886466 0.000000 0.000000 1 0 /autogroup-204
[Mon Mar 11 14:29:19 2024] S smbd[10.91.30.5 86031 2261157.035605 4401802 120 0.000000 3865085.547867 0.000000 0.000000 1 0 /autogroup-113
[Mon Mar 11 14:29:19 2024] I kworker/25:1 2344356 20633369.242049 136226 120 0.000000 5756.346566 0.000000 0.000000 1 0 /
[Mon Mar 11 14:29:19 2024] S smbd[10.87.27.1 2472086 2260850.549800 45 120 0.000000 39.249706 0.000000 0.000000 1 0 /autogroup-113
[Mon Mar 11 14:29:19 2024] I kworker/25:2 2473359 20624769.626038 6 120 0.000000 0.044961 0.000000 0.000000 1 0 /
[Mon Mar 11 14:29:19 2024] S journalctl 2475167 380.504202 273 120 0.000000 426.435553 0.000000 0.000000 1 0 /autogroup-23274
[Mon Mar 11 14:29:19 2024] I kworker/25:0 2476365 20624781.604785 2 120 0.000000 0.014551 0.000000 0.000000 1 0 /
[Mon Mar 11 14:29:19 2024] cpu#26, 2600.000 MHz
[Mon Mar 11 14:29:19 2024] .nr_running : 0
[Mon Mar 11 14:29:19 2024] .nr_switches : 2457302330
[Mon Mar 11 14:29:19 2024] .nr_uninterruptible : 26766
[Mon Mar 11 14:29:19 2024] .next_balance : 5577.334876
[Mon Mar 11 14:29:19 2024] .curr->pid : 0
[Mon Mar 11 14:29:19 2024] .clock : 1282633452.161932
[Mon Mar 11 14:29:19 2024] .clock_task : 1262053718.432981
[Mon Mar 11 14:29:19 2024] .avg_idle : 1000000
[Mon Mar 11 14:29:19 2024] .max_idle_balance_cost : 500000
[Mon Mar 11 14:29:19 2024] rt_rq[26]:
[Mon Mar 11 14:29:19 2024] .rt_nr_running : 0
[Mon Mar 11 14:29:19 2024] .rt_nr_migratory : 0
[Mon Mar 11 14:29:19 2024] .rt_throttled : 0
[Mon Mar 11 14:29:19 2024] .rt_time : 0.000000
[Mon Mar 11 14:29:19 2024] .rt_runtime : 950.000000
[Mon Mar 11 14:29:19 2024] dl_rq[26]:
[Mon Mar 11 14:29:19 2024] .dl_nr_running : 0
[Mon Mar 11 14:29:19 2024] .dl_nr_migratory : 0
[Mon Mar 11 14:29:19 2024] .dl_bw->bw : 996147
[Mon Mar 11 14:29:19 2024] .dl_bw->total_bw : 0
[Mon Mar 11 14:29:19 2024] runnable tasks:
[Mon Mar 11 14:29:19 2024] S task PID tree-key switches prio wait-time sum-exec sum-sleep
[Mon Mar 11 14:29:19 2024] -------------------------------------------------------------------------------------------------------------
[Mon Mar 11 14:29:19 2024] S cpuhp/26 174 23606.902582 25 120 0.000000 0.760992 0.000000 0.000000 0 0 /
[Mon Mar 11 14:29:19 2024] S idle_inject/26 175 -12.000000 3 49 0.000000 0.000000 0.000000 0.000000 0 0 /
[Mon Mar 11 14:29:19 2024] S migration/26 176 1581.822817 363506 0 0.000000 1914.526254 0.000000 0.000000 0 0 /
[Mon Mar 11 14:29:19 2024] S ksoftirqd/26 177 36791242.577690 7382776 120 0.000000 180914.912083 0.000000 0.000000 0 0 /
[Mon Mar 11 14:29:19 2024] I kworker/26:0H 179 14.881818 4 100 0.000000 0.028332 0.000000 0.000000 0 0 /
[Mon Mar 11 14:29:19 2024] I kworker/26:1H 770 36791236.458131 1649062 100 0.000000 53845.018222 0.000000 0.000000 0 0 /
[Mon Mar 11 14:29:19 2024] S scsi_eh_6 908 2137.437479 25 120 0.000000 6.102952 0.000000 0.000000 0 0 /
[Mon Mar 11 14:29:19 2024] S jbd2/dm-81-8 4308 36776987.299253 30974 120 0.000000 4963.794561 0.000000 0.000000 0 0 /
[Mon Mar 11 14:29:19 2024] S sssd 4316 76246.349498 42954 120 0.000000 2311.007455 0.000000 0.000000 0 0 /autogroup-78
[Mon Mar 11 14:29:19 2024] D nfsd 8845 36723414.897139 8901131 120 0.000000 282223.184362 0.000000 0.000000 0 0 /
[Mon Mar 11 14:29:19 2024] D nfsd 8851 36718568.345560 8652720 120 0.000000 372915.329855 0.000000 0.000000 0 0 /
[Mon Mar 11 14:29:19 2024] S colord 21791 142.073761 1066 120 0.000000 373.718129 0.000000 0.000000 0 0 /autogroup-204
[Mon Mar 11 14:29:19 2024] S automount 88101 11.213237 1 120 0.000000 0.261820 0.000000 0.000000 0 0 /autogroup-222
[Mon Mar 11 14:29:19 2024] S automount 88134 37.019646 1 120 0.000000 0.912702 0.000000 0.000000 0 0 /autogroup-222
[Mon Mar 11 14:29:19 2024] S automount 88146 49.929411 1 120 0.000000 0.909772 0.000000 0.000000 0 0 /autogroup-222
[Mon Mar 11 14:29:19 2024] S smbd[134.58.56. 960684 2068301.895701 5384 120 0.000000 1460.852050 0.000000 0.000000 0 0 /autogroup-113
[Mon Mar 11 14:29:19 2024] S smbd[10.87.28.1 2322491 2068259.870093 5379 120 0.000000 2308.866908 0.000000 0.000000 0 0 /autogroup-113
[Mon Mar 11 14:29:19 2024] I kworker/26:2 2401639 36791243.818499 257596 120 0.000000 11027.920229 0.000000 0.000000 0 0 /
[Mon Mar 11 14:29:19 2024] I kworker/u97:3 2462779 36789246.926220 379069 120 0.000000 15750.222948 0.000000 0.000000 0 0 /
[Mon Mar 11 14:29:19 2024] I kworker/26:1 2474460 36786323.377963 9 120 0.000000 0.049250 0.000000 0.000000 0 0 /
[Mon Mar 11 14:29:19 2024] I kworker/26:0 2477469 36786622.575759 3 120 0.000000 0.061638 0.000000 0.000000 0 0 /
[Mon Mar 11 14:29:19 2024] cpu#27, 2600.000 MHz
[Mon Mar 11 14:29:19 2024] .nr_running : 0
[Mon Mar 11 14:29:19 2024] .nr_switches : 1002480564
[Mon Mar 11 14:29:19 2024] .nr_uninterruptible : -10955
[Mon Mar 11 14:29:19 2024] .next_balance : 5577.334876
[Mon Mar 11 14:29:19 2024] .curr->pid : 0
[Mon Mar 11 14:29:19 2024] .clock : 1282633452.161230
[Mon Mar 11 14:29:19 2024] .clock_task : 1281082968.810087
[Mon Mar 11 14:29:19 2024] .avg_idle : 1000000
[Mon Mar 11 14:29:19 2024] .max_idle_balance_cost : 500000
[Mon Mar 11 14:29:19 2024] rt_rq[27]:
[Mon Mar 11 14:29:19 2024] .rt_nr_running : 0
[Mon Mar 11 14:29:19 2024] .rt_nr_migratory : 0
[Mon Mar 11 14:29:19 2024] .rt_throttled : 0
[Mon Mar 11 14:29:19 2024] .rt_time : 0.000000
[Mon Mar 11 14:29:19 2024] .rt_runtime : 950.000000
[Mon Mar 11 14:29:19 2024] dl_rq[27]:
[Mon Mar 11 14:29:19 2024] .dl_nr_running : 0
[Mon Mar 11 14:29:19 2024] .dl_nr_migratory : 0
[Mon Mar 11 14:29:19 2024] .dl_bw->bw : 996147
[Mon Mar 11 14:29:19 2024] .dl_bw->total_bw : 0
[Mon Mar 11 14:29:19 2024] runnable tasks:
[Mon Mar 11 14:29:19 2024] S task PID tree-key switches prio wait-time sum-exec sum-sleep
[Mon Mar 11 14:29:19 2024] -------------------------------------------------------------------------------------------------------------
[Mon Mar 11 14:29:19 2024] S cpuhp/27 180 26958.097613 25 120 0.000000 0.856703 0.000000 0.000000 1 0 /
[Mon Mar 11 14:29:19 2024] S idle_inject/27 181 -11.994604 3 49 0.000000 0.011110 0.000000 0.000000 1 0 /
[Mon Mar 11 14:29:19 2024] S migration/27 182 1649.822816 323698 0 0.000000 1851.914656 0.000000 0.000000 1 0 /
[Mon Mar 11 14:29:19 2024] S ksoftirqd/27 183 20569460.594109 2122703 120 0.000000 68419.959615 0.000000 0.000000 1 0 /
[Mon Mar 11 14:29:19 2024] I kworker/27:0H 185 58.938747 4 100 0.000000 0.021159 0.000000 0.000000 1 0 /
[Mon Mar 11 14:29:19 2024] I kworker/27:1H 828 20569850.225369 909599 100 0.000000 31243.623932 0.000000 0.000000 1 0 /
[Mon Mar 11 14:29:19 2024] I nvme_fc_wq 889 2067.194452 2 100 0.000000 0.008398 0.000000 0.000000 1 0 /
[Mon Mar 11 14:29:19 2024] I rpciod 2794 6757.087854 2 100 0.000000 0.013709 0.000000 0.000000 1 0 /
[Mon Mar 11 14:29:19 2024] I xprtiod 2795 6769.097604 2 100 0.000000 0.011074 0.000000 0.000000 1 0 /
[Mon Mar 11 14:29:19 2024] S systemd-udevd 2816 98460.616950 760799 120 0.000000 1248069.917916 0.000000 0.000000 1 0 /autogroup-39
[Mon Mar 11 14:29:19 2024] I kdmflush/253:56 3874 27886.389931 2 100 0.000000 0.008810 0.000000 0.000000 1 0 /
[Mon Mar 11 14:29:19 2024] S auditd 4255 731.981661 39721 116 0.000000 1325.433456 0.000000 0.000000 1 0 /autogroup-62
[Mon Mar 11 14:29:19 2024] D nfsd 8786 20569859.045812 74089 120 0.000000 8697.077926 0.000000 0.000000 1 0 /
[Mon Mar 11 14:29:19 2024] D nfsd 8853 20500589.645296 10007686 120 0.000000 442380.593217 0.000000 0.000000 1 0 /
[Mon Mar 11 14:29:19 2024] I kworker/27:2 2144617 20569859.954817 210170 120 0.000000 10137.066621 0.000000 0.000000 1 0 /
[Mon Mar 11 14:29:19 2024] S smbd[10.87.27.1 2343951 1580634.286320 1118 120 0.000000 250.973953 0.000000 0.000000 1 0 /autogroup-113
[Mon Mar 11 14:29:19 2024] S rpcd_winreg 2347933 208314.687886 592979 120 0.000000 150673.165789 0.000000 0.000000 1 0 /autogroup-22820
[Mon Mar 11 14:29:19 2024] S smbd[10.40.138. 2377602 1580633.988057 341 120 0.000000 118.763343 0.000000 0.000000 1 0 /autogroup-113
[Mon Mar 11 14:29:19 2024] S smbd[10.87.18.7 2425212 1580633.990081 354 120 0.000000 152.083427 0.000000 0.000000 1 0 /autogroup-113
[Mon Mar 11 14:29:19 2024] S smbd[10.87.29.8 2459178 1580639.767451 15144 120 0.000000 2988.473061 0.000000 0.000000 1 0 /autogroup-113
[Mon Mar 11 14:29:19 2024] I kworker/27:0 2473569 20558278.057202 5 120 0.000000 0.028884 0.000000 0.000000 1 0 /
[Mon Mar 11 14:29:19 2024] I kworker/27:1 2476181 20558693.970875 4 120 0.000000 0.076907 0.000000 0.000000 1 0 /
[Mon Mar 11 14:29:19 2024] cpu#28, 2600.000 MHz
[Mon Mar 11 14:29:19 2024] .nr_running : 0
[Mon Mar 11 14:29:19 2024] .nr_switches : 2333117815
[Mon Mar 11 14:29:19 2024] .nr_uninterruptible : 36485
[Mon Mar 11 14:29:19 2024] .next_balance : 5577.334876
[Mon Mar 11 14:29:19 2024] .curr->pid : 0
[Mon Mar 11 14:29:19 2024] .clock : 1282633452.161449
[Mon Mar 11 14:29:19 2024] .clock_task : 1266800310.542179
[Mon Mar 11 14:29:19 2024] .avg_idle : 983160
[Mon Mar 11 14:29:19 2024] .max_idle_balance_cost : 500000
[Mon Mar 11 14:29:19 2024] cfs_rq[28]:/autogroup-113
[Mon Mar 11 14:29:19 2024] .exec_clock : 0.000000
[Mon Mar 11 14:29:19 2024] .MIN_vruntime : 0.000001
[Mon Mar 11 14:29:19 2024] .min_vruntime : 2287115.375726
[Mon Mar 11 14:29:19 2024] .max_vruntime : 0.000001
[Mon Mar 11 14:29:19 2024] .spread : 0.000000
[Mon Mar 11 14:29:19 2024] .spread0 : -18130551.132389
[Mon Mar 11 14:29:19 2024] .nr_spread_over : 0
[Mon Mar 11 14:29:19 2024] .nr_running : 0
[Mon Mar 11 14:29:19 2024] .h_nr_running : 0
[Mon Mar 11 14:29:19 2024] .idle_nr_running : 0
[Mon Mar 11 14:29:19 2024] .idle_h_nr_running : 0
[Mon Mar 11 14:29:19 2024] .load : 0
[Mon Mar 11 14:29:19 2024] .load_avg : 3
[Mon Mar 11 14:29:19 2024] .runnable_avg : 3
[Mon Mar 11 14:29:19 2024] .util_avg : 3
[Mon Mar 11 14:29:19 2024] .util_est_enqueued : 0
[Mon Mar 11 14:29:19 2024] .removed.load_avg : 0
[Mon Mar 11 14:29:19 2024] .removed.util_avg : 0
[Mon Mar 11 14:29:19 2024] .removed.runnable_avg : 0
[Mon Mar 11 14:29:19 2024] .tg_load_avg_contrib : 3
[Mon Mar 11 14:29:19 2024] .tg_load_avg : 102
[Mon Mar 11 14:29:19 2024] .throttled : 0
[Mon Mar 11 14:29:19 2024] .throttle_count : 0
[Mon Mar 11 14:29:19 2024] .se->exec_start : 1266800238.818173
[Mon Mar 11 14:29:19 2024] .se->vruntime : 35351482.597983
[Mon Mar 11 14:29:19 2024] .se->sum_exec_runtime : 2168728.352752
[Mon Mar 11 14:29:19 2024] .se->load.weight : 122164
[Mon Mar 11 14:29:19 2024] .se->avg.load_avg : 0
[Mon Mar 11 14:29:19 2024] .se->avg.util_avg : 3
[Mon Mar 11 14:29:19 2024] .se->avg.runnable_avg : 3
[Mon Mar 11 14:29:19 2024] cfs_rq[28]:/
[Mon Mar 11 14:29:19 2024] .exec_clock : 0.000000
[Mon Mar 11 14:29:19 2024] .MIN_vruntime : 0.000001
[Mon Mar 11 14:29:19 2024] .min_vruntime : 35351491.742307
[Mon Mar 11 14:29:19 2024] .max_vruntime : 0.000001
[Mon Mar 11 14:29:19 2024] .spread : 0.000000
[Mon Mar 11 14:29:19 2024] .spread0 : 14933825.234192
[Mon Mar 11 14:29:19 2024] .nr_spread_over : 0
[Mon Mar 11 14:29:19 2024] .nr_running : 0
[Mon Mar 11 14:29:19 2024] .h_nr_running : 0
[Mon Mar 11 14:29:19 2024] .idle_nr_running : 0
[Mon Mar 11 14:29:19 2024] .idle_h_nr_running : 0
[Mon Mar 11 14:29:19 2024] .load : 0
[Mon Mar 11 14:29:19 2024] .load_avg : 6
[Mon Mar 11 14:29:19 2024] .runnable_avg : 3
[Mon Mar 11 14:29:19 2024] .util_avg : 3
[Mon Mar 11 14:29:19 2024] .util_est_enqueued : 0
[Mon Mar 11 14:29:19 2024] .removed.load_avg : 0
[Mon Mar 11 14:29:19 2024] .removed.util_avg : 0
[Mon Mar 11 14:29:19 2024] .removed.runnable_avg : 0
[Mon Mar 11 14:29:19 2024] .tg_load_avg_contrib : 0
[Mon Mar 11 14:29:19 2024] .tg_load_avg : 0
[Mon Mar 11 14:29:19 2024] .throttled : 0
[Mon Mar 11 14:29:19 2024] .throttle_count : 0
[Mon Mar 11 14:29:19 2024] rt_rq[28]:
[Mon Mar 11 14:29:19 2024] .rt_nr_running : 0
[Mon Mar 11 14:29:19 2024] .rt_nr_migratory : 0
[Mon Mar 11 14:29:19 2024] .rt_throttled : 0
[Mon Mar 11 14:29:19 2024] .rt_time : 0.741596
[Mon Mar 11 14:29:19 2024] .rt_runtime : 950.000000
[Mon Mar 11 14:29:19 2024] dl_rq[28]:
[Mon Mar 11 14:29:19 2024] .dl_nr_running : 0
[Mon Mar 11 14:29:19 2024] .dl_nr_migratory : 0
[Mon Mar 11 14:29:19 2024] .dl_bw->bw : 996147
[Mon Mar 11 14:29:19 2024] .dl_bw->total_bw : 0
[Mon Mar 11 14:29:19 2024] runnable tasks:
[Mon Mar 11 14:29:19 2024] S task PID tree-key switches prio wait-time sum-exec sum-sleep
[Mon Mar 11 14:29:19 2024] -------------------------------------------------------------------------------------------------------------
[Mon Mar 11 14:29:19 2024] S cpuhp/28 186 24467.327419 25 120 0.000000 0.931473 0.000000 0.000000 0 0 /
[Mon Mar 11 14:29:19 2024] S idle_inject/28 187 -12.000000 3 49 0.000000 0.000000 0.000000 0.000000 0 0 /
[Mon Mar 11 14:29:19 2024] S migration/28 188 1717.822791 359161 0 0.000000 1918.506629 0.000000 0.000000 0 0 /
[Mon Mar 11 14:29:19 2024] S ksoftirqd/28 189 35351472.574412 6496166 120 0.000000 166193.264532 0.000000 0.000000 0 0 /
[Mon Mar 11 14:29:19 2024] I kworker/28:0H 191 100.572124 4 100 0.000000 0.021484 0.000000 0.000000 0 0 /
[Mon Mar 11 14:29:19 2024] I kthrotld 388 60.224546 2 100 0.000000 0.015372 0.000000 0.000000 0 0 /
[Mon Mar 11 14:29:19 2024] I kworker/28:1H 797 35351479.743890 1473752 100 0.000000 48966.040525 0.000000 0.000000 0 0 /
[Mon Mar 11 14:29:19 2024] I kmpathd 3346 25437.931795 2 100 0.000000 0.063045 0.000000 0.000000 0 0 /
[Mon Mar 11 14:29:19 2024] S jbd2/dm-85-8 4240 35218792.671883 1269 120 0.000000 98.149423 0.000000 0.000000 0 0 /
[Mon Mar 11 14:29:19 2024] D nfsd 8783 35351485.974431 68107 120 0.000000 8304.030129 0.000000 0.000000 0 0 /
[Mon Mar 11 14:29:19 2024] D nfsd 8800 35351491.742307 100784 120 0.000000 11085.091832 0.000000 0.000000 0 0 /
[Mon Mar 11 14:29:19 2024] D nfsd 8811 35351488.913213 188046 120 0.000000 18087.651476 0.000000 0.000000 0 0 /
[Mon Mar 11 14:29:19 2024] D nfsd 8819 35351098.469205 1475231 120 0.000000 80038.218149 0.000000 0.000000 0 0 /
[Mon Mar 11 14:29:19 2024] D nfsd 8821 35346689.963250 1355027 120 0.000000 77645.905253 0.000000 0.000000 0 0 /
[Mon Mar 11 14:29:19 2024] D nfsd 8872 35264053.263806 1371523107 120 0.000000 33490756.107842 0.000000 0.000000 0 0 /
[Mon Mar 11 14:29:19 2024] S gmain 21805 11.207902 2 120 0.000000 0.256487 0.000000 0.000000 0 0 /autogroup-204
[Mon Mar 11 14:29:19 2024] S automount 88149 11.861608 1 120 0.000000 0.910191 0.000000 0.000000 0 0 /autogroup-222
[Mon Mar 11 14:29:19 2024] S smbd[10.87.27.3 2193788 2287000.001916 1035 120 0.000000 279.775300 0.000000 0.000000 0 0 /autogroup-113
[Mon Mar 11 14:29:19 2024] S smbd[10.87.29.8 2346897 2286985.682236 620 120 0.000000 191.621440 0.000000 0.000000 0 0 /autogroup-113
[Mon Mar 11 14:29:19 2024] S smbd[10.87.27.1 2347638 2287106.006442 13988 120 0.000000 5904.163319 0.000000 0.000000 0 0 /autogroup-113
[Mon Mar 11 14:29:19 2024] S smbd[10.86.26.7 2366328 2287088.692190 368 120 0.000000 105.971223 0.000000 0.000000 0 0 /autogroup-113
[Mon Mar 11 14:29:19 2024] I kworker/28:0 2409584 35351479.876784 217220 120 0.000000 11690.359957 0.000000 0.000000 0 0 /
[Mon Mar 11 14:29:19 2024] S smbd[10.87.27.2 2440138 2286999.860277 130 120 0.000000 53.316142 0.000000 0.000000 0 0 /autogroup-113
[Mon Mar 11 14:29:19 2024] I kworker/28:2 2474525 35348048.447079 8 120 0.000000 0.052003 0.000000 0.000000 0 0 /
[Mon Mar 11 14:29:19 2024] I kworker/28:1 2477534 35348532.167907 4 120 0.000000 0.069155 0.000000 0.000000 0 0 /
[Mon Mar 11 14:29:19 2024] cpu#29, 2600.000 MHz
[Mon Mar 11 14:29:19 2024] .nr_running : 0
[Mon Mar 11 14:29:19 2024] .nr_switches : 973699126
[Mon Mar 11 14:29:19 2024] .nr_uninterruptible : -8284
[Mon Mar 11 14:29:19 2024] .next_balance : 5577.334875
[Mon Mar 11 14:29:19 2024] .curr->pid : 0
[Mon Mar 11 14:29:19 2024] .clock : 1282633452.160657
[Mon Mar 11 14:29:19 2024] .clock_task : 1281130905.050549
[Mon Mar 11 14:29:19 2024] .avg_idle : 1000000
[Mon Mar 11 14:29:19 2024] .max_idle_balance_cost : 500000
[Mon Mar 11 14:29:19 2024] rt_rq[29]:
[Mon Mar 11 14:29:19 2024] .rt_nr_running : 0
[Mon Mar 11 14:29:19 2024] .rt_nr_migratory : 0
[Mon Mar 11 14:29:19 2024] .rt_throttled : 0
[Mon Mar 11 14:29:19 2024] .rt_time : 0.000000
[Mon Mar 11 14:29:19 2024] .rt_runtime : 950.000000
[Mon Mar 11 14:29:19 2024] dl_rq[29]:
[Mon Mar 11 14:29:19 2024] .dl_nr_running : 0
[Mon Mar 11 14:29:19 2024] .dl_nr_migratory : 0
[Mon Mar 11 14:29:19 2024] .dl_bw->bw : 996147
[Mon Mar 11 14:29:19 2024] .dl_bw->total_bw : 0
[Mon Mar 11 14:29:19 2024] runnable tasks:
[Mon Mar 11 14:29:19 2024] S task PID tree-key switches prio wait-time sum-exec sum-sleep
[Mon Mar 11 14:29:19 2024] -------------------------------------------------------------------------------------------------------------
[Mon Mar 11 14:29:19 2024] S cpuhp/29 192 27336.035836 25 120 0.000000 0.640881 0.000000 0.000000 1 0 /
[Mon Mar 11 14:29:19 2024] S idle_inject/29 193 -12.000000 3 49 0.000000 0.000000 0.000000 0.000000 1 0 /
[Mon Mar 11 14:29:19 2024] S migration/29 194 1785.822790 323910 0 0.000000 1933.488315 0.000000 0.000000 1 0 /
[Mon Mar 11 14:29:19 2024] S ksoftirqd/29 195 19780704.777553 2053919 120 0.000000 66844.935710 0.000000 0.000000 1 0 /
[Mon Mar 11 14:29:19 2024] I kworker/29:0H 197 644.947209 4 100 0.000000 0.014916 0.000000 0.000000 1 0 /
[Mon Mar 11 14:29:19 2024] I kworker/29:1H 877 19781749.635358 833808 100 0.000000 28560.002417 0.000000 0.000000 1 0 /
[Mon Mar 11 14:29:19 2024] I nvme-reset-wq 880 706.132822 2 100 0.000000 0.006370 0.000000 0.000000 1 0 /
[Mon Mar 11 14:29:19 2024] S scsi_eh_3 902 2343.549794 25 120 0.000000 3.806096 0.000000 0.000000 1 0 /
[Mon Mar 11 14:29:19 2024] I scsi_tmf_3 903 1651.005638 2 100 0.000000 0.014333 0.000000 0.000000 1 0 /
[Mon Mar 11 14:29:19 2024] I scsi_tmf_4 905 1675.023417 2 100 0.000000 0.011040 0.000000 0.000000 1 0 /
[Mon Mar 11 14:29:19 2024] I scsi_tmf_5 907 1687.033632 2 100 0.000000 0.012448 0.000000 0.000000 1 0 /
[Mon Mar 11 14:29:19 2024] I kdmflush/253:18 3534 27900.784849 2 100 0.000000 0.014036 0.000000 0.000000 1 0 /
[Mon Mar 11 14:29:19 2024] I kdmflush/253:27 3612 27948.608761 2 100 0.000000 0.014517 0.000000 0.000000 1 0 /
[Mon Mar 11 14:29:19 2024] I ext4-rsv-conver 4265 28332.975251 2 100 0.000000 0.016037 0.000000 0.000000 1 0 /
[Mon Mar 11 14:29:19 2024] S avahi-daemon 4321 11.468531 2 120 0.000000 0.517116 0.000000 0.000000 1 0 /autogroup-70
[Mon Mar 11 14:29:19 2024] S sssd_autofs 4358 137116.608098 264924 120 0.000000 18254.029467 0.000000 0.000000 1 0 /autogroup-78
[Mon Mar 11 14:29:19 2024] I dm-thin 6395 28875.814440 2 100 0.000000 0.063019 0.000000 0.000000 1 0 /
[Mon Mar 11 14:29:19 2024] S dmeventd 6410 2145.772838 144146 102 0.000000 22017.259595 0.000000 0.000000 1 6410 /autogroup-125
[Mon Mar 11 14:29:19 2024] D nfsd 8793 19781761.546732 81524 120 0.000000 9286.620505 0.000000 0.000000 1 0 /
[Mon Mar 11 14:29:19 2024] D nfsd 8836 19766206.005765 4192253 120 0.000000 241849.341168 0.000000 0.000000 1 0 /
[Mon Mar 11 14:29:19 2024] S gdbus 21426 25.154925 161 120 0.000000 37.257953 0.000000 0.000000 1 21426 /autogroup-154
[Mon Mar 11 14:29:19 2024] S automount 88139 24.640968 1 120 0.000000 0.787324 0.000000 0.000000 1 0 /autogroup-222
[Mon Mar 11 14:29:19 2024] S smbd[10.87.18.7 2428733 1325662.905937 387 120 0.000000 177.665898 0.000000 0.000000 1 0 /autogroup-113
[Mon Mar 11 14:29:19 2024] I kworker/29:0 2436183 19781749.645167 61082 120 0.000000 3797.869688 0.000000 0.000000 1 0 /
[Mon Mar 11 14:29:19 2024] S smbd[10.40.140. 2453653 1325650.602636 96 120 0.000000 51.466211 0.000000 0.000000 1 0 /autogroup-113
[Mon Mar 11 14:29:19 2024] I kworker/29:2 2474780 19775520.754109 3 120 0.000000 0.018825 0.000000 0.000000 1 0 /
[Mon Mar 11 14:29:19 2024] cpu#30, 2600.000 MHz
[Mon Mar 11 14:29:19 2024] .nr_running : 0
[Mon Mar 11 14:29:19 2024] .nr_switches : 3769795270
[Mon Mar 11 14:29:19 2024] .nr_uninterruptible : 39193
[Mon Mar 11 14:29:19 2024] .next_balance : 5577.334875
[Mon Mar 11 14:29:19 2024] .curr->pid : 0
[Mon Mar 11 14:29:19 2024] .clock : 1282633452.161040
[Mon Mar 11 14:29:19 2024] .clock_task : 1252314208.909423
[Mon Mar 11 14:29:19 2024] .avg_idle : 1000000
[Mon Mar 11 14:29:19 2024] .max_idle_balance_cost : 500000
[Mon Mar 11 14:29:19 2024] cfs_rq[30]:/autogroup-113
[Mon Mar 11 14:29:19 2024] .exec_clock : 0.000000
[Mon Mar 11 14:29:19 2024] .MIN_vruntime : 0.000001
[Mon Mar 11 14:29:19 2024] .min_vruntime : 1827811.301253
[Mon Mar 11 14:29:19 2024] .max_vruntime : 0.000001
[Mon Mar 11 14:29:19 2024] .spread : 0.000000
[Mon Mar 11 14:29:19 2024] .spread0 : -18589855.206862
[Mon Mar 11 14:29:19 2024] .nr_spread_over : 0
[Mon Mar 11 14:29:19 2024] .nr_running : 0
[Mon Mar 11 14:29:19 2024] .h_nr_running : 0
[Mon Mar 11 14:29:19 2024] .idle_nr_running : 0
[Mon Mar 11 14:29:19 2024] .idle_h_nr_running : 0
[Mon Mar 11 14:29:19 2024] .load : 0
[Mon Mar 11 14:29:19 2024] .load_avg : 0
[Mon Mar 11 14:29:19 2024] .runnable_avg : 0
[Mon Mar 11 14:29:19 2024] .util_avg : 0
[Mon Mar 11 14:29:19 2024] .util_est_enqueued : 0
[Mon Mar 11 14:29:19 2024] .removed.load_avg : 0
[Mon Mar 11 14:29:19 2024] .removed.util_avg : 0
[Mon Mar 11 14:29:19 2024] .removed.runnable_avg : 0
[Mon Mar 11 14:29:19 2024] .tg_load_avg_contrib : 0
[Mon Mar 11 14:29:19 2024] .tg_load_avg : 102
[Mon Mar 11 14:29:19 2024] .throttled : 0
[Mon Mar 11 14:29:19 2024] .throttle_count : 0
[Mon Mar 11 14:29:19 2024] .se->exec_start : 1252314107.747249
[Mon Mar 11 14:29:19 2024] .se->vruntime : 54713088.078382
[Mon Mar 11 14:29:19 2024] .se->sum_exec_runtime : 1667204.892016
[Mon Mar 11 14:29:19 2024] .se->load.weight : 2
[Mon Mar 11 14:29:19 2024] .se->avg.load_avg : 0
[Mon Mar 11 14:29:19 2024] .se->avg.util_avg : 0
[Mon Mar 11 14:29:19 2024] .se->avg.runnable_avg : 0
[Mon Mar 11 14:29:19 2024] cfs_rq[30]:/
[Mon Mar 11 14:29:19 2024] .exec_clock : 0.000000
[Mon Mar 11 14:29:19 2024] .MIN_vruntime : 0.000001
[Mon Mar 11 14:29:19 2024] .min_vruntime : 54713099.649567
[Mon Mar 11 14:29:19 2024] .max_vruntime : 0.000001
[Mon Mar 11 14:29:19 2024] .spread : 0.000000
[Mon Mar 11 14:29:19 2024] .spread0 : 34295433.141452
[Mon Mar 11 14:29:19 2024] .nr_spread_over : 0
[Mon Mar 11 14:29:19 2024] .nr_running : 0
[Mon Mar 11 14:29:19 2024] .h_nr_running : 0
[Mon Mar 11 14:29:19 2024] .idle_nr_running : 0
[Mon Mar 11 14:29:19 2024] .idle_h_nr_running : 0
[Mon Mar 11 14:29:19 2024] .load : 0
[Mon Mar 11 14:29:19 2024] .load_avg : 0
[Mon Mar 11 14:29:19 2024] .runnable_avg : 0
[Mon Mar 11 14:29:19 2024] .util_avg : 0
[Mon Mar 11 14:29:19 2024] .util_est_enqueued : 0
[Mon Mar 11 14:29:19 2024] .removed.load_avg : 0
[Mon Mar 11 14:29:19 2024] .removed.util_avg : 0
[Mon Mar 11 14:29:19 2024] .removed.runnable_avg : 0
[Mon Mar 11 14:29:19 2024] .tg_load_avg_contrib : 0
[Mon Mar 11 14:29:19 2024] .tg_load_avg : 0
[Mon Mar 11 14:29:19 2024] .throttled : 0
[Mon Mar 11 14:29:19 2024] .throttle_count : 0
[Mon Mar 11 14:29:19 2024] rt_rq[30]:
[Mon Mar 11 14:29:19 2024] .rt_nr_running : 0
[Mon Mar 11 14:29:19 2024] .rt_nr_migratory : 0
[Mon Mar 11 14:29:19 2024] .rt_throttled : 0
[Mon Mar 11 14:29:19 2024] .rt_time : 0.000000
[Mon Mar 11 14:29:19 2024] .rt_runtime : 950.000000
[Mon Mar 11 14:29:19 2024] dl_rq[30]:
[Mon Mar 11 14:29:19 2024] .dl_nr_running : 0
[Mon Mar 11 14:29:19 2024] .dl_nr_migratory : 0
[Mon Mar 11 14:29:19 2024] .dl_bw->bw : 996147
[Mon Mar 11 14:29:19 2024] .dl_bw->total_bw : 0
[Mon Mar 11 14:29:19 2024] runnable tasks:
[Mon Mar 11 14:29:19 2024] S task PID tree-key switches prio wait-time sum-exec sum-sleep
[Mon Mar 11 14:29:19 2024] -------------------------------------------------------------------------------------------------------------
[Mon Mar 11 14:29:19 2024] S cpuhp/30 198 27707.242305 25 120 0.000000 0.883592 0.000000 0.000000 0 0 /
[Mon Mar 11 14:29:19 2024] S idle_inject/30 199 -12.000000 3 49 0.000000 0.000000 0.000000 0.000000 0 0 /
[Mon Mar 11 14:29:19 2024] S migration/30 200 1853.822765 375246 0 0.000000 1914.133523 0.000000 0.000000 0 0 /
[Mon Mar 11 14:29:19 2024] S ksoftirqd/30 201 54713067.203589 12832494 120 0.000000 296928.222491 0.000000 0.000000 0 0 /
[Mon Mar 11 14:29:19 2024] I kworker/30:0H 203 40.610924 4 100 0.000000 0.046357 0.000000 0.000000 0 0 /
[Mon Mar 11 14:29:19 2024] I kworker/30:1H 773 54713087.649879 2047597 100 0.000000 57970.744443 0.000000 0.000000 0 0 /
[Mon Mar 11 14:29:19 2024] S scsi_eh_11 922 2906.284561 25 120 0.000000 3.021279 0.000000 0.000000 0 0 /
[Mon Mar 11 14:29:19 2024] I scsi_wq_15 961 2916.915135 2 100 0.000000 0.006923 0.000000 0.000000 0 0 /
[Mon Mar 11 14:29:19 2024] I ipmi-msghandler 3119 9177.907899 2 100 0.000000 0.007451 0.000000 0.000000 0 0 /
[Mon Mar 11 14:29:19 2024] I kdmflush/253:26 3610 30094.425256 2 100 0.000000 0.015141 0.000000 0.000000 0 0 /
[Mon Mar 11 14:29:19 2024] I kdmflush/253:48 3806 30441.338818 2 100 0.000000 0.009726 0.000000 0.000000 0 0 /
[Mon Mar 11 14:29:19 2024] S ipmievd 4318 1279.475180 257116 120 0.000000 11973.922805 0.000000 0.000000 0 0 /autogroup-74
[Mon Mar 11 14:29:19 2024] D nfsd 8785 54713087.068428 72158 120 0.000000 8555.827205 0.000000 0.000000 0 0 /
[Mon Mar 11 14:29:19 2024] D nfsd 8803 54713084.529861 117561 120 0.000000 12174.146490 0.000000 0.000000 0 0 /
[Mon Mar 11 14:29:19 2024] D nfsd 8814 54713090.411096 326877 120 0.000000 26562.260384 0.000000 0.000000 0 0 /
[Mon Mar 11 14:29:19 2024] D nfsd 8815 54712918.239335 388488 120 0.000000 30889.130054 0.000000 0.000000 0 0 /
[Mon Mar 11 14:29:19 2024] D nfsd 8818 54712552.489522 1013118 120 0.000000 58538.292722 0.000000 0.000000 0 0 /
[Mon Mar 11 14:29:19 2024] D nfsd 8825 54696392.769622 2500570 120 0.000000 130849.129639 0.000000 0.000000 0 0 /
[Mon Mar 11 14:29:19 2024] D nfsd 8828 54600869.267487 249094 120 0.000000 19620.432999 0.000000 0.000000 0 0 /
[Mon Mar 11 14:29:19 2024] D nfsd 8831 54654008.177722 1588536 120 0.000000 95579.359611 0.000000 0.000000 0 0 /
[Mon Mar 11 14:29:19 2024] D nfsd 8861 54591761.355245 29213313 120 0.000000 1208432.697269 0.000000 0.000000 0 0 /
[Mon Mar 11 14:29:19 2024] D nfsd 8875 54569428.184409 7082610996 120 0.000000 175537379.985343 0.000000 0.000000 0 0 /
[Mon Mar 11 14:29:19 2024] S smbd[10.87.31.1 1059267 1827799.697405 5257 120 0.000000 1524.895754 0.000000 0.000000 0 0 /autogroup-113
[Mon Mar 11 14:29:19 2024] D kworker/30:0 2364267 54712989.727112 345513 120 0.000000 14561.467284 0.000000 0.000000 0 0 /
[Mon Mar 11 14:29:19 2024] I kworker/u97:1 2451949 54713099.649567 723816 120 0.000000 15800.529215 0.000000 0.000000 0 0 /
[Mon Mar 11 14:29:19 2024] I kworker/30:1 2473379 54688989.156641 10 120 0.000000 0.059539 0.000000 0.000000 0 0 /
[Mon Mar 11 14:29:19 2024] I kworker/30:2 2475882 54713087.662033 17 120 0.000000 0.125553 0.000000 0.000000 0 0 /
[Mon Mar 11 14:29:19 2024] cpu#31, 2600.000 MHz
[Mon Mar 11 14:29:19 2024] .nr_running : 0
[Mon Mar 11 14:29:19 2024] .nr_switches : 934247363
[Mon Mar 11 14:29:19 2024] .nr_uninterruptible : -5035
[Mon Mar 11 14:29:19 2024] .next_balance : 5577.334877
[Mon Mar 11 14:29:19 2024] .curr->pid : 0
[Mon Mar 11 14:29:19 2024] .clock : 1282633452.162406
[Mon Mar 11 14:29:19 2024] .clock_task : 1281176641.859853
[Mon Mar 11 14:29:19 2024] .avg_idle : 1000000
[Mon Mar 11 14:29:19 2024] .max_idle_balance_cost : 500000
[Mon Mar 11 14:29:19 2024] cfs_rq[31]:/
[Mon Mar 11 14:29:19 2024] .exec_clock : 0.000000
[Mon Mar 11 14:29:19 2024] .MIN_vruntime : 0.000001
[Mon Mar 11 14:29:19 2024] .min_vruntime : 18913167.665467
[Mon Mar 11 14:29:19 2024] .max_vruntime : 0.000001
[Mon Mar 11 14:29:19 2024] .spread : 0.000000
[Mon Mar 11 14:29:19 2024] .spread0 : -1504498.842648
[Mon Mar 11 14:29:19 2024] .nr_spread_over : 0
[Mon Mar 11 14:29:19 2024] .nr_running : 0
[Mon Mar 11 14:29:19 2024] .h_nr_running : 0
[Mon Mar 11 14:29:19 2024] .idle_nr_running : 0
[Mon Mar 11 14:29:19 2024] .idle_h_nr_running : 0
[Mon Mar 11 14:29:19 2024] .load : 0
[Mon Mar 11 14:29:19 2024] .load_avg : 0
[Mon Mar 11 14:29:19 2024] .runnable_avg : 0
[Mon Mar 11 14:29:19 2024] .util_avg : 0
[Mon Mar 11 14:29:19 2024] .util_est_enqueued : 0
[Mon Mar 11 14:29:19 2024] .removed.load_avg : 0
[Mon Mar 11 14:29:19 2024] .removed.util_avg : 0
[Mon Mar 11 14:29:19 2024] .removed.runnable_avg : 0
[Mon Mar 11 14:29:19 2024] .tg_load_avg_contrib : 0
[Mon Mar 11 14:29:19 2024] .tg_load_avg : 0
[Mon Mar 11 14:29:19 2024] .throttled : 0
[Mon Mar 11 14:29:19 2024] .throttle_count : 0
[Mon Mar 11 14:29:19 2024] rt_rq[31]:
[Mon Mar 11 14:29:19 2024] .rt_nr_running : 0
[Mon Mar 11 14:29:19 2024] .rt_nr_migratory : 0
[Mon Mar 11 14:29:19 2024] .rt_throttled : 0
[Mon Mar 11 14:29:19 2024] .rt_time : 0.000000
[Mon Mar 11 14:29:19 2024] .rt_runtime : 950.000000
[Mon Mar 11 14:29:19 2024] dl_rq[31]:
[Mon Mar 11 14:29:19 2024] .dl_nr_running : 0
[Mon Mar 11 14:29:19 2024] .dl_nr_migratory : 0
[Mon Mar 11 14:29:19 2024] .dl_bw->bw : 996147
[Mon Mar 11 14:29:19 2024] .dl_bw->total_bw : 0
[Mon Mar 11 14:29:19 2024] runnable tasks:
[Mon Mar 11 14:29:19 2024] S task PID tree-key switches prio wait-time sum-exec sum-sleep
[Mon Mar 11 14:29:19 2024] -------------------------------------------------------------------------------------------------------------
[Mon Mar 11 14:29:19 2024] S cpuhp/31 204 23183.738052 25 120 0.000000 1.984444 0.000000 0.000000 1 0 /
[Mon Mar 11 14:29:19 2024] S idle_inject/31 205 -11.994447 3 49 0.000000 0.233381 0.000000 0.000000 1 0 /
[Mon Mar 11 14:29:19 2024] S migration/31 207 1918.044378 324029 0 0.000000 1921.749222 0.000000 0.000000 1 0 /
[Mon Mar 11 14:29:19 2024] S ksoftirqd/31 208 18912382.515114 1955950 120 0.000000 63903.878297 0.000000 0.000000 1 0 /
[Mon Mar 11 14:29:19 2024] I kworker/31:0H 210 67.506584 4 100 0.000000 0.034039 0.000000 0.000000 1 0 /
[Mon Mar 11 14:29:19 2024] I kworker/31:1H 803 18912464.158320 782302 100 0.000000 27164.626072 0.000000 0.000000 1 0 /
[Mon Mar 11 14:29:19 2024] I kdmflush/253:87 4166 23293.393835 2 100 0.000000 0.017439 0.000000 0.000000 1 0 /
[Mon Mar 11 14:29:19 2024] S f2b/observer 2470448 23278.646362 1 120 0.000000 0.168234 0.000000 0.000000 1 0 /autogroup-96
[Mon Mar 11 14:29:19 2024] S gdbus 5165 1933.257011 311267 120 0.000000 45341.992410 0.000000 0.000000 1 4667 /autogroup-104
[Mon Mar 11 14:29:19 2024] S JS Helper 5180 11.205943 10 120 0.000000 0.254526 0.000000 0.000000 1 0 /autogroup-104
[Mon Mar 11 14:29:19 2024] S uwsgi 5236 3425.517111 226 120 0.000000 185.220898 0.000000 0.000000 1 0 /autogroup-90
[Mon Mar 11 14:29:19 2024] D nfsd 8792 18913162.419960 84339 120 0.000000 9440.605656 0.000000 0.000000 1 0 /
[Mon Mar 11 14:29:19 2024] D nfsd 8813 18913162.628026 261677 120 0.000000 23211.393110 0.000000 0.000000 1 0 /
[Mon Mar 11 14:29:19 2024] I kworker/31:2 2436168 18913154.991206 61335 120 0.000000 3433.530478 0.000000 0.000000 1 0 /
[Mon Mar 11 14:29:19 2024] S smbd[10.87.28.1 2441978 1216497.375004 220 120 0.000000 92.346038 0.000000 0.000000 1 0 /autogroup-113
[Mon Mar 11 14:29:19 2024] S smbd[10.87.28.1 2457400 1216472.702692 111 120 0.000000 60.372911 0.000000 0.000000 1 0 /autogroup-113
[Mon Mar 11 14:29:19 2024] I kworker/u96:5 2466033 18913167.665467 85607 120 0.000000 32767.313757 0.000000 0.000000 1 0 /
[Mon Mar 11 14:29:19 2024] I kworker/31:0 2474534 18890962.520235 3 120 0.000000 0.029093 0.000000 0.000000 1 0 /
[Mon Mar 11 14:29:19 2024] cpu#32, 2600.000 MHz
[Mon Mar 11 14:29:19 2024] .nr_running : 0
[Mon Mar 11 14:29:19 2024] .nr_switches : 3058688882
[Mon Mar 11 14:29:19 2024] .nr_uninterruptible : 15636
[Mon Mar 11 14:29:19 2024] .next_balance : 5577.334876
[Mon Mar 11 14:29:19 2024] .curr->pid : 0
[Mon Mar 11 14:29:19 2024] .clock : 1282633453.161490
[Mon Mar 11 14:29:19 2024] .clock_task : 1260420965.487018
[Mon Mar 11 14:29:19 2024] .avg_idle : 1000000
[Mon Mar 11 14:29:19 2024] .max_idle_balance_cost : 500000
[Mon Mar 11 14:29:19 2024] rt_rq[32]:
[Mon Mar 11 14:29:19 2024] .rt_nr_running : 0
[Mon Mar 11 14:29:19 2024] .rt_nr_migratory : 0
[Mon Mar 11 14:29:19 2024] .rt_throttled : 0
[Mon Mar 11 14:29:19 2024] .rt_time : 0.000000
[Mon Mar 11 14:29:19 2024] .rt_runtime : 950.000000
[Mon Mar 11 14:29:19 2024] dl_rq[32]:
[Mon Mar 11 14:29:19 2024] .dl_nr_running : 0
[Mon Mar 11 14:29:19 2024] .dl_nr_migratory : 0
[Mon Mar 11 14:29:19 2024] .dl_bw->bw : 996147
[Mon Mar 11 14:29:19 2024] .dl_bw->total_bw : 0
[Mon Mar 11 14:29:19 2024] runnable tasks:
[Mon Mar 11 14:29:19 2024] S task PID tree-key switches prio wait-time sum-exec sum-sleep
[Mon Mar 11 14:29:19 2024] -------------------------------------------------------------------------------------------------------------
[Mon Mar 11 14:29:19 2024] S cpuhp/32 211 24602.849659 25 120 0.000000 0.671807 0.000000 0.000000 0 0 /
[Mon Mar 11 14:29:19 2024] S idle_inject/32 212 -12.000000 3 49 0.000000 0.000000 0.000000 0.000000 0 0 /
[Mon Mar 11 14:29:19 2024] S migration/32 213 1986.044354 380976 0 0.000000 2045.251086 0.000000 0.000000 0 0 /
[Mon Mar 11 14:29:19 2024] S ksoftirqd/32 214 45066090.409469 10342211 120 0.000000 234618.394814 0.000000 0.000000 0 0 /
[Mon Mar 11 14:29:19 2024] I kworker/32:0H 216 39.602572 4 100 0.000000 0.013003 0.000000 0.000000 0 0 /
[Mon Mar 11 14:29:19 2024] I writeback 363 13.400695 2 100 0.000000 0.000000 0.000000 0.000000 0 0 /
[Mon Mar 11 14:29:19 2024] I tpm_dev_wq 372 13.400695 2 100 0.000000 0.000000 0.000000 0.000000 0 0 /
[Mon Mar 11 14:29:19 2024] I kworker/32:1H 406 45066114.141864 1761136 100 0.000000 52233.979953 0.000000 0.000000 0 0 /
[Mon Mar 11 14:29:19 2024] I poll_megasas0_s 931 2038.309746 2 100 0.000000 0.027107 0.000000 0.000000 0 0 /
[Mon Mar 11 14:29:19 2024] I kdmflush/253:51 3837 25593.905418 2 100 0.000000 0.018228 0.000000 0.000000 0 0 /
[Mon Mar 11 14:29:19 2024] I kdmflush/253:89 4172 25787.071043 2 100 0.000000 0.016299 0.000000 0.000000 0 0 /
[Mon Mar 11 14:29:19 2024] S jbd2/dm-83-8 4350 45066114.349151 303135 120 0.000000 18348.000104 0.000000 0.000000 0 0 /
[Mon Mar 11 14:29:19 2024] D nfsd 8782 45066119.378656 67460 120 0.000000 8251.485627 0.000000 0.000000 0 0 /
[Mon Mar 11 14:29:19 2024] D nfsd 8810 45066126.141449 164146 120 0.000000 16587.517234 0.000000 0.000000 0 0 /
[Mon Mar 11 14:29:19 2024] D nfsd 8847 44984526.578282 9036579 120 0.000000 357315.796423 0.000000 0.000000 0 0 /
[Mon Mar 11 14:29:19 2024] S automount 88127 23.939798 1 120 0.000000 0.882379 0.000000 0.000000 0 0 /autogroup-222
[Mon Mar 11 14:29:19 2024] S automount 88159 62.611349 1 120 0.000000 0.850809 0.000000 0.000000 0 0 /autogroup-222
[Mon Mar 11 14:29:19 2024] I kworker/32:0 2344368 45066114.208484 378163 120 0.000000 18597.235107 0.000000 0.000000 0 0 /
[Mon Mar 11 14:29:19 2024] S smbd[10.87.27.1 2344564 2011560.787408 477 120 0.000000 148.359065 0.000000 0.000000 0 0 /autogroup-113
[Mon Mar 11 14:29:19 2024] S smbd[10.87.18.2 2347077 2011637.083704 909 120 0.000000 307.074769 0.000000 0.000000 0 0 /autogroup-113
[Mon Mar 11 14:29:19 2024] S smbd[10.87.29.9 2395603 2011560.760869 310 120 0.000000 98.868814 0.000000 0.000000 0 0 /autogroup-113
[Mon Mar 11 14:29:19 2024] I kworker/32:1 2470837 45050189.265983 21 120 0.000000 0.076223 0.000000 0.000000 0 0 /
[Mon Mar 11 14:29:19 2024] I kworker/32:2 2475859 45059554.686693 21 120 0.000000 0.199284 0.000000 0.000000 0 0 /
[Mon Mar 11 14:29:19 2024] cpu#33, 2600.000 MHz
[Mon Mar 11 14:29:19 2024] .nr_running : 0
[Mon Mar 11 14:29:19 2024] .nr_switches : 911324112
[Mon Mar 11 14:29:19 2024] .nr_uninterruptible : -3649
[Mon Mar 11 14:29:19 2024] .next_balance : 5577.334877
[Mon Mar 11 14:29:19 2024] .curr->pid : 0
[Mon Mar 11 14:29:19 2024] .clock : 1282633453.160511
[Mon Mar 11 14:29:19 2024] .clock_task : 1281205568.591957
[Mon Mar 11 14:29:19 2024] .avg_idle : 1000000
[Mon Mar 11 14:29:19 2024] .max_idle_balance_cost : 500000
[Mon Mar 11 14:29:19 2024] rt_rq[33]:
[Mon Mar 11 14:29:19 2024] .rt_nr_running : 0
[Mon Mar 11 14:29:19 2024] .rt_nr_migratory : 0
[Mon Mar 11 14:29:19 2024] .rt_throttled : 0
[Mon Mar 11 14:29:19 2024] .rt_time : 0.000000
[Mon Mar 11 14:29:19 2024] .rt_runtime : 950.000000
[Mon Mar 11 14:29:19 2024] dl_rq[33]:
[Mon Mar 11 14:29:19 2024] .dl_nr_running : 0
[Mon Mar 11 14:29:19 2024] .dl_nr_migratory : 0
[Mon Mar 11 14:29:19 2024] .dl_bw->bw : 996147
[Mon Mar 11 14:29:19 2024] .dl_bw->total_bw : 0
[Mon Mar 11 14:29:19 2024] runnable tasks:
[Mon Mar 11 14:29:19 2024] S task PID tree-key switches prio wait-time sum-exec sum-sleep
[Mon Mar 11 14:29:19 2024] -------------------------------------------------------------------------------------------------------------
[Mon Mar 11 14:29:19 2024] S cpuhp/33 217 21359.466636 25 120 0.000000 0.676036 0.000000 0.000000 1 0 /
[Mon Mar 11 14:29:19 2024] S idle_inject/33 218 -12.000000 3 49 0.000000 0.000000 0.000000 0.000000 1 0 /
[Mon Mar 11 14:29:19 2024] S migration/33 219 2054.049488 323947 0 0.000000 1940.884327 0.000000 0.000000 1 0 /
[Mon Mar 11 14:29:19 2024] S ksoftirqd/33 220 18522501.468156 1881932 120 0.000000 61610.801756 0.000000 0.000000 1 0 /
[Mon Mar 11 14:29:19 2024] I kworker/33:0H 222 690.274881 4 100 0.000000 0.012629 0.000000 0.000000 1 0 /
[Mon Mar 11 14:29:19 2024] S kauditd 356 18522609.129157 349285 120 0.000000 4925.355065 0.000000 0.000000 1 0 /
[Mon Mar 11 14:29:19 2024] I kworker/33:1H 874 18522609.519504 805238 100 0.000000 27183.307839 0.000000 0.000000 1 0 /
[Mon Mar 11 14:29:19 2024] S writer#3 5331 425961.043751 55300298 120 0.000000 1456522.760828 0.000000 0.000000 1 5327 /autogroup-105
[Mon Mar 11 14:29:19 2024] S jbd2/dm-98-8 6487 18413323.890250 539 120 0.000000 28.676716 0.000000 0.000000 1 0 /
[Mon Mar 11 14:29:19 2024] I kdmflush/253:10 8719 23285.211605 2 100 0.000000 0.007915 0.000000 0.000000 1 0 /
[Mon Mar 11 14:29:19 2024] D nfsd 8844 18469903.898437 7148783 120 0.000000 214334.022179 0.000000 0.000000 1 0 /
[Mon Mar 11 14:29:19 2024] D nfsd 8852 18470784.612419 9758338 120 0.000000 439552.047941 0.000000 0.000000 1 0 /
[Mon Mar 11 14:29:19 2024] D nfsd 8858 18463784.126770 17912269 120 0.000000 783911.070444 0.000000 0.000000 1 0 /
[Mon Mar 11 14:29:19 2024] S smbd[10.87.27.2 2370451 1105561.267752 379 120 0.000000 125.485170 0.000000 0.000000 1 0 /autogroup-113
[Mon Mar 11 14:29:19 2024] S smbd[10.87.28.6 2419992 1105560.908876 405 120 0.000000 107.696804 0.000000 0.000000 1 0 /autogroup-113
[Mon Mar 11 14:29:19 2024] I kworker/33:1 2434371 18522620.201079 70707 120 0.000000 4725.174632 0.000000 0.000000 1 0 /
[Mon Mar 11 14:29:19 2024] I kworker/u98:10 2463433 18522630.962612 224485 120 0.000000 4782.863496 0.000000 0.000000 1 0 /
[Mon Mar 11 14:29:19 2024] I kworker/33:2 2474557 18519125.341171 10 120 0.000000 0.054772 0.000000 0.000000 1 0 /
[Mon Mar 11 14:29:19 2024] I kworker/33:0 2477675 18519137.328252 2 120 0.000000 0.012966 0.000000 0.000000 1 0 /
[Mon Mar 11 14:29:19 2024] cpu#34, 2600.000 MHz
[Mon Mar 11 14:29:19 2024] .nr_running : 0
[Mon Mar 11 14:29:19 2024] .nr_switches : 2857202473
[Mon Mar 11 14:29:19 2024] .nr_uninterruptible : 26823
[Mon Mar 11 14:29:19 2024] .next_balance : 5577.334878
[Mon Mar 11 14:29:19 2024] .curr->pid : 0
[Mon Mar 11 14:29:19 2024] .clock : 1282633453.163247
[Mon Mar 11 14:29:19 2024] .clock_task : 1262053899.391721
[Mon Mar 11 14:29:19 2024] .avg_idle : 1000000
[Mon Mar 11 14:29:19 2024] .max_idle_balance_cost : 500000
[Mon Mar 11 14:29:19 2024] rt_rq[34]:
[Mon Mar 11 14:29:19 2024] .rt_nr_running : 0
[Mon Mar 11 14:29:19 2024] .rt_nr_migratory : 0
[Mon Mar 11 14:29:19 2024] .rt_throttled : 0
[Mon Mar 11 14:29:19 2024] .rt_time : 0.000000
[Mon Mar 11 14:29:19 2024] .rt_runtime : 950.000000
[Mon Mar 11 14:29:19 2024] dl_rq[34]:
[Mon Mar 11 14:29:19 2024] .dl_nr_running : 0
[Mon Mar 11 14:29:19 2024] .dl_nr_migratory : 0
[Mon Mar 11 14:29:19 2024] .dl_bw->bw : 996147
[Mon Mar 11 14:29:19 2024] .dl_bw->total_bw : 0
[Mon Mar 11 14:29:19 2024] runnable tasks:
[Mon Mar 11 14:29:19 2024] S task PID tree-key switches prio wait-time sum-exec sum-sleep
[Mon Mar 11 14:29:19 2024] -------------------------------------------------------------------------------------------------------------
[Mon Mar 11 14:29:19 2024] S cpuhp/34 223 22765.629809 25 120 0.000000 0.814599 0.000000 0.000000 0 0 /
[Mon Mar 11 14:29:19 2024] S idle_inject/34 224 -12.000000 3 49 0.000000 0.000000 0.000000 0.000000 0 0 /
[Mon Mar 11 14:29:19 2024] S migration/34 225 2122.049463 368590 0 0.000000 1978.759386 0.000000 0.000000 0 0 /
[Mon Mar 11 14:29:19 2024] S ksoftirqd/34 226 42202131.591786 9172509 120 0.000000 219827.863105 0.000000 0.000000 0 0 /
[Mon Mar 11 14:29:19 2024] I kworker/34:0H 228 76.825262 4 100 0.000000 0.031952 0.000000 0.000000 0 0 /
[Mon Mar 11 14:29:19 2024] I acpi_thermal_pm 400 76.534164 2 100 0.000000 0.017320 0.000000 0.000000 0 0 /
[Mon Mar 11 14:29:19 2024] I kworker/34:1H 745 42202235.358631 1655644 100 0.000000 49251.726479 0.000000 0.000000 0 0 /
[Mon Mar 11 14:29:19 2024] I kdmflush/253:44 3770 23966.769205 2 100 0.000000 0.017563 0.000000 0.000000 0 0 /
[Mon Mar 11 14:29:19 2024] S jbd2/dm-79-8 4270 42202235.717902 999072 120 0.000000 52365.683002 0.000000 0.000000 0 0 /
[Mon Mar 11 14:29:19 2024] S jbd2/dm-67-8 4276 42202235.528436 523000 120 0.000000 22523.730329 0.000000 0.000000 0 0 /
[Mon Mar 11 14:29:19 2024] S NetworkManager 4349 2686.665908 486205 120 0.000000 65653.587121 0.000000 0.000000 0 4359 /autogroup-82
[Mon Mar 11 14:29:19 2024] S uwsgi 5238 664.290160 391 120 0.000000 492.952488 0.000000 0.000000 0 0 /autogroup-90
[Mon Mar 11 14:29:19 2024] I kworker/34:0 2078485 42202235.384938 931800 120 0.000000 30858.925314 0.000000 0.000000 0 0 /
[Mon Mar 11 14:29:19 2024] S smbd[10.87.29.5 2331098 2229876.546800 517 120 0.000000 145.288568 0.000000 0.000000 0 0 /autogroup-113
[Mon Mar 11 14:29:19 2024] S samba-bgqd 2339818 19243.478080 10803 120 0.000000 16447.852718 0.000000 0.000000 0 0 /autogroup-22820
[Mon Mar 11 14:29:19 2024] S smbd[10.33.22.1 2349497 2229666.987357 486 120 0.000000 154.500876 0.000000 0.000000 0 0 /autogroup-113
[Mon Mar 11 14:29:19 2024] S smbd[10.87.31.1 2396394 2229558.831884 305 120 0.000000 102.112379 0.000000 0.000000 0 0 /autogroup-113
[Mon Mar 11 14:29:19 2024] S smbd[10.40.234. 2404568 2229912.682394 3599 120 0.000000 1319.976481 0.000000 0.000000 0 0 /autogroup-113
[Mon Mar 11 14:29:19 2024] I kworker/34:2 2475317 42197051.969339 18 120 0.000000 0.087815 0.000000 0.000000 0 0 /
[Mon Mar 11 14:29:19 2024] cpu#35, 2600.000 MHz
[Mon Mar 11 14:29:19 2024] .nr_running : 0
[Mon Mar 11 14:29:19 2024] .nr_switches : 874229258
[Mon Mar 11 14:29:19 2024] .nr_uninterruptible : -2501
[Mon Mar 11 14:29:19 2024] .next_balance : 5577.334875
[Mon Mar 11 14:29:19 2024] .curr->pid : 0
[Mon Mar 11 14:29:19 2024] .clock : 1282633453.158114
[Mon Mar 11 14:29:19 2024] .clock_task : 1281231960.582846
[Mon Mar 11 14:29:19 2024] .avg_idle : 1000000
[Mon Mar 11 14:29:19 2024] .max_idle_balance_cost : 500000
[Mon Mar 11 14:29:19 2024] rt_rq[35]:
[Mon Mar 11 14:29:19 2024] .rt_nr_running : 0
[Mon Mar 11 14:29:19 2024] .rt_nr_migratory : 0
[Mon Mar 11 14:29:19 2024] .rt_throttled : 0
[Mon Mar 11 14:29:19 2024] .rt_time : 0.000000
[Mon Mar 11 14:29:19 2024] .rt_runtime : 950.000000
[Mon Mar 11 14:29:19 2024] dl_rq[35]:
[Mon Mar 11 14:29:19 2024] .dl_nr_running : 0
[Mon Mar 11 14:29:19 2024] .dl_nr_migratory : 0
[Mon Mar 11 14:29:19 2024] .dl_bw->bw : 996147
[Mon Mar 11 14:29:19 2024] .dl_bw->total_bw : 0
[Mon Mar 11 14:29:19 2024] runnable tasks:
[Mon Mar 11 14:29:19 2024] S task PID tree-key switches prio wait-time sum-exec sum-sleep
[Mon Mar 11 14:29:19 2024] -------------------------------------------------------------------------------------------------------------
[Mon Mar 11 14:29:19 2024] S cpuhp/35 229 21160.012116 25 120 0.000000 0.792289 0.000000 0.000000 1 0 /
[Mon Mar 11 14:29:19 2024] S idle_inject/35 230 -12.000000 3 49 0.000000 0.001287 0.000000 0.000000 1 0 /
[Mon Mar 11 14:29:19 2024] S migration/35 231 0.000000 323614 0 0.000000 1919.440054 0.000000 0.000000 1 0 /
[Mon Mar 11 14:29:19 2024] S ksoftirqd/35 232 17769333.195118 1765668 120 0.000000 58048.002825 0.000000 0.000000 1 0 /
[Mon Mar 11 14:29:19 2024] I kworker/35:0H 234 46.667683 4 100 0.000000 0.014217 0.000000 0.000000 1 0 /
[Mon Mar 11 14:29:19 2024] I kworker/35:1H 620 17769643.424640 773910 100 0.000000 26580.944892 0.000000 0.000000 1 0 /
[Mon Mar 11 14:29:19 2024] I nvme-delete-wq 881 921.768551 2 100 0.000000 0.007748 0.000000 0.000000 1 0 /
[Mon Mar 11 14:29:19 2024] I scsi_tmf_12 925 1771.749158 2 100 0.000000 0.012430 0.000000 0.000000 1 0 /
[Mon Mar 11 14:29:19 2024] I scsi_tmf_14 929 1795.767151 2 100 0.000000 0.010589 0.000000 0.000000 1 0 /
[Mon Mar 11 14:29:19 2024] I kdmflush/253:79 4153 21813.120382 2 100 0.000000 0.019452 0.000000 0.000000 1 0 /
[Mon Mar 11 14:29:19 2024] S gssproxy 4420 1727.019608 38893 120 0.000000 2837.125562 0.000000 0.000000 1 0 /autogroup-98
[Mon Mar 11 14:29:19 2024] S gssproxy 4426 11.009374 2 120 0.000000 0.057959 0.000000 0.000000 1 0 /autogroup-98
[Mon Mar 11 14:29:19 2024] S gssproxy 4427 23.045582 1 120 0.000000 0.036217 0.000000 0.000000 1 0 /autogroup-98
[Mon Mar 11 14:29:19 2024] S gssproxy 4428 35.079440 1 120 0.000000 0.033867 0.000000 0.000000 1 0 /autogroup-98
[Mon Mar 11 14:29:19 2024] S JS Helper 5181 -5.349679 11 120 0.000000 0.362890 0.000000 0.000000 1 0 /autogroup-104
[Mon Mar 11 14:29:19 2024] D nfsd 8824 17769624.287672 3701991 120 0.000000 191960.796469 0.000000 0.000000 1 0 /
[Mon Mar 11 14:29:19 2024] D nfsd 8865 17694268.332316 68392850 120 0.000000 2480606.148617 0.000000 0.000000 1 0 /
[Mon Mar 11 14:29:19 2024] S smbd[10.87.20.2 3797717 1018863.671099 18954 120 0.000000 7691.558275 0.000000 0.000000 1 0 /autogroup-113
[Mon Mar 11 14:29:19 2024] I kworker/35:0 2344315 17769643.464374 138933 120 0.000000 6886.987572 0.000000 0.000000 1 0 /
[Mon Mar 11 14:29:19 2024] S smbd[10.40.131. 2365796 1018880.141597 391 120 0.000000 128.296416 0.000000 0.000000 1 0 /autogroup-113
[Mon Mar 11 14:29:19 2024] S smbd[10.87.29.9 2458024 1018863.915275 76 120 0.000000 40.091002 0.000000 0.000000 1 0 /autogroup-113
[Mon Mar 11 14:29:19 2024] I kworker/35:1 2475663 17767524.203193 8 120 0.000000 0.032187 0.000000 0.000000 1 0 /
[Mon Mar 11 14:29:19 2024] S rpcd_spoolss 2477388 202938.028752 1626 120 0.000000 520.201286 0.000000 0.000000 1 0 /autogroup-22820
[Mon Mar 11 14:29:19 2024] cpu#36, 2600.000 MHz
[Mon Mar 11 14:29:19 2024] .nr_running : 0
[Mon Mar 11 14:29:19 2024] .nr_switches : 2694265997
[Mon Mar 11 14:29:19 2024] .nr_uninterruptible : 26995
[Mon Mar 11 14:29:19 2024] .next_balance : 5577.334877
[Mon Mar 11 14:29:19 2024] .curr->pid : 0
[Mon Mar 11 14:29:19 2024] .clock : 1282633453.161253
[Mon Mar 11 14:29:19 2024] .clock_task : 1264113913.150948
[Mon Mar 11 14:29:19 2024] .avg_idle : 1000000
[Mon Mar 11 14:29:19 2024] .max_idle_balance_cost : 500000
[Mon Mar 11 14:29:19 2024] rt_rq[36]:
[Mon Mar 11 14:29:19 2024] .rt_nr_running : 0
[Mon Mar 11 14:29:19 2024] .rt_nr_migratory : 0
[Mon Mar 11 14:29:19 2024] .rt_throttled : 0
[Mon Mar 11 14:29:19 2024] .rt_time : 0.000000
[Mon Mar 11 14:29:19 2024] .rt_runtime : 950.000000
[Mon Mar 11 14:29:19 2024] dl_rq[36]:
[Mon Mar 11 14:29:19 2024] .dl_nr_running : 0
[Mon Mar 11 14:29:19 2024] .dl_nr_migratory : 0
[Mon Mar 11 14:29:19 2024] .dl_bw->bw : 996147
[Mon Mar 11 14:29:19 2024] .dl_bw->total_bw : 0
[Mon Mar 11 14:29:19 2024] runnable tasks:
[Mon Mar 11 14:29:19 2024] S task PID tree-key switches prio wait-time sum-exec sum-sleep
[Mon Mar 11 14:29:19 2024] -------------------------------------------------------------------------------------------------------------
[Mon Mar 11 14:29:19 2024] S cpuhp/36 235 26429.055755 25 120 0.000000 0.727715 0.000000 0.000000 0 0 /
[Mon Mar 11 14:29:19 2024] S idle_inject/36 236 -12.000000 3 49 0.000000 0.000000 0.000000 0.000000 0 0 /
[Mon Mar 11 14:29:19 2024] S migration/36 237 2258.054608 362728 0 0.000000 1944.725306 0.000000 0.000000 0 0 /
[Mon Mar 11 14:29:19 2024] S ksoftirqd/36 238 40162136.741591 8034022 120 0.000000 200149.660796 0.000000 0.000000 0 0 /
[Mon Mar 11 14:29:19 2024] I kworker/36:0H 240 318.331682 4 100 0.000000 0.032690 0.000000 0.000000 0 0 /
[Mon Mar 11 14:29:19 2024] I kaluad 402 78.978709 2 100 0.000000 0.059950 0.000000 0.000000 0 0 /
[Mon Mar 11 14:29:19 2024] I kworker/36:1H 801 40162641.173472 1527143 100 0.000000 45688.673396 0.000000 0.000000 0 0 /
[Mon Mar 11 14:29:19 2024] I kdmflush/253:59 3893 28079.646107 2 100 0.000000 0.010937 0.000000 0.000000 0 0 /
[Mon Mar 11 14:29:19 2024] S jbd2/dm-33-8 4130 40162643.394077 1765898 120 0.000000 140931.592836 0.000000 0.000000 0 0 /
[Mon Mar 11 14:29:19 2024] S smbd-notifyd 4819 2321249.010629 3385071 120 0.000000 181688.331842 0.000000 0.000000 0 0 /autogroup-113
[Mon Mar 11 14:29:19 2024] S systemd 5558 7121.970550 16618 120 0.000000 97074.074352 0.000000 0.000000 0 0 /autogroup-121
[Mon Mar 11 14:29:19 2024] D nfsd 8794 40162642.132851 85773 120 0.000000 9585.592461 0.000000 0.000000 0 0 /
[Mon Mar 11 14:29:19 2024] D nfsd 8822 40116193.065430 308578 120 0.000000 28704.577400 0.000000 0.000000 0 0 /
[Mon Mar 11 14:29:19 2024] I kworker/36:1 2364257 40162641.219761 284801 120 0.000000 18733.324188 0.000000 0.000000 0 0 /
[Mon Mar 11 14:29:19 2024] I kworker/36:0 2472450 40150655.534616 8 120 0.000000 0.041589 0.000000 0.000000 0 0 /
[Mon Mar 11 14:29:19 2024] I kworker/u96:6 2472989 40162136.799824 49140 120 0.000000 15610.415321 0.000000 0.000000 0 0 /
[Mon Mar 11 14:29:19 2024] I kworker/36:2 2476104 40156521.875964 4 120 0.000000 0.069811 0.000000 0.000000 0 0 /
[Mon Mar 11 14:29:19 2024] cpu#37, 2600.000 MHz
[Mon Mar 11 14:29:19 2024] .nr_running : 0
[Mon Mar 11 14:29:19 2024] .nr_switches : 863794615
[Mon Mar 11 14:29:19 2024] .nr_uninterruptible : -2274
[Mon Mar 11 14:29:19 2024] .next_balance : 5577.334877
[Mon Mar 11 14:29:19 2024] .curr->pid : 0
[Mon Mar 11 14:29:19 2024] .clock : 1282633453.161099
[Mon Mar 11 14:29:19 2024] .clock_task : 1281269055.831703
[Mon Mar 11 14:29:19 2024] .avg_idle : 1000000
[Mon Mar 11 14:29:19 2024] .max_idle_balance_cost : 500000
[Mon Mar 11 14:29:19 2024] rt_rq[37]:
[Mon Mar 11 14:29:19 2024] .rt_nr_running : 0
[Mon Mar 11 14:29:19 2024] .rt_nr_migratory : 0
[Mon Mar 11 14:29:19 2024] .rt_throttled : 0
[Mon Mar 11 14:29:19 2024] .rt_time : 0.000000
[Mon Mar 11 14:29:19 2024] .rt_runtime : 950.000000
[Mon Mar 11 14:29:19 2024] dl_rq[37]:
[Mon Mar 11 14:29:19 2024] .dl_nr_running : 0
[Mon Mar 11 14:29:19 2024] .dl_nr_migratory : 0
[Mon Mar 11 14:29:19 2024] .dl_bw->bw : 996147
[Mon Mar 11 14:29:19 2024] .dl_bw->total_bw : 0
[Mon Mar 11 14:29:19 2024] runnable tasks:
[Mon Mar 11 14:29:19 2024] S task PID tree-key switches prio wait-time sum-exec sum-sleep
[Mon Mar 11 14:29:19 2024] -------------------------------------------------------------------------------------------------------------
[Mon Mar 11 14:29:19 2024] S cpuhp/37 241 29446.555105 25 120 0.000000 0.857971 0.000000 0.000000 1 0 /
[Mon Mar 11 14:29:19 2024] S idle_inject/37 242 -12.000000 3 49 0.000000 0.000000 0.000000 0.000000 1 0 /
[Mon Mar 11 14:29:19 2024] S migration/37 243 2326.054608 324178 0 0.000000 1949.245636 0.000000 0.000000 1 0 /
[Mon Mar 11 14:29:19 2024] S ksoftirqd/37 244 17661531.607683 1748896 120 0.000000 57209.549970 0.000000 0.000000 1 0 /
[Mon Mar 11 14:29:19 2024] I kworker/37:0H 246 1776.459651 4 100 0.000000 0.029970 0.000000 0.000000 1 0 /
[Mon Mar 11 14:29:19 2024] I nvme-auth-wq 882 967.825675 2 100 0.000000 0.008149 0.000000 0.000000 1 0 /
[Mon Mar 11 14:29:19 2024] S scsi_eh_12 924 1866.356119 26 120 0.000000 2.526508 0.000000 0.000000 1 0 /
[Mon Mar 11 14:29:19 2024] I kworker/37:1H 930 17661889.419846 767198 100 0.000000 26604.845528 0.000000 0.000000 1 0 /
[Mon Mar 11 14:29:19 2024] I lpfc_wq 932 1835.035017 2 100 0.000000 0.010705 0.000000 0.000000 1 0 /
[Mon Mar 11 14:29:19 2024] I kdmflush/253:21 3570 30568.382664 2 100 0.000000 0.009171 0.000000 0.000000 1 0 /
[Mon Mar 11 14:29:19 2024] I kdmflush/253:86 4162 30627.871120 2 100 0.000000 0.017731 0.000000 0.000000 1 0 /
[Mon Mar 11 14:29:19 2024] I ext4-rsv-conver 4327 31101.213258 2 100 0.000000 0.017887 0.000000 0.000000 1 0 /
[Mon Mar 11 14:29:19 2024] S f2b/f.samba 4439 23163.522760 2566672 120 0.000000 331096.791605 0.000000 0.000000 1 4439 /autogroup-96
[Mon Mar 11 14:29:19 2024] S gssproxy 4429 1396.806687 43322 120 0.000000 39309.365681 0.000000 0.000000 1 0 /autogroup-98
[Mon Mar 11 14:29:19 2024] S JS Helper 5179 -12.733809 12 120 0.000000 0.348841 0.000000 0.000000 1 0 /autogroup-104
[Mon Mar 11 14:29:19 2024] S reader#3 5336 425603.275060 495971 120 0.000000 409995.791717 0.000000 0.000000 1 5327 /autogroup-105
[Mon Mar 11 14:29:19 2024] D nfsd 8790 17661822.765779 79087 120 0.000000 9030.227870 0.000000 0.000000 1 0 /
[Mon Mar 11 14:29:19 2024] D nfsd 8855 17591330.602846 12034855 120 0.000000 531625.374692 0.000000 0.000000 1 0 /
[Mon Mar 11 14:29:19 2024] S dbus 21786 70.151293 112 120 0.000000 83.086232 0.000000 0.000000 1 0 /autogroup-87
[Mon Mar 11 14:29:19 2024] S automount 88154 11.945083 1 120 0.000000 0.993666 0.000000 0.000000 1 0 /autogroup-222
[Mon Mar 11 14:29:19 2024] S automount 88155 24.717274 1 120 0.000000 0.772198 0.000000 0.000000 1 0 /autogroup-222
[Mon Mar 11 14:29:19 2024] S smbd[10.87.27.3 2193786 990875.619912 1032 120 0.000000 283.353459 0.000000 0.000000 1 0 /autogroup-113
[Mon Mar 11 14:29:19 2024] S smbd[10.87.28.8 2325987 990875.861352 687 120 0.000000 234.835815 0.000000 0.000000 1 0 /autogroup-113
[Mon Mar 11 14:29:19 2024] S smbd[10.87.28.5 2346299 990851.613523 480 120 0.000000 151.753083 0.000000 0.000000 1 0 /autogroup-113
[Mon Mar 11 14:29:19 2024] I kworker/37:1 2411099 17661810.851206 109814 120 0.000000 7504.494947 0.000000 0.000000 1 0 /
[Mon Mar 11 14:29:19 2024] S smbd[10.40.146. 2456493 990875.742714 100 120 0.000000 56.896699 0.000000 0.000000 1 0 /autogroup-113
[Mon Mar 11 14:29:19 2024] I kworker/37:0 2474527 17657073.411967 10 120 0.000000 0.066711 0.000000 0.000000 1 0 /
[Mon Mar 11 14:29:19 2024] I kworker/37:2 2476855 17660704.148651 5 120 0.000000 0.025531 0.000000 0.000000 1 0 /
[Mon Mar 11 14:29:19 2024] S rpcd_classic 2477044 226134.995644 110 120 0.000000 71.523731 0.000000 0.000000 1 0 /autogroup-22820
[Mon Mar 11 14:29:19 2024] S rpcd_spoolss 2477392 226133.047902 57 120 0.000000 70.037449 0.000000 0.000000 1 0 /autogroup-22820
[Mon Mar 11 14:29:19 2024] cpu#38, 2600.000 MHz
[Mon Mar 11 14:29:19 2024] .nr_running : 0
[Mon Mar 11 14:29:19 2024] .nr_switches : 2411982442
[Mon Mar 11 14:29:19 2024] .nr_uninterruptible : 34366
[Mon Mar 11 14:29:19 2024] .next_balance : 5577.334876
[Mon Mar 11 14:29:19 2024] .curr->pid : 0
[Mon Mar 11 14:29:19 2024] .clock : 1282633454.157240
[Mon Mar 11 14:29:19 2024] .clock_task : 1266729442.249762
[Mon Mar 11 14:29:19 2024] .avg_idle : 840996
[Mon Mar 11 14:29:19 2024] .max_idle_balance_cost : 500000
[Mon Mar 11 14:29:19 2024] rt_rq[38]:
[Mon Mar 11 14:29:19 2024] .rt_nr_running : 0
[Mon Mar 11 14:29:19 2024] .rt_nr_migratory : 0
[Mon Mar 11 14:29:19 2024] .rt_throttled : 0
[Mon Mar 11 14:29:19 2024] .rt_time : 0.000000
[Mon Mar 11 14:29:19 2024] .rt_runtime : 950.000000
[Mon Mar 11 14:29:19 2024] dl_rq[38]:
[Mon Mar 11 14:29:19 2024] .dl_nr_running : 0
[Mon Mar 11 14:29:19 2024] .dl_nr_migratory : 0
[Mon Mar 11 14:29:19 2024] .dl_bw->bw : 996147
[Mon Mar 11 14:29:19 2024] .dl_bw->total_bw : 0
[Mon Mar 11 14:29:19 2024] runnable tasks:
[Mon Mar 11 14:29:19 2024] S task PID tree-key switches prio wait-time sum-exec sum-sleep
[Mon Mar 11 14:29:19 2024] -------------------------------------------------------------------------------------------------------------
[Mon Mar 11 14:29:19 2024] S cpuhp/38 247 29367.584529 25 120 0.000000 1.102400 0.000000 0.000000 0 0 /
[Mon Mar 11 14:29:19 2024] S idle_inject/38 248 -12.000000 3 49 0.000000 0.000000 0.000000 0.000000 0 0 /
[Mon Mar 11 14:29:19 2024] S migration/38 249 2394.054584 360036 0 0.000000 1952.378965 0.000000 0.000000 0 0 /
[Mon Mar 11 14:29:19 2024] S ksoftirqd/38 250 35897296.199335 6740495 120 0.000000 177167.053115 0.000000 0.000000 0 0 /
[Mon Mar 11 14:29:19 2024] I kworker/38:0H 252 275.785696 4 100 0.000000 0.018007 0.000000 0.000000 0 0 /
[Mon Mar 11 14:29:19 2024] S watchdogd 376 0.000000 2 49 0.000000 0.001131 0.000000 0.000000 0 0 /
[Mon Mar 11 14:29:19 2024] I kworker/38:1H 782 35897285.454785 1390661 100 0.000000 43730.675767 0.000000 0.000000 0 0 /
[Mon Mar 11 14:29:19 2024] S scsi_eh_9 918 3099.896301 25 120 0.000000 1.918330 0.000000 0.000000 0 0 /
[Mon Mar 11 14:29:19 2024] I scsi_tmf_13 927 3096.260663 2 100 0.000000 0.049846 0.000000 0.000000 0 0 /
[Mon Mar 11 14:29:19 2024] I kdmflush/253:54 3851 30477.037714 2 100 0.000000 0.016385 0.000000 0.000000 0 0 /
[Mon Mar 11 14:29:19 2024] S uwsgi 5237 400.212737 295 120 0.000000 329.201492 0.000000 0.000000 0 0 /autogroup-90
[Mon Mar 11 14:29:19 2024] S (sd-pam) 5572 11.653767 1 120 0.000000 0.702352 0.000000 0.000000 0 0 /autogroup-121
[Mon Mar 11 14:29:19 2024] D nfsd 8816 35897297.456099 543826 120 0.000000 37992.093535 0.000000 0.000000 0 0 /
[Mon Mar 11 14:29:19 2024] D nfsd 8826 35864240.192841 578969 120 0.000000 45585.767320 0.000000 0.000000 0 0 /
[Mon Mar 11 14:29:19 2024] D nfsd 8839 35833966.301340 1150071 120 0.000000 57664.031683 0.000000 0.000000 0 0 /
[Mon Mar 11 14:29:19 2024] D nfsd 8846 35859524.401393 11806699 120 0.000000 436337.699240 0.000000 0.000000 0 0 /
[Mon Mar 11 14:29:19 2024] S smbd[10.87.28.1 1049691 2391400.329546 13982 120 0.000000 3883.138043 0.000000 0.000000 0 0 /autogroup-113
[Mon Mar 11 14:29:19 2024] I kworker/38:2 2364258 35897336.059223 275122 120 0.000000 14109.874838 0.000000 0.000000 0 0 /
[Mon Mar 11 14:29:19 2024] S sshd 2371682 0.039535 53 120 0.000000 11.867012 0.000000 0.000000 0 0 /autogroup-23021
[Mon Mar 11 14:29:19 2024] S smbd[10.87.19.4 2434016 2391457.809595 842 120 0.000000 301.613891 0.000000 0.000000 0 0 /autogroup-113
[Mon Mar 11 14:29:19 2024] S smbd[10.33.88.9 2464105 2391469.663007 1640 120 0.000000 1354.336849 0.000000 0.000000 0 0 /autogroup-113
[Mon Mar 11 14:29:19 2024] I kworker/38:1 2475692 35893456.662804 6 120 0.000000 0.073610 0.000000 0.000000 0 0 /
[Mon Mar 11 14:29:19 2024] S smbd[10.40.128. 2478178 2391483.986499 625 120 0.000000 54.845725 0.000000 0.000000 0 0 /autogroup-113
[Mon Mar 11 14:29:19 2024] cpu#39, 2600.000 MHz
[Mon Mar 11 14:29:19 2024] .nr_running : 0
[Mon Mar 11 14:29:19 2024] .nr_switches : 845617600
[Mon Mar 11 14:29:19 2024] .nr_uninterruptible : -1266
[Mon Mar 11 14:29:19 2024] .next_balance : 5577.334877
[Mon Mar 11 14:29:19 2024] .curr->pid : 0
[Mon Mar 11 14:29:19 2024] .clock : 1282633454.161323
[Mon Mar 11 14:29:19 2024] .clock_task : 1281284910.810937
[Mon Mar 11 14:29:19 2024] .avg_idle : 1000000
[Mon Mar 11 14:29:19 2024] .max_idle_balance_cost : 500000
[Mon Mar 11 14:29:19 2024] cfs_rq[39]:/
[Mon Mar 11 14:29:19 2024] .exec_clock : 0.000000
[Mon Mar 11 14:29:19 2024] .MIN_vruntime : 0.000001
[Mon Mar 11 14:29:19 2024] .min_vruntime : 17343149.665584
[Mon Mar 11 14:29:19 2024] .max_vruntime : 0.000001
[Mon Mar 11 14:29:19 2024] .spread : 0.000000
[Mon Mar 11 14:29:19 2024] .spread0 : -3074516.842531
[Mon Mar 11 14:29:19 2024] .nr_spread_over : 0
[Mon Mar 11 14:29:19 2024] .nr_running : 0
[Mon Mar 11 14:29:19 2024] .h_nr_running : 0
[Mon Mar 11 14:29:19 2024] .idle_nr_running : 0
[Mon Mar 11 14:29:19 2024] .idle_h_nr_running : 0
[Mon Mar 11 14:29:19 2024] .load : 0
[Mon Mar 11 14:29:19 2024] .load_avg : 2
[Mon Mar 11 14:29:19 2024] .runnable_avg : 0
[Mon Mar 11 14:29:19 2024] .util_avg : 0
[Mon Mar 11 14:29:19 2024] .util_est_enqueued : 0
[Mon Mar 11 14:29:19 2024] .removed.load_avg : 0
[Mon Mar 11 14:29:19 2024] .removed.util_avg : 0
[Mon Mar 11 14:29:19 2024] .removed.runnable_avg : 0
[Mon Mar 11 14:29:19 2024] .tg_load_avg_contrib : 0
[Mon Mar 11 14:29:19 2024] .tg_load_avg : 0
[Mon Mar 11 14:29:19 2024] .throttled : 0
[Mon Mar 11 14:29:19 2024] .throttle_count : 0
[Mon Mar 11 14:29:19 2024] rt_rq[39]:
[Mon Mar 11 14:29:19 2024] .rt_nr_running : 0
[Mon Mar 11 14:29:19 2024] .rt_nr_migratory : 0
[Mon Mar 11 14:29:19 2024] .rt_throttled : 0
[Mon Mar 11 14:29:19 2024] .rt_time : 0.000000
[Mon Mar 11 14:29:19 2024] .rt_runtime : 950.000000
[Mon Mar 11 14:29:19 2024] dl_rq[39]:
[Mon Mar 11 14:29:19 2024] .dl_nr_running : 0
[Mon Mar 11 14:29:19 2024] .dl_nr_migratory : 0
[Mon Mar 11 14:29:19 2024] .dl_bw->bw : 996147
[Mon Mar 11 14:29:19 2024] .dl_bw->total_bw : 0
[Mon Mar 11 14:29:19 2024] runnable tasks:
[Mon Mar 11 14:29:19 2024] S task PID tree-key switches prio wait-time sum-exec sum-sleep
[Mon Mar 11 14:29:19 2024] -------------------------------------------------------------------------------------------------------------
[Mon Mar 11 14:29:19 2024] S cpuhp/39 253 25042.299856 25 120 0.000000 0.639678 0.000000 0.000000 1 0 /
[Mon Mar 11 14:29:19 2024] S idle_inject/39 254 -11.995353 3 49 0.000000 0.010391 0.000000 0.000000 1 0 /
[Mon Mar 11 14:29:19 2024] S migration/39 255 2462.054589 324307 0 0.000000 1941.529853 0.000000 0.000000 1 0 /
[Mon Mar 11 14:29:19 2024] S ksoftirqd/39 256 17342017.237773 1703732 120 0.000000 56103.106457 0.000000 0.000000 1 0 /
[Mon Mar 11 14:29:19 2024] I kworker/39:0H 258 136.379023 4 100 0.000000 0.028311 0.000000 0.000000 1 0 /
[Mon Mar 11 14:29:19 2024] I kworker/39:1H 765 17343137.665903 750770 100 0.000000 26398.818303 0.000000 0.000000 1 0 /
[Mon Mar 11 14:29:19 2024] S lpfc_worker_1 989 17343137.665723 284763 100 0.000000 5426.790937 0.000000 0.000000 1 0 /
[Mon Mar 11 14:29:19 2024] S multipathd 3354 0.000000 12 0 0.000000 11.094414 0.000000 0.000000 1 0 /autogroup-45
[Mon Mar 11 14:29:19 2024] S avahi-daemon 4304 428.281844 93423 120 0.000000 15830.156555 0.000000 0.000000 1 0 /autogroup-70
[Mon Mar 11 14:29:19 2024] S nsrexecd 5339 7672.943725 475447 120 0.000000 34733.376497 0.000000 0.000000 1 5339 /autogroup-117
[Mon Mar 11 14:29:19 2024] S jbd2/dm-95-8 6488 17248764.126110 671 120 0.000000 208.657763 0.000000 0.000000 1 0 /
[Mon Mar 11 14:29:19 2024] I kdmflush/253:99 8694 26600.750779 2 100 0.000000 0.063299 0.000000 0.000000 1 0 /
[Mon Mar 11 14:29:19 2024] I ib-comp-wq 8762 26636.949487 2 100 0.000000 0.065486 0.000000 0.000000 1 0 /
[Mon Mar 11 14:29:19 2024] D nfsd 8798 17343149.665584 93175 120 0.000000 10395.675057 0.000000 0.000000 1 0 /
[Mon Mar 11 14:29:19 2024] D nfsd 8823 17343148.085731 2973818 120 0.000000 147797.809136 0.000000 0.000000 1 0 /
[Mon Mar 11 14:29:19 2024] D nfsd 8859 17303619.863031 20778866 120 0.000000 898391.855561 0.000000 0.000000 1 0 /
[Mon Mar 11 14:29:19 2024] S automount 88121 2653.399424 13 120 0.000000 2.245948 0.000000 0.000000 1 0 /autogroup-222
[Mon Mar 11 14:29:19 2024] S tlsmgr 158515 1539.398420 4346 120 0.000000 953.044802 0.000000 0.000000 1 0 /autogroup-115
[Mon Mar 11 14:29:19 2024] I kworker/39:2 2251290 17343137.733346 174864 120 0.000000 12516.844085 0.000000 0.000000 1 0 /
[Mon Mar 11 14:29:19 2024] S smbd[10.87.27.8 2372510 949322.698281 384 120 0.000000 121.010069 0.000000 0.000000 1 0 /autogroup-113
[Mon Mar 11 14:29:19 2024] I kworker/39:0 2471904 17342219.118298 5 120 0.000000 0.550950 0.000000 0.000000 1 0 /
[Mon Mar 11 14:29:19 2024] I kworker/39:1 2478206 17342231.085874 2 120 0.000000 0.060302 0.000000 0.000000 1 0 /
[Mon Mar 11 14:29:19 2024] cpu#40, 2600.000 MHz
[Mon Mar 11 14:29:19 2024] .nr_running : 0
[Mon Mar 11 14:29:19 2024] .nr_switches : 2980246779
[Mon Mar 11 14:29:19 2024] .nr_uninterruptible : 24150
[Mon Mar 11 14:29:19 2024] .next_balance : 5577.334878
[Mon Mar 11 14:29:19 2024] .curr->pid : 0
[Mon Mar 11 14:29:19 2024] .clock : 1282633454.161538
[Mon Mar 11 14:29:19 2024] .clock_task : 1257562675.387490
[Mon Mar 11 14:29:19 2024] .avg_idle : 1000000
[Mon Mar 11 14:29:19 2024] .max_idle_balance_cost : 500000
[Mon Mar 11 14:29:19 2024] rt_rq[40]:
[Mon Mar 11 14:29:19 2024] .rt_nr_running : 0
[Mon Mar 11 14:29:19 2024] .rt_nr_migratory : 0
[Mon Mar 11 14:29:19 2024] .rt_throttled : 0
[Mon Mar 11 14:29:19 2024] .rt_time : 0.000000
[Mon Mar 11 14:29:19 2024] .rt_runtime : 950.000000
[Mon Mar 11 14:29:19 2024] dl_rq[40]:
[Mon Mar 11 14:29:19 2024] .dl_nr_running : 0
[Mon Mar 11 14:29:19 2024] .dl_nr_migratory : 0
[Mon Mar 11 14:29:19 2024] .dl_bw->bw : 996147
[Mon Mar 11 14:29:19 2024] .dl_bw->total_bw : 0
[Mon Mar 11 14:29:19 2024] runnable tasks:
[Mon Mar 11 14:29:19 2024] S task PID tree-key switches prio wait-time sum-exec sum-sleep
[Mon Mar 11 14:29:19 2024] -------------------------------------------------------------------------------------------------------------
[Mon Mar 11 14:29:19 2024] S cpuhp/40 259 22774.414769 25 120 0.000000 1.054486 0.000000 0.000000 0 0 /
[Mon Mar 11 14:29:19 2024] S idle_inject/40 260 -12.000000 3 49 0.000000 0.000000 0.000000 0.000000 0 0 /
[Mon Mar 11 14:29:19 2024] S migration/40 261 2530.056051 394260 0 0.000000 2106.751273 0.000000 0.000000 0 0 /
[Mon Mar 11 14:29:19 2024] S ksoftirqd/40 262 44019337.022821 10993228 120 0.000000 238958.281123 0.000000 0.000000 0 0 /
[Mon Mar 11 14:29:19 2024] I kworker/40:0H 264 70.589518 4 100 0.000000 0.084035 0.000000 0.000000 0 0 /
[Mon Mar 11 14:29:19 2024] I kintegrityd 369 51.034898 2 100 0.000000 0.000000 0.000000 0.000000 0 0 /
[Mon Mar 11 14:29:19 2024] I md_bitmap 374 51.036105 2 100 0.000000 0.001207 0.000000 0.000000 0 0 /
[Mon Mar 11 14:29:19 2024] I kworker/40:1H 804 44019336.969523 1671018 100 0.000000 47751.898571 0.000000 0.000000 0 0 /
[Mon Mar 11 14:29:19 2024] S scsi_eh_2 900 2889.559709 25 120 0.000000 4.466813 0.000000 0.000000 0 0 /
[Mon Mar 11 14:29:19 2024] S scsi_eh_14 928 2890.311214 25 120 0.000000 4.418888 0.000000 0.000000 0 0 /
[Mon Mar 11 14:29:19 2024] S jbd2/dm-77-8 4353 44019340.410796 1682917 120 0.000000 464110.420386 0.000000 0.000000 0 0 /
[Mon Mar 11 14:29:19 2024] I kdmflush/253:97 6433 26014.861775 2 100 0.000000 0.014046 0.000000 0.000000 0 0 /
[Mon Mar 11 14:29:19 2024] S jbd2/dm-101-8 8746 43860152.185757 13470 120 0.000000 3385.568781 0.000000 0.000000 0 0 /
[Mon Mar 11 14:29:19 2024] D nfsd 8797 44019348.969030 92422 120 0.000000 10285.420367 0.000000 0.000000 0 0 /
[Mon Mar 11 14:29:19 2024] D nfsd 8835 43950074.644499 1033655 120 0.000000 61019.596108 0.000000 0.000000 0 0 /
[Mon Mar 11 14:29:19 2024] D nfsd 8840 43941588.794471 1931440 120 0.000000 84935.159947 0.000000 0.000000 0 0 /
[Mon Mar 11 14:29:19 2024] D nfsd 8857 43932394.829460 15366373 120 0.000000 676417.528097 0.000000 0.000000 0 0 /
[Mon Mar 11 14:29:19 2024] D nfsd 8869 43929480.450118 283550159 120 0.000000 8285285.827066 0.000000 0.000000 0 0 /
[Mon Mar 11 14:29:19 2024] S smbd[10.86.26.7 348298 2282803.750072 36042 120 0.000000 10191.207203 0.000000 0.000000 0 0 /autogroup-113
[Mon Mar 11 14:29:19 2024] S smbd[10.87.27.1 1045161 2282504.598515 5109 120 0.000000 1412.250436 0.000000 0.000000 0 0 /autogroup-113
[Mon Mar 11 14:29:19 2024] I kworker/40:0 2193065 44019337.054720 584792 120 0.000000 21168.813802 0.000000 0.000000 0 0 /
[Mon Mar 11 14:29:19 2024] S smbd[10.87.29.1 2330921 2282464.301928 2283 120 0.000000 793.827340 0.000000 0.000000 0 0 /autogroup-113
[Mon Mar 11 14:29:19 2024] S smbd[10.87.27.1 2339621 2282447.267549 1073 120 0.000000 513.012288 0.000000 0.000000 0 0 /autogroup-113
[Mon Mar 11 14:29:19 2024] S smbd[10.40.231. 2433966 2282521.660267 5322 120 0.000000 2847.313292 0.000000 0.000000 0 0 /autogroup-113
[Mon Mar 11 14:29:19 2024] S smbd[10.33.88.4 2443657 2282726.707646 122 120 0.000000 49.163112 0.000000 0.000000 0 0 /autogroup-113
[Mon Mar 11 14:29:19 2024] I kworker/40:1 2475325 44018722.803022 16 120 0.000000 0.147345 0.000000 0.000000 0 0 /
[Mon Mar 11 14:29:19 2024] I kworker/40:2 2478207 44019024.950793 3 120 0.000000 0.017096 0.000000 0.000000 0 0 /
[Mon Mar 11 14:29:19 2024] S rpcd_spoolss 2478220 18636.984234 143 120 0.000000 96.879171 0.000000 0.000000 0 0 /autogroup-22820
[Mon Mar 11 14:29:19 2024] cpu#41, 2600.000 MHz
[Mon Mar 11 14:29:19 2024] .nr_running : 0
[Mon Mar 11 14:29:19 2024] .nr_switches : 769033688
[Mon Mar 11 14:29:19 2024] .nr_uninterruptible : -2153
[Mon Mar 11 14:29:19 2024] .next_balance : 5577.334876
[Mon Mar 11 14:29:19 2024] .curr->pid : 0
[Mon Mar 11 14:29:19 2024] .clock : 1282633454.160581
[Mon Mar 11 14:29:19 2024] .clock_task : 1281380260.022644
[Mon Mar 11 14:29:19 2024] .avg_idle : 1000000
[Mon Mar 11 14:29:19 2024] .max_idle_balance_cost : 500000
[Mon Mar 11 14:29:19 2024] rt_rq[41]:
[Mon Mar 11 14:29:19 2024] .rt_nr_running : 0
[Mon Mar 11 14:29:19 2024] .rt_nr_migratory : 0
[Mon Mar 11 14:29:19 2024] .rt_throttled : 0
[Mon Mar 11 14:29:19 2024] .rt_time : 0.000000
[Mon Mar 11 14:29:19 2024] .rt_runtime : 950.000000
[Mon Mar 11 14:29:19 2024] dl_rq[41]:
[Mon Mar 11 14:29:19 2024] .dl_nr_running : 0
[Mon Mar 11 14:29:19 2024] .dl_nr_migratory : 0
[Mon Mar 11 14:29:19 2024] .dl_bw->bw : 996147
[Mon Mar 11 14:29:19 2024] .dl_bw->total_bw : 0
[Mon Mar 11 14:29:19 2024] runnable tasks:
[Mon Mar 11 14:29:19 2024] S task PID tree-key switches prio wait-time sum-exec sum-sleep
[Mon Mar 11 14:29:19 2024] -------------------------------------------------------------------------------------------------------------
[Mon Mar 11 14:29:19 2024] S cpuhp/41 265 22916.688052 25 120 0.000000 0.850472 0.000000 0.000000 1 0 /
[Mon Mar 11 14:29:19 2024] S idle_inject/41 266 -12.000000 3 49 0.000000 0.000000 0.000000 0.000000 1 0 /
[Mon Mar 11 14:29:19 2024] S migration/41 267 2598.056055 325773 0 0.000000 2098.472928 0.000000 0.000000 1 0 /
[Mon Mar 11 14:29:19 2024] S ksoftirqd/41 268 15809663.340352 1653137 120 0.000000 54863.923639 0.000000 0.000000 1 0 /
[Mon Mar 11 14:29:19 2024] I kworker/41:0H 270 153.424794 4 100 0.000000 0.054856 0.000000 0.000000 1 0 /
[Mon Mar 11 14:29:19 2024] I kworker/41:1H 760 15809849.989686 942541 100 0.000000 32671.621713 0.000000 0.000000 1 0 /
[Mon Mar 11 14:29:19 2024] I ixgbe 886 2410.191218 2 100 0.000000 0.010429 0.000000 0.000000 1 0 /
[Mon Mar 11 14:29:19 2024] I nfit 3237 22516.800348 2 100 0.000000 0.009425 0.000000 0.000000 1 0 /
[Mon Mar 11 14:29:19 2024] S multipathd 3355 0.000000 42958 0 0.000000 1095.175132 0.000000 0.000000 1 0 /autogroup-45
[Mon Mar 11 14:29:19 2024] I kdmflush/253:25 3598 23654.529860 2 100 0.000000 0.010061 0.000000 0.000000 1 0 /
[Mon Mar 11 14:29:19 2024] I ext4-rsv-conver 4243 23864.506328 2 100 0.000000 0.034406 0.000000 0.000000 1 0 /
[Mon Mar 11 14:29:19 2024] S writer#1 5329 277180.368642 55225428 120 0.000000 1457581.827934 0.000000 0.000000 1 5327 /autogroup-105
[Mon Mar 11 14:29:19 2024] I ext4-rsv-conver 8733 25341.513119 2 100 0.000000 0.065460 0.000000 0.000000 1 0 /
[Mon Mar 11 14:29:19 2024] S nfsclients.sh 91766 289400.280209 212204 120 0.000000 316873.551136 0.000000 0.000000 1 0 /autogroup-292
[Mon Mar 11 14:29:19 2024] S bash 2371683 95.410554 40 120 0.000000 39.418602 0.000000 0.000000 1 0 /autogroup-23022
[Mon Mar 11 14:29:19 2024] S smbd[10.40.147. 2377087 853732.743590 61578 120 0.000000 22680.039593 0.000000 0.000000 1 0 /autogroup-113
[Mon Mar 11 14:29:19 2024] I kworker/41:1 2408233 15809851.021005 177638 120 0.000000 6159.061412 0.000000 0.000000 1 0 /
[Mon Mar 11 14:29:19 2024] S smbd[10.87.28.1 2419997 853787.191465 215 120 0.000000 77.941484 0.000000 0.000000 1 0 /autogroup-113
[Mon Mar 11 14:29:19 2024] S smbd[10.40.136. 2459316 853749.947129 87 120 0.000000 47.726812 0.000000 0.000000 1 0 /autogroup-113
[Mon Mar 11 14:29:19 2024] I kworker/41:0 2475205 15808064.909054 4 120 0.000000 0.031815 0.000000 0.000000 1 0 /
[Mon Mar 11 14:29:19 2024] D kworker/u98:2 2476491 15809851.840965 40438 120 0.000000 1287.117784 0.000000 0.000000 1 0 /
[Mon Mar 11 14:29:19 2024] I kworker/41:2 2477809 15808088.892049 2 120 0.000000 0.002833 0.000000 0.000000 1 0 /
[Mon Mar 11 14:29:19 2024] cpu#42, 2600.000 MHz
[Mon Mar 11 14:29:19 2024] .nr_running : 0
[Mon Mar 11 14:29:19 2024] .nr_switches : 2996328903
[Mon Mar 11 14:29:19 2024] .nr_uninterruptible : 23982
[Mon Mar 11 14:29:19 2024] .next_balance : 5577.334879
[Mon Mar 11 14:29:19 2024] .curr->pid : 0
[Mon Mar 11 14:29:19 2024] .clock : 1282633454.163040
[Mon Mar 11 14:29:19 2024] .clock_task : 1259162399.048103
[Mon Mar 11 14:29:19 2024] .avg_idle : 1000000
[Mon Mar 11 14:29:19 2024] .max_idle_balance_cost : 500000
[Mon Mar 11 14:29:19 2024] rt_rq[42]:
[Mon Mar 11 14:29:19 2024] .rt_nr_running : 0
[Mon Mar 11 14:29:19 2024] .rt_nr_migratory : 0
[Mon Mar 11 14:29:19 2024] .rt_throttled : 0
[Mon Mar 11 14:29:19 2024] .rt_time : 0.000000
[Mon Mar 11 14:29:19 2024] .rt_runtime : 950.000000
[Mon Mar 11 14:29:19 2024] dl_rq[42]:
[Mon Mar 11 14:29:19 2024] .dl_nr_running : 0
[Mon Mar 11 14:29:19 2024] .dl_nr_migratory : 0
[Mon Mar 11 14:29:19 2024] .dl_bw->bw : 996147
[Mon Mar 11 14:29:19 2024] .dl_bw->total_bw : 0
[Mon Mar 11 14:29:19 2024] runnable tasks:
[Mon Mar 11 14:29:19 2024] S task PID tree-key switches prio wait-time sum-exec sum-sleep
[Mon Mar 11 14:29:19 2024] -------------------------------------------------------------------------------------------------------------
[Mon Mar 11 14:29:19 2024] S cpuhp/42 271 23770.590731 25 120 0.000000 1.112376 0.000000 0.000000 0 0 /
[Mon Mar 11 14:29:19 2024] S idle_inject/42 272 -11.996193 3 49 0.000000 0.008634 0.000000 0.000000 0 0 /
[Mon Mar 11 14:29:19 2024] S migration/42 273 2666.056029 376584 0 0.000000 1978.490589 0.000000 0.000000 0 0 /
[Mon Mar 11 14:29:19 2024] S ksoftirqd/42 274 44313378.415395 10284550 120 0.000000 234271.575233 0.000000 0.000000 0 0 /
[Mon Mar 11 14:29:19 2024] I kworker/42:0H 276 405.090127 4 100 0.000000 0.017486 0.000000 0.000000 0 0 /
[Mon Mar 11 14:29:19 2024] I cryptd 368 33.956059 2 100 0.000000 0.001297 0.000000 0.000000 0 0 /
[Mon Mar 11 14:29:19 2024] I edac-poller 375 45.957394 2 100 0.000000 0.001341 0.000000 0.000000 0 0 /
[Mon Mar 11 14:29:19 2024] I kmpath_rdacd 401 70.023013 2 100 0.000000 0.060965 0.000000 0.000000 0 0 /
[Mon Mar 11 14:29:19 2024] I kworker/42:1H 805 44313535.668704 1664246 100 0.000000 47711.767500 0.000000 0.000000 0 0 /
[Mon Mar 11 14:29:19 2024] S scsi_eh_7 914 1981.205596 25 120 0.000000 3.018351 0.000000 0.000000 0 0 /
[Mon Mar 11 14:29:19 2024] I scsi_tmf_7 915 1978.339045 2 100 0.000000 0.061972 0.000000 0.000000 0 0 /
[Mon Mar 11 14:29:19 2024] I kdmflush/253:1 3216 19623.395132 2 100 0.000000 0.008752 0.000000 0.000000 0 0 /
[Mon Mar 11 14:29:19 2024] I ext4-rsv-conver 4241 24415.520577 2 100 0.000000 0.069758 0.000000 0.000000 0 0 /
[Mon Mar 11 14:29:19 2024] D in:imjournal 4837 1018.808474 1549844 120 0.000000 494163.301716 0.000000 0.000000 0 4837 /autogroup-112
[Mon Mar 11 14:29:19 2024] S nsrexecd 5347 4562.820660 256334 120 0.000000 8065.031771 0.000000 0.000000 0 5347 /autogroup-117
[Mon Mar 11 14:29:19 2024] D nfsd 8812 44312943.970944 205781 120 0.000000 20048.371105 0.000000 0.000000 0 0 /
[Mon Mar 11 14:29:19 2024] D nfsd 8870 44214210.515615 460088797 120 0.000000 12545690.671023 0.000000 0.000000 0 0 /
[Mon Mar 11 14:29:19 2024] D nfsd 8871 44212554.801884 778478749 120 0.000000 20032936.803347 0.000000 0.000000 0 0 /
[Mon Mar 11 14:29:19 2024] S gmain 21425 11.129420 1 120 0.000000 0.178004 0.000000 0.000000 0 0 /autogroup-154
[Mon Mar 11 14:29:19 2024] S automount 88140 11.871090 1 120 0.000000 0.919673 0.000000 0.000000 0 0 /autogroup-222
[Mon Mar 11 14:29:19 2024] S automount 88164 24.730710 1 120 0.000000 0.859627 0.000000 0.000000 0 0 /autogroup-222
[Mon Mar 11 14:29:19 2024] S smbd[10.40.230. 2340528 2874680.406835 37051 120 0.000000 13523.823190 0.000000 0.000000 0 0 /autogroup-113
[Mon Mar 11 14:29:19 2024] S smbd[10.39.226. 2411168 2874650.986950 24300 120 0.000000 2759.730043 0.000000 0.000000 0 0 /autogroup-113
[Mon Mar 11 14:29:19 2024] I kworker/42:0 2427884 44313535.694543 133518 120 0.000000 9601.413689 0.000000 0.000000 0 0 /
[Mon Mar 11 14:29:19 2024] I kworker/42:2 2474171 44307997.269374 9 120 0.000000 0.095482 0.000000 0.000000 0 0 /
[Mon Mar 11 14:29:19 2024] I kworker/42:1 2477540 44311426.797706 7 120 0.000000 0.073412 0.000000 0.000000 0 0 /
[Mon Mar 11 14:29:19 2024] S rpcd_spoolss 2478222 18709.324821 69 120 0.000000 69.818768 0.000000 0.000000 0 0 /autogroup-22820
[Mon Mar 11 14:29:19 2024] cpu#43, 2600.000 MHz
[Mon Mar 11 14:29:19 2024] .nr_running : 1
[Mon Mar 11 14:29:19 2024] .nr_switches : 852520640
[Mon Mar 11 14:29:19 2024] .nr_uninterruptible : -1295
[Mon Mar 11 14:29:19 2024] .next_balance : 5577.334931
[Mon Mar 11 14:29:19 2024] .curr->pid : 2792
[Mon Mar 11 14:29:19 2024] .clock : 1282633455.160727
[Mon Mar 11 14:29:19 2024] .clock_task : 1281307282.033299
[Mon Mar 11 14:29:19 2024] .avg_idle : 924089
[Mon Mar 11 14:29:19 2024] .max_idle_balance_cost : 500000
[Mon Mar 11 14:29:19 2024] cfs_rq[43]:/autogroup-29
[Mon Mar 11 14:29:19 2024] .exec_clock : 0.000000
[Mon Mar 11 14:29:19 2024] .MIN_vruntime : 0.000001
[Mon Mar 11 14:29:19 2024] .min_vruntime : 5369.166860
[Mon Mar 11 14:29:19 2024] .max_vruntime : 0.000001
[Mon Mar 11 14:29:19 2024] .spread : 0.000000
[Mon Mar 11 14:29:19 2024] .spread0 : -20412297.341255
[Mon Mar 11 14:29:19 2024] .nr_spread_over : 0
[Mon Mar 11 14:29:19 2024] .nr_running : 1
[Mon Mar 11 14:29:19 2024] .h_nr_running : 1
[Mon Mar 11 14:29:19 2024] .idle_nr_running : 0
[Mon Mar 11 14:29:19 2024] .idle_h_nr_running : 0
[Mon Mar 11 14:29:19 2024] .load : 1048576
[Mon Mar 11 14:29:19 2024] .load_avg : 32
[Mon Mar 11 14:29:19 2024] .runnable_avg : 32
[Mon Mar 11 14:29:19 2024] .util_avg : 32
[Mon Mar 11 14:29:19 2024] .util_est_enqueued : 0
[Mon Mar 11 14:29:19 2024] .removed.load_avg : 0
[Mon Mar 11 14:29:19 2024] .removed.util_avg : 0
[Mon Mar 11 14:29:19 2024] .removed.runnable_avg : 0
[Mon Mar 11 14:29:19 2024] .tg_load_avg_contrib : 32
[Mon Mar 11 14:29:19 2024] .tg_load_avg : 32
[Mon Mar 11 14:29:19 2024] .throttled : 0
[Mon Mar 11 14:29:19 2024] .throttle_count : 0
[Mon Mar 11 14:29:19 2024] .se->exec_start : 1281307282.033299
[Mon Mar 11 14:29:19 2024] .se->vruntime : 17451283.075921
[Mon Mar 11 14:29:19 2024] .se->sum_exec_runtime : 5347.659787
[Mon Mar 11 14:29:19 2024] .se->load.weight : 1048576
[Mon Mar 11 14:29:19 2024] .se->avg.load_avg : 43
[Mon Mar 11 14:29:19 2024] .se->avg.util_avg : 32
[Mon Mar 11 14:29:19 2024] .se->avg.runnable_avg : 32
[Mon Mar 11 14:29:19 2024] cfs_rq[43]:/
[Mon Mar 11 14:29:19 2024] .exec_clock : 0.000000
[Mon Mar 11 14:29:19 2024] .MIN_vruntime : 0.000001
[Mon Mar 11 14:29:19 2024] .min_vruntime : 17451283.075921
[Mon Mar 11 14:29:19 2024] .max_vruntime : 0.000001
[Mon Mar 11 14:29:19 2024] .spread : 0.000000
[Mon Mar 11 14:29:19 2024] .spread0 : -2966383.432194
[Mon Mar 11 14:29:19 2024] .nr_spread_over : 0
[Mon Mar 11 14:29:19 2024] .nr_running : 1
[Mon Mar 11 14:29:19 2024] .h_nr_running : 1
[Mon Mar 11 14:29:19 2024] .idle_nr_running : 0
[Mon Mar 11 14:29:19 2024] .idle_h_nr_running : 0
[Mon Mar 11 14:29:19 2024] .load : 1048576
[Mon Mar 11 14:29:19 2024] .load_avg : 43
[Mon Mar 11 14:29:19 2024] .runnable_avg : 32
[Mon Mar 11 14:29:19 2024] .util_avg : 32
[Mon Mar 11 14:29:19 2024] .util_est_enqueued : 143
[Mon Mar 11 14:29:19 2024] .removed.load_avg : 0
[Mon Mar 11 14:29:19 2024] .removed.util_avg : 0
[Mon Mar 11 14:29:19 2024] .removed.runnable_avg : 0
[Mon Mar 11 14:29:19 2024] .tg_load_avg_contrib : 0
[Mon Mar 11 14:29:19 2024] .tg_load_avg : 0
[Mon Mar 11 14:29:19 2024] .throttled : 0
[Mon Mar 11 14:29:19 2024] .throttle_count : 0
[Mon Mar 11 14:29:19 2024] rt_rq[43]:
[Mon Mar 11 14:29:19 2024] .rt_nr_running : 0
[Mon Mar 11 14:29:19 2024] .rt_nr_migratory : 0
[Mon Mar 11 14:29:19 2024] .rt_throttled : 0
[Mon Mar 11 14:29:19 2024] .rt_time : 0.000000
[Mon Mar 11 14:29:19 2024] .rt_runtime : 950.000000
[Mon Mar 11 14:29:19 2024] dl_rq[43]:
[Mon Mar 11 14:29:19 2024] .dl_nr_running : 0
[Mon Mar 11 14:29:19 2024] .dl_nr_migratory : 0
[Mon Mar 11 14:29:19 2024] .dl_bw->bw : 996147
[Mon Mar 11 14:29:19 2024] .dl_bw->total_bw : 0
[Mon Mar 11 14:29:19 2024] runnable tasks:
[Mon Mar 11 14:29:19 2024] S task PID tree-key switches prio wait-time sum-exec sum-sleep
[Mon Mar 11 14:29:19 2024] -------------------------------------------------------------------------------------------------------------
[Mon Mar 11 14:29:19 2024] S cpuhp/43 277 27597.030351 25 120 0.000000 0.898476 0.000000 0.000000 1 0 /
[Mon Mar 11 14:29:19 2024] S idle_inject/43 278 -12.000000 3 49 0.000000 0.000000 0.000000 0.000000 1 0 /
[Mon Mar 11 14:29:19 2024] S migration/43 279 2734.056023 323730 0 0.000000 1942.974615 0.000000 0.000000 1 0 /
[Mon Mar 11 14:29:19 2024] S ksoftirqd/43 280 17450792.576998 1723775 120 0.000000 56842.083760 0.000000 0.000000 1 0 /
[Mon Mar 11 14:29:19 2024] I kworker/43:0H 282 36.451975 4 100 0.000000 0.154937 0.000000 0.000000 1 0 /
[Mon Mar 11 14:29:19 2024] I kworker/43:1H 728 17450633.834340 886926 100 0.000000 30777.899285 0.000000 0.000000 1 0 /
[Mon Mar 11 14:29:19 2024] >R systemd-journal 2792 5369.166860 7467532 120 0.000000 617171.090881 0.000000 0.000000 1 2792 /autogroup-29
[Mon Mar 11 14:29:19 2024] S dbus-broker-lau 4294 5406.993099 402 120 0.000000 78.073023 0.000000 0.000000 1 0 /autogroup-65
[Mon Mar 11 14:29:19 2024] S jbd2/dm-88-8 4330 17345548.757446 8756 120 0.000000 1451.682965 0.000000 0.000000 1 0 /
[Mon Mar 11 14:29:19 2024] S JS Helper 5177 11.109554 11 120 0.000000 0.158137 0.000000 0.000000 1 0 /autogroup-104
[Mon Mar 11 14:29:19 2024] S master 4982 943.815980 96965 120 0.000000 13883.092393 0.000000 0.000000 1 0 /autogroup-115
[Mon Mar 11 14:29:19 2024] D nfsd 8838 17414572.885648 2472995 120 0.000000 129045.613813 0.000000 0.000000 1 0 /
[Mon Mar 11 14:29:19 2024] S smbd[10.88.18.1 3000371 869127.471874 1362984 120 0.000000 87769.378646 0.000000 0.000000 1 3000371 /autogroup-113
[Mon Mar 11 14:29:19 2024] S smbd[10.87.29.1 2341353 869127.560035 655 120 0.000000 234.201396 0.000000 0.000000 1 0 /autogroup-113
[Mon Mar 11 14:29:19 2024] S sshd 2371675 78.317121 24 120 0.000000 84.581467 0.000000 0.000000 1 0 /autogroup-23021
[Mon Mar 11 14:29:19 2024] I kworker/43:2 2414694 17451263.645045 114830 120 0.000000 6441.889371 0.000000 0.000000 1 0 /
[Mon Mar 11 14:29:19 2024] S smbd[10.87.20.1 2462719 869038.676470 638 120 0.000000 221.309743 0.000000 0.000000 1 0 /autogroup-113
[Mon Mar 11 14:29:19 2024] I kworker/u98:12 2464235 17451120.593081 234066 120 0.000000 6753.054198 0.000000 0.000000 1 0 /
[Mon Mar 11 14:29:19 2024] I kworker/43:1 2471690 17426068.129165 9 120 0.000000 0.052791 0.000000 0.000000 1 0 /
[Mon Mar 11 14:29:19 2024] cpu#44, 2600.000 MHz
[Mon Mar 11 14:29:19 2024] .nr_running : 0
[Mon Mar 11 14:29:19 2024] .nr_switches : 2816030255
[Mon Mar 11 14:29:19 2024] .nr_uninterruptible : 29153
[Mon Mar 11 14:29:19 2024] .next_balance : 5577.334878
[Mon Mar 11 14:29:19 2024] .curr->pid : 0
[Mon Mar 11 14:29:19 2024] .clock : 1282633455.161070
[Mon Mar 11 14:29:19 2024] .clock_task : 1262384626.708904
[Mon Mar 11 14:29:19 2024] .avg_idle : 1000000
[Mon Mar 11 14:29:19 2024] .max_idle_balance_cost : 500000
[Mon Mar 11 14:29:19 2024] rt_rq[44]:
[Mon Mar 11 14:29:19 2024] .rt_nr_running : 0
[Mon Mar 11 14:29:19 2024] .rt_nr_migratory : 0
[Mon Mar 11 14:29:19 2024] .rt_throttled : 0
[Mon Mar 11 14:29:19 2024] .rt_time : 0.000000
[Mon Mar 11 14:29:19 2024] .rt_runtime : 950.000000
[Mon Mar 11 14:29:19 2024] dl_rq[44]:
[Mon Mar 11 14:29:19 2024] .dl_nr_running : 0
[Mon Mar 11 14:29:19 2024] .dl_nr_migratory : 0
[Mon Mar 11 14:29:19 2024] .dl_bw->bw : 996147
[Mon Mar 11 14:29:19 2024] .dl_bw->total_bw : 0
[Mon Mar 11 14:29:19 2024] runnable tasks:
[Mon Mar 11 14:29:19 2024] S task PID tree-key switches prio wait-time sum-exec sum-sleep
[Mon Mar 11 14:29:19 2024] -------------------------------------------------------------------------------------------------------------
[Mon Mar 11 14:29:19 2024] S cpuhp/44 283 23515.869837 25 120 0.000000 0.828902 0.000000 0.000000 0 0 /
[Mon Mar 11 14:29:19 2024] S idle_inject/44 284 -12.000000 3 49 0.000000 0.000000 0.000000 0.000000 0 0 /
[Mon Mar 11 14:29:19 2024] S migration/44 285 2802.055998 367433 0 0.000000 1996.863011 0.000000 0.000000 0 0 /
[Mon Mar 11 14:29:19 2024] S ksoftirqd/44 286 42157696.977957 9077806 120 0.000000 217146.834413 0.000000 0.000000 0 0 /
[Mon Mar 11 14:29:19 2024] I kworker/44:0H 288 280.322767 4 100 0.000000 0.017773 0.000000 0.000000 0 0 /
[Mon Mar 11 14:29:19 2024] S oom_reaper 362 52.198947 2 120 0.000000 0.000000 0.000000 0.000000 0 0 /
[Mon Mar 11 14:29:19 2024] I blkcg_punt_bio 371 52.198947 2 100 0.000000 0.000000 0.000000 0.000000 0 0 /
[Mon Mar 11 14:29:19 2024] I ipv6_addrconf 407 88.735838 2 100 0.000000 0.047464 0.000000 0.000000 0 0 /
[Mon Mar 11 14:29:19 2024] I kworker/u99:0 461 181.140492 2 100 0.000000 0.011316 0.000000 0.000000 0 0 /
[Mon Mar 11 14:29:19 2024] I kworker/44:1H 641 42157696.977051 1535421 100 0.000000 44286.464054 0.000000 0.000000 0 0 /
[Mon Mar 11 14:29:19 2024] S scsi_eh_13 926 2745.566335 25 120 0.000000 3.029101 0.000000 0.000000 0 0 /
[Mon Mar 11 14:29:19 2024] I kdmflush/253:40 3731 24269.855143 2 100 0.000000 0.015202 0.000000 0.000000 0 0 /
[Mon Mar 11 14:29:19 2024] S jbd2/sda2-8 4207 41215155.025274 121 120 0.000000 3.186544 0.000000 0.000000 0 0 /
[Mon Mar 11 14:29:19 2024] D jbd2/dm-96-8 6491 42157696.983533 766787 120 0.000000 187933.894428 0.000000 0.000000 0 0 /
[Mon Mar 11 14:29:19 2024] D nfsd 8787 42157702.140609 74621 120 0.000000 8693.896547 0.000000 0.000000 0 0 /
[Mon Mar 11 14:29:19 2024] D nfsd 8820 42151736.333957 876842 120 0.000000 59665.234451 0.000000 0.000000 0 0 /
[Mon Mar 11 14:29:19 2024] D nfsd 8863 42062268.037734 42914126 120 0.000000 1666453.479846 0.000000 0.000000 0 0 /
[Mon Mar 11 14:29:19 2024] S smbd[10.87.30.2 185208 2791369.985906 33693182 120 0.000000 2608026.833978 0.000000 0.000000 0 185208 /autogroup-113
[Mon Mar 11 14:29:19 2024] S smbd[10.87.29.9 2364172 2791326.948755 33732 120 0.000000 6168.537741 0.000000 0.000000 0 0 /autogroup-113
[Mon Mar 11 14:29:19 2024] S smbd[10.87.30.4 2364895 2791264.056726 416 120 0.000000 133.751437 0.000000 0.000000 0 0 /autogroup-113
[Mon Mar 11 14:29:19 2024] S smbd[10.87.28.1 2398554 2791354.438690 754 120 0.000000 413.025715 0.000000 0.000000 0 0 /autogroup-113
[Mon Mar 11 14:29:19 2024] S smbd[10.87.19.5 2470233 2791356.283812 4581 120 0.000000 1117.537752 0.000000 0.000000 0 0 /autogroup-113
[Mon Mar 11 14:29:19 2024] I kworker/44:2 2472062 42150722.988707 46 120 0.000000 0.306183 0.000000 0.000000 0 0 /
[Mon Mar 11 14:29:19 2024] I kworker/44:0 2474523 42157697.011401 7945 120 0.000000 273.450846 0.000000 0.000000 0 0 /
[Mon Mar 11 14:29:19 2024] I kworker/44:1 2476853 42154640.702730 4 120 0.000000 0.019172 0.000000 0.000000 0 0 /
[Mon Mar 11 14:29:19 2024] S rpcd_spoolss 2478223 16746.774963 53 120 0.000000 75.279481 0.000000 0.000000 0 0 /autogroup-22820
[Mon Mar 11 14:29:19 2024] cpu#45, 2600.000 MHz
[Mon Mar 11 14:29:19 2024] .nr_running : 0
[Mon Mar 11 14:29:19 2024] .nr_switches : 834697063
[Mon Mar 11 14:29:19 2024] .nr_uninterruptible : -150
[Mon Mar 11 14:29:19 2024] .next_balance : 5577.334880
[Mon Mar 11 14:29:19 2024] .curr->pid : 0
[Mon Mar 11 14:29:19 2024] .clock : 1282633455.162037
[Mon Mar 11 14:29:19 2024] .clock_task : 1281324452.811200
[Mon Mar 11 14:29:19 2024] .avg_idle : 1000000
[Mon Mar 11 14:29:19 2024] .max_idle_balance_cost : 500000
[Mon Mar 11 14:29:19 2024] cfs_rq[45]:/
[Mon Mar 11 14:29:19 2024] .exec_clock : 0.000000
[Mon Mar 11 14:29:19 2024] .MIN_vruntime : 0.000001
[Mon Mar 11 14:29:19 2024] .min_vruntime : 16908831.887344
[Mon Mar 11 14:29:19 2024] .max_vruntime : 0.000001
[Mon Mar 11 14:29:19 2024] .spread : 0.000000
[Mon Mar 11 14:29:19 2024] .spread0 : -3508834.620771
[Mon Mar 11 14:29:19 2024] .nr_spread_over : 0
[Mon Mar 11 14:29:19 2024] .nr_running : 0
[Mon Mar 11 14:29:19 2024] .h_nr_running : 0
[Mon Mar 11 14:29:19 2024] .idle_nr_running : 0
[Mon Mar 11 14:29:19 2024] .idle_h_nr_running : 0
[Mon Mar 11 14:29:19 2024] .load : 0
[Mon Mar 11 14:29:19 2024] .load_avg : 1
[Mon Mar 11 14:29:19 2024] .runnable_avg : 0
[Mon Mar 11 14:29:19 2024] .util_avg : 0
[Mon Mar 11 14:29:19 2024] .util_est_enqueued : 0
[Mon Mar 11 14:29:19 2024] .removed.load_avg : 0
[Mon Mar 11 14:29:19 2024] .removed.util_avg : 0
[Mon Mar 11 14:29:19 2024] .removed.runnable_avg : 0
[Mon Mar 11 14:29:19 2024] .tg_load_avg_contrib : 0
[Mon Mar 11 14:29:19 2024] .tg_load_avg : 0
[Mon Mar 11 14:29:19 2024] .throttled : 0
[Mon Mar 11 14:29:19 2024] .throttle_count : 0
[Mon Mar 11 14:29:19 2024] rt_rq[45]:
[Mon Mar 11 14:29:19 2024] .rt_nr_running : 0
[Mon Mar 11 14:29:19 2024] .rt_nr_migratory : 0
[Mon Mar 11 14:29:19 2024] .rt_throttled : 0
[Mon Mar 11 14:29:19 2024] .rt_time : 0.000000
[Mon Mar 11 14:29:19 2024] .rt_runtime : 950.000000
[Mon Mar 11 14:29:19 2024] dl_rq[45]:
[Mon Mar 11 14:29:19 2024] .dl_nr_running : 0
[Mon Mar 11 14:29:19 2024] .dl_nr_migratory : 0
[Mon Mar 11 14:29:19 2024] .dl_bw->bw : 996147
[Mon Mar 11 14:29:19 2024] .dl_bw->total_bw : 0
[Mon Mar 11 14:29:19 2024] runnable tasks:
[Mon Mar 11 14:29:19 2024] S task PID tree-key switches prio wait-time sum-exec sum-sleep
[Mon Mar 11 14:29:19 2024] -------------------------------------------------------------------------------------------------------------
[Mon Mar 11 14:29:19 2024] S cpuhp/45 289 22697.043172 25 120 0.000000 0.696489 0.000000 0.000000 1 0 /
[Mon Mar 11 14:29:19 2024] S idle_inject/45 290 -12.000000 3 49 0.000000 0.000000 0.000000 0.000000 1 0 /
[Mon Mar 11 14:29:19 2024] S migration/45 291 2870.055998 323700 0 0.000000 1897.906226 0.000000 0.000000 1 0 /
[Mon Mar 11 14:29:19 2024] S ksoftirqd/45 292 16908428.714213 1687308 120 0.000000 56094.407051 0.000000 0.000000 1 0 /
[Mon Mar 11 14:29:19 2024] I kworker/45:0H 294 44.345188 4 100 0.000000 0.042061 0.000000 0.000000 1 0 /
[Mon Mar 11 14:29:19 2024] I kworker/45:1H 764 16908819.887511 824074 100 0.000000 29250.996644 0.000000 0.000000 1 0 /
[Mon Mar 11 14:29:19 2024] I nvme-wq 879 688.114634 2 100 0.000000 0.008472 0.000000 0.000000 1 0 /
[Mon Mar 11 14:29:19 2024] I kdmflush/253:46 3782 23634.542679 2 100 0.000000 0.012263 0.000000 0.000000 1 0 /
[Mon Mar 11 14:29:19 2024] I kdmflush/253:61 3919 23764.680184 2 100 0.000000 0.012188 0.000000 0.000000 1 0 /
[Mon Mar 11 14:29:19 2024] I kdmflush/253:72 4007 24047.673019 2 100 0.000000 0.010746 0.000000 0.000000 1 0 /
[Mon Mar 11 14:29:19 2024] S gmain 5162 11.034125 2 120 0.000000 0.082710 0.000000 0.000000 1 0 /autogroup-104
[Mon Mar 11 14:29:19 2024] S polkitd 5197 23.250636 4 120 0.000000 0.216518 0.000000 0.000000 1 0 /autogroup-104
[Mon Mar 11 14:29:19 2024] S writer#2 5330 482320.770660 55147407 120 0.000000 1458194.181012 0.000000 0.000000 1 5327 /autogroup-105
[Mon Mar 11 14:29:19 2024] S uwsgi 5239 4819.060713 643 120 0.000000 1283.521483 0.000000 0.000000 1 0 /autogroup-90
[Mon Mar 11 14:29:19 2024] D nfsd 8789 16908790.199547 78181 120 0.000000 8981.738658 0.000000 0.000000 1 0 /
[Mon Mar 11 14:29:19 2024] D nfsd 8848 16845783.144976 8259682 120 0.000000 342138.812648 0.000000 0.000000 1 0 /
[Mon Mar 11 14:29:19 2024] D nfsd 8867 16830172.894740 125301912 120 0.000000 4135727.762149 0.000000 0.000000 1 0 /
[Mon Mar 11 14:29:19 2024] S automount 88130 1599.204683 23 120 0.000000 3.557280 0.000000 0.000000 1 0 /autogroup-222
[Mon Mar 11 14:29:19 2024] S smbd[10.87.19.5 978979 848284.387452 5378 120 0.000000 1452.966166 0.000000 0.000000 1 0 /autogroup-113
[Mon Mar 11 14:29:19 2024] S smbd[10.87.29.1 2297566 848296.192180 9526 120 0.000000 11178.141828 0.000000 0.000000 1 0 /autogroup-113
[Mon Mar 11 14:29:19 2024] S smbd[10.33.88.4 2322831 848280.165421 12619 120 0.000000 3106.191462 0.000000 0.000000 1 0 /autogroup-113
[Mon Mar 11 14:29:19 2024] S smbd[10.87.30.4 2364892 848235.060798 775 120 0.000000 286.880544 0.000000 0.000000 1 0 /autogroup-113
[Mon Mar 11 14:29:19 2024] S smbd[10.40.142. 2381864 848279.466223 510 120 0.000000 197.729459 0.000000 0.000000 1 0 /autogroup-113
[Mon Mar 11 14:29:19 2024] I kworker/45:0 2385965 16908819.681044 207637 120 0.000000 7487.489092 0.000000 0.000000 1 0 /
[Mon Mar 11 14:29:19 2024] S smbd[10.87.29.1 2440119 848279.477097 177 120 0.000000 75.190257 0.000000 0.000000 1 0 /autogroup-113
[Mon Mar 11 14:29:19 2024] I kworker/45:2 2473358 16896354.894791 5 120 0.000000 0.043559 0.000000 0.000000 1 0 /
[Mon Mar 11 14:29:19 2024] I kworker/u98:1 2473403 16898740.925324 69077 120 0.000000 4976.446729 0.000000 0.000000 1 0 /
[Mon Mar 11 14:29:19 2024] I kworker/45:1 2476033 16905248.134675 8 120 0.000000 0.054582 0.000000 0.000000 1 0 /
[Mon Mar 11 14:29:19 2024] cpu#46, 2600.000 MHz
[Mon Mar 11 14:29:19 2024] .nr_running : 0
[Mon Mar 11 14:29:19 2024] .nr_switches : 2717295473
[Mon Mar 11 14:29:19 2024] .nr_uninterruptible : 37982
[Mon Mar 11 14:29:19 2024] .next_balance : 5577.334877
[Mon Mar 11 14:29:19 2024] .curr->pid : 0
[Mon Mar 11 14:29:19 2024] .clock : 1282633455.161381
[Mon Mar 11 14:29:19 2024] .clock_task : 1263601685.171145
[Mon Mar 11 14:29:19 2024] .avg_idle : 1000000
[Mon Mar 11 14:29:19 2024] .max_idle_balance_cost : 500000
[Mon Mar 11 14:29:19 2024] cfs_rq[46]:/
[Mon Mar 11 14:29:19 2024] .exec_clock : 0.000000
[Mon Mar 11 14:29:19 2024] .MIN_vruntime : 0.000001
[Mon Mar 11 14:29:19 2024] .min_vruntime : 40443164.714160
[Mon Mar 11 14:29:19 2024] .max_vruntime : 0.000001
[Mon Mar 11 14:29:19 2024] .spread : 0.000000
[Mon Mar 11 14:29:19 2024] .spread0 : 20025498.206045
[Mon Mar 11 14:29:19 2024] .nr_spread_over : 0
[Mon Mar 11 14:29:19 2024] .nr_running : 0
[Mon Mar 11 14:29:19 2024] .h_nr_running : 0
[Mon Mar 11 14:29:19 2024] .idle_nr_running : 0
[Mon Mar 11 14:29:19 2024] .idle_h_nr_running : 0
[Mon Mar 11 14:29:19 2024] .load : 0
[Mon Mar 11 14:29:19 2024] .load_avg : 0
[Mon Mar 11 14:29:19 2024] .runnable_avg : 0
[Mon Mar 11 14:29:19 2024] .util_avg : 0
[Mon Mar 11 14:29:19 2024] .util_est_enqueued : 0
[Mon Mar 11 14:29:19 2024] .removed.load_avg : 0
[Mon Mar 11 14:29:19 2024] .removed.util_avg : 0
[Mon Mar 11 14:29:19 2024] .removed.runnable_avg : 0
[Mon Mar 11 14:29:19 2024] .tg_load_avg_contrib : 0
[Mon Mar 11 14:29:19 2024] .tg_load_avg : 0
[Mon Mar 11 14:29:19 2024] .throttled : 0
[Mon Mar 11 14:29:19 2024] .throttle_count : 0
[Mon Mar 11 14:29:19 2024] rt_rq[46]:
[Mon Mar 11 14:29:19 2024] .rt_nr_running : 0
[Mon Mar 11 14:29:19 2024] .rt_nr_migratory : 0
[Mon Mar 11 14:29:19 2024] .rt_throttled : 0
[Mon Mar 11 14:29:19 2024] .rt_time : 0.000000
[Mon Mar 11 14:29:19 2024] .rt_runtime : 950.000000
[Mon Mar 11 14:29:19 2024] dl_rq[46]:
[Mon Mar 11 14:29:19 2024] .dl_nr_running : 0
[Mon Mar 11 14:29:19 2024] .dl_nr_migratory : 0
[Mon Mar 11 14:29:19 2024] .dl_bw->bw : 996147
[Mon Mar 11 14:29:19 2024] .dl_bw->total_bw : 0
[Mon Mar 11 14:29:19 2024] runnable tasks:
[Mon Mar 11 14:29:19 2024] S task PID tree-key switches prio wait-time sum-exec sum-sleep
[Mon Mar 11 14:29:19 2024] -------------------------------------------------------------------------------------------------------------
[Mon Mar 11 14:29:19 2024] S cpuhp/46 295 23217.842829 25 120 0.000000 0.748679 0.000000 0.000000 0 0 /
[Mon Mar 11 14:29:19 2024] S idle_inject/46 296 -11.996217 3 49 0.000000 0.010193 0.000000 0.000000 0 0 /
[Mon Mar 11 14:29:19 2024] S migration/46 297 0.000000 364531 0 0.000000 1954.429126 0.000000 0.000000 0 0 /
[Mon Mar 11 14:29:19 2024] S ksoftirqd/46 298 40442964.446823 8047609 120 0.000000 201628.273289 0.000000 0.000000 0 0 /
[Mon Mar 11 14:29:19 2024] I kworker/46:0H 300 365.751900 4 100 0.000000 0.050581 0.000000 0.000000 0 0 /
[Mon Mar 11 14:29:19 2024] I zswap-shrink 460 239.007746 2 100 0.000000 0.013230 0.000000 0.000000 0 0 /
[Mon Mar 11 14:29:19 2024] I kworker/46:1H 721 40443152.714483 1518121 100 0.000000 44149.622712 0.000000 0.000000 0 0 /
[Mon Mar 11 14:29:19 2024] S nrpe 4740 168196.970426 114215 120 0.000000 25818.707718 0.000000 0.000000 0 0 /autogroup-109
[Mon Mar 11 14:29:19 2024] D nfsd 8809 40443140.629874 157349 120 0.000000 15910.209360 0.000000 0.000000 0 0 /
[Mon Mar 11 14:29:19 2024] D nfsd 8837 40391590.232658 2132154 120 0.000000 114007.722388 0.000000 0.000000 0 0 /
[Mon Mar 11 14:29:19 2024] I kworker/46:1 2050654 40443138.236066 808697 120 0.000000 25467.045857 0.000000 0.000000 0 0 /
[Mon Mar 11 14:29:19 2024] S smbd[10.87.27.3 2318500 2482567.304916 969 120 0.000000 349.279908 0.000000 0.000000 0 0 /autogroup-113
[Mon Mar 11 14:29:19 2024] S smbd[10.87.28.1 2323597 2482627.945363 569 120 0.000000 180.530759 0.000000 0.000000 0 0 /autogroup-113
[Mon Mar 11 14:29:19 2024] S smbd[10.87.28.1 2340410 2482584.037614 476 120 0.000000 146.005053 0.000000 0.000000 0 0 /autogroup-113
[Mon Mar 11 14:29:19 2024] I kworker/46:2 2472151 40426263.236097 12 120 0.000000 0.046309 0.000000 0.000000 0 0 /
[Mon Mar 11 14:29:19 2024] I kworker/46:0 2475460 40438475.372011 7 120 0.000000 0.086767 0.000000 0.000000 0 0 /
[Mon Mar 11 14:29:19 2024] cpu#47, 2600.000 MHz
[Mon Mar 11 14:29:19 2024] .nr_running : 1
[Mon Mar 11 14:29:19 2024] .nr_switches : 838026468
[Mon Mar 11 14:29:19 2024] .nr_uninterruptible : -632
[Mon Mar 11 14:29:19 2024] .next_balance : 5577.334932
[Mon Mar 11 14:29:19 2024] .curr->pid : 17
[Mon Mar 11 14:29:19 2024] .clock : 1282633455.160009
[Mon Mar 11 14:29:19 2024] .clock_task : 1281281464.096361
[Mon Mar 11 14:29:19 2024] .avg_idle : 692134
[Mon Mar 11 14:29:19 2024] .max_idle_balance_cost : 500000
[Mon Mar 11 14:29:19 2024] cfs_rq[47]:/
[Mon Mar 11 14:29:19 2024] .exec_clock : 0.000000
[Mon Mar 11 14:29:19 2024] .MIN_vruntime : 0.000001
[Mon Mar 11 14:29:19 2024] .min_vruntime : 17562030.542460
[Mon Mar 11 14:29:19 2024] .max_vruntime : 0.000001
[Mon Mar 11 14:29:19 2024] .spread : 0.000000
[Mon Mar 11 14:29:19 2024] .spread0 : -2855635.965655
[Mon Mar 11 14:29:19 2024] .nr_spread_over : 0
[Mon Mar 11 14:29:19 2024] .nr_running : 1
[Mon Mar 11 14:29:19 2024] .h_nr_running : 1
[Mon Mar 11 14:29:19 2024] .idle_nr_running : 0
[Mon Mar 11 14:29:19 2024] .idle_h_nr_running : 0
[Mon Mar 11 14:29:19 2024] .load : 1048576
[Mon Mar 11 14:29:19 2024] .load_avg : 41
[Mon Mar 11 14:29:19 2024] .runnable_avg : 41
[Mon Mar 11 14:29:19 2024] .util_avg : 41
[Mon Mar 11 14:29:19 2024] .util_est_enqueued : 27
[Mon Mar 11 14:29:19 2024] .removed.load_avg : 0
[Mon Mar 11 14:29:19 2024] .removed.util_avg : 0
[Mon Mar 11 14:29:19 2024] .removed.runnable_avg : 0
[Mon Mar 11 14:29:19 2024] .tg_load_avg_contrib : 0
[Mon Mar 11 14:29:19 2024] .tg_load_avg : 0
[Mon Mar 11 14:29:19 2024] .throttled : 0
[Mon Mar 11 14:29:19 2024] .throttle_count : 0
[Mon Mar 11 14:29:19 2024] rt_rq[47]:
[Mon Mar 11 14:29:19 2024] .rt_nr_running : 0
[Mon Mar 11 14:29:19 2024] .rt_nr_migratory : 0
[Mon Mar 11 14:29:19 2024] .rt_throttled : 0
[Mon Mar 11 14:29:19 2024] .rt_time : 0.024833
[Mon Mar 11 14:29:19 2024] .rt_runtime : 950.000000
[Mon Mar 11 14:29:19 2024] dl_rq[47]:
[Mon Mar 11 14:29:19 2024] .dl_nr_running : 0
[Mon Mar 11 14:29:19 2024] .dl_nr_migratory : 0
[Mon Mar 11 14:29:19 2024] .dl_bw->bw : 996147
[Mon Mar 11 14:29:19 2024] .dl_bw->total_bw : 0
[Mon Mar 11 14:29:19 2024] runnable tasks:
[Mon Mar 11 14:29:19 2024] S task PID tree-key switches prio wait-time sum-exec sum-sleep
[Mon Mar 11 14:29:19 2024] -------------------------------------------------------------------------------------------------------------
[Mon Mar 11 14:29:19 2024] >R pr/tty0 17 17562025.446738 6172008 120 0.000000 55561.107622 0.000000 0.000000 1 0 /
[Mon Mar 11 14:29:19 2024] S cpuhp/47 301 24226.751130 25 120 0.000000 0.799915 0.000000 0.000000 1 0 /
[Mon Mar 11 14:29:19 2024] S idle_inject/47 302 -12.000000 3 49 0.000000 0.000000 0.000000 0.000000 1 0 /
[Mon Mar 11 14:29:19 2024] S migration/47 303 3006.062858 323084 0 0.000000 2026.298100 0.000000 0.000000 1 0 /
[Mon Mar 11 14:29:19 2024] S ksoftirqd/47 304 17561685.512870 1652852 120 0.000000 54349.339133 0.000000 0.000000 1 0 /
[Mon Mar 11 14:29:19 2024] I kworker/47:0H 306 152.594739 4 100 0.000000 0.029461 0.000000 0.000000 1 0 /
[Mon Mar 11 14:29:19 2024] I kworker/47:1H 749 17562018.542556 1107336 100 0.000000 38223.315965 0.000000 0.000000 1 0 /
[Mon Mar 11 14:29:19 2024] S jbd2/dm-0-8 2716 17562018.765148 1369629 120 0.000000 59603.923972 0.000000 0.000000 1 0 /
[Mon Mar 11 14:29:19 2024] S multipathd 3350 0.000000 2212278 0 0.000000 46065.475878 0.000000 0.000000 1 0 /autogroup-45
[Mon Mar 11 14:29:19 2024] S tuned 4413 14011.121626 1302257 120 0.000000 128009.012523 0.000000 0.000000 1 4413 /autogroup-93
[Mon Mar 11 14:29:19 2024] S tuned 4666 14009.725811 1304993 120 0.000000 123279.129785 0.000000 0.000000 1 4666 /autogroup-93
[Mon Mar 11 14:29:19 2024] S writer#4 5332 577861.987132 55260970 120 0.000000 1455707.168755 0.000000 0.000000 1 5327 /autogroup-105
[Mon Mar 11 14:29:19 2024] S qmgr 5126 1961.675670 10678 120 0.000000 3052.668226 0.000000 0.000000 1 0 /autogroup-115
[Mon Mar 11 14:29:19 2024] I kdmflush/253:10 8699 25697.008861 2 100 0.000000 0.070436 0.000000 0.000000 1 0 /
[Mon Mar 11 14:29:19 2024] D nfsd 8829 17516468.836694 347892 120 0.000000 28554.780021 0.000000 0.000000 1 0 /
[Mon Mar 11 14:29:19 2024] S automount 88143 2104.600765 27 120 0.000000 2.853638 0.000000 0.000000 1 0 /autogroup-222
[Mon Mar 11 14:29:19 2024] S smbd[10.87.29.8 2353738 867963.285101 3377 120 0.000000 846.773417 0.000000 0.000000 1 0 /autogroup-113
[Mon Mar 11 14:29:19 2024] S pickup 2449277 1961.409240 73 120 0.000000 37.307330 0.000000 0.000000 1 0 /autogroup-115
[Mon Mar 11 14:29:19 2024] I kworker/47:2 2452570 17562018.593685 49418 120 0.000000 3118.221796 0.000000 0.000000 1 0 /
[Mon Mar 11 14:29:19 2024] S smbd[10.87.28.1 2457742 867939.476315 118 120 0.000000 68.375658 0.000000 0.000000 1 0 /autogroup-113
[Mon Mar 11 14:29:19 2024] I kworker/u98:3 2461839 17554369.712742 171580 120 0.000000 5403.343331 0.000000 0.000000 1 0 /
[Mon Mar 11 14:29:19 2024] I kworker/47:1 2474397 17556065.443165 9 120 0.000000 0.066801 0.000000 0.000000 1 0 /
[Mon Mar 11 14:29:19 2024] I kworker/47:0 2476852 17556096.226761 3 120 0.000000 0.019199 0.000000 0.000000 1 0 /
[Mon Mar 11 14:29:19 2024] Showing busy workqueues and worker pools:
[Mon Mar 11 14:29:19 2024] workqueue events: flags=0x0
[Mon Mar 11 14:29:19 2024] pwq 60: cpus=30 node=0 flags=0x0 nice=0 active=2/256 refcnt=3
[Mon Mar 11 14:29:19 2024] in-flight: 2364267:delayed_fput delayed_fput
[Mon Mar 11 14:29:19 2024] pwq 28: cpus=14 node=0 flags=0x0 nice=0 active=1/256 refcnt=2
[Mon Mar 11 14:29:19 2024] pending: drm_fb_helper_damage_work [drm_kms_helper]
[Mon Mar 11 14:29:19 2024] pwq 22: cpus=11 node=1 flags=0x0 nice=0 active=1/256 refcnt=2
[Mon Mar 11 14:29:19 2024] pending: kfree_rcu_monitor
[Mon Mar 11 14:29:19 2024] workqueue events_power_efficient: flags=0x80
[Mon Mar 11 14:29:19 2024] pwq 28: cpus=14 node=0 flags=0x0 nice=0 active=1/256 refcnt=2
[Mon Mar 11 14:29:19 2024] pending: fb_flashcursor
[Mon Mar 11 14:29:19 2024] workqueue mm_percpu_wq: flags=0x8
[Mon Mar 11 14:29:19 2024] pwq 28: cpus=14 node=0 flags=0x0 nice=0 active=2/256 refcnt=3
[Mon Mar 11 14:29:19 2024] pending: vmstat_update, lru_add_drain_per_cpu
[Mon Mar 11 14:29:19 2024] pwq 22: cpus=11 node=1 flags=0x0 nice=0 active=2/256 refcnt=4
[Mon Mar 11 14:29:19 2024] pending: vmstat_update, lru_add_drain_per_cpu BAR(367)
[Mon Mar 11 14:29:19 2024] workqueue writeback: flags=0x4a
[Mon Mar 11 14:29:19 2024] pwq 98: cpus=1,3,5,7,9,11,13,15,17,19,21,23,25,27,29,31,33,35,37,39,41,43,45,47 node=1 flags=0x4 nice=0 active=1/256 refcnt=2
[Mon Mar 11 14:29:19 2024] in-flight: 2476491:wb_workfn
[Mon Mar 11 14:29:19 2024] workqueue kblockd: flags=0x18
[Mon Mar 11 14:29:19 2024] pwq 29: cpus=14 node=0 flags=0x0 nice=-20 active=1/256 refcnt=2
[Mon Mar 11 14:29:19 2024] pending: blk_mq_timeout_work
[Mon Mar 11 14:29:19 2024] workqueue lpfc_wq: flags=0x8
[Mon Mar 11 14:29:19 2024] pwq 28: cpus=14 node=0 flags=0x0 nice=0 active=1/256 refcnt=2
[Mon Mar 11 14:29:19 2024] pending: lpfc_sli4_hba_process_cq [lpfc]
[Mon Mar 11 14:29:19 2024] workqueue lpfc_wq: flags=0x8
[Mon Mar 11 14:29:19 2024] pwq 28: cpus=14 node=0 flags=0x0 nice=0 active=1/256 refcnt=2
[Mon Mar 11 14:29:19 2024] pending: lpfc_sli4_hba_process_cq [lpfc]
[Mon Mar 11 14:29:19 2024] workqueue nfsd4: flags=0x2
[Mon Mar 11 14:29:19 2024] pwq 98: cpus=1,3,5,7,9,11,13,15,17,19,21,23,25,27,29,31,33,35,37,39,41,43,45,47 node=1 flags=0x4 nice=0 active=1/256 refcnt=2
[Mon Mar 11 14:29:19 2024] in-flight: 2468089:laundromat_main [nfsd]
[Mon Mar 11 14:29:19 2024] workqueue nfsd4_callbacks: flags=0xa0002
[Mon Mar 11 14:29:19 2024] pwq 96: cpus=0-47 flags=0x4 nice=0 active=1/1 refcnt=326
[Mon Mar 11 14:29:19 2024] in-flight: 2451130:nfsd4_run_cb_work [nfsd]
[Mon Mar 11 14:29:19 2024] inactive: nfsd4_run_cb_work [nfsd], nfsd4_run_cb_work [nfsd], nfsd4_run_cb_work [nfsd], nfsd4_run_cb_work [nfsd], nfsd4_run_cb_work [nfsd], nfsd4_run_cb_work [nfsd], nfsd4_run_cb_work [nfsd], nfsd4_run_cb_work [nfsd], nfsd4_run_cb_work [nfsd], nfsd4_run_cb_work [nfsd], nfsd4_run_cb_work [nfsd], nfsd4_run_cb_work [nfsd], nfsd4_run_cb_work [nfsd], nfsd4_run_cb_work [nfsd], nfsd4_run_cb_work [nfsd], nfsd4_run_cb_work [nfsd], nfsd4_run_cb_work [nfsd], nfsd4_run_cb_work [nfsd], nfsd4_run_cb_work [nfsd], nfsd4_run_cb_work [nfsd], nfsd4_run_cb_work [nfsd], nfsd4_run_cb_work [nfsd], nfsd4_run_cb_work [nfsd], nfsd4_run_cb_work [nfsd], nfsd4_run_cb_work [nfsd], nfsd4_run_cb_work [nfsd], nfsd4_run_cb_work [nfsd], nfsd4_run_cb_work [nfsd], nfsd4_run_cb_work [nfsd], nfsd4_run_cb_work [nfsd], nfsd4_run_cb_work [nfsd], nfsd4_run_cb_work [nfsd], nfsd4_run_cb_work [nfsd], nfsd4_run_cb_work [nfsd], nfsd4_run_cb_work [nfsd], nfsd4_run_cb_work [nfsd], nfsd4_run_cb_work [nfsd]
[Mon Mar 11 14:29:19 2024] , nfsd4_run_cb_work [nfsd], nfsd4_run_cb_work [nfsd], nfsd4_run_cb_work [nfsd], nfsd4_run_cb_work [nfsd], nfsd4_run_cb_work [nfsd], nfsd4_run_cb_work [nfsd], nfsd4_run_cb_work [nfsd], nfsd4_run_cb_work [nfsd], nfsd4_run_cb_work [nfsd], nfsd4_run_cb_work [nfsd], nfsd4_run_cb_work [nfsd], nfsd4_run_cb_work [nfsd], nfsd4_run_cb_work [nfsd], nfsd4_run_cb_work [nfsd], nfsd4_run_cb_work [nfsd], nfsd4_run_cb_work [nfsd], nfsd4_run_cb_work [nfsd], nfsd4_run_cb_work [nfsd], nfsd4_run_cb_work [nfsd], nfsd4_run_cb_work [nfsd], nfsd4_run_cb_work [nfsd], nfsd4_run_cb_work [nfsd], nfsd4_run_cb_work [nfsd], nfsd4_run_cb_work [nfsd], nfsd4_run_cb_work [nfsd], nfsd4_run_cb_work [nfsd], nfsd4_run_cb_work [nfsd], nfsd4_run_cb_work [nfsd], nfsd4_run_cb_work [nfsd], nfsd4_run_cb_work [nfsd], nfsd4_run_cb_work [nfsd], nfsd4_run_cb_work [nfsd], nfsd4_run_cb_work [nfsd], nfsd4_run_cb_work [nfsd], nfsd4_run_cb_work [nfsd], nfsd4_run_cb_work [nfsd], nfsd4_run_cb_work [nfsd], nfsd4_run_cb_work [nfsd]
[Mon Mar 11 14:29:19 2024] , nfsd4_run_cb_work [nfsd], nfsd4_run_cb_work [nfsd], nfsd4_run_cb_work [nfsd], nfsd4_run_cb_work [nfsd], nfsd4_run_cb_work [nfsd], nfsd4_run_cb_work [nfsd], nfsd4_run_cb_work [nfsd], nfsd4_run_cb_work [nfsd], nfsd4_run_cb_work [nfsd], nfsd4_run_cb_work [nfsd], nfsd4_run_cb_work [nfsd], nfsd4_run_cb_work [nfsd], nfsd4_run_cb_work [nfsd], nfsd4_run_cb_work [nfsd], nfsd4_run_cb_work [nfsd], nfsd4_run_cb_work [nfsd], nfsd4_run_cb_work [nfsd], nfsd4_run_cb_work [nfsd], nfsd4_run_cb_work [nfsd], nfsd4_run_cb_work [nfsd], nfsd4_run_cb_work [nfsd], nfsd4_run_cb_work [nfsd], nfsd4_run_cb_work [nfsd], nfsd4_run_cb_work [nfsd], nfsd4_run_cb_work [nfsd], nfsd4_run_cb_work [nfsd], nfsd4_run_cb_work [nfsd], nfsd4_run_cb_work [nfsd], nfsd4_run_cb_work [nfsd], nfsd4_run_cb_work [nfsd], nfsd4_run_cb_work [nfsd], nfsd4_run_cb_work [nfsd], nfsd4_run_cb_work [nfsd], nfsd4_run_cb_work [nfsd], nfsd4_run_cb_work [nfsd], nfsd4_run_cb_work [nfsd], nfsd4_run_cb_work [nfsd], nfsd4_run_cb_work [nfsd]
[Mon Mar 11 14:29:19 2024] , nfsd4_run_cb_work [nfsd], nfsd4_run_cb_work [nfsd], nfsd4_run_cb_work [nfsd], nfsd4_run_cb_work [nfsd], nfsd4_run_cb_work [nfsd], nfsd4_run_cb_work [nfsd], nfsd4_run_cb_work [nfsd], nfsd4_run_cb_work [nfsd], nfsd4_run_cb_work [nfsd], nfsd4_run_cb_work [nfsd], nfsd4_run_cb_work [nfsd], nfsd4_run_cb_work [nfsd], nfsd4_run_cb_work [nfsd], nfsd4_run_cb_work [nfsd], nfsd4_run_cb_work [nfsd], nfsd4_run_cb_work [nfsd], nfsd4_run_cb_work [nfsd], nfsd4_run_cb_work [nfsd], nfsd4_run_cb_work [nfsd], nfsd4_run_cb_work [nfsd], nfsd4_run_cb_work [nfsd], nfsd4_run_cb_work [nfsd], nfsd4_run_cb_work [nfsd], nfsd4_run_cb_work [nfsd], nfsd4_run_cb_work [nfsd], nfsd4_run_cb_work [nfsd], nfsd4_run_cb_work [nfsd], nfsd4_run_cb_work [nfsd], nfsd4_run_cb_work [nfsd], nfsd4_run_cb_work [nfsd], nfsd4_run_cb_work [nfsd], nfsd4_run_cb_work [nfsd], nfsd4_run_cb_work [nfsd], nfsd4_run_cb_work [nfsd], nfsd4_run_cb_work [nfsd], nfsd4_run_cb_work [nfsd], nfsd4_run_cb_work [nfsd], nfsd4_run_cb_work [nfsd]
[Mon Mar 11 14:29:19 2024] , nfsd4_run_cb_work [nfsd], nfsd4_run_cb_work [nfsd], nfsd4_run_cb_work [nfsd], nfsd4_run_cb_work [nfsd], nfsd4_run_cb_work [nfsd], nfsd4_run_cb_work [nfsd], nfsd4_run_cb_work [nfsd], nfsd4_run_cb_work [nfsd], nfsd4_run_cb_work [nfsd], nfsd4_run_cb_work [nfsd], nfsd4_run_cb_work [nfsd], nfsd4_run_cb_work [nfsd], nfsd4_run_cb_work [nfsd], nfsd4_run_cb_work [nfsd], nfsd4_run_cb_work [nfsd], nfsd4_run_cb_work [nfsd], nfsd4_run_cb_work [nfsd], nfsd4_run_cb_work [nfsd], nfsd4_run_cb_work [nfsd], nfsd4_run_cb_work [nfsd], nfsd4_run_cb_work [nfsd], nfsd4_run_cb_work [nfsd], nfsd4_run_cb_work [nfsd], nfsd4_run_cb_work [nfsd], nfsd4_run_cb_work [nfsd], nfsd4_run_cb_work [nfsd], nfsd4_run_cb_work [nfsd], nfsd4_run_cb_work [nfsd], nfsd4_run_cb_work [nfsd], nfsd4_run_cb_work [nfsd], nfsd4_run_cb_work [nfsd], nfsd4_run_cb_work [nfsd], nfsd4_run_cb_work [nfsd], nfsd4_run_cb_work [nfsd], nfsd4_run_cb_work [nfsd], nfsd4_run_cb_work [nfsd], nfsd4_run_cb_work [nfsd], nfsd4_run_cb_work [nfsd]
[Mon Mar 11 14:29:19 2024] , nfsd4_run_cb_work [nfsd], nfsd4_run_cb_work [nfsd], nfsd4_run_cb_work [nfsd], nfsd4_run_cb_work [nfsd], nfsd4_run_cb_work [nfsd], nfsd4_run_cb_work [nfsd], nfsd4_run_cb_work [nfsd], nfsd4_run_cb_work [nfsd], nfsd4_run_cb_work [nfsd], nfsd4_run_cb_work [nfsd], nfsd4_run_cb_work [nfsd], nfsd4_run_cb_work [nfsd], nfsd4_run_cb_work [nfsd], nfsd4_run_cb_work [nfsd], nfsd4_run_cb_work [nfsd], nfsd4_run_cb_work [nfsd], nfsd4_run_cb_work [nfsd], nfsd4_run_cb_work [nfsd], nfsd4_run_cb_work [nfsd], nfsd4_run_cb_work [nfsd], nfsd4_run_cb_work [nfsd], nfsd4_run_cb_work [nfsd], nfsd4_run_cb_work [nfsd], nfsd4_run_cb_work [nfsd], nfsd4_run_cb_work [nfsd], nfsd4_run_cb_work [nfsd], nfsd4_run_cb_work [nfsd], nfsd4_run_cb_work [nfsd], nfsd4_run_cb_work [nfsd], nfsd4_run_cb_work [nfsd], nfsd4_run_cb_work [nfsd], nfsd4_run_cb_work [nfsd], nfsd4_run_cb_work [nfsd], nfsd4_run_cb_work [nfsd], nfsd4_run_cb_work [nfsd], nfsd4_run_cb_work [nfsd], nfsd4_run_cb_work [nfsd], nfsd4_run_cb_work [nfsd]
[Mon Mar 11 14:29:19 2024] , nfsd4_run_cb_work [nfsd], nfsd4_run_cb_work [nfsd], nfsd4_run_cb_work [nfsd], nfsd4_run_cb_work [nfsd], nfsd4_run_cb_work [nfsd], nfsd4_run_cb_work [nfsd], nfsd4_run_cb_work [nfsd], nfsd4_run_cb_work [nfsd], nfsd4_run_cb_work [nfsd], nfsd4_run_cb_work [nfsd], nfsd4_run_cb_work [nfsd], nfsd4_run_cb_work [nfsd], nfsd4_run_cb_work [nfsd], nfsd4_run_cb_work [nfsd], nfsd4_run_cb_work [nfsd], nfsd4_run_cb_work [nfsd], nfsd4_run_cb_work [nfsd], nfsd4_run_cb_work [nfsd], nfsd4_run_cb_work [nfsd], nfsd4_run_cb_work [nfsd], nfsd4_run_cb_work [nfsd], nfsd4_run_cb_work [nfsd], nfsd4_run_cb_work [nfsd], nfsd4_run_cb_work [nfsd], nfsd4_run_cb_work [nfsd], nfsd4_run_cb_work [nfsd], nfsd4_run_cb_work [nfsd], nfsd4_run_cb_work [nfsd], nfsd4_run_cb_work [nfsd], nfsd4_run_cb_work [nfsd], nfsd4_run_cb_work [nfsd], nfsd4_run_cb_work [nfsd], nfsd4_run_cb_work [nfsd], nfsd4_run_cb_work [nfsd], nfsd4_run_cb_work [nfsd], nfsd4_run_cb_work [nfsd], nfsd4_run_cb_work [nfsd], nfsd4_run_cb_work [nfsd]
[Mon Mar 11 14:29:19 2024] , nfsd4_run_cb_work [nfsd], nfsd4_run_cb_work [nfsd], nfsd4_run_cb_work [nfsd], nfsd4_run_cb_work [nfsd], nfsd4_run_cb_work [nfsd], nfsd4_run_cb_work [nfsd], nfsd4_run_cb_work [nfsd], nfsd4_run_cb_work [nfsd], nfsd4_run_cb_work [nfsd], nfsd4_run_cb_work [nfsd], nfsd4_run_cb_work [nfsd], nfsd4_run_cb_work [nfsd], nfsd4_run_cb_work [nfsd], nfsd4_run_cb_work [nfsd], nfsd4_run_cb_work [nfsd], nfsd4_run_cb_work [nfsd], nfsd4_run_cb_work [nfsd], nfsd4_run_cb_work [nfsd], nfsd4_run_cb_work [nfsd], nfsd4_run_cb_work [nfsd], nfsd4_run_cb_work [nfsd], nfsd4_run_cb_work [nfsd], nfsd4_run_cb_work [nfsd], nfsd4_run_cb_work [nfsd], nfsd4_run_cb_work [nfsd], nfsd4_run_cb_work [nfsd], nfsd4_run_cb_work [nfsd], nfsd4_run_cb_work [nfsd], nfsd4_run_cb_work [nfsd], nfsd4_run_cb_work [nfsd], nfsd4_run_cb_work [nfsd], nfsd4_run_cb_work [nfsd], nfsd4_run_cb_work [nfsd], nfsd4_run_cb_work [nfsd], nfsd4_run_cb_work [nfsd], nfsd4_run_cb_work [nfsd], nfsd4_run_cb_work [nfsd], nfsd4_run_cb_work [nfsd]
[Mon Mar 11 14:29:19 2024] , nfsd4_run_cb_work [nfsd], nfsd4_run_cb_work [nfsd], nfsd4_run_cb_work [nfsd], nfsd4_run_cb_work [nfsd], nfsd4_run_cb_work [nfsd], nfsd4_run_cb_work [nfsd], nfsd4_run_cb_work [nfsd], nfsd4_run_cb_work [nfsd], nfsd4_run_cb_work [nfsd], nfsd4_run_cb_work [nfsd], nfsd4_run_cb_work [nfsd], nfsd4_run_cb_work [nfsd], nfsd4_run_cb_work [nfsd], nfsd4_run_cb_work [nfsd], nfsd4_run_cb_work [nfsd], nfsd4_run_cb_work [nfsd], nfsd4_run_cb_work [nfsd], nfsd4_run_cb_work [nfsd], nfsd4_run_cb_work [nfsd]
[Mon Mar 11 14:29:19 2024] pool 60: cpus=30 node=0 flags=0x0 nice=0 hung=1s workers=3 idle: 2475882 2473379
[Mon Mar 11 14:29:19 2024] pool 96: cpus=0-47 flags=0x4 nice=0 hung=0s workers=6 idle: 2466033 2442472 2472989 2470993 2466032
[Mon Mar 11 14:29:19 2024] pool 98: cpus=1,3,5,7,9,11,13,15,17,19,21,23,25,27,29,31,33,35,37,39,41,43,45,47 node=1 flags=0x4 nice=0 hung=0s workers=8 idle: 2463319 2463433 2464235 2473403 2471807 2461839
[-- Attachment #3: rpc_tasks.txt --]
[-- Type: text/plain, Size: 111 bytes --]
18423 2281 0 0x18 0x0 1354 nfsd4_cb_ops [nfsd] nfs4_cbv1 CB_RECALL_ANY a:call_start [sunrpc] q:delayq
[-- Attachment #4: nfs_threads.txt --]
[-- Type: text/plain, Size: 919 bytes --]
/proc/2002/stack:
[<0>] nfs_wait_client_init_complete.part.12+0x3b/0x90 [nfs]
[<0>] nfs41_discover_server_trunking+0x61/0xa0 [nfsv4]
[<0>] nfs4_discover_server_trunking+0x72/0x240 [nfsv4]
[<0>] nfs4_init_client+0xbd/0x150 [nfsv4]
[<0>] nfs4_set_client+0xed/0x150 [nfsv4]
[<0>] nfs4_create_server+0x127/0x2c0 [nfsv4]
[<0>] nfs4_try_get_tree+0x33/0xb0 [nfsv4]
[<0>] vfs_get_tree+0x25/0xc0
[<0>] do_mount+0x2e9/0x950
[<0>] ksys_mount+0xbe/0xe0
[<0>] __x64_sys_mount+0x21/0x30
[<0>] do_syscall_64+0x5b/0x1b0
[<0>] entry_SYSCALL_64_after_hwframe+0x61/0xc6
/proc/2022/stack:
[<0>] nfs41_callback_svc+0x18f/0x1a0 [nfsv4]
[<0>] kthread+0x134/0x150
[<0>] ret_from_fork+0x1f/0x40
/proc/2023/stack:
[<0>] msleep+0x28/0x40
[<0>] nfs4_handle_reclaim_lease_error+0x7e/0x140 [nfsv4]
[<0>] nfs4_state_manager+0x487/0x860 [nfsv4]
[<0>] nfs4_run_state_manager+0x20/0x40 [nfsv4]
[<0>] kthread+0x134/0x150
[<0>] ret_from_fork+0x1f/0x40
^ permalink raw reply [flat|nested] 26+ messages in thread
end of thread, other threads:[~2024-03-21 21:13 UTC | newest]
Thread overview: 26+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2024-03-12 12:24 nfsd hangs and nfsd_break_deleg_cb+0x170/0x190 warning Rik Theys
2024-03-12 12:47 ` Jeff Layton
2024-03-18 20:21 ` Rik Theys
2024-03-18 21:15 ` Rik Theys
2024-03-18 21:54 ` Jeff Layton
2024-03-19 7:58 ` Rik Theys
2024-03-19 10:39 ` Jeff Layton
2024-03-19 10:58 ` Rik Theys
2024-03-19 11:36 ` Jeff Layton
2024-03-20 13:03 ` Charles Hedrick
2024-03-19 17:09 ` Dai Ngo
2024-03-19 19:41 ` Rik Theys
2024-03-19 20:55 ` Jeff Layton
2024-03-19 21:42 ` Dai Ngo
2024-03-20 19:41 ` Rik Theys
2024-03-21 20:48 ` Jeff Layton
2024-03-21 21:13 ` Rik Theys
-- strict thread matches above, loose matches on Subject: below --
2024-03-12 13:30 Rik Theys
2024-03-11 18:43 Rik Theys
2024-03-12 11:22 ` Jeff Layton
2024-03-12 11:37 ` Jeff Layton
2024-03-12 16:43 ` Dai Ngo
2024-03-12 18:23 ` Rik Theys
2024-03-13 18:44 ` Dai Ngo
2024-03-13 19:50 ` Rik Theys
2024-03-13 22:45 ` Dai Ngo
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).