RE: [External] - Re: System hanging when using condition variables

[Russell Johnson <russell.johnson@kratosdefense.com>]

[-- Attachment #1.1: Type: text/plain, Size: 266 bytes --]

>Thanks. These reports show that an assertion spuriously trips 
>in the lock validator due to a missing fix up in the Dovetail support 
>for that code. Please apply [1].

I applied those patches and now I have the following output from the kernel.

Thanks,

Russell

[-- Attachment #1.2: stacktrace_4.txt --]
[-- Type: text/plain, Size: 38322 bytes --]

[Sep29 12:29] ------------[ cut here ]------------
[  +0.000003] WARNING: CPU: 0 PID: 0 at kernel/rcu/tree.c:1129 rcu_is_watching+0x2b/0x60
[  +0.000011] Modules linked in: dma_buf_mgr(OE) rtlstddriver(OE) xt_CHECKSUM xt_MASQUERADE tun ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 xt_conntrack ebtable_nat ip6table_nat ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_nat iptable_mangle iptable_security iptable_raw nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 ip_set nfnetlink ebtable_filter ebtables rfkill ip6table_filter ip6_tables iptable_filter sunrpc dm_mirror dm_region_hash dm_log dm_mod x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel gpio_ich iTCO_wdt iTCO_vendor_support mxm_wmi kvm irqbypass crct10dif_pclmul crc32_pclmul ghash_clmulni_intel aesni_intel crypto_simd cryptd rapl intel_cstate pcspkr i2c_i801 intel_pch_thermal i2c_smbus mei_me sg lpc_ich mfd_core joydev input_leds mei ftdi_sio ioatdma acpi_ipmi ipmi_si ipmi_devintf ipmi_msghandler wmi acpi_pad ip_tables xfs libcrc32c sr_mod sd_mod cdrom t10_pi ast drm_vram_helper drm_kms_helper syscopyarea sysfillrect sysimgblt
[  +0.000051]  fb_sys_fops drm_ttm_helper ttm drm ixgbe igb ahci libahci libata crc32c_intel mdio ptp i2c_algo_bit pps_core dca
[  +0.000010] CPU: 0 PID: 0 Comm: swapper/0 Tainted: G           OE     5.15.64evl-g4a69fe361195 #8
[  +0.000003] Hardware name: Supermicro Super Server/X10SDV-6C-TLN4F, BIOS 1.3 02/13/2018
[  +0.000001] IRQ stage: EVL
[  +0.000001] RIP: 0010:rcu_is_watching+0x2b/0x60
[  +0.000005] Code: b8 01 00 00 00 65 8b 15 33 94 ee 51 48 89 e5 f7 c2 00 00 00 01 75 15 0f b6 05 72 ae 76 02 84 c0 75 10 81 e2 00 00 00 02 74 08 <0f> 0b 5d c3 cc cc cc cc 65 ff 05 06 94 ee 51 48 c7 c0 48 f0 05 00
[  +0.000003] RSP: 0018:ffffffffaf603940 EFLAGS: 00010006
[  +0.000002] RAX: 0000000000000000 RBX: ffffffffafb1bbc9 RCX: ffffffffaf603998
[  +0.000002] RDX: 0000000002000000 RSI: ffffffffaf603e01 RDI: ffffffffafb1bbc9
[  +0.000001] RBP: ffffffffaf603940 R08: ffffffffaf603f48 R09: ffffffffaf61c818
[  +0.000001] R10: ffffffffaf603ad0 R11: ffffffffaf61c840 R12: ffffffffaf603a40
[  +0.000002] R13: 0000000000000000 R14: ffffffffaf61ba00 R15: ffffffffaffc2180
[  +0.000001] FS:  0000000000000000(0000) GS:ffff9e2b6d400000(0000) knlGS:0000000000000000
[  +0.000002] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[  +0.000001] CR2: 00007f8e413d5070 CR3: 0000000148b34005 CR4: 00000000003706f0
[  +0.000002] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[  +0.000001] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[  +0.000002] Call Trace:
[  +0.000000]  <TASK>

[  +0.000006] =============================
[  +0.000001] WARNING: suspicious RCU usage
[  +0.000000] 5.15.64evl-g4a69fe361195 #8 Tainted: G           OE
[  +0.000002] -----------------------------
[  +0.000000] include/linux/rcupdate.h:689 rcu_read_lock() used illegally while idle!
[  +0.000002]
other info that might help us debug this:

[  +0.000001]
rcu_scheduler_active = 2, debug_locks = 1
[  +0.000003] RCU used illegally from extended quiescent state!
[  +0.000000] 2 locks held by swapper/0/0:
[  +0.000002]  #0: ffff9e2b6d458730 (__RAWLOCK(&thread->lock)){....}-{2:2}, at: __evl_schedule+0x99/0x9f0
[  +0.000009]  #1: ffff9e2b6d4586b8 (__RAWLOCK(&rq->lock)){....}-{2:2}, at: __evl_schedule+0x99/0x9f0
[  +0.000006]
stack backtrace:
[  +0.000001] CPU: 0 PID: 0 Comm: swapper/0 Tainted: G           OE     5.15.64evl-g4a69fe361195 #8
[  +0.000002] Hardware name: Supermicro Super Server/X10SDV-6C-TLN4F, BIOS 1.3 02/13/2018
[  +0.000001] IRQ stage: EVL
[  +0.000001] Call Trace:
[  +0.000000]  <TASK>

[  +0.000007] =============================
[  +0.000000] WARNING: suspicious RCU usage
[  +0.000001] 5.15.64evl-g4a69fe361195 #8 Tainted: G           OE
[  +0.000001] -----------------------------
[  +0.000000] include/linux/rcupdate.h:717 rcu_read_unlock() used illegally while idle!
[  +0.000001]
other info that might help us debug this:

[  +0.000001]
rcu_scheduler_active = 2, debug_locks = 1
[  +0.000003] RCU used illegally from extended quiescent state!
[  +0.000000] 2 locks held by swapper/0/0:
[  +0.000001]  #0: ffff9e2b6d458730 (__RA[ 1357.745304] EVL: Main[4625] could not receive HM event #2
WLOCK(&thread->l[ 1359.106986]
[ 1359.106989] ============================================
[ 1359.106990] WARNING: possible recursive locking detected
ock)){....}-{2:2[ 1359.106991] 5.15.64evl-g4a69fe361195 #8 Tainted: G        W  OE
}, at: __evl_sch[ 1359.106993] --------------------------------------------
edule+0x99/0x9f0[ 1359.106993] IRQ_0xe4f_1_0/4641 is trying to acquire lock:

[  +0[ 1359.106995] ffff9e2843d6a018 (__RAWLOCK(&thread->lock)){-.-.}-{2:2}, at: evl_lock_mutex_timeout+0x299/0xe40
.000006]  1359.107007]
[ 1359.107007] but task is already holding lock:
[ 1359.107007] ffff9e28412f0018 (__RAWLOCK(&thread->lock)){-.-.}-{2:2}, at: evl_lock_mutex_timeout+0x1b5/0xe40
3m #1: f[ 1359.107013]
[ 1359.107013] other info that might help us debug this:
[ 1359.107014]  Possible unsafe locking scenario:
[ 1359.107014]
fff9e2b6d4586b8 [ 1359.107015]        CPU0
[ 1359.107015]        ----
[ 1359.107016]   lock(__RAWLOCK(&thread->lock));
[ 1359.107017]   lock(__RAWLOCK(&thread->lock));
[ 1359.107018]
[ 1359.107018]  *** DEADLOCK ***
[ 1359.107018]
[ 1359.107019]  May be due to missing lock nesting notation
[ 1359.107019]
(__RAWLOCK(&rq->[ 1359.107019] 2 locks held by IRQ_0xe4f_1_0/4641:
[ 1359.107021]  #0: ffff9e28454ba9d8 (&mon->mutex){....}-{0:0}, at: evl_lock_mutex_timeout+0x1b5/0xe40
lock)){....}-{2:[ 1359.107027]  #1: ffff9e28412f0018 (__RAWLOCK(&thread->lock)){-.-.}-{2:2}, at: evl_lock_mutex_timeout+0x1b5/0xe40
2}, at: __evl_sc[ 1359.107032]
[ 1359.107032] stack backtrace:
[ 1359.107033] CPU: 10 PID: 4641 Comm: IRQ_0xe4f_1_0 Tainted: G        W  OE     5.15.64evl-g4a69fe361195 #8
hedule+0x99/0x9f[ 1359.107037] Hardware name: Supermicro Super Server/X10SDV-6C-TLN4F, BIOS 1.3 02/13/2018
0
[  +[ 1359.107038] IRQ stage: EVL
[ 1359.107039] Call Trace:
0.000005]  1359.107040]  <TASK>
[ 1359.107054]  dump_stack_lvl+0x82/0xbc
[ 1359.107110]  dump_stack+0x10/0x16
[ 1359.107125]  __lock_acquire.cold.80+0xc3/0x2ab
[ 1359.107317]  lock_acquire+0xb3/0x2a0
[ 1359.107335]  ? evl_lock_mutex_timeout+0x299/0xe40
[ 1359.107541]  evl_lock_mutex_timeout+0x2c1/0xe40
1m
stack backtr[ 1359.107545]  ? evl_lock_mutex_timeout+0x299/0xe40
ace:
[[ 1359.107779]  enter_monitor+0x88/0xc0
  +0.000001]  1359.107820]  monitor_oob_ioctl+0x30c/0xf60
[ 1359.107836]  ? find_held_lock+0x35/0xb0
mCPU 1359.107878]  ? evl_get_file+0x6d/0xd0
m: 0 PID: 0 Comm[ 1359.107920]  ? lock_release+0x13e/0x270
: swapper/0 Tain[ 1359.108100]  EVL_ioctl+0x46/0xa0
ted: G          [ 1359.108152]  do_oob_syscall+0x3bb/0x3d0
 OE     5.15.64e[ 1359.108167]  ? handle_oob_irq+0xa2/0x190
vl-g4a69fe361195[ 1359.108257]  handle_oob_syscall+0xa8/0x100
 #8
[ [ 1359.108360]  pipeline_syscall+0x65/0x130
 +0.000001] [ 1359.108414]  syscall_enter_from_user_mode+0x47/0x110
Hardware na[ 1359.108469]  do_syscall_64+0x15/0xb0
me: Supe[ 1359.108497]  entry_SYSCALL_64_after_hwframe+0x61/0xcb
rmicro Super Ser[ 1359.108501] RIP: 0033:0x7f8e411d2a6b
ver/X10SDV-6C-TL[ 1359.108511] Code: Unable to access opcode bytes at RIP 0x7f8e411d2a41.
N4F, BIOS 1.3 02[ 1359.108512] RSP: 002b:00007f8e2f439320 EFLAGS: 00000246 ORIG_RAX: 000000000000009d
/13/2018
 1359.108515] RAX: ffffffffffffffda RBX: 00007f8e2f4393b0 RCX: 00007f8e411d2a6b
32m[  +0.000001][ 1359.108516] RDX: 0000000040106d00 RSI: 000000000000000d RDI: 0000000010000002
 IRQ st[ 1359.108518] RBP: 0000000040106d00 R08: 0000000000000000 R09: 00007f8e39eae680
age: EVL[ 1359.108519] R10: 00007f8e2f4393b0 R11: 0000000000000246 R12: 000000000000000d

[  +0[ 1359.108520] R13: 00007f8e2f4395b0 R14: 00007f8e39eb3968 R15: 00007f8e2f439578
.000001]  1359.108790]  </TASK>
mCall Trace:
[  +0.000000]  <TASK>

[  +0.000013] =============================
[  +0.000000] WARNING: suspicious RCU usage
[  +0.000000] 5.15.64evl-g4a69fe361195 #8 Tainted: G           OE
[  +0.000001] -----------------------------
[  +0.000001] include/linux/rcupdate.h:689 rcu_read_lock() used illegally while idle!
[  +0.000001]
other info that might help us debug this:

[  +0.000000]
rcu_scheduler_active = 2, debug_locks = 1
[  +0.000003] RCU used illegally from extended quiescent state!
[  +0.000000] 2 locks held by swapper/0/0:
[  +0.000001]  #0: ffff9e2b6d458730 (__RAWLOCK(&thread->lock)){....}-{2:2}, at: __evl_schedule+0x99/0x9f0
[  +0.000006]  #1: ffff9e2b6d4586b8 (__RAWLOCK(&rq->lock)){....}-{2:2}, at: __evl_schedule+0x99/0x9f0
[  +0.000005]
stack backtrace:
[  +0.000000] CPU: 0 PID: 0 Comm: swapper/0 Tainted: G           OE     5.15.64evl-g4a69fe361195 #8
[  +0.000002] Hardware name: Supermicro Super Server/X10SDV-6C-TLN4F, BIOS 1.3 02/13/2018
[  +0.000001] IRQ stage: EVL
[  +0.000000] Call Trace:
[  +0.000001]  <TASK>

[  +0.000013] =============================
[  +0.000001] WARNING: suspicious RCU usage
[  +0.000000] 5.15.64evl-g4a69fe361195 #8 Tainted: G           OE
[  +0.000001] -----------------------------
[  +0.000000] include/linux/rcupdate.h:717 rcu_read_unlock() used illegally while idle!
[  +0.000001]
other info that might help us debug this:

[  +0.000001]
rcu_scheduler_active = 2, debug_locks = 1
[  +0.000002] RCU used illegally from extended quiescent state!
[  +0.000001] 2 locks held by swapper/0/0:
[  +0.000000]  #0: ffff9e2b6d458730 (__RAWLOCK(&thread->lock)){....}-{2:2}, at: __evl_schedule+0x99/0x9f0
[  +0.000006]  #1: ffff9e2b6d4586b8 (__RAWLOCK(&rq->lock)){....}-{2:2}, at: __evl_schedule+0x99/0x9f0
[  +0.000005]
stack backtrace:
[  +0.000000] CPU: 0 PID: 0 Comm: swapper/0 Tainted: G           OE     5.15.64evl-g4a69fe361195 #8
[  +0.000002] Hardware name: Supermicro Super Server/X10SDV-6C-TLN4F, BIOS 1.3 02/13/2018
[  +0.000001] IRQ stage: EVL
[  +0.000000] Call Trace:
[  +0.000000]  <TASK>
[  +0.000014]  dump_stack_lvl+0x82/0xbc
[  +0.000055]  dump_stack+0x10/0x16
[  +0.000015]  lockdep_rcu_suspicious+0xed/0xfc
[  +0.000088]  is_bpf_text_address+0xdf/0xf0
[  +0.000027]  kernel_text_address+0x108/0x120
[  +0.000054]  __kernel_text_address+0xd/0x40
[  +0.000028]  show_trace_log_lvl+0x1d0/0x296
[  +0.000204]  ? dump_stack_lvl+0x82/0xbc
[  +0.000115]  show_stack+0x34/0x3a
[  +0.000016]  dump_stack_lvl+0x82/0xbc
[  +0.000053]  dump_stack+0x10/0x16
[  +0.000015]  lockdep_rcu_suspicious+0xed/0xfc
[  +0.000079]  is_bpf_text_address+0xac/0xf0
[  +0.000027]  kernel_text_address+0x108/0x120
[  +0.000052]  __kernel_text_address+0xd/0x40
[  +0.000028]  show_trace_log_lvl+0x1d0/0x296
[  +0.000203]  ? dump_stack_lvl+0x82/0xbc
[  +0.000121]  show_stack+0x34/0x3a
[  +0.000016]  dump_stack_lvl+0x82/0xbc
[  +0.000052]  dump_stack+0x10/0x16
[  +0.000016]  lockdep_rcu_suspicious+0xed/0xfc
[  +0.000079]  __is_insn_slot_addr+0x118/0x130
[  +0.000041]  kernel_text_address+0xc4/0x120
[  +0.000053]  __kernel_text_address+0xd/0x40
[  +0.000027]  show_trace_log_lvl+0x1d0/0x296
[  +0.000203]  ? dump_stack_lvl+0x82/0xbc
[  +0.000115]  show_stack+0x34/0x3a
[  +0.000016]  dump_stack_lvl+0x82/0xbc
[  +0.000053]  dump_stack+0x10/0x16
[  +0.000015]  lockdep_rcu_suspicious+0xed/0xfc
[  +0.000079]  __is_insn_slot_addr+0xe2/0x130
[  +0.000040]  kernel_text_address+0xc4/0x120
[  +0.000052]  __kernel_text_address+0xd/0x40
[  +0.000028]  show_trace_log_lvl+0x1d0/0x296
[  +0.000208]  ? kernel_text_address+0x2c/0x120
[  +0.000078]  ? rcu_is_watching+0x2b/0x60
[  +0.000029]  show_regs.cold.19+0x1a/0x1f
[  +0.000029]  __warn+0x80/0x100
[  +0.000040]  ? rcu_is_watching+0x2b/0x60
[  +0.000041]  report_bug+0xc5/0x100
[  +0.000093]  handle_bug+0x41/0xa0
[  +0.000041]  exc_invalid_op+0x18/0x70
[  +0.000040]  asm_exc_invalid_op+0x1b/0x20
[  +0.000002] RIP: 0010:rcu_is_watching+0x2b/0x60
[  +0.000004] Code: b8 01 00 00 00 65 8b 15 33 94 ee 51 48 89 e5 f7 c2 00 00 00 01 75 15 0f b6 05 72 ae 76 02 84 c0 75 10 81 e2 00 00 00 02 74 08 <0f> 0b 5d c3 cc cc cc cc 65 ff 05 06 94 ee 51 48 c7 c0 48 f0 05 00
[  +0.000003] RSP: 0018:ffffffffaf603940 EFLAGS: 00010006
[  +0.000002] RAX: 0000000000000000 RBX: ffffffffafb1bbc9 RCX: ffffffffaf603998
[  +0.000001] RDX: 0000000002000000 RSI: ffffffffaf603e01 RDI: ffffffffafb1bbc9
[  +0.000001] RBP: ffffffffaf603940 R08: ffffffffaf603f48 R09: ffffffffaf61c818
[  +0.000001] R10: ffffffffaf603ad0 R11: ffffffffaf61c840 R12: ffffffffaf603a40
[  +0.000001] R13: 0000000000000000 R14: ffffffffaf61ba00 R15: ffffffffaffc2180
[  +0.000077]  ? arch_call_rest_init+0x9/0x14
[  +0.000116]  ? arch_call_rest_init+0x9/0x14
[  +0.000078]  kernel_text_address+0x2c/0x120
[  +0.000015]  ? arch_call_rest_init+0x9/0x14
[  +0.000041]  __kernel_text_address+0xd/0x40
[  +0.000027]  unwind_get_return_address+0x1e/0x40
[  +0.000003]  ? create_prof_cpu_mask+0x30/0x30
[  +0.000017]  arch_stack_walk+0x9c/0xf0
[  +0.000136]  ? arch_call_rest_init+0x9/0x14
[  +0.000103]  stack_trace_save+0x47/0x70
[  +0.000064]  save_trace+0x45/0x390
[  +0.000141]  check_prevs_add+0xa9e/0x1060
[  +0.000178]  __lock_acquire+0x1105/0x1580
[  +0.000004]  ? lock_is_held_type+0x50/0x110
[  +0.000003]  ? subsystem_release+0x19/0x40
[  +0.000166]  lock_acquire+0xb3/0x2a0
[  +0.000016]  ? lock_timer_base.isra.21+0x2f/0x80
[  +0.000202]  lock_timer_base.isra.21+0x57/0x80
[  +0.000003]  ? lock_timer_base.isra.21+0x2f/0x80
[  +0.000083]  evl_start_timer+0x50/0x210
[  +0.000090]  __evl_schedule+0x622/0x9f0
[  +0.000116]  run_oob_call+0x99/0x170
[  +0.000102]  resume_oob_task+0x110/0x1c0
[  +0.000091]  finalize_oob_transition+0x42/0x80
[  +0.000028]  inband_switch_tail+0x3b/0x90
[  +0.000028]  __schedule+0x4bb/0xa70
[  +0.000115]  schedule_idle+0x1b/0x40
[  +0.000016]  do_idle+0x1b3/0x280
[  +0.000114]  cpu_startup_entry+0x18/0x20
[  +0.000028]  rest_init+0x177/0x280
[  +0.000041]  arch_call_rest_init+0x9/0x14
[  +0.000028]  start_kernel+0x6bc/0x6e6
[  +0.000145]  x86_64_start_reservations+0x24/0x2a
[  +0.000029]  x86_64_start_kernel+0x89/0x90
[  +0.000028]  secondary_startup_64_no_verify+0xb0/0xbb
[  +0.000265]  </TASK>
[  +0.000001]  dump_stack_lvl+0x82/0xbc
[  +0.000053]  dump_stack+0x10/0x16
[  +0.000016]  lockdep_rcu_suspicious+0xed/0xfc
[  +0.000079]  is_bpf_text_address+0xac/0xf0
[  +0.000027]  kernel_text_address+0x108/0x120
[  +0.000052]  __kernel_text_address+0xd/0x40
[  +0.000028]  show_trace_log_lvl+0x1d0/0x296
[  +0.000202]  ? dump_stack_lvl+0x82/0xbc
[  +0.000116]  show_stack+0x34/0x3a
[  +0.000015]  dump_stack_lvl+0x82/0xbc
[  +0.000053]  dump_stack+0x10/0x16
[  +0.000015]  lockdep_rcu_suspicious+0xed/0xfc
[  +0.000085]  __is_insn_slot_addr+0x118/0x130
[  +0.000040]  kernel_text_address+0xc4/0x120
[  +0.000052]  __kernel_text_address+0xd/0x40
[  +0.000028]  show_trace_log_lvl+0x1d0/0x296
[  +0.000203]  ? dump_stack_lvl+0x82/0xbc
[  +0.000115]  show_stack+0x34/0x3a
[  +0.000016]  dump_stack_lvl+0x82/0xbc
[  +0.000052]  dump_stack+0x10/0x16
[  +0.000015]  lockdep_rcu_suspicious+0xed/0xfc
[  +0.000079]  __is_insn_slot_addr+0xe2/0x130
[  +0.000041]  kernel_text_address+0xc4/0x120
[  +0.000052]  __kernel_text_address+0xd/0x40
[  +0.000027]  show_trace_log_lvl+0x1d0/0x296
[  +0.000203]  ? kernel_text_address+0x2c/0x120
[  +0.000083]  ? rcu_is_watching+0x2b/0x60
[  +0.000029]  show_regs.cold.19+0x1a/0x1f
[  +0.000029]  __warn+0x80/0x100
[  +0.000039]  ? rcu_is_watching+0x2b/0x60
[  +0.000041]  report_bug+0xc5/0x100
[  +0.000092]  handle_bug+0x41/0xa0
[  +0.000040]  exc_invalid_op+0x18/0x70
[  +0.000040]  asm_exc_invalid_op+0x1b/0x20
[  +0.000002] RIP: 0010:rcu_is_watching+0x2b/0x60
[  +0.000004] Code: b8 01 00 00 00 65 8b 15 33 94 ee 51 48 89 e5 f7 c2 00 00 00 01 75 15 0f b6 05 72 ae 76 02 84 c0 75 10 81 e2 00 00 00 02 74 08 <0f> 0b 5d c3 cc cc cc cc 65 ff 05 06 94 ee 51 48 c7 c0 48 f0 05 00
[  +0.000002] RSP: 0018:ffffffffaf603940 EFLAGS: 00010006
[  +0.000001] RAX: 0000000000000000 RBX: ffffffffafb1bbc9 RCX: ffffffffaf603998
[  +0.000001] RDX: 0000000002000000 RSI: ffffffffaf603e01 RDI: ffffffffafb1bbc9
[  +0.000002] RBP: ffffffffaf603940 R08: ffffffffaf603f48 R09: ffffffffaf61c818
[  +0.000001] R10: ffffffffaf603ad0 R11: ffffffffaf61c840 R12: ffffffffaf603a40
[  +0.000001] R13: 0000000000000000 R14: ffffffffaf61ba00 R15: ffffffffaffc2180
[  +0.000076]  ? arch_call_rest_init+0x9/0x14
[  +0.000115]  ? arch_call_rest_init+0x9/0x14
[  +0.000078]  kernel_text_address+0x2c/0x120
[  +0.000015]  ? arch_call_rest_init+0x9/0x14
[  +0.000041]  __kernel_text_address+0xd/0x40
[  +0.000027]  unwind_get_return_address+0x1e/0x40
[  +0.000002]  ? create_prof_cpu_mask+0x30/0x30
[  +0.000017]  arch_stack_walk+0x9c/0xf0
[  +0.000128]  ? arch_call_rest_init+0x9/0x14
[  +0.000109]  stack_trace_save+0x47/0x70
[  +0.000064]  save_trace+0x45/0x390
[  +0.000140]  check_prevs_add+0xa9e/0x1060
[  +0.000179]  __lock_acquire+0x1105/0x1580
[  +0.000003]  ? lock_is_held_type+0x50/0x110
[  +0.000003]  ? subsystem_release+0x19/0x40
[  +0.000166]  lock_acquire+0xb3/0x2a0
[  +0.000015]  ? lock_timer_base.isra.21+0x2f/0x80
[  +0.000203]  lock_timer_base.isra.21+0x57/0x80
[  +0.000002]  ? lock_timer_base.isra.21+0x2f/0x80
[  +0.000077]  evl_start_timer+0x50/0x210
[  +0.000091]  __evl_schedule+0x622/0x9f0
[  +0.000120]  run_oob_call+0x99/0x170
[  +0.000102]  resume_oob_task+0x110/0x1c0
[  +0.000091]  finalize_oob_transition+0x42/0x80
[  +0.000028]  inband_switch_tail+0x3b/0x90
[  +0.000029]  __schedule+0x4bb/0xa70
[  +0.000115]  schedule_idle+0x1b/0x40
[  +0.000015]  do_idle+0x1b3/0x280
[  +0.000114]  cpu_startup_entry+0x18/0x20
[  +0.000027]  rest_init+0x177/0x280
[  +0.000041]  arch_call_rest_init+0x9/0x14
[  +0.000028]  start_kernel+0x6bc/0x6e6
[  +0.000141]  x86_64_start_reservations+0x24/0x2a
[  +0.000028]  x86_64_start_kernel+0x89/0x90
[  +0.000028]  secondary_startup_64_no_verify+0xb0/0xbb
[  +0.000270]  </TASK>
[  +0.000002]  dump_stack_lvl+0x82/0xbc
[  +0.000053]  dump_stack+0x10/0x16
[  +0.000015]  lockdep_rcu_suspicious+0xed/0xfc
[  +0.000080]  __is_insn_slot_addr+0x118/0x130
[  +0.000040]  kernel_text_address+0xc4/0x120
[  +0.000052]  __kernel_text_address+0xd/0x40
[  +0.000028]  show_trace_log_lvl+0x1d0/0x296
[  +0.000203]  ? dump_stack_lvl+0x82/0xbc
[  +0.000115]  show_stack+0x34/0x3a
[  +0.000016]  dump_stack_lvl+0x82/0xbc
[  +0.000052]  dump_stack+0x10/0x16
[  +0.000015]  lockdep_rcu_suspicious+0xed/0xfc
[  +0.000079]  __is_insn_slot_addr+0xe2/0x130
[  +0.000041]  kernel_text_address+0xc4/0x120
[  +0.000052]  __kernel_text_address+0xd/0x40
[  +0.000029]  show_trace_log_lvl+0x1d0/0x296
[  +0.000206]  ? kernel_text_address+0x2c/0x120
[  +0.000077]  ? rcu_is_watching+0x2b/0x60
[  +0.000029]  show_regs.cold.19+0x1a/0x1f
[  +0.000029]  __warn+0x80/0x100
[  +0.000039]  ? rcu_is_watching+0x2b/0x60
[  +0.000042]  report_bug+0xc5/0x100
[  +0.000091]  handle_bug+0x41/0xa0
[  +0.000040]  exc_invalid_op+0x18/0x70
[  +0.000040]  asm_exc_invalid_op+0x1b/0x20
[  +0.000002] RIP: 0010:rcu_is_watching+0x2b/0x60
[  +0.000004] Code: b8 01 00 00 00 65 8b 15 33 94 ee 51 48 89 e5 f7 c2 00 00 00 01 75 15 0f b6 05 72 ae 76 02 84 c0 75 10 81 e2 00 00 00 02 74 08 <0f> 0b 5d c3 cc cc cc cc 65 ff 05 06 94 ee 51 48 c7 c0 48 f0 05 00
[  +0.000001] RSP: 0018:ffffffffaf603940 EFLAGS: 00010006
[  +0.000002] RAX: 0000000000000000 RBX: ffffffffafb1bbc9 RCX: ffffffffaf603998
[  +0.000001] RDX: 0000000002000000 RSI: ffffffffaf603e01 RDI: ffffffffafb1bbc9
[  +0.000001] RBP: ffffffffaf603940 R08: ffffffffaf603f48 R09: ffffffffaf61c818
[  +0.000001] R10: ffffffffaf603ad0 R11: ffffffffaf61c840 R12: ffffffffaf603a40
[  +0.000001] R13: 0000000000000000 R14: ffffffffaf61ba00 R15: ffffffffaffc2180
[  +0.000076]  ? arch_call_rest_init+0x9/0x14
[  +0.000116]  ? arch_call_rest_init+0x9/0x14
[  +0.000077]  kernel_text_address+0x2c/0x120
[  +0.000015]  ? arch_call_rest_init+0x9/0x14
[  +0.000041]  __kernel_text_address+0xd/0x40
[  +0.000027]  unwind_get_return_address+0x1e/0x40
[  +0.000002]  ? create_prof_cpu_mask+0x30/0x30
[  +0.000017]  arch_stack_walk+0x9c/0xf0
[  +0.000133]  ? arch_call_rest_init+0x9/0x14
[  +0.000103]  stack_trace_save+0x47/0x70
[  +0.000064]  save_trace+0x45/0x390
[  +0.000141]  check_prevs_add+0xa9e/0x1060
[  +0.000178]  __lock_acquire+0x1105/0x1580
[  +0.000003]  ? lock_is_held_type+0x50/0x110
[  +0.000003]  ? subsystem_release+0x19/0x40
[  +0.000165]  lock_acquire+0xb3/0x2a0
[  +0.000016]  ? lock_timer_base.isra.21+0x2f/0x80
[  +0.000202]  lock_timer_base.isra.21+0x57/0x80
[  +0.000003]  ? lock_timer_base.isra.21+0x2f/0x80
[  +0.000082]  evl_start_timer+0x50/0x210
[  +0.000090]  __evl_schedule+0x622/0x9f0
[  +0.000116]  run_oob_call+0x99/0x170
[  +0.000102]  resume_oob_task+0x110/0x1c0
[  +0.000090]  finalize_oob_transition+0x42/0x80
[  +0.000029]  inband_switch_tail+0x3b/0x90
[  +0.000028]  __schedule+0x4bb/0xa70
[  +0.000115]  schedule_idle+0x1b/0x40
[  +0.000015]  do_idle+0x1b3/0x280
[  +0.000114]  cpu_startup_entry+0x18/0x20
[  +0.000027]  rest_init+0x177/0x280
[  +0.000042]  arch_call_rest_init+0x9/0x14
[  +0.000028]  start_kernel+0x6bc/0x6e6
[  +0.000145]  x86_64_start_reservations+0x24/0x2a
[  +0.000028]  x86_64_start_kernel+0x89/0x90
[  +0.000028]  secondary_startup_64_no_verify+0xb0/0xbb
[  +0.000264]  </TASK>
[  +0.000009]  dump_stack_lvl+0x82/0xbc
[  +0.000053]  dump_stack+0x10/0x16
[  +0.000015]  lockdep_rcu_suspicious+0xed/0xfc
[  +0.000079]  __is_insn_slot_addr+0xe2/0x130
[  +0.000040]  kernel_text_address+0xc4/0x120
[  +0.000053]  __kernel_text_address+0xd/0x40
[  +0.000027]  show_trace_log_lvl+0x1d0/0x296
[  +0.000203]  ? kernel_text_address+0x2c/0x120
[  +0.000077]  ? rcu_is_watching+0x2b/0x60
[  +0.000029]  show_regs.cold.19+0x1a/0x1f
[  +0.000029]  __warn+0x80/0x100
[  +0.000039]  ? rcu_is_watching+0x2b/0x60
[  +0.000047]  report_bug+0xc5/0x100
[  +0.000091]  handle_bug+0x41/0xa0
[  +0.000040]  exc_invalid_op+0x18/0x70
[  +0.000040]  asm_exc_invalid_op+0x1b/0x20
[  +0.000002] RIP: 0010:rcu_is_watching+0x2b/0x60
[  +0.000004] Code: b8 01 00 00 00 65 8b 15 33 94 ee 51 48 89 e5 f7 c2 00 00 00 01 75 15 0f b6 05 72 ae 76 02 84 c0 75 10 81 e2 00 00 00 02 74 08 <0f> 0b 5d c3 cc cc cc cc 65 ff 05 06 94 ee 51 48 c7 c0 48 f0 05 00
[  +0.000001] RSP: 0018:ffffffffaf603940 EFLAGS: 00010006
[  +0.000002] RAX: 0000000000000000 RBX: ffffffffafb1bbc9 RCX: ffffffffaf603998
[  +0.000001] RDX: 0000000002000000 RSI: ffffffffaf603e01 RDI: ffffffffafb1bbc9
[  +0.000001] RBP: ffffffffaf603940 R08: ffffffffaf603f48 R09: ffffffffaf61c818
[  +0.000001] R10: ffffffffaf603ad0 R11: ffffffffaf61c840 R12: ffffffffaf603a40
[  +0.000001] R13: 0000000000000000 R14: ffffffffaf61ba00 R15: ffffffffaffc2180
[  +0.000076]  ? arch_call_rest_init+0x9/0x14
[  +0.000116]  ? arch_call_rest_init+0x9/0x14
[  +0.000077]  kernel_text_address+0x2c/0x120
[  +0.000015]  ? arch_call_rest_init+0x9/0x14
[  +0.000041]  __kernel_text_address+0xd/0x40
[  +0.000027]  unwind_get_return_address+0x1e/0x40
[  +0.000002]  ? create_prof_cpu_mask+0x30/0x30
[  +0.000017]  arch_stack_walk+0x9c/0xf0
[  +0.000128]  ? arch_call_rest_init+0x9/0x14
[  +0.000103]  stack_trace_save+0x47/0x70
[  +0.000064]  save_trace+0x45/0x390
[  +0.000146]  check_prevs_add+0xa9e/0x1060
[  +0.000178]  __lock_acquire+0x1105/0x1580
[  +0.000003]  ? lock_is_held_type+0x50/0x110
[  +0.000003]  ? subsystem_release+0x19/0x40
[  +0.000166]  lock_acquire+0xb3/0x2a0
[  +0.000015]  ? lock_timer_base.isra.21+0x2f/0x80
[  +0.000203]  lock_timer_base.isra.21+0x57/0x80
[  +0.000002]  ? lock_timer_base.isra.21+0x2f/0x80
[  +0.000077]  evl_start_timer+0x50/0x210
[  +0.000090]  __evl_schedule+0x622/0x9f0
[  +0.000116]  run_oob_call+0x99/0x170
[  +0.000102]  resume_oob_task+0x110/0x1c0
[  +0.000095]  finalize_oob_transition+0x42/0x80
[  +0.000029]  inband_switch_tail+0x3b/0x90
[  +0.000028]  __schedule+0x4bb/0xa70
[  +0.000115]  schedule_idle+0x1b/0x40
[  +0.000015]  do_idle+0x1b3/0x280
[  +0.000114]  cpu_startup_entry+0x18/0x20
[  +0.000027]  rest_init+0x177/0x280
[  +0.000041]  arch_call_rest_init+0x9/0x14
[  +0.000029]  start_kernel+0x6bc/0x6e6
[  +0.000140]  x86_64_start_reservations+0x24/0x2a
[  +0.000028]  x86_64_start_kernel+0x89/0x90
[  +0.000027]  secondary_startup_64_no_verify+0xb0/0xbb
[  +0.000265]  </TASK>
[  +0.000010]  kernel_text_address+0x2c/0x120
[  +0.000015]  ? arch_call_rest_init+0x9/0x14
[  +0.000044]  __kernel_text_address+0xd/0x40
[  +0.000029]  unwind_get_return_address+0x1e/0x40
[  +0.000002]  ? create_prof_cpu_mask+0x30/0x30
[  +0.000016]  arch_stack_walk+0x9c/0xf0
[  +0.000129]  ? arch_call_rest_init+0x9/0x14
[  +0.000102]  stack_trace_save+0x47/0x70
[  +0.000065]  save_trace+0x45/0x390
[  +0.000140]  check_prevs_add+0xa9e/0x1060
[  +0.000178]  __lock_acquire+0x1105/0x1580
[  +0.000003]  ? lock_is_held_type+0x50/0x110
[  +0.000003]  ? subsystem_release+0x19/0x40
[  +0.000166]  lock_acquire+0xb3/0x2a0
[  +0.000015]  ? lock_timer_base.isra.21+0x2f/0x80
[  +0.000208]  lock_timer_base.isra.21+0x57/0x80
[  +0.000002]  ? lock_timer_base.isra.21+0x2f/0x80
[  +0.000078]  evl_start_timer+0x50/0x210
[  +0.000090]  __evl_schedule+0x622/0x9f0
[  +0.000115]  run_oob_call+0x99/0x170
[  +0.000102]  resume_oob_task+0x110/0x1c0
[  +0.000090]  finalize_oob_transition+0x42/0x80
[  +0.000029]  inband_switch_tail+0x3b/0x90
[  +0.000028]  __schedule+0x4bb/0xa70
[  +0.000115]  schedule_idle+0x1b/0x40
[  +0.000015]  do_idle+0x1b3/0x280
[  +0.000114]  cpu_startup_entry+0x18/0x20
[  +0.000027]  rest_init+0x177/0x280
[  +0.000041]  arch_call_rest_init+0x9/0x14
[  +0.000028]  start_kernel+0x6bc/0x6e6
[  +0.000146]  x86_64_start_reservations+0x24/0x2a
[  +0.000028]  x86_64_start_kernel+0x89/0x90
[  +0.000027]  secondary_startup_64_no_verify+0xb0/0xbb
[  +0.000265]  </TASK>
[  +0.000001] irq event stamp: 8487512
[  +0.000000] hardirqs last  enabled at (8487511): [<ffffffffae126994>] sync_current_irq_stage+0x2a4/0x2d0
[  +0.000003] hardirqs last disabled at (8487512): [<ffffffffaeaf48e2>] __schedule+0x282/0xa70
[  +0.000003] softirqs last  enabled at (8487488): [<ffffffffaee002e9>] __do_softirq+0x2e9/0x421
[  +0.000003] softirqs last disabled at (8487481): [<ffffffffae08aa3d>] irq_exit_rcu+0xcd/0xf0
[  +0.000004] ---[ end trace da74aa0feaab0470 ]---
[  +0.015764] EVL: Main[4625] could not receive HM event #2
[  +0.000968] EVL: Main[4625] could not receive HM event #2
[  +0.000343] EVL: Main[4625] could not receive HM event #2
[  +0.000406] EVL: Main[4625] could not receive HM event #2
[  +0.000403] EVL: Main[4625] could not receive HM event #2
[  +0.000312] EVL: Main[4625] could not receive HM event #2
[  +0.000974] EVL: Main[4625] could not receive HM event #2
[  +0.003818] EVL: Main[4625] could not receive HM event #2
[  +0.018835] EVL: Main[4625] could not receive HM event #2
[  +0.000475] EVL: Main[4625] could not receive HM event #2

[  +1.361685] ============================================
[  +0.000001] WARNING: possible recursive locking detected
[  +0.000001] 5.15.64evl-g4a69fe361195 #8 Tainted: G        W  OE
[  +0.000002] --------------------------------------------
[  +0.000000] IRQ_0xe4f_1_0/4641 is trying to acquire lock:
[  +0.000002] ffff9e2843d6a018 (__RAWLOCK(&thread->lock)){-.-.}-{2:2}, at: evl_lock_mutex_timeout+0x299/0xe40
[  +0.000012]
but task is already holding lock:
[  +0.000000] ffff9e28412f0018 (__RAWLOCK(&thread->lock)){-.-.}-{2:2}, at: evl_lock_mutex_timeout+0x1b5/0xe40
[  +0.000006]
other info that might help us debug this:
[  +0.000001]  Possible unsafe locking scenario:

[  +0.000001]        CPU0
[  +0.000000]        ----
[  +0.000001]   lock(__RAWLOCK(&thread->lock));
[  +0.000001]   lock(__RAWLOCK(&thread->lock));
[  +0.000001]
 *** DEADLOCK ***

[  +0.000001]  May be due to missing lock nesting notation

[  +0.000000] 2 locks held by IRQ_0xe4f_1_0/4641:
[  +0.000002]  #0: ffff9e28454ba9d8 (&mon->mutex){....}-{0:0}, at: evl_lock_mutex_timeout+0x1b5/0xe40
[  +0.000006]  #1: ffff9e28412f0018 (__RAWLOCK(&thread->lock)){-.-.}-{2:2}, at: evl_lock_mutex_timeout+0x1b5/0xe40
[  +0.000005]
stack backtrace:
[  +0.000001] CPU: 10 PID: 4641 Comm: IRQ_0xe4f_1_0 Tainted: G        W  OE     5.15.64evl-g4a69fe361195 #8
[  +0.000004] Hardware name: Supermicro Super Server/X10SDV-6C-TLN4F, BIOS 1.3 02/13/2018
[  +0.000001] IRQ stage: EVL
[  +0.000001] Call Trace:
[  +0.000001]  <TASK>
[  +0.000014]  dump_stack_lvl+0x82/0xbc
[  +0.000056]  dump_stack+0x10/0x16
[  +0.000015]  __lock_acquire.cold.80+0xc3/0x2ab
[  +0.000192]  lock_acquire+0xb3/0x2a0
[  +0.000018]  ? evl_lock_mutex_timeout+0x299/0xe40
[  +0.000206]  evl_lock_mutex_timeout+0x2c1/0xe40
[  +0.000004]  ? evl_lock_mutex_timeout+0x299/0xe40
[  +0.000234]  enter_monitor+0x88/0xc0
[  +0.000041]  monitor_oob_ioctl+0x30c/0xf60
[  +0.000016]  ? find_held_lock+0x35/0xb0
[  +0.000042]  ? evl_get_file+0x6d/0xd0
[  +0.000042]  ? lock_release+0x13e/0x270
[  +0.000180]  EVL_ioctl+0x46/0xa0
[  +0.000052]  do_oob_syscall+0x3bb/0x3d0
[  +0.000015]  ? handle_oob_irq+0xa2/0x190
[  +0.000090]  handle_oob_syscall+0xa8/0x100
[  +0.000103]  pipeline_syscall+0x65/0x130
[  +0.000054]  syscall_enter_from_user_mode+0x47/0x110
[  +0.000055]  do_syscall_64+0x15/0xb0
[  +0.000028]  entry_SYSCALL_64_after_hwframe+0x61/0xcb
[  +0.000004] RIP: 0033:0x7f8e411d2a6b
[  +0.000010] Code: Unable to access opcode bytes at RIP 0x7f8e411d2a41.
[  +0.000001] RSP: 002b:00007f8e2f439320 EFLAGS: 00000246 ORIG_RAX: 000000000000009d
[  +0.000003] RAX: ffffffffffffffda RBX: 00007f8e2f4393b0 RCX: 00007f8e411d2a6b
[  +0.000001] RDX: 0000000040106d00 RSI: 000000000000000d RDI: 0000000010000002
[  +0.000002] RBP: 0000000040106d00 R08: 0000000000000000 R09: 00007f8e39eae680
[  +0.000001] R10: 00007f8e2f4393b0 R11: 0000000000000246 R12: 000000000000000d
[  +0.000001] R13: 00007f8e2f4395b0 R14: 00007f8e39eb3968 R15: 00007f8e2f439578
[  +0.000270]  </TASK>
[ 1407.967142] watchdog: BUG: soft lockup - CPU#7 stuck for 23s! [kworker/7:0:3780]
[ 1407.990020] Modules linked in: dma_buf_mgr(OE) rtlstddriver(OE) xt_CHECKSUM xt_MASQUERADE tun ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 xt_conntrack ebtable_nat ip6table_nat ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_nat iptable_mangle iptable_security iptable_raw nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 ip_set nfnetlink ebtable_filter ebtables rfkill ip6table_filter ip6_tables iptable_filter sunrpc dm_mirror dm_region_hash dm_log dm_mod x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel gpio_ich iTCO_wdt iTCO_vendor_support mxm_wmi kvm irqbypass crct10dif_pclmul crc32_pclmul ghash_clmulni_intel aesni_intel crypto_simd cryptd rapl intel_cstate pcspkr i2c_i801 intel_pch_thermal i2c_smbus mei_me sg lpc_ich mfd_core joydev input_leds mei ftdi_sio ioatdma acpi_ipmi ipmi_si ipmi_devintf ipmi_msghandler wmi acpi_pad ip_tables xfs libcrc32c sr_mod sd_mod cdrom t10_pi ast drm_vram_helper drm_kms_helper syscopyarea sysfillrect sysimgblt
[ 1407.990121]  fb_sys_fops drm_ttm_helper ttm drm ixgbe igb ahci libahci libata crc32c_intel mdio ptp i2c_algo_bit pps_core dca
[ 1408.291430] irq event stamp: 12888
[ 1408.302564] hardirqs last  enabled at (12887): [<ffffffffaeafe622>] _raw_spin_unlock_irq+0x22/0x40
[ 1408.330379] hardirqs last disabled at (12888): [<ffffffffaeaf48e2>] __schedule+0x282/0xa70
[ 1408.356099] softirqs last  enabled at (12848): [<ffffffffaee002e9>] __do_softirq+0x2e9/0x421
[ 1408.382322] softirqs last disabled at (12843): [<ffffffffae08aa3d>] irq_exit_rcu+0xcd/0xf0
[ 1408.408019] CPU: 7 PID: 3780 Comm: kworker/7:0 Tainted: G        W  OE     5.15.64evl-g4a69fe361195 #8
[ 1408.436844] Hardware name: Supermicro Super Server/X10SDV-6C-TLN4F, BIOS 1.3 02/13/2018
[ 1408.461788] IRQ stage: Linux
[ 1408.471385] Workqueue: events netstamp_clear
[ 1408.485177] RIP: 0010:smp_call_function_many_cond+0x154/0x370
[ 1408.503389] Code: 89 c7 e8 1f 32 3e 00 3b 05 4d ea 92 01 73 96 48 63 d0 48 8b 0b 48 03 0c d5 80 b8 3f af 8b 51 08 83 e2 01 74 0a f3 90 8b 51 08 <83> e2 01 75 f6 eb cb 80 3d f6 90 92 01 00 0f 85 f8 fe ff ff 8b 05
[ 1408.561725] RSP: 0018:ffffbab0c2dcfcc0 EFLAGS: 00000202 ORIG_RAX: 0000000000000000
[ 1408.585477] RAX: 0000000000000000 RBX: 0000000000000000 RCX: 0000000000000000
[ 1408.607938] RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000000
[ 1408.630370] RBP: ffffbab0c2dcfd20 R08: 0000000000000000 R09: 0000000000000000
[ 1408.652811] R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000000
[ 1408.675242] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000
[ 1408.697674] FS:  0000000000000000(0000) GS:ffff9e2b6d780000(0000) knlGS:0000000000000000
[ 1408.722984] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 1408.741273] CR2: 0000000001c6cc68 CR3: 00000003a5612004 CR4: 00000000003706e0
[ 1408.763752] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 1408.786220] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[ 1408.808688] Call Trace:
[ 1408.817132]  <TASK>
[ 1408.824513]  ? __text_poke+0x4a0/0x4a0
[ 1408.836834]  ? netif_receive_skb_list_internal+0x3d/0x370
[ 1408.854107]  on_each_cpu_cond_mask+0x24/0x40
[ 1408.868013]  text_poke_bp_batch+0xbc/0x2c0
[ 1408.881371]  ? netif_receive_skb_list_internal+0x3d/0x370
[ 1408.898639]  text_poke_finish+0x1a/0x30
[ 1408.911209]  arch_jump_label_transform_apply+0x17/0x30
[ 1408.927690]  __jump_label_update+0xdd/0x120
[ 1408.941307]  jump_label_update+0xc9/0xf0
[ 1408.954132]  static_key_enable_cpuslocked+0x5f/0x90
[ 1408.969842]  static_key_enable+0x15/0x30
[ 1408.982671]  netstamp_clear+0x28/0x40
[ 1408.994699]  process_one_work+0x29a/0x590
[ 1409.007765]  worker_thread+0x2f/0x3e0
[ 1409.019774]  ? process_one_work+0x590/0x590
[ 1409.033297]  kthread+0x13b/0x160
[ 1409.043923]  ? set_kthread_struct+0x50/0x50
[ 1409.057394]  ret_from_fork+0x22/0x30
[ 1409.069014]  </TASK>
[Sep29 12:30] watchdog: BUG: soft lockup - CPU#7 stuck for 23s! [kworker/7:0:3780]
[  +0.022878] Modules linked in: dma_buf_mgr(OE) rtlstddriver(OE) xt_CHECKSUM xt_MASQUERADE tun ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 xt_conntrack ebtable_nat ip6table_nat ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_nat iptable_mangle iptable_security iptable_raw nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 ip_set nfnetlink ebtable_filter ebtables rfkill ip6table_filter ip6_tables iptable_filter sunrpc dm_mirror dm_region_hash dm_log dm_mod x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel gpio_ich iTCO_wdt iTCO_vendor_support mxm_wmi kvm irqbypass crct10dif_pclmul crc32_pclmul ghash_clmulni_intel aesni_intel crypto_simd cryptd rapl intel_cstate pcspkr i2c_i801 intel_pch_thermal i2c_smbus mei_me sg lpc_ich mfd_core joydev input_leds mei ftdi_sio ioatdma acpi_ipmi ipmi_si ipmi_devintf ipmi_msghandler wmi acpi_pad ip_tables xfs libcrc32c sr_mod sd_mod cdrom t10_pi ast drm_vram_helper drm_kms_helper syscopyarea sysfillrect sysimgblt
[  +0.000101]  fb_sys_fops drm_ttm_helper ttm drm ixgbe igb ahci libahci libata crc32c_intel mdio ptp i2c_algo_bit pps_core dca
[  +0.301309] irq event stamp: 12888
[  +0.011134] hardirqs last  enabled at (12887): [<ffffffffaeafe622>] _raw_spin_unlock_irq+0x22/0x40
[  +0.027815] hardirqs last disabled at (12888): [<ffffffffaeaf48e2>] __schedule+0x282/0xa70
[  +0.025720] softirqs last  enabled at (12848): [<ffffffffaee002e9>] __do_softirq+0x2e9/0x421
[  +0.026223] softirqs last disabled at (12843): [<ffffffffae08aa3d>] irq_exit_rcu+0xcd/0xf0
[  +0.025697] CPU: 7 PID: 3780 Comm: kworker/7:0 Tainted: G        W  OE     5.15.64evl-g4a69fe361195 #8
[  +0.028825] Hardware name: Supermicro Super Server/X10SDV-6C-TLN4F, BIOS 1.3 02/13/2018
[  +0.024944] IRQ stage: Linux
[  +0.009597] Workqueue: events netstamp_clear
[  +0.013792] RIP: 0010:smp_call_function_many_cond+0x154/0x370
[  +0.018212] Code: 89 c7 e8 1f 32 3e 00 3b 05 4d ea 92 01 73 96 48 63 d0 48 8b 0b 48 03 0c d5 80 b8 3f af 8b 51 08 83 e2 01 74 0a f3 90 8b 51 08 <83> e2 01 75 f6 eb cb 80 3d f6 90 92 01 00 0f 85 f8 fe ff ff 8b 05
[  +0.058336] RSP: 0018:ffffbab0c2dcfcc0 EFLAGS: 00000202 ORIG_RAX: 0000000000000000
[  +0.023752] RAX: 0000000000000000 RBX: 0000000000000000 RCX: 0000000000000000
[  +0.022461] RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000000
[  +0.022432] RBP: ffffbab0c2dcfd20 R08: 0000000000000000 R09: 0000000000000000
[  +0.022441] R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000000
[  +0.022431] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000
[  +0.022432] FS:  0000000000000000(0000) GS:ffff9e2b6d780000(0000) knlGS:0000000000000000
[  +0.025310] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[  +0.018289] CR2: 0000000001c6cc68 CR3: 00000003a5612004 CR4: 00000000003706e0
[  +0.022479] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[  +0.022468] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[  +0.022468] Call Trace:
[  +0.008444]  <TASK>
[  +0.007381]  ? __text_poke+0x4a0/0x4a0
[  +0.012321]  ? netif_receive_skb_list_internal+0x3d/0x370
[  +0.017273]  on_each_cpu_cond_mask+0x24/0x40
[  +0.013906]  text_poke_bp_batch+0xbc/0x2c0
[  +0.013358]  ? netif_receive_skb_list_internal+0x3d/0x370
[  +0.017268]  text_poke_finish+0x1a/0x30
[  +0.012570]  arch_jump_label_transform_apply+0x17/0x30
[  +0.016481]  __jump_label_update+0xdd/0x120
[  +0.013617]  jump_label_update+0xc9/0xf0
[  +0.012825]  static_key_enable_cpuslocked+0x5f/0x90
[  +0.015710]  static_key_enable+0x15/0x30
[  +0.012829]  netstamp_clear+0x28/0x40
[  +0.012028]  process_one_work+0x29a/0x590
[  +0.013066]  worker_thread+0x2f/0x3e0
[  +0.012009]  ? process_one_work+0x590/0x590
[  +0.013523]  kthread+0x13b/0x160
[  +0.010626]  ? set_kthread_struct+0x50/0x50
[  +0.013471]  ret_from_fork+0x22/0x30
[  +0.011620]  </TASK>

[-- Attachment #2: smime.p7s --]
[-- Type: application/pkcs7-signature, Size: 6759 bytes --]