EVL Memory

EVL Memory

[Russell Johnson]

[-- Attachment #1.1.1: Type: text/plain, Size: 4554 bytes --]

Hello,

 

We have been running into some memory issues with our realtime EVL
application. Recently, we are seeing a core dump write immediately on
startup, and when running in the debugger,  the call stack just doesn't make
any sense - it almost seems random. I have the latest EVL kernel built with
all the EVL debug on including the EVL_DEBUG_MEM option, and I also turned
on KASAN. I see no output at all from the kernel when this core dump happens
except for a couple basic lines:

 

EVL: RxMain switching in-band [pid=4327, excpt=14, user_pc=0x0]

RxMain[4327]: segfault at 0 ip 0000000000000000 sp 00007f7ab5305468 error 14
in realtime_sfhm[400000+161000]

Code: Unable to access opcode bytes at RIP 0xffffffffffffffd6.

 

I wouldn't say those lines are very helpful. I know that exception 14 on x86
is a page fault and that a pc value of 0 does not seem good (memory
corruption?). 

 

I wanted to get your opinion on our overall memory setup and see if there
are obvious issues or if you had any recommendations of things to try. So we
set up 1 EVL heap on startup of the application, and this heap is used for
all dynamic allocations throughout the entire app as we also overrode global
new/delete to use the EVL heap rather than malloc/free. The reason we
overrode global new/delete is that we have a lot of STL objects as well as
third party libraries that we are using in this app and it would become a
very long and challenging process to actually go through and modify all
those to use custom allocators. So, in main(), we spawn a "Main" EVL thread.
All other threads are spawned from this parent EVL thread. When the EVL Main
thread starts, it sets a static flag that enables using the evl heap in the
global news/deletes, and every library and process for the realtime app is
created and started. We also prefault the EVL heap after it is
initialized/created in order to try to avoid page faults while running the
realtime loop. However, we still see occasional page faults in our realtime
EVL threads while running our main realtime loop. I don't understand how
there could be a page fault if the EVL heap is large enough (verified) and
prefaulted.

 

Now back to the issue mentioned at the beginning of this email - we are
unsure at this time if the page fault error seen there is cause, result, or
primary error causing these core dumps. Typically when I see the occasional
page faults while running other EVL threads in the app, I do not see core
dumps - just the log from the kernel. The only reason why I figured I would
at least run this past you was that I did not see any core dumps when
disabling the flag in the global new/delete and just using malloc/free (of
course I saw some in band switches, but that was it).

 

The issue of occasional page faults, handling dynamic memory allocation
(using global new/delete), and now these possible memory corruption seg
faults is becoming a larger concern for us. We would like to make sure that
we are understanding how to use memory in an EVL application properly, and
we would be interested to know if there are any recommended ways of tracking
these down with an EVL application. I have tried to build/run with the gcc
address sanitizer, but I was seeing issues attaching EVL threads when this
was enabled. I have also tried running valgrind, but that has produced
nothing useful. And, of course, I have run in gdb and the stack traces are
not helpful. At this point, any guidance, thoughts, and/or recommendations
would be greatly appreciated. I added some more clear/specific questions at
the bottom.

 

A few specific questions:

1.	Is this a reasonable model to use for an EVL application, or do you
expect the model to revolve more around static allocation?
2.	Are you aware of people using EVL overriding the global new/delete
to use the EVL heap?
3.	Do you have any tools for debugging the EVL heap or have you adapted
any existing tools (such as valgrind) to debug the EVL heap?
4.	Is there any known way of protecting against a stack overflow?
5.	PC value of 0 is never valid and we have no evidence that we have an
uninitialized pointer in our C++ code. Is there anyway to use info from EVL
to help track down this issue?
6.	We are allocating 2GB of EVL Heap memory and pre-faulting all of it
on startup. We also use pthread_attr_setstacksize to pre-allocate the stack
for each EVL thread we have. EVL still says that we get rare page faults.
How is this possible? Are we missing something? (I have attached our heap
pre-faulting logic)

 

 

Thanks,

 

Russell


[-- Attachment #1.1.2: Type: text/html, Size: 9154 bytes --]

[-- Attachment #1.2: heap_prealloc.txt --]
[-- Type: text/plain, Size: 1362 bytes --]

// Static variables
struct evl_heap heap_manager::s_runtime_heap;
bool heap_manager::s_use_evl = false; // Flag used in global new/delete to switch between Linux and EVL heaps
size_t heap_manager::s_init_bytes = 2147483648; // 2GB

///////////////////////////////////////////////////////////////////////////////
void* heap_manager::alloc_block(size_t bytes)
{
    void* data = evl_alloc_block(&s_runtime_heap, bytes);
    if (data == NULL)
    {
        // Switch back to Linux heap to avoid infinite alloc loop
        s_use_evl = false;
        throw(std::runtime_error("Error allocating a block on the EVL heap"));
    }
    return data;
}

///////////////////////////////////////////////////////////////////////////////
void heap_manager::free_block(void* block)
{
    int ret = evl_free_block(&s_runtime_heap, block);
    if (ret < 0)
    {
        sfhm::rtString msg;
        msg.Format("%s:%d: Error freeing a block on the EVL heap", __FILE__, __LINE__);
        throw(std::system_error(ret, std::generic_category(), msg));
    }
}

///////////////////////////////////////////////////////////////////////////////
void heap_manager::prefault()
{
    char *dummy = (char*)alloc_block(s_init_bytes);
    for (size_t i = 0; i < s_init_bytes; i += sysconf(_SC_PAGESIZE))
        dummy[i] = 1;  
    free_block(dummy);
}


[-- Attachment #2: smime.p7s --]
[-- Type: application/pkcs7-signature, Size: 6759 bytes --]

Re: EVL Memory

[Philippe Gerum]

Russell Johnson <russell.johnson@kratosdefense.com> writes:

> [[S/MIME Signed Part:Undecided]]
> Hello,
>
>  
>
> We have been running into some memory issues with our realtime EVL application. Recently, we are seeing a core dump write immediately on
> startup, and when running in the debugger,  the call stack just doesn’t make any sense – it almost seems random. I have the latest EVL
> kernel built with all the EVL debug on including the EVL_DEBUG_MEM option, and I also turned on KASAN. I see no output at all from the
> kernel when this core dump happens except for a couple basic lines:
>
>  
>
> EVL: RxMain switching in-band [pid=4327, excpt=14, user_pc=0x0]
>
> RxMain[4327]: segfault at 0 ip 0000000000000000 sp 00007f7ab5305468 error 14 in realtime_sfhm[400000+161000]
>
> Code: Unable to access opcode bytes at RIP 0xffffffffffffffd6.
>
>  
>
> I wouldn’t say those lines are very helpful. I know that exception 14 on x86 is a page fault and that a pc value of 0 does not seem good
> (memory corruption?). 
>

Yes, this very much looks like a stack overwrite, since you are running
x86, return addresses may be popped from the stack, so this could lead
to that funky user_pc=0x0 if the IP register is reloaded with a random
value.

>  
>
> I wanted to get your opinion on our overall memory setup and see if there are obvious issues or if you had any recommendations of things to
> try.

Checking and controlling the thread stack consumption comes to mind,
looking for unconstrained writes to stack-based variables too.

> So we set up 1 EVL heap on startup of the application, and this heap is used for all dynamic allocations throughout the entire app as we
> also overrode global new/delete to use the EVL heap rather than malloc/free. The reason we overrode global new/delete is that we have a lot
> of STL objects as well as third party libraries that we are using in this app and it would become a very long and challenging process to actually
> go through and modify all those to use custom allocators.

Hopefully the STL won't use hidden synchronization via *libc mutexes,
otherwise this would be an issue. Re-routing new()/delete() away from
malloc()/free() to a dedicated allocator is common practice.

> So, in main(), we spawn a “Main” EVL thread. All other threads are spawned from
> this parent EVL thread. When the EVL Main thread starts, it sets a static flag that enables using the evl heap in the global news/deletes, and
> every library and process for the realtime app is created and started. We also prefault the EVL heap after it is initialized/created in order to try
> to avoid page faults while running the realtime loop. However, we still see occasional page faults in our realtime EVL threads while running our
> main realtime loop. I don’t understand how there could be a page fault if the EVL heap is large enough (verified) and prefaulted.
>
>  
>
> Now back to the issue mentioned at the beginning of this email – we are unsure at this time if the page fault error seen there is cause, result,
> or primary error causing these core dumps.

This page fault is a symptom, not the issue, the kernel detects that the
app has done something weird, like jumping to 0x0. If the stack is
wrecked, then other registers than IP might be trashed on reload,
leading to other forms of bad accesses. But they would all originate
from the same stack corruption issue.

> Typically when I see the occasional page faults while running other EVL threads in the app, I do
> not see core dumps – just the log from the kernel. The only reason why I figured I would at least run this past you was that I did not see any
> core dumps when disabling the flag in the global new/delete and just using malloc/free (of course I saw some in band switches, but that was
> it).

You may want to make sure to set ulimit to allow unlimited core dumps
for the process.

>
>  
>
> The issue of occasional page faults, handling dynamic memory allocation (using global new/delete), and now these possible memory
> corruption seg faults is becoming a larger concern for us. We would like to make sure that we are understanding how to use memory in an
> EVL application properly, and we would be interested to know if there are any recommended ways of tracking these down with an EVL
> application. I have tried to build/run with the gcc address sanitizer, but I was seeing issues attaching EVL threads when this was enabled. I
> have also tried running valgrind, but that has produced nothing useful. And, of course, I have run in gdb and the stack traces are not helpful.
> At this point, any guidance, thoughts, and/or recommendations would be greatly appreciated. I added some more clear/specific questions at
> the bottom.
>

An EVL application can run over valgrind, you may want to check if that
helps.

>  
>
> A few specific questions:
>
> 1 Is this a reasonable model to use for an EVL application, or do you expect the model to revolve more around static allocation?
>

No requirement for static allocation if that fits the bill.

> 2 Are you aware of people using EVL overriding the global new/delete to use the EVL heap?
>

Yes.

> 3 Do you have any tools for debugging the EVL heap or have you adapted any existing tools (such as valgrind) to debug the EVL heap?
>

As mentioned earlier, valgrind works out of the box for EVL apps.

> 4 Is there any known way of protecting against a stack overflow?
>

EVL threads are merely plain threads on real-time steroïds, so the
read-only canary area / red zone the *libc sets in order to detect
overflows is there too. So maybe the stack is not overflowing, but
trashed by some buffer overflow on some automatic variable.

> 5 PC value of 0 is never valid and we have no evidence that we have an uninitialized pointer in our C++ code. Is there anyway to use info from
>  EVL to help track down this issue?
>

No, it's the worst case, we loose track due to the bad jump, and there
is no link register to locate the caller on this architecture.

> 6 We are allocating 2GB of EVL Heap memory and pre-faulting all of it on startup. We also use pthread_attr_setstacksize to pre-allocate the
>  stack for each EVL thread we have. EVL still says that we get rare page faults. How is this possible? Are we missing something? (I have
>  attached our heap pre-faulting logic)
>

To answer this, we'd need to find out which kind of PF is that, whether
this is a PTE miss and which code actually triggers it. This would be a
Dovetail/x86 issue if any, EVL leaves all the mm stuff to the in-band
kernel.

-- 
Philippe.

RE: [External] - Re: EVL Memory

[Russell Johnson]

[-- Attachment #1.1: Type: text/plain, Size: 788 bytes --]

Thanks for the quick responses to our questions, that is very helpful! After 
more debugging we were able to find an issue on our end that got us farther 
with the issues I mentioned at the beginning of my previous email. Now, it has 
evolved into a something else it appears. I ran with the debug kernel and I am 
getting a ton of kernel splat that seems concerning. The callstacks go all 
throughout EVL, so I was wondering if you could make anything out of the 
output.. I also see two "switched inband while holding mutex" notifications 
from EVL along with a fault notification (grouped together) at regular 
intervals while running. So I am sure the two are related.. Trying to get a 
better understanding of which direction to start looking in. Any input helps - 
thanks!

Russell

[-- Attachment #1.2: corruption_stack.txt --]
[-- Type: text/plain, Size: 104366 bytes --]

Nov 11 21:25:07 10.243.1.49 [ 3877.744174] ------------[ cut here ]------------
Nov 11 21:25:07 10.243.1.49 [ 3877.744180] list_del corruption, ffff888151798240->next is LIST_POISON1 (dead000000000100)
Nov 11 21:25:07 10.243.1.49 [ 3877.744186] EVL: RxWorker0 switching in-band [pid=16134, excpt=6, __list_del_entry_valid+0xc5/0x120]
Nov 11 21:25:07 10.243.1.49 [ 3877.846773] WARNING: CPU: 8 PID: 16134 at lib/list_debug.c:55 __list_del_entry_valid+0xc5/0x120
Nov 11 21:25:07 10.243.1.49 [ 3877.846791] Modules linked in: xt_CHECKSUM xt_MASQUERADE tun netconsole ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 xt_conntrack ebtable_nat ip6table_nat ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_nat iptable_mangle iptable_security iptable_raw nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 ip_set nfnetlink ebtable_filter ebtables rfkill ip6table_filter ip6_tables iptable_filter sunrpc dm_mirror dm_region_hash dm_log dm_mod iTCO_wdt gpio_ich iTCO_vendor_support mxm_wmi x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm irqbypass crct10dif_pclmul crc32_pclmul ghash_clmulni_intel aesni_intel crypto_simd cryptd rapl mei_me i2c_i801 lpc_ich intel_cstate pcspkr intel_pch_thermal input_leds sg joydev ftdi_sio i2c_smbus mfd_core mei acpi_ipmi ioatdma ipmi_si ipmi_devintf ipmi_msghandler wmi ip_tables xfs libcrc32c ast drm_vram_helper sd_mod drm_kms_helper sr_mod cdrom t10_pi syscopyarea sysfillrect sysimgblt fb_sys_fops drm_ttm_helper ttm drm
Nov 11 21:25:07 10.243.1.49
Nov 11 21:25:07 10.243.1.49 [ 3877.846956]  ixgbe igb ahci libahci crc32c_intel libata mdio ptp i2c_algo_bit pps_core dca
Nov 11 21:25:07 10.243.1.49 [ 3877.846978] CPU: 8 PID: 16134 Comm: RxWorker0 Tainted: G        W         5.15.77evl-gae6080e09d9a #1
Nov 11 21:25:07 10.243.1.49 [ 3877.846986] Hardware name: Supermicro Super Server/X10SDV-6C-TLN4F, BIOS 1.3 02/13/2018
Nov 11 21:25:07 10.243.1.49 [ 3877.846989] IRQ stage: Linux
Nov 11 21:25:07 10.243.1.49 [ 3877.846991] RIP: 0010:__list_del_entry_valid+0xc5/0x120
Nov 11 21:25:07 10.243.1.49 [ 3877.847000] Code: c0 eb e0 48 89 de 48 c7 c7 00 0c 82 a5 e8 7c fa 8b 00 0f 0b 31 c0 eb cb 4c 89 e2 48 89 de 48 c7 c7 60 0c 82 a5 e8 64 fa 8b 00 <0f> 0b 31 c0 eb b3 4c 89 ea 48 89 de 48 c7 c7 c0 0c 82 a5 e8 4c fa
Nov 11 21:25:07 10.243.1.49 [ 3877.847005] RSP: 0018:ffff88816588fac8 EFLAGS: 00010082
Nov 11 21:25:07 10.243.1.49 [ 3877.847010] RAX: 0000000000000000 RBX: ffff888151798240 RCX: 0000000000000027
Nov 11 21:25:08 10.243.1.49 [ 3877.847014] RDX: 0000000000000027 RSI: dffffc0000000000 RDI: ffff88841c227b18
Nov 11 21:25:08 10.243.1.49 [ 3877.847018] RBP: ffff88816588fae0 R08: ffffed1083844f64 R09: ffffed1083844f64
Nov 11 21:25:08 10.243.1.49 [ 3877.847022] R10: ffff88841c227b1b R11: ffffed1083844f63 R12: dead000000000100
Nov 11 21:25:08 10.243.1.49 [ 3877.847026] R13: dead000000000122 R14: ffff888151798000 R15: ffff888148b3a000
Nov 11 21:25:08 10.243.1.49 [ 3877.847030] FS:  00007f7313fff700(0000) GS:ffff88841c200000(0000) knlGS:0000000000000000
Nov 11 21:25:08 10.243.1.49 [ 3877.847035] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Nov 11 21:25:08 10.243.1.49 [ 3877.847039] CR2: 00007f4d49b09160 CR3: 000000016e080005 CR4: 00000000003706e0
Nov 11 21:25:08 10.243.1.49 [ 3877.847043] Call Trace:
Nov 11 21:25:08 10.243.1.49 [ 3877.847045]  <TASK>
Nov 11 21:25:08 10.243.1.49 [ 3877.847059]  __untrack_event.part.9+0x26/0xe0
Nov 11 21:25:08 10.243.1.49 [ 3877.847096]  monitor_oob_ioctl+0x1356/0x1490
Nov 11 21:25:08 10.243.1.49 [ 3877.847154]  ? enter_monitor+0x160/0x160
Nov 11 21:25:08 10.243.1.49 [ 3877.847170]  ? __kasan_check_read+0x11/0x20
Nov 11 21:25:08 10.243.1.49 [ 3877.847187]  ? do_raw_spin_unlock+0x97/0x140
Nov 11 21:25:08 10.243.1.49 [ 3877.847213]  ? do_clock_tick+0x40a/0x630
Nov 11 21:25:08 10.243.1.49 [ 3877.847224]  ? rcu_read_lock_held_common+0x25/0x70
Nov 11 21:25:08 10.243.1.49 [ 3877.847265]  ? do_raw_spin_lock+0x119/0x1e0
Nov 11 21:25:08 10.243.1.49 [ 3877.847281]  ? spin_dump+0x90/0x90
Nov 11 21:25:08 10.243.1.49 [ 3877.847316]  ? __kasan_check_read+0x11/0x20
Nov 11 21:25:08 10.243.1.49 [ 3877.847367]  EVL_ioctl+0x81/0xf0
Nov 11 21:25:08 10.243.1.49 [ 3877.847386]  do_oob_syscall+0x699/0x6b0
Nov 11 21:25:08 10.243.1.49 [ 3877.847400]  ? prepare_for_signal+0x180/0x180
Nov 11 21:25:08 10.243.1.49 [ 3877.847414]  ? do_oob_irq+0x283/0x7e0
Nov 11 21:25:08 10.243.1.49 [ 3877.847420]  ? rcu_dynticks_eqs_enter+0xe/0x20
Nov 11 21:25:08 10.243.1.49 [ 3877.847440]  handle_oob_syscall+0x189/0x230
Nov 11 21:25:08 10.243.1.49 [ 3877.847455]  ? handle_pipelined_syscall+0x820/0x820
Nov 11 21:25:08 10.243.1.49 [ 3877.847473]  ? rcu_read_lock_sched_held+0x5b/0xd0
Nov 11 21:25:08 10.243.1.49 [ 3877.847494]  ? rcu_read_lock_any_held.part.15+0x20/0x20
Nov 11 21:25:08 10.243.1.49 [ 3877.847536]  pipeline_syscall+0xa7/0x1c0
Nov 11 21:25:08 10.243.1.49 [ 3877.847575]  syscall_enter_from_user_mode+0x47/0x110
Nov 11 21:25:08 10.243.1.49 [ 3877.847601]  do_syscall_64+0x15/0xb0
Nov 11 21:25:08 10.243.1.49 [ 3877.847622]  entry_SYSCALL_64_after_hwframe+0x61/0xcb
Nov 11 21:25:08 10.243.1.49 [ 3877.847636] RIP: 0033:0x7f735de5a940
Nov 11 21:25:08 10.243.1.49 [ 3877.847648] Code: 77 ff ff bf 02 00 00 10 8b 44 24 0c 48 98 48 89 c6 48 8b 04 24 48 89 c2 4c 8b 54 24 38 41 b8 00 00 00 00 b8 9d 00 00 00 0f 05 <48> 89 44 24 30 48 8b 44 24 30 89 44 24 2c 8b 44 24 28 be 00 00 00
Nov 11 21:25:08 10.243.1.49 [ 3877.847658] RSP: 002b:00007f7313fef340 EFLAGS: 00000202 ORIG_RAX: 000000000000009d
Nov 11 21:25:08 10.243.1.49 [ 3877.847669] RAX: ffffffffffffffda RBX: 0000000000000f26 RCX: 00007f735de5a940
Nov 11 21:25:08 10.243.1.49 [ 3877.847677] RDX: 00000000c0186d03 RSI: 000000000000000d RDI: 0000000010000002
Nov 11 21:25:08 10.243.1.49 [ 3877.847683] RBP: 00007f7313fef520 R08: 0000000000000000 R09: 00007ffc09389080
Nov 11 21:25:08 10.243.1.49 [ 3877.847691] R10: 00007f7313fef4f0 R11: 0000000000000202 R12: 00007f7313fef600
Nov 11 21:25:08 10.243.1.49 [ 3877.847697] R13: 0000000000000000 R14: 00000000022cb5bd R15: 00007f7313fef580
Nov 11 21:25:08 10.243.1.49 [ 3877.847777]  </TASK>
Nov 11 21:25:08 10.243.1.49 [ 3877.847779] irq event stamp: 889
Nov 11 21:25:08 10.243.1.49 [ 3877.847782] hardirqs last  enabled at (889): [<ffffffffa51330bc>] _raw_spin_unlock_irqrestore+0x4c/0x50
Nov 11 21:25:08 10.243.1.49 [ 3877.847791] hardirqs last disabled at (888): [<ffffffffa5132e2b>] _raw_spin_lock_irqsave+0x1b/0x60
Nov 11 21:25:08 10.243.1.49 [ 3877.847798] softirqs last  enabled at (286): [<ffffffffa5400403>] __do_softirq+0x403/0x57b
Nov 11 21:25:09 10.243.1.49 [ 3877.847805] softirqs last disabled at (275): [<ffffffffa40d261d>] irq_exit_rcu+0x7d/0x150
Nov 11 21:25:09 10.243.1.49 [ 3877.847815] ---[ end trace 87ffbd6a737b5b80 ]---
Nov 11 21:25:09 10.243.1.49 [ 3879.476338] EVL: RxWorker0 resuming out-of-band [pid=16134, excpt=6, __list_del_entry_valid+0xc7/0x120]
Nov 11 21:25:09 10.243.1.49 [ 3879.509373] ------------[ cut here ]------------
Nov 11 21:25:09 10.243.1.49 [ 3879.509377] WARNING: CPU: 8 PID: 16134 at kernel/evl/thread.c:580 evl_sleep_on+0x9d/0xb0
Nov 11 21:25:09 10.243.1.49 [ 3879.509391] Modules linked in: xt_CHECKSUM xt_MASQUERADE tun netconsole ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 xt_conntrack ebtable_nat ip6table_nat ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_nat iptable_mangle iptable_security iptable_raw nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 ip_set nfnetlink ebtable_filter ebtables rfkill ip6table_filter ip6_tables iptable_filter sunrpc dm_mirror dm_region_hash dm_log dm_mod iTCO_wdt gpio_ich iTCO_vendor_support mxm_wmi x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm irqbypass crct10dif_pclmul crc32_pclmul ghash_clmulni_intel aesni_intel crypto_simd cryptd rapl mei_me i2c_i801 lpc_ich intel_cstate pcspkr intel_pch_thermal input_leds sg joydev ftdi_sio i2c_smbus mfd_core mei acpi_ipmi ioatdma ipmi_si ipmi_devintf ipmi_msghandler wmi ip_tables xfs libcrc32c ast drm_vram_helper sd_mod drm_kms_helper sr_mod cdrom t10_pi syscopyarea sysfillrect sysimgblt fb_sys_fops drm_ttm_helper ttm drm
Nov 11 21:25:09 10.243.1.49
Nov 11 21:25:09 10.243.1.49 [ 3879.509575]  ixgbe igb ahci libahci crc32c_intel libata mdio ptp i2c_algo_bit pps_core dca
Nov 11 21:25:09 10.243.1.49 [ 3879.509597] CPU: 8 PID: 16134 Comm: RxWorker0 Tainted: G        W         5.15.77evl-gae6080e09d9a #1
Nov 11 21:25:09 10.243.1.49 [ 3879.509605] Hardware name: Supermicro Super Server/X10SDV-6C-TLN4F, BIOS 1.3 02/13/2018
Nov 11 21:25:09 10.243.1.49 [ 3879.509608] IRQ stage: Linux
Nov 11 21:25:09 10.243.1.49 [ 3879.509610] RIP: 0010:evl_sleep_on+0x9d/0xb0
Nov 11 21:25:09 10.243.1.49 [ 3879.509618] Code: f8 ff ff 4c 89 e7 e8 42 97 e9 ff 48 89 df e8 3a 97 e9 ff ff 75 c8 9d 48 83 c4 10 5b 41 5c 41 5d 41 5e 41 5f 5d c3 cc cc cc cc <0f> 0b eb a1 0f 1f 44 00 00 66 2e 0f 1f 84 00 00 00 00 00 55 48 89
Nov 11 21:25:09 10.243.1.49 [ 3879.509623] RSP: 0018:ffff88816588f9a0 EFLAGS: 00010046
Nov 11 21:25:09 10.243.1.49 [ 3879.509628] RAX: 0000000000000000 RBX: ffff888148b3a000 RCX: ffffffffa43034a0
Nov 11 21:25:09 10.243.1.49 [ 3879.509633] RDX: dffffc0000000000 RSI: 0000000000000000 RDI: ffff88816d77c798
Nov 11 21:25:09 10.243.1.49 [ 3879.509637] RBP: ffff88816588f9d8 R08: ffffed1029167401 R09: ffffed1029167401
Nov 11 21:25:09 10.243.1.49 [ 3879.509641] R10: ffff888148b3a003 R11: ffffed1029167400 R12: ffff888148b3a090
Nov 11 21:25:09 10.243.1.49 [ 3879.509644] R13: 0000000000000000 R14: ffffffffa6209b40 R15: ffff88815179a170
Nov 11 21:25:09 10.243.1.49 [ 3879.509648] FS:  00007f7313fff700(0000) GS:ffff88841c200000(0000) knlGS:0000000000000000
Nov 11 21:25:09 10.243.1.49 [ 3879.509653] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Nov 11 21:25:09 10.243.1.49 [ 3879.509657] CR2: 00007f4d49b09160 CR3: 000000016e080005 CR4: 00000000003706e0
Nov 11 21:25:09 10.243.1.49 [ 3879.509661] Call Trace:
Nov 11 21:25:09 10.243.1.49 [ 3879.509663]  <TASK>
Nov 11 21:25:09 10.243.1.49 [ 3879.509679]  evl_lock_mutex_timeout+0x7f2/0x1100
Nov 11 21:25:09 10.243.1.49 [ 3879.509709]  ? evl_trylock_mutex+0x1c0/0x1c0
Nov 11 21:25:09 10.243.1.49 [ 3879.509724]  ? __list_del_entry_valid+0xc7/0x120
Nov 11 21:25:09 10.243.1.49 [ 3879.509739]  ? __list_del_entry_valid+0xc5/0x120
Nov 11 21:25:09 10.243.1.49 [ 3879.509762]  monitor_oob_ioctl+0xf95/0x1490
Nov 11 21:25:09 10.243.1.49 [ 3879.509792]  ? enter_monitor+0x160/0x160
Nov 11 21:25:10 10.243.1.49 [ 3879.509800]  ? __kasan_check_read+0x11/0x20
Nov 11 21:25:10 10.243.1.49 [ 3879.509810]  ? do_raw_spin_unlock+0x97/0x140
Nov 11 21:25:10 10.243.1.49 [ 3879.509823]  ? do_clock_tick+0x40a/0x630
Nov 11 21:25:10 10.243.1.49 [ 3879.509830]  ? rcu_read_lock_held_common+0x25/0x70
Nov 11 21:25:10 10.243.1.49 [ 3879.509853]  ? do_raw_spin_lock+0x119/0x1e0
Nov 11 21:25:10 10.243.1.49 [ 3879.509862]  ? spin_dump+0x90/0x90
Nov 11 21:25:10 10.243.1.49 [ 3879.509879]  ? __kasan_check_read+0x11/0x20
Nov 11 21:25:10 10.243.1.49 [ 3879.509907]  EVL_ioctl+0x81/0xf0
Nov 11 21:25:10 10.243.1.49 [ 3879.509924]  do_oob_syscall+0x699/0x6b0
Nov 11 21:25:10 10.243.1.49 [ 3879.509937]  ? prepare_for_signal+0x180/0x180
Nov 11 21:25:10 10.243.1.49 [ 3879.509951]  ? do_oob_irq+0x283/0x7e0
Nov 11 21:25:10 10.243.1.49 [ 3879.509956]  ? rcu_dynticks_eqs_enter+0xe/0x20
Nov 11 21:25:10 10.243.1.49 [ 3879.509976]  handle_oob_syscall+0x189/0x230
Nov 11 21:25:10 10.243.1.49 [ 3879.509990]  ? handle_pipelined_syscall+0x820/0x820
Nov 11 21:25:10 10.243.1.49 [ 3879.510007]  ? rcu_read_lock_sched_held+0x5b/0xd0
Nov 11 21:25:10 10.243.1.49 [ 3879.510017]  ? rcu_read_lock_any_held.part.15+0x20/0x20
Nov 11 21:25:10 10.243.1.49 [ 3879.510038]  pipeline_syscall+0xa7/0x1c0
Nov 11 21:25:10 10.243.1.49 [ 3879.510057]  syscall_enter_from_user_mode+0x47/0x110
Nov 11 21:25:10 10.243.1.49 [ 3879.510070]  do_syscall_64+0x15/0xb0
Nov 11 21:25:10 10.243.1.49 [ 3879.510080]  entry_SYSCALL_64_after_hwframe+0x61/0xcb
Nov 11 21:25:10 10.243.1.49 [ 3879.510087] RIP: 0033:0x7f735de5a940
Nov 11 21:25:10 10.243.1.49 [ 3879.510094] Code: 77 ff ff bf 02 00 00 10 8b 44 24 0c 48 98 48 89 c6 48 8b 04 24 48 89 c2 4c 8b 54 24 38 41 b8 00 00 00 00 b8 9d 00 00 00 0f 05 <48> 89 44 24 30 48 8b 44 24 30 89 44 24 2c 8b 44 24 28 be 00 00 00
Nov 11 21:25:10 10.243.1.49 [ 3879.510099] RSP: 002b:00007f7313fef340 EFLAGS: 00000202 ORIG_RAX: 000000000000009d
Nov 11 21:25:10 10.243.1.49 [ 3879.510105] RAX: ffffffffffffffda RBX: 0000000000000f26 RCX: 00007f735de5a940
Nov 11 21:25:10 10.243.1.49 [ 3879.510108] RDX: 00000000c0186d03 RSI: 000000000000000d RDI: 0000000010000002
Nov 11 21:25:10 10.243.1.49 [ 3879.510112] RBP: 00007f7313fef520 R08: 0000000000000000 R09: 00007ffc09389080
Nov 11 21:25:10 10.243.1.49 [ 3879.510115] R10: 00007f7313fef4f0 R11: 0000000000000202 R12: 00007f7313fef600
Nov 11 21:25:10 10.243.1.49 [ 3879.510119] R13: 0000000000000000 R14: 00000000022cb5bd R15: 00007f7313fef580
Nov 11 21:25:10 10.243.1.49 [ 3879.510155]  </TASK>
Nov 11 21:25:10 10.243.1.49 [ 3879.510157] irq event stamp: 904
Nov 11 21:25:10 10.243.1.49 [ 3879.510159] hardirqs last  enabled at (903): [<ffffffffa41aaac2>] __up_console_sem+0x62/0x70
Nov 11 21:25:10 10.243.1.49 [ 3879.510169] hardirqs last disabled at (904): [<ffffffffa5115791>] handle_bug+0x81/0xa0
Nov 11 21:25:10 10.243.1.49 [ 3879.510177] softirqs last  enabled at (286): [<ffffffffa5400403>] __do_softirq+0x403/0x57b
Nov 11 21:25:10 10.243.1.49 [ 3879.510185] softirqs last disabled at (275): [<ffffffffa40d261d>] irq_exit_rcu+0x7d/0x150
Nov 11 21:25:10 10.243.1.49 [ 3879.510195] ---[ end trace 87ffbd6a737b5b81 ]---
Nov 11 21:25:10 10.243.1.49 [ 3881.205548] ------------[ cut here ]------------
Nov 11 21:25:10 10.243.1.49 [ 3881.224515] syscall 157 left IRQs disabled
Nov 11 21:25:10 10.243.1.49 [ 3881.224599] WARNING: CPU: 8 PID: 16134 at kernel/entry/common.c:353 syscall_exit_to_user_mode_prepare+0x37/0x270
Nov 11 21:25:11 10.243.1.49 [ 3881.277186] Modules linked in: xt_CHECKSUM xt_MASQUERADE tun netconsole ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 xt_conntrack ebtable_nat ip6table_nat ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_nat iptable_mangle iptable_security iptable_raw nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 ip_set nfnetlink ebtable_filter ebtables rfkill ip6table_filter ip6_tables iptable_filter sunrpc dm_mirror dm_region_hash dm_log dm_mod iTCO_wdt gpio_ich iTCO_vendor_support mxm_wmi x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm irqbypass crct10dif_pclmul crc32_pclmul ghash_clmulni_intel aesni_intel crypto_simd cryptd rapl mei_me i2c_i801 lpc_ich intel_cstate pcspkr intel_pch_thermal input_leds sg joydev ftdi_sio i2c_smbus mfd_core mei acpi_ipmi ioatdma ipmi_si ipmi_devintf ipmi_msghandler wmi ip_tables xfs libcrc32c ast drm_vram_helper sd_mod drm_kms_helper sr_mod cdrom t10_pi syscopyarea sysfillrect sysimgblt fb_sys_fops drm_ttm_helper ttm drm
Nov 11 21:25:11 10.243.1.49
Nov 11 21:25:11 10.243.1.49 [ 3881.277744]  ixgbe igb ahci libahci crc32c_intel libata mdio ptp i2c_algo_bit pps_core dca
Nov 11 21:25:11 10.243.1.49 [ 3881.608230] CPU: 8 PID: 16134 Comm: RxWorker0 Tainted: G        W         5.15.77evl-gae6080e09d9a #1
Nov 11 21:25:11 10.243.1.49 [ 3881.641467] Hardware name: Supermicro Super Server/X10SDV-6C-TLN4F, BIOS 1.3 02/13/2018
Nov 11 21:25:11 10.243.1.49 [ 3881.671175] IRQ stage: Linux
Nov 11 21:25:11 10.243.1.49 [ 3881.685407] RIP: 0010:syscall_exit_to_user_mode_prepare+0x37/0x270
Nov 11 21:25:11 10.243.1.49 [ 3881.709677] Code: 41 55 41 54 49 89 fc 53 4c 8b 68 08 48 8b 5f 78 eb 36 e8 0c fe f2 00 85 c0 74 1c 48 63 f3 48 c7 c7 70 66 ca a5 e8 42 f0 ec 00 <0f> 0b fb e8 e1 a1 0d 00 e8 bc af fd ff 41 f6 c5 76 75 3a 5b 41 5c
Nov 11 21:25:11 10.243.1.49 [ 3881.778001] RSP: 0018:ffff88816588ff00 EFLAGS: 00010282
Nov 11 21:25:11 10.243.1.49 [ 3881.800036] RAX: 0000000000000000 RBX: 000000000000009d RCX: 0000000000000027
Nov 11 21:25:11 10.243.1.49 [ 3881.827850] RDX: 0000000000000027 RSI: dffffc0000000000 RDI: ffff88841c227b18
Nov 11 21:25:11 10.243.1.49 [ 3881.856667] RBP: ffff88816588ff18 R08: ffffed1083844f64 R09: ffffed1083844f64
Nov 11 21:25:11 10.243.1.49 [ 3881.884352] R10: ffff88841c227b1b R11: ffffed1083844f63 R12: ffff88816588ff58
Nov 11 21:25:11 10.243.1.49 [ 3881.912059] R13: 0000000000000010 R14: 0000000000000000 R15: 0000000000000000
Nov 11 21:25:11 10.243.1.49 [ 3881.939817] FS:  00007f7313fff700(0000) GS:ffff88841c200000(0000) knlGS:0000000000000000
Nov 11 21:25:11 10.243.1.49 [ 3881.970540] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Nov 11 21:25:11 10.243.1.49 [ 3881.994183] CR2: 00007f4d49b09160 CR3: 000000016e080005 CR4: 00000000003706e0
Nov 11 21:25:11 10.243.1.49 [ 3882.022110] Call Trace:
Nov 11 21:25:11 10.243.1.49 [ 3882.035924]  <TASK>
Nov 11 21:25:11 10.243.1.49 [ 3882.048991]  syscall_exit_to_user_mode+0xd/0x90
Nov 11 21:25:11 10.243.1.49 [ 3882.069055]  do_syscall_64+0x50/0xb0
Nov 11 21:25:11 10.243.1.49 [ 3882.086281]  entry_SYSCALL_64_after_hwframe+0x61/0xcb
Nov 11 21:25:11 10.243.1.49 [ 3882.108130] RIP: 0033:0x7f735de5a940
Nov 11 21:25:11 10.243.1.49 [ 3882.125201] Code: 77 ff ff bf 02 00 00 10 8b 44 24 0c 48 98 48 89 c6 48 8b 04 24 48 89 c2 4c 8b 54 24 38 41 b8 00 00 00 00 b8 9d 00 00 00 0f 05 <48> 89 44 24 30 48 8b 44 24 30 89 44 24 2c 8b 44 24 28 be 00 00 00
Nov 11 21:25:11 10.243.1.49 [ 3882.194404] RSP: 002b:00007f7313fef340 EFLAGS: 00000202 ORIG_RAX: 000000000000009d
Nov 11 21:25:11 10.243.1.49 [ 3882.223637] RAX: 0000000000000000 RBX: 0000000000000f26 RCX: 00007f735de5a940
Nov 11 21:25:11 10.243.1.49 [ 3882.252758] RDX: 00000000c0186d03 RSI: 000000000000000d RDI: 0000000010000002
Nov 11 21:25:11 10.243.1.49 [ 3882.280720] RBP: 00007f7313fef520 R08: 0000000000000000 R09: 00007ffc09389080
Nov 11 21:25:11 10.243.1.49 [ 3882.308846] R10: 00007f7313fef4f0 R11: 0000000000000202 R12: 00007f7313fef600
Nov 11 21:25:11 10.243.1.49 [ 3882.336822] R13: 0000000000000000 R14: 00000000022cb5bd R15: 00007f7313fef580
Nov 11 21:25:11 10.243.1.49 [ 3882.364830]  </TASK>
Nov 11 21:25:11 10.243.1.49 [ 3882.377882] irq event stamp: 1573
Nov 11 21:25:11 10.243.1.49 [ 3882.394225] hardirqs last  enabled at (1581): [<ffffffffa41aaac2>] __up_console_sem+0x62/0x70
Nov 11 21:25:12 10.243.1.49 [ 3882.426450] hardirqs last disabled at (1588): [<ffffffffa41aaaa7>] __up_console_sem+0x47/0x70
Nov 11 21:25:12 10.243.1.49 [ 3882.458614] softirqs last  enabled at (286): [<ffffffffa5400403>] __do_softirq+0x403/0x57b
Nov 11 21:25:12 10.243.1.49 [ 3882.490022] softirqs last disabled at (275): [<ffffffffa40d261d>] irq_exit_rcu+0x7d/0x150
Nov 11 21:25:12 10.243.1.49 [ 3882.521175] ---[ end trace 87ffbd6a737b5b82 ]---
Nov 11 21:25:31 10.243.1.49 [ 3902.026568] ------------[ cut here ]------------
Nov 11 21:25:31 10.243.1.49 [ 3902.026575] list_del corruption, ffff888151798240->next is LIST_POISON1 (dead000000000100)
Nov 11 21:25:31 10.243.1.49 [ 3902.026585] EVL: RxWorker1 switching in-band [pid=16135, excpt=6, __list_del_entry_valid+0xc5/0x120]
Nov 11 21:25:31 10.243.1.49 [ 3902.131088] WARNING: CPU: 9 PID: 16135 at lib/list_debug.c:55 __list_del_entry_valid+0xc5/0x120
Nov 11 21:25:31 10.243.1.49 [ 3902.131111] Modules linked in: xt_CHECKSUM xt_MASQUERADE tun netconsole ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 xt_conntrack ebtable_nat ip6table_nat ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_nat iptable_mangle iptable_security iptable_raw nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 ip_set nfnetlink ebtable_filter ebtables rfkill ip6table_filter ip6_tables iptable_filter sunrpc dm_mirror dm_region_hash dm_log dm_mod iTCO_wdt gpio_ich iTCO_vendor_support mxm_wmi x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm irqbypass crct10dif_pclmul crc32_pclmul ghash_clmulni_intel aesni_intel crypto_simd cryptd rapl mei_me i2c_i801 lpc_ich intel_cstate pcspkr intel_pch_thermal input_leds sg joydev ftdi_sio i2c_smbus mfd_core mei acpi_ipmi ioatdma ipmi_si ipmi_devintf ipmi_msghandler wmi ip_tables xfs libcrc32c ast drm_vram_helper sd_mod drm_kms_helper sr_mod cdrom t10_pi syscopyarea sysfillrect sysimgblt fb_sys_fops drm_ttm_helper ttm drm
Nov 11 21:25:31 10.243.1.49
Nov 11 21:25:32 10.243.1.49 [ 3902.131434]  ixgbe igb ahci libahci crc32c_intel libata mdio ptp i2c_algo_bit pps_core dca
Nov 11 21:25:32 10.243.1.49 [ 3902.131478] CPU: 9 PID: 16135 Comm: RxWorker1 Tainted: G        W         5.15.77evl-gae6080e09d9a #1
Nov 11 21:25:32 10.243.1.49 [ 3902.131491] Hardware name: Supermicro Super Server/X10SDV-6C-TLN4F, BIOS 1.3 02/13/2018
Nov 11 21:25:32 10.243.1.49 [ 3902.131496] IRQ stage: Linux
Nov 11 21:25:32 10.243.1.49 [ 3902.131501] RIP: 0010:__list_del_entry_valid+0xc5/0x120
Nov 11 21:25:32 10.243.1.49 [ 3902.131516] Code: c0 eb e0 48 89 de 48 c7 c7 00 0c 82 a5 e8 7c fa 8b 00 0f 0b 31 c0 eb cb 4c 89 e2 48 89 de 48 c7 c7 60 0c 82 a5 e8 64 fa 8b 00 <0f> 0b 31 c0 eb b3 4c 89 ea 48 89 de 48 c7 c7 c0 0c 82 a5 e8 4c fa
Nov 11 21:25:32 10.243.1.49 [ 3902.131526] RSP: 0018:ffff88815ef47ac8 EFLAGS: 00010082
Nov 11 21:25:32 10.243.1.49 [ 3902.131535] RAX: 0000000000000000 RBX: ffff888151798240 RCX: 0000000000000027
Nov 11 21:25:32 10.243.1.49 [ 3902.131543] RDX: 0000000000000027 RSI: dffffc0000000000 RDI: ffff88841c2a7b18
Nov 11 21:25:32 10.243.1.49 [ 3902.131551] RBP: ffff88815ef47ae0 R08: ffffed1083854f64 R09: ffffed1083854f64
Nov 11 21:25:32 10.243.1.49 [ 3902.131559] R10: ffff88841c2a7b1b R11: ffffed1083854f63 R12: dead000000000100
Nov 11 21:25:32 10.243.1.49 [ 3902.131567] R13: dead000000000122 R14: ffff888151798000 R15: ffff888145380000
Nov 11 21:25:32 10.243.1.49 [ 3902.131576] FS:  00007f7312ffe700(0000) GS:ffff88841c280000(0000) knlGS:0000000000000000
Nov 11 21:25:32 10.243.1.49 [ 3902.131586] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Nov 11 21:25:32 10.243.1.49 [ 3902.131594] CR2: 0000000000000000 CR3: 000000016e080005 CR4: 00000000003706e0
Nov 11 21:25:32 10.243.1.49 [ 3902.131601] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Nov 11 21:25:32 10.243.1.49 [ 3902.131608] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Nov 11 21:25:32 10.243.1.49 [ 3902.131615] Call Trace:
Nov 11 21:25:32 10.243.1.49 [ 3902.131618]  <TASK>
Nov 11 21:25:32 10.243.1.49 [ 3902.131637]  __untrack_event.part.9+0x26/0xe0
Nov 11 21:25:32 10.243.1.49 [ 3902.131675]  monitor_oob_ioctl+0x1356/0x1490
Nov 11 21:25:32 10.243.1.49 [ 3902.131736]  ? enter_monitor+0x160/0x160
Nov 11 21:25:32 10.243.1.49 [ 3902.131774]  ? __kasan_check_read+0x11/0x20
Nov 11 21:25:32 10.243.1.49 [ 3902.131795]  ? rcu_read_lock_held_common+0x25/0x70
Nov 11 21:25:32 10.243.1.49 [ 3902.131843]  ? do_raw_spin_lock+0x119/0x1e0
Nov 11 21:25:32 10.243.1.49 [ 3902.131861]  ? spin_dump+0x90/0x90
Nov 11 21:25:32 10.243.1.49 [ 3902.131899]  ? __kasan_check_read+0x11/0x20
Nov 11 21:25:32 10.243.1.49 [ 3902.131960]  EVL_ioctl+0x81/0xf0
Nov 11 21:25:32 10.243.1.49 [ 3902.131996]  do_oob_syscall+0x699/0x6b0
Nov 11 21:25:32 10.243.1.49 [ 3902.132025]  ? prepare_for_signal+0x180/0x180
Nov 11 21:25:33 10.243.1.49 [ 3902.132054]  ? do_oob_irq+0x283/0x7e0
Nov 11 21:25:33 10.243.1.49 [ 3902.132097]  handle_oob_syscall+0x189/0x230
Nov 11 21:25:33 10.243.1.49 [ 3902.132126]  ? handle_pipelined_syscall+0x820/0x820
Nov 11 21:25:33 10.243.1.49 [ 3902.132206]  pipeline_syscall+0xa7/0x1c0
Nov 11 21:25:33 10.243.1.49 [ 3902.132245]  syscall_enter_from_user_mode+0x47/0x110
Nov 11 21:25:33 10.243.1.49 [ 3902.132272]  do_syscall_64+0x15/0xb0
Nov 11 21:25:33 10.243.1.49 [ 3902.132292]  entry_SYSCALL_64_after_hwframe+0x61/0xcb
Nov 11 21:25:33 10.243.1.49 [ 3902.132305] RIP: 0033:0x7f735de5a940
Nov 11 21:25:33 10.243.1.49 [ 3902.132316] Code: 77 ff ff bf 02 00 00 10 8b 44 24 0c 48 98 48 89 c6 48 8b 04 24 48 89 c2 4c 8b 54 24 38 41 b8 00 00 00 00 b8 9d 00 00 00 0f 05 <48> 89 44 24 30 48 8b 44 24 30 89 44 24 2c 8b 44 24 28 be 00 00 00
Nov 11 21:25:33 10.243.1.49 [ 3902.132326] RSP: 002b:00007f7312fee340 EFLAGS: 00000202 ORIG_RAX: 000000000000009d
Nov 11 21:25:33 10.243.1.49 [ 3902.132337] RAX: ffffffffffffffda RBX: 0000000000000f3e RCX: 00007f735de5a940
Nov 11 21:25:33 10.243.1.49 [ 3902.132345] RDX: 00000000c0186d03 RSI: 000000000000000d RDI: 0000000010000002
Nov 11 21:25:33 10.243.1.49 [ 3902.132352] RBP: 00007f7312fee520 R08: 0000000000000000 R09: 00007ffc09389080
Nov 11 21:25:33 10.243.1.49 [ 3902.132359] R10: 00007f7312fee4f0 R11: 0000000000000202 R12: 00007f7312fee600
Nov 11 21:25:33 10.243.1.49 [ 3902.132367] R13: 0000000000000000 R14: 000000001302529e R15: 00007f7312fee580
Nov 11 21:25:33 10.243.1.49 [ 3902.132448]  </TASK>
Nov 11 21:25:33 10.243.1.49 [ 3902.132452] irq event stamp: 673
Nov 11 21:25:33 10.243.1.49 [ 3902.132455] hardirqs last  enabled at (673): [<ffffffffa51330bc>] _raw_spin_unlock_irqrestore+0x4c/0x50
Nov 11 21:25:33 10.243.1.49 [ 3902.132470] hardirqs last disabled at (672): [<ffffffffa5132e2b>] _raw_spin_lock_irqsave+0x1b/0x60
Nov 11 21:25:33 10.243.1.49 [ 3902.132484] softirqs last  enabled at (32): [<ffffffffa5400403>] __do_softirq+0x403/0x57b
Nov 11 21:25:33 10.243.1.49 [ 3902.132499] softirqs last disabled at (25): [<ffffffffa40d261d>] irq_exit_rcu+0x7d/0x150
Nov 11 21:25:33 10.243.1.49 [ 3902.132515] ---[ end trace 87ffbd6a737b5b83 ]---
Nov 11 21:25:33 10.243.1.49 [ 3904.104311] EVL: RxWorker1 resuming out-of-band [pid=16135, excpt=6, __list_del_entry_valid+0xc7/0x120]
Nov 11 21:25:33 10.243.1.49 [ 3904.141268] ------------[ cut here ]------------
Nov 11 21:25:33 10.243.1.49 [ 3904.159499] syscall 157 left IRQs disabled
Nov 11 21:25:33 10.243.1.49 [ 3904.159561] WARNING: CPU: 9 PID: 16135 at kernel/entry/common.c:353 syscall_exit_to_user_mode_prepare+0x37/0x270
Nov 11 21:25:34 10.243.1.49 [ 3904.211325] Modules linked in: xt_CHECKSUM xt_MASQUERADE tun netconsole ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 xt_conntrack ebtable_nat ip6table_nat ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_nat iptable_mangle iptable_security iptable_raw nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 ip_set nfnetlink ebtable_filter ebtables rfkill ip6table_filter ip6_tables iptable_filter sunrpc dm_mirror dm_region_hash dm_log dm_mod iTCO_wdt gpio_ich iTCO_vendor_support mxm_wmi x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm irqbypass crct10dif_pclmul crc32_pclmul ghash_clmulni_intel aesni_intel crypto_simd cryptd rapl mei_me i2c_i801 lpc_ich intel_cstate pcspkr intel_pch_thermal input_leds sg joydev ftdi_sio i2c_smbus mfd_core mei acpi_ipmi ioatdma ipmi_si ipmi_devintf ipmi_msghandler wmi ip_tables xfs libcrc32c ast drm_vram_helper sd_mod drm_kms_helper sr_mod cdrom t10_pi syscopyarea sysfillrect sysimgblt fb_sys_fops drm_ttm_helper ttm drm
Nov 11 21:25:34 10.243.1.49
Nov 11 21:25:34 10.243.1.49 [ 3904.211877]  ixgbe igb ahci libahci crc32c_intel libata mdio ptp i2c_algo_bit pps_core dca
Nov 11 21:25:34 10.243.1.49 [ 3904.545198] CPU: 9 PID: 16135 Comm: RxWorker1 Tainted: G        W         5.15.77evl-gae6080e09d9a #1
Nov 11 21:25:34 10.243.1.49 [ 3904.578704] Hardware name: Supermicro Super Server/X10SDV-6C-TLN4F, BIOS 1.3 02/13/2018
Nov 11 21:25:34 10.243.1.49 [ 3904.608607] IRQ stage: Linux
Nov 11 21:25:34 10.243.1.49 [ 3904.623117] RIP: 0010:syscall_exit_to_user_mode_prepare+0x37/0x270
Nov 11 21:25:34 10.243.1.49 [ 3904.648593] Code: 41 55 41 54 49 89 fc 53 4c 8b 68 08 48 8b 5f 78 eb 36 e8 0c fe f2 00 85 c0 74 1c 48 63 f3 48 c7 c7 70 66 ca a5 e8 42 f0 ec 00 <0f> 0b fb e8 e1 a1 0d 00 e8 bc af fd ff 41 f6 c5 76 75 3a 5b 41 5c
Nov 11 21:25:34 10.243.1.49 [ 3904.717388] RSP: 0018:ffff88815ef47f00 EFLAGS: 00010282
Nov 11 21:25:34 10.243.1.49 [ 3904.739421] RAX: 0000000000000000 RBX: 000000000000009d RCX: 0000000000000027
Nov 11 21:25:34 10.243.1.49 [ 3904.767285] RDX: 0000000000000027 RSI: dffffc0000000000 RDI: ffff88841c2a7b18
Nov 11 21:25:34 10.243.1.49 [ 3904.795170] RBP: ffff88815ef47f18 R08: ffffed1083854f64 R09: ffffed1083854f64
Nov 11 21:25:34 10.243.1.49 [ 3904.823094] R10: ffff88841c2a7b1b R11: ffffed1083854f63 R12: ffff88815ef47f58
Nov 11 21:25:34 10.243.1.49 [ 3904.851037] R13: 0000000000000010 R14: 0000000000000000 R15: 0000000000000000
Nov 11 21:25:34 10.243.1.49 [ 3904.879042] FS:  00007f7312ffe700(0000) GS:ffff88841c280000(0000) knlGS:0000000000000000
Nov 11 21:25:34 10.243.1.49 [ 3904.909991] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Nov 11 21:25:34 10.243.1.49 [ 3904.933894] CR2: 0000000000000000 CR3: 000000016e080005 CR4: 00000000003706e0
Nov 11 21:25:34 10.243.1.49 [ 3904.962065] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Nov 11 21:25:34 10.243.1.49 [ 3904.990210] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Nov 11 21:25:34 10.243.1.49 [ 3905.018329] Call Trace:
Nov 11 21:25:34 10.243.1.49 [ 3905.032333]  <TASK>
Nov 11 21:25:34 10.243.1.49 [ 3905.046205]  syscall_exit_to_user_mode+0xd/0x90
Nov 11 21:25:34 10.243.1.49 [ 3905.066327]  do_syscall_64+0x50/0xb0
Nov 11 21:25:34 10.243.1.49 [ 3905.083393]  entry_SYSCALL_64_after_hwframe+0x61/0xcb
Nov 11 21:25:34 10.243.1.49 [ 3905.104867] RIP: 0033:0x7f735de5a940
Nov 11 21:25:34 10.243.1.49 [ 3905.121847] Code: 77 ff ff bf 02 00 00 10 8b 44 24 0c 48 98 48 89 c6 48 8b 04 24 48 89 c2 4c 8b 54 24 38 41 b8 00 00 00 00 b8 9d 00 00 00 0f 05 <48> 89 44 24 30 48 8b 44 24 30 89 44 24 2c 8b 44 24 28 be 00 00 00
Nov 11 21:25:34 10.243.1.49 [ 3905.191272] RSP: 002b:00007f7312fee340 EFLAGS: 00000202 ORIG_RAX: 000000000000009d
Nov 11 21:25:34 10.243.1.49 [ 3905.220715] RAX: 0000000000000000 RBX: 0000000000000f3e RCX: 00007f735de5a940
Nov 11 21:25:34 10.243.1.49 [ 3905.249800] RDX: 00000000c0186d03 RSI: 000000000000000d RDI: 0000000010000002
Nov 11 21:25:34 10.243.1.49 [ 3905.277936] RBP: 00007f7312fee520 R08: 0000000000000000 R09: 00007ffc09389080
Nov 11 21:25:34 10.243.1.49 [ 3905.306096] R10: 00007f7312fee4f0 R11: 0000000000000202 R12: 00007f7312fee600
Nov 11 21:25:34 10.243.1.49 [ 3905.334148] R13: 0000000000000000 R14: 000000001302529e R15: 00007f7312fee580
Nov 11 21:25:34 10.243.1.49 [ 3905.362279]  </TASK>
Nov 11 21:25:34 10.243.1.49 [ 3905.375352] irq event stamp: 1379
Nov 11 21:25:34 10.243.1.49 [ 3905.391835] hardirqs last  enabled at (1387): [<ffffffffa41aaac2>] __up_console_sem+0x62/0x70
Nov 11 21:25:35 10.243.1.49 [ 3905.424075] hardirqs last disabled at (1394): [<ffffffffa41aaaa7>] __up_console_sem+0x47/0x70
Nov 11 21:25:35 10.243.1.49 [ 3905.456192] softirqs last  enabled at (684): [<ffffffffa5400403>] __do_softirq+0x403/0x57b
Nov 11 21:25:35 10.243.1.49 [ 3905.487432] softirqs last disabled at (677): [<ffffffffa40d261d>] irq_exit_rcu+0x7d/0x150
Nov 11 21:25:35 10.243.1.49 [ 3905.518421] ---[ end trace 87ffbd6a737b5b84 ]---
Nov 11 21:25:38 10.243.1.49 [ 3908.717920] ------------[ cut here ]------------
Nov 11 21:25:38 10.243.1.49 [ 3908.717927] list_del corruption, ffff888151798240->next is LIST_POISON1 (dead000000000100)
Nov 11 21:25:38 10.243.1.49 [ 3908.717938] EVL: RxWorker1 switching in-band [pid=16135, excpt=6, __list_del_entry_valid+0xc5/0x120]
Nov 11 21:25:38 10.243.1.49 [ 3908.820609] WARNING: CPU: 9 PID: 16135 at lib/list_debug.c:55 __list_del_entry_valid+0xc5/0x120
Nov 11 21:25:38 10.243.1.49 [ 3908.820633] Modules linked in: xt_CHECKSUM xt_MASQUERADE tun netconsole ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 xt_conntrack ebtable_nat ip6table_nat ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_nat iptable_mangle iptable_security iptable_raw nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 ip_set nfnetlink ebtable_filter ebtables rfkill ip6table_filter ip6_tables iptable_filter sunrpc dm_mirror dm_region_hash dm_log dm_mod iTCO_wdt gpio_ich iTCO_vendor_support mxm_wmi x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm irqbypass crct10dif_pclmul crc32_pclmul ghash_clmulni_intel aesni_intel crypto_simd cryptd rapl mei_me i2c_i801 lpc_ich intel_cstate pcspkr intel_pch_thermal input_leds sg joydev ftdi_sio i2c_smbus mfd_core mei acpi_ipmi ioatdma ipmi_si ipmi_devintf ipmi_msghandler wmi ip_tables xfs libcrc32c ast drm_vram_helper sd_mod drm_kms_helper sr_mod cdrom t10_pi syscopyarea sysfillrect sysimgblt fb_sys_fops drm_ttm_helper ttm drm
Nov 11 21:25:38 10.243.1.49
Nov 11 21:25:38 10.243.1.49 [ 3908.820955]  ixgbe igb ahci libahci crc32c_intel libata mdio ptp i2c_algo_bit pps_core dca
Nov 11 21:25:38 10.243.1.49 [ 3908.820998] CPU: 9 PID: 16135 Comm: RxWorker1 Tainted: G        W         5.15.77evl-gae6080e09d9a #1
Nov 11 21:25:38 10.243.1.49 [ 3908.821011] Hardware name: Supermicro Super Server/X10SDV-6C-TLN4F, BIOS 1.3 02/13/2018
Nov 11 21:25:38 10.243.1.49 [ 3908.821016] IRQ stage: Linux
Nov 11 21:25:38 10.243.1.49 [ 3908.821021] RIP: 0010:__list_del_entry_valid+0xc5/0x120
Nov 11 21:25:38 10.243.1.49 [ 3908.821036] Code: c0 eb e0 48 89 de 48 c7 c7 00 0c 82 a5 e8 7c fa 8b 00 0f 0b 31 c0 eb cb 4c 89 e2 48 89 de 48 c7 c7 60 0c 82 a5 e8 64 fa 8b 00 <0f> 0b 31 c0 eb b3 4c 89 ea 48 89 de 48 c7 c7 c0 0c 82 a5 e8 4c fa
Nov 11 21:25:39 10.243.1.49 [ 3908.821046] RSP: 0018:ffff88815ef47ac8 EFLAGS: 00010082
Nov 11 21:25:39 10.243.1.49 [ 3908.821056] RAX: 0000000000000000 RBX: ffff888151798240 RCX: 0000000000000027
Nov 11 21:25:39 10.243.1.49 [ 3908.821063] RDX: 0000000000000027 RSI: dffffc0000000000 RDI: ffff88841c2a7b18
Nov 11 21:25:39 10.243.1.49 [ 3908.821071] RBP: ffff88815ef47ae0 R08: ffffed1083854f64 R09: ffffed1083854f64
Nov 11 21:25:39 10.243.1.49 [ 3908.821080] R10: ffff88841c2a7b1b R11: ffffed1083854f63 R12: dead000000000100
Nov 11 21:25:39 10.243.1.49 [ 3908.821088] R13: dead000000000122 R14: ffff888151798000 R15: ffff888145380000
Nov 11 21:25:39 10.243.1.49 [ 3908.821096] FS:  00007f7312ffe700(0000) GS:ffff88841c280000(0000) knlGS:0000000000000000
Nov 11 21:25:39 10.243.1.49 [ 3908.821106] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Nov 11 21:25:39 10.243.1.49 [ 3908.821115] CR2: 0000000000000000 CR3: 000000016e080005 CR4: 00000000003706e0
Nov 11 21:25:39 10.243.1.49 [ 3908.821122] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Nov 11 21:25:39 10.243.1.49 [ 3908.821128] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Nov 11 21:25:39 10.243.1.49 [ 3908.821135] Call Trace:
Nov 11 21:25:39 10.243.1.49 [ 3908.821139]  <TASK>
Nov 11 21:25:39 10.243.1.49 [ 3908.821158]  __untrack_event.part.9+0x26/0xe0
Nov 11 21:25:39 10.243.1.49 [ 3908.821196]  monitor_oob_ioctl+0x1356/0x1490
Nov 11 21:25:39 10.243.1.49 [ 3908.821256]  ? enter_monitor+0x160/0x160
Nov 11 21:25:39 10.243.1.49 [ 3908.821277]  ? handle_irq_pipelined_finish+0xbe/0x240
Nov 11 21:25:39 10.243.1.49 [ 3908.821351]  ? do_raw_spin_lock+0x119/0x1e0
Nov 11 21:25:39 10.243.1.49 [ 3908.821369]  ? spin_dump+0x90/0x90
Nov 11 21:25:39 10.243.1.49 [ 3908.821407]  ? __kasan_check_read+0x11/0x20
Nov 11 21:25:39 10.243.1.49 [ 3908.821471]  EVL_ioctl+0x81/0xf0
Nov 11 21:25:39 10.243.1.49 [ 3908.821506]  do_oob_syscall+0x699/0x6b0
Nov 11 21:25:39 10.243.1.49 [ 3908.821536]  ? prepare_for_signal+0x180/0x180
Nov 11 21:25:39 10.243.1.49 [ 3908.821565]  ? do_oob_irq+0x283/0x7e0
Nov 11 21:25:39 10.243.1.49 [ 3908.821607]  handle_oob_syscall+0x189/0x230
Nov 11 21:25:39 10.243.1.49 [ 3908.821637]  ? handle_pipelined_syscall+0x820/0x820
Nov 11 21:25:39 10.243.1.49 [ 3908.821673]  ? rcu_read_lock_sched_held+0x5b/0xd0
Nov 11 21:25:39 10.243.1.49 [ 3908.821695]  ? rcu_read_lock_any_held.part.15+0x20/0x20
Nov 11 21:25:39 10.243.1.49 [ 3908.821738]  pipeline_syscall+0xa7/0x1c0
Nov 11 21:25:39 10.243.1.49 [ 3908.821777]  syscall_enter_from_user_mode+0x47/0x110
Nov 11 21:25:39 10.243.1.49 [ 3908.821804]  do_syscall_64+0x15/0xb0
Nov 11 21:25:39 10.243.1.49 [ 3908.821824]  entry_SYSCALL_64_after_hwframe+0x61/0xcb
Nov 11 21:25:39 10.243.1.49 [ 3908.821838] RIP: 0033:0x7f735de5a940
Nov 11 21:25:39 10.243.1.49 [ 3908.821849] Code: 77 ff ff bf 02 00 00 10 8b 44 24 0c 48 98 48 89 c6 48 8b 04 24 48 89 c2 4c 8b 54 24 38 41 b8 00 00 00 00 b8 9d 00 00 00 0f 05 <48> 89 44 24 30 48 8b 44 24 30 89 44 24 2c 8b 44 24 28 be 00 00 00
Nov 11 21:25:40 10.243.1.49 [ 3908.821858] RSP: 002b:00007f7312fee340 EFLAGS: 00000202 ORIG_RAX: 000000000000009d
Nov 11 21:25:40 10.243.1.49 [ 3908.821869] RAX: ffffffffffffffda RBX: 0000000000000f45 RCX: 00007f735de5a940
Nov 11 21:25:40 10.243.1.49 [ 3908.821877] RDX: 00000000c0186d03 RSI: 000000000000000d RDI: 0000000010000002
Nov 11 21:25:40 10.243.1.49 [ 3908.821884] RBP: 00007f7312fee520 R08: 0000000000000000 R09: 00007ffc09389080
Nov 11 21:25:40 10.243.1.49 [ 3908.821891] R10: 00007f7312fee4f0 R11: 0000000000000202 R12: 00007f7312fee600
Nov 11 21:25:40 10.243.1.49 [ 3908.821899] R13: 0000000000000000 R14: 00000000009cca93 R15: 00007f7312fee580
Nov 11 21:25:40 10.243.1.49 [ 3908.821980]  </TASK>
Nov 11 21:25:40 10.243.1.49 [ 3908.821983] irq event stamp: 1475
Nov 11 21:25:40 10.243.1.49 [ 3908.821987] hardirqs last  enabled at (1475): [<ffffffffa51330bc>] _raw_spin_unlock_irqrestore+0x4c/0x50
Nov 11 21:25:40 10.243.1.49 [ 3908.822002] hardirqs last disabled at (1474): [<ffffffffa5132e2b>] _raw_spin_lock_irqsave+0x1b/0x60
Nov 11 21:25:40 10.243.1.49 [ 3908.822016] softirqs last  enabled at (684): [<ffffffffa5400403>] __do_softirq+0x403/0x57b
Nov 11 21:25:40 10.243.1.49 [ 3908.822031] softirqs last disabled at (677): [<ffffffffa40d261d>] irq_exit_rcu+0x7d/0x150
Nov 11 21:25:40 10.243.1.49 [ 3908.822047] ---[ end trace 87ffbd6a737b5b85 ]---
Nov 11 21:25:40 10.243.1.49 [ 3910.811569] EVL: RxWorker1 resuming out-of-band [pid=16135, excpt=6, __list_del_entry_valid+0xc7/0x120]
Nov 11 21:25:40 10.243.1.49 [ 3910.848547] ------------[ cut here ]------------
Nov 11 21:25:40 10.243.1.49 [ 3910.866790] syscall 157 left IRQs disabled
Nov 11 21:25:40 10.243.1.49 [ 3910.866843] WARNING: CPU: 9 PID: 16135 at kernel/entry/common.c:353 syscall_exit_to_user_mode_prepare+0x37/0x270
Nov 11 21:25:40 10.243.1.49 [ 3910.918617] Modules linked in: xt_CHECKSUM xt_MASQUERADE tun netconsole ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 xt_conntrack ebtable_nat ip6table_nat ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_nat iptable_mangle iptable_security iptable_raw nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 ip_set nfnetlink ebtable_filter ebtables rfkill ip6table_filter ip6_tables iptable_filter sunrpc dm_mirror dm_region_hash dm_log dm_mod iTCO_wdt gpio_ich iTCO_vendor_support mxm_wmi x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm irqbypass crct10dif_pclmul crc32_pclmul ghash_clmulni_intel aesni_intel crypto_simd cryptd rapl mei_me i2c_i801 lpc_ich intel_cstate pcspkr intel_pch_thermal input_leds sg joydev ftdi_sio i2c_smbus mfd_core mei acpi_ipmi ioatdma ipmi_si ipmi_devintf ipmi_msghandler wmi ip_tables xfs libcrc32c ast drm_vram_helper sd_mod drm_kms_helper sr_mod cdrom t10_pi syscopyarea sysfillrect sysimgblt fb_sys_fops drm_ttm_helper ttm drm
Nov 11 21:25:40 10.243.1.49
Nov 11 21:25:40 10.243.1.49 [ 3910.919123]  ixgbe igb ahci libahci crc32c_intel libata mdio ptp i2c_algo_bit pps_core dca
Nov 11 21:25:40 10.243.1.49 [ 3911.252660] CPU: 9 PID: 16135 Comm: RxWorker1 Tainted: G        W         5.15.77evl-gae6080e09d9a #1
Nov 11 21:25:40 10.243.1.49 [ 3911.286188] Hardware name: Supermicro Super Server/X10SDV-6C-TLN4F, BIOS 1.3 02/13/2018
Nov 11 21:25:40 10.243.1.49 [ 3911.316122] IRQ stage: Linux
Nov 11 21:25:40 10.243.1.49 [ 3911.330643] RIP: 0010:syscall_exit_to_user_mode_prepare+0x37/0x270
Nov 11 21:25:40 10.243.1.49 [ 3911.356120] Code: 41 55 41 54 49 89 fc 53 4c 8b 68 08 48 8b 5f 78 eb 36 e8 0c fe f2 00 85 c0 74 1c 48 63 f3 48 c7 c7 70 66 ca a5 e8 42 f0 ec 00 <0f> 0b fb e8 e1 a1 0d 00 e8 bc af fd ff 41 f6 c5 76 75 3a 5b 41 5c
Nov 11 21:25:40 10.243.1.49 [ 3911.424947] RSP: 0018:ffff88815ef47f00 EFLAGS: 00010282
Nov 11 21:25:41 10.243.1.49 [ 3911.446986] RAX: 0000000000000000 RBX: 000000000000009d RCX: 0000000000000027
Nov 11 21:25:41 10.243.1.49 [ 3911.474887] RDX: 0000000000000027 RSI: dffffc0000000000 RDI: ffff88841c2a7b18
Nov 11 21:25:41 10.243.1.49 [ 3911.502796] RBP: ffff88815ef47f18 R08: ffffed1083854f64 R09: ffffed1083854f64
Nov 11 21:25:41 10.243.1.49 [ 3911.530745] R10: ffff88841c2a7b1b R11: ffffed1083854f63 R12: ffff88815ef47f58
Nov 11 21:25:41 10.243.1.49 [ 3911.558728] R13: 0000000000000010 R14: 0000000000000000 R15: 0000000000000000
Nov 11 21:25:41 10.243.1.49 [ 3911.586747] FS:  00007f7312ffe700(0000) GS:ffff88841c280000(0000) knlGS:0000000000000000
Nov 11 21:25:41 10.243.1.49 [ 3911.617694] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Nov 11 21:25:41 10.243.1.49 [ 3911.641647] CR2: 0000000000000000 CR3: 000000016e080005 CR4: 00000000003706e0
Nov 11 21:25:41 10.243.1.49 [ 3911.669831] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Nov 11 21:25:41 10.243.1.49 [ 3911.698009] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Nov 11 21:25:41 10.243.1.49 [ 3911.726180] Call Trace:
Nov 11 21:25:41 10.243.1.49 [ 3911.740157]  <TASK>
Nov 11 21:25:41 10.243.1.49 [ 3911.754393]  syscall_exit_to_user_mode+0xd/0x90
Nov 11 21:25:41 10.243.1.49 [ 3911.774522]  do_syscall_64+0x50/0xb0
Nov 11 21:25:41 10.243.1.49 [ 3911.791628]  entry_SYSCALL_64_after_hwframe+0x61/0xcb
Nov 11 21:25:41 10.243.1.49 [ 3911.813111] RIP: 0033:0x7f735de5a940
Nov 11 21:25:41 10.243.1.49 [ 3911.830102] Code: 77 ff ff bf 02 00 00 10 8b 44 24 0c 48 98 48 89 c6 48 8b 04 24 48 89 c2 4c 8b 54 24 38 41 b8 00 00 00 00 b8 9d 00 00 00 0f 05 <48> 89 44 24 30 48 8b 44 24 30 89 44 24 2c 8b 44 24 28 be 00 00 00
Nov 11 21:25:41 10.243.1.49 [ 3911.899551] RSP: 002b:00007f7312fee340 EFLAGS: 00000202 ORIG_RAX: 000000000000009d
Nov 11 21:25:41 10.243.1.49 [ 3911.929006] RAX: 0000000000000000 RBX: 0000000000000f45 RCX: 00007f735de5a940
Nov 11 21:25:41 10.243.1.49 [ 3911.958507] RDX: 00000000c0186d03 RSI: 000000000000000d RDI: 0000000010000002
Nov 11 21:25:41 10.243.1.49 [ 3911.986639] RBP: 00007f7312fee520 R08: 0000000000000000 R09: 00007ffc09389080
Nov 11 21:25:41 10.243.1.49 [ 3912.014819] R10: 00007f7312fee4f0 R11: 0000000000000202 R12: 00007f7312fee600
Nov 11 21:25:41 10.243.1.49 [ 3912.042905] R13: 0000000000000000 R14: 00000000009cca93 R15: 00007f7312fee580
Nov 11 21:25:41 10.243.1.49 [ 3912.071017]  </TASK>
Nov 11 21:25:41 10.243.1.49 [ 3912.084096] irq event stamp: 2193
Nov 11 21:25:41 10.243.1.49 [ 3912.100606] hardirqs last  enabled at (2201): [<ffffffffa41aaac2>] __up_console_sem+0x62/0x70
Nov 11 21:25:41 10.243.1.49 [ 3912.132860] hardirqs last disabled at (2208): [<ffffffffa41aaaa7>] __up_console_sem+0x47/0x70
Nov 11 21:25:41 10.243.1.49 [ 3912.164996] softirqs last  enabled at (1488): [<ffffffffa5400403>] __do_softirq+0x403/0x57b
Nov 11 21:25:41 10.243.1.49 [ 3912.196532] softirqs last disabled at (1479): [<ffffffffa40d261d>] irq_exit_rcu+0x7d/0x150
Nov 11 21:25:41 10.243.1.49 [ 3912.227804] ---[ end trace 87ffbd6a737b5b86 ]---
Nov 11 21:25:47 10.243.1.49 [ 3917.682604] ------------[ cut here ]------------
Nov 11 21:25:47 10.243.1.49 [ 3917.682611] list_del corruption, ffff888151798240->next is LIST_POISON1 (dead000000000100)
Nov 11 21:25:47 10.243.1.49 [ 3917.682622] EVL: RxWorker0 switching in-band [pid=16134, excpt=6, __list_del_entry_valid+0xc5/0x120]
Nov 11 21:25:47 10.243.1.49 [ 3917.768071] WARNING: CPU: 8 PID: 16134 at lib/list_debug.c:55 __list_del_entry_valid+0xc5/0x120
Nov 11 21:25:47 10.243.1.49 [ 3917.768086] Modules linked in: xt_CHECKSUM xt_MASQUERADE tun netconsole ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 xt_conntrack ebtable_nat ip6table_nat ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_nat iptable_mangle iptable_security iptable_raw nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 ip_set nfnetlink ebtable_filter ebtables rfkill ip6table_filter ip6_tables iptable_filter sunrpc dm_mirror dm_region_hash dm_log dm_mod iTCO_wdt gpio_ich iTCO_vendor_support mxm_wmi x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm irqbypass crct10dif_pclmul crc32_pclmul ghash_clmulni_intel aesni_intel crypto_simd cryptd rapl mei_me i2c_i801 lpc_ich intel_cstate pcspkr intel_pch_thermal input_leds sg joydev ftdi_sio i2c_smbus mfd_core mei acpi_ipmi ioatdma ipmi_si ipmi_devintf ipmi_msghandler wmi ip_tables xfs libcrc32c ast drm_vram_helper sd_mod drm_kms_helper sr_mod cdrom t10_pi syscopyarea sysfillrect sysimgblt fb_sys_fops drm_ttm_helper ttm drm
Nov 11 21:25:47 10.243.1.49
Nov 11 21:25:47 10.243.1.49 [ 3917.768251]  ixgbe igb ahci libahci crc32c_intel libata mdio ptp i2c_algo_bit pps_core dca
Nov 11 21:25:47 10.243.1.49 [ 3917.768274] CPU: 8 PID: 16134 Comm: RxWorker0 Tainted: G        W         5.15.77evl-gae6080e09d9a #1
Nov 11 21:25:47 10.243.1.49 [ 3917.768282] Hardware name: Supermicro Super Server/X10SDV-6C-TLN4F, BIOS 1.3 02/13/2018
Nov 11 21:25:47 10.243.1.49 [ 3917.768285] IRQ stage: Linux
Nov 11 21:25:47 10.243.1.49 [ 3917.768287] RIP: 0010:__list_del_entry_valid+0xc5/0x120
Nov 11 21:25:47 10.243.1.49 [ 3917.768296] Code: c0 eb e0 48 89 de 48 c7 c7 00 0c 82 a5 e8 7c fa 8b 00 0f 0b 31 c0 eb cb 4c 89 e2 48 89 de 48 c7 c7 60 0c 82 a5 e8 64 fa 8b 00 <0f> 0b 31 c0 eb b3 4c 89 ea 48 89 de 48 c7 c7 c0 0c 82 a5 e8 4c fa
Nov 11 21:25:47 10.243.1.49 [ 3917.768301] RSP: 0018:ffff88816588fac8 EFLAGS: 00010082
Nov 11 21:25:47 10.243.1.49 [ 3917.768306] RAX: 0000000000000000 RBX: ffff888151798240 RCX: 0000000000000027
Nov 11 21:25:47 10.243.1.49 [ 3917.768310] RDX: 0000000000000027 RSI: dffffc0000000000 RDI: ffff88841c227b18
Nov 11 21:25:47 10.243.1.49 [ 3917.768314] RBP: ffff88816588fae0 R08: ffffed1083844f64 R09: ffffed1083844f64
Nov 11 21:25:48 10.243.1.49 [ 3917.768318] R10: ffff88841c227b1b R11: ffffed1083844f63 R12: dead000000000100
Nov 11 21:25:48 10.243.1.49 [ 3917.768322] R13: dead000000000122 R14: ffff888151798000 R15: ffff888148b3a000
Nov 11 21:25:48 10.243.1.49 [ 3917.768326] FS:  00007f7313fff700(0000) GS:ffff88841c200000(0000) knlGS:0000000000000000
Nov 11 21:25:48 10.243.1.49 [ 3917.768331] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Nov 11 21:25:48 10.243.1.49 [ 3917.768334] CR2: 000024a61fc9e000 CR3: 000000016e080001 CR4: 00000000003706e0
Nov 11 21:25:48 10.243.1.49 [ 3917.768338] Call Trace:
Nov 11 21:25:48 10.243.1.49 [ 3917.768340]  <TASK>
Nov 11 21:25:48 10.243.1.49 [ 3917.768350]  __untrack_event.part.9+0x26/0xe0
Nov 11 21:25:48 10.243.1.49 [ 3917.768371]  monitor_oob_ioctl+0x1356/0x1490
Nov 11 21:25:48 10.243.1.49 [ 3917.768400]  ? enter_monitor+0x160/0x160
Nov 11 21:25:48 10.243.1.49 [ 3917.768410]  ? handle_irq_pipelined_finish+0xbe/0x240
Nov 11 21:25:48 10.243.1.49 [ 3917.768446]  ? do_raw_spin_lock+0x119/0x1e0
Nov 11 21:25:48 10.243.1.49 [ 3917.768456]  ? spin_dump+0x90/0x90
Nov 11 21:25:48 10.243.1.49 [ 3917.768484]  ? __kasan_check_read+0x11/0x20
Nov 11 21:25:48 10.243.1.49 [ 3917.768540]  EVL_ioctl+0x81/0xf0
Nov 11 21:25:48 10.243.1.49 [ 3917.768558]  do_oob_syscall+0x699/0x6b0
Nov 11 21:25:48 10.243.1.49 [ 3917.768572]  ? prepare_for_signal+0x180/0x180
Nov 11 21:25:48 10.243.1.49 [ 3917.768587]  ? do_oob_irq+0x283/0x7e0
Nov 11 21:25:48 10.243.1.49 [ 3917.768594]  ? arch_pipeline_entry+0xce/0x110
Nov 11 21:25:48 10.243.1.49 [ 3917.768612]  handle_oob_syscall+0x189/0x230
Nov 11 21:25:48 10.243.1.49 [ 3917.768626]  ? handle_pipelined_syscall+0x820/0x820
Nov 11 21:25:48 10.243.1.49 [ 3917.768644]  ? rcu_read_lock_sched_held+0x5b/0xd0
Nov 11 21:25:48 10.243.1.49 [ 3917.768655]  ? rcu_read_lock_any_held.part.15+0x20/0x20
Nov 11 21:25:48 10.243.1.49 [ 3917.768676]  pipeline_syscall+0xa7/0x1c0
Nov 11 21:25:48 10.243.1.49 [ 3917.768696]  syscall_enter_from_user_mode+0x47/0x110
Nov 11 21:25:48 10.243.1.49 [ 3917.768710]  do_syscall_64+0x15/0xb0
Nov 11 21:25:48 10.243.1.49 [ 3917.768721]  entry_SYSCALL_64_after_hwframe+0x61/0xcb
Nov 11 21:25:48 10.243.1.49 [ 3917.768729] RIP: 0033:0x7f735de5a940
Nov 11 21:25:48 10.243.1.49 [ 3917.768735] Code: 77 ff ff bf 02 00 00 10 8b 44 24 0c 48 98 48 89 c6 48 8b 04 24 48 89 c2 4c 8b 54 24 38 41 b8 00 00 00 00 b8 9d 00 00 00 0f 05 <48> 89 44 24 30 48 8b 44 24 30 89 44 24 2c 8b 44 24 28 be 00 00 00
Nov 11 21:25:48 10.243.1.49 [ 3917.768740] RSP: 002b:00007f7313fef340 EFLAGS: 00000202 ORIG_RAX: 000000000000009d
Nov 11 21:25:48 10.243.1.49 [ 3917.768746] RAX: ffffffffffffffda RBX: 0000000000000f4d RCX: 00007f735de5a940
Nov 11 21:25:48 10.243.1.49 [ 3917.768750] RDX: 00000000c0186d03 RSI: 000000000000000d RDI: 0000000010000002
Nov 11 21:25:48 10.243.1.49 [ 3917.768754] RBP: 00007f7313fef520 R08: 0000000000000000 R09: 00007ffc09389080
Nov 11 21:25:48 10.243.1.49 [ 3917.768757] R10: 00007f7313fef4f0 R11: 0000000000000202 R12: 00007f7313fef600
Nov 11 21:25:48 10.243.1.49 [ 3917.768760] R13: 0000000000000000 R14: 000000003a1b30e9 R15: 00007f7313fef580
Nov 11 21:25:48 10.243.1.49 [ 3917.768798]  </TASK>
Nov 11 21:25:48 10.243.1.49 [ 3917.768800] irq event stamp: 1763
Nov 11 21:25:48 10.243.1.49 [ 3917.768802] hardirqs last  enabled at (1763): [<ffffffffa51330bc>] _raw_spin_unlock_irqrestore+0x4c/0x50
Nov 11 21:25:48 10.243.1.49 [ 3917.768811] hardirqs last disabled at (1762): [<ffffffffa5132e2b>] _raw_spin_lock_irqsave+0x1b/0x60
Nov 11 21:25:48 10.243.1.49 [ 3917.768818] softirqs last  enabled at (286): [<ffffffffa5400403>] __do_softirq+0x403/0x57b
Nov 11 21:25:48 10.243.1.49 [ 3917.768826] softirqs last disabled at (275): [<ffffffffa40d261d>] irq_exit_rcu+0x7d/0x150
Nov 11 21:25:48 10.243.1.49 [ 3917.768835] ---[ end trace 87ffbd6a737b5b87 ]---
Nov 11 21:25:48 10.243.1.49 [ 3919.382324] EVL: RxWorker0 resuming out-of-band [pid=16134, excpt=6, __list_del_entry_valid+0xc7/0x120]
Nov 11 21:25:48 10.243.1.49 [ 3919.415123] ------------[ cut here ]------------
Nov 11 21:25:49 10.243.1.49 [ 3919.433389] syscall 157 left IRQs disabled
Nov 11 21:25:49 10.243.1.49 [ 3919.433444] WARNING: CPU: 8 PID: 16134 at kernel/entry/common.c:353 syscall_exit_to_user_mode_prepare+0x37/0x270
Nov 11 21:25:49 10.243.1.49 [ 3919.485450] Modules linked in: xt_CHECKSUM xt_MASQUERADE tun netconsole ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 xt_conntrack ebtable_nat ip6table_nat ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_nat iptable_mangle iptable_security iptable_raw nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 ip_set nfnetlink ebtable_filter ebtables rfkill ip6table_filter ip6_tables iptable_filter sunrpc dm_mirror dm_region_hash dm_log dm_mod iTCO_wdt gpio_ich iTCO_vendor_support mxm_wmi x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm irqbypass crct10dif_pclmul crc32_pclmul ghash_clmulni_intel aesni_intel crypto_simd cryptd rapl mei_me i2c_i801 lpc_ich intel_cstate pcspkr intel_pch_thermal input_leds sg joydev ftdi_sio i2c_smbus mfd_core mei acpi_ipmi ioatdma ipmi_si ipmi_devintf ipmi_msghandler wmi ip_tables xfs libcrc32c ast drm_vram_helper sd_mod drm_kms_helper sr_mod cdrom t10_pi syscopyarea sysfillrect sysimgblt fb_sys_fops drm_ttm_helper ttm drm
Nov 11 21:25:49 10.243.1.49
Nov 11 21:25:49 10.243.1.49 [ 3919.486010]  ixgbe igb ahci libahci crc32c_intel libata mdio ptp i2c_algo_bit pps_core dca
Nov 11 21:25:49 10.243.1.49 [ 3919.819623] CPU: 8 PID: 16134 Comm: RxWorker0 Tainted: G        W         5.15.77evl-gae6080e09d9a #1
Nov 11 21:25:49 10.243.1.49 [ 3919.853138] Hardware name: Supermicro Super Server/X10SDV-6C-TLN4F, BIOS 1.3 02/13/2018
Nov 11 21:25:49 10.243.1.49 [ 3919.883092] IRQ stage: Linux
Nov 11 21:25:49 10.243.1.49 [ 3919.897600] RIP: 0010:syscall_exit_to_user_mode_prepare+0x37/0x270
Nov 11 21:25:49 10.243.1.49 [ 3919.922159] Code: 41 55 41 54 49 89 fc 53 4c 8b 68 08 48 8b 5f 78 eb 36 e8 0c fe f2 00 85 c0 74 1c 48 63 f3 48 c7 c7 70 66 ca a5 e8 42 f0 ec 00 <0f> 0b fb e8 e1 a1 0d 00 e8 bc af fd ff 41 f6 c5 76 75 3a 5b 41 5c
Nov 11 21:25:49 10.243.1.49 [ 3919.991108] RSP: 0018:ffff88816588ff00 EFLAGS: 00010282
Nov 11 21:25:49 10.243.1.49 [ 3920.013245] RAX: 0000000000000000 RBX: 000000000000009d RCX: 0000000000000027
Nov 11 21:25:49 10.243.1.49 [ 3920.041449] RDX: 0000000000000027 RSI: dffffc0000000000 RDI: ffff88841c227b18
Nov 11 21:25:49 10.243.1.49 [ 3920.069436] RBP: ffff88816588ff18 R08: ffffed1083844f64 R09: ffffed1083844f64
Nov 11 21:25:49 10.243.1.49 [ 3920.097424] R10: ffff88841c227b1b R11: ffffed1083844f63 R12: ffff88816588ff58
Nov 11 21:25:49 10.243.1.49 [ 3920.125444] R13: 0000000000000010 R14: 0000000000000000 R15: 0000000000000000
Nov 11 21:25:49 10.243.1.49 [ 3920.153567] FS:  00007f7313fff700(0000) GS:ffff88841c200000(0000) knlGS:0000000000000000
Nov 11 21:25:49 10.243.1.49 [ 3920.184562] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Nov 11 21:25:49 10.243.1.49 [ 3920.208601] CR2: 000024a61fc9e000 CR3: 000000016e080001 CR4: 00000000003706e0
Nov 11 21:25:49 10.243.1.49 [ 3920.236828] Call Trace:
Nov 11 21:25:49 10.243.1.49 [ 3920.250925]  <TASK>
Nov 11 21:25:49 10.243.1.49 [ 3920.264093]  syscall_exit_to_user_mode+0xd/0x90
Nov 11 21:25:49 10.243.1.49 [ 3920.284430]  do_syscall_64+0x50/0xb0
Nov 11 21:25:49 10.243.1.49 [ 3920.301962]  entry_SYSCALL_64_after_hwframe+0x61/0xcb
Nov 11 21:25:49 10.243.1.49 [ 3920.323780] RIP: 0033:0x7f735de5a940
Nov 11 21:25:49 10.243.1.49 [ 3920.340972] Code: 77 ff ff bf 02 00 00 10 8b 44 24 0c 48 98 48 89 c6 48 8b 04 24 48 89 c2 4c 8b 54 24 38 41 b8 00 00 00 00 b8 9d 00 00 00 0f 05 <48> 89 44 24 30 48 8b 44 24 30 89 44 24 2c 8b 44 24 28 be 00 00 00
Nov 11 21:25:50 10.243.1.49 [ 3920.415607] RSP: 002b:00007f7313fef340 EFLAGS: 00000202 ORIG_RAX: 000000000000009d
Nov 11 21:25:50 10.243.1.49 [ 3920.445137] RAX: 0000000000000000 RBX: 0000000000000f4d RCX: 00007f735de5a940
Nov 11 21:25:50 10.243.1.49 [ 3920.473367] RDX: 00000000c0186d03 RSI: 000000000000000d RDI: 0000000010000002
Nov 11 21:25:50 10.243.1.49 [ 3920.501668] RBP: 00007f7313fef520 R08: 0000000000000000 R09: 00007ffc09389080
Nov 11 21:25:50 10.243.1.49 [ 3920.529978] R10: 00007f7313fef4f0 R11: 0000000000000202 R12: 00007f7313fef600
Nov 11 21:25:50 10.243.1.49 [ 3920.559152] R13: 0000000000000000 R14: 000000003a1b30e9 R15: 00007f7313fef580
Nov 11 21:25:50 10.243.1.49 [ 3920.587647]  </TASK>
Nov 11 21:25:50 10.243.1.49 [ 3920.600846] irq event stamp: 2457
Nov 11 21:25:50 10.243.1.49 [ 3920.617396] hardirqs last  enabled at (2465): [<ffffffffa41aaac2>] __up_console_sem+0x62/0x70
Nov 11 21:25:50 10.243.1.49 [ 3920.649771] hardirqs last disabled at (2472): [<ffffffffa41aaaa7>] __up_console_sem+0x47/0x70
Nov 11 21:25:50 10.243.1.49 [ 3920.682095] softirqs last  enabled at (286): [<ffffffffa5400403>] __do_softirq+0x403/0x57b
Nov 11 21:25:50 10.243.1.49 [ 3920.713632] softirqs last disabled at (275): [<ffffffffa40d261d>] irq_exit_rcu+0x7d/0x150
Nov 11 21:25:50 10.243.1.49 [ 3920.744811] ---[ end trace 87ffbd6a737b5b88 ]---
Nov 11 21:26:01 10.243.1.49 [ 3932.344700] ------------[ cut here ]------------
Nov 11 21:26:01 10.243.1.49 [ 3932.344708] list_del corruption, ffff888151798240->next is LIST_POISON1 (dead000000000100)
Nov 11 21:26:02 10.243.1.49 [ 3932.344718] EVL: RxWorker1 switching in-band [pid=16135, excpt=6, __list_del_entry_valid+0xc5/0x120]
Nov 11 21:26:02 10.243.1.49 [ 3932.449176] WARNING: CPU: 9 PID: 16135 at lib/list_debug.c:55 __list_del_entry_valid+0xc5/0x120
Nov 11 21:26:02 10.243.1.49 [ 3932.449200] Modules linked in: xt_CHECKSUM xt_MASQUERADE tun netconsole ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 xt_conntrack ebtable_nat ip6table_nat ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_nat iptable_mangle iptable_security iptable_raw nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 ip_set nfnetlink ebtable_filter ebtables rfkill ip6table_filter ip6_tables iptable_filter sunrpc dm_mirror dm_region_hash dm_log dm_mod iTCO_wdt gpio_ich iTCO_vendor_support mxm_wmi x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm irqbypass crct10dif_pclmul crc32_pclmul ghash_clmulni_intel aesni_intel crypto_simd cryptd rapl mei_me i2c_i801 lpc_ich intel_cstate pcspkr intel_pch_thermal input_leds sg joydev ftdi_sio i2c_smbus mfd_core mei acpi_ipmi ioatdma ipmi_si ipmi_devintf ipmi_msghandler wmi ip_tables xfs libcrc32c ast drm_vram_helper sd_mod drm_kms_helper sr_mod cdrom t10_pi syscopyarea sysfillrect sysimgblt fb_sys_fops drm_ttm_helper ttm drm
Nov 11 21:26:02 10.243.1.49
Nov 11 21:26:02 10.243.1.49 [ 3932.449522]  ixgbe igb ahci libahci crc32c_intel libata mdio ptp i2c_algo_bit pps_core dca
Nov 11 21:26:02 10.243.1.49 [ 3932.449565] CPU: 9 PID: 16135 Comm: RxWorker1 Tainted: G        W         5.15.77evl-gae6080e09d9a #1
Nov 11 21:26:02 10.243.1.49 [ 3932.449578] Hardware name: Supermicro Super Server/X10SDV-6C-TLN4F, BIOS 1.3 02/13/2018
Nov 11 21:26:02 10.243.1.49 [ 3932.449583] IRQ stage: Linux
Nov 11 21:26:02 10.243.1.49 [ 3932.449588] RIP: 0010:__list_del_entry_valid+0xc5/0x120
Nov 11 21:26:02 10.243.1.49 [ 3932.449604] Code: c0 eb e0 48 89 de 48 c7 c7 00 0c 82 a5 e8 7c fa 8b 00 0f 0b 31 c0 eb cb 4c 89 e2 48 89 de 48 c7 c7 60 0c 82 a5 e8 64 fa 8b 00 <0f> 0b 31 c0 eb b3 4c 89 ea 48 89 de 48 c7 c7 c0 0c 82 a5 e8 4c fa
Nov 11 21:26:02 10.243.1.49 [ 3932.449614] RSP: 0018:ffff88815ef47ac8 EFLAGS: 00010082
Nov 11 21:26:02 10.243.1.49 [ 3932.449623] RAX: 0000000000000000 RBX: ffff888151798240 RCX: 0000000000000027
Nov 11 21:26:02 10.243.1.49 [ 3932.449631] RDX: 0000000000000027 RSI: dffffc0000000000 RDI: ffff88841c2a7b18
Nov 11 21:26:02 10.243.1.49 [ 3932.449639] RBP: ffff88815ef47ae0 R08: ffffed1083854f64 R09: ffffed1083854f64
Nov 11 21:26:02 10.243.1.49 [ 3932.449648] R10: ffff88841c2a7b1b R11: ffffed1083854f63 R12: dead000000000100
Nov 11 21:26:02 10.243.1.49 [ 3932.449656] R13: dead000000000122 R14: ffff888151798000 R15: ffff888145380000
Nov 11 21:26:02 10.243.1.49 [ 3932.449664] FS:  00007f7312ffe700(0000) GS:ffff88841c280000(0000) knlGS:0000000000000000
Nov 11 21:26:02 10.243.1.49 [ 3932.449674] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Nov 11 21:26:02 10.243.1.49 [ 3932.449681] CR2: 0000000000000000 CR3: 000000016e080005 CR4: 00000000003706e0
Nov 11 21:26:02 10.243.1.49 [ 3932.449689] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Nov 11 21:26:03 10.243.1.49 [ 3932.449695] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Nov 11 21:26:03 10.243.1.49 [ 3932.449702] Call Trace:
Nov 11 21:26:03 10.243.1.49 [ 3932.449706]  <TASK>
Nov 11 21:26:03 10.243.1.49 [ 3932.449725]  __untrack_event.part.9+0x26/0xe0
Nov 11 21:26:03 10.243.1.49 [ 3932.449762]  monitor_oob_ioctl+0x1356/0x1490
Nov 11 21:26:03 10.243.1.49 [ 3932.449823]  ? enter_monitor+0x160/0x160
Nov 11 21:26:03 10.243.1.49 [ 3932.449872]  ? rcu_read_lock_held_common+0x25/0x70
Nov 11 21:26:03 10.243.1.49 [ 3932.449920]  ? do_raw_spin_lock+0x119/0x1e0
Nov 11 21:26:03 10.243.1.49 [ 3932.449938]  ? spin_dump+0x90/0x90
Nov 11 21:26:03 10.243.1.49 [ 3932.449976]  ? __kasan_check_read+0x11/0x20
Nov 11 21:26:03 10.243.1.49 [ 3932.450040]  EVL_ioctl+0x81/0xf0
Nov 11 21:26:03 10.243.1.49 [ 3932.450075]  do_oob_syscall+0x699/0x6b0
Nov 11 21:26:03 10.243.1.49 [ 3932.450105]  ? prepare_for_signal+0x180/0x180
Nov 11 21:26:03 10.243.1.49 [ 3932.450134]  ? do_oob_irq+0x283/0x7e0
Nov 11 21:26:03 10.243.1.49 [ 3932.450148]  ? arch_pipeline_entry+0xce/0x110
Nov 11 21:26:03 10.243.1.49 [ 3932.450185]  handle_oob_syscall+0x189/0x230
Nov 11 21:26:03 10.243.1.49 [ 3932.450215]  ? handle_pipelined_syscall+0x820/0x820
Nov 11 21:26:03 10.243.1.49 [ 3932.450252]  ? rcu_read_lock_sched_held+0x5b/0xd0
Nov 11 21:26:03 10.243.1.49 [ 3932.450272]  ? rcu_read_lock_any_held.part.15+0x20/0x20
Nov 11 21:26:03 10.243.1.49 [ 3932.450315]  pipeline_syscall+0xa7/0x1c0
Nov 11 21:26:03 10.243.1.49 [ 3932.450354]  syscall_enter_from_user_mode+0x47/0x110
Nov 11 21:26:03 10.243.1.49 [ 3932.450380]  do_syscall_64+0x15/0xb0
Nov 11 21:26:03 10.243.1.49 [ 3932.450400]  entry_SYSCALL_64_after_hwframe+0x61/0xcb
Nov 11 21:26:03 10.243.1.49 [ 3932.450413] RIP: 0033:0x7f735de5a940
Nov 11 21:26:03 10.243.1.49 [ 3932.450425] Code: 77 ff ff bf 02 00 00 10 8b 44 24 0c 48 98 48 89 c6 48 8b 04 24 48 89 c2 4c 8b 54 24 38 41 b8 00 00 00 00 b8 9d 00 00 00 0f 05 <48> 89 44 24 30 48 8b 44 24 30 89 44 24 2c 8b 44 24 28 be 00 00 00
Nov 11 21:26:03 10.243.1.49 [ 3932.450434] RSP: 002b:00007f7312fee340 EFLAGS: 00000202 ORIG_RAX: 000000000000009d
Nov 11 21:26:03 10.243.1.49 [ 3932.450445] RAX: ffffffffffffffda RBX: 0000000000000f5c RCX: 00007f735de5a940
Nov 11 21:26:03 10.243.1.49 [ 3932.450453] RDX: 00000000c0186d03 RSI: 000000000000000d RDI: 0000000010000002
Nov 11 21:26:03 10.243.1.49 [ 3932.450460] RBP: 00007f7312fee520 R08: 0000000000000000 R09: 00007ffc09389080
Nov 11 21:26:03 10.243.1.49 [ 3932.450467] R10: 00007f7312fee4f0 R11: 0000000000000202 R12: 00007f7312fee600
Nov 11 21:26:03 10.243.1.49 [ 3932.450475] R13: 0000000000000000 R14: 0000000025f8f620 R15: 00007f7312fee580
Nov 11 21:26:03 10.243.1.49 [ 3932.450556]  </TASK>
Nov 11 21:26:03 10.243.1.49 [ 3932.450560] irq event stamp: 2335
Nov 11 21:26:03 10.243.1.49 [ 3932.450563] hardirqs last  enabled at (2335): [<ffffffffa51330bc>] _raw_spin_unlock_irqrestore+0x4c/0x50
Nov 11 21:26:03 10.243.1.49 [ 3932.450578] hardirqs last disabled at (2334): [<ffffffffa5132e2b>] _raw_spin_lock_irqsave+0x1b/0x60
Nov 11 21:26:03 10.243.1.49 [ 3932.450591] softirqs last  enabled at (1488): [<ffffffffa5400403>] __do_softirq+0x403/0x57b
Nov 11 21:26:03 10.243.1.49 [ 3932.450606] softirqs last disabled at (1479): [<ffffffffa40d261d>] irq_exit_rcu+0x7d/0x150
Nov 11 21:26:04 10.243.1.49 [ 3932.450623] ---[ end trace 87ffbd6a737b5b89 ]---
Nov 11 21:26:04 10.243.1.49 [ 3934.468351] EVL: RxWorker1 resuming out-of-band [pid=16135, excpt=6, __list_del_entry_valid+0xc7/0x120]
Nov 11 21:26:04 10.243.1.49 [ 3934.505195] ------------[ cut here ]------------
Nov 11 21:26:04 10.243.1.49 [ 3934.523357] syscall 157 left IRQs disabled
Nov 11 21:26:04 10.243.1.49 [ 3934.523412] WARNING: CPU: 9 PID: 16135 at kernel/entry/common.c:353 syscall_exit_to_user_mode_prepare+0x37/0x270
Nov 11 21:26:04 10.243.1.49 [ 3934.575073] Modules linked in: xt_CHECKSUM xt_MASQUERADE tun netconsole ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 xt_conntrack ebtable_nat ip6table_nat ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_nat iptable_mangle iptable_security iptable_raw nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 ip_set nfnetlink ebtable_filter ebtables rfkill ip6table_filter ip6_tables iptable_filter sunrpc dm_mirror dm_region_hash dm_log dm_mod iTCO_wdt gpio_ich iTCO_vendor_support mxm_wmi x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm irqbypass crct10dif_pclmul crc32_pclmul ghash_clmulni_intel aesni_intel crypto_simd cryptd rapl mei_me i2c_i801 lpc_ich intel_cstate pcspkr intel_pch_thermal input_leds sg joydev ftdi_sio i2c_smbus mfd_core mei acpi_ipmi ioatdma ipmi_si ipmi_devintf ipmi_msghandler wmi ip_tables xfs libcrc32c ast drm_vram_helper sd_mod drm_kms_helper sr_mod cdrom t10_pi syscopyarea sysfillrect sysimgblt fb_sys_fops drm_ttm_helper ttm drm
Nov 11 21:26:04 10.243.1.49
Nov 11 21:26:04 10.243.1.49 [ 3934.575624]  ixgbe igb ahci libahci crc32c_intel libata mdio ptp i2c_algo_bit pps_core dca
Nov 11 21:26:04 10.243.1.49 [ 3934.908567] CPU: 9 PID: 16135 Comm: RxWorker1 Tainted: G        W         5.15.77evl-gae6080e09d9a #1
Nov 11 21:26:04 10.243.1.49 [ 3934.942053] Hardware name: Supermicro Super Server/X10SDV-6C-TLN4F, BIOS 1.3 02/13/2018
Nov 11 21:26:04 10.243.1.49 [ 3934.971930] IRQ stage: Linux
Nov 11 21:26:04 10.243.1.49 [ 3934.986372] RIP: 0010:syscall_exit_to_user_mode_prepare+0x37/0x270
Nov 11 21:26:04 10.243.1.49 [ 3935.011814] Code: 41 55 41 54 49 89 fc 53 4c 8b 68 08 48 8b 5f 78 eb 36 e8 0c fe f2 00 85 c0 74 1c 48 63 f3 48 c7 c7 70 66 ca a5 e8 42 f0 ec 00 <0f> 0b fb e8 e1 a1 0d 00 e8 bc af fd ff 41 f6 c5 76 75 3a 5b 41 5c
Nov 11 21:26:04 10.243.1.49 [ 3935.080613] RSP: 0018:ffff88815ef47f00 EFLAGS: 00010282
Nov 11 21:26:04 10.243.1.49 [ 3935.102656] RAX: 0000000000000000 RBX: 000000000000009d RCX: 0000000000000027
Nov 11 21:26:04 10.243.1.49 [ 3935.130510] RDX: 0000000000000027 RSI: dffffc0000000000 RDI: ffff88841c2a7b18
Nov 11 21:26:04 10.243.1.49 [ 3935.158377] RBP: ffff88815ef47f18 R08: ffffed1083854f64 R09: ffffed1083854f64
Nov 11 21:26:04 10.243.1.49 [ 3935.186296] R10: ffff88841c2a7b1b R11: ffffed1083854f63 R12: ffff88815ef47f58
Nov 11 21:26:04 10.243.1.49 [ 3935.214788] R13: 0000000000000010 R14: 0000000000000000 R15: 0000000000000000
Nov 11 21:26:04 10.243.1.49 [ 3935.242756] FS:  00007f7312ffe700(0000) GS:ffff88841c280000(0000) knlGS:0000000000000000
Nov 11 21:26:04 10.243.1.49 [ 3935.273674] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Nov 11 21:26:04 10.243.1.49 [ 3935.297585] CR2: 0000000000000000 CR3: 000000016e080005 CR4: 00000000003706e0
Nov 11 21:26:04 10.243.1.49 [ 3935.325742] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Nov 11 21:26:04 10.243.1.49 [ 3935.353906] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Nov 11 21:26:04 10.243.1.49 [ 3935.382008] Call Trace:
Nov 11 21:26:04 10.243.1.49 [ 3935.395984]  <TASK>
Nov 11 21:26:04 10.243.1.49 [ 3935.409811]  syscall_exit_to_user_mode+0xd/0x90
Nov 11 21:26:04 10.243.1.49 [ 3935.429912]  do_syscall_64+0x50/0xb0
Nov 11 21:26:05 10.243.1.49 [ 3935.446955]  entry_SYSCALL_64_after_hwframe+0x61/0xcb
Nov 11 21:26:05 10.243.1.49 [ 3935.468406] RIP: 0033:0x7f735de5a940
Nov 11 21:26:05 10.243.1.49 [ 3935.485360] Code: 77 ff ff bf 02 00 00 10 8b 44 24 0c 48 98 48 89 c6 48 8b 04 24 48 89 c2 4c 8b 54 24 38 41 b8 00 00 00 00 b8 9d 00 00 00 0f 05 <48> 89 44 24 30 48 8b 44 24 30 89 44 24 2c 8b 44 24 28 be 00 00 00
Nov 11 21:26:05 10.243.1.49 [ 3935.554757] RSP: 002b:00007f7312fee340 EFLAGS: 00000202 ORIG_RAX: 000000000000009d
Nov 11 21:26:05 10.243.1.49 [ 3935.584179] RAX: 0000000000000000 RBX: 0000000000000f5c RCX: 00007f735de5a940
Nov 11 21:26:05 10.243.1.49 [ 3935.613249] RDX: 00000000c0186d03 RSI: 000000000000000d RDI: 0000000010000002
Nov 11 21:26:05 10.243.1.49 [ 3935.641352] RBP: 00007f7312fee520 R08: 0000000000000000 R09: 00007ffc09389080
Nov 11 21:26:05 10.243.1.49 [ 3935.669502] R10: 00007f7312fee4f0 R11: 0000000000000202 R12: 00007f7312fee600
Nov 11 21:26:05 10.243.1.49 [ 3935.697540] R13: 0000000000000000 R14: 0000000025f8f620 R15: 00007f7312fee580
Nov 11 21:26:05 10.243.1.49 [ 3935.725630]  </TASK>
Nov 11 21:26:05 10.243.1.49 [ 3935.738718] irq event stamp: 3045
Nov 11 21:26:05 10.243.1.49 [ 3935.755162] hardirqs last  enabled at (3053): [<ffffffffa41aaac2>] __up_console_sem+0x62/0x70
Nov 11 21:26:05 10.243.1.49 [ 3935.787382] hardirqs last disabled at (3060): [<ffffffffa41aaaa7>] __up_console_sem+0x47/0x70
Nov 11 21:26:05 10.243.1.49 [ 3935.819592] softirqs last  enabled at (2346): [<ffffffffa5400403>] __do_softirq+0x403/0x57b
Nov 11 21:26:05 10.243.1.49 [ 3935.851088] softirqs last disabled at (2339): [<ffffffffa40d261d>] irq_exit_rcu+0x7d/0x150
Nov 11 21:26:05 10.243.1.49 [ 3935.882340] ---[ end trace 87ffbd6a737b5b8a ]---
Nov 11 21:26:10 10.243.1.49 [ 3940.614587] ------------[ cut here ]------------
Nov 11 21:26:10 10.243.1.49 [ 3940.614594] list_del corruption, ffff888151798240->next is LIST_POISON1 (dead000000000100)
Nov 11 21:26:10 10.243.1.49 [ 3940.614604] EVL: RxWorker1 switching in-band [pid=16135, excpt=6, __list_del_entry_valid+0xc5/0x120]
Nov 11 21:26:10 10.243.1.49 [ 3940.717204] WARNING: CPU: 9 PID: 16135 at lib/list_debug.c:55 __list_del_entry_valid+0xc5/0x120
Nov 11 21:26:10 10.243.1.49 [ 3940.717228] Modules linked in: xt_CHECKSUM xt_MASQUERADE tun netconsole ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 xt_conntrack ebtable_nat ip6table_nat ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_nat iptable_mangle iptable_security iptable_raw nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 ip_set nfnetlink ebtable_filter ebtables rfkill ip6table_filter ip6_tables iptable_filter sunrpc dm_mirror dm_region_hash dm_log dm_mod iTCO_wdt gpio_ich iTCO_vendor_support mxm_wmi x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm irqbypass crct10dif_pclmul crc32_pclmul ghash_clmulni_intel aesni_intel crypto_simd cryptd rapl mei_me i2c_i801 lpc_ich intel_cstate pcspkr intel_pch_thermal input_leds sg joydev ftdi_sio i2c_smbus mfd_core mei acpi_ipmi ioatdma ipmi_si ipmi_devintf ipmi_msghandler wmi ip_tables xfs libcrc32c ast drm_vram_helper sd_mod drm_kms_helper sr_mod cdrom t10_pi syscopyarea sysfillrect sysimgblt fb_sys_fops drm_ttm_helper ttm drm
Nov 11 21:26:10 10.243.1.49
Nov 11 21:26:10 10.243.1.49 [ 3940.717553]  ixgbe igb ahci libahci crc32c_intel libata mdio ptp i2c_algo_bit pps_core dca
Nov 11 21:26:10 10.243.1.49 [ 3940.717596] CPU: 9 PID: 16135 Comm: RxWorker1 Tainted: G        W         5.15.77evl-gae6080e09d9a #1
Nov 11 21:26:10 10.243.1.49 [ 3940.717609] Hardware name: Supermicro Super Server/X10SDV-6C-TLN4F, BIOS 1.3 02/13/2018
Nov 11 21:26:10 10.243.1.49 [ 3940.717615] IRQ stage: Linux
Nov 11 21:26:10 10.243.1.49 [ 3940.717619] RIP: 0010:__list_del_entry_valid+0xc5/0x120
Nov 11 21:26:10 10.243.1.49 [ 3940.717634] Code: c0 eb e0 48 89 de 48 c7 c7 00 0c 82 a5 e8 7c fa 8b 00 0f 0b 31 c0 eb cb 4c 89 e2 48 89 de 48 c7 c7 60 0c 82 a5 e8 64 fa 8b 00 <0f> 0b 31 c0 eb b3 4c 89 ea 48 89 de 48 c7 c7 c0 0c 82 a5 e8 4c fa
Nov 11 21:26:10 10.243.1.49 [ 3940.717644] RSP: 0018:ffff88815ef47ac8 EFLAGS: 00010082
Nov 11 21:26:10 10.243.1.49 [ 3940.717654] RAX: 0000000000000000 RBX: ffff888151798240 RCX: 0000000000000027
Nov 11 21:26:10 10.243.1.49 [ 3940.717662] RDX: 0000000000000027 RSI: dffffc0000000000 RDI: ffff88841c2a7b18
Nov 11 21:26:11 10.243.1.49 [ 3940.717669] RBP: ffff88815ef47ae0 R08: ffffed1083854f64 R09: ffffed1083854f64
Nov 11 21:26:11 10.243.1.49 [ 3940.717678] R10: ffff88841c2a7b1b R11: ffffed1083854f63 R12: dead000000000100
Nov 11 21:26:11 10.243.1.49 [ 3940.717686] R13: dead000000000122 R14: ffff888151798000 R15: ffff888145380000
Nov 11 21:26:11 10.243.1.49 [ 3940.717694] FS:  00007f7312ffe700(0000) GS:ffff88841c280000(0000) knlGS:0000000000000000
Nov 11 21:26:11 10.243.1.49 [ 3940.717704] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Nov 11 21:26:11 10.243.1.49 [ 3940.717712] CR2: 0000000000000000 CR3: 000000016e080005 CR4: 00000000003706e0
Nov 11 21:26:11 10.243.1.49 [ 3940.717719] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Nov 11 21:26:11 10.243.1.49 [ 3940.717726] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Nov 11 21:26:11 10.243.1.49 [ 3940.717733] Call Trace:
Nov 11 21:26:11 10.243.1.49 [ 3940.717736]  <TASK>
Nov 11 21:26:11 10.243.1.49 [ 3940.717755]  __untrack_event.part.9+0x26/0xe0
Nov 11 21:26:11 10.243.1.49 [ 3940.717793]  monitor_oob_ioctl+0x1356/0x1490
Nov 11 21:26:11 10.243.1.49 [ 3940.717854]  ? enter_monitor+0x160/0x160
Nov 11 21:26:11 10.243.1.49 [ 3940.717871]  ? __kasan_check_read+0x11/0x20
Nov 11 21:26:11 10.243.1.49 [ 3940.717888]  ? do_raw_spin_unlock+0x97/0x140
Nov 11 21:26:11 10.243.1.49 [ 3940.717914]  ? do_clock_tick+0x40a/0x630
Nov 11 21:26:11 10.243.1.49 [ 3940.717928]  ? rcu_read_lock_held_common+0x25/0x70
Nov 11 21:26:11 10.243.1.49 [ 3940.717975]  ? do_raw_spin_lock+0x119/0x1e0
Nov 11 21:26:11 10.243.1.49 [ 3940.717992]  ? spin_dump+0x90/0x90
Nov 11 21:26:11 10.243.1.49 [ 3940.718030]  ? __kasan_check_read+0x11/0x20
Nov 11 21:26:11 10.243.1.49 [ 3940.718091]  EVL_ioctl+0x81/0xf0
Nov 11 21:26:11 10.243.1.49 [ 3940.718127]  do_oob_syscall+0x699/0x6b0
Nov 11 21:26:11 10.243.1.49 [ 3940.718156]  ? prepare_for_signal+0x180/0x180
Nov 11 21:26:11 10.243.1.49 [ 3940.718186]  ? do_oob_irq+0x283/0x7e0
Nov 11 21:26:11 10.243.1.49 [ 3940.718196]  ? rcu_dynticks_eqs_enter+0xe/0x20
Nov 11 21:26:11 10.243.1.49 [ 3940.718237]  handle_oob_syscall+0x189/0x230
Nov 11 21:26:11 10.243.1.49 [ 3940.718267]  ? handle_pipelined_syscall+0x820/0x820
Nov 11 21:26:11 10.243.1.49 [ 3940.718304]  ? rcu_read_lock_sched_held+0x5b/0xd0
Nov 11 21:26:11 10.243.1.49 [ 3940.718325]  ? rcu_read_lock_any_held.part.15+0x20/0x20
Nov 11 21:26:11 10.243.1.49 [ 3940.718367]  pipeline_syscall+0xa7/0x1c0
Nov 11 21:26:11 10.243.1.49 [ 3940.718406]  syscall_enter_from_user_mode+0x47/0x110
Nov 11 21:26:11 10.243.1.49 [ 3940.718433]  do_syscall_64+0x15/0xb0
Nov 11 21:26:11 10.243.1.49 [ 3940.718453]  entry_SYSCALL_64_after_hwframe+0x61/0xcb
Nov 11 21:26:11 10.243.1.49 [ 3940.718466] RIP: 0033:0x7f735de5a940
Nov 11 21:26:11 10.243.1.49 [ 3940.718477] Code: 77 ff ff bf 02 00 00 10 8b 44 24 0c 48 98 48 89 c6 48 8b 04 24 48 89 c2 4c 8b 54 24 38 41 b8 00 00 00 00 b8 9d 00 00 00 0f 05 <48> 89 44 24 30 48 8b 44 24 30 89 44 24 2c 8b 44 24 28 be 00 00 00
Nov 11 21:26:11 10.243.1.49 [ 3940.718487] RSP: 002b:00007f7312fee340 EFLAGS: 00000202 ORIG_RAX: 000000000000009d
Nov 11 21:26:12 10.243.1.49 [ 3940.718498] RAX: ffffffffffffffda RBX: 0000000000000f64 RCX: 00007f735de5a940
Nov 11 21:26:12 10.243.1.49 [ 3940.718506] RDX: 00000000c0186d03 RSI: 000000000000000d RDI: 0000000010000002
Nov 11 21:26:12 10.243.1.49 [ 3940.718513] RBP: 00007f7312fee520 R08: 0000000000000000 R09: 00007ffc09389080
Nov 11 21:26:12 10.243.1.49 [ 3940.718520] R10: 00007f7312fee4f0 R11: 0000000000000202 R12: 00007f7312fee600
Nov 11 21:26:12 10.243.1.49 [ 3940.718527] R13: 0000000000000000 R14: 00000000360f356d R15: 00007f7312fee580
Nov 11 21:26:12 10.243.1.49 [ 3940.718609]  </TASK>
Nov 11 21:26:12 10.243.1.49 [ 3940.718613] irq event stamp: 3141
Nov 11 21:26:12 10.243.1.49 [ 3940.718616] hardirqs last  enabled at (3141): [<ffffffffa51330bc>] _raw_spin_unlock_irqrestore+0x4c/0x50
Nov 11 21:26:12 10.243.1.49 [ 3940.718631] hardirqs last disabled at (3140): [<ffffffffa5132e2b>] _raw_spin_lock_irqsave+0x1b/0x60
Nov 11 21:26:12 10.243.1.49 [ 3940.718645] softirqs last  enabled at (2346): [<ffffffffa5400403>] __do_softirq+0x403/0x57b
Nov 11 21:26:12 10.243.1.49 [ 3940.718660] softirqs last disabled at (2339): [<ffffffffa40d261d>] irq_exit_rcu+0x7d/0x150
Nov 11 21:26:12 10.243.1.49 [ 3940.718676] ---[ end trace 87ffbd6a737b5b8b ]---
Nov 11 21:26:12 10.243.1.49 [ 3942.781537] EVL: RxWorker1 resuming out-of-band [pid=16135, excpt=6, __list_del_entry_valid+0xc7/0x120]
Nov 11 21:26:12 10.243.1.49 [ 3942.818240] ------------[ cut here ]------------
Nov 11 21:26:12 10.243.1.49 [ 3942.836295] syscall 157 left IRQs disabled
Nov 11 21:26:12 10.243.1.49 [ 3942.836351] WARNING: CPU: 9 PID: 16135 at kernel/entry/common.c:353 syscall_exit_to_user_mode_prepare+0x37/0x270
Nov 11 21:26:12 10.243.1.49 [ 3942.887649] Modules linked in: xt_CHECKSUM xt_MASQUERADE tun netconsole ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 xt_conntrack ebtable_nat ip6table_nat ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_nat iptable_mangle iptable_security iptable_raw nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 ip_set nfnetlink ebtable_filter ebtables rfkill ip6table_filter ip6_tables iptable_filter sunrpc dm_mirror dm_region_hash dm_log dm_mod iTCO_wdt gpio_ich iTCO_vendor_support mxm_wmi x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm irqbypass crct10dif_pclmul crc32_pclmul ghash_clmulni_intel aesni_intel crypto_simd cryptd rapl mei_me i2c_i801 lpc_ich intel_cstate pcspkr intel_pch_thermal input_leds sg joydev ftdi_sio i2c_smbus mfd_core mei acpi_ipmi ioatdma ipmi_si ipmi_devintf ipmi_msghandler wmi ip_tables xfs libcrc32c ast drm_vram_helper sd_mod drm_kms_helper sr_mod cdrom t10_pi syscopyarea sysfillrect sysimgblt fb_sys_fops drm_ttm_helper ttm drm
Nov 11 21:26:12 10.243.1.49
Nov 11 21:26:12 10.243.1.49 [ 3942.888154]  ixgbe igb ahci libahci crc32c_intel libata mdio ptp i2c_algo_bit pps_core dca
Nov 11 21:26:12 10.243.1.49 [ 3943.219885] CPU: 9 PID: 16135 Comm: RxWorker1 Tainted: G        W         5.15.77evl-gae6080e09d9a #1
Nov 11 21:26:12 10.243.1.49 [ 3943.253383] Hardware name: Supermicro Super Server/X10SDV-6C-TLN4F, BIOS 1.3 02/13/2018
Nov 11 21:26:12 10.243.1.49 [ 3943.283317] IRQ stage: Linux
Nov 11 21:26:12 10.243.1.49 [ 3943.297827] RIP: 0010:syscall_exit_to_user_mode_prepare+0x37/0x270
Nov 11 21:26:12 10.243.1.49 [ 3943.323263] Code: 41 55 41 54 49 89 fc 53 4c 8b 68 08 48 8b 5f 78 eb 36 e8 0c fe f2 00 85 c0 74 1c 48 63 f3 48 c7 c7 70 66 ca a5 e8 42 f0 ec 00 <0f> 0b fb e8 e1 a1 0d 00 e8 bc af fd ff 41 f6 c5 76 75 3a 5b 41 5c
Nov 11 21:26:12 10.243.1.49 [ 3943.392085] RSP: 0018:ffff88815ef47f00 EFLAGS: 00010282
Nov 11 21:26:12 10.243.1.49 [ 3943.414133] RAX: 0000000000000000 RBX: 000000000000009d RCX: 0000000000000027
Nov 11 21:26:13 10.243.1.49 [ 3943.441990] RDX: 0000000000000027 RSI: dffffc0000000000 RDI: ffff88841c2a7b18
Nov 11 21:26:13 10.243.1.49 [ 3943.469893] RBP: ffff88815ef47f18 R08: ffffed1083854f64 R09: ffffed1083854f64
Nov 11 21:26:13 10.243.1.49 [ 3943.497833] R10: ffff88841c2a7b1b R11: ffffed1083854f63 R12: ffff88815ef47f58
Nov 11 21:26:13 10.243.1.49 [ 3943.526730] R13: 0000000000000010 R14: 0000000000000000 R15: 0000000000000000
Nov 11 21:26:13 10.243.1.49 [ 3943.554742] FS:  00007f7312ffe700(0000) GS:ffff88841c280000(0000) knlGS:0000000000000000
Nov 11 21:26:13 10.243.1.49 [ 3943.585690] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Nov 11 21:26:13 10.243.1.49 [ 3943.609617] CR2: 0000000000000000 CR3: 000000016e080005 CR4: 00000000003706e0
Nov 11 21:26:13 10.243.1.49 [ 3943.637821] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Nov 11 21:26:13 10.243.1.49 [ 3943.665965] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Nov 11 21:26:13 10.243.1.49 [ 3943.694083] Call Trace:
Nov 11 21:26:13 10.243.1.49 [ 3943.708081]  <TASK>
Nov 11 21:26:13 10.243.1.49 [ 3943.722275]  syscall_exit_to_user_mode+0xd/0x90
Nov 11 21:26:13 10.243.1.49 [ 3943.742387]  do_syscall_64+0x50/0xb0
Nov 11 21:26:13 10.243.1.49 [ 3943.759505]  entry_SYSCALL_64_after_hwframe+0x61/0xcb
Nov 11 21:26:13 10.243.1.49 [ 3943.780956] RIP: 0033:0x7f735de5a940
Nov 11 21:26:13 10.243.1.49 [ 3943.797910] Code: 77 ff ff bf 02 00 00 10 8b 44 24 0c 48 98 48 89 c6 48 8b 04 24 48 89 c2 4c 8b 54 24 38 41 b8 00 00 00 00 b8 9d 00 00 00 0f 05 <48> 89 44 24 30 48 8b 44 24 30 89 44 24 2c 8b 44 24 28 be 00 00 00
Nov 11 21:26:13 10.243.1.49 [ 3943.867297] RSP: 002b:00007f7312fee340 EFLAGS: 00000202 ORIG_RAX: 000000000000009d
Nov 11 21:26:13 10.243.1.49 [ 3943.896714] RAX: 0000000000000000 RBX: 0000000000000f64 RCX: 00007f735de5a940
Nov 11 21:26:13 10.243.1.49 [ 3943.925810] RDX: 00000000c0186d03 RSI: 000000000000000d RDI: 0000000010000002
Nov 11 21:26:13 10.243.1.49 [ 3943.953912] RBP: 00007f7312fee520 R08: 0000000000000000 R09: 00007ffc09389080
Nov 11 21:26:13 10.243.1.49 [ 3943.982042] R10: 00007f7312fee4f0 R11: 0000000000000202 R12: 00007f7312fee600
Nov 11 21:26:13 10.243.1.49 [ 3944.010078] R13: 0000000000000000 R14: 00000000360f356d R15: 00007f7312fee580
Nov 11 21:26:13 10.243.1.49 [ 3944.038147]  </TASK>
Nov 11 21:26:13 10.243.1.49 [ 3944.051224] irq event stamp: 3849
Nov 11 21:26:13 10.243.1.49 [ 3944.067708] hardirqs last  enabled at (3857): [<ffffffffa41aaac2>] __up_console_sem+0x62/0x70
Nov 11 21:26:13 10.243.1.49 [ 3944.099955] hardirqs last disabled at (3864): [<ffffffffa41aaaa7>] __up_console_sem+0x47/0x70
Nov 11 21:26:13 10.243.1.49 [ 3944.132084] softirqs last  enabled at (3156): [<ffffffffa5400403>] __do_softirq+0x403/0x57b
Nov 11 21:26:13 10.243.1.49 [ 3944.163602] softirqs last disabled at (3145): [<ffffffffa40d261d>] irq_exit_rcu+0x7d/0x150
Nov 11 21:26:13 10.243.1.49 [ 3944.194885] ---[ end trace 87ffbd6a737b5b8c ]---
Nov 11 21:26:42 10.243.1.49 [ 3973.340652] ------------[ cut here ]------------
Nov 11 21:26:42 10.243.1.49 [ 3973.340659] list_del corruption, ffff888151798240->next is LIST_POISON1 (dead000000000100)
Nov 11 21:26:42 10.243.1.49 [ 3973.340669] EVL: RxWorker1 switching in-band [pid=16135, excpt=6, __list_del_entry_valid+0xc5/0x120]
Nov 11 21:26:43 10.243.1.49 [ 3973.443226] WARNING: CPU: 9 PID: 16135 at lib/list_debug.c:55 __list_del_entry_valid+0xc5/0x120
Nov 11 21:26:43 10.243.1.49 [ 3973.443250] Modules linked in: xt_CHECKSUM xt_MASQUERADE tun netconsole ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 xt_conntrack ebtable_nat ip6table_nat ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_nat iptable_mangle iptable_security iptable_raw nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 ip_set nfnetlink ebtable_filter ebtables rfkill ip6table_filter ip6_tables iptable_filter sunrpc dm_mirror dm_region_hash dm_log dm_mod iTCO_wdt gpio_ich iTCO_vendor_support mxm_wmi x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm irqbypass crct10dif_pclmul crc32_pclmul ghash_clmulni_intel aesni_intel crypto_simd cryptd rapl mei_me i2c_i801 lpc_ich intel_cstate pcspkr intel_pch_thermal input_leds sg joydev ftdi_sio i2c_smbus mfd_core mei acpi_ipmi ioatdma ipmi_si ipmi_devintf ipmi_msghandler wmi ip_tables xfs libcrc32c ast drm_vram_helper sd_mod drm_kms_helper sr_mod cdrom t10_pi syscopyarea sysfillrect sysimgblt fb_sys_fops drm_ttm_helper ttm drm
Nov 11 21:26:43 10.243.1.49
Nov 11 21:26:43 10.243.1.49 [ 3973.443573]  ixgbe igb ahci libahci crc32c_intel libata mdio ptp i2c_algo_bit pps_core dca
Nov 11 21:26:43 10.243.1.49 [ 3973.443616] CPU: 9 PID: 16135 Comm: RxWorker1 Tainted: G        W         5.15.77evl-gae6080e09d9a #1
Nov 11 21:26:43 10.243.1.49 [ 3973.443629] Hardware name: Supermicro Super Server/X10SDV-6C-TLN4F, BIOS 1.3 02/13/2018
Nov 11 21:26:43 10.243.1.49 [ 3973.443634] IRQ stage: Linux
Nov 11 21:26:43 10.243.1.49 [ 3973.443639] RIP: 0010:__list_del_entry_valid+0xc5/0x120
Nov 11 21:26:43 10.243.1.49 [ 3973.443654] Code: c0 eb e0 48 89 de 48 c7 c7 00 0c 82 a5 e8 7c fa 8b 00 0f 0b 31 c0 eb cb 4c 89 e2 48 89 de 48 c7 c7 60 0c 82 a5 e8 64 fa 8b 00 <0f> 0b 31 c0 eb b3 4c 89 ea 48 89 de 48 c7 c7 c0 0c 82 a5 e8 4c fa
Nov 11 21:26:43 10.243.1.49 [ 3973.443664] RSP: 0018:ffff88815ef47ac8 EFLAGS: 00010082
Nov 11 21:26:43 10.243.1.49 [ 3973.443674] RAX: 0000000000000000 RBX: ffff888151798240 RCX: 0000000000000027
Nov 11 21:26:43 10.243.1.49 [ 3973.443682] RDX: 0000000000000027 RSI: dffffc0000000000 RDI: ffff88841c2a7b18
Nov 11 21:26:43 10.243.1.49 [ 3973.443690] RBP: ffff88815ef47ae0 R08: ffffed1083854f64 R09: ffffed1083854f64
Nov 11 21:26:43 10.243.1.49 [ 3973.443698] R10: ffff88841c2a7b1b R11: ffffed1083854f63 R12: dead000000000100
Nov 11 21:26:43 10.243.1.49 [ 3973.443706] R13: dead000000000122 R14: ffff888151798000 R15: ffff888145380000
Nov 11 21:26:43 10.243.1.49 [ 3973.443714] FS:  00007f7312ffe700(0000) GS:ffff88841c280000(0000) knlGS:0000000000000000
Nov 11 21:26:43 10.243.1.49 [ 3973.443724] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Nov 11 21:26:43 10.243.1.49 [ 3973.443732] CR2: 0000000000000000 CR3: 000000016e080005 CR4: 00000000003706e0
Nov 11 21:26:43 10.243.1.49 [ 3973.443739] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Nov 11 21:26:43 10.243.1.49 [ 3973.443746] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Nov 11 21:26:44 10.243.1.49 [ 3973.443753] Call Trace:
Nov 11 21:26:44 10.243.1.49 [ 3973.443756]  <TASK>
Nov 11 21:26:44 10.243.1.49 [ 3973.443775]  __untrack_event.part.9+0x26/0xe0
Nov 11 21:26:44 10.243.1.49 [ 3973.443812]  monitor_oob_ioctl+0x1356/0x1490
Nov 11 21:26:44 10.243.1.49 [ 3973.443873]  ? enter_monitor+0x160/0x160
Nov 11 21:26:44 10.243.1.49 [ 3973.443894]  ? handle_irq_pipelined_finish+0xbe/0x240
Nov 11 21:26:44 10.243.1.49 [ 3973.443968]  ? do_raw_spin_lock+0x119/0x1e0
Nov 11 21:26:44 10.243.1.49 [ 3973.443987]  ? spin_dump+0x90/0x90
Nov 11 21:26:44 10.243.1.49 [ 3973.444024]  ? __kasan_check_read+0x11/0x20
Nov 11 21:26:44 10.243.1.49 [ 3973.444087]  EVL_ioctl+0x81/0xf0
Nov 11 21:26:44 10.243.1.49 [ 3973.444123]  do_oob_syscall+0x699/0x6b0
Nov 11 21:26:44 10.243.1.49 [ 3973.444152]  ? prepare_for_signal+0x180/0x180
Nov 11 21:26:44 10.243.1.49 [ 3973.444217]  handle_oob_syscall+0x189/0x230
Nov 11 21:26:44 10.243.1.49 [ 3973.444246]  ? handle_pipelined_syscall+0x820/0x820
Nov 11 21:26:44 10.243.1.49 [ 3973.444283]  ? rcu_read_lock_sched_held+0x5b/0xd0
Nov 11 21:26:44 10.243.1.49 [ 3973.444305]  ? rcu_read_lock_any_held.part.15+0x20/0x20
Nov 11 21:26:44 10.243.1.49 [ 3973.444347]  pipeline_syscall+0xa7/0x1c0
Nov 11 21:26:44 10.243.1.49 [ 3973.444387]  syscall_enter_from_user_mode+0x47/0x110
Nov 11 21:26:44 10.243.1.49 [ 3973.444414]  do_syscall_64+0x15/0xb0
Nov 11 21:26:44 10.243.1.49 [ 3973.444434]  entry_SYSCALL_64_after_hwframe+0x61/0xcb
Nov 11 21:26:44 10.243.1.49 [ 3973.444448] RIP: 0033:0x7f735de5a940
Nov 11 21:26:44 10.243.1.49 [ 3973.444460] Code: 77 ff ff bf 02 00 00 10 8b 44 24 0c 48 98 48 89 c6 48 8b 04 24 48 89 c2 4c 8b 54 24 38 41 b8 00 00 00 00 b8 9d 00 00 00 0f 05 <48> 89 44 24 30 48 8b 44 24 30 89 44 24 2c 8b 44 24 28 be 00 00 00
Nov 11 21:26:44 10.243.1.49 [ 3973.444469] RSP: 002b:00007f7312fee340 EFLAGS: 00000202 ORIG_RAX: 000000000000009d
Nov 11 21:26:44 10.243.1.49 [ 3973.444480] RAX: ffffffffffffffda RBX: 0000000000000f85 RCX: 00007f735de5a940
Nov 11 21:26:44 10.243.1.49 [ 3973.444488] RDX: 00000000c0186d03 RSI: 000000000000000d RDI: 0000000010000002
Nov 11 21:26:44 10.243.1.49 [ 3973.444495] RBP: 00007f7312fee520 R08: 0000000000000000 R09: 00007ffc09389080
Nov 11 21:26:44 10.243.1.49 [ 3973.444502] R10: 00007f7312fee4f0 R11: 0000000000000202 R12: 00007f7312fee600
Nov 11 21:26:44 10.243.1.49 [ 3973.444510] R13: 0000000000000000 R14: 0000000025bbabb3 R15: 00007f7312fee580
Nov 11 21:26:44 10.243.1.49 [ 3973.444591]  </TASK>
Nov 11 21:26:44 10.243.1.49 [ 3973.444595] irq event stamp: 3945
Nov 11 21:26:44 10.243.1.49 [ 3973.444598] hardirqs last  enabled at (3945): [<ffffffffa51330bc>] _raw_spin_unlock_irqrestore+0x4c/0x50
Nov 11 21:26:44 10.243.1.49 [ 3973.444613] hardirqs last disabled at (3944): [<ffffffffa5132e2b>] _raw_spin_lock_irqsave+0x1b/0x60
Nov 11 21:26:44 10.243.1.49 [ 3973.444627] softirqs last  enabled at (3156): [<ffffffffa5400403>] __do_softirq+0x403/0x57b
Nov 11 21:26:44 10.243.1.49 [ 3973.444641] softirqs last disabled at (3145): [<ffffffffa40d261d>] irq_exit_rcu+0x7d/0x150
Nov 11 21:26:44 10.243.1.49 [ 3973.444658] ---[ end trace 87ffbd6a737b5b8d ]---
Nov 11 21:26:45 10.243.1.49 [ 3975.416141] EVL: RxWorker1 resuming out-of-band [pid=16135, excpt=6, __list_del_entry_valid+0xc7/0x120]
Nov 11 21:26:45 10.243.1.49 [ 3975.453142] ------------[ cut here ]------------
Nov 11 21:26:45 10.243.1.49 [ 3975.471367] syscall 157 left IRQs disabled
Nov 11 21:26:45 10.243.1.49 [ 3975.471418] WARNING: CPU: 9 PID: 16135 at kernel/entry/common.c:353 syscall_exit_to_user_mode_prepare+0x37/0x270
Nov 11 21:26:45 10.243.1.49 [ 3975.523213] Modules linked in: xt_CHECKSUM xt_MASQUERADE tun netconsole ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 xt_conntrack ebtable_nat ip6table_nat ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_nat iptable_mangle iptable_security iptable_raw nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 ip_set nfnetlink ebtable_filter ebtables rfkill ip6table_filter ip6_tables iptable_filter sunrpc dm_mirror dm_region_hash dm_log dm_mod iTCO_wdt gpio_ich iTCO_vendor_support mxm_wmi x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm irqbypass crct10dif_pclmul crc32_pclmul ghash_clmulni_intel aesni_intel crypto_simd cryptd rapl mei_me i2c_i801 lpc_ich intel_cstate pcspkr intel_pch_thermal input_leds sg joydev ftdi_sio i2c_smbus mfd_core mei acpi_ipmi ioatdma ipmi_si ipmi_devintf ipmi_msghandler wmi ip_tables xfs libcrc32c ast drm_vram_helper sd_mod drm_kms_helper sr_mod cdrom t10_pi syscopyarea sysfillrect sysimgblt fb_sys_fops drm_ttm_helper ttm drm
Nov 11 21:26:45 10.243.1.49
Nov 11 21:26:45 10.243.1.49 [ 3975.523761]  ixgbe igb ahci libahci crc32c_intel libata mdio ptp i2c_algo_bit pps_core dca
Nov 11 21:26:45 10.243.1.49 [ 3975.857517] CPU: 9 PID: 16135 Comm: RxWorker1 Tainted: G        W         5.15.77evl-gae6080e09d9a #1
Nov 11 21:26:45 10.243.1.49 [ 3975.891006] Hardware name: Supermicro Super Server/X10SDV-6C-TLN4F, BIOS 1.3 02/13/2018
Nov 11 21:26:45 10.243.1.49 [ 3975.920914] IRQ stage: Linux
Nov 11 21:26:45 10.243.1.49 [ 3975.935389] RIP: 0010:syscall_exit_to_user_mode_prepare+0x37/0x270
Nov 11 21:26:45 10.243.1.49 [ 3975.961180] Code: 41 55 41 54 49 89 fc 53 4c 8b 68 08 48 8b 5f 78 eb 36 e8 0c fe f2 00 85 c0 74 1c 48 63 f3 48 c7 c7 70 66 ca a5 e8 42 f0 ec 00 <0f> 0b fb e8 e1 a1 0d 00 e8 bc af fd ff 41 f6 c5 76 75 3a 5b 41 5c
Nov 11 21:26:45 10.243.1.49 [ 3976.029965] RSP: 0018:ffff88815ef47f00 EFLAGS: 00010282
Nov 11 21:26:45 10.243.1.49 [ 3976.051995] RAX: 0000000000000000 RBX: 000000000000009d RCX: 0000000000000027
Nov 11 21:26:45 10.243.1.49 [ 3976.079868] RDX: 0000000000000027 RSI: dffffc0000000000 RDI: ffff88841c2a7b18
Nov 11 21:26:45 10.243.1.49 [ 3976.107756] RBP: ffff88815ef47f18 R08: ffffed1083854f64 R09: ffffed1083854f64
Nov 11 21:26:45 10.243.1.49 [ 3976.135689] R10: ffff88841c2a7b1b R11: ffffed1083854f63 R12: ffff88815ef47f58
Nov 11 21:26:45 10.243.1.49 [ 3976.163639] R13: 0000000000000010 R14: 0000000000000000 R15: 0000000000000000
Nov 11 21:26:45 10.243.1.49 [ 3976.191612] FS:  00007f7312ffe700(0000) GS:ffff88841c280000(0000) knlGS:0000000000000000
Nov 11 21:26:45 10.243.1.49 [ 3976.222528] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Nov 11 21:26:45 10.243.1.49 [ 3976.246420] CR2: 0000000000000000 CR3: 000000016e080005 CR4: 00000000003706e0
Nov 11 21:26:45 10.243.1.49 [ 3976.274590] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Nov 11 21:26:45 10.243.1.49 [ 3976.302728] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Nov 11 21:26:45 10.243.1.49 [ 3976.330855] Call Trace:
Nov 11 21:26:45 10.243.1.49 [ 3976.344849]  <TASK>
Nov 11 21:26:45 10.243.1.49 [ 3976.358680]  syscall_exit_to_user_mode+0xd/0x90
Nov 11 21:26:45 10.243.1.49 [ 3976.378802]  do_syscall_64+0x50/0xb0
Nov 11 21:26:45 10.243.1.49 [ 3976.395857]  entry_SYSCALL_64_after_hwframe+0x61/0xcb
Nov 11 21:26:45 10.243.1.49 [ 3976.417301] RIP: 0033:0x7f735de5a940
Nov 11 21:26:46 10.243.1.49 [ 3976.434233] Code: 77 ff ff bf 02 00 00 10 8b 44 24 0c 48 98 48 89 c6 48 8b 04 24 48 89 c2 4c 8b 54 24 38 41 b8 00 00 00 00 b8 9d 00 00 00 0f 05 <48> 89 44 24 30 48 8b 44 24 30 89 44 24 2c 8b 44 24 28 be 00 00 00
Nov 11 21:26:46 10.243.1.49 [ 3976.503605] RSP: 002b:00007f7312fee340 EFLAGS: 00000202 ORIG_RAX: 000000000000009d
Nov 11 21:26:46 10.243.1.49 [ 3976.533012] RAX: 0000000000000000 RBX: 0000000000000f85 RCX: 00007f735de5a940
Nov 11 21:26:46 10.243.1.49 [ 3976.562080] RDX: 00000000c0186d03 RSI: 000000000000000d RDI: 0000000010000002
Nov 11 21:26:46 10.243.1.49 [ 3976.590164] RBP: 00007f7312fee520 R08: 0000000000000000 R09: 00007ffc09389080
Nov 11 21:26:46 10.243.1.49 [ 3976.618274] R10: 00007f7312fee4f0 R11: 0000000000000202 R12: 00007f7312fee600
Nov 11 21:26:46 10.243.1.49 [ 3976.646281] R13: 0000000000000000 R14: 0000000025bbabb3 R15: 00007f7312fee580
Nov 11 21:26:46 10.243.1.49 [ 3976.674323]  </TASK>
Nov 11 21:26:46 10.243.1.49 [ 3976.687394] irq event stamp: 4671
Nov 11 21:26:46 10.243.1.49 [ 3976.703858] hardirqs last  enabled at (4679): [<ffffffffa41aaac2>] __up_console_sem+0x62/0x70
Nov 11 21:26:46 10.243.1.49 [ 3976.736098] hardirqs last disabled at (4686): [<ffffffffa41aaaa7>] __up_console_sem+0x47/0x70
Nov 11 21:26:46 10.243.1.49 [ 3976.768189] softirqs last  enabled at (3960): [<ffffffffa5400403>] __do_softirq+0x403/0x57b
Nov 11 21:26:46 10.243.1.49 [ 3976.799696] softirqs last disabled at (3949): [<ffffffffa40d261d>] irq_exit_rcu+0x7d/0x150
Nov 11 21:26:46 10.243.1.49 [ 3976.830927] ---[ end trace 87ffbd6a737b5b8e ]---
Nov 11 21:26:47 10.243.1.49 [ 3978.390616] ------------[ cut here ]------------
Nov 11 21:26:48 10.243.1.49 [ 3978.390624] list_del corruption, ffff888151798240->next is LIST_POISON1 (dead000000000100)
Nov 11 21:26:48 10.243.1.49 [ 3978.390634] EVL: RxWorker1 switching in-band [pid=16135, excpt=6, __list_del_entry_valid+0xc5/0x120]
Nov 11 21:26:48 10.243.1.49 [ 3978.493295] WARNING: CPU: 9 PID: 16135 at lib/list_debug.c:55 __list_del_entry_valid+0xc5/0x120
Nov 11 21:26:48 10.243.1.49 [ 3978.493321] Modules linked in: xt_CHECKSUM xt_MASQUERADE tun netconsole ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 xt_conntrack ebtable_nat ip6table_nat ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_nat iptable_mangle iptable_security iptable_raw nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 ip_set nfnetlink ebtable_filter ebtables rfkill ip6table_filter ip6_tables iptable_filter sunrpc dm_mirror dm_region_hash dm_log dm_mod iTCO_wdt gpio_ich iTCO_vendor_support mxm_wmi x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm irqbypass crct10dif_pclmul crc32_pclmul ghash_clmulni_intel aesni_intel crypto_simd cryptd rapl mei_me i2c_i801 lpc_ich intel_cstate pcspkr intel_pch_thermal input_leds sg joydev ftdi_sio i2c_smbus mfd_core mei acpi_ipmi ioatdma ipmi_si ipmi_devintf ipmi_msghandler wmi ip_tables xfs libcrc32c ast drm_vram_helper sd_mod drm_kms_helper sr_mod cdrom t10_pi syscopyarea sysfillrect sysimgblt fb_sys_fops drm_ttm_helper ttm drm
Nov 11 21:26:48 10.243.1.49
Nov 11 21:26:48 10.243.1.49 [ 3978.493644]  ixgbe igb ahci libahci crc32c_intel libata mdio ptp i2c_algo_bit pps_core dca
Nov 11 21:26:48 10.243.1.49 [ 3978.493688] CPU: 9 PID: 16135 Comm: RxWorker1 Tainted: G        W         5.15.77evl-gae6080e09d9a #1
Nov 11 21:26:48 10.243.1.49 [ 3978.493701] Hardware name: Supermicro Super Server/X10SDV-6C-TLN4F, BIOS 1.3 02/13/2018
Nov 11 21:26:48 10.243.1.49 [ 3978.493707] IRQ stage: Linux
Nov 11 21:26:48 10.243.1.49 [ 3978.493711] RIP: 0010:__list_del_entry_valid+0xc5/0x120
Nov 11 21:26:48 10.243.1.49 [ 3978.493726] Code: c0 eb e0 48 89 de 48 c7 c7 00 0c 82 a5 e8 7c fa 8b 00 0f 0b 31 c0 eb cb 4c 89 e2 48 89 de 48 c7 c7 60 0c 82 a5 e8 64 fa 8b 00 <0f> 0b 31 c0 eb b3 4c 89 ea 48 89 de 48 c7 c7 c0 0c 82 a5 e8 4c fa
Nov 11 21:26:48 10.243.1.49 [ 3978.493736] RSP: 0018:ffff88815ef47ac8 EFLAGS: 00010082
Nov 11 21:26:48 10.243.1.49 [ 3978.493745] RAX: 0000000000000000 RBX: ffff888151798240 RCX: 0000000000000027
Nov 11 21:26:48 10.243.1.49 [ 3978.493753] RDX: 0000000000000027 RSI: dffffc0000000000 RDI: ffff88841c2a7b18
Nov 11 21:26:48 10.243.1.49 [ 3978.493761] RBP: ffff88815ef47ae0 R08: ffffed1083854f64 R09: ffffed1083854f64
Nov 11 21:26:48 10.243.1.49 [ 3978.493770] R10: ffff88841c2a7b1b R11: ffffed1083854f63 R12: dead000000000100
Nov 11 21:26:48 10.243.1.49 [ 3978.493778] R13: dead000000000122 R14: ffff888151798000 R15: ffff888145380000
Nov 11 21:26:48 10.243.1.49 [ 3978.493786] FS:  00007f7312ffe700(0000) GS:ffff88841c280000(0000) knlGS:0000000000000000
Nov 11 21:26:48 10.243.1.49 [ 3978.493796] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Nov 11 21:26:48 10.243.1.49 [ 3978.493804] CR2: 0000000000000000 CR3: 000000016e080005 CR4: 00000000003706e0
Nov 11 21:26:49 10.243.1.49 [ 3978.493811] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Nov 11 21:26:49 10.243.1.49 [ 3978.493818] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Nov 11 21:26:49 10.243.1.49 [ 3978.493825] Call Trace:
Nov 11 21:26:49 10.243.1.49 [ 3978.493828]  <TASK>
Nov 11 21:26:49 10.243.1.49 [ 3978.493847]  __untrack_event.part.9+0x26/0xe0
Nov 11 21:26:49 10.243.1.49 [ 3978.493884]  monitor_oob_ioctl+0x1356/0x1490
Nov 11 21:26:49 10.243.1.49 [ 3978.493945]  ? enter_monitor+0x160/0x160
Nov 11 21:26:49 10.243.1.49 [ 3978.493987]  ? rcu_nmi_exit+0x9e/0xc0
Nov 11 21:26:49 10.243.1.49 [ 3978.494010]  ? handle_irq_pipelined_finish+0xbe/0x240
Nov 11 21:26:49 10.243.1.49 [ 3978.494048]  ? do_raw_spin_lock+0x119/0x1e0
Nov 11 21:26:49 10.243.1.49 [ 3978.494066]  ? spin_dump+0x90/0x90
Nov 11 21:26:49 10.243.1.49 [ 3978.494086]  ? asm_sysvec_apic_timer_interrupt+0x1b/0x20
Nov 11 21:26:49 10.243.1.49 [ 3978.494115]  ? __kasan_check_read+0x11/0x20
Nov 11 21:26:49 10.243.1.49 [ 3978.494178]  EVL_ioctl+0x81/0xf0
Nov 11 21:26:49 10.243.1.49 [ 3978.494214]  do_oob_syscall+0x699/0x6b0
Nov 11 21:26:49 10.243.1.49 [ 3978.494243]  ? prepare_for_signal+0x180/0x180
Nov 11 21:26:49 10.243.1.49 [ 3978.494273]  ? do_oob_irq+0x283/0x7e0
Nov 11 21:26:49 10.243.1.49 [ 3978.494315]  handle_oob_syscall+0x189/0x230
Nov 11 21:26:49 10.243.1.49 [ 3978.494344]  ? handle_pipelined_syscall+0x820/0x820
Nov 11 21:26:49 10.243.1.49 [ 3978.494381]  ? rcu_read_lock_sched_held+0x5b/0xd0
Nov 11 21:26:49 10.243.1.49 [ 3978.494403]  ? rcu_read_lock_any_held.part.15+0x20/0x20
Nov 11 21:26:49 10.243.1.49 [ 3978.494446]  pipeline_syscall+0xa7/0x1c0
Nov 11 21:26:49 10.243.1.49 [ 3978.494486]  syscall_enter_from_user_mode+0x47/0x110
Nov 11 21:26:49 10.243.1.49 [ 3978.494512]  do_syscall_64+0x15/0xb0
Nov 11 21:26:49 10.243.1.49 [ 3978.494533]  entry_SYSCALL_64_after_hwframe+0x61/0xcb
Nov 11 21:26:49 10.243.1.49 [ 3978.494546] RIP: 0033:0x7f735de5a940
Nov 11 21:26:49 10.243.1.49 [ 3978.494558] Code: 77 ff ff bf 02 00 00 10 8b 44 24 0c 48 98 48 89 c6 48 8b 04 24 48 89 c2 4c 8b 54 24 38 41 b8 00 00 00 00 b8 9d 00 00 00 0f 05 <48> 89 44 24 30 48 8b 44 24 30 89 44 24 2c 8b 44 24 28 be 00 00 00
Nov 11 21:26:49 10.243.1.49 [ 3978.494567] RSP: 002b:00007f7312fee340 EFLAGS: 00000202 ORIG_RAX: 000000000000009d
Nov 11 21:26:49 10.243.1.49 [ 3978.494578] RAX: ffffffffffffffda RBX: 0000000000000f8a RCX: 00007f735de5a940
Nov 11 21:26:49 10.243.1.49 [ 3978.494586] RDX: 00000000c0186d03 RSI: 000000000000000d RDI: 0000000010000002
Nov 11 21:26:49 10.243.1.49 [ 3978.494593] RBP: 00007f7312fee520 R08: 0000000000000000 R09: 00007ffc09389080
Nov 11 21:26:49 10.243.1.49 [ 3978.494601] R10: 00007f7312fee4f0 R11: 0000000000000202 R12: 00007f7312fee600
Nov 11 21:26:49 10.243.1.49 [ 3978.494608] R13: 0000000000000000 R14: 0000000028b61d36 R15: 00007f7312fee580
Nov 11 21:26:49 10.243.1.49 [ 3978.494689]  </TASK>
Nov 11 21:26:49 10.243.1.49 [ 3978.494693] irq event stamp: 4767
Nov 11 21:26:49 10.243.1.49 [ 3978.494696] hardirqs last  enabled at (4767): [<ffffffffa51330bc>] _raw_spin_unlock_irqrestore+0x4c/0x50
Nov 11 21:26:49 10.243.1.49 [ 3978.494711] hardirqs last disabled at (4766): [<ffffffffa5132e2b>] _raw_spin_lock_irqsave+0x1b/0x60
Nov 11 21:26:50 10.243.1.49 [ 3978.494725] softirqs last  enabled at (3960): [<ffffffffa5400403>] __do_softirq+0x403/0x57b
Nov 11 21:26:50 10.243.1.49 [ 3978.494740] softirqs last disabled at (3949): [<ffffffffa40d261d>] irq_exit_rcu+0x7d/0x150
Nov 11 21:26:50 10.243.1.49 [ 3978.494756] ---[ end trace 87ffbd6a737b5b8f ]---
Nov 11 21:26:50 10.243.1.49 [ 3980.524993] EVL: RxWorker1 resuming out-of-band [pid=16135, excpt=6, __list_del_entry_valid+0xc7/0x120]
Nov 11 21:26:50 10.243.1.49 [ 3980.561838] ------------[ cut here ]------------
Nov 11 21:26:50 10.243.1.49 [ 3980.580024] syscall 157 left IRQs disabled
Nov 11 21:26:50 10.243.1.49 [ 3980.580075] WARNING: CPU: 9 PID: 16135 at kernel/entry/common.c:353 syscall_exit_to_user_mode_prepare+0x37/0x270
Nov 11 21:26:50 10.243.1.49 [ 3980.631745] Modules linked in: xt_CHECKSUM xt_MASQUERADE tun netconsole ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 xt_conntrack ebtable_nat ip6table_nat ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_nat iptable_mangle iptable_security iptable_raw nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 ip_set nfnetlink ebtable_filter ebtables rfkill ip6table_filter ip6_tables iptable_filter sunrpc dm_mirror dm_region_hash dm_log dm_mod iTCO_wdt gpio_ich iTCO_vendor_support mxm_wmi x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm irqbypass crct10dif_pclmul crc32_pclmul ghash_clmulni_intel aesni_intel crypto_simd cryptd rapl mei_me i2c_i801 lpc_ich intel_cstate pcspkr intel_pch_thermal input_leds sg joydev ftdi_sio i2c_smbus mfd_core mei acpi_ipmi ioatdma ipmi_si ipmi_devintf ipmi_msghandler wmi ip_tables xfs libcrc32c ast drm_vram_helper sd_mod drm_kms_helper sr_mod cdrom t10_pi syscopyarea sysfillrect sysimgblt fb_sys_fops drm_ttm_helper ttm drm
Nov 11 21:26:50 10.243.1.49
Nov 11 21:26:50 10.243.1.49 [ 3980.632245]  ixgbe igb ahci libahci crc32c_intel libata mdio ptp i2c_algo_bit pps_core dca
Nov 11 21:26:50 10.243.1.49 [ 3980.965791] CPU: 9 PID: 16135 Comm: RxWorker1 Tainted: G        W         5.15.77evl-gae6080e09d9a #1
Nov 11 21:26:50 10.243.1.49 [ 3980.999328] Hardware name: Supermicro Super Server/X10SDV-6C-TLN4F, BIOS 1.3 02/13/2018
Nov 11 21:26:50 10.243.1.49 [ 3981.029249] IRQ stage: Linux
Nov 11 21:26:50 10.243.1.49 [ 3981.043760] RIP: 0010:syscall_exit_to_user_mode_prepare+0x37/0x270
Nov 11 21:26:50 10.243.1.49 [ 3981.069293] Code: 41 55 41 54 49 89 fc 53 4c 8b 68 08 48 8b 5f 78 eb 36 e8 0c fe f2 00 85 c0 74 1c 48 63 f3 48 c7 c7 70 66 ca a5 e8 42 f0 ec 00 <0f> 0b fb e8 e1 a1 0d 00 e8 bc af fd ff 41 f6 c5 76 75 3a 5b 41 5c
Nov 11 21:26:50 10.243.1.49 [ 3981.138077] RSP: 0018:ffff88815ef47f00 EFLAGS: 00010282
Nov 11 21:26:50 10.243.1.49 [ 3981.160087] RAX: 0000000000000000 RBX: 000000000000009d RCX: 0000000000000027
Nov 11 21:26:50 10.243.1.49 [ 3981.187931] RDX: 0000000000000027 RSI: dffffc0000000000 RDI: ffff88841c2a7b18
Nov 11 21:26:50 10.243.1.49 [ 3981.215800] RBP: ffff88815ef47f18 R08: ffffed1083854f64 R09: ffffed1083854f64
Nov 11 21:26:50 10.243.1.49 [ 3981.243717] R10: ffff88841c2a7b1b R11: ffffed1083854f63 R12: ffff88815ef47f58
Nov 11 21:26:50 10.243.1.49 [ 3981.271663] R13: 0000000000000010 R14: 0000000000000000 R15: 0000000000000000
Nov 11 21:26:50 10.243.1.49 [ 3981.299646] FS:  00007f7312ffe700(0000) GS:ffff88841c280000(0000) knlGS:0000000000000000
Nov 11 21:26:50 10.243.1.49 [ 3981.330571] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Nov 11 21:26:50 10.243.1.49 [ 3981.354488] CR2: 0000000000000000 CR3: 000000016e080005 CR4: 00000000003706e0
Nov 11 21:26:50 10.243.1.49 [ 3981.382620] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Nov 11 21:26:50 10.243.1.49 [ 3981.410763] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Nov 11 21:26:51 10.243.1.49 [ 3981.438857] Call Trace:
Nov 11 21:26:51 10.243.1.49 [ 3981.452814]  <TASK>
Nov 11 21:26:51 10.243.1.49 [ 3981.466641]  syscall_exit_to_user_mode+0xd/0x90
Nov 11 21:26:51 10.243.1.49 [ 3981.486739]  do_syscall_64+0x50/0xb0
Nov 11 21:26:51 10.243.1.49 [ 3981.503788]  entry_SYSCALL_64_after_hwframe+0x61/0xcb
Nov 11 21:26:51 10.243.1.49 [ 3981.525232] RIP: 0033:0x7f735de5a940
Nov 11 21:26:51 10.243.1.49 [ 3981.542188] Code: 77 ff ff bf 02 00 00 10 8b 44 24 0c 48 98 48 89 c6 48 8b 04 24 48 89 c2 4c 8b 54 24 38 41 b8 00 00 00 00 b8 9d 00 00 00 0f 05 <48> 89 44 24 30 48 8b 44 24 30 89 44 24 2c 8b 44 24 28 be 00 00 00
Nov 11 21:26:51 10.243.1.49 [ 3981.611725] RSP: 002b:00007f7312fee340 EFLAGS: 00000202 ORIG_RAX: 000000000000009d
Nov 11 21:26:51 10.243.1.49 [ 3981.641121] RAX: 0000000000000000 RBX: 0000000000000f8a RCX: 00007f735de5a940
Nov 11 21:26:51 10.243.1.49 [ 3981.670236] RDX: 00000000c0186d03 RSI: 000000000000000d RDI: 0000000010000002
Nov 11 21:26:51 10.243.1.49 [ 3981.698325] RBP: 00007f7312fee520 R08: 0000000000000000 R09: 00007ffc09389080
Nov 11 21:26:51 10.243.1.49 [ 3981.726468] R10: 00007f7312fee4f0 R11: 0000000000000202 R12: 00007f7312fee600
Nov 11 21:26:51 10.243.1.49 [ 3981.754484] R13: 0000000000000000 R14: 0000000028b61d36 R15: 00007f7312fee580
Nov 11 21:26:51 10.243.1.49 [ 3981.782554]  </TASK>
Nov 11 21:26:51 10.243.1.49 [ 3981.795620] irq event stamp: 5475
Nov 11 21:26:51 10.243.1.49 [ 3981.812071] hardirqs last  enabled at (5483): [<ffffffffa41aaac2>] __up_console_sem+0x62/0x70
Nov 11 21:26:51 10.243.1.49 [ 3981.844286] hardirqs last disabled at (5490): [<ffffffffa41aaaa7>] __up_console_sem+0x47/0x70
Nov 11 21:26:51 10.243.1.49 [ 3981.876372] softirqs last  enabled at (4782): [<ffffffffa5400403>] __do_softirq+0x403/0x57b
Nov 11 21:26:51 10.243.1.49 [ 3981.907874] softirqs last disabled at (4771): [<ffffffffa40d261d>] irq_exit_rcu+0x7d/0x150
Nov 11 21:26:51 10.243.1.49 [ 3981.939107] ---[ end trace 87ffbd6a737b5b90 ]---

[-- Attachment #2: smime.p7s --]
[-- Type: application/pkcs7-signature, Size: 6759 bytes --]

Re: [External] - Re: EVL Memory

[Philippe Gerum]

Russell Johnson <russell.johnson@kratosdefense.com> writes:

> [[S/MIME Signed Part:Undecided]]
> Thanks for the quick responses to our questions, that is very helpful! After 
> more debugging we were able to find an issue on our end that got us farther 
> with the issues I mentioned at the beginning of my previous email. Now, it has 
> evolved into a something else it appears. I ran with the debug kernel and I am 
> getting a ton of kernel splat that seems concerning. The callstacks go all 
> throughout EVL, so I was wondering if you could make anything out of the 
> output.. I also see two "switched inband while holding mutex" notifications 
> from EVL along with a fault notification (grouped together) at regular 
> intervals while running. So I am sure the two are related.. Trying to get a 
> better understanding of which direction to start looking in. Any input helps - 
> thanks!
>

Clearly a bug in the core, which should be fixed in the next-evl-rebase
branch. Please confirm if that works as expected on your end.

Thanks,

-- 
Philippe.

RE: [External] - Re: EVL Memory

[Russell Johnson]

[-- Attachment #1.1: Type: text/plain, Size: 288 bytes --]

Hi Philippe,

I went ahead and tested out your changes on the "next" branch. Looks like it
runs better (so definitely progress), but it still hangs eventually. I have
attached the output from
the debug kernel (a couple runs because the output was different both
times).

Thanks,

Russell

[-- Attachment #1.2: stack_trace_1.txt --]
[-- Type: text/plain, Size: 29628 bytes --]

Nov 14 15:22:39 10.243.0.116 [  459.485242] rcu: INFO: rcu_sched detected stalls on CPUs/tasks:
Nov 14 15:22:39 10.243.0.116 [  459.485772] rcu: #0118-...0: (98 ticks this GP) idle=fe5/1/0x4000000000000000 softirq=22420/22426 fqs=13882
Nov 14 15:22:39 10.243.0.116 [  459.486409] #011(detected by 6, t=65003 jiffies, g=25653, q=1912)
Nov 14 15:22:39 10.243.0.116 [  459.486634] Sending NMI from CPU 6 to CPUs 8:
Nov 14 15:22:39 10.243.0.116 [  459.490943] NMI backtrace for cpu 8
Nov 14 15:22:39 10.243.0.116 [  459.490950] CPU: 8 PID: 1817 Comm: TxWorker0 Not tainted 5.15.77evl-g6437d24c6dbc #1
Nov 14 15:22:39 10.243.0.116 [  459.490964] Hardware name: Supermicro Super Server/X10SDV-6C+-TLN4F, BIOS 2.1 11/22/2019
Nov 14 15:22:39 10.243.0.116 [  459.490969] IRQ stage: EVL
Nov 14 15:22:39 10.243.0.116 [  459.490974] RIP: 0010:__asan_load4+0x0/0x90
Nov 14 15:22:39 10.243.0.116 [  459.490992] Code: c0 84 c0 75 e1 5d c3 cc cc cc cc 48 c1 e8 03 80 3c 10 00 75 e9 5d c3 cc cc cc cc 0f 1f 44 00 00 66 2e 0f 1f 84 00 00 00 00 00 <55> 48 89 e5 48 8b 4d 08 48 83 ff fb 77 53 48 b8 ff ff ff ff ff 7f
Nov 14 15:22:39 10.243.0.116 [  459.491002] RSP: 0018:ffff88811f7df820 EFLAGS: 00000002
Nov 14 15:22:39 10.243.0.116 [  459.491012] RAX: 0000000000140001 RBX: ffff888125a6a970 RCX: ffffffff9119be53
Nov 14 15:22:39 10.243.0.116 [  459.491021] RDX: 0000000000000003 RSI: dffffc0000000000 RDI: ffff888125a6a970
Nov 14 15:22:39 10.243.0.116 [  459.491028] RBP: ffff88811f7df908 R08: ffffed1024b4d52f R09: ffffed1024b4d52f
Nov 14 15:22:39 10.243.0.116 [  459.491037] R10: ffff888125a6a973 R11: ffffed1024b4d52e R12: 1ffff11023efbf08
Nov 14 15:22:39 10.243.0.116 [  459.491045] R13: ffff8883de66b040 R14: ffff88811f7df8e0 R15: 0000000000000000
Nov 14 15:22:39 10.243.0.116 [  459.491053] FS:  00007f39f3fff700(0000) GS:ffff8883de600000(0000) knlGS:0000000000000000
Nov 14 15:22:39 10.243.0.116 [  459.491063] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Nov 14 15:22:39 10.243.0.116 [  459.491071] CR2: 00007f1197a22000 CR3: 0000000111018003 CR4: 00000000003706e0
Nov 14 15:22:39 10.243.0.116 [  459.491079] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Nov 14 15:22:39 10.243.0.116 [  459.491085] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Nov 14 15:22:39 10.243.0.116 [  459.491092] Call Trace:
Nov 14 15:22:39 10.243.0.116 [  459.491096]  <TASK>
Nov 14 15:22:40 10.243.0.116 [  459.491100]  ? native_queued_spin_lock_slowpath+0x363/0x4a0
Nov 14 15:22:40 10.243.0.116 [  459.491121]  ? .slowpath+0x1d/0x1d
Nov 14 15:22:40 10.243.0.116 [  459.491137]  ? __kasan_check_write+0x14/0x20
Nov 14 15:22:40 10.243.0.116 [  459.491151]  ? set_mutex_owner+0x190/0x4a0
Nov 14 15:22:40 10.243.0.116 [  459.491177]  do_raw_spin_lock+0x1c6/0x1e0
Nov 14 15:22:40 10.243.0.116 [  459.491190]  ? spin_dump+0x90/0x90
Nov 14 15:22:40 10.243.0.116 [  459.491204]  ? __kasan_check_write+0x14/0x20
Nov 14 15:22:40 10.243.0.116 [  459.491217]  ? fast_grab_mutex+0x9c/0x160
Nov 14 15:22:40 10.243.0.116 [  459.491236]  evl_lock_mutex_timeout+0x25d/0xee0
Nov 14 15:22:40 10.243.0.116 [  459.491247]  ? _find_next_bit+0x4a/0x130
Nov 14 15:22:40 10.243.0.116 [  459.491277]  ? evl_trylock_mutex+0x1c0/0x1c0
Nov 14 15:22:40 10.243.0.116 [  459.491296]  ? __kasan_check_read+0x11/0x20
Nov 14 15:22:40 10.243.0.116 [  459.491315]  ? __kasan_check_read+0x11/0x20
Nov 14 15:22:40 10.243.0.116 [  459.491335]  enter_monitor+0x117/0x160
Nov 14 15:22:40 10.243.0.116 [  459.491357]  monitor_oob_ioctl+0x4e7/0x1380
Nov 14 15:22:40 10.243.0.116 [  459.491373]  ? __kasan_check_write+0x14/0x20
Nov 14 15:22:40 10.243.0.116 [  459.491396]  ? enter_monitor+0x160/0x160
Nov 14 15:22:40 10.243.0.116 [  459.491412]  ? handle_irq_pipelined_finish+0xbe/0x240
Nov 14 15:22:40 10.243.0.116 [  459.491444]  ? do_raw_spin_lock+0x119/0x1e0
Nov 14 15:22:40 10.243.0.116 [  459.491457]  ? spin_dump+0x90/0x90
Nov 14 15:22:40 10.243.0.116 [  459.491475]  ? __kasan_check_read+0x11/0x20
Nov 14 15:22:40 10.243.0.116 [  459.491488]  ? do_raw_spin_unlock+0x97/0x140
Nov 14 15:22:40 10.243.0.116 [  459.491512]  EVL_ioctl+0x81/0xf0
Nov 14 15:22:40 10.243.0.116 [  459.491531]  do_oob_syscall+0x699/0x6b0
Nov 14 15:22:40 10.243.0.116 [  459.491548]  ? prepare_for_signal+0x180/0x180
Nov 14 15:22:40 10.243.0.116 [  459.491565]  ? do_oob_irq+0x283/0x7e0
Nov 14 15:22:40 10.243.0.116 [  459.491586]  handle_oob_syscall+0x189/0x230
Nov 14 15:22:40 10.243.0.116 [  459.491603]  ? handle_pipelined_syscall+0x820/0x820
Nov 14 15:22:40 10.243.0.116 [  459.491623]  ? rcu_read_lock_sched_held+0x5b/0xd0
Nov 14 15:22:40 10.243.0.116 [  459.491641]  ? rcu_read_lock_any_held.part.15+0x20/0x20
Nov 14 15:22:40 10.243.0.116 [  459.491664]  pipeline_syscall+0xa7/0x1c0
Nov 14 15:22:40 10.243.0.116 [  459.491686]  syscall_enter_from_user_mode+0x47/0x110
Nov 14 15:22:40 10.243.0.116 [  459.491704]  do_syscall_64+0x15/0xb0
Nov 14 15:22:40 10.243.0.116 [  459.491720]  entry_SYSCALL_64_after_hwframe+0x61/0xcb
Nov 14 15:22:40 10.243.0.116 [  459.491733] RIP: 0033:0x7f3a156e0b4b
Nov 14 15:22:40 10.243.0.116 [  459.491742] Code: Unable to access opcode bytes at RIP 0x7f3a156e0b21.
Nov 14 15:22:40 10.243.0.116 [  459.491747] RSP: 002b:00007f39f3fef010 EFLAGS: 00000246 ORIG_RAX: 000000000000009d
Nov 14 15:22:40 10.243.0.116 [  459.491758] RAX: ffffffffffffffda RBX: 00007f39f3fef0a0 RCX: 00007f3a156e0b4b
Nov 14 15:22:40 10.243.0.116 [  459.491766] RDX: 0000000040106d00 RSI: 0000000000000009 RDI: 0000000010000002
Nov 14 15:22:40 10.243.0.116 [  459.491774] RBP: 0000000040106d00 R08: 0000000000000000 R09: 0000000083a21068
Nov 14 15:22:40 10.243.0.116 [  459.491781] R10: 00007f39f3fef0a0 R11: 0000000000000246 R12: 0000000000000009
Nov 14 15:22:40 10.243.0.116 [  459.491788] R13: 0000000001001000 R14: 0000000000000000 R15: 00007f39f3fff700
Nov 14 15:22:40 10.243.0.116 [  459.491818]  </TASK>
Nov 14 15:23:04 10.243.0.116 [  483.963153] watchdog: BUG: soft lockup - CPU#6 stuck for 41s! [kworker/6:1:144]
Nov 14 15:23:04 10.243.0.116 [  483.966775] Modules linked in: xt_CHECKSUM xt_MASQUERADE tun netconsole ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 xt_conntrack ebtable_nat ebtable_broute ip6table_nat ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_nat iptable_mangle iptable_security iptable_raw nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 libcrc32c ip_set nfnetlink ebtable_filter ebtables rfkill ip6table_filter ip6_tables iptable_filter sunrpc dm_mirror dm_region_hash dm_log dm_mod x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm gpio_ich iTCO_wdt iTCO_vendor_support mxm_wmi irqbypass crct10dif_pclmul crc32_pclmul ghash_clmulni_intel aesni_intel crypto_simd cryptd rapl intel_cstate pcspkr i2c_i801 intel_pch_thermal input_leds i2c_smbus joydev sg ftdi_sio lpc_ich mei_me mfd_core mei ioatdma wmi ip_tables ext4 mbcache jbd2 ast drm_vram_helper drm_kms_helper sd_mod syscopyarea sysfillrect sysimgblt sr_mod fb_sys_fops t10_pi drm_ttm_helper cdrom ttm drm ixgbe igb ahci
Nov 14 15:23:04 10.243.0.116 [  483.967267]  libahci libata crc32c_intel mdio ptp pps_core i2c_algo_bit dca fuse
Nov 14 15:23:04 10.243.0.116 [  484.006663] irq event stamp: 263866
Nov 14 15:23:04 10.243.0.116 [  484.012429] hardirqs last  enabled at (263865): [<ffffffff92117f9e>] arch_pipeline_entry+0x5e/0x110
Nov 14 15:23:04 10.243.0.116 [  484.017600] hardirqs last disabled at (263866): [<ffffffff911c346b>] sync_current_irq_stage+0x4db/0x500
Nov 14 15:23:04 10.243.0.116 [  484.022771] softirqs last  enabled at (28414): [<ffffffff92400403>] __do_softirq+0x403/0x57b
Nov 14 15:23:04 10.243.0.116 [  484.027974] softirqs last disabled at (28379): [<ffffffff910d261d>] irq_exit_rcu+0x7d/0x150
Nov 14 15:23:04 10.243.0.116 [  484.033196] CPU: 6 PID: 144 Comm: kworker/6:1 Not tainted 5.15.77evl-g6437d24c6dbc #1
Nov 14 15:23:04 10.243.0.116 [  484.038521] Hardware name: Supermicro Super Server/X10SDV-6C+-TLN4F, BIOS 2.1 11/22/2019
Nov 14 15:23:04 10.243.0.116 [  484.043973] IRQ stage: Linux
Nov 14 15:23:04 10.243.0.116 [  484.049334] Workqueue: events netstamp_clear
Nov 14 15:23:04 10.243.0.116 [  484.054767] RIP: 0010:__asan_load4+0x0/0x90
Nov 14 15:23:04 10.243.0.116 [  484.060203] Code: c0 84 c0 75 e1 5d c3 cc cc cc cc 48 c1 e8 03 80 3c 10 00 75 e9 5d c3 cc cc cc cc 0f 1f 44 00 00 66 2e 0f 1f 84 00 00 00 00 00 <55> 48 89 e5 48 8b 4d 08 48 83 ff fb 77 53 48 b8 ff ff ff ff ff 7f
Nov 14 15:23:04 10.243.0.116 [  484.071793] RSP: 0018:ffff88810233fb00 EFLAGS: 00000202 ORIG_RAX: 0000000000000000
Nov 14 15:23:04 10.243.0.116 [  484.077714] RAX: 0000000000000000 RBX: 0000000000000000 RCX: 0000000000000000
Nov 14 15:23:04 10.243.0.116 [  484.083650] RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000000
Nov 14 15:23:04 10.243.0.116 [  484.089574] RBP: ffff88810233fb80 R08: 0000000000000000 R09: 0000000000000000
Nov 14 15:23:04 10.243.0.116 [  484.095474] R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000000
Nov 14 15:23:04 10.243.0.116 [  484.101366] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000
Nov 14 15:23:04 10.243.0.116 [  484.107243] FS:  0000000000000000(0000) GS:ffff8883de500000(0000) knlGS:0000000000000000
Nov 14 15:23:04 10.243.0.116 [  484.113203] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Nov 14 15:23:04 10.243.0.116 [  484.119151] CR2: 00007f35560cf830 CR3: 00000002bee16004 CR4: 00000000003706e0
Nov 14 15:23:04 10.243.0.116 [  484.125195] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Nov 14 15:23:04 10.243.0.116 [  484.131250] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Nov 14 15:23:04 10.243.0.116 [  484.137283] Call Trace:
Nov 14 15:23:04 10.243.0.116 [  484.143250]  <TASK>
Nov 14 15:23:04 10.243.0.116 [  484.149152]  ? smp_call_function_many_cond+0x1b0/0x4d0
Nov 14 15:23:04 10.243.0.116 [  484.155158]  ? text_poke_bp_batch+0x350/0x350
Nov 14 15:23:04 10.243.0.116 [  484.161161]  ? netif_receive_skb_list_internal+0x100/0x730
Nov 14 15:23:04 10.243.0.116 [  484.167196]  on_each_cpu_cond_mask+0x24/0x40
Nov 14 15:23:04 10.243.0.116 [  484.173187]  text_poke_bp_batch+0x146/0x350
Nov 14 15:23:04 10.243.0.116 [  484.179151]  ? __text_poke+0x620/0x620
Nov 14 15:23:04 10.243.0.116 [  484.185042]  ? netif_receive_skb_list_internal+0x100/0x730
Nov 14 15:23:04 10.243.0.116 [  484.190927]  ? netif_receive_skb_list_internal+0x10f/0x730
Nov 14 15:23:04 10.243.0.116 [  484.196733]  ? netif_receive_skb_list_internal+0x105/0x730
Nov 14 15:23:04 10.243.0.116 [  484.202525]  ? netif_receive_skb_list_internal+0x100/0x730
Nov 14 15:23:04 10.243.0.116 [  484.208274]  text_poke_finish+0x1a/0x30
Nov 14 15:23:04 10.243.0.116 [  484.213979]  arch_jump_label_transform_apply+0x17/0x30
Nov 14 15:23:04 10.243.0.116 [  484.219692]  __jump_label_update+0x105/0x140
Nov 14 15:23:04 10.243.0.116 [  484.225314]  jump_label_update+0x16b/0x1a0
Nov 14 15:23:04 10.243.0.116 [  484.230826]  static_key_enable_cpuslocked+0xc0/0x100
Nov 14 15:23:04 10.243.0.116 [  484.236186]  static_key_enable+0x15/0x30
Nov 14 15:23:04 10.243.0.116 [  484.241348]  netstamp_clear+0x4b/0x60
Nov 14 15:23:04 10.243.0.116 [  484.246365]  process_one_work+0x517/0xa10
Nov 14 15:23:04 10.243.0.116 [  484.251248]  ? pwq_dec_nr_in_flight+0x120/0x120
Nov 14 15:23:04 10.243.0.116 [  484.255999]  ? lockdep_hardirqs_off+0x8d/0xc0
Nov 14 15:23:04 10.243.0.116 [  484.260592]  worker_thread+0x52/0x580
Nov 14 15:23:04 10.243.0.116 [  484.265003]  ? process_one_work+0xa10/0xa10
Nov 14 15:23:04 10.243.0.116 [  484.269288]  kthread+0x1ee/0x220
Nov 14 15:23:04 10.243.0.116 [  484.273484]  ? set_kthread_struct+0x80/0x80
Nov 14 15:23:04 10.243.0.116 [  484.277620]  ret_from_fork+0x22/0x30
Nov 14 15:23:04 10.243.0.116 [  484.281740]  </TASK>
Nov 14 15:23:32 10.243.0.116 [  511.964554] watchdog: BUG: soft lockup - CPU#6 stuck for 67s! [kworker/6:1:144]
Nov 14 15:23:32 10.243.0.116 [  511.968772] Modules linked in: xt_CHECKSUM xt_MASQUERADE tun netconsole ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 xt_conntrack ebtable_nat ebtable_broute ip6table_nat ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_nat iptable_mangle iptable_security iptable_raw nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 libcrc32c ip_set nfnetlink ebtable_filter ebtables rfkill ip6table_filter ip6_tables iptable_filter sunrpc dm_mirror dm_region_hash dm_log dm_mod x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm gpio_ich iTCO_wdt iTCO_vendor_support mxm_wmi irqbypass crct10dif_pclmul crc32_pclmul ghash_clmulni_intel aesni_intel crypto_simd cryptd rapl intel_cstate pcspkr i2c_i801 intel_pch_thermal input_leds i2c_smbus joydev sg ftdi_sio lpc_ich mei_me mfd_core mei ioatdma wmi ip_tables ext4 mbcache jbd2 ast drm_vram_helper drm_kms_helper sd_mod syscopyarea sysfillrect sysimgblt sr_mod fb_sys_fops t10_pi drm_ttm_helper cdrom ttm drm ixgbe igb ahci
Nov 14 15:23:32 10.243.0.116 [  511.969293]  libahci libata crc32c_intel mdio ptp pps_core i2c_algo_bit dca fuse
Nov 14 15:23:32 10.243.0.116 [  512.011764] irq event stamp: 429970
Nov 14 15:23:32 10.243.0.116 [  512.016583] hardirqs last  enabled at (429969): [<ffffffff92117f9e>] arch_pipeline_entry+0x5e/0x110
Nov 14 15:23:32 10.243.0.116 [  512.021670] hardirqs last disabled at (429970): [<ffffffff911c346b>] sync_current_irq_stage+0x4db/0x500
Nov 14 15:23:32 10.243.0.116 [  512.026822] softirqs last  enabled at (28414): [<ffffffff92400403>] __do_softirq+0x403/0x57b
Nov 14 15:23:32 10.243.0.116 [  512.032059] softirqs last disabled at (28379): [<ffffffff910d261d>] irq_exit_rcu+0x7d/0x150
Nov 14 15:23:32 10.243.0.116 [  512.037247] CPU: 6 PID: 144 Comm: kworker/6:1 Tainted: G             L    5.15.77evl-g6437d24c6dbc #1
Nov 14 15:23:32 10.243.0.116 [  512.042513] Hardware name: Supermicro Super Server/X10SDV-6C+-TLN4F, BIOS 2.1 11/22/2019
Nov 14 15:23:32 10.243.0.116 [  512.047809] IRQ stage: Linux
Nov 14 15:23:32 10.243.0.116 [  512.052993] Workqueue: events netstamp_clear
Nov 14 15:23:32 10.243.0.116 [  512.058501] RIP: 0010:__asan_load4+0x37/0x90
Nov 14 15:23:32 10.243.0.116 [  512.063690] Code: 53 48 b8 ff ff ff ff ff 7f ff ff 48 39 c7 76 44 48 8d 47 03 48 89 c2 83 e2 07 48 83 fa 02 76 1c 48 be 00 00 00 00 00 fc ff df <48> c1 e8 03 0f b6 04 30 84 c0 75 3c 5d c3 cc cc cc cc 48 be 00 00
Nov 14 15:23:32 10.243.0.116 [  512.074803] RSP: 0018:ffff88810233faf8 EFLAGS: 00000202 ORIG_RAX: 0000000000000000
Nov 14 15:23:32 10.243.0.116 [  512.080492] RAX: 0000000000000000 RBX: 0000000000000000 RCX: 0000000000000000
Nov 14 15:23:32 10.243.0.116 [  512.086188] RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000000
Nov 14 15:23:32 10.243.0.116 [  512.091780] RBP: ffff88810233faf8 R08: 0000000000000000 R09: 0000000000000000
Nov 14 15:23:32 10.243.0.116 [  512.097397] R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000000
Nov 14 15:23:32 10.243.0.116 [  512.103002] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000
Nov 14 15:23:32 10.243.0.116 [  512.108511] FS:  0000000000000000(0000) GS:ffff8883de500000(0000) knlGS:0000000000000000
Nov 14 15:23:32 10.243.0.116 [  512.114164] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Nov 14 15:23:32 10.243.0.116 [  512.119846] CR2: 00007f35560cf830 CR3: 00000002bee16004 CR4: 00000000003706e0
Nov 14 15:23:32 10.243.0.116 [  512.125688] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Nov 14 15:23:32 10.243.0.116 [  512.131516] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Nov 14 15:23:32 10.243.0.116 [  512.137310] Call Trace:
Nov 14 15:23:32 10.243.0.116 [  512.144464]  <TASK>
Nov 14 15:23:32 10.243.0.116 [  512.150132]  smp_call_function_many_cond+0x1b0/0x4d0
Nov 14 15:23:32 10.243.0.116 [  512.155855]  ? text_poke_bp_batch+0x350/0x350
Nov 14 15:23:32 10.243.0.116 [  512.161614]  ? netif_receive_skb_list_internal+0x100/0x730
Nov 14 15:23:32 10.243.0.116 [  512.167449]  on_each_cpu_cond_mask+0x24/0x40
Nov 14 15:23:32 10.243.0.116 [  512.173244]  text_poke_bp_batch+0x146/0x350
Nov 14 15:23:32 10.243.0.116 [  512.179047]  ? __text_poke+0x620/0x620
Nov 14 15:23:32 10.243.0.116 [  512.184799]  ? netif_receive_skb_list_internal+0x100/0x730
Nov 14 15:23:32 10.243.0.116 [  512.190662]  ? netif_receive_skb_list_internal+0x10f/0x730
Nov 14 15:23:32 10.243.0.116 [  512.196457]  ? netif_receive_skb_list_internal+0x105/0x730
Nov 14 15:23:32 10.243.0.116 [  512.202211]  ? netif_receive_skb_list_internal+0x100/0x730
Nov 14 15:23:32 10.243.0.116 [  512.207942]  text_poke_finish+0x1a/0x30
Nov 14 15:23:32 10.243.0.116 [  512.213597]  arch_jump_label_transform_apply+0x17/0x30
Nov 14 15:23:32 10.243.0.116 [  512.219363]  __jump_label_update+0x105/0x140
Nov 14 15:23:32 10.243.0.116 [  512.225044]  jump_label_update+0x16b/0x1a0
Nov 14 15:23:32 10.243.0.116 [  512.230557]  static_key_enable_cpuslocked+0xc0/0x100
Nov 14 15:23:32 10.243.0.116 [  512.235980]  static_key_enable+0x15/0x30
Nov 14 15:23:32 10.243.0.116 [  512.241195]  netstamp_clear+0x4b/0x60
Nov 14 15:23:32 10.243.0.116 [  512.246241]  process_one_work+0x517/0xa10
Nov 14 15:23:32 10.243.0.116 [  512.251182]  ? pwq_dec_nr_in_flight+0x120/0x120
Nov 14 15:23:32 10.243.0.116 [  512.255963]  ? lockdep_hardirqs_off+0x8d/0xc0
Nov 14 15:23:32 10.243.0.116 [  512.260571]  worker_thread+0x52/0x580
Nov 14 15:23:32 10.243.0.116 [  512.265041]  ? process_one_work+0xa10/0xa10
Nov 14 15:23:32 10.243.0.116 [  512.269380]  kthread+0x1ee/0x220
Nov 14 15:23:32 10.243.0.116 [  512.273605]  ? set_kthread_struct+0x80/0x80
Nov 14 15:23:32 10.243.0.116 [  512.277772]  ret_from_fork+0x22/0x30
Nov 14 15:23:32 10.243.0.116 [  512.281945]  </TASK>
Nov 14 15:24:00 10.243.0.116 [  539.965956] watchdog: BUG: soft lockup - CPU#6 stuck for 93s! [kworker/6:1:144]
Nov 14 15:24:00 10.243.0.116 [  539.970187] Modules linked in: xt_CHECKSUM xt_MASQUERADE tun netconsole ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 xt_conntrack ebtable_nat ebtable_broute ip6table_nat ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_nat iptable_mangle iptable_security iptable_raw nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 libcrc32c ip_set nfnetlink ebtable_filter ebtables rfkill ip6table_filter ip6_tables iptable_filter sunrpc dm_mirror dm_region_hash dm_log dm_mod x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm gpio_ich iTCO_wdt iTCO_vendor_support mxm_wmi irqbypass crct10dif_pclmul crc32_pclmul ghash_clmulni_intel aesni_intel crypto_simd cryptd rapl intel_cstate pcspkr i2c_i801 intel_pch_thermal input_leds i2c_smbus joydev sg ftdi_sio lpc_ich mei_me mfd_core mei ioatdma wmi ip_tables ext4 mbcache jbd2 ast drm_vram_helper drm_kms_helper sd_mod syscopyarea sysfillrect sysimgblt sr_mod fb_sys_fops t10_pi drm_ttm_helper cdrom ttm drm ixgbe igb ahci
Nov 14 15:24:00 10.243.0.116 [  539.970707]  libahci libata crc32c_intel mdio ptp pps_core i2c_algo_bit dca fuse
Nov 14 15:24:00 10.243.0.116 [  540.012415] irq event stamp: 596080
Nov 14 15:24:00 10.243.0.116 [  540.017214] hardirqs last  enabled at (596079): [<ffffffff92117f9e>] arch_pipeline_entry+0x5e/0x110
Nov 14 15:24:00 10.243.0.116 [  540.022334] hardirqs last disabled at (596080): [<ffffffff911c346b>] sync_current_irq_stage+0x4db/0x500
Nov 14 15:24:00 10.243.0.116 [  540.027485] softirqs last  enabled at (28414): [<ffffffff92400403>] __do_softirq+0x403/0x57b
Nov 14 15:24:00 10.243.0.116 [  540.032694] softirqs last disabled at (28379): [<ffffffff910d261d>] irq_exit_rcu+0x7d/0x150
Nov 14 15:24:00 10.243.0.116 [  540.037936] CPU: 6 PID: 144 Comm: kworker/6:1 Tainted: G             L    5.15.77evl-g6437d24c6dbc #1
Nov 14 15:24:00 10.243.0.116 [  540.043203] Hardware name: Supermicro Super Server/X10SDV-6C+-TLN4F, BIOS 2.1 11/22/2019
Nov 14 15:24:00 10.243.0.116 [  540.048499] IRQ stage: Linux
Nov 14 15:24:00 10.243.0.116 [  540.053696] Workqueue: events netstamp_clear
Nov 14 15:24:00 10.243.0.116 [  540.058927] RIP: 0010:smp_call_function_many_cond+0x1a8/0x4d0
Nov 14 15:24:00 10.243.0.116 [  540.064174] Code: fd 4c 89 e7 e8 79 8c 2a 00 4a 8d 3c fd 80 28 d1 92 49 8b 1c 24 e8 68 8c 2a 00 4a 03 1c fd 80 28 d1 92 4c 8d 7b 08 eb 02 f3 90 <4c> 89 ff e8 20 8b 2a 00 8b 43 08 a8 01 75 ef eb 9f 48 c7 c7 c8 2a
Nov 14 15:24:00 10.243.0.116 [  540.075363] RSP: 0018:ffff88810233fb08 EFLAGS: 00000202 ORIG_RAX: 0000000000000000
Nov 14 15:24:00 10.243.0.116 [  540.081049] RAX: 0000000000000000 RBX: 0000000000000000 RCX: 0000000000000000
Nov 14 15:24:00 10.243.0.116 [  540.086749] RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000000
Nov 14 15:24:00 10.243.0.116 [  540.092403] RBP: ffff88810233fb80 R08: 0000000000000000 R09: 0000000000000000
Nov 14 15:24:00 10.243.0.116 [  540.098020] R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000000
Nov 14 15:24:00 10.243.0.116 [  540.103651] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000
Nov 14 15:24:00 10.243.0.116 [  540.109188] FS:  0000000000000000(0000) GS:ffff8883de500000(0000) knlGS:0000000000000000
Nov 14 15:24:00 10.243.0.116 [  540.114865] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Nov 14 15:24:00 10.243.0.116 [  540.120587] CR2: 00007f35560cf830 CR3: 00000002bee16004 CR4: 00000000003706e0
Nov 14 15:24:00 10.243.0.116 [  540.126464] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Nov 14 15:24:00 10.243.0.116 [  540.132313] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Nov 14 15:24:00 10.243.0.116 [  540.138099] Call Trace:
Nov 14 15:24:00 10.243.0.116 [  540.143805]  <TASK>
Nov 14 15:24:00 10.243.0.116 [  540.149490]  ? text_poke_bp_batch+0x350/0x350
Nov 14 15:24:00 10.243.0.116 [  540.155222]  ? netif_receive_skb_list_internal+0x100/0x730
Nov 14 15:24:00 10.243.0.116 [  540.161020]  on_each_cpu_cond_mask+0x24/0x40
Nov 14 15:24:00 10.243.0.116 [  540.166805]  text_poke_bp_batch+0x146/0x350
Nov 14 15:24:00 10.243.0.116 [  540.172636]  ? __text_poke+0x620/0x620
Nov 14 15:24:00 10.243.0.116 [  540.178411]  ? netif_receive_skb_list_internal+0x100/0x730
Nov 14 15:24:00 10.243.0.116 [  540.184224]  ? netif_receive_skb_list_internal+0x10f/0x730
Nov 14 15:24:00 10.243.0.116 [  540.190013]  ? netif_receive_skb_list_internal+0x105/0x730
Nov 14 15:24:00 10.243.0.116 [  540.195776]  ? netif_receive_skb_list_internal+0x100/0x730
Nov 14 15:24:00 10.243.0.116 [  540.201502]  text_poke_finish+0x1a/0x30
Nov 14 15:24:00 10.243.0.116 [  540.207155]  arch_jump_label_transform_apply+0x17/0x30
Nov 14 15:24:00 10.243.0.116 [  540.212890]  __jump_label_update+0x105/0x140
Nov 14 15:24:00 10.243.0.116 [  540.218648]  jump_label_update+0x16b/0x1a0
Nov 14 15:24:00 10.243.0.116 [  540.224316]  static_key_enable_cpuslocked+0xc0/0x100
Nov 14 15:24:00 10.243.0.116 [  540.229858]  static_key_enable+0x15/0x30
Nov 14 15:24:00 10.243.0.116 [  540.235203]  netstamp_clear+0x4b/0x60
Nov 14 15:24:00 10.243.0.116 [  540.240414]  process_one_work+0x517/0xa10
Nov 14 15:24:00 10.243.0.116 [  540.245493]  ? pwq_dec_nr_in_flight+0x120/0x120
Nov 14 15:24:00 10.243.0.116 [  540.250416]  ? lockdep_hardirqs_off+0x8d/0xc0
Nov 14 15:24:00 10.243.0.116 [  540.255187]  worker_thread+0x52/0x580
Nov 14 15:24:00 10.243.0.116 [  540.259809]  ? process_one_work+0xa10/0xa10
Nov 14 15:24:00 10.243.0.116 [  540.264234]  kthread+0x1ee/0x220
Nov 14 15:24:00 10.243.0.116 [  540.268564]  ? set_kthread_struct+0x80/0x80
Nov 14 15:24:00 10.243.0.116 [  540.272831]  ret_from_fork+0x22/0x30
Nov 14 15:24:00 10.243.0.116 [  540.277007]  </TASK>
Nov 14 15:24:28 10.243.0.116 [  567.967358] watchdog: BUG: soft lockup - CPU#6 stuck for 119s! [kworker/6:1:144]
Nov 14 15:24:28 10.243.0.116 [  567.971610] Modules linked in: xt_CHECKSUM xt_MASQUERADE tun netconsole ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 xt_conntrack ebtable_nat ebtable_broute ip6table_nat ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_nat iptable_mangle iptable_security iptable_raw nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 libcrc32c ip_set nfnetlink ebtable_filter ebtables rfkill ip6table_filter ip6_tables iptable_filter sunrpc dm_mirror dm_region_hash dm_log dm_mod x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm gpio_ich iTCO_wdt iTCO_vendor_support mxm_wmi irqbypass crct10dif_pclmul crc32_pclmul ghash_clmulni_intel aesni_intel crypto_simd cryptd rapl intel_cstate pcspkr i2c_i801 intel_pch_thermal input_leds i2c_smbus joydev sg ftdi_sio lpc_ich mei_me mfd_core mei ioatdma wmi ip_tables ext4 mbcache jbd2 ast drm_vram_helper drm_kms_helper sd_mod syscopyarea sysfillrect sysimgblt sr_mod fb_sys_fops t10_pi drm_ttm_helper cdrom ttm drm ixgbe igb ahci
Nov 14 15:24:28 10.243.0.116 [  567.972130]  libahci libata crc32c_intel mdio ptp pps_core i2c_algo_bit dca fuse
Nov 14 15:24:28 10.243.0.116 [  568.014341] irq event stamp: 762232
Nov 14 15:24:28 10.243.0.116 [  568.019106] hardirqs last  enabled at (762231): [<ffffffff92117f9e>] arch_pipeline_entry+0x5e/0x110
Nov 14 15:24:28 10.243.0.116 [  568.024163] hardirqs last disabled at (762232): [<ffffffff911c346b>] sync_current_irq_stage+0x4db/0x500
Nov 14 15:24:28 10.243.0.116 [  568.029268] softirqs last  enabled at (28414): [<ffffffff92400403>] __do_softirq+0x403/0x57b
Nov 14 15:24:28 10.243.0.116 [  568.034414] softirqs last disabled at (28379): [<ffffffff910d261d>] irq_exit_rcu+0x7d/0x150
Nov 14 15:24:28 10.243.0.116 [  568.039613] CPU: 6 PID: 144 Comm: kworker/6:1 Tainted: G             L    5.15.77evl-g6437d24c6dbc #1
Nov 14 15:24:28 10.243.0.116 [  568.044932] Hardware name: Supermicro Super Server/X10SDV-6C+-TLN4F, BIOS 2.1 11/22/2019
Nov 14 15:24:28 10.243.0.116 [  568.050231] IRQ stage: Linux
Nov 14 15:24:28 10.243.0.116 [  568.055409] Workqueue: events netstamp_clear
Nov 14 15:24:28 10.243.0.116 [  568.060592] RIP: 0010:__asan_load4+0x0/0x90
Nov 14 15:24:28 10.243.0.116 [  568.065844] Code: c0 84 c0 75 e1 5d c3 cc cc cc cc 48 c1 e8 03 80 3c 10 00 75 e9 5d c3 cc cc cc cc 0f 1f 44 00 00 66 2e 0f 1f 84 00 00 00 00 00 <55> 48 89 e5 48 8b 4d 08 48 83 ff fb 77 53 48 b8 ff ff ff ff ff 7f
Nov 14 15:24:28 10.243.0.116 [  568.076879] RSP: 0018:ffff88810233fb00 EFLAGS: 00000202 ORIG_RAX: 0000000000000000
Nov 14 15:24:28 10.243.0.116 [  568.082543] RAX: 0000000000000000 RBX: 0000000000000000 RCX: 0000000000000000
Nov 14 15:24:28 10.243.0.116 [  568.088226] RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000000
Nov 14 15:24:28 10.243.0.116 [  568.093856] RBP: ffff88810233fb80 R08: 0000000000000000 R09: 0000000000000000
Nov 14 15:24:28 10.243.0.116 [  568.099441] R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000000
Nov 14 15:24:28 10.243.0.116 [  568.105039] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000
Nov 14 15:24:28 10.243.0.116 [  568.110555] FS:  0000000000000000(0000) GS:ffff8883de500000(0000) knlGS:0000000000000000
Nov 14 15:24:28 10.243.0.116 [  568.116232] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Nov 14 15:24:28 10.243.0.116 [  568.121930] CR2: 00007f35560cf830 CR3: 00000002bee16004 CR4: 00000000003706e0
Nov 14 15:24:28 10.243.0.116 [  568.127768] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Nov 14 15:24:28 10.243.0.116 [  568.133584] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Nov 14 15:24:28 10.243.0.116 [  568.139391] Call Trace:
Nov 14 15:24:28 10.243.0.116 [  568.145079]  <TASK>
Nov 14 15:24:28 10.243.0.116 [  568.150725]  ? smp_call_function_many_cond+0x1b0/0x4d0
Nov 14 15:24:28 10.243.0.116 [  568.156467]  ? text_poke_bp_batch+0x350/0x350
Nov 14 15:24:28 10.243.0.116 [  568.162243]  ? netif_receive_skb_list_internal+0x100/0x730
Nov 14 15:24:28 10.243.0.116 [  568.168050]  on_each_cpu_cond_mask+0x24/0x40
Nov 14 15:24:28 10.243.0.116 [  568.173844]  text_poke_bp_batch+0x146/0x350
Nov 14 15:24:28 10.243.0.116 [  568.179642]  ? __text_poke+0x620/0x620
Nov 14 15:24:28 10.243.0.116 [  568.185433]  ? netif_receive_skb_list_internal+0x100/0x730
Nov 14 15:24:28 10.243.0.116 [  568.191280]  ? netif_receive_skb_list_internal+0x10f/0x730
Nov 14 15:24:28 10.243.0.116 [  568.197076]  ? netif_receive_skb_list_internal+0x105/0x730
Nov 14 15:24:28 10.243.0.116 [  568.202834]  ? netif_receive_skb_list_internal+0x100/0x730
Nov 14 15:24:28 10.243.0.116 [  568.208545]  text_poke_finish+0x1a/0x30
Nov 14 15:24:28 10.243.0.116 [  568.214238]  arch_jump_label_transform_apply+0x17/0x30
Nov 14 15:24:28 10.243.0.116 [  568.219993]  __jump_label_update+0x105/0x140
Nov 14 15:24:28 10.243.0.116 [  568.225650]  jump_label_update+0x16b/0x1a0
Nov 14 15:24:28 10.243.0.116 [  568.231193]  static_key_enable_cpuslocked+0xc0/0x100
Nov 14 15:24:28 10.243.0.116 [  568.236602]  static_key_enable+0x15/0x30
Nov 14 15:24:28 10.243.0.116 [  568.241834]  netstamp_clear+0x4b/0x60
Nov 14 15:24:28 10.243.0.116 [  568.246877]  process_one_work+0x517/0xa10
Nov 14 15:24:28 10.243.0.116 [  568.251805]  ? pwq_dec_nr_in_flight+0x120/0x120
Nov 14 15:24:28 10.243.0.116 [  568.256579]  ? lockdep_hardirqs_off+0x8d/0xc0
Nov 14 15:24:28 10.243.0.116 [  568.261226]  worker_thread+0x52/0x580
Nov 14 15:24:28 10.243.0.116 [  568.265742]  ? process_one_work+0xa10/0xa10
Nov 14 15:24:28 10.243.0.116 [  568.270071]  kthread+0x1ee/0x220
Nov 14 15:24:28 10.243.0.116 [  568.274299]  ? set_kthread_struct+0x80/0x80
Nov 14 15:24:28 10.243.0.116 [  568.278490]  ret_from_fork+0x22/0x30
Nov 14 15:24:28 10.243.0.116 [  568.282655]  </TASK>

[-- Attachment #1.3: stack_trace_2.txt --]
[-- Type: text/plain, Size: 72805 bytes --]

Nov 14 15:36:46 10.243.0.116 [  528.734461] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/0:0:7]
Nov 14 15:36:46 10.243.0.116 [  528.735538] Modules linked in: xt_CHECKSUM xt_MASQUERADE tun netconsole ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 xt_conntrack ebtable_nat ebtable_broute ip6table_nat ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_nat iptable_mangle iptable_security iptable_raw nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 libcrc32c ip_set nfnetlink ebtable_filter ebtables rfkill ip6table_filter ip6_tables iptable_filter sunrpc dm_mirror dm_region_hash dm_log dm_mod iTCO_wdt gpio_ich iTCO_vendor_support x86_pkg_temp_thermal intel_powerclamp mxm_wmi coretemp kvm_intel kvm irqbypass crct10dif_pclmul crc32_pclmul ghash_clmulni_intel aesni_intel crypto_simd cryptd rapl intel_cstate pcspkr joydev intel_pch_thermal input_leds sg mei_me ftdi_sio i2c_i801 lpc_ich mei mfd_core i2c_smbus ioatdma wmi ip_tables ext4 mbcache jbd2 ast drm_vram_helper drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops drm_ttm_helper sr_mod ttm sd_mod cdrom t10_pi igb drm ixgbe ahci
Nov 14 15:36:46 10.243.0.116 [  528.736553]  libahci libata crc32c_intel mdio ptp i2c_algo_bit pps_core dca fuse
Nov 14 15:36:46 10.243.0.116 [  528.805968] irq event stamp: 4404404
Nov 14 15:36:46 10.243.0.116 [  528.816003] hardirqs last  enabled at (4404403): [<ffffffffbc117f9e>] arch_pipeline_entry+0x5e/0x110
Nov 14 15:36:46 10.243.0.116 [  528.826492] hardirqs last disabled at (4404404): [<ffffffffbb1c346b>] sync_current_irq_stage+0x4db/0x500
Nov 14 15:36:46 10.243.0.116 [  528.836988] softirqs last  enabled at (4297982): [<ffffffffbc400403>] __do_softirq+0x403/0x57b
Nov 14 15:36:46 10.243.0.116 [  528.847475] softirqs last disabled at (4297951): [<ffffffffbb0d261d>] irq_exit_rcu+0x7d/0x150
Nov 14 15:36:46 10.243.0.116 [  528.857999] CPU: 0 PID: 7 Comm: kworker/0:0 Not tainted 5.15.77evl-g6437d24c6dbc #1
Nov 14 15:36:46 10.243.0.116 [  528.868497] Hardware name: Supermicro Super Server/X10SDV-6C+-TLN4F, BIOS 2.1 11/22/2019
Nov 14 15:36:46 10.243.0.116 [  528.879049] IRQ stage: Linux
Nov 14 15:36:46 10.243.0.116 [  528.889330] Workqueue: events netstamp_clear
Nov 14 15:36:46 10.243.0.116 [  528.899642] RIP: 0010:unlock_stage+0x52/0x80
Nov 14 15:36:46 10.243.0.116 [  528.909837] Code: e8 e3 8e 0a ff 48 f7 d0 48 89 c7 48 21 cf e8 d5 8e 0a ff 25 00 02 00 00 48 89 c3 48 85 d2 75 10 48 89 c2 48 85 d2 74 01 fb 5b <5d> c3 cc cc cc cc 65 48 8b 3c 25 80 6f 02 00 48 81 c7 d8 0d 00 00
Nov 14 15:36:46 10.243.0.116 [  528.931388] RSP: 0018:ffff8883de2092c8 EFLAGS: 00000206 ORIG_RAX: 0000000000000000
Nov 14 15:36:46 10.243.0.116 [  528.942274] RAX: 0000000000000000 RBX: 0000000000000000 RCX: 0000000000000000
Nov 14 15:36:46 10.243.0.116 [  528.953099] RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000000
Nov 14 15:36:46 10.243.0.116 [  528.963917] RBP: ffff8883de2095c8 R08: 0000000000000000 R09: 0000000000000000
Nov 14 15:36:46 10.243.0.116 [  528.974628] R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000000
Nov 14 15:36:46 10.243.0.116 [  528.985222] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000
Nov 14 15:36:46 10.243.0.116 [  528.995664] FS:  0000000000000000(0000) GS:ffff8883de200000(0000) knlGS:0000000000000000
Nov 14 15:36:46 10.243.0.116 [  529.006128] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Nov 14 15:36:46 10.243.0.116 [  529.016518] CR2: 00007f5155c96d60 CR3: 0000000122034003 CR4: 00000000003706f0
Nov 14 15:36:46 10.243.0.116 [  529.027064] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Nov 14 15:36:46 10.243.0.116 [  529.037517] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Nov 14 15:36:46 10.243.0.116 [  529.047802] Call Trace:
Nov 14 15:36:46 10.243.0.116 [  529.057827]  <IRQ>
Nov 14 15:36:46 10.243.0.116 [  529.067870]  ? __kasan_check_read+0x11/0x20
Nov 14 15:36:46 10.243.0.116 [  529.078161]  ? mark_lock.part.53+0x119/0x1460
Nov 14 15:36:46 10.243.0.116 [  529.088516]  ? create_prof_cpu_mask+0x30/0x30
Nov 14 15:36:46 10.243.0.116 [  529.098999]  ? arch_stack_walk+0x86/0xf0
Nov 14 15:36:46 10.243.0.116 [  529.109569]  ? print_usage_bug+0x60/0x60
Nov 14 15:36:46 10.243.0.116 [  529.120172]  ret_from_fork+0x22/0x30
Nov 14 15:36:46 10.243.0.116 [  529.130845] WARNING: kernel stack frame pointer at 000000009b1db727 in kworker/0:0:7 has bad value 00000000b3fc3469
Nov 14 15:36:46 10.243.0.116 [  529.130856] unwind stack type:2 next_sp:00000000f67bf0e4 mask:0x4 graph_idx:0
Nov 14 15:36:47 10.243.0.116 [  529.130863] 000000009b1db727: ffffffffbb1ae7b0 (_prb_commit+0x140/0x140)
Nov 14 15:36:47 10.243.0.116 [  529.130878] 00000000b43a18b9: 0000000041b58ab3 (0x41b58ab3)
Nov 14 15:36:47 10.243.0.116 [  529.130890] 000000004d9a86d2: 6234336131386239 (0x6234336131386239)
Nov 14 15:36:47 10.243.0.116 [  529.130902] 00000000d8117c58: ffffffffbb859c00 (put_dec+0xc0/0xc0)
Nov 14 15:36:47 10.243.0.116 [  529.130918] 00000000c14e1bb9: 9fbcf6e6657b4600 (0x9fbcf6e6657b4600)
Nov 14 15:36:47 10.243.0.116 [  529.130930] 000000004f9a0750: ffff103000001005 (0xffff103000001005)
Nov 14 15:36:47 10.243.0.116 [  529.130942] 000000007ec12b4f: ffff88811a250200 (0xffff88811a250200)
Nov 14 15:36:47 10.243.0.116 [  529.130954] 00000000ae08693a: ffffffffbe539583 (__log_buf+0x17f63/0x80020)
Nov 14 15:36:47 10.243.0.116 [  529.130968] 000000005538940a: 0000003000000000 (0x3000000000)
Nov 14 15:36:47 10.243.0.116 [  529.130980] 000000006d8ad422: 9fbcf6e6657b4600 (0x9fbcf6e6657b4600)
Nov 14 15:36:47 10.243.0.116 [  529.130992] 00000000f28336c7: ffffff00ffffffff (0xffffff00ffffffff)
Nov 14 15:36:47 10.243.0.116 [  529.131003] 00000000d311b6ae: 0000003000000000 (0x3000000000)
Nov 14 15:36:47 10.243.0.116 [  529.131014] 0000000083206f42: 1ffff1107bc412bd (0x1ffff1107bc412bd)
Nov 14 15:36:47 10.243.0.116 [  529.131026] 000000005a221ff0: ffffffffbe539920 (__log_buf+0x18300/0x80020)
Nov 14 15:36:47 10.243.0.116 [  529.131039] 00000000702c1abb: ffffffffbe5399a8 (__log_buf+0x18388/0x80020)
Nov 14 15:36:47 10.243.0.116 [  529.131052] 000000001d21a716: ffff8883de2097b8 (0xffff8883de2097b8)
Nov 14 15:36:47 10.243.0.116 [  529.131063] 00000000f69b2341: ffffffffbc653ce9 (__func__.55224+0x129/0xa40)
Nov 14 15:36:47 10.243.0.116 [  529.131080] 00000000816bb13c: ffff8883de2095f0 (0xffff8883de2095f0)
Nov 14 15:36:47 10.243.0.116 [  529.131092] 00000000b92630bf: ffffffffbb4cb4ff (__asan_loadN+0xf/0x20)
Nov 14 15:36:47 10.243.0.116 [  529.131106] 000000007ae54397: ffff8883de209710 (0xffff8883de209710)
Nov 14 15:36:47 10.243.0.116 [  529.131117] 00000000a636e04f: ffffffffbb859fcc (number+0x3cc/0x5e0)
Nov 14 15:36:47 10.243.0.116 [  529.131132] 0000000003f95819: ffffffffbe539ba1 (__log_buf+0x18581/0x80020)
Nov 14 15:36:47 10.243.0.116 [  529.131145] 000000002089889d: ffffffffbe539bc2 (__log_buf+0x185a2/0x80020)
Nov 14 15:36:47 10.243.0.116 [  529.131158] 000000009f7f45b6: 1ffff1107bc412c9 (0x1ffff1107bc412c9)
Nov 14 15:36:47 10.243.0.116 [  529.131170] 000000004a67d2b0: ffffffff001aec1a (0xffffffff001aec1a)
Nov 14 15:36:47 10.243.0.116 [  529.131181] 0000000032a1a535: 0000001010ffc5c8 (0x1010ffc5c8)
Nov 14 15:36:47 10.243.0.116 [  529.131193] 00000000609a6396: 0000001010ffc5c8 (0x1010ffc5c8)
Nov 14 15:36:47 10.243.0.116 [  529.131204] 000000005e203b6c: ffff888300000006 (0xffff888300000006)
Nov 14 15:36:47 10.243.0.116 [  529.131215] 000000006f765dae: 0000001030100000 (0x1030100000)
Nov 14 15:36:47 10.243.0.116 [  529.131227] 0000000085cd1c75: ffffffffbe539dc8 (__log_buf+0x187a8/0x80020)
Nov 14 15:36:47 10.243.0.116 [  529.131239] 0000000098eac725: ffff8883de209778 (0xffff8883de209778)
Nov 14 15:36:47 10.243.0.116 [  529.131251] 0000000010e1f5a2: ffff8883de2096e8 (0xffff8883de2096e8)
Nov 14 15:36:47 10.243.0.116 [  529.131263] 00000000b5ca23dd: ffff8883de209668 (0xffff8883de209668)
Nov 14 15:36:47 10.243.0.116 [  529.131275] 00000000713044db: ffffffffbb4cb4ff (__asan_loadN+0xf/0x20)
Nov 14 15:36:47 10.243.0.116 [  529.131288] 00000000f90beaad: ffff8883de209710 (0xffff8883de209710)
Nov 14 15:36:47 10.243.0.116 [  529.131300] 00000000ff98e030: ffffffffbb85a976 (format_decode+0xd6/0x5c0)
Nov 14 15:36:47 10.243.0.116 [  529.131315] 00000000d0e54bdf: ffffffffbb85fff8 (pointer+0x398/0x5d0)
Nov 14 15:36:47 10.243.0.116 [  529.131327] 00000000c2624c74: 1ffff1107bc412d1 (0x1ffff1107bc412d1)
Nov 14 15:36:47 10.243.0.116 [  529.131339] 00000000af7737fe: 0000000041b58ab3 (0x41b58ab3)
Nov 14 15:36:47 10.243.0.116 [  529.131350] 000000006eca9803: ffffffffbccd37b1 (.LC2+0x80a/0x25639)
Nov 14 15:36:47 10.243.0.116 [  529.131364] 00000000b125007a: ffffffffbb85a8a0 (enable_ptr_key_workfn+0x30/0x30)
Nov 14 15:36:47 10.243.0.116 [  529.131380] 00000000cebc573b: ffff8883de2096d8 (0xffff8883de2096d8)
Nov 14 15:36:47 10.243.0.116 [  529.131392] 00000000a5fc355e: ffffffffbc653cf0 (__func__.55224+0x130/0xa40)
Nov 14 15:36:47 10.243.0.116 [  529.131407] 000000008a64698f: 6666666666666666 (0x6666666666666666)
Nov 14 15:36:47 10.243.0.116 [  529.131419] 00000000ebabac1b: ffffffffbb1aed04 (space_used.isra.9+0x24/0x70)
Nov 14 15:36:47 10.243.0.116 [  529.131432] 00000000b220bd53: 0000000000000010 (0x10)
Nov 14 15:36:47 10.243.0.116 [  529.131442] 000000007e4e2a59: ffff8883de209778 (0xffff8883de209778)
Nov 14 15:36:47 10.243.0.116 [  529.131454] 0000000015d0035c: 0000000000000002 (0x2)
Nov 14 15:36:47 10.243.0.116 [  529.131464] 0000000063b6edda: ffffffffbc653cee (__func__.55224+0x12e/0xa40)
Nov 14 15:36:47 10.243.0.116 [  529.131479] 00000000da7968e1: ffffffffbe53a35e (__log_buf+0x18d3e/0x80020)
Nov 14 15:36:47 10.243.0.116 [  529.131492] 00000000c3628767: ffff8883de209710 (0xffff8883de209710)
Nov 14 15:36:47 10.243.0.116 [  529.131504] 000000006d58dd1e: ffffffffbb4cbccd (memcpy+0x4d/0x70)
Nov 14 15:36:47 10.243.0.116 [  529.131517] 000000003eea0d3f: ffffffffbe53a434 (__log_buf+0x18e14/0x80020)
Nov 14 15:36:47 10.243.0.116 [  529.131530] 0000000022f84f11: ffff8883de2097b8 (0xffff8883de2097b8)
Nov 14 15:36:47 10.243.0.116 [  529.131542] 00000000a5747a88: ffffffffbc653cf0 (__func__.55224+0x130/0xa40)
Nov 14 15:36:47 10.243.0.116 [  529.131557] 000000001f28fbc4: ffff8883de2097e0 (0xffff8883de2097e0)
Nov 14 15:36:47 10.243.0.116 [  529.131569] 0000000050be520f: ffffffffbb860889 (vsnprintf+0x659/0x8e0)
Nov 14 15:36:47 10.243.0.116 [  529.131581] 0000000067024a4c: 09ff8883de209970 (0x9ff8883de209970)
Nov 14 15:36:47 10.243.0.116 [  529.131593] 000000007e4d11f7: 1ffff1107bc412eb (0x1ffff1107bc412eb)
Nov 14 15:36:47 10.243.0.116 [  529.131605] 000000006eb87554: 0000000000000039 (0x39)
Nov 14 15:36:47 10.243.0.116 [  529.131615] 0000000022cdf9cb: ffffffffbe53a638 (__log_buf+0x19018/0x80020)
Nov 14 15:36:47 10.243.0.116 [  529.131628] 00000000d4591335: ffff8883de209ae8 (0xffff8883de209ae8)
Nov 14 15:36:47 10.243.0.116 [  529.131640] 00000000276e56c0: ffff8883de209778 (0xffff8883de209778)
Nov 14 15:36:47 10.243.0.116 [  529.131652] 000000003890a2c2: ffffffffbe53a739 (__log_buf+0x19119/0x80020)
Nov 14 15:36:47 10.243.0.116 [  529.131664] 000000006ebc4411: 0000000041b58ab3 (0x41b58ab3)
Nov 14 15:36:47 10.243.0.116 [  529.131675] 0000000030cb32c4: 80000000ffffc5f0 (0x80000000ffffc5f0)
Nov 14 15:36:47 10.243.0.116 [  529.131687] 00000000c2c357f9: ffff8883de209970 (0xffff8883de209970)
Nov 14 15:36:47 10.243.0.116 [  529.131699] 0000000029bfd7c2: 1ffff1107bc412f4 (0x1ffff1107bc412f4)
Nov 14 15:36:47 10.243.0.116 [  529.131711] 00000000f0bd3f33: ffff8883de209788 (0xffff8883de209788)
Nov 14 15:36:47 10.243.0.116 [  529.131722] 0000000085ee32de: ffffffffbb4cbb64 (__kasan_check_write+0x14/0x20)
Nov 14 15:36:47 10.243.0.116 [  529.131737] 0000000015a2ee2a: ffff8883de209828 (0xffff8883de209828)
Nov 14 15:36:47 10.243.0.116 [  529.131749] 00000000d7c82f7f: 0000000000000087 (0x87)
Nov 14 15:36:47 10.243.0.116 [  529.131759] 00000000eacad8a0: 0000000ede2099b0 (0xede2099b0)
Nov 14 15:36:47 10.243.0.116 [  529.131770] 0000000031e8cb99: 0000000041b58ab3 (0x41b58ab3)
Nov 14 15:36:47 10.243.0.116 [  529.131781] 0000000072321493: ffffffffbcca4e67 (.LC0+0xbef/0x2536)
Nov 14 15:36:47 10.243.0.116 [  529.131793] 00000000ea5e2f62: ffffffffbb1ae670 (desc_make_final+0x80/0x80)
Nov 14 15:36:47 10.243.0.116 [  529.131806] 000000001c55a81f: ffff8883de2097f8 (0xffff8883de2097f8)
Nov 14 15:36:47 10.243.0.116 [  529.131818] 0000000007d3232b: 00000000ffffc5fc (0xffffc5fc)
Nov 14 15:36:47 10.243.0.116 [  529.131829] 000000007005b795: ffff8883de2098b0 (0xffff8883de2098b0)
Nov 14 15:36:47 10.243.0.116 [  529.131840] 000000002d714708: ffffffffbe53ab50 (__log_buf+0x19530/0x80020)
Nov 14 15:36:47 10.243.0.116 [  529.131853] 00000000683d01be: 0000000000000000 ...
Nov 14 15:36:47 10.243.0.116 [  529.131858] 00000000cac33919: ffff8883de2098b0 (0xffff8883de2098b0)
Nov 14 15:36:47 10.243.0.116 [  529.131870] 000000003aac650c: 0000000000000038 (0x38)
Nov 14 15:36:47 10.243.0.116 [  529.131880] 00000000f7e63d6d: 0000000000000029 (0x29)
Nov 14 15:36:47 10.243.0.116 [  529.131890] 00000000b23eb722: 9fbcf6e6657b4600 (0x9fbcf6e6657b4600)
Nov 14 15:36:47 10.243.0.116 [  529.131902] 0000000030a37258: 0000000000000037 (0x37)
Nov 14 15:36:47 10.243.0.116 [  529.131912] 00000000230eda56: ffff8883de227b2d (0xffff8883de227b2d)
Nov 14 15:36:47 10.243.0.116 [  529.131924] 000000004e4ea23f: 0000000000000082 (0x82)
Nov 14 15:36:47 10.243.0.116 [  529.131934] 0000000054b24a9f: 0000000000000002 (0x2)
Nov 14 15:36:47 10.243.0.116 [  529.131944] 000000005d6c8274: ffffffffbcf3fa40 (_printk_rb_static_infos+0x212c0/0x160020)
Nov 14 15:36:47 10.243.0.116 [  529.131961] 00000000ced9b71a: ffff8883de209838 (0xffff8883de209838)
Nov 14 15:36:47 10.243.0.116 [  529.131973] 00000000af2d2482: ffffffffbb1b027e (prb_final_commit+0xe/0x20)
Nov 14 15:36:47 10.243.0.116 [  529.131985] 00000000c8804ec0: ffff8883de209a58 (0xffff8883de209a58)
Nov 14 15:36:47 10.243.0.116 [  529.131997] 000000006aa0f304: 0000000000000246 (0x246)
Nov 14 15:36:47 10.243.0.116 [  529.132008] 00000000db681876: ffff8883de209bf0 (0xffff8883de209bf0)
Nov 14 15:36:47 10.243.0.116 [  529.132019] 0000000042b2a7ab: 0000007b32b9b2e4 (0x7b32b9b2e4)
Nov 14 15:36:47 10.243.0.116 [  529.132031] 0000000011d648e0: ffff8883de209930 (0xffff8883de209930)
Nov 14 15:36:47 10.243.0.116 [  529.132043] 000000007f041534: 0000000080000000 (0x80000000)
Nov 14 15:36:47 10.243.0.116 [  529.132053] 000000001ad273ac: 0000000000000246 (0x246)
Nov 14 15:36:47 10.243.0.116 [  529.132064] 000000006f742d5d: 1ffff1107bc41312 (0x1ffff1107bc41312)
Nov 14 15:36:47 10.243.0.116 [  529.132076] 00000000362fb448: ffff8883de209970 (0xffff8883de209970)
Nov 14 15:36:47 10.243.0.116 [  529.132087] 000000004c798c91: 0000000000000000 ...
Nov 14 15:36:47 10.243.0.116 [  529.132092] 000000001378e64d: 0000000041b58ab3 (0x41b58ab3)
Nov 14 15:36:47 10.243.0.116 [  529.132103] 0000000072098fe2: ffffffffbcca4d68 (.LC0+0xaf0/0x2536)
Nov 14 15:36:47 10.243.0.116 [  529.132115] 00000000dc0ac9a1: ffffffffbb1ad060 (printk_parse_prefix+0x10/0x10)
Nov 14 15:36:47 10.243.0.116 [  529.132128] 000000001acd665a: 0000000041b58ab3 (0x41b58ab3)
Nov 14 15:36:47 10.243.0.116 [  529.132139] 00000000e59ca6a3: ffffffff00000002 (0xffffffff00000002)
Nov 14 15:36:47 10.243.0.116 [  529.132151] 000000000e2442fb: ffffffffbb1ae670 (desc_make_final+0x80/0x80)
Nov 14 15:36:47 10.243.0.116 [  529.132163] 000000005387815d: 0000000000000067 (0x67)
Nov 14 15:36:47 10.243.0.116 [  529.132173] 00000000a4995aa7: 00000000ffffc5ae (0xffffc5ae)
Nov 14 15:36:47 10.243.0.116 [  529.132184] 00000000d43145ee: ffffffffbe539518 (__log_buf+0x17ef8/0x80020)
Nov 14 15:36:47 10.243.0.116 [  529.132197] 000000008984b050: ffff8883de209900 (0xffff8883de209900)
Nov 14 15:36:47 10.243.0.116 [  529.132209] 00000000b96b9174: ffffffffbb4cbc5d (memmove+0x4d/0x70)
Nov 14 15:36:47 10.243.0.116 [  529.132222] 00000000b021f99f: ffffffffbe539518 (__log_buf+0x17ef8/0x80020)
Nov 14 15:36:47 10.243.0.116 [  529.132234] 000000006d7be158: 0000000000000004 (0x4)
Nov 14 15:36:47 10.243.0.116 [  529.132244] 000000008e6df9b1: ffff8883de2099b8 (0xffff8883de2099b8)
Nov 14 15:36:47 10.243.0.116 [  529.132256] 000000004869db8d: 9fbcf6e6657b4600 (0x9fbcf6e6657b4600)
Nov 14 15:36:47 10.243.0.116 [  529.132268] 0000000001aee0cc: 0000000000000067 (0x67)
Nov 14 15:36:47 10.243.0.116 [  529.132278] 000000004fccd22e: ffff8883de227b2d (0xffff8883de227b2d)
Nov 14 15:36:48 10.243.0.116 [  529.132290] 000000004372c524: 0000000000000082 (0x82)
Nov 14 15:36:48 10.243.0.116 [  529.132300] 000000004abbd464: 0000000000000002 (0x2)
Nov 14 15:36:48 10.243.0.116 [  529.132310] 000000006fcd69d2: ffffffffbcf3db50 (_printk_rb_static_infos+0x1f3d0/0x160020)
Nov 14 15:36:48 10.243.0.116 [  529.132326] 000000009439164a: ffffffffbcf40598 (_printk_rb_static_infos+0x21e18/0x160020)
Nov 14 15:36:48 10.243.0.116 [  529.132342] 000000008f5a51aa: ffffffffbe53b6b0 (__log_buf+0x1a090/0x80020)
Nov 14 15:36:48 10.243.0.116 [  529.132354] 00000000ac4d84bc: ffff888300000040 (0xffff888300000040)
Nov 14 15:36:48 10.243.0.116 [  529.132366] 000000009c3ce432: 0000000000000246 (0x246)
Nov 14 15:36:48 10.243.0.116 [  529.132376] 000000007a45154d: ffffffffbcca38cd (.LC11+0x52cd/0x5c78)
Nov 14 15:36:48 10.243.0.116 [  529.132389] 0000000049a0772a: 0000007b32a7f430 (0x7b32a7f430)
Nov 14 15:36:48 10.243.0.116 [  529.132400] 0000000077a21716: ffff8883de209a38 (0xffff8883de209a38)
Nov 14 15:36:48 10.243.0.116 [  529.132412] 00000000d46d4179: ffffffff80000000 (0xffffffff80000000)
Nov 14 15:36:48 10.243.0.116 [  529.132424] 00000000f51abe41: ffffffffbcf1e700 (prb+0x40/0x40)
Nov 14 15:36:48 10.243.0.116 [  529.132437] 000000005388a0e3: 0000000000000097 (0x97)
Nov 14 15:36:48 10.243.0.116 [  529.132448] 000000003a2562c0: 00000000ffffc633 (0xffffc633)
Nov 14 15:36:48 10.243.0.116 [  529.132459] 000000006005e98a: 0000000000000040 (0x40)
Nov 14 15:36:48 10.243.0.116 [  529.132469] 000000003980664b: 0000000000000000 ...
Nov 14 15:36:48 10.243.0.116 [  529.132473] 000000006ac84573: 0000000041b58ab3 (0x41b58ab3)
Nov 14 15:36:48 10.243.0.116 [  529.132484] 000000003f940649: ffffffffbcca4d68 (.LC0+0xaf0/0x2536)
Nov 14 15:36:48 10.243.0.116 [  529.132497] 0000000083fb5b49: ffffffffbb1ad060 (printk_parse_prefix+0x10/0x10)
Nov 14 15:36:48 10.243.0.116 [  529.132509] 00000000d29f9f88: ffff888300000028 (0xffff888300000028)
Nov 14 15:36:48 10.243.0.116 [  529.132521] 00000000af2f612b: ffff8883de209b70 (0xffff8883de209b70)
Nov 14 15:36:48 10.243.0.116 [  529.132533] 000000005c6cdf03: ffff8883de209b28 (0xffff8883de209b28)
Nov 14 15:36:48 10.243.0.116 [  529.132544] 00000000c3716aaf: ffffed107bc41340 (0xffffed107bc41340)
Nov 14 15:36:48 10.243.0.116 [  529.132556] 0000000048930860: 0000000000000000 ...
Nov 14 15:36:48 10.243.0.116 [  529.132561] 000000009e1c9be9: ffffffffc0d34030 (netconsole+0x10/0xffffffffffffdfe0 [netconsole])
Nov 14 15:36:48 10.243.0.116 [  529.132587] 000000000c9d185b: ffff8883de227b18 (0xffff8883de227b18)
Nov 14 15:36:48 10.243.0.116 [  529.132598] 000000005ec3c23d: ffff8883de227b10 (0xffff8883de227b10)
Nov 14 15:36:48 10.243.0.116 [  529.132610] 00000000b3de6f5c: 0000000000000000 ...
Nov 14 15:36:48 10.243.0.116 [  529.132615] 0000000036f9187b: ffff8883de209a08 (0xffff8883de209a08)
Nov 14 15:36:48 10.243.0.116 [  529.132626] 000000001ce1c746: ffffffffbb4cbb64 (__kasan_check_write+0x14/0x20)
Nov 14 15:36:48 10.243.0.116 [  529.132641] 00000000f2e813a1: ffff8883de209a28 (0xffff8883de209a28)
Nov 14 15:36:48 10.243.0.116 [  529.132652] 0000000062195896: ffffffffbb2d8be3 (irq_work_claim+0x23/0x50)
Nov 14 15:36:48 10.243.0.116 [  529.132668] 00000000689e2328: ffff8883de227b10 (0xffff8883de227b10)
Nov 14 15:36:48 10.243.0.116 [  529.132680] 00000000a7abb420: 00000000ffffffff (0xffffffff)
Nov 14 15:36:48 10.243.0.116 [  529.132691] 000000004ecc7499: ffff8883de227b18 (0xffff8883de227b18)
Nov 14 15:36:48 10.243.0.116 [  529.132702] 00000000ca9cf222: ffff8883de227b10 (0xffff8883de227b10)
Nov 14 15:36:48 10.243.0.116 [  529.132714] 00000000b1d7ae89: ffff8883de209d28 (0xffff8883de209d28)
Nov 14 15:36:48 10.243.0.116 [  529.132726] 000000004796b255: ffff8883de209a50 (0xffff8883de209a50)
Nov 14 15:36:48 10.243.0.116 [  529.132738] 000000008546273e: ffffffffbb4cbb64 (__kasan_check_write+0x14/0x20)
Nov 14 15:36:48 10.243.0.116 [  529.132752] 00000000b76fabfe: ffff8883de209a70 (0xffff8883de209a70)
Nov 14 15:36:48 10.243.0.116 [  529.132764] 000000000ae0383e: ffffffffbb2d8be3 (irq_work_claim+0x23/0x50)
Nov 14 15:36:48 10.243.0.116 [  529.132778] 00000000a93f1368: ffff8883de227b10 (0xffff8883de227b10)
Nov 14 15:36:48 10.243.0.116 [  529.132790] 00000000332b6372: 0000000000000000 ...
Nov 14 15:36:48 10.243.0.116 [  529.132795] 00000000f67a8c42: ffff8883de209a90 (0xffff8883de209a90)
Nov 14 15:36:48 10.243.0.116 [  529.132807] 000000000b1d91f0: ffffffffbb2d8d61 (irq_work_queue+0x11/0x40)
Nov 14 15:36:48 10.243.0.116 [  529.132822] 00000000d3b34959: 0000000000000000 ...
Nov 14 15:36:48 10.243.0.116 [  529.132826] 0000000000a68d4f: 0000000000000026 (0x26)
Nov 14 15:36:48 10.243.0.116 [  529.132837] 000000001013493c: ffff8883de209aa0 (0xffff8883de209aa0)
Nov 14 15:36:48 10.243.0.116 [  529.132848] 0000000073d6f50f: ffffffffbb1aaf5b (__wake_up_klogd.part.28+0x3b/0x50)
Nov 14 15:36:48 10.243.0.116 [  529.132865] 00000000c90cd880: ffff8883de209ab8 (0xffff8883de209ab8)
Nov 14 15:36:48 10.243.0.116 [  529.132877] 00000000d8526066: ffffffffbb1ae03e (vprintk_deferred+0x2e/0x40)
Nov 14 15:36:48 10.243.0.116 [  529.132890] 0000000074ae61bd: ffffed107bc41359 (0xffffed107bc41359)
Nov 14 15:36:48 10.243.0.116 [  529.132901] 0000000072d580b9: ffff8883de209b60 (0xffff8883de209b60)
Nov 14 15:36:48 10.243.0.116 [  529.132913] 00000000428b2550: ffffffffbc0c0dbf (_printk_deferred+0x9a/0xc4)
Nov 14 15:36:48 10.243.0.116 [  529.132930] 0000000073070417: 0000000041b58ab3 (0x41b58ab3)
Nov 14 15:36:48 10.243.0.116 [  529.132941] 0000000018bb4b5a: ffffffffbcc91fb2 (msr_save_dmi_table+0x203652/0x2093e4)
Nov 14 15:36:48 10.243.0.116 [  529.132955] 0000000058ad4334: ffffffffbc0c0d25 (suspend_console.cold.46+0x29/0x29)
Nov 14 15:36:48 10.243.0.116 [  529.132970] 00000000f8c8cd34: ffffffffbe520e40 (ext_text.52517+0x2020/0x2020)
Nov 14 15:36:48 10.243.0.116 [  529.132983] 00000000ba49adfa: ffff888300000028 (0xffff888300000028)
Nov 14 15:36:48 10.243.0.116 [  529.132995] 00000000aa3c150a: ffff8883de209b70 (0xffff8883de209b70)
Nov 14 15:36:48 10.243.0.116 [  529.133007] 000000000595b4d8: ffff8883de209b28 (0xffff8883de209b28)
Nov 14 15:36:48 10.243.0.116 [  529.133019] 00000000e0ce4622: ffff8883de209b10 (0xffff8883de209b10)
Nov 14 15:36:48 10.243.0.116 [  529.133031] 0000000041edaae1: ffffffffbb4cbb64 (__kasan_check_write+0x14/0x20)
Nov 14 15:36:48 10.243.0.116 [  529.133045] 00000000aa8959ed: ffff8883de209b30 (0xffff8883de209b30)
Nov 14 15:36:48 10.243.0.116 [  529.133056] 00000000ce0fe119: ffffffffbc1168d7 (get_stack_info_noinstr+0x17/0xf0)
Nov 14 15:36:48 10.243.0.116 [  529.133073] 00000000cf07427a: 9fbcf6e6657b4600 (0x9fbcf6e6657b4600)
Nov 14 15:36:48 10.243.0.116 [  529.133085] 00000000dcd0aea6: ffff8883de209ba0 (0xffff8883de209ba0)
Nov 14 15:36:48 10.243.0.116 [  529.133097] 000000008a053701: ffff8883de209b28 (0xffff8883de209b28)
Nov 14 15:36:48 10.243.0.116 [  529.133108] 00000000c0841b80: 0000000000000010 (0x10)
Nov 14 15:36:48 10.243.0.116 [  529.133119] 00000000225f2906: 0000000000000010 (0x10)
Nov 14 15:36:48 10.243.0.116 [  529.133129] 000000000eb7339a: 0000000000000010 (0x10)
Nov 14 15:36:48 10.243.0.116 [  529.133139] 00000000375afe86: 0000000000000010 (0x10)
Nov 14 15:36:48 10.243.0.116 [  529.133149] 00000000bd2fe889: 1ffff1107bc41370 (0x1ffff1107bc41370)
Nov 14 15:36:48 10.243.0.116 [  529.133161] 0000000075557976: ffff8883de209c68 (0xffff8883de209c68)
Nov 14 15:36:48 10.243.0.116 [  529.133172] 00000000b4a616c5: ffffffffbc0b55b9 (unwind_dump.cold.5+0xcd/0x14a)
Nov 14 15:36:48 10.243.0.116 [  529.133189] 00000000b58a3533: ffffffffbc0b55b9 (unwind_dump.cold.5+0xcd/0x14a)
Nov 14 15:36:48 10.243.0.116 [  529.133205] 000000006aae4f94: ffff8883de209d50 (0xffff8883de209d50)
Nov 14 15:36:48 10.243.0.116 [  529.133216] 0000000095c7ab22: 0000000041b58ab3 (0x41b58ab3)
Nov 14 15:36:48 10.243.0.116 [  529.133227] 000000006bcfb2eb: ffffffffbcc9cce8 (.LC0+0x22e/0xf26)
Nov 14 15:36:48 10.243.0.116 [  529.133240] 00000000fbbc5bab: ffffffffbb09a9f0 (__read_once_word_nocheck+0x10/0x10)
Nov 14 15:36:48 10.243.0.116 [  529.133254] 00000000c8e74825: ffff8883de209ba8 (0xffff8883de209ba8)
Nov 14 15:36:48 10.243.0.116 [  529.133266] 00000000ebaea821: 0000000000000004 (0x4)
Nov 14 15:36:48 10.243.0.116 [  529.133276] 00000000f78d2edd: ffff8883de209bc0 (0xffff8883de209bc0)
Nov 14 15:36:48 10.243.0.116 [  529.133288] 000000009dcf7811: ffffffffbb1ae03e (vprintk_deferred+0x2e/0x40)
Nov 14 15:36:48 10.243.0.116 [  529.133300] 0000000057e33732: ffffed107bc4137a (0xffffed107bc4137a)
Nov 14 15:36:48 10.243.0.116 [  529.133312] 0000000076ef0285: ffff8883de209c68 (0xffff8883de209c68)
Nov 14 15:36:48 10.243.0.116 [  529.133323] 000000008e3560a2: ffffffffbc0c0dbf (_printk_deferred+0x9a/0xc4)
Nov 14 15:36:48 10.243.0.116 [  529.133338] 00000000471659b5: 0000000041b58ab3 (0x41b58ab3)
Nov 14 15:36:48 10.243.0.116 [  529.133349] 0000000016b309ea: ffffffffbcc91fb2 (msr_save_dmi_table+0x203652/0x2093e4)
Nov 14 15:36:48 10.243.0.116 [  529.133363] 00000000380170f6: 0000000000000002 (0x2)
Nov 14 15:36:48 10.243.0.116 [  529.133373] 00000000ae80c40b: ffff8883de202000 (0xffff8883de202000)
Nov 14 15:36:48 10.243.0.116 [  529.133385] 00000000e3d881c7: ffff8883de20a000 (0xffff8883de20a000)
Nov 14 15:36:48 10.243.0.116 [  529.133397] 000000009daee861: ffff8881008ff8d0 (0xffff8881008ff8d0)
Nov 14 15:36:48 10.243.0.116 [  529.133408] 000000000c98541e: ffff8883de209c30 (0xffff8883de209c30)
Nov 14 15:36:48 10.243.0.116 [  529.133420] 00000000627c5592: ffff8883de209c68 (0xffff8883de209c68)
Nov 14 15:36:48 10.243.0.116 [  529.133432] 00000000aff50ac2: ffffffffbb09ac07 (update_stack_state+0x107/0x240)
Nov 14 15:36:48 10.243.0.116 [  529.133445] 000000003db28f9d: ffff8883de209568 (0xffff8883de209568)
Nov 14 15:36:48 10.243.0.116 [  529.133457] 00000000e2261d7e: 0000000000000000 ...
Nov 14 15:36:48 10.243.0.116 [  529.133462] 00000000b72fd78d: 9fbcf6e6657b4600 (0x9fbcf6e6657b4600)
Nov 14 15:36:48 10.243.0.116 [  529.133474] 000000000fc82b45: ffff8883de209508 (0xffff8883de209508)
Nov 14 15:36:48 10.243.0.116 [  529.133485] 00000000d8e03dfa: 9fbcf6e6657b4600 (0x9fbcf6e6657b4600)
Nov 14 15:36:48 10.243.0.116 [  529.133497] 00000000743d17c1: ffff8883de209d28 (0xffff8883de209d28)
Nov 14 15:36:48 10.243.0.116 [  529.133509] 00000000f14ae4fb: ffff8881008f0000 (0xffff8881008f0000)
Nov 14 15:36:48 10.243.0.116 [  529.133521] 00000000a475e855: 0000000000000007 (0x7)
Nov 14 15:36:48 10.243.0.116 [  529.133530] 00000000b01b20d2: ffff8881008f0000 (0xffff8881008f0000)
Nov 14 15:36:48 10.243.0.116 [  529.133542] 00000000b30810fb: ffff8883de209d80 (0xffff8883de209d80)
Nov 14 15:36:48 10.243.0.116 [  529.133554] 00000000b89c7941: ffff8883de209cc8 (0xffff8883de209cc8)
Nov 14 15:36:48 10.243.0.116 [  529.133566] 0000000041ede0f7: ffffffffbb09af74 (unwind_next_frame.part.4+0x1e4/0x3b0)
Nov 14 15:36:48 10.243.0.116 [  529.133580] 000000007c607d29: ffff8883de209ca8 (0xffff8883de209ca8)
Nov 14 15:36:48 10.243.0.116 [  529.133591] 0000000076eb8f83: ffffffffbcc98b0d (.LC1+0xdc9/0x4d76)
Nov 14 15:36:48 10.243.0.116 [  529.133604] 000000004a8c15ad: ffff8883de209d60 (0xffff8883de209d60)
Nov 14 15:36:48 10.243.0.116 [  529.133616] 0000000036baea76: ffff8883de209508 (0xffff8883de209508)
Nov 14 15:36:48 10.243.0.116 [  529.133627] 00000000ae384451: ffff8883de209d50 (0xffff8883de209d50)
Nov 14 15:36:48 10.243.0.116 [  529.133639] 0000000026e69587: ffff8883de209d28 (0xffff8883de209d28)
Nov 14 15:36:48 10.243.0.116 [  529.133651] 00000000de275070: 0000000000000000 ...
Nov 14 15:36:48 10.243.0.116 [  529.133656] 00000000650ef2cc: ffff8881008f0000 (0xffff8881008f0000)
Nov 14 15:36:48 10.243.0.116 [  529.133667] 0000000060521f0e: ffffffffbccbbb31 (.LC115+0x223/0x557)
Nov 14 15:36:48 10.243.0.116 [  529.133680] 00000000fab9cdd6: ffffffffbcc98b0d (.LC1+0xdc9/0x4d76)
Nov 14 15:36:48 10.243.0.116 [  529.133692] 00000000bab62068: ffff8883de209ce0 (0xffff8883de209ce0)
Nov 14 15:36:48 10.243.0.116 [  529.133704] 000000000c7e4825: ffffffffbb09b15b (unwind_next_frame+0x1b/0x30)
Nov 14 15:36:48 10.243.0.116 [  529.133717] 000000001c3965ea: ffff8883de209560 (0xffff8883de209560)
Nov 14 15:36:48 10.243.0.116 [  529.133729] 00000000b7c5c49b: ffff8883de209db8 (0xffff8883de209db8)
Nov 14 15:36:48 10.243.0.116 [  529.133741] 00000000ac071018: ffffffffbc0b17f5 (show_trace_log_lvl+0x1f9/0x296)
Nov 14 15:36:48 10.243.0.116 [  529.133754] 00000000bc9c9dd5: ffffffffbb002712 (ret_from_fork+0x22/0x30)
Nov 14 15:36:48 10.243.0.116 [  529.133768] 000000000f56a718: ffff8883de209560 (0xffff8883de209560)
Nov 14 15:36:48 10.243.0.116 [  529.133779] 00000000e9c18dc2: 0000000000000004 (0x4)
Nov 14 15:36:48 10.243.0.116 [  529.133789] 00000000188c828d: 0000000000000002 (0x2)
Nov 14 15:36:48 10.243.0.116 [  529.133799] 0000000081360797: ffff8883de202000 (0xffff8883de202000)
Nov 14 15:36:48 10.243.0.116 [  529.133811] 0000000016ee3f0e: ffff8883de20a000 (0xffff8883de20a000)
Nov 14 15:36:48 10.243.0.116 [  529.133822] 000000005c8c6dab: ffff8881008ff8d0 (0xffff8881008ff8d0)
Nov 14 15:36:49 10.243.0.116 [  529.133834] 00000000741037a5: 0000000000000002 (0x2)
Nov 14 15:36:49 10.243.0.116 [  529.133844] 00000000f4d844df: ffff8883de202000 (0xffff8883de202000)
Nov 14 15:36:49 10.243.0.116 [  529.133856] 0000000092677788: ffff8883de20a000 (0xffff8883de20a000)
Nov 14 15:36:49 10.243.0.116 [  529.133867] 00000000cf395131: ffff8881008ff8d0 (0xffff8881008ff8d0)
Nov 14 15:36:49 10.243.0.116 [  529.133879] 00000000f682fe51: 0000000000000004 (0x4)
Nov 14 15:36:49 10.243.0.116 [  529.133889] 00000000b0e88e7d: ffff8881008f0000 (0xffff8881008f0000)
Nov 14 15:36:49 10.243.0.116 [  529.133901] 00000000ca2ebd51: 0000010100000000 (0x10100000000)
Nov 14 15:36:49 10.243.0.116 [  529.133912] 00000000af8db30c: ffff8883de209558 (0xffff8883de209558)
Nov 14 15:36:49 10.243.0.116 [  529.133924] 00000000221c07cb: ffff8883de209558 (0xffff8883de209558)
Nov 14 15:36:49 10.243.0.116 [  529.133936] 000000008aac9934: ffff8883de209628 (0xffff8883de209628)
Nov 14 15:36:49 10.243.0.116 [  529.133947] 00000000997756d0: 0000000000000000 ...
Nov 14 15:36:49 10.243.0.116 [  529.133952] 000000003760ac0a: 9fbcf6e6657b4600 (0x9fbcf6e6657b4600)
Nov 14 15:36:49 10.243.0.116 [  529.133964] 00000000ae588beb: ffff8883de258800 (0xffff8883de258800)
Nov 14 15:36:49 10.243.0.116 [  529.133976] 00000000831eed5a: ffff8883de258800 (0xffff8883de258800)
Nov 14 15:36:49 10.243.0.116 [  529.133987] 00000000c7f3bb99: 0000000000000000 ...
Nov 14 15:36:49 10.243.0.116 [  529.133992] 00000000662f8ee6: 0000000000000016 (0x16)
Nov 14 15:36:49 10.243.0.116 [  529.134002] 00000000f2be797d: 00000000ee6b2800 (0xee6b2800)
Nov 14 15:36:49 10.243.0.116 [  529.134013] 0000000060a61a2a: ffff8883de209dd0 (0xffff8883de209dd0)
Nov 14 15:36:49 10.243.0.116 [  529.134025] 0000000038d8a03d: ffffffffbc0b190a (show_regs.cold.17+0x1a/0x1f)
Nov 14 15:36:49 10.243.0.116 [  529.134038] 000000009029f41b: ffff8881008f0000 (0xffff8881008f0000)
Nov 14 15:36:49 10.243.0.116 [  529.134050] 000000000d9c2b39: ffff8883de209e08 (0xffff8883de209e08)
Nov 14 15:36:49 10.243.0.116 [  529.134062] 00000000bd7966f4: ffffffffbb2940b0 (watchdog_timer_fn+0x210/0x280)
Nov 14 15:36:49 10.243.0.116 [  529.134076] 00000000973eb3e4: ffff8883de25c260 (0xffff8883de25c260)
Nov 14 15:36:49 10.243.0.116 [  529.134088] 00000000d856845d: ffff8883de25b3c0 (0xffff8883de25b3c0)
Nov 14 15:36:49 10.243.0.116 [  529.134099] 0000000053bc3834: ffff8883de25b440 (0xffff8883de25b440)
Nov 14 15:36:49 10.243.0.116 [  529.134111] 00000000e9f58748: ffffffffbd7479e0 (__cpu_present_mask+0x400/0x400)
Nov 14 15:36:49 10.243.0.116 [  529.134126] 0000000075cdd591: ffffffffbb293ea0 (lockup_detector_update_enable+0x50/0x50)
Nov 14 15:36:49 10.243.0.116 [  529.134141] 00000000b4e817d1: ffff8883de209ef0 (0xffff8883de209ef0)
Nov 14 15:36:49 10.243.0.116 [  529.134153] 0000000087c03c30: ffffffffbb1f6f6d (__hrtimer_run_queues+0x38d/0x730)
Nov 14 15:36:49 10.243.0.116 [  529.134169] 0000000003c7cd6c: 1ffff1107bc413cd (0x1ffff1107bc413cd)
Nov 14 15:36:49 10.243.0.116 [  529.134181] 00000000a70c59f9: 0000007b0a3a037d (0x7b0a3a037d)
Nov 14 15:36:49 10.243.0.116 [  529.134192] 00000000ae282e6b: 00000000008f0dd8 (0x8f0dd8)
Nov 14 15:36:49 10.243.0.116 [  529.134203] 000000001b694a5a: ffff8883de209e40 (0xffff8883de209e40)
Nov 14 15:36:49 10.243.0.116 [  529.134215] 000000006e1cc746: 0000000000000000 ...
Nov 14 15:36:49 10.243.0.116 [  529.134219] 000000002c95dd6a: ffff8883de25b498 (0xffff8883de25b498)
Nov 14 15:36:49 10.243.0.116 [  529.134231] 00000000375b8266: 0000000000000000 ...
Nov 14 15:36:49 10.243.0.116 [  529.134236] 00000000fde59118: ffff8883de25b450 (0xffff8883de25b450)
Nov 14 15:36:49 10.243.0.116 [  529.134248] 00000000ce81d14c: ffff8883de25b3c0 (0xffff8883de25b3c0)
Nov 14 15:36:49 10.243.0.116 [  529.134259] 000000002c6d5c77: ffff8883de25b488 (0xffff8883de25b488)
Nov 14 15:36:49 10.243.0.116 [  529.134271] 00000000a569fe40: 0000000041b58ab3 (0x41b58ab3)
Nov 14 15:36:49 10.243.0.116 [  529.134282] 00000000310ff610: ffffffffbcca7192 (.LC2+0x9e4/0xa452)
Nov 14 15:36:49 10.243.0.116 [  529.134294] 00000000409a0852: ffffffffbb1f6be0 (enqueue_hrtimer+0x180/0x180)
Nov 14 15:36:49 10.243.0.116 [  529.134310] 0000000083e8f8e7: ffff8883de25b3c0 (0xffff8883de25b3c0)
Nov 14 15:36:49 10.243.0.116 [  529.134322] 00000000f10fc082: 0000007b0a3a037d (0x7b0a3a037d)
Nov 14 15:36:49 10.243.0.116 [  529.134333] 0000000046cceb63: ffffffffbb1c38ee (inband_irq_restore+0xe/0x20)
Nov 14 15:36:49 10.243.0.116 [  529.134346] 0000000047adf15b: ffff8883de209ef0 (0xffff8883de209ef0)
Nov 14 15:36:49 10.243.0.116 [  529.134358] 000000009202353f: ffffffffbb1fd6fe (ktime_get_update_offsets_now+0xee/0x200)
Nov 14 15:36:49 10.243.0.116 [  529.134373] 00000000647e6db1: ffff8883de25b628 (0xffff8883de25b628)
Nov 14 15:36:49 10.243.0.116 [  529.134384] 0000000072b985f8: ffff8883de25b5a8 (0xffff8883de25b5a8)
Nov 14 15:36:49 10.243.0.116 [  529.134396] 00000000f16adde1: ffff8883de25b528 (0xffff8883de25b528)
Nov 14 15:36:49 10.243.0.116 [  529.134408] 00000000cbd7ef02: 9fbcf6e6657b4600 (0x9fbcf6e6657b4600)
Nov 14 15:36:49 10.243.0.116 [  529.134420] 0000000031881ce5: ffff8883de25b3c0 (0xffff8883de25b3c0)
Nov 14 15:36:49 10.243.0.116 [  529.134432] 00000000bdf19510: 0000007b0a3a037d (0x7b0a3a037d)
Nov 14 15:36:49 10.243.0.116 [  529.134443] 000000002a021434: 0000000000000000 ...
Nov 14 15:36:49 10.243.0.116 [  529.134447] 00000000551bbf16: ffff8883de25b40c (0xffff8883de25b40c)
Nov 14 15:36:49 10.243.0.116 [  529.134459] 00000000b0f8a932: ffff8883de25b528 (0xffff8883de25b528)
Nov 14 15:36:49 10.243.0.116 [  529.134471] 0000000090f9dab7: ffff8883de209f88 (0xffff8883de209f88)
Nov 14 15:36:49 10.243.0.116 [  529.134483] 00000000e439fb99: ffffffffbb1f85c5 (hrtimer_interrupt+0x1a5/0x350)
Nov 14 15:36:49 10.243.0.116 [  529.134499] 000000006f1a6586: 0000007b0a3a037d (0x7b0a3a037d)
Nov 14 15:36:49 10.243.0.116 [  529.134510] 0000000051c8bb38: 000000030635b600 (0x30635b600)
Nov 14 15:36:49 10.243.0.116 [  529.134521] 00000000b7f950ad: ffff8883de25b408 (0xffff8883de25b408)
Nov 14 15:36:49 10.243.0.116 [  529.134533] 000000008e7d930d: ffff8883de25b828 (0xffff8883de25b828)
Nov 14 15:36:49 10.243.0.116 [  529.134545] 00000000ef911a43: ffff8883de25b7a8 (0xffff8883de25b7a8)
Nov 14 15:36:49 10.243.0.116 [  529.134557] 00000000ad03c251: ffff8883de25b728 (0xffff8883de25b728)
Nov 14 15:36:49 10.243.0.116 [  529.134568] 00000000beffc89e: ffff8883de25b430 (0xffff8883de25b430)
Nov 14 15:36:49 10.243.0.116 [  529.134580] 00000000d36a9c73: 0000000041b58ab3 (0x41b58ab3)
Nov 14 15:36:49 10.243.0.116 [  529.134591] 00000000476ccfaa: 7fffffffffffffff (0x7fffffffffffffff)
Nov 14 15:36:49 10.243.0.116 [  529.134603] 00000000671568eb: ffff8883de25b420 (0xffff8883de25b420)
Nov 14 15:36:49 10.243.0.116 [  529.134615] 00000000f5c8f455: ffff8883de25b628 (0xffff8883de25b628)
Nov 14 15:36:49 10.243.0.116 [  529.134626] 00000000663e8768: ffff8883de25b5a8 (0xffff8883de25b5a8)
Nov 14 15:36:49 10.243.0.116 [  529.134638] 0000000018aacc2f: ffff8883de25bec0 (0xffff8883de25bec0)
Nov 14 15:36:49 10.243.0.116 [  529.134650] 000000002b12bb4d: ffff88810635b600 (0xffff88810635b600)
Nov 14 15:36:49 10.243.0.116 [  529.134662] 00000000f62261b2: 000000000000001a (0x1a)
Nov 14 15:36:49 10.243.0.116 [  529.134672] 0000000073995e4f: 0000000000000000 ...
Nov 14 15:36:49 10.243.0.116 [  529.134677] 00000000eb7f3bf6: ffff888100063400 (0xffff888100063400)
Nov 14 15:36:49 10.243.0.116 [  529.134689] 00000000b13173bd: ffff8883de209fa0 (0xffff8883de209fa0)
Nov 14 15:36:49 10.243.0.116 [  529.134700] 0000000049b431bb: ffffffffbb218342 (proxy_irq_handler+0x32/0x40)
Nov 14 15:36:49 10.243.0.116 [  529.134716] 00000000131ae12f: ffff8883de25bec0 (0xffff8883de25bec0)
Nov 14 15:36:49 10.243.0.116 [  529.134727] 00000000bb2d04cc: ffff8883de209fd0 (0xffff8883de209fd0)
Nov 14 15:36:49 10.243.0.116 [  529.134739] 00000000930535d7: ffffffffbb1c2882 (handle_synthetic_irq+0xd2/0x300)
Nov 14 15:36:49 10.243.0.116 [  529.134753] 00000000fe23b11c: ffff8881076d6000 (0xffff8881076d6000)
Nov 14 15:36:49 10.243.0.116 [  529.134765] 00000000a415a2b4: 000000000000001a (0x1a)
Nov 14 15:36:49 10.243.0.116 [  529.134775] 0000000020f9babf: 0000000000000000 ...
Nov 14 15:36:49 10.243.0.116 [  529.134780] 00000000c17bcf18: ffff8883de209fe8 (0xffff8883de209fe8)
Nov 14 15:36:49 10.243.0.116 [  529.134791] 0000000081018e16: ffffffffbb0993fa (do_irq_inband+0x2a/0x40)
Nov 14 15:36:49 10.243.0.116 [  529.134804] 00000000732cf632: ffff8883de258800 (0xffff8883de258800)
Nov 14 15:36:49 10.243.0.116 [  529.134816] 00000000c06676cd: ffff8881008ff8f8 (0xffff8881008ff8f8)
Nov 14 15:36:49 10.243.0.116 [  529.134827] 00000000c1f1d9ee: ffffffffbb0995c7 (arch_do_IRQ_pipelined+0xb7/0x5a0)
Nov 14 15:36:49 10.243.0.116 [  529.134841] 000000002752a675: ffff8881008ff8d0 (0xffff8881008ff8d0)
Nov 14 15:36:49 10.243.0.116 [  529.134853] 00000000f67bf0e4: 0000000000000069 (0x69)
Nov 14 15:36:49 10.243.0.116 [  529.134863] 000000001676c077: 000000000000001a (0x1a)
Nov 14 15:36:49 10.243.0.116 [  529.134873] 000000007318ac7d: ffff8881076d6000 (0xffff8881076d6000)
Nov 14 15:36:49 10.243.0.116 [  529.134885] 00000000f30cdb32: ffffffffbce0f4c8 (desc_cache+0x348/0x400)
Nov 14 15:36:49 10.243.0.116 [  529.134900] 0000000030519f56: ffff8883de2587c0 (0xffff8883de2587c0)
Nov 14 15:36:49 10.243.0.116 [  529.134912] 0000000029065785: ffff8881008ff970 (0xffff8881008ff970)
Nov 14 15:36:49 10.243.0.116 [  529.134924] 00000000c42b0c2f: ffffffffbb1c326e (sync_current_irq_stage+0x2de/0x500)
Nov 14 15:36:49 10.243.0.116 [  529.134938] 000000005691593f: 0000000000000000 ...
Nov 14 15:36:49 10.243.0.116 [  529.134942] 0000000097622b82: 0000000000000200 (0x200)
Nov 14 15:36:49 10.243.0.116 [  529.134953] 000000007ae62693: 0000000000000000 ...
Nov 14 15:36:49 10.243.0.116 [  529.134958] 00000000e32336f0: ffff8883de227b60 (0xffff8883de227b60)
Nov 14 15:36:49 10.243.0.116 [  529.134969] 000000005620ed6b: ffff8883de2587c8 (0xffff8883de2587c8)
Nov 14 15:36:49 10.243.0.116 [  529.134981] 00000000137745d7: ffffffffbd0e79e0 (sirq_chip+0x140/0x140)
Nov 14 15:36:49 10.243.0.116 [  529.134996] 000000000e5fa6dd: ffff8883de2587c0 (0xffff8883de2587c0)
Nov 14 15:36:49 10.243.0.116 [  529.135008] 000000004ca75486: ffffffffbd0e79e0 (sirq_chip+0x140/0x140)
Nov 14 15:36:49 10.243.0.116 [  529.135023] 00000000e54fd845: 00000000000587c0 (0x587c0)
Nov 14 15:36:49 10.243.0.116 [  529.135034] 00000000f0eaf87b: ffff8881008f0dd8 (0xffff8881008f0dd8)
Nov 14 15:36:49 10.243.0.116 [  529.135045] 00000000bc84aa23: 00000000000646a0 (0x646a0)
Nov 14 15:36:49 10.243.0.116 [  529.135056] 0000000094d709a1: ffff8881008ff9a0 (0xffff8881008ff9a0)
Nov 14 15:36:49 10.243.0.116 [  529.135068] 000000008b67c0b4: ffffffffbb1c363a (sync_irq_stage+0x1aa/0x1e0)
Nov 14 15:36:49 10.243.0.116 [  529.135081] 00000000d702cf4e: ffff8883de2587c0 (0xffff8883de2587c0)
Nov 14 15:36:49 10.243.0.116 [  529.135093] 00000000c01db597: 0000000000000000 ...
Nov 14 15:36:49 10.243.0.116 [  529.135097] 000000008c242e0b: ffff8881008f0000 (0xffff8881008f0000)
Nov 14 15:36:49 10.243.0.116 [  529.135109] 0000000026f688ad: ffffffffbe5b2d40 (synthetic_irq_domain+0x40/0x40)
Nov 14 15:36:49 10.243.0.116 [  529.135123] 000000009eee558f: ffff8881008ff9d0 (0xffff8881008ff9d0)
Nov 14 15:36:49 10.243.0.116 [  529.135135] 00000000c82ca2ba: ffffffffbb1c36f6 (synchronize_pipeline+0x86/0xe0)
Nov 14 15:36:49 10.243.0.116 [  529.135149] 000000006e8cf4d6: ffff8883de2587c0 (0xffff8883de2587c0)
Nov 14 15:36:49 10.243.0.116 [  529.135161] 00000000f25dd0f4: 0000000000000000 ...
Nov 14 15:36:49 10.243.0.116 [  529.135165] 00000000e94c724e: ffff8881008ffa58 (0xffff8881008ffa58)
Nov 14 15:36:49 10.243.0.116 [  529.135177] 000000003904355e: ffff8883de2587c0 (0xffff8883de2587c0)
Nov 14 15:36:49 10.243.0.116 [  529.135189] 0000000091b84a2c: ffff8881008ffa08 (0xffff8881008ffa08)
Nov 14 15:36:49 10.243.0.116 [  529.135201] 00000000e3d1c0f2: ffffffffbb1c4031 (handle_irq_pipelined_finish+0xf1/0x240)
Nov 14 15:36:49 10.243.0.116 [  529.135215] 000000002ff9b1f8: ffff8881008ffa58 (0xffff8881008ffa58)
Nov 14 15:36:49 10.243.0.116 [  529.135227] 000000009924227a: ffff8881008f0dd8 (0xffff8881008f0dd8)
Nov 14 15:36:49 10.243.0.116 [  529.135239] 000000006c672157: 0000000000000000 ...
Nov 14 15:36:49 10.243.0.116 [  529.135244] 000000002d7e5258: ffff8883de2587c0 (0xffff8883de2587c0)
Nov 14 15:36:49 10.243.0.116 [  529.135255] 00000000c2584c15: 0000000000000000 ...
Nov 14 15:36:49 10.243.0.116 [  529.135260] 00000000f938db81: ffff8881008ffa38 (0xffff8881008ffa38)
Nov 14 15:36:49 10.243.0.116 [  529.135272] 0000000018f28934: ffffffffbc117fc1 (arch_pipeline_entry+0x81/0x110)
Nov 14 15:36:49 10.243.0.116 [  529.135285] 0000000094248c06: 0000000000000000 ...
Nov 14 15:36:49 10.243.0.116 [  529.135290] 000000008d6ef368: ffff8881008ffa48 (0xffff8881008ffa48)
Nov 14 15:36:49 10.243.0.116 [  529.135301] 00000000fd5717a3: ffffffffbc117c5e (sysvec_apic_timer_interrupt+0xe/0x20)
Nov 14 15:36:49 10.243.0.116 [  529.135315] 000000004f38b1c4: ffff8881008ffa59 (0xffff8881008ffa59)
Nov 14 15:36:49 10.243.0.116 [  529.135326] 000000004c20f858: ffffffffbc200d0b (asm_sysvec_apic_timer_interrupt+0x1b/0x20)
Nov 14 15:36:49 10.243.0.116 [  529.135342] 000000009e9bd7f9: ffff8883de3ef7a8 (0xffff8883de3ef7a8)
Nov 14 15:36:49 10.243.0.116 [  529.135354] 00000000cd340f20: ffff8883de26b5c8 (0xffff8883de26b5c8)
Nov 14 15:36:49 10.243.0.116 [  529.135365] 00000000bb23acc3: 0000000000000003 (0x3)
Nov 14 15:36:49 10.243.0.116 [  529.135375] 0000000099bcd81e: ffff8883de26b5c0 (0xffff8883de26b5c0)
Nov 14 15:36:49 10.243.0.116 [  529.135387] 00000000bfd3a3c2: ffff8881008ffb80 (0xffff8881008ffb80)
Nov 14 15:36:50 10.243.0.116 [  529.135399] 000000008f5fba70: ffff8883de3ef7a0 (0xffff8883de3ef7a0)
Nov 14 15:36:50 10.243.0.116 [  529.135411] 00000000a150196b: ffffed102011e1bb (0xffffed102011e1bb)
Nov 14 15:36:50 10.243.0.116 [  529.135422] 000000008ebf103d: ffff8881008f0ddf (0xffff8881008f0ddf)
Nov 14 15:36:50 10.243.0.116 [  529.135434] 000000004721fde9: 0000000000000000 ...
Nov 14 15:36:50 10.243.0.116 [  529.135439] 000000003d7915a2: 0000000000000011 (0x11)
Nov 14 15:36:50 10.243.0.116 [  529.135449] 000000009fd16d91: ffffffffbb222390 (smp_call_function_many_cond+0x1b0/0x4d0)
Nov 14 15:36:50 10.243.0.116 [  529.135463] 0000000044ca1ff3: 0000000000000003 (0x3)
Nov 14 15:36:50 10.243.0.116 [  529.135473] 000000009f2b594f: dffffc0000000000 (0xdffffc0000000000)
Nov 14 15:36:50 10.243.0.116 [  529.135484] 0000000093c8e5a3: ffff8883de3ef7a8 (0xffff8883de3ef7a8)
Nov 14 15:36:50 10.243.0.116 [  529.135496] 00000000a0fa40f5: ffffffffffffffff (0xffffffffffffffff)
Nov 14 15:36:50 10.243.0.116 [  529.135508] 0000000036c467c1: ffffffffbb4caeb0 (__asan_store2+0xa0/0xa0)
Nov 14 15:36:50 10.243.0.116 [  529.135521] 000000002486e91e: 0000000000000010 (0x10)
Nov 14 15:36:50 10.243.0.116 [  529.135531] 00000000e062a523: 0000000000000202 (0x202)
Nov 14 15:36:50 10.243.0.116 [  529.135542] 0000000083436bc2: ffff8881008ffb00 (0xffff8881008ffb00)
Nov 14 15:36:50 10.243.0.116 [  529.135553] 00000000040fb253: 0000000000000018 (0x18)
Nov 14 15:36:50 10.243.0.116 [  529.135564] 0000000000d533e6: ffffffffbb222390 (smp_call_function_many_cond+0x1b0/0x4d0)
Nov 14 15:36:50 10.243.0.116 [  529.135577] 000000005b3ea390: 0000000b41b58ab3 (0xb41b58ab3)
Nov 14 15:36:50 10.243.0.116 [  529.135588] 00000000fd07c06a: ffff8883de26b5c0 (0xffff8883de26b5c0)
Nov 14 15:36:50 10.243.0.116 [  529.135600] 000000001540963b: ffff8883de26b5d0 (0xffff8883de26b5d0)
Nov 14 15:36:50 10.243.0.116 [  529.135612] 0000000095b2d7da: 01ff88810000000b (0x1ff88810000000b)
Nov 14 15:36:50 10.243.0.116 [  529.135623] 00000000a5cf7e88: ffffffffbb04a610 (text_poke_bp_batch+0x350/0x350)
Nov 14 15:36:50 10.243.0.116 [  529.135636] 00000000ac637bae: ffff8883de26b5c8 (0xffff8883de26b5c8)
Nov 14 15:36:50 10.243.0.116 [  529.135648] 00000000c18f0abc: 0000000000000003 (0x3)
Nov 14 15:36:50 10.243.0.116 [  529.135658] 00000000adcb351d: ffffffffbcd128d8 (__per_cpu_offset+0x58/0x10000)
Nov 14 15:36:50 10.243.0.116 [  529.135674] 00000000fbd8f7d0: 0000000000000000 ...
Nov 14 15:36:50 10.243.0.116 [  529.135678] 00000000bea0428c: ffffffffbda41cb0 (tp_vec+0x50/0x1020)
Nov 14 15:36:50 10.243.0.116 [  529.135692] 000000003781f724: ffff8881008ffc78 (0xffff8881008ffc78)
Nov 14 15:36:50 10.243.0.116 [  529.135704] 000000006f6f6b1b: ffffffffbda41cb0 (tp_vec+0x50/0x1020)
Nov 14 15:36:50 10.243.0.116 [  529.135717] 00000000582ff280: ffffffffbbd1cb70 (netif_receive_skb_list_internal+0x100/0x730)
Nov 14 15:36:50 10.243.0.116 [  529.135734] 0000000029c6e685: 000000000000000f (0xf)
Nov 14 15:36:50 10.243.0.116 [  529.135743] 000000003330f591: ffff8881008ffb90 (0xffff8881008ffb90)
Nov 14 15:36:50 10.243.0.116 [  529.135755] 00000000bd1c51c1: ffffffffbb222774 (on_each_cpu_cond_mask+0x24/0x40)
Nov 14 15:36:50 10.243.0.116 [  529.135768] 0000000051d2381a: ffff8881008ffca0 (0xffff8881008ffca0)
Nov 14 15:36:50 10.243.0.116 [  529.135780] 000000004ae7cd28: ffffffffbb04a406 (text_poke_bp_batch+0x146/0x350)
Nov 14 15:36:50 10.243.0.116 [  529.135793] 0000000014a07231: 0000000000000040 (0x40)
Nov 14 15:36:50 10.243.0.116 [  529.135803] 00000000eeb7a175: 1ffff1102011ff7b (0x1ffff1102011ff7b)
Nov 14 15:36:50 10.243.0.116 [  529.135815] 000000004a8f67a1: ffffffffbda41c60 (tp_vec_nr+0x40/0x40)
Nov 14 15:36:50 10.243.0.116 [  529.135828] 000000006fdde61f: 0000000100000000 (0x100000000)
Nov 14 15:36:50 10.243.0.116 [  529.135839] 0000000086891b1b: 0000000000000000 ...
Nov 14 15:36:50 10.243.0.116 [  529.135844] 000000008b335bb2: 00000401000000d7 (0x401000000d7)
Nov 14 15:36:50 10.243.0.116 [  529.135855] 00000000494f712e: 0000000000000000 ...
Nov 14 15:36:50 10.243.0.116 [  529.135860] 000000004596323f: 0000000041b58ab3 (0x41b58ab3)
Nov 14 15:36:50 10.243.0.116 [  529.135871] 00000000a4ae1f10: ffffffffbcc994b8 (.LC1+0x1774/0x4d76)
Nov 14 15:36:50 10.243.0.116 [  529.135883] 000000004c225d53: ffffffffbb04a2c0 (__text_poke+0x620/0x620)
Nov 14 15:36:50 10.243.0.116 [  529.135895] 000000002eca7314: ffffffffbda4154f (insn.37130+0xf/0x40)
Nov 14 15:36:50 10.243.0.116 [  529.135909] 0000000094ed033c: ffffffffbda415cc (page_random+0xc/0x40)
Nov 14 15:36:50 10.243.0.116 [  529.135922] 00000000e2e53e1f: ffffffffbbd1cb70 (netif_receive_skb_list_internal+0x100/0x730)
Nov 14 15:36:50 10.243.0.116 [  529.135937] 00000000f756fb83: ffffffffbbd1cb7f (netif_receive_skb_list_internal+0x10f/0x730)
Nov 14 15:36:50 10.243.0.116 [  529.135952] 00000000fd2d720c: ffffffffbbd1cb75 (netif_receive_skb_list_internal+0x105/0x730)
Nov 14 15:36:50 10.243.0.116 [  529.135967] 000000006374e7ed: 0000000000000000 ...
Nov 14 15:36:50 10.243.0.116 [  529.135972] 00000000a169672f: 0000000000000001 (0x1)
Nov 14 15:36:50 10.243.0.116 [  529.135982] 00000000ea7e3bb8: ffff8881008ffc20 (0xffff8881008ffc20)
Nov 14 15:36:50 10.243.0.116 [  529.135994] 00000000f75b1e26: 0000000000000040 (0x40)
Nov 14 15:36:50 10.243.0.116 [  529.136004] 0000000091283f70: ffffffffbbd1cb70 (netif_receive_skb_list_internal+0x100/0x730)
Nov 14 15:36:50 10.243.0.116 [  529.136019] 0000000010e2cdcb: ffffffffbda41540 (i8259A_auto_eoi+0x40/0x40)
Nov 14 15:36:50 10.243.0.116 [  529.136033] 0000000059857621: 0000000000000005 (0x5)
Nov 14 15:36:50 10.243.0.116 [  529.136043] 0000000064410713: 0000000000000000 ...
Nov 14 15:36:50 10.243.0.116 [  529.136048] 00000000ccda5293: 9fbcf6e6657b4600 (0x9fbcf6e6657b4600)
Nov 14 15:36:50 10.243.0.116 [  529.136060] 0000000078d5cb58: ffffffffbcd3db10 (__start___jump_table+0x16790/0x16d40)
Nov 14 15:36:50 10.243.0.116 [  529.136074] 00000000f0bd57fa: ffff8881008ffcb0 (0xffff8881008ffcb0)
Nov 14 15:36:50 10.243.0.116 [  529.136086] 000000005203f777: 9fbcf6e6657b4600 (0x9fbcf6e6657b4600)
Nov 14 15:36:50 10.243.0.116 [  529.136098] 00000000ed3246c0: ffffffffbcd3db10 (__start___jump_table+0x16790/0x16d40)
Nov 14 15:36:50 10.243.0.116 [  529.136112] 0000000058b38182: ffffffffbcd3e0b0 (__start___jump_table+0x16d30/0x16d40)
Nov 14 15:36:50 10.243.0.116 [  529.136126] 00000000fa162c5f: 0000000000000000 ...
Nov 14 15:36:50 10.243.0.116 [  529.136131] 00000000e5c1eec9: ffffffffbe881d40 (netstamp_needed_deferred+0x40/0x40)
Nov 14 15:36:50 10.243.0.116 [  529.136147] 000000006a57c819: ffffffffbcd3db18 (__start___jump_table+0x16798/0x16d40)
Nov 14 15:36:50 10.243.0.116 [  529.136161] 00000000e54c85fd: ffff8881008ffcb0 (0xffff8881008ffcb0)
Nov 14 15:36:50 10.243.0.116 [  529.136173] 000000009e2b1295: ffffffffbb04ba2a (text_poke_finish+0x1a/0x30)
Nov 14 15:36:50 10.243.0.116 [  529.136185] 00000000dcb0fc2b: ffff8881008ffcc0 (0xffff8881008ffcc0)
Nov 14 15:36:50 10.243.0.116 [  529.136197] 00000000a53a9d84: ffffffffbb0455a7 (arch_jump_label_transform_apply+0x17/0x30)
Nov 14 15:36:50 10.243.0.116 [  529.136214] 00000000dafafc1b: ffff8881008ffcf8 (0xffff8881008ffcf8)
Nov 14 15:36:50 10.243.0.116 [  529.136226] 00000000ed068850: ffffffffbb3c0ea5 (__jump_label_update+0x105/0x140)
Nov 14 15:36:50 10.243.0.116 [  529.136242] 0000000031b257f5: 0000000000000000 ...
Nov 14 15:36:50 10.243.0.116 [  529.136246] 00000000349ae5f3: ffffffffbe881d40 (netstamp_needed_deferred+0x40/0x40)
Nov 14 15:36:50 10.243.0.116 [  529.136262] 00000000aabb79dd: ffffffffbe881d48 (netstamp_needed_key+0x8/0x40)
Nov 14 15:36:50 10.243.0.116 [  529.136277] 00000000d2f5ae61: ffffffffbcd3e0c0 (__start___jump_table+0x16d40/0x16d40)
Nov 14 15:36:50 10.243.0.116 [  529.136291] 000000003328d50e: ffff8883de269a40 (0xffff8883de269a40)
Nov 14 15:36:50 10.243.0.116 [  529.136303] 0000000027d50bb3: ffff8881008ffd30 (0xffff8881008ffd30)
Nov 14 15:36:50 10.243.0.116 [  529.136314] 00000000ac799d20: ffffffffbb3c104b (jump_label_update+0x16b/0x1a0)
Nov 14 15:36:50 10.243.0.116 [  529.136330] 000000000e081f0a: ffffffffbe881d40 (netstamp_needed_deferred+0x40/0x40)
Nov 14 15:36:50 10.243.0.116 [  529.136345] 00000000edf0bd3e: ffff8881000c1300 (0xffff8881000c1300)
Nov 14 15:36:50 10.243.0.116 [  529.136357] 000000004ee4cd45: 0000000000000000 ...
Nov 14 15:36:50 10.243.0.116 [  529.136362] 00000000a2d6abe0: ffff8881008ffe08 (0xffff8881008ffe08)
Nov 14 15:36:50 10.243.0.116 [  529.136373] 000000009615d461: ffff8883de269a40 (0xffff8883de269a40)
Nov 14 15:36:50 10.243.0.116 [  529.136385] 00000000dcfbe5ab: ffff8881008ffd48 (0xffff8881008ffd48)
Nov 14 15:36:50 10.243.0.116 [  529.136397] 00000000fd2c5fd7: ffffffffbb3c1140 (static_key_enable_cpuslocked+0xc0/0x100)
Nov 14 15:36:50 10.243.0.116 [  529.136413] 0000000030211707: ffffffffbe881d40 (netstamp_needed_deferred+0x40/0x40)
Nov 14 15:36:50 10.243.0.116 [  529.136428] 00000000af689e67: ffff8881008ffd60 (0xffff8881008ffd60)
Nov 14 15:36:50 10.243.0.116 [  529.136440] 000000003c0219a1: ffffffffbb3c1275 (static_key_enable+0x15/0x30)
Nov 14 15:36:50 10.243.0.116 [  529.136455] 000000005745c745: 0000000000000001 (0x1)
Nov 14 15:36:50 10.243.0.116 [  529.136465] 00000000dbf9c702: ffff8881008ffd78 (0xffff8881008ffd78)
Nov 14 15:36:50 10.243.0.116 [  529.136477] 00000000fb3eccc9: ffffffffbbd05cbb (netstamp_clear+0x4b/0x60)
Nov 14 15:36:50 10.243.0.116 [  529.136491] 000000003ee22c59: ffffffffbd658280 (xps_map_mutex+0xc0/0xc0)
Nov 14 15:36:50 10.243.0.116 [  529.136504] 00000000015d5008: ffff8881008ffe90 (0xffff8881008ffe90)
Nov 14 15:36:50 10.243.0.116 [  529.136516] 00000000d33d712d: ffffffffbb101417 (process_one_work+0x517/0xa10)
Nov 14 15:36:50 10.243.0.116 [  529.136529] 00000000a9bb019c: ffff8881000c1348 (0xffff8881000c1348)
Nov 14 15:36:50 10.243.0.116 [  529.136541] 00000000b87f47c2: ffff8881000c1308 (0xffff8881000c1308)
Nov 14 15:36:50 10.243.0.116 [  529.136552] 00000000cbe6835b: ffff8881000c1320 (0xffff8881000c1320)
Nov 14 15:36:50 10.243.0.116 [  529.136564] 000000006125dcad: ffff8881000c1328 (0xffff8881000c1328)
Nov 14 15:36:50 10.243.0.116 [  529.136576] 000000001718abfe: ffff8883de26ee05 (0xffff8883de26ee05)
Nov 14 15:36:50 10.243.0.116 [  529.136588] 000000003e8d5470: ffff8881000c1310 (0xffff8881000c1310)
Nov 14 15:36:50 10.243.0.116 [  529.136600] 000000006be3b345: 1ffff1102011ffbd (0x1ffff1102011ffbd)
Nov 14 15:36:50 10.243.0.116 [  529.136611] 00000000575726eb: ffff888100061000 (0xffff888100061000)
Nov 14 15:36:50 10.243.0.116 [  529.136623] 000000000fdff907: 00000000008ffdd8 (0x8ffdd8)
Nov 14 15:36:50 10.243.0.116 [  529.136634] 000000003ca6f586: ffff8881000c1318 (0xffff8881000c1318)
Nov 14 15:36:50 10.243.0.116 [  529.136646] 00000000dede653d: ffff8883de26ee08 (0xffff8883de26ee08)
Nov 14 15:36:50 10.243.0.116 [  529.136657] 000000003ad88f18: ffff8883de26ee00 (0xffff8883de26ee00)
Nov 14 15:36:50 10.243.0.116 [  529.136669] 000000005bb09100: 0000000041b58ab3 (0x41b58ab3)
Nov 14 15:36:50 10.243.0.116 [  529.136680] 0000000065c2f6f8: ffffffffbcca09db (.LC11+0x23db/0x5c78)
Nov 14 15:36:50 10.243.0.116 [  529.136693] 00000000d785747b: ffffffffbb100f00 (pwq_dec_nr_in_flight+0x120/0x120)
Nov 14 15:36:50 10.243.0.116 [  529.136706] 000000002e18affb: ffff8881008ffe20 (0xffff8881008ffe20)
Nov 14 15:36:50 10.243.0.116 [  529.136718] 000000007429389b: ffffffffbd658280 (xps_map_mutex+0xc0/0xc0)
Nov 14 15:36:50 10.243.0.116 [  529.136729] 00000000ae564aab: ffffffffbe21b200 (lock_classes+0x462c0/0x180020)
Nov 14 15:36:50 10.243.0.116 [  529.136746] 00000000b1cf89d7: 0000000000000000 ...
Nov 14 15:36:50 10.243.0.116 [  529.136750] 00000000d330e1fc: ffffffffbca13fc0 (bpf_xdp_link_lops+0x160/0xf60)
Nov 14 15:36:50 10.243.0.116 [  529.136765] 000000002bff5fc4: 0000000000000000 ...
Nov 14 15:36:50 10.243.0.116 [  529.136770] 0000000091e22e0f: ffff8881008ffe48 (0xffff8881008ffe48)
Nov 14 15:36:50 10.243.0.116 [  529.136782] 0000000097c447a8: ffffffffbc11833d (lockdep_hardirqs_off+0x8d/0xc0)
Nov 14 15:36:50 10.243.0.116 [  529.136795] 0000000051ef92a0: 9fbcf6e6657b4600 (0x9fbcf6e6657b4600)
Nov 14 15:36:51 10.243.0.116 [  529.136806] 000000008cb06935: ffff8883de269a40 (0xffff8883de269a40)
Nov 14 15:36:51 10.243.0.116 [  529.136818] 00000000acea9a2b: ffff8881008f0000 (0xffff8881008f0000)
Nov 14 15:36:51 10.243.0.116 [  529.136830] 00000000003242a4: ffff8883de269a40 (0xffff8883de269a40)
Nov 14 15:36:51 10.243.0.116 [  529.136842] 0000000030f27326: 9fbcf6e6657b4600 (0x9fbcf6e6657b4600)
Nov 14 15:36:51 10.243.0.116 [  529.136854] 000000001c6330e6: ffff8881000c1300 (0xffff8881000c1300)
Nov 14 15:36:51 10.243.0.116 [  529.136865] 0000000035ae973b: ffff8881000c1330 (0xffff8881000c1330)
Nov 14 15:36:51 10.243.0.116 [  529.136877] 000000003570eedd: ffff8883de269a40 (0xffff8883de269a40)
Nov 14 15:36:51 10.243.0.116 [  529.136889] 0000000047d1640c: ffff8883de269a98 (0xffff8883de269a98)
Nov 14 15:36:51 10.243.0.116 [  529.136901] 0000000064e5038e: ffffffffbd658280 (xps_map_mutex+0xc0/0xc0)
Nov 14 15:36:51 10.243.0.116 [  529.136912] 00000000ca1eb1d8: ffff8881008fff00 (0xffff8881008fff00)
Nov 14 15:36:51 10.243.0.116 [  529.136924] 00000000b880b75f: ffffffffbb101962 (worker_thread+0x52/0x580)
Nov 14 15:36:51 10.243.0.116 [  529.136936] 000000005672ab19: ffff8881000c1348 (0xffff8881000c1348)
Nov 14 15:36:51 10.243.0.116 [  529.136948] 000000003d6e777a: ffff8881000c1368 (0xffff8881000c1368)
Nov 14 15:36:51 10.243.0.116 [  529.136960] 0000000088a9e75e: ffff8883de269e00 (0xffff8883de269e00)
Nov 14 15:36:51 10.243.0.116 [  529.136972] 00000000a2417c0a: ffff8881000c1340 (0xffff8881000c1340)
Nov 14 15:36:51 10.243.0.116 [  529.136984] 000000005a0a7e10: ffff8883de269a90 (0xffff8883de269a90)
Nov 14 15:36:51 10.243.0.116 [  529.136995] 00000000d628c6c9: ffffffffbd658288 (netstamp_work+0x8/0x80)
Nov 14 15:36:51 10.243.0.116 [  529.137007] 000000009989aa92: 0000000100032708 (0x100032708)
Nov 14 15:36:51 10.243.0.116 [  529.137019] 000000003763d07a: 00000000fffffffc (0xfffffffc)
Nov 14 15:36:51 10.243.0.116 [  529.137029] 00000000019278f1: ffff8881000c6c00 (0xffff8881000c6c00)
Nov 14 15:36:51 10.243.0.116 [  529.137041] 00000000f64d02e5: ffffffffbb101910 (process_one_work+0xa10/0xa10)
Nov 14 15:36:51 10.243.0.116 [  529.137054] 00000000182d84bb: ffff8881000c1300 (0xffff8881000c1300)
Nov 14 15:36:51 10.243.0.116 [  529.137065] 00000000da28dade: ffff8881008f0000 (0xffff8881008f0000)
Nov 14 15:36:51 10.243.0.116 [  529.137077] 00000000f15a1e28: ffff8881008fff48 (0xffff8881008fff48)
Nov 14 15:36:51 10.243.0.116 [  529.137089] 000000003003c269: ffffffffbb10e44e (kthread+0x1ee/0x220)
Nov 14 15:36:51 10.243.0.116 [  529.137104] 000000003179e66c: ffff888100827b70 (0xffff888100827b70)
Nov 14 15:36:51 10.243.0.116 [  529.137115] 000000008637a83d: ffff8881000c6c78 (0xffff8881000c6c78)
Nov 14 15:36:51 10.243.0.116 [  529.137127] 00000000aa291968: ffffffffbb10e260 (set_kthread_struct+0x80/0x80)
Nov 14 15:36:51 10.243.0.116 [  529.137142] 000000008d69b706: ffff88810013a880 (0xffff88810013a880)
Nov 14 15:36:51 10.243.0.116 [  529.137154] 0000000060ac9456: 0000000000000000 ...
Nov 14 15:36:51 10.243.0.116 [  529.137159] 00000000bdb3941d: ffff8881008fff59 (0xffff8881008fff59)
Nov 14 15:36:51 10.243.0.116 [  529.137171] 00000000121fdcef: ffffffffbb002712 (ret_from_fork+0x22/0x30)
Nov 14 15:36:51 10.243.0.116 [  529.137184] 0000000005a17736: 0000000000000000 ...
Nov 14 15:36:51 10.243.0.116 [  529.137194]  ? put_dec+0xc0/0xc0
Nov 14 15:36:51 10.243.0.116 [  533.416172]  ? stack_trace_save+0x82/0xb0
Nov 14 15:36:51 10.243.0.116 [  533.423592]  ? filter_irq_stacks+0x80/0x80
Nov 14 15:36:51 10.243.0.116 [  533.430970]  ? __kasan_check_read+0x11/0x20
Nov 14 15:36:51 10.243.0.116 [  533.438286]  ? mark_lock.part.53+0x119/0x1460
Nov 14 15:36:51 10.243.0.116 [  533.445569]  ? __kasan_check_read+0x11/0x20
Nov 14 15:36:51 10.243.0.116 [  533.452762]  ? print_usage_bug+0x60/0x60
Nov 14 15:36:51 10.243.0.116 [  533.459936]  ? print_usage_bug+0x60/0x60
Nov 14 15:36:51 10.243.0.116 [  533.466916]  ? dma_map_page_attrs+0x1c2/0x3f0
Nov 14 15:36:51 10.243.0.116 [  533.473894]  ? __kasan_check_write+0x14/0x20
Nov 14 15:36:51 10.243.0.116 [  533.480745]  ? __test_and_set_bit.constprop.24+0x12/0x30
Nov 14 15:36:51 10.243.0.116 [  533.487608]  ? test_and_lock_stage+0x98/0xb0
Nov 14 15:36:51 10.243.0.116 [  533.494421]  ? unlock_stage+0x3b/0x80
Nov 14 15:36:51 10.243.0.116 [  533.501106]  ? lock_is_held_type+0xdd/0x110
Nov 14 15:36:51 10.243.0.116 [  533.507765]  ? __kasan_check_write+0x14/0x20
Nov 14 15:36:51 10.243.0.116 [  533.514359]  ? desc_read+0x18f/0x1c0
Nov 14 15:36:51 10.243.0.116 [  533.521000]  ? __kasan_check_write+0x14/0x20
Nov 14 15:36:51 10.243.0.116 [  533.527661]  ? __kasan_check_read+0x11/0x20
Nov 14 15:36:51 10.243.0.116 [  533.534206]  ? mark_lock.part.53+0x119/0x1460
Nov 14 15:36:51 10.243.0.116 [  533.540752]  ? _prb_read_valid+0x358/0x400
Nov 14 15:36:51 10.243.0.116 [  533.547364]  ? print_usage_bug+0x60/0x60
Nov 14 15:36:51 10.243.0.116 [  533.553990]  ? get_data.isra.10+0x1d0/0x1d0
Nov 14 15:36:51 10.243.0.116 [  533.560644]  ? unlock_stage+0x3b/0x80
Nov 14 15:36:51 10.243.0.116 [  533.567226]  ? lock_release+0x297/0x410
Nov 14 15:36:51 10.243.0.116 [  533.573736]  ? up+0x41/0x60
Nov 14 15:36:51 10.243.0.116 [  533.580126]  ? lock_downgrade+0x3c0/0x3c0
Nov 14 15:36:51 10.243.0.116 [  533.586522]  ? do_raw_spin_lock+0x119/0x1e0
Nov 14 15:36:51 10.243.0.116 [  533.593023]  ? __kasan_check_read+0x11/0x20
Nov 14 15:36:51 10.243.0.116 [  533.599541]  ? __kasan_check_read+0x11/0x20
Nov 14 15:36:51 10.243.0.116 [  533.605907]  ? __kasan_check_write+0x14/0x20
Nov 14 15:36:51 10.243.0.116 [  533.612306]  ? __kasan_check_write+0x14/0x20
Nov 14 15:36:51 10.243.0.116 [  533.618520]  ? __test_and_set_bit.constprop.24+0x12/0x30
Nov 14 15:36:51 10.243.0.116 [  533.624769]  ? test_and_lock_stage+0x98/0xb0
Nov 14 15:36:51 10.243.0.116 [  533.630999]  ? unlock_stage+0x3b/0x80
Nov 14 15:36:51 10.243.0.116 [  533.637156]  ? do_syslog.part.29+0x430/0x430
Nov 14 15:36:51 10.243.0.116 [  533.643293]  ? lock_is_held_type+0xdd/0x110
Nov 14 15:36:51 10.243.0.116 [  533.649732]  ? __kasan_check_write+0x14/0x20
Nov 14 15:36:51 10.243.0.116 [  533.655866]  ? find_held_lock+0xca/0xf0
Nov 14 15:36:51 10.243.0.116 [  533.662017]  ? unlock_stage+0x3b/0x80
Nov 14 15:36:51 10.243.0.116 [  533.668082]  ? lock_release+0x297/0x410
Nov 14 15:36:51 10.243.0.116 [  533.674093]  ? is_bpf_text_address+0x50/0xf0
Nov 14 15:36:51 10.243.0.116 [  533.680190]  ? lock_downgrade+0x3c0/0x3c0
Nov 14 15:36:51 10.243.0.116 [  533.686253]  ? bpf_ksym_find+0x58/0xf0
Nov 14 15:36:51 10.243.0.116 [  533.692149]  ? swsusp_write.cold.20+0x1f5/0x1f5
Nov 14 15:36:51 10.243.0.116 [  533.698123]  ? is_bpf_text_address+0x6f/0xf0
Nov 14 15:36:51 10.243.0.116 [  533.704005]  ? kernel_text_address+0x111/0x120
Nov 14 15:36:51 10.243.0.116 [  533.709823]  ? __kernel_text_address+0xd/0x40
Nov 14 15:36:51 10.243.0.116 [  533.715560]  ? show_trace_log_lvl+0x1d0/0x296
Nov 14 15:36:51 10.243.0.116 [  533.721274]  ? show_trace_log_lvl+0x1d0/0x296
Nov 14 15:36:51 10.243.0.116 [  533.727009]  ? show_regs.cold.17+0x1a/0x1f
Nov 14 15:36:51 10.243.0.116 [  533.732605]  ? watchdog_timer_fn+0x210/0x280
Nov 14 15:36:51 10.243.0.116 [  533.738225]  ? lockup_detector_update_enable+0x50/0x50
Nov 14 15:36:51 10.243.0.116 [  533.743912]  ? __hrtimer_run_queues+0x38d/0x730
Nov 14 15:36:51 10.243.0.116 [  533.749654]  ? enqueue_hrtimer+0x180/0x180
Nov 14 15:36:51 10.243.0.116 [  533.755347]  ? inband_irq_restore+0xe/0x20
Nov 14 15:36:51 10.243.0.116 [  533.761100]  ? ktime_get_update_offsets_now+0xee/0x200
Nov 14 15:36:51 10.243.0.116 [  533.766995]  ? hrtimer_interrupt+0x1a5/0x350
Nov 14 15:36:51 10.243.0.116 [  533.772908]  ? proxy_irq_handler+0x32/0x40
Nov 14 15:36:51 10.243.0.116 [  533.778704]  ? handle_synthetic_irq+0xd2/0x300
Nov 14 15:36:51 10.243.0.116 [  533.789202]  ? do_irq_inband+0x2a/0x40
Nov 14 15:36:51 10.243.0.116 [  533.794970]  ? arch_do_IRQ_pipelined+0xb7/0x5a0
Nov 14 15:36:51 10.243.0.116 [  533.800770]  </IRQ>
Nov 14 15:36:51 10.243.0.116 [  533.806430]  <TASK>
Nov 14 15:36:51 10.243.0.116 [  533.812026]  ? sync_current_irq_stage+0x2de/0x500
Nov 14 15:36:51 10.243.0.116 [  533.817848]  ? sync_irq_stage+0x1aa/0x1e0
Nov 14 15:36:51 10.243.0.116 [  533.823627]  ? synchronize_pipeline+0x86/0xe0
Nov 14 15:36:51 10.243.0.116 [  533.829429]  ? handle_irq_pipelined_finish+0xf1/0x240
Nov 14 15:36:51 10.243.0.116 [  533.835288]  ? arch_pipeline_entry+0x81/0x110
Nov 14 15:36:51 10.243.0.116 [  533.841142]  ? sysvec_apic_timer_interrupt+0xe/0x20
Nov 14 15:36:51 10.243.0.116 [  533.847034]  ? asm_sysvec_apic_timer_interrupt+0x1b/0x20
Nov 14 15:36:51 10.243.0.116 [  533.853040]  ? smp_call_function_many_cond+0x1b0/0x4d0
Nov 14 15:36:51 10.243.0.116 [  533.859090]  ? __asan_store2+0xa0/0xa0
Nov 14 15:36:51 10.243.0.116 [  533.865080]  ? smp_call_function_many_cond+0x1b0/0x4d0
Nov 14 15:36:51 10.243.0.116 [  533.871196]  ? text_poke_bp_batch+0x350/0x350
Nov 14 15:36:51 10.243.0.116 [  533.877321]  ? netif_receive_skb_list_internal+0x100/0x730
Nov 14 15:36:51 10.243.0.116 [  533.883535]  ? on_each_cpu_cond_mask+0x24/0x40
Nov 14 15:36:51 10.243.0.116 [  533.889823]  ? text_poke_bp_batch+0x146/0x350
Nov 14 15:36:51 10.243.0.116 [  533.896216]  ? __text_poke+0x620/0x620
Nov 14 15:36:51 10.243.0.116 [  533.902611]  ? netif_receive_skb_list_internal+0x100/0x730
Nov 14 15:36:51 10.243.0.116 [  533.909124]  ? netif_receive_skb_list_internal+0x10f/0x730
Nov 14 15:36:51 10.243.0.116 [  533.915505]  ? netif_receive_skb_list_internal+0x105/0x730
Nov 14 15:36:51 10.243.0.116 [  533.921822]  ? netif_receive_skb_list_internal+0x100/0x730
Nov 14 15:36:51 10.243.0.116 [  533.928099]  ? text_poke_finish+0x1a/0x30
Nov 14 15:36:51 10.243.0.116 [  533.934272]  ? arch_jump_label_transform_apply+0x17/0x30
Nov 14 15:36:51 10.243.0.116 [  533.940568]  ? __jump_label_update+0x105/0x140
Nov 14 15:36:51 10.243.0.116 [  533.946880]  ? jump_label_update+0x16b/0x1a0
Nov 14 15:36:51 10.243.0.116 [  533.953222]  ? static_key_enable_cpuslocked+0xc0/0x100
Nov 14 15:36:51 10.243.0.116 [  533.959635]  ? static_key_enable+0x15/0x30
Nov 14 15:36:51 10.243.0.116 [  533.965956]  ? netstamp_clear+0x4b/0x60
Nov 14 15:36:51 10.243.0.116 [  533.972263]  ? process_one_work+0x517/0xa10
Nov 14 15:36:51 10.243.0.116 [  533.978649]  ? pwq_dec_nr_in_flight+0x120/0x120
Nov 14 15:36:51 10.243.0.116 [  533.985025]  ? lockdep_hardirqs_off+0x8d/0xc0
Nov 14 15:36:51 10.243.0.116 [  533.991395]  ? worker_thread+0x52/0x580
Nov 14 15:36:51 10.243.0.116 [  533.997787]  ? process_one_work+0xa10/0xa10
Nov 14 15:36:51 10.243.0.116 [  534.004145]  ? kthread+0x1ee/0x220
Nov 14 15:36:51 10.243.0.116 [  534.010458]  ? set_kthread_struct+0x80/0x80
Nov 14 15:36:51 10.243.0.116 [  534.016928]  ? ret_from_fork+0x22/0x30
Nov 14 15:36:51 10.243.0.116 [  534.023451]  </TASK>
Nov 14 15:37:03 10.243.0.116 [  545.280292] ------------[ cut here ]------------
Nov 14 15:37:03 10.243.0.116 [  545.283938] NETDEV WATCHDOG: eno1 (igb): transmit queue 4 timed out
Nov 14 15:37:03 10.243.0.116 [  545.287501] WARNING: CPU: 1 PID: 0 at net/sched/sch_generic.c:478 dev_watchdog+0x4e5/0x550
Nov 14 15:37:03 10.243.0.116 [  545.291073] Modules linked in: xt_CHECKSUM xt_MASQUERADE tun netconsole ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 xt_conntrack ebtable_nat ebtable_broute ip6table_nat ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_nat iptable_mangle iptable_security iptable_raw nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 libcrc32c ip_set nfnetlink ebtable_filter ebtables rfkill ip6table_filter ip6_tables iptable_filter sunrpc dm_mirror dm_region_hash dm_log dm_mod iTCO_wdt gpio_ich iTCO_vendor_support x86_pkg_temp_thermal intel_powerclamp mxm_wmi coretemp kvm_intel kvm irqbypass crct10dif_pclmul crc32_pclmul ghash_clmulni_intel aesni_intel crypto_simd cryptd rapl intel_cstate pcspkr joydev intel_pch_thermal input_leds sg mei_me ftdi_sio i2c_i801 lpc_ich mei mfd_core i2c_smbus ioatdma wmi ip_tables ext4 mbcache jbd2 ast drm_vram_helper drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops drm_ttm_helper sr_mod ttm sd_mod cdrom t10_pi igb drm ixgbe ahci
Nov 14 15:37:03 10.243.0.116 [  545.291724]  libahci libata crc32c_intel mdio ptp i2c_algo_bit pps_core dca fuse
Nov 14 15:37:03 10.243.0.116 [  545.331077] CPU: 1 PID: 0 Comm: swapper/1 Tainted: G             L    5.15.77evl-g6437d24c6dbc #1
Nov 14 15:37:03 10.243.0.116 [  545.336167] Hardware name: Supermicro Super Server/X10SDV-6C+-TLN4F, BIOS 2.1 11/22/2019
Nov 14 15:37:03 10.243.0.116 [  545.341369] IRQ stage: Linux
Nov 14 15:37:03 10.243.0.116 [  545.346449] RIP: 0010:dev_watchdog+0x4e5/0x550
Nov 14 15:37:03 10.243.0.116 [  545.351559] Code: ff ff 4c 8b 75 b8 c6 05 fe 16 94 01 01 4c 89 f7 e8 60 23 f5 ff 44 89 e1 4c 89 f6 48 c7 c7 20 2f a3 bc 48 89 c2 e8 24 51 2e 00 <0f> 0b e9 10 ff ff ff 65 ff 05 2d 47 25 44 48 c7 c7 c0 3e 72 bd e8
Nov 14 15:37:03 10.243.0.116 [  545.362667] RSP: 0018:ffff8883de289d28 EFLAGS: 00010282
Nov 14 15:37:03 10.243.0.116 [  545.368165] RAX: 0000000000000000 RBX: ffff888127be4500 RCX: ffffffffbb2172fd
Nov 14 15:37:03 10.243.0.116 [  545.373751] RDX: 1ffff1107bc5b7c1 RSI: 0000000000000008 RDI: ffff8883de2dbe0c
Nov 14 15:37:03 10.243.0.116 [  545.379379] RBP: ffff8883de289d88 R08: ffffed107bc5c8d2 R09: ffffed107bc5c8d2
Nov 14 15:37:03 10.243.0.116 [  545.384992] R10: ffff8883de2e468f R11: ffffed107bc5c8d1 R12: 0000000000000004
Nov 14 15:37:03 10.243.0.116 [  545.390682] R13: ffff888127be4408 R14: ffff888127be4000 R15: 0000000000000001
Nov 14 15:37:03 10.243.0.116 [  545.396307] FS:  0000000000000000(0000) GS:ffff8883de280000(0000) knlGS:0000000000000000
Nov 14 15:37:03 10.243.0.116 [  545.401981] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Nov 14 15:37:03 10.243.0.116 [  545.407732] CR2: 00007f75377db000 CR3: 00000003a4e16001 CR4: 00000000003706e0
Nov 14 15:37:03 10.243.0.116 [  545.413508] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Nov 14 15:37:03 10.243.0.116 [  545.419347] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Nov 14 15:37:03 10.243.0.116 [  545.425110] Call Trace:
Nov 14 15:37:03 10.243.0.116 [  545.430991]  <IRQ>
Nov 14 15:37:03 10.243.0.116 [  545.436780]  ? qdisc_put_unlocked+0x40/0x40
Nov 14 15:37:03 10.243.0.116 [  545.442693]  call_timer_fn+0x118/0x3a0
Nov 14 15:37:03 10.243.0.116 [  545.448463]  ? _raw_spin_unlock_irq+0x22/0x40
Nov 14 15:37:03 10.243.0.116 [  545.454264]  ? add_timer+0x360/0x360
Nov 14 15:37:03 10.243.0.116 [  545.460016]  ? __kasan_check_write+0x14/0x20
Nov 14 15:37:03 10.243.0.116 [  545.465872]  ? qdisc_put_unlocked+0x40/0x40
Nov 14 15:37:03 10.243.0.116 [  545.471638]  run_timer_softirq+0x93b/0x9a0
Nov 14 15:37:03 10.243.0.116 [  545.477455]  ? call_timer_fn+0x3a0/0x3a0
Nov 14 15:37:03 10.243.0.116 [  545.483285]  ? rcu_read_lock_sched_held+0xa0/0xd0
Nov 14 15:37:03 10.243.0.116 [  545.489075]  ? rcu_read_lock_any_held.part.15+0x20/0x20
Nov 14 15:37:03 10.243.0.116 [  545.494936]  ? __kasan_check_write+0x14/0x20
Nov 14 15:37:03 10.243.0.116 [  545.500745]  __do_softirq+0xff/0x57b
Nov 14 15:37:03 10.243.0.116 [  545.506538]  irq_exit_rcu+0x7d/0x150
Nov 14 15:37:03 10.243.0.116 [  545.512353]  arch_do_IRQ_pipelined+0xbc/0x5a0
Nov 14 15:37:03 10.243.0.116 [  545.518111]  </IRQ>
Nov 14 15:37:03 10.243.0.116 [  545.523839]  <TASK>
Nov 14 15:37:03 10.243.0.116 [  545.529534]  sync_current_irq_stage+0x2de/0x500
Nov 14 15:37:03 10.243.0.116 [  545.535377]  __inband_irq_enable+0x6f/0x80
Nov 14 15:37:03 10.243.0.116 [  545.541062]  inband_irq_enable+0x1b/0x30
Nov 14 15:37:03 10.243.0.116 [  545.546677]  cpuidle_enter_state+0x193/0x780
Nov 14 15:37:03 10.243.0.116 [  545.552173]  cpuidle_enter+0x3c/0x70
Nov 14 15:37:03 10.243.0.116 [  545.557550]  call_cpuidle+0x59/0x90
Nov 14 15:37:03 10.243.0.116 [  545.562691]  do_idle+0x370/0x390
Nov 14 15:37:03 10.243.0.116 [  545.567644]  ? arch_cpu_idle_exit+0x40/0x40
Nov 14 15:37:03 10.243.0.116 [  545.572489]  ? schedule_idle+0x43/0x60
Nov 14 15:37:03 10.243.0.116 [  545.577147]  cpu_startup_entry+0x18/0x20
Nov 14 15:37:03 10.243.0.116 [  545.581664]  start_secondary+0x1cc/0x220
Nov 14 15:37:03 10.243.0.116 [  545.586002]  ? set_cpu_sibling_map+0xce0/0xce0
Nov 14 15:37:03 10.243.0.116 [  545.590273]  secondary_startup_64_no_verify+0xb0/0xbb
Nov 14 15:37:03 10.243.0.116 [  545.594490]  </TASK>
Nov 14 15:37:03 10.243.0.116 [  545.598499] irq event stamp: 3264836
Nov 14 15:37:03 10.243.0.116 [  545.602459] hardirqs last  enabled at (3264844): [<ffffffffbb1aaac2>] __up_console_sem+0x62/0x70
Nov 14 15:37:03 10.243.0.116 [  545.606667] hardirqs last disabled at (3264851): [<ffffffffbb1aaaa7>] __up_console_sem+0x47/0x70
Nov 14 15:37:03 10.243.0.116 [  545.610707] softirqs last  enabled at (3263990): [<ffffffffbc400403>] __do_softirq+0x403/0x57b
Nov 14 15:37:03 10.243.0.116 [  545.614604] softirqs last disabled at (3264003): [<ffffffffbb0d261d>] irq_exit_rcu+0x7d/0x150
Nov 14 15:37:03 10.243.0.116 [  545.618433] ---[ end trace 818597eea2f0e48b ]---
Nov 14 15:37:03 10.243.0.116 [  545.623002] igb 0000:05:00.0 eno1: Reset adapter

[-- Attachment #2: smime.p7s --]
[-- Type: application/pkcs7-signature, Size: 6759 bytes --]

Re: [External] - Re: EVL Memory

[Philippe Gerum]

Hi Russell,

Russell Johnson <russell.johnson@kratosdefense.com> writes:

> [[S/MIME Signed Part:Undecided]]
> Hi Philippe,
>
> I went ahead and tested out your changes on the "next" branch. Looks like it
> runs better (so definitely progress), but it still hangs eventually. I
> have

How long does it take to hit this bug?

> attached the output from
> the debug kernel (a couple runs because the output was different both
> times).
>
> Thanks,
>
> Russell
>
> [2. text/plain; stack_trace_1.txt]...
>

This is the log which has the best information so far. There is a lot of
noise added by multiple subsystems though, could you try reproducing the
issue, but this time with the following kernel options set as mentioned
below?

# CONFIG_FTRACE is off
# CONFIG_KASAN is off
# CONFIG_JUMP_LABEL is off
# CONFIG_EVL_DEBUG_WOLI is off
CONFIG_PROVE_LOCKING=y
CONFIG_DEBUG_HARD_LOCKS=y
CONFIG_EVL_DEBUG_CORE=y

Thanks,

-- 
Philippe.

RE: [External] - Re: EVL Memory

[Russell Johnson]

[-- Attachment #1.1: Type: text/plain, Size: 1807 bytes --]

Sometimes I saw this issue in seconds/minutes. However, I did just see it
run over night on one system, and it hasn't died yet. I did see one kernel
splat from that run though, so I attached it here. I am building the kernel
with the parameters you recommended to turn off as we speak. I will share
with you what I see once I get it installed and running.


Thanks,

Russell


-----Original Message-----
From: Philippe Gerum <rpm@xenomai.org> 
Sent: Tuesday, November 15, 2022 1:34 AM
To: Russell Johnson <russell.johnson@kratosdefense.com>
Cc: xenomai@lists.linux.dev; Bryan Butler <Bryan.Butler@kratosdefense.com>;
Shawn McManus <shawn.mcmanus@kratosdefense.com>
Subject: Re: [External] - Re: EVL Memory

CAUTION: This email originated from outside of the organization. Do not
click links or open attachments unless you recognize the sender and know the
content is safe.


Hi Russell,

Russell Johnson <russell.johnson@kratosdefense.com> writes:

> [[S/MIME Signed Part:Undecided]]
> Hi Philippe,
>
> I went ahead and tested out your changes on the "next" branch. Looks 
> like it runs better (so definitely progress), but it still hangs 
> eventually. I have

How long does it take to hit this bug?

> attached the output from
> the debug kernel (a couple runs because the output was different both 
> times).
>
> Thanks,
>
> Russell
>
> [2. text/plain; stack_trace_1.txt]...
>

This is the log which has the best information so far. There is a lot of
noise added by multiple subsystems though, could you try reproducing the
issue, but this time with the following kernel options set as mentioned
below?

# CONFIG_FTRACE is off
# CONFIG_KASAN is off
# CONFIG_JUMP_LABEL is off
# CONFIG_EVL_DEBUG_WOLI is off
CONFIG_PROVE_LOCKING=y
CONFIG_DEBUG_HARD_LOCKS=y
CONFIG_EVL_DEBUG_CORE=y

Thanks,

--
Philippe.

[-- Attachment #1.2: stack_trace_4.txt --]
[-- Type: text/plain, Size: 7325 bytes --]

Nov 15 00:01:55 10.243.1.49 [ 1513.156427] EVL: RxMain switching in-band [pid=11995, excpt=6, adjust_owner_boost+0x17c/0x1a0]
Nov 15 00:01:55 10.243.1.49 [ 1513.203609] EVL: RxMain[11995] could not receive HM event #3
Nov 15 00:01:55 10.243.1.49 [ 1513.203635] ------------[ cut here ]------------
Nov 15 00:01:55 10.243.1.49 [ 1513.203637] WARNING: CPU: 4 PID: 11995 at kernel/evl/mutex.c:274 adjust_owner_boost+0x17c/0x1a0
Nov 15 00:01:55 10.243.1.49 [ 1513.203648] Modules linked in: xt_CHECKSUM xt_MASQUERADE tun netconsole ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 xt_conntrack ebtable_nat ip6table_nat ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_nat iptable_mangle iptable_security iptable_raw nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 ip_set nfnetlink ebtable_filter ebtables rfkill ip6table_filter ip6_tables iptable_filter sunrpc dm_mirror dm_region_hash dm_log dm_mod iTCO_wdt gpio_ich iTCO_vendor_support x86_pkg_temp_thermal mxm_wmi intel_powerclamp coretemp kvm_intel kvm irqbypass crct10dif_pclmul crc32_pclmul ghash_clmulni_intel aesni_intel crypto_simd cryptd rapl mei_me i2c_i801 lpc_ich i2c_smbus input_leds mfd_core joydev intel_pch_thermal pcspkr intel_cstate sg ftdi_sio mei ioatdma acpi_ipmi ipmi_si ipmi_devintf ipmi_msghandler wmi ip_tables xfs libcrc32c sr_mod sd_mod cdrom t10_pi ast drm_vram_helper drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops drm_ttm_helper ttm igb
Nov 15 00:01:55 10.243.1.49  drm
Nov 15 00:01:55 10.243.1.49 [ 1513.203813]  ixgbe ahci libahci libata crc32c_intel mdio ptp i2c_algo_bit pps_core dca
Nov 15 00:01:55 10.243.1.49 [ 1513.203833] CPU: 4 PID: 11995 Comm: RxMain Not tainted 5.15.77evl-g6437d24c6dbc #1
Nov 15 00:01:55 10.243.1.49 [ 1513.203840] Hardware name: Supermicro Super Server/X10SDV-6C-TLN4F, BIOS 1.3 02/13/2018
Nov 15 00:01:55 10.243.1.49 [ 1513.203843] IRQ stage: Linux
Nov 15 00:01:56 10.243.1.49 [ 1513.203845] RIP: 0010:adjust_owner_boost+0x17c/0x1a0
Nov 15 00:01:56 10.243.1.49 [ 1513.203852] Code: 3a c0 ea ff 5b 41 5c 41 5d 41 5e 5d c3 cc cc cc cc 4c 89 e7 e8 45 be ea ff 48 89 df e8 3d be ea ff eb a1 0f 0b e9 b0 fe ff ff <0f> 0b 4c 89 f7 e8 0a c0 ea ff e9 11 ff ff ff 0f 0b e9 6f ff ff ff
Nov 15 00:01:56 10.243.1.49 [ 1513.203857] RSP: 0018:ffff8881659d7a08 EFLAGS: 00010046
Nov 15 00:01:56 10.243.1.49 [ 1513.203862] RAX: ffff8881602ad1d8 RBX: ffff888141d4e000 RCX: ffffffffac2f0b67
Nov 15 00:01:56 10.243.1.49 [ 1513.203866] RDX: dffffc0000000000 RSI: dffffc0000000000 RDI: ffff8881602ad1d8
Nov 15 00:01:56 10.243.1.49 [ 1513.203870] RBP: ffff8881659d7a28 R08: ffffed102c055a2f R09: ffffed102c055a2f
Nov 15 00:01:56 10.243.1.49 [ 1513.203874] R10: ffff8881602ad173 R11: ffffed102c055a2e R12: ffff8881602ad1f0
Nov 15 00:01:56 10.243.1.49 [ 1513.203878] R13: ffff8881602ad150 R14: ffff8881602ad170 R15: ffff888141d4e078
Nov 15 00:01:56 10.243.1.49 [ 1513.203882] FS:  00007fb882ffe700(0000) GS:ffff88841c000000(0000) knlGS:0000000000000000
Nov 15 00:01:56 10.243.1.49 [ 1513.203887] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Nov 15 00:01:56 10.243.1.49 [ 1513.203891] CR2: 0000000000000000 CR3: 0000000174026005 CR4: 00000000003706e0
Nov 15 00:01:56 10.243.1.49 [ 1513.203895] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Nov 15 00:01:56 10.243.1.49 [ 1513.203898] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Nov 15 00:01:56 10.243.1.49 [ 1513.203902] Call Trace:
Nov 15 00:01:56 10.243.1.49 [ 1513.203904]  <TASK>
Nov 15 00:01:56 10.243.1.49 [ 1513.203916]  drop_booster+0x239/0x350
Nov 15 00:01:56 10.243.1.49 [ 1513.203930]  ? evl_requeue_mutex_wait+0x530/0x530
Nov 15 00:01:56 10.243.1.49 [ 1513.203935]  ? rcu_read_lock_sched_held+0x5b/0xd0
Nov 15 00:01:56 10.243.1.49 [ 1513.203949]  ? rcu_read_lock_any_held.part.15+0x20/0x20
Nov 15 00:01:56 10.243.1.49 [ 1513.203959]  ? do_raw_spin_lock+0xb1/0x1e0
Nov 15 00:01:56 10.243.1.49 [ 1513.203979]  __evl_unlock_mutex+0x2bb/0x340
Nov 15 00:01:56 10.243.1.49 [ 1513.203998]  monitor_oob_ioctl+0x80f/0x1380
Nov 15 00:01:56 10.243.1.49 [ 1513.204008]  ? __kasan_check_read+0x11/0x20
Nov 15 00:01:56 10.243.1.49 [ 1513.204023]  ? do_raw_spin_unlock+0x97/0x140
Nov 15 00:01:56 10.243.1.49 [ 1513.204050]  ? enter_monitor+0x160/0x160
Nov 15 00:01:56 10.243.1.49 [ 1513.204097]  ? do_raw_spin_lock+0x119/0x1e0
Nov 15 00:01:56 10.243.1.49 [ 1513.204106]  ? spin_dump+0x90/0x90
Nov 15 00:01:56 10.243.1.49 [ 1513.204123]  ? __kasan_check_read+0x11/0x20
Nov 15 00:01:56 10.243.1.49 [ 1513.204131]  ? do_raw_spin_unlock+0x97/0x140
Nov 15 00:01:56 10.243.1.49 [ 1513.204156]  EVL_ioctl+0x81/0xf0
Nov 15 00:01:56 10.243.1.49 [ 1513.204174]  do_oob_syscall+0x699/0x6b0
Nov 15 00:01:56 10.243.1.49 [ 1513.204188]  ? prepare_for_signal+0x180/0x180
Nov 15 00:01:56 10.243.1.49 [ 1513.204202]  ? do_oob_irq+0x283/0x7e0
Nov 15 00:01:56 10.243.1.49 [ 1513.204223]  handle_oob_syscall+0x189/0x230
Nov 15 00:01:56 10.243.1.49 [ 1513.204237]  ? handle_pipelined_syscall+0x820/0x820
Nov 15 00:01:56 10.243.1.49 [ 1513.204254]  ? rcu_read_lock_sched_held+0x5b/0xd0
Nov 15 00:01:56 10.243.1.49 [ 1513.204265]  ? rcu_read_lock_any_held.part.15+0x20/0x20
Nov 15 00:01:56 10.243.1.49 [ 1513.204286]  pipeline_syscall+0xa7/0x1c0
Nov 15 00:01:56 10.243.1.49 [ 1513.204306]  syscall_enter_from_user_mode+0x47/0x110
Nov 15 00:01:56 10.243.1.49 [ 1513.204321]  do_syscall_64+0x15/0xb0
Nov 15 00:01:56 10.243.1.49 [ 1513.204332]  entry_SYSCALL_64_after_hwframe+0x61/0xcb
Nov 15 00:01:56 10.243.1.49 [ 1513.204340] RIP: 0033:0x7fb8beef1940
Nov 15 00:01:56 10.243.1.49 [ 1513.204347] Code: 77 ff ff bf 02 00 00 10 8b 44 24 0c 48 98 48 89 c6 48 8b 04 24 48 89 c2 4c 8b 54 24 38 41 b8 00 00 00 00 b8 9d 00 00 00 0f 05 <48> 89 44 24 30 48 8b 44 24 30 89 44 24 2c 8b 44 24 28 be 00 00 00
Nov 15 00:01:56 10.243.1.49 [ 1513.204352] RSP: 002b:00007fb882fee300 EFLAGS: 00000202 ORIG_RAX: 000000000000009d
Nov 15 00:01:56 10.243.1.49 [ 1513.204358] RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 00007fb8beef1940
Nov 15 00:01:57 10.243.1.49 [ 1513.204362] RDX: 0000000000006d02 RSI: 000000000000000c RDI: 0000000010000002
Nov 15 00:01:57 10.243.1.49 [ 1513.204365] RBP: 00007fb882fee430 R08: 0000000000000000 R09: 00000000004596a8
Nov 15 00:01:57 10.243.1.49 [ 1513.204369] R10: 00007fb874000dc0 R11: 0000000000000202 R12: 00007fb882fee4e0
Nov 15 00:01:57 10.243.1.49 [ 1513.204372] R13: 0000000083f2eea8 R14: 0000000000000008 R15: 0000000083f2eea8
Nov 15 00:01:57 10.243.1.49 [ 1513.204410]  </TASK>
Nov 15 00:01:57 10.243.1.49 [ 1513.204412] irq event stamp: 537
Nov 15 00:01:57 10.243.1.49 [ 1513.204414] hardirqs last  enabled at (537): [<ffffffffac1aaac2>] __up_console_sem+0x62/0x70
Nov 15 00:01:57 10.243.1.49 [ 1513.204424] hardirqs last disabled at (536): [<ffffffffac1aaaa7>] __up_console_sem+0x47/0x70
Nov 15 00:01:57 10.243.1.49 [ 1513.204432] softirqs last  enabled at (86): [<ffffffffad400403>] __do_softirq+0x403/0x57b
Nov 15 00:01:57 10.243.1.49 [ 1513.204440] softirqs last disabled at (79): [<ffffffffac0d261d>] irq_exit_rcu+0x7d/0x150
Nov 15 00:01:57 10.243.1.49 [ 1513.204449] ---[ end trace e15e7b271a08d237 ]---
Nov 15 00:01:57 10.243.1.49 [ 1514.893764] EVL: RxMain resuming out-of-band [pid=11995, excpt=6, adjust_owner_boost+0x17e/0x1a0]

[-- Attachment #2: smime.p7s --]
[-- Type: application/pkcs7-signature, Size: 6759 bytes --]

Re: [External] - Re: EVL Memory

[Philippe Gerum]

Russell Johnson <russell.johnson@kratosdefense.com> writes:

> [[S/MIME Signed Part:Undecided]]
> Sometimes I saw this issue in seconds/minutes. However, I did just see it
> run over night on one system, and it hasn't died yet. I did see one kernel
> splat from that run though, so I attached it here.

Ok, this splat is a false positive from an assertion which should have
been removed now that the PI walk is preemptible. The case it complains
about is valid, I'll push a fix to address this tomorrow.

> I am building the kernel
> with the parameters you recommended to turn off as we speak. I will share
> with you what I see once I get it installed and running.
>

Thanks,

-- 
Philippe.

Re: [External] - Re: EVL Memory

[Philippe Gerum]

Philippe Gerum <rpm@xenomai.org> writes:

> Russell Johnson <russell.johnson@kratosdefense.com> writes:
>
>> [[S/MIME Signed Part:Undecided]]
>> Sometimes I saw this issue in seconds/minutes. However, I did just see it
>> run over night on one system, and it hasn't died yet. I did see one kernel
>> splat from that run though, so I attached it here.
>
> Ok, this splat is a false positive from an assertion which should have
> been removed now that the PI walk is preemptible. The case it complains
> about is valid, I'll push a fix to address this tomorrow.
>
>> I am building the kernel
>> with the parameters you recommended to turn off as we speak. I will share
>> with you what I see once I get it installed and running.
>>
>
> Thanks,

I caught this on a different x86 hardware today, this looks like the
issue you reported, only with more reliable reporting from lockdep when
running oob, thanks to some improvements just added to Dovetail (see the
EVL -next branch).

The bug is highly time dependent it seems, this would explain why an
overnight run may pass. In fact, slowing down the machine seems to help,
among three systems running the test, only the one enabling CONFIG_KASAN
reported it here.

The trace below is clean, this will help fixing the issue. I'll follow
up on this.

[  170.488660] 
[  170.488669] ======================================================
[  170.488673] WARNING: possible circular locking dependency detected
[  170.488677] 5.15.77+ #112 Not tainted
[  170.488683] ------------------------------------------------------
[  170.488685] monitor-trylock/575 is trying to acquire lock:
[  170.488692] ffff88810182f018 (monitor-trylock-B:573){-...}-{0:0}, at: evl_lock_mutex_timeout+0x39a/0xf00
[  170.488726] 
[  170.488726] but task is already holding lock:
[  170.488728] ffff8881017ee018 (monitor-trylock-D:573){-.-.}-{0:0}, at: evl_lock_mutex_timeout+0x39a/0xf00
[  170.488749] 
[  170.488749] which lock already depends on the new lock.
[  170.488749] 
[  170.488751] 
[  170.488751] the existing dependency chain (in reverse order) is:
[  170.488754] 
[  170.488754] -> #2 (monitor-trylock-D:573){-.-.}-{0:0}:
[  170.488767]        evl_lock_mutex_timeout+0x3eb/0xf00
[  170.488776]        monitor_oob_ioctl+0x387/0x1560
[  170.488784]        EVL_ioctl+0x82/0x100
[  170.488792]        do_oob_syscall+0x264/0x6b0
[  170.488800]        handle_oob_syscall+0x162/0x1f0
[  170.488809]        pipeline_syscall+0xb7/0x1d0
[  170.488816]        syscall_enter_from_user_mode+0x40/0xa0
[  170.488828]        do_syscall_64+0x15/0xf0
[  170.488836]        entry_SYSCALL_64_after_hwframe+0x61/0xcb
[  170.488847] 
[  170.488847] -> #1 (monitor-try-main:573){-.-.}-{0:0}:
[  170.488859]        evl_wakeup_thread+0x45/0xe0
[  170.488869]        do_clock_tick+0x1d8/0x6c0
[  170.488876]        evl_core_tick+0x6e/0xe0
[  170.488883]        lapic_oob_handler+0x65/0x90
[  170.488891]        do_oob_irq+0x2d8/0x420
[  170.488899]        handle_oob_irq+0x118/0x250
[  170.488907]        generic_pipeline_irq_desc+0x1b2/0x400
[  170.488916]        arch_handle_irq+0x6c/0x270
[  170.488925]        arch_pipeline_entry+0xb1/0x110
[  170.488934]        asm_sysvec_apic_timer_interrupt+0x16/0x20
[  170.488945]        unlock_stage+0x4f/0x70
[  170.488953]        lock_acquire+0x18d/0x3f0
[  170.488963]        evl_init_thread+0x66a/0x820
[  170.488971]        thread_factory_build+0x25a/0x960
[  170.488980]        ioctl_clone_device+0x15c/0x330
[  170.488988]        __x64_sys_ioctl+0xce/0x110
[  170.488998]        do_syscall_64+0x44/0xf0
[  170.489005]        entry_SYSCALL_64_after_hwframe+0x61/0xcb
[  170.489015] 
[  170.489015] -> #0 (monitor-trylock-B:573){-...}-{0:0}:
[  170.489027]        __lock_acquire+0x1bef/0x35e0
[  170.489037]        lock_acquire+0x162/0x3f0
[  170.489045]        evl_lock_mutex_timeout+0x3eb/0xf00
[  170.489053]        monitor_oob_ioctl+0x387/0x1560
[  170.489061]        EVL_ioctl+0x82/0x100
[  170.489068]        do_oob_syscall+0x264/0x6b0
[  170.489076]        handle_oob_syscall+0x162/0x1f0
[  170.489085]        pipeline_syscall+0xb7/0x1d0
[  170.489092]        syscall_enter_from_user_mode+0x40/0xa0
[  170.489101]        do_syscall_64+0x15/0xf0
[  170.489109]        entry_SYSCALL_64_after_hwframe+0x61/0xcb
[  170.489119] 
[  170.489119] other info that might help us debug this:
[  170.489119] 
[  170.489121] Chain exists of:
[  170.489121]   monitor-trylock-B:573 --> monitor-try-main:573 --> monitor-trylock-D:573
[  170.489121] 
[  170.489136]  Possible unsafe locking scenario:
[  170.489136] 
[  170.489138]        CPU0                    CPU1
[  170.489140]        ----                    ----
[  170.489142]   lock(monitor-trylock-D:573);
[  170.489148]                                lock(monitor-try-main:573);
[  170.489155]                                lock(monitor-trylock-D:573);
[  170.489161]   lock(monitor-trylock-B:573);
[  170.489166] 
[  170.489166]  *** DEADLOCK ***
[  170.489166] 
[  170.489168] 2 locks held by monitor-trylock/575:
[  170.489173]  #0: ffff888108433988 (test573.0){-...}-{0:0}, at: evl_lock_mutex_timeout+0x23c/0xf00
[  170.489196]  #1: ffff8881017ee018 (monitor-trylock-D:573){-.-.}-{0:0}, at: evl_lock_mutex_timeout+0x39a/0xf00
[  170.489218] 
[  170.489218] stack backtrace:
[  170.489223] CPU: 1 PID: 575 Comm: monitor-trylock Not tainted 5.15.77+ #112
[  170.489232] Hardware name: TQ-Group TQMxE39M/Type2 - Board Product Name, BIOS 5.12.09.16.05 07/26/2017
[  170.489237] IRQ stage: EVL
[  170.489241] Call Trace:
[  170.489246]  <TASK>
[  170.489252]  dump_stack_lvl+0x7d/0xb3
[  170.489269]  check_noncircular+0x1ca/0x200
[  170.489286]  ? print_circular_bug+0x240/0x240
[  170.489306]  ? __lock_acquire+0x85e/0x35e0
[  170.489331]  ? add_chain_block+0x33a/0x360
[  170.489350]  __lock_acquire+0x1bef/0x35e0
[  170.489391]  ? lockdep_hardirqs_on_prepare+0x240/0x240
[  170.489401]  ? check_preemption_disabled+0x12/0xf0
[  170.489414]  ? unlock_stage+0xa/0x70
[  170.489429]  ? __evl_get_element_by_fundle+0xc5/0x1e0
[  170.489449]  lock_acquire+0x162/0x3f0
[  170.489461]  ? evl_lock_mutex_timeout+0x39a/0xf00
[  170.489479]  ? lock_downgrade+0x3a0/0x3a0
[  170.489489]  ? evl_lock_mutex_timeout+0xa63/0xf00
[  170.489502]  ? do_raw_spin_lock+0x106/0x190
[  170.489517]  ? spin_bug+0xa0/0xa0
[  170.489542]  evl_lock_mutex_timeout+0x3eb/0xf00
[  170.489551]  ? evl_lock_mutex_timeout+0x39a/0xf00
[  170.489586]  ? evl_destroy_mutex+0x230/0x230
[  170.489600]  ? reacquire_held_locks+0x290/0x290
[  170.489615]  ? enter_monitor+0xf1/0x170
[  170.489632]  monitor_oob_ioctl+0x387/0x1560
[  170.489641]  ? do_raw_spin_unlock+0x9b/0x100
[  170.489653]  ? check_preemption_disabled+0x12/0xf0
[  170.489665]  ? unlock_stage+0xa/0x70
[  170.489677]  ? lock_acquire+0x18d/0x3f0
[  170.489690]  ? check_preemption_disabled+0x12/0xf0
[  170.489702]  ? unlock_stage+0xa/0x70
[  170.489714]  ? enter_monitor+0x170/0x170
[  170.489723]  ? evl_get_file+0xec/0x130
[  170.489735]  ? reacquire_held_locks+0x290/0x290
[  170.489762]  ? do_raw_spin_unlock+0x9b/0x100
[  170.489789]  EVL_ioctl+0x82/0x100
[  170.489807]  do_oob_syscall+0x264/0x6b0
[  170.489824]  ? EVL_read+0xe0/0xe0
[  170.489836]  ? do_oob_syscall+0x6b0/0x6b0
[  170.489857]  handle_oob_syscall+0x162/0x1f0
[  170.489873]  ? handle_pipelined_syscall+0x770/0x770
[  170.489913]  pipeline_syscall+0xb7/0x1d0
[  170.489931]  syscall_enter_from_user_mode+0x40/0xa0
[  170.489947]  do_syscall_64+0x15/0xf0
[  170.489959]  entry_SYSCALL_64_after_hwframe+0x61/0xcb
[  170.489971] RIP: 0033:0x7fba0aa0bc2b
[  170.489981] Code: Unable to access opcode bytes at RIP 0x7fba0aa0bc01.
[  170.489985] RSP: 002b:00007fba0a80e960 EFLAGS: 00000246 ORIG_RAX: 000000000000009d
[  170.489995] RAX: ffffffffffffffda RBX: 0000000040106d00 RCX: 00007fba0aa0bc2b
[  170.490001] RDX: 0000000040106d00 RSI: 000000000000000b RDI: 0000000010000002
[  170.490006] RBP: 000000000000000b R08: 0000000000000000 R09: 00007ffd93d65090
[  170.490011] R10: 00007fba0a80ea00 R11: 0000000000000246 R12: 00007fba0a80ea00
[  170.490017] R13: 0000000000000000 R14: 00007ffd93c2fb70 R15: 00007fba0a7ff000
[  170.490056]  </TASK>
[  183.001931] EVL: watchdog triggered on CPU #1 -- runaway thread 'post-many-flags:645.2' signaled
[  214.720837] EVL: fault:954 switching in-band [pid=954, excpt=14, user_pc=0x55a1b74461ec]
[  249.980299] sched: RT throttling activated
[  286.528172] EVL: fault:1500 switching in-band [pid=1500, excpt=14, user_pc=0x5575ebc251ec]
[  322.356198] EVL: watchdog triggered on CPU #1 -- runaway thread 'post-many-flags:1716.0' signaled
[  356.456796] EVL: fault:2037 switching in-band [pid=2037, excpt=14, user_pc=0x55892e4ea1ec]
[  392.208582] EVL: watchdog triggered on CPU #2 -- runaway thread 'post-many-flags:2252.0' signaled
[  423.936215] EVL: fault:2564 switching in-band [pid=2564, excpt=14, user_pc=0x55911362b1ec]
[  459.696676] EVL: watchdog triggered on CPU #3 -- runaway thread 'post-many-flags:2778.1' signaled
[  491.385666] EVL: fault:3096 switching in-band [pid=3096, excpt=14, user_pc=0x564c8f17d1ec]
[  527.246324] EVL: watchdog triggered on CPU #1 -- runaway thread 'post-many-flags:3311.15' signaled
[  558.830749] EVL: fault:3617 switching in-band [pid=3617, excpt=14, user_pc=0x5637705481ec]
[  594.531226] EVL: watchdog triggered on CPU #0 -- runaway thread 'post-many-flags:3832.11' signaled
[  626.164629] EVL: fault:4149 switching in-band [pid=4149, excpt=14, user_pc=0x5589652f01ec]
[  662.032655] EVL: watchdog triggered on CPU #0 -- runaway thread 'post-many-flags:4364.11' signaled
[  693.631577] EVL: fault:4675 switching in-band [pid=4675, excpt=14, user_pc=0x55558d6d91ec]
[  729.450287] EVL: watchdog triggered on CPU #1 -- runaway thread 'post-many-flags:4890.1' signaled
[  761.086870] EVL: fault:5202 switching in-band [pid=5202, excpt=14, user_pc=0x561b37a801ec]


-- 
Philippe.

RE: [External] - Re: EVL Memory

[Russell Johnson]

[-- Attachment #1: Type: text/plain, Size: 104 bytes --]

Sounds good. Just let me know when you want me to test, and I will pull your
latest.


Thanks,

Russell

[-- Attachment #2: smime.p7s --]
[-- Type: application/pkcs7-signature, Size: 6759 bytes --]

Re: [External] - Re: EVL Memory

[Philippe Gerum]

Russell Johnson <russell.johnson@kratosdefense.com> writes:

> [[S/MIME Signed Part:Undecided]]
> Sounds good. Just let me know when you want me to test, and I will pull your
> latest.
>
>

Please try the -next branch, commit #b295ac3ad51b so that we can get a
fresh status with the latest updates, and hopefully better trace dumps
if an issue arises.

Thanks,

-- 
Philippe.

RE: [External] - Re: EVL Memory

[Russell Johnson]

[-- Attachment #1: Type: text/plain, Size: 459 bytes --]

Will do. I am currently fighting a weird build issue since pulling the
latest commit of the kernel. I can try to build on a different system next. 

AR      drivers/built-in.a
error: Bad exit status from /var/tmp/rpm-tmp.2BRlM1 (%build)

I typically run "make rpm-pkg" so that I can install the built kernel on
other systems. I am going to assume the issue is on my end for now... Will
let you know what I find once I can get this to build.

Thanks,

Russell

[-- Attachment #2: smime.p7s --]
[-- Type: application/pkcs7-signature, Size: 6759 bytes --]

Re: [External] - Re: EVL Memory

[Philippe Gerum]

Russell Johnson <russell.johnson@kratosdefense.com> writes:

> [[S/MIME Signed Part:Undecided]]
> Will do. I am currently fighting a weird build issue since pulling the
> latest commit of the kernel. I can try to build on a different system next. 
>
> AR      drivers/built-in.a
> error: Bad exit status from /var/tmp/rpm-tmp.2BRlM1 (%build)
>

Patching issue? If you apply the EVL patch series on top of a different
kernel they are based on, maybe some conflicts have appeared with the
latest release, causing make-pkg to break.

> I typically run "make rpm-pkg" so that I can install the built kernel on
> other systems. I am going to assume the issue is on my end for now... Will
> let you know what I find once I can get this to build.
>
> Thanks,
>
> Russell
>
> [[End of S/MIME Signed Part]]


-- 
Philippe.

RE: [External] - Re: EVL Memory

[Russell Johnson]

[-- Attachment #1: Type: text/plain, Size: 87 bytes --]

This is actually a fresh clone/build of the EVL kernel which is what makes
it weird...

[-- Attachment #2: smime.p7s --]
[-- Type: application/pkcs7-signature, Size: 6759 bytes --]

Re: [External] - Re: EVL Memory

[Philippe Gerum]

Russell Johnson <russell.johnson@kratosdefense.com> writes:

> [[S/MIME Signed Part:Undecided]]
> This is actually a fresh clone/build of the EVL kernel which is what makes
> it weird...
>
> [[End of S/MIME Signed Part]]

Any error log produced by your build system would help.

-- 
Philippe.

RE: [External] - Re: EVL Memory

[Russell Johnson]

[-- Attachment #1: Type: text/plain, Size: 2204 bytes --]

I found the error in the make output. There was also a warning that I pasted
below.

----------------------------------------------------------------------------
-----------------

In file included from ./arch/x86/include/asm/special_insns.h:10,
                 from ./arch/x86/include/asm/processor.h:25,
                 from ./arch/x86/include/asm/cpufeature.h:5,
                 from ./arch/x86/include/asm/thread_info.h:53,
                 from ./include/linux/thread_info.h:60,
                 from ./arch/x86/include/asm/preempt.h:7,
                 from ./include/linux/preempt.h:88,
                 from ./include/linux/spinlock.h:55,
                 from kernel/rcu/tree.c:23:
kernel/rcu/tree_plugin.h: In function 'rcu_read_unlock_strict':
./include/linux/irqflags.h:289:3: error: expected expression before '{'
token
  ({      \
   ^
kernel/rcu/tree_plugin.h:826:40: note: in expansion of macro 'irqs_disabled'
   on_pipeline_entry() || running_oob() irqs_disabled() || preempt_count()
|| !rcu_state.gp_kthread)
                                        ^~~~~~~~~~~~~
In file included from kernel/rcu/tree.c:4762:
kernel/rcu/tree_plugin.h:826:26: error: called object is not a function or
function pointer
   on_pipeline_entry() || running_oob() irqs_disabled() || preempt_count()
|| !rcu_state.gp_kthread)

----------------------------------------------------------------------------
-----------------

CC [M]  sound/soc/intel/common/soc-acpi-intel-bxt-match.o
drivers/net/phy/sfp.c:168: warning: "T_WAIT" redefined
 #define T_WAIT   msecs_to_jiffies(50)

In file included from ./include/evl/thread.h:25,
                 from ./include/evl/sched.h:14,
                 from ./include/evl/flag.h:11,
                 from ./include/asm-generic/evl/netdevice.h:10,
                 from ./arch/x86/include/dovetail/netdevice.h:5,
                 from ./include/net/netoob.h:5,
                 from ./include/linux/netdevice.h:43,
                 from ./include/linux/rtnetlink.h:7,
                 from drivers/net/phy/sfp.c:17:
./include/uapi/evl/thread.h:21: note: this is the location of the previous
definition
 #define T_WAIT    0x00000008 /* Periodic wait */

Thanks,

Russell

[-- Attachment #2: smime.p7s --]
[-- Type: application/pkcs7-signature, Size: 6759 bytes --]

RE: [External] - Re: EVL Memory

[Russell Johnson]

[-- Attachment #1: Type: text/plain, Size: 314 bytes --]

So I can build v5.15.y-evl-rebase just fine on multiple systems (using plain
make and using make rpm-pkg). I cannot build the latest next-evl-rebase
branch on any of my systems (using plain make and using make rpm-pkg). There
must have been something that changed in recent commits.. Any ideas?


Thanks,

Russell

[-- Attachment #2: smime.p7s --]
[-- Type: application/pkcs7-signature, Size: 6759 bytes --]

Re: [External] - Re: EVL Memory

[Philippe Gerum]

Russell Johnson <russell.johnson@kratosdefense.com> writes:

> [[S/MIME Signed Part:Undecided]]
> So I can build v5.15.y-evl-rebase just fine on multiple systems (using plain
> make and using make rpm-pkg). I cannot build the latest next-evl-rebase
> branch on any of my systems (using plain make and using make rpm-pkg). There
> must have been something that changed in recent commits.. Any ideas?
>
>

You may have to resolve conflicts between your custom kernel and these
commits which update the RCU code shared with upstream:

402b2c85f52fa rcu: irq_pipeline: deny strict grace periods from out-of-band context
03d19b028241d rcu: irq_pipeline: enable read lock sections from out-of-band context

-- 
Philippe.

Re: [External] - Re: EVL Memory

[Philippe Gerum]

Philippe Gerum <rpm@xenomai.org> writes:

> Russell Johnson <russell.johnson@kratosdefense.com> writes:
>
>> [[S/MIME Signed Part:Undecided]]
>> Sounds good. Just let me know when you want me to test, and I will pull your
>> latest.
>>
>>
>
> Please try the -next branch, commit #b295ac3ad51b so that we can get a
> fresh status with the latest updates, and hopefully better trace dumps
> if an issue arises.
>

Two overnight tests passed here with this stack (octa-core xu4q/armv7
and quad-core x86 SoM), running loops of the condvar app you sent me,
and the EVL test suite in parallel. Although we may not be done yet,
this is a positive outcome given the amount of stress these tests impose
on the EVL core.

However, the evl ps display you sent me, showing a PI boost would
require some investigation if these threads would never switch back to
their normal priority, clearing the 'b' flag accordingly in the STAT
field.

-- 
Philippe.

RE: [External] - Re: EVL Memory

[Russell Johnson]

[-- Attachment #1: Type: text/plain, Size: 484 bytes --]

After fixing that line of code that was causing errors with building the EVL
kernel, I was able to successfully build. I am now running our scenario that
was causing issues previously. I will let you know what issues (if any) I
run into. 

To your other point, I can confirm that the priorities that are being
boosted are never being cleared back to the original base priorities.. What
do you recommend looking at first in order to figure out why this is
happening?

Thanks,

Russell

[-- Attachment #2: smime.p7s --]
[-- Type: application/pkcs7-signature, Size: 6759 bytes --]

Re: [External] - Re: EVL Memory

[Philippe Gerum]

Russell Johnson <russell.johnson@kratosdefense.com> writes:

> [[S/MIME Signed Part:Undecided]]
> After fixing that line of code that was causing errors with building the EVL
> kernel, I was able to successfully build. I am now running our scenario that
> was causing issues previously. I will let you know what issues (if any) I
> run into. 
>
> To your other point, I can confirm that the priorities that are being
> boosted are never being cleared back to the original base priorities.. What
> do you recommend looking at first in order to figure out why this is
> happening?
>

This is something going on into the core due to the scalable PI
implementation for mutexes. Please pull commit 8390add2f7 from the -next
branch, it adds a couple of debug assertions which should trigger when
the issue arises (expect a kernel splat if so).

-- 
Philippe.

RE: [External] - Re: EVL Memory

[Russell Johnson]

[-- Attachment #1: Type: text/plain, Size: 68 bytes --]

Sounds good. I will let you know what I run into.

Thanks,

Russell

[-- Attachment #2: smime.p7s --]
[-- Type: application/pkcs7-signature, Size: 6759 bytes --]

RE: [External] - Re: EVL Memory

[Russell Johnson]

[-- Attachment #1.1: Type: text/plain, Size: 225 bytes --]

I just tried to run the app with the latest "next" branch, and it didn't
make it long at all before I got a ton of dump in netconsole. But the
messages look very strange to me. A lot of "printk messages dropped"...
Thoughts?

[-- Attachment #1.2: kdump.txt --]
[-- Type: text/plain, Size: 156781 bytes --]

Nov 19 11:05:02 10.243.0.116 [  111.107091] EVL: DummyTxSample switching in-band [pid=1563, excpt=6, __evl_unlock_mutex+0x2a6/0x410]
Nov 19 11:05:02 10.243.0.116 [  111.107128] ------------[ cut here ]------------
Nov 19 11:05:02 10.243.0.116 [  111.107130] WARNING: CPU: 5 PID: 1563 at kernel/evl/sched/core.c:1010 __evl_schedule+0x73a/0x1180
Nov 19 11:05:02 10.243.0.116 [  111.107143] Modules linked in: xt_CHECKSUM xt_MASQUERADE tun netconsole ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 xt_conntrack ebtable_nat ebtable_broute ip6table_nat ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_nat iptable_mangle iptable_security iptable_raw nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 libcrc32c ip_set nfnetlink ebtable_filter ebtables rfkill ip6table_filter ip6_tables iptable_filter sunrpc dm_mirror dm_region_hash dm_log dm_mod x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm iTCO_wdt gpio_ich iTCO_vendor_support mxm_wmi irqbypass crct10dif_pclmul crc32_pclmul ghash_clmulni_intel aesni_intel crypto_simd cryptd rapl pcspkr intel_cstate mei_me sg joydev lpc_ich input_leds i2c_i801 ftdi_sio intel_pch_thermal i2c_smbus mei mfd_core ioatdma wmi ip_tables ext4 mbcache jbd2 ast drm_vram_helper drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops drm_ttm_helper sd_mod sr_mod ttm cdrom t10_pi igb drm ixgbe ahci
Nov 19 11:05:02 10.243.0.116 [  111.107359]  libahci libata crc32c_intel mdio ptp pps_core i2c_algo_bit dca fuse
Nov 19 11:05:02 10.243.0.116 [  111.107377] CPU: 5 PID: 1563 Comm: DummyTxSample Not tainted 5.15.77evl-g8390add2f766 #3
Nov 19 11:05:02 10.243.0.116 [  111.107385] Hardware name: Supermicro Super Server/X10SDV-6C+-TLN4F, BIOS 2.1 11/22/2019
Nov 19 11:05:02 10.243.0.116 [  111.107388] IRQ stage: EVL
Nov 19 11:05:02 10.243.0.116 [  111.107390] RIP: 0010:__evl_schedule+0x73a/0x1180
Nov 19 11:05:02 10.243.0.116 [  111.107397] Code: 46 78 48 39 d8 0f 85 45 fa ff ff 4c 8b 75 b0 48 c7 c6 fb d1 31 88 4c 89 f7 e8 12 15 e8 ff 48 8b 5d d0 48 89 df e8 16 63 e8 ff <0f> 0b 68 fb d1 31 88 31 f6 45 31 c9 41 b8 01 00 00 00 31 c9 31 d2
Nov 19 11:05:02 10.243.0.116 ** 302 printk messages dropped **
Nov 19 11:05:02 10.243.0.116 [  111.110085]  ? evl_switch_inband+0x273/0x820
Nov 19 11:05:02 10.243.0.116 ** 1639 printk messages dropped **
Nov 19 11:05:02 10.243.0.116 [  111.129875] EVL: DummyTxSample switching in-band [pid=1563, excpt=6, evl_switch_oob+0x509/0x610]
Nov 19 11:05:02 10.243.0.116 ** 914 printk messages dropped **
Nov 19 11:05:02 10.243.0.116 [  111.151205] EVL: DummyTxSample switching in-band [pid=1563, excpt=6, evl_switch_oob+0x509/0x610]
Nov 19 11:05:02 10.243.0.116 ** 1019 printk messages dropped **
Nov 19 11:05:02 10.243.0.116 [  111.174977] EVL: DummyTxSample switching in-band [pid=1563, excpt=6, evl_switch_oob+0x509/0x610]
Nov 19 11:05:02 10.243.0.116 ** 1232 printk messages dropped **
Nov 19 11:05:02 10.243.0.116 [  111.191438]  ? search_exception_tables+0x58/0x60
Nov 19 11:05:02 10.243.0.116 ** 1718 printk messages dropped **
Nov 19 11:05:02 10.243.0.116 [  111.202869]  ? search_exception_tables+0x58/0x60
Nov 19 11:05:02 10.243.0.116 ** 1643 printk messages dropped **
Nov 19 11:05:02 10.243.0.116 [  111.214246] RSP: 0018:fffffe000016bfa8 EFLAGS: 00010082
Nov 19 11:05:02 10.243.0.116 ** 1682 printk messages dropped **
Nov 19 11:05:02 10.243.0.116 [  111.225557] RBP: fffffe000016c040 R08: 000000000000000d R09: fffffe000016c360
Nov 19 11:05:02 10.243.0.116 ** 1672 printk messages dropped **
Nov 19 11:05:02 10.243.0.116 [  111.237299] RSP: 0018:fffffe000016ddd8 EFLAGS: 00010046
Nov 19 11:05:02 10.243.0.116 ** 1600 printk messages dropped **
Nov 19 11:05:02 10.243.0.116 [  111.248325]  ? do_kern_addr_fault+0x7c/0xb0
Nov 19 11:05:02 10.243.0.116 ** 1757 printk messages dropped **
Nov 19 11:05:02 10.243.0.116 [  111.260561] #PF: supervisor read access in kernel mode
Nov 19 11:05:02 10.243.0.116 ** 1629 printk messages dropped **
Nov 19 11:05:02 10.243.0.116 [  111.272536] RDX: 1ffffffff12d6bd4 RSI: 0000000000000008 RDI: ffffffff896b5ea0
Nov 19 11:05:02 10.243.0.116 ** 1762 printk messages dropped **
Nov 19 11:05:02 10.243.0.116 [  111.284783] RAX: fffffbc00002d7f7 RBX: fffffe000016c36d RCX: ffffffff88877714
Nov 19 11:05:02 10.243.0.116 ** 1771 printk messages dropped **
Nov 19 11:05:02 10.243.0.116 [  111.297699] Code: b1 ff ff 65 ff 05 94 13 f4 76 48 8b 1d 8d a7 0c 01 48 83 eb 08 4c 8d 6b 08 49 81 fd 40 03 1b 8a 74 3e 48 89 df e8 04 7d 3f ff <83> 3b 03 74 1f 48 8d 75 a8 48 89 df e8 93 a3 14 ff 48 8d 73 18 48
Nov 19 11:05:02 10.243.0.116 ** 2861 printk messages dropped **
Nov 19 11:05:02 10.243.0.116 [  111.318270] R10: ffff8883de4a7b1b R11: ffffed107bc94f63 R12: 1fffffc00002ddc3
Nov 19 11:05:02 10.243.0.116 ** 3879 printk messages dropped **
Nov 19 11:05:02 10.243.0.116 [  111.345877] RBP: fffffe000016fac0 R08: ffffed107bc94f64 R09: ffffed107bc94f64
Nov 19 11:05:02 10.243.0.116 ** 3924 printk messages dropped **
Nov 19 11:05:02 10.243.0.116 [  111.373834]  ? page_fault_oops+0x3c0/0x3c0
Nov 19 11:05:02 10.243.0.116 ** 3737 printk messages dropped **
Nov 19 11:05:02 10.243.0.116 [  111.400598]  ? printk_parse_prefix+0x10/0x10
Nov 19 11:05:02 10.243.0.116 ** 3716 printk messages dropped **
Nov 19 11:05:02 10.243.0.116 [  111.427673] RAX: 0000000000000000 RBX: fffffffffffffff8 RCX: ffffffff890e5bcc
Nov 19 11:05:02 10.243.0.116 ** 3743 printk messages dropped **
Nov 19 11:05:02 10.243.0.116 [  111.454397] RSP: 0018:fffffe000016e1e8 EFLAGS: 00010046
Nov 19 11:05:02 10.243.0.116 ** 3629 printk messages dropped **
Nov 19 11:05:02 10.243.0.116 [  111.480345] PGD 30ae1a067 P4D 30ae1a067 PUD 30ae1c067 PMD 0
Nov 19 11:05:02 10.243.0.116 ** 3578 printk messages dropped **
Nov 19 11:05:02 10.243.0.116 [  111.505897] RSP: 0018:fffffe000016ea08 EFLAGS: 00010046
Nov 19 11:05:02 10.243.0.116 ** 3301 printk messages dropped **
Nov 19 11:05:02 10.243.0.116 [  111.529405] RSP: 0018:fffffe000016ec68 EFLAGS: 00010046
Nov 19 11:05:02 10.243.0.116 ** 3295 printk messages dropped **
Nov 19 11:05:02 10.243.0.116 [  111.553339]  ? fixup_exception+0x32/0x420
Nov 19 11:05:02 10.243.0.116 ** 3213 printk messages dropped **
Nov 19 11:05:02 10.243.0.116 [  111.577218]  ? cmp_ex_search+0x1e/0x40
Nov 19 11:05:02 10.243.0.116 ** 3216 printk messages dropped **
Nov 19 11:05:02 10.243.0.116 [  111.601117]  ? format_decode+0x38/0x5c0
Nov 19 11:05:02 10.243.0.116 ** 3182 printk messages dropped **
Nov 19 11:05:02 10.243.0.116 [  111.624834]  ? string_nocheck+0x135/0x190
Nov 19 11:05:02 10.243.0.116 ** 2980 printk messages dropped **
Nov 19 11:05:02 10.243.0.116 [  111.646616]  ? __die+0x25/0x30
Nov 19 11:05:02 10.243.0.116 ** 2958 printk messages dropped **
Nov 19 11:05:02 10.243.0.116 [  111.668190]  ? print_modules+0x88/0x121
Nov 19 11:05:02 10.243.0.116 ** 2970 printk messages dropped **
Nov 19 11:05:02 10.243.0.116 [  111.690089]  ? lock_release+0xd5/0x420
Nov 19 11:05:02 10.243.0.116 ** 2967 printk messages dropped **
Nov 19 11:05:02 10.243.0.116 [  111.711844]  __die_body+0x1f/0x60
Nov 19 11:05:02 10.243.0.116 ** 2865 printk messages dropped **
Nov 19 11:05:02 10.243.0.116 [  111.732997] R10: ffff8883de4a7b1b R11: ffffed107bc94f63 R12: 1fffffc00002dcbf
Nov 19 11:05:02 10.243.0.116 ** 2871 printk messages dropped **
Nov 19 11:05:02 10.243.0.116 [  111.754181] RIP: 0010:format_decode+0x38/0x5c0
Nov 19 11:05:02 10.243.0.116 ** 2850 printk messages dropped **
Nov 19 11:05:02 10.243.0.116 [  111.775220]  __bad_area_nosemaphore+0x247/0x2b0
Nov 19 11:05:02 10.243.0.116 ** 2826 printk messages dropped **
Nov 19 11:05:02 10.243.0.116 [  111.796007]  ? print_modules+0x88/0x121
Nov 19 11:05:02 10.243.0.116 ** 3032 printk messages dropped **
Nov 19 11:05:02 10.243.0.116 [  111.818461]  ? print_modules+0x88/0x121
Nov 19 11:05:02 10.243.0.116 ** 2951 printk messages dropped **
Nov 19 11:05:02 10.243.0.116 [  111.840208] RBP: fffffe000016e260 R08: ffffed107bc94f64 R09: ffffed107bc94f64
Nov 19 11:05:02 10.243.0.116 ** 2930 printk messages dropped **
Nov 19 11:05:02 10.243.0.116 [  111.861910] R10: ffff8883de4a7b1b R11: ffffed107bc94f63 R12: 1fffffc00002df49
Nov 19 11:05:02 10.243.0.116 ** 2750 printk messages dropped **
Nov 19 11:05:02 10.243.0.116 [  111.881442] RBP: fffffe000016f0f0 R08: ffffed107bc94f64 R09: ffffed107bc94f64
Nov 19 11:05:02 10.243.0.116 ** 2702 printk messages dropped **
Nov 19 11:05:02 10.243.0.116 [  111.900716]  ? print_modules+0x88/0x121
Nov 19 11:05:03 10.243.0.116 ** 2632 printk messages dropped **
Nov 19 11:05:03 10.243.0.116 [  111.919390] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Nov 19 11:05:03 10.243.0.116 ** 2711 printk messages dropped **
Nov 19 11:05:03 10.243.0.116 [  111.938557]  __die_body+0x1f/0x60
Nov 19 11:05:03 10.243.0.116 ** 2712 printk messages dropped **
Nov 19 11:05:03 10.243.0.116 [  111.957855]  ? bsearch+0x5e/0x90
Nov 19 11:05:03 10.243.0.116 ** 2667 printk messages dropped **
Nov 19 11:05:03 10.243.0.116 [  111.976807]  __die_body+0x1f/0x60
Nov 19 11:05:03 10.243.0.116 ** 2654 printk messages dropped **
Nov 19 11:05:03 10.243.0.116 [  111.995754] Hardware name: Supermicro Super Server/X10SDV-6C+-TLN4F, BIOS 2.1 11/22/2019
Nov 19 11:05:03 10.243.0.116 ** 2613 printk messages dropped **
Nov 19 11:05:03 10.243.0.116 [  112.014442] R10: fffffe000016c447 R11: fffffbc00002d888 R12: fffffe000016c1b0
Nov 19 11:05:03 10.243.0.116 ** 2583 printk messages dropped **
Nov 19 11:05:03 10.243.0.116 [  112.032807] RIP: 0010:print_modules+0x88/0x121
Nov 19 11:05:03 10.243.0.116 ** 2679 printk messages dropped **
Nov 19 11:05:03 10.243.0.116 [  112.051785] R10: ffff8883de4a7b1b R11: ffffed107bc94f63 R12: 1fffffc00002df13
Nov 19 11:05:03 10.243.0.116 ** 2658 printk messages dropped **
Nov 19 11:05:03 10.243.0.116 [  112.070766] Code: b1 ff ff 65 ff 05 94 13 f4 76 48 8b 1d 8d a7 0c 01 48 83 eb 08 4c 8d 6b 08 49 81 fd 40 03 1b 8a 74 3e 48 89 df e8 04 7d 3f ff <83> 3b 03 74 1f 48 8d 75 a8 48 89 df e8 93 a3 14 ff 48 8d 73 18 48
Nov 19 11:05:03 10.243.0.116 ** 4278 printk messages dropped **
Nov 19 11:05:03 10.243.0.116 [  112.101133] Oops: 0000 [#595] SMP KASAN PTI IRQ_PIPELINE
Nov 19 11:05:03 10.243.0.116 ** 2938 printk messages dropped **
Nov 19 11:05:03 10.243.0.116 [  112.121944] RSP: 0018:fffffe000016ec68 EFLAGS: 00010046
Nov 19 11:05:03 10.243.0.116 ** 2941 printk messages dropped **
Nov 19 11:05:03 10.243.0.116 [  112.142870]  ? spurious_kernel_fault+0x200/0x200
Nov 19 11:05:03 10.243.0.116 ** 2999 printk messages dropped **
Nov 19 11:05:03 10.243.0.116 [  112.164148] RSP: 0018:fffffe000016ff10 EFLAGS: 00010046
Nov 19 11:05:03 10.243.0.116 ** 3044 printk messages dropped **
Nov 19 11:05:03 10.243.0.116 [  112.185763]  ? cmp_ex_search+0x1e/0x40
Nov 19 11:05:03 10.243.0.116 ** 3032 printk messages dropped **
Nov 19 11:05:03 10.243.0.116 [  112.207408] RBP: fffffe000016e0b0 R08: ffffed107bc94f64 R09: ffffed107bc94f64
Nov 19 11:05:03 10.243.0.116 ** 3114 printk messages dropped **
Nov 19 11:05:03 10.243.0.116 [  112.229353]  ? print_modules+0x88/0x121
Nov 19 11:05:03 10.243.0.116 ** 3062 printk messages dropped **
Nov 19 11:05:03 10.243.0.116 [  112.250999] RSP: 0018:fffffe000016e858 EFLAGS: 00010046
Nov 19 11:05:03 10.243.0.116 ** 3051 printk messages dropped **
Nov 19 11:05:03 10.243.0.116 [  112.272551]  kernelmode_fixup_or_oops+0x135/0x160
Nov 19 11:05:03 10.243.0.116 ** 3056 printk messages dropped **
Nov 19 11:05:03 10.243.0.116 [  112.294273] Code: b1 ff ff 65 ff 05 94 13 f4 76 48 8b 1d 8d a7 0c 01 48 83 eb 08 4c 8d 6b 08 49 81 fd 40 03 1b 8a 74 3e 48 89 df e8 04 7d 3f ff <83> 3b 03 74 1f 48 8d 75 a8 48 89 df e8 93 a3 14 ff 48 8d 73 18 48
Nov 19 11:05:03 10.243.0.116 ** 4808 printk messages dropped **
Nov 19 11:05:03 10.243.0.116 [  112.328274] R10: fffffe000016c447 R11: fffffbc00002d888 R12: fffffe000016c1b0
Nov 19 11:05:03 10.243.0.116 ** 3251 printk messages dropped **
Nov 19 11:05:03 10.243.0.116 [  112.351288]  ? cmp_ex_search+0x1e/0x40
Nov 19 11:05:03 10.243.0.116 ** 3238 printk messages dropped **
Nov 19 11:05:03 10.243.0.116 [  112.374322]  ? print_modules+0x88/0x121
Nov 19 11:05:03 10.243.0.116 ** 3110 printk messages dropped **
Nov 19 11:05:03 10.243.0.116 [  112.396569] RAX: 0000000000000000 RBX: fffffffffffffff8 RCX: ffffffff890e5bcc
Nov 19 11:05:03 10.243.0.116 ** 3165 printk messages dropped **
Nov 19 11:05:03 10.243.0.116 [  112.419064]  ? kallsyms_lookup_buildid+0x9/0x170
Nov 19 11:05:03 10.243.0.116 ** 3180 printk messages dropped **
Nov 19 11:05:03 10.243.0.116 [  112.441614]  ? print_modules+0x88/0x121
Nov 19 11:05:03 10.243.0.116 ** 3119 printk messages dropped **
Nov 19 11:05:03 10.243.0.116 [  112.463774]  __die+0x25/0x30
Nov 19 11:05:03 10.243.0.116 ** 3220 printk messages dropped **
Nov 19 11:05:03 10.243.0.116 [  112.486671] BUG: unable to handle page fault for address: fffffffffffffff8
Nov 19 11:05:03 10.243.0.116 ** 3152 printk messages dropped **
Nov 19 11:05:03 10.243.0.116 [  112.508794]  ? page_fault_oops+0x3c0/0x3c0
Nov 19 11:05:03 10.243.0.116 ** 3152 printk messages dropped **
Nov 19 11:05:03 10.243.0.116 [  112.531235] RSP: 0018:fffffe000016f078 EFLAGS: 00010046
Nov 19 11:05:03 10.243.0.116 ** 3091 printk messages dropped **
Nov 19 11:05:03 10.243.0.116 [  112.553257]  ? kallsyms_lookup_buildid+0x9/0x170
Nov 19 11:05:03 10.243.0.116 ** 3072 printk messages dropped **
Nov 19 11:05:03 10.243.0.116 [  112.575168] R10: ffff8883de4a7b1b R11: ffffed107bc94f63 R12: 1fffffc00002dc89
Nov 19 11:05:03 10.243.0.116 ** 3095 printk messages dropped **
Nov 19 11:05:03 10.243.0.116 [  112.597221] RDX: 0000000000000003 RSI: dffffc0000000000 RDI: fffffffffffffff8
Nov 19 11:05:03 10.243.0.116 ** 3020 printk messages dropped **
Nov 19 11:05:03 10.243.0.116 [  112.618727]  ? print_modules+0x88/0x121
Nov 19 11:05:03 10.243.0.116 ** 2968 printk messages dropped **
Nov 19 11:05:03 10.243.0.116 [  112.639688]  ? spurious_kernel_fault+0x200/0x200
Nov 19 11:05:03 10.243.0.116 ** 2957 printk messages dropped **
Nov 19 11:05:03 10.243.0.116 [  112.660824]  __die+0x25/0x30
Nov 19 11:05:03 10.243.0.116 ** 2897 printk messages dropped **
Nov 19 11:05:03 10.243.0.116 [  112.681234]  ? spurious_kernel_fault+0x200/0x200
Nov 19 11:05:03 10.243.0.116 ** 2886 printk messages dropped **
Nov 19 11:05:03 10.243.0.116 [  112.701769]  ? print_modules+0x88/0x121
Nov 19 11:05:03 10.243.0.116 ** 2872 printk messages dropped **
Nov 19 11:05:03 10.243.0.116 [  112.722133]  ? cmp_ex_sort+0x50/0x50
Nov 19 11:05:03 10.243.0.116 ** 2860 printk messages dropped **
Nov 19 11:05:03 10.243.0.116 [  112.742609]  ? bsearch+0x5e/0x90
Nov 19 11:05:03 10.243.0.116 ** 2776 printk messages dropped **
Nov 19 11:05:03 10.243.0.116 [  112.762421]  asm_exc_page_fault+0x27/0x30
Nov 19 11:05:03 10.243.0.116 ** 2801 printk messages dropped **
Nov 19 11:05:03 10.243.0.116 [  112.782342]  __bad_area_nosemaphore+0x247/0x2b0
Nov 19 11:05:03 10.243.0.116 ** 2765 printk messages dropped **
Nov 19 11:05:03 10.243.0.116 [  112.801992]  exc_page_fault+0xcf/0xe0
Nov 19 11:05:03 10.243.0.116 ** 2703 printk messages dropped **
Nov 19 11:05:03 10.243.0.116 [  112.821245]  __bad_area_nosemaphore+0x247/0x2b0
Nov 19 11:05:03 10.243.0.116 ** 2598 printk messages dropped **
Nov 19 11:05:03 10.243.0.116 [  112.839693] BUG: unable to handle page fault for address: fffffffffffffff8
Nov 19 11:05:03 10.243.0.116 ** 2597 printk messages dropped **
Nov 19 11:05:03 10.243.0.116 [  112.858178] RSP: 0018:fffffe000016ff10 EFLAGS: 00010046
Nov 19 11:05:03 10.243.0.116 ** 2521 printk messages dropped **
Nov 19 11:05:03 10.243.0.116 [  112.876132] RDX: 0000000000000003 RSI: dffffc0000000000 RDI: fffffffffffffff8
Nov 19 11:05:03 10.243.0.116 ** 2658 printk messages dropped **
Nov 19 11:05:03 10.243.0.116 [  112.895034] RSP: 0018:fffffe000016ff10 EFLAGS: 00010046
Nov 19 11:05:03 10.243.0.116 ** 2611 printk messages dropped **
Nov 19 11:05:03 10.243.0.116 [  112.913512] RAX: 0000000000000000 RBX: fffffffffffffff8 RCX: ffffffff890e5bcc
Nov 19 11:05:04 10.243.0.116 ** 2687 printk messages dropped **
Nov 19 11:05:04 10.243.0.116 [  112.932577]  ? cmp_ex_search+0x1e/0x40
Nov 19 11:05:04 10.243.0.116 ** 2665 printk messages dropped **
Nov 19 11:05:04 10.243.0.116 [  112.951541] BUG: unable to handle page fault for address: fffffffffffffff8
Nov 19 11:05:04 10.243.0.116 ** 2742 printk messages dropped **
Nov 19 11:05:04 10.243.0.116 [  112.971047]  ? print_modules+0x88/0x121
Nov 19 11:05:04 10.243.0.116 ** 2706 printk messages dropped **
Nov 19 11:05:04 10.243.0.116 [  112.990240] RSP: 0018:fffffe000016ff10 EFLAGS: 00010046
Nov 19 11:05:04 10.243.0.116 ** 2717 printk messages dropped **
Nov 19 11:05:04 10.243.0.116 [  113.009578]  ? load_module.cold.80+0x38d/0x38d
Nov 19 11:05:04 10.243.0.116 ** 2723 printk messages dropped **
Nov 19 11:05:04 10.243.0.116 [  113.029003]  kernelmode_fixup_or_oops+0x135/0x160
Nov 19 11:05:04 10.243.0.116 ** 2702 printk messages dropped **
Nov 19 11:05:04 10.243.0.116 [  113.048235] RAX: 0000000000f00000 RBX: fffffe000016ff00 RCX: fffffe000016c000
Nov 19 11:05:04 10.243.0.116 ** 2693 printk messages dropped **
Nov 19 11:05:04 10.243.0.116 [  113.067387] #PF: supervisor read access in kernel mode
Nov 19 11:05:04 10.243.0.116 ** 2679 printk messages dropped **
Nov 19 11:05:04 10.243.0.116 [  113.086291]  handle_invalid_op+0x3c/0x50
Nov 19 11:05:04 10.243.0.116 ** 2676 printk messages dropped **
Nov 19 11:05:04 10.243.0.116 [  113.105305]  ? cmp_ex_sort+0x50/0x50
Nov 19 11:05:04 10.243.0.116 ** 2671 printk messages dropped **
Nov 19 11:05:04 10.243.0.116 [  113.124466] BUG: unable to handle page fault for address: fffffffffffffff8
Nov 19 11:05:04 10.243.0.116 ** 2626 printk messages dropped **
Nov 19 11:05:04 10.243.0.116 [  113.143121] RBP: fffffe000016e0b0 R08: ffffed107bc94f64 R09: ffffed107bc94f64
Nov 19 11:05:04 10.243.0.116 ** 2769 printk messages dropped **
Nov 19 11:05:04 10.243.0.116 [  113.162675]  __die_body+0x1f/0x60
Nov 19 11:05:04 10.243.0.116 ** 2799 printk messages dropped **
Nov 19 11:05:04 10.243.0.116 [  113.182597] R10: ffff8883de4a7b1b R11: ffffed107bc94f63 R12: 1fffffc00002df13
Nov 19 11:05:04 10.243.0.116 ** 2811 printk messages dropped **
Nov 19 11:05:04 10.243.0.116 [  113.202635]  ? print_modules+0x88/0x121
Nov 19 11:05:04 10.243.0.116 ** 2846 printk messages dropped **
Nov 19 11:05:04 10.243.0.116 [  113.222710] R13: 0000000000000000 R14: 0000000000000000 R15: ffff88811d198000
Nov 19 11:05:04 10.243.0.116 ** 2960 printk messages dropped **
Nov 19 11:05:04 10.243.0.116 [  113.243832] R13: 0000000000000000 R14: 0000000000000000 R15: ffff88811d198000
Nov 19 11:05:04 10.243.0.116 ** 2962 printk messages dropped **
Nov 19 11:05:04 10.243.0.116 [  113.264943]  ? print_modules+0x88/0x121
Nov 19 11:05:04 10.243.0.116 ** 3009 printk messages dropped **
Nov 19 11:05:04 10.243.0.116 [  113.286491] RSP: 0018:fffffe000016d408 EFLAGS: 00010046
Nov 19 11:05:04 10.243.0.116 ** 2966 printk messages dropped **
Nov 19 11:05:04 10.243.0.116 [  113.307454]  ? print_modules+0x88/0x121
Nov 19 11:05:04 10.243.0.116 ** 2959 printk messages dropped **
Nov 19 11:05:04 10.243.0.116 [  113.328493] R13: 0000000000000000 R14: 0000000000000000 R15: ffff88811d198000
Nov 19 11:05:04 10.243.0.116 ** 2988 printk messages dropped **
Nov 19 11:05:04 10.243.0.116 [  113.349712] RSP: 0018:fffffe000016f078 EFLAGS: 00010046
Nov 19 11:05:04 10.243.0.116 ** 2993 printk messages dropped **
Nov 19 11:05:04 10.243.0.116 [  113.371071] RSP: 0018:fffffe000016fca0 EFLAGS: 00010046
Nov 19 11:05:04 10.243.0.116 ** 2934 printk messages dropped **
Nov 19 11:05:04 10.243.0.116 [  113.392004] BUG: unable to handle page fault for address: fffffffffffffff8
Nov 19 11:05:04 10.243.0.116 ** 2949 printk messages dropped **
Nov 19 11:05:04 10.243.0.116 [  113.412866] RSP: 0018:fffffe000016fca0 EFLAGS: 00010046
Nov 19 11:05:04 10.243.0.116 ** 2947 printk messages dropped **
Nov 19 11:05:04 10.243.0.116 [  113.433748] Hardware name: Supermicro Super Server/X10SDV-6C+-TLN4F, BIOS 2.1 11/22/2019
Nov 19 11:05:04 10.243.0.116 ** 2976 printk messages dropped **
Nov 19 11:05:04 10.243.0.116 [  113.454933] BUG: unable to handle page fault for address: fffffffffffffff8
Nov 19 11:05:04 10.243.0.116 ** 3038 printk messages dropped **
Nov 19 11:05:04 10.243.0.116 [  113.476370] Call Trace:
Nov 19 11:05:04 10.243.0.116 ** 3047 printk messages dropped **
Nov 19 11:05:04 10.243.0.116 [  113.497930] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Nov 19 11:05:04 10.243.0.116 ** 3040 printk messages dropped **
Nov 19 11:05:04 10.243.0.116 [  113.519492] RIP: 0010:irqentry_nmi_enter+0x83/0x90
Nov 19 11:05:04 10.243.0.116 ** 3088 printk messages dropped **
Nov 19 11:05:04 10.243.0.116 [  113.541452]  ? fixup_exception+0x32/0x420
Nov 19 11:05:04 10.243.0.116 ** 3041 printk messages dropped **
Nov 19 11:05:04 10.243.0.116 [  113.562986]  ? bsearch+0x5e/0x90
Nov 19 11:05:04 10.243.0.116 ** 2983 printk messages dropped **
Nov 19 11:05:04 10.243.0.116 [  113.584192]  ? print_modules+0x88/0x121
Nov 19 11:05:04 10.243.0.116 ** 2960 printk messages dropped **
Nov 19 11:05:04 10.243.0.116 [  113.605183] Modules linked in: xt_CHECKSUM xt_MASQUERADE tun netconsole ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 xt_conntrack ebtable_nat ebtable_broute ip6table_nat ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_nat iptable_mangle iptable_security iptable_raw nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 libcrc32c ip_set nfnetlink ebtable_filter ebtables â–’â–’#023â–’â–’â–’â–’â–’â–’tâ–’#034â–’â–’â–’â–’â–’tâ–’#034â–’â–’â–’â–’â–’b#023â–’â–’â–’â–’â–’(FOKX)
Nov 19 11:05:04 10.243.0.116 ** 7880 printk messages dropped **
Nov 19 11:05:04 10.243.0.116 [  113.661029] RDX: 0000000000000003 RSI: dffffc0000000000 RDI: fffffffffffffff8
Nov 19 11:05:04 10.243.0.116 ** 3219 printk messages dropped **
Nov 19 11:05:04 10.243.0.116 [  113.683830] R13: fffffe000016c66a R14: ffffffff890e5bcc R15: fffffe000016c0c0
Nov 19 11:05:04 10.243.0.116 ** 3307 printk messages dropped **
Nov 19 11:05:04 10.243.0.116 [  113.707368]  ? print_modules+0x88/0x121
Nov 19 11:05:04 10.243.0.116 ** 3259 printk messages dropped **
Nov 19 11:05:04 10.243.0.116 [  113.730724] RSP: 0018:fffffe000016dc28 EFLAGS: 00010046
Nov 19 11:05:04 10.243.0.116 ** 3257 printk messages dropped **
Nov 19 11:05:04 10.243.0.116 [  113.753810] R13: 0000000000000000 R14: 0000000000000000 R15: ffff88811d198000
Nov 19 11:05:04 10.243.0.116 ** 3275 printk messages dropped **
Nov 19 11:05:04 10.243.0.116 [  113.777306] RSP: 0018:fffffe000016dc28 EFLAGS: 00010046
Nov 19 11:05:04 10.243.0.116 ** 3252 printk messages dropped **
Nov 19 11:05:04 10.243.0.116 [  113.800323] R13: 0000000000000000 R14: 0000000000000000 R15: ffff88811d198000
Nov 19 11:05:04 10.243.0.116 ** 3360 printk messages dropped **
Nov 19 11:05:04 10.243.0.116 [  113.824208] BUG: unable to handle page fault for address: fffffffffffffff8
Nov 19 11:05:04 10.243.0.116 ** 3334 printk messages dropped **
Nov 19 11:05:04 10.243.0.116 [  113.847831]  kernelmode_fixup_or_oops+0x135/0x160
Nov 19 11:05:04 10.243.0.116 ** 3356 printk messages dropped **
Nov 19 11:05:04 10.243.0.116 [  113.871590]  page_fault_oops+0x124/0x3c0
Nov 19 11:05:04 10.243.0.116 ** 3288 printk messages dropped **
Nov 19 11:05:04 10.243.0.116 [  113.895042] RSP: 0018:fffffe000016ff10 EFLAGS: 00010046
Nov 19 11:05:05 10.243.0.116 ** 3260 printk messages dropped **
Nov 19 11:05:05 10.243.0.116 [  113.918243]  __die_body+0x1f/0x60
Nov 19 11:05:05 10.243.0.116 ** 3272 printk messages dropped **
Nov 19 11:05:05 10.243.0.116 [  113.941486] RDX: 0000000000000003 RSI: dffffc0000000000 RDI: fffffffffffffff8
Nov 19 11:05:05 10.243.0.116 ** 3229 printk messages dropped **
Nov 19 11:05:05 10.243.0.116 [  113.964559] BUG: unable to handle page fault for address: fffffffffffffff8
Nov 19 11:05:05 10.243.0.116 ** 3313 printk messages dropped **
Nov 19 11:05:05 10.243.0.116 [  113.988073] BUG: unable to handle page fault for address: fffffffffffffff8
Nov 19 11:05:05 10.243.0.116 ** 3212 printk messages dropped **
Nov 19 11:05:05 10.243.0.116 [  114.010762] CPU: 5 PID: 1563 Comm: DummyTxSample Tainted: G        W         5.15.77evl-g8390add2f766 #3
Nov 19 11:05:05 10.243.0.116 ** 3297 printk messages dropped **
Nov 19 11:05:05 10.243.0.116 [  114.034199]  ? cmp_ex_sort+0x50/0x50
Nov 19 11:05:05 10.243.0.116 ** 3313 printk messages dropped **
Nov 19 11:05:05 10.243.0.116 [  114.057815] Code: b1 ff ff 65 ff 05 94 13 f4 76 48 8b 1d 8d a7 0c 01 48 83 eb 08 4c 8d 6b 08 49 81 fd 40 03 1b 8a 74 3e 48 89 df e8 04 7d 3f ff <83> 3b 03 74 1f 48 8d 75 a8 48 89 df e8 93 a3 14 ff 48 8d 73 18 48
Nov 19 11:05:05 10.243.0.116 ** 5069 printk messages dropped **
Nov 19 11:05:05 10.243.0.116 [  114.093725]  __bad_area_nosemaphore+0x247/0x2b0
Nov 19 11:05:05 10.243.0.116 ** 3438 printk messages dropped **
Nov 19 11:05:05 10.243.0.116 [  114.117993]  ? print_modules+0x88/0x121
Nov 19 11:05:05 10.243.0.116 ** 3429 printk messages dropped **
Nov 19 11:05:05 10.243.0.116 [  114.142381] RSP: 0018:fffffe000016e448 EFLAGS: 00010046
Nov 19 11:05:05 10.243.0.116 ** 3482 printk messages dropped **
Nov 19 11:05:05 10.243.0.116 [  114.167381]  ? print_modules+0x88/0x121
Nov 19 11:05:05 10.243.0.116 ** 3332 printk messages dropped **
Nov 19 11:05:05 10.243.0.116 [  114.191087] RSP: 0018:fffffe000016d408 EFLAGS: 00010046
Nov 19 11:05:05 10.243.0.116 ** 3261 printk messages dropped **
Nov 19 11:05:05 10.243.0.116 [  114.214266]  ? print_modules+0x88/0x121
Nov 19 11:05:05 10.243.0.116 ** 3109 printk messages dropped **
Nov 19 11:05:05 10.243.0.116 [  114.236211] Call Trace:
Nov 19 11:05:05 10.243.0.116 ** 3056 printk messages dropped **
Nov 19 11:05:05 10.243.0.116 [  114.257889]  ? lock_release+0xd5/0x420
Nov 19 11:05:05 10.243.0.116 ** 3024 printk messages dropped **
Nov 19 11:05:05 10.243.0.116 [  114.279406]  ? search_bpf_extables+0x136/0x190
Nov 19 11:05:05 10.243.0.116 ** 3018 printk messages dropped **
Nov 19 11:05:05 10.243.0.116 [  114.300860]  ? irqentry_nmi_enter+0x83/0x90
Nov 19 11:05:05 10.243.0.116 ** 2973 printk messages dropped **
Nov 19 11:05:05 10.243.0.116 [  114.321983] RBP: fffffe000016ece0 R08: ffffed107bc94f64 R09: ffffed107bc94f64
Nov 19 11:05:05 10.243.0.116 ** 3401 printk messages dropped **
Nov 19 11:05:05 10.243.0.116 [  114.346007] RDX: 0000000000000003 RSI: dffffc0000000000 RDI: fffffffffffffff8
Nov 19 11:05:05 10.243.0.116 ** 2996 printk messages dropped **
Nov 19 11:05:05 10.243.0.116 [  114.367282] R13: 0000000000000000 R14: 0000000000000000 R15: ffff88811d198000
Nov 19 11:05:05 10.243.0.116 ** 2986 printk messages dropped **
Nov 19 11:05:05 10.243.0.116 [  114.388519] RAX: 0000000000000000 RBX: 1fffffc00002d804 RCX: fffffe000016c040
Nov 19 11:05:05 10.243.0.116 ** 2957 printk messages dropped **
Nov 19 11:05:05 10.243.0.116 [  114.409618] R13: 0000000000000000 R14: 0000000000000000 R15: ffff88811d198000
Nov 19 11:05:05 10.243.0.116 ** 3092 printk messages dropped **
Nov 19 11:05:05 10.243.0.116 [  114.431683] R13: 0000000000000000 R14: 0000000000000000 R15: ffff88811d198000
Nov 19 11:05:05 10.243.0.116 ** 3110 printk messages dropped **
Nov 19 11:05:05 10.243.0.116 [  114.453777] RSP: 0018:fffffe000016e858 EFLAGS: 00010046
Nov 19 11:05:05 10.243.0.116 ** 3147 printk messages dropped **
Nov 19 11:05:05 10.243.0.116 [  114.476124] Code: b1 ff ff 65 ff 05 94 13 f4 76 48 8b 1d 8d a7 0c 01 48 83 eb 08 4c 8d 6b 08 49 81 fd 40 03 1b 8a 74 3e 48 89 df e8 04 7d 3f ff <83> 3b 03 74 1f 48 8d 75 a8 48 89 df e8 93 a3 14 ff 48 8d 73 18 48
Nov 19 11:05:05 10.243.0.116 ** 4918 printk messages dropped **
Nov 19 11:05:05 10.243.0.116 [  114.510869]  ? print_modules+0x88/0x121
Nov 19 11:05:05 10.243.0.116 ** 3267 printk messages dropped **
Nov 19 11:05:05 10.243.0.116 [  114.534169]  ? kallsyms_lookup_buildid+0x9/0x170
Nov 19 11:05:05 10.243.0.116 ** 3190 printk messages dropped **
Nov 19 11:05:05 10.243.0.116 [  114.556829]  ? is_prefetch.isra.28+0x8c/0x2b0
Nov 19 11:05:05 10.243.0.116 ** 3103 printk messages dropped **
Nov 19 11:05:05 10.243.0.116 [  114.579027]  ? print_modules+0x88/0x121
Nov 19 11:05:05 10.243.0.116 ** 3123 printk messages dropped **
Nov 19 11:05:05 10.243.0.116 [  114.601269] R10: ffff8883de4a7b1b R11: ffffed107bc94f63 R12: 1fffffc00002de91
Nov 19 11:05:05 10.243.0.116 ** 2994 printk messages dropped **
Nov 19 11:05:05 10.243.0.116 [  114.622582]  </#DF>
Nov 19 11:05:05 10.243.0.116 ** 2950 printk messages dropped **
Nov 19 11:05:05 10.243.0.116 [  114.643616] R13: 0000000000000000 R14: 0000000000000000 R15: ffff88811d198000
Nov 19 11:05:05 10.243.0.116 ** 2965 printk messages dropped **
Nov 19 11:05:05 10.243.0.116 [  114.664743]  __die_body+0x1f/0x60
Nov 19 11:05:05 10.243.0.116 ** 3049 printk messages dropped **
Nov 19 11:05:05 10.243.0.116 [  114.686558] RSP: 0018:fffffe000016f078 EFLAGS: 00010046
Nov 19 11:05:05 10.243.0.116 ** 3013 printk messages dropped **
Nov 19 11:05:05 10.243.0.116 [  114.707894] RSP: 0018:fffffe000016ff10 EFLAGS: 00010046
Nov 19 11:05:05 10.243.0.116 ** 2977 printk messages dropped **
Nov 19 11:05:05 10.243.0.116 [  114.728992] CR2: fffffffffffffff8 CR3: 0000000102568002 CR4: 00000000003706e0
Nov 19 11:05:05 10.243.0.116 ** 3032 printk messages dropped **
Nov 19 11:05:05 10.243.0.116 [  114.750679] RSP: 0018:fffffe000016f488 EFLAGS: 00010046
Nov 19 11:05:05 10.243.0.116 ** 3111 printk messages dropped **
Nov 19 11:05:05 10.243.0.116 [  114.772935]  ? fixup_exception+0x32/0x420
Nov 19 11:05:05 10.243.0.116 ** 3126 printk messages dropped **
Nov 19 11:05:05 10.243.0.116 [  114.795159] R13: fffffe000016c66a R14: ffffffff890e5bcc R15: fffffe000016c0c0
Nov 19 11:05:05 10.243.0.116 ** 3151 printk messages dropped **
Nov 19 11:05:05 10.243.0.116 [  114.817580]  ? cmp_ex_search+0x1e/0x40
Nov 19 11:05:05 10.243.0.116 ** 3199 printk messages dropped **
Nov 19 11:05:05 10.243.0.116 [  114.840231] RSP: 0018:fffffe000016e858 EFLAGS: 00010046
Nov 19 11:05:05 10.243.0.116 ** 3151 printk messages dropped **
Nov 19 11:05:05 10.243.0.116 [  114.862704] RBP: fffffe000016e0b0 R08: ffffed107bc94f64 R09: ffffed107bc94f64
Nov 19 11:05:05 10.243.0.116 ** 3155 printk messages dropped **
Nov 19 11:05:05 10.243.0.116 [  114.885217] RBP: fffffe000016c008 R08: fffffe000016c080 R09: fffffe000016c1b0
Nov 19 11:05:05 10.243.0.116 ** 3186 printk messages dropped **
Nov 19 11:05:05 10.243.0.116 [  114.907999] R13: 0000000000000000 R14: 0000000000000000 R15: ffff88811d198000
Nov 19 11:05:06 10.243.0.116 ** 3148 printk messages dropped **
Nov 19 11:05:06 10.243.0.116 [  114.930370] R13: 0000000000000000 R14: 0000000000000000 R15: ffff88811d198000
Nov 19 11:05:06 10.243.0.116 ** 3148 printk messages dropped **
Nov 19 11:05:06 10.243.0.116 [  114.952871] R13: 0000000000000000 R14: 0000000000000000 R15: ffff88811d198000
Nov 19 11:05:06 10.243.0.116 ** 3242 printk messages dropped **
Nov 19 11:05:06 10.243.0.116 [  114.975993] RBP: fffffe000016ff20 R08: 0000000000000000 R09: 0000000000000000
Nov 19 11:05:06 10.243.0.116 ** 3123 printk messages dropped **
Nov 19 11:05:06 10.243.0.116 [  114.998016] Call Trace:
Nov 19 11:05:06 10.243.0.116 ** 3119 printk messages dropped **
Nov 19 11:05:06 10.243.0.116 [  115.020081]  die+0x38/0x60
Nov 19 11:05:06 10.243.0.116 ** 2980 printk messages dropped **
Nov 19 11:05:06 10.243.0.116 [  115.041175]  ? print_modules+0x88/0x121
Nov 19 11:05:06 10.243.0.116 ** 2969 printk messages dropped **
Nov 19 11:05:06 10.243.0.116 [  115.062322]  ? bsearch+0x5e/0x90
Nov 19 11:05:06 10.243.0.116 ** 2928 printk messages dropped **
Nov 19 11:05:06 10.243.0.116 [  115.083075]  asm_exc_invalid_op+0x1b/0x20
Nov 19 11:05:06 10.243.0.116 ** 2934 printk messages dropped **
Nov 19 11:05:06 10.243.0.116 [  115.103944] FS:  00007fdce1ffd700(0000) GS:ffff8883de480000(0000) knlGS:0000000000000000
Nov 19 11:05:06 10.243.0.116 ** 2986 printk messages dropped **
Nov 19 11:05:06 10.243.0.116 [  115.125168]  bad_area_nosemaphore+0x11/0x20
Nov 19 11:05:06 10.243.0.116 ** 2927 printk messages dropped **
Nov 19 11:05:06 10.243.0.116 [  115.145895]  bad_area_nosemaphore+0x11/0x20
Nov 19 11:05:06 10.243.0.116 ** 2955 printk messages dropped **
Nov 19 11:05:06 10.243.0.116 [  115.166794] RDX: 0000000000000000 RSI: 0000000000000000 RDI: fffffe000016ff58
Nov 19 11:05:06 10.243.0.116 ** 2943 printk messages dropped **
Nov 19 11:05:06 10.243.0.116 [  115.187730]  ? cmp_ex_search+0x1e/0x40
Nov 19 11:05:06 10.243.0.116 ** 3000 printk messages dropped **
Nov 19 11:05:06 10.243.0.116 [  115.209175]  ? print_modules+0x88/0x121
Nov 19 11:05:06 10.243.0.116 ** 2965 printk messages dropped **
Nov 19 11:05:06 10.243.0.116 [  115.230204]  bad_area_nosemaphore+0x11/0x20
Nov 19 11:05:06 10.243.0.116 ** 2921 printk messages dropped **
Nov 19 11:05:06 10.243.0.116 [  115.250899] RAX: 0000000000f00000 RBX: fffffe000016ff00 RCX: fffffe000016c000
Nov 19 11:05:06 10.243.0.116 ** 2913 printk messages dropped **
Nov 19 11:05:06 10.243.0.116 [  115.271475] BUG: unable to handle page fault for address: fffffffffffffff8
Nov 19 11:05:06 10.243.0.116 ** 2915 printk messages dropped **
Nov 19 11:05:06 10.243.0.116 [  115.291998] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Nov 19 11:05:06 10.243.0.116 ** 3037 printk messages dropped **
Nov 19 11:05:06 10.243.0.116 [  115.313646]  ? bsearch+0x5e/0x90
Nov 19 11:05:06 10.243.0.116 ** 2956 printk messages dropped **
Nov 19 11:05:06 10.243.0.116 [  115.334619]  page_fault_oops+0x124/0x3c0
Nov 19 11:05:06 10.243.0.116 ** 2947 printk messages dropped **
Nov 19 11:05:06 10.243.0.116 [  115.355550] RDX: fffffe000016c0c0 RSI: fffffe000016c100 RDI: ffffffff890e5bcc
Nov 19 11:05:06 10.243.0.116 ** 2958 printk messages dropped **
Nov 19 11:05:06 10.243.0.116 [  115.376506]  ? print_modules+0x88/0x121
Nov 19 11:05:06 10.243.0.116 ** 2957 printk messages dropped **
Nov 19 11:05:06 10.243.0.116 [  115.397599] R10: ffff8883de4a7b1b R11: ffffed107bc94f63 R12: 1fffffc00002dd0b
Nov 19 11:05:06 10.243.0.116 ** 2929 printk messages dropped **
Nov 19 11:05:06 10.243.0.116 [  115.418345]  ? print_modules+0x88/0x121
Nov 19 11:05:06 10.243.0.116 ** 2837 printk messages dropped **
Nov 19 11:05:06 10.243.0.116 [  115.438354]  ? cmp_ex_search+0x1e/0x40
Nov 19 11:05:06 10.243.0.116 ** 2797 printk messages dropped **
Nov 19 11:05:06 10.243.0.116 [  115.458255] BUG: unable to handle page fault for address: fffffffffffffff8
Nov 19 11:05:06 10.243.0.116 ** 2744 printk messages dropped **
Nov 19 11:05:06 10.243.0.116 [  115.477688]  ? print_modules+0x88/0x121
Nov 19 11:05:06 10.243.0.116 ** 2733 printk messages dropped **
Nov 19 11:05:06 10.243.0.116 [  115.497254]  exc_invalid_op+0x4f/0xa0
Nov 19 11:05:06 10.243.0.116 ** 2678 printk messages dropped **
Nov 19 11:05:06 10.243.0.116 [  115.516293] RBP: fffffe000016e4c0 R08: ffffed107bc94f64 R09: ffffed107bc94f64
Nov 19 11:05:06 10.243.0.116 ** 2764 printk messages dropped **
Nov 19 11:05:06 10.243.0.116 [  115.536044]  ? search_exception_tables+0x58/0x60
Nov 19 11:05:06 10.243.0.116 ** 2797 printk messages dropped **
Nov 19 11:05:06 10.243.0.116 [  115.555905] IRQ stage: Linux
Nov 19 11:05:06 10.243.0.116 ** 2818 printk messages dropped **
Nov 19 11:05:06 10.243.0.116 [  115.575854] R13: 0000000000000000 R14: 0000000000000000 R15: ffff88811d198000
Nov 19 11:05:06 10.243.0.116 ** 2829 printk messages dropped **
Nov 19 11:05:06 10.243.0.116 [  115.596067]  ? print_modules+0x88/0x121
Nov 19 11:05:06 10.243.0.116 ** 2810 printk messages dropped **
Nov 19 11:05:06 10.243.0.116 [  115.615992]  do_trap+0x126/0x150
Nov 19 11:05:06 10.243.0.116 ** 2790 printk messages dropped **
Nov 19 11:05:06 10.243.0.116 [  115.635923] Call Trace:
Nov 19 11:05:06 10.243.0.116 ** 2757 printk messages dropped **
Nov 19 11:05:06 10.243.0.116 [  115.655611] Hardware name: Supermicro Super Server/X10SDV-6C+-TLN4F, BIOS 2.1 11/22/2019
Nov 19 11:05:06 10.243.0.116 ** 2754 printk messages dropped **
Nov 19 11:05:06 10.243.0.116 [  115.675289] RDX: 0000000000000003 RSI: dffffc0000000000 RDI: fffffffffffffff8
Nov 19 11:05:06 10.243.0.116 ** 2827 printk messages dropped **
Nov 19 11:05:06 10.243.0.116 [  115.695412] RSP: 0018:fffffe000016c000 EFLAGS: 00010046
Nov 19 11:05:06 10.243.0.116 ** 3268 printk messages dropped **
Nov 19 11:05:06 10.243.0.116 [  115.718586] RBP: fffffe000016f500 R08: ffffed107bc94f64 R09: ffffed107bc94f64
Nov 19 11:05:06 10.243.0.116 ** 2920 printk messages dropped **
Nov 19 11:05:06 10.243.0.116 [  115.739374]  exc_page_fault+0xcf/0xe0
Nov 19 11:05:06 10.243.0.116 ** 2908 printk messages dropped **
Nov 19 11:05:06 10.243.0.116 [  115.760026]  __die_body+0x1f/0x60
Nov 19 11:05:06 10.243.0.116 ** 2802 printk messages dropped **
Nov 19 11:05:06 10.243.0.116 [  115.779910] BUG: unable to handle page fault for address: fffffffffffffff8
Nov 19 11:05:06 10.243.0.116 ** 2762 printk messages dropped **
Nov 19 11:05:06 10.243.0.116 [  115.799458] RSP: 0018:fffffe000016c000 EFLAGS: 00010046
Nov 19 11:05:06 10.243.0.116 ** 2804 printk messages dropped **
Nov 19 11:05:06 10.243.0.116 [  115.819285] Code: 4e 2a 00 8b 45 c4 c6 03 00 48 83 c4 18 5b 41 5c 41 5d 41 5e 41 5f 5d c3 cc cc cc cc 0f 1f 44 00 00 55 48 89 e5 41 57 49 89 d7 <41> 56 49 89 f6 41 55 49 89 cd 41 54 49 89 fc 49 8d 79 7f 53 4c 89
Nov 19 11:05:06 10.243.0.116 ** 4331 printk messages dropped **
Nov 19 11:05:06 10.243.0.116 [  115.849952]  do_kern_addr_fault+0x7c/0xb0
Nov 19 11:05:06 10.243.0.116 ** 2951 printk messages dropped **
Nov 19 11:05:06 10.243.0.116 [  115.870933]  ? page_fault_oops+0x3c0/0x3c0
Nov 19 11:05:06 10.243.0.116 ** 2947 printk messages dropped **
Nov 19 11:05:06 10.243.0.116 [  115.891917]  ? print_modules+0x88/0x121
Nov 19 11:05:06 10.243.0.116 ** 2866 printk messages dropped **
Nov 19 11:05:06 10.243.0.116 [  115.912342]  __die+0x25/0x30
Nov 19 11:05:07 10.243.0.116 ** 2887 printk messages dropped **
Nov 19 11:05:07 10.243.0.116 [  115.932731] R10: ffff8883de4a7b1b R11: ffffed107bc94f63 R12: 1fffffc00002de0f
Nov 19 11:05:07 10.243.0.116 ** 2894 printk messages dropped **
Nov 19 11:05:07 10.243.0.116 [  115.953192] R10: ffff8883de4a7b1b R11: ffffed107bc94f63 R12: 1fffffc00002de0f
Nov 19 11:05:07 10.243.0.116 ** 2913 printk messages dropped **
Nov 19 11:05:07 10.243.0.116 [  115.973850]  ? search_exception_tables+0x58/0x60
Nov 19 11:05:07 10.243.0.116 ** 2946 printk messages dropped **
Nov 19 11:05:07 10.243.0.116 [  115.994934]  ? kallsyms_lookup_buildid+0x9/0x170
Nov 19 11:05:07 10.243.0.116 ** 2926 printk messages dropped **
Nov 19 11:05:07 10.243.0.116 [  116.015684]  ? kallsyms_lookup_buildid+0x9/0x170
Nov 19 11:05:07 10.243.0.116 ** 2906 printk messages dropped **
Nov 19 11:05:07 10.243.0.116 [  116.036280] RSP: 0018:fffffe000016ff10 EFLAGS: 00010046
Nov 19 11:05:07 10.243.0.116 ** 2925 printk messages dropped **
Nov 19 11:05:07 10.243.0.116 [  116.057062] RSP: 0018:fffffe000016d818 EFLAGS: 00010046
Nov 19 11:05:07 10.243.0.116 ** 2938 printk messages dropped **
Nov 19 11:05:07 10.243.0.116 [  116.077920]  ? print_modules+0x88/0x121
Nov 19 11:05:07 10.243.0.116 ** 2925 printk messages dropped **
Nov 19 11:05:07 10.243.0.116 [  116.098752] R10: ffff8883de4a7b1b R11: ffffed107bc94f63 R12: 1fffffc00002dc07
Nov 19 11:05:07 10.243.0.116 ** 2950 printk messages dropped **
Nov 19 11:05:07 10.243.0.116 [  116.119722]  bad_area_nosemaphore+0x11/0x20
Nov 19 11:05:07 10.243.0.116 ** 3026 printk messages dropped **
Nov 19 11:05:07 10.243.0.116 [  116.141384]  bad_area_nosemaphore+0x11/0x20
Nov 19 11:05:07 10.243.0.116 ** 2984 printk messages dropped **
Nov 19 11:05:07 10.243.0.116 [  116.162672] RSP: 0018:fffffe000016d408 EFLAGS: 00010046
Nov 19 11:05:07 10.243.0.116 ** 2927 printk messages dropped **
Nov 19 11:05:07 10.243.0.116 [  116.183611]  do_kern_addr_fault+0x7c/0xb0
Nov 19 11:05:07 10.243.0.116 ** 2893 printk messages dropped **
Nov 19 11:05:07 10.243.0.116 [  116.204159]  bad_area_nosemaphore+0x11/0x20
Nov 19 11:05:07 10.243.0.116 ** 2847 printk messages dropped **
Nov 19 11:05:07 10.243.0.116 [  116.224287]  ? print_modules+0x88/0x121
Nov 19 11:05:07 10.243.0.116 ** 2816 printk messages dropped **
Nov 19 11:05:07 10.243.0.116 [  116.244401]  ? fixup_exception+0x32/0x420
Nov 19 11:05:07 10.243.0.116 ** 2799 printk messages dropped **
Nov 19 11:05:07 10.243.0.116 [  116.264242]  do_kern_addr_fault+0x7c/0xb0
Nov 19 11:05:07 10.243.0.116 ** 2776 printk messages dropped **
Nov 19 11:05:07 10.243.0.116 [  116.283931]  ? kallsyms_lookup_buildid+0x9/0x170
Nov 19 11:05:07 10.243.0.116 ** 2768 printk messages dropped **
Nov 19 11:05:07 10.243.0.116 [  116.303637] R13: 0000000000000000 R14: 0000000000000000 R15: ffff88811d198000
Nov 19 11:05:07 10.243.0.116 ** 2695 printk messages dropped **
Nov 19 11:05:07 10.243.0.116 [  116.322662] R10: ffff8883de4a7b1b R11: ffffed107bc94f63 R12: 1fffffc00002dc07
Nov 19 11:05:07 10.243.0.116 ** 2728 printk messages dropped **
Nov 19 11:05:07 10.243.0.116 [  116.342050]  __bad_area_nosemaphore+0x247/0x2b0
Nov 19 11:05:07 10.243.0.116 ** 2689 printk messages dropped **
Nov 19 11:05:07 10.243.0.116 [  116.361279]  __die_body+0x1f/0x60
Nov 19 11:05:07 10.243.0.116 ** 2671 printk messages dropped **
Nov 19 11:05:07 10.243.0.116 [  116.380296] BUG: unable to handle page fault for address: fffffffffffffff8
Nov 19 11:05:07 10.243.0.116 ** 2738 printk messages dropped **
Nov 19 11:05:07 10.243.0.116 [  116.399728]  ? bsearch+0x5e/0x90
Nov 19 11:05:07 10.243.0.116 ** 2756 printk messages dropped **
Nov 19 11:05:07 10.243.0.116 [  116.419419] RSP: 0018:fffffe000016f898 EFLAGS: 00010046
Nov 19 11:05:07 10.243.0.116 ** 2683 printk messages dropped **
Nov 19 11:05:07 10.243.0.116 [  116.438509]  __die_body+0x1f/0x60
Nov 19 11:05:07 10.243.0.116 ** 2688 printk messages dropped **
Nov 19 11:05:07 10.243.0.116 [  116.457615]  ? irqentry_nmi_enter+0x83/0x90
Nov 19 11:05:07 10.243.0.116 ** 2546 printk messages dropped **
Nov 19 11:05:07 10.243.0.116 [  116.475678]  ? print_modules+0x88/0x121
Nov 19 11:05:07 10.243.0.116 ** 2631 printk messages dropped **
Nov 19 11:05:07 10.243.0.116 [  116.494370]  ? irqentry_nmi_enter+0x83/0x90
Nov 19 11:05:07 10.243.0.116 ** 2557 printk messages dropped **
Nov 19 11:05:07 10.243.0.116 [  116.512592] Call Trace:
Nov 19 11:05:07 10.243.0.116 ** 2541 printk messages dropped **
Nov 19 11:05:07 10.243.0.116 [  116.530691] Call Trace:
Nov 19 11:05:07 10.243.0.116 ** 2537 printk messages dropped **
Nov 19 11:05:07 10.243.0.116 [  116.548781] R13: ffff888118d24780 R14: 0000000102568002 R15: 0000000000000000
Nov 19 11:05:07 10.243.0.116 ** 2540 printk messages dropped **
Nov 19 11:05:07 10.243.0.116 [  116.566751]  ? fixup_exception+0x32/0x420
Nov 19 11:05:07 10.243.0.116 ** 2505 printk messages dropped **
Nov 19 11:05:07 10.243.0.116 [  116.584704] Thread overran stack, or stack corrupted
Nov 19 11:05:07 10.243.0.116 ** 2526 printk messages dropped **
Nov 19 11:05:07 10.243.0.116 [  116.602653] Code: e0 ee 76 85 c0 75 0c 41 c7 84 24 b8 0d 00 00 00 00 00 00 e8 6f fe ff ff e8 3a 38 19 ff 31 c0 88 d8 5b 41 5c 5d c3 cc cc cc cc <0f> 0b 90 66 2e 0f 1f 84 00 00 00 00 00 55 48 89 e5 40 84 f6 75 3e
Nov 19 11:05:07 10.243.0.116 ** 3951 printk messages dropped **
Nov 19 11:05:07 10.243.0.116 [  116.630726] R10: ffff8883de4a7b1b R11: ffffed107bc94f63 R12: 1fffffc00002df13
Nov 19 11:05:07 10.243.0.116 ** 2702 printk messages dropped **
Nov 19 11:05:07 10.243.0.116 [  116.649819]  __die_body+0x1f/0x60
Nov 19 11:05:07 10.243.0.116 ** 2779 printk messages dropped **
Nov 19 11:05:07 10.243.0.116 [  116.669717]  ? cmp_ex_sort+0x50/0x50
Nov 19 11:05:07 10.243.0.116 ** 2729 printk messages dropped **
Nov 19 11:05:07 10.243.0.116 [  116.688983] BUG: unable to handle page fault for address: fffffffffffffff8
Nov 19 11:05:07 10.243.0.116 ** 2760 printk messages dropped **
Nov 19 11:05:07 10.243.0.116 [  116.708555] R13: 0000000000000000 R14: 0000000000000000 R15: ffff88811d198000
Nov 19 11:05:07 10.243.0.116 ** 2826 printk messages dropped **
Nov 19 11:05:07 10.243.0.116 [  116.728458] CR2: fffffffffffffff8 CR3: 0000000102568002 CR4: 00000000003706e0
Nov 19 11:05:07 10.243.0.116 ** 2905 printk messages dropped **
Nov 19 11:05:07 10.243.0.116 [  116.748990]  ? spurious_kernel_fault+0x200/0x200
Nov 19 11:05:07 10.243.0.116 ** 2921 printk messages dropped **
Nov 19 11:05:07 10.243.0.116 [  116.769761]  ? load_module.cold.80+0x38d/0x38d
Nov 19 11:05:07 10.243.0.116 ** 2833 printk messages dropped **
Nov 19 11:05:07 10.243.0.116 [  116.789895]  ? print_modules+0x88/0x121
Nov 19 11:05:07 10.243.0.116 ** 2794 printk messages dropped **
Nov 19 11:05:07 10.243.0.116 [  116.809787] RDX: 0000000000000003 RSI: dffffc0000000000 RDI: fffffffffffffff8
Nov 19 11:05:07 10.243.0.116 ** 2821 printk messages dropped **
Nov 19 11:05:07 10.243.0.116 [  116.829735] RSP: 0018:fffffe000016ec68 EFLAGS: 00010046
Nov 19 11:05:07 10.243.0.116 ** 2867 printk messages dropped **
Nov 19 11:05:07 10.243.0.116 [  116.850124] R10: 0000000000000000 R11: 0000000000000000 R12: ffff888118d24780
Nov 19 11:05:07 10.243.0.116 ** 2861 printk messages dropped **
Nov 19 11:05:07 10.243.0.116 [  116.870398]  ? fixup_exception+0x32/0x420
Nov 19 11:05:07 10.243.0.116 ** 2907 printk messages dropped **
Nov 19 11:05:07 10.243.0.116 [  116.891124] R10: ffff8883de4a7b1b R11: ffffed107bc94f63 R12: 1fffffc00002df94
Nov 19 11:05:07 10.243.0.116 ** 2926 printk messages dropped **
Nov 19 11:05:07 10.243.0.116 [  116.911916]  ? print_modules+0x88/0x121
Nov 19 11:05:08 10.243.0.116 ** 2984 printk messages dropped **
Nov 19 11:05:08 10.243.0.116 [  116.933105] R10: ffff8883de4a7b1b R11: ffffed107bc94f63 R12: 1fffffc00002de91
Nov 19 11:05:08 10.243.0.116 ** 3004 printk messages dropped **
Nov 19 11:05:08 10.243.0.116 [  116.954490] IRQ stage: Linux
Nov 19 11:05:08 10.243.0.116 ** 3057 printk messages dropped **
Nov 19 11:05:08 10.243.0.116 [  116.976183]  ? search_exception_tables+0x58/0x60
Nov 19 11:05:08 10.243.0.116 ** 3026 printk messages dropped **
Nov 19 11:05:08 10.243.0.116 [  116.997661] RSP: 0018:fffffe000016ec68 EFLAGS: 00010046
Nov 19 11:05:08 10.243.0.116 ** 3538 printk messages dropped **
Nov 19 11:05:08 10.243.0.116 [  117.022750] R10: ffff8883de4a7b1b R11: ffffed107bc94f63 R12: 1fffffc00002de0f
Nov 19 11:05:08 10.243.0.116 ** 3079 printk messages dropped **
Nov 19 11:05:08 10.243.0.116 [  117.044692] RAX: 0000000000000000 RBX: fffffffffffffff8 RCX: ffffffff890e5bcc
Nov 19 11:05:08 10.243.0.116 ** 3166 printk messages dropped **
Nov 19 11:05:08 10.243.0.116 [  117.067261]  __die_body+0x1f/0x60
Nov 19 11:05:08 10.243.0.116 ** 3148 printk messages dropped **
Nov 19 11:05:08 10.243.0.116 [  117.089709]  ? kallsyms_lookup_buildid+0x9/0x170
Nov 19 11:05:08 10.243.0.116 ** 3057 printk messages dropped **
Nov 19 11:05:08 10.243.0.116 [  117.111510] BUG: unable to handle page fault for address: fffffffffffffff8
Nov 19 11:05:08 10.243.0.116 ** 2983 printk messages dropped **
Nov 19 11:05:08 10.243.0.116 [  117.132653]  ? print_modules+0x88/0x121
Nov 19 11:05:08 10.243.0.116 ** 2940 printk messages dropped **
Nov 19 11:05:08 10.243.0.116 [  117.153514]  ? print_modules+0x88/0x121
Nov 19 11:05:08 10.243.0.116 ** 2973 printk messages dropped **
Nov 19 11:05:08 10.243.0.116 [  117.174717] RSP: 0018:fffffe000016e858 EFLAGS: 00010046
Nov 19 11:05:08 10.243.0.116 ** 3024 printk messages dropped **
Nov 19 11:05:08 10.243.0.116 [  117.196172] RIP: 0010:print_modules+0x88/0x121
Nov 19 11:05:08 10.243.0.116 ** 2969 printk messages dropped **
Nov 19 11:05:08 10.243.0.116 [  117.217393] BUG: unable to handle page fault for address: fffffffffffffff8
Nov 19 11:05:08 10.243.0.116 ** 2894 printk messages dropped **
Nov 19 11:05:08 10.243.0.116 [  117.237877] BUG: unable to handle page fault for address: fffffffffffffff8
Nov 19 11:05:08 10.243.0.116 ** 2894 printk messages dropped **
Nov 19 11:05:08 10.243.0.116 [  117.258399] BUG: unable to handle page fault for address: fffffffffffffff8
Nov 19 11:05:08 10.243.0.116 ** 2957 printk messages dropped **
Nov 19 11:05:08 10.243.0.116 [  117.279220] CR2: fffffffffffffff8 CR3: 0000000102568002 CR4: 00000000003706e0
Nov 19 11:05:08 10.243.0.116 ** 3000 printk messages dropped **
Nov 19 11:05:08 10.243.0.116 [  117.300647] Code: e0 ee 76 85 c0 75 0c 41 c7 84 24 b8 0d 00 00 00 00 00 00 e8 6f fe ff ff e8 3a 38 19 ff 31 c0 88 d8 5b 41 5c 5d c3 cc cc cc cc <0f> 0b 90 66 2e 0f 1f 84 00 00 00 00 00 55 48 89 e5 40 84 f6 75 3e
Nov 19 11:05:08 10.243.0.116 ** 4668 printk messages dropped **
Nov 19 11:05:08 10.243.0.116 [  117.333899]  kernelmode_fixup_or_oops+0x135/0x160
Nov 19 11:05:08 10.243.0.116 ** 3123 printk messages dropped **
Nov 19 11:05:08 10.243.0.116 [  117.356100] RSP: 0018:fffffe000016dc28 EFLAGS: 00010046
Nov 19 11:05:08 10.243.0.116 ** 3177 printk messages dropped **
Nov 19 11:05:08 10.243.0.116 [  117.378724] Code: e0 ee 76 85 c0 75 0c 41 c7 84 24 b8 0d 00 00 00 00 00 00 e8 6f fe ff ff e8 3a 38 19 ff 31 c0 88 d8 5b 41 5c 5d c3 cc cc cc cc <0f> 0b 90 66 2e 0f 1f 84 00 00 00 00 00 55 48 89 e5 40 84 f6 75 3e
Nov 19 11:05:08 10.243.0.116 ** 4805 printk messages dropped **
Nov 19 11:05:08 10.243.0.116 [  117.412798] R13: 0000000000000000 R14: 0000000000000000 R15: ffff88811d198000
Nov 19 11:05:08 10.243.0.116 ** 3278 printk messages dropped **
Nov 19 11:05:08 10.243.0.116 [  117.435982] RBP: fffffe000016e4c0 R08: ffffed107bc94f64 R09: ffffed107bc94f64
Nov 19 11:05:08 10.243.0.116 ** 3352 printk messages dropped **
Nov 19 11:05:08 10.243.0.116 [  117.459823]  ? kallsyms_lookup_buildid+0x9/0x170
Nov 19 11:05:08 10.243.0.116 ** 3337 printk messages dropped **
Nov 19 11:05:08 10.243.0.116 [  117.483497] RSP: 0018:fffffe000016f078 EFLAGS: 00010046
Nov 19 11:05:08 10.243.0.116 ** 3356 printk messages dropped **
Nov 19 11:05:08 10.243.0.116 [  117.507257] RSP: 0018:fffffe000016f078 EFLAGS: 00010046
Nov 19 11:05:08 10.243.0.116 ** 3363 printk messages dropped **
Nov 19 11:05:08 10.243.0.116 [  117.531120]  ? print_modules+0x88/0x121
Nov 19 11:05:08 10.243.0.116 ** 3349 printk messages dropped **
Nov 19 11:05:08 10.243.0.116 [  117.554832] RDX: 0000000000000003 RSI: dffffc0000000000 RDI: fffffffffffffff8
Nov 19 11:05:08 10.243.0.116 ** 3313 printk messages dropped **
Nov 19 11:05:08 10.243.0.116 [  117.578463] RDX: 0000000000000003 RSI: dffffc0000000000 RDI: fffffffffffffff8
Nov 19 11:05:08 10.243.0.116 ** 3314 printk messages dropped **
Nov 19 11:05:08 10.243.0.116 [  117.601900]  ? print_modules+0x88/0x121
Nov 19 11:05:08 10.243.0.116 ** 3339 printk messages dropped **
Nov 19 11:05:08 10.243.0.116 [  117.625689]  ? print_modules+0x88/0x121
Nov 19 11:05:08 10.243.0.116 ** 3428 printk messages dropped **
Nov 19 11:05:08 10.243.0.116 [  117.650011] RSP: 0018:fffffe000016ff10 EFLAGS: 00010046
Nov 19 11:05:08 10.243.0.116 ** 3288 printk messages dropped **
Nov 19 11:05:08 10.243.0.116 [  117.673124]  ? cmp_ex_search+0x1e/0x40
Nov 19 11:05:08 10.243.0.116 ** 3299 printk messages dropped **
Nov 19 11:05:08 10.243.0.116 [  117.696451]  __die+0x25/0x30
Nov 19 11:05:08 10.243.0.116 ** 3269 printk messages dropped **
Nov 19 11:05:08 10.243.0.116 [  117.719669] RSP: 0018:fffffe000016f898 EFLAGS: 00010046
Nov 19 11:05:08 10.243.0.116 ** 3252 printk messages dropped **
Nov 19 11:05:08 10.243.0.116 [  117.742707] R13: 0000000000000000 R14: 0000000000000046 R15: 0000000000000004
Nov 19 11:05:08 10.243.0.116 ** 3293 printk messages dropped **
Nov 19 11:05:08 10.243.0.116 [  117.766273] RSP: 0018:fffffe000016d408 EFLAGS: 00010046
Nov 19 11:05:08 10.243.0.116 ** 3284 printk messages dropped **
Nov 19 11:05:08 10.243.0.116 [  117.789625] R10: ffff8883de4a7b1b R11: ffffed107bc94f63 R12: 1fffffc00002d9ff
Nov 19 11:05:08 10.243.0.116 ** 3361 printk messages dropped **
Nov 19 11:05:08 10.243.0.116 [  117.813545] CR2: fffffffffffffff8 CR3: 0000000102568002 CR4: 00000000003706e0
Nov 19 11:05:08 10.243.0.116 ** 3290 printk messages dropped **
Nov 19 11:05:08 10.243.0.116 [  117.836733] RDX: fffffe000016c0c0 RSI: fffffe000016c100 RDI: ffffffff890e5bcc
Nov 19 11:05:08 10.243.0.116 ** 3276 printk messages dropped **
Nov 19 11:05:08 10.243.0.116 [  117.860058] R13: 0000000000000000 R14: 0000000000000046 R15: 0000000000000004
Nov 19 11:05:08 10.243.0.116 ** 3246 printk messages dropped **
Nov 19 11:05:08 10.243.0.116 [  117.883163]  ? spurious_kernel_fault+0x200/0x200
Nov 19 11:05:08 10.243.0.116 ** 3198 printk messages dropped **
Nov 19 11:05:08 10.243.0.116 [  117.905949] RIP: 0010:print_modules+0x88/0x121
Nov 19 11:05:09 10.243.0.116 ** 3186 printk messages dropped **
Nov 19 11:05:09 10.243.0.116 [  117.928575] RBP: fffffe000016dca0 R08: ffffed107bc94f64 R09: ffffed107bc94f64
Nov 19 11:05:09 10.243.0.116 ** 3186 printk messages dropped **
Nov 19 11:05:09 10.243.0.116 [  117.951238] RAX: 0000000000000000 RBX: 1fffffc00002d804 RCX: fffffe000016c040
Nov 19 11:05:09 10.243.0.116 ** 3125 printk messages dropped **
Nov 19 11:05:09 10.243.0.116 [  117.973359]  ? load_module.cold.80+0x38d/0x38d
Nov 19 11:05:09 10.243.0.116 ** 3055 printk messages dropped **
Nov 19 11:05:09 10.243.0.116 [  117.995000]  ? print_modules+0x88/0x121
Nov 19 11:05:09 10.243.0.116 ** 3012 printk messages dropped **
Nov 19 11:05:09 10.243.0.116 [  118.016566] RSP: 0018:fffffe000016dc28 EFLAGS: 00010046
Nov 19 11:05:09 10.243.0.116 ** 3039 printk messages dropped **
Nov 19 11:05:09 10.243.0.116 [  118.038146]  ? lock_release+0xd5/0x420
Nov 19 11:05:09 10.243.0.116 ** 3036 printk messages dropped **
Nov 19 11:05:09 10.243.0.116 [  118.059839] RBP: fffffe000016ff20 R08: 0000000000000000 R09: 0000000000000000
Nov 19 11:05:09 10.243.0.116 ** 2990 printk messages dropped **
Nov 19 11:05:09 10.243.0.116 [  118.080959] RIP: 0010:irqentry_nmi_enter+0x83/0x90
Nov 19 11:05:09 10.243.0.116 ** 3016 printk messages dropped **
Nov 19 11:05:09 10.243.0.116 [  118.102317] RSP: 0018:fffffe000016f488 EFLAGS: 00010046
Nov 19 11:05:09 10.243.0.116 ** 3012 printk messages dropped **
Nov 19 11:05:09 10.243.0.116 [  118.123722] Hardware name: Supermicro Super Server/X10SDV-6C+-TLN4F, BIOS 2.1 11/22/2019
Nov 19 11:05:09 10.243.0.116 ** 2994 printk messages dropped **
Nov 19 11:05:09 10.243.0.116 [  118.144945]  kernelmode_fixup_or_oops+0x135/0x160
Nov 19 11:05:09 10.243.0.116 ** 3029 printk messages dropped **
Nov 19 11:05:09 10.243.0.116 [  118.166381] R13: 0000000000000000 R14: 0000000000000000 R15: ffff88811d198000
Nov 19 11:05:09 10.243.0.116 ** 3067 printk messages dropped **
Nov 19 11:05:09 10.243.0.116 [  118.188200]  ? search_exception_tables+0x58/0x60
Nov 19 11:05:09 10.243.0.116 ** 3066 printk messages dropped **
Nov 19 11:05:09 10.243.0.116 [  118.209925]  <#DF>
Nov 19 11:05:09 10.243.0.116 ** 3097 printk messages dropped **
Nov 19 11:05:09 10.243.0.116 [  118.232086]  do_error_trap+0xce/0x1b0
Nov 19 11:05:09 10.243.0.116 ** 3106 printk messages dropped **
Nov 19 11:05:09 10.243.0.116 [  118.254177] RBP: fffffe000016f500 R08: ffffed107bc94f64 R09: ffffed107bc94f64
Nov 19 11:05:09 10.243.0.116 ** 3051 printk messages dropped **
Nov 19 11:05:09 10.243.0.116 [  118.275866] R13: 0000000000000000 R14: 0000000000000000 R15: ffff88811d198000
Nov 19 11:05:09 10.243.0.116 ** 3140 printk messages dropped **
Nov 19 11:05:09 10.243.0.116 [  118.298411] RSP: 0018:fffffe000016ff10 EFLAGS: 00010046
Nov 19 11:05:09 10.243.0.116 ** 3134 printk messages dropped **
Nov 19 11:05:09 10.243.0.116 [  118.320670]  ? print_modules+0x88/0x121
Nov 19 11:05:09 10.243.0.116 ** 3043 printk messages dropped **
Nov 19 11:05:09 10.243.0.116 [  118.342407] R13: 0000000000000000 R14: 0000000000000000 R15: ffff88811d198000
Nov 19 11:05:09 10.243.0.116 ** 3027 printk messages dropped **
Nov 19 11:05:09 10.243.0.116 [  118.363831] RAX: 0000000000000000 RBX: fffffffffffffff8 RCX: ffffffff890e5bcc
Nov 19 11:05:09 10.243.0.116 ** 3065 printk messages dropped **
Nov 19 11:05:09 10.243.0.116 [  118.385567]  ? print_modules+0x88/0x121
Nov 19 11:05:09 10.243.0.116 ** 2977 printk messages dropped **
Nov 19 11:05:09 10.243.0.116 [  118.406717] FS:  00007fdce1ffd700(0000) GS:ffff8883de480000(0000) knlGS:0000000000000000
Nov 19 11:05:09 10.243.0.116 ** 3022 printk messages dropped **
Nov 19 11:05:09 10.243.0.116 [  118.428266]  asm_exc_page_fault+0x27/0x30
Nov 19 11:05:09 10.243.0.116 ** 3148 printk messages dropped **
Nov 19 11:05:09 10.243.0.116 [  118.450750] RSP: 0018:fffffe000016ff10 EFLAGS: 00010046
Nov 19 11:05:09 10.243.0.116 ** 3455 printk messages dropped **
Nov 19 11:05:09 10.243.0.116 [  118.475210]  kernelmode_fixup_or_oops+0x135/0x160
Nov 19 11:05:09 10.243.0.116 ** 2955 printk messages dropped **
Nov 19 11:05:09 10.243.0.116 [  118.496152]  ? print_modules+0x88/0x121
Nov 19 11:05:09 10.243.0.116 ** 2953 printk messages dropped **
Nov 19 11:05:09 10.243.0.116 [  118.517162] CPU: 5 PID: 1563 Comm: DummyTxSample Tainted: G        W         5.15.77evl-g8390add2f766 #3
Nov 19 11:05:09 10.243.0.116 ** 3029 printk messages dropped **
Nov 19 11:05:09 10.243.0.116 [  118.538698]  __bad_area_nosemaphore+0x247/0x2b0
Nov 19 11:05:09 10.243.0.116 ** 3027 printk messages dropped **
Nov 19 11:05:09 10.243.0.116 [  118.560206] R10: ffff8883de4a7b1b R11: ffffed107bc94f63 R12: 1fffffc00002dd0b
Nov 19 11:05:09 10.243.0.116 ** 3076 printk messages dropped **
Nov 19 11:05:09 10.243.0.116 [  118.582026] RAX: 0000000000f00000 RBX: fffffe000016ff00 RCX: fffffe000016c000
Nov 19 11:05:09 10.243.0.116 ** 3107 printk messages dropped **
Nov 19 11:05:09 10.243.0.116 [  118.604112]  ? lock_release+0xd5/0x420
Nov 19 11:05:09 10.243.0.116 ** 3046 printk messages dropped **
Nov 19 11:05:09 10.243.0.116 [  118.625698]  __die_body+0x1f/0x60
Nov 19 11:05:09 10.243.0.116 ** 3071 printk messages dropped **
Nov 19 11:05:09 10.243.0.116 [  118.647540]  ? print_modules+0x88/0x121
Nov 19 11:05:09 10.243.0.116 ** 2958 printk messages dropped **
Nov 19 11:05:09 10.243.0.116 [  118.668660] RSP: 0018:fffffe000016ff10 EFLAGS: 00010046
Nov 19 11:05:09 10.243.0.116 ** 2932 printk messages dropped **
Nov 19 11:05:09 10.243.0.116 [  118.689492] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Nov 19 11:05:09 10.243.0.116 ** 2929 printk messages dropped **
Nov 19 11:05:09 10.243.0.116 [  118.710382] R10: ffff8883de4a7b1b R11: ffffed107bc94f63 R12: 1fffffc00002db85
Nov 19 11:05:09 10.243.0.116 ** 2945 printk messages dropped **
Nov 19 11:05:09 10.243.0.116 [  118.731274]  ? print_modules+0x88/0x121
Nov 19 11:05:09 10.243.0.116 ** 3027 printk messages dropped **
Nov 19 11:05:09 10.243.0.116 [  118.752744]  ? search_exception_tables+0x58/0x60
Nov 19 11:05:09 10.243.0.116 ** 3001 printk messages dropped **
Nov 19 11:05:09 10.243.0.116 [  118.774022] RAX: 0000000000000000 RBX: 1fffffc00002d804 RCX: fffffe000016c040
Nov 19 11:05:09 10.243.0.116 ** 2962 printk messages dropped **
Nov 19 11:05:09 10.243.0.116 [  118.795066]  __die_body+0x1f/0x60
Nov 19 11:05:09 10.243.0.116 ** 2958 printk messages dropped **
Nov 19 11:05:09 10.243.0.116 [  118.815988]  ? load_module.cold.80+0x38d/0x38d
Nov 19 11:05:09 10.243.0.116 ** 2917 printk messages dropped **
Nov 19 11:05:09 10.243.0.116 [  118.836615] RIP: 0010:print_modules+0x88/0x121
Nov 19 11:05:09 10.243.0.116 ** 2906 printk messages dropped **
Nov 19 11:05:09 10.243.0.116 [  118.857283]  __die_body+0x1f/0x60
Nov 19 11:05:09 10.243.0.116 ** 2853 printk messages dropped **
Nov 19 11:05:09 10.243.0.116 [  118.877497] RDX: 0000000000000003 RSI: dffffc0000000000 RDI: fffffffffffffff8
Nov 19 11:05:09 10.243.0.116 ** 2792 printk messages dropped **
Nov 19 11:05:09 10.243.0.116 [  118.897412] R10: ffff8883de4a7b1b R11: ffffed107bc94f63 R12: 1fffffc00002da81
Nov 19 11:05:10 10.243.0.116 ** 2893 printk messages dropped **
Nov 19 11:05:10 10.243.0.116 [  118.917986] RBP: fffffe000016d480 R08: ffffed107bc94f64 R09: ffffed107bc94f64
Nov 19 11:05:10 10.243.0.116 ** 2891 printk messages dropped **
Nov 19 11:05:10 10.243.0.116 [  118.938541] RSP: 0018:fffffe000016d408 EFLAGS: 00010046
Nov 19 11:05:10 10.243.0.116 ** 2898 printk messages dropped **
Nov 19 11:05:10 10.243.0.116 [  118.959229] R10: ffff8883de4a7b1b R11: ffffed107bc94f63 R12: 1fffffc00002da81
Nov 19 11:05:10 10.243.0.116 ** 2927 printk messages dropped **
Nov 19 11:05:10 10.243.0.116 [  118.980033] Code: b1 ff ff 65 ff 05 94 13 f4 76 48 8b 1d 8d a7 0c 01 48 83 eb 08 4c 8d 6b 08 49 81 fd 40 03 1b 8a 74 3e 48 89 df e8 04 7d 3f ff <83> 3b 03 74 1f 48 8d 75 a8 48 89 df e8 93 a3 14 ff 48 8d 73 18 48
Nov 19 11:05:10 10.243.0.116 ** 4631 printk messages dropped **
Nov 19 11:05:10 10.243.0.116 [  119.012949]  ? print_modules+0x88/0x121
Nov 19 11:05:10 10.243.0.116 ** 3062 printk messages dropped **
Nov 19 11:05:10 10.243.0.116 [  119.034749] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Nov 19 11:05:10 10.243.0.116 ** 3073 printk messages dropped **
Nov 19 11:05:10 10.243.0.116 [  119.056565]  ? bsearch+0x5e/0x90
Nov 19 11:05:10 10.243.0.116 ** 3031 printk messages dropped **
Nov 19 11:05:10 10.243.0.116 [  119.078172] RIP: 0010:print_modules+0x88/0x121
Nov 19 11:05:10 10.243.0.116 ** 2967 printk messages dropped **
Nov 19 11:05:10 10.243.0.116 [  119.099367] R13: 0000000000000000 R14: 0000000000000000 R15: ffff88811d198000
Nov 19 11:05:10 10.243.0.116 ** 2963 printk messages dropped **
Nov 19 11:05:10 10.243.0.116 [  119.120576]  ? load_module.cold.80+0x38d/0x38d
Nov 19 11:05:10 10.243.0.116 ** 3017 printk messages dropped **
Nov 19 11:05:10 10.243.0.116 [  119.142051] RDX: fffffe000016c0c0 RSI: fffffe000016c100 RDI: ffffffff890e5bcc
Nov 19 11:05:10 10.243.0.116 ** 3097 printk messages dropped **
Nov 19 11:05:10 10.243.0.116 [  119.164191]  ? spurious_kernel_fault+0x200/0x200
Nov 19 11:05:10 10.243.0.116 ** 3101 printk messages dropped **
Nov 19 11:05:10 10.243.0.116 [  119.186318] RSP: 0018:fffffe000016d818 EFLAGS: 00010046
Nov 19 11:05:10 10.243.0.116 ** 3113 printk messages dropped **
Nov 19 11:05:10 10.243.0.116 [  119.208391]  ? search_bpf_extables+0x136/0x190
Nov 19 11:05:10 10.243.0.116 ** 3030 printk messages dropped **
Nov 19 11:05:10 10.243.0.116 [  119.229736] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Nov 19 11:05:10 10.243.0.116 ** 3021 printk messages dropped **
Nov 19 11:05:10 10.243.0.116 [  119.251244]  asm_exc_page_fault+0x27/0x30
Nov 19 11:05:10 10.243.0.116 ** 3132 printk messages dropped **
Nov 19 11:05:10 10.243.0.116 [  119.273490] R10: ffff8883de4a7b1b R11: ffffed107bc94f63 R12: 1fffffc00002df94
Nov 19 11:05:10 10.243.0.116 ** 3062 printk messages dropped **
Nov 19 11:05:10 10.243.0.116 [  119.295121]  ? cmp_ex_search+0x1e/0x40
Nov 19 11:05:10 10.243.0.116 ** 3054 printk messages dropped **
Nov 19 11:05:10 10.243.0.116 [  119.316878] RSP: 0018:fffffe000016ff10 EFLAGS: 00010046
Nov 19 11:05:10 10.243.0.116 ** 3053 printk messages dropped **
Nov 19 11:05:10 10.243.0.116 [  119.338584] R10: 0000000000000000 R11: 0000000000000000 R12: ffff888118d24780
Nov 19 11:05:10 10.243.0.116 ** 3080 printk messages dropped **
Nov 19 11:05:10 10.243.0.116 [  119.360494] RDX: 0000000000000000 RSI: 0000000000000000 RDI: fffffe000016ff58
Nov 19 11:05:10 10.243.0.116 ** 3163 printk messages dropped **
Nov 19 11:05:10 10.243.0.116 [  119.382932]  ? spurious_kernel_fault+0x200/0x200
Nov 19 11:05:10 10.243.0.116 ** 3196 printk messages dropped **
Nov 19 11:05:10 10.243.0.116 [  119.405745]  exc_page_fault+0xcf/0xe0
Nov 19 11:05:10 10.243.0.116 ** 3118 printk messages dropped **
Nov 19 11:05:10 10.243.0.116 [  119.427906] R10: ffff8883de4a7b1b R11: ffffed107bc94f63 R12: 1fffffc00002db85
Nov 19 11:05:10 10.243.0.116 ** 3139 printk messages dropped **
Nov 19 11:05:10 10.243.0.116 [  119.450297]  ? lock_release+0xd5/0x420
Nov 19 11:05:10 10.243.0.116 ** 3041 printk messages dropped **
Nov 19 11:05:10 10.243.0.116 [  119.471988] R13: 0000000000000000 R14: 0000000000000000 R15: ffff88811d198000
Nov 19 11:05:10 10.243.0.116 ** 3094 printk messages dropped **
Nov 19 11:05:10 10.243.0.116 [  119.493987]  ? print_modules+0x88/0x121
Nov 19 11:05:10 10.243.0.116 ** 2971 printk messages dropped **
Nov 19 11:05:10 10.243.0.116 [  119.515050] RSP: 0018:fffffe000016d408 EFLAGS: 00010046
Nov 19 11:05:10 10.243.0.116 ** 2900 printk messages dropped **
Nov 19 11:05:10 10.243.0.116 [  119.535544] FS:  00007fdce1ffd700(0000) GS:ffff8883de480000(0000) knlGS:0000000000000000
Nov 19 11:05:10 10.243.0.116 ** 2930 printk messages dropped **
Nov 19 11:05:10 10.243.0.116 [  119.556276] R10: ffff8883de4a7b1b R11: ffffed107bc94f63 R12: 1fffffc00002db03
Nov 19 11:05:10 10.243.0.116 ** 2989 printk messages dropped **
Nov 19 11:05:10 10.243.0.116 [  119.577443] RSP: 0018:fffffe000016e448 EFLAGS: 00010046
Nov 19 11:05:10 10.243.0.116 ** 3052 printk messages dropped **
Nov 19 11:05:10 10.243.0.116 [  119.599025]  __bad_area_nosemaphore+0x247/0x2b0
Nov 19 11:05:10 10.243.0.116 ** 3018 printk messages dropped **
Nov 19 11:05:10 10.243.0.116 [  119.620456]  do_kern_addr_fault+0x7c/0xb0
Nov 19 11:05:10 10.243.0.116 ** 2986 printk messages dropped **
Nov 19 11:05:10 10.243.0.116 [  119.641770]  ? search_bpf_extables+0x136/0x190
Nov 19 11:05:10 10.243.0.116 ** 2992 printk messages dropped **
Nov 19 11:05:10 10.243.0.116 [  119.663068]  ? print_modules+0x88/0x121
Nov 19 11:05:10 10.243.0.116 ** 2921 printk messages dropped **
Nov 19 11:05:10 10.243.0.116 [  119.683888]  ? spurious_kernel_fault+0x200/0x200
Nov 19 11:05:10 10.243.0.116 ** 2913 printk messages dropped **
Nov 19 11:05:10 10.243.0.116 [  119.704591] RSP: 0018:fffffe000016f078 EFLAGS: 00010046
Nov 19 11:05:10 10.243.0.116 ** 2897 printk messages dropped **
Nov 19 11:05:10 10.243.0.116 [  119.725246] RBP: fffffe000016f0f0 R08: ffffed107bc94f64 R09: ffffed107bc94f64
Nov 19 11:05:10 10.243.0.116 ** 2971 printk messages dropped **
Nov 19 11:05:10 10.243.0.116 [  119.746588]  ? spurious_kernel_fault+0x200/0x200
Nov 19 11:05:10 10.243.0.116 ** 2954 printk messages dropped **
Nov 19 11:05:10 10.243.0.116 [  119.767698] #PF: supervisor read access in kernel mode
Nov 19 11:05:10 10.243.0.116 ** 2962 printk messages dropped **
Nov 19 11:05:10 10.243.0.116 [  119.788706]  ? kallsyms_lookup_buildid+0x9/0x170
Nov 19 11:05:10 10.243.0.116 ** 2916 printk messages dropped **
Nov 19 11:05:10 10.243.0.116 [  119.809388] RSP: 0018:fffffe000016c000 EFLAGS: 00010046
Nov 19 11:05:10 10.243.0.116 ** 2916 printk messages dropped **
Nov 19 11:05:10 10.243.0.116 [  119.829956] RAX: 0000000000000000 RBX: fffffffffffffff8 RCX: ffffffff890e5bcc
Nov 19 11:05:10 10.243.0.116 ** 3426 printk messages dropped **
Nov 19 11:05:10 10.243.0.116 [  119.854131] RSP: 0018:fffffe000016c000 EFLAGS: 00010046
Nov 19 11:05:10 10.243.0.116 ** 2953 printk messages dropped **
Nov 19 11:05:10 10.243.0.116 [  119.875124] RSP: 0018:fffffe000016ec68 EFLAGS: 00010046
Nov 19 11:05:10 10.243.0.116 ** 2932 printk messages dropped **
Nov 19 11:05:10 10.243.0.116 [  119.895959] R13: 0000000000000000 R14: 0000000000000000 R15: ffff88811d198000
Nov 19 11:05:11 10.243.0.116 ** 2936 printk messages dropped **
Nov 19 11:05:11 10.243.0.116 [  119.916762]  ? spurious_kernel_fault+0x200/0x200
Nov 19 11:05:11 10.243.0.116 ** 2966 printk messages dropped **
Nov 19 11:05:11 10.243.0.116 [  119.937959] RSP: 0018:fffffe000016ff10 EFLAGS: 00010046
Nov 19 11:05:11 10.243.0.116 ** 2928 printk messages dropped **
Nov 19 11:05:11 10.243.0.116 [  119.958785] RBP: fffffe000016e4c0 R08: ffffed107bc94f64 R09: ffffed107bc94f64
Nov 19 11:05:11 10.243.0.116 ** 2999 printk messages dropped **
Nov 19 11:05:11 10.243.0.116 [  119.980033] R10: ffff8883de4a7b1b R11: ffffed107bc94f63 R12: 1fffffc00002de0f
Nov 19 11:05:11 10.243.0.116 ** 3000 printk messages dropped **
Nov 19 11:05:11 10.243.0.116 [  120.001411]  do_trap+0x126/0x150
Nov 19 11:05:11 10.243.0.116 ** 3005 printk messages dropped **
Nov 19 11:05:11 10.243.0.116 [  120.022691] RSP: 0018:fffffe000016e858 EFLAGS: 00010046
Nov 19 11:05:11 10.243.0.116 ** 3012 printk messages dropped **
Nov 19 11:05:11 10.243.0.116 [  120.044086]  ? page_fault_oops+0x3c0/0x3c0
Nov 19 11:05:11 10.243.0.116 ** 2922 printk messages dropped **
Nov 19 11:05:11 10.243.0.116 [  120.064714]  ? spurious_kernel_fault+0x200/0x200
Nov 19 11:05:11 10.243.0.116 ** 2866 printk messages dropped **
Nov 19 11:05:11 10.243.0.116 [  120.084910]  ? page_fault_oops+0x3c0/0x3c0
Nov 19 11:05:11 10.243.0.116 ** 2828 printk messages dropped **
Nov 19 11:05:11 10.243.0.116 [  120.104949]  ? page_fault_oops+0x3c0/0x3c0
Nov 19 11:05:11 10.243.0.116 ** 2804 printk messages dropped **
Nov 19 11:05:11 10.243.0.116 [  120.124899]  do_kern_addr_fault+0x7c/0xb0
Nov 19 11:05:11 10.243.0.116 ** 2843 printk messages dropped **
Nov 19 11:05:11 10.243.0.116 [  120.145264] RBP: fffffe000016c008 R08: fffffe000016c080 R09: fffffe000016c1b0
Nov 19 11:05:11 10.243.0.116 ** 2832 printk messages dropped **
Nov 19 11:05:11 10.243.0.116 [  120.165329] R10: ffff8883de4a7b1b R11: ffffed107bc94f63 R12: 1fffffc00002df13
Nov 19 11:05:11 10.243.0.116 ** 2923 printk messages dropped **
Nov 19 11:05:11 10.243.0.116 [  120.185987] RSP: 0018:fffffe000016fca0 EFLAGS: 00010046
Nov 19 11:05:11 10.243.0.116 ** 2930 printk messages dropped **
Nov 19 11:05:11 10.243.0.116 [  120.206828]  ? print_modules+0x88/0x121
Nov 19 11:05:11 10.243.0.116 ** 2916 printk messages dropped **
Nov 19 11:05:11 10.243.0.116 [  120.227746] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Nov 19 11:05:11 10.243.0.116 ** 2887 printk messages dropped **
Nov 19 11:05:11 10.243.0.116 [  120.248265] RSP: 0018:fffffe000016e038 EFLAGS: 00010046
Nov 19 11:05:11 10.243.0.116 ** 2894 printk messages dropped **
Nov 19 11:05:11 10.243.0.116 [  120.268736] RSP: 0018:fffffe000016e038 EFLAGS: 00010046
Nov 19 11:05:11 10.243.0.116 ** 2894 printk messages dropped **
Nov 19 11:05:11 10.243.0.116 [  120.289270] RSP: 0018:fffffe000016e038 EFLAGS: 00010046
Nov 19 11:05:11 10.243.0.116 ** 2900 printk messages dropped **
Nov 19 11:05:11 10.243.0.116 [  120.309882] FS:  00007fdce1ffd700(0000) GS:ffff8883de480000(0000) knlGS:0000000000000000
Nov 19 11:05:11 10.243.0.116 ** 2964 printk messages dropped **
Nov 19 11:05:11 10.243.0.116 [  120.330884] R13: 0000000000000000 R14: 0000000000000000 R15: ffff88811d198000
Nov 19 11:05:11 10.243.0.116 ** 2994 printk messages dropped **
Nov 19 11:05:11 10.243.0.116 [  120.352227]  ? print_modules+0x88/0x121
Nov 19 11:05:11 10.243.0.116 ** 2984 printk messages dropped **
Nov 19 11:05:11 10.243.0.116 [  120.373417] RSP: 0018:fffffe000016c000 EFLAGS: 00010046
Nov 19 11:05:11 10.243.0.116 ** 2950 printk messages dropped **
Nov 19 11:05:11 10.243.0.116 [  120.394348]  asm_exc_page_fault+0x27/0x30
Nov 19 11:05:11 10.243.0.116 ** 2930 printk messages dropped **
Nov 19 11:05:11 10.243.0.116 [  120.415006] RSP: 0018:fffffe000016e448 EFLAGS: 00010046
Nov 19 11:05:11 10.243.0.116 ** 2894 printk messages dropped **
Nov 19 11:05:11 10.243.0.116 [  120.435551] RSP: 0018:fffffe000016e448 EFLAGS: 00010046
Nov 19 11:05:11 10.243.0.116 ** 2892 printk messages dropped **
Nov 19 11:05:11 10.243.0.116 [  120.455999] RIP: 0010:print_modules+0x88/0x121
Nov 19 11:05:11 10.243.0.116 ** 2854 printk messages dropped **
Nov 19 11:05:11 10.243.0.116 [  120.476102]  ? fixup_exception+0x32/0x420
Nov 19 11:05:11 10.243.0.116 ** 2814 printk messages dropped **
Nov 19 11:05:11 10.243.0.116 [  120.496116] RSP: 0018:fffffe000016fca0 EFLAGS: 00010046
Nov 19 11:05:11 10.243.0.116 ** 2812 printk messages dropped **
Nov 19 11:05:11 10.243.0.116 [  120.516037]  ? cmp_ex_sort+0x50/0x50
Nov 19 11:05:11 10.243.0.116 ** 2773 printk messages dropped **
Nov 19 11:05:11 10.243.0.116 [  120.535751]  do_kern_addr_fault+0x7c/0xb0
Nov 19 11:05:11 10.243.0.116 ** 2800 printk messages dropped **
Nov 19 11:05:11 10.243.0.116 [  120.555659]  ? search_exception_tables+0x58/0x60
Nov 19 11:05:11 10.243.0.116 ** 2786 printk messages dropped **
Nov 19 11:05:11 10.243.0.116 [  120.575297]  ? spurious_kernel_fault+0x200/0x200
Nov 19 11:05:11 10.243.0.116 ** 2774 printk messages dropped **
Nov 19 11:05:11 10.243.0.116 [  120.594910]  exc_invalid_op+0x4f/0xa0
Nov 19 11:05:11 10.243.0.116 ** 2792 printk messages dropped **
Nov 19 11:05:11 10.243.0.116 [  120.614724]  ? lock_release+0xd5/0x420
Nov 19 11:05:11 10.243.0.116 ** 2799 printk messages dropped **
Nov 19 11:05:11 10.243.0.116 [  120.634567] BUG: unable to handle page fault for address: fffffffffffffff8
Nov 19 11:05:11 10.243.0.116 ** 2774 printk messages dropped **
Nov 19 11:05:11 10.243.0.116 [  120.654337]  ? page_fault_oops+0x3c0/0x3c0
Nov 19 11:05:11 10.243.0.116 ** 2728 printk messages dropped **
Nov 19 11:05:11 10.243.0.116 [  120.673753]  ? load_module.cold.80+0x38d/0x38d
Nov 19 11:05:11 10.243.0.116 ** 2706 printk messages dropped **
Nov 19 11:05:11 10.243.0.116 [  120.692986] RAX: 0000000000000000 RBX: 1fffffc00002d804 RCX: fffffe000016c040
Nov 19 11:05:11 10.243.0.116 ** 2741 printk messages dropped **
Nov 19 11:05:11 10.243.0.116 [  120.712540] Call Trace:
Nov 19 11:05:11 10.243.0.116 ** 2680 printk messages dropped **
Nov 19 11:05:11 10.243.0.116 [  120.731548]  ? page_fault_oops+0x3c0/0x3c0
Nov 19 11:05:11 10.243.0.116 ** 2678 printk messages dropped **
Nov 19 11:05:11 10.243.0.116 [  120.750619] FS:  00007fdce1ffd700(0000) GS:ffff8883de480000(0000) knlGS:0000000000000000
Nov 19 11:05:11 10.243.0.116 ** 2743 printk messages dropped **
Nov 19 11:05:11 10.243.0.116 [  120.770171] R13: 0000000000000000 R14: 0000000000000000 R15: ffff88811d198000
Nov 19 11:05:11 10.243.0.116 ** 2725 printk messages dropped **
Nov 19 11:05:11 10.243.0.116 [  120.789539] RAX: 0000000000000000 RBX: fffffffffffffff8 RCX: ffffffff890e5bcc
Nov 19 11:05:11 10.243.0.116 ** 2789 printk messages dropped **
Nov 19 11:05:11 10.243.0.116 [  120.809264] RIP: 0010:irqentry_nmi_enter+0x83/0x90
Nov 19 11:05:11 10.243.0.116 ** 2711 printk messages dropped **
Nov 19 11:05:11 10.243.0.116 [  120.828663] RSP: 0018:fffffe000016e858 EFLAGS: 00010046
Nov 19 11:05:11 10.243.0.116 ** 2740 printk messages dropped **
Nov 19 11:05:11 10.243.0.116 [  120.848112] R10: fffffe000016c447 R11: fffffbc00002d888 R12: fffffe000016c1b0
Nov 19 11:05:11 10.243.0.116 ** 2787 printk messages dropped **
Nov 19 11:05:11 10.243.0.116 [  120.867854]  __bad_area_nosemaphore+0x247/0x2b0
Nov 19 11:05:11 10.243.0.116 ** 2784 printk messages dropped **
Nov 19 11:05:11 10.243.0.116 [  120.887655]  ? cmp_ex_search+0x1e/0x40
Nov 19 11:05:11 10.243.0.116 ** 2778 printk messages dropped **
Nov 19 11:05:11 10.243.0.116 [  120.907399] RAX: 0000000000f00000 RBX: fffffe000016ff00 RCX: fffffe000016c000
Nov 19 11:05:12 10.243.0.116 ** 2797 printk messages dropped **
Nov 19 11:05:12 10.243.0.116 [  120.927203]  ? search_exception_tables+0x58/0x60
Nov 19 11:05:12 10.243.0.116 ** 2805 printk messages dropped **
Nov 19 11:05:12 10.243.0.116 [  120.947088] RSP: 0018:fffffe000016e858 EFLAGS: 00010046
Nov 19 11:05:12 10.243.0.116 ** 2826 printk messages dropped **
Nov 19 11:05:12 10.243.0.116 [  120.967127] RBP: fffffe000016e0b0 R08: ffffed107bc94f64 R09: ffffed107bc94f64
Nov 19 11:05:12 10.243.0.116 ** 2876 printk messages dropped **
Nov 19 11:05:12 10.243.0.116 [  120.987552]  ? print_modules+0x88/0x121
Nov 19 11:05:12 10.243.0.116 ** 2890 printk messages dropped **
Nov 19 11:05:12 10.243.0.116 [  121.008095] RDX: fffffe000016c0c0 RSI: fffffe000016c100 RDI: ffffffff890e5bcc
Nov 19 11:05:12 10.243.0.116 ** 2902 printk messages dropped **
Nov 19 11:05:12 10.243.0.116 [  121.028720] BUG: unable to handle page fault for address: fffffffffffffff8
Nov 19 11:05:12 10.243.0.116 ** 2970 printk messages dropped **
Nov 19 11:05:12 10.243.0.116 [  121.049842]  bad_area_nosemaphore+0x11/0x20
Nov 19 11:05:12 10.243.0.116 ** 3019 printk messages dropped **
Nov 19 11:05:12 10.243.0.116 [  121.071376]  ? print_modules+0x88/0x121
Nov 19 11:05:12 10.243.0.116 ** 2980 printk messages dropped **
Nov 19 11:05:12 10.243.0.116 [  121.092699] BUG: unable to handle page fault for address: fffffffffffffff8
Nov 19 11:05:12 10.243.0.116 ** 3065 printk messages dropped **
Nov 19 11:05:12 10.243.0.116 [  121.114431]  ? search_exception_tables+0x58/0x60
Nov 19 11:05:12 10.243.0.116 ** 3033 printk messages dropped **
Nov 19 11:05:12 10.243.0.116 [  121.135974] RSP: 0018:fffffe000016c000 EFLAGS: 00010046
Nov 19 11:05:12 10.243.0.116 ** 3497 printk messages dropped **
Nov 19 11:05:12 10.243.0.116 [  121.160859]  __bad_area_nosemaphore+0x247/0x2b0
Nov 19 11:05:12 10.243.0.116 ** 3040 printk messages dropped **
Nov 19 11:05:12 10.243.0.116 [  121.182482] BUG: unable to handle page fault for address: fffffffffffffff8
Nov 19 11:05:12 10.243.0.116 ** 3026 printk messages dropped **
Nov 19 11:05:12 10.243.0.116 [  121.203845]  ? cmp_ex_sort+0x50/0x50
Nov 19 11:05:12 10.243.0.116 ** 3128 printk messages dropped **
Nov 19 11:05:12 10.243.0.116 [  121.226146]  ? print_modules+0x88/0x121
Nov 19 11:05:12 10.243.0.116 ** 3048 printk messages dropped **
Nov 19 11:05:12 10.243.0.116 [  121.247705] BUG: unable to handle page fault for address: fffffffffffffff8
Nov 19 11:05:12 10.243.0.116 ** 3008 printk messages dropped **
Nov 19 11:05:12 10.243.0.116 [  121.268919]  ? kallsyms_lookup_buildid+0x9/0x170
Nov 19 11:05:12 10.243.0.116 ** 2996 printk messages dropped **
Nov 19 11:05:12 10.243.0.116 [  121.290390] Oops: 0000 [#6851] SMP KASAN PTI IRQ_PIPELINE
Nov 19 11:05:12 10.243.0.116 ** 2929 printk messages dropped **
Nov 19 11:05:12 10.243.0.116 [  121.311200]  ? fixup_exception+0x32/0x420
Nov 19 11:05:12 10.243.0.116 ** 2921 printk messages dropped **
Nov 19 11:05:12 10.243.0.116 [  121.331888]  ? is_prefetch.isra.28+0x8c/0x2b0
Nov 19 11:05:12 10.243.0.116 ** 2920 printk messages dropped **
Nov 19 11:05:12 10.243.0.116 [  121.352631]  __die+0x25/0x30
Nov 19 11:05:12 10.243.0.116 ** 2838 printk messages dropped **
Nov 19 11:05:12 10.243.0.116 [  121.372740]  ? search_bpf_extables+0x136/0x190
Nov 19 11:05:12 10.243.0.116 ** 2791 printk messages dropped **
Nov 19 11:05:12 10.243.0.116 [  121.392597]  ? print_modules+0x88/0x121
Nov 19 11:05:12 10.243.0.116 ** 2823 printk messages dropped **
Nov 19 11:05:12 10.243.0.116 [  121.412569] R13: ffff888118d24780 R14: 0000000102568002 R15: 0000000000000000
Nov 19 11:05:12 10.243.0.116 ** 2776 printk messages dropped **
Nov 19 11:05:12 10.243.0.116 [  121.432282] RDX: 0000000000000003 RSI: dffffc0000000000 RDI: fffffffffffffff8
Nov 19 11:05:12 10.243.0.116 ** 2827 printk messages dropped **
Nov 19 11:05:12 10.243.0.116 [  121.452444] RSP: 0018:fffffe000016c000 EFLAGS: 00010046
Nov 19 11:05:12 10.243.0.116 ** 2804 printk messages dropped **
Nov 19 11:05:12 10.243.0.116 [  121.472443]  ? print_modules+0x88/0x121
Nov 19 11:05:12 10.243.0.116 ** 2756 printk messages dropped **
Nov 19 11:05:12 10.243.0.116 [  121.492111] RSP: 0018:fffffe000016e448 EFLAGS: 00010046
Nov 19 11:05:12 10.243.0.116 ** 2795 printk messages dropped **
Nov 19 11:05:12 10.243.0.116 [  121.511901] RSP: 0018:fffffe000016d818 EFLAGS: 00010046
Nov 19 11:05:12 10.243.0.116 ** 2795 printk messages dropped **
Nov 19 11:05:12 10.243.0.116 [  121.531919] RBP: fffffe000016ff20 R08: 0000000000000000 R09: 0000000000000000
Nov 19 11:05:12 10.243.0.116 ** 2772 printk messages dropped **
Nov 19 11:05:12 10.243.0.116 [  121.551658] RSP: 0018:fffffe000016f078 EFLAGS: 00010046
Nov 19 11:05:12 10.243.0.116 ** 2767 printk messages dropped **
Nov 19 11:05:12 10.243.0.116 [  121.571238] R13: 0000000000000000 R14: 0000000000000000 R15: ffff88811d198000
Nov 19 11:05:12 10.243.0.116 ** 2838 printk messages dropped **
Nov 19 11:05:12 10.243.0.116 [  121.591487]  ? cmp_ex_search+0x1e/0x40
Nov 19 11:05:12 10.243.0.116 ** 2830 printk messages dropped **
Nov 19 11:05:12 10.243.0.116 [  121.611642] BUG: unable to handle page fault for address: fffffffffffffff8
Nov 19 11:05:12 10.243.0.116 ** 2865 printk messages dropped **
Nov 19 11:05:12 10.243.0.116 [  121.631887]  do_trap+0x126/0x150
Nov 19 11:05:12 10.243.0.116 ** 2822 printk messages dropped **
Nov 19 11:05:12 10.243.0.116 [  121.651927] R13: 0000000000000000 R14: 0000000000000000 R15: ffff88811d198000
Nov 19 11:05:12 10.243.0.116 ** 2841 printk messages dropped **
Nov 19 11:05:12 10.243.0.116 [  121.672196]  ? is_prefetch.isra.28+0x8c/0x2b0
Nov 19 11:05:12 10.243.0.116 ** 2796 printk messages dropped **
Nov 19 11:05:12 10.243.0.116 [  121.692041]  ? page_fault_oops+0x3c0/0x3c0
Nov 19 11:05:12 10.243.0.116 ** 2813 printk messages dropped **
Nov 19 11:05:12 10.243.0.116 [  121.712070] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Nov 19 11:05:12 10.243.0.116 ** 2874 printk messages dropped **
Nov 19 11:05:12 10.243.0.116 [  121.732563] BUG: unable to handle page fault for address: fffffffffffffff8
Nov 19 11:05:12 10.243.0.116 ** 2894 printk messages dropped **
Nov 19 11:05:12 10.243.0.116 [  121.753209] BUG: unable to handle page fault for address: fffffffffffffff8
Nov 19 11:05:12 10.243.0.116 ** 2915 printk messages dropped **
Nov 19 11:05:12 10.243.0.116 [  121.773845] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Nov 19 11:05:12 10.243.0.116 ** 2957 printk messages dropped **
Nov 19 11:05:12 10.243.0.116 [  121.794886] RDX: 0000000000000003 RSI: dffffc0000000000 RDI: fffffffffffffff8
Nov 19 11:05:12 10.243.0.116 ** 3060 printk messages dropped **
Nov 19 11:05:12 10.243.0.116 [  121.816779] RBP: fffffe000016f500 R08: ffffed107bc94f64 R09: ffffed107bc94f64
Nov 19 11:05:12 10.243.0.116 ** 3052 printk messages dropped **
Nov 19 11:05:12 10.243.0.116 [  121.838390]  ? print_modules+0x88/0x121
Nov 19 11:05:12 10.243.0.116 ** 3156 printk messages dropped **
Nov 19 11:05:12 10.243.0.116 [  121.860988] R10: ffff8883de4a7b1b R11: ffffed107bc94f63 R12: 1fffffc00002df13
Nov 19 11:05:12 10.243.0.116 ** 3128 printk messages dropped **
Nov 19 11:05:12 10.243.0.116 [  121.883235]  ? cmp_ex_sort+0x50/0x50
Nov 19 11:05:12 10.243.0.116 ** 3249 printk messages dropped **
Nov 19 11:05:12 10.243.0.116 [  121.906265] BUG: unable to handle page fault for address: fffffffffffffff8
Nov 19 11:05:13 10.243.0.116 ** 3167 printk messages dropped **
Nov 19 11:05:13 10.243.0.116 [  121.928613]  ? lock_release+0xd5/0x420
Nov 19 11:05:13 10.243.0.116 ** 3124 printk messages dropped **
Nov 19 11:05:13 10.243.0.116 [  121.950865]  ? print_modules+0x88/0x121
Nov 19 11:05:13 10.243.0.116 ** 3081 printk messages dropped **
Nov 19 11:05:13 10.243.0.116 [  121.972804]  ? search_bpf_extables+0x136/0x190
Nov 19 11:05:13 10.243.0.116 ** 3081 printk messages dropped **
Nov 19 11:05:13 10.243.0.116 [  121.994623]  ? print_modules+0x88/0x121
Nov 19 11:05:13 10.243.0.116 ** 3047 printk messages dropped **
Nov 19 11:05:13 10.243.0.116 [  122.016210] #PF: supervisor read access in kernel mode
Nov 19 11:05:13 10.243.0.116 ** 3079 printk messages dropped **
Nov 19 11:05:13 10.243.0.116 [  122.038047] R13: 0000000000000000 R14: 0000000000000000 R15: ffff88811d198000
Nov 19 11:05:13 10.243.0.116 ** 3010 printk messages dropped **
Nov 19 11:05:13 10.243.0.116 [  122.059355] R10: ffff8883de4a7b1b R11: ffffed107bc94f63 R12: 1fffffc00002db85
Nov 19 11:05:13 10.243.0.116 ** 3101 printk messages dropped **
Nov 19 11:05:13 10.243.0.116 [  122.081420]  ? load_module.cold.80+0x38d/0x38d
Nov 19 11:05:13 10.243.0.116 ** 3019 printk messages dropped **
Nov 19 11:05:13 10.243.0.116 [  122.102802]  __die+0x25/0x30
Nov 19 11:05:13 10.243.0.116 ** 3066 printk messages dropped **
Nov 19 11:05:13 10.243.0.116 [  122.124486]  ? is_prefetch.isra.28+0x8c/0x2b0
Nov 19 11:05:13 10.243.0.116 ** 2958 printk messages dropped **
Nov 19 11:05:13 10.243.0.116 [  122.145473]  ? bsearch+0x5e/0x90
Nov 19 11:05:13 10.243.0.116 ** 2960 printk messages dropped **
Nov 19 11:05:13 10.243.0.116 [  122.166469]  exc_invalid_op+0x4f/0xa0
Nov 19 11:05:13 10.243.0.116 ** 2896 printk messages dropped **
Nov 19 11:05:13 10.243.0.116 [  122.187139] RIP: 0010:irqentry_nmi_enter+0x83/0x90
Nov 19 11:05:13 10.243.0.116 ** 2876 printk messages dropped **
Nov 19 11:05:13 10.243.0.116 [  122.207531] RSP: 0018:fffffe000016fca0 EFLAGS: 00010046
Nov 19 11:05:13 10.243.0.116 ** 2902 printk messages dropped **
Nov 19 11:05:13 10.243.0.116 [  122.228173]  ? kallsyms_lookup_buildid+0x9/0x170
Nov 19 11:05:13 10.243.0.116 ** 2869 printk messages dropped **
Nov 19 11:05:13 10.243.0.116 [  122.248551]  ? bsearch+0x5e/0x90
Nov 19 11:05:13 10.243.0.116 ** 2823 printk messages dropped **
Nov 19 11:05:13 10.243.0.116 [  122.268617]  __die+0x25/0x30
Nov 19 11:05:13 10.243.0.116 ** 2717 printk messages dropped **
Nov 19 11:05:13 10.243.0.116 [  122.287816] Code: b1 ff ff 65 ff 05 94 13 f4 76 48 8b 1d 8d a7 0c 01 48 83 eb 08 4c 8d 6b 08 49 81 fd 40 03 1b 8a 74 3e 48 89 df e8 04 7d 3f ff <83> 3b 03 74 1f 48 8d 75 a8 48 89 df e8 93 a3 14 ff 48 8d 73 18 48
Nov 19 11:05:13 10.243.0.116 ** 4177 printk messages dropped **
Nov 19 11:05:13 10.243.0.116 [  122.317589] RBP: fffffe000016ff20 R08: 0000000000000000 R09: 0000000000000000
Nov 19 11:05:13 10.243.0.116 ** 2783 printk messages dropped **
Nov 19 11:05:13 10.243.0.116 [  122.337297] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Nov 19 11:05:13 10.243.0.116 ** 2812 printk messages dropped **
Nov 19 11:05:13 10.243.0.116 [  122.357328] RSP: 0018:fffffe000016ff10 EFLAGS: 00010046
Nov 19 11:05:13 10.243.0.116 ** 2709 printk messages dropped **
Nov 19 11:05:13 10.243.0.116 [  122.376513] RSP: 0018:fffffe000016e858 EFLAGS: 00010046
Nov 19 11:05:13 10.243.0.116 ** 2764 printk messages dropped **
Nov 19 11:05:13 10.243.0.116 [  122.396121] RDX: 0000000000000003 RSI: dffffc0000000000 RDI: fffffffffffffff8
Nov 19 11:05:13 10.243.0.116 ** 2821 printk messages dropped **
Nov 19 11:05:13 10.243.0.116 [  122.416231] RSP: 0018:fffffe000016cff8 EFLAGS: 00010046
Nov 19 11:05:13 10.243.0.116 ** 2828 printk messages dropped **
Nov 19 11:05:13 10.243.0.116 [  122.436349]  ? is_prefetch.isra.28+0x8c/0x2b0
Nov 19 11:05:13 10.243.0.116 ** 2853 printk messages dropped **
Nov 19 11:05:13 10.243.0.116 [  122.456537]  __die_body+0x1f/0x60
Nov 19 11:05:13 10.243.0.116 ** 2846 printk messages dropped **
Nov 19 11:05:13 10.243.0.116 [  122.476751]  do_kern_addr_fault+0x7c/0xb0
Nov 19 11:05:13 10.243.0.116 ** 3258 printk messages dropped **
Nov 19 11:05:13 10.243.0.116 [  122.499901]  ? print_modules+0x88/0x121
Nov 19 11:05:13 10.243.0.116 ** 2799 printk messages dropped **
Nov 19 11:05:13 10.243.0.116 [  122.519806]  __die_body+0x1f/0x60
Nov 19 11:05:13 10.243.0.116 ** 2774 printk messages dropped **
Nov 19 11:05:13 10.243.0.116 [  122.539476] Oops: 0000 [#7699] SMP KASAN PTI IRQ_PIPELINE
Nov 19 11:05:13 10.243.0.116 ** 2727 printk messages dropped **
Nov 19 11:05:13 10.243.0.116 [  122.558733]  __die_body+0x1f/0x60
Nov 19 11:05:13 10.243.0.116 ** 2650 printk messages dropped **
Nov 19 11:05:13 10.243.0.116 [  122.577701] RSP: 0018:fffffe000016ff10 EFLAGS: 00010046
Nov 19 11:05:13 10.243.0.116 ** 2642 printk messages dropped **
Nov 19 11:05:13 10.243.0.116 [  122.596462] Code: b1 ff ff 65 ff 05 94 13 f4 76 48 8b 1d 8d a7 0c 01 48 83 eb 08 4c 8d 6b 08 49 81 fd 40 03 1b 8a 74 3e 48 89 df e8 04 7d 3f ff <83> 3b 03 74 1f 48 8d 75 a8 48 89 df e8 93 a3 14 ff 48 8d 73 18 48
Nov 19 11:05:13 10.243.0.116 ** 4218 printk messages dropped **
Nov 19 11:05:13 10.243.0.116 [  122.626371]  ? load_module.cold.80+0x38d/0x38d
Nov 19 11:05:13 10.243.0.116 ** 2914 printk messages dropped **
Nov 19 11:05:13 10.243.0.116 [  122.647087]  do_kern_addr_fault+0x7c/0xb0
Nov 19 11:05:13 10.243.0.116 ** 2907 printk messages dropped **
Nov 19 11:05:13 10.243.0.116 [  122.667646]  ? kallsyms_lookup_buildid+0x9/0x170
Nov 19 11:05:13 10.243.0.116 ** 2884 printk messages dropped **
Nov 19 11:05:13 10.243.0.116 [  122.688113] RIP: 0010:print_modules+0x88/0x121
Nov 19 11:05:13 10.243.0.116 ** 2884 printk messages dropped **
Nov 19 11:05:13 10.243.0.116 [  122.708617]  ? search_bpf_extables+0x136/0x190
Nov 19 11:05:13 10.243.0.116 ** 2914 printk messages dropped **
Nov 19 11:05:13 10.243.0.116 [  122.729363]  ? kallsyms_lookup_buildid+0x9/0x170
Nov 19 11:05:13 10.243.0.116 ** 2926 printk messages dropped **
Nov 19 11:05:13 10.243.0.116 [  122.750275] BUG: unable to handle page fault for address: fffffffffffffff8
Nov 19 11:05:13 10.243.0.116 ** 2955 printk messages dropped **
Nov 19 11:05:13 10.243.0.116 [  122.771250]  page_fault_oops+0x124/0x3c0
Nov 19 11:05:13 10.243.0.116 ** 2888 printk messages dropped **
Nov 19 11:05:13 10.243.0.116 [  122.791683]  ? print_modules+0x88/0x121
Nov 19 11:05:13 10.243.0.116 ** 2801 printk messages dropped **
Nov 19 11:05:13 10.243.0.116 [  122.811524] RDX: 0000000000000003 RSI: dffffc0000000000 RDI: fffffffffffffff8
Nov 19 11:05:13 10.243.0.116 ** 2791 printk messages dropped **
Nov 19 11:05:13 10.243.0.116 [  122.831307] RAX: 0000000000f00000 RBX: fffffe000016ff00 RCX: fffffe000016c000
Nov 19 11:05:13 10.243.0.116 ** 2802 printk messages dropped **
Nov 19 11:05:13 10.243.0.116 [  122.851167] RIP: 0010:irqentry_nmi_enter+0x83/0x90
Nov 19 11:05:13 10.243.0.116 ** 2805 printk messages dropped **
Nov 19 11:05:13 10.243.0.116 [  122.871100] RSP: 0018:fffffe000016c000 EFLAGS: 00010046
Nov 19 11:05:13 10.243.0.116 ** 2757 printk messages dropped **
Nov 19 11:05:13 10.243.0.116 [  122.890715]  kernelmode_fixup_or_oops+0x135/0x160
Nov 19 11:05:13 10.243.0.116 ** 2721 printk messages dropped **
Nov 19 11:05:13 10.243.0.116 [  122.909976]  ? cmp_ex_sort+0x50/0x50
Nov 19 11:05:14 10.243.0.116 ** 2764 printk messages dropped **
Nov 19 11:05:14 10.243.0.116 [  122.929651]  __die_body+0x1f/0x60
Nov 19 11:05:14 10.243.0.116 ** 2726 printk messages dropped **
Nov 19 11:05:14 10.243.0.116 [  122.948994]  ? print_modules+0x88/0x121
Nov 19 11:05:14 10.243.0.116 ** 2724 printk messages dropped **
Nov 19 11:05:14 10.243.0.116 [  122.968266] FS:  00007fdce1ffd700(0000) GS:ffff8883de480000(0000) knlGS:0000000000000000
Nov 19 11:05:14 10.243.0.116 ** 2806 printk messages dropped **
Nov 19 11:05:14 10.243.0.116 [  122.988286] RDX: 0000000000000003 RSI: dffffc0000000000 RDI: fffffffffffffff8
Nov 19 11:05:14 10.243.0.116 ** 2847 printk messages dropped **
Nov 19 11:05:14 10.243.0.116 [  123.008467]  ? search_bpf_extables+0x136/0x190
Nov 19 11:05:14 10.243.0.116 ** 2911 printk messages dropped **
Nov 19 11:05:14 10.243.0.116 [  123.029034] R13: 0000000000000000 R14: 0000000000000000 R15: ffff88811d198000
Nov 19 11:05:14 10.243.0.116 ** 2945 printk messages dropped **
Nov 19 11:05:14 10.243.0.116 [  123.049993]  ? search_exception_tables+0x58/0x60
Nov 19 11:05:14 10.243.0.116 ** 2955 printk messages dropped **
Nov 19 11:05:14 10.243.0.116 [  123.071112] RSP: 0018:fffffe000016d408 EFLAGS: 00010046
Nov 19 11:05:14 10.243.0.116 ** 2896 printk messages dropped **
Nov 19 11:05:14 10.243.0.116 [  123.091640] RDX: 0000000000000003 RSI: dffffc0000000000 RDI: fffffffffffffff8
Nov 19 11:05:14 10.243.0.116 ** 2877 printk messages dropped **
Nov 19 11:05:14 10.243.0.116 [  123.111928]  ? print_modules+0x88/0x121
Nov 19 11:05:14 10.243.0.116 ** 2877 printk messages dropped **
Nov 19 11:05:14 10.243.0.116 [  123.132370] Code: e0 ee 76 85 c0 75 0c 41 c7 84 24 b8 0d 00 00 00 00 00 00 e8 6f fe ff ff e8 3a 38 19 ff 31 c0 88 d8 5b 41 5c 5d c3 cc cc cc cc <0f> 0b 90 66 2e 0f 1f 84 00 00 00 00 00 55 48 89 e5 40 84 f6 75 3e
Nov 19 11:05:14 10.243.0.116 ** 4470 printk messages dropped **
Nov 19 11:05:14 10.243.0.116 [  123.164375]  ? lock_release+0xd5/0x420
Nov 19 11:05:14 10.243.0.116 ** 3020 printk messages dropped **
Nov 19 11:05:14 10.243.0.116 [  123.185655] BUG: unable to handle page fault for address: fffffffffffffff8
Nov 19 11:05:14 10.243.0.116 ** 3051 printk messages dropped **
Nov 19 11:05:14 10.243.0.116 [  123.207302]  ? kallsyms_lookup_buildid+0x9/0x170
Nov 19 11:05:14 10.243.0.116 ** 3040 printk messages dropped **
Nov 19 11:05:14 10.243.0.116 [  123.228903] RSP: 0018:fffffe000016e448 EFLAGS: 00010046
Nov 19 11:05:14 10.243.0.116 ** 3092 printk messages dropped **
Nov 19 11:05:14 10.243.0.116 [  123.250890] RSP: 0018:fffffe000016fca0 EFLAGS: 00010046
Nov 19 11:05:14 10.243.0.116 ** 3017 printk messages dropped **
Nov 19 11:05:14 10.243.0.116 [  123.272313] RAX: 0000000000000000 RBX: fffffffffffffff8 RCX: ffffffff890e5bcc
Nov 19 11:05:14 10.243.0.116 ** 3110 printk messages dropped **
Nov 19 11:05:14 10.243.0.116 [  123.294462] Code: e0 ee 76 85 c0 75 0c 41 c7 84 24 b8 0d 00 00 00 00 00 00 e8 6f fe ff ff e8 3a 38 19 ff 31 c0 88 d8 5b 41 5c 5d c3 cc cc cc cc <0f> 0b 90 66 2e 0f 1f 84 00 00 00 00 00 55 48 89 e5 40 84 f6 75 3e
Nov 19 11:05:14 10.243.0.116 ** 4803 printk messages dropped **
Nov 19 11:05:14 10.243.0.116 [  123.328403]  __bad_area_nosemaphore+0x247/0x2b0
Nov 19 11:05:14 10.243.0.116 ** 3215 printk messages dropped **
Nov 19 11:05:14 10.243.0.116 [  123.351303]  bad_area_nosemaphore+0x11/0x20
Nov 19 11:05:14 10.243.0.116 ** 3166 printk messages dropped **
Nov 19 11:05:14 10.243.0.116 [  123.373845]  page_fault_oops+0x124/0x3c0
Nov 19 11:05:14 10.243.0.116 ** 3150 printk messages dropped **
Nov 19 11:05:14 10.243.0.116 [  123.396316]  ? spurious_kernel_fault+0x200/0x200
Nov 19 11:05:14 10.243.0.116 ** 3130 printk messages dropped **
Nov 19 11:05:14 10.243.0.116 [  123.418521]  exc_page_fault+0xcf/0xe0
Nov 19 11:05:14 10.243.0.116 ** 3149 printk messages dropped **
Nov 19 11:05:14 10.243.0.116 [  123.440918] RSP: 0018:fffffe000016ff10 EFLAGS: 00010046
Nov 19 11:05:14 10.243.0.116 ** 3185 printk messages dropped **
Nov 19 11:05:14 10.243.0.116 [  123.463632]  ? fixup_exception+0x32/0x420
Nov 19 11:05:14 10.243.0.116 ** 3163 printk messages dropped **
Nov 19 11:05:14 10.243.0.116 [  123.486100]  ? print_modules+0x88/0x121
Nov 19 11:05:14 10.243.0.116 ** 3206 printk messages dropped **
Nov 19 11:05:14 10.243.0.116 [  123.508923] PGD 30ae1a067 P4D 30ae1a067 PUD 30ae1c067 PMD 0
Nov 19 11:05:14 10.243.0.116 ** 3044 printk messages dropped **
Nov 19 11:05:14 10.243.0.116 [  123.530505] Hardware name: Supermicro Super Server/X10SDV-6C+-TLN4F, BIOS 2.1 11/22/2019
Nov 19 11:05:14 10.243.0.116 ** 3117 printk messages dropped **
Nov 19 11:05:14 10.243.0.116 [  123.552614]  __bad_area_nosemaphore+0x247/0x2b0
Nov 19 11:05:14 10.243.0.116 ** 3110 printk messages dropped **
Nov 19 11:05:14 10.243.0.116 [  123.574696] RAX: 0000000000f00000 RBX: fffffe000016ff00 RCX: fffffe000016c000
Nov 19 11:05:14 10.243.0.116 ** 3127 printk messages dropped **
Nov 19 11:05:14 10.243.0.116 [  123.596867] RBP: fffffe000016c008 R08: fffffe000016c080 R09: fffffe000016c1b0
Nov 19 11:05:14 10.243.0.116 ** 3155 printk messages dropped **
Nov 19 11:05:14 10.243.0.116 [  123.619379] BUG: unable to handle page fault for address: fffffffffffffff8
Nov 19 11:05:14 10.243.0.116 ** 3164 printk messages dropped **
Nov 19 11:05:14 10.243.0.116 [  123.641816] RBP: fffffe000016ff20 R08: 0000000000000000 R09: 0000000000000000
Nov 19 11:05:14 10.243.0.116 ** 3231 printk messages dropped **
Nov 19 11:05:14 10.243.0.116 [  123.664757] BUG: unable to handle page fault for address: fffffffffffffff8
Nov 19 11:05:14 10.243.0.116 ** 3198 printk messages dropped **
Nov 19 11:05:14 10.243.0.116 [  123.687415]  ? fixup_exception+0x32/0x420
Nov 19 11:05:14 10.243.0.116 ** 3105 printk messages dropped **
Nov 19 11:05:14 10.243.0.116 [  123.709501]  ? spurious_kernel_fault+0x200/0x200
Nov 19 11:05:14 10.243.0.116 ** 3116 printk messages dropped **
Nov 19 11:05:14 10.243.0.116 [  123.731534]  </#DF>
Nov 19 11:05:14 10.243.0.116 ** 3048 printk messages dropped **
Nov 19 11:05:14 10.243.0.116 [  123.753081] R10: ffff8883de4a7b1b R11: ffffed107bc94f63 R12: 1fffffc00002dc07
Nov 19 11:05:14 10.243.0.116 ** 3163 printk messages dropped **
Nov 19 11:05:14 10.243.0.116 [  123.775756] #PF: error_code(0x0000) - not-present page
Nov 19 11:05:14 10.243.0.116 ** 3093 printk messages dropped **
Nov 19 11:05:14 10.243.0.116 [  123.797603] RSP: 0018:fffffe000016f488 EFLAGS: 00010046
Nov 19 11:05:14 10.243.0.116 ** 3076 printk messages dropped **
Nov 19 11:05:14 10.243.0.116 [  123.819356]  ? print_modules+0x88/0x121
Nov 19 11:05:14 10.243.0.116 ** 2999 printk messages dropped **
Nov 19 11:05:14 10.243.0.116 [  123.840564] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Nov 19 11:05:14 10.243.0.116 ** 2926 printk messages dropped **
Nov 19 11:05:14 10.243.0.116 [  123.861409] R10: ffff8883de4a7b1b R11: ffffed107bc94f63 R12: 1fffffc00002de0f
Nov 19 11:05:14 10.243.0.116 ** 3418 printk messages dropped **
Nov 19 11:05:14 10.243.0.116 [  123.885738]  ? print_modules+0x88/0x121
Nov 19 11:05:14 10.243.0.116 ** 2956 printk messages dropped **
Nov 19 11:05:14 10.243.0.116 [  123.906623]  exc_invalid_op+0x4f/0xa0
Nov 19 11:05:15 10.243.0.116 ** 2937 printk messages dropped **
Nov 19 11:05:15 10.243.0.116 [  123.927399] CR2: fffffffffffffff8 CR3: 0000000102568002 CR4: 00000000003706e0
Nov 19 11:05:15 10.243.0.116 ** 3029 printk messages dropped **
Nov 19 11:05:15 10.243.0.116 [  123.948948]  ? print_modules+0x88/0x121
Nov 19 11:05:15 10.243.0.116 ** 3008 printk messages dropped **
Nov 19 11:05:15 10.243.0.116 [  123.970365] RAX: 0000000000f00000 RBX: fffffe000016ff00 RCX: fffffe000016c000
Nov 19 11:05:15 10.243.0.116 ** 3057 printk messages dropped **
Nov 19 11:05:15 10.243.0.116 [  123.992214]  exc_page_fault+0xcf/0xe0
Nov 19 11:05:15 10.243.0.116 ** 3051 printk messages dropped **
Nov 19 11:05:15 10.243.0.116 [  124.013964] RAX: 0000000000f00000 RBX: fffffe000016ff00 RCX: fffffe000016c000
Nov 19 11:05:15 10.243.0.116 ** 3094 printk messages dropped **
Nov 19 11:05:15 10.243.0.116 [  124.036029] RSP: 0018:fffffe000016ec68 EFLAGS: 00010046
Nov 19 11:05:15 10.243.0.116 ** 3089 printk messages dropped **
Nov 19 11:05:15 10.243.0.116 [  124.057817]  <#DF>
Nov 19 11:05:15 10.243.0.116 ** 3066 printk messages dropped **
Nov 19 11:05:15 10.243.0.116 [  124.079718]  ? spurious_kernel_fault+0x200/0x200
Nov 19 11:05:15 10.243.0.116 ** 3017 printk messages dropped **
Nov 19 11:05:15 10.243.0.116 [  124.101175] R13: 0000000000000000 R14: 0000000000000000 R15: ffff88811d198000
Nov 19 11:05:15 10.243.0.116 ** 2980 printk messages dropped **
Nov 19 11:05:15 10.243.0.116 [  124.122308] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Nov 19 11:05:15 10.243.0.116 ** 2984 printk messages dropped **
Nov 19 11:05:15 10.243.0.116 [  124.143569]  __bad_area_nosemaphore+0x247/0x2b0
Nov 19 11:05:15 10.243.0.116 ** 2942 printk messages dropped **
Nov 19 11:05:15 10.243.0.116 [  124.164529]  ? load_module.cold.80+0x38d/0x38d
Nov 19 11:05:15 10.243.0.116 ** 2892 printk messages dropped **
Nov 19 11:05:15 10.243.0.116 [  124.184991]  ? print_modules+0x88/0x121
Nov 19 11:05:15 10.243.0.116 ** 2828 printk messages dropped **
Nov 19 11:05:15 10.243.0.116 [  124.204921]  ? print_modules+0x88/0x121
Nov 19 11:05:15 10.243.0.116 ** 2814 printk messages dropped **
Nov 19 11:05:15 10.243.0.116 [  124.224938] IRQ stage: Linux
Nov 19 11:05:15 10.243.0.116 ** 2793 printk messages dropped **
Nov 19 11:05:15 10.243.0.116 [  124.244711]  ? lock_release+0xd5/0x420
Nov 19 11:05:15 10.243.0.116 ** 2748 printk messages dropped **
Nov 19 11:05:15 10.243.0.116 [  124.264249] Code: b1 ff ff 65 ff 05 94 13 f4 76 48 8b 1d 8d a7 0c 01 48 83 eb 08 4c 8d 6b 08 49 81 fd 40 03 1b 8a 74 3e 48 89 df e8 04 7d 3f ff <83> 3b 03 74 1f 48 8d 75 a8 48 89 df e8 93 a3 14 ff 48 8d 73 18 48
Nov 19 11:05:15 10.243.0.116 ** 4276 printk messages dropped **
Nov 19 11:05:15 10.243.0.116 [  124.294556]  ? print_modules+0x88/0x121
Nov 19 11:05:15 10.243.0.116 ** 2861 printk messages dropped **
Nov 19 11:05:15 10.243.0.116 [  124.314971]  ? print_modules+0x88/0x121
Nov 19 11:05:15 10.243.0.116 ** 2852 printk messages dropped **
Nov 19 11:05:15 10.243.0.116 [  124.335229] R10: fffffe000016c447 R11: fffffbc00002d888 R12: fffffe000016c1b0
Nov 19 11:05:15 10.243.0.116 ** 2886 printk messages dropped **
Nov 19 11:05:15 10.243.0.116 [  124.355683]  exc_double_fault+0x9f/0x110
Nov 19 11:05:15 10.243.0.116 ** 2834 printk messages dropped **
Nov 19 11:05:15 10.243.0.116 [  124.375804] Code: b1 ff ff 65 ff 05 94 13 f4 76 48 8b 1d 8d a7 0c 01 48 83 eb 08 4c 8d 6b 08 49 81 fd 40 03 1b 8a 74 3e 48 89 df e8 04 7d 3f ff <83> 3b 03 74 1f 48 8d 75 a8 48 89 df e8 93 a3 14 ff 48 8d 73 18 48
Nov 19 11:05:15 10.243.0.116 ** 4322 printk messages dropped **
Nov 19 11:05:15 10.243.0.116 [  124.406518]  kernelmode_fixup_or_oops+0x135/0x160
Nov 19 11:05:15 10.243.0.116 ** 2949 printk messages dropped **
Nov 19 11:05:15 10.243.0.116 [  124.427481]  handle_invalid_op+0x3c/0x50
Nov 19 11:05:15 10.243.0.116 ** 2916 printk messages dropped **
Nov 19 11:05:15 10.243.0.116 [  124.448176]  ? print_modules+0x88/0x121
Nov 19 11:05:15 10.243.0.116 ** 2908 printk messages dropped **
Nov 19 11:05:15 10.243.0.116 [  124.468868] Code: b1 ff ff 65 ff 05 94 13 f4 76 48 8b 1d 8d a7 0c 01 48 83 eb 08 4c 8d 6b 08 49 81 fd 40 03 1b 8a 74 3e 48 89 df e8 04 7d 3f ff <83> 3b 03 74 1f 48 8d 75 a8 48 89 df e8 93 a3 14 ff 48 8d 73 18 48
Nov 19 11:05:15 10.243.0.116 ** 4474 printk messages dropped **
Nov 19 11:05:15 10.243.0.116 [  124.500768]  ? cmp_ex_search+0x1e/0x40
Nov 19 11:05:15 10.243.0.116 ** 2996 printk messages dropped **
Nov 19 11:05:15 10.243.0.116 [  124.522172] RSP: 0018:fffffe000016e448 EFLAGS: 00010046
Nov 19 11:05:15 10.243.0.116 ** 3021 printk messages dropped **
Nov 19 11:05:15 10.243.0.116 [  124.543650]  ? search_exception_tables+0x58/0x60
Nov 19 11:05:15 10.243.0.116 ** 2975 printk messages dropped **
Nov 19 11:05:15 10.243.0.116 [  124.564814] R13: 0000000000000000 R14: 0000000000000046 R15: 0000000000000004
Nov 19 11:05:15 10.243.0.116 ** 3060 printk messages dropped **
Nov 19 11:05:15 10.243.0.116 [  124.586641]  ? bsearch+0x5e/0x90
Nov 19 11:05:15 10.243.0.116 ** 3045 printk messages dropped **
Nov 19 11:05:15 10.243.0.116 [  124.608216] R13: fffffe000016c66a R14: ffffffff890e5bcc R15: fffffe000016c0c0
Nov 19 11:05:15 10.243.0.116 ** 2995 printk messages dropped **
Nov 19 11:05:15 10.243.0.116 [  124.629413]  ? spurious_kernel_fault+0x200/0x200
Nov 19 11:05:15 10.243.0.116 ** 3040 printk messages dropped **
Nov 19 11:05:15 10.243.0.116 [  124.651061]  do_kern_addr_fault+0x7c/0xb0
Nov 19 11:05:15 10.243.0.116 ** 3001 printk messages dropped **
Nov 19 11:05:15 10.243.0.116 [  124.672281]  ? lock_release+0xd5/0x420
Nov 19 11:05:15 10.243.0.116 ** 3005 printk messages dropped **
Nov 19 11:05:15 10.243.0.116 [  124.693771]  kernelmode_fixup_or_oops+0x135/0x160
Nov 19 11:05:15 10.243.0.116 ** 3017 printk messages dropped **
Nov 19 11:05:15 10.243.0.116 [  124.715262]  ? bsearch+0x5e/0x90
Nov 19 11:05:15 10.243.0.116 ** 2995 printk messages dropped **
Nov 19 11:05:15 10.243.0.116 [  124.736502] RSP: 0018:fffffe000016d408 EFLAGS: 00010046
Nov 19 11:05:15 10.243.0.116 ** 3016 printk messages dropped **
Nov 19 11:05:15 10.243.0.116 [  124.757841]  ? is_prefetch.isra.28+0x8c/0x2b0
Nov 19 11:05:15 10.243.0.116 ** 3045 printk messages dropped **
Nov 19 11:05:15 10.243.0.116 [  124.779382] R10: ffff8883de4a7b1b R11: ffffed107bc94f63 R12: 1fffffc00002de91
Nov 19 11:05:15 10.243.0.116 ** 2928 printk messages dropped **
Nov 19 11:05:15 10.243.0.116 [  124.800172] R13: 0000000000000000 R14: 0000000000000000 R15: ffff88811d198000
Nov 19 11:05:15 10.243.0.116 ** 2962 printk messages dropped **
Nov 19 11:05:15 10.243.0.116 [  124.821435] BUG: unable to handle page fault for address: fffffffffffffff8
Nov 19 11:05:15 10.243.0.116 ** 3073 printk messages dropped **
Nov 19 11:05:15 10.243.0.116 [  124.843338] RIP: 0010:print_modules+0x88/0x121
Nov 19 11:05:15 10.243.0.116 ** 3114 printk messages dropped **
Nov 19 11:05:15 10.243.0.116 [  124.865446] RSP: 0018:fffffe000016ff10 EFLAGS: 00010046
Nov 19 11:05:15 10.243.0.116 ** 2999 printk messages dropped **
Nov 19 11:05:15 10.243.0.116 [  124.886619] R10: 0000000000000000 R11: 0000000000000000 R12: ffff888118d24780
Nov 19 11:05:16 10.243.0.116 ** 2988 printk messages dropped **
Nov 19 11:05:16 10.243.0.116 [  124.907803] Code: 4e 2a 00 8b 45 c4 c6 03 00 48 83 c4 18 5b 41 5c 41 5d 41 5e 41 5f 5d c3 cc cc cc cc 0f 1f 44 00 00 55 48 89 e5 41 57 49 89 d7 <41> 56 49 89 f6 41 55 49 89 cd 41 54 49 89 fc 49 8d 79 7f 53 4c 89
Nov 19 11:05:16 10.243.0.116 ** 4504 printk messages dropped **
Nov 19 11:05:16 10.243.0.116 [  124.939699] RBP: fffffe000016dca0 R08: ffffed107bc94f64 R09: ffffed107bc94f64
Nov 19 11:05:16 10.243.0.116 ** 3148 printk messages dropped **
Nov 19 11:05:16 10.243.0.116 [  124.962083]  __bad_area_nosemaphore+0x247/0x2b0
Nov 19 11:05:16 10.243.0.116 ** 3194 printk messages dropped **
Nov 19 11:05:16 10.243.0.116 [  124.984676]  ? print_modules+0x88/0x121
Nov 19 11:05:16 10.243.0.116 ** 3041 printk messages dropped **
Nov 19 11:05:16 10.243.0.116 [  125.006342] RBP: fffffe000016d070 R08: ffffed107bc94f64 R09: ffffed107bc94f64
Nov 19 11:05:16 10.243.0.116 ** 3005 printk messages dropped **
Nov 19 11:05:16 10.243.0.116 [  125.027853]  __die_body+0x1f/0x60
Nov 19 11:05:16 10.243.0.116 ** 3173 printk messages dropped **
Nov 19 11:05:16 10.243.0.116 [  125.050368] R13: ffff888118d24780 R14: 0000000102568002 R15: 0000000000000000
Nov 19 11:05:16 10.243.0.116 ** 3075 printk messages dropped **
Nov 19 11:05:16 10.243.0.116 [  125.072164]  exc_invalid_op+0x4f/0xa0
Nov 19 11:05:16 10.243.0.116 ** 3044 printk messages dropped **
Nov 19 11:05:16 10.243.0.116 [  125.093860] Thread overran stack, or stack corrupted
Nov 19 11:05:16 10.243.0.116 ** 3029 printk messages dropped **
Nov 19 11:05:16 10.243.0.116 [  125.115284] R10: 0000000000000000 R11: 0000000000000000 R12: ffff888118d24780
Nov 19 11:05:16 10.243.0.116 ** 3063 printk messages dropped **
Nov 19 11:05:16 10.243.0.116 [  125.137078] R13: 0000000000000000 R14: 0000000000000046 R15: 0000000000000004
Nov 19 11:05:16 10.243.0.116 ** 3083 printk messages dropped **
Nov 19 11:05:16 10.243.0.116 [  125.158977]  ? print_modules+0x88/0x121
Nov 19 11:05:16 10.243.0.116 ** 3104 printk messages dropped **
Nov 19 11:05:16 10.243.0.116 [  125.181019]  do_kern_addr_fault+0x7c/0xb0
Nov 19 11:05:16 10.243.0.116 ** 3087 printk messages dropped **
Nov 19 11:05:16 10.243.0.116 [  125.202958] RSP: 0018:fffffe000016e858 EFLAGS: 00010046
Nov 19 11:05:16 10.243.0.116 ** 3100 printk messages dropped **
Nov 19 11:05:16 10.243.0.116 [  125.225029] #PF: supervisor read access in kernel mode
Nov 19 11:05:16 10.243.0.116 ** 3177 printk messages dropped **
Nov 19 11:05:16 10.243.0.116 [  125.247594] R10: ffff8883de4a7b1b R11: ffffed107bc94f63 R12: 1fffffc00002df94
Nov 19 11:05:16 10.243.0.116 ** 3190 printk messages dropped **
Nov 19 11:05:16 10.243.0.116 [  125.270255]  ? lock_release+0xd5/0x420
Nov 19 11:05:16 10.243.0.116 ** 3636 printk messages dropped **
Nov 19 11:05:16 10.243.0.116 [  125.296198]  ? bsearch+0x5e/0x90
Nov 19 11:05:16 10.243.0.116 ** 3198 printk messages dropped **
Nov 19 11:05:16 10.243.0.116 [  125.318717] RIP: 0010:print_modules+0x88/0x121
Nov 19 11:05:16 10.243.0.116 ** 3066 printk messages dropped **
Nov 19 11:05:16 10.243.0.116 [  125.340480] IRQ stage: Linux
Nov 19 11:05:16 10.243.0.116 ** 3012 printk messages dropped **
Nov 19 11:05:16 10.243.0.116 [  125.362028]  do_trap+0x126/0x150
Nov 19 11:05:16 10.243.0.116 ** 2966 printk messages dropped **
Nov 19 11:05:16 10.243.0.116 [  125.383075]  bad_area_nosemaphore+0x11/0x20
Nov 19 11:05:16 10.243.0.116 ** 2918 printk messages dropped **
Nov 19 11:05:16 10.243.0.116 [  125.403844]  ? is_prefetch.isra.28+0x8c/0x2b0
Nov 19 11:05:16 10.243.0.116 ** 2896 printk messages dropped **
Nov 19 11:05:16 10.243.0.116 [  125.424296]  ? print_modules+0x88/0x121
Nov 19 11:05:16 10.243.0.116 ** 2814 printk messages dropped **
Nov 19 11:05:16 10.243.0.116 [  125.444234] Code: 4e 2a 00 8b 45 c4 c6 03 00 48 83 c4 18 5b 41 5c 41 5d 41 5e 41 5f 5d c3 cc cc cc cc 0f 1f 44 00 00 55 48 89 e5 41 57 49 89 d7 <41> 56 49 89 f6 41 55 49 89 cd 41 54 49 89 fc 49 8d 79 7f 53 4c 89
Nov 19 11:05:16 10.243.0.116 ** 4146 printk messages dropped **
Nov 19 11:05:16 10.243.0.116 [  125.473743]  do_trap+0x126/0x150
Nov 19 11:05:16 10.243.0.116 ** 2769 printk messages dropped **
Nov 19 11:05:16 10.243.0.116 [  125.493413]  ? is_prefetch.isra.28+0x8c/0x2b0
Nov 19 11:05:16 10.243.0.116 ** 2739 printk messages dropped **
Nov 19 11:05:16 10.243.0.116 [  125.512891] RSP: 0018:fffffe000016dc28 EFLAGS: 00010046
Nov 19 11:05:16 10.243.0.116 ** 2768 printk messages dropped **
Nov 19 11:05:16 10.243.0.116 [  125.532399]  ? fixup_exception+0x32/0x420
Nov 19 11:05:16 10.243.0.116 ** 2716 printk messages dropped **
Nov 19 11:05:16 10.243.0.116 [  125.551749] R10: fffffe000016c447 R11: fffffbc00002d888 R12: fffffe000016c1b0
Nov 19 11:05:16 10.243.0.116 ** 2752 printk messages dropped **
Nov 19 11:05:16 10.243.0.116 [  125.571326]  ? fixup_exception+0x32/0x420
Nov 19 11:05:16 10.243.0.116 ** 2735 printk messages dropped **
Nov 19 11:05:16 10.243.0.116 [  125.590705]  ? print_modules+0x88/0x121
Nov 19 11:05:16 10.243.0.116 ** 2706 printk messages dropped **
Nov 19 11:05:16 10.243.0.116 [  125.609948] RSP: 0018:fffffe000016ff10 EFLAGS: 00010046
Nov 19 11:05:16 10.243.0.116 ** 2722 printk messages dropped **
Nov 19 11:05:16 10.243.0.116 [  125.629292]  ? print_modules+0x88/0x121
Nov 19 11:05:16 10.243.0.116 ** 2664 printk messages dropped **
Nov 19 11:05:16 10.243.0.116 [  125.648353]  bad_area_nosemaphore+0x11/0x20
Nov 19 11:05:16 10.243.0.116 ** 2641 printk messages dropped **
Nov 19 11:05:16 10.243.0.116 [  125.667272] RSP: 0018:fffffe000016ff10 EFLAGS: 00010046
Nov 19 11:05:16 10.243.0.116 ** 2542 printk messages dropped **
Nov 19 11:05:16 10.243.0.116 [  125.685246] RIP: 0010:print_modules+0x88/0x121
Nov 19 11:05:16 10.243.0.116 ** 2646 printk messages dropped **
Nov 19 11:05:16 10.243.0.116 [  125.703968] R10: ffff8883de4a7b1b R11: ffffed107bc94f63 R12: 1fffffc00002dd0b
Nov 19 11:05:16 10.243.0.116 ** 2660 printk messages dropped **
Nov 19 11:05:16 10.243.0.116 [  125.722879]  ? fixup_exception+0x32/0x420
Nov 19 11:05:16 10.243.0.116 ** 2640 printk messages dropped **
Nov 19 11:05:16 10.243.0.116 [  125.741667] RAX: 0000000000000000 RBX: fffffffffffffff8 RCX: ffffffff890e5bcc
Nov 19 11:05:16 10.243.0.116 ** 2679 printk messages dropped **
Nov 19 11:05:16 10.243.0.116 [  125.760683]  ? print_modules+0x88/0x121
Nov 19 11:05:16 10.243.0.116 ** 2684 printk messages dropped **
Nov 19 11:05:16 10.243.0.116 [  125.779793]  ? is_prefetch.isra.28+0x8c/0x2b0
Nov 19 11:05:16 10.243.0.116 ** 2682 printk messages dropped **
Nov 19 11:05:16 10.243.0.116 [  125.798825] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Nov 19 11:05:16 10.243.0.116 ** 2777 printk messages dropped **
Nov 19 11:05:16 10.243.0.116 [  125.818394]  ? kallsyms_lookup_buildid+0x9/0x170
Nov 19 11:05:16 10.243.0.116 ** 2752 printk messages dropped **
Nov 19 11:05:16 10.243.0.116 [  125.837979] R10: ffff8883de4a7b1b R11: ffffed107bc94f63 R12: 1fffffc00002dc89
Nov 19 11:05:16 10.243.0.116 ** 2859 printk messages dropped **
Nov 19 11:05:16 10.243.0.116 [  125.858230] RSP: 0018:fffffe000016ff10 EFLAGS: 00010046
Nov 19 11:05:16 10.243.0.116 ** 2889 printk messages dropped **
Nov 19 11:05:16 10.243.0.116 [  125.878629]  ? irqentry_nmi_enter+0x83/0x90
Nov 19 11:05:16 10.243.0.116 ** 2882 printk messages dropped **
Nov 19 11:05:16 10.243.0.116 [  125.899203] RBP: fffffe000016fd18 R08: ffffed107bc94f64 R09: ffffed107bc94f64
Nov 19 11:05:17 10.243.0.116 ** 2881 printk messages dropped **
Nov 19 11:05:17 10.243.0.116 [  125.919638]  ? search_exception_tables+0x58/0x60
Nov 19 11:05:17 10.243.0.116 ** 2792 printk messages dropped **
Nov 19 11:05:17 10.243.0.116 [  125.939449]  ? print_modules+0x88/0x121
Nov 19 11:05:17 10.243.0.116 ** 2792 printk messages dropped **
Nov 19 11:05:17 10.243.0.116 [  125.959367]  ? load_module.cold.80+0x38d/0x38d
Nov 19 11:05:17 10.243.0.116 ** 2794 printk messages dropped **
Nov 19 11:05:17 10.243.0.116 [  125.979180]  ? print_modules+0x88/0x121
Nov 19 11:05:17 10.243.0.116 ** 2782 printk messages dropped **
Nov 19 11:05:17 10.243.0.116 [  125.998934]  ? bsearch+0x5e/0x90
Nov 19 11:05:17 10.243.0.116 ** 2779 printk messages dropped **
Nov 19 11:05:17 10.243.0.116 [  126.018750] RBP: fffffe000016ff20 R08: 0000000000000000 R09: 0000000000000000
Nov 19 11:05:17 10.243.0.116 ** 2741 printk messages dropped **
Nov 19 11:05:17 10.243.0.116 [  126.038297]  kernelmode_fixup_or_oops+0x135/0x160
Nov 19 11:05:17 10.243.0.116 ** 2772 printk messages dropped **
Nov 19 11:05:17 10.243.0.116 [  126.058093] RBP: fffffe000016ff20 R08: 0000000000000000 R09: 0000000000000000
Nov 19 11:05:17 10.243.0.116 ** 2817 printk messages dropped **
Nov 19 11:05:17 10.243.0.116 [  126.078032]  ? bsearch+0x5e/0x90
Nov 19 11:05:17 10.243.0.116 ** 2747 printk messages dropped **
Nov 19 11:05:17 10.243.0.116 [  126.097492] RAX: 0000000000000000 RBX: fffffffffffffff8 RCX: ffffffff890e5bcc
Nov 19 11:05:17 10.243.0.116 ** 2827 printk messages dropped **
Nov 19 11:05:17 10.243.0.116 [  126.117567] RSP: 0018:fffffe000016e038 EFLAGS: 00010046
Nov 19 11:05:17 10.243.0.116 ** 2890 printk messages dropped **
Nov 19 11:05:17 10.243.0.116 [  126.138057]  exc_page_fault+0xcf/0xe0
Nov 19 11:05:17 10.243.0.116 ** 2860 printk messages dropped **
Nov 19 11:05:17 10.243.0.116 [  126.158350]  do_kern_addr_fault+0x7c/0xb0
Nov 19 11:05:17 10.243.0.116 ** 2862 printk messages dropped **
Nov 19 11:05:17 10.243.0.116 [  126.178688]  exc_page_fault+0xcf/0xe0
Nov 19 11:05:17 10.243.0.116 ** 2797 printk messages dropped **
Nov 19 11:05:17 10.243.0.116 [  126.198516] RAX: 0000000000f00000 RBX: fffffe000016ff00 RCX: fffffe000016c000
Nov 19 11:05:17 10.243.0.116 ** 2811 printk messages dropped **
Nov 19 11:05:17 10.243.0.116 [  126.218508] R10: ffff8883de4a7b1b R11: ffffed107bc94f63 R12: 1fffffc00002de91
Nov 19 11:05:17 10.243.0.116 ** 2825 printk messages dropped **
Nov 19 11:05:17 10.243.0.116 [  126.238677] RAX: 0000000000000000 RBX: fffffffffffffff8 RCX: ffffffff890e5bcc
Nov 19 11:05:17 10.243.0.116 ** 2856 printk messages dropped **
Nov 19 11:05:17 10.243.0.116 [  126.259047]  exc_page_fault+0xcf/0xe0
Nov 19 11:05:17 10.243.0.116 ** 2911 printk messages dropped **
Nov 19 11:05:17 10.243.0.116 [  126.279790]  ? lock_release+0xd5/0x420
Nov 19 11:05:17 10.243.0.116 ** 2824 printk messages dropped **
Nov 19 11:05:17 10.243.0.116 [  126.299784]  ? kallsyms_lookup_buildid+0x9/0x170
Nov 19 11:05:17 10.243.0.116 ** 2828 printk messages dropped **
Nov 19 11:05:17 10.243.0.116 [  126.319835]  ? kallsyms_lookup_buildid+0x9/0x170
Nov 19 11:05:17 10.243.0.116 ** 2816 printk messages dropped **
Nov 19 11:05:17 10.243.0.116 [  126.339828] RSP: 0018:fffffe000016ff10 EFLAGS: 00010046
Nov 19 11:05:17 10.243.0.116 ** 2783 printk messages dropped **
Nov 19 11:05:17 10.243.0.116 [  126.359528]  ? load_module.cold.80+0x38d/0x38d
Nov 19 11:05:17 10.243.0.116 ** 2727 printk messages dropped **
Nov 19 11:05:17 10.243.0.116 [  126.378911]  ? print_modules+0x88/0x121
Nov 19 11:05:17 10.243.0.116 ** 2770 printk messages dropped **
Nov 19 11:05:17 10.243.0.116 [  126.398454] R10: ffff8883de4a7b1b R11: ffffed107bc94f63 R12: 1fffffc00002df13
Nov 19 11:05:17 10.243.0.116 ** 2688 printk messages dropped **
Nov 19 11:05:17 10.243.0.116 [  126.417430]  exc_page_fault+0xcf/0xe0
Nov 19 11:05:17 10.243.0.116 ** 2754 printk messages dropped **
Nov 19 11:05:17 10.243.0.116 [  126.437032]  __die+0x25/0x30
Nov 19 11:05:17 10.243.0.116 ** 2716 printk messages dropped **
Nov 19 11:05:17 10.243.0.116 [  126.456202] RBP: fffffe000016e0b0 R08: ffffed107bc94f64 R09: ffffed107bc94f64
Nov 19 11:05:17 10.243.0.116 ** 2774 printk messages dropped **
Nov 19 11:05:17 10.243.0.116 [  126.475884] RSP: 0018:fffffe000016f488 EFLAGS: 00010046
Nov 19 11:05:17 10.243.0.116 ** 2828 printk messages dropped **
Nov 19 11:05:17 10.243.0.116 [  126.496083] RSP: 0018:fffffe000016ec68 EFLAGS: 00010046
Nov 19 11:05:17 10.243.0.116 ** 2791 printk messages dropped **
Nov 19 11:05:17 10.243.0.116 [  126.515906] Code: e0 ee 76 85 c0 75 0c 41 c7 84 24 b8 0d 00 00 00 00 00 00 e8 6f fe ff ff e8 3a 38 19 ff 31 c0 88 d8 5b 41 5c 5d c3 cc cc cc cc <0f> 0b 90 66 2e 0f 1f 84 00 00 00 00 00 55 48 89 e5 40 84 f6 75 3e
Nov 19 11:05:17 10.243.0.116 ** 4304 printk messages dropped **
Nov 19 11:05:17 10.243.0.116 [  126.546513]  bad_area_nosemaphore+0x11/0x20
Nov 19 11:05:17 10.243.0.116 ** 2911 printk messages dropped **
Nov 19 11:05:17 10.243.0.116 [  126.567270]  __die+0x25/0x30
Nov 19 11:05:17 10.243.0.116 ** 3335 printk messages dropped **
Nov 19 11:05:17 10.243.0.116 [  126.591050] RSP: 0018:fffffe000016ec68 EFLAGS: 00010046
Nov 19 11:05:17 10.243.0.116 ** 2828 printk messages dropped **
Nov 19 11:05:17 10.243.0.116 [  126.611157] RSP: 0018:fffffe000016e448 EFLAGS: 00010046
Nov 19 11:05:17 10.243.0.116 ** 2836 printk messages dropped **
Nov 19 11:05:17 10.243.0.116 [  126.631271]  ? load_module.cold.80+0x38d/0x38d
Nov 19 11:05:17 10.243.0.116 ** 2889 printk messages dropped **
Nov 19 11:05:17 10.243.0.116 [  126.651843] RBP: fffffe000016dca0 R08: ffffed107bc94f64 R09: ffffed107bc94f64
Nov 19 11:05:17 10.243.0.116 ** 2891 printk messages dropped **
Nov 19 11:05:17 10.243.0.116 [  126.672417] RSP: 0018:fffffe000016dc28 EFLAGS: 00010046
Nov 19 11:05:17 10.243.0.116 ** 2959 printk messages dropped **
Nov 19 11:05:17 10.243.0.116 [  126.693470] Code: b1 ff ff 65 ff 05 94 13 f4 76 48 8b 1d 8d a7 0c 01 48 83 eb 08 4c 8d 6b 08 49 81 fd 40 03 1b 8a 74 3e 48 89 df e8 04 7d 3f ff <83> 3b 03 74 1f 48 8d 75 a8 48 89 df e8 93 a3 14 ff 48 8d 73 18 48
Nov 19 11:05:17 10.243.0.116 ** 4526 printk messages dropped **
Nov 19 11:05:17 10.243.0.116 [  126.725592]  exc_invalid_op+0x4f/0xa0
Nov 19 11:05:17 10.243.0.116 ** 2965 printk messages dropped **
Nov 19 11:05:17 10.243.0.116 [  126.746758] RIP: 0010:print_modules+0x88/0x121
Nov 19 11:05:17 10.243.0.116 ** 3005 printk messages dropped **
Nov 19 11:05:17 10.243.0.116 [  126.768094]  __die_body+0x1f/0x60
Nov 19 11:05:17 10.243.0.116 ** 2994 printk messages dropped **
Nov 19 11:05:17 10.243.0.116 [  126.789459]  __die+0x25/0x30
Nov 19 11:05:17 10.243.0.116 ** 2949 printk messages dropped **
Nov 19 11:05:17 10.243.0.116 [  126.810472] RBP: fffffe000016c008 R08: fffffe000016c080 R09: fffffe000016c1b0
Nov 19 11:05:17 10.243.0.116 ** 2951 printk messages dropped **
Nov 19 11:05:17 10.243.0.116 [  126.831386] RAX: 0000000000000000 RBX: fffffffffffffff8 RCX: ffffffff890e5bcc
Nov 19 11:05:17 10.243.0.116 ** 3027 printk messages dropped **
Nov 19 11:05:17 10.243.0.116 [  126.852887] RDX: 0000000000000003 RSI: dffffc0000000000 RDI: fffffffffffffff8
Nov 19 11:05:17 10.243.0.116 ** 3058 printk messages dropped **
Nov 19 11:05:17 10.243.0.116 [  126.874551] R10: fffffe000016c447 R11: fffffbc00002d888 R12: fffffe000016c1b0
Nov 19 11:05:17 10.243.0.116 ** 3140 printk messages dropped **
Nov 19 11:05:17 10.243.0.116 [  126.896980]  __bad_area_nosemaphore+0x247/0x2b0
Nov 19 11:05:18 10.243.0.116 ** 3065 printk messages dropped **
Nov 19 11:05:18 10.243.0.116 [  126.918795]  ? kallsyms_lookup_buildid+0x9/0x170
Nov 19 11:05:18 10.243.0.116 ** 3190 printk messages dropped **
Nov 19 11:05:18 10.243.0.116 [  126.941472]  ? load_module.cold.80+0x38d/0x38d
Nov 19 11:05:18 10.243.0.116 ** 3167 printk messages dropped **
Nov 19 11:05:18 10.243.0.116 [  126.963916] R13: ffff888118d24780 R14: 0000000102568002 R15: 0000000000000000
Nov 19 11:05:18 10.243.0.116 ** 3138 printk messages dropped **
Nov 19 11:05:18 10.243.0.116 [  126.986295]  kernelmode_fixup_or_oops+0x135/0x160
Nov 19 11:05:18 10.243.0.116 ** 3127 printk messages dropped **
Nov 19 11:05:18 10.243.0.116 [  127.008320]  ? print_modules+0x88/0x121
Nov 19 11:05:18 10.243.0.116 ** 3118 printk messages dropped **
Nov 19 11:05:18 10.243.0.116 [  127.030498] BUG: unable to handle page fault for address: fffffffffffffff8
Nov 19 11:05:18 10.243.0.116 ** 3049 printk messages dropped **
Nov 19 11:05:18 10.243.0.116 [  127.052085]  ? print_modules+0x88/0x121
Nov 19 11:05:18 10.243.0.116 ** 2985 printk messages dropped **
Nov 19 11:05:18 10.243.0.116 [  127.073315] RDX: fffffe000016c0c0 RSI: fffffe000016c100 RDI: ffffffff890e5bcc
Nov 19 11:05:18 10.243.0.116 ** 3027 printk messages dropped **
Nov 19 11:05:18 10.243.0.116 [  127.094727]  __die_body+0x1f/0x60
Nov 19 11:05:18 10.243.0.116 ** 3051 printk messages dropped **
Nov 19 11:05:18 10.243.0.116 [  127.116510] R13: fffffe000016c66a R14: ffffffff890e5bcc R15: fffffe000016c0c0
Nov 19 11:05:18 10.243.0.116 ** 3022 printk messages dropped **
Nov 19 11:05:18 10.243.0.116 [  127.137909]  ? load_module.cold.80+0x38d/0x38d
Nov 19 11:05:18 10.243.0.116 ** 3065 printk messages dropped **
Nov 19 11:05:18 10.243.0.116 [  127.159818]  handle_invalid_op+0x3c/0x50
Nov 19 11:05:18 10.243.0.116 ** 3002 printk messages dropped **
Nov 19 11:05:18 10.243.0.116 [  127.181148] RSP: 0018:fffffe000016e038 EFLAGS: 00010046
Nov 19 11:05:18 10.243.0.116 ** 3026 printk messages dropped **
Nov 19 11:05:18 10.243.0.116 [  127.202760] RSP: 0018:fffffe000016f078 EFLAGS: 00010046
Nov 19 11:05:18 10.243.0.116 ** 2954 printk messages dropped **
Nov 19 11:05:18 10.243.0.116 [  127.223704]  bad_area_nosemaphore+0x11/0x20
Nov 19 11:05:18 10.243.0.116 ** 2853 printk messages dropped **
Nov 19 11:05:18 10.243.0.116 [  127.244082]  ? page_fault_oops+0x3c0/0x3c0
Nov 19 11:05:18 10.243.0.116 ** 2849 printk messages dropped **
Nov 19 11:05:18 10.243.0.116 [  127.264375]  ? print_modules+0x88/0x121
Nov 19 11:05:18 10.243.0.116 ** 2788 printk messages dropped **
Nov 19 11:05:18 10.243.0.116 [  127.284164] RSP: 0018:fffffe000016e038 EFLAGS: 00010046
Nov 19 11:05:18 10.243.0.116 ** 2830 printk messages dropped **
Nov 19 11:05:18 10.243.0.116 [  127.304281] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Nov 19 11:05:18 10.243.0.116 ** 2892 printk messages dropped **
Nov 19 11:05:18 10.243.0.116 [  127.324934] R13: 0000000000000000 R14: 0000000000000000 R15: ffff88811d198000
Nov 19 11:05:18 10.243.0.116 ** 2929 printk messages dropped **
Nov 19 11:05:18 10.243.0.116 [  127.345859] RDX: 0000000000000003 RSI: dffffc0000000000 RDI: fffffffffffffff8
Nov 19 11:05:18 10.243.0.116 ** 2919 printk messages dropped **
Nov 19 11:05:18 10.243.0.116 [  127.366734]  bad_area_nosemaphore+0x11/0x20
Nov 19 11:05:18 10.243.0.116 ** 2870 printk messages dropped **
Nov 19 11:05:18 10.243.0.116 [  127.387188] RBP: fffffe000016dca0 R08: ffffed107bc94f64 R09: ffffed107bc94f64
Nov 19 11:05:18 10.243.0.116 ** 2853 printk messages dropped **
Nov 19 11:05:18 10.243.0.116 [  127.407431] RSP: 0018:fffffe000016d818 EFLAGS: 00010046
Nov 19 11:05:18 10.243.0.116 ** 2874 printk messages dropped **
Nov 19 11:05:18 10.243.0.116 [  127.427901] RAX: 0000000000000000 RBX: 1fffffc00002d804 RCX: fffffe000016c040
Nov 19 11:05:18 10.243.0.116 ** 2899 printk messages dropped **
Nov 19 11:05:18 10.243.0.116 [  127.448511]  ? print_modules+0x88/0x121
Nov 19 11:05:18 10.243.0.116 ** 2898 printk messages dropped **
Nov 19 11:05:18 10.243.0.116 [  127.469268] BUG: unable to handle page fault for address: fffffffffffffff8
Nov 19 11:05:18 10.243.0.116 ** 2899 printk messages dropped **
Nov 19 11:05:18 10.243.0.116 [  127.489964] Oops: 0000 [#11071] SMP KASAN PTI IRQ_PIPELINE
Nov 19 11:05:18 10.243.0.116 ** 2910 printk messages dropped **
Nov 19 11:05:18 10.243.0.116 [  127.510510] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Nov 19 11:05:18 10.243.0.116 ** 2962 printk messages dropped **
Nov 19 11:05:18 10.243.0.116 [  127.531595]  ? print_modules+0x88/0x121
Nov 19 11:05:18 10.243.0.116 ** 2991 printk messages dropped **
Nov 19 11:05:18 10.243.0.116 [  127.553017] R13: 0000000000000000 R14: 0000000000000000 R15: ffff88811d198000
Nov 19 11:05:18 10.243.0.116 ** 2955 printk messages dropped **
Nov 19 11:05:18 10.243.0.116 [  127.573985] RSP: 0018:fffffe000016f488 EFLAGS: 00010046
Nov 19 11:05:18 10.243.0.116 ** 2980 printk messages dropped **
Nov 19 11:05:18 10.243.0.116 [  127.595113] RSP: 0018:fffffe000016ff10 EFLAGS: 00010046
Nov 19 11:05:18 10.243.0.116 ** 2961 printk messages dropped **
Nov 19 11:05:18 10.243.0.116 [  127.616138] RIP: 0010:print_modules+0x88/0x121
Nov 19 11:05:18 10.243.0.116 ** 2935 printk messages dropped **
Nov 19 11:05:18 10.243.0.116 [  127.637146]  ? print_modules+0x88/0x121
Nov 19 11:05:18 10.243.0.116 ** 2953 printk messages dropped **
Nov 19 11:05:18 10.243.0.116 [  127.658075] Code: b1 ff ff 65 ff 05 94 13 f4 76 48 8b 1d 8d a7 0c 01 48 83 eb 08 4c 8d 6b 08 49 81 fd 40 03 1b 8a 74 3e 48 89 df e8 04 7d 3f ff <83> 3b 03 74 1f 48 8d 75 a8 48 89 df e8 93 a3 14 ff 48 8d 73 18 48
Nov 19 11:05:18 10.243.0.116 ** 4616 printk messages dropped **
Nov 19 11:05:18 10.243.0.116 [  127.690799]  ? print_modules+0x88/0x121
Nov 19 11:05:18 10.243.0.116 ** 3084 printk messages dropped **
Nov 19 11:05:18 10.243.0.116 [  127.712761]  ? kallsyms_lookup_buildid+0x9/0x170
Nov 19 11:05:18 10.243.0.116 ** 3057 printk messages dropped **
Nov 19 11:05:18 10.243.0.116 [  127.734475] BUG: unable to handle page fault for address: fffffffffffffff8
Nov 19 11:05:18 10.243.0.116 ** 3115 printk messages dropped **
Nov 19 11:05:18 10.243.0.116 [  127.756476]  ? print_modules+0x88/0x121
Nov 19 11:05:18 10.243.0.116 ** 3108 printk messages dropped **
Nov 19 11:05:18 10.243.0.116 [  127.778513] RSP: 0018:fffffe000016e448 EFLAGS: 00010046
Nov 19 11:05:18 10.243.0.116 ** 3172 printk messages dropped **
Nov 19 11:05:18 10.243.0.116 [  127.801076]  ? fixup_exception+0x32/0x420
Nov 19 11:05:18 10.243.0.116 ** 3155 printk messages dropped **
Nov 19 11:05:18 10.243.0.116 [  127.823703]  ? search_bpf_extables+0x136/0x190
Nov 19 11:05:18 10.243.0.116 ** 3059 printk messages dropped **
Nov 19 11:05:18 10.243.0.116 [  127.845560] RIP: 0010:print_modules+0x88/0x121
Nov 19 11:05:18 10.243.0.116 ** 3083 printk messages dropped **
Nov 19 11:05:18 10.243.0.116 [  127.867500]  ? print_modules+0x88/0x121
Nov 19 11:05:18 10.243.0.116 ** 3032 printk messages dropped **
Nov 19 11:05:18 10.243.0.116 [  127.889061] RIP: 0010:kallsyms_lookup_buildid+0x9/0x170
Nov 19 11:05:18 10.243.0.116 ** 2965 printk messages dropped **
Nov 19 11:05:18 10.243.0.116 [  127.909957] R10: ffff8883de4a7b1b R11: ffffed107bc94f63 R12: 1fffffc00002df94
Nov 19 11:05:19 10.243.0.116 ** 2936 printk messages dropped **
Nov 19 11:05:19 10.243.0.116 [  127.930751] BUG: unable to handle page fault for address: fffffffffffffff8
Nov 19 11:05:19 10.243.0.116 ** 2984 printk messages dropped **
Nov 19 11:05:19 10.243.0.116 [  127.951852] #PF: supervisor read access in kernel mode
Nov 19 11:05:19 10.243.0.116 ** 3448 printk messages dropped **
Nov 19 11:05:19 10.243.0.116 [  127.976276]  ? load_module.cold.80+0x38d/0x38d
Nov 19 11:05:19 10.243.0.116 ** 2975 printk messages dropped **
Nov 19 11:05:19 10.243.0.116 [  127.997559]  ? search_exception_tables+0x58/0x60
Nov 19 11:05:19 10.243.0.116 ** 2934 printk messages dropped **
Nov 19 11:05:19 10.243.0.116 [  128.018474] RSP: 0018:fffffe000016c000 EFLAGS: 00010046
Nov 19 11:05:19 10.243.0.116 ** 2919 printk messages dropped **
Nov 19 11:05:19 10.243.0.116 [  128.039321] R10: ffff8883de4a7b1b R11: ffffed107bc94f63 R12: 1fffffc00002dc89
Nov 19 11:05:19 10.243.0.116 ** 2967 printk messages dropped **
Nov 19 11:05:19 10.243.0.116 [  128.060491]  ? print_modules+0x88/0x121
Nov 19 11:05:19 10.243.0.116 ** 2962 printk messages dropped **
Nov 19 11:05:19 10.243.0.116 [  128.081548]  ? load_module.cold.80+0x38d/0x38d
Nov 19 11:05:19 10.243.0.116 ** 2886 printk messages dropped **
Nov 19 11:05:19 10.243.0.116 [  128.101963] RSP: 0018:fffffe000016f488 EFLAGS: 00010046
Nov 19 11:05:19 10.243.0.116 ** 2861 printk messages dropped **
Nov 19 11:05:19 10.243.0.116 [  128.122243] RSP: 0018:fffffe000016f078 EFLAGS: 00010046
Nov 19 11:05:19 10.243.0.116 ** 2879 printk messages dropped **
Nov 19 11:05:19 10.243.0.116 [  128.142539]  ? is_prefetch.isra.28+0x8c/0x2b0
Nov 19 11:05:19 10.243.0.116 ** 2898 printk messages dropped **
Nov 19 11:05:19 10.243.0.116 [  128.163135]  ? print_modules+0x88/0x121
Nov 19 11:05:19 10.243.0.116 ** 2874 printk messages dropped **
Nov 19 11:05:19 10.243.0.116 [  128.183617] RDX: 0000000000000003 RSI: dffffc0000000000 RDI: fffffffffffffff8
Nov 19 11:05:19 10.243.0.116 ** 2760 printk messages dropped **
Nov 19 11:05:19 10.243.0.116 [  128.203339]  ? search_exception_tables+0x58/0x60
Nov 19 11:05:19 10.243.0.116 ** 2783 printk messages dropped **
Nov 19 11:05:19 10.243.0.116 [  128.223112]  __die+0x25/0x30
Nov 19 11:05:19 10.243.0.116 ** 2780 printk messages dropped **
Nov 19 11:05:19 10.243.0.116 [  128.242914]  ? page_fault_oops+0x3c0/0x3c0
Nov 19 11:05:19 10.243.0.116 ** 2700 printk messages dropped **
Nov 19 11:05:19 10.243.0.116 [  128.262082]  ? lock_release+0xd5/0x420
Nov 19 11:05:19 10.243.0.116 ** 2705 printk messages dropped **
Nov 19 11:05:19 10.243.0.116 [  128.281289]  page_fault_oops+0x124/0x3c0
Nov 19 11:05:19 10.243.0.116 ** 2710 printk messages dropped **
Nov 19 11:05:19 10.243.0.116 [  128.300649] RSP: 0018:fffffe000016c000 EFLAGS: 00010046
Nov 19 11:05:19 10.243.0.116 ** 2595 printk messages dropped **
Nov 19 11:05:19 10.243.0.116 [  128.319097]  ? print_modules+0x88/0x121
Nov 19 11:05:19 10.243.0.116 ** 2720 printk messages dropped **
Nov 19 11:05:19 10.243.0.116 [  128.338439] RSP: 0018:fffffe000016ff10 EFLAGS: 00010046
Nov 19 11:05:19 10.243.0.116 ** 2675 printk messages dropped **
Nov 19 11:05:19 10.243.0.116 [  128.357436] Code: b1 ff ff 65 ff 05 94 13 f4 76 48 8b 1d 8d a7 0c 01 48 83 eb 08 4c 8d 6b 08 49 81 fd 40 03 1b 8a 74 3e 48 89 df e8 04 7d 3f ff <83> 3b 03 74 1f 48 8d 75 a8 48 89 df e8 93 a3 14 ff 48 8d 73 18 48
Nov 19 11:05:19 10.243.0.116 ** 4242 printk messages dropped **
Nov 19 11:05:19 10.243.0.116 [  128.387520]  ? page_fault_oops+0x3c0/0x3c0
Nov 19 11:05:19 10.243.0.116 ** 2874 printk messages dropped **
Nov 19 11:05:19 10.243.0.116 [  128.407870] R10: ffff8883de4a7b1b R11: ffffed107bc94f63 R12: 1fffffc00002dd0b
Nov 19 11:05:19 10.243.0.116 ** 2812 printk messages dropped **
Nov 19 11:05:19 10.243.0.116 [  128.427755]  ? print_modules+0x88/0x121
Nov 19 11:05:19 10.243.0.116 ** 2795 printk messages dropped **
Nov 19 11:05:19 10.243.0.116 [  128.447542] Call Trace:
Nov 19 11:05:19 10.243.0.116 ** 2735 printk messages dropped **
Nov 19 11:05:19 10.243.0.116 [  128.466915] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Nov 19 11:05:19 10.243.0.116 ** 2761 printk messages dropped **
Nov 19 11:05:19 10.243.0.116 [  128.486534] Hardware name: Supermicro Super Server/X10SDV-6C+-TLN4F, BIOS 2.1 11/22/2019
Nov 19 11:05:19 10.243.0.116 ** 2794 printk messages dropped **
Nov 19 11:05:19 10.243.0.116 [  128.506347]  ? print_modules+0x88/0x121
Nov 19 11:05:19 10.243.0.116 ** 2804 printk messages dropped **
Nov 19 11:05:19 10.243.0.116 [  128.526462] Hardware name: Supermicro Super Server/X10SDV-6C+-TLN4F, BIOS 2.1 11/22/2019
Nov 19 11:05:19 10.243.0.116 ** 2823 printk messages dropped **
Nov 19 11:05:19 10.243.0.116 [  128.546427]  ? kallsyms_lookup_buildid+0x9/0x170
Nov 19 11:05:19 10.243.0.116 ** 2829 printk messages dropped **
Nov 19 11:05:19 10.243.0.116 [  128.566508]  __die_body+0x1f/0x60
Nov 19 11:05:19 10.243.0.116 ** 2818 printk messages dropped **
Nov 19 11:05:19 10.243.0.116 [  128.586499] RSP: 0018:fffffe000016e858 EFLAGS: 00010046
Nov 19 11:05:19 10.243.0.116 ** 2805 printk messages dropped **
Nov 19 11:05:19 10.243.0.116 [  128.606247]  __die_body+0x1f/0x60
Nov 19 11:05:19 10.243.0.116 ** 2796 printk messages dropped **
Nov 19 11:05:19 10.243.0.116 [  128.626076]  __die+0x25/0x30
Nov 19 11:05:19 10.243.0.116 ** 2809 printk messages dropped **
Nov 19 11:05:19 10.243.0.116 [  128.646127]  ? cmp_ex_search+0x1e/0x40
Nov 19 11:05:19 10.243.0.116 ** 2752 printk messages dropped **
Nov 19 11:05:19 10.243.0.116 [  128.665510] R13: 0000000000000000 R14: 0000000000000000 R15: ffff88811d198000
Nov 19 11:05:19 10.243.0.116 ** 2708 printk messages dropped **
Nov 19 11:05:19 10.243.0.116 [  128.684724] BUG: unable to handle page fault for address: fffffffffffffff8
Nov 19 11:05:19 10.243.0.116 ** 2800 printk messages dropped **
Nov 19 11:05:19 10.243.0.116 [  128.704711]  page_fault_oops+0x124/0x3c0
Nov 19 11:05:19 10.243.0.116 ** 2821 printk messages dropped **
Nov 19 11:05:19 10.243.0.116 [  128.724626] R13: 0000000000000000 R14: 0000000000000000 R15: ffff88811d198000
Nov 19 11:05:19 10.243.0.116 ** 2823 printk messages dropped **
Nov 19 11:05:19 10.243.0.116 [  128.744746] RSP: 0018:fffffe000016e448 EFLAGS: 00010046
Nov 19 11:05:19 10.243.0.116 ** 2960 printk messages dropped **
Nov 19 11:05:19 10.243.0.116 [  128.765709] RSP: 0018:fffffe000016ec68 EFLAGS: 00010046
Nov 19 11:05:19 10.243.0.116 ** 2942 printk messages dropped **
Nov 19 11:05:19 10.243.0.116 [  128.786707]  ? cmp_ex_search+0x1e/0x40
Nov 19 11:05:19 10.243.0.116 ** 2918 printk messages dropped **
Nov 19 11:05:19 10.243.0.116 [  128.807576]  ? print_modules+0x88/0x121
Nov 19 11:05:19 10.243.0.116 ** 2921 printk messages dropped **
Nov 19 11:05:19 10.243.0.116 [  128.828298] RSP: 0018:fffffe000016f898 EFLAGS: 00010046
Nov 19 11:05:19 10.243.0.116 ** 2872 printk messages dropped **
Nov 19 11:05:19 10.243.0.116 [  128.848637]  __die+0x25/0x30
Nov 19 11:05:19 10.243.0.116 ** 2719 printk messages dropped **
Nov 19 11:05:19 10.243.0.116 [  128.867959] RAX: 0000000000000000 RBX: fffffffffffffff8 RCX: ffffffff890e5bcc
Nov 19 11:05:19 10.243.0.116 ** 2780 printk messages dropped **
Nov 19 11:05:19 10.243.0.116 [  128.887778]  __die_body+0x1f/0x60
Nov 19 11:05:19 10.243.0.116 ** 2720 printk messages dropped **
Nov 19 11:05:19 10.243.0.116 [  128.907153] RSP: 0018:fffffe000016e858 EFLAGS: 00010046
Nov 19 11:05:20 10.243.0.116 ** 2731 printk messages dropped **
Nov 19 11:05:20 10.243.0.116 [  128.926522]  kernelmode_fixup_or_oops+0x135/0x160
Nov 19 11:05:20 10.243.0.116 ** 2737 printk messages dropped **
Nov 19 11:05:20 10.243.0.116 [  128.945909] RSP: 0018:fffffe000016e858 EFLAGS: 00010046
Nov 19 11:05:20 10.243.0.116 ** 2759 printk messages dropped **
Nov 19 11:05:20 10.243.0.116 [  128.965519]  ? print_modules+0x88/0x121
Nov 19 11:05:20 10.243.0.116 ** 2692 printk messages dropped **
Nov 19 11:05:20 10.243.0.116 [  128.984785] Modules linked in: xt_CHECKSUM xt_MASQUERADE tun netconsole ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 xt_conntrack ebtable_nat ebtable_broute ip6table_nat ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_nat iptable_mangle iptable_security iptable_raw nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 libcrc32c ip_set nfnetlink ebtable_filter ebtables â–’â–’#023â–’â–’â–’â–’â–’â–’tâ–’#034â–’â–’â–’â–’â–’tâ–’#034â–’â–’â–’â–’â–’b#023â–’â–’â–’â–’â–’(FOKX)
Nov 19 11:05:20 10.243.0.116 ** 7077 printk messages dropped **
Nov 19 11:05:20 10.243.0.116 [  129.035399]  ? fixup_exception+0x32/0x420
Nov 19 11:05:20 10.243.0.116 ** 2909 printk messages dropped **
Nov 19 11:05:20 10.243.0.116 [  129.056057]  ? print_modules+0x88/0x121
Nov 19 11:05:20 10.243.0.116 ** 2922 printk messages dropped **
Nov 19 11:05:20 10.243.0.116 [  129.076627] RAX: 0000000000000000 RBX: fffffffffffffff8 RCX: ffffffff890e5bcc
Nov 19 11:05:20 10.243.0.116 ** 2928 printk messages dropped **
Nov 19 11:05:20 10.243.0.116 [  129.097533] RDX: 0000000000000003 RSI: dffffc0000000000 RDI: fffffffffffffff8
Nov 19 11:05:20 10.243.0.116 ** 3077 printk messages dropped **
Nov 19 11:05:20 10.243.0.116 [  129.119375]  ? print_modules+0x88/0x121
Nov 19 11:05:20 10.243.0.116 ** 3034 printk messages dropped **
Nov 19 11:05:20 10.243.0.116 [  129.140873] R13: 0000000000000000 R14: 0000000000000000 R15: ffff88811d198000
Nov 19 11:05:20 10.243.0.116 ** 3142 printk messages dropped **
Nov 19 11:05:20 10.243.0.116 [  129.163097]  ? print_modules+0x88/0x121
Nov 19 11:05:20 10.243.0.116 ** 3064 printk messages dropped **
Nov 19 11:05:20 10.243.0.116 [  129.184841] R13: ffff888118d24780 R14: 0000000102568002 R15: 0000000000000000
Nov 19 11:05:20 10.243.0.116 ** 3024 printk messages dropped **
Nov 19 11:05:20 10.243.0.116 [  129.206278]  __die_body+0x1f/0x60
Nov 19 11:05:20 10.243.0.116 ** 3049 printk messages dropped **
Nov 19 11:05:20 10.243.0.116 [  129.227852]  __die_body+0x1f/0x60
Nov 19 11:05:20 10.243.0.116 ** 2981 printk messages dropped **
Nov 19 11:05:20 10.243.0.116 [  129.249051] RAX: 0000000000000000 RBX: 1fffffc00002d804 RCX: fffffe000016c040
Nov 19 11:05:20 10.243.0.116 ** 2968 printk messages dropped **
Nov 19 11:05:20 10.243.0.116 [  129.270221]  ? bsearch+0x5e/0x90
Nov 19 11:05:20 10.243.0.116 ** 2993 printk messages dropped **
Nov 19 11:05:20 10.243.0.116 [  129.291567]  exc_invalid_op+0x4f/0xa0
Nov 19 11:05:20 10.243.0.116 ** 3404 printk messages dropped **
Nov 19 11:05:20 10.243.0.116 [  129.315761] RSP: 0018:fffffe000016e038 EFLAGS: 00010046
Nov 19 11:05:20 10.243.0.116 ** 2970 printk messages dropped **
Nov 19 11:05:20 10.243.0.116 [  129.336821] R13: 0000000000000000 R14: 0000000000000000 R15: ffff88811d198000
Nov 19 11:05:20 10.243.0.116 ** 3035 printk messages dropped **
Nov 19 11:05:20 10.243.0.116 [  129.358377]  ? spurious_kernel_fault+0x200/0x200
Nov 19 11:05:20 10.243.0.116 ** 3005 printk messages dropped **
Nov 19 11:05:20 10.243.0.116 [  129.379824] RBP: fffffe000016e0b0 R08: ffffed107bc94f64 R09: ffffed107bc94f64
Nov 19 11:05:20 10.243.0.116 ** 3091 printk messages dropped **
Nov 19 11:05:20 10.243.0.116 [  129.401867] RDX: 0000000000000003 RSI: dffffc0000000000 RDI: fffffffffffffff8
Nov 19 11:05:20 10.243.0.116 ** 3081 printk messages dropped **
Nov 19 11:05:20 10.243.0.116 [  129.423789] RAX: 0000000000000000 RBX: fffffffffffffff8 RCX: ffffffff890e5bcc
Nov 19 11:05:20 10.243.0.116 ** 3180 printk messages dropped **
Nov 19 11:05:20 10.243.0.116 [  129.446513] RSP: 0018:fffffe000016d818 EFLAGS: 00010046
Nov 19 11:05:20 10.243.0.116 ** 3184 printk messages dropped **
Nov 19 11:05:20 10.243.0.116 [  129.469085]  __bad_area_nosemaphore+0x247/0x2b0
Nov 19 11:05:20 10.243.0.116 ** 3191 printk messages dropped **
Nov 19 11:05:20 10.243.0.116 [  129.491800] RBP: fffffe000016ece0 R08: ffffed107bc94f64 R09: ffffed107bc94f64
Nov 19 11:05:20 10.243.0.116 ** 3213 printk messages dropped **
Nov 19 11:05:20 10.243.0.116 [  129.514569] RSP: 0018:fffffe000016fca0 EFLAGS: 00010046
Nov 19 11:05:20 10.243.0.116 ** 3076 printk messages dropped **
Nov 19 11:05:20 10.243.0.116 [  129.536405] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Nov 19 11:05:20 10.243.0.116 ** 3009 printk messages dropped **
Nov 19 11:05:20 10.243.0.116 [  129.557776] RSP: 0018:fffffe000016ff10 EFLAGS: 00010046
Nov 19 11:05:20 10.243.0.116 ** 3023 printk messages dropped **
Nov 19 11:05:20 10.243.0.116 [  129.579352]  bad_area_nosemaphore+0x11/0x20
Nov 19 11:05:20 10.243.0.116 ** 3048 printk messages dropped **
Nov 19 11:05:20 10.243.0.116 [  129.600951]  __bad_area_nosemaphore+0x247/0x2b0
Nov 19 11:05:20 10.243.0.116 ** 3018 printk messages dropped **
Nov 19 11:05:20 10.243.0.116 [  129.622384] RIP: 0010:irqentry_nmi_enter+0x83/0x90
Nov 19 11:05:20 10.243.0.116 ** 2937 printk messages dropped **
Nov 19 11:05:20 10.243.0.116 [  129.643380] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Nov 19 11:05:20 10.243.0.116 ** 3000 printk messages dropped **
Nov 19 11:05:20 10.243.0.116 [  129.664755]  page_fault_oops+0x124/0x3c0
Nov 19 11:05:20 10.243.0.116 ** 2971 printk messages dropped **
Nov 19 11:05:20 10.243.0.116 [  129.685808]  ? search_exception_tables+0x58/0x60
Nov 19 11:05:20 10.243.0.116 ** 2946 printk messages dropped **
Nov 19 11:05:20 10.243.0.116 [  129.706748] #PF: error_code(0x0000) - not-present page
Nov 19 11:05:20 10.243.0.116 ** 2914 printk messages dropped **
Nov 19 11:05:20 10.243.0.116 [  129.727528] Call Trace:
Nov 19 11:05:20 10.243.0.116 ** 2957 printk messages dropped **
Nov 19 11:05:20 10.243.0.116 [  129.748426] RBP: fffffe000016e8d0 R08: ffffed107bc94f64 R09: ffffed107bc94f64
Nov 19 11:05:20 10.243.0.116 ** 2972 printk messages dropped **
Nov 19 11:05:20 10.243.0.116 [  129.769497]  ? cmp_ex_search+0x1e/0x40
Nov 19 11:05:20 10.243.0.116 ** 2951 printk messages dropped **
Nov 19 11:05:20 10.243.0.116 [  129.790557]  ? print_modules+0x88/0x121
Nov 19 11:05:20 10.243.0.116 ** 2927 printk messages dropped **
Nov 19 11:05:20 10.243.0.116 [  129.811359]  ? print_modules+0x88/0x121
Nov 19 11:05:20 10.243.0.116 ** 2941 printk messages dropped **
Nov 19 11:05:20 10.243.0.116 [  129.832298] RDX: 0000000000000003 RSI: dffffc0000000000 RDI: fffffffffffffff8
Nov 19 11:05:20 10.243.0.116 ** 2963 printk messages dropped **
Nov 19 11:05:20 10.243.0.116 [  129.853251] RSP: 0018:fffffe000016e448 EFLAGS: 00010046
Nov 19 11:05:20 10.243.0.116 ** 2979 printk messages dropped **
Nov 19 11:05:20 10.243.0.116 [  129.874498]  ? page_fault_oops+0x3c0/0x3c0
Nov 19 11:05:20 10.243.0.116 ** 2947 printk messages dropped **
Nov 19 11:05:20 10.243.0.116 [  129.895326]  ? print_modules+0x88/0x121
Nov 19 11:05:21 10.243.0.116 ** 2895 printk messages dropped **
Nov 19 11:05:21 10.243.0.116 [  129.915992]  ? print_modules+0x88/0x121
Nov 19 11:05:21 10.243.0.116 ** 2792 printk messages dropped **
Nov 19 11:05:21 10.243.0.116 [  129.935773] R10: ffff8883de4a7b1b R11: ffffed107bc94f63 R12: 1fffffc00002dd0b
Nov 19 11:05:21 10.243.0.116 ** 2784 printk messages dropped **
Nov 19 11:05:21 10.243.0.116 [  129.955520] RIP: 0010:print_modules+0x88/0x121
Nov 19 11:05:21 10.243.0.116 ** 2797 printk messages dropped **
Nov 19 11:05:21 10.243.0.116 [  129.975338]  ? is_prefetch.isra.28+0x8c/0x2b0
Nov 19 11:05:21 10.243.0.116 ** 2765 printk messages dropped **
Nov 19 11:05:21 10.243.0.116 [  129.994969]  ? search_bpf_extables+0x136/0x190
Nov 19 11:05:21 10.243.0.116 ** 2764 printk messages dropped **
Nov 19 11:05:21 10.243.0.116 [  130.014477]  ? lock_release+0xd5/0x420
Nov 19 11:05:21 10.243.0.116 ** 2682 printk messages dropped **
Nov 19 11:05:21 10.243.0.116 [  130.033469] RDX: 0000000000000000 RSI: 0000000000000000 RDI: fffffe000016ff58
Nov 19 11:05:21 10.243.0.116 ** 2666 printk messages dropped **
Nov 19 11:05:21 10.243.0.116 [  130.052353]  ? irqentry_nmi_enter+0x83/0x90
Nov 19 11:05:21 10.243.0.116 ** 2657 printk messages dropped **
Nov 19 11:05:21 10.243.0.116 [  130.071212] Code: b1 ff ff 65 ff 05 94 13 f4 76 48 8b 1d 8d a7 0c 01 48 83 eb 08 4c 8d 6b 08 49 81 fd 40 03 1b 8a 74 3e 48 89 df e8 04 7d 3f ff <83> 3b 03 74 1f 48 8d 75 a8 48 89 df e8 93 a3 14 ff 48 8d 73 18 48
Nov 19 11:05:21 10.243.0.116 ** 4143 printk messages dropped **
Nov 19 11:05:21 10.243.0.116 [  130.100664]  __die+0x25/0x30
Nov 19 11:05:21 10.243.0.116 ** 2786 printk messages dropped **
Nov 19 11:05:21 10.243.0.116 [  130.120460] RDX: 0000000000000003 RSI: dffffc0000000000 RDI: fffffffffffffff8
Nov 19 11:05:21 10.243.0.116 ** 2791 printk messages dropped **
Nov 19 11:05:21 10.243.0.116 [  130.140241] RBP: fffffe000016dca0 R08: ffffed107bc94f64 R09: ffffed107bc94f64
Nov 19 11:05:21 10.243.0.116 ** 2853 printk messages dropped **
Nov 19 11:05:21 10.243.0.116 [  130.160434]  ? irqentry_nmi_enter+0x83/0x90
Nov 19 11:05:21 10.243.0.116 ** 2853 printk messages dropped **
Nov 19 11:05:21 10.243.0.116 [  130.180869] R13: 0000000000000000 R14: 0000000000000000 R15: ffff88811d198000
Nov 19 11:05:21 10.243.0.116 ** 2889 printk messages dropped **
Nov 19 11:05:21 10.243.0.116 [  130.201496] RSP: 0018:fffffe000016f898 EFLAGS: 00010046
Nov 19 11:05:21 10.243.0.116 ** 2918 printk messages dropped **
Nov 19 11:05:21 10.243.0.116 [  130.222195]  ? fixup_exception+0x32/0x420
Nov 19 11:05:21 10.243.0.116 ** 2924 printk messages dropped **
Nov 19 11:05:21 10.243.0.116 [  130.243027] RAX: 0000000000f00000 RBX: fffffe000016ff00 RCX: fffffe000016c000
Nov 19 11:05:21 10.243.0.116 ** 2908 printk messages dropped **
Nov 19 11:05:21 10.243.0.116 [  130.263579] R13: fffffe000016c66a R14: ffffffff890e5bcc R15: fffffe000016c0c0
Nov 19 11:05:21 10.243.0.116 ** 3019 printk messages dropped **
Nov 19 11:05:21 10.243.0.116 [  130.285103] R13: 0000000000000000 R14: 0000000000000000 R15: ffff88811d198000
Nov 19 11:05:21 10.243.0.116 ** 3042 printk messages dropped **
Nov 19 11:05:21 10.243.0.116 [  130.306617]  ? search_bpf_extables+0x136/0x190
Nov 19 11:05:21 10.243.0.116 ** 3073 printk messages dropped **
Nov 19 11:05:21 10.243.0.116 [  130.328446]  page_fault_oops+0x124/0x3c0
Nov 19 11:05:21 10.243.0.116 ** 3050 printk messages dropped **
Nov 19 11:05:21 10.243.0.116 [  130.350006] RBP: fffffe000016f910 R08: ffffed107bc94f64 R09: ffffed107bc94f64
Nov 19 11:05:21 10.243.0.116 ** 2984 printk messages dropped **
Nov 19 11:05:21 10.243.0.116 [  130.371360] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Nov 19 11:05:21 10.243.0.116 ** 3147 printk messages dropped **
Nov 19 11:05:21 10.243.0.116 [  130.393584]  __bad_area_nosemaphore+0x247/0x2b0
Nov 19 11:05:21 10.243.0.116 ** 3127 printk messages dropped **
Nov 19 11:05:21 10.243.0.116 [  130.415886] R13: 0000000000000000 R14: 0000000000000000 R15: ffff88811d198000
Nov 19 11:05:21 10.243.0.116 ** 3274 printk messages dropped **
Nov 19 11:05:21 10.243.0.116 [  130.439177] RSP: 0018:fffffe000016ff10 EFLAGS: 00010046
Nov 19 11:05:21 10.243.0.116 ** 3154 printk messages dropped **
Nov 19 11:05:21 10.243.0.116 [  130.461515]  ? lock_release+0xd5/0x420
Nov 19 11:05:21 10.243.0.116 ** 3174 printk messages dropped **
Nov 19 11:05:21 10.243.0.116 [  130.484105]  ? print_modules+0x88/0x121
Nov 19 11:05:21 10.243.0.116 ** 3143 printk messages dropped **
Nov 19 11:05:21 10.243.0.116 [  130.506386] R10: fffffe000016c447 R11: fffffbc00002d888 R12: fffffe000016c1b0
Nov 19 11:05:21 10.243.0.116 ** 3198 printk messages dropped **
Nov 19 11:05:21 10.243.0.116 [  130.529139]  do_kern_addr_fault+0x7c/0xb0
Nov 19 11:05:21 10.243.0.116 ** 3218 printk messages dropped **
Nov 19 11:05:21 10.243.0.116 [  130.552075] Code: b1 ff ff 65 ff 05 94 13 f4 76 48 8b 1d 8d a7 0c 01 48 83 eb 08 4c 8d 6b 08 49 81 fd 40 03 1b 8a 74 3e 48 89 df e8 04 7d 3f ff <83> 3b 03 74 1f 48 8d 75 a8 48 89 df e8 93 a3 14 ff 48 8d 73 18 48
Nov 19 11:05:21 10.243.0.116 ** 4926 printk messages dropped **
Nov 19 11:05:21 10.243.0.116 [  130.586869] RSP: 0018:fffffe000016ff10 EFLAGS: 00010046
Nov 19 11:05:21 10.243.0.116 ** 3268 printk messages dropped **
Nov 19 11:05:21 10.243.0.116 [  130.610173]  ? is_prefetch.isra.28+0x8c/0x2b0
Nov 19 11:05:21 10.243.0.116 ** 3191 printk messages dropped **
Nov 19 11:05:21 10.243.0.116 [  130.632847]  do_kern_addr_fault+0x7c/0xb0
Nov 19 11:05:21 10.243.0.116 ** 3207 printk messages dropped **
Nov 19 11:05:21 10.243.0.116 [  130.655706]  ? search_bpf_extables+0x136/0x190
Nov 19 11:05:21 10.243.0.116 ** 3097 printk messages dropped **
Nov 19 11:05:21 10.243.0.116 [  130.677602] RBP: fffffe000016e0b0 R08: ffffed107bc94f64 R09: ffffed107bc94f64
Nov 19 11:05:21 10.243.0.116 ** 3556 printk messages dropped **
Nov 19 11:05:21 10.243.0.116 [  130.702753] IRQ stage: Linux
Nov 19 11:05:21 10.243.0.116 ** 3148 printk messages dropped **
Nov 19 11:05:21 10.243.0.116 [  130.725133]  ? cmp_ex_search+0x1e/0x40
Nov 19 11:05:21 10.243.0.116 ** 3075 printk messages dropped **
Nov 19 11:05:21 10.243.0.116 [  130.746888] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Nov 19 11:05:21 10.243.0.116 ** 3025 printk messages dropped **
Nov 19 11:05:21 10.243.0.116 [  130.768384] R13: 0000000000000000 R14: 0000000000000046 R15: 0000000000000004
Nov 19 11:05:21 10.243.0.116 ** 3092 printk messages dropped **
Nov 19 11:05:21 10.243.0.116 [  130.790402]  kernelmode_fixup_or_oops+0x135/0x160
Nov 19 11:05:21 10.243.0.116 ** 3065 printk messages dropped **
Nov 19 11:05:21 10.243.0.116 [  130.812269]  ? load_module.cold.80+0x38d/0x38d
Nov 19 11:05:21 10.243.0.116 ** 3000 printk messages dropped **
Nov 19 11:05:21 10.243.0.116 [  130.833668]  ? irqentry_nmi_enter+0x83/0x90
Nov 19 11:05:21 10.243.0.116 ** 2935 printk messages dropped **
Nov 19 11:05:21 10.243.0.116 [  130.854453] R13: 0000000000000000 R14: 0000000000000000 R15: ffff88811d198000
Nov 19 11:05:21 10.243.0.116 ** 3002 printk messages dropped **
Nov 19 11:05:21 10.243.0.116 [  130.875624]  ? kallsyms_lookup_buildid+0x9/0x170
Nov 19 11:05:21 10.243.0.116 ** 2998 printk messages dropped **
Nov 19 11:05:21 10.243.0.116 [  130.896878] Hardware name: Supermicro Super Server/X10SDV-6C+-TLN4F, BIOS 2.1 11/22/2019
Nov 19 11:05:22 10.243.0.116 ** 2998 printk messages dropped **
Nov 19 11:05:22 10.243.0.116 [  130.918256]  exc_page_fault+0xcf/0xe0
Nov 19 11:05:22 10.243.0.116 ** 2976 printk messages dropped **
Nov 19 11:05:22 10.243.0.116 [  130.939339]  page_fault_oops+0x124/0x3c0
Nov 19 11:05:22 10.243.0.116 ** 2950 printk messages dropped **
Nov 19 11:05:22 10.243.0.116 [  130.960262] R13: fffffe000016c66a R14: ffffffff890e5bcc R15: fffffe000016c0c0
Nov 19 11:05:22 10.243.0.116 ** 2924 printk messages dropped **
Nov 19 11:05:22 10.243.0.116 [  130.981114]  ? kallsyms_lookup_buildid+0x9/0x170
Nov 19 11:05:22 10.243.0.116 ** 2978 printk messages dropped **
Nov 19 11:05:22 10.243.0.116 [  131.002327]  bad_area_nosemaphore+0x11/0x20
Nov 19 11:05:22 10.243.0.116 ** 2971 printk messages dropped **
Nov 19 11:05:22 10.243.0.116 [  131.023363] R13: 0000000000000000 R14: 0000000000000000 R15: ffff88811d198000
Nov 19 11:05:22 10.243.0.116 ** 2988 printk messages dropped **
Nov 19 11:05:22 10.243.0.116 [  131.044532] RBP: fffffe000016c008 R08: fffffe000016c080 R09: fffffe000016c1b0
Nov 19 11:05:22 10.243.0.116 ** 2961 printk messages dropped **
Nov 19 11:05:22 10.243.0.116 [  131.065595]  __die+0x25/0x30
Nov 19 11:05:22 10.243.0.116 ** 3005 printk messages dropped **
Nov 19 11:05:22 10.243.0.116 [  131.086742]  ? search_exception_tables+0x58/0x60
Nov 19 11:05:22 10.243.0.116 ** 2839 printk messages dropped **
Nov 19 11:05:22 10.243.0.116 [  131.106993] RAX: 0000000000000000 RBX: fffffffffffffff8 RCX: ffffffff890e5bcc
Nov 19 11:05:22 10.243.0.116 ** 2857 printk messages dropped **
Nov 19 11:05:22 10.243.0.116 [  131.127331]  asm_exc_page_fault+0x27/0x30
Nov 19 11:05:22 10.243.0.116 ** 2872 printk messages dropped **
Nov 19 11:05:22 10.243.0.116 [  131.147640]  ? load_module.cold.80+0x38d/0x38d
Nov 19 11:05:22 10.243.0.116 ** 2883 printk messages dropped **
Nov 19 11:05:22 10.243.0.116 [  131.168123]  asm_exc_page_fault+0x27/0x30
Nov 19 11:05:22 10.243.0.116 ** 2894 printk messages dropped **
Nov 19 11:05:22 10.243.0.116 [  131.188715]  asm_exc_page_fault+0x27/0x30
Nov 19 11:05:22 10.243.0.116 ** 2828 printk messages dropped **
Nov 19 11:05:22 10.243.0.116 [  131.208854] RSP: 0018:fffffe000016ff10 EFLAGS: 00010046
Nov 19 11:05:22 10.243.0.116 ** 2845 printk messages dropped **
Nov 19 11:05:22 10.243.0.116 [  131.229071] R10: ffff8883de4a7b1b R11: ffffed107bc94f63 R12: 1fffffc00002df13
Nov 19 11:05:22 10.243.0.116 ** 2903 printk messages dropped **
Nov 19 11:05:22 10.243.0.116 [  131.249689]  ? lock_release+0xd5/0x420
Nov 19 11:05:22 10.243.0.116 ** 2887 printk messages dropped **
Nov 19 11:05:22 10.243.0.116 [  131.270295]  ? print_modules+0x88/0x121
Nov 19 11:05:22 10.243.0.116 ** 2813 printk messages dropped **
Nov 19 11:05:22 10.243.0.116 [  131.290296]  ? fixup_exception+0x32/0x420
Nov 19 11:05:22 10.243.0.116 ** 2769 printk messages dropped **
Nov 19 11:05:22 10.243.0.116 [  131.309918] RBP: fffffe000016e0b0 R08: ffffed107bc94f64 R09: ffffed107bc94f64
Nov 19 11:05:22 10.243.0.116 ** 2846 printk messages dropped **
Nov 19 11:05:22 10.243.0.116 [  131.330111]  ? print_modules+0x88/0x121
Nov 19 11:05:22 10.243.0.116 ** 2844 printk messages dropped **
Nov 19 11:05:22 10.243.0.116 [  131.350364]  ? print_modules+0x88/0x121
Nov 19 11:05:22 10.243.0.116 ** 2755 printk messages dropped **
Nov 19 11:05:22 10.243.0.116 [  131.369941]  ? cmp_ex_search+0x1e/0x40
Nov 19 11:05:22 10.243.0.116 ** 2779 printk messages dropped **
Nov 19 11:05:22 10.243.0.116 [  131.389791]  ? print_modules+0x88/0x121
Nov 19 11:05:22 10.243.0.116 ** 2688 printk messages dropped **
Nov 19 11:05:22 10.243.0.116 [  131.408966] R10: ffff8883de4a7b1b R11: ffffed107bc94f63 R12: 1fffffc00002df13
Nov 19 11:05:22 10.243.0.116 ** 2686 printk messages dropped **
Nov 19 11:05:22 10.243.0.116 [  131.428092]  ? cmp_ex_sort+0x50/0x50
Nov 19 11:05:22 10.243.0.116 ** 2735 printk messages dropped **
Nov 19 11:05:22 10.243.0.116 [  131.447595]  ? lock_release+0xd5/0x420
Nov 19 11:05:22 10.243.0.116 ** 2706 printk messages dropped **
Nov 19 11:05:22 10.243.0.116 [  131.466846]  asm_exc_double_fault+0x1f/0x30
Nov 19 11:05:22 10.243.0.116 ** 2568 printk messages dropped **
Nov 19 11:05:22 10.243.0.116 [  131.485191]  ? kallsyms_lookup_buildid+0x9/0x170
Nov 19 11:05:22 10.243.0.116 ** 2688 printk messages dropped **
Nov 19 11:05:22 10.243.0.116 [  131.504276]  ? fixup_exception+0x32/0x420
Nov 19 11:05:22 10.243.0.116 ** 2586 printk messages dropped **
Nov 19 11:05:22 10.243.0.116 [  131.522558]  ? print_modules+0x88/0x121
Nov 19 11:05:22 10.243.0.116 ** 2531 printk messages dropped **
Nov 19 11:05:22 10.243.0.116 [  131.540600]  exc_double_fault+0x9f/0x110
Nov 19 11:05:22 10.243.0.116 ** 2521 printk messages dropped **
Nov 19 11:05:22 10.243.0.116 [  131.558375]  ? print_modules+0x88/0x121
Nov 19 11:05:22 10.243.0.116 ** 2523 printk messages dropped **
Nov 19 11:05:22 10.243.0.116 [  131.576310]  ? search_bpf_extables+0x136/0x190
Nov 19 11:05:22 10.243.0.116 ** 2455 printk messages dropped **
Nov 19 11:05:22 10.243.0.116 [  131.593703]  ? fixup_exception+0x32/0x420
Nov 19 11:05:22 10.243.0.116 ** 2440 printk messages dropped **
Nov 19 11:05:22 10.243.0.116 [  131.611165] R13: ffff888118d24780 R14: 0000000102568002 R15: 0000000000000000
Nov 19 11:05:22 10.243.0.116 ** 2410 printk messages dropped **
Nov 19 11:05:22 10.243.0.116 [  131.628319] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Nov 19 11:05:22 10.243.0.116 ** 2542 printk messages dropped **
Nov 19 11:05:22 10.243.0.116 [  131.646334]  ? print_modules+0x88/0x121
Nov 19 11:05:22 10.243.0.116 ** 2573 printk messages dropped **
Nov 19 11:05:22 10.243.0.116 [  131.664722] R13: 0000000000000000 R14: 0000000000000000 R15: ffff88811d198000
Nov 19 11:05:22 10.243.0.116 ** 2602 printk messages dropped **
Nov 19 11:05:22 10.243.0.116 [  131.683266] RSP: 0018:fffffe000016e858 EFLAGS: 00010046
Nov 19 11:05:22 10.243.0.116 ** 2574 printk messages dropped **
Nov 19 11:05:22 10.243.0.116 [  131.701504] RSP: 0018:fffffe000016e858 EFLAGS: 00010046
Nov 19 11:05:22 10.243.0.116 ** 2571 printk messages dropped **
Nov 19 11:05:22 10.243.0.116 [  131.719811] RDX: 0000000000000003 RSI: dffffc0000000000 RDI: fffffffffffffff8
Nov 19 11:05:22 10.243.0.116 ** 2672 printk messages dropped **
Nov 19 11:05:22 10.243.0.116 [  131.738810] RAX: 0000000000000000 RBX: fffffffffffffff8 RCX: ffffffff890e5bcc
Nov 19 11:05:22 10.243.0.116 ** 2707 printk messages dropped **
Nov 19 11:05:22 10.243.0.116 [  131.758046] RDX: 0000000000000003 RSI: dffffc0000000000 RDI: fffffffffffffff8
Nov 19 11:05:22 10.243.0.116 ** 2737 printk messages dropped **
Nov 19 11:05:22 10.243.0.116 [  131.777405] RSP: 0018:fffffe000016f078 EFLAGS: 00010046
Nov 19 11:05:22 10.243.0.116 ** 2764 printk messages dropped **
Nov 19 11:05:22 10.243.0.116 [  131.797192] PGD 30ae1a067 P4D 30ae1a067 PUD 30ae1c067 PMD 0
Nov 19 11:05:22 10.243.0.116 ** 2813 printk messages dropped **
Nov 19 11:05:22 10.243.0.116 [  131.817224] RSP: 0018:fffffe000016f898 EFLAGS: 00010046
Nov 19 11:05:22 10.243.0.116 ** 2841 printk messages dropped **
Nov 19 11:05:22 10.243.0.116 [  131.837396] R10: ffff8883de4a7b1b R11: ffffed107bc94f63 R12: 1fffffc00002df94

[-- Attachment #2: smime.p7s --]
[-- Type: application/pkcs7-signature, Size: 6759 bytes --]

Re: [External] - Re: EVL Memory

[Philippe Gerum]

Russell Johnson <russell.johnson@kratosdefense.com> writes:

> [[S/MIME Signed Part:Undecided]]
> I just tried to run the app with the latest "next" branch, and it didn't
> make it long at all before I got a ton of dump in netconsole. But the
> messages look very strange to me. A lot of "printk messages dropped"...
> Thoughts?
>

Confirmed, this is the bug I'm chasing ATM, there is an assertion which
triggers in the core, so frequently that the system enters a tight loop
only issuing and (re-)producing it again and again. I discovered it by
artificially slowing down the kernel side, otherwise my board would
simply lockup with no message.

-- 
Philippe.

Re: [External] - Re: EVL Memory

[Philippe Gerum]

Russell Johnson <russell.johnson@kratosdefense.com> writes:

> [[S/MIME Signed Part:Undecided]]
> After fixing that line of code that was causing errors with building the EVL
> kernel, I was able to successfully build. I am now running our scenario that
> was causing issues previously. I will let you know what issues (if any) I
> run into. 
>
> To your other point, I can confirm that the priorities that are being
> boosted are never being cleared back to the original base priorities.. What
> do you recommend looking at first in order to figure out why this is
> happening?
>

I could not reproduce this PI deboosting issue yet, however I eventually
figured out the reason for the lockup - which may translate to RCU
warnings on your system with many CPUs: this is a Dovetail issue wrt
lockdep support. IOW, the bug is in the lock debug infrastructure,
caused by adding interrupt pipelining to the picture.

You can work around this issue until fixed by disabling PROVE_LOCKING
and LOCKDEP entirely (which is also enabled by DEBUG_LOCK_ALLOC and
DEBUG_WW_MUTEX_SLOWPATH indirectly).

-- 
Philippe.

RE: [External] - Re: EVL Memory

[Bryan Butler]

Hi Philippe, just to let you know this is a holiday week here in the US, and Russell is out until next Monday. Those of us left will try to update our build, but we may not be able to test it thoroughly. Just wanted you to know since you may not hear much from us for the next week.

-----Original Message-----
From: Philippe Gerum <rpm@xenomai.org> 
Sent: Monday, November 21, 2022 8:56 AM
To: Russell Johnson <russell.johnson@kratosdefense.com>
Cc: xenomai@lists.linux.dev; Bryan Butler <Bryan.Butler@kratosdefense.com>; Shawn McManus <shawn.mcmanus@kratosdefense.com>
Subject: Re: [External] - Re: EVL Memory

CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you recognize the sender and know the content is safe.


Russell Johnson <russell.johnson@kratosdefense.com> writes:

> [[S/MIME Signed Part:Undecided]]
> After fixing that line of code that was causing errors with building 
> the EVL kernel, I was able to successfully build. I am now running our 
> scenario that was causing issues previously. I will let you know what 
> issues (if any) I run into.
>
> To your other point, I can confirm that the priorities that are being 
> boosted are never being cleared back to the original base priorities.. 
> What do you recommend looking at first in order to figure out why this 
> is happening?
>

I could not reproduce this PI deboosting issue yet, however I eventually figured out the reason for the lockup - which may translate to RCU warnings on your system with many CPUs: this is a Dovetail issue wrt lockdep support. IOW, the bug is in the lock debug infrastructure, caused by adding interrupt pipelining to the picture.

You can work around this issue until fixed by disabling PROVE_LOCKING and LOCKDEP entirely (which is also enabled by DEBUG_LOCK_ALLOC and DEBUG_WW_MUTEX_SLOWPATH indirectly).

--
Philippe.

RE: [External] - Re: EVL Memory

[Russell Johnson]

[-- Attachment #1: Type: text/plain, Size: 231 bytes --]

Hi Philippe,

I am back in the office now. Should I go ahead and build the EVL kernel with
the parameters you mentioned disabled, or has this issue been resolved? I
can pull the latest next branch commit as well.

Thanks,

Russell

[-- Attachment #2: smime.p7s --]
[-- Type: application/pkcs7-signature, Size: 6759 bytes --]

Re: [External] - Re: EVL Memory

[Philippe Gerum]

Russell Johnson <russell.johnson@kratosdefense.com> writes:

> [[S/MIME Signed Part:Undecided]]
> Hi Philippe,
>
> I am back in the office now. Should I go ahead and build the EVL kernel with
> the parameters you mentioned disabled, or has this issue been resolved? I
> can pull the latest next branch commit as well.
>

More improvements have landed, disabling PROVE_LOCKING is no longer
necessary. Please pull v5.15.77-evl2-rebase (rebase/v5.15.y-evl) for
testing.

Thanks,

-- 
Philippe.

RE: [External] - Re: EVL Memory

[Russell Johnson]

[-- Attachment #1.1: Type: text/plain, Size: 427 bytes --]

Sounds good. I built the latest kernel (with lock debugging re-enabled). I
am running our test case as we speak. I have not encountered a crash yet,
but there was a little bit of debug output from the kernel that I have
attached in this email. Also, I am definitely still seeing a lot of the
threads be priority boosted forever. Let me know if you can think of other
ways to help debug that priority issue.


Thanks,

Russell 

[-- Attachment #1.2: stack_trace.txt --]
[-- Type: text/plain, Size: 21012 bytes --]

Nov 28 12:46:20 10.243.0.116 [  605.511149] ------------[ cut here ]------------
Nov 28 12:46:20 10.243.0.116 [  605.511154] DEBUG_LOCKS_WARN_ON(!lockdep_hardirqs_enabled())
Nov 28 12:46:20 10.243.0.116 [  605.511168] WARNING: CPU: 5 PID: 1858 at kernel/locking/lockdep.c:5550 check_flags+0x1a4/0x1d0
Nov 28 12:46:20 10.243.0.116 [  605.511183] Modules linked in: xt_CHECKSUM xt_MASQUERADE tun netconsole ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 xt_conntrack ebtable_nat ebtable_broute ip6table_nat ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_nat iptable_mangle iptable_security iptable_raw nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 libcrc32c ip_set nfnetlink ebtable_filter ebtables rfkill ip6table_filter ip6_tables iptable_filter sunrpc dm_mirror dm_region_hash dm_log dm_mod x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel iTCO_wdt gpio_ich iTCO_vendor_support mxm_wmi kvm irqbypass crct10dif_pclmul crc32_pclmul ghash_clmulni_intel aesni_intel crypto_simd cryptd rapl intel_cstate pcspkr joydev input_leds intel_pch_thermal i2c_i801 i2c_smbus sg ftdi_sio mei_me lpc_ich mfd_core mei ioatdma wmi ip_tables ext4 mbcache jbd2 ast drm_vram_helper drm_kms_helper sd_mod sr_mod syscopyarea sysfillrect sysimgblt cdrom fb_sys_fops t10_pi drm_ttm_helper ttm drm ixgbe ahci libahc
Nov 28 12:46:20 10.243.0.116 [  605.511353]  igb crc32c_intel libata mdio ptp i2c_algo_bit pps_core dca fuse
Nov 28 12:46:20 10.243.0.116 [  605.511372] CPU: 5 PID: 1858 Comm: TxStartup Not tainted 5.15.77evl-gaf7e24df1f51 #3
Nov 28 12:46:20 10.243.0.116 [  605.511380] Hardware name: Supermicro Super Server/X10SDV-6C+-TLN4F, BIOS 2.1 11/22/2019
Nov 28 12:46:20 10.243.0.116 [  605.511383] IRQ stage: Linux
Nov 28 12:46:20 10.243.0.116 [  605.511386] RIP: 0010:check_flags+0x1a4/0x1d0
Nov 28 12:46:20 10.243.0.116 [  605.511393] Code: ff ff e8 9f b4 6a ff 85 c0 74 21 44 8b 05 0c 45 60 01 45 85 c0 75 15 48 c7 c6 40 05 69 ab 48 c7 c7 00 d4 68 ab e8 f0 e2 f9 ff <0f> 0b 48 c7 c7 80 05 69 ab e8 69 7a fa ff e9 5c ff ff ff 65 48 8b
Nov 28 12:46:20 10.243.0.116 [  605.511398] RSP: 0018:ffff888120cb7d90 EFLAGS: 00010082
Nov 28 12:46:20 10.243.0.116 [  605.511404] RAX: 0000000000000000 RBX: 1ffff11024196fb8 RCX: 0000000000000027
Nov 28 12:46:20 10.243.0.116 [  605.511408] RDX: 0000000000000027 RSI: dffffc0000000000 RDI: ffff8883de137b78
Nov 28 12:46:20 10.243.0.116 [  605.511412] RBP: ffff888120cb7d90 R08: ffffed107bc26f70 R09: ffffed107bc26f70
Nov 28 12:46:20 10.243.0.116 [  605.511416] R10: ffff8883de137b7b R11: ffffed107bc26f6f R12: ffff888120cb7e80
Nov 28 12:46:20 10.243.0.116 [  605.511420] R13: 0000000000000005 R14: 0000000000000000 R15: 0000000000000000
Nov 28 12:46:20 10.243.0.116 [  605.511423] FS:  00007f1ddeffe700(0000) GS:ffff8883de100000(0000) knlGS:0000000000000000
Nov 28 12:46:20 10.243.0.116 [  605.511429] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Nov 28 12:46:20 10.243.0.116 [  605.511433] CR2: 0000000000000000 CR3: 000000010af38002 CR4: 00000000003706e0
Nov 28 12:46:20 10.243.0.116 [  605.511436] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Nov 28 12:46:20 10.243.0.116 [  605.511439] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Nov 28 12:46:21 10.243.0.116 [  605.511443] Call Trace:
Nov 28 12:46:21 10.243.0.116 [  605.511445]  <TASK>
Nov 28 12:46:21 10.243.0.116 [  605.511449]  lock_acquire+0x150/0x3f0
Nov 28 12:46:21 10.243.0.116 [  605.511462]  ? lockdep_hardirqs_on_prepare+0x250/0x250
Nov 28 12:46:21 10.243.0.116 [  605.511471]  ? lock_release+0x2ac/0x440
Nov 28 12:46:21 10.243.0.116 [  605.511479]  ? __context_tracking_enter+0x6f/0x80
Nov 28 12:46:21 10.243.0.116 [  605.511487]  ? lock_downgrade+0x3e0/0x3e0
Nov 28 12:46:21 10.243.0.116 [  605.511496]  ? rcu_read_lock_sched_held+0x5b/0xd0
Nov 28 12:46:21 10.243.0.116 [  605.511507]  ? sched_clock+0x9/0x10
Nov 28 12:46:21 10.243.0.116 [  605.511516]  vtime_user_exit+0x73/0x130
Nov 28 12:46:21 10.243.0.116 [  605.511523]  ? __context_tracking_exit+0x60/0x70
Nov 28 12:46:21 10.243.0.116 [  605.511532]  __context_tracking_exit+0x60/0x70
Nov 28 12:46:21 10.243.0.116 [  605.511539]  syscall_enter_from_user_mode+0xd8/0x130
Nov 28 12:46:21 10.243.0.116 [  605.511547]  do_syscall_64+0x1d/0xa0
Nov 28 12:46:21 10.243.0.116 [  605.511557]  entry_SYSCALL_64_after_hwframe+0x61/0xcb
Nov 28 12:46:21 10.243.0.116 [  605.511565] RIP: 0033:0x7f1df05bce44
Nov 28 12:46:21 10.243.0.116 [  605.511571] Code: ff 31 c0 87 83 1c 06 00 00 83 f8 fe 0f 84 42 02 00 00 48 8b 54 24 08 be 18 00 00 00 b8 11 01 00 00 48 8d ba e0 02 00 00 0f 05 <f6> 82 14 06 00 00 04 0f 85 e9 01 00 00 48 8d 7c 24 10 48 c7 44 24
Nov 28 12:46:21 10.243.0.116 [  605.511576] RSP: 002b:00007f1ddefee960 EFLAGS: 00000202 ORIG_RAX: 0000000000000111
Nov 28 12:46:21 10.243.0.116 [  605.511581] RAX: ffffffffffffffda RBX: 00007f1ddeffe700 RCX: 00007f1df05bce44
Nov 28 12:46:21 10.243.0.116 [  605.511585] RDX: 00007f1ddeffe700 RSI: 0000000000000018 RDI: 00007f1ddeffe9e0
Nov 28 12:46:21 10.243.0.116 [  605.511589] RBP: 0000000000000000 R08: 00007f1ddeffe700 R09: 00007f1ddeffe700
Nov 28 12:46:21 10.243.0.116 [  605.511592] R10: 00007f1ddeffe9d0 R11: 0000000000000202 R12: 0000000000000001
Nov 28 12:46:21 10.243.0.116 [  605.511596] R13: 0000000001001000 R14: 0000000000000000 R15: 00007f1ddeffe700
Nov 28 12:46:21 10.243.0.116 [  605.511607]  </TASK>
Nov 28 12:46:21 10.243.0.116 [  605.511610] irq event stamp: 6
Nov 28 12:46:21 10.243.0.116 [  605.511612] hardirqs last  enabled at (5): [<ffffffffaa002725>] ret_from_fork+0x15/0x30
Nov 28 12:46:21 10.243.0.116 [  605.511621] hardirqs last disabled at (6): [<ffffffffab13d46d>] do_syscall_64+0x1d/0xa0
Nov 28 12:46:21 10.243.0.116 [  605.511630] softirqs last  enabled at (0): [<ffffffffaa0c4f7a>] copy_process+0x12fa/0x3750
Nov 28 12:46:21 10.243.0.116 [  605.511638] softirqs last disabled at (0): [<0000000000000000>] 0x0
Nov 28 12:46:21 10.243.0.116 [  605.511645] ---[ end trace 19749a8d325a12e0 ]---
Nov 28 12:46:21 10.243.0.116 [  605.511650] possible reason: unannotated irqs-on.
Nov 28 12:46:21 10.243.0.116 [  605.511653] irq event stamp: 6
Nov 28 12:46:21 10.243.0.116 [  605.511656] hardirqs last  enabled at (5): [<ffffffffaa002725>] ret_from_fork+0x15/0x30
Nov 28 12:46:21 10.243.0.116 [  605.511669] hardirqs last disabled at (6): [<ffffffffab13d46d>] do_syscall_64+0x1d/0xa0
Nov 28 12:46:21 10.243.0.116 [  605.511678] softirqs last  enabled at (0): [<ffffffffaa0c4f7a>] copy_process+0x12fa/0x3750
Nov 28 12:46:21 10.243.0.116 [  605.511684] softirqs last disabled at (0): [<0000000000000000>] 0x0
Nov 28 12:46:21 10.243.0.116 [  606.064136] ------------[ cut here ]------------
Nov 28 12:46:21 10.243.0.116 [  606.064142] WARNING: CPU: 5 PID: 1858 at kernel/evl/mutex.c:846 __evl_unlock_mutex+0x2b4/0x430
Nov 28 12:46:21 10.243.0.116 [  606.064163] Modules linked in: xt_CHECKSUM xt_MASQUERADE tun netconsole ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 xt_conntrack ebtable_nat ebtable_broute ip6table_nat ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_nat iptable_mangle iptable_security iptable_raw nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 libcrc32c ip_set nfnetlink ebtable_filter ebtables rfkill ip6table_filter ip6_tables iptable_filter sunrpc dm_mirror dm_region_hash dm_log dm_mod x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel iTCO_wdt gpio_ich iTCO_vendor_support mxm_wmi kvm irqbypass crct10dif_pclmul crc32_pclmul ghash_clmulni_intel aesni_intel crypto_simd cryptd rapl intel_cstate pcspkr joydev input_leds intel_pch_thermal i2c_i801 i2c_smbus sg ftdi_sio mei_me lpc_ich mfd_core mei ioatdma wmi ip_tables ext4 mbcache jbd2 ast drm_vram_helper drm_kms_helper sd_mod sr_mod syscopyarea sysfillrect sysimgblt cdrom fb_sys_fops t10_pi drm_ttm_helper ttm drm ixgbe ahci libahc
Nov 28 12:46:21 10.243.0.116 [  606.064495]  igb crc32c_intel libata mdio ptp i2c_algo_bit pps_core dca fuse
Nov 28 12:46:21 10.243.0.116 [  606.064532] CPU: 5 PID: 1858 Comm: DummyTxSample Tainted: G        W         5.15.77evl-gaf7e24df1f51 #3
Nov 28 12:46:21 10.243.0.116 [  606.064545] Hardware name: Supermicro Super Server/X10SDV-6C+-TLN4F, BIOS 2.1 11/22/2019
Nov 28 12:46:21 10.243.0.116 [  606.064550] IRQ stage: EVL
Nov 28 12:46:21 10.243.0.116 [  606.064555] RIP: 0010:__evl_unlock_mutex+0x2b4/0x430
Nov 28 12:46:21 10.243.0.116 [  606.064568] Code: f7 e8 10 fb 1d 00 48 c7 c6 13 11 30 aa 48 89 df 4d 8b 7c 24 78 e8 fc f4 e9 ff 4c 89 e7 e8 b4 3f ea ff 41 55 9d 4d 39 fe 75 bc <0f> 0b eb b8 85 c0 75 3c 48 89 df e8 ec d7 ff ff e9 52 fe ff ff 65
Nov 28 12:46:21 10.243.0.116 [  606.064578] RSP: 0018:ffff888120cb7ac8 EFLAGS: 00010046
Nov 28 12:46:21 10.243.0.116 [  606.064588] RAX: 0000000000000000 RBX: ffff888101576018 RCX: ffffffffaa1a5187
Nov 28 12:46:21 10.243.0.116 [  606.064596] RDX: 0000000000000003 RSI: dffffc0000000000 RDI: ffff888101576000
Nov 28 12:46:21 10.243.0.116 [  606.064604] RBP: ffff888120cb7b10 R08: ffffed10202aec01 R09: ffffed10202aec01
Nov 28 12:46:21 10.243.0.116 [  606.064612] R10: ffff888101576003 R11: ffffed10202aec00 R12: ffff888101576000
Nov 28 12:46:21 10.243.0.116 [  606.064620] R13: 0000000000000082 R14: ffff888101576078 R15: ffff888101576078
Nov 28 12:46:21 10.243.0.116 [  606.064629] FS:  00007f1ddeffe700(0000) GS:ffff8883de100000(0000) knlGS:0000000000000000
Nov 28 12:46:21 10.243.0.116 [  606.064638] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Nov 28 12:46:21 10.243.0.116 [  606.064646] CR2: 0000000000000000 CR3: 000000010af38002 CR4: 00000000003706e0
Nov 28 12:46:21 10.243.0.116 [  606.064654] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Nov 28 12:46:21 10.243.0.116 [  606.064660] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Nov 28 12:46:21 10.243.0.116 [  606.064667] Call Trace:
Nov 28 12:46:21 10.243.0.116 [  606.064671]  <TASK>
Nov 28 12:46:21 10.243.0.116 [  606.064690]  monitor_oob_ioctl+0x1044/0x16d0
Nov 28 12:46:21 10.243.0.116 [  606.064701]  ? rcu_read_lock_held_common+0x25/0x70
Nov 28 12:46:21 10.243.0.116 [  606.064735]  ? enter_monitor+0x160/0x160
Nov 28 12:46:21 10.243.0.116 [  606.064756]  ? lock_release+0xd5/0x440
Nov 28 12:46:21 10.243.0.116 [  606.064771]  ? evl_get_file+0xa3/0x130
Nov 28 12:46:21 10.243.0.116 [  606.064788]  ? lock_downgrade+0x3e0/0x3e0
Nov 28 12:46:21 10.243.0.116 [  606.064805]  ? do_raw_spin_lock+0x119/0x1e0
Nov 28 12:46:21 10.243.0.116 [  606.064818]  ? spin_dump+0x90/0x90
Nov 28 12:46:21 10.243.0.116 [  606.064828]  ? asm_sysvec_apic_timer_interrupt+0x1b/0x20
Nov 28 12:46:21 10.243.0.116 [  606.064853]  ? __kasan_check_read+0x11/0x20
Nov 28 12:46:21 10.243.0.116 [  606.064885]  EVL_ioctl+0x81/0xf0
Nov 28 12:46:21 10.243.0.116 [  606.064906]  do_oob_syscall+0x5ac/0x6c0
Nov 28 12:46:21 10.243.0.116 [  606.064924]  ? prepare_for_signal+0x1b0/0x1b0
Nov 28 12:46:21 10.243.0.116 [  606.064956]  handle_oob_syscall+0x189/0x230
Nov 28 12:46:21 10.243.0.116 [  606.064974]  ? handle_pipelined_syscall+0x840/0x840
Nov 28 12:46:21 10.243.0.116 [  606.064996]  ? rcu_read_lock_sched_held+0x5b/0xd0
Nov 28 12:46:21 10.243.0.116 [  606.065013]  ? rcu_read_lock_any_held.part.15+0x20/0x20
Nov 28 12:46:21 10.243.0.116 [  606.065038]  pipeline_syscall+0xa7/0x1c0
Nov 28 12:46:21 10.243.0.116 [  606.065062]  syscall_enter_from_user_mode+0x5f/0x130
Nov 28 12:46:21 10.243.0.116 [  606.065080]  do_syscall_64+0x1d/0xa0
Nov 28 12:46:21 10.243.0.116 [  606.065097]  entry_SYSCALL_64_after_hwframe+0x61/0xcb
Nov 28 12:46:21 10.243.0.116 [  606.065110] RIP: 0033:0x7f1df8382b4b
Nov 28 12:46:21 10.243.0.116 [  606.065119] Code: Unable to access opcode bytes at RIP 0x7f1df8382b21.
Nov 28 12:46:21 10.243.0.116 [  606.065124] RSP: 002b:00007f1ddefee3f0 EFLAGS: 00000246 ORIG_RAX: 000000000000009d
Nov 28 12:46:21 10.243.0.116 [  606.065135] RAX: ffffffffffffffda RBX: 00007f1ddefee490 RCX: 00007f1df8382b4b
Nov 28 12:46:21 10.243.0.116 [  606.065143] RDX: 00000000c0186d03 RSI: 0000000000000049 RDI: 0000000010000002
Nov 28 12:46:21 10.243.0.116 [  606.065150] RBP: 00000000c0186d03 R08: 0000000000000000 R09: 00007fff8256b080
Nov 28 12:46:21 10.243.0.116 [  606.065158] R10: 00007f1ddefee490 R11: 0000000000000246 R12: 0000000000000049
Nov 28 12:46:21 10.243.0.116 [  606.065165] R13: 0000000000000001 R14: 0000000036a20b78 R15: 00007f1ddefee590
Nov 28 12:46:21 10.243.0.116 [  606.065200]  </TASK>
Nov 28 12:46:21 10.243.0.116 [  606.065204] irq event stamp: 6
Nov 28 12:46:21 10.243.0.116 [  606.065208] hardirqs last  enabled at (5): [<ffffffffaa002725>] ret_from_fork+0x15/0x30
Nov 28 12:46:21 10.243.0.116 [  606.065222] hardirqs last disabled at (6): [<ffffffffab13d46d>] do_syscall_64+0x1d/0xa0
Nov 28 12:46:21 10.243.0.116 [  606.065237] softirqs last  enabled at (0): [<ffffffffaa0c4f7a>] copy_process+0x12fa/0x3750
Nov 28 12:46:21 10.243.0.116 [  606.065250] softirqs last disabled at (0): [<0000000000000000>] 0x0
Nov 28 12:46:21 10.243.0.116 [  606.065258] ---[ end trace 19749a8d325a12e1 ]---
Nov 28 12:46:21 10.243.0.116 [  606.065296] ------------[ cut here ]------------
Nov 28 12:46:21 10.243.0.116 [  606.065300] WARNING: CPU: 5 PID: 1858 at kernel/evl/sched/core.c:1015 __evl_schedule+0x73a/0x1180
Nov 28 12:46:21 10.243.0.116 [  606.065314] Modules linked in: xt_CHECKSUM xt_MASQUERADE tun netconsole ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 xt_conntrack ebtable_nat ebtable_broute ip6table_nat ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_nat iptable_mangle iptable_security iptable_raw nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 libcrc32c ip_set nfnetlink ebtable_filter ebtables rfkill ip6table_filter ip6_tables iptable_filter sunrpc dm_mirror dm_region_hash dm_log dm_mod x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel iTCO_wdt gpio_ich iTCO_vendor_support mxm_wmi kvm irqbypass crct10dif_pclmul crc32_pclmul ghash_clmulni_intel aesni_intel crypto_simd cryptd rapl intel_cstate pcspkr joydev input_leds intel_pch_thermal i2c_i801 i2c_smbus sg ftdi_sio mei_me lpc_ich mfd_core mei ioatdma wmi ip_tables ext4 mbcache jbd2 ast drm_vram_helper drm_kms_helper sd_mod sr_mod syscopyarea sysfillrect sysimgblt cdrom fb_sys_fops t10_pi drm_ttm_helper ttm drm ixgbe ahci libahc
Nov 28 12:46:21 10.243.0.116 [  606.065619]  igb crc32c_intel libata mdio ptp i2c_algo_bit pps_core dca fuse
Nov 28 12:46:21 10.243.0.116 [  606.065651] CPU: 5 PID: 1858 Comm: DummyTxSample Tainted: G        W         5.15.77evl-gaf7e24df1f51 #3
Nov 28 12:46:21 10.243.0.116 [  606.065663] Hardware name: Supermicro Super Server/X10SDV-6C+-TLN4F, BIOS 2.1 11/22/2019
Nov 28 12:46:21 10.243.0.116 [  606.065668] IRQ stage: EVL
Nov 28 12:46:21 10.243.0.116 [  606.065672] RIP: 0010:__evl_schedule+0x73a/0x1180
Nov 28 12:46:21 10.243.0.116 [  606.065682] Code: 46 78 48 39 d8 0f 85 45 fa ff ff 4c 8b 75 b0 48 c7 c6 1b f7 31 aa 4c 89 f7 e8 02 0f e8 ff 48 8b 5d d0 48 89 df e8 b6 59 e8 ff <0f> 0b 68 1b f7 31 aa 31 f6 45 31 c9 41 b8 01 00 00 00 31 c9 31 d2
Nov 28 12:46:21 10.243.0.116 [  606.065691] RSP: 0018:ffff888120cb7960 EFLAGS: 00010046
Nov 28 12:46:21 10.243.0.116 [  606.065700] RAX: 0000000000000000 RBX: ffff888101576000 RCX: ffffffffaa1a5187
Nov 28 12:46:21 10.243.0.116 [  606.065707] RDX: 0000000000000003 RSI: dffffc0000000000 RDI: ffff888101576000
Nov 28 12:46:21 10.243.0.116 [  606.065713] RBP: ffff888120cb79f0 R08: ffffed10202aec01 R09: ffffed10202aec01
Nov 28 12:46:21 10.243.0.116 [  606.065721] R10: ffff888101576003 R11: ffffed10202aec00 R12: 0000000010008000
Nov 28 12:46:21 10.243.0.116 [  606.065728] R13: ffff888120cb7a78 R14: ffff888101576018 R15: ffff8883de174720
Nov 28 12:46:21 10.243.0.116 [  606.065736] FS:  00007f1ddeffe700(0000) GS:ffff8883de100000(0000) knlGS:0000000000000000
Nov 28 12:46:21 10.243.0.116 [  606.065745] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Nov 28 12:46:21 10.243.0.116 [  606.065752] CR2: 0000000000000000 CR3: 000000010af38002 CR4: 00000000003706e0
Nov 28 12:46:21 10.243.0.116 [  606.065759] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Nov 28 12:46:21 10.243.0.116 [  606.065765] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Nov 28 12:46:21 10.243.0.116 [  606.065771] Call Trace:
Nov 28 12:46:21 10.243.0.116 [  606.065775]  <TASK>
Nov 28 12:46:21 10.243.0.116 [  606.065802]  handle_irq_pipelined_finish+0x141/0x240
Nov 28 12:46:21 10.243.0.116 [  606.065822]  arch_pipeline_entry+0xce/0x110
Nov 28 12:46:21 10.243.0.116 [  606.065839]  sysvec_apic_timer_interrupt+0xe/0x20
Nov 28 12:46:21 10.243.0.116 [  606.065851]  asm_sysvec_apic_timer_interrupt+0x1b/0x20
Nov 28 12:46:21 10.243.0.116 [  606.065864] RIP: 0010:monitor_oob_ioctl+0x1066/0x16d0
Nov 28 12:46:22 10.243.0.116 [  606.065876] Code: fe ff ff e8 bc 38 00 00 48 8b bd 48 fe ff ff 48 c7 c6 be d5 2f aa e8 49 30 ea ff 48 8b bd 58 fe ff ff e8 fd 7a ea ff 41 55 9d <48> 8b bd 38 fe ff ff e8 4e d2 01 00 41 89 c5 85 c0 0f 84 c9 00 00
Nov 28 12:46:22 10.243.0.116 [  606.065885] RSP: 0018:ffff888120cb7b20 EFLAGS: 00000246
Nov 28 12:46:22 10.243.0.116 [  606.065893] RAX: 0000000000000000 RBX: ffff8881078c2a50 RCX: ffffffffaa1a5187
Nov 28 12:46:22 10.243.0.116 [  606.065900] RDX: 0000000000000003 RSI: dffffc0000000000 RDI: ffff88811a51e220
Nov 28 12:46:22 10.243.0.116 [  606.065904] RBP: ffff888120cb7d08 R08: ffffed10234a3c45 R09: ffffed10234a3c45
Nov 28 12:46:22 10.243.0.116 [  606.065907] R10: ffff88811a51e223 R11: ffffed10234a3c44 R12: 00007f1ddefee490
Nov 28 12:46:22 10.243.0.116 [  606.065911] R13: 0000000000000246 R14: ffff888101576000 R15: ffff88811a51e000
Nov 28 12:46:22 10.243.0.116 [  606.065921]  ? do_raw_spin_unlock+0x97/0x140
Nov 28 12:46:22 10.243.0.116 [  606.065932]  ? rcu_read_lock_held_common+0x25/0x70
Nov 28 12:46:22 10.243.0.116 [  606.065947]  ? enter_monitor+0x160/0x160
Nov 28 12:46:22 10.243.0.116 [  606.065957]  ? lock_release+0xd5/0x440
Nov 28 12:46:22 10.243.0.116 [  606.065964]  ? evl_get_file+0xa3/0x130
Nov 28 12:46:22 10.243.0.116 [  606.065972]  ? lock_downgrade+0x3e0/0x3e0
Nov 28 12:46:22 10.243.0.116 [  606.065981]  ? do_raw_spin_lock+0x119/0x1e0
Nov 28 12:46:22 10.243.0.116 [  606.065987]  ? spin_dump+0x90/0x90
Nov 28 12:46:22 10.243.0.116 [  606.065992]  ? asm_sysvec_apic_timer_interrupt+0x1b/0x20
Nov 28 12:46:22 10.243.0.116 [  606.066004]  ? __kasan_check_read+0x11/0x20
Nov 28 12:46:22 10.243.0.116 [  606.066018]  EVL_ioctl+0x81/0xf0
Nov 28 12:46:22 10.243.0.116 [  606.066028]  do_oob_syscall+0x5ac/0x6c0
Nov 28 12:46:22 10.243.0.116 [  606.066037]  ? prepare_for_signal+0x1b0/0x1b0
Nov 28 12:46:22 10.243.0.116 [  606.066051]  handle_oob_syscall+0x189/0x230
Nov 28 12:46:22 10.243.0.116 [  606.066060]  ? handle_pipelined_syscall+0x840/0x840
Nov 28 12:46:22 10.243.0.116 [  606.066070]  ? rcu_read_lock_sched_held+0x5b/0xd0
Nov 28 12:46:22 10.243.0.116 [  606.066079]  ? rcu_read_lock_any_held.part.15+0x20/0x20
Nov 28 12:46:22 10.243.0.116 [  606.066091]  pipeline_syscall+0xa7/0x1c0
Nov 28 12:46:22 10.243.0.116 [  606.066103]  syscall_enter_from_user_mode+0x5f/0x130
Nov 28 12:46:22 10.243.0.116 [  606.066112]  do_syscall_64+0x1d/0xa0
Nov 28 12:46:22 10.243.0.116 [  606.066120]  entry_SYSCALL_64_after_hwframe+0x61/0xcb
Nov 28 12:46:22 10.243.0.116 [  606.066127] RIP: 0033:0x7f1df8382b4b
Nov 28 12:46:22 10.243.0.116 [  606.066131] Code: Unable to access opcode bytes at RIP 0x7f1df8382b21.
Nov 28 12:46:22 10.243.0.116 [  606.066134] RSP: 002b:00007f1ddefee3f0 EFLAGS: 00000246 ORIG_RAX: 000000000000009d
Nov 28 12:46:22 10.243.0.116 [  606.066139] RAX: ffffffffffffffda RBX: 00007f1ddefee490 RCX: 00007f1df8382b4b
Nov 28 12:46:22 10.243.0.116 [  606.066143] RDX: 00000000c0186d03 RSI: 0000000000000049 RDI: 0000000010000002
Nov 28 12:46:22 10.243.0.116 [  606.066147] RBP: 00000000c0186d03 R08: 0000000000000000 R09: 00007fff8256b080
Nov 28 12:46:22 10.243.0.116 [  606.066150] R10: 00007f1ddefee490 R11: 0000000000000246 R12: 0000000000000049
Nov 28 12:46:22 10.243.0.116 [  606.066153] R13: 0000000000000001 R14: 0000000036a20b78 R15: 00007f1ddefee590
Nov 28 12:46:22 10.243.0.116 [  606.066169]  </TASK>
Nov 28 12:46:22 10.243.0.116 [  606.066171] irq event stamp: 6
Nov 28 12:46:22 10.243.0.116 [  606.066173] hardirqs last  enabled at (5): [<ffffffffaa002725>] ret_from_fork+0x15/0x30
Nov 28 12:46:22 10.243.0.116 [  606.066180] hardirqs last disabled at (6): [<ffffffffab13d46d>] do_syscall_64+0x1d/0xa0
Nov 28 12:46:22 10.243.0.116 [  606.066188] softirqs last  enabled at (0): [<ffffffffaa0c4f7a>] copy_process+0x12fa/0x3750
Nov 28 12:46:22 10.243.0.116 [  606.066194] softirqs last disabled at (0): [<0000000000000000>] 0x0
Nov 28 12:46:22 10.243.0.116 [  606.066198] ---[ end trace 19749a8d325a12e2 ]---

Nov 28 12:47:46 10.243.0.116 [  691.063089] EVL: DummyRxService switching in-band [pid=1994, excpt=14, user_pc=0x7f1df837d580]


[-- Attachment #2: smime.p7s --]
[-- Type: application/pkcs7-signature, Size: 6759 bytes --]

Re: [External] - Re: EVL Memory

[Philippe Gerum]

Philippe Gerum <rpm@xenomai.org> writes:

> Russell Johnson <russell.johnson@kratosdefense.com> writes:
>
>> [[S/MIME Signed Part:Undecided]]
>> Sounds good. I built the latest kernel (with lock debugging re-enabled). I
>> am running our test case as we speak. I have not encountered a crash yet,
>> but there was a little bit of debug output from the kernel that I have
>> attached in this email. Also, I am definitely still seeing a lot of the
>> threads be priority boosted forever. Let me know if you can think of other
>> ways to help debug that priority issue.
>>
>>
>

Working on it.

-- 
Philippe.

Re: [External] - Re: EVL Memory

[Philippe Gerum]

Philippe Gerum <rpm@xenomai.org> writes:

> Philippe Gerum <rpm@xenomai.org> writes:
>
>> Russell Johnson <russell.johnson@kratosdefense.com> writes:
>>
>>> [[S/MIME Signed Part:Undecided]]
>>> Sounds good. I built the latest kernel (with lock debugging re-enabled). I
>>> am running our test case as we speak. I have not encountered a crash yet,
>>> but there was a little bit of debug output from the kernel that I have
>>> attached in this email. Also, I am definitely still seeing a lot of the
>>> threads be priority boosted forever. Let me know if you can think of other
>>> ways to help debug that priority issue.
>>>
>>>
>>
>
> Working on it.

Please pull the latest code from the next-evl-rebase branch. An
overnight stress test has been running here on several machines for a
few hours so far, you may want to check with your app as well.

-- 
Philippe.

RE: [External] - Re: EVL Memory

[Russell Johnson]

[-- Attachment #1.1: Type: text/plain, Size: 505 bytes --]

Hi Philippe,

I pulled your latest commit on the "next" branch (5e681f841fc3), I built it
with the normal debug configuration parameters enabled, and ran my test case
against it. I appear to be seeing the same behavior as before. No crashing,
but many threads that are boosted infinitely. I did get a kernel splat
again; it looks oddly similar to the one I sent previously with the old
kernel.... Doesn't make much sense to me. I verified I have the latest
commit checked out and built.

Thanks,

Russell

[-- Attachment #1.2: stack_trace_12_1.txt --]
[-- Type: text/plain, Size: 19587 bytes --]

Dec  1 12:51:14 10.243.0.116 [  334.869230] ------------[ cut here ]------------
Dec  1 12:51:14 10.243.0.116 [  334.869235] DEBUG_LOCKS_WARN_ON(!lockdep_hardirqs_enabled())
Dec  1 12:51:14 10.243.0.116 [  334.869247] WARNING: CPU: 5 PID: 1999 at kernel/locking/lockdep.c:5550 check_flags+0x1a4/0x1d0
Dec  1 12:51:14 10.243.0.116 [  334.869262] Modules linked in: xt_CHECKSUM xt_MASQUERADE tun netconsole ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 xt_conntrack ebtable_nat ebtable_broute ip6table_nat ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_nat iptable_mangle iptable_security iptable_raw nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 libcrc32c ip_set nfnetlink ebtable_filter ebtables rfkill ip6table_filter ip6_tables iptable_filter sunrpc dm_mirror dm_region_hash dm_log dm_mod x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel iTCO_wdt gpio_ich iTCO_vendor_support mxm_wmi kvm irqbypass crct10dif_pclmul crc32_pclmul ghash_clmulni_intel aesni_intel crypto_simd cryptd rapl joydev intel_cstate input_leds ftdi_sio pcspkr i2c_i801 sg mei_me i2c_smbus intel_pch_thermal lpc_ich mei mfd_core ioatdma wmi ip_tables ext4 mbcache jbd2 ast drm_vram_helper drm_kms_helper ixgbe sr_mod sd_mod syscopyarea cdrom sysfillrect t10_pi sysimgblt fb_sys_fops drm_ttm_helper ttm mdio drm ahci i
Dec  1 12:51:14 10.243.0.116 [  334.869438]  libahci libata crc32c_intel ptp pps_core i2c_algo_bit dca fuse
Dec  1 12:51:14 10.243.0.116 [  334.869456] CPU: 5 PID: 1999 Comm: TxStartup Not tainted 5.15.77evl-g5e681f841fc3 #3
Dec  1 12:51:14 10.243.0.116 [  334.869463] Hardware name: Supermicro Super Server/X10SDV-6C+-TLN4F, BIOS 2.1 11/22/2019
Dec  1 12:51:14 10.243.0.116 [  334.869466] IRQ stage: Linux
Dec  1 12:51:14 10.243.0.116 [  334.869469] RIP: 0010:check_flags+0x1a4/0x1d0
Dec  1 12:51:14 10.243.0.116 [  334.869476] Code: ff ff e8 8f b2 6a ff 85 c0 74 21 44 8b 05 cc 54 60 01 45 85 c0 75 15 48 c7 c6 40 05 69 82 48 c7 c7 00 d4 68 82 e8 f0 e2 f9 ff <0f> 0b 48 c7 c7 80 05 69 82 e8 69 7a fa ff e9 5c ff ff ff 65 48 8b
Dec  1 12:51:14 10.243.0.116 [  334.869482] RSP: 0018:ffff8881161ffd90 EFLAGS: 00010082
Dec  1 12:51:14 10.243.0.116 [  334.869487] RAX: 0000000000000000 RBX: 1ffff11022c3ffb8 RCX: 0000000000000027
Dec  1 12:51:14 10.243.0.116 [  334.869491] RDX: 0000000000000027 RSI: dffffc0000000000 RDI: ffff8883de137b78
Dec  1 12:51:14 10.243.0.116 [  334.869495] RBP: ffff8881161ffd90 R08: ffffed107bc26f70 R09: ffffed107bc26f70
Dec  1 12:51:14 10.243.0.116 [  334.869500] R10: ffff8883de137b7b R11: ffffed107bc26f6f R12: ffff8881161ffe80
Dec  1 12:51:14 10.243.0.116 [  334.869503] R13: 0000000000000005 R14: 0000000000000000 R15: 0000000000000000
Dec  1 12:51:14 10.243.0.116 [  334.869507] FS:  00007f8f72ffe700(0000) GS:ffff8883de100000(0000) knlGS:0000000000000000
Dec  1 12:51:14 10.243.0.116 [  334.869512] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Dec  1 12:51:14 10.243.0.116 [  334.869516] CR2: 0000000000000000 CR3: 0000000110ece002 CR4: 00000000003706e0
Dec  1 12:51:14 10.243.0.116 [  334.869520] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Dec  1 12:51:14 10.243.0.116 [  334.869523] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Dec  1 12:51:14 10.243.0.116 [  334.869526] Call Trace:
Dec  1 12:51:14 10.243.0.116 [  334.869528]  <TASK>
Dec  1 12:51:14 10.243.0.116 [  334.869532]  lock_acquire+0x150/0x3f0
Dec  1 12:51:14 10.243.0.116 [  334.869545]  ? lockdep_hardirqs_on_prepare+0x250/0x250
Dec  1 12:51:14 10.243.0.116 [  334.869555]  ? lock_release+0x2ac/0x440
Dec  1 12:51:14 10.243.0.116 [  334.869562]  ? __context_tracking_enter+0x6f/0x80
Dec  1 12:51:14 10.243.0.116 [  334.869570]  ? lock_downgrade+0x3e0/0x3e0
Dec  1 12:51:14 10.243.0.116 [  334.869579]  ? rcu_read_lock_sched_held+0x5b/0xd0
Dec  1 12:51:14 10.243.0.116 [  334.869590]  ? sched_clock+0x9/0x10
Dec  1 12:51:14 10.243.0.116 [  334.869599]  vtime_user_exit+0x73/0x130
Dec  1 12:51:14 10.243.0.116 [  334.869607]  ? __context_tracking_exit+0x60/0x70
Dec  1 12:51:14 10.243.0.116 [  334.869615]  __context_tracking_exit+0x60/0x70
Dec  1 12:51:14 10.243.0.116 [  334.869622]  syscall_enter_from_user_mode+0xd8/0x130
Dec  1 12:51:14 10.243.0.116 [  334.869631]  do_syscall_64+0x1d/0xa0
Dec  1 12:51:14 10.243.0.116 [  334.869640]  entry_SYSCALL_64_after_hwframe+0x61/0xcb
Dec  1 12:51:14 10.243.0.116 [  334.869650] RIP: 0033:0x7f8f83d43e44
Dec  1 12:51:14 10.243.0.116 [  334.869655] Code: ff 31 c0 87 83 1c 06 00 00 83 f8 fe 0f 84 42 02 00 00 48 8b 54 24 08 be 18 00 00 00 b8 11 01 00 00 48 8d ba e0 02 00 00 0f 05 <f6> 82 14 06 00 00 04 0f 85 e9 01 00 00 48 8d 7c 24 10 48 c7 44 24
Dec  1 12:51:14 10.243.0.116 [  334.869660] RSP: 002b:00007f8f72fee960 EFLAGS: 00000202 ORIG_RAX: 0000000000000111
Dec  1 12:51:14 10.243.0.116 [  334.869666] RAX: ffffffffffffffda RBX: 00007f8f72ffe700 RCX: 00007f8f83d43e44
Dec  1 12:51:14 10.243.0.116 [  334.869670] RDX: 00007f8f72ffe700 RSI: 0000000000000018 RDI: 00007f8f72ffe9e0
Dec  1 12:51:14 10.243.0.116 [  334.869674] RBP: 0000000000000000 R08: 00007f8f72ffe700 R09: 00007f8f72ffe700
Dec  1 12:51:14 10.243.0.116 [  334.869677] R10: 00007f8f72ffe9d0 R11: 0000000000000202 R12: 0000000000000001
Dec  1 12:51:14 10.243.0.116 [  334.869681] R13: 0000000001001000 R14: 0000000000000000 R15: 00007f8f72ffe700
Dec  1 12:51:14 10.243.0.116 [  334.869692]  </TASK>
Dec  1 12:51:14 10.243.0.116 [  334.869694] irq event stamp: 6
Dec  1 12:51:14 10.243.0.116 [  334.869696] hardirqs last  enabled at (5): [<ffffffff81002725>] ret_from_fork+0x15/0x30
Dec  1 12:51:14 10.243.0.116 [  334.869705] hardirqs last disabled at (6): [<ffffffff8213c46d>] do_syscall_64+0x1d/0xa0
Dec  1 12:51:14 10.243.0.116 [  334.869713] softirqs last  enabled at (0): [<ffffffff810c4f7a>] copy_process+0x12fa/0x3750
Dec  1 12:51:14 10.243.0.116 [  334.869720] softirqs last disabled at (0): [<0000000000000000>] 0x0
Dec  1 12:51:14 10.243.0.116 [  334.869724] ---[ end trace fa2b3c78041cd9c9 ]---
Dec  1 12:51:14 10.243.0.116 [  334.869727] possible reason: unannotated irqs-on.
Dec  1 12:51:14 10.243.0.116 [  334.869729] irq event stamp: 6
Dec  1 12:51:14 10.243.0.116 [  334.869730] hardirqs last  enabled at (5): [<ffffffff81002725>] ret_from_fork+0x15/0x30
Dec  1 12:51:14 10.243.0.116 [  334.869737] hardirqs last disabled at (6): [<ffffffff8213c46d>] do_syscall_64+0x1d/0xa0
Dec  1 12:51:14 10.243.0.116 [  334.869745] softirqs last  enabled at (0): [<ffffffff810c4f7a>] copy_process+0x12fa/0x3750
Dec  1 12:51:14 10.243.0.116 [  334.869750] softirqs last disabled at (0): [<0000000000000000>] 0x0
Dec  1 12:51:14 10.243.0.116 [  335.305095] ------------[ cut here ]------------
Dec  1 12:51:14 10.243.0.116 [  335.305100] WARNING: CPU: 5 PID: 1999 at kernel/evl/mutex.c:659 __evl_unlock_mutex+0x2a9/0x430
Dec  1 12:51:14 10.243.0.116 [  335.305114] Modules linked in: xt_CHECKSUM xt_MASQUERADE tun netconsole ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 xt_conntrack ebtable_nat ebtable_broute ip6table_nat ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_nat iptable_mangle iptable_security iptable_raw nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 libcrc32c ip_set nfnetlink ebtable_filter ebtables rfkill ip6table_filter ip6_tables iptable_filter sunrpc dm_mirror dm_region_hash dm_log dm_mod x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel iTCO_wdt gpio_ich iTCO_vendor_support mxm_wmi kvm irqbypass crct10dif_pclmul crc32_pclmul ghash_clmulni_intel aesni_intel crypto_simd cryptd rapl joydev intel_cstate input_leds ftdi_sio pcspkr i2c_i801 sg mei_me i2c_smbus intel_pch_thermal lpc_ich mei mfd_core ioatdma wmi ip_tables ext4 mbcache jbd2 ast drm_vram_helper drm_kms_helper ixgbe sr_mod sd_mod syscopyarea cdrom sysfillrect t10_pi sysimgblt fb_sys_fops drm_ttm_helper ttm mdio drm ahci i
Dec  1 12:51:14 10.243.0.116 [  335.305283]  libahci libata crc32c_intel ptp pps_core i2c_algo_bit dca fuse
Dec  1 12:51:14 10.243.0.116 [  335.305300] CPU: 5 PID: 1999 Comm: DummyTxSample Tainted: G        W         5.15.77evl-g5e681f841fc3 #3
Dec  1 12:51:14 10.243.0.116 [  335.305308] Hardware name: Supermicro Super Server/X10SDV-6C+-TLN4F, BIOS 2.1 11/22/2019
Dec  1 12:51:14 10.243.0.116 [  335.305311] IRQ stage: EVL
Dec  1 12:51:14 10.243.0.116 [  335.305314] RIP: 0010:__evl_unlock_mutex+0x2a9/0x430
Dec  1 12:51:14 10.243.0.116 [  335.305320] Code: f7 e8 db ec 1d 00 48 c7 c6 38 0d 30 81 48 89 df 4d 8b 7c 24 78 e8 d7 f8 e9 ff 4c 89 e7 e8 8f 43 ea ff 41 55 9d 4d 39 fe 75 bc <0f> 0b eb b8 85 c0 75 3c 48 89 df e8 67 d8 ff ff e9 5d fe ff ff 65
Dec  1 12:51:14 10.243.0.116 [  335.305326] RSP: 0018:ffff8881161ffac0 EFLAGS: 00010046
Dec  1 12:51:14 10.243.0.116 [  335.305331] RAX: 0000000000000000 RBX: ffff8881015b6018 RCX: ffffffff811a5187
Dec  1 12:51:14 10.243.0.116 [  335.305335] RDX: 0000000000000003 RSI: dffffc0000000000 RDI: ffff8881015b6000
Dec  1 12:51:14 10.243.0.116 [  335.305339] RBP: ffff8881161ffb08 R08: ffffed10202b6c01 R09: ffffed10202b6c01
Dec  1 12:51:14 10.243.0.116 [  335.305343] R10: ffff8881015b6003 R11: ffffed10202b6c00 R12: ffff8881015b6000
Dec  1 12:51:14 10.243.0.116 [  335.305347] R13: 0000000000000082 R14: ffff8881015b6078 R15: ffff8881015b6078
Dec  1 12:51:14 10.243.0.116 [  335.305351] FS:  00007f8f72ffe700(0000) GS:ffff8883de100000(0000) knlGS:0000000000000000
Dec  1 12:51:14 10.243.0.116 [  335.305356] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Dec  1 12:51:14 10.243.0.116 [  335.305360] CR2: 0000000000000000 CR3: 0000000110ece002 CR4: 00000000003706e0
Dec  1 12:51:14 10.243.0.116 [  335.305363] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Dec  1 12:51:14 10.243.0.116 [  335.305366] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Dec  1 12:51:14 10.243.0.116 [  335.305370] Call Trace:
Dec  1 12:51:14 10.243.0.116 [  335.305372]  <TASK>
Dec  1 12:51:14 10.243.0.116 [  335.305381]  monitor_oob_ioctl+0x1019/0x1670
Dec  1 12:51:14 10.243.0.116 [  335.305388]  ? rcu_read_lock_held_common+0x25/0x70
Dec  1 12:51:14 10.243.0.116 [  335.305405]  ? enter_monitor+0x160/0x160
Dec  1 12:51:14 10.243.0.116 [  335.305415]  ? lock_release+0xd5/0x440
Dec  1 12:51:14 10.243.0.116 [  335.305423]  ? evl_get_file+0xa3/0x130
Dec  1 12:51:14 10.243.0.116 [  335.305432]  ? lock_downgrade+0x3e0/0x3e0
Dec  1 12:51:14 10.243.0.116 [  335.305440]  ? do_raw_spin_lock+0x119/0x1e0
Dec  1 12:51:14 10.243.0.116 [  335.305447]  ? spin_dump+0x90/0x90
Dec  1 12:51:14 10.243.0.116 [  335.305457]  ? __kasan_check_read+0x11/0x20
Dec  1 12:51:14 10.243.0.116 [  335.305473]  EVL_ioctl+0x81/0xf0
Dec  1 12:51:14 10.243.0.116 [  335.305483]  do_oob_syscall+0x5ac/0x6c0
Dec  1 12:51:14 10.243.0.116 [  335.305491]  ? prepare_for_signal+0x1b0/0x1b0
Dec  1 12:51:14 10.243.0.116 [  335.305501]  ? rcu_dynticks_eqs_enter+0xe/0x20
Dec  1 12:51:14 10.243.0.116 [  335.305513]  handle_oob_syscall+0x189/0x230
Dec  1 12:51:14 10.243.0.116 [  335.305522]  ? handle_pipelined_syscall+0x840/0x840
Dec  1 12:51:14 10.243.0.116 [  335.305539]  pipeline_syscall+0xa7/0x1c0
Dec  1 12:51:14 10.243.0.116 [  335.305551]  syscall_enter_from_user_mode+0x5f/0x130
Dec  1 12:51:14 10.243.0.116 [  335.305559]  do_syscall_64+0x1d/0xa0
Dec  1 12:51:14 10.243.0.116 [  335.305568]  entry_SYSCALL_64_after_hwframe+0x61/0xcb
Dec  1 12:51:14 10.243.0.116 [  335.305577] RIP: 0033:0x7f8f8bb09b4b
Dec  1 12:51:14 10.243.0.116 [  335.305582] Code: Unable to access opcode bytes at RIP 0x7f8f8bb09b21.
Dec  1 12:51:14 10.243.0.116 [  335.305584] RSP: 002b:00007f8f72fee3f0 EFLAGS: 00000246 ORIG_RAX: 000000000000009d
Dec  1 12:51:14 10.243.0.116 [  335.305590] RAX: ffffffffffffffda RBX: 00007f8f72fee490 RCX: 00007f8f8bb09b4b
Dec  1 12:51:14 10.243.0.116 [  335.305594] RDX: 00000000c0186d03 RSI: 0000000000000049 RDI: 0000000010000002
Dec  1 12:51:14 10.243.0.116 [  335.305598] RBP: 00000000c0186d03 R08: 0000000000000000 R09: 00007ffc749bd080
Dec  1 12:51:14 10.243.0.116 [  335.305601] R10: 00007f8f72fee490 R11: 0000000000000246 R12: 0000000000000049
Dec  1 12:51:14 10.243.0.116 [  335.305605] R13: 0000000000000001 R14: 0000000006150743 R15: 00007f8f72fee590
Dec  1 12:51:14 10.243.0.116 [  335.305620]  </TASK>
Dec  1 12:51:14 10.243.0.116 [  335.305622] irq event stamp: 6
Dec  1 12:51:14 10.243.0.116 [  335.305624] hardirqs last  enabled at (5): [<ffffffff81002725>] ret_from_fork+0x15/0x30
Dec  1 12:51:14 10.243.0.116 [  335.305633] hardirqs last disabled at (6): [<ffffffff8213c46d>] do_syscall_64+0x1d/0xa0
Dec  1 12:51:14 10.243.0.116 [  335.305641] softirqs last  enabled at (0): [<ffffffff810c4f7a>] copy_process+0x12fa/0x3750
Dec  1 12:51:14 10.243.0.116 [  335.305647] softirqs last disabled at (0): [<0000000000000000>] 0x0
Dec  1 12:51:14 10.243.0.116 [  335.305652] ---[ end trace fa2b3c78041cd9ca ]---
Dec  1 12:51:14 10.243.0.116 [  335.305669] ------------[ cut here ]------------
Dec  1 12:51:14 10.243.0.116 [  335.305671] WARNING: CPU: 5 PID: 1999 at kernel/evl/sched/core.c:989 __evl_schedule+0x73a/0x1180
Dec  1 12:51:14 10.243.0.116 [  335.305679] Modules linked in: xt_CHECKSUM xt_MASQUERADE tun netconsole ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 xt_conntrack ebtable_nat ebtable_broute ip6table_nat ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_nat iptable_mangle iptable_security iptable_raw nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 libcrc32c ip_set nfnetlink ebtable_filter ebtables rfkill ip6table_filter ip6_tables iptable_filter sunrpc dm_mirror dm_region_hash dm_log dm_mod x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel iTCO_wdt gpio_ich iTCO_vendor_support mxm_wmi kvm irqbypass crct10dif_pclmul crc32_pclmul ghash_clmulni_intel aesni_intel crypto_simd cryptd rapl joydev intel_cstate input_leds ftdi_sio pcspkr i2c_i801 sg mei_me i2c_smbus intel_pch_thermal lpc_ich mei mfd_core ioatdma wmi ip_tables ext4 mbcache jbd2 ast drm_vram_helper drm_kms_helper ixgbe sr_mod sd_mod syscopyarea cdrom sysfillrect t10_pi sysimgblt fb_sys_fops drm_ttm_helper ttm mdio drm ahci i
Dec  1 12:51:14 10.243.0.116 [  335.305834]  libahci libata crc32c_intel ptp pps_core i2c_algo_bit dca fuse
Dec  1 12:51:14 10.243.0.116 [  335.305849] CPU: 5 PID: 1999 Comm: DummyTxSample Tainted: G        W         5.15.77evl-g5e681f841fc3 #3
Dec  1 12:51:14 10.243.0.116 [  335.305856] Hardware name: Supermicro Super Server/X10SDV-6C+-TLN4F, BIOS 2.1 11/22/2019
Dec  1 12:51:14 10.243.0.116 [  335.305858] IRQ stage: EVL
Dec  1 12:51:14 10.243.0.116 [  335.305860] RIP: 0010:__evl_schedule+0x73a/0x1180
Dec  1 12:51:14 10.243.0.116 [  335.305870] Code: 46 78 48 39 d8 0f 85 45 fa ff ff 4c 8b 75 b0 48 c7 c6 eb ed 31 81 4c 89 f7 e8 32 18 e8 ff 48 8b 5d d0 48 89 df e8 e6 62 e8 ff <0f> 0b 68 eb ed 31 81 31 f6 45 31 c9 41 b8 01 00 00 00 31 c9 31 d2
Dec  1 12:51:14 10.243.0.116 [  335.305879] RSP: 0018:ffff8881161ffa28 EFLAGS: 00010046
Dec  1 12:51:14 10.243.0.116 [  335.305888] RAX: 0000000000000000 RBX: ffff8881015b6000 RCX: ffffffff811a5187
Dec  1 12:51:14 10.243.0.116 [  335.305895] RDX: 0000000000000003 RSI: dffffc0000000000 RDI: ffff8881015b6000
Dec  1 12:51:14 10.243.0.116 [  335.305903] RBP: ffff8881161ffab8 R08: ffffed10202b6c01 R09: ffffed10202b6c01
Dec  1 12:51:14 10.243.0.116 [  335.305910] R10: ffff8881015b6003 R11: ffffed10202b6c00 R12: ffff8881015b6000
Dec  1 12:51:15 10.243.0.116 [  335.305917] R13: ffff8883de174720 R14: ffff8881015b6018 R15: ffff8883de174720
Dec  1 12:51:15 10.243.0.116 [  335.305925] FS:  00007f8f72ffe700(0000) GS:ffff8883de100000(0000) knlGS:0000000000000000
Dec  1 12:51:15 10.243.0.116 [  335.305934] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Dec  1 12:51:15 10.243.0.116 [  335.305941] CR2: 0000000000000000 CR3: 0000000110ece002 CR4: 00000000003706e0
Dec  1 12:51:15 10.243.0.116 [  335.305948] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Dec  1 12:51:15 10.243.0.116 [  335.305954] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Dec  1 12:51:15 10.243.0.116 [  335.305960] Call Trace:
Dec  1 12:51:15 10.243.0.116 [  335.305964]  <TASK>
Dec  1 12:51:15 10.243.0.116 [  335.305967]  ? monitor_oob_ioctl+0x100d/0x1670
Dec  1 12:51:15 10.243.0.116 [  335.305983]  ? lock_downgrade+0x3e0/0x3e0
Dec  1 12:51:15 10.243.0.116 [  335.305999]  ? do_raw_spin_unlock+0x97/0x140
Dec  1 12:51:15 10.243.0.116 [  335.306020]  __evl_wait_schedule+0x82/0x3c0
Dec  1 12:51:15 10.243.0.116 [  335.306036]  ? __kasan_check_read+0x11/0x20
Dec  1 12:51:15 10.243.0.116 [  335.306054]  monitor_oob_ioctl+0x1047/0x1670
Dec  1 12:51:15 10.243.0.116 [  335.306066]  ? rcu_read_lock_held_common+0x25/0x70
Dec  1 12:51:15 10.243.0.116 [  335.306095]  ? enter_monitor+0x160/0x160
Dec  1 12:51:15 10.243.0.116 [  335.306114]  ? lock_release+0xd5/0x440
Dec  1 12:51:15 10.243.0.116 [  335.306127]  ? evl_get_file+0xa3/0x130
Dec  1 12:51:15 10.243.0.116 [  335.306142]  ? lock_downgrade+0x3e0/0x3e0
Dec  1 12:51:15 10.243.0.116 [  335.306158]  ? do_raw_spin_lock+0x119/0x1e0
Dec  1 12:51:15 10.243.0.116 [  335.306170]  ? spin_dump+0x90/0x90
Dec  1 12:51:15 10.243.0.116 [  335.306190]  ? __kasan_check_read+0x11/0x20
Dec  1 12:51:15 10.243.0.116 [  335.306219]  EVL_ioctl+0x81/0xf0
Dec  1 12:51:15 10.243.0.116 [  335.306238]  do_oob_syscall+0x5ac/0x6c0
Dec  1 12:51:15 10.243.0.116 [  335.306255]  ? prepare_for_signal+0x1b0/0x1b0
Dec  1 12:51:15 10.243.0.116 [  335.306273]  ? rcu_dynticks_eqs_enter+0xe/0x20
Dec  1 12:51:15 10.243.0.116 [  335.306295]  handle_oob_syscall+0x189/0x230
Dec  1 12:51:15 10.243.0.116 [  335.306312]  ? handle_pipelined_syscall+0x840/0x840
Dec  1 12:51:15 10.243.0.116 [  335.306344]  pipeline_syscall+0xa7/0x1c0
Dec  1 12:51:15 10.243.0.116 [  335.306356]  syscall_enter_from_user_mode+0x5f/0x130
Dec  1 12:51:15 10.243.0.116 [  335.306364]  do_syscall_64+0x1d/0xa0
Dec  1 12:51:15 10.243.0.116 [  335.306373]  entry_SYSCALL_64_after_hwframe+0x61/0xcb
Dec  1 12:51:15 10.243.0.116 [  335.306380] RIP: 0033:0x7f8f8bb09b4b
Dec  1 12:51:15 10.243.0.116 [  335.306384] Code: Unable to access opcode bytes at RIP 0x7f8f8bb09b21.
Dec  1 12:51:15 10.243.0.116 [  335.306387] RSP: 002b:00007f8f72fee3f0 EFLAGS: 00000246 ORIG_RAX: 000000000000009d
Dec  1 12:51:15 10.243.0.116 [  335.306393] RAX: ffffffffffffffda RBX: 00007f8f72fee490 RCX: 00007f8f8bb09b4b
Dec  1 12:51:15 10.243.0.116 [  335.306396] RDX: 00000000c0186d03 RSI: 0000000000000049 RDI: 0000000010000002
Dec  1 12:51:15 10.243.0.116 [  335.306400] RBP: 00000000c0186d03 R08: 0000000000000000 R09: 00007ffc749bd080
Dec  1 12:51:15 10.243.0.116 [  335.306403] R10: 00007f8f72fee490 R11: 0000000000000246 R12: 0000000000000049
Dec  1 12:51:15 10.243.0.116 [  335.306407] R13: 0000000000000001 R14: 0000000006150743 R15: 00007f8f72fee590
Dec  1 12:51:15 10.243.0.116 [  335.306422]  </TASK>
Dec  1 12:51:15 10.243.0.116 [  335.306424] irq event stamp: 6
Dec  1 12:51:15 10.243.0.116 [  335.306426] hardirqs last  enabled at (5): [<ffffffff81002725>] ret_from_fork+0x15/0x30
Dec  1 12:51:15 10.243.0.116 [  335.306433] hardirqs last disabled at (6): [<ffffffff8213c46d>] do_syscall_64+0x1d/0xa0
Dec  1 12:51:15 10.243.0.116 [  335.306441] softirqs last  enabled at (0): [<ffffffff810c4f7a>] copy_process+0x12fa/0x3750
Dec  1 12:51:15 10.243.0.116 [  335.306447] softirqs last disabled at (0): [<0000000000000000>] 0x0
Dec  1 12:51:15 10.243.0.116 [  335.306451] ---[ end trace fa2b3c78041cd9cb ]---

[-- Attachment #2: smime.p7s --]
[-- Type: application/pkcs7-signature, Size: 6759 bytes --]

Re: [External] - Re: EVL Memory

[Philippe Gerum]

Russell Johnson <russell.johnson@kratosdefense.com> writes:

> [[S/MIME Signed Part:Undecided]]
> Hi Philippe,
>
> I pulled your latest commit on the "next" branch (5e681f841fc3), I built it
> with the normal debug configuration parameters enabled, and ran my test case
> against it. I appear to be seeing the same behavior as before. No crashing,
> but many threads that are boosted infinitely. I did get a kernel splat
> again; it looks oddly similar to the one I sent previously with the old
> kernel.... Doesn't make much sense to me. I verified I have the latest
> commit checked out and built.
>
> Thanks,
>
> Russell
>
> [2. text/plain; stack_trace_12_1.txt]...
>
> [[End of S/MIME Signed Part]]

Could you enable CONFIG_LOCALVERSION_AUTO, then check that uname reports
the right commit, e.g.?

root@homelab-qemu-x86_64:~# uname -a
Linux homelab-qemu-x86_64 5.15.77-00732-g5e681f841fc3 #9 SMP PREEMPT IRQPIPE Thu Dec 1 15:32:38 CET 2022 x86_64 GNU/Linux
                                         ^^^^^^^^^^^^

Next, I would disable CONFIG_DEBUG_LOCKDEP for now, the first warning in
the series relates to debugging the lockdep infrastructure, which is not
directly related to EVL, I'll address that one later.

-- 
Philippe.

RE: [External] - Re: EVL Memory

[Russell Johnson]

[-- Attachment #1: Type: text/plain, Size: 141 bytes --]

I do already have that enabled and have confirmed the proper commit shows up
in uname. I will disable LOCKDEP and re-run.

Thanks,

Russell


[-- Attachment #2: smime.p7s --]
[-- Type: application/pkcs7-signature, Size: 6759 bytes --]

Re: [External] - Re: EVL Memory

[Philippe Gerum]

Russell Johnson <russell.johnson@kratosdefense.com> writes:

> [[S/MIME Signed Part:Undecided]]
> I do already have that enabled and have confirmed the proper commit shows up
> in uname. I will disable LOCKDEP and re-run.
>
> Thanks,
>
> Russell
>
> [[End of S/MIME Signed Part]]

I realized that the line information from the kernel splats was matching
the latest sources anyway, so you are definitely running the right
kernel. Can you observe that the threads are indeed stuck with a boosted
priority that never changes, using evl ps?

-- 
Philippe.

Re: [External] - Re: EVL Memory

[Philippe Gerum]

Philippe Gerum <rpm@xenomai.org> writes:

> Russell Johnson <russell.johnson@kratosdefense.com> writes:
>
>> [[S/MIME Signed Part:Undecided]]
>> I do already have that enabled and have confirmed the proper commit shows up
>> in uname. I will disable LOCKDEP and re-run.
>>
>> Thanks,
>>
>> Russell
>>
>> [[End of S/MIME Signed Part]]
>
> I realized that the line information from the kernel splats was matching
> the latest sources anyway, so you are definitely running the right
> kernel. Can you observe that the threads are indeed stuck with a boosted
> priority that never changes, using evl ps?


The patch below should capture a trace snapshot when the assertion
trips, with a backtrace of the most recently executed function calls
before this happens. This works with the "evl trace" command [1],

e.g.

# evl trace -e
... wait for kernel splats
...
# evl trace > tracelog.txt

[1] https://evlproject.org/core/commands/#evl-trace-command



diff --git a/kernel/evl/mutex.c b/kernel/evl/mutex.c
index 16750fd193377..c3c1576975c7e 100644
--- a/kernel/evl/mutex.c
+++ b/kernel/evl/mutex.c
@@ -653,6 +653,7 @@ void __evl_unlock_mutex(struct evl_mutex *mutex)
 	 */
 	if (IS_ENABLED(CONFIG_EVL_DEBUG_CORE)) {
 		bool bad;
+		trace_evl_trigger("BOOST1");
 		raw_spin_lock_irqsave(&curr->lock, flags);
 		bad = curr->state & EVL_T_BOOST && list_empty(&curr->boosters);
 		raw_spin_unlock_irqrestore(&curr->lock, flags);
diff --git a/kernel/evl/sched/core.c b/kernel/evl/sched/core.c
index 3bfbe37b1eb7b..d1307fb1b4fd5 100644
--- a/kernel/evl/sched/core.c
+++ b/kernel/evl/sched/core.c
@@ -985,6 +985,7 @@ void __evl_schedule(void) /* oob or/and hard irqs off (CPU migration-safe) */
 	 */
 	if (IS_ENABLED(CONFIG_EVL_DEBUG_CORE) &&
 		curr->state & EVL_T_BOOST && list_empty(&curr->boosters)) {
+		trace_evl_trigger("BOOST2");
 		raw_spin_unlock(&curr->lock);
 		EVL_WARN_ON_ONCE(CORE, 1);
 		raw_spin_lock(&curr->lock);

-- 
Philippe.

RE: [External] - Re: EVL Memory

[Russell Johnson]

[-- Attachment #1.1: Type: text/plain, Size: 207 bytes --]

I enabled all the ftrace stuff recommended by the wiki page, ran my test
app, and the system ended up getting hung so I couldn't get the output of
the ftrace.. I got a ton of kernel splat.

Thanks,

Russell

[-- Attachment #1.2: ftrace_1.txt --]
[-- Type: text/plain, Size: 238435 bytes --]

Dec  2 12:33:29 10.243.0.116 [ 1715.407337] ==================================================================
Dec  2 12:33:29 10.243.0.116 [ 1715.407376] BUG: KASAN: null-ptr-deref in trace_event_raw_event_evl_sleep_on+0xc3/0x380
Dec  2 12:33:29 10.243.0.116 [ 1715.407448] Read of size 8 at addr 0000000000000070 by task DummyTxSample/3577
Dec  2 12:33:29 10.243.0.116 [ 1715.407487]
Dec  2 12:33:29 10.243.0.116 [ 1715.407528] CPU: 5 PID: 3577 Comm: DummyTxSample Tainted: G        W         5.15.77evl-g5e681f841fc3-dirty #3
Dec  2 12:33:29 10.243.0.116 [ 1715.407573] Hardware name: Supermicro Super Server/X10SDV-6C+-TLN4F, BIOS 2.1 11/22/2019
Dec  2 12:33:29 10.243.0.116 [ 1715.407609] IRQ stage: EVL
Dec  2 12:33:29 10.243.0.116 [ 1715.407651] Call Trace:
Dec  2 12:33:29 10.243.0.116 [ 1715.407703]  <TASK>
Dec  2 12:33:29 10.243.0.116 [ 1715.407822]  dump_stack_lvl+0x90/0xd8
Dec  2 12:33:29 10.243.0.116 [ 1715.407895]  ? trace_event_raw_event_evl_sleep_on+0xc3/0x380
Dec  2 12:33:29 10.243.0.116 [ 1715.407929]  ? trace_event_raw_event_evl_sleep_on+0xc3/0x380
Dec  2 12:33:29 10.243.0.116 [ 1715.407998]  kasan_report.cold.16+0x64/0xdb
Dec  2 12:33:29 10.243.0.116 [ 1715.408318]  ? trace_event_raw_event_evl_sleep_on+0xc3/0x380
Dec  2 12:33:29 10.243.0.116 [ 1715.408844]  __asan_load8+0x6d/0xa0
Dec  2 12:33:29 10.243.0.116 [ 1715.408916]  trace_event_raw_event_evl_sleep_on+0xc3/0x380
Dec  2 12:33:29 10.243.0.116 [ 1715.409364]  ? trace_event_raw_event_evl_init_thread+0x330/0x330
Dec  2 12:33:29 10.243.0.116 [ 1715.409766]  ? 0xffffffffc0b38099
Dec  2 12:33:29 10.243.0.116 [ 1715.409859]  ? evl_delay+0x48/0xf0
Dec  2 12:33:29 10.243.0.116 [ 1715.409892]  ? evl_sleep_on_locked+0x730/0x730
Dec  2 12:33:29 10.243.0.116 [ 1715.410065]  ? __kasan_check_write+0x14/0x20
Dec  2 12:33:29 10.243.0.116 [ 1715.410616]  evl_sleep_on_locked+0x629/0x730
Dec  2 12:33:29 10.243.0.116 [ 1715.410772]  ? evl_sleep_on_locked+0x5/0x730
Dec  2 12:33:29 10.243.0.116 [ 1715.411255]  evl_sleep_on+0x7b/0xb0
Dec  2 12:33:29 10.243.0.116 [ 1715.411887]  evl_delay+0x48/0xf0
Dec  2 12:33:29 10.243.0.116 [ 1715.412140]  clock_oob_ioctl+0x182/0x2d0
Dec  2 12:33:29 10.243.0.116 [ 1715.412534]  ? clock_common_ioctl+0x5e0/0x5e0
Dec  2 12:33:30 10.243.0.116 [ 1715.412868]  ? clock_oob_ioctl+0x5/0x2d0
Dec  2 12:33:30 10.243.0.116 [ 1715.413406]  EVL_ioctl+0x86/0x100
Dec  2 12:33:30 10.243.0.116 [ 1715.413919]  do_oob_syscall+0x5b1/0x6d0
Dec  2 12:33:30 10.243.0.116 [ 1715.414133]  ? prepare_for_signal+0x180/0x180
Dec  2 12:33:30 10.243.0.116 [ 1715.414517]  ? do_oob_syscall+0x5/0x6d0
Dec  2 12:33:30 10.243.0.116 [ 1715.414665]  ? handle_pipelined_syscall+0x840/0x840
Dec  2 12:33:30 10.243.0.116 [ 1715.415066]  handle_oob_syscall+0x18e/0x230
Dec  2 12:33:30 10.243.0.116 [ 1715.415464]  ? handle_pipelined_syscall+0x840/0x840
Dec  2 12:33:30 10.243.0.116 [ 1715.415976]  ? handle_oob_syscall+0x5/0x230
Dec  2 12:33:30 10.243.0.116 [ 1715.416646]  pipeline_syscall+0xac/0x1c0
Dec  2 12:33:30 10.243.0.116 [ 1715.417051]  syscall_enter_from_user_mode+0x5f/0x130
Dec  2 12:33:30 10.243.0.116 [ 1715.417358]  do_syscall_64+0x1d/0xa0
Dec  2 12:33:30 10.243.0.116 [ 1715.417595]  entry_SYSCALL_64_after_hwframe+0x61/0xcb
Dec  2 12:33:30 10.243.0.116 [ 1715.417673] RIP: 0033:0x7f302d4b1b9b
Dec  2 12:33:30 10.243.0.116 [ 1715.417784] Code: Unable to access opcode bytes at RIP 0x7f302d4b1b71.
Dec  2 12:33:30 10.243.0.116 [ 1715.417819] RSP: 002b:00007f3013fef5a0 EFLAGS: 00000246 ORIG_RAX: 000000000000009d
Dec  2 12:33:30 10.243.0.116 [ 1715.417859] RAX: ffffffffffffffda RBX: 00007f3013fef630 RCX: 00007f302d4b1b9b
Dec  2 12:33:30 10.243.0.116 [ 1715.417877] RDX: 0000000040106300 RSI: 0000000000000004 RDI: 0000000010000002
Dec  2 12:33:30 10.243.0.116 [ 1715.417895] RBP: 0000000040106300 R08: 0000000000000000 R09: 00007ffd9df03080
Dec  2 12:33:30 10.243.0.116 [ 1715.417912] R10: 00007f3013fef630 R11: 0000000000000246 R12: 0000000000000004
Dec  2 12:33:30 10.243.0.116 [ 1715.417929] R13: 0000000001001000 R14: 0000000000000000 R15: 00007f3013fff700
Dec  2 12:33:30 10.243.0.116 [ 1715.419069]  </TASK>
Dec  2 12:33:30 10.243.0.116 [ 1715.419089] ==================================================================
Dec  2 12:33:30 10.243.0.116 [ 1715.419105] Disabling lock debugging due to kernel taint
Dec  2 12:33:30 10.243.0.116 [ 1715.419141] EVL: DummyTxSample switching in-band [pid=3577, excpt=14, trace_event_raw_event_evl_sleep_on+0xc3/0x380]
Dec  2 12:33:50 10.243.0.116 [ 1736.447142] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [TxStartup:3574]
Dec  2 12:33:50 10.243.0.116 [ 1736.479659] Modules linked in: xt_CHECKSUM xt_MASQUERADE tun netconsole ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 xt_conntrack ebtable_nat ebtable_broute ip6table_nat ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_nat iptable_mangle iptable_security iptable_raw nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 libcrc32c ip_set nfnetlink ebtable_filter ebtables rfkill ip6table_filter ip6_tables iptable_filter sunrpc dm_mirror dm_region_hash dm_log dm_mod x86_pkg_temp_thermal intel_powerclamp coretemp gpio_ich iTCO_wdt iTCO_vendor_support kvm_intel mxm_wmi kvm irqbypass crct10dif_pclmul crc32_pclmul ghash_clmulni_intel aesni_intel crypto_simd cryptd rapl intel_cstate pcspkr sg i2c_i801 ftdi_sio mei_me i2c_smbus joydev input_leds intel_pch_thermal mei lpc_ich mfd_core ioatdma wmi ip_tables ext4 mbcache jbd2 ast drm_vram_helper drm_kms_helper syscopyarea ixgbe sysfillrect sd_mod sr_mod sysimgblt fb_sys_fops cdrom t10_pi drm_ttm_helper ttm mdio drm igb ah
Dec  2 12:33:50 10.243.0.116 [ 1736.493080]  libahci libata crc32c_intel ptp pps_core i2c_algo_bit dca fuse
Dec  2 12:33:50 10.243.0.116 [ 1736.803990] irq event stamp: 24564
Dec  2 12:33:50 10.243.0.116 [ 1736.840292] hardirqs last  enabled at (24563): [<ffffffff8d1d44ad>] sync_current_irq_stage+0x45d/0x500
Dec  2 12:33:50 10.243.0.116 [ 1736.878499] hardirqs last disabled at (24564): [<ffffffff8d49923b>] handle_mm_fault+0x48b/0x520
Dec  2 12:33:50 10.243.0.116 [ 1736.916737] softirqs last  enabled at (0): [<ffffffff8d0ca587>] copy_process+0x1307/0x3770
Dec  2 12:33:50 10.243.0.116 [ 1736.955360] softirqs last disabled at (0): [<0000000000000000>] 0x0
Dec  2 12:33:50 10.243.0.116 [ 1736.993539] CPU: 0 PID: 3574 Comm: TxStartup Tainted: G    B   W         5.15.77evl-g5e681f841fc3-dirty #3
Dec  2 12:33:50 10.243.0.116 [ 1737.047281] Hardware name: Supermicro Super Server/X10SDV-6C+-TLN4F, BIOS 2.1 11/22/2019
Dec  2 12:33:50 10.243.0.116 [ 1737.086618] IRQ stage: Linux
Dec  2 12:33:50 10.243.0.116 [ 1737.124857] RIP: 0010:proxy_set_next_ktime+0x77/0x90
Dec  2 12:33:51 10.243.0.116 [ 1737.163868] Code: 74 37 49 8d 7c 24 30 e8 d7 2d 1d 00 49 8b 44 24 30 4c 89 e7 e8 da 42 0b 01 48 8d 34 03 31 d2 4c 89 ef e8 5c 14 00 00 41 56 9d <31> c0 5b 41 5c 41 5d 41 5e 5d c3 cc cc cc cc e8 25 f5 eb ff eb d9
Dec  2 12:33:51 10.243.0.116 [ 1737.247966] RSP: 0018:ffff88841e209e80 EFLAGS: 00000283 ORIG_RAX: 0000000000000000
Dec  2 12:33:51 10.243.0.116 [ 1737.291844] RAX: 0000000000000000 RBX: 0000000000000000 RCX: 0000000000000000
Dec  2 12:33:51 10.243.0.116 [ 1737.335764] RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000000
Dec  2 12:33:51 10.243.0.116 [ 1737.379722] RBP: ffff88841e209ea0 R08: 0000000000000000 R09: 0000000000000000
Dec  2 12:33:51 10.243.0.116 [ 1737.423854] R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000000
Dec  2 12:33:51 10.243.0.116 [ 1737.468065] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000
Dec  2 12:33:51 10.243.0.116 [ 1737.512227] FS:  00007f301ae2e700(0000) GS:ffff88841e200000(0000) knlGS:0000000000000000
Dec  2 12:33:51 10.243.0.116 [ 1737.557029] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Dec  2 12:33:51 10.243.0.116 [ 1737.601610] CR2: 00007f1e4033b660 CR3: 0000000137d08006 CR4: 00000000003706f0
Dec  2 12:33:51 10.243.0.116 [ 1737.647012] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Dec  2 12:33:51 10.243.0.116 [ 1737.693142] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Dec  2 12:33:51 10.243.0.116 [ 1737.739454] Call Trace:
Dec  2 12:33:51 10.243.0.116 [ 1737.784755]  <IRQ>
Dec  2 12:33:51 10.243.0.116 [ 1737.830140]  ? __hrtimer_run_queues+0x5/0x780
Dec  2 12:33:51 10.243.0.116 [ 1737.876648]  ? hrtimer_interrupt+0x1aa/0x360
Dec  2 12:33:51 10.243.0.116 [ 1737.922863]  ? hrtimer_interrupt+0x5/0x360
Dec  2 12:33:51 10.243.0.116 [ 1737.968896]  ? proxy_irq_handler+0x37/0x50
Dec  2 12:33:51 10.243.0.116 [ 1738.014332]  ? handle_synthetic_irq+0xec/0x320
Dec  2 12:33:51 10.243.0.116 [ 1738.059846]  ? do_irq_inband+0x2f/0x40
Dec  2 12:33:51 10.243.0.116 [ 1738.104678]  ? arch_do_IRQ_pipelined+0xcd/0x5b0
Dec  2 12:33:51 10.243.0.116 [ 1738.149293]  </IRQ>
Dec  2 12:33:52 10.243.0.116 [ 1738.192714]  <TASK>
Dec  2 12:33:52 10.243.0.116 [ 1738.236021]  ? sync_current_irq_stage+0x2e3/0x500
Dec  2 12:33:52 10.243.0.116 [ 1738.280446]  ? sync_irq_stage+0x1af/0x1e0
Dec  2 12:33:52 10.243.0.116 [ 1738.323930]  ? synchronize_pipeline+0x8b/0xe0
Dec  2 12:33:52 10.243.0.116 [ 1738.367375]  ? handle_irq_pipelined_finish+0xf6/0x240
Dec  2 12:33:52 10.243.0.116 [ 1738.410732]  ? arch_pipeline_entry+0x81/0x110
Dec  2 12:33:52 10.243.0.116 [ 1738.453483]  ? sysvec_apic_timer_interrupt+0xe/0x20
Dec  2 12:33:52 10.243.0.116 [ 1738.495505]  ? asm_sysvec_apic_timer_interrupt+0x1b/0x20
Dec  2 12:33:52 10.243.0.116 [ 1738.538163]  ? smp_call_function_many_cond+0x1c7/0x4e0
Dec  2 12:33:52 10.243.0.116 [ 1738.579862]  ? __asan_store2+0xa0/0xa0
Dec  2 12:33:52 10.243.0.116 [ 1738.620658]  ? smp_call_function_many_cond+0x1c7/0x4e0
Dec  2 12:33:52 10.243.0.116 [ 1738.661383]  ? leave_mm+0x40/0x40
Dec  2 12:33:52 10.243.0.116 [ 1738.757429]  ? x86_configure_nx+0x70/0x70
Dec  2 12:33:52 10.243.0.116 [ 1738.797065]  ? on_each_cpu_cond_mask+0x29/0x40
Dec  2 12:33:52 10.243.0.116 [ 1738.836149]  ? native_flush_tlb_multi+0xbe/0x250
Dec  2 12:33:52 10.243.0.116 [ 1738.874960]  ? flush_tlb_mm_range+0x136/0x1f0
Dec  2 12:33:52 10.243.0.116 [ 1738.913432]  ? change_protection_range+0x8bd/0xea0
Dec  2 12:33:52 10.243.0.116 [ 1738.953489]  ? prot_none_pte_entry+0x70/0x70
Dec  2 12:33:52 10.243.0.116 [ 1738.991085]  ? 0xffffffffc0b38099
Dec  2 12:33:52 10.243.0.116 [ 1739.028345]  ? change_protection_range+0x5/0xea0
Dec  2 12:33:52 10.243.0.116 [ 1739.065872]  ? change_protection+0x50/0x90
Dec  2 12:33:52 10.243.0.116 [ 1739.102540]  ? mprotect_fixup+0x2c5/0x490
Dec  2 12:33:52 10.243.0.116 [ 1739.139304]  ? change_protection+0x90/0x90
Dec  2 12:33:52 10.243.0.116 [ 1739.175299]  ? mprotect_fixup+0x5/0x490
Dec  2 12:33:53 10.243.0.116 [ 1739.211544]  ? do_mprotect_pkey+0x32a/0x590
Dec  2 12:33:53 10.243.0.116 [ 1739.246174]  ? mprotect_fixup+0x490/0x490
Dec  2 12:33:53 10.243.0.116 [ 1739.278329]  ? do_mprotect_pkey+0x5/0x590
Dec  2 12:33:53 10.243.0.116 [ 1739.310804]  ? __x64_sys_mprotect+0x48/0x60
Dec  2 12:33:53 10.243.0.116 [ 1739.342598]  ? do_syscall_64+0x4c/0xa0
Dec  2 12:33:53 10.243.0.116 [ 1739.374063]  ? entry_SYSCALL_64_after_hwframe+0x61/0xcb
Dec  2 12:33:53 10.243.0.116 [ 1739.407243]  </TASK>
Dec  2 12:34:18 10.243.0.116 [ 1764.447279] watchdog: BUG: soft lockup - CPU#0 stuck for 49s! [TxStartup:3574]
Dec  2 12:34:18 10.243.0.116 [ 1764.479915] Modules linked in: xt_CHECKSUM xt_MASQUERADE tun netconsole ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 xt_conntrack ebtable_nat ebtable_broute ip6table_nat ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_nat iptable_mangle iptable_security iptable_raw nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 libcrc32c ip_set nfnetlink ebtable_filter ebtables rfkill ip6table_filter ip6_tables iptable_filter sunrpc dm_mirror dm_region_hash dm_log dm_mod x86_pkg_temp_thermal intel_powerclamp coretemp gpio_ich iTCO_wdt iTCO_vendor_support kvm_intel mxm_wmi kvm irqbypass crct10dif_pclmul crc32_pclmul ghash_clmulni_intel aesni_intel crypto_simd cryptd rapl intel_cstate pcspkr sg i2c_i801 ftdi_sio mei_me i2c_smbus joydev input_leds intel_pch_thermal mei lpc_ich mfd_core ioatdma wmi ip_tables ext4 mbcache jbd2 ast drm_vram_helper drm_kms_helper syscopyarea ixgbe sysfillrect sd_mod sr_mod sysimgblt fb_sys_fops cdrom t10_pi drm_ttm_helper ttm mdio drm igb ah
Dec  2 12:34:18 10.243.0.116 [ 1764.493334]  libahci libata crc32c_intel ptp pps_core i2c_algo_bit dca fuse
Dec  2 12:34:18 10.243.0.116 [ 1764.789622] irq event stamp: 24564
Dec  2 12:34:18 10.243.0.116 [ 1764.823694] hardirqs last  enabled at (24563): [<ffffffff8d1d44ad>] sync_current_irq_stage+0x45d/0x500
Dec  2 12:34:18 10.243.0.116 [ 1764.859866] hardirqs last disabled at (24564): [<ffffffff8d49923b>] handle_mm_fault+0x48b/0x520
Dec  2 12:34:18 10.243.0.116 [ 1764.896015] softirqs last  enabled at (0): [<ffffffff8d0ca587>] copy_process+0x1307/0x3770
Dec  2 12:34:18 10.243.0.116 [ 1764.932341] softirqs last disabled at (0): [<0000000000000000>] 0x0
Dec  2 12:34:18 10.243.0.116 [ 1764.968289] CPU: 0 PID: 3574 Comm: TxStartup Tainted: G    B   W    L    5.15.77evl-g5e681f841fc3-dirty #3
Dec  2 12:34:18 10.243.0.116 [ 1765.005626] Hardware name: Supermicro Super Server/X10SDV-6C+-TLN4F, BIOS 2.1 11/22/2019
Dec  2 12:34:18 10.243.0.116 [ 1765.042956] IRQ stage: Linux
Dec  2 12:34:18 10.243.0.116 [ 1765.079153] RIP: 0010:proxy_set_next_ktime+0x77/0x90
Dec  2 12:34:18 10.243.0.116 [ 1765.116160] Code: 74 37 49 8d 7c 24 30 e8 d7 2d 1d 00 49 8b 44 24 30 4c 89 e7 e8 da 42 0b 01 48 8d 34 03 31 d2 4c 89 ef e8 5c 14 00 00 41 56 9d <31> c0 5b 41 5c 41 5d 41 5e 5d c3 cc cc cc cc e8 25 f5 eb ff eb d9
Dec  2 12:34:19 10.243.0.116 [ 1765.196086] RSP: 0018:ffff88841e209e80 EFLAGS: 00000283 ORIG_RAX: 0000000000000000
Dec  2 12:34:19 10.243.0.116 [ 1765.237762] RAX: 0000000000000000 RBX: 0000000000000000 RCX: 0000000000000000
Dec  2 12:34:19 10.243.0.116 [ 1765.279483] RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000000
Dec  2 12:34:19 10.243.0.116 [ 1765.321180] RBP: ffff88841e209ea0 R08: 0000000000000000 R09: 0000000000000000
Dec  2 12:34:19 10.243.0.116 [ 1765.362899] R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000000
Dec  2 12:34:19 10.243.0.116 [ 1765.404731] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000
Dec  2 12:34:19 10.243.0.116 [ 1765.446628] FS:  00007f301ae2e700(0000) GS:ffff88841e200000(0000) knlGS:0000000000000000
Dec  2 12:34:19 10.243.0.116 [ 1765.489146] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Dec  2 12:34:19 10.243.0.116 [ 1765.531366] CR2: 00007f1e4033b660 CR3: 0000000137d08006 CR4: 00000000003706f0
Dec  2 12:34:19 10.243.0.116 [ 1765.574265] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Dec  2 12:34:19 10.243.0.116 [ 1765.617819] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Dec  2 12:34:19 10.243.0.116 [ 1765.661535] Call Trace:
Dec  2 12:34:19 10.243.0.116 [ 1765.704246]  <IRQ>
Dec  2 12:34:19 10.243.0.116 [ 1765.747003]  ? __hrtimer_run_queues+0x5/0x780
Dec  2 12:34:19 10.243.0.116 [ 1765.790969]  ? hrtimer_interrupt+0x1aa/0x360
Dec  2 12:34:19 10.243.0.116 [ 1765.834540]  ? hrtimer_interrupt+0x5/0x360
Dec  2 12:34:19 10.243.0.116 [ 1765.878267]  ? proxy_irq_handler+0x37/0x50
Dec  2 12:34:19 10.243.0.116 [ 1765.921371]  ? handle_synthetic_irq+0xec/0x320
Dec  2 12:34:19 10.243.0.116 [ 1765.964775]  ? do_irq_inband+0x2f/0x40
Dec  2 12:34:19 10.243.0.116 [ 1766.007748]  ? arch_do_IRQ_pipelined+0xcd/0x5b0
Dec  2 12:34:19 10.243.0.116 [ 1766.050937]  </IRQ>
Dec  2 12:34:19 10.243.0.116 [ 1766.093286]  <TASK>
Dec  2 12:34:19 10.243.0.116 [ 1766.135921]  ? sync_current_irq_stage+0x2e3/0x500
Dec  2 12:34:19 10.243.0.116 [ 1766.179998]  ? sync_irq_stage+0x1af/0x1e0
Dec  2 12:34:20 10.243.0.116 [ 1766.223200]  ? synchronize_pipeline+0x8b/0xe0
Dec  2 12:34:20 10.243.0.116 [ 1766.266455]  ? handle_irq_pipelined_finish+0xf6/0x240
Dec  2 12:34:20 10.243.0.116 [ 1766.310033]  ? arch_pipeline_entry+0x81/0x110
Dec  2 12:34:20 10.243.0.116 [ 1766.352702]  ? sysvec_apic_timer_interrupt+0xe/0x20
Dec  2 12:34:20 10.243.0.116 [ 1766.394696]  ? asm_sysvec_apic_timer_interrupt+0x1b/0x20
Dec  2 12:34:20 10.243.0.116 [ 1766.437281]  ? smp_call_function_many_cond+0x1c7/0x4e0
Dec  2 12:34:20 10.243.0.116 [ 1766.478887]  ? smp_call_function_many_cond+0x1bd/0x4e0
Dec  2 12:34:20 10.243.0.116 [ 1766.519883]  ? smp_call_function_many_cond+0x1c7/0x4e0
Dec  2 12:34:20 10.243.0.116 [ 1766.560322]  ? leave_mm+0x40/0x40
Dec  2 12:34:20 10.243.0.116 [ 1766.599782]  ? x86_configure_nx+0x70/0x70
Dec  2 12:34:20 10.243.0.116 [ 1766.639112]  ? on_each_cpu_cond_mask+0x29/0x40
Dec  2 12:34:20 10.243.0.116 [ 1766.781244]  ? native_flush_tlb_multi+0xbe/0x250
Dec  2 12:34:20 10.243.0.116 [ 1766.819877]  ? flush_tlb_mm_range+0x136/0x1f0
Dec  2 12:34:20 10.243.0.116 [ 1766.858215]  ? change_protection_range+0x8bd/0xea0
Dec  2 12:34:20 10.243.0.116 [ 1766.898133]  ? prot_none_pte_entry+0x70/0x70
Dec  2 12:34:20 10.243.0.116 [ 1766.935650]  ? 0xffffffffc0b38099
Dec  2 12:34:20 10.243.0.116 [ 1766.972844]  ? change_protection_range+0x5/0xea0
Dec  2 12:34:20 10.243.0.116 [ 1767.010173]  ? change_protection+0x50/0x90
Dec  2 12:34:20 10.243.0.116 [ 1767.046647]  ? mprotect_fixup+0x2c5/0x490
Dec  2 12:34:20 10.243.0.116 [ 1767.083266]  ? change_protection+0x90/0x90
Dec  2 12:34:20 10.243.0.116 [ 1767.119046]  ? mprotect_fixup+0x5/0x490
Dec  2 12:34:20 10.243.0.116 [ 1767.155109]  ? do_mprotect_pkey+0x32a/0x590
Dec  2 12:34:20 10.243.0.116 [ 1767.189546]  ? mprotect_fixup+0x490/0x490
Dec  2 12:34:21 10.243.0.116 [ 1767.221516]  ? do_mprotect_pkey+0x5/0x590
Dec  2 12:34:21 10.243.0.116 [ 1767.253852]  ? __x64_sys_mprotect+0x48/0x60
Dec  2 12:34:21 10.243.0.116 [ 1767.285437]  ? do_syscall_64+0x4c/0xa0
Dec  2 12:34:21 10.243.0.116 [ 1767.316755]  ? entry_SYSCALL_64_after_hwframe+0x61/0xcb
Dec  2 12:34:21 10.243.0.116 [ 1767.349783]  </TASK>
Dec  2 12:34:34 10.243.0.116 [ 1780.530672] rcu: INFO: rcu_sched detected stalls on CPUs/tasks:
Dec  2 12:34:34 10.243.0.116 [ 1780.554365] rcu: #0115-...!: (1 GPs behind) idle=78b/1/0x4000000000000002 softirq=161154/161154 fqs=1120
Dec  2 12:34:34 10.243.0.116 [ 1780.577036] #011(detected by 2, t=65048 jiffies, g=126833, q=6205)
Dec  2 12:34:34 10.243.0.116 [ 1780.599018] Sending NMI from CPU 2 to CPUs 5:
Dec  2 12:34:34 10.243.0.116 [ 1780.620633] NMI backtrace for cpu 5
Dec  2 12:34:34 10.243.0.116 [ 1780.620676] CPU: 5 PID: 3577 Comm: DummyTxSample Tainted: G    B   W    L    5.15.77evl-g5e681f841fc3-dirty #3
Dec  2 12:34:34 10.243.0.116 [ 1780.620719] Hardware name: Supermicro Super Server/X10SDV-6C+-TLN4F, BIOS 2.1 11/22/2019
Dec  2 12:34:34 10.243.0.116 [ 1780.620754] IRQ stage: EVL
Dec  2 12:34:34 10.243.0.116 [ 1780.620798] RIP: 0010:native_queued_spin_lock_slowpath+0x18c/0x4c0
Dec  2 12:34:34 10.243.0.116 [ 1780.620874] Code: 48 89 df e8 d6 4c 37 00 8b 03 30 e4 44 09 e8 41 89 46 c0 a9 00 01 ff ff 0f 85 0d 03 00 00 85 c0 74 16 8b 03 84 c0 74 10 f3 90 <48> 89 df e8 ac 4c 37 00 8b 03 84 c0 75 f0 48 89 df e8 fe 4b 37 00
Dec  2 12:34:34 10.243.0.116 [ 1780.620914] RSP: 0018:ffff888137737538 EFLAGS: 00000002
Dec  2 12:34:34 10.243.0.116 [ 1780.620982] RAX: 00000000001c0101 RBX: ffff888101aae000 RCX: ffffffff8d1abe74
Dec  2 12:34:34 10.243.0.116 [ 1780.621021] RDX: 0000000000000003 RSI: dffffc0000000000 RDI: ffff888101aae000
Dec  2 12:34:34 10.243.0.116 [ 1780.621054] RBP: ffff888137737618 R08: ffffed1020355c01 R09: ffffed1020355c01
Dec  2 12:34:34 10.243.0.116 [ 1780.621071] R10: ffff888101aae003 R11: ffffed1020355c00 R12: 1ffff11026ee6eaa
Dec  2 12:34:34 10.243.0.116 [ 1780.621089] R13: 0000000000000000 R14: ffff8881377375f0 R15: 0000000000000001
Dec  2 12:34:34 10.243.0.116 [ 1780.621117] FS:  00007f3013fff700(0000) GS:ffff88841e480000(0000) knlGS:0000000000000000
Dec  2 12:34:34 10.243.0.116 [ 1780.621135] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Dec  2 12:34:34 10.243.0.116 [ 1780.621153] CR2: 0000000000000070 CR3: 0000000137d08002 CR4: 00000000003706e0
Dec  2 12:34:34 10.243.0.116 [ 1780.621170] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Dec  2 12:34:34 10.243.0.116 [ 1780.621186] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Dec  2 12:34:34 10.243.0.116 [ 1780.621204] Call Trace:
Dec  2 12:34:34 10.243.0.116 [ 1780.621223]  <TASK>
Dec  2 12:34:34 10.243.0.116 [ 1780.621281]  ? 0xffffffffc0b38099
Dec  2 12:34:34 10.243.0.116 [ 1780.621336]  ? trace_function+0x11e/0x180
Dec  2 12:34:34 10.243.0.116 [ 1780.621457]  ? .slowpath+0x1d/0x1d
Dec  2 12:34:35 10.243.0.116 [ 1780.622062]  ? do_raw_spin_lock+0xe6/0x1e0
Dec  2 12:34:35 10.243.0.116 [ 1780.622247]  ? native_queued_spin_lock_slowpath+0x5/0x4c0
Dec  2 12:34:35 10.243.0.116 [ 1780.622318]  ? 0xffffffffc0b38099
Dec  2 12:34:35 10.243.0.116 [ 1780.622922]  do_raw_spin_lock+0x1cb/0x1e0
Dec  2 12:34:35 10.243.0.116 [ 1780.623097]  ? spin_dump+0x90/0x90
Dec  2 12:34:35 10.243.0.116 [ 1780.623265]  ? do_raw_spin_lock+0x5/0x1e0
Dec  2 12:34:35 10.243.0.116 [ 1780.623903]  evl_switch_inband+0xad/0x7c0
Dec  2 12:34:35 10.243.0.116 [ 1780.624326]  handle_oob_trap_entry+0x10a/0x200
Dec  2 12:34:35 10.243.0.116 [ 1780.624743]  __oob_trap_notify+0x3f/0x50
Dec  2 12:34:35 10.243.0.116 [ 1780.625084]  page_fault_oops+0x1e4/0x3d0
Dec  2 12:34:35 10.243.0.116 [ 1780.625155]  ? trace_function+0x11e/0x180
Dec  2 12:34:35 10.243.0.116 [ 1780.625220]  ? do_user_addr_fault+0x4af/0xa30
Dec  2 12:34:35 10.243.0.116 [ 1780.625355]  ? spurious_kernel_fault+0x200/0x200
Dec  2 12:34:35 10.243.0.116 [ 1780.625497]  ? function_trace_call+0x153/0x1d0
Dec  2 12:34:35 10.243.0.116 [ 1780.625567]  ? do_user_addr_fault+0x4af/0xa30
Dec  2 12:34:35 10.243.0.116 [ 1780.625638]  ? spurious_kernel_fault+0x200/0x200
Dec  2 12:34:35 10.243.0.116 [ 1780.626108]  ? 0xffffffffc0b38099
Dec  2 12:34:35 10.243.0.116 [ 1780.626365]  ? do_user_addr_fault+0xa30/0xa30
Dec  2 12:34:35 10.243.0.116 [ 1780.626673]  ? do_user_addr_fault+0x686/0xa30
Dec  2 12:34:35 10.243.0.116 [ 1780.627051]  ? page_fault_oops+0x5/0x3d0
Dec  2 12:34:35 10.243.0.116 [ 1780.627119]  ? exc_page_fault+0x5c/0xe0
Dec  2 12:34:35 10.243.0.116 [ 1780.627419]  do_user_addr_fault+0x4af/0xa30
Dec  2 12:34:35 10.243.0.116 [ 1780.627493]  ? rcu_read_lock_held_common+0x5/0x70
Dec  2 12:34:35 10.243.0.116 [ 1780.627641]  ? rcu_read_lock_held_common+0x2a/0x70
Dec  2 12:34:35 10.243.0.116 [ 1780.628139]  ? trace_page_fault_user+0x130/0x130
Dec  2 12:34:35 10.243.0.116 [ 1780.628307]  ? fault_in_kernel_space+0x5/0x40
Dec  2 12:34:35 10.243.0.116 [ 1780.628999]  exc_page_fault+0x71/0xe0
Dec  2 12:34:35 10.243.0.116 [ 1780.629231]  asm_exc_page_fault+0x27/0x30
Dec  2 12:34:35 10.243.0.116 [ 1780.629268] RIP: 0010:trace_event_raw_event_evl_sleep_on+0xc3/0x380
Dec  2 12:34:35 10.243.0.116 [ 1780.629303] Code: 80 0f 85 93 02 00 00 41 f6 c7 40 0f 85 10 02 00 00 41 f7 c7 00 02 00 00 0f 85 8f 02 00 00 4d 8d 7d 70 4c 89 ff e8 6d e9 1e 00 <49> 8b 7d 70 48 85 ff 0f 84 2d 02 00 00 e8 ab b4 58 00 8d 50 01 83
Dec  2 12:34:35 10.243.0.116 [ 1780.629322] RSP: 0018:ffff888137737aa0 EFLAGS: 00010082
Dec  2 12:34:35 10.243.0.116 [ 1780.629352] RAX: 0000000000000001 RBX: 1ffff11026ee6f5a RCX: ffffffff8d1acefc
Dec  2 12:34:35 10.243.0.116 [ 1780.629369] RDX: 0000000000000003 RSI: dffffc0000000000 RDI: ffffffff8f2acb40
Dec  2 12:34:35 10.243.0.116 [ 1780.629386] RBP: ffff888137737b78 R08: fffffbfff1e55969 R09: fffffbfff1e55969
Dec  2 12:34:35 10.243.0.116 [ 1780.629404] R10: ffffffff8f2acb43 R11: fffffbfff1e55968 R12: ffff888100195000
Dec  2 12:34:35 10.243.0.116 [ 1780.629421] R13: 0000000000000000 R14: ffffffff8f219d40 R15: 0000000000000070
Dec  2 12:34:35 10.243.0.116 [ 1780.630165]  ? do_raw_spin_unlock+0x9c/0x140
Dec  2 12:34:35 10.243.0.116 [ 1780.631094]  ? trace_event_raw_event_evl_init_thread+0x330/0x330
Dec  2 12:34:36 10.243.0.116 [ 1780.631268]  ? 0xffffffffc0b38099
Dec  2 12:34:36 10.243.0.116 [ 1780.631322]  ? evl_delay+0x48/0xf0
Dec  2 12:34:36 10.243.0.116 [ 1780.631354]  ? evl_sleep_on_locked+0x730/0x730
Dec  2 12:34:36 10.243.0.116 [ 1780.631645]  ? __kasan_check_write+0x14/0x20
Dec  2 12:34:36 10.243.0.116 [ 1780.632141]  evl_sleep_on_locked+0x629/0x730
Dec  2 12:34:36 10.243.0.116 [ 1780.632210]  ? evl_sleep_on_locked+0x5/0x730
Dec  2 12:34:36 10.243.0.116 [ 1780.632776]  evl_sleep_on+0x7b/0xb0
Dec  2 12:34:36 10.243.0.116 [ 1780.633233]  evl_delay+0x48/0xf0
Dec  2 12:34:36 10.243.0.116 [ 1780.633528]  clock_oob_ioctl+0x182/0x2d0
Dec  2 12:34:36 10.243.0.116 [ 1780.633912]  ? clock_common_ioctl+0x5e0/0x5e0
Dec  2 12:34:36 10.243.0.116 [ 1780.634157]  ? clock_oob_ioctl+0x5/0x2d0
Dec  2 12:34:36 10.243.0.116 [ 1780.634659]  EVL_ioctl+0x86/0x100
Dec  2 12:34:36 10.243.0.116 [ 1780.635150]  do_oob_syscall+0x5b1/0x6d0
Dec  2 12:34:36 10.243.0.116 [ 1780.635353]  ? prepare_for_signal+0x180/0x180
Dec  2 12:34:36 10.243.0.116 [ 1780.635642]  ? do_oob_syscall+0x5/0x6d0
Dec  2 12:34:36 10.243.0.116 [ 1780.635789]  ? handle_pipelined_syscall+0x840/0x840
Dec  2 12:34:36 10.243.0.116 [ 1780.636237]  handle_oob_syscall+0x18e/0x230
Dec  2 12:34:36 10.243.0.116 [ 1780.636462]  ? handle_pipelined_syscall+0x840/0x840
Dec  2 12:34:36 10.243.0.116 [ 1780.637093]  ? handle_oob_syscall+0x5/0x230
Dec  2 12:34:36 10.243.0.116 [ 1780.637511]  pipeline_syscall+0xac/0x1c0
Dec  2 12:34:36 10.243.0.116 [ 1780.638085]  syscall_enter_from_user_mode+0x5f/0x130
Dec  2 12:34:36 10.243.0.116 [ 1780.638255]  do_syscall_64+0x1d/0xa0
Dec  2 12:34:36 10.243.0.116 [ 1780.638358]  entry_SYSCALL_64_after_hwframe+0x61/0xcb
Dec  2 12:34:36 10.243.0.116 [ 1780.638393] RIP: 0033:0x7f302d4b1b9b
Dec  2 12:34:36 10.243.0.116 [ 1780.638466] Code: Unable to access opcode bytes at RIP 0x7f302d4b1b71.
Dec  2 12:34:36 10.243.0.116 [ 1780.638501] RSP: 002b:00007f3013fef5a0 EFLAGS: 00000246 ORIG_RAX: 000000000000009d
Dec  2 12:34:36 10.243.0.116 [ 1780.638574] RAX: ffffffffffffffda RBX: 00007f3013fef630 RCX: 00007f302d4b1b9b
Dec  2 12:34:36 10.243.0.116 [ 1780.638612] RDX: 0000000040106300 RSI: 0000000000000004 RDI: 0000000010000002
Dec  2 12:34:36 10.243.0.116 [ 1780.638649] RBP: 0000000040106300 R08: 0000000000000000 R09: 00007ffd9df03080
Dec  2 12:34:36 10.243.0.116 [ 1780.638687] R10: 00007f3013fef630 R11: 0000000000000246 R12: 0000000000000004
Dec  2 12:34:36 10.243.0.116 [ 1780.638724] R13: 0000000001001000 R14: 0000000000000000 R15: 00007f3013fff700
Dec  2 12:34:36 10.243.0.116 [ 1780.639897]  </TASK>
Dec  2 12:34:36 10.243.0.116 [ 1780.639940] INFO: NMI handler (nmi_cpu_backtrace_handler) took too long to run: 19.307 msecs
Dec  2 12:34:36 10.243.0.116 [ 1780.640671] rcu: rcu_sched kthread timer wakeup didn't happen for 55940 jiffies! g126833 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x402
Dec  2 12:34:36 10.243.0.116 [ 1782.865200] rcu: #011Possible timer handling issue on cpu=6 timer-softirq=22974
Dec  2 12:34:36 10.243.0.116 [ 1782.886720] rcu: rcu_sched kthread starved for 58189 jiffies! g126833 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x402 ->cpu=6
Dec  2 12:34:36 10.243.0.116 [ 1782.908748] rcu: #011Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior.
Dec  2 12:34:36 10.243.0.116 [ 1782.930280] rcu: RCU grace-period kthread stack dump:
Dec  2 12:34:36 10.243.0.116 [ 1782.950285] task:rcu_sched       state:I stack:    0 pid:   14 ppid:     2 flags:0x00004000
Dec  2 12:34:36 10.243.0.116 [ 1782.971170] Call Trace:
Dec  2 12:34:36 10.243.0.116 [ 1782.991024]  <TASK>
Dec  2 12:34:36 10.243.0.116 [ 1783.010534]  __schedule+0x6e4/0x1160
Dec  2 12:34:36 10.243.0.116 [ 1783.030474]  ? __sched_text_start+0x8/0x8
Dec  2 12:34:36 10.243.0.116 [ 1783.050180]  ? __inband_irq_enable+0x4f/0x80
Dec  2 12:34:36 10.243.0.116 [ 1783.070137]  schedule+0xd8/0x190
Dec  2 12:34:36 10.243.0.116 [ 1783.089281]  schedule_timeout+0x2fc/0x530
Dec  2 12:34:36 10.243.0.116 [ 1783.108541]  ? function_trace_call+0x153/0x1d0
Dec  2 12:34:36 10.243.0.116 [ 1783.127571]  ? usleep_range_state+0xf0/0xf0
Dec  2 12:34:36 10.243.0.116 [ 1783.146902]  ? 0xffffffffc0b38099
Dec  2 12:34:36 10.243.0.116 [ 1783.166626]  ? __kasan_check_read+0x11/0x20
Dec  2 12:34:36 10.243.0.116 [ 1783.186425]  ? __bpf_trace_tick_stop+0x20/0x20
Dec  2 12:34:36 10.243.0.116 [ 1783.206166]  ? schedule_timeout+0x5/0x530
Dec  2 12:34:37 10.243.0.116 [ 1783.225381]  ? prepare_to_swait_event+0x72/0x180
Dec  2 12:34:37 10.243.0.116 [ 1783.245285]  rcu_gp_fqs_loop+0x150/0x510
Dec  2 12:34:37 10.243.0.116 [ 1783.264887]  ? force_qs_rnp+0x390/0x390
Dec  2 12:34:37 10.243.0.116 [ 1783.284392]  ? rcu_gp_fqs_loop+0x5/0x510
Dec  2 12:34:37 10.243.0.116 [ 1783.304072]  ? rcu_gp_init+0x49d/0x890
Dec  2 12:34:37 10.243.0.116 [ 1783.323572]  rcu_gp_kthread+0x15c/0x240
Dec  2 12:34:37 10.243.0.116 [ 1783.343293]  ? rcu_gp_fqs_loop+0x510/0x510
Dec  2 12:34:37 10.243.0.116 [ 1783.362926]  ? __kasan_check_read+0x11/0x20
Dec  2 12:34:37 10.243.0.116 [ 1783.382419]  ? __kthread_parkme+0x9d/0xc0
Dec  2 12:34:37 10.243.0.116 [ 1783.401964]  ? rcu_gp_fqs_loop+0x510/0x510
Dec  2 12:34:37 10.243.0.116 [ 1783.421204]  kthread+0x1f3/0x220
Dec  2 12:34:37 10.243.0.116 [ 1783.440649]  ? set_kthread_struct+0x90/0x90
Dec  2 12:34:37 10.243.0.116 [ 1783.459969]  ret_from_fork+0x22/0x30
Dec  2 12:34:37 10.243.0.116 [ 1783.480016]  </TASK>
Dec  2 12:34:37 10.243.0.116 [ 1783.498908] rcu: Stack dump where RCU GP kthread last ran:
Dec  2 12:34:37 10.243.0.116 [ 1783.518605] Sending NMI from CPU 2 to CPUs 6:
Dec  2 12:34:37 10.243.0.116 [ 1783.538329] NMI backtrace for cpu 6
Dec  2 12:34:37 10.243.0.116 [ 1783.538375] CPU: 6 PID: 3580 Comm: evl-ps Tainted: G    B   W    L    5.15.77evl-g5e681f841fc3-dirty #3
Dec  2 12:34:37 10.243.0.116 [ 1783.538418] Hardware name: Supermicro Super Server/X10SDV-6C+-TLN4F, BIOS 2.1 11/22/2019
Dec  2 12:34:37 10.243.0.116 [ 1783.538454] IRQ stage: Linux
Dec  2 12:34:37 10.243.0.116 [ 1783.538500] RIP: 0010:__asan_load4+0x21/0x90
Dec  2 12:34:37 10.243.0.116 [ 1783.538570] Code: 2e 0f 1f 84 00 00 00 00 00 55 48 89 e5 48 8b 4d 08 48 83 ff fb 77 53 48 b8 ff ff ff ff ff 7f ff ff 48 39 c7 76 44 48 8d 47 03 <48> 89 c2 83 e2 07 48 83 fa 02 76 1c 48 be 00 00 00 00 00 fc ff df
Dec  2 12:34:37 10.243.0.116 [ 1783.538611] RSP: 0018:ffff888137877890 EFLAGS: 00000012
Dec  2 12:34:37 10.243.0.116 [ 1783.538679] RAX: ffff888101aae003 RBX: ffff888101aae000 RCX: ffffffff8d1ac05d
Dec  2 12:34:37 10.243.0.116 [ 1783.538718] RDX: 0000000000000003 RSI: dffffc0000000000 RDI: ffff888101aae000
Dec  2 12:34:37 10.243.0.116 [ 1783.538756] RBP: ffff888137877890 R08: ffffed1020355c01 R09: ffffed1020355c01
Dec  2 12:34:37 10.243.0.116 [ 1783.538795] R10: ffff888101aae003 R11: ffffed1020355c00 R12: 1ffff11026f0ef17
Dec  2 12:34:37 10.243.0.116 [ 1783.538836] R13: ffff88841e56bb00 R14: ffff888137877958 R15: 0000000000000000
Dec  2 12:34:37 10.243.0.116 [ 1783.538898] FS:  00007fb41f68c740(0000) GS:ffff88841e500000(0000) knlGS:0000000000000000
Dec  2 12:34:37 10.243.0.116 [ 1783.538938] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Dec  2 12:34:37 10.243.0.116 [ 1783.538976] CR2: 00007fb41f126c50 CR3: 0000000104d8e002 CR4: 00000000003706e0
Dec  2 12:34:37 10.243.0.116 [ 1783.539014] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Dec  2 12:34:37 10.243.0.116 [ 1783.539051] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Dec  2 12:34:37 10.243.0.116 [ 1783.539090] Call Trace:
Dec  2 12:34:37 10.243.0.116 [ 1783.539134]  <TASK>
Dec  2 12:34:37 10.243.0.116 [ 1783.539253]  native_queued_spin_lock_slowpath+0x37d/0x4c0
Dec  2 12:34:37 10.243.0.116 [ 1783.539719]  ? .slowpath+0x1d/0x1d
Dec  2 12:34:37 10.243.0.116 [ 1783.540330]  ? do_raw_spin_lock+0xe6/0x1e0
Dec  2 12:34:37 10.243.0.116 [ 1783.540709]  ? native_queued_spin_lock_slowpath+0x5/0x4c0
Dec  2 12:34:37 10.243.0.116 [ 1783.540868]  ? 0xffffffffc0b38099
Dec  2 12:34:37 10.243.0.116 [ 1783.541608]  do_raw_spin_lock+0x1cb/0x1e0
Dec  2 12:34:38 10.243.0.116 [ 1783.541841]  ? spin_dump+0x90/0x90
Dec  2 12:34:38 10.243.0.116 [ 1783.542220]  ? do_raw_spin_lock+0x5/0x1e0
Dec  2 12:34:38 10.243.0.116 [ 1783.543063]  sched_show+0x12d/0x300
Dec  2 12:34:38 10.243.0.116 [ 1783.543225]  ? 0xffffffffc0b38099
Dec  2 12:34:38 10.243.0.116 [ 1783.543733]  ? inband_task_wakeup+0x160/0x160
Dec  2 12:34:38 10.243.0.116 [ 1783.544113]  ? sched_show+0x5/0x300
Dec  2 12:34:38 10.243.0.116 [ 1783.544953]  dev_attr_show+0x3c/0x70
Dec  2 12:34:38 10.243.0.116 [ 1783.545417]  sysfs_kf_seq_show+0x161/0x1f0
Dec  2 12:34:38 10.243.0.116 [ 1783.545574]  ? device_remove_bin_file+0x20/0x20
Dec  2 12:34:38 10.243.0.116 [ 1783.546339]  kernfs_seq_show+0xa4/0xc0
Dec  2 12:34:38 10.243.0.116 [ 1783.546878]  seq_read_iter+0x289/0x740
Dec  2 12:34:38 10.243.0.116 [ 1783.548419]  kernfs_fop_read_iter+0x25a/0x2c0
Dec  2 12:34:38 10.243.0.116 [ 1783.548496]  ? kernfs_fop_read_iter+0x5/0x2c0
Dec  2 12:34:38 10.243.0.116 [ 1783.548799]  ? iov_iter_init+0x7d/0xa0
Dec  2 12:34:38 10.243.0.116 [ 1783.549332]  new_sync_read+0x2ab/0x3e0
Dec  2 12:34:38 10.243.0.116 [ 1783.549872]  ? __x64_sys_llseek+0x240/0x240
Dec  2 12:34:38 10.243.0.116 [ 1783.550176]  ? vfs_read+0x170/0x270
Dec  2 12:34:38 10.243.0.116 [ 1783.550247]  ? __x64_sys_llseek+0x240/0x240
Dec  2 12:34:38 10.243.0.116 [ 1783.551011]  ? selinux_file_permission+0x1cc/0x200
Dec  2 12:34:38 10.243.0.116 [ 1783.552085]  ? new_sync_read+0x5/0x3e0
Dec  2 12:34:38 10.243.0.116 [ 1783.552926]  vfs_read+0x170/0x270
Dec  2 12:34:38 10.243.0.116 [ 1783.553696]  ksys_read+0xcc/0x170
Dec  2 12:34:38 10.243.0.116 [ 1783.554006]  ? vfs_write+0x4b0/0x4b0
Dec  2 12:34:38 10.243.0.116 [ 1783.554384]  ? ksys_read+0x5/0x170
Dec  2 12:34:38 10.243.0.116 [ 1783.555226]  __x64_sys_read+0x43/0x50
Dec  2 12:34:38 10.243.0.116 [ 1783.555612]  do_syscall_64+0x4c/0xa0
Dec  2 12:34:38 10.243.0.116 [ 1783.555845]  entry_SYSCALL_64_after_hwframe+0x61/0xcb
Dec  2 12:34:38 10.243.0.116 [ 1783.555924] RIP: 0033:0x7fb41f1a79a0
Dec  2 12:34:38 10.243.0.116 [ 1783.556043] Code: 0b 31 c0 48 83 c4 08 e9 be fe ff ff 48 8d 3d d7 c3 09 00 e8 42 8c 02 00 66 90 83 3d 1d d6 2d 00 00 75 10 b8 00 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 31 c3 48 83 ec 08 e8 de cc 01 00 48 89 04 24
Dec  2 12:34:38 10.243.0.116 [ 1783.556083] RSP: 002b:00007ffc99d03948 EFLAGS: 00000246 ORIG_RAX: 0000000000000000
Dec  2 12:34:38 10.243.0.116 [ 1783.556156] RAX: ffffffffffffffda RBX: 0000000000b131f0 RCX: 00007fb41f1a79a0
Dec  2 12:34:38 10.243.0.116 [ 1783.556195] RDX: 0000000000001000 RSI: 00007fb41f6a5000 RDI: 0000000000000005
Dec  2 12:34:38 10.243.0.116 [ 1783.556232] RBP: 00007fb41f47c380 R08: ffffffffffffffff R09: 0000000000000000
Dec  2 12:34:38 10.243.0.116 [ 1783.556270] R10: 0000000000000022 R11: 0000000000000246 R12: 00007fb41f47b838
Dec  2 12:34:38 10.243.0.116 [ 1783.556307] R13: 00007fb41f47b838 R14: 0000000000000d70 R15: 00007fb41f47c5a8
Dec  2 12:34:38 10.243.0.116 [ 1783.557962]  </TASK>
Dec  2 12:34:38 10.243.0.116 [ 1783.558036] INFO: NMI handler (nmi_cpu_backtrace_handler) took too long to run: 19.682 msecs
Dec  2 12:34:46 10.243.0.116 [ 1792.447406] watchdog: BUG: soft lockup - CPU#0 stuck for 75s! [TxStartup:3574]
Dec  2 12:34:46 10.243.0.116 [ 1792.483614] Modules linked in: xt_CHECKSUM xt_MASQUERADE tun netconsole ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 xt_conntrack ebtable_nat ebtable_broute ip6table_nat ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_nat iptable_mangle iptable_security iptable_raw nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 libcrc32c ip_set nfnetlink ebtable_filter ebtables rfkill ip6table_filter ip6_tables iptable_filter sunrpc dm_mirror dm_region_hash dm_log dm_mod x86_pkg_temp_thermal intel_powerclamp coretemp gpio_ich iTCO_wdt iTCO_vendor_support kvm_intel mxm_wmi kvm irqbypass crct10dif_pclmul crc32_pclmul ghash_clmulni_intel aesni_intel crypto_simd cryptd rapl intel_cstate pcspkr sg i2c_i801 ftdi_sio mei_me i2c_smbus joydev input_leds intel_pch_thermal mei lpc_ich mfd_core ioatdma wmi ip_tables ext4 mbcache jbd2 ast drm_vram_helper drm_kms_helper syscopyarea ixgbe sysfillrect sd_mod sr_mod sysimgblt fb_sys_fops cdrom t10_pi drm_ttm_helper ttm mdio drm igb ah
Dec  2 12:34:46 10.243.0.116 [ 1792.497034]  libahci libata crc32c_intel ptp pps_core i2c_algo_bit dca fuse
Dec  2 12:34:46 10.243.0.116 [ 1792.840127] irq event stamp: 24564
Dec  2 12:34:46 10.243.0.116 [ 1792.879961] hardirqs last  enabled at (24563): [<ffffffff8d1d44ad>] sync_current_irq_stage+0x45d/0x500
Dec  2 12:34:46 10.243.0.116 [ 1792.921994] hardirqs last disabled at (24564): [<ffffffff8d49923b>] handle_mm_fault+0x48b/0x520
Dec  2 12:34:46 10.243.0.116 [ 1792.964153] softirqs last  enabled at (0): [<ffffffff8d0ca587>] copy_process+0x1307/0x3770
Dec  2 12:34:46 10.243.0.116 [ 1793.006557] softirqs last disabled at (0): [<0000000000000000>] 0x0
Dec  2 12:34:46 10.243.0.116 [ 1793.048711] CPU: 0 PID: 3574 Comm: TxStartup Tainted: G    B   W    L    5.15.77evl-g5e681f841fc3-dirty #3
Dec  2 12:34:46 10.243.0.116 [ 1793.092310] Hardware name: Supermicro Super Server/X10SDV-6C+-TLN4F, BIOS 2.1 11/22/2019
Dec  2 12:34:46 10.243.0.116 [ 1793.135761] IRQ stage: Linux
Dec  2 12:34:46 10.243.0.116 [ 1793.177935] RIP: 0010:proxy_set_next_ktime+0x77/0x90
Dec  2 12:34:47 10.243.0.116 [ 1793.220998] Code: 74 37 49 8d 7c 24 30 e8 d7 2d 1d 00 49 8b 44 24 30 4c 89 e7 e8 da 42 0b 01 48 8d 34 03 31 d2 4c 89 ef e8 5c 14 00 00 41 56 9d <31> c0 5b 41 5c 41 5d 41 5e 5d c3 cc cc cc cc e8 25 f5 eb ff eb d9
Dec  2 12:34:47 10.243.0.116 [ 1793.312708] RSP: 0018:ffff88841e209e80 EFLAGS: 00000283 ORIG_RAX: 0000000000000000
Dec  2 12:34:47 10.243.0.116 [ 1793.360453] RAX: 0000000000000000 RBX: 0000000000000000 RCX: 0000000000000000
Dec  2 12:34:47 10.243.0.116 [ 1793.408216] RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000000
Dec  2 12:34:47 10.243.0.116 [ 1793.455977] RBP: ffff88841e209ea0 R08: 0000000000000000 R09: 0000000000000000
Dec  2 12:34:47 10.243.0.116 [ 1793.503877] R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000000
Dec  2 12:34:47 10.243.0.116 [ 1793.551923] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000
Dec  2 12:34:47 10.243.0.116 [ 1793.600030] FS:  00007f301ae2e700(0000) GS:ffff88841e200000(0000) knlGS:0000000000000000
Dec  2 12:34:47 10.243.0.116 [ 1793.648722] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Dec  2 12:34:47 10.243.0.116 [ 1793.697094] CR2: 00007f1e4033b660 CR3: 0000000137d08006 CR4: 00000000003706f0
Dec  2 12:34:47 10.243.0.116 [ 1793.746376] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Dec  2 12:34:47 10.243.0.116 [ 1793.796333] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Dec  2 12:34:47 10.243.0.116 [ 1793.846653] Call Trace:
Dec  2 12:34:47 10.243.0.116 [ 1793.895855]  <IRQ>
Dec  2 12:34:47 10.243.0.116 [ 1793.945182]  ? __hrtimer_run_queues+0x5/0x780
Dec  2 12:34:47 10.243.0.116 [ 1793.995801]  ? hrtimer_interrupt+0x1aa/0x360
Dec  2 12:34:47 10.243.0.116 [ 1794.046133]  ? hrtimer_interrupt+0x5/0x360
Dec  2 12:34:47 10.243.0.116 [ 1794.096597]  ? proxy_irq_handler+0x37/0x50
Dec  2 12:34:47 10.243.0.116 [ 1794.144668]  ? handle_synthetic_irq+0xec/0x320
Dec  2 12:34:48 10.243.0.116 [ 1794.190960]  ? do_irq_inband+0x2f/0x40
Dec  2 12:34:48 10.243.0.116 [ 1794.236462]  ? arch_do_IRQ_pipelined+0xcd/0x5b0
Dec  2 12:34:48 10.243.0.116 [ 1794.281795]  </IRQ>
Dec  2 12:34:48 10.243.0.116 [ 1794.326004]  <TASK>
Dec  2 12:34:48 10.243.0.116 [ 1794.370207]  ? sync_current_irq_stage+0x2e3/0x500
Dec  2 12:34:48 10.243.0.116 [ 1794.415541]  ? sync_irq_stage+0x1af/0x1e0
Dec  2 12:34:48 10.243.0.116 [ 1794.459875]  ? synchronize_pipeline+0x8b/0xe0
Dec  2 12:34:48 10.243.0.116 [ 1794.503881]  ? handle_irq_pipelined_finish+0xf6/0x240
Dec  2 12:34:48 10.243.0.116 [ 1794.547660]  ? arch_pipeline_entry+0x81/0x110
Dec  2 12:34:48 10.243.0.116 [ 1794.590797]  ? sysvec_apic_timer_interrupt+0xe/0x20
Dec  2 12:34:48 10.243.0.116 [ 1794.633205]  ? asm_sysvec_apic_timer_interrupt+0x1b/0x20
Dec  2 12:34:48 10.243.0.116 [ 1794.676183]  ? smp_call_function_many_cond+0x1c7/0x4e0
Dec  2 12:34:48 10.243.0.116 [ 1794.718209]  ? smp_call_function_many_cond+0x1bf/0x4e0
Dec  2 12:34:48 10.243.0.116 [ 1794.759572]  ? smp_call_function_many_cond+0x1c7/0x4e0
Dec  2 12:34:48 10.243.0.116 [ 1794.800343]  ? leave_mm+0x40/0x40
Dec  2 12:34:48 10.243.0.116 [ 1794.840123]  ? x86_configure_nx+0x70/0x70
Dec  2 12:34:48 10.243.0.116 [ 1794.879829]  ? on_each_cpu_cond_mask+0x29/0x40
Dec  2 12:34:48 10.243.0.116 [ 1794.918982]  ? native_flush_tlb_multi+0xbe/0x250
Dec  2 12:34:48 10.243.0.116 [ 1794.957878]  ? flush_tlb_mm_range+0x136/0x1f0
Dec  2 12:34:48 10.243.0.116 [ 1794.996473]  ? change_protection_range+0x8bd/0xea0
Dec  2 12:34:48 10.243.0.116 [ 1795.036759]  ? prot_none_pte_entry+0x70/0x70
Dec  2 12:34:48 10.243.0.116 [ 1795.074648]  ? 0xffffffffc0b38099
Dec  2 12:34:48 10.243.0.116 [ 1795.112223]  ? change_protection_range+0x5/0xea0
Dec  2 12:34:48 10.243.0.116 [ 1795.149818]  ? change_protection+0x50/0x90
Dec  2 12:34:48 10.243.0.116 [ 1795.186491]  ? mprotect_fixup+0x2c5/0x490
Dec  2 12:34:49 10.243.0.116 [ 1795.223324]  ? change_protection+0x90/0x90
Dec  2 12:34:49 10.243.0.116 [ 1795.259252]  ? mprotect_fixup+0x5/0x490
Dec  2 12:34:49 10.243.0.116 [ 1795.295565]  ? do_mprotect_pkey+0x32a/0x590
Dec  2 12:34:49 10.243.0.116 [ 1795.330273]  ? mprotect_fixup+0x490/0x490
Dec  2 12:34:49 10.243.0.116 [ 1795.362504]  ? do_mprotect_pkey+0x5/0x590
Dec  2 12:34:49 10.243.0.116 [ 1795.395056]  ? __x64_sys_mprotect+0x48/0x60
Dec  2 12:34:49 10.243.0.116 [ 1795.620916]  ? do_syscall_64+0x4c/0xa0
Dec  2 12:34:49 10.243.0.116 [ 1795.652479]  ? entry_SYSCALL_64_after_hwframe+0x61/0xcb
Dec  2 12:34:49 10.243.0.116 [ 1795.685796]  </TASK>
Dec  2 12:35:14 10.243.0.116 [ 1820.447535] watchdog: BUG: soft lockup - CPU#0 stuck for 101s! [TxStartup:3574]
Dec  2 12:35:14 10.243.0.116 [ 1820.480297] Modules linked in: xt_CHECKSUM xt_MASQUERADE tun netconsole ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 xt_conntrack ebtable_nat ebtable_broute ip6table_nat ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_nat iptable_mangle iptable_security iptable_raw nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 libcrc32c ip_set nfnetlink ebtable_filter ebtables rfkill ip6table_filter ip6_tables iptable_filter sunrpc dm_mirror dm_region_hash dm_log dm_mod x86_pkg_temp_thermal intel_powerclamp coretemp gpio_ich iTCO_wdt iTCO_vendor_support kvm_intel mxm_wmi kvm irqbypass crct10dif_pclmul crc32_pclmul ghash_clmulni_intel aesni_intel crypto_simd cryptd rapl intel_cstate pcspkr sg i2c_i801 ftdi_sio mei_me i2c_smbus joydev input_leds intel_pch_thermal mei lpc_ich mfd_core ioatdma wmi ip_tables ext4 mbcache jbd2 ast drm_vram_helper drm_kms_helper syscopyarea ixgbe sysfillrect sd_mod sr_mod sysimgblt fb_sys_fops cdrom t10_pi drm_ttm_helper ttm mdio drm igb ah
Dec  2 12:35:14 10.243.0.116 [ 1820.493722]  libahci libata crc32c_intel ptp pps_core i2c_algo_bit dca fuse
Dec  2 12:35:14 10.243.0.116 [ 1820.790286] irq event stamp: 24564
Dec  2 12:35:14 10.243.0.116 [ 1820.824406] hardirqs last  enabled at (24563): [<ffffffff8d1d44ad>] sync_current_irq_stage+0x45d/0x500
Dec  2 12:35:14 10.243.0.116 [ 1820.860567] hardirqs last disabled at (24564): [<ffffffff8d49923b>] handle_mm_fault+0x48b/0x520
Dec  2 12:35:14 10.243.0.116 [ 1820.896758] softirqs last  enabled at (0): [<ffffffff8d0ca587>] copy_process+0x1307/0x3770
Dec  2 12:35:14 10.243.0.116 [ 1820.933087] softirqs last disabled at (0): [<0000000000000000>] 0x0
Dec  2 12:35:14 10.243.0.116 [ 1820.968972] CPU: 0 PID: 3574 Comm: TxStartup Tainted: G    B   W    L    5.15.77evl-g5e681f841fc3-dirty #3
Dec  2 12:35:14 10.243.0.116 [ 1821.006251] Hardware name: Supermicro Super Server/X10SDV-6C+-TLN4F, BIOS 2.1 11/22/2019
Dec  2 12:35:14 10.243.0.116 [ 1821.043519] IRQ stage: Linux
Dec  2 12:35:14 10.243.0.116 [ 1821.079616] RIP: 0010:proxy_set_next_ktime+0x77/0x90
Dec  2 12:35:14 10.243.0.116 [ 1821.116519] Code: 74 37 49 8d 7c 24 30 e8 d7 2d 1d 00 49 8b 44 24 30 4c 89 e7 e8 da 42 0b 01 48 8d 34 03 31 d2 4c 89 ef e8 5c 14 00 00 41 56 9d <31> c0 5b 41 5c 41 5d 41 5e 5d c3 cc cc cc cc e8 25 f5 eb ff eb d9
Dec  2 12:35:15 10.243.0.116 [ 1821.196115] RSP: 0018:ffff88841e209e80 EFLAGS: 00000293 ORIG_RAX: 0000000000000000
Dec  2 12:35:15 10.243.0.116 [ 1821.237742] RAX: 0000000000000000 RBX: 0000000000000000 RCX: 0000000000000000
Dec  2 12:35:15 10.243.0.116 [ 1821.279413] RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000000
Dec  2 12:35:15 10.243.0.116 [ 1821.320981] RBP: ffff88841e209ea0 R08: 0000000000000000 R09: 0000000000000000
Dec  2 12:35:15 10.243.0.116 [ 1821.362596] R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000000
Dec  2 12:35:15 10.243.0.116 [ 1821.404505] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000
Dec  2 12:35:15 10.243.0.116 [ 1821.446592] FS:  00007f301ae2e700(0000) GS:ffff88841e200000(0000) knlGS:0000000000000000
Dec  2 12:35:15 10.243.0.116 [ 1821.489477] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Dec  2 12:35:15 10.243.0.116 [ 1821.532090] CR2: 00007f1e4033b660 CR3: 0000000137d08006 CR4: 00000000003706f0
Dec  2 12:35:15 10.243.0.116 [ 1821.575248] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Dec  2 12:35:15 10.243.0.116 [ 1821.619017] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Dec  2 12:35:15 10.243.0.116 [ 1821.662935] Call Trace:
Dec  2 12:35:15 10.243.0.116 [ 1821.705794]  <IRQ>
Dec  2 12:35:15 10.243.0.116 [ 1821.748642]  ? __hrtimer_run_queues+0x5/0x780
Dec  2 12:35:15 10.243.0.116 [ 1821.792687]  ? hrtimer_interrupt+0x1aa/0x360
Dec  2 12:35:15 10.243.0.116 [ 1821.836504]  ? hrtimer_interrupt+0x5/0x360
Dec  2 12:35:15 10.243.0.116 [ 1821.880488]  ? proxy_irq_handler+0x37/0x50
Dec  2 12:35:15 10.243.0.116 [ 1821.923848]  ? handle_synthetic_irq+0xec/0x320
Dec  2 12:35:15 10.243.0.116 [ 1821.967496]  ? do_irq_inband+0x2f/0x40
Dec  2 12:35:15 10.243.0.116 [ 1822.010675]  ? arch_do_IRQ_pipelined+0xcd/0x5b0
Dec  2 12:35:15 10.243.0.116 [ 1822.054021]  </IRQ>
Dec  2 12:35:15 10.243.0.116 [ 1822.096506]  <TASK>
Dec  2 12:35:15 10.243.0.116 [ 1822.139286]  ? sync_current_irq_stage+0x2e3/0x500
Dec  2 12:35:15 10.243.0.116 [ 1822.183504]  ? sync_irq_stage+0x1af/0x1e0
Dec  2 12:35:16 10.243.0.116 [ 1822.226784]  ? synchronize_pipeline+0x8b/0xe0
Dec  2 12:35:16 10.243.0.116 [ 1822.270115]  ? handle_irq_pipelined_finish+0xf6/0x240
Dec  2 12:35:16 10.243.0.116 [ 1822.313767]  ? arch_pipeline_entry+0x81/0x110
Dec  2 12:35:16 10.243.0.116 [ 1822.356544]  ? sysvec_apic_timer_interrupt+0xe/0x20
Dec  2 12:35:16 10.243.0.116 [ 1822.398597]  ? asm_sysvec_apic_timer_interrupt+0x1b/0x20
Dec  2 12:35:16 10.243.0.116 [ 1822.441321]  ? smp_call_function_many_cond+0x1c7/0x4e0
Dec  2 12:35:16 10.243.0.116 [ 1822.483104]  ? smp_call_function_many_cond+0x1c2/0x4e0
Dec  2 12:35:16 10.243.0.116 [ 1822.524208]  ? smp_call_function_many_cond+0x1c7/0x4e0
Dec  2 12:35:16 10.243.0.116 [ 1822.564763]  ? leave_mm+0x40/0x40
Dec  2 12:35:16 10.243.0.116 [ 1822.604524]  ? x86_configure_nx+0x70/0x70
Dec  2 12:35:16 10.243.0.116 [ 1822.644094]  ? on_each_cpu_cond_mask+0x29/0x40
Dec  2 12:35:16 10.243.0.116 [ 1822.683101]  ? native_flush_tlb_multi+0xbe/0x250
Dec  2 12:35:16 10.243.0.116 [ 1822.721837]  ? flush_tlb_mm_range+0x136/0x1f0
Dec  2 12:35:16 10.243.0.116 [ 1822.760224]  ? change_protection_range+0x8bd/0xea0
Dec  2 12:35:16 10.243.0.116 [ 1822.800334]  ? prot_none_pte_entry+0x70/0x70
Dec  2 12:35:16 10.243.0.116 [ 1822.837959]  ? 0xffffffffc0b38099
Dec  2 12:35:16 10.243.0.116 [ 1822.875197]  ? change_protection_range+0x5/0xea0
Dec  2 12:35:16 10.243.0.116 [ 1822.912637]  ? change_protection+0x50/0x90
Dec  2 12:35:16 10.243.0.116 [ 1822.949153]  ? mprotect_fixup+0x2c5/0x490
Dec  2 12:35:16 10.243.0.116 [ 1822.985831]  ? change_protection+0x90/0x90
Dec  2 12:35:16 10.243.0.116 [ 1823.021595]  ? mprotect_fixup+0x5/0x490
Dec  2 12:35:16 10.243.0.116 [ 1823.057756]  ? do_mprotect_pkey+0x32a/0x590
Dec  2 12:35:16 10.243.0.116 [ 1823.092300]  ? mprotect_fixup+0x490/0x490
Dec  2 12:35:16 10.243.0.116 [ 1823.124328]  ? do_mprotect_pkey+0x5/0x590
Dec  2 12:35:16 10.243.0.116 [ 1823.156668]  ? __x64_sys_mprotect+0x48/0x60
Dec  2 12:35:16 10.243.0.116 [ 1823.188252]  ? do_syscall_64+0x4c/0xa0
Dec  2 12:35:17 10.243.0.116 [ 1823.219567]  ? entry_SYSCALL_64_after_hwframe+0x61/0xcb
Dec  2 12:35:17 10.243.0.116 [ 1823.252645]  </TASK>
Dec  2 12:35:42 10.243.0.116 [ 1848.447657] watchdog: BUG: soft lockup - CPU#0 stuck for 127s! [TxStartup:3574]
Dec  2 12:35:42 10.243.0.116 [ 1848.480187] Modules linked in: xt_CHECKSUM xt_MASQUERADE tun netconsole ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 xt_conntrack ebtable_nat ebtable_broute ip6table_nat ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_nat iptable_mangle iptable_security iptable_raw nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 libcrc32c ip_set nfnetlink ebtable_filter ebtables rfkill ip6table_filter ip6_tables iptable_filter sunrpc dm_mirror dm_region_hash dm_log dm_mod x86_pkg_temp_thermal intel_powerclamp coretemp gpio_ich iTCO_wdt iTCO_vendor_support kvm_intel mxm_wmi kvm irqbypass crct10dif_pclmul crc32_pclmul ghash_clmulni_intel aesni_intel crypto_simd cryptd rapl intel_cstate pcspkr sg i2c_i801 ftdi_sio mei_me i2c_smbus joydev input_leds intel_pch_thermal mei lpc_ich mfd_core ioatdma wmi ip_tables ext4 mbcache jbd2 ast drm_vram_helper drm_kms_helper syscopyarea ixgbe sysfillrect sd_mod sr_mod sysimgblt fb_sys_fops cdrom t10_pi drm_ttm_helper ttm mdio drm igb ah
Dec  2 12:35:42 10.243.0.116 [ 1848.493609]  libahci libata crc32c_intel ptp pps_core i2c_algo_bit dca fuse
Dec  2 12:35:42 10.243.0.116 [ 1848.788312] irq event stamp: 24564
Dec  2 12:35:42 10.243.0.116 [ 1848.822354] hardirqs last  enabled at (24563): [<ffffffff8d1d44ad>] sync_current_irq_stage+0x45d/0x500
Dec  2 12:35:42 10.243.0.116 [ 1848.858693] hardirqs last disabled at (24564): [<ffffffff8d49923b>] handle_mm_fault+0x48b/0x520
Dec  2 12:35:42 10.243.0.116 [ 1848.894933] softirqs last  enabled at (0): [<ffffffff8d0ca587>] copy_process+0x1307/0x3770
Dec  2 12:35:42 10.243.0.116 [ 1848.931344] softirqs last disabled at (0): [<0000000000000000>] 0x0
Dec  2 12:35:42 10.243.0.116 [ 1848.967316] CPU: 0 PID: 3574 Comm: TxStartup Tainted: G    B   W    L    5.15.77evl-g5e681f841fc3-dirty #3
Dec  2 12:35:42 10.243.0.116 [ 1849.004731] Hardware name: Supermicro Super Server/X10SDV-6C+-TLN4F, BIOS 2.1 11/22/2019
Dec  2 12:35:42 10.243.0.116 [ 1849.042217] IRQ stage: Linux
Dec  2 12:35:42 10.243.0.116 [ 1849.078498] RIP: 0010:proxy_set_next_ktime+0x77/0x90
Dec  2 12:35:42 10.243.0.116 [ 1849.115543] Code: 74 37 49 8d 7c 24 30 e8 d7 2d 1d 00 49 8b 44 24 30 4c 89 e7 e8 da 42 0b 01 48 8d 34 03 31 d2 4c 89 ef e8 5c 14 00 00 41 56 9d <31> c0 5b 41 5c 41 5d 41 5e 5d c3 cc cc cc cc e8 25 f5 eb ff eb d9
Dec  2 12:35:43 10.243.0.116 [ 1849.195313] RSP: 0018:ffff88841e209e80 EFLAGS: 00000293 ORIG_RAX: 0000000000000000
Dec  2 12:35:43 10.243.0.116 [ 1849.237036] RAX: 0000000000000000 RBX: 0000000000000000 RCX: 0000000000000000
Dec  2 12:35:43 10.243.0.116 [ 1849.278803] RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000000
Dec  2 12:35:43 10.243.0.116 [ 1849.320405] RBP: ffff88841e209ea0 R08: 0000000000000000 R09: 0000000000000000
Dec  2 12:35:43 10.243.0.116 [ 1849.362117] R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000000
Dec  2 12:35:43 10.243.0.116 [ 1849.404115] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000
Dec  2 12:35:43 10.243.0.116 [ 1849.446248] FS:  00007f301ae2e700(0000) GS:ffff88841e200000(0000) knlGS:0000000000000000
Dec  2 12:35:43 10.243.0.116 [ 1849.489006] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Dec  2 12:35:43 10.243.0.116 [ 1849.531417] CR2: 00007f1e4033b660 CR3: 0000000137d08006 CR4: 00000000003706f0
Dec  2 12:35:43 10.243.0.116 [ 1849.574502] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Dec  2 12:35:43 10.243.0.116 [ 1849.618197] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Dec  2 12:35:43 10.243.0.116 [ 1849.662044] Call Trace:
Dec  2 12:35:43 10.243.0.116 [ 1849.704888]  <IRQ>
Dec  2 12:35:43 10.243.0.116 [ 1849.747786]  ? __hrtimer_run_queues+0x5/0x780
Dec  2 12:35:43 10.243.0.116 [ 1849.791927]  ? hrtimer_interrupt+0x1aa/0x360
Dec  2 12:35:43 10.243.0.116 [ 1849.835800]  ? hrtimer_interrupt+0x5/0x360
Dec  2 12:35:43 10.243.0.116 [ 1849.879840]  ? proxy_irq_handler+0x37/0x50
Dec  2 12:35:43 10.243.0.116 [ 1849.923446]  ? handle_synthetic_irq+0xec/0x320
Dec  2 12:35:43 10.243.0.116 [ 1849.967256]  ? do_irq_inband+0x2f/0x40
Dec  2 12:35:43 10.243.0.116 [ 1850.010521]  ? arch_do_IRQ_pipelined+0xcd/0x5b0
Dec  2 12:35:43 10.243.0.116 [ 1850.053959]  </IRQ>
Dec  2 12:35:43 10.243.0.116 [ 1850.096538]  <TASK>
Dec  2 12:35:43 10.243.0.116 [ 1850.139360]  ? sync_current_irq_stage+0x2e3/0x500
Dec  2 12:35:43 10.243.0.116 [ 1850.183625]  ? sync_irq_stage+0x1af/0x1e0
Dec  2 12:35:44 10.243.0.116 [ 1850.226955]  ? synchronize_pipeline+0x8b/0xe0
Dec  2 12:35:44 10.243.0.116 [ 1850.270335]  ? handle_irq_pipelined_finish+0xf6/0x240
Dec  2 12:35:44 10.243.0.116 [ 1850.314022]  ? arch_pipeline_entry+0x81/0x110
Dec  2 12:35:44 10.243.0.116 [ 1850.356788]  ? sysvec_apic_timer_interrupt+0xe/0x20
Dec  2 12:35:44 10.243.0.116 [ 1850.398912]  ? asm_sysvec_apic_timer_interrupt+0x1b/0x20
Dec  2 12:35:44 10.243.0.116 [ 1850.441579]  ? smp_call_function_many_cond+0x1c7/0x4e0
Dec  2 12:35:44 10.243.0.116 [ 1850.483300]  ? __asan_store2+0xa0/0xa0
Dec  2 12:35:44 10.243.0.116 [ 1850.524064]  ? smp_call_function_many_cond+0x1c7/0x4e0
Dec  2 12:35:44 10.243.0.116 [ 1850.564755]  ? leave_mm+0x40/0x40
Dec  2 12:35:44 10.243.0.116 [ 1850.604523]  ? x86_configure_nx+0x70/0x70
Dec  2 12:35:44 10.243.0.116 [ 1850.644230]  ? on_each_cpu_cond_mask+0x29/0x40
Dec  2 12:35:44 10.243.0.116 [ 1850.683376]  ? native_flush_tlb_multi+0xbe/0x250
Dec  2 12:35:44 10.243.0.116 [ 1850.722254]  ? flush_tlb_mm_range+0x136/0x1f0
Dec  2 12:35:44 10.243.0.116 [ 1850.760833]  ? change_protection_range+0x8bd/0xea0
Dec  2 12:35:44 10.243.0.116 [ 1850.801013]  ? prot_none_pte_entry+0x70/0x70
Dec  2 12:35:44 10.243.0.116 [ 1850.838725]  ? 0xffffffffc0b38099
Dec  2 12:35:44 10.243.0.116 [ 1850.876052]  ? change_protection_range+0x5/0xea0
Dec  2 12:35:44 10.243.0.116 [ 1850.913582]  ? change_protection+0x50/0x90
Dec  2 12:35:44 10.243.0.116 [ 1850.950280]  ? mprotect_fixup+0x2c5/0x490
Dec  2 12:35:44 10.243.0.116 [ 1850.987037]  ? change_protection+0x90/0x90
Dec  2 12:35:44 10.243.0.116 [ 1851.022894]  ? mprotect_fixup+0x5/0x490
Dec  2 12:35:44 10.243.0.116 [ 1851.059201]  ? do_mprotect_pkey+0x32a/0x590
Dec  2 12:35:44 10.243.0.116 [ 1851.093946]  ? mprotect_fixup+0x490/0x490
Dec  2 12:35:44 10.243.0.116 [ 1851.126216]  ? do_mprotect_pkey+0x5/0x590
Dec  2 12:35:44 10.243.0.116 [ 1851.158752]  ? __x64_sys_mprotect+0x48/0x60
Dec  2 12:35:44 10.243.0.116 [ 1851.190537]  ? do_syscall_64+0x4c/0xa0
Dec  2 12:35:45 10.243.0.116 [ 1851.222048]  ? entry_SYSCALL_64_after_hwframe+0x61/0xcb
Dec  2 12:35:45 10.243.0.116 [ 1851.255277]  </TASK>
Dec  2 12:36:10 10.243.0.116 [ 1876.447790] watchdog: BUG: soft lockup - CPU#0 stuck for 153s! [TxStartup:3574]
Dec  2 12:36:10 10.243.0.116 [ 1876.480408] Modules linked in: xt_CHECKSUM xt_MASQUERADE tun netconsole ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 xt_conntrack ebtable_nat ebtable_broute ip6table_nat ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_nat iptable_mangle iptable_security iptable_raw nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 libcrc32c ip_set nfnetlink ebtable_filter ebtables rfkill ip6table_filter ip6_tables iptable_filter sunrpc dm_mirror dm_region_hash dm_log dm_mod x86_pkg_temp_thermal intel_powerclamp coretemp gpio_ich iTCO_wdt iTCO_vendor_support kvm_intel mxm_wmi kvm irqbypass crct10dif_pclmul crc32_pclmul ghash_clmulni_intel aesni_intel crypto_simd cryptd rapl intel_cstate pcspkr sg i2c_i801 ftdi_sio mei_me i2c_smbus joydev input_leds intel_pch_thermal mei lpc_ich mfd_core ioatdma wmi ip_tables ext4 mbcache jbd2 ast drm_vram_helper drm_kms_helper syscopyarea ixgbe sysfillrect sd_mod sr_mod sysimgblt fb_sys_fops cdrom t10_pi drm_ttm_helper ttm mdio drm igb ah
Dec  2 12:36:10 10.243.0.116 [ 1876.493824]  libahci libata crc32c_intel ptp pps_core i2c_algo_bit dca fuse
Dec  2 12:36:10 10.243.0.116 [ 1876.789813] irq event stamp: 24564
Dec  2 12:36:10 10.243.0.116 [ 1876.824064] hardirqs last  enabled at (24563): [<ffffffff8d1d44ad>] sync_current_irq_stage+0x45d/0x500
Dec  2 12:36:10 10.243.0.116 [ 1876.860438] hardirqs last disabled at (24564): [<ffffffff8d49923b>] handle_mm_fault+0x48b/0x520
Dec  2 12:36:10 10.243.0.116 [ 1876.896715] softirqs last  enabled at (0): [<ffffffff8d0ca587>] copy_process+0x1307/0x3770
Dec  2 12:36:10 10.243.0.116 [ 1876.933187] softirqs last disabled at (0): [<0000000000000000>] 0x0
Dec  2 12:36:10 10.243.0.116 [ 1876.969200] CPU: 0 PID: 3574 Comm: TxStartup Tainted: G    B   W    L    5.15.77evl-g5e681f841fc3-dirty #3
Dec  2 12:36:10 10.243.0.116 [ 1877.006609] Hardware name: Supermicro Super Server/X10SDV-6C+-TLN4F, BIOS 2.1 11/22/2019
Dec  2 12:36:10 10.243.0.116 [ 1877.044067] IRQ stage: Linux
Dec  2 12:36:10 10.243.0.116 [ 1877.080392] RIP: 0010:proxy_set_next_ktime+0x77/0x90
Dec  2 12:36:10 10.243.0.116 [ 1877.117483] Code: 74 37 49 8d 7c 24 30 e8 d7 2d 1d 00 49 8b 44 24 30 4c 89 e7 e8 da 42 0b 01 48 8d 34 03 31 d2 4c 89 ef e8 5c 14 00 00 41 56 9d <31> c0 5b 41 5c 41 5d 41 5e 5d c3 cc cc cc cc e8 25 f5 eb ff eb d9
Dec  2 12:36:11 10.243.0.116 [ 1877.197397] RSP: 0018:ffff88841e209e80 EFLAGS: 00000283 ORIG_RAX: 0000000000000000
Dec  2 12:36:11 10.243.0.116 [ 1877.239296] RAX: 0000000000000000 RBX: 0000000000000000 RCX: 0000000000000000
Dec  2 12:36:11 10.243.0.116 [ 1877.281160] RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000000
Dec  2 12:36:11 10.243.0.116 [ 1877.322917] RBP: ffff88841e209ea0 R08: 0000000000000000 R09: 0000000000000000
Dec  2 12:36:11 10.243.0.116 [ 1877.364716] R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000000
Dec  2 12:36:11 10.243.0.116 [ 1877.406744] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000
Dec  2 12:36:11 10.243.0.116 [ 1877.448842] FS:  00007f301ae2e700(0000) GS:ffff88841e200000(0000) knlGS:0000000000000000
Dec  2 12:36:11 10.243.0.116 [ 1877.491599] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Dec  2 12:36:11 10.243.0.116 [ 1877.534011] CR2: 00007f1e4033b660 CR3: 0000000137d08006 CR4: 00000000003706f0
Dec  2 12:36:11 10.243.0.116 [ 1877.577106] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Dec  2 12:36:11 10.243.0.116 [ 1877.620814] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Dec  2 12:36:11 10.243.0.116 [ 1877.664673] Call Trace:
Dec  2 12:36:11 10.243.0.116 [ 1877.707534]  <IRQ>
Dec  2 12:36:11 10.243.0.116 [ 1877.750487]  ? __hrtimer_run_queues+0x5/0x780
Dec  2 12:36:11 10.243.0.116 [ 1877.794633]  ? hrtimer_interrupt+0x1aa/0x360
Dec  2 12:36:11 10.243.0.116 [ 1877.838506]  ? hrtimer_interrupt+0x5/0x360
Dec  2 12:36:11 10.243.0.116 [ 1877.882548]  ? proxy_irq_handler+0x37/0x50
Dec  2 12:36:11 10.243.0.116 [ 1877.925967]  ? handle_synthetic_irq+0xec/0x320
Dec  2 12:36:11 10.243.0.116 [ 1877.969674]  ? do_irq_inband+0x2f/0x40
Dec  2 12:36:11 10.243.0.116 [ 1878.012847]  ? arch_do_IRQ_pipelined+0xcd/0x5b0
Dec  2 12:36:11 10.243.0.116 [ 1878.056191]  </IRQ>
Dec  2 12:36:11 10.243.0.116 [ 1878.098683]  <TASK>
Dec  2 12:36:11 10.243.0.116 [ 1878.141537]  ? sync_current_irq_stage+0x2e3/0x500
Dec  2 12:36:11 10.243.0.116 [ 1878.185760]  ? sync_irq_stage+0x1af/0x1e0
Dec  2 12:36:12 10.243.0.116 [ 1878.229045]  ? synchronize_pipeline+0x8b/0xe0
Dec  2 12:36:12 10.243.0.116 [ 1878.272395]  ? handle_irq_pipelined_finish+0xf6/0x240
Dec  2 12:36:12 10.243.0.116 [ 1878.316184]  ? arch_pipeline_entry+0x81/0x110
Dec  2 12:36:12 10.243.0.116 [ 1878.358966]  ? sysvec_apic_timer_interrupt+0xe/0x20
Dec  2 12:36:12 10.243.0.116 [ 1878.401033]  ? asm_sysvec_apic_timer_interrupt+0x1b/0x20
Dec  2 12:36:12 10.243.0.116 [ 1878.443713]  ? smp_call_function_many_cond+0x1c7/0x4e0
Dec  2 12:36:12 10.243.0.116 [ 1878.485444]  ? smp_call_function_many_cond+0x1bf/0x4e0
Dec  2 12:36:12 10.243.0.116 [ 1878.526506]  ? smp_call_function_many_cond+0x1c7/0x4e0
Dec  2 12:36:12 10.243.0.116 [ 1878.566963]  ? leave_mm+0x40/0x40
Dec  2 12:36:12 10.243.0.116 [ 1878.606495]  ? x86_configure_nx+0x70/0x70
Dec  2 12:36:12 10.243.0.116 [ 1878.645961]  ? on_each_cpu_cond_mask+0x29/0x40
Dec  2 12:36:12 10.243.0.116 [ 1878.684826]  ? native_flush_tlb_multi+0xbe/0x250
Dec  2 12:36:12 10.243.0.116 [ 1878.723425]  ? flush_tlb_mm_range+0x136/0x1f0
Dec  2 12:36:12 10.243.0.116 [ 1878.761725]  ? change_protection_range+0x8bd/0xea0
Dec  2 12:36:12 10.243.0.116 [ 1878.801660]  ? prot_none_pte_entry+0x70/0x70
Dec  2 12:36:12 10.243.0.116 [ 1878.839184]  ? 0xffffffffc0b38099
Dec  2 12:36:12 10.243.0.116 [ 1878.876328]  ? change_protection_range+0x5/0xea0
Dec  2 12:36:12 10.243.0.116 [ 1878.913679]  ? change_protection+0x50/0x90
Dec  2 12:36:12 10.243.0.116 [ 1878.950108]  ? mprotect_fixup+0x2c5/0x490
Dec  2 12:36:12 10.243.0.116 [ 1878.986635]  ? change_protection+0x90/0x90
Dec  2 12:36:12 10.243.0.116 [ 1879.022250]  ? mprotect_fixup+0x5/0x490
Dec  2 12:36:12 10.243.0.116 [ 1879.058323]  ? do_mprotect_pkey+0x32a/0x590
Dec  2 12:36:12 10.243.0.116 [ 1879.092832]  ? mprotect_fixup+0x490/0x490
Dec  2 12:36:12 10.243.0.116 [ 1879.124822]  ? do_mprotect_pkey+0x5/0x590
Dec  2 12:36:12 10.243.0.116 [ 1879.157175]  ? __x64_sys_mprotect+0x48/0x60
Dec  2 12:36:12 10.243.0.116 [ 1879.188722]  ? do_syscall_64+0x4c/0xa0
Dec  2 12:36:13 10.243.0.116 [ 1879.219994]  ? entry_SYSCALL_64_after_hwframe+0x61/0xcb
Dec  2 12:36:13 10.243.0.116 [ 1879.253178]  </TASK>
Dec  2 12:36:38 10.243.0.116 [ 1904.447919] watchdog: BUG: soft lockup - CPU#0 stuck for 179s! [TxStartup:3574]
Dec  2 12:36:38 10.243.0.116 [ 1904.480296] Modules linked in: xt_CHECKSUM xt_MASQUERADE tun netconsole ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 xt_conntrack ebtable_nat ebtable_broute ip6table_nat ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_nat iptable_mangle iptable_security iptable_raw nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 libcrc32c ip_set nfnetlink ebtable_filter ebtables rfkill ip6table_filter ip6_tables iptable_filter sunrpc dm_mirror dm_region_hash dm_log dm_mod x86_pkg_temp_thermal intel_powerclamp coretemp gpio_ich iTCO_wdt iTCO_vendor_support kvm_intel mxm_wmi kvm irqbypass crct10dif_pclmul crc32_pclmul ghash_clmulni_intel aesni_intel crypto_simd cryptd rapl intel_cstate pcspkr sg i2c_i801 ftdi_sio mei_me i2c_smbus joydev input_leds intel_pch_thermal mei lpc_ich mfd_core ioatdma wmi ip_tables ext4 mbcache jbd2 ast drm_vram_helper drm_kms_helper syscopyarea ixgbe sysfillrect sd_mod sr_mod sysimgblt fb_sys_fops cdrom t10_pi drm_ttm_helper ttm mdio drm igb ah
Dec  2 12:36:38 10.243.0.116 [ 1904.493711]  libahci libata crc32c_intel ptp pps_core i2c_algo_bit dca fuse
Dec  2 12:36:38 10.243.0.116 [ 1904.787213] irq event stamp: 24564
Dec  2 12:36:38 10.243.0.116 [ 1904.821224] hardirqs last  enabled at (24563): [<ffffffff8d1d44ad>] sync_current_irq_stage+0x45d/0x500
Dec  2 12:36:38 10.243.0.116 [ 1904.857335] hardirqs last disabled at (24564): [<ffffffff8d49923b>] handle_mm_fault+0x48b/0x520
Dec  2 12:36:38 10.243.0.116 [ 1904.893475] softirqs last  enabled at (0): [<ffffffff8d0ca587>] copy_process+0x1307/0x3770
Dec  2 12:36:38 10.243.0.116 [ 1904.929794] softirqs last disabled at (0): [<0000000000000000>] 0x0
Dec  2 12:36:38 10.243.0.116 [ 1904.965671] CPU: 0 PID: 3574 Comm: TxStartup Tainted: G    B   W    L    5.15.77evl-g5e681f841fc3-dirty #3
Dec  2 12:36:39 10.243.0.116 [ 1905.002931] Hardware name: Supermicro Super Server/X10SDV-6C+-TLN4F, BIOS 2.1 11/22/2019
Dec  2 12:36:39 10.243.0.116 [ 1905.042665] igb 0000:05:00.0: Detected Tx Unit Hang#012[ 1905.042665]   Tx Queue             <7>#012[ 1905.042665]   TDH                  <4>#012[ 1905.042665]   TDT                  <4>#012[ 1905.042665]   next_to_use          <4>#012[ 1905.042665]   next_to_clean        <9e>#012[ 1905.042665] buffer_info[next_to_clean]#012[ 1905.042665]   time_stamp           <100179b5e>#012[ 1905.042665]   next_to_watch        <00000000b00b2101>#012[ 1905.042665]   jiffies              <10018761e>#012[ 1905.042665]   desc.status          <16c001>
Dec  2 12:36:39 10.243.0.116 [ 1905.770842] IRQ stage: Linux
Dec  2 12:36:39 10.243.0.116 [ 1905.806287] RIP: 0010:proxy_set_next_ktime+0x77/0x90
Dec  2 12:36:39 10.243.0.116 [ 1905.842339] Code: 74 37 49 8d 7c 24 30 e8 d7 2d 1d 00 49 8b 44 24 30 4c 89 e7 e8 da 42 0b 01 48 8d 34 03 31 d2 4c 89 ef e8 5c 14 00 00 41 56 9d <31> c0 5b 41 5c 41 5d 41 5e 5d c3 cc cc cc cc e8 25 f5 eb ff eb d9
Dec  2 12:36:39 10.243.0.116 [ 1905.920072] RSP: 0018:ffff88841e209e80 EFLAGS: 00000283 ORIG_RAX: 0000000000000000
Dec  2 12:36:39 10.243.0.116 [ 1905.960846] RAX: 0000000000000000 RBX: 0000000000000000 RCX: 0000000000000000
Dec  2 12:36:39 10.243.0.116 [ 1906.001673] RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000000
Dec  2 12:36:39 10.243.0.116 [ 1906.042600] RBP: ffff88841e209ea0 R08: 0000000000000000 R09: 0000000000000000
Dec  2 12:36:39 10.243.0.116 [ 1906.083768] R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000000
Dec  2 12:36:39 10.243.0.116 [ 1906.125162] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000
Dec  2 12:36:39 10.243.0.116 [ 1906.166634] FS:  00007f301ae2e700(0000) GS:ffff88841e200000(0000) knlGS:0000000000000000
Dec  2 12:36:40 10.243.0.116 [ 1906.208640] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Dec  2 12:36:40 10.243.0.116 [ 1906.250348] CR2: 00007f1e4033b660 CR3: 0000000137d08006 CR4: 00000000003706f0
Dec  2 12:36:40 10.243.0.116 [ 1906.292967] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Dec  2 12:36:40 10.243.0.116 [ 1906.336130] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Dec  2 12:36:40 10.243.0.116 [ 1906.379390] Call Trace:
Dec  2 12:36:40 10.243.0.116 [ 1906.421672]  <IRQ>
Dec  2 12:36:40 10.243.0.116 [ 1906.464011]  ? __hrtimer_run_queues+0x5/0x780
Dec  2 12:36:40 10.243.0.116 [ 1906.507463]  ? hrtimer_interrupt+0x1aa/0x360
Dec  2 12:36:40 10.243.0.116 [ 1906.550618]  ? hrtimer_interrupt+0x5/0x360
Dec  2 12:36:40 10.243.0.116 [ 1906.593206]  ? proxy_irq_handler+0x37/0x50
Dec  2 12:36:40 10.243.0.116 [ 1906.634680]  ? handle_synthetic_irq+0xec/0x320
Dec  2 12:36:40 10.243.0.116 [ 1906.676127]  ? do_irq_inband+0x2f/0x40
Dec  2 12:36:40 10.243.0.116 [ 1906.716771]  ? arch_do_IRQ_pipelined+0xcd/0x5b0
Dec  2 12:36:40 10.243.0.116 [ 1906.757021]  </IRQ>
Dec  2 12:36:40 10.243.0.116 [ 1906.795940]  <TASK>
Dec  2 12:36:40 10.243.0.116 [ 1906.834764]  ? sync_current_irq_stage+0x2e3/0x500
Dec  2 12:36:40 10.243.0.116 [ 1906.874467]  ? sync_irq_stage+0x1af/0x1e0
Dec  2 12:36:40 10.243.0.116 [ 1906.912909]  ? synchronize_pipeline+0x8b/0xe0
Dec  2 12:36:40 10.243.0.116 [ 1906.950683]  ? handle_irq_pipelined_finish+0xf6/0x240
Dec  2 12:36:40 10.243.0.116 [ 1906.988334]  ? arch_pipeline_entry+0x81/0x110
Dec  2 12:36:40 10.243.0.116 [ 1907.025307]  ? sysvec_apic_timer_interrupt+0xe/0x20
Dec  2 12:36:40 10.243.0.116 [ 1907.062001]  ? asm_sysvec_apic_timer_interrupt+0x1b/0x20
Dec  2 12:36:40 10.243.0.116 [ 1907.099412]  ? smp_call_function_many_cond+0x1c7/0x4e0
Dec  2 12:36:40 10.243.0.116 [ 1907.136226]  ? smp_call_function_many_cond+0x1bf/0x4e0
Dec  2 12:36:40 10.243.0.116 [ 1907.172465]  ? smp_call_function_many_cond+0x1c7/0x4e0
Dec  2 12:36:41 10.243.0.116 [ 1907.208233]  ? leave_mm+0x40/0x40
Dec  2 12:36:41 10.243.0.116 [ 1907.243070]  ? x86_configure_nx+0x70/0x70
Dec  2 12:36:41 10.243.0.116 [ 1907.278191]  ? on_each_cpu_cond_mask+0x29/0x40
Dec  2 12:36:41 10.243.0.116 [ 1907.313310]  ? native_flush_tlb_multi+0xbe/0x250
Dec  2 12:36:41 10.243.0.116 [ 1907.349094]  ? flush_tlb_mm_range+0x136/0x1f0
Dec  2 12:36:41 10.243.0.116 [ 1907.385219]  ? change_protection_range+0x8bd/0xea0
Dec  2 12:36:41 10.243.0.116 [ 1907.423372]  ? prot_none_pte_entry+0x70/0x70
Dec  2 12:36:41 10.243.0.116 [ 1907.459370]  ? 0xffffffffc0b38099
Dec  2 12:36:41 10.243.0.116 [ 1907.495409]  ? change_protection_range+0x5/0xea0
Dec  2 12:36:41 10.243.0.116 [ 1907.531782]  ? change_protection+0x50/0x90
Dec  2 12:36:41 10.243.0.116 [ 1907.567611]  ? mprotect_fixup+0x2c5/0x490
Dec  2 12:36:41 10.243.0.116 [ 1907.604020]  ? change_protection+0x90/0x90
Dec  2 12:36:41 10.243.0.116 [ 1907.639849]  ? mprotect_fixup+0x5/0x490
Dec  2 12:36:41 10.243.0.116 [ 1907.676077]  ? do_mprotect_pkey+0x32a/0x590
Dec  2 12:36:41 10.243.0.116 [ 1907.710738]  ? mprotect_fixup+0x490/0x490
Dec  2 12:36:41 10.243.0.116 [ 1907.742886]  ? do_mprotect_pkey+0x5/0x590
Dec  2 12:36:41 10.243.0.116 [ 1907.775292]  ? __x64_sys_mprotect+0x48/0x60
Dec  2 12:36:41 10.243.0.116 [ 1907.806891]  ? do_syscall_64+0x4c/0xa0
Dec  2 12:36:41 10.243.0.116 [ 1907.838214]  ? entry_SYSCALL_64_after_hwframe+0x61/0xcb
Dec  2 12:36:41 10.243.0.116 [ 1907.871250]  </TASK>
Dec  2 12:36:58 10.243.0.116 [ 1924.447866] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/2:2:259]
Dec  2 12:36:58 10.243.0.116 [ 1924.474882] Modules linked in: xt_CHECKSUM xt_MASQUERADE tun netconsole ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 xt_conntrack ebtable_nat ebtable_broute ip6table_nat ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_nat iptable_mangle iptable_security iptable_raw nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 libcrc32c ip_set nfnetlink ebtable_filter ebtables rfkill ip6table_filter ip6_tables iptable_filter sunrpc dm_mirror dm_region_hash dm_log dm_mod x86_pkg_temp_thermal intel_powerclamp coretemp gpio_ich iTCO_wdt iTCO_vendor_support kvm_intel mxm_wmi kvm irqbypass crct10dif_pclmul crc32_pclmul ghash_clmulni_intel aesni_intel crypto_simd cryptd rapl intel_cstate pcspkr sg i2c_i801 ftdi_sio mei_me i2c_smbus joydev input_leds intel_pch_thermal mei lpc_ich mfd_core ioatdma wmi ip_tables ext4 mbcache jbd2 ast drm_vram_helper drm_kms_helper syscopyarea ixgbe sysfillrect sd_mod sr_mod sysimgblt fb_sys_fops cdrom t10_pi drm_ttm_helper ttm mdio drm igb ah
Dec  2 12:36:58 10.243.0.116 [ 1924.486050]  libahci libata crc32c_intel ptp pps_core i2c_algo_bit dca fuse
Dec  2 12:36:58 10.243.0.116 [ 1924.731137] irq event stamp: 23654
Dec  2 12:36:58 10.243.0.116 [ 1924.759431] hardirqs last  enabled at (23653): [<ffffffff8e1bd0d7>] _raw_spin_unlock_irq+0x27/0x40
Dec  2 12:36:58 10.243.0.116 [ 1924.789736] hardirqs last disabled at (23654): [<ffffffff8e1afe3d>] __schedule+0x90d/0x1160
Dec  2 12:36:58 10.243.0.116 [ 1924.820052] softirqs last  enabled at (23540): [<ffffffff8e400332>] __do_softirq+0x332/0x598
Dec  2 12:36:58 10.243.0.116 [ 1924.850525] softirqs last disabled at (23529): [<ffffffff8d0dc80b>] irq_exit_rcu+0xab/0x180
Dec  2 12:36:58 10.243.0.116 [ 1924.881152] CPU: 2 PID: 259 Comm: kworker/2:2 Tainted: G    B   W    L    5.15.77evl-g5e681f841fc3-dirty #3
Dec  2 12:36:58 10.243.0.116 [ 1924.912713] Hardware name: Supermicro Super Server/X10SDV-6C+-TLN4F, BIOS 2.1 11/22/2019
Dec  2 12:36:58 10.243.0.116 [ 1924.944397] IRQ stage: Linux
Dec  2 12:36:58 10.243.0.116 [ 1924.974695] Workqueue: rcu_gp wait_rcu_exp_gp
Dec  2 12:36:58 10.243.0.116 [ 1925.005121] RIP: 0010:proxy_set_next_ktime+0x77/0x90
Dec  2 12:36:58 10.243.0.116 [ 1925.036876] Code: 74 37 49 8d 7c 24 30 e8 d7 2d 1d 00 49 8b 44 24 30 4c 89 e7 e8 da 42 0b 01 48 8d 34 03 31 d2 4c 89 ef e8 5c 14 00 00 41 56 9d <31> c0 5b 41 5c 41 5d 41 5e 5d c3 cc cc cc cc e8 25 f5 eb ff eb d9
Dec  2 12:36:58 10.243.0.116 [ 1925.103160] RSP: 0018:ffff88841e309e80 EFLAGS: 00000206 ORIG_RAX: 0000000000000000
Dec  2 12:36:58 10.243.0.116 [ 1925.137688] RAX: 0000000000000000 RBX: 0000000000000000 RCX: 0000000000000000
Dec  2 12:36:58 10.243.0.116 [ 1925.172075] RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000000
Dec  2 12:36:59 10.243.0.116 [ 1925.206244] RBP: ffff88841e309ea0 R08: 0000000000000000 R09: 0000000000000000
Dec  2 12:36:59 10.243.0.116 [ 1925.241264] R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000000
Dec  2 12:36:59 10.243.0.116 [ 1925.275971] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000
Dec  2 12:36:59 10.243.0.116 [ 1925.310695] FS:  0000000000000000(0000) GS:ffff88841e300000(0000) knlGS:0000000000000000
Dec  2 12:36:59 10.243.0.116 [ 1925.345999] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Dec  2 12:36:59 10.243.0.116 [ 1925.381463] CR2: 00007f20be593000 CR3: 000000012c0f0004 CR4: 00000000003706e0
Dec  2 12:36:59 10.243.0.116 [ 1925.414834] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Dec  2 12:36:59 10.243.0.116 [ 1925.445684] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Dec  2 12:36:59 10.243.0.116 [ 1925.476363] Call Trace:
Dec  2 12:36:59 10.243.0.116 [ 1925.506356]  <IRQ>
Dec  2 12:36:59 10.243.0.116 [ 1925.536560]  ? __hrtimer_run_queues+0x5/0x780
Dec  2 12:36:59 10.243.0.116 [ 1925.567894]  ? hrtimer_interrupt+0x1aa/0x360
Dec  2 12:36:59 10.243.0.116 [ 1925.599718]  ? hrtimer_interrupt+0x5/0x360
Dec  2 12:36:59 10.243.0.116 [ 1925.636448]  ? proxy_irq_handler+0x37/0x50
Dec  2 12:36:59 10.243.0.116 [ 1925.672463]  ? handle_synthetic_irq+0xec/0x320
Dec  2 12:36:59 10.243.0.116 [ 1925.708339]  ? do_irq_inband+0x2f/0x40
Dec  2 12:36:59 10.243.0.116 [ 1925.743946]  ? arch_do_IRQ_pipelined+0xcd/0x5b0
Dec  2 12:36:59 10.243.0.116 [ 1925.779956]  </IRQ>
Dec  2 12:36:59 10.243.0.116 [ 1925.815138]  <TASK>
Dec  2 12:36:59 10.243.0.116 [ 1925.850818]  ? sync_current_irq_stage+0x2e3/0x500
Dec  2 12:36:59 10.243.0.116 [ 1925.887606]  ? sync_irq_stage+0x1af/0x1e0
Dec  2 12:36:59 10.243.0.116 [ 1925.923337]  ? synchronize_pipeline+0x8b/0xe0
Dec  2 12:36:59 10.243.0.116 [ 1925.960426]  ? handle_irq_pipelined_finish+0xf6/0x240
Dec  2 12:36:59 10.243.0.116 [ 1925.996846]  ? arch_pipeline_entry+0x81/0x110
Dec  2 12:36:59 10.243.0.116 [ 1926.032049]  ? sysvec_apic_timer_interrupt+0xe/0x20
Dec  2 12:36:59 10.243.0.116 [ 1926.067847]  ? asm_sysvec_apic_timer_interrupt+0x1b/0x20
Dec  2 12:36:59 10.243.0.116 [ 1926.101966]  ? rcu_all_qs+0xa0/0xa0
Dec  2 12:36:59 10.243.0.116 [ 1926.135872]  ? smp_call_function_single+0x118/0x210
Dec  2 12:36:59 10.243.0.116 [ 1926.169835]  ? __asan_store2+0xa0/0xa0
Dec  2 12:37:00 10.243.0.116 [ 1926.203129]  ? smp_call_function_single+0x118/0x210
Dec  2 12:37:00 10.243.0.116 [ 1926.235828]  ? 0xffffffffc0b38099
Dec  2 12:37:00 10.243.0.116 [ 1926.268548]  ? generic_exec_single+0x140/0x140
Dec  2 12:37:00 10.243.0.116 [ 1926.300817]  ? rcu_all_qs+0xa0/0xa0
Dec  2 12:37:00 10.243.0.116 [ 1926.332965]  ? smp_call_function_single+0x5/0x210
Dec  2 12:37:00 10.243.0.116 [ 1926.364365]  ? sync_rcu_exp_select_node_cpus+0x3df/0x5c0
Dec  2 12:37:00 10.243.0.116 [ 1926.397394]  ? sync_rcu_exp_select_cpus+0x2c5/0x5c0
Dec  2 12:37:00 10.243.0.116 [ 1926.428585]  ? wait_rcu_exp_gp+0x1f/0x30
Dec  2 12:37:00 10.243.0.116 [ 1926.458830]  ? process_one_work+0x525/0xa30
Dec  2 12:37:00 10.243.0.116 [ 1926.490169]  ? pwq_dec_nr_in_flight+0x120/0x120
Dec  2 12:37:00 10.243.0.116 [ 1926.520083]  ? process_one_work+0x5/0xa30
Dec  2 12:37:00 10.243.0.116 [ 1926.550365]  ? worker_thread+0x57/0x590
Dec  2 12:37:00 10.243.0.116 [ 1926.580743]  ? process_one_work+0xa30/0xa30
Dec  2 12:37:00 10.243.0.116 [ 1926.610735]  ? kthread+0x1f3/0x220
Dec  2 12:37:00 10.243.0.116 [ 1926.638736]  ? set_kthread_struct+0x90/0x90
Dec  2 12:37:00 10.243.0.116 [ 1926.665803]  ? ret_from_fork+0x22/0x30
Dec  2 12:37:00 10.243.0.116 [ 1926.692856]  </TASK>
Dec  2 12:37:06 10.243.0.116 [ 1932.448044] watchdog: BUG: soft lockup - CPU#0 stuck for 205s! [TxStartup:3574]
Dec  2 12:37:06 10.243.0.116 [ 1932.481569] Modules linked in: xt_CHECKSUM xt_MASQUERADE tun netconsole ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 xt_conntrack ebtable_nat ebtable_broute ip6table_nat ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_nat iptable_mangle iptable_security iptable_raw nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 libcrc32c ip_set nfnetlink ebtable_filter ebtables rfkill ip6table_filter ip6_tables iptable_filter sunrpc dm_mirror dm_region_hash dm_log dm_mod x86_pkg_temp_thermal intel_powerclamp coretemp gpio_ich iTCO_wdt iTCO_vendor_support kvm_intel mxm_wmi kvm irqbypass crct10dif_pclmul crc32_pclmul ghash_clmulni_intel aesni_intel crypto_simd cryptd rapl intel_cstate pcspkr sg i2c_i801 ftdi_sio mei_me i2c_smbus joydev input_leds intel_pch_thermal mei lpc_ich mfd_core ioatdma wmi ip_tables ext4 mbcache jbd2 ast drm_vram_helper drm_kms_helper syscopyarea ixgbe sysfillrect sd_mod sr_mod sysimgblt fb_sys_fops cdrom t10_pi drm_ttm_helper ttm mdio drm igb ah
Dec  2 12:37:06 10.243.0.116 [ 1932.494984]  libahci libata crc32c_intel ptp pps_core i2c_algo_bit dca fuse
Dec  2 12:37:06 10.243.0.116 [ 1932.803512] irq event stamp: 24564
Dec  2 12:37:06 10.243.0.116 [ 1932.838158] hardirqs last  enabled at (24563): [<ffffffff8d1d44ad>] sync_current_irq_stage+0x45d/0x500
Dec  2 12:37:06 10.243.0.116 [ 1932.874930] hardirqs last disabled at (24564): [<ffffffff8d49923b>] handle_mm_fault+0x48b/0x520
Dec  2 12:37:06 10.243.0.116 [ 1932.911787] softirqs last  enabled at (0): [<ffffffff8d0ca587>] copy_process+0x1307/0x3770
Dec  2 12:37:06 10.243.0.116 [ 1932.948959] softirqs last disabled at (0): [<0000000000000000>] 0x0
Dec  2 12:37:06 10.243.0.116 [ 1932.985721] CPU: 0 PID: 3574 Comm: TxStartup Tainted: G    B   W    L    5.15.77evl-g5e681f841fc3-dirty #3
Dec  2 12:37:06 10.243.0.116 [ 1933.023777] Hardware name: Supermicro Super Server/X10SDV-6C+-TLN4F, BIOS 2.1 11/22/2019
Dec  2 12:37:06 10.243.0.116 [ 1933.061740] IRQ stage: Linux
Dec  2 12:37:06 10.243.0.116 [ 1933.098584] RIP: 0010:proxy_set_next_ktime+0x77/0x90
Dec  2 12:37:06 10.243.0.116 [ 1933.136128] Code: 74 37 49 8d 7c 24 30 e8 d7 2d 1d 00 49 8b 44 24 30 4c 89 e7 e8 da 42 0b 01 48 8d 34 03 31 d2 4c 89 ef e8 5c 14 00 00 41 56 9d <31> c0 5b 41 5c 41 5d 41 5e 5d c3 cc cc cc cc e8 25 f5 eb ff eb d9
Dec  2 12:37:07 10.243.0.116 [ 1933.217191] RSP: 0018:ffff88841e209e80 EFLAGS: 00000287 ORIG_RAX: 0000000000000000
Dec  2 12:37:07 10.243.0.116 [ 1933.259614] RAX: 0000000000000000 RBX: 0000000000000000 RCX: 0000000000000000
Dec  2 12:37:07 10.243.0.116 [ 1933.302070] RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000000
Dec  2 12:37:07 10.243.0.116 [ 1933.344483] RBP: ffff88841e209ea0 R08: 0000000000000000 R09: 0000000000000000
Dec  2 12:37:07 10.243.0.116 [ 1933.387081] R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000000
Dec  2 12:37:07 10.243.0.116 [ 1933.429714] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000
Dec  2 12:37:07 10.243.0.116 [ 1933.472202] FS:  00007f301ae2e700(0000) GS:ffff88841e200000(0000) knlGS:0000000000000000
Dec  2 12:37:07 10.243.0.116 [ 1933.515298] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Dec  2 12:37:07 10.243.0.116 [ 1933.558143] CR2: 00007f1e4033b660 CR3: 0000000137d08006 CR4: 00000000003706f0
Dec  2 12:37:07 10.243.0.116 [ 1933.601795] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Dec  2 12:37:07 10.243.0.116 [ 1933.646100] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Dec  2 12:37:07 10.243.0.116 [ 1933.690635] Call Trace:
Dec  2 12:37:07 10.243.0.116 [ 1933.734040]  <IRQ>
Dec  2 12:37:07 10.243.0.116 [ 1933.777505]  ? __hrtimer_run_queues+0x5/0x780
Dec  2 12:37:07 10.243.0.116 [ 1933.822038]  ? hrtimer_interrupt+0x1aa/0x360
Dec  2 12:37:07 10.243.0.116 [ 1933.866137]  ? hrtimer_interrupt+0x5/0x360
Dec  2 12:37:07 10.243.0.116 [ 1933.910370]  ? proxy_irq_handler+0x37/0x50
Dec  2 12:37:07 10.243.0.116 [ 1933.954028]  ? handle_synthetic_irq+0xec/0x320
Dec  2 12:37:07 10.243.0.116 [ 1933.998028]  ? do_irq_inband+0x2f/0x40
Dec  2 12:37:07 10.243.0.116 [ 1934.041527]  ? arch_do_IRQ_pipelined+0xcd/0x5b0
Dec  2 12:37:07 10.243.0.116 [ 1934.085139]  </IRQ>
Dec  2 12:37:07 10.243.0.116 [ 1934.127790]  <TASK>
Dec  2 12:37:07 10.243.0.116 [ 1934.170607]  ? sync_current_irq_stage+0x2e3/0x500
Dec  2 12:37:08 10.243.0.116 [ 1934.214757]  ? sync_irq_stage+0x1af/0x1e0
Dec  2 12:37:08 10.243.0.116 [ 1934.257972]  ? synchronize_pipeline+0x8b/0xe0
Dec  2 12:37:08 10.243.0.116 [ 1934.301188]  ? handle_irq_pipelined_finish+0xf6/0x240
Dec  2 12:37:08 10.243.0.116 [ 1934.344721]  ? arch_pipeline_entry+0x81/0x110
Dec  2 12:37:08 10.243.0.116 [ 1934.387389]  ? sysvec_apic_timer_interrupt+0xe/0x20
Dec  2 12:37:08 10.243.0.116 [ 1934.429328]  ? asm_sysvec_apic_timer_interrupt+0x1b/0x20
Dec  2 12:37:08 10.243.0.116 [ 1934.471946]  ? smp_call_function_many_cond+0x1c7/0x4e0
Dec  2 12:37:08 10.243.0.116 [ 1934.513618]  ? __asan_store2+0xa0/0xa0
Dec  2 12:37:08 10.243.0.116 [ 1934.554403]  ? smp_call_function_many_cond+0x1c7/0x4e0
Dec  2 12:37:08 10.243.0.116 [ 1934.595051]  ? leave_mm+0x40/0x40
Dec  2 12:37:08 10.243.0.116 [ 1934.634709]  ? x86_configure_nx+0x70/0x70
Dec  2 12:37:08 10.243.0.116 [ 1934.674264]  ? on_each_cpu_cond_mask+0x29/0x40
Dec  2 12:37:08 10.243.0.116 [ 1934.713336]  ? native_flush_tlb_multi+0xbe/0x250
Dec  2 12:37:08 10.243.0.116 [ 1934.752062]  ? flush_tlb_mm_range+0x136/0x1f0
Dec  2 12:37:08 10.243.0.116 [ 1934.790495]  ? change_protection_range+0x8bd/0xea0
Dec  2 12:37:08 10.243.0.116 [ 1934.830568]  ? prot_none_pte_entry+0x70/0x70
Dec  2 12:37:08 10.243.0.116 [ 1934.868293]  ? 0xffffffffc0b38099
Dec  2 12:37:08 10.243.0.116 [ 1934.905569]  ? change_protection_range+0x5/0xea0
Dec  2 12:37:08 10.243.0.116 [ 1934.943054]  ? change_protection+0x50/0x90
Dec  2 12:37:08 10.243.0.116 [ 1934.979616]  ? mprotect_fixup+0x2c5/0x490
Dec  2 12:37:08 10.243.0.116 [ 1935.016274]  ? change_protection+0x90/0x90
Dec  2 12:37:08 10.243.0.116 [ 1935.052032]  ? mprotect_fixup+0x5/0x490
Dec  2 12:37:08 10.243.0.116 [ 1935.088192]  ? do_mprotect_pkey+0x32a/0x590
Dec  2 12:37:08 10.243.0.116 [ 1935.122778]  ? mprotect_fixup+0x490/0x490
Dec  2 12:37:08 10.243.0.116 [ 1935.154894]  ? do_mprotect_pkey+0x5/0x590
Dec  2 12:37:08 10.243.0.116 [ 1935.187329]  ? __x64_sys_mprotect+0x48/0x60
Dec  2 12:37:09 10.243.0.116 [ 1935.219003]  ? do_syscall_64+0x4c/0xa0
Dec  2 12:37:09 10.243.0.116 [ 1935.250422]  ? entry_SYSCALL_64_after_hwframe+0x61/0xcb
Dec  2 12:37:09 10.243.0.116 [ 1935.283600]  </TASK>
Dec  2 12:37:26 10.243.0.116 [ 1952.447989] watchdog: BUG: soft lockup - CPU#2 stuck for 49s! [kworker/2:2:259]
Dec  2 12:37:26 10.243.0.116 [ 1952.474936] Modules linked in: xt_CHECKSUM xt_MASQUERADE tun netconsole ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 xt_conntrack ebtable_nat ebtable_broute ip6table_nat ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_nat iptable_mangle iptable_security iptable_raw nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 libcrc32c ip_set nfnetlink ebtable_filter ebtables rfkill ip6table_filter ip6_tables iptable_filter sunrpc dm_mirror dm_region_hash dm_log dm_mod x86_pkg_temp_thermal intel_powerclamp coretemp gpio_ich iTCO_wdt iTCO_vendor_support kvm_intel mxm_wmi kvm irqbypass crct10dif_pclmul crc32_pclmul ghash_clmulni_intel aesni_intel crypto_simd cryptd rapl intel_cstate pcspkr sg i2c_i801 ftdi_sio mei_me i2c_smbus joydev input_leds intel_pch_thermal mei lpc_ich mfd_core ioatdma wmi ip_tables ext4 mbcache jbd2 ast drm_vram_helper drm_kms_helper syscopyarea ixgbe sysfillrect sd_mod sr_mod sysimgblt fb_sys_fops cdrom t10_pi drm_ttm_helper ttm mdio drm igb ah
Dec  2 12:37:26 10.243.0.116 [ 1952.485927]  libahci libata crc32c_intel ptp pps_core i2c_algo_bit dca fuse
Dec  2 12:37:26 10.243.0.116 [ 1952.730986] irq event stamp: 23654
Dec  2 12:37:26 10.243.0.116 [ 1952.758774] hardirqs last  enabled at (23653): [<ffffffff8e1bd0d7>] _raw_spin_unlock_irq+0x27/0x40
Dec  2 12:37:26 10.243.0.116 [ 1952.783952] hardirqs last disabled at (23654): [<ffffffff8e1afe3d>] __schedule+0x90d/0x1160
Dec  2 12:37:26 10.243.0.116 [ 1952.813744] softirqs last  enabled at (23540): [<ffffffff8e400332>] __do_softirq+0x332/0x598
Dec  2 12:37:26 10.243.0.116 [ 1952.843412] softirqs last disabled at (23529): [<ffffffff8d0dc80b>] irq_exit_rcu+0xab/0x180
Dec  2 12:37:26 10.243.0.116 [ 1952.873239] CPU: 2 PID: 259 Comm: kworker/2:2 Tainted: G    B   W    L    5.15.77evl-g5e681f841fc3-dirty #3
Dec  2 12:37:26 10.243.0.116 [ 1952.904613] Hardware name: Supermicro Super Server/X10SDV-6C+-TLN4F, BIOS 2.1 11/22/2019
Dec  2 12:37:26 10.243.0.116 [ 1952.935686] IRQ stage: Linux
Dec  2 12:37:26 10.243.0.116 [ 1952.965494] Workqueue: rcu_gp wait_rcu_exp_gp
Dec  2 12:37:26 10.243.0.116 [ 1952.997078] RIP: 0010:proxy_set_next_ktime+0x77/0x90
Dec  2 12:37:26 10.243.0.116 [ 1953.027504] Code: 74 37 49 8d 7c 24 30 e8 d7 2d 1d 00 49 8b 44 24 30 4c 89 e7 e8 da 42 0b 01 48 8d 34 03 31 d2 4c 89 ef e8 5c 14 00 00 41 56 9d <31> c0 5b 41 5c 41 5d 41 5e 5d c3 cc cc cc cc e8 25 f5 eb ff eb d9
Dec  2 12:37:26 10.243.0.116 [ 1953.093929] RSP: 0018:ffff88841e309e80 EFLAGS: 00000206 ORIG_RAX: 0000000000000000
Dec  2 12:37:26 10.243.0.116 [ 1953.128894] RAX: 0000000000000000 RBX: 0000000000000000 RCX: 0000000000000000
Dec  2 12:37:26 10.243.0.116 [ 1953.163657] RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000000
Dec  2 12:37:27 10.243.0.116 [ 1953.198326] RBP: ffff88841e309ea0 R08: 0000000000000000 R09: 0000000000000000
Dec  2 12:37:27 10.243.0.116 [ 1953.233457] R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000000
Dec  2 12:37:27 10.243.0.116 [ 1953.268560] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000
Dec  2 12:37:27 10.243.0.116 [ 1953.303012] FS:  0000000000000000(0000) GS:ffff88841e300000(0000) knlGS:0000000000000000
Dec  2 12:37:27 10.243.0.116 [ 1953.338635] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Dec  2 12:37:27 10.243.0.116 [ 1953.373694] CR2: 00007f20be593000 CR3: 000000012c0f0004 CR4: 00000000003706e0
Dec  2 12:37:27 10.243.0.116 [ 1953.409703] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Dec  2 12:37:27 10.243.0.116 [ 1953.445762] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Dec  2 12:37:27 10.243.0.116 [ 1953.482185] Call Trace:
Dec  2 12:37:27 10.243.0.116 [ 1953.518035]  <IRQ>
Dec  2 12:37:27 10.243.0.116 [ 1953.553529]  ? __hrtimer_run_queues+0x5/0x780
Dec  2 12:37:27 10.243.0.116 [ 1953.589673]  ? hrtimer_interrupt+0x1aa/0x360
Dec  2 12:37:27 10.243.0.116 [ 1953.626461]  ? hrtimer_interrupt+0x5/0x360
Dec  2 12:37:27 10.243.0.116 [ 1953.662586]  ? proxy_irq_handler+0x37/0x50
Dec  2 12:37:27 10.243.0.116 [ 1953.698155]  ? handle_synthetic_irq+0xec/0x320
Dec  2 12:37:27 10.243.0.116 [ 1953.734705]  ? do_irq_inband+0x2f/0x40
Dec  2 12:37:27 10.243.0.116 [ 1953.770931]  ? arch_do_IRQ_pipelined+0xcd/0x5b0
Dec  2 12:37:27 10.243.0.116 [ 1953.806936]  </IRQ>
Dec  2 12:37:27 10.243.0.116 [ 1953.842230]  <TASK>
Dec  2 12:37:27 10.243.0.116 [ 1953.877274]  ? sync_current_irq_stage+0x2e3/0x500
Dec  2 12:37:27 10.243.0.116 [ 1953.914055]  ? sync_irq_stage+0x1af/0x1e0
Dec  2 12:37:27 10.243.0.116 [ 1953.949958]  ? synchronize_pipeline+0x8b/0xe0
Dec  2 12:37:27 10.243.0.116 [ 1953.985819]  ? handle_irq_pipelined_finish+0xf6/0x240
Dec  2 12:37:27 10.243.0.116 [ 1954.021147]  ? arch_pipeline_entry+0x81/0x110
Dec  2 12:37:27 10.243.0.116 [ 1954.056384]  ? sysvec_apic_timer_interrupt+0xe/0x20
Dec  2 12:37:27 10.243.0.116 [ 1954.091256]  ? asm_sysvec_apic_timer_interrupt+0x1b/0x20
Dec  2 12:37:27 10.243.0.116 [ 1954.120285]  ? rcu_all_qs+0xa0/0xa0
Dec  2 12:37:27 10.243.0.116 [ 1954.149372]  ? smp_call_function_single+0x118/0x210
Dec  2 12:37:28 10.243.0.116 [ 1954.178114]  ? __asan_load4+0x8/0x90
Dec  2 12:37:28 10.243.0.116 [ 1954.207467] igb 0000:05:00.0: Detected Tx Unit Hang#012[ 1954.207467]   Tx Queue             <7>#012[ 1954.207467]   TDH                  <ed>#012[ 1954.207467]   TDT                  <ed>#012[ 1954.207467]   next_to_use          <ed>#012[ 1954.207467]   next_to_clean        <84>#012[ 1954.207467] buffer_info[next_to_clean]#012[ 1954.207467]   time_stamp           <10018e37e>#012[ 1954.207467]   next_to_watch        <0000000040fb38c4>#012[ 1954.207467]   jiffies              <10019387d>#012[ 1954.207467]   desc.status          <220001>
Dec  2 12:37:28 10.243.0.116 [ 1954.685801]  ? smp_call_function_single+0x118/0x210
Dec  2 12:37:28 10.243.0.116 [ 1954.710060]  ? 0xffffffffc0b38099
Dec  2 12:37:28 10.243.0.116 [ 1954.734207]  ? generic_exec_single+0x140/0x140
Dec  2 12:37:28 10.243.0.116 [ 1954.757776]  ? rcu_all_qs+0xa0/0xa0
Dec  2 12:37:28 10.243.0.116 [ 1954.779314]  ? smp_call_function_single+0x5/0x210
Dec  2 12:37:28 10.243.0.116 [ 1954.803030]  ? sync_rcu_exp_select_node_cpus+0x3df/0x5c0
Dec  2 12:37:28 10.243.0.116 [ 1954.825421]  ? sync_rcu_exp_select_cpus+0x2c5/0x5c0
Dec  2 12:37:28 10.243.0.116 [ 1954.847117]  ? wait_rcu_exp_gp+0x1f/0x30
Dec  2 12:37:28 10.243.0.116 [ 1954.868608]  ? process_one_work+0x525/0xa30
Dec  2 12:37:28 10.243.0.116 [ 1954.890142]  ? pwq_dec_nr_in_flight+0x120/0x120
Dec  2 12:37:28 10.243.0.116 [ 1954.911808]  ? process_one_work+0x5/0xa30
Dec  2 12:37:28 10.243.0.116 [ 1954.933180]  ? worker_thread+0x57/0x590
Dec  2 12:37:28 10.243.0.116 [ 1954.954188]  ? process_one_work+0xa30/0xa30
Dec  2 12:37:28 10.243.0.116 [ 1954.975074]  ? kthread+0x1f3/0x220
Dec  2 12:37:28 10.243.0.116 [ 1954.995023]  ? set_kthread_struct+0x90/0x90
Dec  2 12:37:28 10.243.0.116 [ 1955.015076]  ? ret_from_fork+0x22/0x30
Dec  2 12:37:28 10.243.0.116 [ 1955.035934]  </TASK>
Dec  2 12:37:34 10.243.0.116 [ 1960.448168] watchdog: BUG: soft lockup - CPU#0 stuck for 231s! [TxStartup:3574]
Dec  2 12:37:34 10.243.0.116 [ 1960.478483] Modules linked in: xt_CHECKSUM xt_MASQUERADE tun netconsole ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 xt_conntrack ebtable_nat ebtable_broute ip6table_nat ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_nat iptable_mangle iptable_security iptable_raw nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 libcrc32c ip_set nfnetlink ebtable_filter ebtables rfkill ip6table_filter ip6_tables iptable_filter sunrpc dm_mirror dm_region_hash dm_log dm_mod x86_pkg_temp_thermal intel_powerclamp coretemp gpio_ich iTCO_wdt iTCO_vendor_support kvm_intel mxm_wmi kvm irqbypass crct10dif_pclmul crc32_pclmul ghash_clmulni_intel aesni_intel crypto_simd cryptd rapl intel_cstate pcspkr sg i2c_i801 ftdi_sio mei_me i2c_smbus joydev input_leds intel_pch_thermal mei lpc_ich mfd_core ioatdma wmi ip_tables ext4 mbcache jbd2 ast drm_vram_helper drm_kms_helper syscopyarea ixgbe sysfillrect sd_mod sr_mod sysimgblt fb_sys_fops cdrom t10_pi drm_ttm_helper ttm mdio drm igb ah
Dec  2 12:37:34 10.243.0.116 [ 1960.491894]  libahci libata crc32c_intel ptp pps_core i2c_algo_bit dca fuse
Dec  2 12:37:34 10.243.0.116 [ 1960.784536] irq event stamp: 24564
Dec  2 12:37:34 10.243.0.116 [ 1960.818817] hardirqs last  enabled at (24563): [<ffffffff8d1d44ad>] sync_current_irq_stage+0x45d/0x500
Dec  2 12:37:34 10.243.0.116 [ 1960.855204] hardirqs last disabled at (24564): [<ffffffff8d49923b>] handle_mm_fault+0x48b/0x520
Dec  2 12:37:34 10.243.0.116 [ 1960.891595] softirqs last  enabled at (0): [<ffffffff8d0ca587>] copy_process+0x1307/0x3770
Dec  2 12:37:34 10.243.0.116 [ 1960.928082] softirqs last disabled at (0): [<0000000000000000>] 0x0
Dec  2 12:37:34 10.243.0.116 [ 1960.964233] CPU: 0 PID: 3574 Comm: TxStartup Tainted: G    B   W    L    5.15.77evl-g5e681f841fc3-dirty #3
Dec  2 12:37:34 10.243.0.116 [ 1961.001650] Hardware name: Supermicro Super Server/X10SDV-6C+-TLN4F, BIOS 2.1 11/22/2019
Dec  2 12:37:34 10.243.0.116 [ 1961.038982] IRQ stage: Linux
Dec  2 12:37:34 10.243.0.116 [ 1961.074975] RIP: 0010:proxy_set_next_ktime+0x77/0x90
Dec  2 12:37:34 10.243.0.116 [ 1961.111655] Code: 74 37 49 8d 7c 24 30 e8 d7 2d 1d 00 49 8b 44 24 30 4c 89 e7 e8 da 42 0b 01 48 8d 34 03 31 d2 4c 89 ef e8 5c 14 00 00 41 56 9d <31> c0 5b 41 5c 41 5d 41 5e 5d c3 cc cc cc cc e8 25 f5 eb ff eb d9
Dec  2 12:37:35 10.243.0.116 [ 1961.190288] RSP: 0018:ffff88841e209e80 EFLAGS: 00000287 ORIG_RAX: 0000000000000000
Dec  2 12:37:35 10.243.0.116 [ 1961.231492] RAX: 0000000000000000 RBX: 0000000000000000 RCX: 0000000000000000
Dec  2 12:37:35 10.243.0.116 [ 1961.272945] RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000000
Dec  2 12:37:35 10.243.0.116 [ 1961.314553] RBP: ffff88841e209ea0 R08: 0000000000000000 R09: 0000000000000000
Dec  2 12:37:35 10.243.0.116 [ 1961.356411] R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000000
Dec  2 12:37:35 10.243.0.116 [ 1961.398485] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000
Dec  2 12:37:35 10.243.0.116 [ 1961.440680] FS:  00007f301ae2e700(0000) GS:ffff88841e200000(0000) knlGS:0000000000000000
Dec  2 12:37:35 10.243.0.116 [ 1961.483538] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Dec  2 12:37:35 10.243.0.116 [ 1961.526168] CR2: 00007f1e4033b660 CR3: 0000000137d08006 CR4: 00000000003706f0
Dec  2 12:37:35 10.243.0.116 [ 1961.569681] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Dec  2 12:37:35 10.243.0.116 [ 1961.613769] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Dec  2 12:37:35 10.243.0.116 [ 1961.658008] Call Trace:
Dec  2 12:37:35 10.243.0.116 [ 1961.701259]  <IRQ>
Dec  2 12:37:35 10.243.0.116 [ 1961.744566]  ? __hrtimer_run_queues+0x5/0x780
Dec  2 12:37:35 10.243.0.116 [ 1961.788936]  ? hrtimer_interrupt+0x1aa/0x360
Dec  2 12:37:35 10.243.0.116 [ 1961.832861]  ? hrtimer_interrupt+0x5/0x360
Dec  2 12:37:35 10.243.0.116 [ 1961.876922]  ? proxy_irq_handler+0x37/0x50
Dec  2 12:37:35 10.243.0.116 [ 1961.920413]  ? handle_synthetic_irq+0xec/0x320
Dec  2 12:37:35 10.243.0.116 [ 1961.964303]  ? do_irq_inband+0x2f/0x40
Dec  2 12:37:35 10.243.0.116 [ 1962.007709]  ? arch_do_IRQ_pipelined+0xcd/0x5b0
Dec  2 12:37:35 10.243.0.116 [ 1962.051271]  </IRQ>
Dec  2 12:37:35 10.243.0.116 [ 1962.093927]  <TASK>
Dec  2 12:37:35 10.243.0.116 [ 1962.136826]  ? sync_current_irq_stage+0x2e3/0x500
Dec  2 12:37:35 10.243.0.116 [ 1962.181099]  ? sync_irq_stage+0x1af/0x1e0
Dec  2 12:37:36 10.243.0.116 [ 1962.224429]  ? synchronize_pipeline+0x8b/0xe0
Dec  2 12:37:36 10.243.0.116 [ 1962.267645]  ? handle_irq_pipelined_finish+0xf6/0x240
Dec  2 12:37:36 10.243.0.116 [ 1962.311183]  ? arch_pipeline_entry+0x81/0x110
Dec  2 12:37:36 10.243.0.116 [ 1962.353851]  ? sysvec_apic_timer_interrupt+0xe/0x20
Dec  2 12:37:36 10.243.0.116 [ 1962.395733]  ? asm_sysvec_apic_timer_interrupt+0x1b/0x20
Dec  2 12:37:36 10.243.0.116 [ 1962.438323]  ? smp_call_function_many_cond+0x1c7/0x4e0
Dec  2 12:37:36 10.243.0.116 [ 1962.479878]  ? __asan_store2+0xa0/0xa0
Dec  2 12:37:36 10.243.0.116 [ 1962.520466]  ? smp_call_function_many_cond+0x1c7/0x4e0
Dec  2 12:37:36 10.243.0.116 [ 1962.561054]  ? leave_mm+0x40/0x40
Dec  2 12:37:36 10.243.0.116 [ 1962.600604]  ? x86_configure_nx+0x70/0x70
Dec  2 12:37:36 10.243.0.116 [ 1962.640035]  ? on_each_cpu_cond_mask+0x29/0x40
Dec  2 12:37:36 10.243.0.116 [ 1962.678918]  ? native_flush_tlb_multi+0xbe/0x250
Dec  2 12:37:36 10.243.0.116 [ 1962.717523]  ? flush_tlb_mm_range+0x136/0x1f0
Dec  2 12:37:36 10.243.0.116 [ 1962.755835]  ? change_protection_range+0x8bd/0xea0
Dec  2 12:37:36 10.243.0.116 [ 1962.795777]  ? prot_none_pte_entry+0x70/0x70
Dec  2 12:37:36 10.243.0.116 [ 1962.833324]  ? 0xffffffffc0b38099
Dec  2 12:37:36 10.243.0.116 [ 1962.870487]  ? change_protection_range+0x5/0xea0
Dec  2 12:37:36 10.243.0.116 [ 1962.907838]  ? change_protection+0x50/0x90
Dec  2 12:37:36 10.243.0.116 [ 1962.944284]  ? mprotect_fixup+0x2c5/0x490
Dec  2 12:37:36 10.243.0.116 [ 1962.980882]  ? change_protection+0x90/0x90
Dec  2 12:37:36 10.243.0.116 [ 1963.016620]  ? mprotect_fixup+0x5/0x490
Dec  2 12:37:36 10.243.0.116 [ 1963.052819]  ? do_mprotect_pkey+0x32a/0x590
Dec  2 12:37:36 10.243.0.116 [ 1963.087401]  ? mprotect_fixup+0x490/0x490
Dec  2 12:37:36 10.243.0.116 [ 1963.119515]  ? do_mprotect_pkey+0x5/0x590
Dec  2 12:37:36 10.243.0.116 [ 1963.151948]  ? __x64_sys_mprotect+0x48/0x60
Dec  2 12:37:36 10.243.0.116 [ 1963.183633]  ? do_syscall_64+0x4c/0xa0
Dec  2 12:37:37 10.243.0.116 [ 1963.215046]  ? entry_SYSCALL_64_after_hwframe+0x61/0xcb
Dec  2 12:37:37 10.243.0.116 [ 1963.248223]  </TASK>
Dec  2 12:37:52 10.243.0.116 [ 1978.563645] rcu: INFO: rcu_sched detected stalls on CPUs/tasks:
Dec  2 12:37:52 10.243.0.116 [ 1978.589999] rcu: #0115-...0: (1 GPs behind) idle=78b/1/0x4000000000000002 softirq=161154/161154 fqs=31605
Dec  2 12:37:52 10.243.0.116 [ 1978.621786] #011(detected by 2, t=263092 jiffies, g=126833, q=17875)
Dec  2 12:37:52 10.243.0.116 [ 1978.648082] Sending NMI from CPU 2 to CPUs 5:
Dec  2 12:37:52 10.243.0.116 [ 1978.678270] NMI backtrace for cpu 5
Dec  2 12:37:52 10.243.0.116 [ 1978.678291] CPU: 5 PID: 3577 Comm: DummyTxSample Tainted: G    B   W    L    5.15.77evl-g5e681f841fc3-dirty #3
Dec  2 12:37:52 10.243.0.116 [ 1978.678311] Hardware name: Supermicro Super Server/X10SDV-6C+-TLN4F, BIOS 2.1 11/22/2019
Dec  2 12:37:52 10.243.0.116 [ 1978.678328] IRQ stage: EVL
Dec  2 12:37:52 10.243.0.116 [ 1978.678372] RIP: 0010:__asan_load4+0x21/0x90
Dec  2 12:37:52 10.243.0.116 [ 1978.678442] Code: 2e 0f 1f 84 00 00 00 00 00 55 48 89 e5 48 8b 4d 08 48 83 ff fb 77 53 48 b8 ff ff ff ff ff 7f ff ff 48 39 c7 76 44 48 8d 47 03 <48> 89 c2 83 e2 07 48 83 fa 02 76 1c 48 be 00 00 00 00 00 fc ff df
Dec  2 12:37:52 10.243.0.116 [ 1978.678482] RSP: 0018:ffff888137737528 EFLAGS: 00000012
Dec  2 12:37:52 10.243.0.116 [ 1978.678550] RAX: ffff888101aae003 RBX: ffff888101aae000 RCX: ffffffff8d1abe74
Dec  2 12:37:52 10.243.0.116 [ 1978.678589] RDX: 0000000000000003 RSI: dffffc0000000000 RDI: ffff888101aae000
Dec  2 12:37:52 10.243.0.116 [ 1978.678626] RBP: ffff888137737528 R08: ffffed1020355c01 R09: ffffed1020355c01
Dec  2 12:37:52 10.243.0.116 [ 1978.678664] R10: ffff888101aae003 R11: ffffed1020355c00 R12: 1ffff11026ee6eaa
Dec  2 12:37:52 10.243.0.116 [ 1978.678704] R13: 0000000000000000 R14: ffff8881377375f0 R15: 0000000000000001
Dec  2 12:37:52 10.243.0.116 [ 1978.678765] FS:  00007f3013fff700(0000) GS:ffff88841e480000(0000) knlGS:0000000000000000
Dec  2 12:37:52 10.243.0.116 [ 1978.678805] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Dec  2 12:37:52 10.243.0.116 [ 1978.678842] CR2: 0000000000000070 CR3: 0000000137d08002 CR4: 00000000003706e0
Dec  2 12:37:52 10.243.0.116 [ 1978.678880] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Dec  2 12:37:53 10.243.0.116 [ 1978.678917] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Dec  2 12:37:53 10.243.0.116 [ 1978.678956] Call Trace:
Dec  2 12:37:53 10.243.0.116 [ 1978.678976]  <TASK>
Dec  2 12:37:53 10.243.0.116 [ 1978.679030]  native_queued_spin_lock_slowpath+0x194/0x4c0
Dec  2 12:37:53 10.243.0.116 [ 1978.679106]  ? 0xffffffffc0b38099
Dec  2 12:37:53 10.243.0.116 [ 1978.679162]  ? trace_function+0x11e/0x180
Dec  2 12:37:53 10.243.0.116 [ 1978.679263]  ? .slowpath+0x1d/0x1d
Dec  2 12:37:53 10.243.0.116 [ 1978.679788]  ? do_raw_spin_lock+0xe6/0x1e0
Dec  2 12:37:53 10.243.0.116 [ 1978.680050]  ? native_queued_spin_lock_slowpath+0x5/0x4c0
Dec  2 12:37:53 10.243.0.116 [ 1978.680121]  ? 0xffffffffc0b38099
Dec  2 12:37:53 10.243.0.116 [ 1978.680592]  do_raw_spin_lock+0x1cb/0x1e0
Dec  2 12:37:53 10.243.0.116 [ 1978.680823]  ? spin_dump+0x90/0x90
Dec  2 12:37:53 10.243.0.116 [ 1978.681064]  ? do_raw_spin_lock+0x5/0x1e0
Dec  2 12:37:53 10.243.0.116 [ 1978.681565]  evl_switch_inband+0xad/0x7c0
Dec  2 12:37:53 10.243.0.116 [ 1978.682123]  handle_oob_trap_entry+0x10a/0x200
Dec  2 12:37:53 10.243.0.116 [ 1978.682405]  __oob_trap_notify+0x3f/0x50
Dec  2 12:37:53 10.243.0.116 [ 1978.682788]  page_fault_oops+0x1e4/0x3d0
Dec  2 12:37:53 10.243.0.116 [ 1978.682941]  ? trace_function+0x11e/0x180
Dec  2 12:37:53 10.243.0.116 [ 1978.683017]  ? do_user_addr_fault+0x4af/0xa30
Dec  2 12:37:53 10.243.0.116 [ 1978.683151]  ? spurious_kernel_fault+0x200/0x200
Dec  2 12:37:53 10.243.0.116 [ 1978.683252]  ? function_trace_call+0x153/0x1d0
Dec  2 12:37:53 10.243.0.116 [ 1978.683283]  ? do_user_addr_fault+0x4af/0xa30
Dec  2 12:37:53 10.243.0.116 [ 1978.683315]  ? spurious_kernel_fault+0x200/0x200
Dec  2 12:37:53 10.243.0.116 [ 1978.683840]  ? 0xffffffffc0b38099
Dec  2 12:37:53 10.243.0.116 [ 1978.684162]  ? do_user_addr_fault+0xa30/0xa30
Dec  2 12:37:53 10.243.0.116 [ 1978.684332]  ? do_user_addr_fault+0x686/0xa30
Dec  2 12:37:53 10.243.0.116 [ 1978.684711]  ? page_fault_oops+0x5/0x3d0
Dec  2 12:37:53 10.243.0.116 [ 1978.684858]  ? exc_page_fault+0x5c/0xe0
Dec  2 12:37:53 10.243.0.116 [ 1978.685214]  do_user_addr_fault+0x4af/0xa30
Dec  2 12:37:53 10.243.0.116 [ 1978.685250]  ? rcu_read_lock_held_common+0x5/0x70
Dec  2 12:37:53 10.243.0.116 [ 1978.685317]  ? rcu_read_lock_held_common+0x2a/0x70
Dec  2 12:37:53 10.243.0.116 [ 1978.685910]  ? trace_page_fault_user+0x130/0x130
Dec  2 12:37:54 10.243.0.116 [ 1978.686104]  ? fault_in_kernel_space+0x5/0x40
Dec  2 12:37:54 10.243.0.116 [ 1978.686659]  exc_page_fault+0x71/0xe0
Dec  2 12:37:54 10.243.0.116 [ 1978.687029]  asm_exc_page_fault+0x27/0x30
Dec  2 12:37:54 10.243.0.116 [ 1978.687066] RIP: 0010:trace_event_raw_event_evl_sleep_on+0xc3/0x380
Dec  2 12:37:54 10.243.0.116 [ 1978.687100] Code: 80 0f 85 93 02 00 00 41 f6 c7 40 0f 85 10 02 00 00 41 f7 c7 00 02 00 00 0f 85 8f 02 00 00 4d 8d 7d 70 4c 89 ff e8 6d e9 1e 00 <49> 8b 7d 70 48 85 ff 0f 84 2d 02 00 00 e8 ab b4 58 00 8d 50 01 83
Dec  2 12:37:54 10.243.0.116 [ 1978.687119] RSP: 0018:ffff888137737aa0 EFLAGS: 00010082
Dec  2 12:37:54 10.243.0.116 [ 1978.687150] RAX: 0000000000000001 RBX: 1ffff11026ee6f5a RCX: ffffffff8d1acefc
Dec  2 12:37:54 10.243.0.116 [ 1978.687167] RDX: 0000000000000003 RSI: dffffc0000000000 RDI: ffffffff8f2acb40
Dec  2 12:37:54 10.243.0.116 [ 1978.687184] RBP: ffff888137737b78 R08: fffffbfff1e55969 R09: fffffbfff1e55969
Dec  2 12:37:54 10.243.0.116 [ 1978.687201] R10: ffffffff8f2acb43 R11: fffffbfff1e55968 R12: ffff888100195000
Dec  2 12:37:54 10.243.0.116 [ 1978.687218] R13: 0000000000000000 R14: ffffffff8f219d40 R15: 0000000000000070
Dec  2 12:37:54 10.243.0.116 [ 1978.687962]  ? do_raw_spin_unlock+0x9c/0x140
Dec  2 12:37:54 10.243.0.116 [ 1978.688807]  ? trace_event_raw_event_evl_init_thread+0x330/0x330
Dec  2 12:37:54 10.243.0.116 [ 1978.689063]  ? 0xffffffffc0b38099
Dec  2 12:37:54 10.243.0.116 [ 1978.689117]  ? evl_delay+0x48/0xf0
Dec  2 12:37:54 10.243.0.116 [ 1978.689149]  ? evl_sleep_on_locked+0x730/0x730
Dec  2 12:37:54 10.243.0.116 [ 1978.689316]  ? __kasan_check_write+0x14/0x20
Dec  2 12:37:54 10.243.0.116 [ 1978.689910]  evl_sleep_on_locked+0x629/0x730
Dec  2 12:37:54 10.243.0.116 [ 1978.690005]  ? evl_sleep_on_locked+0x5/0x730
Dec  2 12:37:54 10.243.0.116 [ 1978.690430]  evl_sleep_on+0x7b/0xb0
Dec  2 12:37:54 10.243.0.116 [ 1978.691029]  evl_delay+0x48/0xf0
Dec  2 12:37:54 10.243.0.116 [ 1978.691266]  clock_oob_ioctl+0x182/0x2d0
Dec  2 12:37:54 10.243.0.116 [ 1978.691573]  ? clock_common_ioctl+0x5e0/0x5e0
Dec  2 12:37:54 10.243.0.116 [ 1978.691952]  ? clock_oob_ioctl+0x5/0x2d0
Dec  2 12:37:54 10.243.0.116 [ 1978.692325]  EVL_ioctl+0x86/0x100
Dec  2 12:37:54 10.243.0.116 [ 1978.692933]  do_oob_syscall+0x5b1/0x6d0
Dec  2 12:37:54 10.243.0.116 [ 1978.693149]  ? prepare_for_signal+0x180/0x180
Dec  2 12:37:54 10.243.0.116 [ 1978.693315]  ? do_oob_syscall+0x5/0x6d0
Dec  2 12:37:54 10.243.0.116 [ 1978.693445]  ? handle_pipelined_syscall+0x840/0x840
Dec  2 12:37:54 10.243.0.116 [ 1978.694034]  handle_oob_syscall+0x18e/0x230
Dec  2 12:37:54 10.243.0.116 [ 1978.694236]  ? handle_pipelined_syscall+0x840/0x840
Dec  2 12:37:54 10.243.0.116 [ 1978.694804]  ? handle_oob_syscall+0x5/0x230
Dec  2 12:37:54 10.243.0.116 [ 1978.695258]  pipeline_syscall+0xac/0x1c0
Dec  2 12:37:55 10.243.0.116 [ 1978.695784]  syscall_enter_from_user_mode+0x5f/0x130
Dec  2 12:37:55 10.243.0.116 [ 1978.696051]  do_syscall_64+0x1d/0xa0
Dec  2 12:37:55 10.243.0.116 [ 1978.696154]  entry_SYSCALL_64_after_hwframe+0x61/0xcb
Dec  2 12:37:55 10.243.0.116 [ 1978.696191] RIP: 0033:0x7f302d4b1b9b
Dec  2 12:37:55 10.243.0.116 [ 1978.696239] Code: Unable to access opcode bytes at RIP 0x7f302d4b1b71.
Dec  2 12:37:55 10.243.0.116 [ 1978.696255] RSP: 002b:00007f3013fef5a0 EFLAGS: 00000246 ORIG_RAX: 000000000000009d
Dec  2 12:37:55 10.243.0.116 [ 1978.696288] RAX: ffffffffffffffda RBX: 00007f3013fef630 RCX: 00007f302d4b1b9b
Dec  2 12:37:55 10.243.0.116 [ 1978.696306] RDX: 0000000040106300 RSI: 0000000000000004 RDI: 0000000010000002
Dec  2 12:37:55 10.243.0.116 [ 1978.696323] RBP: 0000000040106300 R08: 0000000000000000 R09: 00007ffd9df03080
Dec  2 12:37:55 10.243.0.116 [ 1978.696353] R10: 00007f3013fef630 R11: 0000000000000246 R12: 0000000000000004
Dec  2 12:37:55 10.243.0.116 [ 1978.696390] R13: 0000000001001000 R14: 0000000000000000 R15: 00007f3013fff700
Dec  2 12:37:55 10.243.0.116 [ 1978.697564]  </TASK>
Dec  2 12:38:02 10.243.0.116 [ 1988.448303] watchdog: BUG: soft lockup - CPU#0 stuck for 257s! [TxStartup:3574]
Dec  2 12:38:02 10.243.0.116 [ 1988.486820] Modules linked in: xt_CHECKSUM xt_MASQUERADE tun netconsole ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 xt_conntrack ebtable_nat ebtable_broute ip6table_nat ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_nat iptable_mangle iptable_security iptable_raw nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 libcrc32c ip_set nfnetlink ebtable_filter ebtables rfkill ip6table_filter ip6_tables iptable_filter sunrpc dm_mirror dm_region_hash dm_log dm_mod x86_pkg_temp_thermal intel_powerclamp coretemp gpio_ich iTCO_wdt iTCO_vendor_support kvm_intel mxm_wmi kvm irqbypass crct10dif_pclmul crc32_pclmul ghash_clmulni_intel aesni_intel crypto_simd cryptd rapl intel_cstate pcspkr sg i2c_i801 ftdi_sio mei_me i2c_smbus joydev input_leds intel_pch_thermal mei lpc_ich mfd_core ioatdma wmi ip_tables ext4 mbcache jbd2 ast drm_vram_helper drm_kms_helper syscopyarea ixgbe sysfillrect sd_mod sr_mod sysimgblt fb_sys_fops cdrom t10_pi drm_ttm_helper ttm mdio drm igb ah
Dec  2 12:38:02 10.243.0.116 [ 1988.500227]  libahci libata crc32c_intel ptp pps_core i2c_algo_bit dca fuse
Dec  2 12:38:02 10.243.0.116 [ 1988.845134] irq event stamp: 24564
Dec  2 12:38:02 10.243.0.116 [ 1988.881914] hardirqs last  enabled at (24563): [<ffffffff8d1d44ad>] sync_current_irq_stage+0x45d/0x500
Dec  2 12:38:02 10.243.0.116 [ 1988.920476] hardirqs last disabled at (24564): [<ffffffff8d49923b>] handle_mm_fault+0x48b/0x520
Dec  2 12:38:02 10.243.0.116 [ 1988.958723] softirqs last  enabled at (0): [<ffffffff8d0ca587>] copy_process+0x1307/0x3770
Dec  2 12:38:02 10.243.0.116 [ 1988.996927] softirqs last disabled at (0): [<0000000000000000>] 0x0
Dec  2 12:38:02 10.243.0.116 [ 1989.034749] CPU: 0 PID: 3574 Comm: TxStartup Tainted: G    B   W    L    5.15.77evl-g5e681f841fc3-dirty #3
Dec  2 12:38:02 10.243.0.116 [ 1989.073974] Hardware name: Supermicro Super Server/X10SDV-6C+-TLN4F, BIOS 2.1 11/22/2019
Dec  2 12:38:02 10.243.0.116 [ 1989.113043] IRQ stage: Linux
Dec  2 12:38:02 10.243.0.116 [ 1989.151073] RIP: 0010:proxy_set_next_ktime+0x77/0x90
Dec  2 12:38:03 10.243.0.116 [ 1989.189820] Code: 74 37 49 8d 7c 24 30 e8 d7 2d 1d 00 49 8b 44 24 30 4c 89 e7 e8 da 42 0b 01 48 8d 34 03 31 d2 4c 89 ef e8 5c 14 00 00 41 56 9d <31> c0 5b 41 5c 41 5d 41 5e 5d c3 cc cc cc cc e8 25 f5 eb ff eb d9
Dec  2 12:38:03 10.243.0.116 [ 1989.272838] RSP: 0018:ffff88841e209e80 EFLAGS: 00000283 ORIG_RAX: 0000000000000000
Dec  2 12:38:03 10.243.0.116 [ 1989.316267] RAX: 0000000000000000 RBX: 0000000000000000 RCX: 0000000000000000
Dec  2 12:38:03 10.243.0.116 [ 1989.359816] RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000000
Dec  2 12:38:03 10.243.0.116 [ 1989.403347] RBP: ffff88841e209ea0 R08: 0000000000000000 R09: 0000000000000000
Dec  2 12:38:03 10.243.0.116 [ 1989.447076] R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000000
Dec  2 12:38:03 10.243.0.116 [ 1989.490965] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000
Dec  2 12:38:03 10.243.0.116 [ 1989.534863] FS:  00007f301ae2e700(0000) GS:ffff88841e200000(0000) knlGS:0000000000000000
Dec  2 12:38:03 10.243.0.116 [ 1989.579489] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Dec  2 12:38:03 10.243.0.116 [ 1989.623810] CR2: 00007f1e4033b660 CR3: 0000000137d08006 CR4: 00000000003706f0
Dec  2 12:38:03 10.243.0.116 [ 1989.668939] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Dec  2 12:38:03 10.243.0.116 [ 1989.714670] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Dec  2 12:38:03 10.243.0.116 [ 1989.760602] Call Trace:
Dec  2 12:38:03 10.243.0.116 [ 1989.805591]  <IRQ>
Dec  2 12:38:03 10.243.0.116 [ 1989.850632]  ? __hrtimer_run_queues+0x5/0x780
Dec  2 12:38:03 10.243.0.116 [ 1989.896826]  ? hrtimer_interrupt+0x1aa/0x360
Dec  2 12:38:03 10.243.0.116 [ 1989.942735]  ? hrtimer_interrupt+0x5/0x360
Dec  2 12:38:03 10.243.0.116 [ 1989.988730]  ? proxy_irq_handler+0x37/0x50
Dec  2 12:38:03 10.243.0.116 [ 1990.034125]  ? handle_synthetic_irq+0xec/0x320
Dec  2 12:38:03 10.243.0.116 [ 1990.079951]  ? do_irq_inband+0x2f/0x40
Dec  2 12:38:03 10.243.0.116 [ 1990.125092]  ? arch_do_IRQ_pipelined+0xcd/0x5b0
Dec  2 12:38:03 10.243.0.116 [ 1990.170390]  </IRQ>
Dec  2 12:38:04 10.243.0.116 [ 1990.214494]  <TASK>
Dec  2 12:38:04 10.243.0.116 [ 1990.258586]  ? sync_current_irq_stage+0x2e3/0x500
Dec  2 12:38:04 10.243.0.116 [ 1990.303823]  ? sync_irq_stage+0x1af/0x1e0
Dec  2 12:38:04 10.243.0.116 [ 1990.347734]  ? synchronize_pipeline+0x8b/0xe0
Dec  2 12:38:04 10.243.0.116 [ 1990.391415]  ? handle_irq_pipelined_finish+0xf6/0x240
Dec  2 12:38:04 10.243.0.116 [ 1990.435061]  ? arch_pipeline_entry+0x81/0x110
Dec  2 12:38:04 10.243.0.116 [ 1990.477843]  ? sysvec_apic_timer_interrupt+0xe/0x20
Dec  2 12:38:04 10.243.0.116 [ 1990.519893]  ? asm_sysvec_apic_timer_interrupt+0x1b/0x20
Dec  2 12:38:04 10.243.0.116 [ 1990.562628]  ? smp_call_function_many_cond+0x1c7/0x4e0
Dec  2 12:38:04 10.243.0.116 [ 1990.604399]  ? smp_call_function_many_cond+0x1bf/0x4e0
Dec  2 12:38:04 10.243.0.116 [ 1990.645509]  ? smp_call_function_many_cond+0x1c7/0x4e0
Dec  2 12:38:04 10.243.0.116 [ 1990.686011]  ? leave_mm+0x40/0x40
Dec  2 12:38:04 10.243.0.116 [ 1990.725535]  ? x86_configure_nx+0x70/0x70
Dec  2 12:38:04 10.243.0.116 [ 1990.764945]  ? on_each_cpu_cond_mask+0x29/0x40
Dec  2 12:38:04 10.243.0.116 [ 1990.803740]  ? native_flush_tlb_multi+0xbe/0x250
Dec  2 12:38:04 10.243.0.116 [ 1990.842318]  ? flush_tlb_mm_range+0x136/0x1f0
Dec  2 12:38:04 10.243.0.116 [ 1990.880608]  ? change_protection_range+0x8bd/0xea0
Dec  2 12:38:04 10.243.0.116 [ 1990.920540]  ? prot_none_pte_entry+0x70/0x70
Dec  2 12:38:05 10.243.0.116 [ 1990.958070]  ? 0xffffffffc0b38099
Dec  2 12:38:05 10.243.0.116 [ 1990.996945] igb 0000:05:00.0: Detected Tx Unit Hang#012[ 1990.996945]   Tx Queue             <7>#012[ 1990.996945]   TDH                  <d8>#012[ 1990.996945]   TDT                  <d8>#012[ 1990.996945]   next_to_use          <d8>#012[ 1990.996945]   next_to_clean        <6d>#012[ 1990.996945] buffer_info[next_to_clean]#012[ 1990.996945]   time_stamp           <100199d16>#012[ 1990.996945]   next_to_watch        <0000000088ad01cc>#012[ 1990.996945]   jiffies              <10019be3e>#012[ 1990.996945]   desc.status          <13c001>
Dec  2 12:38:05 10.243.0.116 [ 1991.695232]  ? change_protection_range+0x5/0xea0
Dec  2 12:38:05 10.243.0.116 [ 1991.726494]  ? change_protection+0x50/0x90
Dec  2 12:38:05 10.243.0.116 [ 1991.757296]  ? mprotect_fixup+0x2c5/0x490
Dec  2 12:38:05 10.243.0.116 [ 1991.788113]  ? change_protection+0x90/0x90
Dec  2 12:38:05 10.243.0.116 [ 1991.817941]  ? mprotect_fixup+0x5/0x490
Dec  2 12:38:05 10.243.0.116 [ 1991.847781]  ? do_mprotect_pkey+0x32a/0x590
Dec  2 12:38:05 10.243.0.116 [ 1991.877045]  ? mprotect_fixup+0x490/0x490
Dec  2 12:38:05 10.243.0.116 [ 1991.905835]  ? do_mprotect_pkey+0x5/0x590
Dec  2 12:38:05 10.243.0.116 [ 1991.934730]  ? __x64_sys_mprotect+0x48/0x60
Dec  2 12:38:05 10.243.0.116 [ 1991.963315]  ? do_syscall_64+0x4c/0xa0
Dec  2 12:38:05 10.243.0.116 [ 1991.991732]  ? entry_SYSCALL_64_after_hwframe+0x61/0xcb
Dec  2 12:38:05 10.243.0.116 [ 1992.022040]  </TASK>
Dec  2 12:38:18 10.243.0.116 [ 2004.448152] watchdog: BUG: soft lockup - CPU#2 stuck for 97s! [kworker/2:2:259]
Dec  2 12:38:18 10.243.0.116 [ 2004.469650] Modules linked in: xt_CHECKSUM xt_MASQUERADE tun netconsole ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 xt_conntrack ebtable_nat ebtable_broute ip6table_nat ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_nat iptable_mangle iptable_security iptable_raw nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 libcrc32c ip_set nfnetlink ebtable_filter ebtables rfkill ip6table_filter ip6_tables iptable_filter sunrpc dm_mirror dm_region_hash dm_log dm_mod x86_pkg_temp_thermal intel_powerclamp coretemp gpio_ich iTCO_wdt iTCO_vendor_support kvm_intel mxm_wmi kvm irqbypass crct10dif_pclmul crc32_pclmul ghash_clmulni_intel aesni_intel crypto_simd cryptd rapl intel_cstate pcspkr sg i2c_i801 ftdi_sio mei_me i2c_smbus joydev input_leds intel_pch_thermal mei lpc_ich mfd_core ioatdma wmi ip_tables ext4 mbcache jbd2 ast drm_vram_helper drm_kms_helper syscopyarea ixgbe sysfillrect sd_mod sr_mod sysimgblt fb_sys_fops cdrom t10_pi drm_ttm_helper ttm mdio drm igb ah
Dec  2 12:38:18 10.243.0.116 [ 2004.478924]  libahci libata crc32c_intel ptp pps_core i2c_algo_bit dca fuse
Dec  2 12:38:18 10.243.0.116 [ 2004.681952] irq event stamp: 23654
Dec  2 12:38:18 10.243.0.116 [ 2004.705331] hardirqs last  enabled at (23653): [<ffffffff8e1bd0d7>] _raw_spin_unlock_irq+0x27/0x40
Dec  2 12:38:18 10.243.0.116 [ 2004.730286] hardirqs last disabled at (23654): [<ffffffff8e1afe3d>] __schedule+0x90d/0x1160
Dec  2 12:38:18 10.243.0.116 [ 2004.755211] softirqs last  enabled at (23540): [<ffffffff8e400332>] __do_softirq+0x332/0x598
Dec  2 12:38:18 10.243.0.116 [ 2004.780218] softirqs last disabled at (23529): [<ffffffff8d0dc80b>] irq_exit_rcu+0xab/0x180
Dec  2 12:38:18 10.243.0.116 [ 2004.805436] CPU: 2 PID: 259 Comm: kworker/2:2 Tainted: G    B   W    L    5.15.77evl-g5e681f841fc3-dirty #3
Dec  2 12:38:18 10.243.0.116 [ 2004.831372] Hardware name: Supermicro Super Server/X10SDV-6C+-TLN4F, BIOS 2.1 11/22/2019
Dec  2 12:38:18 10.243.0.116 [ 2004.857397] IRQ stage: Linux
Dec  2 12:38:18 10.243.0.116 [ 2004.882945] Workqueue: rcu_gp wait_rcu_exp_gp
Dec  2 12:38:18 10.243.0.116 [ 2004.908295] RIP: 0010:proxy_set_next_ktime+0x77/0x90
Dec  2 12:38:18 10.243.0.116 [ 2004.934098] Code: 74 37 49 8d 7c 24 30 e8 d7 2d 1d 00 49 8b 44 24 30 4c 89 e7 e8 da 42 0b 01 48 8d 34 03 31 d2 4c 89 ef e8 5c 14 00 00 41 56 9d <31> c0 5b 41 5c 41 5d 41 5e 5d c3 cc cc cc cc e8 25 f5 eb ff eb d9
Dec  2 12:38:18 10.243.0.116 [ 2004.989441] RSP: 0018:ffff88841e309e80 EFLAGS: 00000206 ORIG_RAX: 0000000000000000
Dec  2 12:38:18 10.243.0.116 [ 2005.018434] RAX: 0000000000000000 RBX: 0000000000000000 RCX: 0000000000000000
Dec  2 12:38:18 10.243.0.116 [ 2005.047597] RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000000
Dec  2 12:38:18 10.243.0.116 [ 2005.076996] RBP: ffff88841e309ea0 R08: 0000000000000000 R09: 0000000000000000
Dec  2 12:38:18 10.243.0.116 [ 2005.105923] R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000000
Dec  2 12:38:18 10.243.0.116 [ 2005.135182] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000
Dec  2 12:38:18 10.243.0.116 [ 2005.167057] FS:  0000000000000000(0000) GS:ffff88841e300000(0000) knlGS:0000000000000000
Dec  2 12:38:18 10.243.0.116 [ 2005.197851] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Dec  2 12:38:19 10.243.0.116 [ 2005.227899] CR2: 00007f20be593000 CR3: 000000012c0f0004 CR4: 00000000003706e0
Dec  2 12:38:19 10.243.0.116 [ 2005.258724] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Dec  2 12:38:19 10.243.0.116 [ 2005.289997] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Dec  2 12:38:19 10.243.0.116 [ 2005.321112] Call Trace:
Dec  2 12:38:19 10.243.0.116 [ 2005.352747]  <IRQ>
Dec  2 12:38:19 10.243.0.116 [ 2005.383429]  ? __hrtimer_run_queues+0x5/0x780
Dec  2 12:38:19 10.243.0.116 [ 2005.416074]  ? hrtimer_interrupt+0x1aa/0x360
Dec  2 12:38:19 10.243.0.116 [ 2005.447140]  ? hrtimer_interrupt+0x5/0x360
Dec  2 12:38:19 10.243.0.116 [ 2005.478381]  ? proxy_irq_handler+0x37/0x50
Dec  2 12:38:19 10.243.0.116 [ 2005.509129]  ? handle_synthetic_irq+0xec/0x320
Dec  2 12:38:19 10.243.0.116 [ 2005.539900]  ? do_irq_inband+0x2f/0x40
Dec  2 12:38:19 10.243.0.116 [ 2005.570378]  ? arch_do_IRQ_pipelined+0xcd/0x5b0
Dec  2 12:38:19 10.243.0.116 [ 2005.601157]  </IRQ>
Dec  2 12:38:19 10.243.0.116 [ 2005.631146]  <TASK>
Dec  2 12:38:19 10.243.0.116 [ 2005.661412]  ? sync_current_irq_stage+0x2e3/0x500
Dec  2 12:38:19 10.243.0.116 [ 2005.693687]  ? sync_irq_stage+0x1af/0x1e0
Dec  2 12:38:19 10.243.0.116 [ 2005.724231]  ? synchronize_pipeline+0x8b/0xe0
Dec  2 12:38:19 10.243.0.116 [ 2005.754697]  ? handle_irq_pipelined_finish+0xf6/0x240
Dec  2 12:38:19 10.243.0.116 [ 2005.785126]  ? arch_pipeline_entry+0x81/0x110
Dec  2 12:38:19 10.243.0.116 [ 2005.815135]  ? sysvec_apic_timer_interrupt+0xe/0x20
Dec  2 12:38:19 10.243.0.116 [ 2005.844352]  ? asm_sysvec_apic_timer_interrupt+0x1b/0x20
Dec  2 12:38:19 10.243.0.116 [ 2005.873421]  ? rcu_all_qs+0xa0/0xa0
Dec  2 12:38:19 10.243.0.116 [ 2005.903113]  ? smp_call_function_single+0x118/0x210
Dec  2 12:38:19 10.243.0.116 [ 2005.931790]  ? smp_call_function_single+0x110/0x210
Dec  2 12:38:19 10.243.0.116 [ 2005.960105]  ? 0xffffffffc0b38099
Dec  2 12:38:19 10.243.0.116 [ 2005.987507]  ? generic_exec_single+0x140/0x140
Dec  2 12:38:19 10.243.0.116 [ 2006.015303]  ? rcu_all_qs+0xa0/0xa0
Dec  2 12:38:19 10.243.0.116 [ 2006.042180]  ? smp_call_function_single+0x5/0x210
Dec  2 12:38:19 10.243.0.116 [ 2006.069846]  ? sync_rcu_exp_select_node_cpus+0x3df/0x5c0
Dec  2 12:38:19 10.243.0.116 [ 2006.097291]  ? sync_rcu_exp_select_cpus+0x2c5/0x5c0
Dec  2 12:38:19 10.243.0.116 [ 2006.124300]  ? wait_rcu_exp_gp+0x1f/0x30
Dec  2 12:38:19 10.243.0.116 [ 2006.150269]  ? process_one_work+0x525/0xa30
Dec  2 12:38:19 10.243.0.116 [ 2006.176587]  ? pwq_dec_nr_in_flight+0x120/0x120
Dec  2 12:38:19 10.243.0.116 [ 2006.205797]  ? process_one_work+0x5/0xa30
Dec  2 12:38:20 10.243.0.116 [ 2006.231419]  ? worker_thread+0x57/0x590
Dec  2 12:38:20 10.243.0.116 [ 2006.256884]  ? process_one_work+0xa30/0xa30
Dec  2 12:38:20 10.243.0.116 [ 2006.282006]  ? kthread+0x1f3/0x220
Dec  2 12:38:20 10.243.0.116 [ 2006.307148]  ? set_kthread_struct+0x90/0x90
Dec  2 12:38:20 10.243.0.116 [ 2006.331156]  ? ret_from_fork+0x22/0x30
Dec  2 12:38:20 10.243.0.116 [ 2006.354369]  </TASK>
Dec  2 12:38:30 10.243.0.116 [ 2016.448433] watchdog: BUG: soft lockup - CPU#0 stuck for 283s! [TxStartup:3574]
Dec  2 12:38:30 10.243.0.116 [ 2016.481648] Modules linked in: xt_CHECKSUM xt_MASQUERADE tun netconsole ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 xt_conntrack ebtable_nat ebtable_broute ip6table_nat ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_nat iptable_mangle iptable_security iptable_raw nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 libcrc32c ip_set nfnetlink ebtable_filter ebtables rfkill ip6table_filter ip6_tables iptable_filter sunrpc dm_mirror dm_region_hash dm_log dm_mod x86_pkg_temp_thermal intel_powerclamp coretemp gpio_ich iTCO_wdt iTCO_vendor_support kvm_intel mxm_wmi kvm irqbypass crct10dif_pclmul crc32_pclmul ghash_clmulni_intel aesni_intel crypto_simd cryptd rapl intel_cstate pcspkr sg i2c_i801 ftdi_sio mei_me i2c_smbus joydev input_leds intel_pch_thermal mei lpc_ich mfd_core ioatdma wmi ip_tables ext4 mbcache jbd2 ast drm_vram_helper drm_kms_helper syscopyarea ixgbe sysfillrect sd_mod sr_mod sysimgblt fb_sys_fops cdrom t10_pi drm_ttm_helper ttm mdio drm igb ah
Dec  2 12:38:30 10.243.0.116 [ 2016.495055]  libahci libata crc32c_intel ptp pps_core i2c_algo_bit dca fuse
Dec  2 12:38:30 10.243.0.116 [ 2016.802972] irq event stamp: 24564
Dec  2 12:38:30 10.243.0.116 [ 2016.837112] hardirqs last  enabled at (24563): [<ffffffff8d1d44ad>] sync_current_irq_stage+0x45d/0x500
Dec  2 12:38:30 10.243.0.116 [ 2016.873498] hardirqs last disabled at (24564): [<ffffffff8d49923b>] handle_mm_fault+0x48b/0x520
Dec  2 12:38:30 10.243.0.116 [ 2016.909962] softirqs last  enabled at (0): [<ffffffff8d0ca587>] copy_process+0x1307/0x3770
Dec  2 12:38:30 10.243.0.116 [ 2016.946665] softirqs last disabled at (0): [<0000000000000000>] 0x0
Dec  2 12:38:30 10.243.0.116 [ 2016.983174] CPU: 0 PID: 3574 Comm: TxStartup Tainted: G    B   W    L    5.15.77evl-g5e681f841fc3-dirty #3
Dec  2 12:38:30 10.243.0.116 [ 2017.021006] Hardware name: Supermicro Super Server/X10SDV-6C+-TLN4F, BIOS 2.1 11/22/2019
Dec  2 12:38:30 10.243.0.116 [ 2017.058682] IRQ stage: Linux
Dec  2 12:38:30 10.243.0.116 [ 2017.095147] RIP: 0010:proxy_set_next_ktime+0x77/0x90
Dec  2 12:38:30 10.243.0.116 [ 2017.132365] Code: 74 37 49 8d 7c 24 30 e8 d7 2d 1d 00 49 8b 44 24 30 4c 89 e7 e8 da 42 0b 01 48 8d 34 03 31 d2 4c 89 ef e8 5c 14 00 00 41 56 9d <31> c0 5b 41 5c 41 5d 41 5e 5d c3 cc cc cc cc e8 25 f5 eb ff eb d9
Dec  2 12:38:31 10.243.0.116 [ 2017.212507] RSP: 0018:ffff88841e209e80 EFLAGS: 00000287 ORIG_RAX: 0000000000000000
Dec  2 12:38:31 10.243.0.116 [ 2017.254418] RAX: 0000000000000000 RBX: 0000000000000000 RCX: 0000000000000000
Dec  2 12:38:31 10.243.0.116 [ 2017.296412] RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000000
Dec  2 12:38:31 10.243.0.116 [ 2017.338433] RBP: ffff88841e209ea0 R08: 0000000000000000 R09: 0000000000000000
Dec  2 12:38:31 10.243.0.116 [ 2017.380581] R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000000
Dec  2 12:38:31 10.243.0.116 [ 2017.422826] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000
Dec  2 12:38:31 10.243.0.116 [ 2017.465061] FS:  00007f301ae2e700(0000) GS:ffff88841e200000(0000) knlGS:0000000000000000
Dec  2 12:38:31 10.243.0.116 [ 2017.507853] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Dec  2 12:38:31 10.243.0.116 [ 2017.550368] CR2: 00007f1e4033b660 CR3: 0000000137d08006 CR4: 00000000003706f0
Dec  2 12:38:31 10.243.0.116 [ 2017.593586] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Dec  2 12:38:31 10.243.0.116 [ 2017.637571] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Dec  2 12:38:31 10.243.0.116 [ 2017.681765] Call Trace:
Dec  2 12:38:31 10.243.0.116 [ 2017.724913]  <IRQ>
Dec  2 12:38:31 10.243.0.116 [ 2017.767999]  ? __hrtimer_run_queues+0x5/0x780
Dec  2 12:38:31 10.243.0.116 [ 2017.812204]  ? hrtimer_interrupt+0x1aa/0x360
Dec  2 12:38:31 10.243.0.116 [ 2017.855972]  ? hrtimer_interrupt+0x5/0x360
Dec  2 12:38:31 10.243.0.116 [ 2017.899870]  ? proxy_irq_handler+0x37/0x50
Dec  2 12:38:31 10.243.0.116 [ 2017.943316]  ? handle_synthetic_irq+0xec/0x320
Dec  2 12:38:31 10.243.0.116 [ 2017.987103]  ? do_irq_inband+0x2f/0x40
Dec  2 12:38:31 10.243.0.116 [ 2018.030390]  ? arch_do_IRQ_pipelined+0xcd/0x5b0
Dec  2 12:38:31 10.243.0.116 [ 2018.073792]  </IRQ>
Dec  2 12:38:31 10.243.0.116 [ 2018.116430]  <TASK>
Dec  2 12:38:31 10.243.0.116 [ 2018.159197]  ? sync_current_irq_stage+0x2e3/0x500
Dec  2 12:38:32 10.243.0.116 [ 2018.203351]  ? sync_irq_stage+0x1af/0x1e0
Dec  2 12:38:32 10.243.0.116 [ 2018.246629]  ? synchronize_pipeline+0x8b/0xe0
Dec  2 12:38:32 10.243.0.116 [ 2018.289892]  ? handle_irq_pipelined_finish+0xf6/0x240
Dec  2 12:38:32 10.243.0.116 [ 2018.333481]  ? arch_pipeline_entry+0x81/0x110
Dec  2 12:38:32 10.243.0.116 [ 2018.376213]  ? sysvec_apic_timer_interrupt+0xe/0x20
Dec  2 12:38:32 10.243.0.116 [ 2018.418212]  ? asm_sysvec_apic_timer_interrupt+0x1b/0x20
Dec  2 12:38:32 10.243.0.116 [ 2018.460880]  ? smp_call_function_many_cond+0x1c7/0x4e0
Dec  2 12:38:32 10.243.0.116 [ 2018.502596]  ? smp_call_function_many_cond+0x1bf/0x4e0
Dec  2 12:38:32 10.243.0.116 [ 2018.543650]  ? smp_call_function_many_cond+0x1c7/0x4e0
Dec  2 12:38:32 10.243.0.116 [ 2018.584105]  ? leave_mm+0x40/0x40
Dec  2 12:38:32 10.243.0.116 [ 2018.623640]  ? x86_configure_nx+0x70/0x70
Dec  2 12:38:32 10.243.0.116 [ 2018.662997]  ? on_each_cpu_cond_mask+0x29/0x40
Dec  2 12:38:32 10.243.0.116 [ 2018.701747]  ? native_flush_tlb_multi+0xbe/0x250
Dec  2 12:38:32 10.243.0.116 [ 2018.740219]  ? flush_tlb_mm_range+0x136/0x1f0
Dec  2 12:38:32 10.243.0.116 [ 2018.778401]  ? change_protection_range+0x8bd/0xea0
Dec  2 12:38:32 10.243.0.116 [ 2018.818224]  ? prot_none_pte_entry+0x70/0x70
Dec  2 12:38:32 10.243.0.116 [ 2018.855649]  ? 0xffffffffc0b38099
Dec  2 12:38:32 10.243.0.116 [ 2018.892684]  ? change_protection_range+0x5/0xea0
Dec  2 12:38:32 10.243.0.116 [ 2018.929924]  ? change_protection+0x50/0x90
Dec  2 12:38:32 10.243.0.116 [ 2018.966241]  ? mprotect_fixup+0x2c5/0x490
Dec  2 12:38:32 10.243.0.116 [ 2019.002703]  ? change_protection+0x90/0x90
Dec  2 12:38:32 10.243.0.116 [ 2019.038318]  ? mprotect_fixup+0x5/0x490
Dec  2 12:38:32 10.243.0.116 [ 2019.074335]  ? do_mprotect_pkey+0x32a/0x590
Dec  2 12:38:32 10.243.0.116 [ 2019.108837]  ? mprotect_fixup+0x490/0x490
Dec  2 12:38:32 10.243.0.116 [ 2019.140926]  ? do_mprotect_pkey+0x5/0x590
Dec  2 12:38:32 10.243.0.116 [ 2019.173404]  ? __x64_sys_mprotect+0x48/0x60
Dec  2 12:38:33 10.243.0.116 [ 2019.205056]  ? do_syscall_64+0x4c/0xa0
Dec  2 12:38:33 10.243.0.116 [ 2019.236399]  ? entry_SYSCALL_64_after_hwframe+0x61/0xcb
Dec  2 12:38:33 10.243.0.116 [ 2019.269453]  </TASK>
Dec  2 12:38:46 10.243.0.116 [ 2032.448271] watchdog: BUG: soft lockup - CPU#2 stuck for 123s! [kworker/2:2:259]
Dec  2 12:38:46 10.243.0.116 [ 2032.471456] Modules linked in: xt_CHECKSUM xt_MASQUERADE tun netconsole ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 xt_conntrack ebtable_nat ebtable_broute ip6table_nat ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_nat iptable_mangle iptable_security iptable_raw nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 libcrc32c ip_set nfnetlink ebtable_filter ebtables rfkill ip6table_filter ip6_tables iptable_filter sunrpc dm_mirror dm_region_hash dm_log dm_mod x86_pkg_temp_thermal intel_powerclamp coretemp gpio_ich iTCO_wdt iTCO_vendor_support kvm_intel mxm_wmi kvm irqbypass crct10dif_pclmul crc32_pclmul ghash_clmulni_intel aesni_intel crypto_simd cryptd rapl intel_cstate pcspkr sg i2c_i801 ftdi_sio mei_me i2c_smbus joydev input_leds intel_pch_thermal mei lpc_ich mfd_core ioatdma wmi ip_tables ext4 mbcache jbd2 ast drm_vram_helper drm_kms_helper syscopyarea ixgbe sysfillrect sd_mod sr_mod sysimgblt fb_sys_fops cdrom t10_pi drm_ttm_helper ttm mdio drm igb ah
Dec  2 12:38:46 10.243.0.116 [ 2032.480906]  libahci libata crc32c_intel ptp pps_core i2c_algo_bit dca fuse
Dec  2 12:38:46 10.243.0.116 [ 2032.688557] irq event stamp: 23654
Dec  2 12:38:46 10.243.0.116 [ 2032.712383] hardirqs last  enabled at (23653): [<ffffffff8e1bd0d7>] _raw_spin_unlock_irq+0x27/0x40
Dec  2 12:38:46 10.243.0.116 [ 2032.737423] hardirqs last disabled at (23654): [<ffffffff8e1afe3d>] __schedule+0x90d/0x1160
Dec  2 12:38:46 10.243.0.116 [ 2032.762756] softirqs last  enabled at (23540): [<ffffffff8e400332>] __do_softirq+0x332/0x598
Dec  2 12:38:46 10.243.0.116 [ 2032.788209] softirqs last disabled at (23529): [<ffffffff8d0dc80b>] irq_exit_rcu+0xab/0x180
Dec  2 12:38:46 10.243.0.116 [ 2032.813367] CPU: 2 PID: 259 Comm: kworker/2:2 Tainted: G    B   W    L    5.15.77evl-g5e681f841fc3-dirty #3
Dec  2 12:38:46 10.243.0.116 [ 2032.839324] Hardware name: Supermicro Super Server/X10SDV-6C+-TLN4F, BIOS 2.1 11/22/2019
Dec  2 12:38:46 10.243.0.116 [ 2032.865657] IRQ stage: Linux
Dec  2 12:38:46 10.243.0.116 [ 2032.890811] Workqueue: rcu_gp wait_rcu_exp_gp
Dec  2 12:38:46 10.243.0.116 [ 2032.916478] RIP: 0010:proxy_set_next_ktime+0x77/0x90
Dec  2 12:38:46 10.243.0.116 [ 2032.943323] Code: 74 37 49 8d 7c 24 30 e8 d7 2d 1d 00 49 8b 44 24 30 4c 89 e7 e8 da 42 0b 01 48 8d 34 03 31 d2 4c 89 ef e8 5c 14 00 00 41 56 9d <31> c0 5b 41 5c 41 5d 41 5e 5d c3 cc cc cc cc e8 25 f5 eb ff eb d9
Dec  2 12:38:46 10.243.0.116 [ 2032.999052] RSP: 0018:ffff88841e309e80 EFLAGS: 00000202 ORIG_RAX: 0000000000000000
Dec  2 12:38:46 10.243.0.116 [ 2033.028069] RAX: 0000000000000000 RBX: 0000000000000000 RCX: 0000000000000000
Dec  2 12:38:46 10.243.0.116 [ 2033.057407] RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000000
Dec  2 12:38:46 10.243.0.116 [ 2033.086414] RBP: ffff88841e309ea0 R08: 0000000000000000 R09: 0000000000000000
Dec  2 12:38:46 10.243.0.116 [ 2033.115474] R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000000
Dec  2 12:38:46 10.243.0.116 [ 2033.145306] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000
Dec  2 12:38:46 10.243.0.116 [ 2033.174851] FS:  0000000000000000(0000) GS:ffff88841e300000(0000) knlGS:0000000000000000
Dec  2 12:38:47 10.243.0.116 [ 2033.204688] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Dec  2 12:38:47 10.243.0.116 [ 2033.234293] CR2: 00007f20be593000 CR3: 000000012c0f0004 CR4: 00000000003706e0
Dec  2 12:38:47 10.243.0.116 [ 2033.264957] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Dec  2 12:38:47 10.243.0.116 [ 2033.295499] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Dec  2 12:38:47 10.243.0.116 [ 2033.326362] Call Trace:
Dec  2 12:38:47 10.243.0.116 [ 2033.356394]  <IRQ>
Dec  2 12:38:47 10.243.0.116 [ 2033.386713]  ? __hrtimer_run_queues+0x5/0x780
Dec  2 12:38:47 10.243.0.116 [ 2033.417675]  ? hrtimer_interrupt+0x1aa/0x360
Dec  2 12:38:47 10.243.0.116 [ 2033.449274]  ? hrtimer_interrupt+0x5/0x360
Dec  2 12:38:47 10.243.0.116 [ 2033.480308]  ? proxy_irq_handler+0x37/0x50
Dec  2 12:38:47 10.243.0.116 [ 2033.510448]  ? handle_synthetic_irq+0xec/0x320
Dec  2 12:38:47 10.243.0.116 [ 2033.541155]  ? do_irq_inband+0x2f/0x40
Dec  2 12:38:47 10.243.0.116 [ 2033.571516]  ? arch_do_IRQ_pipelined+0xcd/0x5b0
Dec  2 12:38:47 10.243.0.116 [ 2033.601974]  </IRQ>
Dec  2 12:38:47 10.243.0.116 [ 2033.631929]  <TASK>
Dec  2 12:38:47 10.243.0.116 [ 2033.668575]  ? sync_current_irq_stage+0x2e3/0x500
Dec  2 12:38:47 10.243.0.116 [ 2033.706766]  ? sync_irq_stage+0x1af/0x1e0
Dec  2 12:38:47 10.243.0.116 [ 2033.742236]  ? synchronize_pipeline+0x8b/0xe0
Dec  2 12:38:47 10.243.0.116 [ 2033.778014]  ? handle_irq_pipelined_finish+0xf6/0x240
Dec  2 12:38:47 10.243.0.116 [ 2033.814038]  ? arch_pipeline_entry+0x81/0x110
Dec  2 12:38:47 10.243.0.116 [ 2033.849252]  ? sysvec_apic_timer_interrupt+0xe/0x20
Dec  2 12:38:47 10.243.0.116 [ 2033.883939]  ? asm_sysvec_apic_timer_interrupt+0x1b/0x20
Dec  2 12:38:47 10.243.0.116 [ 2033.918267]  ? rcu_all_qs+0xa0/0xa0
Dec  2 12:38:47 10.243.0.116 [ 2033.952426]  ? smp_call_function_single+0x118/0x210
Dec  2 12:38:47 10.243.0.116 [ 2033.981022]  ? __asan_load4+0x44/0x90
Dec  2 12:38:47 10.243.0.116 [ 2034.009445]  ? smp_call_function_single+0x118/0x210
Dec  2 12:38:47 10.243.0.116 [ 2034.037307]  ? 0xffffffffc0b38099
Dec  2 12:38:47 10.243.0.116 [ 2034.064484]  ? generic_exec_single+0x140/0x140
Dec  2 12:38:47 10.243.0.116 [ 2034.092349]  ? rcu_all_qs+0xa0/0xa0
Dec  2 12:38:47 10.243.0.116 [ 2034.118587]  ? smp_call_function_single+0x5/0x210
Dec  2 12:38:47 10.243.0.116 [ 2034.145459]  ? sync_rcu_exp_select_node_cpus+0x3df/0x5c0
Dec  2 12:38:47 10.243.0.116 [ 2034.173190]  ? sync_rcu_exp_select_cpus+0x2c5/0x5c0
Dec  2 12:38:47 10.243.0.116 [ 2034.199875]  ? wait_rcu_exp_gp+0x1f/0x30
Dec  2 12:38:48 10.243.0.116 [ 2034.225646]  ? process_one_work+0x525/0xa30
Dec  2 12:38:48 10.243.0.116 [ 2034.251933]  ? pwq_dec_nr_in_flight+0x120/0x120
Dec  2 12:38:48 10.243.0.116 [ 2034.277454]  ? process_one_work+0x5/0xa30
Dec  2 12:38:48 10.243.0.116 [ 2034.303274]  ? worker_thread+0x57/0x590
Dec  2 12:38:48 10.243.0.116 [ 2034.328480]  ? process_one_work+0xa30/0xa30
Dec  2 12:38:48 10.243.0.116 [ 2034.353562]  ? kthread+0x1f3/0x220
Dec  2 12:38:48 10.243.0.116 [ 2034.377426]  ? set_kthread_struct+0x90/0x90
Dec  2 12:38:48 10.243.0.116 [ 2034.400395]  ? ret_from_fork+0x22/0x30
Dec  2 12:38:48 10.243.0.116 [ 2034.424165]  </TASK>
Dec  2 12:38:58 10.243.0.116 [ 2044.448559] watchdog: BUG: soft lockup - CPU#0 stuck for 310s! [TxStartup:3574]
Dec  2 12:38:58 10.243.0.116 [ 2044.481973] Modules linked in: xt_CHECKSUM xt_MASQUERADE tun netconsole ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 xt_conntrack ebtable_nat ebtable_broute ip6table_nat ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_nat iptable_mangle iptable_security iptable_raw nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 libcrc32c ip_set nfnetlink ebtable_filter ebtables rfkill ip6table_filter ip6_tables iptable_filter sunrpc dm_mirror dm_region_hash dm_log dm_mod x86_pkg_temp_thermal intel_powerclamp coretemp gpio_ich iTCO_wdt iTCO_vendor_support kvm_intel mxm_wmi kvm irqbypass crct10dif_pclmul crc32_pclmul ghash_clmulni_intel aesni_intel crypto_simd cryptd rapl intel_cstate pcspkr sg i2c_i801 ftdi_sio mei_me i2c_smbus joydev input_leds intel_pch_thermal mei lpc_ich mfd_core ioatdma wmi ip_tables ext4 mbcache jbd2 ast drm_vram_helper drm_kms_helper syscopyarea ixgbe sysfillrect sd_mod sr_mod sysimgblt fb_sys_fops cdrom t10_pi drm_ttm_helper ttm mdio drm igb ah
Dec  2 12:38:58 10.243.0.116 [ 2044.495384]  libahci libata crc32c_intel ptp pps_core i2c_algo_bit dca fuse
Dec  2 12:38:58 10.243.0.116 [ 2044.802287] irq event stamp: 24564
Dec  2 12:38:58 10.243.0.116 [ 2044.836791] hardirqs last  enabled at (24563): [<ffffffff8d1d44ad>] sync_current_irq_stage+0x45d/0x500
Dec  2 12:38:58 10.243.0.116 [ 2044.873349] hardirqs last disabled at (24564): [<ffffffff8d49923b>] handle_mm_fault+0x48b/0x520
Dec  2 12:38:58 10.243.0.116 [ 2044.909991] softirqs last  enabled at (0): [<ffffffff8d0ca587>] copy_process+0x1307/0x3770
Dec  2 12:38:58 10.243.0.116 [ 2044.946899] softirqs last disabled at (0): [<0000000000000000>] 0x0
Dec  2 12:38:58 10.243.0.116 [ 2044.983456] CPU: 0 PID: 3574 Comm: TxStartup Tainted: G    B   W    L    5.15.77evl-g5e681f841fc3-dirty #3
Dec  2 12:38:58 10.243.0.116 [ 2045.021303] Hardware name: Supermicro Super Server/X10SDV-6C+-TLN4F, BIOS 2.1 11/22/2019
Dec  2 12:38:58 10.243.0.116 [ 2045.059006] IRQ stage: Linux
Dec  2 12:38:58 10.243.0.116 [ 2045.095537] RIP: 0010:proxy_set_next_ktime+0x77/0x90
Dec  2 12:38:58 10.243.0.116 [ 2045.132758] Code: 74 37 49 8d 7c 24 30 e8 d7 2d 1d 00 49 8b 44 24 30 4c 89 e7 e8 da 42 0b 01 48 8d 34 03 31 d2 4c 89 ef e8 5c 14 00 00 41 56 9d <31> c0 5b 41 5c 41 5d 41 5e 5d c3 cc cc cc cc e8 25 f5 eb ff eb d9
Dec  2 12:38:59 10.243.0.116 [ 2045.213105] RSP: 0018:ffff88841e209e80 EFLAGS: 00000283 ORIG_RAX: 0000000000000000
Dec  2 12:38:59 10.243.0.116 [ 2045.255163] RAX: 0000000000000000 RBX: 0000000000000000 RCX: 0000000000000000
Dec  2 12:38:59 10.243.0.116 [ 2045.297371] RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000000
Dec  2 12:38:59 10.243.0.116 [ 2045.341932] igb 0000:05:00.0: Detected Tx Unit Hang#012[ 2045.341932]   Tx Queue             <7>#012[ 2045.341932]   TDH                  <b7>#012[ 2045.341932]   TDT                  <b7>#012[ 2045.341932]   next_to_use          <b7>#012[ 2045.341932]   next_to_clean        <58>#012[ 2045.341932] buffer_info[next_to_clean]#012[ 2045.341932]   time_stamp           <1001a2b9e>#012[ 2045.341932]   next_to_watch        <0000000099f2880c>#012[ 2045.341932]   jiffies              <1001a98fe>#012[ 2045.341932]   desc.status          <180001>
Dec  2 12:38:59 10.243.0.116 [ 2046.106908] RBP: ffff88841e209ea0 R08: 0000000000000000 R09: 0000000000000000
Dec  2 12:38:59 10.243.0.116 [ 2046.148531] R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000000
Dec  2 12:39:00 10.243.0.116 [ 2046.190377] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000
Dec  2 12:39:00 10.243.0.116 [ 2046.232226] FS:  00007f301ae2e700(0000) GS:ffff88841e200000(0000) knlGS:0000000000000000
Dec  2 12:39:00 10.243.0.116 [ 2046.274785] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Dec  2 12:39:00 10.243.0.116 [ 2046.317038] CR2: 00007f1e4033b660 CR3: 0000000137d08006 CR4: 00000000003706f0
Dec  2 12:39:00 10.243.0.116 [ 2046.360015] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Dec  2 12:39:00 10.243.0.116 [ 2046.403613] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Dec  2 12:39:00 10.243.0.116 [ 2046.447341] Call Trace:
Dec  2 12:39:00 10.243.0.116 [ 2046.490033]  <IRQ>
Dec  2 12:39:00 10.243.0.116 [ 2046.532902]  ? __hrtimer_run_queues+0x5/0x780
Dec  2 12:39:00 10.243.0.116 [ 2046.576758]  ? hrtimer_interrupt+0x1aa/0x360
Dec  2 12:39:00 10.243.0.116 [ 2046.620378]  ? hrtimer_interrupt+0x5/0x360
Dec  2 12:39:00 10.243.0.116 [ 2046.663430]  ? proxy_irq_handler+0x37/0x50
Dec  2 12:39:00 10.243.0.116 [ 2046.705363]  ? handle_synthetic_irq+0xec/0x320
Dec  2 12:39:00 10.243.0.116 [ 2046.747283]  ? do_irq_inband+0x2f/0x40
Dec  2 12:39:00 10.243.0.116 [ 2046.788307]  ? arch_do_IRQ_pipelined+0xcd/0x5b0
Dec  2 12:39:00 10.243.0.116 [ 2046.828966]  </IRQ>
Dec  2 12:39:00 10.243.0.116 [ 2046.868360]  <TASK>
Dec  2 12:39:00 10.243.0.116 [ 2046.907769]  ? sync_current_irq_stage+0x2e3/0x500
Dec  2 12:39:00 10.243.0.116 [ 2046.948106]  ? sync_irq_stage+0x1af/0x1e0
Dec  2 12:39:00 10.243.0.116 [ 2046.987021]  ? synchronize_pipeline+0x8b/0xe0
Dec  2 12:39:00 10.243.0.116 [ 2047.025317]  ? handle_irq_pipelined_finish+0xf6/0x240
Dec  2 12:39:00 10.243.0.116 [ 2047.063551]  ? arch_pipeline_entry+0x81/0x110
Dec  2 12:39:00 10.243.0.116 [ 2047.101173]  ? sysvec_apic_timer_interrupt+0xe/0x20
Dec  2 12:39:00 10.243.0.116 [ 2047.138384]  ? asm_sysvec_apic_timer_interrupt+0x1b/0x20
Dec  2 12:39:00 10.243.0.116 [ 2047.176263]  ? smp_call_function_many_cond+0x1c7/0x4e0
Dec  2 12:39:01 10.243.0.116 [ 2047.213429]  ? smp_call_function_many_cond+0x1bf/0x4e0
Dec  2 12:39:01 10.243.0.116 [ 2047.250086]  ? smp_call_function_many_cond+0x1c7/0x4e0
Dec  2 12:39:01 10.243.0.116 [ 2047.286270]  ? leave_mm+0x40/0x40
Dec  2 12:39:01 10.243.0.116 [ 2047.321531]  ? x86_configure_nx+0x70/0x70
Dec  2 12:39:01 10.243.0.116 [ 2047.357103]  ? on_each_cpu_cond_mask+0x29/0x40
Dec  2 12:39:01 10.243.0.116 [ 2047.390970]  ? native_flush_tlb_multi+0xbe/0x250
Dec  2 12:39:01 10.243.0.116 [ 2047.423165]  ? flush_tlb_mm_range+0x136/0x1f0
Dec  2 12:39:01 10.243.0.116 [ 2047.455372]  ? change_protection_range+0x8bd/0xea0
Dec  2 12:39:01 10.243.0.116 [ 2047.489147]  ? prot_none_pte_entry+0x70/0x70
Dec  2 12:39:01 10.243.0.116 [ 2047.520707]  ? 0xffffffffc0b38099
Dec  2 12:39:01 10.243.0.116 [ 2047.551768]  ? change_protection_range+0x5/0xea0
Dec  2 12:39:01 10.243.0.116 [ 2047.583130]  ? change_protection+0x50/0x90
Dec  2 12:39:01 10.243.0.116 [ 2047.614115]  ? mprotect_fixup+0x2c5/0x490
Dec  2 12:39:01 10.243.0.116 [ 2047.645809]  ? change_protection+0x90/0x90
Dec  2 12:39:01 10.243.0.116 [ 2047.677157]  ? mprotect_fixup+0x5/0x490
Dec  2 12:39:01 10.243.0.116 [ 2047.709020]  ? do_mprotect_pkey+0x32a/0x590
Dec  2 12:39:01 10.243.0.116 [ 2047.740960]  ? mprotect_fixup+0x490/0x490
Dec  2 12:39:01 10.243.0.116 [ 2047.772601]  ? do_mprotect_pkey+0x5/0x590
Dec  2 12:39:01 10.243.0.116 [ 2047.804697]  ? __x64_sys_mprotect+0x48/0x60
Dec  2 12:39:01 10.243.0.116 [ 2047.836395]  ? do_syscall_64+0x4c/0xa0
Dec  2 12:39:01 10.243.0.116 [ 2047.867909]  ? entry_SYSCALL_64_after_hwframe+0x61/0xcb
Dec  2 12:39:01 10.243.0.116 [ 2047.900943]  </TASK>
Dec  2 12:39:14 10.243.0.116 [ 2060.448392] watchdog: BUG: soft lockup - CPU#2 stuck for 149s! [kworker/2:2:259]
Dec  2 12:39:14 10.243.0.116 [ 2060.475591] Modules linked in: xt_CHECKSUM xt_MASQUERADE tun netconsole ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 xt_conntrack ebtable_nat ebtable_broute ip6table_nat ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_nat iptable_mangle iptable_security iptable_raw nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 libcrc32c ip_set nfnetlink ebtable_filter ebtables rfkill ip6table_filter ip6_tables iptable_filter sunrpc dm_mirror dm_region_hash dm_log dm_mod x86_pkg_temp_thermal intel_powerclamp coretemp gpio_ich iTCO_wdt iTCO_vendor_support kvm_intel mxm_wmi kvm irqbypass crct10dif_pclmul crc32_pclmul ghash_clmulni_intel aesni_intel crypto_simd cryptd rapl intel_cstate pcspkr sg i2c_i801 ftdi_sio mei_me i2c_smbus joydev input_leds intel_pch_thermal mei lpc_ich mfd_core ioatdma wmi ip_tables ext4 mbcache jbd2 ast drm_vram_helper drm_kms_helper syscopyarea ixgbe sysfillrect sd_mod sr_mod sysimgblt fb_sys_fops cdrom t10_pi drm_ttm_helper ttm mdio drm igb ah
Dec  2 12:39:14 10.243.0.116 [ 2060.486482]  libahci libata crc32c_intel ptp pps_core i2c_algo_bit dca fuse
Dec  2 12:39:14 10.243.0.116 [ 2060.730068] irq event stamp: 23654
Dec  2 12:39:14 10.243.0.116 [ 2060.758569] hardirqs last  enabled at (23653): [<ffffffff8e1bd0d7>] _raw_spin_unlock_irq+0x27/0x40
Dec  2 12:39:14 10.243.0.116 [ 2060.788809] hardirqs last disabled at (23654): [<ffffffff8e1afe3d>] __schedule+0x90d/0x1160
Dec  2 12:39:14 10.243.0.116 [ 2060.819247] softirqs last  enabled at (23540): [<ffffffff8e400332>] __do_softirq+0x332/0x598
Dec  2 12:39:14 10.243.0.116 [ 2060.850141] softirqs last disabled at (23529): [<ffffffff8d0dc80b>] irq_exit_rcu+0xab/0x180
Dec  2 12:39:14 10.243.0.116 [ 2060.880299] CPU: 2 PID: 259 Comm: kworker/2:2 Tainted: G    B   W    L    5.15.77evl-g5e681f841fc3-dirty #3
Dec  2 12:39:14 10.243.0.116 [ 2060.910500] Hardware name: Supermicro Super Server/X10SDV-6C+-TLN4F, BIOS 2.1 11/22/2019
Dec  2 12:39:14 10.243.0.116 [ 2060.941241] IRQ stage: Linux
Dec  2 12:39:14 10.243.0.116 [ 2060.970686] Workqueue: rcu_gp wait_rcu_exp_gp
Dec  2 12:39:14 10.243.0.116 [ 2061.000770] RIP: 0010:proxy_set_next_ktime+0x77/0x90
Dec  2 12:39:14 10.243.0.116 [ 2061.031233] Code: 74 37 49 8d 7c 24 30 e8 d7 2d 1d 00 49 8b 44 24 30 4c 89 e7 e8 da 42 0b 01 48 8d 34 03 31 d2 4c 89 ef e8 5c 14 00 00 41 56 9d <31> c0 5b 41 5c 41 5d 41 5e 5d c3 cc cc cc cc e8 25 f5 eb ff eb d9
Dec  2 12:39:14 10.243.0.116 [ 2061.097685] RSP: 0018:ffff88841e309e80 EFLAGS: 00000212 ORIG_RAX: 0000000000000000
Dec  2 12:39:14 10.243.0.116 [ 2061.132281] RAX: 0000000000000000 RBX: 0000000000000000 RCX: 0000000000000000
Dec  2 12:39:14 10.243.0.116 [ 2061.167173] RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000000
Dec  2 12:39:15 10.243.0.116 [ 2061.200725] RBP: ffff88841e309ea0 R08: 0000000000000000 R09: 0000000000000000
Dec  2 12:39:15 10.243.0.116 [ 2061.235143] R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000000
Dec  2 12:39:15 10.243.0.116 [ 2061.270298] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000
Dec  2 12:39:15 10.243.0.116 [ 2061.305262] FS:  0000000000000000(0000) GS:ffff88841e300000(0000) knlGS:0000000000000000
Dec  2 12:39:15 10.243.0.116 [ 2061.340219] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Dec  2 12:39:15 10.243.0.116 [ 2061.375425] CR2: 00007f20be593000 CR3: 000000012c0f0004 CR4: 00000000003706e0
Dec  2 12:39:15 10.243.0.116 [ 2061.410750] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Dec  2 12:39:15 10.243.0.116 [ 2061.446647] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Dec  2 12:39:15 10.243.0.116 [ 2061.483639] Call Trace:
Dec  2 12:39:15 10.243.0.116 [ 2061.519200]  <IRQ>
Dec  2 12:39:15 10.243.0.116 [ 2061.555338]  ? __hrtimer_run_queues+0x5/0x780
Dec  2 12:39:15 10.243.0.116 [ 2061.591578]  ? hrtimer_interrupt+0x1aa/0x360
Dec  2 12:39:15 10.243.0.116 [ 2061.628584]  ? hrtimer_interrupt+0x5/0x360
Dec  2 12:39:15 10.243.0.116 [ 2061.665510]  ? proxy_irq_handler+0x37/0x50
Dec  2 12:39:15 10.243.0.116 [ 2061.701603]  ? handle_synthetic_irq+0xec/0x320
Dec  2 12:39:15 10.243.0.116 [ 2061.737600]  ? do_irq_inband+0x2f/0x40
Dec  2 12:39:15 10.243.0.116 [ 2061.773420]  ? arch_do_IRQ_pipelined+0xcd/0x5b0
Dec  2 12:39:15 10.243.0.116 [ 2061.809606]  </IRQ>
Dec  2 12:39:15 10.243.0.116 [ 2061.844564]  <TASK>
Dec  2 12:39:15 10.243.0.116 [ 2061.880756]  ? sync_current_irq_stage+0x2e3/0x500
Dec  2 12:39:15 10.243.0.116 [ 2061.917441]  ? sync_irq_stage+0x1af/0x1e0
Dec  2 12:39:15 10.243.0.116 [ 2061.953222]  ? synchronize_pipeline+0x8b/0xe0
Dec  2 12:39:15 10.243.0.116 [ 2061.988742]  ? handle_irq_pipelined_finish+0xf6/0x240
Dec  2 12:39:15 10.243.0.116 [ 2062.024044]  ? arch_pipeline_entry+0x81/0x110
Dec  2 12:39:15 10.243.0.116 [ 2062.060520]  ? sysvec_apic_timer_interrupt+0xe/0x20
Dec  2 12:39:15 10.243.0.116 [ 2062.095276]  ? asm_sysvec_apic_timer_interrupt+0x1b/0x20
Dec  2 12:39:15 10.243.0.116 [ 2062.129138]  ? rcu_all_qs+0xa0/0xa0
Dec  2 12:39:15 10.243.0.116 [ 2062.162561]  ? smp_call_function_single+0x118/0x210
Dec  2 12:39:15 10.243.0.116 [ 2062.196576]  ? smp_call_function_single+0x110/0x210
Dec  2 12:39:16 10.243.0.116 [ 2062.230191]  ? 0xffffffffc0b38099
Dec  2 12:39:16 10.243.0.116 [ 2062.262961]  ? generic_exec_single+0x140/0x140
Dec  2 12:39:16 10.243.0.116 [ 2062.296062]  ? rcu_all_qs+0xa0/0xa0
Dec  2 12:39:16 10.243.0.116 [ 2062.327694]  ? smp_call_function_single+0x5/0x210
Dec  2 12:39:16 10.243.0.116 [ 2062.359335]  ? sync_rcu_exp_select_node_cpus+0x3df/0x5c0
Dec  2 12:39:16 10.243.0.116 [ 2062.392078]  ? sync_rcu_exp_select_cpus+0x2c5/0x5c0
Dec  2 12:39:16 10.243.0.116 [ 2062.423516]  ? wait_rcu_exp_gp+0x1f/0x30
Dec  2 12:39:16 10.243.0.116 [ 2062.454905]  ? process_one_work+0x525/0xa30
Dec  2 12:39:16 10.243.0.116 [ 2062.486606]  ? pwq_dec_nr_in_flight+0x120/0x120
Dec  2 12:39:16 10.243.0.116 [ 2062.518360]  ? process_one_work+0x5/0xa30
Dec  2 12:39:16 10.243.0.116 [ 2062.548595]  ? worker_thread+0x57/0x590
Dec  2 12:39:16 10.243.0.116 [ 2062.578441]  ? process_one_work+0xa30/0xa30
Dec  2 12:39:16 10.243.0.116 [ 2062.608470]  ? kthread+0x1f3/0x220
Dec  2 12:39:16 10.243.0.116 [ 2062.637626]  ? set_kthread_struct+0x90/0x90
Dec  2 12:39:16 10.243.0.116 [ 2062.665706]  ? ret_from_fork+0x22/0x30
Dec  2 12:39:16 10.243.0.116 [ 2062.693597]  </TASK>
Dec  2 12:39:26 10.243.0.116 [ 2072.448680] watchdog: BUG: soft lockup - CPU#0 stuck for 336s! [TxStartup:3574]
Dec  2 12:39:26 10.243.0.116 [ 2072.481782] Modules linked in: xt_CHECKSUM xt_MASQUERADE tun netconsole ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 xt_conntrack ebtable_nat ebtable_broute ip6table_nat ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_nat iptable_mangle iptable_security iptable_raw nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 libcrc32c ip_set nfnetlink ebtable_filter ebtables rfkill ip6table_filter ip6_tables iptable_filter sunrpc dm_mirror dm_region_hash dm_log dm_mod x86_pkg_temp_thermal intel_powerclamp coretemp gpio_ich iTCO_wdt iTCO_vendor_support kvm_intel mxm_wmi kvm irqbypass crct10dif_pclmul crc32_pclmul ghash_clmulni_intel aesni_intel crypto_simd cryptd rapl intel_cstate pcspkr sg i2c_i801 ftdi_sio mei_me i2c_smbus joydev input_leds intel_pch_thermal mei lpc_ich mfd_core ioatdma wmi ip_tables ext4 mbcache jbd2 ast drm_vram_helper drm_kms_helper syscopyarea ixgbe sysfillrect sd_mod sr_mod sysimgblt fb_sys_fops cdrom t10_pi drm_ttm_helper ttm mdio drm igb ah
Dec  2 12:39:26 10.243.0.116 [ 2072.495188]  libahci libata crc32c_intel ptp pps_core i2c_algo_bit dca fuse
Dec  2 12:39:26 10.243.0.116 [ 2072.803831] irq event stamp: 24564
Dec  2 12:39:26 10.243.0.116 [ 2072.838075] hardirqs last  enabled at (24563): [<ffffffff8d1d44ad>] sync_current_irq_stage+0x45d/0x500
Dec  2 12:39:26 10.243.0.116 [ 2072.874564] hardirqs last disabled at (24564): [<ffffffff8d49923b>] handle_mm_fault+0x48b/0x520
Dec  2 12:39:26 10.243.0.116 [ 2072.911128] softirqs last  enabled at (0): [<ffffffff8d0ca587>] copy_process+0x1307/0x3770
Dec  2 12:39:26 10.243.0.116 [ 2072.947950] softirqs last disabled at (0): [<0000000000000000>] 0x0
Dec  2 12:39:26 10.243.0.116 [ 2072.984461] CPU: 0 PID: 3574 Comm: TxStartup Tainted: G    B   W    L    5.15.77evl-g5e681f841fc3-dirty #3
Dec  2 12:39:26 10.243.0.116 [ 2073.022321] Hardware name: Supermicro Super Server/X10SDV-6C+-TLN4F, BIOS 2.1 11/22/2019
Dec  2 12:39:26 10.243.0.116 [ 2073.059945] IRQ stage: Linux
Dec  2 12:39:26 10.243.0.116 [ 2073.096349] RIP: 0010:proxy_set_next_ktime+0x77/0x90
Dec  2 12:39:26 10.243.0.116 [ 2073.133564] Code: 74 37 49 8d 7c 24 30 e8 d7 2d 1d 00 49 8b 44 24 30 4c 89 e7 e8 da 42 0b 01 48 8d 34 03 31 d2 4c 89 ef e8 5c 14 00 00 41 56 9d <31> c0 5b 41 5c 41 5d 41 5e 5d c3 cc cc cc cc e8 25 f5 eb ff eb d9
Dec  2 12:39:27 10.243.0.116 [ 2073.213832] RSP: 0018:ffff88841e209e80 EFLAGS: 00000283 ORIG_RAX: 0000000000000000
Dec  2 12:39:27 10.243.0.116 [ 2073.255849] RAX: 0000000000000000 RBX: 0000000000000000 RCX: 0000000000000000
Dec  2 12:39:27 10.243.0.116 [ 2073.297994] RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000000
Dec  2 12:39:27 10.243.0.116 [ 2073.340130] RBP: ffff88841e209ea0 R08: 0000000000000000 R09: 0000000000000000
Dec  2 12:39:27 10.243.0.116 [ 2073.382397] R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000000
Dec  2 12:39:27 10.243.0.116 [ 2073.424648] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000
Dec  2 12:39:27 10.243.0.116 [ 2073.466940] FS:  00007f301ae2e700(0000) GS:ffff88841e200000(0000) knlGS:0000000000000000
Dec  2 12:39:27 10.243.0.116 [ 2073.509838] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Dec  2 12:39:27 10.243.0.116 [ 2073.552453] CR2: 00007f1e4033b660 CR3: 0000000137d08006 CR4: 00000000003706f0
Dec  2 12:39:27 10.243.0.116 [ 2073.595831] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Dec  2 12:39:27 10.243.0.116 [ 2073.639918] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Dec  2 12:39:27 10.243.0.116 [ 2073.684143] Call Trace:
Dec  2 12:39:27 10.243.0.116 [ 2073.727390]  <IRQ>
Dec  2 12:39:27 10.243.0.116 [ 2073.770643]  ? __hrtimer_run_queues+0x5/0x780
Dec  2 12:39:27 10.243.0.116 [ 2073.814956]  ? hrtimer_interrupt+0x1aa/0x360
Dec  2 12:39:27 10.243.0.116 [ 2073.858829]  ? hrtimer_interrupt+0x5/0x360
Dec  2 12:39:27 10.243.0.116 [ 2073.902834]  ? proxy_irq_handler+0x37/0x50
Dec  2 12:39:27 10.243.0.116 [ 2073.946324]  ? handle_synthetic_irq+0xec/0x320
Dec  2 12:39:27 10.243.0.116 [ 2073.990214]  ? do_irq_inband+0x2f/0x40
Dec  2 12:39:27 10.243.0.116 [ 2074.033559]  ? arch_do_IRQ_pipelined+0xcd/0x5b0
Dec  2 12:39:27 10.243.0.116 [ 2074.077073]  </IRQ>
Dec  2 12:39:27 10.243.0.116 [ 2074.119560]  <TASK>
Dec  2 12:39:27 10.243.0.116 [ 2074.162223]  ? sync_current_irq_stage+0x2e3/0x500
Dec  2 12:39:28 10.243.0.116 [ 2074.206215]  ? sync_irq_stage+0x1af/0x1e0
Dec  2 12:39:28 10.243.0.116 [ 2074.249379]  ? synchronize_pipeline+0x8b/0xe0
Dec  2 12:39:28 10.243.0.116 [ 2074.292542]  ? handle_irq_pipelined_finish+0xf6/0x240
Dec  2 12:39:28 10.243.0.116 [ 2074.336084]  ? arch_pipeline_entry+0x81/0x110
Dec  2 12:39:28 10.243.0.116 [ 2074.378759]  ? sysvec_apic_timer_interrupt+0xe/0x20
Dec  2 12:39:28 10.243.0.116 [ 2074.420644]  ? asm_sysvec_apic_timer_interrupt+0x1b/0x20
Dec  2 12:39:28 10.243.0.116 [ 2074.463214]  ? smp_call_function_many_cond+0x1c7/0x4e0
Dec  2 12:39:28 10.243.0.116 [ 2074.504766]  ? smp_call_function_many_cond+0x1bf/0x4e0
Dec  2 12:39:28 10.243.0.116 [ 2074.545654]  ? smp_call_function_many_cond+0x1c7/0x4e0
Dec  2 12:39:28 10.243.0.116 [ 2074.585941]  ? leave_mm+0x40/0x40
Dec  2 12:39:28 10.243.0.116 [ 2074.625301]  ? x86_configure_nx+0x70/0x70
Dec  2 12:39:28 10.243.0.116 [ 2074.664598]  ? on_each_cpu_cond_mask+0x29/0x40
Dec  2 12:39:28 10.243.0.116 [ 2074.703284]  ? native_flush_tlb_multi+0xbe/0x250
Dec  2 12:39:28 10.243.0.116 [ 2074.741707]  ? flush_tlb_mm_range+0x136/0x1f0
Dec  2 12:39:28 10.243.0.116 [ 2074.779837]  ? change_protection_range+0x8bd/0xea0
Dec  2 12:39:28 10.243.0.116 [ 2074.819603]  ? prot_none_pte_entry+0x70/0x70
Dec  2 12:39:28 10.243.0.116 [ 2074.856966]  ? 0xffffffffc0b38099
Dec  2 12:39:28 10.243.0.116 [ 2074.893950]  ? change_protection_range+0x5/0xea0
Dec  2 12:39:28 10.243.0.116 [ 2074.931179]  ? change_protection+0x50/0x90
Dec  2 12:39:28 10.243.0.116 [ 2074.967496]  ? mprotect_fixup+0x2c5/0x490
Dec  2 12:39:28 10.243.0.116 [ 2075.004032]  ? change_protection+0x90/0x90
Dec  2 12:39:28 10.243.0.116 [ 2075.039649]  ? mprotect_fixup+0x5/0x490
Dec  2 12:39:28 10.243.0.116 [ 2075.075605]  ? do_mprotect_pkey+0x32a/0x590
Dec  2 12:39:28 10.243.0.116 [ 2075.110053]  ? mprotect_fixup+0x490/0x490
Dec  2 12:39:28 10.243.0.116 [ 2075.142030]  ? do_mprotect_pkey+0x5/0x590
Dec  2 12:39:28 10.243.0.116 [ 2075.174378]  ? __x64_sys_mprotect+0x48/0x60
Dec  2 12:39:29 10.243.0.116 [ 2075.205924]  ? do_syscall_64+0x4c/0xa0
Dec  2 12:39:29 10.243.0.116 [ 2075.237209]  ? entry_SYSCALL_64_after_hwframe+0x61/0xcb
Dec  2 12:39:29 10.243.0.116 [ 2075.270326]  </TASK>
Dec  2 12:39:42 10.243.0.116 [ 2088.448629] watchdog: BUG: soft lockup - CPU#2 stuck for 176s! [kworker/2:2:259]
Dec  2 12:39:42 10.243.0.116 [ 2088.475352] Modules linked in: xt_CHECKSUM xt_MASQUERADE tun netconsole ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 xt_conntrack ebtable_nat ebtable_broute ip6table_nat ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_nat iptable_mangle iptable_security iptable_raw nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 libcrc32c ip_set nfnetlink ebtable_filter ebtables rfkill ip6table_filter ip6_tables iptable_filter sunrpc dm_mirror dm_region_hash dm_log dm_mod x86_pkg_temp_thermal intel_powerclamp coretemp gpio_ich iTCO_wdt iTCO_vendor_support kvm_intel mxm_wmi kvm irqbypass crct10dif_pclmul crc32_pclmul ghash_clmulni_intel aesni_intel crypto_simd cryptd rapl intel_cstate pcspkr sg i2c_i801 ftdi_sio mei_me i2c_smbus joydev input_leds intel_pch_thermal mei lpc_ich mfd_core ioatdma wmi ip_tables ext4 mbcache jbd2 ast drm_vram_helper drm_kms_helper syscopyarea ixgbe sysfillrect sd_mod sr_mod sysimgblt fb_sys_fops cdrom t10_pi drm_ttm_helper ttm mdio drm igb ah
Dec  2 12:39:42 10.243.0.116 [ 2088.486572]  libahci libata crc32c_intel ptp pps_core i2c_algo_bit dca fuse
Dec  2 12:39:42 10.243.0.116 [ 2088.730631] irq event stamp: 23654
Dec  2 12:39:42 10.243.0.116 [ 2088.758626] hardirqs last  enabled at (23653): [<ffffffff8e1bd0d7>] _raw_spin_unlock_irq+0x27/0x40
Dec  2 12:39:42 10.243.0.116 [ 2088.787936] hardirqs last disabled at (23654): [<ffffffff8e1afe3d>] __schedule+0x90d/0x1160
Dec  2 12:39:42 10.243.0.116 [ 2088.817621] softirqs last  enabled at (23540): [<ffffffff8e400332>] __do_softirq+0x332/0x598
Dec  2 12:39:42 10.243.0.116 [ 2088.847524] softirqs last disabled at (23529): [<ffffffff8d0dc80b>] irq_exit_rcu+0xab/0x180
Dec  2 12:39:42 10.243.0.116 [ 2088.877312] CPU: 2 PID: 259 Comm: kworker/2:2 Tainted: G    B   W    L    5.15.77evl-g5e681f841fc3-dirty #3
Dec  2 12:39:42 10.243.0.116 [ 2088.909046] Hardware name: Supermicro Super Server/X10SDV-6C+-TLN4F, BIOS 2.1 11/22/2019
Dec  2 12:39:42 10.243.0.116 [ 2088.940060] IRQ stage: Linux
Dec  2 12:39:42 10.243.0.116 [ 2088.970322] Workqueue: rcu_gp wait_rcu_exp_gp
Dec  2 12:39:42 10.243.0.116 [ 2089.001487] RIP: 0010:proxy_set_next_ktime+0x77/0x90
Dec  2 12:39:42 10.243.0.116 [ 2089.031748] Code: 74 37 49 8d 7c 24 30 e8 d7 2d 1d 00 49 8b 44 24 30 4c 89 e7 e8 da 42 0b 01 48 8d 34 03 31 d2 4c 89 ef e8 5c 14 00 00 41 56 9d <31> c0 5b 41 5c 41 5d 41 5e 5d c3 cc cc cc cc e8 25 f5 eb ff eb d9
Dec  2 12:39:42 10.243.0.116 [ 2089.097870] RSP: 0018:ffff88841e309e80 EFLAGS: 00000206 ORIG_RAX: 0000000000000000
Dec  2 12:39:42 10.243.0.116 [ 2089.132269] RAX: 0000000000000000 RBX: 0000000000000000 RCX: 0000000000000000
Dec  2 12:39:42 10.243.0.116 [ 2089.167738] RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000000
Dec  2 12:39:43 10.243.0.116 [ 2089.202163] RBP: ffff88841e309ea0 R08: 0000000000000000 R09: 0000000000000000
Dec  2 12:39:43 10.243.0.116 [ 2089.236901] R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000000
Dec  2 12:39:43 10.243.0.116 [ 2089.272330] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000
Dec  2 12:39:43 10.243.0.116 [ 2089.307390] FS:  0000000000000000(0000) GS:ffff88841e300000(0000) knlGS:0000000000000000
Dec  2 12:39:43 10.243.0.116 [ 2089.342680] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Dec  2 12:39:43 10.243.0.116 [ 2089.377828] CR2: 00007f20be593000 CR3: 000000012c0f0004 CR4: 00000000003706e0
Dec  2 12:39:43 10.243.0.116 [ 2089.413995] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Dec  2 12:39:43 10.243.0.116 [ 2089.450145] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Dec  2 12:39:43 10.243.0.116 [ 2089.486639] Call Trace:
Dec  2 12:39:43 10.243.0.116 [ 2089.522001]  <IRQ>
Dec  2 12:39:43 10.243.0.116 [ 2089.558066]  ? __hrtimer_run_queues+0x5/0x780
Dec  2 12:39:43 10.243.0.116 [ 2089.594642]  ? hrtimer_interrupt+0x1aa/0x360
Dec  2 12:39:43 10.243.0.116 [ 2089.630634]  ? hrtimer_interrupt+0x5/0x360
Dec  2 12:39:43 10.243.0.116 [ 2089.668391]  ? proxy_irq_handler+0x37/0x50
Dec  2 12:39:43 10.243.0.116 [ 2089.705655]  ? handle_synthetic_irq+0xec/0x320
Dec  2 12:39:43 10.243.0.116 [ 2089.741436]  ? do_irq_inband+0x2f/0x40
Dec  2 12:39:43 10.243.0.116 [ 2089.777024]  ? arch_do_IRQ_pipelined+0xcd/0x5b0
Dec  2 12:39:43 10.243.0.116 [ 2089.812928]  </IRQ>
Dec  2 12:39:43 10.243.0.116 [ 2089.847781]  <TASK>
Dec  2 12:39:43 10.243.0.116 [ 2089.883618]  ? sync_current_irq_stage+0x2e3/0x500
Dec  2 12:39:43 10.243.0.116 [ 2089.920096]  ? sync_irq_stage+0x1af/0x1e0
Dec  2 12:39:43 10.243.0.116 [ 2089.956060]  ? synchronize_pipeline+0x8b/0xe0
Dec  2 12:39:43 10.243.0.116 [ 2089.991818]  ? handle_irq_pipelined_finish+0xf6/0x240
Dec  2 12:39:43 10.243.0.116 [ 2090.027497]  ? arch_pipeline_entry+0x81/0x110
Dec  2 12:39:43 10.243.0.116 [ 2090.062693]  ? sysvec_apic_timer_interrupt+0xe/0x20
Dec  2 12:39:43 10.243.0.116 [ 2090.098887]  ? asm_sysvec_apic_timer_interrupt+0x1b/0x20
Dec  2 12:39:43 10.243.0.116 [ 2090.133472]  ? rcu_all_qs+0xa0/0xa0
Dec  2 12:39:43 10.243.0.116 [ 2090.167668]  ? smp_call_function_single+0x118/0x210
Dec  2 12:39:44 10.243.0.116 [ 2090.201348]  ? smp_call_function_single+0x110/0x210
Dec  2 12:39:44 10.243.0.116 [ 2090.234913]  ? 0xffffffffc0b38099
Dec  2 12:39:44 10.243.0.116 [ 2090.267447]  ? generic_exec_single+0x140/0x140
Dec  2 12:39:44 10.243.0.116 [ 2090.299797]  ? rcu_all_qs+0xa0/0xa0
Dec  2 12:39:44 10.243.0.116 [ 2090.331452]  ? smp_call_function_single+0x5/0x210
Dec  2 12:39:44 10.243.0.116 [ 2090.362830]  ? sync_rcu_exp_select_node_cpus+0x3df/0x5c0
Dec  2 12:39:44 10.243.0.116 [ 2090.395518]  ? sync_rcu_exp_select_cpus+0x2c5/0x5c0
Dec  2 12:39:44 10.243.0.116 [ 2090.426866]  ? wait_rcu_exp_gp+0x1f/0x30
Dec  2 12:39:44 10.243.0.116 [ 2090.458175]  ? process_one_work+0x525/0xa30
Dec  2 12:39:44 10.243.0.116 [ 2090.488716]  ? pwq_dec_nr_in_flight+0x120/0x120
Dec  2 12:39:44 10.243.0.116 [ 2090.521548] igb 0000:05:00.0: Detected Tx Unit Hang#012[ 2090.521548]   Tx Queue             <7>#012[ 2090.521548]   TDH                  <a2>#012[ 2090.521548]   TDT                  <a2>#012[ 2090.521548]   next_to_use          <a2>#012[ 2090.521548]   next_to_clean        <37>#012[ 2090.521548] buffer_info[next_to_clean]#012[ 2090.521548]   time_stamp           <1001b065e>#012[ 2090.521548]   next_to_watch        <00000000a47c25ef>#012[ 2090.521548]   jiffies              <1001b4cf7>#012[ 2090.521548]   desc.status          <200001>
Dec  2 12:39:44 10.243.0.116 [ 2091.049899]  ? process_one_work+0x5/0xa30
Dec  2 12:39:44 10.243.0.116 [ 2091.075135]  ? worker_thread+0x57/0x590
Dec  2 12:39:44 10.243.0.116 [ 2091.100541]  ? process_one_work+0xa30/0xa30
Dec  2 12:39:44 10.243.0.116 [ 2091.124625]  ? kthread+0x1f3/0x220
Dec  2 12:39:44 10.243.0.116 [ 2091.148587]  ? set_kthread_struct+0x90/0x90
Dec  2 12:39:44 10.243.0.116 [ 2091.172035]  ? ret_from_fork+0x22/0x30
Dec  2 12:39:44 10.243.0.116 [ 2091.196708]  </TASK>
Dec  2 12:39:54 10.243.0.116 [ 2100.448815] watchdog: BUG: soft lockup - CPU#0 stuck for 362s! [TxStartup:3574]
Dec  2 12:39:54 10.243.0.116 [ 2100.478530] Modules linked in: xt_CHECKSUM xt_MASQUERADE tun netconsole ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 xt_conntrack ebtable_nat ebtable_broute ip6table_nat ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_nat iptable_mangle iptable_security iptable_raw nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 libcrc32c ip_set nfnetlink ebtable_filter ebtables rfkill ip6table_filter ip6_tables iptable_filter sunrpc dm_mirror dm_region_hash dm_log dm_mod x86_pkg_temp_thermal intel_powerclamp coretemp gpio_ich iTCO_wdt iTCO_vendor_support kvm_intel mxm_wmi kvm irqbypass crct10dif_pclmul crc32_pclmul ghash_clmulni_intel aesni_intel crypto_simd cryptd rapl intel_cstate pcspkr sg i2c_i801 ftdi_sio mei_me i2c_smbus joydev input_leds intel_pch_thermal mei lpc_ich mfd_core ioatdma wmi ip_tables ext4 mbcache jbd2 ast drm_vram_helper drm_kms_helper syscopyarea ixgbe sysfillrect sd_mod sr_mod sysimgblt fb_sys_fops cdrom t10_pi drm_ttm_helper ttm mdio drm igb ah
Dec  2 12:39:54 10.243.0.116 [ 2100.491938]  libahci libata crc32c_intel ptp pps_core i2c_algo_bit dca fuse
Dec  2 12:39:54 10.243.0.116 [ 2100.778576] irq event stamp: 24564
Dec  2 12:39:54 10.243.0.116 [ 2100.812244] hardirqs last  enabled at (24563): [<ffffffff8d1d44ad>] sync_current_irq_stage+0x45d/0x500
Dec  2 12:39:54 10.243.0.116 [ 2100.848101] hardirqs last disabled at (24564): [<ffffffff8d49923b>] handle_mm_fault+0x48b/0x520
Dec  2 12:39:54 10.243.0.116 [ 2100.884023] softirqs last  enabled at (0): [<ffffffff8d0ca587>] copy_process+0x1307/0x3770
Dec  2 12:39:54 10.243.0.116 [ 2100.920047] softirqs last disabled at (0): [<0000000000000000>] 0x0
Dec  2 12:39:54 10.243.0.116 [ 2100.955649] CPU: 0 PID: 3574 Comm: TxStartup Tainted: G    B   W    L    5.15.77evl-g5e681f841fc3-dirty #3
Dec  2 12:39:54 10.243.0.116 [ 2100.992638] Hardware name: Supermicro Super Server/X10SDV-6C+-TLN4F, BIOS 2.1 11/22/2019
Dec  2 12:39:54 10.243.0.116 [ 2101.029393] IRQ stage: Linux
Dec  2 12:39:54 10.243.0.116 [ 2101.065010] RIP: 0010:proxy_set_next_ktime+0x77/0x90
Dec  2 12:39:54 10.243.0.116 [ 2101.101248] Code: 74 37 49 8d 7c 24 30 e8 d7 2d 1d 00 49 8b 44 24 30 4c 89 e7 e8 da 42 0b 01 48 8d 34 03 31 d2 4c 89 ef e8 5c 14 00 00 41 56 9d <31> c0 5b 41 5c 41 5d 41 5e 5d c3 cc cc cc cc e8 25 f5 eb ff eb d9
Dec  2 12:39:54 10.243.0.116 [ 2101.179784] RSP: 0018:ffff88841e209e80 EFLAGS: 00000293 ORIG_RAX: 0000000000000000
Dec  2 12:39:55 10.243.0.116 [ 2101.221019] RAX: 0000000000000000 RBX: 0000000000000000 RCX: 0000000000000000
Dec  2 12:39:55 10.243.0.116 [ 2101.262269] RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000000
Dec  2 12:39:55 10.243.0.116 [ 2101.303492] RBP: ffff88841e209ea0 R08: 0000000000000000 R09: 0000000000000000
Dec  2 12:39:55 10.243.0.116 [ 2101.344652] R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000000
Dec  2 12:39:55 10.243.0.116 [ 2101.386037] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000
Dec  2 12:39:55 10.243.0.116 [ 2101.427404] FS:  00007f301ae2e700(0000) GS:ffff88841e200000(0000) knlGS:0000000000000000
Dec  2 12:39:55 10.243.0.116 [ 2101.469491] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Dec  2 12:39:55 10.243.0.116 [ 2101.511268] CR2: 00007f1e4033b660 CR3: 0000000137d08006 CR4: 00000000003706f0
Dec  2 12:39:55 10.243.0.116 [ 2101.553907] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Dec  2 12:39:55 10.243.0.116 [ 2101.597049] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Dec  2 12:39:55 10.243.0.116 [ 2101.640484] Call Trace:
Dec  2 12:39:55 10.243.0.116 [ 2101.683117]  <IRQ>
Dec  2 12:39:55 10.243.0.116 [ 2101.725983]  ? __hrtimer_run_queues+0x5/0x780
Dec  2 12:39:55 10.243.0.116 [ 2101.770084]  ? hrtimer_interrupt+0x1aa/0x360
Dec  2 12:39:55 10.243.0.116 [ 2101.814028]  ? hrtimer_interrupt+0x5/0x360
Dec  2 12:39:55 10.243.0.116 [ 2101.858141]  ? proxy_irq_handler+0x37/0x50
Dec  2 12:39:55 10.243.0.116 [ 2101.901590]  ? handle_synthetic_irq+0xec/0x320
Dec  2 12:39:55 10.243.0.116 [ 2101.945384]  ? do_irq_inband+0x2f/0x40
Dec  2 12:39:55 10.243.0.116 [ 2101.988805]  ? arch_do_IRQ_pipelined+0xcd/0x5b0
Dec  2 12:39:55 10.243.0.116 [ 2102.032320]  </IRQ>
Dec  2 12:39:55 10.243.0.116 [ 2102.074990]  <TASK>
Dec  2 12:39:55 10.243.0.116 [ 2102.117878]  ? sync_current_irq_stage+0x2e3/0x500
Dec  2 12:39:55 10.243.0.116 [ 2102.162145]  ? sync_irq_stage+0x1af/0x1e0
Dec  2 12:39:56 10.243.0.116 [ 2102.205659]  ? synchronize_pipeline+0x8b/0xe0
Dec  2 12:39:56 10.243.0.116 [ 2102.249046]  ? handle_irq_pipelined_finish+0xf6/0x240
Dec  2 12:39:56 10.243.0.116 [ 2102.292811]  ? arch_pipeline_entry+0x81/0x110
Dec  2 12:39:56 10.243.0.116 [ 2102.335687]  ? sysvec_apic_timer_interrupt+0xe/0x20
Dec  2 12:39:56 10.243.0.116 [ 2102.377800]  ? asm_sysvec_apic_timer_interrupt+0x1b/0x20
Dec  2 12:39:56 10.243.0.116 [ 2102.420525]  ? smp_call_function_many_cond+0x1c7/0x4e0
Dec  2 12:39:56 10.243.0.116 [ 2102.462312]  ? smp_call_function_many_cond+0x1bf/0x4e0
Dec  2 12:39:56 10.243.0.116 [ 2102.503432]  ? smp_call_function_many_cond+0x1c7/0x4e0
Dec  2 12:39:56 10.243.0.116 [ 2102.543940]  ? leave_mm+0x40/0x40
Dec  2 12:39:56 10.243.0.116 [ 2102.583522]  ? x86_configure_nx+0x70/0x70
Dec  2 12:39:56 10.243.0.116 [ 2102.623035]  ? on_each_cpu_cond_mask+0x29/0x40
Dec  2 12:39:56 10.243.0.116 [ 2102.662000]  ? native_flush_tlb_multi+0xbe/0x250
Dec  2 12:39:56 10.243.0.116 [ 2102.700752]  ? flush_tlb_mm_range+0x136/0x1f0
Dec  2 12:39:56 10.243.0.116 [ 2102.739204]  ? change_protection_range+0x8bd/0xea0
Dec  2 12:39:56 10.243.0.116 [ 2102.779304]  ? prot_none_pte_entry+0x70/0x70
Dec  2 12:39:56 10.243.0.116 [ 2102.816984]  ? 0xffffffffc0b38099
Dec  2 12:39:56 10.243.0.116 [ 2102.854285]  ? change_protection_range+0x5/0xea0
Dec  2 12:39:56 10.243.0.116 [ 2102.891782]  ? change_protection+0x50/0x90
Dec  2 12:39:56 10.243.0.116 [ 2102.928355]  ? mprotect_fixup+0x2c5/0x490
Dec  2 12:39:56 10.243.0.116 [ 2102.965099]  ? change_protection+0x90/0x90
Dec  2 12:39:56 10.243.0.116 [ 2103.000868]  ? mprotect_fixup+0x5/0x490
Dec  2 12:39:56 10.243.0.116 [ 2103.037083]  ? do_mprotect_pkey+0x32a/0x590
Dec  2 12:39:56 10.243.0.116 [ 2103.071741]  ? mprotect_fixup+0x490/0x490
Dec  2 12:39:56 10.243.0.116 [ 2103.103817]  ? do_mprotect_pkey+0x5/0x590
Dec  2 12:39:56 10.243.0.116 [ 2103.136229]  ? __x64_sys_mprotect+0x48/0x60
Dec  2 12:39:56 10.243.0.116 [ 2103.167825]  ? do_syscall_64+0x4c/0xa0
Dec  2 12:39:56 10.243.0.116 [ 2103.199202]  ? entry_SYSCALL_64_after_hwframe+0x61/0xcb
Dec  2 12:39:57 10.243.0.116 [ 2103.232352]  </TASK>
Dec  2 12:40:10 10.243.0.116 [ 2116.448664] watchdog: BUG: soft lockup - CPU#2 stuck for 202s! [kworker/2:2:259]
Dec  2 12:40:10 10.243.0.116 [ 2116.472238] Modules linked in: xt_CHECKSUM xt_MASQUERADE tun netconsole ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 xt_conntrack ebtable_nat ebtable_broute ip6table_nat ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_nat iptable_mangle iptable_security iptable_raw nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 libcrc32c ip_set nfnetlink ebtable_filter ebtables rfkill ip6table_filter ip6_tables iptable_filter sunrpc dm_mirror dm_region_hash dm_log dm_mod x86_pkg_temp_thermal intel_powerclamp coretemp gpio_ich iTCO_wdt iTCO_vendor_support kvm_intel mxm_wmi kvm irqbypass crct10dif_pclmul crc32_pclmul ghash_clmulni_intel aesni_intel crypto_simd cryptd rapl intel_cstate pcspkr sg i2c_i801 ftdi_sio mei_me i2c_smbus joydev input_leds intel_pch_thermal mei lpc_ich mfd_core ioatdma wmi ip_tables ext4 mbcache jbd2 ast drm_vram_helper drm_kms_helper syscopyarea ixgbe sysfillrect sd_mod sr_mod sysimgblt fb_sys_fops cdrom t10_pi drm_ttm_helper ttm mdio drm igb ah
Dec  2 12:40:10 10.243.0.116 [ 2116.481515]  libahci libata crc32c_intel ptp pps_core i2c_algo_bit dca fuse
Dec  2 12:40:10 10.243.0.116 [ 2116.690576] irq event stamp: 23654
Dec  2 12:40:10 10.243.0.116 [ 2116.714282] hardirqs last  enabled at (23653): [<ffffffff8e1bd0d7>] _raw_spin_unlock_irq+0x27/0x40
Dec  2 12:40:10 10.243.0.116 [ 2116.739381] hardirqs last disabled at (23654): [<ffffffff8e1afe3d>] __schedule+0x90d/0x1160
Dec  2 12:40:10 10.243.0.116 [ 2116.764607] softirqs last  enabled at (23540): [<ffffffff8e400332>] __do_softirq+0x332/0x598
Dec  2 12:40:10 10.243.0.116 [ 2116.789853] softirqs last disabled at (23529): [<ffffffff8d0dc80b>] irq_exit_rcu+0xab/0x180
Dec  2 12:40:10 10.243.0.116 [ 2116.815610] CPU: 2 PID: 259 Comm: kworker/2:2 Tainted: G    B   W    L    5.15.77evl-g5e681f841fc3-dirty #3
Dec  2 12:40:10 10.243.0.116 [ 2116.843038] Hardware name: Supermicro Super Server/X10SDV-6C+-TLN4F, BIOS 2.1 11/22/2019
Dec  2 12:40:10 10.243.0.116 [ 2116.873343] IRQ stage: Linux
Dec  2 12:40:10 10.243.0.116 [ 2116.902852] Workqueue: rcu_gp wait_rcu_exp_gp
Dec  2 12:40:10 10.243.0.116 [ 2116.933088] RIP: 0010:proxy_set_next_ktime+0x77/0x90
Dec  2 12:40:10 10.243.0.116 [ 2116.963553] Code: 74 37 49 8d 7c 24 30 e8 d7 2d 1d 00 49 8b 44 24 30 4c 89 e7 e8 da 42 0b 01 48 8d 34 03 31 d2 4c 89 ef e8 5c 14 00 00 41 56 9d <31> c0 5b 41 5c 41 5d 41 5e 5d c3 cc cc cc cc e8 25 f5 eb ff eb d9
Dec  2 12:40:10 10.243.0.116 [ 2117.030373] RSP: 0018:ffff88841e309e80 EFLAGS: 00000206 ORIG_RAX: 0000000000000000
Dec  2 12:40:10 10.243.0.116 [ 2117.065225] RAX: 0000000000000000 RBX: 0000000000000000 RCX: 0000000000000000
Dec  2 12:40:10 10.243.0.116 [ 2117.101015] RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000000
Dec  2 12:40:10 10.243.0.116 [ 2117.135637] RBP: ffff88841e309ea0 R08: 0000000000000000 R09: 0000000000000000
Dec  2 12:40:10 10.243.0.116 [ 2117.169933] R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000000
Dec  2 12:40:11 10.243.0.116 [ 2117.204701] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000
Dec  2 12:40:11 10.243.0.116 [ 2117.239608] FS:  0000000000000000(0000) GS:ffff88841e300000(0000) knlGS:0000000000000000
Dec  2 12:40:11 10.243.0.116 [ 2117.274729] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Dec  2 12:40:11 10.243.0.116 [ 2117.309287] CR2: 00007f20be593000 CR3: 000000012c0f0004 CR4: 00000000003706e0
Dec  2 12:40:11 10.243.0.116 [ 2117.344779] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Dec  2 12:40:11 10.243.0.116 [ 2117.380910] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Dec  2 12:40:11 10.243.0.116 [ 2117.416836] Call Trace:
Dec  2 12:40:11 10.243.0.116 [ 2117.452955]  <IRQ>
Dec  2 12:40:11 10.243.0.116 [ 2117.488345]  ? __hrtimer_run_queues+0x5/0x780
Dec  2 12:40:11 10.243.0.116 [ 2117.524823]  ? hrtimer_interrupt+0x1aa/0x360
Dec  2 12:40:11 10.243.0.116 [ 2117.561380]  ? hrtimer_interrupt+0x5/0x360
Dec  2 12:40:11 10.243.0.116 [ 2117.597261]  ? proxy_irq_handler+0x37/0x50
Dec  2 12:40:11 10.243.0.116 [ 2117.634103]  ? handle_synthetic_irq+0xec/0x320
Dec  2 12:40:11 10.243.0.116 [ 2117.669282]  ? do_irq_inband+0x2f/0x40
Dec  2 12:40:11 10.243.0.116 [ 2117.705035]  ? arch_do_IRQ_pipelined+0xcd/0x5b0
Dec  2 12:40:11 10.243.0.116 [ 2117.740104]  </IRQ>
Dec  2 12:40:11 10.243.0.116 [ 2117.774863]  <TASK>
Dec  2 12:40:11 10.243.0.116 [ 2117.809975]  ? sync_current_irq_stage+0x2e3/0x500
Dec  2 12:40:11 10.243.0.116 [ 2117.845972]  ? sync_irq_stage+0x1af/0x1e0
Dec  2 12:40:11 10.243.0.116 [ 2117.881950]  ? synchronize_pipeline+0x8b/0xe0
Dec  2 12:40:11 10.243.0.116 [ 2117.917238]  ? handle_irq_pipelined_finish+0xf6/0x240
Dec  2 12:40:11 10.243.0.116 [ 2117.952517]  ? arch_pipeline_entry+0x81/0x110
Dec  2 12:40:11 10.243.0.116 [ 2117.986932]  ? sysvec_apic_timer_interrupt+0xe/0x20
Dec  2 12:40:11 10.243.0.116 [ 2118.021173]  ? asm_sysvec_apic_timer_interrupt+0x1b/0x20
Dec  2 12:40:11 10.243.0.116 [ 2118.055060]  ? rcu_all_qs+0xa0/0xa0
Dec  2 12:40:11 10.243.0.116 [ 2118.088745]  ? smp_call_function_single+0x118/0x210
Dec  2 12:40:11 10.243.0.116 [ 2118.121946]  ? __asan_load4+0x1/0x90
Dec  2 12:40:11 10.243.0.116 [ 2118.155217]  ? smp_call_function_single+0x118/0x210
Dec  2 12:40:11 10.243.0.116 [ 2118.188678]  ? 0xffffffffc0b38099
Dec  2 12:40:12 10.243.0.116 [ 2118.221441]  ? generic_exec_single+0x140/0x140
Dec  2 12:40:12 10.243.0.116 [ 2118.254434]  ? rcu_all_qs+0xa0/0xa0
Dec  2 12:40:12 10.243.0.116 [ 2118.285958]  ? smp_call_function_single+0x5/0x210
Dec  2 12:40:12 10.243.0.116 [ 2118.317541]  ? sync_rcu_exp_select_node_cpus+0x3df/0x5c0
Dec  2 12:40:12 10.243.0.116 [ 2118.349629]  ? sync_rcu_exp_select_cpus+0x2c5/0x5c0
Dec  2 12:40:12 10.243.0.116 [ 2118.380715]  ? wait_rcu_exp_gp+0x1f/0x30
Dec  2 12:40:12 10.243.0.116 [ 2118.410933]  ? process_one_work+0x525/0xa30
Dec  2 12:40:12 10.243.0.116 [ 2118.441787]  ? pwq_dec_nr_in_flight+0x120/0x120
Dec  2 12:40:12 10.243.0.116 [ 2118.472037]  ? process_one_work+0x5/0xa30
Dec  2 12:40:12 10.243.0.116 [ 2118.502451]  ? worker_thread+0x57/0x590
Dec  2 12:40:12 10.243.0.116 [ 2118.531697]  ? process_one_work+0xa30/0xa30
Dec  2 12:40:12 10.243.0.116 [ 2118.561546]  ? kthread+0x1f3/0x220
Dec  2 12:40:12 10.243.0.116 [ 2118.589568]  ? set_kthread_struct+0x90/0x90
Dec  2 12:40:12 10.243.0.116 [ 2118.616472]  ? ret_from_fork+0x22/0x30
Dec  2 12:40:12 10.243.0.116 [ 2118.644962]  </TASK>
Dec  2 12:40:22 10.243.0.116 [ 2128.448943] watchdog: BUG: soft lockup - CPU#0 stuck for 388s! [TxStartup:3574]
Dec  2 12:40:22 10.243.0.116 [ 2128.482213] Modules linked in: xt_CHECKSUM xt_MASQUERADE tun netconsole ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 xt_conntrack ebtable_nat ebtable_broute ip6table_nat ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_nat iptable_mangle iptable_security iptable_raw nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 libcrc32c ip_set nfnetlink ebtable_filter ebtables rfkill ip6table_filter ip6_tables iptable_filter sunrpc dm_mirror dm_region_hash dm_log dm_mod x86_pkg_temp_thermal intel_powerclamp coretemp gpio_ich iTCO_wdt iTCO_vendor_support kvm_intel mxm_wmi kvm irqbypass crct10dif_pclmul crc32_pclmul ghash_clmulni_intel aesni_intel crypto_simd cryptd rapl intel_cstate pcspkr sg i2c_i801 ftdi_sio mei_me i2c_smbus joydev input_leds intel_pch_thermal mei lpc_ich mfd_core ioatdma wmi ip_tables ext4 mbcache jbd2 ast drm_vram_helper drm_kms_helper syscopyarea ixgbe sysfillrect sd_mod sr_mod sysimgblt fb_sys_fops cdrom t10_pi drm_ttm_helper ttm mdio drm igb ah
Dec  2 12:40:22 10.243.0.116 [ 2128.495619]  libahci libata crc32c_intel ptp pps_core i2c_algo_bit dca fuse
Dec  2 12:40:22 10.243.0.116 [ 2128.802341] irq event stamp: 24564
Dec  2 12:40:22 10.243.0.116 [ 2128.836839] hardirqs last  enabled at (24563): [<ffffffff8d1d44ad>] sync_current_irq_stage+0x45d/0x500
Dec  2 12:40:22 10.243.0.116 [ 2128.873513] hardirqs last disabled at (24564): [<ffffffff8d49923b>] handle_mm_fault+0x48b/0x520
Dec  2 12:40:22 10.243.0.116 [ 2128.910272] softirqs last  enabled at (0): [<ffffffff8d0ca587>] copy_process+0x1307/0x3770
Dec  2 12:40:22 10.243.0.116 [ 2128.947353] softirqs last disabled at (0): [<0000000000000000>] 0x0
Dec  2 12:40:22 10.243.0.116 [ 2128.984026] CPU: 0 PID: 3574 Comm: TxStartup Tainted: G    B   W    L    5.15.77evl-g5e681f841fc3-dirty #3
Dec  2 12:40:22 10.243.0.116 [ 2129.022043] Hardware name: Supermicro Super Server/X10SDV-6C+-TLN4F, BIOS 2.1 11/22/2019
Dec  2 12:40:22 10.243.0.116 [ 2129.059965] IRQ stage: Linux
Dec  2 12:40:22 10.243.0.116 [ 2129.096715] RIP: 0010:proxy_set_next_ktime+0x77/0x90
Dec  2 12:40:22 10.243.0.116 [ 2129.134206] Code: 74 37 49 8d 7c 24 30 e8 d7 2d 1d 00 49 8b 44 24 30 4c 89 e7 e8 da 42 0b 01 48 8d 34 03 31 d2 4c 89 ef e8 5c 14 00 00 41 56 9d <31> c0 5b 41 5c 41 5d 41 5e 5d c3 cc cc cc cc e8 25 f5 eb ff eb d9
Dec  2 12:40:23 10.243.0.116 [ 2129.215161] RSP: 0018:ffff88841e209e80 EFLAGS: 00000283 ORIG_RAX: 0000000000000000
Dec  2 12:40:23 10.243.0.116 [ 2129.257476] RAX: 0000000000000000 RBX: 0000000000000000 RCX: 0000000000000000
Dec  2 12:40:23 10.243.0.116 [ 2129.299779] RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000000
Dec  2 12:40:23 10.243.0.116 [ 2129.342076] RBP: ffff88841e209ea0 R08: 0000000000000000 R09: 0000000000000000
Dec  2 12:40:23 10.243.0.116 [ 2129.384559] R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000000
Dec  2 12:40:23 10.243.0.116 [ 2129.427209] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000
Dec  2 12:40:23 10.243.0.116 [ 2129.469768] FS:  00007f301ae2e700(0000) GS:ffff88841e200000(0000) knlGS:0000000000000000
Dec  2 12:40:23 10.243.0.116 [ 2129.512868] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Dec  2 12:40:23 10.243.0.116 [ 2129.555639] CR2: 00007f1e4033b660 CR3: 0000000137d08006 CR4: 00000000003706f0
Dec  2 12:40:23 10.243.0.116 [ 2129.599174] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Dec  2 12:40:23 10.243.0.116 [ 2129.643299] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Dec  2 12:40:23 10.243.0.116 [ 2129.687572] Call Trace:
Dec  2 12:40:23 10.243.0.116 [ 2129.730861]  <IRQ>
Dec  2 12:40:23 10.243.0.116 [ 2129.774150]  ? __hrtimer_run_queues+0x5/0x780
Dec  2 12:40:23 10.243.0.116 [ 2129.818511]  ? hrtimer_interrupt+0x1aa/0x360
Dec  2 12:40:23 10.243.0.116 [ 2129.862419]  ? hrtimer_interrupt+0x5/0x360
Dec  2 12:40:23 10.243.0.116 [ 2129.906530]  ? proxy_irq_handler+0x37/0x50
Dec  2 12:40:23 10.243.0.116 [ 2129.950058]  ? handle_synthetic_irq+0xec/0x320
Dec  2 12:40:23 10.243.0.116 [ 2129.993931]  ? do_irq_inband+0x2f/0x40
Dec  2 12:40:23 10.243.0.116 [ 2130.037311]  ? arch_do_IRQ_pipelined+0xcd/0x5b0
Dec  2 12:40:23 10.243.0.116 [ 2130.080809]  </IRQ>
Dec  2 12:40:23 10.243.0.116 [ 2130.123341]  <TASK>
Dec  2 12:40:23 10.243.0.116 [ 2130.166162]  ? sync_current_irq_stage+0x2e3/0x500
Dec  2 12:40:24 10.243.0.116 [ 2130.210255]  ? sync_irq_stage+0x1af/0x1e0
Dec  2 12:40:24 10.243.0.116 [ 2130.253415]  ? synchronize_pipeline+0x8b/0xe0
Dec  2 12:40:24 10.243.0.116 [ 2130.296562]  ? handle_irq_pipelined_finish+0xf6/0x240
Dec  2 12:40:24 10.243.0.116 [ 2130.340085]  ? arch_pipeline_entry+0x81/0x110
Dec  2 12:40:24 10.243.0.116 [ 2130.382620]  ? sysvec_apic_timer_interrupt+0xe/0x20
Dec  2 12:40:24 10.243.0.116 [ 2130.424448]  ? asm_sysvec_apic_timer_interrupt+0x1b/0x20
Dec  2 12:40:24 10.243.0.116 [ 2130.466895]  ? smp_call_function_many_cond+0x1c7/0x4e0
Dec  2 12:40:24 10.243.0.116 [ 2130.508458]  ? smp_call_function_many_cond+0x1bf/0x4e0
Dec  2 12:40:24 10.243.0.116 [ 2130.549349]  ? smp_call_function_many_cond+0x1c7/0x4e0
Dec  2 12:40:24 10.243.0.116 [ 2130.589573]  ? leave_mm+0x40/0x40
Dec  2 12:40:24 10.243.0.116 [ 2130.628875]  ? x86_configure_nx+0x70/0x70
Dec  2 12:40:24 10.243.0.116 [ 2130.668113]  ? on_each_cpu_cond_mask+0x29/0x40
Dec  2 12:40:24 10.243.0.116 [ 2130.706794]  ? native_flush_tlb_multi+0xbe/0x250
Dec  2 12:40:24 10.243.0.116 [ 2130.745218]  ? flush_tlb_mm_range+0x136/0x1f0
Dec  2 12:40:24 10.243.0.116 [ 2130.783351]  ? change_protection_range+0x8bd/0xea0
Dec  2 12:40:24 10.243.0.116 [ 2130.823114]  ? prot_none_pte_entry+0x70/0x70
Dec  2 12:40:24 10.243.0.116 [ 2130.860478]  ? 0xffffffffc0b38099
Dec  2 12:40:24 10.243.0.116 [ 2130.897447]  ? change_protection_range+0x5/0xea0
Dec  2 12:40:24 10.243.0.116 [ 2130.934684]  ? change_protection+0x50/0x90
Dec  2 12:40:24 10.243.0.116 [ 2130.971000]  ? mprotect_fixup+0x2c5/0x490
Dec  2 12:40:24 10.243.0.116 [ 2131.007411]  ? change_protection+0x90/0x90
Dec  2 12:40:24 10.243.0.116 [ 2131.042978]  ? mprotect_fixup+0x5/0x490
Dec  2 12:40:24 10.243.0.116 [ 2131.078940]  ? do_mprotect_pkey+0x32a/0x590
Dec  2 12:40:24 10.243.0.116 [ 2131.113390]  ? mprotect_fixup+0x490/0x490
Dec  2 12:40:24 10.243.0.116 [ 2131.145371]  ? do_mprotect_pkey+0x5/0x590
Dec  2 12:40:24 10.243.0.116 [ 2131.177723]  ? __x64_sys_mprotect+0x48/0x60
Dec  2 12:40:25 10.243.0.116 [ 2131.209322]  ? do_syscall_64+0x4c/0xa0
Dec  2 12:40:25 10.243.0.116 [ 2131.240653]  ? entry_SYSCALL_64_after_hwframe+0x61/0xcb
Dec  2 12:40:25 10.243.0.116 [ 2131.273701]  </TASK>
Dec  2 12:40:38 10.243.0.116 [ 2144.448885] watchdog: BUG: soft lockup - CPU#2 stuck for 228s! [kworker/2:2:259]
Dec  2 12:40:38 10.243.0.116 [ 2144.476030] Modules linked in: xt_CHECKSUM xt_MASQUERADE tun netconsole ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 xt_conntrack ebtable_nat ebtable_broute ip6table_nat ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_nat iptable_mangle iptable_security iptable_raw nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 libcrc32c ip_set nfnetlink ebtable_filter ebtables rfkill ip6table_filter ip6_tables iptable_filter sunrpc dm_mirror dm_region_hash dm_log dm_mod x86_pkg_temp_thermal intel_powerclamp coretemp gpio_ich iTCO_wdt iTCO_vendor_support kvm_intel mxm_wmi kvm irqbypass crct10dif_pclmul crc32_pclmul ghash_clmulni_intel aesni_intel crypto_simd cryptd rapl intel_cstate pcspkr sg i2c_i801 ftdi_sio mei_me i2c_smbus joydev input_leds intel_pch_thermal mei lpc_ich mfd_core ioatdma wmi ip_tables ext4 mbcache jbd2 ast drm_vram_helper drm_kms_helper syscopyarea ixgbe sysfillrect sd_mod sr_mod sysimgblt fb_sys_fops cdrom t10_pi drm_ttm_helper ttm mdio drm igb ah
Dec  2 12:40:38 10.243.0.116 [ 2144.486971]  libahci libata crc32c_intel ptp pps_core i2c_algo_bit dca fuse
Dec  2 12:40:38 10.243.0.116 [ 2144.731633] irq event stamp: 23654
Dec  2 12:40:38 10.243.0.116 [ 2144.759283] hardirqs last  enabled at (23653): [<ffffffff8e1bd0d7>] _raw_spin_unlock_irq+0x27/0x40
Dec  2 12:40:38 10.243.0.116 [ 2144.788919] hardirqs last disabled at (23654): [<ffffffff8e1afe3d>] __schedule+0x90d/0x1160
Dec  2 12:40:38 10.243.0.116 [ 2144.818735] softirqs last  enabled at (23540): [<ffffffff8e400332>] __do_softirq+0x332/0x598
Dec  2 12:40:38 10.243.0.116 [ 2144.848402] softirqs last disabled at (23529): [<ffffffff8d0dc80b>] irq_exit_rcu+0xab/0x180
Dec  2 12:40:38 10.243.0.116 [ 2144.878814] CPU: 2 PID: 259 Comm: kworker/2:2 Tainted: G    B   W    L    5.15.77evl-g5e681f841fc3-dirty #3
Dec  2 12:40:38 10.243.0.116 [ 2144.909134] Hardware name: Supermicro Super Server/X10SDV-6C+-TLN4F, BIOS 2.1 11/22/2019
Dec  2 12:40:38 10.243.0.116 [ 2144.939645] IRQ stage: Linux
Dec  2 12:40:38 10.243.0.116 [ 2144.967272] Workqueue: rcu_gp wait_rcu_exp_gp
Dec  2 12:40:38 10.243.0.116 [ 2144.993045] RIP: 0010:proxy_set_next_ktime+0x77/0x90
Dec  2 12:40:38 10.243.0.116 [ 2145.019251] Code: 74 37 49 8d 7c 24 30 e8 d7 2d 1d 00 49 8b 44 24 30 4c 89 e7 e8 da 42 0b 01 48 8d 34 03 31 d2 4c 89 ef e8 5c 14 00 00 41 56 9d <31> c0 5b 41 5c 41 5d 41 5e 5d c3 cc cc cc cc e8 25 f5 eb ff eb d9
Dec  2 12:40:38 10.243.0.116 [ 2145.076784] RSP: 0018:ffff88841e309e80 EFLAGS: 00000206 ORIG_RAX: 0000000000000000
Dec  2 12:40:38 10.243.0.116 [ 2145.106000] RAX: 0000000000000000 RBX: 0000000000000000 RCX: 0000000000000000
Dec  2 12:40:38 10.243.0.116 [ 2145.135196] RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000000
Dec  2 12:40:39 10.243.0.116 [ 2145.164940] RBP: ffff88841e309ea0 R08: 0000000000000000 R09: 0000000000000000
Dec  2 12:40:39 10.243.0.116 [ 2145.195994] igb 0000:05:00.0: Detected Tx Unit Hang#012[ 2145.195994]   Tx Queue             <7>#012[ 2145.195994]   TDH                  <81>#012[ 2145.195994]   TDT                  <81>#012[ 2145.195994]   next_to_use          <81>#012[ 2145.195994]   next_to_clean        <22>#012[ 2145.195994] buffer_info[next_to_clean]#012[ 2145.195994]   time_stamp           <1001bb9c9>#012[ 2145.195994]   next_to_watch        <000000006e41f2ad>#012[ 2145.195994]   jiffies              <1001c2289>#012[ 2145.195994]   desc.status          <184001>
Dec  2 12:40:39 10.243.0.116 [ 2145.705538] R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000000
Dec  2 12:40:39 10.243.0.116 [ 2145.739949] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000
Dec  2 12:40:39 10.243.0.116 [ 2145.774513] FS:  0000000000000000(0000) GS:ffff88841e300000(0000) knlGS:0000000000000000
Dec  2 12:40:39 10.243.0.116 [ 2145.810132] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Dec  2 12:40:39 10.243.0.116 [ 2145.845403] CR2: 00007f20be593000 CR3: 000000012c0f0004 CR4: 00000000003706e0
Dec  2 12:40:39 10.243.0.116 [ 2145.881708] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Dec  2 12:40:39 10.243.0.116 [ 2145.918447] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Dec  2 12:40:39 10.243.0.116 [ 2145.954316] Call Trace:
Dec  2 12:40:39 10.243.0.116 [ 2145.990151]  <IRQ>
Dec  2 12:40:39 10.243.0.116 [ 2146.025115]  ? __hrtimer_run_queues+0x5/0x780
Dec  2 12:40:39 10.243.0.116 [ 2146.062417]  ? hrtimer_interrupt+0x1aa/0x360
Dec  2 12:40:39 10.243.0.116 [ 2146.097515]  ? hrtimer_interrupt+0x5/0x360
Dec  2 12:40:39 10.243.0.116 [ 2146.132641]  ? proxy_irq_handler+0x37/0x50
Dec  2 12:40:39 10.243.0.116 [ 2146.166511]  ? handle_synthetic_irq+0xec/0x320
Dec  2 12:40:40 10.243.0.116 [ 2146.202196]  ? do_irq_inband+0x2f/0x40
Dec  2 12:40:40 10.243.0.116 [ 2146.236678]  ? arch_do_IRQ_pipelined+0xcd/0x5b0
Dec  2 12:40:40 10.243.0.116 [ 2146.270048]  </IRQ>
Dec  2 12:40:40 10.243.0.116 [ 2146.303312]  <TASK>
Dec  2 12:40:40 10.243.0.116 [ 2146.335917]  ? sync_current_irq_stage+0x2e3/0x500
Dec  2 12:40:40 10.243.0.116 [ 2146.368562]  ? sync_irq_stage+0x1af/0x1e0
Dec  2 12:40:40 10.243.0.116 [ 2146.399946]  ? synchronize_pipeline+0x8b/0xe0
Dec  2 12:40:40 10.243.0.116 [ 2146.430861]  ? handle_irq_pipelined_finish+0xf6/0x240
Dec  2 12:40:40 10.243.0.116 [ 2146.462658]  ? arch_pipeline_entry+0x81/0x110
Dec  2 12:40:40 10.243.0.116 [ 2146.488474]  ? sysvec_apic_timer_interrupt+0xe/0x20
Dec  2 12:40:40 10.243.0.116 [ 2146.513800]  ? asm_sysvec_apic_timer_interrupt+0x1b/0x20
Dec  2 12:40:40 10.243.0.116 [ 2146.539438]  ? rcu_all_qs+0xa0/0xa0
Dec  2 12:40:40 10.243.0.116 [ 2146.564914]  ? smp_call_function_single+0x118/0x210
Dec  2 12:40:40 10.243.0.116 [ 2146.589891]  ? __asan_store2+0xa0/0xa0
Dec  2 12:40:40 10.243.0.116 [ 2146.614324]  ? smp_call_function_single+0x118/0x210
Dec  2 12:40:40 10.243.0.116 [ 2146.639044]  ? 0xffffffffc0b38099
Dec  2 12:40:40 10.243.0.116 [ 2146.664836]  ? generic_exec_single+0x140/0x140
Dec  2 12:40:40 10.243.0.116 [ 2146.688627]  ? rcu_all_qs+0xa0/0xa0
Dec  2 12:40:40 10.243.0.116 [ 2146.710007]  ? smp_call_function_single+0x5/0x210
Dec  2 12:40:40 10.243.0.116 [ 2146.732444]  ? sync_rcu_exp_select_node_cpus+0x3df/0x5c0
Dec  2 12:40:40 10.243.0.116 [ 2146.754906]  ? sync_rcu_exp_select_cpus+0x2c5/0x5c0
Dec  2 12:40:40 10.243.0.116 [ 2146.776758]  ? wait_rcu_exp_gp+0x1f/0x30
Dec  2 12:40:40 10.243.0.116 [ 2146.797838]  ? process_one_work+0x525/0xa30
Dec  2 12:40:40 10.243.0.116 [ 2146.819412]  ? pwq_dec_nr_in_flight+0x120/0x120
Dec  2 12:40:40 10.243.0.116 [ 2146.840962]  ? process_one_work+0x5/0xa30
Dec  2 12:40:40 10.243.0.116 [ 2146.862506]  ? worker_thread+0x57/0x590
Dec  2 12:40:40 10.243.0.116 [ 2146.884014]  ? process_one_work+0xa30/0xa30
Dec  2 12:40:40 10.243.0.116 [ 2146.905731]  ? kthread+0x1f3/0x220
Dec  2 12:40:40 10.243.0.116 [ 2146.926954]  ? set_kthread_struct+0x90/0x90
Dec  2 12:40:40 10.243.0.116 [ 2146.950059]  ? ret_from_fork+0x22/0x30
Dec  2 12:40:40 10.243.0.116 [ 2146.977278]  </TASK>
Dec  2 12:40:50 10.243.0.116 [ 2156.449070] watchdog: BUG: soft lockup - CPU#0 stuck for 414s! [TxStartup:3574]
Dec  2 12:40:50 10.243.0.116 [ 2156.481983] Modules linked in: xt_CHECKSUM xt_MASQUERADE tun netconsole ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 xt_conntrack ebtable_nat ebtable_broute ip6table_nat ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_nat iptable_mangle iptable_security iptable_raw nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 libcrc32c ip_set nfnetlink ebtable_filter ebtables rfkill ip6table_filter ip6_tables iptable_filter sunrpc dm_mirror dm_region_hash dm_log dm_mod x86_pkg_temp_thermal intel_powerclamp coretemp gpio_ich iTCO_wdt iTCO_vendor_support kvm_intel mxm_wmi kvm irqbypass crct10dif_pclmul crc32_pclmul ghash_clmulni_intel aesni_intel crypto_simd cryptd rapl intel_cstate pcspkr sg i2c_i801 ftdi_sio mei_me i2c_smbus joydev input_leds intel_pch_thermal mei lpc_ich mfd_core ioatdma wmi ip_tables ext4 mbcache jbd2 ast drm_vram_helper drm_kms_helper syscopyarea ixgbe sysfillrect sd_mod sr_mod sysimgblt fb_sys_fops cdrom t10_pi drm_ttm_helper ttm mdio drm igb ah
Dec  2 12:40:50 10.243.0.116 [ 2156.495393]  libahci libata crc32c_intel ptp pps_core i2c_algo_bit dca fuse
Dec  2 12:40:50 10.243.0.116 [ 2156.799560] irq event stamp: 24564
Dec  2 12:40:50 10.243.0.116 [ 2156.833815] hardirqs last  enabled at (24563): [<ffffffff8d1d44ad>] sync_current_irq_stage+0x45d/0x500
Dec  2 12:40:50 10.243.0.116 [ 2156.870130] hardirqs last disabled at (24564): [<ffffffff8d49923b>] handle_mm_fault+0x48b/0x520
Dec  2 12:40:50 10.243.0.116 [ 2156.906486] softirqs last  enabled at (0): [<ffffffff8d0ca587>] copy_process+0x1307/0x3770
Dec  2 12:40:50 10.243.0.116 [ 2156.943090] softirqs last disabled at (0): [<0000000000000000>] 0x0
Dec  2 12:40:50 10.243.0.116 [ 2156.979285] CPU: 0 PID: 3574 Comm: TxStartup Tainted: G    B   W    L    5.15.77evl-g5e681f841fc3-dirty #3
Dec  2 12:40:50 10.243.0.116 [ 2157.016825] Hardware name: Supermicro Super Server/X10SDV-6C+-TLN4F, BIOS 2.1 11/22/2019
Dec  2 12:40:50 10.243.0.116 [ 2157.054296] IRQ stage: Linux
Dec  2 12:40:50 10.243.0.116 [ 2157.090530] RIP: 0010:proxy_set_next_ktime+0x77/0x90
Dec  2 12:40:50 10.243.0.116 [ 2157.127512] Code: 74 37 49 8d 7c 24 30 e8 d7 2d 1d 00 49 8b 44 24 30 4c 89 e7 e8 da 42 0b 01 48 8d 34 03 31 d2 4c 89 ef e8 5c 14 00 00 41 56 9d <31> c0 5b 41 5c 41 5d 41 5e 5d c3 cc cc cc cc e8 25 f5 eb ff eb d9
Dec  2 12:40:51 10.243.0.116 [ 2157.207540] RSP: 0018:ffff88841e209e80 EFLAGS: 00000283 ORIG_RAX: 0000000000000000
Dec  2 12:40:51 10.243.0.116 [ 2157.249500] RAX: 0000000000000000 RBX: 0000000000000000 RCX: 0000000000000000
Dec  2 12:40:51 10.243.0.116 [ 2157.291551] RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000000
Dec  2 12:40:51 10.243.0.116 [ 2157.333595] RBP: ffff88841e209ea0 R08: 0000000000000000 R09: 0000000000000000
Dec  2 12:40:51 10.243.0.116 [ 2157.375819] R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000000
Dec  2 12:40:51 10.243.0.116 [ 2157.418278] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000
Dec  2 12:40:51 10.243.0.116 [ 2157.460581] FS:  00007f301ae2e700(0000) GS:ffff88841e200000(0000) knlGS:0000000000000000
Dec  2 12:40:51 10.243.0.116 [ 2157.503492] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Dec  2 12:40:51 10.243.0.116 [ 2157.546156] CR2: 00007f1e4033b660 CR3: 0000000137d08006 CR4: 00000000003706f0
Dec  2 12:40:51 10.243.0.116 [ 2157.589617] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Dec  2 12:40:51 10.243.0.116 [ 2157.633717] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Dec  2 12:40:51 10.243.0.116 [ 2157.677966] Call Trace:
Dec  2 12:40:51 10.243.0.116 [ 2157.721222]  <IRQ>
Dec  2 12:40:51 10.243.0.116 [ 2157.764537]  ? __hrtimer_run_queues+0x5/0x780
Dec  2 12:40:51 10.243.0.116 [ 2157.808917]  ? hrtimer_interrupt+0x1aa/0x360
Dec  2 12:40:51 10.243.0.116 [ 2157.852856]  ? hrtimer_interrupt+0x5/0x360
Dec  2 12:40:51 10.243.0.116 [ 2157.896928]  ? proxy_irq_handler+0x37/0x50
Dec  2 12:40:51 10.243.0.116 [ 2157.940431]  ? handle_synthetic_irq+0xec/0x320
Dec  2 12:40:51 10.243.0.116 [ 2157.984332]  ? do_irq_inband+0x2f/0x40
Dec  2 12:40:51 10.243.0.116 [ 2158.027794]  ? arch_do_IRQ_pipelined+0xcd/0x5b0
Dec  2 12:40:51 10.243.0.116 [ 2158.071377]  </IRQ>
Dec  2 12:40:51 10.243.0.116 [ 2158.114053]  <TASK>
Dec  2 12:40:51 10.243.0.116 [ 2158.156950]  ? sync_current_irq_stage+0x2e3/0x500
Dec  2 12:40:52 10.243.0.116 [ 2158.201285]  ? sync_irq_stage+0x1af/0x1e0
Dec  2 12:40:52 10.243.0.116 [ 2158.244735]  ? synchronize_pipeline+0x8b/0xe0
Dec  2 12:40:52 10.243.0.116 [ 2158.288129]  ? handle_irq_pipelined_finish+0xf6/0x240
Dec  2 12:40:52 10.243.0.116 [ 2158.331899]  ? arch_pipeline_entry+0x81/0x110
Dec  2 12:40:52 10.243.0.116 [ 2158.374736]  ? sysvec_apic_timer_interrupt+0xe/0x20
Dec  2 12:40:52 10.243.0.116 [ 2158.416860]  ? asm_sysvec_apic_timer_interrupt+0x1b/0x20
Dec  2 12:40:52 10.243.0.116 [ 2158.459689]  ? smp_call_function_many_cond+0x1c7/0x4e0
Dec  2 12:40:52 10.243.0.116 [ 2158.501474]  ? __asan_load4+0x3f/0x90
Dec  2 12:40:52 10.243.0.116 [ 2158.542485]  ? smp_call_function_many_cond+0x1c7/0x4e0
Dec  2 12:40:52 10.243.0.116 [ 2158.583413]  ? leave_mm+0x40/0x40
Dec  2 12:40:52 10.243.0.116 [ 2158.623416]  ? x86_configure_nx+0x70/0x70
Dec  2 12:40:52 10.243.0.116 [ 2158.663405]  ? on_each_cpu_cond_mask+0x29/0x40
Dec  2 12:40:52 10.243.0.116 [ 2158.702779]  ? native_flush_tlb_multi+0xbe/0x250
Dec  2 12:40:52 10.243.0.116 [ 2158.741956]  ? flush_tlb_mm_range+0x136/0x1f0
Dec  2 12:40:52 10.243.0.116 [ 2158.780713]  ? change_protection_range+0x8bd/0xea0
Dec  2 12:40:52 10.243.0.116 [ 2158.821114]  ? prot_none_pte_entry+0x70/0x70
Dec  2 12:40:52 10.243.0.116 [ 2158.859112]  ? 0xffffffffc0b38099
Dec  2 12:40:52 10.243.0.116 [ 2158.896767]  ? change_protection_range+0x5/0xea0
Dec  2 12:40:52 10.243.0.116 [ 2158.934620]  ? change_protection+0x50/0x90
Dec  2 12:40:52 10.243.0.116 [ 2158.971564]  ? mprotect_fixup+0x2c5/0x490
Dec  2 12:40:52 10.243.0.116 [ 2159.008657]  ? change_protection+0x90/0x90
Dec  2 12:40:52 10.243.0.116 [ 2159.044903]  ? mprotect_fixup+0x5/0x490
Dec  2 12:40:52 10.243.0.116 [ 2159.081538]  ? do_mprotect_pkey+0x32a/0x590
Dec  2 12:40:52 10.243.0.116 [ 2159.116561]  ? mprotect_fixup+0x490/0x490
Dec  2 12:40:52 10.243.0.116 [ 2159.149112]  ? do_mprotect_pkey+0x5/0x590
Dec  2 12:40:52 10.243.0.116 [ 2159.181930]  ? __x64_sys_mprotect+0x48/0x60
Dec  2 12:40:53 10.243.0.116 [ 2159.213945]  ? do_syscall_64+0x4c/0xa0
Dec  2 12:40:53 10.243.0.116 [ 2159.245698]  ? entry_SYSCALL_64_after_hwframe+0x61/0xcb
Dec  2 12:40:53 10.243.0.116 [ 2159.279163]  </TASK>
Dec  2 12:41:06 10.243.0.116 [ 2172.448919] watchdog: BUG: soft lockup - CPU#2 stuck for 254s! [kworker/2:2:259]
Dec  2 12:41:06 10.243.0.116 [ 2172.472779] Modules linked in: xt_CHECKSUM xt_MASQUERADE tun netconsole ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 xt_conntrack ebtable_nat ebtable_broute ip6table_nat ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_nat iptable_mangle iptable_security iptable_raw nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 libcrc32c ip_set nfnetlink ebtable_filter ebtables rfkill ip6table_filter ip6_tables iptable_filter sunrpc dm_mirror dm_region_hash dm_log dm_mod x86_pkg_temp_thermal intel_powerclamp coretemp gpio_ich iTCO_wdt iTCO_vendor_support kvm_intel mxm_wmi kvm irqbypass crct10dif_pclmul crc32_pclmul ghash_clmulni_intel aesni_intel crypto_simd cryptd rapl intel_cstate pcspkr sg i2c_i801 ftdi_sio mei_me i2c_smbus joydev input_leds intel_pch_thermal mei lpc_ich mfd_core ioatdma wmi ip_tables ext4 mbcache jbd2 ast drm_vram_helper drm_kms_helper syscopyarea ixgbe sysfillrect sd_mod sr_mod sysimgblt fb_sys_fops cdrom t10_pi drm_ttm_helper ttm mdio drm igb ah
Dec  2 12:41:06 10.243.0.116 [ 2172.481940]  libahci libata crc32c_intel ptp pps_core i2c_algo_bit dca fuse
Dec  2 12:41:06 10.243.0.116 [ 2172.693098] irq event stamp: 23654
Dec  2 12:41:06 10.243.0.116 [ 2172.717182] hardirqs last  enabled at (23653): [<ffffffff8e1bd0d7>] _raw_spin_unlock_irq+0x27/0x40
Dec  2 12:41:06 10.243.0.116 [ 2172.742865] hardirqs last disabled at (23654): [<ffffffff8e1afe3d>] __schedule+0x90d/0x1160
Dec  2 12:41:06 10.243.0.116 [ 2172.768174] softirqs last  enabled at (23540): [<ffffffff8e400332>] __do_softirq+0x332/0x598
Dec  2 12:41:06 10.243.0.116 [ 2172.794073] softirqs last disabled at (23529): [<ffffffff8d0dc80b>] irq_exit_rcu+0xab/0x180
Dec  2 12:41:06 10.243.0.116 [ 2172.819870] CPU: 2 PID: 259 Comm: kworker/2:2 Tainted: G    B   W    L    5.15.77evl-g5e681f841fc3-dirty #3
Dec  2 12:41:06 10.243.0.116 [ 2172.846125] Hardware name: Supermicro Super Server/X10SDV-6C+-TLN4F, BIOS 2.1 11/22/2019
Dec  2 12:41:06 10.243.0.116 [ 2172.872453] IRQ stage: Linux
Dec  2 12:41:06 10.243.0.116 [ 2172.898319] Workqueue: rcu_gp wait_rcu_exp_gp
Dec  2 12:41:06 10.243.0.116 [ 2172.924692] RIP: 0010:proxy_set_next_ktime+0x77/0x90
Dec  2 12:41:06 10.243.0.116 [ 2172.950849] Code: 74 37 49 8d 7c 24 30 e8 d7 2d 1d 00 49 8b 44 24 30 4c 89 e7 e8 da 42 0b 01 48 8d 34 03 31 d2 4c 89 ef e8 5c 14 00 00 41 56 9d <31> c0 5b 41 5c 41 5d 41 5e 5d c3 cc cc cc cc e8 25 f5 eb ff eb d9
Dec  2 12:41:06 10.243.0.116 [ 2173.008828] RSP: 0018:ffff88841e309e80 EFLAGS: 00000216 ORIG_RAX: 0000000000000000
Dec  2 12:41:06 10.243.0.116 [ 2173.038096] RAX: 0000000000000000 RBX: 0000000000000000 RCX: 0000000000000000
Dec  2 12:41:06 10.243.0.116 [ 2173.067630] RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000000
Dec  2 12:41:06 10.243.0.116 [ 2173.096995] RBP: ffff88841e309ea0 R08: 0000000000000000 R09: 0000000000000000
Dec  2 12:41:06 10.243.0.116 [ 2173.126351] R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000000
Dec  2 12:41:06 10.243.0.116 [ 2173.155911] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000
Dec  2 12:41:06 10.243.0.116 [ 2173.185523] FS:  0000000000000000(0000) GS:ffff88841e300000(0000) knlGS:0000000000000000
Dec  2 12:41:07 10.243.0.116 [ 2173.215510] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Dec  2 12:41:07 10.243.0.116 [ 2173.245034] CR2: 00007f20be593000 CR3: 000000012c0f0004 CR4: 00000000003706e0
Dec  2 12:41:07 10.243.0.116 [ 2173.275141] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Dec  2 12:41:07 10.243.0.116 [ 2173.306146] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Dec  2 12:41:07 10.243.0.116 [ 2173.336932] Call Trace:
Dec  2 12:41:07 10.243.0.116 [ 2173.366903]  <IRQ>
Dec  2 12:41:07 10.243.0.116 [ 2173.397045]  ? __hrtimer_run_queues+0x5/0x780
Dec  2 12:41:07 10.243.0.116 [ 2173.427939]  ? hrtimer_interrupt+0x1aa/0x360
Dec  2 12:41:07 10.243.0.116 [ 2173.458627]  ? hrtimer_interrupt+0x5/0x360
Dec  2 12:41:07 10.243.0.116 [ 2173.489188]  ? proxy_irq_handler+0x37/0x50
Dec  2 12:41:07 10.243.0.116 [ 2173.519684]  ? handle_synthetic_irq+0xec/0x320
Dec  2 12:41:07 10.243.0.116 [ 2173.550090]  ? do_irq_inband+0x2f/0x40
Dec  2 12:41:07 10.243.0.116 [ 2173.580331]  ? arch_do_IRQ_pipelined+0xcd/0x5b0
Dec  2 12:41:07 10.243.0.116 [ 2173.612806]  </IRQ>
Dec  2 12:41:07 10.243.0.116 [ 2173.642398]  <TASK>
Dec  2 12:41:07 10.243.0.116 [ 2173.675088]  ? sync_current_irq_stage+0x2e3/0x500
Dec  2 12:41:07 10.243.0.116 [ 2173.703503] rcu: INFO: rcu_sched detected stalls on CPUs/tasks:
Dec  2 12:41:07 10.243.0.116 [ 2173.706116]  ? sync_irq_stage+0x1af/0x1e0
Dec  2 12:41:07 10.243.0.116 [ 2173.737195] rcu: #0115-...0: (1 GPs behind) idle=78b/1/0x4000000000000002 softirq=161154/161154 fqs=64098
Dec  2 12:41:07 10.243.0.116 [ 2173.767830]  ? synchronize_pipeline+0x8b/0xe0
Dec  2 12:41:07 10.243.0.116 [ 2173.800157] #011(detected by 7, t=458269 jiffies, g=126833, q=20346)
Dec  2 12:41:07 10.243.0.116 [ 2173.830748]  ? handle_irq_pipelined_finish+0xf6/0x240
Dec  2 12:41:07 10.243.0.116 [ 2173.862046] Sending NMI from CPU 7 to CPUs 5:
Dec  2 12:41:07 10.243.0.116 [ 2173.892149]  ? arch_pipeline_entry+0x81/0x110
Dec  2 12:41:07 10.243.0.116 [ 2173.922817] NMI backtrace for cpu 5
Dec  2 12:41:07 10.243.0.116 [ 2173.922859] CPU: 5 PID: 3577 Comm: DummyTxSample Tainted: G    B   W    L    5.15.77evl-g5e681f841fc3-dirty #3
Dec  2 12:41:07 10.243.0.116 [ 2173.922896] Hardware name: Supermicro Super Server/X10SDV-6C+-TLN4F, BIOS 2.1 11/22/2019
Dec  2 12:41:07 10.243.0.116 [ 2173.922912] IRQ stage: EVL
Dec  2 12:41:07 10.243.0.116 [ 2173.922932] RIP: 0010:__asan_load4+0x0/0x90
Dec  2 12:41:07 10.243.0.116 [ 2173.922965] Code: c0 84 c0 75 e1 5d c3 cc cc cc cc 48 c1 e8 03 80 3c 10 00 75 e9 5d c3 cc cc cc cc 0f 1f 44 00 00 66 2e 0f 1f 84 00 00 00 00 00 <55> 48 89 e5 48 8b 4d 08 48 83 ff fb 77 53 48 b8 ff ff ff ff ff 7f
Dec  2 12:41:07 10.243.0.116 [ 2173.922984] RSP: 0018:ffff888137737530 EFLAGS: 00000002
Dec  2 12:41:07 10.243.0.116 [ 2173.923016] RAX: 00000000001c0101 RBX: ffff888101aae000 RCX: ffffffff8d1abe74
Dec  2 12:41:08 10.243.0.116 [ 2173.923034] RDX: 0000000000000003 RSI: dffffc0000000000 RDI: ffff888101aae000
Dec  2 12:41:08 10.243.0.116 [ 2173.923051] RBP: ffff888137737618 R08: ffffed1020355c01 R09: ffffed1020355c01
Dec  2 12:41:08 10.243.0.116 [ 2173.923068] R10: ffff888101aae003 R11: ffffed1020355c00 R12: 1ffff11026ee6eaa
Dec  2 12:41:08 10.243.0.116 [ 2173.923085] R13: 0000000000000000 R14: ffff8881377375f0 R15: 0000000000000001
Dec  2 12:41:08 10.243.0.116 [ 2173.923112] FS:  00007f3013fff700(0000) GS:ffff88841e480000(0000) knlGS:0000000000000000
Dec  2 12:41:08 10.243.0.116 [ 2173.923131] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Dec  2 12:41:08 10.243.0.116 [ 2173.923149] CR2: 0000000000000070 CR3: 0000000137d08002 CR4: 00000000003706e0
Dec  2 12:41:08 10.243.0.116 [ 2173.923166] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Dec  2 12:41:08 10.243.0.116 [ 2173.923182] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Dec  2 12:41:08 10.243.0.116 [ 2173.923200] Call Trace:
Dec  2 12:41:08 10.243.0.116 [ 2173.923219]  <TASK>
Dec  2 12:41:08 10.243.0.116 [ 2173.923257]  ? native_queued_spin_lock_slowpath+0x194/0x4c0
Dec  2 12:41:08 10.243.0.116 [ 2173.923419]  ? 0xffffffffc0b38099
Dec  2 12:41:08 10.243.0.116 [ 2173.923541]  ? trace_function+0x11e/0x180
Dec  2 12:41:08 10.243.0.116 [ 2173.923766]  ? .slowpath+0x1d/0x1d
Dec  2 12:41:08 10.243.0.116 [ 2173.924082]  ? do_raw_spin_lock+0xe6/0x1e0
Dec  2 12:41:08 10.243.0.116 [ 2173.924286]  ? native_queued_spin_lock_slowpath+0x5/0x4c0
Dec  2 12:41:08 10.243.0.116 [ 2173.924443]  ? 0xffffffffc0b38099
Dec  2 12:41:08 10.243.0.116 [ 2173.924996]  do_raw_spin_lock+0x1cb/0x1e0
Dec  2 12:41:08 10.243.0.116 [ 2173.925098]  ? spin_dump+0x90/0x90
Dec  2 12:41:08 10.243.0.116 [ 2173.925320]  ? do_raw_spin_lock+0x5/0x1e0
Dec  2 12:41:08 10.243.0.116 [ 2173.925985]  evl_switch_inband+0xad/0x7c0
Dec  2 12:41:08 10.243.0.116 [ 2173.926460]  handle_oob_trap_entry+0x10a/0x200
Dec  2 12:41:08 10.243.0.116 [ 2173.926915]  __oob_trap_notify+0x3f/0x50
Dec  2 12:41:08 10.243.0.116 [ 2173.927085]  page_fault_oops+0x1e4/0x3d0
Dec  2 12:41:08 10.243.0.116 [ 2173.927155]  ? trace_function+0x11e/0x180
Dec  2 12:41:08 10.243.0.116 [ 2173.927220]  ? do_user_addr_fault+0x4af/0xa30
Dec  2 12:41:08 10.243.0.116 [ 2173.927519]  ? spurious_kernel_fault+0x200/0x200
Dec  2 12:41:08 10.243.0.116 [ 2173.927745]  ? function_trace_call+0x153/0x1d0
Dec  2 12:41:09 10.243.0.116 [ 2173.927814]  ? do_user_addr_fault+0x4af/0xa30
Dec  2 12:41:09 10.243.0.116 [ 2173.927885]  ? spurious_kernel_fault+0x200/0x200
Dec  2 12:41:09 10.243.0.116 [ 2173.928128]  ? 0xffffffffc0b38099
Dec  2 12:41:09 10.243.0.116 [ 2173.928591]  ? do_user_addr_fault+0xa30/0xa30
Dec  2 12:41:09 10.243.0.116 [ 2173.928905]  ? do_user_addr_fault+0x686/0xa30
Dec  2 12:41:09 10.243.0.116 [ 2173.929072]  ? page_fault_oops+0x5/0x3d0
Dec  2 12:41:09 10.243.0.116 [ 2173.929139]  ? exc_page_fault+0x5c/0xe0
Dec  2 12:41:09 10.243.0.116 [ 2173.929718]  do_user_addr_fault+0x4af/0xa30
Dec  2 12:41:09 10.243.0.116 [ 2173.929796]  ? rcu_read_lock_held_common+0x5/0x70
Dec  2 12:41:09 10.243.0.116 [ 2173.929892]  ? rcu_read_lock_held_common+0x2a/0x70
Dec  2 12:41:09 10.243.0.116 [ 2173.930159]  ? trace_page_fault_user+0x130/0x130
Dec  2 12:41:09 10.243.0.116 [ 2173.930461]  ? fault_in_kernel_space+0x5/0x40
Dec  2 12:41:09 10.243.0.116 [ 2173.931048]  exc_page_fault+0x71/0xe0
Dec  2 12:41:09 10.243.0.116 [ 2173.931288]  asm_exc_page_fault+0x27/0x30
Dec  2 12:41:09 10.243.0.116 [ 2173.931366] RIP: 0010:trace_event_raw_event_evl_sleep_on+0xc3/0x380
Dec  2 12:41:09 10.243.0.116 [ 2173.931438] Code: 80 0f 85 93 02 00 00 41 f6 c7 40 0f 85 10 02 00 00 41 f7 c7 00 02 00 00 0f 85 8f 02 00 00 4d 8d 7d 70 4c 89 ff e8 6d e9 1e 00 <49> 8b 7d 70 48 85 ff 0f 84 2d 02 00 00 e8 ab b4 58 00 8d 50 01 83
Dec  2 12:41:09 10.243.0.116 [ 2173.931478] RSP: 0018:ffff888137737aa0 EFLAGS: 00010082
Dec  2 12:41:09 10.243.0.116 [ 2173.931545] RAX: 0000000000000001 RBX: 1ffff11026ee6f5a RCX: ffffffff8d1acefc
Dec  2 12:41:09 10.243.0.116 [ 2173.931583] RDX: 0000000000000003 RSI: dffffc0000000000 RDI: ffffffff8f2acb40
Dec  2 12:41:09 10.243.0.116 [ 2173.931620] RBP: ffff888137737b78 R08: fffffbfff1e55969 R09: fffffbfff1e55969
Dec  2 12:41:09 10.243.0.116 [ 2173.931659] R10: ffffffff8f2acb43 R11: fffffbfff1e55968 R12: ffff888100195000
Dec  2 12:41:09 10.243.0.116 [ 2173.931697] R13: 0000000000000000 R14: ffffffff8f219d40 R15: 0000000000000070
Dec  2 12:41:09 10.243.0.116 [ 2173.932173]  ? do_raw_spin_unlock+0x9c/0x140
Dec  2 12:41:09 10.243.0.116 [ 2173.933102]  ? trace_event_raw_event_evl_init_thread+0x330/0x330
Dec  2 12:41:09 10.243.0.116 [ 2173.933345]  ? 0xffffffffc0b38099
Dec  2 12:41:09 10.243.0.116 [ 2173.933467]  ? evl_delay+0x48/0xf0
Dec  2 12:41:09 10.243.0.116 [ 2173.933536]  ? evl_sleep_on_locked+0x730/0x730
Dec  2 12:41:09 10.243.0.116 [ 2173.933878]  ? __kasan_check_write+0x14/0x20
Dec  2 12:41:09 10.243.0.116 [ 2173.934147]  evl_sleep_on_locked+0x629/0x730
Dec  2 12:41:09 10.243.0.116 [ 2173.934215]  ? evl_sleep_on_locked+0x5/0x730
Dec  2 12:41:09 10.243.0.116 [ 2173.934937]  evl_sleep_on+0x7b/0xb0
Dec  2 12:41:09 10.243.0.116 [ 2173.935266]  evl_delay+0x48/0xf0
Dec  2 12:41:10 10.243.0.116 [ 2173.935801]  clock_oob_ioctl+0x182/0x2d0
Dec  2 12:41:10 10.243.0.116 [ 2173.935998]  ? clock_common_ioctl+0x5e0/0x5e0
Dec  2 12:41:10 10.243.0.116 [ 2173.936166]  ? clock_oob_ioctl+0x5/0x2d0
Dec  2 12:41:10 10.243.0.116 [ 2173.936888]  EVL_ioctl+0x86/0x100
Dec  2 12:41:10 10.243.0.116 [ 2173.937157]  do_oob_syscall+0x5b1/0x6d0
Dec  2 12:41:10 10.243.0.116 [ 2173.937538]  ? prepare_for_signal+0x180/0x180
Dec  2 12:41:10 10.243.0.116 [ 2173.937879]  ? do_oob_syscall+0x5/0x6d0
Dec  2 12:41:10 10.243.0.116 [ 2173.937944]  ? handle_pipelined_syscall+0x840/0x840
Dec  2 12:41:10 10.243.0.116 [ 2173.938277]  handle_oob_syscall+0x18e/0x230
Dec  2 12:41:10 10.243.0.116 [ 2173.938736]  ? handle_pipelined_syscall+0x840/0x840
Dec  2 12:41:10 10.243.0.116 [ 2173.939100]  ? handle_oob_syscall+0x5/0x230
Dec  2 12:41:10 10.243.0.116 [ 2173.939784]  pipeline_syscall+0xac/0x1c0
Dec  2 12:41:10 10.243.0.116 [ 2173.940092]  syscall_enter_from_user_mode+0x5f/0x130
Dec  2 12:41:10 10.243.0.116 [ 2173.940313]  do_syscall_64+0x1d/0xa0
Dec  2 12:41:10 10.243.0.116 [ 2173.940544]  entry_SYSCALL_64_after_hwframe+0x61/0xcb
Dec  2 12:41:10 10.243.0.116 [ 2173.940621] RIP: 0033:0x7f302d4b1b9b
Dec  2 12:41:10 10.243.0.116 [ 2173.940729] Code: Unable to access opcode bytes at RIP 0x7f302d4b1b71.
Dec  2 12:41:10 10.243.0.116 [ 2173.940765] RSP: 002b:00007f3013fef5a0 EFLAGS: 00000246 ORIG_RAX: 000000000000009d
Dec  2 12:41:10 10.243.0.116 [ 2173.940837] RAX: ffffffffffffffda RBX: 00007f3013fef630 RCX: 00007f302d4b1b9b
Dec  2 12:41:10 10.243.0.116 [ 2173.940862] RDX: 0000000040106300 RSI: 0000000000000004 RDI: 0000000010000002
Dec  2 12:41:10 10.243.0.116 [ 2173.940879] RBP: 0000000040106300 R08: 0000000000000000 R09: 00007ffd9df03080
Dec  2 12:41:10 10.243.0.116 [ 2173.940895] R10: 00007f3013fef630 R11: 0000000000000246 R12: 0000000000000004
Dec  2 12:41:10 10.243.0.116 [ 2173.940912] R13: 0000000001001000 R14: 0000000000000000 R15: 00007f3013fff700
Dec  2 12:41:10 10.243.0.116 [ 2173.941989]  </TASK>
Dec  2 12:41:11 10.243.0.116 [ 2176.435476] igb 0000:05:00.0: Detected Tx Unit Hang#012[ 2176.435476]   Tx Queue             <7>#012[ 2176.435476]   TDH                  <6c>#012[ 2176.435476]   TDT                  <6c>#012[ 2176.435476]   next_to_use          <6c>#012[ 2176.435476]   next_to_clean        <1>#012[ 2176.435476] buffer_info[next_to_clean]#012[ 2176.435476]   time_stamp           <1001c8f4b>#012[ 2176.435476]   next_to_watch        <0000000020e55410>#012[ 2176.435476]   jiffies              <1001c9c90>#012[ 2176.435476]   desc.status          <200001>
Dec  2 12:41:11 10.243.0.116 [ 2177.456207]  ? sysvec_apic_timer_interrupt+0xe/0x20
Dec  2 12:41:11 10.243.0.116 [ 2177.481209]  ? asm_sysvec_apic_timer_interrupt+0x1b/0x20
Dec  2 12:41:11 10.243.0.116 [ 2177.507030]  ? rcu_all_qs+0xa0/0xa0
Dec  2 12:41:11 10.243.0.116 [ 2177.531975]  ? smp_call_function_single+0x118/0x210
Dec  2 12:41:11 10.243.0.116 [ 2177.557938]  ? __asan_load4+0x43/0x90
Dec  2 12:41:11 10.243.0.116 [ 2177.582713]  ? smp_call_function_single+0x118/0x210
Dec  2 12:41:11 10.243.0.116 [ 2177.607685]  ? 0xffffffffc0b38099
Dec  2 12:41:11 10.243.0.116 [ 2177.633230]  ? generic_exec_single+0x140/0x140
Dec  2 12:41:11 10.243.0.116 [ 2177.658763]  ? rcu_all_qs+0xa0/0xa0
Dec  2 12:41:11 10.243.0.116 [ 2177.683507]  ? smp_call_function_single+0x5/0x210
Dec  2 12:41:11 10.243.0.116 [ 2177.710811]  ? sync_rcu_exp_select_node_cpus+0x3df/0x5c0
Dec  2 12:41:11 10.243.0.116 [ 2177.737181]  ? sync_rcu_exp_select_cpus+0x2c5/0x5c0
Dec  2 12:41:11 10.243.0.116 [ 2177.763199]  ? wait_rcu_exp_gp+0x1f/0x30
Dec  2 12:41:11 10.243.0.116 [ 2177.789914]  ? process_one_work+0x525/0xa30
Dec  2 12:41:11 10.243.0.116 [ 2177.815411]  ? pwq_dec_nr_in_flight+0x120/0x120
Dec  2 12:41:11 10.243.0.116 [ 2177.841478]  ? process_one_work+0x5/0xa30
Dec  2 12:41:11 10.243.0.116 [ 2177.867761]  ? worker_thread+0x57/0x590
Dec  2 12:41:11 10.243.0.116 [ 2177.894138]  ? process_one_work+0xa30/0xa30
Dec  2 12:41:11 10.243.0.116 [ 2177.919636]  ? kthread+0x1f3/0x220
Dec  2 12:41:11 10.243.0.116 [ 2177.944856]  ? set_kthread_struct+0x90/0x90
Dec  2 12:41:11 10.243.0.116 [ 2177.971028]  ? ret_from_fork+0x22/0x30
Dec  2 12:41:11 10.243.0.116 [ 2177.998311]  </TASK>
Dec  2 12:41:18 10.243.0.116 [ 2184.449193] watchdog: BUG: soft lockup - CPU#0 stuck for 440s! [TxStartup:3574]
Dec  2 12:41:18 10.243.0.116 [ 2184.482092] Modules linked in: xt_CHECKSUM xt_MASQUERADE tun netconsole ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 xt_conntrack ebtable_nat ebtable_broute ip6table_nat ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_nat iptable_mangle iptable_security iptable_raw nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 libcrc32c ip_set nfnetlink ebtable_filter ebtables rfkill ip6table_filter ip6_tables iptable_filter sunrpc dm_mirror dm_region_hash dm_log dm_mod x86_pkg_temp_thermal intel_powerclamp coretemp gpio_ich iTCO_wdt iTCO_vendor_support kvm_intel mxm_wmi kvm irqbypass crct10dif_pclmul crc32_pclmul ghash_clmulni_intel aesni_intel crypto_simd cryptd rapl intel_cstate pcspkr sg i2c_i801 ftdi_sio mei_me i2c_smbus joydev input_leds intel_pch_thermal mei lpc_ich mfd_core ioatdma wmi ip_tables ext4 mbcache jbd2 ast drm_vram_helper drm_kms_helper syscopyarea ixgbe sysfillrect sd_mod sr_mod sysimgblt fb_sys_fops cdrom t10_pi drm_ttm_helper ttm mdio drm igb ah
Dec  2 12:41:18 10.243.0.116 [ 2184.495504]  libahci libata crc32c_intel ptp pps_core i2c_algo_bit dca fuse
Dec  2 12:41:18 10.243.0.116 [ 2184.808210] irq event stamp: 24564
Dec  2 12:41:18 10.243.0.116 [ 2184.843604] hardirqs last  enabled at (24563): [<ffffffff8d1d44ad>] sync_current_irq_stage+0x45d/0x500
Dec  2 12:41:18 10.243.0.116 [ 2184.880807] hardirqs last disabled at (24564): [<ffffffff8d49923b>] handle_mm_fault+0x48b/0x520
Dec  2 12:41:18 10.243.0.116 [ 2184.917645] softirqs last  enabled at (0): [<ffffffff8d0ca587>] copy_process+0x1307/0x3770
Dec  2 12:41:18 10.243.0.116 [ 2184.954352] softirqs last disabled at (0): [<0000000000000000>] 0x0
Dec  2 12:41:18 10.243.0.116 [ 2184.990422] CPU: 0 PID: 3574 Comm: TxStartup Tainted: G    B   W    L    5.15.77evl-g5e681f841fc3-dirty #3
Dec  2 12:41:18 10.243.0.116 [ 2185.027726] Hardware name: Supermicro Super Server/X10SDV-6C+-TLN4F, BIOS 2.1 11/22/2019
Dec  2 12:41:18 10.243.0.116 [ 2185.065085] IRQ stage: Linux
Dec  2 12:41:18 10.243.0.116 [ 2185.101505] RIP: 0010:proxy_set_next_ktime+0x77/0x90
Dec  2 12:41:18 10.243.0.116 [ 2185.138806] Code: 74 37 49 8d 7c 24 30 e8 d7 2d 1d 00 49 8b 44 24 30 4c 89 e7 e8 da 42 0b 01 48 8d 34 03 31 d2 4c 89 ef e8 5c 14 00 00 41 56 9d <31> c0 5b 41 5c 41 5d 41 5e 5d c3 cc cc cc cc e8 25 f5 eb ff eb d9
Dec  2 12:41:19 10.243.0.116 [ 2185.219458] RSP: 0018:ffff88841e209e80 EFLAGS: 00000293 ORIG_RAX: 0000000000000000
Dec  2 12:41:19 10.243.0.116 [ 2185.261756] RAX: 0000000000000000 RBX: 0000000000000000 RCX: 0000000000000000
Dec  2 12:41:19 10.243.0.116 [ 2185.304098] RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000000
Dec  2 12:41:19 10.243.0.116 [ 2185.346507] RBP: ffff88841e209ea0 R08: 0000000000000000 R09: 0000000000000000
Dec  2 12:41:19 10.243.0.116 [ 2185.389030] R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000000
Dec  2 12:41:19 10.243.0.116 [ 2185.431664] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000
Dec  2 12:41:19 10.243.0.116 [ 2185.474153] FS:  00007f301ae2e700(0000) GS:ffff88841e200000(0000) knlGS:0000000000000000
Dec  2 12:41:19 10.243.0.116 [ 2185.517254] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Dec  2 12:41:19 10.243.0.116 [ 2185.560089] CR2: 00007f1e4033b660 CR3: 0000000137d08006 CR4: 00000000003706f0
Dec  2 12:41:19 10.243.0.116 [ 2185.603683] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Dec  2 12:41:19 10.243.0.116 [ 2185.647936] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Dec  2 12:41:19 10.243.0.116 [ 2185.692334] Call Trace:
Dec  2 12:41:19 10.243.0.116 [ 2185.735795]  <IRQ>
Dec  2 12:41:19 10.243.0.116 [ 2185.779309]  ? __hrtimer_run_queues+0x5/0x780
Dec  2 12:41:19 10.243.0.116 [ 2185.823902]  ? hrtimer_interrupt+0x1aa/0x360
Dec  2 12:41:19 10.243.0.116 [ 2185.868093]  ? hrtimer_interrupt+0x5/0x360
Dec  2 12:41:19 10.243.0.116 [ 2185.912562]  ? proxy_irq_handler+0x37/0x50
Dec  2 12:41:19 10.243.0.116 [ 2185.956266]  ? handle_synthetic_irq+0xec/0x320
Dec  2 12:41:19 10.243.0.116 [ 2186.000431]  ? do_irq_inband+0x2f/0x40
Dec  2 12:41:19 10.243.0.116 [ 2186.044110]  ? arch_do_IRQ_pipelined+0xcd/0x5b0
Dec  2 12:41:19 10.243.0.116 [ 2186.087778]  </IRQ>
Dec  2 12:41:19 10.243.0.116 [ 2186.130553]  <TASK>
Dec  2 12:41:19 10.243.0.116 [ 2186.173534]  ? sync_current_irq_stage+0x2e3/0x500
Dec  2 12:41:20 10.243.0.116 [ 2186.217906]  ? sync_irq_stage+0x1af/0x1e0
Dec  2 12:41:20 10.243.0.116 [ 2186.261397]  ? synchronize_pipeline+0x8b/0xe0
Dec  2 12:41:20 10.243.0.116 [ 2186.304887]  ? handle_irq_pipelined_finish+0xf6/0x240
Dec  2 12:41:20 10.243.0.116 [ 2186.348756]  ? arch_pipeline_entry+0x81/0x110
Dec  2 12:41:20 10.243.0.116 [ 2186.391742]  ? sysvec_apic_timer_interrupt+0xe/0x20
Dec  2 12:41:20 10.243.0.116 [ 2186.434005]  ? asm_sysvec_apic_timer_interrupt+0x1b/0x20
Dec  2 12:41:20 10.243.0.116 [ 2186.476894]  ? smp_call_function_many_cond+0x1c7/0x4e0
Dec  2 12:41:20 10.243.0.116 [ 2186.518832]  ? smp_call_function_many_cond+0x1bf/0x4e0
Dec  2 12:41:20 10.243.0.116 [ 2186.560149]  ? smp_call_function_many_cond+0x1c7/0x4e0
Dec  2 12:41:20 10.243.0.116 [ 2186.600865]  ? leave_mm+0x40/0x40
Dec  2 12:41:20 10.243.0.116 [ 2186.640662]  ? x86_configure_nx+0x70/0x70
Dec  2 12:41:20 10.243.0.116 [ 2186.680446]  ? on_each_cpu_cond_mask+0x29/0x40
Dec  2 12:41:20 10.243.0.116 [ 2186.719677]  ? native_flush_tlb_multi+0xbe/0x250
Dec  2 12:41:20 10.243.0.116 [ 2186.758644]  ? flush_tlb_mm_range+0x136/0x1f0
Dec  2 12:41:20 10.243.0.116 [ 2186.797431]  ? change_protection_range+0x8bd/0xea0
Dec  2 12:41:20 10.243.0.116 [ 2186.837789]  ? prot_none_pte_entry+0x70/0x70
Dec  2 12:41:20 10.243.0.116 [ 2186.875699]  ? 0xffffffffc0b38099
Dec  2 12:41:20 10.243.0.116 [ 2186.913217]  ? change_protection_range+0x5/0xea0
Dec  2 12:41:20 10.243.0.116 [ 2186.950934]  ? change_protection+0x50/0x90
Dec  2 12:41:20 10.243.0.116 [ 2186.987737]  ? mprotect_fixup+0x2c5/0x490
Dec  2 12:41:20 10.243.0.116 [ 2187.024688]  ? change_protection+0x90/0x90
Dec  2 12:41:20 10.243.0.116 [ 2187.060788]  ? mprotect_fixup+0x5/0x490
Dec  2 12:41:20 10.243.0.116 [ 2187.097277]  ? do_mprotect_pkey+0x32a/0x590
Dec  2 12:41:20 10.243.0.116 [ 2187.132155]  ? mprotect_fixup+0x490/0x490
Dec  2 12:41:20 10.243.0.116 [ 2187.164626]  ? do_mprotect_pkey+0x5/0x590
Dec  2 12:41:20 10.243.0.116 [ 2187.197302]  ? __x64_sys_mprotect+0x48/0x60
Dec  2 12:41:21 10.243.0.116 [ 2187.229167]  ? do_syscall_64+0x4c/0xa0
Dec  2 12:41:21 10.243.0.116 [ 2187.260767]  ? entry_SYSCALL_64_after_hwframe+0x61/0xcb
Dec  2 12:41:21 10.243.0.116 [ 2187.294132]  </TASK>
Dec  2 12:41:34 10.243.0.116 [ 2200.449047] watchdog: BUG: soft lockup - CPU#2 stuck for 280s! [kworker/2:2:259]
Dec  2 12:41:34 10.243.0.116 [ 2200.472753] Modules linked in: xt_CHECKSUM xt_MASQUERADE tun netconsole ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 xt_conntrack ebtable_nat ebtable_broute ip6table_nat ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_nat iptable_mangle iptable_security iptable_raw nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 libcrc32c ip_set nfnetlink ebtable_filter ebtables rfkill ip6table_filter ip6_tables iptable_filter sunrpc dm_mirror dm_region_hash dm_log dm_mod x86_pkg_temp_thermal intel_powerclamp coretemp gpio_ich iTCO_wdt iTCO_vendor_support kvm_intel mxm_wmi kvm irqbypass crct10dif_pclmul crc32_pclmul ghash_clmulni_intel aesni_intel crypto_simd cryptd rapl intel_cstate pcspkr sg i2c_i801 ftdi_sio mei_me i2c_smbus joydev input_leds intel_pch_thermal mei lpc_ich mfd_core ioatdma wmi ip_tables ext4 mbcache jbd2 ast drm_vram_helper drm_kms_helper syscopyarea ixgbe sysfillrect sd_mod sr_mod sysimgblt fb_sys_fops cdrom t10_pi drm_ttm_helper ttm mdio drm igb ah
Dec  2 12:41:34 10.243.0.116 [ 2200.481997]  libahci libata crc32c_intel ptp pps_core i2c_algo_bit dca fuse
Dec  2 12:41:34 10.243.0.116 [ 2200.691783] irq event stamp: 23654
Dec  2 12:41:34 10.243.0.116 [ 2200.715628] hardirqs last  enabled at (23653): [<ffffffff8e1bd0d7>] _raw_spin_unlock_irq+0x27/0x40
Dec  2 12:41:34 10.243.0.116 [ 2200.741161] hardirqs last disabled at (23654): [<ffffffff8e1afe3d>] __schedule+0x90d/0x1160
Dec  2 12:41:34 10.243.0.116 [ 2200.766289] softirqs last  enabled at (23540): [<ffffffff8e400332>] __do_softirq+0x332/0x598
Dec  2 12:41:34 10.243.0.116 [ 2200.791821] softirqs last disabled at (23529): [<ffffffff8d0dc80b>] irq_exit_rcu+0xab/0x180
Dec  2 12:41:34 10.243.0.116 [ 2200.817167] CPU: 2 PID: 259 Comm: kworker/2:2 Tainted: G    B   W    L    5.15.77evl-g5e681f841fc3-dirty #3
Dec  2 12:41:34 10.243.0.116 [ 2200.843479] Hardware name: Supermicro Super Server/X10SDV-6C+-TLN4F, BIOS 2.1 11/22/2019
Dec  2 12:41:34 10.243.0.116 [ 2200.869540] IRQ stage: Linux
Dec  2 12:41:34 10.243.0.116 [ 2200.894964] Workqueue: rcu_gp wait_rcu_exp_gp
Dec  2 12:41:34 10.243.0.116 [ 2200.920962] RIP: 0010:proxy_set_next_ktime+0x77/0x90
Dec  2 12:41:34 10.243.0.116 [ 2200.946808] Code: 74 37 49 8d 7c 24 30 e8 d7 2d 1d 00 49 8b 44 24 30 4c 89 e7 e8 da 42 0b 01 48 8d 34 03 31 d2 4c 89 ef e8 5c 14 00 00 41 56 9d <31> c0 5b 41 5c 41 5d 41 5e 5d c3 cc cc cc cc e8 25 f5 eb ff eb d9
Dec  2 12:41:34 10.243.0.116 [ 2201.003069] RSP: 0018:ffff88841e309e80 EFLAGS: 00000206 ORIG_RAX: 0000000000000000
Dec  2 12:41:34 10.243.0.116 [ 2201.032233] RAX: 0000000000000000 RBX: 0000000000000000 RCX: 0000000000000000
Dec  2 12:41:34 10.243.0.116 [ 2201.061761] RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000000
Dec  2 12:41:34 10.243.0.116 [ 2201.090855] RBP: ffff88841e309ea0 R08: 0000000000000000 R09: 0000000000000000
Dec  2 12:41:34 10.243.0.116 [ 2201.120085] R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000000
Dec  2 12:41:34 10.243.0.116 [ 2201.149416] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000
Dec  2 12:41:34 10.243.0.116 [ 2201.179358] FS:  0000000000000000(0000) GS:ffff88841e300000(0000) knlGS:0000000000000000
Dec  2 12:41:35 10.243.0.116 [ 2201.209228] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Dec  2 12:41:35 10.243.0.116 [ 2201.238797] CR2: 00007f20be593000 CR3: 000000012c0f0004 CR4: 00000000003706e0
Dec  2 12:41:35 10.243.0.116 [ 2201.269139] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Dec  2 12:41:35 10.243.0.116 [ 2201.299747] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Dec  2 12:41:35 10.243.0.116 [ 2201.330320] Call Trace:
Dec  2 12:41:35 10.243.0.116 [ 2201.360219]  <IRQ>
Dec  2 12:41:35 10.243.0.116 [ 2201.390402]  ? __hrtimer_run_queues+0x5/0x780
Dec  2 12:41:35 10.243.0.116 [ 2201.421189]  ? hrtimer_interrupt+0x1aa/0x360
Dec  2 12:41:35 10.243.0.116 [ 2201.451999]  ? hrtimer_interrupt+0x5/0x360
Dec  2 12:41:35 10.243.0.116 [ 2201.482889]  ? proxy_irq_handler+0x37/0x50
Dec  2 12:41:35 10.243.0.116 [ 2201.513006]  ? handle_synthetic_irq+0xec/0x320
Dec  2 12:41:35 10.243.0.116 [ 2201.543224]  ? do_irq_inband+0x2f/0x40
Dec  2 12:41:35 10.243.0.116 [ 2201.573586]  ? arch_do_IRQ_pipelined+0xcd/0x5b0
Dec  2 12:41:35 10.243.0.116 [ 2201.603958]  </IRQ>
Dec  2 12:41:35 10.243.0.116 [ 2201.633424]  <TASK>
Dec  2 12:41:35 10.243.0.116 [ 2201.665784]  ? sync_current_irq_stage+0x2e3/0x500
Dec  2 12:41:35 10.243.0.116 [ 2201.697221]  ? sync_irq_stage+0x1af/0x1e0
Dec  2 12:41:35 10.243.0.116 [ 2201.734042]  ? synchronize_pipeline+0x8b/0xe0
Dec  2 12:41:35 10.243.0.116 [ 2201.769611]  ? handle_irq_pipelined_finish+0xf6/0x240
Dec  2 12:41:35 10.243.0.116 [ 2201.805219]  ? arch_pipeline_entry+0x81/0x110
Dec  2 12:41:35 10.243.0.116 [ 2201.840034]  ? sysvec_apic_timer_interrupt+0xe/0x20
Dec  2 12:41:35 10.243.0.116 [ 2201.874218]  ? asm_sysvec_apic_timer_interrupt+0x1b/0x20
Dec  2 12:41:35 10.243.0.116 [ 2201.908548]  ? rcu_all_qs+0xa0/0xa0
Dec  2 12:41:35 10.243.0.116 [ 2201.942675]  ? smp_call_function_single+0x118/0x210
Dec  2 12:41:35 10.243.0.116 [ 2201.976159]  ? smp_call_function_single+0x110/0x210
Dec  2 12:41:35 10.243.0.116 [ 2202.009858]  ? 0xffffffffc0b38099
Dec  2 12:41:35 10.243.0.116 [ 2202.042400]  ? generic_exec_single+0x140/0x140
Dec  2 12:41:35 10.243.0.116 [ 2202.075416]  ? rcu_all_qs+0xa0/0xa0
Dec  2 12:41:35 10.243.0.116 [ 2202.107233]  ? smp_call_function_single+0x5/0x210
Dec  2 12:41:35 10.243.0.116 [ 2202.139254]  ? sync_rcu_exp_select_node_cpus+0x3df/0x5c0
Dec  2 12:41:35 10.243.0.116 [ 2202.171882]  ? sync_rcu_exp_select_cpus+0x2c5/0x5c0
Dec  2 12:41:36 10.243.0.116 [ 2202.203417]  ? wait_rcu_exp_gp+0x1f/0x30
Dec  2 12:41:36 10.243.0.116 [ 2202.234017]  ? process_one_work+0x525/0xa30
Dec  2 12:41:36 10.243.0.116 [ 2202.265176]  ? pwq_dec_nr_in_flight+0x120/0x120
Dec  2 12:41:36 10.243.0.116 [ 2202.295857]  ? process_one_work+0x5/0xa30
Dec  2 12:41:36 10.243.0.116 [ 2202.326958]  ? worker_thread+0x57/0x590
Dec  2 12:41:36 10.243.0.116 [ 2202.356729]  ? process_one_work+0xa30/0xa30
Dec  2 12:41:36 10.243.0.116 [ 2202.387120]  ? kthread+0x1f3/0x220
Dec  2 12:41:36 10.243.0.116 [ 2202.416563]  ? set_kthread_struct+0x90/0x90
Dec  2 12:41:36 10.243.0.116 [ 2202.446123]  ? ret_from_fork+0x22/0x30
Dec  2 12:41:36 10.243.0.116 [ 2202.474174]  </TASK>
Dec  2 12:41:46 10.243.0.116 [ 2212.449324] watchdog: BUG: soft lockup - CPU#0 stuck for 466s! [TxStartup:3574]
Dec  2 12:41:46 10.243.0.116 [ 2212.482867] Modules linked in: xt_CHECKSUM xt_MASQUERADE tun netconsole ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 xt_conntrack ebtable_nat ebtable_broute ip6table_nat ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_nat iptable_mangle iptable_security iptable_raw nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 libcrc32c ip_set nfnetlink ebtable_filter ebtables rfkill ip6table_filter ip6_tables iptable_filter sunrpc dm_mirror dm_region_hash dm_log dm_mod x86_pkg_temp_thermal intel_powerclamp coretemp gpio_ich iTCO_wdt iTCO_vendor_support kvm_intel mxm_wmi kvm irqbypass crct10dif_pclmul crc32_pclmul ghash_clmulni_intel aesni_intel crypto_simd cryptd rapl intel_cstate pcspkr sg i2c_i801 ftdi_sio mei_me i2c_smbus joydev input_leds intel_pch_thermal mei lpc_ich mfd_core ioatdma wmi ip_tables ext4 mbcache jbd2 ast drm_vram_helper drm_kms_helper syscopyarea ixgbe sysfillrect sd_mod sr_mod sysimgblt fb_sys_fops cdrom t10_pi drm_ttm_helper ttm mdio drm igb ah
Dec  2 12:41:46 10.243.0.116 [ 2212.496283]  libahci libata crc32c_intel ptp pps_core i2c_algo_bit dca fuse
Dec  2 12:41:46 10.243.0.116 [ 2212.807789] irq event stamp: 24564
Dec  2 12:41:46 10.243.0.116 [ 2212.842413] hardirqs last  enabled at (24563): [<ffffffff8d1d44ad>] sync_current_irq_stage+0x45d/0x500
Dec  2 12:41:46 10.243.0.116 [ 2212.879242] hardirqs last disabled at (24564): [<ffffffff8d49923b>] handle_mm_fault+0x48b/0x520
Dec  2 12:41:46 10.243.0.116 [ 2212.916101] softirqs last  enabled at (0): [<ffffffff8d0ca587>] copy_process+0x1307/0x3770
Dec  2 12:41:46 10.243.0.116 [ 2212.953194] softirqs last disabled at (0): [<0000000000000000>] 0x0
Dec  2 12:41:46 10.243.0.116 [ 2212.990176] CPU: 0 PID: 3574 Comm: TxStartup Tainted: G    B   W    L    5.15.77evl-g5e681f841fc3-dirty #3
Dec  2 12:41:46 10.243.0.116 [ 2213.028396] Hardware name: Supermicro Super Server/X10SDV-6C+-TLN4F, BIOS 2.1 11/22/2019
Dec  2 12:41:46 10.243.0.116 [ 2213.066458] IRQ stage: Linux
Dec  2 12:41:46 10.243.0.116 [ 2213.103306] RIP: 0010:proxy_set_next_ktime+0x77/0x90
Dec  2 12:41:46 10.243.0.116 [ 2213.140801] Code: 74 37 49 8d 7c 24 30 e8 d7 2d 1d 00 49 8b 44 24 30 4c 89 e7 e8 da 42 0b 01 48 8d 34 03 31 d2 4c 89 ef e8 5c 14 00 00 41 56 9d <31> c0 5b 41 5c 41 5d 41 5e 5d c3 cc cc cc cc e8 25 f5 eb ff eb d9
Dec  2 12:41:47 10.243.0.116 [ 2213.221591] RSP: 0018:ffff88841e209e80 EFLAGS: 00000297 ORIG_RAX: 0000000000000000
Dec  2 12:41:47 10.243.0.116 [ 2213.263898] RAX: 0000000000000000 RBX: 0000000000000000 RCX: 0000000000000000
Dec  2 12:41:47 10.243.0.116 [ 2213.306332] RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000000
Dec  2 12:41:47 10.243.0.116 [ 2213.348762] RBP: ffff88841e209ea0 R08: 0000000000000000 R09: 0000000000000000
Dec  2 12:41:47 10.243.0.116 [ 2213.391301] R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000000
Dec  2 12:41:47 10.243.0.116 [ 2213.433897] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000
Dec  2 12:41:47 10.243.0.116 [ 2213.476523] FS:  00007f301ae2e700(0000) GS:ffff88841e200000(0000) knlGS:0000000000000000
Dec  2 12:41:47 10.243.0.116 [ 2213.519709] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Dec  2 12:41:47 10.243.0.116 [ 2213.562679] CR2: 00007f1e4033b660 CR3: 0000000137d08006 CR4: 00000000003706f0
Dec  2 12:41:47 10.243.0.116 [ 2213.606340] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Dec  2 12:41:47 10.243.0.116 [ 2213.650776] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Dec  2 12:41:47 10.243.0.116 [ 2213.695237] Call Trace:
Dec  2 12:41:47 10.243.0.116 [ 2213.738705]  <IRQ>
Dec  2 12:41:47 10.243.0.116 [ 2213.782176]  ? __hrtimer_run_queues+0x5/0x780
Dec  2 12:41:47 10.243.0.116 [ 2213.826716]  ? hrtimer_interrupt+0x1aa/0x360
Dec  2 12:41:47 10.243.0.116 [ 2213.870760]  ? hrtimer_interrupt+0x5/0x360
Dec  2 12:41:47 10.243.0.116 [ 2213.914987]  ? proxy_irq_handler+0x37/0x50
Dec  2 12:41:47 10.243.0.116 [ 2213.958641]  ? handle_synthetic_irq+0xec/0x320
Dec  2 12:41:47 10.243.0.116 [ 2214.002776]  ? do_irq_inband+0x2f/0x40
Dec  2 12:41:47 10.243.0.116 [ 2214.046406]  ? arch_do_IRQ_pipelined+0xcd/0x5b0
Dec  2 12:41:47 10.243.0.116 [ 2214.090029]  </IRQ>
Dec  2 12:41:47 10.243.0.116 [ 2214.132692]  <TASK>
Dec  2 12:41:47 10.243.0.116 [ 2214.175524]  ? sync_current_irq_stage+0x2e3/0x500
Dec  2 12:41:48 10.243.0.116 [ 2214.219692]  ? sync_irq_stage+0x1af/0x1e0
Dec  2 12:41:48 10.243.0.116 [ 2214.262968]  ? synchronize_pipeline+0x8b/0xe0
Dec  2 12:41:48 10.243.0.116 [ 2214.306190]  ? handle_irq_pipelined_finish+0xf6/0x240
Dec  2 12:41:48 10.243.0.116 [ 2214.349747]  ? arch_pipeline_entry+0x81/0x110
Dec  2 12:41:48 10.243.0.116 [ 2214.392439]  ? sysvec_apic_timer_interrupt+0xe/0x20
Dec  2 12:41:48 10.243.0.116 [ 2214.434406]  ? asm_sysvec_apic_timer_interrupt+0x1b/0x20
Dec  2 12:41:48 10.243.0.116 [ 2214.477042]  ? smp_call_function_many_cond+0x1c7/0x4e0
Dec  2 12:41:48 10.243.0.116 [ 2214.518736]  ? __asan_load4+0x37/0x90
Dec  2 12:41:48 10.243.0.116 [ 2214.559653]  ? smp_call_function_many_cond+0x1c7/0x4e0
Dec  2 12:41:48 10.243.0.116 [ 2214.600427]  ? leave_mm+0x40/0x40
Dec  2 12:41:48 10.243.0.116 [ 2214.640220]  ? x86_configure_nx+0x70/0x70
Dec  2 12:41:48 10.243.0.116 [ 2214.680002]  ? on_each_cpu_cond_mask+0x29/0x40
Dec  2 12:41:48 10.243.0.116 [ 2214.719226]  ? native_flush_tlb_multi+0xbe/0x250
Dec  2 12:41:48 10.243.0.116 [ 2214.758188]  ? flush_tlb_mm_range+0x136/0x1f0
Dec  2 12:41:48 10.243.0.116 [ 2214.796803]  ? change_protection_range+0x8bd/0xea0
Dec  2 12:41:48 10.243.0.116 [ 2214.837115]  ? prot_none_pte_entry+0x70/0x70
Dec  2 12:41:48 10.243.0.116 [ 2214.874914]  ? 0xffffffffc0b38099
Dec  2 12:41:48 10.243.0.116 [ 2214.912374]  ? change_protection_range+0x5/0xea0
Dec  2 12:41:48 10.243.0.116 [ 2214.949991]  ? change_protection+0x50/0x90
Dec  2 12:41:48 10.243.0.116 [ 2214.986701]  ? mprotect_fixup+0x2c5/0x490
Dec  2 12:41:48 10.243.0.116 [ 2215.023481]  ? change_protection+0x90/0x90
Dec  2 12:41:48 10.243.0.116 [ 2215.059428]  ? mprotect_fixup+0x5/0x490
Dec  2 12:41:48 10.243.0.116 [ 2215.095808]  ? do_mprotect_pkey+0x32a/0x590
Dec  2 12:41:48 10.243.0.116 [ 2215.130566]  ? mprotect_fixup+0x490/0x490
Dec  2 12:41:48 10.243.0.116 [ 2215.162857]  ? do_mprotect_pkey+0x5/0x590
Dec  2 12:41:48 10.243.0.116 [ 2215.195533]  ? __x64_sys_mprotect+0x48/0x60
Dec  2 12:41:49 10.243.0.116 [ 2215.227339]  ? do_syscall_64+0x4c/0xa0
Dec  2 12:41:49 10.243.0.116 [ 2215.258936]  ? entry_SYSCALL_64_after_hwframe+0x61/0xcb
Dec  2 12:41:49 10.243.0.116 [ 2215.292360]  </TASK>
Dec  2 12:41:55 10.243.0.116 [ 2221.347164] ------------[ cut here ]------------
Dec  2 12:41:55 10.243.0.116 [ 2221.376431] igb 0000:05:00.0: Detected Tx Unit Hang#012[ 2221.376431]   Tx Queue             <7>#012[ 2221.376431]   TDH                  <57>#012[ 2221.376431]   TDT                  <57>#012[ 2221.376431]   next_to_use          <57>#012[ 2221.376431]   next_to_clean        <ec>#012[ 2221.376431] buffer_info[next_to_clean]#012[ 2221.376431]   time_stamp           <1001cfd74>#012[ 2221.376431]   next_to_watch        <00000000a55bfb00>#012[ 2221.376431]   jiffies              <1001d4c1d>#012[ 2221.376431]   desc.status          <1ac001>
Dec  2 12:41:55 10.243.0.116 [ 2221.876167] NETDEV WATCHDOG: eno1 (igb): transmit queue 7 timed out
Dec  2 12:41:55 10.243.0.116 [ 2221.899091] WARNING: CPU: 1 PID: 0 at net/sched/sch_generic.c:478 dev_watchdog+0x4ff/0x570
Dec  2 12:41:55 10.243.0.116 [ 2221.930278] Modules linked in: xt_CHECKSUM xt_MASQUERADE tun netconsole ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 xt_conntrack ebtable_nat ebtable_broute ip6table_nat ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_nat iptable_mangle iptable_security iptable_raw nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 libcrc32c ip_set nfnetlink ebtable_filter ebtables rfkill ip6table_filter ip6_tables iptable_filter sunrpc dm_mirror dm_region_hash dm_log dm_mod x86_pkg_temp_thermal intel_powerclamp coretemp gpio_ich iTCO_wdt iTCO_vendor_support kvm_intel mxm_wmi kvm irqbypass crct10dif_pclmul crc32_pclmul ghash_clmulni_intel aesni_intel crypto_simd cryptd rapl intel_cstate pcspkr sg i2c_i801 ftdi_sio mei_me i2c_smbus joydev input_leds intel_pch_thermal mei lpc_ich mfd_core ioatdma wmi ip_tables ext4 mbcache jbd2 ast drm_vram_helper drm_kms_helper syscopyarea ixgbe sysfillrect sd_mod sr_mod sysimgblt fb_sys_fops cdrom t10_pi drm_ttm_helper ttm mdio drm igb ah
Dec  2 12:41:55 10.243.0.116 [ 2221.984081]  libahci libata crc32c_intel ptp pps_core i2c_algo_bit dca fuse
Dec  2 12:41:56 10.243.0.116 [ 2222.222049] CPU: 1 PID: 0 Comm: swapper/1 Tainted: G    B   W    L    5.15.77evl-g5e681f841fc3-dirty #3
Dec  2 12:41:56 10.243.0.116 [ 2222.258031] Hardware name: Supermicro Super Server/X10SDV-6C+-TLN4F, BIOS 2.1 11/22/2019
Dec  2 12:41:56 10.243.0.116 [ 2222.294311] IRQ stage: Linux
Dec  2 12:41:56 10.243.0.116 [ 2222.330016] RIP: 0010:dev_watchdog+0x4ff/0x570
Dec  2 12:41:56 10.243.0.116 [ 2222.365176] Code: ff ff 4c 8b 75 b8 c6 05 81 68 8d 01 01 4c 89 f7 e8 96 fc f4 ff 44 89 e1 4c 89 f6 48 c7 c7 20 3f a4 8e 48 89 c2 e8 92 eb 2e 00 <0f> 0b e9 10 ff ff ff 65 ff 05 53 5d 1d 72 48 c7 c7 58 25 73 8f e8
Dec  2 12:41:56 10.243.0.116 [ 2222.434240] RSP: 0018:ffff88841e289d20 EFLAGS: 00010286
Dec  2 12:41:56 10.243.0.116 [ 2222.475306] RAX: 0000000000000000 RBX: ffff8881321c8500 RCX: ffffffff8d22a902
Dec  2 12:41:56 10.243.0.116 [ 2222.515381] RDX: 1ffff11083c5b7c1 RSI: 0000000000000008 RDI: ffff88841e2dbe0c
Dec  2 12:41:56 10.243.0.116 [ 2222.556084] RBP: ffff88841e289d80 R08: ffffed1083c5ca29 R09: ffffed1083c5ca29
Dec  2 12:41:56 10.243.0.116 [ 2222.587339] R10: ffff8881001ce107 R11: ffffed1020039c20 R12: 0000000000000007
Dec  2 12:41:56 10.243.0.116 [ 2222.630191] R13: ffff8881321c8408 R14: ffff8881321c8000 R15: 0000000000000001
Dec  2 12:41:56 10.243.0.116 [ 2222.672328] FS:  0000000000000000(0000) GS:ffff88841e280000(0000) knlGS:0000000000000000
Dec  2 12:41:56 10.243.0.116 [ 2222.716123] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Dec  2 12:41:56 10.243.0.116 [ 2222.759259] CR2: 00007f202c70b000 CR3: 0000000407e16004 CR4: 00000000003706e0
Dec  2 12:41:56 10.243.0.116 [ 2222.803194] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Dec  2 12:41:56 10.243.0.116 [ 2222.847256] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Dec  2 12:41:56 10.243.0.116 [ 2222.891345] Call Trace:
Dec  2 12:41:56 10.243.0.116 [ 2222.935339]  <IRQ>
Dec  2 12:41:56 10.243.0.116 [ 2222.980419]  ? qdisc_put_unlocked+0x50/0x50
Dec  2 12:41:56 10.243.0.116 [ 2223.023440]  call_timer_fn+0x224/0x3e0
Dec  2 12:41:56 10.243.0.116 [ 2223.067309]  ? add_timer+0x360/0x360
Dec  2 12:41:56 10.243.0.116 [ 2223.111083]  ? call_timer_fn+0x5/0x3e0
Dec  2 12:41:56 10.243.0.116 [ 2223.143416]  ? qdisc_put_unlocked+0x50/0x50
Dec  2 12:41:57 10.243.0.116 [ 2223.188309]  run_timer_softirq+0x952/0x9c0
Dec  2 12:41:57 10.243.0.116 [ 2223.234313]  ? call_timer_fn+0x3e0/0x3e0
Dec  2 12:41:57 10.243.0.116 [ 2223.278327]  ? rcu_read_lock_held_common+0x5/0x70
Dec  2 12:41:57 10.243.0.116 [ 2223.322433]  ? call_timer_fn+0x3e0/0x3e0
Dec  2 12:41:57 10.243.0.116 [ 2223.365205]  ? __do_softirq+0xf8/0x598
Dec  2 12:41:57 10.243.0.116 [ 2223.398271]  ? run_timer_softirq+0x5/0x9c0
Dec  2 12:41:57 10.243.0.116 [ 2223.443418]  __do_softirq+0x103/0x598
Dec  2 12:41:57 10.243.0.116 [ 2223.489152]  irq_exit_rcu+0xab/0x180
Dec  2 12:41:57 10.243.0.116 [ 2223.533367]  arch_do_IRQ_pipelined+0xd2/0x5b0
Dec  2 12:41:57 10.243.0.116 [ 2223.578142]  </IRQ>
Dec  2 12:41:57 10.243.0.116 [ 2223.621272]  <TASK>
Dec  2 12:41:57 10.243.0.116 [ 2223.664298]  sync_current_irq_stage+0x2e3/0x500
Dec  2 12:41:57 10.243.0.116 [ 2223.699143]  __inband_irq_enable+0x74/0x80
Dec  2 12:41:57 10.243.0.116 [ 2223.743057]  inband_irq_enable+0x1b/0x30
Dec  2 12:41:57 10.243.0.116 [ 2223.786056]  cpuidle_enter_state+0x1c2/0x7b0
Dec  2 12:41:57 10.243.0.116 [ 2223.831370]  cpuidle_enter+0x41/0x70
Dec  2 12:41:57 10.243.0.116 [ 2223.875002]  call_cpuidle+0x5e/0x90
Dec  2 12:41:57 10.243.0.116 [ 2223.917226]  do_idle+0x375/0x390
Dec  2 12:41:57 10.243.0.116 [ 2223.949021]  ? arch_cpu_idle_exit+0x40/0x40
Dec  2 12:41:57 10.243.0.116 [ 2223.991096]  ? do_idle+0x5/0x390
Dec  2 12:41:57 10.243.0.116 [ 2224.032947]  cpu_startup_entry+0x1d/0x20
Dec  2 12:41:57 10.243.0.116 [ 2224.073417]  start_secondary+0x1d4/0x230
Dec  2 12:41:57 10.243.0.116 [ 2224.114018]  ? set_cpu_sibling_map+0xcf0/0xcf0
Dec  2 12:41:57 10.243.0.116 [ 2224.155005]  secondary_startup_64_no_verify+0xb0/0xbb
Dec  2 12:41:57 10.243.0.116 [ 2224.197428]  </TASK>
Dec  2 12:41:58 10.243.0.116 [ 2224.225993] irq event stamp: 8128534
Dec  2 12:41:58 10.243.0.116 [ 2224.254271] hardirqs last  enabled at (8128533): [<ffffffff8d22afe2>] tick_nohz_idle_exit+0x92/0x1a0
Dec  2 12:41:58 10.243.0.116 [ 2224.294358] hardirqs last disabled at (8128534): [<ffffffff8e1afe3d>] __schedule+0x90d/0x1160
Dec  2 12:41:58 10.243.0.116 [ 2224.333354] softirqs last  enabled at (8128500): [<ffffffff8e400332>] __do_softirq+0x332/0x598
Dec  2 12:41:58 10.243.0.116 [ 2224.370023] softirqs last disabled at (8128493): [<ffffffff8d0dc80b>] irq_exit_rcu+0xab/0x180
Dec  2 12:41:58 10.243.0.116 [ 2224.406201] ---[ end trace 579693c5019e495a ]---
Dec  2 12:41:58 10.243.0.116 [ 2224.455183] igb 0000:05:00.0 eno1: Reset adapter

[-- Attachment #2: smime.p7s --]
[-- Type: application/pkcs7-signature, Size: 6759 bytes --]

Re: [External] - Re: EVL Memory

[Philippe Gerum]

Russell Johnson <russell.johnson@kratosdefense.com> writes:

> [[S/MIME Signed Part:Undecided]]
> I enabled all the ftrace stuff recommended by the wiki page, ran my test
> app, and the system ended up getting hung so I couldn't get the output of
> the ftrace.. I got a ton of kernel splat.
>
> Thanks,
>
> Russell
>
> [2. text/plain; ftrace_1.txt]...
>
> [[End of S/MIME Signed Part]]

Fixed in next branch.

-- 
Philippe.

RE: [External] - Re: EVL Memory

[Russell Johnson]

[-- Attachment #1.1: Type: text/plain, Size: 382 bytes --]

Yes I have confirmed that the threads are stuck with boosted priority in
"evl ps". I just went to re-run again with the
kernel I used yesterday (with lockdep still enabled) - since the other
kernel is still building - and I actually got a crash and some memory output
from kasan. I attached the output.

I just saw your new email - I will look into that as well.

Thanks,

Russell


[-- Attachment #1.2: null_ptr.txt --]
[-- Type: text/plain, Size: 14768 bytes --]

Dec  2 08:43:14 10.243.0.116 [71855.650618] ==================================================================
Dec  2 08:43:14 10.243.0.116 [71855.650625] BUG: KASAN: null-ptr-deref in do_raw_spin_lock+0x70/0x1e0
Dec  2 08:43:14 10.243.0.116 [71855.650642] Read of size 4 at addr 0000000000000004 by task RxWorker0/6338
Dec  2 08:43:14 10.243.0.116 [71855.650651]
Dec  2 08:43:14 10.243.0.116 [71855.650655] CPU: 8 PID: 6338 Comm: RxWorker0 Tainted: G        W         5.15.77evl-g5e681f841fc3 #3
Dec  2 08:43:14 10.243.0.116 [71855.650669] Hardware name: Supermicro Super Server/X10SDV-6C+-TLN4F, BIOS 2.1 11/22/2019
Dec  2 08:43:14 10.243.0.116 [71855.650675] IRQ stage: EVL
Dec  2 08:43:14 10.243.0.116 [71855.650680] Call Trace:
Dec  2 08:43:14 10.243.0.116 [71855.650684]  <TASK>
Dec  2 08:43:14 10.243.0.116 [71855.650691]  dump_stack_lvl+0x90/0xd8
Dec  2 08:43:14 10.243.0.116 [71855.650706]  ? do_raw_spin_lock+0x70/0x1e0
Dec  2 08:43:14 10.243.0.116 [71855.650717]  ? do_raw_spin_lock+0x70/0x1e0
Dec  2 08:43:14 10.243.0.116 [71855.650728]  kasan_report.cold.17+0x64/0xdb
Dec  2 08:43:14 10.243.0.116 [71855.650750]  ? do_raw_spin_lock+0x70/0x1e0
Dec  2 08:43:15 10.243.0.116 [71855.650768]  __asan_load4+0x6d/0x90
Dec  2 08:43:15 10.243.0.116 [71855.650782]  do_raw_spin_lock+0x70/0x1e0
Dec  2 08:43:15 10.243.0.116 [71855.650794]  ? spin_dump+0x90/0x90
Dec  2 08:43:15 10.243.0.116 [71855.650809]  ? evl_get_thread_wchan+0x15b/0x270
Dec  2 08:43:15 10.243.0.116 [71855.650836]  evl_requeue_mutex_wait+0x18b/0x500
Dec  2 08:43:15 10.243.0.116 [71855.650847]  ? evl_requeue_mutex_wait+0x15f/0x500
Dec  2 08:43:15 10.243.0.116 [71855.650858]  ? evl_adjust_thread_boost+0x424/0xd70
Dec  2 08:43:15 10.243.0.116 [71855.650882]  evl_walk_lock_chain+0xea/0x280
Dec  2 08:43:15 10.243.0.116 [71855.650908]  evl_lock_mutex_timeout+0x40a/0xca0
Dec  2 08:43:15 10.243.0.116 [71855.650940]  ? evl_destroy_mutex+0x220/0x220
Dec  2 08:43:15 10.243.0.116 [71855.650955]  ? do_raw_spin_unlock+0x97/0x140
Dec  2 08:43:15 10.243.0.116 [71855.650980]  enter_monitor+0x114/0x160
Dec  2 08:43:15 10.243.0.116 [71855.650999]  monitor_oob_ioctl+0x4c2/0x1670
Dec  2 08:43:15 10.243.0.116 [71855.651011]  ? rcu_read_lock_held_common+0x25/0x70
Dec  2 08:43:15 10.243.0.116 [71855.651029]  ? rcu_read_lock_sched_held+0x5b/0xd0
Dec  2 08:43:15 10.243.0.116 [71855.651045]  ? rcu_read_lock_held_common+0x25/0x70
Dec  2 08:43:15 10.243.0.116 [71855.651061]  ? rcu_read_lock_sched_held+0x5b/0xd0
Dec  2 08:43:15 10.243.0.116 [71855.651081]  ? enter_monitor+0x160/0x160
Dec  2 08:43:15 10.243.0.116 [71855.651101]  ? lock_release+0xd5/0x440
Dec  2 08:43:15 10.243.0.116 [71855.651115]  ? evl_get_file+0xa3/0x130
Dec  2 08:43:15 10.243.0.116 [71855.651132]  ? lock_downgrade+0x3e0/0x3e0
Dec  2 08:43:15 10.243.0.116 [71855.651148]  ? do_raw_spin_lock+0x119/0x1e0
Dec  2 08:43:15 10.243.0.116 [71855.651161]  ? spin_dump+0x90/0x90
Dec  2 08:43:15 10.243.0.116 [71855.651183]  ? __kasan_check_read+0x11/0x20
Dec  2 08:43:15 10.243.0.116 [71855.651197]  ? do_raw_spin_unlock+0x97/0x140
Dec  2 08:43:15 10.243.0.116 [71855.651223]  EVL_ioctl+0x81/0xf0
Dec  2 08:43:15 10.243.0.116 [71855.651243]  do_oob_syscall+0x5ac/0x6c0
Dec  2 08:43:15 10.243.0.116 [71855.651262]  ? prepare_for_signal+0x1b0/0x1b0
Dec  2 08:43:15 10.243.0.116 [71855.651281]  ? do_oob_irq+0x2d0/0x880
Dec  2 08:43:15 10.243.0.116 [71855.651293]  ? rcu_read_lock_sched_held+0x5b/0xd0
Dec  2 08:43:15 10.243.0.116 [71855.651314]  handle_oob_syscall+0x189/0x230
Dec  2 08:43:15 10.243.0.116 [71855.651332]  ? handle_pipelined_syscall+0x840/0x840
Dec  2 08:43:15 10.243.0.116 [71855.651353]  ? rcu_read_lock_sched_held+0x5b/0xd0
Dec  2 08:43:15 10.243.0.116 [71855.651370]  ? rcu_read_lock_any_held.part.15+0x20/0x20
Dec  2 08:43:15 10.243.0.116 [71855.651394]  pipeline_syscall+0xa7/0x1c0
Dec  2 08:43:15 10.243.0.116 [71855.651418]  syscall_enter_from_user_mode+0x5f/0x130
Dec  2 08:43:15 10.243.0.116 [71855.651437]  do_syscall_64+0x1d/0xa0
Dec  2 08:43:15 10.243.0.116 [71855.651454]  entry_SYSCALL_64_after_hwframe+0x61/0xcb
Dec  2 08:43:15 10.243.0.116 [71855.651470] RIP: 0033:0x7f5f997a5b4b
Dec  2 08:43:15 10.243.0.116 [71855.651480] Code: Unable to access opcode bytes at RIP 0x7f5f997a5b21.
Dec  2 08:43:15 10.243.0.116 [71855.651486] RSP: 002b:00007f5f55fed510 EFLAGS: 00000246 ORIG_RAX: 000000000000009d
Dec  2 08:43:15 10.243.0.116 [71855.651498] RAX: ffffffffffffffda RBX: 00007f5f55fed5a0 RCX: 00007f5f997a5b4b
Dec  2 08:43:15 10.243.0.116 [71855.651507] RDX: 0000000040106d00 RSI: 000000000000000e RDI: 0000000010000002
Dec  2 08:43:15 10.243.0.116 [71855.651515] RBP: 0000000040106d00 R08: 0000000000000000 R09: 00000000ffffffff
Dec  2 08:43:15 10.243.0.116 [71855.651523] R10: 00007f5f55fed5a0 R11: 0000000000000246 R12: 000000000000000e
Dec  2 08:43:15 10.243.0.116 [71855.651530] R13: 00007f5f55fed600 R14: 0000000000000000 R15: 00007f5f55ffd700
Dec  2 08:43:15 10.243.0.116 [71855.651566]  </TASK>
Dec  2 08:43:15 10.243.0.116 [71855.651570] ==================================================================
Dec  2 08:43:15 10.243.0.116 [71855.651576] EVL: RxWorker0 switching in-band [pid=6338, excpt=14, do_raw_spin_lock+0x70/0x1e0]
Dec  2 08:43:15 10.243.0.116 [71855.838056] EVL: RxWorker0[6338] could not receive HM event #3
Dec  2 08:43:15 10.243.0.116 [71855.838073] BUG: kernel NULL pointer dereference, address: 0000000000000004
Dec  2 08:43:15 10.243.0.116 [71855.838079] #PF: supervisor read access in kernel mode
Dec  2 08:43:15 10.243.0.116 [71855.838084] #PF: error_code(0x0000) - not-present page
Dec  2 08:43:15 10.243.0.116 [71855.838090] PGD 8000000135348067 P4D 8000000135348067 PUD 1e3e6c067 PMD 0
Dec  2 08:43:15 10.243.0.116 [71855.838109] Oops: 0000 [#1] SMP KASAN PTI IRQ_PIPELINE
Dec  2 08:43:15 10.243.0.116 [71855.838122] CPU: 8 PID: 6338 Comm: RxWorker0 Tainted: G    B   W         5.15.77evl-g5e681f841fc3 #3
Dec  2 08:43:15 10.243.0.116 [71855.838134] Hardware name: Supermicro Super Server/X10SDV-6C+-TLN4F, BIOS 2.1 11/22/2019
Dec  2 08:43:15 10.243.0.116 [71855.838140] IRQ stage: Linux
Dec  2 08:43:15 10.243.0.116 [71855.838143] RIP: 0010:do_raw_spin_lock+0x70/0x1e0
Dec  2 08:43:15 10.243.0.116 [71855.838158] Code: 41 48 c7 45 88 10 4f 1a 81 c7 00 f1 f1 f1 f1 c7 40 04 04 f2 f2 f2 65 48 8b 04 25 28 00 00 00 48 89 45 d0 31 c0 e8 70 a9 33 00 <8b> 43 04 3d ad 4e ad de 0f 85 d7 00 00 00 4c 8d 73 10 4c 89 f7 e8
Dec  2 08:43:15 10.243.0.116 [71855.838168] RSP: 0018:ffff888105117868 EFLAGS: 00010082
Dec  2 08:43:15 10.243.0.116 [71855.838178] RAX: 0000000000000001 RBX: 0000000000000000 RCX: ffffffff811a5187
Dec  2 08:43:15 10.243.0.116 [71855.838185] RDX: 0000000000000003 RSI: dffffc0000000000 RDI: ffffffff8329cea0
Dec  2 08:43:15 10.243.0.116 [71855.838193] RBP: ffff8881051178f0 R08: fffffbfff06539d5 R09: fffffbfff06539d5
Dec  2 08:43:15 10.243.0.116 [71855.838201] R10: ffffffff8329cea3 R11: fffffbfff06539d4 R12: 1ffff11020a22f0d
Dec  2 08:43:15 10.243.0.116 [71855.838209] R13: ffff8881051178c8 R14: 0000000000000000 R15: ffff888124d71968
Dec  2 08:43:15 10.243.0.116 [71855.838216] FS:  00007f5f55ffd700(0000) GS:ffff8883de400000(0000) knlGS:0000000000000000
Dec  2 08:43:15 10.243.0.116 [71855.838226] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Dec  2 08:43:15 10.243.0.116 [71855.838234] CR2: 0000000000000004 CR3: 00000001ed95c004 CR4: 00000000003706e0
Dec  2 08:43:15 10.243.0.116 [71855.838241] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Dec  2 08:43:15 10.243.0.116 [71855.838248] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Dec  2 08:43:15 10.243.0.116 [71855.838255] Call Trace:
Dec  2 08:43:15 10.243.0.116 [71855.838258]  <TASK>
Dec  2 08:43:15 10.243.0.116 [71855.838265]  ? spin_dump+0x90/0x90
Dec  2 08:43:15 10.243.0.116 [71855.838279]  ? evl_get_thread_wchan+0x15b/0x270
Dec  2 08:43:15 10.243.0.116 [71855.838301]  evl_requeue_mutex_wait+0x18b/0x500
Dec  2 08:43:15 10.243.0.116 [71855.838313]  ? evl_requeue_mutex_wait+0x15f/0x500
Dec  2 08:43:15 10.243.0.116 [71855.838323]  ? evl_adjust_thread_boost+0x424/0xd70
Dec  2 08:43:15 10.243.0.116 [71855.838345]  evl_walk_lock_chain+0xea/0x280
Dec  2 08:43:15 10.243.0.116 [71855.838368]  evl_lock_mutex_timeout+0x40a/0xca0
Dec  2 08:43:15 10.243.0.116 [71855.838391]  ? evl_destroy_mutex+0x220/0x220
Dec  2 08:43:15 10.243.0.116 [71855.838402]  ? do_raw_spin_unlock+0x97/0x140
Dec  2 08:43:15 10.243.0.116 [71855.838419]  enter_monitor+0x114/0x160
Dec  2 08:43:15 10.243.0.116 [71855.838433]  monitor_oob_ioctl+0x4c2/0x1670
Dec  2 08:43:15 10.243.0.116 [71855.838444]  ? rcu_read_lock_held_common+0x25/0x70
Dec  2 08:43:15 10.243.0.116 [71855.838461]  ? rcu_read_lock_sched_held+0x5b/0xd0
Dec  2 08:43:15 10.243.0.116 [71855.838473]  ? rcu_read_lock_held_common+0x25/0x70
Dec  2 08:43:15 10.243.0.116 [71855.838488]  ? rcu_read_lock_sched_held+0x5b/0xd0
Dec  2 08:43:15 10.243.0.116 [71855.838505]  ? enter_monitor+0x160/0x160
Dec  2 08:43:15 10.243.0.116 [71855.838522]  ? lock_release+0xd5/0x440
Dec  2 08:43:15 10.243.0.116 [71855.838536]  ? evl_get_file+0xa3/0x130
Dec  2 08:43:15 10.243.0.116 [71855.838551]  ? lock_downgrade+0x3e0/0x3e0
Dec  2 08:43:15 10.243.0.116 [71855.838567]  ? do_raw_spin_lock+0x119/0x1e0
Dec  2 08:43:15 10.243.0.116 [71855.838579]  ? spin_dump+0x90/0x90
Dec  2 08:43:15 10.243.0.116 [71855.838597]  ? __kasan_check_read+0x11/0x20
Dec  2 08:43:15 10.243.0.116 [71855.838611]  ? do_raw_spin_unlock+0x97/0x140
Dec  2 08:43:15 10.243.0.116 [71855.838633]  EVL_ioctl+0x81/0xf0
Dec  2 08:43:15 10.243.0.116 [71855.838651]  do_oob_syscall+0x5ac/0x6c0
Dec  2 08:43:15 10.243.0.116 [71855.838667]  ? prepare_for_signal+0x1b0/0x1b0
Dec  2 08:43:15 10.243.0.116 [71855.838683]  ? do_oob_irq+0x2d0/0x880
Dec  2 08:43:15 10.243.0.116 [71855.838695]  ? rcu_read_lock_sched_held+0x5b/0xd0
Dec  2 08:43:15 10.243.0.116 [71855.838715]  handle_oob_syscall+0x189/0x230
Dec  2 08:43:15 10.243.0.116 [71855.838732]  ? handle_pipelined_syscall+0x840/0x840
Dec  2 08:43:15 10.243.0.116 [71855.838751]  ? rcu_read_lock_sched_held+0x5b/0xd0
Dec  2 08:43:15 10.243.0.116 [71855.838767]  ? rcu_read_lock_any_held.part.15+0x20/0x20
Dec  2 08:43:15 10.243.0.116 [71855.838787]  pipeline_syscall+0xa7/0x1c0
Dec  2 08:43:15 10.243.0.116 [71855.838807]  syscall_enter_from_user_mode+0x5f/0x130
Dec  2 08:43:15 10.243.0.116 [71855.838824]  do_syscall_64+0x1d/0xa0
Dec  2 08:43:15 10.243.0.116 [71855.838840]  entry_SYSCALL_64_after_hwframe+0x61/0xcb
Dec  2 08:43:15 10.243.0.116 [71855.838854] RIP: 0033:0x7f5f997a5b4b
Dec  2 08:43:15 10.243.0.116 [71855.838864] Code: 44 24 18 c7 44 24 08 18 00 00 00 e8 0f 98 ff ff b9 9d 00 00 00 49 63 f4 48 89 ea bf 02 00 00 10 49 89 da 45 31 c0 89 c8 0f 05 <8b> 7c 24 04 48 89 c3 31 f6 89 c5 e8 e5 97 ff ff 85 db 75 11 48 83
Dec  2 08:43:15 10.243.0.116 [71855.838873] RSP: 002b:00007f5f55fed510 EFLAGS: 00000246 ORIG_RAX: 000000000000009d
Dec  2 08:43:15 10.243.0.116 [71855.838885] RAX: ffffffffffffffda RBX: 00007f5f55fed5a0 RCX: 00007f5f997a5b4b
Dec  2 08:43:15 10.243.0.116 [71855.838893] RDX: 0000000040106d00 RSI: 000000000000000e RDI: 0000000010000002
Dec  2 08:43:15 10.243.0.116 [71855.838900] RBP: 0000000040106d00 R08: 0000000000000000 R09: 00000000ffffffff
Dec  2 08:43:15 10.243.0.116 [71855.838908] R10: 00007f5f55fed5a0 R11: 0000000000000246 R12: 000000000000000e
Dec  2 08:43:15 10.243.0.116 [71855.838914] R13: 00007f5f55fed600 R14: 0000000000000000 R15: 00007f5f55ffd700
Dec  2 08:43:15 10.243.0.116 [71855.838941]  </TASK>
Dec  2 08:43:15 10.243.0.116 [71855.838945] Modules linked in: binfmt_misc xt_CHECKSUM xt_MASQUERADE tun netconsole ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 xt_conntrack ebtable_nat ebtable_broute ip6table_nat ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_nat iptable_mangle iptable_security iptable_raw nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 libcrc32c ip_set nfnetlink ebtable_filter ebtables rfkill ip6table_filter ip6_tables iptable_filter sunrpc dm_mirror dm_region_hash dm_log dm_mod x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel iTCO_wdt gpio_ich iTCO_vendor_support mxm_wmi kvm irqbypass crct10dif_pclmul crc32_pclmul ghash_clmulni_intel aesni_intel crypto_simd cryptd rapl joydev intel_cstate input_leds ftdi_sio pcspkr i2c_i801 sg mei_me i2c_smbus intel_pch_thermal lpc_ich mei mfd_core ioatdma wmi ip_tables ext4 mbcache jbd2 ast drm_vram_helper drm_kms_helper ixgbe sr_mod sd_mod syscopyarea cdrom sysfillrect t10_pi sysimgblt fb_sys_fops drm_ttm_helper ttm mdi
Dec  2 08:43:15 10.243.0.116 [71855.839277]  drm ahci igb libahci libata crc32c_intel ptp pps_core i2c_algo_bit dca fuse
Dec  2 08:43:15 10.243.0.116 [71855.839318] CR2: 0000000000000004
Dec  2 08:43:16 10.243.0.116 [71856.179377] ---[ end trace fa2b3c78041cd9cc ]---
Dec  2 08:43:16 10.243.0.116 [71856.190886] RIP: 0010:do_raw_spin_lock+0x70/0x1e0
Dec  2 08:43:16 10.243.0.116 [71856.190897] Code: 41 48 c7 45 88 10 4f 1a 81 c7 00 f1 f1 f1 f1 c7 40 04 04 f2 f2 f2 65 48 8b 04 25 28 00 00 00 48 89 45 d0 31 c0 e8 70 a9 33 00 <8b> 43 04 3d ad 4e ad de 0f 85 d7 00 00 00 4c 8d 73 10 4c 89 f7 e8
Dec  2 08:43:16 10.243.0.116 [71856.190903] RSP: 0018:ffff888105117868 EFLAGS: 00010082
Dec  2 08:43:16 10.243.0.116 [71856.190909] RAX: 0000000000000001 RBX: 0000000000000000 RCX: ffffffff811a5187
Dec  2 08:43:16 10.243.0.116 [71856.190913] RDX: 0000000000000003 RSI: dffffc0000000000 RDI: ffffffff8329cea0
Dec  2 08:43:16 10.243.0.116 [71856.190917] RBP: ffff8881051178f0 R08: fffffbfff06539d5 R09: fffffbfff06539d5
Dec  2 08:43:16 10.243.0.116 [71856.190921] R10: ffffffff8329cea3 R11: fffffbfff06539d4 R12: 1ffff11020a22f0d
Dec  2 08:43:16 10.243.0.116 [71856.190925] R13: ffff8881051178c8 R14: 0000000000000000 R15: ffff888124d71968
Dec  2 08:43:16 10.243.0.116 [71856.190929] FS:  00007f5f55ffd700(0000) GS:ffff8883de400000(0000) knlGS:0000000000000000
Dec  2 08:43:16 10.243.0.116 [71856.190934] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Dec  2 08:43:16 10.243.0.116 [71856.190938] CR2: 0000000000000004 CR3: 00000001ed95c004 CR4: 00000000003706e0
Dec  2 08:43:16 10.243.0.116 [71856.190941] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Dec  2 08:43:16 10.243.0.116 [71856.190944] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Dec  2 08:43:16 10.243.0.116 [71856.190949] Kernel panic - not syncing: Fatal exception
Dec  2 08:43:16 10.243.0.116 [71857.263519] Shutting down cpus with NMI
Dec  2 08:43:16 10.243.0.116 [71857.263528] Kernel Offset: 0x0 from 0xffffffff81000000 (relocation range: 0xffffffff80000000-0xffffffffbfffffff)

[-- Attachment #2: smime.p7s --]
[-- Type: application/pkcs7-signature, Size: 6759 bytes --]

Re: [External] - Re: EVL Memory

[Philippe Gerum]

Russell Johnson <russell.johnson@kratosdefense.com> writes:

> [[S/MIME Signed Part:Undecided]]
> Yes I have confirmed that the threads are stuck with boosted priority in
> "evl ps". I just went to re-run again with the
> kernel I used yesterday (with lockdep still enabled) - since the other
> kernel is still building - and I actually got a crash and some memory output
> from kasan. I attached the output.
>
> I just saw your new email - I will look into that as well.
>
> Thanks,
>
> Russell
>
> [2. text/plain; null_ptr.txt]...
>
> [[End of S/MIME Signed Part]]

Ok, the crash in evl_requeue_mutex_wait() visible in this log gives some
hint. I cannot reproduce this though, so this must be a very specific
pattern.

-- 
Philippe.

Re: [External] - Re: EVL Memory

[Philippe Gerum]

Russell Johnson <russell.johnson@kratosdefense.com> writes:

> [[S/MIME Signed Part:Undecided]]
> Yes I have confirmed that the threads are stuck with boosted priority in
> "evl ps". I just went to re-run again with the
> kernel I used yesterday (with lockdep still enabled) - since the other
> kernel is still building - and I actually got a crash and some memory output
> from kasan. I attached the output.
>
> I just saw your new email - I will look into that as well.
>
> Thanks,
>
> Russell
>
> [2. text/plain; null_ptr.txt]...
>
> [[End of S/MIME Signed Part]]

This should fix the null pointer issue. I'm still trying to figure out
how the other issue can ever happen.

diff --git a/kernel/evl/mutex.c b/kernel/evl/mutex.c
index 16750fd193377..a74108fdc5ef6 100644
--- a/kernel/evl/mutex.c
+++ b/kernel/evl/mutex.c
@@ -727,6 +727,13 @@ bool evl_requeue_mutex_wait(struct evl_wait_channel *wchan,
 	assert_hard_lock(&wchan->lock);
 	assert_hard_lock(&waiter->lock);
 
+	/*
+	 * __evl_unlock_mutex() may run concurrently to us, dropping
+	 * ownership. Make sure we still have an owner.
+	 */
+	if (owner == NULL)
+		return false;
+
 	/*
 	 * To prevent ABBA, we may drop the waiter lock prior to
 	 * reacquiring it in a double locking sequence along with

-- 
Philippe.

RE: [External] - Re: EVL Memory

[Russell Johnson]

[-- Attachment #1.1: Type: text/plain, Size: 267 bytes --]

I added in your patches and rebuilt with lockdep disabled and with ftrace
disabled (since the ftrace seems to be killing the system). I did see a
couple brief kernel splats that I figured I would at least ship your way to
see if they are important.

Thanks,

Russell

[-- Attachment #1.2: stack_trace.txt --]
[-- Type: text/plain, Size: 12115 bytes --]

Dec  2 15:23:09 10.243.0.116 [  190.726063] ------------[ cut here ]------------
Dec  2 15:23:09 10.243.0.116 [  190.726069] WARNING: CPU: 5 PID: 1884 at kernel/evl/mutex.c:660 __evl_unlock_mutex+0x310/0x510
Dec  2 15:23:09 10.243.0.116 [  190.726090] Modules linked in: xt_CHECKSUM xt_MASQUERADE tun netconsole ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 xt_conntrack ebtable_nat ebtable_broute ip6table_nat ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_nat iptable_mangle iptable_security iptable_raw nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 libcrc32c ip_set nfnetlink ebtable_filter ebtables rfkill ip6table_filter ip6_tables iptable_filter sunrpc dm_mirror dm_region_hash dm_log dm_mod x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm irqbypass crct10dif_pclmul crc32_pclmul ghash_clmulni_intel gpio_ich iTCO_wdt aesni_intel iTCO_vendor_support crypto_simd mxm_wmi cryptd rapl intel_cstate pcspkr input_leds joydev i2c_i801 i2c_smbus sg ftdi_sio intel_pch_thermal mei_me mei lpc_ich mfd_core ioatdma wmi ip_tables ext4 mbcache jbd2 ast drm_vram_helper drm_kms_helper sd_mod sr_mod syscopyarea sysfillrect cdrom t10_pi sysimgblt fb_sys_fops drm_ttm_helper ttm igb drm ixgbe ahci
Dec  2 15:23:09 10.243.0.116 [  190.726260]  libahci libata crc32c_intel mdio ptp i2c_algo_bit pps_core dca fuse
Dec  2 15:23:09 10.243.0.116 [  190.726278] CPU: 5 PID: 1884 Comm: DummyTxSample Not tainted 5.15.77evl-g5e681f841fc3-dirty #3
Dec  2 15:23:09 10.243.0.116 [  190.726285] Hardware name: Supermicro Super Server/X10SDV-6C+-TLN4F, BIOS 2.1 11/22/2019
Dec  2 15:23:09 10.243.0.116 [  190.726288] IRQ stage: EVL
Dec  2 15:23:09 10.243.0.116 [  190.726291] RIP: 0010:__evl_unlock_mutex+0x310/0x510
Dec  2 15:23:09 10.243.0.116 [  190.726298] Code: 71 af e8 63 bf 1d 00 48 8b 05 84 e6 41 02 e8 07 83 ed ff 85 c0 74 45 65 ff 0d 7c b0 d2 52 e9 6f fd ff ff e8 62 72 ea ff 53 9d <0f> 0b 48 83 c4 18 5b 41 5c 41 5d 41 5e 41 5f 5d c3 cc cc cc cc 4c
Dec  2 15:23:09 10.243.0.116 [  190.726303] RSP: 0018:ffff88812fb37ad8 EFLAGS: 00010082
Dec  2 15:23:09 10.243.0.116 [  190.726308] RAX: 0000000000000000 RBX: 0000000000000082 RCX: ffffffffad1a31c7
Dec  2 15:23:09 10.243.0.116 [  190.726313] RDX: 0000000000000003 RSI: dffffc0000000000 RDI: ffff888101576000
Dec  2 15:23:09 10.243.0.116 [  190.726316] RBP: ffff88812fb37b18 R08: ffffed10202aec01 R09: ffffed10202aec01
Dec  2 15:23:09 10.243.0.116 [  190.726321] R10: ffff888101576003 R11: ffffed10202aec00 R12: ffff888101576000
Dec  2 15:23:09 10.243.0.116 [  190.726324] R13: ffff888101576078 R14: ffff88811fdac1c8 R15: ffff88811fdac158
Dec  2 15:23:09 10.243.0.116 [  190.726328] FS:  00007f1a29659700(0000) GS:ffff8883de480000(0000) knlGS:0000000000000000
Dec  2 15:23:09 10.243.0.116 [  190.726333] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Dec  2 15:23:09 10.243.0.116 [  190.726337] CR2: 0000000000000000 CR3: 000000010aeec002 CR4: 00000000003706e0
Dec  2 15:23:09 10.243.0.116 [  190.726341] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Dec  2 15:23:09 10.243.0.116 [  190.726344] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Dec  2 15:23:09 10.243.0.116 [  190.726347] Call Trace:
Dec  2 15:23:09 10.243.0.116 [  190.726350]  <TASK>
Dec  2 15:23:09 10.243.0.116 [  190.726369]  monitor_oob_ioctl+0xe43/0x13e0
Dec  2 15:23:09 10.243.0.116 [  190.726400]  ? enter_monitor+0x160/0x160
Dec  2 15:23:09 10.243.0.116 [  190.726441]  ? do_raw_spin_lock+0x119/0x1e0
Dec  2 15:23:09 10.243.0.116 [  190.726451]  ? spin_dump+0x90/0x90
Dec  2 15:23:09 10.243.0.116 [  190.726469]  ? __kasan_check_read+0x11/0x20
Dec  2 15:23:09 10.243.0.116 [  190.726499]  EVL_ioctl+0x81/0xf0
Dec  2 15:23:09 10.243.0.116 [  190.726517]  do_oob_syscall+0x5ac/0x6c0
Dec  2 15:23:09 10.243.0.116 [  190.726531]  ? prepare_for_signal+0x180/0x180
Dec  2 15:23:09 10.243.0.116 [  190.726545]  ? do_oob_syscall+0x6c0/0x6c0
Dec  2 15:23:09 10.243.0.116 [  190.726566]  handle_oob_syscall+0x189/0x230
Dec  2 15:23:09 10.243.0.116 [  190.726580]  ? handle_pipelined_syscall+0x840/0x840
Dec  2 15:23:09 10.243.0.116 [  190.726618]  pipeline_syscall+0xa7/0x1c0
Dec  2 15:23:09 10.243.0.116 [  190.726637]  syscall_enter_from_user_mode+0x5f/0x130
Dec  2 15:23:09 10.243.0.116 [  190.726650]  do_syscall_64+0x1d/0xa0
Dec  2 15:23:09 10.243.0.116 [  190.726661]  entry_SYSCALL_64_after_hwframe+0x61/0xcb
Dec  2 15:23:09 10.243.0.116 [  190.726668] RIP: 0033:0x7f1a3dcdeb9b
Dec  2 15:23:09 10.243.0.116 [  190.726674] Code: Unable to access opcode bytes at RIP 0x7f1a3dcdeb71.
Dec  2 15:23:09 10.243.0.116 [  190.726677] RSP: 002b:00007f1a296493f0 EFLAGS: 00000246 ORIG_RAX: 000000000000009d
Dec  2 15:23:09 10.243.0.116 [  190.726682] RAX: ffffffffffffffda RBX: 00007f1a29649490 RCX: 00007f1a3dcdeb9b
Dec  2 15:23:09 10.243.0.116 [  190.726686] RDX: 00000000c0186d03 RSI: 0000000000000049 RDI: 0000000010000002
Dec  2 15:23:09 10.243.0.116 [  190.726690] RBP: 00000000c0186d03 R08: 0000000000000000 R09: 00007ffddbb2f080
Dec  2 15:23:09 10.243.0.116 [  190.726693] R10: 00007f1a29649490 R11: 0000000000000246 R12: 0000000000000049
Dec  2 15:23:09 10.243.0.116 [  190.726697] R13: 0000000000000001 R14: 000000001a2e47cc R15: 00007f1a29649590
Dec  2 15:23:09 10.243.0.116 [  190.726735]  </TASK>
Dec  2 15:23:09 10.243.0.116 [  190.726737] irq event stamp: 556
Dec  2 15:23:09 10.243.0.116 [  190.726739] hardirqs last  enabled at (555): [<ffffffffae14ee3a>] _raw_spin_unlock_irqrestore+0x5a/0x60
Dec  2 15:23:09 10.243.0.116 [  190.726747] hardirqs last disabled at (556): [<ffffffffae141d7d>] __schedule+0x90d/0x1160
Dec  2 15:23:09 10.243.0.116 [  190.726755] softirqs last  enabled at (0): [<ffffffffad0c4f3a>] copy_process+0x12fa/0x3750
Dec  2 15:23:09 10.243.0.116 [  190.726762] softirqs last disabled at (0): [<0000000000000000>] 0x0
Dec  2 15:23:09 10.243.0.116 [  190.726767] ---[ end trace eb204101d74e7ec4 ]---
Dec  2 15:23:09 10.243.0.116 [  190.726785] ------------[ cut here ]------------
Dec  2 15:23:09 10.243.0.116 [  190.726787] WARNING: CPU: 5 PID: 1884 at kernel/evl/sched/core.c:990 __evl_schedule+0x6d7/0x1160
Dec  2 15:23:09 10.243.0.116 [  190.726801] Modules linked in: xt_CHECKSUM xt_MASQUERADE tun netconsole ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 xt_conntrack ebtable_nat ebtable_broute ip6table_nat ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_nat iptable_mangle iptable_security iptable_raw nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 libcrc32c ip_set nfnetlink ebtable_filter ebtables rfkill ip6table_filter ip6_tables iptable_filter sunrpc dm_mirror dm_region_hash dm_log dm_mod x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm irqbypass crct10dif_pclmul crc32_pclmul ghash_clmulni_intel gpio_ich iTCO_wdt aesni_intel iTCO_vendor_support crypto_simd mxm_wmi cryptd rapl intel_cstate pcspkr input_leds joydev i2c_i801 i2c_smbus sg ftdi_sio intel_pch_thermal mei_me mei lpc_ich mfd_core ioatdma wmi ip_tables ext4 mbcache jbd2 ast drm_vram_helper drm_kms_helper sd_mod sr_mod syscopyarea sysfillrect cdrom t10_pi sysimgblt fb_sys_fops drm_ttm_helper ttm igb drm ixgbe ahci
Dec  2 15:23:09 10.243.0.116 [  190.727081]  libahci libata crc32c_intel mdio ptp i2c_algo_bit pps_core dca fuse
Dec  2 15:23:09 10.243.0.116 [  190.727114] CPU: 5 PID: 1884 Comm: DummyTxSample Tainted: G        W         5.15.77evl-g5e681f841fc3-dirty #3
Dec  2 15:23:09 10.243.0.116 [  190.727126] Hardware name: Supermicro Super Server/X10SDV-6C+-TLN4F, BIOS 2.1 11/22/2019
Dec  2 15:23:09 10.243.0.116 [  190.727131] IRQ stage: EVL
Dec  2 15:23:09 10.243.0.116 [  190.727134] RIP: 0010:__evl_schedule+0x6d7/0x1160
Dec  2 15:23:09 10.243.0.116 [  190.727145] Code: c1 f8 06 48 8d 3c c5 e0 c7 73 af e8 d3 0b 1c 00 48 0f a3 1d 4b 4a 42 02 0f 82 54 09 00 00 48 8b 5d d0 48 89 df e8 89 b3 e8 ff <0f> 0b 48 89 df e8 9f b1 e8 ff e9 53 fa ff ff 48 8b 7d c8 e8 21 ff
Dec  2 15:23:09 10.243.0.116 [  190.727154] RSP: 0018:ffff88812fb37a50 EFLAGS: 00010046
Dec  2 15:23:09 10.243.0.116 [  190.727163] RAX: 0000000000000000 RBX: ffff888101576000 RCX: ffffffffad1a31c7
Dec  2 15:23:09 10.243.0.116 [  190.727170] RDX: 0000000000000003 RSI: dffffc0000000000 RDI: ffff888101576000
Dec  2 15:23:09 10.243.0.116 [  190.727177] RBP: ffff88812fb37ad0 R08: ffffed10202aec01 R09: ffffed10202aec01
Dec  2 15:23:09 10.243.0.116 [  190.727185] R10: ffff888101576003 R11: ffffed10202aec00 R12: ffff88811fda9960
Dec  2 15:23:09 10.243.0.116 [  190.727192] R13: ffff8883de4e46a0 R14: ffff888101576000 R15: ffff8883de4e46a0
Dec  2 15:23:09 10.243.0.116 [  190.727200] FS:  00007f1a29659700(0000) GS:ffff8883de480000(0000) knlGS:0000000000000000
Dec  2 15:23:09 10.243.0.116 [  190.727209] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Dec  2 15:23:09 10.243.0.116 [  190.727217] CR2: 0000000000000000 CR3: 000000010aeec002 CR4: 00000000003706e0
Dec  2 15:23:09 10.243.0.116 [  190.727224] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Dec  2 15:23:09 10.243.0.116 [  190.727229] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Dec  2 15:23:09 10.243.0.116 [  190.727236] Call Trace:
Dec  2 15:23:10 10.243.0.116 [  190.727239]  <TASK>
Dec  2 15:23:10 10.243.0.116 [  190.727301]  __evl_wait_schedule+0x80/0x350
Dec  2 15:23:10 10.243.0.116 [  190.727318]  ? __kasan_check_read+0x11/0x20
Dec  2 15:23:10 10.243.0.116 [  190.727350]  monitor_oob_ioctl+0xe62/0x13e0
Dec  2 15:23:10 10.243.0.116 [  190.727406]  ? enter_monitor+0x160/0x160
Dec  2 15:23:10 10.243.0.116 [  190.727486]  ? do_raw_spin_lock+0x119/0x1e0
Dec  2 15:23:10 10.243.0.116 [  190.727502]  ? spin_dump+0x90/0x90
Dec  2 15:23:10 10.243.0.116 [  190.727538]  ? __kasan_check_read+0x11/0x20
Dec  2 15:23:10 10.243.0.116 [  190.727596]  EVL_ioctl+0x81/0xf0
Dec  2 15:23:10 10.243.0.116 [  190.727630]  do_oob_syscall+0x5ac/0x6c0
Dec  2 15:23:10 10.243.0.116 [  190.727650]  ? prepare_for_signal+0x180/0x180
Dec  2 15:23:10 10.243.0.116 [  190.727664]  ? do_oob_syscall+0x6c0/0x6c0
Dec  2 15:23:10 10.243.0.116 [  190.727684]  handle_oob_syscall+0x189/0x230
Dec  2 15:23:10 10.243.0.116 [  190.727699]  ? handle_pipelined_syscall+0x840/0x840
Dec  2 15:23:10 10.243.0.116 [  190.727736]  pipeline_syscall+0xa7/0x1c0
Dec  2 15:23:10 10.243.0.116 [  190.727755]  syscall_enter_from_user_mode+0x5f/0x130
Dec  2 15:23:10 10.243.0.116 [  190.727768]  do_syscall_64+0x1d/0xa0
Dec  2 15:23:10 10.243.0.116 [  190.727778]  entry_SYSCALL_64_after_hwframe+0x61/0xcb
Dec  2 15:23:10 10.243.0.116 [  190.727785] RIP: 0033:0x7f1a3dcdeb9b
Dec  2 15:23:10 10.243.0.116 [  190.727793] Code: Unable to access opcode bytes at RIP 0x7f1a3dcdeb71.
Dec  2 15:23:10 10.243.0.116 [  190.727797] RSP: 002b:00007f1a296493f0 EFLAGS: 00000246 ORIG_RAX: 000000000000009d
Dec  2 15:23:10 10.243.0.116 [  190.727808] RAX: ffffffffffffffda RBX: 00007f1a29649490 RCX: 00007f1a3dcdeb9b
Dec  2 15:23:10 10.243.0.116 [  190.727816] RDX: 00000000c0186d03 RSI: 0000000000000049 RDI: 0000000010000002
Dec  2 15:23:10 10.243.0.116 [  190.727823] RBP: 00000000c0186d03 R08: 0000000000000000 R09: 00007ffddbb2f080
Dec  2 15:23:10 10.243.0.116 [  190.727830] R10: 00007f1a29649490 R11: 0000000000000246 R12: 0000000000000049
Dec  2 15:23:10 10.243.0.116 [  190.727833] R13: 0000000000000001 R14: 000000001a2e47cc R15: 00007f1a29649590
Dec  2 15:23:10 10.243.0.116 [  190.727872]  </TASK>
Dec  2 15:23:10 10.243.0.116 [  190.727874] irq event stamp: 556
Dec  2 15:23:10 10.243.0.116 [  190.727876] hardirqs last  enabled at (555): [<ffffffffae14ee3a>] _raw_spin_unlock_irqrestore+0x5a/0x60
Dec  2 15:23:10 10.243.0.116 [  190.727884] hardirqs last disabled at (556): [<ffffffffae141d7d>] __schedule+0x90d/0x1160
Dec  2 15:23:10 10.243.0.116 [  190.727891] softirqs last  enabled at (0): [<ffffffffad0c4f3a>] copy_process+0x12fa/0x3750
Dec  2 15:23:10 10.243.0.116 [  190.727897] softirqs last disabled at (0): [<0000000000000000>] 0x0
Dec  2 15:23:10 10.243.0.116 [  190.727902] ---[ end trace eb204101d74e7ec5 ]---

[-- Attachment #2: smime.p7s --]
[-- Type: application/pkcs7-signature, Size: 6759 bytes --]

Re: [External] - Re: EVL Memory

[Philippe Gerum]

Russell Johnson <russell.johnson@kratosdefense.com> writes:

> [[S/MIME Signed Part:Undecided]]
> I added in your patches and rebuilt with lockdep disabled and with ftrace
> disabled (since the ftrace seems to be killing the system). I did see a
> couple brief kernel splats that I figured I would at least ship your way to
> see if they are important.
>
> Thanks,
>
> Russell
>
> [2. text/plain; stack_trace.txt]...
>
> [[End of S/MIME Signed Part]]

These warnings relate to the same deboosting bug we have been chasing
lately. Still trying to reproduce this here, no luck so far.

-- 
Philippe.

Re: [External] - Re: EVL Memory

[Philippe Gerum]

Philippe Gerum <rpm@xenomai.org> writes:

> Russell Johnson <russell.johnson@kratosdefense.com> writes:
>
>> [[S/MIME Signed Part:Undecided]]
>> I added in your patches and rebuilt with lockdep disabled and with ftrace
>> disabled (since the ftrace seems to be killing the system). I did see a
>> couple brief kernel splats that I figured I would at least ship your way to
>> see if they are important.
>>
>> Thanks,
>>
>> Russell
>>
>> [2. text/plain; stack_trace.txt]...
>>
>> [[End of S/MIME Signed Part]]
>
> These warnings relate to the same deboosting bug we have been chasing
> lately. Still trying to reproduce this here, no luck so far.


Please try the current tip of next-evl, which includes this fix [1].

[1] https://source.denx.de/Xenomai/xenomai4/linux-evl/-/commit/722ded4782f949e95e36ffad248a8f63c6446c4c

-- 
Philippe.

Re: [External] - Re: EVL Memory

[Philippe Gerum]

a
Philippe Gerum <rpm@xenomai.org> writes:

> Philippe Gerum <rpm@xenomai.org> writes:
>
>> Russell Johnson <russell.johnson@kratosdefense.com> writes:
>>
>>> [[S/MIME Signed Part:Undecided]]
>>> I added in your patches and rebuilt with lockdep disabled and with ftrace
>>> disabled (since the ftrace seems to be killing the system). I did see a
>>> couple brief kernel splats that I figured I would at least ship your way to
>>> see if they are important.
>>>
>>> Thanks,
>>>
>>> Russell
>>>
>>> [2. text/plain; stack_trace.txt]...
>>>
>>> [[End of S/MIME Signed Part]]
>>
>> These warnings relate to the same deboosting bug we have been chasing
>> lately. Still trying to reproduce this here, no luck so far.
>
>
> Please try the current tip of next-evl, which includes this fix [1].
>
> [1] https://source.denx.de/Xenomai/xenomai4/linux-evl/-/commit/722ded4782f949e95e36ffad248a8f63c6446c4c

This patch is ironing an important code path, but I just managed to
reproduce a deboosting issue for the first time. I would suggest to wait
until I can issue a patch fixing it before testing the next-evl branch.

-- 
Philippe.

Re: [External] - Re: EVL Memory

[Philippe Gerum]

Philippe Gerum <rpm@xenomai.org> writes:

> a
> Philippe Gerum <rpm@xenomai.org> writes:
>
>> Philippe Gerum <rpm@xenomai.org> writes:
>>
>>> Russell Johnson <russell.johnson@kratosdefense.com> writes:
>>>
>>>> [[S/MIME Signed Part:Undecided]]
>>>> I added in your patches and rebuilt with lockdep disabled and with ftrace
>>>> disabled (since the ftrace seems to be killing the system). I did see a
>>>> couple brief kernel splats that I figured I would at least ship your way to
>>>> see if they are important.
>>>>
>>>> Thanks,
>>>>
>>>> Russell
>>>>
>>>> [2. text/plain; stack_trace.txt]...
>>>>
>>>> [[End of S/MIME Signed Part]]
>>>
>>> These warnings relate to the same deboosting bug we have been chasing
>>> lately. Still trying to reproduce this here, no luck so far.
>>
>>
>> Please try the current tip of next-evl, which includes this fix [1].
>>
>> [1] https://source.denx.de/Xenomai/xenomai4/linux-evl/-/commit/722ded4782f949e95e36ffad248a8f63c6446c4c
>
> This patch is ironing an important code path, but I just managed to
> reproduce a deboosting issue for the first time. I would suggest to wait
> until I can issue a patch fixing it before testing the next-evl branch.

Fixed now. Please pull from next-evl for testing.

Thanks,

-- 
Philippe.

RE: [External] - Re: EVL Memory

[Russell Johnson]

[-- Attachment #1: Type: text/plain, Size: 142 bytes --]

Hi Philippe,

Thanks for the work. I am pulling the next branch now. I will get you some
updated info tonight or tomorrow.


Thanks,

Russell

[-- Attachment #2: smime.p7s --]
[-- Type: application/pkcs7-signature, Size: 6759 bytes --]

RE: [External] - Re: EVL Memory

[Russell Johnson]

[-- Attachment #1.1: Type: text/plain, Size: 330 bytes --]

Just ran the latest kernel from the next branch with the lockdep turned back
on and ftrace off for now. Good news is that I no longer see a boosting
issue. Thanks again for the work to track that down. All that is left at the
moment is one lockdep kernel splat which may not be important, but I
attached anyway.

Thanks,

Russell

[-- Attachment #1.2: final_lockdep_trace.txt --]
[-- Type: text/plain, Size: 6661 bytes --]

Dec  4 23:43:01 10.243.0.116 [  183.100173] ------------[ cut here ]------------
Dec  4 23:43:01 10.243.0.116 [  183.100180] DEBUG_LOCKS_WARN_ON(!lockdep_hardirqs_enabled())
Dec  4 23:43:01 10.243.0.116 [  183.100200] WARNING: CPU: 5 PID: 1561 at kernel/locking/lockdep.c:5550 check_flags+0x1a4/0x1d0
Dec  4 23:43:01 10.243.0.116 [  183.100214] Modules linked in: xt_CHECKSUM xt_MASQUERADE tun netconsole ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 xt_conntrack ebtable_nat ebtable_broute ip6table_nat ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_nat iptable_mangle iptable_security iptable_raw nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 libcrc32c ip_set nfnetlink ebtable_filter ebtables rfkill ip6table_filter ip6_tables iptable_filter sunrpc dm_mirror dm_region_hash dm_log dm_mod x86_pkg_temp_thermal intel_powerclamp coretemp gpio_ich iTCO_wdt iTCO_vendor_support mxm_wmi kvm_intel kvm irqbypass crct10dif_pclmul crc32_pclmul ghash_clmulni_intel aesni_intel crypto_simd cryptd rapl i2c_i801 intel_cstate joydev ftdi_sio intel_pch_thermal i2c_smbus input_leds sg mei_me lpc_ich pcspkr mfd_core mei ioatdma wmi ip_tables ext4 mbcache jbd2 ast drm_vram_helper drm_kms_helper ixgbe sd_mod sr_mod syscopyarea sysfillrect cdrom t10_pi sysimgblt fb_sys_fops drm_ttm_helper ttm mdio drm ahci
Dec  4 23:43:01 10.243.0.116 [  183.100384]  libahci igb crc32c_intel libata ptp i2c_algo_bit pps_core dca fuse
Dec  4 23:43:01 10.243.0.116 [  183.100403] CPU: 5 PID: 1561 Comm: TxStartup Not tainted 5.15.77evl-g7564b3bed14f #3
Dec  4 23:43:01 10.243.0.116 [  183.100411] Hardware name: Supermicro Super Server/X10SDV-6C+-TLN4F, BIOS 2.1 11/22/2019
Dec  4 23:43:01 10.243.0.116 [  183.100414] IRQ stage: Linux
Dec  4 23:43:01 10.243.0.116 [  183.100417] RIP: 0010:check_flags+0x1a4/0x1d0
Dec  4 23:43:01 10.243.0.116 [  183.100424] Code: ff ff e8 1f d4 6a ff 85 c0 74 21 44 8b 05 3c de 60 01 45 85 c0 75 15 48 c7 c6 80 04 69 af 48 c7 c7 40 d3 68 af e8 e6 e3 f9 ff <0f> 0b 48 c7 c7 c0 04 69 af e8 5f 7b fa ff e9 5c ff ff ff 65 48 8b
Dec  4 23:43:01 10.243.0.116 [  183.100430] RSP: 0018:ffff88810c9d7d90 EFLAGS: 00010082
Dec  4 23:43:01 10.243.0.116 [  183.100435] RAX: 0000000000000000 RBX: 1ffff1102193afb8 RCX: 0000000000000027
Dec  4 23:43:01 10.243.0.116 [  183.100439] RDX: 0000000000000027 RSI: dffffc0000000000 RDI: ffff8883de137b78
Dec  4 23:43:01 10.243.0.116 [  183.100443] RBP: ffff88810c9d7d90 R08: ffffed107bc26f70 R09: ffffed107bc26f70
Dec  4 23:43:01 10.243.0.116 [  183.100448] R10: ffff8883de137b7b R11: ffffed107bc26f6f R12: ffff88810c9d7e80
Dec  4 23:43:01 10.243.0.116 [  183.100452] R13: 0000000000000005 R14: 0000000000000000 R15: 0000000000000000
Dec  4 23:43:01 10.243.0.116 [  183.100455] FS:  00007fec6923c700(0000) GS:ffff8883de100000(0000) knlGS:0000000000000000
Dec  4 23:43:01 10.243.0.116 [  183.100460] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Dec  4 23:43:01 10.243.0.116 [  183.100464] CR2: 0000000000000000 CR3: 0000000133cf8002 CR4: 00000000003706e0
Dec  4 23:43:01 10.243.0.116 [  183.100468] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Dec  4 23:43:01 10.243.0.116 [  183.100471] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Dec  4 23:43:01 10.243.0.116 [  183.100475] Call Trace:
Dec  4 23:43:01 10.243.0.116 [  183.100477]  <TASK>
Dec  4 23:43:01 10.243.0.116 [  183.100481]  lock_acquire+0x150/0x3f0
Dec  4 23:43:01 10.243.0.116 [  183.100494]  ? lockdep_hardirqs_on_prepare+0x250/0x250
Dec  4 23:43:01 10.243.0.116 [  183.100503]  ? lock_release+0x2ac/0x440
Dec  4 23:43:01 10.243.0.116 [  183.100511]  ? __context_tracking_enter+0x6f/0x80
Dec  4 23:43:01 10.243.0.116 [  183.100518]  ? lock_downgrade+0x3e0/0x3e0
Dec  4 23:43:01 10.243.0.116 [  183.100527]  ? rcu_read_lock_sched_held+0x5b/0xd0
Dec  4 23:43:01 10.243.0.116 [  183.100537]  ? sched_clock+0x9/0x10
Dec  4 23:43:01 10.243.0.116 [  183.100546]  vtime_user_exit+0x73/0x150
Dec  4 23:43:01 10.243.0.116 [  183.100553]  ? __context_tracking_exit+0x60/0x70
Dec  4 23:43:01 10.243.0.116 [  183.100561]  __context_tracking_exit+0x60/0x70
Dec  4 23:43:01 10.243.0.116 [  183.100568]  syscall_enter_from_user_mode+0xd8/0x130
Dec  4 23:43:01 10.243.0.116 [  183.100576]  do_syscall_64+0x1d/0xa0
Dec  4 23:43:01 10.243.0.116 [  183.100586]  entry_SYSCALL_64_after_hwframe+0x61/0xcb
Dec  4 23:43:01 10.243.0.116 [  183.100593] RIP: 0033:0x7fec75afbe44
Dec  4 23:43:01 10.243.0.116 [  183.100599] Code: ff 31 c0 87 83 1c 06 00 00 83 f8 fe 0f 84 42 02 00 00 48 8b 54 24 08 be 18 00 00 00 b8 11 01 00 00 48 8d ba e0 02 00 00 0f 05 <f6> 82 14 06 00 00 04 0f 85 e9 01 00 00 48 8d 7c 24 10 48 c7 44 24
Dec  4 23:43:01 10.243.0.116 [  183.100605] RSP: 002b:00007fec6922c960 EFLAGS: 00000202 ORIG_RAX: 0000000000000111
Dec  4 23:43:01 10.243.0.116 [  183.100611] RAX: ffffffffffffffda RBX: 00007fec6923c700 RCX: 00007fec75afbe44
Dec  4 23:43:01 10.243.0.116 [  183.100615] RDX: 00007fec6923c700 RSI: 0000000000000018 RDI: 00007fec6923c9e0
Dec  4 23:43:01 10.243.0.116 [  183.100619] RBP: 0000000000000000 R08: 00007fec6923c700 R09: 00007fec6923c700
Dec  4 23:43:01 10.243.0.116 [  183.100622] R10: 00007fec6923c9d0 R11: 0000000000000202 R12: 0000000000000001
Dec  4 23:43:01 10.243.0.116 [  183.100626] R13: 0000000001001000 R14: 0000000000000000 R15: 00007fec6923c700
Dec  4 23:43:01 10.243.0.116 [  183.100637]  </TASK>
Dec  4 23:43:01 10.243.0.116 [  183.100640] irq event stamp: 6
Dec  4 23:43:01 10.243.0.116 [  183.100642] hardirqs last  enabled at (5): [<ffffffffae002725>] ret_from_fork+0x15/0x30
Dec  4 23:43:01 10.243.0.116 [  183.100651] hardirqs last disabled at (6): [<ffffffffaf13333d>] do_syscall_64+0x1d/0xa0
Dec  4 23:43:01 10.243.0.116 [  183.100659] softirqs last  enabled at (0): [<ffffffffae0c4ee2>] copy_process+0x12c2/0x3740
Dec  4 23:43:01 10.243.0.116 [  183.100666] softirqs last disabled at (0): [<0000000000000000>] 0x0
Dec  4 23:43:01 10.243.0.116 [  183.100670] ---[ end trace f5eed2984649a17b ]---
Dec  4 23:43:01 10.243.0.116 [  183.100674] possible reason: unannotated irqs-on.
Dec  4 23:43:01 10.243.0.116 [  183.100675] irq event stamp: 6
Dec  4 23:43:01 10.243.0.116 [  183.100677] hardirqs last  enabled at (5): [<ffffffffae002725>] ret_from_fork+0x15/0x30
Dec  4 23:43:01 10.243.0.116 [  183.100684] hardirqs last disabled at (6): [<ffffffffaf13333d>] do_syscall_64+0x1d/0xa0
Dec  4 23:43:01 10.243.0.116 [  183.100691] softirqs last  enabled at (0): [<ffffffffae0c4ee2>] copy_process+0x12c2/0x3740
Dec  4 23:43:01 10.243.0.116 [  183.100698] softirqs last disabled at (0): [<0000000000000000>] 0x0

[-- Attachment #2: smime.p7s --]
[-- Type: application/pkcs7-signature, Size: 6759 bytes --]

RE: [External] - Re: EVL Memory

[Russell Johnson]

[-- Attachment #1: Type: text/plain, Size: 207 bytes --]

By the way, I did notice that in "evl ps -l", the WCHAN column lists
"(null)" for all EVL Threads. I believe it used to list stuff like
ksem->wait, so not sure if something changed there.


Thanks,

Russell

[-- Attachment #2: smime.p7s --]
[-- Type: application/pkcs7-signature, Size: 6759 bytes --]

Re: [External] - Re: EVL Memory

[Philippe Gerum]

Russell Johnson <russell.johnson@kratosdefense.com> writes:

> [[S/MIME Signed Part:Undecided]]
> By the way, I did notice that in "evl ps -l", the WCHAN column lists
> "(null)" for all EVL Threads. I believe it used to list stuff like
> ksem->wait, so not sure if something changed there.
>
>

Nothing changed, I've been running evl ps loops here as well and did not
notice any weirdness of that kind, this does not look good since the ABI
did not change with these latest fixes.

Generally speaking, evl ps extracts all the information it displays from
/sys device files, as published by the real-time core. Could you check
the values exported for threads manually? e.g.

root@homelab-qemu-x86_64:/sys# cat /sys/devices/virtual/thread/Thread2/wchan 
Event1

-- 
Philippe.

RE: [External] - Re: EVL Memory

[Russell Johnson]

[-- Attachment #1: Type: text/plain, Size: 311 bytes --]

I had recently updated libevl to the latest "next" branch, but it appears I
had not rebuilt my app. I went ahead an rebuilt, and now I see the wchan
under /sys/devices/virtual/thread/ThreadName. I still do not see it under
evl ps though. My guess is that I have something messed up on my end.

Thanks,

Russell

[-- Attachment #2: smime.p7s --]
[-- Type: application/pkcs7-signature, Size: 6759 bytes --]

RE: [External] - Re: EVL Memory

[Russell Johnson]

[-- Attachment #1: Type: text/plain, Size: 81 bytes --]

Once I run as root, it works fine.. Sorry about the confusion.

Thanks,

Russell

[-- Attachment #2: smime.p7s --]
[-- Type: application/pkcs7-signature, Size: 6759 bytes --]

Re: [External] - Re: EVL Memory

[Philippe Gerum]

Russell Johnson <russell.johnson@kratosdefense.com> writes:

> [[S/MIME Signed Part:Undecided]]
> Once I run as root, it works fine.. Sorry about the confusion.
>
> Thanks,
>
> Russell
>
> [[End of S/MIME Signed Part]]

Nevertheless evl-ps outputs confusing information in this case, if not
plain broken. Ok, I'll have a look.

Thanks,

-- 
Philippe.

Re: [External] - Re: EVL Memory

[Philippe Gerum]

Russell Johnson <russell.johnson@kratosdefense.com> writes:

> [[S/MIME Signed Part:Undecided]]
> Just ran the latest kernel from the next branch with the lockdep turned back
> on and ftrace off for now. Good news is that I no longer see a boosting
> issue. Thanks again for the work to track that down. All that is left at the
> moment is one lockdep kernel splat which may not be important, but I
> attached anyway.
>

I'll have a look at this one too, this is depending on
CONFIG_DEBUG_LOCKDEP. I have not been able to reproduce it yet.  ftrace
should be back to normal as well, I fixed the recently broken tracepoint
causing crashes.

-- 
Philippe.

RE: [External] - Re: EVL Memory

[Russell Johnson]

[-- Attachment #1.1: Type: text/plain, Size: 101 bytes --]

I have attached another stack trace from another un on a different system as
well.

Thanks,

Russell

[-- Attachment #1.2: stack_trace_3.txt --]
[-- Type: text/plain, Size: 18561 bytes --]

Nov 14 23:30:57 10.243.1.49 [ 2957.529589] watchdog: BUG: soft lockup - CPU#0 stuck for 26s! [Main:30755]
Nov 14 23:30:57 10.243.1.49 [ 2957.550451] Modules linked in: xt_CHECKSUM xt_MASQUERADE tun netconsole ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 xt_conntrack ebtable_nat ip6table_nat ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_nat iptable_mangle iptable_security iptable_raw nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 ip_set nfnetlink ebtable_filter ebtables rfkill ip6table_filter ip6_tables iptable_filter sunrpc dm_mirror dm_region_hash dm_log dm_mod iTCO_wdt iTCO_vendor_support gpio_ich x86_pkg_temp_thermal mxm_wmi intel_powerclamp coretemp kvm_intel kvm irqbypass crct10dif_pclmul crc32_pclmul ghash_clmulni_intel aesni_intel crypto_simd cryptd rapl mei_me i2c_i801 lpc_ich intel_cstate pcspkr intel_pch_thermal i2c_smbus input_leds sg joydev ftdi_sio mfd_core mei ioatdma acpi_ipmi ipmi_si ipmi_devintf ipmi_msghandler wmi ip_tables xfs libcrc32c ast drm_vram_helper drm_kms_helper sr_mod sd_mod cdrom t10_pi syscopyarea sysfillrect sysimgblt fb_sys_fops drm_ttm_helper igb ttm
Nov 14 23:30:57 10.243.1.49  drm
Nov 14 23:30:57 10.243.1.49 [ 2957.550952]  ixgbe ahci libahci libata crc32c_intel mdio ptp pps_core i2c_algo_bit dca
Nov 14 23:30:57 10.243.1.49 [ 2957.839229] irq event stamp: 39024920
Nov 14 23:30:57 10.243.1.49 [ 2957.850329] hardirqs last  enabled at (39024919): [<ffffffffaf117f9e>] arch_pipeline_entry+0x5e/0x110
Nov 14 23:30:57 10.243.1.49 [ 2957.878257] hardirqs last disabled at (39024920): [<ffffffffae1c346b>] sync_current_irq_stage+0x4db/0x500
Nov 14 23:30:57 10.243.1.49 [ 2957.907229] softirqs last  enabled at (616958): [<ffffffffaf400403>] __do_softirq+0x403/0x57b
Nov 14 23:30:57 10.243.1.49 [ 2957.933046] softirqs last disabled at (616951): [<ffffffffae0d261d>] irq_exit_rcu+0x7d/0x150
Nov 14 23:30:57 10.243.1.49 [ 2957.958618] CPU: 0 PID: 30755 Comm: Main Not tainted 5.15.77evl-g6437d24c6dbc #1
Nov 14 23:30:57 10.243.1.49 [ 2957.981007] Hardware name: Supermicro Super Server/X10SDV-6C-TLN4F, BIOS 1.3 02/13/2018
Nov 14 23:30:57 10.243.1.49 [ 2958.005248] IRQ stage: Linux
Nov 14 23:30:57 10.243.1.49 [ 2958.014002] RIP: 0010:check_inband_stage+0x78/0xe0
Nov 14 23:30:58 10.243.1.49 [ 2958.028544] Code: 4c 8d ab d8 0d 00 00 be 08 00 00 00 4c 89 ef e8 7e a2 30 00 4c 89 ef e8 26 97 30 00 48 8b 83 d8 0d 00 00 a8 02 75 10 41 54 9d <5b> 41 5c 41 5d 41 5e 5d c3 cc cc cc cc 65 8b 05 64 56 e6 51 a9 00
Nov 14 23:30:58 10.243.1.49 [ 2958.085339] RSP: 0018:ffff8881683a7ad0 EFLAGS: 00000246 ORIG_RAX: 0000000000000000
Nov 14 23:30:58 10.243.1.49 [ 2958.108259] RAX: 0000000000000000 RBX: 0000000000000000 RCX: 0000000000000000
Nov 14 23:30:58 10.243.1.49 [ 2958.129875] RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000000
Nov 14 23:30:58 10.243.1.49 [ 2958.151483] RBP: ffff8881683a7af0 R08: 0000000000000000 R09: 0000000000000000
Nov 14 23:30:58 10.243.1.49 [ 2958.173084] R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000000
Nov 14 23:30:58 10.243.1.49 [ 2958.194711] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000
Nov 14 23:30:58 10.243.1.49 [ 2958.216325] FS:  00007fc7d3fff700(0000) GS:ffff88841be00000(0000) knlGS:0000000000000000
Nov 14 23:30:58 10.243.1.49 [ 2958.240836] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Nov 14 23:30:58 10.243.1.49 [ 2958.258250] CR2: 00005620615ca810 CR3: 0000000145d6e003 CR4: 00000000003706f0
Nov 14 23:30:58 10.243.1.49 [ 2958.279868] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Nov 14 23:30:58 10.243.1.49 [ 2958.301477] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Nov 14 23:30:58 10.243.1.49 [ 2958.323084] Call Trace:
Nov 14 23:30:58 10.243.1.49 [ 2958.335942]  <TASK>
Nov 14 23:30:58 10.243.1.49 [ 2958.347883]  inband_irq_disable+0x13/0x40
Nov 14 23:30:58 10.243.1.49 [ 2958.365423]  _raw_spin_lock_irq+0x50/0x60
Nov 14 23:30:58 10.243.1.49 [ 2958.382965]  rwsem_down_write_slowpath+0x90c/0xb70
Nov 14 23:30:58 10.243.1.49 [ 2958.402827]  ? downgrade_write+0x280/0x280
Nov 14 23:30:58 10.243.1.49 [ 2958.420469]  ? unlock_stage+0x6d/0x80
Nov 14 23:30:58 10.243.1.49 [ 2958.436874]  ? lock_acquire+0x1a4/0x3f0
Nov 14 23:30:58 10.243.1.49 [ 2958.453877]  ? enable_oob_stage+0x51/0x110
Nov 14 23:30:58 10.243.1.49 [ 2958.471683]  ? __kasan_check_read+0x11/0x20
Nov 14 23:30:58 10.243.1.49 [ 2958.489815]  down_write_killable+0x11e/0x140
Nov 14 23:30:58 10.243.1.49 [ 2958.508220]  ? down_write_killable+0xa1/0x140
Nov 14 23:30:58 10.243.1.49 [ 2958.526959]  ? down_write_killable+0x11e/0x140
Nov 14 23:30:58 10.243.1.49 [ 2958.545995]  ? __down_killable+0x230/0x230
Nov 14 23:30:58 10.243.1.49 [ 2958.564032]  ? security_mmap_file+0xa7/0xd0
Nov 14 23:30:58 10.243.1.49 [ 2958.582308]  vm_mmap_pgoff+0x133/0x240
Nov 14 23:30:58 10.243.1.49 [ 2958.599193]  ? __audit_syscall_entry+0x1b5/0x200
Nov 14 23:30:58 10.243.1.49 [ 2958.618559]  ? randomize_page+0x60/0x60
Nov 14 23:30:58 10.243.1.49 [ 2958.635418]  ? ktime_get_coarse_real_ts64+0x109/0x120
Nov 14 23:30:58 10.243.1.49 [ 2958.655814]  ? __kasan_check_read+0x11/0x20
Nov 14 23:30:58 10.243.1.49 [ 2958.673450]  ? __kasan_check_read+0x11/0x20
Nov 14 23:30:58 10.243.1.49 [ 2958.690881]  ? __kasan_check_write+0x14/0x20
Nov 14 23:30:58 10.243.1.49 [ 2958.708376]  ksys_mmap_pgoff+0xa4/0x350
Nov 14 23:30:58 10.243.1.49 [ 2958.724412]  ? mlock_future_check+0x90/0x90
Nov 14 23:30:58 10.243.1.49 [ 2958.741338]  ? __audit_syscall_entry+0x1b5/0x200
Nov 14 23:30:58 10.243.1.49 [ 2958.759492]  __x64_sys_mmap+0x88/0xb0
Nov 14 23:30:58 10.243.1.49 [ 2958.774686]  do_syscall_64+0x44/0xb0
Nov 14 23:30:58 10.243.1.49 [ 2958.789502]  entry_SYSCALL_64_after_hwframe+0x61/0xcb
Nov 14 23:30:58 10.243.1.49 [ 2958.808760] RIP: 0033:0x7fc7e42afd9a
Nov 14 23:30:58 10.243.1.49 [ 2958.823504] Code: cc 55 48 89 f5 53 48 89 fb 74 35 4d 63 f0 4c 63 fa 4d 89 e9 4d 89 f0 4d 63 d4 4c 89 fa 48 89 ee 48 89 df b8 09 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 52 5b 5d 41 5c 41 5d 41 5e 41 5f c3 0f 1f 00
Nov 14 23:30:58 10.243.1.49 [ 2958.888422] RSP: 002b:00007fc7d3fef4d8 EFLAGS: 00000206 ORIG_RAX: 0000000000000009
Nov 14 23:30:58 10.243.1.49 [ 2958.915462] RAX: ffffffffffffffda RBX: 00007fc7b8000000 RCX: 00007fc7e42afd9a
Nov 14 23:30:58 10.243.1.49 [ 2958.941169] RDX: 0000000000000000 RSI: 0000000004000000 RDI: 00007fc7b8000000
Nov 14 23:30:58 10.243.1.49 [ 2958.966914] RBP: 0000000004000000 R08: ffffffffffffffff R09: 0000000000000000
Nov 14 23:30:58 10.243.1.49 [ 2958.992494] R10: 0000000000004022 R11: 0000000000000206 R12: 0000000000004022
Nov 14 23:30:58 10.243.1.49 [ 2959.017924] R13: 0000000000000000 R14: ffffffffffffffff R15: 0000000000000000
Nov 14 23:30:58 10.243.1.49 [ 2959.043331]  </TASK>
Nov 14 23:31:25 10.243.1.49 [ 2985.529490] watchdog: BUG: soft lockup - CPU#0 stuck for 52s! [Main:30755]
Nov 14 23:31:25 10.243.1.49 [ 2985.554225] Modules linked in: xt_CHECKSUM xt_MASQUERADE tun netconsole ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 xt_conntrack ebtable_nat ip6table_nat ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_nat iptable_mangle iptable_security iptable_raw nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 ip_set nfnetlink ebtable_filter ebtables rfkill ip6table_filter ip6_tables iptable_filter sunrpc dm_mirror dm_region_hash dm_log dm_mod iTCO_wdt iTCO_vendor_support gpio_ich x86_pkg_temp_thermal mxm_wmi intel_powerclamp coretemp kvm_intel kvm irqbypass crct10dif_pclmul crc32_pclmul ghash_clmulni_intel aesni_intel crypto_simd cryptd rapl mei_me i2c_i801 lpc_ich intel_cstate pcspkr intel_pch_thermal i2c_smbus input_leds sg joydev ftdi_sio mfd_core mei ioatdma acpi_ipmi ipmi_si ipmi_devintf ipmi_msghandler wmi ip_tables xfs libcrc32c ast drm_vram_helper drm_kms_helper sr_mod sd_mod cdrom t10_pi syscopyarea sysfillrect sysimgblt fb_sys_fops drm_ttm_helper igb ttm
Nov 14 23:31:25 10.243.1.49  drm
Nov 14 23:31:25 10.243.1.49 [ 2985.554777]  ixgbe ahci libahci libata crc32c_intel mdio ptp pps_core i2c_algo_bit dca
Nov 14 23:31:25 10.243.1.49 [ 2985.884044] irq event stamp: 76245688
Nov 14 23:31:25 10.243.1.49 [ 2985.900283] hardirqs last  enabled at (76245687): [<ffffffffaf133052>] _raw_spin_unlock_irq+0x22/0x40
Nov 14 23:31:25 10.243.1.49 [ 2985.933431] hardirqs last disabled at (76245688): [<ffffffffae1c346b>] sync_current_irq_stage+0x4db/0x500
Nov 14 23:31:25 10.243.1.49 [ 2985.967696] softirqs last  enabled at (616958): [<ffffffffaf400403>] __do_softirq+0x403/0x57b
Nov 14 23:31:25 10.243.1.49 [ 2985.998848] softirqs last disabled at (616951): [<ffffffffae0d261d>] irq_exit_rcu+0x7d/0x150
Nov 14 23:31:25 10.243.1.49 [ 2986.029763] CPU: 0 PID: 30755 Comm: Main Tainted: G             L    5.15.77evl-g6437d24c6dbc #1
Nov 14 23:31:26 10.243.1.49 [ 2986.061829] Hardware name: Supermicro Super Server/X10SDV-6C-TLN4F, BIOS 1.3 02/13/2018
Nov 14 23:31:26 10.243.1.49 [ 2986.091634] IRQ stage: Linux
Nov 14 23:31:26 10.243.1.49 [ 2986.106121] RIP: 0010:stage_disabled+0x9/0x50
Nov 14 23:31:26 10.243.1.49 [ 2986.124991] Code: d8 0d 00 00 48 89 e5 e8 15 91 0a ff 5d 0f b6 c0 c3 cc cc cc cc 66 90 66 2e 0f 1f 84 00 00 00 00 00 55 48 89 e5 e8 f7 8f 0a ff <ba> 01 00 00 00 f6 c4 02 74 10 65 8b 05 16 e8 f0 50 31 d2 a9 00 00
Nov 14 23:31:26 10.243.1.49 [ 2986.193530] RSP: 0018:ffff8881683a7ab8 EFLAGS: 00000246 ORIG_RAX: 0000000000000000
Nov 14 23:31:26 10.243.1.49 [ 2986.222546] RAX: 0000000000000000 RBX: 0000000000000000 RCX: 0000000000000000
Nov 14 23:31:26 10.243.1.49 [ 2986.250251] RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000000
Nov 14 23:31:26 10.243.1.49 [ 2986.277945] RBP: ffff8881683a7ab8 R08: 0000000000000000 R09: 0000000000000000
Nov 14 23:31:26 10.243.1.49 [ 2986.305661] R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000000
Nov 14 23:31:26 10.243.1.49 [ 2986.333311] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000
Nov 14 23:31:26 10.243.1.49 [ 2986.360912] FS:  00007fc7d3fff700(0000) GS:ffff88841be00000(0000) knlGS:0000000000000000
Nov 14 23:31:26 10.243.1.49 [ 2986.391456] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Nov 14 23:31:26 10.243.1.49 [ 2986.414993] CR2: 00005620615ca810 CR3: 0000000145d6e003 CR4: 00000000003706f0
Nov 14 23:31:26 10.243.1.49 [ 2986.442836] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Nov 14 23:31:26 10.243.1.49 [ 2986.470728] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Nov 14 23:31:26 10.243.1.49 [ 2986.498600] Call Trace:
Nov 14 23:31:26 10.243.1.49 [ 2986.512327]  <TASK>
Nov 14 23:31:26 10.243.1.49 [ 2986.524956]  ? __inband_irq_enable+0x6f/0x80
Nov 14 23:31:26 10.243.1.49 [ 2986.544164]  ? inband_irq_enable+0x1b/0x30
Nov 14 23:31:26 10.243.1.49 [ 2986.562850]  ? _raw_spin_unlock_irq+0x27/0x40
Nov 14 23:31:26 10.243.1.49 [ 2986.582295]  ? rwsem_down_write_slowpath+0x4df/0xb70
Nov 14 23:31:26 10.243.1.49 [ 2986.603606]  ? downgrade_write+0x280/0x280
Nov 14 23:31:26 10.243.1.49 [ 2986.622099]  ? unlock_stage+0x6d/0x80
Nov 14 23:31:26 10.243.1.49 [ 2986.639128]  ? lock_acquire+0x1a4/0x3f0
Nov 14 23:31:26 10.243.1.49 [ 2986.656627]  ? enable_oob_stage+0x51/0x110
Nov 14 23:31:26 10.243.1.49 [ 2986.674811]  ? __kasan_check_read+0x11/0x20
Nov 14 23:31:26 10.243.1.49 [ 2986.693230]  ? down_write_killable+0x11e/0x140
Nov 14 23:31:26 10.243.1.49 [ 2986.712419]  ? down_write_killable+0xa1/0x140
Nov 14 23:31:26 10.243.1.49 [ 2986.731314]  ? down_write_killable+0x11e/0x140
Nov 14 23:31:26 10.243.1.49 [ 2986.750393]  ? __down_killable+0x230/0x230
Nov 14 23:31:26 10.243.1.49 [ 2986.768375]  ? security_mmap_file+0xa7/0xd0
Nov 14 23:31:26 10.243.1.49 [ 2986.786583]  ? vm_mmap_pgoff+0x133/0x240
Nov 14 23:31:26 10.243.1.49 [ 2986.803839]  ? __audit_syscall_entry+0x1b5/0x200
Nov 14 23:31:26 10.243.1.49 [ 2986.823056]  ? randomize_page+0x60/0x60
Nov 14 23:31:26 10.243.1.49 [ 2986.839753]  ? ktime_get_coarse_real_ts64+0x109/0x120
Nov 14 23:31:26 10.243.1.49 [ 2986.859991]  ? __kasan_check_read+0x11/0x20
Nov 14 23:31:26 10.243.1.49 [ 2986.877444]  ? __kasan_check_read+0x11/0x20
Nov 14 23:31:26 10.243.1.49 [ 2986.894726]  ? __kasan_check_write+0x14/0x20
Nov 14 23:31:26 10.243.1.49 [ 2986.912072]  ? ksys_mmap_pgoff+0xa4/0x350
Nov 14 23:31:26 10.243.1.49 [ 2986.928512]  ? mlock_future_check+0x90/0x90
Nov 14 23:31:26 10.243.1.49 [ 2986.945320]  ? __audit_syscall_entry+0x1b5/0x200
Nov 14 23:31:26 10.243.1.49 [ 2986.963379]  ? __x64_sys_mmap+0x88/0xb0
Nov 14 23:31:26 10.243.1.49 [ 2986.979437]  ? do_syscall_64+0x44/0xb0
Nov 14 23:31:26 10.243.1.49 [ 2986.994732]  ? entry_SYSCALL_64_after_hwframe+0x61/0xcb
Nov 14 23:31:26 10.243.1.49 [ 2987.014465]  </TASK>
Nov 14 23:31:34 10.243.1.49 [ 2994.750477] rcu: INFO: rcu_sched self-detected stall on CPU
Nov 14 23:31:34 10.243.1.49 [ 2994.771198] rcu: #0110-....: (61952 ticks this GP) idle=637/1/0x4000000000000000 softirq=479020/479020 fqs=15011
Nov 14 23:31:34 10.243.1.49 [ 2994.806625] #011(t=65000 jiffies g=779101 q=11085)
Nov 14 23:31:34 10.243.1.49 [ 2994.824337] NMI backtrace for cpu 0
Nov 14 23:31:34 10.243.1.49 [ 2994.838800] CPU: 0 PID: 30755 Comm: Main Tainted: G             L    5.15.77evl-g6437d24c6dbc #1
Nov 14 23:31:34 10.243.1.49 [ 2994.869184] Hardware name: Supermicro Super Server/X10SDV-6C-TLN4F, BIOS 1.3 02/13/2018
Nov 14 23:31:34 10.243.1.49 [ 2994.897430] IRQ stage: Linux
Nov 14 23:31:34 10.243.1.49 [ 2994.909888] Call Trace:
Nov 14 23:31:34 10.243.1.49 [ 2994.920975]  <IRQ>
Nov 14 23:31:34 10.243.1.49 [ 2994.930771]  dump_stack_lvl+0x82/0xbc
Nov 14 23:31:34 10.243.1.49 [ 2994.945563]  dump_stack+0x10/0x16
Nov 14 23:31:34 10.243.1.49 [ 2994.959239]  nmi_cpu_backtrace.cold.10+0x68/0xa9
Nov 14 23:31:34 10.243.1.49 [ 2994.976927]  ? lapic_can_unplug_cpu+0xa0/0xa0
Nov 14 23:31:34 10.243.1.49 [ 2994.993781]  nmi_trigger_cpumask_backtrace+0x170/0x180
Nov 14 23:31:34 10.243.1.49 [ 2995.012954]  arch_trigger_cpumask_backtrace+0x14/0x20
Nov 14 23:31:34 10.243.1.49 [ 2995.031810]  rcu_dump_cpu_stacks+0x138/0x1c8
Nov 14 23:31:34 10.243.1.49 [ 2995.048328]  rcu_sched_clock_irq.cold.99+0x45/0x317
Nov 14 23:31:35 10.243.1.49 [ 2995.067017]  ? __kasan_check_write+0x14/0x20
Nov 14 23:31:35 10.243.1.49 [ 2995.083517]  ? tick_sched_do_timer+0x90/0x90
Nov 14 23:31:35 10.243.1.49 [ 2995.100006]  update_process_times+0xc2/0xf0
Nov 14 23:31:35 10.243.1.49 [ 2995.116252]  tick_sched_handle.isra.21+0x36/0x90
Nov 14 23:31:35 10.243.1.49 [ 2995.133802]  tick_sched_timer+0x7c/0xa0
Nov 14 23:31:35 10.243.1.49 [ 2995.148978]  __hrtimer_run_queues+0x38d/0x730
Nov 14 23:31:35 10.243.1.49 [ 2995.165758]  ? enqueue_hrtimer+0x180/0x180
Nov 14 23:31:35 10.243.1.49 [ 2995.181730]  ? inband_irq_restore+0xe/0x20
Nov 14 23:31:35 10.243.1.49 [ 2995.197681]  ? ktime_get_update_offsets_now+0xee/0x200
Nov 14 23:31:35 10.243.1.49 [ 2995.216815]  hrtimer_interrupt+0x1a5/0x350
Nov 14 23:31:35 10.243.1.49 [ 2995.232848]  proxy_irq_handler+0x32/0x40
Nov 14 23:31:35 10.243.1.49 [ 2995.248314]  handle_synthetic_irq+0xd2/0x300
Nov 14 23:31:35 10.243.1.49 [ 2995.264855]  do_irq_inband+0x2a/0x40
Nov 14 23:31:35 10.243.1.49 [ 2995.279359]  arch_do_IRQ_pipelined+0xb7/0x5a0
Nov 14 23:31:35 10.243.1.49 [ 2995.296235]  </IRQ>
Nov 14 23:31:35 10.243.1.49 [ 2995.306254]  <TASK>
Nov 14 23:31:35 10.243.1.49 [ 2995.316240]  sync_current_irq_stage+0x2de/0x500
Nov 14 23:31:35 10.243.1.49 [ 2995.333723]  __inband_irq_enable+0x6f/0x80
Nov 14 23:31:35 10.243.1.49 [ 2995.349775]  inband_irq_enable+0x1b/0x30
Nov 14 23:31:35 10.243.1.49 [ 2995.365263]  _raw_spin_unlock_irq+0x27/0x40
Nov 14 23:31:35 10.243.1.49 [ 2995.381554]  rwsem_down_write_slowpath+0x4df/0xb70
Nov 14 23:31:35 10.243.1.49 [ 2995.399696]  ? downgrade_write+0x280/0x280
Nov 14 23:31:35 10.243.1.49 [ 2995.415775]  ? unlock_stage+0x6d/0x80
Nov 14 23:31:35 10.243.1.49 [ 2995.430739]  ? lock_acquire+0x1a4/0x3f0
Nov 14 23:31:35 10.243.1.49 [ 2995.446074]  ? enable_oob_stage+0x51/0x110
Nov 14 23:31:35 10.243.1.49 [ 2995.462166]  ? __kasan_check_read+0x11/0x20
Nov 14 23:31:35 10.243.1.49 [ 2995.478523]  down_write_killable+0x11e/0x140
Nov 14 23:31:35 10.243.1.49 [ 2995.495116]  ? down_write_killable+0xa1/0x140
Nov 14 23:31:35 10.243.1.49 [ 2995.511987]  ? down_write_killable+0x11e/0x140
Nov 14 23:31:35 10.243.1.49 [ 2995.529078]  ? __down_killable+0x230/0x230
Nov 14 23:31:35 10.243.1.49 [ 2995.545097]  ? security_mmap_file+0xa7/0xd0
Nov 14 23:31:35 10.243.1.49 [ 2995.561348]  vm_mmap_pgoff+0x133/0x240
Nov 14 23:31:35 10.243.1.49 [ 2995.576380]  ? __audit_syscall_entry+0x1b5/0x200
Nov 14 23:31:35 10.243.1.49 [ 2995.593742]  ? randomize_page+0x60/0x60
Nov 14 23:31:35 10.243.1.49 [ 2995.608709]  ? ktime_get_coarse_real_ts64+0x109/0x120
Nov 14 23:31:35 10.243.1.49 [ 2995.627285]  ? __kasan_check_read+0x11/0x20
Nov 14 23:31:35 10.243.1.49 [ 2995.643179]  ? __kasan_check_read+0x11/0x20
Nov 14 23:31:35 10.243.1.49 [ 2995.658946]  ? __kasan_check_write+0x14/0x20
Nov 14 23:31:35 10.243.1.49 [ 2995.674966]  ksys_mmap_pgoff+0xa4/0x350
Nov 14 23:31:35 10.243.1.49 [ 2995.689710]  ? mlock_future_check+0x90/0x90
Nov 14 23:31:35 10.243.1.49 [ 2995.705527]  ? __audit_syscall_entry+0x1b5/0x200
Nov 14 23:31:35 10.243.1.49 [ 2995.722708]  __x64_sys_mmap+0x88/0xb0
Nov 14 23:31:35 10.243.1.49 [ 2995.736987]  do_syscall_64+0x44/0xb0
Nov 14 23:31:35 10.243.1.49 [ 2995.750955]  entry_SYSCALL_64_after_hwframe+0x61/0xcb
Nov 14 23:31:35 10.243.1.49 [ 2995.769381] RIP: 0033:0x7fc7e42afd9a
Nov 14 23:31:35 10.243.1.49 [ 2995.783350] Code: cc 55 48 89 f5 53 48 89 fb 74 35 4d 63 f0 4c 63 fa 4d 89 e9 4d 89 f0 4d 63 d4 4c 89 fa 48 89 ee 48 89 df b8 09 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 52 5b 5d 41 5c 41 5d 41 5e 41 5f c3 0f 1f 00
Nov 14 23:31:35 10.243.1.49 [ 2995.846809] RSP: 002b:00007fc7d3fef4d8 EFLAGS: 00000206 ORIG_RAX: 0000000000000009
Nov 14 23:31:35 10.243.1.49 [ 2995.873220] RAX: ffffffffffffffda RBX: 00007fc7b8000000 RCX: 00007fc7e42afd9a
Nov 14 23:31:35 10.243.1.49 [ 2995.898402] RDX: 0000000000000000 RSI: 0000000004000000 RDI: 00007fc7b8000000
Nov 14 23:31:35 10.243.1.49 [ 2995.923598] RBP: 0000000004000000 R08: ffffffffffffffff R09: 0000000000000000
Nov 14 23:31:35 10.243.1.49 [ 2995.948794] R10: 0000000000004022 R11: 0000000000000206 R12: 0000000000004022
Nov 14 23:31:35 10.243.1.49 [ 2995.974025] R13: 0000000000000000 R14: ffffffffffffffff R15: 0000000000000000
Nov 14 23:31:35 10.243.1.49 [ 2995.999318]  </TASK>

[-- Attachment #2: smime.p7s --]
[-- Type: application/pkcs7-signature, Size: 6759 bytes --]