[PATCH RFC] panic: Avoid extra noisy messages due to stopped cpus

[PATCH RFC] panic: Avoid extra noisy messages due to stopped cpus

[Feng Tang]

Sometimes when debugging kernel panic, we saw many extra noisy error
messages after the expected end:

[   35.743249] ---[ end Kernel panic - not syncing: Fatal exception
[   35.749975] ------------[ cut here ]------------

These messages may overflow the sceen (framebuffer) and make debugging
much difficulter.

This hack patch just quickly prevent these noisy message, and would
really like to get some comments and suggestions.

I have tried other ways like adding a panic notifier block inside
tick/sched code to cancel tick_sched timer in panic case, which
also works.

These extra messages are of 2 kinds:
a)
	 WARNING: CPU: 1 PID: 280 at kernel/sched/core.c:1198 set_task_cpu+0x183/0x190
	 Call Trace:
	  <IRQ>
	  try_to_wake_up+0x157/0x430
	  default_wake_function+0xd/0x10
	  autoremove_wake_function+0x11/0x60
	  __wake_up_common+0x8a/0x160
	  __wake_up_common_lock+0x6c/0x90
	  __wake_up+0xe/0x10
	  wake_up_klogd_work_func+0x3b/0x60
	  irq_work_run_list+0x4e/0x80
	  irq_work_tick+0x40/0x50
	  update_process_times+0x3d/0x50
	  tick_sched_timer+0x38/0x80
	  __hrtimer_run_queues+0xce/0x200
	  hrtimer_interrupt+0xac/0x1f0
	  smp_apic_timer_interrupt+0x6e/0x140
	  apic_timer_interrupt+0x8e/0xa0

b)
	sched: Unexpected reschedule of offline CPU#0!
	 ------------[ cut here ]------------
	 WARNING: CPU: 1 PID: 300 at arch/x86/kernel/smp.c:141 native_smp_send_reschedule+0x3d/0x50
	  trigger_load_balance+0x125/0x230
	  scheduler_tick+0xa2/0xd0
	  update_process_times+0x42/0x50
	  tick_sched_handle.isra.5+0x21/0x60
	  tick_sched_timer+0x38/0x80
	  __hrtimer_run_queues+0xce/0x200
	  hrtimer_interrupt+0xac/0x1f0
	  smp_apic_timer_interrupt+0x6e/0x140
	  apic_timer_interrupt+0x8e/0xa0

Signed-off-by: Feng Tang <feng.tang@intel.com>
---
 arch/x86/kernel/process.c | 1 +
 kernel/sched/fair.c       | 2 +-
 2 files changed, 2 insertions(+), 1 deletion(-)

diff --git a/arch/x86/kernel/process.c b/arch/x86/kernel/process.c
index c93fcfd..b703862 100644
--- a/arch/x86/kernel/process.c
+++ b/arch/x86/kernel/process.c
@@ -520,6 +520,7 @@ void stop_this_cpu(void *dummy)
 	 * Remove this CPU:
 	 */
 	set_cpu_online(smp_processor_id(), false);
+	set_cpu_active(smp_processor_id(), false);
 	disable_local_APIC();
 	mcheck_cpu_clear(this_cpu_ptr(&cpu_info));
 
diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c
index 7fc4a37..cf41b7b 100644
--- a/kernel/sched/fair.c
+++ b/kernel/sched/fair.c
@@ -9034,7 +9034,7 @@ static inline int find_new_ilb(void)
 {
 	int ilb = cpumask_first(nohz.idle_cpus_mask);
 
-	if (ilb < nr_cpu_ids && idle_cpu(ilb))
+	if (ilb < nr_cpu_ids && idle_cpu(ilb) && cpu_online(ilb))
 		return ilb;
 
 	return nr_cpu_ids;
-- 
2.7.4

Re: [PATCH RFC] panic: Avoid extra noisy messages due to stopped cpus

[Peter Zijlstra]

On Thu, Oct 11, 2018 at 03:17:48PM +0800, Feng Tang wrote:
> Sometimes when debugging kernel panic, we saw many extra noisy error
> messages after the expected end:
> 
> [   35.743249] ---[ end Kernel panic - not syncing: Fatal exception
> [   35.749975] ------------[ cut here ]------------
> 
> These messages may overflow the sceen (framebuffer) and make debugging
> much difficulter.

*blink* you actually using the framebuffer for debugging ?! Why the heck
are you doing that?

> diff --git a/arch/x86/kernel/process.c b/arch/x86/kernel/process.c
> index c93fcfd..b703862 100644
> --- a/arch/x86/kernel/process.c
> +++ b/arch/x86/kernel/process.c
> @@ -520,6 +520,7 @@ void stop_this_cpu(void *dummy)
>  	 * Remove this CPU:
>  	 */
>  	set_cpu_online(smp_processor_id(), false);
> +	set_cpu_active(smp_processor_id(), false);
>  	disable_local_APIC();
>  	mcheck_cpu_clear(this_cpu_ptr(&cpu_info));
>  

WTH is stop_this_cpu() and how do we even get here with active still
set?

> diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c
> index 7fc4a37..cf41b7b 100644
> --- a/kernel/sched/fair.c
> +++ b/kernel/sched/fair.c
> @@ -9034,7 +9034,7 @@ static inline int find_new_ilb(void)
>  {
>  	int ilb = cpumask_first(nohz.idle_cpus_mask);
>  
> -	if (ilb < nr_cpu_ids && idle_cpu(ilb))
> +	if (ilb < nr_cpu_ids && idle_cpu(ilb) && cpu_online(ilb))
>  		return ilb;
>  
>  	return nr_cpu_ids;


Similar, this is the result of taking the CPU away without going through
the normal path. You're doing something dodgy.

Re: [PATCH RFC] panic: Avoid extra noisy messages due to stopped cpus

[Feng Tang]

Hi Peter,

On Thu, Oct 11, 2018 at 11:35:00AM +0200, Peter Zijlstra wrote:
> On Thu, Oct 11, 2018 at 03:17:48PM +0800, Feng Tang wrote:
> > Sometimes when debugging kernel panic, we saw many extra noisy error
> > messages after the expected end:
> > 
> > [   35.743249] ---[ end Kernel panic - not syncing: Fatal exception
> > [   35.749975] ------------[ cut here ]------------
> > 
> > These messages may overflow the sceen (framebuffer) and make debugging
> > much difficulter.
> 
> *blink* you actually using the framebuffer for debugging ?! Why the heck
> are you doing that?

I didn't make it clear.

I just wanted to show some use case, that when I work under command line
mode, if the kernel panic (without reboot configured), it will pop out
the panic message, and these noisy message may keep poping up and I
hardly see the actual panic message on the screen.

> 
> > diff --git a/arch/x86/kernel/process.c b/arch/x86/kernel/process.c
> > index c93fcfd..b703862 100644
> > --- a/arch/x86/kernel/process.c
> > +++ b/arch/x86/kernel/process.c
> > @@ -520,6 +520,7 @@ void stop_this_cpu(void *dummy)
> >  	 * Remove this CPU:
> >  	 */
> >  	set_cpu_online(smp_processor_id(), false);
> > +	set_cpu_active(smp_processor_id(), false);
> >  	disable_local_APIC();
> >  	mcheck_cpu_clear(this_cpu_ptr(&cpu_info));
> >  
> 
> WTH is stop_this_cpu() and how do we even get here with active still
> set?

Current panic() will call
	smp_send_stop
		native_stop_other_cpus
			apic->send_IPI_allbutself(REBOOT_VECTOR)
				smp_reboot_interrupt
					stop_this_cpu

> > diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c
> > index 7fc4a37..cf41b7b 100644
> > --- a/kernel/sched/fair.c
> > +++ b/kernel/sched/fair.c
> > @@ -9034,7 +9034,7 @@ static inline int find_new_ilb(void)
> >  {
> >  	int ilb = cpumask_first(nohz.idle_cpus_mask);
> >  
> > -	if (ilb < nr_cpu_ids && idle_cpu(ilb))
> > +	if (ilb < nr_cpu_ids && idle_cpu(ilb) && cpu_online(ilb))
> >  		return ilb;
> >  
> >  	return nr_cpu_ids;
> 
> 
> Similar, this is the result of taking the CPU away without going through
> the normal path. You're doing something dodgy.

You are right, it's hacky. As I said in the commit log, suggestions are
appreciated.

Current emergercy handling of panic didn't notify the idle subsystem of
other cpus being stopped.

I have tried several other ways:
a) add a panic notifier to cancel the schdule timer when panic happens
b) don't re-enable the local interrupt inside panic() after normal panic
handling.

Thanks,
Feng

Re: [PATCH RFC] panic: Avoid extra noisy messages due to stopped cpus

[Feng Tang]

Hi maintainers,

On Thu, Oct 11, 2018 at 05:59:41PM +0800, Feng Tang wrote:
> Hi Peter,
> 
> On Thu, Oct 11, 2018 at 11:35:00AM +0200, Peter Zijlstra wrote:
> > On Thu, Oct 11, 2018 at 03:17:48PM +0800, Feng Tang wrote:
> > > Sometimes when debugging kernel panic, we saw many extra noisy error
> > > messages after the expected end:
> > > 
> > > [   35.743249] ---[ end Kernel panic - not syncing: Fatal exception
> > > [   35.749975] ------------[ cut here ]------------
> > > 
> > > These messages may overflow the sceen (framebuffer) and make debugging
> > > much difficulter.
> > 
> > *blink* you actually using the framebuffer for debugging ?! Why the heck
> > are you doing that?
> 
> I didn't make it clear.
> 
> I just wanted to show some use case, that when I work under command line
> mode, if the kernel panic (without reboot configured), it will pop out
> the panic message, and these noisy message may keep poping up and I
> hardly see the actual panic message on the screen.
> 
> > 
> > > diff --git a/arch/x86/kernel/process.c b/arch/x86/kernel/process.c
> > > index c93fcfd..b703862 100644
> > > --- a/arch/x86/kernel/process.c
> > > +++ b/arch/x86/kernel/process.c
> > > @@ -520,6 +520,7 @@ void stop_this_cpu(void *dummy)
> > >  	 * Remove this CPU:
> > >  	 */
> > >  	set_cpu_online(smp_processor_id(), false);
> > > +	set_cpu_active(smp_processor_id(), false);
> > >  	disable_local_APIC();
> > >  	mcheck_cpu_clear(this_cpu_ptr(&cpu_info));
> > >  
> > 
> > WTH is stop_this_cpu() and how do we even get here with active still
> > set?
> 
> Current panic() will call
> 	smp_send_stop
> 		native_stop_other_cpus
> 			apic->send_IPI_allbutself(REBOOT_VECTOR)
> 				smp_reboot_interrupt
> 					stop_this_cpu
> 
> > > diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c
> > > index 7fc4a37..cf41b7b 100644
> > > --- a/kernel/sched/fair.c
> > > +++ b/kernel/sched/fair.c
> > > @@ -9034,7 +9034,7 @@ static inline int find_new_ilb(void)
> > >  {
> > >  	int ilb = cpumask_first(nohz.idle_cpus_mask);
> > >  
> > > -	if (ilb < nr_cpu_ids && idle_cpu(ilb))
> > > +	if (ilb < nr_cpu_ids && idle_cpu(ilb) && cpu_online(ilb))
> > >  		return ilb;
> > >  
> > >  	return nr_cpu_ids;
> > 
> > 
> > Similar, this is the result of taking the CPU away without going through
> > the normal path. You're doing something dodgy.
> 
> You are right, it's hacky. As I said in the commit log, suggestions are
> appreciated.
> 
> Current emergercy handling of panic didn't notify the idle subsystem of
> other cpus being stopped.
> 
> I have tried several other ways:
> a) add a panic notifier to cancel the schdule timer when panic happens
> b) don't re-enable the local interrupt inside panic() after normal panic
> handling.

Could you help to review the patch? thanks,

- Feng

[PATCH v2] panic: Avoid the extra noise dmesg

[Feng Tang]

Sometimes when debugging kernel panic, we saw many extra noisy error
messages after the expected end:

[   35.743249] ---[ end Kernel panic - not syncing: Fatal exception
[   35.749975] ------------[ cut here ]------------

When panic happens under console mode, the screen will first show the panic
call stack, but will immediately be overrided by noisy extra messages, like
this for every CPU except the panic one:

	WARNING: CPU: 1 PID: 280 at kernel/sched/core.c:1198 set_task_cpu+0x183/0x190
	Call Trace:
	<IRQ>
	try_to_wake_up
	default_wake_function
	autoremove_wake_function
	__wake_up_common
	__wake_up_common_lock
	__wake_up
	wake_up_klogd_work_func
	irq_work_run_list
	irq_work_tick
	update_process_times
	tick_sched_timer
	__hrtimer_run_queues
	hrtimer_interrupt
	smp_apic_timer_interrupt
	apic_timer_interrupt

Which make debugging more difficult, as the original context is
lost on screen.

Keeping the CPU IRQ disabled will void these messages, as we code
run to this point, the user has chosed not to reboot, nor do
some extra handling with panic notifier, not much point in
re-enabling the interrupt.

Signed-off-by: Feng Tang <feng.tang@intel.com>
Cc: Thomas Gleixner <tglx@linutronix.de>
Cc: Kees Cook <keescook@chromium.org>
Cc: Borislav Petkov <bp@suse.de>
---
 kernel/panic.c | 1 -
 1 file changed, 1 deletion(-)

diff --git a/kernel/panic.c b/kernel/panic.c
index f6d549a29a5c..a616e55ab57e 100644
--- a/kernel/panic.c
+++ b/kernel/panic.c
@@ -295,7 +295,6 @@ void panic(const char *fmt, ...)
 	}
 #endif
 	pr_emerg("---[ end Kernel panic - not syncing: %s ]---\n", buf);
-	local_irq_enable();
 	for (i = 0; ; i += PANIC_TIMER_STEP) {
 		touch_softlockup_watchdog();
 		if (i >= i_next) {
-- 
2.14.1

Re: [PATCH v2] panic: Avoid the extra noise dmesg

[Andrew Morton]

On Thu, 8 Nov 2018 21:05:03 +0800 Feng Tang <feng.tang@intel.com> wrote:

> Sometimes when debugging kernel panic, we saw many extra noisy error
> messages after the expected end:
> 
> [   35.743249] ---[ end Kernel panic - not syncing: Fatal exception
> [   35.749975] ------------[ cut here ]------------

What are these "noisy messages"?  The above looks OK?

> When panic happens under console mode, the screen will first show the panic
> call stack, but will immediately be overrided by noisy extra messages, like
> this for every CPU except the panic one:
> 
> 	WARNING: CPU: 1 PID: 280 at kernel/sched/core.c:1198 set_task_cpu+0x183/0x190
> 	Call Trace:
> 	<IRQ>
> 	try_to_wake_up
> 	default_wake_function
> 	autoremove_wake_function
> 	__wake_up_common
> 	__wake_up_common_lock
> 	__wake_up
> 	wake_up_klogd_work_func
> 	irq_work_run_list
> 	irq_work_tick
> 	update_process_times
> 	tick_sched_timer
> 	__hrtimer_run_queues
> 	hrtimer_interrupt
> 	smp_apic_timer_interrupt
> 	apic_timer_interrupt

And this output looks OK as well.

> Which make debugging more difficult, as the original context is
> lost on screen.
> 
> Keeping the CPU IRQ disabled will void these messages, as we code
> run to this point, the user has chosed not to reboot, nor do
> some extra handling with panic notifier, not much point in
> re-enabling the interrupt.

And please have another attempt at the above paragraph?

> --- a/kernel/panic.c
> +++ b/kernel/panic.c
> @@ -295,7 +295,6 @@ void panic(const char *fmt, ...)
>  	}
>  #endif
>  	pr_emerg("---[ end Kernel panic - not syncing: %s ]---\n", buf);
> -	local_irq_enable();
>  	for (i = 0; ; i += PANIC_TIMER_STEP) {
>  		touch_softlockup_watchdog();
>  		if (i >= i_next) {

The patch looks reasonable, but the description is just too hard for me
to follow, sorry.  Please spend a bit more time over that and resend?

Re: [PATCH v2] panic: Avoid the extra noise dmesg

[Feng Tang]

Hi Andrew,

On Mon, Nov 26, 2018 at 03:59:52PM -0800, Andrew Morton wrote:
> On Thu, 8 Nov 2018 21:05:03 +0800 Feng Tang <feng.tang@intel.com> wrote:
> 
> > Sometimes when debugging kernel panic, we saw many extra noisy error
> > messages after the expected end:
> > 
> > [   35.743249] ---[ end Kernel panic - not syncing: Fatal exception
> > [   35.749975] ------------[ cut here ]------------
> 
> What are these "noisy messages"?  The above looks OK?

Yes, the above is the sane messages :) and contains the necessary context
info for debug.

The noisy messages are those following messages, I tried to capture one with
sysrq and paste it in the tail of this mail as it's big (for a 4 cores
machine) 

> > When panic happens under console mode, the screen will first show the panic
> > call stack, but will immediately be overrided by noisy extra messages, like
> > this for every CPU except the panic one:
> > 
> > 	WARNING: CPU: 1 PID: 280 at kernel/sched/core.c:1198 set_task_cpu+0x183/0x190
> > 	Call Trace:
> > 	<IRQ>
> > 	try_to_wake_up
> > 	default_wake_function
> > 	autoremove_wake_function
> > 	__wake_up_common
> > 	__wake_up_common_lock
> > 	__wake_up
> > 	wake_up_klogd_work_func
> > 	irq_work_run_list
> > 	irq_work_tick
> > 	update_process_times
> > 	tick_sched_timer
> > 	__hrtimer_run_queues
> > 	hrtimer_interrupt
> > 	smp_apic_timer_interrupt
> > 	apic_timer_interrupt
> 
> And this output looks OK as well.
 
Yes, this is fine also. What I mean is it will be a little noisy if each and
every CPU will dump similar info, which will make it hard to find the real panic
context if a user has no serial console but a LCD.

> > Which make debugging more difficult, as the original context is
> > lost on screen.
> > 
> > Keeping the CPU IRQ disabled will void these messages, as we code
> > run to this point, the user has chosed not to reboot, nor do
> > some extra handling with panic notifier, not much point in
> > re-enabling the interrupt.
> 
> And please have another attempt at the above paragraph?

Sorry for the unclear info, will try to sanitize it.

> > --- a/kernel/panic.c
> > +++ b/kernel/panic.c
> > @@ -295,7 +295,6 @@ void panic(const char *fmt, ...)
> >  	}
> >  #endif
> >  	pr_emerg("---[ end Kernel panic - not syncing: %s ]---\n", buf);
> > -	local_irq_enable();
> >  	for (i = 0; ; i += PANIC_TIMER_STEP) {
> >  		touch_softlockup_watchdog();
> >  		if (i >= i_next) {
> 
> The patch looks reasonable, but the description is just too hard for me
> to follow, sorry.  Please spend a bit more time over that and resend?

Sure and thanks for your review! 

- Feng
-------

Panic log:

#echo c > /proc/sysrq-trigger 
[   62.918474] sysrq: SysRq : Trigger a crash
[   62.918643] BUG: unable to handle kernel NULL pointer dereference at 0000000000000000
[   62.918906] PGD 67e36067 P4D 67e36067 PUD 6189f067 PMD 0 
[   62.919092] Oops: 0002 [#1] PREEMPT SMP
[   62.919226] CPU: 3 PID: 336 Comm: bash Tainted: G     U  W         4.19.0-19.iot-lts2018-sos-g6b95688-dirty #44
[   62.919563] RIP: 0010:sysrq_handle_crash+0x16/0x20
[   62.919726] Code: d5 e8 8e c2 bc ff 0f 1f 40 00 66 2e 0f 1f 84 00 00 00 00 00 55 48 89 e5 e8 77 31 c3 ff c7 05 65 9e 16 01 01 00 00 00 0f ae f8 <c6> 04 25 00 00 00 00 01 5d c3 55 48 89 e5 fb 66 0f 1f 44 00 00 e8
[   62.920332] RSP: 0018:ffff9939a5ed3d98 EFLAGS: 00010246
[   62.920508] RAX: 0000000000000000 RBX: 0000000000000063 RCX: 0000000000000000
[   62.920744] RDX: 0000000000000000 RSI: ffffffffba47f75b RDI: 0000000000000063
[   62.920979] RBP: ffff9939a5ed3d98 R08: 0000000000000000 R09: 0000000000000c2e
[   62.921216] R10: 0000000000000000 R11: ffffffffba932f0d R12: 0000000000000003
[   62.921457] R13: 0000000000000000 R14: ffffffffba6807e0 R15: 0000000000000000
[   62.921693] FS:  00007f3d2fb94740(0000) GS:ffff9939b4b80000(0000) knlGS:0000000000000000
[   62.921959] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[   62.922150] CR2: 0000000000000000 CR3: 000000006185d000 CR4: 00000000003406e0
[   62.922385] Call Trace:
[   62.922476]  __handle_sysrq+0xef/0x140
[   62.922605]  write_sysrq_trigger+0x4c/0x50
[   62.922746]  proc_reg_write+0x40/0x70
[   62.922874]  __vfs_write+0x35/0x180
[   62.922997]  ? __this_cpu_preempt_check+0x13/0x20
[   62.923156]  vfs_write+0xc5/0x1c0
[   62.923271]  ksys_write+0x53/0xc0
[   62.923388]  ? __audit_syscall_exit+0x20d/0x2a0
[   62.923541]  __x64_sys_write+0x15/0x20
[   62.923671]  do_syscall_64+0x4e/0x100
[   62.923800]  entry_SYSCALL_64_after_hwframe+0x44/0xa9
[   62.923970] RIP: 0033:0x7f3d2fca0664
[   62.924093] Code: 00 f7 d8 64 89 02 48 c7 c0 ff ff ff ff eb b3 0f 1f 80 00 00 00 00 48 8d 05 99 20 0e 00 8b 00 85 c0 75 13 b8 01 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 54 c3 0f 1f 00 41 54 49 89 d4 55 48 89 f5 53
[   62.924699] RSP: 002b:00007ffcf9071e98 EFLAGS: 00000246 ORIG_RAX: 0000000000000001
[   62.924949] RAX: ffffffffffffffda RBX: 0000000000000002 RCX: 00007f3d2fca0664
[   62.925184] RDX: 0000000000000002 RSI: 0000000000ee8190 RDI: 0000000000000001
[   62.925420] RBP: 0000000000ee8190 R08: 000000000000000a R09: 00007f3d2fb94740
[   62.925655] R10: 00000000004b8f41 R11: 0000000000000246 R12: 00007f3d2fd7e760
[   62.925890] R13: 0000000000000002 R14: 00007f3d2fd7a280 R15: 00007f3d2fd79740
[   62.926126] Modules linked in:
[   62.926234] CR2: 0000000000000000
[   62.926349] ---[ end trace 416678827625f99b ]---
[   62.936473] RIP: 0010:sysrq_handle_crash+0x16/0x20
[   62.936644] Code: d5 e8 8e c2 bc ff 0f 1f 40 00 66 2e 0f 1f 84 00 00 00 00 00 55 48 89 e5 e8 77 31 c3 ff c7 05 65 9e 16 01 01 00 00 00 0f ae f8 <c6> 04 25 00 00 00 00 01 5d c3 55 48 89 e5 fb 66 0f 1f 44 00 00 e8
[   62.937260] RSP: 0018:ffff9939a5ed3d98 EFLAGS: 00010246
[   62.937438] RAX: 0000000000000000 RBX: 0000000000000063 RCX: 0000000000000000
[   62.937677] RDX: 0000000000000000 RSI: ffffffffba47f75b RDI: 0000000000000063
[   62.937917] RBP: ffff9939a5ed3d98 R08: 0000000000000000 R09: 0000000000000c2e
[   62.938158] R10: 0000000000000000 R11: ffffffffba932f0d R12: 0000000000000003
[   62.938396] R13: 0000000000000000 R14: ffffffffba6807e0 R15: 0000000000000000
[   62.938636] FS:  00007f3d2fb94740(0000) GS:ffff9939b4b80000(0000) knlGS:0000000000000000
[   62.938905] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[   62.939099] CR2: 0000000000000000 CR3: 000000006185d000 CR4: 00000000003406e0
[   62.939339] Kernel panic - not syncing: Fatal exception
[   62.939941] reboot: panic mode set: p,w
[   62.940100] Kernel Offset: 0x38000000 from 0xffffffff81000000 (relocation range: 0xffffffff80000000-0xffffffffbfffffff)
[   62.950783] ---[ end Kernel panic - not syncing: Fatal exception ]---
[   62.951034] ------------[ cut here ]------------
[   62.951201] sched: Unexpected reschedule of offline CPU#1!
[   62.951409] WARNING: CPU: 3 PID: 336 at arch/x86/kernel/smp.c:141 native_smp_send_reschedule+0x3a/0x40
[   62.951732] Modules linked in:
[   62.951848] CPU: 3 PID: 336 Comm: bash Tainted: G     UD W         4.19.0-19.iot-lts2018-sos-g6b95688-dirty #44
[   62.952199] RIP: 0010:native_smp_send_reschedule+0x3a/0x40
[   62.952394] Code: c0 84 c0 74 17 48 8b 05 c4 40 54 01 be fd 00 00 00 48 8b 40 30 e8 f6 a8 dc 00 5d c3 89 fe 48 c7 c7 00 7a 45 ba e8 e6 e2 05 00 <0f> 0b 5d c3 66 90 55 48 89 e5 41 54 53 48 83 ec 20 65 48 8b 04 25
[   62.953037] RSP: 0018:ffff9939b4b83cd8 EFLAGS: 00010082
[   62.953222] RAX: 0000000000000000 RBX: 0000000000000001 RCX: 0000000000000000
[   62.953472] RDX: 0000000000000007 RSI: ffffffffba47f75b RDI: 00000000ffffffff
[   62.953721] RBP: ffff9939b4b83cd8 R08: 0000000000000000 R09: 0000000000000c65
[   62.953970] R10: 0000000000000000 R11: ffffffffba932f0d R12: 0000000000000000
[   62.954219] R13: ffff9939a46e28c8 R14: ffff9939a46e2140 R15: 0000000000020480
[   62.954469] FS:  00007f3d2fb94740(0000) GS:ffff9939b4b80000(0000) knlGS:0000000000000000
[   62.954753] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[   62.954955] CR2: 0000000000000000 CR3: 000000006185d000 CR4: 00000000003406e0
[   62.955205] Call Trace:
[   62.955299]  <IRQ>
[   62.955380]  try_to_wake_up+0x386/0x3f0
[   62.955521]  default_wake_function+0xd/0x10
[   62.955673]  autoremove_wake_function+0x11/0x60
[   62.955836]  __wake_up_common+0x82/0x150
[   62.955979]  ? __queue_work+0x125/0x330
[   62.956119]  __wake_up_common_lock+0x7b/0xc0
[   62.956273]  __wake_up+0xe/0x10
[   62.956390]  wake_up_klogd_work_func+0x47/0x70
[   62.956552]  irq_work_run_list+0x55/0x80
[   62.956694]  irq_work_tick+0x55/0x60
[   62.956825]  update_process_times+0x3d/0x50
[   62.956977]  tick_sched_handle+0x24/0x60
[   62.957120]  tick_sched_timer+0x37/0x80
[   62.957259]  ? tick_sched_do_timer+0x60/0x60
[   62.957413]  __hrtimer_run_queues+0xf3/0x190
[   62.957567]  hrtimer_interrupt+0x11d/0x250
[   62.957718]  smp_apic_timer_interrupt+0x60/0xa0
[   63.577258]  apic_timer_interrupt+0xf/0x20
[   63.579677]  </IRQ>
[   63.579759] RIP: 0010:panic+0x19b/0x1f4
[   63.579897] Code: 89 01 90 69 09 b9 48 c7 c6 80 c3 92 ba 48 c7 c7 e0 e1 45 ba c7 05 15 d7 59 01 00 00 00 00 e8 ee 56 05 00 fb 66 0f 1f 44 00 00 <45> 31 e4 45 31 f6 31 db e8 9c d6 0a 00 4c 39 f3 7c 1d 41 83 f4 01
[   63.580536] RSP: 0018:ffff9939a5ed3af0 EFLAGS: 00000286 ORIG_RAX: ffffffffffffff13
[   63.580800] RAX: 0000000000000039 RBX: 0000000000000000 RCX: 0000000000000000
[   63.581048] RDX: 0000000000000000 RSI: ffffffffba47f75b RDI: 00000000ffffffff
[   63.581296] RBP: ffff9939a5ed3b68 R08: 0000000000000000 R09: 0000000000000c63
[   63.581544] R10: 000000000000064b R11: ffffffffba932f0d R12: ffffffffba450f06
[   63.581792] R13: ffff9939a5ed3b78 R14: 0000000000000046 R15: 0000000000000000
[   63.582046]  oops_end+0x9d/0xb0
[   63.582163]  no_context+0x12d/0x320
[   63.582290]  __bad_area_nosemaphore+0x7c/0x1d0
[   63.582450]  bad_area_nosemaphore+0xf/0x20
[   63.582598]  __do_page_fault+0xa7/0x570
[   63.582737]  do_page_fault+0x22/0x30
[   63.582869]  page_fault+0x1e/0x30
[   63.582992] RIP: 0010:sysrq_handle_crash+0x16/0x20
[   63.583163] Code: d5 e8 8e c2 bc ff 0f 1f 40 00 66 2e 0f 1f 84 00 00 00 00 00 55 48 89 e5 e8 77 31 c3 ff c7 05 65 9e 16 01 01 00 00 00 0f ae f8 <c6> 04 25 00 00 00 00 01 5d c3 55 48 89 e5 fb 66 0f 1f 44 00 00 e8
[   63.583802] RSP: 0018:ffff9939a5ed3d98 EFLAGS: 00010246
[   63.583986] RAX: 0000000000000000 RBX: 0000000000000063 RCX: 0000000000000000
[   63.584234] RDX: 0000000000000000 RSI: ffffffffba47f75b RDI: 0000000000000063
[   63.584482] RBP: ffff9939a5ed3d98 R08: 0000000000000000 R09: 0000000000000c2e
[   63.584730] R10: 0000000000000000 R11: ffffffffba932f0d R12: 0000000000000003
[   63.584978] R13: 0000000000000000 R14: ffffffffba6807e0 R15: 0000000000000000
[   63.585229]  ? sysrq_handle_crash+0x9/0x20
[   63.585377]  __handle_sysrq+0xef/0x140
[   63.585513]  write_sysrq_trigger+0x4c/0x50
[   63.585662]  proc_reg_write+0x40/0x70
[   63.585795]  __vfs_write+0x35/0x180
[   63.585925]  ? __this_cpu_preempt_check+0x13/0x20
[   63.586093]  vfs_write+0xc5/0x1c0
[   63.586214]  ksys_write+0x53/0xc0
[   63.586336]  ? __audit_syscall_exit+0x20d/0x2a0
[   63.586498]  __x64_sys_write+0x15/0x20
[   63.586635]  do_syscall_64+0x4e/0x100
[   63.586768]  entry_SYSCALL_64_after_hwframe+0x44/0xa9
[   63.586948] RIP: 0033:0x7f3d2fca0664
[   63.587078] Code: 00 f7 d8 64 89 02 48 c7 c0 ff ff ff ff eb b3 0f 1f 80 00 00 00 00 48 8d 05 99 20 0e 00 8b 00 85 c0 75 13 b8 01 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 54 c3 0f 1f 00 41 54 49 89 d4 55 48 89 f5 53
[   63.587716] RSP: 002b:00007ffcf9071e98 EFLAGS: 00000246 ORIG_RAX: 0000000000000001
[   63.587980] RAX: ffffffffffffffda RBX: 0000000000000002 RCX: 00007f3d2fca0664
[   63.588228] RDX: 0000000000000002 RSI: 0000000000ee8190 RDI: 0000000000000001
[   63.588476] RBP: 0000000000ee8190 R08: 000000000000000a R09: 00007f3d2fb94740
[   63.588724] R10: 00000000004b8f41 R11: 0000000000000246 R12: 00007f3d2fd7e760
[   63.588972] R13: 0000000000000002 R14: 00007f3d2fd7a280 R15: 00007f3d2fd79740
[   63.589222] ---[ end trace 416678827625f99c ]---
[   63.589394] WARNING: CPU: 3 PID: 336 at kernel/sched/core.c:1163 set_task_cpu+0x145/0x150
[   63.589680] Modules linked in:
[   63.589793] CPU: 3 PID: 336 Comm: bash Tainted: G     UD W         4.19.0-19.iot-lts2018-sos-g6b95688-dirty #44
[   63.590143] RIP: 0010:set_task_cpu+0x145/0x150
[   63.590302] Code: 5e 5d c3 80 8b 94 04 00 00 04 e9 76 ff ff ff 0f 0b e9 00 ff ff ff f7 43 60 fd ff ff ff 0f 84 0a ff ff ff 0f 0b e9 03 ff ff ff <0f> 0b e9 12 ff ff ff 0f 1f 40 00 55 48 89 e5 41 56 49 89 f6 41 55
[   63.590943] RSP: 0018:ffff9939b4b83c10 EFLAGS: 00010046
[   63.591128] RAX: 0000000000000200 RBX: ffff9939a7d26180 RCX: ffff9939a7d26180
[   63.591377] RDX: ffff9939b4b20401 RSI: 0000000000000002 RDI: ffff9939a7d26180
[   63.591625] RBP: ffff9939b4b83c30 R08: ffff9939a7d2be78 R09: 0000000000000000
[   63.591874] R10: 0000000000000046 R11: ffffffffba932f0d R12: 0000000000000002
[   63.592122] R13: ffff9939a7d26908 R14: 0000000000000002 R15: ffff9939a7d264f0
[   63.592371] FS:  00007f3d2fb94740(0000) GS:ffff9939b4b80000(0000) knlGS:0000000000000000
[   63.592653] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[   63.592855] CR2: 0000000000000000 CR3: 000000006185d000 CR4: 00000000003406e0
[   63.593103] Call Trace:
[   63.593194]  <IRQ>
[   63.593272]  try_to_wake_up+0x13a/0x3f0
[   63.593413]  default_wake_function+0xd/0x10
[   63.593563]  __wake_up_common+0x82/0x150
[   63.593706]  __wake_up_locked+0x11/0x20
[   63.593847]  ep_poll_callback+0xcc/0x330
[   63.593989]  __wake_up_common+0x82/0x150
[   63.594131]  ? __queue_work+0x125/0x330
[   63.594270]  __wake_up_common_lock+0x7b/0xc0
[   63.594424]  __wake_up+0xe/0x10
[   63.594540]  wake_up_klogd_work_func+0x47/0x70
[   63.594699]  irq_work_run_list+0x55/0x80
[   63.594841]  irq_work_tick+0x55/0x60
[   63.594971]  update_process_times+0x3d/0x50
[   63.595122]  tick_sched_handle+0x24/0x60
[   63.595264]  tick_sched_timer+0x37/0x80
[   63.595403]  ? tick_sched_do_timer+0x60/0x60
[   63.595557]  __hrtimer_run_queues+0xf3/0x190
[   63.595711]  hrtimer_interrupt+0x11d/0x250
[   63.595859]  smp_apic_timer_interrupt+0x60/0xa0
[   63.596021]  apic_timer_interrupt+0xf/0x20
[   63.596168]  </IRQ>
[   63.596249] RIP: 0010:panic+0x19b/0x1f4
[   63.596387] Code: 89 01 90 69 09 b9 48 c7 c6 80 c3 92 ba 48 c7 c7 e0 e1 45 ba c7 05 15 d7 59 01 00 00 00 00 e8 ee 56 05 00 fb 66 0f 1f 44 00 00 <45> 31 e4 45 31 f6 31 db e8 9c d6 0a 00 4c 39 f3 7c 1d 41 83 f4 01
[   63.597027] RSP: 0018:ffff9939a5ed3af0 EFLAGS: 00000286 ORIG_RAX: ffffffffffffff13
[   63.597292] RAX: 0000000000000039 RBX: 0000000000000000 RCX: 0000000000000000
[   63.597540] RDX: 0000000000000000 RSI: ffffffffba47f75b RDI: 00000000ffffffff
[   63.597789] RBP: ffff9939a5ed3b68 R08: 0000000000000000 R09: 0000000000000c63
[   64.191896] R10: 000000000000064b R11: ffffffffba932f0d R12: ffffffffba450f06
[   64.192145] R13: ffff9939a5ed3b78 R14: 0000000000000046 R15: 0000000000000000
[   64.192398]  oops_end+0x9d/0xb0
[   64.192514]  no_context+0x12d/0x320
[   64.192642]  __bad_area_nosemaphore+0x7c/0x1d0
[   64.192802]  bad_area_nosemaphore+0xf/0x20
[   64.192949]  __do_page_fault+0xa7/0x570
[   64.193089]  do_page_fault+0x22/0x30
[   64.193220]  page_fault+0x1e/0x30
[   64.193342] RIP: 0010:sysrq_handle_crash+0x16/0x20
[   64.193513] Code: d5 e8 8e c2 bc ff 0f 1f 40 00 66 2e 0f 1f 84 00 00 00 00 00 55 48 89 e5 e8 77 31 c3 ff c7 05 65 9e 16 01 01 00 00 00 0f ae f8 <c6> 04 25 00 00 00 00 01 5d c3 55 48 89 e5 fb 66 0f 1f 44 00 00 e8
[   64.194154] RSP: 0018:ffff9939a5ed3d98 EFLAGS: 00010246
[   64.194339] RAX: 0000000000000000 RBX: 0000000000000063 RCX: 0000000000000000
[   64.194588] RDX: 0000000000000000 RSI: ffffffffba47f75b RDI: 0000000000000063
[   64.194837] RBP: ffff9939a5ed3d98 R08: 0000000000000000 R09: 0000000000000c2e
[   64.195085] R10: 0000000000000000 R11: ffffffffba932f0d R12: 0000000000000003
[   64.195334] R13: 0000000000000000 R14: ffffffffba6807e0 R15: 0000000000000000
[   64.195585]  ? sysrq_handle_crash+0x9/0x20
[   64.195734]  __handle_sysrq+0xef/0x140
[   64.195870]  write_sysrq_trigger+0x4c/0x50
[   64.196018]  proc_reg_write+0x40/0x70
[   64.196151]  __vfs_write+0x35/0x180
[   64.196279]  ? __this_cpu_preempt_check+0x13/0x20
[   64.196447]  vfs_write+0xc5/0x1c0
[   64.196569]  ksys_write+0x53/0xc0
[   64.196690]  ? __audit_syscall_exit+0x20d/0x2a0
[   64.196853]  __x64_sys_write+0x15/0x20
[   64.196988]  do_syscall_64+0x4e/0x100
[   64.197122]  entry_SYSCALL_64_after_hwframe+0x44/0xa9
[   64.197302] RIP: 0033:0x7f3d2fca0664
[   64.197431] Code: 00 f7 d8 64 89 02 48 c7 c0 ff ff ff ff eb b3 0f 1f 80 00 00 00 00 48 8d 05 99 20 0e 00 8b 00 85 c0 75 13 b8 01 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 54 c3 0f 1f 00 41 54 49 89 d4 55 48 89 f5 53
[   64.394050] RSP: 002b:00007ffcf9071e98 EFLAGS: 00000246 ORIG_RAX: 0000000000000001
[   64.394315] RAX: ffffffffffffffda RBX: 0000000000000002 RCX: 00007f3d2fca0664
[   64.394563] RDX: 0000000000000002 RSI: 0000000000ee8190 RDI: 0000000000000001
[   64.394812] RBP: 0000000000ee8190 R08: 000000000000000a R09: 00007f3d2fb94740
[   64.395061] R10: 00000000004b8f41 R11: 0000000000000246 R12: 00007f3d2fd7e760
[   64.395309] R13: 0000000000000002 R14: 00007f3d2fd7a280 R15: 00007f3d2fd79740
[   64.395559] ---[ end trace 416678827625f99d ]---
[   64.395735] ------------[ cut here ]------------
[   64.395899] sched: Unexpected reschedule of offline CPU#2!
[   64.396095] WARNING: CPU: 3 PID: 336 at arch/x86/kernel/smp.c:141 native_smp_send_reschedule+0x3a/0x40
[   64.396417] Modules linked in:
[   64.396531] CPU: 3 PID: 336 Comm: bash Tainted: G     UD W         4.19.0-19.iot-lts2018-sos-g6b95688-dirty #44
[   64.396880] RIP: 0010:native_smp_send_reschedule+0x3a/0x40
[   64.397074] Code: c0 84 c0 74 17 48 8b 05 c4 40 54 01 be fd 00 00 00 48 8b 40 30 e8 f6 a8 dc 00 5d c3 89 fe 48 c7 c7 00 7a 45 ba e8 e6 e2 05 00 <0f> 0b 5d c3 66 90 55 48 89 e5 41 54 53 48 83 ec 20 65 48 8b 04 25
[   64.397715] RSP: 0018:ffff9939b4b83ba0 EFLAGS: 00010  64.532707]  irq_work_tick+0x55/0x60
[   64.532838]  update_process_times+0x3d/0x50
[   64.532988]  tick_sched_handle+0x24/0x60
[   64.533130]  tick_sched_timer+0x37/0x80
[   64.533270]  ? tick_sched_do_timer+0x60/0x60
[   64.533423]  __hrtimer_run_queues+0xf3/0x190
[   64.533577]  hrtimer_interrupt+0x11d/0x250
[   64.533725]  smp_apic_timer_interrupt+0x60/0xa0
[   64.533887]  apic_timer_interrupt+0xf/0x20
[   64.534034]  </IRQ>
[   64.534115] RIP: 0010:panic+0x19b/0x1f4
[   64.534253] Code: 89 01 90 69 09 b9 48 c7 c6 80 c3 92 ba 48 c7 c7 e0 e1 45 ba c7 05 15 d7 59 01 00 00 00 00 e8 ee 56 05 00 fb 66 0f 1f 44 00 00 <45> 31 e4 45 31 f6 31 db e8 9c d6 0a 00 4c 39 f3 7c 1d 41 83 f4 01
[   64.534894] RSP: 0018:ffff9939a5ed3af0 EFLAGS: 00000286 ORIG_RAX: ffffffffffffff13
[   64.535158] RAX: 0000000000000039 RBX: 0000000000000000 RCX: 0000000000000000
[   64.535407] RDX: 0000000000000000 RSI: ffffffffba47f75b RDI: 00000000ffffffff
[   64.535655] RBP: ffff9939a5ed3b68 R08: 0000000000000000 R09: 0000000000000c63
[   64.535904] R10: 000000000000064b R11: ffffffffba932f0d R12: ffffffffba450f06
[   64.536153] R13: ffff9939a5ed3b78 R14: 0000000000000046 R15: 0000000000000000
[   64.536405]  oops_end+0x9d/0xb0
[   64.536521]  no_context+0x12d/0x320
[   64.536649]  __bad_area_nosemaphore+0x7c/0x1d0
[   64.536809]  bad_area_nosemaphore+0xf/0x20
[   64.536956]  __do_page_fault+0xa7/0x570
[   64.537096]  do_page_fault+0x22/0x30
[   64.537227]  page_fault+0x1e/0x30
[   64.537349] RIP: 0010:sysrq_handle_crash+0x16/0x20
[   64.537520] Code: d5 e8 8e c2 bc ff 0f 1f 40 00 66 2e 0f 1f 84 00 00 00 00 00 55 48 89 e5 e8 77 31 c3 ff c7 05 65 9e 16 01 01 00 00 00 0f ae f8 <c6> 04 25 00 00 00 00 01 5d c3 55 48 89 e5 fb 66 0f 1f 44 00 00 e8
[   64.538164] RSP: 0018:ffff9939a5ed3d98 EFLAGS: 00010246
[   64.538349] RAX: 0000000000000000 RBX: 0000000000000063 RCX: 0000000000000000
[   64.538598] RDX: 0000000000000000 RSI: ffffffffba47f75b RDI: 0000000000000063
[   64.538846] RBP: ffff9939a5ed3d98 R08: 0000000000000000 R09: 0000000000000c2e
[   64.539095] R10: 0000000000000000 R11: ffffffffba932f0d R12: 0000000000000003
[   64.539343] R13: 0000000000000000 R14: ffffffffba6807e0 R15: 0000000000000000
[   64.539594]  ? sysrq_handle_crash+0x9/0x20
[   64.539743]  __handle_sysrq+0xef/0x140
[   64.539879]  write_sysrq_trigger+0x4c/0x50
[   64.540027]  proc_reg_write+0x40/0x70
[   64.540160]  __vfs_write+0x35/0x180
[   64.540289]  ? __this_cpu_preempt_check+0x13/0x20
[   64.540457]  vfs_write+0xc5/0x1c0
[   64.540578]  ksys_write+0x53/0xc0
[   64.540700]  ? __audit_syscall_exit+0x20d/0x2a0
[   64.540862]  __x64_sys_write+0x15/0x20
[   64.540998]  do_syscall_64+0x4e/0x100
[   64.541132]  entry_SYSCALL_64_after_hwframe+0x44/0xa9
[   64.541311] RIP: 0033:0x7f3d2fca0664
[   64.541441] Code: 00 f7 d8 64 89 02 48 c7 c0 ff ff ff ff eb b3 0f 1f 80 00 00 00 00 48 8d 05 99 20 0e 00 8b 00 85 c0 75 13 b8 01 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 54 c3 0f 1f 00 41 54 49 89 d4 55 48 89 f5 53
[   64.542080] RSP: 002b:00007ffcf9071e98 EFLAGS: 00000246 ORIG_RAX: 0000000000000001
[   64.542345] RAX: ffffffffffffffda RBX: 0000000000000002 RCX: 00007f3d2fca0664
[   64.542593] RDX: 0000000000000002 RSI: 0000000000ee8190 RDI: 0000000000000001
[   64.542842] RBP: 0000000000ee8190 R08: 000000000000000a R09: 00007f3d2fb94740
[   64.543090] R10: 00000000004b8f41 R11: 0000000000000246 R12: 00007f3d2fd7e760
[   64.543339] R13: 0000000000000002 R14: 00007f3d2fd7a280 R15: 00007f3d2fd79740
[   64.543589] ---[ end trace 416678827625f99e ]---
[   64.543775] ------------[ cut here ]------------
[   64.543942] sched: Unexpected reschedule of offline CPU#0!
[   64.544138] WARNING: CPU: 3 PID: 336 at arch/x86/kernel/smp.c:141 native_smp_send_reschedule+0x3a/0x40
[   64.544460] Modules linked in:
[   64.544573] CPU: 3 PID: 336 Comm: bash Tainted: G     UD W         4.19.0-19.iot-lts2018-sos-g6b95688-dirty #44
[   64.544923] RIP: 0010:native_smp_send_reschedule+0x3a/0x40
[   64.545117] Code: c0 84 c0 74 17 48 8b 05 c4 40 54 01 be fd 00 00 00 48 8b 40 30 e8 f6 a8 dc 00 5d c3 89 fe 48 c7 c7 00 7a 45 ba e8 e6 e2 05 00 <0f> 0b 5d c3 66 90 55 48 89 e5 41 54 53 48 83 ec 20 65 48 8b 04 25
[   64.545755] RSP: 0018:ffff9939b4b83e28 EFLAGS: 00010086
[   64.545939] RAX: 0000000000000000 RBX: 0000000000000003 RCX: 0000000000000000
[   64.546187] RDX: 0000000000000007 RSI: ffffffffba47f75b RDI: 00000000ffffffff
[   64.546434] RBP: ffff9939b4b83e28 R08: 0000000000000000 R09: 0000000000000d5e
[   64.546682] R10: 0000000000000400 R11: ffffffffba932f0d R12: 0000000000000000
[   64.546930] R13: 0000000000000003 R14: ffff9939a3650280 R15: ffff9939b4b9be00
[   64.547178] FS:  00007f3d2fb94740(0000) GS:ffff9939b4b80000(0000) knlGS:0000000000000000
[   64.547458] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[   64.547659] CR2: 0000000000000000 CR3: 000000006185d000 CR4: 00000000003406e0
[   64.547906] Call Trace:
[   64.547997]  <IRQ>
[   64.548074]  kick_ilb+0x8b/0xa0
[   64.548190]  trigger_load_balance+0xb1/0x1a0
[   64.548344]  scheduler_tick+0xa4/0xd0
[   64.548477]  update_process_times+0x42/0x50
[   64.548627]  tick_sched_handle+0x24/0x60
[   64.548769]  tick_sched_timer+0x37/0x80
[   64.548908]  ? tick_sched_do_timer+0x60/0x60
[   64.549060]  __hrtimer_run_queues+0xf3/0x190
[   64.549214]  hrtimer_interrupt+0x11d/0x250
[   64.549362]  smp_apic_timer_interrupt+0x60/0xa0
[   64.549523]  apic_timer_interrupt+0xf/0x20
[   64.549670]  </IRQ>
[   64.549750] RIP: 0010:panic+0x19b/0x1f4
[   64.549888] Code: 89 01 90 69 09 b9 48 c7 c6 80 c3 92 ba 48 c7 c7 e0 e1 45 ba c7 05 15 d7 59 01 00 00 00 00 e8 ee 56 05 00 fb 66 0f 1f 44 00 00 <45> 31 e4 45 31 f6 31 db e8 9c d6 0a 00 4c 39 f3 7c 1d 41 83 f4 01
[   64.550525] RSP: 0018:ffff9939a5ed3af0 EFLAGS: 00000286 ORIG_RAX: ffffffffffffff13
[   64.550789] RAX: 0000000000000039 RBX: 0000000000000000 RCX: 0000000000000000
[   64.551036] RDX: 0000000000000000 RSI: ffffffffba47f75b RDI: 00000000ffffffff
[   64.551284] RBP: ffff9939a5ed3b68 R08: 0000000000000000 R09: 0000000000000c63
[   64.551532] R10: 000000000000064b R11: ffffffffba932f0d R12: ffffffffba450f06
[   64.551779] R13: ffff9939a5ed3b78 R14: 0000000000000046 R15: 0000000000000000
[   64.552031]  oops_end+0x9d/0xb0
[   64.552146]  no_context+0x12d/0x320
[   64.552274]  __bad_area_nosemaphore+0x7c/0x1d0
[   64.552433]  bad_area_nosemaphore+0xf/0x20
[   64.552580]  __do_page_fault+0xa7/0x570
[   64.552720]  do_page_fault+0x22/0x30
[   64.552850]  page_fault+0x1e/0x30
[   64.552972] RIP: 0010:sysrq_handle_crash+0x16/0x20
[   64.553141] Code: d5 e8 8e c2 bc ff 0f 1f 40 00 66 2e 0f 1f 84 00 00 00 00 00 55 48 89 e5 e8 77 31 c3 ff c7 05 65 9e 16 01 01 00 00 00 0f ae f8 <c6> 04 25 00 00 00 00 01 5d c3 55 48 89 e5 fb 66 0f 1f 44 00 00 e8
[   64.553779] RSP: 0018:ffff9939a5ed3d98 EFLAGS: 00010246
[   64.553964] RAX: 0000000000000000 RBX: 0000000000000063 RCX: 0000000000000000
[   64.554212] RDX: 0000000000000000 RSI: ffffffffba47f75b RDI: 0000000000000063
[   64.554460] RBP: ffff9939a5ed3d98 R08: 0000000000000000 R09: 0000000000000c2e
[   64.554709] R10: 0000000000000000 R11: ffffffffba932f0d R12: 0000000000000003
[   64.554957] R13: 0000000000000000 R14: ffffffffba6807e0 R15: 0000000000000000
[   64.555207]  ? sysrq_handle_crash+0x9/0x20
[   64.555355]  __handle_sysrq+0xef/0x140
[   64.555491]  write_sysrq_trigger+0x4c/0x50
[   64.555638]  proc_reg_write+0x40/0x70
[   64.555771]  __vfs_write+0x35/0x180
[   64.555898]  ? __this_cpu_preempt_check+0x13/0x20
[   64.556066]  vfs_write+0xc5/0x1c0
[   64.556187]  ksys_write+0x53/0xc0
[   64.556308]  ? __audit_syscall_exit+0x20d/0x2a0
[   64.556470]  __x64_sys_write+0x15/0x20
[   64.556606]  do_syscall_64+0x4e/0x100
[   64.556739]  entry_SYSCALL_64_after_hwframe+0x44/0xa9
[   64.556918] RIP: 0033:0x7f3d2fca0664
[   64.557047] Code: 00 f7 d8 64 89 02 48 c7 c0 ff ff ff ff eb b3 0f 1f 80 00 00 00 00 48 8d 05 99 20 0e 00 8b 00 85 c0 75 13 b8 01 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 54 c3 0f 1f 00 41 54 49 89 d4 55 48 89 f5 53
[   64.557685] RSP: 002b:00007ffcf9071e98 EFLAGS: 00000246 ORIG_RAX: 0000000000000001
[   65.374953] RAX: ffffffffffffffda RBX: 0000000000000002 RCX: 00007f3d2fca0664
[   65.376235] RDX: 0000000000000002 RSI: 0000000000ee8190 RDI: 0000000000000001
[   65.376482] RBP: 0000000000ee8190 R08: 000000000000000a R09: 00007f3d2fb94740
[   65.376730] R10: 00000000004b8f41 R11: 0000000000000246 R12: 00007f3d2fd7e760
[   65.376978] R13: 0000000000000002 R14: 00007f3d2fd7a280 R15: 00007f3d2fd79740
[   65.377228] ---[ end trace 416678827625f99f ]---