* [PATCH RFC] panic: Avoid extra noisy messages due to stopped cpus
@ 2018-10-11 7:17 Feng Tang
2018-10-11 9:35 ` Peter Zijlstra
0 siblings, 1 reply; 7+ messages in thread
From: Feng Tang @ 2018-10-11 7:17 UTC (permalink / raw)
To: Thomas Gleixner, Ingo Molnar, H Peter Anvin, Borislav Petkov,
Peter Zijlstra, Andrew Morton, linux-kernel
Cc: Feng Tang
Sometimes when debugging kernel panic, we saw many extra noisy error
messages after the expected end:
[ 35.743249] ---[ end Kernel panic - not syncing: Fatal exception
[ 35.749975] ------------[ cut here ]------------
These messages may overflow the sceen (framebuffer) and make debugging
much difficulter.
This hack patch just quickly prevent these noisy message, and would
really like to get some comments and suggestions.
I have tried other ways like adding a panic notifier block inside
tick/sched code to cancel tick_sched timer in panic case, which
also works.
These extra messages are of 2 kinds:
a)
WARNING: CPU: 1 PID: 280 at kernel/sched/core.c:1198 set_task_cpu+0x183/0x190
Call Trace:
<IRQ>
try_to_wake_up+0x157/0x430
default_wake_function+0xd/0x10
autoremove_wake_function+0x11/0x60
__wake_up_common+0x8a/0x160
__wake_up_common_lock+0x6c/0x90
__wake_up+0xe/0x10
wake_up_klogd_work_func+0x3b/0x60
irq_work_run_list+0x4e/0x80
irq_work_tick+0x40/0x50
update_process_times+0x3d/0x50
tick_sched_timer+0x38/0x80
__hrtimer_run_queues+0xce/0x200
hrtimer_interrupt+0xac/0x1f0
smp_apic_timer_interrupt+0x6e/0x140
apic_timer_interrupt+0x8e/0xa0
b)
sched: Unexpected reschedule of offline CPU#0!
------------[ cut here ]------------
WARNING: CPU: 1 PID: 300 at arch/x86/kernel/smp.c:141 native_smp_send_reschedule+0x3d/0x50
trigger_load_balance+0x125/0x230
scheduler_tick+0xa2/0xd0
update_process_times+0x42/0x50
tick_sched_handle.isra.5+0x21/0x60
tick_sched_timer+0x38/0x80
__hrtimer_run_queues+0xce/0x200
hrtimer_interrupt+0xac/0x1f0
smp_apic_timer_interrupt+0x6e/0x140
apic_timer_interrupt+0x8e/0xa0
Signed-off-by: Feng Tang <feng.tang@intel.com>
---
arch/x86/kernel/process.c | 1 +
kernel/sched/fair.c | 2 +-
2 files changed, 2 insertions(+), 1 deletion(-)
diff --git a/arch/x86/kernel/process.c b/arch/x86/kernel/process.c
index c93fcfd..b703862 100644
--- a/arch/x86/kernel/process.c
+++ b/arch/x86/kernel/process.c
@@ -520,6 +520,7 @@ void stop_this_cpu(void *dummy)
* Remove this CPU:
*/
set_cpu_online(smp_processor_id(), false);
+ set_cpu_active(smp_processor_id(), false);
disable_local_APIC();
mcheck_cpu_clear(this_cpu_ptr(&cpu_info));
diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c
index 7fc4a37..cf41b7b 100644
--- a/kernel/sched/fair.c
+++ b/kernel/sched/fair.c
@@ -9034,7 +9034,7 @@ static inline int find_new_ilb(void)
{
int ilb = cpumask_first(nohz.idle_cpus_mask);
- if (ilb < nr_cpu_ids && idle_cpu(ilb))
+ if (ilb < nr_cpu_ids && idle_cpu(ilb) && cpu_online(ilb))
return ilb;
return nr_cpu_ids;
--
2.7.4
^ permalink raw reply related [flat|nested] 7+ messages in thread
* Re: [PATCH RFC] panic: Avoid extra noisy messages due to stopped cpus
2018-10-11 7:17 [PATCH RFC] panic: Avoid extra noisy messages due to stopped cpus Feng Tang
@ 2018-10-11 9:35 ` Peter Zijlstra
2018-10-11 9:59 ` Feng Tang
0 siblings, 1 reply; 7+ messages in thread
From: Peter Zijlstra @ 2018-10-11 9:35 UTC (permalink / raw)
To: Feng Tang
Cc: Thomas Gleixner, Ingo Molnar, H Peter Anvin, Borislav Petkov,
Andrew Morton, linux-kernel
On Thu, Oct 11, 2018 at 03:17:48PM +0800, Feng Tang wrote:
> Sometimes when debugging kernel panic, we saw many extra noisy error
> messages after the expected end:
>
> [ 35.743249] ---[ end Kernel panic - not syncing: Fatal exception
> [ 35.749975] ------------[ cut here ]------------
>
> These messages may overflow the sceen (framebuffer) and make debugging
> much difficulter.
*blink* you actually using the framebuffer for debugging ?! Why the heck
are you doing that?
> diff --git a/arch/x86/kernel/process.c b/arch/x86/kernel/process.c
> index c93fcfd..b703862 100644
> --- a/arch/x86/kernel/process.c
> +++ b/arch/x86/kernel/process.c
> @@ -520,6 +520,7 @@ void stop_this_cpu(void *dummy)
> * Remove this CPU:
> */
> set_cpu_online(smp_processor_id(), false);
> + set_cpu_active(smp_processor_id(), false);
> disable_local_APIC();
> mcheck_cpu_clear(this_cpu_ptr(&cpu_info));
>
WTH is stop_this_cpu() and how do we even get here with active still
set?
> diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c
> index 7fc4a37..cf41b7b 100644
> --- a/kernel/sched/fair.c
> +++ b/kernel/sched/fair.c
> @@ -9034,7 +9034,7 @@ static inline int find_new_ilb(void)
> {
> int ilb = cpumask_first(nohz.idle_cpus_mask);
>
> - if (ilb < nr_cpu_ids && idle_cpu(ilb))
> + if (ilb < nr_cpu_ids && idle_cpu(ilb) && cpu_online(ilb))
> return ilb;
>
> return nr_cpu_ids;
Similar, this is the result of taking the CPU away without going through
the normal path. You're doing something dodgy.
^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: [PATCH RFC] panic: Avoid extra noisy messages due to stopped cpus
2018-10-11 9:35 ` Peter Zijlstra
@ 2018-10-11 9:59 ` Feng Tang
2018-10-22 9:55 ` Feng Tang
0 siblings, 1 reply; 7+ messages in thread
From: Feng Tang @ 2018-10-11 9:59 UTC (permalink / raw)
To: Peter Zijlstra
Cc: Thomas Gleixner, Ingo Molnar, H Peter Anvin, Borislav Petkov,
Andrew Morton, linux-kernel
Hi Peter,
On Thu, Oct 11, 2018 at 11:35:00AM +0200, Peter Zijlstra wrote:
> On Thu, Oct 11, 2018 at 03:17:48PM +0800, Feng Tang wrote:
> > Sometimes when debugging kernel panic, we saw many extra noisy error
> > messages after the expected end:
> >
> > [ 35.743249] ---[ end Kernel panic - not syncing: Fatal exception
> > [ 35.749975] ------------[ cut here ]------------
> >
> > These messages may overflow the sceen (framebuffer) and make debugging
> > much difficulter.
>
> *blink* you actually using the framebuffer for debugging ?! Why the heck
> are you doing that?
I didn't make it clear.
I just wanted to show some use case, that when I work under command line
mode, if the kernel panic (without reboot configured), it will pop out
the panic message, and these noisy message may keep poping up and I
hardly see the actual panic message on the screen.
>
> > diff --git a/arch/x86/kernel/process.c b/arch/x86/kernel/process.c
> > index c93fcfd..b703862 100644
> > --- a/arch/x86/kernel/process.c
> > +++ b/arch/x86/kernel/process.c
> > @@ -520,6 +520,7 @@ void stop_this_cpu(void *dummy)
> > * Remove this CPU:
> > */
> > set_cpu_online(smp_processor_id(), false);
> > + set_cpu_active(smp_processor_id(), false);
> > disable_local_APIC();
> > mcheck_cpu_clear(this_cpu_ptr(&cpu_info));
> >
>
> WTH is stop_this_cpu() and how do we even get here with active still
> set?
Current panic() will call
smp_send_stop
native_stop_other_cpus
apic->send_IPI_allbutself(REBOOT_VECTOR)
smp_reboot_interrupt
stop_this_cpu
> > diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c
> > index 7fc4a37..cf41b7b 100644
> > --- a/kernel/sched/fair.c
> > +++ b/kernel/sched/fair.c
> > @@ -9034,7 +9034,7 @@ static inline int find_new_ilb(void)
> > {
> > int ilb = cpumask_first(nohz.idle_cpus_mask);
> >
> > - if (ilb < nr_cpu_ids && idle_cpu(ilb))
> > + if (ilb < nr_cpu_ids && idle_cpu(ilb) && cpu_online(ilb))
> > return ilb;
> >
> > return nr_cpu_ids;
>
>
> Similar, this is the result of taking the CPU away without going through
> the normal path. You're doing something dodgy.
You are right, it's hacky. As I said in the commit log, suggestions are
appreciated.
Current emergercy handling of panic didn't notify the idle subsystem of
other cpus being stopped.
I have tried several other ways:
a) add a panic notifier to cancel the schdule timer when panic happens
b) don't re-enable the local interrupt inside panic() after normal panic
handling.
Thanks,
Feng
^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: [PATCH RFC] panic: Avoid extra noisy messages due to stopped cpus
2018-10-11 9:59 ` Feng Tang
@ 2018-10-22 9:55 ` Feng Tang
2018-11-08 13:05 ` [PATCH v2] panic: Avoid the extra noise dmesg Feng Tang
0 siblings, 1 reply; 7+ messages in thread
From: Feng Tang @ 2018-10-22 9:55 UTC (permalink / raw)
To: Peter Zijlstra
Cc: Thomas Gleixner, Ingo Molnar, H Peter Anvin, Borislav Petkov,
Andrew Morton, linux-kernel
Hi maintainers,
On Thu, Oct 11, 2018 at 05:59:41PM +0800, Feng Tang wrote:
> Hi Peter,
>
> On Thu, Oct 11, 2018 at 11:35:00AM +0200, Peter Zijlstra wrote:
> > On Thu, Oct 11, 2018 at 03:17:48PM +0800, Feng Tang wrote:
> > > Sometimes when debugging kernel panic, we saw many extra noisy error
> > > messages after the expected end:
> > >
> > > [ 35.743249] ---[ end Kernel panic - not syncing: Fatal exception
> > > [ 35.749975] ------------[ cut here ]------------
> > >
> > > These messages may overflow the sceen (framebuffer) and make debugging
> > > much difficulter.
> >
> > *blink* you actually using the framebuffer for debugging ?! Why the heck
> > are you doing that?
>
> I didn't make it clear.
>
> I just wanted to show some use case, that when I work under command line
> mode, if the kernel panic (without reboot configured), it will pop out
> the panic message, and these noisy message may keep poping up and I
> hardly see the actual panic message on the screen.
>
> >
> > > diff --git a/arch/x86/kernel/process.c b/arch/x86/kernel/process.c
> > > index c93fcfd..b703862 100644
> > > --- a/arch/x86/kernel/process.c
> > > +++ b/arch/x86/kernel/process.c
> > > @@ -520,6 +520,7 @@ void stop_this_cpu(void *dummy)
> > > * Remove this CPU:
> > > */
> > > set_cpu_online(smp_processor_id(), false);
> > > + set_cpu_active(smp_processor_id(), false);
> > > disable_local_APIC();
> > > mcheck_cpu_clear(this_cpu_ptr(&cpu_info));
> > >
> >
> > WTH is stop_this_cpu() and how do we even get here with active still
> > set?
>
> Current panic() will call
> smp_send_stop
> native_stop_other_cpus
> apic->send_IPI_allbutself(REBOOT_VECTOR)
> smp_reboot_interrupt
> stop_this_cpu
>
> > > diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c
> > > index 7fc4a37..cf41b7b 100644
> > > --- a/kernel/sched/fair.c
> > > +++ b/kernel/sched/fair.c
> > > @@ -9034,7 +9034,7 @@ static inline int find_new_ilb(void)
> > > {
> > > int ilb = cpumask_first(nohz.idle_cpus_mask);
> > >
> > > - if (ilb < nr_cpu_ids && idle_cpu(ilb))
> > > + if (ilb < nr_cpu_ids && idle_cpu(ilb) && cpu_online(ilb))
> > > return ilb;
> > >
> > > return nr_cpu_ids;
> >
> >
> > Similar, this is the result of taking the CPU away without going through
> > the normal path. You're doing something dodgy.
>
> You are right, it's hacky. As I said in the commit log, suggestions are
> appreciated.
>
> Current emergercy handling of panic didn't notify the idle subsystem of
> other cpus being stopped.
>
> I have tried several other ways:
> a) add a panic notifier to cancel the schdule timer when panic happens
> b) don't re-enable the local interrupt inside panic() after normal panic
> handling.
Could you help to review the patch? thanks,
- Feng
^ permalink raw reply [flat|nested] 7+ messages in thread
* [PATCH v2] panic: Avoid the extra noise dmesg
2018-10-22 9:55 ` Feng Tang
@ 2018-11-08 13:05 ` Feng Tang
2018-11-26 23:59 ` Andrew Morton
0 siblings, 1 reply; 7+ messages in thread
From: Feng Tang @ 2018-11-08 13:05 UTC (permalink / raw)
To: Andrew Morton, Peter Zijlstra, Thomas Gleixner
Cc: Ingo Molnar, H Peter Anvin, Borislav Petkov, Kees Cook, linux-kernel
Sometimes when debugging kernel panic, we saw many extra noisy error
messages after the expected end:
[ 35.743249] ---[ end Kernel panic - not syncing: Fatal exception
[ 35.749975] ------------[ cut here ]------------
When panic happens under console mode, the screen will first show the panic
call stack, but will immediately be overrided by noisy extra messages, like
this for every CPU except the panic one:
WARNING: CPU: 1 PID: 280 at kernel/sched/core.c:1198 set_task_cpu+0x183/0x190
Call Trace:
<IRQ>
try_to_wake_up
default_wake_function
autoremove_wake_function
__wake_up_common
__wake_up_common_lock
__wake_up
wake_up_klogd_work_func
irq_work_run_list
irq_work_tick
update_process_times
tick_sched_timer
__hrtimer_run_queues
hrtimer_interrupt
smp_apic_timer_interrupt
apic_timer_interrupt
Which make debugging more difficult, as the original context is
lost on screen.
Keeping the CPU IRQ disabled will void these messages, as we code
run to this point, the user has chosed not to reboot, nor do
some extra handling with panic notifier, not much point in
re-enabling the interrupt.
Signed-off-by: Feng Tang <feng.tang@intel.com>
Cc: Thomas Gleixner <tglx@linutronix.de>
Cc: Kees Cook <keescook@chromium.org>
Cc: Borislav Petkov <bp@suse.de>
---
kernel/panic.c | 1 -
1 file changed, 1 deletion(-)
diff --git a/kernel/panic.c b/kernel/panic.c
index f6d549a29a5c..a616e55ab57e 100644
--- a/kernel/panic.c
+++ b/kernel/panic.c
@@ -295,7 +295,6 @@ void panic(const char *fmt, ...)
}
#endif
pr_emerg("---[ end Kernel panic - not syncing: %s ]---\n", buf);
- local_irq_enable();
for (i = 0; ; i += PANIC_TIMER_STEP) {
touch_softlockup_watchdog();
if (i >= i_next) {
--
2.14.1
^ permalink raw reply related [flat|nested] 7+ messages in thread
* Re: [PATCH v2] panic: Avoid the extra noise dmesg
2018-11-08 13:05 ` [PATCH v2] panic: Avoid the extra noise dmesg Feng Tang
@ 2018-11-26 23:59 ` Andrew Morton
2018-11-27 3:32 ` Feng Tang
0 siblings, 1 reply; 7+ messages in thread
From: Andrew Morton @ 2018-11-26 23:59 UTC (permalink / raw)
To: Feng Tang
Cc: Peter Zijlstra, Thomas Gleixner, Ingo Molnar, H Peter Anvin,
Borislav Petkov, Kees Cook, linux-kernel
On Thu, 8 Nov 2018 21:05:03 +0800 Feng Tang <feng.tang@intel.com> wrote:
> Sometimes when debugging kernel panic, we saw many extra noisy error
> messages after the expected end:
>
> [ 35.743249] ---[ end Kernel panic - not syncing: Fatal exception
> [ 35.749975] ------------[ cut here ]------------
What are these "noisy messages"? The above looks OK?
> When panic happens under console mode, the screen will first show the panic
> call stack, but will immediately be overrided by noisy extra messages, like
> this for every CPU except the panic one:
>
> WARNING: CPU: 1 PID: 280 at kernel/sched/core.c:1198 set_task_cpu+0x183/0x190
> Call Trace:
> <IRQ>
> try_to_wake_up
> default_wake_function
> autoremove_wake_function
> __wake_up_common
> __wake_up_common_lock
> __wake_up
> wake_up_klogd_work_func
> irq_work_run_list
> irq_work_tick
> update_process_times
> tick_sched_timer
> __hrtimer_run_queues
> hrtimer_interrupt
> smp_apic_timer_interrupt
> apic_timer_interrupt
And this output looks OK as well.
> Which make debugging more difficult, as the original context is
> lost on screen.
>
> Keeping the CPU IRQ disabled will void these messages, as we code
> run to this point, the user has chosed not to reboot, nor do
> some extra handling with panic notifier, not much point in
> re-enabling the interrupt.
And please have another attempt at the above paragraph?
> --- a/kernel/panic.c
> +++ b/kernel/panic.c
> @@ -295,7 +295,6 @@ void panic(const char *fmt, ...)
> }
> #endif
> pr_emerg("---[ end Kernel panic - not syncing: %s ]---\n", buf);
> - local_irq_enable();
> for (i = 0; ; i += PANIC_TIMER_STEP) {
> touch_softlockup_watchdog();
> if (i >= i_next) {
The patch looks reasonable, but the description is just too hard for me
to follow, sorry. Please spend a bit more time over that and resend?
^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: [PATCH v2] panic: Avoid the extra noise dmesg
2018-11-26 23:59 ` Andrew Morton
@ 2018-11-27 3:32 ` Feng Tang
0 siblings, 0 replies; 7+ messages in thread
From: Feng Tang @ 2018-11-27 3:32 UTC (permalink / raw)
To: Andrew Morton
Cc: Peter Zijlstra, Thomas Gleixner, Ingo Molnar, H Peter Anvin,
Borislav Petkov, Kees Cook, linux-kernel
Hi Andrew,
On Mon, Nov 26, 2018 at 03:59:52PM -0800, Andrew Morton wrote:
> On Thu, 8 Nov 2018 21:05:03 +0800 Feng Tang <feng.tang@intel.com> wrote:
>
> > Sometimes when debugging kernel panic, we saw many extra noisy error
> > messages after the expected end:
> >
> > [ 35.743249] ---[ end Kernel panic - not syncing: Fatal exception
> > [ 35.749975] ------------[ cut here ]------------
>
> What are these "noisy messages"? The above looks OK?
Yes, the above is the sane messages :) and contains the necessary context
info for debug.
The noisy messages are those following messages, I tried to capture one with
sysrq and paste it in the tail of this mail as it's big (for a 4 cores
machine)
> > When panic happens under console mode, the screen will first show the panic
> > call stack, but will immediately be overrided by noisy extra messages, like
> > this for every CPU except the panic one:
> >
> > WARNING: CPU: 1 PID: 280 at kernel/sched/core.c:1198 set_task_cpu+0x183/0x190
> > Call Trace:
> > <IRQ>
> > try_to_wake_up
> > default_wake_function
> > autoremove_wake_function
> > __wake_up_common
> > __wake_up_common_lock
> > __wake_up
> > wake_up_klogd_work_func
> > irq_work_run_list
> > irq_work_tick
> > update_process_times
> > tick_sched_timer
> > __hrtimer_run_queues
> > hrtimer_interrupt
> > smp_apic_timer_interrupt
> > apic_timer_interrupt
>
> And this output looks OK as well.
Yes, this is fine also. What I mean is it will be a little noisy if each and
every CPU will dump similar info, which will make it hard to find the real panic
context if a user has no serial console but a LCD.
> > Which make debugging more difficult, as the original context is
> > lost on screen.
> >
> > Keeping the CPU IRQ disabled will void these messages, as we code
> > run to this point, the user has chosed not to reboot, nor do
> > some extra handling with panic notifier, not much point in
> > re-enabling the interrupt.
>
> And please have another attempt at the above paragraph?
Sorry for the unclear info, will try to sanitize it.
> > --- a/kernel/panic.c
> > +++ b/kernel/panic.c
> > @@ -295,7 +295,6 @@ void panic(const char *fmt, ...)
> > }
> > #endif
> > pr_emerg("---[ end Kernel panic - not syncing: %s ]---\n", buf);
> > - local_irq_enable();
> > for (i = 0; ; i += PANIC_TIMER_STEP) {
> > touch_softlockup_watchdog();
> > if (i >= i_next) {
>
> The patch looks reasonable, but the description is just too hard for me
> to follow, sorry. Please spend a bit more time over that and resend?
Sure and thanks for your review!
- Feng
-------
Panic log:
#echo c > /proc/sysrq-trigger
[ 62.918474] sysrq: SysRq : Trigger a crash
[ 62.918643] BUG: unable to handle kernel NULL pointer dereference at 0000000000000000
[ 62.918906] PGD 67e36067 P4D 67e36067 PUD 6189f067 PMD 0
[ 62.919092] Oops: 0002 [#1] PREEMPT SMP
[ 62.919226] CPU: 3 PID: 336 Comm: bash Tainted: G U W 4.19.0-19.iot-lts2018-sos-g6b95688-dirty #44
[ 62.919563] RIP: 0010:sysrq_handle_crash+0x16/0x20
[ 62.919726] Code: d5 e8 8e c2 bc ff 0f 1f 40 00 66 2e 0f 1f 84 00 00 00 00 00 55 48 89 e5 e8 77 31 c3 ff c7 05 65 9e 16 01 01 00 00 00 0f ae f8 <c6> 04 25 00 00 00 00 01 5d c3 55 48 89 e5 fb 66 0f 1f 44 00 00 e8
[ 62.920332] RSP: 0018:ffff9939a5ed3d98 EFLAGS: 00010246
[ 62.920508] RAX: 0000000000000000 RBX: 0000000000000063 RCX: 0000000000000000
[ 62.920744] RDX: 0000000000000000 RSI: ffffffffba47f75b RDI: 0000000000000063
[ 62.920979] RBP: ffff9939a5ed3d98 R08: 0000000000000000 R09: 0000000000000c2e
[ 62.921216] R10: 0000000000000000 R11: ffffffffba932f0d R12: 0000000000000003
[ 62.921457] R13: 0000000000000000 R14: ffffffffba6807e0 R15: 0000000000000000
[ 62.921693] FS: 00007f3d2fb94740(0000) GS:ffff9939b4b80000(0000) knlGS:0000000000000000
[ 62.921959] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 62.922150] CR2: 0000000000000000 CR3: 000000006185d000 CR4: 00000000003406e0
[ 62.922385] Call Trace:
[ 62.922476] __handle_sysrq+0xef/0x140
[ 62.922605] write_sysrq_trigger+0x4c/0x50
[ 62.922746] proc_reg_write+0x40/0x70
[ 62.922874] __vfs_write+0x35/0x180
[ 62.922997] ? __this_cpu_preempt_check+0x13/0x20
[ 62.923156] vfs_write+0xc5/0x1c0
[ 62.923271] ksys_write+0x53/0xc0
[ 62.923388] ? __audit_syscall_exit+0x20d/0x2a0
[ 62.923541] __x64_sys_write+0x15/0x20
[ 62.923671] do_syscall_64+0x4e/0x100
[ 62.923800] entry_SYSCALL_64_after_hwframe+0x44/0xa9
[ 62.923970] RIP: 0033:0x7f3d2fca0664
[ 62.924093] Code: 00 f7 d8 64 89 02 48 c7 c0 ff ff ff ff eb b3 0f 1f 80 00 00 00 00 48 8d 05 99 20 0e 00 8b 00 85 c0 75 13 b8 01 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 54 c3 0f 1f 00 41 54 49 89 d4 55 48 89 f5 53
[ 62.924699] RSP: 002b:00007ffcf9071e98 EFLAGS: 00000246 ORIG_RAX: 0000000000000001
[ 62.924949] RAX: ffffffffffffffda RBX: 0000000000000002 RCX: 00007f3d2fca0664
[ 62.925184] RDX: 0000000000000002 RSI: 0000000000ee8190 RDI: 0000000000000001
[ 62.925420] RBP: 0000000000ee8190 R08: 000000000000000a R09: 00007f3d2fb94740
[ 62.925655] R10: 00000000004b8f41 R11: 0000000000000246 R12: 00007f3d2fd7e760
[ 62.925890] R13: 0000000000000002 R14: 00007f3d2fd7a280 R15: 00007f3d2fd79740
[ 62.926126] Modules linked in:
[ 62.926234] CR2: 0000000000000000
[ 62.926349] ---[ end trace 416678827625f99b ]---
[ 62.936473] RIP: 0010:sysrq_handle_crash+0x16/0x20
[ 62.936644] Code: d5 e8 8e c2 bc ff 0f 1f 40 00 66 2e 0f 1f 84 00 00 00 00 00 55 48 89 e5 e8 77 31 c3 ff c7 05 65 9e 16 01 01 00 00 00 0f ae f8 <c6> 04 25 00 00 00 00 01 5d c3 55 48 89 e5 fb 66 0f 1f 44 00 00 e8
[ 62.937260] RSP: 0018:ffff9939a5ed3d98 EFLAGS: 00010246
[ 62.937438] RAX: 0000000000000000 RBX: 0000000000000063 RCX: 0000000000000000
[ 62.937677] RDX: 0000000000000000 RSI: ffffffffba47f75b RDI: 0000000000000063
[ 62.937917] RBP: ffff9939a5ed3d98 R08: 0000000000000000 R09: 0000000000000c2e
[ 62.938158] R10: 0000000000000000 R11: ffffffffba932f0d R12: 0000000000000003
[ 62.938396] R13: 0000000000000000 R14: ffffffffba6807e0 R15: 0000000000000000
[ 62.938636] FS: 00007f3d2fb94740(0000) GS:ffff9939b4b80000(0000) knlGS:0000000000000000
[ 62.938905] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 62.939099] CR2: 0000000000000000 CR3: 000000006185d000 CR4: 00000000003406e0
[ 62.939339] Kernel panic - not syncing: Fatal exception
[ 62.939941] reboot: panic mode set: p,w
[ 62.940100] Kernel Offset: 0x38000000 from 0xffffffff81000000 (relocation range: 0xffffffff80000000-0xffffffffbfffffff)
[ 62.950783] ---[ end Kernel panic - not syncing: Fatal exception ]---
[ 62.951034] ------------[ cut here ]------------
[ 62.951201] sched: Unexpected reschedule of offline CPU#1!
[ 62.951409] WARNING: CPU: 3 PID: 336 at arch/x86/kernel/smp.c:141 native_smp_send_reschedule+0x3a/0x40
[ 62.951732] Modules linked in:
[ 62.951848] CPU: 3 PID: 336 Comm: bash Tainted: G UD W 4.19.0-19.iot-lts2018-sos-g6b95688-dirty #44
[ 62.952199] RIP: 0010:native_smp_send_reschedule+0x3a/0x40
[ 62.952394] Code: c0 84 c0 74 17 48 8b 05 c4 40 54 01 be fd 00 00 00 48 8b 40 30 e8 f6 a8 dc 00 5d c3 89 fe 48 c7 c7 00 7a 45 ba e8 e6 e2 05 00 <0f> 0b 5d c3 66 90 55 48 89 e5 41 54 53 48 83 ec 20 65 48 8b 04 25
[ 62.953037] RSP: 0018:ffff9939b4b83cd8 EFLAGS: 00010082
[ 62.953222] RAX: 0000000000000000 RBX: 0000000000000001 RCX: 0000000000000000
[ 62.953472] RDX: 0000000000000007 RSI: ffffffffba47f75b RDI: 00000000ffffffff
[ 62.953721] RBP: ffff9939b4b83cd8 R08: 0000000000000000 R09: 0000000000000c65
[ 62.953970] R10: 0000000000000000 R11: ffffffffba932f0d R12: 0000000000000000
[ 62.954219] R13: ffff9939a46e28c8 R14: ffff9939a46e2140 R15: 0000000000020480
[ 62.954469] FS: 00007f3d2fb94740(0000) GS:ffff9939b4b80000(0000) knlGS:0000000000000000
[ 62.954753] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 62.954955] CR2: 0000000000000000 CR3: 000000006185d000 CR4: 00000000003406e0
[ 62.955205] Call Trace:
[ 62.955299] <IRQ>
[ 62.955380] try_to_wake_up+0x386/0x3f0
[ 62.955521] default_wake_function+0xd/0x10
[ 62.955673] autoremove_wake_function+0x11/0x60
[ 62.955836] __wake_up_common+0x82/0x150
[ 62.955979] ? __queue_work+0x125/0x330
[ 62.956119] __wake_up_common_lock+0x7b/0xc0
[ 62.956273] __wake_up+0xe/0x10
[ 62.956390] wake_up_klogd_work_func+0x47/0x70
[ 62.956552] irq_work_run_list+0x55/0x80
[ 62.956694] irq_work_tick+0x55/0x60
[ 62.956825] update_process_times+0x3d/0x50
[ 62.956977] tick_sched_handle+0x24/0x60
[ 62.957120] tick_sched_timer+0x37/0x80
[ 62.957259] ? tick_sched_do_timer+0x60/0x60
[ 62.957413] __hrtimer_run_queues+0xf3/0x190
[ 62.957567] hrtimer_interrupt+0x11d/0x250
[ 62.957718] smp_apic_timer_interrupt+0x60/0xa0
[ 63.577258] apic_timer_interrupt+0xf/0x20
[ 63.579677] </IRQ>
[ 63.579759] RIP: 0010:panic+0x19b/0x1f4
[ 63.579897] Code: 89 01 90 69 09 b9 48 c7 c6 80 c3 92 ba 48 c7 c7 e0 e1 45 ba c7 05 15 d7 59 01 00 00 00 00 e8 ee 56 05 00 fb 66 0f 1f 44 00 00 <45> 31 e4 45 31 f6 31 db e8 9c d6 0a 00 4c 39 f3 7c 1d 41 83 f4 01
[ 63.580536] RSP: 0018:ffff9939a5ed3af0 EFLAGS: 00000286 ORIG_RAX: ffffffffffffff13
[ 63.580800] RAX: 0000000000000039 RBX: 0000000000000000 RCX: 0000000000000000
[ 63.581048] RDX: 0000000000000000 RSI: ffffffffba47f75b RDI: 00000000ffffffff
[ 63.581296] RBP: ffff9939a5ed3b68 R08: 0000000000000000 R09: 0000000000000c63
[ 63.581544] R10: 000000000000064b R11: ffffffffba932f0d R12: ffffffffba450f06
[ 63.581792] R13: ffff9939a5ed3b78 R14: 0000000000000046 R15: 0000000000000000
[ 63.582046] oops_end+0x9d/0xb0
[ 63.582163] no_context+0x12d/0x320
[ 63.582290] __bad_area_nosemaphore+0x7c/0x1d0
[ 63.582450] bad_area_nosemaphore+0xf/0x20
[ 63.582598] __do_page_fault+0xa7/0x570
[ 63.582737] do_page_fault+0x22/0x30
[ 63.582869] page_fault+0x1e/0x30
[ 63.582992] RIP: 0010:sysrq_handle_crash+0x16/0x20
[ 63.583163] Code: d5 e8 8e c2 bc ff 0f 1f 40 00 66 2e 0f 1f 84 00 00 00 00 00 55 48 89 e5 e8 77 31 c3 ff c7 05 65 9e 16 01 01 00 00 00 0f ae f8 <c6> 04 25 00 00 00 00 01 5d c3 55 48 89 e5 fb 66 0f 1f 44 00 00 e8
[ 63.583802] RSP: 0018:ffff9939a5ed3d98 EFLAGS: 00010246
[ 63.583986] RAX: 0000000000000000 RBX: 0000000000000063 RCX: 0000000000000000
[ 63.584234] RDX: 0000000000000000 RSI: ffffffffba47f75b RDI: 0000000000000063
[ 63.584482] RBP: ffff9939a5ed3d98 R08: 0000000000000000 R09: 0000000000000c2e
[ 63.584730] R10: 0000000000000000 R11: ffffffffba932f0d R12: 0000000000000003
[ 63.584978] R13: 0000000000000000 R14: ffffffffba6807e0 R15: 0000000000000000
[ 63.585229] ? sysrq_handle_crash+0x9/0x20
[ 63.585377] __handle_sysrq+0xef/0x140
[ 63.585513] write_sysrq_trigger+0x4c/0x50
[ 63.585662] proc_reg_write+0x40/0x70
[ 63.585795] __vfs_write+0x35/0x180
[ 63.585925] ? __this_cpu_preempt_check+0x13/0x20
[ 63.586093] vfs_write+0xc5/0x1c0
[ 63.586214] ksys_write+0x53/0xc0
[ 63.586336] ? __audit_syscall_exit+0x20d/0x2a0
[ 63.586498] __x64_sys_write+0x15/0x20
[ 63.586635] do_syscall_64+0x4e/0x100
[ 63.586768] entry_SYSCALL_64_after_hwframe+0x44/0xa9
[ 63.586948] RIP: 0033:0x7f3d2fca0664
[ 63.587078] Code: 00 f7 d8 64 89 02 48 c7 c0 ff ff ff ff eb b3 0f 1f 80 00 00 00 00 48 8d 05 99 20 0e 00 8b 00 85 c0 75 13 b8 01 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 54 c3 0f 1f 00 41 54 49 89 d4 55 48 89 f5 53
[ 63.587716] RSP: 002b:00007ffcf9071e98 EFLAGS: 00000246 ORIG_RAX: 0000000000000001
[ 63.587980] RAX: ffffffffffffffda RBX: 0000000000000002 RCX: 00007f3d2fca0664
[ 63.588228] RDX: 0000000000000002 RSI: 0000000000ee8190 RDI: 0000000000000001
[ 63.588476] RBP: 0000000000ee8190 R08: 000000000000000a R09: 00007f3d2fb94740
[ 63.588724] R10: 00000000004b8f41 R11: 0000000000000246 R12: 00007f3d2fd7e760
[ 63.588972] R13: 0000000000000002 R14: 00007f3d2fd7a280 R15: 00007f3d2fd79740
[ 63.589222] ---[ end trace 416678827625f99c ]---
[ 63.589394] WARNING: CPU: 3 PID: 336 at kernel/sched/core.c:1163 set_task_cpu+0x145/0x150
[ 63.589680] Modules linked in:
[ 63.589793] CPU: 3 PID: 336 Comm: bash Tainted: G UD W 4.19.0-19.iot-lts2018-sos-g6b95688-dirty #44
[ 63.590143] RIP: 0010:set_task_cpu+0x145/0x150
[ 63.590302] Code: 5e 5d c3 80 8b 94 04 00 00 04 e9 76 ff ff ff 0f 0b e9 00 ff ff ff f7 43 60 fd ff ff ff 0f 84 0a ff ff ff 0f 0b e9 03 ff ff ff <0f> 0b e9 12 ff ff ff 0f 1f 40 00 55 48 89 e5 41 56 49 89 f6 41 55
[ 63.590943] RSP: 0018:ffff9939b4b83c10 EFLAGS: 00010046
[ 63.591128] RAX: 0000000000000200 RBX: ffff9939a7d26180 RCX: ffff9939a7d26180
[ 63.591377] RDX: ffff9939b4b20401 RSI: 0000000000000002 RDI: ffff9939a7d26180
[ 63.591625] RBP: ffff9939b4b83c30 R08: ffff9939a7d2be78 R09: 0000000000000000
[ 63.591874] R10: 0000000000000046 R11: ffffffffba932f0d R12: 0000000000000002
[ 63.592122] R13: ffff9939a7d26908 R14: 0000000000000002 R15: ffff9939a7d264f0
[ 63.592371] FS: 00007f3d2fb94740(0000) GS:ffff9939b4b80000(0000) knlGS:0000000000000000
[ 63.592653] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 63.592855] CR2: 0000000000000000 CR3: 000000006185d000 CR4: 00000000003406e0
[ 63.593103] Call Trace:
[ 63.593194] <IRQ>
[ 63.593272] try_to_wake_up+0x13a/0x3f0
[ 63.593413] default_wake_function+0xd/0x10
[ 63.593563] __wake_up_common+0x82/0x150
[ 63.593706] __wake_up_locked+0x11/0x20
[ 63.593847] ep_poll_callback+0xcc/0x330
[ 63.593989] __wake_up_common+0x82/0x150
[ 63.594131] ? __queue_work+0x125/0x330
[ 63.594270] __wake_up_common_lock+0x7b/0xc0
[ 63.594424] __wake_up+0xe/0x10
[ 63.594540] wake_up_klogd_work_func+0x47/0x70
[ 63.594699] irq_work_run_list+0x55/0x80
[ 63.594841] irq_work_tick+0x55/0x60
[ 63.594971] update_process_times+0x3d/0x50
[ 63.595122] tick_sched_handle+0x24/0x60
[ 63.595264] tick_sched_timer+0x37/0x80
[ 63.595403] ? tick_sched_do_timer+0x60/0x60
[ 63.595557] __hrtimer_run_queues+0xf3/0x190
[ 63.595711] hrtimer_interrupt+0x11d/0x250
[ 63.595859] smp_apic_timer_interrupt+0x60/0xa0
[ 63.596021] apic_timer_interrupt+0xf/0x20
[ 63.596168] </IRQ>
[ 63.596249] RIP: 0010:panic+0x19b/0x1f4
[ 63.596387] Code: 89 01 90 69 09 b9 48 c7 c6 80 c3 92 ba 48 c7 c7 e0 e1 45 ba c7 05 15 d7 59 01 00 00 00 00 e8 ee 56 05 00 fb 66 0f 1f 44 00 00 <45> 31 e4 45 31 f6 31 db e8 9c d6 0a 00 4c 39 f3 7c 1d 41 83 f4 01
[ 63.597027] RSP: 0018:ffff9939a5ed3af0 EFLAGS: 00000286 ORIG_RAX: ffffffffffffff13
[ 63.597292] RAX: 0000000000000039 RBX: 0000000000000000 RCX: 0000000000000000
[ 63.597540] RDX: 0000000000000000 RSI: ffffffffba47f75b RDI: 00000000ffffffff
[ 63.597789] RBP: ffff9939a5ed3b68 R08: 0000000000000000 R09: 0000000000000c63
[ 64.191896] R10: 000000000000064b R11: ffffffffba932f0d R12: ffffffffba450f06
[ 64.192145] R13: ffff9939a5ed3b78 R14: 0000000000000046 R15: 0000000000000000
[ 64.192398] oops_end+0x9d/0xb0
[ 64.192514] no_context+0x12d/0x320
[ 64.192642] __bad_area_nosemaphore+0x7c/0x1d0
[ 64.192802] bad_area_nosemaphore+0xf/0x20
[ 64.192949] __do_page_fault+0xa7/0x570
[ 64.193089] do_page_fault+0x22/0x30
[ 64.193220] page_fault+0x1e/0x30
[ 64.193342] RIP: 0010:sysrq_handle_crash+0x16/0x20
[ 64.193513] Code: d5 e8 8e c2 bc ff 0f 1f 40 00 66 2e 0f 1f 84 00 00 00 00 00 55 48 89 e5 e8 77 31 c3 ff c7 05 65 9e 16 01 01 00 00 00 0f ae f8 <c6> 04 25 00 00 00 00 01 5d c3 55 48 89 e5 fb 66 0f 1f 44 00 00 e8
[ 64.194154] RSP: 0018:ffff9939a5ed3d98 EFLAGS: 00010246
[ 64.194339] RAX: 0000000000000000 RBX: 0000000000000063 RCX: 0000000000000000
[ 64.194588] RDX: 0000000000000000 RSI: ffffffffba47f75b RDI: 0000000000000063
[ 64.194837] RBP: ffff9939a5ed3d98 R08: 0000000000000000 R09: 0000000000000c2e
[ 64.195085] R10: 0000000000000000 R11: ffffffffba932f0d R12: 0000000000000003
[ 64.195334] R13: 0000000000000000 R14: ffffffffba6807e0 R15: 0000000000000000
[ 64.195585] ? sysrq_handle_crash+0x9/0x20
[ 64.195734] __handle_sysrq+0xef/0x140
[ 64.195870] write_sysrq_trigger+0x4c/0x50
[ 64.196018] proc_reg_write+0x40/0x70
[ 64.196151] __vfs_write+0x35/0x180
[ 64.196279] ? __this_cpu_preempt_check+0x13/0x20
[ 64.196447] vfs_write+0xc5/0x1c0
[ 64.196569] ksys_write+0x53/0xc0
[ 64.196690] ? __audit_syscall_exit+0x20d/0x2a0
[ 64.196853] __x64_sys_write+0x15/0x20
[ 64.196988] do_syscall_64+0x4e/0x100
[ 64.197122] entry_SYSCALL_64_after_hwframe+0x44/0xa9
[ 64.197302] RIP: 0033:0x7f3d2fca0664
[ 64.197431] Code: 00 f7 d8 64 89 02 48 c7 c0 ff ff ff ff eb b3 0f 1f 80 00 00 00 00 48 8d 05 99 20 0e 00 8b 00 85 c0 75 13 b8 01 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 54 c3 0f 1f 00 41 54 49 89 d4 55 48 89 f5 53
[ 64.394050] RSP: 002b:00007ffcf9071e98 EFLAGS: 00000246 ORIG_RAX: 0000000000000001
[ 64.394315] RAX: ffffffffffffffda RBX: 0000000000000002 RCX: 00007f3d2fca0664
[ 64.394563] RDX: 0000000000000002 RSI: 0000000000ee8190 RDI: 0000000000000001
[ 64.394812] RBP: 0000000000ee8190 R08: 000000000000000a R09: 00007f3d2fb94740
[ 64.395061] R10: 00000000004b8f41 R11: 0000000000000246 R12: 00007f3d2fd7e760
[ 64.395309] R13: 0000000000000002 R14: 00007f3d2fd7a280 R15: 00007f3d2fd79740
[ 64.395559] ---[ end trace 416678827625f99d ]---
[ 64.395735] ------------[ cut here ]------------
[ 64.395899] sched: Unexpected reschedule of offline CPU#2!
[ 64.396095] WARNING: CPU: 3 PID: 336 at arch/x86/kernel/smp.c:141 native_smp_send_reschedule+0x3a/0x40
[ 64.396417] Modules linked in:
[ 64.396531] CPU: 3 PID: 336 Comm: bash Tainted: G UD W 4.19.0-19.iot-lts2018-sos-g6b95688-dirty #44
[ 64.396880] RIP: 0010:native_smp_send_reschedule+0x3a/0x40
[ 64.397074] Code: c0 84 c0 74 17 48 8b 05 c4 40 54 01 be fd 00 00 00 48 8b 40 30 e8 f6 a8 dc 00 5d c3 89 fe 48 c7 c7 00 7a 45 ba e8 e6 e2 05 00 <0f> 0b 5d c3 66 90 55 48 89 e5 41 54 53 48 83 ec 20 65 48 8b 04 25
[ 64.397715] RSP: 0018:ffff9939b4b83ba0 EFLAGS: 00010 64.532707] irq_work_tick+0x55/0x60
[ 64.532838] update_process_times+0x3d/0x50
[ 64.532988] tick_sched_handle+0x24/0x60
[ 64.533130] tick_sched_timer+0x37/0x80
[ 64.533270] ? tick_sched_do_timer+0x60/0x60
[ 64.533423] __hrtimer_run_queues+0xf3/0x190
[ 64.533577] hrtimer_interrupt+0x11d/0x250
[ 64.533725] smp_apic_timer_interrupt+0x60/0xa0
[ 64.533887] apic_timer_interrupt+0xf/0x20
[ 64.534034] </IRQ>
[ 64.534115] RIP: 0010:panic+0x19b/0x1f4
[ 64.534253] Code: 89 01 90 69 09 b9 48 c7 c6 80 c3 92 ba 48 c7 c7 e0 e1 45 ba c7 05 15 d7 59 01 00 00 00 00 e8 ee 56 05 00 fb 66 0f 1f 44 00 00 <45> 31 e4 45 31 f6 31 db e8 9c d6 0a 00 4c 39 f3 7c 1d 41 83 f4 01
[ 64.534894] RSP: 0018:ffff9939a5ed3af0 EFLAGS: 00000286 ORIG_RAX: ffffffffffffff13
[ 64.535158] RAX: 0000000000000039 RBX: 0000000000000000 RCX: 0000000000000000
[ 64.535407] RDX: 0000000000000000 RSI: ffffffffba47f75b RDI: 00000000ffffffff
[ 64.535655] RBP: ffff9939a5ed3b68 R08: 0000000000000000 R09: 0000000000000c63
[ 64.535904] R10: 000000000000064b R11: ffffffffba932f0d R12: ffffffffba450f06
[ 64.536153] R13: ffff9939a5ed3b78 R14: 0000000000000046 R15: 0000000000000000
[ 64.536405] oops_end+0x9d/0xb0
[ 64.536521] no_context+0x12d/0x320
[ 64.536649] __bad_area_nosemaphore+0x7c/0x1d0
[ 64.536809] bad_area_nosemaphore+0xf/0x20
[ 64.536956] __do_page_fault+0xa7/0x570
[ 64.537096] do_page_fault+0x22/0x30
[ 64.537227] page_fault+0x1e/0x30
[ 64.537349] RIP: 0010:sysrq_handle_crash+0x16/0x20
[ 64.537520] Code: d5 e8 8e c2 bc ff 0f 1f 40 00 66 2e 0f 1f 84 00 00 00 00 00 55 48 89 e5 e8 77 31 c3 ff c7 05 65 9e 16 01 01 00 00 00 0f ae f8 <c6> 04 25 00 00 00 00 01 5d c3 55 48 89 e5 fb 66 0f 1f 44 00 00 e8
[ 64.538164] RSP: 0018:ffff9939a5ed3d98 EFLAGS: 00010246
[ 64.538349] RAX: 0000000000000000 RBX: 0000000000000063 RCX: 0000000000000000
[ 64.538598] RDX: 0000000000000000 RSI: ffffffffba47f75b RDI: 0000000000000063
[ 64.538846] RBP: ffff9939a5ed3d98 R08: 0000000000000000 R09: 0000000000000c2e
[ 64.539095] R10: 0000000000000000 R11: ffffffffba932f0d R12: 0000000000000003
[ 64.539343] R13: 0000000000000000 R14: ffffffffba6807e0 R15: 0000000000000000
[ 64.539594] ? sysrq_handle_crash+0x9/0x20
[ 64.539743] __handle_sysrq+0xef/0x140
[ 64.539879] write_sysrq_trigger+0x4c/0x50
[ 64.540027] proc_reg_write+0x40/0x70
[ 64.540160] __vfs_write+0x35/0x180
[ 64.540289] ? __this_cpu_preempt_check+0x13/0x20
[ 64.540457] vfs_write+0xc5/0x1c0
[ 64.540578] ksys_write+0x53/0xc0
[ 64.540700] ? __audit_syscall_exit+0x20d/0x2a0
[ 64.540862] __x64_sys_write+0x15/0x20
[ 64.540998] do_syscall_64+0x4e/0x100
[ 64.541132] entry_SYSCALL_64_after_hwframe+0x44/0xa9
[ 64.541311] RIP: 0033:0x7f3d2fca0664
[ 64.541441] Code: 00 f7 d8 64 89 02 48 c7 c0 ff ff ff ff eb b3 0f 1f 80 00 00 00 00 48 8d 05 99 20 0e 00 8b 00 85 c0 75 13 b8 01 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 54 c3 0f 1f 00 41 54 49 89 d4 55 48 89 f5 53
[ 64.542080] RSP: 002b:00007ffcf9071e98 EFLAGS: 00000246 ORIG_RAX: 0000000000000001
[ 64.542345] RAX: ffffffffffffffda RBX: 0000000000000002 RCX: 00007f3d2fca0664
[ 64.542593] RDX: 0000000000000002 RSI: 0000000000ee8190 RDI: 0000000000000001
[ 64.542842] RBP: 0000000000ee8190 R08: 000000000000000a R09: 00007f3d2fb94740
[ 64.543090] R10: 00000000004b8f41 R11: 0000000000000246 R12: 00007f3d2fd7e760
[ 64.543339] R13: 0000000000000002 R14: 00007f3d2fd7a280 R15: 00007f3d2fd79740
[ 64.543589] ---[ end trace 416678827625f99e ]---
[ 64.543775] ------------[ cut here ]------------
[ 64.543942] sched: Unexpected reschedule of offline CPU#0!
[ 64.544138] WARNING: CPU: 3 PID: 336 at arch/x86/kernel/smp.c:141 native_smp_send_reschedule+0x3a/0x40
[ 64.544460] Modules linked in:
[ 64.544573] CPU: 3 PID: 336 Comm: bash Tainted: G UD W 4.19.0-19.iot-lts2018-sos-g6b95688-dirty #44
[ 64.544923] RIP: 0010:native_smp_send_reschedule+0x3a/0x40
[ 64.545117] Code: c0 84 c0 74 17 48 8b 05 c4 40 54 01 be fd 00 00 00 48 8b 40 30 e8 f6 a8 dc 00 5d c3 89 fe 48 c7 c7 00 7a 45 ba e8 e6 e2 05 00 <0f> 0b 5d c3 66 90 55 48 89 e5 41 54 53 48 83 ec 20 65 48 8b 04 25
[ 64.545755] RSP: 0018:ffff9939b4b83e28 EFLAGS: 00010086
[ 64.545939] RAX: 0000000000000000 RBX: 0000000000000003 RCX: 0000000000000000
[ 64.546187] RDX: 0000000000000007 RSI: ffffffffba47f75b RDI: 00000000ffffffff
[ 64.546434] RBP: ffff9939b4b83e28 R08: 0000000000000000 R09: 0000000000000d5e
[ 64.546682] R10: 0000000000000400 R11: ffffffffba932f0d R12: 0000000000000000
[ 64.546930] R13: 0000000000000003 R14: ffff9939a3650280 R15: ffff9939b4b9be00
[ 64.547178] FS: 00007f3d2fb94740(0000) GS:ffff9939b4b80000(0000) knlGS:0000000000000000
[ 64.547458] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 64.547659] CR2: 0000000000000000 CR3: 000000006185d000 CR4: 00000000003406e0
[ 64.547906] Call Trace:
[ 64.547997] <IRQ>
[ 64.548074] kick_ilb+0x8b/0xa0
[ 64.548190] trigger_load_balance+0xb1/0x1a0
[ 64.548344] scheduler_tick+0xa4/0xd0
[ 64.548477] update_process_times+0x42/0x50
[ 64.548627] tick_sched_handle+0x24/0x60
[ 64.548769] tick_sched_timer+0x37/0x80
[ 64.548908] ? tick_sched_do_timer+0x60/0x60
[ 64.549060] __hrtimer_run_queues+0xf3/0x190
[ 64.549214] hrtimer_interrupt+0x11d/0x250
[ 64.549362] smp_apic_timer_interrupt+0x60/0xa0
[ 64.549523] apic_timer_interrupt+0xf/0x20
[ 64.549670] </IRQ>
[ 64.549750] RIP: 0010:panic+0x19b/0x1f4
[ 64.549888] Code: 89 01 90 69 09 b9 48 c7 c6 80 c3 92 ba 48 c7 c7 e0 e1 45 ba c7 05 15 d7 59 01 00 00 00 00 e8 ee 56 05 00 fb 66 0f 1f 44 00 00 <45> 31 e4 45 31 f6 31 db e8 9c d6 0a 00 4c 39 f3 7c 1d 41 83 f4 01
[ 64.550525] RSP: 0018:ffff9939a5ed3af0 EFLAGS: 00000286 ORIG_RAX: ffffffffffffff13
[ 64.550789] RAX: 0000000000000039 RBX: 0000000000000000 RCX: 0000000000000000
[ 64.551036] RDX: 0000000000000000 RSI: ffffffffba47f75b RDI: 00000000ffffffff
[ 64.551284] RBP: ffff9939a5ed3b68 R08: 0000000000000000 R09: 0000000000000c63
[ 64.551532] R10: 000000000000064b R11: ffffffffba932f0d R12: ffffffffba450f06
[ 64.551779] R13: ffff9939a5ed3b78 R14: 0000000000000046 R15: 0000000000000000
[ 64.552031] oops_end+0x9d/0xb0
[ 64.552146] no_context+0x12d/0x320
[ 64.552274] __bad_area_nosemaphore+0x7c/0x1d0
[ 64.552433] bad_area_nosemaphore+0xf/0x20
[ 64.552580] __do_page_fault+0xa7/0x570
[ 64.552720] do_page_fault+0x22/0x30
[ 64.552850] page_fault+0x1e/0x30
[ 64.552972] RIP: 0010:sysrq_handle_crash+0x16/0x20
[ 64.553141] Code: d5 e8 8e c2 bc ff 0f 1f 40 00 66 2e 0f 1f 84 00 00 00 00 00 55 48 89 e5 e8 77 31 c3 ff c7 05 65 9e 16 01 01 00 00 00 0f ae f8 <c6> 04 25 00 00 00 00 01 5d c3 55 48 89 e5 fb 66 0f 1f 44 00 00 e8
[ 64.553779] RSP: 0018:ffff9939a5ed3d98 EFLAGS: 00010246
[ 64.553964] RAX: 0000000000000000 RBX: 0000000000000063 RCX: 0000000000000000
[ 64.554212] RDX: 0000000000000000 RSI: ffffffffba47f75b RDI: 0000000000000063
[ 64.554460] RBP: ffff9939a5ed3d98 R08: 0000000000000000 R09: 0000000000000c2e
[ 64.554709] R10: 0000000000000000 R11: ffffffffba932f0d R12: 0000000000000003
[ 64.554957] R13: 0000000000000000 R14: ffffffffba6807e0 R15: 0000000000000000
[ 64.555207] ? sysrq_handle_crash+0x9/0x20
[ 64.555355] __handle_sysrq+0xef/0x140
[ 64.555491] write_sysrq_trigger+0x4c/0x50
[ 64.555638] proc_reg_write+0x40/0x70
[ 64.555771] __vfs_write+0x35/0x180
[ 64.555898] ? __this_cpu_preempt_check+0x13/0x20
[ 64.556066] vfs_write+0xc5/0x1c0
[ 64.556187] ksys_write+0x53/0xc0
[ 64.556308] ? __audit_syscall_exit+0x20d/0x2a0
[ 64.556470] __x64_sys_write+0x15/0x20
[ 64.556606] do_syscall_64+0x4e/0x100
[ 64.556739] entry_SYSCALL_64_after_hwframe+0x44/0xa9
[ 64.556918] RIP: 0033:0x7f3d2fca0664
[ 64.557047] Code: 00 f7 d8 64 89 02 48 c7 c0 ff ff ff ff eb b3 0f 1f 80 00 00 00 00 48 8d 05 99 20 0e 00 8b 00 85 c0 75 13 b8 01 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 54 c3 0f 1f 00 41 54 49 89 d4 55 48 89 f5 53
[ 64.557685] RSP: 002b:00007ffcf9071e98 EFLAGS: 00000246 ORIG_RAX: 0000000000000001
[ 65.374953] RAX: ffffffffffffffda RBX: 0000000000000002 RCX: 00007f3d2fca0664
[ 65.376235] RDX: 0000000000000002 RSI: 0000000000ee8190 RDI: 0000000000000001
[ 65.376482] RBP: 0000000000ee8190 R08: 000000000000000a R09: 00007f3d2fb94740
[ 65.376730] R10: 00000000004b8f41 R11: 0000000000000246 R12: 00007f3d2fd7e760
[ 65.376978] R13: 0000000000000002 R14: 00007f3d2fd7a280 R15: 00007f3d2fd79740
[ 65.377228] ---[ end trace 416678827625f99f ]---
^ permalink raw reply [flat|nested] 7+ messages in thread
end of thread, other threads:[~2018-11-27 3:25 UTC | newest]
Thread overview: 7+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2018-10-11 7:17 [PATCH RFC] panic: Avoid extra noisy messages due to stopped cpus Feng Tang
2018-10-11 9:35 ` Peter Zijlstra
2018-10-11 9:59 ` Feng Tang
2018-10-22 9:55 ` Feng Tang
2018-11-08 13:05 ` [PATCH v2] panic: Avoid the extra noise dmesg Feng Tang
2018-11-26 23:59 ` Andrew Morton
2018-11-27 3:32 ` Feng Tang
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.