All of lore.kernel.org
 help / color / mirror / Atom feed
From: Feng Tang <feng.tang@intel.com>
To: Andrew Morton <akpm@linux-foundation.org>
Cc: Peter Zijlstra <peterz@infradead.org>,
	Thomas Gleixner <tglx@linutronix.de>,
	Ingo Molnar <mingo@kernel.org>,
	H Peter Anvin <hpa@linux.intel.com>,
	Borislav Petkov <bp@alien8.de>, Kees Cook <keescook@chromium.org>,
	linux-kernel@vger.kernel.org
Subject: Re: [PATCH v2] panic: Avoid the extra noise dmesg
Date: Tue, 27 Nov 2018 11:32:56 +0800	[thread overview]
Message-ID: <20181127033256.2xvjvjhmqu3hqhpm@shbuild888> (raw)
In-Reply-To: <20181126155952.46d0c641a69746d33f4212e6@linux-foundation.org>

Hi Andrew,

On Mon, Nov 26, 2018 at 03:59:52PM -0800, Andrew Morton wrote:
> On Thu, 8 Nov 2018 21:05:03 +0800 Feng Tang <feng.tang@intel.com> wrote:
> 
> > Sometimes when debugging kernel panic, we saw many extra noisy error
> > messages after the expected end:
> > 
> > [   35.743249] ---[ end Kernel panic - not syncing: Fatal exception
> > [   35.749975] ------------[ cut here ]------------
> 
> What are these "noisy messages"?  The above looks OK?

Yes, the above is the sane messages :) and contains the necessary context
info for debug.

The noisy messages are those following messages, I tried to capture one with
sysrq and paste it in the tail of this mail as it's big (for a 4 cores
machine) 

> > When panic happens under console mode, the screen will first show the panic
> > call stack, but will immediately be overrided by noisy extra messages, like
> > this for every CPU except the panic one:
> > 
> > 	WARNING: CPU: 1 PID: 280 at kernel/sched/core.c:1198 set_task_cpu+0x183/0x190
> > 	Call Trace:
> > 	<IRQ>
> > 	try_to_wake_up
> > 	default_wake_function
> > 	autoremove_wake_function
> > 	__wake_up_common
> > 	__wake_up_common_lock
> > 	__wake_up
> > 	wake_up_klogd_work_func
> > 	irq_work_run_list
> > 	irq_work_tick
> > 	update_process_times
> > 	tick_sched_timer
> > 	__hrtimer_run_queues
> > 	hrtimer_interrupt
> > 	smp_apic_timer_interrupt
> > 	apic_timer_interrupt
> 
> And this output looks OK as well.
 
Yes, this is fine also. What I mean is it will be a little noisy if each and
every CPU will dump similar info, which will make it hard to find the real panic
context if a user has no serial console but a LCD.

> > Which make debugging more difficult, as the original context is
> > lost on screen.
> > 
> > Keeping the CPU IRQ disabled will void these messages, as we code
> > run to this point, the user has chosed not to reboot, nor do
> > some extra handling with panic notifier, not much point in
> > re-enabling the interrupt.
> 
> And please have another attempt at the above paragraph?

Sorry for the unclear info, will try to sanitize it.

> > --- a/kernel/panic.c
> > +++ b/kernel/panic.c
> > @@ -295,7 +295,6 @@ void panic(const char *fmt, ...)
> >  	}
> >  #endif
> >  	pr_emerg("---[ end Kernel panic - not syncing: %s ]---\n", buf);
> > -	local_irq_enable();
> >  	for (i = 0; ; i += PANIC_TIMER_STEP) {
> >  		touch_softlockup_watchdog();
> >  		if (i >= i_next) {
> 
> The patch looks reasonable, but the description is just too hard for me
> to follow, sorry.  Please spend a bit more time over that and resend?

Sure and thanks for your review! 

- Feng
-------

Panic log:

#echo c > /proc/sysrq-trigger 
[   62.918474] sysrq: SysRq : Trigger a crash
[   62.918643] BUG: unable to handle kernel NULL pointer dereference at 0000000000000000
[   62.918906] PGD 67e36067 P4D 67e36067 PUD 6189f067 PMD 0 
[   62.919092] Oops: 0002 [#1] PREEMPT SMP
[   62.919226] CPU: 3 PID: 336 Comm: bash Tainted: G     U  W         4.19.0-19.iot-lts2018-sos-g6b95688-dirty #44
[   62.919563] RIP: 0010:sysrq_handle_crash+0x16/0x20
[   62.919726] Code: d5 e8 8e c2 bc ff 0f 1f 40 00 66 2e 0f 1f 84 00 00 00 00 00 55 48 89 e5 e8 77 31 c3 ff c7 05 65 9e 16 01 01 00 00 00 0f ae f8 <c6> 04 25 00 00 00 00 01 5d c3 55 48 89 e5 fb 66 0f 1f 44 00 00 e8
[   62.920332] RSP: 0018:ffff9939a5ed3d98 EFLAGS: 00010246
[   62.920508] RAX: 0000000000000000 RBX: 0000000000000063 RCX: 0000000000000000
[   62.920744] RDX: 0000000000000000 RSI: ffffffffba47f75b RDI: 0000000000000063
[   62.920979] RBP: ffff9939a5ed3d98 R08: 0000000000000000 R09: 0000000000000c2e
[   62.921216] R10: 0000000000000000 R11: ffffffffba932f0d R12: 0000000000000003
[   62.921457] R13: 0000000000000000 R14: ffffffffba6807e0 R15: 0000000000000000
[   62.921693] FS:  00007f3d2fb94740(0000) GS:ffff9939b4b80000(0000) knlGS:0000000000000000
[   62.921959] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[   62.922150] CR2: 0000000000000000 CR3: 000000006185d000 CR4: 00000000003406e0
[   62.922385] Call Trace:
[   62.922476]  __handle_sysrq+0xef/0x140
[   62.922605]  write_sysrq_trigger+0x4c/0x50
[   62.922746]  proc_reg_write+0x40/0x70
[   62.922874]  __vfs_write+0x35/0x180
[   62.922997]  ? __this_cpu_preempt_check+0x13/0x20
[   62.923156]  vfs_write+0xc5/0x1c0
[   62.923271]  ksys_write+0x53/0xc0
[   62.923388]  ? __audit_syscall_exit+0x20d/0x2a0
[   62.923541]  __x64_sys_write+0x15/0x20
[   62.923671]  do_syscall_64+0x4e/0x100
[   62.923800]  entry_SYSCALL_64_after_hwframe+0x44/0xa9
[   62.923970] RIP: 0033:0x7f3d2fca0664
[   62.924093] Code: 00 f7 d8 64 89 02 48 c7 c0 ff ff ff ff eb b3 0f 1f 80 00 00 00 00 48 8d 05 99 20 0e 00 8b 00 85 c0 75 13 b8 01 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 54 c3 0f 1f 00 41 54 49 89 d4 55 48 89 f5 53
[   62.924699] RSP: 002b:00007ffcf9071e98 EFLAGS: 00000246 ORIG_RAX: 0000000000000001
[   62.924949] RAX: ffffffffffffffda RBX: 0000000000000002 RCX: 00007f3d2fca0664
[   62.925184] RDX: 0000000000000002 RSI: 0000000000ee8190 RDI: 0000000000000001
[   62.925420] RBP: 0000000000ee8190 R08: 000000000000000a R09: 00007f3d2fb94740
[   62.925655] R10: 00000000004b8f41 R11: 0000000000000246 R12: 00007f3d2fd7e760
[   62.925890] R13: 0000000000000002 R14: 00007f3d2fd7a280 R15: 00007f3d2fd79740
[   62.926126] Modules linked in:
[   62.926234] CR2: 0000000000000000
[   62.926349] ---[ end trace 416678827625f99b ]---
[   62.936473] RIP: 0010:sysrq_handle_crash+0x16/0x20
[   62.936644] Code: d5 e8 8e c2 bc ff 0f 1f 40 00 66 2e 0f 1f 84 00 00 00 00 00 55 48 89 e5 e8 77 31 c3 ff c7 05 65 9e 16 01 01 00 00 00 0f ae f8 <c6> 04 25 00 00 00 00 01 5d c3 55 48 89 e5 fb 66 0f 1f 44 00 00 e8
[   62.937260] RSP: 0018:ffff9939a5ed3d98 EFLAGS: 00010246
[   62.937438] RAX: 0000000000000000 RBX: 0000000000000063 RCX: 0000000000000000
[   62.937677] RDX: 0000000000000000 RSI: ffffffffba47f75b RDI: 0000000000000063
[   62.937917] RBP: ffff9939a5ed3d98 R08: 0000000000000000 R09: 0000000000000c2e
[   62.938158] R10: 0000000000000000 R11: ffffffffba932f0d R12: 0000000000000003
[   62.938396] R13: 0000000000000000 R14: ffffffffba6807e0 R15: 0000000000000000
[   62.938636] FS:  00007f3d2fb94740(0000) GS:ffff9939b4b80000(0000) knlGS:0000000000000000
[   62.938905] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[   62.939099] CR2: 0000000000000000 CR3: 000000006185d000 CR4: 00000000003406e0
[   62.939339] Kernel panic - not syncing: Fatal exception
[   62.939941] reboot: panic mode set: p,w
[   62.940100] Kernel Offset: 0x38000000 from 0xffffffff81000000 (relocation range: 0xffffffff80000000-0xffffffffbfffffff)
[   62.950783] ---[ end Kernel panic - not syncing: Fatal exception ]---
[   62.951034] ------------[ cut here ]------------
[   62.951201] sched: Unexpected reschedule of offline CPU#1!
[   62.951409] WARNING: CPU: 3 PID: 336 at arch/x86/kernel/smp.c:141 native_smp_send_reschedule+0x3a/0x40
[   62.951732] Modules linked in:
[   62.951848] CPU: 3 PID: 336 Comm: bash Tainted: G     UD W         4.19.0-19.iot-lts2018-sos-g6b95688-dirty #44
[   62.952199] RIP: 0010:native_smp_send_reschedule+0x3a/0x40
[   62.952394] Code: c0 84 c0 74 17 48 8b 05 c4 40 54 01 be fd 00 00 00 48 8b 40 30 e8 f6 a8 dc 00 5d c3 89 fe 48 c7 c7 00 7a 45 ba e8 e6 e2 05 00 <0f> 0b 5d c3 66 90 55 48 89 e5 41 54 53 48 83 ec 20 65 48 8b 04 25
[   62.953037] RSP: 0018:ffff9939b4b83cd8 EFLAGS: 00010082
[   62.953222] RAX: 0000000000000000 RBX: 0000000000000001 RCX: 0000000000000000
[   62.953472] RDX: 0000000000000007 RSI: ffffffffba47f75b RDI: 00000000ffffffff
[   62.953721] RBP: ffff9939b4b83cd8 R08: 0000000000000000 R09: 0000000000000c65
[   62.953970] R10: 0000000000000000 R11: ffffffffba932f0d R12: 0000000000000000
[   62.954219] R13: ffff9939a46e28c8 R14: ffff9939a46e2140 R15: 0000000000020480
[   62.954469] FS:  00007f3d2fb94740(0000) GS:ffff9939b4b80000(0000) knlGS:0000000000000000
[   62.954753] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[   62.954955] CR2: 0000000000000000 CR3: 000000006185d000 CR4: 00000000003406e0
[   62.955205] Call Trace:
[   62.955299]  <IRQ>
[   62.955380]  try_to_wake_up+0x386/0x3f0
[   62.955521]  default_wake_function+0xd/0x10
[   62.955673]  autoremove_wake_function+0x11/0x60
[   62.955836]  __wake_up_common+0x82/0x150
[   62.955979]  ? __queue_work+0x125/0x330
[   62.956119]  __wake_up_common_lock+0x7b/0xc0
[   62.956273]  __wake_up+0xe/0x10
[   62.956390]  wake_up_klogd_work_func+0x47/0x70
[   62.956552]  irq_work_run_list+0x55/0x80
[   62.956694]  irq_work_tick+0x55/0x60
[   62.956825]  update_process_times+0x3d/0x50
[   62.956977]  tick_sched_handle+0x24/0x60
[   62.957120]  tick_sched_timer+0x37/0x80
[   62.957259]  ? tick_sched_do_timer+0x60/0x60
[   62.957413]  __hrtimer_run_queues+0xf3/0x190
[   62.957567]  hrtimer_interrupt+0x11d/0x250
[   62.957718]  smp_apic_timer_interrupt+0x60/0xa0
[   63.577258]  apic_timer_interrupt+0xf/0x20
[   63.579677]  </IRQ>
[   63.579759] RIP: 0010:panic+0x19b/0x1f4
[   63.579897] Code: 89 01 90 69 09 b9 48 c7 c6 80 c3 92 ba 48 c7 c7 e0 e1 45 ba c7 05 15 d7 59 01 00 00 00 00 e8 ee 56 05 00 fb 66 0f 1f 44 00 00 <45> 31 e4 45 31 f6 31 db e8 9c d6 0a 00 4c 39 f3 7c 1d 41 83 f4 01
[   63.580536] RSP: 0018:ffff9939a5ed3af0 EFLAGS: 00000286 ORIG_RAX: ffffffffffffff13
[   63.580800] RAX: 0000000000000039 RBX: 0000000000000000 RCX: 0000000000000000
[   63.581048] RDX: 0000000000000000 RSI: ffffffffba47f75b RDI: 00000000ffffffff
[   63.581296] RBP: ffff9939a5ed3b68 R08: 0000000000000000 R09: 0000000000000c63
[   63.581544] R10: 000000000000064b R11: ffffffffba932f0d R12: ffffffffba450f06
[   63.581792] R13: ffff9939a5ed3b78 R14: 0000000000000046 R15: 0000000000000000
[   63.582046]  oops_end+0x9d/0xb0
[   63.582163]  no_context+0x12d/0x320
[   63.582290]  __bad_area_nosemaphore+0x7c/0x1d0
[   63.582450]  bad_area_nosemaphore+0xf/0x20
[   63.582598]  __do_page_fault+0xa7/0x570
[   63.582737]  do_page_fault+0x22/0x30
[   63.582869]  page_fault+0x1e/0x30
[   63.582992] RIP: 0010:sysrq_handle_crash+0x16/0x20
[   63.583163] Code: d5 e8 8e c2 bc ff 0f 1f 40 00 66 2e 0f 1f 84 00 00 00 00 00 55 48 89 e5 e8 77 31 c3 ff c7 05 65 9e 16 01 01 00 00 00 0f ae f8 <c6> 04 25 00 00 00 00 01 5d c3 55 48 89 e5 fb 66 0f 1f 44 00 00 e8
[   63.583802] RSP: 0018:ffff9939a5ed3d98 EFLAGS: 00010246
[   63.583986] RAX: 0000000000000000 RBX: 0000000000000063 RCX: 0000000000000000
[   63.584234] RDX: 0000000000000000 RSI: ffffffffba47f75b RDI: 0000000000000063
[   63.584482] RBP: ffff9939a5ed3d98 R08: 0000000000000000 R09: 0000000000000c2e
[   63.584730] R10: 0000000000000000 R11: ffffffffba932f0d R12: 0000000000000003
[   63.584978] R13: 0000000000000000 R14: ffffffffba6807e0 R15: 0000000000000000
[   63.585229]  ? sysrq_handle_crash+0x9/0x20
[   63.585377]  __handle_sysrq+0xef/0x140
[   63.585513]  write_sysrq_trigger+0x4c/0x50
[   63.585662]  proc_reg_write+0x40/0x70
[   63.585795]  __vfs_write+0x35/0x180
[   63.585925]  ? __this_cpu_preempt_check+0x13/0x20
[   63.586093]  vfs_write+0xc5/0x1c0
[   63.586214]  ksys_write+0x53/0xc0
[   63.586336]  ? __audit_syscall_exit+0x20d/0x2a0
[   63.586498]  __x64_sys_write+0x15/0x20
[   63.586635]  do_syscall_64+0x4e/0x100
[   63.586768]  entry_SYSCALL_64_after_hwframe+0x44/0xa9
[   63.586948] RIP: 0033:0x7f3d2fca0664
[   63.587078] Code: 00 f7 d8 64 89 02 48 c7 c0 ff ff ff ff eb b3 0f 1f 80 00 00 00 00 48 8d 05 99 20 0e 00 8b 00 85 c0 75 13 b8 01 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 54 c3 0f 1f 00 41 54 49 89 d4 55 48 89 f5 53
[   63.587716] RSP: 002b:00007ffcf9071e98 EFLAGS: 00000246 ORIG_RAX: 0000000000000001
[   63.587980] RAX: ffffffffffffffda RBX: 0000000000000002 RCX: 00007f3d2fca0664
[   63.588228] RDX: 0000000000000002 RSI: 0000000000ee8190 RDI: 0000000000000001
[   63.588476] RBP: 0000000000ee8190 R08: 000000000000000a R09: 00007f3d2fb94740
[   63.588724] R10: 00000000004b8f41 R11: 0000000000000246 R12: 00007f3d2fd7e760
[   63.588972] R13: 0000000000000002 R14: 00007f3d2fd7a280 R15: 00007f3d2fd79740
[   63.589222] ---[ end trace 416678827625f99c ]---
[   63.589394] WARNING: CPU: 3 PID: 336 at kernel/sched/core.c:1163 set_task_cpu+0x145/0x150
[   63.589680] Modules linked in:
[   63.589793] CPU: 3 PID: 336 Comm: bash Tainted: G     UD W         4.19.0-19.iot-lts2018-sos-g6b95688-dirty #44
[   63.590143] RIP: 0010:set_task_cpu+0x145/0x150
[   63.590302] Code: 5e 5d c3 80 8b 94 04 00 00 04 e9 76 ff ff ff 0f 0b e9 00 ff ff ff f7 43 60 fd ff ff ff 0f 84 0a ff ff ff 0f 0b e9 03 ff ff ff <0f> 0b e9 12 ff ff ff 0f 1f 40 00 55 48 89 e5 41 56 49 89 f6 41 55
[   63.590943] RSP: 0018:ffff9939b4b83c10 EFLAGS: 00010046
[   63.591128] RAX: 0000000000000200 RBX: ffff9939a7d26180 RCX: ffff9939a7d26180
[   63.591377] RDX: ffff9939b4b20401 RSI: 0000000000000002 RDI: ffff9939a7d26180
[   63.591625] RBP: ffff9939b4b83c30 R08: ffff9939a7d2be78 R09: 0000000000000000
[   63.591874] R10: 0000000000000046 R11: ffffffffba932f0d R12: 0000000000000002
[   63.592122] R13: ffff9939a7d26908 R14: 0000000000000002 R15: ffff9939a7d264f0
[   63.592371] FS:  00007f3d2fb94740(0000) GS:ffff9939b4b80000(0000) knlGS:0000000000000000
[   63.592653] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[   63.592855] CR2: 0000000000000000 CR3: 000000006185d000 CR4: 00000000003406e0
[   63.593103] Call Trace:
[   63.593194]  <IRQ>
[   63.593272]  try_to_wake_up+0x13a/0x3f0
[   63.593413]  default_wake_function+0xd/0x10
[   63.593563]  __wake_up_common+0x82/0x150
[   63.593706]  __wake_up_locked+0x11/0x20
[   63.593847]  ep_poll_callback+0xcc/0x330
[   63.593989]  __wake_up_common+0x82/0x150
[   63.594131]  ? __queue_work+0x125/0x330
[   63.594270]  __wake_up_common_lock+0x7b/0xc0
[   63.594424]  __wake_up+0xe/0x10
[   63.594540]  wake_up_klogd_work_func+0x47/0x70
[   63.594699]  irq_work_run_list+0x55/0x80
[   63.594841]  irq_work_tick+0x55/0x60
[   63.594971]  update_process_times+0x3d/0x50
[   63.595122]  tick_sched_handle+0x24/0x60
[   63.595264]  tick_sched_timer+0x37/0x80
[   63.595403]  ? tick_sched_do_timer+0x60/0x60
[   63.595557]  __hrtimer_run_queues+0xf3/0x190
[   63.595711]  hrtimer_interrupt+0x11d/0x250
[   63.595859]  smp_apic_timer_interrupt+0x60/0xa0
[   63.596021]  apic_timer_interrupt+0xf/0x20
[   63.596168]  </IRQ>
[   63.596249] RIP: 0010:panic+0x19b/0x1f4
[   63.596387] Code: 89 01 90 69 09 b9 48 c7 c6 80 c3 92 ba 48 c7 c7 e0 e1 45 ba c7 05 15 d7 59 01 00 00 00 00 e8 ee 56 05 00 fb 66 0f 1f 44 00 00 <45> 31 e4 45 31 f6 31 db e8 9c d6 0a 00 4c 39 f3 7c 1d 41 83 f4 01
[   63.597027] RSP: 0018:ffff9939a5ed3af0 EFLAGS: 00000286 ORIG_RAX: ffffffffffffff13
[   63.597292] RAX: 0000000000000039 RBX: 0000000000000000 RCX: 0000000000000000
[   63.597540] RDX: 0000000000000000 RSI: ffffffffba47f75b RDI: 00000000ffffffff
[   63.597789] RBP: ffff9939a5ed3b68 R08: 0000000000000000 R09: 0000000000000c63
[   64.191896] R10: 000000000000064b R11: ffffffffba932f0d R12: ffffffffba450f06
[   64.192145] R13: ffff9939a5ed3b78 R14: 0000000000000046 R15: 0000000000000000
[   64.192398]  oops_end+0x9d/0xb0
[   64.192514]  no_context+0x12d/0x320
[   64.192642]  __bad_area_nosemaphore+0x7c/0x1d0
[   64.192802]  bad_area_nosemaphore+0xf/0x20
[   64.192949]  __do_page_fault+0xa7/0x570
[   64.193089]  do_page_fault+0x22/0x30
[   64.193220]  page_fault+0x1e/0x30
[   64.193342] RIP: 0010:sysrq_handle_crash+0x16/0x20
[   64.193513] Code: d5 e8 8e c2 bc ff 0f 1f 40 00 66 2e 0f 1f 84 00 00 00 00 00 55 48 89 e5 e8 77 31 c3 ff c7 05 65 9e 16 01 01 00 00 00 0f ae f8 <c6> 04 25 00 00 00 00 01 5d c3 55 48 89 e5 fb 66 0f 1f 44 00 00 e8
[   64.194154] RSP: 0018:ffff9939a5ed3d98 EFLAGS: 00010246
[   64.194339] RAX: 0000000000000000 RBX: 0000000000000063 RCX: 0000000000000000
[   64.194588] RDX: 0000000000000000 RSI: ffffffffba47f75b RDI: 0000000000000063
[   64.194837] RBP: ffff9939a5ed3d98 R08: 0000000000000000 R09: 0000000000000c2e
[   64.195085] R10: 0000000000000000 R11: ffffffffba932f0d R12: 0000000000000003
[   64.195334] R13: 0000000000000000 R14: ffffffffba6807e0 R15: 0000000000000000
[   64.195585]  ? sysrq_handle_crash+0x9/0x20
[   64.195734]  __handle_sysrq+0xef/0x140
[   64.195870]  write_sysrq_trigger+0x4c/0x50
[   64.196018]  proc_reg_write+0x40/0x70
[   64.196151]  __vfs_write+0x35/0x180
[   64.196279]  ? __this_cpu_preempt_check+0x13/0x20
[   64.196447]  vfs_write+0xc5/0x1c0
[   64.196569]  ksys_write+0x53/0xc0
[   64.196690]  ? __audit_syscall_exit+0x20d/0x2a0
[   64.196853]  __x64_sys_write+0x15/0x20
[   64.196988]  do_syscall_64+0x4e/0x100
[   64.197122]  entry_SYSCALL_64_after_hwframe+0x44/0xa9
[   64.197302] RIP: 0033:0x7f3d2fca0664
[   64.197431] Code: 00 f7 d8 64 89 02 48 c7 c0 ff ff ff ff eb b3 0f 1f 80 00 00 00 00 48 8d 05 99 20 0e 00 8b 00 85 c0 75 13 b8 01 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 54 c3 0f 1f 00 41 54 49 89 d4 55 48 89 f5 53
[   64.394050] RSP: 002b:00007ffcf9071e98 EFLAGS: 00000246 ORIG_RAX: 0000000000000001
[   64.394315] RAX: ffffffffffffffda RBX: 0000000000000002 RCX: 00007f3d2fca0664
[   64.394563] RDX: 0000000000000002 RSI: 0000000000ee8190 RDI: 0000000000000001
[   64.394812] RBP: 0000000000ee8190 R08: 000000000000000a R09: 00007f3d2fb94740
[   64.395061] R10: 00000000004b8f41 R11: 0000000000000246 R12: 00007f3d2fd7e760
[   64.395309] R13: 0000000000000002 R14: 00007f3d2fd7a280 R15: 00007f3d2fd79740
[   64.395559] ---[ end trace 416678827625f99d ]---
[   64.395735] ------------[ cut here ]------------
[   64.395899] sched: Unexpected reschedule of offline CPU#2!
[   64.396095] WARNING: CPU: 3 PID: 336 at arch/x86/kernel/smp.c:141 native_smp_send_reschedule+0x3a/0x40
[   64.396417] Modules linked in:
[   64.396531] CPU: 3 PID: 336 Comm: bash Tainted: G     UD W         4.19.0-19.iot-lts2018-sos-g6b95688-dirty #44
[   64.396880] RIP: 0010:native_smp_send_reschedule+0x3a/0x40
[   64.397074] Code: c0 84 c0 74 17 48 8b 05 c4 40 54 01 be fd 00 00 00 48 8b 40 30 e8 f6 a8 dc 00 5d c3 89 fe 48 c7 c7 00 7a 45 ba e8 e6 e2 05 00 <0f> 0b 5d c3 66 90 55 48 89 e5 41 54 53 48 83 ec 20 65 48 8b 04 25
[   64.397715] RSP: 0018:ffff9939b4b83ba0 EFLAGS: 00010  64.532707]  irq_work_tick+0x55/0x60
[   64.532838]  update_process_times+0x3d/0x50
[   64.532988]  tick_sched_handle+0x24/0x60
[   64.533130]  tick_sched_timer+0x37/0x80
[   64.533270]  ? tick_sched_do_timer+0x60/0x60
[   64.533423]  __hrtimer_run_queues+0xf3/0x190
[   64.533577]  hrtimer_interrupt+0x11d/0x250
[   64.533725]  smp_apic_timer_interrupt+0x60/0xa0
[   64.533887]  apic_timer_interrupt+0xf/0x20
[   64.534034]  </IRQ>
[   64.534115] RIP: 0010:panic+0x19b/0x1f4
[   64.534253] Code: 89 01 90 69 09 b9 48 c7 c6 80 c3 92 ba 48 c7 c7 e0 e1 45 ba c7 05 15 d7 59 01 00 00 00 00 e8 ee 56 05 00 fb 66 0f 1f 44 00 00 <45> 31 e4 45 31 f6 31 db e8 9c d6 0a 00 4c 39 f3 7c 1d 41 83 f4 01
[   64.534894] RSP: 0018:ffff9939a5ed3af0 EFLAGS: 00000286 ORIG_RAX: ffffffffffffff13
[   64.535158] RAX: 0000000000000039 RBX: 0000000000000000 RCX: 0000000000000000
[   64.535407] RDX: 0000000000000000 RSI: ffffffffba47f75b RDI: 00000000ffffffff
[   64.535655] RBP: ffff9939a5ed3b68 R08: 0000000000000000 R09: 0000000000000c63
[   64.535904] R10: 000000000000064b R11: ffffffffba932f0d R12: ffffffffba450f06
[   64.536153] R13: ffff9939a5ed3b78 R14: 0000000000000046 R15: 0000000000000000
[   64.536405]  oops_end+0x9d/0xb0
[   64.536521]  no_context+0x12d/0x320
[   64.536649]  __bad_area_nosemaphore+0x7c/0x1d0
[   64.536809]  bad_area_nosemaphore+0xf/0x20
[   64.536956]  __do_page_fault+0xa7/0x570
[   64.537096]  do_page_fault+0x22/0x30
[   64.537227]  page_fault+0x1e/0x30
[   64.537349] RIP: 0010:sysrq_handle_crash+0x16/0x20
[   64.537520] Code: d5 e8 8e c2 bc ff 0f 1f 40 00 66 2e 0f 1f 84 00 00 00 00 00 55 48 89 e5 e8 77 31 c3 ff c7 05 65 9e 16 01 01 00 00 00 0f ae f8 <c6> 04 25 00 00 00 00 01 5d c3 55 48 89 e5 fb 66 0f 1f 44 00 00 e8
[   64.538164] RSP: 0018:ffff9939a5ed3d98 EFLAGS: 00010246
[   64.538349] RAX: 0000000000000000 RBX: 0000000000000063 RCX: 0000000000000000
[   64.538598] RDX: 0000000000000000 RSI: ffffffffba47f75b RDI: 0000000000000063
[   64.538846] RBP: ffff9939a5ed3d98 R08: 0000000000000000 R09: 0000000000000c2e
[   64.539095] R10: 0000000000000000 R11: ffffffffba932f0d R12: 0000000000000003
[   64.539343] R13: 0000000000000000 R14: ffffffffba6807e0 R15: 0000000000000000
[   64.539594]  ? sysrq_handle_crash+0x9/0x20
[   64.539743]  __handle_sysrq+0xef/0x140
[   64.539879]  write_sysrq_trigger+0x4c/0x50
[   64.540027]  proc_reg_write+0x40/0x70
[   64.540160]  __vfs_write+0x35/0x180
[   64.540289]  ? __this_cpu_preempt_check+0x13/0x20
[   64.540457]  vfs_write+0xc5/0x1c0
[   64.540578]  ksys_write+0x53/0xc0
[   64.540700]  ? __audit_syscall_exit+0x20d/0x2a0
[   64.540862]  __x64_sys_write+0x15/0x20
[   64.540998]  do_syscall_64+0x4e/0x100
[   64.541132]  entry_SYSCALL_64_after_hwframe+0x44/0xa9
[   64.541311] RIP: 0033:0x7f3d2fca0664
[   64.541441] Code: 00 f7 d8 64 89 02 48 c7 c0 ff ff ff ff eb b3 0f 1f 80 00 00 00 00 48 8d 05 99 20 0e 00 8b 00 85 c0 75 13 b8 01 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 54 c3 0f 1f 00 41 54 49 89 d4 55 48 89 f5 53
[   64.542080] RSP: 002b:00007ffcf9071e98 EFLAGS: 00000246 ORIG_RAX: 0000000000000001
[   64.542345] RAX: ffffffffffffffda RBX: 0000000000000002 RCX: 00007f3d2fca0664
[   64.542593] RDX: 0000000000000002 RSI: 0000000000ee8190 RDI: 0000000000000001
[   64.542842] RBP: 0000000000ee8190 R08: 000000000000000a R09: 00007f3d2fb94740
[   64.543090] R10: 00000000004b8f41 R11: 0000000000000246 R12: 00007f3d2fd7e760
[   64.543339] R13: 0000000000000002 R14: 00007f3d2fd7a280 R15: 00007f3d2fd79740
[   64.543589] ---[ end trace 416678827625f99e ]---
[   64.543775] ------------[ cut here ]------------
[   64.543942] sched: Unexpected reschedule of offline CPU#0!
[   64.544138] WARNING: CPU: 3 PID: 336 at arch/x86/kernel/smp.c:141 native_smp_send_reschedule+0x3a/0x40
[   64.544460] Modules linked in:
[   64.544573] CPU: 3 PID: 336 Comm: bash Tainted: G     UD W         4.19.0-19.iot-lts2018-sos-g6b95688-dirty #44
[   64.544923] RIP: 0010:native_smp_send_reschedule+0x3a/0x40
[   64.545117] Code: c0 84 c0 74 17 48 8b 05 c4 40 54 01 be fd 00 00 00 48 8b 40 30 e8 f6 a8 dc 00 5d c3 89 fe 48 c7 c7 00 7a 45 ba e8 e6 e2 05 00 <0f> 0b 5d c3 66 90 55 48 89 e5 41 54 53 48 83 ec 20 65 48 8b 04 25
[   64.545755] RSP: 0018:ffff9939b4b83e28 EFLAGS: 00010086
[   64.545939] RAX: 0000000000000000 RBX: 0000000000000003 RCX: 0000000000000000
[   64.546187] RDX: 0000000000000007 RSI: ffffffffba47f75b RDI: 00000000ffffffff
[   64.546434] RBP: ffff9939b4b83e28 R08: 0000000000000000 R09: 0000000000000d5e
[   64.546682] R10: 0000000000000400 R11: ffffffffba932f0d R12: 0000000000000000
[   64.546930] R13: 0000000000000003 R14: ffff9939a3650280 R15: ffff9939b4b9be00
[   64.547178] FS:  00007f3d2fb94740(0000) GS:ffff9939b4b80000(0000) knlGS:0000000000000000
[   64.547458] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[   64.547659] CR2: 0000000000000000 CR3: 000000006185d000 CR4: 00000000003406e0
[   64.547906] Call Trace:
[   64.547997]  <IRQ>
[   64.548074]  kick_ilb+0x8b/0xa0
[   64.548190]  trigger_load_balance+0xb1/0x1a0
[   64.548344]  scheduler_tick+0xa4/0xd0
[   64.548477]  update_process_times+0x42/0x50
[   64.548627]  tick_sched_handle+0x24/0x60
[   64.548769]  tick_sched_timer+0x37/0x80
[   64.548908]  ? tick_sched_do_timer+0x60/0x60
[   64.549060]  __hrtimer_run_queues+0xf3/0x190
[   64.549214]  hrtimer_interrupt+0x11d/0x250
[   64.549362]  smp_apic_timer_interrupt+0x60/0xa0
[   64.549523]  apic_timer_interrupt+0xf/0x20
[   64.549670]  </IRQ>
[   64.549750] RIP: 0010:panic+0x19b/0x1f4
[   64.549888] Code: 89 01 90 69 09 b9 48 c7 c6 80 c3 92 ba 48 c7 c7 e0 e1 45 ba c7 05 15 d7 59 01 00 00 00 00 e8 ee 56 05 00 fb 66 0f 1f 44 00 00 <45> 31 e4 45 31 f6 31 db e8 9c d6 0a 00 4c 39 f3 7c 1d 41 83 f4 01
[   64.550525] RSP: 0018:ffff9939a5ed3af0 EFLAGS: 00000286 ORIG_RAX: ffffffffffffff13
[   64.550789] RAX: 0000000000000039 RBX: 0000000000000000 RCX: 0000000000000000
[   64.551036] RDX: 0000000000000000 RSI: ffffffffba47f75b RDI: 00000000ffffffff
[   64.551284] RBP: ffff9939a5ed3b68 R08: 0000000000000000 R09: 0000000000000c63
[   64.551532] R10: 000000000000064b R11: ffffffffba932f0d R12: ffffffffba450f06
[   64.551779] R13: ffff9939a5ed3b78 R14: 0000000000000046 R15: 0000000000000000
[   64.552031]  oops_end+0x9d/0xb0
[   64.552146]  no_context+0x12d/0x320
[   64.552274]  __bad_area_nosemaphore+0x7c/0x1d0
[   64.552433]  bad_area_nosemaphore+0xf/0x20
[   64.552580]  __do_page_fault+0xa7/0x570
[   64.552720]  do_page_fault+0x22/0x30
[   64.552850]  page_fault+0x1e/0x30
[   64.552972] RIP: 0010:sysrq_handle_crash+0x16/0x20
[   64.553141] Code: d5 e8 8e c2 bc ff 0f 1f 40 00 66 2e 0f 1f 84 00 00 00 00 00 55 48 89 e5 e8 77 31 c3 ff c7 05 65 9e 16 01 01 00 00 00 0f ae f8 <c6> 04 25 00 00 00 00 01 5d c3 55 48 89 e5 fb 66 0f 1f 44 00 00 e8
[   64.553779] RSP: 0018:ffff9939a5ed3d98 EFLAGS: 00010246
[   64.553964] RAX: 0000000000000000 RBX: 0000000000000063 RCX: 0000000000000000
[   64.554212] RDX: 0000000000000000 RSI: ffffffffba47f75b RDI: 0000000000000063
[   64.554460] RBP: ffff9939a5ed3d98 R08: 0000000000000000 R09: 0000000000000c2e
[   64.554709] R10: 0000000000000000 R11: ffffffffba932f0d R12: 0000000000000003
[   64.554957] R13: 0000000000000000 R14: ffffffffba6807e0 R15: 0000000000000000
[   64.555207]  ? sysrq_handle_crash+0x9/0x20
[   64.555355]  __handle_sysrq+0xef/0x140
[   64.555491]  write_sysrq_trigger+0x4c/0x50
[   64.555638]  proc_reg_write+0x40/0x70
[   64.555771]  __vfs_write+0x35/0x180
[   64.555898]  ? __this_cpu_preempt_check+0x13/0x20
[   64.556066]  vfs_write+0xc5/0x1c0
[   64.556187]  ksys_write+0x53/0xc0
[   64.556308]  ? __audit_syscall_exit+0x20d/0x2a0
[   64.556470]  __x64_sys_write+0x15/0x20
[   64.556606]  do_syscall_64+0x4e/0x100
[   64.556739]  entry_SYSCALL_64_after_hwframe+0x44/0xa9
[   64.556918] RIP: 0033:0x7f3d2fca0664
[   64.557047] Code: 00 f7 d8 64 89 02 48 c7 c0 ff ff ff ff eb b3 0f 1f 80 00 00 00 00 48 8d 05 99 20 0e 00 8b 00 85 c0 75 13 b8 01 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 54 c3 0f 1f 00 41 54 49 89 d4 55 48 89 f5 53
[   64.557685] RSP: 002b:00007ffcf9071e98 EFLAGS: 00000246 ORIG_RAX: 0000000000000001
[   65.374953] RAX: ffffffffffffffda RBX: 0000000000000002 RCX: 00007f3d2fca0664
[   65.376235] RDX: 0000000000000002 RSI: 0000000000ee8190 RDI: 0000000000000001
[   65.376482] RBP: 0000000000ee8190 R08: 000000000000000a R09: 00007f3d2fb94740
[   65.376730] R10: 00000000004b8f41 R11: 0000000000000246 R12: 00007f3d2fd7e760
[   65.376978] R13: 0000000000000002 R14: 00007f3d2fd7a280 R15: 00007f3d2fd79740
[   65.377228] ---[ end trace 416678827625f99f ]---

      reply	other threads:[~2018-11-27  3:25 UTC|newest]

Thread overview: 7+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2018-10-11  7:17 [PATCH RFC] panic: Avoid extra noisy messages due to stopped cpus Feng Tang
2018-10-11  9:35 ` Peter Zijlstra
2018-10-11  9:59   ` Feng Tang
2018-10-22  9:55     ` Feng Tang
2018-11-08 13:05       ` [PATCH v2] panic: Avoid the extra noise dmesg Feng Tang
2018-11-26 23:59         ` Andrew Morton
2018-11-27  3:32           ` Feng Tang [this message]

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20181127033256.2xvjvjhmqu3hqhpm@shbuild888 \
    --to=feng.tang@intel.com \
    --cc=akpm@linux-foundation.org \
    --cc=bp@alien8.de \
    --cc=hpa@linux.intel.com \
    --cc=keescook@chromium.org \
    --cc=linux-kernel@vger.kernel.org \
    --cc=mingo@kernel.org \
    --cc=peterz@infradead.org \
    --cc=tglx@linutronix.de \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.