LKML Archive on lore.kernel.org
 help / color / Atom feed
From: Guenter Roeck <linux@roeck-us.net>
To: x86@kernel.org
Cc: Ingo Molnar <mingo@redhat.com>,
	Thomas Gleixner <tglx@linutronix.de>,
	linux-kernel@vger.kernel.org, Borislav Petkov <bp@alien8.de>
Subject: sched: Unexpected reschedule of offline CPU#2!
Date: Sat, 27 Jul 2019 09:44:50 -0700
Message-ID: <20190727164450.GA11726@roeck-us.net> (raw)

Hi,

I see the following traceback (or similar tracebacks) once in a while
during my boot tests. In this specific case it is with mainline
(v5.3-rc1-195-g3ea54d9b0d65), but I have seen it with other branches
as well. This isn't a new problem; I have seen it for quite some time.
There is no specific action required to make it appear; just running
reboot loops is sufficient. The problem doesn't happen a lot;
non-scientifically I would say I see it maybe once every few hundred
boots.

No specific action requested or asked for; this is just informational.

A complete log is at:
https://kerneltests.org/builders/qemu-x86-master/builds/1285/steps/qemubuildcommand/logs/stdio

Guenter

---
[   61.248329] sd 0:0:0:0: [sda] Synchronizing SCSI cache
[   61.268277] e1000e: EEE TX LPI TIMER: 00000000
[   61.311435] reboot: Restarting system
[   61.312321] reboot: machine restart
[   61.342193] ------------[ cut here ]------------
[   61.342660] sched: Unexpected reschedule of offline CPU#2!
ILLOPC: ce241f83: 0f 0b
[   61.344323] WARNING: CPU: 1 PID: 15 at arch/x86/kernel/smp.c:126 native_smp_send_reschedule+0x33/0x40
[   61.344836] Modules linked in:
[   61.345694] CPU: 1 PID: 15 Comm: ksoftirqd/1 Not tainted 5.3.0-rc1+ #1
[   61.345998] Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS rel-1.12.1-0-ga5cab58e9a3f-prebuilt.qemu.org 04/01/2014
[   61.346569] EIP: native_smp_send_reschedule+0x33/0x40
[   61.347099] Code: cf 73 1c 8b 15 60 54 2b cf 8b 4a 18 ba fd 00 00 00 e8 05 65 c7 00 c9 c3 8d b4 26 00 00 00 00 50 68 04 ca 1a cf e8 fe e3 01 00 <0f> 0b 58 5a c9 c3 8d b4 26 00 00 00 00 55 89 e5 56 53 83 ec 0c 65
[   61.347726] EAX: 0000002e EBX: 00000002 ECX: 00000000 EDX: cdd64140
[   61.347977] ESI: 00000002 EDI: 00000000 EBP: cdd73c88 ESP: cdd73c80
[   61.348234] DS: 007b ES: 007b FS: 00d8 GS: 00e0 SS: 0068 EFLAGS: 00000096
[   61.348514] CR0: 80050033 CR2: b7ee7048 CR3: 0c28f000 CR4: 000006d0
[   61.348866] Call Trace:
[   61.349392]  kick_ilb+0x90/0xa0
[   61.349629]  trigger_load_balance+0xf0/0x5c0
[   61.349859]  ? check_preempt_wakeup+0x1b0/0x1b0
[   61.350057]  scheduler_tick+0xa7/0xd0
[   61.350266]  update_process_times+0x4a/0x60
[   61.350467]  tick_sched_handle+0x3e/0x50
[   61.350650]  tick_sched_timer+0x37/0x90
[   61.350847]  __hrtimer_run_queues+0xf7/0x440
[   61.351056]  ? tick_sched_do_timer+0x70/0x70
[   61.351281]  hrtimer_interrupt+0x10e/0x260
[   61.351541]  smp_apic_timer_interrupt+0x68/0x210
[   61.351750]  apic_timer_interrupt+0x106/0x10c
[   61.352040] EIP: _raw_spin_unlock_irqrestore+0x47/0x50
[   61.352254] Code: 66 40 ff f6 c7 02 75 1b 53 9d e8 c4 67 49 ff 64 ff 0d 84 27 50 cf 5b 5e 5d c3 8d b4 26 00 00 00 00 66 90 e8 ab 69 49 ff 53 9d <eb> e3 8d b4 26 00 00 00 00 55 64 ff 05 84 27 50 cf 89 e5 53 89 c3
[   61.352810] EAX: cdd64140 EBX: 00000282 ECX: 00000003 EDX: 00000002
[   61.353041] ESI: cdc01940 EDI: 00000001 EBP: cdd73e08 ESP: cdd73e00
[   61.353273] DS: 007b ES: 007b FS: 00d8 GS: 00e0 SS: 0068 EFLAGS: 00000282
[   61.353705]  ? _raw_spin_unlock_irqrestore+0x47/0x50
[   61.362142]  free_debug_processing+0x199/0x220
[   61.362413]  __slab_free+0x220/0x3b0
[   61.362599]  ? irq_kobj_release+0x1c/0x20
[   61.362845]  ? kfree+0x1ad/0x270
[   61.363002]  ? kfree+0x1ad/0x270
[   61.363162]  kfree+0x264/0x270
[   61.363305]  ? kfree+0x264/0x270
[   61.363458]  ? irq_kobj_release+0x1c/0x20
[   61.363624]  ? irq_kobj_release+0x1c/0x20
[   61.363824]  irq_kobj_release+0x1c/0x20
[   61.364018]  kobject_put+0x58/0xc0
[   61.364211]  ? hwirq_show+0x50/0x50
[   61.364439]  delayed_free_desc+0xb/0x10
[   61.364621]  rcu_core+0x288/0xb50
[   61.364805]  ? __do_softirq+0x7e/0x3bb
[   61.365042]  rcu_core_si+0x8/0x10
[   61.365209]  __do_softirq+0xa9/0x3bb
[   61.365445]  run_ksoftirqd+0x25/0x50
[   61.365615]  smpboot_thread_fn+0xef/0x1d0
[   61.365834]  kthread+0xf2/0x110
[   61.365986]  ? sort_range+0x20/0x20
[   61.366156]  ? kthread_create_on_node+0x20/0x20
[   61.366360]  ret_from_fork+0x2e/0x38
[   61.366818] irq event stamp: 1267
[   61.367115] hardirqs last  enabled at (1266): [<ceeb37f5>] _raw_spin_unlock_irqrestore+0x45/0x50
[   61.367448] hardirqs last disabled at (1267): [<ce20178a>] trace_hardirqs_off_thunk+0xc/0x12
[   61.367769] softirqs last  enabled at (1232): [<ceeb7a45>] __do_softirq+0x2c5/0x3bb
[   61.368057] softirqs last disabled at (1237): [<ce267605>] run_ksoftirqd+0x25/0x50
[   61.368389] ---[ end trace 3465d631a21844b8 ]---

             reply index

Thread overview: 10+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2019-07-27 16:44 Guenter Roeck [this message]
2019-07-29  9:35 ` Peter Zijlstra
2019-07-29  9:58   ` Thomas Gleixner
2019-07-29 10:13     ` Peter Zijlstra
2019-07-29 10:38       ` Thomas Gleixner
2019-07-29 10:47         ` Peter Zijlstra
2019-07-29 20:50           ` Guenter Roeck
2019-08-16 10:22             ` Thomas Gleixner
2019-08-16 19:32               ` Guenter Roeck
2019-08-17 20:21                 ` Thomas Gleixner

Reply instructions:

You may reply publically to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20190727164450.GA11726@roeck-us.net \
    --to=linux@roeck-us.net \
    --cc=bp@alien8.de \
    --cc=linux-kernel@vger.kernel.org \
    --cc=mingo@redhat.com \
    --cc=tglx@linutronix.de \
    --cc=x86@kernel.org \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link

LKML Archive on lore.kernel.org

Archives are clonable:
	git clone --mirror https://lore.kernel.org/lkml/0 lkml/git/0.git
	git clone --mirror https://lore.kernel.org/lkml/1 lkml/git/1.git
	git clone --mirror https://lore.kernel.org/lkml/2 lkml/git/2.git
	git clone --mirror https://lore.kernel.org/lkml/3 lkml/git/3.git
	git clone --mirror https://lore.kernel.org/lkml/4 lkml/git/4.git
	git clone --mirror https://lore.kernel.org/lkml/5 lkml/git/5.git
	git clone --mirror https://lore.kernel.org/lkml/6 lkml/git/6.git
	git clone --mirror https://lore.kernel.org/lkml/7 lkml/git/7.git
	git clone --mirror https://lore.kernel.org/lkml/8 lkml/git/8.git

	# If you have public-inbox 1.1+ installed, you may
	# initialize and index your mirror using the following commands:
	public-inbox-init -V2 lkml lkml/ https://lore.kernel.org/lkml \
		linux-kernel@vger.kernel.org
	public-inbox-index lkml

Example config snippet for mirrors

Newsgroup available over NNTP:
	nntp://nntp.lore.kernel.org/org.kernel.vger.linux-kernel


AGPL code for this site: git clone https://public-inbox.org/public-inbox.git