From: Guenter Roeck <linux@roeck-us.net> To: x86@kernel.org Cc: Ingo Molnar <mingo@redhat.com>, Thomas Gleixner <tglx@linutronix.de>, linux-kernel@vger.kernel.org, Borislav Petkov <bp@alien8.de> Subject: sched: Unexpected reschedule of offline CPU#2! Date: Sat, 27 Jul 2019 09:44:50 -0700 Message-ID: <20190727164450.GA11726@roeck-us.net> (raw) Hi, I see the following traceback (or similar tracebacks) once in a while during my boot tests. In this specific case it is with mainline (v5.3-rc1-195-g3ea54d9b0d65), but I have seen it with other branches as well. This isn't a new problem; I have seen it for quite some time. There is no specific action required to make it appear; just running reboot loops is sufficient. The problem doesn't happen a lot; non-scientifically I would say I see it maybe once every few hundred boots. No specific action requested or asked for; this is just informational. A complete log is at: https://kerneltests.org/builders/qemu-x86-master/builds/1285/steps/qemubuildcommand/logs/stdio Guenter --- [ 61.248329] sd 0:0:0:0: [sda] Synchronizing SCSI cache [ 61.268277] e1000e: EEE TX LPI TIMER: 00000000 [ 61.311435] reboot: Restarting system [ 61.312321] reboot: machine restart [ 61.342193] ------------[ cut here ]------------ [ 61.342660] sched: Unexpected reschedule of offline CPU#2! ILLOPC: ce241f83: 0f 0b [ 61.344323] WARNING: CPU: 1 PID: 15 at arch/x86/kernel/smp.c:126 native_smp_send_reschedule+0x33/0x40 [ 61.344836] Modules linked in: [ 61.345694] CPU: 1 PID: 15 Comm: ksoftirqd/1 Not tainted 5.3.0-rc1+ #1 [ 61.345998] Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS rel-1.12.1-0-ga5cab58e9a3f-prebuilt.qemu.org 04/01/2014 [ 61.346569] EIP: native_smp_send_reschedule+0x33/0x40 [ 61.347099] Code: cf 73 1c 8b 15 60 54 2b cf 8b 4a 18 ba fd 00 00 00 e8 05 65 c7 00 c9 c3 8d b4 26 00 00 00 00 50 68 04 ca 1a cf e8 fe e3 01 00 <0f> 0b 58 5a c9 c3 8d b4 26 00 00 00 00 55 89 e5 56 53 83 ec 0c 65 [ 61.347726] EAX: 0000002e EBX: 00000002 ECX: 00000000 EDX: cdd64140 [ 61.347977] ESI: 00000002 EDI: 00000000 EBP: cdd73c88 ESP: cdd73c80 [ 61.348234] DS: 007b ES: 007b FS: 00d8 GS: 00e0 SS: 0068 EFLAGS: 00000096 [ 61.348514] CR0: 80050033 CR2: b7ee7048 CR3: 0c28f000 CR4: 000006d0 [ 61.348866] Call Trace: [ 61.349392] kick_ilb+0x90/0xa0 [ 61.349629] trigger_load_balance+0xf0/0x5c0 [ 61.349859] ? check_preempt_wakeup+0x1b0/0x1b0 [ 61.350057] scheduler_tick+0xa7/0xd0 [ 61.350266] update_process_times+0x4a/0x60 [ 61.350467] tick_sched_handle+0x3e/0x50 [ 61.350650] tick_sched_timer+0x37/0x90 [ 61.350847] __hrtimer_run_queues+0xf7/0x440 [ 61.351056] ? tick_sched_do_timer+0x70/0x70 [ 61.351281] hrtimer_interrupt+0x10e/0x260 [ 61.351541] smp_apic_timer_interrupt+0x68/0x210 [ 61.351750] apic_timer_interrupt+0x106/0x10c [ 61.352040] EIP: _raw_spin_unlock_irqrestore+0x47/0x50 [ 61.352254] Code: 66 40 ff f6 c7 02 75 1b 53 9d e8 c4 67 49 ff 64 ff 0d 84 27 50 cf 5b 5e 5d c3 8d b4 26 00 00 00 00 66 90 e8 ab 69 49 ff 53 9d <eb> e3 8d b4 26 00 00 00 00 55 64 ff 05 84 27 50 cf 89 e5 53 89 c3 [ 61.352810] EAX: cdd64140 EBX: 00000282 ECX: 00000003 EDX: 00000002 [ 61.353041] ESI: cdc01940 EDI: 00000001 EBP: cdd73e08 ESP: cdd73e00 [ 61.353273] DS: 007b ES: 007b FS: 00d8 GS: 00e0 SS: 0068 EFLAGS: 00000282 [ 61.353705] ? _raw_spin_unlock_irqrestore+0x47/0x50 [ 61.362142] free_debug_processing+0x199/0x220 [ 61.362413] __slab_free+0x220/0x3b0 [ 61.362599] ? irq_kobj_release+0x1c/0x20 [ 61.362845] ? kfree+0x1ad/0x270 [ 61.363002] ? kfree+0x1ad/0x270 [ 61.363162] kfree+0x264/0x270 [ 61.363305] ? kfree+0x264/0x270 [ 61.363458] ? irq_kobj_release+0x1c/0x20 [ 61.363624] ? irq_kobj_release+0x1c/0x20 [ 61.363824] irq_kobj_release+0x1c/0x20 [ 61.364018] kobject_put+0x58/0xc0 [ 61.364211] ? hwirq_show+0x50/0x50 [ 61.364439] delayed_free_desc+0xb/0x10 [ 61.364621] rcu_core+0x288/0xb50 [ 61.364805] ? __do_softirq+0x7e/0x3bb [ 61.365042] rcu_core_si+0x8/0x10 [ 61.365209] __do_softirq+0xa9/0x3bb [ 61.365445] run_ksoftirqd+0x25/0x50 [ 61.365615] smpboot_thread_fn+0xef/0x1d0 [ 61.365834] kthread+0xf2/0x110 [ 61.365986] ? sort_range+0x20/0x20 [ 61.366156] ? kthread_create_on_node+0x20/0x20 [ 61.366360] ret_from_fork+0x2e/0x38 [ 61.366818] irq event stamp: 1267 [ 61.367115] hardirqs last enabled at (1266): [<ceeb37f5>] _raw_spin_unlock_irqrestore+0x45/0x50 [ 61.367448] hardirqs last disabled at (1267): [<ce20178a>] trace_hardirqs_off_thunk+0xc/0x12 [ 61.367769] softirqs last enabled at (1232): [<ceeb7a45>] __do_softirq+0x2c5/0x3bb [ 61.368057] softirqs last disabled at (1237): [<ce267605>] run_ksoftirqd+0x25/0x50 [ 61.368389] ---[ end trace 3465d631a21844b8 ]---
next reply index Thread overview: 10+ messages / expand[flat|nested] mbox.gz Atom feed top 2019-07-27 16:44 Guenter Roeck [this message] 2019-07-29 9:35 ` Peter Zijlstra 2019-07-29 9:58 ` Thomas Gleixner 2019-07-29 10:13 ` Peter Zijlstra 2019-07-29 10:38 ` Thomas Gleixner 2019-07-29 10:47 ` Peter Zijlstra 2019-07-29 20:50 ` Guenter Roeck 2019-08-16 10:22 ` Thomas Gleixner 2019-08-16 19:32 ` Guenter Roeck 2019-08-17 20:21 ` Thomas Gleixner
Reply instructions: You may reply publicly to this message via plain-text email using any one of the following methods: * Save the following mbox file, import it into your mail client, and reply-to-all from there: mbox Avoid top-posting and favor interleaved quoting: https://en.wikipedia.org/wiki/Posting_style#Interleaved_style * Reply using the --to, --cc, and --in-reply-to switches of git-send-email(1): git send-email \ --in-reply-to=20190727164450.GA11726@roeck-us.net \ --to=linux@roeck-us.net \ --cc=bp@alien8.de \ --cc=linux-kernel@vger.kernel.org \ --cc=mingo@redhat.com \ --cc=tglx@linutronix.de \ --cc=x86@kernel.org \ /path/to/YOUR_REPLY https://kernel.org/pub/software/scm/git/docs/git-send-email.html * If your mail client supports setting the In-Reply-To header via mailto: links, try the mailto: link
LKML Archive on lore.kernel.org Archives are clonable: git clone --mirror https://lore.kernel.org/lkml/0 lkml/git/0.git git clone --mirror https://lore.kernel.org/lkml/1 lkml/git/1.git git clone --mirror https://lore.kernel.org/lkml/2 lkml/git/2.git git clone --mirror https://lore.kernel.org/lkml/3 lkml/git/3.git git clone --mirror https://lore.kernel.org/lkml/4 lkml/git/4.git git clone --mirror https://lore.kernel.org/lkml/5 lkml/git/5.git git clone --mirror https://lore.kernel.org/lkml/6 lkml/git/6.git git clone --mirror https://lore.kernel.org/lkml/7 lkml/git/7.git git clone --mirror https://lore.kernel.org/lkml/8 lkml/git/8.git git clone --mirror https://lore.kernel.org/lkml/9 lkml/git/9.git # If you have public-inbox 1.1+ installed, you may # initialize and index your mirror using the following commands: public-inbox-init -V2 lkml lkml/ https://lore.kernel.org/lkml \ linux-kernel@vger.kernel.org public-inbox-index lkml Example config snippet for mirrors Newsgroup available over NNTP: nntp://nntp.lore.kernel.org/org.kernel.vger.linux-kernel AGPL code for this site: git clone https://public-inbox.org/public-inbox.git