LKML Archive on lore.kernel.org
 help / color / Atom feed
From: Sasha Levin <levinsasha928@gmail.com>
To: Peter Zijlstra <peterz@infradead.org>
Cc: Ingo Molnar <mingo@kernel.org>,
	Thomas Gleixner <tglx@linutronix.de>,
	"linux-kernel@vger.kernel.org List"
	<linux-kernel@vger.kernel.org>,
	alexander.levin@verizon.com
Subject: Re: [PATCH 3/4] sched: WARN when migrating to an offline CPU
Date: Thu, 28 Sep 2017 02:14:15 -0700
Message-ID: <CA+1xoqe0AW9acyvdtE+=CN8FisQrXH2wnwvakSXapg3sKE4khw@mail.gmail.com> (raw)
In-Reply-To: <20170907150614.094206976@infradead.org>

On Thu, Sep 7, 2017 at 8:03 AM, Peter Zijlstra <peterz@infradead.org> wrote:
> Migrating tasks to offline CPUs is a pretty big fail, warn about it.

Hey Peter,

This seems to get hit on the latest -next:

[2035565360.446794] Unregister pv shared memory for cpu 2
[2035565360.467930] numa_remove_cpu cpu 2 node 2: mask now 6
[2035565360.471431] ------------[ cut here ]------------
[2035565360.472548] WARNING: CPU: 2 PID: 24 at
kernel/sched/core.c:1178 set_task_cpu (kernel/sched/core.c:1157)
[2035565360.473840] Modules linked in:
[2035565360.474632] CPU: 2 PID: 24 Comm: migration/2 Not tainted
4.14.0-rc2-next-20170927+ #252
[2035565360.476128] task: ffff8801ab7dc040 task.stack: ffff8801ab7e0000
[2035565360.477108] RIP: 0010:set_task_cpu (??:?)
[2035565360.477850] RSP: 0018:ffff8801ab807b18 EFLAGS: 00010096
[2035565360.478734] RAX: 0000000000000002 RBX: ffff8801a8d94040 RCX:
1ffff1003577cb3c
[2035565360.479872] RDX: 0000000000000004 RSI: 0000000000000002 RDI:
ffffffff9177d940
[2035565360.480998] RBP: ffff8801ab807b58 R08: 000000000000b6b2 R09:
ffff8801abbe5a2c
[2035565360.482356] R10: 0000000000000001 R11: 0000000000000001 R12:
ffff8801a8d94048
[2035565360.483490] R13: ffff8801ab807c88 R14: 0000000000000002 R15:
ffff8801a8d940a4
[2035565360.484617] FS:  0000000000000000(0000)
GS:ffff8801ab800000(0000) knlGS:0000000000000000
[2035565360.485881] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[2035565360.490482] CR2: 00000000012dfae4 CR3: 00000001a30c5002 CR4:
00000000000606a0
[2035565360.491836] Call Trace:
[2035565360.492301]  <IRQ>
[2035565360.492710] detach_task.isra.80 (kernel/sched/fair.c:6816)
[2035565360.493430] load_balance (./include/linux/list.h:78
kernel/sched/fair.c:6902 kernel/sched/fair.c:8201)
[2035565360.494094] ? find_busiest_group (kernel/sched/fair.c:8133)
[2035565360.494861] ? rebalance_domains (kernel/sched/fair.c:8785)
[2035565360.495571] rebalance_domains (kernel/sched/fair.c:8829)
[2035565360.496292] ? find_next_bit (lib/find_bit.c:65)
[2035565360.496961] ? pick_next_task_fair (kernel/sched/fair.c:8779)
[2035565360.497753] ? check_preemption_disabled (lib/smp_processor_id.c:52)
[2035565360.498574] run_rebalance_domains (kernel/sched/fair.c:9052)
[2035565360.499318] ? check_preemption_disabled (lib/smp_processor_id.c:52)
[2035565360.500388] __do_softirq (kernel/softirq.c:284
./include/linux/jump_label.h:141 ./include/trace/events/irq.h:141
kernel/softirq.c:285)
[2035565360.501039] ? hrtimer_interrupt (kernel/time/hrtimer.c:1370)
[2035565360.502014] irq_exit (kernel/softirq.c:364 kernel/softirq.c:405)
[2035565360.502598] smp_apic_timer_interrupt
(./arch/x86/include/asm/irq_regs.h:26
arch/x86/kernel/apic/apic.c:1043)
[2035565360.503605] apic_timer_interrupt (arch/x86/entry/entry_64.S:770)
[2035565360.504313]  </IRQ>
[2035565360.504724] RIP: 0010:multi_cpu_stop (??:?)
[2035565360.505503] RSP: 0018:ffff8801ab7e7d88 EFLAGS: 00000292
ORIG_RAX: ffffffffffffff11
[2035565360.506721] RAX: 0000000000000003 RBX: 0000000000000004 RCX:
000000000000000a
[2035565360.507862] RDX: 0000000000000004 RSI: 0000000000000040 RDI:
0000000000000292
[2035565360.508997] RBP: ffff8801ab7e7de0 R08: ffff8801ab7dc040 R09:
0000000000000000
[2035565360.510566] R10: 0000000000000001 R11: 1ffff1003577cb55 R12:
ffff8801aae2f890
[2035565360.511701] R13: ffff8801aae2f8b4 R14: dffffc0000000000 R15:
0000000000000003
[2035565360.512843] ? cpu_stop_queue_work (kernel/stop_machine.c:176)
[2035565360.513591] cpu_stopper_thread (kernel/stop_machine.c:480)
[2035565360.514316] ? cpu_stop_create (kernel/stop_machine.c:458)
[2035565360.515218] smpboot_thread_fn (kernel/smpboot.c:164)
[2035565360.515918] ? sort_range (kernel/smpboot.c:107)
[2035565360.516528] ? schedule (./arch/x86/include/asm/bitops.h:324
(discriminator 1) ./include/linux/thread_info.h:79 (discriminator 1)
./include/linux/sched.h:1605 (discriminator 1)
kernel/sched/core.c:3435 (discriminator 1))
[2035565360.517119] ? __kthread_parkme (kernel/kthread.c:188)
[2035565360.517835] kthread (kernel/kthread.c:242)
[2035565360.518413] ? sort_range (kernel/smpboot.c:107)
[2035565360.519008] ? kthread_create_on_node (kernel/kthread.c:198)
[2035565360.519792] ret_from_fork (arch/x86/entry/entry_64.S:437)
[2035565360.520407] Code: 09 84 d2 74 05 e8 94 38 4b 00 f7 83 84 00 00
00 fd ff ff ff 0f 84 a0 f9 ff ff 0f ff e9 99 f9 ff ff e8 68 18 ff ff
e9 c7 fd ff ff <0f> ff e9 ea f9 ff ff e8 57 18 ff ff e9 f5 f9 ff ff e8
0d 84 0f
All code
========
   0: 09 84 d2 74 05 e8 94 or     %eax,-0x6b17fa8c(%rdx,%rdx,8)
   7: 38 4b 00             cmp    %cl,0x0(%rbx)
   a: f7 83 84 00 00 00 fd testl  $0xfffffffd,0x84(%rbx)
  11: ff ff ff
  14: 0f 84 a0 f9 ff ff     je     0xfffffffffffff9ba
  1a: 0f ff                 (bad)
  1c: e9 99 f9 ff ff       jmpq   0xfffffffffffff9ba
  21: e8 68 18 ff ff       callq  0xffffffffffff188e
  26: e9 c7 fd ff ff       jmpq   0xfffffffffffffdf2
  2b:* 0f ff                 (bad)   <-- trapping instruction
  2d: e9 ea f9 ff ff       jmpq   0xfffffffffffffa1c
  32: e8 57 18 ff ff       callq  0xffffffffffff188e
  37: e9 f5 f9 ff ff       jmpq   0xfffffffffffffa31
  3c: e8 0d 84 0f 00       callq  0xf844e

Code starting with the faulting instruction
===========================================
   0: 0f ff                 (bad)
   2: e9 ea f9 ff ff       jmpq   0xfffffffffffff9f1
   7: e8 57 18 ff ff       callq  0xffffffffffff1863
   c: e9 f5 f9 ff ff       jmpq   0xfffffffffffffa06
  11: e8 0d 84 0f 00       callq  0xf8423
[2035565360.523585] ---[ end trace 812b7ff74114db6c ]---
[2035565360.524744] smpboot: CPU 2 is now offline


Thanks,
Sasha

  parent reply index

Thread overview: 18+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2017-09-07 15:03 [PATCH 0/4] sched: Fix some load-balancer vs hotplug holes Peter Zijlstra
2017-09-07 15:03 ` [PATCH 1/4] sched/fair: Avoid newidle balance for !active CPUs Peter Zijlstra
2017-09-12 18:04   ` [tip:sched/urgent] " tip-bot for Peter Zijlstra
2017-09-07 15:03 ` [PATCH 2/4] sched/fair: Plug hole between hotplug and active load_balance Peter Zijlstra
2017-09-12 18:05   ` [tip:sched/urgent] sched/fair: Plug hole between hotplug and active_load_balance() tip-bot for Peter Zijlstra
2017-09-07 15:03 ` [PATCH 3/4] sched: WARN when migrating to an offline CPU Peter Zijlstra
2017-09-12 18:05   ` [tip:sched/urgent] sched/core: WARN() " tip-bot for Peter Zijlstra
2017-09-28  9:14   ` Sasha Levin [this message]
2017-09-28 10:35     ` [PATCH 3/4] sched: WARN " Peter Zijlstra
2017-09-28 11:03       ` Levin, Alexander (Sasha Levin)
2017-09-28 11:42         ` Peter Zijlstra
2017-09-29 11:11           ` Peter Zijlstra
2017-10-07  2:07             ` Levin, Alexander (Sasha Levin)
2017-10-07  9:15               ` Peter Zijlstra
     [not found]                 ` <20171007174327.ky6g5viokxg5ysdm@sasha-lappy>
2017-10-09  8:04                   ` Peter Zijlstra
2017-10-10  1:18                     ` Levin, Alexander (Sasha Levin)
2017-09-07 15:03 ` [PATCH 4/4] sched/debug: Add debugfs knob for "sched_debug" Peter Zijlstra
2017-09-12 18:05   ` [tip:sched/urgent] " tip-bot for Peter Zijlstra

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to='CA+1xoqe0AW9acyvdtE+=CN8FisQrXH2wnwvakSXapg3sKE4khw@mail.gmail.com' \
    --to=levinsasha928@gmail.com \
    --cc=alexander.levin@verizon.com \
    --cc=linux-kernel@vger.kernel.org \
    --cc=mingo@kernel.org \
    --cc=peterz@infradead.org \
    --cc=tglx@linutronix.de \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link

LKML Archive on lore.kernel.org

Archives are clonable:
	git clone --mirror https://lore.kernel.org/lkml/0 lkml/git/0.git
	git clone --mirror https://lore.kernel.org/lkml/1 lkml/git/1.git
	git clone --mirror https://lore.kernel.org/lkml/2 lkml/git/2.git
	git clone --mirror https://lore.kernel.org/lkml/3 lkml/git/3.git
	git clone --mirror https://lore.kernel.org/lkml/4 lkml/git/4.git
	git clone --mirror https://lore.kernel.org/lkml/5 lkml/git/5.git
	git clone --mirror https://lore.kernel.org/lkml/6 lkml/git/6.git
	git clone --mirror https://lore.kernel.org/lkml/7 lkml/git/7.git
	git clone --mirror https://lore.kernel.org/lkml/8 lkml/git/8.git

	# If you have public-inbox 1.1+ installed, you may
	# initialize and index your mirror using the following commands:
	public-inbox-init -V2 lkml lkml/ https://lore.kernel.org/lkml \
		linux-kernel@vger.kernel.org
	public-inbox-index lkml

Example config snippet for mirrors

Newsgroup available over NNTP:
	nntp://nntp.lore.kernel.org/org.kernel.vger.linux-kernel


AGPL code for this site: git clone https://public-inbox.org/public-inbox.git