linux-kernel.vger.kernel.org archive mirror
 help / color / mirror / Atom feed
* [PATCH 0/4] sched: Fix some load-balancer vs hotplug holes
@ 2017-09-07 15:03 Peter Zijlstra
  2017-09-07 15:03 ` [PATCH 1/4] sched/fair: Avoid newidle balance for !active CPUs Peter Zijlstra
                   ` (3 more replies)
  0 siblings, 4 replies; 18+ messages in thread
From: Peter Zijlstra @ 2017-09-07 15:03 UTC (permalink / raw)
  To: mingo, tglx; +Cc: linux-kernel, peterz

Hi

Thomas had some hotplug stress funnies and these appear to cure things. Should
go into this merge window thingy, except the last one, which is fine for later.

The first is an obvious regression and has a Fixes line, the second
looks like a pre-existing hole, but might be made easier to hit,
although I can't readily tell you how.

And while I could readily hit the first hole, I didn't manage to trigger
the second, but Thomas had enough trace data to confirm he could.

^ permalink raw reply	[flat|nested] 18+ messages in thread

* [PATCH 1/4] sched/fair: Avoid newidle balance for !active CPUs
  2017-09-07 15:03 [PATCH 0/4] sched: Fix some load-balancer vs hotplug holes Peter Zijlstra
@ 2017-09-07 15:03 ` Peter Zijlstra
  2017-09-12 18:04   ` [tip:sched/urgent] " tip-bot for Peter Zijlstra
  2017-09-07 15:03 ` [PATCH 2/4] sched/fair: Plug hole between hotplug and active load_balance Peter Zijlstra
                   ` (2 subsequent siblings)
  3 siblings, 1 reply; 18+ messages in thread
From: Peter Zijlstra @ 2017-09-07 15:03 UTC (permalink / raw)
  To: mingo, tglx; +Cc: linux-kernel, peterz

[-- Attachment #1: peterz-sched-newidle-hole.patch --]
[-- Type: text/plain, Size: 1208 bytes --]

On CPU hot unplug, when parking the last kthread we'll try and
schedule into idle to kill the CPU. This last schedule can (and does)
trigger newidle balance because at this point the sched domains are
still up because of commit:

  77d1dfda0e79 ("sched/topology, cpuset: Avoid spurious/wrong domain rebuilds")

Obviously pulling tasks to an already offline CPU is a bad idea, and
all balancing operations _should_ be subject to cpu_active_mask, make
it so.

Reported-by: Thomas Gleixner <tglx@linutronix.de>
Fixes: 77d1dfda0e79 ("sched/topology, cpuset: Avoid spurious/wrong domain rebuilds")
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
---
 kernel/sched/fair.c |    6 ++++++
 1 file changed, 6 insertions(+)

--- a/kernel/sched/fair.c
+++ b/kernel/sched/fair.c
@@ -8448,6 +8448,12 @@ static int idle_balance(struct rq *this_
 	this_rq->idle_stamp = rq_clock(this_rq);
 
 	/*
+	 * Do not pull tasks towards !active CPUs...
+	 */
+	if (!cpu_active(this_cpu))
+		return 0;
+
+	/*
 	 * This is OK, because current is on_cpu, which avoids it being picked
 	 * for load-balance and preemption/IRQs are still disabled avoiding
 	 * further scheduler activity on it and we're being very careful to

^ permalink raw reply	[flat|nested] 18+ messages in thread

* [PATCH 2/4] sched/fair: Plug hole between hotplug and active load_balance
  2017-09-07 15:03 [PATCH 0/4] sched: Fix some load-balancer vs hotplug holes Peter Zijlstra
  2017-09-07 15:03 ` [PATCH 1/4] sched/fair: Avoid newidle balance for !active CPUs Peter Zijlstra
@ 2017-09-07 15:03 ` Peter Zijlstra
  2017-09-12 18:05   ` [tip:sched/urgent] sched/fair: Plug hole between hotplug and active_load_balance() tip-bot for Peter Zijlstra
  2017-09-07 15:03 ` [PATCH 3/4] sched: WARN when migrating to an offline CPU Peter Zijlstra
  2017-09-07 15:03 ` [PATCH 4/4] sched/debug: Add debugfs knob for "sched_debug" Peter Zijlstra
  3 siblings, 1 reply; 18+ messages in thread
From: Peter Zijlstra @ 2017-09-07 15:03 UTC (permalink / raw)
  To: mingo, tglx; +Cc: linux-kernel, peterz

[-- Attachment #1: peterz-sched-active-balance-hole.patch --]
[-- Type: text/plain, Size: 1641 bytes --]

The load balancer applies cpu_active_mask to whatever sched_domains it
finds, however in the case of active_balance there is a hole between
setting rq->{active_balance,push_cpu} and running the stop_machine
work doing the actual migration.

The @push_cpu can go offline in this window, which would result in us
moving a task onto a dead cpu, which is a fairly bad thing.

Double check the active mask before the stop work does the migration.


  CPU0					CPU1

  <SoftIRQ>
					stop_machine(takedown_cpu)
    load_balance()			cpu_stopper_thread()
      ...				  work = multi_cpu_stop
      stop_one_cpu_nowait(		    /* wait for CPU0 */
	.func = active_load_balance_cpu_stop
      );
  </SoftIRQ>

  cpu_stopper_thread()
    work = multi_cpu_stop
      /* sync with CPU1 */
					    take_cpu_down()
					<idle>
					  play_dead();

    work = active_load_balance_cpu_stop
      set_task_cpu(p, CPU1); /* oops!! */

Reported-by: Thomas Gleixner <tglx@linutronix.de>
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
---
 kernel/sched/fair.c |    7 +++++++
 1 file changed, 7 insertions(+)

--- a/kernel/sched/fair.c
+++ b/kernel/sched/fair.c
@@ -8560,6 +8560,13 @@ static int active_load_balance_cpu_stop(
 	struct rq_flags rf;
 
 	rq_lock_irq(busiest_rq, &rf);
+	/*
+	 * Between queueing the stop-work and running it is a hole in which
+	 * CPUs can become inactive. We should not move tasks from or to
+	 * inactive CPUs.
+	 */
+	if (!cpu_active(busiest_cpu) || !cpu_active(target_cpu))
+		goto out_unlock;
 
 	/* make sure the requested cpu hasn't gone down in the meantime */
 	if (unlikely(busiest_cpu != smp_processor_id() ||

^ permalink raw reply	[flat|nested] 18+ messages in thread

* [PATCH 3/4] sched: WARN when migrating to an offline CPU
  2017-09-07 15:03 [PATCH 0/4] sched: Fix some load-balancer vs hotplug holes Peter Zijlstra
  2017-09-07 15:03 ` [PATCH 1/4] sched/fair: Avoid newidle balance for !active CPUs Peter Zijlstra
  2017-09-07 15:03 ` [PATCH 2/4] sched/fair: Plug hole between hotplug and active load_balance Peter Zijlstra
@ 2017-09-07 15:03 ` Peter Zijlstra
  2017-09-12 18:05   ` [tip:sched/urgent] sched/core: WARN() " tip-bot for Peter Zijlstra
  2017-09-28  9:14   ` [PATCH 3/4] sched: WARN " Sasha Levin
  2017-09-07 15:03 ` [PATCH 4/4] sched/debug: Add debugfs knob for "sched_debug" Peter Zijlstra
  3 siblings, 2 replies; 18+ messages in thread
From: Peter Zijlstra @ 2017-09-07 15:03 UTC (permalink / raw)
  To: mingo, tglx; +Cc: linux-kernel, peterz

[-- Attachment #1: peterz-sched-warn-migrate-to-offline.patch --]
[-- Type: text/plain, Size: 608 bytes --]

Migrating tasks to offline CPUs is a pretty big fail, warn about it.

Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
---
 kernel/sched/core.c |    4 ++++
 1 file changed, 4 insertions(+)

--- a/kernel/sched/core.c
+++ b/kernel/sched/core.c
@@ -1173,6 +1173,10 @@ void set_task_cpu(struct task_struct *p,
 	WARN_ON_ONCE(debug_locks && !(lockdep_is_held(&p->pi_lock) ||
 				      lockdep_is_held(&task_rq(p)->lock)));
 #endif
+	/*
+	 * Clearly, migrating tasks to offline CPUs is a fairly daft thing.
+	 */
+	WARN_ON_ONCE(!cpu_online(new_cpu));
 #endif
 
 	trace_sched_migrate_task(p, new_cpu);

^ permalink raw reply	[flat|nested] 18+ messages in thread

* [PATCH 4/4] sched/debug: Add debugfs knob for "sched_debug"
  2017-09-07 15:03 [PATCH 0/4] sched: Fix some load-balancer vs hotplug holes Peter Zijlstra
                   ` (2 preceding siblings ...)
  2017-09-07 15:03 ` [PATCH 3/4] sched: WARN when migrating to an offline CPU Peter Zijlstra
@ 2017-09-07 15:03 ` Peter Zijlstra
  2017-09-12 18:05   ` [tip:sched/urgent] " tip-bot for Peter Zijlstra
  3 siblings, 1 reply; 18+ messages in thread
From: Peter Zijlstra @ 2017-09-07 15:03 UTC (permalink / raw)
  To: mingo, tglx; +Cc: linux-kernel, peterz

[-- Attachment #1: peterz-sched-debug-knob.patch --]
[-- Type: text/plain, Size: 1371 bytes --]

I'm forever late for editing my kernel cmdline, add a runtime knob to
disable the "sched_debug" thing.

Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
---

--- a/kernel/sched/debug.c
+++ b/kernel/sched/debug.c
@@ -181,11 +181,16 @@ static const struct file_operations sche
 	.release	= single_release,
 };
 
+__read_mostly bool sched_debug_enabled;
+
 static __init int sched_init_debug(void)
 {
 	debugfs_create_file("sched_features", 0644, NULL, NULL,
 			&sched_feat_fops);
 
+	debugfs_create_bool("sched_debug", 0644, NULL,
+			&sched_debug_enabled);
+
 	return 0;
 }
 late_initcall(sched_init_debug);
--- a/kernel/sched/sched.h
+++ b/kernel/sched/sched.h
@@ -1954,6 +1954,8 @@ extern struct sched_entity *__pick_first
 extern struct sched_entity *__pick_last_entity(struct cfs_rq *cfs_rq);
 
 #ifdef	CONFIG_SCHED_DEBUG
+extern bool sched_debug_enabled;
+
 extern void print_cfs_stats(struct seq_file *m, int cpu);
 extern void print_rt_stats(struct seq_file *m, int cpu);
 extern void print_dl_stats(struct seq_file *m, int cpu);
--- a/kernel/sched/topology.c
+++ b/kernel/sched/topology.c
@@ -14,11 +14,9 @@ cpumask_var_t sched_domains_tmpmask2;
 
 #ifdef CONFIG_SCHED_DEBUG
 
-static __read_mostly int sched_debug_enabled;
-
 static int __init sched_debug_setup(char *str)
 {
-	sched_debug_enabled = 1;
+	sched_debug_enabled = true;
 
 	return 0;
 }

^ permalink raw reply	[flat|nested] 18+ messages in thread

* [tip:sched/urgent] sched/fair: Avoid newidle balance for !active CPUs
  2017-09-07 15:03 ` [PATCH 1/4] sched/fair: Avoid newidle balance for !active CPUs Peter Zijlstra
@ 2017-09-12 18:04   ` tip-bot for Peter Zijlstra
  0 siblings, 0 replies; 18+ messages in thread
From: tip-bot for Peter Zijlstra @ 2017-09-12 18:04 UTC (permalink / raw)
  To: linux-tip-commits; +Cc: hpa, peterz, tglx, linux-kernel, torvalds, mingo

Commit-ID:  2800486ee34825d954f64c6f98037daea328f121
Gitweb:     http://git.kernel.org/tip/2800486ee34825d954f64c6f98037daea328f121
Author:     Peter Zijlstra <peterz@infradead.org>
AuthorDate: Thu, 7 Sep 2017 17:03:50 +0200
Committer:  Ingo Molnar <mingo@kernel.org>
CommitDate: Tue, 12 Sep 2017 17:41:03 +0200

sched/fair: Avoid newidle balance for !active CPUs

On CPU hot unplug, when parking the last kthread we'll try and
schedule into idle to kill the CPU. This last schedule can (and does)
trigger newidle balance because at this point the sched domains are
still up because of commit:

  77d1dfda0e79 ("sched/topology, cpuset: Avoid spurious/wrong domain rebuilds")

Obviously pulling tasks to an already offline CPU is a bad idea, and
all balancing operations _should_ be subject to cpu_active_mask, make
it so.

Reported-by: Thomas Gleixner <tglx@linutronix.de>
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Cc: Linus Torvalds <torvalds@linux-foundation.org>
Cc: Peter Zijlstra <peterz@infradead.org>
Fixes: 77d1dfda0e79 ("sched/topology, cpuset: Avoid spurious/wrong domain rebuilds")
Link: http://lkml.kernel.org/r/20170907150613.994135806@infradead.org
Signed-off-by: Ingo Molnar <mingo@kernel.org>
---
 kernel/sched/fair.c | 6 ++++++
 1 file changed, 6 insertions(+)

diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c
index 8415d1e..3bcea40 100644
--- a/kernel/sched/fair.c
+++ b/kernel/sched/fair.c
@@ -8448,6 +8448,12 @@ static int idle_balance(struct rq *this_rq, struct rq_flags *rf)
 	this_rq->idle_stamp = rq_clock(this_rq);
 
 	/*
+	 * Do not pull tasks towards !active CPUs...
+	 */
+	if (!cpu_active(this_cpu))
+		return 0;
+
+	/*
 	 * This is OK, because current is on_cpu, which avoids it being picked
 	 * for load-balance and preemption/IRQs are still disabled avoiding
 	 * further scheduler activity on it and we're being very careful to

^ permalink raw reply related	[flat|nested] 18+ messages in thread

* [tip:sched/urgent] sched/fair: Plug hole between hotplug and active_load_balance()
  2017-09-07 15:03 ` [PATCH 2/4] sched/fair: Plug hole between hotplug and active load_balance Peter Zijlstra
@ 2017-09-12 18:05   ` tip-bot for Peter Zijlstra
  0 siblings, 0 replies; 18+ messages in thread
From: tip-bot for Peter Zijlstra @ 2017-09-12 18:05 UTC (permalink / raw)
  To: linux-tip-commits; +Cc: mingo, torvalds, hpa, linux-kernel, peterz, tglx

Commit-ID:  edd8e41d2e3cbd6ebe13ead30eb1adc6f48cbb33
Gitweb:     http://git.kernel.org/tip/edd8e41d2e3cbd6ebe13ead30eb1adc6f48cbb33
Author:     Peter Zijlstra <peterz@infradead.org>
AuthorDate: Thu, 7 Sep 2017 17:03:51 +0200
Committer:  Ingo Molnar <mingo@kernel.org>
CommitDate: Tue, 12 Sep 2017 17:41:04 +0200

sched/fair: Plug hole between hotplug and active_load_balance()

The load balancer applies cpu_active_mask to whatever sched_domains it
finds, however in the case of active_balance there is a hole between
setting rq->{active_balance,push_cpu} and running the stop_machine
work doing the actual migration.

The @push_cpu can go offline in this window, which would result in us
moving a task onto a dead cpu, which is a fairly bad thing.

Double check the active mask before the stop work does the migration.

  CPU0					CPU1

  <SoftIRQ>
					stop_machine(takedown_cpu)
    load_balance()			cpu_stopper_thread()
      ...				  work = multi_cpu_stop
      stop_one_cpu_nowait(		    /* wait for CPU0 */
	.func = active_load_balance_cpu_stop
      );
  </SoftIRQ>

  cpu_stopper_thread()
    work = multi_cpu_stop
      /* sync with CPU1 */
					    take_cpu_down()
					<idle>
					  play_dead();

    work = active_load_balance_cpu_stop
      set_task_cpu(p, CPU1); /* oops!! */

Reported-by: Thomas Gleixner <tglx@linutronix.de>
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Cc: Linus Torvalds <torvalds@linux-foundation.org>
Cc: Peter Zijlstra <peterz@infradead.org>
Link: http://lkml.kernel.org/r/20170907150614.044460912@infradead.org
Signed-off-by: Ingo Molnar <mingo@kernel.org>
---
 kernel/sched/fair.c | 7 +++++++
 1 file changed, 7 insertions(+)

diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c
index 3bcea40..efeebed 100644
--- a/kernel/sched/fair.c
+++ b/kernel/sched/fair.c
@@ -8560,6 +8560,13 @@ static int active_load_balance_cpu_stop(void *data)
 	struct rq_flags rf;
 
 	rq_lock_irq(busiest_rq, &rf);
+	/*
+	 * Between queueing the stop-work and running it is a hole in which
+	 * CPUs can become inactive. We should not move tasks from or to
+	 * inactive CPUs.
+	 */
+	if (!cpu_active(busiest_cpu) || !cpu_active(target_cpu))
+		goto out_unlock;
 
 	/* make sure the requested cpu hasn't gone down in the meantime */
 	if (unlikely(busiest_cpu != smp_processor_id() ||

^ permalink raw reply related	[flat|nested] 18+ messages in thread

* [tip:sched/urgent] sched/core: WARN() when migrating to an offline CPU
  2017-09-07 15:03 ` [PATCH 3/4] sched: WARN when migrating to an offline CPU Peter Zijlstra
@ 2017-09-12 18:05   ` tip-bot for Peter Zijlstra
  2017-09-28  9:14   ` [PATCH 3/4] sched: WARN " Sasha Levin
  1 sibling, 0 replies; 18+ messages in thread
From: tip-bot for Peter Zijlstra @ 2017-09-12 18:05 UTC (permalink / raw)
  To: linux-tip-commits; +Cc: peterz, linux-kernel, hpa, mingo, torvalds, tglx

Commit-ID:  4ff9083b8a9a80bdf4ebbbec22cda4cbfb60f7aa
Gitweb:     http://git.kernel.org/tip/4ff9083b8a9a80bdf4ebbbec22cda4cbfb60f7aa
Author:     Peter Zijlstra <peterz@infradead.org>
AuthorDate: Thu, 7 Sep 2017 17:03:52 +0200
Committer:  Ingo Molnar <mingo@kernel.org>
CommitDate: Tue, 12 Sep 2017 17:41:04 +0200

sched/core: WARN() when migrating to an offline CPU

Migrating tasks to offline CPUs is a pretty big fail, warn about it.

Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Cc: Linus Torvalds <torvalds@linux-foundation.org>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Thomas Gleixner <tglx@linutronix.de>
Link: http://lkml.kernel.org/r/20170907150614.094206976@infradead.org
Signed-off-by: Ingo Molnar <mingo@kernel.org>
---
 kernel/sched/core.c | 4 ++++
 1 file changed, 4 insertions(+)

diff --git a/kernel/sched/core.c b/kernel/sched/core.c
index 136a76d..18a6966 100644
--- a/kernel/sched/core.c
+++ b/kernel/sched/core.c
@@ -1173,6 +1173,10 @@ void set_task_cpu(struct task_struct *p, unsigned int new_cpu)
 	WARN_ON_ONCE(debug_locks && !(lockdep_is_held(&p->pi_lock) ||
 				      lockdep_is_held(&task_rq(p)->lock)));
 #endif
+	/*
+	 * Clearly, migrating tasks to offline CPUs is a fairly daft thing.
+	 */
+	WARN_ON_ONCE(!cpu_online(new_cpu));
 #endif
 
 	trace_sched_migrate_task(p, new_cpu);

^ permalink raw reply related	[flat|nested] 18+ messages in thread

* [tip:sched/urgent] sched/debug: Add debugfs knob for "sched_debug"
  2017-09-07 15:03 ` [PATCH 4/4] sched/debug: Add debugfs knob for "sched_debug" Peter Zijlstra
@ 2017-09-12 18:05   ` tip-bot for Peter Zijlstra
  0 siblings, 0 replies; 18+ messages in thread
From: tip-bot for Peter Zijlstra @ 2017-09-12 18:05 UTC (permalink / raw)
  To: linux-tip-commits; +Cc: linux-kernel, tglx, peterz, hpa, torvalds, mingo

Commit-ID:  9469eb01db891b55367ee7539f1b9f7f6fd2819d
Gitweb:     http://git.kernel.org/tip/9469eb01db891b55367ee7539f1b9f7f6fd2819d
Author:     Peter Zijlstra <peterz@infradead.org>
AuthorDate: Thu, 7 Sep 2017 17:03:53 +0200
Committer:  Ingo Molnar <mingo@kernel.org>
CommitDate: Tue, 12 Sep 2017 17:41:04 +0200

sched/debug: Add debugfs knob for "sched_debug"

I'm forever late for editing my kernel cmdline, add a runtime knob to
disable the "sched_debug" thing.

Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Cc: Linus Torvalds <torvalds@linux-foundation.org>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Thomas Gleixner <tglx@linutronix.de>
Link: http://lkml.kernel.org/r/20170907150614.142924283@infradead.org
Signed-off-by: Ingo Molnar <mingo@kernel.org>
---
 kernel/sched/debug.c    | 5 +++++
 kernel/sched/sched.h    | 2 ++
 kernel/sched/topology.c | 4 +---
 3 files changed, 8 insertions(+), 3 deletions(-)

diff --git a/kernel/sched/debug.c b/kernel/sched/debug.c
index 4a23bbc..b19d06ea 100644
--- a/kernel/sched/debug.c
+++ b/kernel/sched/debug.c
@@ -181,11 +181,16 @@ static const struct file_operations sched_feat_fops = {
 	.release	= single_release,
 };
 
+__read_mostly bool sched_debug_enabled;
+
 static __init int sched_init_debug(void)
 {
 	debugfs_create_file("sched_features", 0644, NULL, NULL,
 			&sched_feat_fops);
 
+	debugfs_create_bool("sched_debug", 0644, NULL,
+			&sched_debug_enabled);
+
 	return 0;
 }
 late_initcall(sched_init_debug);
diff --git a/kernel/sched/sched.h b/kernel/sched/sched.h
index ab1c7f5..7ea2a03 100644
--- a/kernel/sched/sched.h
+++ b/kernel/sched/sched.h
@@ -1954,6 +1954,8 @@ extern struct sched_entity *__pick_first_entity(struct cfs_rq *cfs_rq);
 extern struct sched_entity *__pick_last_entity(struct cfs_rq *cfs_rq);
 
 #ifdef	CONFIG_SCHED_DEBUG
+extern bool sched_debug_enabled;
+
 extern void print_cfs_stats(struct seq_file *m, int cpu);
 extern void print_rt_stats(struct seq_file *m, int cpu);
 extern void print_dl_stats(struct seq_file *m, int cpu);
diff --git a/kernel/sched/topology.c b/kernel/sched/topology.c
index 6f7b439..2ab2aa6 100644
--- a/kernel/sched/topology.c
+++ b/kernel/sched/topology.c
@@ -14,11 +14,9 @@ cpumask_var_t sched_domains_tmpmask2;
 
 #ifdef CONFIG_SCHED_DEBUG
 
-static __read_mostly int sched_debug_enabled;
-
 static int __init sched_debug_setup(char *str)
 {
-	sched_debug_enabled = 1;
+	sched_debug_enabled = true;
 
 	return 0;
 }

^ permalink raw reply related	[flat|nested] 18+ messages in thread

* Re: [PATCH 3/4] sched: WARN when migrating to an offline CPU
  2017-09-07 15:03 ` [PATCH 3/4] sched: WARN when migrating to an offline CPU Peter Zijlstra
  2017-09-12 18:05   ` [tip:sched/urgent] sched/core: WARN() " tip-bot for Peter Zijlstra
@ 2017-09-28  9:14   ` Sasha Levin
  2017-09-28 10:35     ` Peter Zijlstra
  1 sibling, 1 reply; 18+ messages in thread
From: Sasha Levin @ 2017-09-28  9:14 UTC (permalink / raw)
  To: Peter Zijlstra
  Cc: Ingo Molnar, Thomas Gleixner, linux-kernel@vger.kernel.org List,
	alexander.levin

On Thu, Sep 7, 2017 at 8:03 AM, Peter Zijlstra <peterz@infradead.org> wrote:
> Migrating tasks to offline CPUs is a pretty big fail, warn about it.

Hey Peter,

This seems to get hit on the latest -next:

[2035565360.446794] Unregister pv shared memory for cpu 2
[2035565360.467930] numa_remove_cpu cpu 2 node 2: mask now 6
[2035565360.471431] ------------[ cut here ]------------
[2035565360.472548] WARNING: CPU: 2 PID: 24 at
kernel/sched/core.c:1178 set_task_cpu (kernel/sched/core.c:1157)
[2035565360.473840] Modules linked in:
[2035565360.474632] CPU: 2 PID: 24 Comm: migration/2 Not tainted
4.14.0-rc2-next-20170927+ #252
[2035565360.476128] task: ffff8801ab7dc040 task.stack: ffff8801ab7e0000
[2035565360.477108] RIP: 0010:set_task_cpu (??:?)
[2035565360.477850] RSP: 0018:ffff8801ab807b18 EFLAGS: 00010096
[2035565360.478734] RAX: 0000000000000002 RBX: ffff8801a8d94040 RCX:
1ffff1003577cb3c
[2035565360.479872] RDX: 0000000000000004 RSI: 0000000000000002 RDI:
ffffffff9177d940
[2035565360.480998] RBP: ffff8801ab807b58 R08: 000000000000b6b2 R09:
ffff8801abbe5a2c
[2035565360.482356] R10: 0000000000000001 R11: 0000000000000001 R12:
ffff8801a8d94048
[2035565360.483490] R13: ffff8801ab807c88 R14: 0000000000000002 R15:
ffff8801a8d940a4
[2035565360.484617] FS:  0000000000000000(0000)
GS:ffff8801ab800000(0000) knlGS:0000000000000000
[2035565360.485881] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[2035565360.490482] CR2: 00000000012dfae4 CR3: 00000001a30c5002 CR4:
00000000000606a0
[2035565360.491836] Call Trace:
[2035565360.492301]  <IRQ>
[2035565360.492710] detach_task.isra.80 (kernel/sched/fair.c:6816)
[2035565360.493430] load_balance (./include/linux/list.h:78
kernel/sched/fair.c:6902 kernel/sched/fair.c:8201)
[2035565360.494094] ? find_busiest_group (kernel/sched/fair.c:8133)
[2035565360.494861] ? rebalance_domains (kernel/sched/fair.c:8785)
[2035565360.495571] rebalance_domains (kernel/sched/fair.c:8829)
[2035565360.496292] ? find_next_bit (lib/find_bit.c:65)
[2035565360.496961] ? pick_next_task_fair (kernel/sched/fair.c:8779)
[2035565360.497753] ? check_preemption_disabled (lib/smp_processor_id.c:52)
[2035565360.498574] run_rebalance_domains (kernel/sched/fair.c:9052)
[2035565360.499318] ? check_preemption_disabled (lib/smp_processor_id.c:52)
[2035565360.500388] __do_softirq (kernel/softirq.c:284
./include/linux/jump_label.h:141 ./include/trace/events/irq.h:141
kernel/softirq.c:285)
[2035565360.501039] ? hrtimer_interrupt (kernel/time/hrtimer.c:1370)
[2035565360.502014] irq_exit (kernel/softirq.c:364 kernel/softirq.c:405)
[2035565360.502598] smp_apic_timer_interrupt
(./arch/x86/include/asm/irq_regs.h:26
arch/x86/kernel/apic/apic.c:1043)
[2035565360.503605] apic_timer_interrupt (arch/x86/entry/entry_64.S:770)
[2035565360.504313]  </IRQ>
[2035565360.504724] RIP: 0010:multi_cpu_stop (??:?)
[2035565360.505503] RSP: 0018:ffff8801ab7e7d88 EFLAGS: 00000292
ORIG_RAX: ffffffffffffff11
[2035565360.506721] RAX: 0000000000000003 RBX: 0000000000000004 RCX:
000000000000000a
[2035565360.507862] RDX: 0000000000000004 RSI: 0000000000000040 RDI:
0000000000000292
[2035565360.508997] RBP: ffff8801ab7e7de0 R08: ffff8801ab7dc040 R09:
0000000000000000
[2035565360.510566] R10: 0000000000000001 R11: 1ffff1003577cb55 R12:
ffff8801aae2f890
[2035565360.511701] R13: ffff8801aae2f8b4 R14: dffffc0000000000 R15:
0000000000000003
[2035565360.512843] ? cpu_stop_queue_work (kernel/stop_machine.c:176)
[2035565360.513591] cpu_stopper_thread (kernel/stop_machine.c:480)
[2035565360.514316] ? cpu_stop_create (kernel/stop_machine.c:458)
[2035565360.515218] smpboot_thread_fn (kernel/smpboot.c:164)
[2035565360.515918] ? sort_range (kernel/smpboot.c:107)
[2035565360.516528] ? schedule (./arch/x86/include/asm/bitops.h:324
(discriminator 1) ./include/linux/thread_info.h:79 (discriminator 1)
./include/linux/sched.h:1605 (discriminator 1)
kernel/sched/core.c:3435 (discriminator 1))
[2035565360.517119] ? __kthread_parkme (kernel/kthread.c:188)
[2035565360.517835] kthread (kernel/kthread.c:242)
[2035565360.518413] ? sort_range (kernel/smpboot.c:107)
[2035565360.519008] ? kthread_create_on_node (kernel/kthread.c:198)
[2035565360.519792] ret_from_fork (arch/x86/entry/entry_64.S:437)
[2035565360.520407] Code: 09 84 d2 74 05 e8 94 38 4b 00 f7 83 84 00 00
00 fd ff ff ff 0f 84 a0 f9 ff ff 0f ff e9 99 f9 ff ff e8 68 18 ff ff
e9 c7 fd ff ff <0f> ff e9 ea f9 ff ff e8 57 18 ff ff e9 f5 f9 ff ff e8
0d 84 0f
All code
========
   0: 09 84 d2 74 05 e8 94 or     %eax,-0x6b17fa8c(%rdx,%rdx,8)
   7: 38 4b 00             cmp    %cl,0x0(%rbx)
   a: f7 83 84 00 00 00 fd testl  $0xfffffffd,0x84(%rbx)
  11: ff ff ff
  14: 0f 84 a0 f9 ff ff     je     0xfffffffffffff9ba
  1a: 0f ff                 (bad)
  1c: e9 99 f9 ff ff       jmpq   0xfffffffffffff9ba
  21: e8 68 18 ff ff       callq  0xffffffffffff188e
  26: e9 c7 fd ff ff       jmpq   0xfffffffffffffdf2
  2b:* 0f ff                 (bad)   <-- trapping instruction
  2d: e9 ea f9 ff ff       jmpq   0xfffffffffffffa1c
  32: e8 57 18 ff ff       callq  0xffffffffffff188e
  37: e9 f5 f9 ff ff       jmpq   0xfffffffffffffa31
  3c: e8 0d 84 0f 00       callq  0xf844e

Code starting with the faulting instruction
===========================================
   0: 0f ff                 (bad)
   2: e9 ea f9 ff ff       jmpq   0xfffffffffffff9f1
   7: e8 57 18 ff ff       callq  0xffffffffffff1863
   c: e9 f5 f9 ff ff       jmpq   0xfffffffffffffa06
  11: e8 0d 84 0f 00       callq  0xf8423
[2035565360.523585] ---[ end trace 812b7ff74114db6c ]---
[2035565360.524744] smpboot: CPU 2 is now offline


Thanks,
Sasha

^ permalink raw reply	[flat|nested] 18+ messages in thread

* Re: [PATCH 3/4] sched: WARN when migrating to an offline CPU
  2017-09-28  9:14   ` [PATCH 3/4] sched: WARN " Sasha Levin
@ 2017-09-28 10:35     ` Peter Zijlstra
  2017-09-28 11:03       ` Levin, Alexander (Sasha Levin)
  0 siblings, 1 reply; 18+ messages in thread
From: Peter Zijlstra @ 2017-09-28 10:35 UTC (permalink / raw)
  To: Sasha Levin
  Cc: Ingo Molnar, Thomas Gleixner, linux-kernel@vger.kernel.org List,
	alexander.levin

On Thu, Sep 28, 2017 at 02:14:15AM -0700, Sasha Levin wrote:
> On Thu, Sep 7, 2017 at 8:03 AM, Peter Zijlstra <peterz@infradead.org> wrote:
> > Migrating tasks to offline CPUs is a pretty big fail, warn about it.
> 
> Hey Peter,
> 
> This seems to get hit on the latest -next:
> 
> [2035565360.446794] Unregister pv shared memory for cpu 2
> [2035565360.467930] numa_remove_cpu cpu 2 node 2: mask now 6
> [2035565360.471431] ------------[ cut here ]------------
> [2035565360.472548] WARNING: CPU: 2 PID: 24 at > kernel/sched/core.c:1178 set_task_cpu (kernel/sched/core.c:1157)
> [2035565360.473840] Modules linked in:
> [2035565360.474632] CPU: 2 PID: 24 Comm: migration/2 Not tainted > 4.14.0-rc2-next-20170927+ #252

Urgh, weird. That really shouldn't happen. Can you easily reproduce?

> [2035565360.491836] Call Trace:
> [2035565360.492301]  <IRQ>
> [2035565360.492710] detach_task.isra.80 (kernel/sched/fair.c:6816)
> [2035565360.493430] load_balance (./include/linux/list.h:78 > kernel/sched/fair.c:6902 kernel/sched/fair.c:8201)
> [2035565360.494094] ? find_busiest_group (kernel/sched/fair.c:8133)
> [2035565360.494861] ? rebalance_domains (kernel/sched/fair.c:8785)
> [2035565360.495571] rebalance_domains (kernel/sched/fair.c:8829)
> [2035565360.496292] ? find_next_bit (lib/find_bit.c:65)
> [2035565360.496961] ? pick_next_task_fair (kernel/sched/fair.c:8779)
> [2035565360.497753] ? check_preemption_disabled (lib/smp_processor_id.c:52)
> [2035565360.498574] run_rebalance_domains (kernel/sched/fair.c:9052)
> [2035565360.499318] ? check_preemption_disabled (lib/smp_processor_id.c:52)
> [2035565360.500388] __do_softirq (kernel/softirq.c:284 > ./include/linux/jump_label.h:141 ./include/trace/events/irq.h:141 > kernel/softirq.c:285)
> [2035565360.501039] ? hrtimer_interrupt (kernel/time/hrtimer.c:1370)
> [2035565360.502014] irq_exit (kernel/softirq.c:364 kernel/softirq.c:405)
> [2035565360.502598] smp_apic_timer_interrupt > (./arch/x86/include/asm/irq_regs.h:26 > arch/x86/kernel/apic/apic.c:1043)
> [2035565360.503605] apic_timer_interrupt (arch/x86/entry/entry_64.S:770)
> [2035565360.504313]  </IRQ>

^ permalink raw reply	[flat|nested] 18+ messages in thread

* Re: [PATCH 3/4] sched: WARN when migrating to an offline CPU
  2017-09-28 10:35     ` Peter Zijlstra
@ 2017-09-28 11:03       ` Levin, Alexander (Sasha Levin)
  2017-09-28 11:42         ` Peter Zijlstra
  0 siblings, 1 reply; 18+ messages in thread
From: Levin, Alexander (Sasha Levin) @ 2017-09-28 11:03 UTC (permalink / raw)
  To: Peter Zijlstra
  Cc: Sasha Levin, Ingo Molnar, Thomas Gleixner,
	linux-kernel@vger.kernel.org List

On Thu, Sep 28, 2017 at 12:35:41PM +0200, Peter Zijlstra wrote:
>On Thu, Sep 28, 2017 at 02:14:15AM -0700, Sasha Levin wrote:
>> On Thu, Sep 7, 2017 at 8:03 AM, Peter Zijlstra <peterz@infradead.org> wrote:
>> > Migrating tasks to offline CPUs is a pretty big fail, warn about it.
>>
>> Hey Peter,
>>
>> This seems to get hit on the latest -next:
>>
>> [2035565360.446794] Unregister pv shared memory for cpu 2
>> [2035565360.467930] numa_remove_cpu cpu 2 node 2: mask now 6
>> [2035565360.471431] ------------[ cut here ]------------
>> [2035565360.472548] WARNING: CPU: 2 PID: 24 at > kernel/sched/core.c:1178 set_task_cpu (kernel/sched/core.c:1157)
>> [2035565360.473840] Modules linked in:
>> [2035565360.474632] CPU: 2 PID: 24 Comm: migration/2 Not tainted > 4.14.0-rc2-next-20170927+ #252
>
>Urgh, weird. That really shouldn't happen. Can you easily reproduce?

Looks like yes. Seems like it's enough to stress CPU hotplug + trinity.

If you have patches you want me try send them my way.

-- 

Thanks,
Sasha

^ permalink raw reply	[flat|nested] 18+ messages in thread

* Re: [PATCH 3/4] sched: WARN when migrating to an offline CPU
  2017-09-28 11:03       ` Levin, Alexander (Sasha Levin)
@ 2017-09-28 11:42         ` Peter Zijlstra
  2017-09-29 11:11           ` Peter Zijlstra
  0 siblings, 1 reply; 18+ messages in thread
From: Peter Zijlstra @ 2017-09-28 11:42 UTC (permalink / raw)
  To: Levin, Alexander (Sasha Levin)
  Cc: Sasha Levin, Ingo Molnar, Thomas Gleixner,
	linux-kernel@vger.kernel.org List

On Thu, Sep 28, 2017 at 11:03:10AM +0000, Levin, Alexander (Sasha Levin) wrote:
> On Thu, Sep 28, 2017 at 12:35:41PM +0200, Peter Zijlstra wrote:
> >On Thu, Sep 28, 2017 at 02:14:15AM -0700, Sasha Levin wrote:

> >> [2035565360.446794] Unregister pv shared memory for cpu 2
> >> [2035565360.467930] numa_remove_cpu cpu 2 node 2: mask now 6
> >> [2035565360.471431] ------------[ cut here ]------------
> >> [2035565360.472548] WARNING: CPU: 2 PID: 24 at > kernel/sched/core.c:1178 set_task_cpu (kernel/sched/core.c:1157)
> >> [2035565360.473840] Modules linked in:
> >> [2035565360.474632] CPU: 2 PID: 24 Comm: migration/2 Not tainted > 4.14.0-rc2-next-20170927+ #252
> >
> >Urgh, weird. That really shouldn't happen. Can you easily reproduce?
> 
> Looks like yes. Seems like it's enough to stress CPU hotplug + trinity.

OK, I'll see if I can reproduce building kernels and hotplug stress.
Otherwise I'll try and cook up some debug patches for you.

^ permalink raw reply	[flat|nested] 18+ messages in thread

* Re: [PATCH 3/4] sched: WARN when migrating to an offline CPU
  2017-09-28 11:42         ` Peter Zijlstra
@ 2017-09-29 11:11           ` Peter Zijlstra
  2017-10-07  2:07             ` Levin, Alexander (Sasha Levin)
  0 siblings, 1 reply; 18+ messages in thread
From: Peter Zijlstra @ 2017-09-29 11:11 UTC (permalink / raw)
  To: Levin, Alexander (Sasha Levin)
  Cc: Sasha Levin, Ingo Molnar, Thomas Gleixner,
	linux-kernel@vger.kernel.org List

On Thu, Sep 28, 2017 at 01:42:49PM +0200, Peter Zijlstra wrote:
> On Thu, Sep 28, 2017 at 11:03:10AM +0000, Levin, Alexander (Sasha Levin) wrote:
> > On Thu, Sep 28, 2017 at 12:35:41PM +0200, Peter Zijlstra wrote:
> > >On Thu, Sep 28, 2017 at 02:14:15AM -0700, Sasha Levin wrote:
> 
> > >> [2035565360.446794] Unregister pv shared memory for cpu 2
> > >> [2035565360.467930] numa_remove_cpu cpu 2 node 2: mask now 6
> > >> [2035565360.471431] ------------[ cut here ]------------
> > >> [2035565360.472548] WARNING: CPU: 2 PID: 24 at > kernel/sched/core.c:1178 set_task_cpu (kernel/sched/core.c:1157)
> > >> [2035565360.473840] Modules linked in:
> > >> [2035565360.474632] CPU: 2 PID: 24 Comm: migration/2 Not tainted > 4.14.0-rc2-next-20170927+ #252
> > >
> > >Urgh, weird. That really shouldn't happen. Can you easily reproduce?
> > 
> > Looks like yes. Seems like it's enough to stress CPU hotplug + trinity.
> 
> OK, I'll see if I can reproduce building kernels and hotplug stress.
> Otherwise I'll try and cook up some debug patches for you.

I can't seem to trigger :-(

Can you please run with the below patch and:

  # echo 1 > /proc/sys/kernel/traceoff_on_warning

---
 kernel/sched/core.c |  3 +++
 kernel/sched/fair.c | 10 ++++++++++
 2 files changed, 13 insertions(+)

diff --git a/kernel/sched/core.c b/kernel/sched/core.c
index 18a6966567da..c613f7756981 100644
--- a/kernel/sched/core.c
+++ b/kernel/sched/core.c
@@ -5592,6 +5592,7 @@ int sched_cpu_activate(unsigned int cpu)
 	struct rq_flags rf;
 
 	set_cpu_active(cpu, true);
+	trace_printk("active: %d mask: %*pbl\n", cpu, cpumask_pr_args(cpu_active_mask));
 
 	if (sched_smp_initialized) {
 		sched_domains_numa_masks_set(cpu);
@@ -5624,6 +5625,7 @@ int sched_cpu_deactivate(unsigned int cpu)
 	int ret;
 
 	set_cpu_active(cpu, false);
+	trace_printk("not-active: %d mask: %*pbl\n", cpu, cpumask_pr_args(cpu_active_mask));
 	/*
 	 * We've cleared cpu_active_mask, wait for all preempt-disabled and RCU
 	 * users of this state to go away such that all new such users will
@@ -5632,6 +5634,7 @@ int sched_cpu_deactivate(unsigned int cpu)
 	 * Do sync before park smpboot threads to take care the rcu boost case.
 	 */
 	synchronize_rcu_mult(call_rcu, call_rcu_sched);
+	trace_printk("rcu-sync: %d\n", cpu);
 
 	if (!sched_smp_initialized)
 		return 0;
diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c
index 70ba32e08a23..cb8f43a59f33 100644
--- a/kernel/sched/fair.c
+++ b/kernel/sched/fair.c
@@ -8150,8 +8150,11 @@ static int load_balance(int this_cpu, struct rq *this_rq,
 		.tasks		= LIST_HEAD_INIT(env.tasks),
 	};
 
+
 	cpumask_and(cpus, sched_domain_span(sd), cpu_active_mask);
 
+	trace_printk("dst_cpu: %d cpus: %*pbl\n", this_cpu, cpumask_pr_args(cpus));
+
 	schedstat_inc(sd->lb_count[idle]);
 
 redo:
@@ -8248,6 +8251,9 @@ static int load_balance(int this_cpu, struct rq *this_rq,
 
 			env.dst_rq	 = cpu_rq(env.new_dst_cpu);
 			env.dst_cpu	 = env.new_dst_cpu;
+
+			trace_printk("dst_cpu: %d\n", env.dst_cpu);
+
 			env.flags	&= ~LBF_DST_PINNED;
 			env.loop	 = 0;
 			env.loop_break	 = sched_nr_migrate_break;
@@ -8465,6 +8471,7 @@ static int idle_balance(struct rq *this_rq, struct rq_flags *rf)
 
 	update_blocked_averages(this_cpu);
 	rcu_read_lock();
+	trace_printk("rcu-read-lock: %d\n", this_cpu);
 	for_each_domain(this_cpu, sd) {
 		int continue_balancing = 1;
 		u64 t0, domain_cost;
@@ -8500,6 +8507,7 @@ static int idle_balance(struct rq *this_rq, struct rq_flags *rf)
 		if (pulled_task || this_rq->nr_running > 0)
 			break;
 	}
+	trace_printk("rcu-read-unlock: %d\n", this_cpu);
 	rcu_read_unlock();
 
 	raw_spin_lock(&this_rq->lock);
@@ -8790,6 +8798,7 @@ static void rebalance_domains(struct rq *rq, enum cpu_idle_type idle)
 	update_blocked_averages(cpu);
 
 	rcu_read_lock();
+	trace_printk("rcu-read-lock: %d\n", cpu);
 	for_each_domain(cpu, sd) {
 		/*
 		 * Decay the newidle max times here because this is a regular
@@ -8853,6 +8862,7 @@ static void rebalance_domains(struct rq *rq, enum cpu_idle_type idle)
 		rq->max_idle_balance_cost =
 			max((u64)sysctl_sched_migration_cost, max_cost);
 	}
+	trace_printk("rcu-read-unlock: %d\n", cpu);
 	rcu_read_unlock();
 
 	/*

^ permalink raw reply related	[flat|nested] 18+ messages in thread

* Re: [PATCH 3/4] sched: WARN when migrating to an offline CPU
  2017-09-29 11:11           ` Peter Zijlstra
@ 2017-10-07  2:07             ` Levin, Alexander (Sasha Levin)
  2017-10-07  9:15               ` Peter Zijlstra
  0 siblings, 1 reply; 18+ messages in thread
From: Levin, Alexander (Sasha Levin) @ 2017-10-07  2:07 UTC (permalink / raw)
  To: Peter Zijlstra
  Cc: Sasha Levin, Ingo Molnar, Thomas Gleixner,
	linux-kernel@vger.kernel.org List

On Fri, Sep 29, 2017 at 01:11:26PM +0200, Peter Zijlstra wrote:
>I can't seem to trigger :-(
>
>Can you please run with the below patch and:
>
>  # echo 1 > /proc/sys/kernel/traceoff_on_warning

The call stack trace looks like so:

[ 2073.492089] Unregister pv shared memory for cpu 2
[ 2073.495414] NOHZ: local_softirq_pending 202
[ 2073.516166] ------------[ cut here ]------------
[ 2073.519047] WARNING: CPU: 2 PID: 24 at kernel/sched/core.c:1178 set_task_cpu (kernel/sched/core.c:1157) 
[ 2073.520317] Modules linked in:
[ 2073.520816] CPU: 2 PID: 24 Comm: migration/2 Not tainted 4.14.0-rc2-next-20170927+ #256
[ 2073.522046] task: ffff8801ab7cc040 task.stack: ffff8801ab7d8000
[ 2073.522963] RIP: 0010:set_task_cpu (??:?) 
[ 2073.523655] RSP: 0018:ffff8801ab807b20 EFLAGS: 00010096
[ 2073.524662] RAX: 0000000000000002 RBX: ffff880189810040 RCX: 1ffff100356f993a
[ 2073.525747] RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000046
[ 2073.526845] RBP: ffff8801ab807b60 R08: 0000000000000000 R09: 0000000000000000
[ 2073.527940] R10: 0000000000000000 R11: 0000000000000000 R12: 00000000001e5900
[ 2073.529027] R13: 0000000000000006 R14: 0000000000000002 R15: ffff8801898100a4
[ 2073.530118] FS:  0000000000000000(0000) GS:ffff8801ab800000(0000) knlGS:0000000000000000
[ 2073.531556] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 2073.532443] CR2: 00007ffd965f1be8 CR3: 0000000187ede001 CR4: 00000000000606a0
[ 2073.533546] Call Trace:
[ 2073.533944]  <IRQ>
[ 2073.534280] detach_task.isra.80 (kernel/sched/fair.c:6816) 
[ 2073.534939] load_balance (./include/linux/list.h:78 kernel/sched/fair.c:6902 kernel/sched/fair.c:8204) 
[ 2073.535543] ? find_busiest_group (kernel/sched/fair.c:8133) 
[ 2073.536252] rebalance_domains (kernel/sched/fair.c:8838) 
[ 2073.536903] ? pick_next_task_fair (kernel/sched/fair.c:8787) 
[ 2073.537628] ? check_preemption_disabled (lib/smp_processor_id.c:52) 
[ 2073.538395] run_rebalance_domains (kernel/sched/fair.c:9062) 
[ 2073.539096] ? check_preemption_disabled (lib/smp_processor_id.c:52) 
[ 2073.539850] __do_softirq (kernel/softirq.c:284 ./include/linux/jump_label.h:141 ./include/trace/events/irq.h:141 kernel/softirq.c:285) 
[ 2073.540424] ? check_preemption_disabled (lib/smp_processor_id.c:52) 
[ 2073.541189] irq_exit (kernel/softirq.c:364 kernel/softirq.c:405) 
[ 2073.541715] smp_apic_timer_interrupt (./arch/x86/include/asm/irq_regs.h:26 arch/x86/kernel/apic/apic.c:1043) 
[ 2073.542452] apic_timer_interrupt (arch/x86/entry/entry_64.S:770) 
[ 2073.543116]  </IRQ>
[ 2073.543463] RIP: 0010:multi_cpu_stop (??:?) 
[ 2073.544173] RSP: 0018:ffff8801ab7dfd88 EFLAGS: 00000292 ORIG_RAX: ffffffffffffff11
[ 2073.545326] RAX: 0000000000000007 RBX: 0000000000000004 RCX: 000000000000000a
[ 2073.546406] RDX: 0000000000000000 RSI: ffffffff9c46eee0 RDI: 0000000000000292
[ 2073.547503] RBP: ffff8801ab7dfde0 R08: 0000000000000000 R09: 0000000000000000
[ 2073.548587] R10: 0000000000000000 R11: 0000000000000000 R12: ffff88012aaf7890
[ 2073.549668] R13: ffff88012aaf78b4 R14: dffffc0000000000 R15: 0000000000000003
[ 2073.550765] ? __this_cpu_preempt_check (lib/smp_processor_id.c:63) 
[ 2073.551501] ? cpu_stop_queue_work (kernel/stop_machine.c:176) 
[ 2073.552195] cpu_stopper_thread (kernel/stop_machine.c:480) 
[ 2073.552845] ? cpu_stop_create (kernel/stop_machine.c:458) 
[ 2073.553462] smpboot_thread_fn (kernel/smpboot.c:164) 
[ 2073.554104] ? sort_range (kernel/smpboot.c:107) 
[ 2073.554671] ? schedule (./arch/x86/include/asm/bitops.h:324 (discriminator 1) ./include/linux/thread_info.h:79 (discriminator 1) ./include/linux/sched.h:1605 (discriminator 1) kernel/sched/core.c:3435 (discriminator 1)) 
[ 2073.555230] ? __kthread_parkme (kernel/kthread.c:188) 
[ 2073.555892] kthread (kernel/kthread.c:242) 
[ 2073.556408] ? sort_range (kernel/smpboot.c:107) 
[ 2073.556963] ? kthread_create_on_node (kernel/kthread.c:198) 
[ 2073.557693] ret_from_fork (arch/x86/entry/entry_64.S:437) 
[ 2073.558261] Code: 09 84 d2 74 05 e8 84 3a 4b 00 f7 83 84 00 00 00 fd ff ff ff 0f 84 a0 f9 ff ff 0f ff e9 99 f9 ff ff e8 78 18 ff ff e9 c7 fd ff ff <0f> ff e9 ea f9 ff ff e8 67 18 ff ff e9 f5 f9 ff ff e8 fd 85 0f 

All code
========
   0:	09 84 d2 74 05 e8 84 	or     %eax,-0x7b17fa8c(%rdx,%rdx,8)
   7:	3a 4b 00             	cmp    0x0(%rbx),%cl
   a:	f7 83 84 00 00 00 fd 	testl  $0xfffffffd,0x84(%rbx)
  11:	ff ff ff 
  14:	0f 84 a0 f9 ff ff    	je     0xfffffffffffff9ba
  1a:	0f ff                	(bad)  
  1c:	e9 99 f9 ff ff       	jmpq   0xfffffffffffff9ba
  21:	e8 78 18 ff ff       	callq  0xffffffffffff189e
  26:	e9 c7 fd ff ff       	jmpq   0xfffffffffffffdf2
  2b:*	0f ff                	(bad)  		<-- trapping instruction
  2d:	e9 ea f9 ff ff       	jmpq   0xfffffffffffffa1c
  32:	e8 67 18 ff ff       	callq  0xffffffffffff189e
  37:	e9 f5 f9 ff ff       	jmpq   0xfffffffffffffa31
  3c:	e8 fd 85 0f 00       	callq  0xf863e

Code starting with the faulting instruction
===========================================
   0:	0f ff                	(bad)  
   2:	e9 ea f9 ff ff       	jmpq   0xfffffffffffff9f1
   7:	e8 67 18 ff ff       	callq  0xffffffffffff1873
   c:	e9 f5 f9 ff ff       	jmpq   0xfffffffffffffa06
  11:	e8 fd 85 0f 00       	callq  0xf8613
[ 2073.561289] ---[ end trace 88455b894f38cbc5 ]---

And quite a few lines of your added trace (lmk if you need more, or all):

<idle>-0     [000] ..s2   104.829161: load_balance: dst_cpu: 6 cpus: 0-1,3-4,6-7,9
<idle>-0     [000] ..s1   104.829172: rebalance_domains: rcu-read-unlock: 6
<idle>-0     [000] ..s1   104.829177: rebalance_domains: rcu-read-lock: 0
<idle>-0     [000] ..s1   104.829179: load_balance: dst_cpu: 0 cpus: 0-1,3-4,6-7,9
<idle>-0     [000] ..s1   104.829183: rebalance_domains: rcu-read-unlock: 0
<idle>-0     [001] .Ns1   104.829313: rebalance_domains: rcu-read-lock: 1
<idle>-0     [001] .Ns1   104.829317: rebalance_domains: rcu-read-unlock: 1
<idle>-0     [006] .Ns1   104.829446: rebalance_domains: rcu-read-lock: 6
<idle>-0     [006] .Ns1   104.829450: rebalance_domains: rcu-read-unlock: 6
trinity-c28-3303  [006] d..1   104.829663: pick_next_task_fair: rcu-read-lock: 6
trinity-c28-3303  [006] d..1   104.829667: load_balance: dst_cpu: 6 cpus: 1,3-4,6-7,9
trinity-c28-3303  [006] d..1   104.829671: load_balance: dst_cpu: 6 cpus: 1,3-4,6-7,9
trinity-c28-3303  [006] d..1   104.829702: pick_next_task_fair: rcu-read-unlock: 6
kdevtmpfs-72    [001] d..1   104.830079: pick_next_task_fair: rcu-read-lock: 1
kdevtmpfs-72    [001] d..1   104.830083: load_balance: dst_cpu: 1 cpus: 1,9
kdevtmpfs-72    [001] d..1   104.830087: load_balance: dst_cpu: 1 cpus: 1,9
kdevtmpfs-72    [001] d..1   104.830097: pick_next_task_fair: rcu-read-unlock: 1
trinity-c37-3313  [000] d..1   104.830218: pick_next_task_fair: rcu-read-lock: 0
trinity-c37-3313  [000] d..1   104.830221: load_balance: dst_cpu: 0 cpus: 0-1,3-4,6-7,9
trinity-c37-3313  [000] d..1   104.830231: load_balance: dst_cpu: 0 cpus: 0-1,3-4,6-7,9
trinity-c37-3313  [000] d..1   104.830237: pick_next_task_fair: rcu-read-unlock: 0
trinity-c28-3303  [006] d..1   104.830912: pick_next_task_fair: rcu-read-lock: 6
trinity-c28-3303  [006] d..1   104.830916: load_balance: dst_cpu: 6 cpus: 1,3-4,6-7,9
trinity-c28-3303  [006] d..1   104.830919: load_balance: dst_cpu: 6 cpus: 1,3-4,6-7,9
trinity-c28-3303  [006] d..1   104.830950: pick_next_task_fair: rcu-read-unlock: 6
kworker/u26:2-413   [005] ..s1   104.831066: rebalance_domains: rcu-read-lock: 5
kworker/u26:2-413   [005] ..s1   104.831071: rebalance_domains: rcu-read-unlock: 5
rcu_preempt-9     [006] d..1   104.831127: pick_next_task_fair: rcu-read-lock: 6
rcu_preempt-9     [006] d..1   104.831130: load_balance: dst_cpu: 6 cpus: 1,3-4,6-7,9
rcu_preempt-9     [006] d..1   104.831133: load_balance: dst_cpu: 6 cpus: 1,3-4,6-7,9
rcu_preempt-9     [006] d..1   104.831138: pick_next_task_fair: rcu-read-unlock: 6
ksoftirqd/6-51    [006] d..1   104.831284: pick_next_task_fair: rcu-read-lock: 6
ksoftirqd/6-51    [006] d..1   104.831287: load_balance: dst_cpu: 6 cpus: 1,3-4,6-7,9
ksoftirqd/6-51    [006] d..1   104.831290: load_balance: dst_cpu: 6 cpus: 1,3-4,6-7,9
ksoftirqd/6-51    [006] d..1   104.831307: pick_next_task_fair: rcu-read-unlock: 6
kworker/u26:2-5519  [005] d..1   104.831458: pick_next_task_fair: rcu-read-lock: 5
kworker/u26:2-5519  [005] d..1   104.831462: load_balance: dst_cpu: 5 cpus: 1,9
kworker/u26:2-5519  [005] d..1   104.831473: load_balance: dst_cpu: 5 cpus: 1,9
kworker/u26:2-5519  [005] d..1   104.831479: pick_next_task_fair: rcu-read-unlock: 5
kworker/u26:2-5519  [005] d..1   104.831796: pick_next_task_fair: rcu-read-lock: 5
kworker/u26:2-5519  [005] d..1   104.831799: load_balance: dst_cpu: 5 cpus: 1,9
kworker/u26:2-5519  [005] dN.1   104.831823: pick_next_task_fair: rcu-read-unlock: 5
ksoftirqd/1-18    [001] d..1   104.831914: pick_next_task_fair: rcu-read-lock: 1
ksoftirqd/1-18    [001] d..1   104.831918: load_balance: dst_cpu: 1 cpus: 1,9
ksoftirqd/1-18    [001] d..1   104.831928: load_balance: dst_cpu: 1 cpus: 1,9
ksoftirqd/1-18    [001] d..1   104.831934: pick_next_task_fair: rcu-read-unlock: 1
kworker/u26:2-5519  [005] ..s2   104.832087: rebalance_domains: rcu-read-lock: 5
kworker/u26:2-5519  [005] ..s2   104.832091: rebalance_domains: rcu-read-unlock: 5
trinity-c37-3313  [000] ..s2   104.832097: rebalance_domains: rcu-read-lock: 0
trinity-c37-3313  [000] ..s2   104.832101: rebalance_domains: rcu-read-unlock: 0
trinity-c37-3313  [000] d..1   104.832188: pick_next_task_fair: rcu-read-lock: 0
trinity-c37-3313  [000] d..1   104.832190: load_balance: dst_cpu: 0 cpus: 0-1,3-4,6-7,9
trinity-c37-3313  [000] d..1   104.832195: load_balance: dst_cpu: 0 cpus: 0-1,3-4,6-7,9
trinity-c37-3313  [000] d..1   104.832206: pick_next_task_fair: rcu-read-unlock: 0
kworker/u26:2-5519  [005] d..1   104.832226: pick_next_task_fair: rcu-read-lock: 5
kworker/u26:2-5519  [005] d..1   104.832230: load_balance: dst_cpu: 5 cpus: 1,9
kworker/u26:2-5519  [005] d..1   104.832239: load_balance: dst_cpu: 5 cpus: 1,9
kworker/u26:2-5519  [005] d..1   104.832244: pick_next_task_fair: rcu-read-unlock: 5
trinity-c12-5517  [008] d..1   104.832468: pick_next_task_fair: rcu-read-lock: 8
trinity-c12-5517  [008] d..1   104.832472: load_balance: dst_cpu: 8 cpus: 0-1,3-4,6-7,9
trinity-c12-5517  [008] d..1   104.832479: load_balance: dst_cpu: 8 cpus: 0-1,3-4,6-7,9
kworker/u26:2-5519  [005] d..1   104.832499: pick_next_task_fair: rcu-read-lock: 5
kworker/u26:2-5519  [005] d..1   104.832502: load_balance: dst_cpu: 5 cpus: 1,9
kworker/u26:2-5519  [005] d..1   104.832506: load_balance: dst_cpu: 5 cpus: 1,9
kworker/u26:2-5519  [005] d..1   104.832520: pick_next_task_fair: rcu-read-unlock: 5
trinity-c28-3303  [006] d..1   104.832533: pick_next_task_fair: rcu-read-lock: 6
trinity-c28-3303  [006] d..1   104.832537: load_balance: dst_cpu: 6 cpus: 1,3-4,6-7,9
trinity-c12-5517  [008] d..1   104.832537: pick_next_task_fair: rcu-read-unlock: 8
trinity-c28-3303  [006] d..1   104.832540: load_balance: dst_cpu: 6 cpus: 1,3-4,6-7,9
trinity-c28-3303  [006] d..1   104.832550: pick_next_task_fair: rcu-read-unlock: 6
ksoftirqd/8-63    [008] d..1   104.832790: pick_next_task_fair: rcu-read-lock: 8
ksoftirqd/8-63    [008] d..1   104.832793: load_balance: dst_cpu: 8 cpus: 0-1,3-4,6-7,9
ksoftirqd/8-63    [008] d..1   104.832797: load_balance: dst_cpu: 8 cpus: 0-1,3-4,6-7,9
ksoftirqd/8-63    [008] d..1   104.832809: pick_next_task_fair: rcu-read-unlock: 8
<idle>-0     [005] .Ns1   104.833083: rebalance_domains: rcu-read-lock: 5
<idle>-0     [005] .Ns1   104.833087: rebalance_domains: rcu-read-unlock: 5
trinity-c78-4927  [000] d..1   104.833225: pick_next_task_fair: rcu-read-lock: 0
trinity-c78-4927  [000] d..1   104.833228: load_balance: dst_cpu: 0 cpus: 0-1,3-4,6-7,9
trinity-c78-4927  [000] d..1   104.833239: load_balance: dst_cpu: 0 cpus: 0-1,3-4,6-7,9
trinity-c78-4927  [000] d..1   104.833260: pick_next_task_fair: rcu-read-unlock: 0
kworker/u26:2-5519  [005] d..1   104.833545: pick_next_task_fair: rcu-read-lock: 5
kworker/u26:2-5519  [005] d..1   104.833548: load_balance: dst_cpu: 5 cpus: 1,9
kworker/u26:2-5519  [005] d..1   104.833785: load_balance: dst_cpu: 5 cpus: 1,9
kworker/u26:2-5519  [005] d..1   104.833799: pick_next_task_fair: rcu-read-unlock: 5
trinity-c28-3303  [006] ..s1   104.834090: rebalance_domains: rcu-read-lock: 6
trinity-c28-3303  [006] ..s1   104.834094: rebalance_domains: rcu-read-unlock: 6
rcu_preempt-9     [006] d..1   104.834193: pick_next_task_fair: rcu-read-lock: 6
rcu_preempt-9     [006] d..1   104.834196: load_balance: dst_cpu: 6 cpus: 1,3-4,6-7,9
rcu_preempt-9     [006] d..1   104.834200: load_balance: dst_cpu: 6 cpus: 1,3-4,6-7,9
rcu_preempt-9     [006] d..1   104.834210: pick_next_task_fair: rcu-read-unlock: 6
ksoftirqd/6-51    [006] d..1   104.834309: pick_next_task_fair: rcu-read-lock: 6
ksoftirqd/6-51    [006] d..1   104.834312: load_balance: dst_cpu: 6 cpus: 1,3-4,6-7,9
ksoftirqd/6-51    [006] d..1   104.834315: load_balance: dst_cpu: 6 cpus: 1,3-4,6-7,9
ksoftirqd/6-51    [006] d..1   104.834323: pick_next_task_fair: rcu-read-unlock: 6
<idle>-0     [000] .Ns1   104.834449: rebalance_domains: rcu-read-lock: 0
<idle>-0     [000] .Ns1   104.834452: load_balance: dst_cpu: 0 cpus: 0-1,3-4,6-7,9
<idle>-0     [000] .Ns2   104.834457: load_balance: dst_cpu: 0 cpus: 0-1,3-4,6-7,9
<idle>-0     [000] .Ns1   104.834459: rebalance_domains: rcu-read-unlock: 0
trinity-c37-3313  [000] d..1   104.834761: pick_next_task_fair: rcu-read-lock: 0
trinity-c37-3313  [000] d..1   104.834764: load_balance: dst_cpu: 0 cpus: 0-1,3-4,6-7,9
trinity-c37-3313  [000] d..1   104.834769: load_balance: dst_cpu: 0 cpus: 0-1,3-4,6-7,9
trinity-c37-3313  [000] d..1   104.834780: pick_next_task_fair: rcu-read-unlock: 0
trinity-c79-3355  [005] .Ns1   104.835721: rebalance_domains: rcu-read-lock: 5
trinity-c79-3355  [005] .Ns1   104.835725: rebalance_domains: rcu-read-unlock: 5
<idle>-0     [008] .Ns1   104.835746: rebalance_domains: rcu-read-lock: 8
<idle>-0     [008] .Ns1   104.835751: load_balance: dst_cpu: 8 cpus: 0-1,3-4,6-7,9
<idle>-0     [008] .Ns2   104.835759: load_balance: dst_cpu: 8 cpus: 0-1,3-4,6-7,9
<idle>-0     [008] .Ns1   104.835761: rebalance_domains: rcu-read-unlock: 8
trinity-c28-3303  [006] d..1   104.835817: pick_next_task_fair: rcu-read-lock: 6
trinity-c28-3303  [006] d..1   104.835820: load_balance: dst_cpu: 6 cpus: 1,3-4,6-7,9
trinity-c28-3303  [006] d..1   104.835824: load_balance: dst_cpu: 6 cpus: 1,3-4,6-7,9
trinity-c28-3303  [006] d..1   104.835829: pick_next_task_fair: rcu-read-unlock: 6
rcu_sched-10    [008] d..1   104.835871: pick_next_task_fair: rcu-read-lock: 8
rcu_sched-10    [008] d..1   104.835874: load_balance: dst_cpu: 8 cpus: 0-1,3-4,6-7,9
rcu_sched-10    [008] d..1   104.835879: load_balance: dst_cpu: 8 cpus: 0-1,3-4,6-7,9
rcu_sched-10    [008] d..1   104.835915: pick_next_task_fair: rcu-read-unlock: 8
trinity-c63-3339  [008] d..1   104.836114: pick_next_task_fair: rcu-read-lock: 8
trinity-c63-3339  [008] d..1   104.836117: load_balance: dst_cpu: 8 cpus: 0-1,3-4,6-7,9
trinity-c63-3339  [008] d..1   104.836122: load_balance: dst_cpu: 8 cpus: 0-1,3-4,6-7,9
trinity-c63-3339  [008] d..1   104.836133: pick_next_task_fair: rcu-read-unlock: 8
ksoftirqd/1-18    [001] ..s.   104.836352: rebalance_domains: rcu-read-lock: 1
ksoftirqd/1-18    [001] ..s.   104.836356: rebalance_domains: rcu-read-unlock: 1
<idle>-0     [000] .Ns1   104.836602: rebalance_domains: rcu-read-lock: 0
<idle>-0     [000] .Ns1   104.836606: rebalance_domains: rcu-read-unlock: 0
migration/0-12    [000] ..s1   104.837059: rebalance_domains: rcu-read-lock: 0
migration/0-12    [000] ..s1   104.837063: rebalance_domains: rcu-read-unlock: 0
migration/6-50    [006] ..s2   104.837087: rebalance_domains: rcu-read-lock: 6
migration/6-50    [006] ..s2   104.837092: rebalance_domains: rcu-read-unlock: 6
<idle>-0     [001] .Ns1   104.843088: rebalance_domains: rcu-read-lock: 1
<idle>-0     [001] .Ns1   104.843093: rebalance_domains: rcu-read-unlock: 1
<idle>-0     [008] .Ns1   104.843269: rebalance_domains: rcu-read-lock: 8
<idle>-0     [008] .Ns1   104.843273: rebalance_domains: rcu-read-unlock: 8
migration/2-24    [002] ..s1   104.855994: rebalance_domains: rcu-read-lock: 2
migration/2-24    [002] ..s1   104.856000: load_balance: dst_cpu: 2 cpus: 6

-- 

Thanks,
Sasha

^ permalink raw reply	[flat|nested] 18+ messages in thread

* Re: [PATCH 3/4] sched: WARN when migrating to an offline CPU
  2017-10-07  2:07             ` Levin, Alexander (Sasha Levin)
@ 2017-10-07  9:15               ` Peter Zijlstra
       [not found]                 ` <20171007174327.ky6g5viokxg5ysdm@sasha-lappy>
  0 siblings, 1 reply; 18+ messages in thread
From: Peter Zijlstra @ 2017-10-07  9:15 UTC (permalink / raw)
  To: Levin, Alexander (Sasha Levin)
  Cc: Sasha Levin, Ingo Molnar, Thomas Gleixner,
	linux-kernel@vger.kernel.org List

On Sat, Oct 07, 2017 at 02:07:26AM +0000, Levin, Alexander (Sasha Levin) wrote:
> And quite a few lines of your added trace (lmk if you need more, or all):

Yeah, could you please upload all of it somewhere? That chunk didn't
include any hotplug bits at all.

^ permalink raw reply	[flat|nested] 18+ messages in thread

* Re: [PATCH 3/4] sched: WARN when migrating to an offline CPU
       [not found]                 ` <20171007174327.ky6g5viokxg5ysdm@sasha-lappy>
@ 2017-10-09  8:04                   ` Peter Zijlstra
  2017-10-10  1:18                     ` Levin, Alexander (Sasha Levin)
  0 siblings, 1 reply; 18+ messages in thread
From: Peter Zijlstra @ 2017-10-09  8:04 UTC (permalink / raw)
  To: Levin, Alexander (Sasha Levin)
  Cc: Sasha Levin, Ingo Molnar, Thomas Gleixner,
	linux-kernel@vger.kernel.org List

On Sat, Oct 07, 2017 at 05:43:32PM +0000, Levin, Alexander (Sasha Levin) wrote:
> On Sat, Oct 07, 2017 at 11:15:17AM +0200, Peter Zijlstra wrote:
> >On Sat, Oct 07, 2017 at 02:07:26AM +0000, Levin, Alexander (Sasha Levin) wrote:
> >> And quite a few lines of your added trace (lmk if you need more, or all):
> >
> >Yeah, could you please upload all of it somewhere? That chunk didn't
> >include any hotplug bits at all.
> 
> Attached. It's the stack trace followed by everything else.

Much thanks, clue:

[ 2073.495414] NOHZ: local_softirq_pending 202 -- (TIMER_SOFTIRQ | RCU_SOFTIRQ)

cpuhp/2-22    [002] ....   104.797971: sched_cpu_deactivate: not-active: 2 mask: 0-1,3-4,6-7,9

cpuhp/2-22    [002] ....   104.825166: sched_cpu_deactivate: rcu-sync: 2

migration/2-24    [002] ..s1   104.855994: rebalance_domains: rcu-read-lock: 2
migration/2-24    [002] ..s1   104.856000: load_balance: dst_cpu: 2 cpus: 6


supposed fix; could you please verify?

---
 kernel/sched/fair.c | 7 +++++++
 1 file changed, 7 insertions(+)

diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c
index 350dbec01523..35f0168fb609 100644
--- a/kernel/sched/fair.c
+++ b/kernel/sched/fair.c
@@ -8392,6 +8392,13 @@ static int should_we_balance(struct lb_env *env)
 	struct sched_group *sg = env->sd->groups;
 	int cpu, balance_cpu = -1;
 
+	/*
+	 * Ensure the balancing environment is consistent; can happen
+	 * when the softirq triggers 'during' hotplug.
+	 */
+	if (!cpumask_test_cpu(env->dst_cpu, env->cpus))
+		return 0;
+
 	/*
 	 * In the newly idle case, we will allow all the cpu's
 	 * to do the newly idle load balance.

^ permalink raw reply related	[flat|nested] 18+ messages in thread

* Re: [PATCH 3/4] sched: WARN when migrating to an offline CPU
  2017-10-09  8:04                   ` Peter Zijlstra
@ 2017-10-10  1:18                     ` Levin, Alexander (Sasha Levin)
  0 siblings, 0 replies; 18+ messages in thread
From: Levin, Alexander (Sasha Levin) @ 2017-10-10  1:18 UTC (permalink / raw)
  To: Peter Zijlstra
  Cc: Sasha Levin, Ingo Molnar, Thomas Gleixner,
	linux-kernel@vger.kernel.org List

On Mon, Oct 09, 2017 at 10:04:45AM +0200, Peter Zijlstra wrote:
>On Sat, Oct 07, 2017 at 05:43:32PM +0000, Levin, Alexander (Sasha Levin) wrote:
>> On Sat, Oct 07, 2017 at 11:15:17AM +0200, Peter Zijlstra wrote:
>> >On Sat, Oct 07, 2017 at 02:07:26AM +0000, Levin, Alexander (Sasha Levin) wrote:
>> >> And quite a few lines of your added trace (lmk if you need more, or all):
>> >
>> >Yeah, could you please upload all of it somewhere? That chunk didn't
>> >include any hotplug bits at all.
>>
>> Attached. It's the stack trace followed by everything else.
>
>Much thanks, clue:
>
>[ 2073.495414] NOHZ: local_softirq_pending 202 -- (TIMER_SOFTIRQ | RCU_SOFTIRQ)
>
>cpuhp/2-22    [002] ....   104.797971: sched_cpu_deactivate: not-active: 2 mask: 0-1,3-4,6-7,9
>
>cpuhp/2-22    [002] ....   104.825166: sched_cpu_deactivate: rcu-sync: 2
>
>migration/2-24    [002] ..s1   104.855994: rebalance_domains: rcu-read-lock: 2
>migration/2-24    [002] ..s1   104.856000: load_balance: dst_cpu: 2 cpus: 6
>
>
>supposed fix; could you please verify?

Patch works, thanks!

-- 

Thanks,
Sasha

^ permalink raw reply	[flat|nested] 18+ messages in thread

end of thread, other threads:[~2017-10-10  1:19 UTC | newest]

Thread overview: 18+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2017-09-07 15:03 [PATCH 0/4] sched: Fix some load-balancer vs hotplug holes Peter Zijlstra
2017-09-07 15:03 ` [PATCH 1/4] sched/fair: Avoid newidle balance for !active CPUs Peter Zijlstra
2017-09-12 18:04   ` [tip:sched/urgent] " tip-bot for Peter Zijlstra
2017-09-07 15:03 ` [PATCH 2/4] sched/fair: Plug hole between hotplug and active load_balance Peter Zijlstra
2017-09-12 18:05   ` [tip:sched/urgent] sched/fair: Plug hole between hotplug and active_load_balance() tip-bot for Peter Zijlstra
2017-09-07 15:03 ` [PATCH 3/4] sched: WARN when migrating to an offline CPU Peter Zijlstra
2017-09-12 18:05   ` [tip:sched/urgent] sched/core: WARN() " tip-bot for Peter Zijlstra
2017-09-28  9:14   ` [PATCH 3/4] sched: WARN " Sasha Levin
2017-09-28 10:35     ` Peter Zijlstra
2017-09-28 11:03       ` Levin, Alexander (Sasha Levin)
2017-09-28 11:42         ` Peter Zijlstra
2017-09-29 11:11           ` Peter Zijlstra
2017-10-07  2:07             ` Levin, Alexander (Sasha Levin)
2017-10-07  9:15               ` Peter Zijlstra
     [not found]                 ` <20171007174327.ky6g5viokxg5ysdm@sasha-lappy>
2017-10-09  8:04                   ` Peter Zijlstra
2017-10-10  1:18                     ` Levin, Alexander (Sasha Levin)
2017-09-07 15:03 ` [PATCH 4/4] sched/debug: Add debugfs knob for "sched_debug" Peter Zijlstra
2017-09-12 18:05   ` [tip:sched/urgent] " tip-bot for Peter Zijlstra

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).