From: "tip-bot2 for Peter Zijlstra" <tip-bot2@linutronix.de>
To: linux-tip-commits@vger.kernel.org
Cc: "Peter Zijlstra (Intel)" <peterz@infradead.org>,
Valentin Schneider <valentin.schneider@arm.com>,
x86@kernel.org, linux-kernel@vger.kernel.org
Subject: [tip: sched/core] sched: Use cpu_dying() to fix balance_push vs hotplug-rollback
Date: Fri, 16 Apr 2021 15:53:44 -0000 [thread overview]
Message-ID: <161858842443.29796.7574048202320693227.tip-bot2@tip-bot2> (raw)
In-Reply-To: <YHgAYef83VQhKdC2@hirez.programming.kicks-ass.net>
The following commit has been merged into the sched/core branch of tip:
Commit-ID: b5c4477366fb5e6a2f0f38742c33acd666c07698
Gitweb: https://git.kernel.org/tip/b5c4477366fb5e6a2f0f38742c33acd666c07698
Author: Peter Zijlstra <peterz@infradead.org>
AuthorDate: Thu, 21 Jan 2021 16:09:32 +01:00
Committer: Peter Zijlstra <peterz@infradead.org>
CommitterDate: Fri, 16 Apr 2021 17:06:32 +02:00
sched: Use cpu_dying() to fix balance_push vs hotplug-rollback
Use the new cpu_dying() state to simplify and fix the balance_push()
vs CPU hotplug rollback state.
Specifically, we currently rely on notifiers sched_cpu_dying() /
sched_cpu_activate() to terminate balance_push, however if the
cpu_down() fails when we're past sched_cpu_deactivate(), it should
terminate balance_push at that point and not wait until we hit
sched_cpu_activate().
Similarly, when cpu_up() fails and we're going back down, balance_push
should be active, where it currently is not.
So instead, make sure balance_push is enabled below SCHED_AP_ACTIVE
(when !cpu_active()), and gate it's utility with cpu_dying().
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Reviewed-by: Valentin Schneider <valentin.schneider@arm.com>
Link: https://lkml.kernel.org/r/YHgAYef83VQhKdC2@hirez.programming.kicks-ass.net
---
kernel/sched/core.c | 26 +++++++++++++++-----------
kernel/sched/sched.h | 1 -
2 files changed, 15 insertions(+), 12 deletions(-)
diff --git a/kernel/sched/core.c b/kernel/sched/core.c
index 95bd6ab..7d031da 100644
--- a/kernel/sched/core.c
+++ b/kernel/sched/core.c
@@ -1811,7 +1811,7 @@ static inline bool is_cpu_allowed(struct task_struct *p, int cpu)
return cpu_online(cpu);
/* Regular kernel threads don't get to stay during offline. */
- if (cpu_rq(cpu)->balance_push)
+ if (cpu_dying(cpu))
return false;
/* But are allowed during online. */
@@ -7638,6 +7638,9 @@ static DEFINE_PER_CPU(struct cpu_stop_work, push_work);
/*
* Ensure we only run per-cpu kthreads once the CPU goes !active.
+ *
+ * This is enabled below SCHED_AP_ACTIVE; when !cpu_active(), but only
+ * effective when the hotplug motion is down.
*/
static void balance_push(struct rq *rq)
{
@@ -7645,12 +7648,19 @@ static void balance_push(struct rq *rq)
lockdep_assert_held(&rq->lock);
SCHED_WARN_ON(rq->cpu != smp_processor_id());
+
/*
* Ensure the thing is persistent until balance_push_set(.on = false);
*/
rq->balance_callback = &balance_push_callback;
/*
+ * Only active while going offline.
+ */
+ if (!cpu_dying(rq->cpu))
+ return;
+
+ /*
* Both the cpu-hotplug and stop task are in this case and are
* required to complete the hotplug process.
*
@@ -7703,7 +7713,6 @@ static void balance_push_set(int cpu, bool on)
struct rq_flags rf;
rq_lock_irqsave(rq, &rf);
- rq->balance_push = on;
if (on) {
WARN_ON_ONCE(rq->balance_callback);
rq->balance_callback = &balance_push_callback;
@@ -7828,8 +7837,8 @@ int sched_cpu_activate(unsigned int cpu)
struct rq_flags rf;
/*
- * Make sure that when the hotplug state machine does a roll-back
- * we clear balance_push. Ideally that would happen earlier...
+ * Clear the balance_push callback and prepare to schedule
+ * regular tasks.
*/
balance_push_set(cpu, false);
@@ -8014,12 +8023,6 @@ int sched_cpu_dying(unsigned int cpu)
}
rq_unlock_irqrestore(rq, &rf);
- /*
- * Now that the CPU is offline, make sure we're welcome
- * to new tasks once we come back up.
- */
- balance_push_set(cpu, false);
-
calc_load_migrate(rq);
update_max_interval();
hrtick_clear(rq);
@@ -8204,7 +8207,7 @@ void __init sched_init(void)
rq->sd = NULL;
rq->rd = NULL;
rq->cpu_capacity = rq->cpu_capacity_orig = SCHED_CAPACITY_SCALE;
- rq->balance_callback = NULL;
+ rq->balance_callback = &balance_push_callback;
rq->active_balance = 0;
rq->next_balance = jiffies;
rq->push_cpu = 0;
@@ -8251,6 +8254,7 @@ void __init sched_init(void)
#ifdef CONFIG_SMP
idle_thread_set_boot_cpu();
+ balance_push_set(smp_processor_id(), false);
#endif
init_sched_fair_class();
diff --git a/kernel/sched/sched.h b/kernel/sched/sched.h
index cbb0b01..7e7e936 100644
--- a/kernel/sched/sched.h
+++ b/kernel/sched/sched.h
@@ -983,7 +983,6 @@ struct rq {
unsigned long cpu_capacity_orig;
struct callback_head *balance_callback;
- unsigned char balance_push;
unsigned char nohz_idle_balance;
unsigned char idle_balance;
prev parent reply other threads:[~2021-04-16 15:54 UTC|newest]
Thread overview: 29+ messages / expand[flat|nested] mbox.gz Atom feed top
2021-03-10 14:52 [PATCH 0/3] sched: Fix remaining balance_push vs hotplug hole Peter Zijlstra
2021-03-10 14:52 ` [PATCH 1/3] cpumask: Make cpu_{online,possible,present,active}() inline Peter Zijlstra
2021-04-16 15:53 ` [tip: sched/core] " tip-bot2 for Peter Zijlstra
2021-03-10 14:53 ` [PATCH 2/3] cpumask: Introduce DYING mask Peter Zijlstra
2021-03-21 19:30 ` Qais Yousef
2021-03-22 15:07 ` Steven Rostedt
2021-04-12 10:55 ` Peter Zijlstra
2021-04-12 11:16 ` Qais Yousef
2021-04-16 15:53 ` [tip: sched/core] " tip-bot2 for Peter Zijlstra
2021-03-10 14:53 ` [PATCH 3/3] sched: Use cpu_dying() to fix balance_push vs hotplug-rollback Peter Zijlstra
2021-03-11 15:13 ` Valentin Schneider
2021-03-11 16:42 ` Peter Zijlstra
2021-04-12 12:03 ` Peter Zijlstra
2021-04-12 17:22 ` Valentin Schneider
2021-04-13 6:51 ` Peter Zijlstra
2021-04-15 8:59 ` Peter Zijlstra
2021-04-15 14:32 ` Valentin Schneider
2021-04-15 15:29 ` Peter Zijlstra
2021-04-15 15:34 ` Valentin Schneider
2021-04-19 10:56 ` Vincent Donnefort
2021-04-20 9:46 ` Vincent Donnefort
2021-04-20 14:20 ` Peter Zijlstra
2021-04-20 14:39 ` Peter Zijlstra
2021-04-20 14:58 ` Peter Zijlstra
2021-04-20 16:53 ` Vincent Donnefort
2021-04-20 18:07 ` Peter Zijlstra
2021-04-21 9:32 ` Valentin Schneider
2021-04-22 7:36 ` [tip: sched/core] cpumask/hotplug: Fix cpu_dying() state tracking tip-bot2 for Peter Zijlstra
2021-04-16 15:53 ` tip-bot2 for Peter Zijlstra [this message]
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=161858842443.29796.7574048202320693227.tip-bot2@tip-bot2 \
--to=tip-bot2@linutronix.de \
--cc=linux-kernel@vger.kernel.org \
--cc=linux-tip-commits@vger.kernel.org \
--cc=peterz@infradead.org \
--cc=valentin.schneider@arm.com \
--cc=x86@kernel.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).