From: Peter Zijlstra <peterz@infradead.org>
To: tglx@linutronix.de, frederic@kernel.org
Cc: linux-kernel@vger.kernel.org, x86@kernel.org, cai@lca.pw,
mgorman@techsingularity.net, peterz@infradead.org
Subject: [RFC][PATCH 6/7] sched: Add rq::ttwu_pending
Date: Tue, 26 May 2020 18:11:03 +0200 [thread overview]
Message-ID: <20200526161908.070399698@infradead.org> (raw)
In-Reply-To: 20200526161057.531933155@infradead.org
In preparation of removing rq->wake_list, replace the
!list_empty(rq->wake_list) with rq->ttwu_pending. This is not fully
equivalent as this new variable is racy.
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
---
kernel/sched/core.c | 13 +++++++++++--
kernel/sched/debug.c | 1 -
kernel/sched/fair.c | 2 +-
kernel/sched/sched.h | 4 +++-
4 files changed, 15 insertions(+), 5 deletions(-)
--- a/kernel/sched/core.c
+++ b/kernel/sched/core.c
@@ -2275,13 +2275,21 @@ static int ttwu_remote(struct task_struc
void sched_ttwu_pending(void)
{
struct rq *rq = this_rq();
- struct llist_node *llist = llist_del_all(&rq->wake_list);
+ struct llist_node *llist;
struct task_struct *p, *t;
struct rq_flags rf;
+ llist = llist_del_all(&rq->wake_list);
if (!llist)
return;
+ /*
+ * rq::ttwu_pending racy indication of out-standing wakeups.
+ * Races such that false-negatives are possible, since they
+ * are shorter lived that false-positives would be.
+ */
+ WRITE_ONCE(rq->ttwu_pending, 0);
+
rq_lock_irqsave(rq, &rf);
update_rq_clock(rq);
@@ -2312,6 +2320,7 @@ static void ttwu_queue_remote(struct tas
p->sched_remote_wakeup = !!(wake_flags & WF_MIGRATED);
+ WRITE_ONCE(rq->ttwu_pending, 1);
if (llist_add(&p->wake_entry, &rq->wake_list)) {
if (!set_nr_if_polling(rq->idle))
smp_call_function_single_async(cpu, &rq->wake_csd);
@@ -4668,7 +4677,7 @@ int idle_cpu(int cpu)
return 0;
#ifdef CONFIG_SMP
- if (!llist_empty(&rq->wake_list))
+ if (rq->ttwu_pending)
return 0;
#endif
--- a/kernel/sched/debug.c
+++ b/kernel/sched/debug.c
@@ -638,7 +638,6 @@ do { \
P(nr_running);
P(nr_switches);
- P(nr_load_updates);
P(nr_uninterruptible);
PN(next_balance);
SEQ_printf(m, " .%-30s: %ld\n", "curr->pid", (long)(task_pid_nr(rq->curr)));
--- a/kernel/sched/fair.c
+++ b/kernel/sched/fair.c
@@ -8590,7 +8590,7 @@ static int idle_cpu_without(int cpu, str
*/
#ifdef CONFIG_SMP
- if (!llist_empty(&rq->wake_list))
+ if (rq->ttwu_pending)
return 0;
#endif
--- a/kernel/sched/sched.h
+++ b/kernel/sched/sched.h
@@ -895,7 +895,9 @@ struct rq {
atomic_t nohz_flags;
#endif /* CONFIG_NO_HZ_COMMON */
- unsigned long nr_load_updates;
+#ifdef CONFIG_SMP
+ unsigned int ttwu_pending;
+#endif
u64 nr_switches;
#ifdef CONFIG_UCLAMP_TASK
next prev parent reply other threads:[~2020-05-26 16:24 UTC|newest]
Thread overview: 62+ messages / expand[flat|nested] mbox.gz Atom feed top
2020-05-26 16:10 [RFC][PATCH 0/7] Fix the scheduler-IPI mess Peter Zijlstra
2020-05-26 16:10 ` [RFC][PATCH 1/7] sched: Fix smp_call_function_single_async() usage for ILB Peter Zijlstra
2020-05-26 23:56 ` Frederic Weisbecker
2020-05-27 10:23 ` Vincent Guittot
2020-05-27 11:28 ` Frederic Weisbecker
2020-05-27 12:07 ` Vincent Guittot
2020-05-29 15:26 ` Valentin Schneider
2020-06-01 9:52 ` [tip: sched/core] " tip-bot2 for Peter Zijlstra
2020-06-01 11:40 ` Frederic Weisbecker
2020-05-26 16:10 ` [RFC][PATCH 2/7] smp: Optimize flush_smp_call_function_queue() Peter Zijlstra
2020-05-28 12:28 ` Frederic Weisbecker
2020-06-01 9:52 ` [tip: sched/core] " tip-bot2 for Peter Zijlstra
2020-05-26 16:11 ` [RFC][PATCH 3/7] smp: Move irq_work_run() out of flush_smp_call_function_queue() Peter Zijlstra
2020-05-29 13:04 ` Frederic Weisbecker
2020-06-01 9:52 ` [tip: sched/core] " tip-bot2 for Peter Zijlstra
2020-05-26 16:11 ` [RFC][PATCH 4/7] smp: Optimize send_call_function_single_ipi() Peter Zijlstra
2020-05-27 9:56 ` Peter Zijlstra
2020-05-27 10:15 ` Peter Zijlstra
2020-05-27 15:56 ` Paul E. McKenney
2020-05-27 16:35 ` Peter Zijlstra
2020-05-27 17:12 ` Peter Zijlstra
2020-05-27 19:39 ` Paul E. McKenney
2020-05-28 1:35 ` Joel Fernandes
2020-05-28 8:59 ` [tip: core/rcu] rcu: Allow for smp_call_function() running callbacks from idle tip-bot2 for Peter Zijlstra
2021-01-21 16:56 ` [RFC][PATCH 4/7] smp: Optimize send_call_function_single_ipi() Peter Zijlstra
2021-01-22 0:20 ` Paul E. McKenney
2021-01-22 8:31 ` Peter Zijlstra
2021-01-22 15:35 ` Paul E. McKenney
2020-05-29 13:01 ` Frederic Weisbecker
2020-06-01 9:52 ` [tip: sched/core] " tip-bot2 for Peter Zijlstra
2020-05-26 16:11 ` [RFC][PATCH 5/7] irq_work, smp: Allow irq_work on call_single_queue Peter Zijlstra
2020-05-28 23:40 ` Frederic Weisbecker
2020-05-29 13:36 ` Peter Zijlstra
2020-06-05 9:37 ` Peter Zijlstra
2020-06-05 15:02 ` Frederic Weisbecker
2020-06-05 16:17 ` Peter Zijlstra
2020-06-05 15:24 ` Kees Cook
2020-06-10 13:24 ` Frederic Weisbecker
2020-06-01 9:52 ` [tip: sched/core] " tip-bot2 for Peter Zijlstra
2020-05-26 16:11 ` Peter Zijlstra [this message]
2020-06-01 9:52 ` [tip: sched/core] sched: Add rq::ttwu_pending tip-bot2 for Peter Zijlstra
2020-05-26 16:11 ` [RFC][PATCH 7/7] sched: Replace rq::wake_list Peter Zijlstra
2020-05-29 15:10 ` Valdis Klētnieks
2020-06-01 9:52 ` [tip: sched/core] " tip-bot2 for Peter Zijlstra
2020-06-02 15:16 ` Frederic Weisbecker
2020-06-04 14:18 ` [RFC][PATCH 7/7] " Guenter Roeck
2020-06-05 0:24 ` Eric Biggers
2020-06-05 7:41 ` Peter Zijlstra
2020-06-05 16:15 ` Eric Biggers
2020-06-06 23:13 ` Guenter Roeck
2020-06-09 20:21 ` Eric Biggers
2020-06-09 21:25 ` Guenter Roeck
2020-06-09 21:38 ` Eric Biggers
2020-06-09 22:06 ` Peter Zijlstra
2020-06-09 23:03 ` Guenter Roeck
2020-06-10 9:09 ` Peter Zijlstra
2020-06-18 17:57 ` Steven Rostedt
2020-06-18 19:06 ` Guenter Roeck
2020-06-09 22:07 ` Peter Zijlstra
2020-06-05 8:10 ` Peter Zijlstra
2020-06-05 13:33 ` Guenter Roeck
2020-06-05 14:09 ` Peter Zijlstra
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20200526161908.070399698@infradead.org \
--to=peterz@infradead.org \
--cc=cai@lca.pw \
--cc=frederic@kernel.org \
--cc=linux-kernel@vger.kernel.org \
--cc=mgorman@techsingularity.net \
--cc=tglx@linutronix.de \
--cc=x86@kernel.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).