From: "tip-bot2 for Peter Zijlstra" <tip-bot2@linutronix.de>
To: linux-tip-commits@vger.kernel.org
Cc: Mel Gorman <mgorman@techsingularity.net>,
"Peter Zijlstra (Intel)" <peterz@infradead.org>,
x86@kernel.org, linux-kernel@vger.kernel.org
Subject: [tip: sched/urgent] sched: Fix data-race in wakeup
Date: Thu, 19 Nov 2020 09:55:05 -0000 [thread overview]
Message-ID: <160577970512.11244.15276468517015655452.tip-bot2@tip-bot2> (raw)
In-Reply-To: <20201117083016.GK3121392@hirez.programming.kicks-ass.net>
The following commit has been merged into the sched/urgent branch of tip:
Commit-ID: f97bb5272d9e95d400d6c8643ebb146b3e3e7842
Gitweb: https://git.kernel.org/tip/f97bb5272d9e95d400d6c8643ebb146b3e3e7842
Author: Peter Zijlstra <peterz@infradead.org>
AuthorDate: Tue, 17 Nov 2020 09:08:41 +01:00
Committer: Peter Zijlstra <peterz@infradead.org>
CommitterDate: Tue, 17 Nov 2020 13:15:27 +01:00
sched: Fix data-race in wakeup
Mel reported that on some ARM64 platforms loadavg goes bananas and
Will tracked it down to the following race:
CPU0 CPU1
schedule()
prev->sched_contributes_to_load = X;
deactivate_task(prev);
try_to_wake_up()
if (p->on_rq &&) // false
if (smp_load_acquire(&p->on_cpu) && // true
ttwu_queue_wakelist())
p->sched_remote_wakeup = Y;
smp_store_release(prev->on_cpu, 0);
where both p->sched_contributes_to_load and p->sched_remote_wakeup are
in the same word, and thus the stores X and Y race (and can clobber
one another's data).
Whereas prior to commit c6e7bd7afaeb ("sched/core: Optimize ttwu()
spinning on p->on_cpu") the p->on_cpu handoff serialized access to
p->sched_remote_wakeup (just as it still does with
p->sched_contributes_to_load) that commit broke that by calling
ttwu_queue_wakelist() with p->on_cpu != 0.
However, due to
p->XXX = X ttwu()
schedule() if (p->on_rq && ...) // false
smp_mb__after_spinlock() if (smp_load_acquire(&p->on_cpu) &&
deactivate_task() ttwu_queue_wakelist())
p->on_rq = 0; p->sched_remote_wakeup = Y;
We can be sure any 'current' store is complete and 'current' is
guaranteed asleep. Therefore we can move p->sched_remote_wakeup into
the current flags word.
Note: while the observed failure was loadavg accounting gone wrong due
to ttwu() cobbering p->sched_contributes_to_load, the reverse problem
is also possible where schedule() clobbers p->sched_remote_wakeup,
this could result in enqueue_entity() wrecking ->vruntime and causing
scheduling artifacts.
Fixes: c6e7bd7afaeb ("sched/core: Optimize ttwu() spinning on p->on_cpu")
Reported-by: Mel Gorman <mgorman@techsingularity.net>
Debugged-by: Will Deacon <will@kernel.org>
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Link: https://lkml.kernel.org/r/20201117083016.GK3121392@hirez.programming.kicks-ass.net
---
include/linux/sched.h | 16 +++++++++++++++-
1 file changed, 15 insertions(+), 1 deletion(-)
diff --git a/include/linux/sched.h b/include/linux/sched.h
index d383cf0..0e91b45 100644
--- a/include/linux/sched.h
+++ b/include/linux/sched.h
@@ -769,7 +769,6 @@ struct task_struct {
unsigned sched_reset_on_fork:1;
unsigned sched_contributes_to_load:1;
unsigned sched_migrated:1;
- unsigned sched_remote_wakeup:1;
#ifdef CONFIG_PSI
unsigned sched_psi_wake_requeue:1;
#endif
@@ -779,6 +778,21 @@ struct task_struct {
/* Unserialized, strictly 'current' */
+ /*
+ * This field must not be in the scheduler word above due to wakelist
+ * queueing no longer being serialized by p->on_cpu. However:
+ *
+ * p->XXX = X; ttwu()
+ * schedule() if (p->on_rq && ..) // false
+ * smp_mb__after_spinlock(); if (smp_load_acquire(&p->on_cpu) && //true
+ * deactivate_task() ttwu_queue_wakelist())
+ * p->on_rq = 0; p->sched_remote_wakeup = Y;
+ *
+ * guarantees all stores of 'current' are visible before
+ * ->sched_remote_wakeup gets used, so it can be in this word.
+ */
+ unsigned sched_remote_wakeup:1;
+
/* Bit to tell LSMs we're in execve(): */
unsigned in_execve:1;
unsigned in_iowait:1;
prev parent reply other threads:[~2020-11-19 9:55 UTC|newest]
Thread overview: 36+ messages / expand[flat|nested] mbox.gz Atom feed top
2020-11-16 9:10 Loadavg accounting error on arm64 Mel Gorman
2020-11-16 11:49 ` Mel Gorman
2020-11-16 12:00 ` Mel Gorman
2020-11-16 12:53 ` Peter Zijlstra
2020-11-16 12:58 ` Peter Zijlstra
2020-11-16 15:29 ` Mel Gorman
2020-11-16 16:42 ` Mel Gorman
2020-11-16 16:49 ` Peter Zijlstra
2020-11-16 17:24 ` Mel Gorman
2020-11-16 17:41 ` Will Deacon
2020-11-16 12:46 ` Peter Zijlstra
2020-11-16 12:58 ` Mel Gorman
2020-11-16 13:11 ` Will Deacon
2020-11-16 13:37 ` Mel Gorman
2020-11-16 14:20 ` Peter Zijlstra
2020-11-16 15:52 ` Mel Gorman
2020-11-16 16:54 ` Peter Zijlstra
2020-11-16 17:16 ` Mel Gorman
2020-11-16 19:31 ` Mel Gorman
2020-11-17 8:30 ` [PATCH] sched: Fix data-race in wakeup Peter Zijlstra
2020-11-17 9:15 ` Will Deacon
2020-11-17 9:29 ` Peter Zijlstra
2020-11-17 9:46 ` Peter Zijlstra
2020-11-17 10:36 ` Will Deacon
2020-11-17 12:52 ` Valentin Schneider
2020-11-17 15:37 ` Valentin Schneider
2020-11-17 16:13 ` Peter Zijlstra
2020-11-17 19:32 ` Valentin Schneider
2020-11-18 8:05 ` Peter Zijlstra
2020-11-18 9:51 ` Valentin Schneider
2020-11-18 13:33 ` Marco Elver
2020-11-17 9:38 ` [PATCH] sched: Fix rq->nr_iowait ordering Peter Zijlstra
2020-11-17 11:43 ` Mel Gorman
2020-11-19 9:55 ` [tip: sched/urgent] " tip-bot2 for Peter Zijlstra
2020-11-17 12:40 ` [PATCH] sched: Fix data-race in wakeup Mel Gorman
2020-11-19 9:55 ` tip-bot2 for Peter Zijlstra [this message]
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=160577970512.11244.15276468517015655452.tip-bot2@tip-bot2 \
--to=tip-bot2@linutronix.de \
--cc=linux-kernel@vger.kernel.org \
--cc=linux-tip-commits@vger.kernel.org \
--cc=mgorman@techsingularity.net \
--cc=peterz@infradead.org \
--cc=x86@kernel.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).