From: Peter Zijlstra <peterz@infradead.org>
To: mingo@kernel.org, oleg@redhat.com
Cc: linux-kernel@vger.kernel.org, peterz@infradead.org,
paulmck@linux.vnet.ibm.com, boqun.feng@gmail.com, corbet@lwn.net,
mhocko@kernel.org, dhowells@redhat.com,
torvalds@linux-foundation.org, will.deacon@arm.com
Subject: [PATCH 3/4] sched: Fix a race in try_to_wake_up() vs schedule()
Date: Mon, 02 Nov 2015 14:29:04 +0100 [thread overview]
Message-ID: <20151102134940.944089740@infradead.org> (raw)
In-Reply-To: 20151102132901.157178466@infradead.org
[-- Attachment #1: peterz-sched-fix-ttwu-race.patch --]
[-- Type: text/plain, Size: 2503 bytes --]
Oleg noticed that its possible to falsely observe p->on_cpu == 0 such
that we'll prematurely continue with the wakeup and effectively run p on
two CPUs at the same time.
Even though the overlap is very limited; the task is in the middle of
being scheduled out; it could still result in corruption of the
scheduler data structures.
CPU0 CPU1
set_current_state(...)
<preempt_schedule>
context_switch(X, Y)
prepare_lock_switch(Y)
Y->on_cpu = 1;
finish_lock_switch(X)
store_release(X->on_cpu, 0);
try_to_wake_up(X)
LOCK(p->pi_lock);
t = X->on_cpu; // 0
context_switch(Y, X)
prepare_lock_switch(X)
X->on_cpu = 1;
finish_lock_switch(Y)
store_release(Y->on_cpu, 0);
</preempt_schedule>
schedule();
deactivate_task(X);
X->on_rq = 0;
if (X->on_rq) // false
if (t) while (X->on_cpu)
cpu_relax();
context_switch(X, ..)
finish_lock_switch(X)
store_release(X->on_cpu, 0);
Avoid the load of X->on_cpu being hoisted over the X->on_rq load.
Reported-by: Oleg Nesterov <oleg@redhat.com>
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
---
kernel/sched/core.c | 19 +++++++++++++++++++
1 file changed, 19 insertions(+)
--- a/kernel/sched/core.c
+++ b/kernel/sched/core.c
@@ -2084,6 +2084,25 @@ try_to_wake_up(struct task_struct *p, un
#ifdef CONFIG_SMP
/*
+ * Ensure we load p->on_cpu _after_ p->on_rq, otherwise it would be
+ * possible to, falsely, observe p->on_cpu == 0.
+ *
+ * One must be running (->on_cpu == 1) in order to remove oneself
+ * from the runqueue.
+ *
+ * [S] ->on_cpu = 1; [L] ->on_rq
+ * UNLOCK rq->lock
+ * RMB
+ * LOCK rq->lock
+ * [S] ->on_rq = 0; [L] ->on_cpu
+ *
+ * Pairs with the full barrier implied in the UNLOCK+LOCK on rq->lock
+ * from the consecutive calls to schedule(); the first switching to our
+ * task, the second putting it to sleep.
+ */
+ smp_rmb();
+
+ /*
* If the owning (remote) cpu is still in the middle of schedule() with
* this task as prev, wait until its done referencing the task.
*/
next prev parent reply other threads:[~2015-11-02 13:52 UTC|newest]
Thread overview: 78+ messages / expand[flat|nested] mbox.gz Atom feed top
2015-11-02 13:29 [PATCH 0/4] scheduler ordering bits Peter Zijlstra
2015-11-02 13:29 ` [PATCH 1/4] sched: Better document the try_to_wake_up() barriers Peter Zijlstra
2015-12-04 0:09 ` Byungchul Park
2015-12-04 0:58 ` Byungchul Park
2015-11-02 13:29 ` [PATCH 2/4] sched: Document Program-Order guarantees Peter Zijlstra
2015-11-02 20:27 ` Paul Turner
2015-11-02 20:34 ` Peter Zijlstra
2015-11-02 22:09 ` Paul Turner
2015-11-02 22:12 ` Peter Zijlstra
2015-11-20 10:02 ` Peter Zijlstra
2015-11-20 14:08 ` Boqun Feng
2015-11-20 14:18 ` Peter Zijlstra
2015-11-20 14:21 ` Boqun Feng
2015-11-20 19:41 ` Peter Zijlstra
2015-11-02 13:29 ` Peter Zijlstra [this message]
2015-11-02 13:29 ` [PATCH 4/4] locking: Introduce smp_cond_acquire() Peter Zijlstra
2015-11-02 13:57 ` Peter Zijlstra
2015-11-02 17:43 ` Will Deacon
2015-11-03 1:14 ` Paul E. McKenney
2015-11-03 1:25 ` Linus Torvalds
2015-11-02 17:42 ` Will Deacon
2015-11-02 18:08 ` Linus Torvalds
2015-11-02 18:37 ` Will Deacon
2015-11-02 19:17 ` Linus Torvalds
2015-11-02 19:57 ` Will Deacon
2015-11-02 20:23 ` Peter Zijlstra
2015-11-02 21:56 ` Peter Zijlstra
2015-11-03 1:57 ` Paul E. McKenney
2015-11-03 19:40 ` Linus Torvalds
2015-11-04 3:57 ` Paul E. McKenney
2015-11-04 4:43 ` Linus Torvalds
2015-11-04 12:54 ` Paul E. McKenney
2015-11-02 20:36 ` David Howells
2015-11-02 20:40 ` Peter Zijlstra
2015-11-02 21:11 ` Linus Torvalds
2015-11-03 17:59 ` Oleg Nesterov
2015-11-03 18:23 ` Peter Zijlstra
2015-11-11 9:39 ` Boqun Feng
2015-11-11 10:34 ` Boqun Feng
2015-11-11 19:53 ` Oleg Nesterov
2015-11-12 13:50 ` Paul E. McKenney
2015-11-11 12:12 ` Peter Zijlstra
2015-11-11 19:39 ` Oleg Nesterov
2015-11-11 21:23 ` Linus Torvalds
2015-11-12 7:14 ` Boqun Feng
2015-11-12 10:28 ` Peter Zijlstra
2015-11-12 15:00 ` Oleg Nesterov
2015-11-12 14:40 ` Paul E. McKenney
2015-11-12 14:49 ` Boqun Feng
2015-11-12 15:02 ` Paul E. McKenney
2015-11-12 21:53 ` Will Deacon
2015-11-12 14:50 ` Peter Zijlstra
2015-11-12 15:01 ` Paul E. McKenney
2015-11-12 15:08 ` Peter Zijlstra
2015-11-12 15:20 ` Paul E. McKenney
2015-11-12 21:25 ` Will Deacon
2015-11-12 15:18 ` Boqun Feng
2015-11-12 18:38 ` Oleg Nesterov
2015-11-12 18:02 ` Peter Zijlstra
2015-11-12 19:33 ` Oleg Nesterov
2015-11-12 18:59 ` Paul E. McKenney
2015-11-12 21:33 ` Will Deacon
2015-11-12 23:43 ` Paul E. McKenney
2015-11-16 13:58 ` Will Deacon
2015-11-12 18:21 ` Linus Torvalds
2015-11-12 22:09 ` Will Deacon
2015-11-16 15:56 ` Peter Zijlstra
2015-11-16 16:04 ` Peter Zijlstra
2015-11-16 16:24 ` Will Deacon
2015-11-16 16:44 ` Paul E. McKenney
2015-11-16 16:46 ` Will Deacon
2015-11-16 17:15 ` Paul E. McKenney
2015-11-16 21:58 ` Linus Torvalds
2015-11-17 11:51 ` Will Deacon
2015-11-17 21:01 ` Paul E. McKenney
2015-11-18 11:25 ` Will Deacon
2015-11-19 18:01 ` Will Deacon
2015-11-20 10:09 ` Peter Zijlstra
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20151102134940.944089740@infradead.org \
--to=peterz@infradead.org \
--cc=boqun.feng@gmail.com \
--cc=corbet@lwn.net \
--cc=dhowells@redhat.com \
--cc=linux-kernel@vger.kernel.org \
--cc=mhocko@kernel.org \
--cc=mingo@kernel.org \
--cc=oleg@redhat.com \
--cc=paulmck@linux.vnet.ibm.com \
--cc=torvalds@linux-foundation.org \
--cc=will.deacon@arm.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.