From: "tip-bot2 for Peter Zijlstra" <tip-bot2@linutronix.de>
To: linux-tip-commits@vger.kernel.org
Cc: Tejun Heo <tj@kernel.org>,
"Peter Zijlstra (Intel)" <peterz@infradead.org>,
Mel Gorman <mgorman@techsingularity.net>,
x86@kernel.org, linux-kernel@vger.kernel.org
Subject: [tip: sched/urgent] sched: Fix rq->nr_iowait ordering
Date: Thu, 19 Nov 2020 09:55:04 -0000 [thread overview]
Message-ID: <160577970455.11244.7243891330140821350.tip-bot2@tip-bot2> (raw)
In-Reply-To: <20201117093829.GD3121429@hirez.programming.kicks-ass.net>
The following commit has been merged into the sched/urgent branch of tip:
Commit-ID: ec618b84f6e15281cc3660664d34cd0dd2f2579e
Gitweb: https://git.kernel.org/tip/ec618b84f6e15281cc3660664d34cd0dd2f2579e
Author: Peter Zijlstra <peterz@infradead.org>
AuthorDate: Thu, 24 Sep 2020 13:50:42 +02:00
Committer: Peter Zijlstra <peterz@infradead.org>
CommitterDate: Tue, 17 Nov 2020 13:15:28 +01:00
sched: Fix rq->nr_iowait ordering
schedule() ttwu()
deactivate_task(); if (p->on_rq && ...) // false
atomic_dec(&task_rq(p)->nr_iowait);
if (prev->in_iowait)
atomic_inc(&rq->nr_iowait);
Allows nr_iowait to be decremented before it gets incremented,
resulting in more dodgy IO-wait numbers than usual.
Note that because we can now do ttwu_queue_wakelist() before
p->on_cpu==0, we lose the natural ordering and have to further delay
the decrement.
Fixes: c6e7bd7afaeb ("sched/core: Optimize ttwu() spinning on p->on_cpu")
Reported-by: Tejun Heo <tj@kernel.org>
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Acked-by: Mel Gorman <mgorman@techsingularity.net>
Link: https://lkml.kernel.org/r/20201117093829.GD3121429@hirez.programming.kicks-ass.net
---
kernel/sched/core.c | 15 ++++++++++-----
1 file changed, 10 insertions(+), 5 deletions(-)
diff --git a/kernel/sched/core.c b/kernel/sched/core.c
index d2003a7..9f0ebfb 100644
--- a/kernel/sched/core.c
+++ b/kernel/sched/core.c
@@ -2501,7 +2501,12 @@ ttwu_do_activate(struct rq *rq, struct task_struct *p, int wake_flags,
#ifdef CONFIG_SMP
if (wake_flags & WF_MIGRATED)
en_flags |= ENQUEUE_MIGRATED;
+ else
#endif
+ if (p->in_iowait) {
+ delayacct_blkio_end(p);
+ atomic_dec(&task_rq(p)->nr_iowait);
+ }
activate_task(rq, p, en_flags);
ttwu_do_wakeup(rq, p, wake_flags, rf);
@@ -2888,11 +2893,6 @@ try_to_wake_up(struct task_struct *p, unsigned int state, int wake_flags)
if (READ_ONCE(p->on_rq) && ttwu_runnable(p, wake_flags))
goto unlock;
- if (p->in_iowait) {
- delayacct_blkio_end(p);
- atomic_dec(&task_rq(p)->nr_iowait);
- }
-
#ifdef CONFIG_SMP
/*
* Ensure we load p->on_cpu _after_ p->on_rq, otherwise it would be
@@ -2963,6 +2963,11 @@ try_to_wake_up(struct task_struct *p, unsigned int state, int wake_flags)
cpu = select_task_rq(p, p->wake_cpu, SD_BALANCE_WAKE, wake_flags);
if (task_cpu(p) != cpu) {
+ if (p->in_iowait) {
+ delayacct_blkio_end(p);
+ atomic_dec(&task_rq(p)->nr_iowait);
+ }
+
wake_flags |= WF_MIGRATED;
psi_ttwu_dequeue(p);
set_task_cpu(p, cpu);
next prev parent reply other threads:[~2020-11-19 9:55 UTC|newest]
Thread overview: 70+ messages / expand[flat|nested] mbox.gz Atom feed top
2020-11-16 9:10 Loadavg accounting error on arm64 Mel Gorman
2020-11-16 9:10 ` Mel Gorman
2020-11-16 11:49 ` Mel Gorman
2020-11-16 11:49 ` Mel Gorman
2020-11-16 12:00 ` Mel Gorman
2020-11-16 12:00 ` Mel Gorman
2020-11-16 12:53 ` Peter Zijlstra
2020-11-16 12:53 ` Peter Zijlstra
2020-11-16 12:58 ` Peter Zijlstra
2020-11-16 12:58 ` Peter Zijlstra
2020-11-16 15:29 ` Mel Gorman
2020-11-16 15:29 ` Mel Gorman
2020-11-16 16:42 ` Mel Gorman
2020-11-16 16:42 ` Mel Gorman
2020-11-16 16:49 ` Peter Zijlstra
2020-11-16 16:49 ` Peter Zijlstra
2020-11-16 17:24 ` Mel Gorman
2020-11-16 17:24 ` Mel Gorman
2020-11-16 17:41 ` Will Deacon
2020-11-16 17:41 ` Will Deacon
2020-11-16 12:46 ` Peter Zijlstra
2020-11-16 12:46 ` Peter Zijlstra
2020-11-16 12:58 ` Mel Gorman
2020-11-16 12:58 ` Mel Gorman
2020-11-16 13:11 ` Will Deacon
2020-11-16 13:11 ` Will Deacon
2020-11-16 13:37 ` Mel Gorman
2020-11-16 13:37 ` Mel Gorman
2020-11-16 14:20 ` Peter Zijlstra
2020-11-16 14:20 ` Peter Zijlstra
2020-11-16 15:52 ` Mel Gorman
2020-11-16 15:52 ` Mel Gorman
2020-11-16 16:54 ` Peter Zijlstra
2020-11-16 16:54 ` Peter Zijlstra
2020-11-16 17:16 ` Mel Gorman
2020-11-16 17:16 ` Mel Gorman
2020-11-16 19:31 ` Mel Gorman
2020-11-16 19:31 ` Mel Gorman
2020-11-17 8:30 ` [PATCH] sched: Fix data-race in wakeup Peter Zijlstra
2020-11-17 8:30 ` Peter Zijlstra
2020-11-17 9:15 ` Will Deacon
2020-11-17 9:15 ` Will Deacon
2020-11-17 9:29 ` Peter Zijlstra
2020-11-17 9:29 ` Peter Zijlstra
2020-11-17 9:46 ` Peter Zijlstra
2020-11-17 9:46 ` Peter Zijlstra
2020-11-17 10:36 ` Will Deacon
2020-11-17 10:36 ` Will Deacon
2020-11-17 12:52 ` Valentin Schneider
2020-11-17 12:52 ` Valentin Schneider
2020-11-17 15:37 ` Valentin Schneider
2020-11-17 15:37 ` Valentin Schneider
2020-11-17 16:13 ` Peter Zijlstra
2020-11-17 16:13 ` Peter Zijlstra
2020-11-17 19:32 ` Valentin Schneider
2020-11-17 19:32 ` Valentin Schneider
2020-11-18 8:05 ` Peter Zijlstra
2020-11-18 8:05 ` Peter Zijlstra
2020-11-18 9:51 ` Valentin Schneider
2020-11-18 9:51 ` Valentin Schneider
2020-11-18 13:33 ` Marco Elver
2020-11-18 13:33 ` Marco Elver
2020-11-17 9:38 ` [PATCH] sched: Fix rq->nr_iowait ordering Peter Zijlstra
2020-11-17 9:38 ` Peter Zijlstra
2020-11-17 11:43 ` Mel Gorman
2020-11-17 11:43 ` Mel Gorman
2020-11-19 9:55 ` tip-bot2 for Peter Zijlstra [this message]
2020-11-17 12:40 ` [PATCH] sched: Fix data-race in wakeup Mel Gorman
2020-11-17 12:40 ` Mel Gorman
2020-11-19 9:55 ` [tip: sched/urgent] " tip-bot2 for Peter Zijlstra
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=160577970455.11244.7243891330140821350.tip-bot2@tip-bot2 \
--to=tip-bot2@linutronix.de \
--cc=linux-kernel@vger.kernel.org \
--cc=linux-tip-commits@vger.kernel.org \
--cc=mgorman@techsingularity.net \
--cc=peterz@infradead.org \
--cc=tj@kernel.org \
--cc=x86@kernel.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.