From: Peter Zijlstra <peterz@infradead.org> To: Mel Gorman <mgorman@techsingularity.net> Cc: Will Deacon <will@kernel.org>, Davidlohr Bueso <dave@stgolabs.net>, linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, Tejun Heo <tj@kernel.org> Subject: [PATCH] sched: Fix rq->nr_iowait ordering Date: Tue, 17 Nov 2020 10:38:29 +0100 [thread overview] Message-ID: <20201117093829.GD3121429@hirez.programming.kicks-ass.net> (raw) In-Reply-To: <20201117083016.GK3121392@hirez.programming.kicks-ass.net> And poking at this reminded me of an order email from TJ that seems to have stagnated. --- Subject: sched: Fix rq->nr_iowait ordering From: Peter Zijlstra <peterz@infradead.org> Date: Thu, 24 Sep 2020 13:50:42 +0200 schedule() ttwu() deactivate_task(); if (p->on_rq && ...) // false atomic_dec(&task_rq(p)->nr_iowait); if (prev->in_iowait) atomic_inc(&rq->nr_iowait); Allows nr_iowait to be decremented before it gets incremented, resulting in more dodgy IO-wait numbers than usual. Note that because we can now do ttwu_queue_wakelist() before p->on_cpu==0, we lose the natural ordering and have to further delay the decrement. Fixes: Fixes: c6e7bd7afaeb ("sched/core: Optimize ttwu() spinning on p->on_cpu") Reported-by: Tejun Heo <tj@kernel.org> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> --- kernel/sched/core.c | 15 ++++++++++----- 1 file changed, 10 insertions(+), 5 deletions(-) --- a/kernel/sched/core.c +++ b/kernel/sched/core.c @@ -2949,7 +2949,12 @@ ttwu_do_activate(struct rq *rq, struct t #ifdef CONFIG_SMP if (wake_flags & WF_MIGRATED) en_flags |= ENQUEUE_MIGRATED; + else #endif + if (p->in_iowait) { + delayacct_blkio_end(p); + atomic_dec(&task_rq(p)->nr_iowait); + } activate_task(rq, p, en_flags); ttwu_do_wakeup(rq, p, wake_flags, rf); @@ -3336,11 +3341,6 @@ try_to_wake_up(struct task_struct *p, un if (READ_ONCE(p->on_rq) && ttwu_runnable(p, wake_flags)) goto unlock; - if (p->in_iowait) { - delayacct_blkio_end(p); - atomic_dec(&task_rq(p)->nr_iowait); - } - #ifdef CONFIG_SMP /* * Ensure we load p->on_cpu _after_ p->on_rq, otherwise it would be @@ -3411,6 +3411,11 @@ try_to_wake_up(struct task_struct *p, un cpu = select_task_rq(p, p->wake_cpu, wake_flags | WF_TTWU); if (task_cpu(p) != cpu) { + if (p->in_iowait) { + delayacct_blkio_end(p); + atomic_dec(&task_rq(p)->nr_iowait); + } + wake_flags |= WF_MIGRATED; psi_ttwu_dequeue(p); set_task_cpu(p, cpu);
WARNING: multiple messages have this Message-ID (diff)
From: Peter Zijlstra <peterz@infradead.org> To: Mel Gorman <mgorman@techsingularity.net> Cc: Tejun Heo <tj@kernel.org>, Davidlohr Bueso <dave@stgolabs.net>, Will Deacon <will@kernel.org>, linux-kernel@vger.kernel.org, linux-arm-kernel@lists.infradead.org Subject: [PATCH] sched: Fix rq->nr_iowait ordering Date: Tue, 17 Nov 2020 10:38:29 +0100 [thread overview] Message-ID: <20201117093829.GD3121429@hirez.programming.kicks-ass.net> (raw) In-Reply-To: <20201117083016.GK3121392@hirez.programming.kicks-ass.net> And poking at this reminded me of an order email from TJ that seems to have stagnated. --- Subject: sched: Fix rq->nr_iowait ordering From: Peter Zijlstra <peterz@infradead.org> Date: Thu, 24 Sep 2020 13:50:42 +0200 schedule() ttwu() deactivate_task(); if (p->on_rq && ...) // false atomic_dec(&task_rq(p)->nr_iowait); if (prev->in_iowait) atomic_inc(&rq->nr_iowait); Allows nr_iowait to be decremented before it gets incremented, resulting in more dodgy IO-wait numbers than usual. Note that because we can now do ttwu_queue_wakelist() before p->on_cpu==0, we lose the natural ordering and have to further delay the decrement. Fixes: Fixes: c6e7bd7afaeb ("sched/core: Optimize ttwu() spinning on p->on_cpu") Reported-by: Tejun Heo <tj@kernel.org> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> --- kernel/sched/core.c | 15 ++++++++++----- 1 file changed, 10 insertions(+), 5 deletions(-) --- a/kernel/sched/core.c +++ b/kernel/sched/core.c @@ -2949,7 +2949,12 @@ ttwu_do_activate(struct rq *rq, struct t #ifdef CONFIG_SMP if (wake_flags & WF_MIGRATED) en_flags |= ENQUEUE_MIGRATED; + else #endif + if (p->in_iowait) { + delayacct_blkio_end(p); + atomic_dec(&task_rq(p)->nr_iowait); + } activate_task(rq, p, en_flags); ttwu_do_wakeup(rq, p, wake_flags, rf); @@ -3336,11 +3341,6 @@ try_to_wake_up(struct task_struct *p, un if (READ_ONCE(p->on_rq) && ttwu_runnable(p, wake_flags)) goto unlock; - if (p->in_iowait) { - delayacct_blkio_end(p); - atomic_dec(&task_rq(p)->nr_iowait); - } - #ifdef CONFIG_SMP /* * Ensure we load p->on_cpu _after_ p->on_rq, otherwise it would be @@ -3411,6 +3411,11 @@ try_to_wake_up(struct task_struct *p, un cpu = select_task_rq(p, p->wake_cpu, wake_flags | WF_TTWU); if (task_cpu(p) != cpu) { + if (p->in_iowait) { + delayacct_blkio_end(p); + atomic_dec(&task_rq(p)->nr_iowait); + } + wake_flags |= WF_MIGRATED; psi_ttwu_dequeue(p); set_task_cpu(p, cpu); _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
next prev parent reply other threads:[~2020-11-17 9:38 UTC|newest] Thread overview: 70+ messages / expand[flat|nested] mbox.gz Atom feed top 2020-11-16 9:10 Loadavg accounting error on arm64 Mel Gorman 2020-11-16 9:10 ` Mel Gorman 2020-11-16 11:49 ` Mel Gorman 2020-11-16 11:49 ` Mel Gorman 2020-11-16 12:00 ` Mel Gorman 2020-11-16 12:00 ` Mel Gorman 2020-11-16 12:53 ` Peter Zijlstra 2020-11-16 12:53 ` Peter Zijlstra 2020-11-16 12:58 ` Peter Zijlstra 2020-11-16 12:58 ` Peter Zijlstra 2020-11-16 15:29 ` Mel Gorman 2020-11-16 15:29 ` Mel Gorman 2020-11-16 16:42 ` Mel Gorman 2020-11-16 16:42 ` Mel Gorman 2020-11-16 16:49 ` Peter Zijlstra 2020-11-16 16:49 ` Peter Zijlstra 2020-11-16 17:24 ` Mel Gorman 2020-11-16 17:24 ` Mel Gorman 2020-11-16 17:41 ` Will Deacon 2020-11-16 17:41 ` Will Deacon 2020-11-16 12:46 ` Peter Zijlstra 2020-11-16 12:46 ` Peter Zijlstra 2020-11-16 12:58 ` Mel Gorman 2020-11-16 12:58 ` Mel Gorman 2020-11-16 13:11 ` Will Deacon 2020-11-16 13:11 ` Will Deacon 2020-11-16 13:37 ` Mel Gorman 2020-11-16 13:37 ` Mel Gorman 2020-11-16 14:20 ` Peter Zijlstra 2020-11-16 14:20 ` Peter Zijlstra 2020-11-16 15:52 ` Mel Gorman 2020-11-16 15:52 ` Mel Gorman 2020-11-16 16:54 ` Peter Zijlstra 2020-11-16 16:54 ` Peter Zijlstra 2020-11-16 17:16 ` Mel Gorman 2020-11-16 17:16 ` Mel Gorman 2020-11-16 19:31 ` Mel Gorman 2020-11-16 19:31 ` Mel Gorman 2020-11-17 8:30 ` [PATCH] sched: Fix data-race in wakeup Peter Zijlstra 2020-11-17 8:30 ` Peter Zijlstra 2020-11-17 9:15 ` Will Deacon 2020-11-17 9:15 ` Will Deacon 2020-11-17 9:29 ` Peter Zijlstra 2020-11-17 9:29 ` Peter Zijlstra 2020-11-17 9:46 ` Peter Zijlstra 2020-11-17 9:46 ` Peter Zijlstra 2020-11-17 10:36 ` Will Deacon 2020-11-17 10:36 ` Will Deacon 2020-11-17 12:52 ` Valentin Schneider 2020-11-17 12:52 ` Valentin Schneider 2020-11-17 15:37 ` Valentin Schneider 2020-11-17 15:37 ` Valentin Schneider 2020-11-17 16:13 ` Peter Zijlstra 2020-11-17 16:13 ` Peter Zijlstra 2020-11-17 19:32 ` Valentin Schneider 2020-11-17 19:32 ` Valentin Schneider 2020-11-18 8:05 ` Peter Zijlstra 2020-11-18 8:05 ` Peter Zijlstra 2020-11-18 9:51 ` Valentin Schneider 2020-11-18 9:51 ` Valentin Schneider 2020-11-18 13:33 ` Marco Elver 2020-11-18 13:33 ` Marco Elver 2020-11-17 9:38 ` Peter Zijlstra [this message] 2020-11-17 9:38 ` [PATCH] sched: Fix rq->nr_iowait ordering Peter Zijlstra 2020-11-17 11:43 ` Mel Gorman 2020-11-17 11:43 ` Mel Gorman 2020-11-19 9:55 ` [tip: sched/urgent] " tip-bot2 for Peter Zijlstra 2020-11-17 12:40 ` [PATCH] sched: Fix data-race in wakeup Mel Gorman 2020-11-17 12:40 ` Mel Gorman 2020-11-19 9:55 ` [tip: sched/urgent] " tip-bot2 for Peter Zijlstra
Reply instructions: You may reply publicly to this message via plain-text email using any one of the following methods: * Save the following mbox file, import it into your mail client, and reply-to-all from there: mbox Avoid top-posting and favor interleaved quoting: https://en.wikipedia.org/wiki/Posting_style#Interleaved_style * Reply using the --to, --cc, and --in-reply-to switches of git-send-email(1): git send-email \ --in-reply-to=20201117093829.GD3121429@hirez.programming.kicks-ass.net \ --to=peterz@infradead.org \ --cc=dave@stgolabs.net \ --cc=linux-arm-kernel@lists.infradead.org \ --cc=linux-kernel@vger.kernel.org \ --cc=mgorman@techsingularity.net \ --cc=tj@kernel.org \ --cc=will@kernel.org \ /path/to/YOUR_REPLY https://kernel.org/pub/software/scm/git/docs/git-send-email.html * If your mail client supports setting the In-Reply-To header via mailto: links, try the mailto: linkBe sure your reply has a Subject: header at the top and a blank line before the message body.
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.