From: Rik van Riel <riel@surriel.com>
To: peterz@infradead.org
Cc: mingo@redhat.com, linux-kernel@vger.kernel.org,
kernel-team@fb.com, morten.rasmussen@arm.com, tglx@linutronix.de,
dietmar.eggeman@arm.com, mgorman@techsingularity.com,
vincent.guittot@linaro.org, Rik van Riel <riel@surriel.com>
Subject: [PATCH 4/8] sched,fair: remove cfs rqs from leaf_cfs_rq_list bottom up
Date: Wed, 12 Jun 2019 15:32:23 -0400 [thread overview]
Message-ID: <20190612193227.993-5-riel@surriel.com> (raw)
In-Reply-To: <20190612193227.993-1-riel@surriel.com>
Reducing the overhead of the CPU controller is achieved by not walking
all the sched_entities every time a task is enqueued or dequeued.
One of the things being checked every single time is whether the cfs_rq
is on the rq->leaf_cfs_rq_list.
By only removing a cfs_rq from the list once it no longer has children
on the list, we can avoid walking the sched_entity hierarchy if the bottom
cfs_rq is on the list, once the runqueues have been flattened.
Signed-off-by: Rik van Riel <riel@surriel.com>
---
kernel/sched/fair.c | 17 +++++++++++++++++
kernel/sched/sched.h | 1 +
2 files changed, 18 insertions(+)
diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c
index aebd43d74468..dcc521d251e3 100644
--- a/kernel/sched/fair.c
+++ b/kernel/sched/fair.c
@@ -285,6 +285,13 @@ static inline bool list_add_leaf_cfs_rq(struct cfs_rq *cfs_rq)
cfs_rq->on_list = 1;
+ /*
+ * If the tmp_alone_branch cursor was moved, it means a child cfs_rq
+ * is already on the list ahead of us.
+ */
+ if (rq->tmp_alone_branch != &rq->leaf_cfs_rq_list)
+ cfs_rq->children_on_list++;
+
/*
* Ensure we either appear before our parent (if already
* enqueued) or force our parent to appear after us when it is
@@ -310,6 +317,7 @@ static inline bool list_add_leaf_cfs_rq(struct cfs_rq *cfs_rq)
* list.
*/
rq->tmp_alone_branch = &rq->leaf_cfs_rq_list;
+ cfs_rq->tg->parent->cfs_rq[cpu]->children_on_list++;
return true;
}
@@ -358,6 +366,11 @@ static inline void list_del_leaf_cfs_rq(struct cfs_rq *cfs_rq)
if (rq->tmp_alone_branch == &cfs_rq->leaf_cfs_rq_list)
rq->tmp_alone_branch = cfs_rq->leaf_cfs_rq_list.prev;
+ if (cfs_rq->tg->parent) {
+ int cpu = cpu_of(rq);
+ cfs_rq->tg->parent->cfs_rq[cpu]->children_on_list--;
+ }
+
list_del_rcu(&cfs_rq->leaf_cfs_rq_list);
cfs_rq->on_list = 0;
}
@@ -7688,6 +7701,10 @@ static inline bool cfs_rq_is_decayed(struct cfs_rq *cfs_rq)
if (cfs_rq->avg.util_sum)
return false;
+ /* Remove decayed parents once their decayed children are gone. */
+ if (cfs_rq->children_on_list)
+ return false;
+
return true;
}
diff --git a/kernel/sched/sched.h b/kernel/sched/sched.h
index 5be14cee61f9..18494b1a9bac 100644
--- a/kernel/sched/sched.h
+++ b/kernel/sched/sched.h
@@ -556,6 +556,7 @@ struct cfs_rq {
* This list is used during load balance.
*/
int on_list;
+ int children_on_list;
struct list_head leaf_cfs_rq_list;
struct task_group *tg; /* group that "owns" this runqueue */
--
2.20.1
next prev parent reply other threads:[~2019-06-12 19:33 UTC|newest]
Thread overview: 24+ messages / expand[flat|nested] mbox.gz Atom feed top
2019-06-12 19:32 [RFC] sched,cfs: flatten CPU controller runqueues Rik van Riel
2019-06-12 19:32 ` [PATCH 1/8] sched: introduce task_se_h_load helper Rik van Riel
2019-06-19 12:52 ` Dietmar Eggemann
2019-06-19 13:57 ` Rik van Riel
2019-06-19 15:18 ` Dietmar Eggemann
2019-06-19 15:55 ` Rik van Riel
2019-06-12 19:32 ` [PATCH 2/8] sched: change /proc/sched_debug fields Rik van Riel
2019-06-12 19:32 ` [PATCH 3/8] sched,fair: redefine runnable_load_avg as the sum of task_h_load Rik van Riel
2019-06-18 9:08 ` Dietmar Eggemann
2019-06-26 14:34 ` Dietmar Eggemann
2019-06-12 19:32 ` Rik van Riel [this message]
2019-06-12 19:32 ` [PATCH 5/8] sched,cfs: use explicit cfs_rq of parent se helper Rik van Riel
2019-06-20 16:23 ` Dietmar Eggemann
2019-06-20 16:29 ` Rik van Riel
2019-06-24 11:24 ` Dietmar Eggemann
2019-06-26 15:58 ` Dietmar Eggemann
2019-06-26 16:15 ` Rik van Riel
2019-06-12 19:32 ` [PATCH 6/8] sched,cfs: fix zero length timeslice calculation Rik van Riel
2019-06-12 19:32 ` [PATCH 7/8] sched,fair: refactor enqueue/dequeue_entity Rik van Riel
2019-06-12 19:32 ` [PATCH 8/8] sched,fair: flatten hierarchical runqueues Rik van Riel
2019-06-25 9:50 ` Dietmar Eggemann
2019-06-25 13:51 ` Rik van Riel
2019-06-28 10:26 ` Dietmar Eggemann
2019-06-28 19:36 ` Rik van Riel
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20190612193227.993-5-riel@surriel.com \
--to=riel@surriel.com \
--cc=dietmar.eggeman@arm.com \
--cc=kernel-team@fb.com \
--cc=linux-kernel@vger.kernel.org \
--cc=mgorman@techsingularity.com \
--cc=mingo@redhat.com \
--cc=morten.rasmussen@arm.com \
--cc=peterz@infradead.org \
--cc=tglx@linutronix.de \
--cc=vincent.guittot@linaro.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).