From: Mel Gorman <mgorman@techsingularity.net> To: LKML <linux-kernel@vger.kernel.org> Cc: Aubrey Li <aubrey.li@linux.intel.com>, Barry Song <song.bao.hua@hisilicon.com>, Ingo Molnar <mingo@redhat.com>, Peter Ziljstra <peterz@infradead.org>, Juri Lelli <juri.lelli@redhat.com>, Vincent Guittot <vincent.guittot@linaro.org>, Valentin Schneider <valentin.schneider@arm.com>, Linux-ARM <linux-arm-kernel@lists.infradead.org> Subject: [PATCH 10/10] sched/fair: Avoid revisiting CPUs multiple times during select_idle_sibling Date: Thu, 3 Dec 2020 14:20:11 +0000 [thread overview] Message-ID: <20201203142011.GW3371@techsingularity.net> (raw) In-Reply-To: <20201203141124.7391-1-mgorman@techsingularity.net> Note: While this is done in the context of select_idle_core(), I would not expect it to be done like this. The intent is to illustrate how idle_cpu_mask could be filtered before select_idle_cpus() scans the rest of a domain or a wider scan was done across a cluster. select_idle_core() potentially searches a number of CPUs for idle candidates before select_idle_cpu() clears the mask and revisits the same CPUs. This patch moves the initialisation of select_idle_mask to the top-level and reuses the same mask across both select_idle_core and select_idle_cpu. select_idle_smt() is left alone as the cost of checking one SMT sibling is marginal relative to calling __clear_cpumask_cpu() for evey CPU visited by select_idle_core(). Signed-off-by: Mel Gorman <mgorman@techsingularity.net> --- kernel/sched/fair.c | 29 ++++++++++++++++------------- 1 file changed, 16 insertions(+), 13 deletions(-) diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c index cd95daf9f53e..af2e108c20c0 100644 --- a/kernel/sched/fair.c +++ b/kernel/sched/fair.c @@ -6096,10 +6096,9 @@ void __update_idle_core(struct rq *rq) * sd_llc->shared->has_idle_cores and enabled through update_idle_core() above. */ static int select_idle_core(struct task_struct *p, struct sched_domain *sd, - int target, int nr) + int target, int nr, struct cpumask *cpus) { int idle_candidate = -1; - struct cpumask *cpus = this_cpu_cpumask_var_ptr(select_idle_mask); int core, cpu; if (!static_branch_likely(&sched_smt_present)) @@ -6108,9 +6107,6 @@ static int select_idle_core(struct task_struct *p, struct sched_domain *sd, if (!test_idle_cores(target, false)) return -1; - cpumask_and(cpus, sched_domain_span(sd), p->cpus_ptr); - __cpumask_clear_cpu(target, cpus); - for_each_cpu_wrap(core, cpus, target) { bool idle = true; @@ -6175,7 +6171,7 @@ static int select_idle_smt(struct task_struct *p, struct sched_domain *sd, int t #else /* CONFIG_SCHED_SMT */ static inline int select_idle_core(struct task_struct *p, struct sched_domain *sd, - int target, int nr) + int target, int nr, struct cpumask *cpus) { return -1; } @@ -6193,14 +6189,10 @@ static inline int select_idle_smt(struct task_struct *p, struct sched_domain *sd * average idle time for this rq (as found in rq->avg_idle). */ static int select_idle_cpu(struct task_struct *p, struct sched_domain *sd, - int target, int nr) + int target, int nr, struct cpumask *cpus) { - struct cpumask *cpus = this_cpu_cpumask_var_ptr(select_idle_mask); int cpu; - cpumask_and(cpus, sched_domain_span(sd), p->cpus_ptr); - __cpumask_clear_cpu(target, cpus); - for_each_cpu_wrap(cpu, cpus, target) { schedstat_inc(this_rq()->sis_scanned); if (!--nr) @@ -6260,6 +6252,7 @@ static inline bool asym_fits_capacity(int task_util, int cpu) static int select_idle_sibling(struct task_struct *p, int prev, int target) { struct sched_domain *sd, *this_sd; + struct cpumask *cpus_visited; unsigned long task_util; int i, recent_used_cpu, depth; u64 time; @@ -6358,13 +6351,23 @@ static int select_idle_sibling(struct task_struct *p, int prev, int target) depth = sis_search_depth(sd, this_sd); + /* + * Init the select_idle_mask. select_idle_core() will mask + * out the CPUs that have already been limited to limit the + * search in select_idle_cpu(). Further clearing is not + * done as select_idle_smt checks only one CPU. + */ + cpus_visited = this_cpu_cpumask_var_ptr(select_idle_mask); + cpumask_and(cpus_visited, sched_domain_span(sd), p->cpus_ptr); + __cpumask_clear_cpu(target, cpus_visited); + schedstat_inc(this_rq()->sis_domain_search); - i = select_idle_core(p, sd, target, depth); + i = select_idle_core(p, sd, target, depth, cpus_visited); if ((unsigned)i < nr_cpumask_bits) return i; time = cpu_clock(smp_processor_id()); - i = select_idle_cpu(p, sd, target, depth); + i = select_idle_cpu(p, sd, target, depth, cpus_visited); if ((unsigned)i < nr_cpumask_bits) goto acct_cost; -- 2.26.2
WARNING: multiple messages have this Message-ID (diff)
From: Mel Gorman <mgorman@techsingularity.net> To: LKML <linux-kernel@vger.kernel.org> Cc: Barry Song <song.bao.hua@hisilicon.com>, Juri Lelli <juri.lelli@redhat.com>, Vincent Guittot <vincent.guittot@linaro.org>, Peter Ziljstra <peterz@infradead.org>, Aubrey Li <aubrey.li@linux.intel.com>, Ingo Molnar <mingo@redhat.com>, Valentin Schneider <valentin.schneider@arm.com>, Linux-ARM <linux-arm-kernel@lists.infradead.org> Subject: [PATCH 10/10] sched/fair: Avoid revisiting CPUs multiple times during select_idle_sibling Date: Thu, 3 Dec 2020 14:20:11 +0000 [thread overview] Message-ID: <20201203142011.GW3371@techsingularity.net> (raw) In-Reply-To: <20201203141124.7391-1-mgorman@techsingularity.net> Note: While this is done in the context of select_idle_core(), I would not expect it to be done like this. The intent is to illustrate how idle_cpu_mask could be filtered before select_idle_cpus() scans the rest of a domain or a wider scan was done across a cluster. select_idle_core() potentially searches a number of CPUs for idle candidates before select_idle_cpu() clears the mask and revisits the same CPUs. This patch moves the initialisation of select_idle_mask to the top-level and reuses the same mask across both select_idle_core and select_idle_cpu. select_idle_smt() is left alone as the cost of checking one SMT sibling is marginal relative to calling __clear_cpumask_cpu() for evey CPU visited by select_idle_core(). Signed-off-by: Mel Gorman <mgorman@techsingularity.net> --- kernel/sched/fair.c | 29 ++++++++++++++++------------- 1 file changed, 16 insertions(+), 13 deletions(-) diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c index cd95daf9f53e..af2e108c20c0 100644 --- a/kernel/sched/fair.c +++ b/kernel/sched/fair.c @@ -6096,10 +6096,9 @@ void __update_idle_core(struct rq *rq) * sd_llc->shared->has_idle_cores and enabled through update_idle_core() above. */ static int select_idle_core(struct task_struct *p, struct sched_domain *sd, - int target, int nr) + int target, int nr, struct cpumask *cpus) { int idle_candidate = -1; - struct cpumask *cpus = this_cpu_cpumask_var_ptr(select_idle_mask); int core, cpu; if (!static_branch_likely(&sched_smt_present)) @@ -6108,9 +6107,6 @@ static int select_idle_core(struct task_struct *p, struct sched_domain *sd, if (!test_idle_cores(target, false)) return -1; - cpumask_and(cpus, sched_domain_span(sd), p->cpus_ptr); - __cpumask_clear_cpu(target, cpus); - for_each_cpu_wrap(core, cpus, target) { bool idle = true; @@ -6175,7 +6171,7 @@ static int select_idle_smt(struct task_struct *p, struct sched_domain *sd, int t #else /* CONFIG_SCHED_SMT */ static inline int select_idle_core(struct task_struct *p, struct sched_domain *sd, - int target, int nr) + int target, int nr, struct cpumask *cpus) { return -1; } @@ -6193,14 +6189,10 @@ static inline int select_idle_smt(struct task_struct *p, struct sched_domain *sd * average idle time for this rq (as found in rq->avg_idle). */ static int select_idle_cpu(struct task_struct *p, struct sched_domain *sd, - int target, int nr) + int target, int nr, struct cpumask *cpus) { - struct cpumask *cpus = this_cpu_cpumask_var_ptr(select_idle_mask); int cpu; - cpumask_and(cpus, sched_domain_span(sd), p->cpus_ptr); - __cpumask_clear_cpu(target, cpus); - for_each_cpu_wrap(cpu, cpus, target) { schedstat_inc(this_rq()->sis_scanned); if (!--nr) @@ -6260,6 +6252,7 @@ static inline bool asym_fits_capacity(int task_util, int cpu) static int select_idle_sibling(struct task_struct *p, int prev, int target) { struct sched_domain *sd, *this_sd; + struct cpumask *cpus_visited; unsigned long task_util; int i, recent_used_cpu, depth; u64 time; @@ -6358,13 +6351,23 @@ static int select_idle_sibling(struct task_struct *p, int prev, int target) depth = sis_search_depth(sd, this_sd); + /* + * Init the select_idle_mask. select_idle_core() will mask + * out the CPUs that have already been limited to limit the + * search in select_idle_cpu(). Further clearing is not + * done as select_idle_smt checks only one CPU. + */ + cpus_visited = this_cpu_cpumask_var_ptr(select_idle_mask); + cpumask_and(cpus_visited, sched_domain_span(sd), p->cpus_ptr); + __cpumask_clear_cpu(target, cpus_visited); + schedstat_inc(this_rq()->sis_domain_search); - i = select_idle_core(p, sd, target, depth); + i = select_idle_core(p, sd, target, depth, cpus_visited); if ((unsigned)i < nr_cpumask_bits) return i; time = cpu_clock(smp_processor_id()); - i = select_idle_cpu(p, sd, target, depth); + i = select_idle_cpu(p, sd, target, depth, cpus_visited); if ((unsigned)i < nr_cpumask_bits) goto acct_cost; -- 2.26.2 _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
next prev parent reply other threads:[~2020-12-03 14:21 UTC|newest] Thread overview: 60+ messages / expand[flat|nested] mbox.gz Atom feed top 2020-12-03 14:11 [RFC PATCH 00/10] Reduce time complexity of select_idle_sibling Mel Gorman 2020-12-03 14:11 ` Mel Gorman 2020-12-03 14:11 ` [PATCH 01/10] sched/fair: Track efficiency " Mel Gorman 2020-12-03 14:11 ` Mel Gorman 2020-12-03 14:11 ` [PATCH 02/10] sched/fair: Track efficiency of task recent_used_cpu Mel Gorman 2020-12-03 14:11 ` Mel Gorman 2020-12-03 14:11 ` [PATCH 03/10] sched/fair: Remove SIS_AVG_CPU Mel Gorman 2020-12-03 14:11 ` Mel Gorman 2020-12-03 14:11 ` [PATCH 04/10] sched/fair: Return an idle cpu if one is found after a failed search for an idle core Mel Gorman 2020-12-03 14:11 ` Mel Gorman 2020-12-03 16:35 ` Vincent Guittot 2020-12-03 16:35 ` Vincent Guittot 2020-12-03 17:50 ` Mel Gorman 2020-12-03 17:50 ` Mel Gorman 2020-12-03 14:11 ` [PATCH 05/10] sched/fair: Do not replace recent_used_cpu with the new target Mel Gorman 2020-12-03 14:11 ` Mel Gorman 2020-12-03 14:11 ` [PATCH 06/10] sched/fair: Clear the target CPU from the cpumask of CPUs searched Mel Gorman 2020-12-03 14:11 ` Mel Gorman 2020-12-03 16:38 ` Vincent Guittot 2020-12-03 16:38 ` Vincent Guittot 2020-12-03 17:52 ` Mel Gorman 2020-12-03 17:52 ` Mel Gorman 2020-12-04 10:56 ` Vincent Guittot 2020-12-04 10:56 ` Vincent Guittot 2020-12-04 11:30 ` Mel Gorman 2020-12-04 11:30 ` Mel Gorman 2020-12-04 13:13 ` Vincent Guittot 2020-12-04 13:13 ` Vincent Guittot 2020-12-04 13:17 ` Vincent Guittot 2020-12-04 13:17 ` Vincent Guittot 2020-12-04 13:40 ` Li, Aubrey 2020-12-04 13:40 ` Li, Aubrey 2020-12-04 13:47 ` Li, Aubrey 2020-12-04 13:47 ` Li, Aubrey 2020-12-04 13:47 ` Vincent Guittot 2020-12-04 13:47 ` Vincent Guittot 2020-12-04 14:07 ` Li, Aubrey 2020-12-04 14:07 ` Li, Aubrey 2020-12-04 14:31 ` Mel Gorman 2020-12-04 14:31 ` Mel Gorman 2020-12-04 15:23 ` Vincent Guittot 2020-12-04 15:23 ` Vincent Guittot 2020-12-04 15:40 ` Mel Gorman 2020-12-04 15:40 ` Mel Gorman 2020-12-04 15:43 ` Vincent Guittot 2020-12-04 15:43 ` Vincent Guittot 2020-12-04 18:41 ` Mel Gorman 2020-12-04 18:41 ` Mel Gorman 2020-12-04 14:27 ` Mel Gorman 2020-12-04 14:27 ` Mel Gorman 2020-12-03 14:11 ` [PATCH 07/10] sched/fair: Account for the idle cpu/smt search cost Mel Gorman 2020-12-03 14:11 ` Mel Gorman 2020-12-03 14:11 ` [PATCH 08/10] sched/fair: Reintroduce SIS_AVG_CPU but in the context of SIS_PROP to reduce search depth Mel Gorman 2020-12-03 14:11 ` Mel Gorman 2020-12-03 14:11 ` [PATCH 09/10] sched/fair: Limit the search for an idle core Mel Gorman 2020-12-03 14:11 ` Mel Gorman 2020-12-03 14:19 ` Mel Gorman 2020-12-03 14:19 ` Mel Gorman 2020-12-03 14:20 ` Mel Gorman [this message] 2020-12-03 14:20 ` [PATCH 10/10] sched/fair: Avoid revisiting CPUs multiple times during select_idle_sibling Mel Gorman
Reply instructions: You may reply publicly to this message via plain-text email using any one of the following methods: * Save the following mbox file, import it into your mail client, and reply-to-all from there: mbox Avoid top-posting and favor interleaved quoting: https://en.wikipedia.org/wiki/Posting_style#Interleaved_style * Reply using the --to, --cc, and --in-reply-to switches of git-send-email(1): git send-email \ --in-reply-to=20201203142011.GW3371@techsingularity.net \ --to=mgorman@techsingularity.net \ --cc=aubrey.li@linux.intel.com \ --cc=juri.lelli@redhat.com \ --cc=linux-arm-kernel@lists.infradead.org \ --cc=linux-kernel@vger.kernel.org \ --cc=mingo@redhat.com \ --cc=peterz@infradead.org \ --cc=song.bao.hua@hisilicon.com \ --cc=valentin.schneider@arm.com \ --cc=vincent.guittot@linaro.org \ /path/to/YOUR_REPLY https://kernel.org/pub/software/scm/git/docs/git-send-email.html * If your mail client supports setting the In-Reply-To header via mailto: links, try the mailto: linkBe sure your reply has a Subject: header at the top and a blank line before the message body.
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.