From: Joel Fernandes <joel@joelfernandes.org>
To: Vincent Guittot <vincent.guittot@linaro.org>
Cc: mingo@redhat.com, peterz@infradead.org, juri.lelli@redhat.com,
dietmar.eggemann@arm.com, rostedt@goodmis.org,
bsegall@google.com, mgorman@suse.de, bristot@redhat.com,
linux-kernel@vger.kernel.org, tim.c.chen@linux.intel.com,
joelaf@google.com, dianders@google.com, qais.yousef@arm.com,
Chris.Redpath@arm.com
Subject: Re: [PATCH v3 0/5] Improve newidle lb cost tracking and early abort
Date: Thu, 28 Oct 2021 19:25:18 -0400 [thread overview]
Message-ID: <YXsxXqkLx+fKA1Ab@google.com> (raw)
In-Reply-To: <20211019123537.17146-1-vincent.guittot@linaro.org>
Hi, Vincent, Peter,
On Tue, Oct 19, 2021 at 02:35:32PM +0200, Vincent Guittot wrote:
> This patchset updates newidle lb cost tracking and early abort:
>
> The time spent running update_blocked_averages is now accounted in the 1st
> sched_domain level. This time can be significant and move the cost of
> newidle lb above the avg_idle time.
>
> The decay of max_newidle_lb_cost is modified to start only when the field
> has not been updated for a while. Recent update will not be decayed
> immediatlybut only after a while.
>
> The condition of an avg_idle lower than sysctl_sched_migration_cost has
> been removed as the 500us value is quite large and prevent opportunity to
> pull task on the newly idle CPU for at least 1st domain levels.
It appears this series is not yet in upstream Linus's tree. What's the latest on it?
I see a lot of times on ARM64 devices that load balance is skipped due to the
high the sysctl_sched_migration_cost. I saw another thread as well where
someone complained the performance varies and the default might be too high:
https://lkml.org/lkml/2021/9/14/150
Any other thoughts? Happy to help on any progress on this series as well. Thanks,
- Joel
>
> Monitoring sd->max_newidle_lb_cost on cpu0 of a Arm64 system
> THX2 (2 nodes * 28 cores * 4 cpus) during the benchmarks gives the
> following results:
> min avg max
> SMT: 1us 33us 273us - this one includes the update of blocked load
> MC: 7us 49us 398us
> NUMA: 10us 45us 158us
>
>
> Some results for hackbench -l $LOOPS -g $group :
> group tip/sched/core + this patchset
> 1 15.189(+/- 2%) 14.987(+/- 2%) +1%
> 4 4.336(+/- 3%) 4.322(+/- 5%) +0%
> 16 3.654(+/- 1%) 2.922(+/- 3%) +20%
> 32 3.209(+/- 1%) 2.919(+/- 3%) +9%
> 64 2.965(+/- 1%) 2.826(+/- 1%) +4%
> 128 2.954(+/- 1%) 2.993(+/- 8%) -1%
> 256 2.951(+/- 1%) 2.894(+/- 1%) +2%
>
> tbench and reaim have not shown any difference
>
> Change since v2:
> - Update and decay of sd->last_decay_max_lb_cost are gathered in
> update_newidle_cost(). The behavior remains almost the same except that
> the decay can happen during newidle_balance now.
>
> Tests results haven't shown any differences
>
> I haven't modified rq->max_idle_balance_cost. It acts as the max value
> for avg_idle and prevents the latter to reach high value during long
> idle phase. Moving on an IIR filter instead, could delay the convergence
> of avg_idle to a reasonnable value that reflect current situation.
>
> - Added a minor cleanup of newidle_balance
>
> Change since v1:
> - account the time spent in update_blocked_averages() in the 1st domain
>
> - reduce number of call of sched_clock_cpu()
>
> - change the way max_newidle_lb_cost is decayed. Peter suggested to use a
> IIR but keeping a track of the current max value gave the best result
>
> - removed the condition (this_rq->avg_idle < sysctl_sched_migration_cost)
> as suggested by Peter
>
> Vincent Guittot (5):
> sched/fair: Account update_blocked_averages in newidle_balance cost
> sched/fair: Skip update_blocked_averages if we are defering load
> balance
> sched/fair: Wait before decaying max_newidle_lb_cost
> sched/fair: Remove sysctl_sched_migration_cost condition
> sched/fair: cleanup newidle_balance
>
> include/linux/sched/topology.h | 2 +-
> kernel/sched/fair.c | 65 ++++++++++++++++++++++------------
> kernel/sched/topology.c | 2 +-
> 3 files changed, 45 insertions(+), 24 deletions(-)
>
> --
> 2.17.1
>
next prev parent reply other threads:[~2021-10-28 23:25 UTC|newest]
Thread overview: 34+ messages / expand[flat|nested] mbox.gz Atom feed top
2021-10-19 12:35 [PATCH v3 0/5] Improve newidle lb cost tracking and early abort Vincent Guittot
2021-10-19 12:35 ` [PATCH v3 1/5] sched/fair: Account update_blocked_averages in newidle_balance cost Vincent Guittot
2021-10-21 9:43 ` Mel Gorman
2021-10-31 10:16 ` [tip: sched/core] " tip-bot2 for Vincent Guittot
2021-10-19 12:35 ` [PATCH v3 2/5] sched/fair: Skip update_blocked_averages if we are defering load balance Vincent Guittot
2021-10-21 9:44 ` Mel Gorman
2021-10-21 12:45 ` Vincent Guittot
2021-10-31 10:16 ` [tip: sched/core] " tip-bot2 for Vincent Guittot
2021-10-19 12:35 ` [PATCH 3/5] sched/fair: Wait before decaying max_newidle_lb_cost Vincent Guittot
2021-10-21 9:45 ` Mel Gorman
2021-10-29 10:01 ` Dietmar Eggemann
2021-10-29 12:19 ` Vincent Guittot
2021-10-31 10:16 ` [tip: sched/core] " tip-bot2 for Vincent Guittot
2021-10-19 12:35 ` [PATCH v3 4/5] sched/fair: Remove sysctl_sched_migration_cost condition Vincent Guittot
2021-10-21 9:46 ` Mel Gorman
2021-10-29 10:01 ` Dietmar Eggemann
2021-10-29 12:30 ` Vincent Guittot
2021-10-31 10:16 ` [tip: sched/core] " tip-bot2 for Vincent Guittot
2021-10-19 12:35 ` [PATCH v3 5/5] sched/fair: cleanup newidle_balance Vincent Guittot
2021-10-21 9:46 ` Mel Gorman
2021-10-31 10:16 ` [tip: sched/core] sched/fair: Cleanup newidle_balance tip-bot2 for Vincent Guittot
2021-10-21 9:52 ` [PATCH v3 0/5] Improve newidle lb cost tracking and early abort Mel Gorman
2021-10-21 12:42 ` Vincent Guittot
2021-10-26 17:25 ` Tim Chen
2021-10-27 8:49 ` Vincent Guittot
2021-10-27 20:53 ` Tim Chen
2021-10-28 12:15 ` Vincent Guittot
2021-10-28 21:36 ` Tim Chen
2021-10-29 17:00 ` Tim Chen
2021-10-31 10:18 ` Vincent Guittot
2021-10-28 23:25 ` Joel Fernandes [this message]
2021-10-29 7:49 ` Vincent Guittot
2021-10-29 19:37 ` Joel Fernandes
2021-10-29 10:01 ` Dietmar Eggemann
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=YXsxXqkLx+fKA1Ab@google.com \
--to=joel@joelfernandes.org \
--cc=Chris.Redpath@arm.com \
--cc=bristot@redhat.com \
--cc=bsegall@google.com \
--cc=dianders@google.com \
--cc=dietmar.eggemann@arm.com \
--cc=joelaf@google.com \
--cc=juri.lelli@redhat.com \
--cc=linux-kernel@vger.kernel.org \
--cc=mgorman@suse.de \
--cc=mingo@redhat.com \
--cc=peterz@infradead.org \
--cc=qais.yousef@arm.com \
--cc=rostedt@goodmis.org \
--cc=tim.c.chen@linux.intel.com \
--cc=vincent.guittot@linaro.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).