From: Nick Desaulniers <ndesaulniers@google.com>
To: Josh Don <joshdon@google.com>
Cc: Ingo Molnar <mingo@redhat.com>,
Peter Zijlstra <peterz@infradead.org>,
Juri Lelli <juri.lelli@redhat.com>,
Vincent Guittot <vincent.guittot@linaro.org>,
Dietmar Eggemann <dietmar.eggemann@arm.com>,
Steven Rostedt <rostedt@goodmis.org>,
Ben Segall <bsegall@google.com>, Mel Gorman <mgorman@suse.de>,
Daniel Bristot de Oliveira <bristot@redhat.com>,
Nathan Chancellor <nathan@kernel.org>,
LKML <linux-kernel@vger.kernel.org>,
clang-built-linux <clang-built-linux@googlegroups.com>,
Clement Courbet <courbet@google.com>,
Oleg Rombakh <olegrom@google.com>,
Bill Wendling <morbo@google.com>
Subject: Re: [PATCH v2] sched: Optimize __calc_delta.
Date: Thu, 4 Mar 2021 09:34:29 -0800 [thread overview]
Message-ID: <CAKwvOdmijctJfM3gNfwEVjaQyp3LZkhnAwgsT7EBhsSBJyfLAA@mail.gmail.com> (raw)
In-Reply-To: <20210303224653.2579656-1-joshdon@google.com>
On Wed, Mar 3, 2021 at 2:48 PM Josh Don <joshdon@google.com> wrote:
>
> From: Clement Courbet <courbet@google.com>
>
> A significant portion of __calc_delta time is spent in the loop
> shifting a u64 by 32 bits. Use `fls` instead of iterating.
>
> This is ~7x faster on benchmarks.
>
> The generic `fls` implementation (`generic_fls`) is still ~4x faster
> than the loop.
> Architectures that have a better implementation will make use of it. For
> example, on X86 we get an additional factor 2 in speed without dedicated
> implementation.
>
> On gcc, the asm versions of `fls` are about the same speed as the
> builtin. On clang, the versions that use fls are more than twice as
> slow as the builtin. This is because the way the `fls` function is
> written, clang puts the value in memory:
> https://godbolt.org/z/EfMbYe. This bug is filed at
> https://bugs.llvm.org/show_bug.cgi?id=49406.
Hi Josh, Thanks for helping get this patch across the finish line.
Would you mind updating the commit message to point to
https://bugs.llvm.org/show_bug.cgi?id=20197?
>
> ```
> name cpu/op
> BM_Calc<__calc_delta_loop> 9.57ms ±12%
> BM_Calc<__calc_delta_generic_fls> 2.36ms ±13%
> BM_Calc<__calc_delta_asm_fls> 2.45ms ±13%
> BM_Calc<__calc_delta_asm_fls_nomem> 1.66ms ±12%
> BM_Calc<__calc_delta_asm_fls64> 2.46ms ±13%
> BM_Calc<__calc_delta_asm_fls64_nomem> 1.34ms ±15%
> BM_Calc<__calc_delta_builtin> 1.32ms ±11%
> ```
>
> Signed-off-by: Clement Courbet <courbet@google.com>
> Signed-off-by: Josh Don <joshdon@google.com>
> ---
> kernel/sched/fair.c | 19 +++++++++++--------
> kernel/sched/sched.h | 1 +
> 2 files changed, 12 insertions(+), 8 deletions(-)
>
> diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c
> index 8a8bd7b13634..a691371960ae 100644
> --- a/kernel/sched/fair.c
> +++ b/kernel/sched/fair.c
> @@ -229,22 +229,25 @@ static void __update_inv_weight(struct load_weight *lw)
> static u64 __calc_delta(u64 delta_exec, unsigned long weight, struct load_weight *lw)
> {
> u64 fact = scale_load_down(weight);
> + u32 fact_hi = (u32)(fact >> 32);
> int shift = WMULT_SHIFT;
> + int fs;
>
> __update_inv_weight(lw);
>
> - if (unlikely(fact >> 32)) {
> - while (fact >> 32) {
> - fact >>= 1;
> - shift--;
> - }
> + if (unlikely(fact_hi)) {
> + fs = fls(fact_hi);
> + shift -= fs;
> + fact >>= fs;
> }
>
> fact = mul_u32_u32(fact, lw->inv_weight);
>
> - while (fact >> 32) {
> - fact >>= 1;
> - shift--;
> + fact_hi = (u32)(fact >> 32);
> + if (fact_hi) {
> + fs = fls(fact_hi);
> + shift -= fs;
> + fact >>= fs;
> }
>
> return mul_u64_u32_shr(delta_exec, fact, shift);
> diff --git a/kernel/sched/sched.h b/kernel/sched/sched.h
> index 10a1522b1e30..714af71cf983 100644
> --- a/kernel/sched/sched.h
> +++ b/kernel/sched/sched.h
> @@ -36,6 +36,7 @@
> #include <uapi/linux/sched/types.h>
>
> #include <linux/binfmts.h>
> +#include <linux/bitops.h>
This hunk of the patch is curious. I assume that bitops.h is needed
for fls(); if so, why not #include it in kernel/sched/fair.c?
Otherwise this potentially hurts compile time for all TUs that include
kernel/sched/sched.h.
> #include <linux/blkdev.h>
> #include <linux/compat.h>
> #include <linux/context_tracking.h>
> --
> 2.30.1.766.gb4fecdf3b7-goog
>
--
Thanks,
~Nick Desaulniers
next prev parent reply other threads:[~2021-03-04 17:36 UTC|newest]
Thread overview: 18+ messages / expand[flat|nested] mbox.gz Atom feed top
2021-02-26 19:52 [PATCH] sched: Optimize __calc_delta Josh Don
2021-02-26 21:03 ` Peter Zijlstra
2021-03-02 20:55 ` Josh Don
2021-03-02 20:57 ` Josh Don
2021-03-03 9:56 ` Peter Zijlstra
2021-03-03 21:59 ` Josh Don
2021-03-03 22:46 ` [PATCH v2] " Josh Don
2021-03-04 8:31 ` Peter Zijlstra
2021-03-04 17:34 ` Nick Desaulniers [this message]
2021-03-04 18:24 ` Sedat Dilek
2021-03-04 19:21 ` Sedat Dilek
2021-03-05 1:04 ` Josh Don
2021-03-05 17:13 ` David Laight
2021-03-10 11:26 ` [tip: sched/core] sched: Optimize __calc_delta() tip-bot2 for Clement Courbet
2021-03-03 10:02 ` [PATCH] sched: Optimize __calc_delta Peter Zijlstra
2021-03-03 22:00 ` Josh Don
2021-03-03 9:49 ` Peter Zijlstra
2021-02-28 15:15 ` [sched] 4112549ee5: WARNING:at_kernel/rcu/rcutorture.c:#rcu_torture_fwd_prog_nr[rcutorture] kernel test robot
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=CAKwvOdmijctJfM3gNfwEVjaQyp3LZkhnAwgsT7EBhsSBJyfLAA@mail.gmail.com \
--to=ndesaulniers@google.com \
--cc=bristot@redhat.com \
--cc=bsegall@google.com \
--cc=clang-built-linux@googlegroups.com \
--cc=courbet@google.com \
--cc=dietmar.eggemann@arm.com \
--cc=joshdon@google.com \
--cc=juri.lelli@redhat.com \
--cc=linux-kernel@vger.kernel.org \
--cc=mgorman@suse.de \
--cc=mingo@redhat.com \
--cc=morbo@google.com \
--cc=nathan@kernel.org \
--cc=olegrom@google.com \
--cc=peterz@infradead.org \
--cc=rostedt@goodmis.org \
--cc=vincent.guittot@linaro.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).