All of lore.kernel.org
 help / color / mirror / Atom feed
* [PATCH] sched: Optimize __calc_delta.
@ 2021-02-26 19:52 Josh Don
  2021-02-26 21:03 ` Peter Zijlstra
  2021-02-28 15:15   ` kernel test robot
  0 siblings, 2 replies; 19+ messages in thread
From: Josh Don @ 2021-02-26 19:52 UTC (permalink / raw)
  To: Ingo Molnar, Peter Zijlstra, Juri Lelli, Vincent Guittot
  Cc: Dietmar Eggemann, Steven Rostedt, Ben Segall, Mel Gorman,
	Daniel Bristot de Oliveira, linux-kernel, Clement Courbet,
	Oleg Rombakh, Josh Don

From: Clement Courbet <courbet@google.com>

A significant portion of __calc_delta time is spent in the loop
shifting a u64 by 32 bits. Use a __builtin_clz instead of iterating.

This is ~7x faster on benchmarks.

Signed-off-by: Clement Courbet <courbet@google.com>
Signed-off-by: Josh Don <joshdon@google.com>
---
 kernel/sched/fair.c | 30 ++++++++++++++++++++++--------
 1 file changed, 22 insertions(+), 8 deletions(-)

diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c
index 8a8bd7b13634..dbd1ae203f7c 100644
--- a/kernel/sched/fair.c
+++ b/kernel/sched/fair.c
@@ -214,6 +214,16 @@ static void __update_inv_weight(struct load_weight *lw)
 		lw->inv_weight = WMULT_CONST / w;
 }
 
+/*
+ * A __builtin_clz that handles a u32 value on architectures
+ * where `sizeof(unsigned int) < 32`.
+ */
+#if (__SIZEOF_INT__ < 32)
+# define BUILTIN_CLZ32(v) __builtin_clzl(v)
+#else
+# define BUILTIN_CLZ32(v) __builtin_clz(v)
+#endif
+
 /*
  * delta_exec * weight / lw.weight
  *   OR
@@ -229,27 +239,31 @@ static void __update_inv_weight(struct load_weight *lw)
 static u64 __calc_delta(u64 delta_exec, unsigned long weight, struct load_weight *lw)
 {
 	u64 fact = scale_load_down(weight);
+	u32 fact_hi = (u32)(fact >> 32);
 	int shift = WMULT_SHIFT;
+	int fs;
 
 	__update_inv_weight(lw);
 
-	if (unlikely(fact >> 32)) {
-		while (fact >> 32) {
-			fact >>= 1;
-			shift--;
-		}
+	if (unlikely(fact_hi)) {
+		fs = 32 - BUILTIN_CLZ32(fact_hi);
+		shift -= fs;
+		fact >>= fs;
 	}
 
 	fact = mul_u32_u32(fact, lw->inv_weight);
 
-	while (fact >> 32) {
-		fact >>= 1;
-		shift--;
+	fact_hi = (u32)(fact >> 32);
+	if (fact_hi) {
+		fs = 32 - BUILTIN_CLZ32(fact_hi);
+		shift -= fs;
+		fact >>= fs;
 	}
 
 	return mul_u64_u32_shr(delta_exec, fact, shift);
 }
 
+#undef BUILTIN_CLZ32
 
 const struct sched_class fair_sched_class;
 
-- 
2.30.1.766.gb4fecdf3b7-goog


^ permalink raw reply related	[flat|nested] 19+ messages in thread

end of thread, other threads:[~2021-03-10 11:27 UTC | newest]

Thread overview: 19+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2021-02-26 19:52 [PATCH] sched: Optimize __calc_delta Josh Don
2021-02-26 21:03 ` Peter Zijlstra
2021-03-02 20:55   ` Josh Don
2021-03-02 20:57     ` Josh Don
2021-03-03  9:56       ` Peter Zijlstra
2021-03-03 21:59         ` Josh Don
2021-03-03 22:46         ` [PATCH v2] " Josh Don
2021-03-04  8:31           ` Peter Zijlstra
2021-03-04 17:34           ` Nick Desaulniers
2021-03-04 18:24             ` Sedat Dilek
2021-03-04 19:21               ` Sedat Dilek
2021-03-05  1:04             ` Josh Don
2021-03-05 17:13             ` David Laight
2021-03-10 11:26           ` [tip: sched/core] sched: Optimize __calc_delta() tip-bot2 for Clement Courbet
2021-03-03 10:02       ` [PATCH] sched: Optimize __calc_delta Peter Zijlstra
2021-03-03 22:00         ` Josh Don
2021-03-03  9:49     ` Peter Zijlstra
2021-02-28 15:15 ` [sched] 4112549ee5: WARNING:at_kernel/rcu/rcutorture.c:#rcu_torture_fwd_prog_nr[rcutorture] kernel test robot
2021-02-28 15:15   ` kernel test robot

This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.