From: Erich Focht <efocht@ess.nec.de>
To: Ingo Molnar <mingo@elte.hu>, "Martin J. Bligh" <mbligh@aracnet.com>
Cc: Christoph Hellwig <hch@infradead.org>,
Robert Love <rml@tech9.net>,
Michael Hohnbaum <hohnbaum@us.ibm.com>,
Andrew Theurer <habanero@us.ibm.com>,
Linus Torvalds <torvalds@transmeta.com>,
linux-kernel <linux-kernel@vger.kernel.org>,
lse-tech <lse-tech@lists.sourceforge.net>
Subject: Re: [patch] sched-2.5.59-A2
Date: Fri, 17 Jan 2003 15:35:21 +0100 [thread overview]
Message-ID: <200301171535.21226.efocht@ess.nec.de> (raw)
In-Reply-To: <Pine.LNX.4.44.0301170921430.3723-100000@localhost.localdomain>
Hi Ingo,
I tried your patch on the small NEC TX7 I can access easilly (8
Itanium2 CPUs on 4 nodes). I actually used your patch on top of the
2.5.52-ia64 kernel, but that shouldn't matter.
I like the cleanup of the topology.h. Also the renaming to
prev_cpu_load. There was a mistake (I think) in the call to
load_balance() in the idle path, guess you wanted to have:
+ load_balance(this_rq, 1, __node_to_cpu_mask(this_node));
instead of
+ load_balance(this_rq, 1, this_cpumask);
otherwise you won't load balance at all for idle cpus.
Here are the results:
kernbench (average of 5 kernel compiles) (standard error in brackets)
---------
Elapsed UserTime SysTime
orig 134.43(1.79) 944.79(0.43) 21.41(0.28)
ingo 136.74(1.58) 951.55(0.73) 21.16(0.32)
ingofix 135.22(0.59) 952.17(0.78) 21.16(0.19)
hackbench (chat benchmark alike) (elapsed time for N groups of 20
--------- senders & receivers, stats from 10 measurements)
N=10 N=25 N=50 N=100
orig 0.77(0.03) 1.91(0.06) 3.77(0.06) 7.78(0.21)
ingo 1.70(0.35) 3.11(0.47) 4.85(0.55) 8.80(0.98)
ingofix 1.16(0.14) 2.67(0.53) 5.05(0.26) 9.99(0.13)
numabench (N memory intensive tasks running in parallel, disturbed for
--------- a short time by a "hackbench 10" call)
numa_test N=4 ElapsedTime TotalUserTime TotalSysTime
orig: 26.13(2.54) 86.10(4.47) 0.09(0.01)
ingo: 27.60(2.16) 88.06(4.58) 0.11(0.01)
ingofix: 25.51(3.05) 83.55(2.78) 0.10(0.01)
numa_test N=8 ElapsedTime TotalUserTime TotalSysTime
orig: 24.81(2.71) 164.94(4.82) 0.17(0.01)
ingo: 27.38(3.01) 170.06(5.60) 0.30(0.03)
ingofix: 29.08(2.79) 172.10(4.48) 0.32(0.03)
numa_test N=16 ElapsedTime TotalUserTime TotalSysTime
orig: 45.19(3.42) 332.07(5.89) 0.32(0.01)
ingo: 50.18(0.38) 359.46(9.31) 0.46(0.04)
ingofix: 50.30(0.42) 357.38(9.12) 0.46(0.01)
numa_test N=32 ElapsedTime TotalUserTime TotalSysTime
orig: 86.84(1.83) 671.99(9.98) 0.65(0.02)
ingo: 93.44(2.13) 704.90(16.91) 0.82(0.06)
ingofix: 93.92(1.28) 727.58(9.26) 0.77(0.03)
>From these results I would prefer to either leave the numa scheduler
as it is or to introduce an IDLE_NODEBALANCE_TICK and a
BUSY_NODEBALANCE_TICK instead of just having one NODE_REBALANCE_TICK
which balances very rarely. In that case it would make sense to keep
the cpus_to_balance() function.
Best regards,
Erich
On Friday 17 January 2003 09:47, Ingo Molnar wrote:
> Martin, Erich,
>
> could you give the attached patch a go, it implements my
> cleanup/reorganization suggestions ontop of 2.5.59:
>
> - decouple the 'slow' rebalancer from the 'fast' rebalancer and attach it
> to the scheduler tick. Remove rq->nr_balanced.
>
> - do idle-rebalancing every 1 msec, intra-node rebalancing every 200
> msecs and inter-node rebalancing every 400 msecs.
>
> - move the tick-based rebalancing logic into rebalance_tick(), it looks
> more organized this way and we have all related code in one spot.
>
> - clean up the topology.h include file structure. Since generic kernel
> code uses all the defines already, there's no reason to keep them in
> asm-generic.h. I've created a linux/topology.h file that includes
> asm/topology.h and takes care of the default and generic definitions.
> Moved over a generic topology definition from mmzone.h.
>
> - renamed rq->prev_nr_running[] to rq->prev_cpu_load[] - this further
> unifies the SMP and NUMA balancers and is more in line with the
> prev_node_load name.
>
> If performance drops due to this patch then the benchmarking goal would be
> to tune the following frequencies:
>
> #define IDLE_REBALANCE_TICK (HZ/1000 ?: 1)
> #define BUSY_REBALANCE_TICK (HZ/5 ?: 1)
> #define NODE_REBALANCE_TICK (BUSY_REBALANCE_TICK * 2)
>
> In theory NODE_REBALANCE_TICK could be defined by the NUMA platform,
> although in the past such per-platform scheduler tunings used to end
> 'orphaned' after some time. 400 msecs is pretty conservative at the
> moment, it could be made more frequent if benchmark results support it.
>
> the patch compiles and boots on UP and SMP, it compiles on x86-NUMA.
>
> Ingo
next prev parent reply other threads:[~2003-01-17 14:25 UTC|newest]
Thread overview: 96+ messages / expand[flat|nested] mbox.gz Atom feed top
2003-01-09 23:54 Minature NUMA scheduler Martin J. Bligh
2003-01-10 5:36 ` [Lse-tech] " Michael Hohnbaum
2003-01-10 16:34 ` Erich Focht
2003-01-10 16:57 ` Martin J. Bligh
2003-01-12 23:35 ` Erich Focht
2003-01-12 23:55 ` NUMA scheduler 2nd approach Erich Focht
2003-01-13 8:02 ` Christoph Hellwig
2003-01-13 11:32 ` Erich Focht
2003-01-13 15:26 ` [Lse-tech] " Christoph Hellwig
2003-01-13 15:46 ` Erich Focht
2003-01-13 19:03 ` Michael Hohnbaum
2003-01-14 1:23 ` Michael Hohnbaum
2003-01-14 4:45 ` [Lse-tech] " Andrew Theurer
2003-01-14 4:56 ` Martin J. Bligh
2003-01-14 11:14 ` Erich Focht
2003-01-14 15:55 ` [PATCH 2.5.58] new NUMA scheduler Erich Focht
2003-01-14 16:07 ` [Lse-tech] " Christoph Hellwig
2003-01-14 16:23 ` [PATCH 2.5.58] new NUMA scheduler: fix Erich Focht
2003-01-14 16:43 ` Erich Focht
2003-01-14 19:02 ` Michael Hohnbaum
2003-01-14 21:56 ` [Lse-tech] " Michael Hohnbaum
2003-01-15 15:10 ` Erich Focht
2003-01-16 0:14 ` Michael Hohnbaum
2003-01-16 6:05 ` Martin J. Bligh
2003-01-16 16:47 ` Erich Focht
2003-01-16 18:07 ` Robert Love
2003-01-16 18:48 ` Martin J. Bligh
2003-01-16 19:07 ` Ingo Molnar
2003-01-16 18:59 ` Martin J. Bligh
2003-01-16 19:10 ` Christoph Hellwig
2003-01-16 19:44 ` Ingo Molnar
2003-01-16 19:43 ` Martin J. Bligh
2003-01-16 20:19 ` Ingo Molnar
2003-01-16 20:29 ` [Lse-tech] " Rick Lindsley
2003-01-16 23:31 ` Martin J. Bligh
2003-01-17 7:23 ` Ingo Molnar
2003-01-17 8:47 ` [patch] sched-2.5.59-A2 Ingo Molnar
2003-01-17 14:35 ` Erich Focht [this message]
2003-01-17 15:11 ` Ingo Molnar
2003-01-17 15:30 ` Erich Focht
2003-01-17 16:58 ` Martin J. Bligh
2003-01-18 20:54 ` NUMA sched -> pooling scheduler (inc HT) Martin J. Bligh
2003-01-18 21:34 ` [Lse-tech] " Martin J. Bligh
2003-01-19 0:13 ` Andrew Theurer
2003-01-17 18:19 ` [patch] sched-2.5.59-A2 Michael Hohnbaum
2003-01-18 7:08 ` William Lee Irwin III
2003-01-18 8:12 ` Martin J. Bligh
2003-01-18 8:16 ` William Lee Irwin III
2003-01-19 4:22 ` William Lee Irwin III
2003-01-17 17:21 ` Martin J. Bligh
2003-01-17 17:23 ` Martin J. Bligh
2003-01-17 18:11 ` Erich Focht
2003-01-17 19:04 ` Martin J. Bligh
2003-01-17 19:26 ` [Lse-tech] " Martin J. Bligh
2003-01-18 0:13 ` Michael Hohnbaum
2003-01-18 13:31 ` [patch] tunable rebalance rates for sched-2.5.59-B0 Erich Focht
2003-01-18 23:09 ` [patch] sched-2.5.59-A2 Erich Focht
2003-01-20 9:28 ` Ingo Molnar
2003-01-20 12:07 ` Erich Focht
2003-01-20 16:56 ` Ingo Molnar
2003-01-20 17:04 ` Ingo Molnar
2003-01-20 17:10 ` Martin J. Bligh
2003-01-20 17:24 ` Ingo Molnar
2003-01-20 19:13 ` Andrew Theurer
2003-01-20 19:33 ` Martin J. Bligh
2003-01-20 19:52 ` Andrew Theurer
2003-01-20 19:52 ` Martin J. Bligh
2003-01-20 21:18 ` [patch] HT scheduler, sched-2.5.59-D7 Ingo Molnar
2003-01-20 22:28 ` Andrew Morton
2003-01-21 1:11 ` Michael Hohnbaum
2003-01-22 3:15 ` Michael Hohnbaum
2003-01-22 16:41 ` Andrew Theurer
2003-01-22 16:17 ` Martin J. Bligh
2003-01-22 16:20 ` Andrew Theurer
2003-01-22 16:35 ` Michael Hohnbaum
2003-02-03 18:23 ` [patch] HT scheduler, sched-2.5.59-E2 Ingo Molnar
2003-02-03 20:47 ` Robert Love
2003-02-04 9:31 ` Erich Focht
2003-01-20 17:04 ` [patch] sched-2.5.59-A2 Martin J. Bligh
2003-01-21 17:44 ` Erich Focht
2003-01-20 16:23 ` Martin J. Bligh
2003-01-20 16:59 ` Ingo Molnar
2003-01-17 23:09 ` Matthew Dobson
2003-01-16 23:45 ` [PATCH 2.5.58] new NUMA scheduler: fix Michael Hohnbaum
2003-01-17 11:10 ` Erich Focht
2003-01-17 14:07 ` Ingo Molnar
2003-01-16 19:44 ` John Bradford
2003-01-14 16:51 ` Christoph Hellwig
2003-01-15 0:05 ` Michael Hohnbaum
2003-01-15 7:47 ` Martin J. Bligh
2003-01-14 5:50 ` [Lse-tech] Re: NUMA scheduler 2nd approach Michael Hohnbaum
2003-01-14 16:52 ` Andrew Theurer
2003-01-14 15:13 ` Erich Focht
2003-01-14 10:56 ` Erich Focht
2003-01-11 14:43 ` [Lse-tech] Minature NUMA scheduler Bill Davidsen
2003-01-12 23:24 ` Erich Focht
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=200301171535.21226.efocht@ess.nec.de \
--to=efocht@ess.nec.de \
--cc=habanero@us.ibm.com \
--cc=hch@infradead.org \
--cc=hohnbaum@us.ibm.com \
--cc=linux-kernel@vger.kernel.org \
--cc=lse-tech@lists.sourceforge.net \
--cc=mbligh@aracnet.com \
--cc=mingo@elte.hu \
--cc=rml@tech9.net \
--cc=torvalds@transmeta.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).