Re: sched domains kernbench improvements

Re: sched domains kernbench improvements

[Con Kolivas]

Hi Nick

> So it is more a matter of tuning than anything fundamental

Geez I know how you feel... :-D


I tried it on the X440 with sched smt disabled

better than before but still slower than vanilla on half load; however better 
than vanilla on optimal and full load now! I wonder whether the worse result 
on half load is as relevant since this is 8x HT cpus?

Full details:


vanilla:
Average Half Load Run:
Elapsed Time 120.186
User Time 800.614
System Time 92.604
Percent CPU 742.4
Context Switches 10430.6
Sleeps 26554.6

Average Optimum Load Run:
Elapsed Time 81.67
User Time 1009.54
System Time 113.614
Percent CPU 1374.6
Context Switches 63728
Sleeps 41399.4

Average Maximum Load Run:
Elapsed Time 83.148
User Time 1014.54
System Time 123.806
Percent CPU 1368.4
Context Switches 45229.6
Sleeps 22077.6


-mm no SMT (sorry didnt do max load):
Average Half Load Run:
Elapsed Time 133.51
User Time 799.268
System Time 92.784
Percent CPU 669
Context Switches 19340.8
Sleeps 24427.4

Average Optimum Load Run:
Elapsed Time 81.486
User Time 1006.37
System Time 106.952
Percent CPU 1366.8
Context Switches 33939
Sleeps 32453.4


-mm less idle:
Average Half Load Run:
Elapsed Time 128.23
User Time 819.256
System Time 93.09
Percent CPU 713
Context Switches 18566.2
Sleeps 24663.8

Average Optimum Load Run:
Elapsed Time 79.844
User Time 1004.26
System Time 106.74
Percent CPU 1391.8
Context Switches 33718
Sleeps 33007.8

Average Maximum Load Run:
Elapsed Time 81.47
User Time 1008.32
System Time 119.652
Percent CPU 1384.4
Context Switches 31296.6
Sleeps 22667.6

Con

Re: sched domains kernbench improvements

[Nick Piggin]

Con Kolivas wrote:

>Hi Nick
>
>
>>So it is more a matter of tuning than anything fundamental
>>
>
>Geez I know how you feel... :-D
>
>
>I tried it on the X440 with sched smt disabled
>
>better than before but still slower than vanilla on half load; however better 
>than vanilla on optimal and full load now! I wonder whether the worse result 
>on half load is as relevant since this is 8x HT cpus?
>
>

Thanks. Yep the drop off at half load is to be expected with
CONFIG_SCHED_SMT turned off.

Re: sched domains kernbench improvements

[Con Kolivas]

On Sat, 28 Feb 2004 22:15, Nick Piggin wrote:
> Con Kolivas wrote:
> >Hi Nick
> >
> >>So it is more a matter of tuning than anything fundamental
> >
> >Geez I know how you feel... :-D
> >
> >
> >I tried it on the X440 with sched smt disabled
> >
> >better than before but still slower than vanilla on half load; however
> > better than vanilla on optimal and full load now! I wonder whether the
> > worse result on half load is as relevant since this is 8x HT cpus?
>
> Thanks. Yep the drop off at half load is to be expected with
> CONFIG_SCHED_SMT turned off.

Will this affect the SCHED_SMT performance and should I do a round of benchies 
with this enabled?

Con

Re: sched domains kernbench improvements

[Nick Piggin]

Con Kolivas wrote:

>On Sat, 28 Feb 2004 22:15, Nick Piggin wrote:
>
>>Con Kolivas wrote:
>>
>>>Hi Nick
>>>
>>>
>>>>So it is more a matter of tuning than anything fundamental
>>>>
>>>Geez I know how you feel... :-D
>>>
>>>
>>>I tried it on the X440 with sched smt disabled
>>>
>>>better than before but still slower than vanilla on half load; however
>>>better than vanilla on optimal and full load now! I wonder whether the
>>>worse result on half load is as relevant since this is 8x HT cpus?
>>>
>>Thanks. Yep the drop off at half load is to be expected with
>>CONFIG_SCHED_SMT turned off.
>>
>
>Will this affect the SCHED_SMT performance and should I do a round of benchies 
>with this enabled?
>
>

It will as far as balancing between physical CPUs, yes. It probably
doesn't make quite a big difference because it is less of a problem
if one sibling goes idle than if one CPU (in the 8-way) goes idle).

But if you could do a round with SCHED_SMT enabled it would be very
nice of you ;)

Re: sched domains kernbench improvements

[Con Kolivas]

On Sat, 28 Feb 2004 22:23, Nick Piggin wrote:
> Con Kolivas wrote:
> >Will this affect the SCHED_SMT performance and should I do a round of
> > benchies with this enabled?
>
> It will as far as balancing between physical CPUs, yes. It probably
> doesn't make quite a big difference because it is less of a problem
> if one sibling goes idle than if one CPU (in the 8-way) goes idle).
>
> But if you could do a round with SCHED_SMT enabled it would be very
> nice of you ;)

One wonders sometimes why one asks a question if one already knows the 
answer ;-) 

better on half load

sched_SMT:
Average Half Load Run:
Elapsed Time 113.008
User Time 742.786
System Time 90.65
Percent CPU 738
Context Switches 28062.6
Sleeps 24571.8

Average Optimum Load Run:
Elapsed Time 79.278
User Time 1007.69
System Time 107.388
Percent CPU 1407
Context Switches 33355
Sleeps 32720

Average Maximum Load Run:
Elapsed Time 80.33
User Time 1009.89
System Time 121.518
Percent CPU 1408.4
Context Switches 31802.4
Sleeps 22905


sched_SMT-lessidle:
Average Half Load Run:
Elapsed Time 112.084
User Time 722.972
System Time 91.946
Percent CPU 727.2
Context Switches 13016.6
Sleeps 24991.2

Average Optimum Load Run:
Elapsed Time 79.07
User Time 1007.33
System Time 107.482
Percent CPU 1410.2
Context Switches 33007.2
Sleeps 32159.6

Average Maximum Load Run:
Elapsed Time 80.926
User Time 1010.12
System Time 121.472
Percent CPU 1399
Context Switches 31268.4
Sleeps 22479

Con

sched domains kernbench improvements

[Nick Piggin]

[-- Attachment #1: Type: text/plain, Size: 818 bytes --]

Con,
I was able to reproduce your half-load kernbench problems
on the non-NUMA stp 8-way.

I made a pretty simple "lessidle" patch which tweaks some
sched domain parameters to be more inclined to move tasks,
especially when idle. That brought performance to nearly
exactly the same as 2.6.3.

Context switches are still up, but user and system time
is down a bit. So indicates it is still less balance-happy
but is obviously enough to bring the idle time down.

2.6.3:                http://khack.osdl.org/stp/288459/
2.6.3-mm4-lessidle:   http://khack.osdl.org/stp/288995/

Phew! So it is more a matter of tuning than anything
fundamental. It may be that the patch now makes balancing
too aggressive, but it is probably better to err on the
side that is closer to 2.6 behaviour.

I haven't tested this on much else.


[-- Attachment #2: sched-lessidle.patch --]
[-- Type: text/plain, Size: 1818 bytes --]

 linux-2.6-npiggin/include/linux/sched.h |   16 ++++++++--------
 linux-2.6-npiggin/kernel/sched.c        |    5 +++++
 2 files changed, 13 insertions(+), 8 deletions(-)

diff -puN include/linux/sched.h~sched-lessidle include/linux/sched.h
--- linux-2.6/include/linux/sched.h~sched-lessidle	2004-02-21 10:57:07.000000000 +1100
+++ linux-2.6-npiggin/include/linux/sched.h	2004-02-21 10:59:46.000000000 +1100
@@ -598,11 +598,11 @@ struct sched_domain {
 	.parent			= NULL,			\
 	.groups			= NULL,			\
 	.min_interval		= 1,			\
-	.max_interval		= 8,			\
-	.busy_factor		= 32,			\
+	.max_interval		= 4,			\
+	.busy_factor		= 64,			\
 	.imbalance_pct		= 125,			\
-	.cache_hot_time		= (5*1000000),		\
-	.cache_nice_tries	= 2,			\
+	.cache_hot_time		= (5*1000000/2),	\
+	.cache_nice_tries	= 1,			\
 	.flags			= SD_FLAG_FASTMIGRATE | SD_FLAG_NEWIDLE,\
 	.last_balance		= jiffies,		\
 	.balance_interval	= 1,			\
@@ -615,11 +615,11 @@ struct sched_domain {
 	.span			= CPU_MASK_NONE,	\
 	.parent			= NULL,			\
 	.groups			= NULL,			\
-	.min_interval		= 20,			\
-	.max_interval		= 1000*fls(num_online_cpus()),\
-	.busy_factor		= 4,			\
+	.min_interval		= 8,			\
+	.max_interval		= 256*fls(num_online_cpus()),\
+	.busy_factor		= 8,			\
 	.imbalance_pct		= 125,			\
-	.cache_hot_time		= (5*1000000),		\
+	.cache_hot_time		= (10*1000000),		\
 	.cache_nice_tries	= 1,			\
 	.flags			= SD_FLAG_EXEC,		\
 	.last_balance		= jiffies,		\
diff -puN kernel/sched.c~sched-lessidle kernel/sched.c
--- linux-2.6/kernel/sched.c~sched-lessidle	2004-02-21 10:57:10.000000000 +1100
+++ linux-2.6-npiggin/kernel/sched.c	2004-02-21 16:15:18.000000000 +1100
@@ -1493,6 +1493,11 @@ nextgroup:
 	return busiest;
 
 out_balanced:
+	if (busiest && idle == NEWLY_IDLE) {
+		*imbalance = 1;
+		return busiest;
+	}
+
 	*imbalance = 0;
 	return NULL;
 }

_