2.6.0-test2-mm3 osdl-aim-7 regression

2.6.0-test2-mm3 osdl-aim-7 regression

[Cliff White]

I see 2.6.0-test2-mm4 is already in our queue, so this may be
Old News. ( serves me right for taking a weekend off )
Performance of -mm3 falls off on the 4-cpu machines. 

2-cpu ssytems
Kernel 			JPM 
2.6.0-test2-mm3		1313.53
linux-2.6.0-test2	1320.68 (0.54 % +)

4-cpu systems
2.6.0-test2-mm3		4824.96
linux-2.6.0-test2	5381.20 ( 11.53 % + )

Full details at
http://developer.osdl.org/cliffw/reaim/index.html
code at 
bk://developer.osdl.org/osdl-aim-7

cliffw

Re: 2.6.0-test2-mm3 osdl-aim-7 regression

[Andrew Morton]

Cliff White <cliffw@osdl.org> wrote:
>
> 
> I see 2.6.0-test2-mm4 is already in our queue, so this may be
> Old News. ( serves me right for taking a weekend off )
> Performance of -mm3 falls off on the 4-cpu machines. 
> 
> 2-cpu ssytems
> Kernel 			JPM 
> 2.6.0-test2-mm3		1313.53
> linux-2.6.0-test2	1320.68 (0.54 % +)
> 
> 4-cpu systems
> 2.6.0-test2-mm3		4824.96
> linux-2.6.0-test2	5381.20 ( 11.53 % + )
> 
> Full details at
> http://developer.osdl.org/cliffw/reaim/index.html
> code at 
> bk://developer.osdl.org/osdl-aim-7
> 

OK, I can reproduce this on 4way.

Binary searching (insert gratuitous rant about benchmarks that take more
than two minutes to complete) reveals that the slowdown is due to
sched-2.6.0-test2-mm2-A3.

So mm4 with everthing up to but not including sched-2.6.0-test2-mm2-A3:

	Max Jobs per Minute 1467.06
	Max Jobs per Minute 1478.82
	Max Jobs per Minute 1473.36

	3853.55s user 264.31s system 370% cpu 18:31.95 total

After adding sched-2.6.0-test2-mm2-A3:

        Max Jobs per Minute 1375.63
        Max Jobs per Minute 1278.40
        Max Jobs per Minute 1293.11

        4416.70s user 275.61s system 374% cpu 20:53.58 total

A 10% regression there, mainly user time.


The test is:

- build bk://developer.osdl.org/osdl-aim-7

- cd src

- time ./reaim -s4 -q -t -i4 -f./workfile.new_dbase -r3 -b -l./reaim.config

Re: 2.6.0-test2-mm3 osdl-aim-7 regression

[Cliff White]

> Cliff White <cliffw@osdl.org> wrote:
> >
> > 
> > I see 2.6.0-test2-mm4 is already in our queue, so this may be
> > Old News. ( serves me right for taking a weekend off )
> > Performance of -mm3 falls off on the 4-cpu machines. 
> > 
> > 2-cpu ssytems
> > Kernel 			JPM 
> > 2.6.0-test2-mm3		1313.53
> > linux-2.6.0-test2	1320.68 (0.54 % +)
> > 
> > 4-cpu systems
> > 2.6.0-test2-mm3		4824.96
> > linux-2.6.0-test2	5381.20 ( 11.53 % + )
> > 
> > Full details at
> > http://developer.osdl.org/cliffw/reaim/index.html
> > code at 
> > bk://developer.osdl.org/osdl-aim-7
> > 
> 
> OK, I can reproduce this on 4way.
> 
> Binary searching (insert gratuitous rant about benchmarks that take more
> than two minutes to complete) reveals that the slowdown is due to
> sched-2.6.0-test2-mm2-A3.

[Rant response]
For a short test run, you can run a small number of iterations like this:
./reaim -s2 -e10 -i2 -f./workfile.new_dbase

( 2->10 users, increment by 2)

That takes about 5 minutes on our 4-way.

Or, run one iteration with a large user count:

./reaim -s25 -e25 -f ./workifile.foo

cliffw


> 
> So mm4 with everthing up to but not including sched-2.6.0-test2-mm2-A3:
> 
> 	Max Jobs per Minute 1467.06
> 	Max Jobs per Minute 1478.82
> 	Max Jobs per Minute 1473.36
> 
> 	3853.55s user 264.31s system 370% cpu 18:31.95 total
> 
> After adding sched-2.6.0-test2-mm2-A3:
> 
>         Max Jobs per Minute 1375.63
>         Max Jobs per Minute 1278.40
>         Max Jobs per Minute 1293.11
> 
>         4416.70s user 275.61s system 374% cpu 20:53.58 total
> 
> A 10% regression there, mainly user time.
> 
> 
> The test is:
> 
> - build bk://developer.osdl.org/osdl-aim-7
> 
> - cd src
> 
> - time ./reaim -s4 -q -t -i4 -f./workfile.new_dbase -r3 -b -l./reaim.config
> 
> 
> 

Re: 2.6.0-test2-mm3 osdl-aim-7 regression

[Con Kolivas]

On Thu, 7 Aug 2003 05:10, Cliff White wrote:
> > Binary searching (insert gratuitous rant about benchmarks that take more
> > than two minutes to complete) reveals that the slowdown is due to
> > sched-2.6.0-test2-mm2-A3.

This is most likely the round robinning of tasks every 25ms. The extra 
overhead of nanosecond timing I doubt could make that size difference (but I 
could be wrong). There is some tweaking of this round robinning in my code 
which may help this, but it won't bring it back up to original performance I 
believe. Two things to try are add my patches up to O12.3int first to see how 
much (if at all!) it helps, and change TIMESLICE_GRANULARITY in sched.c to 
(MAX_TIMESLICE) which basically disables it completely. If there is still  a 
drop in performance with this, the remainder is the extra locking/overhead in 
nanosecond timing.

Con

Re: 2.6.0-test2-mm3 osdl-aim-7 regression

[Nick Piggin]

Con Kolivas wrote:

>On Thu, 7 Aug 2003 05:10, Cliff White wrote:
>
>>>Binary searching (insert gratuitous rant about benchmarks that take more
>>>than two minutes to complete) reveals that the slowdown is due to
>>>sched-2.6.0-test2-mm2-A3.
>>>
>
>This is most likely the round robinning of tasks every 25ms. The extra 
>overhead of nanosecond timing I doubt could make that size difference (but I 
>could be wrong). There is some tweaking of this round robinning in my code 
>which may help this, but it won't bring it back up to original performance I 
>believe. Two things to try are add my patches up to O12.3int first to see how 
>much (if at all!) it helps, and change TIMESLICE_GRANULARITY in sched.c to 
>(MAX_TIMESLICE) which basically disables it completely. If there is still  a 
>drop in performance with this, the remainder is the extra locking/overhead in 
>nanosecond timing.
>
>
What is the need for this round robining? Don't processes get a calculated
timeslice anyway?

Re: 2.6.0-test2-mm3 osdl-aim-7 regression

[Con Kolivas]

On Thu, 7 Aug 2003 15:11, Nick Piggin wrote:
> What is the need for this round robining? Don't processes get a calculated
> timeslice anyway?

Nice to see you taking an unhealthy interest in the scheduler tweaks Nick. 
This issue has been discussed before but it never hurts to review things. 
I've uncc'ed the rest of the people in case we get carried away again. First 
let me show you Ingo's comment in the relevant code section:

		 * Prevent a too long timeslice allowing a task to monopolize
		 * the CPU. We do this by splitting up the timeslice into
		 * smaller pieces.
		 *
		 * Note: this does not mean the task's timeslices expire or
		 * get lost in any way, they just might be preempted by
		 * another task of equal priority. (one with higher
		 * priority would have preempted this task already.) We
		 * requeue this task to the end of the list on this priority
		 * level, which is in essence a round-robin of tasks with
		 * equal priority.

I was gonna say second blah blah but I think the first paragraph explains the 
issue. 

Must we do this? No. 

Should we? Probably. 

How frequently should we do it? Once again I'll quote Ingo who said it's a 
difficult question to answer. 

The more frequently you round robin the lower the scheduler latency between 
SCHED_OTHER tasks of the same priority. However, the longer the timeslice the 
more benefit you get from cpu cache. Where is the sweet spot? Depends on the 
hardware and your usage requirements of course, but Ingo has empirically 
chosen 25ms after 50ms seemed too long. Basically cache trashing becomes a 
real problem with timeslices below ~7ms on modern hardware in my limited 
testing. A minor quirk in Ingo's original code means _occasionally_ a task 
will be requeued with <3ms to go. It will be interesting to see if fixing 
this (which O12.2+ does) makes a big difference or whether we need to 
reconsider how frequently (if at all) we round robin tasks.  

Con

Re: 2.6.0-test2-mm3 osdl-aim-7 regression

[Nick Piggin]

Con Kolivas wrote:

>On Thu, 7 Aug 2003 15:11, Nick Piggin wrote:
>
>>What is the need for this round robining? Don't processes get a calculated
>>timeslice anyway?
>>
>
>Nice to see you taking an unhealthy interest in the scheduler tweaks Nick. 
>This issue has been discussed before but it never hurts to review things. 
>I've uncc'ed the rest of the people in case we get carried away again. First 
>let me show you Ingo's comment in the relevant code section:
>
>		 * Prevent a too long timeslice allowing a task to monopolize
>		 * the CPU. We do this by splitting up the timeslice into
>		 * smaller pieces.
>		 *
>		 * Note: this does not mean the task's timeslices expire or
>		 * get lost in any way, they just might be preempted by
>		 * another task of equal priority. (one with higher
>		 * priority would have preempted this task already.) We
>		 * requeue this task to the end of the list on this priority
>		 * level, which is in essence a round-robin of tasks with
>		 * equal priority.
>
>I was gonna say second blah blah but I think the first paragraph explains the 
>issue. 
>
>Must we do this? No. 
>
>Should we? Probably. 
>
>How frequently should we do it? Once again I'll quote Ingo who said it's a 
>difficult question to answer. 
>

OK, I was just thinking it should get done automatically by virtue
of the regular timeslice allocation, dynamic priorities, etc.

It just sounds like another workaround due to the scheduler's inability
to properly manage priorities and (the large range of length of) timeslices.

>
>
>The more frequently you round robin the lower the scheduler latency between 
>SCHED_OTHER tasks of the same priority. However, the longer the timeslice the 
>more benefit you get from cpu cache. Where is the sweet spot? Depends on the 
>hardware and your usage requirements of course, but Ingo has empirically 
>chosen 25ms after 50ms seemed too long. Basically cache trashing becomes a 
>real problem with timeslices below ~7ms on modern hardware in my limited 
>testing. A minor quirk in Ingo's original code means _occasionally_ a task 
>will be requeued with <3ms to go. It will be interesting to see if fixing 
>this (which O12.2+ does) makes a big difference or whether we need to 
>reconsider how frequently (if at all) we round robin tasks.  
>

Why not have it dynamic? CPU hogs get longer timeslices (but of course
can be preempted by higher priorities).

Re: 2.6.0-test2-mm3 osdl-aim-7 regression

[Con Kolivas]

On Thu, 7 Aug 2003 18:25, Nick Piggin wrote:
> >The more frequently you round robin the lower the scheduler latency
> > between SCHED_OTHER tasks of the same priority. However, the longer the
> > timeslice the more benefit you get from cpu cache. Where is the sweet
> > spot? Depends on the hardware and your usage requirements of course, but
> > Ingo has empirically chosen 25ms after 50ms seemed too long. Basically
> > cache trashing becomes a real problem with timeslices below ~7ms on
> > modern hardware in my limited testing. A minor quirk in Ingo's original
> > code means _occasionally_ a task will be requeued with <3ms to go. It
> > will be interesting to see if fixing this (which O12.2+ does) makes a big
> > difference or whether we need to reconsider how frequently (if at all) we
> > round robin tasks.
>
> Why not have it dynamic? CPU hogs get longer timeslices (but of course
> can be preempted by higher priorities).

Funny you should say that. Before Ingo merged his A3 changes, that's what my 
version of them did.

Con

Re: 2.6.0-test2-mm3 osdl-aim-7 regression

[Nick Piggin]

Con Kolivas wrote:

>On Thu, 7 Aug 2003 18:25, Nick Piggin wrote:
>
>>>The more frequently you round robin the lower the scheduler latency
>>>between SCHED_OTHER tasks of the same priority. However, the longer the
>>>timeslice the more benefit you get from cpu cache. Where is the sweet
>>>spot? Depends on the hardware and your usage requirements of course, but
>>>Ingo has empirically chosen 25ms after 50ms seemed too long. Basically
>>>cache trashing becomes a real problem with timeslices below ~7ms on
>>>modern hardware in my limited testing. A minor quirk in Ingo's original
>>>code means _occasionally_ a task will be requeued with <3ms to go. It
>>>will be interesting to see if fixing this (which O12.2+ does) makes a big
>>>difference or whether we need to reconsider how frequently (if at all) we
>>>round robin tasks.
>>>
>>Why not have it dynamic? CPU hogs get longer timeslices (but of course
>>can be preempted by higher priorities).
>>
>
>Funny you should say that. Before Ingo merged his A3 changes, that's what my 
>version of them did.
>
>

Between you and me, I think this would be the right way to go if it
could be done right. I don't think wli, mjb and the rest of their
clique appreciate the 25ms reschedule!

Re: 2.6.0-test2-mm3 osdl-aim-7 regression

[Cliff White]

> On Thu, 7 Aug 2003 05:10, Cliff White wrote:
> > > Binary searching (insert gratuitous rant about benchmarks that take more
> > > than two minutes to complete) reveals that the slowdown is due to
> > > sched-2.6.0-test2-mm2-A3.
> 
> This is most likely the round robinning of tasks every 25ms. The extra 
> overhead of nanosecond timing I doubt could make that size difference (but I 
> could be wrong). There is some tweaking of this round robinning in my code 
> which may help this, but it won't bring it back up to original performance I 
> believe. Two things to try are add my patches up to O12.3int first to see how 
> much (if at all!) it helps, and change TIMESLICE_GRANULARITY in sched.c to 
> (MAX_TIMESLICE) which basically disables it completely. If there is still  a 
> drop in performance with this, the remainder is the extra locking/overhead in 
> nanosecond timing.
> 
> Con
> 
Added your patches to PLM, from your web site. We've had other issues slowing 
up the
4-cpu queue, but the two CPU tests ran. On these smaller platforms, not seeing 
big
difference between the patches.

STP id PLM# Kernel Name         Workfile   MaxJPM  MaxUser Host     %Change
 277231 2042 CK-O13-O13.1int-1  new_dbase  1333.60  22     stp2-002 0.00
 277230 2041 CK-O12.3-O13int-1  new_dbase  1344.23  24     stp2-003 0.80
 277228 2040 CK-012.2-O12.3int-1 new_dbase 1328.86  22     stp2-002 -0.36
All are a bit better than stock:
276572 2020 linux-2.6.0-test2    new_dbase  1320.68 22	   stp2-000 -0.96
---- 
Code location:
bk://developer.osdl.org/osdl-aim-7
More results:
http://developer.osdl.org/cliffw/reaim/index.html

Run parameters: 

./reaim -s2 -x -t -i2 -f workfile.new_dbase -r3 -b -l./stp.config

cliffw