Nick's scheduler v16

Nick's scheduler v16

[Nick Piggin]

Hi

http://www.kerneltrap.org/~npiggin/v16/

I'm starting to do some large SMP / NUMA testing. Fixed and changed quite
a bit. It isn't too bad, although I'm only testing dbench, tbench, and
volanomark at the moment.

These SMP and NUMA changes are not tied to my interactivity stuff, so its
possible they could get included if they turn out well. If you find any
problems with it (high end or interactivity), please let me know.

Re: Nick's scheduler v16

[Nick Piggin]

Nick Piggin wrote:

> Hi
>
> http://www.kerneltrap.org/~npiggin/v16/
>
> I'm starting to do some large SMP / NUMA testing. Fixed and changed quite
> a bit. It isn't too bad, although I'm only testing dbench, tbench, and
> volanomark at the moment.
>

Oh, if anyone can suggest other benchmarks I could use it would be good
(I'm currently getting osdl's dbt2 set up). System is 16-way 16GB ia32
NUMA and very minimum disk bandwidth. Thanks.

Re: Nick's scheduler v16

[Martin J. Bligh]

> I'm starting to do some large SMP / NUMA testing. Fixed and changed quite
> a bit. It isn't too bad, although I'm only testing dbench, tbench, and
> volanomark at the moment.
> 
> These SMP and NUMA changes are not tied to my interactivity stuff, so its
> possible they could get included if they turn out well. If you find any
> problems with it (high end or interactivity), please let me know.

Interesting ... some things get getter, some worse:

Kernbench: (make -j N vmlinux, where N = 2 x num_cpus)
                              Elapsed      System        User         CPU
              2.6.0-test8       45.20      100.97      566.65     1476.25
         2.6.0-test8-nick       44.81       93.98      568.49     1477.50
        2.6.0-test8-nick2       44.78       94.69      568.81     1482.00

elapsed is a tiny bit faster, system is significantly less, but with
higher parallelism:


Kernbench: (make -j vmlinux, maximal tasks)
                              Elapsed      System        User         CPU
              2.6.0-test8       45.86      119.41      569.66     1502.00
         2.6.0-test8-nick       47.00      112.75      590.40     1495.00
        2.6.0-test8-nick2       47.11      112.86      590.31     1491.50

elapsed is definitely worse now.
SDET is a happy bunny though:

SDET 128  (see disclaimer)
                           Throughput    Std. Dev
              2.6.0-test8       100.0%         0.3%
         2.6.0-test8-nick       109.9%         0.2%

Much of the changes there might just be backing out Con's interactivity
changes ...

M.

Re: Nick's scheduler v16

[Nick Piggin]

Martin J. Bligh wrote:

>>I'm starting to do some large SMP / NUMA testing. Fixed and changed quite
>>a bit. It isn't too bad, although I'm only testing dbench, tbench, and
>>volanomark at the moment.
>>
>>These SMP and NUMA changes are not tied to my interactivity stuff, so its
>>possible they could get included if they turn out well. If you find any
>>problems with it (high end or interactivity), please let me know.
>>
>
>Interesting ... some things get getter, some worse:
>
>Kernbench: (make -j N vmlinux, where N = 2 x num_cpus)
>                              Elapsed      System        User         CPU
>              2.6.0-test8       45.20      100.97      566.65     1476.25
>         2.6.0-test8-nick       44.81       93.98      568.49     1477.50
>        2.6.0-test8-nick2       44.78       94.69      568.81     1482.00
>
>elapsed is a tiny bit faster, system is significantly less, but with
>higher parallelism:
>
>
>Kernbench: (make -j vmlinux, maximal tasks)
>                              Elapsed      System        User         CPU
>              2.6.0-test8       45.86      119.41      569.66     1502.00
>         2.6.0-test8-nick       47.00      112.75      590.40     1495.00
>        2.6.0-test8-nick2       47.11      112.86      590.31     1491.50
>
>elapsed is definitely worse now.
>

I'll see if I can reproduce this and work out what is going on.
Thanks Martin.

Re: Nick's scheduler v16 - reaim

[Cliff White]

> 
> 
> 
> >>I'm starting to do some large SMP / NUMA testing. Fixed and changed quite
> >>a bit. It isn't too bad, although I'm only testing dbench, tbench, and
> >>volanomark at the moment.
> >>
> >>These SMP and NUMA changes are not tied to my interactivity stuff, so its
> >>possible they could get included if they turn out well. If you find any
> >>problems with it (high end or interactivity), please let me know.
> >>

Results from reaim aren't encouraging.
patch was applied against 2.6.0-test8 - the result
is PLM #2232

STP id  kernal name       Max JPM   Max User Pct    elevator
281932	nick_v16          4923.08    60	     0.00   AS
281933	nick_v16          5196.06    68	     5.54   deadline
281722	linux-2.6.0-test8 5432.77    92	     9.38   deadline
281792	2.6.0-test8-mm1   5384.41    92	     8.56   deadline
281790	2.6.0-test8-mm1   5392.65    88      8.7    AS

The kernel doesn't perform well at larger user numbers.
Notice the different in max users.
Compare the graphs for jobs per minute, the usual graph
is quite flat, see:
http://khack.osdl.org/stp/281932/results/jpm.png

With the v16 scheduler, jobs per minute falls off rapidly
as user number increases giving graph with a steep slope, 
not good.
http://khack.osdl.org/stp/281790/results/jpm.png
Further results: http://www.developer.osdl.org/reaim/index.html

cliffw

Re: Nick's scheduler v16 - reaim

[Nick Piggin]

Cliff White wrote:

>>
>>
>>>>I'm starting to do some large SMP / NUMA testing. Fixed and changed quite
>>>>a bit. It isn't too bad, although I'm only testing dbench, tbench, and
>>>>volanomark at the moment.
>>>>
>>>>These SMP and NUMA changes are not tied to my interactivity stuff, so its
>>>>possible they could get included if they turn out well. If you find any
>>>>problems with it (high end or interactivity), please let me know.
>>>>
>>>>
>
>Results from reaim aren't encouraging.
>patch was applied against 2.6.0-test8 - the result
>is PLM #2232
>
>STP id  kernal name       Max JPM   Max User Pct    elevator
>281932	nick_v16          4923.08    60	     0.00   AS
>281933	nick_v16          5196.06    68	     5.54   deadline
>281722	linux-2.6.0-test8 5432.77    92	     9.38   deadline
>281792	2.6.0-test8-mm1   5384.41    92	     8.56   deadline
>281790	2.6.0-test8-mm1   5392.65    88      8.7    AS
>
>The kernel doesn't perform well at larger user numbers.
>Notice the different in max users.
>Compare the graphs for jobs per minute, the usual graph
>is quite flat, see:
>http://khack.osdl.org/stp/281932/results/jpm.png
>
>With the v16 scheduler, jobs per minute falls off rapidly
>as user number increases giving graph with a steep slope, 
>not good.
>http://khack.osdl.org/stp/281790/results/jpm.png
>Further results: http://www.developer.osdl.org/reaim/index.html
>

Hi Cliff,

Yeah, unfortunately this graph has been a feature of my scheduler for
quite a while :( I think it is caused either by too much balancing
or timeslices getting too small because there are a lot of threads
blocking. Anyway, if you had the time to test this one would be really
good.

http://www.kerneltrap.org/~npiggin/v16/sched-rollup-nopolicy-v16.gz

Nick

Nick's scheduler v17a

[Nick Piggin]

http://www.kerneltrap.org/~npiggin/v17a/

More balancing fixes. I also incorporated some of Andrew Theurer's
ideas. I'm generally getting good numbers now, but using fairly
synthetic benchmarks.

Now would be a good time to test if anyone is interested. Thanks.

Re: Nick's scheduler v17a

[Rhino]

[-- Attachment #1: Type: text/plain, Size: 625 bytes --]

On Thu, 30 Oct 2003 14:41:07 +1100
Nick Piggin <piggin@cyberone.com.au> wrote:

> http://www.kerneltrap.org/~npiggin/v17a/
> 
> More balancing fixes. I also incorporated some of Andrew Theurer's
> ideas. I'm generally getting good numbers now, but using fairly
> synthetic benchmarks.
> 
> Now would be a good time to test if anyone is interested. Thanks.

well i didn't have the time to make extensive tests yet, but the behaviour improved a lot since v16,
and *looks* quite  better compared to test9 on a 2 way xeon p4 with hyperthreading enabled,
seeing  4 cpu's.

when i get back home, I'll try to make a few benchmarks.

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: Nick's scheduler v17a

[Nick Piggin]

Rhino wrote:

>On Thu, 30 Oct 2003 14:41:07 +1100
>Nick Piggin <piggin@cyberone.com.au> wrote:
>
>
>>http://www.kerneltrap.org/~npiggin/v17a/
>>
>>More balancing fixes. I also incorporated some of Andrew Theurer's
>>ideas. I'm generally getting good numbers now, but using fairly
>>synthetic benchmarks.
>>
>>Now would be a good time to test if anyone is interested. Thanks.
>>
>
>well i didn't have the time to make extensive tests yet, but the behaviour improved a lot since v16,
>and *looks* quite  better compared to test9 on a 2 way xeon p4 with hyperthreading enabled,
>seeing  4 cpu's.
>
>when i get back home, I'll try to make a few benchmarks.
>

That would be nice. There still isn't a lot of work done in the HT
department, but something simple and easy like the shared runqueues
patch might be reasonable for 2.6.