[SCHEDULER] Performance drop in 4.19 compared to 4.18 kernel

[SCHEDULER] Performance drop in 4.19 compared to 4.18 kernel

[Jirka Hladky]

Hi Srikar,

I work at Red Hat in the Kernel Performance Team. I would like to ask
you for help.

We have detected a significant performance drop (20% and more) with
4.19rc1 relatively to 4.18 vanilla. We see the regression on different
2 NUMA and 4 NUMA boxes with pretty much all the benchmarks we use -
NAS, Stream, SPECjbb2005, SPECjvm2008.

Mel Gorman has suggested checking
2d4056fafa196e1ab4e7161bae4df76f9602d56d commit - with it reverted we
got some performance back but not entirely:

 * Compared to 4.18, there is still performance regression -
especially with NAS (sp_C_x subtest) and SPECjvm2008. On 4 NUMA
systems, regression is around 10-15%
  * Compared to 4.19rc1 there is a clear gain across all benchmarks, up to 20%.

We are investigating the issue further, Mel has suggested to check
305c1fac3225dfa7eeb89bfe91b7335a6edd5172 as next.

Do you have any further recommendations, which commits have possibly
caused the performance degradation?

I want to discuss with you how can we collaborate on performance
testing for the upstream kernel. Does your testing show as well
performance drop in 4.19? If so, do you have any plans for the fix? If
no, can we send you some more information about our tests so that you
can try to reproduce it?

We would also be more than happy to test the new patches for the
performance - please let us know if you are interested.  We have a
pool of 1 NUMA up to 8 NUMA boxes for that, both AMD and Intel,
covering different CPU generations from Sandy Bridge till Skylake.

I'm looking forward to hearing from you.
Jirka

Re: [SCHEDULER] Performance drop in 4.19 compared to 4.18 kernel

[Srikar Dronamraju]

* Jirka Hladky <jhladky@redhat.com> [2018-09-07 11:34:49]:

Hi Jirka,

> 
> We have detected a significant performance drop (20% and more) with
> 4.19rc1 relatively to 4.18 vanilla. We see the regression on different
> 2 NUMA and 4 NUMA boxes with pretty much all the benchmarks we use -
> NAS, Stream, SPECjbb2005, SPECjvm2008.
> 

Do you run single instance of these benchmarks?
I generally run specjbb2005 (single and multi instance). I should be able to
run stream.  I have tried running NAS but I couldnt set it up properly.

I also run a set of perf bench scripts but thats not a real workload.
However perf bench gives me a visual perspective of how things are
converging. I also run an internal benchmark that mimics a trading
application.

> Mel Gorman has suggested checking
> 2d4056fafa196e1ab4e7161bae4df76f9602d56d commit - with it reverted we
> got some performance back but not entirely:
> 
>  * Compared to 4.18, there is still performance regression -
> especially with NAS (sp_C_x subtest) and SPECjvm2008. On 4 NUMA
> systems, regression is around 10-15%
>   * Compared to 4.19rc1 there is a clear gain across all benchmarks, up to 20%.
> 
> We are investigating the issue further, Mel has suggested to check
> 305c1fac3225dfa7eeb89bfe91b7335a6edd5172 as next.

Can you please pick


1. 69bb3230297e881c797bbc4b3dbf73514078bc9d sched/numa: Stop multiple tasks
from moving to the cpu at the same time
2. dc62cfdac5e5b7a61cd8a2bd4190e80b9bb408fc sched/numa: Avoid task migration
for small numa improvement
3. 76e18a67cdd9e3609716c8a074c03168734736f9 sched/numa: Pass destination cpu as
a parameter to migrate_task_rq
4.  489c19b440ebdbabffe530b9a41389d0a8b315d9 sched/numa: Reset scan rate
whenever task moves across nodes
5.  b7e9ae1ae3825f35cd0f38f1f0c8e91ea145bc30 sched/numa: Limit the
conditions where scan period is reset

from https://git.kernel.org/pub/scm/linux/kernel/git/peterz/queue.git/commit/kernel/sched


You may also want to try with reverting
f03bb6760b8e5e2bcecc88d2a2ef41c09adcab39 sched/numa: Use task faults only if
numa_group is not yet set

> I want to discuss with you how can we collaborate on performance
> testing for the upstream kernel. Does your testing show as well
> performance drop in 4.19? If so, do you have any plans for the fix? If
> no, can we send you some more information about our tests so that you
> can try to reproduce it?

While I have not kept record of the performance numbers on the upstream
kernel, I have some rough patches on scheduler from a performance point of
view. I will try to clean up and send them out soon. (Will copy you when
sending them out).

> 
> We would also be more than happy to test the new patches for the
> performance - please let us know if you are interested.  We have a
> pool of 1 NUMA up to 8 NUMA boxes for that, both AMD and Intel,
> covering different CPU generations from Sandy Bridge till Skylake.
> 

I generally test on Power8 (4 node, 16 node), 2 node Power 9, 2 node skylake
and 4 node Power 7.  Surely I will keep you informed and eager to know the
results of your experiments.

-- 
Thanks and Regards
Srikar Dronamraju

Re: [SCHEDULER] Performance drop in 4.19 compared to 4.18 kernel

[Peter Zijlstra]

On Fri, Sep 07, 2018 at 06:26:49PM +0530, Srikar Dronamraju wrote:

> Can you please pick
> 
> 
> 1. 69bb3230297e881c797bbc4b3dbf73514078bc9d sched/numa: Stop multiple tasks
> from moving to the cpu at the same time
> 2. dc62cfdac5e5b7a61cd8a2bd4190e80b9bb408fc sched/numa: Avoid task migration
> for small numa improvement
> 3. 76e18a67cdd9e3609716c8a074c03168734736f9 sched/numa: Pass destination cpu as
> a parameter to migrate_task_rq
> 4.  489c19b440ebdbabffe530b9a41389d0a8b315d9 sched/numa: Reset scan rate
> whenever task moves across nodes
> 5.  b7e9ae1ae3825f35cd0f38f1f0c8e91ea145bc30 sched/numa: Limit the
> conditions where scan period is reset
> 
> from https://git.kernel.org/pub/scm/linux/kernel/git/peterz/queue.git/commit/kernel/sched

That is not a stable tree; the whole thing is re-generated from my quilt
set every time I feel like it.

It is likely those commit ids will no longer exist in a few hours.

Re: [SCHEDULER] Performance drop in 4.19 compared to 4.18 kernel

[Srikar Dronamraju]

* Peter Zijlstra <peterz@infradead.org> [2018-09-07 15:19:23]:

> On Fri, Sep 07, 2018 at 06:26:49PM +0530, Srikar Dronamraju wrote:
> 
> > Can you please pick
> > 
> > 
> > 1. 69bb3230297e881c797bbc4b3dbf73514078bc9d sched/numa: Stop multiple tasks
> > from moving to the cpu at the same time
> > 2. dc62cfdac5e5b7a61cd8a2bd4190e80b9bb408fc sched/numa: Avoid task migration
> > for small numa improvement
> > 3. 76e18a67cdd9e3609716c8a074c03168734736f9 sched/numa: Pass destination cpu as
> > a parameter to migrate_task_rq
> > 4.  489c19b440ebdbabffe530b9a41389d0a8b315d9 sched/numa: Reset scan rate
> > whenever task moves across nodes
> > 5.  b7e9ae1ae3825f35cd0f38f1f0c8e91ea145bc30 sched/numa: Limit the
> > conditions where scan period is reset
> > 
> > from https://git.kernel.org/pub/scm/linux/kernel/git/peterz/queue.git/commit/kernel/sched
> 
> That is not a stable tree; the whole thing is re-generated from my quilt
> set every time I feel like it.
> 
> It is likely those commit ids will no longer exist in a few hours.
> 

Okay, I will forward the relevant mails to Jirka.

-- 
Thanks and Regards
Srikar Dronamraju

Re: [SCHEDULER] Performance drop in 4.19 compared to 4.18 kernel

[Peter Zijlstra]

On Fri, Sep 07, 2018 at 07:14:20PM +0530, Srikar Dronamraju wrote:
> * Peter Zijlstra <peterz@infradead.org> [2018-09-07 15:19:23]:
> 
> > On Fri, Sep 07, 2018 at 06:26:49PM +0530, Srikar Dronamraju wrote:
> > 
> > > Can you please pick
> > > 
> > > 
> > > 1. 69bb3230297e881c797bbc4b3dbf73514078bc9d sched/numa: Stop multiple tasks
> > > from moving to the cpu at the same time
> > > 2. dc62cfdac5e5b7a61cd8a2bd4190e80b9bb408fc sched/numa: Avoid task migration
> > > for small numa improvement
> > > 3. 76e18a67cdd9e3609716c8a074c03168734736f9 sched/numa: Pass destination cpu as
> > > a parameter to migrate_task_rq
> > > 4.  489c19b440ebdbabffe530b9a41389d0a8b315d9 sched/numa: Reset scan rate
> > > whenever task moves across nodes
> > > 5.  b7e9ae1ae3825f35cd0f38f1f0c8e91ea145bc30 sched/numa: Limit the
> > > conditions where scan period is reset
> > > 
> > > from https://git.kernel.org/pub/scm/linux/kernel/git/peterz/queue.git/commit/kernel/sched
> > 
> > That is not a stable tree; the whole thing is re-generated from my quilt
> > set every time I feel like it.
> > 
> > It is likely those commit ids will no longer exist in a few hours.
> > 
> 
> Okay, I will forward the relevant mails to Jirka.

Or he can click on that link and find new IDs :-)

Re: [SCHEDULER] Performance drop in 4.19 compared to 4.18 kernel

[Jirka Hladky]

Hi Peter and Srikar,

thanks a lot for the information and for the patches to test!

> I have bounced the 5 patches to you, (one of the 6 has not been applied by
> Peter) so I have skipped that.
> They can also be fetched from
> http://lore.kernel.org/lkml/1533276841-16341-1-git-send-email-srikar@linux.vnet.ibm.com

We have started the benchmarks, I will report the results on Monday.

> I generally run specjbb2005 (single and multi instance).
We also run a single and multi-instance specjbb2005 test.

> I have tried running NAS but I couldn't set it up properly.
We run the OMP variant and we control the number of threads with the
OMP_NUM_THREADS env variable. The setup is quite simple:

cd NPB_sources/config/
mv suite_x86_64.def suite.def
cd ..
make suite

FYI - starting from 4.17 kernel there is a significant performance
drop compared to 4.16 kernel. Mel has come up with a
sched-numa-fast-crossnode-v1r12 patch series
git.kernel.org/pub/scm/linux/kernel/git/mel/linux.git which we have
tested extensively and with it, the benchmarks results are back at
4.16 level. As I understand it, Mel's patch series depends on your
patch series and can be only merged when your patches are completed.

Thanks!
Jirka


On Fri, Sep 7, 2018 at 3:52 PM, Peter Zijlstra <peterz@infradead.org> wrote:
>
> On Fri, Sep 07, 2018 at 07:14:20PM +0530, Srikar Dronamraju wrote:
> > * Peter Zijlstra <peterz@infradead.org> [2018-09-07 15:19:23]:
> >
> > > On Fri, Sep 07, 2018 at 06:26:49PM +0530, Srikar Dronamraju wrote:
> > >
> > > > Can you please pick
> > > >
> > > >
> > > > 1. 69bb3230297e881c797bbc4b3dbf73514078bc9d sched/numa: Stop multiple tasks
> > > > from moving to the cpu at the same time
> > > > 2. dc62cfdac5e5b7a61cd8a2bd4190e80b9bb408fc sched/numa: Avoid task migration
> > > > for small numa improvement
> > > > 3. 76e18a67cdd9e3609716c8a074c03168734736f9 sched/numa: Pass destination cpu as
> > > > a parameter to migrate_task_rq
> > > > 4.  489c19b440ebdbabffe530b9a41389d0a8b315d9 sched/numa: Reset scan rate
> > > > whenever task moves across nodes
> > > > 5.  b7e9ae1ae3825f35cd0f38f1f0c8e91ea145bc30 sched/numa: Limit the
> > > > conditions where scan period is reset
> > > >
> > > > from https://git.kernel.org/pub/scm/linux/kernel/git/peterz/queue.git/commit/kernel/sched
> > >
> > > That is not a stable tree; the whole thing is re-generated from my quilt
> > > set every time I feel like it.
> > >
> > > It is likely those commit ids will no longer exist in a few hours.
> > >
> >
> > Okay, I will forward the relevant mails to Jirka.
>
> Or he can click on that link and find new IDs :-)

Re: [SCHEDULER] Performance drop in 4.19 compared to 4.18 kernel

[Jirka Hladky]

Hi Peter and Srikar,

> I have bounced the 5 patches to you, (one of the 6 has not been applied by
> Peter) so I have skipped that.
> They can also be fetched from
> http://lore.kernel.org/lkml/1533276841-16341-1-git-send-email-srikar@linux.vnet.ibm.com

I'm sorry for the delay, we have finally the results for the above
kernel. The performance results look good compared to 4.19 vanilla and
are about the same as Mel's sched-numa-fast-crossnode-v1r12 patch set
for 4.18:

Compared to kernel-4.19.0-0.rc1.1

  * Improvement upto 20% for SPECjbb2005, SPECjvm2008 benchmarks
  * Improvement upto 50% for stream benchmark
  * Improvement upto 100% for the NAS benchmark (sp_C subtest, 8
threads on 4 NUMA system with 4x E5-4610 v2 @ 2.30GHz, 64 cores in
total)

When I compare it against Mel's patchset
(git://git.kernel.org/pub/scm/linux/kernel/git/mel/linux.git
sched-numa-fast-crossnode-v1r12)

  * Mel's kernel is about 15% faster for stream benchmarks
  * The other benchmarks show very similar results with both kernels

Mel's patchset eliminates NUMA migration rate limits, this is
presumbly the reason for the good stream results.

Do you have any update when the current patchset could be merged into
the upstream kernel?

Thanks a lot!
Jirka

On Sun, Sep 9, 2018 at 4:03 PM, Jirka Hladky <jhladky@redhat.com> wrote:
>
> Hi Peter and Srikar,
>
> thanks a lot for the information and for the patches to test!
>
> > I have bounced the 5 patches to you, (one of the 6 has not been applied by
> > Peter) so I have skipped that.
> > They can also be fetched from
> > http://lore.kernel.org/lkml/1533276841-16341-1-git-send-email-srikar@linux.vnet.ibm.com
>
> We have started the benchmarks, I will report the results on Monday.
>
> > I generally run specjbb2005 (single and multi instance).
> We also run a single and multi-instance specjbb2005 test.
>
> > I have tried running NAS but I couldn't set it up properly.
> We run the OMP variant and we control the number of threads with the
> OMP_NUM_THREADS env variable. The setup is quite simple:
>
> cd NPB_sources/config/
> mv suite_x86_64.def suite.def
> cd ..
> make suite
>
> FYI - starting from 4.17 kernel there is a significant performance
> drop compared to 4.16 kernel. Mel has come up with a
> sched-numa-fast-crossnode-v1r12 patch series
> git.kernel.org/pub/scm/linux/kernel/git/mel/linux.git which we have
> tested extensively and with it, the benchmarks results are back at
> 4.16 level. As I understand it, Mel's patch series depends on your
> patch series and can be only merged when your patches are completed.
>
> Thanks!
> Jirka
>
>
> On Fri, Sep 7, 2018 at 3:52 PM, Peter Zijlstra <peterz@infradead.org> wrote:
> >
> > On Fri, Sep 07, 2018 at 07:14:20PM +0530, Srikar Dronamraju wrote:
> > > * Peter Zijlstra <peterz@infradead.org> [2018-09-07 15:19:23]:
> > >
> > > > On Fri, Sep 07, 2018 at 06:26:49PM +0530, Srikar Dronamraju wrote:
> > > >
> > > > > Can you please pick
> > > > >
> > > > >
> > > > > 1. 69bb3230297e881c797bbc4b3dbf73514078bc9d sched/numa: Stop multiple tasks
> > > > > from moving to the cpu at the same time
> > > > > 2. dc62cfdac5e5b7a61cd8a2bd4190e80b9bb408fc sched/numa: Avoid task migration
> > > > > for small numa improvement
> > > > > 3. 76e18a67cdd9e3609716c8a074c03168734736f9 sched/numa: Pass destination cpu as
> > > > > a parameter to migrate_task_rq
> > > > > 4.  489c19b440ebdbabffe530b9a41389d0a8b315d9 sched/numa: Reset scan rate
> > > > > whenever task moves across nodes
> > > > > 5.  b7e9ae1ae3825f35cd0f38f1f0c8e91ea145bc30 sched/numa: Limit the
> > > > > conditions where scan period is reset
> > > > >
> > > > > from https://git.kernel.org/pub/scm/linux/kernel/git/peterz/queue.git/commit/kernel/sched
> > > >
> > > > That is not a stable tree; the whole thing is re-generated from my quilt
> > > > set every time I feel like it.
> > > >
> > > > It is likely those commit ids will no longer exist in a few hours.
> > > >
> > >
> > > Okay, I will forward the relevant mails to Jirka.
> >
> > Or he can click on that link and find new IDs :-)

Re: [SCHEDULER] Performance drop in 4.19 compared to 4.18 kernel

[Mel Gorman]

On Fri, Sep 14, 2018 at 04:50:20PM +0200, Jirka Hladky wrote:
> Hi Peter and Srikar,
> 
> > I have bounced the 5 patches to you, (one of the 6 has not been applied by
> > Peter) so I have skipped that.
> > They can also be fetched from
> > http://lore.kernel.org/lkml/1533276841-16341-1-git-send-email-srikar@linux.vnet.ibm.com
> 
> I'm sorry for the delay, we have finally the results for the above
> kernel. The performance results look good compared to 4.19 vanilla and
> are about the same as Mel's sched-numa-fast-crossnode-v1r12 patch set
> for 4.18:
> 
> Compared to kernel-4.19.0-0.rc1.1
> 
>   * Improvement upto 20% for SPECjbb2005, SPECjvm2008 benchmarks
>   * Improvement upto 50% for stream benchmark
>   * Improvement upto 100% for the NAS benchmark (sp_C subtest, 8
> threads on 4 NUMA system with 4x E5-4610 v2 @ 2.30GHz, 64 cores in
> total)
> 
> When I compare it against Mel's patchset
> (git://git.kernel.org/pub/scm/linux/kernel/git/mel/linux.git
> sched-numa-fast-crossnode-v1r12)
> 
>   * Mel's kernel is about 15% faster for stream benchmarks
>   * The other benchmarks show very similar results with both kernels
> 
> Mel's patchset eliminates NUMA migration rate limits, this is
> presumbly the reason for the good stream results.
> 
> Do you have any update when the current patchset could be merged into
> the upstream kernel?
> 

I'm travelling at the moment but when I get back, I'll see what's in the
tip tree with respect to Srikar's patches and then rebase the fast-migration
patches on top and reconfirm they still behave as expected. Assuming
they do, I'll resend them.

Re: [SCHEDULER] Performance drop in 4.19 compared to 4.18 kernel

[Jirka Hladky]

Resending in the plain text mode.

> I'm travelling at the moment but when I get back, I'll see what's in the
> tip tree with respect to Srikar's patches and then rebase the fast-migration
> patches on top and reconfirm they still behave as expected. Assuming
> they do, I'll resend them.


Sounds great, thank you!

If you want me to retest the rebased patch set, just let me know, we
would be more than happy to run the tests on our side.

Jirka

On Mon, Sep 17, 2018 at 3:06 PM, Mel Gorman <mgorman@techsingularity.net> wrote:
> On Fri, Sep 14, 2018 at 04:50:20PM +0200, Jirka Hladky wrote:
>> Hi Peter and Srikar,
>>
>> > I have bounced the 5 patches to you, (one of the 6 has not been applied by
>> > Peter) so I have skipped that.
>> > They can also be fetched from
>> > http://lore.kernel.org/lkml/1533276841-16341-1-git-send-email-srikar@linux.vnet.ibm.com
>>
>> I'm sorry for the delay, we have finally the results for the above
>> kernel. The performance results look good compared to 4.19 vanilla and
>> are about the same as Mel's sched-numa-fast-crossnode-v1r12 patch set
>> for 4.18:
>>
>> Compared to kernel-4.19.0-0.rc1.1
>>
>>   * Improvement upto 20% for SPECjbb2005, SPECjvm2008 benchmarks
>>   * Improvement upto 50% for stream benchmark
>>   * Improvement upto 100% for the NAS benchmark (sp_C subtest, 8
>> threads on 4 NUMA system with 4x E5-4610 v2 @ 2.30GHz, 64 cores in
>> total)
>>
>> When I compare it against Mel's patchset
>> (git://git.kernel.org/pub/scm/linux/kernel/git/mel/linux.git
>> sched-numa-fast-crossnode-v1r12)
>>
>>   * Mel's kernel is about 15% faster for stream benchmarks
>>   * The other benchmarks show very similar results with both kernels
>>
>> Mel's patchset eliminates NUMA migration rate limits, this is
>> presumbly the reason for the good stream results.
>>
>> Do you have any update when the current patchset could be merged into
>> the upstream kernel?
>>
>
> I'm travelling at the moment but when I get back, I'll see what's in the
> tip tree with respect to Srikar's patches and then rebase the fast-migration
> patches on top and reconfirm they still behave as expected. Assuming
> they do, I'll resend them.

Re: [SCHEDULER] Performance drop in 4.19 compared to 4.18 kernel

[Jon Masters]

On 9/7/18 5:34 AM, Jirka Hladky wrote:

> We would also be more than happy to test the new patches for the
> performance - please let us know if you are interested.  We have a
> pool of 1 NUMA up to 8 NUMA boxes for that, both AMD and Intel,
> covering different CPU generations from Sandy Bridge till Skylake.

I've followed up internally on ensuring Arm is added to your mix. Can
get you suitable 1 and 2P systems for internal use.

Jon.