sched : performance regression 24% between 4.4rc4 and 4.3 kernel

sched : performance regression 24% between 4.4rc4 and 4.3 kernel

[Jirka Hladky]

Hello,

we are doing performance testing of the new kernel scheduler (commit
53528695ff6d8b77011bc818407c13e30914a946). In most cases we see
performance improvements compared to 4.3 kernel with the exception of
stream benchmark when running on 4 NUMA node server.

When we run 4 stream benchmark processes on 4 NUMA node server and we
compare the total performance we see drop about 24% compared to 4.3
kernel. This is caused by the fact that 2 stream benchmarks are
running on the same NUMA node while 1 NUMA node does not run any
stream benchmark. With kernel 4.3, load is distributed evenly among
all 4 NUMA nodes. When two stream benchmarks are running on the same
NUMA node then the runtime is almost twice as long compared to one
stream bench running on one NUMA node. See log files [1] bellow.

Please see the graph comparing stream benchmark results between kernel
4.3 and 4.4rc4 (for legend see [2] bellow).
https://jhladky.fedorapeople.org/sched_stream_kernel_4.3vs4.4rc4/Stream_benchmark_on_4_NUMA_node_server_4.3vs4.4rc4_kernel.png

Could you please help us to identify the root cause of this
regression? We don't have the skills to fix the problem ourselves but
we will be more than happy to test any proposed patch for this issue.

Thanks a lot for your help on that!
Jirka

Further details:

[1] Log files can be downloaded here:
https://jhladky.fedorapeople.org/sched_stream_kernel_4.3vs4.4rc4/4.4RC4_stream_log_files.tar.bz2

$grep "User time" *log
stream.defaultRun.004streams.loop01.instance001.log:User time:  12.370 seconds
stream.defaultRun.004streams.loop01.instance002.log:User time:  10.560 seconds
stream.defaultRun.004streams.loop01.instance003.log:User time:  19.330 seconds
stream.defaultRun.004streams.loop01.instance004.log:User time:  17.820 seconds


$grep "NUMA nodes:" *log
stream.defaultRun.004streams.loop01.instance001.log:NUMA nodes:     2
2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 1 1 1 1 1 1 2 2 2 2 2 2 2 2 2 2
2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2
2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2
2 2 2 2 2 2 2 2 2 2 2 2 2 2
stream.defaultRun.004streams.loop01.instance002.log:NUMA nodes:     0
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0 0 0 0 0
stream.defaultRun.004streams.loop01.instance003.log:NUMA nodes:     3
3 3 3 3 3 3 3 3 0 0 0 0 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3
3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 0 0 0 0 0 0 0 0 0 0 3 3 3 3
3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3
3 3 3 3 3 3 3 3 3 3 3 3 3 3
stream.defaultRun.004streams.loop01.instance004.log:NUMA nodes:     3
3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3
3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3
3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3
3 3 0 0 0 0 0 0 0 0 0 0 0 0

=> please note that NO bench is running on NUMA node #1 and instances
#3 and #4 are running both on NUMA node #3. This has huge performance
impact as stream instances on node #3 need 19 and 17 seconds to finish
compared to 10 and 12 seconds for instances running alone on one NUMA
node.

[2] Graph:
https://jhladky.fedorapeople.org/sched_stream_kernel_4.3vs4.4rc4/Stream_benchmark_on_4_NUMA_node_server_4.3vs4.4rc4_kernel.png

Graph Legend:
GREEN line => kernel 4.3
BLUE line =>    kernel 4.4rc4
x-axis      =>     number of parallel stream instances
y-axis      =>     Sum [1/runtime] over all stream instances


Details on server: DELL PowerEdge R820, 4x E5-4607 0 @ 2.20GHz and 128GB RAM
http://ark.intel.com/products/64604

Re: sched : performance regression 24% between 4.4rc4 and 4.3 kernel

[Mike Galbraith]

(it's always a good idea to CC subsystem maintainers when reporting) 

On Fri, 2015-12-11 at 15:17 +0100, Jirka Hladky wrote:
> Hello,
> 
> we are doing performance testing of the new kernel scheduler (commit
> 53528695ff6d8b77011bc818407c13e30914a946). In most cases we see
> performance improvements compared to 4.3 kernel with the exception of
> stream benchmark when running on 4 NUMA node server.
> 
> When we run 4 stream benchmark processes on 4 NUMA node server and we
> compare the total performance we see drop about 24% compared to 4.3
> kernel. This is caused by the fact that 2 stream benchmarks are
> running on the same NUMA node while 1 NUMA node does not run any
> stream benchmark. With kernel 4.3, load is distributed evenly among
> all 4 NUMA nodes. When two stream benchmarks are running on the same
> NUMA node then the runtime is almost twice as long compared to one
> stream bench running on one NUMA node. See log files [1] bellow.
> 
> Please see the graph comparing stream benchmark results between
> kernel
> 4.3 and 4.4rc4 (for legend see [2] bellow).
> https://jhladky.fedorapeople.org/sched_stream_kernel_4.3vs4.4rc4/Stre
> am_benchmark_on_4_NUMA_node_server_4.3vs4.4rc4_kernel.png
> 
> Could you please help us to identify the root cause of this
> regression? We don't have the skills to fix the problem ourselves but
> we will be more than happy to test any proposed patch for this issue.

A bisection doesn't require any special skills, but may give busy
maintainers a single change to eyeball vs the entire lot. 

	-Mike

Re: sched : performance regression 24% between 4.4rc4 and 4.3 kernel

[Jirka Hladky]

> A bisection doesn't require any special skills, but may give busy
> maintainers a single change to eyeball vs the entire lot.

They have been couple of merges which makes git revert difficult but I
will try to produce patch files for kernel/sched directory only with

git diff v4.4-rc4..fe19159 -- sched

and let you the outcome.



On Sat, Dec 12, 2015 at 8:04 AM, Mike Galbraith
<umgwanakikbuti@gmail.com> wrote:
> (it's always a good idea to CC subsystem maintainers when reporting)
>
> On Fri, 2015-12-11 at 15:17 +0100, Jirka Hladky wrote:
>> Hello,
>>
>> we are doing performance testing of the new kernel scheduler (commit
>> 53528695ff6d8b77011bc818407c13e30914a946). In most cases we see
>> performance improvements compared to 4.3 kernel with the exception of
>> stream benchmark when running on 4 NUMA node server.
>>
>> When we run 4 stream benchmark processes on 4 NUMA node server and we
>> compare the total performance we see drop about 24% compared to 4.3
>> kernel. This is caused by the fact that 2 stream benchmarks are
>> running on the same NUMA node while 1 NUMA node does not run any
>> stream benchmark. With kernel 4.3, load is distributed evenly among
>> all 4 NUMA nodes. When two stream benchmarks are running on the same
>> NUMA node then the runtime is almost twice as long compared to one
>> stream bench running on one NUMA node. See log files [1] bellow.
>>
>> Please see the graph comparing stream benchmark results between
>> kernel
>> 4.3 and 4.4rc4 (for legend see [2] bellow).
>> https://jhladky.fedorapeople.org/sched_stream_kernel_4.3vs4.4rc4/Stre
>> am_benchmark_on_4_NUMA_node_server_4.3vs4.4rc4_kernel.png
>>
>> Could you please help us to identify the root cause of this
>> regression? We don't have the skills to fix the problem ourselves but
>> we will be more than happy to test any proposed patch for this issue.
>
> A bisection doesn't require any special skills, but may give busy
> maintainers a single change to eyeball vs the entire lot.
>
>         -Mike

Re: sched : performance regression 24% between 4.4rc4 and 4.3 kernel

[Mike Galbraith]

On Sat, 2015-12-12 at 15:16 +0100, Jirka Hladky wrote:
> > A bisection doesn't require any special skills, but may give busy
> > maintainers a single change to eyeball vs the entire lot.
> 
> They have been couple of merges which makes git revert difficult...

You could try https://git-scm.com/docs/git-bisect instead.

	-Mike

Re: sched : performance regression 24% between 4.4rc4 and 4.3 kernel

[Jirka Hladky]

Hi all,

I have the results of bisecting:

first bad commit: [973759c80db96ed4b4c5cb85ac7d48107f801371] Merge tag
'v4.3-rc1' into sched/core, to refresh the branch

Could you please have a look at this commit why it has caused the
performance regression when running 4 stream benchmarks in parallel on
4 NUMA node server?

Please let me know if you need additional data. git bisect log is bellow.

Thanks a lot!
Jirka



$ git bisect log
git bisect start '--' 'kernel/sched'
# good: [6a13feb9c82803e2b815eca72fa7a9f5561d7861] Linux 4.3
git bisect good 6a13feb9c82803e2b815eca72fa7a9f5561d7861
# bad: [527e9316f8ec44bd53d90fb9f611fa7ffff52bb9] Linux 4.4-rc4
git bisect bad 527e9316f8ec44bd53d90fb9f611fa7ffff52bb9
# bad: [b99def8b961448f5b9a550dddeeb718e3975e7a6] sched/core: Rework
TASK_DEAD preemption exception
git bisect bad b99def8b961448f5b9a550dddeeb718e3975e7a6
# skip: [8cd5601c50603caa195ce86cc465cb04079ed488] sched/fair: Convert
arch_scale_cpu_capacity() from weak function to #define
git bisect skip 8cd5601c50603caa195ce86cc465cb04079ed488
# bad: [fe19159225d8516f3f57a5fe8f735c01684f0ddd] Merge branch
'sched/urgent' into sched/core, to pick up fixes before applying new
changes
git bisect bad fe19159225d8516f3f57a5fe8f735c01684f0ddd
# bad: [78a9c54649ea220065aad9902460a1d137c7eafd] sched/numa: Rename
numabalancing_enabled to sched_numa_balancing
git bisect bad 78a9c54649ea220065aad9902460a1d137c7eafd
# bad: [6efdb105d392da3ad5cb4ef951aed373cd049813] sched/fair: Fix
switched_to_fair()'s per entity load tracking
git bisect bad 6efdb105d392da3ad5cb4ef951aed373cd049813
# bad: [50a2a3b246149d041065a67ccb3e98145f780a2f] sched/fair: Have
task_move_group_fair() unconditionally add the entity load to the
runqueue
git bisect bad 50a2a3b246149d041065a67ccb3e98145f780a2f
# bad: [973759c80db96ed4b4c5cb85ac7d48107f801371] Merge tag 'v4.3-rc1'
into sched/core, to refresh the branch
git bisect bad 973759c80db96ed4b4c5cb85ac7d48107f801371
# first bad commit: [973759c80db96ed4b4c5cb85ac7d48107f801371] Merge
tag 'v4.3-rc1' into sched/core, to refresh the branch

On Sat, Dec 12, 2015 at 3:37 PM, Mike Galbraith
<umgwanakikbuti@gmail.com> wrote:
> On Sat, 2015-12-12 at 15:16 +0100, Jirka Hladky wrote:
>> > A bisection doesn't require any special skills, but may give busy
>> > maintainers a single change to eyeball vs the entire lot.
>>
>> They have been couple of merges which makes git revert difficult...
>
> You could try https://git-scm.com/docs/git-bisect instead.
>
>         -Mike

Re: sched : performance regression 24% between 4.4rc4 and 4.3 kernel

[Jirka Hladky]

Hi all,

I have the results of bisecting:

first bad commit: [973759c80db96ed4b4c5cb85ac7d48107f801371] Merge tag
'v4.3-rc1' into sched/core, to refresh the branch

Could you please have a look at this commit why it has caused the
performance regression when running 4 stream benchmarks in parallel on
4 NUMA node server?

Please let me know if you need additional data. git bisect log is bellow.

Thanks a lot!
Jirka



$ git bisect log
git bisect start '--' 'kernel/sched'
# good: [6a13feb9c82803e2b815eca72fa7a9f5561d7861] Linux 4.3
git bisect good 6a13feb9c82803e2b815eca72fa7a9f5561d7861
# bad: [527e9316f8ec44bd53d90fb9f611fa7ffff52bb9] Linux 4.4-rc4
git bisect bad 527e9316f8ec44bd53d90fb9f611fa7ffff52bb9
# bad: [b99def8b961448f5b9a550dddeeb718e3975e7a6] sched/core: Rework
TASK_DEAD preemption exception
git bisect bad b99def8b961448f5b9a550dddeeb718e3975e7a6
# skip: [8cd5601c50603caa195ce86cc465cb04079ed488] sched/fair: Convert
arch_scale_cpu_capacity() from weak function to #define
git bisect skip 8cd5601c50603caa195ce86cc465cb04079ed488
# bad: [fe19159225d8516f3f57a5fe8f735c01684f0ddd] Merge branch
'sched/urgent' into sched/core, to pick up fixes before applying new
changes
git bisect bad fe19159225d8516f3f57a5fe8f735c01684f0ddd
# bad: [78a9c54649ea220065aad9902460a1d137c7eafd] sched/numa: Rename
numabalancing_enabled to sched_numa_balancing
git bisect bad 78a9c54649ea220065aad9902460a1d137c7eafd
# bad: [6efdb105d392da3ad5cb4ef951aed373cd049813] sched/fair: Fix
switched_to_fair()'s per entity load tracking
git bisect bad 6efdb105d392da3ad5cb4ef951aed373cd049813
# bad: [50a2a3b246149d041065a67ccb3e98145f780a2f] sched/fair: Have
task_move_group_fair() unconditionally add the entity load to the
runqueue
git bisect bad 50a2a3b246149d041065a67ccb3e98145f780a2f
# bad: [973759c80db96ed4b4c5cb85ac7d48107f801371] Merge tag 'v4.3-rc1'
into sched/core, to refresh the branch
git bisect bad 973759c80db96ed4b4c5cb85ac7d48107f801371
# first bad commit: [973759c80db96ed4b4c5cb85ac7d48107f801371] Merge
tag 'v4.3-rc1' into sched/core, to refresh the branch

On Sat, Dec 12, 2015 at 3:37 PM, Mike Galbraith
<umgwanakikbuti@gmail.com> wrote:
> On Sat, 2015-12-12 at 15:16 +0100, Jirka Hladky wrote:
>> > A bisection doesn't require any special skills, but may give busy
>> > maintainers a single change to eyeball vs the entire lot.
>>
>> They have been couple of merges which makes git revert difficult...
>
> You could try https://git-scm.com/docs/git-bisect instead.
>
>         -Mike

Re: sched : performance regression 24% between 4.4rc4 and 4.3 kernel

[Rik van Riel]

On 12/14/2015 06:52 PM, Jirka Hladky wrote:
> Hi all,
> 
> I have the results of bisecting:
> 
> first bad commit: [973759c80db96ed4b4c5cb85ac7d48107f801371] Merge tag
> 'v4.3-rc1' into sched/core, to refresh the branch
> 
> Could you please have a look at this commit why it has caused the
> performance regression when running 4 stream benchmarks in parallel on 4
> NUMA node server? 

That is a merge commit. It contains no actual code changes.

> Please let me know if you need additional data. git bisect log is bellow. 

It looks like "git bisect" may have led you astray.

I am not sure what debugging tool to use to figure out which
of the patches from some merged-in branch caused the issue,
but hopefully one of the people reading this email know a trick.

-- 
All rights reversed

Re: sched : performance regression 24% between 4.4rc4 and 4.3 kernel

[Jirka Hladky]

Hi Rik,

I have reviewed the data and you are right. The trouble is that even
with 4.3 kernel there is 20% change that results will be bad. I have
repeated tests 100 times on 4.3 kernel over the night. In 20 cases I
see that runtime went up from 12 seconds to 28 seconds due to the
wrong NUMA placement. I will try to replay the bisect once again.

Jirka

On Tue, Dec 15, 2015 at 3:12 AM, Rik van Riel <riel@redhat.com> wrote:
> On 12/14/2015 06:52 PM, Jirka Hladky wrote:
>> Hi all,
>>
>> I have the results of bisecting:
>>
>> first bad commit: [973759c80db96ed4b4c5cb85ac7d48107f801371] Merge tag
>> 'v4.3-rc1' into sched/core, to refresh the branch
>>
>> Could you please have a look at this commit why it has caused the
>> performance regression when running 4 stream benchmarks in parallel on 4
>> NUMA node server?
>
> That is a merge commit. It contains no actual code changes.
>
>> Please let me know if you need additional data. git bisect log is bellow.
>
> It looks like "git bisect" may have led you astray.
>
> I am not sure what debugging tool to use to figure out which
> of the patches from some merged-in branch caused the issue,
> but hopefully one of the people reading this email know a trick.
>
> --
> All rights reversed

Re: sched : performance regression 24% between 4.4rc4 and 4.3 kernel

[Jirka Hladky]

Hi Rik,

I have redone the bisecting and have new results:

# first bad commit: [2a595721a1fa6b684c1c818f379bef834ac3d65e]
sched/numa: Convert sched_numa_balancing to a static_branch

Could you please have a look what went wrong?

Thanks a lot!
Jirka

git bisect start '--' 'kernel/sched'
# good: [6a13feb9c82803e2b815eca72fa7a9f5561d7861] Linux 4.3
git bisect good 6a13feb9c82803e2b815eca72fa7a9f5561d7861
# bad: [527e9316f8ec44bd53d90fb9f611fa7ffff52bb9] Linux 4.4-rc4
git bisect bad 527e9316f8ec44bd53d90fb9f611fa7ffff52bb9
# bad: [b99def8b961448f5b9a550dddeeb718e3975e7a6] sched/core: Rework
TASK_DEAD preemption exception
git bisect bad b99def8b961448f5b9a550dddeeb718e3975e7a6
# skip: [8cd5601c50603caa195ce86cc465cb04079ed488] sched/fair: Convert
arch_scale_cpu_capacity() from weak function to #define
git bisect skip 8cd5601c50603caa195ce86cc465cb04079ed488
# bad: [fe19159225d8516f3f57a5fe8f735c01684f0ddd] Merge branch
'sched/urgent' into sched/core, to pick up fixes before applying new
changes
git bisect bad fe19159225d8516f3f57a5fe8f735c01684f0ddd
# good: [78a9c54649ea220065aad9902460a1d137c7eafd] sched/numa: Rename
numabalancing_enabled to sched_numa_balancing
git bisect good 78a9c54649ea220065aad9902460a1d137c7eafd
# bad: [54a21385facbdcd89a78e8c3e5025f04c5f2b59c] sched/fair: Rename
scale() to cap_scale()
git bisect bad 54a21385facbdcd89a78e8c3e5025f04c5f2b59c
# bad: [9e91d61d9b0ca8d865dbd59af8d0d5c5b68003e9] sched/fair: Name
utilization related data and functions consistently
git bisect bad 9e91d61d9b0ca8d865dbd59af8d0d5c5b68003e9
# bad: [2a595721a1fa6b684c1c818f379bef834ac3d65e] sched/numa: Convert
sched_numa_balancing to a static_branch
git bisect bad 2a595721a1fa6b684c1c818f379bef834ac3d65e
# good: [2b49d84b259fc18e131026e5d38e7855352f71b9] sched/numa: Remove
the NUMA sched_feature
git bisect good 2b49d84b259fc18e131026e5d38e7855352f71b9
# first bad commit: [2a595721a1fa6b684c1c818f379bef834ac3d65e]
sched/numa: Convert sched_numa_balancing to a static_branch

On Tue, Dec 15, 2015 at 9:49 AM, Jirka Hladky <jhladky@redhat.com> wrote:
> Hi Rik,
>
> I have reviewed the data and you are right. The trouble is that even
> with 4.3 kernel there is 20% change that results will be bad. I have
> repeated tests 100 times on 4.3 kernel over the night. In 20 cases I
> see that runtime went up from 12 seconds to 28 seconds due to the
> wrong NUMA placement. I will try to replay the bisect once again.
>
> Jirka
>
> On Tue, Dec 15, 2015 at 3:12 AM, Rik van Riel <riel@redhat.com> wrote:
>> On 12/14/2015 06:52 PM, Jirka Hladky wrote:
>>> Hi all,
>>>
>>> I have the results of bisecting:
>>>
>>> first bad commit: [973759c80db96ed4b4c5cb85ac7d48107f801371] Merge tag
>>> 'v4.3-rc1' into sched/core, to refresh the branch
>>>
>>> Could you please have a look at this commit why it has caused the
>>> performance regression when running 4 stream benchmarks in parallel on 4
>>> NUMA node server?
>>
>> That is a merge commit. It contains no actual code changes.
>>
>>> Please let me know if you need additional data. git bisect log is bellow.
>>
>> It looks like "git bisect" may have led you astray.
>>
>> I am not sure what debugging tool to use to figure out which
>> of the patches from some merged-in branch caused the issue,
>> but hopefully one of the people reading this email know a trick.
>>
>> --
>> All rights reversed

Re: sched : performance regression 24% between 4.4rc4 and 4.3 kernel

[Peter Zijlstra]

On Wed, Dec 16, 2015 at 01:56:17PM +0100, Jirka Hladky wrote:
> Hi Rik,
> 
> I have redone the bisecting and have new results:
> 
> # first bad commit: [2a595721a1fa6b684c1c818f379bef834ac3d65e]
> sched/numa: Convert sched_numa_balancing to a static_branch
> 
> Could you please have a look what went wrong?

The below is obviously wrong, but your kernel should have that patch.

So if you revert this patch (ie. go back to the regular variable) it
works again?

---

commit b52da86e0ad58f096710977fcda856fd84da9233
Author: Srikar Dronamraju <srikar@linux.vnet.ibm.com>
Date:   Fri Oct 2 07:48:25 2015 +0530

    sched/numa: Fix task_tick_fair() from disabling numa_balancing
    
    If static branch 'sched_numa_balancing' is enabled, it should kickstart
    NUMA balancing through task_tick_numa(). However the following commit:
    
      2a595721a1fa ("sched/numa: Convert sched_numa_balancing to a static_branch")
    
    erroneously disables this.
    
    Fix this anomaly by enabling task_tick_numa() when the static branch
    'sched_numa_balancing' is enabled.
    
    Signed-off-by: Srikar Dronamraju <srikar@linux.vnet.ibm.com>
    Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
    Cc: Linus Torvalds <torvalds@linux-foundation.org>
    Cc: Mel Gorman <mgorman@suse.de>
    Cc: Mike Galbraith <efault@gmx.de>
    Cc: Peter Zijlstra <peterz@infradead.org>
    Cc: Rik van Riel <riel@redhat.com>
    Cc: Thomas Gleixner <tglx@linutronix.de>
    Cc: linux-kernel@vger.kernel.org
    Link: http://lkml.kernel.org/r/1443752305-27413-1-git-send-email-srikar@linux.vnet.ibm.com
    Signed-off-by: Ingo Molnar <mingo@kernel.org>

diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c
index 4df37a48f499..3bdc3da7bc6a 100644
--- a/kernel/sched/fair.c
+++ b/kernel/sched/fair.c
@@ -7881,7 +7881,7 @@ static void task_tick_fair(struct rq *rq, struct task_struct *curr, int queued)
 		entity_tick(cfs_rq, se, queued);
 	}
 
-	if (!static_branch_unlikely(&sched_numa_balancing))
+	if (static_branch_unlikely(&sched_numa_balancing))
 		task_tick_numa(rq, curr);
 }
 

Re: sched : performance regression 24% between 4.4rc4 and 4.3 kernel

[Jirka Hladky]

Hi Peter,

you are right the kernel  4.4-rc4 has it already fixed. It seems I
will need to redo the bisecting once again, starting with
2a595721a1fa6b684c1c818f379bef834ac3d65e

git bisect start -- kernel/sched
git bisect bad v4.4-rc4
git bisect good 2b49d84b259fc18e131026e5d38e7855352f71b9
Bisecting: 32 revisions left to test after this (roughly 5 steps)
[da7142e2ed735e1c1bef5a757dc55de35c65fbd6] sched/core: Simplify
preempt_count tests

I will let you know the outcome.

Jirka


On Wed, Dec 16, 2015 at 2:50 PM, Peter Zijlstra <peterz@infradead.org> wrote:
> On Wed, Dec 16, 2015 at 01:56:17PM +0100, Jirka Hladky wrote:
>> Hi Rik,
>>
>> I have redone the bisecting and have new results:
>>
>> # first bad commit: [2a595721a1fa6b684c1c818f379bef834ac3d65e]
>> sched/numa: Convert sched_numa_balancing to a static_branch
>>
>> Could you please have a look what went wrong?
>
> The below is obviously wrong, but your kernel should have that patch.
>
> So if you revert this patch (ie. go back to the regular variable) it
> works again?
>
> ---
>
> commit b52da86e0ad58f096710977fcda856fd84da9233
> Author: Srikar Dronamraju <srikar@linux.vnet.ibm.com>
> Date:   Fri Oct 2 07:48:25 2015 +0530
>
>     sched/numa: Fix task_tick_fair() from disabling numa_balancing
>
>     If static branch 'sched_numa_balancing' is enabled, it should kickstart
>     NUMA balancing through task_tick_numa(). However the following commit:
>
>       2a595721a1fa ("sched/numa: Convert sched_numa_balancing to a static_branch")
>
>     erroneously disables this.
>
>     Fix this anomaly by enabling task_tick_numa() when the static branch
>     'sched_numa_balancing' is enabled.
>
>     Signed-off-by: Srikar Dronamraju <srikar@linux.vnet.ibm.com>
>     Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
>     Cc: Linus Torvalds <torvalds@linux-foundation.org>
>     Cc: Mel Gorman <mgorman@suse.de>
>     Cc: Mike Galbraith <efault@gmx.de>
>     Cc: Peter Zijlstra <peterz@infradead.org>
>     Cc: Rik van Riel <riel@redhat.com>
>     Cc: Thomas Gleixner <tglx@linutronix.de>
>     Cc: linux-kernel@vger.kernel.org
>     Link: http://lkml.kernel.org/r/1443752305-27413-1-git-send-email-srikar@linux.vnet.ibm.com
>     Signed-off-by: Ingo Molnar <mingo@kernel.org>
>
> diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c
> index 4df37a48f499..3bdc3da7bc6a 100644
> --- a/kernel/sched/fair.c
> +++ b/kernel/sched/fair.c
> @@ -7881,7 +7881,7 @@ static void task_tick_fair(struct rq *rq, struct task_struct *curr, int queued)
>                 entity_tick(cfs_rq, se, queued);
>         }
>
> -       if (!static_branch_unlikely(&sched_numa_balancing))
> +       if (static_branch_unlikely(&sched_numa_balancing))
>                 task_tick_numa(rq, curr);
>  }
>

Re: sched : performance regression 24% between 4.4rc4 and 4.3 kernel

[Jirka Hladky]

Hi Peter,

I'm not sure how to do the bisecting and avoid landing at:

[2a595721a1fa6b684c1c818f379bef834ac3d65e] sched/numa: Convert
sched_numa_balancing to a static_branch

I have redone the bisecting but I have landed again at this commit.
Can you please help me to identify the commit which has fixed for
2a595721a1fa6b684c1c818f379bef834ac3d65e ? I think I will need to
start the bisecting from there.

Thanks
Jirka

>
>
> On Wed, Dec 16, 2015 at 6:04 PM, Jirka Hladky <jhladky@redhat.com> wrote:
>>
>> Hi Peter,
>>
>> you are right the kernel  4.4-rc4 has it already fixed. It seems I
>> will need to redo the bisecting once again, starting with
>> 2a595721a1fa6b684c1c818f379bef834ac3d65e
>>
>> git bisect start -- kernel/sched
>> git bisect bad v4.4-rc4
>> git bisect good 2b49d84b259fc18e131026e5d38e7855352f71b9
>> Bisecting: 32 revisions left to test after this (roughly 5 steps)
>> [da7142e2ed735e1c1bef5a757dc55de35c65fbd6] sched/core: Simplify
>> preempt_count tests
>>
>> I will let you know the outcome.
>>
>> Jirka
>>
>>
>> On Wed, Dec 16, 2015 at 2:50 PM, Peter Zijlstra <peterz@infradead.org> wrote:
>> > On Wed, Dec 16, 2015 at 01:56:17PM +0100, Jirka Hladky wrote:
>> >> Hi Rik,
>> >>
>> >> I have redone the bisecting and have new results:
>> >>
>> >> # first bad commit: [2a595721a1fa6b684c1c818f379bef834ac3d65e]
>> >> sched/numa: Convert sched_numa_balancing to a static_branch
>> >>
>> >> Could you please have a look what went wrong?
>> >
>> > The below is obviously wrong, but your kernel should have that patch.
>> >
>> > So if you revert this patch (ie. go back to the regular variable) it
>> > works again?
>> >
>> > ---
>> >
>> > commit b52da86e0ad58f096710977fcda856fd84da9233
>> > Author: Srikar Dronamraju <srikar@linux.vnet.ibm.com>
>> > Date:   Fri Oct 2 07:48:25 2015 +0530
>> >
>> >     sched/numa: Fix task_tick_fair() from disabling numa_balancing
>> >
>> >     If static branch 'sched_numa_balancing' is enabled, it should kickstart
>> >     NUMA balancing through task_tick_numa(). However the following commit:
>> >
>> >       2a595721a1fa ("sched/numa: Convert sched_numa_balancing to a static_branch")
>> >
>> >     erroneously disables this.
>> >
>> >     Fix this anomaly by enabling task_tick_numa() when the static branch
>> >     'sched_numa_balancing' is enabled.
>> >
>> >     Signed-off-by: Srikar Dronamraju <srikar@linux.vnet.ibm.com>
>> >     Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
>> >     Cc: Linus Torvalds <torvalds@linux-foundation.org>
>> >     Cc: Mel Gorman <mgorman@suse.de>
>> >     Cc: Mike Galbraith <efault@gmx.de>
>> >     Cc: Peter Zijlstra <peterz@infradead.org>
>> >     Cc: Rik van Riel <riel@redhat.com>
>> >     Cc: Thomas Gleixner <tglx@linutronix.de>
>> >     Cc: linux-kernel@vger.kernel.org
>> >     Link: http://lkml.kernel.org/r/1443752305-27413-1-git-send-email-srikar@linux.vnet.ibm.com
>> >     Signed-off-by: Ingo Molnar <mingo@kernel.org>
>> >
>> > diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c
>> > index 4df37a48f499..3bdc3da7bc6a 100644
>> > --- a/kernel/sched/fair.c
>> > +++ b/kernel/sched/fair.c
>> > @@ -7881,7 +7881,7 @@ static void task_tick_fair(struct rq *rq, struct task_struct *curr, int queued)
>> >                 entity_tick(cfs_rq, se, queued);
>> >         }
>> >
>> > -       if (!static_branch_unlikely(&sched_numa_balancing))
>> > +       if (static_branch_unlikely(&sched_numa_balancing))
>> >                 task_tick_numa(rq, curr);
>> >  }
>> >
>
>

Re: sched : performance regression 24% between 4.4rc4 and 4.3 kernel

[Mike Galbraith]

On Thu, 2015-12-17 at 16:43 +0100, Jirka Hladky wrote:
> Hi Peter,
> 
> I'm not sure how to do the bisecting and avoid landing at:
> 
> [2a595721a1fa6b684c1c818f379bef834ac3d65e] sched/numa: Convert
> sched_numa_balancing to a static_branch
> 
> I have redone the bisecting but I have landed again at this commit.
> Can you please help me to identify the commit which has fixed for
> 2a595721a1fa6b684c1c818f379bef834ac3d65e ? I think I will need to
> start the bisecting from there.

One way...

Put b52da86e0ad58f096710977fcda856fd84da9233 in quilt queue, try to
quilt push before each build, ignoring whether it applies or not.  If
it does apply, quilt pop before saying git bisect [good/bad] lest git
refuse to play due to you mucking about with its source its source.

	-Mike