[PATCH] nohz: Revert "nohz: Set isolcpus when nohz_full is set"

[PATCH] nohz: Revert "nohz: Set isolcpus when nohz_full is set"

[Frederic Weisbecker]

This reverts commit 8cb9764fc88b41db11f251e8b2a0d006578b7eb4.

We assumed that nohz full users always want scheduler isolation on full
dynticks CPUs, therefore we included nohz full CPUs on cpu_isolated_map.
This means that tasks run by default on CPUs outside the nohz_full range
unless their affinity is explicity overwritten.

This suits pure isolation workloads but when the machine is needed to
run common workloads, the available sets of CPUs to run common tasks
becomes reduced.

We reach an extreme case when CONFIG_NO_HZ_FULL_ALL is enabled as it
leaves only CPU 0 for non-isolation tasks, which makes people think that
their supercomputer regressed to 90's UP.

Some nohz full users appear to be interested in running normal workloads
either before or after an isolation workload. Nohz full isn't optimized
toward normal workloads but it's still better than UP performance.

We are reaching a limitation in kernel presets here. Lets revert this
cpu_isolated_map inclusion and let userspace do its own scheduler
isolation using cpusets or explicit affinity settings.

Reported-by: Ingo Molnar <mingo@kernel.org>
Reported-by: Mike Galbraith <umgwanakikbuti@gmail.com>
Cc: Chris Metcalf <cmetcalf@ezchip.com>
Cc: Rik van Riel <riel@redhat.com>
Cc: Christoph Lameter <cl@linux.com>
Cc: Mike Galbraith <umgwanakikbuti@gmail.com>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Dave Jones <davej@redhat.com>
Cc: Oleg Nesterov <oleg@redhat.com>
Cc: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
Cc: Thomas Gleixner <tglx@linutronix.de>
Cc: Ingo Molnar <mingo@kernel.org>
Cc: Alexey Dobriyan <adobriyan@gmail.com>
Cc: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Frederic Weisbecker <fweisbec@gmail.com>
---
 kernel/sched/core.c | 3 ---
 1 file changed, 3 deletions(-)

diff --git a/kernel/sched/core.c b/kernel/sched/core.c
index 6159531..3c35b5f 100644
--- a/kernel/sched/core.c
+++ b/kernel/sched/core.c
@@ -7238,9 +7238,6 @@ void __init sched_init_smp(void)
 	alloc_cpumask_var(&non_isolated_cpus, GFP_KERNEL);
 	alloc_cpumask_var(&fallback_doms, GFP_KERNEL);
 
-	/* nohz_full won't take effect without isolating the cpus. */
-	tick_nohz_full_add_cpus_to(cpu_isolated_map);
-
 	sched_init_numa();
 
 	/*
-- 
2.5.3

Re: [PATCH] nohz: Revert "nohz: Set isolcpus when nohz_full is set"

[Paul E. McKenney]

On Mon, Oct 12, 2015 at 05:21:23PM +0200, Frederic Weisbecker wrote:
> This reverts commit 8cb9764fc88b41db11f251e8b2a0d006578b7eb4.
> 
> We assumed that nohz full users always want scheduler isolation on full
> dynticks CPUs, therefore we included nohz full CPUs on cpu_isolated_map.
> This means that tasks run by default on CPUs outside the nohz_full range
> unless their affinity is explicity overwritten.
> 
> This suits pure isolation workloads but when the machine is needed to
> run common workloads, the available sets of CPUs to run common tasks
> becomes reduced.
> 
> We reach an extreme case when CONFIG_NO_HZ_FULL_ALL is enabled as it
> leaves only CPU 0 for non-isolation tasks, which makes people think that
> their supercomputer regressed to 90's UP.
> 
> Some nohz full users appear to be interested in running normal workloads
> either before or after an isolation workload. Nohz full isn't optimized
> toward normal workloads but it's still better than UP performance.
> 
> We are reaching a limitation in kernel presets here. Lets revert this
> cpu_isolated_map inclusion and let userspace do its own scheduler
> isolation using cpusets or explicit affinity settings.
> 
> Reported-by: Ingo Molnar <mingo@kernel.org>
> Reported-by: Mike Galbraith <umgwanakikbuti@gmail.com>
> Cc: Chris Metcalf <cmetcalf@ezchip.com>
> Cc: Rik van Riel <riel@redhat.com>
> Cc: Christoph Lameter <cl@linux.com>
> Cc: Mike Galbraith <umgwanakikbuti@gmail.com>
> Cc: Peter Zijlstra <peterz@infradead.org>
> Cc: Dave Jones <davej@redhat.com>
> Cc: Oleg Nesterov <oleg@redhat.com>
> Cc: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
> Cc: Thomas Gleixner <tglx@linutronix.de>
> Cc: Ingo Molnar <mingo@kernel.org>
> Cc: Alexey Dobriyan <adobriyan@gmail.com>
> Cc: Andrew Morton <akpm@linux-foundation.org>
> Signed-off-by: Frederic Weisbecker <fweisbec@gmail.com>
> ---
>  kernel/sched/core.c | 3 ---
>  1 file changed, 3 deletions(-)
> 
> diff --git a/kernel/sched/core.c b/kernel/sched/core.c
> index 6159531..3c35b5f 100644
> --- a/kernel/sched/core.c
> +++ b/kernel/sched/core.c
> @@ -7238,9 +7238,6 @@ void __init sched_init_smp(void)
>  	alloc_cpumask_var(&non_isolated_cpus, GFP_KERNEL);
>  	alloc_cpumask_var(&fallback_doms, GFP_KERNEL);
> 
> -	/* nohz_full won't take effect without isolating the cpus. */
> -	tick_nohz_full_add_cpus_to(cpu_isolated_map);
> -

Why not make this controlled by a boot parameter?  That preserves
the ease of use for those needing it, but avoids problems from people
doing "make randconfig".

							Thanx, Paul

>  	sched_init_numa();
> 
>  	/*
> -- 
> 2.5.3
> 

Re: [PATCH] nohz: Revert "nohz: Set isolcpus when nohz_full is set"

[Frederic Weisbecker]

On Mon, Oct 12, 2015 at 08:32:02AM -0700, Paul E. McKenney wrote:
> On Mon, Oct 12, 2015 at 05:21:23PM +0200, Frederic Weisbecker wrote:
> > This reverts commit 8cb9764fc88b41db11f251e8b2a0d006578b7eb4.
> > 
> > We assumed that nohz full users always want scheduler isolation on full
> > dynticks CPUs, therefore we included nohz full CPUs on cpu_isolated_map.
> > This means that tasks run by default on CPUs outside the nohz_full range
> > unless their affinity is explicity overwritten.
> > 
> > This suits pure isolation workloads but when the machine is needed to
> > run common workloads, the available sets of CPUs to run common tasks
> > becomes reduced.
> > 
> > We reach an extreme case when CONFIG_NO_HZ_FULL_ALL is enabled as it
> > leaves only CPU 0 for non-isolation tasks, which makes people think that
> > their supercomputer regressed to 90's UP.
> > 
> > Some nohz full users appear to be interested in running normal workloads
> > either before or after an isolation workload. Nohz full isn't optimized
> > toward normal workloads but it's still better than UP performance.
> > 
> > We are reaching a limitation in kernel presets here. Lets revert this
> > cpu_isolated_map inclusion and let userspace do its own scheduler
> > isolation using cpusets or explicit affinity settings.
> > 
> > Reported-by: Ingo Molnar <mingo@kernel.org>
> > Reported-by: Mike Galbraith <umgwanakikbuti@gmail.com>
> > Cc: Chris Metcalf <cmetcalf@ezchip.com>
> > Cc: Rik van Riel <riel@redhat.com>
> > Cc: Christoph Lameter <cl@linux.com>
> > Cc: Mike Galbraith <umgwanakikbuti@gmail.com>
> > Cc: Peter Zijlstra <peterz@infradead.org>
> > Cc: Dave Jones <davej@redhat.com>
> > Cc: Oleg Nesterov <oleg@redhat.com>
> > Cc: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
> > Cc: Thomas Gleixner <tglx@linutronix.de>
> > Cc: Ingo Molnar <mingo@kernel.org>
> > Cc: Alexey Dobriyan <adobriyan@gmail.com>
> > Cc: Andrew Morton <akpm@linux-foundation.org>
> > Signed-off-by: Frederic Weisbecker <fweisbec@gmail.com>
> > ---
> >  kernel/sched/core.c | 3 ---
> >  1 file changed, 3 deletions(-)
> > 
> > diff --git a/kernel/sched/core.c b/kernel/sched/core.c
> > index 6159531..3c35b5f 100644
> > --- a/kernel/sched/core.c
> > +++ b/kernel/sched/core.c
> > @@ -7238,9 +7238,6 @@ void __init sched_init_smp(void)
> >  	alloc_cpumask_var(&non_isolated_cpus, GFP_KERNEL);
> >  	alloc_cpumask_var(&fallback_doms, GFP_KERNEL);
> > 
> > -	/* nohz_full won't take effect without isolating the cpus. */
> > -	tick_nohz_full_add_cpus_to(cpu_isolated_map);
> > -
> 
> Why not make this controlled by a boot parameter?  That preserves
> the ease of use for those needing it, but avoids problems from people
> doing "make randconfig".

Well it is already. As you pass nohz_full=1-32, you can pass as well isolcpus=1-32

Re: [PATCH] nohz: Revert "nohz: Set isolcpus when nohz_full is set"

[Paul E. McKenney]

On Mon, Oct 12, 2015 at 06:20:03PM +0200, Frederic Weisbecker wrote:
> On Mon, Oct 12, 2015 at 08:32:02AM -0700, Paul E. McKenney wrote:
> > On Mon, Oct 12, 2015 at 05:21:23PM +0200, Frederic Weisbecker wrote:
> > > This reverts commit 8cb9764fc88b41db11f251e8b2a0d006578b7eb4.
> > > 
> > > We assumed that nohz full users always want scheduler isolation on full
> > > dynticks CPUs, therefore we included nohz full CPUs on cpu_isolated_map.
> > > This means that tasks run by default on CPUs outside the nohz_full range
> > > unless their affinity is explicity overwritten.
> > > 
> > > This suits pure isolation workloads but when the machine is needed to
> > > run common workloads, the available sets of CPUs to run common tasks
> > > becomes reduced.
> > > 
> > > We reach an extreme case when CONFIG_NO_HZ_FULL_ALL is enabled as it
> > > leaves only CPU 0 for non-isolation tasks, which makes people think that
> > > their supercomputer regressed to 90's UP.
> > > 
> > > Some nohz full users appear to be interested in running normal workloads
> > > either before or after an isolation workload. Nohz full isn't optimized
> > > toward normal workloads but it's still better than UP performance.
> > > 
> > > We are reaching a limitation in kernel presets here. Lets revert this
> > > cpu_isolated_map inclusion and let userspace do its own scheduler
> > > isolation using cpusets or explicit affinity settings.
> > > 
> > > Reported-by: Ingo Molnar <mingo@kernel.org>
> > > Reported-by: Mike Galbraith <umgwanakikbuti@gmail.com>
> > > Cc: Chris Metcalf <cmetcalf@ezchip.com>
> > > Cc: Rik van Riel <riel@redhat.com>
> > > Cc: Christoph Lameter <cl@linux.com>
> > > Cc: Mike Galbraith <umgwanakikbuti@gmail.com>
> > > Cc: Peter Zijlstra <peterz@infradead.org>
> > > Cc: Dave Jones <davej@redhat.com>
> > > Cc: Oleg Nesterov <oleg@redhat.com>
> > > Cc: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
> > > Cc: Thomas Gleixner <tglx@linutronix.de>
> > > Cc: Ingo Molnar <mingo@kernel.org>
> > > Cc: Alexey Dobriyan <adobriyan@gmail.com>
> > > Cc: Andrew Morton <akpm@linux-foundation.org>
> > > Signed-off-by: Frederic Weisbecker <fweisbec@gmail.com>
> > > ---
> > >  kernel/sched/core.c | 3 ---
> > >  1 file changed, 3 deletions(-)
> > > 
> > > diff --git a/kernel/sched/core.c b/kernel/sched/core.c
> > > index 6159531..3c35b5f 100644
> > > --- a/kernel/sched/core.c
> > > +++ b/kernel/sched/core.c
> > > @@ -7238,9 +7238,6 @@ void __init sched_init_smp(void)
> > >  	alloc_cpumask_var(&non_isolated_cpus, GFP_KERNEL);
> > >  	alloc_cpumask_var(&fallback_doms, GFP_KERNEL);
> > > 
> > > -	/* nohz_full won't take effect without isolating the cpus. */
> > > -	tick_nohz_full_add_cpus_to(cpu_isolated_map);
> > > -
> > 
> > Why not make this controlled by a boot parameter?  That preserves
> > the ease of use for those needing it, but avoids problems from people
> > doing "make randconfig".
> 
> Well it is already. As you pass nohz_full=1-32, you can pass as well isolcpus=1-32

True enough.  Not sure that having to repeat the CPU list twice qualifies as
"easy to use", though.  Why not a nohz_full_iso or some such that isolates
whatever CPUs you specified?

But we really need people using this to weigh in.

							Thanx, Paul

Re: [PATCH] nohz: Revert "nohz: Set isolcpus when nohz_full is set"

[Chris Metcalf]

On 10/12/2015 12:53 PM, Paul E. McKenney wrote:
> On Mon, Oct 12, 2015 at 06:20:03PM +0200, Frederic Weisbecker wrote:
>> On Mon, Oct 12, 2015 at 08:32:02AM -0700, Paul E. McKenney wrote:
>>> On Mon, Oct 12, 2015 at 05:21:23PM +0200, Frederic Weisbecker wrote:
>>>> This reverts commit 8cb9764fc88b41db11f251e8b2a0d006578b7eb4.
>>>>
>>>> We assumed that nohz full users always want scheduler isolation on full
>>>> dynticks CPUs, therefore we included nohz full CPUs on cpu_isolated_map.
>>>> This means that tasks run by default on CPUs outside the nohz_full range
>>>> unless their affinity is explicity overwritten.
>>>>
>>>> This suits pure isolation workloads but when the machine is needed to
>>>> run common workloads, the available sets of CPUs to run common tasks
>>>> becomes reduced.
>>>>
>>>> We reach an extreme case when CONFIG_NO_HZ_FULL_ALL is enabled as it
>>>> leaves only CPU 0 for non-isolation tasks, which makes people think that
>>>> their supercomputer regressed to 90's UP.
>>>>
>>>> Some nohz full users appear to be interested in running normal workloads
>>>> either before or after an isolation workload. Nohz full isn't optimized
>>>> toward normal workloads but it's still better than UP performance.
>>>>
>>>> We are reaching a limitation in kernel presets here. Lets revert this
>>>> cpu_isolated_map inclusion and let userspace do its own scheduler
>>>> isolation using cpusets or explicit affinity settings.
>>>>
>>>> Reported-by: Ingo Molnar <mingo@kernel.org>
>>>> Reported-by: Mike Galbraith <umgwanakikbuti@gmail.com>
>>>> Cc: Chris Metcalf <cmetcalf@ezchip.com>
>>>> Cc: Rik van Riel <riel@redhat.com>
>>>> Cc: Christoph Lameter <cl@linux.com>
>>>> Cc: Mike Galbraith <umgwanakikbuti@gmail.com>
>>>> Cc: Peter Zijlstra <peterz@infradead.org>
>>>> Cc: Dave Jones <davej@redhat.com>
>>>> Cc: Oleg Nesterov <oleg@redhat.com>
>>>> Cc: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
>>>> Cc: Thomas Gleixner <tglx@linutronix.de>
>>>> Cc: Ingo Molnar <mingo@kernel.org>
>>>> Cc: Alexey Dobriyan <adobriyan@gmail.com>
>>>> Cc: Andrew Morton <akpm@linux-foundation.org>
>>>> Signed-off-by: Frederic Weisbecker <fweisbec@gmail.com>
>>>> ---
>>>>   kernel/sched/core.c | 3 ---
>>>>   1 file changed, 3 deletions(-)
>>>>
>>>> diff --git a/kernel/sched/core.c b/kernel/sched/core.c
>>>> index 6159531..3c35b5f 100644
>>>> --- a/kernel/sched/core.c
>>>> +++ b/kernel/sched/core.c
>>>> @@ -7238,9 +7238,6 @@ void __init sched_init_smp(void)
>>>>   	alloc_cpumask_var(&non_isolated_cpus, GFP_KERNEL);
>>>>   	alloc_cpumask_var(&fallback_doms, GFP_KERNEL);
>>>>
>>>> -	/* nohz_full won't take effect without isolating the cpus. */
>>>> -	tick_nohz_full_add_cpus_to(cpu_isolated_map);
>>>> -
>>> Why not make this controlled by a boot parameter?  That preserves
>>> the ease of use for those needing it, but avoids problems from people
>>> doing "make randconfig".
>> Well it is already. As you pass nohz_full=1-32, you can pass as well isolcpus=1-32
> True enough.  Not sure that having to repeat the CPU list twice qualifies as
> "easy to use", though.  Why not a nohz_full_iso or some such that isolates
> whatever CPUs you specified?

Is it worth starting to think about grouping things under the
"task isolation" model somehow?  "task_isolation_cpus=1-31"
or some such for this, and then that just sets up the nohz_full
and isolcpus options under the hood?

-- 
Chris Metcalf, EZChip Semiconductor
http://www.ezchip.com

Re: [PATCH] nohz: Revert "nohz: Set isolcpus when nohz_full is set"

[Frederic Weisbecker]

On Mon, Oct 12, 2015 at 12:55:24PM -0400, Chris Metcalf wrote:
> On 10/12/2015 12:53 PM, Paul E. McKenney wrote:
> 
> Is it worth starting to think about grouping things under the
> "task isolation" model somehow?  "task_isolation_cpus=1-31"
> or some such for this, and then that just sets up the nohz_full
> and isolcpus options under the hood?

Yeah if I could do it again, I would have rather created something like
cpu_isolation= (which name would conflict with isolcpus though) instead
of nohz_full=, because nohz_full= is really just a subset of what people
want.

But yeah if you guys want to create a new parameter that gathers nohz
and isolcpus I think we can.

task_isolation is really just about tasks so it should be another name.

Re: [PATCH] nohz: Revert "nohz: Set isolcpus when nohz_full is set"

[Christoph Lameter]

On Mon, 12 Oct 2015, Frederic Weisbecker wrote:

> But yeah if you guys want to create a new parameter that gathers nohz
> and isolcpus I think we can.

Could we also add the rcu settings?

> task_isolation is really just about tasks so it should be another name.

no_os_cpus?

os_free_cpus?

Re: [PATCH] nohz: Revert "nohz: Set isolcpus when nohz_full is set"

[Frederic Weisbecker]

On Mon, Oct 12, 2015 at 12:45:08PM -0500, Christoph Lameter wrote:
> On Mon, 12 Oct 2015, Frederic Weisbecker wrote:
> 
> > But yeah if you guys want to create a new parameter that gathers nohz
> > and isolcpus I think we can.
> 
> Could we also add the rcu settings?

Yeah, those are implied by nohz_full anyway.

Now nohz_full shouldn't imply anything else than nohz but I'm not sure
I can change that. Kernel parameters are ABIs of some sort.

Ideally we should have something like cpu_isolation= which implies everything.

> 
> > task_isolation is really just about tasks so it should be another name.
> 
> no_os_cpus?
>
> os_free_cpus?

No, we can still do syscalls :-)

But noise_free_cpus= ?

Re: [PATCH] nohz: Revert "nohz: Set isolcpus when nohz_full is set"

[Paul E. McKenney]

On Mon, Oct 12, 2015 at 12:45:08PM -0500, Christoph Lameter wrote:
> On Mon, 12 Oct 2015, Frederic Weisbecker wrote:
> 
> > But yeah if you guys want to create a new parameter that gathers nohz
> > and isolcpus I think we can.
> 
> Could we also add the rcu settings?

I thought that I already had.  What am I still missing?

> > task_isolation is really just about tasks so it should be another name.
> 
> no_os_cpus?
> 
> os_free_cpus?

os_jitter_free_cpus?

clean_cpus?

quiet_cpus?

I have no problem with any of the names proposed thus far, but obviously
could not resist the urge to propose a few more.

							Thanx, Paul

Re: [PATCH] nohz: Revert "nohz: Set isolcpus when nohz_full is set"

[Frederic Weisbecker]

On Mon, Oct 12, 2015 at 10:52:08AM -0700, Paul E. McKenney wrote:
> On Mon, Oct 12, 2015 at 12:45:08PM -0500, Christoph Lameter wrote:
> > On Mon, 12 Oct 2015, Frederic Weisbecker wrote:
> > 
> > > But yeah if you guys want to create a new parameter that gathers nohz
> > > and isolcpus I think we can.
> > 
> > Could we also add the rcu settings?
> 
> I thought that I already had.  What am I still missing?
> 
> > > task_isolation is really just about tasks so it should be another name.
> > 
> > no_os_cpus?
> > 
> > os_free_cpus?
> 
> os_jitter_free_cpus?
> 
> clean_cpus?
> 
> quiet_cpus?
> 
> I have no problem with any of the names proposed thus far, but obviously
> could not resist the urge to propose a few more.

I moved forward quite slowly with nohz full over the years but at least I'm proud
that this subsystem introduced some of the most interesting naming challenges!

Re: [PATCH] nohz: Revert "nohz: Set isolcpus when nohz_full is set"

[Paul E. McKenney]

On Mon, Oct 12, 2015 at 07:55:01PM +0200, Frederic Weisbecker wrote:
> On Mon, Oct 12, 2015 at 10:52:08AM -0700, Paul E. McKenney wrote:
> > On Mon, Oct 12, 2015 at 12:45:08PM -0500, Christoph Lameter wrote:
> > > On Mon, 12 Oct 2015, Frederic Weisbecker wrote:
> > > 
> > > > But yeah if you guys want to create a new parameter that gathers nohz
> > > > and isolcpus I think we can.
> > > 
> > > Could we also add the rcu settings?
> > 
> > I thought that I already had.  What am I still missing?
> > 
> > > > task_isolation is really just about tasks so it should be another name.
> > > 
> > > no_os_cpus?
> > > 
> > > os_free_cpus?
> > 
> > os_jitter_free_cpus?
> > 
> > clean_cpus?
> > 
> > quiet_cpus?
> > 
> > I have no problem with any of the names proposed thus far, but obviously
> > could not resist the urge to propose a few more.
> 
> I moved forward quite slowly with nohz full over the years but at least I'm proud
> that this subsystem introduced some of the most interesting naming challenges!

;-) ;-) ;-)

							Thanx, Paul

Re: [PATCH] nohz: Revert "nohz: Set isolcpus when nohz_full is set"

[Christoph Lameter]

On Mon, 12 Oct 2015, Paul E. McKenney wrote:

> ;-) ;-) ;-)

bare_metal_cpus=?

metal=

iron_maiden=

uhh skip the last one.

Re: [PATCH] nohz: Revert "nohz: Set isolcpus when nohz_full is set"

[Chris Metcalf]

On 10/12/2015 01:42 PM, Frederic Weisbecker wrote:
> On Mon, Oct 12, 2015 at 12:55:24PM -0400, Chris Metcalf wrote:
>> >On 10/12/2015 12:53 PM, Paul E. McKenney wrote:
>> >
>> >Is it worth starting to think about grouping things under the
>> >"task isolation" model somehow?  "task_isolation_cpus=1-31"
>> >or some such for this, and then that just sets up the nohz_full
>> >and isolcpus options under the hood?
> Yeah if I could do it again, I would have rather created something like
> cpu_isolation= (which name would conflict with isolcpus though)

Well, it only sort of conflicts.  I think given that they would be clearly
documented they are different enough not to create too much
confusion.  And cpu_isolation really does seem like a good name.

-- 
Chris Metcalf, EZChip Semiconductor
http://www.ezchip.com

Re: [PATCH] nohz: Revert "nohz: Set isolcpus when nohz_full is set"

[Thomas Gleixner]

On Mon, 12 Oct 2015, Frederic Weisbecker wrote:
> This reverts commit 8cb9764fc88b41db11f251e8b2a0d006578b7eb4.
> 
> We assumed that nohz full users always want scheduler isolation on full
> dynticks CPUs, therefore we included nohz full CPUs on cpu_isolated_map.
> This means that tasks run by default on CPUs outside the nohz_full range
> unless their affinity is explicity overwritten.
> 
> This suits pure isolation workloads but when the machine is needed to
> run common workloads, the available sets of CPUs to run common tasks
> becomes reduced.
> 
> We reach an extreme case when CONFIG_NO_HZ_FULL_ALL is enabled as it
> leaves only CPU 0 for non-isolation tasks, which makes people think that
> their supercomputer regressed to 90's UP.
> 
> Some nohz full users appear to be interested in running normal workloads
> either before or after an isolation workload. Nohz full isn't optimized
> toward normal workloads but it's still better than UP performance.
> 
> We are reaching a limitation in kernel presets here. Lets revert this
> cpu_isolated_map inclusion and let userspace do its own scheduler
> isolation using cpusets or explicit affinity settings.

Acked-by: Thomas Gleixner <tglx@linutronix.de>

[tip:sched/core] nohz: Revert "nohz: Set isolcpus when nohz_full is set"

[tip-bot for Frederic Weisbecker]

Commit-ID:  0baabb385eb4bce699ddab0db015112be6cf1e6a
Gitweb:     http://git.kernel.org/tip/0baabb385eb4bce699ddab0db015112be6cf1e6a
Author:     Frederic Weisbecker <fweisbec@gmail.com>
AuthorDate: Mon, 12 Oct 2015 17:21:23 +0200
Committer:  Ingo Molnar <mingo@kernel.org>
CommitDate: Tue, 20 Oct 2015 10:13:36 +0200

nohz: Revert "nohz: Set isolcpus when nohz_full is set"

This reverts:

  8cb9764fc88b ("nohz: Set isolcpus when nohz_full is set")

We assumed that full-nohz users always want scheduler isolation on full
dynticks CPUs, therefore we included full-nohz CPUs on cpu_isolated_map.

This means that tasks run by default on CPUs outside the nohz_full range
unless their affinity is explicity overwritten.

This suits pure isolation workloads but when the machine is needed to
run common workloads, the available sets of CPUs to run common tasks
becomes reduced.

We reach an extreme case when CONFIG_NO_HZ_FULL_ALL is enabled as it
leaves only CPU 0 for non-isolation tasks, which makes people think that
their supercomputer regressed to 90's UP - which is true in a sense.

Some full-nohz users appear to be interested in running normal workloads
either before or after an isolation workload. Full-nohz isn't optimized
toward normal workloads but it's still better than UP performance.

We are reaching a limitation in kernel presets here. Lets revert this
cpu_isolated_map inclusion and let userspace do its own scheduler
isolation using cpusets or explicit affinity settings.

Reported-by: Ingo Molnar <mingo@kernel.org>
Reported-by: Mike Galbraith <umgwanakikbuti@gmail.com>
Signed-off-by: Frederic Weisbecker <fweisbec@gmail.com>
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Acked-by: Thomas Gleixner <tglx@linutronix.de>
Cc: Alexey Dobriyan <adobriyan@gmail.com>
Cc: Andrew Morton <akpm@linux-foundation.org>
Cc: Chris Metcalf <cmetcalf@ezchip.com>
Cc: Christoph Lameter <cl@linux.com>
Cc: Dave Jones <davej@redhat.com>
Cc: Linus Torvalds <torvalds@linux-foundation.org>
Cc: Mike Galbraith <efault@gmx.de>
Cc: Oleg Nesterov <oleg@redhat.com>
Cc: Paul E . McKenney <paulmck@linux.vnet.ibm.com>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Rik van Riel <riel@redhat.com>
Link: http://lkml.kernel.org/r/1444663283-30068-1-git-send-email-fweisbec@gmail.com
Signed-off-by: Ingo Molnar <mingo@kernel.org>
---
 kernel/sched/core.c | 3 ---
 1 file changed, 3 deletions(-)

diff --git a/kernel/sched/core.c b/kernel/sched/core.c
index 10a8faa..5bd7d60 100644
--- a/kernel/sched/core.c
+++ b/kernel/sched/core.c
@@ -7238,9 +7238,6 @@ void __init sched_init_smp(void)
 	alloc_cpumask_var(&non_isolated_cpus, GFP_KERNEL);
 	alloc_cpumask_var(&fallback_doms, GFP_KERNEL);
 
-	/* nohz_full won't take effect without isolating the cpus. */
-	tick_nohz_full_add_cpus_to(cpu_isolated_map);
-
 	sched_init_numa();
 
 	/*