All of lore.kernel.org
 help / color / mirror / Atom feed
From: Michal Hocko <mhocko@kernel.org>
To: Srikar Dronamraju <srikar@linux.vnet.ibm.com>
Cc: Ingo Molnar <mingo@kernel.org>,
	Peter Zijlstra <peterz@infradead.org>,
	LKML <linux-kernel@vger.kernel.org>,
	Mel Gorman <mgorman@techsingularity.net>,
	Rik van Riel <riel@redhat.com>
Subject: Re: [PATCH] sched: Fix numabalancing to work with isolated cpus
Date: Thu, 6 Apr 2017 09:34:36 +0200	[thread overview]
Message-ID: <20170406073436.GD5497@dhcp22.suse.cz> (raw)
In-Reply-To: <20170406071950.GA5843@linux.vnet.ibm.com>

On Thu 06-04-17 12:49:50, Srikar Dronamraju wrote:
> > > > > The isolated cpus are part of the cpus allowed list. In the above case,
> > > > > numabalancing ends up scheduling some of these tasks on isolated cpus.
> > > > 
> > > > Why is this bad? If the task is allowed to run on isolated CPUs then why
> > > 
> > > 1. kernel-parameters.txt states: isolcpus as "Isolate CPUs from the
> > > general scheduler." So the expectation that numabalancing can schedule
> > > tasks on it is wrong.
> > 
> > Right but if the task is allowed to run on isolated cpus then the numa
> > balancing for this taks should be allowed to run on those cpus, no?
> 
> No numabalancing or any other scheduler balancing should be looking at
> tasks that are bound to isolated cpus.

Is this documented anywhere? My understanding of isolcpus is to make
sure that nothing _outside_ of the dedicated workload interfers. But why
shouldn't the dedicated workload be numa balanced is not clear to me at
all.

> Similar example that I gave in my reply to Mel.
> 
> Lets consider 2 node, 24 core with 12 cores in each node.
> Cores 0-11 in Node 1 and cores 12-23 in the other node.
> Lets also disable smt/hyperthreading, enable isolcpus from core
> 6-11,12-17.  Lets run 48 thread ebizzy workload and give it a cpu list
> of say 11,12-17 using taskset.
> 
> Now all the 48 ebizzy threads will only run on core 11. It will never
> spread to other cores even in the same node(or in the same node/but
> isolated cpus) or to the different nodes. i.e even if numabalancing is
> running or not, even if my fix is around or not, all threads will be
> confined to core 11, even though the cpus_allowed is 11,12-17.

Isn't that a bug in isolcpus implementation? It is certainly an
unexpected behavior I would say. Is this documented anywhere?

> > Say your application would be bound _only_ to isolated cpus. Should that
> > imply no numa balancing at all?
> 
> Yes, it implies no numa balancing.
> 
> > 
> > > 2. If numabalancing was disabled, the task would never run on the
> > > isolated CPUs.
> > 
> > I am confused. I thought you said "However a task might call
> > sched_setaffinity() that includes all possible cpus in the system
> > including the isolated cpus." So the task is allowed to run there.
> > Or am I missing something?
> > 
> 
> Peter, Rik, Ingo can correct me here.
> 
> I feel most programs that call sched_setaffinity including perf bench
> are written with an assumption that they are never run with isolcpus.

Isn't sched_setaffinity the only way how to actually make it possible to
run on isolcpus?

> > Please note that I do not claim the patch is wrong. I am still not sure
> > myself but the chagelog is missing the most important information "why
> > the change is the right thing".
> 
>  I am open to editing the changelog, I assumed that isolcpus kernel
>  parameter was clear that no scheduling algorithms can interfere with
>  isolcpus. Would stating this in the changelog clarify to you that this
>  change is right?

I would really like to see it confirmed by the scheduler maintainers and
documented properly as well. What you are claiming here is rather
surprising to my understanding of what isolcpus acutally is.
-- 
Michal Hocko
SUSE Labs

  reply	other threads:[~2017-04-06  7:34 UTC|newest]

Thread overview: 18+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2017-04-04 17:27 Srikar Dronamraju
2017-04-04 18:56 ` Rik van Riel
2017-04-04 20:37 ` Mel Gorman
2017-04-05  1:50   ` Srikar Dronamraju
2017-04-05  8:09     ` Mel Gorman
2017-04-05 12:57 ` Michal Hocko
2017-04-05 15:22   ` Srikar Dronamraju
2017-04-05 16:44     ` Michal Hocko
2017-04-06  7:19       ` Srikar Dronamraju
2017-04-06  7:34         ` Michal Hocko [this message]
2017-04-06  9:23           ` Peter Zijlstra
2017-04-06 10:13             ` Michal Hocko
2017-04-06 10:29               ` Peter Zijlstra
2017-04-06 10:42                 ` Michal Hocko
2017-04-06 10:47                   ` Peter Zijlstra
2017-04-06 13:44                     ` Michal Hocko
2017-04-06  7:36 ` Mike Galbraith
2017-04-06  7:36 ` Peter Zijlstra

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20170406073436.GD5497@dhcp22.suse.cz \
    --to=mhocko@kernel.org \
    --cc=linux-kernel@vger.kernel.org \
    --cc=mgorman@techsingularity.net \
    --cc=mingo@kernel.org \
    --cc=peterz@infradead.org \
    --cc=riel@redhat.com \
    --cc=srikar@linux.vnet.ibm.com \
    --subject='Re: [PATCH] sched: Fix numabalancing to work with isolated cpus' \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link

This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.