Re: [PATCH v2] sched/core: Don't mix isolcpus and housekeeping CPUs

From: Peter Zijlstra <peterz@infradead.org>
To: Srikar Dronamraju <srikar@linux.vnet.ibm.com>
Cc: Ingo Molnar <mingo@kernel.org>,
	LKML <linux-kernel@vger.kernel.org>,
	Mel Gorman <mgorman@techsingularity.net>,
	Rik van Riel <riel@surriel.com>, Yi Wang <wang.yi59@zte.com.cn>,
	zhong.weidong@zte.com.cn, Yi Liu <liu.yi24@zte.com.cn>,
	Frederic Weisbecker <frederic@kernel.org>,
	Thomas Gleixner <tglx@linutronix.de>
Subject: Re: [PATCH v2] sched/core: Don't mix isolcpus and housekeeping CPUs
Date: Fri, 26 Oct 2018 10:41:05 +0200	[thread overview]
Message-ID: <20181026084105.GY3109@worktop.c.hoisthospitality.com> (raw)
In-Reply-To: <20181025173058.GD18466@linux.vnet.ibm.com>

On Thu, Oct 25, 2018 at 11:00:58PM +0530, Srikar Dronamraju wrote:
> > But it doesn't solve the problem.
> > 
> > You can create multiple partitions with cpusets but still have an
> > unbound task in the root cgroup. That would suffer the exact same
> > problems.
> > 
> > Thing is, load-balancing, of any kind, should respect sched_domains, and
> > currently numa balancing barely looks at it.
> 
> Agreed that we should have looked at sched_domains. However I still believe
> we can't have task->cpus_allowed with a mix of isolcpus and non-isolcpus.
> won't it lead to inconsistent behaviour?

We currently can; and it depends on what you call inconsistent.

> > The proposed patch puts the minimal constraints on the numa balancer to
> > respect sched_domains; but doesn't yet correctly deal with hotplug.
> 
> I was also thinking about hotplug. Also your proposed patch and even my
> proposed patch don't seem to work well with the below scenario.
> 
> # cat /sys/devices/system/cpu/possible
> 0-31
> # cat /sys/devices/system/cpu/isolated
> 1,5,9,13
> # cat hist.sh
> echo 0 > /proc/sys/kernel/numa_balancing
> cd /sys/fs/cgroup/cpuset
> mkdir -p student
> cp cpuset.mems student/
> cd student
> echo "0-31" > cpuset.cpus
> echo $$ > cgroup.procs 
> echo "1-8" > cpuset.cpus
> /home/srikar/work/ebizzy-0.3/ebizzy -S 1000 &
> PID=$!
> sleep 10
> pidstat -p $! -t |tail -n +3 |head -n 10
> pidstat -p $$ -t |tail -n +3
> pkill ebizzy
> #
> # ./hist.sh
> 10:35:21  IST   UID      TGID       TID    %usr %system  %guest    %CPU   CPU  Command
> 10:35:21  IST     0      2645         -    8.70    0.01    0.00    8.71     1  ebizzy
> 10:35:21  IST     0         -      2645    0.01    0.00    0.00    0.01     1  |__ebizzy
> 10:35:21  IST     0         -      2647    0.14    0.00    0.00    0.14     1  |__ebizzy
> 10:35:21  IST     0         -      2648    0.13    0.00    0.00    0.13     1  |__ebizzy
> 10:35:21  IST     0         -      2649    0.13    0.00    0.00    0.13     1  |__ebizzy
> 10:35:21  IST     0         -      2650    0.13    0.00    0.00    0.13     1  |__ebizzy
> 10:35:21  IST     0         -      2651    0.13    0.00    0.00    0.13     1  |__ebizzy
> 10:35:21  IST     0         -      2652    0.13    0.00    0.00    0.13     1  |__ebizzy
> 10:35:21  IST     0         -      2653    0.13    0.00    0.00    0.13     1  |__ebizzy
> 10:35:23  IST   UID      TGID       TID    %usr %system  %guest    %CPU   CPU  Command
> 10:35:23  IST     0      2642         -    0.00    0.00    0.00    0.00     1  hist.sh
> 10:35:23  IST     0         -      2642    0.00    0.00    0.00    0.00     1  |__hist.sh
> #

That's correct and specified behaviour.

cpu1 has no sched domain (or the singleton domaon, which is the same)
and thus its tasks will not be migrated.

If there is a problem, it is with cpuset and its interaction with
isolcpus. I still have to look at the last version of cpuset-v2, but
iirc that was better in this regard.

I'm stil saying we should kill isolcpus entirely, its rubbish, has
always been.