linux-kernel.vger.kernel.org archive mirror
 help / color / mirror / Atom feed
From: Valentin Schneider <valentin.schneider@arm.com>
To: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>,
	Li Zefan <lizefan@huawei.com>
Cc: linux-kernel <linux-kernel@vger.kernel.org>,
	Peter Zijlstra <peterz@infradead.org>,
	Ingo Molnar <mingo@kernel.org>
Subject: Re: [regression] cpuset: offlined CPUs removed from affinity masks
Date: Thu, 16 Jan 2020 18:27:02 +0000	[thread overview]
Message-ID: <3010d5a9-cce9-bc79-22c0-365d88f1135c@arm.com> (raw)
In-Reply-To: <1251528473.590671.1579196495905.JavaMail.zimbra@efficios.com>

On 16/01/2020 17:41, Mathieu Desnoyers wrote:
> Hi,
> 
> I noticed the following regression with CONFIG_CPUSET=y. Note that
> I am not using cpusets at all (only using the root cpuset I'm given
> at boot), it's just configured in. I am currently working on a 5.2.5
> kernel. I am simply combining use of taskset(1) (setting the affinity
> mask of a process) and cpu hotplug. The result is that with
> CONFIG_CPUSET=y, setting the affinity mask including an offline CPU number
> don't keep that CPU in the affinity mask, and it is never put back when the
> CPU comes back online. CONFIG_CPUSET=n behaves as expected, and puts back
> the CPU into the affinity mask reported to user-space when it comes back
> online.
> 
> 
> * With CONFIG_CPUSET=y (unexpected behavior):
> 
> # echo 0 > /sys/devices/system/cpu/cpu1/online
> 
> % taskset 0x7 ./loop &
> % taskset -p $!
> pid 1341's current affinity mask: 5
> 
> # echo 1 > /sys/devices/system/cpu/cpu1/online
> 
> taskset -p $!
> pid 1341's current affinity mask: 5
> 
> kill $!
> 

As discussed on IRC, this is because we have in sched_setaffinity():

  cpuset_cpus_allowed(p, cpus_allowed);
  cpumask_and(new_mask, in_mask, cpus_allowed);

Another source of issue is that CPUs are taken out of cpusets when
hotplugged out, and not put back in when hotplugged back in (except for the
root cpuset which follows cpu_active_mask).

Both cpuset.effective_cpus and cpuset.allowed_cpus seem to only span
online CPUs:

  root@valsch-juno:~# cat /sys/fs/cgroup/cpuset/cpuset.effective_cpus 
  0-5
  root@valsch-juno:~# cat /sys/fs/cgroup/cpuset/cpuset.cpus
  0-5
  root@valsch-juno:~# echo 0 > /sys/devices/system/cpu/cpu3/online                                     
  [93418.733050] CPU3: shutdown
  [93418.735815] psci: CPU3 killed (polled 0 ms)
  root@valsch-juno:~# cat /sys/fs/cgroup/cpuset/cpuset.cpus                                            
  0-2,4-5
  root@valsch-juno:~# cat /sys/fs/cgroup/cpuset/cpuset.effective_cpus                                  
  0-2,4-5

The thing is, with CONFIG_CPUSET=n, we can absolutely cope with p->cpus_ptr
spanning CPUs that are offline because we still check the active/online
mask (is_cpu_allowed()). So one thing I'd like to know is why do cpusets
remove offline cpus from their mask? I could see cpuset.allowed containing
both online & offline CPUs, and cpuset.effective containing just the online
ones.

That way in sched_setaffinity() we can still check for cpuset.allowed, and
we still have the online/active check in __set_cpus_allowed_ptr() to deny
stupid requests.

  reply	other threads:[~2020-01-16 18:27 UTC|newest]

Thread overview: 16+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2020-01-16 17:41 [regression] cpuset: offlined CPUs removed from affinity masks Mathieu Desnoyers
2020-01-16 18:27 ` Valentin Schneider [this message]
2020-02-17 16:03 ` Mathieu Desnoyers
2020-02-19 15:19   ` Tejun Heo
2020-02-19 15:43     ` Mathieu Desnoyers
2020-02-19 15:47       ` Tejun Heo
2020-02-19 15:50         ` Mathieu Desnoyers
2020-02-19 15:52           ` Tejun Heo
2020-02-19 16:08             ` Mathieu Desnoyers
2020-02-19 16:12               ` Tejun Heo
2020-03-07 16:06                 ` Mathieu Desnoyers
2020-03-12 18:26                   ` Tejun Heo
2020-03-12 19:47                     ` Mathieu Desnoyers
2020-03-24 18:01                       ` Tejun Heo
2020-03-24 19:30                         ` Mathieu Desnoyers
2020-03-30 19:53                           ` Mathieu Desnoyers

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=3010d5a9-cce9-bc79-22c0-365d88f1135c@arm.com \
    --to=valentin.schneider@arm.com \
    --cc=linux-kernel@vger.kernel.org \
    --cc=lizefan@huawei.com \
    --cc=mathieu.desnoyers@efficios.com \
    --cc=mingo@kernel.org \
    --cc=peterz@infradead.org \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).