From: Waiman Long <longman@redhat.com>
To: Tejun Heo <tj@kernel.org>, Zefan Li <lizefan.x@bytedance.com>,
Johannes Weiner <hannes@cmpxchg.org>,
Jonathan Corbet <corbet@lwn.net>, Shuah Khan <shuah@kernel.org>
Cc: cgroups@vger.kernel.org, linux-kernel@vger.kernel.org,
linux-doc@vger.kernel.org, linux-kselftest@vger.kernel.org,
"Andrew Morton" <akpm@linux-foundation.org>,
"Roman Gushchin" <guro@fb.com>, "Phil Auld" <pauld@redhat.com>,
"Peter Zijlstra" <peterz@infradead.org>,
"Juri Lelli" <juri.lelli@redhat.com>,
"Frederic Weisbecker" <frederic@kernel.org>,
"Marcelo Tosatti" <mtosatti@redhat.com>,
"Michal Koutný" <mkoutny@suse.com>,
"Waiman Long" <longman@redhat.com>
Subject: [PATCH v8 5/6] cgroup/cpuset: Update description of cpuset.cpus.partition in cgroup-v2.rst
Date: Mon, 18 Oct 2021 10:36:18 -0400 [thread overview]
Message-ID: <20211018143619.205065-6-longman@redhat.com> (raw)
In-Reply-To: <20211018143619.205065-1-longman@redhat.com>
Update Documentation/admin-guide/cgroup-v2.rst on the newly introduced
"isolated" cpuset partition type as well as other changes made in other
cpuset patches.
Signed-off-by: Waiman Long <longman@redhat.com>
---
Documentation/admin-guide/cgroup-v2.rst | 153 ++++++++++++++----------
1 file changed, 93 insertions(+), 60 deletions(-)
diff --git a/Documentation/admin-guide/cgroup-v2.rst b/Documentation/admin-guide/cgroup-v2.rst
index 4d8c27eca96b..40d39562a8dd 100644
--- a/Documentation/admin-guide/cgroup-v2.rst
+++ b/Documentation/admin-guide/cgroup-v2.rst
@@ -2091,8 +2091,9 @@ Cpuset Interface Files
It accepts only the following input values when written to.
======== ================================
- "root" a partition root
- "member" a non-root member of a partition
+ "member" Non-root member of a partition
+ "root" Partition root
+ "isolated" Partition root without load balancing
======== ================================
When set to be a partition root, the current cgroup is the
@@ -2101,64 +2102,96 @@ Cpuset Interface Files
partition roots themselves and their descendants. The root
cgroup is always a partition root.
- There are constraints on where a partition root can be set.
- It can only be set in a cgroup if all the following conditions
- are true.
-
- 1) The "cpuset.cpus" is not empty and the list of CPUs are
- exclusive, i.e. they are not shared by any of its siblings.
- 2) The parent cgroup is a partition root.
- 3) The "cpuset.cpus" is also a proper subset of the parent's
- "cpuset.cpus.effective".
- 4) There is no child cgroups with cpuset enabled. This is for
- eliminating corner cases that have to be handled if such a
- condition is allowed.
-
- Setting it to partition root will take the CPUs away from the
- effective CPUs of the parent cgroup. Once it is set, this
- file cannot be reverted back to "member" if there are any child
- cgroups with cpuset enabled.
-
- A parent partition cannot distribute all its CPUs to its
- child partitions. There must be at least one cpu left in the
- parent partition.
-
- Once becoming a partition root, changes to "cpuset.cpus" is
- generally allowed as long as the first condition above is true,
- the change will not take away all the CPUs from the parent
- partition and the new "cpuset.cpus" value is a superset of its
- children's "cpuset.cpus" values.
-
- Sometimes, external factors like changes to ancestors'
- "cpuset.cpus" or cpu hotplug can cause the state of the partition
- root to change. On read, the "cpuset.sched.partition" file
- can show the following values.
-
- ============== ==============================
- "member" Non-root member of a partition
- "root" Partition root
- "root invalid" Invalid partition root
- ============== ==============================
-
- It is a partition root if the first 2 partition root conditions
- above are true and at least one CPU from "cpuset.cpus" is
- granted by the parent cgroup.
-
- A partition root can become invalid if none of CPUs requested
- in "cpuset.cpus" can be granted by the parent cgroup or the
- parent cgroup is no longer a partition root itself. In this
- case, it is not a real partition even though the restriction
- of the first partition root condition above will still apply.
- The cpu affinity of all the tasks in the cgroup will then be
- associated with CPUs in the nearest ancestor partition.
-
- An invalid partition root can be transitioned back to a
- real partition root if at least one of the requested CPUs
- can now be granted by its parent. In this case, the cpu
- affinity of all the tasks in the formerly invalid partition
- will be associated to the CPUs of the newly formed partition.
- Changing the partition state of an invalid partition root to
- "member" is always allowed even if child cpusets are present.
+ When set to "isolated", the CPUs in that partition root will
+ be in an isolated state without any load balancing from the
+ scheduler. Tasks in such a partition must be explicitly bound
+ to each individual CPU.
+
+ "cpuset.cpus" must always be set up first before enabling
+ partition. Unlike "member" whose "cpuset.cpus.effective" can
+ contain CPUs not in "cpuset.cpus", this can never happen with a
+ valid partition root. In other words, "cpuset.cpus.effective"
+ is always a subset of "cpuset.cpus" for a valid partition root.
+
+ When a parent partition root cannot exclusively grant any of
+ the CPUs specified in "cpuset.cpus", "cpuset.cpus.effective"
+ becomes empty. If there are tasks in the partition root, the
+ partition root becomes invalid and "cpuset.cpus.effective"
+ is reset to that of the nearest non-empty ancestor.
+
+ Note that a task cannot be moved to a cgroup with empty
+ "cpuset.cpus.effective".
+
+ There are additional constraints on where a partition root can
+ be enabled ("root" or "isolated"). It can only be enabled in
+ a cgroup if all the following conditions are met.
+
+ 1) The "cpuset.cpus" is non-empty and exclusive, i.e. they are
+ not shared by any of its siblings.
+ 2) The parent cgroup is a valid partition root.
+ 3) The "cpuset.cpus" is a subset of parent's "cpuset.cpus".
+ 4) There is no child cgroups with cpuset enabled. This avoids
+ cpu migrations of multiple cgroups simultaneously which can
+ be problematic.
+
+ On read, the "cpuset.cpus.partition" file can show the following
+ values.
+
+ ====================== ==============================
+ "member" Non-root member of a partition
+ "root" Partition root
+ "isolated" Partition root without load balancing
+ "root invalid (<reason>)" Invalid partition root
+ ====================== ==============================
+
+ In the case of an invalid partition root, a descriptive string on
+ why the partition is invalid is included within parentheses.
+
+ Once becoming a partition root, changes to "cpuset.cpus"
+ is generally allowed as long as the cpu list is exclusive,
+ non-empty and is a superset of children's cpu lists.
+
+ The constraints of a valid partition root are as follows:
+
+ 1) The parent cgroup is a valid partition root.
+ 2) "cpuset.cpus.effective" is a subset of "cpuset.cpus"
+ 3) "cpuset.cpus.effective" is non-empty when there are tasks
+ in the partition.
+
+ Changes to "cpuset.cpus" or cpu hotplug may cause the state
+ of a valid partition root to become invalid when one or more
+ constraints of a valid partition root are violated. Therefore,
+ user space agents that manage partition roots should avoid
+ unnecessary changes to "cpuset.cpus" and always check the state
+ of "cpuset.cpus.partition" after making changes to make sure
+ that the partitions are functioning properly as expected.
+
+ Changing a partition root to "member" is always allowed.
+ If there are child partition roots underneath it, however,
+ they will be forced to be switched back to "member" too and
+ lose their partitions. So care must be taken to double check
+ for this condition before disabling a partition root.
+
+ Setting a cgroup to a valid partition root will take the CPUs
+ away from the effective CPUs of the parent partition.
+
+ A valid parent partition may distribute out all its CPUs to
+ its child partitions as long as it is not the root cgroup as
+ we need some house-keeping CPUs in the root cgroup.
+
+ An invalid partition is not a real partition even though some
+ internal states may still be kept.
+
+ An invalid partition root can be reverted back to a real
+ partition root if none of the constraints of a valid partition
+ root are violated.
+
+ Poll and inotify events are triggered whenever the state of
+ "cpuset.cpus.partition" changes. That includes changes caused by
+ write to "cpuset.cpus.partition", cpu hotplug and other changes
+ that make the partition invalid. This will allow user space
+ agents to monitor unexpected changes to "cpuset.cpus.partition"
+ without the need to do continuous polling.
Device controller
--
2.27.0
next prev parent reply other threads:[~2021-10-18 14:39 UTC|newest]
Thread overview: 39+ messages / expand[flat|nested] mbox.gz Atom feed top
2021-10-18 14:36 [PATCH v8 0/6] cgroup/cpuset: Add new cpuset partition type & empty effecitve cpus Waiman Long
2021-10-18 14:36 ` [PATCH v8 1/6] cgroup/cpuset: Allow no-task partition to have empty cpuset.cpus.effective Waiman Long
2021-10-18 14:36 ` [PATCH v8 2/6] cgroup/cpuset: Refining features and constraints of a partition Waiman Long
2021-10-18 14:36 ` [PATCH v8 3/6] cgroup/cpuset: Add a new isolated cpus.partition type Waiman Long
2021-10-18 14:36 ` [PATCH v8 4/6] cgroup/cpuset: Show invalid partition reason string Waiman Long
2021-10-18 14:36 ` Waiman Long [this message]
2021-11-15 19:31 ` [PATCH v8 5/6] cgroup/cpuset: Update description of cpuset.cpus.partition in cgroup-v2.rst Michal Koutný
2021-11-15 20:11 ` Tejun Heo
2021-11-15 21:27 ` Waiman Long
2021-11-15 21:10 ` Waiman Long
2021-11-16 17:54 ` Michal Koutný
2021-11-30 15:35 ` Waiman Long
2021-11-30 17:11 ` Tejun Heo
2021-12-01 3:56 ` Waiman Long
2021-12-01 14:13 ` Michal Koutný
2021-12-01 14:56 ` Waiman Long
2021-12-01 16:39 ` Tejun Heo
2021-12-01 17:49 ` Waiman Long
2021-12-01 14:26 ` Waiman Long
2021-12-01 16:46 ` Tejun Heo
2021-12-01 18:05 ` Waiman Long
2021-12-02 1:28 ` Waiman Long
2021-12-03 18:25 ` Michal Koutný
2021-12-03 19:27 ` Waiman Long
2021-10-18 14:36 ` [PATCH v8 6/6] kselftest/cgroup: Add cpuset v2 partition root state test Waiman Long
2021-10-27 23:05 ` [PATCH v8 0/6] cgroup/cpuset: Add new cpuset partition type & empty effecitve cpus Waiman Long
2021-11-10 11:13 ` Felix Moessbauer
2021-11-10 13:21 ` Marcelo Tosatti
2021-11-10 13:56 ` Michal Koutný
2021-11-10 15:21 ` Moessbauer, Felix
2021-11-10 16:10 ` Marcelo Tosatti
2021-11-10 16:14 ` Marcelo Tosatti
2021-11-10 16:15 ` Jan Kiszka
2021-11-10 17:29 ` Marcelo Tosatti
2021-11-10 18:30 ` Waiman Long
2021-11-10 17:52 ` Michal Koutný
2021-11-10 18:04 ` Jan Kiszka
2021-11-10 18:15 ` Michal Koutný
2021-11-10 15:20 ` Waiman Long
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20211018143619.205065-6-longman@redhat.com \
--to=longman@redhat.com \
--cc=akpm@linux-foundation.org \
--cc=cgroups@vger.kernel.org \
--cc=corbet@lwn.net \
--cc=frederic@kernel.org \
--cc=guro@fb.com \
--cc=hannes@cmpxchg.org \
--cc=juri.lelli@redhat.com \
--cc=linux-doc@vger.kernel.org \
--cc=linux-kernel@vger.kernel.org \
--cc=linux-kselftest@vger.kernel.org \
--cc=lizefan.x@bytedance.com \
--cc=mkoutny@suse.com \
--cc=mtosatti@redhat.com \
--cc=pauld@redhat.com \
--cc=peterz@infradead.org \
--cc=shuah@kernel.org \
--cc=tj@kernel.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).