[PATCH 0/2] isolation: Exclude dynamically isolated CPUs from housekeeping masks

[PATCH 0/2] isolation: Exclude dynamically isolated CPUs from housekeeping masks

[Waiman Long]

The housekeeping CPU masks, set up by the "isolcpus" and "nohz_full"
boot command line options, are used at boot time to exclude selected
CPUs from running some kernel housekeeping processes to minimize
disturbance to latency sensitive userspace applications. However, some
of housekeeping CPU masks are also checked at run time to avoid using
those isolated CPUs.

The purpose of this patch series is to exclude dynamically isolated
CPUs from some housekeeping masks so that subsystems that check the
housekeeping masks at run time will not see those isolated CPUs. It does
not migrate the housekeeping processes that have been running on those
newly isolated CPUs since bootup to other CPUs. That will hopefully be
done in the near future.

This patch series only updates the HK_TYPE_TIMER and HK_TYPE_RCU
housekeeping masks for the time being, though this is subject to change.

Waiman Long (2):
  sched/isolation: Exclude dynamically isolated CPUs from housekeeping
    masks
  cgroup/cpuset: Exclude isolated CPUs from housekeeping CPU masks

 include/linux/sched/isolation.h |   8 +++
 kernel/cgroup/cpuset.c          |  30 +++++++---
 kernel/sched/isolation.c        | 101 +++++++++++++++++++++++++++++++-
 3 files changed, 131 insertions(+), 8 deletions(-)

-- 
2.39.3

[PATCH 1/2] sched/isolation: Exclude dynamically isolated CPUs from housekeeping masks

[Waiman Long]

The housekeeping CPU masks, set up by the "isolcpus" and "nohz_full"
boot command line options, are used at boot time to exclude selected CPUs
from running some kernel background processes to minimize disturbance
to latency sensitive userspace applications. Some of housekeeping CPU
masks are also checked at run time to avoid using those isolated CPUs.

The cpuset subsystem is now able to dynamically create a set of isolated
CPUs to be used in isolated cpuset partitions. The long term goal is
to make the degree of isolation as close as possible to what can be
done statically using those boot command line options.

This patch is a step in that direction by making the housekeeping CPU
mask APIs exclude the dynamically isolated CPUs when they are called
at run time. The housekeeping CPU masks will fall back to the bootup
default when all the dynamically isolated CPUs are released.

A new housekeeping_exlude_isolcpus() function is added which is to be
called by the cpuset subsystem to provide a list of isolated CPUs to
be excluded.

Signed-off-by: Waiman Long <longman@redhat.com>
---
 include/linux/sched/isolation.h |   8 +++
 kernel/sched/isolation.c        | 101 +++++++++++++++++++++++++++++++-
 2 files changed, 108 insertions(+), 1 deletion(-)

diff --git a/include/linux/sched/isolation.h b/include/linux/sched/isolation.h
index 2b461129d1fa..d64fa4e60138 100644
--- a/include/linux/sched/isolation.h
+++ b/include/linux/sched/isolation.h
@@ -27,6 +27,8 @@ extern bool housekeeping_enabled(enum hk_type type);
 extern void housekeeping_affine(struct task_struct *t, enum hk_type type);
 extern bool housekeeping_test_cpu(int cpu, enum hk_type type);
 extern void __init housekeeping_init(void);
+extern int housekeeping_exlude_isolcpus(const struct cpumask *isolcpus,
+					unsigned long flags);
 
 #else
 
@@ -54,6 +56,12 @@ static inline bool housekeeping_test_cpu(int cpu, enum hk_type type)
 }
 
 static inline void housekeeping_init(void) { }
+
+static inline int housekeeping_exlude_isolcpus(struct cpumask *isolcpus,
+					       unsigned long flags)
+{
+	return -EOPNOTSUPP;
+}
 #endif /* CONFIG_CPU_ISOLATION */
 
 static inline bool housekeeping_cpu(int cpu, enum hk_type type)
diff --git a/kernel/sched/isolation.c b/kernel/sched/isolation.c
index 373d42c707bc..f47ff04ac77c 100644
--- a/kernel/sched/isolation.c
+++ b/kernel/sched/isolation.c
@@ -28,7 +28,8 @@ struct housekeeping {
 	unsigned long flags;
 };
 
-static struct housekeeping housekeeping;
+static struct housekeeping housekeeping __read_mostly;
+static struct housekeeping housekeeping_boot __read_mostly;
 
 bool housekeeping_enabled(enum hk_type type)
 {
@@ -239,3 +240,101 @@ static int __init housekeeping_isolcpus_setup(char *str)
 	return housekeeping_setup(str, flags);
 }
 __setup("isolcpus=", housekeeping_isolcpus_setup);
+
+/*
+ * Save init housekeeping masks to housekeeping_boot
+ */
+static int housekeeping_copy2_boot(void)
+{
+	enum hk_type type;
+
+	housekeeping_boot.flags = housekeeping.flags;
+	for_each_set_bit(type, &housekeeping.flags, HK_TYPE_MAX) {
+		if (!alloc_cpumask_var(&housekeeping_boot.cpumasks[type],
+				       GFP_KERNEL))
+			return -ENOMEM;
+		cpumask_copy(housekeeping_boot.cpumasks[type],
+			     housekeeping.cpumasks[type]);
+	}
+	return 0;
+}
+
+/*
+ * Exclude the given dynamically isolated CPUs from the housekeeping CPUs
+ * External synchronization is required to make sure that concurrent call to
+ * this function will not happen.
+ *
+ * Return: 0 if successful, an error code if not
+ */
+int housekeeping_exlude_isolcpus(const struct cpumask *isolcpus, unsigned long flags)
+{
+	static unsigned long alloc_flags;
+	static cpumask_var_t tmp_mask;
+	static bool excluded;	/* @true if some CPUs have been excluded */
+	static bool inited;	/* @true if called before */
+
+	bool mask_empty = !isolcpus || cpumask_empty(isolcpus);
+	enum hk_type type;
+
+	lockdep_assert_cpus_held();
+
+	if (mask_empty && (!inited || !excluded))
+		return 0;
+
+	if (unlikely(!inited)) {
+		if (!alloc_cpumask_var(&tmp_mask, GFP_KERNEL))
+			return -ENOMEM;
+		if (housekeeping.flags) {
+			int err = housekeeping_copy2_boot();
+
+			if (err)
+				return err;
+		}
+		alloc_flags = housekeeping.flags;
+		inited = true;
+	}
+
+	if (mask_empty) {
+		excluded = false;
+
+		/*
+		 * Reset housekeeping to bootup default
+		 */
+		for_each_set_bit(type, &housekeeping_boot.flags, HK_TYPE_MAX)
+			cpumask_copy(housekeeping.cpumasks[type],
+				     housekeeping_boot.cpumasks[type]);
+
+		WRITE_ONCE(housekeeping.flags, housekeeping_boot.flags);
+		if (!housekeeping_boot.flags &&
+		    static_branch_unlikely(&housekeeping_overridden))
+			static_key_disable_cpuslocked(&housekeeping_overridden.key);
+		return 0;
+	}
+
+	/*
+	 * Setting up the new housekeeping cpumasks
+	 */
+	for_each_set_bit(type, &flags, HK_TYPE_MAX) {
+		const struct cpumask *src_mask;
+
+		if (!(BIT(type) & alloc_flags)) {
+			if (!alloc_cpumask_var(&housekeeping.cpumasks[type], GFP_KERNEL))
+				return -ENOMEM;
+			alloc_flags |= BIT(type);
+		}
+		src_mask = (BIT(type) & housekeeping_boot.flags)
+			 ? housekeeping_boot.cpumasks[type] : cpu_possible_mask;
+		/*
+		 * Make sure there is at least one online housekeeping CPU
+		 */
+		cpumask_andnot(tmp_mask, src_mask, isolcpus);
+		if (!cpumask_intersects(tmp_mask, cpu_online_mask))
+			return -EINVAL;	/* Invalid isolated CPUs */
+		cpumask_copy(housekeeping.cpumasks[type], tmp_mask);
+	}
+	WRITE_ONCE(housekeeping.flags, housekeeping_boot.flags | flags);
+	excluded = true;
+	if (!static_branch_unlikely(&housekeeping_overridden))
+		static_key_enable_cpuslocked(&housekeeping_overridden.key);
+	return 0;
+}
-- 
2.39.3

[PATCH 2/2] cgroup/cpuset: Exclude isolated CPUs from housekeeping CPU masks

[Waiman Long]

Call the newly introduced housekeeping_exlude_isolcpus() function to
exclude isolated CPUs from the selected housekeeping CPU masks. This
is in addition to the exclusion of isolated CPUs from the workqueue
unbound CPU mask.

Right now only HK_TYPE_TIMER and HK_TYPE_RCU CPU masks are updated,
but more may be added in the future when appropriate.

Signed-off-by: Waiman Long <longman@redhat.com>
---
 kernel/cgroup/cpuset.c | 30 +++++++++++++++++++++++-------
 1 file changed, 23 insertions(+), 7 deletions(-)

diff --git a/kernel/cgroup/cpuset.c b/kernel/cgroup/cpuset.c
index ba36c073304a..d2541af71c22 100644
--- a/kernel/cgroup/cpuset.c
+++ b/kernel/cgroup/cpuset.c
@@ -214,6 +214,11 @@ static cpumask_var_t	isolated_cpus;
 /* List of remote partition root children */
 static struct list_head remote_children;
 
+/*
+ * The set of housekeeping flags to be updated for CPU isolation
+ */
+#define	HOUSEKEEPING_FLAGS	(BIT(HK_TYPE_TIMER) | BIT(HK_TYPE_RCU))
+
 /*
  * Partition root states:
  *
@@ -1505,7 +1510,15 @@ static bool partition_xcpus_del(int old_prs, struct cpuset *parent,
 	return isolcpus_updated;
 }
 
-static void update_unbound_workqueue_cpumask(bool isolcpus_updated)
+/**
+ * update_isolation_cpumasks - Update external isolation CPU masks
+ * @isolcpus_updated - @true if isolation CPU masks update needed
+ *
+ * The following external CPU masks will be updated if necessary:
+ * - workqueue unbound cpumask
+ * - housekeeping cpumasks
+ */
+static void update_isolation_cpumasks(bool isolcpus_updated)
 {
 	int ret;
 
@@ -1515,7 +1528,10 @@ static void update_unbound_workqueue_cpumask(bool isolcpus_updated)
 		return;
 
 	ret = workqueue_unbound_exclude_cpumask(isolated_cpus);
-	WARN_ON_ONCE(ret < 0);
+	if (WARN_ON_ONCE(ret < 0))
+		return;
+	ret = housekeeping_exlude_isolcpus(isolated_cpus, HOUSEKEEPING_FLAGS);
+	WARN_ON_ONCE((ret < 0) && (ret != -EOPNOTSUPP));
 }
 
 /**
@@ -1609,7 +1625,7 @@ static int remote_partition_enable(struct cpuset *cs, int new_prs,
 		parent->child_ecpus_count--;
 	}
 	spin_unlock_irq(&callback_lock);
-	update_unbound_workqueue_cpumask(isolcpus_updated);
+	update_isolation_cpumasks(isolcpus_updated);
 
 	/*
 	 * Proprogate changes in top_cpuset's effective_cpus down the hierarchy.
@@ -1645,7 +1661,7 @@ static void remote_partition_disable(struct cpuset *cs, struct tmpmasks *tmp)
 		cs->prs_err = PERR_INVCPUS;
 	reset_partition_data(cs);
 	spin_unlock_irq(&callback_lock);
-	update_unbound_workqueue_cpumask(isolcpus_updated);
+	update_isolation_cpumasks(isolcpus_updated);
 
 	/*
 	 * Proprogate changes in top_cpuset's effective_cpus down the hierarchy.
@@ -1697,7 +1713,7 @@ static void remote_cpus_update(struct cpuset *cs, struct cpumask *newmask,
 	if (deleting)
 		isolcpus_updated += partition_xcpus_del(prs, NULL, tmp->delmask);
 	spin_unlock_irq(&callback_lock);
-	update_unbound_workqueue_cpumask(isolcpus_updated);
+	update_isolation_cpumasks(isolcpus_updated);
 
 	/*
 	 * Proprogate changes in top_cpuset's effective_cpus down the hierarchy.
@@ -2067,7 +2083,7 @@ static int update_parent_effective_cpumask(struct cpuset *cs, int cmd,
 		WARN_ON_ONCE(parent->nr_subparts < 0);
 	}
 	spin_unlock_irq(&callback_lock);
-	update_unbound_workqueue_cpumask(isolcpus_updated);
+	update_isolation_cpumasks(isolcpus_updated);
 
 	if ((old_prs != new_prs) && (cmd == partcmd_update))
 		update_partition_exclusive(cs, new_prs);
@@ -3131,7 +3147,7 @@ static int update_prstate(struct cpuset *cs, int new_prs)
 	else if (new_xcpus_state)
 		partition_xcpus_newstate(old_prs, new_prs, cs->effective_xcpus);
 	spin_unlock_irq(&callback_lock);
-	update_unbound_workqueue_cpumask(new_xcpus_state);
+	update_isolation_cpumasks(new_xcpus_state);
 
 	/* Force update if switching back to member */
 	update_cpumasks_hier(cs, &tmpmask, !new_prs ? HIER_CHECKALL : 0);
-- 
2.39.3

Re: [PATCH 2/2] cgroup/cpuset: Exclude isolated CPUs from housekeeping CPU masks

[Tejun Heo]

On Wed, Feb 28, 2024 at 09:14:14PM -0500, Waiman Long wrote:
> Call the newly introduced housekeeping_exlude_isolcpus() function to
> exclude isolated CPUs from the selected housekeeping CPU masks. This
> is in addition to the exclusion of isolated CPUs from the workqueue
> unbound CPU mask.
> 
> Right now only HK_TYPE_TIMER and HK_TYPE_RCU CPU masks are updated,
> but more may be added in the future when appropriate.
> 
> Signed-off-by: Waiman Long <longman@redhat.com>

This looks fine to me from cgroup POV. Please feel free to route the patch
any way you see fit.

  Acked-by: Tejun Heo <tj@kernel.org>

Thanks.

-- 
tejun

Re: [PATCH 1/2] sched/isolation: Exclude dynamically isolated CPUs from housekeeping masks

[Frederic Weisbecker]

Le Wed, Feb 28, 2024 at 09:14:13PM -0500, Waiman Long a écrit :
> The housekeeping CPU masks, set up by the "isolcpus" and "nohz_full"
> boot command line options, are used at boot time to exclude selected CPUs
> from running some kernel background processes to minimize disturbance
> to latency sensitive userspace applications. Some of housekeeping CPU
> masks are also checked at run time to avoid using those isolated CPUs.
> 
> The cpuset subsystem is now able to dynamically create a set of isolated
> CPUs to be used in isolated cpuset partitions. The long term goal is
> to make the degree of isolation as close as possible to what can be
> done statically using those boot command line options.
> 
> This patch is a step in that direction by making the housekeeping CPU
> mask APIs exclude the dynamically isolated CPUs when they are called
> at run time. The housekeeping CPU masks will fall back to the bootup
> default when all the dynamically isolated CPUs are released.
> 
> A new housekeeping_exlude_isolcpus() function is added which is to be
> called by the cpuset subsystem to provide a list of isolated CPUs to
> be excluded.

Can we instead overwrite housekeeping_boot.cpumasks from cpusets entirely and
forget about the original boot value forever?

Thanks.

Re: [PATCH 1/2] sched/isolation: Exclude dynamically isolated CPUs from housekeeping masks

[Waiman Long]

On 3/25/24 19:21, Frederic Weisbecker wrote:
> Le Wed, Feb 28, 2024 at 09:14:13PM -0500, Waiman Long a écrit :
>> The housekeeping CPU masks, set up by the "isolcpus" and "nohz_full"
>> boot command line options, are used at boot time to exclude selected CPUs
>> from running some kernel background processes to minimize disturbance
>> to latency sensitive userspace applications. Some of housekeeping CPU
>> masks are also checked at run time to avoid using those isolated CPUs.
>>
>> The cpuset subsystem is now able to dynamically create a set of isolated
>> CPUs to be used in isolated cpuset partitions. The long term goal is
>> to make the degree of isolation as close as possible to what can be
>> done statically using those boot command line options.
>>
>> This patch is a step in that direction by making the housekeeping CPU
>> mask APIs exclude the dynamically isolated CPUs when they are called
>> at run time. The housekeeping CPU masks will fall back to the bootup
>> default when all the dynamically isolated CPUs are released.
>>
>> A new housekeeping_exlude_isolcpus() function is added which is to be
>> called by the cpuset subsystem to provide a list of isolated CPUs to
>> be excluded.
> Can we instead overwrite housekeeping_boot.cpumasks from cpusets entirely and
> forget about the original boot value forever?

That is an interesting question.

I think we will still need to keep the boot values somewhere, at least 
until we can dynamically isolate CPUs as good as what can be done at 
boot time. At this point, what can be done dynamically is still limited. 
So we want to keep the statically isolated CPUs around without breaking 
its isolation. Of course, we can do all these within cpuset with minimal 
change in kernel/sched/isolation.c.

Cheers,
Longman