[PATCH 0/5] handle kthread_prio kernel cmdline parameter consistently

[PATCH 0/5] handle kthread_prio kernel cmdline parameter consistently

[Alison Chaiken]

When CONFIG_PREEMPT_RT=y, RCU_SOFTIRQ work is moved to dedicated rcuc
per-core threads.  The rcutree.kthread_prio kernel cmdline parameter
controls the priority of those threads.

A provided kthread_prio kernel cmdline parameter also elevates the
priority of rcuog threads, but only if they are launched from
rcu_spawn_gp_kthread(), not if they are launched from
rcu_spawn_one_nocb_kthread().  Fix this inconsistency.

When CONFIG_RCU_NOCB_CPU=y, an rcu_nocbs command-line parameter
offloads the work of rcuc on designated cores to new rcuop threads
that are not pinned to the cores whose expired grace-period timer
callbacks they run.  While rcuop threads have the same function as
rcuc threads, their priority is not controlled by the kthread_prio
parameter.  Add this feature and update the documentation accordingly.

Alison Chaiken (4):
  RCU: move kthread_prio bounds-check to a separate function
  RCU: make priority of grace-period thread consistent
  RCU: elevate priority of offloaded callback threads
  RCU: update documentation regarding kthread_prio cmdline parameter

 .../admin-guide/kernel-parameters.txt         |  2 +
 kernel/rcu/tree.c                             | 42 +++++++++++--------
 kernel/rcu/tree_nocb.h                        |  7 ++++
 3 files changed, 34 insertions(+), 17 deletions(-)

-- 
2.32.0

[PATCH 1/4] RCU: move kthread_prio bounds-check to a separate function

[Alison Chaiken]

Move the bounds-check of the kthread_prio cmdline parameter to a new
function in order to faciliate a different callsite.

Signed-off-by: Alison Chaiken <achaiken@aurora.tech>
---
 kernel/rcu/tree.c | 39 +++++++++++++++++++++++----------------
 1 file changed, 23 insertions(+), 16 deletions(-)

diff --git a/kernel/rcu/tree.c b/kernel/rcu/tree.c
index 9a04550cc54b..15cabb288a1c 100644
--- a/kernel/rcu/tree.c
+++ b/kernel/rcu/tree.c
@@ -4404,26 +4404,10 @@ static int rcu_pm_notify(struct notifier_block *self,
 static int __init rcu_spawn_gp_kthread(void)
 {
 	unsigned long flags;
-	int kthread_prio_in = kthread_prio;
 	struct rcu_node *rnp;
 	struct sched_param sp;
 	struct task_struct *t;
 
-	/* Force priority into range. */
-	if (IS_ENABLED(CONFIG_RCU_BOOST) && kthread_prio < 2
-	    && IS_BUILTIN(CONFIG_RCU_TORTURE_TEST))
-		kthread_prio = 2;
-	else if (IS_ENABLED(CONFIG_RCU_BOOST) && kthread_prio < 1)
-		kthread_prio = 1;
-	else if (kthread_prio < 0)
-		kthread_prio = 0;
-	else if (kthread_prio > 99)
-		kthread_prio = 99;
-
-	if (kthread_prio != kthread_prio_in)
-		pr_alert("rcu_spawn_gp_kthread(): Limited prio to %d from %d\n",
-			 kthread_prio, kthread_prio_in);
-
 	rcu_scheduler_fully_active = 1;
 	t = kthread_create(rcu_gp_kthread, NULL, "%s", rcu_state.name);
 	if (WARN_ONCE(IS_ERR(t), "%s: Could not start grace-period kthread, OOM is now expected behavior\n", __func__))
@@ -4548,6 +4532,28 @@ static void __init rcu_init_one(void)
 	}
 }
 
+/*
+ * Force priority from the kernel command-line into range.
+ */
+static void __init sanitize_kthread_prio(void)
+{
+	int kthread_prio_in = kthread_prio;
+
+	if (IS_ENABLED(CONFIG_RCU_BOOST) && kthread_prio < 2
+	    && IS_BUILTIN(CONFIG_RCU_TORTURE_TEST))
+		kthread_prio = 2;
+	else if (IS_ENABLED(CONFIG_RCU_BOOST) && kthread_prio < 1)
+		kthread_prio = 1;
+	else if (kthread_prio < 0)
+		kthread_prio = 0;
+	else if (kthread_prio > 99)
+		kthread_prio = 99;
+
+	if (kthread_prio != kthread_prio_in)
+		pr_alert("%s: Limited prio to %d from %d\n",
+			 __func__, kthread_prio, kthread_prio_in);
+}
+
 /*
  * Compute the rcu_node tree geometry from kernel parameters.  This cannot
  * replace the definitions in tree.h because those are needed to size
@@ -4708,6 +4714,7 @@ void __init rcu_init(void)
 
 	kfree_rcu_batch_init();
 	rcu_bootup_announce();
+	sanitize_kthread_prio();
 	rcu_init_geometry();
 	rcu_init_one();
 	if (dump_tree)
-- 
2.32.0

[PATCH 2/4] RCU: make priority of grace-period thread consistent

[Alison Chaiken]

The priority of RCU grace period threads is set to kthread_prio when
they are launched from rcu_spawn_gp_kthread().  The same is not true
of rcu_spawn_one_nocb_kthread().  Accordingly, add priority elevation
to rcu_spawn_one_nocb_kthread().

Signed-off-by: Alison Chaiken <achaiken@aurora.tech>
---
 kernel/rcu/tree_nocb.h | 4 ++++
 1 file changed, 4 insertions(+)

diff --git a/kernel/rcu/tree_nocb.h b/kernel/rcu/tree_nocb.h
index 8fdf44f8523f..08b5c0042979 100644
--- a/kernel/rcu/tree_nocb.h
+++ b/kernel/rcu/tree_nocb.h
@@ -1183,7 +1183,9 @@ static void rcu_spawn_one_nocb_kthread(int cpu)
 	struct rcu_data *rdp = per_cpu_ptr(&rcu_data, cpu);
 	struct rcu_data *rdp_gp;
 	struct task_struct *t;
+	struct sched_param sp;
 
+	sp.sched_priority = kthread_prio;
 	/*
 	 * If this isn't a no-CBs CPU or if it already has an rcuo kthread,
 	 * then nothing to do.
@@ -1199,6 +1201,8 @@ static void rcu_spawn_one_nocb_kthread(int cpu)
 		if (WARN_ONCE(IS_ERR(t), "%s: Could not start rcuo GP kthread, OOM is now expected behavior\n", __func__))
 			return;
 		WRITE_ONCE(rdp_gp->nocb_gp_kthread, t);
+		if (kthread_prio)
+			sched_setscheduler_nocheck(t, SCHED_FIFO, &sp);
 	}
 
 	/* Spawn the kthread for this CPU. */
-- 
2.32.0

[PATCH 3/4] RCU: elevate priority of offloaded callback threads

[Alison Chaiken]

When CONFIG_PREEMPT_RT=y, the rcutree.kthread_prio command-line
parameter signals initialization code to boost the priority of rcuc
callbacks to the designated value.  With the additional
CONFIG_RCU_NOCB_CPU=y configuration and an additional rcu_nocbs
command-line parameter, the callbacks on the listed cores are
offloaded to new rcuop threads that are not pinned to the cores whose
post-grace-period work is performed.  While the rcuop threads perform
the same function as the rcuc threads they offload, the kthread_prio
parameter only boosts the priority of the rcuc threads.  Fix this
inconsistency by elevating rcuop threads to the same priority as rcuc
ones.

Signed-off-by: Alison Chaiken <achaiken@aurora.tech>
---
 kernel/rcu/tree.c      | 2 +-
 kernel/rcu/tree_nocb.h | 3 +++
 2 files changed, 4 insertions(+), 1 deletion(-)

diff --git a/kernel/rcu/tree.c b/kernel/rcu/tree.c
index 15cabb288a1c..ba9eaa9a72b0 100644
--- a/kernel/rcu/tree.c
+++ b/kernel/rcu/tree.c
@@ -153,7 +153,7 @@ static void sync_sched_exp_online_cleanup(int cpu);
 static void check_cb_ovld_locked(struct rcu_data *rdp, struct rcu_node *rnp);
 static bool rcu_rdp_is_offloaded(struct rcu_data *rdp);
 
-/* rcuc/rcub kthread realtime priority */
+/* rcuc/rcub/rcuop kthread realtime priority */
 static int kthread_prio = IS_ENABLED(CONFIG_RCU_BOOST) ? 1 : 0;
 module_param(kthread_prio, int, 0444);
 
diff --git a/kernel/rcu/tree_nocb.h b/kernel/rcu/tree_nocb.h
index 08b5c0042979..3343e42685dc 100644
--- a/kernel/rcu/tree_nocb.h
+++ b/kernel/rcu/tree_nocb.h
@@ -1210,6 +1210,9 @@ static void rcu_spawn_one_nocb_kthread(int cpu)
 			"rcuo%c/%d", rcu_state.abbr, cpu);
 	if (WARN_ONCE(IS_ERR(t), "%s: Could not start rcuo CB kthread, OOM is now expected behavior\n", __func__))
 		return;
+
+	if (kthread_prio)
+		sched_setscheduler_nocheck(t, SCHED_FIFO, &sp);
 	WRITE_ONCE(rdp->nocb_cb_kthread, t);
 	WRITE_ONCE(rdp->nocb_gp_kthread, rdp_gp->nocb_gp_kthread);
 }
-- 
2.32.0

[PATCH 4/4] RCU: update documentation regarding kthread_prio cmdline parameter

[Alison Chaiken]

Inform readers that the priority of RCU no-callback threads will also
be boosted.

Signed-off-by: Alison Chaiken <achaiken@aurora.tech>
---
 Documentation/admin-guide/kernel-parameters.txt | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/Documentation/admin-guide/kernel-parameters.txt b/Documentation/admin-guide/kernel-parameters.txt
index 1396fd2d9031..b844978e1bdf 100644
--- a/Documentation/admin-guide/kernel-parameters.txt
+++ b/Documentation/admin-guide/kernel-parameters.txt
@@ -4439,6 +4439,8 @@
 			(the least-favored priority).  Otherwise, when
 			RCU_BOOST is not set, valid values are 0-99 and
 			the default is zero (non-realtime operation).
+			When RCU_NOCB_CPU is set, also adjust the
+			priority of NOCB callback kthreads.
 
 	rcutree.rcu_nocb_gp_stride= [KNL]
 			Set the number of NOCB callback kthreads in
-- 
2.32.0

Re: [PATCH 0/5] handle kthread_prio kernel cmdline parameter consistently

[Paul E. McKenney]

On Tue, Jan 11, 2022 at 03:32:49PM -0800, Alison Chaiken wrote:
> When CONFIG_PREEMPT_RT=y, RCU_SOFTIRQ work is moved to dedicated rcuc
> per-core threads.  The rcutree.kthread_prio kernel cmdline parameter
> controls the priority of those threads.
> 
> A provided kthread_prio kernel cmdline parameter also elevates the
> priority of rcuog threads, but only if they are launched from
> rcu_spawn_gp_kthread(), not if they are launched from
> rcu_spawn_one_nocb_kthread().  Fix this inconsistency.
> 
> When CONFIG_RCU_NOCB_CPU=y, an rcu_nocbs command-line parameter
> offloads the work of rcuc on designated cores to new rcuop threads
> that are not pinned to the cores whose expired grace-period timer
> callbacks they run.  While rcuop threads have the same function as
> rcuc threads, their priority is not controlled by the kthread_prio
> parameter.  Add this feature and update the documentation accordingly.
> 
> Alison Chaiken (4):
>   RCU: move kthread_prio bounds-check to a separate function
>   RCU: make priority of grace-period thread consistent
>   RCU: elevate priority of offloaded callback threads
>   RCU: update documentation regarding kthread_prio cmdline parameter

I have queued these for review and testing, thank you!

2/4 appears to have been produced against a pre-merge-window mainline,
so I had to adjust it a bit to queue it on -rcu.  Please check to make
sure that I did not mess anything up.

							Thanx, Paul