* [PATCH rcu 0/9] Miscellaneous fixes for v5.18
@ 2022-02-04 23:07 Paul E. McKenney
2022-02-04 23:07 ` [PATCH rcu 1/9] MAINTAINERS: Add Frederic and Neeraj to their RCU files Paul E. McKenney
` (8 more replies)
0 siblings, 9 replies; 17+ messages in thread
From: Paul E. McKenney @ 2022-02-04 23:07 UTC (permalink / raw)
To: rcu; +Cc: linux-kernel, kernel-team, rostedt
Hello!
This series provides miscellaneous fixes:
1. MAINTAINERS: Add Frederic and Neeraj to their RCU files.
2. Fix description of kvfree_rcu(), courtesy of "Uladzislau Rezki
(Sony)".
3. Add mutex for rcu boost kthread spawning and affinity setting,
courtesy of David Woodhouse.
4. Inline __call_rcu() into call_rcu().
5. Record work creation stack trace with interrupts enabled,
courtesy of Zqiang.
6. Mark writes to the rcu_segcblist structure's ->flags field.
7. Uninline multi-use function: finish_rcuwait(), courtesy of
Ingo Molnar.
8. Remove __read_mostly annotations from rcu_scheduler_active
externs, courtesy of Ingo Molnar.
9. Replace cpumask_weight with cpumask_empty where appropriate,
courtesy of Yury Norov.
Thanx, Paul
------------------------------------------------------------------------
b/MAINTAINERS | 2
b/include/linux/rcupdate.h | 2
b/include/linux/rcutree.h | 2
b/include/linux/rcuwait.h | 6 --
b/kernel/rcu/rcu_segcblist.h | 4 -
b/kernel/rcu/tree.c | 1
b/kernel/rcu/tree.h | 3 +
b/kernel/rcu/tree_nocb.h | 4 -
b/kernel/rcu/tree_plugin.h | 10 +++-
b/kernel/rcu/update.c | 7 +++
include/linux/rcupdate.h | 2
kernel/rcu/tree.c | 93 +++++++++++++++++++------------------------
kernel/rcu/tree_plugin.h | 2
13 files changed, 73 insertions(+), 65 deletions(-)
^ permalink raw reply [flat|nested] 17+ messages in thread
* [PATCH rcu 1/9] MAINTAINERS: Add Frederic and Neeraj to their RCU files
2022-02-04 23:07 [PATCH rcu 0/9] Miscellaneous fixes for v5.18 Paul E. McKenney
@ 2022-02-04 23:07 ` Paul E. McKenney
2022-02-11 15:15 ` Frederic Weisbecker
2022-02-04 23:07 ` [PATCH rcu 2/9] rcu: Fix description of kvfree_rcu() Paul E. McKenney
` (7 subsequent siblings)
8 siblings, 1 reply; 17+ messages in thread
From: Paul E. McKenney @ 2022-02-04 23:07 UTC (permalink / raw)
To: rcu
Cc: linux-kernel, kernel-team, rostedt, Paul E. McKenney,
Neeraj Upadhyay, Frederic Weisbecker
Adding Frederic as an RCU maintainer for kernel/rcu/tree_nocb.h given his
work with offloading and de-offloading callbacks from CPUs. Also adding
Neeraj for kernel/rcu/tasks.h given his focused work on RCU Tasks Trace.
As in I am reasonably certain that each understands the full contents
of the corresponding file.
Cc: Neeraj Upadhyay <quic_neeraju@quicinc.com>
Cc: Frederic Weisbecker <frederic@kernel.org>
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
---
MAINTAINERS | 2 ++
1 file changed, 2 insertions(+)
diff --git a/MAINTAINERS b/MAINTAINERS
index ea3e6c9143848..5ad49de532b1a 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -16298,6 +16298,8 @@ F: tools/testing/selftests/resctrl/
READ-COPY UPDATE (RCU)
M: "Paul E. McKenney" <paulmck@kernel.org>
+M: Frederic Weisbecker <frederic@kernel.org> (kernel/rcu/tree_nocb.h)
+M: Neeraj Upadhyay <quic_neeraju@quicinc.com> (kernel/rcu/tasks.h)
M: Josh Triplett <josh@joshtriplett.org>
R: Steven Rostedt <rostedt@goodmis.org>
R: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
--
2.31.1.189.g2e36527f23
^ permalink raw reply related [flat|nested] 17+ messages in thread
* [PATCH rcu 2/9] rcu: Fix description of kvfree_rcu()
2022-02-04 23:07 [PATCH rcu 0/9] Miscellaneous fixes for v5.18 Paul E. McKenney
2022-02-04 23:07 ` [PATCH rcu 1/9] MAINTAINERS: Add Frederic and Neeraj to their RCU files Paul E. McKenney
@ 2022-02-04 23:07 ` Paul E. McKenney
2022-02-04 23:07 ` [PATCH rcu 3/9] rcu: Add mutex for rcu boost kthread spawning and affinity setting Paul E. McKenney
` (6 subsequent siblings)
8 siblings, 0 replies; 17+ messages in thread
From: Paul E. McKenney @ 2022-02-04 23:07 UTC (permalink / raw)
To: rcu
Cc: linux-kernel, kernel-team, rostedt, Uladzislau Rezki (Sony),
Paul E . McKenney
From: "Uladzislau Rezki (Sony)" <urezki@gmail.com>
The kvfree_rcu() header comment's description of the "ptr" parameter
is unclear, therefore rephrase it to make it clear that it is a pointer
to the memory to eventually be passed to kvfree().
Reported-by: Steven Rostedt <rostedt@goodmis.org>
Signed-off-by: Uladzislau Rezki (Sony) <urezki@gmail.com>
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
---
include/linux/rcupdate.h | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/include/linux/rcupdate.h b/include/linux/rcupdate.h
index 88b42eb464068..9d7df8d36af07 100644
--- a/include/linux/rcupdate.h
+++ b/include/linux/rcupdate.h
@@ -924,7 +924,7 @@ static inline notrace void rcu_read_unlock_sched_notrace(void)
*
* kvfree_rcu(ptr);
*
- * where @ptr is a pointer to kvfree().
+ * where @ptr is the pointer to be freed by kvfree().
*
* Please note, head-less way of freeing is permitted to
* use from a context that has to follow might_sleep()
--
2.31.1.189.g2e36527f23
^ permalink raw reply related [flat|nested] 17+ messages in thread
* [PATCH rcu 3/9] rcu: Add mutex for rcu boost kthread spawning and affinity setting
2022-02-04 23:07 [PATCH rcu 0/9] Miscellaneous fixes for v5.18 Paul E. McKenney
2022-02-04 23:07 ` [PATCH rcu 1/9] MAINTAINERS: Add Frederic and Neeraj to their RCU files Paul E. McKenney
2022-02-04 23:07 ` [PATCH rcu 2/9] rcu: Fix description of kvfree_rcu() Paul E. McKenney
@ 2022-02-04 23:07 ` Paul E. McKenney
2022-02-11 14:57 ` Frederic Weisbecker
2022-02-04 23:08 ` [PATCH rcu 4/9] rcu: Inline __call_rcu() into call_rcu() Paul E. McKenney
` (5 subsequent siblings)
8 siblings, 1 reply; 17+ messages in thread
From: Paul E. McKenney @ 2022-02-04 23:07 UTC (permalink / raw)
To: rcu
Cc: linux-kernel, kernel-team, rostedt, David Woodhouse, Paul E . McKenney
From: David Woodhouse <dwmw@amazon.co.uk>
As we handle parallel CPU bringup, we will need to take care to avoid
spawning multiple boost threads, or race conditions when setting their
affinity. Spotted by Paul McKenney.
Signed-off-by: David Woodhouse <dwmw@amazon.co.uk>
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
---
kernel/rcu/tree.c | 1 +
kernel/rcu/tree.h | 3 +++
kernel/rcu/tree_plugin.h | 10 ++++++++--
3 files changed, 12 insertions(+), 2 deletions(-)
diff --git a/kernel/rcu/tree.c b/kernel/rcu/tree.c
index a4c25a6283b0b..d1d1a8c51223b 100644
--- a/kernel/rcu/tree.c
+++ b/kernel/rcu/tree.c
@@ -4570,6 +4570,7 @@ static void __init rcu_init_one(void)
init_waitqueue_head(&rnp->exp_wq[2]);
init_waitqueue_head(&rnp->exp_wq[3]);
spin_lock_init(&rnp->exp_lock);
+ mutex_init(&rnp->boost_kthread_mutex);
}
}
diff --git a/kernel/rcu/tree.h b/kernel/rcu/tree.h
index 486fc901bd085..3b8b60de07c38 100644
--- a/kernel/rcu/tree.h
+++ b/kernel/rcu/tree.h
@@ -110,6 +110,9 @@ struct rcu_node {
/* side effect, not as a lock. */
unsigned long boost_time;
/* When to start boosting (jiffies). */
+ struct mutex boost_kthread_mutex;
+ /* Exclusion for thread spawning and affinity */
+ /* manipulation. */
struct task_struct *boost_kthread_task;
/* kthread that takes care of priority */
/* boosting for this rcu_node structure. */
diff --git a/kernel/rcu/tree_plugin.h b/kernel/rcu/tree_plugin.h
index c5b45c2f68a15..07845dcd33c5e 100644
--- a/kernel/rcu/tree_plugin.h
+++ b/kernel/rcu/tree_plugin.h
@@ -1172,15 +1172,16 @@ static void rcu_spawn_one_boost_kthread(struct rcu_node *rnp)
struct sched_param sp;
struct task_struct *t;
+ mutex_lock(&rnp->boost_kthread_mutex);
if (rnp->boost_kthread_task || !rcu_scheduler_fully_active)
- return;
+ goto out;
rcu_state.boost = 1;
t = kthread_create(rcu_boost_kthread, (void *)rnp,
"rcub/%d", rnp_index);
if (WARN_ON_ONCE(IS_ERR(t)))
- return;
+ goto out;
raw_spin_lock_irqsave_rcu_node(rnp, flags);
rnp->boost_kthread_task = t;
@@ -1188,6 +1189,9 @@ static void rcu_spawn_one_boost_kthread(struct rcu_node *rnp)
sp.sched_priority = kthread_prio;
sched_setscheduler_nocheck(t, SCHED_FIFO, &sp);
wake_up_process(t); /* get to TASK_INTERRUPTIBLE quickly. */
+
+ out:
+ mutex_unlock(&rnp->boost_kthread_mutex);
}
/*
@@ -1210,6 +1214,7 @@ static void rcu_boost_kthread_setaffinity(struct rcu_node *rnp, int outgoingcpu)
return;
if (!zalloc_cpumask_var(&cm, GFP_KERNEL))
return;
+ mutex_lock(&rnp->boost_kthread_mutex);
for_each_leaf_node_possible_cpu(rnp, cpu)
if ((mask & leaf_node_cpu_bit(rnp, cpu)) &&
cpu != outgoingcpu)
@@ -1218,6 +1223,7 @@ static void rcu_boost_kthread_setaffinity(struct rcu_node *rnp, int outgoingcpu)
if (cpumask_weight(cm) == 0)
cpumask_copy(cm, housekeeping_cpumask(HK_FLAG_RCU));
set_cpus_allowed_ptr(t, cm);
+ mutex_unlock(&rnp->boost_kthread_mutex);
free_cpumask_var(cm);
}
--
2.31.1.189.g2e36527f23
^ permalink raw reply related [flat|nested] 17+ messages in thread
* [PATCH rcu 4/9] rcu: Inline __call_rcu() into call_rcu()
2022-02-04 23:07 [PATCH rcu 0/9] Miscellaneous fixes for v5.18 Paul E. McKenney
` (2 preceding siblings ...)
2022-02-04 23:07 ` [PATCH rcu 3/9] rcu: Add mutex for rcu boost kthread spawning and affinity setting Paul E. McKenney
@ 2022-02-04 23:08 ` Paul E. McKenney
2022-02-04 23:08 ` [PATCH rcu 5/9] kasan: Record work creation stack trace with interrupts enabled Paul E. McKenney
` (4 subsequent siblings)
8 siblings, 0 replies; 17+ messages in thread
From: Paul E. McKenney @ 2022-02-04 23:08 UTC (permalink / raw)
To: rcu; +Cc: linux-kernel, kernel-team, rostedt, Paul E. McKenney
Because __call_rcu() is invoked only by call_rcu(), this commit inlines
the former into the latter.
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
---
kernel/rcu/tree.c | 91 ++++++++++++++++++++++-------------------------
1 file changed, 42 insertions(+), 49 deletions(-)
diff --git a/kernel/rcu/tree.c b/kernel/rcu/tree.c
index d1d1a8c51223b..f1bb7ccc00847 100644
--- a/kernel/rcu/tree.c
+++ b/kernel/rcu/tree.c
@@ -2995,9 +2995,47 @@ static void check_cb_ovld(struct rcu_data *rdp)
raw_spin_unlock_rcu_node(rnp);
}
-/* Helper function for call_rcu() and friends. */
-static void
-__call_rcu(struct rcu_head *head, rcu_callback_t func)
+/**
+ * call_rcu() - Queue an RCU callback for invocation after a grace period.
+ * @head: structure to be used for queueing the RCU updates.
+ * @func: actual callback function to be invoked after the grace period
+ *
+ * The callback function will be invoked some time after a full grace
+ * period elapses, in other words after all pre-existing RCU read-side
+ * critical sections have completed. However, the callback function
+ * might well execute concurrently with RCU read-side critical sections
+ * that started after call_rcu() was invoked.
+ *
+ * RCU read-side critical sections are delimited by rcu_read_lock()
+ * and rcu_read_unlock(), and may be nested. In addition, but only in
+ * v5.0 and later, regions of code across which interrupts, preemption,
+ * or softirqs have been disabled also serve as RCU read-side critical
+ * sections. This includes hardware interrupt handlers, softirq handlers,
+ * and NMI handlers.
+ *
+ * Note that all CPUs must agree that the grace period extended beyond
+ * all pre-existing RCU read-side critical section. On systems with more
+ * than one CPU, this means that when "func()" is invoked, each CPU is
+ * guaranteed to have executed a full memory barrier since the end of its
+ * last RCU read-side critical section whose beginning preceded the call
+ * to call_rcu(). It also means that each CPU executing an RCU read-side
+ * critical section that continues beyond the start of "func()" must have
+ * executed a memory barrier after the call_rcu() but before the beginning
+ * of that RCU read-side critical section. Note that these guarantees
+ * include CPUs that are offline, idle, or executing in user mode, as
+ * well as CPUs that are executing in the kernel.
+ *
+ * Furthermore, if CPU A invoked call_rcu() and CPU B invoked the
+ * resulting RCU callback function "func()", then both CPU A and CPU B are
+ * guaranteed to execute a full memory barrier during the time interval
+ * between the call to call_rcu() and the invocation of "func()" -- even
+ * if CPU A and CPU B are the same CPU (but again only if the system has
+ * more than one CPU).
+ *
+ * Implementation of these memory-ordering guarantees is described here:
+ * Documentation/RCU/Design/Memory-Ordering/Tree-RCU-Memory-Ordering.rst.
+ */
+void call_rcu(struct rcu_head *head, rcu_callback_t func)
{
static atomic_t doublefrees;
unsigned long flags;
@@ -3011,7 +3049,7 @@ __call_rcu(struct rcu_head *head, rcu_callback_t func)
/*
* Probable double call_rcu(), so leak the callback.
* Use rcu:rcu_callback trace event to find the previous
- * time callback was passed to __call_rcu().
+ * time callback was passed to call_rcu().
*/
if (atomic_inc_return(&doublefrees) < 4) {
pr_err("%s(): Double-freed CB %p->%pS()!!! ", __func__, head, head->func);
@@ -3060,51 +3098,6 @@ __call_rcu(struct rcu_head *head, rcu_callback_t func)
local_irq_restore(flags);
}
}
-
-/**
- * call_rcu() - Queue an RCU callback for invocation after a grace period.
- * @head: structure to be used for queueing the RCU updates.
- * @func: actual callback function to be invoked after the grace period
- *
- * The callback function will be invoked some time after a full grace
- * period elapses, in other words after all pre-existing RCU read-side
- * critical sections have completed. However, the callback function
- * might well execute concurrently with RCU read-side critical sections
- * that started after call_rcu() was invoked.
- *
- * RCU read-side critical sections are delimited by rcu_read_lock()
- * and rcu_read_unlock(), and may be nested. In addition, but only in
- * v5.0 and later, regions of code across which interrupts, preemption,
- * or softirqs have been disabled also serve as RCU read-side critical
- * sections. This includes hardware interrupt handlers, softirq handlers,
- * and NMI handlers.
- *
- * Note that all CPUs must agree that the grace period extended beyond
- * all pre-existing RCU read-side critical section. On systems with more
- * than one CPU, this means that when "func()" is invoked, each CPU is
- * guaranteed to have executed a full memory barrier since the end of its
- * last RCU read-side critical section whose beginning preceded the call
- * to call_rcu(). It also means that each CPU executing an RCU read-side
- * critical section that continues beyond the start of "func()" must have
- * executed a memory barrier after the call_rcu() but before the beginning
- * of that RCU read-side critical section. Note that these guarantees
- * include CPUs that are offline, idle, or executing in user mode, as
- * well as CPUs that are executing in the kernel.
- *
- * Furthermore, if CPU A invoked call_rcu() and CPU B invoked the
- * resulting RCU callback function "func()", then both CPU A and CPU B are
- * guaranteed to execute a full memory barrier during the time interval
- * between the call to call_rcu() and the invocation of "func()" -- even
- * if CPU A and CPU B are the same CPU (but again only if the system has
- * more than one CPU).
- *
- * Implementation of these memory-ordering guarantees is described here:
- * Documentation/RCU/Design/Memory-Ordering/Tree-RCU-Memory-Ordering.rst.
- */
-void call_rcu(struct rcu_head *head, rcu_callback_t func)
-{
- __call_rcu(head, func);
-}
EXPORT_SYMBOL_GPL(call_rcu);
--
2.31.1.189.g2e36527f23
^ permalink raw reply related [flat|nested] 17+ messages in thread
* [PATCH rcu 5/9] kasan: Record work creation stack trace with interrupts enabled
2022-02-04 23:07 [PATCH rcu 0/9] Miscellaneous fixes for v5.18 Paul E. McKenney
` (3 preceding siblings ...)
2022-02-04 23:08 ` [PATCH rcu 4/9] rcu: Inline __call_rcu() into call_rcu() Paul E. McKenney
@ 2022-02-04 23:08 ` Paul E. McKenney
2022-02-04 23:08 ` [PATCH rcu 6/9] rcu: Mark writes to the rcu_segcblist structure's ->flags field Paul E. McKenney
` (3 subsequent siblings)
8 siblings, 0 replies; 17+ messages in thread
From: Paul E. McKenney @ 2022-02-04 23:08 UTC (permalink / raw)
To: rcu
Cc: linux-kernel, kernel-team, rostedt, Zqiang, Marco Elver,
Paul E . McKenney
From: Zqiang <qiang1.zhang@intel.com>
Recording the work creation stack trace for KASAN reports in
call_rcu() is expensive, due to unwinding the stack, but also
due to acquiring depot_lock inside stackdepot (which may be contended).
Because calling kasan_record_aux_stack_noalloc() does not require
interrupts to already be disabled, this may unnecessarily extend
the time with interrupts disabled.
Therefore, move calling kasan_record_aux_stack() before the section
with interrupts disabled.
Acked-by: Marco Elver <elver@google.com>
Signed-off-by: Zqiang <qiang1.zhang@intel.com>
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
---
kernel/rcu/tree.c | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/kernel/rcu/tree.c b/kernel/rcu/tree.c
index f1bb7ccc00847..ca8d7dd026eeb 100644
--- a/kernel/rcu/tree.c
+++ b/kernel/rcu/tree.c
@@ -3060,8 +3060,8 @@ void call_rcu(struct rcu_head *head, rcu_callback_t func)
}
head->func = func;
head->next = NULL;
- local_irq_save(flags);
kasan_record_aux_stack_noalloc(head);
+ local_irq_save(flags);
rdp = this_cpu_ptr(&rcu_data);
/* Add the callback to our list. */
--
2.31.1.189.g2e36527f23
^ permalink raw reply related [flat|nested] 17+ messages in thread
* [PATCH rcu 6/9] rcu: Mark writes to the rcu_segcblist structure's ->flags field
2022-02-04 23:07 [PATCH rcu 0/9] Miscellaneous fixes for v5.18 Paul E. McKenney
` (4 preceding siblings ...)
2022-02-04 23:08 ` [PATCH rcu 5/9] kasan: Record work creation stack trace with interrupts enabled Paul E. McKenney
@ 2022-02-04 23:08 ` Paul E. McKenney
2022-02-04 23:08 ` [PATCH rcu 7/9] rcu: Uninline multi-use function: finish_rcuwait() Paul E. McKenney
` (2 subsequent siblings)
8 siblings, 0 replies; 17+ messages in thread
From: Paul E. McKenney @ 2022-02-04 23:08 UTC (permalink / raw)
To: rcu
Cc: linux-kernel, kernel-team, rostedt, Paul E. McKenney,
Zhouyi Zhou, Frederic Weisbecker
KCSAN reports data races between the rcu_segcblist_clear_flags() and
rcu_segcblist_set_flags() functions, though misreporting the latter
as a call to rcu_segcblist_is_enabled() from call_rcu(). This commit
converts the updates of this field to WRITE_ONCE(), relying on the
resulting unmarked reads to continue to detect buggy concurrent writes
to this field.
Reported-by: Zhouyi Zhou <zhouzhouyi@gmail.com>
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
Cc: Frederic Weisbecker <frederic@kernel.org>
---
kernel/rcu/rcu_segcblist.h | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git a/kernel/rcu/rcu_segcblist.h b/kernel/rcu/rcu_segcblist.h
index e373fbe44da5e..431cee212467d 100644
--- a/kernel/rcu/rcu_segcblist.h
+++ b/kernel/rcu/rcu_segcblist.h
@@ -56,13 +56,13 @@ static inline long rcu_segcblist_n_cbs(struct rcu_segcblist *rsclp)
static inline void rcu_segcblist_set_flags(struct rcu_segcblist *rsclp,
int flags)
{
- rsclp->flags |= flags;
+ WRITE_ONCE(rsclp->flags, rsclp->flags | flags);
}
static inline void rcu_segcblist_clear_flags(struct rcu_segcblist *rsclp,
int flags)
{
- rsclp->flags &= ~flags;
+ WRITE_ONCE(rsclp->flags, rsclp->flags & ~flags);
}
static inline bool rcu_segcblist_test_flags(struct rcu_segcblist *rsclp,
--
2.31.1.189.g2e36527f23
^ permalink raw reply related [flat|nested] 17+ messages in thread
* [PATCH rcu 7/9] rcu: Uninline multi-use function: finish_rcuwait()
2022-02-04 23:07 [PATCH rcu 0/9] Miscellaneous fixes for v5.18 Paul E. McKenney
` (5 preceding siblings ...)
2022-02-04 23:08 ` [PATCH rcu 6/9] rcu: Mark writes to the rcu_segcblist structure's ->flags field Paul E. McKenney
@ 2022-02-04 23:08 ` Paul E. McKenney
2022-02-04 23:08 ` [PATCH rcu 8/9] rcu: Remove __read_mostly annotations from rcu_scheduler_active externs Paul E. McKenney
2022-02-04 23:08 ` [PATCH rcu 9/9] rcu: Replace cpumask_weight with cpumask_empty where appropriate Paul E. McKenney
8 siblings, 0 replies; 17+ messages in thread
From: Paul E. McKenney @ 2022-02-04 23:08 UTC (permalink / raw)
To: rcu; +Cc: linux-kernel, kernel-team, rostedt, Ingo Molnar, Paul E . McKenney
From: Ingo Molnar <mingo@kernel.org>
This is a rarely used function, so uninlining its 3 instructions
is probably a win or a wash - but the main motivation is to
make <linux/rcuwait.h> independent of task_struct details.
Signed-off-by: Ingo Molnar <mingo@kernel.org>
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
---
include/linux/rcuwait.h | 6 +-----
kernel/rcu/update.c | 7 +++++++
2 files changed, 8 insertions(+), 5 deletions(-)
diff --git a/include/linux/rcuwait.h b/include/linux/rcuwait.h
index 61c56cca95c43..8052d34da7826 100644
--- a/include/linux/rcuwait.h
+++ b/include/linux/rcuwait.h
@@ -47,11 +47,7 @@ static inline void prepare_to_rcuwait(struct rcuwait *w)
rcu_assign_pointer(w->task, current);
}
-static inline void finish_rcuwait(struct rcuwait *w)
-{
- rcu_assign_pointer(w->task, NULL);
- __set_current_state(TASK_RUNNING);
-}
+extern void finish_rcuwait(struct rcuwait *w);
#define rcuwait_wait_event(w, condition, state) \
({ \
diff --git a/kernel/rcu/update.c b/kernel/rcu/update.c
index 156892c22bb56..df6197f0d5ce8 100644
--- a/kernel/rcu/update.c
+++ b/kernel/rcu/update.c
@@ -407,6 +407,13 @@ void __wait_rcu_gp(bool checktiny, int n, call_rcu_func_t *crcu_array,
}
EXPORT_SYMBOL_GPL(__wait_rcu_gp);
+void finish_rcuwait(struct rcuwait *w)
+{
+ rcu_assign_pointer(w->task, NULL);
+ __set_current_state(TASK_RUNNING);
+}
+EXPORT_SYMBOL_GPL(finish_rcuwait);
+
#ifdef CONFIG_DEBUG_OBJECTS_RCU_HEAD
void init_rcu_head(struct rcu_head *head)
{
--
2.31.1.189.g2e36527f23
^ permalink raw reply related [flat|nested] 17+ messages in thread
* [PATCH rcu 8/9] rcu: Remove __read_mostly annotations from rcu_scheduler_active externs
2022-02-04 23:07 [PATCH rcu 0/9] Miscellaneous fixes for v5.18 Paul E. McKenney
` (6 preceding siblings ...)
2022-02-04 23:08 ` [PATCH rcu 7/9] rcu: Uninline multi-use function: finish_rcuwait() Paul E. McKenney
@ 2022-02-04 23:08 ` Paul E. McKenney
2022-02-04 23:08 ` [PATCH rcu 9/9] rcu: Replace cpumask_weight with cpumask_empty where appropriate Paul E. McKenney
8 siblings, 0 replies; 17+ messages in thread
From: Paul E. McKenney @ 2022-02-04 23:08 UTC (permalink / raw)
To: rcu; +Cc: linux-kernel, kernel-team, rostedt, Ingo Molnar, Paul E . McKenney
From: Ingo Molnar <mingo@kernel.org>
Remove the __read_mostly attributes from the rcu_scheduler_active
extern declarations, because these attributes are ignored for
prototypes and we'd have to include the full <linux/cache.h> header
to gain this functionally pointless attribute defined.
Signed-off-by: Ingo Molnar <mingo@kernel.org>
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
---
include/linux/rcupdate.h | 2 +-
include/linux/rcutree.h | 2 +-
2 files changed, 2 insertions(+), 2 deletions(-)
diff --git a/include/linux/rcupdate.h b/include/linux/rcupdate.h
index 9d7df8d36af07..e7c39c200e2b0 100644
--- a/include/linux/rcupdate.h
+++ b/include/linux/rcupdate.h
@@ -84,7 +84,7 @@ static inline int rcu_preempt_depth(void)
/* Internal to kernel */
void rcu_init(void);
-extern int rcu_scheduler_active __read_mostly;
+extern int rcu_scheduler_active;
void rcu_sched_clock_irq(int user);
void rcu_report_dead(unsigned int cpu);
void rcutree_migrate_callbacks(int cpu);
diff --git a/include/linux/rcutree.h b/include/linux/rcutree.h
index 53209d6694001..76665db179fa1 100644
--- a/include/linux/rcutree.h
+++ b/include/linux/rcutree.h
@@ -62,7 +62,7 @@ static inline void rcu_irq_exit_check_preempt(void) { }
void exit_rcu(void);
void rcu_scheduler_starting(void);
-extern int rcu_scheduler_active __read_mostly;
+extern int rcu_scheduler_active;
void rcu_end_inkernel_boot(void);
bool rcu_inkernel_boot_has_ended(void);
bool rcu_is_watching(void);
--
2.31.1.189.g2e36527f23
^ permalink raw reply related [flat|nested] 17+ messages in thread
* [PATCH rcu 9/9] rcu: Replace cpumask_weight with cpumask_empty where appropriate
2022-02-04 23:07 [PATCH rcu 0/9] Miscellaneous fixes for v5.18 Paul E. McKenney
` (7 preceding siblings ...)
2022-02-04 23:08 ` [PATCH rcu 8/9] rcu: Remove __read_mostly annotations from rcu_scheduler_active externs Paul E. McKenney
@ 2022-02-04 23:08 ` Paul E. McKenney
2022-02-11 15:17 ` Frederic Weisbecker
8 siblings, 1 reply; 17+ messages in thread
From: Paul E. McKenney @ 2022-02-04 23:08 UTC (permalink / raw)
To: rcu; +Cc: linux-kernel, kernel-team, rostedt, Yury Norov, Paul E . McKenney
From: Yury Norov <yury.norov@gmail.com>
In some places, RCU code calls cpumask_weight() to check if any bit of a
given cpumask is set. We can do it more efficiently with cpumask_empty()
because cpumask_empty() stops traversing the cpumask as soon as it finds
first set bit, while cpumask_weight() counts all bits unconditionally.
Signed-off-by: Yury Norov <yury.norov@gmail.com>
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
---
kernel/rcu/tree_nocb.h | 4 ++--
kernel/rcu/tree_plugin.h | 2 +-
2 files changed, 3 insertions(+), 3 deletions(-)
diff --git a/kernel/rcu/tree_nocb.h b/kernel/rcu/tree_nocb.h
index eeafb546a7a09..f83c7b1d61103 100644
--- a/kernel/rcu/tree_nocb.h
+++ b/kernel/rcu/tree_nocb.h
@@ -1169,7 +1169,7 @@ void __init rcu_init_nohz(void)
struct rcu_data *rdp;
#if defined(CONFIG_NO_HZ_FULL)
- if (tick_nohz_full_running && cpumask_weight(tick_nohz_full_mask))
+ if (tick_nohz_full_running && !cpumask_empty(tick_nohz_full_mask))
need_rcu_nocb_mask = true;
#endif /* #if defined(CONFIG_NO_HZ_FULL) */
@@ -1348,7 +1348,7 @@ static void __init rcu_organize_nocb_kthreads(void)
*/
void rcu_bind_current_to_nocb(void)
{
- if (cpumask_available(rcu_nocb_mask) && cpumask_weight(rcu_nocb_mask))
+ if (cpumask_available(rcu_nocb_mask) && !cpumask_empty(rcu_nocb_mask))
WARN_ON(sched_setaffinity(current->pid, rcu_nocb_mask));
}
EXPORT_SYMBOL_GPL(rcu_bind_current_to_nocb);
diff --git a/kernel/rcu/tree_plugin.h b/kernel/rcu/tree_plugin.h
index 07845dcd33c5e..efd0c87d2ffae 100644
--- a/kernel/rcu/tree_plugin.h
+++ b/kernel/rcu/tree_plugin.h
@@ -1220,7 +1220,7 @@ static void rcu_boost_kthread_setaffinity(struct rcu_node *rnp, int outgoingcpu)
cpu != outgoingcpu)
cpumask_set_cpu(cpu, cm);
cpumask_and(cm, cm, housekeeping_cpumask(HK_FLAG_RCU));
- if (cpumask_weight(cm) == 0)
+ if (cpumask_empty(cm))
cpumask_copy(cm, housekeeping_cpumask(HK_FLAG_RCU));
set_cpus_allowed_ptr(t, cm);
mutex_unlock(&rnp->boost_kthread_mutex);
--
2.31.1.189.g2e36527f23
^ permalink raw reply related [flat|nested] 17+ messages in thread
* Re: [PATCH rcu 3/9] rcu: Add mutex for rcu boost kthread spawning and affinity setting
2022-02-04 23:07 ` [PATCH rcu 3/9] rcu: Add mutex for rcu boost kthread spawning and affinity setting Paul E. McKenney
@ 2022-02-11 14:57 ` Frederic Weisbecker
2022-02-11 15:11 ` Frederic Weisbecker
0 siblings, 1 reply; 17+ messages in thread
From: Frederic Weisbecker @ 2022-02-11 14:57 UTC (permalink / raw)
To: Paul E. McKenney; +Cc: rcu, linux-kernel, kernel-team, rostedt, David Woodhouse
On Fri, Feb 04, 2022 at 03:07:59PM -0800, Paul E. McKenney wrote:
> From: David Woodhouse <dwmw@amazon.co.uk>
>
> As we handle parallel CPU bringup, we will need to take care to avoid
> spawning multiple boost threads, or race conditions when setting their
> affinity. Spotted by Paul McKenney.
>
> Signed-off-by: David Woodhouse <dwmw@amazon.co.uk>
> Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
Reviewed-by: Frederic Weisbecker <frederic@kernel.org>
Speaking of, we have:
rcu_init()
for_each_online_cpu(cpu) // should be boot CPU only at this stage ?
rcutree_prepare_cpu(cpu)
rcu_spawn_one_boost_kthread(cpu)
early_initcall()
rcu_spawn_gp_kthread()
rcu_spawn_boost_kthreads()
rcu_for_each_leaf_node(rnp)
rcu_rnp_online_cpus(rnp) // as above, only boot CPU at this stage.
rcu_spawn_one_boost_kthread(cpu)
cpu_up()
rcutree_prepare_cpu(cpu)
rcu_spawn_one_boost_kthread(cpu)
My guess is that we could remove rcu_spawn_boost_kthreads() and simplify
rcu_init(). Something like this (untested yet):
diff --git a/kernel/rcu/tree.c b/kernel/rcu/tree.c
index 86eec6a0f1a1..da8ac2b6f8cc 100644
--- a/kernel/rcu/tree.c
+++ b/kernel/rcu/tree.c
@@ -4526,7 +4526,6 @@ static int __init rcu_spawn_gp_kthread(void)
raw_spin_unlock_irqrestore_rcu_node(rnp, flags);
wake_up_process(t);
rcu_spawn_nocb_kthreads();
- rcu_spawn_boost_kthreads();
rcu_spawn_core_kthreads();
return 0;
}
@@ -4813,7 +4812,7 @@ static void __init kfree_rcu_batch_init(void)
void __init rcu_init(void)
{
- int cpu;
+ int cpu = smp_processor_id();
rcu_early_boot_tests();
@@ -4833,11 +4832,10 @@ void __init rcu_init(void)
* or the scheduler are operational.
*/
pm_notifier(rcu_pm_notify, 0);
- for_each_online_cpu(cpu) {
- rcutree_prepare_cpu(cpu);
- rcu_cpu_starting(cpu);
- rcutree_online_cpu(cpu);
- }
+
+ rcutree_prepare_cpu(cpu);
+ rcu_cpu_starting(cpu);
+ rcutree_online_cpu(cpu);
/* Create workqueue for Tree SRCU and for expedited GPs. */
rcu_gp_wq = alloc_workqueue("rcu_gp", WQ_MEM_RECLAIM, 0);
diff --git a/kernel/rcu/tree_plugin.h b/kernel/rcu/tree_plugin.h
index 6082dd23408f..90925a589774 100644
--- a/kernel/rcu/tree_plugin.h
+++ b/kernel/rcu/tree_plugin.h
@@ -1226,18 +1226,6 @@ static void rcu_boost_kthread_setaffinity(struct rcu_node *rnp, int outgoingcpu)
free_cpumask_var(cm);
}
-/*
- * Spawn boost kthreads -- called as soon as the scheduler is running.
- */
-static void __init rcu_spawn_boost_kthreads(void)
-{
- struct rcu_node *rnp;
-
- rcu_for_each_leaf_node(rnp)
- if (rcu_rnp_online_cpus(rnp))
- rcu_spawn_one_boost_kthread(rnp);
-}
-
#else /* #ifdef CONFIG_RCU_BOOST */
static void rcu_initiate_boost(struct rcu_node *rnp, unsigned long flags)
@@ -1263,10 +1251,6 @@ static void rcu_boost_kthread_setaffinity(struct rcu_node *rnp, int outgoingcpu)
{
}
-static void __init rcu_spawn_boost_kthreads(void)
-{
-}
-
#endif /* #else #ifdef CONFIG_RCU_BOOST */
/*
^ permalink raw reply related [flat|nested] 17+ messages in thread
* Re: [PATCH rcu 3/9] rcu: Add mutex for rcu boost kthread spawning and affinity setting
2022-02-11 14:57 ` Frederic Weisbecker
@ 2022-02-11 15:11 ` Frederic Weisbecker
2022-02-11 15:42 ` Paul E. McKenney
0 siblings, 1 reply; 17+ messages in thread
From: Frederic Weisbecker @ 2022-02-11 15:11 UTC (permalink / raw)
To: Paul E. McKenney; +Cc: rcu, linux-kernel, kernel-team, rostedt, David Woodhouse
On Fri, Feb 11, 2022 at 03:57:57PM +0100, Frederic Weisbecker wrote:
> On Fri, Feb 04, 2022 at 03:07:59PM -0800, Paul E. McKenney wrote:
> > From: David Woodhouse <dwmw@amazon.co.uk>
> >
> > As we handle parallel CPU bringup, we will need to take care to avoid
> > spawning multiple boost threads, or race conditions when setting their
> > affinity. Spotted by Paul McKenney.
> >
> > Signed-off-by: David Woodhouse <dwmw@amazon.co.uk>
> > Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
>
> Reviewed-by: Frederic Weisbecker <frederic@kernel.org>
>
> Speaking of, we have:
>
> rcu_init()
> for_each_online_cpu(cpu) // should be boot CPU only at this stage ?
> rcutree_prepare_cpu(cpu)
> rcu_spawn_one_boost_kthread(cpu)
>
>
> early_initcall()
> rcu_spawn_gp_kthread()
> rcu_spawn_boost_kthreads()
> rcu_for_each_leaf_node(rnp)
> rcu_rnp_online_cpus(rnp) // as above, only boot CPU at this stage.
> rcu_spawn_one_boost_kthread(cpu)
>
> cpu_up()
> rcutree_prepare_cpu(cpu)
> rcu_spawn_one_boost_kthread(cpu)
>
>
> My guess is that we could remove rcu_spawn_boost_kthreads() and simplify
> rcu_init(). Something like this (untested yet):
>
> diff --git a/kernel/rcu/tree.c b/kernel/rcu/tree.c
> index 86eec6a0f1a1..da8ac2b6f8cc 100644
> --- a/kernel/rcu/tree.c
> +++ b/kernel/rcu/tree.c
> @@ -4526,7 +4526,6 @@ static int __init rcu_spawn_gp_kthread(void)
> raw_spin_unlock_irqrestore_rcu_node(rnp, flags);
> wake_up_process(t);
> rcu_spawn_nocb_kthreads();
> - rcu_spawn_boost_kthreads();
> rcu_spawn_core_kthreads();
> return 0;
> }
> @@ -4813,7 +4812,7 @@ static void __init kfree_rcu_batch_init(void)
>
> void __init rcu_init(void)
> {
> - int cpu;
> + int cpu = smp_processor_id();
>
> rcu_early_boot_tests();
>
> @@ -4833,11 +4832,10 @@ void __init rcu_init(void)
> * or the scheduler are operational.
> */
> pm_notifier(rcu_pm_notify, 0);
> - for_each_online_cpu(cpu) {
> - rcutree_prepare_cpu(cpu);
> - rcu_cpu_starting(cpu);
> - rcutree_online_cpu(cpu);
> - }
> +
> + rcutree_prepare_cpu(cpu);
> + rcu_cpu_starting(cpu);
> + rcutree_online_cpu(cpu);
>
> /* Create workqueue for Tree SRCU and for expedited GPs. */
> rcu_gp_wq = alloc_workqueue("rcu_gp", WQ_MEM_RECLAIM, 0);
> diff --git a/kernel/rcu/tree_plugin.h b/kernel/rcu/tree_plugin.h
> index 6082dd23408f..90925a589774 100644
> --- a/kernel/rcu/tree_plugin.h
> +++ b/kernel/rcu/tree_plugin.h
> @@ -1226,18 +1226,6 @@ static void rcu_boost_kthread_setaffinity(struct rcu_node *rnp, int outgoingcpu)
> free_cpumask_var(cm);
> }
>
> -/*
> - * Spawn boost kthreads -- called as soon as the scheduler is running.
> - */
> -static void __init rcu_spawn_boost_kthreads(void)
> -{
> - struct rcu_node *rnp;
> -
> - rcu_for_each_leaf_node(rnp)
> - if (rcu_rnp_online_cpus(rnp))
> - rcu_spawn_one_boost_kthread(rnp);
> -}
> -
> #else /* #ifdef CONFIG_RCU_BOOST */
>
> static void rcu_initiate_boost(struct rcu_node *rnp, unsigned long flags)
> @@ -1263,10 +1251,6 @@ static void rcu_boost_kthread_setaffinity(struct rcu_node *rnp, int outgoingcpu)
> {
> }
>
> -static void __init rcu_spawn_boost_kthreads(void)
> -{
> -}
> -
> #endif /* #else #ifdef CONFIG_RCU_BOOST */
>
> /*
nocb kthread creation is similar but it depends on the gp kthread.
So we can't rely on rcu_init() -> rcu_prepare_cpu() and we must keep
the early_initcall() -> rcu_spawn_gp_kthread().
That would become (untested again):
diff --git a/kernel/rcu/tree.c b/kernel/rcu/tree.c
index da8ac2b6f8cc..9284625a9a50 100644
--- a/kernel/rcu/tree.c
+++ b/kernel/rcu/tree.c
@@ -4525,7 +4525,7 @@ static int __init rcu_spawn_gp_kthread(void)
smp_store_release(&rcu_state.gp_kthread, t); /* ^^^ */
raw_spin_unlock_irqrestore_rcu_node(rnp, flags);
wake_up_process(t);
- rcu_spawn_nocb_kthreads();
+ rcu_spawn_cpu_nocb_kthread(smp_processor_id());
rcu_spawn_core_kthreads();
return 0;
}
diff --git a/kernel/rcu/tree_nocb.h b/kernel/rcu/tree_nocb.h
index 636d0546a4e9..711f6eb7f7e1 100644
--- a/kernel/rcu/tree_nocb.h
+++ b/kernel/rcu/tree_nocb.h
@@ -1277,22 +1277,6 @@ static void rcu_spawn_cpu_nocb_kthread(int cpu)
WRITE_ONCE(rdp->nocb_gp_kthread, rdp_gp->nocb_gp_kthread);
}
-/*
- * Once the scheduler is running, spawn rcuo kthreads for all online
- * no-CBs CPUs. This assumes that the early_initcall()s happen before
- * non-boot CPUs come online -- if this changes, we will need to add
- * some mutual exclusion.
- */
-static void __init rcu_spawn_nocb_kthreads(void)
-{
- int cpu;
-
- if (rcu_nocb_is_setup) {
- for_each_online_cpu(cpu)
- rcu_spawn_cpu_nocb_kthread(cpu);
- }
-}
-
/* How many CB CPU IDs per GP kthread? Default of -1 for sqrt(nr_cpu_ids). */
static int rcu_nocb_gp_stride = -1;
module_param(rcu_nocb_gp_stride, int, 0444);
@@ -1549,10 +1533,6 @@ static void rcu_spawn_cpu_nocb_kthread(int cpu)
{
}
-static void __init rcu_spawn_nocb_kthreads(void)
-{
-}
-
static void show_rcu_nocb_state(struct rcu_data *rdp)
{
}
^ permalink raw reply related [flat|nested] 17+ messages in thread
* Re: [PATCH rcu 1/9] MAINTAINERS: Add Frederic and Neeraj to their RCU files
2022-02-04 23:07 ` [PATCH rcu 1/9] MAINTAINERS: Add Frederic and Neeraj to their RCU files Paul E. McKenney
@ 2022-02-11 15:15 ` Frederic Weisbecker
2022-02-11 16:40 ` Paul E. McKenney
0 siblings, 1 reply; 17+ messages in thread
From: Frederic Weisbecker @ 2022-02-11 15:15 UTC (permalink / raw)
To: Paul E. McKenney; +Cc: rcu, linux-kernel, kernel-team, rostedt, Neeraj Upadhyay
On Fri, Feb 04, 2022 at 03:07:57PM -0800, Paul E. McKenney wrote:
> Adding Frederic as an RCU maintainer for kernel/rcu/tree_nocb.h given his
> work with offloading and de-offloading callbacks from CPUs. Also adding
> Neeraj for kernel/rcu/tasks.h given his focused work on RCU Tasks Trace.
> As in I am reasonably certain that each understands the full contents
> of the corresponding file.
>
> Cc: Neeraj Upadhyay <quic_neeraju@quicinc.com>
> Cc: Frederic Weisbecker <frederic@kernel.org>
> Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
Acked-by: Frederic Weisbecker <frederic@kernel.org>
Thanks! :)
^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: [PATCH rcu 9/9] rcu: Replace cpumask_weight with cpumask_empty where appropriate
2022-02-04 23:08 ` [PATCH rcu 9/9] rcu: Replace cpumask_weight with cpumask_empty where appropriate Paul E. McKenney
@ 2022-02-11 15:17 ` Frederic Weisbecker
0 siblings, 0 replies; 17+ messages in thread
From: Frederic Weisbecker @ 2022-02-11 15:17 UTC (permalink / raw)
To: Paul E. McKenney; +Cc: rcu, linux-kernel, kernel-team, rostedt, Yury Norov
On Fri, Feb 04, 2022 at 03:08:05PM -0800, Paul E. McKenney wrote:
> From: Yury Norov <yury.norov@gmail.com>
>
> In some places, RCU code calls cpumask_weight() to check if any bit of a
> given cpumask is set. We can do it more efficiently with cpumask_empty()
> because cpumask_empty() stops traversing the cpumask as soon as it finds
> first set bit, while cpumask_weight() counts all bits unconditionally.
>
> Signed-off-by: Yury Norov <yury.norov@gmail.com>
> Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
Acked-by: Frederic Weisbecker <frederic@kernel.org>
^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: [PATCH rcu 3/9] rcu: Add mutex for rcu boost kthread spawning and affinity setting
2022-02-11 15:11 ` Frederic Weisbecker
@ 2022-02-11 15:42 ` Paul E. McKenney
2022-02-11 15:43 ` Frederic Weisbecker
0 siblings, 1 reply; 17+ messages in thread
From: Paul E. McKenney @ 2022-02-11 15:42 UTC (permalink / raw)
To: Frederic Weisbecker
Cc: rcu, linux-kernel, kernel-team, rostedt, David Woodhouse
On Fri, Feb 11, 2022 at 04:11:48PM +0100, Frederic Weisbecker wrote:
> On Fri, Feb 11, 2022 at 03:57:57PM +0100, Frederic Weisbecker wrote:
> > On Fri, Feb 04, 2022 at 03:07:59PM -0800, Paul E. McKenney wrote:
> > > From: David Woodhouse <dwmw@amazon.co.uk>
> > >
> > > As we handle parallel CPU bringup, we will need to take care to avoid
> > > spawning multiple boost threads, or race conditions when setting their
> > > affinity. Spotted by Paul McKenney.
> > >
> > > Signed-off-by: David Woodhouse <dwmw@amazon.co.uk>
> > > Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
> >
> > Reviewed-by: Frederic Weisbecker <frederic@kernel.org>
> >
> > Speaking of, we have:
> >
> > rcu_init()
> > for_each_online_cpu(cpu) // should be boot CPU only at this stage ?
> > rcutree_prepare_cpu(cpu)
> > rcu_spawn_one_boost_kthread(cpu)
> >
> >
> > early_initcall()
> > rcu_spawn_gp_kthread()
> > rcu_spawn_boost_kthreads()
> > rcu_for_each_leaf_node(rnp)
> > rcu_rnp_online_cpus(rnp) // as above, only boot CPU at this stage.
> > rcu_spawn_one_boost_kthread(cpu)
> >
> > cpu_up()
> > rcutree_prepare_cpu(cpu)
> > rcu_spawn_one_boost_kthread(cpu)
> >
> >
> > My guess is that we could remove rcu_spawn_boost_kthreads() and simplify
> > rcu_init(). Something like this (untested yet):
If you also add a WARN_ON() for more than one CPU being online at
rcu_init() time, I am good with this approach. The main danger that
the pre-rcu_init() portion of the boot time becomes the long straw in
the eternal quest to make systems boot faster, but it is not hard to
put it back. Hence the WARN_ON() to make it clear that adjustment
is needed. ;-)
Thanx, Paul
> > diff --git a/kernel/rcu/tree.c b/kernel/rcu/tree.c
> > index 86eec6a0f1a1..da8ac2b6f8cc 100644
> > --- a/kernel/rcu/tree.c
> > +++ b/kernel/rcu/tree.c
> > @@ -4526,7 +4526,6 @@ static int __init rcu_spawn_gp_kthread(void)
> > raw_spin_unlock_irqrestore_rcu_node(rnp, flags);
> > wake_up_process(t);
> > rcu_spawn_nocb_kthreads();
> > - rcu_spawn_boost_kthreads();
> > rcu_spawn_core_kthreads();
> > return 0;
> > }
> > @@ -4813,7 +4812,7 @@ static void __init kfree_rcu_batch_init(void)
> >
> > void __init rcu_init(void)
> > {
> > - int cpu;
> > + int cpu = smp_processor_id();
> >
> > rcu_early_boot_tests();
> >
> > @@ -4833,11 +4832,10 @@ void __init rcu_init(void)
> > * or the scheduler are operational.
> > */
> > pm_notifier(rcu_pm_notify, 0);
> > - for_each_online_cpu(cpu) {
> > - rcutree_prepare_cpu(cpu);
> > - rcu_cpu_starting(cpu);
> > - rcutree_online_cpu(cpu);
> > - }
> > +
> > + rcutree_prepare_cpu(cpu);
> > + rcu_cpu_starting(cpu);
> > + rcutree_online_cpu(cpu);
> >
> > /* Create workqueue for Tree SRCU and for expedited GPs. */
> > rcu_gp_wq = alloc_workqueue("rcu_gp", WQ_MEM_RECLAIM, 0);
> > diff --git a/kernel/rcu/tree_plugin.h b/kernel/rcu/tree_plugin.h
> > index 6082dd23408f..90925a589774 100644
> > --- a/kernel/rcu/tree_plugin.h
> > +++ b/kernel/rcu/tree_plugin.h
> > @@ -1226,18 +1226,6 @@ static void rcu_boost_kthread_setaffinity(struct rcu_node *rnp, int outgoingcpu)
> > free_cpumask_var(cm);
> > }
> >
> > -/*
> > - * Spawn boost kthreads -- called as soon as the scheduler is running.
> > - */
> > -static void __init rcu_spawn_boost_kthreads(void)
> > -{
> > - struct rcu_node *rnp;
> > -
> > - rcu_for_each_leaf_node(rnp)
> > - if (rcu_rnp_online_cpus(rnp))
> > - rcu_spawn_one_boost_kthread(rnp);
> > -}
> > -
> > #else /* #ifdef CONFIG_RCU_BOOST */
> >
> > static void rcu_initiate_boost(struct rcu_node *rnp, unsigned long flags)
> > @@ -1263,10 +1251,6 @@ static void rcu_boost_kthread_setaffinity(struct rcu_node *rnp, int outgoingcpu)
> > {
> > }
> >
> > -static void __init rcu_spawn_boost_kthreads(void)
> > -{
> > -}
> > -
> > #endif /* #else #ifdef CONFIG_RCU_BOOST */
> >
> > /*
>
> nocb kthread creation is similar but it depends on the gp kthread.
> So we can't rely on rcu_init() -> rcu_prepare_cpu() and we must keep
> the early_initcall() -> rcu_spawn_gp_kthread().
>
> That would become (untested again):
>
> diff --git a/kernel/rcu/tree.c b/kernel/rcu/tree.c
> index da8ac2b6f8cc..9284625a9a50 100644
> --- a/kernel/rcu/tree.c
> +++ b/kernel/rcu/tree.c
> @@ -4525,7 +4525,7 @@ static int __init rcu_spawn_gp_kthread(void)
> smp_store_release(&rcu_state.gp_kthread, t); /* ^^^ */
> raw_spin_unlock_irqrestore_rcu_node(rnp, flags);
> wake_up_process(t);
> - rcu_spawn_nocb_kthreads();
> + rcu_spawn_cpu_nocb_kthread(smp_processor_id());
> rcu_spawn_core_kthreads();
> return 0;
> }
> diff --git a/kernel/rcu/tree_nocb.h b/kernel/rcu/tree_nocb.h
> index 636d0546a4e9..711f6eb7f7e1 100644
> --- a/kernel/rcu/tree_nocb.h
> +++ b/kernel/rcu/tree_nocb.h
> @@ -1277,22 +1277,6 @@ static void rcu_spawn_cpu_nocb_kthread(int cpu)
> WRITE_ONCE(rdp->nocb_gp_kthread, rdp_gp->nocb_gp_kthread);
> }
>
> -/*
> - * Once the scheduler is running, spawn rcuo kthreads for all online
> - * no-CBs CPUs. This assumes that the early_initcall()s happen before
> - * non-boot CPUs come online -- if this changes, we will need to add
> - * some mutual exclusion.
> - */
> -static void __init rcu_spawn_nocb_kthreads(void)
> -{
> - int cpu;
> -
> - if (rcu_nocb_is_setup) {
> - for_each_online_cpu(cpu)
> - rcu_spawn_cpu_nocb_kthread(cpu);
> - }
> -}
> -
> /* How many CB CPU IDs per GP kthread? Default of -1 for sqrt(nr_cpu_ids). */
> static int rcu_nocb_gp_stride = -1;
> module_param(rcu_nocb_gp_stride, int, 0444);
> @@ -1549,10 +1533,6 @@ static void rcu_spawn_cpu_nocb_kthread(int cpu)
> {
> }
>
> -static void __init rcu_spawn_nocb_kthreads(void)
> -{
> -}
> -
> static void show_rcu_nocb_state(struct rcu_data *rdp)
> {
> }
^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: [PATCH rcu 3/9] rcu: Add mutex for rcu boost kthread spawning and affinity setting
2022-02-11 15:42 ` Paul E. McKenney
@ 2022-02-11 15:43 ` Frederic Weisbecker
0 siblings, 0 replies; 17+ messages in thread
From: Frederic Weisbecker @ 2022-02-11 15:43 UTC (permalink / raw)
To: Paul E. McKenney; +Cc: rcu, linux-kernel, kernel-team, rostedt, David Woodhouse
On Fri, Feb 11, 2022 at 07:42:03AM -0800, Paul E. McKenney wrote:
> On Fri, Feb 11, 2022 at 04:11:48PM +0100, Frederic Weisbecker wrote:
> > On Fri, Feb 11, 2022 at 03:57:57PM +0100, Frederic Weisbecker wrote:
> > > On Fri, Feb 04, 2022 at 03:07:59PM -0800, Paul E. McKenney wrote:
> > > > From: David Woodhouse <dwmw@amazon.co.uk>
> > > >
> > > > As we handle parallel CPU bringup, we will need to take care to avoid
> > > > spawning multiple boost threads, or race conditions when setting their
> > > > affinity. Spotted by Paul McKenney.
> > > >
> > > > Signed-off-by: David Woodhouse <dwmw@amazon.co.uk>
> > > > Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
> > >
> > > Reviewed-by: Frederic Weisbecker <frederic@kernel.org>
> > >
> > > Speaking of, we have:
> > >
> > > rcu_init()
> > > for_each_online_cpu(cpu) // should be boot CPU only at this stage ?
> > > rcutree_prepare_cpu(cpu)
> > > rcu_spawn_one_boost_kthread(cpu)
> > >
> > >
> > > early_initcall()
> > > rcu_spawn_gp_kthread()
> > > rcu_spawn_boost_kthreads()
> > > rcu_for_each_leaf_node(rnp)
> > > rcu_rnp_online_cpus(rnp) // as above, only boot CPU at this stage.
> > > rcu_spawn_one_boost_kthread(cpu)
> > >
> > > cpu_up()
> > > rcutree_prepare_cpu(cpu)
> > > rcu_spawn_one_boost_kthread(cpu)
> > >
> > >
> > > My guess is that we could remove rcu_spawn_boost_kthreads() and simplify
> > > rcu_init(). Something like this (untested yet):
>
> If you also add a WARN_ON() for more than one CPU being online at
> rcu_init() time, I am good with this approach. The main danger that
> the pre-rcu_init() portion of the boot time becomes the long straw in
> the eternal quest to make systems boot faster, but it is not hard to
> put it back. Hence the WARN_ON() to make it clear that adjustment
> is needed. ;-)
>
> Thanx, Paul
Make sense. On my TODO list then.
Thanks!
^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: [PATCH rcu 1/9] MAINTAINERS: Add Frederic and Neeraj to their RCU files
2022-02-11 15:15 ` Frederic Weisbecker
@ 2022-02-11 16:40 ` Paul E. McKenney
0 siblings, 0 replies; 17+ messages in thread
From: Paul E. McKenney @ 2022-02-11 16:40 UTC (permalink / raw)
To: Frederic Weisbecker
Cc: rcu, linux-kernel, kernel-team, rostedt, Neeraj Upadhyay
On Fri, Feb 11, 2022 at 04:15:29PM +0100, Frederic Weisbecker wrote:
> On Fri, Feb 04, 2022 at 03:07:57PM -0800, Paul E. McKenney wrote:
> > Adding Frederic as an RCU maintainer for kernel/rcu/tree_nocb.h given his
> > work with offloading and de-offloading callbacks from CPUs. Also adding
> > Neeraj for kernel/rcu/tasks.h given his focused work on RCU Tasks Trace.
> > As in I am reasonably certain that each understands the full contents
> > of the corresponding file.
> >
> > Cc: Neeraj Upadhyay <quic_neeraju@quicinc.com>
> > Cc: Frederic Weisbecker <frederic@kernel.org>
> > Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
>
> Acked-by: Frederic Weisbecker <frederic@kernel.org>
>
> Thanks! :)
Thank -you- for all the help with RCU!
I will apply on my next rebase.
Thanx, Paul
^ permalink raw reply [flat|nested] 17+ messages in thread
end of thread, other threads:[~2022-02-11 16:40 UTC | newest]
Thread overview: 17+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2022-02-04 23:07 [PATCH rcu 0/9] Miscellaneous fixes for v5.18 Paul E. McKenney
2022-02-04 23:07 ` [PATCH rcu 1/9] MAINTAINERS: Add Frederic and Neeraj to their RCU files Paul E. McKenney
2022-02-11 15:15 ` Frederic Weisbecker
2022-02-11 16:40 ` Paul E. McKenney
2022-02-04 23:07 ` [PATCH rcu 2/9] rcu: Fix description of kvfree_rcu() Paul E. McKenney
2022-02-04 23:07 ` [PATCH rcu 3/9] rcu: Add mutex for rcu boost kthread spawning and affinity setting Paul E. McKenney
2022-02-11 14:57 ` Frederic Weisbecker
2022-02-11 15:11 ` Frederic Weisbecker
2022-02-11 15:42 ` Paul E. McKenney
2022-02-11 15:43 ` Frederic Weisbecker
2022-02-04 23:08 ` [PATCH rcu 4/9] rcu: Inline __call_rcu() into call_rcu() Paul E. McKenney
2022-02-04 23:08 ` [PATCH rcu 5/9] kasan: Record work creation stack trace with interrupts enabled Paul E. McKenney
2022-02-04 23:08 ` [PATCH rcu 6/9] rcu: Mark writes to the rcu_segcblist structure's ->flags field Paul E. McKenney
2022-02-04 23:08 ` [PATCH rcu 7/9] rcu: Uninline multi-use function: finish_rcuwait() Paul E. McKenney
2022-02-04 23:08 ` [PATCH rcu 8/9] rcu: Remove __read_mostly annotations from rcu_scheduler_active externs Paul E. McKenney
2022-02-04 23:08 ` [PATCH rcu 9/9] rcu: Replace cpumask_weight with cpumask_empty where appropriate Paul E. McKenney
2022-02-11 15:17 ` Frederic Weisbecker
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).