* [RFC PATCH 00/12] rcu/nocb: De-offload and re-offload support v2
@ 2020-09-21 12:43 Frederic Weisbecker
2020-09-21 12:43 ` [RFC PATCH 01/12] rcu: Implement rcu_segcblist_is_offloaded() config dependent Frederic Weisbecker
` (11 more replies)
0 siblings, 12 replies; 32+ messages in thread
From: Frederic Weisbecker @ 2020-09-21 12:43 UTC (permalink / raw)
To: Paul E . McKenney
Cc: LKML, Frederic Weisbecker, Steven Rostedt, Mathieu Desnoyers,
Lai Jiangshan, Joel Fernandes, Josh Triplett
Hi,
This is the second attempt to make rcu-nocb mutable at runtime. See
v1 there: https://lwn.net/Articles/820544/
This time a new approach, as per Paul's suggestion, is to ensure that
callbacks are still served locally until nocb kthreads are fully ready
or fully asleep. This is using a state machine implemented with a mask
(see patch 3/12 for details).
There is still some small polishing to do, as can be seen in the last
patches, to make sure that local callbacks processing is correctly
handled outside of strict softirq (or rcuc) mode.
Thanks.
git://git.kernel.org/pub/scm/linux/kernel/git/frederic/linux-dynticks.git
rcu/nocb
HEAD: 6abe8408307eaeb03b4a0470945943c1decbc4b3
Thanks,
Frederic
---
Frederic Weisbecker (12):
rcu: Implement rcu_segcblist_is_offloaded() config dependent
rcu: Turn enabled/offload states into a common flag
rcu: Provide basic callback offloading state machine bits
rcu: De-offloading CB kthread
rcu: De-offloading GP kthread
rcu: Re-offload support
rcu: Shutdown nocb timer on de-offloading
rcu: Flush bypass before setting SEGCBLIST_SOFTIRQ_ONLY
rcu: Set SEGCBLIST_SOFTIRQ_ONLY at the very last stage of de-offloading
rcu: Process batch locally as long as offloading isn't complete
rcu: Locally accelerate callbacks as long as offloading isn't complete
rcu: Nocb (de)activate through sysfs
include/linux/rcu_segcblist.h | 119 ++++++++++++++++-
include/linux/rcupdate.h | 4 +
kernel/cpu.c | 23 ++++
kernel/rcu/rcu_segcblist.c | 13 +-
kernel/rcu/rcu_segcblist.h | 45 ++++++-
kernel/rcu/tree.c | 32 ++---
kernel/rcu/tree.h | 2 +
kernel/rcu/tree_plugin.h | 297 +++++++++++++++++++++++++++++++++++++++---
8 files changed, 491 insertions(+), 44 deletions(-)
^ permalink raw reply [flat|nested] 32+ messages in thread
* [RFC PATCH 01/12] rcu: Implement rcu_segcblist_is_offloaded() config dependent
2020-09-21 12:43 [RFC PATCH 00/12] rcu/nocb: De-offload and re-offload support v2 Frederic Weisbecker
@ 2020-09-21 12:43 ` Frederic Weisbecker
2020-09-22 0:27 ` Paul E. McKenney
2020-09-21 12:43 ` [RFC PATCH 02/12] rcu: Turn enabled/offload states into a common flag Frederic Weisbecker
` (10 subsequent siblings)
11 siblings, 1 reply; 32+ messages in thread
From: Frederic Weisbecker @ 2020-09-21 12:43 UTC (permalink / raw)
To: Paul E . McKenney
Cc: LKML, Frederic Weisbecker, Steven Rostedt, Mathieu Desnoyers,
Lai Jiangshan, Joel Fernandes, Josh Triplett
This simplify the usage of this API and avoid checking the kernel
config from the callers.
Suggested-by: Paul E. McKenney <paulmck@kernel.org>
Signed-off-by: Frederic Weisbecker <frederic@kernel.org>
Cc: Paul E. McKenney <paulmck@kernel.org>
Cc: Josh Triplett <josh@joshtriplett.org>
Cc: Steven Rostedt <rostedt@goodmis.org>
Cc: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Cc: Lai Jiangshan <jiangshanlai@gmail.com>
Cc: Joel Fernandes <joel@joelfernandes.org>
---
kernel/rcu/rcu_segcblist.h | 2 +-
kernel/rcu/tree.c | 21 +++++++--------------
2 files changed, 8 insertions(+), 15 deletions(-)
diff --git a/kernel/rcu/rcu_segcblist.h b/kernel/rcu/rcu_segcblist.h
index 5c293afc07b8..492262bcb591 100644
--- a/kernel/rcu/rcu_segcblist.h
+++ b/kernel/rcu/rcu_segcblist.h
@@ -62,7 +62,7 @@ static inline bool rcu_segcblist_is_enabled(struct rcu_segcblist *rsclp)
/* Is the specified rcu_segcblist offloaded? */
static inline bool rcu_segcblist_is_offloaded(struct rcu_segcblist *rsclp)
{
- return rsclp->offloaded;
+ return IS_ENABLED(CONFIG_RCU_NOCB_CPU) && rsclp->offloaded;
}
/*
diff --git a/kernel/rcu/tree.c b/kernel/rcu/tree.c
index 8ce77d9ac716..c0286ce8fc03 100644
--- a/kernel/rcu/tree.c
+++ b/kernel/rcu/tree.c
@@ -1583,8 +1583,7 @@ static bool __note_gp_changes(struct rcu_node *rnp, struct rcu_data *rdp)
{
bool ret = false;
bool need_qs;
- const bool offloaded = IS_ENABLED(CONFIG_RCU_NOCB_CPU) &&
- rcu_segcblist_is_offloaded(&rdp->cblist);
+ const bool offloaded = rcu_segcblist_is_offloaded(&rdp->cblist);
raw_lockdep_assert_held_rcu_node(rnp);
@@ -2011,8 +2010,7 @@ static void rcu_gp_cleanup(void)
needgp = true;
}
/* Advance CBs to reduce false positives below. */
- offloaded = IS_ENABLED(CONFIG_RCU_NOCB_CPU) &&
- rcu_segcblist_is_offloaded(&rdp->cblist);
+ offloaded = rcu_segcblist_is_offloaded(&rdp->cblist);
if ((offloaded || !rcu_accelerate_cbs(rnp, rdp)) && needgp) {
WRITE_ONCE(rcu_state.gp_flags, RCU_GP_FLAG_INIT);
WRITE_ONCE(rcu_state.gp_req_activity, jiffies);
@@ -2207,8 +2205,7 @@ rcu_report_qs_rdp(int cpu, struct rcu_data *rdp)
unsigned long flags;
unsigned long mask;
bool needwake = false;
- const bool offloaded = IS_ENABLED(CONFIG_RCU_NOCB_CPU) &&
- rcu_segcblist_is_offloaded(&rdp->cblist);
+ const bool offloaded = rcu_segcblist_is_offloaded(&rdp->cblist);
struct rcu_node *rnp;
rnp = rdp->mynode;
@@ -2375,8 +2372,7 @@ int rcutree_dead_cpu(unsigned int cpu)
static void rcu_do_batch(struct rcu_data *rdp)
{
unsigned long flags;
- const bool offloaded = IS_ENABLED(CONFIG_RCU_NOCB_CPU) &&
- rcu_segcblist_is_offloaded(&rdp->cblist);
+ const bool offloaded = rcu_segcblist_is_offloaded(&rdp->cblist);
struct rcu_head *rhp;
struct rcu_cblist rcl = RCU_CBLIST_INITIALIZER(rcl);
long bl, count;
@@ -2620,8 +2616,7 @@ static __latent_entropy void rcu_core(void)
unsigned long flags;
struct rcu_data *rdp = raw_cpu_ptr(&rcu_data);
struct rcu_node *rnp = rdp->mynode;
- const bool offloaded = IS_ENABLED(CONFIG_RCU_NOCB_CPU) &&
- rcu_segcblist_is_offloaded(&rdp->cblist);
+ const bool offloaded = rcu_segcblist_is_offloaded(&rdp->cblist);
if (cpu_is_offline(smp_processor_id()))
return;
@@ -2919,8 +2914,7 @@ __call_rcu(struct rcu_head *head, rcu_callback_t func)
rcu_segcblist_n_cbs(&rdp->cblist));
/* Go handle any RCU core processing required. */
- if (IS_ENABLED(CONFIG_RCU_NOCB_CPU) &&
- unlikely(rcu_segcblist_is_offloaded(&rdp->cblist))) {
+ if (unlikely(rcu_segcblist_is_offloaded(&rdp->cblist))) {
__call_rcu_nocb_wake(rdp, was_alldone, flags); /* unlocks */
} else {
__call_rcu_core(rdp, head, flags);
@@ -3655,8 +3649,7 @@ static int rcu_pending(int user)
/* Has RCU gone idle with this CPU needing another grace period? */
if (!gp_in_progress && rcu_segcblist_is_enabled(&rdp->cblist) &&
- (!IS_ENABLED(CONFIG_RCU_NOCB_CPU) ||
- !rcu_segcblist_is_offloaded(&rdp->cblist)) &&
+ !rcu_segcblist_is_offloaded(&rdp->cblist) &&
!rcu_segcblist_restempty(&rdp->cblist, RCU_NEXT_READY_TAIL))
return 1;
--
2.28.0
^ permalink raw reply related [flat|nested] 32+ messages in thread
* [RFC PATCH 02/12] rcu: Turn enabled/offload states into a common flag
2020-09-21 12:43 [RFC PATCH 00/12] rcu/nocb: De-offload and re-offload support v2 Frederic Weisbecker
2020-09-21 12:43 ` [RFC PATCH 01/12] rcu: Implement rcu_segcblist_is_offloaded() config dependent Frederic Weisbecker
@ 2020-09-21 12:43 ` Frederic Weisbecker
2020-09-21 12:43 ` [RFC PATCH 03/12] rcu: Provide basic callback offloading state machine bits Frederic Weisbecker
` (9 subsequent siblings)
11 siblings, 0 replies; 32+ messages in thread
From: Frederic Weisbecker @ 2020-09-21 12:43 UTC (permalink / raw)
To: Paul E . McKenney
Cc: LKML, Frederic Weisbecker, Steven Rostedt, Mathieu Desnoyers,
Lai Jiangshan, Joel Fernandes, Josh Triplett
Gather the segcblist properties in a common map to avoid spreading
booleans in the structure. And this prepares for the offloaded state to
be mutable on runtime.
Inspired-by: Paul E. McKenney <paulmck@kernel.org>
Signed-off-by: Frederic Weisbecker <frederic@kernel.org>
Cc: Paul E. McKenney <paulmck@kernel.org>
Cc: Josh Triplett <josh@joshtriplett.org>
Cc: Steven Rostedt <rostedt@goodmis.org>
Cc: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Cc: Lai Jiangshan <jiangshanlai@gmail.com>
Cc: Joel Fernandes <joel@joelfernandes.org>
---
include/linux/rcu_segcblist.h | 6 ++++--
kernel/rcu/rcu_segcblist.c | 6 +++---
kernel/rcu/rcu_segcblist.h | 23 +++++++++++++++++++++--
3 files changed, 28 insertions(+), 7 deletions(-)
diff --git a/include/linux/rcu_segcblist.h b/include/linux/rcu_segcblist.h
index b36afe7b22c9..dca2f39ee67f 100644
--- a/include/linux/rcu_segcblist.h
+++ b/include/linux/rcu_segcblist.h
@@ -63,6 +63,9 @@ struct rcu_cblist {
#define RCU_NEXT_TAIL 3
#define RCU_CBLIST_NSEGS 4
+#define SEGCBLIST_ENABLED BIT(0)
+#define SEGCBLIST_OFFLOADED BIT(1)
+
struct rcu_segcblist {
struct rcu_head *head;
struct rcu_head **tails[RCU_CBLIST_NSEGS];
@@ -72,8 +75,7 @@ struct rcu_segcblist {
#else
long len;
#endif
- u8 enabled;
- u8 offloaded;
+ u8 flags;
};
#define RCU_SEGCBLIST_INITIALIZER(n) \
diff --git a/kernel/rcu/rcu_segcblist.c b/kernel/rcu/rcu_segcblist.c
index 9a0f66133b4b..d131ef8940a0 100644
--- a/kernel/rcu/rcu_segcblist.c
+++ b/kernel/rcu/rcu_segcblist.c
@@ -152,7 +152,7 @@ void rcu_segcblist_init(struct rcu_segcblist *rsclp)
for (i = 0; i < RCU_CBLIST_NSEGS; i++)
rsclp->tails[i] = &rsclp->head;
rcu_segcblist_set_len(rsclp, 0);
- rsclp->enabled = 1;
+ rcu_segcblist_set_flags(rsclp, SEGCBLIST_ENABLED);
}
/*
@@ -163,7 +163,7 @@ void rcu_segcblist_disable(struct rcu_segcblist *rsclp)
{
WARN_ON_ONCE(!rcu_segcblist_empty(rsclp));
WARN_ON_ONCE(rcu_segcblist_n_cbs(rsclp));
- rsclp->enabled = 0;
+ rcu_segcblist_clear_flags(rsclp, SEGCBLIST_ENABLED);
}
/*
@@ -172,7 +172,7 @@ void rcu_segcblist_disable(struct rcu_segcblist *rsclp)
*/
void rcu_segcblist_offload(struct rcu_segcblist *rsclp)
{
- rsclp->offloaded = 1;
+ rcu_segcblist_set_flags(rsclp, SEGCBLIST_OFFLOADED);
}
/*
diff --git a/kernel/rcu/rcu_segcblist.h b/kernel/rcu/rcu_segcblist.h
index 492262bcb591..fc98761e3ee9 100644
--- a/kernel/rcu/rcu_segcblist.h
+++ b/kernel/rcu/rcu_segcblist.h
@@ -50,19 +50,38 @@ static inline long rcu_segcblist_n_cbs(struct rcu_segcblist *rsclp)
#endif
}
+static inline void rcu_segcblist_set_flags(struct rcu_segcblist *rsclp,
+ int flags)
+{
+ rsclp->flags |= flags;
+}
+
+static inline void rcu_segcblist_clear_flags(struct rcu_segcblist *rsclp,
+ int flags)
+{
+ rsclp->flags &= ~flags;
+}
+
+static inline bool rcu_segcblist_test_flags(struct rcu_segcblist *rsclp,
+ int flags)
+{
+ return READ_ONCE(rsclp->flags) & flags;
+}
+
/*
* Is the specified rcu_segcblist enabled, for example, not corresponding
* to an offline CPU?
*/
static inline bool rcu_segcblist_is_enabled(struct rcu_segcblist *rsclp)
{
- return rsclp->enabled;
+ return rcu_segcblist_test_flags(rsclp, SEGCBLIST_ENABLED);
}
/* Is the specified rcu_segcblist offloaded? */
static inline bool rcu_segcblist_is_offloaded(struct rcu_segcblist *rsclp)
{
- return IS_ENABLED(CONFIG_RCU_NOCB_CPU) && rsclp->offloaded;
+ return IS_ENABLED(CONFIG_RCU_NOCB_CPU) &&
+ rcu_segcblist_test_flags(rsclp, SEGCBLIST_OFFLOADED);
}
/*
--
2.28.0
^ permalink raw reply related [flat|nested] 32+ messages in thread
* [RFC PATCH 03/12] rcu: Provide basic callback offloading state machine bits
2020-09-21 12:43 [RFC PATCH 00/12] rcu/nocb: De-offload and re-offload support v2 Frederic Weisbecker
2020-09-21 12:43 ` [RFC PATCH 01/12] rcu: Implement rcu_segcblist_is_offloaded() config dependent Frederic Weisbecker
2020-09-21 12:43 ` [RFC PATCH 02/12] rcu: Turn enabled/offload states into a common flag Frederic Weisbecker
@ 2020-09-21 12:43 ` Frederic Weisbecker
2020-09-21 23:50 ` Paul E. McKenney
2020-09-21 12:43 ` [RFC PATCH 04/12] rcu: De-offloading CB kthread Frederic Weisbecker
` (8 subsequent siblings)
11 siblings, 1 reply; 32+ messages in thread
From: Frederic Weisbecker @ 2020-09-21 12:43 UTC (permalink / raw)
To: Paul E . McKenney
Cc: LKML, Frederic Weisbecker, Steven Rostedt, Mathieu Desnoyers,
Lai Jiangshan, Joel Fernandes, Josh Triplett
We'll need to be able to runtime offload and de-offload the processing
of callback for a given CPU. In order to support a smooth transition
from unlocked local processing (softirq/rcuc) to locked offloaded
processing (rcuop/rcuog) and the reverse, provide the necessary bits and
documentation for the state machine that will carry up all the steps to
enforce correctness while serving callbacks processing all along.
Inspired-by: Paul E. McKenney <paulmck@kernel.org>
Signed-off-by: Frederic Weisbecker <frederic@kernel.org>
Cc: Paul E. McKenney <paulmck@kernel.org>
Cc: Josh Triplett <josh@joshtriplett.org>
Cc: Steven Rostedt <rostedt@goodmis.org>
Cc: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Cc: Lai Jiangshan <jiangshanlai@gmail.com>
Cc: Joel Fernandes <joel@joelfernandes.org>
---
include/linux/rcu_segcblist.h | 115 +++++++++++++++++++++++++++++++++-
kernel/rcu/rcu_segcblist.c | 1 +
kernel/rcu/rcu_segcblist.h | 12 +++-
kernel/rcu/tree.c | 3 +
4 files changed, 128 insertions(+), 3 deletions(-)
diff --git a/include/linux/rcu_segcblist.h b/include/linux/rcu_segcblist.h
index dca2f39ee67f..67f09a912f96 100644
--- a/include/linux/rcu_segcblist.h
+++ b/include/linux/rcu_segcblist.h
@@ -63,8 +63,121 @@ struct rcu_cblist {
#define RCU_NEXT_TAIL 3
#define RCU_CBLIST_NSEGS 4
+
+/*
+ * ==NOCB Offloading state machine==
+ *
+ *
+ * ----------------------------------------------------------------------------
+ * | SEGCBLIST_SOFTIRQ_ONLY |
+ * | |
+ * | Callbacks processed by rcu_core() from softirqs or local |
+ * | rcuc kthread, without holding nocb_lock. |
+ * ----------------------------------------------------------------------------
+ * |
+ * v
+ * ----------------------------------------------------------------------------
+ * | SEGCBLIST_OFFLOADED |
+ * | |
+ * | Callbacks processed by rcu_core() from softirqs or local |
+ * | rcuc kthread, while holding nocb_lock. Waking up CB and GP kthreads, |
+ * | allowing nocb_timer to be armed. |
+ * ----------------------------------------------------------------------------
+ * |
+ * v
+ * -----------------------------------
+ * | |
+ * v v
+ * --------------------------------------- ----------------------------------|
+ * | SEGCBLIST_OFFLOADED | | | SEGCBLIST_OFFLOADED | |
+ * | SEGCBLIST_KTHREAD_CB | | SEGCBLIST_KTHREAD_GP |
+ * | | | |
+ * | | | |
+ * | CB kthread woke up and | | GP kthread woke up and |
+ * | acknowledged SEGCBLIST_OFFLOADED. | | acknowledged SEGCBLIST_OFFLOADED|
+ * | Processes callbacks concurrently | | |
+ * | with rcu_core(), holding | | |
+ * | nocb_lock. | | |
+ * --------------------------------------- -----------------------------------
+ * | |
+ * -----------------------------------
+ * |
+ * v
+ * |--------------------------------------------------------------------------|
+ * | SEGCBLIST_OFFLOADED | |
+ * | SEGCBLIST_KTHREAD_CB | |
+ * | SEGCBLIST_KTHREAD_GP |
+ * | |
+ * | Kthreads handle callbacks holding nocb_lock, local rcu_core() stops |
+ * | handling callbacks. |
+ * ----------------------------------------------------------------------------
+ */
+
+
+
+/*
+ * ==NOCB De-Offloading state machine==
+ *
+ *
+ * |--------------------------------------------------------------------------|
+ * | SEGCBLIST_OFFLOADED | |
+ * | SEGCBLIST_KTHREAD_CB | |
+ * | SEGCBLIST_KTHREAD_GP |
+ * | |
+ * | CB/GP kthreads handle callbacks holding nocb_lock, local rcu_core() |
+ * | ignores callbacks. |
+ * ----------------------------------------------------------------------------
+ * |
+ * v
+ * |--------------------------------------------------------------------------|
+ * | SEGCBLIST_KTHREAD_CB | |
+ * | SEGCBLIST_KTHREAD_GP |
+ * | |
+ * | CB/GP kthreads and local rcu_core() handle callbacks concurrently |
+ * | holding nocb_lock. Wake up CB and GP kthreads if necessary. |
+ * ----------------------------------------------------------------------------
+ * |
+ * v
+ * -----------------------------------
+ * | |
+ * v v
+ * ---------------------------------------------------------------------------|
+ * | |
+ * | SEGCBLIST_KTHREAD_CB | SEGCBLIST_KTHREAD_GP |
+ * | | |
+ * | GP kthread woke up and | CB kthread woke up and |
+ * | acknowledged the fact that | acknowledged the fact that |
+ * | SEGCBLIST_OFFLOADED got cleared. | SEGCBLIST_OFFLOADED got cleared. |
+ * | | The CB kthread goes to sleep |
+ * | The callbacks from the target CPU | until it ever gets re-offloaded. |
+ * | will be ignored from the GP kthread | |
+ * | loop. | |
+ * ----------------------------------------------------------------------------
+ * | |
+ * -----------------------------------
+ * |
+ * v
+ * ----------------------------------------------------------------------------
+ * | 0 |
+ * | |
+ * | Callbacks processed by rcu_core() from softirqs or local |
+ * | rcuc kthread, while holding nocb_lock. Forbid nocb_timer to be armed. |
+ * | Flush pending nocb_timer. Flush nocb bypass callbacks. |
+ * ----------------------------------------------------------------------------
+ * |
+ * v
+ * ----------------------------------------------------------------------------
+ * | SEGCBLIST_SOFTIRQ_ONLY |
+ * | |
+ * | Callbacks processed by rcu_core() from softirqs or local |
+ * | rcuc kthread, without holding nocb_lock. |
+ * ----------------------------------------------------------------------------
+ */
#define SEGCBLIST_ENABLED BIT(0)
-#define SEGCBLIST_OFFLOADED BIT(1)
+#define SEGCBLIST_SOFTIRQ_ONLY BIT(1)
+#define SEGCBLIST_KTHREAD_CB BIT(2)
+#define SEGCBLIST_KTHREAD_GP BIT(3)
+#define SEGCBLIST_OFFLOADED BIT(4)
struct rcu_segcblist {
struct rcu_head *head;
diff --git a/kernel/rcu/rcu_segcblist.c b/kernel/rcu/rcu_segcblist.c
index d131ef8940a0..31cc27ee98d8 100644
--- a/kernel/rcu/rcu_segcblist.c
+++ b/kernel/rcu/rcu_segcblist.c
@@ -172,6 +172,7 @@ void rcu_segcblist_disable(struct rcu_segcblist *rsclp)
*/
void rcu_segcblist_offload(struct rcu_segcblist *rsclp)
{
+ rcu_segcblist_clear_flags(rsclp, SEGCBLIST_SOFTIRQ_ONLY);
rcu_segcblist_set_flags(rsclp, SEGCBLIST_OFFLOADED);
}
diff --git a/kernel/rcu/rcu_segcblist.h b/kernel/rcu/rcu_segcblist.h
index fc98761e3ee9..575896a2518b 100644
--- a/kernel/rcu/rcu_segcblist.h
+++ b/kernel/rcu/rcu_segcblist.h
@@ -80,8 +80,16 @@ static inline bool rcu_segcblist_is_enabled(struct rcu_segcblist *rsclp)
/* Is the specified rcu_segcblist offloaded? */
static inline bool rcu_segcblist_is_offloaded(struct rcu_segcblist *rsclp)
{
- return IS_ENABLED(CONFIG_RCU_NOCB_CPU) &&
- rcu_segcblist_test_flags(rsclp, SEGCBLIST_OFFLOADED);
+ if (IS_ENABLED(CONFIG_RCU_NOCB_CPU)) {
+ /*
+ * Complete de-offloading happens only when SEGCBLIST_SOFTIRQ_ONLY
+ * is set.
+ */
+ if (!rcu_segcblist_test_flags(rsclp, SEGCBLIST_SOFTIRQ_ONLY))
+ return true;
+ }
+
+ return false;
}
/*
diff --git a/kernel/rcu/tree.c b/kernel/rcu/tree.c
index c0286ce8fc03..b4292489db0c 100644
--- a/kernel/rcu/tree.c
+++ b/kernel/rcu/tree.c
@@ -96,6 +96,9 @@ static DEFINE_PER_CPU_SHARED_ALIGNED(struct rcu_data, rcu_data) = {
.dynticks_nesting = 1,
.dynticks_nmi_nesting = DYNTICK_IRQ_NONIDLE,
.dynticks = ATOMIC_INIT(RCU_DYNTICK_CTRL_CTR),
+#ifdef CONFIG_RCU_NOCB_CPU
+ .cblist.flags = SEGCBLIST_SOFTIRQ_ONLY,
+#endif
};
static struct rcu_state rcu_state = {
.level = { &rcu_state.node[0] },
--
2.28.0
^ permalink raw reply related [flat|nested] 32+ messages in thread
* [RFC PATCH 04/12] rcu: De-offloading CB kthread
2020-09-21 12:43 [RFC PATCH 00/12] rcu/nocb: De-offload and re-offload support v2 Frederic Weisbecker
` (2 preceding siblings ...)
2020-09-21 12:43 ` [RFC PATCH 03/12] rcu: Provide basic callback offloading state machine bits Frederic Weisbecker
@ 2020-09-21 12:43 ` Frederic Weisbecker
2020-09-22 0:07 ` Paul E. McKenney
2020-09-21 12:43 ` [RFC PATCH 05/12] rcu: De-offloading GP kthread Frederic Weisbecker
` (7 subsequent siblings)
11 siblings, 1 reply; 32+ messages in thread
From: Frederic Weisbecker @ 2020-09-21 12:43 UTC (permalink / raw)
To: Paul E . McKenney
Cc: LKML, Frederic Weisbecker, Steven Rostedt, Mathieu Desnoyers,
Lai Jiangshan, Joel Fernandes, Josh Triplett
In order to de-offload the callbacks processing of an rdp, we must clear
SEGCBLIST_OFFLOAD and notify the CB kthread so that it clears its own
bit flag and goes to sleep to stop handling callbacks. The GP kthread
will also be notified the same way. Whoever acknowledges and clears its
own bit last must notify the de-offloading worker so that it can resume
the de-offloading while being sure that callbacks won't be handled
remotely anymore.
Inspired-by: Paul E. McKenney <paulmck@kernel.org>
Signed-off-by: Frederic Weisbecker <frederic@kernel.org>
Cc: Paul E. McKenney <paulmck@kernel.org>
Cc: Josh Triplett <josh@joshtriplett.org>
Cc: Steven Rostedt <rostedt@goodmis.org>
Cc: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Cc: Lai Jiangshan <jiangshanlai@gmail.com>
Cc: Joel Fernandes <joel@joelfernandes.org>
---
include/linux/rcupdate.h | 2 +
kernel/rcu/rcu_segcblist.c | 10 ++-
kernel/rcu/rcu_segcblist.h | 2 +-
kernel/rcu/tree.h | 1 +
kernel/rcu/tree_plugin.h | 124 ++++++++++++++++++++++++++++++++-----
5 files changed, 119 insertions(+), 20 deletions(-)
diff --git a/include/linux/rcupdate.h b/include/linux/rcupdate.h
index d15d46db61f7..44ddee921441 100644
--- a/include/linux/rcupdate.h
+++ b/include/linux/rcupdate.h
@@ -97,8 +97,10 @@ static inline void rcu_user_exit(void) { }
#ifdef CONFIG_RCU_NOCB_CPU
void rcu_init_nohz(void);
+void rcu_nocb_cpu_deoffload(int cpu);
#else /* #ifdef CONFIG_RCU_NOCB_CPU */
static inline void rcu_init_nohz(void) { }
+static inline void rcu_nocb_cpu_deoffload(int cpu) { }
#endif /* #else #ifdef CONFIG_RCU_NOCB_CPU */
/**
diff --git a/kernel/rcu/rcu_segcblist.c b/kernel/rcu/rcu_segcblist.c
index 31cc27ee98d8..b70032d77025 100644
--- a/kernel/rcu/rcu_segcblist.c
+++ b/kernel/rcu/rcu_segcblist.c
@@ -170,10 +170,14 @@ void rcu_segcblist_disable(struct rcu_segcblist *rsclp)
* Mark the specified rcu_segcblist structure as offloaded. This
* structure must be empty.
*/
-void rcu_segcblist_offload(struct rcu_segcblist *rsclp)
+void rcu_segcblist_offload(struct rcu_segcblist *rsclp, bool offload)
{
- rcu_segcblist_clear_flags(rsclp, SEGCBLIST_SOFTIRQ_ONLY);
- rcu_segcblist_set_flags(rsclp, SEGCBLIST_OFFLOADED);
+ if (offload) {
+ rcu_segcblist_clear_flags(rsclp, SEGCBLIST_SOFTIRQ_ONLY);
+ rcu_segcblist_set_flags(rsclp, SEGCBLIST_OFFLOADED);
+ } else {
+ rcu_segcblist_clear_flags(rsclp, SEGCBLIST_OFFLOADED);
+ }
}
/*
diff --git a/kernel/rcu/rcu_segcblist.h b/kernel/rcu/rcu_segcblist.h
index 575896a2518b..00ebeb8d39b7 100644
--- a/kernel/rcu/rcu_segcblist.h
+++ b/kernel/rcu/rcu_segcblist.h
@@ -105,7 +105,7 @@ static inline bool rcu_segcblist_restempty(struct rcu_segcblist *rsclp, int seg)
void rcu_segcblist_inc_len(struct rcu_segcblist *rsclp);
void rcu_segcblist_init(struct rcu_segcblist *rsclp);
void rcu_segcblist_disable(struct rcu_segcblist *rsclp);
-void rcu_segcblist_offload(struct rcu_segcblist *rsclp);
+void rcu_segcblist_offload(struct rcu_segcblist *rsclp, bool offload);
bool rcu_segcblist_ready_cbs(struct rcu_segcblist *rsclp);
bool rcu_segcblist_pend_cbs(struct rcu_segcblist *rsclp);
struct rcu_head *rcu_segcblist_first_cb(struct rcu_segcblist *rsclp);
diff --git a/kernel/rcu/tree.h b/kernel/rcu/tree.h
index c96ae351688b..5b37f7103b0d 100644
--- a/kernel/rcu/tree.h
+++ b/kernel/rcu/tree.h
@@ -198,6 +198,7 @@ struct rcu_data {
/* 5) Callback offloading. */
#ifdef CONFIG_RCU_NOCB_CPU
struct swait_queue_head nocb_cb_wq; /* For nocb kthreads to sleep on. */
+ struct swait_queue_head nocb_state_wq; /* For offloading state changes */
struct task_struct *nocb_gp_kthread;
raw_spinlock_t nocb_lock; /* Guard following pair of fields. */
atomic_t nocb_lock_contended; /* Contention experienced. */
diff --git a/kernel/rcu/tree_plugin.h b/kernel/rcu/tree_plugin.h
index 982fc5be5269..3ea43c75c7a3 100644
--- a/kernel/rcu/tree_plugin.h
+++ b/kernel/rcu/tree_plugin.h
@@ -2052,6 +2052,17 @@ static int rcu_nocb_gp_kthread(void *arg)
return 0;
}
+static inline bool nocb_cb_can_run(struct rcu_data *rdp)
+{
+ u8 flags = SEGCBLIST_OFFLOADED | SEGCBLIST_KTHREAD_CB;
+ return rcu_segcblist_test_flags(&rdp->cblist, flags);
+}
+
+static inline bool nocb_cb_wait_cond(struct rcu_data *rdp)
+{
+ return nocb_cb_can_run(rdp) && !READ_ONCE(rdp->nocb_cb_sleep);
+}
+
/*
* Invoke any ready callbacks from the corresponding no-CBs CPU,
* then, if there are no more, wait for more to appear.
@@ -2061,7 +2072,9 @@ static void nocb_cb_wait(struct rcu_data *rdp)
unsigned long cur_gp_seq;
unsigned long flags;
bool needwake_gp = false;
+ bool needwake_state = false;
struct rcu_node *rnp = rdp->mynode;
+ struct rcu_segcblist *cblist = &rdp->cblist;
local_irq_save(flags);
rcu_momentary_dyntick_idle();
@@ -2071,32 +2084,50 @@ static void nocb_cb_wait(struct rcu_data *rdp)
local_bh_enable();
lockdep_assert_irqs_enabled();
rcu_nocb_lock_irqsave(rdp, flags);
- if (rcu_segcblist_nextgp(&rdp->cblist, &cur_gp_seq) &&
+ if (rcu_segcblist_nextgp(cblist, &cur_gp_seq) &&
rcu_seq_done(&rnp->gp_seq, cur_gp_seq) &&
raw_spin_trylock_rcu_node(rnp)) { /* irqs already disabled. */
needwake_gp = rcu_advance_cbs(rdp->mynode, rdp);
raw_spin_unlock_rcu_node(rnp); /* irqs remain disabled. */
}
- if (rcu_segcblist_ready_cbs(&rdp->cblist)) {
- rcu_nocb_unlock_irqrestore(rdp, flags);
- if (needwake_gp)
- rcu_gp_kthread_wake();
- return;
- }
- trace_rcu_nocb_wake(rcu_state.name, rdp->cpu, TPS("CBSleep"));
WRITE_ONCE(rdp->nocb_cb_sleep, true);
+
+ if (rcu_segcblist_test_flags(cblist, SEGCBLIST_OFFLOADED)) {
+ if (rcu_segcblist_ready_cbs(cblist))
+ WRITE_ONCE(rdp->nocb_cb_sleep, false);
+ } else {
+ /*
+ * De-offloading. Clear our flag and notify the de-offload worker.
+ * We won't touch the callbacks and keep sleeping until we ever
+ * get re-offloaded.
+ */
+ WARN_ON_ONCE(!rcu_segcblist_test_flags(cblist, SEGCBLIST_KTHREAD_CB));
+ rcu_segcblist_clear_flags(cblist, SEGCBLIST_KTHREAD_CB);
+ if (!rcu_segcblist_test_flags(cblist, SEGCBLIST_KTHREAD_GP))
+ needwake_state = true;
+ }
+
+ if (rdp->nocb_cb_sleep)
+ trace_rcu_nocb_wake(rcu_state.name, rdp->cpu, TPS("CBSleep"));
+
rcu_nocb_unlock_irqrestore(rdp, flags);
if (needwake_gp)
rcu_gp_kthread_wake();
- swait_event_interruptible_exclusive(rdp->nocb_cb_wq,
- !READ_ONCE(rdp->nocb_cb_sleep));
- if (!smp_load_acquire(&rdp->nocb_cb_sleep)) { /* VVV */
+
+ if (needwake_state)
+ swake_up_one(&rdp->nocb_state_wq);
+
+ do {
+ swait_event_interruptible_exclusive(rdp->nocb_cb_wq,
+ nocb_cb_wait_cond(rdp));
+
/* ^^^ Ensure CB invocation follows _sleep test. */
- return;
- }
- WARN_ON(signal_pending(current));
- trace_rcu_nocb_wake(rcu_state.name, rdp->cpu, TPS("WokeEmpty"));
+ if (smp_load_acquire(&rdp->nocb_cb_sleep)) {
+ WARN_ON(signal_pending(current));
+ trace_rcu_nocb_wake(rcu_state.name, rdp->cpu, TPS("WokeEmpty"));
+ }
+ } while (!nocb_cb_can_run(rdp));
}
/*
@@ -2158,6 +2189,65 @@ static void do_nocb_deferred_wakeup(struct rcu_data *rdp)
do_nocb_deferred_wakeup_common(rdp);
}
+static void __rcu_nocb_rdp_deoffload(struct rcu_data *rdp)
+{
+ unsigned long flags;
+ struct rcu_node *rnp = rdp->mynode;
+ struct rcu_segcblist *cblist = &rdp->cblist;
+ bool wake_cb = false;
+
+ printk("De-offloading %d\n", rdp->cpu);
+
+ rcu_nocb_lock_irqsave(rdp, flags);
+ raw_spin_lock_rcu_node(rnp);
+ rcu_segcblist_offload(cblist, false);
+ raw_spin_unlock_rcu_node(rnp);
+
+ if (rdp->nocb_cb_sleep) {
+ rdp->nocb_cb_sleep = false;
+ wake_cb = true;
+ }
+ rcu_nocb_unlock_irqrestore(rdp, flags);
+
+ if (wake_cb)
+ swake_up_one(&rdp->nocb_cb_wq);
+
+ swait_event_exclusive(rdp->nocb_state_wq,
+ !rcu_segcblist_test_flags(cblist, SEGCBLIST_KTHREAD_CB));
+}
+
+static long rcu_nocb_rdp_deoffload(void *arg)
+{
+ struct rcu_data *rdp = arg;
+
+ WARN_ON_ONCE(rdp->cpu != raw_smp_processor_id());
+ __rcu_nocb_rdp_deoffload(rdp);
+
+ return 0;
+}
+
+void rcu_nocb_cpu_deoffload(int cpu)
+{
+ struct rcu_data *rdp = per_cpu_ptr(&rcu_data, cpu);
+
+ if (rdp == rdp->nocb_gp_rdp) {
+ pr_info("Can't deoffload an rdp GP leader (yet)\n");
+ return;
+ }
+ mutex_lock(&rcu_state.barrier_mutex);
+ cpus_read_lock();
+ if (rcu_segcblist_is_offloaded(&rdp->cblist)) {
+ if (cpu_online(cpu)) {
+ work_on_cpu(cpu, rcu_nocb_rdp_deoffload, rdp);
+ } else {
+ __rcu_nocb_rdp_deoffload(rdp);
+ }
+ cpumask_clear_cpu(cpu, rcu_nocb_mask);
+ }
+ cpus_read_unlock();
+ mutex_unlock(&rcu_state.barrier_mutex);
+}
+
void __init rcu_init_nohz(void)
{
int cpu;
@@ -2200,7 +2290,8 @@ void __init rcu_init_nohz(void)
rdp = per_cpu_ptr(&rcu_data, cpu);
if (rcu_segcblist_empty(&rdp->cblist))
rcu_segcblist_init(&rdp->cblist);
- rcu_segcblist_offload(&rdp->cblist);
+ rcu_segcblist_offload(&rdp->cblist, true);
+ rcu_segcblist_set_flags(&rdp->cblist, SEGCBLIST_KTHREAD_CB);
}
rcu_organize_nocb_kthreads();
}
@@ -2210,6 +2301,7 @@ static void __init rcu_boot_init_nocb_percpu_data(struct rcu_data *rdp)
{
init_swait_queue_head(&rdp->nocb_cb_wq);
init_swait_queue_head(&rdp->nocb_gp_wq);
+ init_swait_queue_head(&rdp->nocb_state_wq);
raw_spin_lock_init(&rdp->nocb_lock);
raw_spin_lock_init(&rdp->nocb_bypass_lock);
raw_spin_lock_init(&rdp->nocb_gp_lock);
--
2.28.0
^ permalink raw reply related [flat|nested] 32+ messages in thread
* [RFC PATCH 05/12] rcu: De-offloading GP kthread
2020-09-21 12:43 [RFC PATCH 00/12] rcu/nocb: De-offload and re-offload support v2 Frederic Weisbecker
` (3 preceding siblings ...)
2020-09-21 12:43 ` [RFC PATCH 04/12] rcu: De-offloading CB kthread Frederic Weisbecker
@ 2020-09-21 12:43 ` Frederic Weisbecker
2020-09-22 0:10 ` Paul E. McKenney
2020-09-21 12:43 ` [RFC PATCH 06/12] rcu: Re-offload support Frederic Weisbecker
` (6 subsequent siblings)
11 siblings, 1 reply; 32+ messages in thread
From: Frederic Weisbecker @ 2020-09-21 12:43 UTC (permalink / raw)
To: Paul E . McKenney
Cc: LKML, Frederic Weisbecker, Steven Rostedt, Mathieu Desnoyers,
Lai Jiangshan, Joel Fernandes, Josh Triplett
In order to de-offload the callbacks processing of an rdp, we must clear
SEGCBLIST_OFFLOAD and notify the GP kthread so that it clears its own
bit flag and ignore the target rdp from its loop. The CB kthread
is also notified the same way. Whoever acknowledges and clears its
own bit last must notify the de-offloading worker so that it can resume
the de-offloading while being sure that callbacks won't be handled
remotely anymore.
Inspired-by: Paul E. McKenney <paulmck@kernel.org>
Signed-off-by: Frederic Weisbecker <frederic@kernel.org>
Cc: Paul E. McKenney <paulmck@kernel.org>
Cc: Josh Triplett <josh@joshtriplett.org>
Cc: Steven Rostedt <rostedt@goodmis.org>
Cc: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Cc: Lai Jiangshan <jiangshanlai@gmail.com>
Cc: Joel Fernandes <joel@joelfernandes.org>
---
kernel/rcu/tree_plugin.h | 53 ++++++++++++++++++++++++++++++++++++++--
1 file changed, 51 insertions(+), 2 deletions(-)
diff --git a/kernel/rcu/tree_plugin.h b/kernel/rcu/tree_plugin.h
index 3ea43c75c7a3..e4e7e7c4bfb6 100644
--- a/kernel/rcu/tree_plugin.h
+++ b/kernel/rcu/tree_plugin.h
@@ -1900,6 +1900,33 @@ static void do_nocb_bypass_wakeup_timer(struct timer_list *t)
__call_rcu_nocb_wake(rdp, true, flags);
}
+static inline bool nocb_gp_enabled_cb(struct rcu_data *rdp)
+{
+ u8 flags = SEGCBLIST_OFFLOADED | SEGCBLIST_KTHREAD_GP;
+
+ return rcu_segcblist_test_flags(&rdp->cblist, flags);
+}
+
+static inline bool nocb_gp_update_state(struct rcu_data *rdp, bool *needwake_state)
+{
+ struct rcu_segcblist *cblist = &rdp->cblist;
+
+ if (rcu_segcblist_test_flags(cblist, SEGCBLIST_OFFLOADED)) {
+ return true;
+ } else {
+ /*
+ * De-offloading. Clear our flag and notify the de-offload worker.
+ * We will ignore this rdp until it ever gets re-offloaded.
+ */
+ WARN_ON_ONCE(!rcu_segcblist_test_flags(cblist, SEGCBLIST_KTHREAD_GP));
+ rcu_segcblist_clear_flags(cblist, SEGCBLIST_KTHREAD_GP);
+ if (!rcu_segcblist_test_flags(cblist, SEGCBLIST_KTHREAD_CB))
+ *needwake_state = true;
+ return false;
+ }
+}
+
+
/*
* No-CBs GP kthreads come here to wait for additional callbacks to show up
* or for grace periods to end.
@@ -1927,8 +1954,17 @@ static void nocb_gp_wait(struct rcu_data *my_rdp)
* and the global grace-period kthread are awakened if needed.
*/
for (rdp = my_rdp; rdp; rdp = rdp->nocb_next_cb_rdp) {
+ bool needwake_state = false;
+ if (!nocb_gp_enabled_cb(rdp))
+ continue;
trace_rcu_nocb_wake(rcu_state.name, rdp->cpu, TPS("Check"));
rcu_nocb_lock_irqsave(rdp, flags);
+ if (!nocb_gp_update_state(rdp, &needwake_state)) {
+ rcu_nocb_unlock_irqrestore(rdp, flags);
+ if (needwake_state)
+ swake_up_one(&rdp->nocb_state_wq);
+ continue;
+ }
bypass_ncbs = rcu_cblist_n_cbs(&rdp->nocb_bypass);
if (bypass_ncbs &&
(time_after(j, READ_ONCE(rdp->nocb_bypass_first) + 1) ||
@@ -2194,7 +2230,8 @@ static void __rcu_nocb_rdp_deoffload(struct rcu_data *rdp)
unsigned long flags;
struct rcu_node *rnp = rdp->mynode;
struct rcu_segcblist *cblist = &rdp->cblist;
- bool wake_cb = false;
+ struct rcu_data *rdp_gp = rdp->nocb_gp_rdp;
+ bool wake_cb = false, wake_gp = false;
printk("De-offloading %d\n", rdp->cpu);
@@ -2212,8 +2249,19 @@ static void __rcu_nocb_rdp_deoffload(struct rcu_data *rdp)
if (wake_cb)
swake_up_one(&rdp->nocb_cb_wq);
+ raw_spin_lock_irqsave(&rdp_gp->nocb_gp_lock, flags);
+ if (rdp_gp->nocb_gp_sleep) {
+ rdp_gp->nocb_gp_sleep = false;
+ wake_gp = true;
+ }
+ raw_spin_unlock_irqrestore(&rdp_gp->nocb_gp_lock, flags);
+
+ if (wake_gp)
+ wake_up_process(rdp_gp->nocb_gp_kthread);
+
swait_event_exclusive(rdp->nocb_state_wq,
- !rcu_segcblist_test_flags(cblist, SEGCBLIST_KTHREAD_CB));
+ !rcu_segcblist_test_flags(cblist, SEGCBLIST_KTHREAD_CB |
+ SEGCBLIST_KTHREAD_GP));
}
static long rcu_nocb_rdp_deoffload(void *arg)
@@ -2292,6 +2340,7 @@ void __init rcu_init_nohz(void)
rcu_segcblist_init(&rdp->cblist);
rcu_segcblist_offload(&rdp->cblist, true);
rcu_segcblist_set_flags(&rdp->cblist, SEGCBLIST_KTHREAD_CB);
+ rcu_segcblist_set_flags(&rdp->cblist, SEGCBLIST_KTHREAD_GP);
}
rcu_organize_nocb_kthreads();
}
--
2.28.0
^ permalink raw reply related [flat|nested] 32+ messages in thread
* [RFC PATCH 06/12] rcu: Re-offload support
2020-09-21 12:43 [RFC PATCH 00/12] rcu/nocb: De-offload and re-offload support v2 Frederic Weisbecker
` (4 preceding siblings ...)
2020-09-21 12:43 ` [RFC PATCH 05/12] rcu: De-offloading GP kthread Frederic Weisbecker
@ 2020-09-21 12:43 ` Frederic Weisbecker
2020-09-21 12:43 ` [RFC PATCH 07/12] rcu: Shutdown nocb timer on de-offloading Frederic Weisbecker
` (5 subsequent siblings)
11 siblings, 0 replies; 32+ messages in thread
From: Frederic Weisbecker @ 2020-09-21 12:43 UTC (permalink / raw)
To: Paul E . McKenney
Cc: LKML, Frederic Weisbecker, Steven Rostedt, Mathieu Desnoyers,
Lai Jiangshan, Joel Fernandes, Josh Triplett
In order to re-offload the callbacks processing of an rdp, we must clear
SEGCBLIST_SOFTIRQ_ONLY, set SEGCBLIST_OFFLOADED and notify the CB and GP
kthreads so that they both set their own bit flag and start processing
the callbacks remotely. The re-offloading worker is then notified that
it can stop processing the callbacks locally.
Ordering must be carefully enforced so that the callbacks that used to
be processed locally without locking must have their latest updates
visible by the time they get processed by the kthreads.
Inspired-by: Paul E. McKenney <paulmck@kernel.org>
Signed-off-by: Frederic Weisbecker <frederic@kernel.org>
Cc: Paul E. McKenney <paulmck@kernel.org>
Cc: Josh Triplett <josh@joshtriplett.org>
Cc: Steven Rostedt <rostedt@goodmis.org>
Cc: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Cc: Lai Jiangshan <jiangshanlai@gmail.com>
Cc: Joel Fernandes <joel@joelfernandes.org>
---
include/linux/rcupdate.h | 2 +
kernel/rcu/tree_plugin.h | 123 +++++++++++++++++++++++++++++++++++----
2 files changed, 113 insertions(+), 12 deletions(-)
diff --git a/include/linux/rcupdate.h b/include/linux/rcupdate.h
index 44ddee921441..4749f743398a 100644
--- a/include/linux/rcupdate.h
+++ b/include/linux/rcupdate.h
@@ -97,9 +97,11 @@ static inline void rcu_user_exit(void) { }
#ifdef CONFIG_RCU_NOCB_CPU
void rcu_init_nohz(void);
+void rcu_nocb_cpu_offload(int cpu);
void rcu_nocb_cpu_deoffload(int cpu);
#else /* #ifdef CONFIG_RCU_NOCB_CPU */
static inline void rcu_init_nohz(void) { }
+static inline void rcu_nocb_cpu_offload(int cpu) { }
static inline void rcu_nocb_cpu_deoffload(int cpu) { }
#endif /* #else #ifdef CONFIG_RCU_NOCB_CPU */
diff --git a/kernel/rcu/tree_plugin.h b/kernel/rcu/tree_plugin.h
index e4e7e7c4bfb6..eceef6dade9a 100644
--- a/kernel/rcu/tree_plugin.h
+++ b/kernel/rcu/tree_plugin.h
@@ -1900,6 +1900,20 @@ static void do_nocb_bypass_wakeup_timer(struct timer_list *t)
__call_rcu_nocb_wake(rdp, true, flags);
}
+/*
+ * Check if we ignore this rdp.
+ *
+ * We check that without holding the nocb lock but
+ * we make sure not to miss a freshly offloaded rdp
+ * with the current ordering:
+ *
+ * rdp_offload_toggle() nocb_gp_enabled_cb()
+ * ------------------------- ----------------------------
+ * WRITE flags LOCK nocb_gp_lock
+ * LOCK nocb_gp_lock READ/WRITE nocb_gp_sleep
+ * READ/WRITE nocb_gp_sleep UNLOCK nocb_gp_lock
+ * UNLOCK nocb_gp_lock READ flags
+ */
static inline bool nocb_gp_enabled_cb(struct rcu_data *rdp)
{
u8 flags = SEGCBLIST_OFFLOADED | SEGCBLIST_KTHREAD_GP;
@@ -1912,6 +1926,11 @@ static inline bool nocb_gp_update_state(struct rcu_data *rdp, bool *needwake_sta
struct rcu_segcblist *cblist = &rdp->cblist;
if (rcu_segcblist_test_flags(cblist, SEGCBLIST_OFFLOADED)) {
+ if (!rcu_segcblist_test_flags(cblist, SEGCBLIST_KTHREAD_GP)) {
+ rcu_segcblist_set_flags(cblist, SEGCBLIST_KTHREAD_GP);
+ if (rcu_segcblist_test_flags(cblist, SEGCBLIST_KTHREAD_CB))
+ *needwake_state = true;
+ }
return true;
} else {
/*
@@ -1974,6 +1993,8 @@ static void nocb_gp_wait(struct rcu_data *my_rdp)
bypass_ncbs = rcu_cblist_n_cbs(&rdp->nocb_bypass);
} else if (!bypass_ncbs && rcu_segcblist_empty(&rdp->cblist)) {
rcu_nocb_unlock_irqrestore(rdp, flags);
+ if (needwake_state)
+ swake_up_one(&rdp->nocb_state_wq);
continue; /* No callbacks here, try next. */
}
if (bypass_ncbs) {
@@ -2025,6 +2046,8 @@ static void nocb_gp_wait(struct rcu_data *my_rdp)
}
if (needwake_gp)
rcu_gp_kthread_wake();
+ if (needwake_state)
+ swake_up_one(&rdp->nocb_state_wq);
}
my_rdp->nocb_gp_bypass = bypass;
@@ -2130,6 +2153,11 @@ static void nocb_cb_wait(struct rcu_data *rdp)
WRITE_ONCE(rdp->nocb_cb_sleep, true);
if (rcu_segcblist_test_flags(cblist, SEGCBLIST_OFFLOADED)) {
+ if (!rcu_segcblist_test_flags(cblist, SEGCBLIST_KTHREAD_CB)) {
+ rcu_segcblist_set_flags(cblist, SEGCBLIST_KTHREAD_CB);
+ if (rcu_segcblist_test_flags(cblist, SEGCBLIST_KTHREAD_GP))
+ needwake_state = true;
+ }
if (rcu_segcblist_ready_cbs(cblist))
WRITE_ONCE(rdp->nocb_cb_sleep, false);
} else {
@@ -2225,29 +2253,28 @@ static void do_nocb_deferred_wakeup(struct rcu_data *rdp)
do_nocb_deferred_wakeup_common(rdp);
}
-static void __rcu_nocb_rdp_deoffload(struct rcu_data *rdp)
+static void rdp_offload_toggle(struct rcu_data *rdp,
+ bool offload, unsigned long flags)
+ __releases(rdp->nocb_lock)
{
- unsigned long flags;
struct rcu_node *rnp = rdp->mynode;
struct rcu_segcblist *cblist = &rdp->cblist;
struct rcu_data *rdp_gp = rdp->nocb_gp_rdp;
- bool wake_cb = false, wake_gp = false;
+ bool wake_gp = false;
- printk("De-offloading %d\n", rdp->cpu);
-
- rcu_nocb_lock_irqsave(rdp, flags);
raw_spin_lock_rcu_node(rnp);
- rcu_segcblist_offload(cblist, false);
+ rcu_segcblist_offload(cblist, offload);
raw_spin_unlock_rcu_node(rnp);
- if (rdp->nocb_cb_sleep) {
+ if (rdp->nocb_cb_sleep)
rdp->nocb_cb_sleep = false;
- wake_cb = true;
- }
rcu_nocb_unlock_irqrestore(rdp, flags);
- if (wake_cb)
- swake_up_one(&rdp->nocb_cb_wq);
+ /*
+ * Ignore former value of nocb_cb_sleep and force wake up as it could
+ * have been spuriously set to false already.
+ */
+ swake_up_one(&rdp->nocb_cb_wq);
raw_spin_lock_irqsave(&rdp_gp->nocb_gp_lock, flags);
if (rdp_gp->nocb_gp_sleep) {
@@ -2258,7 +2285,17 @@ static void __rcu_nocb_rdp_deoffload(struct rcu_data *rdp)
if (wake_gp)
wake_up_process(rdp_gp->nocb_gp_kthread);
+}
+static void __rcu_nocb_rdp_deoffload(struct rcu_data *rdp)
+{
+ struct rcu_segcblist *cblist = &rdp->cblist;
+ unsigned long flags;
+
+ printk("De-offloading %d\n", rdp->cpu);
+
+ rcu_nocb_lock_irqsave(rdp, flags);
+ rdp_offload_toggle(rdp, false, flags);
swait_event_exclusive(rdp->nocb_state_wq,
!rcu_segcblist_test_flags(cblist, SEGCBLIST_KTHREAD_CB |
SEGCBLIST_KTHREAD_GP));
@@ -2296,6 +2333,68 @@ void rcu_nocb_cpu_deoffload(int cpu)
mutex_unlock(&rcu_state.barrier_mutex);
}
+static void __rcu_nocb_rdp_offload(struct rcu_data *rdp)
+{
+ struct rcu_segcblist *cblist = &rdp->cblist;
+ unsigned long flags;
+
+ printk("Offloading %d\n", rdp->cpu);
+ /*
+ * Can't use rcu_nocb_lock_irqsave() while we are in
+ * SEGCBLIST_SOFTIRQ_ONLY mode.
+ */
+ raw_spin_lock_irqsave(&rdp->nocb_lock, flags);
+ /*
+ * We didn't take the nocb lock while working on the
+ * rdp->cblist in SEGCBLIST_SOFTIRQ_ONLY mode.
+ * Every modifications that have been done previously on
+ * rdp->cblist must be visible remotely by the nocb kthreads
+ * upon wake up after reading the cblist flags.
+ *
+ * The layout against nocb_lock enforces that ordering:
+ *
+ * __rcu_nocb_rdp_offload() nocb_cb_wait()/nocb_gp_wait()
+ * ------------------------- ----------------------------
+ * WRITE callbacks rcu_nocb_lock()
+ * rcu_nocb_lock() READ flags
+ * WRITE flags READ callbacks
+ * rcu_nocb_unlock() rcu_nocb_unlock()
+ */
+ rdp_offload_toggle(rdp, true, flags);
+ swait_event_exclusive(rdp->nocb_state_wq,
+ rcu_segcblist_test_flags(cblist, SEGCBLIST_KTHREAD_CB) &&
+ rcu_segcblist_test_flags(cblist, SEGCBLIST_KTHREAD_GP));
+}
+
+static long rcu_nocb_rdp_offload(void *arg)
+{
+ struct rcu_data *rdp = arg;
+
+ WARN_ON_ONCE(rdp->cpu != raw_smp_processor_id());
+ __rcu_nocb_rdp_offload(rdp);
+
+ return 0;
+}
+
+void rcu_nocb_cpu_offload(int cpu)
+{
+ struct rcu_data *rdp = per_cpu_ptr(&rcu_data, cpu);
+
+ mutex_lock(&rcu_state.barrier_mutex);
+ cpus_read_lock();
+ if (!rcu_segcblist_is_offloaded(&rdp->cblist)) {
+ if (cpu_online(cpu)) {
+ work_on_cpu(cpu, rcu_nocb_rdp_offload, rdp);
+ } else {
+ __rcu_nocb_rdp_offload(rdp);
+ }
+ cpumask_set_cpu(cpu, rcu_nocb_mask);
+ }
+ cpus_read_unlock();
+ mutex_unlock(&rcu_state.barrier_mutex);
+}
+
+
void __init rcu_init_nohz(void)
{
int cpu;
--
2.28.0
^ permalink raw reply related [flat|nested] 32+ messages in thread
* [RFC PATCH 07/12] rcu: Shutdown nocb timer on de-offloading
2020-09-21 12:43 [RFC PATCH 00/12] rcu/nocb: De-offload and re-offload support v2 Frederic Weisbecker
` (5 preceding siblings ...)
2020-09-21 12:43 ` [RFC PATCH 06/12] rcu: Re-offload support Frederic Weisbecker
@ 2020-09-21 12:43 ` Frederic Weisbecker
2020-09-22 0:17 ` Paul E. McKenney
2020-09-21 12:43 ` [RFC PATCH 08/12] rcu: Flush bypass before setting SEGCBLIST_SOFTIRQ_ONLY Frederic Weisbecker
` (4 subsequent siblings)
11 siblings, 1 reply; 32+ messages in thread
From: Frederic Weisbecker @ 2020-09-21 12:43 UTC (permalink / raw)
To: Paul E . McKenney
Cc: LKML, Frederic Weisbecker, Steven Rostedt, Mathieu Desnoyers,
Lai Jiangshan, Joel Fernandes, Josh Triplett
Make sure the nocb timer can't fire anymore before we reach the final
de-offload state. Spuriously waking up the GP kthread is no big deal but
we must prevent from executing the timer callback without nocb locking.
Inspired-by: Paul E. McKenney <paulmck@kernel.org>
Signed-off-by: Frederic Weisbecker <frederic@kernel.org>
Cc: Paul E. McKenney <paulmck@kernel.org>
Cc: Josh Triplett <josh@joshtriplett.org>
Cc: Steven Rostedt <rostedt@goodmis.org>
Cc: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Cc: Lai Jiangshan <jiangshanlai@gmail.com>
Cc: Joel Fernandes <joel@joelfernandes.org>
---
kernel/rcu/tree.h | 1 +
kernel/rcu/tree_plugin.h | 12 +++++++++++-
2 files changed, 12 insertions(+), 1 deletion(-)
diff --git a/kernel/rcu/tree.h b/kernel/rcu/tree.h
index 5b37f7103b0d..ca69238e2227 100644
--- a/kernel/rcu/tree.h
+++ b/kernel/rcu/tree.h
@@ -254,6 +254,7 @@ struct rcu_data {
};
/* Values for nocb_defer_wakeup field in struct rcu_data. */
+#define RCU_NOCB_WAKE_OFF -1
#define RCU_NOCB_WAKE_NOT 0
#define RCU_NOCB_WAKE 1
#define RCU_NOCB_WAKE_FORCE 2
diff --git a/kernel/rcu/tree_plugin.h b/kernel/rcu/tree_plugin.h
index eceef6dade9a..3361bd351f3b 100644
--- a/kernel/rcu/tree_plugin.h
+++ b/kernel/rcu/tree_plugin.h
@@ -1637,6 +1637,8 @@ static void wake_nocb_gp(struct rcu_data *rdp, bool force,
static void wake_nocb_gp_defer(struct rcu_data *rdp, int waketype,
const char *reason)
{
+ if (rdp->nocb_defer_wakeup == RCU_NOCB_WAKE_OFF)
+ return;
if (rdp->nocb_defer_wakeup == RCU_NOCB_WAKE_NOT)
mod_timer(&rdp->nocb_timer, jiffies + 1);
if (rdp->nocb_defer_wakeup < waketype)
@@ -2214,7 +2216,7 @@ static int rcu_nocb_cb_kthread(void *arg)
/* Is a deferred wakeup of rcu_nocb_kthread() required? */
static int rcu_nocb_need_deferred_wakeup(struct rcu_data *rdp)
{
- return READ_ONCE(rdp->nocb_defer_wakeup);
+ return READ_ONCE(rdp->nocb_defer_wakeup) > RCU_NOCB_WAKE_NOT;
}
/* Do a deferred wakeup of rcu_nocb_kthread(). */
@@ -2299,6 +2301,12 @@ static void __rcu_nocb_rdp_deoffload(struct rcu_data *rdp)
swait_event_exclusive(rdp->nocb_state_wq,
!rcu_segcblist_test_flags(cblist, SEGCBLIST_KTHREAD_CB |
SEGCBLIST_KTHREAD_GP));
+
+ /* Make sure nocb timer won't stay around */
+ rcu_nocb_lock_irqsave(rdp, flags);
+ WRITE_ONCE(rdp->nocb_defer_wakeup, RCU_NOCB_WAKE_OFF);
+ rcu_nocb_unlock_irqrestore(rdp, flags);
+ del_timer_sync(&rdp->nocb_timer);
}
static long rcu_nocb_rdp_deoffload(void *arg)
@@ -2344,6 +2352,8 @@ static void __rcu_nocb_rdp_offload(struct rcu_data *rdp)
* SEGCBLIST_SOFTIRQ_ONLY mode.
*/
raw_spin_lock_irqsave(&rdp->nocb_lock, flags);
+ /* Re-enable nocb timer */
+ WRITE_ONCE(rdp->nocb_defer_wakeup, RCU_NOCB_WAKE_NOT);
/*
* We didn't take the nocb lock while working on the
* rdp->cblist in SEGCBLIST_SOFTIRQ_ONLY mode.
--
2.28.0
^ permalink raw reply related [flat|nested] 32+ messages in thread
* [RFC PATCH 08/12] rcu: Flush bypass before setting SEGCBLIST_SOFTIRQ_ONLY
2020-09-21 12:43 [RFC PATCH 00/12] rcu/nocb: De-offload and re-offload support v2 Frederic Weisbecker
` (6 preceding siblings ...)
2020-09-21 12:43 ` [RFC PATCH 07/12] rcu: Shutdown nocb timer on de-offloading Frederic Weisbecker
@ 2020-09-21 12:43 ` Frederic Weisbecker
2020-09-21 12:43 ` [RFC PATCH 09/12] rcu: Set SEGCBLIST_SOFTIRQ_ONLY at the very last stage of de-offloading Frederic Weisbecker
` (3 subsequent siblings)
11 siblings, 0 replies; 32+ messages in thread
From: Frederic Weisbecker @ 2020-09-21 12:43 UTC (permalink / raw)
To: Paul E . McKenney
Cc: LKML, Frederic Weisbecker, Steven Rostedt, Mathieu Desnoyers,
Lai Jiangshan, Joel Fernandes, Josh Triplett
Make sure to handle the pending bypass queue before we switch to the
final de-offload state. We'll have to be careful and later set
SEGCBLIST_SOFTIRQ_ONLY before re-enabling again IRQs, or new bypass
callbacks could be queued in the meantine.
Inspired-by: Paul E. McKenney <paulmck@kernel.org>
Signed-off-by: Frederic Weisbecker <frederic@kernel.org>
Cc: Paul E. McKenney <paulmck@kernel.org>
Cc: Josh Triplett <josh@joshtriplett.org>
Cc: Steven Rostedt <rostedt@goodmis.org>
Cc: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Cc: Lai Jiangshan <jiangshanlai@gmail.com>
Cc: Joel Fernandes <joel@joelfernandes.org>
---
kernel/rcu/tree_plugin.h | 10 ++++++++--
1 file changed, 8 insertions(+), 2 deletions(-)
diff --git a/kernel/rcu/tree_plugin.h b/kernel/rcu/tree_plugin.h
index 3361bd351f3b..cee15841c178 100644
--- a/kernel/rcu/tree_plugin.h
+++ b/kernel/rcu/tree_plugin.h
@@ -2302,11 +2302,17 @@ static void __rcu_nocb_rdp_deoffload(struct rcu_data *rdp)
!rcu_segcblist_test_flags(cblist, SEGCBLIST_KTHREAD_CB |
SEGCBLIST_KTHREAD_GP));
+ rcu_nocb_lock_irqsave(rdp, flags);
/* Make sure nocb timer won't stay around */
- rcu_nocb_lock_irqsave(rdp, flags);
WRITE_ONCE(rdp->nocb_defer_wakeup, RCU_NOCB_WAKE_OFF);
- rcu_nocb_unlock_irqrestore(rdp, flags);
del_timer_sync(&rdp->nocb_timer);
+ /*
+ * Flush bypass. While IRQs are disabled and once we set
+ * SEGCBLIST_SOFTIRQ_ONLY, no callback is supposed to be
+ * enqueued on bypass.
+ */
+ rcu_nocb_flush_bypass(rdp, NULL, jiffies);
+ rcu_nocb_unlock_irqrestore(rdp, flags);
}
static long rcu_nocb_rdp_deoffload(void *arg)
--
2.28.0
^ permalink raw reply related [flat|nested] 32+ messages in thread
* [RFC PATCH 09/12] rcu: Set SEGCBLIST_SOFTIRQ_ONLY at the very last stage of de-offloading
2020-09-21 12:43 [RFC PATCH 00/12] rcu/nocb: De-offload and re-offload support v2 Frederic Weisbecker
` (7 preceding siblings ...)
2020-09-21 12:43 ` [RFC PATCH 08/12] rcu: Flush bypass before setting SEGCBLIST_SOFTIRQ_ONLY Frederic Weisbecker
@ 2020-09-21 12:43 ` Frederic Weisbecker
2020-09-21 12:43 ` [RFC PATCH 10/12] rcu: Process batch locally as long as offloading isn't complete Frederic Weisbecker
` (2 subsequent siblings)
11 siblings, 0 replies; 32+ messages in thread
From: Frederic Weisbecker @ 2020-09-21 12:43 UTC (permalink / raw)
To: Paul E . McKenney
Cc: LKML, Frederic Weisbecker, Steven Rostedt, Mathieu Desnoyers,
Lai Jiangshan, Joel Fernandes, Josh Triplett
Set SEGCBLIST_SOFTIRQ_ONLY once everything is settled. After that, the
callbacks are handled locklessly and locally.
Inspired-by: Paul E. McKenney <paulmck@kernel.org>
Signed-off-by: Frederic Weisbecker <frederic@kernel.org>
Cc: Paul E. McKenney <paulmck@kernel.org>
Cc: Josh Triplett <josh@joshtriplett.org>
Cc: Steven Rostedt <rostedt@goodmis.org>
Cc: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Cc: Lai Jiangshan <jiangshanlai@gmail.com>
Cc: Joel Fernandes <joel@joelfernandes.org>
---
kernel/rcu/tree_plugin.h | 9 ++++++++-
1 file changed, 8 insertions(+), 1 deletion(-)
diff --git a/kernel/rcu/tree_plugin.h b/kernel/rcu/tree_plugin.h
index cee15841c178..ef09897c8f2c 100644
--- a/kernel/rcu/tree_plugin.h
+++ b/kernel/rcu/tree_plugin.h
@@ -2312,7 +2312,14 @@ static void __rcu_nocb_rdp_deoffload(struct rcu_data *rdp)
* enqueued on bypass.
*/
rcu_nocb_flush_bypass(rdp, NULL, jiffies);
- rcu_nocb_unlock_irqrestore(rdp, flags);
+ rcu_segcblist_set_flags(cblist, SEGCBLIST_SOFTIRQ_ONLY);
+ /*
+ * With SEGCBLIST_SOFTIRQ_ONLY, we can't use
+ * rcu_nocb_unlock_irqrestore() anymore. Theoretically we
+ * could set SEGCBLIST_SOFTIRQ_ONLY with cb unlocked and IRQs
+ * disabled now, but let's be paranoid.
+ */
+ raw_spin_unlock_irqrestore(&rdp->nocb_lock, flags);
}
static long rcu_nocb_rdp_deoffload(void *arg)
--
2.28.0
^ permalink raw reply related [flat|nested] 32+ messages in thread
* [RFC PATCH 10/12] rcu: Process batch locally as long as offloading isn't complete
2020-09-21 12:43 [RFC PATCH 00/12] rcu/nocb: De-offload and re-offload support v2 Frederic Weisbecker
` (8 preceding siblings ...)
2020-09-21 12:43 ` [RFC PATCH 09/12] rcu: Set SEGCBLIST_SOFTIRQ_ONLY at the very last stage of de-offloading Frederic Weisbecker
@ 2020-09-21 12:43 ` Frederic Weisbecker
2020-09-22 0:22 ` Paul E. McKenney
2020-09-21 12:43 ` [RFC PATCH 11/12] rcu: Locally accelerate callbacks " Frederic Weisbecker
2020-09-21 12:43 ` [RFC PATCH 12/12] rcu: Nocb (de)activate through sysfs Frederic Weisbecker
11 siblings, 1 reply; 32+ messages in thread
From: Frederic Weisbecker @ 2020-09-21 12:43 UTC (permalink / raw)
To: Paul E . McKenney
Cc: LKML, Frederic Weisbecker, Steven Rostedt, Mathieu Desnoyers,
Lai Jiangshan, Joel Fernandes, Josh Triplett
During the offloading or de-offloading process, make sure to process
the callbacks batch locally whenever the segcblist isn't entirely
offloaded. This enforces callback service processing while we are still
in intermediate (de-)offloading state.
FIXME: Note that __call_rcu_core() isn't called during these intermediate
states. Some pieces there may still be necessary.
Inspired-by: Paul E. McKenney <paulmck@kernel.org>
Signed-off-by: Frederic Weisbecker <frederic@kernel.org>
Cc: Paul E. McKenney <paulmck@kernel.org>
Cc: Josh Triplett <josh@joshtriplett.org>
Cc: Steven Rostedt <rostedt@goodmis.org>
Cc: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Cc: Lai Jiangshan <jiangshanlai@gmail.com>
Cc: Joel Fernandes <joel@joelfernandes.org>
---
kernel/rcu/rcu_segcblist.h | 12 ++++++++++++
kernel/rcu/tree.c | 3 ++-
2 files changed, 14 insertions(+), 1 deletion(-)
diff --git a/kernel/rcu/rcu_segcblist.h b/kernel/rcu/rcu_segcblist.h
index 00ebeb8d39b7..f7da3d535888 100644
--- a/kernel/rcu/rcu_segcblist.h
+++ b/kernel/rcu/rcu_segcblist.h
@@ -92,6 +92,18 @@ static inline bool rcu_segcblist_is_offloaded(struct rcu_segcblist *rsclp)
return false;
}
+static inline bool rcu_segcblist_completely_offloaded(struct rcu_segcblist *rsclp)
+{
+ int flags = SEGCBLIST_KTHREAD_CB | SEGCBLIST_KTHREAD_GP | SEGCBLIST_OFFLOADED;
+
+ if (IS_ENABLED(CONFIG_RCU_NOCB_CPU)) {
+ if ((rsclp->flags & flags) == flags)
+ return true;
+ }
+
+ return false;
+}
+
/*
* Are all segments following the specified segment of the specified
* rcu_segcblist structure empty of callbacks? (The specified
diff --git a/kernel/rcu/tree.c b/kernel/rcu/tree.c
index b4292489db0c..928907e9ba94 100644
--- a/kernel/rcu/tree.c
+++ b/kernel/rcu/tree.c
@@ -2620,6 +2620,7 @@ static __latent_entropy void rcu_core(void)
struct rcu_data *rdp = raw_cpu_ptr(&rcu_data);
struct rcu_node *rnp = rdp->mynode;
const bool offloaded = rcu_segcblist_is_offloaded(&rdp->cblist);
+ const bool do_batch = !rcu_segcblist_completely_offloaded(&rdp->cblist);
if (cpu_is_offline(smp_processor_id()))
return;
@@ -2649,7 +2650,7 @@ static __latent_entropy void rcu_core(void)
rcu_check_gp_start_stall(rnp, rdp, rcu_jiffies_till_stall_check());
/* If there are callbacks ready, invoke them. */
- if (!offloaded && rcu_segcblist_ready_cbs(&rdp->cblist) &&
+ if (do_batch && rcu_segcblist_ready_cbs(&rdp->cblist) &&
likely(READ_ONCE(rcu_scheduler_fully_active)))
rcu_do_batch(rdp);
--
2.28.0
^ permalink raw reply related [flat|nested] 32+ messages in thread
* [RFC PATCH 11/12] rcu: Locally accelerate callbacks as long as offloading isn't complete
2020-09-21 12:43 [RFC PATCH 00/12] rcu/nocb: De-offload and re-offload support v2 Frederic Weisbecker
` (9 preceding siblings ...)
2020-09-21 12:43 ` [RFC PATCH 10/12] rcu: Process batch locally as long as offloading isn't complete Frederic Weisbecker
@ 2020-09-21 12:43 ` Frederic Weisbecker
2020-09-22 0:24 ` Paul E. McKenney
2020-09-21 12:43 ` [RFC PATCH 12/12] rcu: Nocb (de)activate through sysfs Frederic Weisbecker
11 siblings, 1 reply; 32+ messages in thread
From: Frederic Weisbecker @ 2020-09-21 12:43 UTC (permalink / raw)
To: Paul E . McKenney
Cc: LKML, Frederic Weisbecker, Steven Rostedt, Mathieu Desnoyers,
Lai Jiangshan, Joel Fernandes, Josh Triplett
I have no idea what I'm doing but doing that looks necessary to me.
Inspired-by: Paul E. McKenney <paulmck@kernel.org>
Signed-off-by: Frederic Weisbecker <frederic@kernel.org>
Cc: Paul E. McKenney <paulmck@kernel.org>
Cc: Josh Triplett <josh@joshtriplett.org>
Cc: Steven Rostedt <rostedt@goodmis.org>
Cc: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Cc: Lai Jiangshan <jiangshanlai@gmail.com>
Cc: Joel Fernandes <joel@joelfernandes.org>
---
kernel/rcu/tree.c | 7 +++----
1 file changed, 3 insertions(+), 4 deletions(-)
diff --git a/kernel/rcu/tree.c b/kernel/rcu/tree.c
index 928907e9ba94..8cef5ea8d1f0 100644
--- a/kernel/rcu/tree.c
+++ b/kernel/rcu/tree.c
@@ -2619,7 +2619,6 @@ static __latent_entropy void rcu_core(void)
unsigned long flags;
struct rcu_data *rdp = raw_cpu_ptr(&rcu_data);
struct rcu_node *rnp = rdp->mynode;
- const bool offloaded = rcu_segcblist_is_offloaded(&rdp->cblist);
const bool do_batch = !rcu_segcblist_completely_offloaded(&rdp->cblist);
if (cpu_is_offline(smp_processor_id()))
@@ -2640,11 +2639,11 @@ static __latent_entropy void rcu_core(void)
/* No grace period and unregistered callbacks? */
if (!rcu_gp_in_progress() &&
- rcu_segcblist_is_enabled(&rdp->cblist) && !offloaded) {
- local_irq_save(flags);
+ rcu_segcblist_is_enabled(&rdp->cblist) && do_batch) {
+ rcu_nocb_lock_irqsave(rdp, flags);
if (!rcu_segcblist_restempty(&rdp->cblist, RCU_NEXT_READY_TAIL))
rcu_accelerate_cbs_unlocked(rnp, rdp);
- local_irq_restore(flags);
+ rcu_nocb_unlock_irqrestore(rdp, flags);
}
rcu_check_gp_start_stall(rnp, rdp, rcu_jiffies_till_stall_check());
--
2.28.0
^ permalink raw reply related [flat|nested] 32+ messages in thread
* [RFC PATCH 12/12] rcu: Nocb (de)activate through sysfs
2020-09-21 12:43 [RFC PATCH 00/12] rcu/nocb: De-offload and re-offload support v2 Frederic Weisbecker
` (10 preceding siblings ...)
2020-09-21 12:43 ` [RFC PATCH 11/12] rcu: Locally accelerate callbacks " Frederic Weisbecker
@ 2020-09-21 12:43 ` Frederic Weisbecker
2020-09-22 0:26 ` Paul E. McKenney
11 siblings, 1 reply; 32+ messages in thread
From: Frederic Weisbecker @ 2020-09-21 12:43 UTC (permalink / raw)
To: Paul E . McKenney
Cc: LKML, Frederic Weisbecker, Steven Rostedt, Mathieu Desnoyers,
Lai Jiangshan, Joel Fernandes, Josh Triplett
Not for merge.
Make nocb toggable for a given CPU using:
/sys/devices/system/cpu/cpu*/hotplug/nocb
This is only intended for those who want to test this patchset. The real
interfaces will be cpuset/isolation and rcutorture.
Not-Signed-off-by: Frederic Weisbecker <frederic@kernel.org>
Cc: Paul E. McKenney <paulmck@kernel.org>
Cc: Josh Triplett <josh@joshtriplett.org>
Cc: Steven Rostedt <rostedt@goodmis.org>
Cc: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Cc: Lai Jiangshan <jiangshanlai@gmail.com>
Cc: Joel Fernandes <joel@joelfernandes.org>
---
kernel/cpu.c | 23 +++++++++++++++++++++++
1 file changed, 23 insertions(+)
diff --git a/kernel/cpu.c b/kernel/cpu.c
index 6ff2578ecf17..a36634113b8e 100644
--- a/kernel/cpu.c
+++ b/kernel/cpu.c
@@ -2224,10 +2224,33 @@ static ssize_t show_cpuhp_fail(struct device *dev,
static DEVICE_ATTR(fail, 0644, show_cpuhp_fail, write_cpuhp_fail);
+static ssize_t write_nocb(struct device *dev,
+ struct device_attribute *attr,
+ const char *buf, size_t count)
+{
+ int val, ret;
+
+ ret = kstrtoint(buf, 10, &val);
+ if (ret)
+ return ret;
+
+ if (val == 0)
+ rcu_nocb_cpu_deoffload(dev->id);
+ else if (val == 1)
+ rcu_nocb_cpu_offload(dev->id);
+ else
+ return -EINVAL;
+
+ return count;
+}
+
+static DEVICE_ATTR(nocb, 0644, NULL, write_nocb);
+
static struct attribute *cpuhp_cpu_attrs[] = {
&dev_attr_state.attr,
&dev_attr_target.attr,
&dev_attr_fail.attr,
+ &dev_attr_nocb.attr,
NULL
};
--
2.28.0
^ permalink raw reply related [flat|nested] 32+ messages in thread
* Re: [RFC PATCH 03/12] rcu: Provide basic callback offloading state machine bits
2020-09-21 12:43 ` [RFC PATCH 03/12] rcu: Provide basic callback offloading state machine bits Frederic Weisbecker
@ 2020-09-21 23:50 ` Paul E. McKenney
0 siblings, 0 replies; 32+ messages in thread
From: Paul E. McKenney @ 2020-09-21 23:50 UTC (permalink / raw)
To: Frederic Weisbecker
Cc: LKML, Steven Rostedt, Mathieu Desnoyers, Lai Jiangshan,
Joel Fernandes, Josh Triplett
On Mon, Sep 21, 2020 at 02:43:42PM +0200, Frederic Weisbecker wrote:
> We'll need to be able to runtime offload and de-offload the processing
> of callback for a given CPU. In order to support a smooth transition
> from unlocked local processing (softirq/rcuc) to locked offloaded
> processing (rcuop/rcuog) and the reverse, provide the necessary bits and
> documentation for the state machine that will carry up all the steps to
> enforce correctness while serving callbacks processing all along.
>
> Inspired-by: Paul E. McKenney <paulmck@kernel.org>
> Signed-off-by: Frederic Weisbecker <frederic@kernel.org>
> Cc: Paul E. McKenney <paulmck@kernel.org>
> Cc: Josh Triplett <josh@joshtriplett.org>
> Cc: Steven Rostedt <rostedt@goodmis.org>
> Cc: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
> Cc: Lai Jiangshan <jiangshanlai@gmail.com>
> Cc: Joel Fernandes <joel@joelfernandes.org>
> ---
> include/linux/rcu_segcblist.h | 115 +++++++++++++++++++++++++++++++++-
> kernel/rcu/rcu_segcblist.c | 1 +
> kernel/rcu/rcu_segcblist.h | 12 +++-
> kernel/rcu/tree.c | 3 +
> 4 files changed, 128 insertions(+), 3 deletions(-)
>
> diff --git a/include/linux/rcu_segcblist.h b/include/linux/rcu_segcblist.h
> index dca2f39ee67f..67f09a912f96 100644
> --- a/include/linux/rcu_segcblist.h
> +++ b/include/linux/rcu_segcblist.h
> @@ -63,8 +63,121 @@ struct rcu_cblist {
> #define RCU_NEXT_TAIL 3
> #define RCU_CBLIST_NSEGS 4
>
> +
> +/*
> + * ==NOCB Offloading state machine==
> + *
> + *
> + * ----------------------------------------------------------------------------
> + * | SEGCBLIST_SOFTIRQ_ONLY |
> + * | |
> + * | Callbacks processed by rcu_core() from softirqs or local |
> + * | rcuc kthread, without holding nocb_lock. |
> + * ----------------------------------------------------------------------------
> + * |
> + * v
> + * ----------------------------------------------------------------------------
> + * | SEGCBLIST_OFFLOADED |
> + * | |
> + * | Callbacks processed by rcu_core() from softirqs or local |
> + * | rcuc kthread, while holding nocb_lock. Waking up CB and GP kthreads, |
> + * | allowing nocb_timer to be armed. |
Whitespace nit before "allowing", just to show that I am paying attention. ;-)
Don't we need to acquire rcu_state.barrier_mutex at this point? Otherwise
rcu_barrier() could fail at this point.
> + * ----------------------------------------------------------------------------
> + * |
> + * v
> + * -----------------------------------
> + * | |
> + * v v
> + * --------------------------------------- ----------------------------------|
> + * | SEGCBLIST_OFFLOADED | | | SEGCBLIST_OFFLOADED | |
> + * | SEGCBLIST_KTHREAD_CB | | SEGCBLIST_KTHREAD_GP |
> + * | | | |
> + * | | | |
> + * | CB kthread woke up and | | GP kthread woke up and |
> + * | acknowledged SEGCBLIST_OFFLOADED. | | acknowledged SEGCBLIST_OFFLOADED|
> + * | Processes callbacks concurrently | | |
> + * | with rcu_core(), holding | | |
> + * | nocb_lock. | | |
> + * --------------------------------------- -----------------------------------
> + * | |
> + * -----------------------------------
> + * |
> + * v
> + * |--------------------------------------------------------------------------|
> + * | SEGCBLIST_OFFLOADED | |
> + * | SEGCBLIST_KTHREAD_CB | |
> + * | SEGCBLIST_KTHREAD_GP |
> + * | |
> + * | Kthreads handle callbacks holding nocb_lock, local rcu_core() stops |
> + * | handling callbacks. |
And rcu_state.barrier_mutex is dropped here.
Or am I missing a trick here? I guess I will look at the later patches to
get an initial estimate of an answer to this question.
> + * ----------------------------------------------------------------------------
> + */
> +
> +
> +
> +/*
> + * ==NOCB De-Offloading state machine==
> + *
> + *
> + * |--------------------------------------------------------------------------|
> + * | SEGCBLIST_OFFLOADED | |
> + * | SEGCBLIST_KTHREAD_CB | |
> + * | SEGCBLIST_KTHREAD_GP |
> + * | |
> + * | CB/GP kthreads handle callbacks holding nocb_lock, local rcu_core() |
> + * | ignores callbacks. |
And don't we also need to acquire rcu_state.barrier_mutex here?
> + * ----------------------------------------------------------------------------
> + * |
> + * v
> + * |--------------------------------------------------------------------------|
> + * | SEGCBLIST_KTHREAD_CB | |
> + * | SEGCBLIST_KTHREAD_GP |
> + * | |
> + * | CB/GP kthreads and local rcu_core() handle callbacks concurrently |
> + * | holding nocb_lock. Wake up CB and GP kthreads if necessary. |
> + * ----------------------------------------------------------------------------
> + * |
> + * v
> + * -----------------------------------
> + * | |
> + * v v
> + * ---------------------------------------------------------------------------|
> + * | |
> + * | SEGCBLIST_KTHREAD_CB | SEGCBLIST_KTHREAD_GP |
> + * | | |
> + * | GP kthread woke up and | CB kthread woke up and |
> + * | acknowledged the fact that | acknowledged the fact that |
> + * | SEGCBLIST_OFFLOADED got cleared. | SEGCBLIST_OFFLOADED got cleared. |
> + * | | The CB kthread goes to sleep |
> + * | The callbacks from the target CPU | until it ever gets re-offloaded. |
> + * | will be ignored from the GP kthread | |
> + * | loop. | |
> + * ----------------------------------------------------------------------------
> + * | |
> + * -----------------------------------
> + * |
> + * v
> + * ----------------------------------------------------------------------------
> + * | 0 |
> + * | |
> + * | Callbacks processed by rcu_core() from softirqs or local |
> + * | rcuc kthread, while holding nocb_lock. Forbid nocb_timer to be armed. |
> + * | Flush pending nocb_timer. Flush nocb bypass callbacks. |
And release rcu_state.barrier_mutex here?
Thanx, Paul
> + * ----------------------------------------------------------------------------
> + * |
> + * v
> + * ----------------------------------------------------------------------------
> + * | SEGCBLIST_SOFTIRQ_ONLY |
> + * | |
> + * | Callbacks processed by rcu_core() from softirqs or local |
> + * | rcuc kthread, without holding nocb_lock. |
> + * ----------------------------------------------------------------------------
> + */
> #define SEGCBLIST_ENABLED BIT(0)
> -#define SEGCBLIST_OFFLOADED BIT(1)
> +#define SEGCBLIST_SOFTIRQ_ONLY BIT(1)
> +#define SEGCBLIST_KTHREAD_CB BIT(2)
> +#define SEGCBLIST_KTHREAD_GP BIT(3)
> +#define SEGCBLIST_OFFLOADED BIT(4)
>
> struct rcu_segcblist {
> struct rcu_head *head;
> diff --git a/kernel/rcu/rcu_segcblist.c b/kernel/rcu/rcu_segcblist.c
> index d131ef8940a0..31cc27ee98d8 100644
> --- a/kernel/rcu/rcu_segcblist.c
> +++ b/kernel/rcu/rcu_segcblist.c
> @@ -172,6 +172,7 @@ void rcu_segcblist_disable(struct rcu_segcblist *rsclp)
> */
> void rcu_segcblist_offload(struct rcu_segcblist *rsclp)
> {
> + rcu_segcblist_clear_flags(rsclp, SEGCBLIST_SOFTIRQ_ONLY);
> rcu_segcblist_set_flags(rsclp, SEGCBLIST_OFFLOADED);
> }
>
> diff --git a/kernel/rcu/rcu_segcblist.h b/kernel/rcu/rcu_segcblist.h
> index fc98761e3ee9..575896a2518b 100644
> --- a/kernel/rcu/rcu_segcblist.h
> +++ b/kernel/rcu/rcu_segcblist.h
> @@ -80,8 +80,16 @@ static inline bool rcu_segcblist_is_enabled(struct rcu_segcblist *rsclp)
> /* Is the specified rcu_segcblist offloaded? */
> static inline bool rcu_segcblist_is_offloaded(struct rcu_segcblist *rsclp)
> {
> - return IS_ENABLED(CONFIG_RCU_NOCB_CPU) &&
> - rcu_segcblist_test_flags(rsclp, SEGCBLIST_OFFLOADED);
> + if (IS_ENABLED(CONFIG_RCU_NOCB_CPU)) {
> + /*
> + * Complete de-offloading happens only when SEGCBLIST_SOFTIRQ_ONLY
> + * is set.
> + */
> + if (!rcu_segcblist_test_flags(rsclp, SEGCBLIST_SOFTIRQ_ONLY))
> + return true;
> + }
> +
> + return false;
> }
>
> /*
> diff --git a/kernel/rcu/tree.c b/kernel/rcu/tree.c
> index c0286ce8fc03..b4292489db0c 100644
> --- a/kernel/rcu/tree.c
> +++ b/kernel/rcu/tree.c
> @@ -96,6 +96,9 @@ static DEFINE_PER_CPU_SHARED_ALIGNED(struct rcu_data, rcu_data) = {
> .dynticks_nesting = 1,
> .dynticks_nmi_nesting = DYNTICK_IRQ_NONIDLE,
> .dynticks = ATOMIC_INIT(RCU_DYNTICK_CTRL_CTR),
> +#ifdef CONFIG_RCU_NOCB_CPU
> + .cblist.flags = SEGCBLIST_SOFTIRQ_ONLY,
> +#endif
> };
> static struct rcu_state rcu_state = {
> .level = { &rcu_state.node[0] },
> --
> 2.28.0
>
^ permalink raw reply [flat|nested] 32+ messages in thread
* Re: [RFC PATCH 04/12] rcu: De-offloading CB kthread
2020-09-21 12:43 ` [RFC PATCH 04/12] rcu: De-offloading CB kthread Frederic Weisbecker
@ 2020-09-22 0:07 ` Paul E. McKenney
0 siblings, 0 replies; 32+ messages in thread
From: Paul E. McKenney @ 2020-09-22 0:07 UTC (permalink / raw)
To: Frederic Weisbecker
Cc: LKML, Steven Rostedt, Mathieu Desnoyers, Lai Jiangshan,
Joel Fernandes, Josh Triplett
On Mon, Sep 21, 2020 at 02:43:43PM +0200, Frederic Weisbecker wrote:
> In order to de-offload the callbacks processing of an rdp, we must clear
> SEGCBLIST_OFFLOAD and notify the CB kthread so that it clears its own
> bit flag and goes to sleep to stop handling callbacks. The GP kthread
> will also be notified the same way. Whoever acknowledges and clears its
> own bit last must notify the de-offloading worker so that it can resume
> the de-offloading while being sure that callbacks won't be handled
> remotely anymore.
>
> Inspired-by: Paul E. McKenney <paulmck@kernel.org>
> Signed-off-by: Frederic Weisbecker <frederic@kernel.org>
> Cc: Paul E. McKenney <paulmck@kernel.org>
> Cc: Josh Triplett <josh@joshtriplett.org>
> Cc: Steven Rostedt <rostedt@goodmis.org>
> Cc: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
> Cc: Lai Jiangshan <jiangshanlai@gmail.com>
> Cc: Joel Fernandes <joel@joelfernandes.org>
> ---
> include/linux/rcupdate.h | 2 +
> kernel/rcu/rcu_segcblist.c | 10 ++-
> kernel/rcu/rcu_segcblist.h | 2 +-
> kernel/rcu/tree.h | 1 +
> kernel/rcu/tree_plugin.h | 124 ++++++++++++++++++++++++++++++++-----
> 5 files changed, 119 insertions(+), 20 deletions(-)
>
> diff --git a/include/linux/rcupdate.h b/include/linux/rcupdate.h
> index d15d46db61f7..44ddee921441 100644
> --- a/include/linux/rcupdate.h
> +++ b/include/linux/rcupdate.h
> @@ -97,8 +97,10 @@ static inline void rcu_user_exit(void) { }
>
> #ifdef CONFIG_RCU_NOCB_CPU
> void rcu_init_nohz(void);
> +void rcu_nocb_cpu_deoffload(int cpu);
> #else /* #ifdef CONFIG_RCU_NOCB_CPU */
> static inline void rcu_init_nohz(void) { }
> +static inline void rcu_nocb_cpu_deoffload(int cpu) { }
> #endif /* #else #ifdef CONFIG_RCU_NOCB_CPU */
>
> /**
> diff --git a/kernel/rcu/rcu_segcblist.c b/kernel/rcu/rcu_segcblist.c
> index 31cc27ee98d8..b70032d77025 100644
> --- a/kernel/rcu/rcu_segcblist.c
> +++ b/kernel/rcu/rcu_segcblist.c
> @@ -170,10 +170,14 @@ void rcu_segcblist_disable(struct rcu_segcblist *rsclp)
> * Mark the specified rcu_segcblist structure as offloaded. This
> * structure must be empty.
> */
> -void rcu_segcblist_offload(struct rcu_segcblist *rsclp)
> +void rcu_segcblist_offload(struct rcu_segcblist *rsclp, bool offload)
> {
> - rcu_segcblist_clear_flags(rsclp, SEGCBLIST_SOFTIRQ_ONLY);
> - rcu_segcblist_set_flags(rsclp, SEGCBLIST_OFFLOADED);
> + if (offload) {
> + rcu_segcblist_clear_flags(rsclp, SEGCBLIST_SOFTIRQ_ONLY);
> + rcu_segcblist_set_flags(rsclp, SEGCBLIST_OFFLOADED);
> + } else {
> + rcu_segcblist_clear_flags(rsclp, SEGCBLIST_OFFLOADED);
> + }
> }
>
> /*
> diff --git a/kernel/rcu/rcu_segcblist.h b/kernel/rcu/rcu_segcblist.h
> index 575896a2518b..00ebeb8d39b7 100644
> --- a/kernel/rcu/rcu_segcblist.h
> +++ b/kernel/rcu/rcu_segcblist.h
> @@ -105,7 +105,7 @@ static inline bool rcu_segcblist_restempty(struct rcu_segcblist *rsclp, int seg)
> void rcu_segcblist_inc_len(struct rcu_segcblist *rsclp);
> void rcu_segcblist_init(struct rcu_segcblist *rsclp);
> void rcu_segcblist_disable(struct rcu_segcblist *rsclp);
> -void rcu_segcblist_offload(struct rcu_segcblist *rsclp);
> +void rcu_segcblist_offload(struct rcu_segcblist *rsclp, bool offload);
> bool rcu_segcblist_ready_cbs(struct rcu_segcblist *rsclp);
> bool rcu_segcblist_pend_cbs(struct rcu_segcblist *rsclp);
> struct rcu_head *rcu_segcblist_first_cb(struct rcu_segcblist *rsclp);
> diff --git a/kernel/rcu/tree.h b/kernel/rcu/tree.h
> index c96ae351688b..5b37f7103b0d 100644
> --- a/kernel/rcu/tree.h
> +++ b/kernel/rcu/tree.h
> @@ -198,6 +198,7 @@ struct rcu_data {
> /* 5) Callback offloading. */
> #ifdef CONFIG_RCU_NOCB_CPU
> struct swait_queue_head nocb_cb_wq; /* For nocb kthreads to sleep on. */
> + struct swait_queue_head nocb_state_wq; /* For offloading state changes */
> struct task_struct *nocb_gp_kthread;
> raw_spinlock_t nocb_lock; /* Guard following pair of fields. */
> atomic_t nocb_lock_contended; /* Contention experienced. */
> diff --git a/kernel/rcu/tree_plugin.h b/kernel/rcu/tree_plugin.h
> index 982fc5be5269..3ea43c75c7a3 100644
> --- a/kernel/rcu/tree_plugin.h
> +++ b/kernel/rcu/tree_plugin.h
> @@ -2052,6 +2052,17 @@ static int rcu_nocb_gp_kthread(void *arg)
> return 0;
> }
>
> +static inline bool nocb_cb_can_run(struct rcu_data *rdp)
> +{
> + u8 flags = SEGCBLIST_OFFLOADED | SEGCBLIST_KTHREAD_CB;
> + return rcu_segcblist_test_flags(&rdp->cblist, flags);
> +}
> +
> +static inline bool nocb_cb_wait_cond(struct rcu_data *rdp)
> +{
> + return nocb_cb_can_run(rdp) && !READ_ONCE(rdp->nocb_cb_sleep);
> +}
> +
> /*
> * Invoke any ready callbacks from the corresponding no-CBs CPU,
> * then, if there are no more, wait for more to appear.
> @@ -2061,7 +2072,9 @@ static void nocb_cb_wait(struct rcu_data *rdp)
> unsigned long cur_gp_seq;
> unsigned long flags;
> bool needwake_gp = false;
> + bool needwake_state = false;
> struct rcu_node *rnp = rdp->mynode;
> + struct rcu_segcblist *cblist = &rdp->cblist;
This is just a convenience variable, correct? As in to avoid typing
"&rdp->" repeatedly? If so, please put a "const" on it.
>
> local_irq_save(flags);
> rcu_momentary_dyntick_idle();
> @@ -2071,32 +2084,50 @@ static void nocb_cb_wait(struct rcu_data *rdp)
> local_bh_enable();
> lockdep_assert_irqs_enabled();
> rcu_nocb_lock_irqsave(rdp, flags);
> - if (rcu_segcblist_nextgp(&rdp->cblist, &cur_gp_seq) &&
> + if (rcu_segcblist_nextgp(cblist, &cur_gp_seq) &&
> rcu_seq_done(&rnp->gp_seq, cur_gp_seq) &&
> raw_spin_trylock_rcu_node(rnp)) { /* irqs already disabled. */
> needwake_gp = rcu_advance_cbs(rdp->mynode, rdp);
> raw_spin_unlock_rcu_node(rnp); /* irqs remain disabled. */
> }
> - if (rcu_segcblist_ready_cbs(&rdp->cblist)) {
> - rcu_nocb_unlock_irqrestore(rdp, flags);
> - if (needwake_gp)
> - rcu_gp_kthread_wake();
> - return;
> - }
>
> - trace_rcu_nocb_wake(rcu_state.name, rdp->cpu, TPS("CBSleep"));
> WRITE_ONCE(rdp->nocb_cb_sleep, true);
> +
> + if (rcu_segcblist_test_flags(cblist, SEGCBLIST_OFFLOADED)) {
> + if (rcu_segcblist_ready_cbs(cblist))
> + WRITE_ONCE(rdp->nocb_cb_sleep, false);
> + } else {
> + /*
> + * De-offloading. Clear our flag and notify the de-offload worker.
> + * We won't touch the callbacks and keep sleeping until we ever
> + * get re-offloaded.
> + */
> + WARN_ON_ONCE(!rcu_segcblist_test_flags(cblist, SEGCBLIST_KTHREAD_CB));
> + rcu_segcblist_clear_flags(cblist, SEGCBLIST_KTHREAD_CB);
> + if (!rcu_segcblist_test_flags(cblist, SEGCBLIST_KTHREAD_GP))
> + needwake_state = true;
> + }
> +
> + if (rdp->nocb_cb_sleep)
> + trace_rcu_nocb_wake(rcu_state.name, rdp->cpu, TPS("CBSleep"));
> +
> rcu_nocb_unlock_irqrestore(rdp, flags);
> if (needwake_gp)
> rcu_gp_kthread_wake();
> - swait_event_interruptible_exclusive(rdp->nocb_cb_wq,
> - !READ_ONCE(rdp->nocb_cb_sleep));
> - if (!smp_load_acquire(&rdp->nocb_cb_sleep)) { /* VVV */
> +
> + if (needwake_state)
> + swake_up_one(&rdp->nocb_state_wq);
> +
> + do {
> + swait_event_interruptible_exclusive(rdp->nocb_cb_wq,
> + nocb_cb_wait_cond(rdp));
> +
> /* ^^^ Ensure CB invocation follows _sleep test. */
> - return;
> - }
> - WARN_ON(signal_pending(current));
> - trace_rcu_nocb_wake(rcu_state.name, rdp->cpu, TPS("WokeEmpty"));
> + if (smp_load_acquire(&rdp->nocb_cb_sleep)) {
> + WARN_ON(signal_pending(current));
> + trace_rcu_nocb_wake(rcu_state.name, rdp->cpu, TPS("WokeEmpty"));
> + }
> + } while (!nocb_cb_can_run(rdp));
> }
>
> /*
> @@ -2158,6 +2189,65 @@ static void do_nocb_deferred_wakeup(struct rcu_data *rdp)
> do_nocb_deferred_wakeup_common(rdp);
> }
>
> +static void __rcu_nocb_rdp_deoffload(struct rcu_data *rdp)
> +{
> + unsigned long flags;
> + struct rcu_node *rnp = rdp->mynode;
> + struct rcu_segcblist *cblist = &rdp->cblist;
> + bool wake_cb = false;
> +
> + printk("De-offloading %d\n", rdp->cpu);
> +
> + rcu_nocb_lock_irqsave(rdp, flags);
> + raw_spin_lock_rcu_node(rnp);
> + rcu_segcblist_offload(cblist, false);
> + raw_spin_unlock_rcu_node(rnp);
> +
> + if (rdp->nocb_cb_sleep) {
> + rdp->nocb_cb_sleep = false;
> + wake_cb = true;
> + }
> + rcu_nocb_unlock_irqrestore(rdp, flags);
> +
> + if (wake_cb)
> + swake_up_one(&rdp->nocb_cb_wq);
> +
> + swait_event_exclusive(rdp->nocb_state_wq,
> + !rcu_segcblist_test_flags(cblist, SEGCBLIST_KTHREAD_CB));
> +}
> +
> +static long rcu_nocb_rdp_deoffload(void *arg)
> +{
> + struct rcu_data *rdp = arg;
> +
> + WARN_ON_ONCE(rdp->cpu != raw_smp_processor_id());
> + __rcu_nocb_rdp_deoffload(rdp);
> +
> + return 0;
> +}
> +
> +void rcu_nocb_cpu_deoffload(int cpu)
> +{
> + struct rcu_data *rdp = per_cpu_ptr(&rcu_data, cpu);
> +
> + if (rdp == rdp->nocb_gp_rdp) {
> + pr_info("Can't deoffload an rdp GP leader (yet)\n");
> + return;
> + }
> + mutex_lock(&rcu_state.barrier_mutex);
> + cpus_read_lock();
And you do in fact acquire rcu_state.barrier_mutex as well as blocking
CPU-hotplug operations. Very good!
> + if (rcu_segcblist_is_offloaded(&rdp->cblist)) {
> + if (cpu_online(cpu)) {
> + work_on_cpu(cpu, rcu_nocb_rdp_deoffload, rdp);
> + } else {
> + __rcu_nocb_rdp_deoffload(rdp);
> + }
> + cpumask_clear_cpu(cpu, rcu_nocb_mask);
In the direct-call case, I get it. But in the work_on_cpu() case,
how do you know that the work has completed? Ah, because work_on_cpu()
is defined to wait until the work is done. Never mind! ;-)
Thanx, Paul
> + }
> + cpus_read_unlock();
> + mutex_unlock(&rcu_state.barrier_mutex);
> +}
> +
> void __init rcu_init_nohz(void)
> {
> int cpu;
> @@ -2200,7 +2290,8 @@ void __init rcu_init_nohz(void)
> rdp = per_cpu_ptr(&rcu_data, cpu);
> if (rcu_segcblist_empty(&rdp->cblist))
> rcu_segcblist_init(&rdp->cblist);
> - rcu_segcblist_offload(&rdp->cblist);
> + rcu_segcblist_offload(&rdp->cblist, true);
> + rcu_segcblist_set_flags(&rdp->cblist, SEGCBLIST_KTHREAD_CB);
> }
> rcu_organize_nocb_kthreads();
> }
> @@ -2210,6 +2301,7 @@ static void __init rcu_boot_init_nocb_percpu_data(struct rcu_data *rdp)
> {
> init_swait_queue_head(&rdp->nocb_cb_wq);
> init_swait_queue_head(&rdp->nocb_gp_wq);
> + init_swait_queue_head(&rdp->nocb_state_wq);
> raw_spin_lock_init(&rdp->nocb_lock);
> raw_spin_lock_init(&rdp->nocb_bypass_lock);
> raw_spin_lock_init(&rdp->nocb_gp_lock);
> --
> 2.28.0
>
^ permalink raw reply [flat|nested] 32+ messages in thread
* Re: [RFC PATCH 05/12] rcu: De-offloading GP kthread
2020-09-21 12:43 ` [RFC PATCH 05/12] rcu: De-offloading GP kthread Frederic Weisbecker
@ 2020-09-22 0:10 ` Paul E. McKenney
2020-09-23 15:31 ` Frederic Weisbecker
0 siblings, 1 reply; 32+ messages in thread
From: Paul E. McKenney @ 2020-09-22 0:10 UTC (permalink / raw)
To: Frederic Weisbecker
Cc: LKML, Steven Rostedt, Mathieu Desnoyers, Lai Jiangshan,
Joel Fernandes, Josh Triplett
On Mon, Sep 21, 2020 at 02:43:44PM +0200, Frederic Weisbecker wrote:
> In order to de-offload the callbacks processing of an rdp, we must clear
> SEGCBLIST_OFFLOAD and notify the GP kthread so that it clears its own
> bit flag and ignore the target rdp from its loop. The CB kthread
> is also notified the same way. Whoever acknowledges and clears its
> own bit last must notify the de-offloading worker so that it can resume
> the de-offloading while being sure that callbacks won't be handled
> remotely anymore.
>
> Inspired-by: Paul E. McKenney <paulmck@kernel.org>
> Signed-off-by: Frederic Weisbecker <frederic@kernel.org>
> Cc: Paul E. McKenney <paulmck@kernel.org>
> Cc: Josh Triplett <josh@joshtriplett.org>
> Cc: Steven Rostedt <rostedt@goodmis.org>
> Cc: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
> Cc: Lai Jiangshan <jiangshanlai@gmail.com>
> Cc: Joel Fernandes <joel@joelfernandes.org>
> ---
> kernel/rcu/tree_plugin.h | 53 ++++++++++++++++++++++++++++++++++++++--
> 1 file changed, 51 insertions(+), 2 deletions(-)
>
> diff --git a/kernel/rcu/tree_plugin.h b/kernel/rcu/tree_plugin.h
> index 3ea43c75c7a3..e4e7e7c4bfb6 100644
> --- a/kernel/rcu/tree_plugin.h
> +++ b/kernel/rcu/tree_plugin.h
> @@ -1900,6 +1900,33 @@ static void do_nocb_bypass_wakeup_timer(struct timer_list *t)
> __call_rcu_nocb_wake(rdp, true, flags);
> }
>
> +static inline bool nocb_gp_enabled_cb(struct rcu_data *rdp)
> +{
> + u8 flags = SEGCBLIST_OFFLOADED | SEGCBLIST_KTHREAD_GP;
> +
> + return rcu_segcblist_test_flags(&rdp->cblist, flags);
> +}
> +
> +static inline bool nocb_gp_update_state(struct rcu_data *rdp, bool *needwake_state)
> +{
> + struct rcu_segcblist *cblist = &rdp->cblist;
> +
> + if (rcu_segcblist_test_flags(cblist, SEGCBLIST_OFFLOADED)) {
> + return true;
> + } else {
> + /*
> + * De-offloading. Clear our flag and notify the de-offload worker.
> + * We will ignore this rdp until it ever gets re-offloaded.
> + */
> + WARN_ON_ONCE(!rcu_segcblist_test_flags(cblist, SEGCBLIST_KTHREAD_GP));
> + rcu_segcblist_clear_flags(cblist, SEGCBLIST_KTHREAD_GP);
> + if (!rcu_segcblist_test_flags(cblist, SEGCBLIST_KTHREAD_CB))
> + *needwake_state = true;
> + return false;
> + }
> +}
> +
> +
> /*
> * No-CBs GP kthreads come here to wait for additional callbacks to show up
> * or for grace periods to end.
> @@ -1927,8 +1954,17 @@ static void nocb_gp_wait(struct rcu_data *my_rdp)
> * and the global grace-period kthread are awakened if needed.
> */
> for (rdp = my_rdp; rdp; rdp = rdp->nocb_next_cb_rdp) {
> + bool needwake_state = false;
> + if (!nocb_gp_enabled_cb(rdp))
> + continue;
> trace_rcu_nocb_wake(rcu_state.name, rdp->cpu, TPS("Check"));
> rcu_nocb_lock_irqsave(rdp, flags);
> + if (!nocb_gp_update_state(rdp, &needwake_state)) {
> + rcu_nocb_unlock_irqrestore(rdp, flags);
> + if (needwake_state)
> + swake_up_one(&rdp->nocb_state_wq);
> + continue;
> + }
> bypass_ncbs = rcu_cblist_n_cbs(&rdp->nocb_bypass);
> if (bypass_ncbs &&
> (time_after(j, READ_ONCE(rdp->nocb_bypass_first) + 1) ||
> @@ -2194,7 +2230,8 @@ static void __rcu_nocb_rdp_deoffload(struct rcu_data *rdp)
> unsigned long flags;
> struct rcu_node *rnp = rdp->mynode;
> struct rcu_segcblist *cblist = &rdp->cblist;
> - bool wake_cb = false;
> + struct rcu_data *rdp_gp = rdp->nocb_gp_rdp;
> + bool wake_cb = false, wake_gp = false;
>
> printk("De-offloading %d\n", rdp->cpu);
>
> @@ -2212,8 +2249,19 @@ static void __rcu_nocb_rdp_deoffload(struct rcu_data *rdp)
> if (wake_cb)
> swake_up_one(&rdp->nocb_cb_wq);
>
> + raw_spin_lock_irqsave(&rdp_gp->nocb_gp_lock, flags);
> + if (rdp_gp->nocb_gp_sleep) {
> + rdp_gp->nocb_gp_sleep = false;
> + wake_gp = true;
> + }
> + raw_spin_unlock_irqrestore(&rdp_gp->nocb_gp_lock, flags);
> +
> + if (wake_gp)
> + wake_up_process(rdp_gp->nocb_gp_kthread);
> +
> swait_event_exclusive(rdp->nocb_state_wq,
> - !rcu_segcblist_test_flags(cblist, SEGCBLIST_KTHREAD_CB));
> + !rcu_segcblist_test_flags(cblist, SEGCBLIST_KTHREAD_CB |
> + SEGCBLIST_KTHREAD_GP));
> }
>
> static long rcu_nocb_rdp_deoffload(void *arg)
> @@ -2292,6 +2340,7 @@ void __init rcu_init_nohz(void)
> rcu_segcblist_init(&rdp->cblist);
> rcu_segcblist_offload(&rdp->cblist, true);
> rcu_segcblist_set_flags(&rdp->cblist, SEGCBLIST_KTHREAD_CB);
> + rcu_segcblist_set_flags(&rdp->cblist, SEGCBLIST_KTHREAD_GP);
OK, I will bite at this nit...
Why not "SEGCBLIST_KTHREAD_CB | SEGCBLIST_KTHREAD_GP"?
Thanx, Paul
> }
> rcu_organize_nocb_kthreads();
> }
> --
> 2.28.0
>
^ permalink raw reply [flat|nested] 32+ messages in thread
* Re: [RFC PATCH 07/12] rcu: Shutdown nocb timer on de-offloading
2020-09-21 12:43 ` [RFC PATCH 07/12] rcu: Shutdown nocb timer on de-offloading Frederic Weisbecker
@ 2020-09-22 0:17 ` Paul E. McKenney
2020-09-23 15:29 ` Frederic Weisbecker
0 siblings, 1 reply; 32+ messages in thread
From: Paul E. McKenney @ 2020-09-22 0:17 UTC (permalink / raw)
To: Frederic Weisbecker
Cc: LKML, Steven Rostedt, Mathieu Desnoyers, Lai Jiangshan,
Joel Fernandes, Josh Triplett
On Mon, Sep 21, 2020 at 02:43:46PM +0200, Frederic Weisbecker wrote:
> Make sure the nocb timer can't fire anymore before we reach the final
> de-offload state. Spuriously waking up the GP kthread is no big deal but
> we must prevent from executing the timer callback without nocb locking.
If we had just the previous patch and not this patch, would things break?
Or are you relying on the fact that there is not yet a connection to
userspace controls for this functionality?
Just looking out for bisectability...
Thanx, Paul
> Inspired-by: Paul E. McKenney <paulmck@kernel.org>
> Signed-off-by: Frederic Weisbecker <frederic@kernel.org>
> Cc: Paul E. McKenney <paulmck@kernel.org>
> Cc: Josh Triplett <josh@joshtriplett.org>
> Cc: Steven Rostedt <rostedt@goodmis.org>
> Cc: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
> Cc: Lai Jiangshan <jiangshanlai@gmail.com>
> Cc: Joel Fernandes <joel@joelfernandes.org>
> ---
> kernel/rcu/tree.h | 1 +
> kernel/rcu/tree_plugin.h | 12 +++++++++++-
> 2 files changed, 12 insertions(+), 1 deletion(-)
>
> diff --git a/kernel/rcu/tree.h b/kernel/rcu/tree.h
> index 5b37f7103b0d..ca69238e2227 100644
> --- a/kernel/rcu/tree.h
> +++ b/kernel/rcu/tree.h
> @@ -254,6 +254,7 @@ struct rcu_data {
> };
>
> /* Values for nocb_defer_wakeup field in struct rcu_data. */
> +#define RCU_NOCB_WAKE_OFF -1
> #define RCU_NOCB_WAKE_NOT 0
> #define RCU_NOCB_WAKE 1
> #define RCU_NOCB_WAKE_FORCE 2
> diff --git a/kernel/rcu/tree_plugin.h b/kernel/rcu/tree_plugin.h
> index eceef6dade9a..3361bd351f3b 100644
> --- a/kernel/rcu/tree_plugin.h
> +++ b/kernel/rcu/tree_plugin.h
> @@ -1637,6 +1637,8 @@ static void wake_nocb_gp(struct rcu_data *rdp, bool force,
> static void wake_nocb_gp_defer(struct rcu_data *rdp, int waketype,
> const char *reason)
> {
> + if (rdp->nocb_defer_wakeup == RCU_NOCB_WAKE_OFF)
> + return;
> if (rdp->nocb_defer_wakeup == RCU_NOCB_WAKE_NOT)
> mod_timer(&rdp->nocb_timer, jiffies + 1);
> if (rdp->nocb_defer_wakeup < waketype)
> @@ -2214,7 +2216,7 @@ static int rcu_nocb_cb_kthread(void *arg)
> /* Is a deferred wakeup of rcu_nocb_kthread() required? */
> static int rcu_nocb_need_deferred_wakeup(struct rcu_data *rdp)
> {
> - return READ_ONCE(rdp->nocb_defer_wakeup);
> + return READ_ONCE(rdp->nocb_defer_wakeup) > RCU_NOCB_WAKE_NOT;
> }
>
> /* Do a deferred wakeup of rcu_nocb_kthread(). */
> @@ -2299,6 +2301,12 @@ static void __rcu_nocb_rdp_deoffload(struct rcu_data *rdp)
> swait_event_exclusive(rdp->nocb_state_wq,
> !rcu_segcblist_test_flags(cblist, SEGCBLIST_KTHREAD_CB |
> SEGCBLIST_KTHREAD_GP));
> +
> + /* Make sure nocb timer won't stay around */
> + rcu_nocb_lock_irqsave(rdp, flags);
> + WRITE_ONCE(rdp->nocb_defer_wakeup, RCU_NOCB_WAKE_OFF);
> + rcu_nocb_unlock_irqrestore(rdp, flags);
> + del_timer_sync(&rdp->nocb_timer);
> }
>
> static long rcu_nocb_rdp_deoffload(void *arg)
> @@ -2344,6 +2352,8 @@ static void __rcu_nocb_rdp_offload(struct rcu_data *rdp)
> * SEGCBLIST_SOFTIRQ_ONLY mode.
> */
> raw_spin_lock_irqsave(&rdp->nocb_lock, flags);
> + /* Re-enable nocb timer */
> + WRITE_ONCE(rdp->nocb_defer_wakeup, RCU_NOCB_WAKE_NOT);
> /*
> * We didn't take the nocb lock while working on the
> * rdp->cblist in SEGCBLIST_SOFTIRQ_ONLY mode.
> --
> 2.28.0
>
^ permalink raw reply [flat|nested] 32+ messages in thread
* Re: [RFC PATCH 10/12] rcu: Process batch locally as long as offloading isn't complete
2020-09-21 12:43 ` [RFC PATCH 10/12] rcu: Process batch locally as long as offloading isn't complete Frederic Weisbecker
@ 2020-09-22 0:22 ` Paul E. McKenney
0 siblings, 0 replies; 32+ messages in thread
From: Paul E. McKenney @ 2020-09-22 0:22 UTC (permalink / raw)
To: Frederic Weisbecker
Cc: LKML, Steven Rostedt, Mathieu Desnoyers, Lai Jiangshan,
Joel Fernandes, Josh Triplett
On Mon, Sep 21, 2020 at 02:43:49PM +0200, Frederic Weisbecker wrote:
> During the offloading or de-offloading process, make sure to process
> the callbacks batch locally whenever the segcblist isn't entirely
> offloaded. This enforces callback service processing while we are still
> in intermediate (de-)offloading state.
>
> FIXME: Note that __call_rcu_core() isn't called during these intermediate
> states. Some pieces there may still be necessary.
Joel's per-segment callback-count work might help there, though I believe
that to be a separate issue.
The key point is that if there are a lot of callbacks, but all of them
are ready to be invoked, then there isn't much point in making the
grace periods complete faster. Right now, lacking per-segment counts,
RCU just assumes that none of the callbacks are ready to be invoked.
Thanx, Paul
> Inspired-by: Paul E. McKenney <paulmck@kernel.org>
> Signed-off-by: Frederic Weisbecker <frederic@kernel.org>
> Cc: Paul E. McKenney <paulmck@kernel.org>
> Cc: Josh Triplett <josh@joshtriplett.org>
> Cc: Steven Rostedt <rostedt@goodmis.org>
> Cc: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
> Cc: Lai Jiangshan <jiangshanlai@gmail.com>
> Cc: Joel Fernandes <joel@joelfernandes.org>
> ---
> kernel/rcu/rcu_segcblist.h | 12 ++++++++++++
> kernel/rcu/tree.c | 3 ++-
> 2 files changed, 14 insertions(+), 1 deletion(-)
>
> diff --git a/kernel/rcu/rcu_segcblist.h b/kernel/rcu/rcu_segcblist.h
> index 00ebeb8d39b7..f7da3d535888 100644
> --- a/kernel/rcu/rcu_segcblist.h
> +++ b/kernel/rcu/rcu_segcblist.h
> @@ -92,6 +92,18 @@ static inline bool rcu_segcblist_is_offloaded(struct rcu_segcblist *rsclp)
> return false;
> }
>
> +static inline bool rcu_segcblist_completely_offloaded(struct rcu_segcblist *rsclp)
> +{
> + int flags = SEGCBLIST_KTHREAD_CB | SEGCBLIST_KTHREAD_GP | SEGCBLIST_OFFLOADED;
> +
> + if (IS_ENABLED(CONFIG_RCU_NOCB_CPU)) {
> + if ((rsclp->flags & flags) == flags)
> + return true;
> + }
> +
> + return false;
> +}
> +
> /*
> * Are all segments following the specified segment of the specified
> * rcu_segcblist structure empty of callbacks? (The specified
> diff --git a/kernel/rcu/tree.c b/kernel/rcu/tree.c
> index b4292489db0c..928907e9ba94 100644
> --- a/kernel/rcu/tree.c
> +++ b/kernel/rcu/tree.c
> @@ -2620,6 +2620,7 @@ static __latent_entropy void rcu_core(void)
> struct rcu_data *rdp = raw_cpu_ptr(&rcu_data);
> struct rcu_node *rnp = rdp->mynode;
> const bool offloaded = rcu_segcblist_is_offloaded(&rdp->cblist);
> + const bool do_batch = !rcu_segcblist_completely_offloaded(&rdp->cblist);
>
> if (cpu_is_offline(smp_processor_id()))
> return;
> @@ -2649,7 +2650,7 @@ static __latent_entropy void rcu_core(void)
> rcu_check_gp_start_stall(rnp, rdp, rcu_jiffies_till_stall_check());
>
> /* If there are callbacks ready, invoke them. */
> - if (!offloaded && rcu_segcblist_ready_cbs(&rdp->cblist) &&
> + if (do_batch && rcu_segcblist_ready_cbs(&rdp->cblist) &&
> likely(READ_ONCE(rcu_scheduler_fully_active)))
> rcu_do_batch(rdp);
>
> --
> 2.28.0
>
^ permalink raw reply [flat|nested] 32+ messages in thread
* Re: [RFC PATCH 11/12] rcu: Locally accelerate callbacks as long as offloading isn't complete
2020-09-21 12:43 ` [RFC PATCH 11/12] rcu: Locally accelerate callbacks " Frederic Weisbecker
@ 2020-09-22 0:24 ` Paul E. McKenney
0 siblings, 0 replies; 32+ messages in thread
From: Paul E. McKenney @ 2020-09-22 0:24 UTC (permalink / raw)
To: Frederic Weisbecker
Cc: LKML, Steven Rostedt, Mathieu Desnoyers, Lai Jiangshan,
Joel Fernandes, Josh Triplett
On Mon, Sep 21, 2020 at 02:43:50PM +0200, Frederic Weisbecker wrote:
> I have no idea what I'm doing but doing that looks necessary to me.
Now -there- is a commit log that inspires confidence!!! ;-)
> Inspired-by: Paul E. McKenney <paulmck@kernel.org>
> Signed-off-by: Frederic Weisbecker <frederic@kernel.org>
> Cc: Paul E. McKenney <paulmck@kernel.org>
> Cc: Josh Triplett <josh@joshtriplett.org>
> Cc: Steven Rostedt <rostedt@goodmis.org>
> Cc: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
> Cc: Lai Jiangshan <jiangshanlai@gmail.com>
> Cc: Joel Fernandes <joel@joelfernandes.org>
> ---
> kernel/rcu/tree.c | 7 +++----
> 1 file changed, 3 insertions(+), 4 deletions(-)
>
> diff --git a/kernel/rcu/tree.c b/kernel/rcu/tree.c
> index 928907e9ba94..8cef5ea8d1f0 100644
> --- a/kernel/rcu/tree.c
> +++ b/kernel/rcu/tree.c
> @@ -2619,7 +2619,6 @@ static __latent_entropy void rcu_core(void)
> unsigned long flags;
> struct rcu_data *rdp = raw_cpu_ptr(&rcu_data);
> struct rcu_node *rnp = rdp->mynode;
> - const bool offloaded = rcu_segcblist_is_offloaded(&rdp->cblist);
> const bool do_batch = !rcu_segcblist_completely_offloaded(&rdp->cblist);
>
> if (cpu_is_offline(smp_processor_id()))
> @@ -2640,11 +2639,11 @@ static __latent_entropy void rcu_core(void)
>
> /* No grace period and unregistered callbacks? */
> if (!rcu_gp_in_progress() &&
> - rcu_segcblist_is_enabled(&rdp->cblist) && !offloaded) {
> - local_irq_save(flags);
> + rcu_segcblist_is_enabled(&rdp->cblist) && do_batch) {
> + rcu_nocb_lock_irqsave(rdp, flags);
> if (!rcu_segcblist_restempty(&rdp->cblist, RCU_NEXT_READY_TAIL))
> rcu_accelerate_cbs_unlocked(rnp, rdp);
It might also be that rcu_advance_cbs() is needed somewhere. I will
need to look carefully to work out whether either of them need to be
invoked at this particular point in the code.
Thanx, Paul
> - local_irq_restore(flags);
> + rcu_nocb_unlock_irqrestore(rdp, flags);
> }
>
> rcu_check_gp_start_stall(rnp, rdp, rcu_jiffies_till_stall_check());
> --
> 2.28.0
>
^ permalink raw reply [flat|nested] 32+ messages in thread
* Re: [RFC PATCH 12/12] rcu: Nocb (de)activate through sysfs
2020-09-21 12:43 ` [RFC PATCH 12/12] rcu: Nocb (de)activate through sysfs Frederic Weisbecker
@ 2020-09-22 0:26 ` Paul E. McKenney
2020-09-23 15:27 ` Frederic Weisbecker
0 siblings, 1 reply; 32+ messages in thread
From: Paul E. McKenney @ 2020-09-22 0:26 UTC (permalink / raw)
To: Frederic Weisbecker
Cc: LKML, Steven Rostedt, Mathieu Desnoyers, Lai Jiangshan,
Joel Fernandes, Josh Triplett
On Mon, Sep 21, 2020 at 02:43:51PM +0200, Frederic Weisbecker wrote:
> Not for merge.
>
> Make nocb toggable for a given CPU using:
> /sys/devices/system/cpu/cpu*/hotplug/nocb
>
> This is only intended for those who want to test this patchset. The real
> interfaces will be cpuset/isolation and rcutorture.
Excellent choice for testing in the near term!
How does rcutorture fare with this series if an additional kthread toggles
offloading on and off continually?
All questions and concerns aside, excellent progress and nice work!!!
Thanx, Paul
> Not-Signed-off-by: Frederic Weisbecker <frederic@kernel.org>
> Cc: Paul E. McKenney <paulmck@kernel.org>
> Cc: Josh Triplett <josh@joshtriplett.org>
> Cc: Steven Rostedt <rostedt@goodmis.org>
> Cc: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
> Cc: Lai Jiangshan <jiangshanlai@gmail.com>
> Cc: Joel Fernandes <joel@joelfernandes.org>
> ---
> kernel/cpu.c | 23 +++++++++++++++++++++++
> 1 file changed, 23 insertions(+)
>
> diff --git a/kernel/cpu.c b/kernel/cpu.c
> index 6ff2578ecf17..a36634113b8e 100644
> --- a/kernel/cpu.c
> +++ b/kernel/cpu.c
> @@ -2224,10 +2224,33 @@ static ssize_t show_cpuhp_fail(struct device *dev,
>
> static DEVICE_ATTR(fail, 0644, show_cpuhp_fail, write_cpuhp_fail);
>
> +static ssize_t write_nocb(struct device *dev,
> + struct device_attribute *attr,
> + const char *buf, size_t count)
> +{
> + int val, ret;
> +
> + ret = kstrtoint(buf, 10, &val);
> + if (ret)
> + return ret;
> +
> + if (val == 0)
> + rcu_nocb_cpu_deoffload(dev->id);
> + else if (val == 1)
> + rcu_nocb_cpu_offload(dev->id);
> + else
> + return -EINVAL;
> +
> + return count;
> +}
> +
> +static DEVICE_ATTR(nocb, 0644, NULL, write_nocb);
> +
> static struct attribute *cpuhp_cpu_attrs[] = {
> &dev_attr_state.attr,
> &dev_attr_target.attr,
> &dev_attr_fail.attr,
> + &dev_attr_nocb.attr,
> NULL
> };
>
> --
> 2.28.0
>
^ permalink raw reply [flat|nested] 32+ messages in thread
* Re: [RFC PATCH 01/12] rcu: Implement rcu_segcblist_is_offloaded() config dependent
2020-09-21 12:43 ` [RFC PATCH 01/12] rcu: Implement rcu_segcblist_is_offloaded() config dependent Frederic Weisbecker
@ 2020-09-22 0:27 ` Paul E. McKenney
2020-09-22 21:43 ` Frederic Weisbecker
0 siblings, 1 reply; 32+ messages in thread
From: Paul E. McKenney @ 2020-09-22 0:27 UTC (permalink / raw)
To: Frederic Weisbecker
Cc: LKML, Steven Rostedt, Mathieu Desnoyers, Lai Jiangshan,
Joel Fernandes, Josh Triplett
On Mon, Sep 21, 2020 at 02:43:40PM +0200, Frederic Weisbecker wrote:
> This simplify the usage of this API and avoid checking the kernel
> config from the callers.
>
> Suggested-by: Paul E. McKenney <paulmck@kernel.org>
> Signed-off-by: Frederic Weisbecker <frederic@kernel.org>
> Cc: Paul E. McKenney <paulmck@kernel.org>
> Cc: Josh Triplett <josh@joshtriplett.org>
> Cc: Steven Rostedt <rostedt@goodmis.org>
> Cc: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
> Cc: Lai Jiangshan <jiangshanlai@gmail.com>
> Cc: Joel Fernandes <joel@joelfernandes.org>
Nice cleanup! I clearly should have done it this way to start with.
Any reason I shouldn't pull this into -rcu right now?
Thanx, Paul
> ---
> kernel/rcu/rcu_segcblist.h | 2 +-
> kernel/rcu/tree.c | 21 +++++++--------------
> 2 files changed, 8 insertions(+), 15 deletions(-)
>
> diff --git a/kernel/rcu/rcu_segcblist.h b/kernel/rcu/rcu_segcblist.h
> index 5c293afc07b8..492262bcb591 100644
> --- a/kernel/rcu/rcu_segcblist.h
> +++ b/kernel/rcu/rcu_segcblist.h
> @@ -62,7 +62,7 @@ static inline bool rcu_segcblist_is_enabled(struct rcu_segcblist *rsclp)
> /* Is the specified rcu_segcblist offloaded? */
> static inline bool rcu_segcblist_is_offloaded(struct rcu_segcblist *rsclp)
> {
> - return rsclp->offloaded;
> + return IS_ENABLED(CONFIG_RCU_NOCB_CPU) && rsclp->offloaded;
> }
>
> /*
> diff --git a/kernel/rcu/tree.c b/kernel/rcu/tree.c
> index 8ce77d9ac716..c0286ce8fc03 100644
> --- a/kernel/rcu/tree.c
> +++ b/kernel/rcu/tree.c
> @@ -1583,8 +1583,7 @@ static bool __note_gp_changes(struct rcu_node *rnp, struct rcu_data *rdp)
> {
> bool ret = false;
> bool need_qs;
> - const bool offloaded = IS_ENABLED(CONFIG_RCU_NOCB_CPU) &&
> - rcu_segcblist_is_offloaded(&rdp->cblist);
> + const bool offloaded = rcu_segcblist_is_offloaded(&rdp->cblist);
>
> raw_lockdep_assert_held_rcu_node(rnp);
>
> @@ -2011,8 +2010,7 @@ static void rcu_gp_cleanup(void)
> needgp = true;
> }
> /* Advance CBs to reduce false positives below. */
> - offloaded = IS_ENABLED(CONFIG_RCU_NOCB_CPU) &&
> - rcu_segcblist_is_offloaded(&rdp->cblist);
> + offloaded = rcu_segcblist_is_offloaded(&rdp->cblist);
> if ((offloaded || !rcu_accelerate_cbs(rnp, rdp)) && needgp) {
> WRITE_ONCE(rcu_state.gp_flags, RCU_GP_FLAG_INIT);
> WRITE_ONCE(rcu_state.gp_req_activity, jiffies);
> @@ -2207,8 +2205,7 @@ rcu_report_qs_rdp(int cpu, struct rcu_data *rdp)
> unsigned long flags;
> unsigned long mask;
> bool needwake = false;
> - const bool offloaded = IS_ENABLED(CONFIG_RCU_NOCB_CPU) &&
> - rcu_segcblist_is_offloaded(&rdp->cblist);
> + const bool offloaded = rcu_segcblist_is_offloaded(&rdp->cblist);
> struct rcu_node *rnp;
>
> rnp = rdp->mynode;
> @@ -2375,8 +2372,7 @@ int rcutree_dead_cpu(unsigned int cpu)
> static void rcu_do_batch(struct rcu_data *rdp)
> {
> unsigned long flags;
> - const bool offloaded = IS_ENABLED(CONFIG_RCU_NOCB_CPU) &&
> - rcu_segcblist_is_offloaded(&rdp->cblist);
> + const bool offloaded = rcu_segcblist_is_offloaded(&rdp->cblist);
> struct rcu_head *rhp;
> struct rcu_cblist rcl = RCU_CBLIST_INITIALIZER(rcl);
> long bl, count;
> @@ -2620,8 +2616,7 @@ static __latent_entropy void rcu_core(void)
> unsigned long flags;
> struct rcu_data *rdp = raw_cpu_ptr(&rcu_data);
> struct rcu_node *rnp = rdp->mynode;
> - const bool offloaded = IS_ENABLED(CONFIG_RCU_NOCB_CPU) &&
> - rcu_segcblist_is_offloaded(&rdp->cblist);
> + const bool offloaded = rcu_segcblist_is_offloaded(&rdp->cblist);
>
> if (cpu_is_offline(smp_processor_id()))
> return;
> @@ -2919,8 +2914,7 @@ __call_rcu(struct rcu_head *head, rcu_callback_t func)
> rcu_segcblist_n_cbs(&rdp->cblist));
>
> /* Go handle any RCU core processing required. */
> - if (IS_ENABLED(CONFIG_RCU_NOCB_CPU) &&
> - unlikely(rcu_segcblist_is_offloaded(&rdp->cblist))) {
> + if (unlikely(rcu_segcblist_is_offloaded(&rdp->cblist))) {
> __call_rcu_nocb_wake(rdp, was_alldone, flags); /* unlocks */
> } else {
> __call_rcu_core(rdp, head, flags);
> @@ -3655,8 +3649,7 @@ static int rcu_pending(int user)
>
> /* Has RCU gone idle with this CPU needing another grace period? */
> if (!gp_in_progress && rcu_segcblist_is_enabled(&rdp->cblist) &&
> - (!IS_ENABLED(CONFIG_RCU_NOCB_CPU) ||
> - !rcu_segcblist_is_offloaded(&rdp->cblist)) &&
> + !rcu_segcblist_is_offloaded(&rdp->cblist) &&
> !rcu_segcblist_restempty(&rdp->cblist, RCU_NEXT_READY_TAIL))
> return 1;
>
> --
> 2.28.0
>
^ permalink raw reply [flat|nested] 32+ messages in thread
* Re: [RFC PATCH 01/12] rcu: Implement rcu_segcblist_is_offloaded() config dependent
2020-09-22 0:27 ` Paul E. McKenney
@ 2020-09-22 21:43 ` Frederic Weisbecker
2020-09-22 23:11 ` Paul E. McKenney
0 siblings, 1 reply; 32+ messages in thread
From: Frederic Weisbecker @ 2020-09-22 21:43 UTC (permalink / raw)
To: Paul E. McKenney
Cc: LKML, Steven Rostedt, Mathieu Desnoyers, Lai Jiangshan,
Joel Fernandes, Josh Triplett
On Mon, Sep 21, 2020 at 05:27:32PM -0700, Paul E. McKenney wrote:
> On Mon, Sep 21, 2020 at 02:43:40PM +0200, Frederic Weisbecker wrote:
> > This simplify the usage of this API and avoid checking the kernel
> > config from the callers.
> >
> > Suggested-by: Paul E. McKenney <paulmck@kernel.org>
> > Signed-off-by: Frederic Weisbecker <frederic@kernel.org>
> > Cc: Paul E. McKenney <paulmck@kernel.org>
> > Cc: Josh Triplett <josh@joshtriplett.org>
> > Cc: Steven Rostedt <rostedt@goodmis.org>
> > Cc: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
> > Cc: Lai Jiangshan <jiangshanlai@gmail.com>
> > Cc: Joel Fernandes <joel@joelfernandes.org>
>
> Nice cleanup! I clearly should have done it this way to start with.
>
> Any reason I shouldn't pull this into -rcu right now?
I think that very one can be applied indeed.
Thanks!
^ permalink raw reply [flat|nested] 32+ messages in thread
* Re: [RFC PATCH 01/12] rcu: Implement rcu_segcblist_is_offloaded() config dependent
2020-09-22 21:43 ` Frederic Weisbecker
@ 2020-09-22 23:11 ` Paul E. McKenney
2020-09-23 15:25 ` Frederic Weisbecker
0 siblings, 1 reply; 32+ messages in thread
From: Paul E. McKenney @ 2020-09-22 23:11 UTC (permalink / raw)
To: Frederic Weisbecker
Cc: LKML, Steven Rostedt, Mathieu Desnoyers, Lai Jiangshan,
Joel Fernandes, Josh Triplett
On Tue, Sep 22, 2020 at 11:43:26PM +0200, Frederic Weisbecker wrote:
> On Mon, Sep 21, 2020 at 05:27:32PM -0700, Paul E. McKenney wrote:
> > On Mon, Sep 21, 2020 at 02:43:40PM +0200, Frederic Weisbecker wrote:
> > > This simplify the usage of this API and avoid checking the kernel
> > > config from the callers.
> > >
> > > Suggested-by: Paul E. McKenney <paulmck@kernel.org>
> > > Signed-off-by: Frederic Weisbecker <frederic@kernel.org>
> > > Cc: Paul E. McKenney <paulmck@kernel.org>
> > > Cc: Josh Triplett <josh@joshtriplett.org>
> > > Cc: Steven Rostedt <rostedt@goodmis.org>
> > > Cc: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
> > > Cc: Lai Jiangshan <jiangshanlai@gmail.com>
> > > Cc: Joel Fernandes <joel@joelfernandes.org>
> >
> > Nice cleanup! I clearly should have done it this way to start with.
> >
> > Any reason I shouldn't pull this into -rcu right now?
>
> I think that very one can be applied indeed.
Very well! I had to hand-apply it due to recent -rcu thrash, and as
usual I could not resist wordsmithing the commit log. Please let
me know if I messed anything up.
Thanx, Paul
------------------------------------------------------------------------
commit 99a2a9b00b1f44019e5667cdbd44b8eb82e9f0a9
Author: Frederic Weisbecker <frederic@kernel.org>
Date: Mon Sep 21 14:43:40 2020 +0200
rcu: Implement rcu_segcblist_is_offloaded() config dependent
This commit simplifies the use of the rcu_segcblist_is_offloaded() API so
that its callers no longer need to check the RCU_NOCB_CPU Kconfig option.
Note that rcu_segcblist_is_offloaded() is defined in the header file,
which means that the generated code should be just as efficient as before.
Suggested-by: Paul E. McKenney <paulmck@kernel.org>
Signed-off-by: Frederic Weisbecker <frederic@kernel.org>
Cc: Paul E. McKenney <paulmck@kernel.org>
Cc: Josh Triplett <josh@joshtriplett.org>
Cc: Steven Rostedt <rostedt@goodmis.org>
Cc: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Cc: Lai Jiangshan <jiangshanlai@gmail.com>
Cc: Joel Fernandes <joel@joelfernandes.org>
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
diff --git a/kernel/rcu/rcu_segcblist.h b/kernel/rcu/rcu_segcblist.h
index 5c293af..492262b 100644
--- a/kernel/rcu/rcu_segcblist.h
+++ b/kernel/rcu/rcu_segcblist.h
@@ -62,7 +62,7 @@ static inline bool rcu_segcblist_is_enabled(struct rcu_segcblist *rsclp)
/* Is the specified rcu_segcblist offloaded? */
static inline bool rcu_segcblist_is_offloaded(struct rcu_segcblist *rsclp)
{
- return rsclp->offloaded;
+ return IS_ENABLED(CONFIG_RCU_NOCB_CPU) && rsclp->offloaded;
}
/*
diff --git a/kernel/rcu/tree.c b/kernel/rcu/tree.c
index 2424e2a..6d9ec8e 100644
--- a/kernel/rcu/tree.c
+++ b/kernel/rcu/tree.c
@@ -1609,8 +1609,7 @@ static bool __note_gp_changes(struct rcu_node *rnp, struct rcu_data *rdp)
{
bool ret = false;
bool need_qs;
- const bool offloaded = IS_ENABLED(CONFIG_RCU_NOCB_CPU) &&
- rcu_segcblist_is_offloaded(&rdp->cblist);
+ const bool offloaded = rcu_segcblist_is_offloaded(&rdp->cblist);
raw_lockdep_assert_held_rcu_node(rnp);
@@ -2054,8 +2053,7 @@ static void rcu_gp_cleanup(void)
needgp = true;
}
/* Advance CBs to reduce false positives below. */
- offloaded = IS_ENABLED(CONFIG_RCU_NOCB_CPU) &&
- rcu_segcblist_is_offloaded(&rdp->cblist);
+ offloaded = rcu_segcblist_is_offloaded(&rdp->cblist);
if ((offloaded || !rcu_accelerate_cbs(rnp, rdp)) && needgp) {
WRITE_ONCE(rcu_state.gp_flags, RCU_GP_FLAG_INIT);
WRITE_ONCE(rcu_state.gp_req_activity, jiffies);
@@ -2254,8 +2252,7 @@ rcu_report_qs_rdp(struct rcu_data *rdp)
unsigned long flags;
unsigned long mask;
bool needwake = false;
- const bool offloaded = IS_ENABLED(CONFIG_RCU_NOCB_CPU) &&
- rcu_segcblist_is_offloaded(&rdp->cblist);
+ const bool offloaded = rcu_segcblist_is_offloaded(&rdp->cblist);
struct rcu_node *rnp;
WARN_ON_ONCE(rdp->cpu != smp_processor_id());
@@ -2423,8 +2420,7 @@ static void rcu_do_batch(struct rcu_data *rdp)
{
int div;
unsigned long flags;
- const bool offloaded = IS_ENABLED(CONFIG_RCU_NOCB_CPU) &&
- rcu_segcblist_is_offloaded(&rdp->cblist);
+ const bool offloaded = rcu_segcblist_is_offloaded(&rdp->cblist);
struct rcu_head *rhp;
struct rcu_cblist rcl = RCU_CBLIST_INITIALIZER(rcl);
long bl, count;
@@ -2681,8 +2677,7 @@ static __latent_entropy void rcu_core(void)
unsigned long flags;
struct rcu_data *rdp = raw_cpu_ptr(&rcu_data);
struct rcu_node *rnp = rdp->mynode;
- const bool offloaded = IS_ENABLED(CONFIG_RCU_NOCB_CPU) &&
- rcu_segcblist_is_offloaded(&rdp->cblist);
+ const bool offloaded = rcu_segcblist_is_offloaded(&rdp->cblist);
if (cpu_is_offline(smp_processor_id()))
return;
@@ -2984,8 +2979,7 @@ __call_rcu(struct rcu_head *head, rcu_callback_t func)
rcu_segcblist_n_cbs(&rdp->cblist));
/* Go handle any RCU core processing required. */
- if (IS_ENABLED(CONFIG_RCU_NOCB_CPU) &&
- unlikely(rcu_segcblist_is_offloaded(&rdp->cblist))) {
+ if (unlikely(rcu_segcblist_is_offloaded(&rdp->cblist))) {
__call_rcu_nocb_wake(rdp, was_alldone, flags); /* unlocks */
} else {
__call_rcu_core(rdp, head, flags);
@@ -3739,8 +3733,7 @@ static int rcu_pending(int user)
/* Has RCU gone idle with this CPU needing another grace period? */
if (!gp_in_progress && rcu_segcblist_is_enabled(&rdp->cblist) &&
- (!IS_ENABLED(CONFIG_RCU_NOCB_CPU) ||
- !rcu_segcblist_is_offloaded(&rdp->cblist)) &&
+ !rcu_segcblist_is_offloaded(&rdp->cblist) &&
!rcu_segcblist_restempty(&rdp->cblist, RCU_NEXT_READY_TAIL))
return 1;
^ permalink raw reply related [flat|nested] 32+ messages in thread
* Re: [RFC PATCH 01/12] rcu: Implement rcu_segcblist_is_offloaded() config dependent
2020-09-22 23:11 ` Paul E. McKenney
@ 2020-09-23 15:25 ` Frederic Weisbecker
0 siblings, 0 replies; 32+ messages in thread
From: Frederic Weisbecker @ 2020-09-23 15:25 UTC (permalink / raw)
To: Paul E. McKenney
Cc: LKML, Steven Rostedt, Mathieu Desnoyers, Lai Jiangshan,
Joel Fernandes, Josh Triplett
On Tue, Sep 22, 2020 at 04:11:50PM -0700, Paul E. McKenney wrote:
> On Tue, Sep 22, 2020 at 11:43:26PM +0200, Frederic Weisbecker wrote:
> > On Mon, Sep 21, 2020 at 05:27:32PM -0700, Paul E. McKenney wrote:
> > > On Mon, Sep 21, 2020 at 02:43:40PM +0200, Frederic Weisbecker wrote:
> > > > This simplify the usage of this API and avoid checking the kernel
> > > > config from the callers.
> > > >
> > > > Suggested-by: Paul E. McKenney <paulmck@kernel.org>
> > > > Signed-off-by: Frederic Weisbecker <frederic@kernel.org>
> > > > Cc: Paul E. McKenney <paulmck@kernel.org>
> > > > Cc: Josh Triplett <josh@joshtriplett.org>
> > > > Cc: Steven Rostedt <rostedt@goodmis.org>
> > > > Cc: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
> > > > Cc: Lai Jiangshan <jiangshanlai@gmail.com>
> > > > Cc: Joel Fernandes <joel@joelfernandes.org>
> > >
> > > Nice cleanup! I clearly should have done it this way to start with.
> > >
> > > Any reason I shouldn't pull this into -rcu right now?
> >
> > I think that very one can be applied indeed.
>
> Very well! I had to hand-apply it due to recent -rcu thrash, and as
> usual I could not resist wordsmithing the commit log. Please let
> me know if I messed anything up.
Looks very good, thanks!
^ permalink raw reply [flat|nested] 32+ messages in thread
* Re: [RFC PATCH 12/12] rcu: Nocb (de)activate through sysfs
2020-09-22 0:26 ` Paul E. McKenney
@ 2020-09-23 15:27 ` Frederic Weisbecker
2020-09-23 15:47 ` Paul E. McKenney
0 siblings, 1 reply; 32+ messages in thread
From: Frederic Weisbecker @ 2020-09-23 15:27 UTC (permalink / raw)
To: Paul E. McKenney
Cc: LKML, Steven Rostedt, Mathieu Desnoyers, Lai Jiangshan,
Joel Fernandes, Josh Triplett
On Mon, Sep 21, 2020 at 05:26:29PM -0700, Paul E. McKenney wrote:
> On Mon, Sep 21, 2020 at 02:43:51PM +0200, Frederic Weisbecker wrote:
> > Not for merge.
> >
> > Make nocb toggable for a given CPU using:
> > /sys/devices/system/cpu/cpu*/hotplug/nocb
> >
> > This is only intended for those who want to test this patchset. The real
> > interfaces will be cpuset/isolation and rcutorture.
>
> Excellent choice for testing in the near term!
>
> How does rcutorture fare with this series if an additional kthread toggles
> offloading on and off continually?
So yeah that would be the plan: having something that randomly toggles
offloading from the RCU tests themselves. I'm not quite familiar with
rcutorture internals yet though.
>
> All questions and concerns aside, excellent progress and nice work!!!
Thanks! We are slowly making progress :)
^ permalink raw reply [flat|nested] 32+ messages in thread
* Re: [RFC PATCH 07/12] rcu: Shutdown nocb timer on de-offloading
2020-09-22 0:17 ` Paul E. McKenney
@ 2020-09-23 15:29 ` Frederic Weisbecker
2020-09-23 15:47 ` Paul E. McKenney
0 siblings, 1 reply; 32+ messages in thread
From: Frederic Weisbecker @ 2020-09-23 15:29 UTC (permalink / raw)
To: Paul E. McKenney
Cc: LKML, Steven Rostedt, Mathieu Desnoyers, Lai Jiangshan,
Joel Fernandes, Josh Triplett
On Mon, Sep 21, 2020 at 05:17:56PM -0700, Paul E. McKenney wrote:
> On Mon, Sep 21, 2020 at 02:43:46PM +0200, Frederic Weisbecker wrote:
> > Make sure the nocb timer can't fire anymore before we reach the final
> > de-offload state. Spuriously waking up the GP kthread is no big deal but
> > we must prevent from executing the timer callback without nocb locking.
>
> If we had just the previous patch and not this patch, would things break?
> Or are you relying on the fact that there is not yet a connection to
> userspace controls for this functionality?
Exactly it shouldn't break because only the last patch makes the code
eventually used.
Thanks.
^ permalink raw reply [flat|nested] 32+ messages in thread
* Re: [RFC PATCH 05/12] rcu: De-offloading GP kthread
2020-09-22 0:10 ` Paul E. McKenney
@ 2020-09-23 15:31 ` Frederic Weisbecker
2020-09-23 15:48 ` Paul E. McKenney
0 siblings, 1 reply; 32+ messages in thread
From: Frederic Weisbecker @ 2020-09-23 15:31 UTC (permalink / raw)
To: Paul E. McKenney
Cc: LKML, Steven Rostedt, Mathieu Desnoyers, Lai Jiangshan,
Joel Fernandes, Josh Triplett
On Mon, Sep 21, 2020 at 05:10:15PM -0700, Paul E. McKenney wrote:
> On Mon, Sep 21, 2020 at 02:43:44PM +0200, Frederic Weisbecker wrote:
> > @@ -2292,6 +2340,7 @@ void __init rcu_init_nohz(void)
> > rcu_segcblist_init(&rdp->cblist);
> > rcu_segcblist_offload(&rdp->cblist, true);
> > rcu_segcblist_set_flags(&rdp->cblist, SEGCBLIST_KTHREAD_CB);
> > + rcu_segcblist_set_flags(&rdp->cblist, SEGCBLIST_KTHREAD_GP);
>
> OK, I will bite at this nit...
>
> Why not "SEGCBLIST_KTHREAD_CB | SEGCBLIST_KTHREAD_GP"?
That spared a broken line :o)
But you're right, I'll do that.
>
> Thanx, Paul
>
> > }
> > rcu_organize_nocb_kthreads();
> > }
> > --
> > 2.28.0
> >
^ permalink raw reply [flat|nested] 32+ messages in thread
* Re: [RFC PATCH 12/12] rcu: Nocb (de)activate through sysfs
2020-09-23 15:27 ` Frederic Weisbecker
@ 2020-09-23 15:47 ` Paul E. McKenney
2020-09-23 19:00 ` Frederic Weisbecker
0 siblings, 1 reply; 32+ messages in thread
From: Paul E. McKenney @ 2020-09-23 15:47 UTC (permalink / raw)
To: Frederic Weisbecker
Cc: LKML, Steven Rostedt, Mathieu Desnoyers, Lai Jiangshan,
Joel Fernandes, Josh Triplett
On Wed, Sep 23, 2020 at 05:27:46PM +0200, Frederic Weisbecker wrote:
> On Mon, Sep 21, 2020 at 05:26:29PM -0700, Paul E. McKenney wrote:
> > On Mon, Sep 21, 2020 at 02:43:51PM +0200, Frederic Weisbecker wrote:
> > > Not for merge.
> > >
> > > Make nocb toggable for a given CPU using:
> > > /sys/devices/system/cpu/cpu*/hotplug/nocb
> > >
> > > This is only intended for those who want to test this patchset. The real
> > > interfaces will be cpuset/isolation and rcutorture.
> >
> > Excellent choice for testing in the near term!
> >
> > How does rcutorture fare with this series if an additional kthread toggles
> > offloading on and off continually?
>
> So yeah that would be the plan: having something that randomly toggles
> offloading from the RCU tests themselves. I'm not quite familiar with
> rcutorture internals yet though.
If you supply a function that selects a CPU and does the toggle, I
can quickly and easily wire it into rcutorture. I of course have no
objection to your learning to do this yourself, but the offer stands.
> > All questions and concerns aside, excellent progress and nice work!!!
>
> Thanks! We are slowly making progress :)
Better than quickly failing to make progress, to be sure! ;-)
Thanx, Paul
^ permalink raw reply [flat|nested] 32+ messages in thread
* Re: [RFC PATCH 07/12] rcu: Shutdown nocb timer on de-offloading
2020-09-23 15:29 ` Frederic Weisbecker
@ 2020-09-23 15:47 ` Paul E. McKenney
0 siblings, 0 replies; 32+ messages in thread
From: Paul E. McKenney @ 2020-09-23 15:47 UTC (permalink / raw)
To: Frederic Weisbecker
Cc: LKML, Steven Rostedt, Mathieu Desnoyers, Lai Jiangshan,
Joel Fernandes, Josh Triplett
On Wed, Sep 23, 2020 at 05:29:52PM +0200, Frederic Weisbecker wrote:
> On Mon, Sep 21, 2020 at 05:17:56PM -0700, Paul E. McKenney wrote:
> > On Mon, Sep 21, 2020 at 02:43:46PM +0200, Frederic Weisbecker wrote:
> > > Make sure the nocb timer can't fire anymore before we reach the final
> > > de-offload state. Spuriously waking up the GP kthread is no big deal but
> > > we must prevent from executing the timer callback without nocb locking.
> >
> > If we had just the previous patch and not this patch, would things break?
> > Or are you relying on the fact that there is not yet a connection to
> > userspace controls for this functionality?
>
> Exactly it shouldn't break because only the last patch makes the code
> eventually used.
That works, thank you!
Thanx, Paul
^ permalink raw reply [flat|nested] 32+ messages in thread
* Re: [RFC PATCH 05/12] rcu: De-offloading GP kthread
2020-09-23 15:31 ` Frederic Weisbecker
@ 2020-09-23 15:48 ` Paul E. McKenney
0 siblings, 0 replies; 32+ messages in thread
From: Paul E. McKenney @ 2020-09-23 15:48 UTC (permalink / raw)
To: Frederic Weisbecker
Cc: LKML, Steven Rostedt, Mathieu Desnoyers, Lai Jiangshan,
Joel Fernandes, Josh Triplett
On Wed, Sep 23, 2020 at 05:31:32PM +0200, Frederic Weisbecker wrote:
> On Mon, Sep 21, 2020 at 05:10:15PM -0700, Paul E. McKenney wrote:
> > On Mon, Sep 21, 2020 at 02:43:44PM +0200, Frederic Weisbecker wrote:
> > > @@ -2292,6 +2340,7 @@ void __init rcu_init_nohz(void)
> > > rcu_segcblist_init(&rdp->cblist);
> > > rcu_segcblist_offload(&rdp->cblist, true);
> > > rcu_segcblist_set_flags(&rdp->cblist, SEGCBLIST_KTHREAD_CB);
> > > + rcu_segcblist_set_flags(&rdp->cblist, SEGCBLIST_KTHREAD_GP);
> >
> > OK, I will bite at this nit...
> >
> > Why not "SEGCBLIST_KTHREAD_CB | SEGCBLIST_KTHREAD_GP"?
>
> That spared a broken line :o)
>
> But you're right, I'll do that.
Either way is fine, just curious. Your choice!
Thanx, Paul
^ permalink raw reply [flat|nested] 32+ messages in thread
* Re: [RFC PATCH 12/12] rcu: Nocb (de)activate through sysfs
2020-09-23 15:47 ` Paul E. McKenney
@ 2020-09-23 19:00 ` Frederic Weisbecker
2020-09-24 0:38 ` Paul E. McKenney
0 siblings, 1 reply; 32+ messages in thread
From: Frederic Weisbecker @ 2020-09-23 19:00 UTC (permalink / raw)
To: Paul E. McKenney
Cc: LKML, Steven Rostedt, Mathieu Desnoyers, Lai Jiangshan,
Joel Fernandes, Josh Triplett
On Wed, Sep 23, 2020 at 08:47:16AM -0700, Paul E. McKenney wrote:
> On Wed, Sep 23, 2020 at 05:27:46PM +0200, Frederic Weisbecker wrote:
> > On Mon, Sep 21, 2020 at 05:26:29PM -0700, Paul E. McKenney wrote:
> > > On Mon, Sep 21, 2020 at 02:43:51PM +0200, Frederic Weisbecker wrote:
> > > > Not for merge.
> > > >
> > > > Make nocb toggable for a given CPU using:
> > > > /sys/devices/system/cpu/cpu*/hotplug/nocb
> > > >
> > > > This is only intended for those who want to test this patchset. The real
> > > > interfaces will be cpuset/isolation and rcutorture.
> > >
> > > Excellent choice for testing in the near term!
> > >
> > > How does rcutorture fare with this series if an additional kthread toggles
> > > offloading on and off continually?
> >
> > So yeah that would be the plan: having something that randomly toggles
> > offloading from the RCU tests themselves. I'm not quite familiar with
> > rcutorture internals yet though.
>
> If you supply a function that selects a CPU and does the toggle, I
> can quickly and easily wire it into rcutorture. I of course have no
> objection to your learning to do this yourself, but the offer stands.
I gladly accept your proposal! This way I will integrate it in the set
and start testing.
The functions to call would be:
void rcu_nocb_cpu_offload(int cpu);
void rcu_nocb_cpu_deoffload(int cpu);
>
> > > All questions and concerns aside, excellent progress and nice work!!!
> >
> > Thanks! We are slowly making progress :)
>
> Better than quickly failing to make progress, to be sure! ;-)
;-)
Thanks a lot!
^ permalink raw reply [flat|nested] 32+ messages in thread
* Re: [RFC PATCH 12/12] rcu: Nocb (de)activate through sysfs
2020-09-23 19:00 ` Frederic Weisbecker
@ 2020-09-24 0:38 ` Paul E. McKenney
0 siblings, 0 replies; 32+ messages in thread
From: Paul E. McKenney @ 2020-09-24 0:38 UTC (permalink / raw)
To: Frederic Weisbecker
Cc: LKML, Steven Rostedt, Mathieu Desnoyers, Lai Jiangshan,
Joel Fernandes, Josh Triplett
On Wed, Sep 23, 2020 at 09:00:50PM +0200, Frederic Weisbecker wrote:
> On Wed, Sep 23, 2020 at 08:47:16AM -0700, Paul E. McKenney wrote:
> > On Wed, Sep 23, 2020 at 05:27:46PM +0200, Frederic Weisbecker wrote:
> > > On Mon, Sep 21, 2020 at 05:26:29PM -0700, Paul E. McKenney wrote:
> > > > On Mon, Sep 21, 2020 at 02:43:51PM +0200, Frederic Weisbecker wrote:
> > > > > Not for merge.
> > > > >
> > > > > Make nocb toggable for a given CPU using:
> > > > > /sys/devices/system/cpu/cpu*/hotplug/nocb
> > > > >
> > > > > This is only intended for those who want to test this patchset. The real
> > > > > interfaces will be cpuset/isolation and rcutorture.
> > > >
> > > > Excellent choice for testing in the near term!
> > > >
> > > > How does rcutorture fare with this series if an additional kthread toggles
> > > > offloading on and off continually?
> > >
> > > So yeah that would be the plan: having something that randomly toggles
> > > offloading from the RCU tests themselves. I'm not quite familiar with
> > > rcutorture internals yet though.
> >
> > If you supply a function that selects a CPU and does the toggle, I
> > can quickly and easily wire it into rcutorture. I of course have no
> > objection to your learning to do this yourself, but the offer stands.
>
> I gladly accept your proposal! This way I will integrate it in the set
> and start testing.
>
> The functions to call would be:
>
> void rcu_nocb_cpu_offload(int cpu);
> void rcu_nocb_cpu_deoffload(int cpu);
I freely confess that I am surprised that there are no error returns
(offline CPU? non-existent CPU?), but here is a first-cut patch to
rcutorture that tests the above interfaces.
Thoughts?
Thanx, Paul
------------------------------------------------------------------------
diff --git a/Documentation/admin-guide/kernel-parameters.txt b/Documentation/admin-guide/kernel-parameters.txt
index 156f033..6528c48 100644
--- a/Documentation/admin-guide/kernel-parameters.txt
+++ b/Documentation/admin-guide/kernel-parameters.txt
@@ -4302,6 +4302,14 @@
stress RCU, they don't participate in the actual
test, hence the "fake".
+ rcutorture.nocbs_nthreads= [KNL]
+ Set number of RCU callback-offload togglers.
+ Zero (the default) disables toggling.
+
+ rcutorture.nocbs_toggle= [KNL]
+ Set the delay in milliseconds between successive
+ callback-offload toggling attempts.
+
rcutorture.nreaders= [KNL]
Set number of RCU readers. The value -1 selects
N-1, where N is the number of CPUs. A value
diff --git a/kernel/rcu/rcutorture.c b/kernel/rcu/rcutorture.c
index 1f1db7a..beba9e7 100644
--- a/kernel/rcu/rcutorture.c
+++ b/kernel/rcu/rcutorture.c
@@ -97,6 +97,8 @@ torture_param(int, object_debug, 0,
torture_param(int, onoff_holdoff, 0, "Time after boot before CPU hotplugs (s)");
torture_param(int, onoff_interval, 0,
"Time between CPU hotplugs (jiffies), 0=disable");
+torture_param(int, nocbs_nthreads, 0, "Number of NOCB toggle threads, 0 to disable");
+torture_param(int, nocbs_toggle, 1000, "Time between toggling nocb state (ms)");
torture_param(int, read_exit_delay, 13,
"Delay between read-then-exit episodes (s)");
torture_param(int, read_exit_burst, 16,
@@ -127,10 +129,12 @@ static char *torture_type = "rcu";
module_param(torture_type, charp, 0444);
MODULE_PARM_DESC(torture_type, "Type of RCU to torture (rcu, srcu, ...)");
+static int nrealnocbers;
static int nrealreaders;
static struct task_struct *writer_task;
static struct task_struct **fakewriter_tasks;
static struct task_struct **reader_tasks;
+static struct task_struct **nocb_tasks;
static struct task_struct *stats_task;
static struct task_struct *fqs_task;
static struct task_struct *boost_tasks[NR_CPUS];
@@ -174,6 +178,8 @@ static unsigned long n_read_exits;
static struct list_head rcu_torture_removed;
static unsigned long shutdown_jiffies;
static unsigned long start_gp_seq;
+static atomic_long_t n_nocb_offload;
+static atomic_long_t n_nocb_deoffload;
static int rcu_torture_writer_state;
#define RTWS_FIXED_DELAY 0
@@ -1498,6 +1504,56 @@ rcu_torture_reader(void *arg)
return 0;
}
+void __weak rcu_nocb_cpu_offload(int cpu) {}
+void __weak rcu_nocb_cpu_deoffload(int cpu) {}
+
+/*
+ * Randomly Toggle CPUs' callback-offload state. This uses hrtimers to
+ * increase race probabilities and fuzzes the interval between toggling.
+ */
+static int rcu_nocb_toggle(void *arg)
+{
+ int cpu;
+ int maxcpu = -1;
+ int oldnice = task_nice(current);
+ long r;
+ DEFINE_TORTURE_RANDOM(rand);
+ ktime_t toggle_delay;
+ unsigned long toggle_fuzz;
+ ktime_t toggle_interval = ms_to_ktime(nocbs_toggle);
+
+ VERBOSE_TOROUT_STRING("rcu_nocb_toggle task started");
+ while (!rcu_inkernel_boot_has_ended())
+ schedule_timeout_interruptible(HZ / 10);
+ for_each_online_cpu(cpu)
+ maxcpu = cpu;
+ WARN_ON(maxcpu < 0);
+ if (toggle_interval > ULONG_MAX)
+ toggle_fuzz = ULONG_MAX >> 3;
+ else
+ toggle_fuzz = toggle_interval >> 3;
+ if (toggle_fuzz <= 0)
+ toggle_fuzz = NSEC_PER_USEC;
+ do {
+ r = torture_random(&rand);
+ cpu = (r >> 4) % (maxcpu + 1);
+ if (r & 0x1) {
+ rcu_nocb_cpu_offload(cpu);
+ atomic_long_inc(&n_nocb_offload);
+ } else {
+ rcu_nocb_cpu_deoffload(cpu);
+ atomic_long_inc(&n_nocb_deoffload);
+ }
+ toggle_delay = torture_random(&rand) % toggle_fuzz + toggle_interval;
+ set_current_state(TASK_INTERRUPTIBLE);
+ schedule_hrtimeout(&toggle_delay, HRTIMER_MODE_REL);
+ if (stutter_wait("rcu_nocb_toggle"))
+ sched_set_normal(current, oldnice);
+ } while (!torture_must_stop());
+ torture_kthread_stopping("rcu_nocb_toggle");
+ return 0;
+}
+
/*
* Print torture statistics. Caller must ensure that there is only
* one call to this function at a given time!!! This is normally
@@ -1553,7 +1609,9 @@ rcu_torture_stats_print(void)
data_race(n_barrier_successes),
data_race(n_barrier_attempts),
data_race(n_rcu_torture_barrier_error));
- pr_cont("read-exits: %ld\n", data_race(n_read_exits));
+ pr_cont("read-exits: %ld ", data_race(n_read_exits));
+ pr_cont("nocb-toggles: %ld:%ld\n",
+ atomic_long_read(&n_nocb_offload), atomic_long_read(&n_nocb_deoffload));
pr_alert("%s%s ", torture_type, TORTURE_FLAG);
if (atomic_read(&n_rcu_torture_mberror) ||
@@ -1647,7 +1705,8 @@ rcu_torture_print_module_parms(struct rcu_torture_ops *cur_ops, const char *tag)
"stall_cpu_block=%d "
"n_barrier_cbs=%d "
"onoff_interval=%d onoff_holdoff=%d "
- "read_exit_delay=%d read_exit_burst=%d\n",
+ "read_exit_delay=%d read_exit_burst=%d "
+ "nocbs_nthreads=%d nocbs_toggle=%d\n",
torture_type, tag, nrealreaders, nfakewriters,
stat_interval, verbose, test_no_idle_hz, shuffle_interval,
stutter, irqreader, fqs_duration, fqs_holdoff, fqs_stutter,
@@ -1657,7 +1716,8 @@ rcu_torture_print_module_parms(struct rcu_torture_ops *cur_ops, const char *tag)
stall_cpu_block,
n_barrier_cbs,
onoff_interval, onoff_holdoff,
- read_exit_delay, read_exit_burst);
+ read_exit_delay, read_exit_burst,
+ nocbs_nthreads, nocbs_toggle);
}
static int rcutorture_booster_cleanup(unsigned int cpu)
@@ -2498,6 +2558,13 @@ rcu_torture_cleanup(void)
torture_stop_kthread(rcu_torture_stall, stall_task);
torture_stop_kthread(rcu_torture_writer, writer_task);
+ if (nocb_tasks) {
+ for (i = 0; i < nrealnocbers; i++)
+ torture_stop_kthread(rcu_nocb_toggle, nocb_tasks[i]);
+ kfree(nocb_tasks);
+ nocb_tasks = NULL;
+ }
+
if (reader_tasks) {
for (i = 0; i < nrealreaders; i++)
torture_stop_kthread(rcu_torture_reader,
@@ -2760,6 +2827,22 @@ rcu_torture_init(void)
if (firsterr)
goto unwind;
}
+ nrealnocbers = nocbs_nthreads;
+ if (WARN_ON(nrealnocbers < 0))
+ nrealnocbers = 1;
+ if (WARN_ON(nocbs_toggle < 0))
+ nocbs_toggle = HZ;
+ nocb_tasks = kcalloc(nrealnocbers, sizeof(nocb_tasks[0]), GFP_KERNEL);
+ if (nocb_tasks == NULL) {
+ VERBOSE_TOROUT_ERRSTRING("out of memory");
+ firsterr = -ENOMEM;
+ goto unwind;
+ }
+ for (i = 0; i < nrealnocbers; i++) {
+ firsterr = torture_create_kthread(rcu_nocb_toggle, NULL, nocb_tasks[i]);
+ if (firsterr)
+ goto unwind;
+ }
if (stat_interval > 0) {
firsterr = torture_create_kthread(rcu_torture_stats, NULL,
stats_task);
^ permalink raw reply related [flat|nested] 32+ messages in thread
end of thread, other threads:[~2020-09-24 0:38 UTC | newest]
Thread overview: 32+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2020-09-21 12:43 [RFC PATCH 00/12] rcu/nocb: De-offload and re-offload support v2 Frederic Weisbecker
2020-09-21 12:43 ` [RFC PATCH 01/12] rcu: Implement rcu_segcblist_is_offloaded() config dependent Frederic Weisbecker
2020-09-22 0:27 ` Paul E. McKenney
2020-09-22 21:43 ` Frederic Weisbecker
2020-09-22 23:11 ` Paul E. McKenney
2020-09-23 15:25 ` Frederic Weisbecker
2020-09-21 12:43 ` [RFC PATCH 02/12] rcu: Turn enabled/offload states into a common flag Frederic Weisbecker
2020-09-21 12:43 ` [RFC PATCH 03/12] rcu: Provide basic callback offloading state machine bits Frederic Weisbecker
2020-09-21 23:50 ` Paul E. McKenney
2020-09-21 12:43 ` [RFC PATCH 04/12] rcu: De-offloading CB kthread Frederic Weisbecker
2020-09-22 0:07 ` Paul E. McKenney
2020-09-21 12:43 ` [RFC PATCH 05/12] rcu: De-offloading GP kthread Frederic Weisbecker
2020-09-22 0:10 ` Paul E. McKenney
2020-09-23 15:31 ` Frederic Weisbecker
2020-09-23 15:48 ` Paul E. McKenney
2020-09-21 12:43 ` [RFC PATCH 06/12] rcu: Re-offload support Frederic Weisbecker
2020-09-21 12:43 ` [RFC PATCH 07/12] rcu: Shutdown nocb timer on de-offloading Frederic Weisbecker
2020-09-22 0:17 ` Paul E. McKenney
2020-09-23 15:29 ` Frederic Weisbecker
2020-09-23 15:47 ` Paul E. McKenney
2020-09-21 12:43 ` [RFC PATCH 08/12] rcu: Flush bypass before setting SEGCBLIST_SOFTIRQ_ONLY Frederic Weisbecker
2020-09-21 12:43 ` [RFC PATCH 09/12] rcu: Set SEGCBLIST_SOFTIRQ_ONLY at the very last stage of de-offloading Frederic Weisbecker
2020-09-21 12:43 ` [RFC PATCH 10/12] rcu: Process batch locally as long as offloading isn't complete Frederic Weisbecker
2020-09-22 0:22 ` Paul E. McKenney
2020-09-21 12:43 ` [RFC PATCH 11/12] rcu: Locally accelerate callbacks " Frederic Weisbecker
2020-09-22 0:24 ` Paul E. McKenney
2020-09-21 12:43 ` [RFC PATCH 12/12] rcu: Nocb (de)activate through sysfs Frederic Weisbecker
2020-09-22 0:26 ` Paul E. McKenney
2020-09-23 15:27 ` Frederic Weisbecker
2020-09-23 15:47 ` Paul E. McKenney
2020-09-23 19:00 ` Frederic Weisbecker
2020-09-24 0:38 ` Paul E. McKenney
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).