From: Frederic Weisbecker <frederic@kernel.org>
To: "Paul E . McKenney" <paulmck@kernel.org>
Cc: LKML <linux-kernel@vger.kernel.org>,
Frederic Weisbecker <frederic@kernel.org>,
Steven Rostedt <rostedt@goodmis.org>,
Mathieu Desnoyers <mathieu.desnoyers@efficios.com>,
Lai Jiangshan <jiangshanlai@gmail.com>,
Joel Fernandes <joel@joelfernandes.org>,
Josh Triplett <josh@joshtriplett.org>
Subject: [RFC PATCH 03/12] rcu: Provide basic callback offloading state machine bits
Date: Mon, 21 Sep 2020 14:43:42 +0200 [thread overview]
Message-ID: <20200921124351.24035-4-frederic@kernel.org> (raw)
In-Reply-To: <20200921124351.24035-1-frederic@kernel.org>
We'll need to be able to runtime offload and de-offload the processing
of callback for a given CPU. In order to support a smooth transition
from unlocked local processing (softirq/rcuc) to locked offloaded
processing (rcuop/rcuog) and the reverse, provide the necessary bits and
documentation for the state machine that will carry up all the steps to
enforce correctness while serving callbacks processing all along.
Inspired-by: Paul E. McKenney <paulmck@kernel.org>
Signed-off-by: Frederic Weisbecker <frederic@kernel.org>
Cc: Paul E. McKenney <paulmck@kernel.org>
Cc: Josh Triplett <josh@joshtriplett.org>
Cc: Steven Rostedt <rostedt@goodmis.org>
Cc: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Cc: Lai Jiangshan <jiangshanlai@gmail.com>
Cc: Joel Fernandes <joel@joelfernandes.org>
---
include/linux/rcu_segcblist.h | 115 +++++++++++++++++++++++++++++++++-
kernel/rcu/rcu_segcblist.c | 1 +
kernel/rcu/rcu_segcblist.h | 12 +++-
kernel/rcu/tree.c | 3 +
4 files changed, 128 insertions(+), 3 deletions(-)
diff --git a/include/linux/rcu_segcblist.h b/include/linux/rcu_segcblist.h
index dca2f39ee67f..67f09a912f96 100644
--- a/include/linux/rcu_segcblist.h
+++ b/include/linux/rcu_segcblist.h
@@ -63,8 +63,121 @@ struct rcu_cblist {
#define RCU_NEXT_TAIL 3
#define RCU_CBLIST_NSEGS 4
+
+/*
+ * ==NOCB Offloading state machine==
+ *
+ *
+ * ----------------------------------------------------------------------------
+ * | SEGCBLIST_SOFTIRQ_ONLY |
+ * | |
+ * | Callbacks processed by rcu_core() from softirqs or local |
+ * | rcuc kthread, without holding nocb_lock. |
+ * ----------------------------------------------------------------------------
+ * |
+ * v
+ * ----------------------------------------------------------------------------
+ * | SEGCBLIST_OFFLOADED |
+ * | |
+ * | Callbacks processed by rcu_core() from softirqs or local |
+ * | rcuc kthread, while holding nocb_lock. Waking up CB and GP kthreads, |
+ * | allowing nocb_timer to be armed. |
+ * ----------------------------------------------------------------------------
+ * |
+ * v
+ * -----------------------------------
+ * | |
+ * v v
+ * --------------------------------------- ----------------------------------|
+ * | SEGCBLIST_OFFLOADED | | | SEGCBLIST_OFFLOADED | |
+ * | SEGCBLIST_KTHREAD_CB | | SEGCBLIST_KTHREAD_GP |
+ * | | | |
+ * | | | |
+ * | CB kthread woke up and | | GP kthread woke up and |
+ * | acknowledged SEGCBLIST_OFFLOADED. | | acknowledged SEGCBLIST_OFFLOADED|
+ * | Processes callbacks concurrently | | |
+ * | with rcu_core(), holding | | |
+ * | nocb_lock. | | |
+ * --------------------------------------- -----------------------------------
+ * | |
+ * -----------------------------------
+ * |
+ * v
+ * |--------------------------------------------------------------------------|
+ * | SEGCBLIST_OFFLOADED | |
+ * | SEGCBLIST_KTHREAD_CB | |
+ * | SEGCBLIST_KTHREAD_GP |
+ * | |
+ * | Kthreads handle callbacks holding nocb_lock, local rcu_core() stops |
+ * | handling callbacks. |
+ * ----------------------------------------------------------------------------
+ */
+
+
+
+/*
+ * ==NOCB De-Offloading state machine==
+ *
+ *
+ * |--------------------------------------------------------------------------|
+ * | SEGCBLIST_OFFLOADED | |
+ * | SEGCBLIST_KTHREAD_CB | |
+ * | SEGCBLIST_KTHREAD_GP |
+ * | |
+ * | CB/GP kthreads handle callbacks holding nocb_lock, local rcu_core() |
+ * | ignores callbacks. |
+ * ----------------------------------------------------------------------------
+ * |
+ * v
+ * |--------------------------------------------------------------------------|
+ * | SEGCBLIST_KTHREAD_CB | |
+ * | SEGCBLIST_KTHREAD_GP |
+ * | |
+ * | CB/GP kthreads and local rcu_core() handle callbacks concurrently |
+ * | holding nocb_lock. Wake up CB and GP kthreads if necessary. |
+ * ----------------------------------------------------------------------------
+ * |
+ * v
+ * -----------------------------------
+ * | |
+ * v v
+ * ---------------------------------------------------------------------------|
+ * | |
+ * | SEGCBLIST_KTHREAD_CB | SEGCBLIST_KTHREAD_GP |
+ * | | |
+ * | GP kthread woke up and | CB kthread woke up and |
+ * | acknowledged the fact that | acknowledged the fact that |
+ * | SEGCBLIST_OFFLOADED got cleared. | SEGCBLIST_OFFLOADED got cleared. |
+ * | | The CB kthread goes to sleep |
+ * | The callbacks from the target CPU | until it ever gets re-offloaded. |
+ * | will be ignored from the GP kthread | |
+ * | loop. | |
+ * ----------------------------------------------------------------------------
+ * | |
+ * -----------------------------------
+ * |
+ * v
+ * ----------------------------------------------------------------------------
+ * | 0 |
+ * | |
+ * | Callbacks processed by rcu_core() from softirqs or local |
+ * | rcuc kthread, while holding nocb_lock. Forbid nocb_timer to be armed. |
+ * | Flush pending nocb_timer. Flush nocb bypass callbacks. |
+ * ----------------------------------------------------------------------------
+ * |
+ * v
+ * ----------------------------------------------------------------------------
+ * | SEGCBLIST_SOFTIRQ_ONLY |
+ * | |
+ * | Callbacks processed by rcu_core() from softirqs or local |
+ * | rcuc kthread, without holding nocb_lock. |
+ * ----------------------------------------------------------------------------
+ */
#define SEGCBLIST_ENABLED BIT(0)
-#define SEGCBLIST_OFFLOADED BIT(1)
+#define SEGCBLIST_SOFTIRQ_ONLY BIT(1)
+#define SEGCBLIST_KTHREAD_CB BIT(2)
+#define SEGCBLIST_KTHREAD_GP BIT(3)
+#define SEGCBLIST_OFFLOADED BIT(4)
struct rcu_segcblist {
struct rcu_head *head;
diff --git a/kernel/rcu/rcu_segcblist.c b/kernel/rcu/rcu_segcblist.c
index d131ef8940a0..31cc27ee98d8 100644
--- a/kernel/rcu/rcu_segcblist.c
+++ b/kernel/rcu/rcu_segcblist.c
@@ -172,6 +172,7 @@ void rcu_segcblist_disable(struct rcu_segcblist *rsclp)
*/
void rcu_segcblist_offload(struct rcu_segcblist *rsclp)
{
+ rcu_segcblist_clear_flags(rsclp, SEGCBLIST_SOFTIRQ_ONLY);
rcu_segcblist_set_flags(rsclp, SEGCBLIST_OFFLOADED);
}
diff --git a/kernel/rcu/rcu_segcblist.h b/kernel/rcu/rcu_segcblist.h
index fc98761e3ee9..575896a2518b 100644
--- a/kernel/rcu/rcu_segcblist.h
+++ b/kernel/rcu/rcu_segcblist.h
@@ -80,8 +80,16 @@ static inline bool rcu_segcblist_is_enabled(struct rcu_segcblist *rsclp)
/* Is the specified rcu_segcblist offloaded? */
static inline bool rcu_segcblist_is_offloaded(struct rcu_segcblist *rsclp)
{
- return IS_ENABLED(CONFIG_RCU_NOCB_CPU) &&
- rcu_segcblist_test_flags(rsclp, SEGCBLIST_OFFLOADED);
+ if (IS_ENABLED(CONFIG_RCU_NOCB_CPU)) {
+ /*
+ * Complete de-offloading happens only when SEGCBLIST_SOFTIRQ_ONLY
+ * is set.
+ */
+ if (!rcu_segcblist_test_flags(rsclp, SEGCBLIST_SOFTIRQ_ONLY))
+ return true;
+ }
+
+ return false;
}
/*
diff --git a/kernel/rcu/tree.c b/kernel/rcu/tree.c
index c0286ce8fc03..b4292489db0c 100644
--- a/kernel/rcu/tree.c
+++ b/kernel/rcu/tree.c
@@ -96,6 +96,9 @@ static DEFINE_PER_CPU_SHARED_ALIGNED(struct rcu_data, rcu_data) = {
.dynticks_nesting = 1,
.dynticks_nmi_nesting = DYNTICK_IRQ_NONIDLE,
.dynticks = ATOMIC_INIT(RCU_DYNTICK_CTRL_CTR),
+#ifdef CONFIG_RCU_NOCB_CPU
+ .cblist.flags = SEGCBLIST_SOFTIRQ_ONLY,
+#endif
};
static struct rcu_state rcu_state = {
.level = { &rcu_state.node[0] },
--
2.28.0
next prev parent reply other threads:[~2020-09-21 12:44 UTC|newest]
Thread overview: 32+ messages / expand[flat|nested] mbox.gz Atom feed top
2020-09-21 12:43 [RFC PATCH 00/12] rcu/nocb: De-offload and re-offload support v2 Frederic Weisbecker
2020-09-21 12:43 ` [RFC PATCH 01/12] rcu: Implement rcu_segcblist_is_offloaded() config dependent Frederic Weisbecker
2020-09-22 0:27 ` Paul E. McKenney
2020-09-22 21:43 ` Frederic Weisbecker
2020-09-22 23:11 ` Paul E. McKenney
2020-09-23 15:25 ` Frederic Weisbecker
2020-09-21 12:43 ` [RFC PATCH 02/12] rcu: Turn enabled/offload states into a common flag Frederic Weisbecker
2020-09-21 12:43 ` Frederic Weisbecker [this message]
2020-09-21 23:50 ` [RFC PATCH 03/12] rcu: Provide basic callback offloading state machine bits Paul E. McKenney
2020-09-21 12:43 ` [RFC PATCH 04/12] rcu: De-offloading CB kthread Frederic Weisbecker
2020-09-22 0:07 ` Paul E. McKenney
2020-09-21 12:43 ` [RFC PATCH 05/12] rcu: De-offloading GP kthread Frederic Weisbecker
2020-09-22 0:10 ` Paul E. McKenney
2020-09-23 15:31 ` Frederic Weisbecker
2020-09-23 15:48 ` Paul E. McKenney
2020-09-21 12:43 ` [RFC PATCH 06/12] rcu: Re-offload support Frederic Weisbecker
2020-09-21 12:43 ` [RFC PATCH 07/12] rcu: Shutdown nocb timer on de-offloading Frederic Weisbecker
2020-09-22 0:17 ` Paul E. McKenney
2020-09-23 15:29 ` Frederic Weisbecker
2020-09-23 15:47 ` Paul E. McKenney
2020-09-21 12:43 ` [RFC PATCH 08/12] rcu: Flush bypass before setting SEGCBLIST_SOFTIRQ_ONLY Frederic Weisbecker
2020-09-21 12:43 ` [RFC PATCH 09/12] rcu: Set SEGCBLIST_SOFTIRQ_ONLY at the very last stage of de-offloading Frederic Weisbecker
2020-09-21 12:43 ` [RFC PATCH 10/12] rcu: Process batch locally as long as offloading isn't complete Frederic Weisbecker
2020-09-22 0:22 ` Paul E. McKenney
2020-09-21 12:43 ` [RFC PATCH 11/12] rcu: Locally accelerate callbacks " Frederic Weisbecker
2020-09-22 0:24 ` Paul E. McKenney
2020-09-21 12:43 ` [RFC PATCH 12/12] rcu: Nocb (de)activate through sysfs Frederic Weisbecker
2020-09-22 0:26 ` Paul E. McKenney
2020-09-23 15:27 ` Frederic Weisbecker
2020-09-23 15:47 ` Paul E. McKenney
2020-09-23 19:00 ` Frederic Weisbecker
2020-09-24 0:38 ` Paul E. McKenney
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20200921124351.24035-4-frederic@kernel.org \
--to=frederic@kernel.org \
--cc=jiangshanlai@gmail.com \
--cc=joel@joelfernandes.org \
--cc=josh@joshtriplett.org \
--cc=linux-kernel@vger.kernel.org \
--cc=mathieu.desnoyers@efficios.com \
--cc=paulmck@kernel.org \
--cc=rostedt@goodmis.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).