From: Frederic Weisbecker <frederic@kernel.org>
To: LKML <linux-kernel@vger.kernel.org>
Cc: Frederic Weisbecker <frederic@kernel.org>,
Peter Zijlstra <peterz@infradead.org>,
Phil Auld <pauld@redhat.com>, Alex Belits <abelits@marvell.com>,
Nicolas Saenz Julienne <nsaenz@kernel.org>,
Xiongfeng Wang <wangxiongfeng2@huawei.com>,
Neeraj Upadhyay <quic_neeraju@quicinc.com>,
Thomas Gleixner <tglx@linutronix.de>,
Yu Liao <liaoyu15@huawei.com>, Boqun Feng <boqun.feng@gmail.com>,
"Paul E . McKenney" <paulmck@kernel.org>,
Marcelo Tosatti <mtosatti@redhat.com>,
Paul Gortmaker <paul.gortmaker@windriver.com>,
Uladzislau Rezki <uladzislau.rezki@sony.com>,
Joel Fernandes <joel@joelfernandes.org>
Subject: [PATCH 15/20] rcu/context-tracking: Move deferred nocb resched to context tracking
Date: Wed, 8 Jun 2022 16:40:32 +0200 [thread overview]
Message-ID: <20220608144037.1765000-16-frederic@kernel.org> (raw)
In-Reply-To: <20220608144037.1765000-1-frederic@kernel.org>
To prepare for migrating the RCU eqs accounting code to context tracking,
split the last-resort deferred nocb resched from rcu_user_enter() and
move it into a separate call from context tracking.
Acked-by: Paul E. McKenney <paulmck@kernel.org>
Signed-off-by: Frederic Weisbecker <frederic@kernel.org>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Thomas Gleixner <tglx@linutronix.de>
Cc: Neeraj Upadhyay <quic_neeraju@quicinc.com>
Cc: Uladzislau Rezki <uladzislau.rezki@sony.com>
Cc: Joel Fernandes <joel@joelfernandes.org>
Cc: Boqun Feng <boqun.feng@gmail.com>
Cc: Nicolas Saenz Julienne <nsaenz@kernel.org>
Cc: Marcelo Tosatti <mtosatti@redhat.com>
Cc: Xiongfeng Wang <wangxiongfeng2@huawei.com>
Cc: Yu Liao<liaoyu15@huawei.com>
Cc: Phil Auld <pauld@redhat.com>
Cc: Paul Gortmaker<paul.gortmaker@windriver.com>
Cc: Alex Belits <abelits@marvell.com>
---
include/linux/rcupdate.h | 6 ++++++
kernel/context_tracking.c | 8 ++++++++
kernel/rcu/tree.c | 15 ++-------------
3 files changed, 16 insertions(+), 13 deletions(-)
diff --git a/include/linux/rcupdate.h b/include/linux/rcupdate.h
index 910328f2f51a..1d120dca6548 100644
--- a/include/linux/rcupdate.h
+++ b/include/linux/rcupdate.h
@@ -113,6 +113,12 @@ static inline void rcu_user_enter(void) { }
static inline void rcu_user_exit(void) { }
#endif /* CONFIG_NO_HZ_FULL */
+#if defined(CONFIG_NO_HZ_FULL) && (!defined(CONFIG_GENERIC_ENTRY) || !defined(CONFIG_KVM_XFER_TO_GUEST_WORK))
+void rcu_irq_work_resched(void);
+#else
+static inline void rcu_irq_work_resched(void) { }
+#endif
+
#ifdef CONFIG_RCU_NOCB_CPU
void rcu_init_nohz(void);
int rcu_nocb_cpu_offload(int cpu);
diff --git a/kernel/context_tracking.c b/kernel/context_tracking.c
index 95ce83904287..d782c868f36f 100644
--- a/kernel/context_tracking.c
+++ b/kernel/context_tracking.c
@@ -177,6 +177,8 @@ static __always_inline void context_tracking_recursion_exit(void)
*/
void noinstr __ct_user_enter(enum ctx_state state)
{
+ lockdep_assert_irqs_disabled();
+
/* Kernel threads aren't supposed to go to userspace */
WARN_ON_ONCE(!current->mm);
@@ -198,6 +200,12 @@ void noinstr __ct_user_enter(enum ctx_state state)
vtime_user_enter(current);
instrumentation_end();
}
+ /*
+ * Other than generic entry implementation, we may be past the last
+ * rescheduling opportunity in the entry code. Trigger a self IPI
+ * that will fire and reschedule once we resume in user/guest mode.
+ */
+ rcu_irq_work_resched();
rcu_user_enter();
}
/*
diff --git a/kernel/rcu/tree.c b/kernel/rcu/tree.c
index ef734577b63d..beddb4fb5d07 100644
--- a/kernel/rcu/tree.c
+++ b/kernel/rcu/tree.c
@@ -685,7 +685,7 @@ static DEFINE_PER_CPU(struct irq_work, late_wakeup_work) =
* last resort is to fire a local irq_work that will trigger a reschedule once IRQs
* get re-enabled again.
*/
-noinstr static void rcu_irq_work_resched(void)
+noinstr void rcu_irq_work_resched(void)
{
struct rcu_data *rdp = this_cpu_ptr(&rcu_data);
@@ -701,10 +701,7 @@ noinstr static void rcu_irq_work_resched(void)
}
instrumentation_end();
}
-
-#else
-static inline void rcu_irq_work_resched(void) { }
-#endif
+#endif /* #if !defined(CONFIG_GENERIC_ENTRY) || !defined(CONFIG_KVM_XFER_TO_GUEST_WORK) */
/**
* rcu_user_enter - inform RCU that we are resuming userspace.
@@ -719,14 +716,6 @@ static inline void rcu_irq_work_resched(void) { }
*/
noinstr void rcu_user_enter(void)
{
- lockdep_assert_irqs_disabled();
-
- /*
- * Other than generic entry implementation, we may be past the last
- * rescheduling opportunity in the entry code. Trigger a self IPI
- * that will fire and reschedule once we resume in user/guest mode.
- */
- rcu_irq_work_resched();
rcu_eqs_enter(true);
}
--
2.25.1
next prev parent reply other threads:[~2022-06-08 14:44 UTC|newest]
Thread overview: 25+ messages / expand[flat|nested] mbox.gz Atom feed top
2022-06-08 14:40 [PATCH 00/20] rcu/context-tracking: Merge RCU eqs-dynticks counter to context tracking v4 Frederic Weisbecker
2022-06-08 14:40 ` [PATCH 01/20] context_tracking: Remove unused context_tracking_in_user() Frederic Weisbecker
2022-06-08 14:40 ` [PATCH 02/20] context_tracking: Add a note about noinstr VS unsafe context tracking functions Frederic Weisbecker
2022-06-09 10:50 ` Peter Zijlstra
2022-06-08 14:40 ` [PATCH 03/20] context_tracking: Rename __context_tracking_enter/exit() to __ct_user_enter/exit() Frederic Weisbecker
2022-06-08 14:40 ` [PATCH 04/20] context_tracking: Rename context_tracking_user_enter/exit() to user_enter/exit_callable() Frederic Weisbecker
2022-06-08 14:40 ` [PATCH 05/20] context_tracking: Rename context_tracking_enter/exit() to ct_user_enter/exit() Frederic Weisbecker
2022-06-08 14:40 ` [PATCH 06/20] context_tracking: Rename context_tracking_cpu_set() to ct_cpu_track_user() Frederic Weisbecker
2022-06-08 14:40 ` [PATCH 07/20] context_tracking: Split user tracking Kconfig Frederic Weisbecker
2022-06-08 14:40 ` [PATCH 08/20] context_tracking: Take idle eqs entrypoints over RCU Frederic Weisbecker
2022-06-08 14:40 ` [PATCH 09/20] context_tracking: Take IRQ " Frederic Weisbecker
2022-06-08 14:40 ` [PATCH 10/20] context_tracking: Take NMI " Frederic Weisbecker
2022-06-08 14:40 ` [PATCH 11/20] rcu/context-tracking: Remove rcu_irq_enter/exit() Frederic Weisbecker
2022-06-08 14:40 ` [PATCH 12/20] rcu/context_tracking: Move dynticks counter to context tracking Frederic Weisbecker
2022-06-08 14:40 ` [PATCH 13/20] rcu/context_tracking: Move dynticks_nesting " Frederic Weisbecker
2022-06-08 14:40 ` [PATCH 14/20] rcu/context_tracking: Move dynticks_nmi_nesting " Frederic Weisbecker
2022-06-08 14:40 ` Frederic Weisbecker [this message]
2022-06-08 14:40 ` [PATCH 16/20] rcu/context-tracking: Move RCU-dynticks internal functions to context_tracking Frederic Weisbecker
2022-06-08 14:40 ` [PATCH 17/20] rcu/context-tracking: Remove unused and/or unecessary middle functions Frederic Weisbecker
2022-06-08 14:40 ` [PATCH 18/20] context_tracking: Convert state to atomic_t Frederic Weisbecker
2022-06-08 14:40 ` [PATCH 19/20] rcu/context_tracking: Merge dynticks counter and context tracking states Frederic Weisbecker
2022-06-08 14:40 ` [PATCH 20/20] MAINTAINERS: Add Paul as context tracking maintainer Frederic Weisbecker
2022-06-20 10:14 ` [PATCH 00/20] rcu/context-tracking: Merge RCU eqs-dynticks counter to context tracking v4 nicolas saenz julienne
2022-06-20 17:16 ` Paul E. McKenney
2022-06-28 13:15 [PATCH 00/20] rcu/context-tracking: Merge RCU eqs-dynticks counter to context tracking v5 Frederic Weisbecker
2022-06-28 13:16 ` [PATCH 15/20] rcu/context-tracking: Move deferred nocb resched to context tracking Frederic Weisbecker
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20220608144037.1765000-16-frederic@kernel.org \
--to=frederic@kernel.org \
--cc=abelits@marvell.com \
--cc=boqun.feng@gmail.com \
--cc=joel@joelfernandes.org \
--cc=liaoyu15@huawei.com \
--cc=linux-kernel@vger.kernel.org \
--cc=mtosatti@redhat.com \
--cc=nsaenz@kernel.org \
--cc=paul.gortmaker@windriver.com \
--cc=pauld@redhat.com \
--cc=paulmck@kernel.org \
--cc=peterz@infradead.org \
--cc=quic_neeraju@quicinc.com \
--cc=tglx@linutronix.de \
--cc=uladzislau.rezki@sony.com \
--cc=wangxiongfeng2@huawei.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).