From: "tip-bot2 for Paul E. McKenney" <tip-bot2@linutronix.de>
To: linux-tip-commits@vger.kernel.org
Cc: "Paul E. McKenney" <paulmck@kernel.org>,
"Peter Zijlstra (Intel)" <peterz@infradead.org>,
Thomas Gleixner <tglx@linutronix.de>,
Alexandre Chartre <alexandre.chartre@oracle.com>,
x86 <x86@kernel.org>, LKML <linux-kernel@vger.kernel.org>
Subject: [tip: core/rcu] rcu: Make RCU IRQ enter/exit functions rely on in_nmi()
Date: Tue, 19 May 2020 19:52:32 -0000 [thread overview]
Message-ID: <158991795254.17951.5903947139387946517.tip-bot2@tip-bot2> (raw)
In-Reply-To: <20200505134101.617130349@linutronix.de>
The following commit has been merged into the core/rcu branch of tip:
Commit-ID: 9ea366f669ded353ae49754216c042e7d2f72ba6
Gitweb: https://git.kernel.org/tip/9ea366f669ded353ae49754216c042e7d2f72ba6
Author: Paul E. McKenney <paulmck@kernel.org>
AuthorDate: Thu, 13 Feb 2020 12:31:16 -08:00
Committer: Thomas Gleixner <tglx@linutronix.de>
CommitterDate: Tue, 19 May 2020 15:51:21 +02:00
rcu: Make RCU IRQ enter/exit functions rely on in_nmi()
The rcu_nmi_enter_common() and rcu_nmi_exit_common() functions take an
"irq" parameter that indicates whether these functions have been invoked from
an irq handler (irq==true) or an NMI handler (irq==false).
However, recent changes have applied notrace to a few critical functions
such that rcu_nmi_enter_common() and rcu_nmi_exit_common() many now rely on
in_nmi(). Note that in_nmi() works no differently than before, but rather
that tracing is now prohibited in code regions where in_nmi() would
incorrectly report NMI state.
Therefore remove the "irq" parameter and inline rcu_nmi_enter_common() and
rcu_nmi_exit_common() into rcu_nmi_enter() and rcu_nmi_exit(),
respectively.
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Signed-off-by: Thomas Gleixner <tglx@linutronix.de>
Reviewed-by: Alexandre Chartre <alexandre.chartre@oracle.com>
Link: https://lkml.kernel.org/r/20200505134101.617130349@linutronix.de
---
kernel/rcu/tree.c | 47 ++++++++++++++--------------------------------
1 file changed, 15 insertions(+), 32 deletions(-)
diff --git a/kernel/rcu/tree.c b/kernel/rcu/tree.c
index 0713ef3..9454016 100644
--- a/kernel/rcu/tree.c
+++ b/kernel/rcu/tree.c
@@ -664,16 +664,18 @@ noinstr void rcu_user_enter(void)
}
#endif /* CONFIG_NO_HZ_FULL */
-/*
+/**
+ * rcu_nmi_exit - inform RCU of exit from NMI context
+ *
* If we are returning from the outermost NMI handler that interrupted an
* RCU-idle period, update rdp->dynticks and rdp->dynticks_nmi_nesting
* to let the RCU grace-period handling know that the CPU is back to
* being RCU-idle.
*
- * If you add or remove a call to rcu_nmi_exit_common(), be sure to test
+ * If you add or remove a call to rcu_nmi_exit(), be sure to test
* with CONFIG_RCU_EQS_DEBUG=y.
*/
-static __always_inline void rcu_nmi_exit_common(bool irq)
+noinstr void rcu_nmi_exit(void)
{
struct rcu_data *rdp = this_cpu_ptr(&rcu_data);
@@ -704,7 +706,7 @@ static __always_inline void rcu_nmi_exit_common(bool irq)
trace_rcu_dyntick(TPS("Startirq"), rdp->dynticks_nmi_nesting, 0, atomic_read(&rdp->dynticks));
WRITE_ONCE(rdp->dynticks_nmi_nesting, 0); /* Avoid store tearing. */
- if (irq)
+ if (!in_nmi())
rcu_prepare_for_idle();
instrumentation_end();
@@ -712,22 +714,11 @@ static __always_inline void rcu_nmi_exit_common(bool irq)
rcu_dynticks_eqs_enter();
// ... but is no longer watching here.
- if (irq)
+ if (!in_nmi())
rcu_dynticks_task_enter();
}
/**
- * rcu_nmi_exit - inform RCU of exit from NMI context
- *
- * If you add or remove a call to rcu_nmi_exit(), be sure to test
- * with CONFIG_RCU_EQS_DEBUG=y.
- */
-void noinstr rcu_nmi_exit(void)
-{
- rcu_nmi_exit_common(false);
-}
-
-/**
* rcu_irq_exit - inform RCU that current CPU is exiting irq towards idle
*
* Exit from an interrupt handler, which might possibly result in entering
@@ -749,7 +740,7 @@ void noinstr rcu_nmi_exit(void)
void noinstr rcu_irq_exit(void)
{
lockdep_assert_irqs_disabled();
- rcu_nmi_exit_common(true);
+ rcu_nmi_exit();
}
/*
@@ -838,7 +829,7 @@ void noinstr rcu_user_exit(void)
#endif /* CONFIG_NO_HZ_FULL */
/**
- * rcu_nmi_enter_common - inform RCU of entry to NMI context
+ * rcu_nmi_enter - inform RCU of entry to NMI context
* @irq: Is this call from rcu_irq_enter?
*
* If the CPU was idle from RCU's viewpoint, update rdp->dynticks and
@@ -847,10 +838,10 @@ void noinstr rcu_user_exit(void)
* long as the nesting level does not overflow an int. (You will probably
* run out of stack space first.)
*
- * If you add or remove a call to rcu_nmi_enter_common(), be sure to test
+ * If you add or remove a call to rcu_nmi_enter(), be sure to test
* with CONFIG_RCU_EQS_DEBUG=y.
*/
-static __always_inline void rcu_nmi_enter_common(bool irq)
+noinstr void rcu_nmi_enter(void)
{
long incby = 2;
struct rcu_data *rdp = this_cpu_ptr(&rcu_data);
@@ -868,18 +859,18 @@ static __always_inline void rcu_nmi_enter_common(bool irq)
*/
if (rcu_dynticks_curr_cpu_in_eqs()) {
- if (irq)
+ if (!in_nmi())
rcu_dynticks_task_exit();
// RCU is not watching here ...
rcu_dynticks_eqs_exit();
// ... but is watching here.
- if (irq)
+ if (!in_nmi())
rcu_cleanup_after_idle();
incby = 1;
- } else if (irq) {
+ } else if (!in_nmi()) {
instrumentation_begin();
if (tick_nohz_full_cpu(rdp->cpu) &&
rdp->dynticks_nmi_nesting == DYNTICK_IRQ_NONIDLE &&
@@ -914,14 +905,6 @@ static __always_inline void rcu_nmi_enter_common(bool irq)
}
/**
- * rcu_nmi_enter - inform RCU of entry to NMI context
- */
-noinstr void rcu_nmi_enter(void)
-{
- rcu_nmi_enter_common(false);
-}
-
-/**
* rcu_irq_enter - inform RCU that current CPU is entering irq away from idle
*
* Enter an interrupt handler, which might possibly result in exiting
@@ -946,7 +929,7 @@ noinstr void rcu_nmi_enter(void)
noinstr void rcu_irq_enter(void)
{
lockdep_assert_irqs_disabled();
- rcu_nmi_enter_common(true);
+ rcu_nmi_enter();
}
/*
next prev parent reply other threads:[~2020-05-19 19:52 UTC|newest]
Thread overview: 178+ messages / expand[flat|nested] mbox.gz Atom feed top
2020-05-05 13:16 [patch V4 part 1 00/36] x86/entry: Entry/exception code rework, preparatory patches Thomas Gleixner
2020-05-05 13:16 ` [patch V4 part 1 01/36] rcu: Add comments marking transitions between RCU watching and not Thomas Gleixner
2020-05-05 13:16 ` [patch V4 part 1 02/36] x86/hw_breakpoint: Prevent data breakpoints on cpu_entry_area Thomas Gleixner
2020-05-06 8:14 ` Borislav Petkov
2020-05-06 12:11 ` Alexandre Chartre
2020-05-09 9:00 ` Lai Jiangshan
2020-05-09 9:23 ` Lai Jiangshan
2020-05-09 19:08 ` Andy Lutomirski
2020-05-19 19:58 ` [tip: x86/entry] " tip-bot2 for Andy Lutomirski
2020-05-05 13:16 ` [patch V4 part 1 03/36] sched: Clean up scheduler_ipi() Thomas Gleixner
2020-05-06 8:32 ` Thomas Gleixner
2020-05-06 8:40 ` Borislav Petkov
2020-05-06 9:12 ` Thomas Gleixner
2020-05-06 10:02 ` Borislav Petkov
2020-05-06 12:37 ` Alexandre Chartre
2020-05-06 15:03 ` Thomas Gleixner
2020-05-06 15:33 ` Peter Zijlstra
2020-05-06 18:28 ` Paul E. McKenney
2020-05-06 18:37 ` Peter Zijlstra
2020-05-06 18:46 ` Paul E. McKenney
2020-05-12 15:13 ` [tip: sched/core] " tip-bot2 for Peter Zijlstra (Intel)
2020-05-05 13:16 ` [patch V4 part 1 04/36] sched: Make scheduler_ipi inline Thomas Gleixner
2020-05-06 12:42 ` Alexandre Chartre
2020-05-12 15:13 ` [tip: sched/core] " tip-bot2 for Thomas Gleixner
2020-05-05 13:16 ` [patch V4 part 1 05/36] x86/entry: Flip _TIF_SIGPENDING and _TIF_NOTIFY_RESUME handling Thomas Gleixner
2020-05-06 11:53 ` Miroslav Benes
2020-05-06 12:06 ` Thomas Gleixner
2020-05-06 15:35 ` Peter Zijlstra
2020-05-06 13:06 ` Alexandre Chartre
2020-05-06 16:26 ` Borislav Petkov
2020-05-07 17:35 ` Andy Lutomirski
2020-05-13 20:56 ` Mathieu Desnoyers
2020-05-13 21:10 ` Steven Rostedt
2020-05-13 22:48 ` Mathieu Desnoyers
2020-05-14 0:12 ` Thomas Gleixner
2020-05-14 0:37 ` Steven Rostedt
2020-05-14 0:49 ` Thomas Gleixner
2020-05-14 1:22 ` Andy Lutomirski
2020-05-14 2:51 ` Mathieu Desnoyers
2020-05-14 9:19 ` Peter Zijlstra
2020-05-05 13:16 ` [patch V4 part 1 06/36] compiler: Simple READ/WRITE_ONCE() implementations Thomas Gleixner
2020-05-06 13:11 ` Alexandre Chartre
2020-05-06 13:33 ` Will Deacon
2020-05-06 15:36 ` Peter Zijlstra
2020-05-06 16:33 ` Borislav Petkov
2020-05-05 13:16 ` [patch V4 part 1 07/36] locking/atomics: Flip fallbacks and instrumentation Thomas Gleixner
2020-05-05 16:04 ` Mark Rutland
2020-05-07 23:41 ` Steven Rostedt
2020-05-08 8:40 ` Peter Zijlstra
2020-05-12 14:36 ` [tip: locking/kcsan] " tip-bot2 for Peter Zijlstra
2020-05-05 13:16 ` [patch V4 part 1 08/36] x86/doublefault: Remove memmove() call Thomas Gleixner
2020-05-06 13:47 ` Alexandre Chartre
2020-05-19 19:58 ` [tip: x86/entry] " tip-bot2 for Peter Zijlstra
2020-05-05 13:16 ` [patch V4 part 1 09/36] x86/entry/64: Avoid pointless code when CONTEXT_TRACKING=n Thomas Gleixner
2020-05-19 19:58 ` [tip: x86/entry] " tip-bot2 for Thomas Gleixner
2020-05-05 13:16 ` [patch V4 part 1 10/36] x86/entry: Remove the unused LOCKDEP_SYSEXIT cruft Thomas Gleixner
2020-05-06 13:52 ` Alexandre Chartre
2020-05-19 19:58 ` [tip: x86/entry] " tip-bot2 for Thomas Gleixner
2020-05-05 13:16 ` [patch V4 part 1 11/36] x86/kvm: Handle async page faults directly through do_page_fault() Thomas Gleixner
2020-05-06 7:00 ` Paolo Bonzini
2020-05-06 14:05 ` Alexandre Chartre
2020-05-19 19:58 ` [tip: x86/entry] " tip-bot2 for Andy Lutomirski
2020-05-05 13:16 ` [patch V4 part 1 12/36] x86/kvm: Sanitize kvm_async_pf_task_wait() Thomas Gleixner
2020-05-05 17:54 ` Paul E. McKenney
2020-05-05 21:50 ` Thomas Gleixner
2020-05-06 7:00 ` Paolo Bonzini
2020-05-06 12:53 ` Steven Rostedt
2020-05-06 15:13 ` Alexandre Chartre
2020-05-19 19:58 ` [tip: x86/entry] " tip-bot2 for Thomas Gleixner
2020-05-05 13:16 ` [patch V4 part 1 13/36] x86/kvm: Restrict ASYNC_PF to user space Thomas Gleixner
2020-05-06 7:00 ` Paolo Bonzini
2020-05-06 15:29 ` Alexandre Chartre
2020-05-19 19:58 ` [tip: x86/entry] " tip-bot2 for Thomas Gleixner
2020-05-05 13:16 ` [patch V4 part 1 14/36] x86/entry: Get rid of ist_begin/end_non_atomic() Thomas Gleixner
2020-05-06 15:34 ` Alexandre Chartre
2020-05-07 17:46 ` Andy Lutomirski
2020-05-13 22:57 ` Mathieu Desnoyers
2020-05-14 0:13 ` Steven Rostedt
2020-05-15 9:34 ` Thomas Gleixner
2020-05-15 13:11 ` Mathieu Desnoyers
2020-05-19 19:52 ` [tip: core/rcu] " tip-bot2 for Thomas Gleixner
2020-05-05 13:16 ` [patch V4 part 1 15/36] kprobes: Lock kprobe_mutex while showing kprobe_blacklist Thomas Gleixner
2020-05-06 15:38 ` Alexandre Chartre
2020-05-12 15:18 ` [tip: core/kprobes] " tip-bot2 for Masami Hiramatsu
2020-05-05 13:16 ` [patch V4 part 1 16/36] kprobes: Support __kprobes blacklist in modules Thomas Gleixner
2020-05-06 15:47 ` Alexandre Chartre
2020-05-12 15:18 ` [tip: core/kprobes] " tip-bot2 for Masami Hiramatsu
2020-05-05 13:16 ` [patch V4 part 1 17/36] kprobes: Support NOKPROBE_SYMBOL() " Thomas Gleixner
2020-05-06 15:54 ` Alexandre Chartre
2020-05-12 15:18 ` [tip: core/kprobes] " tip-bot2 for Masami Hiramatsu
2020-05-05 13:16 ` [patch V4 part 1 18/36] samples/kprobes: Add __kprobes and NOKPROBE_SYMBOL() for handlers Thomas Gleixner
2020-05-06 15:57 ` Alexandre Chartre
2020-05-12 15:18 ` [tip: core/kprobes] " tip-bot2 for Masami Hiramatsu
2020-05-05 13:16 ` [patch V4 part 1 19/36] x86/entry: Exclude low level entry code from sanitizing Thomas Gleixner
2020-05-05 20:39 ` Brian Gerst
2020-05-06 15:42 ` Peter Zijlstra
2020-05-06 16:03 ` Alexandre Chartre
2020-05-13 22:58 ` Mathieu Desnoyers
2020-05-19 19:58 ` [tip: x86/entry] " tip-bot2 for Peter Zijlstra
2020-05-05 13:16 ` [patch V4 part 1 20/36] vmlinux.lds.h: Create section for protection against instrumentation Thomas Gleixner
2020-05-06 16:08 ` Sean Christopherson
2020-05-06 16:28 ` Peter Zijlstra
2020-05-06 16:57 ` Thomas Gleixner
2020-05-19 19:52 ` [tip: core/rcu] " tip-bot2 for Thomas Gleixner
2020-05-05 13:16 ` [patch V4 part 1 21/36] kprobes: Prevent probes in .noinstr.text section Thomas Gleixner
2020-05-08 6:30 ` Masami Hiramatsu
2020-05-19 19:52 ` [tip: core/kprobes] " tip-bot2 for Thomas Gleixner
2020-05-05 13:16 ` [patch V4 part 1 22/36] tracing: Provide lockdep less trace_hardirqs_on/off() variants Thomas Gleixner
2020-05-07 17:55 ` Andy Lutomirski
2020-05-07 18:52 ` Thomas Gleixner
2020-05-19 19:58 ` [tip: x86/entry] " tip-bot2 for Thomas Gleixner
2020-05-05 13:16 ` [patch V4 part 1 23/36] bug: Annotate WARN/BUG/stackfail as noinstr safe Thomas Gleixner
2020-05-13 23:12 ` Mathieu Desnoyers
2020-05-19 19:58 ` [tip: x86/entry] " tip-bot2 for Thomas Gleixner
2020-05-05 13:16 ` [patch V4 part 1 24/36] lockdep: Prepare for noinstr sections Thomas Gleixner
2020-05-08 0:23 ` Steven Rostedt
2020-05-08 8:44 ` Peter Zijlstra
2020-05-19 19:58 ` [tip: x86/entry] " tip-bot2 for Peter Zijlstra
2020-05-05 13:16 ` [patch V4 part 1 25/36] rcu/tree: Mark the idle relevant functions noinstr Thomas Gleixner
2020-05-05 18:07 ` Paul E. McKenney
2020-05-19 19:48 ` Joel Fernandes
2020-05-19 19:52 ` [tip: core/rcu] " tip-bot2 for Thomas Gleixner
2020-09-28 22:22 ` Kim Phillips
2020-09-28 22:55 ` Paul E. McKenney
2020-09-29 7:25 ` Peter Zijlstra
2020-09-29 11:25 ` Peter Zijlstra
2020-09-29 14:34 ` Steven Rostedt
2020-09-29 14:52 ` Peter Zijlstra
2020-05-05 13:16 ` [patch V4 part 1 26/36] printk: Prepare for nested printk_nmi_enter() Thomas Gleixner
2020-05-19 19:52 ` [tip: core/rcu] " tip-bot2 for Petr Mladek
2020-05-05 13:16 ` [patch V4 part 1 27/36] arm64: Prepare arch_nmi_enter() for recursion Thomas Gleixner
2020-05-13 23:28 ` Mathieu Desnoyers
2020-05-15 14:04 ` Frederic Weisbecker
2020-05-15 15:45 ` Will Deacon
2020-05-15 16:01 ` Mathieu Desnoyers
2020-05-15 21:29 ` Thomas Gleixner
2020-05-15 21:31 ` Frederic Weisbecker
2020-05-19 19:52 ` [tip: core/rcu] " tip-bot2 for Frederic Weisbecker
2020-05-05 13:16 ` [patch V4 part 1 28/36] hardirq/nmi: Allow nested nmi_enter() Thomas Gleixner
2020-05-19 19:52 ` [tip: core/rcu] " tip-bot2 for Peter Zijlstra
2020-05-05 13:16 ` [patch V4 part 1 29/36] x86/mce: Send #MC singal from task work Thomas Gleixner
2020-05-07 18:02 ` Andy Lutomirski
2020-05-08 8:48 ` Peter Zijlstra
2020-05-08 21:30 ` Andy Lutomirski
2020-05-14 14:16 ` Borislav Petkov
2020-05-13 23:42 ` Mathieu Desnoyers
2020-05-14 17:38 ` Thomas Gleixner
2020-05-14 17:42 ` Mathieu Desnoyers
2020-05-14 14:17 ` Borislav Petkov
2020-05-14 16:03 ` Mathieu Desnoyers
2020-05-14 16:19 ` Andy Lutomirski
2020-05-14 16:39 ` Borislav Petkov
2020-05-14 17:05 ` Mathieu Desnoyers
2020-05-19 19:52 ` [tip: core/rcu] " tip-bot2 for Peter Zijlstra
2020-05-05 13:16 ` [patch V4 part 1 30/36] lockdep: Always inline lockdep_{off,on}() Thomas Gleixner
2020-05-13 23:46 ` Mathieu Desnoyers
2020-05-19 19:52 ` [tip: core/rcu] " tip-bot2 for Peter Zijlstra
2020-05-05 13:16 ` [patch V4 part 1 31/36] printk: Disallow instrumenting print_nmi_enter() Thomas Gleixner
2020-05-19 19:52 ` [tip: core/rcu] " tip-bot2 for Peter Zijlstra
2020-05-05 13:16 ` [patch V4 part 1 32/36] sh/ftrace: Move arch_ftrace_nmi_{enter,exit} into nmi exception Thomas Gleixner
2020-05-08 0:34 ` Steven Rostedt
2020-05-19 19:52 ` [tip: core/rcu] " tip-bot2 for Peter Zijlstra
2020-05-05 13:16 ` [patch V4 part 1 33/36] x86,tracing: Robustify ftrace_nmi_enter() Thomas Gleixner
2020-05-08 6:19 ` Masami Hiramatsu
2020-05-05 13:16 ` [patch V4 part 1 34/36] sched,rcu,tracing: Avoid tracing before in_nmi() is correct Thomas Gleixner
2020-05-19 19:52 ` [tip: core/rcu] " tip-bot2 for Peter Zijlstra
2020-05-05 13:16 ` [patch V4 part 1 35/36] x86: Replace ist_enter() with nmi_enter() Thomas Gleixner
2020-05-07 18:04 ` Andy Lutomirski
2020-05-07 18:17 ` Mathieu Desnoyers
2020-05-08 8:50 ` Peter Zijlstra
2020-05-08 17:12 ` Josh Poimboeuf
2020-05-14 0:12 ` Mathieu Desnoyers
2020-05-19 19:52 ` [tip: core/rcu] " tip-bot2 for Peter Zijlstra
2020-05-05 13:16 ` [patch V4 part 1 36/36] rcu: Make RCU IRQ enter/exit functions rely on in_nmi() Thomas Gleixner
2020-05-05 18:13 ` Paul E. McKenney
2020-05-06 17:09 ` Alexandre Chartre
2020-05-19 19:52 ` tip-bot2 for Paul E. McKenney [this message]
2020-05-07 18:05 ` [patch V4 part 1 00/36] x86/entry: Entry/exception code rework, preparatory patches Andy Lutomirski
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=158991795254.17951.5903947139387946517.tip-bot2@tip-bot2 \
--to=tip-bot2@linutronix.de \
--cc=alexandre.chartre@oracle.com \
--cc=linux-kernel@vger.kernel.org \
--cc=linux-tip-commits@vger.kernel.org \
--cc=paulmck@kernel.org \
--cc=peterz@infradead.org \
--cc=tglx@linutronix.de \
--cc=x86@kernel.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).