From: "tip-bot2 for Thomas Gleixner" <tip-bot2@linutronix.de>
To: linux-tip-commits@vger.kernel.org
Cc: Thomas Gleixner <tglx@linutronix.de>,
Ingo Molnar <mingo@kernel.org>,
"Paul E. McKenney" <paulmck@kernel.org>,
Andy Lutomirski <luto@kernel.org>, x86 <x86@kernel.org>,
LKML <linux-kernel@vger.kernel.org>
Subject: [tip: x86/entry] x86/entry: Provide idtentry_entry/exit_cond_rcu()
Date: Wed, 27 May 2020 08:12:00 -0000 [thread overview]
Message-ID: <159056712087.17951.10894256787068855437.tip-bot2@tip-bot2> (raw)
In-Reply-To: <20200515235125.628629605@linutronix.de>
The following commit has been merged into the x86/entry branch of tip:
Commit-ID: 8fbf48a5cdb83a1ae4285920713facef72639641
Gitweb: https://git.kernel.org/tip/8fbf48a5cdb83a1ae4285920713facef72639641
Author: Thomas Gleixner <tglx@linutronix.de>
AuthorDate: Thu, 21 May 2020 22:05:17 +02:00
Committer: Ingo Molnar <mingo@kernel.org>
CommitterDate: Tue, 26 May 2020 19:06:27 +02:00
x86/entry: Provide idtentry_entry/exit_cond_rcu()
After a lengthy discussion [1] it turned out that RCU does not need a full
rcu_irq_enter/exit() when RCU is already watching. All it needs if
NOHZ_FULL is active is to check whether the tick needs to be restarted.
This allows to avoid a separate variant for the pagefault handler which
cannot invoke rcu_irq_enter() on a kernel pagefault which might sleep.
The cond_rcu argument is only temporary and will be removed once the
existing users of idtentry_enter/exit() have been cleaned up. After that
the code can be significantly simplified.
[ mingo: Simplified the control flow ]
Signed-off-by: Thomas Gleixner <tglx@linutronix.de>
Signed-off-by: Ingo Molnar <mingo@kernel.org>
Acked-by: "Paul E. McKenney" <paulmck@kernel.org>
Acked-by: Andy Lutomirski <luto@kernel.org>
Link: [1] https://lkml.kernel.org/r/20200515235125.628629605@linutronix.de
Link: https://lore.kernel.org/r/20200521202117.181397835@linutronix.de
---
arch/x86/entry/common.c | 79 +++++++++++++++++++++++++-------
arch/x86/include/asm/idtentry.h | 14 +++++-
2 files changed, 76 insertions(+), 17 deletions(-)
diff --git a/arch/x86/entry/common.c b/arch/x86/entry/common.c
index 9ebe334..a7f5846 100644
--- a/arch/x86/entry/common.c
+++ b/arch/x86/entry/common.c
@@ -512,8 +512,10 @@ SYSCALL_DEFINE0(ni_syscall)
}
/**
- * idtentry_enter - Handle state tracking on idtentry
+ * idtentry_enter_cond_rcu - Handle state tracking on idtentry with conditional
+ * RCU handling
* @regs: Pointer to pt_regs of interrupted context
+ * @cond_rcu: Invoke rcu_irq_enter() only if RCU is not watching
*
* Invokes:
* - lockdep irqflag state tracking as low level ASM entry disabled
@@ -521,40 +523,84 @@ SYSCALL_DEFINE0(ni_syscall)
*
* - Context tracking if the exception hit user mode.
*
- * - RCU notification if the exception hit kernel mode.
- *
* - The hardirq tracer to keep the state consistent as low level ASM
* entry disabled interrupts.
+ *
+ * For kernel mode entries RCU handling is done conditional. If RCU is
+ * watching then the only RCU requirement is to check whether the tick has
+ * to be restarted. If RCU is not watching then rcu_irq_enter() has to be
+ * invoked on entry and rcu_irq_exit() on exit.
+ *
+ * Avoiding the rcu_irq_enter/exit() calls is an optimization but also
+ * solves the problem of kernel mode pagefaults which can schedule, which
+ * is not possible after invoking rcu_irq_enter() without undoing it.
+ *
+ * For user mode entries enter_from_user_mode() must be invoked to
+ * establish the proper context for NOHZ_FULL. Otherwise scheduling on exit
+ * would not be possible.
+ *
+ * Returns: True if RCU has been adjusted on a kernel entry
+ * False otherwise
+ *
+ * The return value must be fed into the rcu_exit argument of
+ * idtentry_exit_cond_rcu().
*/
-void noinstr idtentry_enter(struct pt_regs *regs)
+bool noinstr idtentry_enter_cond_rcu(struct pt_regs *regs, bool cond_rcu)
{
if (user_mode(regs)) {
enter_from_user_mode();
- } else {
+ return false;
+ }
+
+ if (!cond_rcu || !__rcu_is_watching()) {
+ /*
+ * If RCU is not watching then the same careful
+ * sequence vs. lockdep and tracing is required
+ * as in enter_from_user_mode().
+ *
+ * This only happens for IRQs that hit the idle
+ * loop, i.e. if idle is not using MWAIT.
+ */
lockdep_hardirqs_off(CALLER_ADDR0);
rcu_irq_enter();
instrumentation_begin();
trace_hardirqs_off_prepare();
instrumentation_end();
+
+ return true;
}
+
+ /*
+ * If RCU is watching then RCU only wants to check
+ * whether it needs to restart the tick in NOHZ
+ * mode.
+ */
+ instrumentation_begin();
+ rcu_irq_enter_check_tick();
+ /* Use the combo lockdep/tracing function */
+ trace_hardirqs_off();
+ instrumentation_end();
+
+ return false;
}
/**
- * idtentry_exit - Common code to handle return from exceptions
+ * idtentry_exit_cond_rcu - Handle return from exception with conditional RCU
+ * handling
* @regs: Pointer to pt_regs (exception entry regs)
+ * @rcu_exit: Invoke rcu_irq_exit() if true
*
* Depending on the return target (kernel/user) this runs the necessary
- * preemption and work checks if possible and required and returns to
+ * preemption and work checks if possible and reguired and returns to
* the caller with interrupts disabled and no further work pending.
*
* This is the last action before returning to the low level ASM code which
* just needs to return to the appropriate context.
*
- * Invoked by all exception/interrupt IDTENTRY handlers which are not
- * returning through the paranoid exit path (all except NMI, #DF and the IST
- * variants of #MC and #DB) and are therefore on the thread stack.
+ * Counterpart to idtentry_enter_cond_rcu(). The return value of the entry
+ * function must be fed into the @rcu_exit argument.
*/
-void noinstr idtentry_exit(struct pt_regs *regs)
+void noinstr idtentry_exit_cond_rcu(struct pt_regs *regs, bool rcu_exit)
{
lockdep_assert_irqs_disabled();
@@ -580,7 +626,8 @@ void noinstr idtentry_exit(struct pt_regs *regs)
if (IS_ENABLED(CONFIG_DEBUG_ENTRY))
WARN_ON_ONCE(!on_thread_stack());
instrumentation_begin();
- rcu_irq_exit_preempt();
+ if (rcu_exit)
+ rcu_irq_exit_preempt();
if (need_resched())
preempt_schedule_irq();
/* Covers both tracing and lockdep */
@@ -602,10 +649,12 @@ void noinstr idtentry_exit(struct pt_regs *regs)
trace_hardirqs_on_prepare();
lockdep_hardirqs_on_prepare(CALLER_ADDR0);
instrumentation_end();
- rcu_irq_exit();
+ if (rcu_exit)
+ rcu_irq_exit();
lockdep_hardirqs_on(CALLER_ADDR0);
} else {
- /* IRQ flags state is correct already. Just tell RCU */
- rcu_irq_exit();
+ /* IRQ flags state is correct already. Just tell RCU. */
+ if (rcu_exit)
+ rcu_irq_exit();
}
}
diff --git a/arch/x86/include/asm/idtentry.h b/arch/x86/include/asm/idtentry.h
index ce97478..a116b80 100644
--- a/arch/x86/include/asm/idtentry.h
+++ b/arch/x86/include/asm/idtentry.h
@@ -7,8 +7,18 @@
#ifndef __ASSEMBLY__
-void idtentry_enter(struct pt_regs *regs);
-void idtentry_exit(struct pt_regs *regs);
+bool idtentry_enter_cond_rcu(struct pt_regs *regs, bool cond_rcu);
+void idtentry_exit_cond_rcu(struct pt_regs *regs, bool rcu_exit);
+
+static __always_inline void idtentry_enter(struct pt_regs *regs)
+{
+ idtentry_enter_cond_rcu(regs, false);
+}
+
+static __always_inline void idtentry_exit(struct pt_regs *regs)
+{
+ idtentry_exit_cond_rcu(regs, true);
+}
/**
* DECLARE_IDTENTRY - Declare functions for simple IDT entry points
next prev parent reply other threads:[~2020-05-27 8:12 UTC|newest]
Thread overview: 159+ messages / expand[flat|nested] mbox.gz Atom feed top
2020-05-15 23:45 [patch V6 00/37] x86/entry: Rework leftovers and merge plan Thomas Gleixner
2020-05-15 23:45 ` [patch V6 01/37] tracing/hwlat: Use ktime_get_mono_fast_ns() Thomas Gleixner
2020-05-19 21:26 ` Steven Rostedt
2020-05-19 21:45 ` Thomas Gleixner
2020-05-19 22:18 ` Steven Rostedt
2020-05-20 19:51 ` Thomas Gleixner
2020-05-20 20:14 ` Peter Zijlstra
2020-05-20 22:20 ` Thomas Gleixner
2020-05-15 23:45 ` [patch V6 02/37] tracing/hwlat: Split ftrace_nmi_enter/exit() Thomas Gleixner
2020-05-19 22:23 ` Steven Rostedt
2020-05-15 23:45 ` [patch V6 03/37] nmi, tracing: Provide nmi_enter/exit_notrace() Thomas Gleixner
2020-05-17 5:12 ` Andy Lutomirski
2020-05-19 22:24 ` Steven Rostedt
2020-05-15 23:45 ` [patch V6 04/37] x86: Make hardware latency tracing explicit Thomas Gleixner
2020-05-17 5:36 ` Andy Lutomirski
2020-05-17 8:48 ` Thomas Gleixner
2020-05-18 5:50 ` Andy Lutomirski
2020-05-18 8:03 ` Thomas Gleixner
2020-05-18 20:42 ` Andy Lutomirski
2020-05-18 8:01 ` Peter Zijlstra
2020-05-18 8:05 ` Thomas Gleixner
2020-05-18 8:08 ` Peter Zijlstra
2020-05-20 20:09 ` Thomas Gleixner
2020-05-20 20:14 ` Andy Lutomirski
2020-05-20 22:20 ` Thomas Gleixner
2020-05-15 23:45 ` [patch V6 05/37] genirq: Provide irq_enter/exit_rcu() Thomas Gleixner
2020-05-18 23:06 ` Andy Lutomirski
2020-05-15 23:45 ` [patch V6 06/37] genirq: Provde __irq_enter/exit_raw() Thomas Gleixner
2020-05-18 23:07 ` Andy Lutomirski
2020-05-15 23:45 ` [patch V6 07/37] x86/entry: Provide helpers for execute on irqstack Thomas Gleixner
2020-05-18 23:11 ` Andy Lutomirski
2020-05-18 23:46 ` Andy Lutomirski
2020-05-18 23:53 ` Thomas Gleixner
2020-05-18 23:56 ` Andy Lutomirski
2020-05-20 12:35 ` Thomas Gleixner
2020-05-20 15:09 ` Andy Lutomirski
2020-05-20 15:27 ` Thomas Gleixner
2020-05-20 15:36 ` Andy Lutomirski
2020-05-18 23:51 ` Thomas Gleixner
2020-05-15 23:45 ` [patch V6 08/37] x86/entry/64: Move do_softirq_own_stack() to C Thomas Gleixner
2020-05-18 23:48 ` Andy Lutomirski
2020-05-15 23:45 ` [patch V6 09/37] x86/entry: Split idtentry_enter/exit() Thomas Gleixner
2020-05-18 23:49 ` Andy Lutomirski
2020-05-19 8:25 ` Thomas Gleixner
2020-05-15 23:45 ` [patch V6 10/37] x86/entry: Switch XEN/PV hypercall entry to IDTENTRY Thomas Gleixner
2020-05-19 17:06 ` Andy Lutomirski
2020-05-19 18:57 ` Thomas Gleixner
2020-05-19 19:44 ` Andy Lutomirski
2020-05-20 8:06 ` Jürgen Groß
2020-05-20 11:31 ` Andrew Cooper
2020-05-20 14:13 ` Thomas Gleixner
2020-05-20 15:16 ` Andy Lutomirski
2020-05-20 17:22 ` Andy Lutomirski
2020-05-20 19:16 ` Thomas Gleixner
2020-05-20 23:21 ` Andy Lutomirski
2020-05-21 10:45 ` Thomas Gleixner
2020-05-21 2:23 ` Boris Ostrovsky
2020-05-21 7:08 ` Thomas Gleixner
2020-05-15 23:45 ` [patch V6 11/37] x86/entry/64: Simplify idtentry_body Thomas Gleixner
2020-05-19 17:06 ` Andy Lutomirski
2020-05-15 23:45 ` [patch V6 12/37] x86/entry: Provide idtentry_entry/exit_cond_rcu() Thomas Gleixner
2020-05-19 17:08 ` Andy Lutomirski
2020-05-19 19:00 ` Thomas Gleixner
2020-05-19 20:20 ` Thomas Gleixner
2020-05-19 20:24 ` Andy Lutomirski
2020-05-19 21:20 ` Thomas Gleixner
2020-05-20 0:26 ` Andy Lutomirski
2020-05-20 2:23 ` Paul E. McKenney
2020-05-20 15:36 ` Andy Lutomirski
2020-05-20 16:51 ` Andy Lutomirski
2020-05-20 18:05 ` Paul E. McKenney
2020-05-20 19:49 ` Thomas Gleixner
2020-05-20 22:15 ` Paul E. McKenney
2020-05-20 23:25 ` Paul E. McKenney
2020-05-21 8:31 ` Thomas Gleixner
2020-05-21 13:39 ` Paul E. McKenney
2020-05-21 18:41 ` Thomas Gleixner
2020-05-21 19:04 ` Paul E. McKenney
2020-05-20 18:32 ` Thomas Gleixner
2020-05-20 19:24 ` Thomas Gleixner
2020-05-20 19:42 ` Paul E. McKenney
2020-05-20 17:38 ` Paul E. McKenney
2020-05-20 17:47 ` Andy Lutomirski
2020-05-20 18:11 ` Paul E. McKenney
2020-05-20 14:19 ` Thomas Gleixner
2020-05-27 8:12 ` tip-bot2 for Thomas Gleixner [this message]
2020-05-15 23:46 ` [patch V6 13/37] x86/entry: Switch page fault exception to IDTENTRY_RAW Thomas Gleixner
2020-05-19 20:12 ` Andy Lutomirski
2020-05-15 23:46 ` [patch V6 14/37] x86/entry: Remove the transition leftovers Thomas Gleixner
2020-05-19 20:13 ` Andy Lutomirski
2020-05-15 23:46 ` [patch V6 15/37] x86/entry: Change exit path of xen_failsafe_callback Thomas Gleixner
2020-05-19 20:14 ` Andy Lutomirski
2020-05-15 23:46 ` [patch V6 16/37] x86/entry/64: Remove error_exit Thomas Gleixner
2020-05-19 20:14 ` Andy Lutomirski
2020-05-15 23:46 ` [patch V6 17/37] x86/entry/32: Remove common_exception Thomas Gleixner
2020-05-19 20:14 ` Andy Lutomirski
2020-05-15 23:46 ` [patch V6 18/37] x86/irq: Use generic irq_regs implementation Thomas Gleixner
2020-05-15 23:46 ` [patch V6 19/37] x86/irq: Convey vector as argument and not in ptregs Thomas Gleixner
2020-05-19 20:19 ` Andy Lutomirski
2020-05-21 13:22 ` Thomas Gleixner
2020-05-22 18:48 ` Boris Ostrovsky
2020-05-22 19:26 ` Josh Poimboeuf
2020-05-22 19:54 ` Thomas Gleixner
2020-05-15 23:46 ` [patch V6 20/37] x86/irq/64: Provide handle_irq() Thomas Gleixner
2020-05-19 20:21 ` Andy Lutomirski
2020-05-15 23:46 ` [patch V6 21/37] x86/entry: Add IRQENTRY_IRQ macro Thomas Gleixner
2020-05-19 20:27 ` Andy Lutomirski
2020-05-15 23:46 ` [patch V6 22/37] x86/entry: Use idtentry for interrupts Thomas Gleixner
2020-05-19 20:28 ` Andy Lutomirski
2020-05-15 23:46 ` [patch V6 23/37] x86/entry: Provide IDTENTRY_SYSVEC Thomas Gleixner
2020-05-20 0:29 ` Andy Lutomirski
2020-05-20 15:07 ` Thomas Gleixner
2020-05-15 23:46 ` [patch V6 24/37] x86/entry: Convert APIC interrupts to IDTENTRY_SYSVEC Thomas Gleixner
2020-05-20 0:27 ` Andy Lutomirski
2020-05-15 23:46 ` [patch V6 25/37] x86/entry: Convert SMP system vectors " Thomas Gleixner
2020-05-20 0:28 ` Andy Lutomirski
2020-05-15 23:46 ` [patch V6 26/37] x86/entry: Convert various system vectors Thomas Gleixner
2020-05-20 0:30 ` Andy Lutomirski
2020-05-15 23:46 ` [patch V6 27/37] x86/entry: Convert KVM vectors to IDTENTRY_SYSVEC Thomas Gleixner
2020-05-20 0:30 ` Andy Lutomirski
2020-05-15 23:46 ` [patch V6 28/37] x86/entry: Convert various hypervisor " Thomas Gleixner
2020-05-20 0:31 ` Andy Lutomirski
2020-05-15 23:46 ` [patch V6 29/37] x86/entry: Convert XEN hypercall vector " Thomas Gleixner
2020-05-20 0:31 ` Andy Lutomirski
2020-05-15 23:46 ` [patch V6 30/37] x86/entry: Convert reschedule interrupt to IDTENTRY_RAW Thomas Gleixner
2020-05-19 23:57 ` Andy Lutomirski
2020-05-20 15:08 ` Thomas Gleixner
2020-05-15 23:46 ` [patch V6 31/37] x86/entry: Remove the apic/BUILD interrupt leftovers Thomas Gleixner
2020-05-20 0:32 ` Andy Lutomirski
2020-05-15 23:46 ` [patch V6 32/37] x86/entry/64: Remove IRQ stack switching ASM Thomas Gleixner
2020-05-20 0:33 ` Andy Lutomirski
2020-05-15 23:46 ` [patch V6 33/37] x86/entry: Make enter_from_user_mode() static Thomas Gleixner
2020-05-20 0:34 ` Andy Lutomirski
2020-05-15 23:46 ` [patch V6 34/37] x86/entry/32: Remove redundant irq disable code Thomas Gleixner
2020-05-20 0:35 ` Andy Lutomirski
2020-05-15 23:46 ` [patch V6 35/37] x86/entry/64: Remove TRACE_IRQS_*_DEBUG Thomas Gleixner
2020-05-20 0:46 ` Andy Lutomirski
2020-05-15 23:46 ` [patch V6 36/37] x86/entry: Move paranoid irq tracing out of ASM code Thomas Gleixner
2020-05-20 0:53 ` Andy Lutomirski
2020-05-20 15:16 ` Thomas Gleixner
2020-05-20 17:13 ` Andy Lutomirski
2020-05-20 18:33 ` Thomas Gleixner
2020-05-15 23:46 ` [patch V6 37/37] x86/entry: Remove the TRACE_IRQS cruft Thomas Gleixner
2020-05-18 23:07 ` Andy Lutomirski
2020-05-16 17:18 ` [patch V6 00/37] x86/entry: Rework leftovers and merge plan Paul E. McKenney
2020-05-19 12:28 ` Joel Fernandes
2020-05-18 16:07 ` Peter Zijlstra
2020-05-18 18:53 ` Thomas Gleixner
2020-05-19 8:29 ` Peter Zijlstra
2020-05-18 20:24 ` Thomas Gleixner
2020-05-19 8:38 ` Peter Zijlstra
2020-05-19 9:02 ` Peter Zijlstra
2020-05-23 2:52 ` Lai Jiangshan
2020-05-23 13:08 ` Peter Zijlstra
2020-06-15 16:17 ` Peter Zijlstra
2020-05-19 9:06 ` Thomas Gleixner
2020-05-19 18:37 ` Steven Rostedt
2020-05-19 19:09 ` Thomas Gleixner
2020-05-19 19:13 ` Steven Rostedt
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=159056712087.17951.10894256787068855437.tip-bot2@tip-bot2 \
--to=tip-bot2@linutronix.de \
--cc=linux-kernel@vger.kernel.org \
--cc=linux-tip-commits@vger.kernel.org \
--cc=luto@kernel.org \
--cc=mingo@kernel.org \
--cc=paulmck@kernel.org \
--cc=tglx@linutronix.de \
--cc=x86@kernel.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).