From: Thomas Gleixner <tglx@linutronix.de>
To: LKML <linux-kernel@vger.kernel.org>
Cc: x86@kernel.org, Josh Poimboeuf <jpoimboe@redhat.com>,
Kees Cook <keescook@chromium.org>
Subject: [patch 02/12] x86/irq: Sanitize irq stack tracking
Date: Thu, 04 Feb 2021 21:49:05 +0100 [thread overview]
Message-ID: <20210204211154.257738890@linutronix.de> (raw)
In-Reply-To: 20210204204903.350275743@linutronix.de
The recursion protection for hard interrupt stacks is an unsigned int per
CPU variable initialized to -1 named __irq_count.
The irq stack switching is only done when the variable is -1, which creates
worse code than just checking for 0. When the stack switching happens it
uses this_cpu_add/sub(1), but there is no reason to do so. It simply can
use straight writes. This is a historical leftover from the low level ASM
code which used inc and jz to make a decision.
Rename it to hardirq_stack_inuse, make it a bool and use plain stores.
Signed-off-by: Thomas Gleixner <tglx@linutronix.de>
---
arch/x86/include/asm/irq_stack.h | 14 +++++++-------
arch/x86/include/asm/processor.h | 2 +-
arch/x86/kernel/cpu/common.c | 2 +-
arch/x86/kernel/process_64.c | 2 +-
4 files changed, 10 insertions(+), 10 deletions(-)
--- a/arch/x86/include/asm/irq_stack.h
+++ b/arch/x86/include/asm/irq_stack.h
@@ -9,7 +9,7 @@
#ifdef CONFIG_X86_64
static __always_inline bool irqstack_active(void)
{
- return __this_cpu_read(irq_count) != -1;
+ return __this_cpu_read(hardirq_stack_inuse);
}
void asm_call_on_stack(void *sp, void (*func)(void), void *arg);
@@ -22,9 +22,9 @@ static __always_inline void __run_on_irq
{
void *tos = __this_cpu_read(hardirq_stack_ptr);
- __this_cpu_add(irq_count, 1);
+ __this_cpu_write(hardirq_stack_inuse, true);
asm_call_on_stack(tos - 8, func, NULL);
- __this_cpu_sub(irq_count, 1);
+ __this_cpu_write(hardirq_stack_inuse, false);
}
static __always_inline void
@@ -33,9 +33,9 @@ static __always_inline void
{
void *tos = __this_cpu_read(hardirq_stack_ptr);
- __this_cpu_add(irq_count, 1);
+ __this_cpu_write(hardirq_stack_inuse, true);
asm_call_sysvec_on_stack(tos - 8, func, regs);
- __this_cpu_sub(irq_count, 1);
+ __this_cpu_write(hardirq_stack_inuse, false);
}
static __always_inline void
@@ -44,9 +44,9 @@ static __always_inline void
{
void *tos = __this_cpu_read(hardirq_stack_ptr);
- __this_cpu_add(irq_count, 1);
+ __this_cpu_write(hardirq_stack_inuse, true);
asm_call_irq_on_stack(tos - 8, func, desc);
- __this_cpu_sub(irq_count, 1);
+ __this_cpu_write(hardirq_stack_inuse, false);
}
#else /* CONFIG_X86_64 */
--- a/arch/x86/include/asm/processor.h
+++ b/arch/x86/include/asm/processor.h
@@ -454,7 +454,7 @@ static inline unsigned long cpu_kernelmo
return (unsigned long)per_cpu(fixed_percpu_data.gs_base, cpu);
}
-DECLARE_PER_CPU(unsigned int, irq_count);
+DECLARE_PER_CPU(bool, hardirq_stack_inuse);
extern asmlinkage void ignore_sysret(void);
/* Save actual FS/GS selectors and bases to current->thread */
--- a/arch/x86/kernel/cpu/common.c
+++ b/arch/x86/kernel/cpu/common.c
@@ -1740,7 +1740,7 @@ DEFINE_PER_CPU(struct task_struct *, cur
EXPORT_PER_CPU_SYMBOL(current_task);
DEFINE_PER_CPU(struct irq_stack *, hardirq_stack_ptr);
-DEFINE_PER_CPU(unsigned int, irq_count) __visible = -1;
+DEFINE_PER_CPU(bool, hardirq_stack_inuse);
DEFINE_PER_CPU(int, __preempt_count) = INIT_PREEMPT_COUNT;
EXPORT_PER_CPU_SYMBOL(__preempt_count);
--- a/arch/x86/kernel/process_64.c
+++ b/arch/x86/kernel/process_64.c
@@ -539,7 +539,7 @@ void compat_start_thread(struct pt_regs
int cpu = smp_processor_id();
WARN_ON_ONCE(IS_ENABLED(CONFIG_DEBUG_ENTRY) &&
- this_cpu_read(irq_count) != -1);
+ this_cpu_read(hardirq_stack_inuse));
if (!test_thread_flag(TIF_NEED_FPU_LOAD))
switch_fpu_prepare(prev_fpu, cpu);
next prev parent reply other threads:[~2021-02-05 0:58 UTC|newest]
Thread overview: 32+ messages / expand[flat|nested] mbox.gz Atom feed top
2021-02-04 20:49 [patch 00/12] x86/irq/64: Inline irq stack switching Thomas Gleixner
2021-02-04 20:49 ` [patch 01/12] x86/entry: Fix instrumentation annotation Thomas Gleixner
2021-02-05 12:02 ` Borislav Petkov
2021-02-04 20:49 ` Thomas Gleixner [this message]
2021-02-04 20:49 ` [patch 03/12] x86/irq/64: Adjust the per CPU irq stack pointer by 8 Thomas Gleixner
2021-02-05 13:53 ` Peter Zijlstra
2021-02-05 15:00 ` Thomas Gleixner
2021-02-04 20:49 ` [patch 04/12] x86/apic: Split out spurious handling code Thomas Gleixner
2021-02-04 20:49 ` [patch 05/12] x86/irq: Provide macro for inlining irq stack switching Thomas Gleixner
2021-02-08 15:06 ` Borislav Petkov
2021-02-08 20:42 ` Josh Poimboeuf
2021-02-09 15:12 ` Thomas Gleixner
2021-02-09 16:22 ` Josh Poimboeuf
2021-02-09 18:14 ` Thomas Gleixner
2021-02-04 20:49 ` [patch 06/12] x86/entry: Convert system vectors to irq stack macro Thomas Gleixner
2021-02-05 3:52 ` kernel test robot
2021-02-05 14:13 ` Peter Zijlstra
2021-02-07 8:15 ` [kbuild-all] " Rong Chen
2021-02-08 14:19 ` Borislav Petkov
2021-02-09 8:57 ` Rong Chen
2021-02-09 10:46 ` Borislav Petkov
2021-02-04 20:49 ` [patch 07/12] x86/entry: Convert device interrupts to inline stack switching Thomas Gleixner
2021-02-04 20:49 ` [patch 08/12] x86/entry: Use run_sysvec_on_irqstack_cond() for XEN upcall Thomas Gleixner
2021-02-05 7:28 ` kernel test robot
2021-02-04 20:49 ` [patch 09/12] x86/softirq: Remove indirection in do_softirq_own_stack() Thomas Gleixner
2021-02-04 20:49 ` [patch 10/12] x86: Select CONFIG_HAVE_IRQ_EXIT_ON_IRQ_STACK Thomas Gleixner
2021-02-04 20:49 ` [patch 11/12] softirq: Allow inlining do_softirq_own_stack() Thomas Gleixner
2021-02-05 10:14 ` Lai Jiangshan
2021-02-05 11:38 ` Thomas Gleixner
2021-02-09 20:43 ` Thomas Gleixner
2021-02-04 20:49 ` [patch 12/12] x86/softirq/64: Inline do_softirq_own_stack() Thomas Gleixner
2021-02-08 21:47 ` [patch 00/12] x86/irq/64: Inline irq stack switching Kees Cook
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20210204211154.257738890@linutronix.de \
--to=tglx@linutronix.de \
--cc=jpoimboe@redhat.com \
--cc=keescook@chromium.org \
--cc=linux-kernel@vger.kernel.org \
--cc=x86@kernel.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).