* [patch V2 01/13] x86/entry: Fix instrumentation annotation
2021-02-09 23:40 [patch V2 00/13] x86/irq/64: Inline irq stack switching Thomas Gleixner
@ 2021-02-09 23:40 ` Thomas Gleixner
2021-02-11 0:50 ` [tip: x86/entry] " tip-bot2 for Thomas Gleixner
2021-02-09 23:40 ` [patch V2 02/13] x86/irq: Sanitize irq stack tracking Thomas Gleixner
` (11 subsequent siblings)
12 siblings, 1 reply; 31+ messages in thread
From: Thomas Gleixner @ 2021-02-09 23:40 UTC (permalink / raw)
To: LKML
Cc: x86, Josh Poimboeuf, Helge Deller, David S. Miller,
Michael Ellerman, Rich Felker, Heiko Carstens, Kees Cook, stable,
Lai Jiangshan
From: Thomas Gleixner <tglx@linutronix.de>
Embracing a callout into instrumentation_begin() / instrumentation_begin()
does not really make sense. Make the latter instrumentation_end().
Fixes: 2f6474e4636b ("x86/entry: Switch XEN/PV hypercall entry to IDTENTRY")
Signed-off-by: Thomas Gleixner <tglx@linutronix.de>
Reviewed-by: Kees Cook <keescook@chromium.org>
Cc: stable@vger.kernel.org
---
arch/x86/entry/common.c | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
--- a/arch/x86/entry/common.c
+++ b/arch/x86/entry/common.c
@@ -270,7 +270,7 @@ static void __xen_pv_evtchn_do_upcall(vo
instrumentation_begin();
run_on_irqstack_cond(__xen_pv_evtchn_do_upcall, regs);
- instrumentation_begin();
+ instrumentation_end();
set_irq_regs(old_regs);
^ permalink raw reply [flat|nested] 31+ messages in thread* [tip: x86/entry] x86/entry: Fix instrumentation annotation
2021-02-09 23:40 ` [patch V2 01/13] x86/entry: Fix instrumentation annotation Thomas Gleixner
@ 2021-02-11 0:50 ` tip-bot2 for Thomas Gleixner
0 siblings, 0 replies; 31+ messages in thread
From: tip-bot2 for Thomas Gleixner @ 2021-02-11 0:50 UTC (permalink / raw)
To: linux-tip-commits; +Cc: Thomas Gleixner, Kees Cook, stable, x86, linux-kernel
The following commit has been merged into the x86/entry branch of tip:
Commit-ID: 15f720aabe71a5662c4198b22532d95bbeec80ef
Gitweb: https://git.kernel.org/tip/15f720aabe71a5662c4198b22532d95bbeec80ef
Author: Thomas Gleixner <tglx@linutronix.de>
AuthorDate: Wed, 10 Feb 2021 00:40:42 +01:00
Committer: Thomas Gleixner <tglx@linutronix.de>
CommitterDate: Wed, 10 Feb 2021 23:34:13 +01:00
x86/entry: Fix instrumentation annotation
Embracing a callout into instrumentation_begin() / instrumentation_begin()
does not really make sense. Make the latter instrumentation_end().
Fixes: 2f6474e4636b ("x86/entry: Switch XEN/PV hypercall entry to IDTENTRY")
Signed-off-by: Thomas Gleixner <tglx@linutronix.de>
Reviewed-by: Kees Cook <keescook@chromium.org>
Cc: stable@vger.kernel.org
Link: https://lore.kernel.org/r/20210210002512.106502464@linutronix.de
---
arch/x86/entry/common.c | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/arch/x86/entry/common.c b/arch/x86/entry/common.c
index 18d8f17..c4efffb 100644
--- a/arch/x86/entry/common.c
+++ b/arch/x86/entry/common.c
@@ -266,7 +266,7 @@ __visible noinstr void xen_pv_evtchn_do_upcall(struct pt_regs *regs)
instrumentation_begin();
run_on_irqstack_cond(__xen_pv_evtchn_do_upcall, regs);
- instrumentation_begin();
+ instrumentation_end();
set_irq_regs(old_regs);
^ permalink raw reply related [flat|nested] 31+ messages in thread
* [patch V2 02/13] x86/irq: Sanitize irq stack tracking
2021-02-09 23:40 [patch V2 00/13] x86/irq/64: Inline irq stack switching Thomas Gleixner
2021-02-09 23:40 ` [patch V2 01/13] x86/entry: Fix instrumentation annotation Thomas Gleixner
@ 2021-02-09 23:40 ` Thomas Gleixner
2021-02-11 0:50 ` [tip: x86/entry] " tip-bot2 for Thomas Gleixner
2021-02-09 23:40 ` [patch V2 03/13] x86/irq/64: Adjust the per CPU irq stack pointer by 8 Thomas Gleixner
` (10 subsequent siblings)
12 siblings, 1 reply; 31+ messages in thread
From: Thomas Gleixner @ 2021-02-09 23:40 UTC (permalink / raw)
To: LKML
Cc: x86, Josh Poimboeuf, Helge Deller, David S. Miller,
Michael Ellerman, Rich Felker, Heiko Carstens, Kees Cook,
Lai Jiangshan
From: Thomas Gleixner <tglx@linutronix.de>
The recursion protection for hard interrupt stacks is an unsigned int per
CPU variable initialized to -1 named __irq_count.
The irq stack switching is only done when the variable is -1, which creates
worse code than just checking for 0. When the stack switching happens it
uses this_cpu_add/sub(1), but there is no reason to do so. It simply can
use straight writes. This is a historical leftover from the low level ASM
code which used inc and jz to make a decision.
Rename it to hardirq_stack_inuse, make it a bool and use plain stores.
Signed-off-by: Thomas Gleixner <tglx@linutronix.de>
Reviewed-by: Kees Cook <keescook@chromium.org>
---
arch/x86/include/asm/irq_stack.h | 14 +++++++-------
arch/x86/include/asm/processor.h | 2 +-
arch/x86/kernel/cpu/common.c | 2 +-
arch/x86/kernel/process_64.c | 2 +-
4 files changed, 10 insertions(+), 10 deletions(-)
--- a/arch/x86/include/asm/irq_stack.h
+++ b/arch/x86/include/asm/irq_stack.h
@@ -9,7 +9,7 @@
#ifdef CONFIG_X86_64
static __always_inline bool irqstack_active(void)
{
- return __this_cpu_read(irq_count) != -1;
+ return __this_cpu_read(hardirq_stack_inuse);
}
void asm_call_on_stack(void *sp, void (*func)(void), void *arg);
@@ -22,9 +22,9 @@ static __always_inline void __run_on_irq
{
void *tos = __this_cpu_read(hardirq_stack_ptr);
- __this_cpu_add(irq_count, 1);
+ __this_cpu_write(hardirq_stack_inuse, true);
asm_call_on_stack(tos - 8, func, NULL);
- __this_cpu_sub(irq_count, 1);
+ __this_cpu_write(hardirq_stack_inuse, false);
}
static __always_inline void
@@ -33,9 +33,9 @@ static __always_inline void
{
void *tos = __this_cpu_read(hardirq_stack_ptr);
- __this_cpu_add(irq_count, 1);
+ __this_cpu_write(hardirq_stack_inuse, true);
asm_call_sysvec_on_stack(tos - 8, func, regs);
- __this_cpu_sub(irq_count, 1);
+ __this_cpu_write(hardirq_stack_inuse, false);
}
static __always_inline void
@@ -44,9 +44,9 @@ static __always_inline void
{
void *tos = __this_cpu_read(hardirq_stack_ptr);
- __this_cpu_add(irq_count, 1);
+ __this_cpu_write(hardirq_stack_inuse, true);
asm_call_irq_on_stack(tos - 8, func, desc);
- __this_cpu_sub(irq_count, 1);
+ __this_cpu_write(hardirq_stack_inuse, false);
}
#else /* CONFIG_X86_64 */
--- a/arch/x86/include/asm/processor.h
+++ b/arch/x86/include/asm/processor.h
@@ -454,7 +454,7 @@ static inline unsigned long cpu_kernelmo
return (unsigned long)per_cpu(fixed_percpu_data.gs_base, cpu);
}
-DECLARE_PER_CPU(unsigned int, irq_count);
+DECLARE_PER_CPU(bool, hardirq_stack_inuse);
extern asmlinkage void ignore_sysret(void);
/* Save actual FS/GS selectors and bases to current->thread */
--- a/arch/x86/kernel/cpu/common.c
+++ b/arch/x86/kernel/cpu/common.c
@@ -1740,7 +1740,7 @@ DEFINE_PER_CPU(struct task_struct *, cur
EXPORT_PER_CPU_SYMBOL(current_task);
DEFINE_PER_CPU(struct irq_stack *, hardirq_stack_ptr);
-DEFINE_PER_CPU(unsigned int, irq_count) __visible = -1;
+DEFINE_PER_CPU(bool, hardirq_stack_inuse);
DEFINE_PER_CPU(int, __preempt_count) = INIT_PREEMPT_COUNT;
EXPORT_PER_CPU_SYMBOL(__preempt_count);
--- a/arch/x86/kernel/process_64.c
+++ b/arch/x86/kernel/process_64.c
@@ -539,7 +539,7 @@ void compat_start_thread(struct pt_regs
int cpu = smp_processor_id();
WARN_ON_ONCE(IS_ENABLED(CONFIG_DEBUG_ENTRY) &&
- this_cpu_read(irq_count) != -1);
+ this_cpu_read(hardirq_stack_inuse));
if (!test_thread_flag(TIF_NEED_FPU_LOAD))
switch_fpu_prepare(prev_fpu, cpu);
^ permalink raw reply [flat|nested] 31+ messages in thread* [tip: x86/entry] x86/irq: Sanitize irq stack tracking
2021-02-09 23:40 ` [patch V2 02/13] x86/irq: Sanitize irq stack tracking Thomas Gleixner
@ 2021-02-11 0:50 ` tip-bot2 for Thomas Gleixner
0 siblings, 0 replies; 31+ messages in thread
From: tip-bot2 for Thomas Gleixner @ 2021-02-11 0:50 UTC (permalink / raw)
To: linux-tip-commits; +Cc: Thomas Gleixner, Kees Cook, x86, linux-kernel
The following commit has been merged into the x86/entry branch of tip:
Commit-ID: e7f89001797148e8dc7060c335df2c56e73a8c7a
Gitweb: https://git.kernel.org/tip/e7f89001797148e8dc7060c335df2c56e73a8c7a
Author: Thomas Gleixner <tglx@linutronix.de>
AuthorDate: Wed, 10 Feb 2021 00:40:43 +01:00
Committer: Thomas Gleixner <tglx@linutronix.de>
CommitterDate: Wed, 10 Feb 2021 23:34:13 +01:00
x86/irq: Sanitize irq stack tracking
The recursion protection for hard interrupt stacks is an unsigned int per
CPU variable initialized to -1 named __irq_count.
The irq stack switching is only done when the variable is -1, which creates
worse code than just checking for 0. When the stack switching happens it
uses this_cpu_add/sub(1), but there is no reason to do so. It simply can
use straight writes. This is a historical leftover from the low level ASM
code which used inc and jz to make a decision.
Rename it to hardirq_stack_inuse, make it a bool and use plain stores.
Signed-off-by: Thomas Gleixner <tglx@linutronix.de>
Reviewed-by: Kees Cook <keescook@chromium.org>
Link: https://lore.kernel.org/r/20210210002512.228830141@linutronix.de
---
arch/x86/include/asm/irq_stack.h | 14 +++++++-------
arch/x86/include/asm/processor.h | 2 +-
arch/x86/kernel/cpu/common.c | 2 +-
arch/x86/kernel/process_64.c | 2 +-
4 files changed, 10 insertions(+), 10 deletions(-)
diff --git a/arch/x86/include/asm/irq_stack.h b/arch/x86/include/asm/irq_stack.h
index 7758169..4487bb9 100644
--- a/arch/x86/include/asm/irq_stack.h
+++ b/arch/x86/include/asm/irq_stack.h
@@ -9,7 +9,7 @@
#ifdef CONFIG_X86_64
static __always_inline bool irqstack_active(void)
{
- return __this_cpu_read(irq_count) != -1;
+ return __this_cpu_read(hardirq_stack_inuse);
}
void asm_call_on_stack(void *sp, void (*func)(void), void *arg);
@@ -22,9 +22,9 @@ static __always_inline void __run_on_irqstack(void (*func)(void))
{
void *tos = __this_cpu_read(hardirq_stack_ptr);
- __this_cpu_add(irq_count, 1);
+ __this_cpu_write(hardirq_stack_inuse, true);
asm_call_on_stack(tos - 8, func, NULL);
- __this_cpu_sub(irq_count, 1);
+ __this_cpu_write(hardirq_stack_inuse, false);
}
static __always_inline void
@@ -33,9 +33,9 @@ __run_sysvec_on_irqstack(void (*func)(struct pt_regs *regs),
{
void *tos = __this_cpu_read(hardirq_stack_ptr);
- __this_cpu_add(irq_count, 1);
+ __this_cpu_write(hardirq_stack_inuse, true);
asm_call_sysvec_on_stack(tos - 8, func, regs);
- __this_cpu_sub(irq_count, 1);
+ __this_cpu_write(hardirq_stack_inuse, false);
}
static __always_inline void
@@ -44,9 +44,9 @@ __run_irq_on_irqstack(void (*func)(struct irq_desc *desc),
{
void *tos = __this_cpu_read(hardirq_stack_ptr);
- __this_cpu_add(irq_count, 1);
+ __this_cpu_write(hardirq_stack_inuse, true);
asm_call_irq_on_stack(tos - 8, func, desc);
- __this_cpu_sub(irq_count, 1);
+ __this_cpu_write(hardirq_stack_inuse, false);
}
#else /* CONFIG_X86_64 */
diff --git a/arch/x86/include/asm/processor.h b/arch/x86/include/asm/processor.h
index c20a52b..11d10f4 100644
--- a/arch/x86/include/asm/processor.h
+++ b/arch/x86/include/asm/processor.h
@@ -454,7 +454,7 @@ static inline unsigned long cpu_kernelmode_gs_base(int cpu)
return (unsigned long)per_cpu(fixed_percpu_data.gs_base, cpu);
}
-DECLARE_PER_CPU(unsigned int, irq_count);
+DECLARE_PER_CPU(bool, hardirq_stack_inuse);
extern asmlinkage void ignore_sysret(void);
/* Save actual FS/GS selectors and bases to current->thread */
diff --git a/arch/x86/kernel/cpu/common.c b/arch/x86/kernel/cpu/common.c
index 35ad848..845c8a4 100644
--- a/arch/x86/kernel/cpu/common.c
+++ b/arch/x86/kernel/cpu/common.c
@@ -1740,7 +1740,7 @@ DEFINE_PER_CPU(struct task_struct *, current_task) ____cacheline_aligned =
EXPORT_PER_CPU_SYMBOL(current_task);
DEFINE_PER_CPU(struct irq_stack *, hardirq_stack_ptr);
-DEFINE_PER_CPU(unsigned int, irq_count) __visible = -1;
+DEFINE_PER_CPU(bool, hardirq_stack_inuse);
DEFINE_PER_CPU(int, __preempt_count) = INIT_PREEMPT_COUNT;
EXPORT_PER_CPU_SYMBOL(__preempt_count);
diff --git a/arch/x86/kernel/process_64.c b/arch/x86/kernel/process_64.c
index ad582f9..d08307d 100644
--- a/arch/x86/kernel/process_64.c
+++ b/arch/x86/kernel/process_64.c
@@ -539,7 +539,7 @@ __switch_to(struct task_struct *prev_p, struct task_struct *next_p)
int cpu = smp_processor_id();
WARN_ON_ONCE(IS_ENABLED(CONFIG_DEBUG_ENTRY) &&
- this_cpu_read(irq_count) != -1);
+ this_cpu_read(hardirq_stack_inuse));
if (!test_thread_flag(TIF_NEED_FPU_LOAD))
switch_fpu_prepare(prev_fpu, cpu);
^ permalink raw reply related [flat|nested] 31+ messages in thread
* [patch V2 03/13] x86/irq/64: Adjust the per CPU irq stack pointer by 8
2021-02-09 23:40 [patch V2 00/13] x86/irq/64: Inline irq stack switching Thomas Gleixner
2021-02-09 23:40 ` [patch V2 01/13] x86/entry: Fix instrumentation annotation Thomas Gleixner
2021-02-09 23:40 ` [patch V2 02/13] x86/irq: Sanitize irq stack tracking Thomas Gleixner
@ 2021-02-09 23:40 ` Thomas Gleixner
2021-02-10 11:41 ` David Laight
2021-02-11 0:50 ` [tip: x86/entry] " tip-bot2 for Thomas Gleixner
2021-02-09 23:40 ` [patch V2 04/13] x86/apic: Split out spurious handling code Thomas Gleixner
` (9 subsequent siblings)
12 siblings, 2 replies; 31+ messages in thread
From: Thomas Gleixner @ 2021-02-09 23:40 UTC (permalink / raw)
To: LKML
Cc: x86, Josh Poimboeuf, Helge Deller, David S. Miller,
Michael Ellerman, Rich Felker, Heiko Carstens, Kees Cook,
Lai Jiangshan
From: Thomas Gleixner <tglx@linutronix.de>
The per CPU hardirq_stack_ptr contains the pointer to the irq stack in the
form that it is ready to be assigned to [ER]SP so that the first push ends
up on the top entry of the stack.
But the stack switching on 64 bit has the following rules:
1) Store the current stack pointer (RSP) in the top most stack entry
to allow the unwinder to link back to the previous stack
2) Set RSP to the top most stack entry
3) Invoke functions on the irq stack
4) Pop RSP from the top most stack entry (stored in #1) so it's back
to the original stack.
That requires all stack switching code to decrement the stored pointer by 8
in order to be able to store the current RSP and then set RSP to that
location. That's a pointless exercise.
Do the -8 adjustment right when storing the pointer and make the data type
a void pointer to avoid confusion vs. the struct irq_stack data type which
is on 64bit only used to declare the backing store. Move the definition
next to the inuse flag so they likely end up in the same cache
line. Sticking them into a struct to enforce it is a seperate change.
Signed-off-by: Thomas Gleixner <tglx@linutronix.de>
Reviewed-by: Kees Cook <keescook@chromium.org>
---
V2: Reworded changelog so that it's clear that ending up in the cache line
is not guaranteed.
---
arch/x86/include/asm/irq_stack.h | 6 +++---
arch/x86/include/asm/processor.h | 7 +++----
arch/x86/kernel/cpu/common.c | 2 +-
arch/x86/kernel/dumpstack_64.c | 22 ++++++++++++++++------
arch/x86/kernel/irq_64.c | 6 ++++--
5 files changed, 27 insertions(+), 16 deletions(-)
--- a/arch/x86/include/asm/irq_stack.h
+++ b/arch/x86/include/asm/irq_stack.h
@@ -23,7 +23,7 @@ static __always_inline void __run_on_irq
void *tos = __this_cpu_read(hardirq_stack_ptr);
__this_cpu_write(hardirq_stack_inuse, true);
- asm_call_on_stack(tos - 8, func, NULL);
+ asm_call_on_stack(tos, func, NULL);
__this_cpu_write(hardirq_stack_inuse, false);
}
@@ -34,7 +34,7 @@ static __always_inline void
void *tos = __this_cpu_read(hardirq_stack_ptr);
__this_cpu_write(hardirq_stack_inuse, true);
- asm_call_sysvec_on_stack(tos - 8, func, regs);
+ asm_call_sysvec_on_stack(tos, func, regs);
__this_cpu_write(hardirq_stack_inuse, false);
}
@@ -45,7 +45,7 @@ static __always_inline void
void *tos = __this_cpu_read(hardirq_stack_ptr);
__this_cpu_write(hardirq_stack_inuse, true);
- asm_call_irq_on_stack(tos - 8, func, desc);
+ asm_call_irq_on_stack(tos, func, desc);
__this_cpu_write(hardirq_stack_inuse, false);
}
--- a/arch/x86/include/asm/processor.h
+++ b/arch/x86/include/asm/processor.h
@@ -426,8 +426,6 @@ struct irq_stack {
char stack[IRQ_STACK_SIZE];
} __aligned(IRQ_STACK_SIZE);
-DECLARE_PER_CPU(struct irq_stack *, hardirq_stack_ptr);
-
#ifdef CONFIG_X86_32
DECLARE_PER_CPU(unsigned long, cpu_current_top_of_stack);
#else
@@ -454,6 +452,7 @@ static inline unsigned long cpu_kernelmo
return (unsigned long)per_cpu(fixed_percpu_data.gs_base, cpu);
}
+DECLARE_PER_CPU(void *, hardirq_stack_ptr);
DECLARE_PER_CPU(bool, hardirq_stack_inuse);
extern asmlinkage void ignore_sysret(void);
@@ -473,9 +472,9 @@ struct stack_canary {
};
DECLARE_PER_CPU_ALIGNED(struct stack_canary, stack_canary);
#endif
-/* Per CPU softirq stack pointer */
+DECLARE_PER_CPU(struct irq_stack *, hardirq_stack_ptr);
DECLARE_PER_CPU(struct irq_stack *, softirq_stack_ptr);
-#endif /* X86_64 */
+#endif /* !X86_64 */
extern unsigned int fpu_kernel_xstate_size;
extern unsigned int fpu_user_xstate_size;
--- a/arch/x86/kernel/cpu/common.c
+++ b/arch/x86/kernel/cpu/common.c
@@ -1739,7 +1739,7 @@ DEFINE_PER_CPU(struct task_struct *, cur
&init_task;
EXPORT_PER_CPU_SYMBOL(current_task);
-DEFINE_PER_CPU(struct irq_stack *, hardirq_stack_ptr);
+DEFINE_PER_CPU(void *, hardirq_stack_ptr);
DEFINE_PER_CPU(bool, hardirq_stack_inuse);
DEFINE_PER_CPU(int, __preempt_count) = INIT_PREEMPT_COUNT;
--- a/arch/x86/kernel/dumpstack_64.c
+++ b/arch/x86/kernel/dumpstack_64.c
@@ -128,12 +128,21 @@ static __always_inline bool in_exception
static __always_inline bool in_irq_stack(unsigned long *stack, struct stack_info *info)
{
- unsigned long *end = (unsigned long *)this_cpu_read(hardirq_stack_ptr);
- unsigned long *begin = end - (IRQ_STACK_SIZE / sizeof(long));
+ unsigned long *end = (unsigned long *)this_cpu_read(hardirq_stack_ptr);
+ unsigned long *begin;
/*
- * This is a software stack, so 'end' can be a valid stack pointer.
- * It just means the stack is empty.
+ * @end points directly to the top most stack entry to avoid a -8
+ * adjustment in the stack switch hotpath. Adjust it back before
+ * calculating @begin.
+ */
+ end++;
+ begin = end - (IRQ_STACK_SIZE / sizeof(long));
+
+ /*
+ * Due to the switching logic RSP can never be == @end because the
+ * final operation is 'popq %rsp' which means after that RSP points
+ * to the original stack and not to @end.
*/
if (stack < begin || stack >= end)
return false;
@@ -143,8 +152,9 @@ static __always_inline bool in_irq_stack
info->end = end;
/*
- * The next stack pointer is the first thing pushed by the entry code
- * after switching to the irq stack.
+ * The next stack pointer is stored at the top of the irq stack
+ * before switching to the irq stack. Actual stack entries are all
+ * below that.
*/
info->next_sp = (unsigned long *)*(end - 1);
--- a/arch/x86/kernel/irq_64.c
+++ b/arch/x86/kernel/irq_64.c
@@ -48,7 +48,8 @@ static int map_irq_stack(unsigned int cp
if (!va)
return -ENOMEM;
- per_cpu(hardirq_stack_ptr, cpu) = va + IRQ_STACK_SIZE;
+ /* Store actual TOS to avoid adjustment in the hotpath */
+ per_cpu(hardirq_stack_ptr, cpu) = va + IRQ_STACK_SIZE - 8;
return 0;
}
#else
@@ -60,7 +61,8 @@ static int map_irq_stack(unsigned int cp
{
void *va = per_cpu_ptr(&irq_stack_backing_store, cpu);
- per_cpu(hardirq_stack_ptr, cpu) = va + IRQ_STACK_SIZE;
+ /* Store actual TOS to avoid adjustment in the hotpath */
+ per_cpu(hardirq_stack_ptr, cpu) = va + IRQ_STACK_SIZE - 8;
return 0;
}
#endif
^ permalink raw reply [flat|nested] 31+ messages in thread* RE: [patch V2 03/13] x86/irq/64: Adjust the per CPU irq stack pointer by 8
2021-02-09 23:40 ` [patch V2 03/13] x86/irq/64: Adjust the per CPU irq stack pointer by 8 Thomas Gleixner
@ 2021-02-10 11:41 ` David Laight
2021-02-11 0:50 ` [tip: x86/entry] " tip-bot2 for Thomas Gleixner
1 sibling, 0 replies; 31+ messages in thread
From: David Laight @ 2021-02-10 11:41 UTC (permalink / raw)
To: 'Thomas Gleixner', LKML
Cc: x86, Josh Poimboeuf, Helge Deller, David S. Miller,
Michael Ellerman, Rich Felker, Heiko Carstens, Kees Cook,
Lai Jiangshan
From: Thomas Gleixner
> Sent: 09 February 2021 23:41
> From: Thomas Gleixner <tglx@linutronix.de>
>
> The per CPU hardirq_stack_ptr contains the pointer to the irq stack in the
> form that it is ready to be assigned to [ER]SP so that the first push ends
> up on the top entry of the stack.
...
> Do the -8 adjustment right when storing the pointer and make the data type
> a void pointer to avoid confusion vs. the struct irq_stack data type which
> is on 64bit only used to declare the backing store...
It might be better to make it a pointer to an undefined struct.
That will stop all accidental assignments - except those that
are already 'void *'.
David
-
Registered Address Lakeside, Bramley Road, Mount Farm, Milton Keynes, MK1 1PT, UK
Registration No: 1397386 (Wales)
^ permalink raw reply [flat|nested] 31+ messages in thread
* [tip: x86/entry] x86/irq/64: Adjust the per CPU irq stack pointer by 8
2021-02-09 23:40 ` [patch V2 03/13] x86/irq/64: Adjust the per CPU irq stack pointer by 8 Thomas Gleixner
2021-02-10 11:41 ` David Laight
@ 2021-02-11 0:50 ` tip-bot2 for Thomas Gleixner
1 sibling, 0 replies; 31+ messages in thread
From: tip-bot2 for Thomas Gleixner @ 2021-02-11 0:50 UTC (permalink / raw)
To: linux-tip-commits; +Cc: Thomas Gleixner, Kees Cook, x86, linux-kernel
The following commit has been merged into the x86/entry branch of tip:
Commit-ID: 951c2a51ae75382d519839e2308394ad43ce4b40
Gitweb: https://git.kernel.org/tip/951c2a51ae75382d519839e2308394ad43ce4b40
Author: Thomas Gleixner <tglx@linutronix.de>
AuthorDate: Wed, 10 Feb 2021 00:40:44 +01:00
Committer: Thomas Gleixner <tglx@linutronix.de>
CommitterDate: Wed, 10 Feb 2021 23:34:14 +01:00
x86/irq/64: Adjust the per CPU irq stack pointer by 8
The per CPU hardirq_stack_ptr contains the pointer to the irq stack in the
form that it is ready to be assigned to [ER]SP so that the first push ends
up on the top entry of the stack.
But the stack switching on 64 bit has the following rules:
1) Store the current stack pointer (RSP) in the top most stack entry
to allow the unwinder to link back to the previous stack
2) Set RSP to the top most stack entry
3) Invoke functions on the irq stack
4) Pop RSP from the top most stack entry (stored in #1) so it's back
to the original stack.
That requires all stack switching code to decrement the stored pointer by 8
in order to be able to store the current RSP and then set RSP to that
location. That's a pointless exercise.
Do the -8 adjustment right when storing the pointer and make the data type
a void pointer to avoid confusion vs. the struct irq_stack data type which
is on 64bit only used to declare the backing store. Move the definition
next to the inuse flag so they likely end up in the same cache
line. Sticking them into a struct to enforce it is a seperate change.
Signed-off-by: Thomas Gleixner <tglx@linutronix.de>
Reviewed-by: Kees Cook <keescook@chromium.org>
Link: https://lore.kernel.org/r/20210210002512.354260928@linutronix.de
---
arch/x86/include/asm/irq_stack.h | 6 +++---
arch/x86/include/asm/processor.h | 7 +++----
arch/x86/kernel/cpu/common.c | 2 +-
arch/x86/kernel/dumpstack_64.c | 22 ++++++++++++++++------
arch/x86/kernel/irq_64.c | 6 ++++--
5 files changed, 27 insertions(+), 16 deletions(-)
diff --git a/arch/x86/include/asm/irq_stack.h b/arch/x86/include/asm/irq_stack.h
index 4487bb9..554f84b 100644
--- a/arch/x86/include/asm/irq_stack.h
+++ b/arch/x86/include/asm/irq_stack.h
@@ -23,7 +23,7 @@ static __always_inline void __run_on_irqstack(void (*func)(void))
void *tos = __this_cpu_read(hardirq_stack_ptr);
__this_cpu_write(hardirq_stack_inuse, true);
- asm_call_on_stack(tos - 8, func, NULL);
+ asm_call_on_stack(tos, func, NULL);
__this_cpu_write(hardirq_stack_inuse, false);
}
@@ -34,7 +34,7 @@ __run_sysvec_on_irqstack(void (*func)(struct pt_regs *regs),
void *tos = __this_cpu_read(hardirq_stack_ptr);
__this_cpu_write(hardirq_stack_inuse, true);
- asm_call_sysvec_on_stack(tos - 8, func, regs);
+ asm_call_sysvec_on_stack(tos, func, regs);
__this_cpu_write(hardirq_stack_inuse, false);
}
@@ -45,7 +45,7 @@ __run_irq_on_irqstack(void (*func)(struct irq_desc *desc),
void *tos = __this_cpu_read(hardirq_stack_ptr);
__this_cpu_write(hardirq_stack_inuse, true);
- asm_call_irq_on_stack(tos - 8, func, desc);
+ asm_call_irq_on_stack(tos, func, desc);
__this_cpu_write(hardirq_stack_inuse, false);
}
diff --git a/arch/x86/include/asm/processor.h b/arch/x86/include/asm/processor.h
index 11d10f4..dc6d149 100644
--- a/arch/x86/include/asm/processor.h
+++ b/arch/x86/include/asm/processor.h
@@ -426,8 +426,6 @@ struct irq_stack {
char stack[IRQ_STACK_SIZE];
} __aligned(IRQ_STACK_SIZE);
-DECLARE_PER_CPU(struct irq_stack *, hardirq_stack_ptr);
-
#ifdef CONFIG_X86_32
DECLARE_PER_CPU(unsigned long, cpu_current_top_of_stack);
#else
@@ -454,6 +452,7 @@ static inline unsigned long cpu_kernelmode_gs_base(int cpu)
return (unsigned long)per_cpu(fixed_percpu_data.gs_base, cpu);
}
+DECLARE_PER_CPU(void *, hardirq_stack_ptr);
DECLARE_PER_CPU(bool, hardirq_stack_inuse);
extern asmlinkage void ignore_sysret(void);
@@ -473,9 +472,9 @@ struct stack_canary {
};
DECLARE_PER_CPU_ALIGNED(struct stack_canary, stack_canary);
#endif
-/* Per CPU softirq stack pointer */
+DECLARE_PER_CPU(struct irq_stack *, hardirq_stack_ptr);
DECLARE_PER_CPU(struct irq_stack *, softirq_stack_ptr);
-#endif /* X86_64 */
+#endif /* !X86_64 */
extern unsigned int fpu_kernel_xstate_size;
extern unsigned int fpu_user_xstate_size;
diff --git a/arch/x86/kernel/cpu/common.c b/arch/x86/kernel/cpu/common.c
index 845c8a4..c5e23f3 100644
--- a/arch/x86/kernel/cpu/common.c
+++ b/arch/x86/kernel/cpu/common.c
@@ -1739,7 +1739,7 @@ DEFINE_PER_CPU(struct task_struct *, current_task) ____cacheline_aligned =
&init_task;
EXPORT_PER_CPU_SYMBOL(current_task);
-DEFINE_PER_CPU(struct irq_stack *, hardirq_stack_ptr);
+DEFINE_PER_CPU(void *, hardirq_stack_ptr);
DEFINE_PER_CPU(bool, hardirq_stack_inuse);
DEFINE_PER_CPU(int, __preempt_count) = INIT_PREEMPT_COUNT;
diff --git a/arch/x86/kernel/dumpstack_64.c b/arch/x86/kernel/dumpstack_64.c
index 1dd8513..5601b95 100644
--- a/arch/x86/kernel/dumpstack_64.c
+++ b/arch/x86/kernel/dumpstack_64.c
@@ -128,12 +128,21 @@ static __always_inline bool in_exception_stack(unsigned long *stack, struct stac
static __always_inline bool in_irq_stack(unsigned long *stack, struct stack_info *info)
{
- unsigned long *end = (unsigned long *)this_cpu_read(hardirq_stack_ptr);
- unsigned long *begin = end - (IRQ_STACK_SIZE / sizeof(long));
+ unsigned long *end = (unsigned long *)this_cpu_read(hardirq_stack_ptr);
+ unsigned long *begin;
/*
- * This is a software stack, so 'end' can be a valid stack pointer.
- * It just means the stack is empty.
+ * @end points directly to the top most stack entry to avoid a -8
+ * adjustment in the stack switch hotpath. Adjust it back before
+ * calculating @begin.
+ */
+ end++;
+ begin = end - (IRQ_STACK_SIZE / sizeof(long));
+
+ /*
+ * Due to the switching logic RSP can never be == @end because the
+ * final operation is 'popq %rsp' which means after that RSP points
+ * to the original stack and not to @end.
*/
if (stack < begin || stack >= end)
return false;
@@ -143,8 +152,9 @@ static __always_inline bool in_irq_stack(unsigned long *stack, struct stack_info
info->end = end;
/*
- * The next stack pointer is the first thing pushed by the entry code
- * after switching to the irq stack.
+ * The next stack pointer is stored at the top of the irq stack
+ * before switching to the irq stack. Actual stack entries are all
+ * below that.
*/
info->next_sp = (unsigned long *)*(end - 1);
diff --git a/arch/x86/kernel/irq_64.c b/arch/x86/kernel/irq_64.c
index 440eed5..7103f98 100644
--- a/arch/x86/kernel/irq_64.c
+++ b/arch/x86/kernel/irq_64.c
@@ -48,7 +48,8 @@ static int map_irq_stack(unsigned int cpu)
if (!va)
return -ENOMEM;
- per_cpu(hardirq_stack_ptr, cpu) = va + IRQ_STACK_SIZE;
+ /* Store actual TOS to avoid adjustment in the hotpath */
+ per_cpu(hardirq_stack_ptr, cpu) = va + IRQ_STACK_SIZE - 8;
return 0;
}
#else
@@ -60,7 +61,8 @@ static int map_irq_stack(unsigned int cpu)
{
void *va = per_cpu_ptr(&irq_stack_backing_store, cpu);
- per_cpu(hardirq_stack_ptr, cpu) = va + IRQ_STACK_SIZE;
+ /* Store actual TOS to avoid adjustment in the hotpath */
+ per_cpu(hardirq_stack_ptr, cpu) = va + IRQ_STACK_SIZE - 8;
return 0;
}
#endif
^ permalink raw reply related [flat|nested] 31+ messages in thread
* [patch V2 04/13] x86/apic: Split out spurious handling code
2021-02-09 23:40 [patch V2 00/13] x86/irq/64: Inline irq stack switching Thomas Gleixner
` (2 preceding siblings ...)
2021-02-09 23:40 ` [patch V2 03/13] x86/irq/64: Adjust the per CPU irq stack pointer by 8 Thomas Gleixner
@ 2021-02-09 23:40 ` Thomas Gleixner
2021-02-11 0:50 ` [tip: x86/entry] " tip-bot2 for Thomas Gleixner
2021-02-09 23:40 ` [patch V2 05/13] x86/irq: Provide macro for inlining irq stack switching Thomas Gleixner
` (8 subsequent siblings)
12 siblings, 1 reply; 31+ messages in thread
From: Thomas Gleixner @ 2021-02-09 23:40 UTC (permalink / raw)
To: LKML
Cc: x86, Josh Poimboeuf, Helge Deller, David S. Miller,
Michael Ellerman, Rich Felker, Heiko Carstens, Kees Cook,
Lai Jiangshan
From: Thomas Gleixner <tglx@linutronix.de>
sysvec_spurious_apic_interrupt() calls into the handling body of
__spurious_interrupt() which is not obvious as that function is declared
inside the DEFINE_IDTENTRY_IRQ(spurious_interrupt) macro.
As __spurious_interrupt() is currently always inlined this ends up with two
copies of the same code for no reason.
Split the handling function out and invoke it from both entry points.
Signed-off-by: Thomas Gleixner <tglx@linutronix.de>
Reviewed-by: Kees Cook <keescook@chromium.org>
---
arch/x86/kernel/apic/apic.c | 31 +++++++++++++++++++------------
1 file changed, 19 insertions(+), 12 deletions(-)
--- a/arch/x86/kernel/apic/apic.c
+++ b/arch/x86/kernel/apic/apic.c
@@ -2137,18 +2137,11 @@ void __init register_lapic_address(unsig
* Local APIC interrupts
*/
-/**
- * spurious_interrupt - Catch all for interrupts raised on unused vectors
- * @regs: Pointer to pt_regs on stack
- * @vector: The vector number
- *
- * This is invoked from ASM entry code to catch all interrupts which
- * trigger on an entry which is routed to the common_spurious idtentry
- * point.
- *
- * Also called from sysvec_spurious_apic_interrupt().
+/*
+ * Common handling code for spurious_interrupt and spurious_vector entry
+ * points below. No point in allowing the compiler to inline it twice.
*/
-DEFINE_IDTENTRY_IRQ(spurious_interrupt)
+static noinline void handle_spurious_interrupt(u8 vector)
{
u32 v;
@@ -2183,9 +2176,23 @@ DEFINE_IDTENTRY_IRQ(spurious_interrupt)
trace_spurious_apic_exit(vector);
}
+/**
+ * spurious_interrupt - Catch all for interrupts raised on unused vectors
+ * @regs: Pointer to pt_regs on stack
+ * @vector: The vector number
+ *
+ * This is invoked from ASM entry code to catch all interrupts which
+ * trigger on an entry which is routed to the common_spurious idtentry
+ * point.
+ */
+DEFINE_IDTENTRY_IRQ(spurious_interrupt)
+{
+ handle_spurious_interrupt(vector);
+}
+
DEFINE_IDTENTRY_SYSVEC(sysvec_spurious_apic_interrupt)
{
- __spurious_interrupt(regs, SPURIOUS_APIC_VECTOR);
+ handle_spurious_interrupt(SPURIOUS_APIC_VECTOR);
}
/*
^ permalink raw reply [flat|nested] 31+ messages in thread* [tip: x86/entry] x86/apic: Split out spurious handling code
2021-02-09 23:40 ` [patch V2 04/13] x86/apic: Split out spurious handling code Thomas Gleixner
@ 2021-02-11 0:50 ` tip-bot2 for Thomas Gleixner
0 siblings, 0 replies; 31+ messages in thread
From: tip-bot2 for Thomas Gleixner @ 2021-02-11 0:50 UTC (permalink / raw)
To: linux-tip-commits; +Cc: Thomas Gleixner, Kees Cook, x86, linux-kernel
The following commit has been merged into the x86/entry branch of tip:
Commit-ID: 3c5e0267ec3e6ed7d3f1793273cbf0beb4f86a74
Gitweb: https://git.kernel.org/tip/3c5e0267ec3e6ed7d3f1793273cbf0beb4f86a74
Author: Thomas Gleixner <tglx@linutronix.de>
AuthorDate: Wed, 10 Feb 2021 00:40:45 +01:00
Committer: Thomas Gleixner <tglx@linutronix.de>
CommitterDate: Wed, 10 Feb 2021 23:34:14 +01:00
x86/apic: Split out spurious handling code
sysvec_spurious_apic_interrupt() calls into the handling body of
__spurious_interrupt() which is not obvious as that function is declared
inside the DEFINE_IDTENTRY_IRQ(spurious_interrupt) macro.
As __spurious_interrupt() is currently always inlined this ends up with two
copies of the same code for no reason.
Split the handling function out and invoke it from both entry points.
Signed-off-by: Thomas Gleixner <tglx@linutronix.de>
Reviewed-by: Kees Cook <keescook@chromium.org>
Link: https://lore.kernel.org/r/20210210002512.469379641@linutronix.de
---
arch/x86/kernel/apic/apic.c | 31 +++++++++++++++++++------------
1 file changed, 19 insertions(+), 12 deletions(-)
diff --git a/arch/x86/kernel/apic/apic.c b/arch/x86/kernel/apic/apic.c
index 6bd20c0..02956c0 100644
--- a/arch/x86/kernel/apic/apic.c
+++ b/arch/x86/kernel/apic/apic.c
@@ -2133,18 +2133,11 @@ void __init register_lapic_address(unsigned long address)
* Local APIC interrupts
*/
-/**
- * spurious_interrupt - Catch all for interrupts raised on unused vectors
- * @regs: Pointer to pt_regs on stack
- * @vector: The vector number
- *
- * This is invoked from ASM entry code to catch all interrupts which
- * trigger on an entry which is routed to the common_spurious idtentry
- * point.
- *
- * Also called from sysvec_spurious_apic_interrupt().
+/*
+ * Common handling code for spurious_interrupt and spurious_vector entry
+ * points below. No point in allowing the compiler to inline it twice.
*/
-DEFINE_IDTENTRY_IRQ(spurious_interrupt)
+static noinline void handle_spurious_interrupt(u8 vector)
{
u32 v;
@@ -2179,9 +2172,23 @@ out:
trace_spurious_apic_exit(vector);
}
+/**
+ * spurious_interrupt - Catch all for interrupts raised on unused vectors
+ * @regs: Pointer to pt_regs on stack
+ * @vector: The vector number
+ *
+ * This is invoked from ASM entry code to catch all interrupts which
+ * trigger on an entry which is routed to the common_spurious idtentry
+ * point.
+ */
+DEFINE_IDTENTRY_IRQ(spurious_interrupt)
+{
+ handle_spurious_interrupt(vector);
+}
+
DEFINE_IDTENTRY_SYSVEC(sysvec_spurious_apic_interrupt)
{
- __spurious_interrupt(regs, SPURIOUS_APIC_VECTOR);
+ handle_spurious_interrupt(SPURIOUS_APIC_VECTOR);
}
/*
^ permalink raw reply related [flat|nested] 31+ messages in thread
* [patch V2 05/13] x86/irq: Provide macro for inlining irq stack switching
2021-02-09 23:40 [patch V2 00/13] x86/irq/64: Inline irq stack switching Thomas Gleixner
` (3 preceding siblings ...)
2021-02-09 23:40 ` [patch V2 04/13] x86/apic: Split out spurious handling code Thomas Gleixner
@ 2021-02-09 23:40 ` Thomas Gleixner
2021-02-11 0:50 ` [tip: x86/entry] " tip-bot2 for Thomas Gleixner
2021-02-09 23:40 ` [patch V2 06/13] x86/entry: Convert system vectors to irq stack macro Thomas Gleixner
` (7 subsequent siblings)
12 siblings, 1 reply; 31+ messages in thread
From: Thomas Gleixner @ 2021-02-09 23:40 UTC (permalink / raw)
To: LKML
Cc: x86, Josh Poimboeuf, Helge Deller, David S. Miller,
Michael Ellerman, Rich Felker, Heiko Carstens, Kees Cook,
Lai Jiangshan
From: Thomas Gleixner <tglx@linutronix.de>
The effort to make the ASM entry code slim and unified moved the irq stack
switching out of the low level ASM code so that the whole return from
interrupt work and state handling can be done in C and the ASM code just
handles the low level details of entry and exit.
This ended up being a suboptimal implementation for various reasons
(including tooling). The main pain points are:
- The indirect call which is expensive thanks to retpoline
- The inability to stay on the irq stack for softirq processing on return
from interrupt
- The fact that the stack switching code ends up being an easy to target
exploit gadget.
Prepare for inlining the stack switching logic into the C entry points by
providing a ASM macro which contains the guts of the switching mechanism:
1) Store RSP at the top of the irq stack
2) Switch RSP to the irq stack
3) Invoke code
4) Pop the original RSP back
Document the unholy asm() logic while at it to reduce the amount of head
scratching required a half year from now.
Signed-off-by: Thomas Gleixner <tglx@linutronix.de>
Reviewed-by: Kees Cook <keescook@chromium.org>
---
V2: Cosmetic changes - Borislav
Have ASM_CALL_CONSTRAINT unconditional - Josh
---
arch/x86/include/asm/irq_stack.h | 98 +++++++++++++++++++++++++++++++++++++++
1 file changed, 98 insertions(+)
--- a/arch/x86/include/asm/irq_stack.h
+++ b/arch/x86/include/asm/irq_stack.h
@@ -7,6 +7,104 @@
#include <asm/processor.h>
#ifdef CONFIG_X86_64
+
+/*
+ * Macro to inline switching to an interrupt stack and invoking function
+ * calls from there. The following rules apply:
+ *
+ * - Ordering:
+ *
+ * 1. Write the stack pointer into the top most place of the irq
+ * stack. This ensures that the various unwinders can link back to the
+ * original stack.
+ *
+ * 2. Switch the stack pointer to the top of the irq stack.
+ *
+ * 3. Invoke whatever needs to be done (@asm_call argument)
+ *
+ * 4. Pop the original stack pointer from the top of the irq stack
+ * which brings it back to the original stack where it left off.
+ *
+ * - Function invocation:
+ *
+ * To allow flexible usage of the macro, the actual function code including
+ * the store of the arguments in the call ABI registers is handed in via
+ * the @asm_call argument.
+ *
+ * - Local variables:
+ *
+ * @tos:
+ * The @tos variable holds a pointer to the top of the irq stack and
+ * _must_ be allocated in a non-callee saved register as this is a
+ * restriction coming from objtool.
+ *
+ * Note, that (tos) is both in input and output constraints to ensure
+ * that the compiler does not assume that R11 is left untouched in
+ * case this macro is used in some place where the per cpu interrupt
+ * stack pointer is used again afterwards
+ *
+ * - Function arguments:
+ * The function argument(s), if any, have to be defined in register
+ * variables at the place where this is invoked. Storing the
+ * argument(s) in the proper register(s) is part of the @asm_call
+ *
+ * - Constraints:
+ *
+ * The constraints have to be done very carefully because the compiler
+ * does not know about the assembly call.
+ *
+ * output:
+ * As documented already above the @tos variable is required to be in
+ * the output constraints to make the compiler aware that R11 cannot be
+ * reused after the asm() statement.
+ *
+ * For builds with CONFIG_UNWIND_FRAME_POINTER ASM_CALL_CONSTRAINT is
+ * required as well as this prevents certain creative GCC variants from
+ * misplacing the ASM code.
+ *
+ * input:
+ * - func:
+ * Immediate, which tells the compiler that the function is referenced.
+ *
+ * - tos:
+ * Register. The actual register is defined by the variable declaration.
+ *
+ * - function arguments:
+ * The constraints are handed in via the 'argconstr' argument list. They
+ * describe the register arguments which are used in @asm_call.
+ *
+ * clobbers:
+ * Function calls can clobber anything except the callee-saved
+ * registers. Tell the compiler.
+ */
+#define call_on_irqstack(func, asm_call, argconstr...) \
+{ \
+ register void *tos asm("r11"); \
+ \
+ tos = ((void *)__this_cpu_read(hardirq_stack_ptr)); \
+ \
+ asm_inline volatile( \
+ "movq %%rsp, (%[tos]) \n" \
+ "movq %[tos], %%rsp \n" \
+ \
+ asm_call \
+ \
+ "popq %%rsp \n" \
+ \
+ : "+r" (tos), ASM_CALL_CONSTRAINT \
+ : [__func] "i" (func), [tos] "r" (tos) argconstr \
+ : "cc", "rax", "rcx", "rdx", "rsi", "rdi", "r8", "r9", "r10", \
+ "memory" \
+ ); \
+}
+
+/* Macros to assert type correctness for run_*_on_irqstack macros */
+#define assert_function_type(func, proto) \
+ static_assert(__builtin_types_compatible_p(typeof(&func), proto))
+
+#define assert_arg_type(arg, proto) \
+ static_assert(__builtin_types_compatible_p(typeof(arg), proto))
+
static __always_inline bool irqstack_active(void)
{
return __this_cpu_read(hardirq_stack_inuse);
^ permalink raw reply [flat|nested] 31+ messages in thread* [tip: x86/entry] x86/irq: Provide macro for inlining irq stack switching
2021-02-09 23:40 ` [patch V2 05/13] x86/irq: Provide macro for inlining irq stack switching Thomas Gleixner
@ 2021-02-11 0:50 ` tip-bot2 for Thomas Gleixner
0 siblings, 0 replies; 31+ messages in thread
From: tip-bot2 for Thomas Gleixner @ 2021-02-11 0:50 UTC (permalink / raw)
To: linux-tip-commits; +Cc: Thomas Gleixner, Kees Cook, x86, linux-kernel
The following commit has been merged into the x86/entry branch of tip:
Commit-ID: a0cfc74d0b00c5201e1c09e28b2dc01c8088f809
Gitweb: https://git.kernel.org/tip/a0cfc74d0b00c5201e1c09e28b2dc01c8088f809
Author: Thomas Gleixner <tglx@linutronix.de>
AuthorDate: Wed, 10 Feb 2021 00:40:46 +01:00
Committer: Thomas Gleixner <tglx@linutronix.de>
CommitterDate: Wed, 10 Feb 2021 23:34:14 +01:00
x86/irq: Provide macro for inlining irq stack switching
The effort to make the ASM entry code slim and unified moved the irq stack
switching out of the low level ASM code so that the whole return from
interrupt work and state handling can be done in C and the ASM code just
handles the low level details of entry and exit.
This ended up being a suboptimal implementation for various reasons
(including tooling). The main pain points are:
- The indirect call which is expensive thanks to retpoline
- The inability to stay on the irq stack for softirq processing on return
from interrupt
- The fact that the stack switching code ends up being an easy to target
exploit gadget.
Prepare for inlining the stack switching logic into the C entry points by
providing a ASM macro which contains the guts of the switching mechanism:
1) Store RSP at the top of the irq stack
2) Switch RSP to the irq stack
3) Invoke code
4) Pop the original RSP back
Document the unholy asm() logic while at it to reduce the amount of head
scratching required a half year from now.
Signed-off-by: Thomas Gleixner <tglx@linutronix.de>
Reviewed-by: Kees Cook <keescook@chromium.org>
Link: https://lore.kernel.org/r/20210210002512.578371068@linutronix.de
---
arch/x86/include/asm/irq_stack.h | 98 +++++++++++++++++++++++++++++++-
1 file changed, 98 insertions(+)
diff --git a/arch/x86/include/asm/irq_stack.h b/arch/x86/include/asm/irq_stack.h
index 554f84b..d5b7bc6 100644
--- a/arch/x86/include/asm/irq_stack.h
+++ b/arch/x86/include/asm/irq_stack.h
@@ -7,6 +7,104 @@
#include <asm/processor.h>
#ifdef CONFIG_X86_64
+
+/*
+ * Macro to inline switching to an interrupt stack and invoking function
+ * calls from there. The following rules apply:
+ *
+ * - Ordering:
+ *
+ * 1. Write the stack pointer into the top most place of the irq
+ * stack. This ensures that the various unwinders can link back to the
+ * original stack.
+ *
+ * 2. Switch the stack pointer to the top of the irq stack.
+ *
+ * 3. Invoke whatever needs to be done (@asm_call argument)
+ *
+ * 4. Pop the original stack pointer from the top of the irq stack
+ * which brings it back to the original stack where it left off.
+ *
+ * - Function invocation:
+ *
+ * To allow flexible usage of the macro, the actual function code including
+ * the store of the arguments in the call ABI registers is handed in via
+ * the @asm_call argument.
+ *
+ * - Local variables:
+ *
+ * @tos:
+ * The @tos variable holds a pointer to the top of the irq stack and
+ * _must_ be allocated in a non-callee saved register as this is a
+ * restriction coming from objtool.
+ *
+ * Note, that (tos) is both in input and output constraints to ensure
+ * that the compiler does not assume that R11 is left untouched in
+ * case this macro is used in some place where the per cpu interrupt
+ * stack pointer is used again afterwards
+ *
+ * - Function arguments:
+ * The function argument(s), if any, have to be defined in register
+ * variables at the place where this is invoked. Storing the
+ * argument(s) in the proper register(s) is part of the @asm_call
+ *
+ * - Constraints:
+ *
+ * The constraints have to be done very carefully because the compiler
+ * does not know about the assembly call.
+ *
+ * output:
+ * As documented already above the @tos variable is required to be in
+ * the output constraints to make the compiler aware that R11 cannot be
+ * reused after the asm() statement.
+ *
+ * For builds with CONFIG_UNWIND_FRAME_POINTER ASM_CALL_CONSTRAINT is
+ * required as well as this prevents certain creative GCC variants from
+ * misplacing the ASM code.
+ *
+ * input:
+ * - func:
+ * Immediate, which tells the compiler that the function is referenced.
+ *
+ * - tos:
+ * Register. The actual register is defined by the variable declaration.
+ *
+ * - function arguments:
+ * The constraints are handed in via the 'argconstr' argument list. They
+ * describe the register arguments which are used in @asm_call.
+ *
+ * clobbers:
+ * Function calls can clobber anything except the callee-saved
+ * registers. Tell the compiler.
+ */
+#define call_on_irqstack(func, asm_call, argconstr...) \
+{ \
+ register void *tos asm("r11"); \
+ \
+ tos = ((void *)__this_cpu_read(hardirq_stack_ptr)); \
+ \
+ asm_inline volatile( \
+ "movq %%rsp, (%[tos]) \n" \
+ "movq %[tos], %%rsp \n" \
+ \
+ asm_call \
+ \
+ "popq %%rsp \n" \
+ \
+ : "+r" (tos), ASM_CALL_CONSTRAINT \
+ : [__func] "i" (func), [tos] "r" (tos) argconstr \
+ : "cc", "rax", "rcx", "rdx", "rsi", "rdi", "r8", "r9", "r10", \
+ "memory" \
+ ); \
+}
+
+/* Macros to assert type correctness for run_*_on_irqstack macros */
+#define assert_function_type(func, proto) \
+ static_assert(__builtin_types_compatible_p(typeof(&func), proto))
+
+#define assert_arg_type(arg, proto) \
+ static_assert(__builtin_types_compatible_p(typeof(arg), proto))
+
static __always_inline bool irqstack_active(void)
{
return __this_cpu_read(hardirq_stack_inuse);
^ permalink raw reply related [flat|nested] 31+ messages in thread
* [patch V2 06/13] x86/entry: Convert system vectors to irq stack macro
2021-02-09 23:40 [patch V2 00/13] x86/irq/64: Inline irq stack switching Thomas Gleixner
` (4 preceding siblings ...)
2021-02-09 23:40 ` [patch V2 05/13] x86/irq: Provide macro for inlining irq stack switching Thomas Gleixner
@ 2021-02-09 23:40 ` Thomas Gleixner
2021-02-11 0:50 ` [tip: x86/entry] " tip-bot2 for Thomas Gleixner
2021-02-09 23:40 ` [patch V2 07/13] x86/entry: Convert device interrupts to inline stack switching Thomas Gleixner
` (6 subsequent siblings)
12 siblings, 1 reply; 31+ messages in thread
From: Thomas Gleixner @ 2021-02-09 23:40 UTC (permalink / raw)
To: LKML
Cc: x86, Josh Poimboeuf, Helge Deller, David S. Miller,
Michael Ellerman, Rich Felker, Heiko Carstens, Kees Cook,
Lai Jiangshan
From: Thomas Gleixner <tglx@linutronix.de>
To inline the stack switching and to prepare for enabling
CONFIG_HAVE_IRQ_EXIT_ON_IRQ_STACK provide a macro template for system
vectors and device interrupts and convert the system vectors over to it.
Signed-off-by: Thomas Gleixner <tglx@linutronix.de>
Reviewed-by: Kees Cook <keescook@chromium.org>
---
V2: Adopt to underscore removal
---
arch/x86/entry/entry_64.S | 1
arch/x86/include/asm/idtentry.h | 2
arch/x86/include/asm/irq_stack.h | 93 +++++++++++++++++++++++++++------------
3 files changed, 66 insertions(+), 30 deletions(-)
--- a/arch/x86/entry/entry_64.S
+++ b/arch/x86/entry/entry_64.S
@@ -762,7 +762,6 @@ SYM_CODE_END(.Lbad_gs)
* rdx: Function argument (can be NULL if none)
*/
SYM_FUNC_START(asm_call_on_stack)
-SYM_INNER_LABEL(asm_call_sysvec_on_stack, SYM_L_GLOBAL)
SYM_INNER_LABEL(asm_call_irq_on_stack, SYM_L_GLOBAL)
/*
* Save the frame pointer unconditionally. This allows the ORC
--- a/arch/x86/include/asm/idtentry.h
+++ b/arch/x86/include/asm/idtentry.h
@@ -237,10 +237,8 @@ static void __##func(struct pt_regs *reg
irqentry_state_t state = irqentry_enter(regs); \
\
instrumentation_begin(); \
- irq_enter_rcu(); \
kvm_set_cpu_l1tf_flush_l1d(); \
run_sysvec_on_irqstack_cond(__##func, regs); \
- irq_exit_rcu(); \
instrumentation_end(); \
irqentry_exit(regs, state); \
} \
--- a/arch/x86/include/asm/irq_stack.h
+++ b/arch/x86/include/asm/irq_stack.h
@@ -105,14 +105,69 @@
#define assert_arg_type(arg, proto) \
static_assert(__builtin_types_compatible_p(typeof(arg), proto))
+/*
+ * Macro to invoke system vector and device interrupt C handlers.
+ */
+#define call_on_irqstack_cond(func, regs, asm_call, constr, c_args...) \
+{ \
+ /* \
+ * User mode entry and interrupt on the irq stack do not \
+ * switch stacks. If from user mode the task stack is empty. \
+ */ \
+ if (user_mode(regs) || __this_cpu_read(hardirq_stack_inuse)) { \
+ irq_enter_rcu(); \
+ func(c_args); \
+ irq_exit_rcu(); \
+ } else { \
+ /* \
+ * Mark the irq stack inuse _before_ and unmark _after_ \
+ * switching stacks. Interrupts are disabled in both \
+ * places. Invoke the stack switch macro with the call \
+ * sequence which matches the above direct invocation. \
+ */ \
+ __this_cpu_write(hardirq_stack_inuse, true); \
+ call_on_irqstack(func, asm_call, constr); \
+ __this_cpu_write(hardirq_stack_inuse, false); \
+ } \
+}
+
+/*
+ * Function call sequence for __call_on_irqstack() for system vectors.
+ *
+ * Note that irq_enter_rcu() and irq_exit_rcu() do not use the input
+ * mechanism because these functions are global and cannot be optimized out
+ * when compiling a particular source file which uses one of these macros.
+ *
+ * The argument (regs) does not need to be pushed or stashed in a callee
+ * saved register to be safe vs. the irq_enter_rcu() call because the
+ * clobbers already prevent the compiler from storing it in a callee
+ * clobbered register. As the compiler has to preserve @regs for the final
+ * call to idtentry_exit() anyway, it's likely that it does not cause extra
+ * effort for this asm magic.
+ */
+#define ASM_CALL_SYSVEC \
+ "call irq_enter_rcu \n" \
+ "movq %[arg1], %%rdi \n" \
+ "call %P[__func] \n" \
+ "call irq_exit_rcu \n"
+
+#define SYSVEC_CONSTRAINTS , [arg1] "r" (regs)
+
+#define run_sysvec_on_irqstack_cond(func, regs) \
+{ \
+ assert_function_type(func, void (*)(struct pt_regs *)); \
+ assert_arg_type(regs, struct pt_regs *); \
+ \
+ call_on_irqstack_cond(func, regs, ASM_CALL_SYSVEC, \
+ SYSVEC_CONSTRAINTS, regs); \
+}
+
static __always_inline bool irqstack_active(void)
{
return __this_cpu_read(hardirq_stack_inuse);
}
void asm_call_on_stack(void *sp, void (*func)(void), void *arg);
-void asm_call_sysvec_on_stack(void *sp, void (*func)(struct pt_regs *regs),
- struct pt_regs *regs);
void asm_call_irq_on_stack(void *sp, void (*func)(struct irq_desc *desc),
struct irq_desc *desc);
@@ -126,17 +181,6 @@ static __always_inline void __run_on_irq
}
static __always_inline void
-__run_sysvec_on_irqstack(void (*func)(struct pt_regs *regs),
- struct pt_regs *regs)
-{
- void *tos = __this_cpu_read(hardirq_stack_ptr);
-
- __this_cpu_write(hardirq_stack_inuse, true);
- asm_call_sysvec_on_stack(tos, func, regs);
- __this_cpu_write(hardirq_stack_inuse, false);
-}
-
-static __always_inline void
__run_irq_on_irqstack(void (*func)(struct irq_desc *desc),
struct irq_desc *desc)
{
@@ -148,10 +192,17 @@ static __always_inline void
}
#else /* CONFIG_X86_64 */
+
+/* System vector handlers always run on the stack they interrupted. */
+#define run_sysvec_on_irqstack_cond(func, regs) \
+{ \
+ irq_enter_rcu(); \
+ func(regs); \
+ irq_exit_rcu(); \
+}
+
static inline bool irqstack_active(void) { return false; }
static inline void __run_on_irqstack(void (*func)(void)) { }
-static inline void __run_sysvec_on_irqstack(void (*func)(struct pt_regs *regs),
- struct pt_regs *regs) { }
static inline void __run_irq_on_irqstack(void (*func)(struct irq_desc *desc),
struct irq_desc *desc) { }
#endif /* !CONFIG_X86_64 */
@@ -178,18 +229,6 @@ static __always_inline void run_on_irqst
}
static __always_inline void
-run_sysvec_on_irqstack_cond(void (*func)(struct pt_regs *regs),
- struct pt_regs *regs)
-{
- lockdep_assert_irqs_disabled();
-
- if (irq_needs_irq_stack(regs))
- __run_sysvec_on_irqstack(func, regs);
- else
- func(regs);
-}
-
-static __always_inline void
run_irq_on_irqstack_cond(void (*func)(struct irq_desc *desc), struct irq_desc *desc,
struct pt_regs *regs)
{
^ permalink raw reply [flat|nested] 31+ messages in thread* [tip: x86/entry] x86/entry: Convert system vectors to irq stack macro
2021-02-09 23:40 ` [patch V2 06/13] x86/entry: Convert system vectors to irq stack macro Thomas Gleixner
@ 2021-02-11 0:50 ` tip-bot2 for Thomas Gleixner
0 siblings, 0 replies; 31+ messages in thread
From: tip-bot2 for Thomas Gleixner @ 2021-02-11 0:50 UTC (permalink / raw)
To: linux-tip-commits; +Cc: Thomas Gleixner, Kees Cook, x86, linux-kernel
The following commit has been merged into the x86/entry branch of tip:
Commit-ID: 569dd8b4eb7ef666b467c41b8e8e4f2820d07f67
Gitweb: https://git.kernel.org/tip/569dd8b4eb7ef666b467c41b8e8e4f2820d07f67
Author: Thomas Gleixner <tglx@linutronix.de>
AuthorDate: Wed, 10 Feb 2021 00:40:47 +01:00
Committer: Thomas Gleixner <tglx@linutronix.de>
CommitterDate: Wed, 10 Feb 2021 23:34:15 +01:00
x86/entry: Convert system vectors to irq stack macro
To inline the stack switching and to prepare for enabling
CONFIG_HAVE_IRQ_EXIT_ON_IRQ_STACK provide a macro template for system
vectors and device interrupts and convert the system vectors over to it.
Signed-off-by: Thomas Gleixner <tglx@linutronix.de>
Reviewed-by: Kees Cook <keescook@chromium.org>
Link: https://lore.kernel.org/r/20210210002512.676197354@linutronix.de
---
arch/x86/entry/entry_64.S | 1 +-
arch/x86/include/asm/idtentry.h | 2 +-
arch/x86/include/asm/irq_stack.h | 93 ++++++++++++++++++++++---------
3 files changed, 66 insertions(+), 30 deletions(-)
diff --git a/arch/x86/entry/entry_64.S b/arch/x86/entry/entry_64.S
index cad0870..68643d3 100644
--- a/arch/x86/entry/entry_64.S
+++ b/arch/x86/entry/entry_64.S
@@ -762,7 +762,6 @@ SYM_CODE_END(.Lbad_gs)
* rdx: Function argument (can be NULL if none)
*/
SYM_FUNC_START(asm_call_on_stack)
-SYM_INNER_LABEL(asm_call_sysvec_on_stack, SYM_L_GLOBAL)
SYM_INNER_LABEL(asm_call_irq_on_stack, SYM_L_GLOBAL)
/*
* Save the frame pointer unconditionally. This allows the ORC
diff --git a/arch/x86/include/asm/idtentry.h b/arch/x86/include/asm/idtentry.h
index 247a60a..712b3c8 100644
--- a/arch/x86/include/asm/idtentry.h
+++ b/arch/x86/include/asm/idtentry.h
@@ -237,10 +237,8 @@ __visible noinstr void func(struct pt_regs *regs) \
irqentry_state_t state = irqentry_enter(regs); \
\
instrumentation_begin(); \
- irq_enter_rcu(); \
kvm_set_cpu_l1tf_flush_l1d(); \
run_sysvec_on_irqstack_cond(__##func, regs); \
- irq_exit_rcu(); \
instrumentation_end(); \
irqentry_exit(regs, state); \
} \
diff --git a/arch/x86/include/asm/irq_stack.h b/arch/x86/include/asm/irq_stack.h
index d5b7bc6..05c37e7 100644
--- a/arch/x86/include/asm/irq_stack.h
+++ b/arch/x86/include/asm/irq_stack.h
@@ -105,14 +105,69 @@
#define assert_arg_type(arg, proto) \
static_assert(__builtin_types_compatible_p(typeof(arg), proto))
+/*
+ * Macro to invoke system vector and device interrupt C handlers.
+ */
+#define call_on_irqstack_cond(func, regs, asm_call, constr, c_args...) \
+{ \
+ /* \
+ * User mode entry and interrupt on the irq stack do not \
+ * switch stacks. If from user mode the task stack is empty. \
+ */ \
+ if (user_mode(regs) || __this_cpu_read(hardirq_stack_inuse)) { \
+ irq_enter_rcu(); \
+ func(c_args); \
+ irq_exit_rcu(); \
+ } else { \
+ /* \
+ * Mark the irq stack inuse _before_ and unmark _after_ \
+ * switching stacks. Interrupts are disabled in both \
+ * places. Invoke the stack switch macro with the call \
+ * sequence which matches the above direct invocation. \
+ */ \
+ __this_cpu_write(hardirq_stack_inuse, true); \
+ call_on_irqstack(func, asm_call, constr); \
+ __this_cpu_write(hardirq_stack_inuse, false); \
+ } \
+}
+
+/*
+ * Function call sequence for __call_on_irqstack() for system vectors.
+ *
+ * Note that irq_enter_rcu() and irq_exit_rcu() do not use the input
+ * mechanism because these functions are global and cannot be optimized out
+ * when compiling a particular source file which uses one of these macros.
+ *
+ * The argument (regs) does not need to be pushed or stashed in a callee
+ * saved register to be safe vs. the irq_enter_rcu() call because the
+ * clobbers already prevent the compiler from storing it in a callee
+ * clobbered register. As the compiler has to preserve @regs for the final
+ * call to idtentry_exit() anyway, it's likely that it does not cause extra
+ * effort for this asm magic.
+ */
+#define ASM_CALL_SYSVEC \
+ "call irq_enter_rcu \n" \
+ "movq %[arg1], %%rdi \n" \
+ "call %P[__func] \n" \
+ "call irq_exit_rcu \n"
+
+#define SYSVEC_CONSTRAINTS , [arg1] "r" (regs)
+
+#define run_sysvec_on_irqstack_cond(func, regs) \
+{ \
+ assert_function_type(func, void (*)(struct pt_regs *)); \
+ assert_arg_type(regs, struct pt_regs *); \
+ \
+ call_on_irqstack_cond(func, regs, ASM_CALL_SYSVEC, \
+ SYSVEC_CONSTRAINTS, regs); \
+}
+
static __always_inline bool irqstack_active(void)
{
return __this_cpu_read(hardirq_stack_inuse);
}
void asm_call_on_stack(void *sp, void (*func)(void), void *arg);
-void asm_call_sysvec_on_stack(void *sp, void (*func)(struct pt_regs *regs),
- struct pt_regs *regs);
void asm_call_irq_on_stack(void *sp, void (*func)(struct irq_desc *desc),
struct irq_desc *desc);
@@ -126,17 +181,6 @@ static __always_inline void __run_on_irqstack(void (*func)(void))
}
static __always_inline void
-__run_sysvec_on_irqstack(void (*func)(struct pt_regs *regs),
- struct pt_regs *regs)
-{
- void *tos = __this_cpu_read(hardirq_stack_ptr);
-
- __this_cpu_write(hardirq_stack_inuse, true);
- asm_call_sysvec_on_stack(tos, func, regs);
- __this_cpu_write(hardirq_stack_inuse, false);
-}
-
-static __always_inline void
__run_irq_on_irqstack(void (*func)(struct irq_desc *desc),
struct irq_desc *desc)
{
@@ -148,10 +192,17 @@ __run_irq_on_irqstack(void (*func)(struct irq_desc *desc),
}
#else /* CONFIG_X86_64 */
+
+/* System vector handlers always run on the stack they interrupted. */
+#define run_sysvec_on_irqstack_cond(func, regs) \
+{ \
+ irq_enter_rcu(); \
+ func(regs); \
+ irq_exit_rcu(); \
+}
+
static inline bool irqstack_active(void) { return false; }
static inline void __run_on_irqstack(void (*func)(void)) { }
-static inline void __run_sysvec_on_irqstack(void (*func)(struct pt_regs *regs),
- struct pt_regs *regs) { }
static inline void __run_irq_on_irqstack(void (*func)(struct irq_desc *desc),
struct irq_desc *desc) { }
#endif /* !CONFIG_X86_64 */
@@ -178,18 +229,6 @@ static __always_inline void run_on_irqstack_cond(void (*func)(void),
}
static __always_inline void
-run_sysvec_on_irqstack_cond(void (*func)(struct pt_regs *regs),
- struct pt_regs *regs)
-{
- lockdep_assert_irqs_disabled();
-
- if (irq_needs_irq_stack(regs))
- __run_sysvec_on_irqstack(func, regs);
- else
- func(regs);
-}
-
-static __always_inline void
run_irq_on_irqstack_cond(void (*func)(struct irq_desc *desc), struct irq_desc *desc,
struct pt_regs *regs)
{
^ permalink raw reply related [flat|nested] 31+ messages in thread
* [patch V2 07/13] x86/entry: Convert device interrupts to inline stack switching
2021-02-09 23:40 [patch V2 00/13] x86/irq/64: Inline irq stack switching Thomas Gleixner
` (5 preceding siblings ...)
2021-02-09 23:40 ` [patch V2 06/13] x86/entry: Convert system vectors to irq stack macro Thomas Gleixner
@ 2021-02-09 23:40 ` Thomas Gleixner
2021-02-11 0:50 ` [tip: x86/entry] " tip-bot2 for Thomas Gleixner
2021-02-09 23:40 ` [patch V2 08/13] x86/entry: Use run_sysvec_on_irqstack_cond() for XEN upcall Thomas Gleixner
` (5 subsequent siblings)
12 siblings, 1 reply; 31+ messages in thread
From: Thomas Gleixner @ 2021-02-09 23:40 UTC (permalink / raw)
To: LKML
Cc: x86, Josh Poimboeuf, Helge Deller, David S. Miller,
Michael Ellerman, Rich Felker, Heiko Carstens, Kees Cook,
Lai Jiangshan
From: Thomas Gleixner <tglx@linutronix.de>
Convert device interrupts to inline stack switching by replacing the
existing macro implementation with the new inline version. Tweak the
function signature of the actual handler function to have the vector
argument as u32. That allows the inline macro to avoid extra intermediates
and lets the compiler be smarter about the whole thing.
Signed-off-by: Thomas Gleixner <tglx@linutronix.de>
Reviewed-by: Kees Cook <keescook@chromium.org>
---
arch/x86/entry/entry_64.S | 1
arch/x86/include/asm/idtentry.h | 9 ++----
arch/x86/include/asm/irq_stack.h | 58 ++++++++++++++++++++-------------------
arch/x86/kernel/irq.c | 2 -
4 files changed, 36 insertions(+), 34 deletions(-)
--- a/arch/x86/entry/entry_64.S
+++ b/arch/x86/entry/entry_64.S
@@ -762,7 +762,6 @@ SYM_CODE_END(.Lbad_gs)
* rdx: Function argument (can be NULL if none)
*/
SYM_FUNC_START(asm_call_on_stack)
-SYM_INNER_LABEL(asm_call_irq_on_stack, SYM_L_GLOBAL)
/*
* Save the frame pointer unconditionally. This allows the ORC
* unwinder to handle the stack switch.
--- a/arch/x86/include/asm/idtentry.h
+++ b/arch/x86/include/asm/idtentry.h
@@ -187,23 +187,22 @@ static __always_inline void __##func(str
* has to be done in the function body if necessary.
*/
#define DEFINE_IDTENTRY_IRQ(func) \
-static __always_inline void __##func(struct pt_regs *regs, u8 vector); \
+static void __##func(struct pt_regs *regs, u32 vector); \
\
__visible noinstr void func(struct pt_regs *regs, \
unsigned long error_code) \
{ \
irqentry_state_t state = irqentry_enter(regs); \
+ u32 vector = (u32)(u8)error_code; \
\
instrumentation_begin(); \
- irq_enter_rcu(); \
kvm_set_cpu_l1tf_flush_l1d(); \
- __##func (regs, (u8)error_code); \
- irq_exit_rcu(); \
+ run_irq_on_irqstack_cond(__##func, regs, vector); \
instrumentation_end(); \
irqentry_exit(regs, state); \
} \
\
-static __always_inline void __##func(struct pt_regs *regs, u8 vector)
+static noinline void __##func(struct pt_regs *regs, u32 vector)
/**
* DECLARE_IDTENTRY_SYSVEC - Declare functions for system vector entry points
--- a/arch/x86/include/asm/irq_stack.h
+++ b/arch/x86/include/asm/irq_stack.h
@@ -162,14 +162,35 @@
SYSVEC_CONSTRAINTS, regs); \
}
+/*
+ * As in ASM_CALL_SYSVEC above the clobbers force the compiler to store
+ * @regs and @vector in callee saved registers.
+ */
+#define ASM_CALL_IRQ \
+ "call irq_enter_rcu \n" \
+ "movq %[arg1], %%rdi \n" \
+ "movl %[arg2], %%esi \n" \
+ "call %P[__func] \n" \
+ "call irq_exit_rcu \n"
+
+#define IRQ_CONSTRAINTS , [arg1] "r" (regs), [arg2] "r" (vector)
+
+#define run_irq_on_irqstack_cond(func, regs, vector) \
+{ \
+ assert_function_type(func, void (*)(struct pt_regs *, u32)); \
+ assert_arg_type(regs, struct pt_regs *); \
+ assert_arg_type(vector, u32); \
+ \
+ call_on_irqstack_cond(func, regs, ASM_CALL_IRQ, \
+ IRQ_CONSTRAINTS, regs, vector); \
+}
+
static __always_inline bool irqstack_active(void)
{
return __this_cpu_read(hardirq_stack_inuse);
}
void asm_call_on_stack(void *sp, void (*func)(void), void *arg);
-void asm_call_irq_on_stack(void *sp, void (*func)(struct irq_desc *desc),
- struct irq_desc *desc);
static __always_inline void __run_on_irqstack(void (*func)(void))
{
@@ -180,17 +201,6 @@ static __always_inline void __run_on_irq
__this_cpu_write(hardirq_stack_inuse, false);
}
-static __always_inline void
-__run_irq_on_irqstack(void (*func)(struct irq_desc *desc),
- struct irq_desc *desc)
-{
- void *tos = __this_cpu_read(hardirq_stack_ptr);
-
- __this_cpu_write(hardirq_stack_inuse, true);
- asm_call_irq_on_stack(tos, func, desc);
- __this_cpu_write(hardirq_stack_inuse, false);
-}
-
#else /* CONFIG_X86_64 */
/* System vector handlers always run on the stack they interrupted. */
@@ -201,10 +211,16 @@ static __always_inline void
irq_exit_rcu(); \
}
+/* Switches to the irq stack within func() */
+#define run_irq_on_irqstack_cond(func, regs, vector) \
+{ \
+ irq_enter_rcu(); \
+ func(regs, vector); \
+ irq_exit_rcu(); \
+}
+
static inline bool irqstack_active(void) { return false; }
static inline void __run_on_irqstack(void (*func)(void)) { }
-static inline void __run_irq_on_irqstack(void (*func)(struct irq_desc *desc),
- struct irq_desc *desc) { }
#endif /* !CONFIG_X86_64 */
static __always_inline bool irq_needs_irq_stack(struct pt_regs *regs)
@@ -228,16 +244,4 @@ static __always_inline void run_on_irqst
func();
}
-static __always_inline void
-run_irq_on_irqstack_cond(void (*func)(struct irq_desc *desc), struct irq_desc *desc,
- struct pt_regs *regs)
-{
- lockdep_assert_irqs_disabled();
-
- if (irq_needs_irq_stack(regs))
- __run_irq_on_irqstack(func, desc);
- else
- func(desc);
-}
-
#endif
--- a/arch/x86/kernel/irq.c
+++ b/arch/x86/kernel/irq.c
@@ -227,7 +227,7 @@ static __always_inline void handle_irq(s
struct pt_regs *regs)
{
if (IS_ENABLED(CONFIG_X86_64))
- run_irq_on_irqstack_cond(desc->handle_irq, desc, regs);
+ generic_handle_irq_desc(desc);
else
__handle_irq(desc, regs);
}
^ permalink raw reply [flat|nested] 31+ messages in thread* [tip: x86/entry] x86/entry: Convert device interrupts to inline stack switching
2021-02-09 23:40 ` [patch V2 07/13] x86/entry: Convert device interrupts to inline stack switching Thomas Gleixner
@ 2021-02-11 0:50 ` tip-bot2 for Thomas Gleixner
0 siblings, 0 replies; 31+ messages in thread
From: tip-bot2 for Thomas Gleixner @ 2021-02-11 0:50 UTC (permalink / raw)
To: linux-tip-commits; +Cc: Thomas Gleixner, Kees Cook, x86, linux-kernel
The following commit has been merged into the x86/entry branch of tip:
Commit-ID: 5b51e1db9bdc312d53087a0c97d54ea150111c0d
Gitweb: https://git.kernel.org/tip/5b51e1db9bdc312d53087a0c97d54ea150111c0d
Author: Thomas Gleixner <tglx@linutronix.de>
AuthorDate: Wed, 10 Feb 2021 00:40:48 +01:00
Committer: Thomas Gleixner <tglx@linutronix.de>
CommitterDate: Wed, 10 Feb 2021 23:34:15 +01:00
x86/entry: Convert device interrupts to inline stack switching
Convert device interrupts to inline stack switching by replacing the
existing macro implementation with the new inline version. Tweak the
function signature of the actual handler function to have the vector
argument as u32. That allows the inline macro to avoid extra intermediates
and lets the compiler be smarter about the whole thing.
Signed-off-by: Thomas Gleixner <tglx@linutronix.de>
Reviewed-by: Kees Cook <keescook@chromium.org>
Link: https://lore.kernel.org/r/20210210002512.769728139@linutronix.de
---
arch/x86/entry/entry_64.S | 1 +-
arch/x86/include/asm/idtentry.h | 9 ++---
arch/x86/include/asm/irq_stack.h | 58 ++++++++++++++++---------------
arch/x86/kernel/irq.c | 2 +-
4 files changed, 36 insertions(+), 34 deletions(-)
diff --git a/arch/x86/entry/entry_64.S b/arch/x86/entry/entry_64.S
index 68643d3..f446e90 100644
--- a/arch/x86/entry/entry_64.S
+++ b/arch/x86/entry/entry_64.S
@@ -762,7 +762,6 @@ SYM_CODE_END(.Lbad_gs)
* rdx: Function argument (can be NULL if none)
*/
SYM_FUNC_START(asm_call_on_stack)
-SYM_INNER_LABEL(asm_call_irq_on_stack, SYM_L_GLOBAL)
/*
* Save the frame pointer unconditionally. This allows the ORC
* unwinder to handle the stack switch.
diff --git a/arch/x86/include/asm/idtentry.h b/arch/x86/include/asm/idtentry.h
index 712b3c8..f294637 100644
--- a/arch/x86/include/asm/idtentry.h
+++ b/arch/x86/include/asm/idtentry.h
@@ -187,23 +187,22 @@ __visible noinstr void func(struct pt_regs *regs, unsigned long error_code)
* has to be done in the function body if necessary.
*/
#define DEFINE_IDTENTRY_IRQ(func) \
-static __always_inline void __##func(struct pt_regs *regs, u8 vector); \
+static void __##func(struct pt_regs *regs, u32 vector); \
\
__visible noinstr void func(struct pt_regs *regs, \
unsigned long error_code) \
{ \
irqentry_state_t state = irqentry_enter(regs); \
+ u32 vector = (u32)(u8)error_code; \
\
instrumentation_begin(); \
- irq_enter_rcu(); \
kvm_set_cpu_l1tf_flush_l1d(); \
- __##func (regs, (u8)error_code); \
- irq_exit_rcu(); \
+ run_irq_on_irqstack_cond(__##func, regs, vector); \
instrumentation_end(); \
irqentry_exit(regs, state); \
} \
\
-static __always_inline void __##func(struct pt_regs *regs, u8 vector)
+static noinline void __##func(struct pt_regs *regs, u32 vector)
/**
* DECLARE_IDTENTRY_SYSVEC - Declare functions for system vector entry points
diff --git a/arch/x86/include/asm/irq_stack.h b/arch/x86/include/asm/irq_stack.h
index 05c37e7..dabc0cf 100644
--- a/arch/x86/include/asm/irq_stack.h
+++ b/arch/x86/include/asm/irq_stack.h
@@ -162,14 +162,35 @@
SYSVEC_CONSTRAINTS, regs); \
}
+/*
+ * As in ASM_CALL_SYSVEC above the clobbers force the compiler to store
+ * @regs and @vector in callee saved registers.
+ */
+#define ASM_CALL_IRQ \
+ "call irq_enter_rcu \n" \
+ "movq %[arg1], %%rdi \n" \
+ "movl %[arg2], %%esi \n" \
+ "call %P[__func] \n" \
+ "call irq_exit_rcu \n"
+
+#define IRQ_CONSTRAINTS , [arg1] "r" (regs), [arg2] "r" (vector)
+
+#define run_irq_on_irqstack_cond(func, regs, vector) \
+{ \
+ assert_function_type(func, void (*)(struct pt_regs *, u32)); \
+ assert_arg_type(regs, struct pt_regs *); \
+ assert_arg_type(vector, u32); \
+ \
+ call_on_irqstack_cond(func, regs, ASM_CALL_IRQ, \
+ IRQ_CONSTRAINTS, regs, vector); \
+}
+
static __always_inline bool irqstack_active(void)
{
return __this_cpu_read(hardirq_stack_inuse);
}
void asm_call_on_stack(void *sp, void (*func)(void), void *arg);
-void asm_call_irq_on_stack(void *sp, void (*func)(struct irq_desc *desc),
- struct irq_desc *desc);
static __always_inline void __run_on_irqstack(void (*func)(void))
{
@@ -180,17 +201,6 @@ static __always_inline void __run_on_irqstack(void (*func)(void))
__this_cpu_write(hardirq_stack_inuse, false);
}
-static __always_inline void
-__run_irq_on_irqstack(void (*func)(struct irq_desc *desc),
- struct irq_desc *desc)
-{
- void *tos = __this_cpu_read(hardirq_stack_ptr);
-
- __this_cpu_write(hardirq_stack_inuse, true);
- asm_call_irq_on_stack(tos, func, desc);
- __this_cpu_write(hardirq_stack_inuse, false);
-}
-
#else /* CONFIG_X86_64 */
/* System vector handlers always run on the stack they interrupted. */
@@ -201,10 +211,16 @@ __run_irq_on_irqstack(void (*func)(struct irq_desc *desc),
irq_exit_rcu(); \
}
+/* Switches to the irq stack within func() */
+#define run_irq_on_irqstack_cond(func, regs, vector) \
+{ \
+ irq_enter_rcu(); \
+ func(regs, vector); \
+ irq_exit_rcu(); \
+}
+
static inline bool irqstack_active(void) { return false; }
static inline void __run_on_irqstack(void (*func)(void)) { }
-static inline void __run_irq_on_irqstack(void (*func)(struct irq_desc *desc),
- struct irq_desc *desc) { }
#endif /* !CONFIG_X86_64 */
static __always_inline bool irq_needs_irq_stack(struct pt_regs *regs)
@@ -228,16 +244,4 @@ static __always_inline void run_on_irqstack_cond(void (*func)(void),
func();
}
-static __always_inline void
-run_irq_on_irqstack_cond(void (*func)(struct irq_desc *desc), struct irq_desc *desc,
- struct pt_regs *regs)
-{
- lockdep_assert_irqs_disabled();
-
- if (irq_needs_irq_stack(regs))
- __run_irq_on_irqstack(func, desc);
- else
- func(desc);
-}
-
#endif
diff --git a/arch/x86/kernel/irq.c b/arch/x86/kernel/irq.c
index c5dd503..1507b98 100644
--- a/arch/x86/kernel/irq.c
+++ b/arch/x86/kernel/irq.c
@@ -227,7 +227,7 @@ static __always_inline void handle_irq(struct irq_desc *desc,
struct pt_regs *regs)
{
if (IS_ENABLED(CONFIG_X86_64))
- run_irq_on_irqstack_cond(desc->handle_irq, desc, regs);
+ generic_handle_irq_desc(desc);
else
__handle_irq(desc, regs);
}
^ permalink raw reply related [flat|nested] 31+ messages in thread
* [patch V2 08/13] x86/entry: Use run_sysvec_on_irqstack_cond() for XEN upcall
2021-02-09 23:40 [patch V2 00/13] x86/irq/64: Inline irq stack switching Thomas Gleixner
` (6 preceding siblings ...)
2021-02-09 23:40 ` [patch V2 07/13] x86/entry: Convert device interrupts to inline stack switching Thomas Gleixner
@ 2021-02-09 23:40 ` Thomas Gleixner
2021-02-11 0:50 ` [tip: x86/entry] " tip-bot2 for Thomas Gleixner
2021-02-09 23:40 ` [patch V2 09/13] x86/softirq: Remove indirection in do_softirq_own_stack() Thomas Gleixner
` (4 subsequent siblings)
12 siblings, 1 reply; 31+ messages in thread
From: Thomas Gleixner @ 2021-02-09 23:40 UTC (permalink / raw)
To: LKML
Cc: x86, Josh Poimboeuf, Helge Deller, David S. Miller,
Michael Ellerman, Rich Felker, Heiko Carstens, Kees Cook,
Lai Jiangshan
From: Thomas Gleixner <tglx@linutronix.de>
To avoid yet another macro implementation reuse the existing
run_sysvec_on_irqstack_cond() and move the set_irq_regs() handling into the
called function. Makes the code even simpler.
Signed-off-by: Thomas Gleixner <tglx@linutronix.de>
Reviewed-by: Kees Cook <keescook@chromium.org>
---
arch/x86/entry/common.c | 19 ++++++-------------
1 file changed, 6 insertions(+), 13 deletions(-)
--- a/arch/x86/entry/common.c
+++ b/arch/x86/entry/common.c
@@ -249,30 +249,23 @@ static __always_inline bool get_and_clea
static __always_inline void restore_inhcall(bool inhcall) { }
#endif
-static void __xen_pv_evtchn_do_upcall(void)
+static void __xen_pv_evtchn_do_upcall(struct pt_regs *regs)
{
- irq_enter_rcu();
+ struct pt_regs *old_regs = set_irq_regs(regs);
+
inc_irq_stat(irq_hv_callback_count);
xen_hvm_evtchn_do_upcall();
- irq_exit_rcu();
+ set_irq_regs(old_regs);
}
__visible noinstr void xen_pv_evtchn_do_upcall(struct pt_regs *regs)
{
- struct pt_regs *old_regs;
+ irqentry_state_t state = irqentry_enter(regs);
bool inhcall;
- irqentry_state_t state;
-
- state = irqentry_enter(regs);
- old_regs = set_irq_regs(regs);
- instrumentation_begin();
- run_on_irqstack_cond(__xen_pv_evtchn_do_upcall, regs);
- instrumentation_end();
-
- set_irq_regs(old_regs);
+ run_sysvec_on_irqstack_cond(__xen_pv_evtchn_do_upcall, regs);
inhcall = get_and_clear_inhcall();
if (inhcall && !WARN_ON_ONCE(state.exit_rcu)) {
^ permalink raw reply [flat|nested] 31+ messages in thread* [tip: x86/entry] x86/entry: Use run_sysvec_on_irqstack_cond() for XEN upcall
2021-02-09 23:40 ` [patch V2 08/13] x86/entry: Use run_sysvec_on_irqstack_cond() for XEN upcall Thomas Gleixner
@ 2021-02-11 0:50 ` tip-bot2 for Thomas Gleixner
0 siblings, 0 replies; 31+ messages in thread
From: tip-bot2 for Thomas Gleixner @ 2021-02-11 0:50 UTC (permalink / raw)
To: linux-tip-commits; +Cc: Thomas Gleixner, Kees Cook, x86, linux-kernel
The following commit has been merged into the x86/entry branch of tip:
Commit-ID: 359f01d1816fc1ea0161e6c30722bef1ed6b8abb
Gitweb: https://git.kernel.org/tip/359f01d1816fc1ea0161e6c30722bef1ed6b8abb
Author: Thomas Gleixner <tglx@linutronix.de>
AuthorDate: Wed, 10 Feb 2021 00:40:49 +01:00
Committer: Thomas Gleixner <tglx@linutronix.de>
CommitterDate: Wed, 10 Feb 2021 23:34:15 +01:00
x86/entry: Use run_sysvec_on_irqstack_cond() for XEN upcall
To avoid yet another macro implementation reuse the existing
run_sysvec_on_irqstack_cond() and move the set_irq_regs() handling into the
called function. Makes the code even simpler.
Signed-off-by: Thomas Gleixner <tglx@linutronix.de>
Reviewed-by: Kees Cook <keescook@chromium.org>
Link: https://lore.kernel.org/r/20210210002512.869753106@linutronix.de
---
arch/x86/entry/common.c | 19 ++++++-------------
1 file changed, 6 insertions(+), 13 deletions(-)
diff --git a/arch/x86/entry/common.c b/arch/x86/entry/common.c
index c4efffb..60a5fc4 100644
--- a/arch/x86/entry/common.c
+++ b/arch/x86/entry/common.c
@@ -245,30 +245,23 @@ static __always_inline bool get_and_clear_inhcall(void) { return false; }
static __always_inline void restore_inhcall(bool inhcall) { }
#endif
-static void __xen_pv_evtchn_do_upcall(void)
+static void __xen_pv_evtchn_do_upcall(struct pt_regs *regs)
{
- irq_enter_rcu();
+ struct pt_regs *old_regs = set_irq_regs(regs);
+
inc_irq_stat(irq_hv_callback_count);
xen_hvm_evtchn_do_upcall();
- irq_exit_rcu();
+ set_irq_regs(old_regs);
}
__visible noinstr void xen_pv_evtchn_do_upcall(struct pt_regs *regs)
{
- struct pt_regs *old_regs;
+ irqentry_state_t state = irqentry_enter(regs);
bool inhcall;
- irqentry_state_t state;
- state = irqentry_enter(regs);
- old_regs = set_irq_regs(regs);
-
- instrumentation_begin();
- run_on_irqstack_cond(__xen_pv_evtchn_do_upcall, regs);
- instrumentation_end();
-
- set_irq_regs(old_regs);
+ run_sysvec_on_irqstack_cond(__xen_pv_evtchn_do_upcall, regs);
inhcall = get_and_clear_inhcall();
if (inhcall && !WARN_ON_ONCE(state.exit_rcu)) {
^ permalink raw reply related [flat|nested] 31+ messages in thread
* [patch V2 09/13] x86/softirq: Remove indirection in do_softirq_own_stack()
2021-02-09 23:40 [patch V2 00/13] x86/irq/64: Inline irq stack switching Thomas Gleixner
` (7 preceding siblings ...)
2021-02-09 23:40 ` [patch V2 08/13] x86/entry: Use run_sysvec_on_irqstack_cond() for XEN upcall Thomas Gleixner
@ 2021-02-09 23:40 ` Thomas Gleixner
2021-02-11 0:50 ` [tip: x86/entry] " tip-bot2 for Thomas Gleixner
2021-02-09 23:40 ` [patch V2 10/13] x86: Select CONFIG_HAVE_IRQ_EXIT_ON_IRQ_STACK Thomas Gleixner
` (3 subsequent siblings)
12 siblings, 1 reply; 31+ messages in thread
From: Thomas Gleixner @ 2021-02-09 23:40 UTC (permalink / raw)
To: LKML
Cc: x86, Josh Poimboeuf, Helge Deller, David S. Miller,
Michael Ellerman, Rich Felker, Heiko Carstens, Kees Cook,
Lai Jiangshan
From: Thomas Gleixner <tglx@linutronix.de>
Use the new inline stack switching and remove the old ASM indirect call
implementation.
Signed-off-by: Thomas Gleixner <tglx@linutronix.de>
Reviewed-by: Kees Cook <keescook@chromium.org>
---
arch/x86/entry/entry_64.S | 39 -----------------------------
arch/x86/include/asm/irq_stack.h | 52 ++++++++++++---------------------------
arch/x86/kernel/irq_64.c | 2 -
3 files changed, 17 insertions(+), 76 deletions(-)
--- a/arch/x86/entry/entry_64.S
+++ b/arch/x86/entry/entry_64.S
@@ -756,45 +756,6 @@ SYM_CODE_START_LOCAL_NOALIGN(.Lbad_gs)
SYM_CODE_END(.Lbad_gs)
.previous
-/*
- * rdi: New stack pointer points to the top word of the stack
- * rsi: Function pointer
- * rdx: Function argument (can be NULL if none)
- */
-SYM_FUNC_START(asm_call_on_stack)
- /*
- * Save the frame pointer unconditionally. This allows the ORC
- * unwinder to handle the stack switch.
- */
- pushq %rbp
- mov %rsp, %rbp
-
- /*
- * The unwinder relies on the word at the top of the new stack
- * page linking back to the previous RSP.
- */
- mov %rsp, (%rdi)
- mov %rdi, %rsp
- /* Move the argument to the right place */
- mov %rdx, %rdi
-
-1:
- .pushsection .discard.instr_begin
- .long 1b - .
- .popsection
-
- CALL_NOSPEC rsi
-
-2:
- .pushsection .discard.instr_end
- .long 2b - .
- .popsection
-
- /* Restore the previous stack pointer from RBP. */
- leaveq
- ret
-SYM_FUNC_END(asm_call_on_stack)
-
#ifdef CONFIG_XEN_PV
/*
* A note on the "critical region" in our callback handler.
--- a/arch/x86/include/asm/irq_stack.h
+++ b/arch/x86/include/asm/irq_stack.h
@@ -185,20 +185,23 @@
IRQ_CONSTRAINTS, regs, vector); \
}
-static __always_inline bool irqstack_active(void)
-{
- return __this_cpu_read(hardirq_stack_inuse);
-}
-
-void asm_call_on_stack(void *sp, void (*func)(void), void *arg);
+#define ASM_CALL_SOFTIRQ \
+ "call %P[__func] \n"
-static __always_inline void __run_on_irqstack(void (*func)(void))
-{
- void *tos = __this_cpu_read(hardirq_stack_ptr);
-
- __this_cpu_write(hardirq_stack_inuse, true);
- asm_call_on_stack(tos, func, NULL);
- __this_cpu_write(hardirq_stack_inuse, false);
+/*
+ * Macro to invoke __do_softirq on the irq stack. Contrary to the above
+ * the only check which is necessary is whether the interrupt stack is
+ * in use already.
+ */
+#define run_softirq_on_irqstack_cond() \
+{ \
+ if (__this_cpu_read(hardirq_stack_inuse)) { \
+ __do_softirq(); \
+ } else { \
+ __this_cpu_write(hardirq_stack_inuse, true); \
+ call_on_irqstack(__do_softirq, ASM_CALL_SOFTIRQ); \
+ __this_cpu_write(hardirq_stack_inuse, false); \
+ } \
}
#else /* CONFIG_X86_64 */
@@ -219,29 +222,6 @@ static __always_inline void __run_on_irq
irq_exit_rcu(); \
}
-static inline bool irqstack_active(void) { return false; }
-static inline void __run_on_irqstack(void (*func)(void)) { }
#endif /* !CONFIG_X86_64 */
-static __always_inline bool irq_needs_irq_stack(struct pt_regs *regs)
-{
- if (IS_ENABLED(CONFIG_X86_32))
- return false;
- if (!regs)
- return !irqstack_active();
- return !user_mode(regs) && !irqstack_active();
-}
-
-
-static __always_inline void run_on_irqstack_cond(void (*func)(void),
- struct pt_regs *regs)
-{
- lockdep_assert_irqs_disabled();
-
- if (irq_needs_irq_stack(regs))
- __run_on_irqstack(func);
- else
- func();
-}
-
#endif
--- a/arch/x86/kernel/irq_64.c
+++ b/arch/x86/kernel/irq_64.c
@@ -76,5 +76,5 @@ int irq_init_percpu_irqstack(unsigned in
void do_softirq_own_stack(void)
{
- run_on_irqstack_cond(__do_softirq, NULL);
+ run_softirq_on_irqstack_cond();
}
^ permalink raw reply [flat|nested] 31+ messages in thread* [tip: x86/entry] x86/softirq: Remove indirection in do_softirq_own_stack()
2021-02-09 23:40 ` [patch V2 09/13] x86/softirq: Remove indirection in do_softirq_own_stack() Thomas Gleixner
@ 2021-02-11 0:50 ` tip-bot2 for Thomas Gleixner
0 siblings, 0 replies; 31+ messages in thread
From: tip-bot2 for Thomas Gleixner @ 2021-02-11 0:50 UTC (permalink / raw)
To: linux-tip-commits; +Cc: Thomas Gleixner, Kees Cook, x86, linux-kernel
The following commit has been merged into the x86/entry branch of tip:
Commit-ID: 52d743f3b71265e14560a38f4c835d07b9c6fc4c
Gitweb: https://git.kernel.org/tip/52d743f3b71265e14560a38f4c835d07b9c6fc4c
Author: Thomas Gleixner <tglx@linutronix.de>
AuthorDate: Wed, 10 Feb 2021 00:40:50 +01:00
Committer: Thomas Gleixner <tglx@linutronix.de>
CommitterDate: Wed, 10 Feb 2021 23:34:15 +01:00
x86/softirq: Remove indirection in do_softirq_own_stack()
Use the new inline stack switching and remove the old ASM indirect call
implementation.
Signed-off-by: Thomas Gleixner <tglx@linutronix.de>
Reviewed-by: Kees Cook <keescook@chromium.org>
Link: https://lore.kernel.org/r/20210210002512.972714001@linutronix.de
---
arch/x86/entry/entry_64.S | 39 +-----------------------
arch/x86/include/asm/irq_stack.h | 52 +++++++++----------------------
arch/x86/kernel/irq_64.c | 2 +-
3 files changed, 17 insertions(+), 76 deletions(-)
diff --git a/arch/x86/entry/entry_64.S b/arch/x86/entry/entry_64.S
index f446e90..bd52f67 100644
--- a/arch/x86/entry/entry_64.S
+++ b/arch/x86/entry/entry_64.S
@@ -756,45 +756,6 @@ SYM_CODE_START_LOCAL_NOALIGN(.Lbad_gs)
SYM_CODE_END(.Lbad_gs)
.previous
-/*
- * rdi: New stack pointer points to the top word of the stack
- * rsi: Function pointer
- * rdx: Function argument (can be NULL if none)
- */
-SYM_FUNC_START(asm_call_on_stack)
- /*
- * Save the frame pointer unconditionally. This allows the ORC
- * unwinder to handle the stack switch.
- */
- pushq %rbp
- mov %rsp, %rbp
-
- /*
- * The unwinder relies on the word at the top of the new stack
- * page linking back to the previous RSP.
- */
- mov %rsp, (%rdi)
- mov %rdi, %rsp
- /* Move the argument to the right place */
- mov %rdx, %rdi
-
-1:
- .pushsection .discard.instr_begin
- .long 1b - .
- .popsection
-
- CALL_NOSPEC rsi
-
-2:
- .pushsection .discard.instr_end
- .long 2b - .
- .popsection
-
- /* Restore the previous stack pointer from RBP. */
- leaveq
- ret
-SYM_FUNC_END(asm_call_on_stack)
-
#ifdef CONFIG_XEN_PV
/*
* A note on the "critical region" in our callback handler.
diff --git a/arch/x86/include/asm/irq_stack.h b/arch/x86/include/asm/irq_stack.h
index dabc0cf..fa444c2 100644
--- a/arch/x86/include/asm/irq_stack.h
+++ b/arch/x86/include/asm/irq_stack.h
@@ -185,20 +185,23 @@
IRQ_CONSTRAINTS, regs, vector); \
}
-static __always_inline bool irqstack_active(void)
-{
- return __this_cpu_read(hardirq_stack_inuse);
-}
-
-void asm_call_on_stack(void *sp, void (*func)(void), void *arg);
+#define ASM_CALL_SOFTIRQ \
+ "call %P[__func] \n"
-static __always_inline void __run_on_irqstack(void (*func)(void))
-{
- void *tos = __this_cpu_read(hardirq_stack_ptr);
-
- __this_cpu_write(hardirq_stack_inuse, true);
- asm_call_on_stack(tos, func, NULL);
- __this_cpu_write(hardirq_stack_inuse, false);
+/*
+ * Macro to invoke __do_softirq on the irq stack. Contrary to the above
+ * the only check which is necessary is whether the interrupt stack is
+ * in use already.
+ */
+#define run_softirq_on_irqstack_cond() \
+{ \
+ if (__this_cpu_read(hardirq_stack_inuse)) { \
+ __do_softirq(); \
+ } else { \
+ __this_cpu_write(hardirq_stack_inuse, true); \
+ call_on_irqstack(__do_softirq, ASM_CALL_SOFTIRQ); \
+ __this_cpu_write(hardirq_stack_inuse, false); \
+ } \
}
#else /* CONFIG_X86_64 */
@@ -219,29 +222,6 @@ static __always_inline void __run_on_irqstack(void (*func)(void))
irq_exit_rcu(); \
}
-static inline bool irqstack_active(void) { return false; }
-static inline void __run_on_irqstack(void (*func)(void)) { }
#endif /* !CONFIG_X86_64 */
-static __always_inline bool irq_needs_irq_stack(struct pt_regs *regs)
-{
- if (IS_ENABLED(CONFIG_X86_32))
- return false;
- if (!regs)
- return !irqstack_active();
- return !user_mode(regs) && !irqstack_active();
-}
-
-
-static __always_inline void run_on_irqstack_cond(void (*func)(void),
- struct pt_regs *regs)
-{
- lockdep_assert_irqs_disabled();
-
- if (irq_needs_irq_stack(regs))
- __run_on_irqstack(func);
- else
- func();
-}
-
#endif
diff --git a/arch/x86/kernel/irq_64.c b/arch/x86/kernel/irq_64.c
index 7103f98..8d9f9a1 100644
--- a/arch/x86/kernel/irq_64.c
+++ b/arch/x86/kernel/irq_64.c
@@ -76,5 +76,5 @@ int irq_init_percpu_irqstack(unsigned int cpu)
void do_softirq_own_stack(void)
{
- run_on_irqstack_cond(__do_softirq, NULL);
+ run_softirq_on_irqstack_cond();
}
^ permalink raw reply related [flat|nested] 31+ messages in thread
* [patch V2 10/13] x86: Select CONFIG_HAVE_IRQ_EXIT_ON_IRQ_STACK
2021-02-09 23:40 [patch V2 00/13] x86/irq/64: Inline irq stack switching Thomas Gleixner
` (8 preceding siblings ...)
2021-02-09 23:40 ` [patch V2 09/13] x86/softirq: Remove indirection in do_softirq_own_stack() Thomas Gleixner
@ 2021-02-09 23:40 ` Thomas Gleixner
2021-02-11 0:50 ` [tip: x86/entry] " tip-bot2 for Thomas Gleixner
2021-02-09 23:40 ` [patch V2 11/13] softirq: Move __ARCH_HAS_DO_SOFTIRQ to Kconfig Thomas Gleixner
` (2 subsequent siblings)
12 siblings, 1 reply; 31+ messages in thread
From: Thomas Gleixner @ 2021-02-09 23:40 UTC (permalink / raw)
To: LKML
Cc: x86, Josh Poimboeuf, Helge Deller, David S. Miller,
Michael Ellerman, Rich Felker, Heiko Carstens, Kees Cook,
Lai Jiangshan
From: Thomas Gleixner <tglx@linutronix.de>
Now that all invocations of irq_exit_rcu() happen on the irq stack, turn on
CONFIG_HAVE_IRQ_EXIT_ON_IRQ_STACK which causes the core code to invoke
__do_softirq() directly without going through do_softirq_own_stack().
That means do_softirq_own_stack() is only invoked from task context which
means it can't be on the irq stack. Remove the conditional from
run_softirq_on_irqstack_cond() and rename the function accordingly.
Signed-off-by: Thomas Gleixner <tglx@linutronix.de>
Reviewed-by: Kees Cook <keescook@chromium.org>
---
arch/x86/Kconfig | 1 +
arch/x86/include/asm/irq_stack.h | 19 ++++++++-----------
arch/x86/kernel/irq_64.c | 2 +-
3 files changed, 10 insertions(+), 12 deletions(-)
--- a/arch/x86/Kconfig
+++ b/arch/x86/Kconfig
@@ -187,6 +187,7 @@ config X86
select HAVE_HW_BREAKPOINT
select HAVE_IDE
select HAVE_IOREMAP_PROT
+ select HAVE_IRQ_EXIT_ON_IRQ_STACK if X86_64
select HAVE_IRQ_TIME_ACCOUNTING
select HAVE_KERNEL_BZIP2
select HAVE_KERNEL_GZIP
--- a/arch/x86/include/asm/irq_stack.h
+++ b/arch/x86/include/asm/irq_stack.h
@@ -189,19 +189,16 @@
"call %P[__func] \n"
/*
- * Macro to invoke __do_softirq on the irq stack. Contrary to the above
- * the only check which is necessary is whether the interrupt stack is
- * in use already.
+ * Macro to invoke __do_softirq on the irq stack. This is only called from
+ * task context when bottom halfs are about to be reenabled and soft
+ * interrupts are pending to be processed. The interrupt stack cannot be in
+ * use here.
*/
-#define run_softirq_on_irqstack_cond() \
+#define run_softirq_on_irqstack() \
{ \
- if (__this_cpu_read(hardirq_stack_inuse)) { \
- __do_softirq(); \
- } else { \
- __this_cpu_write(hardirq_stack_inuse, true); \
- call_on_irqstack(__do_softirq, ASM_CALL_SOFTIRQ); \
- __this_cpu_write(hardirq_stack_inuse, false); \
- } \
+ __this_cpu_write(hardirq_stack_inuse, true); \
+ call_on_irqstack(__do_softirq, ASM_CALL_SOFTIRQ); \
+ __this_cpu_write(hardirq_stack_inuse, false); \
}
#else /* CONFIG_X86_64 */
--- a/arch/x86/kernel/irq_64.c
+++ b/arch/x86/kernel/irq_64.c
@@ -76,5 +76,5 @@ int irq_init_percpu_irqstack(unsigned in
void do_softirq_own_stack(void)
{
- run_softirq_on_irqstack_cond();
+ run_softirq_on_irqstack();
}
^ permalink raw reply [flat|nested] 31+ messages in thread* [tip: x86/entry] x86: Select CONFIG_HAVE_IRQ_EXIT_ON_IRQ_STACK
2021-02-09 23:40 ` [patch V2 10/13] x86: Select CONFIG_HAVE_IRQ_EXIT_ON_IRQ_STACK Thomas Gleixner
@ 2021-02-11 0:50 ` tip-bot2 for Thomas Gleixner
0 siblings, 0 replies; 31+ messages in thread
From: tip-bot2 for Thomas Gleixner @ 2021-02-11 0:50 UTC (permalink / raw)
To: linux-tip-commits; +Cc: Thomas Gleixner, Kees Cook, x86, linux-kernel
The following commit has been merged into the x86/entry branch of tip:
Commit-ID: 624db9eabc74597f682c0651047a25b54f7260a1
Gitweb: https://git.kernel.org/tip/624db9eabc74597f682c0651047a25b54f7260a1
Author: Thomas Gleixner <tglx@linutronix.de>
AuthorDate: Wed, 10 Feb 2021 00:40:51 +01:00
Committer: Thomas Gleixner <tglx@linutronix.de>
CommitterDate: Wed, 10 Feb 2021 23:34:16 +01:00
x86: Select CONFIG_HAVE_IRQ_EXIT_ON_IRQ_STACK
Now that all invocations of irq_exit_rcu() happen on the irq stack, turn on
CONFIG_HAVE_IRQ_EXIT_ON_IRQ_STACK which causes the core code to invoke
__do_softirq() directly without going through do_softirq_own_stack().
That means do_softirq_own_stack() is only invoked from task context which
means it can't be on the irq stack. Remove the conditional from
run_softirq_on_irqstack_cond() and rename the function accordingly.
Signed-off-by: Thomas Gleixner <tglx@linutronix.de>
Reviewed-by: Kees Cook <keescook@chromium.org>
Link: https://lore.kernel.org/r/20210210002513.068033456@linutronix.de
---
arch/x86/Kconfig | 1 +
arch/x86/include/asm/irq_stack.h | 19 ++++++++-----------
arch/x86/kernel/irq_64.c | 2 +-
3 files changed, 10 insertions(+), 12 deletions(-)
diff --git a/arch/x86/Kconfig b/arch/x86/Kconfig
index 21f8511..e17ce87 100644
--- a/arch/x86/Kconfig
+++ b/arch/x86/Kconfig
@@ -187,6 +187,7 @@ config X86
select HAVE_HW_BREAKPOINT
select HAVE_IDE
select HAVE_IOREMAP_PROT
+ select HAVE_IRQ_EXIT_ON_IRQ_STACK if X86_64
select HAVE_IRQ_TIME_ACCOUNTING
select HAVE_KERNEL_BZIP2
select HAVE_KERNEL_GZIP
diff --git a/arch/x86/include/asm/irq_stack.h b/arch/x86/include/asm/irq_stack.h
index fa444c2..1b82f92 100644
--- a/arch/x86/include/asm/irq_stack.h
+++ b/arch/x86/include/asm/irq_stack.h
@@ -189,19 +189,16 @@
"call %P[__func] \n"
/*
- * Macro to invoke __do_softirq on the irq stack. Contrary to the above
- * the only check which is necessary is whether the interrupt stack is
- * in use already.
+ * Macro to invoke __do_softirq on the irq stack. This is only called from
+ * task context when bottom halfs are about to be reenabled and soft
+ * interrupts are pending to be processed. The interrupt stack cannot be in
+ * use here.
*/
-#define run_softirq_on_irqstack_cond() \
+#define run_softirq_on_irqstack() \
{ \
- if (__this_cpu_read(hardirq_stack_inuse)) { \
- __do_softirq(); \
- } else { \
- __this_cpu_write(hardirq_stack_inuse, true); \
- call_on_irqstack(__do_softirq, ASM_CALL_SOFTIRQ); \
- __this_cpu_write(hardirq_stack_inuse, false); \
- } \
+ __this_cpu_write(hardirq_stack_inuse, true); \
+ call_on_irqstack(__do_softirq, ASM_CALL_SOFTIRQ); \
+ __this_cpu_write(hardirq_stack_inuse, false); \
}
#else /* CONFIG_X86_64 */
diff --git a/arch/x86/kernel/irq_64.c b/arch/x86/kernel/irq_64.c
index 8d9f9a1..b88fdb9 100644
--- a/arch/x86/kernel/irq_64.c
+++ b/arch/x86/kernel/irq_64.c
@@ -76,5 +76,5 @@ int irq_init_percpu_irqstack(unsigned int cpu)
void do_softirq_own_stack(void)
{
- run_softirq_on_irqstack_cond();
+ run_softirq_on_irqstack();
}
^ permalink raw reply related [flat|nested] 31+ messages in thread
* [patch V2 11/13] softirq: Move __ARCH_HAS_DO_SOFTIRQ to Kconfig
2021-02-09 23:40 [patch V2 00/13] x86/irq/64: Inline irq stack switching Thomas Gleixner
` (9 preceding siblings ...)
2021-02-09 23:40 ` [patch V2 10/13] x86: Select CONFIG_HAVE_IRQ_EXIT_ON_IRQ_STACK Thomas Gleixner
@ 2021-02-09 23:40 ` Thomas Gleixner
2021-02-10 7:03 ` Kees Cook
2021-02-11 0:50 ` [tip: x86/entry] " tip-bot2 for Thomas Gleixner
2021-02-09 23:40 ` [patch V2 12/13] softirq: Move do_softirq_own_stack() to generic asm header Thomas Gleixner
2021-02-09 23:40 ` [patch V2 13/13] x86/softirq/64: Inline do_softirq_own_stack() Thomas Gleixner
12 siblings, 2 replies; 31+ messages in thread
From: Thomas Gleixner @ 2021-02-09 23:40 UTC (permalink / raw)
To: LKML
Cc: x86, Josh Poimboeuf, Helge Deller, David S. Miller,
Michael Ellerman, Rich Felker, Heiko Carstens, Lai Jiangshan,
Kees Cook
To prepare for inlining do_softirq_own_stack() replace
__ARCH_HAS_DO_SOFTIRQ with a Kconfig switch and select it in the affected
architectures.
This allows in the next step to move the function prototype and the inline
stub into a seperate asm-generic header file which is required to avoid
include recursion.
Signed-off-by: Thomas Gleixner <tglx@linutronix.de>
Cc: Lai Jiangshan <jiangshanlai+lkml@gmail.com>
---
V2: New patch to get rid of the nested #ifdeffery - Lai
---
arch/Kconfig | 6 ++++++
arch/parisc/Kconfig | 1 +
arch/parisc/include/asm/hardirq.h | 4 ----
arch/powerpc/Kconfig | 1 +
arch/powerpc/include/asm/irq.h | 2 --
arch/s390/Kconfig | 1 +
arch/s390/include/asm/hardirq.h | 1 -
arch/sh/Kconfig | 1 +
arch/sh/include/asm/irq.h | 1 -
arch/sparc/Kconfig | 1 +
arch/sparc/include/asm/irq_64.h | 1 -
arch/x86/Kconfig | 1 +
arch/x86/include/asm/irq.h | 2 --
include/linux/interrupt.h | 2 +-
14 files changed, 13 insertions(+), 12 deletions(-)
--- a/arch/Kconfig
+++ b/arch/Kconfig
@@ -759,6 +759,12 @@ config HAVE_IRQ_EXIT_ON_IRQ_STACK
This spares a stack switch and improves cache usage on softirq
processing.
+config HAVE_SOFTIRQ_ON_OWN_STACK
+ bool
+ help
+ Architecture provides a function to run __do_softirq() on a
+ seperate stack.
+
config PGTABLE_LEVELS
int
default 2
--- a/arch/parisc/Kconfig
+++ b/arch/parisc/Kconfig
@@ -63,6 +63,7 @@ config PARISC
select HAVE_FTRACE_MCOUNT_RECORD if HAVE_DYNAMIC_FTRACE
select HAVE_KPROBES_ON_FTRACE
select HAVE_DYNAMIC_FTRACE_WITH_REGS
+ select HAVE_SOFTIRQ_ON_OWN_STACK if IRQSTACKS
select SET_FS
help
--- a/arch/parisc/include/asm/hardirq.h
+++ b/arch/parisc/include/asm/hardirq.h
@@ -12,10 +12,6 @@
#include <linux/threads.h>
#include <linux/irq.h>
-#ifdef CONFIG_IRQSTACKS
-#define __ARCH_HAS_DO_SOFTIRQ
-#endif
-
typedef struct {
unsigned int __softirq_pending;
unsigned int kernel_stack_usage;
--- a/arch/powerpc/Kconfig
+++ b/arch/powerpc/Kconfig
@@ -237,6 +237,7 @@ config PPC
select MMU_GATHER_PAGE_SIZE
select HAVE_REGS_AND_STACK_ACCESS_API
select HAVE_RELIABLE_STACKTRACE if PPC_BOOK3S_64 && CPU_LITTLE_ENDIAN
+ select HAVE_SOFTIRQ_ON_OWN_STACK
select HAVE_SYSCALL_TRACEPOINTS
select HAVE_VIRT_CPU_ACCOUNTING
select HAVE_IRQ_TIME_ACCOUNTING
--- a/arch/powerpc/include/asm/irq.h
+++ b/arch/powerpc/include/asm/irq.h
@@ -37,8 +37,6 @@ extern int distribute_irqs;
struct pt_regs;
-#define __ARCH_HAS_DO_SOFTIRQ
-
#if defined(CONFIG_BOOKE) || defined(CONFIG_40x)
/*
* Per-cpu stacks for handling critical, debug and machine check
--- a/arch/s390/Kconfig
+++ b/arch/s390/Kconfig
@@ -182,6 +182,7 @@ config S390
select HAVE_REGS_AND_STACK_ACCESS_API
select HAVE_RELIABLE_STACKTRACE
select HAVE_RSEQ
+ select HAVE_SOFTIRQ_ON_OWN_STACK
select HAVE_SYSCALL_TRACEPOINTS
select HAVE_VIRT_CPU_ACCOUNTING
select HAVE_VIRT_CPU_ACCOUNTING_IDLE
--- a/arch/s390/include/asm/hardirq.h
+++ b/arch/s390/include/asm/hardirq.h
@@ -18,7 +18,6 @@
#define or_softirq_pending(x) (S390_lowcore.softirq_pending |= (x))
#define __ARCH_IRQ_STAT
-#define __ARCH_HAS_DO_SOFTIRQ
#define __ARCH_IRQ_EXIT_IRQS_DISABLED
static inline void ack_bad_irq(unsigned int irq)
--- a/arch/sh/Kconfig
+++ b/arch/sh/Kconfig
@@ -55,6 +55,7 @@ config SUPERH
select HAVE_PERF_EVENTS
select HAVE_REGS_AND_STACK_ACCESS_API
select HAVE_UID16
+ select HAVE_SOFTIRQ_ON_OWN_STACK if IRQSTACKS
select HAVE_STACKPROTECTOR
select HAVE_SYSCALL_TRACEPOINTS
select IRQ_FORCED_THREADING
--- a/arch/sh/include/asm/irq.h
+++ b/arch/sh/include/asm/irq.h
@@ -51,7 +51,6 @@ asmlinkage int do_IRQ(unsigned int irq,
#ifdef CONFIG_IRQSTACKS
extern void irq_ctx_init(int cpu);
extern void irq_ctx_exit(int cpu);
-# define __ARCH_HAS_DO_SOFTIRQ
#else
# define irq_ctx_init(cpu) do { } while (0)
# define irq_ctx_exit(cpu) do { } while (0)
--- a/arch/sparc/Kconfig
+++ b/arch/sparc/Kconfig
@@ -97,6 +97,7 @@ config SPARC64
select ARCH_HAS_PTE_SPECIAL
select PCI_DOMAINS if PCI
select ARCH_HAS_GIGANTIC_PAGE
+ select HAVE_SOFTIRQ_ON_OWN_STACK
config ARCH_PROC_KCORE_TEXT
def_bool y
--- a/arch/sparc/include/asm/irq_64.h
+++ b/arch/sparc/include/asm/irq_64.h
@@ -93,7 +93,6 @@ void arch_trigger_cpumask_backtrace(cons
extern void *hardirq_stack[NR_CPUS];
extern void *softirq_stack[NR_CPUS];
-#define __ARCH_HAS_DO_SOFTIRQ
#define NO_IRQ 0xffffffff
--- a/arch/x86/Kconfig
+++ b/arch/x86/Kconfig
@@ -221,6 +221,7 @@ config X86
select HAVE_REGS_AND_STACK_ACCESS_API
select HAVE_RELIABLE_STACKTRACE if X86_64 && (UNWINDER_FRAME_POINTER || UNWINDER_ORC) && STACK_VALIDATION
select HAVE_FUNCTION_ARG_ACCESS_API
+ select HAVE_SOFTIRQ_ON_OWN_STACK
select HAVE_STACKPROTECTOR if CC_HAS_SANE_STACKPROTECTOR
select HAVE_STACK_VALIDATION if X86_64
select HAVE_STATIC_CALL
--- a/arch/x86/include/asm/irq.h
+++ b/arch/x86/include/asm/irq.h
@@ -25,8 +25,6 @@ static inline int irq_canonicalize(int i
extern int irq_init_percpu_irqstack(unsigned int cpu);
-#define __ARCH_HAS_DO_SOFTIRQ
-
struct irq_desc;
extern void fixup_irqs(void);
--- a/include/linux/interrupt.h
+++ b/include/linux/interrupt.h
@@ -569,7 +569,7 @@ struct softirq_action
asmlinkage void do_softirq(void);
asmlinkage void __do_softirq(void);
-#ifdef __ARCH_HAS_DO_SOFTIRQ
+#ifdef CONFIG_HAVE_SOFTIRQ_ON_OWN_STACK
void do_softirq_own_stack(void);
#else
static inline void do_softirq_own_stack(void)
^ permalink raw reply [flat|nested] 31+ messages in thread* Re: [patch V2 11/13] softirq: Move __ARCH_HAS_DO_SOFTIRQ to Kconfig
2021-02-09 23:40 ` [patch V2 11/13] softirq: Move __ARCH_HAS_DO_SOFTIRQ to Kconfig Thomas Gleixner
@ 2021-02-10 7:03 ` Kees Cook
2021-02-11 0:50 ` [tip: x86/entry] " tip-bot2 for Thomas Gleixner
1 sibling, 0 replies; 31+ messages in thread
From: Kees Cook @ 2021-02-10 7:03 UTC (permalink / raw)
To: Thomas Gleixner
Cc: LKML, x86, Josh Poimboeuf, Helge Deller, David S. Miller,
Michael Ellerman, Rich Felker, Heiko Carstens, Lai Jiangshan
On Wed, Feb 10, 2021 at 12:40:52AM +0100, Thomas Gleixner wrote:
> To prepare for inlining do_softirq_own_stack() replace
> __ARCH_HAS_DO_SOFTIRQ with a Kconfig switch and select it in the affected
> architectures.
>
> This allows in the next step to move the function prototype and the inline
> stub into a seperate asm-generic header file which is required to avoid
> include recursion.
>
> Signed-off-by: Thomas Gleixner <tglx@linutronix.de>
Reviewed-by: Kees Cook <keescook@chromium.org>
--
Kees Cook
^ permalink raw reply [flat|nested] 31+ messages in thread
* [tip: x86/entry] softirq: Move __ARCH_HAS_DO_SOFTIRQ to Kconfig
2021-02-09 23:40 ` [patch V2 11/13] softirq: Move __ARCH_HAS_DO_SOFTIRQ to Kconfig Thomas Gleixner
2021-02-10 7:03 ` Kees Cook
@ 2021-02-11 0:50 ` tip-bot2 for Thomas Gleixner
1 sibling, 0 replies; 31+ messages in thread
From: tip-bot2 for Thomas Gleixner @ 2021-02-11 0:50 UTC (permalink / raw)
To: linux-tip-commits; +Cc: Thomas Gleixner, Kees Cook, x86, linux-kernel
The following commit has been merged into the x86/entry branch of tip:
Commit-ID: cd1a41ceba8a4caef4d18a3a14d6d0f8c656efe4
Gitweb: https://git.kernel.org/tip/cd1a41ceba8a4caef4d18a3a14d6d0f8c656efe4
Author: Thomas Gleixner <tglx@linutronix.de>
AuthorDate: Wed, 10 Feb 2021 00:40:52 +01:00
Committer: Thomas Gleixner <tglx@linutronix.de>
CommitterDate: Wed, 10 Feb 2021 23:34:16 +01:00
softirq: Move __ARCH_HAS_DO_SOFTIRQ to Kconfig
To prepare for inlining do_softirq_own_stack() replace
__ARCH_HAS_DO_SOFTIRQ with a Kconfig switch and select it in the affected
architectures.
This allows in the next step to move the function prototype and the inline
stub into a seperate asm-generic header file which is required to avoid
include recursion.
Signed-off-by: Thomas Gleixner <tglx@linutronix.de>
Reviewed-by: Kees Cook <keescook@chromium.org>
Link: https://lore.kernel.org/r/20210210002513.181713427@linutronix.de
---
arch/Kconfig | 6 ++++++
arch/parisc/Kconfig | 1 +
arch/parisc/include/asm/hardirq.h | 4 ----
arch/powerpc/Kconfig | 1 +
arch/powerpc/include/asm/irq.h | 2 --
arch/s390/Kconfig | 1 +
arch/s390/include/asm/hardirq.h | 1 -
arch/sh/Kconfig | 1 +
arch/sh/include/asm/irq.h | 1 -
arch/sparc/Kconfig | 1 +
arch/sparc/include/asm/irq_64.h | 1 -
arch/x86/Kconfig | 1 +
arch/x86/include/asm/irq.h | 2 --
include/linux/interrupt.h | 2 +-
14 files changed, 13 insertions(+), 12 deletions(-)
diff --git a/arch/Kconfig b/arch/Kconfig
index 24862d1..7eab9d0 100644
--- a/arch/Kconfig
+++ b/arch/Kconfig
@@ -759,6 +759,12 @@ config HAVE_IRQ_EXIT_ON_IRQ_STACK
This spares a stack switch and improves cache usage on softirq
processing.
+config HAVE_SOFTIRQ_ON_OWN_STACK
+ bool
+ help
+ Architecture provides a function to run __do_softirq() on a
+ seperate stack.
+
config PGTABLE_LEVELS
int
default 2
diff --git a/arch/parisc/Kconfig b/arch/parisc/Kconfig
index 78b1762..7cd88c0 100644
--- a/arch/parisc/Kconfig
+++ b/arch/parisc/Kconfig
@@ -63,6 +63,7 @@ config PARISC
select HAVE_FTRACE_MCOUNT_RECORD if HAVE_DYNAMIC_FTRACE
select HAVE_KPROBES_ON_FTRACE
select HAVE_DYNAMIC_FTRACE_WITH_REGS
+ select HAVE_SOFTIRQ_ON_OWN_STACK if IRQSTACKS
select SET_FS
help
diff --git a/arch/parisc/include/asm/hardirq.h b/arch/parisc/include/asm/hardirq.h
index fad29aa..1e4fbd0 100644
--- a/arch/parisc/include/asm/hardirq.h
+++ b/arch/parisc/include/asm/hardirq.h
@@ -12,10 +12,6 @@
#include <linux/threads.h>
#include <linux/irq.h>
-#ifdef CONFIG_IRQSTACKS
-#define __ARCH_HAS_DO_SOFTIRQ
-#endif
-
typedef struct {
unsigned int __softirq_pending;
unsigned int kernel_stack_usage;
diff --git a/arch/powerpc/Kconfig b/arch/powerpc/Kconfig
index 107bb43..888d1ad 100644
--- a/arch/powerpc/Kconfig
+++ b/arch/powerpc/Kconfig
@@ -237,6 +237,7 @@ config PPC
select MMU_GATHER_PAGE_SIZE
select HAVE_REGS_AND_STACK_ACCESS_API
select HAVE_RELIABLE_STACKTRACE if PPC_BOOK3S_64 && CPU_LITTLE_ENDIAN
+ select HAVE_SOFTIRQ_ON_OWN_STACK
select HAVE_SYSCALL_TRACEPOINTS
select HAVE_VIRT_CPU_ACCOUNTING
select HAVE_IRQ_TIME_ACCOUNTING
diff --git a/arch/powerpc/include/asm/irq.h b/arch/powerpc/include/asm/irq.h
index 4f983ca..f3f264e 100644
--- a/arch/powerpc/include/asm/irq.h
+++ b/arch/powerpc/include/asm/irq.h
@@ -37,8 +37,6 @@ extern int distribute_irqs;
struct pt_regs;
-#define __ARCH_HAS_DO_SOFTIRQ
-
#if defined(CONFIG_BOOKE) || defined(CONFIG_40x)
/*
* Per-cpu stacks for handling critical, debug and machine check
diff --git a/arch/s390/Kconfig b/arch/s390/Kconfig
index c72874f..e00c02a 100644
--- a/arch/s390/Kconfig
+++ b/arch/s390/Kconfig
@@ -182,6 +182,7 @@ config S390
select HAVE_REGS_AND_STACK_ACCESS_API
select HAVE_RELIABLE_STACKTRACE
select HAVE_RSEQ
+ select HAVE_SOFTIRQ_ON_OWN_STACK
select HAVE_SYSCALL_TRACEPOINTS
select HAVE_VIRT_CPU_ACCOUNTING
select HAVE_VIRT_CPU_ACCOUNTING_IDLE
diff --git a/arch/s390/include/asm/hardirq.h b/arch/s390/include/asm/hardirq.h
index dfbc3c6..58668ff 100644
--- a/arch/s390/include/asm/hardirq.h
+++ b/arch/s390/include/asm/hardirq.h
@@ -18,7 +18,6 @@
#define or_softirq_pending(x) (S390_lowcore.softirq_pending |= (x))
#define __ARCH_IRQ_STAT
-#define __ARCH_HAS_DO_SOFTIRQ
#define __ARCH_IRQ_EXIT_IRQS_DISABLED
static inline void ack_bad_irq(unsigned int irq)
diff --git a/arch/sh/Kconfig b/arch/sh/Kconfig
index 5fa5802..b49d5e0 100644
--- a/arch/sh/Kconfig
+++ b/arch/sh/Kconfig
@@ -56,6 +56,7 @@ config SUPERH
select HAVE_PERF_EVENTS
select HAVE_REGS_AND_STACK_ACCESS_API
select HAVE_UID16
+ select HAVE_SOFTIRQ_ON_OWN_STACK if IRQSTACKS
select HAVE_STACKPROTECTOR
select HAVE_SYSCALL_TRACEPOINTS
select IRQ_FORCED_THREADING
diff --git a/arch/sh/include/asm/irq.h b/arch/sh/include/asm/irq.h
index 6d44c32..839551c 100644
--- a/arch/sh/include/asm/irq.h
+++ b/arch/sh/include/asm/irq.h
@@ -51,7 +51,6 @@ asmlinkage int do_IRQ(unsigned int irq, struct pt_regs *regs);
#ifdef CONFIG_IRQSTACKS
extern void irq_ctx_init(int cpu);
extern void irq_ctx_exit(int cpu);
-# define __ARCH_HAS_DO_SOFTIRQ
#else
# define irq_ctx_init(cpu) do { } while (0)
# define irq_ctx_exit(cpu) do { } while (0)
diff --git a/arch/sparc/Kconfig b/arch/sparc/Kconfig
index c9c34dc..f0d22d5 100644
--- a/arch/sparc/Kconfig
+++ b/arch/sparc/Kconfig
@@ -97,6 +97,7 @@ config SPARC64
select ARCH_HAS_PTE_SPECIAL
select PCI_DOMAINS if PCI
select ARCH_HAS_GIGANTIC_PAGE
+ select HAVE_SOFTIRQ_ON_OWN_STACK
config ARCH_PROC_KCORE_TEXT
def_bool y
diff --git a/arch/sparc/include/asm/irq_64.h b/arch/sparc/include/asm/irq_64.h
index 4d748e9..154df2c 100644
--- a/arch/sparc/include/asm/irq_64.h
+++ b/arch/sparc/include/asm/irq_64.h
@@ -93,7 +93,6 @@ void arch_trigger_cpumask_backtrace(const struct cpumask *mask,
extern void *hardirq_stack[NR_CPUS];
extern void *softirq_stack[NR_CPUS];
-#define __ARCH_HAS_DO_SOFTIRQ
#define NO_IRQ 0xffffffff
diff --git a/arch/x86/Kconfig b/arch/x86/Kconfig
index e17ce87..019e0e1 100644
--- a/arch/x86/Kconfig
+++ b/arch/x86/Kconfig
@@ -221,6 +221,7 @@ config X86
select HAVE_REGS_AND_STACK_ACCESS_API
select HAVE_RELIABLE_STACKTRACE if X86_64 && (UNWINDER_FRAME_POINTER || UNWINDER_ORC) && STACK_VALIDATION
select HAVE_FUNCTION_ARG_ACCESS_API
+ select HAVE_SOFTIRQ_ON_OWN_STACK
select HAVE_STACKPROTECTOR if CC_HAS_SANE_STACKPROTECTOR
select HAVE_STACK_VALIDATION if X86_64
select HAVE_STATIC_CALL
diff --git a/arch/x86/include/asm/irq.h b/arch/x86/include/asm/irq.h
index 528c8a7..f70d2ca 100644
--- a/arch/x86/include/asm/irq.h
+++ b/arch/x86/include/asm/irq.h
@@ -25,8 +25,6 @@ static inline int irq_canonicalize(int irq)
extern int irq_init_percpu_irqstack(unsigned int cpu);
-#define __ARCH_HAS_DO_SOFTIRQ
-
struct irq_desc;
extern void fixup_irqs(void);
diff --git a/include/linux/interrupt.h b/include/linux/interrupt.h
index bb8ff90..f0b918f 100644
--- a/include/linux/interrupt.h
+++ b/include/linux/interrupt.h
@@ -569,7 +569,7 @@ struct softirq_action
asmlinkage void do_softirq(void);
asmlinkage void __do_softirq(void);
-#ifdef __ARCH_HAS_DO_SOFTIRQ
+#ifdef CONFIG_HAVE_SOFTIRQ_ON_OWN_STACK
void do_softirq_own_stack(void);
#else
static inline void do_softirq_own_stack(void)
^ permalink raw reply related [flat|nested] 31+ messages in thread
* [patch V2 12/13] softirq: Move do_softirq_own_stack() to generic asm header
2021-02-09 23:40 [patch V2 00/13] x86/irq/64: Inline irq stack switching Thomas Gleixner
` (10 preceding siblings ...)
2021-02-09 23:40 ` [patch V2 11/13] softirq: Move __ARCH_HAS_DO_SOFTIRQ to Kconfig Thomas Gleixner
@ 2021-02-09 23:40 ` Thomas Gleixner
2021-02-10 7:04 ` Kees Cook
2021-02-11 0:50 ` [tip: x86/entry] " tip-bot2 for Thomas Gleixner
2021-02-09 23:40 ` [patch V2 13/13] x86/softirq/64: Inline do_softirq_own_stack() Thomas Gleixner
12 siblings, 2 replies; 31+ messages in thread
From: Thomas Gleixner @ 2021-02-09 23:40 UTC (permalink / raw)
To: LKML
Cc: x86, Josh Poimboeuf, Helge Deller, David S. Miller,
Michael Ellerman, Rich Felker, Heiko Carstens, Lai Jiangshan,
Kees Cook
To avoid include recursion hell move the do_softirq_own_stack() related
content into a generic asm header and include it from all places in arch/
which need the prototype.
This allows architectures to provide an inline implementation of
do_softirq_own_stack() without introducing a lot of #ifdeffery all over the
place.
Signed-off-by: Thomas Gleixner <tglx@linutronix.de>
Cc: Lai Jiangshan <jiangshanlai+lkml@gmail.com>
---
V2: New patch to get rid of the nested #ifdeffery - Lai
---
arch/parisc/kernel/irq.c | 1 +
arch/powerpc/kernel/irq.c | 1 +
arch/s390/kernel/irq.c | 1 +
arch/sh/kernel/irq.c | 1 +
arch/sparc/kernel/irq_64.c | 1 +
arch/x86/kernel/irq_32.c | 1 +
arch/x86/kernel/irq_64.c | 1 +
include/asm-generic/Kbuild | 1 +
include/asm-generic/softirq_stack.h | 14 ++++++++++++++
include/linux/interrupt.h | 9 ---------
kernel/softirq.c | 2 ++
11 files changed, 24 insertions(+), 9 deletions(-)
--- a/arch/parisc/kernel/irq.c
+++ b/arch/parisc/kernel/irq.c
@@ -17,6 +17,7 @@
#include <linux/types.h>
#include <asm/io.h>
+#include <asm/softirq_stack.h>
#include <asm/smp.h>
#include <asm/ldcw.h>
--- a/arch/powerpc/kernel/irq.c
+++ b/arch/powerpc/kernel/irq.c
@@ -65,6 +65,7 @@
#include <asm/livepatch.h>
#include <asm/asm-prototypes.h>
#include <asm/hw_irq.h>
+#include <asm/softirq_stack.h>
#ifdef CONFIG_PPC64
#include <asm/paca.h>
--- a/arch/s390/kernel/irq.c
+++ b/arch/s390/kernel/irq.c
@@ -27,6 +27,7 @@
#include <asm/irq.h>
#include <asm/hw_irq.h>
#include <asm/stacktrace.h>
+#include <asm/softirq_stack.h>
#include "entry.h"
DEFINE_PER_CPU_SHARED_ALIGNED(struct irq_stat, irq_stat);
--- a/arch/sh/kernel/irq.c
+++ b/arch/sh/kernel/irq.c
@@ -20,6 +20,7 @@
#include <linux/uaccess.h>
#include <asm/thread_info.h>
#include <cpu/mmu_context.h>
+#include <asm/softirq_stack.h>
atomic_t irq_err_count;
--- a/arch/sparc/kernel/irq_64.c
+++ b/arch/sparc/kernel/irq_64.c
@@ -42,6 +42,7 @@
#include <asm/head.h>
#include <asm/hypervisor.h>
#include <asm/cacheflush.h>
+#include <asm/softirq_stack.h>
#include "entry.h"
#include "cpumap.h"
--- a/arch/x86/kernel/irq_32.c
+++ b/arch/x86/kernel/irq_32.c
@@ -22,6 +22,7 @@
#include <asm/apic.h>
#include <asm/nospec-branch.h>
+#include <asm/softirq_stack.h>
#ifdef CONFIG_DEBUG_STACKOVERFLOW
--- a/arch/x86/kernel/irq_64.c
+++ b/arch/x86/kernel/irq_64.c
@@ -20,6 +20,7 @@
#include <linux/sched/task_stack.h>
#include <asm/cpu_entry_area.h>
+#include <asm/softirq_stack.h>
#include <asm/irq_stack.h>
#include <asm/io_apic.h>
#include <asm/apic.h>
--- a/include/asm-generic/Kbuild
+++ b/include/asm-generic/Kbuild
@@ -51,6 +51,7 @@ mandatory-y += sections.h
mandatory-y += serial.h
mandatory-y += shmparam.h
mandatory-y += simd.h
+mandatory-y += softirq_stack.h
mandatory-y += switch_to.h
mandatory-y += timex.h
mandatory-y += tlbflush.h
--- /dev/null
+++ b/include/asm-generic/softirq_stack.h
@@ -0,0 +1,14 @@
+/* SPDX-License-Identifier: GPL-2.0-or-later */
+#ifndef __ASM_GENERIC_SOFTIRQ_STACK_H
+#define __ASM_GENERIC_SOFTIRQ_STACK_H
+
+#ifdef CONFIG_HAVE_SOFTIRQ_ON_OWN_STACK
+void do_softirq_own_stack(void);
+#else
+static inline void do_softirq_own_stack(void)
+{
+ __do_softirq();
+}
+#endif
+
+#endif
--- a/include/linux/interrupt.h
+++ b/include/linux/interrupt.h
@@ -569,15 +569,6 @@ struct softirq_action
asmlinkage void do_softirq(void);
asmlinkage void __do_softirq(void);
-#ifdef CONFIG_HAVE_SOFTIRQ_ON_OWN_STACK
-void do_softirq_own_stack(void);
-#else
-static inline void do_softirq_own_stack(void)
-{
- __do_softirq();
-}
-#endif
-
extern void open_softirq(int nr, void (*action)(struct softirq_action *));
extern void softirq_init(void);
extern void __raise_softirq_irqoff(unsigned int nr);
--- a/kernel/softirq.c
+++ b/kernel/softirq.c
@@ -26,6 +26,8 @@
#include <linux/tick.h>
#include <linux/irq.h>
+#include <asm/softirq_stack.h>
+
#define CREATE_TRACE_POINTS
#include <trace/events/irq.h>
^ permalink raw reply [flat|nested] 31+ messages in thread* Re: [patch V2 12/13] softirq: Move do_softirq_own_stack() to generic asm header
2021-02-09 23:40 ` [patch V2 12/13] softirq: Move do_softirq_own_stack() to generic asm header Thomas Gleixner
@ 2021-02-10 7:04 ` Kees Cook
2021-02-11 0:50 ` [tip: x86/entry] " tip-bot2 for Thomas Gleixner
1 sibling, 0 replies; 31+ messages in thread
From: Kees Cook @ 2021-02-10 7:04 UTC (permalink / raw)
To: Thomas Gleixner
Cc: LKML, x86, Josh Poimboeuf, Helge Deller, David S. Miller,
Michael Ellerman, Rich Felker, Heiko Carstens, Lai Jiangshan
On Wed, Feb 10, 2021 at 12:40:53AM +0100, Thomas Gleixner wrote:
> To avoid include recursion hell move the do_softirq_own_stack() related
> content into a generic asm header and include it from all places in arch/
> which need the prototype.
>
> This allows architectures to provide an inline implementation of
> do_softirq_own_stack() without introducing a lot of #ifdeffery all over the
> place.
>
> Signed-off-by: Thomas Gleixner <tglx@linutronix.de>
Reviewed-by: Kees Cook <keescook@chromium.org>
--
Kees Cook
^ permalink raw reply [flat|nested] 31+ messages in thread
* [tip: x86/entry] softirq: Move do_softirq_own_stack() to generic asm header
2021-02-09 23:40 ` [patch V2 12/13] softirq: Move do_softirq_own_stack() to generic asm header Thomas Gleixner
2021-02-10 7:04 ` Kees Cook
@ 2021-02-11 0:50 ` tip-bot2 for Thomas Gleixner
1 sibling, 0 replies; 31+ messages in thread
From: tip-bot2 for Thomas Gleixner @ 2021-02-11 0:50 UTC (permalink / raw)
To: linux-tip-commits; +Cc: Thomas Gleixner, Kees Cook, x86, linux-kernel
The following commit has been merged into the x86/entry branch of tip:
Commit-ID: db1cc7aede37eb9235759131ddfefd9c0ea5136f
Gitweb: https://git.kernel.org/tip/db1cc7aede37eb9235759131ddfefd9c0ea5136f
Author: Thomas Gleixner <tglx@linutronix.de>
AuthorDate: Wed, 10 Feb 2021 00:40:53 +01:00
Committer: Thomas Gleixner <tglx@linutronix.de>
CommitterDate: Wed, 10 Feb 2021 23:34:16 +01:00
softirq: Move do_softirq_own_stack() to generic asm header
To avoid include recursion hell move the do_softirq_own_stack() related
content into a generic asm header and include it from all places in arch/
which need the prototype.
This allows architectures to provide an inline implementation of
do_softirq_own_stack() without introducing a lot of #ifdeffery all over the
place.
Signed-off-by: Thomas Gleixner <tglx@linutronix.de>
Reviewed-by: Kees Cook <keescook@chromium.org>
Link: https://lore.kernel.org/r/20210210002513.289960691@linutronix.de
---
arch/parisc/kernel/irq.c | 1 +
arch/powerpc/kernel/irq.c | 1 +
arch/s390/kernel/irq.c | 1 +
arch/sh/kernel/irq.c | 1 +
arch/sparc/kernel/irq_64.c | 1 +
arch/x86/kernel/irq_32.c | 1 +
arch/x86/kernel/irq_64.c | 1 +
include/asm-generic/Kbuild | 1 +
include/asm-generic/softirq_stack.h | 14 ++++++++++++++
include/linux/interrupt.h | 9 ---------
kernel/softirq.c | 2 ++
11 files changed, 24 insertions(+), 9 deletions(-)
create mode 100644 include/asm-generic/softirq_stack.h
diff --git a/arch/parisc/kernel/irq.c b/arch/parisc/kernel/irq.c
index 49cd6d2..1632d52 100644
--- a/arch/parisc/kernel/irq.c
+++ b/arch/parisc/kernel/irq.c
@@ -17,6 +17,7 @@
#include <linux/types.h>
#include <asm/io.h>
+#include <asm/softirq_stack.h>
#include <asm/smp.h>
#include <asm/ldcw.h>
diff --git a/arch/powerpc/kernel/irq.c b/arch/powerpc/kernel/irq.c
index 6b1eca5..96296d3 100644
--- a/arch/powerpc/kernel/irq.c
+++ b/arch/powerpc/kernel/irq.c
@@ -65,6 +65,7 @@
#include <asm/livepatch.h>
#include <asm/asm-prototypes.h>
#include <asm/hw_irq.h>
+#include <asm/softirq_stack.h>
#ifdef CONFIG_PPC64
#include <asm/paca.h>
diff --git a/arch/s390/kernel/irq.c b/arch/s390/kernel/irq.c
index f8a8b94..a1a2f75 100644
--- a/arch/s390/kernel/irq.c
+++ b/arch/s390/kernel/irq.c
@@ -27,6 +27,7 @@
#include <asm/irq.h>
#include <asm/hw_irq.h>
#include <asm/stacktrace.h>
+#include <asm/softirq_stack.h>
#include "entry.h"
DEFINE_PER_CPU_SHARED_ALIGNED(struct irq_stat, irq_stat);
diff --git a/arch/sh/kernel/irq.c b/arch/sh/kernel/irq.c
index ab5f790..ef0f082 100644
--- a/arch/sh/kernel/irq.c
+++ b/arch/sh/kernel/irq.c
@@ -20,6 +20,7 @@
#include <linux/uaccess.h>
#include <asm/thread_info.h>
#include <cpu/mmu_context.h>
+#include <asm/softirq_stack.h>
atomic_t irq_err_count;
diff --git a/arch/sparc/kernel/irq_64.c b/arch/sparc/kernel/irq_64.c
index 3ec9f14..c8848bb 100644
--- a/arch/sparc/kernel/irq_64.c
+++ b/arch/sparc/kernel/irq_64.c
@@ -42,6 +42,7 @@
#include <asm/head.h>
#include <asm/hypervisor.h>
#include <asm/cacheflush.h>
+#include <asm/softirq_stack.h>
#include "entry.h"
#include "cpumap.h"
diff --git a/arch/x86/kernel/irq_32.c b/arch/x86/kernel/irq_32.c
index 0b79efc..044902d 100644
--- a/arch/x86/kernel/irq_32.c
+++ b/arch/x86/kernel/irq_32.c
@@ -22,6 +22,7 @@
#include <asm/apic.h>
#include <asm/nospec-branch.h>
+#include <asm/softirq_stack.h>
#ifdef CONFIG_DEBUG_STACKOVERFLOW
diff --git a/arch/x86/kernel/irq_64.c b/arch/x86/kernel/irq_64.c
index b88fdb9..f335c39 100644
--- a/arch/x86/kernel/irq_64.c
+++ b/arch/x86/kernel/irq_64.c
@@ -20,6 +20,7 @@
#include <linux/sched/task_stack.h>
#include <asm/cpu_entry_area.h>
+#include <asm/softirq_stack.h>
#include <asm/irq_stack.h>
#include <asm/io_apic.h>
#include <asm/apic.h>
diff --git a/include/asm-generic/Kbuild b/include/asm-generic/Kbuild
index 267f6df..bd68418 100644
--- a/include/asm-generic/Kbuild
+++ b/include/asm-generic/Kbuild
@@ -51,6 +51,7 @@ mandatory-y += sections.h
mandatory-y += serial.h
mandatory-y += shmparam.h
mandatory-y += simd.h
+mandatory-y += softirq_stack.h
mandatory-y += switch_to.h
mandatory-y += timex.h
mandatory-y += tlbflush.h
diff --git a/include/asm-generic/softirq_stack.h b/include/asm-generic/softirq_stack.h
new file mode 100644
index 0000000..eceeecf
--- /dev/null
+++ b/include/asm-generic/softirq_stack.h
@@ -0,0 +1,14 @@
+/* SPDX-License-Identifier: GPL-2.0-or-later */
+#ifndef __ASM_GENERIC_SOFTIRQ_STACK_H
+#define __ASM_GENERIC_SOFTIRQ_STACK_H
+
+#ifdef CONFIG_HAVE_SOFTIRQ_ON_OWN_STACK
+void do_softirq_own_stack(void);
+#else
+static inline void do_softirq_own_stack(void)
+{
+ __do_softirq();
+}
+#endif
+
+#endif
diff --git a/include/linux/interrupt.h b/include/linux/interrupt.h
index f0b918f..967e257 100644
--- a/include/linux/interrupt.h
+++ b/include/linux/interrupt.h
@@ -569,15 +569,6 @@ struct softirq_action
asmlinkage void do_softirq(void);
asmlinkage void __do_softirq(void);
-#ifdef CONFIG_HAVE_SOFTIRQ_ON_OWN_STACK
-void do_softirq_own_stack(void);
-#else
-static inline void do_softirq_own_stack(void)
-{
- __do_softirq();
-}
-#endif
-
extern void open_softirq(int nr, void (*action)(struct softirq_action *));
extern void softirq_init(void);
extern void __raise_softirq_irqoff(unsigned int nr);
diff --git a/kernel/softirq.c b/kernel/softirq.c
index 9d71046..9908ec4 100644
--- a/kernel/softirq.c
+++ b/kernel/softirq.c
@@ -26,6 +26,8 @@
#include <linux/tick.h>
#include <linux/irq.h>
+#include <asm/softirq_stack.h>
+
#define CREATE_TRACE_POINTS
#include <trace/events/irq.h>
^ permalink raw reply related [flat|nested] 31+ messages in thread
* [patch V2 13/13] x86/softirq/64: Inline do_softirq_own_stack()
2021-02-09 23:40 [patch V2 00/13] x86/irq/64: Inline irq stack switching Thomas Gleixner
` (11 preceding siblings ...)
2021-02-09 23:40 ` [patch V2 12/13] softirq: Move do_softirq_own_stack() to generic asm header Thomas Gleixner
@ 2021-02-09 23:40 ` Thomas Gleixner
2021-02-11 0:50 ` [tip: x86/entry] " tip-bot2 for Thomas Gleixner
2021-02-15 16:39 ` [patch V2 13/13] " Guenter Roeck
12 siblings, 2 replies; 31+ messages in thread
From: Thomas Gleixner @ 2021-02-09 23:40 UTC (permalink / raw)
To: LKML
Cc: x86, Josh Poimboeuf, Helge Deller, David S. Miller,
Michael Ellerman, Rich Felker, Heiko Carstens, Kees Cook,
Lai Jiangshan
From: Thomas Gleixner <tglx@linutronix.de>
There is no reason to have this as a seperate function for a single caller.
Signed-off-by: Thomas Gleixner <tglx@linutronix.de>
Reviewed-by: Kees Cook <keescook@chromium.org>
---
V2: Adopt to the new header file.
---
arch/x86/include/asm/irq_stack.h | 3 +--
arch/x86/include/asm/softirq_stack.h | 11 +++++++++++
arch/x86/kernel/irq_64.c | 5 -----
3 files changed, 12 insertions(+), 7 deletions(-)
--- a/arch/x86/include/asm/irq_stack.h
+++ b/arch/x86/include/asm/irq_stack.h
@@ -194,7 +194,7 @@
* interrupts are pending to be processed. The interrupt stack cannot be in
* use here.
*/
-#define run_softirq_on_irqstack() \
+#define do_softirq_own_stack() \
{ \
__this_cpu_write(hardirq_stack_inuse, true); \
call_on_irqstack(__do_softirq, ASM_CALL_SOFTIRQ); \
@@ -202,7 +202,6 @@
}
#else /* CONFIG_X86_64 */
-
/* System vector handlers always run on the stack they interrupted. */
#define run_sysvec_on_irqstack_cond(func, regs) \
{ \
--- /dev/null
+++ b/arch/x86/include/asm/softirq_stack.h
@@ -0,0 +1,11 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+#ifndef _ASM_X86_SOFTIRQ_STACK_H
+#define _ASM_X86_SOFTIRQ_STACK_H
+
+#ifdef CONFIG_X86_64
+# include <asm/irq_stack.h>
+#else
+# include <asm-generic/softirq_stack.h>
+#endif
+
+#endif
--- a/arch/x86/kernel/irq_64.c
+++ b/arch/x86/kernel/irq_64.c
@@ -74,8 +74,3 @@ int irq_init_percpu_irqstack(unsigned in
return 0;
return map_irq_stack(cpu);
}
-
-void do_softirq_own_stack(void)
-{
- run_softirq_on_irqstack();
-}
^ permalink raw reply [flat|nested] 31+ messages in thread* [tip: x86/entry] x86/softirq/64: Inline do_softirq_own_stack()
2021-02-09 23:40 ` [patch V2 13/13] x86/softirq/64: Inline do_softirq_own_stack() Thomas Gleixner
@ 2021-02-11 0:50 ` tip-bot2 for Thomas Gleixner
2021-02-15 16:39 ` [patch V2 13/13] " Guenter Roeck
1 sibling, 0 replies; 31+ messages in thread
From: tip-bot2 for Thomas Gleixner @ 2021-02-11 0:50 UTC (permalink / raw)
To: linux-tip-commits; +Cc: Thomas Gleixner, Kees Cook, x86, linux-kernel
The following commit has been merged into the x86/entry branch of tip:
Commit-ID: 72f40a2823d6e16229ab58b898c6f22044e5222f
Gitweb: https://git.kernel.org/tip/72f40a2823d6e16229ab58b898c6f22044e5222f
Author: Thomas Gleixner <tglx@linutronix.de>
AuthorDate: Wed, 10 Feb 2021 00:40:54 +01:00
Committer: Thomas Gleixner <tglx@linutronix.de>
CommitterDate: Wed, 10 Feb 2021 23:34:17 +01:00
x86/softirq/64: Inline do_softirq_own_stack()
There is no reason to have this as a seperate function for a single caller.
Signed-off-by: Thomas Gleixner <tglx@linutronix.de>
Reviewed-by: Kees Cook <keescook@chromium.org>
Link: https://lore.kernel.org/r/20210210002513.382806685@linutronix.de
---
arch/x86/include/asm/irq_stack.h | 3 +--
arch/x86/include/asm/softirq_stack.h | 11 +++++++++++
arch/x86/kernel/irq_64.c | 5 -----
3 files changed, 12 insertions(+), 7 deletions(-)
create mode 100644 arch/x86/include/asm/softirq_stack.h
diff --git a/arch/x86/include/asm/irq_stack.h b/arch/x86/include/asm/irq_stack.h
index 1b82f92..9b2a0ff 100644
--- a/arch/x86/include/asm/irq_stack.h
+++ b/arch/x86/include/asm/irq_stack.h
@@ -194,7 +194,7 @@
* interrupts are pending to be processed. The interrupt stack cannot be in
* use here.
*/
-#define run_softirq_on_irqstack() \
+#define do_softirq_own_stack() \
{ \
__this_cpu_write(hardirq_stack_inuse, true); \
call_on_irqstack(__do_softirq, ASM_CALL_SOFTIRQ); \
@@ -202,7 +202,6 @@
}
#else /* CONFIG_X86_64 */
-
/* System vector handlers always run on the stack they interrupted. */
#define run_sysvec_on_irqstack_cond(func, regs) \
{ \
diff --git a/arch/x86/include/asm/softirq_stack.h b/arch/x86/include/asm/softirq_stack.h
new file mode 100644
index 0000000..889d53d
--- /dev/null
+++ b/arch/x86/include/asm/softirq_stack.h
@@ -0,0 +1,11 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+#ifndef _ASM_X86_SOFTIRQ_STACK_H
+#define _ASM_X86_SOFTIRQ_STACK_H
+
+#ifdef CONFIG_X86_64
+# include <asm/irq_stack.h>
+#else
+# include <asm-generic/softirq_stack.h>
+#endif
+
+#endif
diff --git a/arch/x86/kernel/irq_64.c b/arch/x86/kernel/irq_64.c
index f335c39..1c0fb96 100644
--- a/arch/x86/kernel/irq_64.c
+++ b/arch/x86/kernel/irq_64.c
@@ -74,8 +74,3 @@ int irq_init_percpu_irqstack(unsigned int cpu)
return 0;
return map_irq_stack(cpu);
}
-
-void do_softirq_own_stack(void)
-{
- run_softirq_on_irqstack();
-}
^ permalink raw reply related [flat|nested] 31+ messages in thread* Re: [patch V2 13/13] x86/softirq/64: Inline do_softirq_own_stack()
2021-02-09 23:40 ` [patch V2 13/13] x86/softirq/64: Inline do_softirq_own_stack() Thomas Gleixner
2021-02-11 0:50 ` [tip: x86/entry] " tip-bot2 for Thomas Gleixner
@ 2021-02-15 16:39 ` Guenter Roeck
1 sibling, 0 replies; 31+ messages in thread
From: Guenter Roeck @ 2021-02-15 16:39 UTC (permalink / raw)
To: Thomas Gleixner
Cc: LKML, x86, Josh Poimboeuf, Helge Deller, David S. Miller,
Michael Ellerman, Rich Felker, Heiko Carstens, Kees Cook,
Lai Jiangshan
On Wed, Feb 10, 2021 at 12:40:54AM +0100, Thomas Gleixner wrote:
> From: Thomas Gleixner <tglx@linutronix.de>
>
> There is no reason to have this as a seperate function for a single caller.
>
> Signed-off-by: Thomas Gleixner <tglx@linutronix.de>
> Reviewed-by: Kees Cook <keescook@chromium.org>
When building ARCH=um SUBARCH="x86_64" defconfig:
kernel/softirq.c: In function 'invoke_softirq':
arch/x86/include/asm/irq_stack.h:199:19: error: 'hardirq_stack_inuse' undeclared
./arch/x86/include/asm/irq_stack.h:84:33: error: ‘hardirq_stack_ptr’ undeclared (first use in this function)
Guenter
---
bisect log:
# bad: [52a0bcb60e40f30211cb5cbbb0f582ec4e91d896] Add linux-next specific files for 20210215
# good: [92bf22614b21a2706f4993b278017e437f7785b3] Linux 5.11-rc7
git bisect start 'HEAD' 'v5.11-rc7'
# good: [9f44811a4ca6db01e73017e565381549b3c3fc60] next-20210212/mtd
git bisect good 9f44811a4ca6db01e73017e565381549b3c3fc60
# good: [88a516b8c681d666e252abfa5e346606d94653a3] Merge remote-tracking branch 'devicetree/for-next'
git bisect good 88a516b8c681d666e252abfa5e346606d94653a3
# bad: [1c623c776046c4b9d47f70f07928980815578cc1] Merge remote-tracking branch 'char-misc/char-misc-next'
git bisect bad 1c623c776046c4b9d47f70f07928980815578cc1
# bad: [f70ed845051885a4887dd76cb7c616b8f4a2942c] Merge remote-tracking branch 'drivers-x86/for-next'
git bisect bad f70ed845051885a4887dd76cb7c616b8f4a2942c
# bad: [f1b61f7b4fb971f281978fb905507e9ac9b2d973] Merge branch 'core/mm'
git bisect bad f1b61f7b4fb971f281978fb905507e9ac9b2d973
# bad: [364af1939d1ec3512b69bcdc035063a60f9c0b3e] Merge branch 'ras/core'
git bisect bad 364af1939d1ec3512b69bcdc035063a60f9c0b3e
# bad: [9db733e3c78c9c00fa93860078674f23d6dd8b7d] Merge branch 'x86/entry'
git bisect bad 9db733e3c78c9c00fa93860078674f23d6dd8b7d
# bad: [a3251c1a36f595046bea03935ebe37a1e1f1f1d7] Merge branch 'x86/paravirt' into x86/entry
git bisect bad a3251c1a36f595046bea03935ebe37a1e1f1f1d7
# good: [f83d1a0190bf6ac788a848454d21daee2299d085] x86/acpi: Annotate indirect branch as safe
git bisect good f83d1a0190bf6ac788a848454d21daee2299d085
# good: [a0cfc74d0b00c5201e1c09e28b2dc01c8088f809] x86/irq: Provide macro for inlining irq stack switching
git bisect good a0cfc74d0b00c5201e1c09e28b2dc01c8088f809
# good: [db1cc7aede37eb9235759131ddfefd9c0ea5136f] softirq: Move do_softirq_own_stack() to generic asm header
git bisect good db1cc7aede37eb9235759131ddfefd9c0ea5136f
# good: [53c9d9240944088274aadbbbafc6138ca462db4f] x86/pv: Switch SWAPGS to ALTERNATIVE
git bisect good 53c9d9240944088274aadbbbafc6138ca462db4f
# good: [ab234a260b1f625b26cbefa93ca365b0ae66df33] x86/pv: Rework arch_local_irq_restore() to not use popf
git bisect good ab234a260b1f625b26cbefa93ca365b0ae66df33
# bad: [72f40a2823d6e16229ab58b898c6f22044e5222f] x86/softirq/64: Inline do_softirq_own_stack()
git bisect bad 72f40a2823d6e16229ab58b898c6f22044e5222f
# first bad commit: [72f40a2823d6e16229ab58b898c6f22044e5222f] x86/softirq/64: Inline do_softirq_own_stack()
^ permalink raw reply [flat|nested] 31+ messages in thread