[patch 00/12] x86/irq/64: Inline irq stack switching

[patch 00/12] x86/irq/64: Inline irq stack switching

[Thomas Gleixner]

The recent effort to make the ASM entry code slim and unified moved
the irq stack switching out of the low level ASM code so that the
whole return from interrupt work and state handling can be done in C
and the ASM code just handles the true low level details of entry and
exit (which is horrible enough already due to the well thought out
architeture).

The main goal at this point was to get instrumentation and RCU state
under control in a validated way. Inlining the switch mechanism was
attempted back then, but that caused more objtool and unwinder trouble
than we had already on our plate, so we ended up with a simple,
functional but suboptimal implementation. The main issues are:

  - The unnecessary indirect call which is expensive thanks to
    retpoline
    
  - The inability to stay on the irq stack for softirq processing on return
    from interrupt which requires another stack switch operation.
    
  - The fact that the stack switching code ended up being an easy to find
    exploit gadget.

This series revisits the problem and reimplements the stack switch
mechanics via evil inline assembly. Peter Zijlstra provided the required
objtool and unwinder changes already. These are available here:

  https://lore.kernel.org/r/20210203120222.451068583@infradead.org

The full series (including Peter's series) is also available from git:

  git://git.kernel.org/pub/scm/linux/kernel/git/tglx/devel.git x86/entry

All function calls are now direct and fully inlined including the single
instance in the softirq code which is invoked from local_bh_enable() in
task context.

The extra 100 lines in the diffstat are pretty much the extensive commentry
for the whole magic to spare everyone including myself to scratch heads 2
weeks down the road. 

The text size impact is in the noise and looking at the actual entry
functions there is depending on the compiler variant even a small size
decrease.

The patches have been tested with gcc8, gcc10 and clang-13 (fresh from
git). The difference between the output of these compilers is minimal.
gcc8 being slightly worse due to stupid register selection and random
NOPs injected.

Thanks,

	tglx
---
 arch/x86/Kconfig                 |    1 
 arch/x86/entry/common.c          |   19 --
 arch/x86/entry/entry_64.S        |   41 -----
 arch/x86/include/asm/idtentry.h  |   11 -
 arch/x86/include/asm/irq.h       |    3 
 arch/x86/include/asm/irq_stack.h |  283 +++++++++++++++++++++++++++------------
 arch/x86/include/asm/processor.h |    9 -
 arch/x86/kernel/apic/apic.c      |   31 ++--
 arch/x86/kernel/cpu/common.c     |    4 
 arch/x86/kernel/dumpstack_64.c   |   22 ++-
 arch/x86/kernel/irq.c            |    2 
 arch/x86/kernel/irq_64.c         |   11 -
 arch/x86/kernel/process_64.c     |    2 
 include/linux/interrupt.h        |    2 
 kernel/softirq.c                 |    4 
 15 files changed, 270 insertions(+), 175 deletions(-)

[patch 01/12] x86/entry: Fix instrumentation annotation

[Thomas Gleixner]

Embracing a callout into instrumentation_begin() / instrumentation_begin()
does not really make sense. Make the latter instrumentation_end().

Signed-off-by: Thomas Gleixner <tglx@linutronix.de>
---
 arch/x86/entry/common.c |    2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

--- a/arch/x86/entry/common.c
+++ b/arch/x86/entry/common.c
@@ -270,7 +270,7 @@ static void __xen_pv_evtchn_do_upcall(vo
 
 	instrumentation_begin();
 	run_on_irqstack_cond(__xen_pv_evtchn_do_upcall, regs);
-	instrumentation_begin();
+	instrumentation_end();
 
 	set_irq_regs(old_regs);
 

Re: [patch 01/12] x86/entry: Fix instrumentation annotation

[Borislav Petkov]

On Thu, Feb 04, 2021 at 09:49:04PM +0100, Thomas Gleixner wrote:
> Embracing a callout into instrumentation_begin() / instrumentation_begin()
> does not really make sense. Make the latter instrumentation_end().
> 
> Signed-off-by: Thomas Gleixner <tglx@linutronix.de>
> ---
>  arch/x86/entry/common.c |    2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
> 
> --- a/arch/x86/entry/common.c
> +++ b/arch/x86/entry/common.c
> @@ -270,7 +270,7 @@ static void __xen_pv_evtchn_do_upcall(vo
>  
>  	instrumentation_begin();
>  	run_on_irqstack_cond(__xen_pv_evtchn_do_upcall, regs);
> -	instrumentation_begin();
> +	instrumentation_end();
>  
>  	set_irq_regs(old_regs);

Cc: <stable@vger.kernel.org> ?

-- 
Regards/Gruss,
    Boris.

https://people.kernel.org/tglx/notes-about-netiquette

[patch 02/12] x86/irq: Sanitize irq stack tracking

[Thomas Gleixner]

The recursion protection for hard interrupt stacks is an unsigned int per
CPU variable initialized to -1 named __irq_count. 

The irq stack switching is only done when the variable is -1, which creates
worse code than just checking for 0. When the stack switching happens it
uses this_cpu_add/sub(1), but there is no reason to do so. It simply can
use straight writes. This is a historical leftover from the low level ASM
code which used inc and jz to make a decision.

Rename it to hardirq_stack_inuse, make it a bool and use plain stores.

Signed-off-by: Thomas Gleixner <tglx@linutronix.de>
---
 arch/x86/include/asm/irq_stack.h |   14 +++++++-------
 arch/x86/include/asm/processor.h |    2 +-
 arch/x86/kernel/cpu/common.c     |    2 +-
 arch/x86/kernel/process_64.c     |    2 +-
 4 files changed, 10 insertions(+), 10 deletions(-)

--- a/arch/x86/include/asm/irq_stack.h
+++ b/arch/x86/include/asm/irq_stack.h
@@ -9,7 +9,7 @@
 #ifdef CONFIG_X86_64
 static __always_inline bool irqstack_active(void)
 {
-	return __this_cpu_read(irq_count) != -1;
+	return __this_cpu_read(hardirq_stack_inuse);
 }
 
 void asm_call_on_stack(void *sp, void (*func)(void), void *arg);
@@ -22,9 +22,9 @@ static __always_inline void __run_on_irq
 {
 	void *tos = __this_cpu_read(hardirq_stack_ptr);
 
-	__this_cpu_add(irq_count, 1);
+	__this_cpu_write(hardirq_stack_inuse, true);
 	asm_call_on_stack(tos - 8, func, NULL);
-	__this_cpu_sub(irq_count, 1);
+	__this_cpu_write(hardirq_stack_inuse, false);
 }
 
 static __always_inline void
@@ -33,9 +33,9 @@ static __always_inline void
 {
 	void *tos = __this_cpu_read(hardirq_stack_ptr);
 
-	__this_cpu_add(irq_count, 1);
+	__this_cpu_write(hardirq_stack_inuse, true);
 	asm_call_sysvec_on_stack(tos - 8, func, regs);
-	__this_cpu_sub(irq_count, 1);
+	__this_cpu_write(hardirq_stack_inuse, false);
 }
 
 static __always_inline void
@@ -44,9 +44,9 @@ static __always_inline void
 {
 	void *tos = __this_cpu_read(hardirq_stack_ptr);
 
-	__this_cpu_add(irq_count, 1);
+	__this_cpu_write(hardirq_stack_inuse, true);
 	asm_call_irq_on_stack(tos - 8, func, desc);
-	__this_cpu_sub(irq_count, 1);
+	__this_cpu_write(hardirq_stack_inuse, false);
 }
 
 #else /* CONFIG_X86_64 */
--- a/arch/x86/include/asm/processor.h
+++ b/arch/x86/include/asm/processor.h
@@ -454,7 +454,7 @@ static inline unsigned long cpu_kernelmo
 	return (unsigned long)per_cpu(fixed_percpu_data.gs_base, cpu);
 }
 
-DECLARE_PER_CPU(unsigned int, irq_count);
+DECLARE_PER_CPU(bool, hardirq_stack_inuse);
 extern asmlinkage void ignore_sysret(void);
 
 /* Save actual FS/GS selectors and bases to current->thread */
--- a/arch/x86/kernel/cpu/common.c
+++ b/arch/x86/kernel/cpu/common.c
@@ -1740,7 +1740,7 @@ DEFINE_PER_CPU(struct task_struct *, cur
 EXPORT_PER_CPU_SYMBOL(current_task);
 
 DEFINE_PER_CPU(struct irq_stack *, hardirq_stack_ptr);
-DEFINE_PER_CPU(unsigned int, irq_count) __visible = -1;
+DEFINE_PER_CPU(bool, hardirq_stack_inuse);
 
 DEFINE_PER_CPU(int, __preempt_count) = INIT_PREEMPT_COUNT;
 EXPORT_PER_CPU_SYMBOL(__preempt_count);
--- a/arch/x86/kernel/process_64.c
+++ b/arch/x86/kernel/process_64.c
@@ -539,7 +539,7 @@ void compat_start_thread(struct pt_regs
 	int cpu = smp_processor_id();
 
 	WARN_ON_ONCE(IS_ENABLED(CONFIG_DEBUG_ENTRY) &&
-		     this_cpu_read(irq_count) != -1);
+		     this_cpu_read(hardirq_stack_inuse));
 
 	if (!test_thread_flag(TIF_NEED_FPU_LOAD))
 		switch_fpu_prepare(prev_fpu, cpu);

[patch 03/12] x86/irq/64: Adjust the per CPU irq stack pointer by 8

[Thomas Gleixner]

The per CPU hardirq_stack_ptr contains the pointer to the irq stack in the
form that it is ready to be assigned to [ER]SP so that the first push ends
up on the top entry of the stack.

But the stack switching on 64 bit has the following rules:

    1) Store the current stack pointer (RSP) in the top most stack entry
       to allow the unwinder to link back to the previous stack

    2) Set RSP to the top most stack entry

    3) Invoke functions on the irq stack

    4) Pop RSP from the top most stack entry (stored in #1) so it's back
       to the original stack.

That requires all stack switching code to decrement the stored pointer by 8
in order to be able to store the current RSP and then set RSP to that
location. That's a pointless exercise.

Do the -8 adjustment right when storing the pointer and make the data type
a void pointer to avoid confusion vs. the struct irq_stack data type which
is on 64bit only used to declare the backing store. Move the definition
next to the inuse flag so they end up in the same cache line.

Signed-off-by: Thomas Gleixner <tglx@linutronix.de>
---
 arch/x86/include/asm/irq_stack.h |    6 +++---
 arch/x86/include/asm/processor.h |    7 +++----
 arch/x86/kernel/cpu/common.c     |    2 +-
 arch/x86/kernel/dumpstack_64.c   |   22 ++++++++++++++++------
 arch/x86/kernel/irq_64.c         |    6 ++++--
 5 files changed, 27 insertions(+), 16 deletions(-)

--- a/arch/x86/include/asm/irq_stack.h
+++ b/arch/x86/include/asm/irq_stack.h
@@ -23,7 +23,7 @@ static __always_inline void __run_on_irq
 	void *tos = __this_cpu_read(hardirq_stack_ptr);
 
 	__this_cpu_write(hardirq_stack_inuse, true);
-	asm_call_on_stack(tos - 8, func, NULL);
+	asm_call_on_stack(tos, func, NULL);
 	__this_cpu_write(hardirq_stack_inuse, false);
 }
 
@@ -34,7 +34,7 @@ static __always_inline void
 	void *tos = __this_cpu_read(hardirq_stack_ptr);
 
 	__this_cpu_write(hardirq_stack_inuse, true);
-	asm_call_sysvec_on_stack(tos - 8, func, regs);
+	asm_call_sysvec_on_stack(tos, func, regs);
 	__this_cpu_write(hardirq_stack_inuse, false);
 }
 
@@ -45,7 +45,7 @@ static __always_inline void
 	void *tos = __this_cpu_read(hardirq_stack_ptr);
 
 	__this_cpu_write(hardirq_stack_inuse, true);
-	asm_call_irq_on_stack(tos - 8, func, desc);
+	asm_call_irq_on_stack(tos, func, desc);
 	__this_cpu_write(hardirq_stack_inuse, false);
 }
 
--- a/arch/x86/include/asm/processor.h
+++ b/arch/x86/include/asm/processor.h
@@ -426,8 +426,6 @@ struct irq_stack {
 	char		stack[IRQ_STACK_SIZE];
 } __aligned(IRQ_STACK_SIZE);
 
-DECLARE_PER_CPU(struct irq_stack *, hardirq_stack_ptr);
-
 #ifdef CONFIG_X86_32
 DECLARE_PER_CPU(unsigned long, cpu_current_top_of_stack);
 #else
@@ -454,6 +452,7 @@ static inline unsigned long cpu_kernelmo
 	return (unsigned long)per_cpu(fixed_percpu_data.gs_base, cpu);
 }
 
+DECLARE_PER_CPU(void *, hardirq_stack_ptr);
 DECLARE_PER_CPU(bool, hardirq_stack_inuse);
 extern asmlinkage void ignore_sysret(void);
 
@@ -473,9 +472,9 @@ struct stack_canary {
 };
 DECLARE_PER_CPU_ALIGNED(struct stack_canary, stack_canary);
 #endif
-/* Per CPU softirq stack pointer */
+DECLARE_PER_CPU(struct irq_stack *, hardirq_stack_ptr);
 DECLARE_PER_CPU(struct irq_stack *, softirq_stack_ptr);
-#endif	/* X86_64 */
+#endif	/* !X86_64 */
 
 extern unsigned int fpu_kernel_xstate_size;
 extern unsigned int fpu_user_xstate_size;
--- a/arch/x86/kernel/cpu/common.c
+++ b/arch/x86/kernel/cpu/common.c
@@ -1739,7 +1739,7 @@ DEFINE_PER_CPU(struct task_struct *, cur
 	&init_task;
 EXPORT_PER_CPU_SYMBOL(current_task);
 
-DEFINE_PER_CPU(struct irq_stack *, hardirq_stack_ptr);
+DEFINE_PER_CPU(void *, hardirq_stack_ptr);
 DEFINE_PER_CPU(bool, hardirq_stack_inuse);
 
 DEFINE_PER_CPU(int, __preempt_count) = INIT_PREEMPT_COUNT;
--- a/arch/x86/kernel/dumpstack_64.c
+++ b/arch/x86/kernel/dumpstack_64.c
@@ -128,12 +128,21 @@ static __always_inline bool in_exception
 
 static __always_inline bool in_irq_stack(unsigned long *stack, struct stack_info *info)
 {
-	unsigned long *end   = (unsigned long *)this_cpu_read(hardirq_stack_ptr);
-	unsigned long *begin = end - (IRQ_STACK_SIZE / sizeof(long));
+	unsigned long *end = (unsigned long *)this_cpu_read(hardirq_stack_ptr);
+	unsigned long *begin;
 
 	/*
-	 * This is a software stack, so 'end' can be a valid stack pointer.
-	 * It just means the stack is empty.
+	 * @end points directly to the top most stack entry to avoid a -8
+	 * adjustment in the stack switch hotpath. Adjust it back before
+	 * calculating @begin.
+	 */
+	end++;
+	begin = end - (IRQ_STACK_SIZE / sizeof(long));
+
+	/*
+	 * Due to the switching logic RSP can never be == @end because the
+	 * final operation is 'popq %rsp' which means after that RSP points
+	 * to the original stack and not to @end.
 	 */
 	if (stack < begin || stack >= end)
 		return false;
@@ -143,8 +152,9 @@ static __always_inline bool in_irq_stack
 	info->end	= end;
 
 	/*
-	 * The next stack pointer is the first thing pushed by the entry code
-	 * after switching to the irq stack.
+	 * The next stack pointer is stored at the top of the irq stack
+	 * before switching to the irq stack. Actual stack entries are all
+	 * below that.
 	 */
 	info->next_sp = (unsigned long *)*(end - 1);
 
--- a/arch/x86/kernel/irq_64.c
+++ b/arch/x86/kernel/irq_64.c
@@ -48,7 +48,8 @@ static int map_irq_stack(unsigned int cp
 	if (!va)
 		return -ENOMEM;
 
-	per_cpu(hardirq_stack_ptr, cpu) = va + IRQ_STACK_SIZE;
+	/* Store actual TOS to avoid adjustment in the hotpath */
+	per_cpu(hardirq_stack_ptr, cpu) = va + IRQ_STACK_SIZE - 8;
 	return 0;
 }
 #else
@@ -60,7 +61,8 @@ static int map_irq_stack(unsigned int cp
 {
 	void *va = per_cpu_ptr(&irq_stack_backing_store, cpu);
 
-	per_cpu(hardirq_stack_ptr, cpu) = va + IRQ_STACK_SIZE;
+	/* Store actual TOS to avoid adjustment in the hotpath */
+	per_cpu(hardirq_stack_ptr, cpu) = va + IRQ_STACK_SIZE - 8;
 	return 0;
 }
 #endif

Re: [patch 03/12] x86/irq/64: Adjust the per CPU irq stack pointer by 8

[Peter Zijlstra]

On Thu, Feb 04, 2021 at 09:49:06PM +0100, Thomas Gleixner wrote:
>  Move the definition
> next to the inuse flag so they end up in the same cache line.

> -DEFINE_PER_CPU(struct irq_stack *, hardirq_stack_ptr);
> +DEFINE_PER_CPU(void *, hardirq_stack_ptr);
>  DEFINE_PER_CPU(bool, hardirq_stack_inuse);

Not strictly guaranteed they end up in the same line. If you stick them
in a struct and force alignment on the instance you'll have better
guarantees.

Re: [patch 03/12] x86/irq/64: Adjust the per CPU irq stack pointer by 8

[Thomas Gleixner]

On Fri, Feb 05 2021 at 14:53, Peter Zijlstra wrote:
> On Thu, Feb 04, 2021 at 09:49:06PM +0100, Thomas Gleixner wrote:
>>  Move the definition
>> next to the inuse flag so they end up in the same cache line.
>
>> -DEFINE_PER_CPU(struct irq_stack *, hardirq_stack_ptr);
>> +DEFINE_PER_CPU(void *, hardirq_stack_ptr);
>>  DEFINE_PER_CPU(bool, hardirq_stack_inuse);
>
> Not strictly guaranteed they end up in the same line. If you stick them
> in a struct and force alignment on the instance you'll have better
> guarantees.

You're right. There are a bunch of per cpu variables which are randomly
defined all over the place which should stay in the same cache line.

[patch 04/12] x86/apic: Split out spurious handling code

[Thomas Gleixner]

sysvec_spurious_apic_interrupt() calls into the handling body of
__spurious_interrupt() which is not obvious as that function is declared
inside the DEFINE_IDTENTRY_IRQ(spurious_interrupt) macro.

As __spurious_interrupt() is currently always inlined this ends up with two
copies of the same code for no reason.

Split the handling function out and invoke it from both entry points.

Signed-off-by: Thomas Gleixner <tglx@linutronix.de>
---
 arch/x86/kernel/apic/apic.c |   31 +++++++++++++++++++------------
 1 file changed, 19 insertions(+), 12 deletions(-)

--- a/arch/x86/kernel/apic/apic.c
+++ b/arch/x86/kernel/apic/apic.c
@@ -2133,18 +2133,11 @@ void __init register_lapic_address(unsig
  * Local APIC interrupts
  */
 
-/**
- * spurious_interrupt - Catch all for interrupts raised on unused vectors
- * @regs:	Pointer to pt_regs on stack
- * @vector:	The vector number
- *
- * This is invoked from ASM entry code to catch all interrupts which
- * trigger on an entry which is routed to the common_spurious idtentry
- * point.
- *
- * Also called from sysvec_spurious_apic_interrupt().
+/*
+ * Common handling code for spurious_interrupt and spurious_vector entry
+ * points below. No point in allowing the compiler to inline it twice.
  */
-DEFINE_IDTENTRY_IRQ(spurious_interrupt)
+static noinline void handle_spurious_interrupt(u8 vector)
 {
 	u32 v;
 
@@ -2179,9 +2172,23 @@ DEFINE_IDTENTRY_IRQ(spurious_interrupt)
 	trace_spurious_apic_exit(vector);
 }
 
+/**
+ * spurious_interrupt - Catch all for interrupts raised on unused vectors
+ * @regs:	Pointer to pt_regs on stack
+ * @vector:	The vector number
+ *
+ * This is invoked from ASM entry code to catch all interrupts which
+ * trigger on an entry which is routed to the common_spurious idtentry
+ * point.
+ */
+DEFINE_IDTENTRY_IRQ(spurious_interrupt)
+{
+	handle_spurious_interrupt(vector);
+}
+
 DEFINE_IDTENTRY_SYSVEC(sysvec_spurious_apic_interrupt)
 {
-	__spurious_interrupt(regs, SPURIOUS_APIC_VECTOR);
+	handle_spurious_interrupt(SPURIOUS_APIC_VECTOR);
 }
 
 /*

[patch 05/12] x86/irq: Provide macro for inlining irq stack switching

[Thomas Gleixner]

The effort to make the ASM entry code slim and unified moved the irq stack
switching out of the low level ASM code so that the whole return from
interrupt work and state handling can be done in C and the ASM code just
handles the low level details of entry and exit.

This ended up being a suboptimal implementation for various reasons
(including tooling). The main pain points are:

 - The indirect call which is expensive thanks to retpoline

 - The inability to stay on the irq stack for softirq processing on return
   from interrupt

 - The fact that the stack switching code ends up being an easy to target
   exploit gadget.

Prepare for inlining the stack switching logic into the C entry points by
providing a ASM macro which contains the guts of the switching mechanism:

  1) Store RSP at the top of the irq stack
  2) Switch RSP to the irq stack
  3) Invoke code
  4) Pop the original RSP back

Document the unholy asm() logic while at it to reduce the amount of head
scratching required a half year from now.

Signed-off-by: Thomas Gleixner <tglx@linutronix.de>
---
 arch/x86/include/asm/irq_stack.h |  104 +++++++++++++++++++++++++++++++++++++++
 1 file changed, 104 insertions(+)

--- a/arch/x86/include/asm/irq_stack.h
+++ b/arch/x86/include/asm/irq_stack.h
@@ -7,6 +7,110 @@
 #include <asm/processor.h>
 
 #ifdef CONFIG_X86_64
+
+#ifdef CONFIG_UNWINDER_FRAME_POINTER
+# define IRQSTACK_CALL_CONSTRAINT	, ASM_CALL_CONSTRAINT
+#else
+# define IRQSTACK_CALL_CONSTRAINT
+#endif
+
+/*
+ * Macro to inline switching to an interrupt stack and invoking function
+ * calls from there. The following rules apply:
+ *
+ * - Ordering:
+ *
+ *   1. Write the stack pointer content into the top most place of
+ *	the irq stack. This ensures that the various unwinders can
+ *	link back to the original stack.
+ *
+ *   2. Switch the stack pointer to the top of the irq stack.
+ *
+ *   3. Invoke whatever needs to be done (@asm_call argument)
+ *
+ *   4. Pop the original stack pointer from the top of the irq stack
+ *	which brings it back to the original stack where it left off.
+ *
+ * - Function invocation:
+ *
+ *   To allow flexible usage of the macro, the actual function code including
+ *   the store of the arguments in the call ABI registers is handed in via
+ *   the @asm_call argument.
+ *
+ * - Local variables:
+ *
+ *   @tos:
+ *	The @tos variable holds a pointer to the top of the irq stack and
+ *	_must_ be allocated in a non-callee saved register as this is a
+ *	restriction coming from objtool.
+ *
+ *	Note, that (tos) is both in input and output constraints to ensure
+ *	that the compiler does not assume that R11 is left untouched in
+ *	case this macro is used in some place where the per cpu interrupt
+ *	stack pointer is used again afterwards
+ *
+ * - Function arguments:
+ *        The function argument(s) if any have to be defined in register
+ *	  variables at the place where this is invoked. Storing the
+ *	  argument(s) in the proper register(s) is part of the @asm_call
+ *
+ * - Constraints:
+ *
+ *   The constraints have to be done very carefully because the compiler
+ *   does not know about the assembly call.
+ *
+ *   output:
+ *     As documented already above the @tos variable is required to be in
+ *     the output constraints to make the compiler aware that R11 cannot be
+ *     reused after the asm() statement.
+ *
+ *     For builds with CONFIG_UNWIND_FRAME_POINTER ASM_CALL_CONSTRAINT is
+ *     required as well as this prevents certain creative GCC variants from
+ *     misplacing the ASM code.
+ *
+ *  input:
+ *    - func:
+ *	  Immediate, which tells the compiler that the function is referenced.
+ *
+ *    - tos:
+ *	  Register. The actual register is defined by the variable declaration.
+ *
+ *    - function arguments:
+ *	  The constraints are handed in via the 'argconstr' argument list. They
+ *	  describe the register arguments which are used in @asm_call.
+ *
+ *  clobbers:
+ *     Function calls can clobber anything except the callee-saved
+ *     registers. Tell the compiler.
+ */
+#define __call_on_irqstack(func, asm_call, constr...)			\
+{									\
+	register void *tos asm("r11");					\
+									\
+	tos = ((void *)__this_cpu_read(hardirq_stack_ptr));		\
+									\
+	asm_inline volatile(						\
+	"movq	%%rsp, (%[__tos])			\n"		\
+	"movq	%[__tos], %%rsp				\n"		\
+									\
+	asm_call							\
+									\
+	"popq	%%rsp					\n"		\
+									\
+	: "+r" (tos) IRQSTACK_CALL_CONSTRAINT				\
+	: [__func] "i" (func), [__tos] "r" (tos) constr			\
+	: "cc", "rax", "rcx", "rdx", "rsi", "rdi", "r8", "r9", "r10",	\
+	  "memory"							\
+	);								\
+}
+
+/* Macros to assert type correctness for run_*_on_irqstack macros */
+#define assert_function_type(func, proto)				\
+	static_assert(__builtin_types_compatible_p(typeof(&func), proto))
+
+#define assert_arg_type(arg, proto)					\
+	static_assert(__builtin_types_compatible_p(typeof(arg), proto))
+
 static __always_inline bool irqstack_active(void)
 {
 	return __this_cpu_read(hardirq_stack_inuse);

Re: [patch 05/12] x86/irq: Provide macro for inlining irq stack switching

[Borislav Petkov]

On Thu, Feb 04, 2021 at 09:49:08PM +0100, Thomas Gleixner wrote:
> The effort to make the ASM entry code slim and unified moved the irq stack
> switching out of the low level ASM code so that the whole return from
> interrupt work and state handling can be done in C and the ASM code just
> handles the low level details of entry and exit.
> 
> This ended up being a suboptimal implementation for various reasons
> (including tooling). The main pain points are:
> 
>  - The indirect call which is expensive thanks to retpoline
> 
>  - The inability to stay on the irq stack for softirq processing on return
>    from interrupt
> 
>  - The fact that the stack switching code ends up being an easy to target
>    exploit gadget.
> 
> Prepare for inlining the stack switching logic into the C entry points by
> providing a ASM macro which contains the guts of the switching mechanism:
> 
>   1) Store RSP at the top of the irq stack
>   2) Switch RSP to the irq stack
>   3) Invoke code
>   4) Pop the original RSP back
> 
> Document the unholy asm() logic while at it to reduce the amount of head
> scratching required a half year from now.
> 
> Signed-off-by: Thomas Gleixner <tglx@linutronix.de>
> ---
>  arch/x86/include/asm/irq_stack.h |  104 +++++++++++++++++++++++++++++++++++++++
>  1 file changed, 104 insertions(+)
> 
> --- a/arch/x86/include/asm/irq_stack.h
> +++ b/arch/x86/include/asm/irq_stack.h
> @@ -7,6 +7,110 @@
>  #include <asm/processor.h>
>  
>  #ifdef CONFIG_X86_64
> +
> +#ifdef CONFIG_UNWINDER_FRAME_POINTER
> +# define IRQSTACK_CALL_CONSTRAINT	, ASM_CALL_CONSTRAINT
> +#else
> +# define IRQSTACK_CALL_CONSTRAINT
> +#endif
> +
> +/*
> + * Macro to inline switching to an interrupt stack and invoking function
> + * calls from there. The following rules apply:
> + *
> + * - Ordering:
> + *
> + *   1. Write the stack pointer content into the top most place of

I think "content" is not needed here - just "Write the stack pointer".

> + *	the irq stack. This ensures that the various unwinders can
> + *	link back to the original stack.
> + *
> + *   2. Switch the stack pointer to the top of the irq stack.
> + *
> + *   3. Invoke whatever needs to be done (@asm_call argument)
> + *
> + *   4. Pop the original stack pointer from the top of the irq stack
> + *	which brings it back to the original stack where it left off.
> + *
> + * - Function invocation:
> + *
> + *   To allow flexible usage of the macro, the actual function code including
> + *   the store of the arguments in the call ABI registers is handed in via
> + *   the @asm_call argument.
> + *
> + * - Local variables:
> + *
> + *   @tos:
> + *	The @tos variable holds a pointer to the top of the irq stack and
> + *	_must_ be allocated in a non-callee saved register as this is a
> + *	restriction coming from objtool.
> + *
> + *	Note, that (tos) is both in input and output constraints to ensure
> + *	that the compiler does not assume that R11 is left untouched in
> + *	case this macro is used in some place where the per cpu interrupt
> + *	stack pointer is used again afterwards
> + *
> + * - Function arguments:
> + *        The function argument(s) if any have to be defined in register

Commas:

The function argument(s), if any, ...

> + *	  variables at the place where this is invoked. Storing the
> + *	  argument(s) in the proper register(s) is part of the @asm_call
> + *
> + * - Constraints:
> + *
> + *   The constraints have to be done very carefully because the compiler
> + *   does not know about the assembly call.
> + *
> + *   output:
> + *     As documented already above the @tos variable is required to be in
> + *     the output constraints to make the compiler aware that R11 cannot be
> + *     reused after the asm() statement.
> + *
> + *     For builds with CONFIG_UNWIND_FRAME_POINTER ASM_CALL_CONSTRAINT is
> + *     required as well as this prevents certain creative GCC variants from
> + *     misplacing the ASM code.
> + *
> + *  input:
> + *    - func:
> + *	  Immediate, which tells the compiler that the function is referenced.
> + *
> + *    - tos:
> + *	  Register. The actual register is defined by the variable declaration.
> + *
> + *    - function arguments:
> + *	  The constraints are handed in via the 'argconstr' argument list. They

"argconstr" or "constr"?

> + *	  describe the register arguments which are used in @asm_call.
> + *
> + *  clobbers:
> + *     Function calls can clobber anything except the callee-saved
> + *     registers. Tell the compiler.
> + */
> +#define __call_on_irqstack(func, asm_call, constr...)			\

Does the name need to be prepended with "__"? I don't see a
"call_on_irqstack" variant...

> +{									\
> +	register void *tos asm("r11");					\
> +									\
> +	tos = ((void *)__this_cpu_read(hardirq_stack_ptr));		\
> +									\
> +	asm_inline volatile(						\
> +	"movq	%%rsp, (%[__tos])			\n"		\
> +	"movq	%[__tos], %%rsp				\n"		\
> +									\
> +	asm_call							\
> +									\
> +	"popq	%%rsp					\n"		\
> +									\
> +	: "+r" (tos) IRQSTACK_CALL_CONSTRAINT				\
> +	: [__func] "i" (func), [__tos] "r" (tos) constr			\

I think you can call the symbolic names the same as the variables -
i.e., without the "__" so that there's less confusion when looking at
the code.

> +	: "cc", "rax", "rcx", "rdx", "rsi", "rdi", "r8", "r9", "r10",	\
> +	  "memory"							\
> +	);								\
> +}
> +
> +/* Macros to assert type correctness for run_*_on_irqstack macros */
> +#define assert_function_type(func, proto)				\
> +	static_assert(__builtin_types_compatible_p(typeof(&func), proto))
> +
> +#define assert_arg_type(arg, proto)					\
> +	static_assert(__builtin_types_compatible_p(typeof(arg), proto))
> +
>  static __always_inline bool irqstack_active(void)
>  {
>  	return __this_cpu_read(hardirq_stack_inuse);
> 

-- 
Regards/Gruss,
    Boris.

https://people.kernel.org/tglx/notes-about-netiquette

Re: [patch 05/12] x86/irq: Provide macro for inlining irq stack switching

[Josh Poimboeuf]

On Thu, Feb 04, 2021 at 09:49:08PM +0100, Thomas Gleixner wrote:
>  #ifdef CONFIG_X86_64
> +
> +#ifdef CONFIG_UNWINDER_FRAME_POINTER
> +# define IRQSTACK_CALL_CONSTRAINT	, ASM_CALL_CONSTRAINT
> +#else
> +# define IRQSTACK_CALL_CONSTRAINT
> +#endif

Is this really needed?  i.e. does ASM_CALL_CONSTRAINT actually affect
code generation with !FRAME_POINTER?

If so then we should rework ASM_CALL_CONSTRAINT itself to be something
similar to the above, as it's only ever needed with frame pointers.

-- 
Josh

Re: [patch 05/12] x86/irq: Provide macro for inlining irq stack switching

[Thomas Gleixner]

On Mon, Feb 08 2021 at 14:42, Josh Poimboeuf wrote:
> On Thu, Feb 04, 2021 at 09:49:08PM +0100, Thomas Gleixner wrote:
>>  #ifdef CONFIG_X86_64
>> +
>> +#ifdef CONFIG_UNWINDER_FRAME_POINTER
>> +# define IRQSTACK_CALL_CONSTRAINT	, ASM_CALL_CONSTRAINT
>> +#else
>> +# define IRQSTACK_CALL_CONSTRAINT
>> +#endif
>
> Is this really needed?  i.e. does ASM_CALL_CONSTRAINT actually affect
> code generation with !FRAME_POINTER?

The problem is that if the asm inline is the first operation in a
function some compilers insert the asm inline before setting up the
frame pointer.

That's actualy irrelevant here as the compiler cannot reorder against
the C code leading to the asm inline. So we can probably replace it with
a big fat comment.

Thanks,

        tglx

Re: [patch 05/12] x86/irq: Provide macro for inlining irq stack switching

[Josh Poimboeuf]

On Tue, Feb 09, 2021 at 04:12:33PM +0100, Thomas Gleixner wrote:
> On Mon, Feb 08 2021 at 14:42, Josh Poimboeuf wrote:
> > On Thu, Feb 04, 2021 at 09:49:08PM +0100, Thomas Gleixner wrote:
> >>  #ifdef CONFIG_X86_64
> >> +
> >> +#ifdef CONFIG_UNWINDER_FRAME_POINTER
> >> +# define IRQSTACK_CALL_CONSTRAINT	, ASM_CALL_CONSTRAINT
> >> +#else
> >> +# define IRQSTACK_CALL_CONSTRAINT
> >> +#endif
> >
> > Is this really needed?  i.e. does ASM_CALL_CONSTRAINT actually affect
> > code generation with !FRAME_POINTER?
> 
> The problem is that if the asm inline is the first operation in a
> function some compilers insert the asm inline before setting up the
> frame pointer.
> 
> That's actualy irrelevant here as the compiler cannot reorder against
> the C code leading to the asm inline. So we can probably replace it with
> a big fat comment.

Actually, I think keeping ASM_CALL_CONSTRAINT is a good idea.

What I meant was, is the #ifdef needed?  My previous understanding was
that ASM_CALL_CONSTRAINT has no effect for !FRAME_POINTER (i.e., ORC).

So is there any reason to *not* have ASM_CALL_CONSTRAINT with ORC?

-- 
Josh

Re: [patch 05/12] x86/irq: Provide macro for inlining irq stack switching

[Thomas Gleixner]

On Tue, Feb 09 2021 at 10:22, Josh Poimboeuf wrote:
> On Tue, Feb 09, 2021 at 04:12:33PM +0100, Thomas Gleixner wrote:
>> On Mon, Feb 08 2021 at 14:42, Josh Poimboeuf wrote:
>> > On Thu, Feb 04, 2021 at 09:49:08PM +0100, Thomas Gleixner wrote:
>> >>  #ifdef CONFIG_X86_64
>> >> +
>> >> +#ifdef CONFIG_UNWINDER_FRAME_POINTER
>> >> +# define IRQSTACK_CALL_CONSTRAINT	, ASM_CALL_CONSTRAINT
>> >> +#else
>> >> +# define IRQSTACK_CALL_CONSTRAINT
>> >> +#endif
>> >
>> > Is this really needed?  i.e. does ASM_CALL_CONSTRAINT actually affect
>> > code generation with !FRAME_POINTER?
>> 
>> The problem is that if the asm inline is the first operation in a
>> function some compilers insert the asm inline before setting up the
>> frame pointer.
>> 
>> That's actualy irrelevant here as the compiler cannot reorder against
>> the C code leading to the asm inline. So we can probably replace it with
>> a big fat comment.
>
> Actually, I think keeping ASM_CALL_CONSTRAINT is a good idea.
>
> What I meant was, is the #ifdef needed?  My previous understanding was
> that ASM_CALL_CONSTRAINT has no effect for !FRAME_POINTER (i.e., ORC).
>
> So is there any reason to *not* have ASM_CALL_CONSTRAINT with ORC?

You're right. No idea how I ended up with that ifdef.

Thanks,

        tglx

[patch 06/12] x86/entry: Convert system vectors to irq stack macro

[Thomas Gleixner]

To inline the stack switching and to prepare for enabling
CONFIG_HAVE_IRQ_EXIT_ON_IRQ_STACK provide a macro template for system
vectors and device interrupts and convert the system vectors over to it.

Signed-off-by: Thomas Gleixner <tglx@linutronix.de>
---
 arch/x86/entry/entry_64.S        |    1 
 arch/x86/include/asm/idtentry.h  |    2 
 arch/x86/include/asm/irq_stack.h |   93 +++++++++++++++++++++++++++------------
 3 files changed, 66 insertions(+), 30 deletions(-)

--- a/arch/x86/entry/entry_64.S
+++ b/arch/x86/entry/entry_64.S
@@ -762,7 +762,6 @@ SYM_CODE_END(.Lbad_gs)
  * rdx: Function argument (can be NULL if none)
  */
 SYM_FUNC_START(asm_call_on_stack)
-SYM_INNER_LABEL(asm_call_sysvec_on_stack, SYM_L_GLOBAL)
 SYM_INNER_LABEL(asm_call_irq_on_stack, SYM_L_GLOBAL)
 	/*
 	 * Save the frame pointer unconditionally. This allows the ORC
--- a/arch/x86/include/asm/idtentry.h
+++ b/arch/x86/include/asm/idtentry.h
@@ -237,10 +237,8 @@ static void __##func(struct pt_regs *reg
 	irqentry_state_t state = irqentry_enter(regs);			\
 									\
 	instrumentation_begin();					\
-	irq_enter_rcu();						\
 	kvm_set_cpu_l1tf_flush_l1d();					\
 	run_sysvec_on_irqstack_cond(__##func, regs);			\
-	irq_exit_rcu();							\
 	instrumentation_end();						\
 	irqentry_exit(regs, state);					\
 }									\
--- a/arch/x86/include/asm/irq_stack.h
+++ b/arch/x86/include/asm/irq_stack.h
@@ -111,14 +111,69 @@
 #define assert_arg_type(arg, proto)					\
 	static_assert(__builtin_types_compatible_p(typeof(arg), proto))
 
+/*
+ * Macro to invoke system vector and device interrupt C handlers.
+ */
+#define call_on_irqstack_cond(func, regs, asm_call, constr, c_args...)	\
+{									\
+	/*								\
+	 * User mode entry and interrupt on the irq stack do not	\
+	 * switch stacks. If from user mode the task stack is empty.	\
+	 */								\
+	if (user_mode(regs) || __this_cpu_read(hardirq_stack_inuse)) {	\
+		irq_enter_rcu();					\
+		func(c_args);						\
+		irq_exit_rcu();						\
+	} else {							\
+		/*							\
+		 * Mark the irq stack inuse _before_ and unmark _after_	\
+		 * switching stacks. Interrupts are disabled in both	\
+		 * places. Invoke the stack switch macro with the call	\
+		 * sequence which matches the above direct invocation.	\
+		 */							\
+		__this_cpu_write(hardirq_stack_inuse, true);		\
+		__call_on_irqstack(func, asm_call, constr);		\
+		__this_cpu_write(hardirq_stack_inuse, false);		\
+	}								\
+}
+
+/*
+ * Function call sequence for __call_on_irqstack() for system vectors.
+ *
+ * Note that irq_enter_rcu() and irq_exit_rcu() do not use the input
+ * mechanism because these functions are global and cannot be optimized out
+ * when compiling a particular source file which uses one of these macros.
+ *
+ * The argument (regs) does not need to be pushed or stashed in a callee
+ * saved register to be safe vs. the irq_enter_rcu() call because the
+ * clobbers already prevent the compiler from storing it in a callee
+ * clobbered register. As the compiler has to preserve @regs for the final
+ * call to idtentry_exit() anyway, it's likely that it does not cause extra
+ * effort for this asm magic.
+ */
+#define ASM_CALL_SYSVEC							\
+	"call irq_enter_rcu				\n"		\
+	"movq	%[__arg1], %%rdi			\n"		\
+	"call %P[__func]				\n"		\
+	"call irq_exit_rcu				\n"
+
+#define SYSVEC_CONSTRAINTS	, [__arg1] "r" (regs)
+
+#define run_sysvec_on_irqstack_cond(func, regs)				\
+{									\
+	assert_function_type(func, void (*)(struct pt_regs *));		\
+	assert_arg_type(regs, struct pt_regs *);			\
+									\
+	call_on_irqstack_cond(func, regs, ASM_CALL_SYSVEC,		\
+			      SYSVEC_CONSTRAINTS, regs);		\
+}
+
 static __always_inline bool irqstack_active(void)
 {
 	return __this_cpu_read(hardirq_stack_inuse);
 }
 
 void asm_call_on_stack(void *sp, void (*func)(void), void *arg);
-void asm_call_sysvec_on_stack(void *sp, void (*func)(struct pt_regs *regs),
-			      struct pt_regs *regs);
 void asm_call_irq_on_stack(void *sp, void (*func)(struct irq_desc *desc),
 			   struct irq_desc *desc);
 
@@ -132,17 +187,6 @@ static __always_inline void __run_on_irq
 }
 
 static __always_inline void
-__run_sysvec_on_irqstack(void (*func)(struct pt_regs *regs),
-			 struct pt_regs *regs)
-{
-	void *tos = __this_cpu_read(hardirq_stack_ptr);
-
-	__this_cpu_write(hardirq_stack_inuse, true);
-	asm_call_sysvec_on_stack(tos, func, regs);
-	__this_cpu_write(hardirq_stack_inuse, false);
-}
-
-static __always_inline void
 __run_irq_on_irqstack(void (*func)(struct irq_desc *desc),
 		      struct irq_desc *desc)
 {
@@ -154,10 +198,17 @@ static __always_inline void
 }
 
 #else /* CONFIG_X86_64 */
+
+/* System vector handlers always run on the stack they interrupted. */
+#define run_sysvec_on_irqstack_cond(func, regs)				\
+{									\
+	irq_enter_rcu();						\
+	func(regs);							\
+	irq_exit_rcu();							\
+}
+
 static inline bool irqstack_active(void) { return false; }
 static inline void __run_on_irqstack(void (*func)(void)) { }
-static inline void __run_sysvec_on_irqstack(void (*func)(struct pt_regs *regs),
-					    struct pt_regs *regs) { }
 static inline void __run_irq_on_irqstack(void (*func)(struct irq_desc *desc),
 					 struct irq_desc *desc) { }
 #endif /* !CONFIG_X86_64 */
@@ -184,18 +235,6 @@ static __always_inline void run_on_irqst
 }
 
 static __always_inline void
-run_sysvec_on_irqstack_cond(void (*func)(struct pt_regs *regs),
-			    struct pt_regs *regs)
-{
-	lockdep_assert_irqs_disabled();
-
-	if (irq_needs_irq_stack(regs))
-		__run_sysvec_on_irqstack(func, regs);
-	else
-		func(regs);
-}
-
-static __always_inline void
 run_irq_on_irqstack_cond(void (*func)(struct irq_desc *desc), struct irq_desc *desc,
 			 struct pt_regs *regs)
 {

Re: [patch 06/12] x86/entry: Convert system vectors to irq stack macro

[kernel test robot]

[-- Attachment #1: Type: text/plain, Size: 1750 bytes --]

Hi Thomas,

I love your patch! Perhaps something to improve:

[auto build test WARNING on tip/x86/asm]
[also build test WARNING on tip/master linus/master tip/x86/core v5.11-rc6 next-20210125]
[If your patch is applied to the wrong git tree, kindly drop us a note.
And when submitting patch, we suggest to use '--base' as documented in
https://git-scm.com/docs/git-format-patch]

url:    https://github.com/0day-ci/linux/commits/Thomas-Gleixner/x86-irq-64-Inline-irq-stack-switching/20210205-091059
base:   https://git.kernel.org/pub/scm/linux/kernel/git/tip/tip.git 5c99720b28381bb400d4f546734c34ddaf608761
config: x86_64-randconfig-r026-20210204 (attached as .config)
compiler: gcc-9 (Debian 9.3.0-15) 9.3.0
reproduce (this is a W=1 build):
        # https://github.com/0day-ci/linux/commit/d91ff58e804175dd59e483c7cf236e1fe66c2187
        git remote add linux-review https://github.com/0day-ci/linux
        git fetch --no-tags linux-review Thomas-Gleixner/x86-irq-64-Inline-irq-stack-switching/20210205-091059
        git checkout d91ff58e804175dd59e483c7cf236e1fe66c2187
        # save the attached .config to linux build tree
        make W=1 ARCH=x86_64 

If you fix the issue, kindly add following tag as appropriate
Reported-by: kernel test robot <lkp@intel.com>

All warnings (new ones prefixed by >>):

>> arch/x86/hyperv/hv_init.o: warning: objtool: sysvec_hyperv_reenlightenment()+0x7f: undefined stack state
--
>> arch/x86/kernel/cpu/mshyperv.o: warning: objtool: sysvec_hyperv_callback()+0x7f: undefined stack state
>> arch/x86/kernel/cpu/mshyperv.o: warning: objtool: sysvec_hyperv_stimer0()+0x7f: undefined stack state

---
0-DAY CI Kernel Test Service, Intel Corporation
https://lists.01.org/hyperkitty/list/kbuild-all@lists.01.org

[-- Attachment #2: .config.gz --]
[-- Type: application/gzip, Size: 37569 bytes --]

Re: [patch 06/12] x86/entry: Convert system vectors to irq stack macro

[Peter Zijlstra]

On Fri, Feb 05, 2021 at 11:52:40AM +0800, kernel test robot wrote:
> Hi Thomas,
> 
> I love your patch! Perhaps something to improve:
> 
> [auto build test WARNING on tip/x86/asm]
> [also build test WARNING on tip/master linus/master tip/x86/core v5.11-rc6 next-20210125]
> [If your patch is applied to the wrong git tree, kindly drop us a note.
> And when submitting patch, we suggest to use '--base' as documented in
> https://git-scm.com/docs/git-format-patch]
> 
> url:    https://github.com/0day-ci/linux/commits/Thomas-Gleixner/x86-irq-64-Inline-irq-stack-switching/20210205-091059
> base:   https://git.kernel.org/pub/scm/linux/kernel/git/tip/tip.git 5c99720b28381bb400d4f546734c34ddaf608761
> config: x86_64-randconfig-r026-20210204 (attached as .config)
> compiler: gcc-9 (Debian 9.3.0-15) 9.3.0
> reproduce (this is a W=1 build):
>         # https://github.com/0day-ci/linux/commit/d91ff58e804175dd59e483c7cf236e1fe66c2187
>         git remote add linux-review https://github.com/0day-ci/linux
>         git fetch --no-tags linux-review Thomas-Gleixner/x86-irq-64-Inline-irq-stack-switching/20210205-091059
>         git checkout d91ff58e804175dd59e483c7cf236e1fe66c2187
>         # save the attached .config to linux build tree
>         make W=1 ARCH=x86_64 
> 
> If you fix the issue, kindly add following tag as appropriate
> Reported-by: kernel test robot <lkp@intel.com>
> 
> All warnings (new ones prefixed by >>):
> 
> >> arch/x86/hyperv/hv_init.o: warning: objtool: sysvec_hyperv_reenlightenment()+0x7f: undefined stack state
> --
> >> arch/x86/kernel/cpu/mshyperv.o: warning: objtool: sysvec_hyperv_callback()+0x7f: undefined stack state
> >> arch/x86/kernel/cpu/mshyperv.o: warning: objtool: sysvec_hyperv_stimer0()+0x7f: undefined stack state

It would help if you'd actually applied the patches to a tree that had
the required objtool patches as described in 0/n. Or better yet, don't
scrape emails if the 0/n includes a git link which you'll run on anyway.

Re: [kbuild-all] Re: [patch 06/12] x86/entry: Convert system vectors to irq stack macro

[Rong Chen]

On 2/5/21 10:13 PM, Peter Zijlstra wrote:
> On Fri, Feb 05, 2021 at 11:52:40AM +0800, kernel test robot wrote:
>> Hi Thomas,
>>
>> I love your patch! Perhaps something to improve:
>>
>> [auto build test WARNING on tip/x86/asm]
>> [also build test WARNING on tip/master linus/master tip/x86/core v5.11-rc6 next-20210125]
>> [If your patch is applied to the wrong git tree, kindly drop us a note.
>> And when submitting patch, we suggest to use '--base' as documented in
>> https://git-scm.com/docs/git-format-patch]
>>
>> url:    https://github.com/0day-ci/linux/commits/Thomas-Gleixner/x86-irq-64-Inline-irq-stack-switching/20210205-091059
>> base:   https://git.kernel.org/pub/scm/linux/kernel/git/tip/tip.git 5c99720b28381bb400d4f546734c34ddaf608761
>> config: x86_64-randconfig-r026-20210204 (attached as .config)
>> compiler: gcc-9 (Debian 9.3.0-15) 9.3.0
>> reproduce (this is a W=1 build):
>>          # https://github.com/0day-ci/linux/commit/d91ff58e804175dd59e483c7cf236e1fe66c2187
>>          git remote add linux-review https://github.com/0day-ci/linux
>>          git fetch --no-tags linux-review Thomas-Gleixner/x86-irq-64-Inline-irq-stack-switching/20210205-091059
>>          git checkout d91ff58e804175dd59e483c7cf236e1fe66c2187
>>          # save the attached .config to linux build tree
>>          make W=1 ARCH=x86_64
>>
>> If you fix the issue, kindly add following tag as appropriate
>> Reported-by: kernel test robot <lkp@intel.com>
>>
>> All warnings (new ones prefixed by >>):
>>
>>>> arch/x86/hyperv/hv_init.o: warning: objtool: sysvec_hyperv_reenlightenment()+0x7f: undefined stack state
>> --
>>>> arch/x86/kernel/cpu/mshyperv.o: warning: objtool: sysvec_hyperv_callback()+0x7f: undefined stack state
>>>> arch/x86/kernel/cpu/mshyperv.o: warning: objtool: sysvec_hyperv_stimer0()+0x7f: undefined stack state
> It would help if you'd actually applied the patches to a tree that had
> the required objtool patches as described in 0/n. Or better yet, don't
> scrape emails if the 0/n includes a git link which you'll run on anyway.

Hi Peter,

Thanks for the advice, we'll add the check to our cluster,
and sorry for the inconvenience.

Best Regards,
Rong Chen

Re: [kbuild-all] Re: [patch 06/12] x86/entry: Convert system vectors to irq stack macro

[Borislav Petkov]

On Sun, Feb 07, 2021 at 04:15:11PM +0800, Rong Chen wrote:
> Thanks for the advice, we'll add the check to our cluster,
> and sorry for the inconvenience.

When it comes to the tip tree, I'd say you guys are much better off not
scraping any patches from the mailing list but simply testing the tip
branches. That would be more than enough and you already do that anyway.

Thx.

-- 
Regards/Gruss,
    Boris.

https://people.kernel.org/tglx/notes-about-netiquette

Re: [kbuild-all] Re: [patch 06/12] x86/entry: Convert system vectors to irq stack macro

[Rong Chen]

On 2/8/21 10:19 PM, Borislav Petkov wrote:
> On Sun, Feb 07, 2021 at 04:15:11PM +0800, Rong Chen wrote:
>> Thanks for the advice, we'll add the check to our cluster,
>> and sorry for the inconvenience.
> When it comes to the tip tree, I'd say you guys are much better off not
> scraping any patches from the mailing list but simply testing the tip
> branches. That would be more than enough and you already do that anyway.
>
> Thx.
>

Hi Borislav,

Thanks for the help, how can we identify the patches for tip tree,
could you please guide us?

Best Regards,
Rong Chen

Re: [kbuild-all] Re: [patch 06/12] x86/entry: Convert system vectors to irq stack macro

[Borislav Petkov]

On Tue, Feb 09, 2021 at 04:57:09PM +0800, Rong Chen wrote:
> Thanks for the help, how can we identify the patches for tip tree,
> could you please guide us?

I guess something like:

cat patch | ./scripts/get_maintainer.pl -m --no-r --no-l

For a tip patch, it should give, among others:

x86@kernel.org (maintainer:X86 ARCHITECTURE (32-BIT AND 64-BIT))

which is the tip ML.

You'd need to play with this a bit to see what works best.

Also, I'm wondering if it would make sense to have some special syntax
to let the 0day bot know on which tree to apply the patches after
scraping or to even be able to say: "do not test" when sending just an
example patch which is not supposed to be used anyway and thus you don't
have to waste infra cycles on it.

HTH.

-- 
Regards/Gruss,
    Boris.

https://people.kernel.org/tglx/notes-about-netiquette

[patch 07/12] x86/entry: Convert device interrupts to inline stack switching

[Thomas Gleixner]

Convert device interrupts to inline stack switching by replacing the
existing macro implementation with the new inline version. Tweak the
function signature of the actual handler function to have the vector
argument as u32. That allows the inline macro to avoid extra intermediates
and lets the compiler be smarter about the whole thing.

Signed-off-by: Thomas Gleixner <tglx@linutronix.de>
---
 arch/x86/entry/entry_64.S        |    1 
 arch/x86/include/asm/idtentry.h  |    9 ++----
 arch/x86/include/asm/irq_stack.h |   58 ++++++++++++++++++++-------------------
 arch/x86/kernel/irq.c            |    2 -
 4 files changed, 36 insertions(+), 34 deletions(-)

--- a/arch/x86/entry/entry_64.S
+++ b/arch/x86/entry/entry_64.S
@@ -762,7 +762,6 @@ SYM_CODE_END(.Lbad_gs)
  * rdx: Function argument (can be NULL if none)
  */
 SYM_FUNC_START(asm_call_on_stack)
-SYM_INNER_LABEL(asm_call_irq_on_stack, SYM_L_GLOBAL)
 	/*
 	 * Save the frame pointer unconditionally. This allows the ORC
 	 * unwinder to handle the stack switch.
--- a/arch/x86/include/asm/idtentry.h
+++ b/arch/x86/include/asm/idtentry.h
@@ -187,23 +187,22 @@ static __always_inline void __##func(str
  * has to be done in the function body if necessary.
  */
 #define DEFINE_IDTENTRY_IRQ(func)					\
-static __always_inline void __##func(struct pt_regs *regs, u8 vector);	\
+static void __##func(struct pt_regs *regs, u32 vector);			\
 									\
 __visible noinstr void func(struct pt_regs *regs,			\
 			    unsigned long error_code)			\
 {									\
 	irqentry_state_t state = irqentry_enter(regs);			\
+	u32 vector = (u32)(u8)error_code;				\
 									\
 	instrumentation_begin();					\
-	irq_enter_rcu();						\
 	kvm_set_cpu_l1tf_flush_l1d();					\
-	__##func (regs, (u8)error_code);				\
-	irq_exit_rcu();							\
+	run_irq_on_irqstack_cond(__##func, regs, vector);		\
 	instrumentation_end();						\
 	irqentry_exit(regs, state);					\
 }									\
 									\
-static __always_inline void __##func(struct pt_regs *regs, u8 vector)
+static noinline void __##func(struct pt_regs *regs, u32 vector)
 
 /**
  * DECLARE_IDTENTRY_SYSVEC - Declare functions for system vector entry points
--- a/arch/x86/include/asm/irq_stack.h
+++ b/arch/x86/include/asm/irq_stack.h
@@ -168,14 +168,35 @@
 			      SYSVEC_CONSTRAINTS, regs);		\
 }
 
+/*
+ * As in ASM_CALL_SYSVEC above the clobbers force the compiler to store
+ * @regs and @vector in callee saved registers.
+ */
+#define ASM_CALL_IRQ							\
+	"call irq_enter_rcu				\n"		\
+	"movq	%[__arg1], %%rdi			\n"		\
+	"movl	%[__arg2], %%esi			\n"		\
+	"call %P[__func]				\n"		\
+	"call irq_exit_rcu				\n"
+
+#define IRQ_CONSTRAINTS	, [__arg1] "r" (regs), [__arg2] "r" (vector)
+
+#define run_irq_on_irqstack_cond(func, regs, vector)			\
+{									\
+	assert_function_type(func, void (*)(struct pt_regs *, u32));	\
+	assert_arg_type(regs, struct pt_regs *);			\
+	assert_arg_type(vector, u32);					\
+									\
+	call_on_irqstack_cond(func, regs, ASM_CALL_IRQ,			\
+			      IRQ_CONSTRAINTS, regs, vector);		\
+}
+
 static __always_inline bool irqstack_active(void)
 {
 	return __this_cpu_read(hardirq_stack_inuse);
 }
 
 void asm_call_on_stack(void *sp, void (*func)(void), void *arg);
-void asm_call_irq_on_stack(void *sp, void (*func)(struct irq_desc *desc),
-			   struct irq_desc *desc);
 
 static __always_inline void __run_on_irqstack(void (*func)(void))
 {
@@ -186,17 +207,6 @@ static __always_inline void __run_on_irq
 	__this_cpu_write(hardirq_stack_inuse, false);
 }
 
-static __always_inline void
-__run_irq_on_irqstack(void (*func)(struct irq_desc *desc),
-		      struct irq_desc *desc)
-{
-	void *tos = __this_cpu_read(hardirq_stack_ptr);
-
-	__this_cpu_write(hardirq_stack_inuse, true);
-	asm_call_irq_on_stack(tos, func, desc);
-	__this_cpu_write(hardirq_stack_inuse, false);
-}
-
 #else /* CONFIG_X86_64 */
 
 /* System vector handlers always run on the stack they interrupted. */
@@ -207,10 +217,16 @@ static __always_inline void
 	irq_exit_rcu();							\
 }
 
+/* Switches to the irq stack within func() */
+#define run_irq_on_irqstack_cond(func, regs, vector)			\
+{									\
+	irq_enter_rcu();						\
+	func(regs, vector);						\
+	irq_exit_rcu();							\
+}
+
 static inline bool irqstack_active(void) { return false; }
 static inline void __run_on_irqstack(void (*func)(void)) { }
-static inline void __run_irq_on_irqstack(void (*func)(struct irq_desc *desc),
-					 struct irq_desc *desc) { }
 #endif /* !CONFIG_X86_64 */
 
 static __always_inline bool irq_needs_irq_stack(struct pt_regs *regs)
@@ -234,16 +250,4 @@ static __always_inline void run_on_irqst
 		func();
 }
 
-static __always_inline void
-run_irq_on_irqstack_cond(void (*func)(struct irq_desc *desc), struct irq_desc *desc,
-			 struct pt_regs *regs)
-{
-	lockdep_assert_irqs_disabled();
-
-	if (irq_needs_irq_stack(regs))
-		__run_irq_on_irqstack(func, desc);
-	else
-		func(desc);
-}
-
 #endif
--- a/arch/x86/kernel/irq.c
+++ b/arch/x86/kernel/irq.c
@@ -227,7 +227,7 @@ static __always_inline void handle_irq(s
 				       struct pt_regs *regs)
 {
 	if (IS_ENABLED(CONFIG_X86_64))
-		run_irq_on_irqstack_cond(desc->handle_irq, desc, regs);
+		generic_handle_irq_desc(desc);
 	else
 		__handle_irq(desc, regs);
 }

[patch 08/12] x86/entry: Use run_sysvec_on_irqstack_cond() for XEN upcall

[Thomas Gleixner]

To avoid yet another macro implementation reuse the existing
run_sysvec_on_irqstack_cond() and move the set_irq_regs() handling into the
called function. Makes the code even simpler.

Signed-off-by: Thomas Gleixner <tglx@linutronix.de>
---
 arch/x86/entry/common.c |   19 ++++++-------------
 1 file changed, 6 insertions(+), 13 deletions(-)

--- a/arch/x86/entry/common.c
+++ b/arch/x86/entry/common.c
@@ -249,30 +249,23 @@ static __always_inline bool get_and_clea
 static __always_inline void restore_inhcall(bool inhcall) { }
 #endif
 
-static void __xen_pv_evtchn_do_upcall(void)
+static void __xen_pv_evtchn_do_upcall(struct pt_regs *regs)
 {
-	irq_enter_rcu();
+	struct pt_regs *old_regs = set_irq_regs(regs);
+
 	inc_irq_stat(irq_hv_callback_count);
 
 	xen_hvm_evtchn_do_upcall();
 
-	irq_exit_rcu();
+	set_irq_regs(old_regs);
 }
 
 __visible noinstr void xen_pv_evtchn_do_upcall(struct pt_regs *regs)
 {
-	struct pt_regs *old_regs;
+	irqentry_state_t state = irqentry_enter(regs);
 	bool inhcall;
-	irqentry_state_t state;
-
-	state = irqentry_enter(regs);
-	old_regs = set_irq_regs(regs);
 
-	instrumentation_begin();
-	run_on_irqstack_cond(__xen_pv_evtchn_do_upcall, regs);
-	instrumentation_end();
-
-	set_irq_regs(old_regs);
+	run_sysvec_on_irqstack_cond(__xen_pv_evtchn_do_upcall, regs);
 
 	inhcall = get_and_clear_inhcall();
 	if (inhcall && !WARN_ON_ONCE(state.exit_rcu)) {

Re: [patch 08/12] x86/entry: Use run_sysvec_on_irqstack_cond() for XEN upcall

[kernel test robot]

[-- Attachment #1: Type: text/plain, Size: 1508 bytes --]

Hi Thomas,

I love your patch! Perhaps something to improve:

[auto build test WARNING on tip/x86/asm]
[also build test WARNING on tip/master linus/master tip/x86/core v5.11-rc6 next-20210125]
[If your patch is applied to the wrong git tree, kindly drop us a note.
And when submitting patch, we suggest to use '--base' as documented in
https://git-scm.com/docs/git-format-patch]

url:    https://github.com/0day-ci/linux/commits/Thomas-Gleixner/x86-irq-64-Inline-irq-stack-switching/20210205-091059
base:   https://git.kernel.org/pub/scm/linux/kernel/git/tip/tip.git 5c99720b28381bb400d4f546734c34ddaf608761
config: x86_64-rhel (attached as .config)
compiler: gcc-9 (Debian 9.3.0-15) 9.3.0
reproduce (this is a W=1 build):
        # https://github.com/0day-ci/linux/commit/ab3144e8d40d2a97101655f99804baa009bcdc36
        git remote add linux-review https://github.com/0day-ci/linux
        git fetch --no-tags linux-review Thomas-Gleixner/x86-irq-64-Inline-irq-stack-switching/20210205-091059
        git checkout ab3144e8d40d2a97101655f99804baa009bcdc36
        # save the attached .config to linux build tree
        make W=1 ARCH=x86_64 

If you fix the issue, kindly add following tag as appropriate
Reported-by: kernel test robot <lkp@intel.com>

All warnings (new ones prefixed by >>):

>> arch/x86/entry/common.o: warning: objtool: xen_pv_evtchn_do_upcall()+0xa3: undefined stack state

---
0-DAY CI Kernel Test Service, Intel Corporation
https://lists.01.org/hyperkitty/list/kbuild-all@lists.01.org

[-- Attachment #2: .config.gz --]
[-- Type: application/gzip, Size: 45960 bytes --]

[patch 09/12] x86/softirq: Remove indirection in do_softirq_own_stack()

[Thomas Gleixner]

Use the new inline stack switching and remove the old ASM indirect call
implementation.

Signed-off-by: Thomas Gleixner <tglx@linutronix.de>
---
 arch/x86/entry/entry_64.S        |   39 -----------------------------
 arch/x86/include/asm/irq_stack.h |   52 ++++++++++++---------------------------
 arch/x86/kernel/irq_64.c         |    2 -
 3 files changed, 17 insertions(+), 76 deletions(-)

--- a/arch/x86/entry/entry_64.S
+++ b/arch/x86/entry/entry_64.S
@@ -756,45 +756,6 @@ SYM_CODE_START_LOCAL_NOALIGN(.Lbad_gs)
 SYM_CODE_END(.Lbad_gs)
 	.previous
 
-/*
- * rdi: New stack pointer points to the top word of the stack
- * rsi: Function pointer
- * rdx: Function argument (can be NULL if none)
- */
-SYM_FUNC_START(asm_call_on_stack)
-	/*
-	 * Save the frame pointer unconditionally. This allows the ORC
-	 * unwinder to handle the stack switch.
-	 */
-	pushq		%rbp
-	mov		%rsp, %rbp
-
-	/*
-	 * The unwinder relies on the word at the top of the new stack
-	 * page linking back to the previous RSP.
-	 */
-	mov		%rsp, (%rdi)
-	mov		%rdi, %rsp
-	/* Move the argument to the right place */
-	mov		%rdx, %rdi
-
-1:
-	.pushsection .discard.instr_begin
-	.long 1b - .
-	.popsection
-
-	CALL_NOSPEC	rsi
-
-2:
-	.pushsection .discard.instr_end
-	.long 2b - .
-	.popsection
-
-	/* Restore the previous stack pointer from RBP. */
-	leaveq
-	ret
-SYM_FUNC_END(asm_call_on_stack)
-
 #ifdef CONFIG_XEN_PV
 /*
  * A note on the "critical region" in our callback handler.
--- a/arch/x86/include/asm/irq_stack.h
+++ b/arch/x86/include/asm/irq_stack.h
@@ -191,20 +191,23 @@
 			      IRQ_CONSTRAINTS, regs, vector);		\
 }
 
-static __always_inline bool irqstack_active(void)
-{
-	return __this_cpu_read(hardirq_stack_inuse);
-}
-
-void asm_call_on_stack(void *sp, void (*func)(void), void *arg);
+#define ASM_CALL_SOFTIRQ						\
+	"call %P[__func]				\n"
 
-static __always_inline void __run_on_irqstack(void (*func)(void))
-{
-	void *tos = __this_cpu_read(hardirq_stack_ptr);
-
-	__this_cpu_write(hardirq_stack_inuse, true);
-	asm_call_on_stack(tos, func, NULL);
-	__this_cpu_write(hardirq_stack_inuse, false);
+/*
+ * Macro to invoke __do_softirq on the irq stack. Contrary to the above
+ * the only check which is necessary is whether the interrupt stack is
+ * in use already.
+ */
+#define run_softirq_on_irqstack_cond()					\
+{									\
+	if (__this_cpu_read(hardirq_stack_inuse)) {			\
+		__do_softirq();						\
+	} else {							\
+		__this_cpu_write(hardirq_stack_inuse, true);		\
+		__call_on_irqstack(__do_softirq, ASM_CALL_SOFTIRQ);	\
+		__this_cpu_write(hardirq_stack_inuse, false);		\
+	}								\
 }
 
 #else /* CONFIG_X86_64 */
@@ -225,29 +228,6 @@ static __always_inline void __run_on_irq
 	irq_exit_rcu();							\
 }
 
-static inline bool irqstack_active(void) { return false; }
-static inline void __run_on_irqstack(void (*func)(void)) { }
 #endif /* !CONFIG_X86_64 */
 
-static __always_inline bool irq_needs_irq_stack(struct pt_regs *regs)
-{
-	if (IS_ENABLED(CONFIG_X86_32))
-		return false;
-	if (!regs)
-		return !irqstack_active();
-	return !user_mode(regs) && !irqstack_active();
-}
-
-
-static __always_inline void run_on_irqstack_cond(void (*func)(void),
-						 struct pt_regs *regs)
-{
-	lockdep_assert_irqs_disabled();
-
-	if (irq_needs_irq_stack(regs))
-		__run_on_irqstack(func);
-	else
-		func();
-}
-
 #endif
--- a/arch/x86/kernel/irq_64.c
+++ b/arch/x86/kernel/irq_64.c
@@ -76,5 +76,5 @@ int irq_init_percpu_irqstack(unsigned in
 
 void do_softirq_own_stack(void)
 {
-	run_on_irqstack_cond(__do_softirq, NULL);
+	run_softirq_on_irqstack_cond();
 }

[patch 10/12] x86: Select CONFIG_HAVE_IRQ_EXIT_ON_IRQ_STACK

[Thomas Gleixner]

Now that all invocations of irq_exit_rcu() happen on the irq stack, turn on
CONFIG_HAVE_IRQ_EXIT_ON_IRQ_STACK which causes the core code to invoke
__do_softirq() directly without going through do_softirq_own_stack().

That means do_softirq_own_stack() is only invoked from task context which
means it can't be on the irq stack. Remove the conditional from
run_softirq_on_irqstack_cond() and rename the function accordingly.

Signed-off-by: Thomas Gleixner <tglx@linutronix.de>
---
 arch/x86/Kconfig                 |    1 +
 arch/x86/include/asm/irq_stack.h |   19 ++++++++-----------
 arch/x86/kernel/irq_64.c         |    2 +-
 3 files changed, 10 insertions(+), 12 deletions(-)

--- a/arch/x86/Kconfig
+++ b/arch/x86/Kconfig
@@ -187,6 +187,7 @@ config X86
 	select HAVE_HW_BREAKPOINT
 	select HAVE_IDE
 	select HAVE_IOREMAP_PROT
+	select HAVE_IRQ_EXIT_ON_IRQ_STACK	if X86_64
 	select HAVE_IRQ_TIME_ACCOUNTING
 	select HAVE_KERNEL_BZIP2
 	select HAVE_KERNEL_GZIP
--- a/arch/x86/include/asm/irq_stack.h
+++ b/arch/x86/include/asm/irq_stack.h
@@ -195,19 +195,16 @@
 	"call %P[__func]				\n"
 
 /*
- * Macro to invoke __do_softirq on the irq stack. Contrary to the above
- * the only check which is necessary is whether the interrupt stack is
- * in use already.
+ * Macro to invoke __do_softirq on the irq stack. This is only called from
+ * task context when bottom halfs are about to be reenabled and soft
+ * interrupts are pending to be processed. The interrupt stack cannot be in
+ * use here.
  */
-#define run_softirq_on_irqstack_cond()					\
+#define run_softirq_on_irqstack()					\
 {									\
-	if (__this_cpu_read(hardirq_stack_inuse)) {			\
-		__do_softirq();						\
-	} else {							\
-		__this_cpu_write(hardirq_stack_inuse, true);		\
-		__call_on_irqstack(__do_softirq, ASM_CALL_SOFTIRQ);	\
-		__this_cpu_write(hardirq_stack_inuse, false);		\
-	}								\
+	__this_cpu_write(hardirq_stack_inuse, true);			\
+	__call_on_irqstack(__do_softirq, ASM_CALL_SOFTIRQ);		\
+	__this_cpu_write(hardirq_stack_inuse, false);			\
 }
 
 #else /* CONFIG_X86_64 */
--- a/arch/x86/kernel/irq_64.c
+++ b/arch/x86/kernel/irq_64.c
@@ -76,5 +76,5 @@ int irq_init_percpu_irqstack(unsigned in
 
 void do_softirq_own_stack(void)
 {
-	run_softirq_on_irqstack_cond();
+	run_softirq_on_irqstack();
 }

[patch 11/12] softirq: Allow inlining do_softirq_own_stack()

[Thomas Gleixner]

The function to switch to the irq stack on x86 is now minimal and there is
only a single caller. Allow the stack switch to be inlined.

Signed-off-by: Thomas Gleixner <tglx@linutronix.de>
---
 include/linux/interrupt.h |    2 ++
 kernel/softirq.c          |    4 ++++
 2 files changed, 6 insertions(+)

--- a/include/linux/interrupt.h
+++ b/include/linux/interrupt.h
@@ -570,7 +570,9 @@ asmlinkage void do_softirq(void);
 asmlinkage void __do_softirq(void);
 
 #ifdef __ARCH_HAS_DO_SOFTIRQ
+# ifndef __ARCH_HAS_DO_SOFTIRQ_INLINE
 void do_softirq_own_stack(void);
+# endif
 #else
 static inline void do_softirq_own_stack(void)
 {
--- a/kernel/softirq.c
+++ b/kernel/softirq.c
@@ -26,6 +26,10 @@
 #include <linux/tick.h>
 #include <linux/irq.h>
 
+#ifdef __ARCH_HAS_DO_SOFTIRQ_INLINE
+# include <asm/irq_stack.h>
+#endif
+
 #define CREATE_TRACE_POINTS
 #include <trace/events/irq.h>
 

Re: [patch 11/12] softirq: Allow inlining do_softirq_own_stack()

[Lai Jiangshan]

On Fri, Feb 5, 2021 at 10:04 AM Thomas Gleixner <tglx@linutronix.de> wrote:
>
> The function to switch to the irq stack on x86 is now minimal and there is
> only a single caller. Allow the stack switch to be inlined.
>
> Signed-off-by: Thomas Gleixner <tglx@linutronix.de>
> ---
>  include/linux/interrupt.h |    2 ++
>  kernel/softirq.c          |    4 ++++
>  2 files changed, 6 insertions(+)
>
> --- a/include/linux/interrupt.h
> +++ b/include/linux/interrupt.h
> @@ -570,7 +570,9 @@ asmlinkage void do_softirq(void);
>  asmlinkage void __do_softirq(void);
>
>  #ifdef __ARCH_HAS_DO_SOFTIRQ
> +# ifndef __ARCH_HAS_DO_SOFTIRQ_INLINE
>  void do_softirq_own_stack(void);
> +# endif
>  #else
>  static inline void do_softirq_own_stack(void)
>  {

Hello

This patch and the next patch have three "#if[n]def" with
__ARCH_HAS_DO_SOFTIRQ_INLINE and this one is nested in
__ARCH_HAS_DO_SOFTIRQ.

I wonder if we can use __ARCH_HAS_DO_SOFTIRQ only.

For example, we can move "void do_softirq_own_stack(void);" to around
the code where __ARCH_HAS_DO_SOFTIRQ are defined in very ARCHs.
(And for x86, do_softirq_own_stack() is a macro instead of function
declaration as next patch shows)

Thanks
Lai

> --- a/kernel/softirq.c
> +++ b/kernel/softirq.c
> @@ -26,6 +26,10 @@
>  #include <linux/tick.h>
>  #include <linux/irq.h>
>
> +#ifdef __ARCH_HAS_DO_SOFTIRQ_INLINE
> +# include <asm/irq_stack.h>
> +#endif
> +
>  #define CREATE_TRACE_POINTS
>  #include <trace/events/irq.h>
>
>

Re: [patch 11/12] softirq: Allow inlining do_softirq_own_stack()

[Thomas Gleixner]

On Fri, Feb 05 2021 at 18:14, Lai Jiangshan wrote:
> On Fri, Feb 5, 2021 at 10:04 AM Thomas Gleixner <tglx@linutronix.de> wrote:
>>  static inline void do_softirq_own_stack(void)
>>  {
>
> Hello
>
> This patch and the next patch have three "#if[n]def" with
> __ARCH_HAS_DO_SOFTIRQ_INLINE and this one is nested in
> __ARCH_HAS_DO_SOFTIRQ.
>
> I wonder if we can use __ARCH_HAS_DO_SOFTIRQ only.
>
> For example, we can move "void do_softirq_own_stack(void);" to around
> the code where __ARCH_HAS_DO_SOFTIRQ are defined in very ARCHs.
> (And for x86, do_softirq_own_stack() is a macro instead of function
> declaration as next patch shows)

We can do that as well. No strong preference.

Thanks,

        tglx

Re: [patch 11/12] softirq: Allow inlining do_softirq_own_stack()

[Thomas Gleixner]

Lai,

On Fri, Feb 05 2021 at 12:38, Thomas Gleixner wrote:
> On Fri, Feb 05 2021 at 18:14, Lai Jiangshan wrote:
>> On Fri, Feb 5, 2021 at 10:04 AM Thomas Gleixner <tglx@linutronix.de> wrote:
>>>  static inline void do_softirq_own_stack(void)
>>>  {
>>
>> Hello
>>
>> This patch and the next patch have three "#if[n]def" with
>> __ARCH_HAS_DO_SOFTIRQ_INLINE and this one is nested in
>> __ARCH_HAS_DO_SOFTIRQ.
>>
>> I wonder if we can use __ARCH_HAS_DO_SOFTIRQ only.
>>
>> For example, we can move "void do_softirq_own_stack(void);" to around
>> the code where __ARCH_HAS_DO_SOFTIRQ are defined in very ARCHs.
>> (And for x86, do_softirq_own_stack() is a macro instead of function
>> declaration as next patch shows)
>
> We can do that as well. No strong preference.

actually it's not that trivial. It ends up in include hell and then
pulls the whole irq stack macro mess into every file which includes
interrupt.h.

I've moved the default function prototype and the stub into
asm-generic/softirq_stack.h and let x86 override it.

Thanks,

        tglx

[patch 12/12] x86/softirq/64: Inline do_softirq_own_stack()

[Thomas Gleixner]

There is no reason to have this as a seperate function for a single caller.

Signed-off-by: Thomas Gleixner <tglx@linutronix.de>
---
 arch/x86/include/asm/irq.h       |    3 +++
 arch/x86/include/asm/irq_stack.h |    3 +--
 arch/x86/kernel/irq_64.c         |    5 -----
 3 files changed, 4 insertions(+), 7 deletions(-)

--- a/arch/x86/include/asm/irq.h
+++ b/arch/x86/include/asm/irq.h
@@ -26,6 +26,9 @@ static inline int irq_canonicalize(int i
 extern int irq_init_percpu_irqstack(unsigned int cpu);
 
 #define __ARCH_HAS_DO_SOFTIRQ
+#ifdef CONFIG_X86_64
+#define __ARCH_HAS_DO_SOFTIRQ_INLINE
+#endif
 
 struct irq_desc;
 
--- a/arch/x86/include/asm/irq_stack.h
+++ b/arch/x86/include/asm/irq_stack.h
@@ -200,7 +200,7 @@
  * interrupts are pending to be processed. The interrupt stack cannot be in
  * use here.
  */
-#define run_softirq_on_irqstack()					\
+#define do_softirq_own_stack()						\
 {									\
 	__this_cpu_write(hardirq_stack_inuse, true);			\
 	__call_on_irqstack(__do_softirq, ASM_CALL_SOFTIRQ);		\
@@ -208,7 +208,6 @@
 }
 
 #else /* CONFIG_X86_64 */
-
 /* System vector handlers always run on the stack they interrupted. */
 #define run_sysvec_on_irqstack_cond(func, regs)				\
 {									\
--- a/arch/x86/kernel/irq_64.c
+++ b/arch/x86/kernel/irq_64.c
@@ -73,8 +73,3 @@ int irq_init_percpu_irqstack(unsigned in
 		return 0;
 	return map_irq_stack(cpu);
 }
-
-void do_softirq_own_stack(void)
-{
-	run_softirq_on_irqstack();
-}

Re: [patch 00/12] x86/irq/64: Inline irq stack switching

[Kees Cook]

On Thu, Feb 04, 2021 at 09:49:03PM +0100, Thomas Gleixner wrote:
>   - The fact that the stack switching code ended up being an easy to find
>     exploit gadget.

Thanks for cleaning this up! It looks much much safer now. :)

> The patches have been tested with gcc8, gcc10 and clang-13 (fresh from
> git). The difference between the output of these compilers is minimal.
> gcc8 being slightly worse due to stupid register selection and random
> NOPs injected.

Awesome. Please consider the series:

Reviewed-by: Kees Cook <keescook@chromium.org>

-- 
Kees Cook

Re: [patch 05/12] x86/irq: Provide macro for inlining irq stack switching

[Uros Bizjak]

> The effort to make the ASM entry code slim and unified moved the irq stack
> switching out of the low level ASM code so that the whole return from
> interrupt work and state handling can be done in C and the ASM code just
> handles the low level details of entry and exit.
>
> This ended up being a suboptimal implementation for various reasons
> (including tooling). The main pain points are:
>
>  - The indirect call which is expensive thanks to retpoline
>
>  - The inability to stay on the irq stack for softirq processing on return
>    from interrupt
>
>  - The fact that the stack switching code ends up being an easy to target
>    exploit gadget.
>
> Prepare for inlining the stack switching logic into the C entry points by
> providing a ASM macro which contains the guts of the switching mechanism:
>
>   1) Store RSP at the top of the irq stack
>   2) Switch RSP to the irq stack
>   3) Invoke code
>   4) Pop the original RSP back
>
> Document the unholy asm() logic while at it to reduce the amount of head
> scratching required a half year from now.

#define __call_on_irqstack(func, asm_call, constr...) \
+{ \
+ register void *tos asm("r11"); \
+ \
+ tos = ((void *)__this_cpu_read(hardirq_stack_ptr)); \
+ \
+ asm_inline volatile( \
+ "movq %%rsp, (%[__tos]) \n" \
+ "movq %[__tos], %%rsp \n" \
+ \
+ asm_call \
+ \
+ "popq %%rsp \n" \
+ \
+ : "+r" (tos) IRQSTACK_CALL_CONSTRAINT \

Please note that GCC documents "U" register constraint that can be
used here instead of declaring hard register in the variable
declaration:

    'U'
         The call-clobbered integer registers.

+ : [__func] "i" (func), [__tos] "r" (tos) constr \

There is no need to declare "tos" as read operand again, it is already
declared above as readwrite (+) operand.

Considering that (according to the above documentation) it is
necessary to list all input registers that pass function arguments,
the compiler is free to allocate any remaining register from "U"
register class, not only r11. Using an earlyclobber modifier prevents
the compiler from allocating a register that carries input argument,
so:

: [__tos] "+&U" (tos) IRQSTACK_CALL_CONSTRAINT \
: [__func] "i" (func) constr \

could be used.

Also note that functions with variable arguments pass information
about the number of vector registers used in %rax, so %rax should be
listed as input argument in this case. But this should be of no issue
here.

Uros.

+ : "cc", "rax", "rcx", "rdx", "rsi", "rdi", "r8", "r9", "r10", \
+  "memory" \
+ ); \

Re: [patch 05/12] x86/irq: Provide macro for inlining irq stack switching

[Thomas Gleixner]

Uros,

On Fri, Feb 05 2021 at 12:03, Uros Bizjak wrote:

can you please fix your mail client to generate at least the 'In-Reply-to:'
header? Lacking that header breaks threading on lore:

        https://lore.kernel.org/lkml/20210204204903.350275743@linutronix.de/

Your mail is missing there. Ideally it also emits 'References'.

> #define __call_on_irqstack(func, asm_call, constr...) \
> +{ \
> + register void *tos asm("r11"); \
> + \
> + tos = ((void *)__this_cpu_read(hardirq_stack_ptr)); \
> + \
> + asm_inline volatile( \
> + "movq %%rsp, (%[__tos]) \n" \
> + "movq %[__tos], %%rsp \n" \
> + \
> + asm_call \
> + \
> + "popq %%rsp \n" \
> + \
> + : "+r" (tos) IRQSTACK_CALL_CONSTRAINT \
>
> Please note that GCC documents "U" register constraint that can be
> used here instead of declaring hard register in the variable
> declaration:
>
>     'U'
>          The call-clobbered integer registers.

That's not really helpful because clang does not support 'U'.

> + : [__func] "i" (func), [__tos] "r" (tos) constr \
>
> There is no need to declare "tos" as read operand again, it is already
> declared above as readwrite (+) operand.

It makes clang builds fail.

> Considering that (according to the above documentation) it is
> necessary to list all input registers that pass function arguments,
> the compiler is free to allocate any remaining register from "U"
> register class, not only r11. Using an earlyclobber modifier prevents
> the compiler from allocating a register that carries input argument,
> so:
>
> : [__tos] "+&U" (tos) IRQSTACK_CALL_CONSTRAINT \
> : [__func] "i" (func) constr \
>
> could be used.

See above. Without the U constraint we can't rely on the compiler to do
the right thing without the explicit register asm("r11"); And even with
'U' we need to enforce that there is only one U register left to use.

The problem is that the compiler does not know about the call. So we
need to ensure via the clobbers and input/output arguments that it can't
use any of the callee clobbered registers accross the inline asm.

With

        void *tos = this_cpu_read(...);

        : "cc", .... "r9", "r10"

the compiler could still use "r11" for some other stuff and stick tos
into a callee saved register, e.g. r15. If the called function then
clobbers "r11" everything goes south.

There is no point in being extra smart here. The functions have no
register pressure as they are small so enforcing the register allocation
is not restricting the compiler freedom to much. But it ensures that the
compiler can't do anything subtly wrong which would end up being a hard
to debug disaster.

> Also note that functions with variable arguments pass information
> about the number of vector registers used in %rax, so %rax should be
> listed as input argument in this case. But this should be of no issue
> here.

That's really irrelevant as it's a very narrow use case for functions
with 0..2 arguments.

Thanks,

        tglx