* [patch 01/16] x86/entry: Provide IDTENTRY_IST
2020-02-25 22:33 [patch 00/16] x86/entry: Consolidation - Part IV Thomas Gleixner
@ 2020-02-25 22:33 ` Thomas Gleixner
2020-02-25 22:33 ` [patch 02/16] x86/entry: Convert Machine Check to IDTENTRY_IST Thomas Gleixner
` (14 subsequent siblings)
15 siblings, 0 replies; 31+ messages in thread
From: Thomas Gleixner @ 2020-02-25 22:33 UTC (permalink / raw)
To: LKML
Cc: x86, Steven Rostedt, Brian Gerst, Juergen Gross, Paolo Bonzini,
Arnd Bergmann
Same as IDTENTRY but for exceptions which run on Interrupt STacks (IST) on
64bit. For 32bit this maps to IDTENTRY.
There are 3 variants which will be used:
IDTENTRY_MCE
IDTENTRY_DB
IDTENTRY_NMI
These map to IDTENTRY_IST, but only the MCE and DB variants are emitting
ASM code as the NMI entry needs hand crafted ASM still.
The function defines do not contain any idtenter/exit calls as these
exceptions need special treatment.
Signed-off-by: Thomas Gleixner <tglx@linutronix.de>
---
arch/x86/include/asm/idtentry.h | 70 ++++++++++++++++++++++++++++++++++++++++
1 file changed, 70 insertions(+)
--- a/arch/x86/include/asm/idtentry.h
+++ b/arch/x86/include/asm/idtentry.h
@@ -111,6 +111,58 @@ NOKPROBE_SYMBOL(func); \
static __always_inline void __##func(struct pt_regs *regs, \
unsigned long error_code)
+#ifdef CONFIG_X86_64
+/**
+ * DECLARE_IDTENTRY_IST - Declare functions for IST handling IDT entry points
+ * @vector: Vector number (ignored for C)
+ * @func: Function name of the entry point
+ *
+ * Declares three functions:
+ * - The ASM entry point: asm_##func
+ * - The XEN PV trap entry point: xen_##func (maybe unused)
+ * - The C handler called from the ASM entry point
+ */
+#define DECLARE_IDTENTRY_IST(vector, func) \
+ asmlinkage void asm_##func(void); \
+ asmlinkage void xen_asm_##func(void); \
+ __visible void func(struct pt_regs *regs)
+
+/**
+ * DEFINE_IDTENTRY_IST - Emit code for IST entry points
+ * @func: Function name of the entry point
+ *
+ * This provides two entry points:
+ * - The real IST based entry
+ * - The regular stack based entry invoked when coming from user mode
+ * or XEN_PV
+ */
+#define DEFINE_IDTENTRY_IST(func) \
+static __always_inline void __##func(struct pt_regs *regs); \
+ \
+__visible notrace void func(struct pt_regs *regs) \
+{ \
+ __##func (regs); \
+} \
+NOKPROBE_SYMBOL(func); \
+ \
+static __always_inline void __##func(struct pt_regs *regs)
+
+#else /* CONFIG_X86_64 */
+/* Maps to a regular IDTENTRY on 32bit for now */
+# define DECLARE_IDTENTRY_IST DECLARE_IDTENTRY
+# define DEFINE_IDTENTRY_IST DEFINE_IDTENTRY
+#endif /* !CONFIG_X86_64 */
+
+/* C-Code mapping */
+#define DECLARE_IDTENTRY_MCE DECLARE_IDTENTRY_IST
+#define DEFINE_IDTENTRY_MCE DEFINE_IDTENTRY_IST
+
+#define DECLARE_IDTENTRY_NMI DECLARE_IDTENTRY_IST
+#define DEFINE_IDTENTRY_NMI DEFINE_IDTENTRY_IST
+
+#define DECLARE_IDTENTRY_DEBUG DECLARE_IDTENTRY_IST
+#define DEFINE_IDTENTRY_DEBUG DEFINE_IDTENTRY_IST
+
#else /* !__ASSEMBLY__ */
/* Defines for ASM code to construct the IDT entries */
@@ -123,6 +175,24 @@ static __always_inline void __##func(str
/* Special case for 32bit IRET 'trap'. Do not emit ASM code */
#define DECLARE_IDTENTRY_SW(vector, func)
+#ifdef CONFIG_X86_64
+# define DECLARE_IDTENTRY_MCE(vector, func) \
+ idtentry_mce_db vector asm_##func func
+
+# define DECLARE_IDTENTRY_DEBUG(vector, func) \
+ idtentry_mce_db vector asm_##func func
+
+#else
+# define DECLARE_IDTENTRY_MCE(vector, func) \
+ DECLARE_IDTENTRY(vector, func)
+
+# define DECLARE_IDTENTRY_DEBUG(vector, func) \
+ DECLARE_IDTENTRY(vector, func)
+#endif
+
+/* No ASM code emitted for NMI */
+# define DECLARE_IDTENTRY_NMI(vector, func)
+
#endif /* __ASSEMBLY__ */
/* Simple exception entries: */
^ permalink raw reply [flat|nested] 31+ messages in thread
* [patch 02/16] x86/entry: Convert Machine Check to IDTENTRY_IST
2020-02-25 22:33 [patch 00/16] x86/entry: Consolidation - Part IV Thomas Gleixner
2020-02-25 22:33 ` [patch 01/16] x86/entry: Provide IDTENTRY_IST Thomas Gleixner
@ 2020-02-25 22:33 ` Thomas Gleixner
2020-02-25 22:33 ` [patch 03/16] x86/idtentry: Provide IDTENTRY_XEN for XEN/PV Thomas Gleixner
` (13 subsequent siblings)
15 siblings, 0 replies; 31+ messages in thread
From: Thomas Gleixner @ 2020-02-25 22:33 UTC (permalink / raw)
To: LKML
Cc: x86, Steven Rostedt, Brian Gerst, Juergen Gross, Paolo Bonzini,
Arnd Bergmann
Convert #MC to IDTENTRY_MCE:
- Implement the C entry points with DEFINE_IDTENTRY_MCE
- Emit the ASM stub with DECLARE_IDTENTRY_MCE
- Remove the ASM idtentry in 64bit
- Remove the open coded ASM entry code in 32bit
- Fixup the XEN/PV code
- Remove the old prototyoes
- Remove the error code from *machine_check_vector() as
it is always 0 and not used by any of the functions
it can point to. Fixup all the functions as well.
No functional change.
Signed-off-by: Thomas Gleixner <tglx@linutronix.de>
---
arch/x86/entry/entry_32.S | 9 ---------
arch/x86/entry/entry_64.S | 3 ---
arch/x86/include/asm/idtentry.h | 5 +++++
arch/x86/include/asm/mce.h | 2 +-
arch/x86/include/asm/traps.h | 7 -------
arch/x86/kernel/cpu/mce/core.c | 14 ++++++--------
arch/x86/kernel/cpu/mce/inject.c | 4 ++--
arch/x86/kernel/cpu/mce/internal.h | 2 +-
arch/x86/kernel/cpu/mce/p5.c | 2 +-
arch/x86/kernel/cpu/mce/winchip.c | 2 +-
arch/x86/kernel/idt.c | 10 +++++-----
arch/x86/kvm/vmx/vmx.c | 2 +-
arch/x86/xen/enlighten_pv.c | 2 +-
arch/x86/xen/xen-asm_64.S | 2 +-
14 files changed, 25 insertions(+), 41 deletions(-)
--- a/arch/x86/entry/entry_32.S
+++ b/arch/x86/entry/entry_32.S
@@ -1306,15 +1306,6 @@ SYM_CODE_START(native_iret)
SYM_CODE_END(native_iret)
#endif
-#ifdef CONFIG_X86_MCE
-SYM_CODE_START(machine_check)
- ASM_CLAC
- pushl $0
- pushl $do_mce
- jmp common_exception
-SYM_CODE_END(machine_check)
-#endif
-
#ifdef CONFIG_XEN_PV
SYM_FUNC_START(xen_hypervisor_callback)
/*
--- a/arch/x86/entry/entry_64.S
+++ b/arch/x86/entry/entry_64.S
@@ -1066,9 +1066,6 @@ idtentry X86_TRAP_PF page_fault do_pag
idtentry X86_TRAP_PF async_page_fault do_async_page_fault has_error_code=1
#endif
-#ifdef CONFIG_X86_MCE
-idtentry_mce_db X86_TRAP_MCE machine_check do_mce
-#endif
idtentry_mce_db X86_TRAP_DB debug do_debug
idtentry_df X86_TRAP_DF double_fault do_double_fault
--- a/arch/x86/include/asm/idtentry.h
+++ b/arch/x86/include/asm/idtentry.h
@@ -217,4 +217,9 @@ DECLARE_IDTENTRY_ERRORCODE(X86_TRAP_SS,
DECLARE_IDTENTRY_ERRORCODE(X86_TRAP_GP, exc_general_protection);
DECLARE_IDTENTRY_ERRORCODE(X86_TRAP_AC, exc_alignment_check);
+#ifdef CONFIG_X86_MCE
+/* Machine check */
+DECLARE_IDTENTRY_MCE(X86_TRAP_MC, exc_machine_check);
+#endif
+
#endif
--- a/arch/x86/include/asm/mce.h
+++ b/arch/x86/include/asm/mce.h
@@ -238,7 +238,7 @@ extern void mce_disable_bank(int bank);
/*
* Exception handler
*/
-void do_machine_check(struct pt_regs *, long);
+void do_machine_check(struct pt_regs *pt_regs);
/*
* Threshold handler
--- a/arch/x86/include/asm/traps.h
+++ b/arch/x86/include/asm/traps.h
@@ -18,18 +18,12 @@ asmlinkage void double_fault(void);
#endif
asmlinkage void page_fault(void);
asmlinkage void async_page_fault(void);
-#ifdef CONFIG_X86_MCE
-asmlinkage void machine_check(void);
-#endif /* CONFIG_X86_MCE */
#if defined(CONFIG_X86_64) && defined(CONFIG_XEN_PV)
asmlinkage void xen_xennmi(void);
asmlinkage void xen_xendebug(void);
asmlinkage void xen_double_fault(void);
asmlinkage void xen_page_fault(void);
-#ifdef CONFIG_X86_MCE
-asmlinkage void xen_machine_check(void);
-#endif /* CONFIG_X86_MCE */
#endif
dotraplinkage void do_debug(struct pt_regs *regs, long error_code);
@@ -38,7 +32,6 @@ dotraplinkage void do_nmi(struct pt_regs
dotraplinkage void do_double_fault(struct pt_regs *regs, long error_code, unsigned long cr2);
#endif
dotraplinkage void do_page_fault(struct pt_regs *regs, unsigned long error_code, unsigned long address);
-dotraplinkage void do_mce(struct pt_regs *regs, long error_code);
#ifdef CONFIG_X86_64
asmlinkage __visible notrace struct pt_regs *sync_regs(struct pt_regs *eregs);
--- a/arch/x86/kernel/cpu/mce/core.c
+++ b/arch/x86/kernel/cpu/mce/core.c
@@ -1220,7 +1220,7 @@ static void __mc_scan_banks(struct mce *
* backing the user stack, tracing that reads the user stack will cause
* potentially infinite recursion.
*/
-void notrace do_machine_check(struct pt_regs *regs, long error_code)
+void notrace do_machine_check(struct pt_regs *regs)
{
DECLARE_BITMAP(valid_banks, MAX_NR_BANKS);
DECLARE_BITMAP(toclear, MAX_NR_BANKS);
@@ -1358,7 +1358,7 @@ void notrace do_machine_check(struct pt_
local_irq_disable();
ist_end_non_atomic();
} else {
- if (!fixup_exception(regs, X86_TRAP_MC, error_code, 0))
+ if (!fixup_exception(regs, X86_TRAP_MC, 0, 0))
mce_panic("Failed kernel mode recovery", &m, msg);
}
@@ -1889,21 +1889,19 @@ bool filter_mce(struct mce *m)
}
/* Handle unconfigured int18 (should never happen) */
-static void unexpected_machine_check(struct pt_regs *regs, long error_code)
+static void unexpected_machine_check(struct pt_regs *regs)
{
pr_err("CPU#%d: Unexpected int18 (Machine Check)\n",
smp_processor_id());
}
/* Call the installed machine check handler for this CPU setup. */
-void (*machine_check_vector)(struct pt_regs *, long error_code) =
- unexpected_machine_check;
+void (*machine_check_vector)(struct pt_regs *) = unexpected_machine_check;
-dotraplinkage notrace void do_mce(struct pt_regs *regs, long error_code)
+DEFINE_IDTENTRY_MCE(exc_machine_check)
{
- machine_check_vector(regs, error_code);
+ machine_check_vector(regs);
}
-NOKPROBE_SYMBOL(do_mce);
/*
* Called for each booted CPU to set up machine checks.
--- a/arch/x86/kernel/cpu/mce/inject.c
+++ b/arch/x86/kernel/cpu/mce/inject.c
@@ -146,9 +146,9 @@ static void raise_exception(struct mce *
regs.cs = m->cs;
pregs = ®s;
}
- /* in mcheck exeception handler, irq will be disabled */
+ /* do_machine_check() expects interrupts disabled -- at least */
local_irq_save(flags);
- do_machine_check(pregs, 0);
+ do_machine_check(pregs);
local_irq_restore(flags);
m->finished = 0;
}
--- a/arch/x86/kernel/cpu/mce/internal.h
+++ b/arch/x86/kernel/cpu/mce/internal.h
@@ -9,7 +9,7 @@
#include <asm/mce.h>
/* Pointer to the installed machine check handler for this CPU setup. */
-extern void (*machine_check_vector)(struct pt_regs *, long error_code);
+extern void (*machine_check_vector)(struct pt_regs *);
enum severity_level {
MCE_NO_SEVERITY,
--- a/arch/x86/kernel/cpu/mce/p5.c
+++ b/arch/x86/kernel/cpu/mce/p5.c
@@ -20,7 +20,7 @@
int mce_p5_enabled __read_mostly;
/* Machine check handler for Pentium class Intel CPUs: */
-static void pentium_machine_check(struct pt_regs *regs, long error_code)
+static void pentium_machine_check(struct pt_regs *regs)
{
u32 loaddr, hi, lotype;
--- a/arch/x86/kernel/cpu/mce/winchip.c
+++ b/arch/x86/kernel/cpu/mce/winchip.c
@@ -16,7 +16,7 @@
#include "internal.h"
/* Machine check handler for WinChip C6: */
-static void winchip_machine_check(struct pt_regs *regs, long error_code)
+static void winchip_machine_check(struct pt_regs *regs)
{
ist_enter(regs);
--- a/arch/x86/kernel/idt.c
+++ b/arch/x86/kernel/idt.c
@@ -93,7 +93,7 @@ static const __initconst struct idt_data
INTG(X86_TRAP_DB, debug),
#ifdef CONFIG_X86_MCE
- INTG(X86_TRAP_MC, &machine_check),
+ INTG(X86_TRAP_MC, asm_exc_machine_check),
#endif
SYSG(X86_TRAP_OF, asm_exc_overflow),
@@ -182,11 +182,11 @@ gate_desc debug_idt_table[IDT_ENTRIES] _
* cpu_init() when the TSS has been initialized.
*/
static const __initconst struct idt_data ist_idts[] = {
- ISTG(X86_TRAP_DB, debug, IST_INDEX_DB),
- ISTG(X86_TRAP_NMI, nmi, IST_INDEX_NMI),
- ISTG(X86_TRAP_DF, double_fault, IST_INDEX_DF),
+ ISTG(X86_TRAP_DB, debug, IST_INDEX_DB),
+ ISTG(X86_TRAP_NMI, nmi, IST_INDEX_NMI),
+ ISTG(X86_TRAP_DF, double_fault, IST_INDEX_DF),
#ifdef CONFIG_X86_MCE
- ISTG(X86_TRAP_MC, &machine_check, IST_INDEX_MCE),
+ ISTG(X86_TRAP_MC, asm_exc_machine_check, IST_INDEX_MCE),
#endif
};
--- a/arch/x86/kvm/vmx/vmx.c
+++ b/arch/x86/kvm/vmx/vmx.c
@@ -4554,7 +4554,7 @@ static void kvm_machine_check(void)
.flags = X86_EFLAGS_IF,
};
- do_machine_check(®s, 0);
+ do_machine_check(®s);
#endif
}
--- a/arch/x86/xen/enlighten_pv.c
+++ b/arch/x86/xen/enlighten_pv.c
@@ -611,7 +611,7 @@ static struct trap_array_entry trap_arra
{ debug, xen_xendebug, true },
{ double_fault, xen_double_fault, true },
#ifdef CONFIG_X86_MCE
- { machine_check, xen_machine_check, true },
+ TRAP_ENTRY(exc_machine_check, true ),
#endif
{ nmi, xen_xennmi, true },
TRAP_ENTRY(exc_int3, false ),
--- a/arch/x86/xen/xen-asm_64.S
+++ b/arch/x86/xen/xen-asm_64.S
@@ -48,7 +48,7 @@ xen_pv_trap asm_exc_spurious_interrupt_b
xen_pv_trap asm_exc_coprocessor_error
xen_pv_trap asm_exc_alignment_check
#ifdef CONFIG_X86_MCE
-xen_pv_trap machine_check
+xen_pv_trap asm_exc_machine_check
#endif /* CONFIG_X86_MCE */
xen_pv_trap asm_exc_simd_coprocessor_error
#ifdef CONFIG_IA32_EMULATION
^ permalink raw reply [flat|nested] 31+ messages in thread
* [patch 03/16] x86/idtentry: Provide IDTENTRY_XEN for XEN/PV
2020-02-25 22:33 [patch 00/16] x86/entry: Consolidation - Part IV Thomas Gleixner
2020-02-25 22:33 ` [patch 01/16] x86/entry: Provide IDTENTRY_IST Thomas Gleixner
2020-02-25 22:33 ` [patch 02/16] x86/entry: Convert Machine Check to IDTENTRY_IST Thomas Gleixner
@ 2020-02-25 22:33 ` Thomas Gleixner
2020-02-25 22:33 ` [patch 04/16] x86/entry: Convert NMI to IDTENTRY_NMI Thomas Gleixner
` (12 subsequent siblings)
15 siblings, 0 replies; 31+ messages in thread
From: Thomas Gleixner @ 2020-02-25 22:33 UTC (permalink / raw)
To: LKML
Cc: x86, Steven Rostedt, Brian Gerst, Juergen Gross, Paolo Bonzini,
Arnd Bergmann
XEN/PV has special wrappers for NMI and DB exceptions. They redirect these
exceptions through regular IDTENTRY points. Provide the necessary IDTENTRY
macros to make this work
Signed-off-by: Thomas Gleixner <tglx@linutronix.de>
---
arch/x86/include/asm/idtentry.h | 18 +++++++++++++++++-
1 file changed, 17 insertions(+), 1 deletion(-)
--- a/arch/x86/include/asm/idtentry.h
+++ b/arch/x86/include/asm/idtentry.h
@@ -163,6 +163,18 @@ static __always_inline void __##func(str
#define DECLARE_IDTENTRY_DEBUG DECLARE_IDTENTRY_IST
#define DEFINE_IDTENTRY_DEBUG DEFINE_IDTENTRY_IST
+/**
+ * DECLARE_IDTENTRY_XEN - Declare functions for XEN redirect IDT entry points
+ * @vector: Vector number (ignored for C)
+ * @func: Function name of the entry point
+ *
+ * Used for xennmi and xendebug redirections. No DEFINE as this is all
+ * indirection magic.
+ */
+#define DECLARE_IDTENTRY_XEN(vector, func) \
+ asmlinkage void xen_asm_exc_xen##func(void); \
+ asmlinkage void asm_exc_xen##func(void)
+
#else /* !__ASSEMBLY__ */
/* Defines for ASM code to construct the IDT entries */
@@ -191,7 +203,11 @@ static __always_inline void __##func(str
#endif
/* No ASM code emitted for NMI */
-# define DECLARE_IDTENTRY_NMI(vector, func)
+#define DECLARE_IDTENTRY_NMI(vector, func)
+
+/* XEN NMI and DB wrapper */
+#define DECLARE_IDTENTRY_XEN(vector, func) \
+ idtentry vector asm_exc_xen##func exc_##func has_error_code=0
#endif /* __ASSEMBLY__ */
^ permalink raw reply [flat|nested] 31+ messages in thread
* [patch 04/16] x86/entry: Convert NMI to IDTENTRY_NMI
2020-02-25 22:33 [patch 00/16] x86/entry: Consolidation - Part IV Thomas Gleixner
` (2 preceding siblings ...)
2020-02-25 22:33 ` [patch 03/16] x86/idtentry: Provide IDTENTRY_XEN for XEN/PV Thomas Gleixner
@ 2020-02-25 22:33 ` Thomas Gleixner
2020-02-25 22:33 ` [patch 05/16] x86/entry: Convert Debug exception to IDTENTRY_DB Thomas Gleixner
` (11 subsequent siblings)
15 siblings, 0 replies; 31+ messages in thread
From: Thomas Gleixner @ 2020-02-25 22:33 UTC (permalink / raw)
To: LKML
Cc: x86, Steven Rostedt, Brian Gerst, Juergen Gross, Paolo Bonzini,
Arnd Bergmann
Convert #NMI to IDTENTRY_NMI:
- Implement the C entry point with DEFINE_IDTENTRY_NMI
- Fixup the XEN/PV code
- Remove the old prototyoes
No functional change.
Signed-off-by: Thomas Gleixner <tglx@linutronix.de>
---
arch/x86/entry/entry_32.S | 8 ++++----
arch/x86/entry/entry_64.S | 15 +++++++--------
arch/x86/include/asm/idtentry.h | 4 ++++
arch/x86/include/asm/traps.h | 3 ---
arch/x86/kernel/idt.c | 4 ++--
arch/x86/kernel/nmi.c | 4 +---
arch/x86/xen/enlighten_pv.c | 7 ++++++-
arch/x86/xen/xen-asm_64.S | 2 +-
8 files changed, 25 insertions(+), 22 deletions(-)
--- a/arch/x86/entry/entry_32.S
+++ b/arch/x86/entry/entry_32.S
@@ -1514,7 +1514,7 @@ SYM_CODE_END(double_fault)
* switched stacks. We handle both conditions by simply checking whether we
* interrupted kernel code running on the SYSENTER stack.
*/
-SYM_CODE_START(nmi)
+SYM_CODE_START(asm_exc_nmi)
ASM_CLAC
#ifdef CONFIG_X86_ESPFIX32
@@ -1543,7 +1543,7 @@ SYM_CODE_START(nmi)
jb .Lnmi_from_sysenter_stack
/* Not on SYSENTER stack. */
- call do_nmi
+ call exc_nmi
jmp .Lnmi_return
.Lnmi_from_sysenter_stack:
@@ -1553,7 +1553,7 @@ SYM_CODE_START(nmi)
*/
movl %esp, %ebx
movl PER_CPU_VAR(cpu_current_top_of_stack), %esp
- call do_nmi
+ call exc_nmi
movl %ebx, %esp
.Lnmi_return:
@@ -1607,7 +1607,7 @@ SYM_CODE_START(nmi)
lss (1+5+6)*4(%esp), %esp # back to espfix stack
jmp .Lirq_return
#endif
-SYM_CODE_END(nmi)
+SYM_CODE_END(asm_exc_nmi)
#ifdef CONFIG_KVM_GUEST
SYM_CODE_START(async_page_fault)
--- a/arch/x86/entry/entry_64.S
+++ b/arch/x86/entry/entry_64.S
@@ -1071,7 +1071,6 @@ idtentry_df X86_TRAP_DF double_fault d
#ifdef CONFIG_XEN_PV
idtentry 512 /* dummy */ hypervisor_callback xen_do_hypervisor_callback has_error_code=0
-idtentry X86_TRAP_NMI xennmi do_nmi has_error_code=0
idtentry X86_TRAP_DB xendebug do_debug has_error_code=0
#endif
@@ -1400,7 +1399,7 @@ SYM_CODE_END(error_exit)
* %r14: Used to save/restore the CR3 of the interrupted context
* when PAGE_TABLE_ISOLATION is in use. Do not clobber.
*/
-SYM_CODE_START(nmi)
+SYM_CODE_START(asm_exc_nmi)
UNWIND_HINT_IRET_REGS
/*
@@ -1485,7 +1484,7 @@ SYM_CODE_START(nmi)
movq %rsp, %rdi
movq $-1, %rsi
- call do_nmi
+ call exc_nmi
/*
* Return back to user mode. We must *not* do the normal exit
@@ -1542,7 +1541,7 @@ SYM_CODE_START(nmi)
* end_repeat_nmi, then we are a nested NMI. We must not
* modify the "iret" frame because it's being written by
* the outer NMI. That's okay; the outer NMI handler is
- * about to about to call do_nmi anyway, so we can just
+ * about to about to call exc_nmi() anyway, so we can just
* resume the outer NMI.
*/
@@ -1661,7 +1660,7 @@ SYM_CODE_START(nmi)
* RSP is pointing to "outermost RIP". gsbase is unknown, but, if
* we're repeating an NMI, gsbase has the same value that it had on
* the first iteration. paranoid_entry will load the kernel
- * gsbase if needed before we call do_nmi. "NMI executing"
+ * gsbase if needed before we call exc_nmi(). "NMI executing"
* is zero.
*/
movq $1, 10*8(%rsp) /* Set "NMI executing". */
@@ -1695,10 +1694,10 @@ SYM_CODE_START(nmi)
call paranoid_entry
UNWIND_HINT_REGS
- /* paranoidentry do_nmi, 0; without TRACE_IRQS_OFF */
+ /* paranoidentry exc_nmi(), 0; without TRACE_IRQS_OFF */
movq %rsp, %rdi
movq $-1, %rsi
- call do_nmi
+ call exc_nmi
/* Always restore stashed CR3 value (see paranoid_entry) */
RESTORE_CR3 scratch_reg=%r15 save_reg=%r14
@@ -1735,7 +1734,7 @@ SYM_CODE_START(nmi)
* about espfix64 on the way back to kernel mode.
*/
iretq
-SYM_CODE_END(nmi)
+SYM_CODE_END(asm_exc_nmi)
#ifndef CONFIG_IA32_EMULATION
/*
--- a/arch/x86/include/asm/idtentry.h
+++ b/arch/x86/include/asm/idtentry.h
@@ -238,4 +238,8 @@ DECLARE_IDTENTRY_ERRORCODE(X86_TRAP_AC,
DECLARE_IDTENTRY_MCE(X86_TRAP_MC, exc_machine_check);
#endif
+/* NMI */
+DECLARE_IDTENTRY_NMI(X86_TRAP_NMI, exc_nmi);
+DECLARE_IDTENTRY_XEN(X86_TRAP_NMI, nmi);
+
#endif
--- a/arch/x86/include/asm/traps.h
+++ b/arch/x86/include/asm/traps.h
@@ -12,7 +12,6 @@
#define dotraplinkage __visible
asmlinkage void debug(void);
-asmlinkage void nmi(void);
#ifdef CONFIG_X86_64
asmlinkage void double_fault(void);
#endif
@@ -20,14 +19,12 @@ asmlinkage void page_fault(void);
asmlinkage void async_page_fault(void);
#if defined(CONFIG_X86_64) && defined(CONFIG_XEN_PV)
-asmlinkage void xen_xennmi(void);
asmlinkage void xen_xendebug(void);
asmlinkage void xen_double_fault(void);
asmlinkage void xen_page_fault(void);
#endif
dotraplinkage void do_debug(struct pt_regs *regs, long error_code);
-dotraplinkage void do_nmi(struct pt_regs *regs, long error_code);
#if defined(CONFIG_X86_64) || defined(CONFIG_DOUBLEFAULT)
dotraplinkage void do_double_fault(struct pt_regs *regs, long error_code, unsigned long cr2);
#endif
--- a/arch/x86/kernel/idt.c
+++ b/arch/x86/kernel/idt.c
@@ -71,7 +71,7 @@ static const __initconst struct idt_data
*/
static const __initconst struct idt_data def_idts[] = {
INTG(X86_TRAP_DE, asm_exc_divide_error),
- INTG(X86_TRAP_NMI, nmi),
+ INTG(X86_TRAP_NMI, asm_exc_nmi),
INTG(X86_TRAP_BR, asm_exc_bounds),
INTG(X86_TRAP_UD, asm_exc_invalid_op),
INTG(X86_TRAP_NM, asm_exc_device_not_available),
@@ -183,7 +183,7 @@ gate_desc debug_idt_table[IDT_ENTRIES] _
*/
static const __initconst struct idt_data ist_idts[] = {
ISTG(X86_TRAP_DB, debug, IST_INDEX_DB),
- ISTG(X86_TRAP_NMI, nmi, IST_INDEX_NMI),
+ ISTG(X86_TRAP_NMI, asm_exc_nmi, IST_INDEX_NMI),
ISTG(X86_TRAP_DF, double_fault, IST_INDEX_DF),
#ifdef CONFIG_X86_MCE
ISTG(X86_TRAP_MC, asm_exc_machine_check, IST_INDEX_MCE),
--- a/arch/x86/kernel/nmi.c
+++ b/arch/x86/kernel/nmi.c
@@ -507,8 +507,7 @@ static bool notrace is_debug_stack(unsig
NOKPROBE_SYMBOL(is_debug_stack);
#endif
-dotraplinkage notrace void
-do_nmi(struct pt_regs *regs, long error_code)
+DEFINE_IDTENTRY_NMI(exc_nmi)
{
if (IS_ENABLED(CONFIG_SMP) && cpu_is_offline(smp_processor_id()))
return;
@@ -558,7 +557,6 @@ do_nmi(struct pt_regs *regs, long error_
if (user_mode(regs))
mds_user_clear_cpu_buffers();
}
-NOKPROBE_SYMBOL(do_nmi);
void stop_nmi(void)
{
--- a/arch/x86/xen/enlighten_pv.c
+++ b/arch/x86/xen/enlighten_pv.c
@@ -607,13 +607,18 @@ struct trap_array_entry {
.xen = xen_asm_##func, \
.ist_okay = ist_ok }
+#define TRAP_ENTRY_REDIR(func, xenfunc, ist_ok) { \
+ .orig = asm_##func, \
+ .xen = xen_asm_##xenfunc, \
+ .ist_okay = ist_ok }
+
static struct trap_array_entry trap_array[] = {
{ debug, xen_xendebug, true },
{ double_fault, xen_double_fault, true },
#ifdef CONFIG_X86_MCE
TRAP_ENTRY(exc_machine_check, true ),
#endif
- { nmi, xen_xennmi, true },
+ TRAP_ENTRY_REDIR(exc_nmi, exc_xennmi, true ),
TRAP_ENTRY(exc_int3, false ),
TRAP_ENTRY(exc_overflow, false ),
#ifdef CONFIG_IA32_EMULATION
--- a/arch/x86/xen/xen-asm_64.S
+++ b/arch/x86/xen/xen-asm_64.S
@@ -32,7 +32,7 @@ xen_pv_trap asm_exc_divide_error
xen_pv_trap debug
xen_pv_trap xendebug
xen_pv_trap asm_exc_int3
-xen_pv_trap xennmi
+xen_pv_trap asm_exc_xennmi
xen_pv_trap asm_exc_overflow
xen_pv_trap asm_exc_bounds
xen_pv_trap asm_exc_invalid_op
^ permalink raw reply [flat|nested] 31+ messages in thread
* [patch 05/16] x86/entry: Convert Debug exception to IDTENTRY_DB
2020-02-25 22:33 [patch 00/16] x86/entry: Consolidation - Part IV Thomas Gleixner
` (3 preceding siblings ...)
2020-02-25 22:33 ` [patch 04/16] x86/entry: Convert NMI to IDTENTRY_NMI Thomas Gleixner
@ 2020-02-25 22:33 ` Thomas Gleixner
2020-02-25 22:33 ` [patch 06/16] x86/entry/64: Remove error code clearing from #DB and #MCE ASM stub Thomas Gleixner
` (10 subsequent siblings)
15 siblings, 0 replies; 31+ messages in thread
From: Thomas Gleixner @ 2020-02-25 22:33 UTC (permalink / raw)
To: LKML
Cc: x86, Steven Rostedt, Brian Gerst, Juergen Gross, Paolo Bonzini,
Arnd Bergmann
Convert #DB to IDTENTRY_ERRORCODE:
- Implement the C entry point with DEFINE_IDTENTRY_DB
- Emit the ASM stub with DECLARE_IDTENTRY
- Remove the ASM idtentry in 64bit
- Remove the open coded ASM entry code in 32bit
- Fixup the XEN/PV code
- Remove the old prototyoes
No functional change.
Signed-off-by: Thomas Gleixner <tglx@linutronix.de>
---
arch/x86/entry/entry_32.S | 10 ----------
arch/x86/entry/entry_64.S | 2 --
arch/x86/include/asm/idtentry.h | 4 ++++
arch/x86/include/asm/traps.h | 3 ---
arch/x86/kernel/idt.c | 8 ++++----
arch/x86/kernel/traps.c | 13 ++++++-------
arch/x86/xen/enlighten_pv.c | 2 +-
arch/x86/xen/xen-asm_64.S | 4 ++--
8 files changed, 17 insertions(+), 29 deletions(-)
--- a/arch/x86/entry/entry_32.S
+++ b/arch/x86/entry/entry_32.S
@@ -1455,16 +1455,6 @@ SYM_CODE_START_LOCAL_NOALIGN(common_exce
jmp ret_from_exception
SYM_CODE_END(common_exception)
-SYM_CODE_START(debug)
- /*
- * Entry from sysenter is now handled in common_exception
- */
- ASM_CLAC
- pushl $0
- pushl $do_debug
- jmp common_exception
-SYM_CODE_END(debug)
-
#ifdef CONFIG_DOUBLEFAULT
SYM_CODE_START(double_fault)
1:
--- a/arch/x86/entry/entry_64.S
+++ b/arch/x86/entry/entry_64.S
@@ -1066,12 +1066,10 @@ idtentry X86_TRAP_PF page_fault do_pag
idtentry X86_TRAP_PF async_page_fault do_async_page_fault has_error_code=1
#endif
-idtentry_mce_db X86_TRAP_DB debug do_debug
idtentry_df X86_TRAP_DF double_fault do_double_fault
#ifdef CONFIG_XEN_PV
idtentry 512 /* dummy */ hypervisor_callback xen_do_hypervisor_callback has_error_code=0
-idtentry X86_TRAP_DB xendebug do_debug has_error_code=0
#endif
/*
--- a/arch/x86/include/asm/idtentry.h
+++ b/arch/x86/include/asm/idtentry.h
@@ -242,4 +242,8 @@ DECLARE_IDTENTRY_MCE(X86_TRAP_MC, exc_ma
DECLARE_IDTENTRY_NMI(X86_TRAP_NMI, exc_nmi);
DECLARE_IDTENTRY_XEN(X86_TRAP_NMI, nmi);
+/* #DB */
+DECLARE_IDTENTRY_DEBUG(X86_TRAP_DB, exc_debug);
+DECLARE_IDTENTRY_XEN(X86_TRAP_DB, debug);
+
#endif
--- a/arch/x86/include/asm/traps.h
+++ b/arch/x86/include/asm/traps.h
@@ -11,7 +11,6 @@
#define dotraplinkage __visible
-asmlinkage void debug(void);
#ifdef CONFIG_X86_64
asmlinkage void double_fault(void);
#endif
@@ -19,12 +18,10 @@ asmlinkage void page_fault(void);
asmlinkage void async_page_fault(void);
#if defined(CONFIG_X86_64) && defined(CONFIG_XEN_PV)
-asmlinkage void xen_xendebug(void);
asmlinkage void xen_double_fault(void);
asmlinkage void xen_page_fault(void);
#endif
-dotraplinkage void do_debug(struct pt_regs *regs, long error_code);
#if defined(CONFIG_X86_64) || defined(CONFIG_DOUBLEFAULT)
dotraplinkage void do_double_fault(struct pt_regs *regs, long error_code, unsigned long cr2);
#endif
--- a/arch/x86/kernel/idt.c
+++ b/arch/x86/kernel/idt.c
@@ -56,7 +56,7 @@ struct idt_data {
* stacks work only after cpu_init().
*/
static const __initconst struct idt_data early_idts[] = {
- INTG(X86_TRAP_DB, debug),
+ INTG(X86_TRAP_DB, asm_exc_debug),
SYSG(X86_TRAP_BP, asm_exc_int3),
#ifdef CONFIG_X86_32
INTG(X86_TRAP_PF, page_fault),
@@ -90,7 +90,7 @@ static const __initconst struct idt_data
#else
INTG(X86_TRAP_DF, double_fault),
#endif
- INTG(X86_TRAP_DB, debug),
+ INTG(X86_TRAP_DB, asm_exc_debug),
#ifdef CONFIG_X86_MCE
INTG(X86_TRAP_MC, asm_exc_machine_check),
@@ -161,7 +161,7 @@ static const __initconst struct idt_data
* stack set to DEFAULT_STACK (0). Required for NMI trap handling.
*/
static const __initconst struct idt_data dbg_idts[] = {
- INTG(X86_TRAP_DB, debug),
+ INTG(X86_TRAP_DB, asm_exc_debug),
};
#endif
@@ -182,7 +182,7 @@ gate_desc debug_idt_table[IDT_ENTRIES] _
* cpu_init() when the TSS has been initialized.
*/
static const __initconst struct idt_data ist_idts[] = {
- ISTG(X86_TRAP_DB, debug, IST_INDEX_DB),
+ ISTG(X86_TRAP_DB, asm_exc_debug, IST_INDEX_DB),
ISTG(X86_TRAP_NMI, asm_exc_nmi, IST_INDEX_NMI),
ISTG(X86_TRAP_DF, double_fault, IST_INDEX_DF),
#ifdef CONFIG_X86_MCE
--- a/arch/x86/kernel/traps.c
+++ b/arch/x86/kernel/traps.c
@@ -739,7 +739,7 @@ static bool is_sysenter_singlestep(struc
*
* May run on IST stack.
*/
-dotraplinkage void do_debug(struct pt_regs *regs, long error_code)
+DEFINE_IDTENTRY_DEBUG(exc_debug)
{
struct task_struct *tsk = current;
int user_icebp = 0;
@@ -800,8 +800,8 @@ dotraplinkage void do_debug(struct pt_re
goto exit;
#endif
- if (notify_die(DIE_DEBUG, "debug", regs, (long)&dr6, error_code,
- SIGTRAP) == NOTIFY_STOP)
+ if (notify_die(DIE_DEBUG, "debug", regs, (long)&dr6, 0,
+ SIGTRAP) == NOTIFY_STOP)
goto exit;
/*
@@ -814,8 +814,8 @@ dotraplinkage void do_debug(struct pt_re
cond_local_irq_enable(regs);
if (v8086_mode(regs)) {
- handle_vm86_trap((struct kernel_vm86_regs *) regs, error_code,
- X86_TRAP_DB);
+ handle_vm86_trap((struct kernel_vm86_regs *) regs, 0,
+ X86_TRAP_DB);
cond_local_irq_disable(regs);
debug_stack_usage_dec();
goto exit;
@@ -834,14 +834,13 @@ dotraplinkage void do_debug(struct pt_re
}
si_code = get_si_code(tsk->thread.debugreg6);
if (tsk->thread.debugreg6 & (DR_STEP | DR_TRAP_BITS) || user_icebp)
- send_sigtrap(regs, error_code, si_code);
+ send_sigtrap(regs, 0, si_code);
cond_local_irq_disable(regs);
debug_stack_usage_dec();
exit:
ist_exit(regs);
}
-NOKPROBE_SYMBOL(do_debug);
/*
* Note that we play around with the 'TS' bit in an attempt to get
--- a/arch/x86/xen/enlighten_pv.c
+++ b/arch/x86/xen/enlighten_pv.c
@@ -613,7 +613,7 @@ struct trap_array_entry {
.ist_okay = ist_ok }
static struct trap_array_entry trap_array[] = {
- { debug, xen_xendebug, true },
+ TRAP_ENTRY_REDIR(exc_debug, exc_xendebug, true ),
{ double_fault, xen_double_fault, true },
#ifdef CONFIG_X86_MCE
TRAP_ENTRY(exc_machine_check, true ),
--- a/arch/x86/xen/xen-asm_64.S
+++ b/arch/x86/xen/xen-asm_64.S
@@ -29,8 +29,8 @@ SYM_CODE_END(xen_\name)
.endm
xen_pv_trap asm_exc_divide_error
-xen_pv_trap debug
-xen_pv_trap xendebug
+xen_pv_trap asm_exc_debug
+xen_pv_trap asm_exc_xendebug
xen_pv_trap asm_exc_int3
xen_pv_trap asm_exc_xennmi
xen_pv_trap asm_exc_overflow
^ permalink raw reply [flat|nested] 31+ messages in thread
* [patch 06/16] x86/entry/64: Remove error code clearing from #DB and #MCE ASM stub
2020-02-25 22:33 [patch 00/16] x86/entry: Consolidation - Part IV Thomas Gleixner
` (4 preceding siblings ...)
2020-02-25 22:33 ` [patch 05/16] x86/entry: Convert Debug exception to IDTENTRY_DB Thomas Gleixner
@ 2020-02-25 22:33 ` Thomas Gleixner
2020-02-25 22:33 ` [patch 07/16] x86/entry: Provide IDTRENTRY_NOIST variants for #DB and #MC Thomas Gleixner
` (9 subsequent siblings)
15 siblings, 0 replies; 31+ messages in thread
From: Thomas Gleixner @ 2020-02-25 22:33 UTC (permalink / raw)
To: LKML
Cc: x86, Steven Rostedt, Brian Gerst, Juergen Gross, Paolo Bonzini,
Arnd Bergmann
The C entry point do not expect an error code.
Signed-off-by: Thomas Gleixner <tglx@linutronix.de>
---
arch/x86/entry/entry_64.S | 1 -
1 file changed, 1 deletion(-)
--- a/arch/x86/entry/entry_64.S
+++ b/arch/x86/entry/entry_64.S
@@ -631,7 +631,6 @@ SYM_CODE_START(\asmsym)
.endif
movq %rsp, %rdi /* pt_regs pointer */
- xorl %esi, %esi /* Clear the error code */
.if \vector == X86_TRAP_DB
subq $DB_STACK_OFFSET, CPU_TSS_IST(IST_INDEX_DB)
^ permalink raw reply [flat|nested] 31+ messages in thread
* [patch 07/16] x86/entry: Provide IDTRENTRY_NOIST variants for #DB and #MC
2020-02-25 22:33 [patch 00/16] x86/entry: Consolidation - Part IV Thomas Gleixner
` (5 preceding siblings ...)
2020-02-25 22:33 ` [patch 06/16] x86/entry/64: Remove error code clearing from #DB and #MCE ASM stub Thomas Gleixner
@ 2020-02-25 22:33 ` Thomas Gleixner
2020-02-25 22:33 ` [patch 08/16] x86/entry: Implement user mode C entry points for #DB and #MCE Thomas Gleixner
` (8 subsequent siblings)
15 siblings, 0 replies; 31+ messages in thread
From: Thomas Gleixner @ 2020-02-25 22:33 UTC (permalink / raw)
To: LKML
Cc: x86, Steven Rostedt, Brian Gerst, Juergen Gross, Paolo Bonzini,
Arnd Bergmann
Provide NOIST entry point macros which allows to implement NOIST variants
of the C entry points. These are invoked when #DB or #MC enter from user
space. This allows explicit handling of the difference between user mode
and kernel mode entry later.
Signed-off-by: Thomas Gleixner <tglx@linutronix.de>
---
arch/x86/include/asm/idtentry.h | 17 ++++++++++++++++-
1 file changed, 16 insertions(+), 1 deletion(-)
--- a/arch/x86/include/asm/idtentry.h
+++ b/arch/x86/include/asm/idtentry.h
@@ -117,14 +117,16 @@ static __always_inline void __##func(str
* @vector: Vector number (ignored for C)
* @func: Function name of the entry point
*
- * Declares three functions:
+ * Declares four functions:
* - The ASM entry point: asm_##func
* - The XEN PV trap entry point: xen_##func (maybe unused)
+ * - The NOIST C handler called from the ASM entry point on user mode entry
* - The C handler called from the ASM entry point
*/
#define DECLARE_IDTENTRY_IST(vector, func) \
asmlinkage void asm_##func(void); \
asmlinkage void xen_asm_##func(void); \
+ __visible void noist_##func(struct pt_regs *regs); \
__visible void func(struct pt_regs *regs)
/**
@@ -147,6 +149,17 @@ NOKPROBE_SYMBOL(func); \
\
static __always_inline void __##func(struct pt_regs *regs)
+/**
+ * DEFINE_IDTENTRY_NOIST - Emit code for NOIST entry points which
+ * belong to a IST entry point (MCE, DB)
+ * @func: Function name of the entry point. Must be the same as
+ * the function name of the corresponding IST variant
+ *
+ * Maps to DEFINE_IDTENTRY().
+ */
+#define DEFINE_IDTENTRY_NOIST(func) \
+ DEFINE_IDTENTRY(noist_##func)
+
#else /* CONFIG_X86_64 */
/* Maps to a regular IDTENTRY on 32bit for now */
# define DECLARE_IDTENTRY_IST DECLARE_IDTENTRY
@@ -156,12 +169,14 @@ static __always_inline void __##func(str
/* C-Code mapping */
#define DECLARE_IDTENTRY_MCE DECLARE_IDTENTRY_IST
#define DEFINE_IDTENTRY_MCE DEFINE_IDTENTRY_IST
+#define DEFINE_IDTENTRY_MCE_USER DEFINE_IDTENTRY_NOIST
#define DECLARE_IDTENTRY_NMI DECLARE_IDTENTRY_IST
#define DEFINE_IDTENTRY_NMI DEFINE_IDTENTRY_IST
#define DECLARE_IDTENTRY_DEBUG DECLARE_IDTENTRY_IST
#define DEFINE_IDTENTRY_DEBUG DEFINE_IDTENTRY_IST
+#define DEFINE_IDTENTRY_DEBUG_USER DEFINE_IDTENTRY_NOIST
/**
* DECLARE_IDTENTRY_XEN - Declare functions for XEN redirect IDT entry points
^ permalink raw reply [flat|nested] 31+ messages in thread
* [patch 08/16] x86/entry: Implement user mode C entry points for #DB and #MCE
2020-02-25 22:33 [patch 00/16] x86/entry: Consolidation - Part IV Thomas Gleixner
` (6 preceding siblings ...)
2020-02-25 22:33 ` [patch 07/16] x86/entry: Provide IDTRENTRY_NOIST variants for #DB and #MC Thomas Gleixner
@ 2020-02-25 22:33 ` Thomas Gleixner
2020-02-25 22:33 ` [patch 09/16] x86/entry: Provide IDTENTRY_DF Thomas Gleixner
` (7 subsequent siblings)
15 siblings, 0 replies; 31+ messages in thread
From: Thomas Gleixner @ 2020-02-25 22:33 UTC (permalink / raw)
To: LKML
Cc: x86, Steven Rostedt, Brian Gerst, Juergen Gross, Paolo Bonzini,
Arnd Bergmann
The MCE entry point uses the same mechanism as the IST entry point for
now. For #DB split the inner workings and just keep the ist_enter/exit
magic in the IST variant. Fixup the ASM code to emit the proper
noist_##cfunc call.
Signed-off-by: Thomas Gleixner <tglx@linutronix.de>
---
arch/x86/entry/entry_64.S | 2 +-
arch/x86/kernel/cpu/mce/core.c | 11 +++++++++++
arch/x86/kernel/traps.c | 30 ++++++++++++++++++++++--------
3 files changed, 34 insertions(+), 9 deletions(-)
--- a/arch/x86/entry/entry_64.S
+++ b/arch/x86/entry/entry_64.S
@@ -646,7 +646,7 @@ SYM_CODE_START(\asmsym)
/* Switch to the regular task stack and use the noist entry point */
.Lfrom_usermode_switch_stack_\@:
- idtentry_body vector \cfunc, has_error_code=0
+ idtentry_body vector noist_\cfunc, has_error_code=0
_ASM_NOKPROBE(\asmsym)
SYM_CODE_END(\asmsym)
--- a/arch/x86/kernel/cpu/mce/core.c
+++ b/arch/x86/kernel/cpu/mce/core.c
@@ -1898,11 +1898,22 @@ static void unexpected_machine_check(str
/* Call the installed machine check handler for this CPU setup. */
void (*machine_check_vector)(struct pt_regs *) = unexpected_machine_check;
+/* MCE hit kernel mode */
DEFINE_IDTENTRY_MCE(exc_machine_check)
{
machine_check_vector(regs);
}
+#ifdef CONFIG_X86_64
+/*
+ * The user mode variant (same content for now).
+ */
+DEFINE_IDTENTRY_MCE_USER(exc_machine_check)
+{
+ machine_check_vector(regs);
+}
+#endif
+
/*
* Called for each booted CPU to set up machine checks.
* Must be called with preempt off:
--- a/arch/x86/kernel/traps.c
+++ b/arch/x86/kernel/traps.c
@@ -739,15 +739,13 @@ static bool is_sysenter_singlestep(struc
*
* May run on IST stack.
*/
-DEFINE_IDTENTRY_DEBUG(exc_debug)
+static void notrace handle_debug(struct pt_regs *regs)
{
struct task_struct *tsk = current;
int user_icebp = 0;
unsigned long dr6;
int si_code;
- ist_enter(regs);
-
get_debugreg(dr6, 6);
/*
* The Intel SDM says:
@@ -776,7 +774,7 @@ DEFINE_IDTENTRY_DEBUG(exc_debug)
is_sysenter_singlestep(regs))) {
dr6 &= ~DR_STEP;
if (!dr6)
- goto exit;
+ return;
/*
* else we might have gotten a single-step trap and hit a
* watchpoint at the same time, in which case we should fall
@@ -797,12 +795,12 @@ DEFINE_IDTENTRY_DEBUG(exc_debug)
#ifdef CONFIG_KPROBES
if (kprobe_debug_handler(regs))
- goto exit;
+ return;
#endif
if (notify_die(DIE_DEBUG, "debug", regs, (long)&dr6, 0,
SIGTRAP) == NOTIFY_STOP)
- goto exit;
+ return;
/*
* Let others (NMI) know that the debug stack is in use
@@ -818,7 +816,7 @@ DEFINE_IDTENTRY_DEBUG(exc_debug)
X86_TRAP_DB);
cond_local_irq_disable(regs);
debug_stack_usage_dec();
- goto exit;
+ return;
}
if (WARN_ON_ONCE((dr6 & DR_STEP) && !user_mode(regs))) {
@@ -837,11 +835,27 @@ DEFINE_IDTENTRY_DEBUG(exc_debug)
send_sigtrap(regs, 0, si_code);
cond_local_irq_disable(regs);
debug_stack_usage_dec();
+}
+NOKPROBE_SYMBOL(handle_debug);
-exit:
+/*
+ * IST stack entry.
+ */
+DEFINE_IDTENTRY_DEBUG(exc_debug)
+{
+ ist_enter(regs);
+ handle_debug(regs);
ist_exit(regs);
}
+#ifdef CONFIG_X86_64
+/* User entry, runs on regular task stack */
+DEFINE_IDTENTRY_DEBUG_USER(exc_debug)
+{
+ handle_debug(regs);
+}
+#endif
+
/*
* Note that we play around with the 'TS' bit in an attempt to get
* the correct behaviour even in the presence of the asynchronous
^ permalink raw reply [flat|nested] 31+ messages in thread
* [patch 09/16] x86/entry: Provide IDTENTRY_DF
2020-02-25 22:33 [patch 00/16] x86/entry: Consolidation - Part IV Thomas Gleixner
` (7 preceding siblings ...)
2020-02-25 22:33 ` [patch 08/16] x86/entry: Implement user mode C entry points for #DB and #MCE Thomas Gleixner
@ 2020-02-25 22:33 ` Thomas Gleixner
2020-02-25 22:33 ` [patch 10/16] x86/entry: Convert double fault exception to IDTENTRY_DF Thomas Gleixner
` (6 subsequent siblings)
15 siblings, 0 replies; 31+ messages in thread
From: Thomas Gleixner @ 2020-02-25 22:33 UTC (permalink / raw)
To: LKML
Cc: x86, Steven Rostedt, Brian Gerst, Juergen Gross, Paolo Bonzini,
Arnd Bergmann
Provide a separate macro for #DF as this needs to emit paranoid only code
and has also a special ASM stub in 32bit.
Signed-off-by: Thomas Gleixner <tglx@linutronix.de>
---
arch/x86/include/asm/idtentry.h | 82 ++++++++++++++++++++++++++++++++++++++++
1 file changed, 82 insertions(+)
--- a/arch/x86/include/asm/idtentry.h
+++ b/arch/x86/include/asm/idtentry.h
@@ -160,10 +160,85 @@ static __always_inline void __##func(str
#define DEFINE_IDTENTRY_NOIST(func) \
DEFINE_IDTENTRY(noist_##func)
+/**
+ * DECLARE_IDTENTRY_DF - Declare functions for double fault
+ * @vector: Vector number (ignored for C)
+ * @func: Function name of the entry point
+ *
+ * Maps to DECLARE_IDTENTRY_ERRORCODE
+ */
+#define DECLARE_IDTENTRY_DF(vector, func) \
+ DECLARE_IDTENTRY_ERRORCODE(vector, func)
+
+/**
+ * DEFINE_IDTENTRY_DF - Emit code for double fault
+ * @func: Function name of the entry point
+ *
+ * @func is called from ASM entry code with interrupts disabled.
+ */
+#define DEFINE_IDTENTRY_DF(func) \
+static __always_inline void __##func(struct pt_regs *regs, \
+ unsigned long error_code, \
+ unsigned long address); \
+ \
+__visible notrace void func(struct pt_regs *regs, \
+ unsigned long error_code) \
+{ \
+ unsigned long address = read_cr2(); \
+ \
+ trace_hardirqs_off(); \
+ __##func (regs, error_code, address); \
+} \
+NOKPROBE_SYMBOL(func); \
+ \
+static __always_inline void __##func(struct pt_regs *regs, \
+ unsigned long error_code, \
+ unsigned long address)
+
#else /* CONFIG_X86_64 */
/* Maps to a regular IDTENTRY on 32bit for now */
# define DECLARE_IDTENTRY_IST DECLARE_IDTENTRY
# define DEFINE_IDTENTRY_IST DEFINE_IDTENTRY
+
+/**
+ * DECLARE_IDTENTRY_DF - Declare functions for double fault 32bit variant
+ * @vector: Vector number (ignored for C)
+ * @func: Function name of the entry point
+ *
+ * Declares three functions:
+ * - The ASM entry point: asm_##func
+ * - The C handler called from the C shim
+ */
+#define DECLARE_IDTENTRY_DF(vector, func) \
+ asmlinkage void asm_##func(void); \
+ __visible void func(struct pt_regs *regs, \
+ unsigned long error_code, \
+ unsigned long address)
+
+/**
+ * DEFINE_IDTENTRY_DF - Emit code for double fault on 32bit
+ * @func: Function name of the entry point
+ *
+ * This is called through the doublefault shim which already provides
+ * cr2 in the address argument.
+ */
+#define DEFINE_IDTENTRY_DF(func) \
+static __always_inline void __##func(struct pt_regs *regs, \
+ unsigned long error_code, \
+ unsigned long address); \
+ \
+__visible notrace void func(struct pt_regs *regs, \
+ unsigned long error_code, \
+ unsigned long address) \
+{ \
+ __##func (regs, error_code, address); \
+} \
+NOKPROBE_SYMBOL(func); \
+ \
+static __always_inline void __##func(struct pt_regs *regs, \
+ unsigned long error_code, \
+ unsigned long address)
+
#endif /* !CONFIG_X86_64 */
/* C-Code mapping */
@@ -209,12 +284,19 @@ static __always_inline void __##func(str
# define DECLARE_IDTENTRY_DEBUG(vector, func) \
idtentry_mce_db vector asm_##func func
+# define DECLARE_IDTENTRY_DF(vector, func) \
+ idtentry_df vector asm_##func func
+
#else
# define DECLARE_IDTENTRY_MCE(vector, func) \
DECLARE_IDTENTRY(vector, func)
# define DECLARE_IDTENTRY_DEBUG(vector, func) \
DECLARE_IDTENTRY(vector, func)
+
+/* No ASM emitted for DF as this goes through a C shim */
+# define DECLARE_IDTENTRY_DF(vector, func)
+
#endif
/* No ASM code emitted for NMI */
^ permalink raw reply [flat|nested] 31+ messages in thread
* [patch 10/16] x86/entry: Convert double fault exception to IDTENTRY_DF
2020-02-25 22:33 [patch 00/16] x86/entry: Consolidation - Part IV Thomas Gleixner
` (8 preceding siblings ...)
2020-02-25 22:33 ` [patch 09/16] x86/entry: Provide IDTENTRY_DF Thomas Gleixner
@ 2020-02-25 22:33 ` Thomas Gleixner
2020-02-25 22:33 ` [patch 11/16] x86/entry: Switch XEN/PV hypercall entry to IDTENTRY Thomas Gleixner
` (5 subsequent siblings)
15 siblings, 0 replies; 31+ messages in thread
From: Thomas Gleixner @ 2020-02-25 22:33 UTC (permalink / raw)
To: LKML
Cc: x86, Steven Rostedt, Brian Gerst, Juergen Gross, Paolo Bonzini,
Arnd Bergmann
Convert #DF to IDTENTRY_DF
- Implement the C entry point with DEFINE_IDTENTRY_DF
- Emit the ASM stub with DECLARE_IDTENTRY_DF on 64bit
- Remove the ASM idtentry in 64bit
- Adjust the 32bit shim code
- Fixup the XEN/PV code
- Remove the old prototyoes
No functional change.
Signed-off-by: Thomas Gleixner <tglx@linutronix.de>
---
arch/x86/entry/entry_32.S | 4 ++--
arch/x86/entry/entry_64.S | 10 +---------
arch/x86/include/asm/idtentry.h | 5 +++++
arch/x86/include/asm/traps.h | 7 -------
arch/x86/kernel/doublefault_32.c | 7 +++----
arch/x86/kernel/idt.c | 4 ++--
arch/x86/kernel/traps.c | 8 +++++---
arch/x86/xen/enlighten_pv.c | 4 ++--
arch/x86/xen/xen-asm_64.S | 2 +-
9 files changed, 21 insertions(+), 30 deletions(-)
--- a/arch/x86/entry/entry_32.S
+++ b/arch/x86/entry/entry_32.S
@@ -1456,7 +1456,7 @@ SYM_CODE_START_LOCAL_NOALIGN(common_exce
SYM_CODE_END(common_exception)
#ifdef CONFIG_DOUBLEFAULT
-SYM_CODE_START(double_fault)
+SYM_CODE_START(asm_exc_double_fault)
1:
/*
* This is a task gate handler, not an interrupt gate handler.
@@ -1494,7 +1494,7 @@ SYM_CODE_START(double_fault)
1:
hlt
jmp 1b
-SYM_CODE_END(double_fault)
+SYM_CODE_END(asm_exc_double_fault)
#endif
/*
--- a/arch/x86/entry/entry_64.S
+++ b/arch/x86/entry/entry_64.S
@@ -669,15 +669,9 @@ SYM_CODE_START(\asmsym)
call paranoid_entry
UNWIND_HINT_REGS
- /* Read CR2 early */
- GET_CR2_INTO(%r12);
-
- TRACE_IRQS_OFF
-
movq %rsp, %rdi /* pt_regs pointer into first argument */
movq ORIG_RAX(%rsp), %rsi /* get error code into 2nd argument*/
movq $-1, ORIG_RAX(%rsp) /* no syscall to restart */
- movq %r12, %rdx /* Move CR2 into 3rd argument */
call \cfunc
jmp paranoid_exit
@@ -906,7 +900,7 @@ SYM_INNER_LABEL(native_irq_return_iret,
/*
* This may fault. Non-paranoid faults on return to userspace are
* handled by fixup_bad_iret. These include #SS, #GP, and #NP.
- * Double-faults due to espfix64 are handled in do_double_fault.
+ * Double-faults due to espfix64 are handled in exc_double_fault.
* Other faults here are fatal.
*/
iretq
@@ -1065,8 +1059,6 @@ idtentry X86_TRAP_PF page_fault do_pag
idtentry X86_TRAP_PF async_page_fault do_async_page_fault has_error_code=1
#endif
-idtentry_df X86_TRAP_DF double_fault do_double_fault
-
#ifdef CONFIG_XEN_PV
idtentry 512 /* dummy */ hypervisor_callback xen_do_hypervisor_callback has_error_code=0
#endif
--- a/arch/x86/include/asm/idtentry.h
+++ b/arch/x86/include/asm/idtentry.h
@@ -343,4 +343,9 @@ DECLARE_IDTENTRY_XEN(X86_TRAP_NMI, nmi);
DECLARE_IDTENTRY_DEBUG(X86_TRAP_DB, exc_debug);
DECLARE_IDTENTRY_XEN(X86_TRAP_DB, debug);
+/* #DF */
+#if defined(CONFIG_X86_64) || defined(CONFIG_DOUBLEFAULT)
+DECLARE_IDTENTRY_DF(X86_TRAP_DF, exc_double_fault);
+#endif
+
#endif
--- a/arch/x86/include/asm/traps.h
+++ b/arch/x86/include/asm/traps.h
@@ -11,20 +11,13 @@
#define dotraplinkage __visible
-#ifdef CONFIG_X86_64
-asmlinkage void double_fault(void);
-#endif
asmlinkage void page_fault(void);
asmlinkage void async_page_fault(void);
#if defined(CONFIG_X86_64) && defined(CONFIG_XEN_PV)
-asmlinkage void xen_double_fault(void);
asmlinkage void xen_page_fault(void);
#endif
-#if defined(CONFIG_X86_64) || defined(CONFIG_DOUBLEFAULT)
-dotraplinkage void do_double_fault(struct pt_regs *regs, long error_code, unsigned long cr2);
-#endif
dotraplinkage void do_page_fault(struct pt_regs *regs, unsigned long error_code, unsigned long address);
#ifdef CONFIG_X86_64
--- a/arch/x86/kernel/doublefault_32.c
+++ b/arch/x86/kernel/doublefault_32.c
@@ -11,7 +11,6 @@
#include <asm/desc.h>
#include <asm/traps.h>
-extern void double_fault(void);
#define ptr_ok(x) ((x) > PAGE_OFFSET && (x) < PAGE_OFFSET + MAXMEM)
#define TSS(x) this_cpu_read(cpu_tss_rw.x86_tss.x)
@@ -41,7 +40,7 @@ asmlinkage notrace void __noreturn doubl
* Fill in pt_regs. A downside of doing this in C is that the unwinder
* won't see it (no ENCODE_FRAME_POINTER), so a nested stack dump
* won't successfully unwind to the source of the double fault.
- * The main dump from do_double_fault() is fine, though, since it
+ * The main dump from exc_double_fault() is fine, though, since it
* uses these regs directly.
*
* If anyone ever cares, this could be moved to asm.
@@ -71,7 +70,7 @@ asmlinkage notrace void __noreturn doubl
regs.cx = TSS(cx);
regs.bx = TSS(bx);
- do_double_fault(®s, 0, cr2);
+ exc_double_fault(®s, 0, cr2);
/*
* x86_32 does not save the original CR3 anywhere on a task switch.
@@ -96,7 +95,7 @@ DEFINE_PER_CPU_PAGE_ALIGNED(struct doubl
.ldt = 0,
.io_bitmap_base = IO_BITMAP_OFFSET_INVALID,
- .ip = (unsigned long) double_fault,
+ .ip = (unsigned long) asm_exc_double_fault,
.flags = X86_EFLAGS_FIXED,
.es = __USER_DS,
.cs = __KERNEL_CS,
--- a/arch/x86/kernel/idt.c
+++ b/arch/x86/kernel/idt.c
@@ -88,7 +88,7 @@ static const __initconst struct idt_data
#ifdef CONFIG_X86_32
TSKG(X86_TRAP_DF, GDT_ENTRY_DOUBLEFAULT_TSS),
#else
- INTG(X86_TRAP_DF, double_fault),
+ INTG(X86_TRAP_DF, asm_exc_double_fault),
#endif
INTG(X86_TRAP_DB, asm_exc_debug),
@@ -184,7 +184,7 @@ gate_desc debug_idt_table[IDT_ENTRIES] _
static const __initconst struct idt_data ist_idts[] = {
ISTG(X86_TRAP_DB, asm_exc_debug, IST_INDEX_DB),
ISTG(X86_TRAP_NMI, asm_exc_nmi, IST_INDEX_NMI),
- ISTG(X86_TRAP_DF, double_fault, IST_INDEX_DF),
+ ISTG(X86_TRAP_DF, asm_exc_double_fault, IST_INDEX_DF),
#ifdef CONFIG_X86_MCE
ISTG(X86_TRAP_MC, asm_exc_machine_check, IST_INDEX_MCE),
#endif
--- a/arch/x86/kernel/traps.c
+++ b/arch/x86/kernel/traps.c
@@ -358,7 +358,7 @@ DEFINE_IDTENTRY_ERRORCODE(exc_alignment_
* be lost. If, for some reason, we need to return to a context with modified
* regs, the shim code could be adjusted to synchronize the registers.
*/
-dotraplinkage void do_double_fault(struct pt_regs *regs, long error_code, unsigned long cr2)
+DEFINE_IDTENTRY_DF(exc_double_fault)
{
static const char str[] = "double fault";
struct task_struct *tsk = current;
@@ -457,8 +457,10 @@ dotraplinkage void do_double_fault(struc
* stack even if the actual trigger for the double fault was
* something else.
*/
- if ((unsigned long)task_stack_page(tsk) - 1 - cr2 < PAGE_SIZE)
- handle_stack_overflow("kernel stack overflow (double-fault)", regs, cr2);
+ if ((unsigned long)task_stack_page(tsk) - 1 - address < PAGE_SIZE) {
+ handle_stack_overflow("kernel stack overflow (double-fault)",
+ regs, address);
+ }
#endif
pr_emerg("PANIC: double fault, error_code: 0x%lx\n", error_code);
--- a/arch/x86/xen/enlighten_pv.c
+++ b/arch/x86/xen/enlighten_pv.c
@@ -614,7 +614,7 @@ struct trap_array_entry {
static struct trap_array_entry trap_array[] = {
TRAP_ENTRY_REDIR(exc_debug, exc_xendebug, true ),
- { double_fault, xen_double_fault, true },
+ TRAP_ENTRY(exc_double_fault, true ),
#ifdef CONFIG_X86_MCE
TRAP_ENTRY(exc_machine_check, true ),
#endif
@@ -649,7 +649,7 @@ static bool __ref get_trap_addr(void **a
* Replace trap handler addresses by Xen specific ones.
* Check for known traps using IST and whitelist them.
* The debugger ones are the only ones we care about.
- * Xen will handle faults like double_fault, * so we should never see
+ * Xen will handle faults like double_fault, so we should never see
* them. Warn if there's an unexpected IST-using fault handler.
*/
for (nr = 0; nr < ARRAY_SIZE(trap_array); nr++) {
--- a/arch/x86/xen/xen-asm_64.S
+++ b/arch/x86/xen/xen-asm_64.S
@@ -37,7 +37,7 @@ xen_pv_trap asm_exc_overflow
xen_pv_trap asm_exc_bounds
xen_pv_trap asm_exc_invalid_op
xen_pv_trap asm_exc_device_not_available
-xen_pv_trap double_fault
+xen_pv_trap asm_exc_double_fault
xen_pv_trap asm_exc_coproc_segment_overrun
xen_pv_trap asm_exc_invalid_tss
xen_pv_trap asm_exc_segment_not_present
^ permalink raw reply [flat|nested] 31+ messages in thread
* [patch 11/16] x86/entry: Switch XEN/PV hypercall entry to IDTENTRY
2020-02-25 22:33 [patch 00/16] x86/entry: Consolidation - Part IV Thomas Gleixner
` (9 preceding siblings ...)
2020-02-25 22:33 ` [patch 10/16] x86/entry: Convert double fault exception to IDTENTRY_DF Thomas Gleixner
@ 2020-02-25 22:33 ` Thomas Gleixner
2020-02-25 22:33 ` [patch 12/16] x86/entry/64: Simplify idtentry_body Thomas Gleixner
` (4 subsequent siblings)
15 siblings, 0 replies; 31+ messages in thread
From: Thomas Gleixner @ 2020-02-25 22:33 UTC (permalink / raw)
To: LKML
Cc: x86, Steven Rostedt, Brian Gerst, Juergen Gross, Paolo Bonzini,
Arnd Bergmann
Convert # to IDTENTRY:
- Implement the C entry point with DEFINE_IDTENTRY
- Emit the ASM stub with DECLARE_IDTENTRY
- Remove the ASM idtentry in 64bit
- Remove the open coded ASM entry code in 32bit
- Fixup the XEN/PV code
- Remove the old prototyoes
No functional change.
Signed-off-by: Thomas Gleixner <tglx@linutronix.de>
---
arch/x86/entry/entry_32.S | 7 +++++--
arch/x86/entry/entry_64.S | 11 ++++-------
arch/x86/include/asm/idtentry.h | 13 +++++++++++++
arch/x86/xen/setup.c | 4 +++-
arch/x86/xen/smp_pv.c | 3 ++-
arch/x86/xen/xen-asm_32.S | 8 ++++----
arch/x86/xen/xen-asm_64.S | 2 +-
arch/x86/xen/xen-ops.h | 1 -
8 files changed, 32 insertions(+), 17 deletions(-)
--- a/arch/x86/entry/entry_32.S
+++ b/arch/x86/entry/entry_32.S
@@ -1307,7 +1307,10 @@ SYM_CODE_END(native_iret)
#endif
#ifdef CONFIG_XEN_PV
-SYM_FUNC_START(xen_hypervisor_callback)
+/*
+ * See comment in entry_64.S for further explanation
+ */
+SYM_FUNC_START(exc_xen_hypervisor_callback)
/*
* Check to see if we got the event in the critical
* region in xen_iret_direct, after we've reenabled
@@ -1331,7 +1334,7 @@ SYM_FUNC_START(xen_hypervisor_callback)
call xen_maybe_preempt_hcall
#endif
jmp ret_from_intr
-SYM_FUNC_END(xen_hypervisor_callback)
+SYM_FUNC_END(exc_xen_hypervisor_callback)
/*
* Hypervisor uses this for application faults while it executes.
--- a/arch/x86/entry/entry_64.S
+++ b/arch/x86/entry/entry_64.S
@@ -1059,10 +1059,6 @@ idtentry X86_TRAP_PF page_fault do_pag
idtentry X86_TRAP_PF async_page_fault do_async_page_fault has_error_code=1
#endif
-#ifdef CONFIG_XEN_PV
-idtentry 512 /* dummy */ hypervisor_callback xen_do_hypervisor_callback has_error_code=0
-#endif
-
/*
* Reload gs selector with exception handling
* edi: new selector
@@ -1125,9 +1121,10 @@ SYM_FUNC_END(do_softirq_own_stack)
* So, on entry to the handler we detect whether we interrupted an
* existing activation in its critical region -- if so, we pop the current
* activation and restart the handler using the previous one.
+ *
+ * C calling convention: exc_xen_hypervisor_callback(struct *pt_regs)
*/
-/* do_hypervisor_callback(struct *pt_regs) */
-SYM_CODE_START_LOCAL(xen_do_hypervisor_callback)
+SYM_CODE_START_LOCAL(exc_xen_hypervisor_callback)
/*
* Since we don't modify %rdi, evtchn_do_upall(struct *pt_regs) will
@@ -1145,7 +1142,7 @@ SYM_CODE_START_LOCAL(xen_do_hypervisor_c
call xen_maybe_preempt_hcall
#endif
jmp error_exit
-SYM_CODE_END(xen_do_hypervisor_callback)
+SYM_CODE_END(exc_xen_hypervisor_callback)
/*
* Hypervisor uses this for application faults while it executes.
--- a/arch/x86/include/asm/idtentry.h
+++ b/arch/x86/include/asm/idtentry.h
@@ -308,6 +308,13 @@ static __always_inline void __##func(str
#endif /* __ASSEMBLY__ */
+/*
+ * Dummy trap number so the low level ASM macro vector number checks do not
+ * match which results in emitting plain IDTENTRY stubs without bells and
+ * whistels.
+ */
+#define X86_TRAP_OTHER ~0uL
+
/* Simple exception entries: */
DECLARE_IDTENTRY(X86_TRAP_DE, exc_divide_error);
DECLARE_IDTENTRY(X86_TRAP_BP, exc_int3);
@@ -348,4 +355,10 @@ DECLARE_IDTENTRY_XEN(X86_TRAP_DB, debug)
DECLARE_IDTENTRY_DF(X86_TRAP_DF, exc_double_fault);
#endif
+#ifdef CONFIG_XEN_PV
+DECLARE_IDTENTRY(X6_TRAP_OTHER, exc_xen_hypervisor_callback);
+#endif
+
+#undef X86_TRAP_OTHER
+
#endif
--- a/arch/x86/xen/setup.c
+++ b/arch/x86/xen/setup.c
@@ -20,6 +20,7 @@
#include <asm/setup.h>
#include <asm/acpi.h>
#include <asm/numa.h>
+#include <asm/idtentry.h>
#include <asm/xen/hypervisor.h>
#include <asm/xen/hypercall.h>
@@ -993,7 +994,8 @@ void __init xen_pvmmu_arch_setup(void)
HYPERVISOR_vm_assist(VMASST_CMD_enable,
VMASST_TYPE_pae_extended_cr3);
- if (register_callback(CALLBACKTYPE_event, xen_hypervisor_callback) ||
+ if (register_callback(CALLBACKTYPE_event,
+ asm_exc_xen_hypervisor_callback) ||
register_callback(CALLBACKTYPE_failsafe, xen_failsafe_callback))
BUG();
--- a/arch/x86/xen/smp_pv.c
+++ b/arch/x86/xen/smp_pv.c
@@ -27,6 +27,7 @@
#include <asm/paravirt.h>
#include <asm/desc.h>
#include <asm/pgtable.h>
+#include <asm/idtentry.h>
#include <asm/cpu.h>
#include <xen/interface/xen.h>
@@ -346,7 +347,7 @@ cpu_initialize_context(unsigned int cpu,
ctxt->gs_base_kernel = per_cpu_offset(cpu);
#endif
ctxt->event_callback_eip =
- (unsigned long)xen_hypervisor_callback;
+ (unsigned long)asm_exc_xen_hypervisor_callback;
ctxt->failsafe_callback_eip =
(unsigned long)xen_failsafe_callback;
per_cpu(xen_cr3, cpu) = __pa(swapper_pg_dir);
--- a/arch/x86/xen/xen-asm_32.S
+++ b/arch/x86/xen/xen-asm_32.S
@@ -93,7 +93,7 @@ SYM_CODE_START(xen_iret)
/*
* If there's something pending, mask events again so we can
- * jump back into xen_hypervisor_callback. Otherwise do not
+ * jump back into exc_xen_hypervisor_callback. Otherwise do not
* touch XEN_vcpu_info_mask.
*/
jne 1f
@@ -113,7 +113,7 @@ SYM_CODE_START(xen_iret)
* Events are masked, so jumping out of the critical region is
* OK.
*/
- je xen_hypervisor_callback
+ je asm_exc_xen_hypervisor_callback
1: iret
xen_iret_end_crit:
@@ -127,7 +127,7 @@ SYM_CODE_END(xen_iret)
.globl xen_iret_start_crit, xen_iret_end_crit
/*
- * This is called by xen_hypervisor_callback in entry_32.S when it sees
+ * This is called by exc_xen_hypervisor_callback in entry_32.S when it sees
* that the EIP at the time of interrupt was between
* xen_iret_start_crit and xen_iret_end_crit.
*
@@ -144,7 +144,7 @@ SYM_CODE_END(xen_iret)
* eflags }
* cs } nested exception info
* eip }
- * return address : (into xen_hypervisor_callback)
+ * return address : (into asm_exc_xen_hypervisor_callback)
*
* In order to deliver the nested exception properly, we need to discard the
* nested exception frame such that when we handle the exception, we do it
--- a/arch/x86/xen/xen-asm_64.S
+++ b/arch/x86/xen/xen-asm_64.S
@@ -54,7 +54,7 @@ xen_pv_trap asm_exc_simd_coprocessor_err
#ifdef CONFIG_IA32_EMULATION
xen_pv_trap entry_INT80_compat
#endif
-xen_pv_trap hypervisor_callback
+xen_pv_trap asm_exc_xen_hypervisor_callback
__INIT
SYM_CODE_START(xen_early_idt_handler_array)
--- a/arch/x86/xen/xen-ops.h
+++ b/arch/x86/xen/xen-ops.h
@@ -8,7 +8,6 @@
#include <xen/xen-ops.h>
/* These are code, but not functions. Defined in entry.S */
-extern const char xen_hypervisor_callback[];
extern const char xen_failsafe_callback[];
void xen_sysenter_target(void);
^ permalink raw reply [flat|nested] 31+ messages in thread
* [patch 12/16] x86/entry/64: Simplify idtentry_body
2020-02-25 22:33 [patch 00/16] x86/entry: Consolidation - Part IV Thomas Gleixner
` (10 preceding siblings ...)
2020-02-25 22:33 ` [patch 11/16] x86/entry: Switch XEN/PV hypercall entry to IDTENTRY Thomas Gleixner
@ 2020-02-25 22:33 ` Thomas Gleixner
2020-02-25 22:33 ` [patch 13/16] x86/entry: Move irqflags and context tracking to C for simple idtentries Thomas Gleixner
` (3 subsequent siblings)
15 siblings, 0 replies; 31+ messages in thread
From: Thomas Gleixner @ 2020-02-25 22:33 UTC (permalink / raw)
To: LKML
Cc: x86, Steven Rostedt, Brian Gerst, Juergen Gross, Paolo Bonzini,
Arnd Bergmann
All C functions which do not have an error code have been converted to the
new IDTENTRY interface which does not expect an error code in the
arguments. Spare the XORL.
Signed-off-by: Thomas Gleixner <tglx@linutronix.de>
---
arch/x86/entry/entry_64.S | 2 --
1 file changed, 2 deletions(-)
--- a/arch/x86/entry/entry_64.S
+++ b/arch/x86/entry/entry_64.S
@@ -525,8 +525,6 @@ SYM_CODE_END(spurious_entries_start)
.if \has_error_code == 1
movq ORIG_RAX(%rsp), %rsi /* get error code into 2nd argument*/
movq $-1, ORIG_RAX(%rsp) /* no syscall to restart */
- .else
- xorl %esi, %esi /* Clear the error code */
.endif
.if \vector == X86_TRAP_PF
^ permalink raw reply [flat|nested] 31+ messages in thread
* [patch 13/16] x86/entry: Move irqflags and context tracking to C for simple idtentries
2020-02-25 22:33 [patch 00/16] x86/entry: Consolidation - Part IV Thomas Gleixner
` (11 preceding siblings ...)
2020-02-25 22:33 ` [patch 12/16] x86/entry/64: Simplify idtentry_body Thomas Gleixner
@ 2020-02-25 22:33 ` Thomas Gleixner
2020-02-26 8:05 ` Peter Zijlstra
2020-02-25 22:33 ` [patch 14/16] x86/entry: Provide IDTENTRY_CR2 Thomas Gleixner
` (2 subsequent siblings)
15 siblings, 1 reply; 31+ messages in thread
From: Thomas Gleixner @ 2020-02-25 22:33 UTC (permalink / raw)
To: LKML
Cc: x86, Steven Rostedt, Brian Gerst, Juergen Gross, Paolo Bonzini,
Arnd Bergmann
Now that everything is converted to the new IDTENTRY mechansim, move the
irq tracing and the invocation of enter_to_user_mode() to C code, i.e. into
the idtentry_enter() inline which is invoked through the IDTENTRY magic.
Signed-off-by: Thomas Gleixner <tglx@linutronix.de>
---
arch/x86/entry/entry_32.S | 1 -
arch/x86/entry/entry_64.S | 9 ---------
arch/x86/include/asm/idtentry.h | 19 ++++++++++++++++++-
3 files changed, 18 insertions(+), 11 deletions(-)
--- a/arch/x86/entry/entry_32.S
+++ b/arch/x86/entry/entry_32.S
@@ -1452,7 +1452,6 @@ SYM_CODE_START_LOCAL_NOALIGN(common_exce
movl PT_ORIG_EAX(%esp), %edx # get the error code
movl $-1, PT_ORIG_EAX(%esp) # no syscall to restart
- TRACE_IRQS_OFF
movl %esp, %eax # pt_regs pointer
CALL_NOSPEC %edi
jmp ret_from_exception
--- a/arch/x86/entry/entry_64.S
+++ b/arch/x86/entry/entry_64.S
@@ -511,15 +511,6 @@ SYM_CODE_END(spurious_entries_start)
GET_CR2_INTO(%r12);
.endif
- TRACE_IRQS_OFF
-
-#ifdef CONFIG_CONTEXT_TRACKING
- testb $3, CS(%rsp)
- jz .Lfrom_kernel_no_ctxt_tracking_\@
- CALL_enter_from_user_mode
-.Lfrom_kernel_no_ctxt_tracking_\@:
-#endif
-
movq %rsp, %rdi /* pt_regs pointer into 1st argument*/
.if \has_error_code == 1
--- a/arch/x86/include/asm/idtentry.h
+++ b/arch/x86/include/asm/idtentry.h
@@ -7,14 +7,31 @@
#ifndef __ASSEMBLY__
+#ifdef CONFIG_CONTEXT_TRACKING
+static __always_inline void enter_from_user_context(void)
+{
+ CT_WARN_ON(ct_state() != CONTEXT_USER);
+ user_exit_irqoff();
+}
+#else
+static __always_inline void enter_from_user_context(void) { }
+#endif
+
/**
* idtentry_enter - Handle state tracking on idtentry
* @regs: Pointer to pt_regs of interrupted context
*
- * Place holder for now.
+ * Invokes:
+ * - The hardirq tracer to keep the state consistent as low level ASM
+ * entry disabled interrupts.
+ *
+ * - Context tracking if the exception hit user mode
*/
static __always_inline void idtentry_enter(struct pt_regs *regs)
{
+ trace_hardirqs_off();
+ if (user_mode(regs))
+ enter_from_user_context();
}
/**
^ permalink raw reply [flat|nested] 31+ messages in thread
* Re: [patch 13/16] x86/entry: Move irqflags and context tracking to C for simple idtentries
2020-02-25 22:33 ` [patch 13/16] x86/entry: Move irqflags and context tracking to C for simple idtentries Thomas Gleixner
@ 2020-02-26 8:05 ` Peter Zijlstra
2020-02-26 9:20 ` Peter Zijlstra
2020-02-26 17:05 ` Frederic Weisbecker
0 siblings, 2 replies; 31+ messages in thread
From: Peter Zijlstra @ 2020-02-26 8:05 UTC (permalink / raw)
To: Thomas Gleixner
Cc: LKML, x86, Steven Rostedt, Brian Gerst, Juergen Gross,
Paolo Bonzini, Arnd Bergmann
On Tue, Feb 25, 2020 at 11:33:34PM +0100, Thomas Gleixner wrote:
> --- a/arch/x86/include/asm/idtentry.h
> +++ b/arch/x86/include/asm/idtentry.h
> @@ -7,14 +7,31 @@
>
> #ifndef __ASSEMBLY__
>
> +#ifdef CONFIG_CONTEXT_TRACKING
> +static __always_inline void enter_from_user_context(void)
> +{
> + CT_WARN_ON(ct_state() != CONTEXT_USER);
> + user_exit_irqoff();
> +}
> +#else
> +static __always_inline void enter_from_user_context(void) { }
> +#endif
> +
> /**
> * idtentry_enter - Handle state tracking on idtentry
> * @regs: Pointer to pt_regs of interrupted context
> *
> - * Place holder for now.
> + * Invokes:
> + * - The hardirq tracer to keep the state consistent as low level ASM
> + * entry disabled interrupts.
> + *
> + * - Context tracking if the exception hit user mode
> */
> static __always_inline void idtentry_enter(struct pt_regs *regs)
> {
> + trace_hardirqs_off();
> + if (user_mode(regs))
> + enter_from_user_context();
> }
So:
asm_exc_int3
exc_int3
idtentry_enter()
enter_from_user_context
if (context_tracking_enabled())
poke_int3_handler();
Is, AFAICT, completely buggered.
You can't have a static_branch before the poke_int3_handler that deals
with text_poke.
^ permalink raw reply [flat|nested] 31+ messages in thread
* Re: [patch 13/16] x86/entry: Move irqflags and context tracking to C for simple idtentries
2020-02-26 8:05 ` Peter Zijlstra
@ 2020-02-26 9:20 ` Peter Zijlstra
2020-02-26 15:11 ` Andy Lutomirski
2020-02-26 17:05 ` Frederic Weisbecker
1 sibling, 1 reply; 31+ messages in thread
From: Peter Zijlstra @ 2020-02-26 9:20 UTC (permalink / raw)
To: Thomas Gleixner
Cc: LKML, x86, Steven Rostedt, Brian Gerst, Juergen Gross,
Paolo Bonzini, Arnd Bergmann
On Wed, Feb 26, 2020 at 09:05:38AM +0100, Peter Zijlstra wrote:
> On Tue, Feb 25, 2020 at 11:33:34PM +0100, Thomas Gleixner wrote:
>
> > --- a/arch/x86/include/asm/idtentry.h
> > +++ b/arch/x86/include/asm/idtentry.h
> > @@ -7,14 +7,31 @@
> >
> > #ifndef __ASSEMBLY__
> >
> > +#ifdef CONFIG_CONTEXT_TRACKING
> > +static __always_inline void enter_from_user_context(void)
> > +{
> > + CT_WARN_ON(ct_state() != CONTEXT_USER);
> > + user_exit_irqoff();
> > +}
> > +#else
> > +static __always_inline void enter_from_user_context(void) { }
> > +#endif
> > +
> > /**
> > * idtentry_enter - Handle state tracking on idtentry
> > * @regs: Pointer to pt_regs of interrupted context
> > *
> > - * Place holder for now.
> > + * Invokes:
> > + * - The hardirq tracer to keep the state consistent as low level ASM
> > + * entry disabled interrupts.
> > + *
> > + * - Context tracking if the exception hit user mode
> > */
> > static __always_inline void idtentry_enter(struct pt_regs *regs)
> > {
> > + trace_hardirqs_off();
> > + if (user_mode(regs))
> > + enter_from_user_context();
> > }
>
> So:
>
> asm_exc_int3
> exc_int3
> idtentry_enter()
> enter_from_user_context
> if (context_tracking_enabled())
>
> poke_int3_handler();
>
> Is, AFAICT, completely buggered.
>
> You can't have a static_branch before the poke_int3_handler that deals
> with text_poke.
Forgot to say that this isn't new with this patch, just noticed it when
chasing after tracepoints.
After my patch series we can actually fix this by moving the
poke_int3_handler() before idtentry_enter().
^ permalink raw reply [flat|nested] 31+ messages in thread
* Re: [patch 13/16] x86/entry: Move irqflags and context tracking to C for simple idtentries
2020-02-26 9:20 ` Peter Zijlstra
@ 2020-02-26 15:11 ` Andy Lutomirski
2020-02-26 16:28 ` Peter Zijlstra
0 siblings, 1 reply; 31+ messages in thread
From: Andy Lutomirski @ 2020-02-26 15:11 UTC (permalink / raw)
To: Peter Zijlstra
Cc: Thomas Gleixner, LKML, X86 ML, Steven Rostedt, Brian Gerst,
Juergen Gross, Paolo Bonzini, Arnd Bergmann
On Wed, Feb 26, 2020 at 1:20 AM Peter Zijlstra <peterz@infradead.org> wrote:
>
> On Wed, Feb 26, 2020 at 09:05:38AM +0100, Peter Zijlstra wrote:
> > On Tue, Feb 25, 2020 at 11:33:34PM +0100, Thomas Gleixner wrote:
> >
> > > --- a/arch/x86/include/asm/idtentry.h
> > > +++ b/arch/x86/include/asm/idtentry.h
> > > @@ -7,14 +7,31 @@
> > >
> > > #ifndef __ASSEMBLY__
> > >
> > > +#ifdef CONFIG_CONTEXT_TRACKING
> > > +static __always_inline void enter_from_user_context(void)
> > > +{
> > > + CT_WARN_ON(ct_state() != CONTEXT_USER);
> > > + user_exit_irqoff();
> > > +}
> > > +#else
> > > +static __always_inline void enter_from_user_context(void) { }
> > > +#endif
> > > +
> > > /**
> > > * idtentry_enter - Handle state tracking on idtentry
> > > * @regs: Pointer to pt_regs of interrupted context
> > > *
> > > - * Place holder for now.
> > > + * Invokes:
> > > + * - The hardirq tracer to keep the state consistent as low level ASM
> > > + * entry disabled interrupts.
> > > + *
> > > + * - Context tracking if the exception hit user mode
> > > */
> > > static __always_inline void idtentry_enter(struct pt_regs *regs)
> > > {
> > > + trace_hardirqs_off();
> > > + if (user_mode(regs))
> > > + enter_from_user_context();
> > > }
> >
> > So:
> >
> > asm_exc_int3
> > exc_int3
> > idtentry_enter()
> > enter_from_user_context
> > if (context_tracking_enabled())
> >
> > poke_int3_handler();
> >
> > Is, AFAICT, completely buggered.
> >
> > You can't have a static_branch before the poke_int3_handler that deals
> > with text_poke.
>
> Forgot to say that this isn't new with this patch, just noticed it when
> chasing after tracepoints.
>
> After my patch series we can actually fix this by moving the
> poke_int3_handler() before idtentry_enter().
In some sense, this is a weakness of the magic macro approach. Some
of the entries just want to have code that runs before all the entry
fixups. This is an example of it. So are the cr2 reads. It can all
be made to work, but it's a bit gross.
^ permalink raw reply [flat|nested] 31+ messages in thread
* Re: [patch 13/16] x86/entry: Move irqflags and context tracking to C for simple idtentries
2020-02-26 15:11 ` Andy Lutomirski
@ 2020-02-26 16:28 ` Peter Zijlstra
2020-02-26 19:15 ` Andy Lutomirski
0 siblings, 1 reply; 31+ messages in thread
From: Peter Zijlstra @ 2020-02-26 16:28 UTC (permalink / raw)
To: Andy Lutomirski
Cc: Thomas Gleixner, LKML, X86 ML, Steven Rostedt, Brian Gerst,
Juergen Gross, Paolo Bonzini, Arnd Bergmann
On Wed, Feb 26, 2020 at 07:11:39AM -0800, Andy Lutomirski wrote:
> In some sense, this is a weakness of the magic macro approach. Some
> of the entries just want to have code that runs before all the entry
> fixups. This is an example of it. So are the cr2 reads. It can all
> be made to work, but it's a bit gross.
Right. In my current pile (new patche since last posting) I also have
one that makes #DB save-clear/restore DR7.
I got it early enough that only a watchpoint on the task stack can still
screw us over, since I also included your patch that excludes
cpu_entry_area.
Pushing it earlier still would require calling into C from the entry
stack, which I know is on your todo list, but we're not quite there yet.
^ permalink raw reply [flat|nested] 31+ messages in thread
* Re: [patch 13/16] x86/entry: Move irqflags and context tracking to C for simple idtentries
2020-02-26 16:28 ` Peter Zijlstra
@ 2020-02-26 19:15 ` Andy Lutomirski
2020-02-26 20:25 ` Thomas Gleixner
0 siblings, 1 reply; 31+ messages in thread
From: Andy Lutomirski @ 2020-02-26 19:15 UTC (permalink / raw)
To: Peter Zijlstra
Cc: Thomas Gleixner, LKML, X86 ML, Steven Rostedt, Brian Gerst,
Juergen Gross, Paolo Bonzini, Arnd Bergmann
> On Feb 26, 2020, at 8:28 AM, Peter Zijlstra <peterz@infradead.org> wrote:
>
> On Wed, Feb 26, 2020 at 07:11:39AM -0800, Andy Lutomirski wrote:
>
>> In some sense, this is a weakness of the magic macro approach. Some
>> of the entries just want to have code that runs before all the entry
>> fixups. This is an example of it. So are the cr2 reads. It can all
>> be made to work, but it's a bit gross.
>
> Right. In my current pile (new patche since last posting) I also have
> one that makes #DB save-clear/restore DR7.
>
> I got it early enough that only a watchpoint on the task stack can still
> screw us over, since I also included your patch that excludes
> cpu_entry_area.
Hmm. It would be nice to prevent watchpoints on the task stack, but that would need some trickery. It could be done.
>
> Pushing it earlier still would require calling into C from the entry
> stack, which I know is on your todo list, but we're not quite there yet.
Indeed.
This is my main objection to the DEFINE_IDTENTRY stuff. It’s *great* for the easy cases, but it’s not so great for the nasty cases. Maybe we should open code PF, MC, DB, etc. (And kill the kvm special case for PF. I have a working patch for that and I can send it.)
Anyway, this isn’t actually what I was concerned about. I meant DR6, not DR7. Specifically, if we get traced too early in do_debug / exc_debug, we can recursively debug and clobber DR6. The result will be incorrect behavior in the outer do_debug. We can fix this the same way we handle CR2. I just haven’t done it on the existing entry code because it’s too messy.
^ permalink raw reply [flat|nested] 31+ messages in thread
* Re: [patch 13/16] x86/entry: Move irqflags and context tracking to C for simple idtentries
2020-02-26 19:15 ` Andy Lutomirski
@ 2020-02-26 20:25 ` Thomas Gleixner
0 siblings, 0 replies; 31+ messages in thread
From: Thomas Gleixner @ 2020-02-26 20:25 UTC (permalink / raw)
To: Andy Lutomirski, Peter Zijlstra
Cc: LKML, X86 ML, Steven Rostedt, Brian Gerst, Juergen Gross,
Paolo Bonzini, Arnd Bergmann
Andy Lutomirski <luto@amacapital.net> writes:
>> On Feb 26, 2020, at 8:28 AM, Peter Zijlstra <peterz@infradead.org> wrote:
>>
>> On Wed, Feb 26, 2020 at 07:11:39AM -0800, Andy Lutomirski wrote:
>>
>>> In some sense, this is a weakness of the magic macro approach. Some
>>> of the entries just want to have code that runs before all the entry
>>> fixups. This is an example of it. So are the cr2 reads. It can all
>>> be made to work, but it's a bit gross.
>>
>> Right. In my current pile (new patche since last posting) I also have
>> one that makes #DB save-clear/restore DR7.
>>
>> I got it early enough that only a watchpoint on the task stack can still
>> screw us over, since I also included your patch that excludes
>> cpu_entry_area.
>
> Hmm. It would be nice to prevent watchpoints on the task stack, but that would need some trickery. It could be done.
>
>>
>> Pushing it earlier still would require calling into C from the entry
>> stack, which I know is on your todo list, but we're not quite there yet.
>
> Indeed.
>
> This is my main objection to the DEFINE_IDTENTRY stuff. It’s *great*
> for the easy cases, but it’s not so great for the nasty cases. Maybe
> we should open code PF, MC, DB, etc. (And kill the kvm special case
> for PF. I have a working patch for that and I can send it.)
I'm fine with that. I like the obvious easy ones as it spares to
duplicate all the same crap all over the place.
Making the nasty ones have:
#define DEFINE_IDTENTRY_$NASTY(exc_name) \
__visible void notrace exc_name(args....)
would solve this and you can stick whatever you want into it and have
the NOPROBE added manually. Hmm?
Thanks,
tglx
^ permalink raw reply [flat|nested] 31+ messages in thread
* Re: [patch 13/16] x86/entry: Move irqflags and context tracking to C for simple idtentries
2020-02-26 8:05 ` Peter Zijlstra
2020-02-26 9:20 ` Peter Zijlstra
@ 2020-02-26 17:05 ` Frederic Weisbecker
2020-02-26 17:09 ` Andy Lutomirski
1 sibling, 1 reply; 31+ messages in thread
From: Frederic Weisbecker @ 2020-02-26 17:05 UTC (permalink / raw)
To: Peter Zijlstra
Cc: Thomas Gleixner, LKML, x86, Steven Rostedt, Brian Gerst,
Juergen Gross, Paolo Bonzini, Arnd Bergmann
On Wed, Feb 26, 2020 at 09:05:38AM +0100, Peter Zijlstra wrote:
> On Tue, Feb 25, 2020 at 11:33:34PM +0100, Thomas Gleixner wrote:
>
> > --- a/arch/x86/include/asm/idtentry.h
> > +++ b/arch/x86/include/asm/idtentry.h
> > @@ -7,14 +7,31 @@
> >
> > #ifndef __ASSEMBLY__
> >
> > +#ifdef CONFIG_CONTEXT_TRACKING
> > +static __always_inline void enter_from_user_context(void)
> > +{
> > + CT_WARN_ON(ct_state() != CONTEXT_USER);
> > + user_exit_irqoff();
> > +}
> > +#else
> > +static __always_inline void enter_from_user_context(void) { }
> > +#endif
> > +
> > /**
> > * idtentry_enter - Handle state tracking on idtentry
> > * @regs: Pointer to pt_regs of interrupted context
> > *
> > - * Place holder for now.
> > + * Invokes:
> > + * - The hardirq tracer to keep the state consistent as low level ASM
> > + * entry disabled interrupts.
> > + *
> > + * - Context tracking if the exception hit user mode
> > */
> > static __always_inline void idtentry_enter(struct pt_regs *regs)
> > {
> > + trace_hardirqs_off();
> > + if (user_mode(regs))
> > + enter_from_user_context();
> > }
>
> So:
>
> asm_exc_int3
> exc_int3
> idtentry_enter()
> enter_from_user_context
> if (context_tracking_enabled())
>
> poke_int3_handler();
>
> Is, AFAICT, completely buggered.
>
> You can't have a static_branch before the poke_int3_handler that deals
> with text_poke.
#BP is treated like an NMI in your patchset IIRC?
In that case and since that implies we can't schedule, we can remove
the call to context tracking there once we have nmi_enter()/nmi_exit()
called unconditionally.
^ permalink raw reply [flat|nested] 31+ messages in thread
* Re: [patch 13/16] x86/entry: Move irqflags and context tracking to C for simple idtentries
2020-02-26 17:05 ` Frederic Weisbecker
@ 2020-02-26 17:09 ` Andy Lutomirski
2020-02-26 17:17 ` Frederic Weisbecker
0 siblings, 1 reply; 31+ messages in thread
From: Andy Lutomirski @ 2020-02-26 17:09 UTC (permalink / raw)
To: Frederic Weisbecker
Cc: Peter Zijlstra, Thomas Gleixner, LKML, x86, Steven Rostedt,
Brian Gerst, Juergen Gross, Paolo Bonzini, Arnd Bergmann
> On Feb 26, 2020, at 9:05 AM, Frederic Weisbecker <frederic@kernel.org> wrote:
>
> On Wed, Feb 26, 2020 at 09:05:38AM +0100, Peter Zijlstra wrote:
>>> On Tue, Feb 25, 2020 at 11:33:34PM +0100, Thomas Gleixner wrote:
>>>
>>> --- a/arch/x86/include/asm/idtentry.h
>>> +++ b/arch/x86/include/asm/idtentry.h
>>> @@ -7,14 +7,31 @@
>>>
>>> #ifndef __ASSEMBLY__
>>>
>>> +#ifdef CONFIG_CONTEXT_TRACKING
>>> +static __always_inline void enter_from_user_context(void)
>>> +{
>>> + CT_WARN_ON(ct_state() != CONTEXT_USER);
>>> + user_exit_irqoff();
>>> +}
>>> +#else
>>> +static __always_inline void enter_from_user_context(void) { }
>>> +#endif
>>> +
>>> /**
>>> * idtentry_enter - Handle state tracking on idtentry
>>> * @regs: Pointer to pt_regs of interrupted context
>>> *
>>> - * Place holder for now.
>>> + * Invokes:
>>> + * - The hardirq tracer to keep the state consistent as low level ASM
>>> + * entry disabled interrupts.
>>> + *
>>> + * - Context tracking if the exception hit user mode
>>> */
>>> static __always_inline void idtentry_enter(struct pt_regs *regs)
>>> {
>>> + trace_hardirqs_off();
>>> + if (user_mode(regs))
>>> + enter_from_user_context();
>>> }
>>
>> So:
>>
>> asm_exc_int3
>> exc_int3
>> idtentry_enter()
>> enter_from_user_context
>> if (context_tracking_enabled())
>>
>> poke_int3_handler();
>>
>> Is, AFAICT, completely buggered.
>>
>> You can't have a static_branch before the poke_int3_handler that deals
>> with text_poke.
>
> #BP is treated like an NMI in your patchset IIRC?
> In that case and since that implies we can't schedule, we can remove
> the call to context tracking there once we have nmi_enter()/nmi_exit()
> called unconditionally.
int3 from user mode can send signals. This has better be able to schedule by the time it hits prepare_exit_to_usermode.
^ permalink raw reply [flat|nested] 31+ messages in thread
* Re: [patch 13/16] x86/entry: Move irqflags and context tracking to C for simple idtentries
2020-02-26 17:09 ` Andy Lutomirski
@ 2020-02-26 17:17 ` Frederic Weisbecker
0 siblings, 0 replies; 31+ messages in thread
From: Frederic Weisbecker @ 2020-02-26 17:17 UTC (permalink / raw)
To: Andy Lutomirski
Cc: Peter Zijlstra, Thomas Gleixner, LKML, x86, Steven Rostedt,
Brian Gerst, Juergen Gross, Paolo Bonzini, Arnd Bergmann
On Wed, Feb 26, 2020 at 09:09:34AM -0800, Andy Lutomirski wrote:
>
>
> > On Feb 26, 2020, at 9:05 AM, Frederic Weisbecker <frederic@kernel.org> wrote:
> >
> > On Wed, Feb 26, 2020 at 09:05:38AM +0100, Peter Zijlstra wrote:
> >>> On Tue, Feb 25, 2020 at 11:33:34PM +0100, Thomas Gleixner wrote:
> >>>
> >>> --- a/arch/x86/include/asm/idtentry.h
> >>> +++ b/arch/x86/include/asm/idtentry.h
> >>> @@ -7,14 +7,31 @@
> >>>
> >>> #ifndef __ASSEMBLY__
> >>>
> >>> +#ifdef CONFIG_CONTEXT_TRACKING
> >>> +static __always_inline void enter_from_user_context(void)
> >>> +{
> >>> + CT_WARN_ON(ct_state() != CONTEXT_USER);
> >>> + user_exit_irqoff();
> >>> +}
> >>> +#else
> >>> +static __always_inline void enter_from_user_context(void) { }
> >>> +#endif
> >>> +
> >>> /**
> >>> * idtentry_enter - Handle state tracking on idtentry
> >>> * @regs: Pointer to pt_regs of interrupted context
> >>> *
> >>> - * Place holder for now.
> >>> + * Invokes:
> >>> + * - The hardirq tracer to keep the state consistent as low level ASM
> >>> + * entry disabled interrupts.
> >>> + *
> >>> + * - Context tracking if the exception hit user mode
> >>> */
> >>> static __always_inline void idtentry_enter(struct pt_regs *regs)
> >>> {
> >>> + trace_hardirqs_off();
> >>> + if (user_mode(regs))
> >>> + enter_from_user_context();
> >>> }
> >>
> >> So:
> >>
> >> asm_exc_int3
> >> exc_int3
> >> idtentry_enter()
> >> enter_from_user_context
> >> if (context_tracking_enabled())
> >>
> >> poke_int3_handler();
> >>
> >> Is, AFAICT, completely buggered.
> >>
> >> You can't have a static_branch before the poke_int3_handler that deals
> >> with text_poke.
> >
> > #BP is treated like an NMI in your patchset IIRC?
> > In that case and since that implies we can't schedule, we can remove
> > the call to context tracking there once we have nmi_enter()/nmi_exit()
> > called unconditionally.
>
> int3 from user mode can send signals. This has better be able to schedule by the time it hits prepare_exit_to_usermode.
Oh right...
^ permalink raw reply [flat|nested] 31+ messages in thread
* [patch 14/16] x86/entry: Provide IDTENTRY_CR2
2020-02-25 22:33 [patch 00/16] x86/entry: Consolidation - Part IV Thomas Gleixner
` (12 preceding siblings ...)
2020-02-25 22:33 ` [patch 13/16] x86/entry: Move irqflags and context tracking to C for simple idtentries Thomas Gleixner
@ 2020-02-25 22:33 ` Thomas Gleixner
2020-02-25 22:33 ` [patch 15/16] x86/entry: Switch page fault exceptions to idtentry_simple Thomas Gleixner
2020-02-25 22:33 ` [patch 16/16] x86/entry: Disable interrupts in IDTENTRY Thomas Gleixner
15 siblings, 0 replies; 31+ messages in thread
From: Thomas Gleixner @ 2020-02-25 22:33 UTC (permalink / raw)
To: LKML
Cc: x86, Steven Rostedt, Brian Gerst, Juergen Gross, Paolo Bonzini,
Arnd Bergmann
Provide a CR2 aware variant of IDTENTRY which reads CR2 right at the begin
of the C entry point before invoking irq tracing and
enter_from_user_mode(). This allows to move the CR2 handling to C code.
Signed-off-by: Thomas Gleixner <tglx@linutronix.de>
---
arch/x86/include/asm/idtentry.h | 45 ++++++++++++++++++++++++++++++++++++++++
1 file changed, 45 insertions(+)
--- a/arch/x86/include/asm/idtentry.h
+++ b/arch/x86/include/asm/idtentry.h
@@ -128,6 +128,48 @@ NOKPROBE_SYMBOL(func); \
static __always_inline void __##func(struct pt_regs *regs, \
unsigned long error_code)
+/**
+ * DECLARE_IDTENTRY_CR2 - Declare functions for fault handling IDT entry points
+ * @vector: Vector number (ignored for C)
+ * @func: Function name of the entry point
+ *
+ * Declares three functions:
+ * - The ASM entry point: asm_##func
+ * - The XEN PV trap entry point: xen_##func (maybe unused)
+ * - The C handler called from the ASM entry point
+ */
+#define DECLARE_IDTENTRY_CR2(vector, func) \
+ asmlinkage void asm_##func(void); \
+ asmlinkage void xen_asm_##func(void); \
+ __visible void func(struct pt_regs *regs, unsigned long error_code)
+
+/**
+ * DEFINE_IDTENTRY_CR2 - Emit code for fault handling IDT entry points
+ * @func: Function name of the entry point
+ *
+ * Same as IDTENTRY_ERRORCODE but reads CR2 before invoking
+ * idtentry_enter() and hands the CR2 address into the function body.
+ */
+#define DEFINE_IDTENTRY_CR2(func) \
+static __always_inline void __##func(struct pt_regs *regs, \
+ unsigned long error_code, \
+ unsigned long address); \
+ \
+__visible notrace void func(struct pt_regs *regs, \
+ unsigned long error_code) \
+{ \
+ unsigned long address = read_cr2(); \
+ \
+ idtentry_enter(regs); \
+ __##func (regs, error_code, address); \
+ idtentry_exit(regs); \
+} \
+NOKPROBE_SYMBOL(func); \
+ \
+static __always_inline void __##func(struct pt_regs *regs, \
+ unsigned long error_code, \
+ unsigned long address)
+
#ifdef CONFIG_X86_64
/**
* DECLARE_IDTENTRY_IST - Declare functions for IST handling IDT entry points
@@ -291,6 +333,9 @@ static __always_inline void __##func(str
#define DECLARE_IDTENTRY_ERRORCODE(vector, func) \
idtentry vector asm_##func func has_error_code=1
+#define DECLARE_IDTENTRY_CR2(vector, func) \
+ DECLARE_IDTENTRY_ERRORCODE(vector, func)
+
/* Special case for 32bit IRET 'trap'. Do not emit ASM code */
#define DECLARE_IDTENTRY_SW(vector, func)
^ permalink raw reply [flat|nested] 31+ messages in thread
* [patch 15/16] x86/entry: Switch page fault exceptions to idtentry_simple
2020-02-25 22:33 [patch 00/16] x86/entry: Consolidation - Part IV Thomas Gleixner
` (13 preceding siblings ...)
2020-02-25 22:33 ` [patch 14/16] x86/entry: Provide IDTENTRY_CR2 Thomas Gleixner
@ 2020-02-25 22:33 ` Thomas Gleixner
2020-03-05 21:51 ` Andy Lutomirski
2020-02-25 22:33 ` [patch 16/16] x86/entry: Disable interrupts in IDTENTRY Thomas Gleixner
15 siblings, 1 reply; 31+ messages in thread
From: Thomas Gleixner @ 2020-02-25 22:33 UTC (permalink / raw)
To: LKML
Cc: x86, Steven Rostedt, Brian Gerst, Juergen Gross, Paolo Bonzini,
Arnd Bergmann
Convert page fault exceptions to IDTENTRY_CR2:
- Implement the C entry point with DEFINE_IDTENTRY
- Emit the ASM stub with DECLARE_IDTENTRY
- Remove the ASM idtentry in 64bit
- Remove the CR2 read from 64bit
- Remove the open coded ASM entry code in 32bit
- Fixup the XEN/PV code
- Remove the old prototyoes
No functional change.
Signed-off-by: Thomas Gleixner <tglx@linutronix.de>
---
arch/x86/entry/entry_32.S | 38 --------------------------------------
arch/x86/entry/entry_64.S | 33 +++++----------------------------
arch/x86/include/asm/idtentry.h | 6 ++++++
arch/x86/include/asm/kvm_para.h | 1 -
arch/x86/include/asm/traps.h | 14 +++-----------
arch/x86/kernel/idt.c | 4 ++--
arch/x86/kernel/kvm.c | 8 +++-----
arch/x86/mm/fault.c | 20 +++++++++++++-------
arch/x86/xen/enlighten_pv.c | 2 +-
arch/x86/xen/xen-asm_64.S | 2 +-
10 files changed, 34 insertions(+), 94 deletions(-)
--- a/arch/x86/entry/entry_32.S
+++ b/arch/x86/entry/entry_32.S
@@ -1407,36 +1407,6 @@ BUILD_INTERRUPT3(hv_stimer0_callback_vec
#endif /* CONFIG_HYPERV */
-SYM_CODE_START(page_fault)
- ASM_CLAC
- pushl $do_page_fault
- jmp common_exception_read_cr2
-SYM_CODE_END(page_fault)
-
-SYM_CODE_START_LOCAL_NOALIGN(common_exception_read_cr2)
- /* the function address is in %gs's slot on the stack */
- SAVE_ALL switch_stacks=1 skip_gs=1 unwind_espfix=1
-
- ENCODE_FRAME_POINTER
-
- /* fixup %gs */
- GS_TO_REG %ecx
- movl PT_GS(%esp), %edi
- REG_TO_PTGS %ecx
- SET_KERNEL_GS %ecx
-
- GET_CR2_INTO(%ecx) # might clobber %eax
-
- /* fixup orig %eax */
- movl PT_ORIG_EAX(%esp), %edx # get the error code
- movl $-1, PT_ORIG_EAX(%esp) # no syscall to restart
-
- TRACE_IRQS_OFF
- movl %esp, %eax # pt_regs pointer
- CALL_NOSPEC %edi
- jmp ret_from_exception
-SYM_CODE_END(common_exception_read_cr2)
-
SYM_CODE_START_LOCAL_NOALIGN(common_exception)
/* the function address is in %gs's slot on the stack */
SAVE_ALL switch_stacks=1 skip_gs=1 unwind_espfix=1
@@ -1601,14 +1571,6 @@ SYM_CODE_START(asm_exc_nmi)
#endif
SYM_CODE_END(asm_exc_nmi)
-#ifdef CONFIG_KVM_GUEST
-SYM_CODE_START(async_page_fault)
- ASM_CLAC
- pushl $do_async_page_fault
- jmp common_exception_read_cr2
-SYM_CODE_END(async_page_fault)
-#endif
-
SYM_CODE_START(rewind_stack_do_exit)
/* Prevent any naive code from trying to unwind to our caller. */
xorl %ebp, %ebp
--- a/arch/x86/entry/entry_64.S
+++ b/arch/x86/entry/entry_64.S
@@ -493,24 +493,14 @@ SYM_CODE_END(spurious_entries_start)
/**
* idtentry_body - Macro to emit code calling the C function
- * @vector: Vector number
* @cfunc: C function to be called
* @has_error_code: Hardware pushed error code on stack
*/
-.macro idtentry_body vector cfunc has_error_code:req
+.macro idtentry_body cfunc has_error_code:req
call error_entry
UNWIND_HINT_REGS
- .if \vector == X86_TRAP_PF
- /*
- * Store CR2 early so subsequent faults cannot clobber it. Use R12 as
- * intermediate storage as RDX can be clobbered in enter_from_user_mode().
- * GET_CR2_INTO can clobber RAX.
- */
- GET_CR2_INTO(%r12);
- .endif
-
movq %rsp, %rdi /* pt_regs pointer into 1st argument*/
.if \has_error_code == 1
@@ -518,10 +508,6 @@ SYM_CODE_END(spurious_entries_start)
movq $-1, ORIG_RAX(%rsp) /* no syscall to restart */
.endif
- .if \vector == X86_TRAP_PF
- movq %r12, %rdx /* Move CR2 into 3rd argument */
- .endif
-
call \cfunc
jmp error_exit
@@ -560,7 +546,7 @@ SYM_CODE_START(\asmsym)
.Lfrom_usermode_no_gap_\@:
.endif
- idtentry_body \vector \cfunc \has_error_code
+ idtentry_body \cfunc \has_error_code
_ASM_NOKPROBE(\asmsym)
SYM_CODE_END(\asmsym)
@@ -635,7 +621,7 @@ SYM_CODE_START(\asmsym)
/* Switch to the regular task stack and use the noist entry point */
.Lfrom_usermode_switch_stack_\@:
- idtentry_body vector noist_\cfunc, has_error_code=0
+ idtentry_body noist_\cfunc, has_error_code=0
_ASM_NOKPROBE(\asmsym)
SYM_CODE_END(\asmsym)
@@ -1040,18 +1026,9 @@ apicinterrupt IRQ_WORK_VECTOR irq_work
#endif
/*
- * Exception entry points.
+ * Reload gs selector with exception handling
+ * edi: new selector
*/
-
-idtentry X86_TRAP_PF page_fault do_page_fault has_error_code=1
-#ifdef CONFIG_KVM_GUEST
-idtentry X86_TRAP_PF async_page_fault do_async_page_fault has_error_code=1
-#endif
-
- /*
- * Reload gs selector with exception handling
- * edi: new selector
- */
SYM_FUNC_START(native_load_gs_index)
FRAME_BEGIN
pushfq
--- a/arch/x86/include/asm/idtentry.h
+++ b/arch/x86/include/asm/idtentry.h
@@ -399,6 +399,12 @@ DECLARE_IDTENTRY_ERRORCODE(X86_TRAP_SS,
DECLARE_IDTENTRY_ERRORCODE(X86_TRAP_GP, exc_general_protection);
DECLARE_IDTENTRY_ERRORCODE(X86_TRAP_AC, exc_alignment_check);
+/* Page fault entry points */
+DECLARE_IDTENTRY_CR2(X86_TRAP_PF, exc_page_fault);
+#ifdef CONFIG_KVM_GUEST
+DECLARE_IDTENTRY_CR2(X86_TRAP_PF, exc_async_page_fault);
+#endif
+
#ifdef CONFIG_X86_MCE
/* Machine check */
DECLARE_IDTENTRY_MCE(X86_TRAP_MC, exc_machine_check);
--- a/arch/x86/include/asm/kvm_para.h
+++ b/arch/x86/include/asm/kvm_para.h
@@ -92,7 +92,6 @@ void kvm_async_pf_task_wait(u32 token, i
void kvm_async_pf_task_wake(u32 token);
u32 kvm_read_and_reset_pf_reason(void);
extern void kvm_disable_steal_time(void);
-void do_async_page_fault(struct pt_regs *regs, unsigned long error_code, unsigned long address);
#ifdef CONFIG_PARAVIRT_SPINLOCKS
void __init kvm_spinlock_init(void);
--- a/arch/x86/include/asm/traps.h
+++ b/arch/x86/include/asm/traps.h
@@ -9,17 +9,6 @@
#include <asm/idtentry.h>
#include <asm/siginfo.h> /* TRAP_TRACE, ... */
-#define dotraplinkage __visible
-
-asmlinkage void page_fault(void);
-asmlinkage void async_page_fault(void);
-
-#if defined(CONFIG_X86_64) && defined(CONFIG_XEN_PV)
-asmlinkage void xen_page_fault(void);
-#endif
-
-dotraplinkage void do_page_fault(struct pt_regs *regs, unsigned long error_code, unsigned long address);
-
#ifdef CONFIG_X86_64
asmlinkage __visible notrace struct pt_regs *sync_regs(struct pt_regs *eregs);
asmlinkage __visible notrace
@@ -27,6 +16,9 @@ struct bad_iret_stack *fixup_bad_iret(st
void __init trap_init(void);
#endif
+void native_do_page_fault(struct pt_regs *regs, unsigned long error_code,
+ unsigned long address);
+
static inline int get_si_code(unsigned long condition)
{
if (condition & DR_STEP)
--- a/arch/x86/kernel/idt.c
+++ b/arch/x86/kernel/idt.c
@@ -59,7 +59,7 @@ static const __initconst struct idt_data
INTG(X86_TRAP_DB, asm_exc_debug),
SYSG(X86_TRAP_BP, asm_exc_int3),
#ifdef CONFIG_X86_32
- INTG(X86_TRAP_PF, page_fault),
+ INTG(X86_TRAP_PF, asm_exc_page_fault),
#endif
};
@@ -153,7 +153,7 @@ static const __initconst struct idt_data
* stacks work only after cpu_init().
*/
static const __initconst struct idt_data early_pf_idts[] = {
- INTG(X86_TRAP_PF, page_fault),
+ INTG(X86_TRAP_PF, asm_exc_page_fault),
};
/*
--- a/arch/x86/kernel/kvm.c
+++ b/arch/x86/kernel/kvm.c
@@ -242,12 +242,11 @@ u32 kvm_read_and_reset_pf_reason(void)
EXPORT_SYMBOL_GPL(kvm_read_and_reset_pf_reason);
NOKPROBE_SYMBOL(kvm_read_and_reset_pf_reason);
-dotraplinkage void
-do_async_page_fault(struct pt_regs *regs, unsigned long error_code, unsigned long address)
+DEFINE_IDTENTRY_CR2(exc_async_page_fault)
{
switch (kvm_read_and_reset_pf_reason()) {
default:
- do_page_fault(regs, error_code, address);
+ native_do_page_fault(regs, error_code, address);
break;
case KVM_PV_REASON_PAGE_NOT_PRESENT:
/* page is swapped out by the host. */
@@ -260,7 +259,6 @@ do_async_page_fault(struct pt_regs *regs
break;
}
}
-NOKPROBE_SYMBOL(do_async_page_fault);
static void __init paravirt_ops_setup(void)
{
@@ -572,7 +570,7 @@ static int kvm_cpu_down_prepare(unsigned
static void __init kvm_apf_trap_init(void)
{
- update_intr_gate(X86_TRAP_PF, async_page_fault);
+ update_intr_gate(X86_TRAP_PF, asm_exc_async_page_fault);
}
static DEFINE_PER_CPU(cpumask_var_t, __pv_tlb_mask);
--- a/arch/x86/mm/fault.c
+++ b/arch/x86/mm/fault.c
@@ -1500,20 +1500,26 @@ trace_page_fault_entries(struct pt_regs
trace_page_fault_kernel(address, regs, error_code);
}
-dotraplinkage void
-do_page_fault(struct pt_regs *regs, unsigned long hw_error_code,
- unsigned long address)
+/* Invoked from do_aync_page_fault() */
+void notrace native_do_page_fault(struct pt_regs *regs,
+ unsigned long error_code,
+ unsigned long address)
{
prefetchw(¤t->mm->mmap_sem);
- trace_page_fault_entries(regs, hw_error_code, address);
+ trace_page_fault_entries(regs, error_code, address);
if (unlikely(kmmio_fault(regs, address)))
return;
/* Was the fault on kernel-controlled part of the address space? */
if (unlikely(fault_in_kernel_space(address)))
- do_kern_addr_fault(regs, hw_error_code, address);
+ do_kern_addr_fault(regs, error_code, address);
else
- do_user_addr_fault(regs, hw_error_code, address);
+ do_user_addr_fault(regs, error_code, address);
+}
+NOKPROBE_SYMBOL(native_do_page_fault);
+
+DEFINE_IDTENTRY_CR2(exc_page_fault)
+{
+ native_do_page_fault(regs, error_code, address);
}
-NOKPROBE_SYMBOL(do_page_fault);
--- a/arch/x86/xen/enlighten_pv.c
+++ b/arch/x86/xen/enlighten_pv.c
@@ -624,7 +624,7 @@ static struct trap_array_entry trap_arra
#ifdef CONFIG_IA32_EMULATION
{ entry_INT80_compat, xen_entry_INT80_compat, false },
#endif
- { page_fault, xen_page_fault, false },
+ TRAP_ENTRY(exc_page_fault, false ),
TRAP_ENTRY(exc_divide_error, false ),
TRAP_ENTRY(exc_bounds, false ),
TRAP_ENTRY(exc_invalid_op, false ),
--- a/arch/x86/xen/xen-asm_64.S
+++ b/arch/x86/xen/xen-asm_64.S
@@ -43,7 +43,7 @@ xen_pv_trap asm_exc_invalid_tss
xen_pv_trap asm_exc_segment_not_present
xen_pv_trap asm_exc_stack_segment
xen_pv_trap asm_exc_general_protection
-xen_pv_trap page_fault
+xen_pv_trap asm_exc_page_fault
xen_pv_trap asm_exc_spurious_interrupt_bug
xen_pv_trap asm_exc_coprocessor_error
xen_pv_trap asm_exc_alignment_check
^ permalink raw reply [flat|nested] 31+ messages in thread
* Re: [patch 15/16] x86/entry: Switch page fault exceptions to idtentry_simple
2020-02-25 22:33 ` [patch 15/16] x86/entry: Switch page fault exceptions to idtentry_simple Thomas Gleixner
@ 2020-03-05 21:51 ` Andy Lutomirski
2020-03-05 23:02 ` Thomas Gleixner
0 siblings, 1 reply; 31+ messages in thread
From: Andy Lutomirski @ 2020-03-05 21:51 UTC (permalink / raw)
To: Thomas Gleixner, LKML
Cc: x86, Steven Rostedt, Brian Gerst, Juergen Gross, Paolo Bonzini,
Arnd Bergmann
On 2/25/20 2:33 PM, Thomas Gleixner wrote:
> Convert page fault exceptions to IDTENTRY_CR2:
> - Implement the C entry point with DEFINE_IDTENTRY
> - Emit the ASM stub with DECLARE_IDTENTRY
> - Remove the ASM idtentry in 64bit
> - Remove the CR2 read from 64bit
> - Remove the open coded ASM entry code in 32bit
> - Fixup the XEN/PV code
> - Remove the old prototyoes
$SUBJECT says idtentry_simple. I think you meant IDTENTRY_CR2.
(I typed this a while ago and apparently failed to hit send. I'm not
sure it's still relevant.)
^ permalink raw reply [flat|nested] 31+ messages in thread
* Re: [patch 15/16] x86/entry: Switch page fault exceptions to idtentry_simple
2020-03-05 21:51 ` Andy Lutomirski
@ 2020-03-05 23:02 ` Thomas Gleixner
0 siblings, 0 replies; 31+ messages in thread
From: Thomas Gleixner @ 2020-03-05 23:02 UTC (permalink / raw)
To: Andy Lutomirski, LKML
Cc: x86, Steven Rostedt, Brian Gerst, Juergen Gross, Paolo Bonzini,
Arnd Bergmann
Andy Lutomirski <luto@kernel.org> writes:
> On 2/25/20 2:33 PM, Thomas Gleixner wrote:
>> Convert page fault exceptions to IDTENTRY_CR2:
>> - Implement the C entry point with DEFINE_IDTENTRY
>> - Emit the ASM stub with DECLARE_IDTENTRY
>> - Remove the ASM idtentry in 64bit
>> - Remove the CR2 read from 64bit
>> - Remove the open coded ASM entry code in 32bit
>> - Fixup the XEN/PV code
>> - Remove the old prototyoes
>
> $SUBJECT says idtentry_simple. I think you meant IDTENTRY_CR2.
Yes.
> (I typed this a while ago and apparently failed to hit send. I'm not
> sure it's still relevant.)
No, as we agreed to create raw IDTENTRY points for stuff which does not
fall into the trivial category.
I've reworked the mess, but now I'm tripping over the brilliant stuff in
that async PF code. The rcu_irq_enter()/exit() dance there is really
great and of course done unconditionally even if called from the VM exit
PF handling code. Oh well...
Thanks,
tglx
^ permalink raw reply [flat|nested] 31+ messages in thread
* [patch 16/16] x86/entry: Disable interrupts in IDTENTRY
2020-02-25 22:33 [patch 00/16] x86/entry: Consolidation - Part IV Thomas Gleixner
` (14 preceding siblings ...)
2020-02-25 22:33 ` [patch 15/16] x86/entry: Switch page fault exceptions to idtentry_simple Thomas Gleixner
@ 2020-02-25 22:33 ` Thomas Gleixner
2020-02-26 9:23 ` Peter Zijlstra
15 siblings, 1 reply; 31+ messages in thread
From: Thomas Gleixner @ 2020-02-25 22:33 UTC (permalink / raw)
To: LKML
Cc: x86, Steven Rostedt, Brian Gerst, Juergen Gross, Paolo Bonzini,
Arnd Bergmann
Not all exceptions guarantee to return with interrupts disabled. Disable
them in idtentry_exit() which is invoked on all regular (non IST) exception
entry points. Preparatory patch for further consolidation of the return
code.
Signed-off-by: Thomas Gleixner <tglx@linutronix.de>
---
arch/x86/entry/entry_64.S | 3 +--
arch/x86/include/asm/idtentry.h | 3 ++-
2 files changed, 3 insertions(+), 3 deletions(-)
--- a/arch/x86/entry/entry_64.S
+++ b/arch/x86/entry/entry_64.S
@@ -1336,8 +1336,7 @@ SYM_CODE_END(error_entry)
SYM_CODE_START_LOCAL(error_exit)
UNWIND_HINT_REGS
- DISABLE_INTERRUPTS(CLBR_ANY)
- TRACE_IRQS_OFF
+ DEBUG_ENTRY_ASSERT_IRQS_OFF
testb $3, CS(%rsp)
jz retint_kernel
jmp .Lretint_user
--- a/arch/x86/include/asm/idtentry.h
+++ b/arch/x86/include/asm/idtentry.h
@@ -37,10 +37,11 @@ static __always_inline void idtentry_ent
/**
* idtentry_exit - Prepare returning to low level ASM code
*
- * Place holder for now.
+ * Disables interrupts before returning
*/
static __always_inline void idtentry_exit(struct pt_regs *regs)
{
+ local_irq_disable();
}
/**
^ permalink raw reply [flat|nested] 31+ messages in thread
* Re: [patch 16/16] x86/entry: Disable interrupts in IDTENTRY
2020-02-25 22:33 ` [patch 16/16] x86/entry: Disable interrupts in IDTENTRY Thomas Gleixner
@ 2020-02-26 9:23 ` Peter Zijlstra
2020-02-26 20:21 ` Thomas Gleixner
0 siblings, 1 reply; 31+ messages in thread
From: Peter Zijlstra @ 2020-02-26 9:23 UTC (permalink / raw)
To: Thomas Gleixner
Cc: LKML, x86, Steven Rostedt, Brian Gerst, Juergen Gross,
Paolo Bonzini, Arnd Bergmann
On Tue, Feb 25, 2020 at 11:33:37PM +0100, Thomas Gleixner wrote:
> Not all exceptions guarantee to return with interrupts disabled. Disable
> them in idtentry_exit() which is invoked on all regular (non IST) exception
> entry points. Preparatory patch for further consolidation of the return
> code.
ISTR a patch that undoes cond_local_irq_enable() in the various trap
handlers. Did that get lost or is that still on the TODO list somewhere?
Once we do that, this can turn into an assertion that IRQs are in fact
disabled.
^ permalink raw reply [flat|nested] 31+ messages in thread
* Re: [patch 16/16] x86/entry: Disable interrupts in IDTENTRY
2020-02-26 9:23 ` Peter Zijlstra
@ 2020-02-26 20:21 ` Thomas Gleixner
2020-02-27 8:41 ` Peter Zijlstra
0 siblings, 1 reply; 31+ messages in thread
From: Thomas Gleixner @ 2020-02-26 20:21 UTC (permalink / raw)
To: Peter Zijlstra
Cc: LKML, x86, Steven Rostedt, Brian Gerst, Juergen Gross,
Paolo Bonzini, Arnd Bergmann
Peter Zijlstra <peterz@infradead.org> writes:
> On Tue, Feb 25, 2020 at 11:33:37PM +0100, Thomas Gleixner wrote:
>> Not all exceptions guarantee to return with interrupts disabled. Disable
>> them in idtentry_exit() which is invoked on all regular (non IST) exception
>> entry points. Preparatory patch for further consolidation of the return
>> code.
>
> ISTR a patch that undoes cond_local_irq_enable() in the various trap
> handlers. Did that get lost or is that still on the TODO list
> somewhere?
Hmm. I ditched it after we decided that fixing the #PF cases is ugly as
hell. Lemme find that stuff again.
> Once we do that, this can turn into an assertion that IRQs are in fact
> disabled.
Right.
Thanks,
tglx
^ permalink raw reply [flat|nested] 31+ messages in thread
* Re: [patch 16/16] x86/entry: Disable interrupts in IDTENTRY
2020-02-26 20:21 ` Thomas Gleixner
@ 2020-02-27 8:41 ` Peter Zijlstra
0 siblings, 0 replies; 31+ messages in thread
From: Peter Zijlstra @ 2020-02-27 8:41 UTC (permalink / raw)
To: Thomas Gleixner
Cc: LKML, x86, Steven Rostedt, Brian Gerst, Juergen Gross,
Paolo Bonzini, Arnd Bergmann
On Wed, Feb 26, 2020 at 09:21:11PM +0100, Thomas Gleixner wrote:
> Peter Zijlstra <peterz@infradead.org> writes:
>
> > On Tue, Feb 25, 2020 at 11:33:37PM +0100, Thomas Gleixner wrote:
> >> Not all exceptions guarantee to return with interrupts disabled. Disable
> >> them in idtentry_exit() which is invoked on all regular (non IST) exception
> >> entry points. Preparatory patch for further consolidation of the return
> >> code.
> >
> > ISTR a patch that undoes cond_local_irq_enable() in the various trap
> > handlers. Did that get lost or is that still on the TODO list
> > somewhere?
>
> Hmm. I ditched it after we decided that fixing the #PF cases is ugly as
> hell. Lemme find that stuff again.
Hmm, vague memories. Anyway, we can leave that for later, but maybe put
in a comment so we don't forget.
^ permalink raw reply [flat|nested] 31+ messages in thread