* [PATCH v3 0/7] x86/entry: simplify and unify SAVE/POP_REGS
@ 2018-02-11 10:49 Dominik Brodowski
2018-02-11 10:49 ` [PATCH v3 1/7] x86/entry: merge SAVE_C_REGS and SAVE_EXTRA_REGS, remove unused extensions Dominik Brodowski
` (7 more replies)
0 siblings, 8 replies; 33+ messages in thread
From: Dominik Brodowski @ 2018-02-11 10:49 UTC (permalink / raw)
To: linux-kernel, mingo, x86; +Cc: dan.j.williams, tglx, ak, torvalds, luto
The starting point for this series was the intention to interleave
the register clearing with register PUSH or MOV instructions, in order
to minimize the cost of the additional instructions required for the
register clearing. While at it, I noticed that a couple of macros in
arch/x86/entry/calling.h are unused and can be cleaned up.
Based on a preliminary version of this patch,[1] Linus suggested to
merge further codepaths and to use PUSH instead of MOV, as this should
be faster on newer CPUs.
Those who are worried about the increase in text size will find a
patch 8/7 which, based on a suggestion by Liunus, plays some trickery to
move PUSH_AND_CLEAR_REGS from the idtentry macro to the error_entry and
paranoid_entry functions. As things stand, this patch is not meant for
inclusion upstream, as it is unclear whether this additional trickery is
really worth the effort.
Changes since v2:[2]
- update commit message to patch 6/7 to include an explanation by
Ingo Molnar on why we should not be worried too much about the increase
in text size
- fix objtool warning introduced in patch 6/7
- unwind hint improvements [jpoimboe@redhat.com]
- add testing-only patch 8/7
Changes since v1:[1]
- complete rework
[1] http://lkml.kernel.org/r/20180206212546.GA2026@light.dominikbrodowski.net and
http://lkml.kernel.org/r/20180206213202.GB2026@light.dominikbrodowski.net
[2] http://lkml.kernel.org/r/20180207201517.6518-1-linux@dominikbrodowski.net
NOTE / WARNING: Please be *extremely* stringent in reviewing these
patches, *especially* concerning patch 6/7 (x86/entry: get rid of
ALLOC_PT_GPREGS_ON_STACK and SAVE_AND_CLEAR_REGS).
Dominik Brodowski (7):
x86/entry: merge SAVE_C_REGS and SAVE_EXTRA_REGS, remove unused
extensions
x86/entry: merge POP_C_REGS and POP_EXTRA_REGS
x86/entry: interleave XOR register clearing with PUSH instructions
x86/entry: introduce PUSH_AND_CLEAN_REGS
x86/entry: use PUSH_AND_CLEAN_REGS in more cases
x86/entry: get rid of ALLOC_PT_GPREGS_ON_STACK and SAVE_AND_CLEAR_REGS
x86/entry: indent PUSH_AND_CLEAR_REGS and POP_REGS properly
arch/x86/entry/calling.h | 118 ++++++++++++++++++----------------------------
arch/x86/entry/entry_64.S | 109 +++++++-----------------------------------
2 files changed, 64 insertions(+), 163 deletions(-)
--
2.16.1
^ permalink raw reply [flat|nested] 33+ messages in thread
* [PATCH v3 1/7] x86/entry: merge SAVE_C_REGS and SAVE_EXTRA_REGS, remove unused extensions
2018-02-11 10:49 [PATCH v3 0/7] x86/entry: simplify and unify SAVE/POP_REGS Dominik Brodowski
@ 2018-02-11 10:49 ` Dominik Brodowski
2018-02-12 10:15 ` [tip:x86/pti] x86/entry/64: Merge " tip-bot for Dominik Brodowski
2018-02-13 9:00 ` tip-bot for Dominik Brodowski
2018-02-11 10:49 ` [PATCH v3 2/7] x86/entry: merge POP_C_REGS and POP_EXTRA_REGS Dominik Brodowski
` (6 subsequent siblings)
7 siblings, 2 replies; 33+ messages in thread
From: Dominik Brodowski @ 2018-02-11 10:49 UTC (permalink / raw)
To: linux-kernel, mingo, x86; +Cc: dan.j.williams, tglx, ak, torvalds, luto
All current code paths call SAVE_C_REGS and then immediately
SAVE_EXTRA_REGS. Therefore, merge these two macros and order the MOV
sequeneces properly.
While at it, remove the macros to save all except specific registers,
as these macros have been unused for a long time.
Suggested-by: Linus Torvalds <torvalds@linux-foundation.org>
Signed-off-by: Dominik Brodowski <linux@dominikbrodowski.net>
---
arch/x86/entry/calling.h | 57 +++++++++++++----------------------------------
arch/x86/entry/entry_64.S | 12 ++++------
2 files changed, 19 insertions(+), 50 deletions(-)
diff --git a/arch/x86/entry/calling.h b/arch/x86/entry/calling.h
index f4b129d4af42..8907a6593b42 100644
--- a/arch/x86/entry/calling.h
+++ b/arch/x86/entry/calling.h
@@ -101,49 +101,22 @@ For 32-bit we have the following conventions - kernel is built with
addq $-(15*8), %rsp
.endm
- .macro SAVE_C_REGS_HELPER offset=0 rax=1 rcx=1 r8910=1 r11=1
- .if \r11
- movq %r11, 6*8+\offset(%rsp)
- .endif
- .if \r8910
- movq %r10, 7*8+\offset(%rsp)
- movq %r9, 8*8+\offset(%rsp)
- movq %r8, 9*8+\offset(%rsp)
- .endif
- .if \rax
- movq %rax, 10*8+\offset(%rsp)
- .endif
- .if \rcx
- movq %rcx, 11*8+\offset(%rsp)
- .endif
- movq %rdx, 12*8+\offset(%rsp)
- movq %rsi, 13*8+\offset(%rsp)
+ .macro SAVE_REGS offset=0
movq %rdi, 14*8+\offset(%rsp)
- UNWIND_HINT_REGS offset=\offset extra=0
- .endm
- .macro SAVE_C_REGS offset=0
- SAVE_C_REGS_HELPER \offset, 1, 1, 1, 1
- .endm
- .macro SAVE_C_REGS_EXCEPT_RAX_RCX offset=0
- SAVE_C_REGS_HELPER \offset, 0, 0, 1, 1
- .endm
- .macro SAVE_C_REGS_EXCEPT_R891011
- SAVE_C_REGS_HELPER 0, 1, 1, 0, 0
- .endm
- .macro SAVE_C_REGS_EXCEPT_RCX_R891011
- SAVE_C_REGS_HELPER 0, 1, 0, 0, 0
- .endm
- .macro SAVE_C_REGS_EXCEPT_RAX_RCX_R11
- SAVE_C_REGS_HELPER 0, 0, 0, 1, 0
- .endm
-
- .macro SAVE_EXTRA_REGS offset=0
- movq %r15, 0*8+\offset(%rsp)
- movq %r14, 1*8+\offset(%rsp)
- movq %r13, 2*8+\offset(%rsp)
- movq %r12, 3*8+\offset(%rsp)
- movq %rbp, 4*8+\offset(%rsp)
+ movq %rsi, 13*8+\offset(%rsp)
+ movq %rdx, 12*8+\offset(%rsp)
+ movq %rcx, 11*8+\offset(%rsp)
+ movq %rax, 10*8+\offset(%rsp)
+ movq %r8, 9*8+\offset(%rsp)
+ movq %r9, 8*8+\offset(%rsp)
+ movq %r10, 7*8+\offset(%rsp)
+ movq %r11, 6*8+\offset(%rsp)
movq %rbx, 5*8+\offset(%rsp)
+ movq %rbp, 4*8+\offset(%rsp)
+ movq %r12, 3*8+\offset(%rsp)
+ movq %r13, 2*8+\offset(%rsp)
+ movq %r14, 1*8+\offset(%rsp)
+ movq %r15, 0*8+\offset(%rsp)
UNWIND_HINT_REGS offset=\offset
.endm
@@ -197,7 +170,7 @@ For 32-bit we have the following conventions - kernel is built with
* is just setting the LSB, which makes it an invalid stack address and is also
* a signal to the unwinder that it's a pt_regs pointer in disguise.
*
- * NOTE: This macro must be used *after* SAVE_EXTRA_REGS because it corrupts
+ * NOTE: This macro must be used *after* SAVE_REGS because it corrupts
* the original rbp.
*/
.macro ENCODE_FRAME_POINTER ptregs_offset=0
diff --git a/arch/x86/entry/entry_64.S b/arch/x86/entry/entry_64.S
index 9e48002b953b..91e8d84c2496 100644
--- a/arch/x86/entry/entry_64.S
+++ b/arch/x86/entry/entry_64.S
@@ -573,8 +573,7 @@ END(irq_entries_start)
1:
ALLOC_PT_GPREGS_ON_STACK
- SAVE_C_REGS
- SAVE_EXTRA_REGS
+ SAVE_REGS
CLEAR_REGS_NOSPEC
ENCODE_FRAME_POINTER
@@ -1132,8 +1131,7 @@ ENTRY(xen_failsafe_callback)
UNWIND_HINT_IRET_REGS
pushq $-1 /* orig_ax = -1 => not a system call */
ALLOC_PT_GPREGS_ON_STACK
- SAVE_C_REGS
- SAVE_EXTRA_REGS
+ SAVE_REGS
CLEAR_REGS_NOSPEC
ENCODE_FRAME_POINTER
jmp error_exit
@@ -1178,8 +1176,7 @@ idtentry machine_check do_mce has_error_code=0 paranoid=1
ENTRY(paranoid_entry)
UNWIND_HINT_FUNC
cld
- SAVE_C_REGS 8
- SAVE_EXTRA_REGS 8
+ SAVE_REGS 8
CLEAR_REGS_NOSPEC
ENCODE_FRAME_POINTER 8
movl $1, %ebx
@@ -1231,8 +1228,7 @@ END(paranoid_exit)
ENTRY(error_entry)
UNWIND_HINT_FUNC
cld
- SAVE_C_REGS 8
- SAVE_EXTRA_REGS 8
+ SAVE_REGS 8
CLEAR_REGS_NOSPEC
ENCODE_FRAME_POINTER 8
testb $3, CS+8(%rsp)
--
2.16.1
^ permalink raw reply related [flat|nested] 33+ messages in thread
* [PATCH v3 2/7] x86/entry: merge POP_C_REGS and POP_EXTRA_REGS
2018-02-11 10:49 [PATCH v3 0/7] x86/entry: simplify and unify SAVE/POP_REGS Dominik Brodowski
2018-02-11 10:49 ` [PATCH v3 1/7] x86/entry: merge SAVE_C_REGS and SAVE_EXTRA_REGS, remove unused extensions Dominik Brodowski
@ 2018-02-11 10:49 ` Dominik Brodowski
2018-02-12 10:16 ` [tip:x86/pti] x86/entry/64: Merge the POP_C_REGS and POP_EXTRA_REGS macros into a single POP_REGS macro tip-bot for Dominik Brodowski
2018-02-13 9:01 ` tip-bot for Dominik Brodowski
2018-02-11 10:49 ` [PATCH v3 3/7] x86/entry: interleave XOR register clearing with PUSH instructions Dominik Brodowski
` (5 subsequent siblings)
7 siblings, 2 replies; 33+ messages in thread
From: Dominik Brodowski @ 2018-02-11 10:49 UTC (permalink / raw)
To: linux-kernel, mingo, x86; +Cc: dan.j.williams, tglx, ak, torvalds, luto
The two special, opencoded cases for POP_C_REGS can be handled by ASM
macros.
Signed-off-by: Dominik Brodowski <linux@dominikbrodowski.net>
---
arch/x86/entry/calling.h | 15 +++++++++++----
arch/x86/entry/entry_64.S | 26 ++++----------------------
2 files changed, 15 insertions(+), 26 deletions(-)
diff --git a/arch/x86/entry/calling.h b/arch/x86/entry/calling.h
index 8907a6593b42..3bda31736a7b 100644
--- a/arch/x86/entry/calling.h
+++ b/arch/x86/entry/calling.h
@@ -139,25 +139,32 @@ For 32-bit we have the following conventions - kernel is built with
xorq %r15, %r15
.endm
- .macro POP_EXTRA_REGS
+ .macro POP_REGS pop_rdi=1 skip_r11rcx=0
popq %r15
popq %r14
popq %r13
popq %r12
popq %rbp
popq %rbx
- .endm
-
- .macro POP_C_REGS
+ .if \skip_r11rcx
+ popq %rsi
+ .else
popq %r11
+ .endif
popq %r10
popq %r9
popq %r8
popq %rax
+ .if \skip_r11rcx
+ popq %rsi
+ .else
popq %rcx
+ .endif
popq %rdx
popq %rsi
+ .if \pop_rdi
popq %rdi
+ .endif
.endm
.macro icebp
diff --git a/arch/x86/entry/entry_64.S b/arch/x86/entry/entry_64.S
index 91e8d84c2496..286b24b650da 100644
--- a/arch/x86/entry/entry_64.S
+++ b/arch/x86/entry/entry_64.S
@@ -334,15 +334,7 @@ GLOBAL(entry_SYSCALL_64_after_hwframe)
syscall_return_via_sysret:
/* rcx and r11 are already restored (see code above) */
UNWIND_HINT_EMPTY
- POP_EXTRA_REGS
- popq %rsi /* skip r11 */
- popq %r10
- popq %r9
- popq %r8
- popq %rax
- popq %rsi /* skip rcx */
- popq %rdx
- popq %rsi
+ POP_REGS pop_rdi=0 skip_r11rcx=1
/*
* Now all regs are restored except RSP and RDI.
@@ -635,15 +627,7 @@ GLOBAL(swapgs_restore_regs_and_return_to_usermode)
ud2
1:
#endif
- POP_EXTRA_REGS
- popq %r11
- popq %r10
- popq %r9
- popq %r8
- popq %rax
- popq %rcx
- popq %rdx
- popq %rsi
+ POP_REGS pop_rdi=0
/*
* The stack is now user RDI, orig_ax, RIP, CS, EFLAGS, RSP, SS.
@@ -701,8 +685,7 @@ GLOBAL(restore_regs_and_return_to_kernel)
ud2
1:
#endif
- POP_EXTRA_REGS
- POP_C_REGS
+ POP_REGS
addq $8, %rsp /* skip regs->orig_ax */
INTERRUPT_RETURN
@@ -1661,8 +1644,7 @@ end_repeat_nmi:
nmi_swapgs:
SWAPGS_UNSAFE_STACK
nmi_restore:
- POP_EXTRA_REGS
- POP_C_REGS
+ POP_REGS
/*
* Skip orig_ax and the "outermost" frame to point RSP at the "iret"
--
2.16.1
^ permalink raw reply related [flat|nested] 33+ messages in thread
* [PATCH v3 3/7] x86/entry: interleave XOR register clearing with PUSH instructions
2018-02-11 10:49 [PATCH v3 0/7] x86/entry: simplify and unify SAVE/POP_REGS Dominik Brodowski
2018-02-11 10:49 ` [PATCH v3 1/7] x86/entry: merge SAVE_C_REGS and SAVE_EXTRA_REGS, remove unused extensions Dominik Brodowski
2018-02-11 10:49 ` [PATCH v3 2/7] x86/entry: merge POP_C_REGS and POP_EXTRA_REGS Dominik Brodowski
@ 2018-02-11 10:49 ` Dominik Brodowski
2018-02-12 10:16 ` [tip:x86/pti] x86/entry/64: Interleave " tip-bot for Dominik Brodowski
2018-02-13 9:01 ` tip-bot for Dominik Brodowski
2018-02-11 10:49 ` [PATCH v3 4/7] x86/entry: introduce PUSH_AND_CLEAN_REGS Dominik Brodowski
` (4 subsequent siblings)
7 siblings, 2 replies; 33+ messages in thread
From: Dominik Brodowski @ 2018-02-11 10:49 UTC (permalink / raw)
To: linux-kernel, mingo, x86; +Cc: dan.j.williams, tglx, ak, torvalds, luto
Same as is done for syscalls, interleave XOR with PUSH instructions
for exceptions/interrupts, in order to minimize the cost of the
additional instructions required for register clearing.
Signed-off-by: Dominik Brodowski <linux@dominikbrodowski.net>
---
arch/x86/entry/calling.h | 40 +++++++++++++++++++---------------------
arch/x86/entry/entry_64.S | 30 +++++++++++++++++++++---------
2 files changed, 40 insertions(+), 30 deletions(-)
diff --git a/arch/x86/entry/calling.h b/arch/x86/entry/calling.h
index 3bda31736a7b..a05cbb81268d 100644
--- a/arch/x86/entry/calling.h
+++ b/arch/x86/entry/calling.h
@@ -101,44 +101,42 @@ For 32-bit we have the following conventions - kernel is built with
addq $-(15*8), %rsp
.endm
- .macro SAVE_REGS offset=0
+ .macro SAVE_AND_CLEAR_REGS offset=0
+ /*
+ * Save registers and sanitize registers of values that a
+ * speculation attack might otherwise want to exploit. The
+ * lower registers are likely clobbered well before they
+ * could be put to use in a speculative execution gadget.
+ * Interleave XOR with PUSH for better uop scheduling:
+ */
movq %rdi, 14*8+\offset(%rsp)
movq %rsi, 13*8+\offset(%rsp)
movq %rdx, 12*8+\offset(%rsp)
movq %rcx, 11*8+\offset(%rsp)
movq %rax, 10*8+\offset(%rsp)
movq %r8, 9*8+\offset(%rsp)
+ xorq %r8, %r8 /* nospec r8 */
movq %r9, 8*8+\offset(%rsp)
+ xorq %r9, %r9 /* nospec r9 */
movq %r10, 7*8+\offset(%rsp)
+ xorq %r10, %r10 /* nospec r10 */
movq %r11, 6*8+\offset(%rsp)
+ xorq %r11, %r11 /* nospec r11 */
movq %rbx, 5*8+\offset(%rsp)
+ xorl %ebx, %ebx /* nospec rbx */
movq %rbp, 4*8+\offset(%rsp)
+ xorl %ebp, %ebp /* nospec rbp */
movq %r12, 3*8+\offset(%rsp)
+ xorq %r12, %r12 /* nospec r12 */
movq %r13, 2*8+\offset(%rsp)
+ xorq %r13, %r13 /* nospec r13 */
movq %r14, 1*8+\offset(%rsp)
+ xorq %r14, %r14 /* nospec r14 */
movq %r15, 0*8+\offset(%rsp)
+ xorq %r15, %r15 /* nospec r15 */
UNWIND_HINT_REGS offset=\offset
.endm
- /*
- * Sanitize registers of values that a speculation attack
- * might otherwise want to exploit. The lower registers are
- * likely clobbered well before they could be put to use in
- * a speculative execution gadget:
- */
- .macro CLEAR_REGS_NOSPEC
- xorl %ebp, %ebp
- xorl %ebx, %ebx
- xorq %r8, %r8
- xorq %r9, %r9
- xorq %r10, %r10
- xorq %r11, %r11
- xorq %r12, %r12
- xorq %r13, %r13
- xorq %r14, %r14
- xorq %r15, %r15
- .endm
-
.macro POP_REGS pop_rdi=1 skip_r11rcx=0
popq %r15
popq %r14
@@ -177,7 +175,7 @@ For 32-bit we have the following conventions - kernel is built with
* is just setting the LSB, which makes it an invalid stack address and is also
* a signal to the unwinder that it's a pt_regs pointer in disguise.
*
- * NOTE: This macro must be used *after* SAVE_REGS because it corrupts
+ * NOTE: This macro must be used *after* SAVE_AND_CLEAR_REGS because it corrupts
* the original rbp.
*/
.macro ENCODE_FRAME_POINTER ptregs_offset=0
diff --git a/arch/x86/entry/entry_64.S b/arch/x86/entry/entry_64.S
index 286b24b650da..1194814ee12b 100644
--- a/arch/x86/entry/entry_64.S
+++ b/arch/x86/entry/entry_64.S
@@ -565,8 +565,7 @@ END(irq_entries_start)
1:
ALLOC_PT_GPREGS_ON_STACK
- SAVE_REGS
- CLEAR_REGS_NOSPEC
+ SAVE_AND_CLEAR_REGS
ENCODE_FRAME_POINTER
testb $3, CS(%rsp)
@@ -1114,8 +1113,7 @@ ENTRY(xen_failsafe_callback)
UNWIND_HINT_IRET_REGS
pushq $-1 /* orig_ax = -1 => not a system call */
ALLOC_PT_GPREGS_ON_STACK
- SAVE_REGS
- CLEAR_REGS_NOSPEC
+ SAVE_AND_CLEAR_REGS
ENCODE_FRAME_POINTER
jmp error_exit
END(xen_failsafe_callback)
@@ -1159,8 +1157,7 @@ idtentry machine_check do_mce has_error_code=0 paranoid=1
ENTRY(paranoid_entry)
UNWIND_HINT_FUNC
cld
- SAVE_REGS 8
- CLEAR_REGS_NOSPEC
+ SAVE_AND_CLEAR_REGS 8
ENCODE_FRAME_POINTER 8
movl $1, %ebx
movl $MSR_GS_BASE, %ecx
@@ -1211,8 +1208,7 @@ END(paranoid_exit)
ENTRY(error_entry)
UNWIND_HINT_FUNC
cld
- SAVE_REGS 8
- CLEAR_REGS_NOSPEC
+ SAVE_AND_CLEAR_REGS 8
ENCODE_FRAME_POINTER 8
testb $3, CS+8(%rsp)
jz .Lerror_kernelspace
@@ -1399,18 +1395,34 @@ ENTRY(nmi)
pushq (%rdx) /* pt_regs->dx */
pushq %rcx /* pt_regs->cx */
pushq %rax /* pt_regs->ax */
+ /*
+ * Sanitize registers of values that a speculation attack
+ * might otherwise want to exploit. The lower registers are
+ * likely clobbered well before they could be put to use in
+ * a speculative execution gadget. Interleave XOR with PUSH
+ * for better uop scheduling:
+ */
pushq %r8 /* pt_regs->r8 */
+ xorq %r8, %r8 /* nospec r8 */
pushq %r9 /* pt_regs->r9 */
+ xorq %r9, %r9 /* nospec r9 */
pushq %r10 /* pt_regs->r10 */
+ xorq %r10, %r10 /* nospec r10 */
pushq %r11 /* pt_regs->r11 */
+ xorq %r11, %r11 /* nospec r11*/
pushq %rbx /* pt_regs->rbx */
+ xorl %ebx, %ebx /* nospec rbx*/
pushq %rbp /* pt_regs->rbp */
+ xorl %ebp, %ebp /* nospec rbp*/
pushq %r12 /* pt_regs->r12 */
+ xorq %r12, %r12 /* nospec r12*/
pushq %r13 /* pt_regs->r13 */
+ xorq %r13, %r13 /* nospec r13*/
pushq %r14 /* pt_regs->r14 */
+ xorq %r14, %r14 /* nospec r14*/
pushq %r15 /* pt_regs->r15 */
+ xorq %r15, %r15 /* nospec r15*/
UNWIND_HINT_REGS
- CLEAR_REGS_NOSPEC
ENCODE_FRAME_POINTER
/*
--
2.16.1
^ permalink raw reply related [flat|nested] 33+ messages in thread
* [PATCH v3 4/7] x86/entry: introduce PUSH_AND_CLEAN_REGS
2018-02-11 10:49 [PATCH v3 0/7] x86/entry: simplify and unify SAVE/POP_REGS Dominik Brodowski
` (2 preceding siblings ...)
2018-02-11 10:49 ` [PATCH v3 3/7] x86/entry: interleave XOR register clearing with PUSH instructions Dominik Brodowski
@ 2018-02-11 10:49 ` Dominik Brodowski
2018-02-12 10:17 ` [tip:x86/pti] x86/entry/64: Introduce the PUSH_AND_CLEAN_REGS macro tip-bot for Dominik Brodowski
2018-02-13 9:01 ` tip-bot for Dominik Brodowski
2018-02-11 10:49 ` [PATCH v3 5/7] x86/entry: use PUSH_AND_CLEAN_REGS in more cases Dominik Brodowski
` (3 subsequent siblings)
7 siblings, 2 replies; 33+ messages in thread
From: Dominik Brodowski @ 2018-02-11 10:49 UTC (permalink / raw)
To: linux-kernel, mingo, x86; +Cc: dan.j.williams, tglx, ak, torvalds, luto
Those instances where ALLOC_PT_GPREGS_ON_STACK is called just before
SAVE_AND_CLEAR_REGS can trivially be replaced by PUSH_AND_CLEAN_REGS.
This macro uses PUSH instead of MOV and should therefore be faster, at
least on newer CPUs.
Suggested-by: Linus Torvalds <torvalds@linux-foundation.org>
Signed-off-by: Dominik Brodowski <linux@dominikbrodowski.net>
---
arch/x86/entry/calling.h | 36 ++++++++++++++++++++++++++++++++++++
arch/x86/entry/entry_64.S | 6 ++----
2 files changed, 38 insertions(+), 4 deletions(-)
diff --git a/arch/x86/entry/calling.h b/arch/x86/entry/calling.h
index a05cbb81268d..57b1b87a04f0 100644
--- a/arch/x86/entry/calling.h
+++ b/arch/x86/entry/calling.h
@@ -137,6 +137,42 @@ For 32-bit we have the following conventions - kernel is built with
UNWIND_HINT_REGS offset=\offset
.endm
+ .macro PUSH_AND_CLEAR_REGS
+ /*
+ * Push registers and sanitize registers of values that a
+ * speculation attack might otherwise want to exploit. The
+ * lower registers are likely clobbered well before they
+ * could be put to use in a speculative execution gadget.
+ * Interleave XOR with PUSH for better uop scheduling:
+ */
+ pushq %rdi /* pt_regs->di */
+ pushq %rsi /* pt_regs->si */
+ pushq %rdx /* pt_regs->dx */
+ pushq %rcx /* pt_regs->cx */
+ pushq %rax /* pt_regs->ax */
+ pushq %r8 /* pt_regs->r8 */
+ xorq %r8, %r8 /* nospec r8 */
+ pushq %r9 /* pt_regs->r9 */
+ xorq %r9, %r9 /* nospec r9 */
+ pushq %r10 /* pt_regs->r10 */
+ xorq %r10, %r10 /* nospec r10 */
+ pushq %r11 /* pt_regs->r11 */
+ xorq %r11, %r11 /* nospec r11*/
+ pushq %rbx /* pt_regs->rbx */
+ xorl %ebx, %ebx /* nospec rbx*/
+ pushq %rbp /* pt_regs->rbp */
+ xorl %ebp, %ebp /* nospec rbp*/
+ pushq %r12 /* pt_regs->r12 */
+ xorq %r12, %r12 /* nospec r12*/
+ pushq %r13 /* pt_regs->r13 */
+ xorq %r13, %r13 /* nospec r13*/
+ pushq %r14 /* pt_regs->r14 */
+ xorq %r14, %r14 /* nospec r14*/
+ pushq %r15 /* pt_regs->r15 */
+ xorq %r15, %r15 /* nospec r15*/
+ UNWIND_HINT_REGS
+ .endm
+
.macro POP_REGS pop_rdi=1 skip_r11rcx=0
popq %r15
popq %r14
diff --git a/arch/x86/entry/entry_64.S b/arch/x86/entry/entry_64.S
index 1194814ee12b..9dd3fbfdc75d 100644
--- a/arch/x86/entry/entry_64.S
+++ b/arch/x86/entry/entry_64.S
@@ -564,8 +564,7 @@ END(irq_entries_start)
call switch_to_thread_stack
1:
- ALLOC_PT_GPREGS_ON_STACK
- SAVE_AND_CLEAR_REGS
+ PUSH_AND_CLEAR_REGS
ENCODE_FRAME_POINTER
testb $3, CS(%rsp)
@@ -1112,8 +1111,7 @@ ENTRY(xen_failsafe_callback)
addq $0x30, %rsp
UNWIND_HINT_IRET_REGS
pushq $-1 /* orig_ax = -1 => not a system call */
- ALLOC_PT_GPREGS_ON_STACK
- SAVE_AND_CLEAR_REGS
+ PUSH_AND_CLEAR_REGS
ENCODE_FRAME_POINTER
jmp error_exit
END(xen_failsafe_callback)
--
2.16.1
^ permalink raw reply related [flat|nested] 33+ messages in thread
* [PATCH v3 5/7] x86/entry: use PUSH_AND_CLEAN_REGS in more cases
2018-02-11 10:49 [PATCH v3 0/7] x86/entry: simplify and unify SAVE/POP_REGS Dominik Brodowski
` (3 preceding siblings ...)
2018-02-11 10:49 ` [PATCH v3 4/7] x86/entry: introduce PUSH_AND_CLEAN_REGS Dominik Brodowski
@ 2018-02-11 10:49 ` Dominik Brodowski
2018-02-12 10:17 ` [tip:x86/pti] x86/entry/64: Use " tip-bot for Dominik Brodowski
2018-02-13 9:02 ` tip-bot for Dominik Brodowski
2018-02-11 10:49 ` [PATCH v3 6/7] x86/entry: get rid of ALLOC_PT_GPREGS_ON_STACK and SAVE_AND_CLEAR_REGS Dominik Brodowski
` (2 subsequent siblings)
7 siblings, 2 replies; 33+ messages in thread
From: Dominik Brodowski @ 2018-02-11 10:49 UTC (permalink / raw)
To: linux-kernel, mingo, x86; +Cc: dan.j.williams, tglx, ak, torvalds, luto
entry_SYSCALL_64_after_hwframe and nmi can be converted to use
PUSH_AND_CLEAN_REGS instead of opencoded variants thereof. Due to
the interleaving, the additional XOR-based clearing of r8 and r9
in entry_SYSCALL_64_after_hwframe should not have any noticeable
negative implications.
Suggested-by: Linus Torvalds <torvalds@linux-foundation.org>
Signed-off-by: Dominik Brodowski <linux@dominikbrodowski.net>
---
arch/x86/entry/calling.h | 6 ++---
arch/x86/entry/entry_64.S | 65 +++--------------------------------------------
2 files changed, 6 insertions(+), 65 deletions(-)
diff --git a/arch/x86/entry/calling.h b/arch/x86/entry/calling.h
index 57b1b87a04f0..d6a97e2945ee 100644
--- a/arch/x86/entry/calling.h
+++ b/arch/x86/entry/calling.h
@@ -137,7 +137,7 @@ For 32-bit we have the following conventions - kernel is built with
UNWIND_HINT_REGS offset=\offset
.endm
- .macro PUSH_AND_CLEAR_REGS
+ .macro PUSH_AND_CLEAR_REGS rdx=%rdx rax=%rax
/*
* Push registers and sanitize registers of values that a
* speculation attack might otherwise want to exploit. The
@@ -147,9 +147,9 @@ For 32-bit we have the following conventions - kernel is built with
*/
pushq %rdi /* pt_regs->di */
pushq %rsi /* pt_regs->si */
- pushq %rdx /* pt_regs->dx */
+ pushq \rdx /* pt_regs->dx */
pushq %rcx /* pt_regs->cx */
- pushq %rax /* pt_regs->ax */
+ pushq \rax /* pt_regs->ax */
pushq %r8 /* pt_regs->r8 */
xorq %r8, %r8 /* nospec r8 */
pushq %r9 /* pt_regs->r9 */
diff --git a/arch/x86/entry/entry_64.S b/arch/x86/entry/entry_64.S
index 9dd3fbfdc75d..9c4fe360db42 100644
--- a/arch/x86/entry/entry_64.S
+++ b/arch/x86/entry/entry_64.S
@@ -227,35 +227,8 @@ ENTRY(entry_SYSCALL_64)
pushq %rcx /* pt_regs->ip */
GLOBAL(entry_SYSCALL_64_after_hwframe)
pushq %rax /* pt_regs->orig_ax */
- pushq %rdi /* pt_regs->di */
- pushq %rsi /* pt_regs->si */
- pushq %rdx /* pt_regs->dx */
- pushq %rcx /* pt_regs->cx */
- pushq $-ENOSYS /* pt_regs->ax */
- pushq %r8 /* pt_regs->r8 */
- pushq %r9 /* pt_regs->r9 */
- pushq %r10 /* pt_regs->r10 */
- /*
- * Clear extra registers that a speculation attack might
- * otherwise want to exploit. Interleave XOR with PUSH
- * for better uop scheduling:
- */
- xorq %r10, %r10 /* nospec r10 */
- pushq %r11 /* pt_regs->r11 */
- xorq %r11, %r11 /* nospec r11 */
- pushq %rbx /* pt_regs->rbx */
- xorl %ebx, %ebx /* nospec rbx */
- pushq %rbp /* pt_regs->rbp */
- xorl %ebp, %ebp /* nospec rbp */
- pushq %r12 /* pt_regs->r12 */
- xorq %r12, %r12 /* nospec r12 */
- pushq %r13 /* pt_regs->r13 */
- xorq %r13, %r13 /* nospec r13 */
- pushq %r14 /* pt_regs->r14 */
- xorq %r14, %r14 /* nospec r14 */
- pushq %r15 /* pt_regs->r15 */
- xorq %r15, %r15 /* nospec r15 */
- UNWIND_HINT_REGS
+
+ PUSH_AND_CLEAR_REGS rax=$-ENOSYS
TRACE_IRQS_OFF
@@ -1388,39 +1361,7 @@ ENTRY(nmi)
pushq 1*8(%rdx) /* pt_regs->rip */
UNWIND_HINT_IRET_REGS
pushq $-1 /* pt_regs->orig_ax */
- pushq %rdi /* pt_regs->di */
- pushq %rsi /* pt_regs->si */
- pushq (%rdx) /* pt_regs->dx */
- pushq %rcx /* pt_regs->cx */
- pushq %rax /* pt_regs->ax */
- /*
- * Sanitize registers of values that a speculation attack
- * might otherwise want to exploit. The lower registers are
- * likely clobbered well before they could be put to use in
- * a speculative execution gadget. Interleave XOR with PUSH
- * for better uop scheduling:
- */
- pushq %r8 /* pt_regs->r8 */
- xorq %r8, %r8 /* nospec r8 */
- pushq %r9 /* pt_regs->r9 */
- xorq %r9, %r9 /* nospec r9 */
- pushq %r10 /* pt_regs->r10 */
- xorq %r10, %r10 /* nospec r10 */
- pushq %r11 /* pt_regs->r11 */
- xorq %r11, %r11 /* nospec r11*/
- pushq %rbx /* pt_regs->rbx */
- xorl %ebx, %ebx /* nospec rbx*/
- pushq %rbp /* pt_regs->rbp */
- xorl %ebp, %ebp /* nospec rbp*/
- pushq %r12 /* pt_regs->r12 */
- xorq %r12, %r12 /* nospec r12*/
- pushq %r13 /* pt_regs->r13 */
- xorq %r13, %r13 /* nospec r13*/
- pushq %r14 /* pt_regs->r14 */
- xorq %r14, %r14 /* nospec r14*/
- pushq %r15 /* pt_regs->r15 */
- xorq %r15, %r15 /* nospec r15*/
- UNWIND_HINT_REGS
+ PUSH_AND_CLEAR_REGS rdx=(%rdx)
ENCODE_FRAME_POINTER
/*
--
2.16.1
^ permalink raw reply related [flat|nested] 33+ messages in thread
* [PATCH v3 6/7] x86/entry: get rid of ALLOC_PT_GPREGS_ON_STACK and SAVE_AND_CLEAR_REGS
2018-02-11 10:49 [PATCH v3 0/7] x86/entry: simplify and unify SAVE/POP_REGS Dominik Brodowski
` (4 preceding siblings ...)
2018-02-11 10:49 ` [PATCH v3 5/7] x86/entry: use PUSH_AND_CLEAN_REGS in more cases Dominik Brodowski
@ 2018-02-11 10:49 ` Dominik Brodowski
2018-02-12 10:18 ` [tip:x86/pti] x86/entry/64: Get rid of the ALLOC_PT_GPREGS_ON_STACK and SAVE_AND_CLEAR_REGS macros tip-bot for Dominik Brodowski
2018-02-13 9:02 ` tip-bot for Dominik Brodowski
2018-02-11 10:49 ` [PATCH v3 7/7] x86/entry: indent PUSH_AND_CLEAR_REGS and POP_REGS properly Dominik Brodowski
2018-02-11 10:49 ` [PATCH v3 8/7] TESTING_ONLY x86/entry: reduce static footprint of idtentry Dominik Brodowski
7 siblings, 2 replies; 33+ messages in thread
From: Dominik Brodowski @ 2018-02-11 10:49 UTC (permalink / raw)
To: linux-kernel, mingo, x86; +Cc: dan.j.williams, tglx, ak, torvalds, luto
Previously, error_entry() and paranoid_entry() saved the GP registers
onto stack space previously allocated by its callers. Combine these two
steps in the callers, and use the generic PUSH_AND_CLEAR_REGS macro
for that.
This adds a significant amount ot text size. However, Ingo Molnar points
out that
"these numbers also _very_ significantly over-represent the
extra footprint. The assumptions that resulted in
us compressing the IRQ entry code have changed very
significantly with the new x86 IRQ allocation code we
introduced in the last year:
- IRQ vectors are usually populated in tightly clustered
groups.
With our new vector allocator code the typical per CPU
allocation percentage on x86 systems is ~3 device vectors
and ~10 fixed vectors out of ~220 vectors - i.e. a very
low ~6% utilization (!). [...]
The days where we allocated a lot of vectors on every
CPU and the compression of the IRQ entry code text
mattered are over.
- Another issue is that only a small minority of vectors
is frequent enough to actually matter to cache utilization
in practice: 3-4 key IPIs and 1-2 device IRQs at most - and
those vectors tend to be tightly clustered as well into about
two groups, and are probably already on 2-3 cache lines in
practice.
For the common case of 'cache cold' IRQs it's the depth of
the call chain and the fragmentation of the resulting I$
that should be the main performance limit - not the overall
size of it.
- The CPU side cost of IRQ delivery is still very expensive
even in the best, most cached case, as in 'over a thousand
cycles'. So much stuff is done that maybe contemporary x86
IRQ entry microcode already prefetches the IDT entry and its
expected call target address."[*]
[*] http://lkml.kernel.org/r/20180208094710.qnjixhm6hybebdv7@gmail.com
The testb $3, CS(%rsp) instruction in the idtentry macro does not need
modification. Previously, %rsp was manually decreased by 15*8; with
this patch, %rsp is decreased by 15 pushq instructions.
Suggested-by: Linus Torvalds <torvalds@linux-foundation.org>
Signed-off-by: Dominik Brodowski <linux@dominikbrodowski.net>
[jpoimboe@redhat.com: unwind hint improvements]
---
arch/x86/entry/calling.h | 42 +-----------------------------------------
arch/x86/entry/entry_64.S | 20 +++++++++-----------
2 files changed, 10 insertions(+), 52 deletions(-)
diff --git a/arch/x86/entry/calling.h b/arch/x86/entry/calling.h
index d6a97e2945ee..59675010c9a0 100644
--- a/arch/x86/entry/calling.h
+++ b/arch/x86/entry/calling.h
@@ -97,46 +97,6 @@ For 32-bit we have the following conventions - kernel is built with
#define SIZEOF_PTREGS 21*8
- .macro ALLOC_PT_GPREGS_ON_STACK
- addq $-(15*8), %rsp
- .endm
-
- .macro SAVE_AND_CLEAR_REGS offset=0
- /*
- * Save registers and sanitize registers of values that a
- * speculation attack might otherwise want to exploit. The
- * lower registers are likely clobbered well before they
- * could be put to use in a speculative execution gadget.
- * Interleave XOR with PUSH for better uop scheduling:
- */
- movq %rdi, 14*8+\offset(%rsp)
- movq %rsi, 13*8+\offset(%rsp)
- movq %rdx, 12*8+\offset(%rsp)
- movq %rcx, 11*8+\offset(%rsp)
- movq %rax, 10*8+\offset(%rsp)
- movq %r8, 9*8+\offset(%rsp)
- xorq %r8, %r8 /* nospec r8 */
- movq %r9, 8*8+\offset(%rsp)
- xorq %r9, %r9 /* nospec r9 */
- movq %r10, 7*8+\offset(%rsp)
- xorq %r10, %r10 /* nospec r10 */
- movq %r11, 6*8+\offset(%rsp)
- xorq %r11, %r11 /* nospec r11 */
- movq %rbx, 5*8+\offset(%rsp)
- xorl %ebx, %ebx /* nospec rbx */
- movq %rbp, 4*8+\offset(%rsp)
- xorl %ebp, %ebp /* nospec rbp */
- movq %r12, 3*8+\offset(%rsp)
- xorq %r12, %r12 /* nospec r12 */
- movq %r13, 2*8+\offset(%rsp)
- xorq %r13, %r13 /* nospec r13 */
- movq %r14, 1*8+\offset(%rsp)
- xorq %r14, %r14 /* nospec r14 */
- movq %r15, 0*8+\offset(%rsp)
- xorq %r15, %r15 /* nospec r15 */
- UNWIND_HINT_REGS offset=\offset
- .endm
-
.macro PUSH_AND_CLEAR_REGS rdx=%rdx rax=%rax
/*
* Push registers and sanitize registers of values that a
@@ -211,7 +171,7 @@ For 32-bit we have the following conventions - kernel is built with
* is just setting the LSB, which makes it an invalid stack address and is also
* a signal to the unwinder that it's a pt_regs pointer in disguise.
*
- * NOTE: This macro must be used *after* SAVE_AND_CLEAR_REGS because it corrupts
+ * NOTE: This macro must be used *after* PUSH_AND_CLEAR_REGS because it corrupts
* the original rbp.
*/
.macro ENCODE_FRAME_POINTER ptregs_offset=0
diff --git a/arch/x86/entry/entry_64.S b/arch/x86/entry/entry_64.S
index 9c4fe360db42..b868433d7e68 100644
--- a/arch/x86/entry/entry_64.S
+++ b/arch/x86/entry/entry_64.S
@@ -871,7 +871,9 @@ ENTRY(\sym)
pushq $-1 /* ORIG_RAX: no syscall to restart */
.endif
- ALLOC_PT_GPREGS_ON_STACK
+ /* Save all registers in pt_regs */
+ PUSH_AND_CLEAR_REGS
+ ENCODE_FRAME_POINTER
.if \paranoid < 2
testb $3, CS(%rsp) /* If coming from userspace, switch stacks */
@@ -1121,15 +1123,12 @@ idtentry machine_check do_mce has_error_code=0 paranoid=1
#endif
/*
- * Save all registers in pt_regs, and switch gs if needed.
+ * Switch gs if needed.
* Use slow, but surefire "are we in kernel?" check.
* Return: ebx=0: need swapgs on exit, ebx=1: otherwise
*/
ENTRY(paranoid_entry)
- UNWIND_HINT_FUNC
cld
- SAVE_AND_CLEAR_REGS 8
- ENCODE_FRAME_POINTER 8
movl $1, %ebx
movl $MSR_GS_BASE, %ecx
rdmsr
@@ -1142,7 +1141,7 @@ ENTRY(paranoid_entry)
SAVE_AND_SWITCH_TO_KERNEL_CR3 scratch_reg=%rax save_reg=%r14
ret
-END(paranoid_entry)
+ENDPROC(paranoid_entry)
/*
* "Paranoid" exit path from exception stack. This is invoked
@@ -1173,14 +1172,12 @@ ENTRY(paranoid_exit)
END(paranoid_exit)
/*
- * Save all registers in pt_regs, and switch gs if needed.
+ * Switch gs if needed.
* Return: EBX=0: came from user mode; EBX=1: otherwise
*/
ENTRY(error_entry)
- UNWIND_HINT_FUNC
+ UNWIND_HINT_REGS offset=8
cld
- SAVE_AND_CLEAR_REGS 8
- ENCODE_FRAME_POINTER 8
testb $3, CS+8(%rsp)
jz .Lerror_kernelspace
@@ -1571,7 +1568,8 @@ end_repeat_nmi:
* frame to point back to repeat_nmi.
*/
pushq $-1 /* ORIG_RAX: no syscall to restart */
- ALLOC_PT_GPREGS_ON_STACK
+ PUSH_AND_CLEAR_REGS
+ ENCODE_FRAME_POINTER
/*
* Use paranoid_entry to handle SWAPGS, but no need to use paranoid_exit
--
2.16.1
^ permalink raw reply related [flat|nested] 33+ messages in thread
* [PATCH v3 7/7] x86/entry: indent PUSH_AND_CLEAR_REGS and POP_REGS properly
2018-02-11 10:49 [PATCH v3 0/7] x86/entry: simplify and unify SAVE/POP_REGS Dominik Brodowski
` (5 preceding siblings ...)
2018-02-11 10:49 ` [PATCH v3 6/7] x86/entry: get rid of ALLOC_PT_GPREGS_ON_STACK and SAVE_AND_CLEAR_REGS Dominik Brodowski
@ 2018-02-11 10:49 ` Dominik Brodowski
2018-02-12 10:18 ` [tip:x86/pti] x86/entry/64: Indent " tip-bot for Dominik Brodowski
2018-02-13 9:03 ` [tip:x86/pti] x86/entry/64: Indent PUSH_AND_CLEAR_REGS and POP_REGS properly tip-bot for Dominik Brodowski
2018-02-11 10:49 ` [PATCH v3 8/7] TESTING_ONLY x86/entry: reduce static footprint of idtentry Dominik Brodowski
7 siblings, 2 replies; 33+ messages in thread
From: Dominik Brodowski @ 2018-02-11 10:49 UTC (permalink / raw)
To: linux-kernel, mingo, x86; +Cc: dan.j.williams, tglx, ak, torvalds, luto
... same as the other macros in arch/x86/entry/calling.h
Signed-off-by: Dominik Brodowski <linux@dominikbrodowski.net>
---
arch/x86/entry/calling.h | 8 ++++----
1 file changed, 4 insertions(+), 4 deletions(-)
diff --git a/arch/x86/entry/calling.h b/arch/x86/entry/calling.h
index 59675010c9a0..6985440c68fa 100644
--- a/arch/x86/entry/calling.h
+++ b/arch/x86/entry/calling.h
@@ -97,7 +97,7 @@ For 32-bit we have the following conventions - kernel is built with
#define SIZEOF_PTREGS 21*8
- .macro PUSH_AND_CLEAR_REGS rdx=%rdx rax=%rax
+.macro PUSH_AND_CLEAR_REGS rdx=%rdx rax=%rax
/*
* Push registers and sanitize registers of values that a
* speculation attack might otherwise want to exploit. The
@@ -131,9 +131,9 @@ For 32-bit we have the following conventions - kernel is built with
pushq %r15 /* pt_regs->r15 */
xorq %r15, %r15 /* nospec r15*/
UNWIND_HINT_REGS
- .endm
+.endm
- .macro POP_REGS pop_rdi=1 skip_r11rcx=0
+.macro POP_REGS pop_rdi=1 skip_r11rcx=0
popq %r15
popq %r14
popq %r13
@@ -163,7 +163,7 @@ For 32-bit we have the following conventions - kernel is built with
.macro icebp
.byte 0xf1
- .endm
+.endm
/*
* This is a sneaky trick to help the unwinder find pt_regs on the stack. The
--
2.16.1
^ permalink raw reply related [flat|nested] 33+ messages in thread
* [PATCH v3 8/7] TESTING_ONLY x86/entry: reduce static footprint of idtentry
2018-02-11 10:49 [PATCH v3 0/7] x86/entry: simplify and unify SAVE/POP_REGS Dominik Brodowski
` (6 preceding siblings ...)
2018-02-11 10:49 ` [PATCH v3 7/7] x86/entry: indent PUSH_AND_CLEAR_REGS and POP_REGS properly Dominik Brodowski
@ 2018-02-11 10:49 ` Dominik Brodowski
2018-02-12 9:37 ` Ingo Molnar
7 siblings, 1 reply; 33+ messages in thread
From: Dominik Brodowski @ 2018-02-11 10:49 UTC (permalink / raw)
To: linux-kernel, mingo, x86; +Cc: dan.j.williams, tglx, ak, torvalds, luto
Play a little trick in the generic PUSH_AND_CLEAR_REGS macro
to insert the GP registers "above" the original return address.
This allows us to (re-)insert the macro in error_entry() and
paranoid_entry() and to remove it from the idtentry macro. This
reduces the static footprint significantly.
NOTE: It still needs to be evaluated whether the reduced static
footprint at the cost of code readability is really needed here.
Co-developed-by: Linus Torvalds <torvalds@linux-foundation.org>
Signed-off-by: Dominik Brodowski <linux@dominikbrodowski.net>
---
arch/x86/entry/calling.h | 11 ++++++++++-
arch/x86/entry/entry_64.S | 21 ++++++++++-----------
2 files changed, 20 insertions(+), 12 deletions(-)
diff --git a/arch/x86/entry/calling.h b/arch/x86/entry/calling.h
index 6985440c68fa..2d4523ba04a8 100644
--- a/arch/x86/entry/calling.h
+++ b/arch/x86/entry/calling.h
@@ -97,7 +97,7 @@ For 32-bit we have the following conventions - kernel is built with
#define SIZEOF_PTREGS 21*8
-.macro PUSH_AND_CLEAR_REGS rdx=%rdx rax=%rax
+.macro PUSH_AND_CLEAR_REGS rdx=%rdx rax=%rax save_ret=0
/*
* Push registers and sanitize registers of values that a
* speculation attack might otherwise want to exploit. The
@@ -105,8 +105,14 @@ For 32-bit we have the following conventions - kernel is built with
* could be put to use in a speculative execution gadget.
* Interleave XOR with PUSH for better uop scheduling:
*/
+ .if \save_ret
+ pushq %rsi /* pt_regs->si */
+ movq 8(%rsp), %rsi /* temporarily store ret address in %rsi */
+ movq %rdi, 8(%rsp) /* pt_regs->di (overwriting original ret) */
+ .else
pushq %rdi /* pt_regs->di */
pushq %rsi /* pt_regs->si */
+ .endif
pushq \rdx /* pt_regs->dx */
pushq %rcx /* pt_regs->cx */
pushq \rax /* pt_regs->ax */
@@ -131,6 +137,9 @@ For 32-bit we have the following conventions - kernel is built with
pushq %r15 /* pt_regs->r15 */
xorq %r15, %r15 /* nospec r15*/
UNWIND_HINT_REGS
+ .if \save_ret
+ pushq %rsi /* return address on top of stack */
+ .endif
.endm
.macro POP_REGS pop_rdi=1 skip_r11rcx=0
diff --git a/arch/x86/entry/entry_64.S b/arch/x86/entry/entry_64.S
index b868433d7e68..a2e41177e390 100644
--- a/arch/x86/entry/entry_64.S
+++ b/arch/x86/entry/entry_64.S
@@ -871,12 +871,8 @@ ENTRY(\sym)
pushq $-1 /* ORIG_RAX: no syscall to restart */
.endif
- /* Save all registers in pt_regs */
- PUSH_AND_CLEAR_REGS
- ENCODE_FRAME_POINTER
-
.if \paranoid < 2
- testb $3, CS(%rsp) /* If coming from userspace, switch stacks */
+ testb $3, CS-ORIG_RAX(%rsp) /* If coming from userspace, switch stacks */
jnz .Lfrom_usermode_switch_stack_\@
.endif
@@ -1123,12 +1119,15 @@ idtentry machine_check do_mce has_error_code=0 paranoid=1
#endif
/*
- * Switch gs if needed.
+ * Save all registers in pt_regs, and switch gs if needed.
* Use slow, but surefire "are we in kernel?" check.
* Return: ebx=0: need swapgs on exit, ebx=1: otherwise
*/
ENTRY(paranoid_entry)
+ UNWIND_HINT_FUNC
cld
+ PUSH_AND_CLEAR_REGS save_ret=1
+ ENCODE_FRAME_POINTER 8
movl $1, %ebx
movl $MSR_GS_BASE, %ecx
rdmsr
@@ -1141,7 +1140,7 @@ ENTRY(paranoid_entry)
SAVE_AND_SWITCH_TO_KERNEL_CR3 scratch_reg=%rax save_reg=%r14
ret
-ENDPROC(paranoid_entry)
+END(paranoid_entry)
/*
* "Paranoid" exit path from exception stack. This is invoked
@@ -1172,12 +1171,14 @@ ENTRY(paranoid_exit)
END(paranoid_exit)
/*
- * Switch gs if needed.
+ * Save all registers in pt_regs, and switch gs if needed.
* Return: EBX=0: came from user mode; EBX=1: otherwise
*/
ENTRY(error_entry)
- UNWIND_HINT_REGS offset=8
+ UNWIND_HINT_FUNC
cld
+ PUSH_AND_CLEAR_REGS save_ret=1
+ ENCODE_FRAME_POINTER 8
testb $3, CS+8(%rsp)
jz .Lerror_kernelspace
@@ -1568,8 +1569,6 @@ end_repeat_nmi:
* frame to point back to repeat_nmi.
*/
pushq $-1 /* ORIG_RAX: no syscall to restart */
- PUSH_AND_CLEAR_REGS
- ENCODE_FRAME_POINTER
/*
* Use paranoid_entry to handle SWAPGS, but no need to use paranoid_exit
--
2.16.1
^ permalink raw reply related [flat|nested] 33+ messages in thread
* Re: [PATCH v3 8/7] TESTING_ONLY x86/entry: reduce static footprint of idtentry
2018-02-11 10:49 ` [PATCH v3 8/7] TESTING_ONLY x86/entry: reduce static footprint of idtentry Dominik Brodowski
@ 2018-02-12 9:37 ` Ingo Molnar
2018-02-12 19:17 ` Linus Torvalds
0 siblings, 1 reply; 33+ messages in thread
From: Ingo Molnar @ 2018-02-12 9:37 UTC (permalink / raw)
To: Dominik Brodowski
Cc: linux-kernel, x86, dan.j.williams, tglx, ak, torvalds, luto,
Peter Zijlstra, Denys Vlasenko, Josh Poimboeuf, Brian Gerst,
Borislav Petkov
* Dominik Brodowski <linux@dominikbrodowski.net> wrote:
> Play a little trick in the generic PUSH_AND_CLEAR_REGS macro
> to insert the GP registers "above" the original return address.
> This allows us to (re-)insert the macro in error_entry() and
> paranoid_entry() and to remove it from the idtentry macro. This
> reduces the static footprint significantly.
>
> NOTE: It still needs to be evaluated whether the reduced static
> footprint at the cost of code readability is really needed here.
>
> Co-developed-by: Linus Torvalds <torvalds@linux-foundation.org>
> Signed-off-by: Dominik Brodowski <linux@dominikbrodowski.net>
> ---
> arch/x86/entry/calling.h | 11 ++++++++++-
> arch/x86/entry/entry_64.S | 21 ++++++++++-----------
> 2 files changed, 20 insertions(+), 12 deletions(-)
>
> diff --git a/arch/x86/entry/calling.h b/arch/x86/entry/calling.h
> index 6985440c68fa..2d4523ba04a8 100644
> --- a/arch/x86/entry/calling.h
> +++ b/arch/x86/entry/calling.h
> @@ -97,7 +97,7 @@ For 32-bit we have the following conventions - kernel is built with
>
> #define SIZEOF_PTREGS 21*8
>
> -.macro PUSH_AND_CLEAR_REGS rdx=%rdx rax=%rax
> +.macro PUSH_AND_CLEAR_REGS rdx=%rdx rax=%rax save_ret=0
> /*
> * Push registers and sanitize registers of values that a
> * speculation attack might otherwise want to exploit. The
> @@ -105,8 +105,14 @@ For 32-bit we have the following conventions - kernel is built with
> * could be put to use in a speculative execution gadget.
> * Interleave XOR with PUSH for better uop scheduling:
> */
> + .if \save_ret
> + pushq %rsi /* pt_regs->si */
> + movq 8(%rsp), %rsi /* temporarily store ret address in %rsi */
> + movq %rdi, 8(%rsp) /* pt_regs->di (overwriting original ret) */
> + .else
> pushq %rdi /* pt_regs->di */
> pushq %rsi /* pt_regs->si */
> + .endif
> pushq \rdx /* pt_regs->dx */
> pushq %rcx /* pt_regs->cx */
> pushq \rax /* pt_regs->ax */
> @@ -131,6 +137,9 @@ For 32-bit we have the following conventions - kernel is built with
> pushq %r15 /* pt_regs->r15 */
> xorq %r15, %r15 /* nospec r15*/
> UNWIND_HINT_REGS
> + .if \save_ret
> + pushq %rsi /* return address on top of stack */
> + .endif
> .endm
>
> .macro POP_REGS pop_rdi=1 skip_r11rcx=0
> diff --git a/arch/x86/entry/entry_64.S b/arch/x86/entry/entry_64.S
> index b868433d7e68..a2e41177e390 100644
> --- a/arch/x86/entry/entry_64.S
> +++ b/arch/x86/entry/entry_64.S
> @@ -871,12 +871,8 @@ ENTRY(\sym)
> pushq $-1 /* ORIG_RAX: no syscall to restart */
> .endif
>
> - /* Save all registers in pt_regs */
> - PUSH_AND_CLEAR_REGS
> - ENCODE_FRAME_POINTER
> -
> .if \paranoid < 2
> - testb $3, CS(%rsp) /* If coming from userspace, switch stacks */
> + testb $3, CS-ORIG_RAX(%rsp) /* If coming from userspace, switch stacks */
> jnz .Lfrom_usermode_switch_stack_\@
> .endif
>
> @@ -1123,12 +1119,15 @@ idtentry machine_check do_mce has_error_code=0 paranoid=1
> #endif
>
> /*
> - * Switch gs if needed.
> + * Save all registers in pt_regs, and switch gs if needed.
> * Use slow, but surefire "are we in kernel?" check.
> * Return: ebx=0: need swapgs on exit, ebx=1: otherwise
> */
> ENTRY(paranoid_entry)
> + UNWIND_HINT_FUNC
> cld
> + PUSH_AND_CLEAR_REGS save_ret=1
> + ENCODE_FRAME_POINTER 8
> movl $1, %ebx
> movl $MSR_GS_BASE, %ecx
> rdmsr
> @@ -1141,7 +1140,7 @@ ENTRY(paranoid_entry)
> SAVE_AND_SWITCH_TO_KERNEL_CR3 scratch_reg=%rax save_reg=%r14
>
> ret
> -ENDPROC(paranoid_entry)
> +END(paranoid_entry)
>
> /*
> * "Paranoid" exit path from exception stack. This is invoked
> @@ -1172,12 +1171,14 @@ ENTRY(paranoid_exit)
> END(paranoid_exit)
>
> /*
> - * Switch gs if needed.
> + * Save all registers in pt_regs, and switch gs if needed.
> * Return: EBX=0: came from user mode; EBX=1: otherwise
> */
> ENTRY(error_entry)
> - UNWIND_HINT_REGS offset=8
> + UNWIND_HINT_FUNC
> cld
> + PUSH_AND_CLEAR_REGS save_ret=1
> + ENCODE_FRAME_POINTER 8
> testb $3, CS+8(%rsp)
> jz .Lerror_kernelspace
>
> @@ -1568,8 +1569,6 @@ end_repeat_nmi:
> * frame to point back to repeat_nmi.
> */
> pushq $-1 /* ORIG_RAX: no syscall to restart */
> - PUSH_AND_CLEAR_REGS
> - ENCODE_FRAME_POINTER
>
> /*
> * Use paranoid_entry to handle SWAPGS, but no need to use paranoid_exit
Ok, so this does not look _that_ complicated, and the .text savings are
significant:
text data bss dec hex filename
24182 0 0 24182 5e76 entry_64.o.before
20878 0 0 20878 518e entry_64.o.after
But, let's try to estimate the dynamic I$ impact through the static entry points.
I'm using the following list of 'static' IDT entry points:
divide_error
overflow
bounds
invalid_op
device_not_available
double_fault
coprocessor_segment_overrun
invalid_TSS
segment_not_present
spurious_interrupt_bug
coprocessor_error
alignment_check
simd_coprocessor_error
debug
int3
stack_segment
general_protection
page_fault
machine_check
nmi
apic_timer_interrupt
x86_platform_ipi
threshold_interrupt
deferred_error_interrupt
thermal_interrupt
call_function_single_interrupt
call_function_interrupt
reschedule_interrupt
error_interrupt
spurious_interrupt
irq_work_interrupt
Which are present in the vmlinux with well-known addresses - this makes I$
estimates easier.
But let's first filter these entries - a big chunk of them are legacy vectors,
never used on a modern x86 CPU built in the last decade or so. Another chunk are
infrequent entries like <machine_check>.
The entry points that are relevant to an I$ analysis are just 11 entry points:
divide_error
# overflow
# bounds
# invalid_op
# device_not_available
# double_fault
# coprocessor_segment_overrun
# invalid_TSS
# segment_not_present
# spurious_interrupt_bug
# coprocessor_error
# alignment_check
# simd_coprocessor_error
debug
int3
# stack_segment
general_protection
page_fault
# machine_check
nmi
apic_timer_interrupt
# x86_platform_ipi
# threshold_interrupt
# deferred_error_interrupt
# thermal_interrupt
call_function_single_interrupt
call_function_interrupt
reschedule_interrupt
# error_interrupt
# spurious_interrupt
irq_work_interrupt
( Also note that even this list is probably generous: for example
<apic_timer_interrupt> will only arrive once per timer IRQ, by which time it's
probably already out of the L1 cache - and <debug> only matters to a small
minority of users. )
Here's the x86-64 defconfig addresses of these entry points, on a 'vanilla' kernel
that does not have any entry point bloating (and debloating) patches applied, sha1
b7a60525e83a:
address symbol # size # offs
---------------- ------------------------------------ -------------
ffffffff81a00e70: <divide_error> # 40 # 0
ffffffff81a01250: <debug> # 60 # 3e0
ffffffff81a012b0: <int3> # 60 # 60
ffffffff81a01370: <general_protection> # 60 # c0
ffffffff81a013d0: <page_fault> # 60 # 60
ffffffff81a01680: <nmi> # cc # 2b0
ffffffff81a01a40: <apic_timer_interrupt> # c0 # 3c0
ffffffff81a01e00: <call_function_single_interrupt> # c0 # 3c0
ffffffff81a01ec0: <call_function_interrupt> # c0 # c0
ffffffff81a01f80: <reschedule_interrupt> # c0 # c0
ffffffff81a021c0: <irq_work_interrupt> # c0 # 240
The 'size' column is the size of the entry function, the 'offs' column shows the
symbol's offset from the previous entry.
As can be seen in the table:
- even in the 'compressed' layout the entries are already far enough from each
other to make the dynamic I$ footprint 27 cachelines, assuming 64 byte
cachelines:
27 == 1 + 2 + 2 + 2 + 2 + 3 + 3 + 3 + 3 + 3
After applying the simplification patches (sha: 5a10e729bc0a) the static footprint
of the entry code becomes more bloated:
text data bss dec hex filename
19693 0 0 19693 4ced arch/x86/entry/entry_64.o.before
24212 0 0 24212 5e94 arch/x86/entry/entry_64.o.after (+4519 bytes)
But the dynamic I$ footprint does not change nearly as much:
address symbol # size # offs
---------------- ------------------------------------ -------------
ffffffff81a00d00: <divide_error> # 70 # 0
ffffffff81a01320: <debug> # 90 # 620
ffffffff81a013b0: <int3> # 90 # 90
ffffffff81a014c0: <general_protection> # 80 # 110
ffffffff81a01540: <page_fault> # 80 # 80
ffffffff81a01780: <nmi> # cc # 240
ffffffff81a01b70: <apic_timer_interrupt> # 80 # 3f0
ffffffff81a01df0: <call_function_single_interrupt> # 80 # 280
ffffffff81a01e70: <call_function_interrupt> # 80 # 80
ffffffff81a01ef0: <reschedule_interrupt> # 80 # 80
ffffffff81a02070: <irq_work_interrupt> # 80 # 180
The footprint is in fact 25 cache lines:
25 == 2 + 3 + 3 + 2 + 2 + 3 + 2 + 2 + 2 + 2 + 2
The reason is that while exception entry size increased, the IRQ entry size
actually shrunk, because the PUSH based sequences are more compact than the MOVs
they used before:
# BEFORE:
ffffffff81a01f80 <reschedule_interrupt>:
[...]
ffffffff81a01f98: 48 83 c4 88 add $0xffffffffffffff88,%rsp
ffffffff81a01f9c: 4c 89 5c 24 30 mov %r11,0x30(%rsp)
ffffffff81a01fa1: 4c 89 54 24 38 mov %r10,0x38(%rsp)
ffffffff81a01fa6: 4c 89 4c 24 40 mov %r9,0x40(%rsp)
ffffffff81a01fab: 4c 89 44 24 48 mov %r8,0x48(%rsp)
ffffffff81a01fb0: 48 89 44 24 50 mov %rax,0x50(%rsp)
ffffffff81a01fb5: 48 89 4c 24 58 mov %rcx,0x58(%rsp)
ffffffff81a01fba: 48 89 54 24 60 mov %rdx,0x60(%rsp)
ffffffff81a01fbf: 48 89 74 24 68 mov %rsi,0x68(%rsp)
ffffffff81a01fc4: 48 89 7c 24 70 mov %rdi,0x70(%rsp)
ffffffff81a01fc9: 4c 89 3c 24 mov %r15,(%rsp)
ffffffff81a01fcd: 4c 89 74 24 08 mov %r14,0x8(%rsp)
ffffffff81a01fd2: 4c 89 6c 24 10 mov %r13,0x10(%rsp)
ffffffff81a01fd7: 4c 89 64 24 18 mov %r12,0x18(%rsp)
ffffffff81a01fdc: 48 89 6c 24 20 mov %rbp,0x20(%rsp)
ffffffff81a01fe1: 48 89 5c 24 28 mov %rbx,0x28(%rsp)
ffffffff81a01fe6: 31 ed xor %ebp,%ebp
ffffffff81a01fe8: 31 db xor %ebx,%ebx
ffffffff81a01fea: 4d 31 c0 xor %r8,%r8
ffffffff81a01fed: 4d 31 c9 xor %r9,%r9
ffffffff81a01ff0: 4d 31 d2 xor %r10,%r10
ffffffff81a01ff3: 4d 31 db xor %r11,%r11
ffffffff81a01ff6: 4d 31 e4 xor %r12,%r12
ffffffff81a01ff9: 4d 31 ed xor %r13,%r13
ffffffff81a01ffc: 4d 31 f6 xor %r14,%r14
ffffffff81a01fff: 4d 31 ff xor %r15,%r15
# AFTER:
ffffffff81a01ef0 <reschedule_interrupt>:
[...]
ffffffff81a01f08: 57 push %rdi
ffffffff81a01f09: 56 push %rsi
ffffffff81a01f0a: 52 push %rdx
ffffffff81a01f0b: 51 push %rcx
ffffffff81a01f0c: 50 push %rax
ffffffff81a01f0d: 41 50 push %r8
ffffffff81a01f0f: 4d 31 c0 xor %r8,%r8
ffffffff81a01f12: 41 51 push %r9
ffffffff81a01f14: 4d 31 c9 xor %r9,%r9
ffffffff81a01f17: 41 52 push %r10
ffffffff81a01f19: 4d 31 d2 xor %r10,%r10
ffffffff81a01f1c: 41 53 push %r11
ffffffff81a01f1e: 4d 31 db xor %r11,%r11
ffffffff81a01f21: 53 push %rbx
ffffffff81a01f22: 31 db xor %ebx,%ebx
ffffffff81a01f24: 55 push %rbp
ffffffff81a01f25: 31 ed xor %ebp,%ebp
ffffffff81a01f27: 41 54 push %r12
ffffffff81a01f29: 4d 31 e4 xor %r12,%r12
ffffffff81a01f2c: 41 55 push %r13
ffffffff81a01f2e: 4d 31 ed xor %r13,%r13
ffffffff81a01f31: 41 56 push %r14
ffffffff81a01f33: 4d 31 f6 xor %r14,%r14
ffffffff81a01f36: 41 57 push %r15
ffffffff81a01f38: 4d 31 ff xor %r15,%r15
Finally, applying the 8/7 debloating patch gives the following result:
address symbol # size # offs
---------------- ------------------------------------ -------------
ffffffff81a00d00: <divide_error> # 40 # 0
ffffffff81a010b0: <debug> # 50 # 3b0
ffffffff81a01100: <int3> # 50 # 50
ffffffff81a011a0: <general_protection> # 50 # a0
ffffffff81a011f0: <page_fault> # 50 # 50
ffffffff81a01450: <nmi> # cc # 260
ffffffff81a01810: <apic_timer_interrupt> # 80 # 3c0
ffffffff81a01a90: <call_function_single_interrupt> # 80 # 280
ffffffff81a01b10: <call_function_interrupt> # 80 # 80
ffffffff81a01b90: <reschedule_interrupt> # 80 # 80
ffffffff81a01d10: <irq_work_interrupt> # 80 # 180
That's 22 cachelines:
22 == 1 + 2 + 2 + 2 + 2 + 3 + 2 + 2 + 2 + 2 + 2
25 == 2 + 3 + 3 + 2 + 2 + 3 + 2 + 2 + 2 + 2 + 2
So it's a 3 cachelines improvements, coming from the smaller size of these three
entry points:
ffffffff81a00d00: <divide_error> # 40 # 0
ffffffff81a010b0: <debug> # 50 # 3b0
ffffffff81a01100: <int3> # 50 # 50
... which are all debugging/instrumentation related and are thus performance
critical only for a small minority of our users.
So I'm torn whether to apply this last patch, but the interesting, unintuitive
result is that the existing MOV->PUSH simplification patches already appear to
have improved the dynamic I$ footprint from 27 cachelines to 22 cachelines!
Thanks,
Ingo
^ permalink raw reply [flat|nested] 33+ messages in thread
* [tip:x86/pti] x86/entry/64: Merge SAVE_C_REGS and SAVE_EXTRA_REGS, remove unused extensions
2018-02-11 10:49 ` [PATCH v3 1/7] x86/entry: merge SAVE_C_REGS and SAVE_EXTRA_REGS, remove unused extensions Dominik Brodowski
@ 2018-02-12 10:15 ` tip-bot for Dominik Brodowski
2018-02-13 9:00 ` tip-bot for Dominik Brodowski
1 sibling, 0 replies; 33+ messages in thread
From: tip-bot for Dominik Brodowski @ 2018-02-12 10:15 UTC (permalink / raw)
To: linux-tip-commits
Cc: linux, tglx, hpa, dvlasenk, jpoimboe, luto, bp, peterz, torvalds,
mingo, linux-kernel, brgerst
Commit-ID: 0305150fd7ab4d181c6ad874cb19f8c2b4465888
Gitweb: https://git.kernel.org/tip/0305150fd7ab4d181c6ad874cb19f8c2b4465888
Author: Dominik Brodowski <linux@dominikbrodowski.net>
AuthorDate: Sun, 11 Feb 2018 11:49:42 +0100
Committer: Ingo Molnar <mingo@kernel.org>
CommitDate: Mon, 12 Feb 2018 08:06:35 +0100
x86/entry/64: Merge SAVE_C_REGS and SAVE_EXTRA_REGS, remove unused extensions
All current code paths call SAVE_C_REGS and then immediately
SAVE_EXTRA_REGS. Therefore, merge these two macros and order the MOV
sequeneces properly.
While at it, remove the macros to save all except specific registers,
as these macros have been unused for a long time.
Suggested-by: Linus Torvalds <torvalds@linux-foundation.org>
Signed-off-by: Dominik Brodowski <linux@dominikbrodowski.net>
Cc: Andy Lutomirski <luto@kernel.org>
Cc: Borislav Petkov <bp@alien8.de>
Cc: Brian Gerst <brgerst@gmail.com>
Cc: Denys Vlasenko <dvlasenk@redhat.com>
Cc: H. Peter Anvin <hpa@zytor.com>
Cc: Josh Poimboeuf <jpoimboe@redhat.com>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Thomas Gleixner <tglx@linutronix.de>
Cc: dan.j.williams@intel.com
Link: http://lkml.kernel.org/r/20180211104949.12992-2-linux@dominikbrodowski.net
Signed-off-by: Ingo Molnar <mingo@kernel.org>
---
arch/x86/entry/calling.h | 57 +++++++++++++----------------------------------
arch/x86/entry/entry_64.S | 12 ++++------
2 files changed, 19 insertions(+), 50 deletions(-)
diff --git a/arch/x86/entry/calling.h b/arch/x86/entry/calling.h
index f4b129d..8907a65 100644
--- a/arch/x86/entry/calling.h
+++ b/arch/x86/entry/calling.h
@@ -101,49 +101,22 @@ For 32-bit we have the following conventions - kernel is built with
addq $-(15*8), %rsp
.endm
- .macro SAVE_C_REGS_HELPER offset=0 rax=1 rcx=1 r8910=1 r11=1
- .if \r11
- movq %r11, 6*8+\offset(%rsp)
- .endif
- .if \r8910
- movq %r10, 7*8+\offset(%rsp)
- movq %r9, 8*8+\offset(%rsp)
- movq %r8, 9*8+\offset(%rsp)
- .endif
- .if \rax
- movq %rax, 10*8+\offset(%rsp)
- .endif
- .if \rcx
- movq %rcx, 11*8+\offset(%rsp)
- .endif
- movq %rdx, 12*8+\offset(%rsp)
- movq %rsi, 13*8+\offset(%rsp)
+ .macro SAVE_REGS offset=0
movq %rdi, 14*8+\offset(%rsp)
- UNWIND_HINT_REGS offset=\offset extra=0
- .endm
- .macro SAVE_C_REGS offset=0
- SAVE_C_REGS_HELPER \offset, 1, 1, 1, 1
- .endm
- .macro SAVE_C_REGS_EXCEPT_RAX_RCX offset=0
- SAVE_C_REGS_HELPER \offset, 0, 0, 1, 1
- .endm
- .macro SAVE_C_REGS_EXCEPT_R891011
- SAVE_C_REGS_HELPER 0, 1, 1, 0, 0
- .endm
- .macro SAVE_C_REGS_EXCEPT_RCX_R891011
- SAVE_C_REGS_HELPER 0, 1, 0, 0, 0
- .endm
- .macro SAVE_C_REGS_EXCEPT_RAX_RCX_R11
- SAVE_C_REGS_HELPER 0, 0, 0, 1, 0
- .endm
-
- .macro SAVE_EXTRA_REGS offset=0
- movq %r15, 0*8+\offset(%rsp)
- movq %r14, 1*8+\offset(%rsp)
- movq %r13, 2*8+\offset(%rsp)
- movq %r12, 3*8+\offset(%rsp)
- movq %rbp, 4*8+\offset(%rsp)
+ movq %rsi, 13*8+\offset(%rsp)
+ movq %rdx, 12*8+\offset(%rsp)
+ movq %rcx, 11*8+\offset(%rsp)
+ movq %rax, 10*8+\offset(%rsp)
+ movq %r8, 9*8+\offset(%rsp)
+ movq %r9, 8*8+\offset(%rsp)
+ movq %r10, 7*8+\offset(%rsp)
+ movq %r11, 6*8+\offset(%rsp)
movq %rbx, 5*8+\offset(%rsp)
+ movq %rbp, 4*8+\offset(%rsp)
+ movq %r12, 3*8+\offset(%rsp)
+ movq %r13, 2*8+\offset(%rsp)
+ movq %r14, 1*8+\offset(%rsp)
+ movq %r15, 0*8+\offset(%rsp)
UNWIND_HINT_REGS offset=\offset
.endm
@@ -197,7 +170,7 @@ For 32-bit we have the following conventions - kernel is built with
* is just setting the LSB, which makes it an invalid stack address and is also
* a signal to the unwinder that it's a pt_regs pointer in disguise.
*
- * NOTE: This macro must be used *after* SAVE_EXTRA_REGS because it corrupts
+ * NOTE: This macro must be used *after* SAVE_REGS because it corrupts
* the original rbp.
*/
.macro ENCODE_FRAME_POINTER ptregs_offset=0
diff --git a/arch/x86/entry/entry_64.S b/arch/x86/entry/entry_64.S
index 932a445..1a6fc01 100644
--- a/arch/x86/entry/entry_64.S
+++ b/arch/x86/entry/entry_64.S
@@ -573,8 +573,7 @@ END(irq_entries_start)
1:
ALLOC_PT_GPREGS_ON_STACK
- SAVE_C_REGS
- SAVE_EXTRA_REGS
+ SAVE_REGS
CLEAR_REGS_NOSPEC
ENCODE_FRAME_POINTER
@@ -1132,8 +1131,7 @@ ENTRY(xen_failsafe_callback)
UNWIND_HINT_IRET_REGS
pushq $-1 /* orig_ax = -1 => not a system call */
ALLOC_PT_GPREGS_ON_STACK
- SAVE_C_REGS
- SAVE_EXTRA_REGS
+ SAVE_REGS
CLEAR_REGS_NOSPEC
ENCODE_FRAME_POINTER
jmp error_exit
@@ -1178,8 +1176,7 @@ idtentry machine_check do_mce has_error_code=0 paranoid=1
ENTRY(paranoid_entry)
UNWIND_HINT_FUNC
cld
- SAVE_C_REGS 8
- SAVE_EXTRA_REGS 8
+ SAVE_REGS 8
CLEAR_REGS_NOSPEC
ENCODE_FRAME_POINTER 8
movl $1, %ebx
@@ -1231,8 +1228,7 @@ END(paranoid_exit)
ENTRY(error_entry)
UNWIND_HINT_FUNC
cld
- SAVE_C_REGS 8
- SAVE_EXTRA_REGS 8
+ SAVE_REGS 8
CLEAR_REGS_NOSPEC
ENCODE_FRAME_POINTER 8
testb $3, CS+8(%rsp)
^ permalink raw reply related [flat|nested] 33+ messages in thread
* [tip:x86/pti] x86/entry/64: Merge the POP_C_REGS and POP_EXTRA_REGS macros into a single POP_REGS macro
2018-02-11 10:49 ` [PATCH v3 2/7] x86/entry: merge POP_C_REGS and POP_EXTRA_REGS Dominik Brodowski
@ 2018-02-12 10:16 ` tip-bot for Dominik Brodowski
2018-02-13 9:01 ` tip-bot for Dominik Brodowski
1 sibling, 0 replies; 33+ messages in thread
From: tip-bot for Dominik Brodowski @ 2018-02-12 10:16 UTC (permalink / raw)
To: linux-tip-commits
Cc: linux, luto, torvalds, brgerst, tglx, mingo, dvlasenk,
linux-kernel, peterz, jpoimboe, hpa, bp
Commit-ID: a1490086a98e51e3358be8eaf2bc5743c73c6b6e
Gitweb: https://git.kernel.org/tip/a1490086a98e51e3358be8eaf2bc5743c73c6b6e
Author: Dominik Brodowski <linux@dominikbrodowski.net>
AuthorDate: Sun, 11 Feb 2018 11:49:43 +0100
Committer: Ingo Molnar <mingo@kernel.org>
CommitDate: Mon, 12 Feb 2018 08:06:35 +0100
x86/entry/64: Merge the POP_C_REGS and POP_EXTRA_REGS macros into a single POP_REGS macro
The two special, opencoded cases for POP_C_REGS can be handled by ASM
macros.
Signed-off-by: Dominik Brodowski <linux@dominikbrodowski.net>
Cc: Andy Lutomirski <luto@kernel.org>
Cc: Borislav Petkov <bp@alien8.de>
Cc: Brian Gerst <brgerst@gmail.com>
Cc: Denys Vlasenko <dvlasenk@redhat.com>
Cc: H. Peter Anvin <hpa@zytor.com>
Cc: Josh Poimboeuf <jpoimboe@redhat.com>
Cc: Linus Torvalds <torvalds@linux-foundation.org>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Thomas Gleixner <tglx@linutronix.de>
Cc: dan.j.williams@intel.com
Link: http://lkml.kernel.org/r/20180211104949.12992-3-linux@dominikbrodowski.net
Signed-off-by: Ingo Molnar <mingo@kernel.org>
---
arch/x86/entry/calling.h | 15 +++++++++++----
arch/x86/entry/entry_64.S | 26 ++++----------------------
2 files changed, 15 insertions(+), 26 deletions(-)
diff --git a/arch/x86/entry/calling.h b/arch/x86/entry/calling.h
index 8907a65..3bda317 100644
--- a/arch/x86/entry/calling.h
+++ b/arch/x86/entry/calling.h
@@ -139,25 +139,32 @@ For 32-bit we have the following conventions - kernel is built with
xorq %r15, %r15
.endm
- .macro POP_EXTRA_REGS
+ .macro POP_REGS pop_rdi=1 skip_r11rcx=0
popq %r15
popq %r14
popq %r13
popq %r12
popq %rbp
popq %rbx
- .endm
-
- .macro POP_C_REGS
+ .if \skip_r11rcx
+ popq %rsi
+ .else
popq %r11
+ .endif
popq %r10
popq %r9
popq %r8
popq %rax
+ .if \skip_r11rcx
+ popq %rsi
+ .else
popq %rcx
+ .endif
popq %rdx
popq %rsi
+ .if \pop_rdi
popq %rdi
+ .endif
.endm
.macro icebp
diff --git a/arch/x86/entry/entry_64.S b/arch/x86/entry/entry_64.S
index 1a6fc01..7351c91 100644
--- a/arch/x86/entry/entry_64.S
+++ b/arch/x86/entry/entry_64.S
@@ -334,15 +334,7 @@ GLOBAL(entry_SYSCALL_64_after_hwframe)
syscall_return_via_sysret:
/* rcx and r11 are already restored (see code above) */
UNWIND_HINT_EMPTY
- POP_EXTRA_REGS
- popq %rsi /* skip r11 */
- popq %r10
- popq %r9
- popq %r8
- popq %rax
- popq %rsi /* skip rcx */
- popq %rdx
- popq %rsi
+ POP_REGS pop_rdi=0 skip_r11rcx=1
/*
* Now all regs are restored except RSP and RDI.
@@ -635,15 +627,7 @@ GLOBAL(swapgs_restore_regs_and_return_to_usermode)
ud2
1:
#endif
- POP_EXTRA_REGS
- popq %r11
- popq %r10
- popq %r9
- popq %r8
- popq %rax
- popq %rcx
- popq %rdx
- popq %rsi
+ POP_REGS pop_rdi=0
/*
* The stack is now user RDI, orig_ax, RIP, CS, EFLAGS, RSP, SS.
@@ -701,8 +685,7 @@ GLOBAL(restore_regs_and_return_to_kernel)
ud2
1:
#endif
- POP_EXTRA_REGS
- POP_C_REGS
+ POP_REGS
addq $8, %rsp /* skip regs->orig_ax */
INTERRUPT_RETURN
@@ -1661,8 +1644,7 @@ end_repeat_nmi:
nmi_swapgs:
SWAPGS_UNSAFE_STACK
nmi_restore:
- POP_EXTRA_REGS
- POP_C_REGS
+ POP_REGS
/*
* Skip orig_ax and the "outermost" frame to point RSP at the "iret"
^ permalink raw reply related [flat|nested] 33+ messages in thread
* [tip:x86/pti] x86/entry/64: Interleave XOR register clearing with PUSH instructions
2018-02-11 10:49 ` [PATCH v3 3/7] x86/entry: interleave XOR register clearing with PUSH instructions Dominik Brodowski
@ 2018-02-12 10:16 ` tip-bot for Dominik Brodowski
2018-02-13 9:01 ` tip-bot for Dominik Brodowski
1 sibling, 0 replies; 33+ messages in thread
From: tip-bot for Dominik Brodowski @ 2018-02-12 10:16 UTC (permalink / raw)
To: linux-tip-commits
Cc: bp, tglx, hpa, torvalds, jpoimboe, luto, dvlasenk, brgerst,
linux, mingo, linux-kernel, peterz
Commit-ID: e922c091d34db9dec1eeb6e7b923043ae4d366e3
Gitweb: https://git.kernel.org/tip/e922c091d34db9dec1eeb6e7b923043ae4d366e3
Author: Dominik Brodowski <linux@dominikbrodowski.net>
AuthorDate: Sun, 11 Feb 2018 11:49:44 +0100
Committer: Ingo Molnar <mingo@kernel.org>
CommitDate: Mon, 12 Feb 2018 08:06:35 +0100
x86/entry/64: Interleave XOR register clearing with PUSH instructions
Same as is done for syscalls, interleave XOR with PUSH instructions
for exceptions/interrupts, in order to minimize the cost of the
additional instructions required for register clearing.
Signed-off-by: Dominik Brodowski <linux@dominikbrodowski.net>
Cc: Andy Lutomirski <luto@kernel.org>
Cc: Borislav Petkov <bp@alien8.de>
Cc: Brian Gerst <brgerst@gmail.com>
Cc: Denys Vlasenko <dvlasenk@redhat.com>
Cc: H. Peter Anvin <hpa@zytor.com>
Cc: Josh Poimboeuf <jpoimboe@redhat.com>
Cc: Linus Torvalds <torvalds@linux-foundation.org>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Thomas Gleixner <tglx@linutronix.de>
Cc: dan.j.williams@intel.com
Link: http://lkml.kernel.org/r/20180211104949.12992-4-linux@dominikbrodowski.net
Signed-off-by: Ingo Molnar <mingo@kernel.org>
---
arch/x86/entry/calling.h | 40 +++++++++++++++++++---------------------
arch/x86/entry/entry_64.S | 30 +++++++++++++++++++++---------
2 files changed, 40 insertions(+), 30 deletions(-)
diff --git a/arch/x86/entry/calling.h b/arch/x86/entry/calling.h
index 3bda317..a05cbb8 100644
--- a/arch/x86/entry/calling.h
+++ b/arch/x86/entry/calling.h
@@ -101,44 +101,42 @@ For 32-bit we have the following conventions - kernel is built with
addq $-(15*8), %rsp
.endm
- .macro SAVE_REGS offset=0
+ .macro SAVE_AND_CLEAR_REGS offset=0
+ /*
+ * Save registers and sanitize registers of values that a
+ * speculation attack might otherwise want to exploit. The
+ * lower registers are likely clobbered well before they
+ * could be put to use in a speculative execution gadget.
+ * Interleave XOR with PUSH for better uop scheduling:
+ */
movq %rdi, 14*8+\offset(%rsp)
movq %rsi, 13*8+\offset(%rsp)
movq %rdx, 12*8+\offset(%rsp)
movq %rcx, 11*8+\offset(%rsp)
movq %rax, 10*8+\offset(%rsp)
movq %r8, 9*8+\offset(%rsp)
+ xorq %r8, %r8 /* nospec r8 */
movq %r9, 8*8+\offset(%rsp)
+ xorq %r9, %r9 /* nospec r9 */
movq %r10, 7*8+\offset(%rsp)
+ xorq %r10, %r10 /* nospec r10 */
movq %r11, 6*8+\offset(%rsp)
+ xorq %r11, %r11 /* nospec r11 */
movq %rbx, 5*8+\offset(%rsp)
+ xorl %ebx, %ebx /* nospec rbx */
movq %rbp, 4*8+\offset(%rsp)
+ xorl %ebp, %ebp /* nospec rbp */
movq %r12, 3*8+\offset(%rsp)
+ xorq %r12, %r12 /* nospec r12 */
movq %r13, 2*8+\offset(%rsp)
+ xorq %r13, %r13 /* nospec r13 */
movq %r14, 1*8+\offset(%rsp)
+ xorq %r14, %r14 /* nospec r14 */
movq %r15, 0*8+\offset(%rsp)
+ xorq %r15, %r15 /* nospec r15 */
UNWIND_HINT_REGS offset=\offset
.endm
- /*
- * Sanitize registers of values that a speculation attack
- * might otherwise want to exploit. The lower registers are
- * likely clobbered well before they could be put to use in
- * a speculative execution gadget:
- */
- .macro CLEAR_REGS_NOSPEC
- xorl %ebp, %ebp
- xorl %ebx, %ebx
- xorq %r8, %r8
- xorq %r9, %r9
- xorq %r10, %r10
- xorq %r11, %r11
- xorq %r12, %r12
- xorq %r13, %r13
- xorq %r14, %r14
- xorq %r15, %r15
- .endm
-
.macro POP_REGS pop_rdi=1 skip_r11rcx=0
popq %r15
popq %r14
@@ -177,7 +175,7 @@ For 32-bit we have the following conventions - kernel is built with
* is just setting the LSB, which makes it an invalid stack address and is also
* a signal to the unwinder that it's a pt_regs pointer in disguise.
*
- * NOTE: This macro must be used *after* SAVE_REGS because it corrupts
+ * NOTE: This macro must be used *after* SAVE_AND_CLEAR_REGS because it corrupts
* the original rbp.
*/
.macro ENCODE_FRAME_POINTER ptregs_offset=0
diff --git a/arch/x86/entry/entry_64.S b/arch/x86/entry/entry_64.S
index 7351c91..07692b4 100644
--- a/arch/x86/entry/entry_64.S
+++ b/arch/x86/entry/entry_64.S
@@ -565,8 +565,7 @@ END(irq_entries_start)
1:
ALLOC_PT_GPREGS_ON_STACK
- SAVE_REGS
- CLEAR_REGS_NOSPEC
+ SAVE_AND_CLEAR_REGS
ENCODE_FRAME_POINTER
testb $3, CS(%rsp)
@@ -1114,8 +1113,7 @@ ENTRY(xen_failsafe_callback)
UNWIND_HINT_IRET_REGS
pushq $-1 /* orig_ax = -1 => not a system call */
ALLOC_PT_GPREGS_ON_STACK
- SAVE_REGS
- CLEAR_REGS_NOSPEC
+ SAVE_AND_CLEAR_REGS
ENCODE_FRAME_POINTER
jmp error_exit
END(xen_failsafe_callback)
@@ -1159,8 +1157,7 @@ idtentry machine_check do_mce has_error_code=0 paranoid=1
ENTRY(paranoid_entry)
UNWIND_HINT_FUNC
cld
- SAVE_REGS 8
- CLEAR_REGS_NOSPEC
+ SAVE_AND_CLEAR_REGS 8
ENCODE_FRAME_POINTER 8
movl $1, %ebx
movl $MSR_GS_BASE, %ecx
@@ -1211,8 +1208,7 @@ END(paranoid_exit)
ENTRY(error_entry)
UNWIND_HINT_FUNC
cld
- SAVE_REGS 8
- CLEAR_REGS_NOSPEC
+ SAVE_AND_CLEAR_REGS 8
ENCODE_FRAME_POINTER 8
testb $3, CS+8(%rsp)
jz .Lerror_kernelspace
@@ -1399,18 +1395,34 @@ ENTRY(nmi)
pushq (%rdx) /* pt_regs->dx */
pushq %rcx /* pt_regs->cx */
pushq %rax /* pt_regs->ax */
+ /*
+ * Sanitize registers of values that a speculation attack
+ * might otherwise want to exploit. The lower registers are
+ * likely clobbered well before they could be put to use in
+ * a speculative execution gadget. Interleave XOR with PUSH
+ * for better uop scheduling:
+ */
pushq %r8 /* pt_regs->r8 */
+ xorq %r8, %r8 /* nospec r8 */
pushq %r9 /* pt_regs->r9 */
+ xorq %r9, %r9 /* nospec r9 */
pushq %r10 /* pt_regs->r10 */
+ xorq %r10, %r10 /* nospec r10 */
pushq %r11 /* pt_regs->r11 */
+ xorq %r11, %r11 /* nospec r11*/
pushq %rbx /* pt_regs->rbx */
+ xorl %ebx, %ebx /* nospec rbx*/
pushq %rbp /* pt_regs->rbp */
+ xorl %ebp, %ebp /* nospec rbp*/
pushq %r12 /* pt_regs->r12 */
+ xorq %r12, %r12 /* nospec r12*/
pushq %r13 /* pt_regs->r13 */
+ xorq %r13, %r13 /* nospec r13*/
pushq %r14 /* pt_regs->r14 */
+ xorq %r14, %r14 /* nospec r14*/
pushq %r15 /* pt_regs->r15 */
+ xorq %r15, %r15 /* nospec r15*/
UNWIND_HINT_REGS
- CLEAR_REGS_NOSPEC
ENCODE_FRAME_POINTER
/*
^ permalink raw reply related [flat|nested] 33+ messages in thread
* [tip:x86/pti] x86/entry/64: Introduce the PUSH_AND_CLEAN_REGS macro
2018-02-11 10:49 ` [PATCH v3 4/7] x86/entry: introduce PUSH_AND_CLEAN_REGS Dominik Brodowski
@ 2018-02-12 10:17 ` tip-bot for Dominik Brodowski
2018-02-12 13:29 ` Denys Vlasenko
2018-02-13 9:01 ` tip-bot for Dominik Brodowski
1 sibling, 1 reply; 33+ messages in thread
From: tip-bot for Dominik Brodowski @ 2018-02-12 10:17 UTC (permalink / raw)
To: linux-tip-commits
Cc: hpa, torvalds, mingo, luto, linux, bp, linux-kernel, peterz,
brgerst, jpoimboe, dvlasenk, tglx
Commit-ID: 7b7b09f110f06f3c006e9b3f4590f7d9ba91345b
Gitweb: https://git.kernel.org/tip/7b7b09f110f06f3c006e9b3f4590f7d9ba91345b
Author: Dominik Brodowski <linux@dominikbrodowski.net>
AuthorDate: Sun, 11 Feb 2018 11:49:45 +0100
Committer: Ingo Molnar <mingo@kernel.org>
CommitDate: Mon, 12 Feb 2018 08:06:36 +0100
x86/entry/64: Introduce the PUSH_AND_CLEAN_REGS macro
Those instances where ALLOC_PT_GPREGS_ON_STACK is called just before
SAVE_AND_CLEAR_REGS can trivially be replaced by PUSH_AND_CLEAN_REGS.
This macro uses PUSH instead of MOV and should therefore be faster, at
least on newer CPUs.
Suggested-by: Linus Torvalds <torvalds@linux-foundation.org>
Signed-off-by: Dominik Brodowski <linux@dominikbrodowski.net>
Cc: Andy Lutomirski <luto@kernel.org>
Cc: Borislav Petkov <bp@alien8.de>
Cc: Brian Gerst <brgerst@gmail.com>
Cc: Denys Vlasenko <dvlasenk@redhat.com>
Cc: H. Peter Anvin <hpa@zytor.com>
Cc: Josh Poimboeuf <jpoimboe@redhat.com>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Thomas Gleixner <tglx@linutronix.de>
Cc: dan.j.williams@intel.com
Link: http://lkml.kernel.org/r/20180211104949.12992-5-linux@dominikbrodowski.net
Signed-off-by: Ingo Molnar <mingo@kernel.org>
---
arch/x86/entry/calling.h | 36 ++++++++++++++++++++++++++++++++++++
arch/x86/entry/entry_64.S | 6 ++----
2 files changed, 38 insertions(+), 4 deletions(-)
diff --git a/arch/x86/entry/calling.h b/arch/x86/entry/calling.h
index a05cbb8..57b1b87 100644
--- a/arch/x86/entry/calling.h
+++ b/arch/x86/entry/calling.h
@@ -137,6 +137,42 @@ For 32-bit we have the following conventions - kernel is built with
UNWIND_HINT_REGS offset=\offset
.endm
+ .macro PUSH_AND_CLEAR_REGS
+ /*
+ * Push registers and sanitize registers of values that a
+ * speculation attack might otherwise want to exploit. The
+ * lower registers are likely clobbered well before they
+ * could be put to use in a speculative execution gadget.
+ * Interleave XOR with PUSH for better uop scheduling:
+ */
+ pushq %rdi /* pt_regs->di */
+ pushq %rsi /* pt_regs->si */
+ pushq %rdx /* pt_regs->dx */
+ pushq %rcx /* pt_regs->cx */
+ pushq %rax /* pt_regs->ax */
+ pushq %r8 /* pt_regs->r8 */
+ xorq %r8, %r8 /* nospec r8 */
+ pushq %r9 /* pt_regs->r9 */
+ xorq %r9, %r9 /* nospec r9 */
+ pushq %r10 /* pt_regs->r10 */
+ xorq %r10, %r10 /* nospec r10 */
+ pushq %r11 /* pt_regs->r11 */
+ xorq %r11, %r11 /* nospec r11*/
+ pushq %rbx /* pt_regs->rbx */
+ xorl %ebx, %ebx /* nospec rbx*/
+ pushq %rbp /* pt_regs->rbp */
+ xorl %ebp, %ebp /* nospec rbp*/
+ pushq %r12 /* pt_regs->r12 */
+ xorq %r12, %r12 /* nospec r12*/
+ pushq %r13 /* pt_regs->r13 */
+ xorq %r13, %r13 /* nospec r13*/
+ pushq %r14 /* pt_regs->r14 */
+ xorq %r14, %r14 /* nospec r14*/
+ pushq %r15 /* pt_regs->r15 */
+ xorq %r15, %r15 /* nospec r15*/
+ UNWIND_HINT_REGS
+ .endm
+
.macro POP_REGS pop_rdi=1 skip_r11rcx=0
popq %r15
popq %r14
diff --git a/arch/x86/entry/entry_64.S b/arch/x86/entry/entry_64.S
index 07692b4..cf4a9ae 100644
--- a/arch/x86/entry/entry_64.S
+++ b/arch/x86/entry/entry_64.S
@@ -564,8 +564,7 @@ END(irq_entries_start)
call switch_to_thread_stack
1:
- ALLOC_PT_GPREGS_ON_STACK
- SAVE_AND_CLEAR_REGS
+ PUSH_AND_CLEAR_REGS
ENCODE_FRAME_POINTER
testb $3, CS(%rsp)
@@ -1112,8 +1111,7 @@ ENTRY(xen_failsafe_callback)
addq $0x30, %rsp
UNWIND_HINT_IRET_REGS
pushq $-1 /* orig_ax = -1 => not a system call */
- ALLOC_PT_GPREGS_ON_STACK
- SAVE_AND_CLEAR_REGS
+ PUSH_AND_CLEAR_REGS
ENCODE_FRAME_POINTER
jmp error_exit
END(xen_failsafe_callback)
^ permalink raw reply related [flat|nested] 33+ messages in thread
* [tip:x86/pti] x86/entry/64: Use PUSH_AND_CLEAN_REGS in more cases
2018-02-11 10:49 ` [PATCH v3 5/7] x86/entry: use PUSH_AND_CLEAN_REGS in more cases Dominik Brodowski
@ 2018-02-12 10:17 ` tip-bot for Dominik Brodowski
2018-02-13 9:02 ` tip-bot for Dominik Brodowski
1 sibling, 0 replies; 33+ messages in thread
From: tip-bot for Dominik Brodowski @ 2018-02-12 10:17 UTC (permalink / raw)
To: linux-tip-commits
Cc: torvalds, mingo, peterz, tglx, dvlasenk, bp, linux-kernel, hpa,
linux, brgerst, jpoimboe, luto
Commit-ID: 0e61b0142a4bdbe855c1d80342d2ca00241488b6
Gitweb: https://git.kernel.org/tip/0e61b0142a4bdbe855c1d80342d2ca00241488b6
Author: Dominik Brodowski <linux@dominikbrodowski.net>
AuthorDate: Sun, 11 Feb 2018 11:49:46 +0100
Committer: Ingo Molnar <mingo@kernel.org>
CommitDate: Mon, 12 Feb 2018 08:06:36 +0100
x86/entry/64: Use PUSH_AND_CLEAN_REGS in more cases
entry_SYSCALL_64_after_hwframe() and nmi() can be converted to use
PUSH_AND_CLEAN_REGS instead of opencoded variants thereof. Due to
the interleaving, the additional XOR-based clearing of R8 and R9
in entry_SYSCALL_64_after_hwframe() should not have any noticeable
negative implications.
Suggested-by: Linus Torvalds <torvalds@linux-foundation.org>
Signed-off-by: Dominik Brodowski <linux@dominikbrodowski.net>
Cc: Andy Lutomirski <luto@kernel.org>
Cc: Borislav Petkov <bp@alien8.de>
Cc: Brian Gerst <brgerst@gmail.com>
Cc: Denys Vlasenko <dvlasenk@redhat.com>
Cc: H. Peter Anvin <hpa@zytor.com>
Cc: Josh Poimboeuf <jpoimboe@redhat.com>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Thomas Gleixner <tglx@linutronix.de>
Cc: dan.j.williams@intel.com
Link: http://lkml.kernel.org/r/20180211104949.12992-6-linux@dominikbrodowski.net
Signed-off-by: Ingo Molnar <mingo@kernel.org>
---
arch/x86/entry/calling.h | 6 ++---
arch/x86/entry/entry_64.S | 65 +++--------------------------------------------
2 files changed, 6 insertions(+), 65 deletions(-)
diff --git a/arch/x86/entry/calling.h b/arch/x86/entry/calling.h
index 57b1b87..d6a97e2 100644
--- a/arch/x86/entry/calling.h
+++ b/arch/x86/entry/calling.h
@@ -137,7 +137,7 @@ For 32-bit we have the following conventions - kernel is built with
UNWIND_HINT_REGS offset=\offset
.endm
- .macro PUSH_AND_CLEAR_REGS
+ .macro PUSH_AND_CLEAR_REGS rdx=%rdx rax=%rax
/*
* Push registers and sanitize registers of values that a
* speculation attack might otherwise want to exploit. The
@@ -147,9 +147,9 @@ For 32-bit we have the following conventions - kernel is built with
*/
pushq %rdi /* pt_regs->di */
pushq %rsi /* pt_regs->si */
- pushq %rdx /* pt_regs->dx */
+ pushq \rdx /* pt_regs->dx */
pushq %rcx /* pt_regs->cx */
- pushq %rax /* pt_regs->ax */
+ pushq \rax /* pt_regs->ax */
pushq %r8 /* pt_regs->r8 */
xorq %r8, %r8 /* nospec r8 */
pushq %r9 /* pt_regs->r9 */
diff --git a/arch/x86/entry/entry_64.S b/arch/x86/entry/entry_64.S
index cf4a9ae..b06a4b5 100644
--- a/arch/x86/entry/entry_64.S
+++ b/arch/x86/entry/entry_64.S
@@ -227,35 +227,8 @@ ENTRY(entry_SYSCALL_64)
pushq %rcx /* pt_regs->ip */
GLOBAL(entry_SYSCALL_64_after_hwframe)
pushq %rax /* pt_regs->orig_ax */
- pushq %rdi /* pt_regs->di */
- pushq %rsi /* pt_regs->si */
- pushq %rdx /* pt_regs->dx */
- pushq %rcx /* pt_regs->cx */
- pushq $-ENOSYS /* pt_regs->ax */
- pushq %r8 /* pt_regs->r8 */
- pushq %r9 /* pt_regs->r9 */
- pushq %r10 /* pt_regs->r10 */
- /*
- * Clear extra registers that a speculation attack might
- * otherwise want to exploit. Interleave XOR with PUSH
- * for better uop scheduling:
- */
- xorq %r10, %r10 /* nospec r10 */
- pushq %r11 /* pt_regs->r11 */
- xorq %r11, %r11 /* nospec r11 */
- pushq %rbx /* pt_regs->rbx */
- xorl %ebx, %ebx /* nospec rbx */
- pushq %rbp /* pt_regs->rbp */
- xorl %ebp, %ebp /* nospec rbp */
- pushq %r12 /* pt_regs->r12 */
- xorq %r12, %r12 /* nospec r12 */
- pushq %r13 /* pt_regs->r13 */
- xorq %r13, %r13 /* nospec r13 */
- pushq %r14 /* pt_regs->r14 */
- xorq %r14, %r14 /* nospec r14 */
- pushq %r15 /* pt_regs->r15 */
- xorq %r15, %r15 /* nospec r15 */
- UNWIND_HINT_REGS
+
+ PUSH_AND_CLEAR_REGS rax=$-ENOSYS
TRACE_IRQS_OFF
@@ -1388,39 +1361,7 @@ ENTRY(nmi)
pushq 1*8(%rdx) /* pt_regs->rip */
UNWIND_HINT_IRET_REGS
pushq $-1 /* pt_regs->orig_ax */
- pushq %rdi /* pt_regs->di */
- pushq %rsi /* pt_regs->si */
- pushq (%rdx) /* pt_regs->dx */
- pushq %rcx /* pt_regs->cx */
- pushq %rax /* pt_regs->ax */
- /*
- * Sanitize registers of values that a speculation attack
- * might otherwise want to exploit. The lower registers are
- * likely clobbered well before they could be put to use in
- * a speculative execution gadget. Interleave XOR with PUSH
- * for better uop scheduling:
- */
- pushq %r8 /* pt_regs->r8 */
- xorq %r8, %r8 /* nospec r8 */
- pushq %r9 /* pt_regs->r9 */
- xorq %r9, %r9 /* nospec r9 */
- pushq %r10 /* pt_regs->r10 */
- xorq %r10, %r10 /* nospec r10 */
- pushq %r11 /* pt_regs->r11 */
- xorq %r11, %r11 /* nospec r11*/
- pushq %rbx /* pt_regs->rbx */
- xorl %ebx, %ebx /* nospec rbx*/
- pushq %rbp /* pt_regs->rbp */
- xorl %ebp, %ebp /* nospec rbp*/
- pushq %r12 /* pt_regs->r12 */
- xorq %r12, %r12 /* nospec r12*/
- pushq %r13 /* pt_regs->r13 */
- xorq %r13, %r13 /* nospec r13*/
- pushq %r14 /* pt_regs->r14 */
- xorq %r14, %r14 /* nospec r14*/
- pushq %r15 /* pt_regs->r15 */
- xorq %r15, %r15 /* nospec r15*/
- UNWIND_HINT_REGS
+ PUSH_AND_CLEAR_REGS rdx=(%rdx)
ENCODE_FRAME_POINTER
/*
^ permalink raw reply related [flat|nested] 33+ messages in thread
* [tip:x86/pti] x86/entry/64: Get rid of the ALLOC_PT_GPREGS_ON_STACK and SAVE_AND_CLEAR_REGS macros
2018-02-11 10:49 ` [PATCH v3 6/7] x86/entry: get rid of ALLOC_PT_GPREGS_ON_STACK and SAVE_AND_CLEAR_REGS Dominik Brodowski
@ 2018-02-12 10:18 ` tip-bot for Dominik Brodowski
2018-02-13 9:02 ` tip-bot for Dominik Brodowski
1 sibling, 0 replies; 33+ messages in thread
From: tip-bot for Dominik Brodowski @ 2018-02-12 10:18 UTC (permalink / raw)
To: linux-tip-commits
Cc: mingo, brgerst, tglx, dvlasenk, luto, torvalds, jpoimboe, hpa,
bp, peterz, linux, linux-kernel
Commit-ID: f09d160992d129b6d8fada7543761d40f5b33596
Gitweb: https://git.kernel.org/tip/f09d160992d129b6d8fada7543761d40f5b33596
Author: Dominik Brodowski <linux@dominikbrodowski.net>
AuthorDate: Sun, 11 Feb 2018 11:49:47 +0100
Committer: Ingo Molnar <mingo@kernel.org>
CommitDate: Mon, 12 Feb 2018 08:06:36 +0100
x86/entry/64: Get rid of the ALLOC_PT_GPREGS_ON_STACK and SAVE_AND_CLEAR_REGS macros
Previously, error_entry() and paranoid_entry() saved the GP registers
onto stack space previously allocated by its callers. Combine these two
steps in the callers, and use the generic PUSH_AND_CLEAR_REGS macro
for that.
This adds a significant amount ot text size. However, Ingo Molnar points
out that:
"these numbers also _very_ significantly over-represent the
extra footprint. The assumptions that resulted in
us compressing the IRQ entry code have changed very
significantly with the new x86 IRQ allocation code we
introduced in the last year:
- IRQ vectors are usually populated in tightly clustered
groups.
With our new vector allocator code the typical per CPU
allocation percentage on x86 systems is ~3 device vectors
and ~10 fixed vectors out of ~220 vectors - i.e. a very
low ~6% utilization (!). [...]
The days where we allocated a lot of vectors on every
CPU and the compression of the IRQ entry code text
mattered are over.
- Another issue is that only a small minority of vectors
is frequent enough to actually matter to cache utilization
in practice: 3-4 key IPIs and 1-2 device IRQs at most - and
those vectors tend to be tightly clustered as well into about
two groups, and are probably already on 2-3 cache lines in
practice.
For the common case of 'cache cold' IRQs it's the depth of
the call chain and the fragmentation of the resulting I$
that should be the main performance limit - not the overall
size of it.
- The CPU side cost of IRQ delivery is still very expensive
even in the best, most cached case, as in 'over a thousand
cycles'. So much stuff is done that maybe contemporary x86
IRQ entry microcode already prefetches the IDT entry and its
expected call target address."[*]
[*] http://lkml.kernel.org/r/20180208094710.qnjixhm6hybebdv7@gmail.com
The "testb $3, CS(%rsp)" instruction in the idtentry macro does not need
modification. Previously, %rsp was manually decreased by 15*8; with
this patch, %rsp is decreased by 15 pushq instructions.
[jpoimboe@redhat.com: unwind hint improvements]
Suggested-by: Linus Torvalds <torvalds@linux-foundation.org>
Signed-off-by: Dominik Brodowski <linux@dominikbrodowski.net>
Cc: Andy Lutomirski <luto@kernel.org>
Cc: Borislav Petkov <bp@alien8.de>
Cc: Brian Gerst <brgerst@gmail.com>
Cc: Denys Vlasenko <dvlasenk@redhat.com>
Cc: H. Peter Anvin <hpa@zytor.com>
Cc: Josh Poimboeuf <jpoimboe@redhat.com>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Thomas Gleixner <tglx@linutronix.de>
Cc: dan.j.williams@intel.com
Link: http://lkml.kernel.org/r/20180211104949.12992-7-linux@dominikbrodowski.net
Signed-off-by: Ingo Molnar <mingo@kernel.org>
---
arch/x86/entry/calling.h | 42 +-----------------------------------------
arch/x86/entry/entry_64.S | 20 +++++++++-----------
2 files changed, 10 insertions(+), 52 deletions(-)
diff --git a/arch/x86/entry/calling.h b/arch/x86/entry/calling.h
index d6a97e2..5967501 100644
--- a/arch/x86/entry/calling.h
+++ b/arch/x86/entry/calling.h
@@ -97,46 +97,6 @@ For 32-bit we have the following conventions - kernel is built with
#define SIZEOF_PTREGS 21*8
- .macro ALLOC_PT_GPREGS_ON_STACK
- addq $-(15*8), %rsp
- .endm
-
- .macro SAVE_AND_CLEAR_REGS offset=0
- /*
- * Save registers and sanitize registers of values that a
- * speculation attack might otherwise want to exploit. The
- * lower registers are likely clobbered well before they
- * could be put to use in a speculative execution gadget.
- * Interleave XOR with PUSH for better uop scheduling:
- */
- movq %rdi, 14*8+\offset(%rsp)
- movq %rsi, 13*8+\offset(%rsp)
- movq %rdx, 12*8+\offset(%rsp)
- movq %rcx, 11*8+\offset(%rsp)
- movq %rax, 10*8+\offset(%rsp)
- movq %r8, 9*8+\offset(%rsp)
- xorq %r8, %r8 /* nospec r8 */
- movq %r9, 8*8+\offset(%rsp)
- xorq %r9, %r9 /* nospec r9 */
- movq %r10, 7*8+\offset(%rsp)
- xorq %r10, %r10 /* nospec r10 */
- movq %r11, 6*8+\offset(%rsp)
- xorq %r11, %r11 /* nospec r11 */
- movq %rbx, 5*8+\offset(%rsp)
- xorl %ebx, %ebx /* nospec rbx */
- movq %rbp, 4*8+\offset(%rsp)
- xorl %ebp, %ebp /* nospec rbp */
- movq %r12, 3*8+\offset(%rsp)
- xorq %r12, %r12 /* nospec r12 */
- movq %r13, 2*8+\offset(%rsp)
- xorq %r13, %r13 /* nospec r13 */
- movq %r14, 1*8+\offset(%rsp)
- xorq %r14, %r14 /* nospec r14 */
- movq %r15, 0*8+\offset(%rsp)
- xorq %r15, %r15 /* nospec r15 */
- UNWIND_HINT_REGS offset=\offset
- .endm
-
.macro PUSH_AND_CLEAR_REGS rdx=%rdx rax=%rax
/*
* Push registers and sanitize registers of values that a
@@ -211,7 +171,7 @@ For 32-bit we have the following conventions - kernel is built with
* is just setting the LSB, which makes it an invalid stack address and is also
* a signal to the unwinder that it's a pt_regs pointer in disguise.
*
- * NOTE: This macro must be used *after* SAVE_AND_CLEAR_REGS because it corrupts
+ * NOTE: This macro must be used *after* PUSH_AND_CLEAR_REGS because it corrupts
* the original rbp.
*/
.macro ENCODE_FRAME_POINTER ptregs_offset=0
diff --git a/arch/x86/entry/entry_64.S b/arch/x86/entry/entry_64.S
index b06a4b5..cfbf433 100644
--- a/arch/x86/entry/entry_64.S
+++ b/arch/x86/entry/entry_64.S
@@ -871,7 +871,9 @@ ENTRY(\sym)
pushq $-1 /* ORIG_RAX: no syscall to restart */
.endif
- ALLOC_PT_GPREGS_ON_STACK
+ /* Save all registers in pt_regs */
+ PUSH_AND_CLEAR_REGS
+ ENCODE_FRAME_POINTER
.if \paranoid < 2
testb $3, CS(%rsp) /* If coming from userspace, switch stacks */
@@ -1121,15 +1123,12 @@ idtentry machine_check do_mce has_error_code=0 paranoid=1
#endif
/*
- * Save all registers in pt_regs, and switch gs if needed.
+ * Switch gs if needed.
* Use slow, but surefire "are we in kernel?" check.
* Return: ebx=0: need swapgs on exit, ebx=1: otherwise
*/
ENTRY(paranoid_entry)
- UNWIND_HINT_FUNC
cld
- SAVE_AND_CLEAR_REGS 8
- ENCODE_FRAME_POINTER 8
movl $1, %ebx
movl $MSR_GS_BASE, %ecx
rdmsr
@@ -1142,7 +1141,7 @@ ENTRY(paranoid_entry)
SAVE_AND_SWITCH_TO_KERNEL_CR3 scratch_reg=%rax save_reg=%r14
ret
-END(paranoid_entry)
+ENDPROC(paranoid_entry)
/*
* "Paranoid" exit path from exception stack. This is invoked
@@ -1173,14 +1172,12 @@ ENTRY(paranoid_exit)
END(paranoid_exit)
/*
- * Save all registers in pt_regs, and switch gs if needed.
+ * Switch gs if needed.
* Return: EBX=0: came from user mode; EBX=1: otherwise
*/
ENTRY(error_entry)
- UNWIND_HINT_FUNC
+ UNWIND_HINT_REGS offset=8
cld
- SAVE_AND_CLEAR_REGS 8
- ENCODE_FRAME_POINTER 8
testb $3, CS+8(%rsp)
jz .Lerror_kernelspace
@@ -1571,7 +1568,8 @@ end_repeat_nmi:
* frame to point back to repeat_nmi.
*/
pushq $-1 /* ORIG_RAX: no syscall to restart */
- ALLOC_PT_GPREGS_ON_STACK
+ PUSH_AND_CLEAR_REGS
+ ENCODE_FRAME_POINTER
/*
* Use paranoid_entry to handle SWAPGS, but no need to use paranoid_exit
^ permalink raw reply related [flat|nested] 33+ messages in thread
* [tip:x86/pti] x86/entry/64: Indent PUSH_AND_CLEAR_REGS and POP_REGS properly
2018-02-11 10:49 ` [PATCH v3 7/7] x86/entry: indent PUSH_AND_CLEAR_REGS and POP_REGS properly Dominik Brodowski
@ 2018-02-12 10:18 ` tip-bot for Dominik Brodowski
2018-02-12 20:13 ` [PATCH] x86/entry/64: Remove unused icebp macro Borislav Petkov
2018-02-13 9:03 ` [tip:x86/pti] x86/entry/64: Indent PUSH_AND_CLEAR_REGS and POP_REGS properly tip-bot for Dominik Brodowski
1 sibling, 1 reply; 33+ messages in thread
From: tip-bot for Dominik Brodowski @ 2018-02-12 10:18 UTC (permalink / raw)
To: linux-tip-commits
Cc: linux-kernel, dvlasenk, luto, torvalds, jpoimboe, hpa, brgerst,
linux, mingo, peterz, tglx, bp
Commit-ID: 5a10e729bc0aa9420a23a70b169eadd229c27bc1
Gitweb: https://git.kernel.org/tip/5a10e729bc0aa9420a23a70b169eadd229c27bc1
Author: Dominik Brodowski <linux@dominikbrodowski.net>
AuthorDate: Sun, 11 Feb 2018 11:49:48 +0100
Committer: Ingo Molnar <mingo@kernel.org>
CommitDate: Mon, 12 Feb 2018 08:06:37 +0100
x86/entry/64: Indent PUSH_AND_CLEAR_REGS and POP_REGS properly
... same as the other macros in arch/x86/entry/calling.h
Signed-off-by: Dominik Brodowski <linux@dominikbrodowski.net>
Cc: Andy Lutomirski <luto@kernel.org>
Cc: Borislav Petkov <bp@alien8.de>
Cc: Brian Gerst <brgerst@gmail.com>
Cc: Denys Vlasenko <dvlasenk@redhat.com>
Cc: H. Peter Anvin <hpa@zytor.com>
Cc: Josh Poimboeuf <jpoimboe@redhat.com>
Cc: Linus Torvalds <torvalds@linux-foundation.org>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Thomas Gleixner <tglx@linutronix.de>
Cc: dan.j.williams@intel.com
Link: http://lkml.kernel.org/r/20180211104949.12992-8-linux@dominikbrodowski.net
Signed-off-by: Ingo Molnar <mingo@kernel.org>
---
arch/x86/entry/calling.h | 8 ++++----
1 file changed, 4 insertions(+), 4 deletions(-)
diff --git a/arch/x86/entry/calling.h b/arch/x86/entry/calling.h
index 5967501..6985440 100644
--- a/arch/x86/entry/calling.h
+++ b/arch/x86/entry/calling.h
@@ -97,7 +97,7 @@ For 32-bit we have the following conventions - kernel is built with
#define SIZEOF_PTREGS 21*8
- .macro PUSH_AND_CLEAR_REGS rdx=%rdx rax=%rax
+.macro PUSH_AND_CLEAR_REGS rdx=%rdx rax=%rax
/*
* Push registers and sanitize registers of values that a
* speculation attack might otherwise want to exploit. The
@@ -131,9 +131,9 @@ For 32-bit we have the following conventions - kernel is built with
pushq %r15 /* pt_regs->r15 */
xorq %r15, %r15 /* nospec r15*/
UNWIND_HINT_REGS
- .endm
+.endm
- .macro POP_REGS pop_rdi=1 skip_r11rcx=0
+.macro POP_REGS pop_rdi=1 skip_r11rcx=0
popq %r15
popq %r14
popq %r13
@@ -163,7 +163,7 @@ For 32-bit we have the following conventions - kernel is built with
.macro icebp
.byte 0xf1
- .endm
+.endm
/*
* This is a sneaky trick to help the unwinder find pt_regs on the stack. The
^ permalink raw reply related [flat|nested] 33+ messages in thread
* Re: [tip:x86/pti] x86/entry/64: Introduce the PUSH_AND_CLEAN_REGS macro
2018-02-12 10:17 ` [tip:x86/pti] x86/entry/64: Introduce the PUSH_AND_CLEAN_REGS macro tip-bot for Dominik Brodowski
@ 2018-02-12 13:29 ` Denys Vlasenko
2018-02-12 13:36 ` David Laight
2018-02-12 16:51 ` Linus Torvalds
0 siblings, 2 replies; 33+ messages in thread
From: Denys Vlasenko @ 2018-02-12 13:29 UTC (permalink / raw)
To: hpa, torvalds, luto, mingo, bp, linux-kernel, linux, brgerst,
peterz, tglx, jpoimboe, linux-tip-commits
On 02/12/2018 11:17 AM, tip-bot for Dominik Brodowski wrote:
> Commit-ID: 7b7b09f110f06f3c006e9b3f4590f7d9ba91345b
> Gitweb: https://git.kernel.org/tip/7b7b09f110f06f3c006e9b3f4590f7d9ba91345b
> Author: Dominik Brodowski <linux@dominikbrodowski.net>
> AuthorDate: Sun, 11 Feb 2018 11:49:45 +0100
> Committer: Ingo Molnar <mingo@kernel.org>
> CommitDate: Mon, 12 Feb 2018 08:06:36 +0100
>
> x86/entry/64: Introduce the PUSH_AND_CLEAN_REGS macro
>
> Those instances where ALLOC_PT_GPREGS_ON_STACK is called just before
> SAVE_AND_CLEAR_REGS can trivially be replaced by PUSH_AND_CLEAN_REGS.
> This macro uses PUSH instead of MOV and should therefore be faster, at
> least on newer CPUs.
>
> Suggested-by: Linus Torvalds <torvalds@linux-foundation.org>
> Signed-off-by: Dominik Brodowski <linux@dominikbrodowski.net>
> Cc: Andy Lutomirski <luto@kernel.org>
> Cc: Borislav Petkov <bp@alien8.de>
> Cc: Brian Gerst <brgerst@gmail.com>
> Cc: Denys Vlasenko <dvlasenk@redhat.com>
> Cc: H. Peter Anvin <hpa@zytor.com>
> Cc: Josh Poimboeuf <jpoimboe@redhat.com>
> Cc: Peter Zijlstra <peterz@infradead.org>
> Cc: Thomas Gleixner <tglx@linutronix.de>
> Cc: dan.j.williams@intel.com
> Link: http://lkml.kernel.org/r/20180211104949.12992-5-linux@dominikbrodowski.net
> Signed-off-by: Ingo Molnar <mingo@kernel.org>
> ---
> arch/x86/entry/calling.h | 36 ++++++++++++++++++++++++++++++++++++
> arch/x86/entry/entry_64.S | 6 ++----
> 2 files changed, 38 insertions(+), 4 deletions(-)
>
> diff --git a/arch/x86/entry/calling.h b/arch/x86/entry/calling.h
> index a05cbb8..57b1b87 100644
> --- a/arch/x86/entry/calling.h
> +++ b/arch/x86/entry/calling.h
> @@ -137,6 +137,42 @@ For 32-bit we have the following conventions - kernel is built with
> UNWIND_HINT_REGS offset=\offset
> .endm
>
> + .macro PUSH_AND_CLEAR_REGS
> + /*
> + * Push registers and sanitize registers of values that a
> + * speculation attack might otherwise want to exploit. The
> + * lower registers are likely clobbered well before they
> + * could be put to use in a speculative execution gadget.
> + * Interleave XOR with PUSH for better uop scheduling:
> + */
> + pushq %rdi /* pt_regs->di */
> + pushq %rsi /* pt_regs->si */
> + pushq %rdx /* pt_regs->dx */
> + pushq %rcx /* pt_regs->cx */
> + pushq %rax /* pt_regs->ax */
> + pushq %r8 /* pt_regs->r8 */
> + xorq %r8, %r8 /* nospec r8 */
xorq's are slower than xorl's on Silvermont/Knights Landing.
I propose using xorl instead.
> + pushq %r9 /* pt_regs->r9 */
> + xorq %r9, %r9 /* nospec r9 */
> + pushq %r10 /* pt_regs->r10 */
> + xorq %r10, %r10 /* nospec r10 */
> + pushq %r11 /* pt_regs->r11 */
> + xorq %r11, %r11 /* nospec r11*/
> + pushq %rbx /* pt_regs->rbx */
> + xorl %ebx, %ebx /* nospec rbx*/
> + pushq %rbp /* pt_regs->rbp */
> + xorl %ebp, %ebp /* nospec rbp*/
> + pushq %r12 /* pt_regs->r12 */
> + xorq %r12, %r12 /* nospec r12*/
> + pushq %r13 /* pt_regs->r13 */
> + xorq %r13, %r13 /* nospec r13*/
> + pushq %r14 /* pt_regs->r14 */
> + xorq %r14, %r14 /* nospec r14*/
> + pushq %r15 /* pt_regs->r15 */
> + xorq %r15, %r15 /* nospec r15*/
^ permalink raw reply [flat|nested] 33+ messages in thread
* RE: [tip:x86/pti] x86/entry/64: Introduce the PUSH_AND_CLEAN_REGS macro
2018-02-12 13:29 ` Denys Vlasenko
@ 2018-02-12 13:36 ` David Laight
2018-02-12 13:43 ` Denys Vlasenko
2018-02-12 16:51 ` Linus Torvalds
1 sibling, 1 reply; 33+ messages in thread
From: David Laight @ 2018-02-12 13:36 UTC (permalink / raw)
To: 'Denys Vlasenko',
hpa, torvalds, luto, mingo, bp, linux-kernel, linux, brgerst,
peterz, tglx, jpoimboe, linux-tip-commits
From: Denys Vlasenko
> Sent: 12 February 2018 13:29
...
> >
> > x86/entry/64: Introduce the PUSH_AND_CLEAN_REGS macro
> >
> > Those instances where ALLOC_PT_GPREGS_ON_STACK is called just before
> > SAVE_AND_CLEAR_REGS can trivially be replaced by PUSH_AND_CLEAN_REGS.
> > This macro uses PUSH instead of MOV and should therefore be faster, at
> > least on newer CPUs.
...
> > Link: http://lkml.kernel.org/r/20180211104949.12992-5-linux@dominikbrodowski.net
> > Signed-off-by: Ingo Molnar <mingo@kernel.org>
> > ---
> > arch/x86/entry/calling.h | 36 ++++++++++++++++++++++++++++++++++++
> > arch/x86/entry/entry_64.S | 6 ++----
> > 2 files changed, 38 insertions(+), 4 deletions(-)
> >
> > diff --git a/arch/x86/entry/calling.h b/arch/x86/entry/calling.h
> > index a05cbb8..57b1b87 100644
> > --- a/arch/x86/entry/calling.h
> > +++ b/arch/x86/entry/calling.h
> > @@ -137,6 +137,42 @@ For 32-bit we have the following conventions - kernel is built with
> > UNWIND_HINT_REGS offset=\offset
> > .endm
> >
> > + .macro PUSH_AND_CLEAR_REGS
> > + /*
> > + * Push registers and sanitize registers of values that a
> > + * speculation attack might otherwise want to exploit. The
> > + * lower registers are likely clobbered well before they
> > + * could be put to use in a speculative execution gadget.
> > + * Interleave XOR with PUSH for better uop scheduling:
> > + */
> > + pushq %rdi /* pt_regs->di */
> > + pushq %rsi /* pt_regs->si */
> > + pushq %rdx /* pt_regs->dx */
> > + pushq %rcx /* pt_regs->cx */
> > + pushq %rax /* pt_regs->ax */
> > + pushq %r8 /* pt_regs->r8 */
> > + xorq %r8, %r8 /* nospec r8 */
>
> xorq's are slower than xorl's on Silvermont/Knights Landing.
> I propose using xorl instead.
Does using movq to copy the first zero to the other registers make
the code any faster?
ISTR mov reg-reg is often implemented as a register rename rather than an
alu operation.
David
^ permalink raw reply [flat|nested] 33+ messages in thread
* Re: [tip:x86/pti] x86/entry/64: Introduce the PUSH_AND_CLEAN_REGS macro
2018-02-12 13:36 ` David Laight
@ 2018-02-12 13:43 ` Denys Vlasenko
0 siblings, 0 replies; 33+ messages in thread
From: Denys Vlasenko @ 2018-02-12 13:43 UTC (permalink / raw)
To: David Laight, hpa, torvalds, luto, mingo, bp, linux-kernel,
linux, brgerst, peterz, tglx, jpoimboe, linux-tip-commits
On 02/12/2018 02:36 PM, David Laight wrote:
> From: Denys Vlasenko
>> Sent: 12 February 2018 13:29
> ...
>>>
>>> x86/entry/64: Introduce the PUSH_AND_CLEAN_REGS macro
>>>
>>> Those instances where ALLOC_PT_GPREGS_ON_STACK is called just before
>>> SAVE_AND_CLEAR_REGS can trivially be replaced by PUSH_AND_CLEAN_REGS.
>>> This macro uses PUSH instead of MOV and should therefore be faster, at
>>> least on newer CPUs.
> ...
>>> Link: http://lkml.kernel.org/r/20180211104949.12992-5-linux@dominikbrodowski.net
>>> Signed-off-by: Ingo Molnar <mingo@kernel.org>
>>> ---
>>> arch/x86/entry/calling.h | 36 ++++++++++++++++++++++++++++++++++++
>>> arch/x86/entry/entry_64.S | 6 ++----
>>> 2 files changed, 38 insertions(+), 4 deletions(-)
>>>
>>> diff --git a/arch/x86/entry/calling.h b/arch/x86/entry/calling.h
>>> index a05cbb8..57b1b87 100644
>>> --- a/arch/x86/entry/calling.h
>>> +++ b/arch/x86/entry/calling.h
>>> @@ -137,6 +137,42 @@ For 32-bit we have the following conventions - kernel is built with
>>> UNWIND_HINT_REGS offset=\offset
>>> .endm
>>>
>>> + .macro PUSH_AND_CLEAR_REGS
>>> + /*
>>> + * Push registers and sanitize registers of values that a
>>> + * speculation attack might otherwise want to exploit. The
>>> + * lower registers are likely clobbered well before they
>>> + * could be put to use in a speculative execution gadget.
>>> + * Interleave XOR with PUSH for better uop scheduling:
>>> + */
>>> + pushq %rdi /* pt_regs->di */
>>> + pushq %rsi /* pt_regs->si */
>>> + pushq %rdx /* pt_regs->dx */
>>> + pushq %rcx /* pt_regs->cx */
>>> + pushq %rax /* pt_regs->ax */
>>> + pushq %r8 /* pt_regs->r8 */
>>> + xorq %r8, %r8 /* nospec r8 */
>>
>> xorq's are slower than xorl's on Silvermont/Knights Landing.
>> I propose using xorl instead.
>
> Does using movq to copy the first zero to the other registers make
> the code any faster?
>
> ISTR mov reg-reg is often implemented as a register rename rather than an
> alu operation.
xorl is implemented in register rename as well. Just, for some reason,
xorq did not get the same treatment on those CPUs.
^ permalink raw reply [flat|nested] 33+ messages in thread
* Re: [tip:x86/pti] x86/entry/64: Introduce the PUSH_AND_CLEAN_REGS macro
2018-02-12 13:29 ` Denys Vlasenko
2018-02-12 13:36 ` David Laight
@ 2018-02-12 16:51 ` Linus Torvalds
1 sibling, 0 replies; 33+ messages in thread
From: Linus Torvalds @ 2018-02-12 16:51 UTC (permalink / raw)
To: Denys Vlasenko
Cc: Peter Anvin, Andrew Lutomirski, Ingo Molnar, Borislav Petkov,
Linux Kernel Mailing List, Dominik Brodowski, Brian Gerst,
Peter Zijlstra, Thomas Gleixner, Josh Poimboeuf,
linux-tip-commits
On Mon, Feb 12, 2018 at 5:29 AM, Denys Vlasenko <dvlasenk@redhat.com> wrote:
>
> xorq's are slower than xorl's on Silvermont/Knights Landing.
> I propose using xorl instead.
Makes sense, and matches the other 'xorl'.
I suspect the only reason it uses 'xorq' for the high regs is that the
register names for the high regs are "odd" and don't match the legacy
register names. You need to use "%r15d'" for the 32-bit version.
Linus
^ permalink raw reply [flat|nested] 33+ messages in thread
* Re: [PATCH v3 8/7] TESTING_ONLY x86/entry: reduce static footprint of idtentry
2018-02-12 9:37 ` Ingo Molnar
@ 2018-02-12 19:17 ` Linus Torvalds
2018-02-13 12:41 ` Ingo Molnar
0 siblings, 1 reply; 33+ messages in thread
From: Linus Torvalds @ 2018-02-12 19:17 UTC (permalink / raw)
To: Ingo Molnar
Cc: Dominik Brodowski, Linux Kernel Mailing List,
the arch/x86 maintainers, Dan Williams, Thomas Gleixner,
Andi Kleen, Andrew Lutomirski, Peter Zijlstra, Denys Vlasenko,
Josh Poimboeuf, Brian Gerst, Borislav Petkov
On Mon, Feb 12, 2018 at 1:37 AM, Ingo Molnar <mingo@kernel.org> wrote:
>
> Ok, so this does not look _that_ complicated, and the .text savings are
> significant:
Honestly, I think we should do it. 3kB of assembly code is noticeable.
Also, that patch actually allows more cleanups and simplifications.
Look at the "interrupt" macro, which is used by 'apicinterrupt3', and
has a number of uses that way.
That code could be unified a lot, right now it does:
testb $3, CS-ORIG_RAX(%rsp)
jz 1f
SWAPGS
call switch_to_thread_stack
1:
ALLOC_PT_GPREGS_ON_STACK
SAVE_C_REGS
SAVE_EXTRA_REGS
ENCODE_FRAME_POINTER
testb $3, CS(%rsp)
jz 1f
/*
* IRQ from user mode.
*
* We need to tell lockdep that IRQs are off. We can't do this until
* we fix gsbase, and we should do it before enter_from_user_mode
* (which can take locks). Since TRACE_IRQS_OFF idempotent,
* the simplest way to handle it is to just call it twice if
* we enter from user mode. There's no reason to optimize this since
* TRACE_IRQS_OFF is a no-op if lockdep is off.
*/
TRACE_IRQS_OFF
CALL_enter_from_user_mode
1:
ENTER_IRQ_STACK old_rsp=%rdi
/* We entered an interrupt context - irqs are off: */
TRACE_IRQS_OFF
and *all* of that could be in a helper function rather than be
duplicated. and the apicinterrupt3 macro should end up just expanding
to
callq helper
pushq $~(\num)
callq \fn
jmp ret_from_intr
instead of expanding to all that code.
But that would require that same "save_ret" logic.
So it's not just the idtentry cases that can use this trick.
I admit that the trick isn't pretty, but it's not *horribly* ugly either.
Linus
^ permalink raw reply [flat|nested] 33+ messages in thread
* [PATCH] x86/entry/64: Remove unused icebp macro
2018-02-12 10:18 ` [tip:x86/pti] x86/entry/64: Indent " tip-bot for Dominik Brodowski
@ 2018-02-12 20:13 ` Borislav Petkov
2018-02-13 7:36 ` Ingo Molnar
2018-02-13 9:04 ` [tip:x86/pti] x86/entry/64: Remove the unused 'icebp' macro tip-bot for Borislav Petkov
0 siblings, 2 replies; 33+ messages in thread
From: Borislav Petkov @ 2018-02-12 20:13 UTC (permalink / raw)
To: x86-ml
Cc: linux-tip-commits, linux-kernel, dvlasenk, luto, torvalds,
jpoimboe, hpa, brgerst, linux, mingo, peterz, tglx
On Mon, Feb 12, 2018 at 02:18:55AM -0800, tip-bot for Dominik Brodowski wrote:
> Commit-ID: 5a10e729bc0aa9420a23a70b169eadd229c27bc1
> Gitweb: https://git.kernel.org/tip/5a10e729bc0aa9420a23a70b169eadd229c27bc1
> Author: Dominik Brodowski <linux@dominikbrodowski.net>
> AuthorDate: Sun, 11 Feb 2018 11:49:48 +0100
> Committer: Ingo Molnar <mingo@kernel.org>
> CommitDate: Mon, 12 Feb 2018 08:06:37 +0100
>
> x86/entry/64: Indent PUSH_AND_CLEAR_REGS and POP_REGS properly
Ontop:
---
From: Borislav Petkov <bp@suse.de>
That macro was touched around 2.5.8 times, judging by the full history
linux repo, but it was unused even then. Get rid of it already.
Signed-off-by: Borislav Petkov <bp@suse.de>
---
arch/x86/entry/calling.h | 4 ----
1 file changed, 4 deletions(-)
diff --git a/arch/x86/entry/calling.h b/arch/x86/entry/calling.h
index 6985440c68fa..dce7092ab24a 100644
--- a/arch/x86/entry/calling.h
+++ b/arch/x86/entry/calling.h
@@ -159,10 +159,6 @@ For 32-bit we have the following conventions - kernel is built with
.if \pop_rdi
popq %rdi
.endif
- .endm
-
- .macro icebp
- .byte 0xf1
.endm
/*
--
2.13.0
--
Regards/Gruss,
Boris.
Good mailing practices for 400: avoid top-posting and trim the reply.
^ permalink raw reply related [flat|nested] 33+ messages in thread
* Re: [PATCH] x86/entry/64: Remove unused icebp macro
2018-02-12 20:13 ` [PATCH] x86/entry/64: Remove unused icebp macro Borislav Petkov
@ 2018-02-13 7:36 ` Ingo Molnar
2018-02-13 9:04 ` [tip:x86/pti] x86/entry/64: Remove the unused 'icebp' macro tip-bot for Borislav Petkov
1 sibling, 0 replies; 33+ messages in thread
From: Ingo Molnar @ 2018-02-13 7:36 UTC (permalink / raw)
To: Borislav Petkov
Cc: x86-ml, linux-tip-commits, linux-kernel, dvlasenk, luto,
torvalds, jpoimboe, hpa, brgerst, linux, peterz, tglx
* Borislav Petkov <bp@alien8.de> wrote:
> On Mon, Feb 12, 2018 at 02:18:55AM -0800, tip-bot for Dominik Brodowski wrote:
> > Commit-ID: 5a10e729bc0aa9420a23a70b169eadd229c27bc1
> > Gitweb: https://git.kernel.org/tip/5a10e729bc0aa9420a23a70b169eadd229c27bc1
> > Author: Dominik Brodowski <linux@dominikbrodowski.net>
> > AuthorDate: Sun, 11 Feb 2018 11:49:48 +0100
> > Committer: Ingo Molnar <mingo@kernel.org>
> > CommitDate: Mon, 12 Feb 2018 08:06:37 +0100
> >
> > x86/entry/64: Indent PUSH_AND_CLEAR_REGS and POP_REGS properly
>
> Ontop:
>
> ---
> From: Borislav Petkov <bp@suse.de>
>
> That macro was touched around 2.5.8 times, judging by the full history
> linux repo, but it was unused even then. Get rid of it already.
>
> Signed-off-by: Borislav Petkov <bp@suse.de>
> ---
> arch/x86/entry/calling.h | 4 ----
> 1 file changed, 4 deletions(-)
>
> diff --git a/arch/x86/entry/calling.h b/arch/x86/entry/calling.h
> index 6985440c68fa..dce7092ab24a 100644
> --- a/arch/x86/entry/calling.h
> +++ b/arch/x86/entry/calling.h
> @@ -159,10 +159,6 @@ For 32-bit we have the following conventions - kernel is built with
> .if \pop_rdi
> popq %rdi
> .endif
> - .endm
> -
> - .macro icebp
> - .byte 0xf1
> .endm
Good find!
I applied this separately, and added your SOB: let me know if that's not OK.
Thanks,
Ingo
^ permalink raw reply [flat|nested] 33+ messages in thread
* [tip:x86/pti] x86/entry/64: Merge SAVE_C_REGS and SAVE_EXTRA_REGS, remove unused extensions
2018-02-11 10:49 ` [PATCH v3 1/7] x86/entry: merge SAVE_C_REGS and SAVE_EXTRA_REGS, remove unused extensions Dominik Brodowski
2018-02-12 10:15 ` [tip:x86/pti] x86/entry/64: Merge " tip-bot for Dominik Brodowski
@ 2018-02-13 9:00 ` tip-bot for Dominik Brodowski
1 sibling, 0 replies; 33+ messages in thread
From: tip-bot for Dominik Brodowski @ 2018-02-13 9:00 UTC (permalink / raw)
To: linux-tip-commits
Cc: luto, linux-kernel, mingo, hpa, tglx, brgerst, jpoimboe, peterz,
linux, dvlasenk, torvalds, bp
Commit-ID: 2e3f0098bc45f710a2f4cbcc94b80a1fae7a99a1
Gitweb: https://git.kernel.org/tip/2e3f0098bc45f710a2f4cbcc94b80a1fae7a99a1
Author: Dominik Brodowski <linux@dominikbrodowski.net>
AuthorDate: Sun, 11 Feb 2018 11:49:42 +0100
Committer: Ingo Molnar <mingo@kernel.org>
CommitDate: Tue, 13 Feb 2018 09:04:52 +0100
x86/entry/64: Merge SAVE_C_REGS and SAVE_EXTRA_REGS, remove unused extensions
All current code paths call SAVE_C_REGS and then immediately
SAVE_EXTRA_REGS. Therefore, merge these two macros and order the MOV
sequeneces properly.
While at it, remove the macros to save all except specific registers,
as these macros have been unused for a long time.
Suggested-by: Linus Torvalds <torvalds@linux-foundation.org>
Signed-off-by: Dominik Brodowski <linux@dominikbrodowski.net>
Cc: Andy Lutomirski <luto@kernel.org>
Cc: Borislav Petkov <bp@alien8.de>
Cc: Brian Gerst <brgerst@gmail.com>
Cc: Denys Vlasenko <dvlasenk@redhat.com>
Cc: H. Peter Anvin <hpa@zytor.com>
Cc: Josh Poimboeuf <jpoimboe@redhat.com>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Thomas Gleixner <tglx@linutronix.de>
Cc: dan.j.williams@intel.com
Link: http://lkml.kernel.org/r/20180211104949.12992-2-linux@dominikbrodowski.net
Signed-off-by: Ingo Molnar <mingo@kernel.org>
---
arch/x86/entry/calling.h | 57 +++++++++++++----------------------------------
arch/x86/entry/entry_64.S | 12 ++++------
2 files changed, 19 insertions(+), 50 deletions(-)
diff --git a/arch/x86/entry/calling.h b/arch/x86/entry/calling.h
index f4b129d..8907a65 100644
--- a/arch/x86/entry/calling.h
+++ b/arch/x86/entry/calling.h
@@ -101,49 +101,22 @@ For 32-bit we have the following conventions - kernel is built with
addq $-(15*8), %rsp
.endm
- .macro SAVE_C_REGS_HELPER offset=0 rax=1 rcx=1 r8910=1 r11=1
- .if \r11
- movq %r11, 6*8+\offset(%rsp)
- .endif
- .if \r8910
- movq %r10, 7*8+\offset(%rsp)
- movq %r9, 8*8+\offset(%rsp)
- movq %r8, 9*8+\offset(%rsp)
- .endif
- .if \rax
- movq %rax, 10*8+\offset(%rsp)
- .endif
- .if \rcx
- movq %rcx, 11*8+\offset(%rsp)
- .endif
- movq %rdx, 12*8+\offset(%rsp)
- movq %rsi, 13*8+\offset(%rsp)
+ .macro SAVE_REGS offset=0
movq %rdi, 14*8+\offset(%rsp)
- UNWIND_HINT_REGS offset=\offset extra=0
- .endm
- .macro SAVE_C_REGS offset=0
- SAVE_C_REGS_HELPER \offset, 1, 1, 1, 1
- .endm
- .macro SAVE_C_REGS_EXCEPT_RAX_RCX offset=0
- SAVE_C_REGS_HELPER \offset, 0, 0, 1, 1
- .endm
- .macro SAVE_C_REGS_EXCEPT_R891011
- SAVE_C_REGS_HELPER 0, 1, 1, 0, 0
- .endm
- .macro SAVE_C_REGS_EXCEPT_RCX_R891011
- SAVE_C_REGS_HELPER 0, 1, 0, 0, 0
- .endm
- .macro SAVE_C_REGS_EXCEPT_RAX_RCX_R11
- SAVE_C_REGS_HELPER 0, 0, 0, 1, 0
- .endm
-
- .macro SAVE_EXTRA_REGS offset=0
- movq %r15, 0*8+\offset(%rsp)
- movq %r14, 1*8+\offset(%rsp)
- movq %r13, 2*8+\offset(%rsp)
- movq %r12, 3*8+\offset(%rsp)
- movq %rbp, 4*8+\offset(%rsp)
+ movq %rsi, 13*8+\offset(%rsp)
+ movq %rdx, 12*8+\offset(%rsp)
+ movq %rcx, 11*8+\offset(%rsp)
+ movq %rax, 10*8+\offset(%rsp)
+ movq %r8, 9*8+\offset(%rsp)
+ movq %r9, 8*8+\offset(%rsp)
+ movq %r10, 7*8+\offset(%rsp)
+ movq %r11, 6*8+\offset(%rsp)
movq %rbx, 5*8+\offset(%rsp)
+ movq %rbp, 4*8+\offset(%rsp)
+ movq %r12, 3*8+\offset(%rsp)
+ movq %r13, 2*8+\offset(%rsp)
+ movq %r14, 1*8+\offset(%rsp)
+ movq %r15, 0*8+\offset(%rsp)
UNWIND_HINT_REGS offset=\offset
.endm
@@ -197,7 +170,7 @@ For 32-bit we have the following conventions - kernel is built with
* is just setting the LSB, which makes it an invalid stack address and is also
* a signal to the unwinder that it's a pt_regs pointer in disguise.
*
- * NOTE: This macro must be used *after* SAVE_EXTRA_REGS because it corrupts
+ * NOTE: This macro must be used *after* SAVE_REGS because it corrupts
* the original rbp.
*/
.macro ENCODE_FRAME_POINTER ptregs_offset=0
diff --git a/arch/x86/entry/entry_64.S b/arch/x86/entry/entry_64.S
index 932a445..1a6fc01 100644
--- a/arch/x86/entry/entry_64.S
+++ b/arch/x86/entry/entry_64.S
@@ -573,8 +573,7 @@ END(irq_entries_start)
1:
ALLOC_PT_GPREGS_ON_STACK
- SAVE_C_REGS
- SAVE_EXTRA_REGS
+ SAVE_REGS
CLEAR_REGS_NOSPEC
ENCODE_FRAME_POINTER
@@ -1132,8 +1131,7 @@ ENTRY(xen_failsafe_callback)
UNWIND_HINT_IRET_REGS
pushq $-1 /* orig_ax = -1 => not a system call */
ALLOC_PT_GPREGS_ON_STACK
- SAVE_C_REGS
- SAVE_EXTRA_REGS
+ SAVE_REGS
CLEAR_REGS_NOSPEC
ENCODE_FRAME_POINTER
jmp error_exit
@@ -1178,8 +1176,7 @@ idtentry machine_check do_mce has_error_code=0 paranoid=1
ENTRY(paranoid_entry)
UNWIND_HINT_FUNC
cld
- SAVE_C_REGS 8
- SAVE_EXTRA_REGS 8
+ SAVE_REGS 8
CLEAR_REGS_NOSPEC
ENCODE_FRAME_POINTER 8
movl $1, %ebx
@@ -1231,8 +1228,7 @@ END(paranoid_exit)
ENTRY(error_entry)
UNWIND_HINT_FUNC
cld
- SAVE_C_REGS 8
- SAVE_EXTRA_REGS 8
+ SAVE_REGS 8
CLEAR_REGS_NOSPEC
ENCODE_FRAME_POINTER 8
testb $3, CS+8(%rsp)
^ permalink raw reply related [flat|nested] 33+ messages in thread
* [tip:x86/pti] x86/entry/64: Merge the POP_C_REGS and POP_EXTRA_REGS macros into a single POP_REGS macro
2018-02-11 10:49 ` [PATCH v3 2/7] x86/entry: merge POP_C_REGS and POP_EXTRA_REGS Dominik Brodowski
2018-02-12 10:16 ` [tip:x86/pti] x86/entry/64: Merge the POP_C_REGS and POP_EXTRA_REGS macros into a single POP_REGS macro tip-bot for Dominik Brodowski
@ 2018-02-13 9:01 ` tip-bot for Dominik Brodowski
1 sibling, 0 replies; 33+ messages in thread
From: tip-bot for Dominik Brodowski @ 2018-02-13 9:01 UTC (permalink / raw)
To: linux-tip-commits
Cc: hpa, brgerst, linux-kernel, mingo, torvalds, bp, dvlasenk, linux,
luto, jpoimboe, tglx, peterz
Commit-ID: 502af0d70843c2a9405d7ba1f79b4b0305aaf5f5
Gitweb: https://git.kernel.org/tip/502af0d70843c2a9405d7ba1f79b4b0305aaf5f5
Author: Dominik Brodowski <linux@dominikbrodowski.net>
AuthorDate: Sun, 11 Feb 2018 11:49:43 +0100
Committer: Ingo Molnar <mingo@kernel.org>
CommitDate: Tue, 13 Feb 2018 09:04:52 +0100
x86/entry/64: Merge the POP_C_REGS and POP_EXTRA_REGS macros into a single POP_REGS macro
The two special, opencoded cases for POP_C_REGS can be handled by ASM
macros.
Signed-off-by: Dominik Brodowski <linux@dominikbrodowski.net>
Cc: Andy Lutomirski <luto@kernel.org>
Cc: Borislav Petkov <bp@alien8.de>
Cc: Brian Gerst <brgerst@gmail.com>
Cc: Denys Vlasenko <dvlasenk@redhat.com>
Cc: H. Peter Anvin <hpa@zytor.com>
Cc: Josh Poimboeuf <jpoimboe@redhat.com>
Cc: Linus Torvalds <torvalds@linux-foundation.org>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Thomas Gleixner <tglx@linutronix.de>
Cc: dan.j.williams@intel.com
Link: http://lkml.kernel.org/r/20180211104949.12992-3-linux@dominikbrodowski.net
Signed-off-by: Ingo Molnar <mingo@kernel.org>
---
arch/x86/entry/calling.h | 15 +++++++++++----
arch/x86/entry/entry_64.S | 26 ++++----------------------
2 files changed, 15 insertions(+), 26 deletions(-)
diff --git a/arch/x86/entry/calling.h b/arch/x86/entry/calling.h
index 8907a65..3bda317 100644
--- a/arch/x86/entry/calling.h
+++ b/arch/x86/entry/calling.h
@@ -139,25 +139,32 @@ For 32-bit we have the following conventions - kernel is built with
xorq %r15, %r15
.endm
- .macro POP_EXTRA_REGS
+ .macro POP_REGS pop_rdi=1 skip_r11rcx=0
popq %r15
popq %r14
popq %r13
popq %r12
popq %rbp
popq %rbx
- .endm
-
- .macro POP_C_REGS
+ .if \skip_r11rcx
+ popq %rsi
+ .else
popq %r11
+ .endif
popq %r10
popq %r9
popq %r8
popq %rax
+ .if \skip_r11rcx
+ popq %rsi
+ .else
popq %rcx
+ .endif
popq %rdx
popq %rsi
+ .if \pop_rdi
popq %rdi
+ .endif
.endm
.macro icebp
diff --git a/arch/x86/entry/entry_64.S b/arch/x86/entry/entry_64.S
index 1a6fc01..7351c91 100644
--- a/arch/x86/entry/entry_64.S
+++ b/arch/x86/entry/entry_64.S
@@ -334,15 +334,7 @@ GLOBAL(entry_SYSCALL_64_after_hwframe)
syscall_return_via_sysret:
/* rcx and r11 are already restored (see code above) */
UNWIND_HINT_EMPTY
- POP_EXTRA_REGS
- popq %rsi /* skip r11 */
- popq %r10
- popq %r9
- popq %r8
- popq %rax
- popq %rsi /* skip rcx */
- popq %rdx
- popq %rsi
+ POP_REGS pop_rdi=0 skip_r11rcx=1
/*
* Now all regs are restored except RSP and RDI.
@@ -635,15 +627,7 @@ GLOBAL(swapgs_restore_regs_and_return_to_usermode)
ud2
1:
#endif
- POP_EXTRA_REGS
- popq %r11
- popq %r10
- popq %r9
- popq %r8
- popq %rax
- popq %rcx
- popq %rdx
- popq %rsi
+ POP_REGS pop_rdi=0
/*
* The stack is now user RDI, orig_ax, RIP, CS, EFLAGS, RSP, SS.
@@ -701,8 +685,7 @@ GLOBAL(restore_regs_and_return_to_kernel)
ud2
1:
#endif
- POP_EXTRA_REGS
- POP_C_REGS
+ POP_REGS
addq $8, %rsp /* skip regs->orig_ax */
INTERRUPT_RETURN
@@ -1661,8 +1644,7 @@ end_repeat_nmi:
nmi_swapgs:
SWAPGS_UNSAFE_STACK
nmi_restore:
- POP_EXTRA_REGS
- POP_C_REGS
+ POP_REGS
/*
* Skip orig_ax and the "outermost" frame to point RSP at the "iret"
^ permalink raw reply related [flat|nested] 33+ messages in thread
* [tip:x86/pti] x86/entry/64: Interleave XOR register clearing with PUSH instructions
2018-02-11 10:49 ` [PATCH v3 3/7] x86/entry: interleave XOR register clearing with PUSH instructions Dominik Brodowski
2018-02-12 10:16 ` [tip:x86/pti] x86/entry/64: Interleave " tip-bot for Dominik Brodowski
@ 2018-02-13 9:01 ` tip-bot for Dominik Brodowski
1 sibling, 0 replies; 33+ messages in thread
From: tip-bot for Dominik Brodowski @ 2018-02-13 9:01 UTC (permalink / raw)
To: linux-tip-commits
Cc: linux-kernel, dvlasenk, brgerst, torvalds, luto, bp, mingo,
jpoimboe, tglx, peterz, linux, hpa
Commit-ID: f7bafa2b05ef25eda1d9179fd930b0330cf2b7d1
Gitweb: https://git.kernel.org/tip/f7bafa2b05ef25eda1d9179fd930b0330cf2b7d1
Author: Dominik Brodowski <linux@dominikbrodowski.net>
AuthorDate: Sun, 11 Feb 2018 11:49:44 +0100
Committer: Ingo Molnar <mingo@kernel.org>
CommitDate: Tue, 13 Feb 2018 09:04:53 +0100
x86/entry/64: Interleave XOR register clearing with PUSH instructions
Same as is done for syscalls, interleave XOR with PUSH instructions
for exceptions/interrupts, in order to minimize the cost of the
additional instructions required for register clearing.
Signed-off-by: Dominik Brodowski <linux@dominikbrodowski.net>
Cc: Andy Lutomirski <luto@kernel.org>
Cc: Borislav Petkov <bp@alien8.de>
Cc: Brian Gerst <brgerst@gmail.com>
Cc: Denys Vlasenko <dvlasenk@redhat.com>
Cc: H. Peter Anvin <hpa@zytor.com>
Cc: Josh Poimboeuf <jpoimboe@redhat.com>
Cc: Linus Torvalds <torvalds@linux-foundation.org>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Thomas Gleixner <tglx@linutronix.de>
Cc: dan.j.williams@intel.com
Link: http://lkml.kernel.org/r/20180211104949.12992-4-linux@dominikbrodowski.net
Signed-off-by: Ingo Molnar <mingo@kernel.org>
---
arch/x86/entry/calling.h | 40 +++++++++++++++++++---------------------
arch/x86/entry/entry_64.S | 30 +++++++++++++++++++++---------
2 files changed, 40 insertions(+), 30 deletions(-)
diff --git a/arch/x86/entry/calling.h b/arch/x86/entry/calling.h
index 3bda317..a05cbb8 100644
--- a/arch/x86/entry/calling.h
+++ b/arch/x86/entry/calling.h
@@ -101,44 +101,42 @@ For 32-bit we have the following conventions - kernel is built with
addq $-(15*8), %rsp
.endm
- .macro SAVE_REGS offset=0
+ .macro SAVE_AND_CLEAR_REGS offset=0
+ /*
+ * Save registers and sanitize registers of values that a
+ * speculation attack might otherwise want to exploit. The
+ * lower registers are likely clobbered well before they
+ * could be put to use in a speculative execution gadget.
+ * Interleave XOR with PUSH for better uop scheduling:
+ */
movq %rdi, 14*8+\offset(%rsp)
movq %rsi, 13*8+\offset(%rsp)
movq %rdx, 12*8+\offset(%rsp)
movq %rcx, 11*8+\offset(%rsp)
movq %rax, 10*8+\offset(%rsp)
movq %r8, 9*8+\offset(%rsp)
+ xorq %r8, %r8 /* nospec r8 */
movq %r9, 8*8+\offset(%rsp)
+ xorq %r9, %r9 /* nospec r9 */
movq %r10, 7*8+\offset(%rsp)
+ xorq %r10, %r10 /* nospec r10 */
movq %r11, 6*8+\offset(%rsp)
+ xorq %r11, %r11 /* nospec r11 */
movq %rbx, 5*8+\offset(%rsp)
+ xorl %ebx, %ebx /* nospec rbx */
movq %rbp, 4*8+\offset(%rsp)
+ xorl %ebp, %ebp /* nospec rbp */
movq %r12, 3*8+\offset(%rsp)
+ xorq %r12, %r12 /* nospec r12 */
movq %r13, 2*8+\offset(%rsp)
+ xorq %r13, %r13 /* nospec r13 */
movq %r14, 1*8+\offset(%rsp)
+ xorq %r14, %r14 /* nospec r14 */
movq %r15, 0*8+\offset(%rsp)
+ xorq %r15, %r15 /* nospec r15 */
UNWIND_HINT_REGS offset=\offset
.endm
- /*
- * Sanitize registers of values that a speculation attack
- * might otherwise want to exploit. The lower registers are
- * likely clobbered well before they could be put to use in
- * a speculative execution gadget:
- */
- .macro CLEAR_REGS_NOSPEC
- xorl %ebp, %ebp
- xorl %ebx, %ebx
- xorq %r8, %r8
- xorq %r9, %r9
- xorq %r10, %r10
- xorq %r11, %r11
- xorq %r12, %r12
- xorq %r13, %r13
- xorq %r14, %r14
- xorq %r15, %r15
- .endm
-
.macro POP_REGS pop_rdi=1 skip_r11rcx=0
popq %r15
popq %r14
@@ -177,7 +175,7 @@ For 32-bit we have the following conventions - kernel is built with
* is just setting the LSB, which makes it an invalid stack address and is also
* a signal to the unwinder that it's a pt_regs pointer in disguise.
*
- * NOTE: This macro must be used *after* SAVE_REGS because it corrupts
+ * NOTE: This macro must be used *after* SAVE_AND_CLEAR_REGS because it corrupts
* the original rbp.
*/
.macro ENCODE_FRAME_POINTER ptregs_offset=0
diff --git a/arch/x86/entry/entry_64.S b/arch/x86/entry/entry_64.S
index 7351c91..07692b4 100644
--- a/arch/x86/entry/entry_64.S
+++ b/arch/x86/entry/entry_64.S
@@ -565,8 +565,7 @@ END(irq_entries_start)
1:
ALLOC_PT_GPREGS_ON_STACK
- SAVE_REGS
- CLEAR_REGS_NOSPEC
+ SAVE_AND_CLEAR_REGS
ENCODE_FRAME_POINTER
testb $3, CS(%rsp)
@@ -1114,8 +1113,7 @@ ENTRY(xen_failsafe_callback)
UNWIND_HINT_IRET_REGS
pushq $-1 /* orig_ax = -1 => not a system call */
ALLOC_PT_GPREGS_ON_STACK
- SAVE_REGS
- CLEAR_REGS_NOSPEC
+ SAVE_AND_CLEAR_REGS
ENCODE_FRAME_POINTER
jmp error_exit
END(xen_failsafe_callback)
@@ -1159,8 +1157,7 @@ idtentry machine_check do_mce has_error_code=0 paranoid=1
ENTRY(paranoid_entry)
UNWIND_HINT_FUNC
cld
- SAVE_REGS 8
- CLEAR_REGS_NOSPEC
+ SAVE_AND_CLEAR_REGS 8
ENCODE_FRAME_POINTER 8
movl $1, %ebx
movl $MSR_GS_BASE, %ecx
@@ -1211,8 +1208,7 @@ END(paranoid_exit)
ENTRY(error_entry)
UNWIND_HINT_FUNC
cld
- SAVE_REGS 8
- CLEAR_REGS_NOSPEC
+ SAVE_AND_CLEAR_REGS 8
ENCODE_FRAME_POINTER 8
testb $3, CS+8(%rsp)
jz .Lerror_kernelspace
@@ -1399,18 +1395,34 @@ ENTRY(nmi)
pushq (%rdx) /* pt_regs->dx */
pushq %rcx /* pt_regs->cx */
pushq %rax /* pt_regs->ax */
+ /*
+ * Sanitize registers of values that a speculation attack
+ * might otherwise want to exploit. The lower registers are
+ * likely clobbered well before they could be put to use in
+ * a speculative execution gadget. Interleave XOR with PUSH
+ * for better uop scheduling:
+ */
pushq %r8 /* pt_regs->r8 */
+ xorq %r8, %r8 /* nospec r8 */
pushq %r9 /* pt_regs->r9 */
+ xorq %r9, %r9 /* nospec r9 */
pushq %r10 /* pt_regs->r10 */
+ xorq %r10, %r10 /* nospec r10 */
pushq %r11 /* pt_regs->r11 */
+ xorq %r11, %r11 /* nospec r11*/
pushq %rbx /* pt_regs->rbx */
+ xorl %ebx, %ebx /* nospec rbx*/
pushq %rbp /* pt_regs->rbp */
+ xorl %ebp, %ebp /* nospec rbp*/
pushq %r12 /* pt_regs->r12 */
+ xorq %r12, %r12 /* nospec r12*/
pushq %r13 /* pt_regs->r13 */
+ xorq %r13, %r13 /* nospec r13*/
pushq %r14 /* pt_regs->r14 */
+ xorq %r14, %r14 /* nospec r14*/
pushq %r15 /* pt_regs->r15 */
+ xorq %r15, %r15 /* nospec r15*/
UNWIND_HINT_REGS
- CLEAR_REGS_NOSPEC
ENCODE_FRAME_POINTER
/*
^ permalink raw reply related [flat|nested] 33+ messages in thread
* [tip:x86/pti] x86/entry/64: Introduce the PUSH_AND_CLEAN_REGS macro
2018-02-11 10:49 ` [PATCH v3 4/7] x86/entry: introduce PUSH_AND_CLEAN_REGS Dominik Brodowski
2018-02-12 10:17 ` [tip:x86/pti] x86/entry/64: Introduce the PUSH_AND_CLEAN_REGS macro tip-bot for Dominik Brodowski
@ 2018-02-13 9:01 ` tip-bot for Dominik Brodowski
1 sibling, 0 replies; 33+ messages in thread
From: tip-bot for Dominik Brodowski @ 2018-02-13 9:01 UTC (permalink / raw)
To: linux-tip-commits
Cc: peterz, torvalds, jpoimboe, tglx, linux-kernel, linux, bp,
brgerst, dvlasenk, mingo, luto, hpa
Commit-ID: 3f01daecd545e818098d84fd1ad43e19a508d705
Gitweb: https://git.kernel.org/tip/3f01daecd545e818098d84fd1ad43e19a508d705
Author: Dominik Brodowski <linux@dominikbrodowski.net>
AuthorDate: Sun, 11 Feb 2018 11:49:45 +0100
Committer: Ingo Molnar <mingo@kernel.org>
CommitDate: Tue, 13 Feb 2018 09:04:53 +0100
x86/entry/64: Introduce the PUSH_AND_CLEAN_REGS macro
Those instances where ALLOC_PT_GPREGS_ON_STACK is called just before
SAVE_AND_CLEAR_REGS can trivially be replaced by PUSH_AND_CLEAN_REGS.
This macro uses PUSH instead of MOV and should therefore be faster, at
least on newer CPUs.
Suggested-by: Linus Torvalds <torvalds@linux-foundation.org>
Signed-off-by: Dominik Brodowski <linux@dominikbrodowski.net>
Cc: Andy Lutomirski <luto@kernel.org>
Cc: Borislav Petkov <bp@alien8.de>
Cc: Brian Gerst <brgerst@gmail.com>
Cc: Denys Vlasenko <dvlasenk@redhat.com>
Cc: H. Peter Anvin <hpa@zytor.com>
Cc: Josh Poimboeuf <jpoimboe@redhat.com>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Thomas Gleixner <tglx@linutronix.de>
Cc: dan.j.williams@intel.com
Link: http://lkml.kernel.org/r/20180211104949.12992-5-linux@dominikbrodowski.net
Signed-off-by: Ingo Molnar <mingo@kernel.org>
---
arch/x86/entry/calling.h | 36 ++++++++++++++++++++++++++++++++++++
arch/x86/entry/entry_64.S | 6 ++----
2 files changed, 38 insertions(+), 4 deletions(-)
diff --git a/arch/x86/entry/calling.h b/arch/x86/entry/calling.h
index a05cbb8..57b1b87 100644
--- a/arch/x86/entry/calling.h
+++ b/arch/x86/entry/calling.h
@@ -137,6 +137,42 @@ For 32-bit we have the following conventions - kernel is built with
UNWIND_HINT_REGS offset=\offset
.endm
+ .macro PUSH_AND_CLEAR_REGS
+ /*
+ * Push registers and sanitize registers of values that a
+ * speculation attack might otherwise want to exploit. The
+ * lower registers are likely clobbered well before they
+ * could be put to use in a speculative execution gadget.
+ * Interleave XOR with PUSH for better uop scheduling:
+ */
+ pushq %rdi /* pt_regs->di */
+ pushq %rsi /* pt_regs->si */
+ pushq %rdx /* pt_regs->dx */
+ pushq %rcx /* pt_regs->cx */
+ pushq %rax /* pt_regs->ax */
+ pushq %r8 /* pt_regs->r8 */
+ xorq %r8, %r8 /* nospec r8 */
+ pushq %r9 /* pt_regs->r9 */
+ xorq %r9, %r9 /* nospec r9 */
+ pushq %r10 /* pt_regs->r10 */
+ xorq %r10, %r10 /* nospec r10 */
+ pushq %r11 /* pt_regs->r11 */
+ xorq %r11, %r11 /* nospec r11*/
+ pushq %rbx /* pt_regs->rbx */
+ xorl %ebx, %ebx /* nospec rbx*/
+ pushq %rbp /* pt_regs->rbp */
+ xorl %ebp, %ebp /* nospec rbp*/
+ pushq %r12 /* pt_regs->r12 */
+ xorq %r12, %r12 /* nospec r12*/
+ pushq %r13 /* pt_regs->r13 */
+ xorq %r13, %r13 /* nospec r13*/
+ pushq %r14 /* pt_regs->r14 */
+ xorq %r14, %r14 /* nospec r14*/
+ pushq %r15 /* pt_regs->r15 */
+ xorq %r15, %r15 /* nospec r15*/
+ UNWIND_HINT_REGS
+ .endm
+
.macro POP_REGS pop_rdi=1 skip_r11rcx=0
popq %r15
popq %r14
diff --git a/arch/x86/entry/entry_64.S b/arch/x86/entry/entry_64.S
index 07692b4..cf4a9ae 100644
--- a/arch/x86/entry/entry_64.S
+++ b/arch/x86/entry/entry_64.S
@@ -564,8 +564,7 @@ END(irq_entries_start)
call switch_to_thread_stack
1:
- ALLOC_PT_GPREGS_ON_STACK
- SAVE_AND_CLEAR_REGS
+ PUSH_AND_CLEAR_REGS
ENCODE_FRAME_POINTER
testb $3, CS(%rsp)
@@ -1112,8 +1111,7 @@ ENTRY(xen_failsafe_callback)
addq $0x30, %rsp
UNWIND_HINT_IRET_REGS
pushq $-1 /* orig_ax = -1 => not a system call */
- ALLOC_PT_GPREGS_ON_STACK
- SAVE_AND_CLEAR_REGS
+ PUSH_AND_CLEAR_REGS
ENCODE_FRAME_POINTER
jmp error_exit
END(xen_failsafe_callback)
^ permalink raw reply related [flat|nested] 33+ messages in thread
* [tip:x86/pti] x86/entry/64: Use PUSH_AND_CLEAN_REGS in more cases
2018-02-11 10:49 ` [PATCH v3 5/7] x86/entry: use PUSH_AND_CLEAN_REGS in more cases Dominik Brodowski
2018-02-12 10:17 ` [tip:x86/pti] x86/entry/64: Use " tip-bot for Dominik Brodowski
@ 2018-02-13 9:02 ` tip-bot for Dominik Brodowski
1 sibling, 0 replies; 33+ messages in thread
From: tip-bot for Dominik Brodowski @ 2018-02-13 9:02 UTC (permalink / raw)
To: linux-tip-commits
Cc: hpa, linux-kernel, luto, torvalds, tglx, bp, brgerst, linux,
mingo, jpoimboe, peterz, dvlasenk
Commit-ID: 30907fd13bb593202574bb20af58d67c70a1ee14
Gitweb: https://git.kernel.org/tip/30907fd13bb593202574bb20af58d67c70a1ee14
Author: Dominik Brodowski <linux@dominikbrodowski.net>
AuthorDate: Sun, 11 Feb 2018 11:49:46 +0100
Committer: Ingo Molnar <mingo@kernel.org>
CommitDate: Tue, 13 Feb 2018 09:04:53 +0100
x86/entry/64: Use PUSH_AND_CLEAN_REGS in more cases
entry_SYSCALL_64_after_hwframe() and nmi() can be converted to use
PUSH_AND_CLEAN_REGS instead of opencoded variants thereof. Due to
the interleaving, the additional XOR-based clearing of R8 and R9
in entry_SYSCALL_64_after_hwframe() should not have any noticeable
negative implications.
Suggested-by: Linus Torvalds <torvalds@linux-foundation.org>
Signed-off-by: Dominik Brodowski <linux@dominikbrodowski.net>
Cc: Andy Lutomirski <luto@kernel.org>
Cc: Borislav Petkov <bp@alien8.de>
Cc: Brian Gerst <brgerst@gmail.com>
Cc: Denys Vlasenko <dvlasenk@redhat.com>
Cc: H. Peter Anvin <hpa@zytor.com>
Cc: Josh Poimboeuf <jpoimboe@redhat.com>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Thomas Gleixner <tglx@linutronix.de>
Cc: dan.j.williams@intel.com
Link: http://lkml.kernel.org/r/20180211104949.12992-6-linux@dominikbrodowski.net
Signed-off-by: Ingo Molnar <mingo@kernel.org>
---
arch/x86/entry/calling.h | 6 ++---
arch/x86/entry/entry_64.S | 65 +++--------------------------------------------
2 files changed, 6 insertions(+), 65 deletions(-)
diff --git a/arch/x86/entry/calling.h b/arch/x86/entry/calling.h
index 57b1b87..d6a97e2 100644
--- a/arch/x86/entry/calling.h
+++ b/arch/x86/entry/calling.h
@@ -137,7 +137,7 @@ For 32-bit we have the following conventions - kernel is built with
UNWIND_HINT_REGS offset=\offset
.endm
- .macro PUSH_AND_CLEAR_REGS
+ .macro PUSH_AND_CLEAR_REGS rdx=%rdx rax=%rax
/*
* Push registers and sanitize registers of values that a
* speculation attack might otherwise want to exploit. The
@@ -147,9 +147,9 @@ For 32-bit we have the following conventions - kernel is built with
*/
pushq %rdi /* pt_regs->di */
pushq %rsi /* pt_regs->si */
- pushq %rdx /* pt_regs->dx */
+ pushq \rdx /* pt_regs->dx */
pushq %rcx /* pt_regs->cx */
- pushq %rax /* pt_regs->ax */
+ pushq \rax /* pt_regs->ax */
pushq %r8 /* pt_regs->r8 */
xorq %r8, %r8 /* nospec r8 */
pushq %r9 /* pt_regs->r9 */
diff --git a/arch/x86/entry/entry_64.S b/arch/x86/entry/entry_64.S
index cf4a9ae..b06a4b5 100644
--- a/arch/x86/entry/entry_64.S
+++ b/arch/x86/entry/entry_64.S
@@ -227,35 +227,8 @@ ENTRY(entry_SYSCALL_64)
pushq %rcx /* pt_regs->ip */
GLOBAL(entry_SYSCALL_64_after_hwframe)
pushq %rax /* pt_regs->orig_ax */
- pushq %rdi /* pt_regs->di */
- pushq %rsi /* pt_regs->si */
- pushq %rdx /* pt_regs->dx */
- pushq %rcx /* pt_regs->cx */
- pushq $-ENOSYS /* pt_regs->ax */
- pushq %r8 /* pt_regs->r8 */
- pushq %r9 /* pt_regs->r9 */
- pushq %r10 /* pt_regs->r10 */
- /*
- * Clear extra registers that a speculation attack might
- * otherwise want to exploit. Interleave XOR with PUSH
- * for better uop scheduling:
- */
- xorq %r10, %r10 /* nospec r10 */
- pushq %r11 /* pt_regs->r11 */
- xorq %r11, %r11 /* nospec r11 */
- pushq %rbx /* pt_regs->rbx */
- xorl %ebx, %ebx /* nospec rbx */
- pushq %rbp /* pt_regs->rbp */
- xorl %ebp, %ebp /* nospec rbp */
- pushq %r12 /* pt_regs->r12 */
- xorq %r12, %r12 /* nospec r12 */
- pushq %r13 /* pt_regs->r13 */
- xorq %r13, %r13 /* nospec r13 */
- pushq %r14 /* pt_regs->r14 */
- xorq %r14, %r14 /* nospec r14 */
- pushq %r15 /* pt_regs->r15 */
- xorq %r15, %r15 /* nospec r15 */
- UNWIND_HINT_REGS
+
+ PUSH_AND_CLEAR_REGS rax=$-ENOSYS
TRACE_IRQS_OFF
@@ -1388,39 +1361,7 @@ ENTRY(nmi)
pushq 1*8(%rdx) /* pt_regs->rip */
UNWIND_HINT_IRET_REGS
pushq $-1 /* pt_regs->orig_ax */
- pushq %rdi /* pt_regs->di */
- pushq %rsi /* pt_regs->si */
- pushq (%rdx) /* pt_regs->dx */
- pushq %rcx /* pt_regs->cx */
- pushq %rax /* pt_regs->ax */
- /*
- * Sanitize registers of values that a speculation attack
- * might otherwise want to exploit. The lower registers are
- * likely clobbered well before they could be put to use in
- * a speculative execution gadget. Interleave XOR with PUSH
- * for better uop scheduling:
- */
- pushq %r8 /* pt_regs->r8 */
- xorq %r8, %r8 /* nospec r8 */
- pushq %r9 /* pt_regs->r9 */
- xorq %r9, %r9 /* nospec r9 */
- pushq %r10 /* pt_regs->r10 */
- xorq %r10, %r10 /* nospec r10 */
- pushq %r11 /* pt_regs->r11 */
- xorq %r11, %r11 /* nospec r11*/
- pushq %rbx /* pt_regs->rbx */
- xorl %ebx, %ebx /* nospec rbx*/
- pushq %rbp /* pt_regs->rbp */
- xorl %ebp, %ebp /* nospec rbp*/
- pushq %r12 /* pt_regs->r12 */
- xorq %r12, %r12 /* nospec r12*/
- pushq %r13 /* pt_regs->r13 */
- xorq %r13, %r13 /* nospec r13*/
- pushq %r14 /* pt_regs->r14 */
- xorq %r14, %r14 /* nospec r14*/
- pushq %r15 /* pt_regs->r15 */
- xorq %r15, %r15 /* nospec r15*/
- UNWIND_HINT_REGS
+ PUSH_AND_CLEAR_REGS rdx=(%rdx)
ENCODE_FRAME_POINTER
/*
^ permalink raw reply related [flat|nested] 33+ messages in thread
* [tip:x86/pti] x86/entry/64: Get rid of the ALLOC_PT_GPREGS_ON_STACK and SAVE_AND_CLEAR_REGS macros
2018-02-11 10:49 ` [PATCH v3 6/7] x86/entry: get rid of ALLOC_PT_GPREGS_ON_STACK and SAVE_AND_CLEAR_REGS Dominik Brodowski
2018-02-12 10:18 ` [tip:x86/pti] x86/entry/64: Get rid of the ALLOC_PT_GPREGS_ON_STACK and SAVE_AND_CLEAR_REGS macros tip-bot for Dominik Brodowski
@ 2018-02-13 9:02 ` tip-bot for Dominik Brodowski
1 sibling, 0 replies; 33+ messages in thread
From: tip-bot for Dominik Brodowski @ 2018-02-13 9:02 UTC (permalink / raw)
To: linux-tip-commits
Cc: hpa, brgerst, linux-kernel, torvalds, linux, jpoimboe, tglx,
luto, dvlasenk, peterz, bp, mingo
Commit-ID: dde3036d62ba3375840b10ab9ec0d568fd773b07
Gitweb: https://git.kernel.org/tip/dde3036d62ba3375840b10ab9ec0d568fd773b07
Author: Dominik Brodowski <linux@dominikbrodowski.net>
AuthorDate: Sun, 11 Feb 2018 11:49:47 +0100
Committer: Ingo Molnar <mingo@kernel.org>
CommitDate: Tue, 13 Feb 2018 09:04:54 +0100
x86/entry/64: Get rid of the ALLOC_PT_GPREGS_ON_STACK and SAVE_AND_CLEAR_REGS macros
Previously, error_entry() and paranoid_entry() saved the GP registers
onto stack space previously allocated by its callers. Combine these two
steps in the callers, and use the generic PUSH_AND_CLEAR_REGS macro
for that.
This adds a significant amount ot text size. However, Ingo Molnar points
out that:
"these numbers also _very_ significantly over-represent the
extra footprint. The assumptions that resulted in
us compressing the IRQ entry code have changed very
significantly with the new x86 IRQ allocation code we
introduced in the last year:
- IRQ vectors are usually populated in tightly clustered
groups.
With our new vector allocator code the typical per CPU
allocation percentage on x86 systems is ~3 device vectors
and ~10 fixed vectors out of ~220 vectors - i.e. a very
low ~6% utilization (!). [...]
The days where we allocated a lot of vectors on every
CPU and the compression of the IRQ entry code text
mattered are over.
- Another issue is that only a small minority of vectors
is frequent enough to actually matter to cache utilization
in practice: 3-4 key IPIs and 1-2 device IRQs at most - and
those vectors tend to be tightly clustered as well into about
two groups, and are probably already on 2-3 cache lines in
practice.
For the common case of 'cache cold' IRQs it's the depth of
the call chain and the fragmentation of the resulting I$
that should be the main performance limit - not the overall
size of it.
- The CPU side cost of IRQ delivery is still very expensive
even in the best, most cached case, as in 'over a thousand
cycles'. So much stuff is done that maybe contemporary x86
IRQ entry microcode already prefetches the IDT entry and its
expected call target address."[*]
[*] http://lkml.kernel.org/r/20180208094710.qnjixhm6hybebdv7@gmail.com
The "testb $3, CS(%rsp)" instruction in the idtentry macro does not need
modification. Previously, %rsp was manually decreased by 15*8; with
this patch, %rsp is decreased by 15 pushq instructions.
[jpoimboe@redhat.com: unwind hint improvements]
Suggested-by: Linus Torvalds <torvalds@linux-foundation.org>
Signed-off-by: Dominik Brodowski <linux@dominikbrodowski.net>
Cc: Andy Lutomirski <luto@kernel.org>
Cc: Borislav Petkov <bp@alien8.de>
Cc: Brian Gerst <brgerst@gmail.com>
Cc: Denys Vlasenko <dvlasenk@redhat.com>
Cc: H. Peter Anvin <hpa@zytor.com>
Cc: Josh Poimboeuf <jpoimboe@redhat.com>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Thomas Gleixner <tglx@linutronix.de>
Cc: dan.j.williams@intel.com
Link: http://lkml.kernel.org/r/20180211104949.12992-7-linux@dominikbrodowski.net
Signed-off-by: Ingo Molnar <mingo@kernel.org>
---
arch/x86/entry/calling.h | 42 +-----------------------------------------
arch/x86/entry/entry_64.S | 20 +++++++++-----------
2 files changed, 10 insertions(+), 52 deletions(-)
diff --git a/arch/x86/entry/calling.h b/arch/x86/entry/calling.h
index d6a97e2..5967501 100644
--- a/arch/x86/entry/calling.h
+++ b/arch/x86/entry/calling.h
@@ -97,46 +97,6 @@ For 32-bit we have the following conventions - kernel is built with
#define SIZEOF_PTREGS 21*8
- .macro ALLOC_PT_GPREGS_ON_STACK
- addq $-(15*8), %rsp
- .endm
-
- .macro SAVE_AND_CLEAR_REGS offset=0
- /*
- * Save registers and sanitize registers of values that a
- * speculation attack might otherwise want to exploit. The
- * lower registers are likely clobbered well before they
- * could be put to use in a speculative execution gadget.
- * Interleave XOR with PUSH for better uop scheduling:
- */
- movq %rdi, 14*8+\offset(%rsp)
- movq %rsi, 13*8+\offset(%rsp)
- movq %rdx, 12*8+\offset(%rsp)
- movq %rcx, 11*8+\offset(%rsp)
- movq %rax, 10*8+\offset(%rsp)
- movq %r8, 9*8+\offset(%rsp)
- xorq %r8, %r8 /* nospec r8 */
- movq %r9, 8*8+\offset(%rsp)
- xorq %r9, %r9 /* nospec r9 */
- movq %r10, 7*8+\offset(%rsp)
- xorq %r10, %r10 /* nospec r10 */
- movq %r11, 6*8+\offset(%rsp)
- xorq %r11, %r11 /* nospec r11 */
- movq %rbx, 5*8+\offset(%rsp)
- xorl %ebx, %ebx /* nospec rbx */
- movq %rbp, 4*8+\offset(%rsp)
- xorl %ebp, %ebp /* nospec rbp */
- movq %r12, 3*8+\offset(%rsp)
- xorq %r12, %r12 /* nospec r12 */
- movq %r13, 2*8+\offset(%rsp)
- xorq %r13, %r13 /* nospec r13 */
- movq %r14, 1*8+\offset(%rsp)
- xorq %r14, %r14 /* nospec r14 */
- movq %r15, 0*8+\offset(%rsp)
- xorq %r15, %r15 /* nospec r15 */
- UNWIND_HINT_REGS offset=\offset
- .endm
-
.macro PUSH_AND_CLEAR_REGS rdx=%rdx rax=%rax
/*
* Push registers and sanitize registers of values that a
@@ -211,7 +171,7 @@ For 32-bit we have the following conventions - kernel is built with
* is just setting the LSB, which makes it an invalid stack address and is also
* a signal to the unwinder that it's a pt_regs pointer in disguise.
*
- * NOTE: This macro must be used *after* SAVE_AND_CLEAR_REGS because it corrupts
+ * NOTE: This macro must be used *after* PUSH_AND_CLEAR_REGS because it corrupts
* the original rbp.
*/
.macro ENCODE_FRAME_POINTER ptregs_offset=0
diff --git a/arch/x86/entry/entry_64.S b/arch/x86/entry/entry_64.S
index b06a4b5..cfbf433 100644
--- a/arch/x86/entry/entry_64.S
+++ b/arch/x86/entry/entry_64.S
@@ -871,7 +871,9 @@ ENTRY(\sym)
pushq $-1 /* ORIG_RAX: no syscall to restart */
.endif
- ALLOC_PT_GPREGS_ON_STACK
+ /* Save all registers in pt_regs */
+ PUSH_AND_CLEAR_REGS
+ ENCODE_FRAME_POINTER
.if \paranoid < 2
testb $3, CS(%rsp) /* If coming from userspace, switch stacks */
@@ -1121,15 +1123,12 @@ idtentry machine_check do_mce has_error_code=0 paranoid=1
#endif
/*
- * Save all registers in pt_regs, and switch gs if needed.
+ * Switch gs if needed.
* Use slow, but surefire "are we in kernel?" check.
* Return: ebx=0: need swapgs on exit, ebx=1: otherwise
*/
ENTRY(paranoid_entry)
- UNWIND_HINT_FUNC
cld
- SAVE_AND_CLEAR_REGS 8
- ENCODE_FRAME_POINTER 8
movl $1, %ebx
movl $MSR_GS_BASE, %ecx
rdmsr
@@ -1142,7 +1141,7 @@ ENTRY(paranoid_entry)
SAVE_AND_SWITCH_TO_KERNEL_CR3 scratch_reg=%rax save_reg=%r14
ret
-END(paranoid_entry)
+ENDPROC(paranoid_entry)
/*
* "Paranoid" exit path from exception stack. This is invoked
@@ -1173,14 +1172,12 @@ ENTRY(paranoid_exit)
END(paranoid_exit)
/*
- * Save all registers in pt_regs, and switch gs if needed.
+ * Switch gs if needed.
* Return: EBX=0: came from user mode; EBX=1: otherwise
*/
ENTRY(error_entry)
- UNWIND_HINT_FUNC
+ UNWIND_HINT_REGS offset=8
cld
- SAVE_AND_CLEAR_REGS 8
- ENCODE_FRAME_POINTER 8
testb $3, CS+8(%rsp)
jz .Lerror_kernelspace
@@ -1571,7 +1568,8 @@ end_repeat_nmi:
* frame to point back to repeat_nmi.
*/
pushq $-1 /* ORIG_RAX: no syscall to restart */
- ALLOC_PT_GPREGS_ON_STACK
+ PUSH_AND_CLEAR_REGS
+ ENCODE_FRAME_POINTER
/*
* Use paranoid_entry to handle SWAPGS, but no need to use paranoid_exit
^ permalink raw reply related [flat|nested] 33+ messages in thread
* [tip:x86/pti] x86/entry/64: Indent PUSH_AND_CLEAR_REGS and POP_REGS properly
2018-02-11 10:49 ` [PATCH v3 7/7] x86/entry: indent PUSH_AND_CLEAR_REGS and POP_REGS properly Dominik Brodowski
2018-02-12 10:18 ` [tip:x86/pti] x86/entry/64: Indent " tip-bot for Dominik Brodowski
@ 2018-02-13 9:03 ` tip-bot for Dominik Brodowski
1 sibling, 0 replies; 33+ messages in thread
From: tip-bot for Dominik Brodowski @ 2018-02-13 9:03 UTC (permalink / raw)
To: linux-tip-commits
Cc: hpa, bp, tglx, torvalds, brgerst, jpoimboe, luto, linux,
linux-kernel, dvlasenk, peterz, mingo
Commit-ID: 92816f571af81e9a71cc6f3dc8ce1e2fcdf7b6b8
Gitweb: https://git.kernel.org/tip/92816f571af81e9a71cc6f3dc8ce1e2fcdf7b6b8
Author: Dominik Brodowski <linux@dominikbrodowski.net>
AuthorDate: Sun, 11 Feb 2018 11:49:48 +0100
Committer: Ingo Molnar <mingo@kernel.org>
CommitDate: Tue, 13 Feb 2018 09:04:54 +0100
x86/entry/64: Indent PUSH_AND_CLEAR_REGS and POP_REGS properly
... same as the other macros in arch/x86/entry/calling.h
Signed-off-by: Dominik Brodowski <linux@dominikbrodowski.net>
Cc: Andy Lutomirski <luto@kernel.org>
Cc: Borislav Petkov <bp@alien8.de>
Cc: Brian Gerst <brgerst@gmail.com>
Cc: Denys Vlasenko <dvlasenk@redhat.com>
Cc: H. Peter Anvin <hpa@zytor.com>
Cc: Josh Poimboeuf <jpoimboe@redhat.com>
Cc: Linus Torvalds <torvalds@linux-foundation.org>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Thomas Gleixner <tglx@linutronix.de>
Cc: dan.j.williams@intel.com
Link: http://lkml.kernel.org/r/20180211104949.12992-8-linux@dominikbrodowski.net
Signed-off-by: Ingo Molnar <mingo@kernel.org>
---
arch/x86/entry/calling.h | 8 ++++----
1 file changed, 4 insertions(+), 4 deletions(-)
diff --git a/arch/x86/entry/calling.h b/arch/x86/entry/calling.h
index 5967501..6985440 100644
--- a/arch/x86/entry/calling.h
+++ b/arch/x86/entry/calling.h
@@ -97,7 +97,7 @@ For 32-bit we have the following conventions - kernel is built with
#define SIZEOF_PTREGS 21*8
- .macro PUSH_AND_CLEAR_REGS rdx=%rdx rax=%rax
+.macro PUSH_AND_CLEAR_REGS rdx=%rdx rax=%rax
/*
* Push registers and sanitize registers of values that a
* speculation attack might otherwise want to exploit. The
@@ -131,9 +131,9 @@ For 32-bit we have the following conventions - kernel is built with
pushq %r15 /* pt_regs->r15 */
xorq %r15, %r15 /* nospec r15*/
UNWIND_HINT_REGS
- .endm
+.endm
- .macro POP_REGS pop_rdi=1 skip_r11rcx=0
+.macro POP_REGS pop_rdi=1 skip_r11rcx=0
popq %r15
popq %r14
popq %r13
@@ -163,7 +163,7 @@ For 32-bit we have the following conventions - kernel is built with
.macro icebp
.byte 0xf1
- .endm
+.endm
/*
* This is a sneaky trick to help the unwinder find pt_regs on the stack. The
^ permalink raw reply related [flat|nested] 33+ messages in thread
* [tip:x86/pti] x86/entry/64: Remove the unused 'icebp' macro
2018-02-12 20:13 ` [PATCH] x86/entry/64: Remove unused icebp macro Borislav Petkov
2018-02-13 7:36 ` Ingo Molnar
@ 2018-02-13 9:04 ` tip-bot for Borislav Petkov
1 sibling, 0 replies; 33+ messages in thread
From: tip-bot for Borislav Petkov @ 2018-02-13 9:04 UTC (permalink / raw)
To: linux-tip-commits
Cc: bp, jpoimboe, bp, tglx, torvalds, linux-kernel, luto, dvlasenk,
peterz, hpa, brgerst, mingo
Commit-ID: b498c261107461d5c42140dfddd05df83d8ca078
Gitweb: https://git.kernel.org/tip/b498c261107461d5c42140dfddd05df83d8ca078
Author: Borislav Petkov <bp@suse.de>
AuthorDate: Mon, 12 Feb 2018 21:13:18 +0100
Committer: Ingo Molnar <mingo@kernel.org>
CommitDate: Tue, 13 Feb 2018 09:04:55 +0100
x86/entry/64: Remove the unused 'icebp' macro
That macro was touched around 2.5.8 times, judging by the full history
linux repo, but it was unused even then. Get rid of it already.
Signed-off-by: Borislav Petkov <bp@suse.de>
Cc: Andy Lutomirski <luto@kernel.org>
Cc: Borislav Petkov <bp@alien8.de>
Cc: Brian Gerst <brgerst@gmail.com>
Cc: Denys Vlasenko <dvlasenk@redhat.com>
Cc: H. Peter Anvin <hpa@zytor.com>
Cc: Josh Poimboeuf <jpoimboe@redhat.com>
Cc: Linus Torvalds <torvalds@linux-foundation.org>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Thomas Gleixner <tglx@linutronix.de>
Cc: linux@dominikbrodowski.net
Link: http://lkml.kernel.org/r/20180212201318.GD14640@pd.tnic
Signed-off-by: Ingo Molnar <mingo@kernel.org>
---
arch/x86/entry/calling.h | 4 ----
1 file changed, 4 deletions(-)
diff --git a/arch/x86/entry/calling.h b/arch/x86/entry/calling.h
index 6985440..dce7092 100644
--- a/arch/x86/entry/calling.h
+++ b/arch/x86/entry/calling.h
@@ -159,10 +159,6 @@ For 32-bit we have the following conventions - kernel is built with
.if \pop_rdi
popq %rdi
.endif
- .endm
-
- .macro icebp
- .byte 0xf1
.endm
/*
^ permalink raw reply related [flat|nested] 33+ messages in thread
* Re: [PATCH v3 8/7] TESTING_ONLY x86/entry: reduce static footprint of idtentry
2018-02-12 19:17 ` Linus Torvalds
@ 2018-02-13 12:41 ` Ingo Molnar
0 siblings, 0 replies; 33+ messages in thread
From: Ingo Molnar @ 2018-02-13 12:41 UTC (permalink / raw)
To: Linus Torvalds
Cc: Dominik Brodowski, Linux Kernel Mailing List,
the arch/x86 maintainers, Dan Williams, Thomas Gleixner,
Andi Kleen, Andrew Lutomirski, Peter Zijlstra, Denys Vlasenko,
Josh Poimboeuf, Brian Gerst, Borislav Petkov
* Linus Torvalds <torvalds@linux-foundation.org> wrote:
> On Mon, Feb 12, 2018 at 1:37 AM, Ingo Molnar <mingo@kernel.org> wrote:
> >
> > Ok, so this does not look _that_ complicated, and the .text savings are
> > significant:
>
> Honestly, I think we should do it. 3kB of assembly code is noticeable.
>
> Also, that patch actually allows more cleanups and simplifications.
> Look at the "interrupt" macro, which is used by 'apicinterrupt3', and
> has a number of uses that way.
>
> That code could be unified a lot, right now it does:
>
> testb $3, CS-ORIG_RAX(%rsp)
> jz 1f
> SWAPGS
> call switch_to_thread_stack
> 1:
>
> ALLOC_PT_GPREGS_ON_STACK
> SAVE_C_REGS
> SAVE_EXTRA_REGS
> ENCODE_FRAME_POINTER
>
> testb $3, CS(%rsp)
> jz 1f
>
> /*
> * IRQ from user mode.
> *
> * We need to tell lockdep that IRQs are off. We can't do this until
> * we fix gsbase, and we should do it before enter_from_user_mode
> * (which can take locks). Since TRACE_IRQS_OFF idempotent,
> * the simplest way to handle it is to just call it twice if
> * we enter from user mode. There's no reason to optimize this since
> * TRACE_IRQS_OFF is a no-op if lockdep is off.
> */
> TRACE_IRQS_OFF
>
> CALL_enter_from_user_mode
>
> 1:
> ENTER_IRQ_STACK old_rsp=%rdi
> /* We entered an interrupt context - irqs are off: */
> TRACE_IRQS_OFF
>
> and *all* of that could be in a helper function rather than be
> duplicated. and the apicinterrupt3 macro should end up just expanding
> to
>
> callq helper
> pushq $~(\num)
> callq \fn
> jmp ret_from_intr
>
> instead of expanding to all that code.
>
> But that would require that same "save_ret" logic.
>
> So it's not just the idtentry cases that can use this trick.
>
> I admit that the trick isn't pretty, but it's not *horribly* ugly either.
Ok, agreed!
Dominik, could you please do this on top latest tip:master? I have applied all the
other patches. Note that patch #8 now conflicts with recent annotation fixes.
Thanks,
Ingo
^ permalink raw reply [flat|nested] 33+ messages in thread
end of thread, other threads:[~2018-02-13 12:41 UTC | newest]
Thread overview: 33+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2018-02-11 10:49 [PATCH v3 0/7] x86/entry: simplify and unify SAVE/POP_REGS Dominik Brodowski
2018-02-11 10:49 ` [PATCH v3 1/7] x86/entry: merge SAVE_C_REGS and SAVE_EXTRA_REGS, remove unused extensions Dominik Brodowski
2018-02-12 10:15 ` [tip:x86/pti] x86/entry/64: Merge " tip-bot for Dominik Brodowski
2018-02-13 9:00 ` tip-bot for Dominik Brodowski
2018-02-11 10:49 ` [PATCH v3 2/7] x86/entry: merge POP_C_REGS and POP_EXTRA_REGS Dominik Brodowski
2018-02-12 10:16 ` [tip:x86/pti] x86/entry/64: Merge the POP_C_REGS and POP_EXTRA_REGS macros into a single POP_REGS macro tip-bot for Dominik Brodowski
2018-02-13 9:01 ` tip-bot for Dominik Brodowski
2018-02-11 10:49 ` [PATCH v3 3/7] x86/entry: interleave XOR register clearing with PUSH instructions Dominik Brodowski
2018-02-12 10:16 ` [tip:x86/pti] x86/entry/64: Interleave " tip-bot for Dominik Brodowski
2018-02-13 9:01 ` tip-bot for Dominik Brodowski
2018-02-11 10:49 ` [PATCH v3 4/7] x86/entry: introduce PUSH_AND_CLEAN_REGS Dominik Brodowski
2018-02-12 10:17 ` [tip:x86/pti] x86/entry/64: Introduce the PUSH_AND_CLEAN_REGS macro tip-bot for Dominik Brodowski
2018-02-12 13:29 ` Denys Vlasenko
2018-02-12 13:36 ` David Laight
2018-02-12 13:43 ` Denys Vlasenko
2018-02-12 16:51 ` Linus Torvalds
2018-02-13 9:01 ` tip-bot for Dominik Brodowski
2018-02-11 10:49 ` [PATCH v3 5/7] x86/entry: use PUSH_AND_CLEAN_REGS in more cases Dominik Brodowski
2018-02-12 10:17 ` [tip:x86/pti] x86/entry/64: Use " tip-bot for Dominik Brodowski
2018-02-13 9:02 ` tip-bot for Dominik Brodowski
2018-02-11 10:49 ` [PATCH v3 6/7] x86/entry: get rid of ALLOC_PT_GPREGS_ON_STACK and SAVE_AND_CLEAR_REGS Dominik Brodowski
2018-02-12 10:18 ` [tip:x86/pti] x86/entry/64: Get rid of the ALLOC_PT_GPREGS_ON_STACK and SAVE_AND_CLEAR_REGS macros tip-bot for Dominik Brodowski
2018-02-13 9:02 ` tip-bot for Dominik Brodowski
2018-02-11 10:49 ` [PATCH v3 7/7] x86/entry: indent PUSH_AND_CLEAR_REGS and POP_REGS properly Dominik Brodowski
2018-02-12 10:18 ` [tip:x86/pti] x86/entry/64: Indent " tip-bot for Dominik Brodowski
2018-02-12 20:13 ` [PATCH] x86/entry/64: Remove unused icebp macro Borislav Petkov
2018-02-13 7:36 ` Ingo Molnar
2018-02-13 9:04 ` [tip:x86/pti] x86/entry/64: Remove the unused 'icebp' macro tip-bot for Borislav Petkov
2018-02-13 9:03 ` [tip:x86/pti] x86/entry/64: Indent PUSH_AND_CLEAR_REGS and POP_REGS properly tip-bot for Dominik Brodowski
2018-02-11 10:49 ` [PATCH v3 8/7] TESTING_ONLY x86/entry: reduce static footprint of idtentry Dominik Brodowski
2018-02-12 9:37 ` Ingo Molnar
2018-02-12 19:17 ` Linus Torvalds
2018-02-13 12:41 ` Ingo Molnar
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.