From: Borislav Petkov <bp@alien8.de>
To: David Woodhouse <dwmw2@infradead.org>
Cc: Josh Poimboeuf <jpoimboe@redhat.com>,
tim.c.chen@linux.intel.com, pjt@google.com, jikos@kernel.org,
gregkh@linux-foundation.org, dave.hansen@intel.com,
mingo@kernel.org, riel@redhat.com, luto@amacapital.net,
torvalds@linux-foundation.org, ak@linux.intel.com,
keescook@google.com, peterz@infradead.org, tglx@linutronix.de,
hpa@zytor.com, linux-kernel@vger.kernel.org,
linux-tip-commits@vger.kernel.org
Subject: Re: [tip:x86/pti] x86/retpoline: Fill return stack buffer on vmexit
Date: Thu, 25 Jan 2018 20:07:29 +0100 [thread overview]
Message-ID: <20180125190729.jxey2k44sptuk42t@pd.tnic> (raw)
In-Reply-To: <1516903463.30244.148.camel@infradead.org>
On Thu, Jan 25, 2018 at 06:04:23PM +0000, David Woodhouse wrote:
> Yep, I'll buy that. But first we need Josh to work out what he's having
> for lunch.
>
> Although just another marker to tell objtool "ignore this whole
> function" might be sufficient to allow us to have an out-of-line RSB-
> stuffing function.
Ok, I think we solved it all on IRC.
I'm adding a conglomerate patch at the end. It builds and boots here in
a vm. Here's what I did:
The barrier:
static inline void indirect_branch_prediction_barrier(void)
{
alternative_input("", "call __ibp_barrier", X86_FEATURE_IBPB, ASM_NO_INPUT_CLOBBER("eax", "ecx", "edx", "memory"));
}
with
void __ibp_barrier(void)
{
wrmsr(MSR_IA32_PRED_CMD, PRED_CMD_IBPB, 0);
}
Alternatives applies correctly:
[ 1.040965] apply_alternatives: feat: 7*32+16, old: (ffffffff8259d806 len: 5), repl: (ffffffff826aa88b, len: 5), pad: 5
[ 1.042798] ffffffff8259d806: old_insn: 90 90 90 90 90
[ 1.044001] ffffffff826aa88b: rpl_insn: e8 80 c3 9c fe
[ 1.044938] apply_alternatives: Fix CALL offset: 0xfead9405, CALL 0xffffffff81076c10
[ 1.046343] ffffffff8259d806: final_insn: e8 05 94 ad fe
with final_insn becoming:
ffffffff8259d806: e8 80 c3 9c fe callq ffffffff81076c10 <__ibp_barrier>
When we look at the __ibp_barrier:
ffffffff81076c10 <__ibp_barrier>:
ffffffff81076c10: e8 bb ac 98 00 callq ffffffff81a018d0 <__fentry__>
ffffffff81076c15: b9 49 00 00 00 mov $0x49,%ecx
ffffffff81076c1a: b8 01 00 00 00 mov $0x1,%eax
ffffffff81076c1f: 31 d2 xor %edx,%edx
ffffffff81076c21: 0f 30 wrmsr
ffffffff81076c23: 0f 1f 44 00 00 nopl 0x0(%rax,%rax,1)
ffffffff81076c28: c3 retq
ffffffff81076c29: 31 d2 xor %edx,%edx
ffffffff81076c2b: be 01 00 00 00 mov $0x1,%esi
ffffffff81076c30: bf 49 00 00 00 mov $0x49,%edi
ffffffff81076c35: e9 76 64 3f 00 jmpq ffffffff8146d0b0 <do_trace_write_msr>
ffffffff81076c3a: 90 nop
ffffffff81076c3b: 90 nop
ffffffff81076c3c: 90 nop
ffffffff81076c3d: 90 nop
ffffffff81076c3e: 90 nop
It has the tracing shit too. We could replace that with __wrmsr() which
does exception handling. And I think we should do that instead.
/me does it.
The RSB filler became:
static inline void vmexit_fill_RSB(void)
{
#ifdef CONFIG_RETPOLINE
alternative_input("", "call __fill_rsb_clobber_ax", X86_FEATURE_RETPOLINE, ASM_NO_INPUT_CLOBBER("memory"));
#endif
}
and __fill_rsb_clobber_ax looks like:
ENTRY(__fill_rsb_clobber_ax)
___FILL_RETURN_BUFFER %_ASM_AX, RSB_CLEAR_LOOPS, %_ASM_SP
END(__fill_rsb_clobber_ax)
where:
.macro ___FILL_RETURN_BUFFER reg:req nr:req sp:req
mov (\nr / 2), \reg
.align 16
771:
call 772f
773: /* speculation trap */
pause
lfence
jmp 773b
.align 16
772:
call 774f
775: /* speculation trap */
pause
lfence
jmp 775b
.align 16
774:
dec \reg
jnz 771b
add (BITS_PER_LONG /8) * \nr, \sp
.endm
Please check the alignment directives are at the right places.
Application looks correct too:
[ 0.428974] apply_alternatives: feat: 7*32+12, old: (ffffffff8105fe50 len: 5), repl: (ffffffff826aa69f, len: 5), pad: 5
[ 0.432003] ffffffff8105fe50: old_insn: 90 90 90 90 90
[ 0.432977] ffffffff826aa69f: rpl_insn: e8 5c 8b 55 ff
[ 0.433943] apply_alternatives: Fix CALL offset: 0xba33ab, CALL 0xffffffff81c03200
[ 0.435368] ffffffff8105fe50: final_insn: e8 ab 33 ba 00
final_insn becomes
ffffffff8105fe50: e8 5c 8b 55 ff callq ffffffff81c03200 <__fill_rsb_clobber_ax>
and the __fill_rsb_clobber_ax asm look good to me too:
ffffffff81c03200 <__fill_rsb_clobber_ax>:
ffffffff81c03200: 48 8b 04 25 10 00 00 mov 0x10,%rax
ffffffff81c03207: 00
ffffffff81c03208: 0f 1f 84 00 00 00 00 nopl 0x0(%rax,%rax,1)
ffffffff81c0320f: 00
ffffffff81c03210: e8 0b 00 00 00 callq ffffffff81c03220 <__fill_rsb_clobber_ax+0x20>
ffffffff81c03215: f3 90 pause
ffffffff81c03217: 0f ae e8 lfence
ffffffff81c0321a: eb f9 jmp ffffffff81c03215 <__fill_rsb_clobber_ax+0x15>
ffffffff81c0321c: 0f 1f 40 00 nopl 0x0(%rax)
ffffffff81c03220: e8 0b 00 00 00 callq ffffffff81c03230 <__fill_rsb_clobber_ax+0x30>
ffffffff81c03225: f3 90 pause
ffffffff81c03227: 0f ae e8 lfence
ffffffff81c0322a: eb f9 jmp ffffffff81c03225 <__fill_rsb_clobber_ax+0x25>
ffffffff81c0322c: 0f 1f 40 00 nopl 0x0(%rax)
ffffffff81c03230: 48 ff c8 dec %rax
ffffffff81c03233: 75 db jne ffffffff81c03210 <__fill_rsb_clobber_ax+0x10>
ffffffff81c03235: 48 03 24 25 00 01 00 add 0x100,%rsp
ffffffff81c0323c: 00
with the respective alignment.
Btw, I've forced
setup_force_cpu_cap(X86_FEATURE_IBPB);
so that I can see it applies only - nothing else.
Here's the full diff I have so far:
diff --git a/arch/x86/include/asm/cpufeatures.h b/arch/x86/include/asm/cpufeatures.h
index 67bbfaa1448b..a7c5602847b1 100644
--- a/arch/x86/include/asm/cpufeatures.h
+++ b/arch/x86/include/asm/cpufeatures.h
@@ -210,6 +210,7 @@
#define X86_FEATURE_AVX512_4VNNIW ( 7*32+16) /* AVX-512 Neural Network Instructions */
#define X86_FEATURE_AVX512_4FMAPS ( 7*32+17) /* AVX-512 Multiply Accumulation Single precision */
+#define X86_FEATURE_IBPB ( 7*32+16) /* Using Indirect Branch Prediction Barrier */
#define X86_FEATURE_MBA ( 7*32+18) /* Memory Bandwidth Allocation */
#define X86_FEATURE_RSB_CTXSW ( 7*32+19) /* Fill RSB on context switches */
diff --git a/arch/x86/include/asm/msr-index.h b/arch/x86/include/asm/msr-index.h
index e7b983a35506..1c4e87b73789 100644
--- a/arch/x86/include/asm/msr-index.h
+++ b/arch/x86/include/asm/msr-index.h
@@ -39,6 +39,9 @@
/* Intel MSRs. Some also available on other CPUs */
+#define MSR_IA32_PRED_CMD 0x00000049 /* Prediction Command */
+#define PRED_CMD_IBPB (1 << 0) /* Indirect Branch Prediction Barrier */
+
#define MSR_PPIN_CTL 0x0000004e
#define MSR_PPIN 0x0000004f
diff --git a/arch/x86/include/asm/nospec-branch.h b/arch/x86/include/asm/nospec-branch.h
index 4ad41087ce0e..3e4d4996374e 100644
--- a/arch/x86/include/asm/nospec-branch.h
+++ b/arch/x86/include/asm/nospec-branch.h
@@ -1,7 +1,7 @@
/* SPDX-License-Identifier: GPL-2.0 */
-#ifndef __NOSPEC_BRANCH_H__
-#define __NOSPEC_BRANCH_H__
+#ifndef __X86_ASM_NOSPEC_BRANCH_H__
+#define __X86_ASM_NOSPEC_BRANCH_H__
#include <asm/alternative.h>
#include <asm/alternative-asm.h>
@@ -53,6 +53,9 @@
#ifdef __ASSEMBLY__
+#include <asm/bitsperlong.h>
+
+
/*
* This should be used immediately before a retpoline alternative. It tells
* objtool where the retpolines are so that it can make sense of the control
@@ -121,6 +124,29 @@
#endif
.endm
+.macro ___FILL_RETURN_BUFFER reg:req nr:req sp:req
+ mov (\nr / 2), \reg
+ .align 16
+771:
+ call 772f
+773: /* speculation trap */
+ pause
+ lfence
+ jmp 773b
+ .align 16
+772:
+ call 774f
+775: /* speculation trap */
+ pause
+ lfence
+ jmp 775b
+ .align 16
+774:
+ dec \reg
+ jnz 771b
+ add (BITS_PER_LONG /8) * \nr, \sp
+.endm
+
/*
* A simpler FILL_RETURN_BUFFER macro. Don't make people use the CPP
* monstrosity above, manually.
@@ -206,17 +232,14 @@ extern char __indirect_thunk_end[];
static inline void vmexit_fill_RSB(void)
{
#ifdef CONFIG_RETPOLINE
- unsigned long loops;
-
- asm volatile (ANNOTATE_NOSPEC_ALTERNATIVE
- ALTERNATIVE("jmp 910f",
- __stringify(__FILL_RETURN_BUFFER(%0, RSB_CLEAR_LOOPS, %1)),
- X86_FEATURE_RETPOLINE)
- "910:"
- : "=r" (loops), ASM_CALL_CONSTRAINT
- : : "memory" );
+ alternative_input("", "call __fill_rsb_clobber_ax", X86_FEATURE_RETPOLINE, ASM_NO_INPUT_CLOBBER("memory"));
#endif
}
+static inline void indirect_branch_prediction_barrier(void)
+{
+ alternative_input("", "call __ibp_barrier", X86_FEATURE_IBPB, ASM_NO_INPUT_CLOBBER("eax", "ecx", "edx", "memory"));
+}
+
#endif /* __ASSEMBLY__ */
-#endif /* __NOSPEC_BRANCH_H__ */
+#endif /* __X86_ASM_NOSPEC_BRANCH_H__ */
diff --git a/arch/x86/include/asm/processor.h b/arch/x86/include/asm/processor.h
index d3a67fba200a..624bc37dc696 100644
--- a/arch/x86/include/asm/processor.h
+++ b/arch/x86/include/asm/processor.h
@@ -971,4 +971,9 @@ bool xen_set_default_idle(void);
void stop_this_cpu(void *dummy);
void df_debug(struct pt_regs *regs, long error_code);
+
+#ifdef CONFIG_RETPOLINE
+asmlinkage void __fill_rsb_clobber_ax(void);
+void __ibp_barrier(void);
+#endif
#endif /* _ASM_X86_PROCESSOR_H */
diff --git a/arch/x86/kernel/cpu/bugs.c b/arch/x86/kernel/cpu/bugs.c
index 390b3dc3d438..35f60e5b93bf 100644
--- a/arch/x86/kernel/cpu/bugs.c
+++ b/arch/x86/kernel/cpu/bugs.c
@@ -249,6 +249,10 @@ static void __init spectre_v2_select_mitigation(void)
setup_force_cpu_cap(X86_FEATURE_RSB_CTXSW);
pr_info("Filling RSB on context switch\n");
}
+
+ setup_force_cpu_cap(X86_FEATURE_IBPB);
+
+ indirect_branch_prediction_barrier();
}
#undef pr_fmt
@@ -281,3 +285,10 @@ ssize_t cpu_show_spectre_v2(struct device *dev,
return sprintf(buf, "%s\n", spectre_v2_strings[spectre_v2_enabled]);
}
#endif
+
+#ifdef CONFIG_RETPOLINE
+void __ibp_barrier(void)
+{
+ __wrmsr(MSR_IA32_PRED_CMD, PRED_CMD_IBPB, 0);
+}
+#endif
diff --git a/arch/x86/lib/Makefile b/arch/x86/lib/Makefile
index f23934bbaf4e..69a473919260 100644
--- a/arch/x86/lib/Makefile
+++ b/arch/x86/lib/Makefile
@@ -27,6 +27,7 @@ lib-$(CONFIG_RWSEM_XCHGADD_ALGORITHM) += rwsem.o
lib-$(CONFIG_INSTRUCTION_DECODER) += insn.o inat.o insn-eval.o
lib-$(CONFIG_RANDOMIZE_BASE) += kaslr.o
lib-$(CONFIG_RETPOLINE) += retpoline.o
+OBJECT_FILES_NON_STANDARD_retpoline.o :=y
obj-y += msr.o msr-reg.o msr-reg-export.o hweight.o
diff --git a/arch/x86/lib/retpoline.S b/arch/x86/lib/retpoline.S
index dfb2ba91b670..62eaa8f80d30 100644
--- a/arch/x86/lib/retpoline.S
+++ b/arch/x86/lib/retpoline.S
@@ -47,3 +47,7 @@ GENERATE_THUNK(r13)
GENERATE_THUNK(r14)
GENERATE_THUNK(r15)
#endif
+
+ENTRY(__fill_rsb_clobber_ax)
+ ___FILL_RETURN_BUFFER %_ASM_AX, RSB_CLEAR_LOOPS, %_ASM_SP
+END(__fill_rsb_clobber_ax)
--
2.13.0
--
Regards/Gruss,
Boris.
Good mailing practices for 400: avoid top-posting and trim the reply.
next prev parent reply other threads:[~2018-01-25 19:07 UTC|newest]
Thread overview: 85+ messages / expand[flat|nested] mbox.gz Atom feed top
2018-01-11 21:46 [PATCH v8 00/12] Retpoline: Avoid speculative indirect calls in kernel David Woodhouse
2018-01-11 21:46 ` [PATCH v8 01/12] objtool: Detect jumps to retpoline thunks David Woodhouse
2018-01-11 23:22 ` [tip:x86/pti] " tip-bot for Josh Poimboeuf
2018-01-11 21:46 ` [PATCH v8 02/12] objtool: Allow alternatives to be ignored David Woodhouse
2018-01-11 23:22 ` [tip:x86/pti] " tip-bot for Josh Poimboeuf
2018-01-18 19:09 ` [v8,02/12] " Guenter Roeck
2018-01-18 19:33 ` Josh Poimboeuf
2018-01-18 19:41 ` Guenter Roeck
2018-01-22 19:34 ` David Woodhouse
2018-01-22 20:25 ` Guenter Roeck
2018-01-22 20:27 ` David Woodhouse
2018-01-28 21:06 ` Josh Poimboeuf
2018-01-29 1:26 ` Guenter Roeck
2018-01-29 17:15 ` Guenter Roeck
2018-01-29 17:30 ` Josh Poimboeuf
2018-01-22 19:27 ` Guenter Roeck
2018-01-11 21:46 ` [PATCH v8 03/12] x86/retpoline: Add initial retpoline support David Woodhouse
2018-01-11 23:23 ` [tip:x86/pti] " tip-bot for David Woodhouse
2018-01-11 23:58 ` [PATCH v8 03/12] " Tom Lendacky
2018-01-12 10:28 ` David Woodhouse
2018-01-12 14:02 ` Tom Lendacky
2018-01-14 15:02 ` Borislav Petkov
2018-01-14 15:53 ` Josh Poimboeuf
2018-01-14 15:59 ` Borislav Petkov
2018-01-11 21:46 ` [PATCH v8 04/12] x86/spectre: Add boot time option to select Spectre v2 mitigation David Woodhouse
2018-01-11 23:23 ` [tip:x86/pti] " tip-bot for David Woodhouse
2018-01-23 22:40 ` [PATCH v8 04/12] " Borislav Petkov
2018-01-23 22:53 ` David Woodhouse
2018-01-23 23:05 ` Andi Kleen
2018-01-23 22:55 ` Jiri Kosina
2018-01-23 23:05 ` Borislav Petkov
2018-01-24 0:32 ` Kees Cook
2018-01-24 9:58 ` Borislav Petkov
2018-01-23 23:06 ` Jiri Kosina
2018-01-23 23:21 ` Andi Kleen
2018-01-23 23:24 ` Jiri Kosina
2018-01-23 23:45 ` Andi Kleen
2018-01-23 23:49 ` Jiri Kosina
2018-01-24 4:26 ` Greg Kroah-Hartman
2018-01-24 9:56 ` Jiri Kosina
2018-01-24 13:58 ` Greg Kroah-Hartman
2018-01-24 14:03 ` Jiri Kosina
2018-01-24 14:22 ` Greg Kroah-Hartman
2018-01-11 21:46 ` [PATCH v8 05/12] x86/retpoline/crypto: Convert crypto assembler indirect jumps David Woodhouse
2018-01-11 23:24 ` [tip:x86/pti] " tip-bot for David Woodhouse
2018-01-11 21:46 ` [PATCH v8 06/12] x86/retpoline/entry: Convert entry " David Woodhouse
2018-01-11 23:24 ` [tip:x86/pti] " tip-bot for David Woodhouse
2018-01-11 21:46 ` [PATCH v8 07/12] x86/retpoline/ftrace: Convert ftrace " David Woodhouse
2018-01-11 23:25 ` [tip:x86/pti] " tip-bot for David Woodhouse
2018-01-11 21:46 ` [PATCH v8 08/12] x86/retpoline/hyperv: Convert " David Woodhouse
2018-01-11 23:25 ` [tip:x86/pti] " tip-bot for David Woodhouse
2018-01-11 21:46 ` [PATCH v8 09/12] x86/retpoline/xen: Convert Xen hypercall " David Woodhouse
2018-01-11 23:25 ` [tip:x86/pti] " tip-bot for David Woodhouse
2018-01-11 21:46 ` [PATCH v8 10/12] x86/retpoline/checksum32: Convert assembler " David Woodhouse
2018-01-11 23:26 ` [tip:x86/pti] " tip-bot for David Woodhouse
2018-01-11 21:46 ` [PATCH v8 11/12] x86/retpoline/irq32: " David Woodhouse
2018-01-11 23:26 ` [tip:x86/pti] " tip-bot for Andi Kleen
2018-01-11 21:46 ` [PATCH v8 12/12] x86/retpoline: Fill return stack buffer on vmexit David Woodhouse
2018-01-11 23:27 ` [tip:x86/pti] " tip-bot for David Woodhouse
2018-01-11 23:51 ` [PATCH v8 12/12] " Andi Kleen
2018-01-12 11:11 ` [PATCH v8.1 " David Woodhouse
2018-01-12 11:15 ` Thomas Gleixner
2018-01-12 11:21 ` Woodhouse, David
2018-01-12 11:37 ` [tip:x86/pti] " tip-bot for David Woodhouse
2018-01-14 14:50 ` Borislav Petkov
2018-01-14 15:28 ` Thomas Gleixner
2018-01-14 15:35 ` Borislav Petkov
2018-01-25 12:07 ` Borislav Petkov
2018-01-25 12:20 ` David Woodhouse
2018-01-25 12:45 ` Borislav Petkov
2018-01-25 15:10 ` Josh Poimboeuf
2018-01-25 15:51 ` Borislav Petkov
2018-01-25 16:03 ` David Woodhouse
2018-01-25 16:56 ` Josh Poimboeuf
2018-01-25 17:00 ` David Woodhouse
2018-01-25 17:05 ` Andy Lutomirski
2018-01-25 17:44 ` Josh Poimboeuf
2018-01-25 18:41 ` Jiri Kosina
2018-01-25 17:10 ` Thomas Gleixner
2018-01-25 17:32 ` Josh Poimboeuf
2018-01-25 17:53 ` Borislav Petkov
2018-01-25 18:04 ` David Woodhouse
2018-01-25 18:32 ` Josh Poimboeuf
2018-01-25 19:07 ` Borislav Petkov [this message]
2018-01-25 19:10 ` Borislav Petkov
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20180125190729.jxey2k44sptuk42t@pd.tnic \
--to=bp@alien8.de \
--cc=ak@linux.intel.com \
--cc=dave.hansen@intel.com \
--cc=dwmw2@infradead.org \
--cc=gregkh@linux-foundation.org \
--cc=hpa@zytor.com \
--cc=jikos@kernel.org \
--cc=jpoimboe@redhat.com \
--cc=keescook@google.com \
--cc=linux-kernel@vger.kernel.org \
--cc=linux-tip-commits@vger.kernel.org \
--cc=luto@amacapital.net \
--cc=mingo@kernel.org \
--cc=peterz@infradead.org \
--cc=pjt@google.com \
--cc=riel@redhat.com \
--cc=tglx@linutronix.de \
--cc=tim.c.chen@linux.intel.com \
--cc=torvalds@linux-foundation.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).