[PATCH v1 tip/master 0/3] kprobes/x86: retpoline: Fix kprobes for retpoline

[PATCH v1 tip/master 0/3] kprobes/x86: retpoline: Fix kprobes for retpoline

[Masami Hiramatsu]

Hi,

This is a series of patches to fix kprobes issues on the
kernel with CONFIG_RETPOLINE=y.

- [1/3]: This introduces __x86_indirect_thunk_* boundary
	 symbols so that kprobes easily identify those functions.
- [2/3]: Mark __x86_indirect_thunk_* as blacklisted function
	 for kprobes, since it can be called from other
	 blacklisted functions.
- [3/3]: Check jmp instructions in the probe target function
	 whether it jumps into the __x86_indirect_thunk_*,
	 because it is equal to an indirect jump instruction.

Side effect: [1/3] will move __x86_indirect_thunk_* functions
in kernel text area. Of course those functions were in the
.text area, but placed in right after _etext. This just moves
it right before the _etext.

Thank you,

---

Masami Hiramatsu (3):
      retpoline: Introduce start/end markers of indirect thunk
      kprobes/x86: Blacklist indirect thunk functions for kprobes
      kprobes/x86: Disable optimizing on the function jumps to indirect thunk


 arch/x86/include/asm/nospec-branch.h |    3 +++
 arch/x86/kernel/kprobes/opt.c        |   23 +++++++++++++++++++++-
 arch/x86/kernel/vmlinux.lds.S        |   35 ++++++++++++++++++++++++++++++++++
 arch/x86/lib/retpoline.S             |    3 ++-
 4 files changed, 62 insertions(+), 2 deletions(-)

--
Masami Hiramatsu (Linaro) <mhiramat@kernel.org>

[PATCH v1 tip/master 1/3] retpoline: Introduce start/end markers of indirect thunk

[Masami Hiramatsu]

Introduce start/end markers of __x86_indirect_thunk_* functions.
These thunk functions are placed in .text.__x86.indirect_thunk.*
sections. So this puts those sections in the end of kernel text
and adds __indirect_thunk_start/end so that other subsystem
(e.g. kprobes) can identify it.

Signed-off-by: Masami Hiramatsu <mhiramat@kernel.org>
---
 arch/x86/include/asm/nospec-branch.h |    3 +++
 arch/x86/kernel/vmlinux.lds.S        |   35 ++++++++++++++++++++++++++++++++++
 2 files changed, 38 insertions(+)

diff --git a/arch/x86/include/asm/nospec-branch.h b/arch/x86/include/asm/nospec-branch.h
index 7b45d8424150..19ba5ad19c65 100644
--- a/arch/x86/include/asm/nospec-branch.h
+++ b/arch/x86/include/asm/nospec-branch.h
@@ -194,6 +194,9 @@ enum spectre_v2_mitigation {
 	SPECTRE_V2_IBRS,
 };
 
+extern char __indirect_thunk_start[];
+extern char __indirect_thunk_end[];
+
 /*
  * On VMEXIT we must ensure that no RSB predictions learned in the guest
  * can be followed in the host, by overwriting the RSB completely. Both
diff --git a/arch/x86/kernel/vmlinux.lds.S b/arch/x86/kernel/vmlinux.lds.S
index 1e413a9326aa..1a5a663620ce 100644
--- a/arch/x86/kernel/vmlinux.lds.S
+++ b/arch/x86/kernel/vmlinux.lds.S
@@ -74,6 +74,9 @@ jiffies_64 = jiffies;
 
 #endif
 
+#define X86_INDIRECT_THUNK(reg)	\
+		*(.text.__x86.indirect_thunk.##reg)
+
 PHDRS {
 	text PT_LOAD FLAGS(5);          /* R_E */
 	data PT_LOAD FLAGS(6);          /* RW_ */
@@ -124,6 +127,38 @@ SECTIONS
 		ASSERT(. - _entry_trampoline == PAGE_SIZE, "entry trampoline is too big");
 #endif
 
+#ifdef CONFIG_RETPOLINE
+		__indirect_thunk_start = .;
+#ifdef CONFIG_X86_64
+		X86_INDIRECT_THUNK(rax)
+		X86_INDIRECT_THUNK(rbx)
+		X86_INDIRECT_THUNK(rcx)
+		X86_INDIRECT_THUNK(rdx)
+		X86_INDIRECT_THUNK(rsi)
+		X86_INDIRECT_THUNK(rdi)
+		X86_INDIRECT_THUNK(rbp)
+		X86_INDIRECT_THUNK(rsp)
+		X86_INDIRECT_THUNK(r8)
+		X86_INDIRECT_THUNK(r9)
+		X86_INDIRECT_THUNK(r10)
+		X86_INDIRECT_THUNK(r11)
+		X86_INDIRECT_THUNK(r12)
+		X86_INDIRECT_THUNK(r13)
+		X86_INDIRECT_THUNK(r14)
+		X86_INDIRECT_THUNK(r15)
+#else
+		X86_INDIRECT_THUNK(eax)
+		X86_INDIRECT_THUNK(ebx)
+		X86_INDIRECT_THUNK(ecx)
+		X86_INDIRECT_THUNK(edx)
+		X86_INDIRECT_THUNK(esi)
+		X86_INDIRECT_THUNK(edi)
+		X86_INDIRECT_THUNK(ebp)
+		X86_INDIRECT_THUNK(esp)
+#endif
+		__indirect_thunk_end = .;
+#endif
+
 		/* End of text section */
 		_etext = .;
 	} :text = 0x9090

[PATCH v1 tip/master 2/3] kprobes/x86: Blacklist indirect thunk functions for kprobes

[Masami Hiramatsu]

Mark __x86_indirect_thunk_* functions as blacklist for kprobes
because those functions can be called from anywhere in the kernel
including blacklist functions of kprobes.

Signed-off-by: Masami Hiramatsu <mhiramat@kernel.org>
---
 arch/x86/lib/retpoline.S |    3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/arch/x86/lib/retpoline.S b/arch/x86/lib/retpoline.S
index cb45c6cb465f..9d964f6795b8 100644
--- a/arch/x86/lib/retpoline.S
+++ b/arch/x86/lib/retpoline.S
@@ -25,7 +25,8 @@ ENDPROC(__x86_indirect_thunk_\reg)
  * than one per register with the correct names. So we do it
  * the simple and nasty way...
  */
-#define EXPORT_THUNK(reg) EXPORT_SYMBOL(__x86_indirect_thunk_ ## reg)
+#define __EXPORT_THUNK(sym) _ASM_NOKPROBE(sym); EXPORT_SYMBOL(sym)
+#define EXPORT_THUNK(reg) __EXPORT_THUNK(__x86_indirect_thunk_ ## reg)
 #define GENERATE_THUNK(reg) THUNK reg ; EXPORT_THUNK(reg)
 
 GENERATE_THUNK(_ASM_AX)

[PATCH v1 tip/master 3/3] kprobes/x86: Disable optimizing on the function jumps to indirect thunk

[Masami Hiramatsu]

Since indirect jump instructions will be replaced by jump
to __x86_indirect_thunk_*, those jmp instruction must be
treated as an indirect jump. Since optprobe prohibits to
optimize probes in the function which uses an indirect jump,
it also needs to find out the function which jump to
__x86_indirect_thunk_* and disable optimization.

This adds a check that the jump target address is between
the __indirect_thunk_start/end when optimizing kprobe.

Signed-off-by: Masami Hiramatsu <mhiramat@kernel.org>
---
 arch/x86/kernel/kprobes/opt.c |   23 ++++++++++++++++++++++-
 1 file changed, 22 insertions(+), 1 deletion(-)

diff --git a/arch/x86/kernel/kprobes/opt.c b/arch/x86/kernel/kprobes/opt.c
index e941136e24d8..203d398802a3 100644
--- a/arch/x86/kernel/kprobes/opt.c
+++ b/arch/x86/kernel/kprobes/opt.c
@@ -40,6 +40,7 @@
 #include <asm/debugreg.h>
 #include <asm/set_memory.h>
 #include <asm/sections.h>
+#include <asm/nospec-branch.h>
 
 #include "common.h"
 
@@ -203,7 +204,7 @@ static int copy_optimized_instructions(u8 *dest, u8 *src, u8 *real)
 }
 
 /* Check whether insn is indirect jump */
-static int insn_is_indirect_jump(struct insn *insn)
+static int __insn_is_indirect_jump(struct insn *insn)
 {
 	return ((insn->opcode.bytes[0] == 0xff &&
 		(X86_MODRM_REG(insn->modrm.value) & 6) == 4) || /* Jump */
@@ -237,6 +238,26 @@ static int insn_jump_into_range(struct insn *insn, unsigned long start, int len)
 	return (start <= target && target <= start + len);
 }
 
+static int insn_is_indirect_jump(struct insn *insn)
+{
+	int ret = __insn_is_indirect_jump(insn);
+
+#ifdef CONFIG_RETPOLINE
+	/*
+	 * Jump to x86_indirect_thunk_* is treated as an indirect jump.
+	 * Note that even with CONFIG_RETPOLINE=y, the kernel compiled with
+	 * older gcc may use indirect jump. So we add this check instead of
+	 * replace indirect-jump check.
+	 */
+	if (!ret)
+		ret = insn_jump_into_range(insn,
+				(unsigned long)__indirect_thunk_start,
+				(unsigned long)__indirect_thunk_end -
+				(unsigned long)__indirect_thunk_start);
+#endif
+	return ret;
+}
+
 /* Decode whole function to ensure any instructions don't jump into target */
 static int can_optimize(unsigned long paddr)
 {

Re: [PATCH v1 tip/master 1/3] retpoline: Introduce start/end markers of indirect thunk

[Woodhouse, David]

[-- Attachment #1.1: Type: text/plain, Size: 445 bytes --]

On Thu, 2018-01-18 at 21:01 +0900, Masami Hiramatsu wrote:
> 
> +#define X86_INDIRECT_THUNK(reg)        \
> +               *(.text.__x86.indirect_thunk.##reg)

Note that we don't actually care about those being in their own
section, named after the register. That was just a hangover from the
initial GCC COMDAT implementation.

If it makes your life easier just to put them all in a single section,
go ahead.

[-- Attachment #1.2: smime.p7s --]
[-- Type: application/x-pkcs7-signature, Size: 5210 bytes --]

[-- Attachment #2.1: Type: text/plain, Size: 197 bytes --]




Amazon Web Services UK Limited. Registered in England and Wales with registration number 08650665 and which has its registered office at 60 Holborn Viaduct, London EC1A 2FD, United Kingdom.

[-- Attachment #2.2: Type: text/html, Size: 197 bytes --]

Re: [PATCH v1 tip/master 0/3] kprobes/x86: retpoline: Fix kprobes for retpoline

[Andi Kleen]

> Side effect: [1/3] will move __x86_indirect_thunk_* functions
> in kernel text area. Of course those functions were in the
> .text area, but placed in right after _etext. This just moves
> it right before the _etext.

I assume you tested that with page table isolation on?

The thunks need to be accessible from the trampoline.

-Andi

Re: [PATCH v1 tip/master 0/3] kprobes/x86: retpoline: Fix kprobes for retpoline

[David Woodhouse]

[-- Attachment #1: Type: text/plain, Size: 457 bytes --]

On Thu, 2018-01-18 at 05:01 -0800, Andi Kleen wrote:
> > 
> > Side effect: [1/3] will move __x86_indirect_thunk_* functions
> > in kernel text area. Of course those functions were in the
> > .text area, but placed in right after _etext. This just moves
> > it right before the _etext.
> I assume you tested that with page table isolation on?
> 
> The thunks need to be accessible from the trampoline.

I thought we put it inline in the trampoline.

[-- Attachment #2: smime.p7s --]
[-- Type: application/x-pkcs7-signature, Size: 5213 bytes --]

Re: [PATCH v1 tip/master 1/3] retpoline: Introduce start/end markers of indirect thunk

[Masami Hiramatsu]

On Thu, 18 Jan 2018 12:06:10 +0000
"Woodhouse, David" <dwmw@amazon.co.uk> wrote:

> On Thu, 2018-01-18 at 21:01 +0900, Masami Hiramatsu wrote:
> > 
> > +#define X86_INDIRECT_THUNK(reg)        \
> > +               *(.text.__x86.indirect_thunk.##reg)
> 
> Note that we don't actually care about those being in their own
> section, named after the register. That was just a hangover from the
> initial GCC COMDAT implementation.
> 
> If it makes your life easier just to put them all in a single section,
> go ahead.

Got it. I just concerned that gcc was using this section. 
I'll make it ".text.__x86.indirect_thunk" to consolidate.

Thanks!

-- 
Masami Hiramatsu <mhiramat@kernel.org>

Re: [PATCH v1 tip/master 0/3] kprobes/x86: retpoline: Fix kprobes for retpoline

[Masami Hiramatsu]

On Thu, 18 Jan 2018 05:01:57 -0800
Andi Kleen <ak@linux.intel.com> wrote:

> > Side effect: [1/3] will move __x86_indirect_thunk_* functions
> > in kernel text area. Of course those functions were in the
> > .text area, but placed in right after _etext. This just moves
> > it right before the _etext.
> 
> I assume you tested that with page table isolation on?
> 
> The thunks need to be accessible from the trampoline.

Yes, I've tested the kernel with CONFIG_PAGE_TABLE_ISOLATION=y.
As David pointed, maybe all those points are using JMP/CALL_NOSPEC macro directly?

Thanks,

> 
> -Andi


-- 
Masami Hiramatsu <mhiramat@kernel.org>

Re: [PATCH v1 tip/master 0/3] kprobes/x86: retpoline: Fix kprobes for retpoline

[Andi Kleen]

On Thu, Jan 18, 2018 at 02:03:07PM +0100, David Woodhouse wrote:
> On Thu, 2018-01-18 at 05:01 -0800, Andi Kleen wrote:
> > > 
> > > Side effect: [1/3] will move __x86_indirect_thunk_* functions
> > > in kernel text area. Of course those functions were in the
> > > .text area, but placed in right after _etext. This just moves
> > > it right before the _etext.
> > I assume you tested that with page table isolation on?
> > 
> > The thunks need to be accessible from the trampoline.
> 
> I thought we put it inline in the trampoline.

Yes we did, just want to make sure everything still works.

If it's tested that's fine

-Andi