From: Andy Lutomirski <luto@kernel.org> To: x86@kernel.org, LKML <linux-kernel@vger.kernel.org> Cc: Linus Torvalds <torvalds@linux-foundation.org>, Greg Kroah-Hartman <gregkh@linuxfoundation.org>, Alan Cox <alan@linux.intel.com>, Jann Horn <jannh@google.com>, Samuel Neves <samuel.c.p.neves@gmail.com>, Dan Williams <dan.j.williams@intel.com>, Kernel Hardening <kernel-hardening@lists.openwall.com>, Borislav Petkov <bp@alien8.de>, Andy Lutomirski <luto@kernel.org> Subject: [PATCH] x86/retpoline/entry: Disable the entire SYSCALL64 fast path with retpolines on Date: Mon, 22 Jan 2018 10:04:29 -0800 [thread overview] Message-ID: <503224b776b9513885453756e44bab235221124e.1516644136.git.luto@kernel.org> (raw) The existing retpoline code carefully and awkwardly retpolinifies the SYSCALL64 slow path. This stops the fast path from being particularly fast, and it's IMO rather messy. Instead, just bypass the fast path entirely if retpolines are on. This seems to be a speedup on a "minimal" retpoline kernel, mainly because do_syscall_64() ends up calling the syscall handler without using a slow retpoline thunk. As an added benefit, we won't need to apply further Spectre mitigations to the fast path. The current fast path spectre mitigations may have a hole: if the syscall nr is out of bounds, it is plausible that the CPU would mispredict the bounds check and, load a bogus function pointer, and speculatively execute it right though the retpoline. If this is indeed a problem, we need to fix it in the slow paths anyway, but with this patch applied, we can at least leave the fast path alone. Cleans-up: 2641f08bb7fc ("x86/retpoline/entry: Convert entry assembler indirect jumps") Signed-off-by: Andy Lutomirski <luto@kernel.org> --- arch/x86/entry/entry_64.S | 23 +++++++++++++++++------ 1 file changed, 17 insertions(+), 6 deletions(-) diff --git a/arch/x86/entry/entry_64.S b/arch/x86/entry/entry_64.S index 4f8e1d35a97c..b915bad58754 100644 --- a/arch/x86/entry/entry_64.S +++ b/arch/x86/entry/entry_64.S @@ -245,6 +245,9 @@ GLOBAL(entry_SYSCALL_64_after_hwframe) * If we need to do entry work or if we guess we'll need to do * exit work, go straight to the slow path. */ +#ifdef CONFIG_RETPOLINE + ALTERNATIVE "", "jmp entry_SYSCALL64_slow_path", X86_FEATURE_RETPOLINE +#endif movq PER_CPU_VAR(current_task), %r11 testl $_TIF_WORK_SYSCALL_ENTRY|_TIF_ALLWORK_MASK, TASK_TI_flags(%r11) jnz entry_SYSCALL64_slow_path @@ -270,13 +273,11 @@ entry_SYSCALL_64_fastpath: * This call instruction is handled specially in stub_ptregs_64. * It might end up jumping to the slow path. If it jumps, RAX * and all argument registers are clobbered. + * + * NB: no retpoline needed -- we don't execute this code with + * retpolines enabled. */ -#ifdef CONFIG_RETPOLINE - movq sys_call_table(, %rax, 8), %rax - call __x86_indirect_thunk_rax -#else call *sys_call_table(, %rax, 8) -#endif .Lentry_SYSCALL_64_after_fastpath_call: movq %rax, RAX(%rsp) @@ -431,6 +432,9 @@ ENTRY(stub_ptregs_64) * which we achieve by trying again on the slow path. If we are on * the slow path, the extra regs are already saved. * + * This code is unreachable (even via mispredicted conditional branches) + * if we're using retpolines. + * * RAX stores a pointer to the C function implementing the syscall. * IRQs are on. */ @@ -448,12 +452,19 @@ ENTRY(stub_ptregs_64) jmp entry_SYSCALL64_slow_path 1: - JMP_NOSPEC %rax /* Called from C */ + jmp *%rax /* Called from C */ END(stub_ptregs_64) .macro ptregs_stub func ENTRY(ptregs_\func) UNWIND_HINT_FUNC +#ifdef CONFIG_RETPOLINE + /* + * If retpolines are enabled, we don't use the syscall fast path, + * so just jump straight to the syscall body. + */ + ALTERNATIVE "", __stringify(jmp \func), X86_FEATURE_RETPOLINE +#endif leaq \func(%rip), %rax jmp stub_ptregs_64 END(ptregs_\func) -- 2.13.6
WARNING: multiple messages have this Message-ID (diff)
From: Andy Lutomirski <luto@kernel.org> To: x86@kernel.org, LKML <linux-kernel@vger.kernel.org> Cc: Linus Torvalds <torvalds@linux-foundation.org>, Greg Kroah-Hartman <gregkh@linuxfoundation.org>, Alan Cox <alan@linux.intel.com>, Jann Horn <jannh@google.com>, Samuel Neves <samuel.c.p.neves@gmail.com>, Dan Williams <dan.j.williams@intel.com>, Kernel Hardening <kernel-hardening@lists.openwall.com>, Borislav Petkov <bp@alien8.de>, Andy Lutomirski <luto@kernel.org> Subject: [kernel-hardening] [PATCH] x86/retpoline/entry: Disable the entire SYSCALL64 fast path with retpolines on Date: Mon, 22 Jan 2018 10:04:29 -0800 [thread overview] Message-ID: <503224b776b9513885453756e44bab235221124e.1516644136.git.luto@kernel.org> (raw) The existing retpoline code carefully and awkwardly retpolinifies the SYSCALL64 slow path. This stops the fast path from being particularly fast, and it's IMO rather messy. Instead, just bypass the fast path entirely if retpolines are on. This seems to be a speedup on a "minimal" retpoline kernel, mainly because do_syscall_64() ends up calling the syscall handler without using a slow retpoline thunk. As an added benefit, we won't need to apply further Spectre mitigations to the fast path. The current fast path spectre mitigations may have a hole: if the syscall nr is out of bounds, it is plausible that the CPU would mispredict the bounds check and, load a bogus function pointer, and speculatively execute it right though the retpoline. If this is indeed a problem, we need to fix it in the slow paths anyway, but with this patch applied, we can at least leave the fast path alone. Cleans-up: 2641f08bb7fc ("x86/retpoline/entry: Convert entry assembler indirect jumps") Signed-off-by: Andy Lutomirski <luto@kernel.org> --- arch/x86/entry/entry_64.S | 23 +++++++++++++++++------ 1 file changed, 17 insertions(+), 6 deletions(-) diff --git a/arch/x86/entry/entry_64.S b/arch/x86/entry/entry_64.S index 4f8e1d35a97c..b915bad58754 100644 --- a/arch/x86/entry/entry_64.S +++ b/arch/x86/entry/entry_64.S @@ -245,6 +245,9 @@ GLOBAL(entry_SYSCALL_64_after_hwframe) * If we need to do entry work or if we guess we'll need to do * exit work, go straight to the slow path. */ +#ifdef CONFIG_RETPOLINE + ALTERNATIVE "", "jmp entry_SYSCALL64_slow_path", X86_FEATURE_RETPOLINE +#endif movq PER_CPU_VAR(current_task), %r11 testl $_TIF_WORK_SYSCALL_ENTRY|_TIF_ALLWORK_MASK, TASK_TI_flags(%r11) jnz entry_SYSCALL64_slow_path @@ -270,13 +273,11 @@ entry_SYSCALL_64_fastpath: * This call instruction is handled specially in stub_ptregs_64. * It might end up jumping to the slow path. If it jumps, RAX * and all argument registers are clobbered. + * + * NB: no retpoline needed -- we don't execute this code with + * retpolines enabled. */ -#ifdef CONFIG_RETPOLINE - movq sys_call_table(, %rax, 8), %rax - call __x86_indirect_thunk_rax -#else call *sys_call_table(, %rax, 8) -#endif .Lentry_SYSCALL_64_after_fastpath_call: movq %rax, RAX(%rsp) @@ -431,6 +432,9 @@ ENTRY(stub_ptregs_64) * which we achieve by trying again on the slow path. If we are on * the slow path, the extra regs are already saved. * + * This code is unreachable (even via mispredicted conditional branches) + * if we're using retpolines. + * * RAX stores a pointer to the C function implementing the syscall. * IRQs are on. */ @@ -448,12 +452,19 @@ ENTRY(stub_ptregs_64) jmp entry_SYSCALL64_slow_path 1: - JMP_NOSPEC %rax /* Called from C */ + jmp *%rax /* Called from C */ END(stub_ptregs_64) .macro ptregs_stub func ENTRY(ptregs_\func) UNWIND_HINT_FUNC +#ifdef CONFIG_RETPOLINE + /* + * If retpolines are enabled, we don't use the syscall fast path, + * so just jump straight to the syscall body. + */ + ALTERNATIVE "", __stringify(jmp \func), X86_FEATURE_RETPOLINE +#endif leaq \func(%rip), %rax jmp stub_ptregs_64 END(ptregs_\func) -- 2.13.6
next reply other threads:[~2018-01-22 18:04 UTC|newest] Thread overview: 55+ messages / expand[flat|nested] mbox.gz Atom feed top 2018-01-22 18:04 Andy Lutomirski [this message] 2018-01-22 18:04 ` [kernel-hardening] [PATCH] x86/retpoline/entry: Disable the entire SYSCALL64 fast path with retpolines on Andy Lutomirski 2018-01-22 18:55 ` Linus Torvalds 2018-01-22 18:55 ` [kernel-hardening] " Linus Torvalds 2018-01-23 8:01 ` Ingo Molnar 2018-01-23 8:01 ` [kernel-hardening] " Ingo Molnar 2018-01-23 18:36 ` Alan Cox 2018-01-23 18:36 ` [kernel-hardening] " Alan Cox 2018-01-25 18:48 ` Linus Torvalds 2018-01-25 18:48 ` [kernel-hardening] " Linus Torvalds 2018-01-25 19:16 ` Linus Torvalds 2018-01-25 20:04 ` Brian Gerst 2018-01-25 20:04 ` [kernel-hardening] " Brian Gerst 2018-01-25 20:54 ` Linus Torvalds 2018-01-25 20:54 ` [kernel-hardening] " Linus Torvalds 2018-01-25 21:02 ` Andy Lutomirski 2018-01-25 21:02 ` [kernel-hardening] " Andy Lutomirski 2018-01-25 21:05 ` Thomas Gleixner 2018-01-25 21:05 ` [kernel-hardening] " Thomas Gleixner 2018-01-25 21:06 ` Linus Torvalds 2018-01-25 21:06 ` [kernel-hardening] " Linus Torvalds 2018-01-25 21:08 ` Andy Lutomirski 2018-01-25 21:08 ` [kernel-hardening] " Andy Lutomirski 2018-01-25 21:20 ` Linus Torvalds 2018-01-25 21:20 ` [kernel-hardening] " Linus Torvalds 2018-01-25 21:31 ` Andy Lutomirski 2018-01-25 21:31 ` [kernel-hardening] " Andy Lutomirski 2018-01-25 21:39 ` Dan Williams 2018-01-25 21:39 ` [kernel-hardening] " Dan Williams 2018-01-25 21:53 ` Andy Lutomirski 2018-01-25 21:53 ` [kernel-hardening] " Andy Lutomirski 2018-01-25 21:53 ` Linus Torvalds 2018-01-25 21:53 ` [kernel-hardening] " Linus Torvalds 2018-01-26 11:17 ` David Laight 2018-01-26 11:17 ` [kernel-hardening] " David Laight 2018-01-26 14:24 ` Alan Cox 2018-01-26 14:24 ` [kernel-hardening] " Alan Cox 2018-01-26 15:57 ` Andy Lutomirski 2018-01-26 15:57 ` [kernel-hardening] " Andy Lutomirski 2018-01-26 17:40 ` Linus Torvalds 2018-01-26 17:40 ` [kernel-hardening] " Linus Torvalds 2018-01-26 18:07 ` Al Viro 2018-01-26 18:07 ` [kernel-hardening] " Al Viro 2018-01-26 18:13 ` Linus Torvalds 2018-01-26 18:13 ` [kernel-hardening] " Linus Torvalds 2018-01-26 18:23 ` Andy Lutomirski 2018-01-26 18:23 ` [kernel-hardening] " Andy Lutomirski 2018-01-26 18:54 ` Linus Torvalds 2018-01-26 18:54 ` [kernel-hardening] " Linus Torvalds 2018-01-26 19:02 ` Andy Lutomirski 2018-01-26 19:02 ` [kernel-hardening] " Andy Lutomirski 2018-01-29 13:19 ` Will Deacon 2018-01-29 13:19 ` [kernel-hardening] " Will Deacon 2018-01-29 15:23 ` David Laight 2018-01-29 15:23 ` [kernel-hardening] " David Laight
Reply instructions: You may reply publicly to this message via plain-text email using any one of the following methods: * Save the following mbox file, import it into your mail client, and reply-to-all from there: mbox Avoid top-posting and favor interleaved quoting: https://en.wikipedia.org/wiki/Posting_style#Interleaved_style * Reply using the --to, --cc, and --in-reply-to switches of git-send-email(1): git send-email \ --in-reply-to=503224b776b9513885453756e44bab235221124e.1516644136.git.luto@kernel.org \ --to=luto@kernel.org \ --cc=alan@linux.intel.com \ --cc=bp@alien8.de \ --cc=dan.j.williams@intel.com \ --cc=gregkh@linuxfoundation.org \ --cc=jannh@google.com \ --cc=kernel-hardening@lists.openwall.com \ --cc=linux-kernel@vger.kernel.org \ --cc=samuel.c.p.neves@gmail.com \ --cc=torvalds@linux-foundation.org \ --cc=x86@kernel.org \ /path/to/YOUR_REPLY https://kernel.org/pub/software/scm/git/docs/git-send-email.html * If your mail client supports setting the In-Reply-To header via mailto: links, try the mailto: linkBe sure your reply has a Subject: header at the top and a blank line before the message body.
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.