From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1753701AbdKXJU7 (ORCPT ); Fri, 24 Nov 2017 04:20:59 -0500 Received: from mail-wm0-f65.google.com ([74.125.82.65]:34289 "EHLO mail-wm0-f65.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1753057AbdKXJPk (ORCPT ); Fri, 24 Nov 2017 04:15:40 -0500 X-Google-Smtp-Source: AGs4zMZzfP8s0cFy2CEmB2uYm1nY/FDW+D9K6iB8S50ST2alrn0XGFwOwG9MReI7eR75aD4c12LZCA== From: Ingo Molnar To: linux-kernel@vger.kernel.org Cc: Dave Hansen , Andy Lutomirski , Thomas Gleixner , "H . Peter Anvin" , Peter Zijlstra , Borislav Petkov , Linus Torvalds Subject: [PATCH 22/43] x86/mm/kaiser: Prepare assembly for entry/exit CR3 switching Date: Fri, 24 Nov 2017 10:14:27 +0100 Message-Id: <20171124091448.7649-23-mingo@kernel.org> X-Mailer: git-send-email 2.14.1 In-Reply-To: <20171124091448.7649-1-mingo@kernel.org> References: <20171124091448.7649-1-mingo@kernel.org> Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org From: Dave Hansen This is largely code from Andy Lutomirski. I fixed a few bugs in it, and added a few SWITCH_TO_* spots. KAISER needs to switch to a different CR3 value when it enters the kernel and switch back when it exits. This essentially needs to be done before leaving assembly code. This is extra challenging because the switching context is tricky: the registers that can be clobbered can vary. It is also hard to store things on the stack because there is an established ABI (ptregs) or the stack is entirely unsafe to use. This patch establishes a set of macros that allow changing to the user and kernel CR3 values. Interactions with SWAPGS: previous versions of the KAISER code relied on having per-cpu scratch space to save/restore a register that can be used for the CR3 MOV. The %GS register is used to index into our per-cpu space, so SWAPGS *had* to be done before the CR3 switch. That scratch space is gone now, but the semantic that SWAPGS must be done before the CR3 MOV is retained. This is good to keep because it is not that hard to do and it allows us to do things like add per-cpu debugging information to help us figure out what goes wrong sometimes. What this does in the NMI code is worth pointing out. NMIs can interrupt *any* context and they can also be nested with NMIs interrupting other NMIs. The comments below ".Lnmi_from_kernel" explain the format of the stack during this situation. Changing the format of this stack is not a fun exercise: I tried. Instead of storing the old CR3 value on the stack, this patch depend on the *regular* register save/restore mechanism and then uses %r14 to keep CR3 during the NMI. It is callee-saved and will not be clobbered by the C NMI handlers that get called. Signed-off-by: Dave Hansen Cc: Andy Lutomirski Cc: Borislav Petkov Cc: Brian Gerst Cc: Daniel Gruss Cc: Denys Vlasenko Cc: H. Peter Anvin Cc: Hugh Dickins Cc: Josh Poimboeuf Cc: Kees Cook Cc: Linus Torvalds Cc: Michael Schwarz Cc: Moritz Lipp Cc: Peter Zijlstra Cc: Richard Fellner Cc: Thomas Gleixner Cc: linux-mm@kvack.org Link: http://lkml.kernel.org/r/20171123003442.2D047A7D@viggo.jf.intel.com Signed-off-by: Ingo Molnar --- arch/x86/entry/calling.h | 65 ++++++++++++++++++++++++++++++++++++++++ arch/x86/entry/entry_64.S | 44 +++++++++++++++++++++++++-- arch/x86/entry/entry_64_compat.S | 32 +++++++++++++++++++- 3 files changed, 137 insertions(+), 4 deletions(-) diff --git a/arch/x86/entry/calling.h b/arch/x86/entry/calling.h index 3fd8bc560fae..e1650da01323 100644 --- a/arch/x86/entry/calling.h +++ b/arch/x86/entry/calling.h @@ -1,6 +1,7 @@ /* SPDX-License-Identifier: GPL-2.0 */ #include #include +#include /* @@ -187,6 +188,70 @@ For 32-bit we have the following conventions - kernel is built with #endif .endm +#ifdef CONFIG_KAISER + +/* KAISER PGDs are 8k. Flip bit 12 to switch between the two halves: */ +#define KAISER_SWITCH_MASK (1<ax) */ movq %r10, %rcx + /* Must wait until we have the kernel CR3 to call C functions: */ + TRACE_IRQS_OFF + /* * This call instruction is handled specially in stub_ptregs_64. * It might end up jumping to the slow path. If it jumps, RAX @@ -393,6 +408,7 @@ GLOBAL(entry_SYSCALL_64_after_hwframe) * We are on the trampoline stack. All regs except RDI are live. * We can do future final exit work right here. */ + SWITCH_TO_USER_CR3 scratch_reg=%rdi popq %rdi popq %rsp @@ -729,6 +745,8 @@ GLOBAL(swapgs_restore_regs_and_return_to_usermode) * We can do future final exit work right here. */ + SWITCH_TO_USER_CR3 scratch_reg=%rdi + /* Restore RDI. */ popq %rdi SWAPGS @@ -937,6 +955,9 @@ apicinterrupt IRQ_WORK_VECTOR irq_work_interrupt smp_irq_work_interrupt ENTRY(switch_to_thread_stack) UNWIND_HINT_FUNC + /* Need to switch before accessing the thread stack. */ + SWITCH_TO_KERNEL_CR3 scratch_reg=%rdi + pushq %rdi movq %rsp, %rdi movq PER_CPU_VAR(cpu_current_top_of_stack), %rsp @@ -1239,7 +1260,11 @@ ENTRY(paranoid_entry) js 1f /* negative -> in kernel */ SWAPGS xorl %ebx, %ebx -1: ret + +1: + SAVE_AND_SWITCH_TO_KERNEL_CR3 scratch_reg=ax save_reg=%r14 + + ret END(paranoid_entry) /* @@ -1261,6 +1286,7 @@ ENTRY(paranoid_exit) testl %ebx, %ebx /* swapgs needed? */ jnz .Lparanoid_exit_no_swapgs TRACE_IRQS_IRETQ + RESTORE_CR3 %r14 SWAPGS_UNSAFE_STACK jmp .Lparanoid_exit_restore .Lparanoid_exit_no_swapgs: @@ -1289,6 +1315,9 @@ ENTRY(error_entry) */ SWAPGS + /* We have user CR3. Change to kernel CR3. */ + SWITCH_TO_KERNEL_CR3 scratch_reg=%rax + .Lerror_entry_from_usermode_after_swapgs: /* Put us onto the real thread stack. */ popq %r12 /* save return addr in %12 */ @@ -1333,6 +1362,7 @@ ENTRY(error_entry) * gsbase and proceed. We'll fix up the exception and land in * .Lgs_change's error handler with kernel gsbase. */ + SWITCH_TO_KERNEL_CR3 scratch_reg=%rax SWAPGS jmp .Lerror_entry_done @@ -1343,9 +1373,10 @@ ENTRY(error_entry) .Lerror_bad_iret: /* - * We came from an IRET to user mode, so we have user gsbase. - * Switch to kernel gsbase: + * We came from an IRET to user mode, so we have user + * gsbase and CR3. Switch to kernel gsbase and CR3: */ + SWITCH_TO_KERNEL_CR3 scratch_reg=%rax SWAPGS /* @@ -1378,6 +1409,10 @@ END(error_exit) /* * Runs on exception stack. Xen PV does not go through this path at all, * so we can use real assembly here. + * + * Registers: + * %r14: Used to save/restore the CR3 of the interrupted context + * when KAISER is in use. Do not clobber. */ ENTRY(nmi) UNWIND_HINT_IRET_REGS @@ -1441,6 +1476,7 @@ ENTRY(nmi) swapgs cld + SWITCH_TO_KERNEL_CR3 scratch_reg=%rdx movq %rsp, %rdx movq PER_CPU_VAR(cpu_current_top_of_stack), %rsp UNWIND_HINT_IRET_REGS base=%rdx offset=8 @@ -1693,6 +1729,8 @@ ENTRY(nmi) movq $-1, %rsi call do_nmi + RESTORE_CR3 save_reg=%r14 + testl %ebx, %ebx /* swapgs needed? */ jnz nmi_restore nmi_swapgs: diff --git a/arch/x86/entry/entry_64_compat.S b/arch/x86/entry/entry_64_compat.S index 95ad40eb7eff..57cd353c0667 100644 --- a/arch/x86/entry/entry_64_compat.S +++ b/arch/x86/entry/entry_64_compat.S @@ -49,6 +49,10 @@ ENTRY(entry_SYSENTER_compat) /* Interrupts are off on entry. */ SWAPGS + + /* We are about to clobber %rsp anyway, clobbering here is OK */ + SWITCH_TO_KERNEL_CR3 scratch_reg=%rsp + movq PER_CPU_VAR(cpu_current_top_of_stack), %rsp /* @@ -215,6 +219,12 @@ GLOBAL(entry_SYSCALL_compat_after_hwframe) pushq $0 /* pt_regs->r14 = 0 */ pushq $0 /* pt_regs->r15 = 0 */ + /* + * We just saved %rdi so it is safe to clobber. It is not + * preserved during the C calls inside TRACE_IRQS_OFF anyway. + */ + SWITCH_TO_KERNEL_CR3 scratch_reg=%rdi + /* * User mode is traced as though IRQs are on, and SYSENTER * turned them off. @@ -256,10 +266,22 @@ GLOBAL(entry_SYSCALL_compat_after_hwframe) * when the system call started, which is already known to user * code. We zero R8-R10 to avoid info leaks. */ + movq RSP-ORIG_RAX(%rsp), %rsp + + /* + * The original userspace %rsp (RSP-ORIG_RAX(%rsp)) is stored + * on the process stack which is not mapped to userspace and + * not readable after we SWITCH_TO_USER_CR3. Delay the CR3 + * switch until after after the last reference to the process + * stack. + * + * %r8 is zeroed before the sysret, thus safe to clobber. + */ + SWITCH_TO_USER_CR3 scratch_reg=%r8 + xorq %r8, %r8 xorq %r9, %r9 xorq %r10, %r10 - movq RSP-ORIG_RAX(%rsp), %rsp swapgs sysretl END(entry_SYSCALL_compat) @@ -297,6 +319,14 @@ ENTRY(entry_INT80_compat) ASM_CLAC /* Do this early to minimize exposure */ SWAPGS + /* + * Must switch CR3 before thread stack is used. %r8 itself + * is not saved into pt_regs and is not preserved across + * function calls (like TRACE_IRQS_OFF calls), thus should + * be safe to use. + */ + SWITCH_TO_KERNEL_CR3 scratch_reg=%r8 + /* * User tracing code (ptrace or signal handlers) might assume that * the saved RAX contains a 32-bit number when we're invoking a 32-bit -- 2.14.1