* [PATCH v7 00/72] x86: SEV-ES Guest Support
@ 2020-09-07 13:15 ` Joerg Roedel
0 siblings, 0 replies; 264+ messages in thread
From: Joerg Roedel @ 2020-09-07 13:15 UTC (permalink / raw)
To: x86
Cc: Joerg Roedel, Joerg Roedel, hpa, Andy Lutomirski, Dave Hansen,
Peter Zijlstra, Jiri Slaby, Dan Williams, Tom Lendacky,
Juergen Gross, Kees Cook, David Rientjes, Cfir Cohen,
Erdem Aktas, Masami Hiramatsu, Mike Stunes, Sean Christopherson,
Martin Radev, linux-kernel, kvm, virtualization
From: Joerg Roedel <jroedel@suse.de>
Hi,
here is a new version of the SEV-ES Guest Support patches for x86. The
previous versions can be found as a linked list starting here:
https://lore.kernel.org/lkml/20200824085511.7553-1-joro@8bytes.org/
I updated the patch-set based on ther review comments I got and the
discussions around it.
Another important change is that the early IDT setup code is now
completly moved to arch/x86/kernel/head64.c. This makes the whole
early exception handling setup code more robust for kernels that have
KASAN and/or Tracing enabled.
A side effect of this change is that secondary CPU now don't use the
idt_table at early boot, which means that on the secondary CPUs the
early handler does not use IST or paranoid_entry() anymore. This
allowed to remove a couple of patches from this series which were only
needed to setup relevant processor starte early enough for IST
exceptions. In particular this means that the early FSGSBASE and TSS
setup is gone now. Also the patch which moved the idt_table to the
data segement is now removed.
A related change was necessary in the boot path of secondary CPUs,
because those loaded the runtime IDT before setting up the TSS and the
getcpu GDT entry. This is now in proper order so that IST exceptions
will work when the runtime IDT is loaded for the first time. This
setup is added in patch 67.
The cpu_init() function is untouched so that it still act as the
intended cpu-state barrier, regardless of what happens before.
The code survived the usual load test of running one x86-selftest loop
on each core of the guest in parallel with perf (for NMI load). This
runs for several (4+) hours now without any issues.
Please review.
Thanks,
Joerg
Borislav Petkov (1):
KVM: SVM: Use __packed shorthand
Doug Covelli (1):
x86/vmware: Add VMware specific handling for VMMCALL under SEV-ES
Joerg Roedel (50):
KVM: SVM: nested: Don't allocate VMCB structures on stack
KVM: SVM: Add GHCB Accessor functions
x86/traps: Move pf error codes to <asm/trap_pf.h>
x86/insn: Make inat-tables.c suitable for pre-decompression code
x86/umip: Factor out instruction fetch
x86/umip: Factor out instruction decoding
x86/insn: Add insn_get_modrm_reg_off()
x86/insn: Add insn_has_rep_prefix() helper
x86/boot/compressed/64: Disable red-zone usage
x86/boot/compressed/64: Add IDT Infrastructure
x86/boot/compressed/64: Rename kaslr_64.c to ident_map_64.c
x86/boot/compressed/64: Add page-fault handler
x86/boot/compressed/64: Always switch to own page-table
x86/boot/compressed/64: Don't pre-map memory in KASLR code
x86/boot/compressed/64: Change add_identity_map() to take start and
end
x86/boot/compressed/64: Add stage1 #VC handler
x86/boot/compressed/64: Call set_sev_encryption_mask() earlier
x86/boot/compressed/64: Check return value of
kernel_ident_mapping_init()
x86/boot/compressed/64: Add set_page_en/decrypted() helpers
x86/boot/compressed/64: Setup GHCB Based VC Exception handler
x86/boot/compressed/64: Unmap GHCB page before booting the kernel
x86/fpu: Move xgetbv()/xsetbv() into separate header
x86/idt: Split idt_data setup out of set_intr_gate()
x86/head/64: Install startup GDT
x86/head/64: Load GDT after switch to virtual addresses
x86/head/64: Load segment registers earlier
x86/head/64: Switch to initial stack earlier
x86/head/64: Install a CPU bringup IDT
x86/idt: Move two function from k/idt.c to i/a/desc.h
x86/head/64: Move early exception dispatch to C code
x86/sev-es: Add SEV-ES Feature Detection
x86/sev-es: Print SEV-ES info into kernel log
x86/sev-es: Compile early handler code into kernel image
x86/sev-es: Setup early #VC handler
x86/sev-es: Setup GHCB based boot #VC handler
x86/sev-es: Allocate and Map IST stack for #VC handler
x86/sev-es: Adjust #VC IST Stack on entering NMI handler
x86/dumpstack/64: Add noinstr version of get_stack_info()
x86/entry/64: Add entry code for #VC handler
x86/sev-es: Wire up existing #VC exit-code handlers
x86/sev-es: Handle instruction fetches from user-space
x86/sev-es: Handle MMIO String Instructions
x86/sev-es: Handle #AC Events
x86/sev-es: Handle #DB Events
x86/paravirt: Allow hypervisor specific VMMCALL handling under SEV-ES
x86/realmode: Add SEV-ES specific trampoline entry point
x86/smpboot: Load TSS and getcpu GDT entry before loading IDT
x86/head/64: Don't call verify_cpu() on starting APs
x86/sev-es: Support CPU offline/online
x86/sev-es: Handle NMI State
Martin Radev (1):
x86/sev-es: Check required CPU features for SEV-ES
Tom Lendacky (19):
KVM: SVM: Add GHCB definitions
x86/cpufeatures: Add SEV-ES CPU feature
x86/sev-es: Add support for handling IOIO exceptions
x86/sev-es: Add CPUID handling to #VC handler
x86/sev-es: Setup per-cpu GHCBs for the runtime handler
x86/sev-es: Add Runtime #VC Exception Handler
x86/sev-es: Handle MMIO events
x86/sev-es: Handle MSR events
x86/sev-es: Handle DR7 read/write events
x86/sev-es: Handle WBINVD Events
x86/sev-es: Handle RDTSC(P) Events
x86/sev-es: Handle RDPMC Events
x86/sev-es: Handle INVD Events
x86/sev-es: Handle MONITOR/MONITORX Events
x86/sev-es: Handle MWAIT/MWAITX Events
x86/sev-es: Handle VMMCALL Events
x86/kvm: Add KVM specific VMMCALL handling under SEV-ES
x86/realmode: Setup AP jump table
x86/efi: Add GHCB mappings when SEV-ES is active
arch/x86/Kconfig | 1 +
arch/x86/boot/compressed/Makefile | 11 +-
arch/x86/boot/compressed/cpuflags.c | 4 -
arch/x86/boot/compressed/head_64.S | 33 +-
arch/x86/boot/compressed/ident_map_64.c | 349 +++++
arch/x86/boot/compressed/idt_64.c | 54 +
arch/x86/boot/compressed/idt_handlers_64.S | 77 ++
arch/x86/boot/compressed/kaslr.c | 36 +-
arch/x86/boot/compressed/kaslr_64.c | 153 ---
arch/x86/boot/compressed/misc.c | 7 +
arch/x86/boot/compressed/misc.h | 50 +-
arch/x86/boot/compressed/sev-es.c | 214 +++
arch/x86/entry/entry_64.S | 80 ++
arch/x86/include/asm/cpu_entry_area.h | 33 +-
arch/x86/include/asm/cpufeatures.h | 1 +
arch/x86/include/asm/desc.h | 27 +
arch/x86/include/asm/desc_defs.h | 10 +
arch/x86/include/asm/fpu/internal.h | 30 +-
arch/x86/include/asm/fpu/xcr.h | 34 +
arch/x86/include/asm/idtentry.h | 50 +
arch/x86/include/asm/insn-eval.h | 6 +
arch/x86/include/asm/mem_encrypt.h | 5 +
arch/x86/include/asm/msr-index.h | 3 +
arch/x86/include/asm/page_64_types.h | 1 +
arch/x86/include/asm/pgtable.h | 2 +-
arch/x86/include/asm/processor.h | 1 +
arch/x86/include/asm/proto.h | 1 +
arch/x86/include/asm/realmode.h | 7 +
arch/x86/include/asm/segment.h | 2 +-
arch/x86/include/asm/setup.h | 6 +-
arch/x86/include/asm/sev-es.h | 114 ++
arch/x86/include/asm/stacktrace.h | 2 +
arch/x86/include/asm/svm.h | 106 +-
arch/x86/include/asm/trap_pf.h | 24 +
arch/x86/include/asm/trapnr.h | 1 +
arch/x86/include/asm/traps.h | 20 +-
arch/x86/include/asm/x86_init.h | 16 +-
arch/x86/include/uapi/asm/svm.h | 11 +
arch/x86/kernel/Makefile | 3 +
arch/x86/kernel/cpu/amd.c | 3 +-
arch/x86/kernel/cpu/common.c | 25 +
arch/x86/kernel/cpu/scattered.c | 1 +
arch/x86/kernel/cpu/vmware.c | 50 +-
arch/x86/kernel/dumpstack.c | 7 +-
arch/x86/kernel/dumpstack_64.c | 46 +-
arch/x86/kernel/head64.c | 122 +-
arch/x86/kernel/head_64.S | 160 ++-
arch/x86/kernel/idt.c | 41 +-
arch/x86/kernel/kvm.c | 35 +-
arch/x86/kernel/nmi.c | 15 +
arch/x86/kernel/sev-es-shared.c | 507 +++++++
arch/x86/kernel/sev-es.c | 1404 ++++++++++++++++++++
arch/x86/kernel/smpboot.c | 2 +-
arch/x86/kernel/traps.c | 48 +
arch/x86/kernel/umip.c | 49 +-
arch/x86/kvm/svm/nested.c | 47 +-
arch/x86/kvm/svm/svm.c | 2 +
arch/x86/lib/insn-eval.c | 130 ++
arch/x86/mm/cpu_entry_area.c | 3 +-
arch/x86/mm/extable.c | 1 +
arch/x86/mm/mem_encrypt.c | 38 +-
arch/x86/mm/mem_encrypt_identity.c | 3 +
arch/x86/platform/efi/efi_64.c | 10 +
arch/x86/realmode/init.c | 24 +-
arch/x86/realmode/rm/header.S | 3 +
arch/x86/realmode/rm/trampoline_64.S | 20 +
arch/x86/tools/gen-insn-attr-x86.awk | 50 +-
tools/arch/x86/tools/gen-insn-attr-x86.awk | 50 +-
68 files changed, 4030 insertions(+), 451 deletions(-)
create mode 100644 arch/x86/boot/compressed/ident_map_64.c
create mode 100644 arch/x86/boot/compressed/idt_64.c
create mode 100644 arch/x86/boot/compressed/idt_handlers_64.S
delete mode 100644 arch/x86/boot/compressed/kaslr_64.c
create mode 100644 arch/x86/boot/compressed/sev-es.c
create mode 100644 arch/x86/include/asm/fpu/xcr.h
create mode 100644 arch/x86/include/asm/sev-es.h
create mode 100644 arch/x86/include/asm/trap_pf.h
create mode 100644 arch/x86/kernel/sev-es-shared.c
create mode 100644 arch/x86/kernel/sev-es.c
--
2.28.0
^ permalink raw reply [flat|nested] 264+ messages in thread
* [PATCH v7 00/72] x86: SEV-ES Guest Support
@ 2020-09-07 13:15 ` Joerg Roedel
0 siblings, 0 replies; 264+ messages in thread
From: Joerg Roedel @ 2020-09-07 13:15 UTC (permalink / raw)
To: x86
Cc: Juergen Gross, Tom Lendacky, Joerg Roedel, Mike Stunes,
Kees Cook, kvm, Peter Zijlstra, Cfir Cohen, Joerg Roedel,
Dave Hansen, linux-kernel, Sean Christopherson, virtualization,
Martin Radev, Masami Hiramatsu, Andy Lutomirski, hpa,
Erdem Aktas, David Rientjes, Dan Williams, Jiri Slaby
From: Joerg Roedel <jroedel@suse.de>
Hi,
here is a new version of the SEV-ES Guest Support patches for x86. The
previous versions can be found as a linked list starting here:
https://lore.kernel.org/lkml/20200824085511.7553-1-joro@8bytes.org/
I updated the patch-set based on ther review comments I got and the
discussions around it.
Another important change is that the early IDT setup code is now
completly moved to arch/x86/kernel/head64.c. This makes the whole
early exception handling setup code more robust for kernels that have
KASAN and/or Tracing enabled.
A side effect of this change is that secondary CPU now don't use the
idt_table at early boot, which means that on the secondary CPUs the
early handler does not use IST or paranoid_entry() anymore. This
allowed to remove a couple of patches from this series which were only
needed to setup relevant processor starte early enough for IST
exceptions. In particular this means that the early FSGSBASE and TSS
setup is gone now. Also the patch which moved the idt_table to the
data segement is now removed.
A related change was necessary in the boot path of secondary CPUs,
because those loaded the runtime IDT before setting up the TSS and the
getcpu GDT entry. This is now in proper order so that IST exceptions
will work when the runtime IDT is loaded for the first time. This
setup is added in patch 67.
The cpu_init() function is untouched so that it still act as the
intended cpu-state barrier, regardless of what happens before.
The code survived the usual load test of running one x86-selftest loop
on each core of the guest in parallel with perf (for NMI load). This
runs for several (4+) hours now without any issues.
Please review.
Thanks,
Joerg
Borislav Petkov (1):
KVM: SVM: Use __packed shorthand
Doug Covelli (1):
x86/vmware: Add VMware specific handling for VMMCALL under SEV-ES
Joerg Roedel (50):
KVM: SVM: nested: Don't allocate VMCB structures on stack
KVM: SVM: Add GHCB Accessor functions
x86/traps: Move pf error codes to <asm/trap_pf.h>
x86/insn: Make inat-tables.c suitable for pre-decompression code
x86/umip: Factor out instruction fetch
x86/umip: Factor out instruction decoding
x86/insn: Add insn_get_modrm_reg_off()
x86/insn: Add insn_has_rep_prefix() helper
x86/boot/compressed/64: Disable red-zone usage
x86/boot/compressed/64: Add IDT Infrastructure
x86/boot/compressed/64: Rename kaslr_64.c to ident_map_64.c
x86/boot/compressed/64: Add page-fault handler
x86/boot/compressed/64: Always switch to own page-table
x86/boot/compressed/64: Don't pre-map memory in KASLR code
x86/boot/compressed/64: Change add_identity_map() to take start and
end
x86/boot/compressed/64: Add stage1 #VC handler
x86/boot/compressed/64: Call set_sev_encryption_mask() earlier
x86/boot/compressed/64: Check return value of
kernel_ident_mapping_init()
x86/boot/compressed/64: Add set_page_en/decrypted() helpers
x86/boot/compressed/64: Setup GHCB Based VC Exception handler
x86/boot/compressed/64: Unmap GHCB page before booting the kernel
x86/fpu: Move xgetbv()/xsetbv() into separate header
x86/idt: Split idt_data setup out of set_intr_gate()
x86/head/64: Install startup GDT
x86/head/64: Load GDT after switch to virtual addresses
x86/head/64: Load segment registers earlier
x86/head/64: Switch to initial stack earlier
x86/head/64: Install a CPU bringup IDT
x86/idt: Move two function from k/idt.c to i/a/desc.h
x86/head/64: Move early exception dispatch to C code
x86/sev-es: Add SEV-ES Feature Detection
x86/sev-es: Print SEV-ES info into kernel log
x86/sev-es: Compile early handler code into kernel image
x86/sev-es: Setup early #VC handler
x86/sev-es: Setup GHCB based boot #VC handler
x86/sev-es: Allocate and Map IST stack for #VC handler
x86/sev-es: Adjust #VC IST Stack on entering NMI handler
x86/dumpstack/64: Add noinstr version of get_stack_info()
x86/entry/64: Add entry code for #VC handler
x86/sev-es: Wire up existing #VC exit-code handlers
x86/sev-es: Handle instruction fetches from user-space
x86/sev-es: Handle MMIO String Instructions
x86/sev-es: Handle #AC Events
x86/sev-es: Handle #DB Events
x86/paravirt: Allow hypervisor specific VMMCALL handling under SEV-ES
x86/realmode: Add SEV-ES specific trampoline entry point
x86/smpboot: Load TSS and getcpu GDT entry before loading IDT
x86/head/64: Don't call verify_cpu() on starting APs
x86/sev-es: Support CPU offline/online
x86/sev-es: Handle NMI State
Martin Radev (1):
x86/sev-es: Check required CPU features for SEV-ES
Tom Lendacky (19):
KVM: SVM: Add GHCB definitions
x86/cpufeatures: Add SEV-ES CPU feature
x86/sev-es: Add support for handling IOIO exceptions
x86/sev-es: Add CPUID handling to #VC handler
x86/sev-es: Setup per-cpu GHCBs for the runtime handler
x86/sev-es: Add Runtime #VC Exception Handler
x86/sev-es: Handle MMIO events
x86/sev-es: Handle MSR events
x86/sev-es: Handle DR7 read/write events
x86/sev-es: Handle WBINVD Events
x86/sev-es: Handle RDTSC(P) Events
x86/sev-es: Handle RDPMC Events
x86/sev-es: Handle INVD Events
x86/sev-es: Handle MONITOR/MONITORX Events
x86/sev-es: Handle MWAIT/MWAITX Events
x86/sev-es: Handle VMMCALL Events
x86/kvm: Add KVM specific VMMCALL handling under SEV-ES
x86/realmode: Setup AP jump table
x86/efi: Add GHCB mappings when SEV-ES is active
arch/x86/Kconfig | 1 +
arch/x86/boot/compressed/Makefile | 11 +-
arch/x86/boot/compressed/cpuflags.c | 4 -
arch/x86/boot/compressed/head_64.S | 33 +-
arch/x86/boot/compressed/ident_map_64.c | 349 +++++
arch/x86/boot/compressed/idt_64.c | 54 +
arch/x86/boot/compressed/idt_handlers_64.S | 77 ++
arch/x86/boot/compressed/kaslr.c | 36 +-
arch/x86/boot/compressed/kaslr_64.c | 153 ---
arch/x86/boot/compressed/misc.c | 7 +
arch/x86/boot/compressed/misc.h | 50 +-
arch/x86/boot/compressed/sev-es.c | 214 +++
arch/x86/entry/entry_64.S | 80 ++
arch/x86/include/asm/cpu_entry_area.h | 33 +-
arch/x86/include/asm/cpufeatures.h | 1 +
arch/x86/include/asm/desc.h | 27 +
arch/x86/include/asm/desc_defs.h | 10 +
arch/x86/include/asm/fpu/internal.h | 30 +-
arch/x86/include/asm/fpu/xcr.h | 34 +
arch/x86/include/asm/idtentry.h | 50 +
arch/x86/include/asm/insn-eval.h | 6 +
arch/x86/include/asm/mem_encrypt.h | 5 +
arch/x86/include/asm/msr-index.h | 3 +
arch/x86/include/asm/page_64_types.h | 1 +
arch/x86/include/asm/pgtable.h | 2 +-
arch/x86/include/asm/processor.h | 1 +
arch/x86/include/asm/proto.h | 1 +
arch/x86/include/asm/realmode.h | 7 +
arch/x86/include/asm/segment.h | 2 +-
arch/x86/include/asm/setup.h | 6 +-
arch/x86/include/asm/sev-es.h | 114 ++
arch/x86/include/asm/stacktrace.h | 2 +
arch/x86/include/asm/svm.h | 106 +-
arch/x86/include/asm/trap_pf.h | 24 +
arch/x86/include/asm/trapnr.h | 1 +
arch/x86/include/asm/traps.h | 20 +-
arch/x86/include/asm/x86_init.h | 16 +-
arch/x86/include/uapi/asm/svm.h | 11 +
arch/x86/kernel/Makefile | 3 +
arch/x86/kernel/cpu/amd.c | 3 +-
arch/x86/kernel/cpu/common.c | 25 +
arch/x86/kernel/cpu/scattered.c | 1 +
arch/x86/kernel/cpu/vmware.c | 50 +-
arch/x86/kernel/dumpstack.c | 7 +-
arch/x86/kernel/dumpstack_64.c | 46 +-
arch/x86/kernel/head64.c | 122 +-
arch/x86/kernel/head_64.S | 160 ++-
arch/x86/kernel/idt.c | 41 +-
arch/x86/kernel/kvm.c | 35 +-
arch/x86/kernel/nmi.c | 15 +
arch/x86/kernel/sev-es-shared.c | 507 +++++++
arch/x86/kernel/sev-es.c | 1404 ++++++++++++++++++++
arch/x86/kernel/smpboot.c | 2 +-
arch/x86/kernel/traps.c | 48 +
arch/x86/kernel/umip.c | 49 +-
arch/x86/kvm/svm/nested.c | 47 +-
arch/x86/kvm/svm/svm.c | 2 +
arch/x86/lib/insn-eval.c | 130 ++
arch/x86/mm/cpu_entry_area.c | 3 +-
arch/x86/mm/extable.c | 1 +
arch/x86/mm/mem_encrypt.c | 38 +-
arch/x86/mm/mem_encrypt_identity.c | 3 +
arch/x86/platform/efi/efi_64.c | 10 +
arch/x86/realmode/init.c | 24 +-
arch/x86/realmode/rm/header.S | 3 +
arch/x86/realmode/rm/trampoline_64.S | 20 +
arch/x86/tools/gen-insn-attr-x86.awk | 50 +-
tools/arch/x86/tools/gen-insn-attr-x86.awk | 50 +-
68 files changed, 4030 insertions(+), 451 deletions(-)
create mode 100644 arch/x86/boot/compressed/ident_map_64.c
create mode 100644 arch/x86/boot/compressed/idt_64.c
create mode 100644 arch/x86/boot/compressed/idt_handlers_64.S
delete mode 100644 arch/x86/boot/compressed/kaslr_64.c
create mode 100644 arch/x86/boot/compressed/sev-es.c
create mode 100644 arch/x86/include/asm/fpu/xcr.h
create mode 100644 arch/x86/include/asm/sev-es.h
create mode 100644 arch/x86/include/asm/trap_pf.h
create mode 100644 arch/x86/kernel/sev-es-shared.c
create mode 100644 arch/x86/kernel/sev-es.c
--
2.28.0
_______________________________________________
Virtualization mailing list
Virtualization@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/virtualization
^ permalink raw reply [flat|nested] 264+ messages in thread
* [PATCH v7 01/72] KVM: SVM: nested: Don't allocate VMCB structures on stack
2020-09-07 13:15 ` Joerg Roedel
@ 2020-09-07 13:15 ` Joerg Roedel
-1 siblings, 0 replies; 264+ messages in thread
From: Joerg Roedel @ 2020-09-07 13:15 UTC (permalink / raw)
To: x86
Cc: Joerg Roedel, Joerg Roedel, hpa, Andy Lutomirski, Dave Hansen,
Peter Zijlstra, Jiri Slaby, Dan Williams, Tom Lendacky,
Juergen Gross, Kees Cook, David Rientjes, Cfir Cohen,
Erdem Aktas, Masami Hiramatsu, Mike Stunes, Sean Christopherson,
Martin Radev, linux-kernel, kvm, virtualization
From: Joerg Roedel <jroedel@suse.de>
Do not allocate a vmcb_control_area and a vmcb_save_area on the stack,
as these structures will become larger with future extenstions of
SVM and thus the svm_set_nested_state() function will become a too large
stack frame.
Signed-off-by: Joerg Roedel <jroedel@suse.de>
---
arch/x86/kvm/svm/nested.c | 47 +++++++++++++++++++++++++++------------
1 file changed, 33 insertions(+), 14 deletions(-)
diff --git a/arch/x86/kvm/svm/nested.c b/arch/x86/kvm/svm/nested.c
index fb68467e6049..28036629abf8 100644
--- a/arch/x86/kvm/svm/nested.c
+++ b/arch/x86/kvm/svm/nested.c
@@ -1060,10 +1060,14 @@ static int svm_set_nested_state(struct kvm_vcpu *vcpu,
struct vmcb *hsave = svm->nested.hsave;
struct vmcb __user *user_vmcb = (struct vmcb __user *)
&user_kvm_nested_state->data.svm[0];
- struct vmcb_control_area ctl;
- struct vmcb_save_area save;
+ struct vmcb_control_area *ctl;
+ struct vmcb_save_area *save;
+ int ret;
u32 cr0;
+ BUILD_BUG_ON(sizeof(struct vmcb_control_area) + sizeof(struct vmcb_save_area) >
+ KVM_STATE_NESTED_SVM_VMCB_SIZE);
+
if (kvm_state->format != KVM_STATE_NESTED_FORMAT_SVM)
return -EINVAL;
@@ -1095,13 +1099,22 @@ static int svm_set_nested_state(struct kvm_vcpu *vcpu,
return -EINVAL;
if (kvm_state->size < sizeof(*kvm_state) + KVM_STATE_NESTED_SVM_VMCB_SIZE)
return -EINVAL;
- if (copy_from_user(&ctl, &user_vmcb->control, sizeof(ctl)))
- return -EFAULT;
- if (copy_from_user(&save, &user_vmcb->save, sizeof(save)))
- return -EFAULT;
- if (!nested_vmcb_check_controls(&ctl))
- return -EINVAL;
+ ret = -ENOMEM;
+ ctl = kzalloc(sizeof(*ctl), GFP_KERNEL);
+ save = kzalloc(sizeof(*save), GFP_KERNEL);
+ if (!ctl || !save)
+ goto out_free;
+
+ ret = -EFAULT;
+ if (copy_from_user(ctl, &user_vmcb->control, sizeof(*ctl)))
+ goto out_free;
+ if (copy_from_user(save, &user_vmcb->save, sizeof(*save)))
+ goto out_free;
+
+ ret = -EINVAL;
+ if (!nested_vmcb_check_controls(ctl))
+ goto out_free;
/*
* Processor state contains L2 state. Check that it is
@@ -1109,15 +1122,15 @@ static int svm_set_nested_state(struct kvm_vcpu *vcpu,
*/
cr0 = kvm_read_cr0(vcpu);
if (((cr0 & X86_CR0_CD) == 0) && (cr0 & X86_CR0_NW))
- return -EINVAL;
+ goto out_free;
/*
* Validate host state saved from before VMRUN (see
* nested_svm_check_permissions).
* TODO: validate reserved bits for all saved state.
*/
- if (!(save.cr0 & X86_CR0_PG))
- return -EINVAL;
+ if (!(save->cr0 & X86_CR0_PG))
+ goto out_free;
/*
* All checks done, we can enter guest mode. L1 control fields
@@ -1126,15 +1139,21 @@ static int svm_set_nested_state(struct kvm_vcpu *vcpu,
* contains saved L1 state.
*/
copy_vmcb_control_area(&hsave->control, &svm->vmcb->control);
- hsave->save = save;
+ hsave->save = *save;
svm->nested.vmcb = kvm_state->hdr.svm.vmcb_pa;
- load_nested_vmcb_control(svm, &ctl);
+ load_nested_vmcb_control(svm, ctl);
nested_prepare_vmcb_control(svm);
out_set_gif:
svm_set_gif(svm, !!(kvm_state->flags & KVM_STATE_NESTED_GIF_SET));
- return 0;
+
+ ret = 0;
+out_free:
+ kfree(save);
+ kfree(ctl);
+
+ return ret;
}
struct kvm_x86_nested_ops svm_nested_ops = {
--
2.28.0
^ permalink raw reply related [flat|nested] 264+ messages in thread
* [PATCH v7 01/72] KVM: SVM: nested: Don't allocate VMCB structures on stack
@ 2020-09-07 13:15 ` Joerg Roedel
0 siblings, 0 replies; 264+ messages in thread
From: Joerg Roedel @ 2020-09-07 13:15 UTC (permalink / raw)
To: x86
Cc: Juergen Gross, Tom Lendacky, Joerg Roedel, Mike Stunes,
Kees Cook, kvm, Peter Zijlstra, Cfir Cohen, Joerg Roedel,
Dave Hansen, linux-kernel, Sean Christopherson, virtualization,
Martin Radev, Masami Hiramatsu, Andy Lutomirski, hpa,
Erdem Aktas, David Rientjes, Dan Williams, Jiri Slaby
From: Joerg Roedel <jroedel@suse.de>
Do not allocate a vmcb_control_area and a vmcb_save_area on the stack,
as these structures will become larger with future extenstions of
SVM and thus the svm_set_nested_state() function will become a too large
stack frame.
Signed-off-by: Joerg Roedel <jroedel@suse.de>
---
arch/x86/kvm/svm/nested.c | 47 +++++++++++++++++++++++++++------------
1 file changed, 33 insertions(+), 14 deletions(-)
diff --git a/arch/x86/kvm/svm/nested.c b/arch/x86/kvm/svm/nested.c
index fb68467e6049..28036629abf8 100644
--- a/arch/x86/kvm/svm/nested.c
+++ b/arch/x86/kvm/svm/nested.c
@@ -1060,10 +1060,14 @@ static int svm_set_nested_state(struct kvm_vcpu *vcpu,
struct vmcb *hsave = svm->nested.hsave;
struct vmcb __user *user_vmcb = (struct vmcb __user *)
&user_kvm_nested_state->data.svm[0];
- struct vmcb_control_area ctl;
- struct vmcb_save_area save;
+ struct vmcb_control_area *ctl;
+ struct vmcb_save_area *save;
+ int ret;
u32 cr0;
+ BUILD_BUG_ON(sizeof(struct vmcb_control_area) + sizeof(struct vmcb_save_area) >
+ KVM_STATE_NESTED_SVM_VMCB_SIZE);
+
if (kvm_state->format != KVM_STATE_NESTED_FORMAT_SVM)
return -EINVAL;
@@ -1095,13 +1099,22 @@ static int svm_set_nested_state(struct kvm_vcpu *vcpu,
return -EINVAL;
if (kvm_state->size < sizeof(*kvm_state) + KVM_STATE_NESTED_SVM_VMCB_SIZE)
return -EINVAL;
- if (copy_from_user(&ctl, &user_vmcb->control, sizeof(ctl)))
- return -EFAULT;
- if (copy_from_user(&save, &user_vmcb->save, sizeof(save)))
- return -EFAULT;
- if (!nested_vmcb_check_controls(&ctl))
- return -EINVAL;
+ ret = -ENOMEM;
+ ctl = kzalloc(sizeof(*ctl), GFP_KERNEL);
+ save = kzalloc(sizeof(*save), GFP_KERNEL);
+ if (!ctl || !save)
+ goto out_free;
+
+ ret = -EFAULT;
+ if (copy_from_user(ctl, &user_vmcb->control, sizeof(*ctl)))
+ goto out_free;
+ if (copy_from_user(save, &user_vmcb->save, sizeof(*save)))
+ goto out_free;
+
+ ret = -EINVAL;
+ if (!nested_vmcb_check_controls(ctl))
+ goto out_free;
/*
* Processor state contains L2 state. Check that it is
@@ -1109,15 +1122,15 @@ static int svm_set_nested_state(struct kvm_vcpu *vcpu,
*/
cr0 = kvm_read_cr0(vcpu);
if (((cr0 & X86_CR0_CD) == 0) && (cr0 & X86_CR0_NW))
- return -EINVAL;
+ goto out_free;
/*
* Validate host state saved from before VMRUN (see
* nested_svm_check_permissions).
* TODO: validate reserved bits for all saved state.
*/
- if (!(save.cr0 & X86_CR0_PG))
- return -EINVAL;
+ if (!(save->cr0 & X86_CR0_PG))
+ goto out_free;
/*
* All checks done, we can enter guest mode. L1 control fields
@@ -1126,15 +1139,21 @@ static int svm_set_nested_state(struct kvm_vcpu *vcpu,
* contains saved L1 state.
*/
copy_vmcb_control_area(&hsave->control, &svm->vmcb->control);
- hsave->save = save;
+ hsave->save = *save;
svm->nested.vmcb = kvm_state->hdr.svm.vmcb_pa;
- load_nested_vmcb_control(svm, &ctl);
+ load_nested_vmcb_control(svm, ctl);
nested_prepare_vmcb_control(svm);
out_set_gif:
svm_set_gif(svm, !!(kvm_state->flags & KVM_STATE_NESTED_GIF_SET));
- return 0;
+
+ ret = 0;
+out_free:
+ kfree(save);
+ kfree(ctl);
+
+ return ret;
}
struct kvm_x86_nested_ops svm_nested_ops = {
--
2.28.0
_______________________________________________
Virtualization mailing list
Virtualization@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/virtualization
^ permalink raw reply related [flat|nested] 264+ messages in thread
* [PATCH v7 02/72] KVM: SVM: Add GHCB definitions
2020-09-07 13:15 ` Joerg Roedel
@ 2020-09-07 13:15 ` Joerg Roedel
-1 siblings, 0 replies; 264+ messages in thread
From: Joerg Roedel @ 2020-09-07 13:15 UTC (permalink / raw)
To: x86
Cc: Joerg Roedel, Joerg Roedel, Tom Lendacky, hpa, Andy Lutomirski,
Dave Hansen, Peter Zijlstra, Jiri Slaby, Dan Williams,
Juergen Gross, Kees Cook, David Rientjes, Cfir Cohen,
Erdem Aktas, Masami Hiramatsu, Mike Stunes, Sean Christopherson,
Martin Radev, linux-kernel, kvm, virtualization
From: Tom Lendacky <thomas.lendacky@amd.com>
Extend the vmcb_safe_area with SEV-ES fields and add a new
'struct ghcb' which will be used for guest-hypervisor communication.
Signed-off-by: Tom Lendacky <thomas.lendacky@amd.com>
Signed-off-by: Joerg Roedel <jroedel@suse.de>
---
arch/x86/include/asm/svm.h | 51 ++++++++++++++++++++++++++++++++++++--
arch/x86/kvm/svm/svm.c | 2 ++
2 files changed, 51 insertions(+), 2 deletions(-)
diff --git a/arch/x86/include/asm/svm.h b/arch/x86/include/asm/svm.h
index 8a1f5382a4ea..acac55d6f941 100644
--- a/arch/x86/include/asm/svm.h
+++ b/arch/x86/include/asm/svm.h
@@ -200,13 +200,60 @@ struct __attribute__ ((__packed__)) vmcb_save_area {
u64 br_to;
u64 last_excp_from;
u64 last_excp_to;
+
+ /*
+ * The following part of the save area is valid only for
+ * SEV-ES guests when referenced through the GHCB.
+ */
+ u8 reserved_7[104];
+ u64 reserved_8; /* rax already available at 0x01f8 */
+ u64 rcx;
+ u64 rdx;
+ u64 rbx;
+ u64 reserved_9; /* rsp already available at 0x01d8 */
+ u64 rbp;
+ u64 rsi;
+ u64 rdi;
+ u64 r8;
+ u64 r9;
+ u64 r10;
+ u64 r11;
+ u64 r12;
+ u64 r13;
+ u64 r14;
+ u64 r15;
+ u8 reserved_10[16];
+ u64 sw_exit_code;
+ u64 sw_exit_info_1;
+ u64 sw_exit_info_2;
+ u64 sw_scratch;
+ u8 reserved_11[56];
+ u64 xcr0;
+ u8 valid_bitmap[16];
+ u64 x87_state_gpa;
};
+struct ghcb {
+ struct vmcb_save_area save;
+ u8 reserved_save[2048 - sizeof(struct vmcb_save_area)];
+
+ u8 shared_buffer[2032];
+
+ u8 reserved_1[10];
+ u16 protocol_version; /* negotiated SEV-ES/GHCB protocol version */
+ u32 ghcb_usage;
+} __packed;
+
+
+#define EXPECTED_VMCB_SAVE_AREA_SIZE 1032
+#define EXPECTED_VMCB_CONTROL_AREA_SIZE 256
+#define EXPECTED_GHCB_SIZE PAGE_SIZE
static inline void __unused_size_checks(void)
{
- BUILD_BUG_ON(sizeof(struct vmcb_save_area) != 0x298);
- BUILD_BUG_ON(sizeof(struct vmcb_control_area) != 256);
+ BUILD_BUG_ON(sizeof(struct vmcb_save_area) != EXPECTED_VMCB_SAVE_AREA_SIZE);
+ BUILD_BUG_ON(sizeof(struct vmcb_control_area) != EXPECTED_VMCB_CONTROL_AREA_SIZE);
+ BUILD_BUG_ON(sizeof(struct ghcb) != EXPECTED_GHCB_SIZE);
}
struct __attribute__ ((__packed__)) vmcb {
diff --git a/arch/x86/kvm/svm/svm.c b/arch/x86/kvm/svm/svm.c
index 0194336b64a4..1db4fdcb4906 100644
--- a/arch/x86/kvm/svm/svm.c
+++ b/arch/x86/kvm/svm/svm.c
@@ -4164,6 +4164,8 @@ static struct kvm_x86_init_ops svm_init_ops __initdata = {
static int __init svm_init(void)
{
+ __unused_size_checks();
+
return kvm_init(&svm_init_ops, sizeof(struct vcpu_svm),
__alignof__(struct vcpu_svm), THIS_MODULE);
}
--
2.28.0
^ permalink raw reply related [flat|nested] 264+ messages in thread
* [PATCH v7 02/72] KVM: SVM: Add GHCB definitions
@ 2020-09-07 13:15 ` Joerg Roedel
0 siblings, 0 replies; 264+ messages in thread
From: Joerg Roedel @ 2020-09-07 13:15 UTC (permalink / raw)
To: x86
Cc: Juergen Gross, Tom Lendacky, Joerg Roedel, Mike Stunes,
Kees Cook, kvm, Peter Zijlstra, Cfir Cohen, Joerg Roedel,
Dave Hansen, linux-kernel, Sean Christopherson, virtualization,
Martin Radev, Masami Hiramatsu, Andy Lutomirski, hpa,
Erdem Aktas, David Rientjes, Dan Williams, Jiri Slaby
From: Tom Lendacky <thomas.lendacky@amd.com>
Extend the vmcb_safe_area with SEV-ES fields and add a new
'struct ghcb' which will be used for guest-hypervisor communication.
Signed-off-by: Tom Lendacky <thomas.lendacky@amd.com>
Signed-off-by: Joerg Roedel <jroedel@suse.de>
---
arch/x86/include/asm/svm.h | 51 ++++++++++++++++++++++++++++++++++++--
arch/x86/kvm/svm/svm.c | 2 ++
2 files changed, 51 insertions(+), 2 deletions(-)
diff --git a/arch/x86/include/asm/svm.h b/arch/x86/include/asm/svm.h
index 8a1f5382a4ea..acac55d6f941 100644
--- a/arch/x86/include/asm/svm.h
+++ b/arch/x86/include/asm/svm.h
@@ -200,13 +200,60 @@ struct __attribute__ ((__packed__)) vmcb_save_area {
u64 br_to;
u64 last_excp_from;
u64 last_excp_to;
+
+ /*
+ * The following part of the save area is valid only for
+ * SEV-ES guests when referenced through the GHCB.
+ */
+ u8 reserved_7[104];
+ u64 reserved_8; /* rax already available at 0x01f8 */
+ u64 rcx;
+ u64 rdx;
+ u64 rbx;
+ u64 reserved_9; /* rsp already available at 0x01d8 */
+ u64 rbp;
+ u64 rsi;
+ u64 rdi;
+ u64 r8;
+ u64 r9;
+ u64 r10;
+ u64 r11;
+ u64 r12;
+ u64 r13;
+ u64 r14;
+ u64 r15;
+ u8 reserved_10[16];
+ u64 sw_exit_code;
+ u64 sw_exit_info_1;
+ u64 sw_exit_info_2;
+ u64 sw_scratch;
+ u8 reserved_11[56];
+ u64 xcr0;
+ u8 valid_bitmap[16];
+ u64 x87_state_gpa;
};
+struct ghcb {
+ struct vmcb_save_area save;
+ u8 reserved_save[2048 - sizeof(struct vmcb_save_area)];
+
+ u8 shared_buffer[2032];
+
+ u8 reserved_1[10];
+ u16 protocol_version; /* negotiated SEV-ES/GHCB protocol version */
+ u32 ghcb_usage;
+} __packed;
+
+
+#define EXPECTED_VMCB_SAVE_AREA_SIZE 1032
+#define EXPECTED_VMCB_CONTROL_AREA_SIZE 256
+#define EXPECTED_GHCB_SIZE PAGE_SIZE
static inline void __unused_size_checks(void)
{
- BUILD_BUG_ON(sizeof(struct vmcb_save_area) != 0x298);
- BUILD_BUG_ON(sizeof(struct vmcb_control_area) != 256);
+ BUILD_BUG_ON(sizeof(struct vmcb_save_area) != EXPECTED_VMCB_SAVE_AREA_SIZE);
+ BUILD_BUG_ON(sizeof(struct vmcb_control_area) != EXPECTED_VMCB_CONTROL_AREA_SIZE);
+ BUILD_BUG_ON(sizeof(struct ghcb) != EXPECTED_GHCB_SIZE);
}
struct __attribute__ ((__packed__)) vmcb {
diff --git a/arch/x86/kvm/svm/svm.c b/arch/x86/kvm/svm/svm.c
index 0194336b64a4..1db4fdcb4906 100644
--- a/arch/x86/kvm/svm/svm.c
+++ b/arch/x86/kvm/svm/svm.c
@@ -4164,6 +4164,8 @@ static struct kvm_x86_init_ops svm_init_ops __initdata = {
static int __init svm_init(void)
{
+ __unused_size_checks();
+
return kvm_init(&svm_init_ops, sizeof(struct vcpu_svm),
__alignof__(struct vcpu_svm), THIS_MODULE);
}
--
2.28.0
_______________________________________________
Virtualization mailing list
Virtualization@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/virtualization
^ permalink raw reply related [flat|nested] 264+ messages in thread
* [PATCH v7 03/72] KVM: SVM: Add GHCB Accessor functions
2020-09-07 13:15 ` Joerg Roedel
@ 2020-09-07 13:15 ` Joerg Roedel
-1 siblings, 0 replies; 264+ messages in thread
From: Joerg Roedel @ 2020-09-07 13:15 UTC (permalink / raw)
To: x86
Cc: Joerg Roedel, Joerg Roedel, hpa, Andy Lutomirski, Dave Hansen,
Peter Zijlstra, Jiri Slaby, Dan Williams, Tom Lendacky,
Juergen Gross, Kees Cook, David Rientjes, Cfir Cohen,
Erdem Aktas, Masami Hiramatsu, Mike Stunes, Sean Christopherson,
Martin Radev, linux-kernel, kvm, virtualization
From: Joerg Roedel <jroedel@suse.de>
Building a correct GHCB for the hypervisor requires setting valid bits
in the GHCB. Simplify that process by providing accessor functions to
set values and to update the valid bitmap and to check the valid bitmap
in KVM.
Signed-off-by: Joerg Roedel <jroedel@suse.de>
---
arch/x86/include/asm/svm.h | 43 ++++++++++++++++++++++++++++++++++++++
1 file changed, 43 insertions(+)
diff --git a/arch/x86/include/asm/svm.h b/arch/x86/include/asm/svm.h
index acac55d6f941..06e52585aed3 100644
--- a/arch/x86/include/asm/svm.h
+++ b/arch/x86/include/asm/svm.h
@@ -345,4 +345,47 @@ struct __attribute__ ((__packed__)) vmcb {
#define SVM_CR0_SELECTIVE_MASK (X86_CR0_TS | X86_CR0_MP)
+/* GHCB Accessor functions */
+
+#define GHCB_BITMAP_IDX(field) \
+ (offsetof(struct vmcb_save_area, field) / sizeof(u64))
+
+#define DEFINE_GHCB_ACCESSORS(field) \
+ static inline bool ghcb_##field##_is_valid(const struct ghcb *ghcb) \
+ { \
+ return test_bit(GHCB_BITMAP_IDX(field), \
+ (unsigned long *)&ghcb->save.valid_bitmap); \
+ } \
+ \
+ static inline void ghcb_set_##field(struct ghcb *ghcb, u64 value) \
+ { \
+ __set_bit(GHCB_BITMAP_IDX(field), \
+ (unsigned long *)&ghcb->save.valid_bitmap); \
+ ghcb->save.field = value; \
+ }
+
+DEFINE_GHCB_ACCESSORS(cpl)
+DEFINE_GHCB_ACCESSORS(rip)
+DEFINE_GHCB_ACCESSORS(rsp)
+DEFINE_GHCB_ACCESSORS(rax)
+DEFINE_GHCB_ACCESSORS(rcx)
+DEFINE_GHCB_ACCESSORS(rdx)
+DEFINE_GHCB_ACCESSORS(rbx)
+DEFINE_GHCB_ACCESSORS(rbp)
+DEFINE_GHCB_ACCESSORS(rsi)
+DEFINE_GHCB_ACCESSORS(rdi)
+DEFINE_GHCB_ACCESSORS(r8)
+DEFINE_GHCB_ACCESSORS(r9)
+DEFINE_GHCB_ACCESSORS(r10)
+DEFINE_GHCB_ACCESSORS(r11)
+DEFINE_GHCB_ACCESSORS(r12)
+DEFINE_GHCB_ACCESSORS(r13)
+DEFINE_GHCB_ACCESSORS(r14)
+DEFINE_GHCB_ACCESSORS(r15)
+DEFINE_GHCB_ACCESSORS(sw_exit_code)
+DEFINE_GHCB_ACCESSORS(sw_exit_info_1)
+DEFINE_GHCB_ACCESSORS(sw_exit_info_2)
+DEFINE_GHCB_ACCESSORS(sw_scratch)
+DEFINE_GHCB_ACCESSORS(xcr0)
+
#endif
--
2.28.0
^ permalink raw reply related [flat|nested] 264+ messages in thread
* [PATCH v7 03/72] KVM: SVM: Add GHCB Accessor functions
@ 2020-09-07 13:15 ` Joerg Roedel
0 siblings, 0 replies; 264+ messages in thread
From: Joerg Roedel @ 2020-09-07 13:15 UTC (permalink / raw)
To: x86
Cc: Juergen Gross, Tom Lendacky, Joerg Roedel, Mike Stunes,
Kees Cook, kvm, Peter Zijlstra, Cfir Cohen, Joerg Roedel,
Dave Hansen, linux-kernel, Sean Christopherson, virtualization,
Martin Radev, Masami Hiramatsu, Andy Lutomirski, hpa,
Erdem Aktas, David Rientjes, Dan Williams, Jiri Slaby
From: Joerg Roedel <jroedel@suse.de>
Building a correct GHCB for the hypervisor requires setting valid bits
in the GHCB. Simplify that process by providing accessor functions to
set values and to update the valid bitmap and to check the valid bitmap
in KVM.
Signed-off-by: Joerg Roedel <jroedel@suse.de>
---
arch/x86/include/asm/svm.h | 43 ++++++++++++++++++++++++++++++++++++++
1 file changed, 43 insertions(+)
diff --git a/arch/x86/include/asm/svm.h b/arch/x86/include/asm/svm.h
index acac55d6f941..06e52585aed3 100644
--- a/arch/x86/include/asm/svm.h
+++ b/arch/x86/include/asm/svm.h
@@ -345,4 +345,47 @@ struct __attribute__ ((__packed__)) vmcb {
#define SVM_CR0_SELECTIVE_MASK (X86_CR0_TS | X86_CR0_MP)
+/* GHCB Accessor functions */
+
+#define GHCB_BITMAP_IDX(field) \
+ (offsetof(struct vmcb_save_area, field) / sizeof(u64))
+
+#define DEFINE_GHCB_ACCESSORS(field) \
+ static inline bool ghcb_##field##_is_valid(const struct ghcb *ghcb) \
+ { \
+ return test_bit(GHCB_BITMAP_IDX(field), \
+ (unsigned long *)&ghcb->save.valid_bitmap); \
+ } \
+ \
+ static inline void ghcb_set_##field(struct ghcb *ghcb, u64 value) \
+ { \
+ __set_bit(GHCB_BITMAP_IDX(field), \
+ (unsigned long *)&ghcb->save.valid_bitmap); \
+ ghcb->save.field = value; \
+ }
+
+DEFINE_GHCB_ACCESSORS(cpl)
+DEFINE_GHCB_ACCESSORS(rip)
+DEFINE_GHCB_ACCESSORS(rsp)
+DEFINE_GHCB_ACCESSORS(rax)
+DEFINE_GHCB_ACCESSORS(rcx)
+DEFINE_GHCB_ACCESSORS(rdx)
+DEFINE_GHCB_ACCESSORS(rbx)
+DEFINE_GHCB_ACCESSORS(rbp)
+DEFINE_GHCB_ACCESSORS(rsi)
+DEFINE_GHCB_ACCESSORS(rdi)
+DEFINE_GHCB_ACCESSORS(r8)
+DEFINE_GHCB_ACCESSORS(r9)
+DEFINE_GHCB_ACCESSORS(r10)
+DEFINE_GHCB_ACCESSORS(r11)
+DEFINE_GHCB_ACCESSORS(r12)
+DEFINE_GHCB_ACCESSORS(r13)
+DEFINE_GHCB_ACCESSORS(r14)
+DEFINE_GHCB_ACCESSORS(r15)
+DEFINE_GHCB_ACCESSORS(sw_exit_code)
+DEFINE_GHCB_ACCESSORS(sw_exit_info_1)
+DEFINE_GHCB_ACCESSORS(sw_exit_info_2)
+DEFINE_GHCB_ACCESSORS(sw_scratch)
+DEFINE_GHCB_ACCESSORS(xcr0)
+
#endif
--
2.28.0
_______________________________________________
Virtualization mailing list
Virtualization@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/virtualization
^ permalink raw reply related [flat|nested] 264+ messages in thread
* [PATCH v7 04/72] KVM: SVM: Use __packed shorthand
2020-09-07 13:15 ` Joerg Roedel
@ 2020-09-07 13:15 ` Joerg Roedel
-1 siblings, 0 replies; 264+ messages in thread
From: Joerg Roedel @ 2020-09-07 13:15 UTC (permalink / raw)
To: x86
Cc: Joerg Roedel, Joerg Roedel, Borislav Petkov, hpa,
Andy Lutomirski, Dave Hansen, Peter Zijlstra, Jiri Slaby,
Dan Williams, Tom Lendacky, Juergen Gross, Kees Cook,
David Rientjes, Cfir Cohen, Erdem Aktas, Masami Hiramatsu,
Mike Stunes, Sean Christopherson, Martin Radev, linux-kernel,
kvm, virtualization
From: Borislav Petkov <bp@alien8.de>
Use the shorthand to make it more readable.
No functional changes.
Signed-off-by: Borislav Petkov <bp@alien8.de>
Signed-off-by: Joerg Roedel <jroedel@suse.de>
---
arch/x86/include/asm/svm.h | 12 ++++++------
1 file changed, 6 insertions(+), 6 deletions(-)
diff --git a/arch/x86/include/asm/svm.h b/arch/x86/include/asm/svm.h
index 06e52585aed3..cf13f9e78585 100644
--- a/arch/x86/include/asm/svm.h
+++ b/arch/x86/include/asm/svm.h
@@ -150,14 +150,14 @@ struct __attribute__ ((__packed__)) vmcb_control_area {
#define SVM_NESTED_CTL_NP_ENABLE BIT(0)
#define SVM_NESTED_CTL_SEV_ENABLE BIT(1)
-struct __attribute__ ((__packed__)) vmcb_seg {
+struct vmcb_seg {
u16 selector;
u16 attrib;
u32 limit;
u64 base;
-};
+} __packed;
-struct __attribute__ ((__packed__)) vmcb_save_area {
+struct vmcb_save_area {
struct vmcb_seg es;
struct vmcb_seg cs;
struct vmcb_seg ss;
@@ -231,7 +231,7 @@ struct __attribute__ ((__packed__)) vmcb_save_area {
u64 xcr0;
u8 valid_bitmap[16];
u64 x87_state_gpa;
-};
+} __packed;
struct ghcb {
struct vmcb_save_area save;
@@ -256,11 +256,11 @@ static inline void __unused_size_checks(void)
BUILD_BUG_ON(sizeof(struct ghcb) != EXPECTED_GHCB_SIZE);
}
-struct __attribute__ ((__packed__)) vmcb {
+struct vmcb {
struct vmcb_control_area control;
u8 reserved_control[1024 - sizeof(struct vmcb_control_area)];
struct vmcb_save_area save;
-};
+} __packed;
#define SVM_CPUID_FUNC 0x8000000a
--
2.28.0
^ permalink raw reply related [flat|nested] 264+ messages in thread
* [PATCH v7 04/72] KVM: SVM: Use __packed shorthand
@ 2020-09-07 13:15 ` Joerg Roedel
0 siblings, 0 replies; 264+ messages in thread
From: Joerg Roedel @ 2020-09-07 13:15 UTC (permalink / raw)
To: x86
Cc: kvm, Peter Zijlstra, Dave Hansen, virtualization, hpa,
Jiri Slaby, Joerg Roedel, David Rientjes, Martin Radev,
Tom Lendacky, Joerg Roedel, Kees Cook, Cfir Cohen,
Borislav Petkov, Andy Lutomirski, Dan Williams, Juergen Gross,
Mike Stunes, linux-kernel, Sean Christopherson, Masami Hiramatsu,
Erdem Aktas
From: Borislav Petkov <bp@alien8.de>
Use the shorthand to make it more readable.
No functional changes.
Signed-off-by: Borislav Petkov <bp@alien8.de>
Signed-off-by: Joerg Roedel <jroedel@suse.de>
---
arch/x86/include/asm/svm.h | 12 ++++++------
1 file changed, 6 insertions(+), 6 deletions(-)
diff --git a/arch/x86/include/asm/svm.h b/arch/x86/include/asm/svm.h
index 06e52585aed3..cf13f9e78585 100644
--- a/arch/x86/include/asm/svm.h
+++ b/arch/x86/include/asm/svm.h
@@ -150,14 +150,14 @@ struct __attribute__ ((__packed__)) vmcb_control_area {
#define SVM_NESTED_CTL_NP_ENABLE BIT(0)
#define SVM_NESTED_CTL_SEV_ENABLE BIT(1)
-struct __attribute__ ((__packed__)) vmcb_seg {
+struct vmcb_seg {
u16 selector;
u16 attrib;
u32 limit;
u64 base;
-};
+} __packed;
-struct __attribute__ ((__packed__)) vmcb_save_area {
+struct vmcb_save_area {
struct vmcb_seg es;
struct vmcb_seg cs;
struct vmcb_seg ss;
@@ -231,7 +231,7 @@ struct __attribute__ ((__packed__)) vmcb_save_area {
u64 xcr0;
u8 valid_bitmap[16];
u64 x87_state_gpa;
-};
+} __packed;
struct ghcb {
struct vmcb_save_area save;
@@ -256,11 +256,11 @@ static inline void __unused_size_checks(void)
BUILD_BUG_ON(sizeof(struct ghcb) != EXPECTED_GHCB_SIZE);
}
-struct __attribute__ ((__packed__)) vmcb {
+struct vmcb {
struct vmcb_control_area control;
u8 reserved_control[1024 - sizeof(struct vmcb_control_area)];
struct vmcb_save_area save;
-};
+} __packed;
#define SVM_CPUID_FUNC 0x8000000a
--
2.28.0
_______________________________________________
Virtualization mailing list
Virtualization@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/virtualization
^ permalink raw reply related [flat|nested] 264+ messages in thread
* [PATCH v7 05/72] x86/cpufeatures: Add SEV-ES CPU feature
2020-09-07 13:15 ` Joerg Roedel
@ 2020-09-07 13:15 ` Joerg Roedel
-1 siblings, 0 replies; 264+ messages in thread
From: Joerg Roedel @ 2020-09-07 13:15 UTC (permalink / raw)
To: x86
Cc: Joerg Roedel, Joerg Roedel, Tom Lendacky, hpa, Andy Lutomirski,
Dave Hansen, Peter Zijlstra, Jiri Slaby, Dan Williams,
Juergen Gross, Kees Cook, David Rientjes, Cfir Cohen,
Erdem Aktas, Masami Hiramatsu, Mike Stunes, Sean Christopherson,
Martin Radev, linux-kernel, kvm, virtualization
From: Tom Lendacky <thomas.lendacky@amd.com>
Add CPU feature detection for Secure Encrypted Virtualization with
Encrypted State. This feature enhances SEV by also encrypting the
guest register state, making it in-accessible to the hypervisor.
Signed-off-by: Tom Lendacky <thomas.lendacky@amd.com>
Signed-off-by: Joerg Roedel <jroedel@suse.de>
---
arch/x86/include/asm/cpufeatures.h | 1 +
arch/x86/kernel/cpu/amd.c | 3 ++-
arch/x86/kernel/cpu/scattered.c | 1 +
3 files changed, 4 insertions(+), 1 deletion(-)
diff --git a/arch/x86/include/asm/cpufeatures.h b/arch/x86/include/asm/cpufeatures.h
index e1013056eb12..0f1da4a6e886 100644
--- a/arch/x86/include/asm/cpufeatures.h
+++ b/arch/x86/include/asm/cpufeatures.h
@@ -236,6 +236,7 @@
#define X86_FEATURE_EPT_AD ( 8*32+17) /* Intel Extended Page Table access-dirty bit */
#define X86_FEATURE_VMCALL ( 8*32+18) /* "" Hypervisor supports the VMCALL instruction */
#define X86_FEATURE_VMW_VMMCALL ( 8*32+19) /* "" VMware prefers VMMCALL hypercall instruction */
+#define X86_FEATURE_SEV_ES ( 8*32+20) /* AMD Secure Encrypted Virtualization - Encrypted State */
/* Intel-defined CPU features, CPUID level 0x00000007:0 (EBX), word 9 */
#define X86_FEATURE_FSGSBASE ( 9*32+ 0) /* RDFSBASE, WRFSBASE, RDGSBASE, WRGSBASE instructions*/
diff --git a/arch/x86/kernel/cpu/amd.c b/arch/x86/kernel/cpu/amd.c
index dcc3d943c68f..6062ce586b95 100644
--- a/arch/x86/kernel/cpu/amd.c
+++ b/arch/x86/kernel/cpu/amd.c
@@ -614,7 +614,7 @@ static void early_detect_mem_encrypt(struct cpuinfo_x86 *c)
* If BIOS has not enabled SME then don't advertise the
* SME feature (set in scattered.c).
* For SEV: If BIOS has not enabled SEV then don't advertise the
- * SEV feature (set in scattered.c).
+ * SEV and SEV_ES feature (set in scattered.c).
*
* In all cases, since support for SME and SEV requires long mode,
* don't advertise the feature under CONFIG_X86_32.
@@ -645,6 +645,7 @@ static void early_detect_mem_encrypt(struct cpuinfo_x86 *c)
setup_clear_cpu_cap(X86_FEATURE_SME);
clear_sev:
setup_clear_cpu_cap(X86_FEATURE_SEV);
+ setup_clear_cpu_cap(X86_FEATURE_SEV_ES);
}
}
diff --git a/arch/x86/kernel/cpu/scattered.c b/arch/x86/kernel/cpu/scattered.c
index bccfc9ff3cc1..bf01a9693c6c 100644
--- a/arch/x86/kernel/cpu/scattered.c
+++ b/arch/x86/kernel/cpu/scattered.c
@@ -42,6 +42,7 @@ static const struct cpuid_bit cpuid_bits[] = {
{ X86_FEATURE_MBA, CPUID_EBX, 6, 0x80000008, 0 },
{ X86_FEATURE_SME, CPUID_EAX, 0, 0x8000001f, 0 },
{ X86_FEATURE_SEV, CPUID_EAX, 1, 0x8000001f, 0 },
+ { X86_FEATURE_SEV_ES, CPUID_EAX, 3, 0x8000001f, 0 },
{ 0, 0, 0, 0, 0 }
};
--
2.28.0
^ permalink raw reply related [flat|nested] 264+ messages in thread
* [PATCH v7 05/72] x86/cpufeatures: Add SEV-ES CPU feature
@ 2020-09-07 13:15 ` Joerg Roedel
0 siblings, 0 replies; 264+ messages in thread
From: Joerg Roedel @ 2020-09-07 13:15 UTC (permalink / raw)
To: x86
Cc: Juergen Gross, Tom Lendacky, Joerg Roedel, Mike Stunes,
Kees Cook, kvm, Peter Zijlstra, Cfir Cohen, Joerg Roedel,
Dave Hansen, linux-kernel, Sean Christopherson, virtualization,
Martin Radev, Masami Hiramatsu, Andy Lutomirski, hpa,
Erdem Aktas, David Rientjes, Dan Williams, Jiri Slaby
From: Tom Lendacky <thomas.lendacky@amd.com>
Add CPU feature detection for Secure Encrypted Virtualization with
Encrypted State. This feature enhances SEV by also encrypting the
guest register state, making it in-accessible to the hypervisor.
Signed-off-by: Tom Lendacky <thomas.lendacky@amd.com>
Signed-off-by: Joerg Roedel <jroedel@suse.de>
---
arch/x86/include/asm/cpufeatures.h | 1 +
arch/x86/kernel/cpu/amd.c | 3 ++-
arch/x86/kernel/cpu/scattered.c | 1 +
3 files changed, 4 insertions(+), 1 deletion(-)
diff --git a/arch/x86/include/asm/cpufeatures.h b/arch/x86/include/asm/cpufeatures.h
index e1013056eb12..0f1da4a6e886 100644
--- a/arch/x86/include/asm/cpufeatures.h
+++ b/arch/x86/include/asm/cpufeatures.h
@@ -236,6 +236,7 @@
#define X86_FEATURE_EPT_AD ( 8*32+17) /* Intel Extended Page Table access-dirty bit */
#define X86_FEATURE_VMCALL ( 8*32+18) /* "" Hypervisor supports the VMCALL instruction */
#define X86_FEATURE_VMW_VMMCALL ( 8*32+19) /* "" VMware prefers VMMCALL hypercall instruction */
+#define X86_FEATURE_SEV_ES ( 8*32+20) /* AMD Secure Encrypted Virtualization - Encrypted State */
/* Intel-defined CPU features, CPUID level 0x00000007:0 (EBX), word 9 */
#define X86_FEATURE_FSGSBASE ( 9*32+ 0) /* RDFSBASE, WRFSBASE, RDGSBASE, WRGSBASE instructions*/
diff --git a/arch/x86/kernel/cpu/amd.c b/arch/x86/kernel/cpu/amd.c
index dcc3d943c68f..6062ce586b95 100644
--- a/arch/x86/kernel/cpu/amd.c
+++ b/arch/x86/kernel/cpu/amd.c
@@ -614,7 +614,7 @@ static void early_detect_mem_encrypt(struct cpuinfo_x86 *c)
* If BIOS has not enabled SME then don't advertise the
* SME feature (set in scattered.c).
* For SEV: If BIOS has not enabled SEV then don't advertise the
- * SEV feature (set in scattered.c).
+ * SEV and SEV_ES feature (set in scattered.c).
*
* In all cases, since support for SME and SEV requires long mode,
* don't advertise the feature under CONFIG_X86_32.
@@ -645,6 +645,7 @@ static void early_detect_mem_encrypt(struct cpuinfo_x86 *c)
setup_clear_cpu_cap(X86_FEATURE_SME);
clear_sev:
setup_clear_cpu_cap(X86_FEATURE_SEV);
+ setup_clear_cpu_cap(X86_FEATURE_SEV_ES);
}
}
diff --git a/arch/x86/kernel/cpu/scattered.c b/arch/x86/kernel/cpu/scattered.c
index bccfc9ff3cc1..bf01a9693c6c 100644
--- a/arch/x86/kernel/cpu/scattered.c
+++ b/arch/x86/kernel/cpu/scattered.c
@@ -42,6 +42,7 @@ static const struct cpuid_bit cpuid_bits[] = {
{ X86_FEATURE_MBA, CPUID_EBX, 6, 0x80000008, 0 },
{ X86_FEATURE_SME, CPUID_EAX, 0, 0x8000001f, 0 },
{ X86_FEATURE_SEV, CPUID_EAX, 1, 0x8000001f, 0 },
+ { X86_FEATURE_SEV_ES, CPUID_EAX, 3, 0x8000001f, 0 },
{ 0, 0, 0, 0, 0 }
};
--
2.28.0
_______________________________________________
Virtualization mailing list
Virtualization@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/virtualization
^ permalink raw reply related [flat|nested] 264+ messages in thread
* [PATCH v7 06/72] x86/traps: Move pf error codes to <asm/trap_pf.h>
2020-09-07 13:15 ` Joerg Roedel
@ 2020-09-07 13:15 ` Joerg Roedel
-1 siblings, 0 replies; 264+ messages in thread
From: Joerg Roedel @ 2020-09-07 13:15 UTC (permalink / raw)
To: x86
Cc: Joerg Roedel, Joerg Roedel, hpa, Andy Lutomirski, Dave Hansen,
Peter Zijlstra, Jiri Slaby, Dan Williams, Tom Lendacky,
Juergen Gross, Kees Cook, David Rientjes, Cfir Cohen,
Erdem Aktas, Masami Hiramatsu, Mike Stunes, Sean Christopherson,
Martin Radev, linux-kernel, kvm, virtualization
From: Joerg Roedel <jroedel@suse.de>
Move the definition of the x86 page-fault error code bits to the new
header file asm/trap_pf.h. This makes it easier to include them into
pre-decompression boot code. No functional changes.
Signed-off-by: Joerg Roedel <jroedel@suse.de>
---
arch/x86/include/asm/trap_pf.h | 24 ++++++++++++++++++++++++
arch/x86/include/asm/traps.h | 19 +------------------
2 files changed, 25 insertions(+), 18 deletions(-)
create mode 100644 arch/x86/include/asm/trap_pf.h
diff --git a/arch/x86/include/asm/trap_pf.h b/arch/x86/include/asm/trap_pf.h
new file mode 100644
index 000000000000..305bc1214aef
--- /dev/null
+++ b/arch/x86/include/asm/trap_pf.h
@@ -0,0 +1,24 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+#ifndef _ASM_X86_TRAP_PF_H
+#define _ASM_X86_TRAP_PF_H
+
+/*
+ * Page fault error code bits:
+ *
+ * bit 0 == 0: no page found 1: protection fault
+ * bit 1 == 0: read access 1: write access
+ * bit 2 == 0: kernel-mode access 1: user-mode access
+ * bit 3 == 1: use of reserved bit detected
+ * bit 4 == 1: fault was an instruction fetch
+ * bit 5 == 1: protection keys block access
+ */
+enum x86_pf_error_code {
+ X86_PF_PROT = 1 << 0,
+ X86_PF_WRITE = 1 << 1,
+ X86_PF_USER = 1 << 2,
+ X86_PF_RSVD = 1 << 3,
+ X86_PF_INSTR = 1 << 4,
+ X86_PF_PK = 1 << 5,
+};
+
+#endif /* _ASM_X86_TRAP_PF_H */
diff --git a/arch/x86/include/asm/traps.h b/arch/x86/include/asm/traps.h
index 714b1a30e7b0..6a308355ea29 100644
--- a/arch/x86/include/asm/traps.h
+++ b/arch/x86/include/asm/traps.h
@@ -8,6 +8,7 @@
#include <asm/debugreg.h>
#include <asm/idtentry.h>
#include <asm/siginfo.h> /* TRAP_TRACE, ... */
+#include <asm/trap_pf.h>
#ifdef CONFIG_X86_64
asmlinkage __visible notrace struct pt_regs *sync_regs(struct pt_regs *eregs);
@@ -41,22 +42,4 @@ void __noreturn handle_stack_overflow(const char *message,
unsigned long fault_address);
#endif
-/*
- * Page fault error code bits:
- *
- * bit 0 == 0: no page found 1: protection fault
- * bit 1 == 0: read access 1: write access
- * bit 2 == 0: kernel-mode access 1: user-mode access
- * bit 3 == 1: use of reserved bit detected
- * bit 4 == 1: fault was an instruction fetch
- * bit 5 == 1: protection keys block access
- */
-enum x86_pf_error_code {
- X86_PF_PROT = 1 << 0,
- X86_PF_WRITE = 1 << 1,
- X86_PF_USER = 1 << 2,
- X86_PF_RSVD = 1 << 3,
- X86_PF_INSTR = 1 << 4,
- X86_PF_PK = 1 << 5,
-};
#endif /* _ASM_X86_TRAPS_H */
--
2.28.0
^ permalink raw reply related [flat|nested] 264+ messages in thread
* [PATCH v7 06/72] x86/traps: Move pf error codes to <asm/trap_pf.h>
@ 2020-09-07 13:15 ` Joerg Roedel
0 siblings, 0 replies; 264+ messages in thread
From: Joerg Roedel @ 2020-09-07 13:15 UTC (permalink / raw)
To: x86
Cc: Juergen Gross, Tom Lendacky, Joerg Roedel, Mike Stunes,
Kees Cook, kvm, Peter Zijlstra, Cfir Cohen, Joerg Roedel,
Dave Hansen, linux-kernel, Sean Christopherson, virtualization,
Martin Radev, Masami Hiramatsu, Andy Lutomirski, hpa,
Erdem Aktas, David Rientjes, Dan Williams, Jiri Slaby
From: Joerg Roedel <jroedel@suse.de>
Move the definition of the x86 page-fault error code bits to the new
header file asm/trap_pf.h. This makes it easier to include them into
pre-decompression boot code. No functional changes.
Signed-off-by: Joerg Roedel <jroedel@suse.de>
---
arch/x86/include/asm/trap_pf.h | 24 ++++++++++++++++++++++++
arch/x86/include/asm/traps.h | 19 +------------------
2 files changed, 25 insertions(+), 18 deletions(-)
create mode 100644 arch/x86/include/asm/trap_pf.h
diff --git a/arch/x86/include/asm/trap_pf.h b/arch/x86/include/asm/trap_pf.h
new file mode 100644
index 000000000000..305bc1214aef
--- /dev/null
+++ b/arch/x86/include/asm/trap_pf.h
@@ -0,0 +1,24 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+#ifndef _ASM_X86_TRAP_PF_H
+#define _ASM_X86_TRAP_PF_H
+
+/*
+ * Page fault error code bits:
+ *
+ * bit 0 == 0: no page found 1: protection fault
+ * bit 1 == 0: read access 1: write access
+ * bit 2 == 0: kernel-mode access 1: user-mode access
+ * bit 3 == 1: use of reserved bit detected
+ * bit 4 == 1: fault was an instruction fetch
+ * bit 5 == 1: protection keys block access
+ */
+enum x86_pf_error_code {
+ X86_PF_PROT = 1 << 0,
+ X86_PF_WRITE = 1 << 1,
+ X86_PF_USER = 1 << 2,
+ X86_PF_RSVD = 1 << 3,
+ X86_PF_INSTR = 1 << 4,
+ X86_PF_PK = 1 << 5,
+};
+
+#endif /* _ASM_X86_TRAP_PF_H */
diff --git a/arch/x86/include/asm/traps.h b/arch/x86/include/asm/traps.h
index 714b1a30e7b0..6a308355ea29 100644
--- a/arch/x86/include/asm/traps.h
+++ b/arch/x86/include/asm/traps.h
@@ -8,6 +8,7 @@
#include <asm/debugreg.h>
#include <asm/idtentry.h>
#include <asm/siginfo.h> /* TRAP_TRACE, ... */
+#include <asm/trap_pf.h>
#ifdef CONFIG_X86_64
asmlinkage __visible notrace struct pt_regs *sync_regs(struct pt_regs *eregs);
@@ -41,22 +42,4 @@ void __noreturn handle_stack_overflow(const char *message,
unsigned long fault_address);
#endif
-/*
- * Page fault error code bits:
- *
- * bit 0 == 0: no page found 1: protection fault
- * bit 1 == 0: read access 1: write access
- * bit 2 == 0: kernel-mode access 1: user-mode access
- * bit 3 == 1: use of reserved bit detected
- * bit 4 == 1: fault was an instruction fetch
- * bit 5 == 1: protection keys block access
- */
-enum x86_pf_error_code {
- X86_PF_PROT = 1 << 0,
- X86_PF_WRITE = 1 << 1,
- X86_PF_USER = 1 << 2,
- X86_PF_RSVD = 1 << 3,
- X86_PF_INSTR = 1 << 4,
- X86_PF_PK = 1 << 5,
-};
#endif /* _ASM_X86_TRAPS_H */
--
2.28.0
_______________________________________________
Virtualization mailing list
Virtualization@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/virtualization
^ permalink raw reply related [flat|nested] 264+ messages in thread
* [PATCH v7 07/72] x86/insn: Make inat-tables.c suitable for pre-decompression code
2020-09-07 13:15 ` Joerg Roedel
@ 2020-09-07 13:15 ` Joerg Roedel
-1 siblings, 0 replies; 264+ messages in thread
From: Joerg Roedel @ 2020-09-07 13:15 UTC (permalink / raw)
To: x86
Cc: Joerg Roedel, Joerg Roedel, Masami Hiramatsu, hpa,
Andy Lutomirski, Dave Hansen, Peter Zijlstra, Jiri Slaby,
Dan Williams, Tom Lendacky, Juergen Gross, Kees Cook,
David Rientjes, Cfir Cohen, Erdem Aktas, Mike Stunes,
Sean Christopherson, Martin Radev, linux-kernel, kvm,
virtualization
From: Joerg Roedel <jroedel@suse.de>
The inat-tables.c file has some arrays in it that contain pointers to
other arrays. These pointers need to be relocated when the kernel
image is moved to a different location.
The pre-decompression boot-code has no support for applying ELF
relocations, so initialize these arrays at runtime in the
pre-decompression code to make sure all pointers are correctly
initialized.
Signed-off-by: Joerg Roedel <jroedel@suse.de>
Acked-by: Masami Hiramatsu <mhiramat@kernel.org>
---
arch/x86/tools/gen-insn-attr-x86.awk | 50 +++++++++++++++++++++-
tools/arch/x86/tools/gen-insn-attr-x86.awk | 50 +++++++++++++++++++++-
2 files changed, 98 insertions(+), 2 deletions(-)
diff --git a/arch/x86/tools/gen-insn-attr-x86.awk b/arch/x86/tools/gen-insn-attr-x86.awk
index a42015b305f4..af38469afd14 100644
--- a/arch/x86/tools/gen-insn-attr-x86.awk
+++ b/arch/x86/tools/gen-insn-attr-x86.awk
@@ -362,6 +362,9 @@ function convert_operands(count,opnd, i,j,imm,mod)
END {
if (awkchecked != "")
exit 1
+
+ print "#ifndef __BOOT_COMPRESSED\n"
+
# print escape opcode map's array
print "/* Escape opcode map array */"
print "const insn_attr_t * const inat_escape_tables[INAT_ESC_MAX + 1]" \
@@ -388,6 +391,51 @@ END {
for (j = 0; j < max_lprefix; j++)
if (atable[i,j])
print " ["i"]["j"] = "atable[i,j]","
- print "};"
+ print "};\n"
+
+ print "#else /* !__BOOT_COMPRESSED */\n"
+
+ print "/* Escape opcode map array */"
+ print "static const insn_attr_t *inat_escape_tables[INAT_ESC_MAX + 1]" \
+ "[INAT_LSTPFX_MAX + 1];"
+ print ""
+
+ print "/* Group opcode map array */"
+ print "static const insn_attr_t *inat_group_tables[INAT_GRP_MAX + 1]"\
+ "[INAT_LSTPFX_MAX + 1];"
+ print ""
+
+ print "/* AVX opcode map array */"
+ print "static const insn_attr_t *inat_avx_tables[X86_VEX_M_MAX + 1]"\
+ "[INAT_LSTPFX_MAX + 1];"
+ print ""
+
+ print "static void inat_init_tables(void)"
+ print "{"
+
+ # print escape opcode map's array
+ print "\t/* Print Escape opcode map array */"
+ for (i = 0; i < geid; i++)
+ for (j = 0; j < max_lprefix; j++)
+ if (etable[i,j])
+ print "\tinat_escape_tables["i"]["j"] = "etable[i,j]";"
+ print ""
+
+ # print group opcode map's array
+ print "\t/* Print Group opcode map array */"
+ for (i = 0; i < ggid; i++)
+ for (j = 0; j < max_lprefix; j++)
+ if (gtable[i,j])
+ print "\tinat_group_tables["i"]["j"] = "gtable[i,j]";"
+ print ""
+ # print AVX opcode map's array
+ print "\t/* Print AVX opcode map array */"
+ for (i = 0; i < gaid; i++)
+ for (j = 0; j < max_lprefix; j++)
+ if (atable[i,j])
+ print "\tinat_avx_tables["i"]["j"] = "atable[i,j]";"
+
+ print "}"
+ print "#endif"
}
diff --git a/tools/arch/x86/tools/gen-insn-attr-x86.awk b/tools/arch/x86/tools/gen-insn-attr-x86.awk
index a42015b305f4..af38469afd14 100644
--- a/tools/arch/x86/tools/gen-insn-attr-x86.awk
+++ b/tools/arch/x86/tools/gen-insn-attr-x86.awk
@@ -362,6 +362,9 @@ function convert_operands(count,opnd, i,j,imm,mod)
END {
if (awkchecked != "")
exit 1
+
+ print "#ifndef __BOOT_COMPRESSED\n"
+
# print escape opcode map's array
print "/* Escape opcode map array */"
print "const insn_attr_t * const inat_escape_tables[INAT_ESC_MAX + 1]" \
@@ -388,6 +391,51 @@ END {
for (j = 0; j < max_lprefix; j++)
if (atable[i,j])
print " ["i"]["j"] = "atable[i,j]","
- print "};"
+ print "};\n"
+
+ print "#else /* !__BOOT_COMPRESSED */\n"
+
+ print "/* Escape opcode map array */"
+ print "static const insn_attr_t *inat_escape_tables[INAT_ESC_MAX + 1]" \
+ "[INAT_LSTPFX_MAX + 1];"
+ print ""
+
+ print "/* Group opcode map array */"
+ print "static const insn_attr_t *inat_group_tables[INAT_GRP_MAX + 1]"\
+ "[INAT_LSTPFX_MAX + 1];"
+ print ""
+
+ print "/* AVX opcode map array */"
+ print "static const insn_attr_t *inat_avx_tables[X86_VEX_M_MAX + 1]"\
+ "[INAT_LSTPFX_MAX + 1];"
+ print ""
+
+ print "static void inat_init_tables(void)"
+ print "{"
+
+ # print escape opcode map's array
+ print "\t/* Print Escape opcode map array */"
+ for (i = 0; i < geid; i++)
+ for (j = 0; j < max_lprefix; j++)
+ if (etable[i,j])
+ print "\tinat_escape_tables["i"]["j"] = "etable[i,j]";"
+ print ""
+
+ # print group opcode map's array
+ print "\t/* Print Group opcode map array */"
+ for (i = 0; i < ggid; i++)
+ for (j = 0; j < max_lprefix; j++)
+ if (gtable[i,j])
+ print "\tinat_group_tables["i"]["j"] = "gtable[i,j]";"
+ print ""
+ # print AVX opcode map's array
+ print "\t/* Print AVX opcode map array */"
+ for (i = 0; i < gaid; i++)
+ for (j = 0; j < max_lprefix; j++)
+ if (atable[i,j])
+ print "\tinat_avx_tables["i"]["j"] = "atable[i,j]";"
+
+ print "}"
+ print "#endif"
}
--
2.28.0
^ permalink raw reply related [flat|nested] 264+ messages in thread
* [PATCH v7 07/72] x86/insn: Make inat-tables.c suitable for pre-decompression code
@ 2020-09-07 13:15 ` Joerg Roedel
0 siblings, 0 replies; 264+ messages in thread
From: Joerg Roedel @ 2020-09-07 13:15 UTC (permalink / raw)
To: x86
Cc: Juergen Gross, Tom Lendacky, Joerg Roedel, Mike Stunes,
Kees Cook, kvm, Peter Zijlstra, Cfir Cohen, Joerg Roedel,
Dave Hansen, linux-kernel, Sean Christopherson, virtualization,
Martin Radev, Masami Hiramatsu, Andy Lutomirski, hpa,
Erdem Aktas, David Rientjes, Dan Williams, Jiri Slaby
From: Joerg Roedel <jroedel@suse.de>
The inat-tables.c file has some arrays in it that contain pointers to
other arrays. These pointers need to be relocated when the kernel
image is moved to a different location.
The pre-decompression boot-code has no support for applying ELF
relocations, so initialize these arrays at runtime in the
pre-decompression code to make sure all pointers are correctly
initialized.
Signed-off-by: Joerg Roedel <jroedel@suse.de>
Acked-by: Masami Hiramatsu <mhiramat@kernel.org>
---
arch/x86/tools/gen-insn-attr-x86.awk | 50 +++++++++++++++++++++-
tools/arch/x86/tools/gen-insn-attr-x86.awk | 50 +++++++++++++++++++++-
2 files changed, 98 insertions(+), 2 deletions(-)
diff --git a/arch/x86/tools/gen-insn-attr-x86.awk b/arch/x86/tools/gen-insn-attr-x86.awk
index a42015b305f4..af38469afd14 100644
--- a/arch/x86/tools/gen-insn-attr-x86.awk
+++ b/arch/x86/tools/gen-insn-attr-x86.awk
@@ -362,6 +362,9 @@ function convert_operands(count,opnd, i,j,imm,mod)
END {
if (awkchecked != "")
exit 1
+
+ print "#ifndef __BOOT_COMPRESSED\n"
+
# print escape opcode map's array
print "/* Escape opcode map array */"
print "const insn_attr_t * const inat_escape_tables[INAT_ESC_MAX + 1]" \
@@ -388,6 +391,51 @@ END {
for (j = 0; j < max_lprefix; j++)
if (atable[i,j])
print " ["i"]["j"] = "atable[i,j]","
- print "};"
+ print "};\n"
+
+ print "#else /* !__BOOT_COMPRESSED */\n"
+
+ print "/* Escape opcode map array */"
+ print "static const insn_attr_t *inat_escape_tables[INAT_ESC_MAX + 1]" \
+ "[INAT_LSTPFX_MAX + 1];"
+ print ""
+
+ print "/* Group opcode map array */"
+ print "static const insn_attr_t *inat_group_tables[INAT_GRP_MAX + 1]"\
+ "[INAT_LSTPFX_MAX + 1];"
+ print ""
+
+ print "/* AVX opcode map array */"
+ print "static const insn_attr_t *inat_avx_tables[X86_VEX_M_MAX + 1]"\
+ "[INAT_LSTPFX_MAX + 1];"
+ print ""
+
+ print "static void inat_init_tables(void)"
+ print "{"
+
+ # print escape opcode map's array
+ print "\t/* Print Escape opcode map array */"
+ for (i = 0; i < geid; i++)
+ for (j = 0; j < max_lprefix; j++)
+ if (etable[i,j])
+ print "\tinat_escape_tables["i"]["j"] = "etable[i,j]";"
+ print ""
+
+ # print group opcode map's array
+ print "\t/* Print Group opcode map array */"
+ for (i = 0; i < ggid; i++)
+ for (j = 0; j < max_lprefix; j++)
+ if (gtable[i,j])
+ print "\tinat_group_tables["i"]["j"] = "gtable[i,j]";"
+ print ""
+ # print AVX opcode map's array
+ print "\t/* Print AVX opcode map array */"
+ for (i = 0; i < gaid; i++)
+ for (j = 0; j < max_lprefix; j++)
+ if (atable[i,j])
+ print "\tinat_avx_tables["i"]["j"] = "atable[i,j]";"
+
+ print "}"
+ print "#endif"
}
diff --git a/tools/arch/x86/tools/gen-insn-attr-x86.awk b/tools/arch/x86/tools/gen-insn-attr-x86.awk
index a42015b305f4..af38469afd14 100644
--- a/tools/arch/x86/tools/gen-insn-attr-x86.awk
+++ b/tools/arch/x86/tools/gen-insn-attr-x86.awk
@@ -362,6 +362,9 @@ function convert_operands(count,opnd, i,j,imm,mod)
END {
if (awkchecked != "")
exit 1
+
+ print "#ifndef __BOOT_COMPRESSED\n"
+
# print escape opcode map's array
print "/* Escape opcode map array */"
print "const insn_attr_t * const inat_escape_tables[INAT_ESC_MAX + 1]" \
@@ -388,6 +391,51 @@ END {
for (j = 0; j < max_lprefix; j++)
if (atable[i,j])
print " ["i"]["j"] = "atable[i,j]","
- print "};"
+ print "};\n"
+
+ print "#else /* !__BOOT_COMPRESSED */\n"
+
+ print "/* Escape opcode map array */"
+ print "static const insn_attr_t *inat_escape_tables[INAT_ESC_MAX + 1]" \
+ "[INAT_LSTPFX_MAX + 1];"
+ print ""
+
+ print "/* Group opcode map array */"
+ print "static const insn_attr_t *inat_group_tables[INAT_GRP_MAX + 1]"\
+ "[INAT_LSTPFX_MAX + 1];"
+ print ""
+
+ print "/* AVX opcode map array */"
+ print "static const insn_attr_t *inat_avx_tables[X86_VEX_M_MAX + 1]"\
+ "[INAT_LSTPFX_MAX + 1];"
+ print ""
+
+ print "static void inat_init_tables(void)"
+ print "{"
+
+ # print escape opcode map's array
+ print "\t/* Print Escape opcode map array */"
+ for (i = 0; i < geid; i++)
+ for (j = 0; j < max_lprefix; j++)
+ if (etable[i,j])
+ print "\tinat_escape_tables["i"]["j"] = "etable[i,j]";"
+ print ""
+
+ # print group opcode map's array
+ print "\t/* Print Group opcode map array */"
+ for (i = 0; i < ggid; i++)
+ for (j = 0; j < max_lprefix; j++)
+ if (gtable[i,j])
+ print "\tinat_group_tables["i"]["j"] = "gtable[i,j]";"
+ print ""
+ # print AVX opcode map's array
+ print "\t/* Print AVX opcode map array */"
+ for (i = 0; i < gaid; i++)
+ for (j = 0; j < max_lprefix; j++)
+ if (atable[i,j])
+ print "\tinat_avx_tables["i"]["j"] = "atable[i,j]";"
+
+ print "}"
+ print "#endif"
}
--
2.28.0
_______________________________________________
Virtualization mailing list
Virtualization@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/virtualization
^ permalink raw reply related [flat|nested] 264+ messages in thread
* [PATCH v7 08/72] x86/umip: Factor out instruction fetch
2020-09-07 13:15 ` Joerg Roedel
@ 2020-09-07 13:15 ` Joerg Roedel
-1 siblings, 0 replies; 264+ messages in thread
From: Joerg Roedel @ 2020-09-07 13:15 UTC (permalink / raw)
To: x86
Cc: Joerg Roedel, Joerg Roedel, hpa, Andy Lutomirski, Dave Hansen,
Peter Zijlstra, Jiri Slaby, Dan Williams, Tom Lendacky,
Juergen Gross, Kees Cook, David Rientjes, Cfir Cohen,
Erdem Aktas, Masami Hiramatsu, Mike Stunes, Sean Christopherson,
Martin Radev, linux-kernel, kvm, virtualization
From: Joerg Roedel <jroedel@suse.de>
Factor out the code to fetch the instruction from user-space to a helper
function.
No functional changes.
Signed-off-by: Joerg Roedel <jroedel@suse.de>
---
arch/x86/include/asm/insn-eval.h | 2 ++
arch/x86/kernel/umip.c | 26 +++++-----------------
arch/x86/lib/insn-eval.c | 38 ++++++++++++++++++++++++++++++++
3 files changed, 46 insertions(+), 20 deletions(-)
diff --git a/arch/x86/include/asm/insn-eval.h b/arch/x86/include/asm/insn-eval.h
index 2b6ccf2c49f1..b8b9ef1bbd06 100644
--- a/arch/x86/include/asm/insn-eval.h
+++ b/arch/x86/include/asm/insn-eval.h
@@ -19,5 +19,7 @@ void __user *insn_get_addr_ref(struct insn *insn, struct pt_regs *regs);
int insn_get_modrm_rm_off(struct insn *insn, struct pt_regs *regs);
unsigned long insn_get_seg_base(struct pt_regs *regs, int seg_reg_idx);
int insn_get_code_seg_params(struct pt_regs *regs);
+int insn_fetch_from_user(struct pt_regs *regs,
+ unsigned char buf[MAX_INSN_SIZE]);
#endif /* _ASM_X86_INSN_EVAL_H */
diff --git a/arch/x86/kernel/umip.c b/arch/x86/kernel/umip.c
index 2c304fd0bb1a..ad135be4f1f0 100644
--- a/arch/x86/kernel/umip.c
+++ b/arch/x86/kernel/umip.c
@@ -335,11 +335,11 @@ static void force_sig_info_umip_fault(void __user *addr, struct pt_regs *regs)
*/
bool fixup_umip_exception(struct pt_regs *regs)
{
- int not_copied, nr_copied, reg_offset, dummy_data_size, umip_inst;
- unsigned long seg_base = 0, *reg_addr;
+ int nr_copied, reg_offset, dummy_data_size, umip_inst;
/* 10 bytes is the maximum size of the result of UMIP instructions */
unsigned char dummy_data[10] = { 0 };
unsigned char buf[MAX_INSN_SIZE];
+ unsigned long *reg_addr;
void __user *uaddr;
struct insn insn;
int seg_defs;
@@ -347,26 +347,12 @@ bool fixup_umip_exception(struct pt_regs *regs)
if (!regs)
return false;
- /*
- * If not in user-space long mode, a custom code segment could be in
- * use. This is true in protected mode (if the process defined a local
- * descriptor table), or virtual-8086 mode. In most of the cases
- * seg_base will be zero as in USER_CS.
- */
- if (!user_64bit_mode(regs))
- seg_base = insn_get_seg_base(regs, INAT_SEG_REG_CS);
-
- if (seg_base == -1L)
- return false;
-
- not_copied = copy_from_user(buf, (void __user *)(seg_base + regs->ip),
- sizeof(buf));
- nr_copied = sizeof(buf) - not_copied;
+ nr_copied = insn_fetch_from_user(regs, buf);
/*
- * The copy_from_user above could have failed if user code is protected
- * by a memory protection key. Give up on emulation in such a case.
- * Should we issue a page fault?
+ * The insn_fetch_from_user above could have failed if user code
+ * is protected by a memory protection key. Give up on emulation
+ * in such a case. Should we issue a page fault?
*/
if (!nr_copied)
return false;
diff --git a/arch/x86/lib/insn-eval.c b/arch/x86/lib/insn-eval.c
index 5e69603ff63f..947b7f1a0042 100644
--- a/arch/x86/lib/insn-eval.c
+++ b/arch/x86/lib/insn-eval.c
@@ -1367,3 +1367,41 @@ void __user *insn_get_addr_ref(struct insn *insn, struct pt_regs *regs)
return (void __user *)-1L;
}
}
+
+/**
+ * insn_fetch_from_user() - Copy instruction bytes from user-space memory
+ * @regs: Structure with register values as seen when entering kernel mode
+ * @buf: Array to store the fetched instruction
+ *
+ * Gets the linear address of the instruction and copies the instruction bytes
+ * to the buf.
+ *
+ * Returns:
+ *
+ * Number of instruction bytes copied.
+ *
+ * 0 if nothing was copied.
+ */
+int insn_fetch_from_user(struct pt_regs *regs, unsigned char buf[MAX_INSN_SIZE])
+{
+ unsigned long seg_base = 0;
+ int not_copied;
+
+ /*
+ * If not in user-space long mode, a custom code segment could be in
+ * use. This is true in protected mode (if the process defined a local
+ * descriptor table), or virtual-8086 mode. In most of the cases
+ * seg_base will be zero as in USER_CS.
+ */
+ if (!user_64bit_mode(regs)) {
+ seg_base = insn_get_seg_base(regs, INAT_SEG_REG_CS);
+ if (seg_base == -1L)
+ return 0;
+ }
+
+
+ not_copied = copy_from_user(buf, (void __user *)(seg_base + regs->ip),
+ MAX_INSN_SIZE);
+
+ return MAX_INSN_SIZE - not_copied;
+}
--
2.28.0
^ permalink raw reply related [flat|nested] 264+ messages in thread
* [PATCH v7 08/72] x86/umip: Factor out instruction fetch
@ 2020-09-07 13:15 ` Joerg Roedel
0 siblings, 0 replies; 264+ messages in thread
From: Joerg Roedel @ 2020-09-07 13:15 UTC (permalink / raw)
To: x86
Cc: Juergen Gross, Tom Lendacky, Joerg Roedel, Mike Stunes,
Kees Cook, kvm, Peter Zijlstra, Cfir Cohen, Joerg Roedel,
Dave Hansen, linux-kernel, Sean Christopherson, virtualization,
Martin Radev, Masami Hiramatsu, Andy Lutomirski, hpa,
Erdem Aktas, David Rientjes, Dan Williams, Jiri Slaby
From: Joerg Roedel <jroedel@suse.de>
Factor out the code to fetch the instruction from user-space to a helper
function.
No functional changes.
Signed-off-by: Joerg Roedel <jroedel@suse.de>
---
arch/x86/include/asm/insn-eval.h | 2 ++
arch/x86/kernel/umip.c | 26 +++++-----------------
arch/x86/lib/insn-eval.c | 38 ++++++++++++++++++++++++++++++++
3 files changed, 46 insertions(+), 20 deletions(-)
diff --git a/arch/x86/include/asm/insn-eval.h b/arch/x86/include/asm/insn-eval.h
index 2b6ccf2c49f1..b8b9ef1bbd06 100644
--- a/arch/x86/include/asm/insn-eval.h
+++ b/arch/x86/include/asm/insn-eval.h
@@ -19,5 +19,7 @@ void __user *insn_get_addr_ref(struct insn *insn, struct pt_regs *regs);
int insn_get_modrm_rm_off(struct insn *insn, struct pt_regs *regs);
unsigned long insn_get_seg_base(struct pt_regs *regs, int seg_reg_idx);
int insn_get_code_seg_params(struct pt_regs *regs);
+int insn_fetch_from_user(struct pt_regs *regs,
+ unsigned char buf[MAX_INSN_SIZE]);
#endif /* _ASM_X86_INSN_EVAL_H */
diff --git a/arch/x86/kernel/umip.c b/arch/x86/kernel/umip.c
index 2c304fd0bb1a..ad135be4f1f0 100644
--- a/arch/x86/kernel/umip.c
+++ b/arch/x86/kernel/umip.c
@@ -335,11 +335,11 @@ static void force_sig_info_umip_fault(void __user *addr, struct pt_regs *regs)
*/
bool fixup_umip_exception(struct pt_regs *regs)
{
- int not_copied, nr_copied, reg_offset, dummy_data_size, umip_inst;
- unsigned long seg_base = 0, *reg_addr;
+ int nr_copied, reg_offset, dummy_data_size, umip_inst;
/* 10 bytes is the maximum size of the result of UMIP instructions */
unsigned char dummy_data[10] = { 0 };
unsigned char buf[MAX_INSN_SIZE];
+ unsigned long *reg_addr;
void __user *uaddr;
struct insn insn;
int seg_defs;
@@ -347,26 +347,12 @@ bool fixup_umip_exception(struct pt_regs *regs)
if (!regs)
return false;
- /*
- * If not in user-space long mode, a custom code segment could be in
- * use. This is true in protected mode (if the process defined a local
- * descriptor table), or virtual-8086 mode. In most of the cases
- * seg_base will be zero as in USER_CS.
- */
- if (!user_64bit_mode(regs))
- seg_base = insn_get_seg_base(regs, INAT_SEG_REG_CS);
-
- if (seg_base == -1L)
- return false;
-
- not_copied = copy_from_user(buf, (void __user *)(seg_base + regs->ip),
- sizeof(buf));
- nr_copied = sizeof(buf) - not_copied;
+ nr_copied = insn_fetch_from_user(regs, buf);
/*
- * The copy_from_user above could have failed if user code is protected
- * by a memory protection key. Give up on emulation in such a case.
- * Should we issue a page fault?
+ * The insn_fetch_from_user above could have failed if user code
+ * is protected by a memory protection key. Give up on emulation
+ * in such a case. Should we issue a page fault?
*/
if (!nr_copied)
return false;
diff --git a/arch/x86/lib/insn-eval.c b/arch/x86/lib/insn-eval.c
index 5e69603ff63f..947b7f1a0042 100644
--- a/arch/x86/lib/insn-eval.c
+++ b/arch/x86/lib/insn-eval.c
@@ -1367,3 +1367,41 @@ void __user *insn_get_addr_ref(struct insn *insn, struct pt_regs *regs)
return (void __user *)-1L;
}
}
+
+/**
+ * insn_fetch_from_user() - Copy instruction bytes from user-space memory
+ * @regs: Structure with register values as seen when entering kernel mode
+ * @buf: Array to store the fetched instruction
+ *
+ * Gets the linear address of the instruction and copies the instruction bytes
+ * to the buf.
+ *
+ * Returns:
+ *
+ * Number of instruction bytes copied.
+ *
+ * 0 if nothing was copied.
+ */
+int insn_fetch_from_user(struct pt_regs *regs, unsigned char buf[MAX_INSN_SIZE])
+{
+ unsigned long seg_base = 0;
+ int not_copied;
+
+ /*
+ * If not in user-space long mode, a custom code segment could be in
+ * use. This is true in protected mode (if the process defined a local
+ * descriptor table), or virtual-8086 mode. In most of the cases
+ * seg_base will be zero as in USER_CS.
+ */
+ if (!user_64bit_mode(regs)) {
+ seg_base = insn_get_seg_base(regs, INAT_SEG_REG_CS);
+ if (seg_base == -1L)
+ return 0;
+ }
+
+
+ not_copied = copy_from_user(buf, (void __user *)(seg_base + regs->ip),
+ MAX_INSN_SIZE);
+
+ return MAX_INSN_SIZE - not_copied;
+}
--
2.28.0
_______________________________________________
Virtualization mailing list
Virtualization@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/virtualization
^ permalink raw reply related [flat|nested] 264+ messages in thread
* [PATCH v7 09/72] x86/umip: Factor out instruction decoding
2020-09-07 13:15 ` Joerg Roedel
@ 2020-09-07 13:15 ` Joerg Roedel
-1 siblings, 0 replies; 264+ messages in thread
From: Joerg Roedel @ 2020-09-07 13:15 UTC (permalink / raw)
To: x86
Cc: Joerg Roedel, Joerg Roedel, hpa, Andy Lutomirski, Dave Hansen,
Peter Zijlstra, Jiri Slaby, Dan Williams, Tom Lendacky,
Juergen Gross, Kees Cook, David Rientjes, Cfir Cohen,
Erdem Aktas, Masami Hiramatsu, Mike Stunes, Sean Christopherson,
Martin Radev, linux-kernel, kvm, virtualization
From: Joerg Roedel <jroedel@suse.de>
Factor out the code used to decode an instruction with the correct
address and operand sizes to a helper function.
No functional changes.
Signed-off-by: Joerg Roedel <jroedel@suse.de>
---
arch/x86/include/asm/insn-eval.h | 2 ++
arch/x86/kernel/umip.c | 23 +---------------
arch/x86/lib/insn-eval.c | 45 ++++++++++++++++++++++++++++++++
3 files changed, 48 insertions(+), 22 deletions(-)
diff --git a/arch/x86/include/asm/insn-eval.h b/arch/x86/include/asm/insn-eval.h
index b8b9ef1bbd06..392b4fe377f9 100644
--- a/arch/x86/include/asm/insn-eval.h
+++ b/arch/x86/include/asm/insn-eval.h
@@ -21,5 +21,7 @@ unsigned long insn_get_seg_base(struct pt_regs *regs, int seg_reg_idx);
int insn_get_code_seg_params(struct pt_regs *regs);
int insn_fetch_from_user(struct pt_regs *regs,
unsigned char buf[MAX_INSN_SIZE]);
+bool insn_decode(struct insn *insn, struct pt_regs *regs,
+ unsigned char buf[MAX_INSN_SIZE], int buf_size);
#endif /* _ASM_X86_INSN_EVAL_H */
diff --git a/arch/x86/kernel/umip.c b/arch/x86/kernel/umip.c
index ad135be4f1f0..f6225bf22c02 100644
--- a/arch/x86/kernel/umip.c
+++ b/arch/x86/kernel/umip.c
@@ -342,7 +342,6 @@ bool fixup_umip_exception(struct pt_regs *regs)
unsigned long *reg_addr;
void __user *uaddr;
struct insn insn;
- int seg_defs;
if (!regs)
return false;
@@ -357,27 +356,7 @@ bool fixup_umip_exception(struct pt_regs *regs)
if (!nr_copied)
return false;
- insn_init(&insn, buf, nr_copied, user_64bit_mode(regs));
-
- /*
- * Override the default operand and address sizes with what is specified
- * in the code segment descriptor. The instruction decoder only sets
- * the address size it to either 4 or 8 address bytes and does nothing
- * for the operand bytes. This OK for most of the cases, but we could
- * have special cases where, for instance, a 16-bit code segment
- * descriptor is used.
- * If there is an address override prefix, the instruction decoder
- * correctly updates these values, even for 16-bit defaults.
- */
- seg_defs = insn_get_code_seg_params(regs);
- if (seg_defs == -EINVAL)
- return false;
-
- insn.addr_bytes = INSN_CODE_SEG_ADDR_SZ(seg_defs);
- insn.opnd_bytes = INSN_CODE_SEG_OPND_SZ(seg_defs);
-
- insn_get_length(&insn);
- if (nr_copied < insn.length)
+ if (!insn_decode(&insn, regs, buf, nr_copied))
return false;
umip_inst = identify_insn(&insn);
diff --git a/arch/x86/lib/insn-eval.c b/arch/x86/lib/insn-eval.c
index 947b7f1a0042..2323c85132cf 100644
--- a/arch/x86/lib/insn-eval.c
+++ b/arch/x86/lib/insn-eval.c
@@ -1405,3 +1405,48 @@ int insn_fetch_from_user(struct pt_regs *regs, unsigned char buf[MAX_INSN_SIZE])
return MAX_INSN_SIZE - not_copied;
}
+
+/**
+ * insn_decode() - Decode an instruction
+ * @insn: Structure to store decoded instruction
+ * @regs: Structure with register values as seen when entering kernel mode
+ * @buf: Buffer containing the instruction bytes
+ * @buf_size: Number of instruction bytes available in buf
+ *
+ * Decodes the instruction provided in buf and stores the decoding results in
+ * insn. Also determines the correct address and operand sizes.
+ *
+ * Returns:
+ *
+ * True if instruction was decoded, False otherwise.
+ */
+bool insn_decode(struct insn *insn, struct pt_regs *regs,
+ unsigned char buf[MAX_INSN_SIZE], int buf_size)
+{
+ int seg_defs;
+
+ insn_init(insn, buf, buf_size, user_64bit_mode(regs));
+
+ /*
+ * Override the default operand and address sizes with what is specified
+ * in the code segment descriptor. The instruction decoder only sets
+ * the address size it to either 4 or 8 address bytes and does nothing
+ * for the operand bytes. This OK for most of the cases, but we could
+ * have special cases where, for instance, a 16-bit code segment
+ * descriptor is used.
+ * If there is an address override prefix, the instruction decoder
+ * correctly updates these values, even for 16-bit defaults.
+ */
+ seg_defs = insn_get_code_seg_params(regs);
+ if (seg_defs == -EINVAL)
+ return false;
+
+ insn->addr_bytes = INSN_CODE_SEG_ADDR_SZ(seg_defs);
+ insn->opnd_bytes = INSN_CODE_SEG_OPND_SZ(seg_defs);
+
+ insn_get_length(insn);
+ if (buf_size < insn->length)
+ return false;
+
+ return true;
+}
--
2.28.0
^ permalink raw reply related [flat|nested] 264+ messages in thread
* [PATCH v7 09/72] x86/umip: Factor out instruction decoding
@ 2020-09-07 13:15 ` Joerg Roedel
0 siblings, 0 replies; 264+ messages in thread
From: Joerg Roedel @ 2020-09-07 13:15 UTC (permalink / raw)
To: x86
Cc: Juergen Gross, Tom Lendacky, Joerg Roedel, Mike Stunes,
Kees Cook, kvm, Peter Zijlstra, Cfir Cohen, Joerg Roedel,
Dave Hansen, linux-kernel, Sean Christopherson, virtualization,
Martin Radev, Masami Hiramatsu, Andy Lutomirski, hpa,
Erdem Aktas, David Rientjes, Dan Williams, Jiri Slaby
From: Joerg Roedel <jroedel@suse.de>
Factor out the code used to decode an instruction with the correct
address and operand sizes to a helper function.
No functional changes.
Signed-off-by: Joerg Roedel <jroedel@suse.de>
---
arch/x86/include/asm/insn-eval.h | 2 ++
arch/x86/kernel/umip.c | 23 +---------------
arch/x86/lib/insn-eval.c | 45 ++++++++++++++++++++++++++++++++
3 files changed, 48 insertions(+), 22 deletions(-)
diff --git a/arch/x86/include/asm/insn-eval.h b/arch/x86/include/asm/insn-eval.h
index b8b9ef1bbd06..392b4fe377f9 100644
--- a/arch/x86/include/asm/insn-eval.h
+++ b/arch/x86/include/asm/insn-eval.h
@@ -21,5 +21,7 @@ unsigned long insn_get_seg_base(struct pt_regs *regs, int seg_reg_idx);
int insn_get_code_seg_params(struct pt_regs *regs);
int insn_fetch_from_user(struct pt_regs *regs,
unsigned char buf[MAX_INSN_SIZE]);
+bool insn_decode(struct insn *insn, struct pt_regs *regs,
+ unsigned char buf[MAX_INSN_SIZE], int buf_size);
#endif /* _ASM_X86_INSN_EVAL_H */
diff --git a/arch/x86/kernel/umip.c b/arch/x86/kernel/umip.c
index ad135be4f1f0..f6225bf22c02 100644
--- a/arch/x86/kernel/umip.c
+++ b/arch/x86/kernel/umip.c
@@ -342,7 +342,6 @@ bool fixup_umip_exception(struct pt_regs *regs)
unsigned long *reg_addr;
void __user *uaddr;
struct insn insn;
- int seg_defs;
if (!regs)
return false;
@@ -357,27 +356,7 @@ bool fixup_umip_exception(struct pt_regs *regs)
if (!nr_copied)
return false;
- insn_init(&insn, buf, nr_copied, user_64bit_mode(regs));
-
- /*
- * Override the default operand and address sizes with what is specified
- * in the code segment descriptor. The instruction decoder only sets
- * the address size it to either 4 or 8 address bytes and does nothing
- * for the operand bytes. This OK for most of the cases, but we could
- * have special cases where, for instance, a 16-bit code segment
- * descriptor is used.
- * If there is an address override prefix, the instruction decoder
- * correctly updates these values, even for 16-bit defaults.
- */
- seg_defs = insn_get_code_seg_params(regs);
- if (seg_defs == -EINVAL)
- return false;
-
- insn.addr_bytes = INSN_CODE_SEG_ADDR_SZ(seg_defs);
- insn.opnd_bytes = INSN_CODE_SEG_OPND_SZ(seg_defs);
-
- insn_get_length(&insn);
- if (nr_copied < insn.length)
+ if (!insn_decode(&insn, regs, buf, nr_copied))
return false;
umip_inst = identify_insn(&insn);
diff --git a/arch/x86/lib/insn-eval.c b/arch/x86/lib/insn-eval.c
index 947b7f1a0042..2323c85132cf 100644
--- a/arch/x86/lib/insn-eval.c
+++ b/arch/x86/lib/insn-eval.c
@@ -1405,3 +1405,48 @@ int insn_fetch_from_user(struct pt_regs *regs, unsigned char buf[MAX_INSN_SIZE])
return MAX_INSN_SIZE - not_copied;
}
+
+/**
+ * insn_decode() - Decode an instruction
+ * @insn: Structure to store decoded instruction
+ * @regs: Structure with register values as seen when entering kernel mode
+ * @buf: Buffer containing the instruction bytes
+ * @buf_size: Number of instruction bytes available in buf
+ *
+ * Decodes the instruction provided in buf and stores the decoding results in
+ * insn. Also determines the correct address and operand sizes.
+ *
+ * Returns:
+ *
+ * True if instruction was decoded, False otherwise.
+ */
+bool insn_decode(struct insn *insn, struct pt_regs *regs,
+ unsigned char buf[MAX_INSN_SIZE], int buf_size)
+{
+ int seg_defs;
+
+ insn_init(insn, buf, buf_size, user_64bit_mode(regs));
+
+ /*
+ * Override the default operand and address sizes with what is specified
+ * in the code segment descriptor. The instruction decoder only sets
+ * the address size it to either 4 or 8 address bytes and does nothing
+ * for the operand bytes. This OK for most of the cases, but we could
+ * have special cases where, for instance, a 16-bit code segment
+ * descriptor is used.
+ * If there is an address override prefix, the instruction decoder
+ * correctly updates these values, even for 16-bit defaults.
+ */
+ seg_defs = insn_get_code_seg_params(regs);
+ if (seg_defs == -EINVAL)
+ return false;
+
+ insn->addr_bytes = INSN_CODE_SEG_ADDR_SZ(seg_defs);
+ insn->opnd_bytes = INSN_CODE_SEG_OPND_SZ(seg_defs);
+
+ insn_get_length(insn);
+ if (buf_size < insn->length)
+ return false;
+
+ return true;
+}
--
2.28.0
_______________________________________________
Virtualization mailing list
Virtualization@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/virtualization
^ permalink raw reply related [flat|nested] 264+ messages in thread
* [PATCH v7 10/72] x86/insn: Add insn_get_modrm_reg_off()
2020-09-07 13:15 ` Joerg Roedel
@ 2020-09-07 13:15 ` Joerg Roedel
-1 siblings, 0 replies; 264+ messages in thread
From: Joerg Roedel @ 2020-09-07 13:15 UTC (permalink / raw)
To: x86
Cc: Joerg Roedel, Joerg Roedel, Masami Hiramatsu, hpa,
Andy Lutomirski, Dave Hansen, Peter Zijlstra, Jiri Slaby,
Dan Williams, Tom Lendacky, Juergen Gross, Kees Cook,
David Rientjes, Cfir Cohen, Erdem Aktas, Mike Stunes,
Sean Christopherson, Martin Radev, linux-kernel, kvm,
virtualization
From: Joerg Roedel <jroedel@suse.de>
Add a function to the instruction decoder which returns the pt_regs
offset of the register specified in the reg field of the modrm byte.
Signed-off-by: Joerg Roedel <jroedel@suse.de>
Acked-by: Masami Hiramatsu <mhiramat@kernel.org>
---
arch/x86/include/asm/insn-eval.h | 1 +
arch/x86/lib/insn-eval.c | 23 +++++++++++++++++++++++
2 files changed, 24 insertions(+)
diff --git a/arch/x86/include/asm/insn-eval.h b/arch/x86/include/asm/insn-eval.h
index 392b4fe377f9..f748f57f1491 100644
--- a/arch/x86/include/asm/insn-eval.h
+++ b/arch/x86/include/asm/insn-eval.h
@@ -17,6 +17,7 @@
void __user *insn_get_addr_ref(struct insn *insn, struct pt_regs *regs);
int insn_get_modrm_rm_off(struct insn *insn, struct pt_regs *regs);
+int insn_get_modrm_reg_off(struct insn *insn, struct pt_regs *regs);
unsigned long insn_get_seg_base(struct pt_regs *regs, int seg_reg_idx);
int insn_get_code_seg_params(struct pt_regs *regs);
int insn_fetch_from_user(struct pt_regs *regs,
diff --git a/arch/x86/lib/insn-eval.c b/arch/x86/lib/insn-eval.c
index 2323c85132cf..f20942ce76c6 100644
--- a/arch/x86/lib/insn-eval.c
+++ b/arch/x86/lib/insn-eval.c
@@ -20,6 +20,7 @@
enum reg_type {
REG_TYPE_RM = 0,
+ REG_TYPE_REG,
REG_TYPE_INDEX,
REG_TYPE_BASE,
};
@@ -439,6 +440,13 @@ static int get_reg_offset(struct insn *insn, struct pt_regs *regs,
regno += 8;
break;
+ case REG_TYPE_REG:
+ regno = X86_MODRM_REG(insn->modrm.value);
+
+ if (X86_REX_R(insn->rex_prefix.value))
+ regno += 8;
+ break;
+
case REG_TYPE_INDEX:
regno = X86_SIB_INDEX(insn->sib.value);
if (X86_REX_X(insn->rex_prefix.value))
@@ -807,6 +815,21 @@ int insn_get_modrm_rm_off(struct insn *insn, struct pt_regs *regs)
return get_reg_offset(insn, regs, REG_TYPE_RM);
}
+/**
+ * insn_get_modrm_reg_off() - Obtain register in reg part of the ModRM byte
+ * @insn: Instruction containing the ModRM byte
+ * @regs: Register values as seen when entering kernel mode
+ *
+ * Returns:
+ *
+ * The register indicated by the reg part of the ModRM byte. The
+ * register is obtained as an offset from the base of pt_regs.
+ */
+int insn_get_modrm_reg_off(struct insn *insn, struct pt_regs *regs)
+{
+ return get_reg_offset(insn, regs, REG_TYPE_REG);
+}
+
/**
* get_seg_base_limit() - obtain base address and limit of a segment
* @insn: Instruction. Must be valid.
--
2.28.0
^ permalink raw reply related [flat|nested] 264+ messages in thread
* [PATCH v7 10/72] x86/insn: Add insn_get_modrm_reg_off()
@ 2020-09-07 13:15 ` Joerg Roedel
0 siblings, 0 replies; 264+ messages in thread
From: Joerg Roedel @ 2020-09-07 13:15 UTC (permalink / raw)
To: x86
Cc: Juergen Gross, Tom Lendacky, Joerg Roedel, Mike Stunes,
Kees Cook, kvm, Peter Zijlstra, Cfir Cohen, Joerg Roedel,
Dave Hansen, linux-kernel, Sean Christopherson, virtualization,
Martin Radev, Masami Hiramatsu, Andy Lutomirski, hpa,
Erdem Aktas, David Rientjes, Dan Williams, Jiri Slaby
From: Joerg Roedel <jroedel@suse.de>
Add a function to the instruction decoder which returns the pt_regs
offset of the register specified in the reg field of the modrm byte.
Signed-off-by: Joerg Roedel <jroedel@suse.de>
Acked-by: Masami Hiramatsu <mhiramat@kernel.org>
---
arch/x86/include/asm/insn-eval.h | 1 +
arch/x86/lib/insn-eval.c | 23 +++++++++++++++++++++++
2 files changed, 24 insertions(+)
diff --git a/arch/x86/include/asm/insn-eval.h b/arch/x86/include/asm/insn-eval.h
index 392b4fe377f9..f748f57f1491 100644
--- a/arch/x86/include/asm/insn-eval.h
+++ b/arch/x86/include/asm/insn-eval.h
@@ -17,6 +17,7 @@
void __user *insn_get_addr_ref(struct insn *insn, struct pt_regs *regs);
int insn_get_modrm_rm_off(struct insn *insn, struct pt_regs *regs);
+int insn_get_modrm_reg_off(struct insn *insn, struct pt_regs *regs);
unsigned long insn_get_seg_base(struct pt_regs *regs, int seg_reg_idx);
int insn_get_code_seg_params(struct pt_regs *regs);
int insn_fetch_from_user(struct pt_regs *regs,
diff --git a/arch/x86/lib/insn-eval.c b/arch/x86/lib/insn-eval.c
index 2323c85132cf..f20942ce76c6 100644
--- a/arch/x86/lib/insn-eval.c
+++ b/arch/x86/lib/insn-eval.c
@@ -20,6 +20,7 @@
enum reg_type {
REG_TYPE_RM = 0,
+ REG_TYPE_REG,
REG_TYPE_INDEX,
REG_TYPE_BASE,
};
@@ -439,6 +440,13 @@ static int get_reg_offset(struct insn *insn, struct pt_regs *regs,
regno += 8;
break;
+ case REG_TYPE_REG:
+ regno = X86_MODRM_REG(insn->modrm.value);
+
+ if (X86_REX_R(insn->rex_prefix.value))
+ regno += 8;
+ break;
+
case REG_TYPE_INDEX:
regno = X86_SIB_INDEX(insn->sib.value);
if (X86_REX_X(insn->rex_prefix.value))
@@ -807,6 +815,21 @@ int insn_get_modrm_rm_off(struct insn *insn, struct pt_regs *regs)
return get_reg_offset(insn, regs, REG_TYPE_RM);
}
+/**
+ * insn_get_modrm_reg_off() - Obtain register in reg part of the ModRM byte
+ * @insn: Instruction containing the ModRM byte
+ * @regs: Register values as seen when entering kernel mode
+ *
+ * Returns:
+ *
+ * The register indicated by the reg part of the ModRM byte. The
+ * register is obtained as an offset from the base of pt_regs.
+ */
+int insn_get_modrm_reg_off(struct insn *insn, struct pt_regs *regs)
+{
+ return get_reg_offset(insn, regs, REG_TYPE_REG);
+}
+
/**
* get_seg_base_limit() - obtain base address and limit of a segment
* @insn: Instruction. Must be valid.
--
2.28.0
_______________________________________________
Virtualization mailing list
Virtualization@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/virtualization
^ permalink raw reply related [flat|nested] 264+ messages in thread
* [PATCH v7 11/72] x86/insn: Add insn_has_rep_prefix() helper
2020-09-07 13:15 ` Joerg Roedel
@ 2020-09-07 13:15 ` Joerg Roedel
-1 siblings, 0 replies; 264+ messages in thread
From: Joerg Roedel @ 2020-09-07 13:15 UTC (permalink / raw)
To: x86
Cc: Joerg Roedel, Joerg Roedel, Masami Hiramatsu, hpa,
Andy Lutomirski, Dave Hansen, Peter Zijlstra, Jiri Slaby,
Dan Williams, Tom Lendacky, Juergen Gross, Kees Cook,
David Rientjes, Cfir Cohen, Erdem Aktas, Mike Stunes,
Sean Christopherson, Martin Radev, linux-kernel, kvm,
virtualization
From: Joerg Roedel <jroedel@suse.de>
Add a function to check whether an instruction has a REP prefix.
Signed-off-by: Joerg Roedel <jroedel@suse.de>
Reviewed-by: Masami Hiramatsu <mhiramat@kernel.org>
---
arch/x86/include/asm/insn-eval.h | 1 +
arch/x86/lib/insn-eval.c | 24 ++++++++++++++++++++++++
2 files changed, 25 insertions(+)
diff --git a/arch/x86/include/asm/insn-eval.h b/arch/x86/include/asm/insn-eval.h
index f748f57f1491..a0f839aa144d 100644
--- a/arch/x86/include/asm/insn-eval.h
+++ b/arch/x86/include/asm/insn-eval.h
@@ -15,6 +15,7 @@
#define INSN_CODE_SEG_OPND_SZ(params) (params & 0xf)
#define INSN_CODE_SEG_PARAMS(oper_sz, addr_sz) (oper_sz | (addr_sz << 4))
+bool insn_has_rep_prefix(struct insn *insn);
void __user *insn_get_addr_ref(struct insn *insn, struct pt_regs *regs);
int insn_get_modrm_rm_off(struct insn *insn, struct pt_regs *regs);
int insn_get_modrm_reg_off(struct insn *insn, struct pt_regs *regs);
diff --git a/arch/x86/lib/insn-eval.c b/arch/x86/lib/insn-eval.c
index f20942ce76c6..58f7fb95c7f4 100644
--- a/arch/x86/lib/insn-eval.c
+++ b/arch/x86/lib/insn-eval.c
@@ -53,6 +53,30 @@ static bool is_string_insn(struct insn *insn)
}
}
+/**
+ * insn_has_rep_prefix() - Determine if instruction has a REP prefix
+ * @insn: Instruction containing the prefix to inspect
+ *
+ * Returns:
+ *
+ * true if the instruction has a REP prefix, false if not.
+ */
+bool insn_has_rep_prefix(struct insn *insn)
+{
+ int i;
+
+ insn_get_prefixes(insn);
+
+ for (i = 0; i < insn->prefixes.nbytes; i++) {
+ insn_byte_t p = insn->prefixes.bytes[i];
+
+ if (p == 0xf2 || p == 0xf3)
+ return true;
+ }
+
+ return false;
+}
+
/**
* get_seg_reg_override_idx() - obtain segment register override index
* @insn: Valid instruction with segment override prefixes
--
2.28.0
^ permalink raw reply related [flat|nested] 264+ messages in thread
* [PATCH v7 11/72] x86/insn: Add insn_has_rep_prefix() helper
@ 2020-09-07 13:15 ` Joerg Roedel
0 siblings, 0 replies; 264+ messages in thread
From: Joerg Roedel @ 2020-09-07 13:15 UTC (permalink / raw)
To: x86
Cc: Juergen Gross, Tom Lendacky, Joerg Roedel, Mike Stunes,
Kees Cook, kvm, Peter Zijlstra, Cfir Cohen, Joerg Roedel,
Dave Hansen, linux-kernel, Sean Christopherson, virtualization,
Martin Radev, Masami Hiramatsu, Andy Lutomirski, hpa,
Erdem Aktas, David Rientjes, Dan Williams, Jiri Slaby
From: Joerg Roedel <jroedel@suse.de>
Add a function to check whether an instruction has a REP prefix.
Signed-off-by: Joerg Roedel <jroedel@suse.de>
Reviewed-by: Masami Hiramatsu <mhiramat@kernel.org>
---
arch/x86/include/asm/insn-eval.h | 1 +
arch/x86/lib/insn-eval.c | 24 ++++++++++++++++++++++++
2 files changed, 25 insertions(+)
diff --git a/arch/x86/include/asm/insn-eval.h b/arch/x86/include/asm/insn-eval.h
index f748f57f1491..a0f839aa144d 100644
--- a/arch/x86/include/asm/insn-eval.h
+++ b/arch/x86/include/asm/insn-eval.h
@@ -15,6 +15,7 @@
#define INSN_CODE_SEG_OPND_SZ(params) (params & 0xf)
#define INSN_CODE_SEG_PARAMS(oper_sz, addr_sz) (oper_sz | (addr_sz << 4))
+bool insn_has_rep_prefix(struct insn *insn);
void __user *insn_get_addr_ref(struct insn *insn, struct pt_regs *regs);
int insn_get_modrm_rm_off(struct insn *insn, struct pt_regs *regs);
int insn_get_modrm_reg_off(struct insn *insn, struct pt_regs *regs);
diff --git a/arch/x86/lib/insn-eval.c b/arch/x86/lib/insn-eval.c
index f20942ce76c6..58f7fb95c7f4 100644
--- a/arch/x86/lib/insn-eval.c
+++ b/arch/x86/lib/insn-eval.c
@@ -53,6 +53,30 @@ static bool is_string_insn(struct insn *insn)
}
}
+/**
+ * insn_has_rep_prefix() - Determine if instruction has a REP prefix
+ * @insn: Instruction containing the prefix to inspect
+ *
+ * Returns:
+ *
+ * true if the instruction has a REP prefix, false if not.
+ */
+bool insn_has_rep_prefix(struct insn *insn)
+{
+ int i;
+
+ insn_get_prefixes(insn);
+
+ for (i = 0; i < insn->prefixes.nbytes; i++) {
+ insn_byte_t p = insn->prefixes.bytes[i];
+
+ if (p == 0xf2 || p == 0xf3)
+ return true;
+ }
+
+ return false;
+}
+
/**
* get_seg_reg_override_idx() - obtain segment register override index
* @insn: Valid instruction with segment override prefixes
--
2.28.0
_______________________________________________
Virtualization mailing list
Virtualization@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/virtualization
^ permalink raw reply related [flat|nested] 264+ messages in thread
* [PATCH v7 12/72] x86/boot/compressed/64: Disable red-zone usage
2020-09-07 13:15 ` Joerg Roedel
@ 2020-09-07 13:15 ` Joerg Roedel
-1 siblings, 0 replies; 264+ messages in thread
From: Joerg Roedel @ 2020-09-07 13:15 UTC (permalink / raw)
To: x86
Cc: Joerg Roedel, Joerg Roedel, Kees Cook, hpa, Andy Lutomirski,
Dave Hansen, Peter Zijlstra, Jiri Slaby, Dan Williams,
Tom Lendacky, Juergen Gross, David Rientjes, Cfir Cohen,
Erdem Aktas, Masami Hiramatsu, Mike Stunes, Sean Christopherson,
Martin Radev, linux-kernel, kvm, virtualization
From: Joerg Roedel <jroedel@suse.de>
The x86-64 ABI defines a red-zone on the stack:
The 128-byte area beyond the location pointed to by %rsp is considered
to be reserved and shall not be modified by signal or interrupt
handlers. Therefore, functions may use this area for temporary data
that is not needed across function calls. In particular, leaf
functions may use this area for their entire stack frame, rather than
adjusting the stack pointer in the prologue and epilogue. This area is
known as the red zone.
This is not compatible with exception handling, because the IRET frame
written by the hardware at the stack pointer and the functions to handle
the exception will overwrite the temporary variables of the interrupted
function, causing undefined behavior. So disable red-zones for the
pre-decompression boot code.
Signed-off-by: Joerg Roedel <jroedel@suse.de>
Reviewed-by: Kees Cook <keescook@chromium.org>
---
arch/x86/boot/compressed/Makefile | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/arch/x86/boot/compressed/Makefile b/arch/x86/boot/compressed/Makefile
index 871cc071c925..258a2fca6659 100644
--- a/arch/x86/boot/compressed/Makefile
+++ b/arch/x86/boot/compressed/Makefile
@@ -32,7 +32,7 @@ KBUILD_CFLAGS := -m$(BITS) -O2
KBUILD_CFLAGS += -fno-strict-aliasing -fPIE
KBUILD_CFLAGS += -DDISABLE_BRANCH_PROFILING
cflags-$(CONFIG_X86_32) := -march=i386
-cflags-$(CONFIG_X86_64) := -mcmodel=small
+cflags-$(CONFIG_X86_64) := -mcmodel=small -mno-red-zone
KBUILD_CFLAGS += $(cflags-y)
KBUILD_CFLAGS += -mno-mmx -mno-sse
KBUILD_CFLAGS += -ffreestanding
--
2.28.0
^ permalink raw reply related [flat|nested] 264+ messages in thread
* [PATCH v7 12/72] x86/boot/compressed/64: Disable red-zone usage
@ 2020-09-07 13:15 ` Joerg Roedel
0 siblings, 0 replies; 264+ messages in thread
From: Joerg Roedel @ 2020-09-07 13:15 UTC (permalink / raw)
To: x86
Cc: Juergen Gross, Tom Lendacky, Joerg Roedel, Mike Stunes,
Kees Cook, kvm, Peter Zijlstra, Cfir Cohen, Joerg Roedel,
Dave Hansen, linux-kernel, Sean Christopherson, virtualization,
Martin Radev, Masami Hiramatsu, Andy Lutomirski, hpa,
Erdem Aktas, David Rientjes, Dan Williams, Jiri Slaby
From: Joerg Roedel <jroedel@suse.de>
The x86-64 ABI defines a red-zone on the stack:
The 128-byte area beyond the location pointed to by %rsp is considered
to be reserved and shall not be modified by signal or interrupt
handlers. Therefore, functions may use this area for temporary data
that is not needed across function calls. In particular, leaf
functions may use this area for their entire stack frame, rather than
adjusting the stack pointer in the prologue and epilogue. This area is
known as the red zone.
This is not compatible with exception handling, because the IRET frame
written by the hardware at the stack pointer and the functions to handle
the exception will overwrite the temporary variables of the interrupted
function, causing undefined behavior. So disable red-zones for the
pre-decompression boot code.
Signed-off-by: Joerg Roedel <jroedel@suse.de>
Reviewed-by: Kees Cook <keescook@chromium.org>
---
arch/x86/boot/compressed/Makefile | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/arch/x86/boot/compressed/Makefile b/arch/x86/boot/compressed/Makefile
index 871cc071c925..258a2fca6659 100644
--- a/arch/x86/boot/compressed/Makefile
+++ b/arch/x86/boot/compressed/Makefile
@@ -32,7 +32,7 @@ KBUILD_CFLAGS := -m$(BITS) -O2
KBUILD_CFLAGS += -fno-strict-aliasing -fPIE
KBUILD_CFLAGS += -DDISABLE_BRANCH_PROFILING
cflags-$(CONFIG_X86_32) := -march=i386
-cflags-$(CONFIG_X86_64) := -mcmodel=small
+cflags-$(CONFIG_X86_64) := -mcmodel=small -mno-red-zone
KBUILD_CFLAGS += $(cflags-y)
KBUILD_CFLAGS += -mno-mmx -mno-sse
KBUILD_CFLAGS += -ffreestanding
--
2.28.0
_______________________________________________
Virtualization mailing list
Virtualization@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/virtualization
^ permalink raw reply related [flat|nested] 264+ messages in thread
* [PATCH v7 13/72] x86/boot/compressed/64: Add IDT Infrastructure
2020-09-07 13:15 ` Joerg Roedel
@ 2020-09-07 13:15 ` Joerg Roedel
-1 siblings, 0 replies; 264+ messages in thread
From: Joerg Roedel @ 2020-09-07 13:15 UTC (permalink / raw)
To: x86
Cc: Joerg Roedel, Joerg Roedel, hpa, Andy Lutomirski, Dave Hansen,
Peter Zijlstra, Jiri Slaby, Dan Williams, Tom Lendacky,
Juergen Gross, Kees Cook, David Rientjes, Cfir Cohen,
Erdem Aktas, Masami Hiramatsu, Mike Stunes, Sean Christopherson,
Martin Radev, linux-kernel, kvm, virtualization
From: Joerg Roedel <jroedel@suse.de>
Add code needed to setup an IDT in the early pre-decompression
boot-code. The IDT is loaded first in startup_64, which is after
EfiExitBootServices() has been called, and later reloaded when the
kernel image has been relocated to the end of the decompression area.
This allows to setup different IDT handlers before and after the
relocation.
Signed-off-by: Joerg Roedel <jroedel@suse.de>
---
arch/x86/boot/compressed/Makefile | 1 +
arch/x86/boot/compressed/head_64.S | 25 +++++++-
arch/x86/boot/compressed/idt_64.c | 44 ++++++++++++++
arch/x86/boot/compressed/idt_handlers_64.S | 70 ++++++++++++++++++++++
arch/x86/boot/compressed/misc.h | 5 ++
arch/x86/include/asm/desc_defs.h | 3 +
6 files changed, 147 insertions(+), 1 deletion(-)
create mode 100644 arch/x86/boot/compressed/idt_64.c
create mode 100644 arch/x86/boot/compressed/idt_handlers_64.S
diff --git a/arch/x86/boot/compressed/Makefile b/arch/x86/boot/compressed/Makefile
index 258a2fca6659..b378fb14d87c 100644
--- a/arch/x86/boot/compressed/Makefile
+++ b/arch/x86/boot/compressed/Makefile
@@ -80,6 +80,7 @@ vmlinux-objs-$(CONFIG_EARLY_PRINTK) += $(obj)/early_serial_console.o
vmlinux-objs-$(CONFIG_RANDOMIZE_BASE) += $(obj)/kaslr.o
ifdef CONFIG_X86_64
vmlinux-objs-$(CONFIG_RANDOMIZE_BASE) += $(obj)/kaslr_64.o
+ vmlinux-objs-y += $(obj)/idt_64.o $(obj)/idt_handlers_64.o
vmlinux-objs-y += $(obj)/mem_encrypt.o
vmlinux-objs-y += $(obj)/pgtable_64.o
endif
diff --git a/arch/x86/boot/compressed/head_64.S b/arch/x86/boot/compressed/head_64.S
index 9e46729cf162..1fffaec1c069 100644
--- a/arch/x86/boot/compressed/head_64.S
+++ b/arch/x86/boot/compressed/head_64.S
@@ -33,6 +33,7 @@
#include <asm/processor-flags.h>
#include <asm/asm-offsets.h>
#include <asm/bootparam.h>
+#include <asm/desc_defs.h>
#include "pgtable.h"
/*
@@ -415,6 +416,10 @@ SYM_CODE_START(startup_64)
.Lon_kernel_cs:
+ pushq %rsi
+ call load_stage1_idt
+ popq %rsi
+
/*
* paging_prepare() sets up the trampoline and checks if we need to
* enable 5-level paging.
@@ -527,6 +532,13 @@ SYM_FUNC_START_LOCAL_NOALIGN(.Lrelocated)
shrq $3, %rcx
rep stosq
+/*
+ * Load stage2 IDT
+ */
+ pushq %rsi
+ call load_stage2_idt
+ popq %rsi
+
/*
* Do the extraction, and jump to the new kernel..
*/
@@ -659,10 +671,21 @@ SYM_DATA_START_LOCAL(gdt)
.quad 0x0000000000000000 /* TS continued */
SYM_DATA_END_LABEL(gdt, SYM_L_LOCAL, gdt_end)
+SYM_DATA_START(boot_idt_desc)
+ .word boot_idt_end - boot_idt - 1
+ .quad 0
+SYM_DATA_END(boot_idt_desc)
+ .balign 8
+SYM_DATA_START(boot_idt)
+ .rept BOOT_IDT_ENTRIES
+ .quad 0
+ .quad 0
+ .endr
+SYM_DATA_END_LABEL(boot_idt, SYM_L_GLOBAL, boot_idt_end)
+
#ifdef CONFIG_EFI_STUB
SYM_DATA(image_offset, .long 0)
#endif
-
#ifdef CONFIG_EFI_MIXED
SYM_DATA_LOCAL(efi32_boot_args, .long 0, 0, 0)
SYM_DATA(efi_is64, .byte 1)
diff --git a/arch/x86/boot/compressed/idt_64.c b/arch/x86/boot/compressed/idt_64.c
new file mode 100644
index 000000000000..082cd6bca033
--- /dev/null
+++ b/arch/x86/boot/compressed/idt_64.c
@@ -0,0 +1,44 @@
+// SPDX-License-Identifier: GPL-2.0-only
+#include <asm/trap_pf.h>
+#include <asm/segment.h>
+#include <asm/trapnr.h>
+#include "misc.h"
+
+static void set_idt_entry(int vector, void (*handler)(void))
+{
+ unsigned long address = (unsigned long)handler;
+ gate_desc entry;
+
+ memset(&entry, 0, sizeof(entry));
+
+ entry.offset_low = (u16)(address & 0xffff);
+ entry.segment = __KERNEL_CS;
+ entry.bits.type = GATE_TRAP;
+ entry.bits.p = 1;
+ entry.offset_middle = (u16)((address >> 16) & 0xffff);
+ entry.offset_high = (u32)(address >> 32);
+
+ memcpy(&boot_idt[vector], &entry, sizeof(entry));
+}
+
+/* Have this here so we don't need to include <asm/desc.h> */
+static void load_boot_idt(const struct desc_ptr *dtr)
+{
+ asm volatile("lidt %0"::"m" (*dtr));
+}
+
+/* Setup IDT before kernel jumping to .Lrelocated */
+void load_stage1_idt(void)
+{
+ boot_idt_desc.address = (unsigned long)boot_idt;
+
+ load_boot_idt(&boot_idt_desc);
+}
+
+/* Setup IDT after kernel jumping to .Lrelocated */
+void load_stage2_idt(void)
+{
+ boot_idt_desc.address = (unsigned long)boot_idt;
+
+ load_boot_idt(&boot_idt_desc);
+}
diff --git a/arch/x86/boot/compressed/idt_handlers_64.S b/arch/x86/boot/compressed/idt_handlers_64.S
new file mode 100644
index 000000000000..36dee2f40a8b
--- /dev/null
+++ b/arch/x86/boot/compressed/idt_handlers_64.S
@@ -0,0 +1,70 @@
+/* SPDX-License-Identifier: GPL-2.0-only */
+/*
+ * Early IDT handler entry points
+ *
+ * Copyright (C) 2019 SUSE
+ *
+ * Author: Joerg Roedel <jroedel@suse.de>
+ */
+
+#include <asm/segment.h>
+
+/* For ORIG_RAX */
+#include "../../entry/calling.h"
+
+.macro EXCEPTION_HANDLER name function error_code=0
+SYM_FUNC_START(\name)
+
+ /* Build pt_regs */
+ .if \error_code == 0
+ pushq $0
+ .endif
+
+ pushq %rdi
+ pushq %rsi
+ pushq %rdx
+ pushq %rcx
+ pushq %rax
+ pushq %r8
+ pushq %r9
+ pushq %r10
+ pushq %r11
+ pushq %rbx
+ pushq %rbp
+ pushq %r12
+ pushq %r13
+ pushq %r14
+ pushq %r15
+
+ /* Call handler with pt_regs */
+ movq %rsp, %rdi
+ /* Error code is second parameter */
+ movq ORIG_RAX(%rsp), %rsi
+ call \function
+
+ /* Restore regs */
+ popq %r15
+ popq %r14
+ popq %r13
+ popq %r12
+ popq %rbp
+ popq %rbx
+ popq %r11
+ popq %r10
+ popq %r9
+ popq %r8
+ popq %rax
+ popq %rcx
+ popq %rdx
+ popq %rsi
+ popq %rdi
+
+ /* Remove error code and return */
+ addq $8, %rsp
+
+ iretq
+SYM_FUNC_END(\name)
+ .endm
+
+ .text
+ .code64
diff --git a/arch/x86/boot/compressed/misc.h b/arch/x86/boot/compressed/misc.h
index 3efce27ba35c..8feb5f6f329e 100644
--- a/arch/x86/boot/compressed/misc.h
+++ b/arch/x86/boot/compressed/misc.h
@@ -23,6 +23,7 @@
#include <asm/page.h>
#include <asm/boot.h>
#include <asm/bootparam.h>
+#include <asm/desc_defs.h>
#define BOOT_CTYPE_H
#include <linux/acpi.h>
@@ -133,4 +134,8 @@ int count_immovable_mem_regions(void);
static inline int count_immovable_mem_regions(void) { return 0; }
#endif
+/* idt_64.c */
+extern gate_desc boot_idt[BOOT_IDT_ENTRIES];
+extern struct desc_ptr boot_idt_desc;
+
#endif /* BOOT_COMPRESSED_MISC_H */
diff --git a/arch/x86/include/asm/desc_defs.h b/arch/x86/include/asm/desc_defs.h
index a91f3b6e4f2a..5621fb3f2d1a 100644
--- a/arch/x86/include/asm/desc_defs.h
+++ b/arch/x86/include/asm/desc_defs.h
@@ -109,6 +109,9 @@ struct desc_ptr {
#endif /* !__ASSEMBLY__ */
+/* Boot IDT definitions */
+#define BOOT_IDT_ENTRIES 32
+
/* Access rights as returned by LAR */
#define AR_TYPE_RODATA (0 * (1 << 9))
#define AR_TYPE_RWDATA (1 * (1 << 9))
--
2.28.0
^ permalink raw reply related [flat|nested] 264+ messages in thread
* [PATCH v7 13/72] x86/boot/compressed/64: Add IDT Infrastructure
@ 2020-09-07 13:15 ` Joerg Roedel
0 siblings, 0 replies; 264+ messages in thread
From: Joerg Roedel @ 2020-09-07 13:15 UTC (permalink / raw)
To: x86
Cc: Juergen Gross, Tom Lendacky, Joerg Roedel, Mike Stunes,
Kees Cook, kvm, Peter Zijlstra, Cfir Cohen, Joerg Roedel,
Dave Hansen, linux-kernel, Sean Christopherson, virtualization,
Martin Radev, Masami Hiramatsu, Andy Lutomirski, hpa,
Erdem Aktas, David Rientjes, Dan Williams, Jiri Slaby
From: Joerg Roedel <jroedel@suse.de>
Add code needed to setup an IDT in the early pre-decompression
boot-code. The IDT is loaded first in startup_64, which is after
EfiExitBootServices() has been called, and later reloaded when the
kernel image has been relocated to the end of the decompression area.
This allows to setup different IDT handlers before and after the
relocation.
Signed-off-by: Joerg Roedel <jroedel@suse.de>
---
arch/x86/boot/compressed/Makefile | 1 +
arch/x86/boot/compressed/head_64.S | 25 +++++++-
arch/x86/boot/compressed/idt_64.c | 44 ++++++++++++++
arch/x86/boot/compressed/idt_handlers_64.S | 70 ++++++++++++++++++++++
arch/x86/boot/compressed/misc.h | 5 ++
arch/x86/include/asm/desc_defs.h | 3 +
6 files changed, 147 insertions(+), 1 deletion(-)
create mode 100644 arch/x86/boot/compressed/idt_64.c
create mode 100644 arch/x86/boot/compressed/idt_handlers_64.S
diff --git a/arch/x86/boot/compressed/Makefile b/arch/x86/boot/compressed/Makefile
index 258a2fca6659..b378fb14d87c 100644
--- a/arch/x86/boot/compressed/Makefile
+++ b/arch/x86/boot/compressed/Makefile
@@ -80,6 +80,7 @@ vmlinux-objs-$(CONFIG_EARLY_PRINTK) += $(obj)/early_serial_console.o
vmlinux-objs-$(CONFIG_RANDOMIZE_BASE) += $(obj)/kaslr.o
ifdef CONFIG_X86_64
vmlinux-objs-$(CONFIG_RANDOMIZE_BASE) += $(obj)/kaslr_64.o
+ vmlinux-objs-y += $(obj)/idt_64.o $(obj)/idt_handlers_64.o
vmlinux-objs-y += $(obj)/mem_encrypt.o
vmlinux-objs-y += $(obj)/pgtable_64.o
endif
diff --git a/arch/x86/boot/compressed/head_64.S b/arch/x86/boot/compressed/head_64.S
index 9e46729cf162..1fffaec1c069 100644
--- a/arch/x86/boot/compressed/head_64.S
+++ b/arch/x86/boot/compressed/head_64.S
@@ -33,6 +33,7 @@
#include <asm/processor-flags.h>
#include <asm/asm-offsets.h>
#include <asm/bootparam.h>
+#include <asm/desc_defs.h>
#include "pgtable.h"
/*
@@ -415,6 +416,10 @@ SYM_CODE_START(startup_64)
.Lon_kernel_cs:
+ pushq %rsi
+ call load_stage1_idt
+ popq %rsi
+
/*
* paging_prepare() sets up the trampoline and checks if we need to
* enable 5-level paging.
@@ -527,6 +532,13 @@ SYM_FUNC_START_LOCAL_NOALIGN(.Lrelocated)
shrq $3, %rcx
rep stosq
+/*
+ * Load stage2 IDT
+ */
+ pushq %rsi
+ call load_stage2_idt
+ popq %rsi
+
/*
* Do the extraction, and jump to the new kernel..
*/
@@ -659,10 +671,21 @@ SYM_DATA_START_LOCAL(gdt)
.quad 0x0000000000000000 /* TS continued */
SYM_DATA_END_LABEL(gdt, SYM_L_LOCAL, gdt_end)
+SYM_DATA_START(boot_idt_desc)
+ .word boot_idt_end - boot_idt - 1
+ .quad 0
+SYM_DATA_END(boot_idt_desc)
+ .balign 8
+SYM_DATA_START(boot_idt)
+ .rept BOOT_IDT_ENTRIES
+ .quad 0
+ .quad 0
+ .endr
+SYM_DATA_END_LABEL(boot_idt, SYM_L_GLOBAL, boot_idt_end)
+
#ifdef CONFIG_EFI_STUB
SYM_DATA(image_offset, .long 0)
#endif
-
#ifdef CONFIG_EFI_MIXED
SYM_DATA_LOCAL(efi32_boot_args, .long 0, 0, 0)
SYM_DATA(efi_is64, .byte 1)
diff --git a/arch/x86/boot/compressed/idt_64.c b/arch/x86/boot/compressed/idt_64.c
new file mode 100644
index 000000000000..082cd6bca033
--- /dev/null
+++ b/arch/x86/boot/compressed/idt_64.c
@@ -0,0 +1,44 @@
+// SPDX-License-Identifier: GPL-2.0-only
+#include <asm/trap_pf.h>
+#include <asm/segment.h>
+#include <asm/trapnr.h>
+#include "misc.h"
+
+static void set_idt_entry(int vector, void (*handler)(void))
+{
+ unsigned long address = (unsigned long)handler;
+ gate_desc entry;
+
+ memset(&entry, 0, sizeof(entry));
+
+ entry.offset_low = (u16)(address & 0xffff);
+ entry.segment = __KERNEL_CS;
+ entry.bits.type = GATE_TRAP;
+ entry.bits.p = 1;
+ entry.offset_middle = (u16)((address >> 16) & 0xffff);
+ entry.offset_high = (u32)(address >> 32);
+
+ memcpy(&boot_idt[vector], &entry, sizeof(entry));
+}
+
+/* Have this here so we don't need to include <asm/desc.h> */
+static void load_boot_idt(const struct desc_ptr *dtr)
+{
+ asm volatile("lidt %0"::"m" (*dtr));
+}
+
+/* Setup IDT before kernel jumping to .Lrelocated */
+void load_stage1_idt(void)
+{
+ boot_idt_desc.address = (unsigned long)boot_idt;
+
+ load_boot_idt(&boot_idt_desc);
+}
+
+/* Setup IDT after kernel jumping to .Lrelocated */
+void load_stage2_idt(void)
+{
+ boot_idt_desc.address = (unsigned long)boot_idt;
+
+ load_boot_idt(&boot_idt_desc);
+}
diff --git a/arch/x86/boot/compressed/idt_handlers_64.S b/arch/x86/boot/compressed/idt_handlers_64.S
new file mode 100644
index 000000000000..36dee2f40a8b
--- /dev/null
+++ b/arch/x86/boot/compressed/idt_handlers_64.S
@@ -0,0 +1,70 @@
+/* SPDX-License-Identifier: GPL-2.0-only */
+/*
+ * Early IDT handler entry points
+ *
+ * Copyright (C) 2019 SUSE
+ *
+ * Author: Joerg Roedel <jroedel@suse.de>
+ */
+
+#include <asm/segment.h>
+
+/* For ORIG_RAX */
+#include "../../entry/calling.h"
+
+.macro EXCEPTION_HANDLER name function error_code=0
+SYM_FUNC_START(\name)
+
+ /* Build pt_regs */
+ .if \error_code == 0
+ pushq $0
+ .endif
+
+ pushq %rdi
+ pushq %rsi
+ pushq %rdx
+ pushq %rcx
+ pushq %rax
+ pushq %r8
+ pushq %r9
+ pushq %r10
+ pushq %r11
+ pushq %rbx
+ pushq %rbp
+ pushq %r12
+ pushq %r13
+ pushq %r14
+ pushq %r15
+
+ /* Call handler with pt_regs */
+ movq %rsp, %rdi
+ /* Error code is second parameter */
+ movq ORIG_RAX(%rsp), %rsi
+ call \function
+
+ /* Restore regs */
+ popq %r15
+ popq %r14
+ popq %r13
+ popq %r12
+ popq %rbp
+ popq %rbx
+ popq %r11
+ popq %r10
+ popq %r9
+ popq %r8
+ popq %rax
+ popq %rcx
+ popq %rdx
+ popq %rsi
+ popq %rdi
+
+ /* Remove error code and return */
+ addq $8, %rsp
+
+ iretq
+SYM_FUNC_END(\name)
+ .endm
+
+ .text
+ .code64
diff --git a/arch/x86/boot/compressed/misc.h b/arch/x86/boot/compressed/misc.h
index 3efce27ba35c..8feb5f6f329e 100644
--- a/arch/x86/boot/compressed/misc.h
+++ b/arch/x86/boot/compressed/misc.h
@@ -23,6 +23,7 @@
#include <asm/page.h>
#include <asm/boot.h>
#include <asm/bootparam.h>
+#include <asm/desc_defs.h>
#define BOOT_CTYPE_H
#include <linux/acpi.h>
@@ -133,4 +134,8 @@ int count_immovable_mem_regions(void);
static inline int count_immovable_mem_regions(void) { return 0; }
#endif
+/* idt_64.c */
+extern gate_desc boot_idt[BOOT_IDT_ENTRIES];
+extern struct desc_ptr boot_idt_desc;
+
#endif /* BOOT_COMPRESSED_MISC_H */
diff --git a/arch/x86/include/asm/desc_defs.h b/arch/x86/include/asm/desc_defs.h
index a91f3b6e4f2a..5621fb3f2d1a 100644
--- a/arch/x86/include/asm/desc_defs.h
+++ b/arch/x86/include/asm/desc_defs.h
@@ -109,6 +109,9 @@ struct desc_ptr {
#endif /* !__ASSEMBLY__ */
+/* Boot IDT definitions */
+#define BOOT_IDT_ENTRIES 32
+
/* Access rights as returned by LAR */
#define AR_TYPE_RODATA (0 * (1 << 9))
#define AR_TYPE_RWDATA (1 * (1 << 9))
--
2.28.0
_______________________________________________
Virtualization mailing list
Virtualization@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/virtualization
^ permalink raw reply related [flat|nested] 264+ messages in thread
* [PATCH v7 14/72] x86/boot/compressed/64: Rename kaslr_64.c to ident_map_64.c
2020-09-07 13:15 ` Joerg Roedel
@ 2020-09-07 13:15 ` Joerg Roedel
-1 siblings, 0 replies; 264+ messages in thread
From: Joerg Roedel @ 2020-09-07 13:15 UTC (permalink / raw)
To: x86
Cc: Joerg Roedel, Joerg Roedel, Kees Cook, hpa, Andy Lutomirski,
Dave Hansen, Peter Zijlstra, Jiri Slaby, Dan Williams,
Tom Lendacky, Juergen Gross, David Rientjes, Cfir Cohen,
Erdem Aktas, Masami Hiramatsu, Mike Stunes, Sean Christopherson,
Martin Radev, linux-kernel, kvm, virtualization
From: Joerg Roedel <jroedel@suse.de>
The file contains only code related to identity mapped page-tables.
Rename the file and compile it always in.
Signed-off-by: Joerg Roedel <jroedel@suse.de>
Reviewed-by: Kees Cook <keescook@chromium.org>
---
arch/x86/boot/compressed/Makefile | 2 +-
arch/x86/boot/compressed/{kaslr_64.c => ident_map_64.c} | 9 +++++++++
arch/x86/boot/compressed/kaslr.c | 9 ---------
arch/x86/boot/compressed/misc.h | 8 ++++++++
4 files changed, 18 insertions(+), 10 deletions(-)
rename arch/x86/boot/compressed/{kaslr_64.c => ident_map_64.c} (95%)
diff --git a/arch/x86/boot/compressed/Makefile b/arch/x86/boot/compressed/Makefile
index b378fb14d87c..c12c9223a1fb 100644
--- a/arch/x86/boot/compressed/Makefile
+++ b/arch/x86/boot/compressed/Makefile
@@ -79,7 +79,7 @@ vmlinux-objs-y := $(obj)/vmlinux.lds $(obj)/kernel_info.o $(obj)/head_$(BITS).o
vmlinux-objs-$(CONFIG_EARLY_PRINTK) += $(obj)/early_serial_console.o
vmlinux-objs-$(CONFIG_RANDOMIZE_BASE) += $(obj)/kaslr.o
ifdef CONFIG_X86_64
- vmlinux-objs-$(CONFIG_RANDOMIZE_BASE) += $(obj)/kaslr_64.o
+ vmlinux-objs-y += $(obj)/ident_map_64.o
vmlinux-objs-y += $(obj)/idt_64.o $(obj)/idt_handlers_64.o
vmlinux-objs-y += $(obj)/mem_encrypt.o
vmlinux-objs-y += $(obj)/pgtable_64.o
diff --git a/arch/x86/boot/compressed/kaslr_64.c b/arch/x86/boot/compressed/ident_map_64.c
similarity index 95%
rename from arch/x86/boot/compressed/kaslr_64.c
rename to arch/x86/boot/compressed/ident_map_64.c
index f9c5c13d979b..d9932a133ac9 100644
--- a/arch/x86/boot/compressed/kaslr_64.c
+++ b/arch/x86/boot/compressed/ident_map_64.c
@@ -29,6 +29,15 @@
#define __PAGE_OFFSET __PAGE_OFFSET_BASE
#include "../../mm/ident_map.c"
+#ifdef CONFIG_X86_5LEVEL
+unsigned int __pgtable_l5_enabled;
+unsigned int pgdir_shift = 39;
+unsigned int ptrs_per_p4d = 1;
+#endif
+
+/* Used by PAGE_KERN* macros: */
+pteval_t __default_kernel_pte_mask __read_mostly = ~0;
+
/* Used to track our page table allocation area. */
struct alloc_pgt_data {
unsigned char *pgt_buf;
diff --git a/arch/x86/boot/compressed/kaslr.c b/arch/x86/boot/compressed/kaslr.c
index 877970d76249..e27de98ed038 100644
--- a/arch/x86/boot/compressed/kaslr.c
+++ b/arch/x86/boot/compressed/kaslr.c
@@ -40,17 +40,8 @@
#include <asm/setup.h> /* For COMMAND_LINE_SIZE */
#undef _SETUP
-#ifdef CONFIG_X86_5LEVEL
-unsigned int __pgtable_l5_enabled;
-unsigned int pgdir_shift __ro_after_init = 39;
-unsigned int ptrs_per_p4d __ro_after_init = 1;
-#endif
-
extern unsigned long get_cmd_line_ptr(void);
-/* Used by PAGE_KERN* macros: */
-pteval_t __default_kernel_pte_mask __read_mostly = ~0;
-
/* Simplified build-specific string for starting entropy. */
static const char build_str[] = UTS_RELEASE " (" LINUX_COMPILE_BY "@"
LINUX_COMPILE_HOST ") (" LINUX_COMPILER ") " UTS_VERSION;
diff --git a/arch/x86/boot/compressed/misc.h b/arch/x86/boot/compressed/misc.h
index 8feb5f6f329e..98b7a1df9c59 100644
--- a/arch/x86/boot/compressed/misc.h
+++ b/arch/x86/boot/compressed/misc.h
@@ -134,6 +134,14 @@ int count_immovable_mem_regions(void);
static inline int count_immovable_mem_regions(void) { return 0; }
#endif
+/* ident_map_64.c */
+#ifdef CONFIG_X86_5LEVEL
+extern unsigned int __pgtable_l5_enabled, pgdir_shift, ptrs_per_p4d;
+#endif
+
+/* Used by PAGE_KERN* macros: */
+extern pteval_t __default_kernel_pte_mask;
+
/* idt_64.c */
extern gate_desc boot_idt[BOOT_IDT_ENTRIES];
extern struct desc_ptr boot_idt_desc;
--
2.28.0
^ permalink raw reply related [flat|nested] 264+ messages in thread
* [PATCH v7 14/72] x86/boot/compressed/64: Rename kaslr_64.c to ident_map_64.c
@ 2020-09-07 13:15 ` Joerg Roedel
0 siblings, 0 replies; 264+ messages in thread
From: Joerg Roedel @ 2020-09-07 13:15 UTC (permalink / raw)
To: x86
Cc: Juergen Gross, Tom Lendacky, Joerg Roedel, Mike Stunes,
Kees Cook, kvm, Peter Zijlstra, Cfir Cohen, Joerg Roedel,
Dave Hansen, linux-kernel, Sean Christopherson, virtualization,
Martin Radev, Masami Hiramatsu, Andy Lutomirski, hpa,
Erdem Aktas, David Rientjes, Dan Williams, Jiri Slaby
From: Joerg Roedel <jroedel@suse.de>
The file contains only code related to identity mapped page-tables.
Rename the file and compile it always in.
Signed-off-by: Joerg Roedel <jroedel@suse.de>
Reviewed-by: Kees Cook <keescook@chromium.org>
---
arch/x86/boot/compressed/Makefile | 2 +-
arch/x86/boot/compressed/{kaslr_64.c => ident_map_64.c} | 9 +++++++++
arch/x86/boot/compressed/kaslr.c | 9 ---------
arch/x86/boot/compressed/misc.h | 8 ++++++++
4 files changed, 18 insertions(+), 10 deletions(-)
rename arch/x86/boot/compressed/{kaslr_64.c => ident_map_64.c} (95%)
diff --git a/arch/x86/boot/compressed/Makefile b/arch/x86/boot/compressed/Makefile
index b378fb14d87c..c12c9223a1fb 100644
--- a/arch/x86/boot/compressed/Makefile
+++ b/arch/x86/boot/compressed/Makefile
@@ -79,7 +79,7 @@ vmlinux-objs-y := $(obj)/vmlinux.lds $(obj)/kernel_info.o $(obj)/head_$(BITS).o
vmlinux-objs-$(CONFIG_EARLY_PRINTK) += $(obj)/early_serial_console.o
vmlinux-objs-$(CONFIG_RANDOMIZE_BASE) += $(obj)/kaslr.o
ifdef CONFIG_X86_64
- vmlinux-objs-$(CONFIG_RANDOMIZE_BASE) += $(obj)/kaslr_64.o
+ vmlinux-objs-y += $(obj)/ident_map_64.o
vmlinux-objs-y += $(obj)/idt_64.o $(obj)/idt_handlers_64.o
vmlinux-objs-y += $(obj)/mem_encrypt.o
vmlinux-objs-y += $(obj)/pgtable_64.o
diff --git a/arch/x86/boot/compressed/kaslr_64.c b/arch/x86/boot/compressed/ident_map_64.c
similarity index 95%
rename from arch/x86/boot/compressed/kaslr_64.c
rename to arch/x86/boot/compressed/ident_map_64.c
index f9c5c13d979b..d9932a133ac9 100644
--- a/arch/x86/boot/compressed/kaslr_64.c
+++ b/arch/x86/boot/compressed/ident_map_64.c
@@ -29,6 +29,15 @@
#define __PAGE_OFFSET __PAGE_OFFSET_BASE
#include "../../mm/ident_map.c"
+#ifdef CONFIG_X86_5LEVEL
+unsigned int __pgtable_l5_enabled;
+unsigned int pgdir_shift = 39;
+unsigned int ptrs_per_p4d = 1;
+#endif
+
+/* Used by PAGE_KERN* macros: */
+pteval_t __default_kernel_pte_mask __read_mostly = ~0;
+
/* Used to track our page table allocation area. */
struct alloc_pgt_data {
unsigned char *pgt_buf;
diff --git a/arch/x86/boot/compressed/kaslr.c b/arch/x86/boot/compressed/kaslr.c
index 877970d76249..e27de98ed038 100644
--- a/arch/x86/boot/compressed/kaslr.c
+++ b/arch/x86/boot/compressed/kaslr.c
@@ -40,17 +40,8 @@
#include <asm/setup.h> /* For COMMAND_LINE_SIZE */
#undef _SETUP
-#ifdef CONFIG_X86_5LEVEL
-unsigned int __pgtable_l5_enabled;
-unsigned int pgdir_shift __ro_after_init = 39;
-unsigned int ptrs_per_p4d __ro_after_init = 1;
-#endif
-
extern unsigned long get_cmd_line_ptr(void);
-/* Used by PAGE_KERN* macros: */
-pteval_t __default_kernel_pte_mask __read_mostly = ~0;
-
/* Simplified build-specific string for starting entropy. */
static const char build_str[] = UTS_RELEASE " (" LINUX_COMPILE_BY "@"
LINUX_COMPILE_HOST ") (" LINUX_COMPILER ") " UTS_VERSION;
diff --git a/arch/x86/boot/compressed/misc.h b/arch/x86/boot/compressed/misc.h
index 8feb5f6f329e..98b7a1df9c59 100644
--- a/arch/x86/boot/compressed/misc.h
+++ b/arch/x86/boot/compressed/misc.h
@@ -134,6 +134,14 @@ int count_immovable_mem_regions(void);
static inline int count_immovable_mem_regions(void) { return 0; }
#endif
+/* ident_map_64.c */
+#ifdef CONFIG_X86_5LEVEL
+extern unsigned int __pgtable_l5_enabled, pgdir_shift, ptrs_per_p4d;
+#endif
+
+/* Used by PAGE_KERN* macros: */
+extern pteval_t __default_kernel_pte_mask;
+
/* idt_64.c */
extern gate_desc boot_idt[BOOT_IDT_ENTRIES];
extern struct desc_ptr boot_idt_desc;
--
2.28.0
_______________________________________________
Virtualization mailing list
Virtualization@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/virtualization
^ permalink raw reply related [flat|nested] 264+ messages in thread
* [PATCH v7 15/72] x86/boot/compressed/64: Add page-fault handler
2020-09-07 13:15 ` Joerg Roedel
@ 2020-09-07 13:15 ` Joerg Roedel
-1 siblings, 0 replies; 264+ messages in thread
From: Joerg Roedel @ 2020-09-07 13:15 UTC (permalink / raw)
To: x86
Cc: Joerg Roedel, Joerg Roedel, Kees Cook, hpa, Andy Lutomirski,
Dave Hansen, Peter Zijlstra, Jiri Slaby, Dan Williams,
Tom Lendacky, Juergen Gross, David Rientjes, Cfir Cohen,
Erdem Aktas, Masami Hiramatsu, Mike Stunes, Sean Christopherson,
Martin Radev, linux-kernel, kvm, virtualization
From: Joerg Roedel <jroedel@suse.de>
Install a page-fault handler to add an identity mapping to addresses
not yet mapped. Also do some checking whether the error code is sane.
This makes non SEV-ES machines use the exception handling
infrastructure in the pre-decompressions boot code too, making it less
likely to break in the future.
Signed-off-by: Joerg Roedel <jroedel@suse.de>
Reviewed-by: Kees Cook <keescook@chromium.org>
---
arch/x86/boot/compressed/ident_map_64.c | 39 ++++++++++++++++++++++
arch/x86/boot/compressed/idt_64.c | 2 ++
arch/x86/boot/compressed/idt_handlers_64.S | 2 ++
arch/x86/boot/compressed/misc.h | 6 ++++
4 files changed, 49 insertions(+)
diff --git a/arch/x86/boot/compressed/ident_map_64.c b/arch/x86/boot/compressed/ident_map_64.c
index d9932a133ac9..e3d980ae9c2b 100644
--- a/arch/x86/boot/compressed/ident_map_64.c
+++ b/arch/x86/boot/compressed/ident_map_64.c
@@ -19,10 +19,13 @@
/* No PAGE_TABLE_ISOLATION support needed either: */
#undef CONFIG_PAGE_TABLE_ISOLATION
+#include "error.h"
#include "misc.h"
/* These actually do the work of building the kernel identity maps. */
#include <linux/pgtable.h>
+#include <asm/trap_pf.h>
+#include <asm/trapnr.h>
#include <asm/init.h>
/* Use the static base for this part of the boot process */
#undef __PAGE_OFFSET
@@ -160,3 +163,39 @@ void finalize_identity_maps(void)
{
write_cr3(top_level_pgt);
}
+
+static void do_pf_error(const char *msg, unsigned long error_code,
+ unsigned long address, unsigned long ip)
+{
+ error_putstr(msg);
+
+ error_putstr("\nError Code: ");
+ error_puthex(error_code);
+ error_putstr("\nCR2: 0x");
+ error_puthex(address);
+ error_putstr("\nRIP relative to _head: 0x");
+ error_puthex(ip - (unsigned long)_head);
+ error_putstr("\n");
+
+ error("Stopping.\n");
+}
+
+void do_boot_page_fault(struct pt_regs *regs, unsigned long error_code)
+{
+ unsigned long address = native_read_cr2();
+
+ /*
+ * Check for unexpected error codes. Unexpected are:
+ * - Faults on present pages
+ * - User faults
+ * - Reserved bits set
+ */
+ if (error_code & (X86_PF_PROT | X86_PF_USER | X86_PF_RSVD))
+ do_pf_error("Unexpected page-fault:", error_code, address, regs->ip);
+
+ /*
+ * Error code is sane - now identity map the 2M region around
+ * the faulting address.
+ */
+ add_identity_map(address & PMD_MASK, PMD_SIZE);
+}
diff --git a/arch/x86/boot/compressed/idt_64.c b/arch/x86/boot/compressed/idt_64.c
index 082cd6bca033..5f083092a86d 100644
--- a/arch/x86/boot/compressed/idt_64.c
+++ b/arch/x86/boot/compressed/idt_64.c
@@ -40,5 +40,7 @@ void load_stage2_idt(void)
{
boot_idt_desc.address = (unsigned long)boot_idt;
+ set_idt_entry(X86_TRAP_PF, boot_page_fault);
+
load_boot_idt(&boot_idt_desc);
}
diff --git a/arch/x86/boot/compressed/idt_handlers_64.S b/arch/x86/boot/compressed/idt_handlers_64.S
index 36dee2f40a8b..b20e57504a94 100644
--- a/arch/x86/boot/compressed/idt_handlers_64.S
+++ b/arch/x86/boot/compressed/idt_handlers_64.S
@@ -68,3 +68,5 @@ SYM_FUNC_END(\name)
.text
.code64
+
+EXCEPTION_HANDLER boot_page_fault do_boot_page_fault error_code=1
diff --git a/arch/x86/boot/compressed/misc.h b/arch/x86/boot/compressed/misc.h
index 98b7a1df9c59..f0e199174c5f 100644
--- a/arch/x86/boot/compressed/misc.h
+++ b/arch/x86/boot/compressed/misc.h
@@ -37,6 +37,9 @@
#define memptr unsigned
#endif
+/* boot/compressed/vmlinux start and end markers */
+extern char _head[], _end[];
+
/* misc.c */
extern memptr free_mem_ptr;
extern memptr free_mem_end_ptr;
@@ -146,4 +149,7 @@ extern pteval_t __default_kernel_pte_mask;
extern gate_desc boot_idt[BOOT_IDT_ENTRIES];
extern struct desc_ptr boot_idt_desc;
+/* IDT Entry Points */
+void boot_page_fault(void);
+
#endif /* BOOT_COMPRESSED_MISC_H */
--
2.28.0
^ permalink raw reply related [flat|nested] 264+ messages in thread
* [PATCH v7 15/72] x86/boot/compressed/64: Add page-fault handler
@ 2020-09-07 13:15 ` Joerg Roedel
0 siblings, 0 replies; 264+ messages in thread
From: Joerg Roedel @ 2020-09-07 13:15 UTC (permalink / raw)
To: x86
Cc: Juergen Gross, Tom Lendacky, Joerg Roedel, Mike Stunes,
Kees Cook, kvm, Peter Zijlstra, Cfir Cohen, Joerg Roedel,
Dave Hansen, linux-kernel, Sean Christopherson, virtualization,
Martin Radev, Masami Hiramatsu, Andy Lutomirski, hpa,
Erdem Aktas, David Rientjes, Dan Williams, Jiri Slaby
From: Joerg Roedel <jroedel@suse.de>
Install a page-fault handler to add an identity mapping to addresses
not yet mapped. Also do some checking whether the error code is sane.
This makes non SEV-ES machines use the exception handling
infrastructure in the pre-decompressions boot code too, making it less
likely to break in the future.
Signed-off-by: Joerg Roedel <jroedel@suse.de>
Reviewed-by: Kees Cook <keescook@chromium.org>
---
arch/x86/boot/compressed/ident_map_64.c | 39 ++++++++++++++++++++++
arch/x86/boot/compressed/idt_64.c | 2 ++
arch/x86/boot/compressed/idt_handlers_64.S | 2 ++
arch/x86/boot/compressed/misc.h | 6 ++++
4 files changed, 49 insertions(+)
diff --git a/arch/x86/boot/compressed/ident_map_64.c b/arch/x86/boot/compressed/ident_map_64.c
index d9932a133ac9..e3d980ae9c2b 100644
--- a/arch/x86/boot/compressed/ident_map_64.c
+++ b/arch/x86/boot/compressed/ident_map_64.c
@@ -19,10 +19,13 @@
/* No PAGE_TABLE_ISOLATION support needed either: */
#undef CONFIG_PAGE_TABLE_ISOLATION
+#include "error.h"
#include "misc.h"
/* These actually do the work of building the kernel identity maps. */
#include <linux/pgtable.h>
+#include <asm/trap_pf.h>
+#include <asm/trapnr.h>
#include <asm/init.h>
/* Use the static base for this part of the boot process */
#undef __PAGE_OFFSET
@@ -160,3 +163,39 @@ void finalize_identity_maps(void)
{
write_cr3(top_level_pgt);
}
+
+static void do_pf_error(const char *msg, unsigned long error_code,
+ unsigned long address, unsigned long ip)
+{
+ error_putstr(msg);
+
+ error_putstr("\nError Code: ");
+ error_puthex(error_code);
+ error_putstr("\nCR2: 0x");
+ error_puthex(address);
+ error_putstr("\nRIP relative to _head: 0x");
+ error_puthex(ip - (unsigned long)_head);
+ error_putstr("\n");
+
+ error("Stopping.\n");
+}
+
+void do_boot_page_fault(struct pt_regs *regs, unsigned long error_code)
+{
+ unsigned long address = native_read_cr2();
+
+ /*
+ * Check for unexpected error codes. Unexpected are:
+ * - Faults on present pages
+ * - User faults
+ * - Reserved bits set
+ */
+ if (error_code & (X86_PF_PROT | X86_PF_USER | X86_PF_RSVD))
+ do_pf_error("Unexpected page-fault:", error_code, address, regs->ip);
+
+ /*
+ * Error code is sane - now identity map the 2M region around
+ * the faulting address.
+ */
+ add_identity_map(address & PMD_MASK, PMD_SIZE);
+}
diff --git a/arch/x86/boot/compressed/idt_64.c b/arch/x86/boot/compressed/idt_64.c
index 082cd6bca033..5f083092a86d 100644
--- a/arch/x86/boot/compressed/idt_64.c
+++ b/arch/x86/boot/compressed/idt_64.c
@@ -40,5 +40,7 @@ void load_stage2_idt(void)
{
boot_idt_desc.address = (unsigned long)boot_idt;
+ set_idt_entry(X86_TRAP_PF, boot_page_fault);
+
load_boot_idt(&boot_idt_desc);
}
diff --git a/arch/x86/boot/compressed/idt_handlers_64.S b/arch/x86/boot/compressed/idt_handlers_64.S
index 36dee2f40a8b..b20e57504a94 100644
--- a/arch/x86/boot/compressed/idt_handlers_64.S
+++ b/arch/x86/boot/compressed/idt_handlers_64.S
@@ -68,3 +68,5 @@ SYM_FUNC_END(\name)
.text
.code64
+
+EXCEPTION_HANDLER boot_page_fault do_boot_page_fault error_code=1
diff --git a/arch/x86/boot/compressed/misc.h b/arch/x86/boot/compressed/misc.h
index 98b7a1df9c59..f0e199174c5f 100644
--- a/arch/x86/boot/compressed/misc.h
+++ b/arch/x86/boot/compressed/misc.h
@@ -37,6 +37,9 @@
#define memptr unsigned
#endif
+/* boot/compressed/vmlinux start and end markers */
+extern char _head[], _end[];
+
/* misc.c */
extern memptr free_mem_ptr;
extern memptr free_mem_end_ptr;
@@ -146,4 +149,7 @@ extern pteval_t __default_kernel_pte_mask;
extern gate_desc boot_idt[BOOT_IDT_ENTRIES];
extern struct desc_ptr boot_idt_desc;
+/* IDT Entry Points */
+void boot_page_fault(void);
+
#endif /* BOOT_COMPRESSED_MISC_H */
--
2.28.0
_______________________________________________
Virtualization mailing list
Virtualization@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/virtualization
^ permalink raw reply related [flat|nested] 264+ messages in thread
* [PATCH v7 16/72] x86/boot/compressed/64: Always switch to own page-table
2020-09-07 13:15 ` Joerg Roedel
@ 2020-09-07 13:15 ` Joerg Roedel
-1 siblings, 0 replies; 264+ messages in thread
From: Joerg Roedel @ 2020-09-07 13:15 UTC (permalink / raw)
To: x86
Cc: Joerg Roedel, Joerg Roedel, Kees Cook, hpa, Andy Lutomirski,
Dave Hansen, Peter Zijlstra, Jiri Slaby, Dan Williams,
Tom Lendacky, Juergen Gross, David Rientjes, Cfir Cohen,
Erdem Aktas, Masami Hiramatsu, Mike Stunes, Sean Christopherson,
Martin Radev, linux-kernel, kvm, virtualization
From: Joerg Roedel <jroedel@suse.de>
When booted through startup_64 the kernel keeps running on the EFI
page-table until the KASLR code sets up its own page-table. Without
KASLR the pre-decompression boot code never switches off the EFI
page-table. Change that by unconditionally switching to a kernel
controlled page-table after relocation.
This makes sure we can make changes to the mapping when necessary, for
example map pages unencrypted in SEV and SEV-ES guests.
Also remove the debug_putstr() calls in initialize_identity_maps()
because the function now runs before console_init() is called.
Signed-off-by: Joerg Roedel <jroedel@suse.de>
Reviewed-by: Kees Cook <keescook@chromium.org>
---
arch/x86/boot/compressed/head_64.S | 3 +-
arch/x86/boot/compressed/ident_map_64.c | 51 +++++++++++++++----------
arch/x86/boot/compressed/kaslr.c | 3 --
3 files changed, 32 insertions(+), 25 deletions(-)
diff --git a/arch/x86/boot/compressed/head_64.S b/arch/x86/boot/compressed/head_64.S
index 1fffaec1c069..37c8c2c8f8b6 100644
--- a/arch/x86/boot/compressed/head_64.S
+++ b/arch/x86/boot/compressed/head_64.S
@@ -533,10 +533,11 @@ SYM_FUNC_START_LOCAL_NOALIGN(.Lrelocated)
rep stosq
/*
- * Load stage2 IDT
+ * Load stage2 IDT and switch to our own page-table
*/
pushq %rsi
call load_stage2_idt
+ call initialize_identity_maps
popq %rsi
/*
diff --git a/arch/x86/boot/compressed/ident_map_64.c b/arch/x86/boot/compressed/ident_map_64.c
index e3d980ae9c2b..ecf9353b064d 100644
--- a/arch/x86/boot/compressed/ident_map_64.c
+++ b/arch/x86/boot/compressed/ident_map_64.c
@@ -86,9 +86,31 @@ phys_addr_t physical_mask = (1ULL << __PHYSICAL_MASK_SHIFT) - 1;
*/
static struct x86_mapping_info mapping_info;
+/*
+ * Adds the specified range to what will become the new identity mappings.
+ * Once all ranges have been added, the new mapping is activated by calling
+ * finalize_identity_maps() below.
+ */
+void add_identity_map(unsigned long start, unsigned long size)
+{
+ unsigned long end = start + size;
+
+ /* Align boundary to 2M. */
+ start = round_down(start, PMD_SIZE);
+ end = round_up(end, PMD_SIZE);
+ if (start >= end)
+ return;
+
+ /* Build the mapping. */
+ kernel_ident_mapping_init(&mapping_info, (pgd_t *)top_level_pgt,
+ start, end);
+}
+
/* Locates and clears a region for a new top level page table. */
void initialize_identity_maps(void)
{
+ unsigned long start, size;
+
/* If running as an SEV guest, the encryption mask is required. */
set_sev_encryption_mask();
@@ -121,37 +143,24 @@ void initialize_identity_maps(void)
*/
top_level_pgt = read_cr3_pa();
if (p4d_offset((pgd_t *)top_level_pgt, 0) == (p4d_t *)_pgtable) {
- debug_putstr("booted via startup_32()\n");
pgt_data.pgt_buf = _pgtable + BOOT_INIT_PGT_SIZE;
pgt_data.pgt_buf_size = BOOT_PGT_SIZE - BOOT_INIT_PGT_SIZE;
memset(pgt_data.pgt_buf, 0, pgt_data.pgt_buf_size);
} else {
- debug_putstr("booted via startup_64()\n");
pgt_data.pgt_buf = _pgtable;
pgt_data.pgt_buf_size = BOOT_PGT_SIZE;
memset(pgt_data.pgt_buf, 0, pgt_data.pgt_buf_size);
top_level_pgt = (unsigned long)alloc_pgt_page(&pgt_data);
}
-}
-/*
- * Adds the specified range to what will become the new identity mappings.
- * Once all ranges have been added, the new mapping is activated by calling
- * finalize_identity_maps() below.
- */
-void add_identity_map(unsigned long start, unsigned long size)
-{
- unsigned long end = start + size;
-
- /* Align boundary to 2M. */
- start = round_down(start, PMD_SIZE);
- end = round_up(end, PMD_SIZE);
- if (start >= end)
- return;
-
- /* Build the mapping. */
- kernel_ident_mapping_init(&mapping_info, (pgd_t *)top_level_pgt,
- start, end);
+ /*
+ * New page-table is set up - map the kernel image and load it
+ * into cr3.
+ */
+ start = (unsigned long)_head;
+ size = _end - _head;
+ add_identity_map(start, size);
+ write_cr3(top_level_pgt);
}
/*
diff --git a/arch/x86/boot/compressed/kaslr.c b/arch/x86/boot/compressed/kaslr.c
index e27de98ed038..82662869c4cb 100644
--- a/arch/x86/boot/compressed/kaslr.c
+++ b/arch/x86/boot/compressed/kaslr.c
@@ -861,9 +861,6 @@ void choose_random_location(unsigned long input,
boot_params->hdr.loadflags |= KASLR_FLAG;
- /* Prepare to add new identity pagetables on demand. */
- initialize_identity_maps();
-
if (IS_ENABLED(CONFIG_X86_32))
mem_limit = KERNEL_IMAGE_SIZE;
else
--
2.28.0
^ permalink raw reply related [flat|nested] 264+ messages in thread
* [PATCH v7 16/72] x86/boot/compressed/64: Always switch to own page-table
@ 2020-09-07 13:15 ` Joerg Roedel
0 siblings, 0 replies; 264+ messages in thread
From: Joerg Roedel @ 2020-09-07 13:15 UTC (permalink / raw)
To: x86
Cc: Juergen Gross, Tom Lendacky, Joerg Roedel, Mike Stunes,
Kees Cook, kvm, Peter Zijlstra, Cfir Cohen, Joerg Roedel,
Dave Hansen, linux-kernel, Sean Christopherson, virtualization,
Martin Radev, Masami Hiramatsu, Andy Lutomirski, hpa,
Erdem Aktas, David Rientjes, Dan Williams, Jiri Slaby
From: Joerg Roedel <jroedel@suse.de>
When booted through startup_64 the kernel keeps running on the EFI
page-table until the KASLR code sets up its own page-table. Without
KASLR the pre-decompression boot code never switches off the EFI
page-table. Change that by unconditionally switching to a kernel
controlled page-table after relocation.
This makes sure we can make changes to the mapping when necessary, for
example map pages unencrypted in SEV and SEV-ES guests.
Also remove the debug_putstr() calls in initialize_identity_maps()
because the function now runs before console_init() is called.
Signed-off-by: Joerg Roedel <jroedel@suse.de>
Reviewed-by: Kees Cook <keescook@chromium.org>
---
arch/x86/boot/compressed/head_64.S | 3 +-
arch/x86/boot/compressed/ident_map_64.c | 51 +++++++++++++++----------
arch/x86/boot/compressed/kaslr.c | 3 --
3 files changed, 32 insertions(+), 25 deletions(-)
diff --git a/arch/x86/boot/compressed/head_64.S b/arch/x86/boot/compressed/head_64.S
index 1fffaec1c069..37c8c2c8f8b6 100644
--- a/arch/x86/boot/compressed/head_64.S
+++ b/arch/x86/boot/compressed/head_64.S
@@ -533,10 +533,11 @@ SYM_FUNC_START_LOCAL_NOALIGN(.Lrelocated)
rep stosq
/*
- * Load stage2 IDT
+ * Load stage2 IDT and switch to our own page-table
*/
pushq %rsi
call load_stage2_idt
+ call initialize_identity_maps
popq %rsi
/*
diff --git a/arch/x86/boot/compressed/ident_map_64.c b/arch/x86/boot/compressed/ident_map_64.c
index e3d980ae9c2b..ecf9353b064d 100644
--- a/arch/x86/boot/compressed/ident_map_64.c
+++ b/arch/x86/boot/compressed/ident_map_64.c
@@ -86,9 +86,31 @@ phys_addr_t physical_mask = (1ULL << __PHYSICAL_MASK_SHIFT) - 1;
*/
static struct x86_mapping_info mapping_info;
+/*
+ * Adds the specified range to what will become the new identity mappings.
+ * Once all ranges have been added, the new mapping is activated by calling
+ * finalize_identity_maps() below.
+ */
+void add_identity_map(unsigned long start, unsigned long size)
+{
+ unsigned long end = start + size;
+
+ /* Align boundary to 2M. */
+ start = round_down(start, PMD_SIZE);
+ end = round_up(end, PMD_SIZE);
+ if (start >= end)
+ return;
+
+ /* Build the mapping. */
+ kernel_ident_mapping_init(&mapping_info, (pgd_t *)top_level_pgt,
+ start, end);
+}
+
/* Locates and clears a region for a new top level page table. */
void initialize_identity_maps(void)
{
+ unsigned long start, size;
+
/* If running as an SEV guest, the encryption mask is required. */
set_sev_encryption_mask();
@@ -121,37 +143,24 @@ void initialize_identity_maps(void)
*/
top_level_pgt = read_cr3_pa();
if (p4d_offset((pgd_t *)top_level_pgt, 0) == (p4d_t *)_pgtable) {
- debug_putstr("booted via startup_32()\n");
pgt_data.pgt_buf = _pgtable + BOOT_INIT_PGT_SIZE;
pgt_data.pgt_buf_size = BOOT_PGT_SIZE - BOOT_INIT_PGT_SIZE;
memset(pgt_data.pgt_buf, 0, pgt_data.pgt_buf_size);
} else {
- debug_putstr("booted via startup_64()\n");
pgt_data.pgt_buf = _pgtable;
pgt_data.pgt_buf_size = BOOT_PGT_SIZE;
memset(pgt_data.pgt_buf, 0, pgt_data.pgt_buf_size);
top_level_pgt = (unsigned long)alloc_pgt_page(&pgt_data);
}
-}
-/*
- * Adds the specified range to what will become the new identity mappings.
- * Once all ranges have been added, the new mapping is activated by calling
- * finalize_identity_maps() below.
- */
-void add_identity_map(unsigned long start, unsigned long size)
-{
- unsigned long end = start + size;
-
- /* Align boundary to 2M. */
- start = round_down(start, PMD_SIZE);
- end = round_up(end, PMD_SIZE);
- if (start >= end)
- return;
-
- /* Build the mapping. */
- kernel_ident_mapping_init(&mapping_info, (pgd_t *)top_level_pgt,
- start, end);
+ /*
+ * New page-table is set up - map the kernel image and load it
+ * into cr3.
+ */
+ start = (unsigned long)_head;
+ size = _end - _head;
+ add_identity_map(start, size);
+ write_cr3(top_level_pgt);
}
/*
diff --git a/arch/x86/boot/compressed/kaslr.c b/arch/x86/boot/compressed/kaslr.c
index e27de98ed038..82662869c4cb 100644
--- a/arch/x86/boot/compressed/kaslr.c
+++ b/arch/x86/boot/compressed/kaslr.c
@@ -861,9 +861,6 @@ void choose_random_location(unsigned long input,
boot_params->hdr.loadflags |= KASLR_FLAG;
- /* Prepare to add new identity pagetables on demand. */
- initialize_identity_maps();
-
if (IS_ENABLED(CONFIG_X86_32))
mem_limit = KERNEL_IMAGE_SIZE;
else
--
2.28.0
_______________________________________________
Virtualization mailing list
Virtualization@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/virtualization
^ permalink raw reply related [flat|nested] 264+ messages in thread
* [PATCH v7 17/72] x86/boot/compressed/64: Don't pre-map memory in KASLR code
2020-09-07 13:15 ` Joerg Roedel
@ 2020-09-07 13:15 ` Joerg Roedel
-1 siblings, 0 replies; 264+ messages in thread
From: Joerg Roedel @ 2020-09-07 13:15 UTC (permalink / raw)
To: x86
Cc: Joerg Roedel, Joerg Roedel, Kees Cook, hpa, Andy Lutomirski,
Dave Hansen, Peter Zijlstra, Jiri Slaby, Dan Williams,
Tom Lendacky, Juergen Gross, David Rientjes, Cfir Cohen,
Erdem Aktas, Masami Hiramatsu, Mike Stunes, Sean Christopherson,
Martin Radev, linux-kernel, kvm, virtualization
From: Joerg Roedel <jroedel@suse.de>
With the page-fault handler in place the identity mapping can be built
on-demand. So remove the code which manually creates the mappings and
unexport/remove the functions used for it.
Signed-off-by: Joerg Roedel <jroedel@suse.de>
Reviewed-by: Kees Cook <keescook@chromium.org>
---
arch/x86/boot/compressed/ident_map_64.c | 6 ++----
arch/x86/boot/compressed/kaslr.c | 24 +-----------------------
arch/x86/boot/compressed/misc.h | 10 ----------
3 files changed, 3 insertions(+), 37 deletions(-)
diff --git a/arch/x86/boot/compressed/ident_map_64.c b/arch/x86/boot/compressed/ident_map_64.c
index ecf9353b064d..c63257bf8373 100644
--- a/arch/x86/boot/compressed/ident_map_64.c
+++ b/arch/x86/boot/compressed/ident_map_64.c
@@ -87,11 +87,9 @@ phys_addr_t physical_mask = (1ULL << __PHYSICAL_MASK_SHIFT) - 1;
static struct x86_mapping_info mapping_info;
/*
- * Adds the specified range to what will become the new identity mappings.
- * Once all ranges have been added, the new mapping is activated by calling
- * finalize_identity_maps() below.
+ * Adds the specified range to the identity mappings.
*/
-void add_identity_map(unsigned long start, unsigned long size)
+static void add_identity_map(unsigned long start, unsigned long size)
{
unsigned long end = start + size;
diff --git a/arch/x86/boot/compressed/kaslr.c b/arch/x86/boot/compressed/kaslr.c
index 82662869c4cb..b59547ce5b19 100644
--- a/arch/x86/boot/compressed/kaslr.c
+++ b/arch/x86/boot/compressed/kaslr.c
@@ -397,8 +397,6 @@ static void mem_avoid_init(unsigned long input, unsigned long input_size,
*/
mem_avoid[MEM_AVOID_ZO_RANGE].start = input;
mem_avoid[MEM_AVOID_ZO_RANGE].size = (output + init_size) - input;
- add_identity_map(mem_avoid[MEM_AVOID_ZO_RANGE].start,
- mem_avoid[MEM_AVOID_ZO_RANGE].size);
/* Avoid initrd. */
initrd_start = (u64)boot_params->ext_ramdisk_image << 32;
@@ -416,15 +414,11 @@ static void mem_avoid_init(unsigned long input, unsigned long input_size,
cmd_line_size = strnlen((char *)cmd_line, COMMAND_LINE_SIZE-1) + 1;
mem_avoid[MEM_AVOID_CMDLINE].start = cmd_line;
mem_avoid[MEM_AVOID_CMDLINE].size = cmd_line_size;
- add_identity_map(mem_avoid[MEM_AVOID_CMDLINE].start,
- mem_avoid[MEM_AVOID_CMDLINE].size);
}
/* Avoid boot parameters. */
mem_avoid[MEM_AVOID_BOOTPARAMS].start = (unsigned long)boot_params;
mem_avoid[MEM_AVOID_BOOTPARAMS].size = sizeof(*boot_params);
- add_identity_map(mem_avoid[MEM_AVOID_BOOTPARAMS].start,
- mem_avoid[MEM_AVOID_BOOTPARAMS].size);
/* We don't need to set a mapping for setup_data. */
@@ -433,11 +427,6 @@ static void mem_avoid_init(unsigned long input, unsigned long input_size,
/* Enumerate the immovable memory regions */
num_immovable_mem = count_immovable_mem_regions();
-
-#ifdef CONFIG_X86_VERBOSE_BOOTUP
- /* Make sure video RAM can be used. */
- add_identity_map(0, PMD_SIZE);
-#endif
}
/*
@@ -884,19 +873,8 @@ void choose_random_location(unsigned long input,
warn("Physical KASLR disabled: no suitable memory region!");
} else {
/* Update the new physical address location. */
- if (*output != random_addr) {
- add_identity_map(random_addr, output_size);
+ if (*output != random_addr)
*output = random_addr;
- }
-
- /*
- * This loads the identity mapping page table.
- * This should only be done if a new physical address
- * is found for the kernel, otherwise we should keep
- * the old page table to make it be like the "nokaslr"
- * case.
- */
- finalize_identity_maps();
}
diff --git a/arch/x86/boot/compressed/misc.h b/arch/x86/boot/compressed/misc.h
index f0e199174c5f..9840c82a39f1 100644
--- a/arch/x86/boot/compressed/misc.h
+++ b/arch/x86/boot/compressed/misc.h
@@ -98,17 +98,7 @@ static inline void choose_random_location(unsigned long input,
#endif
#ifdef CONFIG_X86_64
-void initialize_identity_maps(void);
-void add_identity_map(unsigned long start, unsigned long size);
-void finalize_identity_maps(void);
extern unsigned char _pgtable[];
-#else
-static inline void initialize_identity_maps(void)
-{ }
-static inline void add_identity_map(unsigned long start, unsigned long size)
-{ }
-static inline void finalize_identity_maps(void)
-{ }
#endif
#ifdef CONFIG_EARLY_PRINTK
--
2.28.0
^ permalink raw reply related [flat|nested] 264+ messages in thread
* [PATCH v7 17/72] x86/boot/compressed/64: Don't pre-map memory in KASLR code
@ 2020-09-07 13:15 ` Joerg Roedel
0 siblings, 0 replies; 264+ messages in thread
From: Joerg Roedel @ 2020-09-07 13:15 UTC (permalink / raw)
To: x86
Cc: Juergen Gross, Tom Lendacky, Joerg Roedel, Mike Stunes,
Kees Cook, kvm, Peter Zijlstra, Cfir Cohen, Joerg Roedel,
Dave Hansen, linux-kernel, Sean Christopherson, virtualization,
Martin Radev, Masami Hiramatsu, Andy Lutomirski, hpa,
Erdem Aktas, David Rientjes, Dan Williams, Jiri Slaby
From: Joerg Roedel <jroedel@suse.de>
With the page-fault handler in place the identity mapping can be built
on-demand. So remove the code which manually creates the mappings and
unexport/remove the functions used for it.
Signed-off-by: Joerg Roedel <jroedel@suse.de>
Reviewed-by: Kees Cook <keescook@chromium.org>
---
arch/x86/boot/compressed/ident_map_64.c | 6 ++----
arch/x86/boot/compressed/kaslr.c | 24 +-----------------------
arch/x86/boot/compressed/misc.h | 10 ----------
3 files changed, 3 insertions(+), 37 deletions(-)
diff --git a/arch/x86/boot/compressed/ident_map_64.c b/arch/x86/boot/compressed/ident_map_64.c
index ecf9353b064d..c63257bf8373 100644
--- a/arch/x86/boot/compressed/ident_map_64.c
+++ b/arch/x86/boot/compressed/ident_map_64.c
@@ -87,11 +87,9 @@ phys_addr_t physical_mask = (1ULL << __PHYSICAL_MASK_SHIFT) - 1;
static struct x86_mapping_info mapping_info;
/*
- * Adds the specified range to what will become the new identity mappings.
- * Once all ranges have been added, the new mapping is activated by calling
- * finalize_identity_maps() below.
+ * Adds the specified range to the identity mappings.
*/
-void add_identity_map(unsigned long start, unsigned long size)
+static void add_identity_map(unsigned long start, unsigned long size)
{
unsigned long end = start + size;
diff --git a/arch/x86/boot/compressed/kaslr.c b/arch/x86/boot/compressed/kaslr.c
index 82662869c4cb..b59547ce5b19 100644
--- a/arch/x86/boot/compressed/kaslr.c
+++ b/arch/x86/boot/compressed/kaslr.c
@@ -397,8 +397,6 @@ static void mem_avoid_init(unsigned long input, unsigned long input_size,
*/
mem_avoid[MEM_AVOID_ZO_RANGE].start = input;
mem_avoid[MEM_AVOID_ZO_RANGE].size = (output + init_size) - input;
- add_identity_map(mem_avoid[MEM_AVOID_ZO_RANGE].start,
- mem_avoid[MEM_AVOID_ZO_RANGE].size);
/* Avoid initrd. */
initrd_start = (u64)boot_params->ext_ramdisk_image << 32;
@@ -416,15 +414,11 @@ static void mem_avoid_init(unsigned long input, unsigned long input_size,
cmd_line_size = strnlen((char *)cmd_line, COMMAND_LINE_SIZE-1) + 1;
mem_avoid[MEM_AVOID_CMDLINE].start = cmd_line;
mem_avoid[MEM_AVOID_CMDLINE].size = cmd_line_size;
- add_identity_map(mem_avoid[MEM_AVOID_CMDLINE].start,
- mem_avoid[MEM_AVOID_CMDLINE].size);
}
/* Avoid boot parameters. */
mem_avoid[MEM_AVOID_BOOTPARAMS].start = (unsigned long)boot_params;
mem_avoid[MEM_AVOID_BOOTPARAMS].size = sizeof(*boot_params);
- add_identity_map(mem_avoid[MEM_AVOID_BOOTPARAMS].start,
- mem_avoid[MEM_AVOID_BOOTPARAMS].size);
/* We don't need to set a mapping for setup_data. */
@@ -433,11 +427,6 @@ static void mem_avoid_init(unsigned long input, unsigned long input_size,
/* Enumerate the immovable memory regions */
num_immovable_mem = count_immovable_mem_regions();
-
-#ifdef CONFIG_X86_VERBOSE_BOOTUP
- /* Make sure video RAM can be used. */
- add_identity_map(0, PMD_SIZE);
-#endif
}
/*
@@ -884,19 +873,8 @@ void choose_random_location(unsigned long input,
warn("Physical KASLR disabled: no suitable memory region!");
} else {
/* Update the new physical address location. */
- if (*output != random_addr) {
- add_identity_map(random_addr, output_size);
+ if (*output != random_addr)
*output = random_addr;
- }
-
- /*
- * This loads the identity mapping page table.
- * This should only be done if a new physical address
- * is found for the kernel, otherwise we should keep
- * the old page table to make it be like the "nokaslr"
- * case.
- */
- finalize_identity_maps();
}
diff --git a/arch/x86/boot/compressed/misc.h b/arch/x86/boot/compressed/misc.h
index f0e199174c5f..9840c82a39f1 100644
--- a/arch/x86/boot/compressed/misc.h
+++ b/arch/x86/boot/compressed/misc.h
@@ -98,17 +98,7 @@ static inline void choose_random_location(unsigned long input,
#endif
#ifdef CONFIG_X86_64
-void initialize_identity_maps(void);
-void add_identity_map(unsigned long start, unsigned long size);
-void finalize_identity_maps(void);
extern unsigned char _pgtable[];
-#else
-static inline void initialize_identity_maps(void)
-{ }
-static inline void add_identity_map(unsigned long start, unsigned long size)
-{ }
-static inline void finalize_identity_maps(void)
-{ }
#endif
#ifdef CONFIG_EARLY_PRINTK
--
2.28.0
_______________________________________________
Virtualization mailing list
Virtualization@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/virtualization
^ permalink raw reply related [flat|nested] 264+ messages in thread
* [PATCH v7 18/72] x86/boot/compressed/64: Change add_identity_map() to take start and end
2020-09-07 13:15 ` Joerg Roedel
@ 2020-09-07 13:15 ` Joerg Roedel
-1 siblings, 0 replies; 264+ messages in thread
From: Joerg Roedel @ 2020-09-07 13:15 UTC (permalink / raw)
To: x86
Cc: Joerg Roedel, Joerg Roedel, Kees Cook, hpa, Andy Lutomirski,
Dave Hansen, Peter Zijlstra, Jiri Slaby, Dan Williams,
Tom Lendacky, Juergen Gross, David Rientjes, Cfir Cohen,
Erdem Aktas, Masami Hiramatsu, Mike Stunes, Sean Christopherson,
Martin Radev, linux-kernel, kvm, virtualization
From: Joerg Roedel <jroedel@suse.de>
Changing the function to take start and end as parameters instead of
start and size simplifies the callers, which don't need to calculate
the size if they already have start and end.
Signed-off-by: Joerg Roedel <jroedel@suse.de>
Reviewed-by: Kees Cook <keescook@chromium.org>
---
arch/x86/boot/compressed/ident_map_64.c | 15 +++++----------
1 file changed, 5 insertions(+), 10 deletions(-)
diff --git a/arch/x86/boot/compressed/ident_map_64.c b/arch/x86/boot/compressed/ident_map_64.c
index c63257bf8373..62e42c11a336 100644
--- a/arch/x86/boot/compressed/ident_map_64.c
+++ b/arch/x86/boot/compressed/ident_map_64.c
@@ -89,10 +89,8 @@ static struct x86_mapping_info mapping_info;
/*
* Adds the specified range to the identity mappings.
*/
-static void add_identity_map(unsigned long start, unsigned long size)
+static void add_identity_map(unsigned long start, unsigned long end)
{
- unsigned long end = start + size;
-
/* Align boundary to 2M. */
start = round_down(start, PMD_SIZE);
end = round_up(end, PMD_SIZE);
@@ -107,8 +105,6 @@ static void add_identity_map(unsigned long start, unsigned long size)
/* Locates and clears a region for a new top level page table. */
void initialize_identity_maps(void)
{
- unsigned long start, size;
-
/* If running as an SEV guest, the encryption mask is required. */
set_sev_encryption_mask();
@@ -155,9 +151,7 @@ void initialize_identity_maps(void)
* New page-table is set up - map the kernel image and load it
* into cr3.
*/
- start = (unsigned long)_head;
- size = _end - _head;
- add_identity_map(start, size);
+ add_identity_map((unsigned long)_head, (unsigned long)_end);
write_cr3(top_level_pgt);
}
@@ -189,7 +183,8 @@ static void do_pf_error(const char *msg, unsigned long error_code,
void do_boot_page_fault(struct pt_regs *regs, unsigned long error_code)
{
- unsigned long address = native_read_cr2();
+ unsigned long address = native_read_cr2() & PMD_MASK;
+ unsigned long end = address + PMD_SIZE;
/*
* Check for unexpected error codes. Unexpected are:
@@ -204,5 +199,5 @@ void do_boot_page_fault(struct pt_regs *regs, unsigned long error_code)
* Error code is sane - now identity map the 2M region around
* the faulting address.
*/
- add_identity_map(address & PMD_MASK, PMD_SIZE);
+ add_identity_map(address, end);
}
--
2.28.0
^ permalink raw reply related [flat|nested] 264+ messages in thread
* [PATCH v7 18/72] x86/boot/compressed/64: Change add_identity_map() to take start and end
@ 2020-09-07 13:15 ` Joerg Roedel
0 siblings, 0 replies; 264+ messages in thread
From: Joerg Roedel @ 2020-09-07 13:15 UTC (permalink / raw)
To: x86
Cc: Juergen Gross, Tom Lendacky, Joerg Roedel, Mike Stunes,
Kees Cook, kvm, Peter Zijlstra, Cfir Cohen, Joerg Roedel,
Dave Hansen, linux-kernel, Sean Christopherson, virtualization,
Martin Radev, Masami Hiramatsu, Andy Lutomirski, hpa,
Erdem Aktas, David Rientjes, Dan Williams, Jiri Slaby
From: Joerg Roedel <jroedel@suse.de>
Changing the function to take start and end as parameters instead of
start and size simplifies the callers, which don't need to calculate
the size if they already have start and end.
Signed-off-by: Joerg Roedel <jroedel@suse.de>
Reviewed-by: Kees Cook <keescook@chromium.org>
---
arch/x86/boot/compressed/ident_map_64.c | 15 +++++----------
1 file changed, 5 insertions(+), 10 deletions(-)
diff --git a/arch/x86/boot/compressed/ident_map_64.c b/arch/x86/boot/compressed/ident_map_64.c
index c63257bf8373..62e42c11a336 100644
--- a/arch/x86/boot/compressed/ident_map_64.c
+++ b/arch/x86/boot/compressed/ident_map_64.c
@@ -89,10 +89,8 @@ static struct x86_mapping_info mapping_info;
/*
* Adds the specified range to the identity mappings.
*/
-static void add_identity_map(unsigned long start, unsigned long size)
+static void add_identity_map(unsigned long start, unsigned long end)
{
- unsigned long end = start + size;
-
/* Align boundary to 2M. */
start = round_down(start, PMD_SIZE);
end = round_up(end, PMD_SIZE);
@@ -107,8 +105,6 @@ static void add_identity_map(unsigned long start, unsigned long size)
/* Locates and clears a region for a new top level page table. */
void initialize_identity_maps(void)
{
- unsigned long start, size;
-
/* If running as an SEV guest, the encryption mask is required. */
set_sev_encryption_mask();
@@ -155,9 +151,7 @@ void initialize_identity_maps(void)
* New page-table is set up - map the kernel image and load it
* into cr3.
*/
- start = (unsigned long)_head;
- size = _end - _head;
- add_identity_map(start, size);
+ add_identity_map((unsigned long)_head, (unsigned long)_end);
write_cr3(top_level_pgt);
}
@@ -189,7 +183,8 @@ static void do_pf_error(const char *msg, unsigned long error_code,
void do_boot_page_fault(struct pt_regs *regs, unsigned long error_code)
{
- unsigned long address = native_read_cr2();
+ unsigned long address = native_read_cr2() & PMD_MASK;
+ unsigned long end = address + PMD_SIZE;
/*
* Check for unexpected error codes. Unexpected are:
@@ -204,5 +199,5 @@ void do_boot_page_fault(struct pt_regs *regs, unsigned long error_code)
* Error code is sane - now identity map the 2M region around
* the faulting address.
*/
- add_identity_map(address & PMD_MASK, PMD_SIZE);
+ add_identity_map(address, end);
}
--
2.28.0
_______________________________________________
Virtualization mailing list
Virtualization@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/virtualization
^ permalink raw reply related [flat|nested] 264+ messages in thread
* [PATCH v7 19/72] x86/boot/compressed/64: Add stage1 #VC handler
2020-09-07 13:15 ` Joerg Roedel
@ 2020-09-07 13:15 ` Joerg Roedel
-1 siblings, 0 replies; 264+ messages in thread
From: Joerg Roedel @ 2020-09-07 13:15 UTC (permalink / raw)
To: x86
Cc: Joerg Roedel, Joerg Roedel, hpa, Andy Lutomirski, Dave Hansen,
Peter Zijlstra, Jiri Slaby, Dan Williams, Tom Lendacky,
Juergen Gross, Kees Cook, David Rientjes, Cfir Cohen,
Erdem Aktas, Masami Hiramatsu, Mike Stunes, Sean Christopherson,
Martin Radev, linux-kernel, kvm, virtualization
From: Joerg Roedel <jroedel@suse.de>
Add the first handler for #VC exceptions. At stage 1 there is no GHCB
yet becaue the kernel might still be running on the EFI page table.
The stage 1 handler is limited to the MSR based protocol to talk to
the hypervisor and can only support CPUID exit-codes, but that is
enough to get to stage 2.
Signed-off-by: Joerg Roedel <jroedel@suse.de>
---
arch/x86/boot/compressed/Makefile | 1 +
arch/x86/boot/compressed/idt_64.c | 4 ++
arch/x86/boot/compressed/idt_handlers_64.S | 4 ++
arch/x86/boot/compressed/misc.h | 1 +
arch/x86/boot/compressed/sev-es.c | 45 +++++++++++++++
arch/x86/include/asm/msr-index.h | 1 +
arch/x86/include/asm/sev-es.h | 37 ++++++++++++
arch/x86/include/asm/trapnr.h | 1 +
arch/x86/kernel/sev-es-shared.c | 66 ++++++++++++++++++++++
9 files changed, 160 insertions(+)
create mode 100644 arch/x86/boot/compressed/sev-es.c
create mode 100644 arch/x86/include/asm/sev-es.h
create mode 100644 arch/x86/kernel/sev-es-shared.c
diff --git a/arch/x86/boot/compressed/Makefile b/arch/x86/boot/compressed/Makefile
index c12c9223a1fb..7302f184bece 100644
--- a/arch/x86/boot/compressed/Makefile
+++ b/arch/x86/boot/compressed/Makefile
@@ -83,6 +83,7 @@ ifdef CONFIG_X86_64
vmlinux-objs-y += $(obj)/idt_64.o $(obj)/idt_handlers_64.o
vmlinux-objs-y += $(obj)/mem_encrypt.o
vmlinux-objs-y += $(obj)/pgtable_64.o
+ vmlinux-objs-$(CONFIG_AMD_MEM_ENCRYPT) += $(obj)/sev-es.o
endif
vmlinux-objs-$(CONFIG_ACPI) += $(obj)/acpi.o
diff --git a/arch/x86/boot/compressed/idt_64.c b/arch/x86/boot/compressed/idt_64.c
index 5f083092a86d..f3ca7324be44 100644
--- a/arch/x86/boot/compressed/idt_64.c
+++ b/arch/x86/boot/compressed/idt_64.c
@@ -32,6 +32,10 @@ void load_stage1_idt(void)
{
boot_idt_desc.address = (unsigned long)boot_idt;
+
+ if (IS_ENABLED(CONFIG_AMD_MEM_ENCRYPT))
+ set_idt_entry(X86_TRAP_VC, boot_stage1_vc);
+
load_boot_idt(&boot_idt_desc);
}
diff --git a/arch/x86/boot/compressed/idt_handlers_64.S b/arch/x86/boot/compressed/idt_handlers_64.S
index b20e57504a94..92eb4df478a1 100644
--- a/arch/x86/boot/compressed/idt_handlers_64.S
+++ b/arch/x86/boot/compressed/idt_handlers_64.S
@@ -70,3 +70,7 @@ SYM_FUNC_END(\name)
.code64
EXCEPTION_HANDLER boot_page_fault do_boot_page_fault error_code=1
+
+#ifdef CONFIG_AMD_MEM_ENCRYPT
+EXCEPTION_HANDLER boot_stage1_vc do_vc_no_ghcb error_code=1
+#endif
diff --git a/arch/x86/boot/compressed/misc.h b/arch/x86/boot/compressed/misc.h
index 9840c82a39f1..eaa8b45ebccb 100644
--- a/arch/x86/boot/compressed/misc.h
+++ b/arch/x86/boot/compressed/misc.h
@@ -141,5 +141,6 @@ extern struct desc_ptr boot_idt_desc;
/* IDT Entry Points */
void boot_page_fault(void);
+void boot_stage1_vc(void);
#endif /* BOOT_COMPRESSED_MISC_H */
diff --git a/arch/x86/boot/compressed/sev-es.c b/arch/x86/boot/compressed/sev-es.c
new file mode 100644
index 000000000000..bb91cbb5920e
--- /dev/null
+++ b/arch/x86/boot/compressed/sev-es.c
@@ -0,0 +1,45 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * AMD Encrypted Register State Support
+ *
+ * Author: Joerg Roedel <jroedel@suse.de>
+ */
+
+/*
+ * misc.h needs to be first because it knows how to include the other kernel
+ * headers in the pre-decompression code in a way that does not break
+ * compilation.
+ */
+#include "misc.h"
+
+#include <asm/sev-es.h>
+#include <asm/msr-index.h>
+#include <asm/ptrace.h>
+#include <asm/svm.h>
+
+static inline u64 sev_es_rd_ghcb_msr(void)
+{
+ unsigned long low, high;
+
+ asm volatile("rdmsr\n" : "=a" (low), "=d" (high) :
+ "c" (MSR_AMD64_SEV_ES_GHCB));
+
+ return ((high << 32) | low);
+}
+
+static inline void sev_es_wr_ghcb_msr(u64 val)
+{
+ u32 low, high;
+
+ low = val & 0xffffffffUL;
+ high = val >> 32;
+
+ asm volatile("wrmsr\n" : : "c" (MSR_AMD64_SEV_ES_GHCB),
+ "a"(low), "d" (high) : "memory");
+}
+
+#undef __init
+#define __init
+
+/* Include code for early handlers */
+#include "../../kernel/sev-es-shared.c"
diff --git a/arch/x86/include/asm/msr-index.h b/arch/x86/include/asm/msr-index.h
index dc131b84ac3a..cd6d651ff730 100644
--- a/arch/x86/include/asm/msr-index.h
+++ b/arch/x86/include/asm/msr-index.h
@@ -466,6 +466,7 @@
#define MSR_AMD64_IBSBRTARGET 0xc001103b
#define MSR_AMD64_IBSOPDATA4 0xc001103d
#define MSR_AMD64_IBS_REG_COUNT_MAX 8 /* includes MSR_AMD64_IBSBRTARGET */
+#define MSR_AMD64_SEV_ES_GHCB 0xc0010130
#define MSR_AMD64_SEV 0xc0010131
#define MSR_AMD64_SEV_ENABLED_BIT 0
#define MSR_AMD64_SEV_ENABLED BIT_ULL(MSR_AMD64_SEV_ENABLED_BIT)
diff --git a/arch/x86/include/asm/sev-es.h b/arch/x86/include/asm/sev-es.h
new file mode 100644
index 000000000000..48a44038b5d1
--- /dev/null
+++ b/arch/x86/include/asm/sev-es.h
@@ -0,0 +1,37 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/*
+ * AMD Encrypted Register State Support
+ *
+ * Author: Joerg Roedel <jroedel@suse.de>
+ */
+
+#ifndef __ASM_ENCRYPTED_STATE_H
+#define __ASM_ENCRYPTED_STATE_H
+
+#include <linux/types.h>
+
+#define GHCB_SEV_CPUID_REQ 0x004UL
+#define GHCB_CPUID_REQ_EAX 0
+#define GHCB_CPUID_REQ_EBX 1
+#define GHCB_CPUID_REQ_ECX 2
+#define GHCB_CPUID_REQ_EDX 3
+#define GHCB_CPUID_REQ(fn, reg) (GHCB_SEV_CPUID_REQ | \
+ (((unsigned long)reg & 3) << 30) | \
+ (((unsigned long)fn) << 32))
+
+#define GHCB_SEV_CPUID_RESP 0x005UL
+#define GHCB_SEV_TERMINATE 0x100UL
+
+#define GHCB_SEV_GHCB_RESP_CODE(v) ((v) & 0xfff)
+#define VMGEXIT() { asm volatile("rep; vmmcall\n\r"); }
+
+void do_vc_no_ghcb(struct pt_regs *regs, unsigned long exit_code);
+
+static inline u64 lower_bits(u64 val, unsigned int bits)
+{
+ u64 mask = (1ULL << bits) - 1;
+
+ return (val & mask);
+}
+
+#endif
diff --git a/arch/x86/include/asm/trapnr.h b/arch/x86/include/asm/trapnr.h
index 082f45631fa9..f5d2325aa0b7 100644
--- a/arch/x86/include/asm/trapnr.h
+++ b/arch/x86/include/asm/trapnr.h
@@ -26,6 +26,7 @@
#define X86_TRAP_XF 19 /* SIMD Floating-Point Exception */
#define X86_TRAP_VE 20 /* Virtualization Exception */
#define X86_TRAP_CP 21 /* Control Protection Exception */
+#define X86_TRAP_VC 29 /* VMM Communication Exception */
#define X86_TRAP_IRET 32 /* IRET Exception */
#endif
diff --git a/arch/x86/kernel/sev-es-shared.c b/arch/x86/kernel/sev-es-shared.c
new file mode 100644
index 000000000000..0bea32341afa
--- /dev/null
+++ b/arch/x86/kernel/sev-es-shared.c
@@ -0,0 +1,66 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * AMD Encrypted Register State Support
+ *
+ * Author: Joerg Roedel <jroedel@suse.de>
+ *
+ * This file is not compiled stand-alone. It contains code shared
+ * between the pre-decompression boot code and the running Linux kernel
+ * and is included directly into both code-bases.
+ */
+
+/*
+ * Boot VC Handler - This is the first VC handler during boot, there is no GHCB
+ * page yet, so it only supports the MSR based communication with the
+ * hypervisor and only the CPUID exit-code.
+ */
+void __init do_vc_no_ghcb(struct pt_regs *regs, unsigned long exit_code)
+{
+ unsigned int fn = lower_bits(regs->ax, 32);
+ unsigned long val;
+
+ /* Only CPUID is supported via MSR protocol */
+ if (exit_code != SVM_EXIT_CPUID)
+ goto fail;
+
+ sev_es_wr_ghcb_msr(GHCB_CPUID_REQ(fn, GHCB_CPUID_REQ_EAX));
+ VMGEXIT();
+ val = sev_es_rd_ghcb_msr();
+ if (GHCB_SEV_GHCB_RESP_CODE(val) != GHCB_SEV_CPUID_RESP)
+ goto fail;
+ regs->ax = val >> 32;
+
+ sev_es_wr_ghcb_msr(GHCB_CPUID_REQ(fn, GHCB_CPUID_REQ_EBX));
+ VMGEXIT();
+ val = sev_es_rd_ghcb_msr();
+ if (GHCB_SEV_GHCB_RESP_CODE(val) != GHCB_SEV_CPUID_RESP)
+ goto fail;
+ regs->bx = val >> 32;
+
+ sev_es_wr_ghcb_msr(GHCB_CPUID_REQ(fn, GHCB_CPUID_REQ_ECX));
+ VMGEXIT();
+ val = sev_es_rd_ghcb_msr();
+ if (GHCB_SEV_GHCB_RESP_CODE(val) != GHCB_SEV_CPUID_RESP)
+ goto fail;
+ regs->cx = val >> 32;
+
+ sev_es_wr_ghcb_msr(GHCB_CPUID_REQ(fn, GHCB_CPUID_REQ_EDX));
+ VMGEXIT();
+ val = sev_es_rd_ghcb_msr();
+ if (GHCB_SEV_GHCB_RESP_CODE(val) != GHCB_SEV_CPUID_RESP)
+ goto fail;
+ regs->dx = val >> 32;
+
+ /* Skip over the CPUID two-byte opcode */
+ regs->ip += 2;
+
+ return;
+
+fail:
+ sev_es_wr_ghcb_msr(GHCB_SEV_TERMINATE);
+ VMGEXIT();
+
+ /* Shouldn't get here - if we do halt the machine */
+ while (true)
+ asm volatile("hlt\n");
+}
--
2.28.0
^ permalink raw reply related [flat|nested] 264+ messages in thread
* [PATCH v7 19/72] x86/boot/compressed/64: Add stage1 #VC handler
@ 2020-09-07 13:15 ` Joerg Roedel
0 siblings, 0 replies; 264+ messages in thread
From: Joerg Roedel @ 2020-09-07 13:15 UTC (permalink / raw)
To: x86
Cc: Juergen Gross, Tom Lendacky, Joerg Roedel, Mike Stunes,
Kees Cook, kvm, Peter Zijlstra, Cfir Cohen, Joerg Roedel,
Dave Hansen, linux-kernel, Sean Christopherson, virtualization,
Martin Radev, Masami Hiramatsu, Andy Lutomirski, hpa,
Erdem Aktas, David Rientjes, Dan Williams, Jiri Slaby
From: Joerg Roedel <jroedel@suse.de>
Add the first handler for #VC exceptions. At stage 1 there is no GHCB
yet becaue the kernel might still be running on the EFI page table.
The stage 1 handler is limited to the MSR based protocol to talk to
the hypervisor and can only support CPUID exit-codes, but that is
enough to get to stage 2.
Signed-off-by: Joerg Roedel <jroedel@suse.de>
---
arch/x86/boot/compressed/Makefile | 1 +
arch/x86/boot/compressed/idt_64.c | 4 ++
arch/x86/boot/compressed/idt_handlers_64.S | 4 ++
arch/x86/boot/compressed/misc.h | 1 +
arch/x86/boot/compressed/sev-es.c | 45 +++++++++++++++
arch/x86/include/asm/msr-index.h | 1 +
arch/x86/include/asm/sev-es.h | 37 ++++++++++++
arch/x86/include/asm/trapnr.h | 1 +
arch/x86/kernel/sev-es-shared.c | 66 ++++++++++++++++++++++
9 files changed, 160 insertions(+)
create mode 100644 arch/x86/boot/compressed/sev-es.c
create mode 100644 arch/x86/include/asm/sev-es.h
create mode 100644 arch/x86/kernel/sev-es-shared.c
diff --git a/arch/x86/boot/compressed/Makefile b/arch/x86/boot/compressed/Makefile
index c12c9223a1fb..7302f184bece 100644
--- a/arch/x86/boot/compressed/Makefile
+++ b/arch/x86/boot/compressed/Makefile
@@ -83,6 +83,7 @@ ifdef CONFIG_X86_64
vmlinux-objs-y += $(obj)/idt_64.o $(obj)/idt_handlers_64.o
vmlinux-objs-y += $(obj)/mem_encrypt.o
vmlinux-objs-y += $(obj)/pgtable_64.o
+ vmlinux-objs-$(CONFIG_AMD_MEM_ENCRYPT) += $(obj)/sev-es.o
endif
vmlinux-objs-$(CONFIG_ACPI) += $(obj)/acpi.o
diff --git a/arch/x86/boot/compressed/idt_64.c b/arch/x86/boot/compressed/idt_64.c
index 5f083092a86d..f3ca7324be44 100644
--- a/arch/x86/boot/compressed/idt_64.c
+++ b/arch/x86/boot/compressed/idt_64.c
@@ -32,6 +32,10 @@ void load_stage1_idt(void)
{
boot_idt_desc.address = (unsigned long)boot_idt;
+
+ if (IS_ENABLED(CONFIG_AMD_MEM_ENCRYPT))
+ set_idt_entry(X86_TRAP_VC, boot_stage1_vc);
+
load_boot_idt(&boot_idt_desc);
}
diff --git a/arch/x86/boot/compressed/idt_handlers_64.S b/arch/x86/boot/compressed/idt_handlers_64.S
index b20e57504a94..92eb4df478a1 100644
--- a/arch/x86/boot/compressed/idt_handlers_64.S
+++ b/arch/x86/boot/compressed/idt_handlers_64.S
@@ -70,3 +70,7 @@ SYM_FUNC_END(\name)
.code64
EXCEPTION_HANDLER boot_page_fault do_boot_page_fault error_code=1
+
+#ifdef CONFIG_AMD_MEM_ENCRYPT
+EXCEPTION_HANDLER boot_stage1_vc do_vc_no_ghcb error_code=1
+#endif
diff --git a/arch/x86/boot/compressed/misc.h b/arch/x86/boot/compressed/misc.h
index 9840c82a39f1..eaa8b45ebccb 100644
--- a/arch/x86/boot/compressed/misc.h
+++ b/arch/x86/boot/compressed/misc.h
@@ -141,5 +141,6 @@ extern struct desc_ptr boot_idt_desc;
/* IDT Entry Points */
void boot_page_fault(void);
+void boot_stage1_vc(void);
#endif /* BOOT_COMPRESSED_MISC_H */
diff --git a/arch/x86/boot/compressed/sev-es.c b/arch/x86/boot/compressed/sev-es.c
new file mode 100644
index 000000000000..bb91cbb5920e
--- /dev/null
+++ b/arch/x86/boot/compressed/sev-es.c
@@ -0,0 +1,45 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * AMD Encrypted Register State Support
+ *
+ * Author: Joerg Roedel <jroedel@suse.de>
+ */
+
+/*
+ * misc.h needs to be first because it knows how to include the other kernel
+ * headers in the pre-decompression code in a way that does not break
+ * compilation.
+ */
+#include "misc.h"
+
+#include <asm/sev-es.h>
+#include <asm/msr-index.h>
+#include <asm/ptrace.h>
+#include <asm/svm.h>
+
+static inline u64 sev_es_rd_ghcb_msr(void)
+{
+ unsigned long low, high;
+
+ asm volatile("rdmsr\n" : "=a" (low), "=d" (high) :
+ "c" (MSR_AMD64_SEV_ES_GHCB));
+
+ return ((high << 32) | low);
+}
+
+static inline void sev_es_wr_ghcb_msr(u64 val)
+{
+ u32 low, high;
+
+ low = val & 0xffffffffUL;
+ high = val >> 32;
+
+ asm volatile("wrmsr\n" : : "c" (MSR_AMD64_SEV_ES_GHCB),
+ "a"(low), "d" (high) : "memory");
+}
+
+#undef __init
+#define __init
+
+/* Include code for early handlers */
+#include "../../kernel/sev-es-shared.c"
diff --git a/arch/x86/include/asm/msr-index.h b/arch/x86/include/asm/msr-index.h
index dc131b84ac3a..cd6d651ff730 100644
--- a/arch/x86/include/asm/msr-index.h
+++ b/arch/x86/include/asm/msr-index.h
@@ -466,6 +466,7 @@
#define MSR_AMD64_IBSBRTARGET 0xc001103b
#define MSR_AMD64_IBSOPDATA4 0xc001103d
#define MSR_AMD64_IBS_REG_COUNT_MAX 8 /* includes MSR_AMD64_IBSBRTARGET */
+#define MSR_AMD64_SEV_ES_GHCB 0xc0010130
#define MSR_AMD64_SEV 0xc0010131
#define MSR_AMD64_SEV_ENABLED_BIT 0
#define MSR_AMD64_SEV_ENABLED BIT_ULL(MSR_AMD64_SEV_ENABLED_BIT)
diff --git a/arch/x86/include/asm/sev-es.h b/arch/x86/include/asm/sev-es.h
new file mode 100644
index 000000000000..48a44038b5d1
--- /dev/null
+++ b/arch/x86/include/asm/sev-es.h
@@ -0,0 +1,37 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/*
+ * AMD Encrypted Register State Support
+ *
+ * Author: Joerg Roedel <jroedel@suse.de>
+ */
+
+#ifndef __ASM_ENCRYPTED_STATE_H
+#define __ASM_ENCRYPTED_STATE_H
+
+#include <linux/types.h>
+
+#define GHCB_SEV_CPUID_REQ 0x004UL
+#define GHCB_CPUID_REQ_EAX 0
+#define GHCB_CPUID_REQ_EBX 1
+#define GHCB_CPUID_REQ_ECX 2
+#define GHCB_CPUID_REQ_EDX 3
+#define GHCB_CPUID_REQ(fn, reg) (GHCB_SEV_CPUID_REQ | \
+ (((unsigned long)reg & 3) << 30) | \
+ (((unsigned long)fn) << 32))
+
+#define GHCB_SEV_CPUID_RESP 0x005UL
+#define GHCB_SEV_TERMINATE 0x100UL
+
+#define GHCB_SEV_GHCB_RESP_CODE(v) ((v) & 0xfff)
+#define VMGEXIT() { asm volatile("rep; vmmcall\n\r"); }
+
+void do_vc_no_ghcb(struct pt_regs *regs, unsigned long exit_code);
+
+static inline u64 lower_bits(u64 val, unsigned int bits)
+{
+ u64 mask = (1ULL << bits) - 1;
+
+ return (val & mask);
+}
+
+#endif
diff --git a/arch/x86/include/asm/trapnr.h b/arch/x86/include/asm/trapnr.h
index 082f45631fa9..f5d2325aa0b7 100644
--- a/arch/x86/include/asm/trapnr.h
+++ b/arch/x86/include/asm/trapnr.h
@@ -26,6 +26,7 @@
#define X86_TRAP_XF 19 /* SIMD Floating-Point Exception */
#define X86_TRAP_VE 20 /* Virtualization Exception */
#define X86_TRAP_CP 21 /* Control Protection Exception */
+#define X86_TRAP_VC 29 /* VMM Communication Exception */
#define X86_TRAP_IRET 32 /* IRET Exception */
#endif
diff --git a/arch/x86/kernel/sev-es-shared.c b/arch/x86/kernel/sev-es-shared.c
new file mode 100644
index 000000000000..0bea32341afa
--- /dev/null
+++ b/arch/x86/kernel/sev-es-shared.c
@@ -0,0 +1,66 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * AMD Encrypted Register State Support
+ *
+ * Author: Joerg Roedel <jroedel@suse.de>
+ *
+ * This file is not compiled stand-alone. It contains code shared
+ * between the pre-decompression boot code and the running Linux kernel
+ * and is included directly into both code-bases.
+ */
+
+/*
+ * Boot VC Handler - This is the first VC handler during boot, there is no GHCB
+ * page yet, so it only supports the MSR based communication with the
+ * hypervisor and only the CPUID exit-code.
+ */
+void __init do_vc_no_ghcb(struct pt_regs *regs, unsigned long exit_code)
+{
+ unsigned int fn = lower_bits(regs->ax, 32);
+ unsigned long val;
+
+ /* Only CPUID is supported via MSR protocol */
+ if (exit_code != SVM_EXIT_CPUID)
+ goto fail;
+
+ sev_es_wr_ghcb_msr(GHCB_CPUID_REQ(fn, GHCB_CPUID_REQ_EAX));
+ VMGEXIT();
+ val = sev_es_rd_ghcb_msr();
+ if (GHCB_SEV_GHCB_RESP_CODE(val) != GHCB_SEV_CPUID_RESP)
+ goto fail;
+ regs->ax = val >> 32;
+
+ sev_es_wr_ghcb_msr(GHCB_CPUID_REQ(fn, GHCB_CPUID_REQ_EBX));
+ VMGEXIT();
+ val = sev_es_rd_ghcb_msr();
+ if (GHCB_SEV_GHCB_RESP_CODE(val) != GHCB_SEV_CPUID_RESP)
+ goto fail;
+ regs->bx = val >> 32;
+
+ sev_es_wr_ghcb_msr(GHCB_CPUID_REQ(fn, GHCB_CPUID_REQ_ECX));
+ VMGEXIT();
+ val = sev_es_rd_ghcb_msr();
+ if (GHCB_SEV_GHCB_RESP_CODE(val) != GHCB_SEV_CPUID_RESP)
+ goto fail;
+ regs->cx = val >> 32;
+
+ sev_es_wr_ghcb_msr(GHCB_CPUID_REQ(fn, GHCB_CPUID_REQ_EDX));
+ VMGEXIT();
+ val = sev_es_rd_ghcb_msr();
+ if (GHCB_SEV_GHCB_RESP_CODE(val) != GHCB_SEV_CPUID_RESP)
+ goto fail;
+ regs->dx = val >> 32;
+
+ /* Skip over the CPUID two-byte opcode */
+ regs->ip += 2;
+
+ return;
+
+fail:
+ sev_es_wr_ghcb_msr(GHCB_SEV_TERMINATE);
+ VMGEXIT();
+
+ /* Shouldn't get here - if we do halt the machine */
+ while (true)
+ asm volatile("hlt\n");
+}
--
2.28.0
_______________________________________________
Virtualization mailing list
Virtualization@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/virtualization
^ permalink raw reply related [flat|nested] 264+ messages in thread
* [PATCH v7 20/72] x86/boot/compressed/64: Call set_sev_encryption_mask() earlier
2020-09-07 13:15 ` Joerg Roedel
@ 2020-09-07 13:15 ` Joerg Roedel
-1 siblings, 0 replies; 264+ messages in thread
From: Joerg Roedel @ 2020-09-07 13:15 UTC (permalink / raw)
To: x86
Cc: Joerg Roedel, Joerg Roedel, hpa, Andy Lutomirski, Dave Hansen,
Peter Zijlstra, Jiri Slaby, Dan Williams, Tom Lendacky,
Juergen Gross, Kees Cook, David Rientjes, Cfir Cohen,
Erdem Aktas, Masami Hiramatsu, Mike Stunes, Sean Christopherson,
Martin Radev, linux-kernel, kvm, virtualization
From: Joerg Roedel <jroedel@suse.de>
Call set_sev_encryption_mask() while still on the stage 1 #VC-handler,
because the stage 2 handler needs the kernel's own page-tables to be
set up, to which calling set_sev_encryption_mask() is a prerequisite.
Signed-off-by: Joerg Roedel <jroedel@suse.de>
---
arch/x86/boot/compressed/head_64.S | 9 ++++++++-
arch/x86/boot/compressed/ident_map_64.c | 3 ---
2 files changed, 8 insertions(+), 4 deletions(-)
diff --git a/arch/x86/boot/compressed/head_64.S b/arch/x86/boot/compressed/head_64.S
index 37c8c2c8f8b6..1c80f1738fd9 100644
--- a/arch/x86/boot/compressed/head_64.S
+++ b/arch/x86/boot/compressed/head_64.S
@@ -533,9 +533,16 @@ SYM_FUNC_START_LOCAL_NOALIGN(.Lrelocated)
rep stosq
/*
- * Load stage2 IDT and switch to our own page-table
+ * If running as an SEV guest, the encryption mask is required in the
+ * page-table setup code below. When the guest also has SEV-ES enabled
+ * set_sev_encryption_mask() will cause #VC exceptions, but the stage2
+ * handler can't map its GHCB because the page-table is not set up yet.
+ * So set up the encryption mask here while still on the stage1 #VC
+ * handler. Then load stage2 IDT and switch to the kernel's own
+ * page-table.
*/
pushq %rsi
+ call set_sev_encryption_mask
call load_stage2_idt
call initialize_identity_maps
popq %rsi
diff --git a/arch/x86/boot/compressed/ident_map_64.c b/arch/x86/boot/compressed/ident_map_64.c
index 62e42c11a336..b4f2a5f503cd 100644
--- a/arch/x86/boot/compressed/ident_map_64.c
+++ b/arch/x86/boot/compressed/ident_map_64.c
@@ -105,9 +105,6 @@ static void add_identity_map(unsigned long start, unsigned long end)
/* Locates and clears a region for a new top level page table. */
void initialize_identity_maps(void)
{
- /* If running as an SEV guest, the encryption mask is required. */
- set_sev_encryption_mask();
-
/* Exclude the encryption mask from __PHYSICAL_MASK */
physical_mask &= ~sme_me_mask;
--
2.28.0
^ permalink raw reply related [flat|nested] 264+ messages in thread
* [PATCH v7 20/72] x86/boot/compressed/64: Call set_sev_encryption_mask() earlier
@ 2020-09-07 13:15 ` Joerg Roedel
0 siblings, 0 replies; 264+ messages in thread
From: Joerg Roedel @ 2020-09-07 13:15 UTC (permalink / raw)
To: x86
Cc: Juergen Gross, Tom Lendacky, Joerg Roedel, Mike Stunes,
Kees Cook, kvm, Peter Zijlstra, Cfir Cohen, Joerg Roedel,
Dave Hansen, linux-kernel, Sean Christopherson, virtualization,
Martin Radev, Masami Hiramatsu, Andy Lutomirski, hpa,
Erdem Aktas, David Rientjes, Dan Williams, Jiri Slaby
From: Joerg Roedel <jroedel@suse.de>
Call set_sev_encryption_mask() while still on the stage 1 #VC-handler,
because the stage 2 handler needs the kernel's own page-tables to be
set up, to which calling set_sev_encryption_mask() is a prerequisite.
Signed-off-by: Joerg Roedel <jroedel@suse.de>
---
arch/x86/boot/compressed/head_64.S | 9 ++++++++-
arch/x86/boot/compressed/ident_map_64.c | 3 ---
2 files changed, 8 insertions(+), 4 deletions(-)
diff --git a/arch/x86/boot/compressed/head_64.S b/arch/x86/boot/compressed/head_64.S
index 37c8c2c8f8b6..1c80f1738fd9 100644
--- a/arch/x86/boot/compressed/head_64.S
+++ b/arch/x86/boot/compressed/head_64.S
@@ -533,9 +533,16 @@ SYM_FUNC_START_LOCAL_NOALIGN(.Lrelocated)
rep stosq
/*
- * Load stage2 IDT and switch to our own page-table
+ * If running as an SEV guest, the encryption mask is required in the
+ * page-table setup code below. When the guest also has SEV-ES enabled
+ * set_sev_encryption_mask() will cause #VC exceptions, but the stage2
+ * handler can't map its GHCB because the page-table is not set up yet.
+ * So set up the encryption mask here while still on the stage1 #VC
+ * handler. Then load stage2 IDT and switch to the kernel's own
+ * page-table.
*/
pushq %rsi
+ call set_sev_encryption_mask
call load_stage2_idt
call initialize_identity_maps
popq %rsi
diff --git a/arch/x86/boot/compressed/ident_map_64.c b/arch/x86/boot/compressed/ident_map_64.c
index 62e42c11a336..b4f2a5f503cd 100644
--- a/arch/x86/boot/compressed/ident_map_64.c
+++ b/arch/x86/boot/compressed/ident_map_64.c
@@ -105,9 +105,6 @@ static void add_identity_map(unsigned long start, unsigned long end)
/* Locates and clears a region for a new top level page table. */
void initialize_identity_maps(void)
{
- /* If running as an SEV guest, the encryption mask is required. */
- set_sev_encryption_mask();
-
/* Exclude the encryption mask from __PHYSICAL_MASK */
physical_mask &= ~sme_me_mask;
--
2.28.0
_______________________________________________
Virtualization mailing list
Virtualization@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/virtualization
^ permalink raw reply related [flat|nested] 264+ messages in thread
* [PATCH v7 21/72] x86/boot/compressed/64: Check return value of kernel_ident_mapping_init()
2020-09-07 13:15 ` Joerg Roedel
@ 2020-09-07 13:15 ` Joerg Roedel
-1 siblings, 0 replies; 264+ messages in thread
From: Joerg Roedel @ 2020-09-07 13:15 UTC (permalink / raw)
To: x86
Cc: Joerg Roedel, Joerg Roedel, hpa, Andy Lutomirski, Dave Hansen,
Peter Zijlstra, Jiri Slaby, Dan Williams, Tom Lendacky,
Juergen Gross, Kees Cook, David Rientjes, Cfir Cohen,
Erdem Aktas, Masami Hiramatsu, Mike Stunes, Sean Christopherson,
Martin Radev, linux-kernel, kvm, virtualization
From: Joerg Roedel <jroedel@suse.de>
The function can fail to create an identity mapping, check for that
and bail out if it happens.
Signed-off-by: Joerg Roedel <jroedel@suse.de>
---
arch/x86/boot/compressed/ident_map_64.c | 7 +++++--
1 file changed, 5 insertions(+), 2 deletions(-)
diff --git a/arch/x86/boot/compressed/ident_map_64.c b/arch/x86/boot/compressed/ident_map_64.c
index b4f2a5f503cd..aa91bebc0fe9 100644
--- a/arch/x86/boot/compressed/ident_map_64.c
+++ b/arch/x86/boot/compressed/ident_map_64.c
@@ -91,6 +91,8 @@ static struct x86_mapping_info mapping_info;
*/
static void add_identity_map(unsigned long start, unsigned long end)
{
+ int ret;
+
/* Align boundary to 2M. */
start = round_down(start, PMD_SIZE);
end = round_up(end, PMD_SIZE);
@@ -98,8 +100,9 @@ static void add_identity_map(unsigned long start, unsigned long end)
return;
/* Build the mapping. */
- kernel_ident_mapping_init(&mapping_info, (pgd_t *)top_level_pgt,
- start, end);
+ ret = kernel_ident_mapping_init(&mapping_info, (pgd_t *)top_level_pgt, start, end);
+ if (ret)
+ error("Error: kernel_ident_mapping_init() failed\n");
}
/* Locates and clears a region for a new top level page table. */
--
2.28.0
^ permalink raw reply related [flat|nested] 264+ messages in thread
* [PATCH v7 21/72] x86/boot/compressed/64: Check return value of kernel_ident_mapping_init()
@ 2020-09-07 13:15 ` Joerg Roedel
0 siblings, 0 replies; 264+ messages in thread
From: Joerg Roedel @ 2020-09-07 13:15 UTC (permalink / raw)
To: x86
Cc: Juergen Gross, Tom Lendacky, Joerg Roedel, Mike Stunes,
Kees Cook, kvm, Peter Zijlstra, Cfir Cohen, Joerg Roedel,
Dave Hansen, linux-kernel, Sean Christopherson, virtualization,
Martin Radev, Masami Hiramatsu, Andy Lutomirski, hpa,
Erdem Aktas, David Rientjes, Dan Williams, Jiri Slaby
From: Joerg Roedel <jroedel@suse.de>
The function can fail to create an identity mapping, check for that
and bail out if it happens.
Signed-off-by: Joerg Roedel <jroedel@suse.de>
---
arch/x86/boot/compressed/ident_map_64.c | 7 +++++--
1 file changed, 5 insertions(+), 2 deletions(-)
diff --git a/arch/x86/boot/compressed/ident_map_64.c b/arch/x86/boot/compressed/ident_map_64.c
index b4f2a5f503cd..aa91bebc0fe9 100644
--- a/arch/x86/boot/compressed/ident_map_64.c
+++ b/arch/x86/boot/compressed/ident_map_64.c
@@ -91,6 +91,8 @@ static struct x86_mapping_info mapping_info;
*/
static void add_identity_map(unsigned long start, unsigned long end)
{
+ int ret;
+
/* Align boundary to 2M. */
start = round_down(start, PMD_SIZE);
end = round_up(end, PMD_SIZE);
@@ -98,8 +100,9 @@ static void add_identity_map(unsigned long start, unsigned long end)
return;
/* Build the mapping. */
- kernel_ident_mapping_init(&mapping_info, (pgd_t *)top_level_pgt,
- start, end);
+ ret = kernel_ident_mapping_init(&mapping_info, (pgd_t *)top_level_pgt, start, end);
+ if (ret)
+ error("Error: kernel_ident_mapping_init() failed\n");
}
/* Locates and clears a region for a new top level page table. */
--
2.28.0
_______________________________________________
Virtualization mailing list
Virtualization@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/virtualization
^ permalink raw reply related [flat|nested] 264+ messages in thread
* [PATCH v7 22/72] x86/boot/compressed/64: Add set_page_en/decrypted() helpers
2020-09-07 13:15 ` Joerg Roedel
@ 2020-09-07 13:15 ` Joerg Roedel
-1 siblings, 0 replies; 264+ messages in thread
From: Joerg Roedel @ 2020-09-07 13:15 UTC (permalink / raw)
To: x86
Cc: Joerg Roedel, Joerg Roedel, hpa, Andy Lutomirski, Dave Hansen,
Peter Zijlstra, Jiri Slaby, Dan Williams, Tom Lendacky,
Juergen Gross, Kees Cook, David Rientjes, Cfir Cohen,
Erdem Aktas, Masami Hiramatsu, Mike Stunes, Sean Christopherson,
Martin Radev, linux-kernel, kvm, virtualization
From: Joerg Roedel <jroedel@suse.de>
The functions are needed to map the GHCB for SEV-ES guests. The GHCB is
used for communication with the hypervisor, so its content must not be
encrypted. After the GHCB is not needed anymore it must be mapped
encrypted again so that the running kernel image can safely re-use the
memory.
Signed-off-by: Joerg Roedel <jroedel@suse.de>
---
arch/x86/boot/compressed/ident_map_64.c | 133 ++++++++++++++++++++++++
arch/x86/boot/compressed/misc.h | 2 +
2 files changed, 135 insertions(+)
diff --git a/arch/x86/boot/compressed/ident_map_64.c b/arch/x86/boot/compressed/ident_map_64.c
index aa91bebc0fe9..05742f641a06 100644
--- a/arch/x86/boot/compressed/ident_map_64.c
+++ b/arch/x86/boot/compressed/ident_map_64.c
@@ -24,6 +24,7 @@
/* These actually do the work of building the kernel identity maps. */
#include <linux/pgtable.h>
+#include <asm/cmpxchg.h>
#include <asm/trap_pf.h>
#include <asm/trapnr.h>
#include <asm/init.h>
@@ -165,6 +166,138 @@ void finalize_identity_maps(void)
write_cr3(top_level_pgt);
}
+static pte_t *split_large_pmd(struct x86_mapping_info *info,
+ pmd_t *pmdp, unsigned long __address)
+{
+ unsigned long page_flags;
+ unsigned long address;
+ pte_t *pte;
+ pmd_t pmd;
+ int i;
+
+ pte = (pte_t *)info->alloc_pgt_page(info->context);
+ if (!pte)
+ return NULL;
+
+ address = __address & PMD_MASK;
+ /* No large page - clear PSE flag */
+ page_flags = info->page_flag & ~_PAGE_PSE;
+
+ /* Populate the PTEs */
+ for (i = 0; i < PTRS_PER_PMD; i++) {
+ set_pte(&pte[i], __pte(address | page_flags));
+ address += PAGE_SIZE;
+ }
+
+ /*
+ * Ideally we need to clear the large PMD first and do a TLB
+ * flush before we write the new PMD. But the 2M range of the
+ * PMD might contain the code we execute and/or the stack
+ * we are on, so we can't do that. But that should be safe here
+ * because we are going from large to small mappings and we are
+ * also the only user of the page-table, so there is no chance
+ * of a TLB multihit.
+ */
+ pmd = __pmd((unsigned long)pte | info->kernpg_flag);
+ set_pmd(pmdp, pmd);
+ /* Flush TLB to establish the new PMD */
+ write_cr3(top_level_pgt);
+
+ return pte + pte_index(__address);
+}
+
+static void clflush_page(unsigned long address)
+{
+ unsigned int flush_size;
+ char *cl, *start, *end;
+
+ /*
+ * Hardcode cl-size to 64 - CPUID can't be used here because that might
+ * cause another #VC exception and the GHCB is not ready to use yet.
+ */
+ flush_size = 64;
+ start = (char *)(address & PAGE_MASK);
+ end = start + PAGE_SIZE;
+
+ /*
+ * First make sure there are no pending writes on the cache-lines to
+ * flush.
+ */
+ asm volatile("mfence" : : : "memory");
+
+ for (cl = start; cl != end; cl += flush_size)
+ clflush(cl);
+}
+
+static int set_clr_page_flags(struct x86_mapping_info *info,
+ unsigned long address,
+ pteval_t set, pteval_t clr)
+{
+ pgd_t *pgdp = (pgd_t *)top_level_pgt;
+ p4d_t *p4dp;
+ pud_t *pudp;
+ pmd_t *pmdp;
+ pte_t *ptep, pte;
+
+ /*
+ * First make sure there is a PMD mapping for 'address'.
+ * It should already exist, but keep things generic.
+ *
+ * To map the page just read from it and fault it in if there is no
+ * mapping yet. add_identity_map() can't be called here because that
+ * would unconditionally map the address on PMD level, destroying any
+ * PTE-level mappings that might already exist. Use assembly here so
+ * the access won't be optimized away.
+ */
+ asm volatile("mov %[address], %%r9"
+ :: [address] "g" (*(unsigned long *)address)
+ : "r9", "memory");
+
+ /*
+ * The page is mapped at least with PMD size - so skip checks and walk
+ * directly to the PMD.
+ */
+ p4dp = p4d_offset(pgdp, address);
+ pudp = pud_offset(p4dp, address);
+ pmdp = pmd_offset(pudp, address);
+
+ if (pmd_large(*pmdp))
+ ptep = split_large_pmd(info, pmdp, address);
+ else
+ ptep = pte_offset_kernel(pmdp, address);
+
+ if (!ptep)
+ return -ENOMEM;
+
+ /*
+ * Changing encryption attributes of a page requires to flush it from
+ * the caches.
+ */
+ if ((set | clr) & _PAGE_ENC)
+ clflush_page(address);
+
+ /* Update PTE */
+ pte = *ptep;
+ pte = pte_set_flags(pte, set);
+ pte = pte_clear_flags(pte, clr);
+ set_pte(ptep, pte);
+
+ /* Flush TLB after changing encryption attribute */
+ write_cr3(top_level_pgt);
+
+ return 0;
+}
+
+int set_page_decrypted(unsigned long address)
+{
+ return set_clr_page_flags(&mapping_info, address, 0, _PAGE_ENC);
+}
+
+int set_page_encrypted(unsigned long address)
+{
+ return set_clr_page_flags(&mapping_info, address, _PAGE_ENC, 0);
+}
+
static void do_pf_error(const char *msg, unsigned long error_code,
unsigned long address, unsigned long ip)
{
diff --git a/arch/x86/boot/compressed/misc.h b/arch/x86/boot/compressed/misc.h
index eaa8b45ebccb..01c0fb3417ca 100644
--- a/arch/x86/boot/compressed/misc.h
+++ b/arch/x86/boot/compressed/misc.h
@@ -98,6 +98,8 @@ static inline void choose_random_location(unsigned long input,
#endif
#ifdef CONFIG_X86_64
+extern int set_page_decrypted(unsigned long address);
+extern int set_page_encrypted(unsigned long address);
extern unsigned char _pgtable[];
#endif
--
2.28.0
^ permalink raw reply related [flat|nested] 264+ messages in thread
* [PATCH v7 22/72] x86/boot/compressed/64: Add set_page_en/decrypted() helpers
@ 2020-09-07 13:15 ` Joerg Roedel
0 siblings, 0 replies; 264+ messages in thread
From: Joerg Roedel @ 2020-09-07 13:15 UTC (permalink / raw)
To: x86
Cc: Juergen Gross, Tom Lendacky, Joerg Roedel, Mike Stunes,
Kees Cook, kvm, Peter Zijlstra, Cfir Cohen, Joerg Roedel,
Dave Hansen, linux-kernel, Sean Christopherson, virtualization,
Martin Radev, Masami Hiramatsu, Andy Lutomirski, hpa,
Erdem Aktas, David Rientjes, Dan Williams, Jiri Slaby
From: Joerg Roedel <jroedel@suse.de>
The functions are needed to map the GHCB for SEV-ES guests. The GHCB is
used for communication with the hypervisor, so its content must not be
encrypted. After the GHCB is not needed anymore it must be mapped
encrypted again so that the running kernel image can safely re-use the
memory.
Signed-off-by: Joerg Roedel <jroedel@suse.de>
---
arch/x86/boot/compressed/ident_map_64.c | 133 ++++++++++++++++++++++++
arch/x86/boot/compressed/misc.h | 2 +
2 files changed, 135 insertions(+)
diff --git a/arch/x86/boot/compressed/ident_map_64.c b/arch/x86/boot/compressed/ident_map_64.c
index aa91bebc0fe9..05742f641a06 100644
--- a/arch/x86/boot/compressed/ident_map_64.c
+++ b/arch/x86/boot/compressed/ident_map_64.c
@@ -24,6 +24,7 @@
/* These actually do the work of building the kernel identity maps. */
#include <linux/pgtable.h>
+#include <asm/cmpxchg.h>
#include <asm/trap_pf.h>
#include <asm/trapnr.h>
#include <asm/init.h>
@@ -165,6 +166,138 @@ void finalize_identity_maps(void)
write_cr3(top_level_pgt);
}
+static pte_t *split_large_pmd(struct x86_mapping_info *info,
+ pmd_t *pmdp, unsigned long __address)
+{
+ unsigned long page_flags;
+ unsigned long address;
+ pte_t *pte;
+ pmd_t pmd;
+ int i;
+
+ pte = (pte_t *)info->alloc_pgt_page(info->context);
+ if (!pte)
+ return NULL;
+
+ address = __address & PMD_MASK;
+ /* No large page - clear PSE flag */
+ page_flags = info->page_flag & ~_PAGE_PSE;
+
+ /* Populate the PTEs */
+ for (i = 0; i < PTRS_PER_PMD; i++) {
+ set_pte(&pte[i], __pte(address | page_flags));
+ address += PAGE_SIZE;
+ }
+
+ /*
+ * Ideally we need to clear the large PMD first and do a TLB
+ * flush before we write the new PMD. But the 2M range of the
+ * PMD might contain the code we execute and/or the stack
+ * we are on, so we can't do that. But that should be safe here
+ * because we are going from large to small mappings and we are
+ * also the only user of the page-table, so there is no chance
+ * of a TLB multihit.
+ */
+ pmd = __pmd((unsigned long)pte | info->kernpg_flag);
+ set_pmd(pmdp, pmd);
+ /* Flush TLB to establish the new PMD */
+ write_cr3(top_level_pgt);
+
+ return pte + pte_index(__address);
+}
+
+static void clflush_page(unsigned long address)
+{
+ unsigned int flush_size;
+ char *cl, *start, *end;
+
+ /*
+ * Hardcode cl-size to 64 - CPUID can't be used here because that might
+ * cause another #VC exception and the GHCB is not ready to use yet.
+ */
+ flush_size = 64;
+ start = (char *)(address & PAGE_MASK);
+ end = start + PAGE_SIZE;
+
+ /*
+ * First make sure there are no pending writes on the cache-lines to
+ * flush.
+ */
+ asm volatile("mfence" : : : "memory");
+
+ for (cl = start; cl != end; cl += flush_size)
+ clflush(cl);
+}
+
+static int set_clr_page_flags(struct x86_mapping_info *info,
+ unsigned long address,
+ pteval_t set, pteval_t clr)
+{
+ pgd_t *pgdp = (pgd_t *)top_level_pgt;
+ p4d_t *p4dp;
+ pud_t *pudp;
+ pmd_t *pmdp;
+ pte_t *ptep, pte;
+
+ /*
+ * First make sure there is a PMD mapping for 'address'.
+ * It should already exist, but keep things generic.
+ *
+ * To map the page just read from it and fault it in if there is no
+ * mapping yet. add_identity_map() can't be called here because that
+ * would unconditionally map the address on PMD level, destroying any
+ * PTE-level mappings that might already exist. Use assembly here so
+ * the access won't be optimized away.
+ */
+ asm volatile("mov %[address], %%r9"
+ :: [address] "g" (*(unsigned long *)address)
+ : "r9", "memory");
+
+ /*
+ * The page is mapped at least with PMD size - so skip checks and walk
+ * directly to the PMD.
+ */
+ p4dp = p4d_offset(pgdp, address);
+ pudp = pud_offset(p4dp, address);
+ pmdp = pmd_offset(pudp, address);
+
+ if (pmd_large(*pmdp))
+ ptep = split_large_pmd(info, pmdp, address);
+ else
+ ptep = pte_offset_kernel(pmdp, address);
+
+ if (!ptep)
+ return -ENOMEM;
+
+ /*
+ * Changing encryption attributes of a page requires to flush it from
+ * the caches.
+ */
+ if ((set | clr) & _PAGE_ENC)
+ clflush_page(address);
+
+ /* Update PTE */
+ pte = *ptep;
+ pte = pte_set_flags(pte, set);
+ pte = pte_clear_flags(pte, clr);
+ set_pte(ptep, pte);
+
+ /* Flush TLB after changing encryption attribute */
+ write_cr3(top_level_pgt);
+
+ return 0;
+}
+
+int set_page_decrypted(unsigned long address)
+{
+ return set_clr_page_flags(&mapping_info, address, 0, _PAGE_ENC);
+}
+
+int set_page_encrypted(unsigned long address)
+{
+ return set_clr_page_flags(&mapping_info, address, _PAGE_ENC, 0);
+}
+
static void do_pf_error(const char *msg, unsigned long error_code,
unsigned long address, unsigned long ip)
{
diff --git a/arch/x86/boot/compressed/misc.h b/arch/x86/boot/compressed/misc.h
index eaa8b45ebccb..01c0fb3417ca 100644
--- a/arch/x86/boot/compressed/misc.h
+++ b/arch/x86/boot/compressed/misc.h
@@ -98,6 +98,8 @@ static inline void choose_random_location(unsigned long input,
#endif
#ifdef CONFIG_X86_64
+extern int set_page_decrypted(unsigned long address);
+extern int set_page_encrypted(unsigned long address);
extern unsigned char _pgtable[];
#endif
--
2.28.0
_______________________________________________
Virtualization mailing list
Virtualization@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/virtualization
^ permalink raw reply related [flat|nested] 264+ messages in thread
* [PATCH v7 23/72] x86/boot/compressed/64: Setup GHCB Based VC Exception handler
2020-09-07 13:15 ` Joerg Roedel
@ 2020-09-07 13:15 ` Joerg Roedel
-1 siblings, 0 replies; 264+ messages in thread
From: Joerg Roedel @ 2020-09-07 13:15 UTC (permalink / raw)
To: x86
Cc: Joerg Roedel, Joerg Roedel, hpa, Andy Lutomirski, Dave Hansen,
Peter Zijlstra, Jiri Slaby, Dan Williams, Tom Lendacky,
Juergen Gross, Kees Cook, David Rientjes, Cfir Cohen,
Erdem Aktas, Masami Hiramatsu, Mike Stunes, Sean Christopherson,
Martin Radev, linux-kernel, kvm, virtualization
From: Joerg Roedel <jroedel@suse.de>
Install an exception handler for #VC exception that uses a GHCB. Also
add the infrastructure for handling different exit-codes by decoding
the instruction that caused the exception and error handling.
Signed-off-by: Joerg Roedel <jroedel@suse.de>
---
arch/x86/Kconfig | 1 +
arch/x86/boot/compressed/Makefile | 5 +
arch/x86/boot/compressed/idt_64.c | 4 +
arch/x86/boot/compressed/idt_handlers_64.S | 3 +-
arch/x86/boot/compressed/misc.c | 7 +
arch/x86/boot/compressed/misc.h | 7 +
arch/x86/boot/compressed/sev-es.c | 111 +++++++++++++++
arch/x86/include/asm/sev-es.h | 39 ++++++
arch/x86/include/uapi/asm/svm.h | 1 +
arch/x86/kernel/sev-es-shared.c | 154 +++++++++++++++++++++
10 files changed, 331 insertions(+), 1 deletion(-)
diff --git a/arch/x86/Kconfig b/arch/x86/Kconfig
index 8a48d3eedb84..5a604353ec42 100644
--- a/arch/x86/Kconfig
+++ b/arch/x86/Kconfig
@@ -1523,6 +1523,7 @@ config AMD_MEM_ENCRYPT
select DYNAMIC_PHYSICAL_MASK
select ARCH_USE_MEMREMAP_PROT
select ARCH_HAS_FORCE_DMA_UNENCRYPTED
+ select INSTRUCTION_DECODER
help
Say yes to enable support for the encryption of system memory.
This requires an AMD processor that supports Secure Memory
diff --git a/arch/x86/boot/compressed/Makefile b/arch/x86/boot/compressed/Makefile
index 7302f184bece..3224f1d9a5cd 100644
--- a/arch/x86/boot/compressed/Makefile
+++ b/arch/x86/boot/compressed/Makefile
@@ -45,6 +45,11 @@ KBUILD_CFLAGS += -fno-asynchronous-unwind-tables
KBUILD_CFLAGS += -D__DISABLE_EXPORTS
KBUILD_CFLAGS += -include $(srctree)/include/linux/hidden.h
+# sev-es.c indirectly inludes inat-table.h which is generated during
+# compilation and stored in $(objtree). Add the directory to the includes so
+# that the compiler finds it even with out-of-tree builds (make O=/some/path).
+CFLAGS_sev-es.o += -I$(objtree)/arch/x86/lib/
+
KBUILD_AFLAGS := $(KBUILD_CFLAGS) -D__ASSEMBLY__
GCOV_PROFILE := n
UBSAN_SANITIZE :=n
diff --git a/arch/x86/boot/compressed/idt_64.c b/arch/x86/boot/compressed/idt_64.c
index f3ca7324be44..804a502ee0d2 100644
--- a/arch/x86/boot/compressed/idt_64.c
+++ b/arch/x86/boot/compressed/idt_64.c
@@ -46,5 +46,9 @@ void load_stage2_idt(void)
set_idt_entry(X86_TRAP_PF, boot_page_fault);
+#ifdef CONFIG_AMD_MEM_ENCRYPT
+ set_idt_entry(X86_TRAP_VC, boot_stage2_vc);
+#endif
+
load_boot_idt(&boot_idt_desc);
}
diff --git a/arch/x86/boot/compressed/idt_handlers_64.S b/arch/x86/boot/compressed/idt_handlers_64.S
index 92eb4df478a1..22890e199f5b 100644
--- a/arch/x86/boot/compressed/idt_handlers_64.S
+++ b/arch/x86/boot/compressed/idt_handlers_64.S
@@ -72,5 +72,6 @@ SYM_FUNC_END(\name)
EXCEPTION_HANDLER boot_page_fault do_boot_page_fault error_code=1
#ifdef CONFIG_AMD_MEM_ENCRYPT
-EXCEPTION_HANDLER boot_stage1_vc do_vc_no_ghcb error_code=1
+EXCEPTION_HANDLER boot_stage1_vc do_vc_no_ghcb error_code=1
+EXCEPTION_HANDLER boot_stage2_vc do_boot_stage2_vc error_code=1
#endif
diff --git a/arch/x86/boot/compressed/misc.c b/arch/x86/boot/compressed/misc.c
index e478e40fbe5a..267e7f93050e 100644
--- a/arch/x86/boot/compressed/misc.c
+++ b/arch/x86/boot/compressed/misc.c
@@ -442,6 +442,13 @@ asmlinkage __visible void *extract_kernel(void *rmode, memptr heap,
parse_elf(output);
handle_relocations(output, output_len, virt_addr);
debug_putstr("done.\nBooting the kernel.\n");
+
+ /*
+ * Flush GHCB from cache and map it encrypted again when running as
+ * SEV-ES guest.
+ */
+ sev_es_shutdown_ghcb();
+
return output;
}
diff --git a/arch/x86/boot/compressed/misc.h b/arch/x86/boot/compressed/misc.h
index 01c0fb3417ca..9995c70ca813 100644
--- a/arch/x86/boot/compressed/misc.h
+++ b/arch/x86/boot/compressed/misc.h
@@ -115,6 +115,12 @@ static inline void console_init(void)
void set_sev_encryption_mask(void);
+#ifdef CONFIG_AMD_MEM_ENCRYPT
+void sev_es_shutdown_ghcb(void);
+#else
+static inline void sev_es_shutdown_ghcb(void) { }
+#endif
+
/* acpi.c */
#ifdef CONFIG_ACPI
acpi_physical_address get_rsdp_addr(void);
@@ -144,5 +150,6 @@ extern struct desc_ptr boot_idt_desc;
/* IDT Entry Points */
void boot_page_fault(void);
void boot_stage1_vc(void);
+void boot_stage2_vc(void);
#endif /* BOOT_COMPRESSED_MISC_H */
diff --git a/arch/x86/boot/compressed/sev-es.c b/arch/x86/boot/compressed/sev-es.c
index bb91cbb5920e..633d47df702f 100644
--- a/arch/x86/boot/compressed/sev-es.c
+++ b/arch/x86/boot/compressed/sev-es.c
@@ -13,10 +13,17 @@
#include "misc.h"
#include <asm/sev-es.h>
+#include <asm/trapnr.h>
+#include <asm/trap_pf.h>
#include <asm/msr-index.h>
#include <asm/ptrace.h>
#include <asm/svm.h>
+#include "error.h"
+
+struct ghcb boot_ghcb_page __aligned(PAGE_SIZE);
+struct ghcb *boot_ghcb;
+
static inline u64 sev_es_rd_ghcb_msr(void)
{
unsigned long low, high;
@@ -38,8 +45,112 @@ static inline void sev_es_wr_ghcb_msr(u64 val)
"a"(low), "d" (high) : "memory");
}
+static enum es_result vc_decode_insn(struct es_em_ctxt *ctxt)
+{
+ char buffer[MAX_INSN_SIZE];
+ enum es_result ret;
+
+ memcpy(buffer, (unsigned char *)ctxt->regs->ip, MAX_INSN_SIZE);
+
+ insn_init(&ctxt->insn, buffer, MAX_INSN_SIZE, 1);
+ insn_get_length(&ctxt->insn);
+
+ ret = ctxt->insn.immediate.got ? ES_OK : ES_DECODE_FAILED;
+
+ return ret;
+}
+
+static enum es_result vc_write_mem(struct es_em_ctxt *ctxt,
+ void *dst, char *buf, size_t size)
+{
+ memcpy(dst, buf, size);
+
+ return ES_OK;
+}
+
+static enum es_result vc_read_mem(struct es_em_ctxt *ctxt,
+ void *src, char *buf, size_t size)
+{
+ memcpy(buf, src, size);
+
+ return ES_OK;
+}
+
#undef __init
+#undef __pa
#define __init
+#define __pa(x) ((unsigned long)(x))
+
+#define __BOOT_COMPRESSED
+
+/* Basic instruction decoding support needed */
+#include "../../lib/inat.c"
+#include "../../lib/insn.c"
/* Include code for early handlers */
#include "../../kernel/sev-es-shared.c"
+
+static bool early_setup_sev_es(void)
+{
+ if (!sev_es_negotiate_protocol())
+ sev_es_terminate(GHCB_SEV_ES_REASON_PROTOCOL_UNSUPPORTED);
+
+ if (set_page_decrypted((unsigned long)&boot_ghcb_page))
+ return false;
+
+ /* Page is now mapped decrypted, clear it */
+ memset(&boot_ghcb_page, 0, sizeof(boot_ghcb_page));
+
+ boot_ghcb = &boot_ghcb_page;
+
+ /* Initialize lookup tables for the instruction decoder */
+ inat_init_tables();
+
+ return true;
+}
+
+void sev_es_shutdown_ghcb(void)
+{
+ if (!boot_ghcb)
+ return;
+
+ /*
+ * GHCB Page must be flushed from the cache and mapped encrypted again.
+ * Otherwise the running kernel will see strange cache effects when
+ * trying to use that page.
+ */
+ if (set_page_encrypted((unsigned long)&boot_ghcb_page))
+ error("Can't map GHCB page encrypted");
+}
+
+void do_boot_stage2_vc(struct pt_regs *regs, unsigned long exit_code)
+{
+ struct es_em_ctxt ctxt;
+ enum es_result result;
+
+ if (!boot_ghcb && !early_setup_sev_es())
+ sev_es_terminate(GHCB_SEV_ES_REASON_GENERAL_REQUEST);
+
+ vc_ghcb_invalidate(boot_ghcb);
+ result = vc_init_em_ctxt(&ctxt, regs, exit_code);
+ if (result != ES_OK)
+ goto finish;
+
+ switch (exit_code) {
+ default:
+ result = ES_UNSUPPORTED;
+ break;
+ }
+
+finish:
+ if (result == ES_OK) {
+ vc_finish_insn(&ctxt);
+ } else if (result != ES_RETRY) {
+ /*
+ * For now, just halt the machine. That makes debugging easier,
+ * later we just call sev_es_terminate() here.
+ */
+ while (true)
+ asm volatile("hlt\n");
+ }
+}
diff --git a/arch/x86/include/asm/sev-es.h b/arch/x86/include/asm/sev-es.h
index 48a44038b5d1..6dc52440c4b4 100644
--- a/arch/x86/include/asm/sev-es.h
+++ b/arch/x86/include/asm/sev-es.h
@@ -9,7 +9,14 @@
#define __ASM_ENCRYPTED_STATE_H
#include <linux/types.h>
+#include <asm/insn.h>
+#define GHCB_SEV_INFO 0x001UL
+#define GHCB_SEV_INFO_REQ 0x002UL
+#define GHCB_INFO(v) ((v) & 0xfffUL)
+#define GHCB_PROTO_MAX(v) (((v) >> 48) & 0xffffUL)
+#define GHCB_PROTO_MIN(v) (((v) >> 32) & 0xffffUL)
+#define GHCB_PROTO_OUR 0x0001UL
#define GHCB_SEV_CPUID_REQ 0x004UL
#define GHCB_CPUID_REQ_EAX 0
#define GHCB_CPUID_REQ_EBX 1
@@ -19,12 +26,44 @@
(((unsigned long)reg & 3) << 30) | \
(((unsigned long)fn) << 32))
+#define GHCB_PROTOCOL_MAX 0x0001UL
+#define GHCB_DEFAULT_USAGE 0x0000UL
+
#define GHCB_SEV_CPUID_RESP 0x005UL
#define GHCB_SEV_TERMINATE 0x100UL
+#define GHCB_SEV_TERMINATE_REASON(reason_set, reason_val) \
+ (((((u64)reason_set) & 0x7) << 12) | \
+ ((((u64)reason_val) & 0xff) << 16))
+#define GHCB_SEV_ES_REASON_GENERAL_REQUEST 0
+#define GHCB_SEV_ES_REASON_PROTOCOL_UNSUPPORTED 1
#define GHCB_SEV_GHCB_RESP_CODE(v) ((v) & 0xfff)
#define VMGEXIT() { asm volatile("rep; vmmcall\n\r"); }
+enum es_result {
+ ES_OK, /* All good */
+ ES_UNSUPPORTED, /* Requested operation not supported */
+ ES_VMM_ERROR, /* Unexpected state from the VMM */
+ ES_DECODE_FAILED, /* Instruction decoding failed */
+ ES_EXCEPTION, /* Instruction caused exception */
+ ES_RETRY, /* Retry instruction emulation */
+};
+
+struct es_fault_info {
+ unsigned long vector;
+ unsigned long error_code;
+ unsigned long cr2;
+};
+
+struct pt_regs;
+
+/* ES instruction emulation context */
+struct es_em_ctxt {
+ struct pt_regs *regs;
+ struct insn insn;
+ struct es_fault_info fi;
+};
+
void do_vc_no_ghcb(struct pt_regs *regs, unsigned long exit_code);
static inline u64 lower_bits(u64 val, unsigned int bits)
diff --git a/arch/x86/include/uapi/asm/svm.h b/arch/x86/include/uapi/asm/svm.h
index 2e8a30f06c74..c68d1618c9b0 100644
--- a/arch/x86/include/uapi/asm/svm.h
+++ b/arch/x86/include/uapi/asm/svm.h
@@ -29,6 +29,7 @@
#define SVM_EXIT_WRITE_DR6 0x036
#define SVM_EXIT_WRITE_DR7 0x037
#define SVM_EXIT_EXCP_BASE 0x040
+#define SVM_EXIT_LAST_EXCP 0x05f
#define SVM_EXIT_INTR 0x060
#define SVM_EXIT_NMI 0x061
#define SVM_EXIT_SMI 0x062
diff --git a/arch/x86/kernel/sev-es-shared.c b/arch/x86/kernel/sev-es-shared.c
index 0bea32341afa..7ac6e6b0ae57 100644
--- a/arch/x86/kernel/sev-es-shared.c
+++ b/arch/x86/kernel/sev-es-shared.c
@@ -9,6 +9,118 @@
* and is included directly into both code-bases.
*/
+static void sev_es_terminate(unsigned int reason)
+{
+ u64 val = GHCB_SEV_TERMINATE;
+
+ /*
+ * Tell the hypervisor what went wrong - only reason-set 0 is
+ * currently supported.
+ */
+ val |= GHCB_SEV_TERMINATE_REASON(0, reason);
+
+ /* Request Guest Termination from Hypvervisor */
+ sev_es_wr_ghcb_msr(val);
+ VMGEXIT();
+
+ while (true)
+ asm volatile("hlt\n" : : : "memory");
+}
+
+static bool sev_es_negotiate_protocol(void)
+{
+ u64 val;
+
+ /* Do the GHCB protocol version negotiation */
+ sev_es_wr_ghcb_msr(GHCB_SEV_INFO_REQ);
+ VMGEXIT();
+ val = sev_es_rd_ghcb_msr();
+
+ if (GHCB_INFO(val) != GHCB_SEV_INFO)
+ return false;
+
+ if (GHCB_PROTO_MAX(val) < GHCB_PROTO_OUR ||
+ GHCB_PROTO_MIN(val) > GHCB_PROTO_OUR)
+ return false;
+
+ return true;
+}
+
+static void vc_ghcb_invalidate(struct ghcb *ghcb)
+{
+ memset(ghcb->save.valid_bitmap, 0, sizeof(ghcb->save.valid_bitmap));
+}
+
+static bool vc_decoding_needed(unsigned long exit_code)
+{
+ /* Exceptions don't require to decode the instruction */
+ return !(exit_code >= SVM_EXIT_EXCP_BASE &&
+ exit_code <= SVM_EXIT_LAST_EXCP);
+}
+
+static enum es_result vc_init_em_ctxt(struct es_em_ctxt *ctxt,
+ struct pt_regs *regs,
+ unsigned long exit_code)
+{
+ enum es_result ret = ES_OK;
+
+ memset(ctxt, 0, sizeof(*ctxt));
+ ctxt->regs = regs;
+
+ if (vc_decoding_needed(exit_code))
+ ret = vc_decode_insn(ctxt);
+
+ return ret;
+}
+
+static void vc_finish_insn(struct es_em_ctxt *ctxt)
+{
+ ctxt->regs->ip += ctxt->insn.length;
+}
+
+static enum es_result sev_es_ghcb_hv_call(struct ghcb *ghcb,
+ struct es_em_ctxt *ctxt,
+ u64 exit_code, u64 exit_info_1,
+ u64 exit_info_2)
+{
+ enum es_result ret;
+
+ /* Fill in protocol and format specifiers */
+ ghcb->protocol_version = GHCB_PROTOCOL_MAX;
+ ghcb->ghcb_usage = GHCB_DEFAULT_USAGE;
+
+ ghcb_set_sw_exit_code(ghcb, exit_code);
+ ghcb_set_sw_exit_info_1(ghcb, exit_info_1);
+ ghcb_set_sw_exit_info_2(ghcb, exit_info_2);
+
+ sev_es_wr_ghcb_msr(__pa(ghcb));
+ VMGEXIT();
+
+ if ((ghcb->save.sw_exit_info_1 & 0xffffffff) == 1) {
+ u64 info = ghcb->save.sw_exit_info_2;
+ unsigned long v;
+
+ info = ghcb->save.sw_exit_info_2;
+ v = info & SVM_EVTINJ_VEC_MASK;
+
+ /* Check if exception information from hypervisor is sane. */
+ if ((info & SVM_EVTINJ_VALID) &&
+ ((v == X86_TRAP_GP) || (v == X86_TRAP_UD)) &&
+ ((info & SVM_EVTINJ_TYPE_MASK) == SVM_EVTINJ_TYPE_EXEPT)) {
+ ctxt->fi.vector = v;
+ if (info & SVM_EVTINJ_VALID_ERR)
+ ctxt->fi.error_code = info >> 32;
+ ret = ES_EXCEPTION;
+ } else {
+ ret = ES_VMM_ERROR;
+ }
+ } else {
+ ret = ES_OK;
+ }
+
+ return ret;
+}
+
/*
* Boot VC Handler - This is the first VC handler during boot, there is no GHCB
* page yet, so it only supports the MSR based communication with the
@@ -64,3 +176,45 @@ void __init do_vc_no_ghcb(struct pt_regs *regs, unsigned long exit_code)
while (true)
asm volatile("hlt\n");
}
+
+static enum es_result vc_insn_string_read(struct es_em_ctxt *ctxt,
+ void *src, char *buf,
+ unsigned int data_size,
+ unsigned int count,
+ bool backwards)
+{
+ int i, b = backwards ? -1 : 1;
+ enum es_result ret = ES_OK;
+
+ for (i = 0; i < count; i++) {
+ void *s = src + (i * data_size * b);
+ char *d = buf + (i * data_size);
+
+ ret = vc_read_mem(ctxt, s, d, data_size);
+ if (ret != ES_OK)
+ break;
+ }
+
+ return ret;
+}
+
+static enum es_result vc_insn_string_write(struct es_em_ctxt *ctxt,
+ void *dst, char *buf,
+ unsigned int data_size,
+ unsigned int count,
+ bool backwards)
+{
+ int i, s = backwards ? -1 : 1;
+ enum es_result ret = ES_OK;
+
+ for (i = 0; i < count; i++) {
+ void *d = dst + (i * data_size * s);
+ char *b = buf + (i * data_size);
+
+ ret = vc_write_mem(ctxt, d, b, data_size);
+ if (ret != ES_OK)
+ break;
+ }
+
+ return ret;
+}
--
2.28.0
^ permalink raw reply related [flat|nested] 264+ messages in thread
* [PATCH v7 23/72] x86/boot/compressed/64: Setup GHCB Based VC Exception handler
@ 2020-09-07 13:15 ` Joerg Roedel
0 siblings, 0 replies; 264+ messages in thread
From: Joerg Roedel @ 2020-09-07 13:15 UTC (permalink / raw)
To: x86
Cc: Juergen Gross, Tom Lendacky, Joerg Roedel, Mike Stunes,
Kees Cook, kvm, Peter Zijlstra, Cfir Cohen, Joerg Roedel,
Dave Hansen, linux-kernel, Sean Christopherson, virtualization,
Martin Radev, Masami Hiramatsu, Andy Lutomirski, hpa,
Erdem Aktas, David Rientjes, Dan Williams, Jiri Slaby
From: Joerg Roedel <jroedel@suse.de>
Install an exception handler for #VC exception that uses a GHCB. Also
add the infrastructure for handling different exit-codes by decoding
the instruction that caused the exception and error handling.
Signed-off-by: Joerg Roedel <jroedel@suse.de>
---
arch/x86/Kconfig | 1 +
arch/x86/boot/compressed/Makefile | 5 +
arch/x86/boot/compressed/idt_64.c | 4 +
arch/x86/boot/compressed/idt_handlers_64.S | 3 +-
arch/x86/boot/compressed/misc.c | 7 +
arch/x86/boot/compressed/misc.h | 7 +
arch/x86/boot/compressed/sev-es.c | 111 +++++++++++++++
arch/x86/include/asm/sev-es.h | 39 ++++++
arch/x86/include/uapi/asm/svm.h | 1 +
arch/x86/kernel/sev-es-shared.c | 154 +++++++++++++++++++++
10 files changed, 331 insertions(+), 1 deletion(-)
diff --git a/arch/x86/Kconfig b/arch/x86/Kconfig
index 8a48d3eedb84..5a604353ec42 100644
--- a/arch/x86/Kconfig
+++ b/arch/x86/Kconfig
@@ -1523,6 +1523,7 @@ config AMD_MEM_ENCRYPT
select DYNAMIC_PHYSICAL_MASK
select ARCH_USE_MEMREMAP_PROT
select ARCH_HAS_FORCE_DMA_UNENCRYPTED
+ select INSTRUCTION_DECODER
help
Say yes to enable support for the encryption of system memory.
This requires an AMD processor that supports Secure Memory
diff --git a/arch/x86/boot/compressed/Makefile b/arch/x86/boot/compressed/Makefile
index 7302f184bece..3224f1d9a5cd 100644
--- a/arch/x86/boot/compressed/Makefile
+++ b/arch/x86/boot/compressed/Makefile
@@ -45,6 +45,11 @@ KBUILD_CFLAGS += -fno-asynchronous-unwind-tables
KBUILD_CFLAGS += -D__DISABLE_EXPORTS
KBUILD_CFLAGS += -include $(srctree)/include/linux/hidden.h
+# sev-es.c indirectly inludes inat-table.h which is generated during
+# compilation and stored in $(objtree). Add the directory to the includes so
+# that the compiler finds it even with out-of-tree builds (make O=/some/path).
+CFLAGS_sev-es.o += -I$(objtree)/arch/x86/lib/
+
KBUILD_AFLAGS := $(KBUILD_CFLAGS) -D__ASSEMBLY__
GCOV_PROFILE := n
UBSAN_SANITIZE :=n
diff --git a/arch/x86/boot/compressed/idt_64.c b/arch/x86/boot/compressed/idt_64.c
index f3ca7324be44..804a502ee0d2 100644
--- a/arch/x86/boot/compressed/idt_64.c
+++ b/arch/x86/boot/compressed/idt_64.c
@@ -46,5 +46,9 @@ void load_stage2_idt(void)
set_idt_entry(X86_TRAP_PF, boot_page_fault);
+#ifdef CONFIG_AMD_MEM_ENCRYPT
+ set_idt_entry(X86_TRAP_VC, boot_stage2_vc);
+#endif
+
load_boot_idt(&boot_idt_desc);
}
diff --git a/arch/x86/boot/compressed/idt_handlers_64.S b/arch/x86/boot/compressed/idt_handlers_64.S
index 92eb4df478a1..22890e199f5b 100644
--- a/arch/x86/boot/compressed/idt_handlers_64.S
+++ b/arch/x86/boot/compressed/idt_handlers_64.S
@@ -72,5 +72,6 @@ SYM_FUNC_END(\name)
EXCEPTION_HANDLER boot_page_fault do_boot_page_fault error_code=1
#ifdef CONFIG_AMD_MEM_ENCRYPT
-EXCEPTION_HANDLER boot_stage1_vc do_vc_no_ghcb error_code=1
+EXCEPTION_HANDLER boot_stage1_vc do_vc_no_ghcb error_code=1
+EXCEPTION_HANDLER boot_stage2_vc do_boot_stage2_vc error_code=1
#endif
diff --git a/arch/x86/boot/compressed/misc.c b/arch/x86/boot/compressed/misc.c
index e478e40fbe5a..267e7f93050e 100644
--- a/arch/x86/boot/compressed/misc.c
+++ b/arch/x86/boot/compressed/misc.c
@@ -442,6 +442,13 @@ asmlinkage __visible void *extract_kernel(void *rmode, memptr heap,
parse_elf(output);
handle_relocations(output, output_len, virt_addr);
debug_putstr("done.\nBooting the kernel.\n");
+
+ /*
+ * Flush GHCB from cache and map it encrypted again when running as
+ * SEV-ES guest.
+ */
+ sev_es_shutdown_ghcb();
+
return output;
}
diff --git a/arch/x86/boot/compressed/misc.h b/arch/x86/boot/compressed/misc.h
index 01c0fb3417ca..9995c70ca813 100644
--- a/arch/x86/boot/compressed/misc.h
+++ b/arch/x86/boot/compressed/misc.h
@@ -115,6 +115,12 @@ static inline void console_init(void)
void set_sev_encryption_mask(void);
+#ifdef CONFIG_AMD_MEM_ENCRYPT
+void sev_es_shutdown_ghcb(void);
+#else
+static inline void sev_es_shutdown_ghcb(void) { }
+#endif
+
/* acpi.c */
#ifdef CONFIG_ACPI
acpi_physical_address get_rsdp_addr(void);
@@ -144,5 +150,6 @@ extern struct desc_ptr boot_idt_desc;
/* IDT Entry Points */
void boot_page_fault(void);
void boot_stage1_vc(void);
+void boot_stage2_vc(void);
#endif /* BOOT_COMPRESSED_MISC_H */
diff --git a/arch/x86/boot/compressed/sev-es.c b/arch/x86/boot/compressed/sev-es.c
index bb91cbb5920e..633d47df702f 100644
--- a/arch/x86/boot/compressed/sev-es.c
+++ b/arch/x86/boot/compressed/sev-es.c
@@ -13,10 +13,17 @@
#include "misc.h"
#include <asm/sev-es.h>
+#include <asm/trapnr.h>
+#include <asm/trap_pf.h>
#include <asm/msr-index.h>
#include <asm/ptrace.h>
#include <asm/svm.h>
+#include "error.h"
+
+struct ghcb boot_ghcb_page __aligned(PAGE_SIZE);
+struct ghcb *boot_ghcb;
+
static inline u64 sev_es_rd_ghcb_msr(void)
{
unsigned long low, high;
@@ -38,8 +45,112 @@ static inline void sev_es_wr_ghcb_msr(u64 val)
"a"(low), "d" (high) : "memory");
}
+static enum es_result vc_decode_insn(struct es_em_ctxt *ctxt)
+{
+ char buffer[MAX_INSN_SIZE];
+ enum es_result ret;
+
+ memcpy(buffer, (unsigned char *)ctxt->regs->ip, MAX_INSN_SIZE);
+
+ insn_init(&ctxt->insn, buffer, MAX_INSN_SIZE, 1);
+ insn_get_length(&ctxt->insn);
+
+ ret = ctxt->insn.immediate.got ? ES_OK : ES_DECODE_FAILED;
+
+ return ret;
+}
+
+static enum es_result vc_write_mem(struct es_em_ctxt *ctxt,
+ void *dst, char *buf, size_t size)
+{
+ memcpy(dst, buf, size);
+
+ return ES_OK;
+}
+
+static enum es_result vc_read_mem(struct es_em_ctxt *ctxt,
+ void *src, char *buf, size_t size)
+{
+ memcpy(buf, src, size);
+
+ return ES_OK;
+}
+
#undef __init
+#undef __pa
#define __init
+#define __pa(x) ((unsigned long)(x))
+
+#define __BOOT_COMPRESSED
+
+/* Basic instruction decoding support needed */
+#include "../../lib/inat.c"
+#include "../../lib/insn.c"
/* Include code for early handlers */
#include "../../kernel/sev-es-shared.c"
+
+static bool early_setup_sev_es(void)
+{
+ if (!sev_es_negotiate_protocol())
+ sev_es_terminate(GHCB_SEV_ES_REASON_PROTOCOL_UNSUPPORTED);
+
+ if (set_page_decrypted((unsigned long)&boot_ghcb_page))
+ return false;
+
+ /* Page is now mapped decrypted, clear it */
+ memset(&boot_ghcb_page, 0, sizeof(boot_ghcb_page));
+
+ boot_ghcb = &boot_ghcb_page;
+
+ /* Initialize lookup tables for the instruction decoder */
+ inat_init_tables();
+
+ return true;
+}
+
+void sev_es_shutdown_ghcb(void)
+{
+ if (!boot_ghcb)
+ return;
+
+ /*
+ * GHCB Page must be flushed from the cache and mapped encrypted again.
+ * Otherwise the running kernel will see strange cache effects when
+ * trying to use that page.
+ */
+ if (set_page_encrypted((unsigned long)&boot_ghcb_page))
+ error("Can't map GHCB page encrypted");
+}
+
+void do_boot_stage2_vc(struct pt_regs *regs, unsigned long exit_code)
+{
+ struct es_em_ctxt ctxt;
+ enum es_result result;
+
+ if (!boot_ghcb && !early_setup_sev_es())
+ sev_es_terminate(GHCB_SEV_ES_REASON_GENERAL_REQUEST);
+
+ vc_ghcb_invalidate(boot_ghcb);
+ result = vc_init_em_ctxt(&ctxt, regs, exit_code);
+ if (result != ES_OK)
+ goto finish;
+
+ switch (exit_code) {
+ default:
+ result = ES_UNSUPPORTED;
+ break;
+ }
+
+finish:
+ if (result == ES_OK) {
+ vc_finish_insn(&ctxt);
+ } else if (result != ES_RETRY) {
+ /*
+ * For now, just halt the machine. That makes debugging easier,
+ * later we just call sev_es_terminate() here.
+ */
+ while (true)
+ asm volatile("hlt\n");
+ }
+}
diff --git a/arch/x86/include/asm/sev-es.h b/arch/x86/include/asm/sev-es.h
index 48a44038b5d1..6dc52440c4b4 100644
--- a/arch/x86/include/asm/sev-es.h
+++ b/arch/x86/include/asm/sev-es.h
@@ -9,7 +9,14 @@
#define __ASM_ENCRYPTED_STATE_H
#include <linux/types.h>
+#include <asm/insn.h>
+#define GHCB_SEV_INFO 0x001UL
+#define GHCB_SEV_INFO_REQ 0x002UL
+#define GHCB_INFO(v) ((v) & 0xfffUL)
+#define GHCB_PROTO_MAX(v) (((v) >> 48) & 0xffffUL)
+#define GHCB_PROTO_MIN(v) (((v) >> 32) & 0xffffUL)
+#define GHCB_PROTO_OUR 0x0001UL
#define GHCB_SEV_CPUID_REQ 0x004UL
#define GHCB_CPUID_REQ_EAX 0
#define GHCB_CPUID_REQ_EBX 1
@@ -19,12 +26,44 @@
(((unsigned long)reg & 3) << 30) | \
(((unsigned long)fn) << 32))
+#define GHCB_PROTOCOL_MAX 0x0001UL
+#define GHCB_DEFAULT_USAGE 0x0000UL
+
#define GHCB_SEV_CPUID_RESP 0x005UL
#define GHCB_SEV_TERMINATE 0x100UL
+#define GHCB_SEV_TERMINATE_REASON(reason_set, reason_val) \
+ (((((u64)reason_set) & 0x7) << 12) | \
+ ((((u64)reason_val) & 0xff) << 16))
+#define GHCB_SEV_ES_REASON_GENERAL_REQUEST 0
+#define GHCB_SEV_ES_REASON_PROTOCOL_UNSUPPORTED 1
#define GHCB_SEV_GHCB_RESP_CODE(v) ((v) & 0xfff)
#define VMGEXIT() { asm volatile("rep; vmmcall\n\r"); }
+enum es_result {
+ ES_OK, /* All good */
+ ES_UNSUPPORTED, /* Requested operation not supported */
+ ES_VMM_ERROR, /* Unexpected state from the VMM */
+ ES_DECODE_FAILED, /* Instruction decoding failed */
+ ES_EXCEPTION, /* Instruction caused exception */
+ ES_RETRY, /* Retry instruction emulation */
+};
+
+struct es_fault_info {
+ unsigned long vector;
+ unsigned long error_code;
+ unsigned long cr2;
+};
+
+struct pt_regs;
+
+/* ES instruction emulation context */
+struct es_em_ctxt {
+ struct pt_regs *regs;
+ struct insn insn;
+ struct es_fault_info fi;
+};
+
void do_vc_no_ghcb(struct pt_regs *regs, unsigned long exit_code);
static inline u64 lower_bits(u64 val, unsigned int bits)
diff --git a/arch/x86/include/uapi/asm/svm.h b/arch/x86/include/uapi/asm/svm.h
index 2e8a30f06c74..c68d1618c9b0 100644
--- a/arch/x86/include/uapi/asm/svm.h
+++ b/arch/x86/include/uapi/asm/svm.h
@@ -29,6 +29,7 @@
#define SVM_EXIT_WRITE_DR6 0x036
#define SVM_EXIT_WRITE_DR7 0x037
#define SVM_EXIT_EXCP_BASE 0x040
+#define SVM_EXIT_LAST_EXCP 0x05f
#define SVM_EXIT_INTR 0x060
#define SVM_EXIT_NMI 0x061
#define SVM_EXIT_SMI 0x062
diff --git a/arch/x86/kernel/sev-es-shared.c b/arch/x86/kernel/sev-es-shared.c
index 0bea32341afa..7ac6e6b0ae57 100644
--- a/arch/x86/kernel/sev-es-shared.c
+++ b/arch/x86/kernel/sev-es-shared.c
@@ -9,6 +9,118 @@
* and is included directly into both code-bases.
*/
+static void sev_es_terminate(unsigned int reason)
+{
+ u64 val = GHCB_SEV_TERMINATE;
+
+ /*
+ * Tell the hypervisor what went wrong - only reason-set 0 is
+ * currently supported.
+ */
+ val |= GHCB_SEV_TERMINATE_REASON(0, reason);
+
+ /* Request Guest Termination from Hypvervisor */
+ sev_es_wr_ghcb_msr(val);
+ VMGEXIT();
+
+ while (true)
+ asm volatile("hlt\n" : : : "memory");
+}
+
+static bool sev_es_negotiate_protocol(void)
+{
+ u64 val;
+
+ /* Do the GHCB protocol version negotiation */
+ sev_es_wr_ghcb_msr(GHCB_SEV_INFO_REQ);
+ VMGEXIT();
+ val = sev_es_rd_ghcb_msr();
+
+ if (GHCB_INFO(val) != GHCB_SEV_INFO)
+ return false;
+
+ if (GHCB_PROTO_MAX(val) < GHCB_PROTO_OUR ||
+ GHCB_PROTO_MIN(val) > GHCB_PROTO_OUR)
+ return false;
+
+ return true;
+}
+
+static void vc_ghcb_invalidate(struct ghcb *ghcb)
+{
+ memset(ghcb->save.valid_bitmap, 0, sizeof(ghcb->save.valid_bitmap));
+}
+
+static bool vc_decoding_needed(unsigned long exit_code)
+{
+ /* Exceptions don't require to decode the instruction */
+ return !(exit_code >= SVM_EXIT_EXCP_BASE &&
+ exit_code <= SVM_EXIT_LAST_EXCP);
+}
+
+static enum es_result vc_init_em_ctxt(struct es_em_ctxt *ctxt,
+ struct pt_regs *regs,
+ unsigned long exit_code)
+{
+ enum es_result ret = ES_OK;
+
+ memset(ctxt, 0, sizeof(*ctxt));
+ ctxt->regs = regs;
+
+ if (vc_decoding_needed(exit_code))
+ ret = vc_decode_insn(ctxt);
+
+ return ret;
+}
+
+static void vc_finish_insn(struct es_em_ctxt *ctxt)
+{
+ ctxt->regs->ip += ctxt->insn.length;
+}
+
+static enum es_result sev_es_ghcb_hv_call(struct ghcb *ghcb,
+ struct es_em_ctxt *ctxt,
+ u64 exit_code, u64 exit_info_1,
+ u64 exit_info_2)
+{
+ enum es_result ret;
+
+ /* Fill in protocol and format specifiers */
+ ghcb->protocol_version = GHCB_PROTOCOL_MAX;
+ ghcb->ghcb_usage = GHCB_DEFAULT_USAGE;
+
+ ghcb_set_sw_exit_code(ghcb, exit_code);
+ ghcb_set_sw_exit_info_1(ghcb, exit_info_1);
+ ghcb_set_sw_exit_info_2(ghcb, exit_info_2);
+
+ sev_es_wr_ghcb_msr(__pa(ghcb));
+ VMGEXIT();
+
+ if ((ghcb->save.sw_exit_info_1 & 0xffffffff) == 1) {
+ u64 info = ghcb->save.sw_exit_info_2;
+ unsigned long v;
+
+ info = ghcb->save.sw_exit_info_2;
+ v = info & SVM_EVTINJ_VEC_MASK;
+
+ /* Check if exception information from hypervisor is sane. */
+ if ((info & SVM_EVTINJ_VALID) &&
+ ((v == X86_TRAP_GP) || (v == X86_TRAP_UD)) &&
+ ((info & SVM_EVTINJ_TYPE_MASK) == SVM_EVTINJ_TYPE_EXEPT)) {
+ ctxt->fi.vector = v;
+ if (info & SVM_EVTINJ_VALID_ERR)
+ ctxt->fi.error_code = info >> 32;
+ ret = ES_EXCEPTION;
+ } else {
+ ret = ES_VMM_ERROR;
+ }
+ } else {
+ ret = ES_OK;
+ }
+
+ return ret;
+}
+
/*
* Boot VC Handler - This is the first VC handler during boot, there is no GHCB
* page yet, so it only supports the MSR based communication with the
@@ -64,3 +176,45 @@ void __init do_vc_no_ghcb(struct pt_regs *regs, unsigned long exit_code)
while (true)
asm volatile("hlt\n");
}
+
+static enum es_result vc_insn_string_read(struct es_em_ctxt *ctxt,
+ void *src, char *buf,
+ unsigned int data_size,
+ unsigned int count,
+ bool backwards)
+{
+ int i, b = backwards ? -1 : 1;
+ enum es_result ret = ES_OK;
+
+ for (i = 0; i < count; i++) {
+ void *s = src + (i * data_size * b);
+ char *d = buf + (i * data_size);
+
+ ret = vc_read_mem(ctxt, s, d, data_size);
+ if (ret != ES_OK)
+ break;
+ }
+
+ return ret;
+}
+
+static enum es_result vc_insn_string_write(struct es_em_ctxt *ctxt,
+ void *dst, char *buf,
+ unsigned int data_size,
+ unsigned int count,
+ bool backwards)
+{
+ int i, s = backwards ? -1 : 1;
+ enum es_result ret = ES_OK;
+
+ for (i = 0; i < count; i++) {
+ void *d = dst + (i * data_size * s);
+ char *b = buf + (i * data_size);
+
+ ret = vc_write_mem(ctxt, d, b, data_size);
+ if (ret != ES_OK)
+ break;
+ }
+
+ return ret;
+}
--
2.28.0
_______________________________________________
Virtualization mailing list
Virtualization@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/virtualization
^ permalink raw reply related [flat|nested] 264+ messages in thread
* [PATCH v7 24/72] x86/boot/compressed/64: Unmap GHCB page before booting the kernel
2020-09-07 13:15 ` Joerg Roedel
@ 2020-09-07 13:15 ` Joerg Roedel
-1 siblings, 0 replies; 264+ messages in thread
From: Joerg Roedel @ 2020-09-07 13:15 UTC (permalink / raw)
To: x86
Cc: Joerg Roedel, Joerg Roedel, hpa, Andy Lutomirski, Dave Hansen,
Peter Zijlstra, Jiri Slaby, Dan Williams, Tom Lendacky,
Juergen Gross, Kees Cook, David Rientjes, Cfir Cohen,
Erdem Aktas, Masami Hiramatsu, Mike Stunes, Sean Christopherson,
Martin Radev, linux-kernel, kvm, virtualization
From: Joerg Roedel <jroedel@suse.de>
Force a page-fault on any further accesses to the GHCB page when they
shouldn't happen anymore. This will catch the bugs where a #VC exception
is raised when no one is expected anymore.
Signed-off-by: Joerg Roedel <jroedel@suse.de>
---
arch/x86/boot/compressed/ident_map_64.c | 17 +++++++++++++++--
arch/x86/boot/compressed/misc.h | 6 ++++++
arch/x86/boot/compressed/sev-es.c | 14 ++++++++++++++
3 files changed, 35 insertions(+), 2 deletions(-)
diff --git a/arch/x86/boot/compressed/ident_map_64.c b/arch/x86/boot/compressed/ident_map_64.c
index 05742f641a06..063a60edcf99 100644
--- a/arch/x86/boot/compressed/ident_map_64.c
+++ b/arch/x86/boot/compressed/ident_map_64.c
@@ -298,6 +298,11 @@ int set_page_encrypted(unsigned long address)
return set_clr_page_flags(&mapping_info, address, _PAGE_ENC, 0);
}
+int set_page_non_present(unsigned long address)
+{
+ return set_clr_page_flags(&mapping_info, address, 0, _PAGE_PRESENT);
+}
+
static void do_pf_error(const char *msg, unsigned long error_code,
unsigned long address, unsigned long ip)
{
@@ -316,8 +321,14 @@ static void do_pf_error(const char *msg, unsigned long error_code,
void do_boot_page_fault(struct pt_regs *regs, unsigned long error_code)
{
- unsigned long address = native_read_cr2() & PMD_MASK;
- unsigned long end = address + PMD_SIZE;
+ unsigned long address = native_read_cr2();
+ unsigned long end;
+ bool ghcb_fault;
+
+ ghcb_fault = sev_es_check_ghcb_fault(address);
+
+ address &= PMD_MASK;
+ end = address + PMD_SIZE;
/*
* Check for unexpected error codes. Unexpected are:
@@ -327,6 +338,8 @@ void do_boot_page_fault(struct pt_regs *regs, unsigned long error_code)
*/
if (error_code & (X86_PF_PROT | X86_PF_USER | X86_PF_RSVD))
do_pf_error("Unexpected page-fault:", error_code, address, regs->ip);
+ else if (ghcb_fault)
+ do_pf_error("Page-fault on GHCB page:", error_code, address, regs->ip);
/*
* Error code is sane - now identity map the 2M region around
diff --git a/arch/x86/boot/compressed/misc.h b/arch/x86/boot/compressed/misc.h
index 9995c70ca813..c0e0ffeee50a 100644
--- a/arch/x86/boot/compressed/misc.h
+++ b/arch/x86/boot/compressed/misc.h
@@ -100,6 +100,7 @@ static inline void choose_random_location(unsigned long input,
#ifdef CONFIG_X86_64
extern int set_page_decrypted(unsigned long address);
extern int set_page_encrypted(unsigned long address);
+extern int set_page_non_present(unsigned long address);
extern unsigned char _pgtable[];
#endif
@@ -117,8 +118,13 @@ void set_sev_encryption_mask(void);
#ifdef CONFIG_AMD_MEM_ENCRYPT
void sev_es_shutdown_ghcb(void);
+extern bool sev_es_check_ghcb_fault(unsigned long address);
#else
static inline void sev_es_shutdown_ghcb(void) { }
+static inline bool sev_es_check_ghcb_fault(unsigned long address)
+{
+ return false;
+}
#endif
/* acpi.c */
diff --git a/arch/x86/boot/compressed/sev-es.c b/arch/x86/boot/compressed/sev-es.c
index 633d47df702f..ea656b22743b 100644
--- a/arch/x86/boot/compressed/sev-es.c
+++ b/arch/x86/boot/compressed/sev-es.c
@@ -121,6 +121,20 @@ void sev_es_shutdown_ghcb(void)
*/
if (set_page_encrypted((unsigned long)&boot_ghcb_page))
error("Can't map GHCB page encrypted");
+
+ /*
+ * GHCB page is mapped encrypted again and flushed from the cache.
+ * Mark it non-present now to catch bugs when #VC exceptions trigger
+ * after this point.
+ */
+ if (set_page_non_present((unsigned long)&boot_ghcb_page))
+ error("Can't unmap GHCB page");
+}
+
+bool sev_es_check_ghcb_fault(unsigned long address)
+{
+ /* Check whether the fault was on the GHCB page */
+ return ((address & PAGE_MASK) == (unsigned long)&boot_ghcb_page);
}
void do_boot_stage2_vc(struct pt_regs *regs, unsigned long exit_code)
--
2.28.0
^ permalink raw reply related [flat|nested] 264+ messages in thread
* [PATCH v7 24/72] x86/boot/compressed/64: Unmap GHCB page before booting the kernel
@ 2020-09-07 13:15 ` Joerg Roedel
0 siblings, 0 replies; 264+ messages in thread
From: Joerg Roedel @ 2020-09-07 13:15 UTC (permalink / raw)
To: x86
Cc: Juergen Gross, Tom Lendacky, Joerg Roedel, Mike Stunes,
Kees Cook, kvm, Peter Zijlstra, Cfir Cohen, Joerg Roedel,
Dave Hansen, linux-kernel, Sean Christopherson, virtualization,
Martin Radev, Masami Hiramatsu, Andy Lutomirski, hpa,
Erdem Aktas, David Rientjes, Dan Williams, Jiri Slaby
From: Joerg Roedel <jroedel@suse.de>
Force a page-fault on any further accesses to the GHCB page when they
shouldn't happen anymore. This will catch the bugs where a #VC exception
is raised when no one is expected anymore.
Signed-off-by: Joerg Roedel <jroedel@suse.de>
---
arch/x86/boot/compressed/ident_map_64.c | 17 +++++++++++++++--
arch/x86/boot/compressed/misc.h | 6 ++++++
arch/x86/boot/compressed/sev-es.c | 14 ++++++++++++++
3 files changed, 35 insertions(+), 2 deletions(-)
diff --git a/arch/x86/boot/compressed/ident_map_64.c b/arch/x86/boot/compressed/ident_map_64.c
index 05742f641a06..063a60edcf99 100644
--- a/arch/x86/boot/compressed/ident_map_64.c
+++ b/arch/x86/boot/compressed/ident_map_64.c
@@ -298,6 +298,11 @@ int set_page_encrypted(unsigned long address)
return set_clr_page_flags(&mapping_info, address, _PAGE_ENC, 0);
}
+int set_page_non_present(unsigned long address)
+{
+ return set_clr_page_flags(&mapping_info, address, 0, _PAGE_PRESENT);
+}
+
static void do_pf_error(const char *msg, unsigned long error_code,
unsigned long address, unsigned long ip)
{
@@ -316,8 +321,14 @@ static void do_pf_error(const char *msg, unsigned long error_code,
void do_boot_page_fault(struct pt_regs *regs, unsigned long error_code)
{
- unsigned long address = native_read_cr2() & PMD_MASK;
- unsigned long end = address + PMD_SIZE;
+ unsigned long address = native_read_cr2();
+ unsigned long end;
+ bool ghcb_fault;
+
+ ghcb_fault = sev_es_check_ghcb_fault(address);
+
+ address &= PMD_MASK;
+ end = address + PMD_SIZE;
/*
* Check for unexpected error codes. Unexpected are:
@@ -327,6 +338,8 @@ void do_boot_page_fault(struct pt_regs *regs, unsigned long error_code)
*/
if (error_code & (X86_PF_PROT | X86_PF_USER | X86_PF_RSVD))
do_pf_error("Unexpected page-fault:", error_code, address, regs->ip);
+ else if (ghcb_fault)
+ do_pf_error("Page-fault on GHCB page:", error_code, address, regs->ip);
/*
* Error code is sane - now identity map the 2M region around
diff --git a/arch/x86/boot/compressed/misc.h b/arch/x86/boot/compressed/misc.h
index 9995c70ca813..c0e0ffeee50a 100644
--- a/arch/x86/boot/compressed/misc.h
+++ b/arch/x86/boot/compressed/misc.h
@@ -100,6 +100,7 @@ static inline void choose_random_location(unsigned long input,
#ifdef CONFIG_X86_64
extern int set_page_decrypted(unsigned long address);
extern int set_page_encrypted(unsigned long address);
+extern int set_page_non_present(unsigned long address);
extern unsigned char _pgtable[];
#endif
@@ -117,8 +118,13 @@ void set_sev_encryption_mask(void);
#ifdef CONFIG_AMD_MEM_ENCRYPT
void sev_es_shutdown_ghcb(void);
+extern bool sev_es_check_ghcb_fault(unsigned long address);
#else
static inline void sev_es_shutdown_ghcb(void) { }
+static inline bool sev_es_check_ghcb_fault(unsigned long address)
+{
+ return false;
+}
#endif
/* acpi.c */
diff --git a/arch/x86/boot/compressed/sev-es.c b/arch/x86/boot/compressed/sev-es.c
index 633d47df702f..ea656b22743b 100644
--- a/arch/x86/boot/compressed/sev-es.c
+++ b/arch/x86/boot/compressed/sev-es.c
@@ -121,6 +121,20 @@ void sev_es_shutdown_ghcb(void)
*/
if (set_page_encrypted((unsigned long)&boot_ghcb_page))
error("Can't map GHCB page encrypted");
+
+ /*
+ * GHCB page is mapped encrypted again and flushed from the cache.
+ * Mark it non-present now to catch bugs when #VC exceptions trigger
+ * after this point.
+ */
+ if (set_page_non_present((unsigned long)&boot_ghcb_page))
+ error("Can't unmap GHCB page");
+}
+
+bool sev_es_check_ghcb_fault(unsigned long address)
+{
+ /* Check whether the fault was on the GHCB page */
+ return ((address & PAGE_MASK) == (unsigned long)&boot_ghcb_page);
}
void do_boot_stage2_vc(struct pt_regs *regs, unsigned long exit_code)
--
2.28.0
_______________________________________________
Virtualization mailing list
Virtualization@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/virtualization
^ permalink raw reply related [flat|nested] 264+ messages in thread
* [PATCH v7 25/72] x86/sev-es: Add support for handling IOIO exceptions
2020-09-07 13:15 ` Joerg Roedel
@ 2020-09-07 13:15 ` Joerg Roedel
-1 siblings, 0 replies; 264+ messages in thread
From: Joerg Roedel @ 2020-09-07 13:15 UTC (permalink / raw)
To: x86
Cc: Joerg Roedel, Joerg Roedel, Tom Lendacky, hpa, Andy Lutomirski,
Dave Hansen, Peter Zijlstra, Jiri Slaby, Dan Williams,
Juergen Gross, Kees Cook, David Rientjes, Cfir Cohen,
Erdem Aktas, Masami Hiramatsu, Mike Stunes, Sean Christopherson,
Martin Radev, linux-kernel, kvm, virtualization
From: Tom Lendacky <thomas.lendacky@amd.com>
Add support for decoding and handling #VC exceptions for IOIO events.
Signed-off-by: Tom Lendacky <thomas.lendacky@amd.com>
[ jroedel@suse.de: Adapted code to #VC handling framework ]
Co-developed-by: Joerg Roedel <jroedel@suse.de>
Signed-off-by: Joerg Roedel <jroedel@suse.de>
---
arch/x86/boot/compressed/sev-es.c | 32 +++++
arch/x86/kernel/sev-es-shared.c | 214 ++++++++++++++++++++++++++++++
2 files changed, 246 insertions(+)
diff --git a/arch/x86/boot/compressed/sev-es.c b/arch/x86/boot/compressed/sev-es.c
index ea656b22743b..d7ab212180b6 100644
--- a/arch/x86/boot/compressed/sev-es.c
+++ b/arch/x86/boot/compressed/sev-es.c
@@ -24,6 +24,35 @@
struct ghcb boot_ghcb_page __aligned(PAGE_SIZE);
struct ghcb *boot_ghcb;
+/*
+ * Copy a version of this function here - insn-eval.c can't be used in
+ * pre-decompression code.
+ */
+static bool insn_has_rep_prefix(struct insn *insn)
+{
+ int i;
+
+ insn_get_prefixes(insn);
+
+ for (i = 0; i < insn->prefixes.nbytes; i++) {
+ insn_byte_t p = insn->prefixes.bytes[i];
+
+ if (p == 0xf2 || p == 0xf3)
+ return true;
+ }
+
+ return false;
+}
+
+/*
+ * Only a dummy for insn_get_seg_base() - Early boot-code is 64bit only and
+ * doesn't use segments.
+ */
+static unsigned long insn_get_seg_base(struct pt_regs *regs, int seg_reg_idx)
+{
+ return 0UL;
+}
+
static inline u64 sev_es_rd_ghcb_msr(void)
{
unsigned long low, high;
@@ -151,6 +180,9 @@ void do_boot_stage2_vc(struct pt_regs *regs, unsigned long exit_code)
goto finish;
switch (exit_code) {
+ case SVM_EXIT_IOIO:
+ result = vc_handle_ioio(boot_ghcb, &ctxt);
+ break;
default:
result = ES_UNSUPPORTED;
break;
diff --git a/arch/x86/kernel/sev-es-shared.c b/arch/x86/kernel/sev-es-shared.c
index 7ac6e6b0ae57..bae7cf28455b 100644
--- a/arch/x86/kernel/sev-es-shared.c
+++ b/arch/x86/kernel/sev-es-shared.c
@@ -218,3 +218,217 @@ static enum es_result vc_insn_string_write(struct es_em_ctxt *ctxt,
return ret;
}
+
+#define IOIO_TYPE_STR BIT(2)
+#define IOIO_TYPE_IN 1
+#define IOIO_TYPE_INS (IOIO_TYPE_IN | IOIO_TYPE_STR)
+#define IOIO_TYPE_OUT 0
+#define IOIO_TYPE_OUTS (IOIO_TYPE_OUT | IOIO_TYPE_STR)
+
+#define IOIO_REP BIT(3)
+
+#define IOIO_ADDR_64 BIT(9)
+#define IOIO_ADDR_32 BIT(8)
+#define IOIO_ADDR_16 BIT(7)
+
+#define IOIO_DATA_32 BIT(6)
+#define IOIO_DATA_16 BIT(5)
+#define IOIO_DATA_8 BIT(4)
+
+#define IOIO_SEG_ES (0 << 10)
+#define IOIO_SEG_DS (3 << 10)
+
+static enum es_result vc_ioio_exitinfo(struct es_em_ctxt *ctxt, u64 *exitinfo)
+{
+ struct insn *insn = &ctxt->insn;
+ *exitinfo = 0;
+
+ switch (insn->opcode.bytes[0]) {
+ /* INS opcodes */
+ case 0x6c:
+ case 0x6d:
+ *exitinfo |= IOIO_TYPE_INS;
+ *exitinfo |= IOIO_SEG_ES;
+ *exitinfo |= (ctxt->regs->dx & 0xffff) << 16;
+ break;
+
+ /* OUTS opcodes */
+ case 0x6e:
+ case 0x6f:
+ *exitinfo |= IOIO_TYPE_OUTS;
+ *exitinfo |= IOIO_SEG_DS;
+ *exitinfo |= (ctxt->regs->dx & 0xffff) << 16;
+ break;
+
+ /* IN immediate opcodes */
+ case 0xe4:
+ case 0xe5:
+ *exitinfo |= IOIO_TYPE_IN;
+ *exitinfo |= (u64)insn->immediate.value << 16;
+ break;
+
+ /* OUT immediate opcodes */
+ case 0xe6:
+ case 0xe7:
+ *exitinfo |= IOIO_TYPE_OUT;
+ *exitinfo |= (u64)insn->immediate.value << 16;
+ break;
+
+ /* IN register opcodes */
+ case 0xec:
+ case 0xed:
+ *exitinfo |= IOIO_TYPE_IN;
+ *exitinfo |= (ctxt->regs->dx & 0xffff) << 16;
+ break;
+
+ /* OUT register opcodes */
+ case 0xee:
+ case 0xef:
+ *exitinfo |= IOIO_TYPE_OUT;
+ *exitinfo |= (ctxt->regs->dx & 0xffff) << 16;
+ break;
+
+ default:
+ return ES_DECODE_FAILED;
+ }
+
+ switch (insn->opcode.bytes[0]) {
+ case 0x6c:
+ case 0x6e:
+ case 0xe4:
+ case 0xe6:
+ case 0xec:
+ case 0xee:
+ /* Single byte opcodes */
+ *exitinfo |= IOIO_DATA_8;
+ break;
+ default:
+ /* Length determined by instruction parsing */
+ *exitinfo |= (insn->opnd_bytes == 2) ? IOIO_DATA_16
+ : IOIO_DATA_32;
+ }
+ switch (insn->addr_bytes) {
+ case 2:
+ *exitinfo |= IOIO_ADDR_16;
+ break;
+ case 4:
+ *exitinfo |= IOIO_ADDR_32;
+ break;
+ case 8:
+ *exitinfo |= IOIO_ADDR_64;
+ break;
+ }
+
+ if (insn_has_rep_prefix(insn))
+ *exitinfo |= IOIO_REP;
+
+ return ES_OK;
+}
+
+static enum es_result vc_handle_ioio(struct ghcb *ghcb, struct es_em_ctxt *ctxt)
+{
+ struct pt_regs *regs = ctxt->regs;
+ u64 exit_info_1, exit_info_2;
+ enum es_result ret;
+
+ ret = vc_ioio_exitinfo(ctxt, &exit_info_1);
+ if (ret != ES_OK)
+ return ret;
+
+ if (exit_info_1 & IOIO_TYPE_STR) {
+
+ /* (REP) INS/OUTS */
+
+ bool df = ((regs->flags & X86_EFLAGS_DF) == X86_EFLAGS_DF);
+ unsigned int io_bytes, exit_bytes;
+ unsigned int ghcb_count, op_count;
+ unsigned long es_base;
+ u64 sw_scratch;
+
+ /*
+ * For the string variants with rep prefix the amount of in/out
+ * operations per #VC exception is limited so that the kernel
+ * has a chance to take interrupts and re-schedule while the
+ * instruction is emulated.
+ */
+ io_bytes = (exit_info_1 >> 4) & 0x7;
+ ghcb_count = sizeof(ghcb->shared_buffer) / io_bytes;
+
+ op_count = (exit_info_1 & IOIO_REP) ? regs->cx : 1;
+ exit_info_2 = min(op_count, ghcb_count);
+ exit_bytes = exit_info_2 * io_bytes;
+
+ es_base = insn_get_seg_base(ctxt->regs, INAT_SEG_REG_ES);
+
+ /* Read bytes of OUTS into the shared buffer */
+ if (!(exit_info_1 & IOIO_TYPE_IN)) {
+ ret = vc_insn_string_read(ctxt,
+ (void *)(es_base + regs->si),
+ ghcb->shared_buffer, io_bytes,
+ exit_info_2, df);
+ if (ret)
+ return ret;
+ }
+
+ /*
+ * Issue an VMGEXIT to the HV to consume the bytes from the
+ * shared buffer or to have it write them into the shared buffer
+ * depending on the instruction: OUTS or INS.
+ */
+ sw_scratch = __pa(ghcb) + offsetof(struct ghcb, shared_buffer);
+ ghcb_set_sw_scratch(ghcb, sw_scratch);
+ ret = sev_es_ghcb_hv_call(ghcb, ctxt, SVM_EXIT_IOIO,
+ exit_info_1, exit_info_2);
+ if (ret != ES_OK)
+ return ret;
+
+ /* Read bytes from shared buffer into the guest's destination. */
+ if (exit_info_1 & IOIO_TYPE_IN) {
+ ret = vc_insn_string_write(ctxt,
+ (void *)(es_base + regs->di),
+ ghcb->shared_buffer, io_bytes,
+ exit_info_2, df);
+ if (ret)
+ return ret;
+
+ if (df)
+ regs->di -= exit_bytes;
+ else
+ regs->di += exit_bytes;
+ } else {
+ if (df)
+ regs->si -= exit_bytes;
+ else
+ regs->si += exit_bytes;
+ }
+
+ if (exit_info_1 & IOIO_REP)
+ regs->cx -= exit_info_2;
+
+ ret = regs->cx ? ES_RETRY : ES_OK;
+
+ } else {
+
+ /* IN/OUT into/from rAX */
+
+ int bits = (exit_info_1 & 0x70) >> 1;
+ u64 rax = 0;
+
+ if (!(exit_info_1 & IOIO_TYPE_IN))
+ rax = lower_bits(regs->ax, bits);
+
+ ghcb_set_rax(ghcb, rax);
+
+ ret = sev_es_ghcb_hv_call(ghcb, ctxt, SVM_EXIT_IOIO, exit_info_1, 0);
+ if (ret != ES_OK)
+ return ret;
+
+ if (exit_info_1 & IOIO_TYPE_IN) {
+ if (!ghcb_rax_is_valid(ghcb))
+ return ES_VMM_ERROR;
+ regs->ax = lower_bits(ghcb->save.rax, bits);
+ }
+ }
+
+ return ret;
+}
--
2.28.0
^ permalink raw reply related [flat|nested] 264+ messages in thread
* [PATCH v7 25/72] x86/sev-es: Add support for handling IOIO exceptions
@ 2020-09-07 13:15 ` Joerg Roedel
0 siblings, 0 replies; 264+ messages in thread
From: Joerg Roedel @ 2020-09-07 13:15 UTC (permalink / raw)
To: x86
Cc: Juergen Gross, Tom Lendacky, Joerg Roedel, Mike Stunes,
Kees Cook, kvm, Peter Zijlstra, Cfir Cohen, Joerg Roedel,
Dave Hansen, linux-kernel, Sean Christopherson, virtualization,
Martin Radev, Masami Hiramatsu, Andy Lutomirski, hpa,
Erdem Aktas, David Rientjes, Dan Williams, Jiri Slaby
From: Tom Lendacky <thomas.lendacky@amd.com>
Add support for decoding and handling #VC exceptions for IOIO events.
Signed-off-by: Tom Lendacky <thomas.lendacky@amd.com>
[ jroedel@suse.de: Adapted code to #VC handling framework ]
Co-developed-by: Joerg Roedel <jroedel@suse.de>
Signed-off-by: Joerg Roedel <jroedel@suse.de>
---
arch/x86/boot/compressed/sev-es.c | 32 +++++
arch/x86/kernel/sev-es-shared.c | 214 ++++++++++++++++++++++++++++++
2 files changed, 246 insertions(+)
diff --git a/arch/x86/boot/compressed/sev-es.c b/arch/x86/boot/compressed/sev-es.c
index ea656b22743b..d7ab212180b6 100644
--- a/arch/x86/boot/compressed/sev-es.c
+++ b/arch/x86/boot/compressed/sev-es.c
@@ -24,6 +24,35 @@
struct ghcb boot_ghcb_page __aligned(PAGE_SIZE);
struct ghcb *boot_ghcb;
+/*
+ * Copy a version of this function here - insn-eval.c can't be used in
+ * pre-decompression code.
+ */
+static bool insn_has_rep_prefix(struct insn *insn)
+{
+ int i;
+
+ insn_get_prefixes(insn);
+
+ for (i = 0; i < insn->prefixes.nbytes; i++) {
+ insn_byte_t p = insn->prefixes.bytes[i];
+
+ if (p == 0xf2 || p == 0xf3)
+ return true;
+ }
+
+ return false;
+}
+
+/*
+ * Only a dummy for insn_get_seg_base() - Early boot-code is 64bit only and
+ * doesn't use segments.
+ */
+static unsigned long insn_get_seg_base(struct pt_regs *regs, int seg_reg_idx)
+{
+ return 0UL;
+}
+
static inline u64 sev_es_rd_ghcb_msr(void)
{
unsigned long low, high;
@@ -151,6 +180,9 @@ void do_boot_stage2_vc(struct pt_regs *regs, unsigned long exit_code)
goto finish;
switch (exit_code) {
+ case SVM_EXIT_IOIO:
+ result = vc_handle_ioio(boot_ghcb, &ctxt);
+ break;
default:
result = ES_UNSUPPORTED;
break;
diff --git a/arch/x86/kernel/sev-es-shared.c b/arch/x86/kernel/sev-es-shared.c
index 7ac6e6b0ae57..bae7cf28455b 100644
--- a/arch/x86/kernel/sev-es-shared.c
+++ b/arch/x86/kernel/sev-es-shared.c
@@ -218,3 +218,217 @@ static enum es_result vc_insn_string_write(struct es_em_ctxt *ctxt,
return ret;
}
+
+#define IOIO_TYPE_STR BIT(2)
+#define IOIO_TYPE_IN 1
+#define IOIO_TYPE_INS (IOIO_TYPE_IN | IOIO_TYPE_STR)
+#define IOIO_TYPE_OUT 0
+#define IOIO_TYPE_OUTS (IOIO_TYPE_OUT | IOIO_TYPE_STR)
+
+#define IOIO_REP BIT(3)
+
+#define IOIO_ADDR_64 BIT(9)
+#define IOIO_ADDR_32 BIT(8)
+#define IOIO_ADDR_16 BIT(7)
+
+#define IOIO_DATA_32 BIT(6)
+#define IOIO_DATA_16 BIT(5)
+#define IOIO_DATA_8 BIT(4)
+
+#define IOIO_SEG_ES (0 << 10)
+#define IOIO_SEG_DS (3 << 10)
+
+static enum es_result vc_ioio_exitinfo(struct es_em_ctxt *ctxt, u64 *exitinfo)
+{
+ struct insn *insn = &ctxt->insn;
+ *exitinfo = 0;
+
+ switch (insn->opcode.bytes[0]) {
+ /* INS opcodes */
+ case 0x6c:
+ case 0x6d:
+ *exitinfo |= IOIO_TYPE_INS;
+ *exitinfo |= IOIO_SEG_ES;
+ *exitinfo |= (ctxt->regs->dx & 0xffff) << 16;
+ break;
+
+ /* OUTS opcodes */
+ case 0x6e:
+ case 0x6f:
+ *exitinfo |= IOIO_TYPE_OUTS;
+ *exitinfo |= IOIO_SEG_DS;
+ *exitinfo |= (ctxt->regs->dx & 0xffff) << 16;
+ break;
+
+ /* IN immediate opcodes */
+ case 0xe4:
+ case 0xe5:
+ *exitinfo |= IOIO_TYPE_IN;
+ *exitinfo |= (u64)insn->immediate.value << 16;
+ break;
+
+ /* OUT immediate opcodes */
+ case 0xe6:
+ case 0xe7:
+ *exitinfo |= IOIO_TYPE_OUT;
+ *exitinfo |= (u64)insn->immediate.value << 16;
+ break;
+
+ /* IN register opcodes */
+ case 0xec:
+ case 0xed:
+ *exitinfo |= IOIO_TYPE_IN;
+ *exitinfo |= (ctxt->regs->dx & 0xffff) << 16;
+ break;
+
+ /* OUT register opcodes */
+ case 0xee:
+ case 0xef:
+ *exitinfo |= IOIO_TYPE_OUT;
+ *exitinfo |= (ctxt->regs->dx & 0xffff) << 16;
+ break;
+
+ default:
+ return ES_DECODE_FAILED;
+ }
+
+ switch (insn->opcode.bytes[0]) {
+ case 0x6c:
+ case 0x6e:
+ case 0xe4:
+ case 0xe6:
+ case 0xec:
+ case 0xee:
+ /* Single byte opcodes */
+ *exitinfo |= IOIO_DATA_8;
+ break;
+ default:
+ /* Length determined by instruction parsing */
+ *exitinfo |= (insn->opnd_bytes == 2) ? IOIO_DATA_16
+ : IOIO_DATA_32;
+ }
+ switch (insn->addr_bytes) {
+ case 2:
+ *exitinfo |= IOIO_ADDR_16;
+ break;
+ case 4:
+ *exitinfo |= IOIO_ADDR_32;
+ break;
+ case 8:
+ *exitinfo |= IOIO_ADDR_64;
+ break;
+ }
+
+ if (insn_has_rep_prefix(insn))
+ *exitinfo |= IOIO_REP;
+
+ return ES_OK;
+}
+
+static enum es_result vc_handle_ioio(struct ghcb *ghcb, struct es_em_ctxt *ctxt)
+{
+ struct pt_regs *regs = ctxt->regs;
+ u64 exit_info_1, exit_info_2;
+ enum es_result ret;
+
+ ret = vc_ioio_exitinfo(ctxt, &exit_info_1);
+ if (ret != ES_OK)
+ return ret;
+
+ if (exit_info_1 & IOIO_TYPE_STR) {
+
+ /* (REP) INS/OUTS */
+
+ bool df = ((regs->flags & X86_EFLAGS_DF) == X86_EFLAGS_DF);
+ unsigned int io_bytes, exit_bytes;
+ unsigned int ghcb_count, op_count;
+ unsigned long es_base;
+ u64 sw_scratch;
+
+ /*
+ * For the string variants with rep prefix the amount of in/out
+ * operations per #VC exception is limited so that the kernel
+ * has a chance to take interrupts and re-schedule while the
+ * instruction is emulated.
+ */
+ io_bytes = (exit_info_1 >> 4) & 0x7;
+ ghcb_count = sizeof(ghcb->shared_buffer) / io_bytes;
+
+ op_count = (exit_info_1 & IOIO_REP) ? regs->cx : 1;
+ exit_info_2 = min(op_count, ghcb_count);
+ exit_bytes = exit_info_2 * io_bytes;
+
+ es_base = insn_get_seg_base(ctxt->regs, INAT_SEG_REG_ES);
+
+ /* Read bytes of OUTS into the shared buffer */
+ if (!(exit_info_1 & IOIO_TYPE_IN)) {
+ ret = vc_insn_string_read(ctxt,
+ (void *)(es_base + regs->si),
+ ghcb->shared_buffer, io_bytes,
+ exit_info_2, df);
+ if (ret)
+ return ret;
+ }
+
+ /*
+ * Issue an VMGEXIT to the HV to consume the bytes from the
+ * shared buffer or to have it write them into the shared buffer
+ * depending on the instruction: OUTS or INS.
+ */
+ sw_scratch = __pa(ghcb) + offsetof(struct ghcb, shared_buffer);
+ ghcb_set_sw_scratch(ghcb, sw_scratch);
+ ret = sev_es_ghcb_hv_call(ghcb, ctxt, SVM_EXIT_IOIO,
+ exit_info_1, exit_info_2);
+ if (ret != ES_OK)
+ return ret;
+
+ /* Read bytes from shared buffer into the guest's destination. */
+ if (exit_info_1 & IOIO_TYPE_IN) {
+ ret = vc_insn_string_write(ctxt,
+ (void *)(es_base + regs->di),
+ ghcb->shared_buffer, io_bytes,
+ exit_info_2, df);
+ if (ret)
+ return ret;
+
+ if (df)
+ regs->di -= exit_bytes;
+ else
+ regs->di += exit_bytes;
+ } else {
+ if (df)
+ regs->si -= exit_bytes;
+ else
+ regs->si += exit_bytes;
+ }
+
+ if (exit_info_1 & IOIO_REP)
+ regs->cx -= exit_info_2;
+
+ ret = regs->cx ? ES_RETRY : ES_OK;
+
+ } else {
+
+ /* IN/OUT into/from rAX */
+
+ int bits = (exit_info_1 & 0x70) >> 1;
+ u64 rax = 0;
+
+ if (!(exit_info_1 & IOIO_TYPE_IN))
+ rax = lower_bits(regs->ax, bits);
+
+ ghcb_set_rax(ghcb, rax);
+
+ ret = sev_es_ghcb_hv_call(ghcb, ctxt, SVM_EXIT_IOIO, exit_info_1, 0);
+ if (ret != ES_OK)
+ return ret;
+
+ if (exit_info_1 & IOIO_TYPE_IN) {
+ if (!ghcb_rax_is_valid(ghcb))
+ return ES_VMM_ERROR;
+ regs->ax = lower_bits(ghcb->save.rax, bits);
+ }
+ }
+
+ return ret;
+}
--
2.28.0
_______________________________________________
Virtualization mailing list
Virtualization@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/virtualization
^ permalink raw reply related [flat|nested] 264+ messages in thread
* [PATCH v7 26/72] x86/fpu: Move xgetbv()/xsetbv() into separate header
2020-09-07 13:15 ` Joerg Roedel
@ 2020-09-07 13:15 ` Joerg Roedel
-1 siblings, 0 replies; 264+ messages in thread
From: Joerg Roedel @ 2020-09-07 13:15 UTC (permalink / raw)
To: x86
Cc: Joerg Roedel, Joerg Roedel, hpa, Andy Lutomirski, Dave Hansen,
Peter Zijlstra, Jiri Slaby, Dan Williams, Tom Lendacky,
Juergen Gross, Kees Cook, David Rientjes, Cfir Cohen,
Erdem Aktas, Masami Hiramatsu, Mike Stunes, Sean Christopherson,
Martin Radev, linux-kernel, kvm, virtualization
From: Joerg Roedel <jroedel@suse.de>
The xgetbv() function is needed in pre-decompression boot code, but
asm/fpu/internal.h can't be included there directly. Doing so opens
the door to include-hell due to various include-magic in
boot/compressed/misc.h.
Avoid that by moving xgetbv()/xsetbv() to a separate header file and
include this instead.
Signed-off-by: Joerg Roedel <jroedel@suse.de>
---
arch/x86/include/asm/fpu/internal.h | 30 +------------------------
arch/x86/include/asm/fpu/xcr.h | 34 +++++++++++++++++++++++++++++
2 files changed, 35 insertions(+), 29 deletions(-)
create mode 100644 arch/x86/include/asm/fpu/xcr.h
diff --git a/arch/x86/include/asm/fpu/internal.h b/arch/x86/include/asm/fpu/internal.h
index 21a8b5259477..ceeba9f63172 100644
--- a/arch/x86/include/asm/fpu/internal.h
+++ b/arch/x86/include/asm/fpu/internal.h
@@ -19,6 +19,7 @@
#include <asm/user.h>
#include <asm/fpu/api.h>
#include <asm/fpu/xstate.h>
+#include <asm/fpu/xcr.h>
#include <asm/cpufeature.h>
#include <asm/trace/fpu.h>
@@ -585,33 +586,4 @@ static inline void switch_fpu_finish(struct fpu *new_fpu)
__write_pkru(pkru_val);
}
-/*
- * MXCSR and XCR definitions:
- */
-
-static inline void ldmxcsr(u32 mxcsr)
-{
- asm volatile("ldmxcsr %0" :: "m" (mxcsr));
-}
-
-extern unsigned int mxcsr_feature_mask;
-
-#define XCR_XFEATURE_ENABLED_MASK 0x00000000
-
-static inline u64 xgetbv(u32 index)
-{
- u32 eax, edx;
-
- asm volatile("xgetbv" : "=a" (eax), "=d" (edx) : "c" (index));
- return eax + ((u64)edx << 32);
-}
-
-static inline void xsetbv(u32 index, u64 value)
-{
- u32 eax = value;
- u32 edx = value >> 32;
-
- asm volatile("xsetbv" :: "a" (eax), "d" (edx), "c" (index));
-}
-
#endif /* _ASM_X86_FPU_INTERNAL_H */
diff --git a/arch/x86/include/asm/fpu/xcr.h b/arch/x86/include/asm/fpu/xcr.h
new file mode 100644
index 000000000000..1c7ab8d95da5
--- /dev/null
+++ b/arch/x86/include/asm/fpu/xcr.h
@@ -0,0 +1,34 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+#ifndef _ASM_X86_FPU_XCR_H
+#define _ASM_X86_FPU_XCR_H
+
+/*
+ * MXCSR and XCR definitions:
+ */
+
+static inline void ldmxcsr(u32 mxcsr)
+{
+ asm volatile("ldmxcsr %0" :: "m" (mxcsr));
+}
+
+extern unsigned int mxcsr_feature_mask;
+
+#define XCR_XFEATURE_ENABLED_MASK 0x00000000
+
+static inline u64 xgetbv(u32 index)
+{
+ u32 eax, edx;
+
+ asm volatile("xgetbv" : "=a" (eax), "=d" (edx) : "c" (index));
+ return eax + ((u64)edx << 32);
+}
+
+static inline void xsetbv(u32 index, u64 value)
+{
+ u32 eax = value;
+ u32 edx = value >> 32;
+
+ asm volatile("xsetbv" :: "a" (eax), "d" (edx), "c" (index));
+}
+
+#endif /* _ASM_X86_FPU_XCR_H */
--
2.28.0
^ permalink raw reply related [flat|nested] 264+ messages in thread
* [PATCH v7 26/72] x86/fpu: Move xgetbv()/xsetbv() into separate header
@ 2020-09-07 13:15 ` Joerg Roedel
0 siblings, 0 replies; 264+ messages in thread
From: Joerg Roedel @ 2020-09-07 13:15 UTC (permalink / raw)
To: x86
Cc: Juergen Gross, Tom Lendacky, Joerg Roedel, Mike Stunes,
Kees Cook, kvm, Peter Zijlstra, Cfir Cohen, Joerg Roedel,
Dave Hansen, linux-kernel, Sean Christopherson, virtualization,
Martin Radev, Masami Hiramatsu, Andy Lutomirski, hpa,
Erdem Aktas, David Rientjes, Dan Williams, Jiri Slaby
From: Joerg Roedel <jroedel@suse.de>
The xgetbv() function is needed in pre-decompression boot code, but
asm/fpu/internal.h can't be included there directly. Doing so opens
the door to include-hell due to various include-magic in
boot/compressed/misc.h.
Avoid that by moving xgetbv()/xsetbv() to a separate header file and
include this instead.
Signed-off-by: Joerg Roedel <jroedel@suse.de>
---
arch/x86/include/asm/fpu/internal.h | 30 +------------------------
arch/x86/include/asm/fpu/xcr.h | 34 +++++++++++++++++++++++++++++
2 files changed, 35 insertions(+), 29 deletions(-)
create mode 100644 arch/x86/include/asm/fpu/xcr.h
diff --git a/arch/x86/include/asm/fpu/internal.h b/arch/x86/include/asm/fpu/internal.h
index 21a8b5259477..ceeba9f63172 100644
--- a/arch/x86/include/asm/fpu/internal.h
+++ b/arch/x86/include/asm/fpu/internal.h
@@ -19,6 +19,7 @@
#include <asm/user.h>
#include <asm/fpu/api.h>
#include <asm/fpu/xstate.h>
+#include <asm/fpu/xcr.h>
#include <asm/cpufeature.h>
#include <asm/trace/fpu.h>
@@ -585,33 +586,4 @@ static inline void switch_fpu_finish(struct fpu *new_fpu)
__write_pkru(pkru_val);
}
-/*
- * MXCSR and XCR definitions:
- */
-
-static inline void ldmxcsr(u32 mxcsr)
-{
- asm volatile("ldmxcsr %0" :: "m" (mxcsr));
-}
-
-extern unsigned int mxcsr_feature_mask;
-
-#define XCR_XFEATURE_ENABLED_MASK 0x00000000
-
-static inline u64 xgetbv(u32 index)
-{
- u32 eax, edx;
-
- asm volatile("xgetbv" : "=a" (eax), "=d" (edx) : "c" (index));
- return eax + ((u64)edx << 32);
-}
-
-static inline void xsetbv(u32 index, u64 value)
-{
- u32 eax = value;
- u32 edx = value >> 32;
-
- asm volatile("xsetbv" :: "a" (eax), "d" (edx), "c" (index));
-}
-
#endif /* _ASM_X86_FPU_INTERNAL_H */
diff --git a/arch/x86/include/asm/fpu/xcr.h b/arch/x86/include/asm/fpu/xcr.h
new file mode 100644
index 000000000000..1c7ab8d95da5
--- /dev/null
+++ b/arch/x86/include/asm/fpu/xcr.h
@@ -0,0 +1,34 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+#ifndef _ASM_X86_FPU_XCR_H
+#define _ASM_X86_FPU_XCR_H
+
+/*
+ * MXCSR and XCR definitions:
+ */
+
+static inline void ldmxcsr(u32 mxcsr)
+{
+ asm volatile("ldmxcsr %0" :: "m" (mxcsr));
+}
+
+extern unsigned int mxcsr_feature_mask;
+
+#define XCR_XFEATURE_ENABLED_MASK 0x00000000
+
+static inline u64 xgetbv(u32 index)
+{
+ u32 eax, edx;
+
+ asm volatile("xgetbv" : "=a" (eax), "=d" (edx) : "c" (index));
+ return eax + ((u64)edx << 32);
+}
+
+static inline void xsetbv(u32 index, u64 value)
+{
+ u32 eax = value;
+ u32 edx = value >> 32;
+
+ asm volatile("xsetbv" :: "a" (eax), "d" (edx), "c" (index));
+}
+
+#endif /* _ASM_X86_FPU_XCR_H */
--
2.28.0
_______________________________________________
Virtualization mailing list
Virtualization@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/virtualization
^ permalink raw reply related [flat|nested] 264+ messages in thread
* [PATCH v7 27/72] x86/sev-es: Add CPUID handling to #VC handler
2020-09-07 13:15 ` Joerg Roedel
@ 2020-09-07 13:15 ` Joerg Roedel
-1 siblings, 0 replies; 264+ messages in thread
From: Joerg Roedel @ 2020-09-07 13:15 UTC (permalink / raw)
To: x86
Cc: Joerg Roedel, Joerg Roedel, Tom Lendacky, hpa, Andy Lutomirski,
Dave Hansen, Peter Zijlstra, Jiri Slaby, Dan Williams,
Juergen Gross, Kees Cook, David Rientjes, Cfir Cohen,
Erdem Aktas, Masami Hiramatsu, Mike Stunes, Sean Christopherson,
Martin Radev, linux-kernel, kvm, virtualization
From: Tom Lendacky <thomas.lendacky@amd.com>
Handle #VC exceptions caused by CPUID instructions. These happen in
early boot code when the KASLR code checks for RDTSC.
Signed-off-by: Tom Lendacky <thomas.lendacky@amd.com>
[ jroedel@suse.de: Adapt to #VC handling framework ]
Co-developed-by: Joerg Roedel <jroedel@suse.de>
Signed-off-by: Joerg Roedel <jroedel@suse.de>
---
arch/x86/boot/compressed/sev-es.c | 4 ++++
arch/x86/kernel/sev-es-shared.c | 35 +++++++++++++++++++++++++++++++
2 files changed, 39 insertions(+)
diff --git a/arch/x86/boot/compressed/sev-es.c b/arch/x86/boot/compressed/sev-es.c
index d7ab212180b6..57c41c368681 100644
--- a/arch/x86/boot/compressed/sev-es.c
+++ b/arch/x86/boot/compressed/sev-es.c
@@ -16,6 +16,7 @@
#include <asm/trapnr.h>
#include <asm/trap_pf.h>
#include <asm/msr-index.h>
+#include <asm/fpu/xcr.h>
#include <asm/ptrace.h>
#include <asm/svm.h>
@@ -183,6 +184,9 @@ void do_boot_stage2_vc(struct pt_regs *regs, unsigned long exit_code)
case SVM_EXIT_IOIO:
result = vc_handle_ioio(boot_ghcb, &ctxt);
break;
+ case SVM_EXIT_CPUID:
+ result = vc_handle_cpuid(boot_ghcb, &ctxt);
+ break;
default:
result = ES_UNSUPPORTED;
break;
diff --git a/arch/x86/kernel/sev-es-shared.c b/arch/x86/kernel/sev-es-shared.c
index bae7cf28455b..a6b41910b8ab 100644
--- a/arch/x86/kernel/sev-es-shared.c
+++ b/arch/x86/kernel/sev-es-shared.c
@@ -432,3 +432,38 @@ static enum es_result vc_handle_ioio(struct ghcb *ghcb, struct es_em_ctxt *ctxt)
return ret;
}
+
+static enum es_result vc_handle_cpuid(struct ghcb *ghcb,
+ struct es_em_ctxt *ctxt)
+{
+ struct pt_regs *regs = ctxt->regs;
+ u32 cr4 = native_read_cr4();
+ enum es_result ret;
+
+ ghcb_set_rax(ghcb, regs->ax);
+ ghcb_set_rcx(ghcb, regs->cx);
+
+ if (cr4 & X86_CR4_OSXSAVE)
+ /* Safe to read xcr0 */
+ ghcb_set_xcr0(ghcb, xgetbv(XCR_XFEATURE_ENABLED_MASK));
+ else
+ /* xgetbv will cause #GP - use reset value for xcr0 */
+ ghcb_set_xcr0(ghcb, 1);
+
+ ret = sev_es_ghcb_hv_call(ghcb, ctxt, SVM_EXIT_CPUID, 0, 0);
+ if (ret != ES_OK)
+ return ret;
+
+ if (!(ghcb_rax_is_valid(ghcb) &&
+ ghcb_rbx_is_valid(ghcb) &&
+ ghcb_rcx_is_valid(ghcb) &&
+ ghcb_rdx_is_valid(ghcb)))
+ return ES_VMM_ERROR;
+
+ regs->ax = ghcb->save.rax;
+ regs->bx = ghcb->save.rbx;
+ regs->cx = ghcb->save.rcx;
+ regs->dx = ghcb->save.rdx;
+
+ return ES_OK;
+}
--
2.28.0
^ permalink raw reply related [flat|nested] 264+ messages in thread
* [PATCH v7 27/72] x86/sev-es: Add CPUID handling to #VC handler
@ 2020-09-07 13:15 ` Joerg Roedel
0 siblings, 0 replies; 264+ messages in thread
From: Joerg Roedel @ 2020-09-07 13:15 UTC (permalink / raw)
To: x86
Cc: Juergen Gross, Tom Lendacky, Joerg Roedel, Mike Stunes,
Kees Cook, kvm, Peter Zijlstra, Cfir Cohen, Joerg Roedel,
Dave Hansen, linux-kernel, Sean Christopherson, virtualization,
Martin Radev, Masami Hiramatsu, Andy Lutomirski, hpa,
Erdem Aktas, David Rientjes, Dan Williams, Jiri Slaby
From: Tom Lendacky <thomas.lendacky@amd.com>
Handle #VC exceptions caused by CPUID instructions. These happen in
early boot code when the KASLR code checks for RDTSC.
Signed-off-by: Tom Lendacky <thomas.lendacky@amd.com>
[ jroedel@suse.de: Adapt to #VC handling framework ]
Co-developed-by: Joerg Roedel <jroedel@suse.de>
Signed-off-by: Joerg Roedel <jroedel@suse.de>
---
arch/x86/boot/compressed/sev-es.c | 4 ++++
arch/x86/kernel/sev-es-shared.c | 35 +++++++++++++++++++++++++++++++
2 files changed, 39 insertions(+)
diff --git a/arch/x86/boot/compressed/sev-es.c b/arch/x86/boot/compressed/sev-es.c
index d7ab212180b6..57c41c368681 100644
--- a/arch/x86/boot/compressed/sev-es.c
+++ b/arch/x86/boot/compressed/sev-es.c
@@ -16,6 +16,7 @@
#include <asm/trapnr.h>
#include <asm/trap_pf.h>
#include <asm/msr-index.h>
+#include <asm/fpu/xcr.h>
#include <asm/ptrace.h>
#include <asm/svm.h>
@@ -183,6 +184,9 @@ void do_boot_stage2_vc(struct pt_regs *regs, unsigned long exit_code)
case SVM_EXIT_IOIO:
result = vc_handle_ioio(boot_ghcb, &ctxt);
break;
+ case SVM_EXIT_CPUID:
+ result = vc_handle_cpuid(boot_ghcb, &ctxt);
+ break;
default:
result = ES_UNSUPPORTED;
break;
diff --git a/arch/x86/kernel/sev-es-shared.c b/arch/x86/kernel/sev-es-shared.c
index bae7cf28455b..a6b41910b8ab 100644
--- a/arch/x86/kernel/sev-es-shared.c
+++ b/arch/x86/kernel/sev-es-shared.c
@@ -432,3 +432,38 @@ static enum es_result vc_handle_ioio(struct ghcb *ghcb, struct es_em_ctxt *ctxt)
return ret;
}
+
+static enum es_result vc_handle_cpuid(struct ghcb *ghcb,
+ struct es_em_ctxt *ctxt)
+{
+ struct pt_regs *regs = ctxt->regs;
+ u32 cr4 = native_read_cr4();
+ enum es_result ret;
+
+ ghcb_set_rax(ghcb, regs->ax);
+ ghcb_set_rcx(ghcb, regs->cx);
+
+ if (cr4 & X86_CR4_OSXSAVE)
+ /* Safe to read xcr0 */
+ ghcb_set_xcr0(ghcb, xgetbv(XCR_XFEATURE_ENABLED_MASK));
+ else
+ /* xgetbv will cause #GP - use reset value for xcr0 */
+ ghcb_set_xcr0(ghcb, 1);
+
+ ret = sev_es_ghcb_hv_call(ghcb, ctxt, SVM_EXIT_CPUID, 0, 0);
+ if (ret != ES_OK)
+ return ret;
+
+ if (!(ghcb_rax_is_valid(ghcb) &&
+ ghcb_rbx_is_valid(ghcb) &&
+ ghcb_rcx_is_valid(ghcb) &&
+ ghcb_rdx_is_valid(ghcb)))
+ return ES_VMM_ERROR;
+
+ regs->ax = ghcb->save.rax;
+ regs->bx = ghcb->save.rbx;
+ regs->cx = ghcb->save.rcx;
+ regs->dx = ghcb->save.rdx;
+
+ return ES_OK;
+}
--
2.28.0
_______________________________________________
Virtualization mailing list
Virtualization@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/virtualization
^ permalink raw reply related [flat|nested] 264+ messages in thread
* [PATCH v7 28/72] x86/idt: Split idt_data setup out of set_intr_gate()
2020-09-07 13:15 ` Joerg Roedel
@ 2020-09-07 13:15 ` Joerg Roedel
-1 siblings, 0 replies; 264+ messages in thread
From: Joerg Roedel @ 2020-09-07 13:15 UTC (permalink / raw)
To: x86
Cc: Joerg Roedel, Joerg Roedel, Kees Cook, hpa, Andy Lutomirski,
Dave Hansen, Peter Zijlstra, Jiri Slaby, Dan Williams,
Tom Lendacky, Juergen Gross, David Rientjes, Cfir Cohen,
Erdem Aktas, Masami Hiramatsu, Mike Stunes, Sean Christopherson,
Martin Radev, linux-kernel, kvm, virtualization
From: Joerg Roedel <jroedel@suse.de>
The code to setup idt_data is needed for early exception handling, but
set_intr_gate() can't be used that early because it has pv-ops in its
code path, which don't work that early.
Split out the idt_data initialization part from set_intr_gate() so
that it can be used separately.
Signed-off-by: Joerg Roedel <jroedel@suse.de>
Reviewed-by: Kees Cook <keescook@chromium.org>
---
arch/x86/kernel/idt.c | 22 ++++++++++++++--------
1 file changed, 14 insertions(+), 8 deletions(-)
diff --git a/arch/x86/kernel/idt.c b/arch/x86/kernel/idt.c
index 7ecf9babf0cb..53946c104fa0 100644
--- a/arch/x86/kernel/idt.c
+++ b/arch/x86/kernel/idt.c
@@ -205,18 +205,24 @@ idt_setup_from_table(gate_desc *idt, const struct idt_data *t, int size, bool sy
}
}
+static void init_idt_data(struct idt_data *data, unsigned int n,
+ const void *addr)
+{
+ BUG_ON(n > 0xFF);
+
+ memset(data, 0, sizeof(*data));
+ data->vector = n;
+ data->addr = addr;
+ data->segment = __KERNEL_CS;
+ data->bits.type = GATE_INTERRUPT;
+ data->bits.p = 1;
+}
+
static __init void set_intr_gate(unsigned int n, const void *addr)
{
struct idt_data data;
- BUG_ON(n > 0xFF);
-
- memset(&data, 0, sizeof(data));
- data.vector = n;
- data.addr = addr;
- data.segment = __KERNEL_CS;
- data.bits.type = GATE_INTERRUPT;
- data.bits.p = 1;
+ init_idt_data(&data, n, addr);
idt_setup_from_table(idt_table, &data, 1, false);
}
--
2.28.0
^ permalink raw reply related [flat|nested] 264+ messages in thread
* [PATCH v7 28/72] x86/idt: Split idt_data setup out of set_intr_gate()
@ 2020-09-07 13:15 ` Joerg Roedel
0 siblings, 0 replies; 264+ messages in thread
From: Joerg Roedel @ 2020-09-07 13:15 UTC (permalink / raw)
To: x86
Cc: Juergen Gross, Tom Lendacky, Joerg Roedel, Mike Stunes,
Kees Cook, kvm, Peter Zijlstra, Cfir Cohen, Joerg Roedel,
Dave Hansen, linux-kernel, Sean Christopherson, virtualization,
Martin Radev, Masami Hiramatsu, Andy Lutomirski, hpa,
Erdem Aktas, David Rientjes, Dan Williams, Jiri Slaby
From: Joerg Roedel <jroedel@suse.de>
The code to setup idt_data is needed for early exception handling, but
set_intr_gate() can't be used that early because it has pv-ops in its
code path, which don't work that early.
Split out the idt_data initialization part from set_intr_gate() so
that it can be used separately.
Signed-off-by: Joerg Roedel <jroedel@suse.de>
Reviewed-by: Kees Cook <keescook@chromium.org>
---
arch/x86/kernel/idt.c | 22 ++++++++++++++--------
1 file changed, 14 insertions(+), 8 deletions(-)
diff --git a/arch/x86/kernel/idt.c b/arch/x86/kernel/idt.c
index 7ecf9babf0cb..53946c104fa0 100644
--- a/arch/x86/kernel/idt.c
+++ b/arch/x86/kernel/idt.c
@@ -205,18 +205,24 @@ idt_setup_from_table(gate_desc *idt, const struct idt_data *t, int size, bool sy
}
}
+static void init_idt_data(struct idt_data *data, unsigned int n,
+ const void *addr)
+{
+ BUG_ON(n > 0xFF);
+
+ memset(data, 0, sizeof(*data));
+ data->vector = n;
+ data->addr = addr;
+ data->segment = __KERNEL_CS;
+ data->bits.type = GATE_INTERRUPT;
+ data->bits.p = 1;
+}
+
static __init void set_intr_gate(unsigned int n, const void *addr)
{
struct idt_data data;
- BUG_ON(n > 0xFF);
-
- memset(&data, 0, sizeof(data));
- data.vector = n;
- data.addr = addr;
- data.segment = __KERNEL_CS;
- data.bits.type = GATE_INTERRUPT;
- data.bits.p = 1;
+ init_idt_data(&data, n, addr);
idt_setup_from_table(idt_table, &data, 1, false);
}
--
2.28.0
_______________________________________________
Virtualization mailing list
Virtualization@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/virtualization
^ permalink raw reply related [flat|nested] 264+ messages in thread
* [PATCH v7 29/72] x86/head/64: Install startup GDT
2020-09-07 13:15 ` Joerg Roedel
@ 2020-09-07 13:15 ` Joerg Roedel
-1 siblings, 0 replies; 264+ messages in thread
From: Joerg Roedel @ 2020-09-07 13:15 UTC (permalink / raw)
To: x86
Cc: Joerg Roedel, Joerg Roedel, hpa, Andy Lutomirski, Dave Hansen,
Peter Zijlstra, Jiri Slaby, Dan Williams, Tom Lendacky,
Juergen Gross, Kees Cook, David Rientjes, Cfir Cohen,
Erdem Aktas, Masami Hiramatsu, Mike Stunes, Sean Christopherson,
Martin Radev, linux-kernel, kvm, virtualization
From: Joerg Roedel <jroedel@suse.de>
Handling exceptions during boot requires a working GDT. The kernel GDT
can't be used on the direct mapping, so load a startup GDT and setup
segments.
Signed-off-by: Joerg Roedel <jroedel@suse.de>
---
arch/x86/include/asm/setup.h | 1 +
arch/x86/kernel/head64.c | 33 +++++++++++++++++++++++++++++++++
arch/x86/kernel/head_64.S | 14 ++++++++++++++
3 files changed, 48 insertions(+)
diff --git a/arch/x86/include/asm/setup.h b/arch/x86/include/asm/setup.h
index 84b645cc8bc9..5c2fd05bd52c 100644
--- a/arch/x86/include/asm/setup.h
+++ b/arch/x86/include/asm/setup.h
@@ -48,6 +48,7 @@ extern void reserve_standard_io_resources(void);
extern void i386_reserve_resources(void);
extern unsigned long __startup_64(unsigned long physaddr, struct boot_params *bp);
extern unsigned long __startup_secondary_64(void);
+extern void startup_64_setup_env(unsigned long physbase);
extern int early_make_pgtable(unsigned long address);
#ifdef CONFIG_X86_INTEL_MID
diff --git a/arch/x86/kernel/head64.c b/arch/x86/kernel/head64.c
index cbb71c1b574f..8c82be44be94 100644
--- a/arch/x86/kernel/head64.c
+++ b/arch/x86/kernel/head64.c
@@ -61,6 +61,24 @@ unsigned long vmemmap_base __ro_after_init = __VMEMMAP_BASE_L4;
EXPORT_SYMBOL(vmemmap_base);
#endif
+/*
+ * GDT used on the boot CPU before switching to virtual addresses.
+ */
+static struct desc_struct startup_gdt[GDT_ENTRIES] = {
+ [GDT_ENTRY_KERNEL32_CS] = GDT_ENTRY_INIT(0xc09b, 0, 0xfffff),
+ [GDT_ENTRY_KERNEL_CS] = GDT_ENTRY_INIT(0xa09b, 0, 0xfffff),
+ [GDT_ENTRY_KERNEL_DS] = GDT_ENTRY_INIT(0xc093, 0, 0xfffff),
+};
+
+/*
+ * Address needs to be set at runtime because it references the startup_gdt
+ * while the kernel still uses a direct mapping.
+ */
+static struct desc_ptr startup_gdt_descr = {
+ .size = sizeof(startup_gdt),
+ .address = 0,
+};
+
#define __head __section(.head.text)
static void __head *fixup_pointer(void *ptr, unsigned long physaddr)
@@ -489,3 +507,18 @@ void __init x86_64_start_reservations(char *real_mode_data)
start_kernel();
}
+
+/*
+ * Setup boot CPU state needed before kernel switches to virtual addresses.
+ */
+void __head startup_64_setup_env(unsigned long physbase)
+{
+ /* Load GDT */
+ startup_gdt_descr.address = (unsigned long)fixup_pointer(startup_gdt, physbase);
+ native_load_gdt(&startup_gdt_descr);
+
+ /* New GDT is live - reload data segment registers */
+ asm volatile("movl %%eax, %%ds\n"
+ "movl %%eax, %%ss\n"
+ "movl %%eax, %%es\n" : : "a"(__KERNEL_DS) : "memory");
+}
diff --git a/arch/x86/kernel/head_64.S b/arch/x86/kernel/head_64.S
index 16da4ac01597..2b2e91627221 100644
--- a/arch/x86/kernel/head_64.S
+++ b/arch/x86/kernel/head_64.S
@@ -73,6 +73,20 @@ SYM_CODE_START_NOALIGN(startup_64)
/* Set up the stack for verify_cpu(), similar to initial_stack below */
leaq (__end_init_task - SIZEOF_PTREGS)(%rip), %rsp
+ leaq _text(%rip), %rdi
+ pushq %rsi
+ call startup_64_setup_env
+ popq %rsi
+
+ /* Now switch to __KERNEL_CS so IRET works reliably */
+ pushq $__KERNEL_CS
+ leaq .Lon_kernel_cs(%rip), %rax
+ pushq %rax
+ lretq
+
+.Lon_kernel_cs:
+ UNWIND_HINT_EMPTY
+
/* Sanitize CPU configuration */
call verify_cpu
--
2.28.0
^ permalink raw reply related [flat|nested] 264+ messages in thread
* [PATCH v7 29/72] x86/head/64: Install startup GDT
@ 2020-09-07 13:15 ` Joerg Roedel
0 siblings, 0 replies; 264+ messages in thread
From: Joerg Roedel @ 2020-09-07 13:15 UTC (permalink / raw)
To: x86
Cc: Juergen Gross, Tom Lendacky, Joerg Roedel, Mike Stunes,
Kees Cook, kvm, Peter Zijlstra, Cfir Cohen, Joerg Roedel,
Dave Hansen, linux-kernel, Sean Christopherson, virtualization,
Martin Radev, Masami Hiramatsu, Andy Lutomirski, hpa,
Erdem Aktas, David Rientjes, Dan Williams, Jiri Slaby
From: Joerg Roedel <jroedel@suse.de>
Handling exceptions during boot requires a working GDT. The kernel GDT
can't be used on the direct mapping, so load a startup GDT and setup
segments.
Signed-off-by: Joerg Roedel <jroedel@suse.de>
---
arch/x86/include/asm/setup.h | 1 +
arch/x86/kernel/head64.c | 33 +++++++++++++++++++++++++++++++++
arch/x86/kernel/head_64.S | 14 ++++++++++++++
3 files changed, 48 insertions(+)
diff --git a/arch/x86/include/asm/setup.h b/arch/x86/include/asm/setup.h
index 84b645cc8bc9..5c2fd05bd52c 100644
--- a/arch/x86/include/asm/setup.h
+++ b/arch/x86/include/asm/setup.h
@@ -48,6 +48,7 @@ extern void reserve_standard_io_resources(void);
extern void i386_reserve_resources(void);
extern unsigned long __startup_64(unsigned long physaddr, struct boot_params *bp);
extern unsigned long __startup_secondary_64(void);
+extern void startup_64_setup_env(unsigned long physbase);
extern int early_make_pgtable(unsigned long address);
#ifdef CONFIG_X86_INTEL_MID
diff --git a/arch/x86/kernel/head64.c b/arch/x86/kernel/head64.c
index cbb71c1b574f..8c82be44be94 100644
--- a/arch/x86/kernel/head64.c
+++ b/arch/x86/kernel/head64.c
@@ -61,6 +61,24 @@ unsigned long vmemmap_base __ro_after_init = __VMEMMAP_BASE_L4;
EXPORT_SYMBOL(vmemmap_base);
#endif
+/*
+ * GDT used on the boot CPU before switching to virtual addresses.
+ */
+static struct desc_struct startup_gdt[GDT_ENTRIES] = {
+ [GDT_ENTRY_KERNEL32_CS] = GDT_ENTRY_INIT(0xc09b, 0, 0xfffff),
+ [GDT_ENTRY_KERNEL_CS] = GDT_ENTRY_INIT(0xa09b, 0, 0xfffff),
+ [GDT_ENTRY_KERNEL_DS] = GDT_ENTRY_INIT(0xc093, 0, 0xfffff),
+};
+
+/*
+ * Address needs to be set at runtime because it references the startup_gdt
+ * while the kernel still uses a direct mapping.
+ */
+static struct desc_ptr startup_gdt_descr = {
+ .size = sizeof(startup_gdt),
+ .address = 0,
+};
+
#define __head __section(.head.text)
static void __head *fixup_pointer(void *ptr, unsigned long physaddr)
@@ -489,3 +507,18 @@ void __init x86_64_start_reservations(char *real_mode_data)
start_kernel();
}
+
+/*
+ * Setup boot CPU state needed before kernel switches to virtual addresses.
+ */
+void __head startup_64_setup_env(unsigned long physbase)
+{
+ /* Load GDT */
+ startup_gdt_descr.address = (unsigned long)fixup_pointer(startup_gdt, physbase);
+ native_load_gdt(&startup_gdt_descr);
+
+ /* New GDT is live - reload data segment registers */
+ asm volatile("movl %%eax, %%ds\n"
+ "movl %%eax, %%ss\n"
+ "movl %%eax, %%es\n" : : "a"(__KERNEL_DS) : "memory");
+}
diff --git a/arch/x86/kernel/head_64.S b/arch/x86/kernel/head_64.S
index 16da4ac01597..2b2e91627221 100644
--- a/arch/x86/kernel/head_64.S
+++ b/arch/x86/kernel/head_64.S
@@ -73,6 +73,20 @@ SYM_CODE_START_NOALIGN(startup_64)
/* Set up the stack for verify_cpu(), similar to initial_stack below */
leaq (__end_init_task - SIZEOF_PTREGS)(%rip), %rsp
+ leaq _text(%rip), %rdi
+ pushq %rsi
+ call startup_64_setup_env
+ popq %rsi
+
+ /* Now switch to __KERNEL_CS so IRET works reliably */
+ pushq $__KERNEL_CS
+ leaq .Lon_kernel_cs(%rip), %rax
+ pushq %rax
+ lretq
+
+.Lon_kernel_cs:
+ UNWIND_HINT_EMPTY
+
/* Sanitize CPU configuration */
call verify_cpu
--
2.28.0
_______________________________________________
Virtualization mailing list
Virtualization@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/virtualization
^ permalink raw reply related [flat|nested] 264+ messages in thread
* [PATCH v7 30/72] x86/head/64: Load GDT after switch to virtual addresses
2020-09-07 13:15 ` Joerg Roedel
@ 2020-09-07 13:15 ` Joerg Roedel
-1 siblings, 0 replies; 264+ messages in thread
From: Joerg Roedel @ 2020-09-07 13:15 UTC (permalink / raw)
To: x86
Cc: Joerg Roedel, Joerg Roedel, Kees Cook, hpa, Andy Lutomirski,
Dave Hansen, Peter Zijlstra, Jiri Slaby, Dan Williams,
Tom Lendacky, Juergen Gross, David Rientjes, Cfir Cohen,
Erdem Aktas, Masami Hiramatsu, Mike Stunes, Sean Christopherson,
Martin Radev, linux-kernel, kvm, virtualization
From: Joerg Roedel <jroedel@suse.de>
Load the GDT right after switching to virtual addresses to make sure
there is a defined GDT for exception handling.
Signed-off-by: Joerg Roedel <jroedel@suse.de>
Reviewed-by: Kees Cook <keescook@chromium.org>
---
arch/x86/kernel/head_64.S | 16 ++++++++--------
1 file changed, 8 insertions(+), 8 deletions(-)
diff --git a/arch/x86/kernel/head_64.S b/arch/x86/kernel/head_64.S
index 2b2e91627221..03b03f266dc1 100644
--- a/arch/x86/kernel/head_64.S
+++ b/arch/x86/kernel/head_64.S
@@ -158,6 +158,14 @@ SYM_CODE_START(secondary_startup_64)
1:
UNWIND_HINT_EMPTY
+ /*
+ * We must switch to a new descriptor in kernel space for the GDT
+ * because soon the kernel won't have access anymore to the userspace
+ * addresses where we're currently running on. We have to do that here
+ * because in 32bit we couldn't load a 64bit linear address.
+ */
+ lgdt early_gdt_descr(%rip)
+
/* Check if nx is implemented */
movl $0x80000001, %eax
cpuid
@@ -185,14 +193,6 @@ SYM_CODE_START(secondary_startup_64)
pushq $0
popfq
- /*
- * We must switch to a new descriptor in kernel space for the GDT
- * because soon the kernel won't have access anymore to the userspace
- * addresses where we're currently running on. We have to do that here
- * because in 32bit we couldn't load a 64bit linear address.
- */
- lgdt early_gdt_descr(%rip)
-
/* set up data segments */
xorl %eax,%eax
movl %eax,%ds
--
2.28.0
^ permalink raw reply related [flat|nested] 264+ messages in thread
* [PATCH v7 30/72] x86/head/64: Load GDT after switch to virtual addresses
@ 2020-09-07 13:15 ` Joerg Roedel
0 siblings, 0 replies; 264+ messages in thread
From: Joerg Roedel @ 2020-09-07 13:15 UTC (permalink / raw)
To: x86
Cc: Juergen Gross, Tom Lendacky, Joerg Roedel, Mike Stunes,
Kees Cook, kvm, Peter Zijlstra, Cfir Cohen, Joerg Roedel,
Dave Hansen, linux-kernel, Sean Christopherson, virtualization,
Martin Radev, Masami Hiramatsu, Andy Lutomirski, hpa,
Erdem Aktas, David Rientjes, Dan Williams, Jiri Slaby
From: Joerg Roedel <jroedel@suse.de>
Load the GDT right after switching to virtual addresses to make sure
there is a defined GDT for exception handling.
Signed-off-by: Joerg Roedel <jroedel@suse.de>
Reviewed-by: Kees Cook <keescook@chromium.org>
---
arch/x86/kernel/head_64.S | 16 ++++++++--------
1 file changed, 8 insertions(+), 8 deletions(-)
diff --git a/arch/x86/kernel/head_64.S b/arch/x86/kernel/head_64.S
index 2b2e91627221..03b03f266dc1 100644
--- a/arch/x86/kernel/head_64.S
+++ b/arch/x86/kernel/head_64.S
@@ -158,6 +158,14 @@ SYM_CODE_START(secondary_startup_64)
1:
UNWIND_HINT_EMPTY
+ /*
+ * We must switch to a new descriptor in kernel space for the GDT
+ * because soon the kernel won't have access anymore to the userspace
+ * addresses where we're currently running on. We have to do that here
+ * because in 32bit we couldn't load a 64bit linear address.
+ */
+ lgdt early_gdt_descr(%rip)
+
/* Check if nx is implemented */
movl $0x80000001, %eax
cpuid
@@ -185,14 +193,6 @@ SYM_CODE_START(secondary_startup_64)
pushq $0
popfq
- /*
- * We must switch to a new descriptor in kernel space for the GDT
- * because soon the kernel won't have access anymore to the userspace
- * addresses where we're currently running on. We have to do that here
- * because in 32bit we couldn't load a 64bit linear address.
- */
- lgdt early_gdt_descr(%rip)
-
/* set up data segments */
xorl %eax,%eax
movl %eax,%ds
--
2.28.0
_______________________________________________
Virtualization mailing list
Virtualization@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/virtualization
^ permalink raw reply related [flat|nested] 264+ messages in thread
* [PATCH v7 31/72] x86/head/64: Load segment registers earlier
2020-09-07 13:15 ` Joerg Roedel
@ 2020-09-07 13:15 ` Joerg Roedel
-1 siblings, 0 replies; 264+ messages in thread
From: Joerg Roedel @ 2020-09-07 13:15 UTC (permalink / raw)
To: x86
Cc: Joerg Roedel, Joerg Roedel, Kees Cook, hpa, Andy Lutomirski,
Dave Hansen, Peter Zijlstra, Jiri Slaby, Dan Williams,
Tom Lendacky, Juergen Gross, David Rientjes, Cfir Cohen,
Erdem Aktas, Masami Hiramatsu, Mike Stunes, Sean Christopherson,
Martin Radev, linux-kernel, kvm, virtualization
From: Joerg Roedel <jroedel@suse.de>
Make sure segments are properly set up before setting up an IDT and
doing anything that might cause a #VC exception. This is later needed
for early exception handling.
Signed-off-by: Joerg Roedel <jroedel@suse.de>
Reviewed-by: Kees Cook <keescook@chromium.org>
---
arch/x86/kernel/head_64.S | 52 +++++++++++++++++++--------------------
1 file changed, 26 insertions(+), 26 deletions(-)
diff --git a/arch/x86/kernel/head_64.S b/arch/x86/kernel/head_64.S
index 03b03f266dc1..f402087a02ac 100644
--- a/arch/x86/kernel/head_64.S
+++ b/arch/x86/kernel/head_64.S
@@ -166,6 +166,32 @@ SYM_CODE_START(secondary_startup_64)
*/
lgdt early_gdt_descr(%rip)
+ /* set up data segments */
+ xorl %eax,%eax
+ movl %eax,%ds
+ movl %eax,%ss
+ movl %eax,%es
+
+ /*
+ * We don't really need to load %fs or %gs, but load them anyway
+ * to kill any stale realmode selectors. This allows execution
+ * under VT hardware.
+ */
+ movl %eax,%fs
+ movl %eax,%gs
+
+ /* Set up %gs.
+ *
+ * The base of %gs always points to fixed_percpu_data. If the
+ * stack protector canary is enabled, it is located at %gs:40.
+ * Note that, on SMP, the boot cpu uses init data section until
+ * the per cpu areas are set up.
+ */
+ movl $MSR_GS_BASE,%ecx
+ movl initial_gs(%rip),%eax
+ movl initial_gs+4(%rip),%edx
+ wrmsr
+
/* Check if nx is implemented */
movl $0x80000001, %eax
cpuid
@@ -193,32 +219,6 @@ SYM_CODE_START(secondary_startup_64)
pushq $0
popfq
- /* set up data segments */
- xorl %eax,%eax
- movl %eax,%ds
- movl %eax,%ss
- movl %eax,%es
-
- /*
- * We don't really need to load %fs or %gs, but load them anyway
- * to kill any stale realmode selectors. This allows execution
- * under VT hardware.
- */
- movl %eax,%fs
- movl %eax,%gs
-
- /* Set up %gs.
- *
- * The base of %gs always points to fixed_percpu_data. If the
- * stack protector canary is enabled, it is located at %gs:40.
- * Note that, on SMP, the boot cpu uses init data section until
- * the per cpu areas are set up.
- */
- movl $MSR_GS_BASE,%ecx
- movl initial_gs(%rip),%eax
- movl initial_gs+4(%rip),%edx
- wrmsr
-
/* rsi is pointer to real mode structure with interesting info.
pass it to C */
movq %rsi, %rdi
--
2.28.0
^ permalink raw reply related [flat|nested] 264+ messages in thread
* [PATCH v7 31/72] x86/head/64: Load segment registers earlier
@ 2020-09-07 13:15 ` Joerg Roedel
0 siblings, 0 replies; 264+ messages in thread
From: Joerg Roedel @ 2020-09-07 13:15 UTC (permalink / raw)
To: x86
Cc: Juergen Gross, Tom Lendacky, Joerg Roedel, Mike Stunes,
Kees Cook, kvm, Peter Zijlstra, Cfir Cohen, Joerg Roedel,
Dave Hansen, linux-kernel, Sean Christopherson, virtualization,
Martin Radev, Masami Hiramatsu, Andy Lutomirski, hpa,
Erdem Aktas, David Rientjes, Dan Williams, Jiri Slaby
From: Joerg Roedel <jroedel@suse.de>
Make sure segments are properly set up before setting up an IDT and
doing anything that might cause a #VC exception. This is later needed
for early exception handling.
Signed-off-by: Joerg Roedel <jroedel@suse.de>
Reviewed-by: Kees Cook <keescook@chromium.org>
---
arch/x86/kernel/head_64.S | 52 +++++++++++++++++++--------------------
1 file changed, 26 insertions(+), 26 deletions(-)
diff --git a/arch/x86/kernel/head_64.S b/arch/x86/kernel/head_64.S
index 03b03f266dc1..f402087a02ac 100644
--- a/arch/x86/kernel/head_64.S
+++ b/arch/x86/kernel/head_64.S
@@ -166,6 +166,32 @@ SYM_CODE_START(secondary_startup_64)
*/
lgdt early_gdt_descr(%rip)
+ /* set up data segments */
+ xorl %eax,%eax
+ movl %eax,%ds
+ movl %eax,%ss
+ movl %eax,%es
+
+ /*
+ * We don't really need to load %fs or %gs, but load them anyway
+ * to kill any stale realmode selectors. This allows execution
+ * under VT hardware.
+ */
+ movl %eax,%fs
+ movl %eax,%gs
+
+ /* Set up %gs.
+ *
+ * The base of %gs always points to fixed_percpu_data. If the
+ * stack protector canary is enabled, it is located at %gs:40.
+ * Note that, on SMP, the boot cpu uses init data section until
+ * the per cpu areas are set up.
+ */
+ movl $MSR_GS_BASE,%ecx
+ movl initial_gs(%rip),%eax
+ movl initial_gs+4(%rip),%edx
+ wrmsr
+
/* Check if nx is implemented */
movl $0x80000001, %eax
cpuid
@@ -193,32 +219,6 @@ SYM_CODE_START(secondary_startup_64)
pushq $0
popfq
- /* set up data segments */
- xorl %eax,%eax
- movl %eax,%ds
- movl %eax,%ss
- movl %eax,%es
-
- /*
- * We don't really need to load %fs or %gs, but load them anyway
- * to kill any stale realmode selectors. This allows execution
- * under VT hardware.
- */
- movl %eax,%fs
- movl %eax,%gs
-
- /* Set up %gs.
- *
- * The base of %gs always points to fixed_percpu_data. If the
- * stack protector canary is enabled, it is located at %gs:40.
- * Note that, on SMP, the boot cpu uses init data section until
- * the per cpu areas are set up.
- */
- movl $MSR_GS_BASE,%ecx
- movl initial_gs(%rip),%eax
- movl initial_gs+4(%rip),%edx
- wrmsr
-
/* rsi is pointer to real mode structure with interesting info.
pass it to C */
movq %rsi, %rdi
--
2.28.0
_______________________________________________
Virtualization mailing list
Virtualization@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/virtualization
^ permalink raw reply related [flat|nested] 264+ messages in thread
* [PATCH v7 32/72] x86/head/64: Switch to initial stack earlier
2020-09-07 13:15 ` Joerg Roedel
@ 2020-09-07 13:15 ` Joerg Roedel
-1 siblings, 0 replies; 264+ messages in thread
From: Joerg Roedel @ 2020-09-07 13:15 UTC (permalink / raw)
To: x86
Cc: Joerg Roedel, Joerg Roedel, Kees Cook, hpa, Andy Lutomirski,
Dave Hansen, Peter Zijlstra, Jiri Slaby, Dan Williams,
Tom Lendacky, Juergen Gross, David Rientjes, Cfir Cohen,
Erdem Aktas, Masami Hiramatsu, Mike Stunes, Sean Christopherson,
Martin Radev, linux-kernel, kvm, virtualization
From: Joerg Roedel <jroedel@suse.de>
Make sure there is a stack once the kernel runs from virual addresses.
At this stage any secondary CPU which boots will have lost its stack
because the kernel switched to a new page-table which does not map the
real-mode stack anymore.
This is needed for handling early #VC exceptions caused by instructions
like CPUID.
Signed-off-by: Joerg Roedel <jroedel@suse.de>
Reviewed-by: Kees Cook <keescook@chromium.org>
---
arch/x86/kernel/head_64.S | 9 ++++++---
1 file changed, 6 insertions(+), 3 deletions(-)
diff --git a/arch/x86/kernel/head_64.S b/arch/x86/kernel/head_64.S
index f402087a02ac..83050c9e54d9 100644
--- a/arch/x86/kernel/head_64.S
+++ b/arch/x86/kernel/head_64.S
@@ -192,6 +192,12 @@ SYM_CODE_START(secondary_startup_64)
movl initial_gs+4(%rip),%edx
wrmsr
+ /*
+ * Setup a boot time stack - Any secondary CPU will have lost its stack
+ * by now because the cr3-switch above unmaps the real-mode stack
+ */
+ movq initial_stack(%rip), %rsp
+
/* Check if nx is implemented */
movl $0x80000001, %eax
cpuid
@@ -212,9 +218,6 @@ SYM_CODE_START(secondary_startup_64)
/* Make changes effective */
movq %rax, %cr0
- /* Setup a boot time stack */
- movq initial_stack(%rip), %rsp
-
/* zero EFLAGS after setting rsp */
pushq $0
popfq
--
2.28.0
^ permalink raw reply related [flat|nested] 264+ messages in thread
* [PATCH v7 32/72] x86/head/64: Switch to initial stack earlier
@ 2020-09-07 13:15 ` Joerg Roedel
0 siblings, 0 replies; 264+ messages in thread
From: Joerg Roedel @ 2020-09-07 13:15 UTC (permalink / raw)
To: x86
Cc: Juergen Gross, Tom Lendacky, Joerg Roedel, Mike Stunes,
Kees Cook, kvm, Peter Zijlstra, Cfir Cohen, Joerg Roedel,
Dave Hansen, linux-kernel, Sean Christopherson, virtualization,
Martin Radev, Masami Hiramatsu, Andy Lutomirski, hpa,
Erdem Aktas, David Rientjes, Dan Williams, Jiri Slaby
From: Joerg Roedel <jroedel@suse.de>
Make sure there is a stack once the kernel runs from virual addresses.
At this stage any secondary CPU which boots will have lost its stack
because the kernel switched to a new page-table which does not map the
real-mode stack anymore.
This is needed for handling early #VC exceptions caused by instructions
like CPUID.
Signed-off-by: Joerg Roedel <jroedel@suse.de>
Reviewed-by: Kees Cook <keescook@chromium.org>
---
arch/x86/kernel/head_64.S | 9 ++++++---
1 file changed, 6 insertions(+), 3 deletions(-)
diff --git a/arch/x86/kernel/head_64.S b/arch/x86/kernel/head_64.S
index f402087a02ac..83050c9e54d9 100644
--- a/arch/x86/kernel/head_64.S
+++ b/arch/x86/kernel/head_64.S
@@ -192,6 +192,12 @@ SYM_CODE_START(secondary_startup_64)
movl initial_gs+4(%rip),%edx
wrmsr
+ /*
+ * Setup a boot time stack - Any secondary CPU will have lost its stack
+ * by now because the cr3-switch above unmaps the real-mode stack
+ */
+ movq initial_stack(%rip), %rsp
+
/* Check if nx is implemented */
movl $0x80000001, %eax
cpuid
@@ -212,9 +218,6 @@ SYM_CODE_START(secondary_startup_64)
/* Make changes effective */
movq %rax, %cr0
- /* Setup a boot time stack */
- movq initial_stack(%rip), %rsp
-
/* zero EFLAGS after setting rsp */
pushq $0
popfq
--
2.28.0
_______________________________________________
Virtualization mailing list
Virtualization@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/virtualization
^ permalink raw reply related [flat|nested] 264+ messages in thread
* [PATCH v7 33/72] x86/head/64: Install a CPU bringup IDT
2020-09-07 13:15 ` Joerg Roedel
@ 2020-09-07 13:15 ` Joerg Roedel
-1 siblings, 0 replies; 264+ messages in thread
From: Joerg Roedel @ 2020-09-07 13:15 UTC (permalink / raw)
To: x86
Cc: Joerg Roedel, Joerg Roedel, hpa, Andy Lutomirski, Dave Hansen,
Peter Zijlstra, Jiri Slaby, Dan Williams, Tom Lendacky,
Juergen Gross, Kees Cook, David Rientjes, Cfir Cohen,
Erdem Aktas, Masami Hiramatsu, Mike Stunes, Sean Christopherson,
Martin Radev, linux-kernel, kvm, virtualization
From: Joerg Roedel <jroedel@suse.de>
Add a separate bringup IDT used by CPU bringup code that will be used
until the kernel switches to the idt_table. There are two reasons for
a separate IDT:
1) When the idt_table is set up and the secondary CPUs are
booted, it contains entries (e.g. IST entries) which
require certain CPU state to be set up. This includes a
working TSS (for IST), MSR_GS_BASE (for stack protector) or
CR4.FSGSBASE (for paranoid_entry) path. By using a
dedicated IDT for early boot this state need not to be set
up early.
2) The idt_table is static to idt.c, so any function
using/modifying must be in idt.c too. That means that all
compiler driven instrumentation like tracing or KASAN is
also active in this code. But during early CPU bringup the
environment is not set up for this instrumentation to work
correctly.
To avoid all of these hassles and make early exception handling
robust, use a dedicated bringup IDT.
The IDT is loaded two times, first on the boot CPU while the kernel is
still running on direct mapped addresses, and again later when the
switch to kernel addresses happened. The second IDT load happens on
the boot and secondary CPUs.
Signed-off-by: Joerg Roedel <jroedel@suse.de>
---
arch/x86/include/asm/setup.h | 1 +
arch/x86/kernel/head64.c | 39 ++++++++++++++++++++++++++++++++++++
arch/x86/kernel/head_64.S | 5 +++++
3 files changed, 45 insertions(+)
diff --git a/arch/x86/include/asm/setup.h b/arch/x86/include/asm/setup.h
index 5c2fd05bd52c..4b3ca5ade2fd 100644
--- a/arch/x86/include/asm/setup.h
+++ b/arch/x86/include/asm/setup.h
@@ -50,6 +50,7 @@ extern unsigned long __startup_64(unsigned long physaddr, struct boot_params *bp
extern unsigned long __startup_secondary_64(void);
extern void startup_64_setup_env(unsigned long physbase);
extern int early_make_pgtable(unsigned long address);
+extern void early_setup_idt(void);
#ifdef CONFIG_X86_INTEL_MID
extern void x86_intel_mid_early_setup(void);
diff --git a/arch/x86/kernel/head64.c b/arch/x86/kernel/head64.c
index 8c82be44be94..7bfd5c27c773 100644
--- a/arch/x86/kernel/head64.c
+++ b/arch/x86/kernel/head64.c
@@ -36,6 +36,8 @@
#include <asm/microcode.h>
#include <asm/kasan.h>
#include <asm/fixmap.h>
+#include <asm/realmode.h>
+#include <asm/desc.h>
/*
* Manage page tables very early on.
@@ -508,6 +510,41 @@ void __init x86_64_start_reservations(char *real_mode_data)
start_kernel();
}
+/*
+ * Data structures and code used for IDT setup in head_64.S. The bringup-IDT is
+ * used until the idt_table takes over. On the boot CPU this happens in
+ * x86_64_start_kernel(), on secondary CPUs in start_secondary(). In both cases
+ * this happens in the functions called from head_64.S.
+ *
+ * The idt_table can't be used that early because all the code modifying it is
+ * in idt.c and can be instrumented by tracing or KASAN, which both don't work
+ * during early CPU bringup. Also the idt_table has the runtime vectors
+ * configured which require certain CPU state to be setup already (like TSS),
+ * which also hasn't happened yet in early CPU bringup.
+ */
+static gate_desc bringup_idt_table[NUM_EXCEPTION_VECTORS] __page_aligned_data;
+
+static struct desc_ptr bringup_idt_descr = {
+ .size = (NUM_EXCEPTION_VECTORS * sizeof(gate_desc)) - 1,
+ .address = 0, /* Set at runtime */
+};
+
+/* This runs while still in the direct mapping */
+static void startup_64_load_idt(unsigned long physbase)
+{
+ struct desc_ptr *desc = fixup_pointer(&bringup_idt_descr, physbase);
+
+ desc->address = (unsigned long)fixup_pointer(bringup_idt_table, physbase);
+ native_load_idt(desc);
+}
+
+/* This is used when running on kernel addresses */
+void early_setup_idt(void)
+{
+ bringup_idt_descr.address = (unsigned long)bringup_idt_table;
+ native_load_idt(&bringup_idt_descr);
+}
+
/*
* Setup boot CPU state needed before kernel switches to virtual addresses.
*/
@@ -521,4 +558,6 @@ void __head startup_64_setup_env(unsigned long physbase)
asm volatile("movl %%eax, %%ds\n"
"movl %%eax, %%ss\n"
"movl %%eax, %%es\n" : : "a"(__KERNEL_DS) : "memory");
+
+ startup_64_load_idt(physbase);
}
diff --git a/arch/x86/kernel/head_64.S b/arch/x86/kernel/head_64.S
index 83050c9e54d9..1de09b58e578 100644
--- a/arch/x86/kernel/head_64.S
+++ b/arch/x86/kernel/head_64.S
@@ -198,6 +198,11 @@ SYM_CODE_START(secondary_startup_64)
*/
movq initial_stack(%rip), %rsp
+ /* Setup and Load IDT */
+ pushq %rsi
+ call early_setup_idt
+ popq %rsi
+
/* Check if nx is implemented */
movl $0x80000001, %eax
cpuid
--
2.28.0
^ permalink raw reply related [flat|nested] 264+ messages in thread
* [PATCH v7 33/72] x86/head/64: Install a CPU bringup IDT
@ 2020-09-07 13:15 ` Joerg Roedel
0 siblings, 0 replies; 264+ messages in thread
From: Joerg Roedel @ 2020-09-07 13:15 UTC (permalink / raw)
To: x86
Cc: Juergen Gross, Tom Lendacky, Joerg Roedel, Mike Stunes,
Kees Cook, kvm, Peter Zijlstra, Cfir Cohen, Joerg Roedel,
Dave Hansen, linux-kernel, Sean Christopherson, virtualization,
Martin Radev, Masami Hiramatsu, Andy Lutomirski, hpa,
Erdem Aktas, David Rientjes, Dan Williams, Jiri Slaby
From: Joerg Roedel <jroedel@suse.de>
Add a separate bringup IDT used by CPU bringup code that will be used
until the kernel switches to the idt_table. There are two reasons for
a separate IDT:
1) When the idt_table is set up and the secondary CPUs are
booted, it contains entries (e.g. IST entries) which
require certain CPU state to be set up. This includes a
working TSS (for IST), MSR_GS_BASE (for stack protector) or
CR4.FSGSBASE (for paranoid_entry) path. By using a
dedicated IDT for early boot this state need not to be set
up early.
2) The idt_table is static to idt.c, so any function
using/modifying must be in idt.c too. That means that all
compiler driven instrumentation like tracing or KASAN is
also active in this code. But during early CPU bringup the
environment is not set up for this instrumentation to work
correctly.
To avoid all of these hassles and make early exception handling
robust, use a dedicated bringup IDT.
The IDT is loaded two times, first on the boot CPU while the kernel is
still running on direct mapped addresses, and again later when the
switch to kernel addresses happened. The second IDT load happens on
the boot and secondary CPUs.
Signed-off-by: Joerg Roedel <jroedel@suse.de>
---
arch/x86/include/asm/setup.h | 1 +
arch/x86/kernel/head64.c | 39 ++++++++++++++++++++++++++++++++++++
arch/x86/kernel/head_64.S | 5 +++++
3 files changed, 45 insertions(+)
diff --git a/arch/x86/include/asm/setup.h b/arch/x86/include/asm/setup.h
index 5c2fd05bd52c..4b3ca5ade2fd 100644
--- a/arch/x86/include/asm/setup.h
+++ b/arch/x86/include/asm/setup.h
@@ -50,6 +50,7 @@ extern unsigned long __startup_64(unsigned long physaddr, struct boot_params *bp
extern unsigned long __startup_secondary_64(void);
extern void startup_64_setup_env(unsigned long physbase);
extern int early_make_pgtable(unsigned long address);
+extern void early_setup_idt(void);
#ifdef CONFIG_X86_INTEL_MID
extern void x86_intel_mid_early_setup(void);
diff --git a/arch/x86/kernel/head64.c b/arch/x86/kernel/head64.c
index 8c82be44be94..7bfd5c27c773 100644
--- a/arch/x86/kernel/head64.c
+++ b/arch/x86/kernel/head64.c
@@ -36,6 +36,8 @@
#include <asm/microcode.h>
#include <asm/kasan.h>
#include <asm/fixmap.h>
+#include <asm/realmode.h>
+#include <asm/desc.h>
/*
* Manage page tables very early on.
@@ -508,6 +510,41 @@ void __init x86_64_start_reservations(char *real_mode_data)
start_kernel();
}
+/*
+ * Data structures and code used for IDT setup in head_64.S. The bringup-IDT is
+ * used until the idt_table takes over. On the boot CPU this happens in
+ * x86_64_start_kernel(), on secondary CPUs in start_secondary(). In both cases
+ * this happens in the functions called from head_64.S.
+ *
+ * The idt_table can't be used that early because all the code modifying it is
+ * in idt.c and can be instrumented by tracing or KASAN, which both don't work
+ * during early CPU bringup. Also the idt_table has the runtime vectors
+ * configured which require certain CPU state to be setup already (like TSS),
+ * which also hasn't happened yet in early CPU bringup.
+ */
+static gate_desc bringup_idt_table[NUM_EXCEPTION_VECTORS] __page_aligned_data;
+
+static struct desc_ptr bringup_idt_descr = {
+ .size = (NUM_EXCEPTION_VECTORS * sizeof(gate_desc)) - 1,
+ .address = 0, /* Set at runtime */
+};
+
+/* This runs while still in the direct mapping */
+static void startup_64_load_idt(unsigned long physbase)
+{
+ struct desc_ptr *desc = fixup_pointer(&bringup_idt_descr, physbase);
+
+ desc->address = (unsigned long)fixup_pointer(bringup_idt_table, physbase);
+ native_load_idt(desc);
+}
+
+/* This is used when running on kernel addresses */
+void early_setup_idt(void)
+{
+ bringup_idt_descr.address = (unsigned long)bringup_idt_table;
+ native_load_idt(&bringup_idt_descr);
+}
+
/*
* Setup boot CPU state needed before kernel switches to virtual addresses.
*/
@@ -521,4 +558,6 @@ void __head startup_64_setup_env(unsigned long physbase)
asm volatile("movl %%eax, %%ds\n"
"movl %%eax, %%ss\n"
"movl %%eax, %%es\n" : : "a"(__KERNEL_DS) : "memory");
+
+ startup_64_load_idt(physbase);
}
diff --git a/arch/x86/kernel/head_64.S b/arch/x86/kernel/head_64.S
index 83050c9e54d9..1de09b58e578 100644
--- a/arch/x86/kernel/head_64.S
+++ b/arch/x86/kernel/head_64.S
@@ -198,6 +198,11 @@ SYM_CODE_START(secondary_startup_64)
*/
movq initial_stack(%rip), %rsp
+ /* Setup and Load IDT */
+ pushq %rsi
+ call early_setup_idt
+ popq %rsi
+
/* Check if nx is implemented */
movl $0x80000001, %eax
cpuid
--
2.28.0
_______________________________________________
Virtualization mailing list
Virtualization@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/virtualization
^ permalink raw reply related [flat|nested] 264+ messages in thread
* [PATCH v7 34/72] x86/idt: Move two function from k/idt.c to i/a/desc.h
2020-09-07 13:15 ` Joerg Roedel
@ 2020-09-07 13:15 ` Joerg Roedel
-1 siblings, 0 replies; 264+ messages in thread
From: Joerg Roedel @ 2020-09-07 13:15 UTC (permalink / raw)
To: x86
Cc: Joerg Roedel, Joerg Roedel, hpa, Andy Lutomirski, Dave Hansen,
Peter Zijlstra, Jiri Slaby, Dan Williams, Tom Lendacky,
Juergen Gross, Kees Cook, David Rientjes, Cfir Cohen,
Erdem Aktas, Masami Hiramatsu, Mike Stunes, Sean Christopherson,
Martin Radev, linux-kernel, kvm, virtualization
From: Joerg Roedel <jroedel@suse.de>
Move these two functions from kernel/idt.c to include/asm/desc.h:
* init_idt_data()
* idt_init_desc()
These functions are needed to setup IDT entries very early and need to
be called from head64.c. To be usable this early these functions need to
be compiled without instrumentation and the stack-protector feature.
These features need to be kept enabled for kernel/idt.c, so head64.c
must use its own versions.
Signed-off-by: Joerg Roedel <jroedel@suse.de>
---
arch/x86/include/asm/desc.h | 27 +++++++++++++++++++++++++
arch/x86/include/asm/desc_defs.h | 7 +++++++
arch/x86/kernel/idt.c | 34 --------------------------------
3 files changed, 34 insertions(+), 34 deletions(-)
diff --git a/arch/x86/include/asm/desc.h b/arch/x86/include/asm/desc.h
index 1ced11d31932..476082a83d1c 100644
--- a/arch/x86/include/asm/desc.h
+++ b/arch/x86/include/asm/desc.h
@@ -383,6 +383,33 @@ static inline void set_desc_limit(struct desc_struct *desc, unsigned long limit)
void alloc_intr_gate(unsigned int n, const void *addr);
+static inline void init_idt_data(struct idt_data *data, unsigned int n,
+ const void *addr)
+{
+ BUG_ON(n > 0xFF);
+
+ memset(data, 0, sizeof(*data));
+ data->vector = n;
+ data->addr = addr;
+ data->segment = __KERNEL_CS;
+ data->bits.type = GATE_INTERRUPT;
+ data->bits.p = 1;
+}
+
+static inline void idt_init_desc(gate_desc *gate, const struct idt_data *d)
+{
+ unsigned long addr = (unsigned long) d->addr;
+
+ gate->offset_low = (u16) addr;
+ gate->segment = (u16) d->segment;
+ gate->bits = d->bits;
+ gate->offset_middle = (u16) (addr >> 16);
+#ifdef CONFIG_X86_64
+ gate->offset_high = (u32) (addr >> 32);
+ gate->reserved = 0;
+#endif
+}
+
extern unsigned long system_vectors[];
extern void load_current_idt(void);
diff --git a/arch/x86/include/asm/desc_defs.h b/arch/x86/include/asm/desc_defs.h
index 5621fb3f2d1a..f7e7099af595 100644
--- a/arch/x86/include/asm/desc_defs.h
+++ b/arch/x86/include/asm/desc_defs.h
@@ -74,6 +74,13 @@ struct idt_bits {
p : 1;
} __attribute__((packed));
+struct idt_data {
+ unsigned int vector;
+ unsigned int segment;
+ struct idt_bits bits;
+ const void *addr;
+};
+
struct gate_struct {
u16 offset_low;
u16 segment;
diff --git a/arch/x86/kernel/idt.c b/arch/x86/kernel/idt.c
index 53946c104fa0..4bb4e3d6099e 100644
--- a/arch/x86/kernel/idt.c
+++ b/arch/x86/kernel/idt.c
@@ -11,13 +11,6 @@
#include <asm/desc.h>
#include <asm/hw_irq.h>
-struct idt_data {
- unsigned int vector;
- unsigned int segment;
- struct idt_bits bits;
- const void *addr;
-};
-
#define DPL0 0x0
#define DPL3 0x3
@@ -178,20 +171,6 @@ bool idt_is_f00f_address(unsigned long address)
}
#endif
-static inline void idt_init_desc(gate_desc *gate, const struct idt_data *d)
-{
- unsigned long addr = (unsigned long) d->addr;
-
- gate->offset_low = (u16) addr;
- gate->segment = (u16) d->segment;
- gate->bits = d->bits;
- gate->offset_middle = (u16) (addr >> 16);
-#ifdef CONFIG_X86_64
- gate->offset_high = (u32) (addr >> 32);
- gate->reserved = 0;
-#endif
-}
-
static __init void
idt_setup_from_table(gate_desc *idt, const struct idt_data *t, int size, bool sys)
{
@@ -205,19 +184,6 @@ idt_setup_from_table(gate_desc *idt, const struct idt_data *t, int size, bool sy
}
}
-static void init_idt_data(struct idt_data *data, unsigned int n,
- const void *addr)
-{
- BUG_ON(n > 0xFF);
-
- memset(data, 0, sizeof(*data));
- data->vector = n;
- data->addr = addr;
- data->segment = __KERNEL_CS;
- data->bits.type = GATE_INTERRUPT;
- data->bits.p = 1;
-}
-
static __init void set_intr_gate(unsigned int n, const void *addr)
{
struct idt_data data;
--
2.28.0
^ permalink raw reply related [flat|nested] 264+ messages in thread
* [PATCH v7 34/72] x86/idt: Move two function from k/idt.c to i/a/desc.h
@ 2020-09-07 13:15 ` Joerg Roedel
0 siblings, 0 replies; 264+ messages in thread
From: Joerg Roedel @ 2020-09-07 13:15 UTC (permalink / raw)
To: x86
Cc: Juergen Gross, Tom Lendacky, Joerg Roedel, Mike Stunes,
Kees Cook, kvm, Peter Zijlstra, Cfir Cohen, Joerg Roedel,
Dave Hansen, linux-kernel, Sean Christopherson, virtualization,
Martin Radev, Masami Hiramatsu, Andy Lutomirski, hpa,
Erdem Aktas, David Rientjes, Dan Williams, Jiri Slaby
From: Joerg Roedel <jroedel@suse.de>
Move these two functions from kernel/idt.c to include/asm/desc.h:
* init_idt_data()
* idt_init_desc()
These functions are needed to setup IDT entries very early and need to
be called from head64.c. To be usable this early these functions need to
be compiled without instrumentation and the stack-protector feature.
These features need to be kept enabled for kernel/idt.c, so head64.c
must use its own versions.
Signed-off-by: Joerg Roedel <jroedel@suse.de>
---
arch/x86/include/asm/desc.h | 27 +++++++++++++++++++++++++
arch/x86/include/asm/desc_defs.h | 7 +++++++
arch/x86/kernel/idt.c | 34 --------------------------------
3 files changed, 34 insertions(+), 34 deletions(-)
diff --git a/arch/x86/include/asm/desc.h b/arch/x86/include/asm/desc.h
index 1ced11d31932..476082a83d1c 100644
--- a/arch/x86/include/asm/desc.h
+++ b/arch/x86/include/asm/desc.h
@@ -383,6 +383,33 @@ static inline void set_desc_limit(struct desc_struct *desc, unsigned long limit)
void alloc_intr_gate(unsigned int n, const void *addr);
+static inline void init_idt_data(struct idt_data *data, unsigned int n,
+ const void *addr)
+{
+ BUG_ON(n > 0xFF);
+
+ memset(data, 0, sizeof(*data));
+ data->vector = n;
+ data->addr = addr;
+ data->segment = __KERNEL_CS;
+ data->bits.type = GATE_INTERRUPT;
+ data->bits.p = 1;
+}
+
+static inline void idt_init_desc(gate_desc *gate, const struct idt_data *d)
+{
+ unsigned long addr = (unsigned long) d->addr;
+
+ gate->offset_low = (u16) addr;
+ gate->segment = (u16) d->segment;
+ gate->bits = d->bits;
+ gate->offset_middle = (u16) (addr >> 16);
+#ifdef CONFIG_X86_64
+ gate->offset_high = (u32) (addr >> 32);
+ gate->reserved = 0;
+#endif
+}
+
extern unsigned long system_vectors[];
extern void load_current_idt(void);
diff --git a/arch/x86/include/asm/desc_defs.h b/arch/x86/include/asm/desc_defs.h
index 5621fb3f2d1a..f7e7099af595 100644
--- a/arch/x86/include/asm/desc_defs.h
+++ b/arch/x86/include/asm/desc_defs.h
@@ -74,6 +74,13 @@ struct idt_bits {
p : 1;
} __attribute__((packed));
+struct idt_data {
+ unsigned int vector;
+ unsigned int segment;
+ struct idt_bits bits;
+ const void *addr;
+};
+
struct gate_struct {
u16 offset_low;
u16 segment;
diff --git a/arch/x86/kernel/idt.c b/arch/x86/kernel/idt.c
index 53946c104fa0..4bb4e3d6099e 100644
--- a/arch/x86/kernel/idt.c
+++ b/arch/x86/kernel/idt.c
@@ -11,13 +11,6 @@
#include <asm/desc.h>
#include <asm/hw_irq.h>
-struct idt_data {
- unsigned int vector;
- unsigned int segment;
- struct idt_bits bits;
- const void *addr;
-};
-
#define DPL0 0x0
#define DPL3 0x3
@@ -178,20 +171,6 @@ bool idt_is_f00f_address(unsigned long address)
}
#endif
-static inline void idt_init_desc(gate_desc *gate, const struct idt_data *d)
-{
- unsigned long addr = (unsigned long) d->addr;
-
- gate->offset_low = (u16) addr;
- gate->segment = (u16) d->segment;
- gate->bits = d->bits;
- gate->offset_middle = (u16) (addr >> 16);
-#ifdef CONFIG_X86_64
- gate->offset_high = (u32) (addr >> 32);
- gate->reserved = 0;
-#endif
-}
-
static __init void
idt_setup_from_table(gate_desc *idt, const struct idt_data *t, int size, bool sys)
{
@@ -205,19 +184,6 @@ idt_setup_from_table(gate_desc *idt, const struct idt_data *t, int size, bool sy
}
}
-static void init_idt_data(struct idt_data *data, unsigned int n,
- const void *addr)
-{
- BUG_ON(n > 0xFF);
-
- memset(data, 0, sizeof(*data));
- data->vector = n;
- data->addr = addr;
- data->segment = __KERNEL_CS;
- data->bits.type = GATE_INTERRUPT;
- data->bits.p = 1;
-}
-
static __init void set_intr_gate(unsigned int n, const void *addr)
{
struct idt_data data;
--
2.28.0
_______________________________________________
Virtualization mailing list
Virtualization@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/virtualization
^ permalink raw reply related [flat|nested] 264+ messages in thread
* [PATCH v7 35/72] x86/head/64: Move early exception dispatch to C code
2020-09-07 13:15 ` Joerg Roedel
@ 2020-09-07 13:15 ` Joerg Roedel
-1 siblings, 0 replies; 264+ messages in thread
From: Joerg Roedel @ 2020-09-07 13:15 UTC (permalink / raw)
To: x86
Cc: Joerg Roedel, Joerg Roedel, hpa, Andy Lutomirski, Dave Hansen,
Peter Zijlstra, Jiri Slaby, Dan Williams, Tom Lendacky,
Juergen Gross, Kees Cook, David Rientjes, Cfir Cohen,
Erdem Aktas, Masami Hiramatsu, Mike Stunes, Sean Christopherson,
Martin Radev, linux-kernel, kvm, virtualization
From: Joerg Roedel <jroedel@suse.de>
Move the assembly coded dispatch between page-faults and all other
exceptions to C code to make it easier to maintain and extend.
Also change the return-type of early_make_pgtable() to bool and make it
static.
Signed-off-by: Joerg Roedel <jroedel@suse.de>
---
arch/x86/include/asm/pgtable.h | 2 +-
arch/x86/include/asm/setup.h | 4 +++-
arch/x86/kernel/head64.c | 19 +++++++++++++++----
arch/x86/kernel/head_64.S | 11 +----------
4 files changed, 20 insertions(+), 16 deletions(-)
diff --git a/arch/x86/include/asm/pgtable.h b/arch/x86/include/asm/pgtable.h
index 5e0dcc20614d..a02c67291cfc 100644
--- a/arch/x86/include/asm/pgtable.h
+++ b/arch/x86/include/asm/pgtable.h
@@ -28,7 +28,7 @@
#include <asm-generic/pgtable_uffd.h>
extern pgd_t early_top_pgt[PTRS_PER_PGD];
-int __init __early_make_pgtable(unsigned long address, pmdval_t pmd);
+bool __init __early_make_pgtable(unsigned long address, pmdval_t pmd);
void ptdump_walk_pgd_level(struct seq_file *m, struct mm_struct *mm);
void ptdump_walk_pgd_level_debugfs(struct seq_file *m, struct mm_struct *mm,
diff --git a/arch/x86/include/asm/setup.h b/arch/x86/include/asm/setup.h
index 4b3ca5ade2fd..7d7a064af6ff 100644
--- a/arch/x86/include/asm/setup.h
+++ b/arch/x86/include/asm/setup.h
@@ -39,6 +39,8 @@ void vsmp_init(void);
static inline void vsmp_init(void) { }
#endif
+struct pt_regs;
+
void setup_bios_corruption_check(void);
void early_platform_quirks(void);
@@ -49,8 +51,8 @@ extern void i386_reserve_resources(void);
extern unsigned long __startup_64(unsigned long physaddr, struct boot_params *bp);
extern unsigned long __startup_secondary_64(void);
extern void startup_64_setup_env(unsigned long physbase);
-extern int early_make_pgtable(unsigned long address);
extern void early_setup_idt(void);
+extern void __init do_early_exception(struct pt_regs *regs, int trapnr);
#ifdef CONFIG_X86_INTEL_MID
extern void x86_intel_mid_early_setup(void);
diff --git a/arch/x86/kernel/head64.c b/arch/x86/kernel/head64.c
index 7bfd5c27c773..4282dac694c3 100644
--- a/arch/x86/kernel/head64.c
+++ b/arch/x86/kernel/head64.c
@@ -38,6 +38,8 @@
#include <asm/fixmap.h>
#include <asm/realmode.h>
#include <asm/desc.h>
+#include <asm/extable.h>
+#include <asm/trapnr.h>
/*
* Manage page tables very early on.
@@ -317,7 +319,7 @@ static void __init reset_early_page_tables(void)
}
/* Create a new PMD entry */
-int __init __early_make_pgtable(unsigned long address, pmdval_t pmd)
+bool __init __early_make_pgtable(unsigned long address, pmdval_t pmd)
{
unsigned long physaddr = address - __PAGE_OFFSET;
pgdval_t pgd, *pgd_p;
@@ -327,7 +329,7 @@ int __init __early_make_pgtable(unsigned long address, pmdval_t pmd)
/* Invalid address or early pgt is done ? */
if (physaddr >= MAXMEM || read_cr3_pa() != __pa_nodebug(early_top_pgt))
- return -1;
+ return false;
again:
pgd_p = &early_top_pgt[pgd_index(address)].pgd;
@@ -384,10 +386,10 @@ int __init __early_make_pgtable(unsigned long address, pmdval_t pmd)
}
pmd_p[pmd_index(address)] = pmd;
- return 0;
+ return true;
}
-int __init early_make_pgtable(unsigned long address)
+static bool __init early_make_pgtable(unsigned long address)
{
unsigned long physaddr = address - __PAGE_OFFSET;
pmdval_t pmd;
@@ -397,6 +399,15 @@ int __init early_make_pgtable(unsigned long address)
return __early_make_pgtable(address, pmd);
}
+void __init do_early_exception(struct pt_regs *regs, int trapnr)
+{
+ if (trapnr == X86_TRAP_PF &&
+ early_make_pgtable(native_read_cr2()))
+ return;
+
+ early_fixup_exception(regs, trapnr);
+}
+
/* Don't add a printk in there. printk relies on the PDA which is not initialized
yet. */
static void __init clear_bss(void)
diff --git a/arch/x86/kernel/head_64.S b/arch/x86/kernel/head_64.S
index 1de09b58e578..3b40ec44a67d 100644
--- a/arch/x86/kernel/head_64.S
+++ b/arch/x86/kernel/head_64.S
@@ -341,18 +341,9 @@ SYM_CODE_START_LOCAL(early_idt_handler_common)
pushq %r15 /* pt_regs->r15 */
UNWIND_HINT_REGS
- cmpq $14,%rsi /* Page fault? */
- jnz 10f
- GET_CR2_INTO(%rdi) /* can clobber %rax if pv */
- call early_make_pgtable
- andl %eax,%eax
- jz 20f /* All good */
-
-10:
movq %rsp,%rdi /* RDI = pt_regs; RSI is already trapnr */
- call early_fixup_exception
+ call do_early_exception
-20:
decl early_recursion_flag(%rip)
jmp restore_regs_and_return_to_kernel
SYM_CODE_END(early_idt_handler_common)
--
2.28.0
^ permalink raw reply related [flat|nested] 264+ messages in thread
* [PATCH v7 35/72] x86/head/64: Move early exception dispatch to C code
@ 2020-09-07 13:15 ` Joerg Roedel
0 siblings, 0 replies; 264+ messages in thread
From: Joerg Roedel @ 2020-09-07 13:15 UTC (permalink / raw)
To: x86
Cc: Juergen Gross, Tom Lendacky, Joerg Roedel, Mike Stunes,
Kees Cook, kvm, Peter Zijlstra, Cfir Cohen, Joerg Roedel,
Dave Hansen, linux-kernel, Sean Christopherson, virtualization,
Martin Radev, Masami Hiramatsu, Andy Lutomirski, hpa,
Erdem Aktas, David Rientjes, Dan Williams, Jiri Slaby
From: Joerg Roedel <jroedel@suse.de>
Move the assembly coded dispatch between page-faults and all other
exceptions to C code to make it easier to maintain and extend.
Also change the return-type of early_make_pgtable() to bool and make it
static.
Signed-off-by: Joerg Roedel <jroedel@suse.de>
---
arch/x86/include/asm/pgtable.h | 2 +-
arch/x86/include/asm/setup.h | 4 +++-
arch/x86/kernel/head64.c | 19 +++++++++++++++----
arch/x86/kernel/head_64.S | 11 +----------
4 files changed, 20 insertions(+), 16 deletions(-)
diff --git a/arch/x86/include/asm/pgtable.h b/arch/x86/include/asm/pgtable.h
index 5e0dcc20614d..a02c67291cfc 100644
--- a/arch/x86/include/asm/pgtable.h
+++ b/arch/x86/include/asm/pgtable.h
@@ -28,7 +28,7 @@
#include <asm-generic/pgtable_uffd.h>
extern pgd_t early_top_pgt[PTRS_PER_PGD];
-int __init __early_make_pgtable(unsigned long address, pmdval_t pmd);
+bool __init __early_make_pgtable(unsigned long address, pmdval_t pmd);
void ptdump_walk_pgd_level(struct seq_file *m, struct mm_struct *mm);
void ptdump_walk_pgd_level_debugfs(struct seq_file *m, struct mm_struct *mm,
diff --git a/arch/x86/include/asm/setup.h b/arch/x86/include/asm/setup.h
index 4b3ca5ade2fd..7d7a064af6ff 100644
--- a/arch/x86/include/asm/setup.h
+++ b/arch/x86/include/asm/setup.h
@@ -39,6 +39,8 @@ void vsmp_init(void);
static inline void vsmp_init(void) { }
#endif
+struct pt_regs;
+
void setup_bios_corruption_check(void);
void early_platform_quirks(void);
@@ -49,8 +51,8 @@ extern void i386_reserve_resources(void);
extern unsigned long __startup_64(unsigned long physaddr, struct boot_params *bp);
extern unsigned long __startup_secondary_64(void);
extern void startup_64_setup_env(unsigned long physbase);
-extern int early_make_pgtable(unsigned long address);
extern void early_setup_idt(void);
+extern void __init do_early_exception(struct pt_regs *regs, int trapnr);
#ifdef CONFIG_X86_INTEL_MID
extern void x86_intel_mid_early_setup(void);
diff --git a/arch/x86/kernel/head64.c b/arch/x86/kernel/head64.c
index 7bfd5c27c773..4282dac694c3 100644
--- a/arch/x86/kernel/head64.c
+++ b/arch/x86/kernel/head64.c
@@ -38,6 +38,8 @@
#include <asm/fixmap.h>
#include <asm/realmode.h>
#include <asm/desc.h>
+#include <asm/extable.h>
+#include <asm/trapnr.h>
/*
* Manage page tables very early on.
@@ -317,7 +319,7 @@ static void __init reset_early_page_tables(void)
}
/* Create a new PMD entry */
-int __init __early_make_pgtable(unsigned long address, pmdval_t pmd)
+bool __init __early_make_pgtable(unsigned long address, pmdval_t pmd)
{
unsigned long physaddr = address - __PAGE_OFFSET;
pgdval_t pgd, *pgd_p;
@@ -327,7 +329,7 @@ int __init __early_make_pgtable(unsigned long address, pmdval_t pmd)
/* Invalid address or early pgt is done ? */
if (physaddr >= MAXMEM || read_cr3_pa() != __pa_nodebug(early_top_pgt))
- return -1;
+ return false;
again:
pgd_p = &early_top_pgt[pgd_index(address)].pgd;
@@ -384,10 +386,10 @@ int __init __early_make_pgtable(unsigned long address, pmdval_t pmd)
}
pmd_p[pmd_index(address)] = pmd;
- return 0;
+ return true;
}
-int __init early_make_pgtable(unsigned long address)
+static bool __init early_make_pgtable(unsigned long address)
{
unsigned long physaddr = address - __PAGE_OFFSET;
pmdval_t pmd;
@@ -397,6 +399,15 @@ int __init early_make_pgtable(unsigned long address)
return __early_make_pgtable(address, pmd);
}
+void __init do_early_exception(struct pt_regs *regs, int trapnr)
+{
+ if (trapnr == X86_TRAP_PF &&
+ early_make_pgtable(native_read_cr2()))
+ return;
+
+ early_fixup_exception(regs, trapnr);
+}
+
/* Don't add a printk in there. printk relies on the PDA which is not initialized
yet. */
static void __init clear_bss(void)
diff --git a/arch/x86/kernel/head_64.S b/arch/x86/kernel/head_64.S
index 1de09b58e578..3b40ec44a67d 100644
--- a/arch/x86/kernel/head_64.S
+++ b/arch/x86/kernel/head_64.S
@@ -341,18 +341,9 @@ SYM_CODE_START_LOCAL(early_idt_handler_common)
pushq %r15 /* pt_regs->r15 */
UNWIND_HINT_REGS
- cmpq $14,%rsi /* Page fault? */
- jnz 10f
- GET_CR2_INTO(%rdi) /* can clobber %rax if pv */
- call early_make_pgtable
- andl %eax,%eax
- jz 20f /* All good */
-
-10:
movq %rsp,%rdi /* RDI = pt_regs; RSI is already trapnr */
- call early_fixup_exception
+ call do_early_exception
-20:
decl early_recursion_flag(%rip)
jmp restore_regs_and_return_to_kernel
SYM_CODE_END(early_idt_handler_common)
--
2.28.0
_______________________________________________
Virtualization mailing list
Virtualization@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/virtualization
^ permalink raw reply related [flat|nested] 264+ messages in thread
* [PATCH v7 36/72] x86/sev-es: Add SEV-ES Feature Detection
2020-09-07 13:15 ` Joerg Roedel
@ 2020-09-07 13:15 ` Joerg Roedel
-1 siblings, 0 replies; 264+ messages in thread
From: Joerg Roedel @ 2020-09-07 13:15 UTC (permalink / raw)
To: x86
Cc: Joerg Roedel, Joerg Roedel, Kees Cook, hpa, Andy Lutomirski,
Dave Hansen, Peter Zijlstra, Jiri Slaby, Dan Williams,
Tom Lendacky, Juergen Gross, David Rientjes, Cfir Cohen,
Erdem Aktas, Masami Hiramatsu, Mike Stunes, Sean Christopherson,
Martin Radev, linux-kernel, kvm, virtualization
From: Joerg Roedel <jroedel@suse.de>
Add the sev_es_active function for checking whether SEV-ES is enabled.
Also cache the value of MSR_AMD64_SEV at boot to speed up the feature
checking in the running code.
Signed-off-by: Joerg Roedel <jroedel@suse.de>
Reviewed-by: Kees Cook <keescook@chromium.org>
---
arch/x86/include/asm/mem_encrypt.h | 3 +++
arch/x86/include/asm/msr-index.h | 2 ++
arch/x86/mm/mem_encrypt.c | 9 ++++++++-
arch/x86/mm/mem_encrypt_identity.c | 3 +++
4 files changed, 16 insertions(+), 1 deletion(-)
diff --git a/arch/x86/include/asm/mem_encrypt.h b/arch/x86/include/asm/mem_encrypt.h
index 5049f6c22683..4e72b73a9cb5 100644
--- a/arch/x86/include/asm/mem_encrypt.h
+++ b/arch/x86/include/asm/mem_encrypt.h
@@ -19,6 +19,7 @@
#ifdef CONFIG_AMD_MEM_ENCRYPT
extern u64 sme_me_mask;
+extern u64 sev_status;
extern bool sev_enabled;
void sme_encrypt_execute(unsigned long encrypted_kernel_vaddr,
@@ -50,6 +51,7 @@ void __init mem_encrypt_init(void);
bool sme_active(void);
bool sev_active(void);
+bool sev_es_active(void);
#define __bss_decrypted __attribute__((__section__(".bss..decrypted")))
@@ -72,6 +74,7 @@ static inline void __init sme_enable(struct boot_params *bp) { }
static inline bool sme_active(void) { return false; }
static inline bool sev_active(void) { return false; }
+static inline bool sev_es_active(void) { return false; }
static inline int __init
early_set_memory_decrypted(unsigned long vaddr, unsigned long size) { return 0; }
diff --git a/arch/x86/include/asm/msr-index.h b/arch/x86/include/asm/msr-index.h
index cd6d651ff730..95871defba91 100644
--- a/arch/x86/include/asm/msr-index.h
+++ b/arch/x86/include/asm/msr-index.h
@@ -469,7 +469,9 @@
#define MSR_AMD64_SEV_ES_GHCB 0xc0010130
#define MSR_AMD64_SEV 0xc0010131
#define MSR_AMD64_SEV_ENABLED_BIT 0
+#define MSR_AMD64_SEV_ES_ENABLED_BIT 1
#define MSR_AMD64_SEV_ENABLED BIT_ULL(MSR_AMD64_SEV_ENABLED_BIT)
+#define MSR_AMD64_SEV_ES_ENABLED BIT_ULL(MSR_AMD64_SEV_ES_ENABLED_BIT)
#define MSR_AMD64_VIRT_SPEC_CTRL 0xc001011f
diff --git a/arch/x86/mm/mem_encrypt.c b/arch/x86/mm/mem_encrypt.c
index 9f1177edc2e7..d0d4ebcec1be 100644
--- a/arch/x86/mm/mem_encrypt.c
+++ b/arch/x86/mm/mem_encrypt.c
@@ -38,6 +38,7 @@
* section is later cleared.
*/
u64 sme_me_mask __section(.data) = 0;
+u64 sev_status __section(.data) = 0;
EXPORT_SYMBOL(sme_me_mask);
DEFINE_STATIC_KEY_FALSE(sev_enable_key);
EXPORT_SYMBOL_GPL(sev_enable_key);
@@ -347,7 +348,13 @@ bool sme_active(void)
bool sev_active(void)
{
- return sme_me_mask && sev_enabled;
+ return !!(sev_status & MSR_AMD64_SEV_ENABLED);
+}
+
+/* Needs to be called from non-instrumentable code */
+bool noinstr sev_es_active(void)
+{
+ return sev_status & MSR_AMD64_SEV_ES_ENABLED;
}
/* Override for DMA direct allocation check - ARCH_HAS_FORCE_DMA_UNENCRYPTED */
diff --git a/arch/x86/mm/mem_encrypt_identity.c b/arch/x86/mm/mem_encrypt_identity.c
index e2b0e2ac07bb..68d75379e06a 100644
--- a/arch/x86/mm/mem_encrypt_identity.c
+++ b/arch/x86/mm/mem_encrypt_identity.c
@@ -540,6 +540,9 @@ void __init sme_enable(struct boot_params *bp)
if (!(msr & MSR_AMD64_SEV_ENABLED))
return;
+ /* Save SEV_STATUS to avoid reading MSR again */
+ sev_status = msr;
+
/* SEV state cannot be controlled by a command line option */
sme_me_mask = me_mask;
sev_enabled = true;
--
2.28.0
^ permalink raw reply related [flat|nested] 264+ messages in thread
* [PATCH v7 36/72] x86/sev-es: Add SEV-ES Feature Detection
@ 2020-09-07 13:15 ` Joerg Roedel
0 siblings, 0 replies; 264+ messages in thread
From: Joerg Roedel @ 2020-09-07 13:15 UTC (permalink / raw)
To: x86
Cc: Juergen Gross, Tom Lendacky, Joerg Roedel, Mike Stunes,
Kees Cook, kvm, Peter Zijlstra, Cfir Cohen, Joerg Roedel,
Dave Hansen, linux-kernel, Sean Christopherson, virtualization,
Martin Radev, Masami Hiramatsu, Andy Lutomirski, hpa,
Erdem Aktas, David Rientjes, Dan Williams, Jiri Slaby
From: Joerg Roedel <jroedel@suse.de>
Add the sev_es_active function for checking whether SEV-ES is enabled.
Also cache the value of MSR_AMD64_SEV at boot to speed up the feature
checking in the running code.
Signed-off-by: Joerg Roedel <jroedel@suse.de>
Reviewed-by: Kees Cook <keescook@chromium.org>
---
arch/x86/include/asm/mem_encrypt.h | 3 +++
arch/x86/include/asm/msr-index.h | 2 ++
arch/x86/mm/mem_encrypt.c | 9 ++++++++-
arch/x86/mm/mem_encrypt_identity.c | 3 +++
4 files changed, 16 insertions(+), 1 deletion(-)
diff --git a/arch/x86/include/asm/mem_encrypt.h b/arch/x86/include/asm/mem_encrypt.h
index 5049f6c22683..4e72b73a9cb5 100644
--- a/arch/x86/include/asm/mem_encrypt.h
+++ b/arch/x86/include/asm/mem_encrypt.h
@@ -19,6 +19,7 @@
#ifdef CONFIG_AMD_MEM_ENCRYPT
extern u64 sme_me_mask;
+extern u64 sev_status;
extern bool sev_enabled;
void sme_encrypt_execute(unsigned long encrypted_kernel_vaddr,
@@ -50,6 +51,7 @@ void __init mem_encrypt_init(void);
bool sme_active(void);
bool sev_active(void);
+bool sev_es_active(void);
#define __bss_decrypted __attribute__((__section__(".bss..decrypted")))
@@ -72,6 +74,7 @@ static inline void __init sme_enable(struct boot_params *bp) { }
static inline bool sme_active(void) { return false; }
static inline bool sev_active(void) { return false; }
+static inline bool sev_es_active(void) { return false; }
static inline int __init
early_set_memory_decrypted(unsigned long vaddr, unsigned long size) { return 0; }
diff --git a/arch/x86/include/asm/msr-index.h b/arch/x86/include/asm/msr-index.h
index cd6d651ff730..95871defba91 100644
--- a/arch/x86/include/asm/msr-index.h
+++ b/arch/x86/include/asm/msr-index.h
@@ -469,7 +469,9 @@
#define MSR_AMD64_SEV_ES_GHCB 0xc0010130
#define MSR_AMD64_SEV 0xc0010131
#define MSR_AMD64_SEV_ENABLED_BIT 0
+#define MSR_AMD64_SEV_ES_ENABLED_BIT 1
#define MSR_AMD64_SEV_ENABLED BIT_ULL(MSR_AMD64_SEV_ENABLED_BIT)
+#define MSR_AMD64_SEV_ES_ENABLED BIT_ULL(MSR_AMD64_SEV_ES_ENABLED_BIT)
#define MSR_AMD64_VIRT_SPEC_CTRL 0xc001011f
diff --git a/arch/x86/mm/mem_encrypt.c b/arch/x86/mm/mem_encrypt.c
index 9f1177edc2e7..d0d4ebcec1be 100644
--- a/arch/x86/mm/mem_encrypt.c
+++ b/arch/x86/mm/mem_encrypt.c
@@ -38,6 +38,7 @@
* section is later cleared.
*/
u64 sme_me_mask __section(.data) = 0;
+u64 sev_status __section(.data) = 0;
EXPORT_SYMBOL(sme_me_mask);
DEFINE_STATIC_KEY_FALSE(sev_enable_key);
EXPORT_SYMBOL_GPL(sev_enable_key);
@@ -347,7 +348,13 @@ bool sme_active(void)
bool sev_active(void)
{
- return sme_me_mask && sev_enabled;
+ return !!(sev_status & MSR_AMD64_SEV_ENABLED);
+}
+
+/* Needs to be called from non-instrumentable code */
+bool noinstr sev_es_active(void)
+{
+ return sev_status & MSR_AMD64_SEV_ES_ENABLED;
}
/* Override for DMA direct allocation check - ARCH_HAS_FORCE_DMA_UNENCRYPTED */
diff --git a/arch/x86/mm/mem_encrypt_identity.c b/arch/x86/mm/mem_encrypt_identity.c
index e2b0e2ac07bb..68d75379e06a 100644
--- a/arch/x86/mm/mem_encrypt_identity.c
+++ b/arch/x86/mm/mem_encrypt_identity.c
@@ -540,6 +540,9 @@ void __init sme_enable(struct boot_params *bp)
if (!(msr & MSR_AMD64_SEV_ENABLED))
return;
+ /* Save SEV_STATUS to avoid reading MSR again */
+ sev_status = msr;
+
/* SEV state cannot be controlled by a command line option */
sme_me_mask = me_mask;
sev_enabled = true;
--
2.28.0
_______________________________________________
Virtualization mailing list
Virtualization@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/virtualization
^ permalink raw reply related [flat|nested] 264+ messages in thread
* [PATCH v7 37/72] x86/sev-es: Print SEV-ES info into kernel log
2020-09-07 13:15 ` Joerg Roedel
@ 2020-09-07 13:15 ` Joerg Roedel
-1 siblings, 0 replies; 264+ messages in thread
From: Joerg Roedel @ 2020-09-07 13:15 UTC (permalink / raw)
To: x86
Cc: Joerg Roedel, Joerg Roedel, Kees Cook, hpa, Andy Lutomirski,
Dave Hansen, Peter Zijlstra, Jiri Slaby, Dan Williams,
Tom Lendacky, Juergen Gross, David Rientjes, Cfir Cohen,
Erdem Aktas, Masami Hiramatsu, Mike Stunes, Sean Christopherson,
Martin Radev, linux-kernel, kvm, virtualization
From: Joerg Roedel <jroedel@suse.de>
Refactor the message printed to the kernel log which indicates whether
SEV or SME is active to print a list of enabled encryption features.
This will scale better in the future when more memory encryption
features might be added. Also add SEV-ES to the list of features.
Signed-off-by: Joerg Roedel <jroedel@suse.de>
Reviewed-by: Kees Cook <keescook@chromium.org>
---
arch/x86/mm/mem_encrypt.c | 29 ++++++++++++++++++++++++++---
1 file changed, 26 insertions(+), 3 deletions(-)
diff --git a/arch/x86/mm/mem_encrypt.c b/arch/x86/mm/mem_encrypt.c
index d0d4ebcec1be..d6b8f4c1d3fa 100644
--- a/arch/x86/mm/mem_encrypt.c
+++ b/arch/x86/mm/mem_encrypt.c
@@ -407,6 +407,31 @@ void __init mem_encrypt_free_decrypted_mem(void)
free_init_pages("unused decrypted", vaddr, vaddr_end);
}
+static void print_mem_encrypt_feature_info(void)
+{
+ pr_info("AMD Memory Encryption Features active:");
+
+ /* Secure Memory Encryption */
+ if (sme_active()) {
+ /*
+ * SME is mutually exclusive with any of the SEV
+ * features below.
+ */
+ pr_cont(" SME\n");
+ return;
+ }
+
+ /* Secure Encrypted Virtualization */
+ if (sev_active())
+ pr_cont(" SEV");
+
+ /* Encrypted Register State */
+ if (sev_es_active())
+ pr_cont(" SEV-ES");
+
+ pr_cont("\n");
+}
+
/* Architecture __weak replacement functions */
void __init mem_encrypt_init(void)
{
@@ -422,8 +447,6 @@ void __init mem_encrypt_init(void)
if (sev_active())
static_branch_enable(&sev_enable_key);
- pr_info("AMD %s active\n",
- sev_active() ? "Secure Encrypted Virtualization (SEV)"
- : "Secure Memory Encryption (SME)");
+ print_mem_encrypt_feature_info();
}
--
2.28.0
^ permalink raw reply related [flat|nested] 264+ messages in thread
* [PATCH v7 37/72] x86/sev-es: Print SEV-ES info into kernel log
@ 2020-09-07 13:15 ` Joerg Roedel
0 siblings, 0 replies; 264+ messages in thread
From: Joerg Roedel @ 2020-09-07 13:15 UTC (permalink / raw)
To: x86
Cc: Juergen Gross, Tom Lendacky, Joerg Roedel, Mike Stunes,
Kees Cook, kvm, Peter Zijlstra, Cfir Cohen, Joerg Roedel,
Dave Hansen, linux-kernel, Sean Christopherson, virtualization,
Martin Radev, Masami Hiramatsu, Andy Lutomirski, hpa,
Erdem Aktas, David Rientjes, Dan Williams, Jiri Slaby
From: Joerg Roedel <jroedel@suse.de>
Refactor the message printed to the kernel log which indicates whether
SEV or SME is active to print a list of enabled encryption features.
This will scale better in the future when more memory encryption
features might be added. Also add SEV-ES to the list of features.
Signed-off-by: Joerg Roedel <jroedel@suse.de>
Reviewed-by: Kees Cook <keescook@chromium.org>
---
arch/x86/mm/mem_encrypt.c | 29 ++++++++++++++++++++++++++---
1 file changed, 26 insertions(+), 3 deletions(-)
diff --git a/arch/x86/mm/mem_encrypt.c b/arch/x86/mm/mem_encrypt.c
index d0d4ebcec1be..d6b8f4c1d3fa 100644
--- a/arch/x86/mm/mem_encrypt.c
+++ b/arch/x86/mm/mem_encrypt.c
@@ -407,6 +407,31 @@ void __init mem_encrypt_free_decrypted_mem(void)
free_init_pages("unused decrypted", vaddr, vaddr_end);
}
+static void print_mem_encrypt_feature_info(void)
+{
+ pr_info("AMD Memory Encryption Features active:");
+
+ /* Secure Memory Encryption */
+ if (sme_active()) {
+ /*
+ * SME is mutually exclusive with any of the SEV
+ * features below.
+ */
+ pr_cont(" SME\n");
+ return;
+ }
+
+ /* Secure Encrypted Virtualization */
+ if (sev_active())
+ pr_cont(" SEV");
+
+ /* Encrypted Register State */
+ if (sev_es_active())
+ pr_cont(" SEV-ES");
+
+ pr_cont("\n");
+}
+
/* Architecture __weak replacement functions */
void __init mem_encrypt_init(void)
{
@@ -422,8 +447,6 @@ void __init mem_encrypt_init(void)
if (sev_active())
static_branch_enable(&sev_enable_key);
- pr_info("AMD %s active\n",
- sev_active() ? "Secure Encrypted Virtualization (SEV)"
- : "Secure Memory Encryption (SME)");
+ print_mem_encrypt_feature_info();
}
--
2.28.0
_______________________________________________
Virtualization mailing list
Virtualization@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/virtualization
^ permalink raw reply related [flat|nested] 264+ messages in thread
* [PATCH v7 38/72] x86/sev-es: Compile early handler code into kernel image
2020-09-07 13:15 ` Joerg Roedel
@ 2020-09-07 13:15 ` Joerg Roedel
-1 siblings, 0 replies; 264+ messages in thread
From: Joerg Roedel @ 2020-09-07 13:15 UTC (permalink / raw)
To: x86
Cc: Joerg Roedel, Joerg Roedel, hpa, Andy Lutomirski, Dave Hansen,
Peter Zijlstra, Jiri Slaby, Dan Williams, Tom Lendacky,
Juergen Gross, Kees Cook, David Rientjes, Cfir Cohen,
Erdem Aktas, Masami Hiramatsu, Mike Stunes, Sean Christopherson,
Martin Radev, linux-kernel, kvm, virtualization
From: Joerg Roedel <jroedel@suse.de>
Setup sev-es.c and include the code from the
pre-decompression stage to also build it into the image of the running
kernel. Temporarily add __maybe_unused annotations to avoid build
warnings until the functions get used.
Signed-off-by: Joerg Roedel <jroedel@suse.de>
---
arch/x86/kernel/Makefile | 1 +
arch/x86/kernel/sev-es-shared.c | 21 ++--
arch/x86/kernel/sev-es.c | 163 ++++++++++++++++++++++++++++++++
3 files changed, 175 insertions(+), 10 deletions(-)
create mode 100644 arch/x86/kernel/sev-es.c
diff --git a/arch/x86/kernel/Makefile b/arch/x86/kernel/Makefile
index de09af019e23..3bcdd8d2bbdd 100644
--- a/arch/x86/kernel/Makefile
+++ b/arch/x86/kernel/Makefile
@@ -146,6 +146,7 @@ obj-$(CONFIG_UNWINDER_ORC) += unwind_orc.o
obj-$(CONFIG_UNWINDER_FRAME_POINTER) += unwind_frame.o
obj-$(CONFIG_UNWINDER_GUESS) += unwind_guess.o
+obj-$(CONFIG_AMD_MEM_ENCRYPT) += sev-es.o
###
# 64 bit specific files
ifeq ($(CONFIG_X86_64),y)
diff --git a/arch/x86/kernel/sev-es-shared.c b/arch/x86/kernel/sev-es-shared.c
index a6b41910b8ab..18619279a46f 100644
--- a/arch/x86/kernel/sev-es-shared.c
+++ b/arch/x86/kernel/sev-es-shared.c
@@ -9,7 +9,7 @@
* and is included directly into both code-bases.
*/
-static void sev_es_terminate(unsigned int reason)
+static void __maybe_unused sev_es_terminate(unsigned int reason)
{
u64 val = GHCB_SEV_TERMINATE;
@@ -27,7 +27,7 @@ static void sev_es_terminate(unsigned int reason)
asm volatile("hlt\n" : : : "memory");
}
-static bool sev_es_negotiate_protocol(void)
+static bool __maybe_unused sev_es_negotiate_protocol(void)
{
u64 val;
@@ -46,7 +46,7 @@ static bool sev_es_negotiate_protocol(void)
return true;
}
-static void vc_ghcb_invalidate(struct ghcb *ghcb)
+static void __maybe_unused vc_ghcb_invalidate(struct ghcb *ghcb)
{
memset(ghcb->save.valid_bitmap, 0, sizeof(ghcb->save.valid_bitmap));
}
@@ -58,9 +58,9 @@ static bool vc_decoding_needed(unsigned long exit_code)
exit_code <= SVM_EXIT_LAST_EXCP);
}
-static enum es_result vc_init_em_ctxt(struct es_em_ctxt *ctxt,
- struct pt_regs *regs,
- unsigned long exit_code)
+static enum es_result __maybe_unused vc_init_em_ctxt(struct es_em_ctxt *ctxt,
+ struct pt_regs *regs,
+ unsigned long exit_code)
{
enum es_result ret = ES_OK;
@@ -73,7 +73,7 @@ static enum es_result vc_init_em_ctxt(struct es_em_ctxt *ctxt,
return ret;
}
-static void vc_finish_insn(struct es_em_ctxt *ctxt)
+static void __maybe_unused vc_finish_insn(struct es_em_ctxt *ctxt)
{
ctxt->regs->ip += ctxt->insn.length;
}
@@ -325,7 +325,8 @@ static enum es_result vc_ioio_exitinfo(struct es_em_ctxt *ctxt, u64 *exitinfo)
return ES_OK;
}
-static enum es_result vc_handle_ioio(struct ghcb *ghcb, struct es_em_ctxt *ctxt)
+static enum es_result __maybe_unused
+vc_handle_ioio(struct ghcb *ghcb, struct es_em_ctxt *ctxt)
{
struct pt_regs *regs = ctxt->regs;
u64 exit_info_1, exit_info_2;
@@ -433,8 +434,8 @@ static enum es_result vc_handle_ioio(struct ghcb *ghcb, struct es_em_ctxt *ctxt)
return ret;
}
-static enum es_result vc_handle_cpuid(struct ghcb *ghcb,
- struct es_em_ctxt *ctxt)
+static enum es_result __maybe_unused vc_handle_cpuid(struct ghcb *ghcb,
+ struct es_em_ctxt *ctxt)
{
struct pt_regs *regs = ctxt->regs;
u32 cr4 = native_read_cr4();
diff --git a/arch/x86/kernel/sev-es.c b/arch/x86/kernel/sev-es.c
new file mode 100644
index 000000000000..0b698b653c0b
--- /dev/null
+++ b/arch/x86/kernel/sev-es.c
@@ -0,0 +1,163 @@
+// SPDX-License-Identifier: GPL-2.0-only
+/*
+ * AMD Memory Encryption Support
+ *
+ * Copyright (C) 2019 SUSE
+ *
+ * Author: Joerg Roedel <jroedel@suse.de>
+ */
+
+#include <linux/kernel.h>
+#include <linux/mm.h>
+
+#include <asm/sev-es.h>
+#include <asm/insn-eval.h>
+#include <asm/fpu/internal.h>
+#include <asm/processor.h>
+#include <asm/trap_pf.h>
+#include <asm/trapnr.h>
+#include <asm/svm.h>
+
+static inline u64 sev_es_rd_ghcb_msr(void)
+{
+ return native_read_msr(MSR_AMD64_SEV_ES_GHCB);
+}
+
+static inline void sev_es_wr_ghcb_msr(u64 val)
+{
+ u32 low, high;
+
+ low = (u32)(val);
+ high = (u32)(val >> 32);
+
+ native_write_msr(MSR_AMD64_SEV_ES_GHCB, low, high);
+}
+
+static int vc_fetch_insn_kernel(struct es_em_ctxt *ctxt,
+ unsigned char *buffer)
+{
+ return copy_from_kernel_nofault(buffer, (unsigned char *)ctxt->regs->ip, MAX_INSN_SIZE);
+}
+
+static enum es_result vc_decode_insn(struct es_em_ctxt *ctxt)
+{
+ char buffer[MAX_INSN_SIZE];
+ enum es_result ret;
+ int res;
+
+ res = vc_fetch_insn_kernel(ctxt, buffer);
+ if (unlikely(res == -EFAULT)) {
+ ctxt->fi.vector = X86_TRAP_PF;
+ ctxt->fi.error_code = 0;
+ ctxt->fi.cr2 = ctxt->regs->ip;
+ return ES_EXCEPTION;
+ }
+
+ insn_init(&ctxt->insn, buffer, MAX_INSN_SIZE - res, 1);
+ insn_get_length(&ctxt->insn);
+
+ ret = ctxt->insn.immediate.got ? ES_OK : ES_DECODE_FAILED;
+
+ return ret;
+}
+
+static enum es_result vc_write_mem(struct es_em_ctxt *ctxt,
+ char *dst, char *buf, size_t size)
+{
+ unsigned long error_code = X86_PF_PROT | X86_PF_WRITE;
+ char __user *target = (char __user *)dst;
+ u64 d8;
+ u32 d4;
+ u16 d2;
+ u8 d1;
+
+ switch (size) {
+ case 1:
+ memcpy(&d1, buf, 1);
+ if (put_user(d1, target))
+ goto fault;
+ break;
+ case 2:
+ memcpy(&d2, buf, 2);
+ if (put_user(d2, target))
+ goto fault;
+ break;
+ case 4:
+ memcpy(&d4, buf, 4);
+ if (put_user(d4, target))
+ goto fault;
+ break;
+ case 8:
+ memcpy(&d8, buf, 8);
+ if (put_user(d8, target))
+ goto fault;
+ break;
+ default:
+ WARN_ONCE(1, "%s: Invalid size: %zu\n", __func__, size);
+ return ES_UNSUPPORTED;
+ }
+
+ return ES_OK;
+
+fault:
+ if (user_mode(ctxt->regs))
+ error_code |= X86_PF_USER;
+
+ ctxt->fi.vector = X86_TRAP_PF;
+ ctxt->fi.error_code = error_code;
+ ctxt->fi.cr2 = (unsigned long)dst;
+
+ return ES_EXCEPTION;
+}
+
+static enum es_result vc_read_mem(struct es_em_ctxt *ctxt,
+ char *src, char *buf, size_t size)
+{
+ unsigned long error_code = X86_PF_PROT;
+ char __user *s = (char __user *)src;
+ u64 d8;
+ u32 d4;
+ u16 d2;
+ u8 d1;
+
+ switch (size) {
+ case 1:
+ if (get_user(d1, s))
+ goto fault;
+ memcpy(buf, &d1, 1);
+ break;
+ case 2:
+ if (get_user(d2, s))
+ goto fault;
+ memcpy(buf, &d2, 2);
+ break;
+ case 4:
+ if (get_user(d4, s))
+ goto fault;
+ memcpy(buf, &d4, 4);
+ break;
+ case 8:
+ if (get_user(d8, s))
+ goto fault;
+ memcpy(buf, &d8, 8);
+ break;
+ default:
+ WARN_ONCE(1, "%s: Invalid size: %zu\n", __func__, size);
+ return ES_UNSUPPORTED;
+ }
+
+ return ES_OK;
+
+fault:
+ if (user_mode(ctxt->regs))
+ error_code |= X86_PF_USER;
+
+ ctxt->fi.vector = X86_TRAP_PF;
+ ctxt->fi.error_code = error_code;
+ ctxt->fi.cr2 = (unsigned long)src;
+
+ return ES_EXCEPTION;
+}
+
+/* Include code shared with pre-decompression boot stage */
+#include "sev-es-shared.c"
--
2.28.0
^ permalink raw reply related [flat|nested] 264+ messages in thread
* [PATCH v7 38/72] x86/sev-es: Compile early handler code into kernel image
@ 2020-09-07 13:15 ` Joerg Roedel
0 siblings, 0 replies; 264+ messages in thread
From: Joerg Roedel @ 2020-09-07 13:15 UTC (permalink / raw)
To: x86
Cc: Juergen Gross, Tom Lendacky, Joerg Roedel, Mike Stunes,
Kees Cook, kvm, Peter Zijlstra, Cfir Cohen, Joerg Roedel,
Dave Hansen, linux-kernel, Sean Christopherson, virtualization,
Martin Radev, Masami Hiramatsu, Andy Lutomirski, hpa,
Erdem Aktas, David Rientjes, Dan Williams, Jiri Slaby
From: Joerg Roedel <jroedel@suse.de>
Setup sev-es.c and include the code from the
pre-decompression stage to also build it into the image of the running
kernel. Temporarily add __maybe_unused annotations to avoid build
warnings until the functions get used.
Signed-off-by: Joerg Roedel <jroedel@suse.de>
---
arch/x86/kernel/Makefile | 1 +
arch/x86/kernel/sev-es-shared.c | 21 ++--
arch/x86/kernel/sev-es.c | 163 ++++++++++++++++++++++++++++++++
3 files changed, 175 insertions(+), 10 deletions(-)
create mode 100644 arch/x86/kernel/sev-es.c
diff --git a/arch/x86/kernel/Makefile b/arch/x86/kernel/Makefile
index de09af019e23..3bcdd8d2bbdd 100644
--- a/arch/x86/kernel/Makefile
+++ b/arch/x86/kernel/Makefile
@@ -146,6 +146,7 @@ obj-$(CONFIG_UNWINDER_ORC) += unwind_orc.o
obj-$(CONFIG_UNWINDER_FRAME_POINTER) += unwind_frame.o
obj-$(CONFIG_UNWINDER_GUESS) += unwind_guess.o
+obj-$(CONFIG_AMD_MEM_ENCRYPT) += sev-es.o
###
# 64 bit specific files
ifeq ($(CONFIG_X86_64),y)
diff --git a/arch/x86/kernel/sev-es-shared.c b/arch/x86/kernel/sev-es-shared.c
index a6b41910b8ab..18619279a46f 100644
--- a/arch/x86/kernel/sev-es-shared.c
+++ b/arch/x86/kernel/sev-es-shared.c
@@ -9,7 +9,7 @@
* and is included directly into both code-bases.
*/
-static void sev_es_terminate(unsigned int reason)
+static void __maybe_unused sev_es_terminate(unsigned int reason)
{
u64 val = GHCB_SEV_TERMINATE;
@@ -27,7 +27,7 @@ static void sev_es_terminate(unsigned int reason)
asm volatile("hlt\n" : : : "memory");
}
-static bool sev_es_negotiate_protocol(void)
+static bool __maybe_unused sev_es_negotiate_protocol(void)
{
u64 val;
@@ -46,7 +46,7 @@ static bool sev_es_negotiate_protocol(void)
return true;
}
-static void vc_ghcb_invalidate(struct ghcb *ghcb)
+static void __maybe_unused vc_ghcb_invalidate(struct ghcb *ghcb)
{
memset(ghcb->save.valid_bitmap, 0, sizeof(ghcb->save.valid_bitmap));
}
@@ -58,9 +58,9 @@ static bool vc_decoding_needed(unsigned long exit_code)
exit_code <= SVM_EXIT_LAST_EXCP);
}
-static enum es_result vc_init_em_ctxt(struct es_em_ctxt *ctxt,
- struct pt_regs *regs,
- unsigned long exit_code)
+static enum es_result __maybe_unused vc_init_em_ctxt(struct es_em_ctxt *ctxt,
+ struct pt_regs *regs,
+ unsigned long exit_code)
{
enum es_result ret = ES_OK;
@@ -73,7 +73,7 @@ static enum es_result vc_init_em_ctxt(struct es_em_ctxt *ctxt,
return ret;
}
-static void vc_finish_insn(struct es_em_ctxt *ctxt)
+static void __maybe_unused vc_finish_insn(struct es_em_ctxt *ctxt)
{
ctxt->regs->ip += ctxt->insn.length;
}
@@ -325,7 +325,8 @@ static enum es_result vc_ioio_exitinfo(struct es_em_ctxt *ctxt, u64 *exitinfo)
return ES_OK;
}
-static enum es_result vc_handle_ioio(struct ghcb *ghcb, struct es_em_ctxt *ctxt)
+static enum es_result __maybe_unused
+vc_handle_ioio(struct ghcb *ghcb, struct es_em_ctxt *ctxt)
{
struct pt_regs *regs = ctxt->regs;
u64 exit_info_1, exit_info_2;
@@ -433,8 +434,8 @@ static enum es_result vc_handle_ioio(struct ghcb *ghcb, struct es_em_ctxt *ctxt)
return ret;
}
-static enum es_result vc_handle_cpuid(struct ghcb *ghcb,
- struct es_em_ctxt *ctxt)
+static enum es_result __maybe_unused vc_handle_cpuid(struct ghcb *ghcb,
+ struct es_em_ctxt *ctxt)
{
struct pt_regs *regs = ctxt->regs;
u32 cr4 = native_read_cr4();
diff --git a/arch/x86/kernel/sev-es.c b/arch/x86/kernel/sev-es.c
new file mode 100644
index 000000000000..0b698b653c0b
--- /dev/null
+++ b/arch/x86/kernel/sev-es.c
@@ -0,0 +1,163 @@
+// SPDX-License-Identifier: GPL-2.0-only
+/*
+ * AMD Memory Encryption Support
+ *
+ * Copyright (C) 2019 SUSE
+ *
+ * Author: Joerg Roedel <jroedel@suse.de>
+ */
+
+#include <linux/kernel.h>
+#include <linux/mm.h>
+
+#include <asm/sev-es.h>
+#include <asm/insn-eval.h>
+#include <asm/fpu/internal.h>
+#include <asm/processor.h>
+#include <asm/trap_pf.h>
+#include <asm/trapnr.h>
+#include <asm/svm.h>
+
+static inline u64 sev_es_rd_ghcb_msr(void)
+{
+ return native_read_msr(MSR_AMD64_SEV_ES_GHCB);
+}
+
+static inline void sev_es_wr_ghcb_msr(u64 val)
+{
+ u32 low, high;
+
+ low = (u32)(val);
+ high = (u32)(val >> 32);
+
+ native_write_msr(MSR_AMD64_SEV_ES_GHCB, low, high);
+}
+
+static int vc_fetch_insn_kernel(struct es_em_ctxt *ctxt,
+ unsigned char *buffer)
+{
+ return copy_from_kernel_nofault(buffer, (unsigned char *)ctxt->regs->ip, MAX_INSN_SIZE);
+}
+
+static enum es_result vc_decode_insn(struct es_em_ctxt *ctxt)
+{
+ char buffer[MAX_INSN_SIZE];
+ enum es_result ret;
+ int res;
+
+ res = vc_fetch_insn_kernel(ctxt, buffer);
+ if (unlikely(res == -EFAULT)) {
+ ctxt->fi.vector = X86_TRAP_PF;
+ ctxt->fi.error_code = 0;
+ ctxt->fi.cr2 = ctxt->regs->ip;
+ return ES_EXCEPTION;
+ }
+
+ insn_init(&ctxt->insn, buffer, MAX_INSN_SIZE - res, 1);
+ insn_get_length(&ctxt->insn);
+
+ ret = ctxt->insn.immediate.got ? ES_OK : ES_DECODE_FAILED;
+
+ return ret;
+}
+
+static enum es_result vc_write_mem(struct es_em_ctxt *ctxt,
+ char *dst, char *buf, size_t size)
+{
+ unsigned long error_code = X86_PF_PROT | X86_PF_WRITE;
+ char __user *target = (char __user *)dst;
+ u64 d8;
+ u32 d4;
+ u16 d2;
+ u8 d1;
+
+ switch (size) {
+ case 1:
+ memcpy(&d1, buf, 1);
+ if (put_user(d1, target))
+ goto fault;
+ break;
+ case 2:
+ memcpy(&d2, buf, 2);
+ if (put_user(d2, target))
+ goto fault;
+ break;
+ case 4:
+ memcpy(&d4, buf, 4);
+ if (put_user(d4, target))
+ goto fault;
+ break;
+ case 8:
+ memcpy(&d8, buf, 8);
+ if (put_user(d8, target))
+ goto fault;
+ break;
+ default:
+ WARN_ONCE(1, "%s: Invalid size: %zu\n", __func__, size);
+ return ES_UNSUPPORTED;
+ }
+
+ return ES_OK;
+
+fault:
+ if (user_mode(ctxt->regs))
+ error_code |= X86_PF_USER;
+
+ ctxt->fi.vector = X86_TRAP_PF;
+ ctxt->fi.error_code = error_code;
+ ctxt->fi.cr2 = (unsigned long)dst;
+
+ return ES_EXCEPTION;
+}
+
+static enum es_result vc_read_mem(struct es_em_ctxt *ctxt,
+ char *src, char *buf, size_t size)
+{
+ unsigned long error_code = X86_PF_PROT;
+ char __user *s = (char __user *)src;
+ u64 d8;
+ u32 d4;
+ u16 d2;
+ u8 d1;
+
+ switch (size) {
+ case 1:
+ if (get_user(d1, s))
+ goto fault;
+ memcpy(buf, &d1, 1);
+ break;
+ case 2:
+ if (get_user(d2, s))
+ goto fault;
+ memcpy(buf, &d2, 2);
+ break;
+ case 4:
+ if (get_user(d4, s))
+ goto fault;
+ memcpy(buf, &d4, 4);
+ break;
+ case 8:
+ if (get_user(d8, s))
+ goto fault;
+ memcpy(buf, &d8, 8);
+ break;
+ default:
+ WARN_ONCE(1, "%s: Invalid size: %zu\n", __func__, size);
+ return ES_UNSUPPORTED;
+ }
+
+ return ES_OK;
+
+fault:
+ if (user_mode(ctxt->regs))
+ error_code |= X86_PF_USER;
+
+ ctxt->fi.vector = X86_TRAP_PF;
+ ctxt->fi.error_code = error_code;
+ ctxt->fi.cr2 = (unsigned long)src;
+
+ return ES_EXCEPTION;
+}
+
+/* Include code shared with pre-decompression boot stage */
+#include "sev-es-shared.c"
--
2.28.0
_______________________________________________
Virtualization mailing list
Virtualization@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/virtualization
^ permalink raw reply related [flat|nested] 264+ messages in thread
* [PATCH v7 39/72] x86/sev-es: Setup early #VC handler
2020-09-07 13:15 ` Joerg Roedel
@ 2020-09-07 13:15 ` Joerg Roedel
-1 siblings, 0 replies; 264+ messages in thread
From: Joerg Roedel @ 2020-09-07 13:15 UTC (permalink / raw)
To: x86
Cc: Joerg Roedel, Joerg Roedel, hpa, Andy Lutomirski, Dave Hansen,
Peter Zijlstra, Jiri Slaby, Dan Williams, Tom Lendacky,
Juergen Gross, Kees Cook, David Rientjes, Cfir Cohen,
Erdem Aktas, Masami Hiramatsu, Mike Stunes, Sean Christopherson,
Martin Radev, linux-kernel, kvm, virtualization
From: Joerg Roedel <jroedel@suse.de>
Setup an early handler for #VC exceptions. There is no GHCB mapped
yet, so just re-use the vc_no_ghcb_handler. It can only handle CPUID
exit-codes, but that should be enough to get the kernel through
verify_cpu() and __startup_64() until it runs on virtual addresses.
Signed-off-by: Joerg Roedel <jroedel@suse.de>
---
arch/x86/include/asm/sev-es.h | 3 +++
arch/x86/kernel/head64.c | 20 ++++++++++++++++++++
arch/x86/kernel/head_64.S | 27 +++++++++++++++++++++++++++
3 files changed, 50 insertions(+)
diff --git a/arch/x86/include/asm/sev-es.h b/arch/x86/include/asm/sev-es.h
index 6dc52440c4b4..7175d432ebfe 100644
--- a/arch/x86/include/asm/sev-es.h
+++ b/arch/x86/include/asm/sev-es.h
@@ -73,4 +73,7 @@ static inline u64 lower_bits(u64 val, unsigned int bits)
return (val & mask);
}
+/* Early IDT entry points for #VC handler */
+extern void vc_no_ghcb(void);
+
#endif
diff --git a/arch/x86/kernel/head64.c b/arch/x86/kernel/head64.c
index 4282dac694c3..621b5e8aef4c 100644
--- a/arch/x86/kernel/head64.c
+++ b/arch/x86/kernel/head64.c
@@ -40,6 +40,7 @@
#include <asm/desc.h>
#include <asm/extable.h>
#include <asm/trapnr.h>
+#include <asm/sev-es.h>
/*
* Manage page tables very early on.
@@ -540,10 +541,29 @@ static struct desc_ptr bringup_idt_descr = {
.address = 0, /* Set at runtime */
};
+#ifdef CONFIG_AMD_MEM_ENCRYPT
+static void set_bringup_idt_handler(int n, void *handler)
+{
+ struct idt_data data;
+ gate_desc desc;
+
+ init_idt_data(&data, n, handler);
+ idt_init_desc(&desc, &data);
+ native_write_idt_entry(bringup_idt_table, n, &desc);
+}
+#endif
+
/* This runs while still in the direct mapping */
static void startup_64_load_idt(unsigned long physbase)
{
struct desc_ptr *desc = fixup_pointer(&bringup_idt_descr, physbase);
+ void __maybe_unused *handler;
+
+#ifdef CONFIG_AMD_MEM_ENCRYPT
+ /* VMM Communication Exception */
+ handler = fixup_pointer(vc_no_ghcb, physbase);
+ set_bringup_idt_handler(X86_TRAP_VC, handler);
+#endif
desc->address = (unsigned long)fixup_pointer(bringup_idt_table, physbase);
native_load_idt(desc);
diff --git a/arch/x86/kernel/head_64.S b/arch/x86/kernel/head_64.S
index 3b40ec44a67d..2e5aa03b4321 100644
--- a/arch/x86/kernel/head_64.S
+++ b/arch/x86/kernel/head_64.S
@@ -348,6 +348,33 @@ SYM_CODE_START_LOCAL(early_idt_handler_common)
jmp restore_regs_and_return_to_kernel
SYM_CODE_END(early_idt_handler_common)
+#ifdef CONFIG_AMD_MEM_ENCRYPT
+/*
+ * VC Exception handler used during very early boot. The
+ * early_idt_handler_array can't be used because it returns via the
+ * paravirtualized INTERRUPT_RETURN and pv-ops don't work that early.
+ */
+SYM_CODE_START_NOALIGN(vc_no_ghcb)
+ UNWIND_HINT_IRET_REGS offset=8
+
+ /* Build pt_regs */
+ PUSH_AND_CLEAR_REGS
+
+ /* Call C handler */
+ movq %rsp, %rdi
+ movq ORIG_RAX(%rsp), %rsi
+ call do_vc_no_ghcb
+
+ /* Unwind pt_regs */
+ POP_REGS
+
+ /* Remove Error Code */
+ addq $8, %rsp
+
+ /* Pure iret required here - don't use INTERRUPT_RETURN */
+ iretq
+SYM_CODE_END(vc_no_ghcb)
+#endif
#define SYM_DATA_START_PAGE_ALIGNED(name) \
SYM_START(name, SYM_L_GLOBAL, .balign PAGE_SIZE)
--
2.28.0
^ permalink raw reply related [flat|nested] 264+ messages in thread
* [PATCH v7 39/72] x86/sev-es: Setup early #VC handler
@ 2020-09-07 13:15 ` Joerg Roedel
0 siblings, 0 replies; 264+ messages in thread
From: Joerg Roedel @ 2020-09-07 13:15 UTC (permalink / raw)
To: x86
Cc: Juergen Gross, Tom Lendacky, Joerg Roedel, Mike Stunes,
Kees Cook, kvm, Peter Zijlstra, Cfir Cohen, Joerg Roedel,
Dave Hansen, linux-kernel, Sean Christopherson, virtualization,
Martin Radev, Masami Hiramatsu, Andy Lutomirski, hpa,
Erdem Aktas, David Rientjes, Dan Williams, Jiri Slaby
From: Joerg Roedel <jroedel@suse.de>
Setup an early handler for #VC exceptions. There is no GHCB mapped
yet, so just re-use the vc_no_ghcb_handler. It can only handle CPUID
exit-codes, but that should be enough to get the kernel through
verify_cpu() and __startup_64() until it runs on virtual addresses.
Signed-off-by: Joerg Roedel <jroedel@suse.de>
---
arch/x86/include/asm/sev-es.h | 3 +++
arch/x86/kernel/head64.c | 20 ++++++++++++++++++++
arch/x86/kernel/head_64.S | 27 +++++++++++++++++++++++++++
3 files changed, 50 insertions(+)
diff --git a/arch/x86/include/asm/sev-es.h b/arch/x86/include/asm/sev-es.h
index 6dc52440c4b4..7175d432ebfe 100644
--- a/arch/x86/include/asm/sev-es.h
+++ b/arch/x86/include/asm/sev-es.h
@@ -73,4 +73,7 @@ static inline u64 lower_bits(u64 val, unsigned int bits)
return (val & mask);
}
+/* Early IDT entry points for #VC handler */
+extern void vc_no_ghcb(void);
+
#endif
diff --git a/arch/x86/kernel/head64.c b/arch/x86/kernel/head64.c
index 4282dac694c3..621b5e8aef4c 100644
--- a/arch/x86/kernel/head64.c
+++ b/arch/x86/kernel/head64.c
@@ -40,6 +40,7 @@
#include <asm/desc.h>
#include <asm/extable.h>
#include <asm/trapnr.h>
+#include <asm/sev-es.h>
/*
* Manage page tables very early on.
@@ -540,10 +541,29 @@ static struct desc_ptr bringup_idt_descr = {
.address = 0, /* Set at runtime */
};
+#ifdef CONFIG_AMD_MEM_ENCRYPT
+static void set_bringup_idt_handler(int n, void *handler)
+{
+ struct idt_data data;
+ gate_desc desc;
+
+ init_idt_data(&data, n, handler);
+ idt_init_desc(&desc, &data);
+ native_write_idt_entry(bringup_idt_table, n, &desc);
+}
+#endif
+
/* This runs while still in the direct mapping */
static void startup_64_load_idt(unsigned long physbase)
{
struct desc_ptr *desc = fixup_pointer(&bringup_idt_descr, physbase);
+ void __maybe_unused *handler;
+
+#ifdef CONFIG_AMD_MEM_ENCRYPT
+ /* VMM Communication Exception */
+ handler = fixup_pointer(vc_no_ghcb, physbase);
+ set_bringup_idt_handler(X86_TRAP_VC, handler);
+#endif
desc->address = (unsigned long)fixup_pointer(bringup_idt_table, physbase);
native_load_idt(desc);
diff --git a/arch/x86/kernel/head_64.S b/arch/x86/kernel/head_64.S
index 3b40ec44a67d..2e5aa03b4321 100644
--- a/arch/x86/kernel/head_64.S
+++ b/arch/x86/kernel/head_64.S
@@ -348,6 +348,33 @@ SYM_CODE_START_LOCAL(early_idt_handler_common)
jmp restore_regs_and_return_to_kernel
SYM_CODE_END(early_idt_handler_common)
+#ifdef CONFIG_AMD_MEM_ENCRYPT
+/*
+ * VC Exception handler used during very early boot. The
+ * early_idt_handler_array can't be used because it returns via the
+ * paravirtualized INTERRUPT_RETURN and pv-ops don't work that early.
+ */
+SYM_CODE_START_NOALIGN(vc_no_ghcb)
+ UNWIND_HINT_IRET_REGS offset=8
+
+ /* Build pt_regs */
+ PUSH_AND_CLEAR_REGS
+
+ /* Call C handler */
+ movq %rsp, %rdi
+ movq ORIG_RAX(%rsp), %rsi
+ call do_vc_no_ghcb
+
+ /* Unwind pt_regs */
+ POP_REGS
+
+ /* Remove Error Code */
+ addq $8, %rsp
+
+ /* Pure iret required here - don't use INTERRUPT_RETURN */
+ iretq
+SYM_CODE_END(vc_no_ghcb)
+#endif
#define SYM_DATA_START_PAGE_ALIGNED(name) \
SYM_START(name, SYM_L_GLOBAL, .balign PAGE_SIZE)
--
2.28.0
_______________________________________________
Virtualization mailing list
Virtualization@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/virtualization
^ permalink raw reply related [flat|nested] 264+ messages in thread
* [PATCH v7 40/72] x86/sev-es: Setup GHCB based boot #VC handler
2020-09-07 13:15 ` Joerg Roedel
@ 2020-09-07 13:15 ` Joerg Roedel
-1 siblings, 0 replies; 264+ messages in thread
From: Joerg Roedel @ 2020-09-07 13:15 UTC (permalink / raw)
To: x86
Cc: Joerg Roedel, Joerg Roedel, hpa, Andy Lutomirski, Dave Hansen,
Peter Zijlstra, Jiri Slaby, Dan Williams, Tom Lendacky,
Juergen Gross, Kees Cook, David Rientjes, Cfir Cohen,
Erdem Aktas, Masami Hiramatsu, Mike Stunes, Sean Christopherson,
Martin Radev, linux-kernel, kvm, virtualization
From: Joerg Roedel <jroedel@suse.de>
Add the infrastructure to handle #VC exceptions when the kernel runs
on virtual addresses and has a GHCB mapped. This handler will be used
until the runtime #VC handler takes over.
Since the handler runs very early, disable instrumentation for sev-es.c.
Signed-off-by: Joerg Roedel <jroedel@suse.de>
---
arch/x86/include/asm/realmode.h | 3 +
arch/x86/include/asm/segment.h | 2 +-
arch/x86/include/asm/sev-es.h | 2 +
arch/x86/kernel/Makefile | 2 +
arch/x86/kernel/head64.c | 11 +++
arch/x86/kernel/head_64.S | 34 ++++++++++
arch/x86/kernel/sev-es-shared.c | 14 ++--
arch/x86/kernel/sev-es.c | 116 ++++++++++++++++++++++++++++++++
arch/x86/mm/extable.c | 1 +
9 files changed, 177 insertions(+), 8 deletions(-)
diff --git a/arch/x86/include/asm/realmode.h b/arch/x86/include/asm/realmode.h
index b35030eeec36..96118fb041b8 100644
--- a/arch/x86/include/asm/realmode.h
+++ b/arch/x86/include/asm/realmode.h
@@ -57,6 +57,9 @@ extern unsigned char real_mode_blob_end[];
extern unsigned long initial_code;
extern unsigned long initial_gs;
extern unsigned long initial_stack;
+#ifdef CONFIG_AMD_MEM_ENCRYPT
+extern unsigned long initial_vc_handler;
+#endif
extern unsigned char real_mode_blob[];
extern unsigned char real_mode_relocs[];
diff --git a/arch/x86/include/asm/segment.h b/arch/x86/include/asm/segment.h
index 517920928989..7fdd4facfce7 100644
--- a/arch/x86/include/asm/segment.h
+++ b/arch/x86/include/asm/segment.h
@@ -226,7 +226,7 @@
#define NUM_EXCEPTION_VECTORS 32
/* Bitmask of exception vectors which push an error code on the stack: */
-#define EXCEPTION_ERRCODE_MASK 0x00027d00
+#define EXCEPTION_ERRCODE_MASK 0x20027d00
#define GDT_SIZE (GDT_ENTRIES*8)
#define GDT_ENTRY_TLS_ENTRIES 3
diff --git a/arch/x86/include/asm/sev-es.h b/arch/x86/include/asm/sev-es.h
index 7175d432ebfe..9fbeedaa66ee 100644
--- a/arch/x86/include/asm/sev-es.h
+++ b/arch/x86/include/asm/sev-es.h
@@ -75,5 +75,7 @@ static inline u64 lower_bits(u64 val, unsigned int bits)
/* Early IDT entry points for #VC handler */
extern void vc_no_ghcb(void);
+extern void vc_boot_ghcb(void);
+extern bool handle_vc_boot_ghcb(struct pt_regs *regs);
#endif
diff --git a/arch/x86/kernel/Makefile b/arch/x86/kernel/Makefile
index 3bcdd8d2bbdd..04ceea8f4a89 100644
--- a/arch/x86/kernel/Makefile
+++ b/arch/x86/kernel/Makefile
@@ -20,6 +20,7 @@ CFLAGS_REMOVE_kvmclock.o = -pg
CFLAGS_REMOVE_ftrace.o = -pg
CFLAGS_REMOVE_early_printk.o = -pg
CFLAGS_REMOVE_head64.o = -pg
+CFLAGS_REMOVE_sev-es.o = -pg
endif
KASAN_SANITIZE_head$(BITS).o := n
@@ -27,6 +28,7 @@ KASAN_SANITIZE_dumpstack.o := n
KASAN_SANITIZE_dumpstack_$(BITS).o := n
KASAN_SANITIZE_stacktrace.o := n
KASAN_SANITIZE_paravirt.o := n
+KASAN_SANITIZE_sev-es.o := n
# With some compiler versions the generated code results in boot hangs, caused
# by several compilation units. To be safe, disable all instrumentation.
diff --git a/arch/x86/kernel/head64.c b/arch/x86/kernel/head64.c
index 621b5e8aef4c..7c6165fd8851 100644
--- a/arch/x86/kernel/head64.c
+++ b/arch/x86/kernel/head64.c
@@ -406,6 +406,12 @@ void __init do_early_exception(struct pt_regs *regs, int trapnr)
early_make_pgtable(native_read_cr2()))
return;
+#ifdef CONFIG_AMD_MEM_ENCRYPT
+ if (trapnr == X86_TRAP_VC &&
+ handle_vc_boot_ghcb(regs))
+ return;
+#endif
+
early_fixup_exception(regs, trapnr);
}
@@ -572,6 +578,11 @@ static void startup_64_load_idt(unsigned long physbase)
/* This is used when running on kernel addresses */
void early_setup_idt(void)
{
+#ifdef CONFIG_AMD_MEM_ENCRYPT
+ /* VMM Communication Exception */
+ set_bringup_idt_handler(X86_TRAP_VC, vc_boot_ghcb);
+#endif
+
bringup_idt_descr.address = (unsigned long)bringup_idt_table;
native_load_idt(&bringup_idt_descr);
}
diff --git a/arch/x86/kernel/head_64.S b/arch/x86/kernel/head_64.S
index 2e5aa03b4321..41057ff79284 100644
--- a/arch/x86/kernel/head_64.S
+++ b/arch/x86/kernel/head_64.S
@@ -279,6 +279,37 @@ SYM_CODE_START(start_cpu0)
movq initial_stack(%rip), %rsp
jmp .Ljump_to_C_code
SYM_CODE_END(start_cpu0)
+#endif
+
+#ifdef CONFIG_AMD_MEM_ENCRYPT
+/*
+ * VC Exception handler used during early boot when running on kernel
+ * addresses, but before the switch to the idt_table can be mase. The
+ * early_idt_handler_array can't be used here because it calls into a lot of
+ * __init code and this handler is also used during CPU offlining/onlining.
+ */
+SYM_CODE_START_NOALIGN(vc_boot_ghcb)
+ UNWIND_HINT_IRET_REGS offset=8
+
+ /* Build pt_regs */
+ PUSH_AND_CLEAR_REGS
+
+ /* Call C handler */
+ movq %rsp, %rdi
+ movq ORIG_RAX(%rsp), %rsi
+ movq initial_vc_handler(%rip), %rax
+ ANNOTATE_RETPOLINE_SAFE
+ call *%rax
+
+ /* Unwind pt_regs */
+ POP_REGS
+
+ /* Remove Error Code */
+ addq $8, %rsp
+
+ /* Pure iret required here - don't use INTERRUPT_RETURN */
+ iretq
+SYM_CODE_END(vc_boot_ghcb)
#endif
/* Both SMP bootup and ACPI suspend change these variables */
@@ -286,6 +317,9 @@ SYM_CODE_END(start_cpu0)
.balign 8
SYM_DATA(initial_code, .quad x86_64_start_kernel)
SYM_DATA(initial_gs, .quad INIT_PER_CPU_VAR(fixed_percpu_data))
+#ifdef CONFIG_AMD_MEM_ENCRYPT
+SYM_DATA(initial_vc_handler, .quad handle_vc_boot_ghcb)
+#endif
/*
* The SIZEOF_PTREGS gap is a convention which helps the in-kernel unwinder
diff --git a/arch/x86/kernel/sev-es-shared.c b/arch/x86/kernel/sev-es-shared.c
index 18619279a46f..aa77f2eb8d88 100644
--- a/arch/x86/kernel/sev-es-shared.c
+++ b/arch/x86/kernel/sev-es-shared.c
@@ -9,7 +9,7 @@
* and is included directly into both code-bases.
*/
-static void __maybe_unused sev_es_terminate(unsigned int reason)
+static void sev_es_terminate(unsigned int reason)
{
u64 val = GHCB_SEV_TERMINATE;
@@ -27,7 +27,7 @@ static void __maybe_unused sev_es_terminate(unsigned int reason)
asm volatile("hlt\n" : : : "memory");
}
-static bool __maybe_unused sev_es_negotiate_protocol(void)
+static bool sev_es_negotiate_protocol(void)
{
u64 val;
@@ -46,7 +46,7 @@ static bool __maybe_unused sev_es_negotiate_protocol(void)
return true;
}
-static void __maybe_unused vc_ghcb_invalidate(struct ghcb *ghcb)
+static void vc_ghcb_invalidate(struct ghcb *ghcb)
{
memset(ghcb->save.valid_bitmap, 0, sizeof(ghcb->save.valid_bitmap));
}
@@ -58,9 +58,9 @@ static bool vc_decoding_needed(unsigned long exit_code)
exit_code <= SVM_EXIT_LAST_EXCP);
}
-static enum es_result __maybe_unused vc_init_em_ctxt(struct es_em_ctxt *ctxt,
- struct pt_regs *regs,
- unsigned long exit_code)
+static enum es_result vc_init_em_ctxt(struct es_em_ctxt *ctxt,
+ struct pt_regs *regs,
+ unsigned long exit_code)
{
enum es_result ret = ES_OK;
@@ -73,7 +73,7 @@ static enum es_result __maybe_unused vc_init_em_ctxt(struct es_em_ctxt *ctxt,
return ret;
}
-static void __maybe_unused vc_finish_insn(struct es_em_ctxt *ctxt)
+static void vc_finish_insn(struct es_em_ctxt *ctxt)
{
ctxt->regs->ip += ctxt->insn.length;
}
diff --git a/arch/x86/kernel/sev-es.c b/arch/x86/kernel/sev-es.c
index 0b698b653c0b..bb3e702a71eb 100644
--- a/arch/x86/kernel/sev-es.c
+++ b/arch/x86/kernel/sev-es.c
@@ -7,7 +7,9 @@
* Author: Joerg Roedel <jroedel@suse.de>
*/
+#include <linux/sched/debug.h> /* For show_regs() */
#include <linux/kernel.h>
+#include <linux/printk.h>
#include <linux/mm.h>
#include <asm/sev-es.h>
@@ -18,6 +20,18 @@
#include <asm/trapnr.h>
#include <asm/svm.h>
+/* For early boot hypervisor communication in SEV-ES enabled guests */
+static struct ghcb boot_ghcb_page __bss_decrypted __aligned(PAGE_SIZE);
+
+/*
+ * Needs to be in the .data section because we need it NULL before bss is
+ * cleared
+ */
+static struct ghcb __initdata *boot_ghcb;
+
+/* Needed in vc_early_forward_exception */
+void do_early_exception(struct pt_regs *regs, int trapnr);
+
static inline u64 sev_es_rd_ghcb_msr(void)
{
return native_read_msr(MSR_AMD64_SEV_ES_GHCB);
@@ -161,3 +175,105 @@ static enum es_result vc_read_mem(struct es_em_ctxt *ctxt,
/* Include code shared with pre-decompression boot stage */
#include "sev-es-shared.c"
+
+/*
+ * This function runs on the first #VC exception after the kernel
+ * switched to virtual addresses.
+ */
+static bool __init sev_es_setup_ghcb(void)
+{
+ /* First make sure the hypervisor talks a supported protocol. */
+ if (!sev_es_negotiate_protocol())
+ return false;
+
+ /*
+ * Clear the boot_ghcb. The first exception comes in before the bss
+ * section is cleared.
+ */
+ memset(&boot_ghcb_page, 0, PAGE_SIZE);
+
+ /* Alright - Make the boot-ghcb public */
+ boot_ghcb = &boot_ghcb_page;
+
+ return true;
+}
+
+static void __init vc_early_forward_exception(struct es_em_ctxt *ctxt)
+{
+ int trapnr = ctxt->fi.vector;
+
+ if (trapnr == X86_TRAP_PF)
+ native_write_cr2(ctxt->fi.cr2);
+
+ ctxt->regs->orig_ax = ctxt->fi.error_code;
+ do_early_exception(ctxt->regs, trapnr);
+}
+
+static enum es_result vc_handle_exitcode(struct es_em_ctxt *ctxt,
+ struct ghcb *ghcb,
+ unsigned long exit_code)
+{
+ enum es_result result;
+
+ switch (exit_code) {
+ default:
+ /*
+ * Unexpected #VC exception
+ */
+ result = ES_UNSUPPORTED;
+ }
+
+ return result;
+}
+
+bool __init handle_vc_boot_ghcb(struct pt_regs *regs)
+{
+ unsigned long exit_code = regs->orig_ax;
+ struct es_em_ctxt ctxt;
+ enum es_result result;
+
+ /* Do initial setup or terminate the guest */
+ if (unlikely(boot_ghcb == NULL && !sev_es_setup_ghcb()))
+ sev_es_terminate(GHCB_SEV_ES_REASON_GENERAL_REQUEST);
+
+ vc_ghcb_invalidate(boot_ghcb);
+
+ result = vc_init_em_ctxt(&ctxt, regs, exit_code);
+ if (result == ES_OK)
+ result = vc_handle_exitcode(&ctxt, boot_ghcb, exit_code);
+
+ /* Done - now check the result */
+ switch (result) {
+ case ES_OK:
+ vc_finish_insn(&ctxt);
+ break;
+ case ES_UNSUPPORTED:
+ early_printk("PANIC: Unsupported exit-code 0x%02lx in early #VC exception (IP: 0x%lx)\n",
+ exit_code, regs->ip);
+ goto fail;
+ case ES_VMM_ERROR:
+ early_printk("PANIC: Failure in communication with VMM (exit-code 0x%02lx IP: 0x%lx)\n",
+ exit_code, regs->ip);
+ goto fail;
+ case ES_DECODE_FAILED:
+ early_printk("PANIC: Failed to decode instruction (exit-code 0x%02lx IP: 0x%lx)\n",
+ exit_code, regs->ip);
+ goto fail;
+ case ES_EXCEPTION:
+ vc_early_forward_exception(&ctxt);
+ break;
+ case ES_RETRY:
+ /* Nothing to do */
+ break;
+ default:
+ BUG();
+ }
+
+ return true;
+
+fail:
+ show_regs(regs);
+
+ while (true)
+ halt();
+}
diff --git a/arch/x86/mm/extable.c b/arch/x86/mm/extable.c
index 1d6cb07f4f86..3966749d07ac 100644
--- a/arch/x86/mm/extable.c
+++ b/arch/x86/mm/extable.c
@@ -5,6 +5,7 @@
#include <xen/xen.h>
#include <asm/fpu/internal.h>
+#include <asm/sev-es.h>
#include <asm/traps.h>
#include <asm/kdebug.h>
--
2.28.0
^ permalink raw reply related [flat|nested] 264+ messages in thread
* [PATCH v7 40/72] x86/sev-es: Setup GHCB based boot #VC handler
@ 2020-09-07 13:15 ` Joerg Roedel
0 siblings, 0 replies; 264+ messages in thread
From: Joerg Roedel @ 2020-09-07 13:15 UTC (permalink / raw)
To: x86
Cc: Juergen Gross, Tom Lendacky, Joerg Roedel, Mike Stunes,
Kees Cook, kvm, Peter Zijlstra, Cfir Cohen, Joerg Roedel,
Dave Hansen, linux-kernel, Sean Christopherson, virtualization,
Martin Radev, Masami Hiramatsu, Andy Lutomirski, hpa,
Erdem Aktas, David Rientjes, Dan Williams, Jiri Slaby
From: Joerg Roedel <jroedel@suse.de>
Add the infrastructure to handle #VC exceptions when the kernel runs
on virtual addresses and has a GHCB mapped. This handler will be used
until the runtime #VC handler takes over.
Since the handler runs very early, disable instrumentation for sev-es.c.
Signed-off-by: Joerg Roedel <jroedel@suse.de>
---
arch/x86/include/asm/realmode.h | 3 +
arch/x86/include/asm/segment.h | 2 +-
arch/x86/include/asm/sev-es.h | 2 +
arch/x86/kernel/Makefile | 2 +
arch/x86/kernel/head64.c | 11 +++
arch/x86/kernel/head_64.S | 34 ++++++++++
arch/x86/kernel/sev-es-shared.c | 14 ++--
arch/x86/kernel/sev-es.c | 116 ++++++++++++++++++++++++++++++++
arch/x86/mm/extable.c | 1 +
9 files changed, 177 insertions(+), 8 deletions(-)
diff --git a/arch/x86/include/asm/realmode.h b/arch/x86/include/asm/realmode.h
index b35030eeec36..96118fb041b8 100644
--- a/arch/x86/include/asm/realmode.h
+++ b/arch/x86/include/asm/realmode.h
@@ -57,6 +57,9 @@ extern unsigned char real_mode_blob_end[];
extern unsigned long initial_code;
extern unsigned long initial_gs;
extern unsigned long initial_stack;
+#ifdef CONFIG_AMD_MEM_ENCRYPT
+extern unsigned long initial_vc_handler;
+#endif
extern unsigned char real_mode_blob[];
extern unsigned char real_mode_relocs[];
diff --git a/arch/x86/include/asm/segment.h b/arch/x86/include/asm/segment.h
index 517920928989..7fdd4facfce7 100644
--- a/arch/x86/include/asm/segment.h
+++ b/arch/x86/include/asm/segment.h
@@ -226,7 +226,7 @@
#define NUM_EXCEPTION_VECTORS 32
/* Bitmask of exception vectors which push an error code on the stack: */
-#define EXCEPTION_ERRCODE_MASK 0x00027d00
+#define EXCEPTION_ERRCODE_MASK 0x20027d00
#define GDT_SIZE (GDT_ENTRIES*8)
#define GDT_ENTRY_TLS_ENTRIES 3
diff --git a/arch/x86/include/asm/sev-es.h b/arch/x86/include/asm/sev-es.h
index 7175d432ebfe..9fbeedaa66ee 100644
--- a/arch/x86/include/asm/sev-es.h
+++ b/arch/x86/include/asm/sev-es.h
@@ -75,5 +75,7 @@ static inline u64 lower_bits(u64 val, unsigned int bits)
/* Early IDT entry points for #VC handler */
extern void vc_no_ghcb(void);
+extern void vc_boot_ghcb(void);
+extern bool handle_vc_boot_ghcb(struct pt_regs *regs);
#endif
diff --git a/arch/x86/kernel/Makefile b/arch/x86/kernel/Makefile
index 3bcdd8d2bbdd..04ceea8f4a89 100644
--- a/arch/x86/kernel/Makefile
+++ b/arch/x86/kernel/Makefile
@@ -20,6 +20,7 @@ CFLAGS_REMOVE_kvmclock.o = -pg
CFLAGS_REMOVE_ftrace.o = -pg
CFLAGS_REMOVE_early_printk.o = -pg
CFLAGS_REMOVE_head64.o = -pg
+CFLAGS_REMOVE_sev-es.o = -pg
endif
KASAN_SANITIZE_head$(BITS).o := n
@@ -27,6 +28,7 @@ KASAN_SANITIZE_dumpstack.o := n
KASAN_SANITIZE_dumpstack_$(BITS).o := n
KASAN_SANITIZE_stacktrace.o := n
KASAN_SANITIZE_paravirt.o := n
+KASAN_SANITIZE_sev-es.o := n
# With some compiler versions the generated code results in boot hangs, caused
# by several compilation units. To be safe, disable all instrumentation.
diff --git a/arch/x86/kernel/head64.c b/arch/x86/kernel/head64.c
index 621b5e8aef4c..7c6165fd8851 100644
--- a/arch/x86/kernel/head64.c
+++ b/arch/x86/kernel/head64.c
@@ -406,6 +406,12 @@ void __init do_early_exception(struct pt_regs *regs, int trapnr)
early_make_pgtable(native_read_cr2()))
return;
+#ifdef CONFIG_AMD_MEM_ENCRYPT
+ if (trapnr == X86_TRAP_VC &&
+ handle_vc_boot_ghcb(regs))
+ return;
+#endif
+
early_fixup_exception(regs, trapnr);
}
@@ -572,6 +578,11 @@ static void startup_64_load_idt(unsigned long physbase)
/* This is used when running on kernel addresses */
void early_setup_idt(void)
{
+#ifdef CONFIG_AMD_MEM_ENCRYPT
+ /* VMM Communication Exception */
+ set_bringup_idt_handler(X86_TRAP_VC, vc_boot_ghcb);
+#endif
+
bringup_idt_descr.address = (unsigned long)bringup_idt_table;
native_load_idt(&bringup_idt_descr);
}
diff --git a/arch/x86/kernel/head_64.S b/arch/x86/kernel/head_64.S
index 2e5aa03b4321..41057ff79284 100644
--- a/arch/x86/kernel/head_64.S
+++ b/arch/x86/kernel/head_64.S
@@ -279,6 +279,37 @@ SYM_CODE_START(start_cpu0)
movq initial_stack(%rip), %rsp
jmp .Ljump_to_C_code
SYM_CODE_END(start_cpu0)
+#endif
+
+#ifdef CONFIG_AMD_MEM_ENCRYPT
+/*
+ * VC Exception handler used during early boot when running on kernel
+ * addresses, but before the switch to the idt_table can be mase. The
+ * early_idt_handler_array can't be used here because it calls into a lot of
+ * __init code and this handler is also used during CPU offlining/onlining.
+ */
+SYM_CODE_START_NOALIGN(vc_boot_ghcb)
+ UNWIND_HINT_IRET_REGS offset=8
+
+ /* Build pt_regs */
+ PUSH_AND_CLEAR_REGS
+
+ /* Call C handler */
+ movq %rsp, %rdi
+ movq ORIG_RAX(%rsp), %rsi
+ movq initial_vc_handler(%rip), %rax
+ ANNOTATE_RETPOLINE_SAFE
+ call *%rax
+
+ /* Unwind pt_regs */
+ POP_REGS
+
+ /* Remove Error Code */
+ addq $8, %rsp
+
+ /* Pure iret required here - don't use INTERRUPT_RETURN */
+ iretq
+SYM_CODE_END(vc_boot_ghcb)
#endif
/* Both SMP bootup and ACPI suspend change these variables */
@@ -286,6 +317,9 @@ SYM_CODE_END(start_cpu0)
.balign 8
SYM_DATA(initial_code, .quad x86_64_start_kernel)
SYM_DATA(initial_gs, .quad INIT_PER_CPU_VAR(fixed_percpu_data))
+#ifdef CONFIG_AMD_MEM_ENCRYPT
+SYM_DATA(initial_vc_handler, .quad handle_vc_boot_ghcb)
+#endif
/*
* The SIZEOF_PTREGS gap is a convention which helps the in-kernel unwinder
diff --git a/arch/x86/kernel/sev-es-shared.c b/arch/x86/kernel/sev-es-shared.c
index 18619279a46f..aa77f2eb8d88 100644
--- a/arch/x86/kernel/sev-es-shared.c
+++ b/arch/x86/kernel/sev-es-shared.c
@@ -9,7 +9,7 @@
* and is included directly into both code-bases.
*/
-static void __maybe_unused sev_es_terminate(unsigned int reason)
+static void sev_es_terminate(unsigned int reason)
{
u64 val = GHCB_SEV_TERMINATE;
@@ -27,7 +27,7 @@ static void __maybe_unused sev_es_terminate(unsigned int reason)
asm volatile("hlt\n" : : : "memory");
}
-static bool __maybe_unused sev_es_negotiate_protocol(void)
+static bool sev_es_negotiate_protocol(void)
{
u64 val;
@@ -46,7 +46,7 @@ static bool __maybe_unused sev_es_negotiate_protocol(void)
return true;
}
-static void __maybe_unused vc_ghcb_invalidate(struct ghcb *ghcb)
+static void vc_ghcb_invalidate(struct ghcb *ghcb)
{
memset(ghcb->save.valid_bitmap, 0, sizeof(ghcb->save.valid_bitmap));
}
@@ -58,9 +58,9 @@ static bool vc_decoding_needed(unsigned long exit_code)
exit_code <= SVM_EXIT_LAST_EXCP);
}
-static enum es_result __maybe_unused vc_init_em_ctxt(struct es_em_ctxt *ctxt,
- struct pt_regs *regs,
- unsigned long exit_code)
+static enum es_result vc_init_em_ctxt(struct es_em_ctxt *ctxt,
+ struct pt_regs *regs,
+ unsigned long exit_code)
{
enum es_result ret = ES_OK;
@@ -73,7 +73,7 @@ static enum es_result __maybe_unused vc_init_em_ctxt(struct es_em_ctxt *ctxt,
return ret;
}
-static void __maybe_unused vc_finish_insn(struct es_em_ctxt *ctxt)
+static void vc_finish_insn(struct es_em_ctxt *ctxt)
{
ctxt->regs->ip += ctxt->insn.length;
}
diff --git a/arch/x86/kernel/sev-es.c b/arch/x86/kernel/sev-es.c
index 0b698b653c0b..bb3e702a71eb 100644
--- a/arch/x86/kernel/sev-es.c
+++ b/arch/x86/kernel/sev-es.c
@@ -7,7 +7,9 @@
* Author: Joerg Roedel <jroedel@suse.de>
*/
+#include <linux/sched/debug.h> /* For show_regs() */
#include <linux/kernel.h>
+#include <linux/printk.h>
#include <linux/mm.h>
#include <asm/sev-es.h>
@@ -18,6 +20,18 @@
#include <asm/trapnr.h>
#include <asm/svm.h>
+/* For early boot hypervisor communication in SEV-ES enabled guests */
+static struct ghcb boot_ghcb_page __bss_decrypted __aligned(PAGE_SIZE);
+
+/*
+ * Needs to be in the .data section because we need it NULL before bss is
+ * cleared
+ */
+static struct ghcb __initdata *boot_ghcb;
+
+/* Needed in vc_early_forward_exception */
+void do_early_exception(struct pt_regs *regs, int trapnr);
+
static inline u64 sev_es_rd_ghcb_msr(void)
{
return native_read_msr(MSR_AMD64_SEV_ES_GHCB);
@@ -161,3 +175,105 @@ static enum es_result vc_read_mem(struct es_em_ctxt *ctxt,
/* Include code shared with pre-decompression boot stage */
#include "sev-es-shared.c"
+
+/*
+ * This function runs on the first #VC exception after the kernel
+ * switched to virtual addresses.
+ */
+static bool __init sev_es_setup_ghcb(void)
+{
+ /* First make sure the hypervisor talks a supported protocol. */
+ if (!sev_es_negotiate_protocol())
+ return false;
+
+ /*
+ * Clear the boot_ghcb. The first exception comes in before the bss
+ * section is cleared.
+ */
+ memset(&boot_ghcb_page, 0, PAGE_SIZE);
+
+ /* Alright - Make the boot-ghcb public */
+ boot_ghcb = &boot_ghcb_page;
+
+ return true;
+}
+
+static void __init vc_early_forward_exception(struct es_em_ctxt *ctxt)
+{
+ int trapnr = ctxt->fi.vector;
+
+ if (trapnr == X86_TRAP_PF)
+ native_write_cr2(ctxt->fi.cr2);
+
+ ctxt->regs->orig_ax = ctxt->fi.error_code;
+ do_early_exception(ctxt->regs, trapnr);
+}
+
+static enum es_result vc_handle_exitcode(struct es_em_ctxt *ctxt,
+ struct ghcb *ghcb,
+ unsigned long exit_code)
+{
+ enum es_result result;
+
+ switch (exit_code) {
+ default:
+ /*
+ * Unexpected #VC exception
+ */
+ result = ES_UNSUPPORTED;
+ }
+
+ return result;
+}
+
+bool __init handle_vc_boot_ghcb(struct pt_regs *regs)
+{
+ unsigned long exit_code = regs->orig_ax;
+ struct es_em_ctxt ctxt;
+ enum es_result result;
+
+ /* Do initial setup or terminate the guest */
+ if (unlikely(boot_ghcb == NULL && !sev_es_setup_ghcb()))
+ sev_es_terminate(GHCB_SEV_ES_REASON_GENERAL_REQUEST);
+
+ vc_ghcb_invalidate(boot_ghcb);
+
+ result = vc_init_em_ctxt(&ctxt, regs, exit_code);
+ if (result == ES_OK)
+ result = vc_handle_exitcode(&ctxt, boot_ghcb, exit_code);
+
+ /* Done - now check the result */
+ switch (result) {
+ case ES_OK:
+ vc_finish_insn(&ctxt);
+ break;
+ case ES_UNSUPPORTED:
+ early_printk("PANIC: Unsupported exit-code 0x%02lx in early #VC exception (IP: 0x%lx)\n",
+ exit_code, regs->ip);
+ goto fail;
+ case ES_VMM_ERROR:
+ early_printk("PANIC: Failure in communication with VMM (exit-code 0x%02lx IP: 0x%lx)\n",
+ exit_code, regs->ip);
+ goto fail;
+ case ES_DECODE_FAILED:
+ early_printk("PANIC: Failed to decode instruction (exit-code 0x%02lx IP: 0x%lx)\n",
+ exit_code, regs->ip);
+ goto fail;
+ case ES_EXCEPTION:
+ vc_early_forward_exception(&ctxt);
+ break;
+ case ES_RETRY:
+ /* Nothing to do */
+ break;
+ default:
+ BUG();
+ }
+
+ return true;
+
+fail:
+ show_regs(regs);
+
+ while (true)
+ halt();
+}
diff --git a/arch/x86/mm/extable.c b/arch/x86/mm/extable.c
index 1d6cb07f4f86..3966749d07ac 100644
--- a/arch/x86/mm/extable.c
+++ b/arch/x86/mm/extable.c
@@ -5,6 +5,7 @@
#include <xen/xen.h>
#include <asm/fpu/internal.h>
+#include <asm/sev-es.h>
#include <asm/traps.h>
#include <asm/kdebug.h>
--
2.28.0
_______________________________________________
Virtualization mailing list
Virtualization@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/virtualization
^ permalink raw reply related [flat|nested] 264+ messages in thread
* [PATCH v7 41/72] x86/sev-es: Setup per-cpu GHCBs for the runtime handler
2020-09-07 13:15 ` Joerg Roedel
@ 2020-09-07 13:15 ` Joerg Roedel
-1 siblings, 0 replies; 264+ messages in thread
From: Joerg Roedel @ 2020-09-07 13:15 UTC (permalink / raw)
To: x86
Cc: Joerg Roedel, Joerg Roedel, Tom Lendacky, hpa, Andy Lutomirski,
Dave Hansen, Peter Zijlstra, Jiri Slaby, Dan Williams,
Juergen Gross, Kees Cook, David Rientjes, Cfir Cohen,
Erdem Aktas, Masami Hiramatsu, Mike Stunes, Sean Christopherson,
Martin Radev, linux-kernel, kvm, virtualization
From: Tom Lendacky <thomas.lendacky@amd.com>
The runtime handler needs a GHCB per CPU. Set them up and map them
unencrypted.
Signed-off-by: Tom Lendacky <thomas.lendacky@amd.com>
Signed-off-by: Joerg Roedel <jroedel@suse.de>
---
arch/x86/include/asm/mem_encrypt.h | 2 ++
arch/x86/kernel/sev-es.c | 56 +++++++++++++++++++++++++++++-
arch/x86/kernel/traps.c | 3 ++
3 files changed, 60 insertions(+), 1 deletion(-)
diff --git a/arch/x86/include/asm/mem_encrypt.h b/arch/x86/include/asm/mem_encrypt.h
index 4e72b73a9cb5..c9f5df0a1c10 100644
--- a/arch/x86/include/asm/mem_encrypt.h
+++ b/arch/x86/include/asm/mem_encrypt.h
@@ -49,6 +49,7 @@ void __init mem_encrypt_free_decrypted_mem(void);
/* Architecture __weak replacement functions */
void __init mem_encrypt_init(void);
+void __init sev_es_init_vc_handling(void);
bool sme_active(void);
bool sev_active(void);
bool sev_es_active(void);
@@ -72,6 +73,7 @@ static inline void __init sme_early_init(void) { }
static inline void __init sme_encrypt_kernel(struct boot_params *bp) { }
static inline void __init sme_enable(struct boot_params *bp) { }
+static inline void sev_es_init_vc_handling(void) { }
static inline bool sme_active(void) { return false; }
static inline bool sev_active(void) { return false; }
static inline bool sev_es_active(void) { return false; }
diff --git a/arch/x86/kernel/sev-es.c b/arch/x86/kernel/sev-es.c
index bb3e702a71eb..0f28bb1c0022 100644
--- a/arch/x86/kernel/sev-es.c
+++ b/arch/x86/kernel/sev-es.c
@@ -8,8 +8,13 @@
*/
#include <linux/sched/debug.h> /* For show_regs() */
-#include <linux/kernel.h>
+#include <linux/percpu-defs.h>
+#include <linux/mem_encrypt.h>
#include <linux/printk.h>
+#include <linux/mm_types.h>
+#include <linux/set_memory.h>
+#include <linux/memblock.h>
+#include <linux/kernel.h>
#include <linux/mm.h>
#include <asm/sev-es.h>
@@ -29,6 +34,13 @@ static struct ghcb boot_ghcb_page __bss_decrypted __aligned(PAGE_SIZE);
*/
static struct ghcb __initdata *boot_ghcb;
+/* #VC handler runtime per-cpu data */
+struct sev_es_runtime_data {
+ struct ghcb ghcb_page;
+};
+
+static DEFINE_PER_CPU(struct sev_es_runtime_data*, runtime_data);
+
/* Needed in vc_early_forward_exception */
void do_early_exception(struct pt_regs *regs, int trapnr);
@@ -198,6 +210,48 @@ static bool __init sev_es_setup_ghcb(void)
return true;
}
+static void __init alloc_runtime_data(int cpu)
+{
+ struct sev_es_runtime_data *data;
+
+ data = memblock_alloc(sizeof(*data), PAGE_SIZE);
+ if (!data)
+ panic("Can't allocate SEV-ES runtime data");
+
+ per_cpu(runtime_data, cpu) = data;
+}
+
+static void __init init_ghcb(int cpu)
+{
+ struct sev_es_runtime_data *data;
+ int err;
+
+ data = per_cpu(runtime_data, cpu);
+
+ err = early_set_memory_decrypted((unsigned long)&data->ghcb_page,
+ sizeof(data->ghcb_page));
+ if (err)
+ panic("Can not map GHCBs unencrypted");
+
+ memset(&data->ghcb_page, 0, sizeof(data->ghcb_page));
+}
+
+void __init sev_es_init_vc_handling(void)
+{
+ int cpu;
+
+ BUILD_BUG_ON((offsetof(struct sev_es_runtime_data, ghcb_page) % PAGE_SIZE) != 0);
+
+ if (!sev_es_active())
+ return;
+
+ /* Initialize per-cpu GHCB pages */
+ for_each_possible_cpu(cpu) {
+ alloc_runtime_data(cpu);
+ init_ghcb(cpu);
+ }
+}
+
static void __init vc_early_forward_exception(struct es_em_ctxt *ctxt)
{
int trapnr = ctxt->fi.vector;
diff --git a/arch/x86/kernel/traps.c b/arch/x86/kernel/traps.c
index df9c6554f83e..e121e7c5f831 100644
--- a/arch/x86/kernel/traps.c
+++ b/arch/x86/kernel/traps.c
@@ -1082,6 +1082,9 @@ void __init trap_init(void)
/* Init cpu_entry_area before IST entries are set up */
setup_cpu_entry_areas();
+ /* Init GHCB memory pages when running as an SEV-ES guest */
+ sev_es_init_vc_handling();
+
idt_setup_traps();
/*
--
2.28.0
^ permalink raw reply related [flat|nested] 264+ messages in thread
* [PATCH v7 41/72] x86/sev-es: Setup per-cpu GHCBs for the runtime handler
@ 2020-09-07 13:15 ` Joerg Roedel
0 siblings, 0 replies; 264+ messages in thread
From: Joerg Roedel @ 2020-09-07 13:15 UTC (permalink / raw)
To: x86
Cc: Juergen Gross, Tom Lendacky, Joerg Roedel, Mike Stunes,
Kees Cook, kvm, Peter Zijlstra, Cfir Cohen, Joerg Roedel,
Dave Hansen, linux-kernel, Sean Christopherson, virtualization,
Martin Radev, Masami Hiramatsu, Andy Lutomirski, hpa,
Erdem Aktas, David Rientjes, Dan Williams, Jiri Slaby
From: Tom Lendacky <thomas.lendacky@amd.com>
The runtime handler needs a GHCB per CPU. Set them up and map them
unencrypted.
Signed-off-by: Tom Lendacky <thomas.lendacky@amd.com>
Signed-off-by: Joerg Roedel <jroedel@suse.de>
---
arch/x86/include/asm/mem_encrypt.h | 2 ++
arch/x86/kernel/sev-es.c | 56 +++++++++++++++++++++++++++++-
arch/x86/kernel/traps.c | 3 ++
3 files changed, 60 insertions(+), 1 deletion(-)
diff --git a/arch/x86/include/asm/mem_encrypt.h b/arch/x86/include/asm/mem_encrypt.h
index 4e72b73a9cb5..c9f5df0a1c10 100644
--- a/arch/x86/include/asm/mem_encrypt.h
+++ b/arch/x86/include/asm/mem_encrypt.h
@@ -49,6 +49,7 @@ void __init mem_encrypt_free_decrypted_mem(void);
/* Architecture __weak replacement functions */
void __init mem_encrypt_init(void);
+void __init sev_es_init_vc_handling(void);
bool sme_active(void);
bool sev_active(void);
bool sev_es_active(void);
@@ -72,6 +73,7 @@ static inline void __init sme_early_init(void) { }
static inline void __init sme_encrypt_kernel(struct boot_params *bp) { }
static inline void __init sme_enable(struct boot_params *bp) { }
+static inline void sev_es_init_vc_handling(void) { }
static inline bool sme_active(void) { return false; }
static inline bool sev_active(void) { return false; }
static inline bool sev_es_active(void) { return false; }
diff --git a/arch/x86/kernel/sev-es.c b/arch/x86/kernel/sev-es.c
index bb3e702a71eb..0f28bb1c0022 100644
--- a/arch/x86/kernel/sev-es.c
+++ b/arch/x86/kernel/sev-es.c
@@ -8,8 +8,13 @@
*/
#include <linux/sched/debug.h> /* For show_regs() */
-#include <linux/kernel.h>
+#include <linux/percpu-defs.h>
+#include <linux/mem_encrypt.h>
#include <linux/printk.h>
+#include <linux/mm_types.h>
+#include <linux/set_memory.h>
+#include <linux/memblock.h>
+#include <linux/kernel.h>
#include <linux/mm.h>
#include <asm/sev-es.h>
@@ -29,6 +34,13 @@ static struct ghcb boot_ghcb_page __bss_decrypted __aligned(PAGE_SIZE);
*/
static struct ghcb __initdata *boot_ghcb;
+/* #VC handler runtime per-cpu data */
+struct sev_es_runtime_data {
+ struct ghcb ghcb_page;
+};
+
+static DEFINE_PER_CPU(struct sev_es_runtime_data*, runtime_data);
+
/* Needed in vc_early_forward_exception */
void do_early_exception(struct pt_regs *regs, int trapnr);
@@ -198,6 +210,48 @@ static bool __init sev_es_setup_ghcb(void)
return true;
}
+static void __init alloc_runtime_data(int cpu)
+{
+ struct sev_es_runtime_data *data;
+
+ data = memblock_alloc(sizeof(*data), PAGE_SIZE);
+ if (!data)
+ panic("Can't allocate SEV-ES runtime data");
+
+ per_cpu(runtime_data, cpu) = data;
+}
+
+static void __init init_ghcb(int cpu)
+{
+ struct sev_es_runtime_data *data;
+ int err;
+
+ data = per_cpu(runtime_data, cpu);
+
+ err = early_set_memory_decrypted((unsigned long)&data->ghcb_page,
+ sizeof(data->ghcb_page));
+ if (err)
+ panic("Can not map GHCBs unencrypted");
+
+ memset(&data->ghcb_page, 0, sizeof(data->ghcb_page));
+}
+
+void __init sev_es_init_vc_handling(void)
+{
+ int cpu;
+
+ BUILD_BUG_ON((offsetof(struct sev_es_runtime_data, ghcb_page) % PAGE_SIZE) != 0);
+
+ if (!sev_es_active())
+ return;
+
+ /* Initialize per-cpu GHCB pages */
+ for_each_possible_cpu(cpu) {
+ alloc_runtime_data(cpu);
+ init_ghcb(cpu);
+ }
+}
+
static void __init vc_early_forward_exception(struct es_em_ctxt *ctxt)
{
int trapnr = ctxt->fi.vector;
diff --git a/arch/x86/kernel/traps.c b/arch/x86/kernel/traps.c
index df9c6554f83e..e121e7c5f831 100644
--- a/arch/x86/kernel/traps.c
+++ b/arch/x86/kernel/traps.c
@@ -1082,6 +1082,9 @@ void __init trap_init(void)
/* Init cpu_entry_area before IST entries are set up */
setup_cpu_entry_areas();
+ /* Init GHCB memory pages when running as an SEV-ES guest */
+ sev_es_init_vc_handling();
+
idt_setup_traps();
/*
--
2.28.0
_______________________________________________
Virtualization mailing list
Virtualization@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/virtualization
^ permalink raw reply related [flat|nested] 264+ messages in thread
* [PATCH v7 42/72] x86/sev-es: Allocate and Map IST stack for #VC handler
2020-09-07 13:15 ` Joerg Roedel
@ 2020-09-07 13:15 ` Joerg Roedel
-1 siblings, 0 replies; 264+ messages in thread
From: Joerg Roedel @ 2020-09-07 13:15 UTC (permalink / raw)
To: x86
Cc: Joerg Roedel, Joerg Roedel, hpa, Andy Lutomirski, Dave Hansen,
Peter Zijlstra, Jiri Slaby, Dan Williams, Tom Lendacky,
Juergen Gross, Kees Cook, David Rientjes, Cfir Cohen,
Erdem Aktas, Masami Hiramatsu, Mike Stunes, Sean Christopherson,
Martin Radev, linux-kernel, kvm, virtualization
From: Joerg Roedel <jroedel@suse.de>
Allocate and map an IST stack and an additional fall-back stack for
the #VC handler. The memory for the stacks is allocated only when
SEV-ES is active.
The #VC handler needs to use an IST stack because it could be raised
from kernel space with unsafe stack, e.g. in the SYSCALL entry path.
Since the #VC exception can be nested, the #VC handler switches back to
the interrupted stack when entered from kernel space. If switching back
is not possible the fall-back stack is used.
Signed-off-by: Joerg Roedel <jroedel@suse.de>
---
arch/x86/include/asm/cpu_entry_area.h | 33 +++++++++++++++++----------
arch/x86/include/asm/page_64_types.h | 1 +
arch/x86/kernel/cpu/common.c | 2 ++
arch/x86/kernel/dumpstack_64.c | 8 +++++--
arch/x86/kernel/sev-es.c | 33 +++++++++++++++++++++++++++
5 files changed, 63 insertions(+), 14 deletions(-)
diff --git a/arch/x86/include/asm/cpu_entry_area.h b/arch/x86/include/asm/cpu_entry_area.h
index 8902fdb7de13..3d52b094850a 100644
--- a/arch/x86/include/asm/cpu_entry_area.h
+++ b/arch/x86/include/asm/cpu_entry_area.h
@@ -11,25 +11,29 @@
#ifdef CONFIG_X86_64
/* Macro to enforce the same ordering and stack sizes */
-#define ESTACKS_MEMBERS(guardsize) \
- char DF_stack_guard[guardsize]; \
- char DF_stack[EXCEPTION_STKSZ]; \
- char NMI_stack_guard[guardsize]; \
- char NMI_stack[EXCEPTION_STKSZ]; \
- char DB_stack_guard[guardsize]; \
- char DB_stack[EXCEPTION_STKSZ]; \
- char MCE_stack_guard[guardsize]; \
- char MCE_stack[EXCEPTION_STKSZ]; \
- char IST_top_guard[guardsize]; \
+#define ESTACKS_MEMBERS(guardsize, optional_stack_size) \
+ char DF_stack_guard[guardsize]; \
+ char DF_stack[EXCEPTION_STKSZ]; \
+ char NMI_stack_guard[guardsize]; \
+ char NMI_stack[EXCEPTION_STKSZ]; \
+ char DB_stack_guard[guardsize]; \
+ char DB_stack[EXCEPTION_STKSZ]; \
+ char MCE_stack_guard[guardsize]; \
+ char MCE_stack[EXCEPTION_STKSZ]; \
+ char VC_stack_guard[guardsize]; \
+ char VC_stack[optional_stack_size]; \
+ char VC2_stack_guard[guardsize]; \
+ char VC2_stack[optional_stack_size]; \
+ char IST_top_guard[guardsize]; \
/* The exception stacks' physical storage. No guard pages required */
struct exception_stacks {
- ESTACKS_MEMBERS(0)
+ ESTACKS_MEMBERS(0, 0)
};
/* The effective cpu entry area mapping with guard pages. */
struct cea_exception_stacks {
- ESTACKS_MEMBERS(PAGE_SIZE)
+ ESTACKS_MEMBERS(PAGE_SIZE, EXCEPTION_STKSZ)
};
/*
@@ -40,6 +44,8 @@ enum exception_stack_ordering {
ESTACK_NMI,
ESTACK_DB,
ESTACK_MCE,
+ ESTACK_VC,
+ ESTACK_VC2,
N_EXCEPTION_STACKS
};
@@ -139,4 +145,7 @@ static inline struct entry_stack *cpu_entry_stack(int cpu)
#define __this_cpu_ist_top_va(name) \
CEA_ESTACK_TOP(__this_cpu_read(cea_exception_stacks), name)
+#define __this_cpu_ist_bottom_va(name) \
+ CEA_ESTACK_BOT(__this_cpu_read(cea_exception_stacks), name)
+
#endif
diff --git a/arch/x86/include/asm/page_64_types.h b/arch/x86/include/asm/page_64_types.h
index 288b065955b7..d0c6c10c18a0 100644
--- a/arch/x86/include/asm/page_64_types.h
+++ b/arch/x86/include/asm/page_64_types.h
@@ -28,6 +28,7 @@
#define IST_INDEX_NMI 1
#define IST_INDEX_DB 2
#define IST_INDEX_MCE 3
+#define IST_INDEX_VC 4
/*
* Set __PAGE_OFFSET to the most negative possible address +
diff --git a/arch/x86/kernel/cpu/common.c b/arch/x86/kernel/cpu/common.c
index 8aa20bc2f1ca..1d65365363a1 100644
--- a/arch/x86/kernel/cpu/common.c
+++ b/arch/x86/kernel/cpu/common.c
@@ -1821,6 +1821,8 @@ static inline void tss_setup_ist(struct tss_struct *tss)
tss->x86_tss.ist[IST_INDEX_NMI] = __this_cpu_ist_top_va(NMI);
tss->x86_tss.ist[IST_INDEX_DB] = __this_cpu_ist_top_va(DB);
tss->x86_tss.ist[IST_INDEX_MCE] = __this_cpu_ist_top_va(MCE);
+ /* Only mapped when SEV-ES is active */
+ tss->x86_tss.ist[IST_INDEX_VC] = __this_cpu_ist_top_va(VC);
}
#else /* CONFIG_X86_64 */
diff --git a/arch/x86/kernel/dumpstack_64.c b/arch/x86/kernel/dumpstack_64.c
index 4a94d38cd141..c49cf594714b 100644
--- a/arch/x86/kernel/dumpstack_64.c
+++ b/arch/x86/kernel/dumpstack_64.c
@@ -24,11 +24,13 @@ static const char * const exception_stack_names[] = {
[ ESTACK_NMI ] = "NMI",
[ ESTACK_DB ] = "#DB",
[ ESTACK_MCE ] = "#MC",
+ [ ESTACK_VC ] = "#VC",
+ [ ESTACK_VC2 ] = "#VC2",
};
const char *stack_type_name(enum stack_type type)
{
- BUILD_BUG_ON(N_EXCEPTION_STACKS != 4);
+ BUILD_BUG_ON(N_EXCEPTION_STACKS != 6);
if (type == STACK_TYPE_IRQ)
return "IRQ";
@@ -79,6 +81,8 @@ struct estack_pages estack_pages[CEA_ESTACK_PAGES] ____cacheline_aligned = {
EPAGERANGE(NMI),
EPAGERANGE(DB),
EPAGERANGE(MCE),
+ EPAGERANGE(VC),
+ EPAGERANGE(VC2),
};
static bool in_exception_stack(unsigned long *stack, struct stack_info *info)
@@ -88,7 +92,7 @@ static bool in_exception_stack(unsigned long *stack, struct stack_info *info)
struct pt_regs *regs;
unsigned int k;
- BUILD_BUG_ON(N_EXCEPTION_STACKS != 4);
+ BUILD_BUG_ON(N_EXCEPTION_STACKS != 6);
begin = (unsigned long)__this_cpu_read(cea_exception_stacks);
/*
diff --git a/arch/x86/kernel/sev-es.c b/arch/x86/kernel/sev-es.c
index 0f28bb1c0022..5541788420ce 100644
--- a/arch/x86/kernel/sev-es.c
+++ b/arch/x86/kernel/sev-es.c
@@ -17,6 +17,7 @@
#include <linux/kernel.h>
#include <linux/mm.h>
+#include <asm/cpu_entry_area.h>
#include <asm/sev-es.h>
#include <asm/insn-eval.h>
#include <asm/fpu/internal.h>
@@ -37,10 +38,41 @@ static struct ghcb __initdata *boot_ghcb;
/* #VC handler runtime per-cpu data */
struct sev_es_runtime_data {
struct ghcb ghcb_page;
+
+ /* Physical storage for the per-cpu IST stack of the #VC handler */
+ char ist_stack[EXCEPTION_STKSZ] __aligned(PAGE_SIZE);
+
+ /*
+ * Physical storage for the per-cpu fall-back stack of the #VC handler.
+ * The fall-back stack is used when it is not safe to switch back to the
+ * interrupted stack in the #VC entry code.
+ */
+ char fallback_stack[EXCEPTION_STKSZ] __aligned(PAGE_SIZE);
};
static DEFINE_PER_CPU(struct sev_es_runtime_data*, runtime_data);
+static void __init setup_vc_stacks(int cpu)
+{
+ struct sev_es_runtime_data *data;
+ struct cpu_entry_area *cea;
+ unsigned long vaddr;
+ phys_addr_t pa;
+
+ data = per_cpu(runtime_data, cpu);
+ cea = get_cpu_entry_area(cpu);
+
+ /* Map #VC IST stack */
+ vaddr = CEA_ESTACK_BOT(&cea->estacks, VC);
+ pa = __pa(data->ist_stack);
+ cea_set_pte((void *)vaddr, pa, PAGE_KERNEL);
+
+ /* Map VC fall-back stack */
+ vaddr = CEA_ESTACK_BOT(&cea->estacks, VC2);
+ pa = __pa(data->fallback_stack);
+ cea_set_pte((void *)vaddr, pa, PAGE_KERNEL);
+}
+
/* Needed in vc_early_forward_exception */
void do_early_exception(struct pt_regs *regs, int trapnr);
@@ -249,6 +281,7 @@ void __init sev_es_init_vc_handling(void)
for_each_possible_cpu(cpu) {
alloc_runtime_data(cpu);
init_ghcb(cpu);
+ setup_vc_stacks(cpu);
}
}
--
2.28.0
^ permalink raw reply related [flat|nested] 264+ messages in thread
* [PATCH v7 42/72] x86/sev-es: Allocate and Map IST stack for #VC handler
@ 2020-09-07 13:15 ` Joerg Roedel
0 siblings, 0 replies; 264+ messages in thread
From: Joerg Roedel @ 2020-09-07 13:15 UTC (permalink / raw)
To: x86
Cc: Juergen Gross, Tom Lendacky, Joerg Roedel, Mike Stunes,
Kees Cook, kvm, Peter Zijlstra, Cfir Cohen, Joerg Roedel,
Dave Hansen, linux-kernel, Sean Christopherson, virtualization,
Martin Radev, Masami Hiramatsu, Andy Lutomirski, hpa,
Erdem Aktas, David Rientjes, Dan Williams, Jiri Slaby
From: Joerg Roedel <jroedel@suse.de>
Allocate and map an IST stack and an additional fall-back stack for
the #VC handler. The memory for the stacks is allocated only when
SEV-ES is active.
The #VC handler needs to use an IST stack because it could be raised
from kernel space with unsafe stack, e.g. in the SYSCALL entry path.
Since the #VC exception can be nested, the #VC handler switches back to
the interrupted stack when entered from kernel space. If switching back
is not possible the fall-back stack is used.
Signed-off-by: Joerg Roedel <jroedel@suse.de>
---
arch/x86/include/asm/cpu_entry_area.h | 33 +++++++++++++++++----------
arch/x86/include/asm/page_64_types.h | 1 +
arch/x86/kernel/cpu/common.c | 2 ++
arch/x86/kernel/dumpstack_64.c | 8 +++++--
arch/x86/kernel/sev-es.c | 33 +++++++++++++++++++++++++++
5 files changed, 63 insertions(+), 14 deletions(-)
diff --git a/arch/x86/include/asm/cpu_entry_area.h b/arch/x86/include/asm/cpu_entry_area.h
index 8902fdb7de13..3d52b094850a 100644
--- a/arch/x86/include/asm/cpu_entry_area.h
+++ b/arch/x86/include/asm/cpu_entry_area.h
@@ -11,25 +11,29 @@
#ifdef CONFIG_X86_64
/* Macro to enforce the same ordering and stack sizes */
-#define ESTACKS_MEMBERS(guardsize) \
- char DF_stack_guard[guardsize]; \
- char DF_stack[EXCEPTION_STKSZ]; \
- char NMI_stack_guard[guardsize]; \
- char NMI_stack[EXCEPTION_STKSZ]; \
- char DB_stack_guard[guardsize]; \
- char DB_stack[EXCEPTION_STKSZ]; \
- char MCE_stack_guard[guardsize]; \
- char MCE_stack[EXCEPTION_STKSZ]; \
- char IST_top_guard[guardsize]; \
+#define ESTACKS_MEMBERS(guardsize, optional_stack_size) \
+ char DF_stack_guard[guardsize]; \
+ char DF_stack[EXCEPTION_STKSZ]; \
+ char NMI_stack_guard[guardsize]; \
+ char NMI_stack[EXCEPTION_STKSZ]; \
+ char DB_stack_guard[guardsize]; \
+ char DB_stack[EXCEPTION_STKSZ]; \
+ char MCE_stack_guard[guardsize]; \
+ char MCE_stack[EXCEPTION_STKSZ]; \
+ char VC_stack_guard[guardsize]; \
+ char VC_stack[optional_stack_size]; \
+ char VC2_stack_guard[guardsize]; \
+ char VC2_stack[optional_stack_size]; \
+ char IST_top_guard[guardsize]; \
/* The exception stacks' physical storage. No guard pages required */
struct exception_stacks {
- ESTACKS_MEMBERS(0)
+ ESTACKS_MEMBERS(0, 0)
};
/* The effective cpu entry area mapping with guard pages. */
struct cea_exception_stacks {
- ESTACKS_MEMBERS(PAGE_SIZE)
+ ESTACKS_MEMBERS(PAGE_SIZE, EXCEPTION_STKSZ)
};
/*
@@ -40,6 +44,8 @@ enum exception_stack_ordering {
ESTACK_NMI,
ESTACK_DB,
ESTACK_MCE,
+ ESTACK_VC,
+ ESTACK_VC2,
N_EXCEPTION_STACKS
};
@@ -139,4 +145,7 @@ static inline struct entry_stack *cpu_entry_stack(int cpu)
#define __this_cpu_ist_top_va(name) \
CEA_ESTACK_TOP(__this_cpu_read(cea_exception_stacks), name)
+#define __this_cpu_ist_bottom_va(name) \
+ CEA_ESTACK_BOT(__this_cpu_read(cea_exception_stacks), name)
+
#endif
diff --git a/arch/x86/include/asm/page_64_types.h b/arch/x86/include/asm/page_64_types.h
index 288b065955b7..d0c6c10c18a0 100644
--- a/arch/x86/include/asm/page_64_types.h
+++ b/arch/x86/include/asm/page_64_types.h
@@ -28,6 +28,7 @@
#define IST_INDEX_NMI 1
#define IST_INDEX_DB 2
#define IST_INDEX_MCE 3
+#define IST_INDEX_VC 4
/*
* Set __PAGE_OFFSET to the most negative possible address +
diff --git a/arch/x86/kernel/cpu/common.c b/arch/x86/kernel/cpu/common.c
index 8aa20bc2f1ca..1d65365363a1 100644
--- a/arch/x86/kernel/cpu/common.c
+++ b/arch/x86/kernel/cpu/common.c
@@ -1821,6 +1821,8 @@ static inline void tss_setup_ist(struct tss_struct *tss)
tss->x86_tss.ist[IST_INDEX_NMI] = __this_cpu_ist_top_va(NMI);
tss->x86_tss.ist[IST_INDEX_DB] = __this_cpu_ist_top_va(DB);
tss->x86_tss.ist[IST_INDEX_MCE] = __this_cpu_ist_top_va(MCE);
+ /* Only mapped when SEV-ES is active */
+ tss->x86_tss.ist[IST_INDEX_VC] = __this_cpu_ist_top_va(VC);
}
#else /* CONFIG_X86_64 */
diff --git a/arch/x86/kernel/dumpstack_64.c b/arch/x86/kernel/dumpstack_64.c
index 4a94d38cd141..c49cf594714b 100644
--- a/arch/x86/kernel/dumpstack_64.c
+++ b/arch/x86/kernel/dumpstack_64.c
@@ -24,11 +24,13 @@ static const char * const exception_stack_names[] = {
[ ESTACK_NMI ] = "NMI",
[ ESTACK_DB ] = "#DB",
[ ESTACK_MCE ] = "#MC",
+ [ ESTACK_VC ] = "#VC",
+ [ ESTACK_VC2 ] = "#VC2",
};
const char *stack_type_name(enum stack_type type)
{
- BUILD_BUG_ON(N_EXCEPTION_STACKS != 4);
+ BUILD_BUG_ON(N_EXCEPTION_STACKS != 6);
if (type == STACK_TYPE_IRQ)
return "IRQ";
@@ -79,6 +81,8 @@ struct estack_pages estack_pages[CEA_ESTACK_PAGES] ____cacheline_aligned = {
EPAGERANGE(NMI),
EPAGERANGE(DB),
EPAGERANGE(MCE),
+ EPAGERANGE(VC),
+ EPAGERANGE(VC2),
};
static bool in_exception_stack(unsigned long *stack, struct stack_info *info)
@@ -88,7 +92,7 @@ static bool in_exception_stack(unsigned long *stack, struct stack_info *info)
struct pt_regs *regs;
unsigned int k;
- BUILD_BUG_ON(N_EXCEPTION_STACKS != 4);
+ BUILD_BUG_ON(N_EXCEPTION_STACKS != 6);
begin = (unsigned long)__this_cpu_read(cea_exception_stacks);
/*
diff --git a/arch/x86/kernel/sev-es.c b/arch/x86/kernel/sev-es.c
index 0f28bb1c0022..5541788420ce 100644
--- a/arch/x86/kernel/sev-es.c
+++ b/arch/x86/kernel/sev-es.c
@@ -17,6 +17,7 @@
#include <linux/kernel.h>
#include <linux/mm.h>
+#include <asm/cpu_entry_area.h>
#include <asm/sev-es.h>
#include <asm/insn-eval.h>
#include <asm/fpu/internal.h>
@@ -37,10 +38,41 @@ static struct ghcb __initdata *boot_ghcb;
/* #VC handler runtime per-cpu data */
struct sev_es_runtime_data {
struct ghcb ghcb_page;
+
+ /* Physical storage for the per-cpu IST stack of the #VC handler */
+ char ist_stack[EXCEPTION_STKSZ] __aligned(PAGE_SIZE);
+
+ /*
+ * Physical storage for the per-cpu fall-back stack of the #VC handler.
+ * The fall-back stack is used when it is not safe to switch back to the
+ * interrupted stack in the #VC entry code.
+ */
+ char fallback_stack[EXCEPTION_STKSZ] __aligned(PAGE_SIZE);
};
static DEFINE_PER_CPU(struct sev_es_runtime_data*, runtime_data);
+static void __init setup_vc_stacks(int cpu)
+{
+ struct sev_es_runtime_data *data;
+ struct cpu_entry_area *cea;
+ unsigned long vaddr;
+ phys_addr_t pa;
+
+ data = per_cpu(runtime_data, cpu);
+ cea = get_cpu_entry_area(cpu);
+
+ /* Map #VC IST stack */
+ vaddr = CEA_ESTACK_BOT(&cea->estacks, VC);
+ pa = __pa(data->ist_stack);
+ cea_set_pte((void *)vaddr, pa, PAGE_KERNEL);
+
+ /* Map VC fall-back stack */
+ vaddr = CEA_ESTACK_BOT(&cea->estacks, VC2);
+ pa = __pa(data->fallback_stack);
+ cea_set_pte((void *)vaddr, pa, PAGE_KERNEL);
+}
+
/* Needed in vc_early_forward_exception */
void do_early_exception(struct pt_regs *regs, int trapnr);
@@ -249,6 +281,7 @@ void __init sev_es_init_vc_handling(void)
for_each_possible_cpu(cpu) {
alloc_runtime_data(cpu);
init_ghcb(cpu);
+ setup_vc_stacks(cpu);
}
}
--
2.28.0
_______________________________________________
Virtualization mailing list
Virtualization@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/virtualization
^ permalink raw reply related [flat|nested] 264+ messages in thread
* [PATCH v7 43/72] x86/sev-es: Adjust #VC IST Stack on entering NMI handler
2020-09-07 13:15 ` Joerg Roedel
@ 2020-09-07 13:15 ` Joerg Roedel
-1 siblings, 0 replies; 264+ messages in thread
From: Joerg Roedel @ 2020-09-07 13:15 UTC (permalink / raw)
To: x86
Cc: Joerg Roedel, Joerg Roedel, hpa, Andy Lutomirski, Dave Hansen,
Peter Zijlstra, Jiri Slaby, Dan Williams, Tom Lendacky,
Juergen Gross, Kees Cook, David Rientjes, Cfir Cohen,
Erdem Aktas, Masami Hiramatsu, Mike Stunes, Sean Christopherson,
Martin Radev, linux-kernel, kvm, virtualization
From: Joerg Roedel <jroedel@suse.de>
When an NMI hits in the #VC handler entry code before it switched to
another stack, any subsequent #VC exception in the NMI code-path will
overwrite the interrupted #VC handlers stack.
Make sure this doesn't happen by explicitly adjusting the #VC IST entry
in the NMI handler for the time in can cause #VC exceptions.
Signed-off-by: Joerg Roedel <jroedel@suse.de>
---
arch/x86/include/asm/sev-es.h | 19 +++++++++++++
arch/x86/kernel/nmi.c | 9 ++++++
arch/x86/kernel/sev-es.c | 53 +++++++++++++++++++++++++++++++++++
3 files changed, 81 insertions(+)
diff --git a/arch/x86/include/asm/sev-es.h b/arch/x86/include/asm/sev-es.h
index 9fbeedaa66ee..59176e8c6b81 100644
--- a/arch/x86/include/asm/sev-es.h
+++ b/arch/x86/include/asm/sev-es.h
@@ -78,4 +78,23 @@ extern void vc_no_ghcb(void);
extern void vc_boot_ghcb(void);
extern bool handle_vc_boot_ghcb(struct pt_regs *regs);
+#ifdef CONFIG_AMD_MEM_ENCRYPT
+extern struct static_key_false sev_es_enable_key;
+extern void __sev_es_ist_enter(struct pt_regs *regs);
+extern void __sev_es_ist_exit(void);
+static __always_inline void sev_es_ist_enter(struct pt_regs *regs)
+{
+ if (static_branch_unlikely(&sev_es_enable_key))
+ __sev_es_ist_enter(regs);
+}
+static __always_inline void sev_es_ist_exit(void)
+{
+ if (static_branch_unlikely(&sev_es_enable_key))
+ __sev_es_ist_exit();
+}
+#else
+static inline void sev_es_ist_enter(struct pt_regs *regs) { }
+static inline void sev_es_ist_exit(void) { }
+#endif
+
#endif
diff --git a/arch/x86/kernel/nmi.c b/arch/x86/kernel/nmi.c
index 4fc9954a9560..5859cec774a4 100644
--- a/arch/x86/kernel/nmi.c
+++ b/arch/x86/kernel/nmi.c
@@ -33,6 +33,7 @@
#include <asm/reboot.h>
#include <asm/cache.h>
#include <asm/nospec-branch.h>
+#include <asm/sev-es.h>
#define CREATE_TRACE_POINTS
#include <trace/events/nmi.h>
@@ -488,6 +489,12 @@ DEFINE_IDTENTRY_RAW(exc_nmi)
this_cpu_write(nmi_cr2, read_cr2());
nmi_restart:
+ /*
+ * Needs to happen before DR7 is accessed, because the hypervisor can
+ * intercept DR7 reads/writes, turings those into #VC exceptions.
+ */
+ sev_es_ist_enter(regs);
+
this_cpu_write(nmi_dr7, local_db_save());
irq_state = idtentry_enter_nmi(regs);
@@ -501,6 +508,8 @@ DEFINE_IDTENTRY_RAW(exc_nmi)
local_db_restore(this_cpu_read(nmi_dr7));
+ sev_es_ist_exit();
+
if (unlikely(this_cpu_read(nmi_cr2) != read_cr2()))
write_cr2(this_cpu_read(nmi_cr2));
if (this_cpu_dec_return(nmi_state))
diff --git a/arch/x86/kernel/sev-es.c b/arch/x86/kernel/sev-es.c
index 5541788420ce..69c55f0fdf6a 100644
--- a/arch/x86/kernel/sev-es.c
+++ b/arch/x86/kernel/sev-es.c
@@ -51,6 +51,7 @@ struct sev_es_runtime_data {
};
static DEFINE_PER_CPU(struct sev_es_runtime_data*, runtime_data);
+DEFINE_STATIC_KEY_FALSE(sev_es_enable_key);
static void __init setup_vc_stacks(int cpu)
{
@@ -73,6 +74,55 @@ static void __init setup_vc_stacks(int cpu)
cea_set_pte((void *)vaddr, pa, PAGE_KERNEL);
}
+static __always_inline bool on_vc_stack(unsigned long sp)
+{
+ return ((sp >= __this_cpu_ist_bottom_va(VC)) && (sp < __this_cpu_ist_top_va(VC)));
+}
+
+/*
+ * This function handles the case when an NM is raised in the #VC exception
+ * handler entry code. In this case the IST entry for #VC must be adjusted, so
+ * that any subsequent #VC exception will not overwrite the stack contents of the
+ * interrupted #VC handler.
+ *
+ * The IST entry is adjusted unconditionally so that it can be also be
+ * unconditionally adjusted back in sev_es_ist_exit(). Otherwise a nested
+ * sev_es_ist_exit() call may adjust back the IST entry too early.
+ */
+void noinstr __sev_es_ist_enter(struct pt_regs *regs)
+{
+ unsigned long old_ist, new_ist;
+
+ /* Read old IST entry */
+ old_ist = __this_cpu_read(cpu_tss_rw.x86_tss.ist[IST_INDEX_VC]);
+
+ /* Make room on the IST stack */
+ if (on_vc_stack(regs->sp))
+ new_ist = ALIGN_DOWN(regs->sp, 8) - sizeof(old_ist);
+ else
+ new_ist = old_ist - sizeof(old_ist);
+
+ /* Store old IST entry */
+ *(unsigned long *)new_ist = old_ist;
+
+ /* Set new IST entry */
+ this_cpu_write(cpu_tss_rw.x86_tss.ist[IST_INDEX_VC], new_ist);
+}
+
+void noinstr __sev_es_ist_exit(void)
+{
+ unsigned long ist;
+
+ /* Read IST entry */
+ ist = __this_cpu_read(cpu_tss_rw.x86_tss.ist[IST_INDEX_VC]);
+
+ if (WARN_ON(ist == __this_cpu_ist_top_va(VC)))
+ return;
+
+ /* Read back old IST entry and write it to the TSS */
+ this_cpu_write(cpu_tss_rw.x86_tss.ist[IST_INDEX_VC], *(unsigned long *)ist);
+}
+
/* Needed in vc_early_forward_exception */
void do_early_exception(struct pt_regs *regs, int trapnr);
@@ -277,6 +327,9 @@ void __init sev_es_init_vc_handling(void)
if (!sev_es_active())
return;
+ /* Enable SEV-ES special handling */
+ static_branch_enable(&sev_es_enable_key);
+
/* Initialize per-cpu GHCB pages */
for_each_possible_cpu(cpu) {
alloc_runtime_data(cpu);
--
2.28.0
^ permalink raw reply related [flat|nested] 264+ messages in thread
* [PATCH v7 43/72] x86/sev-es: Adjust #VC IST Stack on entering NMI handler
@ 2020-09-07 13:15 ` Joerg Roedel
0 siblings, 0 replies; 264+ messages in thread
From: Joerg Roedel @ 2020-09-07 13:15 UTC (permalink / raw)
To: x86
Cc: Juergen Gross, Tom Lendacky, Joerg Roedel, Mike Stunes,
Kees Cook, kvm, Peter Zijlstra, Cfir Cohen, Joerg Roedel,
Dave Hansen, linux-kernel, Sean Christopherson, virtualization,
Martin Radev, Masami Hiramatsu, Andy Lutomirski, hpa,
Erdem Aktas, David Rientjes, Dan Williams, Jiri Slaby
From: Joerg Roedel <jroedel@suse.de>
When an NMI hits in the #VC handler entry code before it switched to
another stack, any subsequent #VC exception in the NMI code-path will
overwrite the interrupted #VC handlers stack.
Make sure this doesn't happen by explicitly adjusting the #VC IST entry
in the NMI handler for the time in can cause #VC exceptions.
Signed-off-by: Joerg Roedel <jroedel@suse.de>
---
arch/x86/include/asm/sev-es.h | 19 +++++++++++++
arch/x86/kernel/nmi.c | 9 ++++++
arch/x86/kernel/sev-es.c | 53 +++++++++++++++++++++++++++++++++++
3 files changed, 81 insertions(+)
diff --git a/arch/x86/include/asm/sev-es.h b/arch/x86/include/asm/sev-es.h
index 9fbeedaa66ee..59176e8c6b81 100644
--- a/arch/x86/include/asm/sev-es.h
+++ b/arch/x86/include/asm/sev-es.h
@@ -78,4 +78,23 @@ extern void vc_no_ghcb(void);
extern void vc_boot_ghcb(void);
extern bool handle_vc_boot_ghcb(struct pt_regs *regs);
+#ifdef CONFIG_AMD_MEM_ENCRYPT
+extern struct static_key_false sev_es_enable_key;
+extern void __sev_es_ist_enter(struct pt_regs *regs);
+extern void __sev_es_ist_exit(void);
+static __always_inline void sev_es_ist_enter(struct pt_regs *regs)
+{
+ if (static_branch_unlikely(&sev_es_enable_key))
+ __sev_es_ist_enter(regs);
+}
+static __always_inline void sev_es_ist_exit(void)
+{
+ if (static_branch_unlikely(&sev_es_enable_key))
+ __sev_es_ist_exit();
+}
+#else
+static inline void sev_es_ist_enter(struct pt_regs *regs) { }
+static inline void sev_es_ist_exit(void) { }
+#endif
+
#endif
diff --git a/arch/x86/kernel/nmi.c b/arch/x86/kernel/nmi.c
index 4fc9954a9560..5859cec774a4 100644
--- a/arch/x86/kernel/nmi.c
+++ b/arch/x86/kernel/nmi.c
@@ -33,6 +33,7 @@
#include <asm/reboot.h>
#include <asm/cache.h>
#include <asm/nospec-branch.h>
+#include <asm/sev-es.h>
#define CREATE_TRACE_POINTS
#include <trace/events/nmi.h>
@@ -488,6 +489,12 @@ DEFINE_IDTENTRY_RAW(exc_nmi)
this_cpu_write(nmi_cr2, read_cr2());
nmi_restart:
+ /*
+ * Needs to happen before DR7 is accessed, because the hypervisor can
+ * intercept DR7 reads/writes, turings those into #VC exceptions.
+ */
+ sev_es_ist_enter(regs);
+
this_cpu_write(nmi_dr7, local_db_save());
irq_state = idtentry_enter_nmi(regs);
@@ -501,6 +508,8 @@ DEFINE_IDTENTRY_RAW(exc_nmi)
local_db_restore(this_cpu_read(nmi_dr7));
+ sev_es_ist_exit();
+
if (unlikely(this_cpu_read(nmi_cr2) != read_cr2()))
write_cr2(this_cpu_read(nmi_cr2));
if (this_cpu_dec_return(nmi_state))
diff --git a/arch/x86/kernel/sev-es.c b/arch/x86/kernel/sev-es.c
index 5541788420ce..69c55f0fdf6a 100644
--- a/arch/x86/kernel/sev-es.c
+++ b/arch/x86/kernel/sev-es.c
@@ -51,6 +51,7 @@ struct sev_es_runtime_data {
};
static DEFINE_PER_CPU(struct sev_es_runtime_data*, runtime_data);
+DEFINE_STATIC_KEY_FALSE(sev_es_enable_key);
static void __init setup_vc_stacks(int cpu)
{
@@ -73,6 +74,55 @@ static void __init setup_vc_stacks(int cpu)
cea_set_pte((void *)vaddr, pa, PAGE_KERNEL);
}
+static __always_inline bool on_vc_stack(unsigned long sp)
+{
+ return ((sp >= __this_cpu_ist_bottom_va(VC)) && (sp < __this_cpu_ist_top_va(VC)));
+}
+
+/*
+ * This function handles the case when an NM is raised in the #VC exception
+ * handler entry code. In this case the IST entry for #VC must be adjusted, so
+ * that any subsequent #VC exception will not overwrite the stack contents of the
+ * interrupted #VC handler.
+ *
+ * The IST entry is adjusted unconditionally so that it can be also be
+ * unconditionally adjusted back in sev_es_ist_exit(). Otherwise a nested
+ * sev_es_ist_exit() call may adjust back the IST entry too early.
+ */
+void noinstr __sev_es_ist_enter(struct pt_regs *regs)
+{
+ unsigned long old_ist, new_ist;
+
+ /* Read old IST entry */
+ old_ist = __this_cpu_read(cpu_tss_rw.x86_tss.ist[IST_INDEX_VC]);
+
+ /* Make room on the IST stack */
+ if (on_vc_stack(regs->sp))
+ new_ist = ALIGN_DOWN(regs->sp, 8) - sizeof(old_ist);
+ else
+ new_ist = old_ist - sizeof(old_ist);
+
+ /* Store old IST entry */
+ *(unsigned long *)new_ist = old_ist;
+
+ /* Set new IST entry */
+ this_cpu_write(cpu_tss_rw.x86_tss.ist[IST_INDEX_VC], new_ist);
+}
+
+void noinstr __sev_es_ist_exit(void)
+{
+ unsigned long ist;
+
+ /* Read IST entry */
+ ist = __this_cpu_read(cpu_tss_rw.x86_tss.ist[IST_INDEX_VC]);
+
+ if (WARN_ON(ist == __this_cpu_ist_top_va(VC)))
+ return;
+
+ /* Read back old IST entry and write it to the TSS */
+ this_cpu_write(cpu_tss_rw.x86_tss.ist[IST_INDEX_VC], *(unsigned long *)ist);
+}
+
/* Needed in vc_early_forward_exception */
void do_early_exception(struct pt_regs *regs, int trapnr);
@@ -277,6 +327,9 @@ void __init sev_es_init_vc_handling(void)
if (!sev_es_active())
return;
+ /* Enable SEV-ES special handling */
+ static_branch_enable(&sev_es_enable_key);
+
/* Initialize per-cpu GHCB pages */
for_each_possible_cpu(cpu) {
alloc_runtime_data(cpu);
--
2.28.0
_______________________________________________
Virtualization mailing list
Virtualization@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/virtualization
^ permalink raw reply related [flat|nested] 264+ messages in thread
* [PATCH v7 44/72] x86/dumpstack/64: Add noinstr version of get_stack_info()
2020-09-07 13:15 ` Joerg Roedel
@ 2020-09-07 13:15 ` Joerg Roedel
-1 siblings, 0 replies; 264+ messages in thread
From: Joerg Roedel @ 2020-09-07 13:15 UTC (permalink / raw)
To: x86
Cc: Joerg Roedel, Joerg Roedel, hpa, Andy Lutomirski, Dave Hansen,
Peter Zijlstra, Jiri Slaby, Dan Williams, Tom Lendacky,
Juergen Gross, Kees Cook, David Rientjes, Cfir Cohen,
Erdem Aktas, Masami Hiramatsu, Mike Stunes, Sean Christopherson,
Martin Radev, linux-kernel, kvm, virtualization
From: Joerg Roedel <jroedel@suse.de>
The get_stack_info functionality is needed in the entry code for the #VC
exception handler. Provide a version of it in the .text.noinstr
section which can be called safely from there.
Signed-off-by: Joerg Roedel <jroedel@suse.de>
---
arch/x86/include/asm/stacktrace.h | 2 ++
arch/x86/kernel/dumpstack.c | 7 +++---
arch/x86/kernel/dumpstack_64.c | 38 ++++++++++++++++++-------------
arch/x86/mm/cpu_entry_area.c | 3 ++-
4 files changed, 30 insertions(+), 20 deletions(-)
diff --git a/arch/x86/include/asm/stacktrace.h b/arch/x86/include/asm/stacktrace.h
index 5ae5a68e469d..49600643faba 100644
--- a/arch/x86/include/asm/stacktrace.h
+++ b/arch/x86/include/asm/stacktrace.h
@@ -35,6 +35,8 @@ bool in_entry_stack(unsigned long *stack, struct stack_info *info);
int get_stack_info(unsigned long *stack, struct task_struct *task,
struct stack_info *info, unsigned long *visit_mask);
+bool get_stack_info_noinstr(unsigned long *stack, struct task_struct *task,
+ struct stack_info *info);
const char *stack_type_name(enum stack_type type);
diff --git a/arch/x86/kernel/dumpstack.c b/arch/x86/kernel/dumpstack.c
index 48ce44576947..74147f7b3d82 100644
--- a/arch/x86/kernel/dumpstack.c
+++ b/arch/x86/kernel/dumpstack.c
@@ -29,8 +29,8 @@ static int die_counter;
static struct pt_regs exec_summary_regs;
-bool in_task_stack(unsigned long *stack, struct task_struct *task,
- struct stack_info *info)
+bool noinstr in_task_stack(unsigned long *stack, struct task_struct *task,
+ struct stack_info *info)
{
unsigned long *begin = task_stack_page(task);
unsigned long *end = task_stack_page(task) + THREAD_SIZE;
@@ -46,7 +46,8 @@ bool in_task_stack(unsigned long *stack, struct task_struct *task,
return true;
}
-bool in_entry_stack(unsigned long *stack, struct stack_info *info)
+/* Called from get_stack_info_noinstr - so must be noinstr too */
+bool noinstr in_entry_stack(unsigned long *stack, struct stack_info *info)
{
struct entry_stack *ss = cpu_entry_stack(smp_processor_id());
diff --git a/arch/x86/kernel/dumpstack_64.c b/arch/x86/kernel/dumpstack_64.c
index c49cf594714b..1dd851397bd9 100644
--- a/arch/x86/kernel/dumpstack_64.c
+++ b/arch/x86/kernel/dumpstack_64.c
@@ -85,7 +85,7 @@ struct estack_pages estack_pages[CEA_ESTACK_PAGES] ____cacheline_aligned = {
EPAGERANGE(VC2),
};
-static bool in_exception_stack(unsigned long *stack, struct stack_info *info)
+static __always_inline bool in_exception_stack(unsigned long *stack, struct stack_info *info)
{
unsigned long begin, end, stk = (unsigned long)stack;
const struct estack_pages *ep;
@@ -126,7 +126,7 @@ static bool in_exception_stack(unsigned long *stack, struct stack_info *info)
return true;
}
-static bool in_irq_stack(unsigned long *stack, struct stack_info *info)
+static __always_inline bool in_irq_stack(unsigned long *stack, struct stack_info *info)
{
unsigned long *end = (unsigned long *)this_cpu_read(hardirq_stack_ptr);
unsigned long *begin = end - (IRQ_STACK_SIZE / sizeof(long));
@@ -151,32 +151,38 @@ static bool in_irq_stack(unsigned long *stack, struct stack_info *info)
return true;
}
-int get_stack_info(unsigned long *stack, struct task_struct *task,
- struct stack_info *info, unsigned long *visit_mask)
+bool noinstr get_stack_info_noinstr(unsigned long *stack, struct task_struct *task,
+ struct stack_info *info)
{
- if (!stack)
- goto unknown;
-
- task = task ? : current;
-
if (in_task_stack(stack, task, info))
- goto recursion_check;
+ return true;
if (task != current)
- goto unknown;
+ return false;
if (in_exception_stack(stack, info))
- goto recursion_check;
+ return true;
if (in_irq_stack(stack, info))
- goto recursion_check;
+ return true;
if (in_entry_stack(stack, info))
- goto recursion_check;
+ return true;
+
+ return false;
+}
+
+int get_stack_info(unsigned long *stack, struct task_struct *task,
+ struct stack_info *info, unsigned long *visit_mask)
+{
+ task = task ? : current;
- goto unknown;
+ if (!stack)
+ goto unknown;
+
+ if (!get_stack_info_noinstr(stack, task, info))
+ goto unknown;
-recursion_check:
/*
* Make sure we don't iterate through any given stack more than once.
* If it comes up a second time then there's something wrong going on:
diff --git a/arch/x86/mm/cpu_entry_area.c b/arch/x86/mm/cpu_entry_area.c
index 770b613790b3..f5e1e60c9095 100644
--- a/arch/x86/mm/cpu_entry_area.c
+++ b/arch/x86/mm/cpu_entry_area.c
@@ -21,7 +21,8 @@ DEFINE_PER_CPU(struct cea_exception_stacks*, cea_exception_stacks);
DECLARE_PER_CPU_PAGE_ALIGNED(struct doublefault_stack, doublefault_stack);
#endif
-struct cpu_entry_area *get_cpu_entry_area(int cpu)
+/* Is called from entry code, so must be noinstr */
+noinstr struct cpu_entry_area *get_cpu_entry_area(int cpu)
{
unsigned long va = CPU_ENTRY_AREA_PER_CPU + cpu * CPU_ENTRY_AREA_SIZE;
BUILD_BUG_ON(sizeof(struct cpu_entry_area) % PAGE_SIZE != 0);
--
2.28.0
^ permalink raw reply related [flat|nested] 264+ messages in thread
* [PATCH v7 44/72] x86/dumpstack/64: Add noinstr version of get_stack_info()
@ 2020-09-07 13:15 ` Joerg Roedel
0 siblings, 0 replies; 264+ messages in thread
From: Joerg Roedel @ 2020-09-07 13:15 UTC (permalink / raw)
To: x86
Cc: Juergen Gross, Tom Lendacky, Joerg Roedel, Mike Stunes,
Kees Cook, kvm, Peter Zijlstra, Cfir Cohen, Joerg Roedel,
Dave Hansen, linux-kernel, Sean Christopherson, virtualization,
Martin Radev, Masami Hiramatsu, Andy Lutomirski, hpa,
Erdem Aktas, David Rientjes, Dan Williams, Jiri Slaby
From: Joerg Roedel <jroedel@suse.de>
The get_stack_info functionality is needed in the entry code for the #VC
exception handler. Provide a version of it in the .text.noinstr
section which can be called safely from there.
Signed-off-by: Joerg Roedel <jroedel@suse.de>
---
arch/x86/include/asm/stacktrace.h | 2 ++
arch/x86/kernel/dumpstack.c | 7 +++---
arch/x86/kernel/dumpstack_64.c | 38 ++++++++++++++++++-------------
arch/x86/mm/cpu_entry_area.c | 3 ++-
4 files changed, 30 insertions(+), 20 deletions(-)
diff --git a/arch/x86/include/asm/stacktrace.h b/arch/x86/include/asm/stacktrace.h
index 5ae5a68e469d..49600643faba 100644
--- a/arch/x86/include/asm/stacktrace.h
+++ b/arch/x86/include/asm/stacktrace.h
@@ -35,6 +35,8 @@ bool in_entry_stack(unsigned long *stack, struct stack_info *info);
int get_stack_info(unsigned long *stack, struct task_struct *task,
struct stack_info *info, unsigned long *visit_mask);
+bool get_stack_info_noinstr(unsigned long *stack, struct task_struct *task,
+ struct stack_info *info);
const char *stack_type_name(enum stack_type type);
diff --git a/arch/x86/kernel/dumpstack.c b/arch/x86/kernel/dumpstack.c
index 48ce44576947..74147f7b3d82 100644
--- a/arch/x86/kernel/dumpstack.c
+++ b/arch/x86/kernel/dumpstack.c
@@ -29,8 +29,8 @@ static int die_counter;
static struct pt_regs exec_summary_regs;
-bool in_task_stack(unsigned long *stack, struct task_struct *task,
- struct stack_info *info)
+bool noinstr in_task_stack(unsigned long *stack, struct task_struct *task,
+ struct stack_info *info)
{
unsigned long *begin = task_stack_page(task);
unsigned long *end = task_stack_page(task) + THREAD_SIZE;
@@ -46,7 +46,8 @@ bool in_task_stack(unsigned long *stack, struct task_struct *task,
return true;
}
-bool in_entry_stack(unsigned long *stack, struct stack_info *info)
+/* Called from get_stack_info_noinstr - so must be noinstr too */
+bool noinstr in_entry_stack(unsigned long *stack, struct stack_info *info)
{
struct entry_stack *ss = cpu_entry_stack(smp_processor_id());
diff --git a/arch/x86/kernel/dumpstack_64.c b/arch/x86/kernel/dumpstack_64.c
index c49cf594714b..1dd851397bd9 100644
--- a/arch/x86/kernel/dumpstack_64.c
+++ b/arch/x86/kernel/dumpstack_64.c
@@ -85,7 +85,7 @@ struct estack_pages estack_pages[CEA_ESTACK_PAGES] ____cacheline_aligned = {
EPAGERANGE(VC2),
};
-static bool in_exception_stack(unsigned long *stack, struct stack_info *info)
+static __always_inline bool in_exception_stack(unsigned long *stack, struct stack_info *info)
{
unsigned long begin, end, stk = (unsigned long)stack;
const struct estack_pages *ep;
@@ -126,7 +126,7 @@ static bool in_exception_stack(unsigned long *stack, struct stack_info *info)
return true;
}
-static bool in_irq_stack(unsigned long *stack, struct stack_info *info)
+static __always_inline bool in_irq_stack(unsigned long *stack, struct stack_info *info)
{
unsigned long *end = (unsigned long *)this_cpu_read(hardirq_stack_ptr);
unsigned long *begin = end - (IRQ_STACK_SIZE / sizeof(long));
@@ -151,32 +151,38 @@ static bool in_irq_stack(unsigned long *stack, struct stack_info *info)
return true;
}
-int get_stack_info(unsigned long *stack, struct task_struct *task,
- struct stack_info *info, unsigned long *visit_mask)
+bool noinstr get_stack_info_noinstr(unsigned long *stack, struct task_struct *task,
+ struct stack_info *info)
{
- if (!stack)
- goto unknown;
-
- task = task ? : current;
-
if (in_task_stack(stack, task, info))
- goto recursion_check;
+ return true;
if (task != current)
- goto unknown;
+ return false;
if (in_exception_stack(stack, info))
- goto recursion_check;
+ return true;
if (in_irq_stack(stack, info))
- goto recursion_check;
+ return true;
if (in_entry_stack(stack, info))
- goto recursion_check;
+ return true;
+
+ return false;
+}
+
+int get_stack_info(unsigned long *stack, struct task_struct *task,
+ struct stack_info *info, unsigned long *visit_mask)
+{
+ task = task ? : current;
- goto unknown;
+ if (!stack)
+ goto unknown;
+
+ if (!get_stack_info_noinstr(stack, task, info))
+ goto unknown;
-recursion_check:
/*
* Make sure we don't iterate through any given stack more than once.
* If it comes up a second time then there's something wrong going on:
diff --git a/arch/x86/mm/cpu_entry_area.c b/arch/x86/mm/cpu_entry_area.c
index 770b613790b3..f5e1e60c9095 100644
--- a/arch/x86/mm/cpu_entry_area.c
+++ b/arch/x86/mm/cpu_entry_area.c
@@ -21,7 +21,8 @@ DEFINE_PER_CPU(struct cea_exception_stacks*, cea_exception_stacks);
DECLARE_PER_CPU_PAGE_ALIGNED(struct doublefault_stack, doublefault_stack);
#endif
-struct cpu_entry_area *get_cpu_entry_area(int cpu)
+/* Is called from entry code, so must be noinstr */
+noinstr struct cpu_entry_area *get_cpu_entry_area(int cpu)
{
unsigned long va = CPU_ENTRY_AREA_PER_CPU + cpu * CPU_ENTRY_AREA_SIZE;
BUILD_BUG_ON(sizeof(struct cpu_entry_area) % PAGE_SIZE != 0);
--
2.28.0
_______________________________________________
Virtualization mailing list
Virtualization@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/virtualization
^ permalink raw reply related [flat|nested] 264+ messages in thread
* [PATCH v7 45/72] x86/entry/64: Add entry code for #VC handler
2020-09-07 13:15 ` Joerg Roedel
@ 2020-09-07 13:15 ` Joerg Roedel
-1 siblings, 0 replies; 264+ messages in thread
From: Joerg Roedel @ 2020-09-07 13:15 UTC (permalink / raw)
To: x86
Cc: Joerg Roedel, Joerg Roedel, hpa, Andy Lutomirski, Dave Hansen,
Peter Zijlstra, Jiri Slaby, Dan Williams, Tom Lendacky,
Juergen Gross, Kees Cook, David Rientjes, Cfir Cohen,
Erdem Aktas, Masami Hiramatsu, Mike Stunes, Sean Christopherson,
Martin Radev, linux-kernel, kvm, virtualization
From: Joerg Roedel <jroedel@suse.de>
The #VC handler needs special entry code because:
1. It runs on an IST stack
2. It needs to be able to handle nested #VC exceptions
To make this work the entry code is implemented to pretend it doesn't
use an IST stack. When entered from user-mode or early SYSCALL entry
path it switches to the task stack, if entered from kernel-mode it
tries to switch back to the previous stack in the IRET frame.
The stack found in the IRET frame is validated first, and if it is not
safe to use it for the #VC handler, the code will switch to a
fall-back stack (the #VC2 IST stack). From there it can cause nested
exceptions again.
Signed-off-by: Joerg Roedel <jroedel@suse.de>
---
arch/x86/entry/entry_64.S | 80 +++++++++++++++++++++++++++++++++
arch/x86/include/asm/idtentry.h | 44 ++++++++++++++++++
arch/x86/include/asm/proto.h | 1 +
arch/x86/include/asm/traps.h | 1 +
arch/x86/kernel/traps.c | 45 +++++++++++++++++++
5 files changed, 171 insertions(+)
diff --git a/arch/x86/entry/entry_64.S b/arch/x86/entry/entry_64.S
index 5c5d234d968d..f446e9048d07 100644
--- a/arch/x86/entry/entry_64.S
+++ b/arch/x86/entry/entry_64.S
@@ -101,6 +101,8 @@ SYM_CODE_START(entry_SYSCALL_64)
SWITCH_TO_KERNEL_CR3 scratch_reg=%rsp
movq PER_CPU_VAR(cpu_current_top_of_stack), %rsp
+SYM_INNER_LABEL(entry_SYSCALL_64_safe_stack, SYM_L_GLOBAL)
+
/* Construct struct pt_regs on stack */
pushq $__USER_DS /* pt_regs->ss */
pushq PER_CPU_VAR(cpu_tss_rw + TSS_sp2) /* pt_regs->sp */
@@ -446,6 +448,84 @@ _ASM_NOKPROBE(\asmsym)
SYM_CODE_END(\asmsym)
.endm
+#ifdef CONFIG_AMD_MEM_ENCRYPT
+/**
+ * idtentry_vc - Macro to generate entry stub for #VC
+ * @vector: Vector number
+ * @asmsym: ASM symbol for the entry point
+ * @cfunc: C function to be called
+ *
+ * The macro emits code to set up the kernel context for #VC. The #VC handler
+ * runs on an IST stack and needs to be able to cause nested #VC exceptions.
+ *
+ * To make this work the #VC entry code tries its best to pretend it doesn't use
+ * an IST stack by switching to the task stack if coming from user-space (which
+ * includes early SYSCALL entry path) or back to the stack in the IRET frame if
+ * entered from kernel-mode.
+ *
+ * If entered from kernel-mode the return stack is validated first, and if it is
+ * not safe to use (e.g. because it points to the entry stack) the #VC handler
+ * will switch to a fall-back stack (VC2) and call a special handler function.
+ *
+ * The macro is only used for one vector, but it is planned to be extended in
+ * the future for the #HV exception.
+ */
+.macro idtentry_vc vector asmsym cfunc
+SYM_CODE_START(\asmsym)
+ UNWIND_HINT_IRET_REGS
+ ASM_CLAC
+
+ /*
+ * If the entry is from userspace, switch stacks and treat it as
+ * a normal entry.
+ */
+ testb $3, CS-ORIG_RAX(%rsp)
+ jnz .Lfrom_usermode_switch_stack_\@
+
+ /*
+ * paranoid_entry returns SWAPGS flag for paranoid_exit in EBX.
+ * EBX == 0 -> SWAPGS, EBX == 1 -> no SWAPGS
+ */
+ call paranoid_entry
+
+ UNWIND_HINT_REGS
+
+ /*
+ * Switch off the IST stack to make it free for nested exceptions. The
+ * vc_switch_off_ist() function will switch back to the interrupted
+ * stack if it is safe to do so. If not it switches to the VC fall-back
+ * stack.
+ */
+ movq %rsp, %rdi /* pt_regs pointer */
+ call vc_switch_off_ist
+ movq %rax, %rsp /* Switch to new stack */
+
+ UNWIND_HINT_REGS
+
+ /* Update pt_regs */
+ movq ORIG_RAX(%rsp), %rsi /* get error code into 2nd argument*/
+ movq $-1, ORIG_RAX(%rsp) /* no syscall to restart */
+
+ movq %rsp, %rdi /* pt_regs pointer */
+
+ call \cfunc
+
+ /*
+ * No need to switch back to the IST stack. The current stack is either
+ * identical to the stack in the IRET frame or the VC fall-back stack,
+ * so it is definitly mapped even with PTI enabled.
+ */
+ jmp paranoid_exit
+
+ /* Switch to the regular task stack */
+.Lfrom_usermode_switch_stack_\@:
+ idtentry_body safe_stack_\cfunc, has_error_code=1
+
+_ASM_NOKPROBE(\asmsym)
+SYM_CODE_END(\asmsym)
+.endm
+#endif
+
/*
* Double fault entry. Straight paranoid. No checks from which context
* this comes because for the espfix induced #DF this would do the wrong
diff --git a/arch/x86/include/asm/idtentry.h b/arch/x86/include/asm/idtentry.h
index 337dcfd45472..5ce67113a761 100644
--- a/arch/x86/include/asm/idtentry.h
+++ b/arch/x86/include/asm/idtentry.h
@@ -308,6 +308,18 @@ static __always_inline void __##func(struct pt_regs *regs)
DECLARE_IDTENTRY_RAW(vector, func); \
__visible void noist_##func(struct pt_regs *regs)
+/**
+ * DECLARE_IDTENTRY_VC - Declare functions for the VC entry point
+ * @vector: Vector number (ignored for C)
+ * @func: Function name of the entry point
+ *
+ * Maps to DECLARE_IDTENTRY_RAW_ERRORCODE, but declares also the
+ * safe_stack C handler.
+ */
+#define DECLARE_IDTENTRY_VC(vector, func) \
+ DECLARE_IDTENTRY_RAW_ERRORCODE(vector, func); \
+ __visible noinstr void safe_stack_##func(struct pt_regs *regs, unsigned long error_code)
+
/**
* DEFINE_IDTENTRY_IST - Emit code for IST entry points
* @func: Function name of the entry point
@@ -347,6 +359,35 @@ static __always_inline void __##func(struct pt_regs *regs)
#define DEFINE_IDTENTRY_DF(func) \
DEFINE_IDTENTRY_RAW_ERRORCODE(func)
+/**
+ * DEFINE_IDTENTRY_VC_SAFE_STACK - Emit code for VMM communication handler
+ which runs on a safe stack.
+ * @func: Function name of the entry point
+ *
+ * Maps to DEFINE_IDTENTRY_RAW_ERRORCODE
+ */
+#define DEFINE_IDTENTRY_VC_SAFE_STACK(func) \
+ DEFINE_IDTENTRY_RAW_ERRORCODE(safe_stack_##func)
+
+/**
+ * DEFINE_IDTENTRY_VC_IST - Emit code for VMM communication handler
+ which runs on the VC fall-back stack
+ * @func: Function name of the entry point
+ *
+ * Maps to DEFINE_IDTENTRY_RAW_ERRORCODE
+ */
+#define DEFINE_IDTENTRY_VC_IST(func) \
+ DEFINE_IDTENTRY_RAW_ERRORCODE(ist_##func)
+
+/**
+ * DEFINE_IDTENTRY_VC - Emit code for VMM communication handler
+ * @func: Function name of the entry point
+ *
+ * Maps to DEFINE_IDTENTRY_RAW_ERRORCODE
+ */
+#define DEFINE_IDTENTRY_VC(func) \
+ DEFINE_IDTENTRY_RAW_ERRORCODE(func)
+
#else /* CONFIG_X86_64 */
/**
@@ -433,6 +474,9 @@ __visible noinstr void func(struct pt_regs *regs, \
# define DECLARE_IDTENTRY_XENCB(vector, func) \
DECLARE_IDTENTRY(vector, func)
+# define DECLARE_IDTENTRY_VC(vector, func) \
+ idtentry_vc vector asm_##func func
+
#else
# define DECLARE_IDTENTRY_MCE(vector, func) \
DECLARE_IDTENTRY(vector, func)
diff --git a/arch/x86/include/asm/proto.h b/arch/x86/include/asm/proto.h
index 28996fe19301..2c35f1c01a2d 100644
--- a/arch/x86/include/asm/proto.h
+++ b/arch/x86/include/asm/proto.h
@@ -10,6 +10,7 @@ void syscall_init(void);
#ifdef CONFIG_X86_64
void entry_SYSCALL_64(void);
+void entry_SYSCALL_64_safe_stack(void);
long do_arch_prctl_64(struct task_struct *task, int option, unsigned long arg2);
#endif
diff --git a/arch/x86/include/asm/traps.h b/arch/x86/include/asm/traps.h
index 6a308355ea29..1b86bb3abc56 100644
--- a/arch/x86/include/asm/traps.h
+++ b/arch/x86/include/asm/traps.h
@@ -15,6 +15,7 @@ asmlinkage __visible notrace struct pt_regs *sync_regs(struct pt_regs *eregs);
asmlinkage __visible notrace
struct bad_iret_stack *fixup_bad_iret(struct bad_iret_stack *s);
void __init trap_init(void);
+asmlinkage __visible noinstr struct pt_regs *vc_switch_off_ist(struct pt_regs *eregs);
#endif
#ifdef CONFIG_X86_F00F_BUG
diff --git a/arch/x86/kernel/traps.c b/arch/x86/kernel/traps.c
index e121e7c5f831..e2c6fd4dde8e 100644
--- a/arch/x86/kernel/traps.c
+++ b/arch/x86/kernel/traps.c
@@ -43,6 +43,7 @@
#include <asm/stacktrace.h>
#include <asm/processor.h>
#include <asm/debugreg.h>
+#include <asm/realmode.h>
#include <asm/text-patching.h>
#include <asm/ftrace.h>
#include <asm/traps.h>
@@ -673,6 +674,50 @@ asmlinkage __visible noinstr struct pt_regs *sync_regs(struct pt_regs *eregs)
return regs;
}
+#ifdef CONFIG_AMD_MEM_ENCRYPT
+asmlinkage __visible noinstr struct pt_regs *vc_switch_off_ist(struct pt_regs *regs)
+{
+ unsigned long sp, *stack;
+ struct stack_info info;
+ struct pt_regs *regs_ret;
+
+ /*
+ * In the SYSCALL entry path the RSP value comes from user-space - don't
+ * trust it and switch to the current kernel stack
+ */
+ if (regs->ip >= (unsigned long)entry_SYSCALL_64 &&
+ regs->ip < (unsigned long)entry_SYSCALL_64_safe_stack) {
+ sp = this_cpu_read(cpu_current_top_of_stack);
+ goto sync;
+ }
+
+ /*
+ * From here on the RSP value is trusted. Now check whether entry
+ * happened from a safe stack. Not safe are the entry or unknown stacks,
+ * use the fall-back stack instead in this case.
+ */
+ sp = regs->sp;
+ stack = (unsigned long *)sp;
+
+ if (!get_stack_info_noinstr(stack, current, &info) || info.type == STACK_TYPE_ENTRY ||
+ info.type >= STACK_TYPE_EXCEPTION_LAST)
+ sp = __this_cpu_ist_top_va(VC2);
+
+sync:
+ /*
+ * Found a safe stack - switch to it as if the entry didn't happen via
+ * IST stack. The code below only copies pt_regs, the real switch happens
+ * in assembly code.
+ */
+ sp = ALIGN_DOWN(sp, 8) - sizeof(*regs_ret);
+
+ regs_ret = (struct pt_regs *)sp;
+ *regs_ret = *regs;
+
+ return regs_ret;
+}
+#endif
+
struct bad_iret_stack {
void *error_entry_ret;
struct pt_regs regs;
--
2.28.0
^ permalink raw reply related [flat|nested] 264+ messages in thread
* [PATCH v7 45/72] x86/entry/64: Add entry code for #VC handler
@ 2020-09-07 13:15 ` Joerg Roedel
0 siblings, 0 replies; 264+ messages in thread
From: Joerg Roedel @ 2020-09-07 13:15 UTC (permalink / raw)
To: x86
Cc: Juergen Gross, Tom Lendacky, Joerg Roedel, Mike Stunes,
Kees Cook, kvm, Peter Zijlstra, Cfir Cohen, Joerg Roedel,
Dave Hansen, linux-kernel, Sean Christopherson, virtualization,
Martin Radev, Masami Hiramatsu, Andy Lutomirski, hpa,
Erdem Aktas, David Rientjes, Dan Williams, Jiri Slaby
From: Joerg Roedel <jroedel@suse.de>
The #VC handler needs special entry code because:
1. It runs on an IST stack
2. It needs to be able to handle nested #VC exceptions
To make this work the entry code is implemented to pretend it doesn't
use an IST stack. When entered from user-mode or early SYSCALL entry
path it switches to the task stack, if entered from kernel-mode it
tries to switch back to the previous stack in the IRET frame.
The stack found in the IRET frame is validated first, and if it is not
safe to use it for the #VC handler, the code will switch to a
fall-back stack (the #VC2 IST stack). From there it can cause nested
exceptions again.
Signed-off-by: Joerg Roedel <jroedel@suse.de>
---
arch/x86/entry/entry_64.S | 80 +++++++++++++++++++++++++++++++++
arch/x86/include/asm/idtentry.h | 44 ++++++++++++++++++
arch/x86/include/asm/proto.h | 1 +
arch/x86/include/asm/traps.h | 1 +
arch/x86/kernel/traps.c | 45 +++++++++++++++++++
5 files changed, 171 insertions(+)
diff --git a/arch/x86/entry/entry_64.S b/arch/x86/entry/entry_64.S
index 5c5d234d968d..f446e9048d07 100644
--- a/arch/x86/entry/entry_64.S
+++ b/arch/x86/entry/entry_64.S
@@ -101,6 +101,8 @@ SYM_CODE_START(entry_SYSCALL_64)
SWITCH_TO_KERNEL_CR3 scratch_reg=%rsp
movq PER_CPU_VAR(cpu_current_top_of_stack), %rsp
+SYM_INNER_LABEL(entry_SYSCALL_64_safe_stack, SYM_L_GLOBAL)
+
/* Construct struct pt_regs on stack */
pushq $__USER_DS /* pt_regs->ss */
pushq PER_CPU_VAR(cpu_tss_rw + TSS_sp2) /* pt_regs->sp */
@@ -446,6 +448,84 @@ _ASM_NOKPROBE(\asmsym)
SYM_CODE_END(\asmsym)
.endm
+#ifdef CONFIG_AMD_MEM_ENCRYPT
+/**
+ * idtentry_vc - Macro to generate entry stub for #VC
+ * @vector: Vector number
+ * @asmsym: ASM symbol for the entry point
+ * @cfunc: C function to be called
+ *
+ * The macro emits code to set up the kernel context for #VC. The #VC handler
+ * runs on an IST stack and needs to be able to cause nested #VC exceptions.
+ *
+ * To make this work the #VC entry code tries its best to pretend it doesn't use
+ * an IST stack by switching to the task stack if coming from user-space (which
+ * includes early SYSCALL entry path) or back to the stack in the IRET frame if
+ * entered from kernel-mode.
+ *
+ * If entered from kernel-mode the return stack is validated first, and if it is
+ * not safe to use (e.g. because it points to the entry stack) the #VC handler
+ * will switch to a fall-back stack (VC2) and call a special handler function.
+ *
+ * The macro is only used for one vector, but it is planned to be extended in
+ * the future for the #HV exception.
+ */
+.macro idtentry_vc vector asmsym cfunc
+SYM_CODE_START(\asmsym)
+ UNWIND_HINT_IRET_REGS
+ ASM_CLAC
+
+ /*
+ * If the entry is from userspace, switch stacks and treat it as
+ * a normal entry.
+ */
+ testb $3, CS-ORIG_RAX(%rsp)
+ jnz .Lfrom_usermode_switch_stack_\@
+
+ /*
+ * paranoid_entry returns SWAPGS flag for paranoid_exit in EBX.
+ * EBX == 0 -> SWAPGS, EBX == 1 -> no SWAPGS
+ */
+ call paranoid_entry
+
+ UNWIND_HINT_REGS
+
+ /*
+ * Switch off the IST stack to make it free for nested exceptions. The
+ * vc_switch_off_ist() function will switch back to the interrupted
+ * stack if it is safe to do so. If not it switches to the VC fall-back
+ * stack.
+ */
+ movq %rsp, %rdi /* pt_regs pointer */
+ call vc_switch_off_ist
+ movq %rax, %rsp /* Switch to new stack */
+
+ UNWIND_HINT_REGS
+
+ /* Update pt_regs */
+ movq ORIG_RAX(%rsp), %rsi /* get error code into 2nd argument*/
+ movq $-1, ORIG_RAX(%rsp) /* no syscall to restart */
+
+ movq %rsp, %rdi /* pt_regs pointer */
+
+ call \cfunc
+
+ /*
+ * No need to switch back to the IST stack. The current stack is either
+ * identical to the stack in the IRET frame or the VC fall-back stack,
+ * so it is definitly mapped even with PTI enabled.
+ */
+ jmp paranoid_exit
+
+ /* Switch to the regular task stack */
+.Lfrom_usermode_switch_stack_\@:
+ idtentry_body safe_stack_\cfunc, has_error_code=1
+
+_ASM_NOKPROBE(\asmsym)
+SYM_CODE_END(\asmsym)
+.endm
+#endif
+
/*
* Double fault entry. Straight paranoid. No checks from which context
* this comes because for the espfix induced #DF this would do the wrong
diff --git a/arch/x86/include/asm/idtentry.h b/arch/x86/include/asm/idtentry.h
index 337dcfd45472..5ce67113a761 100644
--- a/arch/x86/include/asm/idtentry.h
+++ b/arch/x86/include/asm/idtentry.h
@@ -308,6 +308,18 @@ static __always_inline void __##func(struct pt_regs *regs)
DECLARE_IDTENTRY_RAW(vector, func); \
__visible void noist_##func(struct pt_regs *regs)
+/**
+ * DECLARE_IDTENTRY_VC - Declare functions for the VC entry point
+ * @vector: Vector number (ignored for C)
+ * @func: Function name of the entry point
+ *
+ * Maps to DECLARE_IDTENTRY_RAW_ERRORCODE, but declares also the
+ * safe_stack C handler.
+ */
+#define DECLARE_IDTENTRY_VC(vector, func) \
+ DECLARE_IDTENTRY_RAW_ERRORCODE(vector, func); \
+ __visible noinstr void safe_stack_##func(struct pt_regs *regs, unsigned long error_code)
+
/**
* DEFINE_IDTENTRY_IST - Emit code for IST entry points
* @func: Function name of the entry point
@@ -347,6 +359,35 @@ static __always_inline void __##func(struct pt_regs *regs)
#define DEFINE_IDTENTRY_DF(func) \
DEFINE_IDTENTRY_RAW_ERRORCODE(func)
+/**
+ * DEFINE_IDTENTRY_VC_SAFE_STACK - Emit code for VMM communication handler
+ which runs on a safe stack.
+ * @func: Function name of the entry point
+ *
+ * Maps to DEFINE_IDTENTRY_RAW_ERRORCODE
+ */
+#define DEFINE_IDTENTRY_VC_SAFE_STACK(func) \
+ DEFINE_IDTENTRY_RAW_ERRORCODE(safe_stack_##func)
+
+/**
+ * DEFINE_IDTENTRY_VC_IST - Emit code for VMM communication handler
+ which runs on the VC fall-back stack
+ * @func: Function name of the entry point
+ *
+ * Maps to DEFINE_IDTENTRY_RAW_ERRORCODE
+ */
+#define DEFINE_IDTENTRY_VC_IST(func) \
+ DEFINE_IDTENTRY_RAW_ERRORCODE(ist_##func)
+
+/**
+ * DEFINE_IDTENTRY_VC - Emit code for VMM communication handler
+ * @func: Function name of the entry point
+ *
+ * Maps to DEFINE_IDTENTRY_RAW_ERRORCODE
+ */
+#define DEFINE_IDTENTRY_VC(func) \
+ DEFINE_IDTENTRY_RAW_ERRORCODE(func)
+
#else /* CONFIG_X86_64 */
/**
@@ -433,6 +474,9 @@ __visible noinstr void func(struct pt_regs *regs, \
# define DECLARE_IDTENTRY_XENCB(vector, func) \
DECLARE_IDTENTRY(vector, func)
+# define DECLARE_IDTENTRY_VC(vector, func) \
+ idtentry_vc vector asm_##func func
+
#else
# define DECLARE_IDTENTRY_MCE(vector, func) \
DECLARE_IDTENTRY(vector, func)
diff --git a/arch/x86/include/asm/proto.h b/arch/x86/include/asm/proto.h
index 28996fe19301..2c35f1c01a2d 100644
--- a/arch/x86/include/asm/proto.h
+++ b/arch/x86/include/asm/proto.h
@@ -10,6 +10,7 @@ void syscall_init(void);
#ifdef CONFIG_X86_64
void entry_SYSCALL_64(void);
+void entry_SYSCALL_64_safe_stack(void);
long do_arch_prctl_64(struct task_struct *task, int option, unsigned long arg2);
#endif
diff --git a/arch/x86/include/asm/traps.h b/arch/x86/include/asm/traps.h
index 6a308355ea29..1b86bb3abc56 100644
--- a/arch/x86/include/asm/traps.h
+++ b/arch/x86/include/asm/traps.h
@@ -15,6 +15,7 @@ asmlinkage __visible notrace struct pt_regs *sync_regs(struct pt_regs *eregs);
asmlinkage __visible notrace
struct bad_iret_stack *fixup_bad_iret(struct bad_iret_stack *s);
void __init trap_init(void);
+asmlinkage __visible noinstr struct pt_regs *vc_switch_off_ist(struct pt_regs *eregs);
#endif
#ifdef CONFIG_X86_F00F_BUG
diff --git a/arch/x86/kernel/traps.c b/arch/x86/kernel/traps.c
index e121e7c5f831..e2c6fd4dde8e 100644
--- a/arch/x86/kernel/traps.c
+++ b/arch/x86/kernel/traps.c
@@ -43,6 +43,7 @@
#include <asm/stacktrace.h>
#include <asm/processor.h>
#include <asm/debugreg.h>
+#include <asm/realmode.h>
#include <asm/text-patching.h>
#include <asm/ftrace.h>
#include <asm/traps.h>
@@ -673,6 +674,50 @@ asmlinkage __visible noinstr struct pt_regs *sync_regs(struct pt_regs *eregs)
return regs;
}
+#ifdef CONFIG_AMD_MEM_ENCRYPT
+asmlinkage __visible noinstr struct pt_regs *vc_switch_off_ist(struct pt_regs *regs)
+{
+ unsigned long sp, *stack;
+ struct stack_info info;
+ struct pt_regs *regs_ret;
+
+ /*
+ * In the SYSCALL entry path the RSP value comes from user-space - don't
+ * trust it and switch to the current kernel stack
+ */
+ if (regs->ip >= (unsigned long)entry_SYSCALL_64 &&
+ regs->ip < (unsigned long)entry_SYSCALL_64_safe_stack) {
+ sp = this_cpu_read(cpu_current_top_of_stack);
+ goto sync;
+ }
+
+ /*
+ * From here on the RSP value is trusted. Now check whether entry
+ * happened from a safe stack. Not safe are the entry or unknown stacks,
+ * use the fall-back stack instead in this case.
+ */
+ sp = regs->sp;
+ stack = (unsigned long *)sp;
+
+ if (!get_stack_info_noinstr(stack, current, &info) || info.type == STACK_TYPE_ENTRY ||
+ info.type >= STACK_TYPE_EXCEPTION_LAST)
+ sp = __this_cpu_ist_top_va(VC2);
+
+sync:
+ /*
+ * Found a safe stack - switch to it as if the entry didn't happen via
+ * IST stack. The code below only copies pt_regs, the real switch happens
+ * in assembly code.
+ */
+ sp = ALIGN_DOWN(sp, 8) - sizeof(*regs_ret);
+
+ regs_ret = (struct pt_regs *)sp;
+ *regs_ret = *regs;
+
+ return regs_ret;
+}
+#endif
+
struct bad_iret_stack {
void *error_entry_ret;
struct pt_regs regs;
--
2.28.0
_______________________________________________
Virtualization mailing list
Virtualization@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/virtualization
^ permalink raw reply related [flat|nested] 264+ messages in thread
* [PATCH v7 46/72] x86/sev-es: Add Runtime #VC Exception Handler
2020-09-07 13:15 ` Joerg Roedel
@ 2020-09-07 13:15 ` Joerg Roedel
-1 siblings, 0 replies; 264+ messages in thread
From: Joerg Roedel @ 2020-09-07 13:15 UTC (permalink / raw)
To: x86
Cc: Joerg Roedel, Joerg Roedel, Tom Lendacky, hpa, Andy Lutomirski,
Dave Hansen, Peter Zijlstra, Jiri Slaby, Dan Williams,
Juergen Gross, Kees Cook, David Rientjes, Cfir Cohen,
Erdem Aktas, Masami Hiramatsu, Mike Stunes, Sean Christopherson,
Martin Radev, linux-kernel, kvm, virtualization
From: Tom Lendacky <thomas.lendacky@amd.com>
Add the handlers for #VC exceptions invoked at runtime.
Signed-off-by: Tom Lendacky <thomas.lendacky@amd.com>
Signed-off-by: Joerg Roedel <jroedel@suse.de>
---
arch/x86/include/asm/idtentry.h | 6 +
arch/x86/kernel/idt.c | 11 +-
arch/x86/kernel/sev-es.c | 246 +++++++++++++++++++++++++++++++-
3 files changed, 255 insertions(+), 8 deletions(-)
diff --git a/arch/x86/include/asm/idtentry.h b/arch/x86/include/asm/idtentry.h
index 5ce67113a761..3d2572760ce4 100644
--- a/arch/x86/include/asm/idtentry.h
+++ b/arch/x86/include/asm/idtentry.h
@@ -318,6 +318,7 @@ static __always_inline void __##func(struct pt_regs *regs)
*/
#define DECLARE_IDTENTRY_VC(vector, func) \
DECLARE_IDTENTRY_RAW_ERRORCODE(vector, func); \
+ __visible noinstr void ist_##func(struct pt_regs *regs, unsigned long error_code); \
__visible noinstr void safe_stack_##func(struct pt_regs *regs, unsigned long error_code)
/**
@@ -608,6 +609,11 @@ DECLARE_IDTENTRY_RAW(X86_TRAP_DB, xenpv_exc_debug);
/* #DF */
DECLARE_IDTENTRY_DF(X86_TRAP_DF, exc_double_fault);
+/* #VC */
+#ifdef CONFIG_AMD_MEM_ENCRYPT
+DECLARE_IDTENTRY_VC(X86_TRAP_VC, exc_vmm_communication);
+#endif
+
#ifdef CONFIG_XEN_PV
DECLARE_IDTENTRY_XENCB(X86_TRAP_OTHER, exc_xen_hypervisor_callback);
#endif
diff --git a/arch/x86/kernel/idt.c b/arch/x86/kernel/idt.c
index 4bb4e3d6099e..e29a6c728001 100644
--- a/arch/x86/kernel/idt.c
+++ b/arch/x86/kernel/idt.c
@@ -229,11 +229,14 @@ static const __initconst struct idt_data early_pf_idts[] = {
* cpu_init() when the TSS has been initialized.
*/
static const __initconst struct idt_data ist_idts[] = {
- ISTG(X86_TRAP_DB, asm_exc_debug, IST_INDEX_DB),
- ISTG(X86_TRAP_NMI, asm_exc_nmi, IST_INDEX_NMI),
- ISTG(X86_TRAP_DF, asm_exc_double_fault, IST_INDEX_DF),
+ ISTG(X86_TRAP_DB, asm_exc_debug, IST_INDEX_DB),
+ ISTG(X86_TRAP_NMI, asm_exc_nmi, IST_INDEX_NMI),
+ ISTG(X86_TRAP_DF, asm_exc_double_fault, IST_INDEX_DF),
#ifdef CONFIG_X86_MCE
- ISTG(X86_TRAP_MC, asm_exc_machine_check, IST_INDEX_MCE),
+ ISTG(X86_TRAP_MC, asm_exc_machine_check, IST_INDEX_MCE),
+#endif
+#ifdef CONFIG_AMD_MEM_ENCRYPT
+ ISTG(X86_TRAP_VC, asm_exc_vmm_communication, IST_INDEX_VC),
#endif
};
diff --git a/arch/x86/kernel/sev-es.c b/arch/x86/kernel/sev-es.c
index 69c55f0fdf6a..cb2f6236a98e 100644
--- a/arch/x86/kernel/sev-es.c
+++ b/arch/x86/kernel/sev-es.c
@@ -7,9 +7,12 @@
* Author: Joerg Roedel <jroedel@suse.de>
*/
+#define pr_fmt(fmt) "SEV-ES: " fmt
+
#include <linux/sched/debug.h> /* For show_regs() */
#include <linux/percpu-defs.h>
#include <linux/mem_encrypt.h>
+#include <linux/lockdep.h>
#include <linux/printk.h>
#include <linux/mm_types.h>
#include <linux/set_memory.h>
@@ -22,8 +25,8 @@
#include <asm/insn-eval.h>
#include <asm/fpu/internal.h>
#include <asm/processor.h>
-#include <asm/trap_pf.h>
-#include <asm/trapnr.h>
+#include <asm/realmode.h>
+#include <asm/traps.h>
#include <asm/svm.h>
/* For early boot hypervisor communication in SEV-ES enabled guests */
@@ -48,11 +51,43 @@ struct sev_es_runtime_data {
* interrupted stack in the #VC entry code.
*/
char fallback_stack[EXCEPTION_STKSZ] __aligned(PAGE_SIZE);
+
+ /*
+ * Reserve one page per CPU as backup storage for the unencrypted GHCB.
+ * It is needed when an NMI happens while the #VC handler uses the real
+ * GHCB, and the NMI handler itself is causing another #VC exception. In
+ * that case the GHCB content of the first handler needs to be backed up
+ * and restored.
+ */
+ struct ghcb backup_ghcb;
+
+ /*
+ * Mark the per-cpu GHCBs as in-use to detect nested #VC exceptions.
+ * There is no need for it to be atomic, because nothing is written to
+ * the GHCB between the read and the write of ghcb_active. So it is safe
+ * to use it when a nested #VC exception happens before the write.
+ *
+ * This is necessary for example in the #VC->NMI->#VC case when the NMI
+ * happens while the first #VC handler uses the GHCB. When the NMI code
+ * raises a second #VC handler it might overwrite the contents of the
+ * GHCB written by the first handler. To avoid this the content of the
+ * GHCB is saved and restored when the GHCB is detected to be in use
+ * already.
+ */
+ bool ghcb_active;
+ bool backup_ghcb_active;
+};
+
+struct ghcb_state {
+ struct ghcb *ghcb;
};
static DEFINE_PER_CPU(struct sev_es_runtime_data*, runtime_data);
DEFINE_STATIC_KEY_FALSE(sev_es_enable_key);
+/* Needed in vc_early_forward_exception */
+void do_early_exception(struct pt_regs *regs, int trapnr);
+
static void __init setup_vc_stacks(int cpu)
{
struct sev_es_runtime_data *data;
@@ -123,8 +158,52 @@ void noinstr __sev_es_ist_exit(void)
this_cpu_write(cpu_tss_rw.x86_tss.ist[IST_INDEX_VC], *(unsigned long *)ist);
}
-/* Needed in vc_early_forward_exception */
-void do_early_exception(struct pt_regs *regs, int trapnr);
+static __always_inline struct ghcb *sev_es_get_ghcb(struct ghcb_state *state)
+{
+ struct sev_es_runtime_data *data;
+ struct ghcb *ghcb;
+
+ data = this_cpu_read(runtime_data);
+ ghcb = &data->ghcb_page;
+
+ if (unlikely(data->ghcb_active)) {
+ /* GHCB is already in use - save its contents */
+
+ if (unlikely(data->backup_ghcb_active))
+ return NULL;
+
+ /* Mark backup_ghcb active before writing to it */
+ data->backup_ghcb_active = true;
+
+ state->ghcb = &data->backup_ghcb;
+
+ /* Backup GHCB content */
+ *state->ghcb = *ghcb;
+ } else {
+ state->ghcb = NULL;
+ data->ghcb_active = true;
+ }
+
+ return ghcb;
+}
+
+static __always_inline void sev_es_put_ghcb(struct ghcb_state *state)
+{
+ struct sev_es_runtime_data *data;
+ struct ghcb *ghcb;
+
+ data = this_cpu_read(runtime_data);
+ ghcb = &data->ghcb_page;
+
+ if (state->ghcb) {
+ /* Restore GHCB from Backup */
+ *ghcb = *state->ghcb;
+ data->backup_ghcb_active = false;
+ state->ghcb = NULL;
+ } else {
+ data->ghcb_active = false;
+ }
+}
static inline u64 sev_es_rd_ghcb_msr(void)
{
@@ -316,6 +395,9 @@ static void __init init_ghcb(int cpu)
panic("Can not map GHCBs unencrypted");
memset(&data->ghcb_page, 0, sizeof(data->ghcb_page));
+
+ data->ghcb_active = false;
+ data->backup_ghcb_active = false;
}
void __init sev_es_init_vc_handling(void)
@@ -336,6 +418,9 @@ void __init sev_es_init_vc_handling(void)
init_ghcb(cpu);
setup_vc_stacks(cpu);
}
+
+ /* Secondary CPUs use the runtime #VC handler */
+ initial_vc_handler = (unsigned long)safe_stack_exc_vmm_communication;
}
static void __init vc_early_forward_exception(struct es_em_ctxt *ctxt)
@@ -366,6 +451,159 @@ static enum es_result vc_handle_exitcode(struct es_em_ctxt *ctxt,
return result;
}
+static __always_inline void vc_forward_exception(struct es_em_ctxt *ctxt)
+{
+ long error_code = ctxt->fi.error_code;
+ int trapnr = ctxt->fi.vector;
+
+ ctxt->regs->orig_ax = ctxt->fi.error_code;
+
+ switch (trapnr) {
+ case X86_TRAP_GP:
+ exc_general_protection(ctxt->regs, error_code);
+ break;
+ case X86_TRAP_UD:
+ exc_invalid_op(ctxt->regs);
+ break;
+ default:
+ pr_emerg("Unsupported exception in #VC instruction emulation - can't continue\n");
+ BUG();
+ }
+}
+
+static __always_inline bool on_vc_fallback_stack(struct pt_regs *regs)
+{
+ unsigned long sp = (unsigned long)regs;
+
+ return (sp >= __this_cpu_ist_bottom_va(VC2) && sp < __this_cpu_ist_top_va(VC2));
+}
+
+/*
+ * Main #VC exception handler. It is called when the entry code was able to
+ * switch off the IST to a safe kernel stack.
+ *
+ * With the current implementation it is always possible to switch to a safe
+ * stack because #VC exceptions only happen at known places, like intercepted
+ * instructions or accesses to MMIO areas/IO ports. They can also happen with
+ * code instrumentation when the hypervisor intercepts #DB, but the critical
+ * paths are forbidden to be instrumented, so #DB exceptions currently also
+ * only happen in safe places.
+ */
+DEFINE_IDTENTRY_VC_SAFE_STACK(exc_vmm_communication)
+{
+ struct sev_es_runtime_data *data = this_cpu_read(runtime_data);
+ struct ghcb_state state;
+ struct es_em_ctxt ctxt;
+ enum es_result result;
+ struct ghcb *ghcb;
+
+ lockdep_assert_irqs_disabled();
+ instrumentation_begin();
+
+ /*
+ * This is invoked through an interrupt gate, so IRQs are disabled. The
+ * code below might walk page-tables for user or kernel addresses, so
+ * keep the IRQs disabled to protect us against concurrent TLB flushes.
+ */
+
+ ghcb = sev_es_get_ghcb(&state);
+ if (!ghcb) {
+ /*
+ * Mark GHCBs inactive so that panic() is able to print the
+ * message.
+ */
+ data->ghcb_active = false;
+ data->backup_ghcb_active = false;
+
+ panic("Unable to handle #VC exception! GHCB and Backup GHCB are already in use");
+ }
+
+ vc_ghcb_invalidate(ghcb);
+ result = vc_init_em_ctxt(&ctxt, regs, error_code);
+
+ if (result == ES_OK)
+ result = vc_handle_exitcode(&ctxt, ghcb, error_code);
+
+ sev_es_put_ghcb(&state);
+
+ /* Done - now check the result */
+ switch (result) {
+ case ES_OK:
+ vc_finish_insn(&ctxt);
+ break;
+ case ES_UNSUPPORTED:
+ pr_err_ratelimited("Unsupported exit-code 0x%02lx in early #VC exception (IP: 0x%lx)\n",
+ error_code, regs->ip);
+ goto fail;
+ case ES_VMM_ERROR:
+ pr_err_ratelimited("Failure in communication with VMM (exit-code 0x%02lx IP: 0x%lx)\n",
+ error_code, regs->ip);
+ goto fail;
+ case ES_DECODE_FAILED:
+ pr_err_ratelimited("Failed to decode instruction (exit-code 0x%02lx IP: 0x%lx)\n",
+ error_code, regs->ip);
+ goto fail;
+ case ES_EXCEPTION:
+ vc_forward_exception(&ctxt);
+ break;
+ case ES_RETRY:
+ /* Nothing to do */
+ break;
+ default:
+ pr_emerg("Unknown result in %s():%d\n", __func__, result);
+ /*
+ * Emulating the instruction which caused the #VC exception
+ * failed - can't continue so print debug information
+ */
+ BUG();
+ }
+
+out:
+ instrumentation_end();
+
+ return;
+
+fail:
+ if (user_mode(regs)) {
+ /*
+ * Do not kill the machine if user-space triggered the
+ * exception. Send SIGBUS instead and let user-space deal with
+ * it.
+ */
+ force_sig_fault(SIGBUS, BUS_OBJERR, (void __user *)0);
+ } else {
+ pr_emerg("PANIC: Unhandled #VC exception in kernel space (result=%d)\n",
+ result);
+
+ /* Show some debug info */
+ show_regs(regs);
+
+ /* Ask hypervisor to sev_es_terminate */
+ sev_es_terminate(GHCB_SEV_ES_REASON_GENERAL_REQUEST);
+
+ /* If that fails and we get here - just panic */
+ panic("Returned from Terminate-Request to Hypervisor\n");
+ }
+
+ goto out;
+}
+
+/* This handler runs on the #VC fall-back stack. It can cause further #VC exceptions */
+DEFINE_IDTENTRY_VC_IST(exc_vmm_communication)
+{
+ instrumentation_begin();
+ panic("Can't handle #VC exception from unsupported context\n");
+ instrumentation_end();
+}
+
+DEFINE_IDTENTRY_VC(exc_vmm_communication)
+{
+ if (likely(!on_vc_fallback_stack(regs)))
+ safe_stack_exc_vmm_communication(regs, error_code);
+ else
+ ist_exc_vmm_communication(regs, error_code);
+}
+
bool __init handle_vc_boot_ghcb(struct pt_regs *regs)
{
unsigned long exit_code = regs->orig_ax;
--
2.28.0
^ permalink raw reply related [flat|nested] 264+ messages in thread
* [PATCH v7 46/72] x86/sev-es: Add Runtime #VC Exception Handler
@ 2020-09-07 13:15 ` Joerg Roedel
0 siblings, 0 replies; 264+ messages in thread
From: Joerg Roedel @ 2020-09-07 13:15 UTC (permalink / raw)
To: x86
Cc: Juergen Gross, Tom Lendacky, Joerg Roedel, Mike Stunes,
Kees Cook, kvm, Peter Zijlstra, Cfir Cohen, Joerg Roedel,
Dave Hansen, linux-kernel, Sean Christopherson, virtualization,
Martin Radev, Masami Hiramatsu, Andy Lutomirski, hpa,
Erdem Aktas, David Rientjes, Dan Williams, Jiri Slaby
From: Tom Lendacky <thomas.lendacky@amd.com>
Add the handlers for #VC exceptions invoked at runtime.
Signed-off-by: Tom Lendacky <thomas.lendacky@amd.com>
Signed-off-by: Joerg Roedel <jroedel@suse.de>
---
arch/x86/include/asm/idtentry.h | 6 +
arch/x86/kernel/idt.c | 11 +-
arch/x86/kernel/sev-es.c | 246 +++++++++++++++++++++++++++++++-
3 files changed, 255 insertions(+), 8 deletions(-)
diff --git a/arch/x86/include/asm/idtentry.h b/arch/x86/include/asm/idtentry.h
index 5ce67113a761..3d2572760ce4 100644
--- a/arch/x86/include/asm/idtentry.h
+++ b/arch/x86/include/asm/idtentry.h
@@ -318,6 +318,7 @@ static __always_inline void __##func(struct pt_regs *regs)
*/
#define DECLARE_IDTENTRY_VC(vector, func) \
DECLARE_IDTENTRY_RAW_ERRORCODE(vector, func); \
+ __visible noinstr void ist_##func(struct pt_regs *regs, unsigned long error_code); \
__visible noinstr void safe_stack_##func(struct pt_regs *regs, unsigned long error_code)
/**
@@ -608,6 +609,11 @@ DECLARE_IDTENTRY_RAW(X86_TRAP_DB, xenpv_exc_debug);
/* #DF */
DECLARE_IDTENTRY_DF(X86_TRAP_DF, exc_double_fault);
+/* #VC */
+#ifdef CONFIG_AMD_MEM_ENCRYPT
+DECLARE_IDTENTRY_VC(X86_TRAP_VC, exc_vmm_communication);
+#endif
+
#ifdef CONFIG_XEN_PV
DECLARE_IDTENTRY_XENCB(X86_TRAP_OTHER, exc_xen_hypervisor_callback);
#endif
diff --git a/arch/x86/kernel/idt.c b/arch/x86/kernel/idt.c
index 4bb4e3d6099e..e29a6c728001 100644
--- a/arch/x86/kernel/idt.c
+++ b/arch/x86/kernel/idt.c
@@ -229,11 +229,14 @@ static const __initconst struct idt_data early_pf_idts[] = {
* cpu_init() when the TSS has been initialized.
*/
static const __initconst struct idt_data ist_idts[] = {
- ISTG(X86_TRAP_DB, asm_exc_debug, IST_INDEX_DB),
- ISTG(X86_TRAP_NMI, asm_exc_nmi, IST_INDEX_NMI),
- ISTG(X86_TRAP_DF, asm_exc_double_fault, IST_INDEX_DF),
+ ISTG(X86_TRAP_DB, asm_exc_debug, IST_INDEX_DB),
+ ISTG(X86_TRAP_NMI, asm_exc_nmi, IST_INDEX_NMI),
+ ISTG(X86_TRAP_DF, asm_exc_double_fault, IST_INDEX_DF),
#ifdef CONFIG_X86_MCE
- ISTG(X86_TRAP_MC, asm_exc_machine_check, IST_INDEX_MCE),
+ ISTG(X86_TRAP_MC, asm_exc_machine_check, IST_INDEX_MCE),
+#endif
+#ifdef CONFIG_AMD_MEM_ENCRYPT
+ ISTG(X86_TRAP_VC, asm_exc_vmm_communication, IST_INDEX_VC),
#endif
};
diff --git a/arch/x86/kernel/sev-es.c b/arch/x86/kernel/sev-es.c
index 69c55f0fdf6a..cb2f6236a98e 100644
--- a/arch/x86/kernel/sev-es.c
+++ b/arch/x86/kernel/sev-es.c
@@ -7,9 +7,12 @@
* Author: Joerg Roedel <jroedel@suse.de>
*/
+#define pr_fmt(fmt) "SEV-ES: " fmt
+
#include <linux/sched/debug.h> /* For show_regs() */
#include <linux/percpu-defs.h>
#include <linux/mem_encrypt.h>
+#include <linux/lockdep.h>
#include <linux/printk.h>
#include <linux/mm_types.h>
#include <linux/set_memory.h>
@@ -22,8 +25,8 @@
#include <asm/insn-eval.h>
#include <asm/fpu/internal.h>
#include <asm/processor.h>
-#include <asm/trap_pf.h>
-#include <asm/trapnr.h>
+#include <asm/realmode.h>
+#include <asm/traps.h>
#include <asm/svm.h>
/* For early boot hypervisor communication in SEV-ES enabled guests */
@@ -48,11 +51,43 @@ struct sev_es_runtime_data {
* interrupted stack in the #VC entry code.
*/
char fallback_stack[EXCEPTION_STKSZ] __aligned(PAGE_SIZE);
+
+ /*
+ * Reserve one page per CPU as backup storage for the unencrypted GHCB.
+ * It is needed when an NMI happens while the #VC handler uses the real
+ * GHCB, and the NMI handler itself is causing another #VC exception. In
+ * that case the GHCB content of the first handler needs to be backed up
+ * and restored.
+ */
+ struct ghcb backup_ghcb;
+
+ /*
+ * Mark the per-cpu GHCBs as in-use to detect nested #VC exceptions.
+ * There is no need for it to be atomic, because nothing is written to
+ * the GHCB between the read and the write of ghcb_active. So it is safe
+ * to use it when a nested #VC exception happens before the write.
+ *
+ * This is necessary for example in the #VC->NMI->#VC case when the NMI
+ * happens while the first #VC handler uses the GHCB. When the NMI code
+ * raises a second #VC handler it might overwrite the contents of the
+ * GHCB written by the first handler. To avoid this the content of the
+ * GHCB is saved and restored when the GHCB is detected to be in use
+ * already.
+ */
+ bool ghcb_active;
+ bool backup_ghcb_active;
+};
+
+struct ghcb_state {
+ struct ghcb *ghcb;
};
static DEFINE_PER_CPU(struct sev_es_runtime_data*, runtime_data);
DEFINE_STATIC_KEY_FALSE(sev_es_enable_key);
+/* Needed in vc_early_forward_exception */
+void do_early_exception(struct pt_regs *regs, int trapnr);
+
static void __init setup_vc_stacks(int cpu)
{
struct sev_es_runtime_data *data;
@@ -123,8 +158,52 @@ void noinstr __sev_es_ist_exit(void)
this_cpu_write(cpu_tss_rw.x86_tss.ist[IST_INDEX_VC], *(unsigned long *)ist);
}
-/* Needed in vc_early_forward_exception */
-void do_early_exception(struct pt_regs *regs, int trapnr);
+static __always_inline struct ghcb *sev_es_get_ghcb(struct ghcb_state *state)
+{
+ struct sev_es_runtime_data *data;
+ struct ghcb *ghcb;
+
+ data = this_cpu_read(runtime_data);
+ ghcb = &data->ghcb_page;
+
+ if (unlikely(data->ghcb_active)) {
+ /* GHCB is already in use - save its contents */
+
+ if (unlikely(data->backup_ghcb_active))
+ return NULL;
+
+ /* Mark backup_ghcb active before writing to it */
+ data->backup_ghcb_active = true;
+
+ state->ghcb = &data->backup_ghcb;
+
+ /* Backup GHCB content */
+ *state->ghcb = *ghcb;
+ } else {
+ state->ghcb = NULL;
+ data->ghcb_active = true;
+ }
+
+ return ghcb;
+}
+
+static __always_inline void sev_es_put_ghcb(struct ghcb_state *state)
+{
+ struct sev_es_runtime_data *data;
+ struct ghcb *ghcb;
+
+ data = this_cpu_read(runtime_data);
+ ghcb = &data->ghcb_page;
+
+ if (state->ghcb) {
+ /* Restore GHCB from Backup */
+ *ghcb = *state->ghcb;
+ data->backup_ghcb_active = false;
+ state->ghcb = NULL;
+ } else {
+ data->ghcb_active = false;
+ }
+}
static inline u64 sev_es_rd_ghcb_msr(void)
{
@@ -316,6 +395,9 @@ static void __init init_ghcb(int cpu)
panic("Can not map GHCBs unencrypted");
memset(&data->ghcb_page, 0, sizeof(data->ghcb_page));
+
+ data->ghcb_active = false;
+ data->backup_ghcb_active = false;
}
void __init sev_es_init_vc_handling(void)
@@ -336,6 +418,9 @@ void __init sev_es_init_vc_handling(void)
init_ghcb(cpu);
setup_vc_stacks(cpu);
}
+
+ /* Secondary CPUs use the runtime #VC handler */
+ initial_vc_handler = (unsigned long)safe_stack_exc_vmm_communication;
}
static void __init vc_early_forward_exception(struct es_em_ctxt *ctxt)
@@ -366,6 +451,159 @@ static enum es_result vc_handle_exitcode(struct es_em_ctxt *ctxt,
return result;
}
+static __always_inline void vc_forward_exception(struct es_em_ctxt *ctxt)
+{
+ long error_code = ctxt->fi.error_code;
+ int trapnr = ctxt->fi.vector;
+
+ ctxt->regs->orig_ax = ctxt->fi.error_code;
+
+ switch (trapnr) {
+ case X86_TRAP_GP:
+ exc_general_protection(ctxt->regs, error_code);
+ break;
+ case X86_TRAP_UD:
+ exc_invalid_op(ctxt->regs);
+ break;
+ default:
+ pr_emerg("Unsupported exception in #VC instruction emulation - can't continue\n");
+ BUG();
+ }
+}
+
+static __always_inline bool on_vc_fallback_stack(struct pt_regs *regs)
+{
+ unsigned long sp = (unsigned long)regs;
+
+ return (sp >= __this_cpu_ist_bottom_va(VC2) && sp < __this_cpu_ist_top_va(VC2));
+}
+
+/*
+ * Main #VC exception handler. It is called when the entry code was able to
+ * switch off the IST to a safe kernel stack.
+ *
+ * With the current implementation it is always possible to switch to a safe
+ * stack because #VC exceptions only happen at known places, like intercepted
+ * instructions or accesses to MMIO areas/IO ports. They can also happen with
+ * code instrumentation when the hypervisor intercepts #DB, but the critical
+ * paths are forbidden to be instrumented, so #DB exceptions currently also
+ * only happen in safe places.
+ */
+DEFINE_IDTENTRY_VC_SAFE_STACK(exc_vmm_communication)
+{
+ struct sev_es_runtime_data *data = this_cpu_read(runtime_data);
+ struct ghcb_state state;
+ struct es_em_ctxt ctxt;
+ enum es_result result;
+ struct ghcb *ghcb;
+
+ lockdep_assert_irqs_disabled();
+ instrumentation_begin();
+
+ /*
+ * This is invoked through an interrupt gate, so IRQs are disabled. The
+ * code below might walk page-tables for user or kernel addresses, so
+ * keep the IRQs disabled to protect us against concurrent TLB flushes.
+ */
+
+ ghcb = sev_es_get_ghcb(&state);
+ if (!ghcb) {
+ /*
+ * Mark GHCBs inactive so that panic() is able to print the
+ * message.
+ */
+ data->ghcb_active = false;
+ data->backup_ghcb_active = false;
+
+ panic("Unable to handle #VC exception! GHCB and Backup GHCB are already in use");
+ }
+
+ vc_ghcb_invalidate(ghcb);
+ result = vc_init_em_ctxt(&ctxt, regs, error_code);
+
+ if (result == ES_OK)
+ result = vc_handle_exitcode(&ctxt, ghcb, error_code);
+
+ sev_es_put_ghcb(&state);
+
+ /* Done - now check the result */
+ switch (result) {
+ case ES_OK:
+ vc_finish_insn(&ctxt);
+ break;
+ case ES_UNSUPPORTED:
+ pr_err_ratelimited("Unsupported exit-code 0x%02lx in early #VC exception (IP: 0x%lx)\n",
+ error_code, regs->ip);
+ goto fail;
+ case ES_VMM_ERROR:
+ pr_err_ratelimited("Failure in communication with VMM (exit-code 0x%02lx IP: 0x%lx)\n",
+ error_code, regs->ip);
+ goto fail;
+ case ES_DECODE_FAILED:
+ pr_err_ratelimited("Failed to decode instruction (exit-code 0x%02lx IP: 0x%lx)\n",
+ error_code, regs->ip);
+ goto fail;
+ case ES_EXCEPTION:
+ vc_forward_exception(&ctxt);
+ break;
+ case ES_RETRY:
+ /* Nothing to do */
+ break;
+ default:
+ pr_emerg("Unknown result in %s():%d\n", __func__, result);
+ /*
+ * Emulating the instruction which caused the #VC exception
+ * failed - can't continue so print debug information
+ */
+ BUG();
+ }
+
+out:
+ instrumentation_end();
+
+ return;
+
+fail:
+ if (user_mode(regs)) {
+ /*
+ * Do not kill the machine if user-space triggered the
+ * exception. Send SIGBUS instead and let user-space deal with
+ * it.
+ */
+ force_sig_fault(SIGBUS, BUS_OBJERR, (void __user *)0);
+ } else {
+ pr_emerg("PANIC: Unhandled #VC exception in kernel space (result=%d)\n",
+ result);
+
+ /* Show some debug info */
+ show_regs(regs);
+
+ /* Ask hypervisor to sev_es_terminate */
+ sev_es_terminate(GHCB_SEV_ES_REASON_GENERAL_REQUEST);
+
+ /* If that fails and we get here - just panic */
+ panic("Returned from Terminate-Request to Hypervisor\n");
+ }
+
+ goto out;
+}
+
+/* This handler runs on the #VC fall-back stack. It can cause further #VC exceptions */
+DEFINE_IDTENTRY_VC_IST(exc_vmm_communication)
+{
+ instrumentation_begin();
+ panic("Can't handle #VC exception from unsupported context\n");
+ instrumentation_end();
+}
+
+DEFINE_IDTENTRY_VC(exc_vmm_communication)
+{
+ if (likely(!on_vc_fallback_stack(regs)))
+ safe_stack_exc_vmm_communication(regs, error_code);
+ else
+ ist_exc_vmm_communication(regs, error_code);
+}
+
bool __init handle_vc_boot_ghcb(struct pt_regs *regs)
{
unsigned long exit_code = regs->orig_ax;
--
2.28.0
_______________________________________________
Virtualization mailing list
Virtualization@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/virtualization
^ permalink raw reply related [flat|nested] 264+ messages in thread
* [PATCH v7 47/72] x86/sev-es: Wire up existing #VC exit-code handlers
2020-09-07 13:15 ` Joerg Roedel
@ 2020-09-07 13:15 ` Joerg Roedel
-1 siblings, 0 replies; 264+ messages in thread
From: Joerg Roedel @ 2020-09-07 13:15 UTC (permalink / raw)
To: x86
Cc: Joerg Roedel, Joerg Roedel, hpa, Andy Lutomirski, Dave Hansen,
Peter Zijlstra, Jiri Slaby, Dan Williams, Tom Lendacky,
Juergen Gross, Kees Cook, David Rientjes, Cfir Cohen,
Erdem Aktas, Masami Hiramatsu, Mike Stunes, Sean Christopherson,
Martin Radev, linux-kernel, kvm, virtualization
From: Joerg Roedel <jroedel@suse.de>
Re-use the handlers for CPUID and IOIO caused #VC exceptions in the
early boot handler.
Signed-off-by: Joerg Roedel <jroedel@suse.de>
---
arch/x86/kernel/sev-es-shared.c | 7 +++----
arch/x86/kernel/sev-es.c | 6 ++++++
2 files changed, 9 insertions(+), 4 deletions(-)
diff --git a/arch/x86/kernel/sev-es-shared.c b/arch/x86/kernel/sev-es-shared.c
index aa77f2eb8d88..a6b41910b8ab 100644
--- a/arch/x86/kernel/sev-es-shared.c
+++ b/arch/x86/kernel/sev-es-shared.c
@@ -325,8 +325,7 @@ static enum es_result vc_ioio_exitinfo(struct es_em_ctxt *ctxt, u64 *exitinfo)
return ES_OK;
}
-static enum es_result __maybe_unused
-vc_handle_ioio(struct ghcb *ghcb, struct es_em_ctxt *ctxt)
+static enum es_result vc_handle_ioio(struct ghcb *ghcb, struct es_em_ctxt *ctxt)
{
struct pt_regs *regs = ctxt->regs;
u64 exit_info_1, exit_info_2;
@@ -434,8 +433,8 @@ vc_handle_ioio(struct ghcb *ghcb, struct es_em_ctxt *ctxt)
return ret;
}
-static enum es_result __maybe_unused vc_handle_cpuid(struct ghcb *ghcb,
- struct es_em_ctxt *ctxt)
+static enum es_result vc_handle_cpuid(struct ghcb *ghcb,
+ struct es_em_ctxt *ctxt)
{
struct pt_regs *regs = ctxt->regs;
u32 cr4 = native_read_cr4();
diff --git a/arch/x86/kernel/sev-es.c b/arch/x86/kernel/sev-es.c
index cb2f6236a98e..42c84b9b6b4f 100644
--- a/arch/x86/kernel/sev-es.c
+++ b/arch/x86/kernel/sev-es.c
@@ -441,6 +441,12 @@ static enum es_result vc_handle_exitcode(struct es_em_ctxt *ctxt,
enum es_result result;
switch (exit_code) {
+ case SVM_EXIT_CPUID:
+ result = vc_handle_cpuid(ghcb, ctxt);
+ break;
+ case SVM_EXIT_IOIO:
+ result = vc_handle_ioio(ghcb, ctxt);
+ break;
default:
/*
* Unexpected #VC exception
--
2.28.0
^ permalink raw reply related [flat|nested] 264+ messages in thread
* [PATCH v7 47/72] x86/sev-es: Wire up existing #VC exit-code handlers
@ 2020-09-07 13:15 ` Joerg Roedel
0 siblings, 0 replies; 264+ messages in thread
From: Joerg Roedel @ 2020-09-07 13:15 UTC (permalink / raw)
To: x86
Cc: Juergen Gross, Tom Lendacky, Joerg Roedel, Mike Stunes,
Kees Cook, kvm, Peter Zijlstra, Cfir Cohen, Joerg Roedel,
Dave Hansen, linux-kernel, Sean Christopherson, virtualization,
Martin Radev, Masami Hiramatsu, Andy Lutomirski, hpa,
Erdem Aktas, David Rientjes, Dan Williams, Jiri Slaby
From: Joerg Roedel <jroedel@suse.de>
Re-use the handlers for CPUID and IOIO caused #VC exceptions in the
early boot handler.
Signed-off-by: Joerg Roedel <jroedel@suse.de>
---
arch/x86/kernel/sev-es-shared.c | 7 +++----
arch/x86/kernel/sev-es.c | 6 ++++++
2 files changed, 9 insertions(+), 4 deletions(-)
diff --git a/arch/x86/kernel/sev-es-shared.c b/arch/x86/kernel/sev-es-shared.c
index aa77f2eb8d88..a6b41910b8ab 100644
--- a/arch/x86/kernel/sev-es-shared.c
+++ b/arch/x86/kernel/sev-es-shared.c
@@ -325,8 +325,7 @@ static enum es_result vc_ioio_exitinfo(struct es_em_ctxt *ctxt, u64 *exitinfo)
return ES_OK;
}
-static enum es_result __maybe_unused
-vc_handle_ioio(struct ghcb *ghcb, struct es_em_ctxt *ctxt)
+static enum es_result vc_handle_ioio(struct ghcb *ghcb, struct es_em_ctxt *ctxt)
{
struct pt_regs *regs = ctxt->regs;
u64 exit_info_1, exit_info_2;
@@ -434,8 +433,8 @@ vc_handle_ioio(struct ghcb *ghcb, struct es_em_ctxt *ctxt)
return ret;
}
-static enum es_result __maybe_unused vc_handle_cpuid(struct ghcb *ghcb,
- struct es_em_ctxt *ctxt)
+static enum es_result vc_handle_cpuid(struct ghcb *ghcb,
+ struct es_em_ctxt *ctxt)
{
struct pt_regs *regs = ctxt->regs;
u32 cr4 = native_read_cr4();
diff --git a/arch/x86/kernel/sev-es.c b/arch/x86/kernel/sev-es.c
index cb2f6236a98e..42c84b9b6b4f 100644
--- a/arch/x86/kernel/sev-es.c
+++ b/arch/x86/kernel/sev-es.c
@@ -441,6 +441,12 @@ static enum es_result vc_handle_exitcode(struct es_em_ctxt *ctxt,
enum es_result result;
switch (exit_code) {
+ case SVM_EXIT_CPUID:
+ result = vc_handle_cpuid(ghcb, ctxt);
+ break;
+ case SVM_EXIT_IOIO:
+ result = vc_handle_ioio(ghcb, ctxt);
+ break;
default:
/*
* Unexpected #VC exception
--
2.28.0
_______________________________________________
Virtualization mailing list
Virtualization@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/virtualization
^ permalink raw reply related [flat|nested] 264+ messages in thread
* [PATCH v7 48/72] x86/sev-es: Handle instruction fetches from user-space
2020-09-07 13:15 ` Joerg Roedel
@ 2020-09-07 13:15 ` Joerg Roedel
-1 siblings, 0 replies; 264+ messages in thread
From: Joerg Roedel @ 2020-09-07 13:15 UTC (permalink / raw)
To: x86
Cc: Joerg Roedel, Joerg Roedel, hpa, Andy Lutomirski, Dave Hansen,
Peter Zijlstra, Jiri Slaby, Dan Williams, Tom Lendacky,
Juergen Gross, Kees Cook, David Rientjes, Cfir Cohen,
Erdem Aktas, Masami Hiramatsu, Mike Stunes, Sean Christopherson,
Martin Radev, linux-kernel, kvm, virtualization
From: Joerg Roedel <jroedel@suse.de>
When a #VC exception is triggered by user-space the instruction decoder
needs to read the instruction bytes from user addresses. Enhance
vc_decode_insn() to safely fetch kernel and user instructions.
Signed-off-by: Joerg Roedel <jroedel@suse.de>
---
arch/x86/kernel/sev-es.c | 31 ++++++++++++++++++++++---------
1 file changed, 22 insertions(+), 9 deletions(-)
diff --git a/arch/x86/kernel/sev-es.c b/arch/x86/kernel/sev-es.c
index 42c84b9b6b4f..bc2854b81a51 100644
--- a/arch/x86/kernel/sev-es.c
+++ b/arch/x86/kernel/sev-es.c
@@ -232,17 +232,30 @@ static enum es_result vc_decode_insn(struct es_em_ctxt *ctxt)
enum es_result ret;
int res;
- res = vc_fetch_insn_kernel(ctxt, buffer);
- if (unlikely(res == -EFAULT)) {
- ctxt->fi.vector = X86_TRAP_PF;
- ctxt->fi.error_code = 0;
- ctxt->fi.cr2 = ctxt->regs->ip;
- return ES_EXCEPTION;
+ if (user_mode(ctxt->regs)) {
+ res = insn_fetch_from_user(ctxt->regs, buffer);
+ if (!res) {
+ ctxt->fi.vector = X86_TRAP_PF;
+ ctxt->fi.error_code = X86_PF_INSTR | X86_PF_USER;
+ ctxt->fi.cr2 = ctxt->regs->ip;
+ return ES_EXCEPTION;
+ }
+
+ if (!insn_decode(&ctxt->insn, ctxt->regs, buffer, res))
+ return ES_DECODE_FAILED;
+ } else {
+ res = vc_fetch_insn_kernel(ctxt, buffer);
+ if (res) {
+ ctxt->fi.vector = X86_TRAP_PF;
+ ctxt->fi.error_code = X86_PF_INSTR;
+ ctxt->fi.cr2 = ctxt->regs->ip;
+ return ES_EXCEPTION;
+ }
+
+ insn_init(&ctxt->insn, buffer, MAX_INSN_SIZE - res, 1);
+ insn_get_length(&ctxt->insn);
}
- insn_init(&ctxt->insn, buffer, MAX_INSN_SIZE - res, 1);
- insn_get_length(&ctxt->insn);
-
ret = ctxt->insn.immediate.got ? ES_OK : ES_DECODE_FAILED;
return ret;
--
2.28.0
^ permalink raw reply related [flat|nested] 264+ messages in thread
* [PATCH v7 48/72] x86/sev-es: Handle instruction fetches from user-space
@ 2020-09-07 13:15 ` Joerg Roedel
0 siblings, 0 replies; 264+ messages in thread
From: Joerg Roedel @ 2020-09-07 13:15 UTC (permalink / raw)
To: x86
Cc: Juergen Gross, Tom Lendacky, Joerg Roedel, Mike Stunes,
Kees Cook, kvm, Peter Zijlstra, Cfir Cohen, Joerg Roedel,
Dave Hansen, linux-kernel, Sean Christopherson, virtualization,
Martin Radev, Masami Hiramatsu, Andy Lutomirski, hpa,
Erdem Aktas, David Rientjes, Dan Williams, Jiri Slaby
From: Joerg Roedel <jroedel@suse.de>
When a #VC exception is triggered by user-space the instruction decoder
needs to read the instruction bytes from user addresses. Enhance
vc_decode_insn() to safely fetch kernel and user instructions.
Signed-off-by: Joerg Roedel <jroedel@suse.de>
---
arch/x86/kernel/sev-es.c | 31 ++++++++++++++++++++++---------
1 file changed, 22 insertions(+), 9 deletions(-)
diff --git a/arch/x86/kernel/sev-es.c b/arch/x86/kernel/sev-es.c
index 42c84b9b6b4f..bc2854b81a51 100644
--- a/arch/x86/kernel/sev-es.c
+++ b/arch/x86/kernel/sev-es.c
@@ -232,17 +232,30 @@ static enum es_result vc_decode_insn(struct es_em_ctxt *ctxt)
enum es_result ret;
int res;
- res = vc_fetch_insn_kernel(ctxt, buffer);
- if (unlikely(res == -EFAULT)) {
- ctxt->fi.vector = X86_TRAP_PF;
- ctxt->fi.error_code = 0;
- ctxt->fi.cr2 = ctxt->regs->ip;
- return ES_EXCEPTION;
+ if (user_mode(ctxt->regs)) {
+ res = insn_fetch_from_user(ctxt->regs, buffer);
+ if (!res) {
+ ctxt->fi.vector = X86_TRAP_PF;
+ ctxt->fi.error_code = X86_PF_INSTR | X86_PF_USER;
+ ctxt->fi.cr2 = ctxt->regs->ip;
+ return ES_EXCEPTION;
+ }
+
+ if (!insn_decode(&ctxt->insn, ctxt->regs, buffer, res))
+ return ES_DECODE_FAILED;
+ } else {
+ res = vc_fetch_insn_kernel(ctxt, buffer);
+ if (res) {
+ ctxt->fi.vector = X86_TRAP_PF;
+ ctxt->fi.error_code = X86_PF_INSTR;
+ ctxt->fi.cr2 = ctxt->regs->ip;
+ return ES_EXCEPTION;
+ }
+
+ insn_init(&ctxt->insn, buffer, MAX_INSN_SIZE - res, 1);
+ insn_get_length(&ctxt->insn);
}
- insn_init(&ctxt->insn, buffer, MAX_INSN_SIZE - res, 1);
- insn_get_length(&ctxt->insn);
-
ret = ctxt->insn.immediate.got ? ES_OK : ES_DECODE_FAILED;
return ret;
--
2.28.0
_______________________________________________
Virtualization mailing list
Virtualization@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/virtualization
^ permalink raw reply related [flat|nested] 264+ messages in thread
* [PATCH v7 49/72] x86/sev-es: Handle MMIO events
2020-09-07 13:15 ` Joerg Roedel
@ 2020-09-07 13:15 ` Joerg Roedel
-1 siblings, 0 replies; 264+ messages in thread
From: Joerg Roedel @ 2020-09-07 13:15 UTC (permalink / raw)
To: x86
Cc: Joerg Roedel, Joerg Roedel, Tom Lendacky, hpa, Andy Lutomirski,
Dave Hansen, Peter Zijlstra, Jiri Slaby, Dan Williams,
Juergen Gross, Kees Cook, David Rientjes, Cfir Cohen,
Erdem Aktas, Masami Hiramatsu, Mike Stunes, Sean Christopherson,
Martin Radev, linux-kernel, kvm, virtualization
From: Tom Lendacky <thomas.lendacky@amd.com>
Add handler for VC exceptions caused by MMIO intercepts. These
intercepts come along as nested page faults on pages with reserved
bits set.
Signed-off-by: Tom Lendacky <thomas.lendacky@amd.com>
[ jroedel@suse.de: Adapt to VC handling framework ]
Co-developed-by: Joerg Roedel <jroedel@suse.de>
Signed-off-by: Joerg Roedel <jroedel@suse.de>
---
arch/x86/include/uapi/asm/svm.h | 5 +
arch/x86/kernel/sev-es.c | 222 ++++++++++++++++++++++++++++++++
2 files changed, 227 insertions(+)
diff --git a/arch/x86/include/uapi/asm/svm.h b/arch/x86/include/uapi/asm/svm.h
index c68d1618c9b0..8f36ae021a7f 100644
--- a/arch/x86/include/uapi/asm/svm.h
+++ b/arch/x86/include/uapi/asm/svm.h
@@ -81,6 +81,11 @@
#define SVM_EXIT_AVIC_INCOMPLETE_IPI 0x401
#define SVM_EXIT_AVIC_UNACCELERATED_ACCESS 0x402
+/* SEV-ES software-defined VMGEXIT events */
+#define SVM_VMGEXIT_MMIO_READ 0x80000001
+#define SVM_VMGEXIT_MMIO_WRITE 0x80000002
+#define SVM_VMGEXIT_UNSUPPORTED_EVENT 0x8000ffff
+
#define SVM_EXIT_ERR -1
#define SVM_EXIT_REASONS \
diff --git a/arch/x86/kernel/sev-es.c b/arch/x86/kernel/sev-es.c
index bc2854b81a51..62fd0ebf2a67 100644
--- a/arch/x86/kernel/sev-es.c
+++ b/arch/x86/kernel/sev-es.c
@@ -359,6 +359,37 @@ static enum es_result vc_read_mem(struct es_em_ctxt *ctxt,
return ES_EXCEPTION;
}
+static bool vc_slow_virt_to_phys(struct ghcb *ghcb, struct es_em_ctxt *ctxt,
+ unsigned long vaddr, phys_addr_t *paddr)
+{
+ unsigned long va = (unsigned long)vaddr;
+ unsigned int level;
+ phys_addr_t pa;
+ pgd_t *pgd;
+ pte_t *pte;
+
+ pgd = __va(read_cr3_pa());
+ pgd = &pgd[pgd_index(va)];
+ pte = lookup_address_in_pgd(pgd, va, &level);
+ if (!pte) {
+ ctxt->fi.vector = X86_TRAP_PF;
+ ctxt->fi.cr2 = vaddr;
+ ctxt->fi.error_code = 0;
+
+ if (user_mode(ctxt->regs))
+ ctxt->fi.error_code |= X86_PF_USER;
+
+ return false;
+ }
+
+ pa = (phys_addr_t)pte_pfn(*pte) << PAGE_SHIFT;
+ pa |= va & ~page_level_mask(level);
+
+ *paddr = pa;
+
+ return true;
+}
+
/* Include code shared with pre-decompression boot stage */
#include "sev-es-shared.c"
@@ -447,6 +478,194 @@ static void __init vc_early_forward_exception(struct es_em_ctxt *ctxt)
do_early_exception(ctxt->regs, trapnr);
}
+static long *vc_insn_get_reg(struct es_em_ctxt *ctxt)
+{
+ long *reg_array;
+ int offset;
+
+ reg_array = (long *)ctxt->regs;
+ offset = insn_get_modrm_reg_off(&ctxt->insn, ctxt->regs);
+
+ if (offset < 0)
+ return NULL;
+
+ offset /= sizeof(long);
+
+ return reg_array + offset;
+}
+
+static enum es_result vc_do_mmio(struct ghcb *ghcb, struct es_em_ctxt *ctxt,
+ unsigned int bytes, bool read)
+{
+ u64 exit_code, exit_info_1, exit_info_2;
+ unsigned long ghcb_pa = __pa(ghcb);
+ phys_addr_t paddr;
+ void __user *ref;
+
+ ref = insn_get_addr_ref(&ctxt->insn, ctxt->regs);
+ if (ref == (void __user *)-1L)
+ return ES_UNSUPPORTED;
+
+ exit_code = read ? SVM_VMGEXIT_MMIO_READ : SVM_VMGEXIT_MMIO_WRITE;
+
+ if (!vc_slow_virt_to_phys(ghcb, ctxt, (unsigned long)ref, &paddr)) {
+ if (!read)
+ ctxt->fi.error_code |= X86_PF_WRITE;
+
+ return ES_EXCEPTION;
+ }
+
+ exit_info_1 = paddr;
+ /* Can never be greater than 8 */
+ exit_info_2 = bytes;
+
+ ghcb->save.sw_scratch = ghcb_pa + offsetof(struct ghcb, shared_buffer);
+
+ return sev_es_ghcb_hv_call(ghcb, ctxt, exit_code, exit_info_1, exit_info_2);
+}
+
+static enum es_result vc_handle_mmio_twobyte_ops(struct ghcb *ghcb,
+ struct es_em_ctxt *ctxt)
+{
+ struct insn *insn = &ctxt->insn;
+ unsigned int bytes = 0;
+ enum es_result ret;
+ int sign_byte;
+ long *reg_data;
+
+ switch (insn->opcode.bytes[1]) {
+ /* MMIO Read w/ zero-extension */
+ case 0xb6:
+ bytes = 1;
+ fallthrough;
+ case 0xb7:
+ if (!bytes)
+ bytes = 2;
+
+ ret = vc_do_mmio(ghcb, ctxt, bytes, true);
+ if (ret)
+ break;
+
+ /* Zero extend based on operand size */
+ reg_data = vc_insn_get_reg(ctxt);
+ if (!reg_data)
+ return ES_DECODE_FAILED;
+
+ memset(reg_data, 0, insn->opnd_bytes);
+
+ memcpy(reg_data, ghcb->shared_buffer, bytes);
+ break;
+
+ /* MMIO Read w/ sign-extension */
+ case 0xbe:
+ bytes = 1;
+ fallthrough;
+ case 0xbf:
+ if (!bytes)
+ bytes = 2;
+
+ ret = vc_do_mmio(ghcb, ctxt, bytes, true);
+ if (ret)
+ break;
+
+ /* Sign extend based on operand size */
+ reg_data = vc_insn_get_reg(ctxt);
+ if (!reg_data)
+ return ES_DECODE_FAILED;
+
+ if (bytes == 1) {
+ u8 *val = (u8 *)ghcb->shared_buffer;
+
+ sign_byte = (*val & 0x80) ? 0xff : 0x00;
+ } else {
+ u16 *val = (u16 *)ghcb->shared_buffer;
+
+ sign_byte = (*val & 0x8000) ? 0xff : 0x00;
+ }
+ memset(reg_data, sign_byte, insn->opnd_bytes);
+
+ memcpy(reg_data, ghcb->shared_buffer, bytes);
+ break;
+
+ default:
+ ret = ES_UNSUPPORTED;
+ }
+
+ return ret;
+}
+
+static enum es_result vc_handle_mmio(struct ghcb *ghcb,
+ struct es_em_ctxt *ctxt)
+{
+ struct insn *insn = &ctxt->insn;
+ unsigned int bytes = 0;
+ enum es_result ret;
+ long *reg_data;
+
+ switch (insn->opcode.bytes[0]) {
+ /* MMIO Write */
+ case 0x88:
+ bytes = 1;
+ fallthrough;
+ case 0x89:
+ if (!bytes)
+ bytes = insn->opnd_bytes;
+
+ reg_data = vc_insn_get_reg(ctxt);
+ if (!reg_data)
+ return ES_DECODE_FAILED;
+
+ memcpy(ghcb->shared_buffer, reg_data, bytes);
+
+ ret = vc_do_mmio(ghcb, ctxt, bytes, false);
+ break;
+
+ case 0xc6:
+ bytes = 1;
+ fallthrough;
+ case 0xc7:
+ if (!bytes)
+ bytes = insn->opnd_bytes;
+
+ memcpy(ghcb->shared_buffer, insn->immediate1.bytes, bytes);
+
+ ret = vc_do_mmio(ghcb, ctxt, bytes, false);
+ break;
+
+ /* MMIO Read */
+ case 0x8a:
+ bytes = 1;
+ fallthrough;
+ case 0x8b:
+ if (!bytes)
+ bytes = insn->opnd_bytes;
+
+ ret = vc_do_mmio(ghcb, ctxt, bytes, true);
+ if (ret)
+ break;
+
+ reg_data = vc_insn_get_reg(ctxt);
+ if (!reg_data)
+ return ES_DECODE_FAILED;
+
+ /* Zero-extend for 32-bit operation */
+ if (bytes == 4)
+ *reg_data = 0;
+
+ memcpy(reg_data, ghcb->shared_buffer, bytes);
+ break;
+
+ /* Two-Byte Opcodes */
+ case 0x0f:
+ ret = vc_handle_mmio_twobyte_ops(ghcb, ctxt);
+ break;
+ default:
+ ret = ES_UNSUPPORTED;
+ }
+
+ return ret;
+}
+
static enum es_result vc_handle_exitcode(struct es_em_ctxt *ctxt,
struct ghcb *ghcb,
unsigned long exit_code)
@@ -460,6 +679,9 @@ static enum es_result vc_handle_exitcode(struct es_em_ctxt *ctxt,
case SVM_EXIT_IOIO:
result = vc_handle_ioio(ghcb, ctxt);
break;
+ case SVM_EXIT_NPF:
+ result = vc_handle_mmio(ghcb, ctxt);
+ break;
default:
/*
* Unexpected #VC exception
--
2.28.0
^ permalink raw reply related [flat|nested] 264+ messages in thread
* [PATCH v7 49/72] x86/sev-es: Handle MMIO events
@ 2020-09-07 13:15 ` Joerg Roedel
0 siblings, 0 replies; 264+ messages in thread
From: Joerg Roedel @ 2020-09-07 13:15 UTC (permalink / raw)
To: x86
Cc: Juergen Gross, Tom Lendacky, Joerg Roedel, Mike Stunes,
Kees Cook, kvm, Peter Zijlstra, Cfir Cohen, Joerg Roedel,
Dave Hansen, linux-kernel, Sean Christopherson, virtualization,
Martin Radev, Masami Hiramatsu, Andy Lutomirski, hpa,
Erdem Aktas, David Rientjes, Dan Williams, Jiri Slaby
From: Tom Lendacky <thomas.lendacky@amd.com>
Add handler for VC exceptions caused by MMIO intercepts. These
intercepts come along as nested page faults on pages with reserved
bits set.
Signed-off-by: Tom Lendacky <thomas.lendacky@amd.com>
[ jroedel@suse.de: Adapt to VC handling framework ]
Co-developed-by: Joerg Roedel <jroedel@suse.de>
Signed-off-by: Joerg Roedel <jroedel@suse.de>
---
arch/x86/include/uapi/asm/svm.h | 5 +
arch/x86/kernel/sev-es.c | 222 ++++++++++++++++++++++++++++++++
2 files changed, 227 insertions(+)
diff --git a/arch/x86/include/uapi/asm/svm.h b/arch/x86/include/uapi/asm/svm.h
index c68d1618c9b0..8f36ae021a7f 100644
--- a/arch/x86/include/uapi/asm/svm.h
+++ b/arch/x86/include/uapi/asm/svm.h
@@ -81,6 +81,11 @@
#define SVM_EXIT_AVIC_INCOMPLETE_IPI 0x401
#define SVM_EXIT_AVIC_UNACCELERATED_ACCESS 0x402
+/* SEV-ES software-defined VMGEXIT events */
+#define SVM_VMGEXIT_MMIO_READ 0x80000001
+#define SVM_VMGEXIT_MMIO_WRITE 0x80000002
+#define SVM_VMGEXIT_UNSUPPORTED_EVENT 0x8000ffff
+
#define SVM_EXIT_ERR -1
#define SVM_EXIT_REASONS \
diff --git a/arch/x86/kernel/sev-es.c b/arch/x86/kernel/sev-es.c
index bc2854b81a51..62fd0ebf2a67 100644
--- a/arch/x86/kernel/sev-es.c
+++ b/arch/x86/kernel/sev-es.c
@@ -359,6 +359,37 @@ static enum es_result vc_read_mem(struct es_em_ctxt *ctxt,
return ES_EXCEPTION;
}
+static bool vc_slow_virt_to_phys(struct ghcb *ghcb, struct es_em_ctxt *ctxt,
+ unsigned long vaddr, phys_addr_t *paddr)
+{
+ unsigned long va = (unsigned long)vaddr;
+ unsigned int level;
+ phys_addr_t pa;
+ pgd_t *pgd;
+ pte_t *pte;
+
+ pgd = __va(read_cr3_pa());
+ pgd = &pgd[pgd_index(va)];
+ pte = lookup_address_in_pgd(pgd, va, &level);
+ if (!pte) {
+ ctxt->fi.vector = X86_TRAP_PF;
+ ctxt->fi.cr2 = vaddr;
+ ctxt->fi.error_code = 0;
+
+ if (user_mode(ctxt->regs))
+ ctxt->fi.error_code |= X86_PF_USER;
+
+ return false;
+ }
+
+ pa = (phys_addr_t)pte_pfn(*pte) << PAGE_SHIFT;
+ pa |= va & ~page_level_mask(level);
+
+ *paddr = pa;
+
+ return true;
+}
+
/* Include code shared with pre-decompression boot stage */
#include "sev-es-shared.c"
@@ -447,6 +478,194 @@ static void __init vc_early_forward_exception(struct es_em_ctxt *ctxt)
do_early_exception(ctxt->regs, trapnr);
}
+static long *vc_insn_get_reg(struct es_em_ctxt *ctxt)
+{
+ long *reg_array;
+ int offset;
+
+ reg_array = (long *)ctxt->regs;
+ offset = insn_get_modrm_reg_off(&ctxt->insn, ctxt->regs);
+
+ if (offset < 0)
+ return NULL;
+
+ offset /= sizeof(long);
+
+ return reg_array + offset;
+}
+
+static enum es_result vc_do_mmio(struct ghcb *ghcb, struct es_em_ctxt *ctxt,
+ unsigned int bytes, bool read)
+{
+ u64 exit_code, exit_info_1, exit_info_2;
+ unsigned long ghcb_pa = __pa(ghcb);
+ phys_addr_t paddr;
+ void __user *ref;
+
+ ref = insn_get_addr_ref(&ctxt->insn, ctxt->regs);
+ if (ref == (void __user *)-1L)
+ return ES_UNSUPPORTED;
+
+ exit_code = read ? SVM_VMGEXIT_MMIO_READ : SVM_VMGEXIT_MMIO_WRITE;
+
+ if (!vc_slow_virt_to_phys(ghcb, ctxt, (unsigned long)ref, &paddr)) {
+ if (!read)
+ ctxt->fi.error_code |= X86_PF_WRITE;
+
+ return ES_EXCEPTION;
+ }
+
+ exit_info_1 = paddr;
+ /* Can never be greater than 8 */
+ exit_info_2 = bytes;
+
+ ghcb->save.sw_scratch = ghcb_pa + offsetof(struct ghcb, shared_buffer);
+
+ return sev_es_ghcb_hv_call(ghcb, ctxt, exit_code, exit_info_1, exit_info_2);
+}
+
+static enum es_result vc_handle_mmio_twobyte_ops(struct ghcb *ghcb,
+ struct es_em_ctxt *ctxt)
+{
+ struct insn *insn = &ctxt->insn;
+ unsigned int bytes = 0;
+ enum es_result ret;
+ int sign_byte;
+ long *reg_data;
+
+ switch (insn->opcode.bytes[1]) {
+ /* MMIO Read w/ zero-extension */
+ case 0xb6:
+ bytes = 1;
+ fallthrough;
+ case 0xb7:
+ if (!bytes)
+ bytes = 2;
+
+ ret = vc_do_mmio(ghcb, ctxt, bytes, true);
+ if (ret)
+ break;
+
+ /* Zero extend based on operand size */
+ reg_data = vc_insn_get_reg(ctxt);
+ if (!reg_data)
+ return ES_DECODE_FAILED;
+
+ memset(reg_data, 0, insn->opnd_bytes);
+
+ memcpy(reg_data, ghcb->shared_buffer, bytes);
+ break;
+
+ /* MMIO Read w/ sign-extension */
+ case 0xbe:
+ bytes = 1;
+ fallthrough;
+ case 0xbf:
+ if (!bytes)
+ bytes = 2;
+
+ ret = vc_do_mmio(ghcb, ctxt, bytes, true);
+ if (ret)
+ break;
+
+ /* Sign extend based on operand size */
+ reg_data = vc_insn_get_reg(ctxt);
+ if (!reg_data)
+ return ES_DECODE_FAILED;
+
+ if (bytes == 1) {
+ u8 *val = (u8 *)ghcb->shared_buffer;
+
+ sign_byte = (*val & 0x80) ? 0xff : 0x00;
+ } else {
+ u16 *val = (u16 *)ghcb->shared_buffer;
+
+ sign_byte = (*val & 0x8000) ? 0xff : 0x00;
+ }
+ memset(reg_data, sign_byte, insn->opnd_bytes);
+
+ memcpy(reg_data, ghcb->shared_buffer, bytes);
+ break;
+
+ default:
+ ret = ES_UNSUPPORTED;
+ }
+
+ return ret;
+}
+
+static enum es_result vc_handle_mmio(struct ghcb *ghcb,
+ struct es_em_ctxt *ctxt)
+{
+ struct insn *insn = &ctxt->insn;
+ unsigned int bytes = 0;
+ enum es_result ret;
+ long *reg_data;
+
+ switch (insn->opcode.bytes[0]) {
+ /* MMIO Write */
+ case 0x88:
+ bytes = 1;
+ fallthrough;
+ case 0x89:
+ if (!bytes)
+ bytes = insn->opnd_bytes;
+
+ reg_data = vc_insn_get_reg(ctxt);
+ if (!reg_data)
+ return ES_DECODE_FAILED;
+
+ memcpy(ghcb->shared_buffer, reg_data, bytes);
+
+ ret = vc_do_mmio(ghcb, ctxt, bytes, false);
+ break;
+
+ case 0xc6:
+ bytes = 1;
+ fallthrough;
+ case 0xc7:
+ if (!bytes)
+ bytes = insn->opnd_bytes;
+
+ memcpy(ghcb->shared_buffer, insn->immediate1.bytes, bytes);
+
+ ret = vc_do_mmio(ghcb, ctxt, bytes, false);
+ break;
+
+ /* MMIO Read */
+ case 0x8a:
+ bytes = 1;
+ fallthrough;
+ case 0x8b:
+ if (!bytes)
+ bytes = insn->opnd_bytes;
+
+ ret = vc_do_mmio(ghcb, ctxt, bytes, true);
+ if (ret)
+ break;
+
+ reg_data = vc_insn_get_reg(ctxt);
+ if (!reg_data)
+ return ES_DECODE_FAILED;
+
+ /* Zero-extend for 32-bit operation */
+ if (bytes == 4)
+ *reg_data = 0;
+
+ memcpy(reg_data, ghcb->shared_buffer, bytes);
+ break;
+
+ /* Two-Byte Opcodes */
+ case 0x0f:
+ ret = vc_handle_mmio_twobyte_ops(ghcb, ctxt);
+ break;
+ default:
+ ret = ES_UNSUPPORTED;
+ }
+
+ return ret;
+}
+
static enum es_result vc_handle_exitcode(struct es_em_ctxt *ctxt,
struct ghcb *ghcb,
unsigned long exit_code)
@@ -460,6 +679,9 @@ static enum es_result vc_handle_exitcode(struct es_em_ctxt *ctxt,
case SVM_EXIT_IOIO:
result = vc_handle_ioio(ghcb, ctxt);
break;
+ case SVM_EXIT_NPF:
+ result = vc_handle_mmio(ghcb, ctxt);
+ break;
default:
/*
* Unexpected #VC exception
--
2.28.0
_______________________________________________
Virtualization mailing list
Virtualization@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/virtualization
^ permalink raw reply related [flat|nested] 264+ messages in thread
* [PATCH v7 50/72] x86/sev-es: Handle MMIO String Instructions
2020-09-07 13:15 ` Joerg Roedel
@ 2020-09-07 13:15 ` Joerg Roedel
-1 siblings, 0 replies; 264+ messages in thread
From: Joerg Roedel @ 2020-09-07 13:15 UTC (permalink / raw)
To: x86
Cc: Joerg Roedel, Joerg Roedel, hpa, Andy Lutomirski, Dave Hansen,
Peter Zijlstra, Jiri Slaby, Dan Williams, Tom Lendacky,
Juergen Gross, Kees Cook, David Rientjes, Cfir Cohen,
Erdem Aktas, Masami Hiramatsu, Mike Stunes, Sean Christopherson,
Martin Radev, linux-kernel, kvm, virtualization
From: Joerg Roedel <jroedel@suse.de>
Add handling for emulation the MOVS instruction on MMIO regions, as done
by the memcpy_toio() and memcpy_fromio() functions.
Signed-off-by: Joerg Roedel <jroedel@suse.de>
---
arch/x86/kernel/sev-es.c | 77 ++++++++++++++++++++++++++++++++++++++++
1 file changed, 77 insertions(+)
diff --git a/arch/x86/kernel/sev-es.c b/arch/x86/kernel/sev-es.c
index 62fd0ebf2a67..a79c7fc32bc6 100644
--- a/arch/x86/kernel/sev-es.c
+++ b/arch/x86/kernel/sev-es.c
@@ -594,6 +594,73 @@ static enum es_result vc_handle_mmio_twobyte_ops(struct ghcb *ghcb,
return ret;
}
+/*
+ * The MOVS instruction has two memory operands, which raises the
+ * problem that it is not known whether the access to the source or the
+ * destination caused the #VC exception (and hence whether an MMIO read
+ * or write operation needs to be emulated).
+ *
+ * Instead of playing games with walking page-tables and trying to guess
+ * whether the source or destination is an MMIO range, split the move
+ * into two operations, a read and a write with only one memory operand.
+ * This will cause a nested #VC exception on the MMIO address which can
+ * then be handled.
+ *
+ * This implementation has the benefit that it also supports MOVS where
+ * source _and_ destination are MMIO regions.
+ *
+ * It will slow MOVS on MMIO down a lot, but in SEV-ES guests it is a
+ * rare operation. If it turns out to be a performance problem the split
+ * operations can be moved to memcpy_fromio() and memcpy_toio().
+ */
+static enum es_result vc_handle_mmio_movs(struct es_em_ctxt *ctxt,
+ unsigned int bytes)
+{
+ unsigned long ds_base, es_base;
+ unsigned char *src, *dst;
+ unsigned char buffer[8];
+ enum es_result ret;
+ bool rep;
+ int off;
+
+ ds_base = insn_get_seg_base(ctxt->regs, INAT_SEG_REG_DS);
+ es_base = insn_get_seg_base(ctxt->regs, INAT_SEG_REG_ES);
+
+ if (ds_base == -1L || es_base == -1L) {
+ ctxt->fi.vector = X86_TRAP_GP;
+ ctxt->fi.error_code = 0;
+ return ES_EXCEPTION;
+ }
+
+ src = ds_base + (unsigned char *)ctxt->regs->si;
+ dst = es_base + (unsigned char *)ctxt->regs->di;
+
+ ret = vc_read_mem(ctxt, src, buffer, bytes);
+ if (ret != ES_OK)
+ return ret;
+
+ ret = vc_write_mem(ctxt, dst, buffer, bytes);
+ if (ret != ES_OK)
+ return ret;
+
+ if (ctxt->regs->flags & X86_EFLAGS_DF)
+ off = -bytes;
+ else
+ off = bytes;
+
+ ctxt->regs->si += off;
+ ctxt->regs->di += off;
+
+ rep = insn_has_rep_prefix(&ctxt->insn);
+ if (rep)
+ ctxt->regs->cx -= 1;
+
+ if (!rep || ctxt->regs->cx == 0)
+ return ES_OK;
+ else
+ return ES_RETRY;
+}
+
static enum es_result vc_handle_mmio(struct ghcb *ghcb,
struct es_em_ctxt *ctxt)
{
@@ -655,6 +722,16 @@ static enum es_result vc_handle_mmio(struct ghcb *ghcb,
memcpy(reg_data, ghcb->shared_buffer, bytes);
break;
+ /* MOVS instruction */
+ case 0xa4:
+ bytes = 1;
+ fallthrough;
+ case 0xa5:
+ if (!bytes)
+ bytes = insn->opnd_bytes;
+
+ ret = vc_handle_mmio_movs(ctxt, bytes);
+ break;
/* Two-Byte Opcodes */
case 0x0f:
ret = vc_handle_mmio_twobyte_ops(ghcb, ctxt);
--
2.28.0
^ permalink raw reply related [flat|nested] 264+ messages in thread
* [PATCH v7 50/72] x86/sev-es: Handle MMIO String Instructions
@ 2020-09-07 13:15 ` Joerg Roedel
0 siblings, 0 replies; 264+ messages in thread
From: Joerg Roedel @ 2020-09-07 13:15 UTC (permalink / raw)
To: x86
Cc: Juergen Gross, Tom Lendacky, Joerg Roedel, Mike Stunes,
Kees Cook, kvm, Peter Zijlstra, Cfir Cohen, Joerg Roedel,
Dave Hansen, linux-kernel, Sean Christopherson, virtualization,
Martin Radev, Masami Hiramatsu, Andy Lutomirski, hpa,
Erdem Aktas, David Rientjes, Dan Williams, Jiri Slaby
From: Joerg Roedel <jroedel@suse.de>
Add handling for emulation the MOVS instruction on MMIO regions, as done
by the memcpy_toio() and memcpy_fromio() functions.
Signed-off-by: Joerg Roedel <jroedel@suse.de>
---
arch/x86/kernel/sev-es.c | 77 ++++++++++++++++++++++++++++++++++++++++
1 file changed, 77 insertions(+)
diff --git a/arch/x86/kernel/sev-es.c b/arch/x86/kernel/sev-es.c
index 62fd0ebf2a67..a79c7fc32bc6 100644
--- a/arch/x86/kernel/sev-es.c
+++ b/arch/x86/kernel/sev-es.c
@@ -594,6 +594,73 @@ static enum es_result vc_handle_mmio_twobyte_ops(struct ghcb *ghcb,
return ret;
}
+/*
+ * The MOVS instruction has two memory operands, which raises the
+ * problem that it is not known whether the access to the source or the
+ * destination caused the #VC exception (and hence whether an MMIO read
+ * or write operation needs to be emulated).
+ *
+ * Instead of playing games with walking page-tables and trying to guess
+ * whether the source or destination is an MMIO range, split the move
+ * into two operations, a read and a write with only one memory operand.
+ * This will cause a nested #VC exception on the MMIO address which can
+ * then be handled.
+ *
+ * This implementation has the benefit that it also supports MOVS where
+ * source _and_ destination are MMIO regions.
+ *
+ * It will slow MOVS on MMIO down a lot, but in SEV-ES guests it is a
+ * rare operation. If it turns out to be a performance problem the split
+ * operations can be moved to memcpy_fromio() and memcpy_toio().
+ */
+static enum es_result vc_handle_mmio_movs(struct es_em_ctxt *ctxt,
+ unsigned int bytes)
+{
+ unsigned long ds_base, es_base;
+ unsigned char *src, *dst;
+ unsigned char buffer[8];
+ enum es_result ret;
+ bool rep;
+ int off;
+
+ ds_base = insn_get_seg_base(ctxt->regs, INAT_SEG_REG_DS);
+ es_base = insn_get_seg_base(ctxt->regs, INAT_SEG_REG_ES);
+
+ if (ds_base == -1L || es_base == -1L) {
+ ctxt->fi.vector = X86_TRAP_GP;
+ ctxt->fi.error_code = 0;
+ return ES_EXCEPTION;
+ }
+
+ src = ds_base + (unsigned char *)ctxt->regs->si;
+ dst = es_base + (unsigned char *)ctxt->regs->di;
+
+ ret = vc_read_mem(ctxt, src, buffer, bytes);
+ if (ret != ES_OK)
+ return ret;
+
+ ret = vc_write_mem(ctxt, dst, buffer, bytes);
+ if (ret != ES_OK)
+ return ret;
+
+ if (ctxt->regs->flags & X86_EFLAGS_DF)
+ off = -bytes;
+ else
+ off = bytes;
+
+ ctxt->regs->si += off;
+ ctxt->regs->di += off;
+
+ rep = insn_has_rep_prefix(&ctxt->insn);
+ if (rep)
+ ctxt->regs->cx -= 1;
+
+ if (!rep || ctxt->regs->cx == 0)
+ return ES_OK;
+ else
+ return ES_RETRY;
+}
+
static enum es_result vc_handle_mmio(struct ghcb *ghcb,
struct es_em_ctxt *ctxt)
{
@@ -655,6 +722,16 @@ static enum es_result vc_handle_mmio(struct ghcb *ghcb,
memcpy(reg_data, ghcb->shared_buffer, bytes);
break;
+ /* MOVS instruction */
+ case 0xa4:
+ bytes = 1;
+ fallthrough;
+ case 0xa5:
+ if (!bytes)
+ bytes = insn->opnd_bytes;
+
+ ret = vc_handle_mmio_movs(ctxt, bytes);
+ break;
/* Two-Byte Opcodes */
case 0x0f:
ret = vc_handle_mmio_twobyte_ops(ghcb, ctxt);
--
2.28.0
_______________________________________________
Virtualization mailing list
Virtualization@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/virtualization
^ permalink raw reply related [flat|nested] 264+ messages in thread
* [PATCH v7 51/72] x86/sev-es: Handle MSR events
2020-09-07 13:15 ` Joerg Roedel
@ 2020-09-07 13:15 ` Joerg Roedel
-1 siblings, 0 replies; 264+ messages in thread
From: Joerg Roedel @ 2020-09-07 13:15 UTC (permalink / raw)
To: x86
Cc: Joerg Roedel, Joerg Roedel, Tom Lendacky, hpa, Andy Lutomirski,
Dave Hansen, Peter Zijlstra, Jiri Slaby, Dan Williams,
Juergen Gross, Kees Cook, David Rientjes, Cfir Cohen,
Erdem Aktas, Masami Hiramatsu, Mike Stunes, Sean Christopherson,
Martin Radev, linux-kernel, kvm, virtualization
From: Tom Lendacky <thomas.lendacky@amd.com>
Implement a handler for #VC exceptions caused by RDMSR/WRMSR
instructions.
Signed-off-by: Tom Lendacky <thomas.lendacky@amd.com>
[ jroedel@suse.de: Adapt to #VC handling infrastructure ]
Co-developed-by: Joerg Roedel <jroedel@suse.de>
Signed-off-by: Joerg Roedel <jroedel@suse.de>
---
arch/x86/kernel/sev-es.c | 28 ++++++++++++++++++++++++++++
1 file changed, 28 insertions(+)
diff --git a/arch/x86/kernel/sev-es.c b/arch/x86/kernel/sev-es.c
index a79c7fc32bc6..439b963ed52b 100644
--- a/arch/x86/kernel/sev-es.c
+++ b/arch/x86/kernel/sev-es.c
@@ -393,6 +393,31 @@ static bool vc_slow_virt_to_phys(struct ghcb *ghcb, struct es_em_ctxt *ctxt,
/* Include code shared with pre-decompression boot stage */
#include "sev-es-shared.c"
+static enum es_result vc_handle_msr(struct ghcb *ghcb, struct es_em_ctxt *ctxt)
+{
+ struct pt_regs *regs = ctxt->regs;
+ enum es_result ret;
+ u64 exit_info_1;
+
+ /* Is it a WRMSR? */
+ exit_info_1 = (ctxt->insn.opcode.bytes[1] == 0x30) ? 1 : 0;
+
+ ghcb_set_rcx(ghcb, regs->cx);
+ if (exit_info_1) {
+ ghcb_set_rax(ghcb, regs->ax);
+ ghcb_set_rdx(ghcb, regs->dx);
+ }
+
+ ret = sev_es_ghcb_hv_call(ghcb, ctxt, SVM_EXIT_MSR, exit_info_1, 0);
+
+ if ((ret == ES_OK) && (!exit_info_1)) {
+ regs->ax = ghcb->save.rax;
+ regs->dx = ghcb->save.rdx;
+ }
+
+ return ret;
+}
+
/*
* This function runs on the first #VC exception after the kernel
* switched to virtual addresses.
@@ -756,6 +781,9 @@ static enum es_result vc_handle_exitcode(struct es_em_ctxt *ctxt,
case SVM_EXIT_IOIO:
result = vc_handle_ioio(ghcb, ctxt);
break;
+ case SVM_EXIT_MSR:
+ result = vc_handle_msr(ghcb, ctxt);
+ break;
case SVM_EXIT_NPF:
result = vc_handle_mmio(ghcb, ctxt);
break;
--
2.28.0
^ permalink raw reply related [flat|nested] 264+ messages in thread
* [PATCH v7 51/72] x86/sev-es: Handle MSR events
@ 2020-09-07 13:15 ` Joerg Roedel
0 siblings, 0 replies; 264+ messages in thread
From: Joerg Roedel @ 2020-09-07 13:15 UTC (permalink / raw)
To: x86
Cc: Juergen Gross, Tom Lendacky, Joerg Roedel, Mike Stunes,
Kees Cook, kvm, Peter Zijlstra, Cfir Cohen, Joerg Roedel,
Dave Hansen, linux-kernel, Sean Christopherson, virtualization,
Martin Radev, Masami Hiramatsu, Andy Lutomirski, hpa,
Erdem Aktas, David Rientjes, Dan Williams, Jiri Slaby
From: Tom Lendacky <thomas.lendacky@amd.com>
Implement a handler for #VC exceptions caused by RDMSR/WRMSR
instructions.
Signed-off-by: Tom Lendacky <thomas.lendacky@amd.com>
[ jroedel@suse.de: Adapt to #VC handling infrastructure ]
Co-developed-by: Joerg Roedel <jroedel@suse.de>
Signed-off-by: Joerg Roedel <jroedel@suse.de>
---
arch/x86/kernel/sev-es.c | 28 ++++++++++++++++++++++++++++
1 file changed, 28 insertions(+)
diff --git a/arch/x86/kernel/sev-es.c b/arch/x86/kernel/sev-es.c
index a79c7fc32bc6..439b963ed52b 100644
--- a/arch/x86/kernel/sev-es.c
+++ b/arch/x86/kernel/sev-es.c
@@ -393,6 +393,31 @@ static bool vc_slow_virt_to_phys(struct ghcb *ghcb, struct es_em_ctxt *ctxt,
/* Include code shared with pre-decompression boot stage */
#include "sev-es-shared.c"
+static enum es_result vc_handle_msr(struct ghcb *ghcb, struct es_em_ctxt *ctxt)
+{
+ struct pt_regs *regs = ctxt->regs;
+ enum es_result ret;
+ u64 exit_info_1;
+
+ /* Is it a WRMSR? */
+ exit_info_1 = (ctxt->insn.opcode.bytes[1] == 0x30) ? 1 : 0;
+
+ ghcb_set_rcx(ghcb, regs->cx);
+ if (exit_info_1) {
+ ghcb_set_rax(ghcb, regs->ax);
+ ghcb_set_rdx(ghcb, regs->dx);
+ }
+
+ ret = sev_es_ghcb_hv_call(ghcb, ctxt, SVM_EXIT_MSR, exit_info_1, 0);
+
+ if ((ret == ES_OK) && (!exit_info_1)) {
+ regs->ax = ghcb->save.rax;
+ regs->dx = ghcb->save.rdx;
+ }
+
+ return ret;
+}
+
/*
* This function runs on the first #VC exception after the kernel
* switched to virtual addresses.
@@ -756,6 +781,9 @@ static enum es_result vc_handle_exitcode(struct es_em_ctxt *ctxt,
case SVM_EXIT_IOIO:
result = vc_handle_ioio(ghcb, ctxt);
break;
+ case SVM_EXIT_MSR:
+ result = vc_handle_msr(ghcb, ctxt);
+ break;
case SVM_EXIT_NPF:
result = vc_handle_mmio(ghcb, ctxt);
break;
--
2.28.0
_______________________________________________
Virtualization mailing list
Virtualization@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/virtualization
^ permalink raw reply related [flat|nested] 264+ messages in thread
* [PATCH v7 52/72] x86/sev-es: Handle DR7 read/write events
2020-09-07 13:15 ` Joerg Roedel
@ 2020-09-07 13:15 ` Joerg Roedel
-1 siblings, 0 replies; 264+ messages in thread
From: Joerg Roedel @ 2020-09-07 13:15 UTC (permalink / raw)
To: x86
Cc: Joerg Roedel, Joerg Roedel, Tom Lendacky, hpa, Andy Lutomirski,
Dave Hansen, Peter Zijlstra, Jiri Slaby, Dan Williams,
Juergen Gross, Kees Cook, David Rientjes, Cfir Cohen,
Erdem Aktas, Masami Hiramatsu, Mike Stunes, Sean Christopherson,
Martin Radev, linux-kernel, kvm, virtualization
From: Tom Lendacky <thomas.lendacky@amd.com>
Add code to handle #VC exceptions on DR7 register reads and writes.
This is needed early because show_regs() reads DR7 to print it out.
Under SEV-ES there is currently no support for saving/restoring the
DRx registers, but software expects to be able to write to the DR7
register. For now, cache the value written to DR7 and return it on
read attempts, but do not touch the real hardware DR7.
Signed-off-by: Tom Lendacky <thomas.lendacky@amd.com>
[ jroedel@suse.de: - Adapt to #VC handling framework
- Support early usage ]
Co-developed-by: Joerg Roedel <jroedel@suse.de>
Signed-off-by: Joerg Roedel <jroedel@suse.de>
---
arch/x86/kernel/sev-es.c | 85 ++++++++++++++++++++++++++++++++++++++++
1 file changed, 85 insertions(+)
diff --git a/arch/x86/kernel/sev-es.c b/arch/x86/kernel/sev-es.c
index 439b963ed52b..7879fe640dcc 100644
--- a/arch/x86/kernel/sev-es.c
+++ b/arch/x86/kernel/sev-es.c
@@ -29,6 +29,8 @@
#include <asm/traps.h>
#include <asm/svm.h>
+#define DR7_RESET_VALUE 0x400
+
/* For early boot hypervisor communication in SEV-ES enabled guests */
static struct ghcb boot_ghcb_page __bss_decrypted __aligned(PAGE_SIZE);
@@ -76,6 +78,13 @@ struct sev_es_runtime_data {
*/
bool ghcb_active;
bool backup_ghcb_active;
+
+ /*
+ * Cached DR7 value - write it on DR7 writes and return it on reads.
+ * That value will never make it to the real hardware DR7 as debugging
+ * is currently unsupported in SEV-ES guests.
+ */
+ unsigned long dr7;
};
struct ghcb_state {
@@ -519,6 +528,21 @@ static long *vc_insn_get_reg(struct es_em_ctxt *ctxt)
return reg_array + offset;
}
+static long *vc_insn_get_rm(struct es_em_ctxt *ctxt)
+{
+ long *reg_array;
+ int offset;
+
+ reg_array = (long *)ctxt->regs;
+ offset = insn_get_modrm_rm_off(&ctxt->insn, ctxt->regs);
+
+ if (offset < 0)
+ return NULL;
+
+ offset /= sizeof(long);
+
+ return reg_array + offset;
+}
static enum es_result vc_do_mmio(struct ghcb *ghcb, struct es_em_ctxt *ctxt,
unsigned int bytes, bool read)
{
@@ -768,6 +792,61 @@ static enum es_result vc_handle_mmio(struct ghcb *ghcb,
return ret;
}
+static enum es_result vc_handle_dr7_write(struct ghcb *ghcb,
+ struct es_em_ctxt *ctxt)
+{
+ struct sev_es_runtime_data *data = this_cpu_read(runtime_data);
+ long val, *reg = vc_insn_get_rm(ctxt);
+ enum es_result ret;
+
+ if (!reg)
+ return ES_DECODE_FAILED;
+
+ val = *reg;
+
+ /* Upper 32 bits must be written as zeroes */
+ if (val >> 32) {
+ ctxt->fi.vector = X86_TRAP_GP;
+ ctxt->fi.error_code = 0;
+ return ES_EXCEPTION;
+ }
+
+ /* Clear out other reserved bits and set bit 10 */
+ val = (val & 0xffff23ffL) | BIT(10);
+
+ /* Early non-zero writes to DR7 are not supported */
+ if (!data && (val & ~DR7_RESET_VALUE))
+ return ES_UNSUPPORTED;
+
+ /* Using a value of 0 for ExitInfo1 means RAX holds the value */
+ ghcb_set_rax(ghcb, val);
+ ret = sev_es_ghcb_hv_call(ghcb, ctxt, SVM_EXIT_WRITE_DR7, 0, 0);
+ if (ret != ES_OK)
+ return ret;
+
+ if (data)
+ data->dr7 = val;
+
+ return ES_OK;
+}
+
+static enum es_result vc_handle_dr7_read(struct ghcb *ghcb,
+ struct es_em_ctxt *ctxt)
+{
+ struct sev_es_runtime_data *data = this_cpu_read(runtime_data);
+ long *reg = vc_insn_get_rm(ctxt);
+
+ if (!reg)
+ return ES_DECODE_FAILED;
+
+ if (data)
+ *reg = data->dr7;
+ else
+ *reg = DR7_RESET_VALUE;
+
+ return ES_OK;
+}
+
static enum es_result vc_handle_exitcode(struct es_em_ctxt *ctxt,
struct ghcb *ghcb,
unsigned long exit_code)
@@ -775,6 +854,12 @@ static enum es_result vc_handle_exitcode(struct es_em_ctxt *ctxt,
enum es_result result;
switch (exit_code) {
+ case SVM_EXIT_READ_DR7:
+ result = vc_handle_dr7_read(ghcb, ctxt);
+ break;
+ case SVM_EXIT_WRITE_DR7:
+ result = vc_handle_dr7_write(ghcb, ctxt);
+ break;
case SVM_EXIT_CPUID:
result = vc_handle_cpuid(ghcb, ctxt);
break;
--
2.28.0
^ permalink raw reply related [flat|nested] 264+ messages in thread
* [PATCH v7 52/72] x86/sev-es: Handle DR7 read/write events
@ 2020-09-07 13:15 ` Joerg Roedel
0 siblings, 0 replies; 264+ messages in thread
From: Joerg Roedel @ 2020-09-07 13:15 UTC (permalink / raw)
To: x86
Cc: Juergen Gross, Tom Lendacky, Joerg Roedel, Mike Stunes,
Kees Cook, kvm, Peter Zijlstra, Cfir Cohen, Joerg Roedel,
Dave Hansen, linux-kernel, Sean Christopherson, virtualization,
Martin Radev, Masami Hiramatsu, Andy Lutomirski, hpa,
Erdem Aktas, David Rientjes, Dan Williams, Jiri Slaby
From: Tom Lendacky <thomas.lendacky@amd.com>
Add code to handle #VC exceptions on DR7 register reads and writes.
This is needed early because show_regs() reads DR7 to print it out.
Under SEV-ES there is currently no support for saving/restoring the
DRx registers, but software expects to be able to write to the DR7
register. For now, cache the value written to DR7 and return it on
read attempts, but do not touch the real hardware DR7.
Signed-off-by: Tom Lendacky <thomas.lendacky@amd.com>
[ jroedel@suse.de: - Adapt to #VC handling framework
- Support early usage ]
Co-developed-by: Joerg Roedel <jroedel@suse.de>
Signed-off-by: Joerg Roedel <jroedel@suse.de>
---
arch/x86/kernel/sev-es.c | 85 ++++++++++++++++++++++++++++++++++++++++
1 file changed, 85 insertions(+)
diff --git a/arch/x86/kernel/sev-es.c b/arch/x86/kernel/sev-es.c
index 439b963ed52b..7879fe640dcc 100644
--- a/arch/x86/kernel/sev-es.c
+++ b/arch/x86/kernel/sev-es.c
@@ -29,6 +29,8 @@
#include <asm/traps.h>
#include <asm/svm.h>
+#define DR7_RESET_VALUE 0x400
+
/* For early boot hypervisor communication in SEV-ES enabled guests */
static struct ghcb boot_ghcb_page __bss_decrypted __aligned(PAGE_SIZE);
@@ -76,6 +78,13 @@ struct sev_es_runtime_data {
*/
bool ghcb_active;
bool backup_ghcb_active;
+
+ /*
+ * Cached DR7 value - write it on DR7 writes and return it on reads.
+ * That value will never make it to the real hardware DR7 as debugging
+ * is currently unsupported in SEV-ES guests.
+ */
+ unsigned long dr7;
};
struct ghcb_state {
@@ -519,6 +528,21 @@ static long *vc_insn_get_reg(struct es_em_ctxt *ctxt)
return reg_array + offset;
}
+static long *vc_insn_get_rm(struct es_em_ctxt *ctxt)
+{
+ long *reg_array;
+ int offset;
+
+ reg_array = (long *)ctxt->regs;
+ offset = insn_get_modrm_rm_off(&ctxt->insn, ctxt->regs);
+
+ if (offset < 0)
+ return NULL;
+
+ offset /= sizeof(long);
+
+ return reg_array + offset;
+}
static enum es_result vc_do_mmio(struct ghcb *ghcb, struct es_em_ctxt *ctxt,
unsigned int bytes, bool read)
{
@@ -768,6 +792,61 @@ static enum es_result vc_handle_mmio(struct ghcb *ghcb,
return ret;
}
+static enum es_result vc_handle_dr7_write(struct ghcb *ghcb,
+ struct es_em_ctxt *ctxt)
+{
+ struct sev_es_runtime_data *data = this_cpu_read(runtime_data);
+ long val, *reg = vc_insn_get_rm(ctxt);
+ enum es_result ret;
+
+ if (!reg)
+ return ES_DECODE_FAILED;
+
+ val = *reg;
+
+ /* Upper 32 bits must be written as zeroes */
+ if (val >> 32) {
+ ctxt->fi.vector = X86_TRAP_GP;
+ ctxt->fi.error_code = 0;
+ return ES_EXCEPTION;
+ }
+
+ /* Clear out other reserved bits and set bit 10 */
+ val = (val & 0xffff23ffL) | BIT(10);
+
+ /* Early non-zero writes to DR7 are not supported */
+ if (!data && (val & ~DR7_RESET_VALUE))
+ return ES_UNSUPPORTED;
+
+ /* Using a value of 0 for ExitInfo1 means RAX holds the value */
+ ghcb_set_rax(ghcb, val);
+ ret = sev_es_ghcb_hv_call(ghcb, ctxt, SVM_EXIT_WRITE_DR7, 0, 0);
+ if (ret != ES_OK)
+ return ret;
+
+ if (data)
+ data->dr7 = val;
+
+ return ES_OK;
+}
+
+static enum es_result vc_handle_dr7_read(struct ghcb *ghcb,
+ struct es_em_ctxt *ctxt)
+{
+ struct sev_es_runtime_data *data = this_cpu_read(runtime_data);
+ long *reg = vc_insn_get_rm(ctxt);
+
+ if (!reg)
+ return ES_DECODE_FAILED;
+
+ if (data)
+ *reg = data->dr7;
+ else
+ *reg = DR7_RESET_VALUE;
+
+ return ES_OK;
+}
+
static enum es_result vc_handle_exitcode(struct es_em_ctxt *ctxt,
struct ghcb *ghcb,
unsigned long exit_code)
@@ -775,6 +854,12 @@ static enum es_result vc_handle_exitcode(struct es_em_ctxt *ctxt,
enum es_result result;
switch (exit_code) {
+ case SVM_EXIT_READ_DR7:
+ result = vc_handle_dr7_read(ghcb, ctxt);
+ break;
+ case SVM_EXIT_WRITE_DR7:
+ result = vc_handle_dr7_write(ghcb, ctxt);
+ break;
case SVM_EXIT_CPUID:
result = vc_handle_cpuid(ghcb, ctxt);
break;
--
2.28.0
_______________________________________________
Virtualization mailing list
Virtualization@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/virtualization
^ permalink raw reply related [flat|nested] 264+ messages in thread
* [PATCH v7 53/72] x86/sev-es: Handle WBINVD Events
2020-09-07 13:15 ` Joerg Roedel
@ 2020-09-07 13:15 ` Joerg Roedel
-1 siblings, 0 replies; 264+ messages in thread
From: Joerg Roedel @ 2020-09-07 13:15 UTC (permalink / raw)
To: x86
Cc: Joerg Roedel, Joerg Roedel, Tom Lendacky, hpa, Andy Lutomirski,
Dave Hansen, Peter Zijlstra, Jiri Slaby, Dan Williams,
Juergen Gross, Kees Cook, David Rientjes, Cfir Cohen,
Erdem Aktas, Masami Hiramatsu, Mike Stunes, Sean Christopherson,
Martin Radev, linux-kernel, kvm, virtualization
From: Tom Lendacky <thomas.lendacky@amd.com>
Implement a handler for #VC exceptions caused by WBINVD instructions.
Signed-off-by: Tom Lendacky <thomas.lendacky@amd.com>
[ jroedel@suse.de: Adapt to #VC handling framework ]
Co-developed-by: Joerg Roedel <jroedel@suse.de>
Signed-off-by: Joerg Roedel <jroedel@suse.de>
---
arch/x86/kernel/sev-es.c | 9 +++++++++
1 file changed, 9 insertions(+)
diff --git a/arch/x86/kernel/sev-es.c b/arch/x86/kernel/sev-es.c
index 7879fe640dcc..c51cb2673af6 100644
--- a/arch/x86/kernel/sev-es.c
+++ b/arch/x86/kernel/sev-es.c
@@ -847,6 +847,12 @@ static enum es_result vc_handle_dr7_read(struct ghcb *ghcb,
return ES_OK;
}
+static enum es_result vc_handle_wbinvd(struct ghcb *ghcb,
+ struct es_em_ctxt *ctxt)
+{
+ return sev_es_ghcb_hv_call(ghcb, ctxt, SVM_EXIT_WBINVD, 0, 0);
+}
+
static enum es_result vc_handle_exitcode(struct es_em_ctxt *ctxt,
struct ghcb *ghcb,
unsigned long exit_code)
@@ -869,6 +875,9 @@ static enum es_result vc_handle_exitcode(struct es_em_ctxt *ctxt,
case SVM_EXIT_MSR:
result = vc_handle_msr(ghcb, ctxt);
break;
+ case SVM_EXIT_WBINVD:
+ result = vc_handle_wbinvd(ghcb, ctxt);
+ break;
case SVM_EXIT_NPF:
result = vc_handle_mmio(ghcb, ctxt);
break;
--
2.28.0
^ permalink raw reply related [flat|nested] 264+ messages in thread
* [PATCH v7 53/72] x86/sev-es: Handle WBINVD Events
@ 2020-09-07 13:15 ` Joerg Roedel
0 siblings, 0 replies; 264+ messages in thread
From: Joerg Roedel @ 2020-09-07 13:15 UTC (permalink / raw)
To: x86
Cc: Juergen Gross, Tom Lendacky, Joerg Roedel, Mike Stunes,
Kees Cook, kvm, Peter Zijlstra, Cfir Cohen, Joerg Roedel,
Dave Hansen, linux-kernel, Sean Christopherson, virtualization,
Martin Radev, Masami Hiramatsu, Andy Lutomirski, hpa,
Erdem Aktas, David Rientjes, Dan Williams, Jiri Slaby
From: Tom Lendacky <thomas.lendacky@amd.com>
Implement a handler for #VC exceptions caused by WBINVD instructions.
Signed-off-by: Tom Lendacky <thomas.lendacky@amd.com>
[ jroedel@suse.de: Adapt to #VC handling framework ]
Co-developed-by: Joerg Roedel <jroedel@suse.de>
Signed-off-by: Joerg Roedel <jroedel@suse.de>
---
arch/x86/kernel/sev-es.c | 9 +++++++++
1 file changed, 9 insertions(+)
diff --git a/arch/x86/kernel/sev-es.c b/arch/x86/kernel/sev-es.c
index 7879fe640dcc..c51cb2673af6 100644
--- a/arch/x86/kernel/sev-es.c
+++ b/arch/x86/kernel/sev-es.c
@@ -847,6 +847,12 @@ static enum es_result vc_handle_dr7_read(struct ghcb *ghcb,
return ES_OK;
}
+static enum es_result vc_handle_wbinvd(struct ghcb *ghcb,
+ struct es_em_ctxt *ctxt)
+{
+ return sev_es_ghcb_hv_call(ghcb, ctxt, SVM_EXIT_WBINVD, 0, 0);
+}
+
static enum es_result vc_handle_exitcode(struct es_em_ctxt *ctxt,
struct ghcb *ghcb,
unsigned long exit_code)
@@ -869,6 +875,9 @@ static enum es_result vc_handle_exitcode(struct es_em_ctxt *ctxt,
case SVM_EXIT_MSR:
result = vc_handle_msr(ghcb, ctxt);
break;
+ case SVM_EXIT_WBINVD:
+ result = vc_handle_wbinvd(ghcb, ctxt);
+ break;
case SVM_EXIT_NPF:
result = vc_handle_mmio(ghcb, ctxt);
break;
--
2.28.0
_______________________________________________
Virtualization mailing list
Virtualization@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/virtualization
^ permalink raw reply related [flat|nested] 264+ messages in thread
* [PATCH v7 54/72] x86/sev-es: Handle RDTSC(P) Events
2020-09-07 13:15 ` Joerg Roedel
@ 2020-09-07 13:15 ` Joerg Roedel
-1 siblings, 0 replies; 264+ messages in thread
From: Joerg Roedel @ 2020-09-07 13:15 UTC (permalink / raw)
To: x86
Cc: Joerg Roedel, Joerg Roedel, Tom Lendacky, hpa, Andy Lutomirski,
Dave Hansen, Peter Zijlstra, Jiri Slaby, Dan Williams,
Juergen Gross, Kees Cook, David Rientjes, Cfir Cohen,
Erdem Aktas, Masami Hiramatsu, Mike Stunes, Sean Christopherson,
Martin Radev, linux-kernel, kvm, virtualization
From: Tom Lendacky <thomas.lendacky@amd.com>
Implement a handler for #VC exceptions caused by RDTSC and RDTSCP
instructions. Also make it available in the pre-decompression stage
because the KASLR code used RDTSC/RDTSCP to gather entropy and some
hypervisors intercept these instructions.
Signed-off-by: Tom Lendacky <thomas.lendacky@amd.com>
[ jroedel@suse.de: - Adapt to #VC handling infrastructure
- Make it available early ]
Co-developed-by: Joerg Roedel <jroedel@suse.de>
Signed-off-by: Joerg Roedel <jroedel@suse.de>
---
arch/x86/boot/compressed/sev-es.c | 4 ++++
arch/x86/kernel/sev-es-shared.c | 23 +++++++++++++++++++++++
arch/x86/kernel/sev-es.c | 4 ++++
3 files changed, 31 insertions(+)
diff --git a/arch/x86/boot/compressed/sev-es.c b/arch/x86/boot/compressed/sev-es.c
index 57c41c368681..45702b866c33 100644
--- a/arch/x86/boot/compressed/sev-es.c
+++ b/arch/x86/boot/compressed/sev-es.c
@@ -181,6 +181,10 @@ void do_boot_stage2_vc(struct pt_regs *regs, unsigned long exit_code)
goto finish;
switch (exit_code) {
+ case SVM_EXIT_RDTSC:
+ case SVM_EXIT_RDTSCP:
+ result = vc_handle_rdtsc(boot_ghcb, &ctxt, exit_code);
+ break;
case SVM_EXIT_IOIO:
result = vc_handle_ioio(boot_ghcb, &ctxt);
break;
diff --git a/arch/x86/kernel/sev-es-shared.c b/arch/x86/kernel/sev-es-shared.c
index a6b41910b8ab..92d77b725ccb 100644
--- a/arch/x86/kernel/sev-es-shared.c
+++ b/arch/x86/kernel/sev-es-shared.c
@@ -467,3 +467,26 @@ static enum es_result vc_handle_cpuid(struct ghcb *ghcb,
return ES_OK;
}
+
+static enum es_result vc_handle_rdtsc(struct ghcb *ghcb,
+ struct es_em_ctxt *ctxt,
+ unsigned long exit_code)
+{
+ bool rdtscp = (exit_code == SVM_EXIT_RDTSCP);
+ enum es_result ret;
+
+ ret = sev_es_ghcb_hv_call(ghcb, ctxt, exit_code, 0, 0);
+ if (ret != ES_OK)
+ return ret;
+
+ if (!(ghcb_rax_is_valid(ghcb) && ghcb_rdx_is_valid(ghcb) &&
+ (!rdtscp || ghcb_rcx_is_valid(ghcb))))
+ return ES_VMM_ERROR;
+
+ ctxt->regs->ax = ghcb->save.rax;
+ ctxt->regs->dx = ghcb->save.rdx;
+ if (rdtscp)
+ ctxt->regs->cx = ghcb->save.rcx;
+
+ return ES_OK;
+}
diff --git a/arch/x86/kernel/sev-es.c b/arch/x86/kernel/sev-es.c
index c51cb2673af6..5c155f307129 100644
--- a/arch/x86/kernel/sev-es.c
+++ b/arch/x86/kernel/sev-es.c
@@ -866,6 +866,10 @@ static enum es_result vc_handle_exitcode(struct es_em_ctxt *ctxt,
case SVM_EXIT_WRITE_DR7:
result = vc_handle_dr7_write(ghcb, ctxt);
break;
+ case SVM_EXIT_RDTSC:
+ case SVM_EXIT_RDTSCP:
+ result = vc_handle_rdtsc(ghcb, ctxt, exit_code);
+ break;
case SVM_EXIT_CPUID:
result = vc_handle_cpuid(ghcb, ctxt);
break;
--
2.28.0
^ permalink raw reply related [flat|nested] 264+ messages in thread
* [PATCH v7 54/72] x86/sev-es: Handle RDTSC(P) Events
@ 2020-09-07 13:15 ` Joerg Roedel
0 siblings, 0 replies; 264+ messages in thread
From: Joerg Roedel @ 2020-09-07 13:15 UTC (permalink / raw)
To: x86
Cc: Juergen Gross, Tom Lendacky, Joerg Roedel, Mike Stunes,
Kees Cook, kvm, Peter Zijlstra, Cfir Cohen, Joerg Roedel,
Dave Hansen, linux-kernel, Sean Christopherson, virtualization,
Martin Radev, Masami Hiramatsu, Andy Lutomirski, hpa,
Erdem Aktas, David Rientjes, Dan Williams, Jiri Slaby
From: Tom Lendacky <thomas.lendacky@amd.com>
Implement a handler for #VC exceptions caused by RDTSC and RDTSCP
instructions. Also make it available in the pre-decompression stage
because the KASLR code used RDTSC/RDTSCP to gather entropy and some
hypervisors intercept these instructions.
Signed-off-by: Tom Lendacky <thomas.lendacky@amd.com>
[ jroedel@suse.de: - Adapt to #VC handling infrastructure
- Make it available early ]
Co-developed-by: Joerg Roedel <jroedel@suse.de>
Signed-off-by: Joerg Roedel <jroedel@suse.de>
---
arch/x86/boot/compressed/sev-es.c | 4 ++++
arch/x86/kernel/sev-es-shared.c | 23 +++++++++++++++++++++++
arch/x86/kernel/sev-es.c | 4 ++++
3 files changed, 31 insertions(+)
diff --git a/arch/x86/boot/compressed/sev-es.c b/arch/x86/boot/compressed/sev-es.c
index 57c41c368681..45702b866c33 100644
--- a/arch/x86/boot/compressed/sev-es.c
+++ b/arch/x86/boot/compressed/sev-es.c
@@ -181,6 +181,10 @@ void do_boot_stage2_vc(struct pt_regs *regs, unsigned long exit_code)
goto finish;
switch (exit_code) {
+ case SVM_EXIT_RDTSC:
+ case SVM_EXIT_RDTSCP:
+ result = vc_handle_rdtsc(boot_ghcb, &ctxt, exit_code);
+ break;
case SVM_EXIT_IOIO:
result = vc_handle_ioio(boot_ghcb, &ctxt);
break;
diff --git a/arch/x86/kernel/sev-es-shared.c b/arch/x86/kernel/sev-es-shared.c
index a6b41910b8ab..92d77b725ccb 100644
--- a/arch/x86/kernel/sev-es-shared.c
+++ b/arch/x86/kernel/sev-es-shared.c
@@ -467,3 +467,26 @@ static enum es_result vc_handle_cpuid(struct ghcb *ghcb,
return ES_OK;
}
+
+static enum es_result vc_handle_rdtsc(struct ghcb *ghcb,
+ struct es_em_ctxt *ctxt,
+ unsigned long exit_code)
+{
+ bool rdtscp = (exit_code == SVM_EXIT_RDTSCP);
+ enum es_result ret;
+
+ ret = sev_es_ghcb_hv_call(ghcb, ctxt, exit_code, 0, 0);
+ if (ret != ES_OK)
+ return ret;
+
+ if (!(ghcb_rax_is_valid(ghcb) && ghcb_rdx_is_valid(ghcb) &&
+ (!rdtscp || ghcb_rcx_is_valid(ghcb))))
+ return ES_VMM_ERROR;
+
+ ctxt->regs->ax = ghcb->save.rax;
+ ctxt->regs->dx = ghcb->save.rdx;
+ if (rdtscp)
+ ctxt->regs->cx = ghcb->save.rcx;
+
+ return ES_OK;
+}
diff --git a/arch/x86/kernel/sev-es.c b/arch/x86/kernel/sev-es.c
index c51cb2673af6..5c155f307129 100644
--- a/arch/x86/kernel/sev-es.c
+++ b/arch/x86/kernel/sev-es.c
@@ -866,6 +866,10 @@ static enum es_result vc_handle_exitcode(struct es_em_ctxt *ctxt,
case SVM_EXIT_WRITE_DR7:
result = vc_handle_dr7_write(ghcb, ctxt);
break;
+ case SVM_EXIT_RDTSC:
+ case SVM_EXIT_RDTSCP:
+ result = vc_handle_rdtsc(ghcb, ctxt, exit_code);
+ break;
case SVM_EXIT_CPUID:
result = vc_handle_cpuid(ghcb, ctxt);
break;
--
2.28.0
_______________________________________________
Virtualization mailing list
Virtualization@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/virtualization
^ permalink raw reply related [flat|nested] 264+ messages in thread
* [PATCH v7 55/72] x86/sev-es: Handle RDPMC Events
2020-09-07 13:15 ` Joerg Roedel
@ 2020-09-07 13:15 ` Joerg Roedel
-1 siblings, 0 replies; 264+ messages in thread
From: Joerg Roedel @ 2020-09-07 13:15 UTC (permalink / raw)
To: x86
Cc: Joerg Roedel, Joerg Roedel, Tom Lendacky, hpa, Andy Lutomirski,
Dave Hansen, Peter Zijlstra, Jiri Slaby, Dan Williams,
Juergen Gross, Kees Cook, David Rientjes, Cfir Cohen,
Erdem Aktas, Masami Hiramatsu, Mike Stunes, Sean Christopherson,
Martin Radev, linux-kernel, kvm, virtualization
From: Tom Lendacky <thomas.lendacky@amd.com>
Implement a handler for #VC exceptions caused by RDPMC instructions.
Signed-off-by: Tom Lendacky <thomas.lendacky@amd.com>
[ jroedel@suse.de: Adapt to #VC handling infrastructure ]
Co-developed-by: Joerg Roedel <jroedel@suse.de>
Signed-off-by: Joerg Roedel <jroedel@suse.de>
---
arch/x86/kernel/sev-es.c | 22 ++++++++++++++++++++++
1 file changed, 22 insertions(+)
diff --git a/arch/x86/kernel/sev-es.c b/arch/x86/kernel/sev-es.c
index 5c155f307129..2ee600f3184b 100644
--- a/arch/x86/kernel/sev-es.c
+++ b/arch/x86/kernel/sev-es.c
@@ -853,6 +853,25 @@ static enum es_result vc_handle_wbinvd(struct ghcb *ghcb,
return sev_es_ghcb_hv_call(ghcb, ctxt, SVM_EXIT_WBINVD, 0, 0);
}
+static enum es_result vc_handle_rdpmc(struct ghcb *ghcb, struct es_em_ctxt *ctxt)
+{
+ enum es_result ret;
+
+ ghcb_set_rcx(ghcb, ctxt->regs->cx);
+
+ ret = sev_es_ghcb_hv_call(ghcb, ctxt, SVM_EXIT_RDPMC, 0, 0);
+ if (ret != ES_OK)
+ return ret;
+
+ if (!(ghcb_rax_is_valid(ghcb) && ghcb_rdx_is_valid(ghcb)))
+ return ES_VMM_ERROR;
+
+ ctxt->regs->ax = ghcb->save.rax;
+ ctxt->regs->dx = ghcb->save.rdx;
+
+ return ES_OK;
+}
+
static enum es_result vc_handle_exitcode(struct es_em_ctxt *ctxt,
struct ghcb *ghcb,
unsigned long exit_code)
@@ -870,6 +889,9 @@ static enum es_result vc_handle_exitcode(struct es_em_ctxt *ctxt,
case SVM_EXIT_RDTSCP:
result = vc_handle_rdtsc(ghcb, ctxt, exit_code);
break;
+ case SVM_EXIT_RDPMC:
+ result = vc_handle_rdpmc(ghcb, ctxt);
+ break;
case SVM_EXIT_CPUID:
result = vc_handle_cpuid(ghcb, ctxt);
break;
--
2.28.0
^ permalink raw reply related [flat|nested] 264+ messages in thread
* [PATCH v7 55/72] x86/sev-es: Handle RDPMC Events
@ 2020-09-07 13:15 ` Joerg Roedel
0 siblings, 0 replies; 264+ messages in thread
From: Joerg Roedel @ 2020-09-07 13:15 UTC (permalink / raw)
To: x86
Cc: Juergen Gross, Tom Lendacky, Joerg Roedel, Mike Stunes,
Kees Cook, kvm, Peter Zijlstra, Cfir Cohen, Joerg Roedel,
Dave Hansen, linux-kernel, Sean Christopherson, virtualization,
Martin Radev, Masami Hiramatsu, Andy Lutomirski, hpa,
Erdem Aktas, David Rientjes, Dan Williams, Jiri Slaby
From: Tom Lendacky <thomas.lendacky@amd.com>
Implement a handler for #VC exceptions caused by RDPMC instructions.
Signed-off-by: Tom Lendacky <thomas.lendacky@amd.com>
[ jroedel@suse.de: Adapt to #VC handling infrastructure ]
Co-developed-by: Joerg Roedel <jroedel@suse.de>
Signed-off-by: Joerg Roedel <jroedel@suse.de>
---
arch/x86/kernel/sev-es.c | 22 ++++++++++++++++++++++
1 file changed, 22 insertions(+)
diff --git a/arch/x86/kernel/sev-es.c b/arch/x86/kernel/sev-es.c
index 5c155f307129..2ee600f3184b 100644
--- a/arch/x86/kernel/sev-es.c
+++ b/arch/x86/kernel/sev-es.c
@@ -853,6 +853,25 @@ static enum es_result vc_handle_wbinvd(struct ghcb *ghcb,
return sev_es_ghcb_hv_call(ghcb, ctxt, SVM_EXIT_WBINVD, 0, 0);
}
+static enum es_result vc_handle_rdpmc(struct ghcb *ghcb, struct es_em_ctxt *ctxt)
+{
+ enum es_result ret;
+
+ ghcb_set_rcx(ghcb, ctxt->regs->cx);
+
+ ret = sev_es_ghcb_hv_call(ghcb, ctxt, SVM_EXIT_RDPMC, 0, 0);
+ if (ret != ES_OK)
+ return ret;
+
+ if (!(ghcb_rax_is_valid(ghcb) && ghcb_rdx_is_valid(ghcb)))
+ return ES_VMM_ERROR;
+
+ ctxt->regs->ax = ghcb->save.rax;
+ ctxt->regs->dx = ghcb->save.rdx;
+
+ return ES_OK;
+}
+
static enum es_result vc_handle_exitcode(struct es_em_ctxt *ctxt,
struct ghcb *ghcb,
unsigned long exit_code)
@@ -870,6 +889,9 @@ static enum es_result vc_handle_exitcode(struct es_em_ctxt *ctxt,
case SVM_EXIT_RDTSCP:
result = vc_handle_rdtsc(ghcb, ctxt, exit_code);
break;
+ case SVM_EXIT_RDPMC:
+ result = vc_handle_rdpmc(ghcb, ctxt);
+ break;
case SVM_EXIT_CPUID:
result = vc_handle_cpuid(ghcb, ctxt);
break;
--
2.28.0
_______________________________________________
Virtualization mailing list
Virtualization@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/virtualization
^ permalink raw reply related [flat|nested] 264+ messages in thread
* [PATCH v7 56/72] x86/sev-es: Handle INVD Events
2020-09-07 13:15 ` Joerg Roedel
@ 2020-09-07 13:15 ` Joerg Roedel
-1 siblings, 0 replies; 264+ messages in thread
From: Joerg Roedel @ 2020-09-07 13:15 UTC (permalink / raw)
To: x86
Cc: Joerg Roedel, Joerg Roedel, Tom Lendacky, hpa, Andy Lutomirski,
Dave Hansen, Peter Zijlstra, Jiri Slaby, Dan Williams,
Juergen Gross, Kees Cook, David Rientjes, Cfir Cohen,
Erdem Aktas, Masami Hiramatsu, Mike Stunes, Sean Christopherson,
Martin Radev, linux-kernel, kvm, virtualization
From: Tom Lendacky <thomas.lendacky@amd.com>
Implement a handler for #VC exceptions caused by INVD instructions.
Since Linux should never use INVD, just mark it as unsupported.
Signed-off-by: Tom Lendacky <thomas.lendacky@amd.com>
[ jroedel@suse.de: Adapt to #VC handling infrastructure ]
Co-developed-by: Joerg Roedel <jroedel@suse.de>
Signed-off-by: Joerg Roedel <jroedel@suse.de>
---
arch/x86/kernel/sev-es.c | 4 ++++
1 file changed, 4 insertions(+)
diff --git a/arch/x86/kernel/sev-es.c b/arch/x86/kernel/sev-es.c
index 2ee600f3184b..7fb17e71ff05 100644
--- a/arch/x86/kernel/sev-es.c
+++ b/arch/x86/kernel/sev-es.c
@@ -892,6 +892,10 @@ static enum es_result vc_handle_exitcode(struct es_em_ctxt *ctxt,
case SVM_EXIT_RDPMC:
result = vc_handle_rdpmc(ghcb, ctxt);
break;
+ case SVM_EXIT_INVD:
+ pr_err_ratelimited("#VC exception for INVD??? Seriously???\n");
+ result = ES_UNSUPPORTED;
+ break;
case SVM_EXIT_CPUID:
result = vc_handle_cpuid(ghcb, ctxt);
break;
--
2.28.0
^ permalink raw reply related [flat|nested] 264+ messages in thread
* [PATCH v7 56/72] x86/sev-es: Handle INVD Events
@ 2020-09-07 13:15 ` Joerg Roedel
0 siblings, 0 replies; 264+ messages in thread
From: Joerg Roedel @ 2020-09-07 13:15 UTC (permalink / raw)
To: x86
Cc: Juergen Gross, Tom Lendacky, Joerg Roedel, Mike Stunes,
Kees Cook, kvm, Peter Zijlstra, Cfir Cohen, Joerg Roedel,
Dave Hansen, linux-kernel, Sean Christopherson, virtualization,
Martin Radev, Masami Hiramatsu, Andy Lutomirski, hpa,
Erdem Aktas, David Rientjes, Dan Williams, Jiri Slaby
From: Tom Lendacky <thomas.lendacky@amd.com>
Implement a handler for #VC exceptions caused by INVD instructions.
Since Linux should never use INVD, just mark it as unsupported.
Signed-off-by: Tom Lendacky <thomas.lendacky@amd.com>
[ jroedel@suse.de: Adapt to #VC handling infrastructure ]
Co-developed-by: Joerg Roedel <jroedel@suse.de>
Signed-off-by: Joerg Roedel <jroedel@suse.de>
---
arch/x86/kernel/sev-es.c | 4 ++++
1 file changed, 4 insertions(+)
diff --git a/arch/x86/kernel/sev-es.c b/arch/x86/kernel/sev-es.c
index 2ee600f3184b..7fb17e71ff05 100644
--- a/arch/x86/kernel/sev-es.c
+++ b/arch/x86/kernel/sev-es.c
@@ -892,6 +892,10 @@ static enum es_result vc_handle_exitcode(struct es_em_ctxt *ctxt,
case SVM_EXIT_RDPMC:
result = vc_handle_rdpmc(ghcb, ctxt);
break;
+ case SVM_EXIT_INVD:
+ pr_err_ratelimited("#VC exception for INVD??? Seriously???\n");
+ result = ES_UNSUPPORTED;
+ break;
case SVM_EXIT_CPUID:
result = vc_handle_cpuid(ghcb, ctxt);
break;
--
2.28.0
_______________________________________________
Virtualization mailing list
Virtualization@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/virtualization
^ permalink raw reply related [flat|nested] 264+ messages in thread
* [PATCH v7 57/72] x86/sev-es: Handle MONITOR/MONITORX Events
2020-09-07 13:15 ` Joerg Roedel
@ 2020-09-07 13:15 ` Joerg Roedel
-1 siblings, 0 replies; 264+ messages in thread
From: Joerg Roedel @ 2020-09-07 13:15 UTC (permalink / raw)
To: x86
Cc: Joerg Roedel, Joerg Roedel, Tom Lendacky, hpa, Andy Lutomirski,
Dave Hansen, Peter Zijlstra, Jiri Slaby, Dan Williams,
Juergen Gross, Kees Cook, David Rientjes, Cfir Cohen,
Erdem Aktas, Masami Hiramatsu, Mike Stunes, Sean Christopherson,
Martin Radev, linux-kernel, kvm, virtualization
From: Tom Lendacky <thomas.lendacky@amd.com>
Implement a handler for #VC exceptions caused by MONITOR and MONITORX
instructions.
Signed-off-by: Tom Lendacky <thomas.lendacky@amd.com>
[ jroedel@suse.de: Adapt to #VC handling infrastructure ]
Co-developed-by: Joerg Roedel <jroedel@suse.de>
Signed-off-by: Joerg Roedel <jroedel@suse.de>
---
arch/x86/kernel/sev-es.c | 13 +++++++++++++
1 file changed, 13 insertions(+)
diff --git a/arch/x86/kernel/sev-es.c b/arch/x86/kernel/sev-es.c
index 7fb17e71ff05..847a578d5158 100644
--- a/arch/x86/kernel/sev-es.c
+++ b/arch/x86/kernel/sev-es.c
@@ -872,6 +872,16 @@ static enum es_result vc_handle_rdpmc(struct ghcb *ghcb, struct es_em_ctxt *ctxt
return ES_OK;
}
+static enum es_result vc_handle_monitor(struct ghcb *ghcb,
+ struct es_em_ctxt *ctxt)
+{
+ /*
+ * Treat it as a NOP and do not leak a physical address to the
+ * hypervisor
+ */
+ return ES_OK;
+}
+
static enum es_result vc_handle_exitcode(struct es_em_ctxt *ctxt,
struct ghcb *ghcb,
unsigned long exit_code)
@@ -908,6 +918,9 @@ static enum es_result vc_handle_exitcode(struct es_em_ctxt *ctxt,
case SVM_EXIT_WBINVD:
result = vc_handle_wbinvd(ghcb, ctxt);
break;
+ case SVM_EXIT_MONITOR:
+ result = vc_handle_monitor(ghcb, ctxt);
+ break;
case SVM_EXIT_NPF:
result = vc_handle_mmio(ghcb, ctxt);
break;
--
2.28.0
^ permalink raw reply related [flat|nested] 264+ messages in thread
* [PATCH v7 57/72] x86/sev-es: Handle MONITOR/MONITORX Events
@ 2020-09-07 13:15 ` Joerg Roedel
0 siblings, 0 replies; 264+ messages in thread
From: Joerg Roedel @ 2020-09-07 13:15 UTC (permalink / raw)
To: x86
Cc: Juergen Gross, Tom Lendacky, Joerg Roedel, Mike Stunes,
Kees Cook, kvm, Peter Zijlstra, Cfir Cohen, Joerg Roedel,
Dave Hansen, linux-kernel, Sean Christopherson, virtualization,
Martin Radev, Masami Hiramatsu, Andy Lutomirski, hpa,
Erdem Aktas, David Rientjes, Dan Williams, Jiri Slaby
From: Tom Lendacky <thomas.lendacky@amd.com>
Implement a handler for #VC exceptions caused by MONITOR and MONITORX
instructions.
Signed-off-by: Tom Lendacky <thomas.lendacky@amd.com>
[ jroedel@suse.de: Adapt to #VC handling infrastructure ]
Co-developed-by: Joerg Roedel <jroedel@suse.de>
Signed-off-by: Joerg Roedel <jroedel@suse.de>
---
arch/x86/kernel/sev-es.c | 13 +++++++++++++
1 file changed, 13 insertions(+)
diff --git a/arch/x86/kernel/sev-es.c b/arch/x86/kernel/sev-es.c
index 7fb17e71ff05..847a578d5158 100644
--- a/arch/x86/kernel/sev-es.c
+++ b/arch/x86/kernel/sev-es.c
@@ -872,6 +872,16 @@ static enum es_result vc_handle_rdpmc(struct ghcb *ghcb, struct es_em_ctxt *ctxt
return ES_OK;
}
+static enum es_result vc_handle_monitor(struct ghcb *ghcb,
+ struct es_em_ctxt *ctxt)
+{
+ /*
+ * Treat it as a NOP and do not leak a physical address to the
+ * hypervisor
+ */
+ return ES_OK;
+}
+
static enum es_result vc_handle_exitcode(struct es_em_ctxt *ctxt,
struct ghcb *ghcb,
unsigned long exit_code)
@@ -908,6 +918,9 @@ static enum es_result vc_handle_exitcode(struct es_em_ctxt *ctxt,
case SVM_EXIT_WBINVD:
result = vc_handle_wbinvd(ghcb, ctxt);
break;
+ case SVM_EXIT_MONITOR:
+ result = vc_handle_monitor(ghcb, ctxt);
+ break;
case SVM_EXIT_NPF:
result = vc_handle_mmio(ghcb, ctxt);
break;
--
2.28.0
_______________________________________________
Virtualization mailing list
Virtualization@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/virtualization
^ permalink raw reply related [flat|nested] 264+ messages in thread
* [PATCH v7 58/72] x86/sev-es: Handle MWAIT/MWAITX Events
2020-09-07 13:15 ` Joerg Roedel
@ 2020-09-07 13:15 ` Joerg Roedel
-1 siblings, 0 replies; 264+ messages in thread
From: Joerg Roedel @ 2020-09-07 13:15 UTC (permalink / raw)
To: x86
Cc: Joerg Roedel, Joerg Roedel, Tom Lendacky, hpa, Andy Lutomirski,
Dave Hansen, Peter Zijlstra, Jiri Slaby, Dan Williams,
Juergen Gross, Kees Cook, David Rientjes, Cfir Cohen,
Erdem Aktas, Masami Hiramatsu, Mike Stunes, Sean Christopherson,
Martin Radev, linux-kernel, kvm, virtualization
From: Tom Lendacky <thomas.lendacky@amd.com>
Implement a handler for #VC exceptions caused by MWAIT and MWAITX
instructions.
Signed-off-by: Tom Lendacky <thomas.lendacky@amd.com>
[ jroedel@suse.de: Adapt to #VC handling infrastructure ]
Co-developed-by: Joerg Roedel <jroedel@suse.de>
Signed-off-by: Joerg Roedel <jroedel@suse.de>
---
arch/x86/kernel/sev-es.c | 10 ++++++++++
1 file changed, 10 insertions(+)
diff --git a/arch/x86/kernel/sev-es.c b/arch/x86/kernel/sev-es.c
index 847a578d5158..0d0b6e15f7d9 100644
--- a/arch/x86/kernel/sev-es.c
+++ b/arch/x86/kernel/sev-es.c
@@ -882,6 +882,13 @@ static enum es_result vc_handle_monitor(struct ghcb *ghcb,
return ES_OK;
}
+static enum es_result vc_handle_mwait(struct ghcb *ghcb,
+ struct es_em_ctxt *ctxt)
+{
+ /* Treat the same as MONITOR/MONITORX */
+ return ES_OK;
+}
+
static enum es_result vc_handle_exitcode(struct es_em_ctxt *ctxt,
struct ghcb *ghcb,
unsigned long exit_code)
@@ -921,6 +928,9 @@ static enum es_result vc_handle_exitcode(struct es_em_ctxt *ctxt,
case SVM_EXIT_MONITOR:
result = vc_handle_monitor(ghcb, ctxt);
break;
+ case SVM_EXIT_MWAIT:
+ result = vc_handle_mwait(ghcb, ctxt);
+ break;
case SVM_EXIT_NPF:
result = vc_handle_mmio(ghcb, ctxt);
break;
--
2.28.0
^ permalink raw reply related [flat|nested] 264+ messages in thread
* [PATCH v7 58/72] x86/sev-es: Handle MWAIT/MWAITX Events
@ 2020-09-07 13:15 ` Joerg Roedel
0 siblings, 0 replies; 264+ messages in thread
From: Joerg Roedel @ 2020-09-07 13:15 UTC (permalink / raw)
To: x86
Cc: Juergen Gross, Tom Lendacky, Joerg Roedel, Mike Stunes,
Kees Cook, kvm, Peter Zijlstra, Cfir Cohen, Joerg Roedel,
Dave Hansen, linux-kernel, Sean Christopherson, virtualization,
Martin Radev, Masami Hiramatsu, Andy Lutomirski, hpa,
Erdem Aktas, David Rientjes, Dan Williams, Jiri Slaby
From: Tom Lendacky <thomas.lendacky@amd.com>
Implement a handler for #VC exceptions caused by MWAIT and MWAITX
instructions.
Signed-off-by: Tom Lendacky <thomas.lendacky@amd.com>
[ jroedel@suse.de: Adapt to #VC handling infrastructure ]
Co-developed-by: Joerg Roedel <jroedel@suse.de>
Signed-off-by: Joerg Roedel <jroedel@suse.de>
---
arch/x86/kernel/sev-es.c | 10 ++++++++++
1 file changed, 10 insertions(+)
diff --git a/arch/x86/kernel/sev-es.c b/arch/x86/kernel/sev-es.c
index 847a578d5158..0d0b6e15f7d9 100644
--- a/arch/x86/kernel/sev-es.c
+++ b/arch/x86/kernel/sev-es.c
@@ -882,6 +882,13 @@ static enum es_result vc_handle_monitor(struct ghcb *ghcb,
return ES_OK;
}
+static enum es_result vc_handle_mwait(struct ghcb *ghcb,
+ struct es_em_ctxt *ctxt)
+{
+ /* Treat the same as MONITOR/MONITORX */
+ return ES_OK;
+}
+
static enum es_result vc_handle_exitcode(struct es_em_ctxt *ctxt,
struct ghcb *ghcb,
unsigned long exit_code)
@@ -921,6 +928,9 @@ static enum es_result vc_handle_exitcode(struct es_em_ctxt *ctxt,
case SVM_EXIT_MONITOR:
result = vc_handle_monitor(ghcb, ctxt);
break;
+ case SVM_EXIT_MWAIT:
+ result = vc_handle_mwait(ghcb, ctxt);
+ break;
case SVM_EXIT_NPF:
result = vc_handle_mmio(ghcb, ctxt);
break;
--
2.28.0
_______________________________________________
Virtualization mailing list
Virtualization@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/virtualization
^ permalink raw reply related [flat|nested] 264+ messages in thread
* [PATCH v7 59/72] x86/sev-es: Handle VMMCALL Events
2020-09-07 13:15 ` Joerg Roedel
@ 2020-09-07 13:16 ` Joerg Roedel
-1 siblings, 0 replies; 264+ messages in thread
From: Joerg Roedel @ 2020-09-07 13:16 UTC (permalink / raw)
To: x86
Cc: Joerg Roedel, Joerg Roedel, Tom Lendacky, hpa, Andy Lutomirski,
Dave Hansen, Peter Zijlstra, Jiri Slaby, Dan Williams,
Juergen Gross, Kees Cook, David Rientjes, Cfir Cohen,
Erdem Aktas, Masami Hiramatsu, Mike Stunes, Sean Christopherson,
Martin Radev, linux-kernel, kvm, virtualization
From: Tom Lendacky <thomas.lendacky@amd.com>
Implement a handler for #VC exceptions caused by VMMCALL instructions.
This patch is only a starting point, VMMCALL emulation under SEV-ES
needs further hypervisor-specific changes to provide additional state.
Signed-off-by: Tom Lendacky <thomas.lendacky@amd.com>
[ jroedel@suse.de: Adapt to #VC handling infrastructure ]
Co-developed-by: Joerg Roedel <jroedel@suse.de>
Signed-off-by: Joerg Roedel <jroedel@suse.de>
---
arch/x86/kernel/sev-es.c | 23 +++++++++++++++++++++++
1 file changed, 23 insertions(+)
diff --git a/arch/x86/kernel/sev-es.c b/arch/x86/kernel/sev-es.c
index 0d0b6e15f7d9..db8e33280f7a 100644
--- a/arch/x86/kernel/sev-es.c
+++ b/arch/x86/kernel/sev-es.c
@@ -889,6 +889,26 @@ static enum es_result vc_handle_mwait(struct ghcb *ghcb,
return ES_OK;
}
+static enum es_result vc_handle_vmmcall(struct ghcb *ghcb,
+ struct es_em_ctxt *ctxt)
+{
+ enum es_result ret;
+
+ ghcb_set_rax(ghcb, ctxt->regs->ax);
+ ghcb_set_cpl(ghcb, user_mode(ctxt->regs) ? 3 : 0);
+
+ ret = sev_es_ghcb_hv_call(ghcb, ctxt, SVM_EXIT_VMMCALL, 0, 0);
+ if (ret != ES_OK)
+ return ret;
+
+ if (!ghcb_rax_is_valid(ghcb))
+ return ES_VMM_ERROR;
+
+ ctxt->regs->ax = ghcb->save.rax;
+
+ return ES_OK;
+}
+
static enum es_result vc_handle_exitcode(struct es_em_ctxt *ctxt,
struct ghcb *ghcb,
unsigned long exit_code)
@@ -922,6 +942,9 @@ static enum es_result vc_handle_exitcode(struct es_em_ctxt *ctxt,
case SVM_EXIT_MSR:
result = vc_handle_msr(ghcb, ctxt);
break;
+ case SVM_EXIT_VMMCALL:
+ result = vc_handle_vmmcall(ghcb, ctxt);
+ break;
case SVM_EXIT_WBINVD:
result = vc_handle_wbinvd(ghcb, ctxt);
break;
--
2.28.0
^ permalink raw reply related [flat|nested] 264+ messages in thread
* [PATCH v7 59/72] x86/sev-es: Handle VMMCALL Events
@ 2020-09-07 13:16 ` Joerg Roedel
0 siblings, 0 replies; 264+ messages in thread
From: Joerg Roedel @ 2020-09-07 13:16 UTC (permalink / raw)
To: x86
Cc: Juergen Gross, Tom Lendacky, Joerg Roedel, Mike Stunes,
Kees Cook, kvm, Peter Zijlstra, Cfir Cohen, Joerg Roedel,
Dave Hansen, linux-kernel, Sean Christopherson, virtualization,
Martin Radev, Masami Hiramatsu, Andy Lutomirski, hpa,
Erdem Aktas, David Rientjes, Dan Williams, Jiri Slaby
From: Tom Lendacky <thomas.lendacky@amd.com>
Implement a handler for #VC exceptions caused by VMMCALL instructions.
This patch is only a starting point, VMMCALL emulation under SEV-ES
needs further hypervisor-specific changes to provide additional state.
Signed-off-by: Tom Lendacky <thomas.lendacky@amd.com>
[ jroedel@suse.de: Adapt to #VC handling infrastructure ]
Co-developed-by: Joerg Roedel <jroedel@suse.de>
Signed-off-by: Joerg Roedel <jroedel@suse.de>
---
arch/x86/kernel/sev-es.c | 23 +++++++++++++++++++++++
1 file changed, 23 insertions(+)
diff --git a/arch/x86/kernel/sev-es.c b/arch/x86/kernel/sev-es.c
index 0d0b6e15f7d9..db8e33280f7a 100644
--- a/arch/x86/kernel/sev-es.c
+++ b/arch/x86/kernel/sev-es.c
@@ -889,6 +889,26 @@ static enum es_result vc_handle_mwait(struct ghcb *ghcb,
return ES_OK;
}
+static enum es_result vc_handle_vmmcall(struct ghcb *ghcb,
+ struct es_em_ctxt *ctxt)
+{
+ enum es_result ret;
+
+ ghcb_set_rax(ghcb, ctxt->regs->ax);
+ ghcb_set_cpl(ghcb, user_mode(ctxt->regs) ? 3 : 0);
+
+ ret = sev_es_ghcb_hv_call(ghcb, ctxt, SVM_EXIT_VMMCALL, 0, 0);
+ if (ret != ES_OK)
+ return ret;
+
+ if (!ghcb_rax_is_valid(ghcb))
+ return ES_VMM_ERROR;
+
+ ctxt->regs->ax = ghcb->save.rax;
+
+ return ES_OK;
+}
+
static enum es_result vc_handle_exitcode(struct es_em_ctxt *ctxt,
struct ghcb *ghcb,
unsigned long exit_code)
@@ -922,6 +942,9 @@ static enum es_result vc_handle_exitcode(struct es_em_ctxt *ctxt,
case SVM_EXIT_MSR:
result = vc_handle_msr(ghcb, ctxt);
break;
+ case SVM_EXIT_VMMCALL:
+ result = vc_handle_vmmcall(ghcb, ctxt);
+ break;
case SVM_EXIT_WBINVD:
result = vc_handle_wbinvd(ghcb, ctxt);
break;
--
2.28.0
_______________________________________________
Virtualization mailing list
Virtualization@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/virtualization
^ permalink raw reply related [flat|nested] 264+ messages in thread
* [PATCH v7 60/72] x86/sev-es: Handle #AC Events
2020-09-07 13:15 ` Joerg Roedel
@ 2020-09-07 13:16 ` Joerg Roedel
-1 siblings, 0 replies; 264+ messages in thread
From: Joerg Roedel @ 2020-09-07 13:16 UTC (permalink / raw)
To: x86
Cc: Joerg Roedel, Joerg Roedel, hpa, Andy Lutomirski, Dave Hansen,
Peter Zijlstra, Jiri Slaby, Dan Williams, Tom Lendacky,
Juergen Gross, Kees Cook, David Rientjes, Cfir Cohen,
Erdem Aktas, Masami Hiramatsu, Mike Stunes, Sean Christopherson,
Martin Radev, linux-kernel, kvm, virtualization
From: Joerg Roedel <jroedel@suse.de>
Implement a handler for #VC exceptions caused by #AC exceptions. The #AC
exception is just forwarded to do_alignment_check() and not pushed down
to the hypervisor, as requested by the SEV-ES GHCB Standardization
Specification.
Signed-off-by: Joerg Roedel <jroedel@suse.de>
---
arch/x86/kernel/sev-es.c | 19 +++++++++++++++++++
1 file changed, 19 insertions(+)
diff --git a/arch/x86/kernel/sev-es.c b/arch/x86/kernel/sev-es.c
index db8e33280f7a..f78ab9369b9c 100644
--- a/arch/x86/kernel/sev-es.c
+++ b/arch/x86/kernel/sev-es.c
@@ -909,6 +909,19 @@ static enum es_result vc_handle_vmmcall(struct ghcb *ghcb,
return ES_OK;
}
+static enum es_result vc_handle_trap_ac(struct ghcb *ghcb,
+ struct es_em_ctxt *ctxt)
+{
+ /*
+ * Calling ecx_alignment_check() directly does not work, because it
+ * enables IRQs and the GHCB is active. Forward the exception and call
+ * it later from vc_forward_exception().
+ */
+ ctxt->fi.vector = X86_TRAP_AC;
+ ctxt->fi.error_code = 0;
+ return ES_EXCEPTION;
+}
+
static enum es_result vc_handle_exitcode(struct es_em_ctxt *ctxt,
struct ghcb *ghcb,
unsigned long exit_code)
@@ -922,6 +935,9 @@ static enum es_result vc_handle_exitcode(struct es_em_ctxt *ctxt,
case SVM_EXIT_WRITE_DR7:
result = vc_handle_dr7_write(ghcb, ctxt);
break;
+ case SVM_EXIT_EXCP_BASE + X86_TRAP_AC:
+ result = vc_handle_trap_ac(ghcb, ctxt);
+ break;
case SVM_EXIT_RDTSC:
case SVM_EXIT_RDTSCP:
result = vc_handle_rdtsc(ghcb, ctxt, exit_code);
@@ -981,6 +997,9 @@ static __always_inline void vc_forward_exception(struct es_em_ctxt *ctxt)
case X86_TRAP_UD:
exc_invalid_op(ctxt->regs);
break;
+ case X86_TRAP_AC:
+ exc_alignment_check(ctxt->regs, error_code);
+ break;
default:
pr_emerg("Unsupported exception in #VC instruction emulation - can't continue\n");
BUG();
--
2.28.0
^ permalink raw reply related [flat|nested] 264+ messages in thread
* [PATCH v7 60/72] x86/sev-es: Handle #AC Events
@ 2020-09-07 13:16 ` Joerg Roedel
0 siblings, 0 replies; 264+ messages in thread
From: Joerg Roedel @ 2020-09-07 13:16 UTC (permalink / raw)
To: x86
Cc: Juergen Gross, Tom Lendacky, Joerg Roedel, Mike Stunes,
Kees Cook, kvm, Peter Zijlstra, Cfir Cohen, Joerg Roedel,
Dave Hansen, linux-kernel, Sean Christopherson, virtualization,
Martin Radev, Masami Hiramatsu, Andy Lutomirski, hpa,
Erdem Aktas, David Rientjes, Dan Williams, Jiri Slaby
From: Joerg Roedel <jroedel@suse.de>
Implement a handler for #VC exceptions caused by #AC exceptions. The #AC
exception is just forwarded to do_alignment_check() and not pushed down
to the hypervisor, as requested by the SEV-ES GHCB Standardization
Specification.
Signed-off-by: Joerg Roedel <jroedel@suse.de>
---
arch/x86/kernel/sev-es.c | 19 +++++++++++++++++++
1 file changed, 19 insertions(+)
diff --git a/arch/x86/kernel/sev-es.c b/arch/x86/kernel/sev-es.c
index db8e33280f7a..f78ab9369b9c 100644
--- a/arch/x86/kernel/sev-es.c
+++ b/arch/x86/kernel/sev-es.c
@@ -909,6 +909,19 @@ static enum es_result vc_handle_vmmcall(struct ghcb *ghcb,
return ES_OK;
}
+static enum es_result vc_handle_trap_ac(struct ghcb *ghcb,
+ struct es_em_ctxt *ctxt)
+{
+ /*
+ * Calling ecx_alignment_check() directly does not work, because it
+ * enables IRQs and the GHCB is active. Forward the exception and call
+ * it later from vc_forward_exception().
+ */
+ ctxt->fi.vector = X86_TRAP_AC;
+ ctxt->fi.error_code = 0;
+ return ES_EXCEPTION;
+}
+
static enum es_result vc_handle_exitcode(struct es_em_ctxt *ctxt,
struct ghcb *ghcb,
unsigned long exit_code)
@@ -922,6 +935,9 @@ static enum es_result vc_handle_exitcode(struct es_em_ctxt *ctxt,
case SVM_EXIT_WRITE_DR7:
result = vc_handle_dr7_write(ghcb, ctxt);
break;
+ case SVM_EXIT_EXCP_BASE + X86_TRAP_AC:
+ result = vc_handle_trap_ac(ghcb, ctxt);
+ break;
case SVM_EXIT_RDTSC:
case SVM_EXIT_RDTSCP:
result = vc_handle_rdtsc(ghcb, ctxt, exit_code);
@@ -981,6 +997,9 @@ static __always_inline void vc_forward_exception(struct es_em_ctxt *ctxt)
case X86_TRAP_UD:
exc_invalid_op(ctxt->regs);
break;
+ case X86_TRAP_AC:
+ exc_alignment_check(ctxt->regs, error_code);
+ break;
default:
pr_emerg("Unsupported exception in #VC instruction emulation - can't continue\n");
BUG();
--
2.28.0
_______________________________________________
Virtualization mailing list
Virtualization@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/virtualization
^ permalink raw reply related [flat|nested] 264+ messages in thread
* [PATCH v7 61/72] x86/sev-es: Handle #DB Events
2020-09-07 13:15 ` Joerg Roedel
@ 2020-09-07 13:16 ` Joerg Roedel
-1 siblings, 0 replies; 264+ messages in thread
From: Joerg Roedel @ 2020-09-07 13:16 UTC (permalink / raw)
To: x86
Cc: Joerg Roedel, Joerg Roedel, hpa, Andy Lutomirski, Dave Hansen,
Peter Zijlstra, Jiri Slaby, Dan Williams, Tom Lendacky,
Juergen Gross, Kees Cook, David Rientjes, Cfir Cohen,
Erdem Aktas, Masami Hiramatsu, Mike Stunes, Sean Christopherson,
Martin Radev, linux-kernel, kvm, virtualization
From: Joerg Roedel <jroedel@suse.de>
Handle #VC exceptions caused by #DB exceptions in the guest. Those
must be handled outside of instrumentation_begin()/end() so that the
handler will not be raised recursively.
Handle them by calling the kernels debug exception handler.
Signed-off-by: Joerg Roedel <jroedel@suse.de>
---
arch/x86/kernel/sev-es.c | 17 +++++++++++++++++
1 file changed, 17 insertions(+)
diff --git a/arch/x86/kernel/sev-es.c b/arch/x86/kernel/sev-es.c
index f78ab9369b9c..3eea4e910fc1 100644
--- a/arch/x86/kernel/sev-es.c
+++ b/arch/x86/kernel/sev-es.c
@@ -922,6 +922,14 @@ static enum es_result vc_handle_trap_ac(struct ghcb *ghcb,
return ES_EXCEPTION;
}
+static __always_inline void vc_handle_trap_db(struct pt_regs *regs)
+{
+ if (user_mode(regs))
+ noist_exc_debug(regs);
+ else
+ exc_debug(regs);
+}
+
static enum es_result vc_handle_exitcode(struct es_em_ctxt *ctxt,
struct ghcb *ghcb,
unsigned long exit_code)
@@ -1033,6 +1041,15 @@ DEFINE_IDTENTRY_VC_SAFE_STACK(exc_vmm_communication)
struct ghcb *ghcb;
lockdep_assert_irqs_disabled();
+
+ /*
+ * Handle #DB before calling into !noinstr code to avoid recursive #DB.
+ */
+ if (error_code == SVM_EXIT_EXCP_BASE + X86_TRAP_DB) {
+ vc_handle_trap_db(regs);
+ return;
+ }
+
instrumentation_begin();
/*
--
2.28.0
^ permalink raw reply related [flat|nested] 264+ messages in thread
* [PATCH v7 61/72] x86/sev-es: Handle #DB Events
@ 2020-09-07 13:16 ` Joerg Roedel
0 siblings, 0 replies; 264+ messages in thread
From: Joerg Roedel @ 2020-09-07 13:16 UTC (permalink / raw)
To: x86
Cc: Juergen Gross, Tom Lendacky, Joerg Roedel, Mike Stunes,
Kees Cook, kvm, Peter Zijlstra, Cfir Cohen, Joerg Roedel,
Dave Hansen, linux-kernel, Sean Christopherson, virtualization,
Martin Radev, Masami Hiramatsu, Andy Lutomirski, hpa,
Erdem Aktas, David Rientjes, Dan Williams, Jiri Slaby
From: Joerg Roedel <jroedel@suse.de>
Handle #VC exceptions caused by #DB exceptions in the guest. Those
must be handled outside of instrumentation_begin()/end() so that the
handler will not be raised recursively.
Handle them by calling the kernels debug exception handler.
Signed-off-by: Joerg Roedel <jroedel@suse.de>
---
arch/x86/kernel/sev-es.c | 17 +++++++++++++++++
1 file changed, 17 insertions(+)
diff --git a/arch/x86/kernel/sev-es.c b/arch/x86/kernel/sev-es.c
index f78ab9369b9c..3eea4e910fc1 100644
--- a/arch/x86/kernel/sev-es.c
+++ b/arch/x86/kernel/sev-es.c
@@ -922,6 +922,14 @@ static enum es_result vc_handle_trap_ac(struct ghcb *ghcb,
return ES_EXCEPTION;
}
+static __always_inline void vc_handle_trap_db(struct pt_regs *regs)
+{
+ if (user_mode(regs))
+ noist_exc_debug(regs);
+ else
+ exc_debug(regs);
+}
+
static enum es_result vc_handle_exitcode(struct es_em_ctxt *ctxt,
struct ghcb *ghcb,
unsigned long exit_code)
@@ -1033,6 +1041,15 @@ DEFINE_IDTENTRY_VC_SAFE_STACK(exc_vmm_communication)
struct ghcb *ghcb;
lockdep_assert_irqs_disabled();
+
+ /*
+ * Handle #DB before calling into !noinstr code to avoid recursive #DB.
+ */
+ if (error_code == SVM_EXIT_EXCP_BASE + X86_TRAP_DB) {
+ vc_handle_trap_db(regs);
+ return;
+ }
+
instrumentation_begin();
/*
--
2.28.0
_______________________________________________
Virtualization mailing list
Virtualization@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/virtualization
^ permalink raw reply related [flat|nested] 264+ messages in thread
* [PATCH v7 62/72] x86/paravirt: Allow hypervisor specific VMMCALL handling under SEV-ES
2020-09-07 13:15 ` Joerg Roedel
@ 2020-09-07 13:16 ` Joerg Roedel
-1 siblings, 0 replies; 264+ messages in thread
From: Joerg Roedel @ 2020-09-07 13:16 UTC (permalink / raw)
To: x86
Cc: Joerg Roedel, Joerg Roedel, hpa, Andy Lutomirski, Dave Hansen,
Peter Zijlstra, Jiri Slaby, Dan Williams, Tom Lendacky,
Juergen Gross, Kees Cook, David Rientjes, Cfir Cohen,
Erdem Aktas, Masami Hiramatsu, Mike Stunes, Sean Christopherson,
Martin Radev, linux-kernel, kvm, virtualization
From: Joerg Roedel <jroedel@suse.de>
Add two new paravirt callbacks to provide hypervisor specific processor
state in the GHCB and to copy state from the hypervisor back to the
processor.
Signed-off-by: Joerg Roedel <jroedel@suse.de>
---
arch/x86/include/asm/x86_init.h | 16 +++++++++++++++-
arch/x86/kernel/sev-es.c | 12 ++++++++++++
2 files changed, 27 insertions(+), 1 deletion(-)
diff --git a/arch/x86/include/asm/x86_init.h b/arch/x86/include/asm/x86_init.h
index 6807153c0410..0304e2931cd3 100644
--- a/arch/x86/include/asm/x86_init.h
+++ b/arch/x86/include/asm/x86_init.h
@@ -4,8 +4,10 @@
#include <asm/bootparam.h>
+struct ghcb;
struct mpc_bus;
struct mpc_cpu;
+struct pt_regs;
struct mpc_table;
struct cpuinfo_x86;
@@ -236,10 +238,22 @@ struct x86_legacy_features {
/**
* struct x86_hyper_runtime - x86 hypervisor specific runtime callbacks
*
- * @pin_vcpu: pin current vcpu to specified physical cpu (run rarely)
+ * @pin_vcpu: pin current vcpu to specified physical
+ * cpu (run rarely)
+ * @sev_es_hcall_prepare: Load additional hypervisor-specific
+ * state into the GHCB when doing a VMMCALL under
+ * SEV-ES. Called from the #VC exception handler.
+ * @sev_es_hcall_finish: Copies state from the GHCB back into the
+ * processor (or pt_regs). Also runs checks on the
+ * state returned from the hypervisor after a
+ * VMMCALL under SEV-ES. Needs to return 'false'
+ * if the checks fail. Called from the #VC
+ * exception handler.
*/
struct x86_hyper_runtime {
void (*pin_vcpu)(int cpu);
+ void (*sev_es_hcall_prepare)(struct ghcb *ghcb, struct pt_regs *regs);
+ bool (*sev_es_hcall_finish)(struct ghcb *ghcb, struct pt_regs *regs);
};
/**
diff --git a/arch/x86/kernel/sev-es.c b/arch/x86/kernel/sev-es.c
index 3eea4e910fc1..9814ac99aea1 100644
--- a/arch/x86/kernel/sev-es.c
+++ b/arch/x86/kernel/sev-es.c
@@ -897,6 +897,9 @@ static enum es_result vc_handle_vmmcall(struct ghcb *ghcb,
ghcb_set_rax(ghcb, ctxt->regs->ax);
ghcb_set_cpl(ghcb, user_mode(ctxt->regs) ? 3 : 0);
+ if (x86_platform.hyper.sev_es_hcall_prepare)
+ x86_platform.hyper.sev_es_hcall_prepare(ghcb, ctxt->regs);
+
ret = sev_es_ghcb_hv_call(ghcb, ctxt, SVM_EXIT_VMMCALL, 0, 0);
if (ret != ES_OK)
return ret;
@@ -906,6 +909,15 @@ static enum es_result vc_handle_vmmcall(struct ghcb *ghcb,
ctxt->regs->ax = ghcb->save.rax;
+ /*
+ * Call sev_es_hcall_finish() after regs->ax is already set.
+ * This allows the hypervisor handler to overwrite it again if
+ * necessary.
+ */
+ if (x86_platform.hyper.sev_es_hcall_finish &&
+ !x86_platform.hyper.sev_es_hcall_finish(ghcb, ctxt->regs))
+ return ES_VMM_ERROR;
+
return ES_OK;
}
--
2.28.0
^ permalink raw reply related [flat|nested] 264+ messages in thread
* [PATCH v7 62/72] x86/paravirt: Allow hypervisor specific VMMCALL handling under SEV-ES
@ 2020-09-07 13:16 ` Joerg Roedel
0 siblings, 0 replies; 264+ messages in thread
From: Joerg Roedel @ 2020-09-07 13:16 UTC (permalink / raw)
To: x86
Cc: Juergen Gross, Tom Lendacky, Joerg Roedel, Mike Stunes,
Kees Cook, kvm, Peter Zijlstra, Cfir Cohen, Joerg Roedel,
Dave Hansen, linux-kernel, Sean Christopherson, virtualization,
Martin Radev, Masami Hiramatsu, Andy Lutomirski, hpa,
Erdem Aktas, David Rientjes, Dan Williams, Jiri Slaby
From: Joerg Roedel <jroedel@suse.de>
Add two new paravirt callbacks to provide hypervisor specific processor
state in the GHCB and to copy state from the hypervisor back to the
processor.
Signed-off-by: Joerg Roedel <jroedel@suse.de>
---
arch/x86/include/asm/x86_init.h | 16 +++++++++++++++-
arch/x86/kernel/sev-es.c | 12 ++++++++++++
2 files changed, 27 insertions(+), 1 deletion(-)
diff --git a/arch/x86/include/asm/x86_init.h b/arch/x86/include/asm/x86_init.h
index 6807153c0410..0304e2931cd3 100644
--- a/arch/x86/include/asm/x86_init.h
+++ b/arch/x86/include/asm/x86_init.h
@@ -4,8 +4,10 @@
#include <asm/bootparam.h>
+struct ghcb;
struct mpc_bus;
struct mpc_cpu;
+struct pt_regs;
struct mpc_table;
struct cpuinfo_x86;
@@ -236,10 +238,22 @@ struct x86_legacy_features {
/**
* struct x86_hyper_runtime - x86 hypervisor specific runtime callbacks
*
- * @pin_vcpu: pin current vcpu to specified physical cpu (run rarely)
+ * @pin_vcpu: pin current vcpu to specified physical
+ * cpu (run rarely)
+ * @sev_es_hcall_prepare: Load additional hypervisor-specific
+ * state into the GHCB when doing a VMMCALL under
+ * SEV-ES. Called from the #VC exception handler.
+ * @sev_es_hcall_finish: Copies state from the GHCB back into the
+ * processor (or pt_regs). Also runs checks on the
+ * state returned from the hypervisor after a
+ * VMMCALL under SEV-ES. Needs to return 'false'
+ * if the checks fail. Called from the #VC
+ * exception handler.
*/
struct x86_hyper_runtime {
void (*pin_vcpu)(int cpu);
+ void (*sev_es_hcall_prepare)(struct ghcb *ghcb, struct pt_regs *regs);
+ bool (*sev_es_hcall_finish)(struct ghcb *ghcb, struct pt_regs *regs);
};
/**
diff --git a/arch/x86/kernel/sev-es.c b/arch/x86/kernel/sev-es.c
index 3eea4e910fc1..9814ac99aea1 100644
--- a/arch/x86/kernel/sev-es.c
+++ b/arch/x86/kernel/sev-es.c
@@ -897,6 +897,9 @@ static enum es_result vc_handle_vmmcall(struct ghcb *ghcb,
ghcb_set_rax(ghcb, ctxt->regs->ax);
ghcb_set_cpl(ghcb, user_mode(ctxt->regs) ? 3 : 0);
+ if (x86_platform.hyper.sev_es_hcall_prepare)
+ x86_platform.hyper.sev_es_hcall_prepare(ghcb, ctxt->regs);
+
ret = sev_es_ghcb_hv_call(ghcb, ctxt, SVM_EXIT_VMMCALL, 0, 0);
if (ret != ES_OK)
return ret;
@@ -906,6 +909,15 @@ static enum es_result vc_handle_vmmcall(struct ghcb *ghcb,
ctxt->regs->ax = ghcb->save.rax;
+ /*
+ * Call sev_es_hcall_finish() after regs->ax is already set.
+ * This allows the hypervisor handler to overwrite it again if
+ * necessary.
+ */
+ if (x86_platform.hyper.sev_es_hcall_finish &&
+ !x86_platform.hyper.sev_es_hcall_finish(ghcb, ctxt->regs))
+ return ES_VMM_ERROR;
+
return ES_OK;
}
--
2.28.0
_______________________________________________
Virtualization mailing list
Virtualization@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/virtualization
^ permalink raw reply related [flat|nested] 264+ messages in thread
* [PATCH v7 63/72] x86/kvm: Add KVM specific VMMCALL handling under SEV-ES
2020-09-07 13:15 ` Joerg Roedel
@ 2020-09-07 13:16 ` Joerg Roedel
-1 siblings, 0 replies; 264+ messages in thread
From: Joerg Roedel @ 2020-09-07 13:16 UTC (permalink / raw)
To: x86
Cc: Joerg Roedel, Joerg Roedel, Tom Lendacky, hpa, Andy Lutomirski,
Dave Hansen, Peter Zijlstra, Jiri Slaby, Dan Williams,
Juergen Gross, Kees Cook, David Rientjes, Cfir Cohen,
Erdem Aktas, Masami Hiramatsu, Mike Stunes, Sean Christopherson,
Martin Radev, linux-kernel, kvm, virtualization
From: Tom Lendacky <thomas.lendacky@amd.com>
Implement the callbacks to copy the processor state required by KVM to
the GHCB.
Signed-off-by: Tom Lendacky <thomas.lendacky@amd.com>
[ jroedel@suse.de: - Split out of a larger patch
- Adapt to different callback functions ]
Co-developed-by: Joerg Roedel <jroedel@suse.de>
Signed-off-by: Joerg Roedel <jroedel@suse.de>
---
arch/x86/kernel/kvm.c | 35 +++++++++++++++++++++++++++++------
1 file changed, 29 insertions(+), 6 deletions(-)
diff --git a/arch/x86/kernel/kvm.c b/arch/x86/kernel/kvm.c
index 08320b0b2b27..0f9597275e9c 100644
--- a/arch/x86/kernel/kvm.c
+++ b/arch/x86/kernel/kvm.c
@@ -36,6 +36,8 @@
#include <asm/hypervisor.h>
#include <asm/tlb.h>
#include <asm/cpuidle_haltpoll.h>
+#include <asm/ptrace.h>
+#include <asm/svm.h>
DEFINE_STATIC_KEY_FALSE(kvm_async_pf_enabled);
@@ -746,13 +748,34 @@ static void __init kvm_init_platform(void)
x86_platform.apic_post_init = kvm_apic_init;
}
+#if defined(CONFIG_AMD_MEM_ENCRYPT)
+static void kvm_sev_es_hcall_prepare(struct ghcb *ghcb, struct pt_regs *regs)
+{
+ /* RAX and CPL are already in the GHCB */
+ ghcb_set_rbx(ghcb, regs->bx);
+ ghcb_set_rcx(ghcb, regs->cx);
+ ghcb_set_rdx(ghcb, regs->dx);
+ ghcb_set_rsi(ghcb, regs->si);
+}
+
+static bool kvm_sev_es_hcall_finish(struct ghcb *ghcb, struct pt_regs *regs)
+{
+ /* No checking of the return state needed */
+ return true;
+}
+#endif
+
const __initconst struct hypervisor_x86 x86_hyper_kvm = {
- .name = "KVM",
- .detect = kvm_detect,
- .type = X86_HYPER_KVM,
- .init.guest_late_init = kvm_guest_init,
- .init.x2apic_available = kvm_para_available,
- .init.init_platform = kvm_init_platform,
+ .name = "KVM",
+ .detect = kvm_detect,
+ .type = X86_HYPER_KVM,
+ .init.guest_late_init = kvm_guest_init,
+ .init.x2apic_available = kvm_para_available,
+ .init.init_platform = kvm_init_platform,
+#if defined(CONFIG_AMD_MEM_ENCRYPT)
+ .runtime.sev_es_hcall_prepare = kvm_sev_es_hcall_prepare,
+ .runtime.sev_es_hcall_finish = kvm_sev_es_hcall_finish,
+#endif
};
static __init int activate_jump_labels(void)
--
2.28.0
^ permalink raw reply related [flat|nested] 264+ messages in thread
* [PATCH v7 63/72] x86/kvm: Add KVM specific VMMCALL handling under SEV-ES
@ 2020-09-07 13:16 ` Joerg Roedel
0 siblings, 0 replies; 264+ messages in thread
From: Joerg Roedel @ 2020-09-07 13:16 UTC (permalink / raw)
To: x86
Cc: Juergen Gross, Tom Lendacky, Joerg Roedel, Mike Stunes,
Kees Cook, kvm, Peter Zijlstra, Cfir Cohen, Joerg Roedel,
Dave Hansen, linux-kernel, Sean Christopherson, virtualization,
Martin Radev, Masami Hiramatsu, Andy Lutomirski, hpa,
Erdem Aktas, David Rientjes, Dan Williams, Jiri Slaby
From: Tom Lendacky <thomas.lendacky@amd.com>
Implement the callbacks to copy the processor state required by KVM to
the GHCB.
Signed-off-by: Tom Lendacky <thomas.lendacky@amd.com>
[ jroedel@suse.de: - Split out of a larger patch
- Adapt to different callback functions ]
Co-developed-by: Joerg Roedel <jroedel@suse.de>
Signed-off-by: Joerg Roedel <jroedel@suse.de>
---
arch/x86/kernel/kvm.c | 35 +++++++++++++++++++++++++++++------
1 file changed, 29 insertions(+), 6 deletions(-)
diff --git a/arch/x86/kernel/kvm.c b/arch/x86/kernel/kvm.c
index 08320b0b2b27..0f9597275e9c 100644
--- a/arch/x86/kernel/kvm.c
+++ b/arch/x86/kernel/kvm.c
@@ -36,6 +36,8 @@
#include <asm/hypervisor.h>
#include <asm/tlb.h>
#include <asm/cpuidle_haltpoll.h>
+#include <asm/ptrace.h>
+#include <asm/svm.h>
DEFINE_STATIC_KEY_FALSE(kvm_async_pf_enabled);
@@ -746,13 +748,34 @@ static void __init kvm_init_platform(void)
x86_platform.apic_post_init = kvm_apic_init;
}
+#if defined(CONFIG_AMD_MEM_ENCRYPT)
+static void kvm_sev_es_hcall_prepare(struct ghcb *ghcb, struct pt_regs *regs)
+{
+ /* RAX and CPL are already in the GHCB */
+ ghcb_set_rbx(ghcb, regs->bx);
+ ghcb_set_rcx(ghcb, regs->cx);
+ ghcb_set_rdx(ghcb, regs->dx);
+ ghcb_set_rsi(ghcb, regs->si);
+}
+
+static bool kvm_sev_es_hcall_finish(struct ghcb *ghcb, struct pt_regs *regs)
+{
+ /* No checking of the return state needed */
+ return true;
+}
+#endif
+
const __initconst struct hypervisor_x86 x86_hyper_kvm = {
- .name = "KVM",
- .detect = kvm_detect,
- .type = X86_HYPER_KVM,
- .init.guest_late_init = kvm_guest_init,
- .init.x2apic_available = kvm_para_available,
- .init.init_platform = kvm_init_platform,
+ .name = "KVM",
+ .detect = kvm_detect,
+ .type = X86_HYPER_KVM,
+ .init.guest_late_init = kvm_guest_init,
+ .init.x2apic_available = kvm_para_available,
+ .init.init_platform = kvm_init_platform,
+#if defined(CONFIG_AMD_MEM_ENCRYPT)
+ .runtime.sev_es_hcall_prepare = kvm_sev_es_hcall_prepare,
+ .runtime.sev_es_hcall_finish = kvm_sev_es_hcall_finish,
+#endif
};
static __init int activate_jump_labels(void)
--
2.28.0
_______________________________________________
Virtualization mailing list
Virtualization@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/virtualization
^ permalink raw reply related [flat|nested] 264+ messages in thread
* [PATCH v7 64/72] x86/vmware: Add VMware specific handling for VMMCALL under SEV-ES
2020-09-07 13:15 ` Joerg Roedel
@ 2020-09-07 13:16 ` Joerg Roedel
-1 siblings, 0 replies; 264+ messages in thread
From: Joerg Roedel @ 2020-09-07 13:16 UTC (permalink / raw)
To: x86
Cc: Joerg Roedel, Joerg Roedel, Doug Covelli, Tom Lendacky, hpa,
Andy Lutomirski, Dave Hansen, Peter Zijlstra, Jiri Slaby,
Dan Williams, Juergen Gross, Kees Cook, David Rientjes,
Cfir Cohen, Erdem Aktas, Masami Hiramatsu, Mike Stunes,
Sean Christopherson, Martin Radev, linux-kernel, kvm,
virtualization
From: Doug Covelli <dcovelli@vmware.com>
Add VMware specific handling for #VC faults caused by VMMCALL
instructions.
Signed-off-by: Doug Covelli <dcovelli@vmware.com>
Signed-off-by: Tom Lendacky <thomas.lendacky@amd.com>
[ jroedel@suse.de: - Adapt to different paravirt interface ]
Co-developed-by: Joerg Roedel <jroedel@suse.de>
Signed-off-by: Joerg Roedel <jroedel@suse.de>
---
arch/x86/kernel/cpu/vmware.c | 50 ++++++++++++++++++++++++++++++++----
1 file changed, 45 insertions(+), 5 deletions(-)
diff --git a/arch/x86/kernel/cpu/vmware.c b/arch/x86/kernel/cpu/vmware.c
index 9b6fafa69be9..924571fe5864 100644
--- a/arch/x86/kernel/cpu/vmware.c
+++ b/arch/x86/kernel/cpu/vmware.c
@@ -33,6 +33,7 @@
#include <asm/timer.h>
#include <asm/apic.h>
#include <asm/vmware.h>
+#include <asm/svm.h>
#undef pr_fmt
#define pr_fmt(fmt) "vmware: " fmt
@@ -476,10 +477,49 @@ static bool __init vmware_legacy_x2apic_available(void)
(eax & (1 << VMWARE_CMD_LEGACY_X2APIC)) != 0;
}
+#ifdef CONFIG_AMD_MEM_ENCRYPT
+static void vmware_sev_es_hcall_prepare(struct ghcb *ghcb,
+ struct pt_regs *regs)
+{
+ /* Copy VMWARE specific Hypercall parameters to the GHCB */
+ ghcb_set_rip(ghcb, regs->ip);
+ ghcb_set_rbx(ghcb, regs->bx);
+ ghcb_set_rcx(ghcb, regs->cx);
+ ghcb_set_rdx(ghcb, regs->dx);
+ ghcb_set_rsi(ghcb, regs->si);
+ ghcb_set_rdi(ghcb, regs->di);
+ ghcb_set_rbp(ghcb, regs->bp);
+}
+
+static bool vmware_sev_es_hcall_finish(struct ghcb *ghcb, struct pt_regs *regs)
+{
+ if (!(ghcb_rbx_is_valid(ghcb) &&
+ ghcb_rcx_is_valid(ghcb) &&
+ ghcb_rdx_is_valid(ghcb) &&
+ ghcb_rsi_is_valid(ghcb) &&
+ ghcb_rdi_is_valid(ghcb) &&
+ ghcb_rbp_is_valid(ghcb)))
+ return false;
+
+ regs->bx = ghcb->save.rbx;
+ regs->cx = ghcb->save.rcx;
+ regs->dx = ghcb->save.rdx;
+ regs->si = ghcb->save.rsi;
+ regs->di = ghcb->save.rdi;
+ regs->bp = ghcb->save.rbp;
+
+ return true;
+}
+#endif
+
const __initconst struct hypervisor_x86 x86_hyper_vmware = {
- .name = "VMware",
- .detect = vmware_platform,
- .type = X86_HYPER_VMWARE,
- .init.init_platform = vmware_platform_setup,
- .init.x2apic_available = vmware_legacy_x2apic_available,
+ .name = "VMware",
+ .detect = vmware_platform,
+ .type = X86_HYPER_VMWARE,
+ .init.init_platform = vmware_platform_setup,
+ .init.x2apic_available = vmware_legacy_x2apic_available,
+#ifdef CONFIG_AMD_MEM_ENCRYPT
+ .runtime.sev_es_hcall_prepare = vmware_sev_es_hcall_prepare,
+ .runtime.sev_es_hcall_finish = vmware_sev_es_hcall_finish,
+#endif
};
--
2.28.0
^ permalink raw reply related [flat|nested] 264+ messages in thread
* [PATCH v7 64/72] x86/vmware: Add VMware specific handling for VMMCALL under SEV-ES
@ 2020-09-07 13:16 ` Joerg Roedel
0 siblings, 0 replies; 264+ messages in thread
From: Joerg Roedel @ 2020-09-07 13:16 UTC (permalink / raw)
To: x86
Cc: kvm, Peter Zijlstra, Dave Hansen, virtualization, Doug Covelli,
hpa, Jiri Slaby, Joerg Roedel, David Rientjes, Martin Radev,
Tom Lendacky, Joerg Roedel, Kees Cook, Cfir Cohen,
Andy Lutomirski, Dan Williams, Juergen Gross, Mike Stunes,
linux-kernel, Sean Christopherson, Masami Hiramatsu, Erdem Aktas
From: Doug Covelli <dcovelli@vmware.com>
Add VMware specific handling for #VC faults caused by VMMCALL
instructions.
Signed-off-by: Doug Covelli <dcovelli@vmware.com>
Signed-off-by: Tom Lendacky <thomas.lendacky@amd.com>
[ jroedel@suse.de: - Adapt to different paravirt interface ]
Co-developed-by: Joerg Roedel <jroedel@suse.de>
Signed-off-by: Joerg Roedel <jroedel@suse.de>
---
arch/x86/kernel/cpu/vmware.c | 50 ++++++++++++++++++++++++++++++++----
1 file changed, 45 insertions(+), 5 deletions(-)
diff --git a/arch/x86/kernel/cpu/vmware.c b/arch/x86/kernel/cpu/vmware.c
index 9b6fafa69be9..924571fe5864 100644
--- a/arch/x86/kernel/cpu/vmware.c
+++ b/arch/x86/kernel/cpu/vmware.c
@@ -33,6 +33,7 @@
#include <asm/timer.h>
#include <asm/apic.h>
#include <asm/vmware.h>
+#include <asm/svm.h>
#undef pr_fmt
#define pr_fmt(fmt) "vmware: " fmt
@@ -476,10 +477,49 @@ static bool __init vmware_legacy_x2apic_available(void)
(eax & (1 << VMWARE_CMD_LEGACY_X2APIC)) != 0;
}
+#ifdef CONFIG_AMD_MEM_ENCRYPT
+static void vmware_sev_es_hcall_prepare(struct ghcb *ghcb,
+ struct pt_regs *regs)
+{
+ /* Copy VMWARE specific Hypercall parameters to the GHCB */
+ ghcb_set_rip(ghcb, regs->ip);
+ ghcb_set_rbx(ghcb, regs->bx);
+ ghcb_set_rcx(ghcb, regs->cx);
+ ghcb_set_rdx(ghcb, regs->dx);
+ ghcb_set_rsi(ghcb, regs->si);
+ ghcb_set_rdi(ghcb, regs->di);
+ ghcb_set_rbp(ghcb, regs->bp);
+}
+
+static bool vmware_sev_es_hcall_finish(struct ghcb *ghcb, struct pt_regs *regs)
+{
+ if (!(ghcb_rbx_is_valid(ghcb) &&
+ ghcb_rcx_is_valid(ghcb) &&
+ ghcb_rdx_is_valid(ghcb) &&
+ ghcb_rsi_is_valid(ghcb) &&
+ ghcb_rdi_is_valid(ghcb) &&
+ ghcb_rbp_is_valid(ghcb)))
+ return false;
+
+ regs->bx = ghcb->save.rbx;
+ regs->cx = ghcb->save.rcx;
+ regs->dx = ghcb->save.rdx;
+ regs->si = ghcb->save.rsi;
+ regs->di = ghcb->save.rdi;
+ regs->bp = ghcb->save.rbp;
+
+ return true;
+}
+#endif
+
const __initconst struct hypervisor_x86 x86_hyper_vmware = {
- .name = "VMware",
- .detect = vmware_platform,
- .type = X86_HYPER_VMWARE,
- .init.init_platform = vmware_platform_setup,
- .init.x2apic_available = vmware_legacy_x2apic_available,
+ .name = "VMware",
+ .detect = vmware_platform,
+ .type = X86_HYPER_VMWARE,
+ .init.init_platform = vmware_platform_setup,
+ .init.x2apic_available = vmware_legacy_x2apic_available,
+#ifdef CONFIG_AMD_MEM_ENCRYPT
+ .runtime.sev_es_hcall_prepare = vmware_sev_es_hcall_prepare,
+ .runtime.sev_es_hcall_finish = vmware_sev_es_hcall_finish,
+#endif
};
--
2.28.0
_______________________________________________
Virtualization mailing list
Virtualization@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/virtualization
^ permalink raw reply related [flat|nested] 264+ messages in thread
* [PATCH v7 65/72] x86/realmode: Add SEV-ES specific trampoline entry point
2020-09-07 13:15 ` Joerg Roedel
@ 2020-09-07 13:16 ` Joerg Roedel
-1 siblings, 0 replies; 264+ messages in thread
From: Joerg Roedel @ 2020-09-07 13:16 UTC (permalink / raw)
To: x86
Cc: Joerg Roedel, Joerg Roedel, hpa, Andy Lutomirski, Dave Hansen,
Peter Zijlstra, Jiri Slaby, Dan Williams, Tom Lendacky,
Juergen Gross, Kees Cook, David Rientjes, Cfir Cohen,
Erdem Aktas, Masami Hiramatsu, Mike Stunes, Sean Christopherson,
Martin Radev, linux-kernel, kvm, virtualization
From: Joerg Roedel <jroedel@suse.de>
The code at the trampoline entry point is executed in real-mode. In
real-mode #VC exceptions can't be handled, so anything that might cause
such an exception must be avoided.
In the standard trampoline entry code this is the WBINVD instruction and
the call to verify_cpu(), which are both not needed anyway when running
as an SEV-ES guest.
Signed-off-by: Joerg Roedel <jroedel@suse.de>
---
arch/x86/include/asm/realmode.h | 3 +++
| 3 +++
arch/x86/realmode/rm/trampoline_64.S | 20 ++++++++++++++++++++
3 files changed, 26 insertions(+)
diff --git a/arch/x86/include/asm/realmode.h b/arch/x86/include/asm/realmode.h
index 96118fb041b8..4d4d853f6841 100644
--- a/arch/x86/include/asm/realmode.h
+++ b/arch/x86/include/asm/realmode.h
@@ -21,6 +21,9 @@ struct real_mode_header {
/* SMP trampoline */
u32 trampoline_start;
u32 trampoline_header;
+#ifdef CONFIG_AMD_MEM_ENCRYPT
+ u32 sev_es_trampoline_start;
+#endif
#ifdef CONFIG_X86_64
u32 trampoline_pgd;
#endif
--git a/arch/x86/realmode/rm/header.S b/arch/x86/realmode/rm/header.S
index af04512c02d9..8c1db5bf5d78 100644
--- a/arch/x86/realmode/rm/header.S
+++ b/arch/x86/realmode/rm/header.S
@@ -20,6 +20,9 @@ SYM_DATA_START(real_mode_header)
/* SMP trampoline */
.long pa_trampoline_start
.long pa_trampoline_header
+#ifdef CONFIG_AMD_MEM_ENCRYPT
+ .long pa_sev_es_trampoline_start
+#endif
#ifdef CONFIG_X86_64
.long pa_trampoline_pgd;
#endif
diff --git a/arch/x86/realmode/rm/trampoline_64.S b/arch/x86/realmode/rm/trampoline_64.S
index 251758ed7443..84c5d1b33d10 100644
--- a/arch/x86/realmode/rm/trampoline_64.S
+++ b/arch/x86/realmode/rm/trampoline_64.S
@@ -56,6 +56,7 @@ SYM_CODE_START(trampoline_start)
testl %eax, %eax # Check for return code
jnz no_longmode
+.Lswitch_to_protected:
/*
* GDT tables in non default location kernel can be beyond 16MB and
* lgdt will not be able to load the address as in real mode default
@@ -80,6 +81,25 @@ no_longmode:
jmp no_longmode
SYM_CODE_END(trampoline_start)
+#ifdef CONFIG_AMD_MEM_ENCRYPT
+/* SEV-ES supports non-zero IP for entry points - no alignment needed */
+SYM_CODE_START(sev_es_trampoline_start)
+ cli # We should be safe anyway
+
+ LJMPW_RM(1f)
+1:
+ mov %cs, %ax # Code and data in the same place
+ mov %ax, %ds
+ mov %ax, %es
+ mov %ax, %ss
+
+ # Setup stack
+ movl $rm_stack_end, %esp
+
+ jmp .Lswitch_to_protected
+SYM_CODE_END(sev_es_trampoline_start)
+#endif /* CONFIG_AMD_MEM_ENCRYPT */
+
#include "../kernel/verify_cpu.S"
.section ".text32","ax"
--
2.28.0
^ permalink raw reply related [flat|nested] 264+ messages in thread
* [PATCH v7 65/72] x86/realmode: Add SEV-ES specific trampoline entry point
@ 2020-09-07 13:16 ` Joerg Roedel
0 siblings, 0 replies; 264+ messages in thread
From: Joerg Roedel @ 2020-09-07 13:16 UTC (permalink / raw)
To: x86
Cc: Juergen Gross, Tom Lendacky, Joerg Roedel, Mike Stunes,
Kees Cook, kvm, Peter Zijlstra, Cfir Cohen, Joerg Roedel,
Dave Hansen, linux-kernel, Sean Christopherson, virtualization,
Martin Radev, Masami Hiramatsu, Andy Lutomirski, hpa,
Erdem Aktas, David Rientjes, Dan Williams, Jiri Slaby
From: Joerg Roedel <jroedel@suse.de>
The code at the trampoline entry point is executed in real-mode. In
real-mode #VC exceptions can't be handled, so anything that might cause
such an exception must be avoided.
In the standard trampoline entry code this is the WBINVD instruction and
the call to verify_cpu(), which are both not needed anyway when running
as an SEV-ES guest.
Signed-off-by: Joerg Roedel <jroedel@suse.de>
---
arch/x86/include/asm/realmode.h | 3 +++
| 3 +++
arch/x86/realmode/rm/trampoline_64.S | 20 ++++++++++++++++++++
3 files changed, 26 insertions(+)
diff --git a/arch/x86/include/asm/realmode.h b/arch/x86/include/asm/realmode.h
index 96118fb041b8..4d4d853f6841 100644
--- a/arch/x86/include/asm/realmode.h
+++ b/arch/x86/include/asm/realmode.h
@@ -21,6 +21,9 @@ struct real_mode_header {
/* SMP trampoline */
u32 trampoline_start;
u32 trampoline_header;
+#ifdef CONFIG_AMD_MEM_ENCRYPT
+ u32 sev_es_trampoline_start;
+#endif
#ifdef CONFIG_X86_64
u32 trampoline_pgd;
#endif
--git a/arch/x86/realmode/rm/header.S b/arch/x86/realmode/rm/header.S
index af04512c02d9..8c1db5bf5d78 100644
--- a/arch/x86/realmode/rm/header.S
+++ b/arch/x86/realmode/rm/header.S
@@ -20,6 +20,9 @@ SYM_DATA_START(real_mode_header)
/* SMP trampoline */
.long pa_trampoline_start
.long pa_trampoline_header
+#ifdef CONFIG_AMD_MEM_ENCRYPT
+ .long pa_sev_es_trampoline_start
+#endif
#ifdef CONFIG_X86_64
.long pa_trampoline_pgd;
#endif
diff --git a/arch/x86/realmode/rm/trampoline_64.S b/arch/x86/realmode/rm/trampoline_64.S
index 251758ed7443..84c5d1b33d10 100644
--- a/arch/x86/realmode/rm/trampoline_64.S
+++ b/arch/x86/realmode/rm/trampoline_64.S
@@ -56,6 +56,7 @@ SYM_CODE_START(trampoline_start)
testl %eax, %eax # Check for return code
jnz no_longmode
+.Lswitch_to_protected:
/*
* GDT tables in non default location kernel can be beyond 16MB and
* lgdt will not be able to load the address as in real mode default
@@ -80,6 +81,25 @@ no_longmode:
jmp no_longmode
SYM_CODE_END(trampoline_start)
+#ifdef CONFIG_AMD_MEM_ENCRYPT
+/* SEV-ES supports non-zero IP for entry points - no alignment needed */
+SYM_CODE_START(sev_es_trampoline_start)
+ cli # We should be safe anyway
+
+ LJMPW_RM(1f)
+1:
+ mov %cs, %ax # Code and data in the same place
+ mov %ax, %ds
+ mov %ax, %es
+ mov %ax, %ss
+
+ # Setup stack
+ movl $rm_stack_end, %esp
+
+ jmp .Lswitch_to_protected
+SYM_CODE_END(sev_es_trampoline_start)
+#endif /* CONFIG_AMD_MEM_ENCRYPT */
+
#include "../kernel/verify_cpu.S"
.section ".text32","ax"
--
2.28.0
_______________________________________________
Virtualization mailing list
Virtualization@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/virtualization
^ permalink raw reply related [flat|nested] 264+ messages in thread
* [PATCH v7 66/72] x86/realmode: Setup AP jump table
2020-09-07 13:15 ` Joerg Roedel
@ 2020-09-07 13:16 ` Joerg Roedel
-1 siblings, 0 replies; 264+ messages in thread
From: Joerg Roedel @ 2020-09-07 13:16 UTC (permalink / raw)
To: x86
Cc: Joerg Roedel, Joerg Roedel, Tom Lendacky, hpa, Andy Lutomirski,
Dave Hansen, Peter Zijlstra, Jiri Slaby, Dan Williams,
Juergen Gross, Kees Cook, David Rientjes, Cfir Cohen,
Erdem Aktas, Masami Hiramatsu, Mike Stunes, Sean Christopherson,
Martin Radev, linux-kernel, kvm, virtualization
From: Tom Lendacky <thomas.lendacky@amd.com>
As part of the GHCB specification, the booting of APs under SEV-ES
requires an AP jump table when transitioning from one layer of code to
another (e.g. when going from UEFI to the OS). As a result, each layer
that parks an AP must provide the physical address of an AP jump table
to the next layer via the hypervisor.
Upon booting of the kernel, read the AP jump table address from the
hypervisor. Under SEV-ES, APs are started using the INIT-SIPI-SIPI
sequence. Before issuing the first SIPI request for an AP, the start
CS and IP is programmed into the AP jump table. Upon issuing the SIPI
request, the AP will awaken and jump to that start CS:IP address.
Signed-off-by: Tom Lendacky <thomas.lendacky@amd.com>
[ jroedel@suse.de: - Adapted to different code base
- Moved AP table setup from SIPI sending path to
real-mode setup code
- Fix sparse warnings ]
Co-developed-by: Joerg Roedel <jroedel@suse.de>
Signed-off-by: Joerg Roedel <jroedel@suse.de>
---
arch/x86/include/asm/sev-es.h | 5 +++
arch/x86/include/uapi/asm/svm.h | 3 ++
arch/x86/kernel/sev-es.c | 69 +++++++++++++++++++++++++++++++++
arch/x86/realmode/init.c | 18 ++++++++-
4 files changed, 93 insertions(+), 2 deletions(-)
diff --git a/arch/x86/include/asm/sev-es.h b/arch/x86/include/asm/sev-es.h
index 59176e8c6b81..db88e1c3442d 100644
--- a/arch/x86/include/asm/sev-es.h
+++ b/arch/x86/include/asm/sev-es.h
@@ -73,6 +73,9 @@ static inline u64 lower_bits(u64 val, unsigned int bits)
return (val & mask);
}
+struct real_mode_header;
+enum stack_type;
+
/* Early IDT entry points for #VC handler */
extern void vc_no_ghcb(void);
extern void vc_boot_ghcb(void);
@@ -92,9 +95,11 @@ static __always_inline void sev_es_ist_exit(void)
if (static_branch_unlikely(&sev_es_enable_key))
__sev_es_ist_exit();
}
+extern int sev_es_setup_ap_jump_table(struct real_mode_header *rmh);
#else
static inline void sev_es_ist_enter(struct pt_regs *regs) { }
static inline void sev_es_ist_exit(void) { }
+static inline int sev_es_setup_ap_jump_table(struct real_mode_header *rmh) { return 0; }
#endif
#endif
diff --git a/arch/x86/include/uapi/asm/svm.h b/arch/x86/include/uapi/asm/svm.h
index 8f36ae021a7f..346b8a7155e8 100644
--- a/arch/x86/include/uapi/asm/svm.h
+++ b/arch/x86/include/uapi/asm/svm.h
@@ -84,6 +84,9 @@
/* SEV-ES software-defined VMGEXIT events */
#define SVM_VMGEXIT_MMIO_READ 0x80000001
#define SVM_VMGEXIT_MMIO_WRITE 0x80000002
+#define SVM_VMGEXIT_AP_JUMP_TABLE 0x80000005
+#define SVM_VMGEXIT_SET_AP_JUMP_TABLE 0
+#define SVM_VMGEXIT_GET_AP_JUMP_TABLE 1
#define SVM_VMGEXIT_UNSUPPORTED_EVENT 0x8000ffff
#define SVM_EXIT_ERR -1
diff --git a/arch/x86/kernel/sev-es.c b/arch/x86/kernel/sev-es.c
index 9814ac99aea1..91e32172d355 100644
--- a/arch/x86/kernel/sev-es.c
+++ b/arch/x86/kernel/sev-es.c
@@ -21,6 +21,7 @@
#include <linux/mm.h>
#include <asm/cpu_entry_area.h>
+#include <asm/stacktrace.h>
#include <asm/sev-es.h>
#include <asm/insn-eval.h>
#include <asm/fpu/internal.h>
@@ -214,6 +215,9 @@ static __always_inline void sev_es_put_ghcb(struct ghcb_state *state)
}
}
+/* Needed in vc_early_forward_exception */
+void do_early_exception(struct pt_regs *regs, int trapnr);
+
static inline u64 sev_es_rd_ghcb_msr(void)
{
return native_read_msr(MSR_AMD64_SEV_ES_GHCB);
@@ -402,6 +406,71 @@ static bool vc_slow_virt_to_phys(struct ghcb *ghcb, struct es_em_ctxt *ctxt,
/* Include code shared with pre-decompression boot stage */
#include "sev-es-shared.c"
+static u64 get_jump_table_addr(void)
+{
+ struct ghcb_state state;
+ unsigned long flags;
+ struct ghcb *ghcb;
+ u64 ret = 0;
+
+ local_irq_save(flags);
+
+ ghcb = sev_es_get_ghcb(&state);
+
+ vc_ghcb_invalidate(ghcb);
+ ghcb_set_sw_exit_code(ghcb, SVM_VMGEXIT_AP_JUMP_TABLE);
+ ghcb_set_sw_exit_info_1(ghcb, SVM_VMGEXIT_GET_AP_JUMP_TABLE);
+ ghcb_set_sw_exit_info_2(ghcb, 0);
+
+ sev_es_wr_ghcb_msr(__pa(ghcb));
+ VMGEXIT();
+
+ if (ghcb_sw_exit_info_1_is_valid(ghcb) &&
+ ghcb_sw_exit_info_2_is_valid(ghcb))
+ ret = ghcb->save.sw_exit_info_2;
+
+ sev_es_put_ghcb(&state);
+
+ local_irq_restore(flags);
+
+ return ret;
+}
+
+int sev_es_setup_ap_jump_table(struct real_mode_header *rmh)
+{
+ u16 startup_cs, startup_ip;
+ phys_addr_t jump_table_pa;
+ u64 jump_table_addr;
+ u16 __iomem *jump_table;
+
+ jump_table_addr = get_jump_table_addr();
+
+ /* On UP guests there is no jump table so this is not a failure */
+ if (!jump_table_addr)
+ return 0;
+
+ /* Check if AP Jump Table is page-aligned */
+ if (jump_table_addr & ~PAGE_MASK)
+ return -EINVAL;
+
+ jump_table_pa = jump_table_addr & PAGE_MASK;
+
+ startup_cs = (u16)(rmh->trampoline_start >> 4);
+ startup_ip = (u16)(rmh->sev_es_trampoline_start -
+ rmh->trampoline_start);
+
+ jump_table = ioremap_encrypted(jump_table_pa, PAGE_SIZE);
+ if (!jump_table)
+ return -EIO;
+
+ writew(startup_ip, &jump_table[0]);
+ writew(startup_cs, &jump_table[1]);
+
+ iounmap(jump_table);
+
+ return 0;
+}
+
static enum es_result vc_handle_msr(struct ghcb *ghcb, struct es_em_ctxt *ctxt)
{
struct pt_regs *regs = ctxt->regs;
diff --git a/arch/x86/realmode/init.c b/arch/x86/realmode/init.c
index 1ed1208931e0..3fb9b60be07a 100644
--- a/arch/x86/realmode/init.c
+++ b/arch/x86/realmode/init.c
@@ -9,6 +9,7 @@
#include <asm/realmode.h>
#include <asm/tlbflush.h>
#include <asm/crash.h>
+#include <asm/sev-es.h>
struct real_mode_header *real_mode_header;
u32 *trampoline_cr4_features;
@@ -38,6 +39,19 @@ void __init reserve_real_mode(void)
crash_reserve_low_1M();
}
+static void sme_sev_setup_real_mode(struct trampoline_header *th)
+{
+#ifdef CONFIG_AMD_MEM_ENCRYPT
+ if (sme_active())
+ th->flags |= TH_FLAGS_SME_ACTIVE;
+
+ if (sev_es_active()) {
+ if (sev_es_setup_ap_jump_table(real_mode_header))
+ panic("Failed to get/update SEV-ES AP Jump Table");
+ }
+#endif
+}
+
static void __init setup_real_mode(void)
{
u16 real_mode_seg;
@@ -104,13 +118,13 @@ static void __init setup_real_mode(void)
*trampoline_cr4_features = mmu_cr4_features;
trampoline_header->flags = 0;
- if (sme_active())
- trampoline_header->flags |= TH_FLAGS_SME_ACTIVE;
trampoline_pgd = (u64 *) __va(real_mode_header->trampoline_pgd);
trampoline_pgd[0] = trampoline_pgd_entry.pgd;
trampoline_pgd[511] = init_top_pgt[511].pgd;
#endif
+
+ sme_sev_setup_real_mode(trampoline_header);
}
/*
--
2.28.0
^ permalink raw reply related [flat|nested] 264+ messages in thread
* [PATCH v7 66/72] x86/realmode: Setup AP jump table
@ 2020-09-07 13:16 ` Joerg Roedel
0 siblings, 0 replies; 264+ messages in thread
From: Joerg Roedel @ 2020-09-07 13:16 UTC (permalink / raw)
To: x86
Cc: Juergen Gross, Tom Lendacky, Joerg Roedel, Mike Stunes,
Kees Cook, kvm, Peter Zijlstra, Cfir Cohen, Joerg Roedel,
Dave Hansen, linux-kernel, Sean Christopherson, virtualization,
Martin Radev, Masami Hiramatsu, Andy Lutomirski, hpa,
Erdem Aktas, David Rientjes, Dan Williams, Jiri Slaby
From: Tom Lendacky <thomas.lendacky@amd.com>
As part of the GHCB specification, the booting of APs under SEV-ES
requires an AP jump table when transitioning from one layer of code to
another (e.g. when going from UEFI to the OS). As a result, each layer
that parks an AP must provide the physical address of an AP jump table
to the next layer via the hypervisor.
Upon booting of the kernel, read the AP jump table address from the
hypervisor. Under SEV-ES, APs are started using the INIT-SIPI-SIPI
sequence. Before issuing the first SIPI request for an AP, the start
CS and IP is programmed into the AP jump table. Upon issuing the SIPI
request, the AP will awaken and jump to that start CS:IP address.
Signed-off-by: Tom Lendacky <thomas.lendacky@amd.com>
[ jroedel@suse.de: - Adapted to different code base
- Moved AP table setup from SIPI sending path to
real-mode setup code
- Fix sparse warnings ]
Co-developed-by: Joerg Roedel <jroedel@suse.de>
Signed-off-by: Joerg Roedel <jroedel@suse.de>
---
arch/x86/include/asm/sev-es.h | 5 +++
arch/x86/include/uapi/asm/svm.h | 3 ++
arch/x86/kernel/sev-es.c | 69 +++++++++++++++++++++++++++++++++
arch/x86/realmode/init.c | 18 ++++++++-
4 files changed, 93 insertions(+), 2 deletions(-)
diff --git a/arch/x86/include/asm/sev-es.h b/arch/x86/include/asm/sev-es.h
index 59176e8c6b81..db88e1c3442d 100644
--- a/arch/x86/include/asm/sev-es.h
+++ b/arch/x86/include/asm/sev-es.h
@@ -73,6 +73,9 @@ static inline u64 lower_bits(u64 val, unsigned int bits)
return (val & mask);
}
+struct real_mode_header;
+enum stack_type;
+
/* Early IDT entry points for #VC handler */
extern void vc_no_ghcb(void);
extern void vc_boot_ghcb(void);
@@ -92,9 +95,11 @@ static __always_inline void sev_es_ist_exit(void)
if (static_branch_unlikely(&sev_es_enable_key))
__sev_es_ist_exit();
}
+extern int sev_es_setup_ap_jump_table(struct real_mode_header *rmh);
#else
static inline void sev_es_ist_enter(struct pt_regs *regs) { }
static inline void sev_es_ist_exit(void) { }
+static inline int sev_es_setup_ap_jump_table(struct real_mode_header *rmh) { return 0; }
#endif
#endif
diff --git a/arch/x86/include/uapi/asm/svm.h b/arch/x86/include/uapi/asm/svm.h
index 8f36ae021a7f..346b8a7155e8 100644
--- a/arch/x86/include/uapi/asm/svm.h
+++ b/arch/x86/include/uapi/asm/svm.h
@@ -84,6 +84,9 @@
/* SEV-ES software-defined VMGEXIT events */
#define SVM_VMGEXIT_MMIO_READ 0x80000001
#define SVM_VMGEXIT_MMIO_WRITE 0x80000002
+#define SVM_VMGEXIT_AP_JUMP_TABLE 0x80000005
+#define SVM_VMGEXIT_SET_AP_JUMP_TABLE 0
+#define SVM_VMGEXIT_GET_AP_JUMP_TABLE 1
#define SVM_VMGEXIT_UNSUPPORTED_EVENT 0x8000ffff
#define SVM_EXIT_ERR -1
diff --git a/arch/x86/kernel/sev-es.c b/arch/x86/kernel/sev-es.c
index 9814ac99aea1..91e32172d355 100644
--- a/arch/x86/kernel/sev-es.c
+++ b/arch/x86/kernel/sev-es.c
@@ -21,6 +21,7 @@
#include <linux/mm.h>
#include <asm/cpu_entry_area.h>
+#include <asm/stacktrace.h>
#include <asm/sev-es.h>
#include <asm/insn-eval.h>
#include <asm/fpu/internal.h>
@@ -214,6 +215,9 @@ static __always_inline void sev_es_put_ghcb(struct ghcb_state *state)
}
}
+/* Needed in vc_early_forward_exception */
+void do_early_exception(struct pt_regs *regs, int trapnr);
+
static inline u64 sev_es_rd_ghcb_msr(void)
{
return native_read_msr(MSR_AMD64_SEV_ES_GHCB);
@@ -402,6 +406,71 @@ static bool vc_slow_virt_to_phys(struct ghcb *ghcb, struct es_em_ctxt *ctxt,
/* Include code shared with pre-decompression boot stage */
#include "sev-es-shared.c"
+static u64 get_jump_table_addr(void)
+{
+ struct ghcb_state state;
+ unsigned long flags;
+ struct ghcb *ghcb;
+ u64 ret = 0;
+
+ local_irq_save(flags);
+
+ ghcb = sev_es_get_ghcb(&state);
+
+ vc_ghcb_invalidate(ghcb);
+ ghcb_set_sw_exit_code(ghcb, SVM_VMGEXIT_AP_JUMP_TABLE);
+ ghcb_set_sw_exit_info_1(ghcb, SVM_VMGEXIT_GET_AP_JUMP_TABLE);
+ ghcb_set_sw_exit_info_2(ghcb, 0);
+
+ sev_es_wr_ghcb_msr(__pa(ghcb));
+ VMGEXIT();
+
+ if (ghcb_sw_exit_info_1_is_valid(ghcb) &&
+ ghcb_sw_exit_info_2_is_valid(ghcb))
+ ret = ghcb->save.sw_exit_info_2;
+
+ sev_es_put_ghcb(&state);
+
+ local_irq_restore(flags);
+
+ return ret;
+}
+
+int sev_es_setup_ap_jump_table(struct real_mode_header *rmh)
+{
+ u16 startup_cs, startup_ip;
+ phys_addr_t jump_table_pa;
+ u64 jump_table_addr;
+ u16 __iomem *jump_table;
+
+ jump_table_addr = get_jump_table_addr();
+
+ /* On UP guests there is no jump table so this is not a failure */
+ if (!jump_table_addr)
+ return 0;
+
+ /* Check if AP Jump Table is page-aligned */
+ if (jump_table_addr & ~PAGE_MASK)
+ return -EINVAL;
+
+ jump_table_pa = jump_table_addr & PAGE_MASK;
+
+ startup_cs = (u16)(rmh->trampoline_start >> 4);
+ startup_ip = (u16)(rmh->sev_es_trampoline_start -
+ rmh->trampoline_start);
+
+ jump_table = ioremap_encrypted(jump_table_pa, PAGE_SIZE);
+ if (!jump_table)
+ return -EIO;
+
+ writew(startup_ip, &jump_table[0]);
+ writew(startup_cs, &jump_table[1]);
+
+ iounmap(jump_table);
+
+ return 0;
+}
+
static enum es_result vc_handle_msr(struct ghcb *ghcb, struct es_em_ctxt *ctxt)
{
struct pt_regs *regs = ctxt->regs;
diff --git a/arch/x86/realmode/init.c b/arch/x86/realmode/init.c
index 1ed1208931e0..3fb9b60be07a 100644
--- a/arch/x86/realmode/init.c
+++ b/arch/x86/realmode/init.c
@@ -9,6 +9,7 @@
#include <asm/realmode.h>
#include <asm/tlbflush.h>
#include <asm/crash.h>
+#include <asm/sev-es.h>
struct real_mode_header *real_mode_header;
u32 *trampoline_cr4_features;
@@ -38,6 +39,19 @@ void __init reserve_real_mode(void)
crash_reserve_low_1M();
}
+static void sme_sev_setup_real_mode(struct trampoline_header *th)
+{
+#ifdef CONFIG_AMD_MEM_ENCRYPT
+ if (sme_active())
+ th->flags |= TH_FLAGS_SME_ACTIVE;
+
+ if (sev_es_active()) {
+ if (sev_es_setup_ap_jump_table(real_mode_header))
+ panic("Failed to get/update SEV-ES AP Jump Table");
+ }
+#endif
+}
+
static void __init setup_real_mode(void)
{
u16 real_mode_seg;
@@ -104,13 +118,13 @@ static void __init setup_real_mode(void)
*trampoline_cr4_features = mmu_cr4_features;
trampoline_header->flags = 0;
- if (sme_active())
- trampoline_header->flags |= TH_FLAGS_SME_ACTIVE;
trampoline_pgd = (u64 *) __va(real_mode_header->trampoline_pgd);
trampoline_pgd[0] = trampoline_pgd_entry.pgd;
trampoline_pgd[511] = init_top_pgt[511].pgd;
#endif
+
+ sme_sev_setup_real_mode(trampoline_header);
}
/*
--
2.28.0
_______________________________________________
Virtualization mailing list
Virtualization@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/virtualization
^ permalink raw reply related [flat|nested] 264+ messages in thread
* [PATCH v7 67/72] x86/smpboot: Load TSS and getcpu GDT entry before loading IDT
2020-09-07 13:15 ` Joerg Roedel
@ 2020-09-07 13:16 ` Joerg Roedel
-1 siblings, 0 replies; 264+ messages in thread
From: Joerg Roedel @ 2020-09-07 13:16 UTC (permalink / raw)
To: x86
Cc: Joerg Roedel, Joerg Roedel, hpa, Andy Lutomirski, Dave Hansen,
Peter Zijlstra, Jiri Slaby, Dan Williams, Tom Lendacky,
Juergen Gross, Kees Cook, David Rientjes, Cfir Cohen,
Erdem Aktas, Masami Hiramatsu, Mike Stunes, Sean Christopherson,
Martin Radev, linux-kernel, kvm, virtualization
From: Joerg Roedel <jroedel@suse.de>
The IDT on 64bit contains vectors which use paranoid_entry() and/or IST
stacks. To make these vectors work the TSS and the getcpu GDT entry need
to be set up before the IDT is loaded.
Signed-off-by: Joerg Roedel <jroedel@suse.de>
---
arch/x86/include/asm/processor.h | 1 +
arch/x86/kernel/cpu/common.c | 23 +++++++++++++++++++++++
arch/x86/kernel/smpboot.c | 2 +-
3 files changed, 25 insertions(+), 1 deletion(-)
diff --git a/arch/x86/include/asm/processor.h b/arch/x86/include/asm/processor.h
index d8a82e650810..5ac507586769 100644
--- a/arch/x86/include/asm/processor.h
+++ b/arch/x86/include/asm/processor.h
@@ -696,6 +696,7 @@ extern void load_direct_gdt(int);
extern void load_fixmap_gdt(int);
extern void load_percpu_segment(int);
extern void cpu_init(void);
+extern void cpu_init_exception_handling(void);
extern void cr4_init(void);
static inline unsigned long get_debugctlmsr(void)
diff --git a/arch/x86/kernel/cpu/common.c b/arch/x86/kernel/cpu/common.c
index 1d65365363a1..a9527c0c38fb 100644
--- a/arch/x86/kernel/cpu/common.c
+++ b/arch/x86/kernel/cpu/common.c
@@ -1854,6 +1854,29 @@ static inline void tss_setup_io_bitmap(struct tss_struct *tss)
#endif
}
+/*
+ * Setup everything needed to handle exceptions from the IDT, including the IST
+ * exceptions which use paranoid_entry()
+ */
+void cpu_init_exception_handling(void)
+{
+ struct tss_struct *tss = this_cpu_ptr(&cpu_tss_rw);
+ int cpu = raw_smp_processor_id();
+
+ /* paranoid_entry() gets the CPU number from the GDT */
+ setup_getcpu(cpu);
+
+ /* IST vectors need TSS to be set up. */
+ tss_setup_ist(tss);
+ tss_setup_io_bitmap(tss);
+ set_tss_desc(cpu, &get_cpu_entry_area(cpu)->tss.x86_tss);
+
+ load_TR_desc();
+
+ /* Finally load the IDT */
+ load_current_idt();
+}
+
/*
* cpu_init() initializes state that is per-CPU. Some data is already
* initialized (naturally) in the bootstrap process, such as the GDT
diff --git a/arch/x86/kernel/smpboot.c b/arch/x86/kernel/smpboot.c
index f5ef689dd62a..de776b2e6046 100644
--- a/arch/x86/kernel/smpboot.c
+++ b/arch/x86/kernel/smpboot.c
@@ -227,7 +227,7 @@ static void notrace start_secondary(void *unused)
load_cr3(swapper_pg_dir);
__flush_tlb_all();
#endif
- load_current_idt();
+ cpu_init_exception_handling();
cpu_init();
x86_cpuinit.early_percpu_clock_init();
preempt_disable();
--
2.28.0
^ permalink raw reply related [flat|nested] 264+ messages in thread
* [PATCH v7 67/72] x86/smpboot: Load TSS and getcpu GDT entry before loading IDT
@ 2020-09-07 13:16 ` Joerg Roedel
0 siblings, 0 replies; 264+ messages in thread
From: Joerg Roedel @ 2020-09-07 13:16 UTC (permalink / raw)
To: x86
Cc: Juergen Gross, Tom Lendacky, Joerg Roedel, Mike Stunes,
Kees Cook, kvm, Peter Zijlstra, Cfir Cohen, Joerg Roedel,
Dave Hansen, linux-kernel, Sean Christopherson, virtualization,
Martin Radev, Masami Hiramatsu, Andy Lutomirski, hpa,
Erdem Aktas, David Rientjes, Dan Williams, Jiri Slaby
From: Joerg Roedel <jroedel@suse.de>
The IDT on 64bit contains vectors which use paranoid_entry() and/or IST
stacks. To make these vectors work the TSS and the getcpu GDT entry need
to be set up before the IDT is loaded.
Signed-off-by: Joerg Roedel <jroedel@suse.de>
---
arch/x86/include/asm/processor.h | 1 +
arch/x86/kernel/cpu/common.c | 23 +++++++++++++++++++++++
arch/x86/kernel/smpboot.c | 2 +-
3 files changed, 25 insertions(+), 1 deletion(-)
diff --git a/arch/x86/include/asm/processor.h b/arch/x86/include/asm/processor.h
index d8a82e650810..5ac507586769 100644
--- a/arch/x86/include/asm/processor.h
+++ b/arch/x86/include/asm/processor.h
@@ -696,6 +696,7 @@ extern void load_direct_gdt(int);
extern void load_fixmap_gdt(int);
extern void load_percpu_segment(int);
extern void cpu_init(void);
+extern void cpu_init_exception_handling(void);
extern void cr4_init(void);
static inline unsigned long get_debugctlmsr(void)
diff --git a/arch/x86/kernel/cpu/common.c b/arch/x86/kernel/cpu/common.c
index 1d65365363a1..a9527c0c38fb 100644
--- a/arch/x86/kernel/cpu/common.c
+++ b/arch/x86/kernel/cpu/common.c
@@ -1854,6 +1854,29 @@ static inline void tss_setup_io_bitmap(struct tss_struct *tss)
#endif
}
+/*
+ * Setup everything needed to handle exceptions from the IDT, including the IST
+ * exceptions which use paranoid_entry()
+ */
+void cpu_init_exception_handling(void)
+{
+ struct tss_struct *tss = this_cpu_ptr(&cpu_tss_rw);
+ int cpu = raw_smp_processor_id();
+
+ /* paranoid_entry() gets the CPU number from the GDT */
+ setup_getcpu(cpu);
+
+ /* IST vectors need TSS to be set up. */
+ tss_setup_ist(tss);
+ tss_setup_io_bitmap(tss);
+ set_tss_desc(cpu, &get_cpu_entry_area(cpu)->tss.x86_tss);
+
+ load_TR_desc();
+
+ /* Finally load the IDT */
+ load_current_idt();
+}
+
/*
* cpu_init() initializes state that is per-CPU. Some data is already
* initialized (naturally) in the bootstrap process, such as the GDT
diff --git a/arch/x86/kernel/smpboot.c b/arch/x86/kernel/smpboot.c
index f5ef689dd62a..de776b2e6046 100644
--- a/arch/x86/kernel/smpboot.c
+++ b/arch/x86/kernel/smpboot.c
@@ -227,7 +227,7 @@ static void notrace start_secondary(void *unused)
load_cr3(swapper_pg_dir);
__flush_tlb_all();
#endif
- load_current_idt();
+ cpu_init_exception_handling();
cpu_init();
x86_cpuinit.early_percpu_clock_init();
preempt_disable();
--
2.28.0
_______________________________________________
Virtualization mailing list
Virtualization@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/virtualization
^ permalink raw reply related [flat|nested] 264+ messages in thread
* [PATCH v7 68/72] x86/head/64: Don't call verify_cpu() on starting APs
2020-09-07 13:15 ` Joerg Roedel
@ 2020-09-07 13:16 ` Joerg Roedel
-1 siblings, 0 replies; 264+ messages in thread
From: Joerg Roedel @ 2020-09-07 13:16 UTC (permalink / raw)
To: x86
Cc: Joerg Roedel, Joerg Roedel, Kees Cook, hpa, Andy Lutomirski,
Dave Hansen, Peter Zijlstra, Jiri Slaby, Dan Williams,
Tom Lendacky, Juergen Gross, David Rientjes, Cfir Cohen,
Erdem Aktas, Masami Hiramatsu, Mike Stunes, Sean Christopherson,
Martin Radev, linux-kernel, kvm, virtualization
From: Joerg Roedel <jroedel@suse.de>
The APs are not ready to handle exceptions when verify_cpu() is called
in secondary_startup_64.
Signed-off-by: Joerg Roedel <jroedel@suse.de>
Reviewed-by: Kees Cook <keescook@chromium.org>
---
arch/x86/include/asm/realmode.h | 1 +
arch/x86/kernel/head_64.S | 12 ++++++++++++
arch/x86/realmode/init.c | 6 ++++++
3 files changed, 19 insertions(+)
diff --git a/arch/x86/include/asm/realmode.h b/arch/x86/include/asm/realmode.h
index 4d4d853f6841..5db5d083c873 100644
--- a/arch/x86/include/asm/realmode.h
+++ b/arch/x86/include/asm/realmode.h
@@ -72,6 +72,7 @@ extern unsigned char startup_32_smp[];
extern unsigned char boot_gdt[];
#else
extern unsigned char secondary_startup_64[];
+extern unsigned char secondary_startup_64_no_verify[];
#endif
static inline size_t real_mode_size_needed(void)
diff --git a/arch/x86/kernel/head_64.S b/arch/x86/kernel/head_64.S
index 41057ff79284..d976a9e6fcba 100644
--- a/arch/x86/kernel/head_64.S
+++ b/arch/x86/kernel/head_64.S
@@ -125,6 +125,18 @@ SYM_CODE_START(secondary_startup_64)
/* Sanitize CPU configuration */
call verify_cpu
+ /*
+ * The secondary_startup_64_no_verify entry point is only used by
+ * SEV-ES guests. In those guests the call to verify_cpu() would cause
+ * #VC exceptions which can not be handled at this stage of secondary
+ * CPU bringup.
+ *
+ * All non SEV-ES systems, especially Intel systems, need to execute
+ * verify_cpu() above to make sure NX is enabled.
+ */
+SYM_INNER_LABEL(secondary_startup_64_no_verify, SYM_L_GLOBAL)
+ UNWIND_HINT_EMPTY
+
/*
* Retrieve the modifier (SME encryption mask if SME is active) to be
* added to the initial pgdir entry that will be programmed into CR3.
diff --git a/arch/x86/realmode/init.c b/arch/x86/realmode/init.c
index 3fb9b60be07a..22fda7d99159 100644
--- a/arch/x86/realmode/init.c
+++ b/arch/x86/realmode/init.c
@@ -46,6 +46,12 @@ static void sme_sev_setup_real_mode(struct trampoline_header *th)
th->flags |= TH_FLAGS_SME_ACTIVE;
if (sev_es_active()) {
+ /*
+ * Skip the call to verify_cpu() in secondary_startup_64 as it
+ * will cause #VC exceptions when the AP can't handle them yet.
+ */
+ th->start = (u64) secondary_startup_64_no_verify;
+
if (sev_es_setup_ap_jump_table(real_mode_header))
panic("Failed to get/update SEV-ES AP Jump Table");
}
--
2.28.0
^ permalink raw reply related [flat|nested] 264+ messages in thread
* [PATCH v7 68/72] x86/head/64: Don't call verify_cpu() on starting APs
@ 2020-09-07 13:16 ` Joerg Roedel
0 siblings, 0 replies; 264+ messages in thread
From: Joerg Roedel @ 2020-09-07 13:16 UTC (permalink / raw)
To: x86
Cc: Juergen Gross, Tom Lendacky, Joerg Roedel, Mike Stunes,
Kees Cook, kvm, Peter Zijlstra, Cfir Cohen, Joerg Roedel,
Dave Hansen, linux-kernel, Sean Christopherson, virtualization,
Martin Radev, Masami Hiramatsu, Andy Lutomirski, hpa,
Erdem Aktas, David Rientjes, Dan Williams, Jiri Slaby
From: Joerg Roedel <jroedel@suse.de>
The APs are not ready to handle exceptions when verify_cpu() is called
in secondary_startup_64.
Signed-off-by: Joerg Roedel <jroedel@suse.de>
Reviewed-by: Kees Cook <keescook@chromium.org>
---
arch/x86/include/asm/realmode.h | 1 +
arch/x86/kernel/head_64.S | 12 ++++++++++++
arch/x86/realmode/init.c | 6 ++++++
3 files changed, 19 insertions(+)
diff --git a/arch/x86/include/asm/realmode.h b/arch/x86/include/asm/realmode.h
index 4d4d853f6841..5db5d083c873 100644
--- a/arch/x86/include/asm/realmode.h
+++ b/arch/x86/include/asm/realmode.h
@@ -72,6 +72,7 @@ extern unsigned char startup_32_smp[];
extern unsigned char boot_gdt[];
#else
extern unsigned char secondary_startup_64[];
+extern unsigned char secondary_startup_64_no_verify[];
#endif
static inline size_t real_mode_size_needed(void)
diff --git a/arch/x86/kernel/head_64.S b/arch/x86/kernel/head_64.S
index 41057ff79284..d976a9e6fcba 100644
--- a/arch/x86/kernel/head_64.S
+++ b/arch/x86/kernel/head_64.S
@@ -125,6 +125,18 @@ SYM_CODE_START(secondary_startup_64)
/* Sanitize CPU configuration */
call verify_cpu
+ /*
+ * The secondary_startup_64_no_verify entry point is only used by
+ * SEV-ES guests. In those guests the call to verify_cpu() would cause
+ * #VC exceptions which can not be handled at this stage of secondary
+ * CPU bringup.
+ *
+ * All non SEV-ES systems, especially Intel systems, need to execute
+ * verify_cpu() above to make sure NX is enabled.
+ */
+SYM_INNER_LABEL(secondary_startup_64_no_verify, SYM_L_GLOBAL)
+ UNWIND_HINT_EMPTY
+
/*
* Retrieve the modifier (SME encryption mask if SME is active) to be
* added to the initial pgdir entry that will be programmed into CR3.
diff --git a/arch/x86/realmode/init.c b/arch/x86/realmode/init.c
index 3fb9b60be07a..22fda7d99159 100644
--- a/arch/x86/realmode/init.c
+++ b/arch/x86/realmode/init.c
@@ -46,6 +46,12 @@ static void sme_sev_setup_real_mode(struct trampoline_header *th)
th->flags |= TH_FLAGS_SME_ACTIVE;
if (sev_es_active()) {
+ /*
+ * Skip the call to verify_cpu() in secondary_startup_64 as it
+ * will cause #VC exceptions when the AP can't handle them yet.
+ */
+ th->start = (u64) secondary_startup_64_no_verify;
+
if (sev_es_setup_ap_jump_table(real_mode_header))
panic("Failed to get/update SEV-ES AP Jump Table");
}
--
2.28.0
_______________________________________________
Virtualization mailing list
Virtualization@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/virtualization
^ permalink raw reply related [flat|nested] 264+ messages in thread
* [PATCH v7 69/72] x86/sev-es: Support CPU offline/online
2020-09-07 13:15 ` Joerg Roedel
@ 2020-09-07 13:16 ` Joerg Roedel
-1 siblings, 0 replies; 264+ messages in thread
From: Joerg Roedel @ 2020-09-07 13:16 UTC (permalink / raw)
To: x86
Cc: Joerg Roedel, Joerg Roedel, hpa, Andy Lutomirski, Dave Hansen,
Peter Zijlstra, Jiri Slaby, Dan Williams, Tom Lendacky,
Juergen Gross, Kees Cook, David Rientjes, Cfir Cohen,
Erdem Aktas, Masami Hiramatsu, Mike Stunes, Sean Christopherson,
Martin Radev, linux-kernel, kvm, virtualization
From: Joerg Roedel <jroedel@suse.de>
Add a play_dead handler when running under SEV-ES. This is needed
because the hypervisor can't deliver an SIPI request to restart the AP.
Instead the kernel has to issue a VMGEXIT to halt the VCPU until the
hypervisor wakes it up again.
Signed-off-by: Joerg Roedel <jroedel@suse.de>
---
arch/x86/include/uapi/asm/svm.h | 1 +
arch/x86/kernel/sev-es.c | 63 +++++++++++++++++++++++++++++++++
2 files changed, 64 insertions(+)
diff --git a/arch/x86/include/uapi/asm/svm.h b/arch/x86/include/uapi/asm/svm.h
index 346b8a7155e8..c1dcf3e114e4 100644
--- a/arch/x86/include/uapi/asm/svm.h
+++ b/arch/x86/include/uapi/asm/svm.h
@@ -84,6 +84,7 @@
/* SEV-ES software-defined VMGEXIT events */
#define SVM_VMGEXIT_MMIO_READ 0x80000001
#define SVM_VMGEXIT_MMIO_WRITE 0x80000002
+#define SVM_VMGEXIT_AP_HLT_LOOP 0x80000004
#define SVM_VMGEXIT_AP_JUMP_TABLE 0x80000005
#define SVM_VMGEXIT_SET_AP_JUMP_TABLE 0
#define SVM_VMGEXIT_GET_AP_JUMP_TABLE 1
diff --git a/arch/x86/kernel/sev-es.c b/arch/x86/kernel/sev-es.c
index 91e32172d355..0b6c4f46ddfc 100644
--- a/arch/x86/kernel/sev-es.c
+++ b/arch/x86/kernel/sev-es.c
@@ -29,6 +29,8 @@
#include <asm/realmode.h>
#include <asm/traps.h>
#include <asm/svm.h>
+#include <asm/smp.h>
+#include <asm/cpu.h>
#define DR7_RESET_VALUE 0x400
@@ -518,6 +520,65 @@ static bool __init sev_es_setup_ghcb(void)
return true;
}
+#ifdef CONFIG_HOTPLUG_CPU
+static void sev_es_ap_hlt_loop(void)
+{
+ struct ghcb_state state;
+ struct ghcb *ghcb;
+
+ ghcb = sev_es_get_ghcb(&state);
+
+ while (true) {
+ vc_ghcb_invalidate(ghcb);
+ ghcb_set_sw_exit_code(ghcb, SVM_VMGEXIT_AP_HLT_LOOP);
+ ghcb_set_sw_exit_info_1(ghcb, 0);
+ ghcb_set_sw_exit_info_2(ghcb, 0);
+
+ sev_es_wr_ghcb_msr(__pa(ghcb));
+ VMGEXIT();
+
+ /* Wakeup signal? */
+ if (ghcb_sw_exit_info_2_is_valid(ghcb) &&
+ ghcb->save.sw_exit_info_2)
+ break;
+ }
+
+ sev_es_put_ghcb(&state);
+}
+
+/*
+ * Play_dead handler when running under SEV-ES. This is needed because
+ * the hypervisor can't deliver an SIPI request to restart the AP.
+ * Instead the kernel has to issue a VMGEXIT to halt the VCPU until the
+ * hypervisor wakes it up again.
+ */
+static void sev_es_play_dead(void)
+{
+ play_dead_common();
+
+ /* IRQs now disabled */
+
+ sev_es_ap_hlt_loop();
+
+ /*
+ * If we get here, the VCPU was woken up again. Jump to CPU
+ * startup code to get it back online.
+ */
+ start_cpu0();
+}
+#else /* CONFIG_HOTPLUG_CPU */
+#define sev_es_play_dead native_play_dead
+#endif /* CONFIG_HOTPLUG_CPU */
+
+#ifdef CONFIG_SMP
+static void __init sev_es_setup_play_dead(void)
+{
+ smp_ops.play_dead = sev_es_play_dead;
+}
+#else
+static inline void sev_es_setup_play_dead(void) { }
+#endif
+
static void __init alloc_runtime_data(int cpu)
{
struct sev_es_runtime_data *data;
@@ -566,6 +627,8 @@ void __init sev_es_init_vc_handling(void)
setup_vc_stacks(cpu);
}
+ sev_es_setup_play_dead();
+
/* Secondary CPUs use the runtime #VC handler */
initial_vc_handler = (unsigned long)safe_stack_exc_vmm_communication;
}
--
2.28.0
^ permalink raw reply related [flat|nested] 264+ messages in thread
* [PATCH v7 69/72] x86/sev-es: Support CPU offline/online
@ 2020-09-07 13:16 ` Joerg Roedel
0 siblings, 0 replies; 264+ messages in thread
From: Joerg Roedel @ 2020-09-07 13:16 UTC (permalink / raw)
To: x86
Cc: Juergen Gross, Tom Lendacky, Joerg Roedel, Mike Stunes,
Kees Cook, kvm, Peter Zijlstra, Cfir Cohen, Joerg Roedel,
Dave Hansen, linux-kernel, Sean Christopherson, virtualization,
Martin Radev, Masami Hiramatsu, Andy Lutomirski, hpa,
Erdem Aktas, David Rientjes, Dan Williams, Jiri Slaby
From: Joerg Roedel <jroedel@suse.de>
Add a play_dead handler when running under SEV-ES. This is needed
because the hypervisor can't deliver an SIPI request to restart the AP.
Instead the kernel has to issue a VMGEXIT to halt the VCPU until the
hypervisor wakes it up again.
Signed-off-by: Joerg Roedel <jroedel@suse.de>
---
arch/x86/include/uapi/asm/svm.h | 1 +
arch/x86/kernel/sev-es.c | 63 +++++++++++++++++++++++++++++++++
2 files changed, 64 insertions(+)
diff --git a/arch/x86/include/uapi/asm/svm.h b/arch/x86/include/uapi/asm/svm.h
index 346b8a7155e8..c1dcf3e114e4 100644
--- a/arch/x86/include/uapi/asm/svm.h
+++ b/arch/x86/include/uapi/asm/svm.h
@@ -84,6 +84,7 @@
/* SEV-ES software-defined VMGEXIT events */
#define SVM_VMGEXIT_MMIO_READ 0x80000001
#define SVM_VMGEXIT_MMIO_WRITE 0x80000002
+#define SVM_VMGEXIT_AP_HLT_LOOP 0x80000004
#define SVM_VMGEXIT_AP_JUMP_TABLE 0x80000005
#define SVM_VMGEXIT_SET_AP_JUMP_TABLE 0
#define SVM_VMGEXIT_GET_AP_JUMP_TABLE 1
diff --git a/arch/x86/kernel/sev-es.c b/arch/x86/kernel/sev-es.c
index 91e32172d355..0b6c4f46ddfc 100644
--- a/arch/x86/kernel/sev-es.c
+++ b/arch/x86/kernel/sev-es.c
@@ -29,6 +29,8 @@
#include <asm/realmode.h>
#include <asm/traps.h>
#include <asm/svm.h>
+#include <asm/smp.h>
+#include <asm/cpu.h>
#define DR7_RESET_VALUE 0x400
@@ -518,6 +520,65 @@ static bool __init sev_es_setup_ghcb(void)
return true;
}
+#ifdef CONFIG_HOTPLUG_CPU
+static void sev_es_ap_hlt_loop(void)
+{
+ struct ghcb_state state;
+ struct ghcb *ghcb;
+
+ ghcb = sev_es_get_ghcb(&state);
+
+ while (true) {
+ vc_ghcb_invalidate(ghcb);
+ ghcb_set_sw_exit_code(ghcb, SVM_VMGEXIT_AP_HLT_LOOP);
+ ghcb_set_sw_exit_info_1(ghcb, 0);
+ ghcb_set_sw_exit_info_2(ghcb, 0);
+
+ sev_es_wr_ghcb_msr(__pa(ghcb));
+ VMGEXIT();
+
+ /* Wakeup signal? */
+ if (ghcb_sw_exit_info_2_is_valid(ghcb) &&
+ ghcb->save.sw_exit_info_2)
+ break;
+ }
+
+ sev_es_put_ghcb(&state);
+}
+
+/*
+ * Play_dead handler when running under SEV-ES. This is needed because
+ * the hypervisor can't deliver an SIPI request to restart the AP.
+ * Instead the kernel has to issue a VMGEXIT to halt the VCPU until the
+ * hypervisor wakes it up again.
+ */
+static void sev_es_play_dead(void)
+{
+ play_dead_common();
+
+ /* IRQs now disabled */
+
+ sev_es_ap_hlt_loop();
+
+ /*
+ * If we get here, the VCPU was woken up again. Jump to CPU
+ * startup code to get it back online.
+ */
+ start_cpu0();
+}
+#else /* CONFIG_HOTPLUG_CPU */
+#define sev_es_play_dead native_play_dead
+#endif /* CONFIG_HOTPLUG_CPU */
+
+#ifdef CONFIG_SMP
+static void __init sev_es_setup_play_dead(void)
+{
+ smp_ops.play_dead = sev_es_play_dead;
+}
+#else
+static inline void sev_es_setup_play_dead(void) { }
+#endif
+
static void __init alloc_runtime_data(int cpu)
{
struct sev_es_runtime_data *data;
@@ -566,6 +627,8 @@ void __init sev_es_init_vc_handling(void)
setup_vc_stacks(cpu);
}
+ sev_es_setup_play_dead();
+
/* Secondary CPUs use the runtime #VC handler */
initial_vc_handler = (unsigned long)safe_stack_exc_vmm_communication;
}
--
2.28.0
_______________________________________________
Virtualization mailing list
Virtualization@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/virtualization
^ permalink raw reply related [flat|nested] 264+ messages in thread
* [PATCH v7 70/72] x86/sev-es: Handle NMI State
2020-09-07 13:15 ` Joerg Roedel
@ 2020-09-07 13:16 ` Joerg Roedel
-1 siblings, 0 replies; 264+ messages in thread
From: Joerg Roedel @ 2020-09-07 13:16 UTC (permalink / raw)
To: x86
Cc: Joerg Roedel, Joerg Roedel, hpa, Andy Lutomirski, Dave Hansen,
Peter Zijlstra, Jiri Slaby, Dan Williams, Tom Lendacky,
Juergen Gross, Kees Cook, David Rientjes, Cfir Cohen,
Erdem Aktas, Masami Hiramatsu, Mike Stunes, Sean Christopherson,
Martin Radev, linux-kernel, kvm, virtualization
From: Joerg Roedel <jroedel@suse.de>
When running under SEV-ES the kernel has to tell the hypervisor when to
open the NMI window again after an NMI was injected. This is done with
an NMI-complete message to the hypervisor.
Add code to the kernels NMI handler to send this message right at the
beginning of do_nmi(). This always allows nesting NMIs.
Signed-off-by: Joerg Roedel <jroedel@suse.de>
---
arch/x86/include/asm/sev-es.h | 7 +++++++
arch/x86/include/uapi/asm/svm.h | 1 +
arch/x86/kernel/nmi.c | 6 ++++++
arch/x86/kernel/sev-es.c | 18 ++++++++++++++++++
4 files changed, 32 insertions(+)
diff --git a/arch/x86/include/asm/sev-es.h b/arch/x86/include/asm/sev-es.h
index db88e1c3442d..e919f09ae33c 100644
--- a/arch/x86/include/asm/sev-es.h
+++ b/arch/x86/include/asm/sev-es.h
@@ -96,10 +96,17 @@ static __always_inline void sev_es_ist_exit(void)
__sev_es_ist_exit();
}
extern int sev_es_setup_ap_jump_table(struct real_mode_header *rmh);
+extern void __sev_es_nmi_complete(void);
+static __always_inline void sev_es_nmi_complete(void)
+{
+ if (static_branch_unlikely(&sev_es_enable_key))
+ __sev_es_nmi_complete();
+}
#else
static inline void sev_es_ist_enter(struct pt_regs *regs) { }
static inline void sev_es_ist_exit(void) { }
static inline int sev_es_setup_ap_jump_table(struct real_mode_header *rmh) { return 0; }
+static inline void sev_es_nmi_complete(void) { }
#endif
#endif
diff --git a/arch/x86/include/uapi/asm/svm.h b/arch/x86/include/uapi/asm/svm.h
index c1dcf3e114e4..a7a3403645e5 100644
--- a/arch/x86/include/uapi/asm/svm.h
+++ b/arch/x86/include/uapi/asm/svm.h
@@ -84,6 +84,7 @@
/* SEV-ES software-defined VMGEXIT events */
#define SVM_VMGEXIT_MMIO_READ 0x80000001
#define SVM_VMGEXIT_MMIO_WRITE 0x80000002
+#define SVM_VMGEXIT_NMI_COMPLETE 0x80000003
#define SVM_VMGEXIT_AP_HLT_LOOP 0x80000004
#define SVM_VMGEXIT_AP_JUMP_TABLE 0x80000005
#define SVM_VMGEXIT_SET_AP_JUMP_TABLE 0
diff --git a/arch/x86/kernel/nmi.c b/arch/x86/kernel/nmi.c
index 5859cec774a4..d961f1b3e251 100644
--- a/arch/x86/kernel/nmi.c
+++ b/arch/x86/kernel/nmi.c
@@ -478,6 +478,12 @@ DEFINE_IDTENTRY_RAW(exc_nmi)
{
bool irq_state;
+ /*
+ * Re-enable NMIs right here when running as an SEV-ES guest. This might
+ * cause nested NMIs, but those can be handled safely.
+ */
+ sev_es_nmi_complete();
+
if (IS_ENABLED(CONFIG_SMP) && arch_cpu_is_offline(smp_processor_id()))
return;
diff --git a/arch/x86/kernel/sev-es.c b/arch/x86/kernel/sev-es.c
index 0b6c4f46ddfc..9ab3a4dfecd8 100644
--- a/arch/x86/kernel/sev-es.c
+++ b/arch/x86/kernel/sev-es.c
@@ -408,6 +408,24 @@ static bool vc_slow_virt_to_phys(struct ghcb *ghcb, struct es_em_ctxt *ctxt,
/* Include code shared with pre-decompression boot stage */
#include "sev-es-shared.c"
+void __sev_es_nmi_complete(void)
+{
+ struct ghcb_state state;
+ struct ghcb *ghcb;
+
+ ghcb = sev_es_get_ghcb(&state);
+
+ vc_ghcb_invalidate(ghcb);
+ ghcb_set_sw_exit_code(ghcb, SVM_VMGEXIT_NMI_COMPLETE);
+ ghcb_set_sw_exit_info_1(ghcb, 0);
+ ghcb_set_sw_exit_info_2(ghcb, 0);
+
+ sev_es_wr_ghcb_msr(__pa(ghcb));
+ VMGEXIT();
+
+ sev_es_put_ghcb(&state);
+}
+
static u64 get_jump_table_addr(void)
{
struct ghcb_state state;
--
2.28.0
^ permalink raw reply related [flat|nested] 264+ messages in thread
* [PATCH v7 70/72] x86/sev-es: Handle NMI State
@ 2020-09-07 13:16 ` Joerg Roedel
0 siblings, 0 replies; 264+ messages in thread
From: Joerg Roedel @ 2020-09-07 13:16 UTC (permalink / raw)
To: x86
Cc: Juergen Gross, Tom Lendacky, Joerg Roedel, Mike Stunes,
Kees Cook, kvm, Peter Zijlstra, Cfir Cohen, Joerg Roedel,
Dave Hansen, linux-kernel, Sean Christopherson, virtualization,
Martin Radev, Masami Hiramatsu, Andy Lutomirski, hpa,
Erdem Aktas, David Rientjes, Dan Williams, Jiri Slaby
From: Joerg Roedel <jroedel@suse.de>
When running under SEV-ES the kernel has to tell the hypervisor when to
open the NMI window again after an NMI was injected. This is done with
an NMI-complete message to the hypervisor.
Add code to the kernels NMI handler to send this message right at the
beginning of do_nmi(). This always allows nesting NMIs.
Signed-off-by: Joerg Roedel <jroedel@suse.de>
---
arch/x86/include/asm/sev-es.h | 7 +++++++
arch/x86/include/uapi/asm/svm.h | 1 +
arch/x86/kernel/nmi.c | 6 ++++++
arch/x86/kernel/sev-es.c | 18 ++++++++++++++++++
4 files changed, 32 insertions(+)
diff --git a/arch/x86/include/asm/sev-es.h b/arch/x86/include/asm/sev-es.h
index db88e1c3442d..e919f09ae33c 100644
--- a/arch/x86/include/asm/sev-es.h
+++ b/arch/x86/include/asm/sev-es.h
@@ -96,10 +96,17 @@ static __always_inline void sev_es_ist_exit(void)
__sev_es_ist_exit();
}
extern int sev_es_setup_ap_jump_table(struct real_mode_header *rmh);
+extern void __sev_es_nmi_complete(void);
+static __always_inline void sev_es_nmi_complete(void)
+{
+ if (static_branch_unlikely(&sev_es_enable_key))
+ __sev_es_nmi_complete();
+}
#else
static inline void sev_es_ist_enter(struct pt_regs *regs) { }
static inline void sev_es_ist_exit(void) { }
static inline int sev_es_setup_ap_jump_table(struct real_mode_header *rmh) { return 0; }
+static inline void sev_es_nmi_complete(void) { }
#endif
#endif
diff --git a/arch/x86/include/uapi/asm/svm.h b/arch/x86/include/uapi/asm/svm.h
index c1dcf3e114e4..a7a3403645e5 100644
--- a/arch/x86/include/uapi/asm/svm.h
+++ b/arch/x86/include/uapi/asm/svm.h
@@ -84,6 +84,7 @@
/* SEV-ES software-defined VMGEXIT events */
#define SVM_VMGEXIT_MMIO_READ 0x80000001
#define SVM_VMGEXIT_MMIO_WRITE 0x80000002
+#define SVM_VMGEXIT_NMI_COMPLETE 0x80000003
#define SVM_VMGEXIT_AP_HLT_LOOP 0x80000004
#define SVM_VMGEXIT_AP_JUMP_TABLE 0x80000005
#define SVM_VMGEXIT_SET_AP_JUMP_TABLE 0
diff --git a/arch/x86/kernel/nmi.c b/arch/x86/kernel/nmi.c
index 5859cec774a4..d961f1b3e251 100644
--- a/arch/x86/kernel/nmi.c
+++ b/arch/x86/kernel/nmi.c
@@ -478,6 +478,12 @@ DEFINE_IDTENTRY_RAW(exc_nmi)
{
bool irq_state;
+ /*
+ * Re-enable NMIs right here when running as an SEV-ES guest. This might
+ * cause nested NMIs, but those can be handled safely.
+ */
+ sev_es_nmi_complete();
+
if (IS_ENABLED(CONFIG_SMP) && arch_cpu_is_offline(smp_processor_id()))
return;
diff --git a/arch/x86/kernel/sev-es.c b/arch/x86/kernel/sev-es.c
index 0b6c4f46ddfc..9ab3a4dfecd8 100644
--- a/arch/x86/kernel/sev-es.c
+++ b/arch/x86/kernel/sev-es.c
@@ -408,6 +408,24 @@ static bool vc_slow_virt_to_phys(struct ghcb *ghcb, struct es_em_ctxt *ctxt,
/* Include code shared with pre-decompression boot stage */
#include "sev-es-shared.c"
+void __sev_es_nmi_complete(void)
+{
+ struct ghcb_state state;
+ struct ghcb *ghcb;
+
+ ghcb = sev_es_get_ghcb(&state);
+
+ vc_ghcb_invalidate(ghcb);
+ ghcb_set_sw_exit_code(ghcb, SVM_VMGEXIT_NMI_COMPLETE);
+ ghcb_set_sw_exit_info_1(ghcb, 0);
+ ghcb_set_sw_exit_info_2(ghcb, 0);
+
+ sev_es_wr_ghcb_msr(__pa(ghcb));
+ VMGEXIT();
+
+ sev_es_put_ghcb(&state);
+}
+
static u64 get_jump_table_addr(void)
{
struct ghcb_state state;
--
2.28.0
_______________________________________________
Virtualization mailing list
Virtualization@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/virtualization
^ permalink raw reply related [flat|nested] 264+ messages in thread
* [PATCH v7 71/72] x86/efi: Add GHCB mappings when SEV-ES is active
2020-09-07 13:15 ` Joerg Roedel
@ 2020-09-07 13:16 ` Joerg Roedel
-1 siblings, 0 replies; 264+ messages in thread
From: Joerg Roedel @ 2020-09-07 13:16 UTC (permalink / raw)
To: x86
Cc: Joerg Roedel, Joerg Roedel, Tom Lendacky, hpa, Andy Lutomirski,
Dave Hansen, Peter Zijlstra, Jiri Slaby, Dan Williams,
Juergen Gross, Kees Cook, David Rientjes, Cfir Cohen,
Erdem Aktas, Masami Hiramatsu, Mike Stunes, Sean Christopherson,
Martin Radev, linux-kernel, kvm, virtualization
From: Tom Lendacky <thomas.lendacky@amd.com>
Calling down to EFI runtime services can result in the firmware performing
VMGEXIT calls. The firmware is likely to use the GHCB of the OS (e.g., for
setting EFI variables), so each GHCB in the system needs to be identity
mapped in the EFI page tables, as unencrypted, to avoid page faults.
Signed-off-by: Tom Lendacky <thomas.lendacky@amd.com>
[ jroedel@suse.de: Moved GHCB mapping loop to sev-es.c ]
Signed-off-by: Joerg Roedel <jroedel@suse.de>
---
arch/x86/boot/compressed/sev-es.c | 1 +
arch/x86/include/asm/sev-es.h | 2 ++
arch/x86/kernel/sev-es.c | 30 ++++++++++++++++++++++++++++++
arch/x86/platform/efi/efi_64.c | 10 ++++++++++
4 files changed, 43 insertions(+)
diff --git a/arch/x86/boot/compressed/sev-es.c b/arch/x86/boot/compressed/sev-es.c
index 45702b866c33..0a9a248ca33d 100644
--- a/arch/x86/boot/compressed/sev-es.c
+++ b/arch/x86/boot/compressed/sev-es.c
@@ -12,6 +12,7 @@
*/
#include "misc.h"
+#include <asm/pgtable_types.h>
#include <asm/sev-es.h>
#include <asm/trapnr.h>
#include <asm/trap_pf.h>
diff --git a/arch/x86/include/asm/sev-es.h b/arch/x86/include/asm/sev-es.h
index e919f09ae33c..cf1d957c7091 100644
--- a/arch/x86/include/asm/sev-es.h
+++ b/arch/x86/include/asm/sev-es.h
@@ -102,11 +102,13 @@ static __always_inline void sev_es_nmi_complete(void)
if (static_branch_unlikely(&sev_es_enable_key))
__sev_es_nmi_complete();
}
+extern int __init sev_es_efi_map_ghcbs(pgd_t *pgd);
#else
static inline void sev_es_ist_enter(struct pt_regs *regs) { }
static inline void sev_es_ist_exit(void) { }
static inline int sev_es_setup_ap_jump_table(struct real_mode_header *rmh) { return 0; }
static inline void sev_es_nmi_complete(void) { }
+static inline int sev_es_efi_map_ghcbs(pgd_t *pgd) { return 0; }
#endif
#endif
diff --git a/arch/x86/kernel/sev-es.c b/arch/x86/kernel/sev-es.c
index 9ab3a4dfecd8..4e2b7e4d9b87 100644
--- a/arch/x86/kernel/sev-es.c
+++ b/arch/x86/kernel/sev-es.c
@@ -491,6 +491,36 @@ int sev_es_setup_ap_jump_table(struct real_mode_header *rmh)
return 0;
}
+/*
+ * This is needed by the OVMF UEFI firmware which will use whatever it finds in
+ * the GHCB MSR as its GHCB to talk to the hypervisor. So make sure the per-cpu
+ * runtime GHCBs used by the kernel are also mapped in the EFI page-table.
+ */
+int __init sev_es_efi_map_ghcbs(pgd_t *pgd)
+{
+ struct sev_es_runtime_data *data;
+ unsigned long address, pflags;
+ int cpu;
+ u64 pfn;
+
+ if (!sev_es_active())
+ return 0;
+
+ pflags = _PAGE_NX | _PAGE_RW;
+
+ for_each_possible_cpu(cpu) {
+ data = per_cpu(runtime_data, cpu);
+
+ address = __pa(&data->ghcb_page);
+ pfn = address >> PAGE_SHIFT;
+
+ if (kernel_map_pages_in_pgd(pgd, pfn, address, 1, pflags))
+ return 1;
+ }
+
+ return 0;
+}
+
static enum es_result vc_handle_msr(struct ghcb *ghcb, struct es_em_ctxt *ctxt)
{
struct pt_regs *regs = ctxt->regs;
diff --git a/arch/x86/platform/efi/efi_64.c b/arch/x86/platform/efi/efi_64.c
index 6af4da1149ba..8f5759df7776 100644
--- a/arch/x86/platform/efi/efi_64.c
+++ b/arch/x86/platform/efi/efi_64.c
@@ -47,6 +47,7 @@
#include <asm/realmode.h>
#include <asm/time.h>
#include <asm/pgalloc.h>
+#include <asm/sev-es.h>
/*
* We allocate runtime services regions top-down, starting from -4G, i.e.
@@ -229,6 +230,15 @@ int __init efi_setup_page_tables(unsigned long pa_memmap, unsigned num_pages)
return 1;
}
+ /*
+ * When SEV-ES is active, the GHCB as set by the kernel will be used
+ * by firmware. Create a 1:1 unencrypted mapping for each GHCB.
+ */
+ if (sev_es_efi_map_ghcbs(pgd)) {
+ pr_err("Failed to create 1:1 mapping for the GHCBs!\n");
+ return 1;
+ }
+
/*
* When making calls to the firmware everything needs to be 1:1
* mapped and addressable with 32-bit pointers. Map the kernel
--
2.28.0
^ permalink raw reply related [flat|nested] 264+ messages in thread
* [PATCH v7 71/72] x86/efi: Add GHCB mappings when SEV-ES is active
@ 2020-09-07 13:16 ` Joerg Roedel
0 siblings, 0 replies; 264+ messages in thread
From: Joerg Roedel @ 2020-09-07 13:16 UTC (permalink / raw)
To: x86
Cc: Juergen Gross, Tom Lendacky, Joerg Roedel, Mike Stunes,
Kees Cook, kvm, Peter Zijlstra, Cfir Cohen, Joerg Roedel,
Dave Hansen, linux-kernel, Sean Christopherson, virtualization,
Martin Radev, Masami Hiramatsu, Andy Lutomirski, hpa,
Erdem Aktas, David Rientjes, Dan Williams, Jiri Slaby
From: Tom Lendacky <thomas.lendacky@amd.com>
Calling down to EFI runtime services can result in the firmware performing
VMGEXIT calls. The firmware is likely to use the GHCB of the OS (e.g., for
setting EFI variables), so each GHCB in the system needs to be identity
mapped in the EFI page tables, as unencrypted, to avoid page faults.
Signed-off-by: Tom Lendacky <thomas.lendacky@amd.com>
[ jroedel@suse.de: Moved GHCB mapping loop to sev-es.c ]
Signed-off-by: Joerg Roedel <jroedel@suse.de>
---
arch/x86/boot/compressed/sev-es.c | 1 +
arch/x86/include/asm/sev-es.h | 2 ++
arch/x86/kernel/sev-es.c | 30 ++++++++++++++++++++++++++++++
arch/x86/platform/efi/efi_64.c | 10 ++++++++++
4 files changed, 43 insertions(+)
diff --git a/arch/x86/boot/compressed/sev-es.c b/arch/x86/boot/compressed/sev-es.c
index 45702b866c33..0a9a248ca33d 100644
--- a/arch/x86/boot/compressed/sev-es.c
+++ b/arch/x86/boot/compressed/sev-es.c
@@ -12,6 +12,7 @@
*/
#include "misc.h"
+#include <asm/pgtable_types.h>
#include <asm/sev-es.h>
#include <asm/trapnr.h>
#include <asm/trap_pf.h>
diff --git a/arch/x86/include/asm/sev-es.h b/arch/x86/include/asm/sev-es.h
index e919f09ae33c..cf1d957c7091 100644
--- a/arch/x86/include/asm/sev-es.h
+++ b/arch/x86/include/asm/sev-es.h
@@ -102,11 +102,13 @@ static __always_inline void sev_es_nmi_complete(void)
if (static_branch_unlikely(&sev_es_enable_key))
__sev_es_nmi_complete();
}
+extern int __init sev_es_efi_map_ghcbs(pgd_t *pgd);
#else
static inline void sev_es_ist_enter(struct pt_regs *regs) { }
static inline void sev_es_ist_exit(void) { }
static inline int sev_es_setup_ap_jump_table(struct real_mode_header *rmh) { return 0; }
static inline void sev_es_nmi_complete(void) { }
+static inline int sev_es_efi_map_ghcbs(pgd_t *pgd) { return 0; }
#endif
#endif
diff --git a/arch/x86/kernel/sev-es.c b/arch/x86/kernel/sev-es.c
index 9ab3a4dfecd8..4e2b7e4d9b87 100644
--- a/arch/x86/kernel/sev-es.c
+++ b/arch/x86/kernel/sev-es.c
@@ -491,6 +491,36 @@ int sev_es_setup_ap_jump_table(struct real_mode_header *rmh)
return 0;
}
+/*
+ * This is needed by the OVMF UEFI firmware which will use whatever it finds in
+ * the GHCB MSR as its GHCB to talk to the hypervisor. So make sure the per-cpu
+ * runtime GHCBs used by the kernel are also mapped in the EFI page-table.
+ */
+int __init sev_es_efi_map_ghcbs(pgd_t *pgd)
+{
+ struct sev_es_runtime_data *data;
+ unsigned long address, pflags;
+ int cpu;
+ u64 pfn;
+
+ if (!sev_es_active())
+ return 0;
+
+ pflags = _PAGE_NX | _PAGE_RW;
+
+ for_each_possible_cpu(cpu) {
+ data = per_cpu(runtime_data, cpu);
+
+ address = __pa(&data->ghcb_page);
+ pfn = address >> PAGE_SHIFT;
+
+ if (kernel_map_pages_in_pgd(pgd, pfn, address, 1, pflags))
+ return 1;
+ }
+
+ return 0;
+}
+
static enum es_result vc_handle_msr(struct ghcb *ghcb, struct es_em_ctxt *ctxt)
{
struct pt_regs *regs = ctxt->regs;
diff --git a/arch/x86/platform/efi/efi_64.c b/arch/x86/platform/efi/efi_64.c
index 6af4da1149ba..8f5759df7776 100644
--- a/arch/x86/platform/efi/efi_64.c
+++ b/arch/x86/platform/efi/efi_64.c
@@ -47,6 +47,7 @@
#include <asm/realmode.h>
#include <asm/time.h>
#include <asm/pgalloc.h>
+#include <asm/sev-es.h>
/*
* We allocate runtime services regions top-down, starting from -4G, i.e.
@@ -229,6 +230,15 @@ int __init efi_setup_page_tables(unsigned long pa_memmap, unsigned num_pages)
return 1;
}
+ /*
+ * When SEV-ES is active, the GHCB as set by the kernel will be used
+ * by firmware. Create a 1:1 unencrypted mapping for each GHCB.
+ */
+ if (sev_es_efi_map_ghcbs(pgd)) {
+ pr_err("Failed to create 1:1 mapping for the GHCBs!\n");
+ return 1;
+ }
+
/*
* When making calls to the firmware everything needs to be 1:1
* mapped and addressable with 32-bit pointers. Map the kernel
--
2.28.0
_______________________________________________
Virtualization mailing list
Virtualization@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/virtualization
^ permalink raw reply related [flat|nested] 264+ messages in thread
* [PATCH v7 72/72] x86/sev-es: Check required CPU features for SEV-ES
2020-09-07 13:15 ` Joerg Roedel
@ 2020-09-07 13:16 ` Joerg Roedel
-1 siblings, 0 replies; 264+ messages in thread
From: Joerg Roedel @ 2020-09-07 13:16 UTC (permalink / raw)
To: x86
Cc: Joerg Roedel, Joerg Roedel, Martin Radev, Kees Cook, hpa,
Andy Lutomirski, Dave Hansen, Peter Zijlstra, Jiri Slaby,
Dan Williams, Tom Lendacky, Juergen Gross, David Rientjes,
Cfir Cohen, Erdem Aktas, Masami Hiramatsu, Mike Stunes,
Sean Christopherson, linux-kernel, kvm, virtualization
From: Martin Radev <martin.b.radev@gmail.com>
Make sure the machine supports RDRAND, otherwise there is no trusted
source of of randomness in the system.
To also check this in the pre-decompression stage, make has_cpuflag
not depend on CONFIG_RANDOMIZE_BASE anymore.
Signed-off-by: Martin Radev <martin.b.radev@gmail.com>
Signed-off-by: Joerg Roedel <jroedel@suse.de>
Reviewed-by: Kees Cook <keescook@chromium.org>
---
arch/x86/boot/compressed/cpuflags.c | 4 ----
arch/x86/boot/compressed/misc.h | 5 +++--
arch/x86/boot/compressed/sev-es.c | 3 +++
arch/x86/kernel/sev-es-shared.c | 15 +++++++++++++++
arch/x86/kernel/sev-es.c | 3 +++
5 files changed, 24 insertions(+), 6 deletions(-)
diff --git a/arch/x86/boot/compressed/cpuflags.c b/arch/x86/boot/compressed/cpuflags.c
index 6448a8196d32..0cc1323896d1 100644
--- a/arch/x86/boot/compressed/cpuflags.c
+++ b/arch/x86/boot/compressed/cpuflags.c
@@ -1,6 +1,4 @@
// SPDX-License-Identifier: GPL-2.0
-#ifdef CONFIG_RANDOMIZE_BASE
-
#include "../cpuflags.c"
bool has_cpuflag(int flag)
@@ -9,5 +7,3 @@ bool has_cpuflag(int flag)
return test_bit(flag, cpu.flags);
}
-
-#endif
diff --git a/arch/x86/boot/compressed/misc.h b/arch/x86/boot/compressed/misc.h
index c0e0ffeee50a..6d31f1b4c4d1 100644
--- a/arch/x86/boot/compressed/misc.h
+++ b/arch/x86/boot/compressed/misc.h
@@ -85,8 +85,6 @@ void choose_random_location(unsigned long input,
unsigned long *output,
unsigned long output_size,
unsigned long *virt_addr);
-/* cpuflags.c */
-bool has_cpuflag(int flag);
#else
static inline void choose_random_location(unsigned long input,
unsigned long input_size,
@@ -97,6 +95,9 @@ static inline void choose_random_location(unsigned long input,
}
#endif
+/* cpuflags.c */
+bool has_cpuflag(int flag);
+
#ifdef CONFIG_X86_64
extern int set_page_decrypted(unsigned long address);
extern int set_page_encrypted(unsigned long address);
diff --git a/arch/x86/boot/compressed/sev-es.c b/arch/x86/boot/compressed/sev-es.c
index 0a9a248ca33d..3c66dad4e62e 100644
--- a/arch/x86/boot/compressed/sev-es.c
+++ b/arch/x86/boot/compressed/sev-es.c
@@ -145,6 +145,9 @@ void sev_es_shutdown_ghcb(void)
if (!boot_ghcb)
return;
+ if (!sev_es_check_cpu_features())
+ error("SEV-ES CPU Features missing.");
+
/*
* GHCB Page must be flushed from the cache and mapped encrypted again.
* Otherwise the running kernel will see strange cache effects when
diff --git a/arch/x86/kernel/sev-es-shared.c b/arch/x86/kernel/sev-es-shared.c
index 92d77b725ccb..ce86d2c9ca7b 100644
--- a/arch/x86/kernel/sev-es-shared.c
+++ b/arch/x86/kernel/sev-es-shared.c
@@ -9,6 +9,21 @@
* and is included directly into both code-bases.
*/
+#ifndef __BOOT_COMPRESSED
+#define error(v) pr_err(v)
+#define has_cpuflag(f) boot_cpu_has(f)
+#endif
+
+static bool __init sev_es_check_cpu_features(void)
+{
+ if (!has_cpuflag(X86_FEATURE_RDRAND)) {
+ error("RDRAND instruction not supported - no trusted source of randomness available\n");
+ return false;
+ }
+
+ return true;
+}
+
static void sev_es_terminate(unsigned int reason)
{
u64 val = GHCB_SEV_TERMINATE;
diff --git a/arch/x86/kernel/sev-es.c b/arch/x86/kernel/sev-es.c
index 4e2b7e4d9b87..70623bbce062 100644
--- a/arch/x86/kernel/sev-es.c
+++ b/arch/x86/kernel/sev-es.c
@@ -665,6 +665,9 @@ void __init sev_es_init_vc_handling(void)
if (!sev_es_active())
return;
+ if (!sev_es_check_cpu_features())
+ panic("SEV-ES CPU Features missing");
+
/* Enable SEV-ES special handling */
static_branch_enable(&sev_es_enable_key);
--
2.28.0
^ permalink raw reply related [flat|nested] 264+ messages in thread
* [PATCH v7 72/72] x86/sev-es: Check required CPU features for SEV-ES
@ 2020-09-07 13:16 ` Joerg Roedel
0 siblings, 0 replies; 264+ messages in thread
From: Joerg Roedel @ 2020-09-07 13:16 UTC (permalink / raw)
To: x86
Cc: Martin Radev, Tom Lendacky, Joerg Roedel, Mike Stunes, Kees Cook,
kvm, Peter Zijlstra, Cfir Cohen, Joerg Roedel, Dave Hansen,
Juergen Gross, Sean Christopherson, linux-kernel,
Masami Hiramatsu, Andy Lutomirski, hpa, Erdem Aktas,
David Rientjes, Dan Williams, virtualization, Jiri Slaby
From: Martin Radev <martin.b.radev@gmail.com>
Make sure the machine supports RDRAND, otherwise there is no trusted
source of of randomness in the system.
To also check this in the pre-decompression stage, make has_cpuflag
not depend on CONFIG_RANDOMIZE_BASE anymore.
Signed-off-by: Martin Radev <martin.b.radev@gmail.com>
Signed-off-by: Joerg Roedel <jroedel@suse.de>
Reviewed-by: Kees Cook <keescook@chromium.org>
---
arch/x86/boot/compressed/cpuflags.c | 4 ----
arch/x86/boot/compressed/misc.h | 5 +++--
arch/x86/boot/compressed/sev-es.c | 3 +++
arch/x86/kernel/sev-es-shared.c | 15 +++++++++++++++
arch/x86/kernel/sev-es.c | 3 +++
5 files changed, 24 insertions(+), 6 deletions(-)
diff --git a/arch/x86/boot/compressed/cpuflags.c b/arch/x86/boot/compressed/cpuflags.c
index 6448a8196d32..0cc1323896d1 100644
--- a/arch/x86/boot/compressed/cpuflags.c
+++ b/arch/x86/boot/compressed/cpuflags.c
@@ -1,6 +1,4 @@
// SPDX-License-Identifier: GPL-2.0
-#ifdef CONFIG_RANDOMIZE_BASE
-
#include "../cpuflags.c"
bool has_cpuflag(int flag)
@@ -9,5 +7,3 @@ bool has_cpuflag(int flag)
return test_bit(flag, cpu.flags);
}
-
-#endif
diff --git a/arch/x86/boot/compressed/misc.h b/arch/x86/boot/compressed/misc.h
index c0e0ffeee50a..6d31f1b4c4d1 100644
--- a/arch/x86/boot/compressed/misc.h
+++ b/arch/x86/boot/compressed/misc.h
@@ -85,8 +85,6 @@ void choose_random_location(unsigned long input,
unsigned long *output,
unsigned long output_size,
unsigned long *virt_addr);
-/* cpuflags.c */
-bool has_cpuflag(int flag);
#else
static inline void choose_random_location(unsigned long input,
unsigned long input_size,
@@ -97,6 +95,9 @@ static inline void choose_random_location(unsigned long input,
}
#endif
+/* cpuflags.c */
+bool has_cpuflag(int flag);
+
#ifdef CONFIG_X86_64
extern int set_page_decrypted(unsigned long address);
extern int set_page_encrypted(unsigned long address);
diff --git a/arch/x86/boot/compressed/sev-es.c b/arch/x86/boot/compressed/sev-es.c
index 0a9a248ca33d..3c66dad4e62e 100644
--- a/arch/x86/boot/compressed/sev-es.c
+++ b/arch/x86/boot/compressed/sev-es.c
@@ -145,6 +145,9 @@ void sev_es_shutdown_ghcb(void)
if (!boot_ghcb)
return;
+ if (!sev_es_check_cpu_features())
+ error("SEV-ES CPU Features missing.");
+
/*
* GHCB Page must be flushed from the cache and mapped encrypted again.
* Otherwise the running kernel will see strange cache effects when
diff --git a/arch/x86/kernel/sev-es-shared.c b/arch/x86/kernel/sev-es-shared.c
index 92d77b725ccb..ce86d2c9ca7b 100644
--- a/arch/x86/kernel/sev-es-shared.c
+++ b/arch/x86/kernel/sev-es-shared.c
@@ -9,6 +9,21 @@
* and is included directly into both code-bases.
*/
+#ifndef __BOOT_COMPRESSED
+#define error(v) pr_err(v)
+#define has_cpuflag(f) boot_cpu_has(f)
+#endif
+
+static bool __init sev_es_check_cpu_features(void)
+{
+ if (!has_cpuflag(X86_FEATURE_RDRAND)) {
+ error("RDRAND instruction not supported - no trusted source of randomness available\n");
+ return false;
+ }
+
+ return true;
+}
+
static void sev_es_terminate(unsigned int reason)
{
u64 val = GHCB_SEV_TERMINATE;
diff --git a/arch/x86/kernel/sev-es.c b/arch/x86/kernel/sev-es.c
index 4e2b7e4d9b87..70623bbce062 100644
--- a/arch/x86/kernel/sev-es.c
+++ b/arch/x86/kernel/sev-es.c
@@ -665,6 +665,9 @@ void __init sev_es_init_vc_handling(void)
if (!sev_es_active())
return;
+ if (!sev_es_check_cpu_features())
+ panic("SEV-ES CPU Features missing");
+
/* Enable SEV-ES special handling */
static_branch_enable(&sev_es_enable_key);
--
2.28.0
_______________________________________________
Virtualization mailing list
Virtualization@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/virtualization
^ permalink raw reply related [flat|nested] 264+ messages in thread
* Re: [PATCH v7 19/72] x86/boot/compressed/64: Add stage1 #VC handler
2020-09-07 13:15 ` Joerg Roedel
@ 2020-09-07 16:58 ` Borislav Petkov
-1 siblings, 0 replies; 264+ messages in thread
From: Borislav Petkov @ 2020-09-07 16:58 UTC (permalink / raw)
To: Joerg Roedel
Cc: x86, Joerg Roedel, hpa, Andy Lutomirski, Dave Hansen,
Peter Zijlstra, Jiri Slaby, Dan Williams, Tom Lendacky,
Juergen Gross, Kees Cook, David Rientjes, Cfir Cohen,
Erdem Aktas, Masami Hiramatsu, Mike Stunes, Sean Christopherson,
Martin Radev, linux-kernel, kvm, virtualization
On Mon, Sep 07, 2020 at 03:15:20PM +0200, Joerg Roedel wrote:
> +static inline u64 sev_es_rd_ghcb_msr(void)
> +{
> + unsigned long low, high;
> +
> + asm volatile("rdmsr\n" : "=a" (low), "=d" (high) :
> + "c" (MSR_AMD64_SEV_ES_GHCB));
> +
> + return ((high << 32) | low);
> +}
> +
> +static inline void sev_es_wr_ghcb_msr(u64 val)
> +{
> + u32 low, high;
> +
> + low = val & 0xffffffffUL;
> + high = val >> 32;
> +
> + asm volatile("wrmsr\n" : : "c" (MSR_AMD64_SEV_ES_GHCB),
^^
No need for that newline and the one above. I've zapped it while applying.
--
Regards/Gruss,
Boris.
https://people.kernel.org/tglx/notes-about-netiquette
^ permalink raw reply [flat|nested] 264+ messages in thread
* Re: [PATCH v7 19/72] x86/boot/compressed/64: Add stage1 #VC handler
@ 2020-09-07 16:58 ` Borislav Petkov
0 siblings, 0 replies; 264+ messages in thread
From: Borislav Petkov @ 2020-09-07 16:58 UTC (permalink / raw)
To: Joerg Roedel
Cc: Juergen Gross, Tom Lendacky, Dave Hansen, Mike Stunes, Kees Cook,
kvm, Peter Zijlstra, Cfir Cohen, x86, linux-kernel,
Sean Christopherson, virtualization, Martin Radev, Joerg Roedel,
Masami Hiramatsu, Andy Lutomirski, hpa, Erdem Aktas,
David Rientjes, Dan Williams, Jiri Slaby
On Mon, Sep 07, 2020 at 03:15:20PM +0200, Joerg Roedel wrote:
> +static inline u64 sev_es_rd_ghcb_msr(void)
> +{
> + unsigned long low, high;
> +
> + asm volatile("rdmsr\n" : "=a" (low), "=d" (high) :
> + "c" (MSR_AMD64_SEV_ES_GHCB));
> +
> + return ((high << 32) | low);
> +}
> +
> +static inline void sev_es_wr_ghcb_msr(u64 val)
> +{
> + u32 low, high;
> +
> + low = val & 0xffffffffUL;
> + high = val >> 32;
> +
> + asm volatile("wrmsr\n" : : "c" (MSR_AMD64_SEV_ES_GHCB),
^^
No need for that newline and the one above. I've zapped it while applying.
--
Regards/Gruss,
Boris.
https://people.kernel.org/tglx/notes-about-netiquette
_______________________________________________
Virtualization mailing list
Virtualization@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/virtualization
^ permalink raw reply [flat|nested] 264+ messages in thread
* Re: [PATCH v7 36/72] x86/sev-es: Add SEV-ES Feature Detection
2020-09-07 13:15 ` Joerg Roedel
@ 2020-09-07 21:02 ` Borislav Petkov
-1 siblings, 0 replies; 264+ messages in thread
From: Borislav Petkov @ 2020-09-07 21:02 UTC (permalink / raw)
To: Joerg Roedel
Cc: x86, Joerg Roedel, Kees Cook, hpa, Andy Lutomirski, Dave Hansen,
Peter Zijlstra, Jiri Slaby, Dan Williams, Tom Lendacky,
Juergen Gross, David Rientjes, Cfir Cohen, Erdem Aktas,
Masami Hiramatsu, Mike Stunes, Sean Christopherson, Martin Radev,
linux-kernel, kvm, virtualization
On Mon, Sep 07, 2020 at 03:15:37PM +0200, Joerg Roedel wrote:
> @@ -347,7 +348,13 @@ bool sme_active(void)
>
> bool sev_active(void)
> {
> - return sme_me_mask && sev_enabled;
> + return !!(sev_status & MSR_AMD64_SEV_ENABLED);
Dropped those "!!" here too while applying.
--
Regards/Gruss,
Boris.
https://people.kernel.org/tglx/notes-about-netiquette
^ permalink raw reply [flat|nested] 264+ messages in thread
* Re: [PATCH v7 36/72] x86/sev-es: Add SEV-ES Feature Detection
@ 2020-09-07 21:02 ` Borislav Petkov
0 siblings, 0 replies; 264+ messages in thread
From: Borislav Petkov @ 2020-09-07 21:02 UTC (permalink / raw)
To: Joerg Roedel
Cc: Juergen Gross, Tom Lendacky, x86, Mike Stunes, Kees Cook, kvm,
Peter Zijlstra, Cfir Cohen, Dave Hansen, linux-kernel,
Sean Christopherson, virtualization, Martin Radev, Joerg Roedel,
Masami Hiramatsu, Andy Lutomirski, hpa, Erdem Aktas,
David Rientjes, Dan Williams, Jiri Slaby
On Mon, Sep 07, 2020 at 03:15:37PM +0200, Joerg Roedel wrote:
> @@ -347,7 +348,13 @@ bool sme_active(void)
>
> bool sev_active(void)
> {
> - return sme_me_mask && sev_enabled;
> + return !!(sev_status & MSR_AMD64_SEV_ENABLED);
Dropped those "!!" here too while applying.
--
Regards/Gruss,
Boris.
https://people.kernel.org/tglx/notes-about-netiquette
_______________________________________________
Virtualization mailing list
Virtualization@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/virtualization
^ permalink raw reply [flat|nested] 264+ messages in thread
* [PATCH v7.1 39/72] x86/sev-es: Setup early #VC handler
2020-09-07 13:15 ` Joerg Roedel
@ 2020-09-08 10:22 ` Joerg Roedel
-1 siblings, 0 replies; 264+ messages in thread
From: Joerg Roedel @ 2020-09-08 10:22 UTC (permalink / raw)
To: x86
Cc: Joerg Roedel, hpa, Andy Lutomirski, Dave Hansen, Peter Zijlstra,
Jiri Slaby, Dan Williams, Tom Lendacky, Juergen Gross, Kees Cook,
David Rientjes, Cfir Cohen, Erdem Aktas, Masami Hiramatsu,
Mike Stunes, Sean Christopherson, Martin Radev, linux-kernel,
kvm, virtualization
From: Joerg Roedel <jroedel@suse.de>
Setup an early handler for #VC exceptions. There is no GHCB mapped
yet, so just re-use the vc_no_ghcb_handler. It can only handle CPUID
exit-codes, but that should be enough to get the kernel through
verify_cpu() and __startup_64() until it runs on virtual addresses.
Signed-off-by: Joerg Roedel <jroedel@suse.de>
---
arch/x86/include/asm/sev-es.h | 3 +++
arch/x86/kernel/head64.c | 21 +++++++++++++++++++++
arch/x86/kernel/head_64.S | 30 ++++++++++++++++++++++++++++++
3 files changed, 54 insertions(+)
diff --git a/arch/x86/include/asm/sev-es.h b/arch/x86/include/asm/sev-es.h
index 6dc52440c4b4..7175d432ebfe 100644
--- a/arch/x86/include/asm/sev-es.h
+++ b/arch/x86/include/asm/sev-es.h
@@ -73,4 +73,7 @@ static inline u64 lower_bits(u64 val, unsigned int bits)
return (val & mask);
}
+/* Early IDT entry points for #VC handler */
+extern void vc_no_ghcb(void);
+
#endif
diff --git a/arch/x86/kernel/head64.c b/arch/x86/kernel/head64.c
index 4282dac694c3..5683bbb555ef 100644
--- a/arch/x86/kernel/head64.c
+++ b/arch/x86/kernel/head64.c
@@ -40,6 +40,7 @@
#include <asm/desc.h>
#include <asm/extable.h>
#include <asm/trapnr.h>
+#include <asm/sev-es.h>
/*
* Manage page tables very early on.
@@ -540,11 +541,31 @@ static struct desc_ptr bringup_idt_descr = {
.address = 0, /* Set at runtime */
};
+static void set_bringup_idt_handler(int n, void *handler)
+{
+#ifdef CONFIG_AMD_MEM_ENCRYPT
+ struct idt_data data;
+ gate_desc desc;
+
+ init_idt_data(&data, n, handler);
+ idt_init_desc(&desc, &data);
+ native_write_idt_entry(bringup_idt_table, n, &desc);
+#endif
+}
+
/* This runs while still in the direct mapping */
static void startup_64_load_idt(unsigned long physbase)
{
struct desc_ptr *desc = fixup_pointer(&bringup_idt_descr, physbase);
+ if (IS_ENABLED(CONFIG_AMD_MEM_ENCRYPT)) {
+ void *handler;
+
+ /* VMM Communication Exception */
+ handler = fixup_pointer(vc_no_ghcb, physbase);
+ set_bringup_idt_handler(X86_TRAP_VC, handler);
+ }
+
desc->address = (unsigned long)fixup_pointer(bringup_idt_table, physbase);
native_load_idt(desc);
}
diff --git a/arch/x86/kernel/head_64.S b/arch/x86/kernel/head_64.S
index 3b40ec44a67d..6e68bca64ae4 100644
--- a/arch/x86/kernel/head_64.S
+++ b/arch/x86/kernel/head_64.S
@@ -348,6 +348,36 @@ SYM_CODE_START_LOCAL(early_idt_handler_common)
jmp restore_regs_and_return_to_kernel
SYM_CODE_END(early_idt_handler_common)
+#ifdef CONFIG_AMD_MEM_ENCRYPT
+/*
+ * VC Exception handler used during very early boot. The
+ * early_idt_handler_array can't be used because it returns via the
+ * paravirtualized INTERRUPT_RETURN and pv-ops don't work that early.
+ *
+ * This handler will end up in the .init.text section and not be
+ * available to boot secondary CPUs.
+ */
+SYM_CODE_START_NOALIGN(vc_no_ghcb)
+ UNWIND_HINT_IRET_REGS offset=8
+
+ /* Build pt_regs */
+ PUSH_AND_CLEAR_REGS
+
+ /* Call C handler */
+ movq %rsp, %rdi
+ movq ORIG_RAX(%rsp), %rsi
+ call do_vc_no_ghcb
+
+ /* Unwind pt_regs */
+ POP_REGS
+
+ /* Remove Error Code */
+ addq $8, %rsp
+
+ /* Pure iret required here - don't use INTERRUPT_RETURN */
+ iretq
+SYM_CODE_END(vc_no_ghcb)
+#endif
#define SYM_DATA_START_PAGE_ALIGNED(name) \
SYM_START(name, SYM_L_GLOBAL, .balign PAGE_SIZE)
--
2.28.0
^ permalink raw reply related [flat|nested] 264+ messages in thread
* [PATCH v7.1 39/72] x86/sev-es: Setup early #VC handler
@ 2020-09-08 10:22 ` Joerg Roedel
0 siblings, 0 replies; 264+ messages in thread
From: Joerg Roedel @ 2020-09-08 10:22 UTC (permalink / raw)
To: x86
Cc: Juergen Gross, Tom Lendacky, Joerg Roedel, Mike Stunes,
Kees Cook, kvm, Peter Zijlstra, Cfir Cohen, Dave Hansen,
linux-kernel, Sean Christopherson, virtualization, Martin Radev,
Masami Hiramatsu, Andy Lutomirski, hpa, Erdem Aktas,
David Rientjes, Dan Williams, Jiri Slaby
From: Joerg Roedel <jroedel@suse.de>
Setup an early handler for #VC exceptions. There is no GHCB mapped
yet, so just re-use the vc_no_ghcb_handler. It can only handle CPUID
exit-codes, but that should be enough to get the kernel through
verify_cpu() and __startup_64() until it runs on virtual addresses.
Signed-off-by: Joerg Roedel <jroedel@suse.de>
---
arch/x86/include/asm/sev-es.h | 3 +++
arch/x86/kernel/head64.c | 21 +++++++++++++++++++++
arch/x86/kernel/head_64.S | 30 ++++++++++++++++++++++++++++++
3 files changed, 54 insertions(+)
diff --git a/arch/x86/include/asm/sev-es.h b/arch/x86/include/asm/sev-es.h
index 6dc52440c4b4..7175d432ebfe 100644
--- a/arch/x86/include/asm/sev-es.h
+++ b/arch/x86/include/asm/sev-es.h
@@ -73,4 +73,7 @@ static inline u64 lower_bits(u64 val, unsigned int bits)
return (val & mask);
}
+/* Early IDT entry points for #VC handler */
+extern void vc_no_ghcb(void);
+
#endif
diff --git a/arch/x86/kernel/head64.c b/arch/x86/kernel/head64.c
index 4282dac694c3..5683bbb555ef 100644
--- a/arch/x86/kernel/head64.c
+++ b/arch/x86/kernel/head64.c
@@ -40,6 +40,7 @@
#include <asm/desc.h>
#include <asm/extable.h>
#include <asm/trapnr.h>
+#include <asm/sev-es.h>
/*
* Manage page tables very early on.
@@ -540,11 +541,31 @@ static struct desc_ptr bringup_idt_descr = {
.address = 0, /* Set at runtime */
};
+static void set_bringup_idt_handler(int n, void *handler)
+{
+#ifdef CONFIG_AMD_MEM_ENCRYPT
+ struct idt_data data;
+ gate_desc desc;
+
+ init_idt_data(&data, n, handler);
+ idt_init_desc(&desc, &data);
+ native_write_idt_entry(bringup_idt_table, n, &desc);
+#endif
+}
+
/* This runs while still in the direct mapping */
static void startup_64_load_idt(unsigned long physbase)
{
struct desc_ptr *desc = fixup_pointer(&bringup_idt_descr, physbase);
+ if (IS_ENABLED(CONFIG_AMD_MEM_ENCRYPT)) {
+ void *handler;
+
+ /* VMM Communication Exception */
+ handler = fixup_pointer(vc_no_ghcb, physbase);
+ set_bringup_idt_handler(X86_TRAP_VC, handler);
+ }
+
desc->address = (unsigned long)fixup_pointer(bringup_idt_table, physbase);
native_load_idt(desc);
}
diff --git a/arch/x86/kernel/head_64.S b/arch/x86/kernel/head_64.S
index 3b40ec44a67d..6e68bca64ae4 100644
--- a/arch/x86/kernel/head_64.S
+++ b/arch/x86/kernel/head_64.S
@@ -348,6 +348,36 @@ SYM_CODE_START_LOCAL(early_idt_handler_common)
jmp restore_regs_and_return_to_kernel
SYM_CODE_END(early_idt_handler_common)
+#ifdef CONFIG_AMD_MEM_ENCRYPT
+/*
+ * VC Exception handler used during very early boot. The
+ * early_idt_handler_array can't be used because it returns via the
+ * paravirtualized INTERRUPT_RETURN and pv-ops don't work that early.
+ *
+ * This handler will end up in the .init.text section and not be
+ * available to boot secondary CPUs.
+ */
+SYM_CODE_START_NOALIGN(vc_no_ghcb)
+ UNWIND_HINT_IRET_REGS offset=8
+
+ /* Build pt_regs */
+ PUSH_AND_CLEAR_REGS
+
+ /* Call C handler */
+ movq %rsp, %rdi
+ movq ORIG_RAX(%rsp), %rsi
+ call do_vc_no_ghcb
+
+ /* Unwind pt_regs */
+ POP_REGS
+
+ /* Remove Error Code */
+ addq $8, %rsp
+
+ /* Pure iret required here - don't use INTERRUPT_RETURN */
+ iretq
+SYM_CODE_END(vc_no_ghcb)
+#endif
#define SYM_DATA_START_PAGE_ALIGNED(name) \
SYM_START(name, SYM_L_GLOBAL, .balign PAGE_SIZE)
--
2.28.0
_______________________________________________
Virtualization mailing list
Virtualization@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/virtualization
^ permalink raw reply related [flat|nested] 264+ messages in thread
* [PATCH v7.1 40/72] x86/sev-es: Setup GHCB based boot #VC handler
2020-09-07 13:15 ` Joerg Roedel
@ 2020-09-08 10:24 ` Joerg Roedel
-1 siblings, 0 replies; 264+ messages in thread
From: Joerg Roedel @ 2020-09-08 10:24 UTC (permalink / raw)
To: x86
Cc: Joerg Roedel, hpa, Andy Lutomirski, Dave Hansen, Peter Zijlstra,
Jiri Slaby, Dan Williams, Tom Lendacky, Juergen Gross, Kees Cook,
David Rientjes, Cfir Cohen, Erdem Aktas, Masami Hiramatsu,
Mike Stunes, Sean Christopherson, Martin Radev, linux-kernel,
kvm, virtualization
From: Joerg Roedel <jroedel@suse.de>
Add the infrastructure to handle #VC exceptions when the kernel runs
on virtual addresses and has a GHCB mapped. This handler will be used
until the runtime #VC handler takes over.
Since the handler runs very early, disable instrumentation for sev-es.c.
Signed-off-by: Joerg Roedel <jroedel@suse.de>
---
arch/x86/include/asm/realmode.h | 3 +
arch/x86/include/asm/segment.h | 2 +-
arch/x86/include/asm/sev-es.h | 2 +
arch/x86/kernel/Makefile | 2 +
arch/x86/kernel/head64.c | 8 +++
arch/x86/kernel/head_64.S | 36 ++++++++++
arch/x86/kernel/sev-es-shared.c | 14 ++--
arch/x86/kernel/sev-es.c | 116 ++++++++++++++++++++++++++++++++
arch/x86/mm/extable.c | 1 +
9 files changed, 176 insertions(+), 8 deletions(-)
diff --git a/arch/x86/include/asm/realmode.h b/arch/x86/include/asm/realmode.h
index b35030eeec36..96118fb041b8 100644
--- a/arch/x86/include/asm/realmode.h
+++ b/arch/x86/include/asm/realmode.h
@@ -57,6 +57,9 @@ extern unsigned char real_mode_blob_end[];
extern unsigned long initial_code;
extern unsigned long initial_gs;
extern unsigned long initial_stack;
+#ifdef CONFIG_AMD_MEM_ENCRYPT
+extern unsigned long initial_vc_handler;
+#endif
extern unsigned char real_mode_blob[];
extern unsigned char real_mode_relocs[];
diff --git a/arch/x86/include/asm/segment.h b/arch/x86/include/asm/segment.h
index 517920928989..7fdd4facfce7 100644
--- a/arch/x86/include/asm/segment.h
+++ b/arch/x86/include/asm/segment.h
@@ -226,7 +226,7 @@
#define NUM_EXCEPTION_VECTORS 32
/* Bitmask of exception vectors which push an error code on the stack: */
-#define EXCEPTION_ERRCODE_MASK 0x00027d00
+#define EXCEPTION_ERRCODE_MASK 0x20027d00
#define GDT_SIZE (GDT_ENTRIES*8)
#define GDT_ENTRY_TLS_ENTRIES 3
diff --git a/arch/x86/include/asm/sev-es.h b/arch/x86/include/asm/sev-es.h
index 7175d432ebfe..9fbeedaa66ee 100644
--- a/arch/x86/include/asm/sev-es.h
+++ b/arch/x86/include/asm/sev-es.h
@@ -75,5 +75,7 @@ static inline u64 lower_bits(u64 val, unsigned int bits)
/* Early IDT entry points for #VC handler */
extern void vc_no_ghcb(void);
+extern void vc_boot_ghcb(void);
+extern bool handle_vc_boot_ghcb(struct pt_regs *regs);
#endif
diff --git a/arch/x86/kernel/Makefile b/arch/x86/kernel/Makefile
index 3bcdd8d2bbdd..04ceea8f4a89 100644
--- a/arch/x86/kernel/Makefile
+++ b/arch/x86/kernel/Makefile
@@ -20,6 +20,7 @@ CFLAGS_REMOVE_kvmclock.o = -pg
CFLAGS_REMOVE_ftrace.o = -pg
CFLAGS_REMOVE_early_printk.o = -pg
CFLAGS_REMOVE_head64.o = -pg
+CFLAGS_REMOVE_sev-es.o = -pg
endif
KASAN_SANITIZE_head$(BITS).o := n
@@ -27,6 +28,7 @@ KASAN_SANITIZE_dumpstack.o := n
KASAN_SANITIZE_dumpstack_$(BITS).o := n
KASAN_SANITIZE_stacktrace.o := n
KASAN_SANITIZE_paravirt.o := n
+KASAN_SANITIZE_sev-es.o := n
# With some compiler versions the generated code results in boot hangs, caused
# by several compilation units. To be safe, disable all instrumentation.
diff --git a/arch/x86/kernel/head64.c b/arch/x86/kernel/head64.c
index 5683bbb555ef..530e055e231b 100644
--- a/arch/x86/kernel/head64.c
+++ b/arch/x86/kernel/head64.c
@@ -406,6 +406,10 @@ void __init do_early_exception(struct pt_regs *regs, int trapnr)
early_make_pgtable(native_read_cr2()))
return;
+ if (IS_ENABLED(CONFIG_AMD_MEM_ENCRYPT) &&
+ trapnr == X86_TRAP_VC && handle_vc_boot_ghcb(regs))
+ return;
+
early_fixup_exception(regs, trapnr);
}
@@ -573,6 +577,10 @@ static void startup_64_load_idt(unsigned long physbase)
/* This is used when running on kernel addresses */
void early_setup_idt(void)
{
+ /* VMM Communication Exception */
+ if (IS_ENABLED(CONFIG_AMD_MEM_ENCRYPT))
+ set_bringup_idt_handler(X86_TRAP_VC, vc_boot_ghcb);
+
bringup_idt_descr.address = (unsigned long)bringup_idt_table;
native_load_idt(&bringup_idt_descr);
}
diff --git a/arch/x86/kernel/head_64.S b/arch/x86/kernel/head_64.S
index 6e68bca64ae4..1a71d0d4d575 100644
--- a/arch/x86/kernel/head_64.S
+++ b/arch/x86/kernel/head_64.S
@@ -279,6 +279,39 @@ SYM_CODE_START(start_cpu0)
movq initial_stack(%rip), %rsp
jmp .Ljump_to_C_code
SYM_CODE_END(start_cpu0)
+#endif
+
+#ifdef CONFIG_AMD_MEM_ENCRYPT
+/*
+ * VC Exception handler used during early boot when running on kernel
+ * addresses, but before the switch to the idt_table can be made.
+ * The early_idt_handler_array can't be used here because it calls into a lot
+ * of __init code and this handler is also used during CPU offlining/onlining.
+ * Therefore this handler ends up in the .text section so that it stays around
+ * when .init.text is freed.
+ */
+SYM_CODE_START_NOALIGN(vc_boot_ghcb)
+ UNWIND_HINT_IRET_REGS offset=8
+
+ /* Build pt_regs */
+ PUSH_AND_CLEAR_REGS
+
+ /* Call C handler */
+ movq %rsp, %rdi
+ movq ORIG_RAX(%rsp), %rsi
+ movq initial_vc_handler(%rip), %rax
+ ANNOTATE_RETPOLINE_SAFE
+ call *%rax
+
+ /* Unwind pt_regs */
+ POP_REGS
+
+ /* Remove Error Code */
+ addq $8, %rsp
+
+ /* Pure iret required here - don't use INTERRUPT_RETURN */
+ iretq
+SYM_CODE_END(vc_boot_ghcb)
#endif
/* Both SMP bootup and ACPI suspend change these variables */
@@ -286,6 +319,9 @@ SYM_CODE_END(start_cpu0)
.balign 8
SYM_DATA(initial_code, .quad x86_64_start_kernel)
SYM_DATA(initial_gs, .quad INIT_PER_CPU_VAR(fixed_percpu_data))
+#ifdef CONFIG_AMD_MEM_ENCRYPT
+SYM_DATA(initial_vc_handler, .quad handle_vc_boot_ghcb)
+#endif
/*
* The SIZEOF_PTREGS gap is a convention which helps the in-kernel unwinder
diff --git a/arch/x86/kernel/sev-es-shared.c b/arch/x86/kernel/sev-es-shared.c
index 18619279a46f..aa77f2eb8d88 100644
--- a/arch/x86/kernel/sev-es-shared.c
+++ b/arch/x86/kernel/sev-es-shared.c
@@ -9,7 +9,7 @@
* and is included directly into both code-bases.
*/
-static void __maybe_unused sev_es_terminate(unsigned int reason)
+static void sev_es_terminate(unsigned int reason)
{
u64 val = GHCB_SEV_TERMINATE;
@@ -27,7 +27,7 @@ static void __maybe_unused sev_es_terminate(unsigned int reason)
asm volatile("hlt\n" : : : "memory");
}
-static bool __maybe_unused sev_es_negotiate_protocol(void)
+static bool sev_es_negotiate_protocol(void)
{
u64 val;
@@ -46,7 +46,7 @@ static bool __maybe_unused sev_es_negotiate_protocol(void)
return true;
}
-static void __maybe_unused vc_ghcb_invalidate(struct ghcb *ghcb)
+static void vc_ghcb_invalidate(struct ghcb *ghcb)
{
memset(ghcb->save.valid_bitmap, 0, sizeof(ghcb->save.valid_bitmap));
}
@@ -58,9 +58,9 @@ static bool vc_decoding_needed(unsigned long exit_code)
exit_code <= SVM_EXIT_LAST_EXCP);
}
-static enum es_result __maybe_unused vc_init_em_ctxt(struct es_em_ctxt *ctxt,
- struct pt_regs *regs,
- unsigned long exit_code)
+static enum es_result vc_init_em_ctxt(struct es_em_ctxt *ctxt,
+ struct pt_regs *regs,
+ unsigned long exit_code)
{
enum es_result ret = ES_OK;
@@ -73,7 +73,7 @@ static enum es_result __maybe_unused vc_init_em_ctxt(struct es_em_ctxt *ctxt,
return ret;
}
-static void __maybe_unused vc_finish_insn(struct es_em_ctxt *ctxt)
+static void vc_finish_insn(struct es_em_ctxt *ctxt)
{
ctxt->regs->ip += ctxt->insn.length;
}
diff --git a/arch/x86/kernel/sev-es.c b/arch/x86/kernel/sev-es.c
index 0b698b653c0b..bb3e702a71eb 100644
--- a/arch/x86/kernel/sev-es.c
+++ b/arch/x86/kernel/sev-es.c
@@ -7,7 +7,9 @@
* Author: Joerg Roedel <jroedel@suse.de>
*/
+#include <linux/sched/debug.h> /* For show_regs() */
#include <linux/kernel.h>
+#include <linux/printk.h>
#include <linux/mm.h>
#include <asm/sev-es.h>
@@ -18,6 +20,18 @@
#include <asm/trapnr.h>
#include <asm/svm.h>
+/* For early boot hypervisor communication in SEV-ES enabled guests */
+static struct ghcb boot_ghcb_page __bss_decrypted __aligned(PAGE_SIZE);
+
+/*
+ * Needs to be in the .data section because we need it NULL before bss is
+ * cleared
+ */
+static struct ghcb __initdata *boot_ghcb;
+
+/* Needed in vc_early_forward_exception */
+void do_early_exception(struct pt_regs *regs, int trapnr);
+
static inline u64 sev_es_rd_ghcb_msr(void)
{
return native_read_msr(MSR_AMD64_SEV_ES_GHCB);
@@ -161,3 +175,105 @@ static enum es_result vc_read_mem(struct es_em_ctxt *ctxt,
/* Include code shared with pre-decompression boot stage */
#include "sev-es-shared.c"
+
+/*
+ * This function runs on the first #VC exception after the kernel
+ * switched to virtual addresses.
+ */
+static bool __init sev_es_setup_ghcb(void)
+{
+ /* First make sure the hypervisor talks a supported protocol. */
+ if (!sev_es_negotiate_protocol())
+ return false;
+
+ /*
+ * Clear the boot_ghcb. The first exception comes in before the bss
+ * section is cleared.
+ */
+ memset(&boot_ghcb_page, 0, PAGE_SIZE);
+
+ /* Alright - Make the boot-ghcb public */
+ boot_ghcb = &boot_ghcb_page;
+
+ return true;
+}
+
+static void __init vc_early_forward_exception(struct es_em_ctxt *ctxt)
+{
+ int trapnr = ctxt->fi.vector;
+
+ if (trapnr == X86_TRAP_PF)
+ native_write_cr2(ctxt->fi.cr2);
+
+ ctxt->regs->orig_ax = ctxt->fi.error_code;
+ do_early_exception(ctxt->regs, trapnr);
+}
+
+static enum es_result vc_handle_exitcode(struct es_em_ctxt *ctxt,
+ struct ghcb *ghcb,
+ unsigned long exit_code)
+{
+ enum es_result result;
+
+ switch (exit_code) {
+ default:
+ /*
+ * Unexpected #VC exception
+ */
+ result = ES_UNSUPPORTED;
+ }
+
+ return result;
+}
+
+bool __init handle_vc_boot_ghcb(struct pt_regs *regs)
+{
+ unsigned long exit_code = regs->orig_ax;
+ struct es_em_ctxt ctxt;
+ enum es_result result;
+
+ /* Do initial setup or terminate the guest */
+ if (unlikely(boot_ghcb == NULL && !sev_es_setup_ghcb()))
+ sev_es_terminate(GHCB_SEV_ES_REASON_GENERAL_REQUEST);
+
+ vc_ghcb_invalidate(boot_ghcb);
+
+ result = vc_init_em_ctxt(&ctxt, regs, exit_code);
+ if (result == ES_OK)
+ result = vc_handle_exitcode(&ctxt, boot_ghcb, exit_code);
+
+ /* Done - now check the result */
+ switch (result) {
+ case ES_OK:
+ vc_finish_insn(&ctxt);
+ break;
+ case ES_UNSUPPORTED:
+ early_printk("PANIC: Unsupported exit-code 0x%02lx in early #VC exception (IP: 0x%lx)\n",
+ exit_code, regs->ip);
+ goto fail;
+ case ES_VMM_ERROR:
+ early_printk("PANIC: Failure in communication with VMM (exit-code 0x%02lx IP: 0x%lx)\n",
+ exit_code, regs->ip);
+ goto fail;
+ case ES_DECODE_FAILED:
+ early_printk("PANIC: Failed to decode instruction (exit-code 0x%02lx IP: 0x%lx)\n",
+ exit_code, regs->ip);
+ goto fail;
+ case ES_EXCEPTION:
+ vc_early_forward_exception(&ctxt);
+ break;
+ case ES_RETRY:
+ /* Nothing to do */
+ break;
+ default:
+ BUG();
+ }
+
+ return true;
+
+fail:
+ show_regs(regs);
+
+ while (true)
+ halt();
+}
diff --git a/arch/x86/mm/extable.c b/arch/x86/mm/extable.c
index 1d6cb07f4f86..3966749d07ac 100644
--- a/arch/x86/mm/extable.c
+++ b/arch/x86/mm/extable.c
@@ -5,6 +5,7 @@
#include <xen/xen.h>
#include <asm/fpu/internal.h>
+#include <asm/sev-es.h>
#include <asm/traps.h>
#include <asm/kdebug.h>
--
2.28.0
^ permalink raw reply related [flat|nested] 264+ messages in thread
* [PATCH v7.1 40/72] x86/sev-es: Setup GHCB based boot #VC handler
@ 2020-09-08 10:24 ` Joerg Roedel
0 siblings, 0 replies; 264+ messages in thread
From: Joerg Roedel @ 2020-09-08 10:24 UTC (permalink / raw)
To: x86
Cc: Juergen Gross, Tom Lendacky, Joerg Roedel, Mike Stunes,
Kees Cook, kvm, Peter Zijlstra, Cfir Cohen, Dave Hansen,
linux-kernel, Sean Christopherson, virtualization, Martin Radev,
Masami Hiramatsu, Andy Lutomirski, hpa, Erdem Aktas,
David Rientjes, Dan Williams, Jiri Slaby
From: Joerg Roedel <jroedel@suse.de>
Add the infrastructure to handle #VC exceptions when the kernel runs
on virtual addresses and has a GHCB mapped. This handler will be used
until the runtime #VC handler takes over.
Since the handler runs very early, disable instrumentation for sev-es.c.
Signed-off-by: Joerg Roedel <jroedel@suse.de>
---
arch/x86/include/asm/realmode.h | 3 +
arch/x86/include/asm/segment.h | 2 +-
arch/x86/include/asm/sev-es.h | 2 +
arch/x86/kernel/Makefile | 2 +
arch/x86/kernel/head64.c | 8 +++
arch/x86/kernel/head_64.S | 36 ++++++++++
arch/x86/kernel/sev-es-shared.c | 14 ++--
arch/x86/kernel/sev-es.c | 116 ++++++++++++++++++++++++++++++++
arch/x86/mm/extable.c | 1 +
9 files changed, 176 insertions(+), 8 deletions(-)
diff --git a/arch/x86/include/asm/realmode.h b/arch/x86/include/asm/realmode.h
index b35030eeec36..96118fb041b8 100644
--- a/arch/x86/include/asm/realmode.h
+++ b/arch/x86/include/asm/realmode.h
@@ -57,6 +57,9 @@ extern unsigned char real_mode_blob_end[];
extern unsigned long initial_code;
extern unsigned long initial_gs;
extern unsigned long initial_stack;
+#ifdef CONFIG_AMD_MEM_ENCRYPT
+extern unsigned long initial_vc_handler;
+#endif
extern unsigned char real_mode_blob[];
extern unsigned char real_mode_relocs[];
diff --git a/arch/x86/include/asm/segment.h b/arch/x86/include/asm/segment.h
index 517920928989..7fdd4facfce7 100644
--- a/arch/x86/include/asm/segment.h
+++ b/arch/x86/include/asm/segment.h
@@ -226,7 +226,7 @@
#define NUM_EXCEPTION_VECTORS 32
/* Bitmask of exception vectors which push an error code on the stack: */
-#define EXCEPTION_ERRCODE_MASK 0x00027d00
+#define EXCEPTION_ERRCODE_MASK 0x20027d00
#define GDT_SIZE (GDT_ENTRIES*8)
#define GDT_ENTRY_TLS_ENTRIES 3
diff --git a/arch/x86/include/asm/sev-es.h b/arch/x86/include/asm/sev-es.h
index 7175d432ebfe..9fbeedaa66ee 100644
--- a/arch/x86/include/asm/sev-es.h
+++ b/arch/x86/include/asm/sev-es.h
@@ -75,5 +75,7 @@ static inline u64 lower_bits(u64 val, unsigned int bits)
/* Early IDT entry points for #VC handler */
extern void vc_no_ghcb(void);
+extern void vc_boot_ghcb(void);
+extern bool handle_vc_boot_ghcb(struct pt_regs *regs);
#endif
diff --git a/arch/x86/kernel/Makefile b/arch/x86/kernel/Makefile
index 3bcdd8d2bbdd..04ceea8f4a89 100644
--- a/arch/x86/kernel/Makefile
+++ b/arch/x86/kernel/Makefile
@@ -20,6 +20,7 @@ CFLAGS_REMOVE_kvmclock.o = -pg
CFLAGS_REMOVE_ftrace.o = -pg
CFLAGS_REMOVE_early_printk.o = -pg
CFLAGS_REMOVE_head64.o = -pg
+CFLAGS_REMOVE_sev-es.o = -pg
endif
KASAN_SANITIZE_head$(BITS).o := n
@@ -27,6 +28,7 @@ KASAN_SANITIZE_dumpstack.o := n
KASAN_SANITIZE_dumpstack_$(BITS).o := n
KASAN_SANITIZE_stacktrace.o := n
KASAN_SANITIZE_paravirt.o := n
+KASAN_SANITIZE_sev-es.o := n
# With some compiler versions the generated code results in boot hangs, caused
# by several compilation units. To be safe, disable all instrumentation.
diff --git a/arch/x86/kernel/head64.c b/arch/x86/kernel/head64.c
index 5683bbb555ef..530e055e231b 100644
--- a/arch/x86/kernel/head64.c
+++ b/arch/x86/kernel/head64.c
@@ -406,6 +406,10 @@ void __init do_early_exception(struct pt_regs *regs, int trapnr)
early_make_pgtable(native_read_cr2()))
return;
+ if (IS_ENABLED(CONFIG_AMD_MEM_ENCRYPT) &&
+ trapnr == X86_TRAP_VC && handle_vc_boot_ghcb(regs))
+ return;
+
early_fixup_exception(regs, trapnr);
}
@@ -573,6 +577,10 @@ static void startup_64_load_idt(unsigned long physbase)
/* This is used when running on kernel addresses */
void early_setup_idt(void)
{
+ /* VMM Communication Exception */
+ if (IS_ENABLED(CONFIG_AMD_MEM_ENCRYPT))
+ set_bringup_idt_handler(X86_TRAP_VC, vc_boot_ghcb);
+
bringup_idt_descr.address = (unsigned long)bringup_idt_table;
native_load_idt(&bringup_idt_descr);
}
diff --git a/arch/x86/kernel/head_64.S b/arch/x86/kernel/head_64.S
index 6e68bca64ae4..1a71d0d4d575 100644
--- a/arch/x86/kernel/head_64.S
+++ b/arch/x86/kernel/head_64.S
@@ -279,6 +279,39 @@ SYM_CODE_START(start_cpu0)
movq initial_stack(%rip), %rsp
jmp .Ljump_to_C_code
SYM_CODE_END(start_cpu0)
+#endif
+
+#ifdef CONFIG_AMD_MEM_ENCRYPT
+/*
+ * VC Exception handler used during early boot when running on kernel
+ * addresses, but before the switch to the idt_table can be made.
+ * The early_idt_handler_array can't be used here because it calls into a lot
+ * of __init code and this handler is also used during CPU offlining/onlining.
+ * Therefore this handler ends up in the .text section so that it stays around
+ * when .init.text is freed.
+ */
+SYM_CODE_START_NOALIGN(vc_boot_ghcb)
+ UNWIND_HINT_IRET_REGS offset=8
+
+ /* Build pt_regs */
+ PUSH_AND_CLEAR_REGS
+
+ /* Call C handler */
+ movq %rsp, %rdi
+ movq ORIG_RAX(%rsp), %rsi
+ movq initial_vc_handler(%rip), %rax
+ ANNOTATE_RETPOLINE_SAFE
+ call *%rax
+
+ /* Unwind pt_regs */
+ POP_REGS
+
+ /* Remove Error Code */
+ addq $8, %rsp
+
+ /* Pure iret required here - don't use INTERRUPT_RETURN */
+ iretq
+SYM_CODE_END(vc_boot_ghcb)
#endif
/* Both SMP bootup and ACPI suspend change these variables */
@@ -286,6 +319,9 @@ SYM_CODE_END(start_cpu0)
.balign 8
SYM_DATA(initial_code, .quad x86_64_start_kernel)
SYM_DATA(initial_gs, .quad INIT_PER_CPU_VAR(fixed_percpu_data))
+#ifdef CONFIG_AMD_MEM_ENCRYPT
+SYM_DATA(initial_vc_handler, .quad handle_vc_boot_ghcb)
+#endif
/*
* The SIZEOF_PTREGS gap is a convention which helps the in-kernel unwinder
diff --git a/arch/x86/kernel/sev-es-shared.c b/arch/x86/kernel/sev-es-shared.c
index 18619279a46f..aa77f2eb8d88 100644
--- a/arch/x86/kernel/sev-es-shared.c
+++ b/arch/x86/kernel/sev-es-shared.c
@@ -9,7 +9,7 @@
* and is included directly into both code-bases.
*/
-static void __maybe_unused sev_es_terminate(unsigned int reason)
+static void sev_es_terminate(unsigned int reason)
{
u64 val = GHCB_SEV_TERMINATE;
@@ -27,7 +27,7 @@ static void __maybe_unused sev_es_terminate(unsigned int reason)
asm volatile("hlt\n" : : : "memory");
}
-static bool __maybe_unused sev_es_negotiate_protocol(void)
+static bool sev_es_negotiate_protocol(void)
{
u64 val;
@@ -46,7 +46,7 @@ static bool __maybe_unused sev_es_negotiate_protocol(void)
return true;
}
-static void __maybe_unused vc_ghcb_invalidate(struct ghcb *ghcb)
+static void vc_ghcb_invalidate(struct ghcb *ghcb)
{
memset(ghcb->save.valid_bitmap, 0, sizeof(ghcb->save.valid_bitmap));
}
@@ -58,9 +58,9 @@ static bool vc_decoding_needed(unsigned long exit_code)
exit_code <= SVM_EXIT_LAST_EXCP);
}
-static enum es_result __maybe_unused vc_init_em_ctxt(struct es_em_ctxt *ctxt,
- struct pt_regs *regs,
- unsigned long exit_code)
+static enum es_result vc_init_em_ctxt(struct es_em_ctxt *ctxt,
+ struct pt_regs *regs,
+ unsigned long exit_code)
{
enum es_result ret = ES_OK;
@@ -73,7 +73,7 @@ static enum es_result __maybe_unused vc_init_em_ctxt(struct es_em_ctxt *ctxt,
return ret;
}
-static void __maybe_unused vc_finish_insn(struct es_em_ctxt *ctxt)
+static void vc_finish_insn(struct es_em_ctxt *ctxt)
{
ctxt->regs->ip += ctxt->insn.length;
}
diff --git a/arch/x86/kernel/sev-es.c b/arch/x86/kernel/sev-es.c
index 0b698b653c0b..bb3e702a71eb 100644
--- a/arch/x86/kernel/sev-es.c
+++ b/arch/x86/kernel/sev-es.c
@@ -7,7 +7,9 @@
* Author: Joerg Roedel <jroedel@suse.de>
*/
+#include <linux/sched/debug.h> /* For show_regs() */
#include <linux/kernel.h>
+#include <linux/printk.h>
#include <linux/mm.h>
#include <asm/sev-es.h>
@@ -18,6 +20,18 @@
#include <asm/trapnr.h>
#include <asm/svm.h>
+/* For early boot hypervisor communication in SEV-ES enabled guests */
+static struct ghcb boot_ghcb_page __bss_decrypted __aligned(PAGE_SIZE);
+
+/*
+ * Needs to be in the .data section because we need it NULL before bss is
+ * cleared
+ */
+static struct ghcb __initdata *boot_ghcb;
+
+/* Needed in vc_early_forward_exception */
+void do_early_exception(struct pt_regs *regs, int trapnr);
+
static inline u64 sev_es_rd_ghcb_msr(void)
{
return native_read_msr(MSR_AMD64_SEV_ES_GHCB);
@@ -161,3 +175,105 @@ static enum es_result vc_read_mem(struct es_em_ctxt *ctxt,
/* Include code shared with pre-decompression boot stage */
#include "sev-es-shared.c"
+
+/*
+ * This function runs on the first #VC exception after the kernel
+ * switched to virtual addresses.
+ */
+static bool __init sev_es_setup_ghcb(void)
+{
+ /* First make sure the hypervisor talks a supported protocol. */
+ if (!sev_es_negotiate_protocol())
+ return false;
+
+ /*
+ * Clear the boot_ghcb. The first exception comes in before the bss
+ * section is cleared.
+ */
+ memset(&boot_ghcb_page, 0, PAGE_SIZE);
+
+ /* Alright - Make the boot-ghcb public */
+ boot_ghcb = &boot_ghcb_page;
+
+ return true;
+}
+
+static void __init vc_early_forward_exception(struct es_em_ctxt *ctxt)
+{
+ int trapnr = ctxt->fi.vector;
+
+ if (trapnr == X86_TRAP_PF)
+ native_write_cr2(ctxt->fi.cr2);
+
+ ctxt->regs->orig_ax = ctxt->fi.error_code;
+ do_early_exception(ctxt->regs, trapnr);
+}
+
+static enum es_result vc_handle_exitcode(struct es_em_ctxt *ctxt,
+ struct ghcb *ghcb,
+ unsigned long exit_code)
+{
+ enum es_result result;
+
+ switch (exit_code) {
+ default:
+ /*
+ * Unexpected #VC exception
+ */
+ result = ES_UNSUPPORTED;
+ }
+
+ return result;
+}
+
+bool __init handle_vc_boot_ghcb(struct pt_regs *regs)
+{
+ unsigned long exit_code = regs->orig_ax;
+ struct es_em_ctxt ctxt;
+ enum es_result result;
+
+ /* Do initial setup or terminate the guest */
+ if (unlikely(boot_ghcb == NULL && !sev_es_setup_ghcb()))
+ sev_es_terminate(GHCB_SEV_ES_REASON_GENERAL_REQUEST);
+
+ vc_ghcb_invalidate(boot_ghcb);
+
+ result = vc_init_em_ctxt(&ctxt, regs, exit_code);
+ if (result == ES_OK)
+ result = vc_handle_exitcode(&ctxt, boot_ghcb, exit_code);
+
+ /* Done - now check the result */
+ switch (result) {
+ case ES_OK:
+ vc_finish_insn(&ctxt);
+ break;
+ case ES_UNSUPPORTED:
+ early_printk("PANIC: Unsupported exit-code 0x%02lx in early #VC exception (IP: 0x%lx)\n",
+ exit_code, regs->ip);
+ goto fail;
+ case ES_VMM_ERROR:
+ early_printk("PANIC: Failure in communication with VMM (exit-code 0x%02lx IP: 0x%lx)\n",
+ exit_code, regs->ip);
+ goto fail;
+ case ES_DECODE_FAILED:
+ early_printk("PANIC: Failed to decode instruction (exit-code 0x%02lx IP: 0x%lx)\n",
+ exit_code, regs->ip);
+ goto fail;
+ case ES_EXCEPTION:
+ vc_early_forward_exception(&ctxt);
+ break;
+ case ES_RETRY:
+ /* Nothing to do */
+ break;
+ default:
+ BUG();
+ }
+
+ return true;
+
+fail:
+ show_regs(regs);
+
+ while (true)
+ halt();
+}
diff --git a/arch/x86/mm/extable.c b/arch/x86/mm/extable.c
index 1d6cb07f4f86..3966749d07ac 100644
--- a/arch/x86/mm/extable.c
+++ b/arch/x86/mm/extable.c
@@ -5,6 +5,7 @@
#include <xen/xen.h>
#include <asm/fpu/internal.h>
+#include <asm/sev-es.h>
#include <asm/traps.h>
#include <asm/kdebug.h>
--
2.28.0
_______________________________________________
Virtualization mailing list
Virtualization@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/virtualization
^ permalink raw reply related [flat|nested] 264+ messages in thread
* Re: [PATCH v7 41/72] x86/sev-es: Setup per-cpu GHCBs for the runtime handler
2020-09-07 13:15 ` Joerg Roedel
@ 2020-09-08 10:51 ` Borislav Petkov
-1 siblings, 0 replies; 264+ messages in thread
From: Borislav Petkov @ 2020-09-08 10:51 UTC (permalink / raw)
To: Joerg Roedel
Cc: x86, Joerg Roedel, Tom Lendacky, hpa, Andy Lutomirski,
Dave Hansen, Peter Zijlstra, Jiri Slaby, Dan Williams,
Juergen Gross, Kees Cook, David Rientjes, Cfir Cohen,
Erdem Aktas, Masami Hiramatsu, Mike Stunes, Sean Christopherson,
Martin Radev, linux-kernel, kvm, virtualization
On Mon, Sep 07, 2020 at 03:15:42PM +0200, Joerg Roedel wrote:
> +void __init sev_es_init_vc_handling(void)
> +{
> + int cpu;
> +
> + BUILD_BUG_ON((offsetof(struct sev_es_runtime_data, ghcb_page) % PAGE_SIZE) != 0);
Simplified that to:
BUILD_BUG_ON(offsetof(struct sev_es_runtime_data, ghcb_page) % PAGE_SIZE);
--
Regards/Gruss,
Boris.
https://people.kernel.org/tglx/notes-about-netiquette
^ permalink raw reply [flat|nested] 264+ messages in thread
* Re: [PATCH v7 41/72] x86/sev-es: Setup per-cpu GHCBs for the runtime handler
@ 2020-09-08 10:51 ` Borislav Petkov
0 siblings, 0 replies; 264+ messages in thread
From: Borislav Petkov @ 2020-09-08 10:51 UTC (permalink / raw)
To: Joerg Roedel
Cc: Juergen Gross, Tom Lendacky, Dave Hansen, Mike Stunes, Kees Cook,
kvm, Peter Zijlstra, Cfir Cohen, x86, linux-kernel,
Sean Christopherson, virtualization, Martin Radev, Joerg Roedel,
Masami Hiramatsu, Andy Lutomirski, hpa, Erdem Aktas,
David Rientjes, Dan Williams, Jiri Slaby
On Mon, Sep 07, 2020 at 03:15:42PM +0200, Joerg Roedel wrote:
> +void __init sev_es_init_vc_handling(void)
> +{
> + int cpu;
> +
> + BUILD_BUG_ON((offsetof(struct sev_es_runtime_data, ghcb_page) % PAGE_SIZE) != 0);
Simplified that to:
BUILD_BUG_ON(offsetof(struct sev_es_runtime_data, ghcb_page) % PAGE_SIZE);
--
Regards/Gruss,
Boris.
https://people.kernel.org/tglx/notes-about-netiquette
_______________________________________________
Virtualization mailing list
Virtualization@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/virtualization
^ permalink raw reply [flat|nested] 264+ messages in thread
* [PATCH v7.2 39/74] x86/sev-es: Setup early #VC handler
2020-09-07 13:15 ` Joerg Roedel
@ 2020-09-08 12:35 ` Joerg Roedel
-1 siblings, 0 replies; 264+ messages in thread
From: Joerg Roedel @ 2020-09-08 12:35 UTC (permalink / raw)
To: x86
Cc: Joerg Roedel, hpa, Andy Lutomirski, Dave Hansen, Peter Zijlstra,
Jiri Slaby, Dan Williams, Tom Lendacky, Juergen Gross, Kees Cook,
David Rientjes, Cfir Cohen, Erdem Aktas, Masami Hiramatsu,
Mike Stunes, Sean Christopherson, Martin Radev, linux-kernel,
kvm, virtualization
From: Joerg Roedel <jroedel@suse.de>
Setup an early handler for #VC exceptions. There is no GHCB mapped
yet, so just re-use the vc_no_ghcb_handler. It can only handle CPUID
exit-codes, but that should be enough to get the kernel through
verify_cpu() and __startup_64() until it runs on virtual addresses.
Signed-off-by: Joerg Roedel <jroedel@suse.de>
---
arch/x86/include/asm/sev-es.h | 3 +++
arch/x86/kernel/head64.c | 25 ++++++++++++++++++++++++-
arch/x86/kernel/head_64.S | 30 ++++++++++++++++++++++++++++++
3 files changed, 57 insertions(+), 1 deletion(-)
diff --git a/arch/x86/include/asm/sev-es.h b/arch/x86/include/asm/sev-es.h
index 6dc52440c4b4..7175d432ebfe 100644
--- a/arch/x86/include/asm/sev-es.h
+++ b/arch/x86/include/asm/sev-es.h
@@ -73,4 +73,7 @@ static inline u64 lower_bits(u64 val, unsigned int bits)
return (val & mask);
}
+/* Early IDT entry points for #VC handler */
+extern void vc_no_ghcb(void);
+
#endif
diff --git a/arch/x86/kernel/head64.c b/arch/x86/kernel/head64.c
index 4282dac694c3..fc55cc9ccb0f 100644
--- a/arch/x86/kernel/head64.c
+++ b/arch/x86/kernel/head64.c
@@ -40,6 +40,7 @@
#include <asm/desc.h>
#include <asm/extable.h>
#include <asm/trapnr.h>
+#include <asm/sev-es.h>
/*
* Manage page tables very early on.
@@ -540,12 +541,34 @@ static struct desc_ptr bringup_idt_descr = {
.address = 0, /* Set at runtime */
};
+static void set_bringup_idt_handler(gate_desc *idt, int n, void *handler)
+{
+#ifdef CONFIG_AMD_MEM_ENCRYPT
+ struct idt_data data;
+ gate_desc desc;
+
+ init_idt_data(&data, n, handler);
+ idt_init_desc(&desc, &data);
+ native_write_idt_entry(idt, n, &desc);
+#endif
+}
+
/* This runs while still in the direct mapping */
static void startup_64_load_idt(unsigned long physbase)
{
struct desc_ptr *desc = fixup_pointer(&bringup_idt_descr, physbase);
+ gate_desc *idt = fixup_pointer(bringup_idt_table, physbase);
+
+
+ if (IS_ENABLED(CONFIG_AMD_MEM_ENCRYPT)) {
+ void *handler;
+
+ /* VMM Communication Exception */
+ handler = fixup_pointer(vc_no_ghcb, physbase);
+ set_bringup_idt_handler(idt, X86_TRAP_VC, handler);
+ }
- desc->address = (unsigned long)fixup_pointer(bringup_idt_table, physbase);
+ desc->address = (unsigned long)idt;
native_load_idt(desc);
}
diff --git a/arch/x86/kernel/head_64.S b/arch/x86/kernel/head_64.S
index 3b40ec44a67d..6e68bca64ae4 100644
--- a/arch/x86/kernel/head_64.S
+++ b/arch/x86/kernel/head_64.S
@@ -348,6 +348,36 @@ SYM_CODE_START_LOCAL(early_idt_handler_common)
jmp restore_regs_and_return_to_kernel
SYM_CODE_END(early_idt_handler_common)
+#ifdef CONFIG_AMD_MEM_ENCRYPT
+/*
+ * VC Exception handler used during very early boot. The
+ * early_idt_handler_array can't be used because it returns via the
+ * paravirtualized INTERRUPT_RETURN and pv-ops don't work that early.
+ *
+ * This handler will end up in the .init.text section and not be
+ * available to boot secondary CPUs.
+ */
+SYM_CODE_START_NOALIGN(vc_no_ghcb)
+ UNWIND_HINT_IRET_REGS offset=8
+
+ /* Build pt_regs */
+ PUSH_AND_CLEAR_REGS
+
+ /* Call C handler */
+ movq %rsp, %rdi
+ movq ORIG_RAX(%rsp), %rsi
+ call do_vc_no_ghcb
+
+ /* Unwind pt_regs */
+ POP_REGS
+
+ /* Remove Error Code */
+ addq $8, %rsp
+
+ /* Pure iret required here - don't use INTERRUPT_RETURN */
+ iretq
+SYM_CODE_END(vc_no_ghcb)
+#endif
#define SYM_DATA_START_PAGE_ALIGNED(name) \
SYM_START(name, SYM_L_GLOBAL, .balign PAGE_SIZE)
--
2.28.0
^ permalink raw reply related [flat|nested] 264+ messages in thread
* [PATCH v7.2 39/74] x86/sev-es: Setup early #VC handler
@ 2020-09-08 12:35 ` Joerg Roedel
0 siblings, 0 replies; 264+ messages in thread
From: Joerg Roedel @ 2020-09-08 12:35 UTC (permalink / raw)
To: x86
Cc: Juergen Gross, Tom Lendacky, Joerg Roedel, Mike Stunes,
Kees Cook, kvm, Peter Zijlstra, Cfir Cohen, Dave Hansen,
linux-kernel, Sean Christopherson, virtualization, Martin Radev,
Masami Hiramatsu, Andy Lutomirski, hpa, Erdem Aktas,
David Rientjes, Dan Williams, Jiri Slaby
From: Joerg Roedel <jroedel@suse.de>
Setup an early handler for #VC exceptions. There is no GHCB mapped
yet, so just re-use the vc_no_ghcb_handler. It can only handle CPUID
exit-codes, but that should be enough to get the kernel through
verify_cpu() and __startup_64() until it runs on virtual addresses.
Signed-off-by: Joerg Roedel <jroedel@suse.de>
---
arch/x86/include/asm/sev-es.h | 3 +++
arch/x86/kernel/head64.c | 25 ++++++++++++++++++++++++-
arch/x86/kernel/head_64.S | 30 ++++++++++++++++++++++++++++++
3 files changed, 57 insertions(+), 1 deletion(-)
diff --git a/arch/x86/include/asm/sev-es.h b/arch/x86/include/asm/sev-es.h
index 6dc52440c4b4..7175d432ebfe 100644
--- a/arch/x86/include/asm/sev-es.h
+++ b/arch/x86/include/asm/sev-es.h
@@ -73,4 +73,7 @@ static inline u64 lower_bits(u64 val, unsigned int bits)
return (val & mask);
}
+/* Early IDT entry points for #VC handler */
+extern void vc_no_ghcb(void);
+
#endif
diff --git a/arch/x86/kernel/head64.c b/arch/x86/kernel/head64.c
index 4282dac694c3..fc55cc9ccb0f 100644
--- a/arch/x86/kernel/head64.c
+++ b/arch/x86/kernel/head64.c
@@ -40,6 +40,7 @@
#include <asm/desc.h>
#include <asm/extable.h>
#include <asm/trapnr.h>
+#include <asm/sev-es.h>
/*
* Manage page tables very early on.
@@ -540,12 +541,34 @@ static struct desc_ptr bringup_idt_descr = {
.address = 0, /* Set at runtime */
};
+static void set_bringup_idt_handler(gate_desc *idt, int n, void *handler)
+{
+#ifdef CONFIG_AMD_MEM_ENCRYPT
+ struct idt_data data;
+ gate_desc desc;
+
+ init_idt_data(&data, n, handler);
+ idt_init_desc(&desc, &data);
+ native_write_idt_entry(idt, n, &desc);
+#endif
+}
+
/* This runs while still in the direct mapping */
static void startup_64_load_idt(unsigned long physbase)
{
struct desc_ptr *desc = fixup_pointer(&bringup_idt_descr, physbase);
+ gate_desc *idt = fixup_pointer(bringup_idt_table, physbase);
+
+
+ if (IS_ENABLED(CONFIG_AMD_MEM_ENCRYPT)) {
+ void *handler;
+
+ /* VMM Communication Exception */
+ handler = fixup_pointer(vc_no_ghcb, physbase);
+ set_bringup_idt_handler(idt, X86_TRAP_VC, handler);
+ }
- desc->address = (unsigned long)fixup_pointer(bringup_idt_table, physbase);
+ desc->address = (unsigned long)idt;
native_load_idt(desc);
}
diff --git a/arch/x86/kernel/head_64.S b/arch/x86/kernel/head_64.S
index 3b40ec44a67d..6e68bca64ae4 100644
--- a/arch/x86/kernel/head_64.S
+++ b/arch/x86/kernel/head_64.S
@@ -348,6 +348,36 @@ SYM_CODE_START_LOCAL(early_idt_handler_common)
jmp restore_regs_and_return_to_kernel
SYM_CODE_END(early_idt_handler_common)
+#ifdef CONFIG_AMD_MEM_ENCRYPT
+/*
+ * VC Exception handler used during very early boot. The
+ * early_idt_handler_array can't be used because it returns via the
+ * paravirtualized INTERRUPT_RETURN and pv-ops don't work that early.
+ *
+ * This handler will end up in the .init.text section and not be
+ * available to boot secondary CPUs.
+ */
+SYM_CODE_START_NOALIGN(vc_no_ghcb)
+ UNWIND_HINT_IRET_REGS offset=8
+
+ /* Build pt_regs */
+ PUSH_AND_CLEAR_REGS
+
+ /* Call C handler */
+ movq %rsp, %rdi
+ movq ORIG_RAX(%rsp), %rsi
+ call do_vc_no_ghcb
+
+ /* Unwind pt_regs */
+ POP_REGS
+
+ /* Remove Error Code */
+ addq $8, %rsp
+
+ /* Pure iret required here - don't use INTERRUPT_RETURN */
+ iretq
+SYM_CODE_END(vc_no_ghcb)
+#endif
#define SYM_DATA_START_PAGE_ALIGNED(name) \
SYM_START(name, SYM_L_GLOBAL, .balign PAGE_SIZE)
--
2.28.0
_______________________________________________
Virtualization mailing list
Virtualization@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/virtualization
^ permalink raw reply related [flat|nested] 264+ messages in thread
* [PATCH v7.2 40/74] x86/sev-es: Setup GHCB based boot #VC handler
2020-09-07 13:15 ` Joerg Roedel
@ 2020-09-08 12:38 ` Joerg Roedel
-1 siblings, 0 replies; 264+ messages in thread
From: Joerg Roedel @ 2020-09-08 12:38 UTC (permalink / raw)
To: x86
Cc: Joerg Roedel, hpa, Andy Lutomirski, Dave Hansen, Peter Zijlstra,
Jiri Slaby, Dan Williams, Tom Lendacky, Juergen Gross, Kees Cook,
David Rientjes, Cfir Cohen, Erdem Aktas, Masami Hiramatsu,
Mike Stunes, Sean Christopherson, Martin Radev, linux-kernel,
kvm, virtualization
From: Joerg Roedel <jroedel@suse.de>
Add the infrastructure to handle #VC exceptions when the kernel runs
on virtual addresses and has a GHCB mapped. This handler will be used
until the runtime #VC handler takes over.
Since the handler runs very early, disable instrumentation for sev-es.c.
Signed-off-by: Joerg Roedel <jroedel@suse.de>
---
arch/x86/include/asm/realmode.h | 3 +
arch/x86/include/asm/segment.h | 2 +-
arch/x86/include/asm/sev-es.h | 2 +
arch/x86/kernel/Makefile | 2 +
arch/x86/kernel/head64.c | 8 +++
arch/x86/kernel/head_64.S | 36 ++++++++++
arch/x86/kernel/sev-es-shared.c | 14 ++--
arch/x86/kernel/sev-es.c | 116 ++++++++++++++++++++++++++++++++
arch/x86/mm/extable.c | 1 +
9 files changed, 176 insertions(+), 8 deletions(-)
diff --git a/arch/x86/include/asm/realmode.h b/arch/x86/include/asm/realmode.h
index b35030eeec36..96118fb041b8 100644
--- a/arch/x86/include/asm/realmode.h
+++ b/arch/x86/include/asm/realmode.h
@@ -57,6 +57,9 @@ extern unsigned char real_mode_blob_end[];
extern unsigned long initial_code;
extern unsigned long initial_gs;
extern unsigned long initial_stack;
+#ifdef CONFIG_AMD_MEM_ENCRYPT
+extern unsigned long initial_vc_handler;
+#endif
extern unsigned char real_mode_blob[];
extern unsigned char real_mode_relocs[];
diff --git a/arch/x86/include/asm/segment.h b/arch/x86/include/asm/segment.h
index 517920928989..7fdd4facfce7 100644
--- a/arch/x86/include/asm/segment.h
+++ b/arch/x86/include/asm/segment.h
@@ -226,7 +226,7 @@
#define NUM_EXCEPTION_VECTORS 32
/* Bitmask of exception vectors which push an error code on the stack: */
-#define EXCEPTION_ERRCODE_MASK 0x00027d00
+#define EXCEPTION_ERRCODE_MASK 0x20027d00
#define GDT_SIZE (GDT_ENTRIES*8)
#define GDT_ENTRY_TLS_ENTRIES 3
diff --git a/arch/x86/include/asm/sev-es.h b/arch/x86/include/asm/sev-es.h
index 7175d432ebfe..9fbeedaa66ee 100644
--- a/arch/x86/include/asm/sev-es.h
+++ b/arch/x86/include/asm/sev-es.h
@@ -75,5 +75,7 @@ static inline u64 lower_bits(u64 val, unsigned int bits)
/* Early IDT entry points for #VC handler */
extern void vc_no_ghcb(void);
+extern void vc_boot_ghcb(void);
+extern bool handle_vc_boot_ghcb(struct pt_regs *regs);
#endif
diff --git a/arch/x86/kernel/Makefile b/arch/x86/kernel/Makefile
index 3bcdd8d2bbdd..04ceea8f4a89 100644
--- a/arch/x86/kernel/Makefile
+++ b/arch/x86/kernel/Makefile
@@ -20,6 +20,7 @@ CFLAGS_REMOVE_kvmclock.o = -pg
CFLAGS_REMOVE_ftrace.o = -pg
CFLAGS_REMOVE_early_printk.o = -pg
CFLAGS_REMOVE_head64.o = -pg
+CFLAGS_REMOVE_sev-es.o = -pg
endif
KASAN_SANITIZE_head$(BITS).o := n
@@ -27,6 +28,7 @@ KASAN_SANITIZE_dumpstack.o := n
KASAN_SANITIZE_dumpstack_$(BITS).o := n
KASAN_SANITIZE_stacktrace.o := n
KASAN_SANITIZE_paravirt.o := n
+KASAN_SANITIZE_sev-es.o := n
# With some compiler versions the generated code results in boot hangs, caused
# by several compilation units. To be safe, disable all instrumentation.
diff --git a/arch/x86/kernel/head64.c b/arch/x86/kernel/head64.c
index fc55cc9ccb0f..4199f25c0063 100644
--- a/arch/x86/kernel/head64.c
+++ b/arch/x86/kernel/head64.c
@@ -406,6 +406,10 @@ void __init do_early_exception(struct pt_regs *regs, int trapnr)
early_make_pgtable(native_read_cr2()))
return;
+ if (IS_ENABLED(CONFIG_AMD_MEM_ENCRYPT) &&
+ trapnr == X86_TRAP_VC && handle_vc_boot_ghcb(regs))
+ return;
+
early_fixup_exception(regs, trapnr);
}
@@ -575,6 +579,10 @@ static void startup_64_load_idt(unsigned long physbase)
/* This is used when running on kernel addresses */
void early_setup_idt(void)
{
+ /* VMM Communication Exception */
+ if (IS_ENABLED(CONFIG_AMD_MEM_ENCRYPT))
+ set_bringup_idt_handler(bringup_idt_table, X86_TRAP_VC, vc_boot_ghcb);
+
bringup_idt_descr.address = (unsigned long)bringup_idt_table;
native_load_idt(&bringup_idt_descr);
}
diff --git a/arch/x86/kernel/head_64.S b/arch/x86/kernel/head_64.S
index 6e68bca64ae4..1a71d0d4d575 100644
--- a/arch/x86/kernel/head_64.S
+++ b/arch/x86/kernel/head_64.S
@@ -279,6 +279,39 @@ SYM_CODE_START(start_cpu0)
movq initial_stack(%rip), %rsp
jmp .Ljump_to_C_code
SYM_CODE_END(start_cpu0)
+#endif
+
+#ifdef CONFIG_AMD_MEM_ENCRYPT
+/*
+ * VC Exception handler used during early boot when running on kernel
+ * addresses, but before the switch to the idt_table can be made.
+ * The early_idt_handler_array can't be used here because it calls into a lot
+ * of __init code and this handler is also used during CPU offlining/onlining.
+ * Therefore this handler ends up in the .text section so that it stays around
+ * when .init.text is freed.
+ */
+SYM_CODE_START_NOALIGN(vc_boot_ghcb)
+ UNWIND_HINT_IRET_REGS offset=8
+
+ /* Build pt_regs */
+ PUSH_AND_CLEAR_REGS
+
+ /* Call C handler */
+ movq %rsp, %rdi
+ movq ORIG_RAX(%rsp), %rsi
+ movq initial_vc_handler(%rip), %rax
+ ANNOTATE_RETPOLINE_SAFE
+ call *%rax
+
+ /* Unwind pt_regs */
+ POP_REGS
+
+ /* Remove Error Code */
+ addq $8, %rsp
+
+ /* Pure iret required here - don't use INTERRUPT_RETURN */
+ iretq
+SYM_CODE_END(vc_boot_ghcb)
#endif
/* Both SMP bootup and ACPI suspend change these variables */
@@ -286,6 +319,9 @@ SYM_CODE_END(start_cpu0)
.balign 8
SYM_DATA(initial_code, .quad x86_64_start_kernel)
SYM_DATA(initial_gs, .quad INIT_PER_CPU_VAR(fixed_percpu_data))
+#ifdef CONFIG_AMD_MEM_ENCRYPT
+SYM_DATA(initial_vc_handler, .quad handle_vc_boot_ghcb)
+#endif
/*
* The SIZEOF_PTREGS gap is a convention which helps the in-kernel unwinder
diff --git a/arch/x86/kernel/sev-es-shared.c b/arch/x86/kernel/sev-es-shared.c
index 18619279a46f..aa77f2eb8d88 100644
--- a/arch/x86/kernel/sev-es-shared.c
+++ b/arch/x86/kernel/sev-es-shared.c
@@ -9,7 +9,7 @@
* and is included directly into both code-bases.
*/
-static void __maybe_unused sev_es_terminate(unsigned int reason)
+static void sev_es_terminate(unsigned int reason)
{
u64 val = GHCB_SEV_TERMINATE;
@@ -27,7 +27,7 @@ static void __maybe_unused sev_es_terminate(unsigned int reason)
asm volatile("hlt\n" : : : "memory");
}
-static bool __maybe_unused sev_es_negotiate_protocol(void)
+static bool sev_es_negotiate_protocol(void)
{
u64 val;
@@ -46,7 +46,7 @@ static bool __maybe_unused sev_es_negotiate_protocol(void)
return true;
}
-static void __maybe_unused vc_ghcb_invalidate(struct ghcb *ghcb)
+static void vc_ghcb_invalidate(struct ghcb *ghcb)
{
memset(ghcb->save.valid_bitmap, 0, sizeof(ghcb->save.valid_bitmap));
}
@@ -58,9 +58,9 @@ static bool vc_decoding_needed(unsigned long exit_code)
exit_code <= SVM_EXIT_LAST_EXCP);
}
-static enum es_result __maybe_unused vc_init_em_ctxt(struct es_em_ctxt *ctxt,
- struct pt_regs *regs,
- unsigned long exit_code)
+static enum es_result vc_init_em_ctxt(struct es_em_ctxt *ctxt,
+ struct pt_regs *regs,
+ unsigned long exit_code)
{
enum es_result ret = ES_OK;
@@ -73,7 +73,7 @@ static enum es_result __maybe_unused vc_init_em_ctxt(struct es_em_ctxt *ctxt,
return ret;
}
-static void __maybe_unused vc_finish_insn(struct es_em_ctxt *ctxt)
+static void vc_finish_insn(struct es_em_ctxt *ctxt)
{
ctxt->regs->ip += ctxt->insn.length;
}
diff --git a/arch/x86/kernel/sev-es.c b/arch/x86/kernel/sev-es.c
index 0b698b653c0b..bb3e702a71eb 100644
--- a/arch/x86/kernel/sev-es.c
+++ b/arch/x86/kernel/sev-es.c
@@ -7,7 +7,9 @@
* Author: Joerg Roedel <jroedel@suse.de>
*/
+#include <linux/sched/debug.h> /* For show_regs() */
#include <linux/kernel.h>
+#include <linux/printk.h>
#include <linux/mm.h>
#include <asm/sev-es.h>
@@ -18,6 +20,18 @@
#include <asm/trapnr.h>
#include <asm/svm.h>
+/* For early boot hypervisor communication in SEV-ES enabled guests */
+static struct ghcb boot_ghcb_page __bss_decrypted __aligned(PAGE_SIZE);
+
+/*
+ * Needs to be in the .data section because we need it NULL before bss is
+ * cleared
+ */
+static struct ghcb __initdata *boot_ghcb;
+
+/* Needed in vc_early_forward_exception */
+void do_early_exception(struct pt_regs *regs, int trapnr);
+
static inline u64 sev_es_rd_ghcb_msr(void)
{
return native_read_msr(MSR_AMD64_SEV_ES_GHCB);
@@ -161,3 +175,105 @@ static enum es_result vc_read_mem(struct es_em_ctxt *ctxt,
/* Include code shared with pre-decompression boot stage */
#include "sev-es-shared.c"
+
+/*
+ * This function runs on the first #VC exception after the kernel
+ * switched to virtual addresses.
+ */
+static bool __init sev_es_setup_ghcb(void)
+{
+ /* First make sure the hypervisor talks a supported protocol. */
+ if (!sev_es_negotiate_protocol())
+ return false;
+
+ /*
+ * Clear the boot_ghcb. The first exception comes in before the bss
+ * section is cleared.
+ */
+ memset(&boot_ghcb_page, 0, PAGE_SIZE);
+
+ /* Alright - Make the boot-ghcb public */
+ boot_ghcb = &boot_ghcb_page;
+
+ return true;
+}
+
+static void __init vc_early_forward_exception(struct es_em_ctxt *ctxt)
+{
+ int trapnr = ctxt->fi.vector;
+
+ if (trapnr == X86_TRAP_PF)
+ native_write_cr2(ctxt->fi.cr2);
+
+ ctxt->regs->orig_ax = ctxt->fi.error_code;
+ do_early_exception(ctxt->regs, trapnr);
+}
+
+static enum es_result vc_handle_exitcode(struct es_em_ctxt *ctxt,
+ struct ghcb *ghcb,
+ unsigned long exit_code)
+{
+ enum es_result result;
+
+ switch (exit_code) {
+ default:
+ /*
+ * Unexpected #VC exception
+ */
+ result = ES_UNSUPPORTED;
+ }
+
+ return result;
+}
+
+bool __init handle_vc_boot_ghcb(struct pt_regs *regs)
+{
+ unsigned long exit_code = regs->orig_ax;
+ struct es_em_ctxt ctxt;
+ enum es_result result;
+
+ /* Do initial setup or terminate the guest */
+ if (unlikely(boot_ghcb == NULL && !sev_es_setup_ghcb()))
+ sev_es_terminate(GHCB_SEV_ES_REASON_GENERAL_REQUEST);
+
+ vc_ghcb_invalidate(boot_ghcb);
+
+ result = vc_init_em_ctxt(&ctxt, regs, exit_code);
+ if (result == ES_OK)
+ result = vc_handle_exitcode(&ctxt, boot_ghcb, exit_code);
+
+ /* Done - now check the result */
+ switch (result) {
+ case ES_OK:
+ vc_finish_insn(&ctxt);
+ break;
+ case ES_UNSUPPORTED:
+ early_printk("PANIC: Unsupported exit-code 0x%02lx in early #VC exception (IP: 0x%lx)\n",
+ exit_code, regs->ip);
+ goto fail;
+ case ES_VMM_ERROR:
+ early_printk("PANIC: Failure in communication with VMM (exit-code 0x%02lx IP: 0x%lx)\n",
+ exit_code, regs->ip);
+ goto fail;
+ case ES_DECODE_FAILED:
+ early_printk("PANIC: Failed to decode instruction (exit-code 0x%02lx IP: 0x%lx)\n",
+ exit_code, regs->ip);
+ goto fail;
+ case ES_EXCEPTION:
+ vc_early_forward_exception(&ctxt);
+ break;
+ case ES_RETRY:
+ /* Nothing to do */
+ break;
+ default:
+ BUG();
+ }
+
+ return true;
+
+fail:
+ show_regs(regs);
+
+ while (true)
+ halt();
+}
diff --git a/arch/x86/mm/extable.c b/arch/x86/mm/extable.c
index 1d6cb07f4f86..3966749d07ac 100644
--- a/arch/x86/mm/extable.c
+++ b/arch/x86/mm/extable.c
@@ -5,6 +5,7 @@
#include <xen/xen.h>
#include <asm/fpu/internal.h>
+#include <asm/sev-es.h>
#include <asm/traps.h>
#include <asm/kdebug.h>
--
2.28.0
^ permalink raw reply related [flat|nested] 264+ messages in thread
* [PATCH v7.2 40/74] x86/sev-es: Setup GHCB based boot #VC handler
@ 2020-09-08 12:38 ` Joerg Roedel
0 siblings, 0 replies; 264+ messages in thread
From: Joerg Roedel @ 2020-09-08 12:38 UTC (permalink / raw)
To: x86
Cc: Juergen Gross, Tom Lendacky, Joerg Roedel, Mike Stunes,
Kees Cook, kvm, Peter Zijlstra, Cfir Cohen, Dave Hansen,
linux-kernel, Sean Christopherson, virtualization, Martin Radev,
Masami Hiramatsu, Andy Lutomirski, hpa, Erdem Aktas,
David Rientjes, Dan Williams, Jiri Slaby
From: Joerg Roedel <jroedel@suse.de>
Add the infrastructure to handle #VC exceptions when the kernel runs
on virtual addresses and has a GHCB mapped. This handler will be used
until the runtime #VC handler takes over.
Since the handler runs very early, disable instrumentation for sev-es.c.
Signed-off-by: Joerg Roedel <jroedel@suse.de>
---
arch/x86/include/asm/realmode.h | 3 +
arch/x86/include/asm/segment.h | 2 +-
arch/x86/include/asm/sev-es.h | 2 +
arch/x86/kernel/Makefile | 2 +
arch/x86/kernel/head64.c | 8 +++
arch/x86/kernel/head_64.S | 36 ++++++++++
arch/x86/kernel/sev-es-shared.c | 14 ++--
arch/x86/kernel/sev-es.c | 116 ++++++++++++++++++++++++++++++++
arch/x86/mm/extable.c | 1 +
9 files changed, 176 insertions(+), 8 deletions(-)
diff --git a/arch/x86/include/asm/realmode.h b/arch/x86/include/asm/realmode.h
index b35030eeec36..96118fb041b8 100644
--- a/arch/x86/include/asm/realmode.h
+++ b/arch/x86/include/asm/realmode.h
@@ -57,6 +57,9 @@ extern unsigned char real_mode_blob_end[];
extern unsigned long initial_code;
extern unsigned long initial_gs;
extern unsigned long initial_stack;
+#ifdef CONFIG_AMD_MEM_ENCRYPT
+extern unsigned long initial_vc_handler;
+#endif
extern unsigned char real_mode_blob[];
extern unsigned char real_mode_relocs[];
diff --git a/arch/x86/include/asm/segment.h b/arch/x86/include/asm/segment.h
index 517920928989..7fdd4facfce7 100644
--- a/arch/x86/include/asm/segment.h
+++ b/arch/x86/include/asm/segment.h
@@ -226,7 +226,7 @@
#define NUM_EXCEPTION_VECTORS 32
/* Bitmask of exception vectors which push an error code on the stack: */
-#define EXCEPTION_ERRCODE_MASK 0x00027d00
+#define EXCEPTION_ERRCODE_MASK 0x20027d00
#define GDT_SIZE (GDT_ENTRIES*8)
#define GDT_ENTRY_TLS_ENTRIES 3
diff --git a/arch/x86/include/asm/sev-es.h b/arch/x86/include/asm/sev-es.h
index 7175d432ebfe..9fbeedaa66ee 100644
--- a/arch/x86/include/asm/sev-es.h
+++ b/arch/x86/include/asm/sev-es.h
@@ -75,5 +75,7 @@ static inline u64 lower_bits(u64 val, unsigned int bits)
/* Early IDT entry points for #VC handler */
extern void vc_no_ghcb(void);
+extern void vc_boot_ghcb(void);
+extern bool handle_vc_boot_ghcb(struct pt_regs *regs);
#endif
diff --git a/arch/x86/kernel/Makefile b/arch/x86/kernel/Makefile
index 3bcdd8d2bbdd..04ceea8f4a89 100644
--- a/arch/x86/kernel/Makefile
+++ b/arch/x86/kernel/Makefile
@@ -20,6 +20,7 @@ CFLAGS_REMOVE_kvmclock.o = -pg
CFLAGS_REMOVE_ftrace.o = -pg
CFLAGS_REMOVE_early_printk.o = -pg
CFLAGS_REMOVE_head64.o = -pg
+CFLAGS_REMOVE_sev-es.o = -pg
endif
KASAN_SANITIZE_head$(BITS).o := n
@@ -27,6 +28,7 @@ KASAN_SANITIZE_dumpstack.o := n
KASAN_SANITIZE_dumpstack_$(BITS).o := n
KASAN_SANITIZE_stacktrace.o := n
KASAN_SANITIZE_paravirt.o := n
+KASAN_SANITIZE_sev-es.o := n
# With some compiler versions the generated code results in boot hangs, caused
# by several compilation units. To be safe, disable all instrumentation.
diff --git a/arch/x86/kernel/head64.c b/arch/x86/kernel/head64.c
index fc55cc9ccb0f..4199f25c0063 100644
--- a/arch/x86/kernel/head64.c
+++ b/arch/x86/kernel/head64.c
@@ -406,6 +406,10 @@ void __init do_early_exception(struct pt_regs *regs, int trapnr)
early_make_pgtable(native_read_cr2()))
return;
+ if (IS_ENABLED(CONFIG_AMD_MEM_ENCRYPT) &&
+ trapnr == X86_TRAP_VC && handle_vc_boot_ghcb(regs))
+ return;
+
early_fixup_exception(regs, trapnr);
}
@@ -575,6 +579,10 @@ static void startup_64_load_idt(unsigned long physbase)
/* This is used when running on kernel addresses */
void early_setup_idt(void)
{
+ /* VMM Communication Exception */
+ if (IS_ENABLED(CONFIG_AMD_MEM_ENCRYPT))
+ set_bringup_idt_handler(bringup_idt_table, X86_TRAP_VC, vc_boot_ghcb);
+
bringup_idt_descr.address = (unsigned long)bringup_idt_table;
native_load_idt(&bringup_idt_descr);
}
diff --git a/arch/x86/kernel/head_64.S b/arch/x86/kernel/head_64.S
index 6e68bca64ae4..1a71d0d4d575 100644
--- a/arch/x86/kernel/head_64.S
+++ b/arch/x86/kernel/head_64.S
@@ -279,6 +279,39 @@ SYM_CODE_START(start_cpu0)
movq initial_stack(%rip), %rsp
jmp .Ljump_to_C_code
SYM_CODE_END(start_cpu0)
+#endif
+
+#ifdef CONFIG_AMD_MEM_ENCRYPT
+/*
+ * VC Exception handler used during early boot when running on kernel
+ * addresses, but before the switch to the idt_table can be made.
+ * The early_idt_handler_array can't be used here because it calls into a lot
+ * of __init code and this handler is also used during CPU offlining/onlining.
+ * Therefore this handler ends up in the .text section so that it stays around
+ * when .init.text is freed.
+ */
+SYM_CODE_START_NOALIGN(vc_boot_ghcb)
+ UNWIND_HINT_IRET_REGS offset=8
+
+ /* Build pt_regs */
+ PUSH_AND_CLEAR_REGS
+
+ /* Call C handler */
+ movq %rsp, %rdi
+ movq ORIG_RAX(%rsp), %rsi
+ movq initial_vc_handler(%rip), %rax
+ ANNOTATE_RETPOLINE_SAFE
+ call *%rax
+
+ /* Unwind pt_regs */
+ POP_REGS
+
+ /* Remove Error Code */
+ addq $8, %rsp
+
+ /* Pure iret required here - don't use INTERRUPT_RETURN */
+ iretq
+SYM_CODE_END(vc_boot_ghcb)
#endif
/* Both SMP bootup and ACPI suspend change these variables */
@@ -286,6 +319,9 @@ SYM_CODE_END(start_cpu0)
.balign 8
SYM_DATA(initial_code, .quad x86_64_start_kernel)
SYM_DATA(initial_gs, .quad INIT_PER_CPU_VAR(fixed_percpu_data))
+#ifdef CONFIG_AMD_MEM_ENCRYPT
+SYM_DATA(initial_vc_handler, .quad handle_vc_boot_ghcb)
+#endif
/*
* The SIZEOF_PTREGS gap is a convention which helps the in-kernel unwinder
diff --git a/arch/x86/kernel/sev-es-shared.c b/arch/x86/kernel/sev-es-shared.c
index 18619279a46f..aa77f2eb8d88 100644
--- a/arch/x86/kernel/sev-es-shared.c
+++ b/arch/x86/kernel/sev-es-shared.c
@@ -9,7 +9,7 @@
* and is included directly into both code-bases.
*/
-static void __maybe_unused sev_es_terminate(unsigned int reason)
+static void sev_es_terminate(unsigned int reason)
{
u64 val = GHCB_SEV_TERMINATE;
@@ -27,7 +27,7 @@ static void __maybe_unused sev_es_terminate(unsigned int reason)
asm volatile("hlt\n" : : : "memory");
}
-static bool __maybe_unused sev_es_negotiate_protocol(void)
+static bool sev_es_negotiate_protocol(void)
{
u64 val;
@@ -46,7 +46,7 @@ static bool __maybe_unused sev_es_negotiate_protocol(void)
return true;
}
-static void __maybe_unused vc_ghcb_invalidate(struct ghcb *ghcb)
+static void vc_ghcb_invalidate(struct ghcb *ghcb)
{
memset(ghcb->save.valid_bitmap, 0, sizeof(ghcb->save.valid_bitmap));
}
@@ -58,9 +58,9 @@ static bool vc_decoding_needed(unsigned long exit_code)
exit_code <= SVM_EXIT_LAST_EXCP);
}
-static enum es_result __maybe_unused vc_init_em_ctxt(struct es_em_ctxt *ctxt,
- struct pt_regs *regs,
- unsigned long exit_code)
+static enum es_result vc_init_em_ctxt(struct es_em_ctxt *ctxt,
+ struct pt_regs *regs,
+ unsigned long exit_code)
{
enum es_result ret = ES_OK;
@@ -73,7 +73,7 @@ static enum es_result __maybe_unused vc_init_em_ctxt(struct es_em_ctxt *ctxt,
return ret;
}
-static void __maybe_unused vc_finish_insn(struct es_em_ctxt *ctxt)
+static void vc_finish_insn(struct es_em_ctxt *ctxt)
{
ctxt->regs->ip += ctxt->insn.length;
}
diff --git a/arch/x86/kernel/sev-es.c b/arch/x86/kernel/sev-es.c
index 0b698b653c0b..bb3e702a71eb 100644
--- a/arch/x86/kernel/sev-es.c
+++ b/arch/x86/kernel/sev-es.c
@@ -7,7 +7,9 @@
* Author: Joerg Roedel <jroedel@suse.de>
*/
+#include <linux/sched/debug.h> /* For show_regs() */
#include <linux/kernel.h>
+#include <linux/printk.h>
#include <linux/mm.h>
#include <asm/sev-es.h>
@@ -18,6 +20,18 @@
#include <asm/trapnr.h>
#include <asm/svm.h>
+/* For early boot hypervisor communication in SEV-ES enabled guests */
+static struct ghcb boot_ghcb_page __bss_decrypted __aligned(PAGE_SIZE);
+
+/*
+ * Needs to be in the .data section because we need it NULL before bss is
+ * cleared
+ */
+static struct ghcb __initdata *boot_ghcb;
+
+/* Needed in vc_early_forward_exception */
+void do_early_exception(struct pt_regs *regs, int trapnr);
+
static inline u64 sev_es_rd_ghcb_msr(void)
{
return native_read_msr(MSR_AMD64_SEV_ES_GHCB);
@@ -161,3 +175,105 @@ static enum es_result vc_read_mem(struct es_em_ctxt *ctxt,
/* Include code shared with pre-decompression boot stage */
#include "sev-es-shared.c"
+
+/*
+ * This function runs on the first #VC exception after the kernel
+ * switched to virtual addresses.
+ */
+static bool __init sev_es_setup_ghcb(void)
+{
+ /* First make sure the hypervisor talks a supported protocol. */
+ if (!sev_es_negotiate_protocol())
+ return false;
+
+ /*
+ * Clear the boot_ghcb. The first exception comes in before the bss
+ * section is cleared.
+ */
+ memset(&boot_ghcb_page, 0, PAGE_SIZE);
+
+ /* Alright - Make the boot-ghcb public */
+ boot_ghcb = &boot_ghcb_page;
+
+ return true;
+}
+
+static void __init vc_early_forward_exception(struct es_em_ctxt *ctxt)
+{
+ int trapnr = ctxt->fi.vector;
+
+ if (trapnr == X86_TRAP_PF)
+ native_write_cr2(ctxt->fi.cr2);
+
+ ctxt->regs->orig_ax = ctxt->fi.error_code;
+ do_early_exception(ctxt->regs, trapnr);
+}
+
+static enum es_result vc_handle_exitcode(struct es_em_ctxt *ctxt,
+ struct ghcb *ghcb,
+ unsigned long exit_code)
+{
+ enum es_result result;
+
+ switch (exit_code) {
+ default:
+ /*
+ * Unexpected #VC exception
+ */
+ result = ES_UNSUPPORTED;
+ }
+
+ return result;
+}
+
+bool __init handle_vc_boot_ghcb(struct pt_regs *regs)
+{
+ unsigned long exit_code = regs->orig_ax;
+ struct es_em_ctxt ctxt;
+ enum es_result result;
+
+ /* Do initial setup or terminate the guest */
+ if (unlikely(boot_ghcb == NULL && !sev_es_setup_ghcb()))
+ sev_es_terminate(GHCB_SEV_ES_REASON_GENERAL_REQUEST);
+
+ vc_ghcb_invalidate(boot_ghcb);
+
+ result = vc_init_em_ctxt(&ctxt, regs, exit_code);
+ if (result == ES_OK)
+ result = vc_handle_exitcode(&ctxt, boot_ghcb, exit_code);
+
+ /* Done - now check the result */
+ switch (result) {
+ case ES_OK:
+ vc_finish_insn(&ctxt);
+ break;
+ case ES_UNSUPPORTED:
+ early_printk("PANIC: Unsupported exit-code 0x%02lx in early #VC exception (IP: 0x%lx)\n",
+ exit_code, regs->ip);
+ goto fail;
+ case ES_VMM_ERROR:
+ early_printk("PANIC: Failure in communication with VMM (exit-code 0x%02lx IP: 0x%lx)\n",
+ exit_code, regs->ip);
+ goto fail;
+ case ES_DECODE_FAILED:
+ early_printk("PANIC: Failed to decode instruction (exit-code 0x%02lx IP: 0x%lx)\n",
+ exit_code, regs->ip);
+ goto fail;
+ case ES_EXCEPTION:
+ vc_early_forward_exception(&ctxt);
+ break;
+ case ES_RETRY:
+ /* Nothing to do */
+ break;
+ default:
+ BUG();
+ }
+
+ return true;
+
+fail:
+ show_regs(regs);
+
+ while (true)
+ halt();
+}
diff --git a/arch/x86/mm/extable.c b/arch/x86/mm/extable.c
index 1d6cb07f4f86..3966749d07ac 100644
--- a/arch/x86/mm/extable.c
+++ b/arch/x86/mm/extable.c
@@ -5,6 +5,7 @@
#include <xen/xen.h>
#include <asm/fpu/internal.h>
+#include <asm/sev-es.h>
#include <asm/traps.h>
#include <asm/kdebug.h>
--
2.28.0
_______________________________________________
Virtualization mailing list
Virtualization@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/virtualization
^ permalink raw reply related [flat|nested] 264+ messages in thread
* Re: [PATCH v7 67/72] x86/smpboot: Load TSS and getcpu GDT entry before loading IDT
2020-09-07 13:16 ` Joerg Roedel
@ 2020-09-08 17:20 ` Borislav Petkov
-1 siblings, 0 replies; 264+ messages in thread
From: Borislav Petkov @ 2020-09-08 17:20 UTC (permalink / raw)
To: Joerg Roedel
Cc: x86, Joerg Roedel, hpa, Andy Lutomirski, Dave Hansen,
Peter Zijlstra, Jiri Slaby, Dan Williams, Tom Lendacky,
Juergen Gross, Kees Cook, David Rientjes, Cfir Cohen,
Erdem Aktas, Masami Hiramatsu, Mike Stunes, Sean Christopherson,
Martin Radev, linux-kernel, kvm, virtualization
On Mon, Sep 07, 2020 at 03:16:08PM +0200, Joerg Roedel wrote:
> From: Joerg Roedel <jroedel@suse.de>
>
> The IDT on 64bit contains vectors which use paranoid_entry() and/or IST
> stacks. To make these vectors work the TSS and the getcpu GDT entry need
> to be set up before the IDT is loaded.
>
> Signed-off-by: Joerg Roedel <jroedel@suse.de>
> ---
> arch/x86/include/asm/processor.h | 1 +
> arch/x86/kernel/cpu/common.c | 23 +++++++++++++++++++++++
> arch/x86/kernel/smpboot.c | 2 +-
> 3 files changed, 25 insertions(+), 1 deletion(-)
>
> diff --git a/arch/x86/include/asm/processor.h b/arch/x86/include/asm/processor.h
> index d8a82e650810..5ac507586769 100644
> --- a/arch/x86/include/asm/processor.h
> +++ b/arch/x86/include/asm/processor.h
> @@ -696,6 +696,7 @@ extern void load_direct_gdt(int);
> extern void load_fixmap_gdt(int);
> extern void load_percpu_segment(int);
> extern void cpu_init(void);
> +extern void cpu_init_exception_handling(void);
> extern void cr4_init(void);
>
> static inline unsigned long get_debugctlmsr(void)
> diff --git a/arch/x86/kernel/cpu/common.c b/arch/x86/kernel/cpu/common.c
> index 1d65365363a1..a9527c0c38fb 100644
> --- a/arch/x86/kernel/cpu/common.c
> +++ b/arch/x86/kernel/cpu/common.c
> @@ -1854,6 +1854,29 @@ static inline void tss_setup_io_bitmap(struct tss_struct *tss)
> #endif
> }
>
> +/*
> + * Setup everything needed to handle exceptions from the IDT, including the IST
> + * exceptions which use paranoid_entry()
> + */
> +void cpu_init_exception_handling(void)
> +{
> + struct tss_struct *tss = this_cpu_ptr(&cpu_tss_rw);
> + int cpu = raw_smp_processor_id();
> +
> + /* paranoid_entry() gets the CPU number from the GDT */
> + setup_getcpu(cpu);
> +
> + /* IST vectors need TSS to be set up. */
> + tss_setup_ist(tss);
> + tss_setup_io_bitmap(tss);
> + set_tss_desc(cpu, &get_cpu_entry_area(cpu)->tss.x86_tss);
> +
> + load_TR_desc();
Aha, this is what you mean here in your 0th message. I'm guessing it is
ok to do those things twice in start_secondary...
--
Regards/Gruss,
Boris.
https://people.kernel.org/tglx/notes-about-netiquette
^ permalink raw reply [flat|nested] 264+ messages in thread
* Re: [PATCH v7 67/72] x86/smpboot: Load TSS and getcpu GDT entry before loading IDT
@ 2020-09-08 17:20 ` Borislav Petkov
0 siblings, 0 replies; 264+ messages in thread
From: Borislav Petkov @ 2020-09-08 17:20 UTC (permalink / raw)
To: Joerg Roedel
Cc: Juergen Gross, Tom Lendacky, Dave Hansen, Mike Stunes, Kees Cook,
kvm, Peter Zijlstra, Cfir Cohen, x86, linux-kernel,
Sean Christopherson, virtualization, Martin Radev, Joerg Roedel,
Masami Hiramatsu, Andy Lutomirski, hpa, Erdem Aktas,
David Rientjes, Dan Williams, Jiri Slaby
On Mon, Sep 07, 2020 at 03:16:08PM +0200, Joerg Roedel wrote:
> From: Joerg Roedel <jroedel@suse.de>
>
> The IDT on 64bit contains vectors which use paranoid_entry() and/or IST
> stacks. To make these vectors work the TSS and the getcpu GDT entry need
> to be set up before the IDT is loaded.
>
> Signed-off-by: Joerg Roedel <jroedel@suse.de>
> ---
> arch/x86/include/asm/processor.h | 1 +
> arch/x86/kernel/cpu/common.c | 23 +++++++++++++++++++++++
> arch/x86/kernel/smpboot.c | 2 +-
> 3 files changed, 25 insertions(+), 1 deletion(-)
>
> diff --git a/arch/x86/include/asm/processor.h b/arch/x86/include/asm/processor.h
> index d8a82e650810..5ac507586769 100644
> --- a/arch/x86/include/asm/processor.h
> +++ b/arch/x86/include/asm/processor.h
> @@ -696,6 +696,7 @@ extern void load_direct_gdt(int);
> extern void load_fixmap_gdt(int);
> extern void load_percpu_segment(int);
> extern void cpu_init(void);
> +extern void cpu_init_exception_handling(void);
> extern void cr4_init(void);
>
> static inline unsigned long get_debugctlmsr(void)
> diff --git a/arch/x86/kernel/cpu/common.c b/arch/x86/kernel/cpu/common.c
> index 1d65365363a1..a9527c0c38fb 100644
> --- a/arch/x86/kernel/cpu/common.c
> +++ b/arch/x86/kernel/cpu/common.c
> @@ -1854,6 +1854,29 @@ static inline void tss_setup_io_bitmap(struct tss_struct *tss)
> #endif
> }
>
> +/*
> + * Setup everything needed to handle exceptions from the IDT, including the IST
> + * exceptions which use paranoid_entry()
> + */
> +void cpu_init_exception_handling(void)
> +{
> + struct tss_struct *tss = this_cpu_ptr(&cpu_tss_rw);
> + int cpu = raw_smp_processor_id();
> +
> + /* paranoid_entry() gets the CPU number from the GDT */
> + setup_getcpu(cpu);
> +
> + /* IST vectors need TSS to be set up. */
> + tss_setup_ist(tss);
> + tss_setup_io_bitmap(tss);
> + set_tss_desc(cpu, &get_cpu_entry_area(cpu)->tss.x86_tss);
> +
> + load_TR_desc();
Aha, this is what you mean here in your 0th message. I'm guessing it is
ok to do those things twice in start_secondary...
--
Regards/Gruss,
Boris.
https://people.kernel.org/tglx/notes-about-netiquette
_______________________________________________
Virtualization mailing list
Virtualization@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/virtualization
^ permalink raw reply [flat|nested] 264+ messages in thread
* Re: [PATCH v7 71/72] x86/efi: Add GHCB mappings when SEV-ES is active
2020-09-07 13:16 ` Joerg Roedel
@ 2020-09-08 17:46 ` Borislav Petkov
-1 siblings, 0 replies; 264+ messages in thread
From: Borislav Petkov @ 2020-09-08 17:46 UTC (permalink / raw)
To: Joerg Roedel, Ard Biesheuvel
Cc: x86, Joerg Roedel, Tom Lendacky, hpa, Andy Lutomirski,
Dave Hansen, Peter Zijlstra, Jiri Slaby, Dan Williams,
Juergen Gross, Kees Cook, David Rientjes, Cfir Cohen,
Erdem Aktas, Masami Hiramatsu, Mike Stunes, Sean Christopherson,
Martin Radev, linux-kernel, kvm, virtualization
+ Ard so that he can ack the efi bits.
On Mon, Sep 07, 2020 at 03:16:12PM +0200, Joerg Roedel wrote:
> From: Tom Lendacky <thomas.lendacky@amd.com>
>
> Calling down to EFI runtime services can result in the firmware performing
> VMGEXIT calls. The firmware is likely to use the GHCB of the OS (e.g., for
> setting EFI variables), so each GHCB in the system needs to be identity
> mapped in the EFI page tables, as unencrypted, to avoid page faults.
>
> Signed-off-by: Tom Lendacky <thomas.lendacky@amd.com>
> [ jroedel@suse.de: Moved GHCB mapping loop to sev-es.c ]
> Signed-off-by: Joerg Roedel <jroedel@suse.de>
> ---
> arch/x86/boot/compressed/sev-es.c | 1 +
> arch/x86/include/asm/sev-es.h | 2 ++
> arch/x86/kernel/sev-es.c | 30 ++++++++++++++++++++++++++++++
> arch/x86/platform/efi/efi_64.c | 10 ++++++++++
> 4 files changed, 43 insertions(+)
>
> diff --git a/arch/x86/boot/compressed/sev-es.c b/arch/x86/boot/compressed/sev-es.c
> index 45702b866c33..0a9a248ca33d 100644
> --- a/arch/x86/boot/compressed/sev-es.c
> +++ b/arch/x86/boot/compressed/sev-es.c
> @@ -12,6 +12,7 @@
> */
> #include "misc.h"
>
> +#include <asm/pgtable_types.h>
> #include <asm/sev-es.h>
> #include <asm/trapnr.h>
> #include <asm/trap_pf.h>
> diff --git a/arch/x86/include/asm/sev-es.h b/arch/x86/include/asm/sev-es.h
> index e919f09ae33c..cf1d957c7091 100644
> --- a/arch/x86/include/asm/sev-es.h
> +++ b/arch/x86/include/asm/sev-es.h
> @@ -102,11 +102,13 @@ static __always_inline void sev_es_nmi_complete(void)
> if (static_branch_unlikely(&sev_es_enable_key))
> __sev_es_nmi_complete();
> }
> +extern int __init sev_es_efi_map_ghcbs(pgd_t *pgd);
> #else
> static inline void sev_es_ist_enter(struct pt_regs *regs) { }
> static inline void sev_es_ist_exit(void) { }
> static inline int sev_es_setup_ap_jump_table(struct real_mode_header *rmh) { return 0; }
> static inline void sev_es_nmi_complete(void) { }
> +static inline int sev_es_efi_map_ghcbs(pgd_t *pgd) { return 0; }
> #endif
>
> #endif
> diff --git a/arch/x86/kernel/sev-es.c b/arch/x86/kernel/sev-es.c
> index 9ab3a4dfecd8..4e2b7e4d9b87 100644
> --- a/arch/x86/kernel/sev-es.c
> +++ b/arch/x86/kernel/sev-es.c
> @@ -491,6 +491,36 @@ int sev_es_setup_ap_jump_table(struct real_mode_header *rmh)
> return 0;
> }
>
> +/*
> + * This is needed by the OVMF UEFI firmware which will use whatever it finds in
> + * the GHCB MSR as its GHCB to talk to the hypervisor. So make sure the per-cpu
> + * runtime GHCBs used by the kernel are also mapped in the EFI page-table.
> + */
> +int __init sev_es_efi_map_ghcbs(pgd_t *pgd)
> +{
> + struct sev_es_runtime_data *data;
> + unsigned long address, pflags;
> + int cpu;
> + u64 pfn;
> +
> + if (!sev_es_active())
> + return 0;
> +
> + pflags = _PAGE_NX | _PAGE_RW;
> +
> + for_each_possible_cpu(cpu) {
> + data = per_cpu(runtime_data, cpu);
> +
> + address = __pa(&data->ghcb_page);
> + pfn = address >> PAGE_SHIFT;
> +
> + if (kernel_map_pages_in_pgd(pgd, pfn, address, 1, pflags))
> + return 1;
> + }
> +
> + return 0;
> +}
> +
> static enum es_result vc_handle_msr(struct ghcb *ghcb, struct es_em_ctxt *ctxt)
> {
> struct pt_regs *regs = ctxt->regs;
> diff --git a/arch/x86/platform/efi/efi_64.c b/arch/x86/platform/efi/efi_64.c
> index 6af4da1149ba..8f5759df7776 100644
> --- a/arch/x86/platform/efi/efi_64.c
> +++ b/arch/x86/platform/efi/efi_64.c
> @@ -47,6 +47,7 @@
> #include <asm/realmode.h>
> #include <asm/time.h>
> #include <asm/pgalloc.h>
> +#include <asm/sev-es.h>
>
> /*
> * We allocate runtime services regions top-down, starting from -4G, i.e.
> @@ -229,6 +230,15 @@ int __init efi_setup_page_tables(unsigned long pa_memmap, unsigned num_pages)
> return 1;
> }
>
> + /*
> + * When SEV-ES is active, the GHCB as set by the kernel will be used
> + * by firmware. Create a 1:1 unencrypted mapping for each GHCB.
> + */
> + if (sev_es_efi_map_ghcbs(pgd)) {
> + pr_err("Failed to create 1:1 mapping for the GHCBs!\n");
> + return 1;
> + }
> +
> /*
> * When making calls to the firmware everything needs to be 1:1
> * mapped and addressable with 32-bit pointers. Map the kernel
> --
> 2.28.0
>
--
Regards/Gruss,
Boris.
https://people.kernel.org/tglx/notes-about-netiquette
^ permalink raw reply [flat|nested] 264+ messages in thread
* Re: [PATCH v7 71/72] x86/efi: Add GHCB mappings when SEV-ES is active
@ 2020-09-08 17:46 ` Borislav Petkov
0 siblings, 0 replies; 264+ messages in thread
From: Borislav Petkov @ 2020-09-08 17:46 UTC (permalink / raw)
To: Joerg Roedel, Ard Biesheuvel
Cc: Juergen Gross, Tom Lendacky, Dave Hansen, Mike Stunes, Kees Cook,
kvm, Peter Zijlstra, Cfir Cohen, x86, linux-kernel,
Sean Christopherson, virtualization, Martin Radev, Joerg Roedel,
Masami Hiramatsu, Andy Lutomirski, hpa, Erdem Aktas,
David Rientjes, Dan Williams, Jiri Slaby
+ Ard so that he can ack the efi bits.
On Mon, Sep 07, 2020 at 03:16:12PM +0200, Joerg Roedel wrote:
> From: Tom Lendacky <thomas.lendacky@amd.com>
>
> Calling down to EFI runtime services can result in the firmware performing
> VMGEXIT calls. The firmware is likely to use the GHCB of the OS (e.g., for
> setting EFI variables), so each GHCB in the system needs to be identity
> mapped in the EFI page tables, as unencrypted, to avoid page faults.
>
> Signed-off-by: Tom Lendacky <thomas.lendacky@amd.com>
> [ jroedel@suse.de: Moved GHCB mapping loop to sev-es.c ]
> Signed-off-by: Joerg Roedel <jroedel@suse.de>
> ---
> arch/x86/boot/compressed/sev-es.c | 1 +
> arch/x86/include/asm/sev-es.h | 2 ++
> arch/x86/kernel/sev-es.c | 30 ++++++++++++++++++++++++++++++
> arch/x86/platform/efi/efi_64.c | 10 ++++++++++
> 4 files changed, 43 insertions(+)
>
> diff --git a/arch/x86/boot/compressed/sev-es.c b/arch/x86/boot/compressed/sev-es.c
> index 45702b866c33..0a9a248ca33d 100644
> --- a/arch/x86/boot/compressed/sev-es.c
> +++ b/arch/x86/boot/compressed/sev-es.c
> @@ -12,6 +12,7 @@
> */
> #include "misc.h"
>
> +#include <asm/pgtable_types.h>
> #include <asm/sev-es.h>
> #include <asm/trapnr.h>
> #include <asm/trap_pf.h>
> diff --git a/arch/x86/include/asm/sev-es.h b/arch/x86/include/asm/sev-es.h
> index e919f09ae33c..cf1d957c7091 100644
> --- a/arch/x86/include/asm/sev-es.h
> +++ b/arch/x86/include/asm/sev-es.h
> @@ -102,11 +102,13 @@ static __always_inline void sev_es_nmi_complete(void)
> if (static_branch_unlikely(&sev_es_enable_key))
> __sev_es_nmi_complete();
> }
> +extern int __init sev_es_efi_map_ghcbs(pgd_t *pgd);
> #else
> static inline void sev_es_ist_enter(struct pt_regs *regs) { }
> static inline void sev_es_ist_exit(void) { }
> static inline int sev_es_setup_ap_jump_table(struct real_mode_header *rmh) { return 0; }
> static inline void sev_es_nmi_complete(void) { }
> +static inline int sev_es_efi_map_ghcbs(pgd_t *pgd) { return 0; }
> #endif
>
> #endif
> diff --git a/arch/x86/kernel/sev-es.c b/arch/x86/kernel/sev-es.c
> index 9ab3a4dfecd8..4e2b7e4d9b87 100644
> --- a/arch/x86/kernel/sev-es.c
> +++ b/arch/x86/kernel/sev-es.c
> @@ -491,6 +491,36 @@ int sev_es_setup_ap_jump_table(struct real_mode_header *rmh)
> return 0;
> }
>
> +/*
> + * This is needed by the OVMF UEFI firmware which will use whatever it finds in
> + * the GHCB MSR as its GHCB to talk to the hypervisor. So make sure the per-cpu
> + * runtime GHCBs used by the kernel are also mapped in the EFI page-table.
> + */
> +int __init sev_es_efi_map_ghcbs(pgd_t *pgd)
> +{
> + struct sev_es_runtime_data *data;
> + unsigned long address, pflags;
> + int cpu;
> + u64 pfn;
> +
> + if (!sev_es_active())
> + return 0;
> +
> + pflags = _PAGE_NX | _PAGE_RW;
> +
> + for_each_possible_cpu(cpu) {
> + data = per_cpu(runtime_data, cpu);
> +
> + address = __pa(&data->ghcb_page);
> + pfn = address >> PAGE_SHIFT;
> +
> + if (kernel_map_pages_in_pgd(pgd, pfn, address, 1, pflags))
> + return 1;
> + }
> +
> + return 0;
> +}
> +
> static enum es_result vc_handle_msr(struct ghcb *ghcb, struct es_em_ctxt *ctxt)
> {
> struct pt_regs *regs = ctxt->regs;
> diff --git a/arch/x86/platform/efi/efi_64.c b/arch/x86/platform/efi/efi_64.c
> index 6af4da1149ba..8f5759df7776 100644
> --- a/arch/x86/platform/efi/efi_64.c
> +++ b/arch/x86/platform/efi/efi_64.c
> @@ -47,6 +47,7 @@
> #include <asm/realmode.h>
> #include <asm/time.h>
> #include <asm/pgalloc.h>
> +#include <asm/sev-es.h>
>
> /*
> * We allocate runtime services regions top-down, starting from -4G, i.e.
> @@ -229,6 +230,15 @@ int __init efi_setup_page_tables(unsigned long pa_memmap, unsigned num_pages)
> return 1;
> }
>
> + /*
> + * When SEV-ES is active, the GHCB as set by the kernel will be used
> + * by firmware. Create a 1:1 unencrypted mapping for each GHCB.
> + */
> + if (sev_es_efi_map_ghcbs(pgd)) {
> + pr_err("Failed to create 1:1 mapping for the GHCBs!\n");
> + return 1;
> + }
> +
> /*
> * When making calls to the firmware everything needs to be 1:1
> * mapped and addressable with 32-bit pointers. Map the kernel
> --
> 2.28.0
>
--
Regards/Gruss,
Boris.
https://people.kernel.org/tglx/notes-about-netiquette
_______________________________________________
Virtualization mailing list
Virtualization@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/virtualization
^ permalink raw reply [flat|nested] 264+ messages in thread
* Re: [PATCH v7 67/72] x86/smpboot: Load TSS and getcpu GDT entry before loading IDT
2020-09-08 17:20 ` Borislav Petkov
@ 2020-09-08 18:42 ` Joerg Roedel
-1 siblings, 0 replies; 264+ messages in thread
From: Joerg Roedel @ 2020-09-08 18:42 UTC (permalink / raw)
To: Borislav Petkov
Cc: Joerg Roedel, x86, hpa, Andy Lutomirski, Dave Hansen,
Peter Zijlstra, Jiri Slaby, Dan Williams, Tom Lendacky,
Juergen Gross, Kees Cook, David Rientjes, Cfir Cohen,
Erdem Aktas, Masami Hiramatsu, Mike Stunes, Sean Christopherson,
Martin Radev, linux-kernel, kvm, virtualization
On Tue, Sep 08, 2020 at 07:20:42PM +0200, Borislav Petkov wrote:
> On Mon, Sep 07, 2020 at 03:16:08PM +0200, Joerg Roedel wrote:
> > +void cpu_init_exception_handling(void)
> > +{
> > + struct tss_struct *tss = this_cpu_ptr(&cpu_tss_rw);
> > + int cpu = raw_smp_processor_id();
> > +
> > + /* paranoid_entry() gets the CPU number from the GDT */
> > + setup_getcpu(cpu);
> > +
> > + /* IST vectors need TSS to be set up. */
> > + tss_setup_ist(tss);
> > + tss_setup_io_bitmap(tss);
> > + set_tss_desc(cpu, &get_cpu_entry_area(cpu)->tss.x86_tss);
> > +
> > + load_TR_desc();
>
> Aha, this is what you mean here in your 0th message. I'm guessing it is
> ok to do those things twice in start_secondary...
Yes, I think its best to do it twice, so that cpu_init() stays the CPU
state barrier it should be, independent of what happens before.
Joerg
^ permalink raw reply [flat|nested] 264+ messages in thread
* Re: [PATCH v7 67/72] x86/smpboot: Load TSS and getcpu GDT entry before loading IDT
@ 2020-09-08 18:42 ` Joerg Roedel
0 siblings, 0 replies; 264+ messages in thread
From: Joerg Roedel @ 2020-09-08 18:42 UTC (permalink / raw)
To: Borislav Petkov
Cc: Juergen Gross, Tom Lendacky, Dave Hansen, Mike Stunes, Kees Cook,
kvm, Peter Zijlstra, Cfir Cohen, Joerg Roedel, x86, linux-kernel,
Sean Christopherson, virtualization, Martin Radev,
Masami Hiramatsu, Andy Lutomirski, hpa, Erdem Aktas,
David Rientjes, Dan Williams, Jiri Slaby
On Tue, Sep 08, 2020 at 07:20:42PM +0200, Borislav Petkov wrote:
> On Mon, Sep 07, 2020 at 03:16:08PM +0200, Joerg Roedel wrote:
> > +void cpu_init_exception_handling(void)
> > +{
> > + struct tss_struct *tss = this_cpu_ptr(&cpu_tss_rw);
> > + int cpu = raw_smp_processor_id();
> > +
> > + /* paranoid_entry() gets the CPU number from the GDT */
> > + setup_getcpu(cpu);
> > +
> > + /* IST vectors need TSS to be set up. */
> > + tss_setup_ist(tss);
> > + tss_setup_io_bitmap(tss);
> > + set_tss_desc(cpu, &get_cpu_entry_area(cpu)->tss.x86_tss);
> > +
> > + load_TR_desc();
>
> Aha, this is what you mean here in your 0th message. I'm guessing it is
> ok to do those things twice in start_secondary...
Yes, I think its best to do it twice, so that cpu_init() stays the CPU
state barrier it should be, independent of what happens before.
Joerg
_______________________________________________
Virtualization mailing list
Virtualization@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/virtualization
^ permalink raw reply [flat|nested] 264+ messages in thread
* Re: [PATCH v7 71/72] x86/efi: Add GHCB mappings when SEV-ES is active
2020-09-08 17:46 ` Borislav Petkov
(?)
@ 2020-09-09 8:27 ` Ard Biesheuvel
2020-09-09 12:44 ` Laszlo Ersek
-1 siblings, 1 reply; 264+ messages in thread
From: Ard Biesheuvel @ 2020-09-09 8:27 UTC (permalink / raw)
To: Borislav Petkov, Laszlo Ersek, brijesh.singh
Cc: Joerg Roedel, X86 ML, Joerg Roedel, Tom Lendacky, H. Peter Anvin,
Andy Lutomirski, Dave Hansen, Peter Zijlstra, Jiri Slaby,
Dan Williams, Juergen Gross, Kees Cook, David Rientjes,
Cfir Cohen, Erdem Aktas, Masami Hiramatsu, Mike Stunes,
Sean Christopherson, Martin Radev, Linux Kernel Mailing List,
kvm, virtualization
(adding Laszlo and Brijesh)
On Tue, 8 Sep 2020 at 20:46, Borislav Petkov <bp@alien8.de> wrote:
>
> + Ard so that he can ack the efi bits.
>
> On Mon, Sep 07, 2020 at 03:16:12PM +0200, Joerg Roedel wrote:
> > From: Tom Lendacky <thomas.lendacky@amd.com>
> >
> > Calling down to EFI runtime services can result in the firmware performing
> > VMGEXIT calls. The firmware is likely to use the GHCB of the OS (e.g., for
> > setting EFI variables), so each GHCB in the system needs to be identity
> > mapped in the EFI page tables, as unencrypted, to avoid page faults.
> >
> > Signed-off-by: Tom Lendacky <thomas.lendacky@amd.com>
> > [ jroedel@suse.de: Moved GHCB mapping loop to sev-es.c ]
> > Signed-off-by: Joerg Roedel <jroedel@suse.de>
This looks like it is papering over a more fundamental issue: any
memory region that the firmware itself needs to access during the
execution of runtime services needs to be described in the UEFI memory
map, with the appropriate annotations so that the OS knows it should
include these in the EFI runtime page tables. So why has this been
omitted in this case?
> > ---
> > arch/x86/boot/compressed/sev-es.c | 1 +
> > arch/x86/include/asm/sev-es.h | 2 ++
> > arch/x86/kernel/sev-es.c | 30 ++++++++++++++++++++++++++++++
> > arch/x86/platform/efi/efi_64.c | 10 ++++++++++
> > 4 files changed, 43 insertions(+)
> >
> > diff --git a/arch/x86/boot/compressed/sev-es.c b/arch/x86/boot/compressed/sev-es.c
> > index 45702b866c33..0a9a248ca33d 100644
> > --- a/arch/x86/boot/compressed/sev-es.c
> > +++ b/arch/x86/boot/compressed/sev-es.c
> > @@ -12,6 +12,7 @@
> > */
> > #include "misc.h"
> >
> > +#include <asm/pgtable_types.h>
> > #include <asm/sev-es.h>
> > #include <asm/trapnr.h>
> > #include <asm/trap_pf.h>
> > diff --git a/arch/x86/include/asm/sev-es.h b/arch/x86/include/asm/sev-es.h
> > index e919f09ae33c..cf1d957c7091 100644
> > --- a/arch/x86/include/asm/sev-es.h
> > +++ b/arch/x86/include/asm/sev-es.h
> > @@ -102,11 +102,13 @@ static __always_inline void sev_es_nmi_complete(void)
> > if (static_branch_unlikely(&sev_es_enable_key))
> > __sev_es_nmi_complete();
> > }
> > +extern int __init sev_es_efi_map_ghcbs(pgd_t *pgd);
> > #else
> > static inline void sev_es_ist_enter(struct pt_regs *regs) { }
> > static inline void sev_es_ist_exit(void) { }
> > static inline int sev_es_setup_ap_jump_table(struct real_mode_header *rmh) { return 0; }
> > static inline void sev_es_nmi_complete(void) { }
> > +static inline int sev_es_efi_map_ghcbs(pgd_t *pgd) { return 0; }
> > #endif
> >
> > #endif
> > diff --git a/arch/x86/kernel/sev-es.c b/arch/x86/kernel/sev-es.c
> > index 9ab3a4dfecd8..4e2b7e4d9b87 100644
> > --- a/arch/x86/kernel/sev-es.c
> > +++ b/arch/x86/kernel/sev-es.c
> > @@ -491,6 +491,36 @@ int sev_es_setup_ap_jump_table(struct real_mode_header *rmh)
> > return 0;
> > }
> >
> > +/*
> > + * This is needed by the OVMF UEFI firmware which will use whatever it finds in
> > + * the GHCB MSR as its GHCB to talk to the hypervisor. So make sure the per-cpu
> > + * runtime GHCBs used by the kernel are also mapped in the EFI page-table.
> > + */
> > +int __init sev_es_efi_map_ghcbs(pgd_t *pgd)
> > +{
> > + struct sev_es_runtime_data *data;
> > + unsigned long address, pflags;
> > + int cpu;
> > + u64 pfn;
> > +
> > + if (!sev_es_active())
> > + return 0;
> > +
> > + pflags = _PAGE_NX | _PAGE_RW;
> > +
> > + for_each_possible_cpu(cpu) {
> > + data = per_cpu(runtime_data, cpu);
> > +
> > + address = __pa(&data->ghcb_page);
> > + pfn = address >> PAGE_SHIFT;
> > +
> > + if (kernel_map_pages_in_pgd(pgd, pfn, address, 1, pflags))
> > + return 1;
> > + }
> > +
> > + return 0;
> > +}
> > +
> > static enum es_result vc_handle_msr(struct ghcb *ghcb, struct es_em_ctxt *ctxt)
> > {
> > struct pt_regs *regs = ctxt->regs;
> > diff --git a/arch/x86/platform/efi/efi_64.c b/arch/x86/platform/efi/efi_64.c
> > index 6af4da1149ba..8f5759df7776 100644
> > --- a/arch/x86/platform/efi/efi_64.c
> > +++ b/arch/x86/platform/efi/efi_64.c
> > @@ -47,6 +47,7 @@
> > #include <asm/realmode.h>
> > #include <asm/time.h>
> > #include <asm/pgalloc.h>
> > +#include <asm/sev-es.h>
> >
> > /*
> > * We allocate runtime services regions top-down, starting from -4G, i.e.
> > @@ -229,6 +230,15 @@ int __init efi_setup_page_tables(unsigned long pa_memmap, unsigned num_pages)
> > return 1;
> > }
> >
> > + /*
> > + * When SEV-ES is active, the GHCB as set by the kernel will be used
> > + * by firmware. Create a 1:1 unencrypted mapping for each GHCB.
> > + */
> > + if (sev_es_efi_map_ghcbs(pgd)) {
> > + pr_err("Failed to create 1:1 mapping for the GHCBs!\n");
> > + return 1;
> > + }
> > +
> > /*
> > * When making calls to the firmware everything needs to be 1:1
> > * mapped and addressable with 32-bit pointers. Map the kernel
> > --
> > 2.28.0
> >
>
> --
> Regards/Gruss,
> Boris.
>
> https://people.kernel.org/tglx/notes-about-netiquette
^ permalink raw reply [flat|nested] 264+ messages in thread
* Re: [PATCH v7 71/72] x86/efi: Add GHCB mappings when SEV-ES is active
2020-09-09 8:27 ` Ard Biesheuvel
@ 2020-09-09 12:44 ` Laszlo Ersek
0 siblings, 0 replies; 264+ messages in thread
From: Laszlo Ersek @ 2020-09-09 12:44 UTC (permalink / raw)
To: Ard Biesheuvel, Borislav Petkov, brijesh.singh
Cc: Joerg Roedel, X86 ML, Joerg Roedel, Tom Lendacky, H. Peter Anvin,
Andy Lutomirski, Dave Hansen, Peter Zijlstra, Jiri Slaby,
Dan Williams, Juergen Gross, Kees Cook, David Rientjes,
Cfir Cohen, Erdem Aktas, Masami Hiramatsu, Mike Stunes,
Sean Christopherson, Martin Radev, Linux Kernel Mailing List,
kvm, virtualization
On 09/09/20 10:27, Ard Biesheuvel wrote:
> (adding Laszlo and Brijesh)
>
> On Tue, 8 Sep 2020 at 20:46, Borislav Petkov <bp@alien8.de> wrote:
>>
>> + Ard so that he can ack the efi bits.
>>
>> On Mon, Sep 07, 2020 at 03:16:12PM +0200, Joerg Roedel wrote:
>>> From: Tom Lendacky <thomas.lendacky@amd.com>
>>>
>>> Calling down to EFI runtime services can result in the firmware
>>> performing VMGEXIT calls. The firmware is likely to use the GHCB of
>>> the OS (e.g., for setting EFI variables),
I've had to stare at this for a while.
Because, normally a VMGEXIT is supposed to occur like this:
- guest does something privileged
- resultant non-automatic exit (NAE) injects a #VC exception
- exception handler figures out what that privileged thing was
- exception handler submits request to hypervisor via GHCB contents plus
VMGEXIT instruction
Point being, the agent that "owns" the exception handler is supposed to
pre-allocate or otherwise provide the GHCB too, for information passing.
So... what is the particular NAE that occurs during the execution of
UEFI runtime services (at OS runtime)?
And assuming it occurs, I'm unsure if the exception handler (IDT) at
that point is owned (temporarily?) by the firmware.
- If the #VC handler comes from the firmware, then I don't know why it
would use the OS's GHCB.
- If the #VC handler comes from the OS, then I don't understand why the
commit message says "firmware performing VMGEXIT", given that in this
case it would be the OS's #VC handler executing VMGEXIT.
So, I think the above commit message implies a VMGEXIT *without* a NAE /
#VC context. (Because, I fail to interpret the commit message in a NAE /
#VC context in any way; see above.)
OK, so let's see where the firmware performs a VMGEXIT *outside* of an
exception handler, *while* at OS runtime. There seems to be one, in file
"OvmfPkg/QemuFlashFvbServicesRuntimeDxe/QemuFlashDxe.c":
> VOID
> QemuFlashPtrWrite (
> IN volatile UINT8 *Ptr,
> IN UINT8 Value
> )
> {
> if (MemEncryptSevEsIsEnabled ()) {
> MSR_SEV_ES_GHCB_REGISTER Msr;
> GHCB *Ghcb;
>
> Msr.GhcbPhysicalAddress = AsmReadMsr64 (MSR_SEV_ES_GHCB);
> Ghcb = Msr.Ghcb;
>
> //
> // Writing to flash is emulated by the hypervisor through the use of write
> // protection. This won't work for an SEV-ES guest because the write won't
> // be recognized as a true MMIO write, which would result in the required
> // #VC exception. Instead, use the the VMGEXIT MMIO write support directly
> // to perform the update.
> //
> VmgInit (Ghcb);
> Ghcb->SharedBuffer[0] = Value;
> Ghcb->SaveArea.SwScratch = (UINT64) (UINTN) Ghcb->SharedBuffer;
> VmgExit (Ghcb, SVM_EXIT_MMIO_WRITE, (UINT64) (UINTN) Ptr, 1);
> VmgDone (Ghcb);
> } else {
> *Ptr = Value;
> }
> }
This function *does* run at OS runtime (as a part of non-volatile UEFI
variable writes).
And note that, wherever MSR_SEV_ES_GHCB points to at the moment, is used
as GHCB.
If the guest kernel allocates its own GHCB and writes the allocation
address to MSR_SEV_ES_GHCB, then indeed the firmware will use the GHCB
of the OS.
I reviewed edk2 commit 437eb3f7a8db
("OvmfPkg/QemuFlashFvbServicesRuntimeDxe: Bypass flash detection with
SEV-ES", 2020-08-17), but I admit I never thought of the guest OS
changing MSR_SEV_ES_GHCB. I'm sorry about that.
As long as this driver is running before OS runtime (i.e., during the
DXE and BDS phases), MSR_SEV_ES_GHCB is supposed to carry the value we
set in "OvmfPkg/PlatformPei/AmdSev.c":
> STATIC
> VOID
> AmdSevEsInitialize (
> VOID
> )
> {
> VOID *GhcbBase;
> PHYSICAL_ADDRESS GhcbBasePa;
> UINTN GhcbPageCount, PageCount;
> RETURN_STATUS PcdStatus, DecryptStatus;
> IA32_DESCRIPTOR Gdtr;
> VOID *Gdt;
>
> if (!MemEncryptSevEsIsEnabled ()) {
> return;
> }
>
> PcdStatus = PcdSetBoolS (PcdSevEsIsEnabled, TRUE);
> ASSERT_RETURN_ERROR (PcdStatus);
>
> //
> // Allocate GHCB and per-CPU variable pages.
> // Since the pages must survive across the UEFI to OS transition
> // make them reserved.
> //
> GhcbPageCount = mMaxCpuCount * 2;
> GhcbBase = AllocateReservedPages (GhcbPageCount);
> ASSERT (GhcbBase != NULL);
>
> GhcbBasePa = (PHYSICAL_ADDRESS)(UINTN) GhcbBase;
>
> //
> // Each vCPU gets two consecutive pages, the first is the GHCB and the
> // second is the per-CPU variable page. Loop through the allocation and
> // only clear the encryption mask for the GHCB pages.
> //
> for (PageCount = 0; PageCount < GhcbPageCount; PageCount += 2) {
> DecryptStatus = MemEncryptSevClearPageEncMask (
> 0,
> GhcbBasePa + EFI_PAGES_TO_SIZE (PageCount),
> 1,
> TRUE
> );
> ASSERT_RETURN_ERROR (DecryptStatus);
> }
>
> ZeroMem (GhcbBase, EFI_PAGES_TO_SIZE (GhcbPageCount));
>
> PcdStatus = PcdSet64S (PcdGhcbBase, GhcbBasePa);
> ASSERT_RETURN_ERROR (PcdStatus);
> PcdStatus = PcdSet64S (PcdGhcbSize, EFI_PAGES_TO_SIZE (GhcbPageCount));
> ASSERT_RETURN_ERROR (PcdStatus);
>
> DEBUG ((DEBUG_INFO,
> "SEV-ES is enabled, %lu GHCB pages allocated starting at 0x%p\n",
> (UINT64)GhcbPageCount, GhcbBase));
>
> AsmWriteMsr64 (MSR_SEV_ES_GHCB, GhcbBasePa);
So what is the *actual* problem at OS runtime:
- Is it that MSR_SEV_ES_GHCB still points at this PEI-phase *reserved*
memory allocation (and so when QemuFlashPtrWrite() tries to access it
during OS runtime, it doesn't have a runtime mapping for it)?
- Or is it that the OS actively changes MSR_SEV_ES_GHCB, pointing to a
memory area that the OS owns -- and *that* area is what
QemuFlashPtrWrite() cannot access at OS runtime?
The first problem statement does *not* seem to apply, given -- again --
that the commit message says, "firmware is likely to use the GHCB of the
OS".
So I think the second problem statement must apply.
(I think the "reserved allocation" above is "reserved" only because we
want to keep the OS out of it around the ExitBootServices() transition.)
Back to the email:
On 09/09/20 10:27, Ard Biesheuvel wrote:
> On Tue, 8 Sep 2020 at 20:46, Borislav Petkov <bp@alien8.de> wrote:
>> On Mon, Sep 07, 2020 at 03:16:12PM +0200, Joerg Roedel wrote:
>>> so each GHCB in the system needs to be identity
>>> mapped in the EFI page tables, as unencrypted, to avoid page faults.
Not sure I agree about this, but at least it seems to confirm my
understanding -- apparently the idea is, for the OS, to satisfy
QemuFlashPtrWrite() in the firmware, by putting the "expected" mapping
-- for wherever MSR_SEV_ES_GHCB is going to point to -- in place.
>>>
>>> Signed-off-by: Tom Lendacky <thomas.lendacky@amd.com>
>>> [ jroedel@suse.de: Moved GHCB mapping loop to sev-es.c ]
>>> Signed-off-by: Joerg Roedel <jroedel@suse.de>
>
>
> This looks like it is papering over a more fundamental issue: any
> memory region that the firmware itself needs to access during the
> execution of runtime services needs to be described in the UEFI memory
> map, with the appropriate annotations so that the OS knows it should
> include these in the EFI runtime page tables. So why has this been
> omitted in this case?
So yeah, the issue seems to be that the QemuFlashFvbServicesRuntimeDxe
driver does not *own* the GHCB that it attempts to use at OS runtime. It
doesn't know where MSR_SEV_ES_GHCB is going to point.
Is QemuFlashFvbServicesRuntimeDxe permitted to change MSR_SEV_ES_GHCB
*temporarily* at OS runtime?
Because, in that case:
- QemuFlashFvbServicesRuntimeDxe should allocate a Runtime Services Data
block for GHCB when it starts up (if SEV-ES is active),
- QemuFlashFvbServicesRuntimeDxe should register a SetVirtualAddressMap
handler, and use EfiConvertPointer() from UefiRuntimeLib to convert
the "runtime GHCB" address to virtual address, in that handler,
- QemuFlashPtrWrite() should call EfiAtRuntime() from UefiRuntimeLib,
and if the latter returns TRUE, then (a) use the runtime-converted
address for populating the GHCB, and (b) temporarily swap
MSR_SEV_ES_GHCB with the address of the self-allocated GHCB. (The MSR
needs a *physical* address, so QemuFlashFvbServicesRuntimeDxe would
have to remember / retain the original (physical) allocation address
too.)
If QemuFlashFvbServicesRuntimeDxe is not permitted to change
MSR_SEV_ES_GHCB even temporarily (at OS runtime), then I think the
approach proposed in this (guest kernel) patch is valid.
Let me skim the code below...
>
>
>
>>> ---
>>> arch/x86/boot/compressed/sev-es.c | 1 +
>>> arch/x86/include/asm/sev-es.h | 2 ++
>>> arch/x86/kernel/sev-es.c | 30 ++++++++++++++++++++++++++++++
>>> arch/x86/platform/efi/efi_64.c | 10 ++++++++++
>>> 4 files changed, 43 insertions(+)
>>>
>>> diff --git a/arch/x86/boot/compressed/sev-es.c b/arch/x86/boot/compressed/sev-es.c
>>> index 45702b866c33..0a9a248ca33d 100644
>>> --- a/arch/x86/boot/compressed/sev-es.c
>>> +++ b/arch/x86/boot/compressed/sev-es.c
>>> @@ -12,6 +12,7 @@
>>> */
>>> #include "misc.h"
>>>
>>> +#include <asm/pgtable_types.h>
>>> #include <asm/sev-es.h>
>>> #include <asm/trapnr.h>
>>> #include <asm/trap_pf.h>
>>> diff --git a/arch/x86/include/asm/sev-es.h b/arch/x86/include/asm/sev-es.h
>>> index e919f09ae33c..cf1d957c7091 100644
>>> --- a/arch/x86/include/asm/sev-es.h
>>> +++ b/arch/x86/include/asm/sev-es.h
>>> @@ -102,11 +102,13 @@ static __always_inline void sev_es_nmi_complete(void)
>>> if (static_branch_unlikely(&sev_es_enable_key))
>>> __sev_es_nmi_complete();
>>> }
>>> +extern int __init sev_es_efi_map_ghcbs(pgd_t *pgd);
>>> #else
>>> static inline void sev_es_ist_enter(struct pt_regs *regs) { }
>>> static inline void sev_es_ist_exit(void) { }
>>> static inline int sev_es_setup_ap_jump_table(struct real_mode_header *rmh) { return 0; }
>>> static inline void sev_es_nmi_complete(void) { }
>>> +static inline int sev_es_efi_map_ghcbs(pgd_t *pgd) { return 0; }
>>> #endif
>>>
>>> #endif
>>> diff --git a/arch/x86/kernel/sev-es.c b/arch/x86/kernel/sev-es.c
>>> index 9ab3a4dfecd8..4e2b7e4d9b87 100644
>>> --- a/arch/x86/kernel/sev-es.c
>>> +++ b/arch/x86/kernel/sev-es.c
>>> @@ -491,6 +491,36 @@ int sev_es_setup_ap_jump_table(struct real_mode_header *rmh)
>>> return 0;
>>> }
>>>
>>> +/*
>>> + * This is needed by the OVMF UEFI firmware which will use whatever it finds in
>>> + * the GHCB MSR as its GHCB to talk to the hypervisor. So make sure the per-cpu
>>> + * runtime GHCBs used by the kernel are also mapped in the EFI page-table.
Yup, this pretty much confirms my suspicion that QemuFlashPtrWrite() is
at the center of this.
(BTW, I don't think that the runtime services data allocation, in
QemuFlashFvbServicesRuntimeDxe, for OS runtime GHCB purposes, would have
to be "per CPU". Refer to "Table 35. Rules for Reentry Into Runtime
Services" in the UEFI spec -- if one processor is executing
SetVariable(), then no other processor must enter SetVariable(). And so
we'll have *at most* one VCPU in QemuFlashPtrWrite(), at any time.)
>>> + */
>>> +int __init sev_es_efi_map_ghcbs(pgd_t *pgd)
>>> +{
>>> + struct sev_es_runtime_data *data;
>>> + unsigned long address, pflags;
>>> + int cpu;
>>> + u64 pfn;
>>> +
>>> + if (!sev_es_active())
>>> + return 0;
>>> +
>>> + pflags = _PAGE_NX | _PAGE_RW;
>>> +
>>> + for_each_possible_cpu(cpu) {
>>> + data = per_cpu(runtime_data, cpu);
>>> +
>>> + address = __pa(&data->ghcb_page);
>>> + pfn = address >> PAGE_SHIFT;
>>> +
>>> + if (kernel_map_pages_in_pgd(pgd, pfn, address, 1, pflags))
>>> + return 1;
>>> + }
>>> +
>>> + return 0;
>>> +}
>>> +
>>> static enum es_result vc_handle_msr(struct ghcb *ghcb, struct es_em_ctxt *ctxt)
>>> {
>>> struct pt_regs *regs = ctxt->regs;
>>> diff --git a/arch/x86/platform/efi/efi_64.c b/arch/x86/platform/efi/efi_64.c
>>> index 6af4da1149ba..8f5759df7776 100644
>>> --- a/arch/x86/platform/efi/efi_64.c
>>> +++ b/arch/x86/platform/efi/efi_64.c
>>> @@ -47,6 +47,7 @@
>>> #include <asm/realmode.h>
>>> #include <asm/time.h>
>>> #include <asm/pgalloc.h>
>>> +#include <asm/sev-es.h>
>>>
>>> /*
>>> * We allocate runtime services regions top-down, starting from -4G, i.e.
>>> @@ -229,6 +230,15 @@ int __init efi_setup_page_tables(unsigned long pa_memmap, unsigned num_pages)
>>> return 1;
>>> }
>>>
>>> + /*
>>> + * When SEV-ES is active, the GHCB as set by the kernel will be used
>>> + * by firmware. Create a 1:1 unencrypted mapping for each GHCB.
>>> + */
>>> + if (sev_es_efi_map_ghcbs(pgd)) {
>>> + pr_err("Failed to create 1:1 mapping for the GHCBs!\n");
>>> + return 1;
>>> + }
>>> +
>>> /*
>>> * When making calls to the firmware everything needs to be 1:1
>>> * mapped and addressable with 32-bit pointers. Map the kernel
Good point!
And it even makes me wonder if the QemuFlashFvbServicesRuntimeDxe
approach, with the runtime services data type memory allocation, is
feasible at all. Namely, a page's encryption status, under SEV, is
controlled through the PTE.
And for this particular UEFI runtime area, it would *not* suffice for
the OS to just virt-map it. The OS would also have to *decrypt* the area
(mark the PTE as "plaintext").
In other words, it would be an "unprecedented" PTE for the OS to set up:
the PTE would not only map the GVA to GPA, but also mark the area as
"plaintext".
Otherwise -- if the OS covers *just* the virt-mapping --,
QemuFlashFvbServicesRuntimeDxe would populate its own "runtime GHCB"
area just fine, but the actual data hitting the host RAM would be
encrypted. And so the hypervisor could not interpret the GHCB.
*If* QemuFlashFvbServicesRuntimeDxe should not change the kernel-owned
PTE at runtime, even temporarily, for marking the GHCB as "plaintext",
then the problem is indeed only solvable in the guest kernel, in my
opinion.
There simply isn't an "architected annotation" for telling the kernel,
"virt-map this runtime services data type memory range, *and* mark it as
plaintext at the same time".
This would be necessary, as both actions affect the exact same PTE, and
the firmware is not really allowed to touch the PTE at runtime. But we
don't have such a hint.
To summarize: for QemuFlashFvbServicesRuntimeDxe to allocate UEFI
Runtime Services Data type memory, for its own runtime GHCB, two
permissions are necessary (together), at OS runtime:
- QemuFlashFvbServicesRuntimeDxe must be allowed to swap MSR_SEV_ES_GHCB
temporarily (before executing VMGEXIT),
- QemuFlashFvbServicesRuntimeDxe must be allowed to change the OS-owned
PTE temporarily (for remapping the GHCB as plaintext, before writing
to it).
Thanks
Laszlo
^ permalink raw reply [flat|nested] 264+ messages in thread
* Re: [PATCH v7 71/72] x86/efi: Add GHCB mappings when SEV-ES is active
@ 2020-09-09 12:44 ` Laszlo Ersek
0 siblings, 0 replies; 264+ messages in thread
From: Laszlo Ersek @ 2020-09-09 12:44 UTC (permalink / raw)
To: Ard Biesheuvel, Borislav Petkov, brijesh.singh
Cc: kvm, Peter Zijlstra, Dave Hansen, virtualization, H. Peter Anvin,
Jiri Slaby, Joerg Roedel, X86 ML, David Rientjes, Martin Radev,
Tom Lendacky, Joerg Roedel, Kees Cook, Cfir Cohen,
Andy Lutomirski, Dan Williams, Juergen Gross, Mike Stunes,
Linux Kernel Mailing List, Sean Christopherson, Masami Hiramatsu,
Erdem Aktas
On 09/09/20 10:27, Ard Biesheuvel wrote:
> (adding Laszlo and Brijesh)
>
> On Tue, 8 Sep 2020 at 20:46, Borislav Petkov <bp@alien8.de> wrote:
>>
>> + Ard so that he can ack the efi bits.
>>
>> On Mon, Sep 07, 2020 at 03:16:12PM +0200, Joerg Roedel wrote:
>>> From: Tom Lendacky <thomas.lendacky@amd.com>
>>>
>>> Calling down to EFI runtime services can result in the firmware
>>> performing VMGEXIT calls. The firmware is likely to use the GHCB of
>>> the OS (e.g., for setting EFI variables),
I've had to stare at this for a while.
Because, normally a VMGEXIT is supposed to occur like this:
- guest does something privileged
- resultant non-automatic exit (NAE) injects a #VC exception
- exception handler figures out what that privileged thing was
- exception handler submits request to hypervisor via GHCB contents plus
VMGEXIT instruction
Point being, the agent that "owns" the exception handler is supposed to
pre-allocate or otherwise provide the GHCB too, for information passing.
So... what is the particular NAE that occurs during the execution of
UEFI runtime services (at OS runtime)?
And assuming it occurs, I'm unsure if the exception handler (IDT) at
that point is owned (temporarily?) by the firmware.
- If the #VC handler comes from the firmware, then I don't know why it
would use the OS's GHCB.
- If the #VC handler comes from the OS, then I don't understand why the
commit message says "firmware performing VMGEXIT", given that in this
case it would be the OS's #VC handler executing VMGEXIT.
So, I think the above commit message implies a VMGEXIT *without* a NAE /
#VC context. (Because, I fail to interpret the commit message in a NAE /
#VC context in any way; see above.)
OK, so let's see where the firmware performs a VMGEXIT *outside* of an
exception handler, *while* at OS runtime. There seems to be one, in file
"OvmfPkg/QemuFlashFvbServicesRuntimeDxe/QemuFlashDxe.c":
> VOID
> QemuFlashPtrWrite (
> IN volatile UINT8 *Ptr,
> IN UINT8 Value
> )
> {
> if (MemEncryptSevEsIsEnabled ()) {
> MSR_SEV_ES_GHCB_REGISTER Msr;
> GHCB *Ghcb;
>
> Msr.GhcbPhysicalAddress = AsmReadMsr64 (MSR_SEV_ES_GHCB);
> Ghcb = Msr.Ghcb;
>
> //
> // Writing to flash is emulated by the hypervisor through the use of write
> // protection. This won't work for an SEV-ES guest because the write won't
> // be recognized as a true MMIO write, which would result in the required
> // #VC exception. Instead, use the the VMGEXIT MMIO write support directly
> // to perform the update.
> //
> VmgInit (Ghcb);
> Ghcb->SharedBuffer[0] = Value;
> Ghcb->SaveArea.SwScratch = (UINT64) (UINTN) Ghcb->SharedBuffer;
> VmgExit (Ghcb, SVM_EXIT_MMIO_WRITE, (UINT64) (UINTN) Ptr, 1);
> VmgDone (Ghcb);
> } else {
> *Ptr = Value;
> }
> }
This function *does* run at OS runtime (as a part of non-volatile UEFI
variable writes).
And note that, wherever MSR_SEV_ES_GHCB points to at the moment, is used
as GHCB.
If the guest kernel allocates its own GHCB and writes the allocation
address to MSR_SEV_ES_GHCB, then indeed the firmware will use the GHCB
of the OS.
I reviewed edk2 commit 437eb3f7a8db
("OvmfPkg/QemuFlashFvbServicesRuntimeDxe: Bypass flash detection with
SEV-ES", 2020-08-17), but I admit I never thought of the guest OS
changing MSR_SEV_ES_GHCB. I'm sorry about that.
As long as this driver is running before OS runtime (i.e., during the
DXE and BDS phases), MSR_SEV_ES_GHCB is supposed to carry the value we
set in "OvmfPkg/PlatformPei/AmdSev.c":
> STATIC
> VOID
> AmdSevEsInitialize (
> VOID
> )
> {
> VOID *GhcbBase;
> PHYSICAL_ADDRESS GhcbBasePa;
> UINTN GhcbPageCount, PageCount;
> RETURN_STATUS PcdStatus, DecryptStatus;
> IA32_DESCRIPTOR Gdtr;
> VOID *Gdt;
>
> if (!MemEncryptSevEsIsEnabled ()) {
> return;
> }
>
> PcdStatus = PcdSetBoolS (PcdSevEsIsEnabled, TRUE);
> ASSERT_RETURN_ERROR (PcdStatus);
>
> //
> // Allocate GHCB and per-CPU variable pages.
> // Since the pages must survive across the UEFI to OS transition
> // make them reserved.
> //
> GhcbPageCount = mMaxCpuCount * 2;
> GhcbBase = AllocateReservedPages (GhcbPageCount);
> ASSERT (GhcbBase != NULL);
>
> GhcbBasePa = (PHYSICAL_ADDRESS)(UINTN) GhcbBase;
>
> //
> // Each vCPU gets two consecutive pages, the first is the GHCB and the
> // second is the per-CPU variable page. Loop through the allocation and
> // only clear the encryption mask for the GHCB pages.
> //
> for (PageCount = 0; PageCount < GhcbPageCount; PageCount += 2) {
> DecryptStatus = MemEncryptSevClearPageEncMask (
> 0,
> GhcbBasePa + EFI_PAGES_TO_SIZE (PageCount),
> 1,
> TRUE
> );
> ASSERT_RETURN_ERROR (DecryptStatus);
> }
>
> ZeroMem (GhcbBase, EFI_PAGES_TO_SIZE (GhcbPageCount));
>
> PcdStatus = PcdSet64S (PcdGhcbBase, GhcbBasePa);
> ASSERT_RETURN_ERROR (PcdStatus);
> PcdStatus = PcdSet64S (PcdGhcbSize, EFI_PAGES_TO_SIZE (GhcbPageCount));
> ASSERT_RETURN_ERROR (PcdStatus);
>
> DEBUG ((DEBUG_INFO,
> "SEV-ES is enabled, %lu GHCB pages allocated starting at 0x%p\n",
> (UINT64)GhcbPageCount, GhcbBase));
>
> AsmWriteMsr64 (MSR_SEV_ES_GHCB, GhcbBasePa);
So what is the *actual* problem at OS runtime:
- Is it that MSR_SEV_ES_GHCB still points at this PEI-phase *reserved*
memory allocation (and so when QemuFlashPtrWrite() tries to access it
during OS runtime, it doesn't have a runtime mapping for it)?
- Or is it that the OS actively changes MSR_SEV_ES_GHCB, pointing to a
memory area that the OS owns -- and *that* area is what
QemuFlashPtrWrite() cannot access at OS runtime?
The first problem statement does *not* seem to apply, given -- again --
that the commit message says, "firmware is likely to use the GHCB of the
OS".
So I think the second problem statement must apply.
(I think the "reserved allocation" above is "reserved" only because we
want to keep the OS out of it around the ExitBootServices() transition.)
Back to the email:
On 09/09/20 10:27, Ard Biesheuvel wrote:
> On Tue, 8 Sep 2020 at 20:46, Borislav Petkov <bp@alien8.de> wrote:
>> On Mon, Sep 07, 2020 at 03:16:12PM +0200, Joerg Roedel wrote:
>>> so each GHCB in the system needs to be identity
>>> mapped in the EFI page tables, as unencrypted, to avoid page faults.
Not sure I agree about this, but at least it seems to confirm my
understanding -- apparently the idea is, for the OS, to satisfy
QemuFlashPtrWrite() in the firmware, by putting the "expected" mapping
-- for wherever MSR_SEV_ES_GHCB is going to point to -- in place.
>>>
>>> Signed-off-by: Tom Lendacky <thomas.lendacky@amd.com>
>>> [ jroedel@suse.de: Moved GHCB mapping loop to sev-es.c ]
>>> Signed-off-by: Joerg Roedel <jroedel@suse.de>
>
>
> This looks like it is papering over a more fundamental issue: any
> memory region that the firmware itself needs to access during the
> execution of runtime services needs to be described in the UEFI memory
> map, with the appropriate annotations so that the OS knows it should
> include these in the EFI runtime page tables. So why has this been
> omitted in this case?
So yeah, the issue seems to be that the QemuFlashFvbServicesRuntimeDxe
driver does not *own* the GHCB that it attempts to use at OS runtime. It
doesn't know where MSR_SEV_ES_GHCB is going to point.
Is QemuFlashFvbServicesRuntimeDxe permitted to change MSR_SEV_ES_GHCB
*temporarily* at OS runtime?
Because, in that case:
- QemuFlashFvbServicesRuntimeDxe should allocate a Runtime Services Data
block for GHCB when it starts up (if SEV-ES is active),
- QemuFlashFvbServicesRuntimeDxe should register a SetVirtualAddressMap
handler, and use EfiConvertPointer() from UefiRuntimeLib to convert
the "runtime GHCB" address to virtual address, in that handler,
- QemuFlashPtrWrite() should call EfiAtRuntime() from UefiRuntimeLib,
and if the latter returns TRUE, then (a) use the runtime-converted
address for populating the GHCB, and (b) temporarily swap
MSR_SEV_ES_GHCB with the address of the self-allocated GHCB. (The MSR
needs a *physical* address, so QemuFlashFvbServicesRuntimeDxe would
have to remember / retain the original (physical) allocation address
too.)
If QemuFlashFvbServicesRuntimeDxe is not permitted to change
MSR_SEV_ES_GHCB even temporarily (at OS runtime), then I think the
approach proposed in this (guest kernel) patch is valid.
Let me skim the code below...
>
>
>
>>> ---
>>> arch/x86/boot/compressed/sev-es.c | 1 +
>>> arch/x86/include/asm/sev-es.h | 2 ++
>>> arch/x86/kernel/sev-es.c | 30 ++++++++++++++++++++++++++++++
>>> arch/x86/platform/efi/efi_64.c | 10 ++++++++++
>>> 4 files changed, 43 insertions(+)
>>>
>>> diff --git a/arch/x86/boot/compressed/sev-es.c b/arch/x86/boot/compressed/sev-es.c
>>> index 45702b866c33..0a9a248ca33d 100644
>>> --- a/arch/x86/boot/compressed/sev-es.c
>>> +++ b/arch/x86/boot/compressed/sev-es.c
>>> @@ -12,6 +12,7 @@
>>> */
>>> #include "misc.h"
>>>
>>> +#include <asm/pgtable_types.h>
>>> #include <asm/sev-es.h>
>>> #include <asm/trapnr.h>
>>> #include <asm/trap_pf.h>
>>> diff --git a/arch/x86/include/asm/sev-es.h b/arch/x86/include/asm/sev-es.h
>>> index e919f09ae33c..cf1d957c7091 100644
>>> --- a/arch/x86/include/asm/sev-es.h
>>> +++ b/arch/x86/include/asm/sev-es.h
>>> @@ -102,11 +102,13 @@ static __always_inline void sev_es_nmi_complete(void)
>>> if (static_branch_unlikely(&sev_es_enable_key))
>>> __sev_es_nmi_complete();
>>> }
>>> +extern int __init sev_es_efi_map_ghcbs(pgd_t *pgd);
>>> #else
>>> static inline void sev_es_ist_enter(struct pt_regs *regs) { }
>>> static inline void sev_es_ist_exit(void) { }
>>> static inline int sev_es_setup_ap_jump_table(struct real_mode_header *rmh) { return 0; }
>>> static inline void sev_es_nmi_complete(void) { }
>>> +static inline int sev_es_efi_map_ghcbs(pgd_t *pgd) { return 0; }
>>> #endif
>>>
>>> #endif
>>> diff --git a/arch/x86/kernel/sev-es.c b/arch/x86/kernel/sev-es.c
>>> index 9ab3a4dfecd8..4e2b7e4d9b87 100644
>>> --- a/arch/x86/kernel/sev-es.c
>>> +++ b/arch/x86/kernel/sev-es.c
>>> @@ -491,6 +491,36 @@ int sev_es_setup_ap_jump_table(struct real_mode_header *rmh)
>>> return 0;
>>> }
>>>
>>> +/*
>>> + * This is needed by the OVMF UEFI firmware which will use whatever it finds in
>>> + * the GHCB MSR as its GHCB to talk to the hypervisor. So make sure the per-cpu
>>> + * runtime GHCBs used by the kernel are also mapped in the EFI page-table.
Yup, this pretty much confirms my suspicion that QemuFlashPtrWrite() is
at the center of this.
(BTW, I don't think that the runtime services data allocation, in
QemuFlashFvbServicesRuntimeDxe, for OS runtime GHCB purposes, would have
to be "per CPU". Refer to "Table 35. Rules for Reentry Into Runtime
Services" in the UEFI spec -- if one processor is executing
SetVariable(), then no other processor must enter SetVariable(). And so
we'll have *at most* one VCPU in QemuFlashPtrWrite(), at any time.)
>>> + */
>>> +int __init sev_es_efi_map_ghcbs(pgd_t *pgd)
>>> +{
>>> + struct sev_es_runtime_data *data;
>>> + unsigned long address, pflags;
>>> + int cpu;
>>> + u64 pfn;
>>> +
>>> + if (!sev_es_active())
>>> + return 0;
>>> +
>>> + pflags = _PAGE_NX | _PAGE_RW;
>>> +
>>> + for_each_possible_cpu(cpu) {
>>> + data = per_cpu(runtime_data, cpu);
>>> +
>>> + address = __pa(&data->ghcb_page);
>>> + pfn = address >> PAGE_SHIFT;
>>> +
>>> + if (kernel_map_pages_in_pgd(pgd, pfn, address, 1, pflags))
>>> + return 1;
>>> + }
>>> +
>>> + return 0;
>>> +}
>>> +
>>> static enum es_result vc_handle_msr(struct ghcb *ghcb, struct es_em_ctxt *ctxt)
>>> {
>>> struct pt_regs *regs = ctxt->regs;
>>> diff --git a/arch/x86/platform/efi/efi_64.c b/arch/x86/platform/efi/efi_64.c
>>> index 6af4da1149ba..8f5759df7776 100644
>>> --- a/arch/x86/platform/efi/efi_64.c
>>> +++ b/arch/x86/platform/efi/efi_64.c
>>> @@ -47,6 +47,7 @@
>>> #include <asm/realmode.h>
>>> #include <asm/time.h>
>>> #include <asm/pgalloc.h>
>>> +#include <asm/sev-es.h>
>>>
>>> /*
>>> * We allocate runtime services regions top-down, starting from -4G, i.e.
>>> @@ -229,6 +230,15 @@ int __init efi_setup_page_tables(unsigned long pa_memmap, unsigned num_pages)
>>> return 1;
>>> }
>>>
>>> + /*
>>> + * When SEV-ES is active, the GHCB as set by the kernel will be used
>>> + * by firmware. Create a 1:1 unencrypted mapping for each GHCB.
>>> + */
>>> + if (sev_es_efi_map_ghcbs(pgd)) {
>>> + pr_err("Failed to create 1:1 mapping for the GHCBs!\n");
>>> + return 1;
>>> + }
>>> +
>>> /*
>>> * When making calls to the firmware everything needs to be 1:1
>>> * mapped and addressable with 32-bit pointers. Map the kernel
Good point!
And it even makes me wonder if the QemuFlashFvbServicesRuntimeDxe
approach, with the runtime services data type memory allocation, is
feasible at all. Namely, a page's encryption status, under SEV, is
controlled through the PTE.
And for this particular UEFI runtime area, it would *not* suffice for
the OS to just virt-map it. The OS would also have to *decrypt* the area
(mark the PTE as "plaintext").
In other words, it would be an "unprecedented" PTE for the OS to set up:
the PTE would not only map the GVA to GPA, but also mark the area as
"plaintext".
Otherwise -- if the OS covers *just* the virt-mapping --,
QemuFlashFvbServicesRuntimeDxe would populate its own "runtime GHCB"
area just fine, but the actual data hitting the host RAM would be
encrypted. And so the hypervisor could not interpret the GHCB.
*If* QemuFlashFvbServicesRuntimeDxe should not change the kernel-owned
PTE at runtime, even temporarily, for marking the GHCB as "plaintext",
then the problem is indeed only solvable in the guest kernel, in my
opinion.
There simply isn't an "architected annotation" for telling the kernel,
"virt-map this runtime services data type memory range, *and* mark it as
plaintext at the same time".
This would be necessary, as both actions affect the exact same PTE, and
the firmware is not really allowed to touch the PTE at runtime. But we
don't have such a hint.
To summarize: for QemuFlashFvbServicesRuntimeDxe to allocate UEFI
Runtime Services Data type memory, for its own runtime GHCB, two
permissions are necessary (together), at OS runtime:
- QemuFlashFvbServicesRuntimeDxe must be allowed to swap MSR_SEV_ES_GHCB
temporarily (before executing VMGEXIT),
- QemuFlashFvbServicesRuntimeDxe must be allowed to change the OS-owned
PTE temporarily (for remapping the GHCB as plaintext, before writing
to it).
Thanks
Laszlo
_______________________________________________
Virtualization mailing list
Virtualization@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/virtualization
^ permalink raw reply [flat|nested] 264+ messages in thread
* Re: [PATCH v7 71/72] x86/efi: Add GHCB mappings when SEV-ES is active
2020-09-09 12:44 ` Laszlo Ersek
@ 2020-09-09 13:24 ` Laszlo Ersek
-1 siblings, 0 replies; 264+ messages in thread
From: Laszlo Ersek @ 2020-09-09 13:24 UTC (permalink / raw)
To: Ard Biesheuvel, Borislav Petkov, brijesh.singh
Cc: Joerg Roedel, X86 ML, Joerg Roedel, Tom Lendacky, H. Peter Anvin,
Andy Lutomirski, Dave Hansen, Peter Zijlstra, Jiri Slaby,
Dan Williams, Juergen Gross, Kees Cook, David Rientjes,
Cfir Cohen, Erdem Aktas, Masami Hiramatsu, Mike Stunes,
Sean Christopherson, Martin Radev, Linux Kernel Mailing List,
kvm, virtualization
On 09/09/20 14:44, Laszlo Ersek wrote:
> To summarize: for QemuFlashFvbServicesRuntimeDxe to allocate UEFI
> Runtime Services Data type memory, for its own runtime GHCB, two
> permissions are necessary (together), at OS runtime:
>
> - QemuFlashFvbServicesRuntimeDxe must be allowed to swap MSR_SEV_ES_GHCB
> temporarily (before executing VMGEXIT),
>
> - QemuFlashFvbServicesRuntimeDxe must be allowed to change the OS-owned
> PTE temporarily (for remapping the GHCB as plaintext, before writing
> to it).
Condition#2 gets worse:
If the firmware-allocated runtime GHCB happens to be virt-mapped by the
OS using a 2MB page (covering other UEFI runtime data areas, perhaps?),
then simply flipping the encryption bit in
QemuFlashFvbServicesRuntimeDxe would mark more runtime memory than just
the GHCB as "plaintext". (2MB-4KB specifically.)
This could result in:
- firmware read accesses outside of the GHCB returning garbage
- firmware write accesses ouside of the GHCB leaking information to the
hypervisor, and reading back later as garbage
In order to prevent those symptoms, the firmware would have to split the
2MB page to 4KB pages, and decrypt just the one (GHCB) page.
But page splitting requires additional memory (for the finer granularity
page table), and fw memory cannot be allocated at OS runtime. So that
extra memory too would have to be pre-allocated by the firmware. Nasty.
I'd recommend sticking with this kernel patch.
Thanks,
Laszlo
^ permalink raw reply [flat|nested] 264+ messages in thread
* Re: [PATCH v7 71/72] x86/efi: Add GHCB mappings when SEV-ES is active
@ 2020-09-09 13:24 ` Laszlo Ersek
0 siblings, 0 replies; 264+ messages in thread
From: Laszlo Ersek @ 2020-09-09 13:24 UTC (permalink / raw)
To: Ard Biesheuvel, Borislav Petkov, brijesh.singh
Cc: kvm, Peter Zijlstra, Dave Hansen, virtualization, H. Peter Anvin,
Jiri Slaby, Joerg Roedel, X86 ML, David Rientjes, Martin Radev,
Tom Lendacky, Joerg Roedel, Kees Cook, Cfir Cohen,
Andy Lutomirski, Dan Williams, Juergen Gross, Mike Stunes,
Linux Kernel Mailing List, Sean Christopherson, Masami Hiramatsu,
Erdem Aktas
On 09/09/20 14:44, Laszlo Ersek wrote:
> To summarize: for QemuFlashFvbServicesRuntimeDxe to allocate UEFI
> Runtime Services Data type memory, for its own runtime GHCB, two
> permissions are necessary (together), at OS runtime:
>
> - QemuFlashFvbServicesRuntimeDxe must be allowed to swap MSR_SEV_ES_GHCB
> temporarily (before executing VMGEXIT),
>
> - QemuFlashFvbServicesRuntimeDxe must be allowed to change the OS-owned
> PTE temporarily (for remapping the GHCB as plaintext, before writing
> to it).
Condition#2 gets worse:
If the firmware-allocated runtime GHCB happens to be virt-mapped by the
OS using a 2MB page (covering other UEFI runtime data areas, perhaps?),
then simply flipping the encryption bit in
QemuFlashFvbServicesRuntimeDxe would mark more runtime memory than just
the GHCB as "plaintext". (2MB-4KB specifically.)
This could result in:
- firmware read accesses outside of the GHCB returning garbage
- firmware write accesses ouside of the GHCB leaking information to the
hypervisor, and reading back later as garbage
In order to prevent those symptoms, the firmware would have to split the
2MB page to 4KB pages, and decrypt just the one (GHCB) page.
But page splitting requires additional memory (for the finer granularity
page table), and fw memory cannot be allocated at OS runtime. So that
extra memory too would have to be pre-allocated by the firmware. Nasty.
I'd recommend sticking with this kernel patch.
Thanks,
Laszlo
_______________________________________________
Virtualization mailing list
Virtualization@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/virtualization
^ permalink raw reply [flat|nested] 264+ messages in thread
* Re: [PATCH v7 71/72] x86/efi: Add GHCB mappings when SEV-ES is active
2020-09-09 12:44 ` Laszlo Ersek
@ 2020-09-09 13:49 ` Tom Lendacky
-1 siblings, 0 replies; 264+ messages in thread
From: Tom Lendacky @ 2020-09-09 13:49 UTC (permalink / raw)
To: Laszlo Ersek, Ard Biesheuvel, Borislav Petkov, brijesh.singh
Cc: Joerg Roedel, X86 ML, Joerg Roedel, H. Peter Anvin,
Andy Lutomirski, Dave Hansen, Peter Zijlstra, Jiri Slaby,
Dan Williams, Juergen Gross, Kees Cook, David Rientjes,
Cfir Cohen, Erdem Aktas, Masami Hiramatsu, Mike Stunes,
Sean Christopherson, Martin Radev, Linux Kernel Mailing List,
kvm, virtualization
On 9/9/20 7:44 AM, Laszlo Ersek wrote:
> On 09/09/20 10:27, Ard Biesheuvel wrote:
>> (adding Laszlo and Brijesh)
>>
>> On Tue, 8 Sep 2020 at 20:46, Borislav Petkov <bp@alien8.de> wrote:
>>>
>>> + Ard so that he can ack the efi bits.
>>>
>>> On Mon, Sep 07, 2020 at 03:16:12PM +0200, Joerg Roedel wrote:
>>>> From: Tom Lendacky <thomas.lendacky@amd.com>
>>>>
>>>> Calling down to EFI runtime services can result in the firmware
>>>> performing VMGEXIT calls. The firmware is likely to use the GHCB of
>>>> the OS (e.g., for setting EFI variables),
>
> I've had to stare at this for a while.
>
> Because, normally a VMGEXIT is supposed to occur like this:
>
> - guest does something privileged
> - resultant non-automatic exit (NAE) injects a #VC exception
> - exception handler figures out what that privileged thing was
> - exception handler submits request to hypervisor via GHCB contents plus
> VMGEXIT instruction
>
> Point being, the agent that "owns" the exception handler is supposed to
> pre-allocate or otherwise provide the GHCB too, for information passing.
>
> So... what is the particular NAE that occurs during the execution of
> UEFI runtime services (at OS runtime)?
>
> And assuming it occurs, I'm unsure if the exception handler (IDT) at
> that point is owned (temporarily?) by the firmware.
>
> - If the #VC handler comes from the firmware, then I don't know why it
> would use the OS's GHCB.
>
> - If the #VC handler comes from the OS, then I don't understand why the
> commit message says "firmware performing VMGEXIT", given that in this
> case it would be the OS's #VC handler executing VMGEXIT.
>
> So, I think the above commit message implies a VMGEXIT *without* a NAE /
> #VC context. (Because, I fail to interpret the commit message in a NAE /
> #VC context in any way; see above.)
Correct.
>
> OK, so let's see where the firmware performs a VMGEXIT *outside* of an
> exception handler, *while* at OS runtime. There seems to be one, in file
> "OvmfPkg/QemuFlashFvbServicesRuntimeDxe/QemuFlashDxe.c":
Again, correct. Basically this is what is invoked when setting UEFI variables.
>
>> VOID
>> QemuFlashPtrWrite (
>> IN volatile UINT8 *Ptr,
>> IN UINT8 Value
>> )
>> {
>> if (MemEncryptSevEsIsEnabled ()) {
>> MSR_SEV_ES_GHCB_REGISTER Msr;
>> GHCB *Ghcb;
>>
>> Msr.GhcbPhysicalAddress = AsmReadMsr64 (MSR_SEV_ES_GHCB);
>> Ghcb = Msr.Ghcb;
>>
>> //
>> // Writing to flash is emulated by the hypervisor through the use of write
>> // protection. This won't work for an SEV-ES guest because the write won't
>> // be recognized as a true MMIO write, which would result in the required
>> // #VC exception. Instead, use the the VMGEXIT MMIO write support directly
>> // to perform the update.
>> //
>> VmgInit (Ghcb);
>> Ghcb->SharedBuffer[0] = Value;
>> Ghcb->SaveArea.SwScratch = (UINT64) (UINTN) Ghcb->SharedBuffer;
>> VmgExit (Ghcb, SVM_EXIT_MMIO_WRITE, (UINT64) (UINTN) Ptr, 1);
>> VmgDone (Ghcb);
>> } else {
>> *Ptr = Value;
>> }
>> }
>
> This function *does* run at OS runtime (as a part of non-volatile UEFI
> variable writes).
>
> And note that, wherever MSR_SEV_ES_GHCB points to at the moment, is used
> as GHCB.
>
> If the guest kernel allocates its own GHCB and writes the allocation
> address to MSR_SEV_ES_GHCB, then indeed the firmware will use the GHCB
> of the OS.
>
> I reviewed edk2 commit 437eb3f7a8db
> ("OvmfPkg/QemuFlashFvbServicesRuntimeDxe: Bypass flash detection with
> SEV-ES", 2020-08-17), but I admit I never thought of the guest OS
> changing MSR_SEV_ES_GHCB. I'm sorry about that.
>
> As long as this driver is running before OS runtime (i.e., during the
> DXE and BDS phases), MSR_SEV_ES_GHCB is supposed to carry the value we
> set in "OvmfPkg/PlatformPei/AmdSev.c":
>
>> STATIC
>> VOID
>> AmdSevEsInitialize (
>> VOID
>> )
>> {
>> VOID *GhcbBase;
>> PHYSICAL_ADDRESS GhcbBasePa;
>> UINTN GhcbPageCount, PageCount;
>> RETURN_STATUS PcdStatus, DecryptStatus;
>> IA32_DESCRIPTOR Gdtr;
>> VOID *Gdt;
>>
>> if (!MemEncryptSevEsIsEnabled ()) {
>> return;
>> }
>>
>> PcdStatus = PcdSetBoolS (PcdSevEsIsEnabled, TRUE);
>> ASSERT_RETURN_ERROR (PcdStatus);
>>
>> //
>> // Allocate GHCB and per-CPU variable pages.
>> // Since the pages must survive across the UEFI to OS transition
>> // make them reserved.
>> //
>> GhcbPageCount = mMaxCpuCount * 2;
>> GhcbBase = AllocateReservedPages (GhcbPageCount);
>> ASSERT (GhcbBase != NULL);
>>
>> GhcbBasePa = (PHYSICAL_ADDRESS)(UINTN) GhcbBase;
>>
>> //
>> // Each vCPU gets two consecutive pages, the first is the GHCB and the
>> // second is the per-CPU variable page. Loop through the allocation and
>> // only clear the encryption mask for the GHCB pages.
>> //
>> for (PageCount = 0; PageCount < GhcbPageCount; PageCount += 2) {
>> DecryptStatus = MemEncryptSevClearPageEncMask (
>> 0,
>> GhcbBasePa + EFI_PAGES_TO_SIZE (PageCount),
>> 1,
>> TRUE
>> );
>> ASSERT_RETURN_ERROR (DecryptStatus);
>> }
>>
>> ZeroMem (GhcbBase, EFI_PAGES_TO_SIZE (GhcbPageCount));
>>
>> PcdStatus = PcdSet64S (PcdGhcbBase, GhcbBasePa);
>> ASSERT_RETURN_ERROR (PcdStatus);
>> PcdStatus = PcdSet64S (PcdGhcbSize, EFI_PAGES_TO_SIZE (GhcbPageCount));
>> ASSERT_RETURN_ERROR (PcdStatus);
>>
>> DEBUG ((DEBUG_INFO,
>> "SEV-ES is enabled, %lu GHCB pages allocated starting at 0x%p\n",
>> (UINT64)GhcbPageCount, GhcbBase));
>>
>> AsmWriteMsr64 (MSR_SEV_ES_GHCB, GhcbBasePa);
>
> So what is the *actual* problem at OS runtime:
>
> - Is it that MSR_SEV_ES_GHCB still points at this PEI-phase *reserved*
> memory allocation (and so when QemuFlashPtrWrite() tries to access it
> during OS runtime, it doesn't have a runtime mapping for it)?
At this point the GHCB MSR points to the OS GHCB, which isn't mapped by
the page tables supplied by the OS and used by UEFI.
>
> - Or is it that the OS actively changes MSR_SEV_ES_GHCB, pointing to a
> memory area that the OS owns -- and *that* area is what
> QemuFlashPtrWrite() cannot access at OS runtime?
Correct.
>
> The first problem statement does *not* seem to apply, given -- again --
> that the commit message says, "firmware is likely to use the GHCB of the
> OS".
>
> So I think the second problem statement must apply.
>
> (I think the "reserved allocation" above is "reserved" only because we
> want to keep the OS out of it around the ExitBootServices() transition.)
>
> Back to the email:
>
> On 09/09/20 10:27, Ard Biesheuvel wrote:
>> On Tue, 8 Sep 2020 at 20:46, Borislav Petkov <bp@alien8.de> wrote:
>>> On Mon, Sep 07, 2020 at 03:16:12PM +0200, Joerg Roedel wrote:
>>>> so each GHCB in the system needs to be identity
>>>> mapped in the EFI page tables, as unencrypted, to avoid page faults.
>
> Not sure I agree about this, but at least it seems to confirm my
> understanding -- apparently the idea is, for the OS, to satisfy
> QemuFlashPtrWrite() in the firmware, by putting the "expected" mapping
> -- for wherever MSR_SEV_ES_GHCB is going to point to -- in place.
>
>>>>
>>>> Signed-off-by: Tom Lendacky <thomas.lendacky@amd.com>
>>>> [ jroedel@suse.de: Moved GHCB mapping loop to sev-es.c ]
>>>> Signed-off-by: Joerg Roedel <jroedel@suse.de>
>>
>>
>> This looks like it is papering over a more fundamental issue: any
>> memory region that the firmware itself needs to access during the
>> execution of runtime services needs to be described in the UEFI memory
>> map, with the appropriate annotations so that the OS knows it should
>> include these in the EFI runtime page tables. So why has this been
>> omitted in this case?
>
> So yeah, the issue seems to be that the QemuFlashFvbServicesRuntimeDxe
> driver does not *own* the GHCB that it attempts to use at OS runtime. It
> doesn't know where MSR_SEV_ES_GHCB is going to point.
>
> Is QemuFlashFvbServicesRuntimeDxe permitted to change MSR_SEV_ES_GHCB
> *temporarily* at OS runtime?
>
> Because, in that case:
>
> - QemuFlashFvbServicesRuntimeDxe should allocate a Runtime Services Data
> block for GHCB when it starts up (if SEV-ES is active),
>
> - QemuFlashFvbServicesRuntimeDxe should register a SetVirtualAddressMap
> handler, and use EfiConvertPointer() from UefiRuntimeLib to convert
> the "runtime GHCB" address to virtual address, in that handler,
>
> - QemuFlashPtrWrite() should call EfiAtRuntime() from UefiRuntimeLib,
> and if the latter returns TRUE, then (a) use the runtime-converted
> address for populating the GHCB, and (b) temporarily swap
> MSR_SEV_ES_GHCB with the address of the self-allocated GHCB. (The MSR
> needs a *physical* address, so QemuFlashFvbServicesRuntimeDxe would
> have to remember / retain the original (physical) allocation address
> too.)
>
> If QemuFlashFvbServicesRuntimeDxe is not permitted to change
> MSR_SEV_ES_GHCB even temporarily (at OS runtime), then I think the
> approach proposed in this (guest kernel) patch is valid.
>
> Let me skim the code below...
>
>>
>>
>>
>>>> ---
>>>> arch/x86/boot/compressed/sev-es.c | 1 +
>>>> arch/x86/include/asm/sev-es.h | 2 ++
>>>> arch/x86/kernel/sev-es.c | 30 ++++++++++++++++++++++++++++++
>>>> arch/x86/platform/efi/efi_64.c | 10 ++++++++++
>>>> 4 files changed, 43 insertions(+)
>>>>
>>>> diff --git a/arch/x86/boot/compressed/sev-es.c b/arch/x86/boot/compressed/sev-es.c
>>>> index 45702b866c33..0a9a248ca33d 100644
>>>> --- a/arch/x86/boot/compressed/sev-es.c
>>>> +++ b/arch/x86/boot/compressed/sev-es.c
>>>> @@ -12,6 +12,7 @@
>>>> */
>>>> #include "misc.h"
>>>>
>>>> +#include <asm/pgtable_types.h>
>>>> #include <asm/sev-es.h>
>>>> #include <asm/trapnr.h>
>>>> #include <asm/trap_pf.h>
>>>> diff --git a/arch/x86/include/asm/sev-es.h b/arch/x86/include/asm/sev-es.h
>>>> index e919f09ae33c..cf1d957c7091 100644
>>>> --- a/arch/x86/include/asm/sev-es.h
>>>> +++ b/arch/x86/include/asm/sev-es.h
>>>> @@ -102,11 +102,13 @@ static __always_inline void sev_es_nmi_complete(void)
>>>> if (static_branch_unlikely(&sev_es_enable_key))
>>>> __sev_es_nmi_complete();
>>>> }
>>>> +extern int __init sev_es_efi_map_ghcbs(pgd_t *pgd);
>>>> #else
>>>> static inline void sev_es_ist_enter(struct pt_regs *regs) { }
>>>> static inline void sev_es_ist_exit(void) { }
>>>> static inline int sev_es_setup_ap_jump_table(struct real_mode_header *rmh) { return 0; }
>>>> static inline void sev_es_nmi_complete(void) { }
>>>> +static inline int sev_es_efi_map_ghcbs(pgd_t *pgd) { return 0; }
>>>> #endif
>>>>
>>>> #endif
>>>> diff --git a/arch/x86/kernel/sev-es.c b/arch/x86/kernel/sev-es.c
>>>> index 9ab3a4dfecd8..4e2b7e4d9b87 100644
>>>> --- a/arch/x86/kernel/sev-es.c
>>>> +++ b/arch/x86/kernel/sev-es.c
>>>> @@ -491,6 +491,36 @@ int sev_es_setup_ap_jump_table(struct real_mode_header *rmh)
>>>> return 0;
>>>> }
>>>>
>>>> +/*
>>>> + * This is needed by the OVMF UEFI firmware which will use whatever it finds in
>>>> + * the GHCB MSR as its GHCB to talk to the hypervisor. So make sure the per-cpu
>>>> + * runtime GHCBs used by the kernel are also mapped in the EFI page-table.
>
> Yup, this pretty much confirms my suspicion that QemuFlashPtrWrite() is
> at the center of this.
>
> (BTW, I don't think that the runtime services data allocation, in
> QemuFlashFvbServicesRuntimeDxe, for OS runtime GHCB purposes, would have
> to be "per CPU". Refer to "Table 35. Rules for Reentry Into Runtime
> Services" in the UEFI spec -- if one processor is executing
> SetVariable(), then no other processor must enter SetVariable(). And so
> we'll have *at most* one VCPU in QemuFlashPtrWrite(), at any time.)
>
>>>> + */
>>>> +int __init sev_es_efi_map_ghcbs(pgd_t *pgd)
>>>> +{
>>>> + struct sev_es_runtime_data *data;
>>>> + unsigned long address, pflags;
>>>> + int cpu;
>>>> + u64 pfn;
>>>> +
>>>> + if (!sev_es_active())
>>>> + return 0;
>>>> +
>>>> + pflags = _PAGE_NX | _PAGE_RW;
>>>> +
>>>> + for_each_possible_cpu(cpu) {
>>>> + data = per_cpu(runtime_data, cpu);
>>>> +
>>>> + address = __pa(&data->ghcb_page);
>>>> + pfn = address >> PAGE_SHIFT;
>>>> +
>>>> + if (kernel_map_pages_in_pgd(pgd, pfn, address, 1, pflags))
>>>> + return 1;
>>>> + }
>>>> +
>>>> + return 0;
>>>> +}
>>>> +
>>>> static enum es_result vc_handle_msr(struct ghcb *ghcb, struct es_em_ctxt *ctxt)
>>>> {
>>>> struct pt_regs *regs = ctxt->regs;
>>>> diff --git a/arch/x86/platform/efi/efi_64.c b/arch/x86/platform/efi/efi_64.c
>>>> index 6af4da1149ba..8f5759df7776 100644
>>>> --- a/arch/x86/platform/efi/efi_64.c
>>>> +++ b/arch/x86/platform/efi/efi_64.c
>>>> @@ -47,6 +47,7 @@
>>>> #include <asm/realmode.h>
>>>> #include <asm/time.h>
>>>> #include <asm/pgalloc.h>
>>>> +#include <asm/sev-es.h>
>>>>
>>>> /*
>>>> * We allocate runtime services regions top-down, starting from -4G, i.e.
>>>> @@ -229,6 +230,15 @@ int __init efi_setup_page_tables(unsigned long pa_memmap, unsigned num_pages)
>>>> return 1;
>>>> }
>>>>
>>>> + /*
>>>> + * When SEV-ES is active, the GHCB as set by the kernel will be used
>>>> + * by firmware. Create a 1:1 unencrypted mapping for each GHCB.
>>>> + */
>>>> + if (sev_es_efi_map_ghcbs(pgd)) {
>>>> + pr_err("Failed to create 1:1 mapping for the GHCBs!\n");
>>>> + return 1;
>>>> + }
>>>> +
>>>> /*
>>>> * When making calls to the firmware everything needs to be 1:1
>>>> * mapped and addressable with 32-bit pointers. Map the kernel
>
> Good point!
>
> And it even makes me wonder if the QemuFlashFvbServicesRuntimeDxe
> approach, with the runtime services data type memory allocation, is
> feasible at all. Namely, a page's encryption status, under SEV, is
> controlled through the PTE.
>
> And for this particular UEFI runtime area, it would *not* suffice for
> the OS to just virt-map it. The OS would also have to *decrypt* the area
> (mark the PTE as "plaintext").
>
> In other words, it would be an "unprecedented" PTE for the OS to set up:
> the PTE would not only map the GVA to GPA, but also mark the area as
> "plaintext".
>
> Otherwise -- if the OS covers *just* the virt-mapping --,
> QemuFlashFvbServicesRuntimeDxe would populate its own "runtime GHCB"
> area just fine, but the actual data hitting the host RAM would be
> encrypted. And so the hypervisor could not interpret the GHCB.
>
> *If* QemuFlashFvbServicesRuntimeDxe should not change the kernel-owned
> PTE at runtime, even temporarily, for marking the GHCB as "plaintext",
> then the problem is indeed only solvable in the guest kernel, in my
> opinion.
>
> There simply isn't an "architected annotation" for telling the kernel,
> "virt-map this runtime services data type memory range, *and* mark it as
> plaintext at the same time".
>
> This would be necessary, as both actions affect the exact same PTE, and
> the firmware is not really allowed to touch the PTE at runtime. But we
> don't have such a hint.
>
>
> To summarize: for QemuFlashFvbServicesRuntimeDxe to allocate UEFI
> Runtime Services Data type memory, for its own runtime GHCB, two
> permissions are necessary (together), at OS runtime:
>
> - QemuFlashFvbServicesRuntimeDxe must be allowed to swap MSR_SEV_ES_GHCB
> temporarily (before executing VMGEXIT),
>
> - QemuFlashFvbServicesRuntimeDxe must be allowed to change the OS-owned
> PTE temporarily (for remapping the GHCB as plaintext, before writing
> to it).
>
Amazing summarization Laszlo!
Thanks,
Tom
> Thanks
> Laszlo
>
^ permalink raw reply [flat|nested] 264+ messages in thread
* Re: [PATCH v7 71/72] x86/efi: Add GHCB mappings when SEV-ES is active
@ 2020-09-09 13:49 ` Tom Lendacky
0 siblings, 0 replies; 264+ messages in thread
From: Tom Lendacky @ 2020-09-09 13:49 UTC (permalink / raw)
To: Laszlo Ersek, Ard Biesheuvel, Borislav Petkov, brijesh.singh
Cc: Juergen Gross, Martin Radev, Dave Hansen, Mike Stunes, Kees Cook,
kvm, Peter Zijlstra, Cfir Cohen, Joerg Roedel, X86 ML,
Linux Kernel Mailing List, Sean Christopherson, virtualization,
Joerg Roedel, Masami Hiramatsu, Andy Lutomirski, H. Peter Anvin,
Erdem Aktas, David Rientjes, Dan Williams, Jiri Slaby
On 9/9/20 7:44 AM, Laszlo Ersek wrote:
> On 09/09/20 10:27, Ard Biesheuvel wrote:
>> (adding Laszlo and Brijesh)
>>
>> On Tue, 8 Sep 2020 at 20:46, Borislav Petkov <bp@alien8.de> wrote:
>>>
>>> + Ard so that he can ack the efi bits.
>>>
>>> On Mon, Sep 07, 2020 at 03:16:12PM +0200, Joerg Roedel wrote:
>>>> From: Tom Lendacky <thomas.lendacky@amd.com>
>>>>
>>>> Calling down to EFI runtime services can result in the firmware
>>>> performing VMGEXIT calls. The firmware is likely to use the GHCB of
>>>> the OS (e.g., for setting EFI variables),
>
> I've had to stare at this for a while.
>
> Because, normally a VMGEXIT is supposed to occur like this:
>
> - guest does something privileged
> - resultant non-automatic exit (NAE) injects a #VC exception
> - exception handler figures out what that privileged thing was
> - exception handler submits request to hypervisor via GHCB contents plus
> VMGEXIT instruction
>
> Point being, the agent that "owns" the exception handler is supposed to
> pre-allocate or otherwise provide the GHCB too, for information passing.
>
> So... what is the particular NAE that occurs during the execution of
> UEFI runtime services (at OS runtime)?
>
> And assuming it occurs, I'm unsure if the exception handler (IDT) at
> that point is owned (temporarily?) by the firmware.
>
> - If the #VC handler comes from the firmware, then I don't know why it
> would use the OS's GHCB.
>
> - If the #VC handler comes from the OS, then I don't understand why the
> commit message says "firmware performing VMGEXIT", given that in this
> case it would be the OS's #VC handler executing VMGEXIT.
>
> So, I think the above commit message implies a VMGEXIT *without* a NAE /
> #VC context. (Because, I fail to interpret the commit message in a NAE /
> #VC context in any way; see above.)
Correct.
>
> OK, so let's see where the firmware performs a VMGEXIT *outside* of an
> exception handler, *while* at OS runtime. There seems to be one, in file
> "OvmfPkg/QemuFlashFvbServicesRuntimeDxe/QemuFlashDxe.c":
Again, correct. Basically this is what is invoked when setting UEFI variables.
>
>> VOID
>> QemuFlashPtrWrite (
>> IN volatile UINT8 *Ptr,
>> IN UINT8 Value
>> )
>> {
>> if (MemEncryptSevEsIsEnabled ()) {
>> MSR_SEV_ES_GHCB_REGISTER Msr;
>> GHCB *Ghcb;
>>
>> Msr.GhcbPhysicalAddress = AsmReadMsr64 (MSR_SEV_ES_GHCB);
>> Ghcb = Msr.Ghcb;
>>
>> //
>> // Writing to flash is emulated by the hypervisor through the use of write
>> // protection. This won't work for an SEV-ES guest because the write won't
>> // be recognized as a true MMIO write, which would result in the required
>> // #VC exception. Instead, use the the VMGEXIT MMIO write support directly
>> // to perform the update.
>> //
>> VmgInit (Ghcb);
>> Ghcb->SharedBuffer[0] = Value;
>> Ghcb->SaveArea.SwScratch = (UINT64) (UINTN) Ghcb->SharedBuffer;
>> VmgExit (Ghcb, SVM_EXIT_MMIO_WRITE, (UINT64) (UINTN) Ptr, 1);
>> VmgDone (Ghcb);
>> } else {
>> *Ptr = Value;
>> }
>> }
>
> This function *does* run at OS runtime (as a part of non-volatile UEFI
> variable writes).
>
> And note that, wherever MSR_SEV_ES_GHCB points to at the moment, is used
> as GHCB.
>
> If the guest kernel allocates its own GHCB and writes the allocation
> address to MSR_SEV_ES_GHCB, then indeed the firmware will use the GHCB
> of the OS.
>
> I reviewed edk2 commit 437eb3f7a8db
> ("OvmfPkg/QemuFlashFvbServicesRuntimeDxe: Bypass flash detection with
> SEV-ES", 2020-08-17), but I admit I never thought of the guest OS
> changing MSR_SEV_ES_GHCB. I'm sorry about that.
>
> As long as this driver is running before OS runtime (i.e., during the
> DXE and BDS phases), MSR_SEV_ES_GHCB is supposed to carry the value we
> set in "OvmfPkg/PlatformPei/AmdSev.c":
>
>> STATIC
>> VOID
>> AmdSevEsInitialize (
>> VOID
>> )
>> {
>> VOID *GhcbBase;
>> PHYSICAL_ADDRESS GhcbBasePa;
>> UINTN GhcbPageCount, PageCount;
>> RETURN_STATUS PcdStatus, DecryptStatus;
>> IA32_DESCRIPTOR Gdtr;
>> VOID *Gdt;
>>
>> if (!MemEncryptSevEsIsEnabled ()) {
>> return;
>> }
>>
>> PcdStatus = PcdSetBoolS (PcdSevEsIsEnabled, TRUE);
>> ASSERT_RETURN_ERROR (PcdStatus);
>>
>> //
>> // Allocate GHCB and per-CPU variable pages.
>> // Since the pages must survive across the UEFI to OS transition
>> // make them reserved.
>> //
>> GhcbPageCount = mMaxCpuCount * 2;
>> GhcbBase = AllocateReservedPages (GhcbPageCount);
>> ASSERT (GhcbBase != NULL);
>>
>> GhcbBasePa = (PHYSICAL_ADDRESS)(UINTN) GhcbBase;
>>
>> //
>> // Each vCPU gets two consecutive pages, the first is the GHCB and the
>> // second is the per-CPU variable page. Loop through the allocation and
>> // only clear the encryption mask for the GHCB pages.
>> //
>> for (PageCount = 0; PageCount < GhcbPageCount; PageCount += 2) {
>> DecryptStatus = MemEncryptSevClearPageEncMask (
>> 0,
>> GhcbBasePa + EFI_PAGES_TO_SIZE (PageCount),
>> 1,
>> TRUE
>> );
>> ASSERT_RETURN_ERROR (DecryptStatus);
>> }
>>
>> ZeroMem (GhcbBase, EFI_PAGES_TO_SIZE (GhcbPageCount));
>>
>> PcdStatus = PcdSet64S (PcdGhcbBase, GhcbBasePa);
>> ASSERT_RETURN_ERROR (PcdStatus);
>> PcdStatus = PcdSet64S (PcdGhcbSize, EFI_PAGES_TO_SIZE (GhcbPageCount));
>> ASSERT_RETURN_ERROR (PcdStatus);
>>
>> DEBUG ((DEBUG_INFO,
>> "SEV-ES is enabled, %lu GHCB pages allocated starting at 0x%p\n",
>> (UINT64)GhcbPageCount, GhcbBase));
>>
>> AsmWriteMsr64 (MSR_SEV_ES_GHCB, GhcbBasePa);
>
> So what is the *actual* problem at OS runtime:
>
> - Is it that MSR_SEV_ES_GHCB still points at this PEI-phase *reserved*
> memory allocation (and so when QemuFlashPtrWrite() tries to access it
> during OS runtime, it doesn't have a runtime mapping for it)?
At this point the GHCB MSR points to the OS GHCB, which isn't mapped by
the page tables supplied by the OS and used by UEFI.
>
> - Or is it that the OS actively changes MSR_SEV_ES_GHCB, pointing to a
> memory area that the OS owns -- and *that* area is what
> QemuFlashPtrWrite() cannot access at OS runtime?
Correct.
>
> The first problem statement does *not* seem to apply, given -- again --
> that the commit message says, "firmware is likely to use the GHCB of the
> OS".
>
> So I think the second problem statement must apply.
>
> (I think the "reserved allocation" above is "reserved" only because we
> want to keep the OS out of it around the ExitBootServices() transition.)
>
> Back to the email:
>
> On 09/09/20 10:27, Ard Biesheuvel wrote:
>> On Tue, 8 Sep 2020 at 20:46, Borislav Petkov <bp@alien8.de> wrote:
>>> On Mon, Sep 07, 2020 at 03:16:12PM +0200, Joerg Roedel wrote:
>>>> so each GHCB in the system needs to be identity
>>>> mapped in the EFI page tables, as unencrypted, to avoid page faults.
>
> Not sure I agree about this, but at least it seems to confirm my
> understanding -- apparently the idea is, for the OS, to satisfy
> QemuFlashPtrWrite() in the firmware, by putting the "expected" mapping
> -- for wherever MSR_SEV_ES_GHCB is going to point to -- in place.
>
>>>>
>>>> Signed-off-by: Tom Lendacky <thomas.lendacky@amd.com>
>>>> [ jroedel@suse.de: Moved GHCB mapping loop to sev-es.c ]
>>>> Signed-off-by: Joerg Roedel <jroedel@suse.de>
>>
>>
>> This looks like it is papering over a more fundamental issue: any
>> memory region that the firmware itself needs to access during the
>> execution of runtime services needs to be described in the UEFI memory
>> map, with the appropriate annotations so that the OS knows it should
>> include these in the EFI runtime page tables. So why has this been
>> omitted in this case?
>
> So yeah, the issue seems to be that the QemuFlashFvbServicesRuntimeDxe
> driver does not *own* the GHCB that it attempts to use at OS runtime. It
> doesn't know where MSR_SEV_ES_GHCB is going to point.
>
> Is QemuFlashFvbServicesRuntimeDxe permitted to change MSR_SEV_ES_GHCB
> *temporarily* at OS runtime?
>
> Because, in that case:
>
> - QemuFlashFvbServicesRuntimeDxe should allocate a Runtime Services Data
> block for GHCB when it starts up (if SEV-ES is active),
>
> - QemuFlashFvbServicesRuntimeDxe should register a SetVirtualAddressMap
> handler, and use EfiConvertPointer() from UefiRuntimeLib to convert
> the "runtime GHCB" address to virtual address, in that handler,
>
> - QemuFlashPtrWrite() should call EfiAtRuntime() from UefiRuntimeLib,
> and if the latter returns TRUE, then (a) use the runtime-converted
> address for populating the GHCB, and (b) temporarily swap
> MSR_SEV_ES_GHCB with the address of the self-allocated GHCB. (The MSR
> needs a *physical* address, so QemuFlashFvbServicesRuntimeDxe would
> have to remember / retain the original (physical) allocation address
> too.)
>
> If QemuFlashFvbServicesRuntimeDxe is not permitted to change
> MSR_SEV_ES_GHCB even temporarily (at OS runtime), then I think the
> approach proposed in this (guest kernel) patch is valid.
>
> Let me skim the code below...
>
>>
>>
>>
>>>> ---
>>>> arch/x86/boot/compressed/sev-es.c | 1 +
>>>> arch/x86/include/asm/sev-es.h | 2 ++
>>>> arch/x86/kernel/sev-es.c | 30 ++++++++++++++++++++++++++++++
>>>> arch/x86/platform/efi/efi_64.c | 10 ++++++++++
>>>> 4 files changed, 43 insertions(+)
>>>>
>>>> diff --git a/arch/x86/boot/compressed/sev-es.c b/arch/x86/boot/compressed/sev-es.c
>>>> index 45702b866c33..0a9a248ca33d 100644
>>>> --- a/arch/x86/boot/compressed/sev-es.c
>>>> +++ b/arch/x86/boot/compressed/sev-es.c
>>>> @@ -12,6 +12,7 @@
>>>> */
>>>> #include "misc.h"
>>>>
>>>> +#include <asm/pgtable_types.h>
>>>> #include <asm/sev-es.h>
>>>> #include <asm/trapnr.h>
>>>> #include <asm/trap_pf.h>
>>>> diff --git a/arch/x86/include/asm/sev-es.h b/arch/x86/include/asm/sev-es.h
>>>> index e919f09ae33c..cf1d957c7091 100644
>>>> --- a/arch/x86/include/asm/sev-es.h
>>>> +++ b/arch/x86/include/asm/sev-es.h
>>>> @@ -102,11 +102,13 @@ static __always_inline void sev_es_nmi_complete(void)
>>>> if (static_branch_unlikely(&sev_es_enable_key))
>>>> __sev_es_nmi_complete();
>>>> }
>>>> +extern int __init sev_es_efi_map_ghcbs(pgd_t *pgd);
>>>> #else
>>>> static inline void sev_es_ist_enter(struct pt_regs *regs) { }
>>>> static inline void sev_es_ist_exit(void) { }
>>>> static inline int sev_es_setup_ap_jump_table(struct real_mode_header *rmh) { return 0; }
>>>> static inline void sev_es_nmi_complete(void) { }
>>>> +static inline int sev_es_efi_map_ghcbs(pgd_t *pgd) { return 0; }
>>>> #endif
>>>>
>>>> #endif
>>>> diff --git a/arch/x86/kernel/sev-es.c b/arch/x86/kernel/sev-es.c
>>>> index 9ab3a4dfecd8..4e2b7e4d9b87 100644
>>>> --- a/arch/x86/kernel/sev-es.c
>>>> +++ b/arch/x86/kernel/sev-es.c
>>>> @@ -491,6 +491,36 @@ int sev_es_setup_ap_jump_table(struct real_mode_header *rmh)
>>>> return 0;
>>>> }
>>>>
>>>> +/*
>>>> + * This is needed by the OVMF UEFI firmware which will use whatever it finds in
>>>> + * the GHCB MSR as its GHCB to talk to the hypervisor. So make sure the per-cpu
>>>> + * runtime GHCBs used by the kernel are also mapped in the EFI page-table.
>
> Yup, this pretty much confirms my suspicion that QemuFlashPtrWrite() is
> at the center of this.
>
> (BTW, I don't think that the runtime services data allocation, in
> QemuFlashFvbServicesRuntimeDxe, for OS runtime GHCB purposes, would have
> to be "per CPU". Refer to "Table 35. Rules for Reentry Into Runtime
> Services" in the UEFI spec -- if one processor is executing
> SetVariable(), then no other processor must enter SetVariable(). And so
> we'll have *at most* one VCPU in QemuFlashPtrWrite(), at any time.)
>
>>>> + */
>>>> +int __init sev_es_efi_map_ghcbs(pgd_t *pgd)
>>>> +{
>>>> + struct sev_es_runtime_data *data;
>>>> + unsigned long address, pflags;
>>>> + int cpu;
>>>> + u64 pfn;
>>>> +
>>>> + if (!sev_es_active())
>>>> + return 0;
>>>> +
>>>> + pflags = _PAGE_NX | _PAGE_RW;
>>>> +
>>>> + for_each_possible_cpu(cpu) {
>>>> + data = per_cpu(runtime_data, cpu);
>>>> +
>>>> + address = __pa(&data->ghcb_page);
>>>> + pfn = address >> PAGE_SHIFT;
>>>> +
>>>> + if (kernel_map_pages_in_pgd(pgd, pfn, address, 1, pflags))
>>>> + return 1;
>>>> + }
>>>> +
>>>> + return 0;
>>>> +}
>>>> +
>>>> static enum es_result vc_handle_msr(struct ghcb *ghcb, struct es_em_ctxt *ctxt)
>>>> {
>>>> struct pt_regs *regs = ctxt->regs;
>>>> diff --git a/arch/x86/platform/efi/efi_64.c b/arch/x86/platform/efi/efi_64.c
>>>> index 6af4da1149ba..8f5759df7776 100644
>>>> --- a/arch/x86/platform/efi/efi_64.c
>>>> +++ b/arch/x86/platform/efi/efi_64.c
>>>> @@ -47,6 +47,7 @@
>>>> #include <asm/realmode.h>
>>>> #include <asm/time.h>
>>>> #include <asm/pgalloc.h>
>>>> +#include <asm/sev-es.h>
>>>>
>>>> /*
>>>> * We allocate runtime services regions top-down, starting from -4G, i.e.
>>>> @@ -229,6 +230,15 @@ int __init efi_setup_page_tables(unsigned long pa_memmap, unsigned num_pages)
>>>> return 1;
>>>> }
>>>>
>>>> + /*
>>>> + * When SEV-ES is active, the GHCB as set by the kernel will be used
>>>> + * by firmware. Create a 1:1 unencrypted mapping for each GHCB.
>>>> + */
>>>> + if (sev_es_efi_map_ghcbs(pgd)) {
>>>> + pr_err("Failed to create 1:1 mapping for the GHCBs!\n");
>>>> + return 1;
>>>> + }
>>>> +
>>>> /*
>>>> * When making calls to the firmware everything needs to be 1:1
>>>> * mapped and addressable with 32-bit pointers. Map the kernel
>
> Good point!
>
> And it even makes me wonder if the QemuFlashFvbServicesRuntimeDxe
> approach, with the runtime services data type memory allocation, is
> feasible at all. Namely, a page's encryption status, under SEV, is
> controlled through the PTE.
>
> And for this particular UEFI runtime area, it would *not* suffice for
> the OS to just virt-map it. The OS would also have to *decrypt* the area
> (mark the PTE as "plaintext").
>
> In other words, it would be an "unprecedented" PTE for the OS to set up:
> the PTE would not only map the GVA to GPA, but also mark the area as
> "plaintext".
>
> Otherwise -- if the OS covers *just* the virt-mapping --,
> QemuFlashFvbServicesRuntimeDxe would populate its own "runtime GHCB"
> area just fine, but the actual data hitting the host RAM would be
> encrypted. And so the hypervisor could not interpret the GHCB.
>
> *If* QemuFlashFvbServicesRuntimeDxe should not change the kernel-owned
> PTE at runtime, even temporarily, for marking the GHCB as "plaintext",
> then the problem is indeed only solvable in the guest kernel, in my
> opinion.
>
> There simply isn't an "architected annotation" for telling the kernel,
> "virt-map this runtime services data type memory range, *and* mark it as
> plaintext at the same time".
>
> This would be necessary, as both actions affect the exact same PTE, and
> the firmware is not really allowed to touch the PTE at runtime. But we
> don't have such a hint.
>
>
> To summarize: for QemuFlashFvbServicesRuntimeDxe to allocate UEFI
> Runtime Services Data type memory, for its own runtime GHCB, two
> permissions are necessary (together), at OS runtime:
>
> - QemuFlashFvbServicesRuntimeDxe must be allowed to swap MSR_SEV_ES_GHCB
> temporarily (before executing VMGEXIT),
>
> - QemuFlashFvbServicesRuntimeDxe must be allowed to change the OS-owned
> PTE temporarily (for remapping the GHCB as plaintext, before writing
> to it).
>
Amazing summarization Laszlo!
Thanks,
Tom
> Thanks
> Laszlo
>
_______________________________________________
Virtualization mailing list
Virtualization@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/virtualization
^ permalink raw reply [flat|nested] 264+ messages in thread
* [tip: x86/seves] x86/efi: Add GHCB mappings when SEV-ES is active
2020-09-07 13:16 ` Joerg Roedel
(?)
(?)
@ 2020-09-10 9:22 ` tip-bot2 for Tom Lendacky
-1 siblings, 0 replies; 264+ messages in thread
From: tip-bot2 for Tom Lendacky @ 2020-09-10 9:22 UTC (permalink / raw)
To: linux-tip-commits; +Cc: Tom Lendacky, Joerg Roedel, Borislav Petkov, x86, LKML
The following commit has been merged into the x86/seves branch of tip:
Commit-ID: 35dcb1ebaf43230637abd2428b9d1fe6c915a78b
Gitweb: https://git.kernel.org/tip/35dcb1ebaf43230637abd2428b9d1fe6c915a78b
Author: Tom Lendacky <thomas.lendacky@amd.com>
AuthorDate: Mon, 07 Sep 2020 15:16:12 +02:00
Committer: Borislav Petkov <bp@suse.de>
CommitterDate: Wed, 09 Sep 2020 18:03:48 +02:00
x86/efi: Add GHCB mappings when SEV-ES is active
Calling down to EFI runtime services can result in the firmware
performing VMGEXIT calls. The firmware is likely to use the GHCB of the
OS (e.g., for setting EFI variables), so each GHCB in the system needs
to be identity-mapped in the EFI page tables, as unencrypted, to avoid
page faults.
Signed-off-by: Tom Lendacky <thomas.lendacky@amd.com>
[ jroedel@suse.de: Moved GHCB mapping loop to sev-es.c ]
Signed-off-by: Joerg Roedel <jroedel@suse.de>
Signed-off-by: Borislav Petkov <bp@suse.de>
Link: https://lkml.kernel.org/r/20200907131613.12703-72-joro@8bytes.org
---
arch/x86/boot/compressed/sev-es.c | 1 +-
arch/x86/include/asm/sev-es.h | 2 ++-
arch/x86/kernel/sev-es.c | 30 ++++++++++++++++++++++++++++++-
arch/x86/platform/efi/efi_64.c | 10 ++++++++++-
4 files changed, 43 insertions(+)
diff --git a/arch/x86/boot/compressed/sev-es.c b/arch/x86/boot/compressed/sev-es.c
index 5f15e58..2a6c7c3 100644
--- a/arch/x86/boot/compressed/sev-es.c
+++ b/arch/x86/boot/compressed/sev-es.c
@@ -12,6 +12,7 @@
*/
#include "misc.h"
+#include <asm/pgtable_types.h>
#include <asm/sev-es.h>
#include <asm/trapnr.h>
#include <asm/trap_pf.h>
diff --git a/arch/x86/include/asm/sev-es.h b/arch/x86/include/asm/sev-es.h
index e919f09..cf1d957 100644
--- a/arch/x86/include/asm/sev-es.h
+++ b/arch/x86/include/asm/sev-es.h
@@ -102,11 +102,13 @@ static __always_inline void sev_es_nmi_complete(void)
if (static_branch_unlikely(&sev_es_enable_key))
__sev_es_nmi_complete();
}
+extern int __init sev_es_efi_map_ghcbs(pgd_t *pgd);
#else
static inline void sev_es_ist_enter(struct pt_regs *regs) { }
static inline void sev_es_ist_exit(void) { }
static inline int sev_es_setup_ap_jump_table(struct real_mode_header *rmh) { return 0; }
static inline void sev_es_nmi_complete(void) { }
+static inline int sev_es_efi_map_ghcbs(pgd_t *pgd) { return 0; }
#endif
#endif
diff --git a/arch/x86/kernel/sev-es.c b/arch/x86/kernel/sev-es.c
index b6518e9..8cac9f8 100644
--- a/arch/x86/kernel/sev-es.c
+++ b/arch/x86/kernel/sev-es.c
@@ -491,6 +491,36 @@ int sev_es_setup_ap_jump_table(struct real_mode_header *rmh)
return 0;
}
+/*
+ * This is needed by the OVMF UEFI firmware which will use whatever it finds in
+ * the GHCB MSR as its GHCB to talk to the hypervisor. So make sure the per-cpu
+ * runtime GHCBs used by the kernel are also mapped in the EFI page-table.
+ */
+int __init sev_es_efi_map_ghcbs(pgd_t *pgd)
+{
+ struct sev_es_runtime_data *data;
+ unsigned long address, pflags;
+ int cpu;
+ u64 pfn;
+
+ if (!sev_es_active())
+ return 0;
+
+ pflags = _PAGE_NX | _PAGE_RW;
+
+ for_each_possible_cpu(cpu) {
+ data = per_cpu(runtime_data, cpu);
+
+ address = __pa(&data->ghcb_page);
+ pfn = address >> PAGE_SHIFT;
+
+ if (kernel_map_pages_in_pgd(pgd, pfn, address, 1, pflags))
+ return 1;
+ }
+
+ return 0;
+}
+
static enum es_result vc_handle_msr(struct ghcb *ghcb, struct es_em_ctxt *ctxt)
{
struct pt_regs *regs = ctxt->regs;
diff --git a/arch/x86/platform/efi/efi_64.c b/arch/x86/platform/efi/efi_64.c
index 6af4da1..8f5759d 100644
--- a/arch/x86/platform/efi/efi_64.c
+++ b/arch/x86/platform/efi/efi_64.c
@@ -47,6 +47,7 @@
#include <asm/realmode.h>
#include <asm/time.h>
#include <asm/pgalloc.h>
+#include <asm/sev-es.h>
/*
* We allocate runtime services regions top-down, starting from -4G, i.e.
@@ -230,6 +231,15 @@ int __init efi_setup_page_tables(unsigned long pa_memmap, unsigned num_pages)
}
/*
+ * When SEV-ES is active, the GHCB as set by the kernel will be used
+ * by firmware. Create a 1:1 unencrypted mapping for each GHCB.
+ */
+ if (sev_es_efi_map_ghcbs(pgd)) {
+ pr_err("Failed to create 1:1 mapping for the GHCBs!\n");
+ return 1;
+ }
+
+ /*
* When making calls to the firmware everything needs to be 1:1
* mapped and addressable with 32-bit pointers. Map the kernel
* text and allocate a new stack because we can't rely on the
^ permalink raw reply related [flat|nested] 264+ messages in thread
* [tip: x86/seves] x86/sev-es: Check required CPU features for SEV-ES
2020-09-07 13:16 ` Joerg Roedel
(?)
@ 2020-09-10 9:22 ` tip-bot2 for Martin Radev
-1 siblings, 0 replies; 264+ messages in thread
From: tip-bot2 for Martin Radev @ 2020-09-10 9:22 UTC (permalink / raw)
To: linux-tip-commits
Cc: Martin Radev, Joerg Roedel, Borislav Petkov, Kees Cook, x86, LKML
The following commit has been merged into the x86/seves branch of tip:
Commit-ID: 1ac0dc494a9796d6f94ff392f16e0c9a8cea7021
Gitweb: https://git.kernel.org/tip/1ac0dc494a9796d6f94ff392f16e0c9a8cea7021
Author: Martin Radev <martin.b.radev@gmail.com>
AuthorDate: Mon, 07 Sep 2020 15:16:13 +02:00
Committer: Borislav Petkov <bp@suse.de>
CommitterDate: Wed, 09 Sep 2020 18:03:48 +02:00
x86/sev-es: Check required CPU features for SEV-ES
Make sure the machine supports RDRAND, otherwise there is no trusted
source of randomness in the system.
To also check this in the pre-decompression stage, make has_cpuflag()
not depend on CONFIG_RANDOMIZE_BASE anymore.
Signed-off-by: Martin Radev <martin.b.radev@gmail.com>
Signed-off-by: Joerg Roedel <jroedel@suse.de>
Signed-off-by: Borislav Petkov <bp@suse.de>
Reviewed-by: Kees Cook <keescook@chromium.org>
Link: https://lkml.kernel.org/r/20200907131613.12703-73-joro@8bytes.org
---
arch/x86/boot/compressed/cpuflags.c | 4 ----
arch/x86/boot/compressed/misc.h | 5 +++--
arch/x86/boot/compressed/sev-es.c | 3 +++
arch/x86/kernel/sev-es-shared.c | 15 +++++++++++++++
arch/x86/kernel/sev-es.c | 3 +++
5 files changed, 24 insertions(+), 6 deletions(-)
diff --git a/arch/x86/boot/compressed/cpuflags.c b/arch/x86/boot/compressed/cpuflags.c
index 6448a81..0cc1323 100644
--- a/arch/x86/boot/compressed/cpuflags.c
+++ b/arch/x86/boot/compressed/cpuflags.c
@@ -1,6 +1,4 @@
// SPDX-License-Identifier: GPL-2.0
-#ifdef CONFIG_RANDOMIZE_BASE
-
#include "../cpuflags.c"
bool has_cpuflag(int flag)
@@ -9,5 +7,3 @@ bool has_cpuflag(int flag)
return test_bit(flag, cpu.flags);
}
-
-#endif
diff --git a/arch/x86/boot/compressed/misc.h b/arch/x86/boot/compressed/misc.h
index c0e0ffe..6d31f1b 100644
--- a/arch/x86/boot/compressed/misc.h
+++ b/arch/x86/boot/compressed/misc.h
@@ -85,8 +85,6 @@ void choose_random_location(unsigned long input,
unsigned long *output,
unsigned long output_size,
unsigned long *virt_addr);
-/* cpuflags.c */
-bool has_cpuflag(int flag);
#else
static inline void choose_random_location(unsigned long input,
unsigned long input_size,
@@ -97,6 +95,9 @@ static inline void choose_random_location(unsigned long input,
}
#endif
+/* cpuflags.c */
+bool has_cpuflag(int flag);
+
#ifdef CONFIG_X86_64
extern int set_page_decrypted(unsigned long address);
extern int set_page_encrypted(unsigned long address);
diff --git a/arch/x86/boot/compressed/sev-es.c b/arch/x86/boot/compressed/sev-es.c
index 2a6c7c3..954cb27 100644
--- a/arch/x86/boot/compressed/sev-es.c
+++ b/arch/x86/boot/compressed/sev-es.c
@@ -145,6 +145,9 @@ void sev_es_shutdown_ghcb(void)
if (!boot_ghcb)
return;
+ if (!sev_es_check_cpu_features())
+ error("SEV-ES CPU Features missing.");
+
/*
* GHCB Page must be flushed from the cache and mapped encrypted again.
* Otherwise the running kernel will see strange cache effects when
diff --git a/arch/x86/kernel/sev-es-shared.c b/arch/x86/kernel/sev-es-shared.c
index 4be8af2..5f83cca 100644
--- a/arch/x86/kernel/sev-es-shared.c
+++ b/arch/x86/kernel/sev-es-shared.c
@@ -9,6 +9,21 @@
* and is included directly into both code-bases.
*/
+#ifndef __BOOT_COMPRESSED
+#define error(v) pr_err(v)
+#define has_cpuflag(f) boot_cpu_has(f)
+#endif
+
+static bool __init sev_es_check_cpu_features(void)
+{
+ if (!has_cpuflag(X86_FEATURE_RDRAND)) {
+ error("RDRAND instruction not supported - no trusted source of randomness available\n");
+ return false;
+ }
+
+ return true;
+}
+
static void sev_es_terminate(unsigned int reason)
{
u64 val = GHCB_SEV_TERMINATE;
diff --git a/arch/x86/kernel/sev-es.c b/arch/x86/kernel/sev-es.c
index 8cac9f8..6fcfdd3 100644
--- a/arch/x86/kernel/sev-es.c
+++ b/arch/x86/kernel/sev-es.c
@@ -665,6 +665,9 @@ void __init sev_es_init_vc_handling(void)
if (!sev_es_active())
return;
+ if (!sev_es_check_cpu_features())
+ panic("SEV-ES CPU Features missing");
+
/* Enable SEV-ES special handling */
static_branch_enable(&sev_es_enable_key);
^ permalink raw reply related [flat|nested] 264+ messages in thread
* [tip: x86/seves] x86/head/64: Don't call verify_cpu() on starting APs
2020-09-07 13:16 ` Joerg Roedel
(?)
@ 2020-09-10 9:22 ` tip-bot2 for Joerg Roedel
-1 siblings, 0 replies; 264+ messages in thread
From: tip-bot2 for Joerg Roedel @ 2020-09-10 9:22 UTC (permalink / raw)
To: linux-tip-commits; +Cc: Joerg Roedel, Borislav Petkov, Kees Cook, x86, LKML
The following commit has been merged into the x86/seves branch of tip:
Commit-ID: 3ecacdbd23956a549d93023f86adc87b4a9d6520
Gitweb: https://git.kernel.org/tip/3ecacdbd23956a549d93023f86adc87b4a9d6520
Author: Joerg Roedel <jroedel@suse.de>
AuthorDate: Mon, 07 Sep 2020 15:16:09 +02:00
Committer: Borislav Petkov <bp@suse.de>
CommitterDate: Wed, 09 Sep 2020 11:33:20 +02:00
x86/head/64: Don't call verify_cpu() on starting APs
The APs are not ready to handle exceptions when verify_cpu() is called
in secondary_startup_64().
Signed-off-by: Joerg Roedel <jroedel@suse.de>
Signed-off-by: Borislav Petkov <bp@suse.de>
Reviewed-by: Kees Cook <keescook@chromium.org>
Link: https://lkml.kernel.org/r/20200907131613.12703-69-joro@8bytes.org
---
arch/x86/include/asm/realmode.h | 1 +
arch/x86/kernel/head_64.S | 12 ++++++++++++
arch/x86/realmode/init.c | 6 ++++++
3 files changed, 19 insertions(+)
diff --git a/arch/x86/include/asm/realmode.h b/arch/x86/include/asm/realmode.h
index 4d4d853..5db5d08 100644
--- a/arch/x86/include/asm/realmode.h
+++ b/arch/x86/include/asm/realmode.h
@@ -72,6 +72,7 @@ extern unsigned char startup_32_smp[];
extern unsigned char boot_gdt[];
#else
extern unsigned char secondary_startup_64[];
+extern unsigned char secondary_startup_64_no_verify[];
#endif
static inline size_t real_mode_size_needed(void)
diff --git a/arch/x86/kernel/head_64.S b/arch/x86/kernel/head_64.S
index 1a71d0d..7eb2a1c 100644
--- a/arch/x86/kernel/head_64.S
+++ b/arch/x86/kernel/head_64.S
@@ -126,6 +126,18 @@ SYM_CODE_START(secondary_startup_64)
call verify_cpu
/*
+ * The secondary_startup_64_no_verify entry point is only used by
+ * SEV-ES guests. In those guests the call to verify_cpu() would cause
+ * #VC exceptions which can not be handled at this stage of secondary
+ * CPU bringup.
+ *
+ * All non SEV-ES systems, especially Intel systems, need to execute
+ * verify_cpu() above to make sure NX is enabled.
+ */
+SYM_INNER_LABEL(secondary_startup_64_no_verify, SYM_L_GLOBAL)
+ UNWIND_HINT_EMPTY
+
+ /*
* Retrieve the modifier (SME encryption mask if SME is active) to be
* added to the initial pgdir entry that will be programmed into CR3.
*/
diff --git a/arch/x86/realmode/init.c b/arch/x86/realmode/init.c
index 3fb9b60..22fda7d 100644
--- a/arch/x86/realmode/init.c
+++ b/arch/x86/realmode/init.c
@@ -46,6 +46,12 @@ static void sme_sev_setup_real_mode(struct trampoline_header *th)
th->flags |= TH_FLAGS_SME_ACTIVE;
if (sev_es_active()) {
+ /*
+ * Skip the call to verify_cpu() in secondary_startup_64 as it
+ * will cause #VC exceptions when the AP can't handle them yet.
+ */
+ th->start = (u64) secondary_startup_64_no_verify;
+
if (sev_es_setup_ap_jump_table(real_mode_header))
panic("Failed to get/update SEV-ES AP Jump Table");
}
^ permalink raw reply related [flat|nested] 264+ messages in thread
* [tip: x86/seves] x86/sev-es: Handle NMI State
2020-09-07 13:16 ` Joerg Roedel
(?)
@ 2020-09-10 9:22 ` tip-bot2 for Joerg Roedel
-1 siblings, 0 replies; 264+ messages in thread
From: tip-bot2 for Joerg Roedel @ 2020-09-10 9:22 UTC (permalink / raw)
To: linux-tip-commits; +Cc: Joerg Roedel, Borislav Petkov, x86, LKML
The following commit has been merged into the x86/seves branch of tip:
Commit-ID: 4ca68e023b11e4d5908bf9ee326fab01111d77d5
Gitweb: https://git.kernel.org/tip/4ca68e023b11e4d5908bf9ee326fab01111d77d5
Author: Joerg Roedel <jroedel@suse.de>
AuthorDate: Mon, 07 Sep 2020 15:16:11 +02:00
Committer: Borislav Petkov <bp@suse.de>
CommitterDate: Wed, 09 Sep 2020 18:02:35 +02:00
x86/sev-es: Handle NMI State
When running under SEV-ES, the kernel has to tell the hypervisor when to
open the NMI window again after an NMI was injected. This is done with
an NMI-complete message to the hypervisor.
Add code to the kernel's NMI handler to send this message right at the
beginning of do_nmi(). This always allows nesting NMIs.
[ bp: Mark __sev_es_nmi_complete() noinstr:
vmlinux.o: warning: objtool: exc_nmi()+0x17: call to __sev_es_nmi_complete()
leaves .noinstr.text section
While at it, use __pa_nodebug() for the same reason due to
CONFIG_DEBUG_VIRTUAL=y:
vmlinux.o: warning: objtool: __sev_es_nmi_complete()+0xd9: call to __phys_addr()
leaves .noinstr.text section ]
Signed-off-by: Joerg Roedel <jroedel@suse.de>
Signed-off-by: Borislav Petkov <bp@suse.de>
Link: https://lkml.kernel.org/r/20200907131613.12703-71-joro@8bytes.org
---
arch/x86/include/asm/sev-es.h | 7 +++++++
arch/x86/include/uapi/asm/svm.h | 1 +
arch/x86/kernel/nmi.c | 6 ++++++
arch/x86/kernel/sev-es.c | 18 ++++++++++++++++++
4 files changed, 32 insertions(+)
diff --git a/arch/x86/include/asm/sev-es.h b/arch/x86/include/asm/sev-es.h
index db88e1c..e919f09 100644
--- a/arch/x86/include/asm/sev-es.h
+++ b/arch/x86/include/asm/sev-es.h
@@ -96,10 +96,17 @@ static __always_inline void sev_es_ist_exit(void)
__sev_es_ist_exit();
}
extern int sev_es_setup_ap_jump_table(struct real_mode_header *rmh);
+extern void __sev_es_nmi_complete(void);
+static __always_inline void sev_es_nmi_complete(void)
+{
+ if (static_branch_unlikely(&sev_es_enable_key))
+ __sev_es_nmi_complete();
+}
#else
static inline void sev_es_ist_enter(struct pt_regs *regs) { }
static inline void sev_es_ist_exit(void) { }
static inline int sev_es_setup_ap_jump_table(struct real_mode_header *rmh) { return 0; }
+static inline void sev_es_nmi_complete(void) { }
#endif
#endif
diff --git a/arch/x86/include/uapi/asm/svm.h b/arch/x86/include/uapi/asm/svm.h
index c1dcf3e..a7a3403 100644
--- a/arch/x86/include/uapi/asm/svm.h
+++ b/arch/x86/include/uapi/asm/svm.h
@@ -84,6 +84,7 @@
/* SEV-ES software-defined VMGEXIT events */
#define SVM_VMGEXIT_MMIO_READ 0x80000001
#define SVM_VMGEXIT_MMIO_WRITE 0x80000002
+#define SVM_VMGEXIT_NMI_COMPLETE 0x80000003
#define SVM_VMGEXIT_AP_HLT_LOOP 0x80000004
#define SVM_VMGEXIT_AP_JUMP_TABLE 0x80000005
#define SVM_VMGEXIT_SET_AP_JUMP_TABLE 0
diff --git a/arch/x86/kernel/nmi.c b/arch/x86/kernel/nmi.c
index 4c89c4d..56b64d7 100644
--- a/arch/x86/kernel/nmi.c
+++ b/arch/x86/kernel/nmi.c
@@ -478,6 +478,12 @@ DEFINE_IDTENTRY_RAW(exc_nmi)
{
bool irq_state;
+ /*
+ * Re-enable NMIs right here when running as an SEV-ES guest. This might
+ * cause nested NMIs, but those can be handled safely.
+ */
+ sev_es_nmi_complete();
+
if (IS_ENABLED(CONFIG_SMP) && arch_cpu_is_offline(smp_processor_id()))
return;
diff --git a/arch/x86/kernel/sev-es.c b/arch/x86/kernel/sev-es.c
index d1bcd0d..b6518e9 100644
--- a/arch/x86/kernel/sev-es.c
+++ b/arch/x86/kernel/sev-es.c
@@ -408,6 +408,24 @@ static bool vc_slow_virt_to_phys(struct ghcb *ghcb, struct es_em_ctxt *ctxt,
/* Include code shared with pre-decompression boot stage */
#include "sev-es-shared.c"
+void noinstr __sev_es_nmi_complete(void)
+{
+ struct ghcb_state state;
+ struct ghcb *ghcb;
+
+ ghcb = sev_es_get_ghcb(&state);
+
+ vc_ghcb_invalidate(ghcb);
+ ghcb_set_sw_exit_code(ghcb, SVM_VMGEXIT_NMI_COMPLETE);
+ ghcb_set_sw_exit_info_1(ghcb, 0);
+ ghcb_set_sw_exit_info_2(ghcb, 0);
+
+ sev_es_wr_ghcb_msr(__pa_nodebug(ghcb));
+ VMGEXIT();
+
+ sev_es_put_ghcb(&state);
+}
+
static u64 get_jump_table_addr(void)
{
struct ghcb_state state;
^ permalink raw reply related [flat|nested] 264+ messages in thread
* [tip: x86/seves] x86/sev-es: Support CPU offline/online
2020-09-07 13:16 ` Joerg Roedel
(?)
@ 2020-09-10 9:22 ` tip-bot2 for Joerg Roedel
-1 siblings, 0 replies; 264+ messages in thread
From: tip-bot2 for Joerg Roedel @ 2020-09-10 9:22 UTC (permalink / raw)
To: linux-tip-commits; +Cc: Joerg Roedel, Borislav Petkov, x86, LKML
The following commit has been merged into the x86/seves branch of tip:
Commit-ID: 094794f59720d7e877a1eeb372ecedeed6b441ab
Gitweb: https://git.kernel.org/tip/094794f59720d7e877a1eeb372ecedeed6b441ab
Author: Joerg Roedel <jroedel@suse.de>
AuthorDate: Mon, 07 Sep 2020 15:16:10 +02:00
Committer: Borislav Petkov <bp@suse.de>
CommitterDate: Wed, 09 Sep 2020 11:33:20 +02:00
x86/sev-es: Support CPU offline/online
Add a play_dead handler when running under SEV-ES. This is needed
because the hypervisor can't deliver an SIPI request to restart the AP.
Instead, the kernel has to issue a VMGEXIT to halt the VCPU until the
hypervisor wakes it up again.
Signed-off-by: Joerg Roedel <jroedel@suse.de>
Signed-off-by: Borislav Petkov <bp@suse.de>
Link: https://lkml.kernel.org/r/20200907131613.12703-70-joro@8bytes.org
---
arch/x86/include/uapi/asm/svm.h | 1 +-
arch/x86/kernel/sev-es.c | 63 ++++++++++++++++++++++++++++++++-
2 files changed, 64 insertions(+)
diff --git a/arch/x86/include/uapi/asm/svm.h b/arch/x86/include/uapi/asm/svm.h
index 346b8a7..c1dcf3e 100644
--- a/arch/x86/include/uapi/asm/svm.h
+++ b/arch/x86/include/uapi/asm/svm.h
@@ -84,6 +84,7 @@
/* SEV-ES software-defined VMGEXIT events */
#define SVM_VMGEXIT_MMIO_READ 0x80000001
#define SVM_VMGEXIT_MMIO_WRITE 0x80000002
+#define SVM_VMGEXIT_AP_HLT_LOOP 0x80000004
#define SVM_VMGEXIT_AP_JUMP_TABLE 0x80000005
#define SVM_VMGEXIT_SET_AP_JUMP_TABLE 0
#define SVM_VMGEXIT_GET_AP_JUMP_TABLE 1
diff --git a/arch/x86/kernel/sev-es.c b/arch/x86/kernel/sev-es.c
index 9ad36ce..d1bcd0d 100644
--- a/arch/x86/kernel/sev-es.c
+++ b/arch/x86/kernel/sev-es.c
@@ -29,6 +29,8 @@
#include <asm/realmode.h>
#include <asm/traps.h>
#include <asm/svm.h>
+#include <asm/smp.h>
+#include <asm/cpu.h>
#define DR7_RESET_VALUE 0x400
@@ -518,6 +520,65 @@ static bool __init sev_es_setup_ghcb(void)
return true;
}
+#ifdef CONFIG_HOTPLUG_CPU
+static void sev_es_ap_hlt_loop(void)
+{
+ struct ghcb_state state;
+ struct ghcb *ghcb;
+
+ ghcb = sev_es_get_ghcb(&state);
+
+ while (true) {
+ vc_ghcb_invalidate(ghcb);
+ ghcb_set_sw_exit_code(ghcb, SVM_VMGEXIT_AP_HLT_LOOP);
+ ghcb_set_sw_exit_info_1(ghcb, 0);
+ ghcb_set_sw_exit_info_2(ghcb, 0);
+
+ sev_es_wr_ghcb_msr(__pa(ghcb));
+ VMGEXIT();
+
+ /* Wakeup signal? */
+ if (ghcb_sw_exit_info_2_is_valid(ghcb) &&
+ ghcb->save.sw_exit_info_2)
+ break;
+ }
+
+ sev_es_put_ghcb(&state);
+}
+
+/*
+ * Play_dead handler when running under SEV-ES. This is needed because
+ * the hypervisor can't deliver an SIPI request to restart the AP.
+ * Instead the kernel has to issue a VMGEXIT to halt the VCPU until the
+ * hypervisor wakes it up again.
+ */
+static void sev_es_play_dead(void)
+{
+ play_dead_common();
+
+ /* IRQs now disabled */
+
+ sev_es_ap_hlt_loop();
+
+ /*
+ * If we get here, the VCPU was woken up again. Jump to CPU
+ * startup code to get it back online.
+ */
+ start_cpu0();
+}
+#else /* CONFIG_HOTPLUG_CPU */
+#define sev_es_play_dead native_play_dead
+#endif /* CONFIG_HOTPLUG_CPU */
+
+#ifdef CONFIG_SMP
+static void __init sev_es_setup_play_dead(void)
+{
+ smp_ops.play_dead = sev_es_play_dead;
+}
+#else
+static inline void sev_es_setup_play_dead(void) { }
+#endif
+
static void __init alloc_runtime_data(int cpu)
{
struct sev_es_runtime_data *data;
@@ -566,6 +627,8 @@ void __init sev_es_init_vc_handling(void)
setup_vc_stacks(cpu);
}
+ sev_es_setup_play_dead();
+
/* Secondary CPUs use the runtime #VC handler */
initial_vc_handler = (unsigned long)safe_stack_exc_vmm_communication;
}
^ permalink raw reply related [flat|nested] 264+ messages in thread
* [tip: x86/seves] x86/realmode: Setup AP jump table
2020-09-07 13:16 ` Joerg Roedel
(?)
@ 2020-09-10 9:22 ` tip-bot2 for Tom Lendacky
-1 siblings, 0 replies; 264+ messages in thread
From: tip-bot2 for Tom Lendacky @ 2020-09-10 9:22 UTC (permalink / raw)
To: linux-tip-commits; +Cc: Tom Lendacky, Joerg Roedel, Borislav Petkov, x86, LKML
The following commit has been merged into the x86/seves branch of tip:
Commit-ID: 8940ac9ced8bc1c48c4e28b0784e3234c9d14469
Gitweb: https://git.kernel.org/tip/8940ac9ced8bc1c48c4e28b0784e3234c9d14469
Author: Tom Lendacky <thomas.lendacky@amd.com>
AuthorDate: Mon, 07 Sep 2020 15:16:07 +02:00
Committer: Borislav Petkov <bp@suse.de>
CommitterDate: Wed, 09 Sep 2020 11:33:20 +02:00
x86/realmode: Setup AP jump table
As part of the GHCB specification, the booting of APs under SEV-ES
requires an AP jump table when transitioning from one layer of code to
another (e.g. when going from UEFI to the OS). As a result, each layer
that parks an AP must provide the physical address of an AP jump table
to the next layer via the hypervisor.
Upon booting of the kernel, read the AP jump table address from the
hypervisor. Under SEV-ES, APs are started using the INIT-SIPI-SIPI
sequence. Before issuing the first SIPI request for an AP, the start
CS and IP is programmed into the AP jump table. Upon issuing the SIPI
request, the AP will awaken and jump to that start CS:IP address.
Signed-off-by: Tom Lendacky <thomas.lendacky@amd.com>
[ jroedel@suse.de: - Adapted to different code base
- Moved AP table setup from SIPI sending path to
real-mode setup code
- Fix sparse warnings ]
Co-developed-by: Joerg Roedel <jroedel@suse.de>
Signed-off-by: Joerg Roedel <jroedel@suse.de>
Signed-off-by: Borislav Petkov <bp@suse.de>
Link: https://lkml.kernel.org/r/20200907131613.12703-67-joro@8bytes.org
---
arch/x86/include/asm/sev-es.h | 5 ++-
arch/x86/include/uapi/asm/svm.h | 3 +-
arch/x86/kernel/sev-es.c | 69 ++++++++++++++++++++++++++++++++-
arch/x86/realmode/init.c | 18 +++++++-
4 files changed, 93 insertions(+), 2 deletions(-)
diff --git a/arch/x86/include/asm/sev-es.h b/arch/x86/include/asm/sev-es.h
index 59176e8..db88e1c 100644
--- a/arch/x86/include/asm/sev-es.h
+++ b/arch/x86/include/asm/sev-es.h
@@ -73,6 +73,9 @@ static inline u64 lower_bits(u64 val, unsigned int bits)
return (val & mask);
}
+struct real_mode_header;
+enum stack_type;
+
/* Early IDT entry points for #VC handler */
extern void vc_no_ghcb(void);
extern void vc_boot_ghcb(void);
@@ -92,9 +95,11 @@ static __always_inline void sev_es_ist_exit(void)
if (static_branch_unlikely(&sev_es_enable_key))
__sev_es_ist_exit();
}
+extern int sev_es_setup_ap_jump_table(struct real_mode_header *rmh);
#else
static inline void sev_es_ist_enter(struct pt_regs *regs) { }
static inline void sev_es_ist_exit(void) { }
+static inline int sev_es_setup_ap_jump_table(struct real_mode_header *rmh) { return 0; }
#endif
#endif
diff --git a/arch/x86/include/uapi/asm/svm.h b/arch/x86/include/uapi/asm/svm.h
index 8f36ae0..346b8a7 100644
--- a/arch/x86/include/uapi/asm/svm.h
+++ b/arch/x86/include/uapi/asm/svm.h
@@ -84,6 +84,9 @@
/* SEV-ES software-defined VMGEXIT events */
#define SVM_VMGEXIT_MMIO_READ 0x80000001
#define SVM_VMGEXIT_MMIO_WRITE 0x80000002
+#define SVM_VMGEXIT_AP_JUMP_TABLE 0x80000005
+#define SVM_VMGEXIT_SET_AP_JUMP_TABLE 0
+#define SVM_VMGEXIT_GET_AP_JUMP_TABLE 1
#define SVM_VMGEXIT_UNSUPPORTED_EVENT 0x8000ffff
#define SVM_EXIT_ERR -1
diff --git a/arch/x86/kernel/sev-es.c b/arch/x86/kernel/sev-es.c
index 6d89df9..9ad36ce 100644
--- a/arch/x86/kernel/sev-es.c
+++ b/arch/x86/kernel/sev-es.c
@@ -21,6 +21,7 @@
#include <linux/mm.h>
#include <asm/cpu_entry_area.h>
+#include <asm/stacktrace.h>
#include <asm/sev-es.h>
#include <asm/insn-eval.h>
#include <asm/fpu/internal.h>
@@ -214,6 +215,9 @@ static __always_inline void sev_es_put_ghcb(struct ghcb_state *state)
}
}
+/* Needed in vc_early_forward_exception */
+void do_early_exception(struct pt_regs *regs, int trapnr);
+
static inline u64 sev_es_rd_ghcb_msr(void)
{
return __rdmsr(MSR_AMD64_SEV_ES_GHCB);
@@ -402,6 +406,71 @@ static bool vc_slow_virt_to_phys(struct ghcb *ghcb, struct es_em_ctxt *ctxt,
/* Include code shared with pre-decompression boot stage */
#include "sev-es-shared.c"
+static u64 get_jump_table_addr(void)
+{
+ struct ghcb_state state;
+ unsigned long flags;
+ struct ghcb *ghcb;
+ u64 ret = 0;
+
+ local_irq_save(flags);
+
+ ghcb = sev_es_get_ghcb(&state);
+
+ vc_ghcb_invalidate(ghcb);
+ ghcb_set_sw_exit_code(ghcb, SVM_VMGEXIT_AP_JUMP_TABLE);
+ ghcb_set_sw_exit_info_1(ghcb, SVM_VMGEXIT_GET_AP_JUMP_TABLE);
+ ghcb_set_sw_exit_info_2(ghcb, 0);
+
+ sev_es_wr_ghcb_msr(__pa(ghcb));
+ VMGEXIT();
+
+ if (ghcb_sw_exit_info_1_is_valid(ghcb) &&
+ ghcb_sw_exit_info_2_is_valid(ghcb))
+ ret = ghcb->save.sw_exit_info_2;
+
+ sev_es_put_ghcb(&state);
+
+ local_irq_restore(flags);
+
+ return ret;
+}
+
+int sev_es_setup_ap_jump_table(struct real_mode_header *rmh)
+{
+ u16 startup_cs, startup_ip;
+ phys_addr_t jump_table_pa;
+ u64 jump_table_addr;
+ u16 __iomem *jump_table;
+
+ jump_table_addr = get_jump_table_addr();
+
+ /* On UP guests there is no jump table so this is not a failure */
+ if (!jump_table_addr)
+ return 0;
+
+ /* Check if AP Jump Table is page-aligned */
+ if (jump_table_addr & ~PAGE_MASK)
+ return -EINVAL;
+
+ jump_table_pa = jump_table_addr & PAGE_MASK;
+
+ startup_cs = (u16)(rmh->trampoline_start >> 4);
+ startup_ip = (u16)(rmh->sev_es_trampoline_start -
+ rmh->trampoline_start);
+
+ jump_table = ioremap_encrypted(jump_table_pa, PAGE_SIZE);
+ if (!jump_table)
+ return -EIO;
+
+ writew(startup_ip, &jump_table[0]);
+ writew(startup_cs, &jump_table[1]);
+
+ iounmap(jump_table);
+
+ return 0;
+}
+
static enum es_result vc_handle_msr(struct ghcb *ghcb, struct es_em_ctxt *ctxt)
{
struct pt_regs *regs = ctxt->regs;
diff --git a/arch/x86/realmode/init.c b/arch/x86/realmode/init.c
index 1ed1208..3fb9b60 100644
--- a/arch/x86/realmode/init.c
+++ b/arch/x86/realmode/init.c
@@ -9,6 +9,7 @@
#include <asm/realmode.h>
#include <asm/tlbflush.h>
#include <asm/crash.h>
+#include <asm/sev-es.h>
struct real_mode_header *real_mode_header;
u32 *trampoline_cr4_features;
@@ -38,6 +39,19 @@ void __init reserve_real_mode(void)
crash_reserve_low_1M();
}
+static void sme_sev_setup_real_mode(struct trampoline_header *th)
+{
+#ifdef CONFIG_AMD_MEM_ENCRYPT
+ if (sme_active())
+ th->flags |= TH_FLAGS_SME_ACTIVE;
+
+ if (sev_es_active()) {
+ if (sev_es_setup_ap_jump_table(real_mode_header))
+ panic("Failed to get/update SEV-ES AP Jump Table");
+ }
+#endif
+}
+
static void __init setup_real_mode(void)
{
u16 real_mode_seg;
@@ -104,13 +118,13 @@ static void __init setup_real_mode(void)
*trampoline_cr4_features = mmu_cr4_features;
trampoline_header->flags = 0;
- if (sme_active())
- trampoline_header->flags |= TH_FLAGS_SME_ACTIVE;
trampoline_pgd = (u64 *) __va(real_mode_header->trampoline_pgd);
trampoline_pgd[0] = trampoline_pgd_entry.pgd;
trampoline_pgd[511] = init_top_pgt[511].pgd;
#endif
+
+ sme_sev_setup_real_mode(trampoline_header);
}
/*
^ permalink raw reply related [flat|nested] 264+ messages in thread
* [tip: x86/seves] x86/smpboot: Load TSS and getcpu GDT entry before loading IDT
2020-09-07 13:16 ` Joerg Roedel
(?)
(?)
@ 2020-09-10 9:22 ` tip-bot2 for Joerg Roedel
-1 siblings, 0 replies; 264+ messages in thread
From: tip-bot2 for Joerg Roedel @ 2020-09-10 9:22 UTC (permalink / raw)
To: linux-tip-commits; +Cc: Joerg Roedel, Borislav Petkov, x86, LKML
The following commit has been merged into the x86/seves branch of tip:
Commit-ID: 520d030852b4c9babfce9a79d8b5320b6b5545e6
Gitweb: https://git.kernel.org/tip/520d030852b4c9babfce9a79d8b5320b6b5545e6
Author: Joerg Roedel <jroedel@suse.de>
AuthorDate: Mon, 07 Sep 2020 15:16:08 +02:00
Committer: Borislav Petkov <bp@suse.de>
CommitterDate: Wed, 09 Sep 2020 11:33:20 +02:00
x86/smpboot: Load TSS and getcpu GDT entry before loading IDT
The IDT on 64-bit contains vectors which use paranoid_entry() and/or IST
stacks. To make these vectors work, the TSS and the getcpu GDT entry need
to be set up before the IDT is loaded.
Signed-off-by: Joerg Roedel <jroedel@suse.de>
Signed-off-by: Borislav Petkov <bp@suse.de>
Link: https://lkml.kernel.org/r/20200907131613.12703-68-joro@8bytes.org
---
arch/x86/include/asm/processor.h | 1 +
arch/x86/kernel/cpu/common.c | 23 +++++++++++++++++++++++
arch/x86/kernel/smpboot.c | 2 +-
3 files changed, 25 insertions(+), 1 deletion(-)
diff --git a/arch/x86/include/asm/processor.h b/arch/x86/include/asm/processor.h
index 97143d8..615dd44 100644
--- a/arch/x86/include/asm/processor.h
+++ b/arch/x86/include/asm/processor.h
@@ -696,6 +696,7 @@ extern void load_direct_gdt(int);
extern void load_fixmap_gdt(int);
extern void load_percpu_segment(int);
extern void cpu_init(void);
+extern void cpu_init_exception_handling(void);
extern void cr4_init(void);
static inline unsigned long get_debugctlmsr(void)
diff --git a/arch/x86/kernel/cpu/common.c b/arch/x86/kernel/cpu/common.c
index 81fba4d..beffea2 100644
--- a/arch/x86/kernel/cpu/common.c
+++ b/arch/x86/kernel/cpu/common.c
@@ -1863,6 +1863,29 @@ static inline void tss_setup_io_bitmap(struct tss_struct *tss)
}
/*
+ * Setup everything needed to handle exceptions from the IDT, including the IST
+ * exceptions which use paranoid_entry().
+ */
+void cpu_init_exception_handling(void)
+{
+ struct tss_struct *tss = this_cpu_ptr(&cpu_tss_rw);
+ int cpu = raw_smp_processor_id();
+
+ /* paranoid_entry() gets the CPU number from the GDT */
+ setup_getcpu(cpu);
+
+ /* IST vectors need TSS to be set up. */
+ tss_setup_ist(tss);
+ tss_setup_io_bitmap(tss);
+ set_tss_desc(cpu, &get_cpu_entry_area(cpu)->tss.x86_tss);
+
+ load_TR_desc();
+
+ /* Finally load the IDT */
+ load_current_idt();
+}
+
+/*
* cpu_init() initializes state that is per-CPU. Some data is already
* initialized (naturally) in the bootstrap process, such as the GDT
* and IDT. We reload them nevertheless, this function acts as a
diff --git a/arch/x86/kernel/smpboot.c b/arch/x86/kernel/smpboot.c
index f5ef689..de776b2 100644
--- a/arch/x86/kernel/smpboot.c
+++ b/arch/x86/kernel/smpboot.c
@@ -227,7 +227,7 @@ static void notrace start_secondary(void *unused)
load_cr3(swapper_pg_dir);
__flush_tlb_all();
#endif
- load_current_idt();
+ cpu_init_exception_handling();
cpu_init();
x86_cpuinit.early_percpu_clock_init();
preempt_disable();
^ permalink raw reply related [flat|nested] 264+ messages in thread
* [tip: x86/seves] x86/vmware: Add VMware-specific handling for VMMCALL under SEV-ES
2020-09-07 13:16 ` Joerg Roedel
(?)
@ 2020-09-10 9:22 ` tip-bot2 for Doug Covelli
2020-10-27 23:19 ` Erdem Aktas
-1 siblings, 1 reply; 264+ messages in thread
From: tip-bot2 for Doug Covelli @ 2020-09-10 9:22 UTC (permalink / raw)
To: linux-tip-commits
Cc: Doug Covelli, Tom Lendacky, Joerg Roedel, Borislav Petkov, x86, LKML
The following commit has been merged into the x86/seves branch of tip:
Commit-ID: 1a222de8dcfb903d039810b0823570ee0be4e6c6
Gitweb: https://git.kernel.org/tip/1a222de8dcfb903d039810b0823570ee0be4e6c6
Author: Doug Covelli <dcovelli@vmware.com>
AuthorDate: Mon, 07 Sep 2020 15:16:05 +02:00
Committer: Borislav Petkov <bp@suse.de>
CommitterDate: Wed, 09 Sep 2020 11:33:20 +02:00
x86/vmware: Add VMware-specific handling for VMMCALL under SEV-ES
Add VMware-specific handling for #VC faults caused by VMMCALL
instructions.
Signed-off-by: Doug Covelli <dcovelli@vmware.com>
Signed-off-by: Tom Lendacky <thomas.lendacky@amd.com>
[ jroedel@suse.de: - Adapt to different paravirt interface ]
Co-developed-by: Joerg Roedel <jroedel@suse.de>
Signed-off-by: Joerg Roedel <jroedel@suse.de>
Signed-off-by: Borislav Petkov <bp@suse.de>
Link: https://lkml.kernel.org/r/20200907131613.12703-65-joro@8bytes.org
---
arch/x86/kernel/cpu/vmware.c | 50 +++++++++++++++++++++++++++++++----
1 file changed, 45 insertions(+), 5 deletions(-)
diff --git a/arch/x86/kernel/cpu/vmware.c b/arch/x86/kernel/cpu/vmware.c
index 9b6fafa..924571f 100644
--- a/arch/x86/kernel/cpu/vmware.c
+++ b/arch/x86/kernel/cpu/vmware.c
@@ -33,6 +33,7 @@
#include <asm/timer.h>
#include <asm/apic.h>
#include <asm/vmware.h>
+#include <asm/svm.h>
#undef pr_fmt
#define pr_fmt(fmt) "vmware: " fmt
@@ -476,10 +477,49 @@ static bool __init vmware_legacy_x2apic_available(void)
(eax & (1 << VMWARE_CMD_LEGACY_X2APIC)) != 0;
}
+#ifdef CONFIG_AMD_MEM_ENCRYPT
+static void vmware_sev_es_hcall_prepare(struct ghcb *ghcb,
+ struct pt_regs *regs)
+{
+ /* Copy VMWARE specific Hypercall parameters to the GHCB */
+ ghcb_set_rip(ghcb, regs->ip);
+ ghcb_set_rbx(ghcb, regs->bx);
+ ghcb_set_rcx(ghcb, regs->cx);
+ ghcb_set_rdx(ghcb, regs->dx);
+ ghcb_set_rsi(ghcb, regs->si);
+ ghcb_set_rdi(ghcb, regs->di);
+ ghcb_set_rbp(ghcb, regs->bp);
+}
+
+static bool vmware_sev_es_hcall_finish(struct ghcb *ghcb, struct pt_regs *regs)
+{
+ if (!(ghcb_rbx_is_valid(ghcb) &&
+ ghcb_rcx_is_valid(ghcb) &&
+ ghcb_rdx_is_valid(ghcb) &&
+ ghcb_rsi_is_valid(ghcb) &&
+ ghcb_rdi_is_valid(ghcb) &&
+ ghcb_rbp_is_valid(ghcb)))
+ return false;
+
+ regs->bx = ghcb->save.rbx;
+ regs->cx = ghcb->save.rcx;
+ regs->dx = ghcb->save.rdx;
+ regs->si = ghcb->save.rsi;
+ regs->di = ghcb->save.rdi;
+ regs->bp = ghcb->save.rbp;
+
+ return true;
+}
+#endif
+
const __initconst struct hypervisor_x86 x86_hyper_vmware = {
- .name = "VMware",
- .detect = vmware_platform,
- .type = X86_HYPER_VMWARE,
- .init.init_platform = vmware_platform_setup,
- .init.x2apic_available = vmware_legacy_x2apic_available,
+ .name = "VMware",
+ .detect = vmware_platform,
+ .type = X86_HYPER_VMWARE,
+ .init.init_platform = vmware_platform_setup,
+ .init.x2apic_available = vmware_legacy_x2apic_available,
+#ifdef CONFIG_AMD_MEM_ENCRYPT
+ .runtime.sev_es_hcall_prepare = vmware_sev_es_hcall_prepare,
+ .runtime.sev_es_hcall_finish = vmware_sev_es_hcall_finish,
+#endif
};
^ permalink raw reply related [flat|nested] 264+ messages in thread
* [tip: x86/seves] x86/realmode: Add SEV-ES specific trampoline entry point
2020-09-07 13:16 ` Joerg Roedel
(?)
@ 2020-09-10 9:22 ` tip-bot2 for Joerg Roedel
-1 siblings, 0 replies; 264+ messages in thread
From: tip-bot2 for Joerg Roedel @ 2020-09-10 9:22 UTC (permalink / raw)
To: linux-tip-commits; +Cc: Joerg Roedel, Borislav Petkov, x86, LKML
The following commit has been merged into the x86/seves branch of tip:
Commit-ID: bf5ff276448f64f1f9ef9ffc9e231026e3887d3d
Gitweb: https://git.kernel.org/tip/bf5ff276448f64f1f9ef9ffc9e231026e3887d3d
Author: Joerg Roedel <jroedel@suse.de>
AuthorDate: Mon, 07 Sep 2020 15:16:06 +02:00
Committer: Borislav Petkov <bp@suse.de>
CommitterDate: Wed, 09 Sep 2020 11:33:20 +02:00
x86/realmode: Add SEV-ES specific trampoline entry point
The code at the trampoline entry point is executed in real-mode. In
real-mode, #VC exceptions can't be handled so anything that might cause
such an exception must be avoided.
In the standard trampoline entry code this is the WBINVD instruction and
the call to verify_cpu(), which are both not needed anyway when running
as an SEV-ES guest.
Signed-off-by: Joerg Roedel <jroedel@suse.de>
Signed-off-by: Borislav Petkov <bp@suse.de>
Link: https://lkml.kernel.org/r/20200907131613.12703-66-joro@8bytes.org
---
arch/x86/include/asm/realmode.h | 3 +++
| 3 +++
arch/x86/realmode/rm/trampoline_64.S | 20 ++++++++++++++++++++
3 files changed, 26 insertions(+)
diff --git a/arch/x86/include/asm/realmode.h b/arch/x86/include/asm/realmode.h
index 96118fb..4d4d853 100644
--- a/arch/x86/include/asm/realmode.h
+++ b/arch/x86/include/asm/realmode.h
@@ -21,6 +21,9 @@ struct real_mode_header {
/* SMP trampoline */
u32 trampoline_start;
u32 trampoline_header;
+#ifdef CONFIG_AMD_MEM_ENCRYPT
+ u32 sev_es_trampoline_start;
+#endif
#ifdef CONFIG_X86_64
u32 trampoline_pgd;
#endif
--git a/arch/x86/realmode/rm/header.S b/arch/x86/realmode/rm/header.S
index af04512..8c1db5b 100644
--- a/arch/x86/realmode/rm/header.S
+++ b/arch/x86/realmode/rm/header.S
@@ -20,6 +20,9 @@ SYM_DATA_START(real_mode_header)
/* SMP trampoline */
.long pa_trampoline_start
.long pa_trampoline_header
+#ifdef CONFIG_AMD_MEM_ENCRYPT
+ .long pa_sev_es_trampoline_start
+#endif
#ifdef CONFIG_X86_64
.long pa_trampoline_pgd;
#endif
diff --git a/arch/x86/realmode/rm/trampoline_64.S b/arch/x86/realmode/rm/trampoline_64.S
index 251758e..84c5d1b 100644
--- a/arch/x86/realmode/rm/trampoline_64.S
+++ b/arch/x86/realmode/rm/trampoline_64.S
@@ -56,6 +56,7 @@ SYM_CODE_START(trampoline_start)
testl %eax, %eax # Check for return code
jnz no_longmode
+.Lswitch_to_protected:
/*
* GDT tables in non default location kernel can be beyond 16MB and
* lgdt will not be able to load the address as in real mode default
@@ -80,6 +81,25 @@ no_longmode:
jmp no_longmode
SYM_CODE_END(trampoline_start)
+#ifdef CONFIG_AMD_MEM_ENCRYPT
+/* SEV-ES supports non-zero IP for entry points - no alignment needed */
+SYM_CODE_START(sev_es_trampoline_start)
+ cli # We should be safe anyway
+
+ LJMPW_RM(1f)
+1:
+ mov %cs, %ax # Code and data in the same place
+ mov %ax, %ds
+ mov %ax, %es
+ mov %ax, %ss
+
+ # Setup stack
+ movl $rm_stack_end, %esp
+
+ jmp .Lswitch_to_protected
+SYM_CODE_END(sev_es_trampoline_start)
+#endif /* CONFIG_AMD_MEM_ENCRYPT */
+
#include "../kernel/verify_cpu.S"
.section ".text32","ax"
^ permalink raw reply related [flat|nested] 264+ messages in thread
* [tip: x86/seves] x86/kvm: Add KVM-specific VMMCALL handling under SEV-ES
2020-09-07 13:16 ` Joerg Roedel
(?)
@ 2020-09-10 9:22 ` tip-bot2 for Tom Lendacky
[not found] ` <CAAYXXYx=Eq4gYfUqdO7u37VRD_GpPYFQgN=GZySmAMcDc2AM=g@mail.gmail.com>
-1 siblings, 1 reply; 264+ messages in thread
From: tip-bot2 for Tom Lendacky @ 2020-09-10 9:22 UTC (permalink / raw)
To: linux-tip-commits; +Cc: Tom Lendacky, Joerg Roedel, Borislav Petkov, x86, LKML
The following commit has been merged into the x86/seves branch of tip:
Commit-ID: 99419b251e5427b89dbfae103d8a2f469efaa4b2
Gitweb: https://git.kernel.org/tip/99419b251e5427b89dbfae103d8a2f469efaa4b2
Author: Tom Lendacky <thomas.lendacky@amd.com>
AuthorDate: Mon, 07 Sep 2020 15:16:04 +02:00
Committer: Borislav Petkov <bp@suse.de>
CommitterDate: Wed, 09 Sep 2020 11:33:20 +02:00
x86/kvm: Add KVM-specific VMMCALL handling under SEV-ES
Implement the callbacks to copy the processor state required by KVM to
the GHCB.
Signed-off-by: Tom Lendacky <thomas.lendacky@amd.com>
[ jroedel@suse.de: - Split out of a larger patch
- Adapt to different callback functions ]
Co-developed-by: Joerg Roedel <jroedel@suse.de>
Signed-off-by: Joerg Roedel <jroedel@suse.de>
Signed-off-by: Borislav Petkov <bp@suse.de>
Link: https://lkml.kernel.org/r/20200907131613.12703-64-joro@8bytes.org
---
arch/x86/kernel/kvm.c | 35 +++++++++++++++++++++++++++++------
1 file changed, 29 insertions(+), 6 deletions(-)
diff --git a/arch/x86/kernel/kvm.c b/arch/x86/kernel/kvm.c
index 08320b0..0f95972 100644
--- a/arch/x86/kernel/kvm.c
+++ b/arch/x86/kernel/kvm.c
@@ -36,6 +36,8 @@
#include <asm/hypervisor.h>
#include <asm/tlb.h>
#include <asm/cpuidle_haltpoll.h>
+#include <asm/ptrace.h>
+#include <asm/svm.h>
DEFINE_STATIC_KEY_FALSE(kvm_async_pf_enabled);
@@ -746,13 +748,34 @@ static void __init kvm_init_platform(void)
x86_platform.apic_post_init = kvm_apic_init;
}
+#if defined(CONFIG_AMD_MEM_ENCRYPT)
+static void kvm_sev_es_hcall_prepare(struct ghcb *ghcb, struct pt_regs *regs)
+{
+ /* RAX and CPL are already in the GHCB */
+ ghcb_set_rbx(ghcb, regs->bx);
+ ghcb_set_rcx(ghcb, regs->cx);
+ ghcb_set_rdx(ghcb, regs->dx);
+ ghcb_set_rsi(ghcb, regs->si);
+}
+
+static bool kvm_sev_es_hcall_finish(struct ghcb *ghcb, struct pt_regs *regs)
+{
+ /* No checking of the return state needed */
+ return true;
+}
+#endif
+
const __initconst struct hypervisor_x86 x86_hyper_kvm = {
- .name = "KVM",
- .detect = kvm_detect,
- .type = X86_HYPER_KVM,
- .init.guest_late_init = kvm_guest_init,
- .init.x2apic_available = kvm_para_available,
- .init.init_platform = kvm_init_platform,
+ .name = "KVM",
+ .detect = kvm_detect,
+ .type = X86_HYPER_KVM,
+ .init.guest_late_init = kvm_guest_init,
+ .init.x2apic_available = kvm_para_available,
+ .init.init_platform = kvm_init_platform,
+#if defined(CONFIG_AMD_MEM_ENCRYPT)
+ .runtime.sev_es_hcall_prepare = kvm_sev_es_hcall_prepare,
+ .runtime.sev_es_hcall_finish = kvm_sev_es_hcall_finish,
+#endif
};
static __init int activate_jump_labels(void)
^ permalink raw reply related [flat|nested] 264+ messages in thread
* [tip: x86/seves] x86/paravirt: Allow hypervisor-specific VMMCALL handling under SEV-ES
2020-09-07 13:16 ` Joerg Roedel
(?)
@ 2020-09-10 9:22 ` tip-bot2 for Joerg Roedel
-1 siblings, 0 replies; 264+ messages in thread
From: tip-bot2 for Joerg Roedel @ 2020-09-10 9:22 UTC (permalink / raw)
To: linux-tip-commits; +Cc: Joerg Roedel, Borislav Petkov, x86, LKML
The following commit has been merged into the x86/seves branch of tip:
Commit-ID: f6a9f8a45810d2914ea422ff39bfe2e0251c50f2
Gitweb: https://git.kernel.org/tip/f6a9f8a45810d2914ea422ff39bfe2e0251c50f2
Author: Joerg Roedel <jroedel@suse.de>
AuthorDate: Mon, 07 Sep 2020 15:16:03 +02:00
Committer: Borislav Petkov <bp@suse.de>
CommitterDate: Wed, 09 Sep 2020 11:33:20 +02:00
x86/paravirt: Allow hypervisor-specific VMMCALL handling under SEV-ES
Add two new paravirt callbacks to provide hypervisor-specific processor
state in the GHCB and to copy state from the hypervisor back to the
processor.
Signed-off-by: Joerg Roedel <jroedel@suse.de>
Signed-off-by: Borislav Petkov <bp@suse.de>
Link: https://lkml.kernel.org/r/20200907131613.12703-63-joro@8bytes.org
---
arch/x86/include/asm/x86_init.h | 16 +++++++++++++++-
arch/x86/kernel/sev-es.c | 12 ++++++++++++
2 files changed, 27 insertions(+), 1 deletion(-)
diff --git a/arch/x86/include/asm/x86_init.h b/arch/x86/include/asm/x86_init.h
index 6807153..0304e29 100644
--- a/arch/x86/include/asm/x86_init.h
+++ b/arch/x86/include/asm/x86_init.h
@@ -4,8 +4,10 @@
#include <asm/bootparam.h>
+struct ghcb;
struct mpc_bus;
struct mpc_cpu;
+struct pt_regs;
struct mpc_table;
struct cpuinfo_x86;
@@ -236,10 +238,22 @@ struct x86_legacy_features {
/**
* struct x86_hyper_runtime - x86 hypervisor specific runtime callbacks
*
- * @pin_vcpu: pin current vcpu to specified physical cpu (run rarely)
+ * @pin_vcpu: pin current vcpu to specified physical
+ * cpu (run rarely)
+ * @sev_es_hcall_prepare: Load additional hypervisor-specific
+ * state into the GHCB when doing a VMMCALL under
+ * SEV-ES. Called from the #VC exception handler.
+ * @sev_es_hcall_finish: Copies state from the GHCB back into the
+ * processor (or pt_regs). Also runs checks on the
+ * state returned from the hypervisor after a
+ * VMMCALL under SEV-ES. Needs to return 'false'
+ * if the checks fail. Called from the #VC
+ * exception handler.
*/
struct x86_hyper_runtime {
void (*pin_vcpu)(int cpu);
+ void (*sev_es_hcall_prepare)(struct ghcb *ghcb, struct pt_regs *regs);
+ bool (*sev_es_hcall_finish)(struct ghcb *ghcb, struct pt_regs *regs);
};
/**
diff --git a/arch/x86/kernel/sev-es.c b/arch/x86/kernel/sev-es.c
index 79d5190..6d89df9 100644
--- a/arch/x86/kernel/sev-es.c
+++ b/arch/x86/kernel/sev-es.c
@@ -897,6 +897,9 @@ static enum es_result vc_handle_vmmcall(struct ghcb *ghcb,
ghcb_set_rax(ghcb, ctxt->regs->ax);
ghcb_set_cpl(ghcb, user_mode(ctxt->regs) ? 3 : 0);
+ if (x86_platform.hyper.sev_es_hcall_prepare)
+ x86_platform.hyper.sev_es_hcall_prepare(ghcb, ctxt->regs);
+
ret = sev_es_ghcb_hv_call(ghcb, ctxt, SVM_EXIT_VMMCALL, 0, 0);
if (ret != ES_OK)
return ret;
@@ -906,6 +909,15 @@ static enum es_result vc_handle_vmmcall(struct ghcb *ghcb,
ctxt->regs->ax = ghcb->save.rax;
+ /*
+ * Call sev_es_hcall_finish() after regs->ax is already set.
+ * This allows the hypervisor handler to overwrite it again if
+ * necessary.
+ */
+ if (x86_platform.hyper.sev_es_hcall_finish &&
+ !x86_platform.hyper.sev_es_hcall_finish(ghcb, ctxt->regs))
+ return ES_VMM_ERROR;
+
return ES_OK;
}
^ permalink raw reply related [flat|nested] 264+ messages in thread
* [tip: x86/seves] x86/sev-es: Handle #DB Events
2020-09-07 13:16 ` Joerg Roedel
(?)
@ 2020-09-10 9:22 ` tip-bot2 for Joerg Roedel
-1 siblings, 0 replies; 264+ messages in thread
From: tip-bot2 for Joerg Roedel @ 2020-09-10 9:22 UTC (permalink / raw)
To: linux-tip-commits; +Cc: Joerg Roedel, Borislav Petkov, x86, LKML
The following commit has been merged into the x86/seves branch of tip:
Commit-ID: cb1ad3ecea959593400dfac4f027dbc005e62c39
Gitweb: https://git.kernel.org/tip/cb1ad3ecea959593400dfac4f027dbc005e62c39
Author: Joerg Roedel <jroedel@suse.de>
AuthorDate: Mon, 07 Sep 2020 15:16:02 +02:00
Committer: Borislav Petkov <bp@suse.de>
CommitterDate: Wed, 09 Sep 2020 11:33:20 +02:00
x86/sev-es: Handle #DB Events
Handle #VC exceptions caused by #DB exceptions in the guest. Those
must be handled outside of instrumentation_begin()/end() so that the
handler will not be raised recursively.
Handle them by calling the kernel's debug exception handler.
Signed-off-by: Joerg Roedel <jroedel@suse.de>
Signed-off-by: Borislav Petkov <bp@suse.de>
Link: https://lkml.kernel.org/r/20200907131613.12703-62-joro@8bytes.org
---
arch/x86/kernel/sev-es.c | 17 +++++++++++++++++
1 file changed, 17 insertions(+)
diff --git a/arch/x86/kernel/sev-es.c b/arch/x86/kernel/sev-es.c
index 8867c48..79d5190 100644
--- a/arch/x86/kernel/sev-es.c
+++ b/arch/x86/kernel/sev-es.c
@@ -922,6 +922,14 @@ static enum es_result vc_handle_trap_ac(struct ghcb *ghcb,
return ES_EXCEPTION;
}
+static __always_inline void vc_handle_trap_db(struct pt_regs *regs)
+{
+ if (user_mode(regs))
+ noist_exc_debug(regs);
+ else
+ exc_debug(regs);
+}
+
static enum es_result vc_handle_exitcode(struct es_em_ctxt *ctxt,
struct ghcb *ghcb,
unsigned long exit_code)
@@ -1033,6 +1041,15 @@ DEFINE_IDTENTRY_VC_SAFE_STACK(exc_vmm_communication)
struct ghcb *ghcb;
lockdep_assert_irqs_disabled();
+
+ /*
+ * Handle #DB before calling into !noinstr code to avoid recursive #DB.
+ */
+ if (error_code == SVM_EXIT_EXCP_BASE + X86_TRAP_DB) {
+ vc_handle_trap_db(regs);
+ return;
+ }
+
instrumentation_begin();
/*
^ permalink raw reply related [flat|nested] 264+ messages in thread
* [tip: x86/seves] x86/sev-es: Handle #AC Events
2020-09-07 13:16 ` Joerg Roedel
(?)
@ 2020-09-10 9:22 ` tip-bot2 for Joerg Roedel
-1 siblings, 0 replies; 264+ messages in thread
From: tip-bot2 for Joerg Roedel @ 2020-09-10 9:22 UTC (permalink / raw)
To: linux-tip-commits; +Cc: Joerg Roedel, Borislav Petkov, x86, LKML
The following commit has been merged into the x86/seves branch of tip:
Commit-ID: a2d0171a9cf59637411281a929900fde80e6c1cb
Gitweb: https://git.kernel.org/tip/a2d0171a9cf59637411281a929900fde80e6c1cb
Author: Joerg Roedel <jroedel@suse.de>
AuthorDate: Mon, 07 Sep 2020 15:16:01 +02:00
Committer: Borislav Petkov <bp@suse.de>
CommitterDate: Wed, 09 Sep 2020 11:33:20 +02:00
x86/sev-es: Handle #AC Events
Implement a handler for #VC exceptions caused by #AC exceptions. The
#AC exception is just forwarded to do_alignment_check() and not pushed
down to the hypervisor, as requested by the SEV-ES GHCB Standardization
Specification.
Signed-off-by: Joerg Roedel <jroedel@suse.de>
Signed-off-by: Borislav Petkov <bp@suse.de>
Link: https://lkml.kernel.org/r/20200907131613.12703-61-joro@8bytes.org
---
arch/x86/kernel/sev-es.c | 19 +++++++++++++++++++
1 file changed, 19 insertions(+)
diff --git a/arch/x86/kernel/sev-es.c b/arch/x86/kernel/sev-es.c
index 86cb4c5..8867c48 100644
--- a/arch/x86/kernel/sev-es.c
+++ b/arch/x86/kernel/sev-es.c
@@ -909,6 +909,19 @@ static enum es_result vc_handle_vmmcall(struct ghcb *ghcb,
return ES_OK;
}
+static enum es_result vc_handle_trap_ac(struct ghcb *ghcb,
+ struct es_em_ctxt *ctxt)
+{
+ /*
+ * Calling ecx_alignment_check() directly does not work, because it
+ * enables IRQs and the GHCB is active. Forward the exception and call
+ * it later from vc_forward_exception().
+ */
+ ctxt->fi.vector = X86_TRAP_AC;
+ ctxt->fi.error_code = 0;
+ return ES_EXCEPTION;
+}
+
static enum es_result vc_handle_exitcode(struct es_em_ctxt *ctxt,
struct ghcb *ghcb,
unsigned long exit_code)
@@ -922,6 +935,9 @@ static enum es_result vc_handle_exitcode(struct es_em_ctxt *ctxt,
case SVM_EXIT_WRITE_DR7:
result = vc_handle_dr7_write(ghcb, ctxt);
break;
+ case SVM_EXIT_EXCP_BASE + X86_TRAP_AC:
+ result = vc_handle_trap_ac(ghcb, ctxt);
+ break;
case SVM_EXIT_RDTSC:
case SVM_EXIT_RDTSCP:
result = vc_handle_rdtsc(ghcb, ctxt, exit_code);
@@ -981,6 +997,9 @@ static __always_inline void vc_forward_exception(struct es_em_ctxt *ctxt)
case X86_TRAP_UD:
exc_invalid_op(ctxt->regs);
break;
+ case X86_TRAP_AC:
+ exc_alignment_check(ctxt->regs, error_code);
+ break;
default:
pr_emerg("Unsupported exception in #VC instruction emulation - can't continue\n");
BUG();
^ permalink raw reply related [flat|nested] 264+ messages in thread
* [tip: x86/seves] x86/sev-es: Handle VMMCALL Events
2020-09-07 13:16 ` Joerg Roedel
(?)
@ 2020-09-10 9:22 ` tip-bot2 for Tom Lendacky
-1 siblings, 0 replies; 264+ messages in thread
From: tip-bot2 for Tom Lendacky @ 2020-09-10 9:22 UTC (permalink / raw)
To: linux-tip-commits; +Cc: Tom Lendacky, Joerg Roedel, Borislav Petkov, x86, LKML
The following commit has been merged into the x86/seves branch of tip:
Commit-ID: 2eb7dcf0ccc40ad3f39b000becf16661abf98102
Gitweb: https://git.kernel.org/tip/2eb7dcf0ccc40ad3f39b000becf16661abf98102
Author: Tom Lendacky <thomas.lendacky@amd.com>
AuthorDate: Mon, 07 Sep 2020 15:16:00 +02:00
Committer: Borislav Petkov <bp@suse.de>
CommitterDate: Wed, 09 Sep 2020 11:33:20 +02:00
x86/sev-es: Handle VMMCALL Events
Implement a handler for #VC exceptions caused by VMMCALL instructions.
This is only a starting point, VMMCALL emulation under SEV-ES needs
further hypervisor-specific changes to provide additional state.
[ bp: Drop "this patch". ]
Signed-off-by: Tom Lendacky <thomas.lendacky@amd.com>
[ jroedel@suse.de: Adapt to #VC handling infrastructure ]
Co-developed-by: Joerg Roedel <jroedel@suse.de>
Signed-off-by: Joerg Roedel <jroedel@suse.de>
Signed-off-by: Borislav Petkov <bp@suse.de>
Link: https://lkml.kernel.org/r/20200907131613.12703-60-joro@8bytes.org
---
arch/x86/kernel/sev-es.c | 23 +++++++++++++++++++++++
1 file changed, 23 insertions(+)
diff --git a/arch/x86/kernel/sev-es.c b/arch/x86/kernel/sev-es.c
index 2aea903..86cb4c5 100644
--- a/arch/x86/kernel/sev-es.c
+++ b/arch/x86/kernel/sev-es.c
@@ -889,6 +889,26 @@ static enum es_result vc_handle_mwait(struct ghcb *ghcb,
return ES_OK;
}
+static enum es_result vc_handle_vmmcall(struct ghcb *ghcb,
+ struct es_em_ctxt *ctxt)
+{
+ enum es_result ret;
+
+ ghcb_set_rax(ghcb, ctxt->regs->ax);
+ ghcb_set_cpl(ghcb, user_mode(ctxt->regs) ? 3 : 0);
+
+ ret = sev_es_ghcb_hv_call(ghcb, ctxt, SVM_EXIT_VMMCALL, 0, 0);
+ if (ret != ES_OK)
+ return ret;
+
+ if (!ghcb_rax_is_valid(ghcb))
+ return ES_VMM_ERROR;
+
+ ctxt->regs->ax = ghcb->save.rax;
+
+ return ES_OK;
+}
+
static enum es_result vc_handle_exitcode(struct es_em_ctxt *ctxt,
struct ghcb *ghcb,
unsigned long exit_code)
@@ -922,6 +942,9 @@ static enum es_result vc_handle_exitcode(struct es_em_ctxt *ctxt,
case SVM_EXIT_MSR:
result = vc_handle_msr(ghcb, ctxt);
break;
+ case SVM_EXIT_VMMCALL:
+ result = vc_handle_vmmcall(ghcb, ctxt);
+ break;
case SVM_EXIT_WBINVD:
result = vc_handle_wbinvd(ghcb, ctxt);
break;
^ permalink raw reply related [flat|nested] 264+ messages in thread
* [tip: x86/seves] x86/sev-es: Handle INVD Events
2020-09-07 13:15 ` Joerg Roedel
(?)
@ 2020-09-10 9:22 ` tip-bot2 for Tom Lendacky
-1 siblings, 0 replies; 264+ messages in thread
From: tip-bot2 for Tom Lendacky @ 2020-09-10 9:22 UTC (permalink / raw)
To: linux-tip-commits; +Cc: Tom Lendacky, Joerg Roedel, Borislav Petkov, x86, LKML
The following commit has been merged into the x86/seves branch of tip:
Commit-ID: 8b4ce83707cb2cff5f6d175ea38f751ac4456096
Gitweb: https://git.kernel.org/tip/8b4ce83707cb2cff5f6d175ea38f751ac4456096
Author: Tom Lendacky <thomas.lendacky@amd.com>
AuthorDate: Mon, 07 Sep 2020 15:15:57 +02:00
Committer: Borislav Petkov <bp@suse.de>
CommitterDate: Wed, 09 Sep 2020 11:33:20 +02:00
x86/sev-es: Handle INVD Events
Implement a handler for #VC exceptions caused by INVD instructions.
Since Linux should never use INVD, just mark it as unsupported.
Signed-off-by: Tom Lendacky <thomas.lendacky@amd.com>
[ jroedel@suse.de: Adapt to #VC handling infrastructure ]
Co-developed-by: Joerg Roedel <jroedel@suse.de>
Signed-off-by: Joerg Roedel <jroedel@suse.de>
Signed-off-by: Borislav Petkov <bp@suse.de>
Link: https://lkml.kernel.org/r/20200907131613.12703-57-joro@8bytes.org
---
arch/x86/kernel/sev-es.c | 4 ++++
1 file changed, 4 insertions(+)
diff --git a/arch/x86/kernel/sev-es.c b/arch/x86/kernel/sev-es.c
index b2d7287..236cfc1 100644
--- a/arch/x86/kernel/sev-es.c
+++ b/arch/x86/kernel/sev-es.c
@@ -892,6 +892,10 @@ static enum es_result vc_handle_exitcode(struct es_em_ctxt *ctxt,
case SVM_EXIT_RDPMC:
result = vc_handle_rdpmc(ghcb, ctxt);
break;
+ case SVM_EXIT_INVD:
+ pr_err_ratelimited("#VC exception for INVD??? Seriously???\n");
+ result = ES_UNSUPPORTED;
+ break;
case SVM_EXIT_CPUID:
result = vc_handle_cpuid(ghcb, ctxt);
break;
^ permalink raw reply related [flat|nested] 264+ messages in thread
* [tip: x86/seves] x86/sev-es: Handle MONITOR/MONITORX Events
2020-09-07 13:15 ` Joerg Roedel
(?)
@ 2020-09-10 9:22 ` tip-bot2 for Tom Lendacky
-1 siblings, 0 replies; 264+ messages in thread
From: tip-bot2 for Tom Lendacky @ 2020-09-10 9:22 UTC (permalink / raw)
To: linux-tip-commits; +Cc: Tom Lendacky, Joerg Roedel, Borislav Petkov, x86, LKML
The following commit has been merged into the x86/seves branch of tip:
Commit-ID: 0c2fd2ef64ef1a91d81c2f61309735ac438b68a4
Gitweb: https://git.kernel.org/tip/0c2fd2ef64ef1a91d81c2f61309735ac438b68a4
Author: Tom Lendacky <thomas.lendacky@amd.com>
AuthorDate: Mon, 07 Sep 2020 15:15:58 +02:00
Committer: Borislav Petkov <bp@suse.de>
CommitterDate: Wed, 09 Sep 2020 11:33:20 +02:00
x86/sev-es: Handle MONITOR/MONITORX Events
Implement a handler for #VC exceptions caused by MONITOR and MONITORX
instructions.
Signed-off-by: Tom Lendacky <thomas.lendacky@amd.com>
[ jroedel@suse.de: Adapt to #VC handling infrastructure ]
Co-developed-by: Joerg Roedel <jroedel@suse.de>
Signed-off-by: Joerg Roedel <jroedel@suse.de>
Signed-off-by: Borislav Petkov <bp@suse.de>
Link: https://lkml.kernel.org/r/20200907131613.12703-58-joro@8bytes.org
---
arch/x86/kernel/sev-es.c | 13 +++++++++++++
1 file changed, 13 insertions(+)
diff --git a/arch/x86/kernel/sev-es.c b/arch/x86/kernel/sev-es.c
index 236cfc1..b9976e7 100644
--- a/arch/x86/kernel/sev-es.c
+++ b/arch/x86/kernel/sev-es.c
@@ -872,6 +872,16 @@ static enum es_result vc_handle_rdpmc(struct ghcb *ghcb, struct es_em_ctxt *ctxt
return ES_OK;
}
+static enum es_result vc_handle_monitor(struct ghcb *ghcb,
+ struct es_em_ctxt *ctxt)
+{
+ /*
+ * Treat it as a NOP and do not leak a physical address to the
+ * hypervisor.
+ */
+ return ES_OK;
+}
+
static enum es_result vc_handle_exitcode(struct es_em_ctxt *ctxt,
struct ghcb *ghcb,
unsigned long exit_code)
@@ -908,6 +918,9 @@ static enum es_result vc_handle_exitcode(struct es_em_ctxt *ctxt,
case SVM_EXIT_WBINVD:
result = vc_handle_wbinvd(ghcb, ctxt);
break;
+ case SVM_EXIT_MONITOR:
+ result = vc_handle_monitor(ghcb, ctxt);
+ break;
case SVM_EXIT_NPF:
result = vc_handle_mmio(ghcb, ctxt);
break;
^ permalink raw reply related [flat|nested] 264+ messages in thread
* [tip: x86/seves] x86/sev-es: Handle MWAIT/MWAITX Events
2020-09-07 13:15 ` Joerg Roedel
(?)
@ 2020-09-10 9:22 ` tip-bot2 for Tom Lendacky
-1 siblings, 0 replies; 264+ messages in thread
From: tip-bot2 for Tom Lendacky @ 2020-09-10 9:22 UTC (permalink / raw)
To: linux-tip-commits; +Cc: Tom Lendacky, Joerg Roedel, Borislav Petkov, x86, LKML
The following commit has been merged into the x86/seves branch of tip:
Commit-ID: ded476bbe203e59cb571d6c576f680efe4a0ddff
Gitweb: https://git.kernel.org/tip/ded476bbe203e59cb571d6c576f680efe4a0ddff
Author: Tom Lendacky <thomas.lendacky@amd.com>
AuthorDate: Mon, 07 Sep 2020 15:15:59 +02:00
Committer: Borislav Petkov <bp@suse.de>
CommitterDate: Wed, 09 Sep 2020 11:33:20 +02:00
x86/sev-es: Handle MWAIT/MWAITX Events
Implement a handler for #VC exceptions caused by MWAIT and MWAITX
instructions.
Signed-off-by: Tom Lendacky <thomas.lendacky@amd.com>
[ jroedel@suse.de: Adapt to #VC handling infrastructure ]
Co-developed-by: Joerg Roedel <jroedel@suse.de>
Signed-off-by: Joerg Roedel <jroedel@suse.de>
Signed-off-by: Borislav Petkov <bp@suse.de>
Link: https://lkml.kernel.org/r/20200907131613.12703-59-joro@8bytes.org
---
arch/x86/kernel/sev-es.c | 10 ++++++++++
1 file changed, 10 insertions(+)
diff --git a/arch/x86/kernel/sev-es.c b/arch/x86/kernel/sev-es.c
index b9976e7..2aea903 100644
--- a/arch/x86/kernel/sev-es.c
+++ b/arch/x86/kernel/sev-es.c
@@ -882,6 +882,13 @@ static enum es_result vc_handle_monitor(struct ghcb *ghcb,
return ES_OK;
}
+static enum es_result vc_handle_mwait(struct ghcb *ghcb,
+ struct es_em_ctxt *ctxt)
+{
+ /* Treat the same as MONITOR/MONITORX */
+ return ES_OK;
+}
+
static enum es_result vc_handle_exitcode(struct es_em_ctxt *ctxt,
struct ghcb *ghcb,
unsigned long exit_code)
@@ -921,6 +928,9 @@ static enum es_result vc_handle_exitcode(struct es_em_ctxt *ctxt,
case SVM_EXIT_MONITOR:
result = vc_handle_monitor(ghcb, ctxt);
break;
+ case SVM_EXIT_MWAIT:
+ result = vc_handle_mwait(ghcb, ctxt);
+ break;
case SVM_EXIT_NPF:
result = vc_handle_mmio(ghcb, ctxt);
break;
^ permalink raw reply related [flat|nested] 264+ messages in thread
* [tip: x86/seves] x86/sev-es: Handle RDPMC Events
2020-09-07 13:15 ` Joerg Roedel
(?)
@ 2020-09-10 9:22 ` tip-bot2 for Tom Lendacky
-1 siblings, 0 replies; 264+ messages in thread
From: tip-bot2 for Tom Lendacky @ 2020-09-10 9:22 UTC (permalink / raw)
To: linux-tip-commits; +Cc: Tom Lendacky, Joerg Roedel, Borislav Petkov, x86, LKML
The following commit has been merged into the x86/seves branch of tip:
Commit-ID: 5d55cf78a8785a7496919462f3d1a4689f62fb1d
Gitweb: https://git.kernel.org/tip/5d55cf78a8785a7496919462f3d1a4689f62fb1d
Author: Tom Lendacky <thomas.lendacky@amd.com>
AuthorDate: Mon, 07 Sep 2020 15:15:56 +02:00
Committer: Borislav Petkov <bp@suse.de>
CommitterDate: Wed, 09 Sep 2020 11:33:20 +02:00
x86/sev-es: Handle RDPMC Events
Implement a handler for #VC exceptions caused by RDPMC instructions.
Signed-off-by: Tom Lendacky <thomas.lendacky@amd.com>
[ jroedel@suse.de: Adapt to #VC handling infrastructure ]
Co-developed-by: Joerg Roedel <jroedel@suse.de>
Signed-off-by: Joerg Roedel <jroedel@suse.de>
Signed-off-by: Borislav Petkov <bp@suse.de>
Link: https://lkml.kernel.org/r/20200907131613.12703-56-joro@8bytes.org
---
arch/x86/kernel/sev-es.c | 22 ++++++++++++++++++++++
1 file changed, 22 insertions(+)
diff --git a/arch/x86/kernel/sev-es.c b/arch/x86/kernel/sev-es.c
index 4d468ec..b2d7287 100644
--- a/arch/x86/kernel/sev-es.c
+++ b/arch/x86/kernel/sev-es.c
@@ -853,6 +853,25 @@ static enum es_result vc_handle_wbinvd(struct ghcb *ghcb,
return sev_es_ghcb_hv_call(ghcb, ctxt, SVM_EXIT_WBINVD, 0, 0);
}
+static enum es_result vc_handle_rdpmc(struct ghcb *ghcb, struct es_em_ctxt *ctxt)
+{
+ enum es_result ret;
+
+ ghcb_set_rcx(ghcb, ctxt->regs->cx);
+
+ ret = sev_es_ghcb_hv_call(ghcb, ctxt, SVM_EXIT_RDPMC, 0, 0);
+ if (ret != ES_OK)
+ return ret;
+
+ if (!(ghcb_rax_is_valid(ghcb) && ghcb_rdx_is_valid(ghcb)))
+ return ES_VMM_ERROR;
+
+ ctxt->regs->ax = ghcb->save.rax;
+ ctxt->regs->dx = ghcb->save.rdx;
+
+ return ES_OK;
+}
+
static enum es_result vc_handle_exitcode(struct es_em_ctxt *ctxt,
struct ghcb *ghcb,
unsigned long exit_code)
@@ -870,6 +889,9 @@ static enum es_result vc_handle_exitcode(struct es_em_ctxt *ctxt,
case SVM_EXIT_RDTSCP:
result = vc_handle_rdtsc(ghcb, ctxt, exit_code);
break;
+ case SVM_EXIT_RDPMC:
+ result = vc_handle_rdpmc(ghcb, ctxt);
+ break;
case SVM_EXIT_CPUID:
result = vc_handle_cpuid(ghcb, ctxt);
break;
^ permalink raw reply related [flat|nested] 264+ messages in thread
* [tip: x86/seves] x86/sev-es: Handle RDTSC(P) Events
2020-09-07 13:15 ` Joerg Roedel
(?)
@ 2020-09-10 9:22 ` tip-bot2 for Tom Lendacky
-1 siblings, 0 replies; 264+ messages in thread
From: tip-bot2 for Tom Lendacky @ 2020-09-10 9:22 UTC (permalink / raw)
To: linux-tip-commits; +Cc: Tom Lendacky, Joerg Roedel, Borislav Petkov, x86, LKML
The following commit has been merged into the x86/seves branch of tip:
Commit-ID: 4711e7acaa125d8cc242f06e1f4d6c74e177454b
Gitweb: https://git.kernel.org/tip/4711e7acaa125d8cc242f06e1f4d6c74e177454b
Author: Tom Lendacky <thomas.lendacky@amd.com>
AuthorDate: Mon, 07 Sep 2020 15:15:55 +02:00
Committer: Borislav Petkov <bp@suse.de>
CommitterDate: Wed, 09 Sep 2020 11:33:20 +02:00
x86/sev-es: Handle RDTSC(P) Events
Implement a handler for #VC exceptions caused by RDTSC and RDTSCP
instructions. Also make it available in the pre-decompression stage
because the KASLR code uses RDTSC/RDTSCP to gather entropy and some
hypervisors intercept these instructions.
Signed-off-by: Tom Lendacky <thomas.lendacky@amd.com>
[ jroedel@suse.de: - Adapt to #VC handling infrastructure
- Make it available early ]
Co-developed-by: Joerg Roedel <jroedel@suse.de>
Signed-off-by: Joerg Roedel <jroedel@suse.de>
Signed-off-by: Borislav Petkov <bp@suse.de>
Link: https://lkml.kernel.org/r/20200907131613.12703-55-joro@8bytes.org
---
arch/x86/boot/compressed/sev-es.c | 4 ++++
arch/x86/kernel/sev-es-shared.c | 23 +++++++++++++++++++++++
arch/x86/kernel/sev-es.c | 4 ++++
3 files changed, 31 insertions(+)
diff --git a/arch/x86/boot/compressed/sev-es.c b/arch/x86/boot/compressed/sev-es.c
index b1790f4..5f15e58 100644
--- a/arch/x86/boot/compressed/sev-es.c
+++ b/arch/x86/boot/compressed/sev-es.c
@@ -181,6 +181,10 @@ void do_boot_stage2_vc(struct pt_regs *regs, unsigned long exit_code)
goto finish;
switch (exit_code) {
+ case SVM_EXIT_RDTSC:
+ case SVM_EXIT_RDTSCP:
+ result = vc_handle_rdtsc(boot_ghcb, &ctxt, exit_code);
+ break;
case SVM_EXIT_IOIO:
result = vc_handle_ioio(boot_ghcb, &ctxt);
break;
diff --git a/arch/x86/kernel/sev-es-shared.c b/arch/x86/kernel/sev-es-shared.c
index 491b557..4be8af2 100644
--- a/arch/x86/kernel/sev-es-shared.c
+++ b/arch/x86/kernel/sev-es-shared.c
@@ -467,3 +467,26 @@ static enum es_result vc_handle_cpuid(struct ghcb *ghcb,
return ES_OK;
}
+
+static enum es_result vc_handle_rdtsc(struct ghcb *ghcb,
+ struct es_em_ctxt *ctxt,
+ unsigned long exit_code)
+{
+ bool rdtscp = (exit_code == SVM_EXIT_RDTSCP);
+ enum es_result ret;
+
+ ret = sev_es_ghcb_hv_call(ghcb, ctxt, exit_code, 0, 0);
+ if (ret != ES_OK)
+ return ret;
+
+ if (!(ghcb_rax_is_valid(ghcb) && ghcb_rdx_is_valid(ghcb) &&
+ (!rdtscp || ghcb_rcx_is_valid(ghcb))))
+ return ES_VMM_ERROR;
+
+ ctxt->regs->ax = ghcb->save.rax;
+ ctxt->regs->dx = ghcb->save.rdx;
+ if (rdtscp)
+ ctxt->regs->cx = ghcb->save.rcx;
+
+ return ES_OK;
+}
diff --git a/arch/x86/kernel/sev-es.c b/arch/x86/kernel/sev-es.c
index aba27c3..4d468ec 100644
--- a/arch/x86/kernel/sev-es.c
+++ b/arch/x86/kernel/sev-es.c
@@ -866,6 +866,10 @@ static enum es_result vc_handle_exitcode(struct es_em_ctxt *ctxt,
case SVM_EXIT_WRITE_DR7:
result = vc_handle_dr7_write(ghcb, ctxt);
break;
+ case SVM_EXIT_RDTSC:
+ case SVM_EXIT_RDTSCP:
+ result = vc_handle_rdtsc(ghcb, ctxt, exit_code);
+ break;
case SVM_EXIT_CPUID:
result = vc_handle_cpuid(ghcb, ctxt);
break;
^ permalink raw reply related [flat|nested] 264+ messages in thread
* [tip: x86/seves] x86/sev-es: Handle MSR events
2020-09-07 13:15 ` Joerg Roedel
(?)
@ 2020-09-10 9:22 ` tip-bot2 for Tom Lendacky
-1 siblings, 0 replies; 264+ messages in thread
From: tip-bot2 for Tom Lendacky @ 2020-09-10 9:22 UTC (permalink / raw)
To: linux-tip-commits; +Cc: Tom Lendacky, Joerg Roedel, Borislav Petkov, x86, LKML
The following commit has been merged into the x86/seves branch of tip:
Commit-ID: a4afa6081c88701635e1e20090f953a25f9444e0
Gitweb: https://git.kernel.org/tip/a4afa6081c88701635e1e20090f953a25f9444e0
Author: Tom Lendacky <thomas.lendacky@amd.com>
AuthorDate: Mon, 07 Sep 2020 15:15:52 +02:00
Committer: Borislav Petkov <bp@suse.de>
CommitterDate: Wed, 09 Sep 2020 11:33:19 +02:00
x86/sev-es: Handle MSR events
Implement a handler for #VC exceptions caused by RDMSR/WRMSR
instructions.
Signed-off-by: Tom Lendacky <thomas.lendacky@amd.com>
[ jroedel@suse.de: Adapt to #VC handling infrastructure. ]
Co-developed-by: Joerg Roedel <jroedel@suse.de>
Signed-off-by: Joerg Roedel <jroedel@suse.de>
Signed-off-by: Borislav Petkov <bp@suse.de>
Link: https://lkml.kernel.org/r/20200907131613.12703-52-joro@8bytes.org
---
arch/x86/kernel/sev-es.c | 28 ++++++++++++++++++++++++++++
1 file changed, 28 insertions(+)
diff --git a/arch/x86/kernel/sev-es.c b/arch/x86/kernel/sev-es.c
index f724b75..6205100 100644
--- a/arch/x86/kernel/sev-es.c
+++ b/arch/x86/kernel/sev-es.c
@@ -393,6 +393,31 @@ static bool vc_slow_virt_to_phys(struct ghcb *ghcb, struct es_em_ctxt *ctxt,
/* Include code shared with pre-decompression boot stage */
#include "sev-es-shared.c"
+static enum es_result vc_handle_msr(struct ghcb *ghcb, struct es_em_ctxt *ctxt)
+{
+ struct pt_regs *regs = ctxt->regs;
+ enum es_result ret;
+ u64 exit_info_1;
+
+ /* Is it a WRMSR? */
+ exit_info_1 = (ctxt->insn.opcode.bytes[1] == 0x30) ? 1 : 0;
+
+ ghcb_set_rcx(ghcb, regs->cx);
+ if (exit_info_1) {
+ ghcb_set_rax(ghcb, regs->ax);
+ ghcb_set_rdx(ghcb, regs->dx);
+ }
+
+ ret = sev_es_ghcb_hv_call(ghcb, ctxt, SVM_EXIT_MSR, exit_info_1, 0);
+
+ if ((ret == ES_OK) && (!exit_info_1)) {
+ regs->ax = ghcb->save.rax;
+ regs->dx = ghcb->save.rdx;
+ }
+
+ return ret;
+}
+
/*
* This function runs on the first #VC exception after the kernel
* switched to virtual addresses.
@@ -756,6 +781,9 @@ static enum es_result vc_handle_exitcode(struct es_em_ctxt *ctxt,
case SVM_EXIT_IOIO:
result = vc_handle_ioio(ghcb, ctxt);
break;
+ case SVM_EXIT_MSR:
+ result = vc_handle_msr(ghcb, ctxt);
+ break;
case SVM_EXIT_NPF:
result = vc_handle_mmio(ghcb, ctxt);
break;
^ permalink raw reply related [flat|nested] 264+ messages in thread
* [tip: x86/seves] x86/sev-es: Handle DR7 read/write events
2020-09-07 13:15 ` Joerg Roedel
(?)
@ 2020-09-10 9:22 ` tip-bot2 for Tom Lendacky
-1 siblings, 0 replies; 264+ messages in thread
From: tip-bot2 for Tom Lendacky @ 2020-09-10 9:22 UTC (permalink / raw)
To: linux-tip-commits; +Cc: Tom Lendacky, Joerg Roedel, Borislav Petkov, x86, LKML
The following commit has been merged into the x86/seves branch of tip:
Commit-ID: 479a7bf5c1f7f14eb37c6d6aeb401562efc1fcad
Gitweb: https://git.kernel.org/tip/479a7bf5c1f7f14eb37c6d6aeb401562efc1fcad
Author: Tom Lendacky <thomas.lendacky@amd.com>
AuthorDate: Mon, 07 Sep 2020 15:15:53 +02:00
Committer: Borislav Petkov <bp@suse.de>
CommitterDate: Wed, 09 Sep 2020 11:33:20 +02:00
x86/sev-es: Handle DR7 read/write events
Add code to handle #VC exceptions on DR7 register reads and writes.
This is needed early because show_regs() reads DR7 to print it out.
Under SEV-ES, there is currently no support for saving/restoring the
DRx registers but software expects to be able to write to the DR7
register. For now, cache the value written to DR7 and return it on
read attempts, but do not touch the real hardware DR7.
Signed-off-by: Tom Lendacky <thomas.lendacky@amd.com>
[ jroedel@suse.de: - Adapt to #VC handling framework
- Support early usage ]
Co-developed-by: Joerg Roedel <jroedel@suse.de>
Signed-off-by: Joerg Roedel <jroedel@suse.de>
Signed-off-by: Borislav Petkov <bp@suse.de>
Link: https://lkml.kernel.org/r/20200907131613.12703-53-joro@8bytes.org
---
arch/x86/kernel/sev-es.c | 85 +++++++++++++++++++++++++++++++++++++++-
1 file changed, 85 insertions(+)
diff --git a/arch/x86/kernel/sev-es.c b/arch/x86/kernel/sev-es.c
index 6205100..391b9c7 100644
--- a/arch/x86/kernel/sev-es.c
+++ b/arch/x86/kernel/sev-es.c
@@ -29,6 +29,8 @@
#include <asm/traps.h>
#include <asm/svm.h>
+#define DR7_RESET_VALUE 0x400
+
/* For early boot hypervisor communication in SEV-ES enabled guests */
static struct ghcb boot_ghcb_page __bss_decrypted __aligned(PAGE_SIZE);
@@ -76,6 +78,13 @@ struct sev_es_runtime_data {
*/
bool ghcb_active;
bool backup_ghcb_active;
+
+ /*
+ * Cached DR7 value - write it on DR7 writes and return it on reads.
+ * That value will never make it to the real hardware DR7 as debugging
+ * is currently unsupported in SEV-ES guests.
+ */
+ unsigned long dr7;
};
struct ghcb_state {
@@ -519,6 +528,21 @@ static long *vc_insn_get_reg(struct es_em_ctxt *ctxt)
return reg_array + offset;
}
+static long *vc_insn_get_rm(struct es_em_ctxt *ctxt)
+{
+ long *reg_array;
+ int offset;
+
+ reg_array = (long *)ctxt->regs;
+ offset = insn_get_modrm_rm_off(&ctxt->insn, ctxt->regs);
+
+ if (offset < 0)
+ return NULL;
+
+ offset /= sizeof(long);
+
+ return reg_array + offset;
+}
static enum es_result vc_do_mmio(struct ghcb *ghcb, struct es_em_ctxt *ctxt,
unsigned int bytes, bool read)
{
@@ -768,6 +792,61 @@ static enum es_result vc_handle_mmio(struct ghcb *ghcb,
return ret;
}
+static enum es_result vc_handle_dr7_write(struct ghcb *ghcb,
+ struct es_em_ctxt *ctxt)
+{
+ struct sev_es_runtime_data *data = this_cpu_read(runtime_data);
+ long val, *reg = vc_insn_get_rm(ctxt);
+ enum es_result ret;
+
+ if (!reg)
+ return ES_DECODE_FAILED;
+
+ val = *reg;
+
+ /* Upper 32 bits must be written as zeroes */
+ if (val >> 32) {
+ ctxt->fi.vector = X86_TRAP_GP;
+ ctxt->fi.error_code = 0;
+ return ES_EXCEPTION;
+ }
+
+ /* Clear out other reserved bits and set bit 10 */
+ val = (val & 0xffff23ffL) | BIT(10);
+
+ /* Early non-zero writes to DR7 are not supported */
+ if (!data && (val & ~DR7_RESET_VALUE))
+ return ES_UNSUPPORTED;
+
+ /* Using a value of 0 for ExitInfo1 means RAX holds the value */
+ ghcb_set_rax(ghcb, val);
+ ret = sev_es_ghcb_hv_call(ghcb, ctxt, SVM_EXIT_WRITE_DR7, 0, 0);
+ if (ret != ES_OK)
+ return ret;
+
+ if (data)
+ data->dr7 = val;
+
+ return ES_OK;
+}
+
+static enum es_result vc_handle_dr7_read(struct ghcb *ghcb,
+ struct es_em_ctxt *ctxt)
+{
+ struct sev_es_runtime_data *data = this_cpu_read(runtime_data);
+ long *reg = vc_insn_get_rm(ctxt);
+
+ if (!reg)
+ return ES_DECODE_FAILED;
+
+ if (data)
+ *reg = data->dr7;
+ else
+ *reg = DR7_RESET_VALUE;
+
+ return ES_OK;
+}
+
static enum es_result vc_handle_exitcode(struct es_em_ctxt *ctxt,
struct ghcb *ghcb,
unsigned long exit_code)
@@ -775,6 +854,12 @@ static enum es_result vc_handle_exitcode(struct es_em_ctxt *ctxt,
enum es_result result;
switch (exit_code) {
+ case SVM_EXIT_READ_DR7:
+ result = vc_handle_dr7_read(ghcb, ctxt);
+ break;
+ case SVM_EXIT_WRITE_DR7:
+ result = vc_handle_dr7_write(ghcb, ctxt);
+ break;
case SVM_EXIT_CPUID:
result = vc_handle_cpuid(ghcb, ctxt);
break;
^ permalink raw reply related [flat|nested] 264+ messages in thread
* [tip: x86/seves] x86/sev-es: Handle WBINVD Events
2020-09-07 13:15 ` Joerg Roedel
(?)
@ 2020-09-10 9:22 ` tip-bot2 for Tom Lendacky
-1 siblings, 0 replies; 264+ messages in thread
From: tip-bot2 for Tom Lendacky @ 2020-09-10 9:22 UTC (permalink / raw)
To: linux-tip-commits; +Cc: Tom Lendacky, Joerg Roedel, Borislav Petkov, x86, LKML
The following commit has been merged into the x86/seves branch of tip:
Commit-ID: a14a92fc4b420ae6713800a8ae8c3686c1a1fa20
Gitweb: https://git.kernel.org/tip/a14a92fc4b420ae6713800a8ae8c3686c1a1fa20
Author: Tom Lendacky <thomas.lendacky@amd.com>
AuthorDate: Mon, 07 Sep 2020 15:15:54 +02:00
Committer: Borislav Petkov <bp@suse.de>
CommitterDate: Wed, 09 Sep 2020 11:33:20 +02:00
x86/sev-es: Handle WBINVD Events
Implement a handler for #VC exceptions caused by WBINVD instructions.
Signed-off-by: Tom Lendacky <thomas.lendacky@amd.com>
[ jroedel@suse.de: Adapt to #VC handling framework ]
Co-developed-by: Joerg Roedel <jroedel@suse.de>
Signed-off-by: Joerg Roedel <jroedel@suse.de>
Signed-off-by: Borislav Petkov <bp@suse.de>
Link: https://lkml.kernel.org/r/20200907131613.12703-54-joro@8bytes.org
---
arch/x86/kernel/sev-es.c | 9 +++++++++
1 file changed, 9 insertions(+)
diff --git a/arch/x86/kernel/sev-es.c b/arch/x86/kernel/sev-es.c
index 391b9c7..aba27c3 100644
--- a/arch/x86/kernel/sev-es.c
+++ b/arch/x86/kernel/sev-es.c
@@ -847,6 +847,12 @@ static enum es_result vc_handle_dr7_read(struct ghcb *ghcb,
return ES_OK;
}
+static enum es_result vc_handle_wbinvd(struct ghcb *ghcb,
+ struct es_em_ctxt *ctxt)
+{
+ return sev_es_ghcb_hv_call(ghcb, ctxt, SVM_EXIT_WBINVD, 0, 0);
+}
+
static enum es_result vc_handle_exitcode(struct es_em_ctxt *ctxt,
struct ghcb *ghcb,
unsigned long exit_code)
@@ -869,6 +875,9 @@ static enum es_result vc_handle_exitcode(struct es_em_ctxt *ctxt,
case SVM_EXIT_MSR:
result = vc_handle_msr(ghcb, ctxt);
break;
+ case SVM_EXIT_WBINVD:
+ result = vc_handle_wbinvd(ghcb, ctxt);
+ break;
case SVM_EXIT_NPF:
result = vc_handle_mmio(ghcb, ctxt);
break;
^ permalink raw reply related [flat|nested] 264+ messages in thread
* [tip: x86/seves] x86/sev-es: Handle MMIO events
2020-09-07 13:15 ` Joerg Roedel
(?)
@ 2020-09-10 9:22 ` tip-bot2 for Tom Lendacky
-1 siblings, 0 replies; 264+ messages in thread
From: tip-bot2 for Tom Lendacky @ 2020-09-10 9:22 UTC (permalink / raw)
To: linux-tip-commits; +Cc: Tom Lendacky, Joerg Roedel, Borislav Petkov, x86, LKML
The following commit has been merged into the x86/seves branch of tip:
Commit-ID: 51ee7d6e3d2b70a3f232cceffab5084a2abd6719
Gitweb: https://git.kernel.org/tip/51ee7d6e3d2b70a3f232cceffab5084a2abd6719
Author: Tom Lendacky <thomas.lendacky@amd.com>
AuthorDate: Mon, 07 Sep 2020 15:15:50 +02:00
Committer: Borislav Petkov <bp@suse.de>
CommitterDate: Wed, 09 Sep 2020 11:33:19 +02:00
x86/sev-es: Handle MMIO events
Add a handler for #VC exceptions caused by MMIO intercepts. These
intercepts come along as nested page faults on pages with reserved
bits set.
Signed-off-by: Tom Lendacky <thomas.lendacky@amd.com>
[ jroedel@suse.de: Adapt to VC handling framework ]
Co-developed-by: Joerg Roedel <jroedel@suse.de>
Signed-off-by: Joerg Roedel <jroedel@suse.de>
Signed-off-by: Borislav Petkov <bp@suse.de>
Link: https://lkml.kernel.org/r/20200907131613.12703-50-joro@8bytes.org
---
arch/x86/include/uapi/asm/svm.h | 5 +-
arch/x86/kernel/sev-es.c | 222 +++++++++++++++++++++++++++++++-
2 files changed, 227 insertions(+)
diff --git a/arch/x86/include/uapi/asm/svm.h b/arch/x86/include/uapi/asm/svm.h
index c68d161..8f36ae0 100644
--- a/arch/x86/include/uapi/asm/svm.h
+++ b/arch/x86/include/uapi/asm/svm.h
@@ -81,6 +81,11 @@
#define SVM_EXIT_AVIC_INCOMPLETE_IPI 0x401
#define SVM_EXIT_AVIC_UNACCELERATED_ACCESS 0x402
+/* SEV-ES software-defined VMGEXIT events */
+#define SVM_VMGEXIT_MMIO_READ 0x80000001
+#define SVM_VMGEXIT_MMIO_WRITE 0x80000002
+#define SVM_VMGEXIT_UNSUPPORTED_EVENT 0x8000ffff
+
#define SVM_EXIT_ERR -1
#define SVM_EXIT_REASONS \
diff --git a/arch/x86/kernel/sev-es.c b/arch/x86/kernel/sev-es.c
index 6c30dbc..a170810 100644
--- a/arch/x86/kernel/sev-es.c
+++ b/arch/x86/kernel/sev-es.c
@@ -359,6 +359,37 @@ fault:
return ES_EXCEPTION;
}
+static bool vc_slow_virt_to_phys(struct ghcb *ghcb, struct es_em_ctxt *ctxt,
+ unsigned long vaddr, phys_addr_t *paddr)
+{
+ unsigned long va = (unsigned long)vaddr;
+ unsigned int level;
+ phys_addr_t pa;
+ pgd_t *pgd;
+ pte_t *pte;
+
+ pgd = __va(read_cr3_pa());
+ pgd = &pgd[pgd_index(va)];
+ pte = lookup_address_in_pgd(pgd, va, &level);
+ if (!pte) {
+ ctxt->fi.vector = X86_TRAP_PF;
+ ctxt->fi.cr2 = vaddr;
+ ctxt->fi.error_code = 0;
+
+ if (user_mode(ctxt->regs))
+ ctxt->fi.error_code |= X86_PF_USER;
+
+ return false;
+ }
+
+ pa = (phys_addr_t)pte_pfn(*pte) << PAGE_SHIFT;
+ pa |= va & ~page_level_mask(level);
+
+ *paddr = pa;
+
+ return true;
+}
+
/* Include code shared with pre-decompression boot stage */
#include "sev-es-shared.c"
@@ -447,6 +478,194 @@ static void __init vc_early_forward_exception(struct es_em_ctxt *ctxt)
do_early_exception(ctxt->regs, trapnr);
}
+static long *vc_insn_get_reg(struct es_em_ctxt *ctxt)
+{
+ long *reg_array;
+ int offset;
+
+ reg_array = (long *)ctxt->regs;
+ offset = insn_get_modrm_reg_off(&ctxt->insn, ctxt->regs);
+
+ if (offset < 0)
+ return NULL;
+
+ offset /= sizeof(long);
+
+ return reg_array + offset;
+}
+
+static enum es_result vc_do_mmio(struct ghcb *ghcb, struct es_em_ctxt *ctxt,
+ unsigned int bytes, bool read)
+{
+ u64 exit_code, exit_info_1, exit_info_2;
+ unsigned long ghcb_pa = __pa(ghcb);
+ phys_addr_t paddr;
+ void __user *ref;
+
+ ref = insn_get_addr_ref(&ctxt->insn, ctxt->regs);
+ if (ref == (void __user *)-1L)
+ return ES_UNSUPPORTED;
+
+ exit_code = read ? SVM_VMGEXIT_MMIO_READ : SVM_VMGEXIT_MMIO_WRITE;
+
+ if (!vc_slow_virt_to_phys(ghcb, ctxt, (unsigned long)ref, &paddr)) {
+ if (!read)
+ ctxt->fi.error_code |= X86_PF_WRITE;
+
+ return ES_EXCEPTION;
+ }
+
+ exit_info_1 = paddr;
+ /* Can never be greater than 8 */
+ exit_info_2 = bytes;
+
+ ghcb->save.sw_scratch = ghcb_pa + offsetof(struct ghcb, shared_buffer);
+
+ return sev_es_ghcb_hv_call(ghcb, ctxt, exit_code, exit_info_1, exit_info_2);
+}
+
+static enum es_result vc_handle_mmio_twobyte_ops(struct ghcb *ghcb,
+ struct es_em_ctxt *ctxt)
+{
+ struct insn *insn = &ctxt->insn;
+ unsigned int bytes = 0;
+ enum es_result ret;
+ int sign_byte;
+ long *reg_data;
+
+ switch (insn->opcode.bytes[1]) {
+ /* MMIO Read w/ zero-extension */
+ case 0xb6:
+ bytes = 1;
+ fallthrough;
+ case 0xb7:
+ if (!bytes)
+ bytes = 2;
+
+ ret = vc_do_mmio(ghcb, ctxt, bytes, true);
+ if (ret)
+ break;
+
+ /* Zero extend based on operand size */
+ reg_data = vc_insn_get_reg(ctxt);
+ if (!reg_data)
+ return ES_DECODE_FAILED;
+
+ memset(reg_data, 0, insn->opnd_bytes);
+
+ memcpy(reg_data, ghcb->shared_buffer, bytes);
+ break;
+
+ /* MMIO Read w/ sign-extension */
+ case 0xbe:
+ bytes = 1;
+ fallthrough;
+ case 0xbf:
+ if (!bytes)
+ bytes = 2;
+
+ ret = vc_do_mmio(ghcb, ctxt, bytes, true);
+ if (ret)
+ break;
+
+ /* Sign extend based on operand size */
+ reg_data = vc_insn_get_reg(ctxt);
+ if (!reg_data)
+ return ES_DECODE_FAILED;
+
+ if (bytes == 1) {
+ u8 *val = (u8 *)ghcb->shared_buffer;
+
+ sign_byte = (*val & 0x80) ? 0xff : 0x00;
+ } else {
+ u16 *val = (u16 *)ghcb->shared_buffer;
+
+ sign_byte = (*val & 0x8000) ? 0xff : 0x00;
+ }
+ memset(reg_data, sign_byte, insn->opnd_bytes);
+
+ memcpy(reg_data, ghcb->shared_buffer, bytes);
+ break;
+
+ default:
+ ret = ES_UNSUPPORTED;
+ }
+
+ return ret;
+}
+
+static enum es_result vc_handle_mmio(struct ghcb *ghcb,
+ struct es_em_ctxt *ctxt)
+{
+ struct insn *insn = &ctxt->insn;
+ unsigned int bytes = 0;
+ enum es_result ret;
+ long *reg_data;
+
+ switch (insn->opcode.bytes[0]) {
+ /* MMIO Write */
+ case 0x88:
+ bytes = 1;
+ fallthrough;
+ case 0x89:
+ if (!bytes)
+ bytes = insn->opnd_bytes;
+
+ reg_data = vc_insn_get_reg(ctxt);
+ if (!reg_data)
+ return ES_DECODE_FAILED;
+
+ memcpy(ghcb->shared_buffer, reg_data, bytes);
+
+ ret = vc_do_mmio(ghcb, ctxt, bytes, false);
+ break;
+
+ case 0xc6:
+ bytes = 1;
+ fallthrough;
+ case 0xc7:
+ if (!bytes)
+ bytes = insn->opnd_bytes;
+
+ memcpy(ghcb->shared_buffer, insn->immediate1.bytes, bytes);
+
+ ret = vc_do_mmio(ghcb, ctxt, bytes, false);
+ break;
+
+ /* MMIO Read */
+ case 0x8a:
+ bytes = 1;
+ fallthrough;
+ case 0x8b:
+ if (!bytes)
+ bytes = insn->opnd_bytes;
+
+ ret = vc_do_mmio(ghcb, ctxt, bytes, true);
+ if (ret)
+ break;
+
+ reg_data = vc_insn_get_reg(ctxt);
+ if (!reg_data)
+ return ES_DECODE_FAILED;
+
+ /* Zero-extend for 32-bit operation */
+ if (bytes == 4)
+ *reg_data = 0;
+
+ memcpy(reg_data, ghcb->shared_buffer, bytes);
+ break;
+
+ /* Two-Byte Opcodes */
+ case 0x0f:
+ ret = vc_handle_mmio_twobyte_ops(ghcb, ctxt);
+ break;
+ default:
+ ret = ES_UNSUPPORTED;
+ }
+
+ return ret;
+}
+
static enum es_result vc_handle_exitcode(struct es_em_ctxt *ctxt,
struct ghcb *ghcb,
unsigned long exit_code)
@@ -460,6 +679,9 @@ static enum es_result vc_handle_exitcode(struct es_em_ctxt *ctxt,
case SVM_EXIT_IOIO:
result = vc_handle_ioio(ghcb, ctxt);
break;
+ case SVM_EXIT_NPF:
+ result = vc_handle_mmio(ghcb, ctxt);
+ break;
default:
/*
* Unexpected #VC exception
^ permalink raw reply related [flat|nested] 264+ messages in thread
* [tip: x86/seves] x86/sev-es: Handle MMIO String Instructions
2020-09-07 13:15 ` Joerg Roedel
(?)
@ 2020-09-10 9:22 ` tip-bot2 for Joerg Roedel
-1 siblings, 0 replies; 264+ messages in thread
From: tip-bot2 for Joerg Roedel @ 2020-09-10 9:22 UTC (permalink / raw)
To: linux-tip-commits; +Cc: Joerg Roedel, Borislav Petkov, x86, LKML
The following commit has been merged into the x86/seves branch of tip:
Commit-ID: 0118b604c2c94c6e34982015cfa7891af4764786
Gitweb: https://git.kernel.org/tip/0118b604c2c94c6e34982015cfa7891af4764786
Author: Joerg Roedel <jroedel@suse.de>
AuthorDate: Mon, 07 Sep 2020 15:15:51 +02:00
Committer: Borislav Petkov <bp@suse.de>
CommitterDate: Wed, 09 Sep 2020 11:33:19 +02:00
x86/sev-es: Handle MMIO String Instructions
Add handling for emulation of the MOVS instruction on MMIO regions, as
done by the memcpy_toio() and memcpy_fromio() functions.
Signed-off-by: Joerg Roedel <jroedel@suse.de>
Signed-off-by: Borislav Petkov <bp@suse.de>
Link: https://lkml.kernel.org/r/20200907131613.12703-51-joro@8bytes.org
---
arch/x86/kernel/sev-es.c | 77 +++++++++++++++++++++++++++++++++++++++-
1 file changed, 77 insertions(+)
diff --git a/arch/x86/kernel/sev-es.c b/arch/x86/kernel/sev-es.c
index a170810..f724b75 100644
--- a/arch/x86/kernel/sev-es.c
+++ b/arch/x86/kernel/sev-es.c
@@ -594,6 +594,73 @@ static enum es_result vc_handle_mmio_twobyte_ops(struct ghcb *ghcb,
return ret;
}
+/*
+ * The MOVS instruction has two memory operands, which raises the
+ * problem that it is not known whether the access to the source or the
+ * destination caused the #VC exception (and hence whether an MMIO read
+ * or write operation needs to be emulated).
+ *
+ * Instead of playing games with walking page-tables and trying to guess
+ * whether the source or destination is an MMIO range, split the move
+ * into two operations, a read and a write with only one memory operand.
+ * This will cause a nested #VC exception on the MMIO address which can
+ * then be handled.
+ *
+ * This implementation has the benefit that it also supports MOVS where
+ * source _and_ destination are MMIO regions.
+ *
+ * It will slow MOVS on MMIO down a lot, but in SEV-ES guests it is a
+ * rare operation. If it turns out to be a performance problem the split
+ * operations can be moved to memcpy_fromio() and memcpy_toio().
+ */
+static enum es_result vc_handle_mmio_movs(struct es_em_ctxt *ctxt,
+ unsigned int bytes)
+{
+ unsigned long ds_base, es_base;
+ unsigned char *src, *dst;
+ unsigned char buffer[8];
+ enum es_result ret;
+ bool rep;
+ int off;
+
+ ds_base = insn_get_seg_base(ctxt->regs, INAT_SEG_REG_DS);
+ es_base = insn_get_seg_base(ctxt->regs, INAT_SEG_REG_ES);
+
+ if (ds_base == -1L || es_base == -1L) {
+ ctxt->fi.vector = X86_TRAP_GP;
+ ctxt->fi.error_code = 0;
+ return ES_EXCEPTION;
+ }
+
+ src = ds_base + (unsigned char *)ctxt->regs->si;
+ dst = es_base + (unsigned char *)ctxt->regs->di;
+
+ ret = vc_read_mem(ctxt, src, buffer, bytes);
+ if (ret != ES_OK)
+ return ret;
+
+ ret = vc_write_mem(ctxt, dst, buffer, bytes);
+ if (ret != ES_OK)
+ return ret;
+
+ if (ctxt->regs->flags & X86_EFLAGS_DF)
+ off = -bytes;
+ else
+ off = bytes;
+
+ ctxt->regs->si += off;
+ ctxt->regs->di += off;
+
+ rep = insn_has_rep_prefix(&ctxt->insn);
+ if (rep)
+ ctxt->regs->cx -= 1;
+
+ if (!rep || ctxt->regs->cx == 0)
+ return ES_OK;
+ else
+ return ES_RETRY;
+}
+
static enum es_result vc_handle_mmio(struct ghcb *ghcb,
struct es_em_ctxt *ctxt)
{
@@ -655,6 +722,16 @@ static enum es_result vc_handle_mmio(struct ghcb *ghcb,
memcpy(reg_data, ghcb->shared_buffer, bytes);
break;
+ /* MOVS instruction */
+ case 0xa4:
+ bytes = 1;
+ fallthrough;
+ case 0xa5:
+ if (!bytes)
+ bytes = insn->opnd_bytes;
+
+ ret = vc_handle_mmio_movs(ctxt, bytes);
+ break;
/* Two-Byte Opcodes */
case 0x0f:
ret = vc_handle_mmio_twobyte_ops(ghcb, ctxt);
^ permalink raw reply related [flat|nested] 264+ messages in thread
* [tip: x86/seves] x86/sev-es: Wire up existing #VC exit-code handlers
2020-09-07 13:15 ` Joerg Roedel
(?)
@ 2020-09-10 9:22 ` tip-bot2 for Joerg Roedel
-1 siblings, 0 replies; 264+ messages in thread
From: tip-bot2 for Joerg Roedel @ 2020-09-10 9:22 UTC (permalink / raw)
To: linux-tip-commits; +Cc: Joerg Roedel, Borislav Petkov, x86, LKML
The following commit has been merged into the x86/seves branch of tip:
Commit-ID: d3529bb73f76d0ec8aafaca505226fa0971c1dc9
Gitweb: https://git.kernel.org/tip/d3529bb73f76d0ec8aafaca505226fa0971c1dc9
Author: Joerg Roedel <jroedel@suse.de>
AuthorDate: Mon, 07 Sep 2020 15:15:48 +02:00
Committer: Borislav Petkov <bp@suse.de>
CommitterDate: Wed, 09 Sep 2020 11:33:19 +02:00
x86/sev-es: Wire up existing #VC exit-code handlers
Re-use the handlers for CPUID- and IOIO-caused #VC exceptions in the
early boot handler.
Signed-off-by: Joerg Roedel <jroedel@suse.de>
Signed-off-by: Borislav Petkov <bp@suse.de>
Link: https://lkml.kernel.org/r/20200907131613.12703-48-joro@8bytes.org
---
arch/x86/kernel/sev-es-shared.c | 7 +++----
arch/x86/kernel/sev-es.c | 6 ++++++
2 files changed, 9 insertions(+), 4 deletions(-)
diff --git a/arch/x86/kernel/sev-es-shared.c b/arch/x86/kernel/sev-es-shared.c
index 53e3de0..491b557 100644
--- a/arch/x86/kernel/sev-es-shared.c
+++ b/arch/x86/kernel/sev-es-shared.c
@@ -325,8 +325,7 @@ static enum es_result vc_ioio_exitinfo(struct es_em_ctxt *ctxt, u64 *exitinfo)
return ES_OK;
}
-static enum es_result __maybe_unused
-vc_handle_ioio(struct ghcb *ghcb, struct es_em_ctxt *ctxt)
+static enum es_result vc_handle_ioio(struct ghcb *ghcb, struct es_em_ctxt *ctxt)
{
struct pt_regs *regs = ctxt->regs;
u64 exit_info_1, exit_info_2;
@@ -434,8 +433,8 @@ vc_handle_ioio(struct ghcb *ghcb, struct es_em_ctxt *ctxt)
return ret;
}
-static enum es_result __maybe_unused vc_handle_cpuid(struct ghcb *ghcb,
- struct es_em_ctxt *ctxt)
+static enum es_result vc_handle_cpuid(struct ghcb *ghcb,
+ struct es_em_ctxt *ctxt)
{
struct pt_regs *regs = ctxt->regs;
u32 cr4 = native_read_cr4();
diff --git a/arch/x86/kernel/sev-es.c b/arch/x86/kernel/sev-es.c
index 0d6b66e..b10a62a 100644
--- a/arch/x86/kernel/sev-es.c
+++ b/arch/x86/kernel/sev-es.c
@@ -441,6 +441,12 @@ static enum es_result vc_handle_exitcode(struct es_em_ctxt *ctxt,
enum es_result result;
switch (exit_code) {
+ case SVM_EXIT_CPUID:
+ result = vc_handle_cpuid(ghcb, ctxt);
+ break;
+ case SVM_EXIT_IOIO:
+ result = vc_handle_ioio(ghcb, ctxt);
+ break;
default:
/*
* Unexpected #VC exception
^ permalink raw reply related [flat|nested] 264+ messages in thread
* [tip: x86/seves] x86/sev-es: Handle instruction fetches from user-space
2020-09-07 13:15 ` Joerg Roedel
(?)
@ 2020-09-10 9:22 ` tip-bot2 for Joerg Roedel
-1 siblings, 0 replies; 264+ messages in thread
From: tip-bot2 for Joerg Roedel @ 2020-09-10 9:22 UTC (permalink / raw)
To: linux-tip-commits; +Cc: Joerg Roedel, Borislav Petkov, x86, LKML
The following commit has been merged into the x86/seves branch of tip:
Commit-ID: 5e3427a7bc432ed2e5de394ac30f160cc6c37a1f
Gitweb: https://git.kernel.org/tip/5e3427a7bc432ed2e5de394ac30f160cc6c37a1f
Author: Joerg Roedel <jroedel@suse.de>
AuthorDate: Mon, 07 Sep 2020 15:15:49 +02:00
Committer: Borislav Petkov <bp@suse.de>
CommitterDate: Wed, 09 Sep 2020 11:33:19 +02:00
x86/sev-es: Handle instruction fetches from user-space
When a #VC exception is triggered by user-space, the instruction decoder
needs to read the instruction bytes from user addresses. Enhance
vc_decode_insn() to safely fetch kernel and user instructions.
Signed-off-by: Joerg Roedel <jroedel@suse.de>
Signed-off-by: Borislav Petkov <bp@suse.de>
Link: https://lkml.kernel.org/r/20200907131613.12703-49-joro@8bytes.org
---
arch/x86/kernel/sev-es.c | 31 ++++++++++++++++++++++---------
1 file changed, 22 insertions(+), 9 deletions(-)
diff --git a/arch/x86/kernel/sev-es.c b/arch/x86/kernel/sev-es.c
index b10a62a..6c30dbc 100644
--- a/arch/x86/kernel/sev-es.c
+++ b/arch/x86/kernel/sev-es.c
@@ -232,17 +232,30 @@ static enum es_result vc_decode_insn(struct es_em_ctxt *ctxt)
enum es_result ret;
int res;
- res = vc_fetch_insn_kernel(ctxt, buffer);
- if (unlikely(res == -EFAULT)) {
- ctxt->fi.vector = X86_TRAP_PF;
- ctxt->fi.error_code = 0;
- ctxt->fi.cr2 = ctxt->regs->ip;
- return ES_EXCEPTION;
+ if (user_mode(ctxt->regs)) {
+ res = insn_fetch_from_user(ctxt->regs, buffer);
+ if (!res) {
+ ctxt->fi.vector = X86_TRAP_PF;
+ ctxt->fi.error_code = X86_PF_INSTR | X86_PF_USER;
+ ctxt->fi.cr2 = ctxt->regs->ip;
+ return ES_EXCEPTION;
+ }
+
+ if (!insn_decode(&ctxt->insn, ctxt->regs, buffer, res))
+ return ES_DECODE_FAILED;
+ } else {
+ res = vc_fetch_insn_kernel(ctxt, buffer);
+ if (res) {
+ ctxt->fi.vector = X86_TRAP_PF;
+ ctxt->fi.error_code = X86_PF_INSTR;
+ ctxt->fi.cr2 = ctxt->regs->ip;
+ return ES_EXCEPTION;
+ }
+
+ insn_init(&ctxt->insn, buffer, MAX_INSN_SIZE - res, 1);
+ insn_get_length(&ctxt->insn);
}
- insn_init(&ctxt->insn, buffer, MAX_INSN_SIZE - res, 1);
- insn_get_length(&ctxt->insn);
-
ret = ctxt->insn.immediate.got ? ES_OK : ES_DECODE_FAILED;
return ret;
^ permalink raw reply related [flat|nested] 264+ messages in thread
* [tip: x86/seves] x86/dumpstack/64: Add noinstr version of get_stack_info()
2020-09-07 13:15 ` Joerg Roedel
(?)
@ 2020-09-10 9:22 ` tip-bot2 for Joerg Roedel
-1 siblings, 0 replies; 264+ messages in thread
From: tip-bot2 for Joerg Roedel @ 2020-09-10 9:22 UTC (permalink / raw)
To: linux-tip-commits; +Cc: Joerg Roedel, Borislav Petkov, x86, LKML
The following commit has been merged into the x86/seves branch of tip:
Commit-ID: 6b27edd74a5e9669120f7bd0ae1f475d124c1042
Gitweb: https://git.kernel.org/tip/6b27edd74a5e9669120f7bd0ae1f475d124c1042
Author: Joerg Roedel <jroedel@suse.de>
AuthorDate: Mon, 07 Sep 2020 15:15:45 +02:00
Committer: Borislav Petkov <bp@suse.de>
CommitterDate: Wed, 09 Sep 2020 11:33:19 +02:00
x86/dumpstack/64: Add noinstr version of get_stack_info()
The get_stack_info() functionality is needed in the entry code for the
#VC exception handler. Provide a version of it in the .text.noinstr
section which can be called safely from there.
Signed-off-by: Joerg Roedel <jroedel@suse.de>
Signed-off-by: Borislav Petkov <bp@suse.de>
Link: https://lkml.kernel.org/r/20200907131613.12703-45-joro@8bytes.org
---
arch/x86/include/asm/stacktrace.h | 2 ++-
arch/x86/kernel/dumpstack.c | 7 +++---
arch/x86/kernel/dumpstack_64.c | 38 +++++++++++++++++-------------
arch/x86/mm/cpu_entry_area.c | 3 +-
4 files changed, 30 insertions(+), 20 deletions(-)
diff --git a/arch/x86/include/asm/stacktrace.h b/arch/x86/include/asm/stacktrace.h
index 5ae5a68..4960064 100644
--- a/arch/x86/include/asm/stacktrace.h
+++ b/arch/x86/include/asm/stacktrace.h
@@ -35,6 +35,8 @@ bool in_entry_stack(unsigned long *stack, struct stack_info *info);
int get_stack_info(unsigned long *stack, struct task_struct *task,
struct stack_info *info, unsigned long *visit_mask);
+bool get_stack_info_noinstr(unsigned long *stack, struct task_struct *task,
+ struct stack_info *info);
const char *stack_type_name(enum stack_type type);
diff --git a/arch/x86/kernel/dumpstack.c b/arch/x86/kernel/dumpstack.c
index 48ce445..74147f7 100644
--- a/arch/x86/kernel/dumpstack.c
+++ b/arch/x86/kernel/dumpstack.c
@@ -29,8 +29,8 @@ static int die_counter;
static struct pt_regs exec_summary_regs;
-bool in_task_stack(unsigned long *stack, struct task_struct *task,
- struct stack_info *info)
+bool noinstr in_task_stack(unsigned long *stack, struct task_struct *task,
+ struct stack_info *info)
{
unsigned long *begin = task_stack_page(task);
unsigned long *end = task_stack_page(task) + THREAD_SIZE;
@@ -46,7 +46,8 @@ bool in_task_stack(unsigned long *stack, struct task_struct *task,
return true;
}
-bool in_entry_stack(unsigned long *stack, struct stack_info *info)
+/* Called from get_stack_info_noinstr - so must be noinstr too */
+bool noinstr in_entry_stack(unsigned long *stack, struct stack_info *info)
{
struct entry_stack *ss = cpu_entry_stack(smp_processor_id());
diff --git a/arch/x86/kernel/dumpstack_64.c b/arch/x86/kernel/dumpstack_64.c
index c49cf59..1dd8513 100644
--- a/arch/x86/kernel/dumpstack_64.c
+++ b/arch/x86/kernel/dumpstack_64.c
@@ -85,7 +85,7 @@ struct estack_pages estack_pages[CEA_ESTACK_PAGES] ____cacheline_aligned = {
EPAGERANGE(VC2),
};
-static bool in_exception_stack(unsigned long *stack, struct stack_info *info)
+static __always_inline bool in_exception_stack(unsigned long *stack, struct stack_info *info)
{
unsigned long begin, end, stk = (unsigned long)stack;
const struct estack_pages *ep;
@@ -126,7 +126,7 @@ static bool in_exception_stack(unsigned long *stack, struct stack_info *info)
return true;
}
-static bool in_irq_stack(unsigned long *stack, struct stack_info *info)
+static __always_inline bool in_irq_stack(unsigned long *stack, struct stack_info *info)
{
unsigned long *end = (unsigned long *)this_cpu_read(hardirq_stack_ptr);
unsigned long *begin = end - (IRQ_STACK_SIZE / sizeof(long));
@@ -151,32 +151,38 @@ static bool in_irq_stack(unsigned long *stack, struct stack_info *info)
return true;
}
-int get_stack_info(unsigned long *stack, struct task_struct *task,
- struct stack_info *info, unsigned long *visit_mask)
+bool noinstr get_stack_info_noinstr(unsigned long *stack, struct task_struct *task,
+ struct stack_info *info)
{
- if (!stack)
- goto unknown;
-
- task = task ? : current;
-
if (in_task_stack(stack, task, info))
- goto recursion_check;
+ return true;
if (task != current)
- goto unknown;
+ return false;
if (in_exception_stack(stack, info))
- goto recursion_check;
+ return true;
if (in_irq_stack(stack, info))
- goto recursion_check;
+ return true;
if (in_entry_stack(stack, info))
- goto recursion_check;
+ return true;
+
+ return false;
+}
+
+int get_stack_info(unsigned long *stack, struct task_struct *task,
+ struct stack_info *info, unsigned long *visit_mask)
+{
+ task = task ? : current;
- goto unknown;
+ if (!stack)
+ goto unknown;
+
+ if (!get_stack_info_noinstr(stack, task, info))
+ goto unknown;
-recursion_check:
/*
* Make sure we don't iterate through any given stack more than once.
* If it comes up a second time then there's something wrong going on:
diff --git a/arch/x86/mm/cpu_entry_area.c b/arch/x86/mm/cpu_entry_area.c
index 770b613..f5e1e60 100644
--- a/arch/x86/mm/cpu_entry_area.c
+++ b/arch/x86/mm/cpu_entry_area.c
@@ -21,7 +21,8 @@ DEFINE_PER_CPU(struct cea_exception_stacks*, cea_exception_stacks);
DECLARE_PER_CPU_PAGE_ALIGNED(struct doublefault_stack, doublefault_stack);
#endif
-struct cpu_entry_area *get_cpu_entry_area(int cpu)
+/* Is called from entry code, so must be noinstr */
+noinstr struct cpu_entry_area *get_cpu_entry_area(int cpu)
{
unsigned long va = CPU_ENTRY_AREA_PER_CPU + cpu * CPU_ENTRY_AREA_SIZE;
BUILD_BUG_ON(sizeof(struct cpu_entry_area) % PAGE_SIZE != 0);
^ permalink raw reply related [flat|nested] 264+ messages in thread
* [tip: x86/seves] x86/entry/64: Add entry code for #VC handler
2020-09-07 13:15 ` Joerg Roedel
(?)
@ 2020-09-10 9:22 ` tip-bot2 for Joerg Roedel
-1 siblings, 0 replies; 264+ messages in thread
From: tip-bot2 for Joerg Roedel @ 2020-09-10 9:22 UTC (permalink / raw)
To: linux-tip-commits; +Cc: Joerg Roedel, Borislav Petkov, x86, LKML
The following commit has been merged into the x86/seves branch of tip:
Commit-ID: a13644f3a53de4e95a7bce6459f834e832ea44c5
Gitweb: https://git.kernel.org/tip/a13644f3a53de4e95a7bce6459f834e832ea44c5
Author: Joerg Roedel <jroedel@suse.de>
AuthorDate: Mon, 07 Sep 2020 15:15:46 +02:00
Committer: Borislav Petkov <bp@suse.de>
CommitterDate: Wed, 09 Sep 2020 11:33:19 +02:00
x86/entry/64: Add entry code for #VC handler
The #VC handler needs special entry code because:
1. It runs on an IST stack
2. It needs to be able to handle nested #VC exceptions
To make this work, the entry code is implemented to pretend it doesn't
use an IST stack. When entered from user-mode or early SYSCALL entry
path it switches to the task stack. If entered from kernel-mode it tries
to switch back to the previous stack in the IRET frame.
The stack found in the IRET frame is validated first, and if it is not
safe to use it for the #VC handler, the code will switch to a
fall-back stack (the #VC2 IST stack). From there, it can cause nested
exceptions again.
Signed-off-by: Joerg Roedel <jroedel@suse.de>
Signed-off-by: Borislav Petkov <bp@suse.de>
Link: https://lkml.kernel.org/r/20200907131613.12703-46-joro@8bytes.org
---
arch/x86/entry/entry_64.S | 80 ++++++++++++++++++++++++++++++++-
arch/x86/include/asm/idtentry.h | 44 ++++++++++++++++++-
arch/x86/include/asm/proto.h | 1 +-
arch/x86/include/asm/traps.h | 1 +-
arch/x86/kernel/traps.c | 45 ++++++++++++++++++-
5 files changed, 171 insertions(+)
diff --git a/arch/x86/entry/entry_64.S b/arch/x86/entry/entry_64.S
index 70dea93..15aa189 100644
--- a/arch/x86/entry/entry_64.S
+++ b/arch/x86/entry/entry_64.S
@@ -101,6 +101,8 @@ SYM_CODE_START(entry_SYSCALL_64)
SWITCH_TO_KERNEL_CR3 scratch_reg=%rsp
movq PER_CPU_VAR(cpu_current_top_of_stack), %rsp
+SYM_INNER_LABEL(entry_SYSCALL_64_safe_stack, SYM_L_GLOBAL)
+
/* Construct struct pt_regs on stack */
pushq $__USER_DS /* pt_regs->ss */
pushq PER_CPU_VAR(cpu_tss_rw + TSS_sp2) /* pt_regs->sp */
@@ -446,6 +448,84 @@ _ASM_NOKPROBE(\asmsym)
SYM_CODE_END(\asmsym)
.endm
+#ifdef CONFIG_AMD_MEM_ENCRYPT
+/**
+ * idtentry_vc - Macro to generate entry stub for #VC
+ * @vector: Vector number
+ * @asmsym: ASM symbol for the entry point
+ * @cfunc: C function to be called
+ *
+ * The macro emits code to set up the kernel context for #VC. The #VC handler
+ * runs on an IST stack and needs to be able to cause nested #VC exceptions.
+ *
+ * To make this work the #VC entry code tries its best to pretend it doesn't use
+ * an IST stack by switching to the task stack if coming from user-space (which
+ * includes early SYSCALL entry path) or back to the stack in the IRET frame if
+ * entered from kernel-mode.
+ *
+ * If entered from kernel-mode the return stack is validated first, and if it is
+ * not safe to use (e.g. because it points to the entry stack) the #VC handler
+ * will switch to a fall-back stack (VC2) and call a special handler function.
+ *
+ * The macro is only used for one vector, but it is planned to be extended in
+ * the future for the #HV exception.
+ */
+.macro idtentry_vc vector asmsym cfunc
+SYM_CODE_START(\asmsym)
+ UNWIND_HINT_IRET_REGS
+ ASM_CLAC
+
+ /*
+ * If the entry is from userspace, switch stacks and treat it as
+ * a normal entry.
+ */
+ testb $3, CS-ORIG_RAX(%rsp)
+ jnz .Lfrom_usermode_switch_stack_\@
+
+ /*
+ * paranoid_entry returns SWAPGS flag for paranoid_exit in EBX.
+ * EBX == 0 -> SWAPGS, EBX == 1 -> no SWAPGS
+ */
+ call paranoid_entry
+
+ UNWIND_HINT_REGS
+
+ /*
+ * Switch off the IST stack to make it free for nested exceptions. The
+ * vc_switch_off_ist() function will switch back to the interrupted
+ * stack if it is safe to do so. If not it switches to the VC fall-back
+ * stack.
+ */
+ movq %rsp, %rdi /* pt_regs pointer */
+ call vc_switch_off_ist
+ movq %rax, %rsp /* Switch to new stack */
+
+ UNWIND_HINT_REGS
+
+ /* Update pt_regs */
+ movq ORIG_RAX(%rsp), %rsi /* get error code into 2nd argument*/
+ movq $-1, ORIG_RAX(%rsp) /* no syscall to restart */
+
+ movq %rsp, %rdi /* pt_regs pointer */
+
+ call \cfunc
+
+ /*
+ * No need to switch back to the IST stack. The current stack is either
+ * identical to the stack in the IRET frame or the VC fall-back stack,
+ * so it is definitly mapped even with PTI enabled.
+ */
+ jmp paranoid_exit
+
+ /* Switch to the regular task stack */
+.Lfrom_usermode_switch_stack_\@:
+ idtentry_body safe_stack_\cfunc, has_error_code=1
+
+_ASM_NOKPROBE(\asmsym)
+SYM_CODE_END(\asmsym)
+.endm
+#endif
+
/*
* Double fault entry. Straight paranoid. No checks from which context
* this comes because for the espfix induced #DF this would do the wrong
diff --git a/arch/x86/include/asm/idtentry.h b/arch/x86/include/asm/idtentry.h
index a433661..840faaf 100644
--- a/arch/x86/include/asm/idtentry.h
+++ b/arch/x86/include/asm/idtentry.h
@@ -309,6 +309,18 @@ static __always_inline void __##func(struct pt_regs *regs)
__visible void noist_##func(struct pt_regs *regs)
/**
+ * DECLARE_IDTENTRY_VC - Declare functions for the VC entry point
+ * @vector: Vector number (ignored for C)
+ * @func: Function name of the entry point
+ *
+ * Maps to DECLARE_IDTENTRY_RAW_ERRORCODE, but declares also the
+ * safe_stack C handler.
+ */
+#define DECLARE_IDTENTRY_VC(vector, func) \
+ DECLARE_IDTENTRY_RAW_ERRORCODE(vector, func); \
+ __visible noinstr void safe_stack_##func(struct pt_regs *regs, unsigned long error_code)
+
+/**
* DEFINE_IDTENTRY_IST - Emit code for IST entry points
* @func: Function name of the entry point
*
@@ -347,6 +359,35 @@ static __always_inline void __##func(struct pt_regs *regs)
#define DEFINE_IDTENTRY_DF(func) \
DEFINE_IDTENTRY_RAW_ERRORCODE(func)
+/**
+ * DEFINE_IDTENTRY_VC_SAFE_STACK - Emit code for VMM communication handler
+ which runs on a safe stack.
+ * @func: Function name of the entry point
+ *
+ * Maps to DEFINE_IDTENTRY_RAW_ERRORCODE
+ */
+#define DEFINE_IDTENTRY_VC_SAFE_STACK(func) \
+ DEFINE_IDTENTRY_RAW_ERRORCODE(safe_stack_##func)
+
+/**
+ * DEFINE_IDTENTRY_VC_IST - Emit code for VMM communication handler
+ which runs on the VC fall-back stack
+ * @func: Function name of the entry point
+ *
+ * Maps to DEFINE_IDTENTRY_RAW_ERRORCODE
+ */
+#define DEFINE_IDTENTRY_VC_IST(func) \
+ DEFINE_IDTENTRY_RAW_ERRORCODE(ist_##func)
+
+/**
+ * DEFINE_IDTENTRY_VC - Emit code for VMM communication handler
+ * @func: Function name of the entry point
+ *
+ * Maps to DEFINE_IDTENTRY_RAW_ERRORCODE
+ */
+#define DEFINE_IDTENTRY_VC(func) \
+ DEFINE_IDTENTRY_RAW_ERRORCODE(func)
+
#else /* CONFIG_X86_64 */
/**
@@ -433,6 +474,9 @@ __visible noinstr void func(struct pt_regs *regs, \
# define DECLARE_IDTENTRY_XENCB(vector, func) \
DECLARE_IDTENTRY(vector, func)
+# define DECLARE_IDTENTRY_VC(vector, func) \
+ idtentry_vc vector asm_##func func
+
#else
# define DECLARE_IDTENTRY_MCE(vector, func) \
DECLARE_IDTENTRY(vector, func)
diff --git a/arch/x86/include/asm/proto.h b/arch/x86/include/asm/proto.h
index 28996fe..2c35f1c 100644
--- a/arch/x86/include/asm/proto.h
+++ b/arch/x86/include/asm/proto.h
@@ -10,6 +10,7 @@ void syscall_init(void);
#ifdef CONFIG_X86_64
void entry_SYSCALL_64(void);
+void entry_SYSCALL_64_safe_stack(void);
long do_arch_prctl_64(struct task_struct *task, int option, unsigned long arg2);
#endif
diff --git a/arch/x86/include/asm/traps.h b/arch/x86/include/asm/traps.h
index 6a30835..1b86bb3 100644
--- a/arch/x86/include/asm/traps.h
+++ b/arch/x86/include/asm/traps.h
@@ -15,6 +15,7 @@ asmlinkage __visible notrace struct pt_regs *sync_regs(struct pt_regs *eregs);
asmlinkage __visible notrace
struct bad_iret_stack *fixup_bad_iret(struct bad_iret_stack *s);
void __init trap_init(void);
+asmlinkage __visible noinstr struct pt_regs *vc_switch_off_ist(struct pt_regs *eregs);
#endif
#ifdef CONFIG_X86_F00F_BUG
diff --git a/arch/x86/kernel/traps.c b/arch/x86/kernel/traps.c
index 2dc32b0..fc1ed40 100644
--- a/arch/x86/kernel/traps.c
+++ b/arch/x86/kernel/traps.c
@@ -43,6 +43,7 @@
#include <asm/stacktrace.h>
#include <asm/processor.h>
#include <asm/debugreg.h>
+#include <asm/realmode.h>
#include <asm/text-patching.h>
#include <asm/ftrace.h>
#include <asm/traps.h>
@@ -673,6 +674,50 @@ asmlinkage __visible noinstr struct pt_regs *sync_regs(struct pt_regs *eregs)
return regs;
}
+#ifdef CONFIG_AMD_MEM_ENCRYPT
+asmlinkage __visible noinstr struct pt_regs *vc_switch_off_ist(struct pt_regs *regs)
+{
+ unsigned long sp, *stack;
+ struct stack_info info;
+ struct pt_regs *regs_ret;
+
+ /*
+ * In the SYSCALL entry path the RSP value comes from user-space - don't
+ * trust it and switch to the current kernel stack
+ */
+ if (regs->ip >= (unsigned long)entry_SYSCALL_64 &&
+ regs->ip < (unsigned long)entry_SYSCALL_64_safe_stack) {
+ sp = this_cpu_read(cpu_current_top_of_stack);
+ goto sync;
+ }
+
+ /*
+ * From here on the RSP value is trusted. Now check whether entry
+ * happened from a safe stack. Not safe are the entry or unknown stacks,
+ * use the fall-back stack instead in this case.
+ */
+ sp = regs->sp;
+ stack = (unsigned long *)sp;
+
+ if (!get_stack_info_noinstr(stack, current, &info) || info.type == STACK_TYPE_ENTRY ||
+ info.type >= STACK_TYPE_EXCEPTION_LAST)
+ sp = __this_cpu_ist_top_va(VC2);
+
+sync:
+ /*
+ * Found a safe stack - switch to it as if the entry didn't happen via
+ * IST stack. The code below only copies pt_regs, the real switch happens
+ * in assembly code.
+ */
+ sp = ALIGN_DOWN(sp, 8) - sizeof(*regs_ret);
+
+ regs_ret = (struct pt_regs *)sp;
+ *regs_ret = *regs;
+
+ return regs_ret;
+}
+#endif
+
struct bad_iret_stack {
void *error_entry_ret;
struct pt_regs regs;
^ permalink raw reply related [flat|nested] 264+ messages in thread
* [tip: x86/seves] x86/sev-es: Add a Runtime #VC Exception Handler
2020-09-07 13:15 ` Joerg Roedel
(?)
@ 2020-09-10 9:22 ` tip-bot2 for Tom Lendacky
-1 siblings, 0 replies; 264+ messages in thread
From: tip-bot2 for Tom Lendacky @ 2020-09-10 9:22 UTC (permalink / raw)
To: linux-tip-commits; +Cc: Tom Lendacky, Joerg Roedel, Borislav Petkov, x86, LKML
The following commit has been merged into the x86/seves branch of tip:
Commit-ID: 0786138c78e79343c7b015d77507cbf9d5f15d00
Gitweb: https://git.kernel.org/tip/0786138c78e79343c7b015d77507cbf9d5f15d00
Author: Tom Lendacky <thomas.lendacky@amd.com>
AuthorDate: Mon, 07 Sep 2020 15:15:47 +02:00
Committer: Borislav Petkov <bp@suse.de>
CommitterDate: Wed, 09 Sep 2020 11:33:19 +02:00
x86/sev-es: Add a Runtime #VC Exception Handler
Add the handlers for #VC exceptions invoked at runtime.
Signed-off-by: Tom Lendacky <thomas.lendacky@amd.com>
Signed-off-by: Joerg Roedel <jroedel@suse.de>
Signed-off-by: Borislav Petkov <bp@suse.de>
Link: https://lkml.kernel.org/r/20200907131613.12703-47-joro@8bytes.org
---
arch/x86/include/asm/idtentry.h | 6 +-
arch/x86/kernel/idt.c | 11 +-
arch/x86/kernel/sev-es.c | 246 ++++++++++++++++++++++++++++++-
3 files changed, 255 insertions(+), 8 deletions(-)
diff --git a/arch/x86/include/asm/idtentry.h b/arch/x86/include/asm/idtentry.h
index 840faaf..58a793b 100644
--- a/arch/x86/include/asm/idtentry.h
+++ b/arch/x86/include/asm/idtentry.h
@@ -318,6 +318,7 @@ static __always_inline void __##func(struct pt_regs *regs)
*/
#define DECLARE_IDTENTRY_VC(vector, func) \
DECLARE_IDTENTRY_RAW_ERRORCODE(vector, func); \
+ __visible noinstr void ist_##func(struct pt_regs *regs, unsigned long error_code); \
__visible noinstr void safe_stack_##func(struct pt_regs *regs, unsigned long error_code)
/**
@@ -608,6 +609,11 @@ DECLARE_IDTENTRY_RAW(X86_TRAP_DB, xenpv_exc_debug);
/* #DF */
DECLARE_IDTENTRY_DF(X86_TRAP_DF, exc_double_fault);
+/* #VC */
+#ifdef CONFIG_AMD_MEM_ENCRYPT
+DECLARE_IDTENTRY_VC(X86_TRAP_VC, exc_vmm_communication);
+#endif
+
#ifdef CONFIG_XEN_PV
DECLARE_IDTENTRY_XENCB(X86_TRAP_OTHER, exc_xen_hypervisor_callback);
#endif
diff --git a/arch/x86/kernel/idt.c b/arch/x86/kernel/idt.c
index 4bb4e3d..e29a6c7 100644
--- a/arch/x86/kernel/idt.c
+++ b/arch/x86/kernel/idt.c
@@ -229,11 +229,14 @@ static const __initconst struct idt_data early_pf_idts[] = {
* cpu_init() when the TSS has been initialized.
*/
static const __initconst struct idt_data ist_idts[] = {
- ISTG(X86_TRAP_DB, asm_exc_debug, IST_INDEX_DB),
- ISTG(X86_TRAP_NMI, asm_exc_nmi, IST_INDEX_NMI),
- ISTG(X86_TRAP_DF, asm_exc_double_fault, IST_INDEX_DF),
+ ISTG(X86_TRAP_DB, asm_exc_debug, IST_INDEX_DB),
+ ISTG(X86_TRAP_NMI, asm_exc_nmi, IST_INDEX_NMI),
+ ISTG(X86_TRAP_DF, asm_exc_double_fault, IST_INDEX_DF),
#ifdef CONFIG_X86_MCE
- ISTG(X86_TRAP_MC, asm_exc_machine_check, IST_INDEX_MCE),
+ ISTG(X86_TRAP_MC, asm_exc_machine_check, IST_INDEX_MCE),
+#endif
+#ifdef CONFIG_AMD_MEM_ENCRYPT
+ ISTG(X86_TRAP_VC, asm_exc_vmm_communication, IST_INDEX_VC),
#endif
};
diff --git a/arch/x86/kernel/sev-es.c b/arch/x86/kernel/sev-es.c
index 39ebb2d..0d6b66e 100644
--- a/arch/x86/kernel/sev-es.c
+++ b/arch/x86/kernel/sev-es.c
@@ -7,9 +7,12 @@
* Author: Joerg Roedel <jroedel@suse.de>
*/
+#define pr_fmt(fmt) "SEV-ES: " fmt
+
#include <linux/sched/debug.h> /* For show_regs() */
#include <linux/percpu-defs.h>
#include <linux/mem_encrypt.h>
+#include <linux/lockdep.h>
#include <linux/printk.h>
#include <linux/mm_types.h>
#include <linux/set_memory.h>
@@ -22,8 +25,8 @@
#include <asm/insn-eval.h>
#include <asm/fpu/internal.h>
#include <asm/processor.h>
-#include <asm/trap_pf.h>
-#include <asm/trapnr.h>
+#include <asm/realmode.h>
+#include <asm/traps.h>
#include <asm/svm.h>
/* For early boot hypervisor communication in SEV-ES enabled guests */
@@ -48,11 +51,43 @@ struct sev_es_runtime_data {
* interrupted stack in the #VC entry code.
*/
char fallback_stack[EXCEPTION_STKSZ] __aligned(PAGE_SIZE);
+
+ /*
+ * Reserve one page per CPU as backup storage for the unencrypted GHCB.
+ * It is needed when an NMI happens while the #VC handler uses the real
+ * GHCB, and the NMI handler itself is causing another #VC exception. In
+ * that case the GHCB content of the first handler needs to be backed up
+ * and restored.
+ */
+ struct ghcb backup_ghcb;
+
+ /*
+ * Mark the per-cpu GHCBs as in-use to detect nested #VC exceptions.
+ * There is no need for it to be atomic, because nothing is written to
+ * the GHCB between the read and the write of ghcb_active. So it is safe
+ * to use it when a nested #VC exception happens before the write.
+ *
+ * This is necessary for example in the #VC->NMI->#VC case when the NMI
+ * happens while the first #VC handler uses the GHCB. When the NMI code
+ * raises a second #VC handler it might overwrite the contents of the
+ * GHCB written by the first handler. To avoid this the content of the
+ * GHCB is saved and restored when the GHCB is detected to be in use
+ * already.
+ */
+ bool ghcb_active;
+ bool backup_ghcb_active;
+};
+
+struct ghcb_state {
+ struct ghcb *ghcb;
};
static DEFINE_PER_CPU(struct sev_es_runtime_data*, runtime_data);
DEFINE_STATIC_KEY_FALSE(sev_es_enable_key);
+/* Needed in vc_early_forward_exception */
+void do_early_exception(struct pt_regs *regs, int trapnr);
+
static void __init setup_vc_stacks(int cpu)
{
struct sev_es_runtime_data *data;
@@ -123,8 +158,52 @@ void noinstr __sev_es_ist_exit(void)
this_cpu_write(cpu_tss_rw.x86_tss.ist[IST_INDEX_VC], *(unsigned long *)ist);
}
-/* Needed in vc_early_forward_exception */
-void do_early_exception(struct pt_regs *regs, int trapnr);
+static __always_inline struct ghcb *sev_es_get_ghcb(struct ghcb_state *state)
+{
+ struct sev_es_runtime_data *data;
+ struct ghcb *ghcb;
+
+ data = this_cpu_read(runtime_data);
+ ghcb = &data->ghcb_page;
+
+ if (unlikely(data->ghcb_active)) {
+ /* GHCB is already in use - save its contents */
+
+ if (unlikely(data->backup_ghcb_active))
+ return NULL;
+
+ /* Mark backup_ghcb active before writing to it */
+ data->backup_ghcb_active = true;
+
+ state->ghcb = &data->backup_ghcb;
+
+ /* Backup GHCB content */
+ *state->ghcb = *ghcb;
+ } else {
+ state->ghcb = NULL;
+ data->ghcb_active = true;
+ }
+
+ return ghcb;
+}
+
+static __always_inline void sev_es_put_ghcb(struct ghcb_state *state)
+{
+ struct sev_es_runtime_data *data;
+ struct ghcb *ghcb;
+
+ data = this_cpu_read(runtime_data);
+ ghcb = &data->ghcb_page;
+
+ if (state->ghcb) {
+ /* Restore GHCB from Backup */
+ *ghcb = *state->ghcb;
+ data->backup_ghcb_active = false;
+ state->ghcb = NULL;
+ } else {
+ data->ghcb_active = false;
+ }
+}
static inline u64 sev_es_rd_ghcb_msr(void)
{
@@ -316,6 +395,9 @@ static void __init init_ghcb(int cpu)
panic("Can't map GHCBs unencrypted");
memset(&data->ghcb_page, 0, sizeof(data->ghcb_page));
+
+ data->ghcb_active = false;
+ data->backup_ghcb_active = false;
}
void __init sev_es_init_vc_handling(void)
@@ -336,6 +418,9 @@ void __init sev_es_init_vc_handling(void)
init_ghcb(cpu);
setup_vc_stacks(cpu);
}
+
+ /* Secondary CPUs use the runtime #VC handler */
+ initial_vc_handler = (unsigned long)safe_stack_exc_vmm_communication;
}
static void __init vc_early_forward_exception(struct es_em_ctxt *ctxt)
@@ -366,6 +451,159 @@ static enum es_result vc_handle_exitcode(struct es_em_ctxt *ctxt,
return result;
}
+static __always_inline void vc_forward_exception(struct es_em_ctxt *ctxt)
+{
+ long error_code = ctxt->fi.error_code;
+ int trapnr = ctxt->fi.vector;
+
+ ctxt->regs->orig_ax = ctxt->fi.error_code;
+
+ switch (trapnr) {
+ case X86_TRAP_GP:
+ exc_general_protection(ctxt->regs, error_code);
+ break;
+ case X86_TRAP_UD:
+ exc_invalid_op(ctxt->regs);
+ break;
+ default:
+ pr_emerg("Unsupported exception in #VC instruction emulation - can't continue\n");
+ BUG();
+ }
+}
+
+static __always_inline bool on_vc_fallback_stack(struct pt_regs *regs)
+{
+ unsigned long sp = (unsigned long)regs;
+
+ return (sp >= __this_cpu_ist_bottom_va(VC2) && sp < __this_cpu_ist_top_va(VC2));
+}
+
+/*
+ * Main #VC exception handler. It is called when the entry code was able to
+ * switch off the IST to a safe kernel stack.
+ *
+ * With the current implementation it is always possible to switch to a safe
+ * stack because #VC exceptions only happen at known places, like intercepted
+ * instructions or accesses to MMIO areas/IO ports. They can also happen with
+ * code instrumentation when the hypervisor intercepts #DB, but the critical
+ * paths are forbidden to be instrumented, so #DB exceptions currently also
+ * only happen in safe places.
+ */
+DEFINE_IDTENTRY_VC_SAFE_STACK(exc_vmm_communication)
+{
+ struct sev_es_runtime_data *data = this_cpu_read(runtime_data);
+ struct ghcb_state state;
+ struct es_em_ctxt ctxt;
+ enum es_result result;
+ struct ghcb *ghcb;
+
+ lockdep_assert_irqs_disabled();
+ instrumentation_begin();
+
+ /*
+ * This is invoked through an interrupt gate, so IRQs are disabled. The
+ * code below might walk page-tables for user or kernel addresses, so
+ * keep the IRQs disabled to protect us against concurrent TLB flushes.
+ */
+
+ ghcb = sev_es_get_ghcb(&state);
+ if (!ghcb) {
+ /*
+ * Mark GHCBs inactive so that panic() is able to print the
+ * message.
+ */
+ data->ghcb_active = false;
+ data->backup_ghcb_active = false;
+
+ panic("Unable to handle #VC exception! GHCB and Backup GHCB are already in use");
+ }
+
+ vc_ghcb_invalidate(ghcb);
+ result = vc_init_em_ctxt(&ctxt, regs, error_code);
+
+ if (result == ES_OK)
+ result = vc_handle_exitcode(&ctxt, ghcb, error_code);
+
+ sev_es_put_ghcb(&state);
+
+ /* Done - now check the result */
+ switch (result) {
+ case ES_OK:
+ vc_finish_insn(&ctxt);
+ break;
+ case ES_UNSUPPORTED:
+ pr_err_ratelimited("Unsupported exit-code 0x%02lx in early #VC exception (IP: 0x%lx)\n",
+ error_code, regs->ip);
+ goto fail;
+ case ES_VMM_ERROR:
+ pr_err_ratelimited("Failure in communication with VMM (exit-code 0x%02lx IP: 0x%lx)\n",
+ error_code, regs->ip);
+ goto fail;
+ case ES_DECODE_FAILED:
+ pr_err_ratelimited("Failed to decode instruction (exit-code 0x%02lx IP: 0x%lx)\n",
+ error_code, regs->ip);
+ goto fail;
+ case ES_EXCEPTION:
+ vc_forward_exception(&ctxt);
+ break;
+ case ES_RETRY:
+ /* Nothing to do */
+ break;
+ default:
+ pr_emerg("Unknown result in %s():%d\n", __func__, result);
+ /*
+ * Emulating the instruction which caused the #VC exception
+ * failed - can't continue so print debug information
+ */
+ BUG();
+ }
+
+out:
+ instrumentation_end();
+
+ return;
+
+fail:
+ if (user_mode(regs)) {
+ /*
+ * Do not kill the machine if user-space triggered the
+ * exception. Send SIGBUS instead and let user-space deal with
+ * it.
+ */
+ force_sig_fault(SIGBUS, BUS_OBJERR, (void __user *)0);
+ } else {
+ pr_emerg("PANIC: Unhandled #VC exception in kernel space (result=%d)\n",
+ result);
+
+ /* Show some debug info */
+ show_regs(regs);
+
+ /* Ask hypervisor to sev_es_terminate */
+ sev_es_terminate(GHCB_SEV_ES_REASON_GENERAL_REQUEST);
+
+ /* If that fails and we get here - just panic */
+ panic("Returned from Terminate-Request to Hypervisor\n");
+ }
+
+ goto out;
+}
+
+/* This handler runs on the #VC fall-back stack. It can cause further #VC exceptions */
+DEFINE_IDTENTRY_VC_IST(exc_vmm_communication)
+{
+ instrumentation_begin();
+ panic("Can't handle #VC exception from unsupported context\n");
+ instrumentation_end();
+}
+
+DEFINE_IDTENTRY_VC(exc_vmm_communication)
+{
+ if (likely(!on_vc_fallback_stack(regs)))
+ safe_stack_exc_vmm_communication(regs, error_code);
+ else
+ ist_exc_vmm_communication(regs, error_code);
+}
+
bool __init handle_vc_boot_ghcb(struct pt_regs *regs)
{
unsigned long exit_code = regs->orig_ax;
^ permalink raw reply related [flat|nested] 264+ messages in thread
* [tip: x86/seves] x86/sev-es: Allocate and map an IST stack for #VC handler
2020-09-07 13:15 ` Joerg Roedel
(?)
@ 2020-09-10 9:22 ` tip-bot2 for Joerg Roedel
-1 siblings, 0 replies; 264+ messages in thread
From: tip-bot2 for Joerg Roedel @ 2020-09-10 9:22 UTC (permalink / raw)
To: linux-tip-commits; +Cc: Joerg Roedel, Borislav Petkov, x86, LKML
The following commit has been merged into the x86/seves branch of tip:
Commit-ID: 02772fb9b68e6a72a5e17f994048df832fe2b15e
Gitweb: https://git.kernel.org/tip/02772fb9b68e6a72a5e17f994048df832fe2b15e
Author: Joerg Roedel <jroedel@suse.de>
AuthorDate: Mon, 07 Sep 2020 15:15:43 +02:00
Committer: Borislav Petkov <bp@suse.de>
CommitterDate: Wed, 09 Sep 2020 11:33:19 +02:00
x86/sev-es: Allocate and map an IST stack for #VC handler
Allocate and map an IST stack and an additional fall-back stack for
the #VC handler. The memory for the stacks is allocated only when
SEV-ES is active.
The #VC handler needs to use an IST stack because a #VC exception can be
raised from kernel space with unsafe stack, e.g. in the SYSCALL entry
path.
Since the #VC exception can be nested, the #VC handler switches back to
the interrupted stack when entered from kernel space. If switching back
is not possible, the fall-back stack is used.
Signed-off-by: Joerg Roedel <jroedel@suse.de>
Signed-off-by: Borislav Petkov <bp@suse.de>
Link: https://lkml.kernel.org/r/20200907131613.12703-43-joro@8bytes.org
---
arch/x86/include/asm/cpu_entry_area.h | 33 ++++++++++++++++----------
arch/x86/include/asm/page_64_types.h | 1 +-
arch/x86/kernel/cpu/common.c | 2 ++-
arch/x86/kernel/dumpstack_64.c | 8 ++++--
arch/x86/kernel/sev-es.c | 33 ++++++++++++++++++++++++++-
5 files changed, 63 insertions(+), 14 deletions(-)
diff --git a/arch/x86/include/asm/cpu_entry_area.h b/arch/x86/include/asm/cpu_entry_area.h
index 8902fdb..3d52b09 100644
--- a/arch/x86/include/asm/cpu_entry_area.h
+++ b/arch/x86/include/asm/cpu_entry_area.h
@@ -11,25 +11,29 @@
#ifdef CONFIG_X86_64
/* Macro to enforce the same ordering and stack sizes */
-#define ESTACKS_MEMBERS(guardsize) \
- char DF_stack_guard[guardsize]; \
- char DF_stack[EXCEPTION_STKSZ]; \
- char NMI_stack_guard[guardsize]; \
- char NMI_stack[EXCEPTION_STKSZ]; \
- char DB_stack_guard[guardsize]; \
- char DB_stack[EXCEPTION_STKSZ]; \
- char MCE_stack_guard[guardsize]; \
- char MCE_stack[EXCEPTION_STKSZ]; \
- char IST_top_guard[guardsize]; \
+#define ESTACKS_MEMBERS(guardsize, optional_stack_size) \
+ char DF_stack_guard[guardsize]; \
+ char DF_stack[EXCEPTION_STKSZ]; \
+ char NMI_stack_guard[guardsize]; \
+ char NMI_stack[EXCEPTION_STKSZ]; \
+ char DB_stack_guard[guardsize]; \
+ char DB_stack[EXCEPTION_STKSZ]; \
+ char MCE_stack_guard[guardsize]; \
+ char MCE_stack[EXCEPTION_STKSZ]; \
+ char VC_stack_guard[guardsize]; \
+ char VC_stack[optional_stack_size]; \
+ char VC2_stack_guard[guardsize]; \
+ char VC2_stack[optional_stack_size]; \
+ char IST_top_guard[guardsize]; \
/* The exception stacks' physical storage. No guard pages required */
struct exception_stacks {
- ESTACKS_MEMBERS(0)
+ ESTACKS_MEMBERS(0, 0)
};
/* The effective cpu entry area mapping with guard pages. */
struct cea_exception_stacks {
- ESTACKS_MEMBERS(PAGE_SIZE)
+ ESTACKS_MEMBERS(PAGE_SIZE, EXCEPTION_STKSZ)
};
/*
@@ -40,6 +44,8 @@ enum exception_stack_ordering {
ESTACK_NMI,
ESTACK_DB,
ESTACK_MCE,
+ ESTACK_VC,
+ ESTACK_VC2,
N_EXCEPTION_STACKS
};
@@ -139,4 +145,7 @@ static inline struct entry_stack *cpu_entry_stack(int cpu)
#define __this_cpu_ist_top_va(name) \
CEA_ESTACK_TOP(__this_cpu_read(cea_exception_stacks), name)
+#define __this_cpu_ist_bottom_va(name) \
+ CEA_ESTACK_BOT(__this_cpu_read(cea_exception_stacks), name)
+
#endif
diff --git a/arch/x86/include/asm/page_64_types.h b/arch/x86/include/asm/page_64_types.h
index 288b065..d0c6c10 100644
--- a/arch/x86/include/asm/page_64_types.h
+++ b/arch/x86/include/asm/page_64_types.h
@@ -28,6 +28,7 @@
#define IST_INDEX_NMI 1
#define IST_INDEX_DB 2
#define IST_INDEX_MCE 3
+#define IST_INDEX_VC 4
/*
* Set __PAGE_OFFSET to the most negative possible address +
diff --git a/arch/x86/kernel/cpu/common.c b/arch/x86/kernel/cpu/common.c
index c5d6f17..81fba4d 100644
--- a/arch/x86/kernel/cpu/common.c
+++ b/arch/x86/kernel/cpu/common.c
@@ -1829,6 +1829,8 @@ static inline void tss_setup_ist(struct tss_struct *tss)
tss->x86_tss.ist[IST_INDEX_NMI] = __this_cpu_ist_top_va(NMI);
tss->x86_tss.ist[IST_INDEX_DB] = __this_cpu_ist_top_va(DB);
tss->x86_tss.ist[IST_INDEX_MCE] = __this_cpu_ist_top_va(MCE);
+ /* Only mapped when SEV-ES is active */
+ tss->x86_tss.ist[IST_INDEX_VC] = __this_cpu_ist_top_va(VC);
}
#else /* CONFIG_X86_64 */
diff --git a/arch/x86/kernel/dumpstack_64.c b/arch/x86/kernel/dumpstack_64.c
index 4a94d38..c49cf59 100644
--- a/arch/x86/kernel/dumpstack_64.c
+++ b/arch/x86/kernel/dumpstack_64.c
@@ -24,11 +24,13 @@ static const char * const exception_stack_names[] = {
[ ESTACK_NMI ] = "NMI",
[ ESTACK_DB ] = "#DB",
[ ESTACK_MCE ] = "#MC",
+ [ ESTACK_VC ] = "#VC",
+ [ ESTACK_VC2 ] = "#VC2",
};
const char *stack_type_name(enum stack_type type)
{
- BUILD_BUG_ON(N_EXCEPTION_STACKS != 4);
+ BUILD_BUG_ON(N_EXCEPTION_STACKS != 6);
if (type == STACK_TYPE_IRQ)
return "IRQ";
@@ -79,6 +81,8 @@ struct estack_pages estack_pages[CEA_ESTACK_PAGES] ____cacheline_aligned = {
EPAGERANGE(NMI),
EPAGERANGE(DB),
EPAGERANGE(MCE),
+ EPAGERANGE(VC),
+ EPAGERANGE(VC2),
};
static bool in_exception_stack(unsigned long *stack, struct stack_info *info)
@@ -88,7 +92,7 @@ static bool in_exception_stack(unsigned long *stack, struct stack_info *info)
struct pt_regs *regs;
unsigned int k;
- BUILD_BUG_ON(N_EXCEPTION_STACKS != 4);
+ BUILD_BUG_ON(N_EXCEPTION_STACKS != 6);
begin = (unsigned long)__this_cpu_read(cea_exception_stacks);
/*
diff --git a/arch/x86/kernel/sev-es.c b/arch/x86/kernel/sev-es.c
index 720b1b6..fae8145 100644
--- a/arch/x86/kernel/sev-es.c
+++ b/arch/x86/kernel/sev-es.c
@@ -17,6 +17,7 @@
#include <linux/kernel.h>
#include <linux/mm.h>
+#include <asm/cpu_entry_area.h>
#include <asm/sev-es.h>
#include <asm/insn-eval.h>
#include <asm/fpu/internal.h>
@@ -37,10 +38,41 @@ static struct ghcb __initdata *boot_ghcb;
/* #VC handler runtime per-CPU data */
struct sev_es_runtime_data {
struct ghcb ghcb_page;
+
+ /* Physical storage for the per-CPU IST stack of the #VC handler */
+ char ist_stack[EXCEPTION_STKSZ] __aligned(PAGE_SIZE);
+
+ /*
+ * Physical storage for the per-CPU fall-back stack of the #VC handler.
+ * The fall-back stack is used when it is not safe to switch back to the
+ * interrupted stack in the #VC entry code.
+ */
+ char fallback_stack[EXCEPTION_STKSZ] __aligned(PAGE_SIZE);
};
static DEFINE_PER_CPU(struct sev_es_runtime_data*, runtime_data);
+static void __init setup_vc_stacks(int cpu)
+{
+ struct sev_es_runtime_data *data;
+ struct cpu_entry_area *cea;
+ unsigned long vaddr;
+ phys_addr_t pa;
+
+ data = per_cpu(runtime_data, cpu);
+ cea = get_cpu_entry_area(cpu);
+
+ /* Map #VC IST stack */
+ vaddr = CEA_ESTACK_BOT(&cea->estacks, VC);
+ pa = __pa(data->ist_stack);
+ cea_set_pte((void *)vaddr, pa, PAGE_KERNEL);
+
+ /* Map VC fall-back stack */
+ vaddr = CEA_ESTACK_BOT(&cea->estacks, VC2);
+ pa = __pa(data->fallback_stack);
+ cea_set_pte((void *)vaddr, pa, PAGE_KERNEL);
+}
+
/* Needed in vc_early_forward_exception */
void do_early_exception(struct pt_regs *regs, int trapnr);
@@ -249,6 +281,7 @@ void __init sev_es_init_vc_handling(void)
for_each_possible_cpu(cpu) {
alloc_runtime_data(cpu);
init_ghcb(cpu);
+ setup_vc_stacks(cpu);
}
}
^ permalink raw reply related [flat|nested] 264+ messages in thread
* [tip: x86/seves] x86/sev-es: Adjust #VC IST Stack on entering NMI handler
2020-09-07 13:15 ` Joerg Roedel
(?)
@ 2020-09-10 9:22 ` tip-bot2 for Joerg Roedel
-1 siblings, 0 replies; 264+ messages in thread
From: tip-bot2 for Joerg Roedel @ 2020-09-10 9:22 UTC (permalink / raw)
To: linux-tip-commits; +Cc: Joerg Roedel, Borislav Petkov, x86, LKML
The following commit has been merged into the x86/seves branch of tip:
Commit-ID: 315562c9af3d583502b35c4b223a08d95ce69864
Gitweb: https://git.kernel.org/tip/315562c9af3d583502b35c4b223a08d95ce69864
Author: Joerg Roedel <jroedel@suse.de>
AuthorDate: Mon, 07 Sep 2020 15:15:44 +02:00
Committer: Borislav Petkov <bp@suse.de>
CommitterDate: Wed, 09 Sep 2020 11:33:19 +02:00
x86/sev-es: Adjust #VC IST Stack on entering NMI handler
When an NMI hits in the #VC handler entry code before it has switched to
another stack, any subsequent #VC exception in the NMI code-path will
overwrite the interrupted #VC handler's stack.
Make sure this doesn't happen by explicitly adjusting the #VC IST entry
in the NMI handler for the time it can cause #VC exceptions.
[ bp: Touchups, spelling fixes. ]
Signed-off-by: Joerg Roedel <jroedel@suse.de>
Signed-off-by: Borislav Petkov <bp@suse.de>
Link: https://lkml.kernel.org/r/20200907131613.12703-44-joro@8bytes.org
---
arch/x86/include/asm/sev-es.h | 19 ++++++++++++-
arch/x86/kernel/nmi.c | 9 ++++++-
arch/x86/kernel/sev-es.c | 53 ++++++++++++++++++++++++++++++++++-
3 files changed, 81 insertions(+)
diff --git a/arch/x86/include/asm/sev-es.h b/arch/x86/include/asm/sev-es.h
index 9fbeeda..59176e8 100644
--- a/arch/x86/include/asm/sev-es.h
+++ b/arch/x86/include/asm/sev-es.h
@@ -78,4 +78,23 @@ extern void vc_no_ghcb(void);
extern void vc_boot_ghcb(void);
extern bool handle_vc_boot_ghcb(struct pt_regs *regs);
+#ifdef CONFIG_AMD_MEM_ENCRYPT
+extern struct static_key_false sev_es_enable_key;
+extern void __sev_es_ist_enter(struct pt_regs *regs);
+extern void __sev_es_ist_exit(void);
+static __always_inline void sev_es_ist_enter(struct pt_regs *regs)
+{
+ if (static_branch_unlikely(&sev_es_enable_key))
+ __sev_es_ist_enter(regs);
+}
+static __always_inline void sev_es_ist_exit(void)
+{
+ if (static_branch_unlikely(&sev_es_enable_key))
+ __sev_es_ist_exit();
+}
+#else
+static inline void sev_es_ist_enter(struct pt_regs *regs) { }
+static inline void sev_es_ist_exit(void) { }
+#endif
+
#endif
diff --git a/arch/x86/kernel/nmi.c b/arch/x86/kernel/nmi.c
index 4fc9954..4c89c4d 100644
--- a/arch/x86/kernel/nmi.c
+++ b/arch/x86/kernel/nmi.c
@@ -33,6 +33,7 @@
#include <asm/reboot.h>
#include <asm/cache.h>
#include <asm/nospec-branch.h>
+#include <asm/sev-es.h>
#define CREATE_TRACE_POINTS
#include <trace/events/nmi.h>
@@ -488,6 +489,12 @@ DEFINE_IDTENTRY_RAW(exc_nmi)
this_cpu_write(nmi_cr2, read_cr2());
nmi_restart:
+ /*
+ * Needs to happen before DR7 is accessed, because the hypervisor can
+ * intercept DR7 reads/writes, turning those into #VC exceptions.
+ */
+ sev_es_ist_enter(regs);
+
this_cpu_write(nmi_dr7, local_db_save());
irq_state = idtentry_enter_nmi(regs);
@@ -501,6 +508,8 @@ nmi_restart:
local_db_restore(this_cpu_read(nmi_dr7));
+ sev_es_ist_exit();
+
if (unlikely(this_cpu_read(nmi_cr2) != read_cr2()))
write_cr2(this_cpu_read(nmi_cr2));
if (this_cpu_dec_return(nmi_state))
diff --git a/arch/x86/kernel/sev-es.c b/arch/x86/kernel/sev-es.c
index fae8145..39ebb2d 100644
--- a/arch/x86/kernel/sev-es.c
+++ b/arch/x86/kernel/sev-es.c
@@ -51,6 +51,7 @@ struct sev_es_runtime_data {
};
static DEFINE_PER_CPU(struct sev_es_runtime_data*, runtime_data);
+DEFINE_STATIC_KEY_FALSE(sev_es_enable_key);
static void __init setup_vc_stacks(int cpu)
{
@@ -73,6 +74,55 @@ static void __init setup_vc_stacks(int cpu)
cea_set_pte((void *)vaddr, pa, PAGE_KERNEL);
}
+static __always_inline bool on_vc_stack(unsigned long sp)
+{
+ return ((sp >= __this_cpu_ist_bottom_va(VC)) && (sp < __this_cpu_ist_top_va(VC)));
+}
+
+/*
+ * This function handles the case when an NMI is raised in the #VC exception
+ * handler entry code. In this case, the IST entry for #VC must be adjusted, so
+ * that any subsequent #VC exception will not overwrite the stack contents of the
+ * interrupted #VC handler.
+ *
+ * The IST entry is adjusted unconditionally so that it can be also be
+ * unconditionally adjusted back in sev_es_ist_exit(). Otherwise a nested
+ * sev_es_ist_exit() call may adjust back the IST entry too early.
+ */
+void noinstr __sev_es_ist_enter(struct pt_regs *regs)
+{
+ unsigned long old_ist, new_ist;
+
+ /* Read old IST entry */
+ old_ist = __this_cpu_read(cpu_tss_rw.x86_tss.ist[IST_INDEX_VC]);
+
+ /* Make room on the IST stack */
+ if (on_vc_stack(regs->sp))
+ new_ist = ALIGN_DOWN(regs->sp, 8) - sizeof(old_ist);
+ else
+ new_ist = old_ist - sizeof(old_ist);
+
+ /* Store old IST entry */
+ *(unsigned long *)new_ist = old_ist;
+
+ /* Set new IST entry */
+ this_cpu_write(cpu_tss_rw.x86_tss.ist[IST_INDEX_VC], new_ist);
+}
+
+void noinstr __sev_es_ist_exit(void)
+{
+ unsigned long ist;
+
+ /* Read IST entry */
+ ist = __this_cpu_read(cpu_tss_rw.x86_tss.ist[IST_INDEX_VC]);
+
+ if (WARN_ON(ist == __this_cpu_ist_top_va(VC)))
+ return;
+
+ /* Read back old IST entry and write it to the TSS */
+ this_cpu_write(cpu_tss_rw.x86_tss.ist[IST_INDEX_VC], *(unsigned long *)ist);
+}
+
/* Needed in vc_early_forward_exception */
void do_early_exception(struct pt_regs *regs, int trapnr);
@@ -277,6 +327,9 @@ void __init sev_es_init_vc_handling(void)
if (!sev_es_active())
return;
+ /* Enable SEV-ES special handling */
+ static_branch_enable(&sev_es_enable_key);
+
/* Initialize per-cpu GHCB pages */
for_each_possible_cpu(cpu) {
alloc_runtime_data(cpu);
^ permalink raw reply related [flat|nested] 264+ messages in thread
* [tip: x86/seves] x86/sev-es: Setup GHCB-based boot #VC handler
2020-09-08 12:38 ` Joerg Roedel
(?)
@ 2020-09-10 9:22 ` tip-bot2 for Joerg Roedel
-1 siblings, 0 replies; 264+ messages in thread
From: tip-bot2 for Joerg Roedel @ 2020-09-10 9:22 UTC (permalink / raw)
To: linux-tip-commits; +Cc: Joerg Roedel, Borislav Petkov, x86, LKML
The following commit has been merged into the x86/seves branch of tip:
Commit-ID: 1aa9aa8ee517e0443b06e816a4fd2d15f2113615
Gitweb: https://git.kernel.org/tip/1aa9aa8ee517e0443b06e816a4fd2d15f2113615
Author: Joerg Roedel <jroedel@suse.de>
AuthorDate: Tue, 08 Sep 2020 14:38:16 +02:00
Committer: Borislav Petkov <bp@suse.de>
CommitterDate: Wed, 09 Sep 2020 11:32:27 +02:00
x86/sev-es: Setup GHCB-based boot #VC handler
Add the infrastructure to handle #VC exceptions when the kernel runs on
virtual addresses and has mapped a GHCB. This handler will be used until
the runtime #VC handler takes over.
Since the handler runs very early, disable instrumentation for sev-es.c.
[ bp: Make vc_ghcb_invalidate() __always_inline so that it can be
inlined in noinstr functions like __sev_es_nmi_complete(). ]
Signed-off-by: Joerg Roedel <jroedel@suse.de>
Signed-off-by: Borislav Petkov <bp@suse.de>
Link: https://lkml.kernel.org/r/20200908123816.GB3764@8bytes.org
---
arch/x86/include/asm/realmode.h | 3 +-
arch/x86/include/asm/segment.h | 2 +-
arch/x86/include/asm/sev-es.h | 2 +-
arch/x86/kernel/Makefile | 2 +-
arch/x86/kernel/head64.c | 8 ++-
arch/x86/kernel/head_64.S | 36 ++++++++++-
arch/x86/kernel/sev-es-shared.c | 14 ++--
arch/x86/kernel/sev-es.c | 116 +++++++++++++++++++++++++++++++-
arch/x86/mm/extable.c | 1 +-
9 files changed, 176 insertions(+), 8 deletions(-)
diff --git a/arch/x86/include/asm/realmode.h b/arch/x86/include/asm/realmode.h
index b35030e..96118fb 100644
--- a/arch/x86/include/asm/realmode.h
+++ b/arch/x86/include/asm/realmode.h
@@ -57,6 +57,9 @@ extern unsigned char real_mode_blob_end[];
extern unsigned long initial_code;
extern unsigned long initial_gs;
extern unsigned long initial_stack;
+#ifdef CONFIG_AMD_MEM_ENCRYPT
+extern unsigned long initial_vc_handler;
+#endif
extern unsigned char real_mode_blob[];
extern unsigned char real_mode_relocs[];
diff --git a/arch/x86/include/asm/segment.h b/arch/x86/include/asm/segment.h
index 9646c30..4e8dec3 100644
--- a/arch/x86/include/asm/segment.h
+++ b/arch/x86/include/asm/segment.h
@@ -230,7 +230,7 @@
#define NUM_EXCEPTION_VECTORS 32
/* Bitmask of exception vectors which push an error code on the stack: */
-#define EXCEPTION_ERRCODE_MASK 0x00027d00
+#define EXCEPTION_ERRCODE_MASK 0x20027d00
#define GDT_SIZE (GDT_ENTRIES*8)
#define GDT_ENTRY_TLS_ENTRIES 3
diff --git a/arch/x86/include/asm/sev-es.h b/arch/x86/include/asm/sev-es.h
index 7175d43..9fbeeda 100644
--- a/arch/x86/include/asm/sev-es.h
+++ b/arch/x86/include/asm/sev-es.h
@@ -75,5 +75,7 @@ static inline u64 lower_bits(u64 val, unsigned int bits)
/* Early IDT entry points for #VC handler */
extern void vc_no_ghcb(void);
+extern void vc_boot_ghcb(void);
+extern bool handle_vc_boot_ghcb(struct pt_regs *regs);
#endif
diff --git a/arch/x86/kernel/Makefile b/arch/x86/kernel/Makefile
index 23bc0f8..4a33272 100644
--- a/arch/x86/kernel/Makefile
+++ b/arch/x86/kernel/Makefile
@@ -20,6 +20,7 @@ CFLAGS_REMOVE_kvmclock.o = -pg
CFLAGS_REMOVE_ftrace.o = -pg
CFLAGS_REMOVE_early_printk.o = -pg
CFLAGS_REMOVE_head64.o = -pg
+CFLAGS_REMOVE_sev-es.o = -pg
endif
KASAN_SANITIZE_head$(BITS).o := n
@@ -27,6 +28,7 @@ KASAN_SANITIZE_dumpstack.o := n
KASAN_SANITIZE_dumpstack_$(BITS).o := n
KASAN_SANITIZE_stacktrace.o := n
KASAN_SANITIZE_paravirt.o := n
+KASAN_SANITIZE_sev-es.o := n
# With some compiler versions the generated code results in boot hangs, caused
# by several compilation units. To be safe, disable all instrumentation.
diff --git a/arch/x86/kernel/head64.c b/arch/x86/kernel/head64.c
index fc55cc9..4199f25 100644
--- a/arch/x86/kernel/head64.c
+++ b/arch/x86/kernel/head64.c
@@ -406,6 +406,10 @@ void __init do_early_exception(struct pt_regs *regs, int trapnr)
early_make_pgtable(native_read_cr2()))
return;
+ if (IS_ENABLED(CONFIG_AMD_MEM_ENCRYPT) &&
+ trapnr == X86_TRAP_VC && handle_vc_boot_ghcb(regs))
+ return;
+
early_fixup_exception(regs, trapnr);
}
@@ -575,6 +579,10 @@ static void startup_64_load_idt(unsigned long physbase)
/* This is used when running on kernel addresses */
void early_setup_idt(void)
{
+ /* VMM Communication Exception */
+ if (IS_ENABLED(CONFIG_AMD_MEM_ENCRYPT))
+ set_bringup_idt_handler(bringup_idt_table, X86_TRAP_VC, vc_boot_ghcb);
+
bringup_idt_descr.address = (unsigned long)bringup_idt_table;
native_load_idt(&bringup_idt_descr);
}
diff --git a/arch/x86/kernel/head_64.S b/arch/x86/kernel/head_64.S
index 6e68bca..1a71d0d 100644
--- a/arch/x86/kernel/head_64.S
+++ b/arch/x86/kernel/head_64.S
@@ -281,11 +281,47 @@ SYM_CODE_START(start_cpu0)
SYM_CODE_END(start_cpu0)
#endif
+#ifdef CONFIG_AMD_MEM_ENCRYPT
+/*
+ * VC Exception handler used during early boot when running on kernel
+ * addresses, but before the switch to the idt_table can be made.
+ * The early_idt_handler_array can't be used here because it calls into a lot
+ * of __init code and this handler is also used during CPU offlining/onlining.
+ * Therefore this handler ends up in the .text section so that it stays around
+ * when .init.text is freed.
+ */
+SYM_CODE_START_NOALIGN(vc_boot_ghcb)
+ UNWIND_HINT_IRET_REGS offset=8
+
+ /* Build pt_regs */
+ PUSH_AND_CLEAR_REGS
+
+ /* Call C handler */
+ movq %rsp, %rdi
+ movq ORIG_RAX(%rsp), %rsi
+ movq initial_vc_handler(%rip), %rax
+ ANNOTATE_RETPOLINE_SAFE
+ call *%rax
+
+ /* Unwind pt_regs */
+ POP_REGS
+
+ /* Remove Error Code */
+ addq $8, %rsp
+
+ /* Pure iret required here - don't use INTERRUPT_RETURN */
+ iretq
+SYM_CODE_END(vc_boot_ghcb)
+#endif
+
/* Both SMP bootup and ACPI suspend change these variables */
__REFDATA
.balign 8
SYM_DATA(initial_code, .quad x86_64_start_kernel)
SYM_DATA(initial_gs, .quad INIT_PER_CPU_VAR(fixed_percpu_data))
+#ifdef CONFIG_AMD_MEM_ENCRYPT
+SYM_DATA(initial_vc_handler, .quad handle_vc_boot_ghcb)
+#endif
/*
* The SIZEOF_PTREGS gap is a convention which helps the in-kernel unwinder
diff --git a/arch/x86/kernel/sev-es-shared.c b/arch/x86/kernel/sev-es-shared.c
index 1861927..53e3de0 100644
--- a/arch/x86/kernel/sev-es-shared.c
+++ b/arch/x86/kernel/sev-es-shared.c
@@ -9,7 +9,7 @@
* and is included directly into both code-bases.
*/
-static void __maybe_unused sev_es_terminate(unsigned int reason)
+static void sev_es_terminate(unsigned int reason)
{
u64 val = GHCB_SEV_TERMINATE;
@@ -27,7 +27,7 @@ static void __maybe_unused sev_es_terminate(unsigned int reason)
asm volatile("hlt\n" : : : "memory");
}
-static bool __maybe_unused sev_es_negotiate_protocol(void)
+static bool sev_es_negotiate_protocol(void)
{
u64 val;
@@ -46,7 +46,7 @@ static bool __maybe_unused sev_es_negotiate_protocol(void)
return true;
}
-static void __maybe_unused vc_ghcb_invalidate(struct ghcb *ghcb)
+static __always_inline void vc_ghcb_invalidate(struct ghcb *ghcb)
{
memset(ghcb->save.valid_bitmap, 0, sizeof(ghcb->save.valid_bitmap));
}
@@ -58,9 +58,9 @@ static bool vc_decoding_needed(unsigned long exit_code)
exit_code <= SVM_EXIT_LAST_EXCP);
}
-static enum es_result __maybe_unused vc_init_em_ctxt(struct es_em_ctxt *ctxt,
- struct pt_regs *regs,
- unsigned long exit_code)
+static enum es_result vc_init_em_ctxt(struct es_em_ctxt *ctxt,
+ struct pt_regs *regs,
+ unsigned long exit_code)
{
enum es_result ret = ES_OK;
@@ -73,7 +73,7 @@ static enum es_result __maybe_unused vc_init_em_ctxt(struct es_em_ctxt *ctxt,
return ret;
}
-static void __maybe_unused vc_finish_insn(struct es_em_ctxt *ctxt)
+static void vc_finish_insn(struct es_em_ctxt *ctxt)
{
ctxt->regs->ip += ctxt->insn.length;
}
diff --git a/arch/x86/kernel/sev-es.c b/arch/x86/kernel/sev-es.c
index 4b1d217..213a427 100644
--- a/arch/x86/kernel/sev-es.c
+++ b/arch/x86/kernel/sev-es.c
@@ -7,7 +7,9 @@
* Author: Joerg Roedel <jroedel@suse.de>
*/
+#include <linux/sched/debug.h> /* For show_regs() */
#include <linux/kernel.h>
+#include <linux/printk.h>
#include <linux/mm.h>
#include <asm/sev-es.h>
@@ -18,6 +20,18 @@
#include <asm/trapnr.h>
#include <asm/svm.h>
+/* For early boot hypervisor communication in SEV-ES enabled guests */
+static struct ghcb boot_ghcb_page __bss_decrypted __aligned(PAGE_SIZE);
+
+/*
+ * Needs to be in the .data section because we need it NULL before bss is
+ * cleared
+ */
+static struct ghcb __initdata *boot_ghcb;
+
+/* Needed in vc_early_forward_exception */
+void do_early_exception(struct pt_regs *regs, int trapnr);
+
static inline u64 sev_es_rd_ghcb_msr(void)
{
return __rdmsr(MSR_AMD64_SEV_ES_GHCB);
@@ -161,3 +175,105 @@ fault:
/* Include code shared with pre-decompression boot stage */
#include "sev-es-shared.c"
+
+/*
+ * This function runs on the first #VC exception after the kernel
+ * switched to virtual addresses.
+ */
+static bool __init sev_es_setup_ghcb(void)
+{
+ /* First make sure the hypervisor talks a supported protocol. */
+ if (!sev_es_negotiate_protocol())
+ return false;
+
+ /*
+ * Clear the boot_ghcb. The first exception comes in before the bss
+ * section is cleared.
+ */
+ memset(&boot_ghcb_page, 0, PAGE_SIZE);
+
+ /* Alright - Make the boot-ghcb public */
+ boot_ghcb = &boot_ghcb_page;
+
+ return true;
+}
+
+static void __init vc_early_forward_exception(struct es_em_ctxt *ctxt)
+{
+ int trapnr = ctxt->fi.vector;
+
+ if (trapnr == X86_TRAP_PF)
+ native_write_cr2(ctxt->fi.cr2);
+
+ ctxt->regs->orig_ax = ctxt->fi.error_code;
+ do_early_exception(ctxt->regs, trapnr);
+}
+
+static enum es_result vc_handle_exitcode(struct es_em_ctxt *ctxt,
+ struct ghcb *ghcb,
+ unsigned long exit_code)
+{
+ enum es_result result;
+
+ switch (exit_code) {
+ default:
+ /*
+ * Unexpected #VC exception
+ */
+ result = ES_UNSUPPORTED;
+ }
+
+ return result;
+}
+
+bool __init handle_vc_boot_ghcb(struct pt_regs *regs)
+{
+ unsigned long exit_code = regs->orig_ax;
+ struct es_em_ctxt ctxt;
+ enum es_result result;
+
+ /* Do initial setup or terminate the guest */
+ if (unlikely(boot_ghcb == NULL && !sev_es_setup_ghcb()))
+ sev_es_terminate(GHCB_SEV_ES_REASON_GENERAL_REQUEST);
+
+ vc_ghcb_invalidate(boot_ghcb);
+
+ result = vc_init_em_ctxt(&ctxt, regs, exit_code);
+ if (result == ES_OK)
+ result = vc_handle_exitcode(&ctxt, boot_ghcb, exit_code);
+
+ /* Done - now check the result */
+ switch (result) {
+ case ES_OK:
+ vc_finish_insn(&ctxt);
+ break;
+ case ES_UNSUPPORTED:
+ early_printk("PANIC: Unsupported exit-code 0x%02lx in early #VC exception (IP: 0x%lx)\n",
+ exit_code, regs->ip);
+ goto fail;
+ case ES_VMM_ERROR:
+ early_printk("PANIC: Failure in communication with VMM (exit-code 0x%02lx IP: 0x%lx)\n",
+ exit_code, regs->ip);
+ goto fail;
+ case ES_DECODE_FAILED:
+ early_printk("PANIC: Failed to decode instruction (exit-code 0x%02lx IP: 0x%lx)\n",
+ exit_code, regs->ip);
+ goto fail;
+ case ES_EXCEPTION:
+ vc_early_forward_exception(&ctxt);
+ break;
+ case ES_RETRY:
+ /* Nothing to do */
+ break;
+ default:
+ BUG();
+ }
+
+ return true;
+
+fail:
+ show_regs(regs);
+
+ while (true)
+ halt();
+}
diff --git a/arch/x86/mm/extable.c b/arch/x86/mm/extable.c
index 1d6cb07..3966749 100644
--- a/arch/x86/mm/extable.c
+++ b/arch/x86/mm/extable.c
@@ -5,6 +5,7 @@
#include <xen/xen.h>
#include <asm/fpu/internal.h>
+#include <asm/sev-es.h>
#include <asm/traps.h>
#include <asm/kdebug.h>
^ permalink raw reply related [flat|nested] 264+ messages in thread
* [tip: x86/seves] x86/sev-es: Setup per-CPU GHCBs for the runtime handler
2020-09-07 13:15 ` Joerg Roedel
(?)
(?)
@ 2020-09-10 9:22 ` tip-bot2 for Tom Lendacky
-1 siblings, 0 replies; 264+ messages in thread
From: tip-bot2 for Tom Lendacky @ 2020-09-10 9:22 UTC (permalink / raw)
To: linux-tip-commits; +Cc: Tom Lendacky, Joerg Roedel, Borislav Petkov, x86, LKML
The following commit has been merged into the x86/seves branch of tip:
Commit-ID: 885689e47dfa1499b756a07237eb645234d93cf9
Gitweb: https://git.kernel.org/tip/885689e47dfa1499b756a07237eb645234d93cf9
Author: Tom Lendacky <thomas.lendacky@amd.com>
AuthorDate: Mon, 07 Sep 2020 15:15:42 +02:00
Committer: Borislav Petkov <bp@suse.de>
CommitterDate: Wed, 09 Sep 2020 11:33:19 +02:00
x86/sev-es: Setup per-CPU GHCBs for the runtime handler
The runtime handler needs one GHCB per-CPU. Set them up and map them
unencrypted.
[ bp: Touchups and simplification. ]
Signed-off-by: Tom Lendacky <thomas.lendacky@amd.com>
Signed-off-by: Joerg Roedel <jroedel@suse.de>
Signed-off-by: Borislav Petkov <bp@suse.de>
Link: https://lkml.kernel.org/r/20200907131613.12703-42-joro@8bytes.org
---
arch/x86/include/asm/mem_encrypt.h | 2 +-
arch/x86/kernel/sev-es.c | 56 ++++++++++++++++++++++++++++-
arch/x86/kernel/traps.c | 3 ++-
3 files changed, 60 insertions(+), 1 deletion(-)
diff --git a/arch/x86/include/asm/mem_encrypt.h b/arch/x86/include/asm/mem_encrypt.h
index 4e72b73..c9f5df0 100644
--- a/arch/x86/include/asm/mem_encrypt.h
+++ b/arch/x86/include/asm/mem_encrypt.h
@@ -49,6 +49,7 @@ void __init mem_encrypt_free_decrypted_mem(void);
/* Architecture __weak replacement functions */
void __init mem_encrypt_init(void);
+void __init sev_es_init_vc_handling(void);
bool sme_active(void);
bool sev_active(void);
bool sev_es_active(void);
@@ -72,6 +73,7 @@ static inline void __init sme_early_init(void) { }
static inline void __init sme_encrypt_kernel(struct boot_params *bp) { }
static inline void __init sme_enable(struct boot_params *bp) { }
+static inline void sev_es_init_vc_handling(void) { }
static inline bool sme_active(void) { return false; }
static inline bool sev_active(void) { return false; }
static inline bool sev_es_active(void) { return false; }
diff --git a/arch/x86/kernel/sev-es.c b/arch/x86/kernel/sev-es.c
index 213a427..720b1b6 100644
--- a/arch/x86/kernel/sev-es.c
+++ b/arch/x86/kernel/sev-es.c
@@ -8,8 +8,13 @@
*/
#include <linux/sched/debug.h> /* For show_regs() */
-#include <linux/kernel.h>
+#include <linux/percpu-defs.h>
+#include <linux/mem_encrypt.h>
#include <linux/printk.h>
+#include <linux/mm_types.h>
+#include <linux/set_memory.h>
+#include <linux/memblock.h>
+#include <linux/kernel.h>
#include <linux/mm.h>
#include <asm/sev-es.h>
@@ -29,6 +34,13 @@ static struct ghcb boot_ghcb_page __bss_decrypted __aligned(PAGE_SIZE);
*/
static struct ghcb __initdata *boot_ghcb;
+/* #VC handler runtime per-CPU data */
+struct sev_es_runtime_data {
+ struct ghcb ghcb_page;
+};
+
+static DEFINE_PER_CPU(struct sev_es_runtime_data*, runtime_data);
+
/* Needed in vc_early_forward_exception */
void do_early_exception(struct pt_regs *regs, int trapnr);
@@ -198,6 +210,48 @@ static bool __init sev_es_setup_ghcb(void)
return true;
}
+static void __init alloc_runtime_data(int cpu)
+{
+ struct sev_es_runtime_data *data;
+
+ data = memblock_alloc(sizeof(*data), PAGE_SIZE);
+ if (!data)
+ panic("Can't allocate SEV-ES runtime data");
+
+ per_cpu(runtime_data, cpu) = data;
+}
+
+static void __init init_ghcb(int cpu)
+{
+ struct sev_es_runtime_data *data;
+ int err;
+
+ data = per_cpu(runtime_data, cpu);
+
+ err = early_set_memory_decrypted((unsigned long)&data->ghcb_page,
+ sizeof(data->ghcb_page));
+ if (err)
+ panic("Can't map GHCBs unencrypted");
+
+ memset(&data->ghcb_page, 0, sizeof(data->ghcb_page));
+}
+
+void __init sev_es_init_vc_handling(void)
+{
+ int cpu;
+
+ BUILD_BUG_ON(offsetof(struct sev_es_runtime_data, ghcb_page) % PAGE_SIZE);
+
+ if (!sev_es_active())
+ return;
+
+ /* Initialize per-cpu GHCB pages */
+ for_each_possible_cpu(cpu) {
+ alloc_runtime_data(cpu);
+ init_ghcb(cpu);
+ }
+}
+
static void __init vc_early_forward_exception(struct es_em_ctxt *ctxt)
{
int trapnr = ctxt->fi.vector;
diff --git a/arch/x86/kernel/traps.c b/arch/x86/kernel/traps.c
index 81a2fb7..2dc32b0 100644
--- a/arch/x86/kernel/traps.c
+++ b/arch/x86/kernel/traps.c
@@ -1074,6 +1074,9 @@ void __init trap_init(void)
/* Init cpu_entry_area before IST entries are set up */
setup_cpu_entry_areas();
+ /* Init GHCB memory pages when running as an SEV-ES guest */
+ sev_es_init_vc_handling();
+
idt_setup_traps();
/*
^ permalink raw reply related [flat|nested] 264+ messages in thread
* [tip: x86/seves] x86/sev-es: Print SEV-ES info into the kernel log
2020-09-07 13:15 ` Joerg Roedel
(?)
@ 2020-09-10 9:22 ` tip-bot2 for Joerg Roedel
-1 siblings, 0 replies; 264+ messages in thread
From: tip-bot2 for Joerg Roedel @ 2020-09-10 9:22 UTC (permalink / raw)
To: linux-tip-commits; +Cc: Joerg Roedel, Borislav Petkov, Kees Cook, x86, LKML
The following commit has been merged into the x86/seves branch of tip:
Commit-ID: c685eb0c12b4d4816d22ee734e91f4005b152fcd
Gitweb: https://git.kernel.org/tip/c685eb0c12b4d4816d22ee734e91f4005b152fcd
Author: Joerg Roedel <jroedel@suse.de>
AuthorDate: Mon, 07 Sep 2020 15:15:38 +02:00
Committer: Borislav Petkov <bp@suse.de>
CommitterDate: Tue, 08 Sep 2020 00:38:01 +02:00
x86/sev-es: Print SEV-ES info into the kernel log
Refactor the message printed to the kernel log which indicates whether
SEV or SME, etc is active. This will scale better in the future when
more memory encryption features might be added. Also add SEV-ES to the
list of features.
[ bp: Massage. ]
Signed-off-by: Joerg Roedel <jroedel@suse.de>
Signed-off-by: Borislav Petkov <bp@suse.de>
Reviewed-by: Kees Cook <keescook@chromium.org>
Link: https://lkml.kernel.org/r/20200907131613.12703-38-joro@8bytes.org
---
arch/x86/mm/mem_encrypt.c | 29 ++++++++++++++++++++++++++---
1 file changed, 26 insertions(+), 3 deletions(-)
diff --git a/arch/x86/mm/mem_encrypt.c b/arch/x86/mm/mem_encrypt.c
index a38f556..ebb7edc 100644
--- a/arch/x86/mm/mem_encrypt.c
+++ b/arch/x86/mm/mem_encrypt.c
@@ -407,6 +407,31 @@ void __init mem_encrypt_free_decrypted_mem(void)
free_init_pages("unused decrypted", vaddr, vaddr_end);
}
+static void print_mem_encrypt_feature_info(void)
+{
+ pr_info("AMD Memory Encryption Features active:");
+
+ /* Secure Memory Encryption */
+ if (sme_active()) {
+ /*
+ * SME is mutually exclusive with any of the SEV
+ * features below.
+ */
+ pr_cont(" SME\n");
+ return;
+ }
+
+ /* Secure Encrypted Virtualization */
+ if (sev_active())
+ pr_cont(" SEV");
+
+ /* Encrypted Register State */
+ if (sev_es_active())
+ pr_cont(" SEV-ES");
+
+ pr_cont("\n");
+}
+
/* Architecture __weak replacement functions */
void __init mem_encrypt_init(void)
{
@@ -422,8 +447,6 @@ void __init mem_encrypt_init(void)
if (sev_active())
static_branch_enable(&sev_enable_key);
- pr_info("AMD %s active\n",
- sev_active() ? "Secure Encrypted Virtualization (SEV)"
- : "Secure Memory Encryption (SME)");
+ print_mem_encrypt_feature_info();
}
^ permalink raw reply related [flat|nested] 264+ messages in thread
* [tip: x86/seves] x86/sev-es: Setup an early #VC handler
2020-09-08 12:35 ` Joerg Roedel
(?)
@ 2020-09-10 9:22 ` tip-bot2 for Joerg Roedel
-1 siblings, 0 replies; 264+ messages in thread
From: tip-bot2 for Joerg Roedel @ 2020-09-10 9:22 UTC (permalink / raw)
To: linux-tip-commits
Cc: Joerg Roedel, Borislav Petkov, kernel test robot, x86, LKML
The following commit has been merged into the x86/seves branch of tip:
Commit-ID: 74d8d9d531b4cc945a9f75aa2fc21d99ca5a9fe3
Gitweb: https://git.kernel.org/tip/74d8d9d531b4cc945a9f75aa2fc21d99ca5a9fe3
Author: Joerg Roedel <jroedel@suse.de>
AuthorDate: Tue, 08 Sep 2020 14:35:17 +02:00
Committer: Borislav Petkov <bp@suse.de>
CommitterDate: Wed, 09 Sep 2020 10:45:24 +02:00
x86/sev-es: Setup an early #VC handler
Setup an early handler for #VC exceptions. There is no GHCB mapped
yet, so just re-use the vc_no_ghcb_handler(). It can only handle
CPUID exit-codes, but that should be enough to get the kernel through
verify_cpu() and __startup_64() until it runs on virtual addresses.
Signed-off-by: Joerg Roedel <jroedel@suse.de>
Signed-off-by: Borislav Petkov <bp@suse.de>
[ boot failure Error: kernel_ident_mapping_init() failed. ]
Reported-by: kernel test robot <lkp@intel.com>
Link: https://lkml.kernel.org/r/20200908123517.GA3764@8bytes.org
---
arch/x86/include/asm/sev-es.h | 3 +++
arch/x86/kernel/head64.c | 25 ++++++++++++++++++++++++-
arch/x86/kernel/head_64.S | 30 ++++++++++++++++++++++++++++++
3 files changed, 57 insertions(+), 1 deletion(-)
diff --git a/arch/x86/include/asm/sev-es.h b/arch/x86/include/asm/sev-es.h
index 6dc5244..7175d43 100644
--- a/arch/x86/include/asm/sev-es.h
+++ b/arch/x86/include/asm/sev-es.h
@@ -73,4 +73,7 @@ static inline u64 lower_bits(u64 val, unsigned int bits)
return (val & mask);
}
+/* Early IDT entry points for #VC handler */
+extern void vc_no_ghcb(void);
+
#endif
diff --git a/arch/x86/kernel/head64.c b/arch/x86/kernel/head64.c
index 4282dac..fc55cc9 100644
--- a/arch/x86/kernel/head64.c
+++ b/arch/x86/kernel/head64.c
@@ -40,6 +40,7 @@
#include <asm/desc.h>
#include <asm/extable.h>
#include <asm/trapnr.h>
+#include <asm/sev-es.h>
/*
* Manage page tables very early on.
@@ -540,12 +541,34 @@ static struct desc_ptr bringup_idt_descr = {
.address = 0, /* Set at runtime */
};
+static void set_bringup_idt_handler(gate_desc *idt, int n, void *handler)
+{
+#ifdef CONFIG_AMD_MEM_ENCRYPT
+ struct idt_data data;
+ gate_desc desc;
+
+ init_idt_data(&data, n, handler);
+ idt_init_desc(&desc, &data);
+ native_write_idt_entry(idt, n, &desc);
+#endif
+}
+
/* This runs while still in the direct mapping */
static void startup_64_load_idt(unsigned long physbase)
{
struct desc_ptr *desc = fixup_pointer(&bringup_idt_descr, physbase);
+ gate_desc *idt = fixup_pointer(bringup_idt_table, physbase);
+
+
+ if (IS_ENABLED(CONFIG_AMD_MEM_ENCRYPT)) {
+ void *handler;
+
+ /* VMM Communication Exception */
+ handler = fixup_pointer(vc_no_ghcb, physbase);
+ set_bringup_idt_handler(idt, X86_TRAP_VC, handler);
+ }
- desc->address = (unsigned long)fixup_pointer(bringup_idt_table, physbase);
+ desc->address = (unsigned long)idt;
native_load_idt(desc);
}
diff --git a/arch/x86/kernel/head_64.S b/arch/x86/kernel/head_64.S
index 3b40ec4..6e68bca 100644
--- a/arch/x86/kernel/head_64.S
+++ b/arch/x86/kernel/head_64.S
@@ -348,6 +348,36 @@ SYM_CODE_START_LOCAL(early_idt_handler_common)
jmp restore_regs_and_return_to_kernel
SYM_CODE_END(early_idt_handler_common)
+#ifdef CONFIG_AMD_MEM_ENCRYPT
+/*
+ * VC Exception handler used during very early boot. The
+ * early_idt_handler_array can't be used because it returns via the
+ * paravirtualized INTERRUPT_RETURN and pv-ops don't work that early.
+ *
+ * This handler will end up in the .init.text section and not be
+ * available to boot secondary CPUs.
+ */
+SYM_CODE_START_NOALIGN(vc_no_ghcb)
+ UNWIND_HINT_IRET_REGS offset=8
+
+ /* Build pt_regs */
+ PUSH_AND_CLEAR_REGS
+
+ /* Call C handler */
+ movq %rsp, %rdi
+ movq ORIG_RAX(%rsp), %rsi
+ call do_vc_no_ghcb
+
+ /* Unwind pt_regs */
+ POP_REGS
+
+ /* Remove Error Code */
+ addq $8, %rsp
+
+ /* Pure iret required here - don't use INTERRUPT_RETURN */
+ iretq
+SYM_CODE_END(vc_no_ghcb)
+#endif
#define SYM_DATA_START_PAGE_ALIGNED(name) \
SYM_START(name, SYM_L_GLOBAL, .balign PAGE_SIZE)
^ permalink raw reply related [flat|nested] 264+ messages in thread
* [tip: x86/seves] x86/sev-es: Compile early handler code into kernel image
2020-09-07 13:15 ` Joerg Roedel
(?)
@ 2020-09-10 9:22 ` tip-bot2 for Joerg Roedel
-1 siblings, 0 replies; 264+ messages in thread
From: tip-bot2 for Joerg Roedel @ 2020-09-10 9:22 UTC (permalink / raw)
To: linux-tip-commits; +Cc: Joerg Roedel, Borislav Petkov, x86, LKML
The following commit has been merged into the x86/seves branch of tip:
Commit-ID: f980f9c31a923e9040dee0bc679a5f5b09e61f40
Gitweb: https://git.kernel.org/tip/f980f9c31a923e9040dee0bc679a5f5b09e61f40
Author: Joerg Roedel <jroedel@suse.de>
AuthorDate: Mon, 07 Sep 2020 15:15:39 +02:00
Committer: Borislav Petkov <bp@suse.de>
CommitterDate: Wed, 09 Sep 2020 10:44:46 +02:00
x86/sev-es: Compile early handler code into kernel image
Setup sev-es.c and include the code from the pre-decompression stage
to also build it into the image of the running kernel. Temporarily add
__maybe_unused annotations to avoid build warnings until the functions
get used.
[ bp: Use the non-tracing rd/wrmsr variants because:
vmlinux.o: warning: objtool: __sev_es_nmi_complete()+0x11f: \
call to do_trace_write_msr() leaves .noinstr.text section
as __sev_es_nmi_complete() is noinstr due to being called from the
NMI handler exc_nmi(). ]
Signed-off-by: Joerg Roedel <jroedel@suse.de>
Signed-off-by: Borislav Petkov <bp@suse.de>
Link: https://lkml.kernel.org/r/20200907131613.12703-39-joro@8bytes.org
---
arch/x86/kernel/Makefile | 1 +-
arch/x86/kernel/sev-es-shared.c | 21 ++--
arch/x86/kernel/sev-es.c | 163 +++++++++++++++++++++++++++++++-
3 files changed, 175 insertions(+), 10 deletions(-)
create mode 100644 arch/x86/kernel/sev-es.c
diff --git a/arch/x86/kernel/Makefile b/arch/x86/kernel/Makefile
index e77261d..23bc0f8 100644
--- a/arch/x86/kernel/Makefile
+++ b/arch/x86/kernel/Makefile
@@ -145,6 +145,7 @@ obj-$(CONFIG_UNWINDER_ORC) += unwind_orc.o
obj-$(CONFIG_UNWINDER_FRAME_POINTER) += unwind_frame.o
obj-$(CONFIG_UNWINDER_GUESS) += unwind_guess.o
+obj-$(CONFIG_AMD_MEM_ENCRYPT) += sev-es.o
###
# 64 bit specific files
ifeq ($(CONFIG_X86_64),y)
diff --git a/arch/x86/kernel/sev-es-shared.c b/arch/x86/kernel/sev-es-shared.c
index a6b4191..1861927 100644
--- a/arch/x86/kernel/sev-es-shared.c
+++ b/arch/x86/kernel/sev-es-shared.c
@@ -9,7 +9,7 @@
* and is included directly into both code-bases.
*/
-static void sev_es_terminate(unsigned int reason)
+static void __maybe_unused sev_es_terminate(unsigned int reason)
{
u64 val = GHCB_SEV_TERMINATE;
@@ -27,7 +27,7 @@ static void sev_es_terminate(unsigned int reason)
asm volatile("hlt\n" : : : "memory");
}
-static bool sev_es_negotiate_protocol(void)
+static bool __maybe_unused sev_es_negotiate_protocol(void)
{
u64 val;
@@ -46,7 +46,7 @@ static bool sev_es_negotiate_protocol(void)
return true;
}
-static void vc_ghcb_invalidate(struct ghcb *ghcb)
+static void __maybe_unused vc_ghcb_invalidate(struct ghcb *ghcb)
{
memset(ghcb->save.valid_bitmap, 0, sizeof(ghcb->save.valid_bitmap));
}
@@ -58,9 +58,9 @@ static bool vc_decoding_needed(unsigned long exit_code)
exit_code <= SVM_EXIT_LAST_EXCP);
}
-static enum es_result vc_init_em_ctxt(struct es_em_ctxt *ctxt,
- struct pt_regs *regs,
- unsigned long exit_code)
+static enum es_result __maybe_unused vc_init_em_ctxt(struct es_em_ctxt *ctxt,
+ struct pt_regs *regs,
+ unsigned long exit_code)
{
enum es_result ret = ES_OK;
@@ -73,7 +73,7 @@ static enum es_result vc_init_em_ctxt(struct es_em_ctxt *ctxt,
return ret;
}
-static void vc_finish_insn(struct es_em_ctxt *ctxt)
+static void __maybe_unused vc_finish_insn(struct es_em_ctxt *ctxt)
{
ctxt->regs->ip += ctxt->insn.length;
}
@@ -325,7 +325,8 @@ static enum es_result vc_ioio_exitinfo(struct es_em_ctxt *ctxt, u64 *exitinfo)
return ES_OK;
}
-static enum es_result vc_handle_ioio(struct ghcb *ghcb, struct es_em_ctxt *ctxt)
+static enum es_result __maybe_unused
+vc_handle_ioio(struct ghcb *ghcb, struct es_em_ctxt *ctxt)
{
struct pt_regs *regs = ctxt->regs;
u64 exit_info_1, exit_info_2;
@@ -433,8 +434,8 @@ static enum es_result vc_handle_ioio(struct ghcb *ghcb, struct es_em_ctxt *ctxt)
return ret;
}
-static enum es_result vc_handle_cpuid(struct ghcb *ghcb,
- struct es_em_ctxt *ctxt)
+static enum es_result __maybe_unused vc_handle_cpuid(struct ghcb *ghcb,
+ struct es_em_ctxt *ctxt)
{
struct pt_regs *regs = ctxt->regs;
u32 cr4 = native_read_cr4();
diff --git a/arch/x86/kernel/sev-es.c b/arch/x86/kernel/sev-es.c
new file mode 100644
index 0000000..4b1d217
--- /dev/null
+++ b/arch/x86/kernel/sev-es.c
@@ -0,0 +1,163 @@
+// SPDX-License-Identifier: GPL-2.0-only
+/*
+ * AMD Memory Encryption Support
+ *
+ * Copyright (C) 2019 SUSE
+ *
+ * Author: Joerg Roedel <jroedel@suse.de>
+ */
+
+#include <linux/kernel.h>
+#include <linux/mm.h>
+
+#include <asm/sev-es.h>
+#include <asm/insn-eval.h>
+#include <asm/fpu/internal.h>
+#include <asm/processor.h>
+#include <asm/trap_pf.h>
+#include <asm/trapnr.h>
+#include <asm/svm.h>
+
+static inline u64 sev_es_rd_ghcb_msr(void)
+{
+ return __rdmsr(MSR_AMD64_SEV_ES_GHCB);
+}
+
+static inline void sev_es_wr_ghcb_msr(u64 val)
+{
+ u32 low, high;
+
+ low = (u32)(val);
+ high = (u32)(val >> 32);
+
+ native_wrmsr(MSR_AMD64_SEV_ES_GHCB, low, high);
+}
+
+static int vc_fetch_insn_kernel(struct es_em_ctxt *ctxt,
+ unsigned char *buffer)
+{
+ return copy_from_kernel_nofault(buffer, (unsigned char *)ctxt->regs->ip, MAX_INSN_SIZE);
+}
+
+static enum es_result vc_decode_insn(struct es_em_ctxt *ctxt)
+{
+ char buffer[MAX_INSN_SIZE];
+ enum es_result ret;
+ int res;
+
+ res = vc_fetch_insn_kernel(ctxt, buffer);
+ if (unlikely(res == -EFAULT)) {
+ ctxt->fi.vector = X86_TRAP_PF;
+ ctxt->fi.error_code = 0;
+ ctxt->fi.cr2 = ctxt->regs->ip;
+ return ES_EXCEPTION;
+ }
+
+ insn_init(&ctxt->insn, buffer, MAX_INSN_SIZE - res, 1);
+ insn_get_length(&ctxt->insn);
+
+ ret = ctxt->insn.immediate.got ? ES_OK : ES_DECODE_FAILED;
+
+ return ret;
+}
+
+static enum es_result vc_write_mem(struct es_em_ctxt *ctxt,
+ char *dst, char *buf, size_t size)
+{
+ unsigned long error_code = X86_PF_PROT | X86_PF_WRITE;
+ char __user *target = (char __user *)dst;
+ u64 d8;
+ u32 d4;
+ u16 d2;
+ u8 d1;
+
+ switch (size) {
+ case 1:
+ memcpy(&d1, buf, 1);
+ if (put_user(d1, target))
+ goto fault;
+ break;
+ case 2:
+ memcpy(&d2, buf, 2);
+ if (put_user(d2, target))
+ goto fault;
+ break;
+ case 4:
+ memcpy(&d4, buf, 4);
+ if (put_user(d4, target))
+ goto fault;
+ break;
+ case 8:
+ memcpy(&d8, buf, 8);
+ if (put_user(d8, target))
+ goto fault;
+ break;
+ default:
+ WARN_ONCE(1, "%s: Invalid size: %zu\n", __func__, size);
+ return ES_UNSUPPORTED;
+ }
+
+ return ES_OK;
+
+fault:
+ if (user_mode(ctxt->regs))
+ error_code |= X86_PF_USER;
+
+ ctxt->fi.vector = X86_TRAP_PF;
+ ctxt->fi.error_code = error_code;
+ ctxt->fi.cr2 = (unsigned long)dst;
+
+ return ES_EXCEPTION;
+}
+
+static enum es_result vc_read_mem(struct es_em_ctxt *ctxt,
+ char *src, char *buf, size_t size)
+{
+ unsigned long error_code = X86_PF_PROT;
+ char __user *s = (char __user *)src;
+ u64 d8;
+ u32 d4;
+ u16 d2;
+ u8 d1;
+
+ switch (size) {
+ case 1:
+ if (get_user(d1, s))
+ goto fault;
+ memcpy(buf, &d1, 1);
+ break;
+ case 2:
+ if (get_user(d2, s))
+ goto fault;
+ memcpy(buf, &d2, 2);
+ break;
+ case 4:
+ if (get_user(d4, s))
+ goto fault;
+ memcpy(buf, &d4, 4);
+ break;
+ case 8:
+ if (get_user(d8, s))
+ goto fault;
+ memcpy(buf, &d8, 8);
+ break;
+ default:
+ WARN_ONCE(1, "%s: Invalid size: %zu\n", __func__, size);
+ return ES_UNSUPPORTED;
+ }
+
+ return ES_OK;
+
+fault:
+ if (user_mode(ctxt->regs))
+ error_code |= X86_PF_USER;
+
+ ctxt->fi.vector = X86_TRAP_PF;
+ ctxt->fi.error_code = error_code;
+ ctxt->fi.cr2 = (unsigned long)src;
+
+ return ES_EXCEPTION;
+}
+
+/* Include code shared with pre-decompression boot stage */
+#include "sev-es-shared.c"
^ permalink raw reply related [flat|nested] 264+ messages in thread
* [tip: x86/seves] x86/head/64: Move early exception dispatch to C code
2020-09-07 13:15 ` Joerg Roedel
(?)
@ 2020-09-10 9:22 ` tip-bot2 for Joerg Roedel
-1 siblings, 0 replies; 264+ messages in thread
From: tip-bot2 for Joerg Roedel @ 2020-09-10 9:22 UTC (permalink / raw)
To: linux-tip-commits; +Cc: Joerg Roedel, Borislav Petkov, x86, LKML
The following commit has been merged into the x86/seves branch of tip:
Commit-ID: 4b47cdbda6f1ad73b08dc7d497bac12b8f26ae0d
Gitweb: https://git.kernel.org/tip/4b47cdbda6f1ad73b08dc7d497bac12b8f26ae0d
Author: Joerg Roedel <jroedel@suse.de>
AuthorDate: Mon, 07 Sep 2020 15:15:36 +02:00
Committer: Borislav Petkov <bp@suse.de>
CommitterDate: Mon, 07 Sep 2020 22:49:18 +02:00
x86/head/64: Move early exception dispatch to C code
Move the assembly coded dispatch between page-faults and all other
exceptions to C code to make it easier to maintain and extend.
Also change the return-type of early_make_pgtable() to bool and make it
static.
Signed-off-by: Joerg Roedel <jroedel@suse.de>
Signed-off-by: Borislav Petkov <bp@suse.de>
Link: https://lkml.kernel.org/r/20200907131613.12703-36-joro@8bytes.org
---
arch/x86/include/asm/pgtable.h | 2 +-
arch/x86/include/asm/setup.h | 4 +++-
arch/x86/kernel/head64.c | 19 +++++++++++++++----
arch/x86/kernel/head_64.S | 11 +----------
4 files changed, 20 insertions(+), 16 deletions(-)
diff --git a/arch/x86/include/asm/pgtable.h b/arch/x86/include/asm/pgtable.h
index b836138..7b8f212 100644
--- a/arch/x86/include/asm/pgtable.h
+++ b/arch/x86/include/asm/pgtable.h
@@ -28,7 +28,7 @@
#include <asm-generic/pgtable_uffd.h>
extern pgd_t early_top_pgt[PTRS_PER_PGD];
-int __init __early_make_pgtable(unsigned long address, pmdval_t pmd);
+bool __init __early_make_pgtable(unsigned long address, pmdval_t pmd);
void ptdump_walk_pgd_level(struct seq_file *m, struct mm_struct *mm);
void ptdump_walk_pgd_level_debugfs(struct seq_file *m, struct mm_struct *mm,
diff --git a/arch/x86/include/asm/setup.h b/arch/x86/include/asm/setup.h
index 4b3ca5a..7d7a064 100644
--- a/arch/x86/include/asm/setup.h
+++ b/arch/x86/include/asm/setup.h
@@ -39,6 +39,8 @@ void vsmp_init(void);
static inline void vsmp_init(void) { }
#endif
+struct pt_regs;
+
void setup_bios_corruption_check(void);
void early_platform_quirks(void);
@@ -49,8 +51,8 @@ extern void i386_reserve_resources(void);
extern unsigned long __startup_64(unsigned long physaddr, struct boot_params *bp);
extern unsigned long __startup_secondary_64(void);
extern void startup_64_setup_env(unsigned long physbase);
-extern int early_make_pgtable(unsigned long address);
extern void early_setup_idt(void);
+extern void __init do_early_exception(struct pt_regs *regs, int trapnr);
#ifdef CONFIG_X86_INTEL_MID
extern void x86_intel_mid_early_setup(void);
diff --git a/arch/x86/kernel/head64.c b/arch/x86/kernel/head64.c
index 7bfd5c2..4282dac 100644
--- a/arch/x86/kernel/head64.c
+++ b/arch/x86/kernel/head64.c
@@ -38,6 +38,8 @@
#include <asm/fixmap.h>
#include <asm/realmode.h>
#include <asm/desc.h>
+#include <asm/extable.h>
+#include <asm/trapnr.h>
/*
* Manage page tables very early on.
@@ -317,7 +319,7 @@ static void __init reset_early_page_tables(void)
}
/* Create a new PMD entry */
-int __init __early_make_pgtable(unsigned long address, pmdval_t pmd)
+bool __init __early_make_pgtable(unsigned long address, pmdval_t pmd)
{
unsigned long physaddr = address - __PAGE_OFFSET;
pgdval_t pgd, *pgd_p;
@@ -327,7 +329,7 @@ int __init __early_make_pgtable(unsigned long address, pmdval_t pmd)
/* Invalid address or early pgt is done ? */
if (physaddr >= MAXMEM || read_cr3_pa() != __pa_nodebug(early_top_pgt))
- return -1;
+ return false;
again:
pgd_p = &early_top_pgt[pgd_index(address)].pgd;
@@ -384,10 +386,10 @@ again:
}
pmd_p[pmd_index(address)] = pmd;
- return 0;
+ return true;
}
-int __init early_make_pgtable(unsigned long address)
+static bool __init early_make_pgtable(unsigned long address)
{
unsigned long physaddr = address - __PAGE_OFFSET;
pmdval_t pmd;
@@ -397,6 +399,15 @@ int __init early_make_pgtable(unsigned long address)
return __early_make_pgtable(address, pmd);
}
+void __init do_early_exception(struct pt_regs *regs, int trapnr)
+{
+ if (trapnr == X86_TRAP_PF &&
+ early_make_pgtable(native_read_cr2()))
+ return;
+
+ early_fixup_exception(regs, trapnr);
+}
+
/* Don't add a printk in there. printk relies on the PDA which is not initialized
yet. */
static void __init clear_bss(void)
diff --git a/arch/x86/kernel/head_64.S b/arch/x86/kernel/head_64.S
index 1de09b5..3b40ec4 100644
--- a/arch/x86/kernel/head_64.S
+++ b/arch/x86/kernel/head_64.S
@@ -341,18 +341,9 @@ SYM_CODE_START_LOCAL(early_idt_handler_common)
pushq %r15 /* pt_regs->r15 */
UNWIND_HINT_REGS
- cmpq $14,%rsi /* Page fault? */
- jnz 10f
- GET_CR2_INTO(%rdi) /* can clobber %rax if pv */
- call early_make_pgtable
- andl %eax,%eax
- jz 20f /* All good */
-
-10:
movq %rsp,%rdi /* RDI = pt_regs; RSI is already trapnr */
- call early_fixup_exception
+ call do_early_exception
-20:
decl early_recursion_flag(%rip)
jmp restore_regs_and_return_to_kernel
SYM_CODE_END(early_idt_handler_common)
^ permalink raw reply related [flat|nested] 264+ messages in thread
* [tip: x86/seves] x86/sev-es: Add SEV-ES Feature Detection
2020-09-07 13:15 ` Joerg Roedel
(?)
(?)
@ 2020-09-10 9:22 ` tip-bot2 for Joerg Roedel
-1 siblings, 0 replies; 264+ messages in thread
From: tip-bot2 for Joerg Roedel @ 2020-09-10 9:22 UTC (permalink / raw)
To: linux-tip-commits; +Cc: Joerg Roedel, Borislav Petkov, Kees Cook, x86, LKML
The following commit has been merged into the x86/seves branch of tip:
Commit-ID: b57de6cd16395be1ebdaa9b489ffbf462bb585c4
Gitweb: https://git.kernel.org/tip/b57de6cd16395be1ebdaa9b489ffbf462bb585c4
Author: Joerg Roedel <jroedel@suse.de>
AuthorDate: Mon, 07 Sep 2020 15:15:37 +02:00
Committer: Borislav Petkov <bp@suse.de>
CommitterDate: Mon, 07 Sep 2020 23:00:20 +02:00
x86/sev-es: Add SEV-ES Feature Detection
Add a sev_es_active() function for checking whether SEV-ES is enabled.
Also cache the value of MSR_AMD64_SEV at boot to speed up the feature
checking in the running code.
[ bp: Remove "!!" in sev_active() too. ]
Signed-off-by: Joerg Roedel <jroedel@suse.de>
Signed-off-by: Borislav Petkov <bp@suse.de>
Reviewed-by: Kees Cook <keescook@chromium.org>
Link: https://lkml.kernel.org/r/20200907131613.12703-37-joro@8bytes.org
---
arch/x86/include/asm/mem_encrypt.h | 3 +++
arch/x86/include/asm/msr-index.h | 2 ++
arch/x86/mm/mem_encrypt.c | 9 ++++++++-
arch/x86/mm/mem_encrypt_identity.c | 3 +++
4 files changed, 16 insertions(+), 1 deletion(-)
diff --git a/arch/x86/include/asm/mem_encrypt.h b/arch/x86/include/asm/mem_encrypt.h
index 5049f6c..4e72b73 100644
--- a/arch/x86/include/asm/mem_encrypt.h
+++ b/arch/x86/include/asm/mem_encrypt.h
@@ -19,6 +19,7 @@
#ifdef CONFIG_AMD_MEM_ENCRYPT
extern u64 sme_me_mask;
+extern u64 sev_status;
extern bool sev_enabled;
void sme_encrypt_execute(unsigned long encrypted_kernel_vaddr,
@@ -50,6 +51,7 @@ void __init mem_encrypt_init(void);
bool sme_active(void);
bool sev_active(void);
+bool sev_es_active(void);
#define __bss_decrypted __attribute__((__section__(".bss..decrypted")))
@@ -72,6 +74,7 @@ static inline void __init sme_enable(struct boot_params *bp) { }
static inline bool sme_active(void) { return false; }
static inline bool sev_active(void) { return false; }
+static inline bool sev_es_active(void) { return false; }
static inline int __init
early_set_memory_decrypted(unsigned long vaddr, unsigned long size) { return 0; }
diff --git a/arch/x86/include/asm/msr-index.h b/arch/x86/include/asm/msr-index.h
index da34fdb..249a414 100644
--- a/arch/x86/include/asm/msr-index.h
+++ b/arch/x86/include/asm/msr-index.h
@@ -469,7 +469,9 @@
#define MSR_AMD64_SEV_ES_GHCB 0xc0010130
#define MSR_AMD64_SEV 0xc0010131
#define MSR_AMD64_SEV_ENABLED_BIT 0
+#define MSR_AMD64_SEV_ES_ENABLED_BIT 1
#define MSR_AMD64_SEV_ENABLED BIT_ULL(MSR_AMD64_SEV_ENABLED_BIT)
+#define MSR_AMD64_SEV_ES_ENABLED BIT_ULL(MSR_AMD64_SEV_ES_ENABLED_BIT)
#define MSR_AMD64_VIRT_SPEC_CTRL 0xc001011f
diff --git a/arch/x86/mm/mem_encrypt.c b/arch/x86/mm/mem_encrypt.c
index 9f1177e..a38f556 100644
--- a/arch/x86/mm/mem_encrypt.c
+++ b/arch/x86/mm/mem_encrypt.c
@@ -38,6 +38,7 @@
* section is later cleared.
*/
u64 sme_me_mask __section(.data) = 0;
+u64 sev_status __section(.data) = 0;
EXPORT_SYMBOL(sme_me_mask);
DEFINE_STATIC_KEY_FALSE(sev_enable_key);
EXPORT_SYMBOL_GPL(sev_enable_key);
@@ -347,7 +348,13 @@ bool sme_active(void)
bool sev_active(void)
{
- return sme_me_mask && sev_enabled;
+ return sev_status & MSR_AMD64_SEV_ENABLED;
+}
+
+/* Needs to be called from non-instrumentable code */
+bool noinstr sev_es_active(void)
+{
+ return sev_status & MSR_AMD64_SEV_ES_ENABLED;
}
/* Override for DMA direct allocation check - ARCH_HAS_FORCE_DMA_UNENCRYPTED */
diff --git a/arch/x86/mm/mem_encrypt_identity.c b/arch/x86/mm/mem_encrypt_identity.c
index e2b0e2a..68d7537 100644
--- a/arch/x86/mm/mem_encrypt_identity.c
+++ b/arch/x86/mm/mem_encrypt_identity.c
@@ -540,6 +540,9 @@ void __init sme_enable(struct boot_params *bp)
if (!(msr & MSR_AMD64_SEV_ENABLED))
return;
+ /* Save SEV_STATUS to avoid reading MSR again */
+ sev_status = msr;
+
/* SEV state cannot be controlled by a command line option */
sme_me_mask = me_mask;
sev_enabled = true;
^ permalink raw reply related [flat|nested] 264+ messages in thread
* [tip: x86/seves] x86/head/64: Install a CPU bringup IDT
2020-09-07 13:15 ` Joerg Roedel
(?)
@ 2020-09-10 9:22 ` tip-bot2 for Joerg Roedel
-1 siblings, 0 replies; 264+ messages in thread
From: tip-bot2 for Joerg Roedel @ 2020-09-10 9:22 UTC (permalink / raw)
To: linux-tip-commits; +Cc: Joerg Roedel, Borislav Petkov, x86, LKML
The following commit has been merged into the x86/seves branch of tip:
Commit-ID: f5963ba7a45fc6ff298a34976064354be437e1d8
Gitweb: https://git.kernel.org/tip/f5963ba7a45fc6ff298a34976064354be437e1d8
Author: Joerg Roedel <jroedel@suse.de>
AuthorDate: Mon, 07 Sep 2020 15:15:34 +02:00
Committer: Borislav Petkov <bp@suse.de>
CommitterDate: Mon, 07 Sep 2020 22:18:38 +02:00
x86/head/64: Install a CPU bringup IDT
Add a separate bringup IDT for the CPU bringup code that will be used
until the kernel switches to the idt_table. There are two reasons for a
separate IDT:
1) When the idt_table is set up and the secondary CPUs are
booted, it contains entries (e.g. IST entries) which
require certain CPU state to be set up. This includes a
working TSS (for IST), MSR_GS_BASE (for stack protector) or
CR4.FSGSBASE (for paranoid_entry) path. By using a
dedicated IDT for early boot this state need not to be set
up early.
2) The idt_table is static to idt.c, so any function
using/modifying must be in idt.c too. That means that all
compiler driven instrumentation like tracing or KASAN is
also active in this code. But during early CPU bringup the
environment is not set up for this instrumentation to work
correctly.
To avoid all of these hassles and make early exception handling robust,
use a dedicated bringup IDT.
The IDT is loaded two times, first on the boot CPU while the kernel is
still running on direct mapped addresses, and again later after the
switch to kernel addresses has happened. The second IDT load happens on
the boot and secondary CPUs.
Signed-off-by: Joerg Roedel <jroedel@suse.de>
Signed-off-by: Borislav Petkov <bp@suse.de>
Link: https://lkml.kernel.org/r/20200907131613.12703-34-joro@8bytes.org
---
arch/x86/include/asm/setup.h | 1 +-
arch/x86/kernel/head64.c | 39 +++++++++++++++++++++++++++++++++++-
arch/x86/kernel/head_64.S | 5 ++++-
3 files changed, 45 insertions(+)
diff --git a/arch/x86/include/asm/setup.h b/arch/x86/include/asm/setup.h
index 5c2fd05..4b3ca5a 100644
--- a/arch/x86/include/asm/setup.h
+++ b/arch/x86/include/asm/setup.h
@@ -50,6 +50,7 @@ extern unsigned long __startup_64(unsigned long physaddr, struct boot_params *bp
extern unsigned long __startup_secondary_64(void);
extern void startup_64_setup_env(unsigned long physbase);
extern int early_make_pgtable(unsigned long address);
+extern void early_setup_idt(void);
#ifdef CONFIG_X86_INTEL_MID
extern void x86_intel_mid_early_setup(void);
diff --git a/arch/x86/kernel/head64.c b/arch/x86/kernel/head64.c
index 8c82be4..7bfd5c2 100644
--- a/arch/x86/kernel/head64.c
+++ b/arch/x86/kernel/head64.c
@@ -36,6 +36,8 @@
#include <asm/microcode.h>
#include <asm/kasan.h>
#include <asm/fixmap.h>
+#include <asm/realmode.h>
+#include <asm/desc.h>
/*
* Manage page tables very early on.
@@ -509,6 +511,41 @@ void __init x86_64_start_reservations(char *real_mode_data)
}
/*
+ * Data structures and code used for IDT setup in head_64.S. The bringup-IDT is
+ * used until the idt_table takes over. On the boot CPU this happens in
+ * x86_64_start_kernel(), on secondary CPUs in start_secondary(). In both cases
+ * this happens in the functions called from head_64.S.
+ *
+ * The idt_table can't be used that early because all the code modifying it is
+ * in idt.c and can be instrumented by tracing or KASAN, which both don't work
+ * during early CPU bringup. Also the idt_table has the runtime vectors
+ * configured which require certain CPU state to be setup already (like TSS),
+ * which also hasn't happened yet in early CPU bringup.
+ */
+static gate_desc bringup_idt_table[NUM_EXCEPTION_VECTORS] __page_aligned_data;
+
+static struct desc_ptr bringup_idt_descr = {
+ .size = (NUM_EXCEPTION_VECTORS * sizeof(gate_desc)) - 1,
+ .address = 0, /* Set at runtime */
+};
+
+/* This runs while still in the direct mapping */
+static void startup_64_load_idt(unsigned long physbase)
+{
+ struct desc_ptr *desc = fixup_pointer(&bringup_idt_descr, physbase);
+
+ desc->address = (unsigned long)fixup_pointer(bringup_idt_table, physbase);
+ native_load_idt(desc);
+}
+
+/* This is used when running on kernel addresses */
+void early_setup_idt(void)
+{
+ bringup_idt_descr.address = (unsigned long)bringup_idt_table;
+ native_load_idt(&bringup_idt_descr);
+}
+
+/*
* Setup boot CPU state needed before kernel switches to virtual addresses.
*/
void __head startup_64_setup_env(unsigned long physbase)
@@ -521,4 +558,6 @@ void __head startup_64_setup_env(unsigned long physbase)
asm volatile("movl %%eax, %%ds\n"
"movl %%eax, %%ss\n"
"movl %%eax, %%es\n" : : "a"(__KERNEL_DS) : "memory");
+
+ startup_64_load_idt(physbase);
}
diff --git a/arch/x86/kernel/head_64.S b/arch/x86/kernel/head_64.S
index 83050c9..1de09b5 100644
--- a/arch/x86/kernel/head_64.S
+++ b/arch/x86/kernel/head_64.S
@@ -198,6 +198,11 @@ SYM_CODE_START(secondary_startup_64)
*/
movq initial_stack(%rip), %rsp
+ /* Setup and Load IDT */
+ pushq %rsi
+ call early_setup_idt
+ popq %rsi
+
/* Check if nx is implemented */
movl $0x80000001, %eax
cpuid
^ permalink raw reply related [flat|nested] 264+ messages in thread
* [tip: x86/seves] x86/idt: Make IDT init functions static inlines
2020-09-07 13:15 ` Joerg Roedel
(?)
@ 2020-09-10 9:22 ` tip-bot2 for Joerg Roedel
-1 siblings, 0 replies; 264+ messages in thread
From: tip-bot2 for Joerg Roedel @ 2020-09-10 9:22 UTC (permalink / raw)
To: linux-tip-commits; +Cc: Joerg Roedel, Borislav Petkov, Kees Cook, x86, LKML
The following commit has been merged into the x86/seves branch of tip:
Commit-ID: 097ee5b778b8970e1c2ed3ca1631b297d90acd61
Gitweb: https://git.kernel.org/tip/097ee5b778b8970e1c2ed3ca1631b297d90acd61
Author: Joerg Roedel <jroedel@suse.de>
AuthorDate: Mon, 07 Sep 2020 15:15:35 +02:00
Committer: Borislav Petkov <bp@suse.de>
CommitterDate: Mon, 07 Sep 2020 22:44:43 +02:00
x86/idt: Make IDT init functions static inlines
Move these two functions from kernel/idt.c to include/asm/desc.h:
* init_idt_data()
* idt_init_desc()
These functions are needed to setup IDT entries very early and need to
be called from head64.c. To be usable this early, these functions need
to be compiled without instrumentation and the stack-protector feature.
These features need to be kept enabled for kernel/idt.c, so head64.c
must use its own versions.
[ bp: Take Kees' suggested patch title and add his Rev-by. ]
Signed-off-by: Joerg Roedel <jroedel@suse.de>
Signed-off-by: Borislav Petkov <bp@suse.de>
Reviewed-by: Kees Cook <keescook@chromium.org>
Link: https://lkml.kernel.org/r/20200907131613.12703-35-joro@8bytes.org
---
arch/x86/include/asm/desc.h | 27 +++++++++++++++++++++++++-
arch/x86/include/asm/desc_defs.h | 7 ++++++-
arch/x86/kernel/idt.c | 34 +-------------------------------
3 files changed, 34 insertions(+), 34 deletions(-)
diff --git a/arch/x86/include/asm/desc.h b/arch/x86/include/asm/desc.h
index 1ced11d..476082a 100644
--- a/arch/x86/include/asm/desc.h
+++ b/arch/x86/include/asm/desc.h
@@ -383,6 +383,33 @@ static inline void set_desc_limit(struct desc_struct *desc, unsigned long limit)
void alloc_intr_gate(unsigned int n, const void *addr);
+static inline void init_idt_data(struct idt_data *data, unsigned int n,
+ const void *addr)
+{
+ BUG_ON(n > 0xFF);
+
+ memset(data, 0, sizeof(*data));
+ data->vector = n;
+ data->addr = addr;
+ data->segment = __KERNEL_CS;
+ data->bits.type = GATE_INTERRUPT;
+ data->bits.p = 1;
+}
+
+static inline void idt_init_desc(gate_desc *gate, const struct idt_data *d)
+{
+ unsigned long addr = (unsigned long) d->addr;
+
+ gate->offset_low = (u16) addr;
+ gate->segment = (u16) d->segment;
+ gate->bits = d->bits;
+ gate->offset_middle = (u16) (addr >> 16);
+#ifdef CONFIG_X86_64
+ gate->offset_high = (u32) (addr >> 32);
+ gate->reserved = 0;
+#endif
+}
+
extern unsigned long system_vectors[];
extern void load_current_idt(void);
diff --git a/arch/x86/include/asm/desc_defs.h b/arch/x86/include/asm/desc_defs.h
index 5621fb3..f7e7099 100644
--- a/arch/x86/include/asm/desc_defs.h
+++ b/arch/x86/include/asm/desc_defs.h
@@ -74,6 +74,13 @@ struct idt_bits {
p : 1;
} __attribute__((packed));
+struct idt_data {
+ unsigned int vector;
+ unsigned int segment;
+ struct idt_bits bits;
+ const void *addr;
+};
+
struct gate_struct {
u16 offset_low;
u16 segment;
diff --git a/arch/x86/kernel/idt.c b/arch/x86/kernel/idt.c
index 53946c1..4bb4e3d 100644
--- a/arch/x86/kernel/idt.c
+++ b/arch/x86/kernel/idt.c
@@ -11,13 +11,6 @@
#include <asm/desc.h>
#include <asm/hw_irq.h>
-struct idt_data {
- unsigned int vector;
- unsigned int segment;
- struct idt_bits bits;
- const void *addr;
-};
-
#define DPL0 0x0
#define DPL3 0x3
@@ -178,20 +171,6 @@ bool idt_is_f00f_address(unsigned long address)
}
#endif
-static inline void idt_init_desc(gate_desc *gate, const struct idt_data *d)
-{
- unsigned long addr = (unsigned long) d->addr;
-
- gate->offset_low = (u16) addr;
- gate->segment = (u16) d->segment;
- gate->bits = d->bits;
- gate->offset_middle = (u16) (addr >> 16);
-#ifdef CONFIG_X86_64
- gate->offset_high = (u32) (addr >> 32);
- gate->reserved = 0;
-#endif
-}
-
static __init void
idt_setup_from_table(gate_desc *idt, const struct idt_data *t, int size, bool sys)
{
@@ -205,19 +184,6 @@ idt_setup_from_table(gate_desc *idt, const struct idt_data *t, int size, bool sy
}
}
-static void init_idt_data(struct idt_data *data, unsigned int n,
- const void *addr)
-{
- BUG_ON(n > 0xFF);
-
- memset(data, 0, sizeof(*data));
- data->vector = n;
- data->addr = addr;
- data->segment = __KERNEL_CS;
- data->bits.type = GATE_INTERRUPT;
- data->bits.p = 1;
-}
-
static __init void set_intr_gate(unsigned int n, const void *addr)
{
struct idt_data data;
^ permalink raw reply related [flat|nested] 264+ messages in thread
* [tip: x86/seves] x86/head/64: Load segment registers earlier
2020-09-07 13:15 ` Joerg Roedel
(?)
@ 2020-09-10 9:22 ` tip-bot2 for Joerg Roedel
-1 siblings, 0 replies; 264+ messages in thread
From: tip-bot2 for Joerg Roedel @ 2020-09-10 9:22 UTC (permalink / raw)
To: linux-tip-commits; +Cc: Joerg Roedel, Borislav Petkov, Kees Cook, x86, LKML
The following commit has been merged into the x86/seves branch of tip:
Commit-ID: 7b99819dfb60268cc1c75f83c949bc4a09221bea
Gitweb: https://git.kernel.org/tip/7b99819dfb60268cc1c75f83c949bc4a09221bea
Author: Joerg Roedel <jroedel@suse.de>
AuthorDate: Mon, 07 Sep 2020 15:15:32 +02:00
Committer: Borislav Petkov <bp@suse.de>
CommitterDate: Mon, 07 Sep 2020 21:40:53 +02:00
x86/head/64: Load segment registers earlier
Make sure segments are properly set up before setting up an IDT and
doing anything that might cause a #VC exception. This is later needed
for early exception handling.
Signed-off-by: Joerg Roedel <jroedel@suse.de>
Signed-off-by: Borislav Petkov <bp@suse.de>
Reviewed-by: Kees Cook <keescook@chromium.org>
Link: https://lkml.kernel.org/r/20200907131613.12703-32-joro@8bytes.org
---
arch/x86/kernel/head_64.S | 52 +++++++++++++++++++-------------------
1 file changed, 26 insertions(+), 26 deletions(-)
diff --git a/arch/x86/kernel/head_64.S b/arch/x86/kernel/head_64.S
index 03b03f2..f402087 100644
--- a/arch/x86/kernel/head_64.S
+++ b/arch/x86/kernel/head_64.S
@@ -166,6 +166,32 @@ SYM_CODE_START(secondary_startup_64)
*/
lgdt early_gdt_descr(%rip)
+ /* set up data segments */
+ xorl %eax,%eax
+ movl %eax,%ds
+ movl %eax,%ss
+ movl %eax,%es
+
+ /*
+ * We don't really need to load %fs or %gs, but load them anyway
+ * to kill any stale realmode selectors. This allows execution
+ * under VT hardware.
+ */
+ movl %eax,%fs
+ movl %eax,%gs
+
+ /* Set up %gs.
+ *
+ * The base of %gs always points to fixed_percpu_data. If the
+ * stack protector canary is enabled, it is located at %gs:40.
+ * Note that, on SMP, the boot cpu uses init data section until
+ * the per cpu areas are set up.
+ */
+ movl $MSR_GS_BASE,%ecx
+ movl initial_gs(%rip),%eax
+ movl initial_gs+4(%rip),%edx
+ wrmsr
+
/* Check if nx is implemented */
movl $0x80000001, %eax
cpuid
@@ -193,32 +219,6 @@ SYM_CODE_START(secondary_startup_64)
pushq $0
popfq
- /* set up data segments */
- xorl %eax,%eax
- movl %eax,%ds
- movl %eax,%ss
- movl %eax,%es
-
- /*
- * We don't really need to load %fs or %gs, but load them anyway
- * to kill any stale realmode selectors. This allows execution
- * under VT hardware.
- */
- movl %eax,%fs
- movl %eax,%gs
-
- /* Set up %gs.
- *
- * The base of %gs always points to fixed_percpu_data. If the
- * stack protector canary is enabled, it is located at %gs:40.
- * Note that, on SMP, the boot cpu uses init data section until
- * the per cpu areas are set up.
- */
- movl $MSR_GS_BASE,%ecx
- movl initial_gs(%rip),%eax
- movl initial_gs+4(%rip),%edx
- wrmsr
-
/* rsi is pointer to real mode structure with interesting info.
pass it to C */
movq %rsi, %rdi
^ permalink raw reply related [flat|nested] 264+ messages in thread
* [tip: x86/seves] x86/head/64: Switch to initial stack earlier
2020-09-07 13:15 ` Joerg Roedel
(?)
@ 2020-09-10 9:22 ` tip-bot2 for Joerg Roedel
-1 siblings, 0 replies; 264+ messages in thread
From: tip-bot2 for Joerg Roedel @ 2020-09-10 9:22 UTC (permalink / raw)
To: linux-tip-commits; +Cc: Joerg Roedel, Borislav Petkov, Kees Cook, x86, LKML
The following commit has been merged into the x86/seves branch of tip:
Commit-ID: 3add38cb96a1ae7d152db69ab4329e809c2af2d4
Gitweb: https://git.kernel.org/tip/3add38cb96a1ae7d152db69ab4329e809c2af2d4
Author: Joerg Roedel <jroedel@suse.de>
AuthorDate: Mon, 07 Sep 2020 15:15:33 +02:00
Committer: Borislav Petkov <bp@suse.de>
CommitterDate: Mon, 07 Sep 2020 21:44:01 +02:00
x86/head/64: Switch to initial stack earlier
Make sure there is a stack once the kernel runs from virtual addresses.
At this stage any secondary CPU which boots will have lost its stack
because the kernel switched to a new page-table which does not map the
real-mode stack anymore.
This is needed for handling early #VC exceptions caused by instructions
like CPUID.
Signed-off-by: Joerg Roedel <jroedel@suse.de>
Signed-off-by: Borislav Petkov <bp@suse.de>
Reviewed-by: Kees Cook <keescook@chromium.org>
Link: https://lkml.kernel.org/r/20200907131613.12703-33-joro@8bytes.org
---
arch/x86/kernel/head_64.S | 9 ++++++---
1 file changed, 6 insertions(+), 3 deletions(-)
diff --git a/arch/x86/kernel/head_64.S b/arch/x86/kernel/head_64.S
index f402087..83050c9 100644
--- a/arch/x86/kernel/head_64.S
+++ b/arch/x86/kernel/head_64.S
@@ -192,6 +192,12 @@ SYM_CODE_START(secondary_startup_64)
movl initial_gs+4(%rip),%edx
wrmsr
+ /*
+ * Setup a boot time stack - Any secondary CPU will have lost its stack
+ * by now because the cr3-switch above unmaps the real-mode stack
+ */
+ movq initial_stack(%rip), %rsp
+
/* Check if nx is implemented */
movl $0x80000001, %eax
cpuid
@@ -212,9 +218,6 @@ SYM_CODE_START(secondary_startup_64)
/* Make changes effective */
movq %rax, %cr0
- /* Setup a boot time stack */
- movq initial_stack(%rip), %rsp
-
/* zero EFLAGS after setting rsp */
pushq $0
popfq
^ permalink raw reply related [flat|nested] 264+ messages in thread
* [tip: x86/seves] x86/idt: Split idt_data setup out of set_intr_gate()
2020-09-07 13:15 ` Joerg Roedel
(?)
@ 2020-09-10 9:22 ` tip-bot2 for Joerg Roedel
-1 siblings, 0 replies; 264+ messages in thread
From: tip-bot2 for Joerg Roedel @ 2020-09-10 9:22 UTC (permalink / raw)
To: linux-tip-commits; +Cc: Joerg Roedel, Borislav Petkov, Kees Cook, x86, LKML
The following commit has been merged into the x86/seves branch of tip:
Commit-ID: 4bed2266cc6f9c3f6cd91378ea4fc76edde674cf
Gitweb: https://git.kernel.org/tip/4bed2266cc6f9c3f6cd91378ea4fc76edde674cf
Author: Joerg Roedel <jroedel@suse.de>
AuthorDate: Mon, 07 Sep 2020 15:15:29 +02:00
Committer: Borislav Petkov <bp@suse.de>
CommitterDate: Mon, 07 Sep 2020 21:30:38 +02:00
x86/idt: Split idt_data setup out of set_intr_gate()
The code to setup idt_data is needed for early exception handling, but
set_intr_gate() can't be used that early because it has pv-ops in its
code path which don't work that early.
Split out the idt_data initialization part from set_intr_gate() so
that it can be used separately.
Signed-off-by: Joerg Roedel <jroedel@suse.de>
Signed-off-by: Borislav Petkov <bp@suse.de>
Reviewed-by: Kees Cook <keescook@chromium.org>
Link: https://lkml.kernel.org/r/20200907131613.12703-29-joro@8bytes.org
---
arch/x86/kernel/idt.c | 22 ++++++++++++++--------
1 file changed, 14 insertions(+), 8 deletions(-)
diff --git a/arch/x86/kernel/idt.c b/arch/x86/kernel/idt.c
index 7ecf9ba..53946c1 100644
--- a/arch/x86/kernel/idt.c
+++ b/arch/x86/kernel/idt.c
@@ -205,18 +205,24 @@ idt_setup_from_table(gate_desc *idt, const struct idt_data *t, int size, bool sy
}
}
+static void init_idt_data(struct idt_data *data, unsigned int n,
+ const void *addr)
+{
+ BUG_ON(n > 0xFF);
+
+ memset(data, 0, sizeof(*data));
+ data->vector = n;
+ data->addr = addr;
+ data->segment = __KERNEL_CS;
+ data->bits.type = GATE_INTERRUPT;
+ data->bits.p = 1;
+}
+
static __init void set_intr_gate(unsigned int n, const void *addr)
{
struct idt_data data;
- BUG_ON(n > 0xFF);
-
- memset(&data, 0, sizeof(data));
- data.vector = n;
- data.addr = addr;
- data.segment = __KERNEL_CS;
- data.bits.type = GATE_INTERRUPT;
- data.bits.p = 1;
+ init_idt_data(&data, n, addr);
idt_setup_from_table(idt_table, &data, 1, false);
}
^ permalink raw reply related [flat|nested] 264+ messages in thread
* [tip: x86/seves] x86/head/64: Load GDT after switch to virtual addresses
2020-09-07 13:15 ` Joerg Roedel
(?)
@ 2020-09-10 9:22 ` tip-bot2 for Joerg Roedel
-1 siblings, 0 replies; 264+ messages in thread
From: tip-bot2 for Joerg Roedel @ 2020-09-10 9:22 UTC (permalink / raw)
To: linux-tip-commits; +Cc: Joerg Roedel, Borislav Petkov, Kees Cook, x86, LKML
The following commit has been merged into the x86/seves branch of tip:
Commit-ID: e04b88336360e101329add0c05e5cb1cebae64fd
Gitweb: https://git.kernel.org/tip/e04b88336360e101329add0c05e5cb1cebae64fd
Author: Joerg Roedel <jroedel@suse.de>
AuthorDate: Mon, 07 Sep 2020 15:15:31 +02:00
Committer: Borislav Petkov <bp@suse.de>
CommitterDate: Mon, 07 Sep 2020 21:35:54 +02:00
x86/head/64: Load GDT after switch to virtual addresses
Load the GDT right after switching to virtual addresses to make sure
there is a defined GDT for exception handling.
Signed-off-by: Joerg Roedel <jroedel@suse.de>
Signed-off-by: Borislav Petkov <bp@suse.de>
Reviewed-by: Kees Cook <keescook@chromium.org>
Link: https://lkml.kernel.org/r/20200907131613.12703-31-joro@8bytes.org
---
arch/x86/kernel/head_64.S | 16 ++++++++--------
1 file changed, 8 insertions(+), 8 deletions(-)
diff --git a/arch/x86/kernel/head_64.S b/arch/x86/kernel/head_64.S
index 2b2e916..03b03f2 100644
--- a/arch/x86/kernel/head_64.S
+++ b/arch/x86/kernel/head_64.S
@@ -158,6 +158,14 @@ SYM_CODE_START(secondary_startup_64)
1:
UNWIND_HINT_EMPTY
+ /*
+ * We must switch to a new descriptor in kernel space for the GDT
+ * because soon the kernel won't have access anymore to the userspace
+ * addresses where we're currently running on. We have to do that here
+ * because in 32bit we couldn't load a 64bit linear address.
+ */
+ lgdt early_gdt_descr(%rip)
+
/* Check if nx is implemented */
movl $0x80000001, %eax
cpuid
@@ -185,14 +193,6 @@ SYM_CODE_START(secondary_startup_64)
pushq $0
popfq
- /*
- * We must switch to a new descriptor in kernel space for the GDT
- * because soon the kernel won't have access anymore to the userspace
- * addresses where we're currently running on. We have to do that here
- * because in 32bit we couldn't load a 64bit linear address.
- */
- lgdt early_gdt_descr(%rip)
-
/* set up data segments */
xorl %eax,%eax
movl %eax,%ds
^ permalink raw reply related [flat|nested] 264+ messages in thread
* [tip: x86/seves] x86/head/64: Install startup GDT
2020-09-07 13:15 ` Joerg Roedel
(?)
@ 2020-09-10 9:22 ` tip-bot2 for Joerg Roedel
-1 siblings, 0 replies; 264+ messages in thread
From: tip-bot2 for Joerg Roedel @ 2020-09-10 9:22 UTC (permalink / raw)
To: linux-tip-commits; +Cc: Joerg Roedel, Borislav Petkov, x86, LKML
The following commit has been merged into the x86/seves branch of tip:
Commit-ID: 866b556efa1295934ed0bc20c2f208c93a873fb0
Gitweb: https://git.kernel.org/tip/866b556efa1295934ed0bc20c2f208c93a873fb0
Author: Joerg Roedel <jroedel@suse.de>
AuthorDate: Mon, 07 Sep 2020 15:15:30 +02:00
Committer: Borislav Petkov <bp@suse.de>
CommitterDate: Mon, 07 Sep 2020 21:33:17 +02:00
x86/head/64: Install startup GDT
Handling exceptions during boot requires a working GDT. The kernel GDT
can't be used on the direct mapping, so load a startup GDT and setup
segments.
Signed-off-by: Joerg Roedel <jroedel@suse.de>
Signed-off-by: Borislav Petkov <bp@suse.de>
Link: https://lkml.kernel.org/r/20200907131613.12703-30-joro@8bytes.org
---
arch/x86/include/asm/setup.h | 1 +
arch/x86/kernel/head64.c | 33 +++++++++++++++++++++++++++++++++
arch/x86/kernel/head_64.S | 14 ++++++++++++++
3 files changed, 48 insertions(+)
diff --git a/arch/x86/include/asm/setup.h b/arch/x86/include/asm/setup.h
index 84b645c..5c2fd05 100644
--- a/arch/x86/include/asm/setup.h
+++ b/arch/x86/include/asm/setup.h
@@ -48,6 +48,7 @@ extern void reserve_standard_io_resources(void);
extern void i386_reserve_resources(void);
extern unsigned long __startup_64(unsigned long physaddr, struct boot_params *bp);
extern unsigned long __startup_secondary_64(void);
+extern void startup_64_setup_env(unsigned long physbase);
extern int early_make_pgtable(unsigned long address);
#ifdef CONFIG_X86_INTEL_MID
diff --git a/arch/x86/kernel/head64.c b/arch/x86/kernel/head64.c
index cbb71c1..8c82be4 100644
--- a/arch/x86/kernel/head64.c
+++ b/arch/x86/kernel/head64.c
@@ -61,6 +61,24 @@ unsigned long vmemmap_base __ro_after_init = __VMEMMAP_BASE_L4;
EXPORT_SYMBOL(vmemmap_base);
#endif
+/*
+ * GDT used on the boot CPU before switching to virtual addresses.
+ */
+static struct desc_struct startup_gdt[GDT_ENTRIES] = {
+ [GDT_ENTRY_KERNEL32_CS] = GDT_ENTRY_INIT(0xc09b, 0, 0xfffff),
+ [GDT_ENTRY_KERNEL_CS] = GDT_ENTRY_INIT(0xa09b, 0, 0xfffff),
+ [GDT_ENTRY_KERNEL_DS] = GDT_ENTRY_INIT(0xc093, 0, 0xfffff),
+};
+
+/*
+ * Address needs to be set at runtime because it references the startup_gdt
+ * while the kernel still uses a direct mapping.
+ */
+static struct desc_ptr startup_gdt_descr = {
+ .size = sizeof(startup_gdt),
+ .address = 0,
+};
+
#define __head __section(.head.text)
static void __head *fixup_pointer(void *ptr, unsigned long physaddr)
@@ -489,3 +507,18 @@ void __init x86_64_start_reservations(char *real_mode_data)
start_kernel();
}
+
+/*
+ * Setup boot CPU state needed before kernel switches to virtual addresses.
+ */
+void __head startup_64_setup_env(unsigned long physbase)
+{
+ /* Load GDT */
+ startup_gdt_descr.address = (unsigned long)fixup_pointer(startup_gdt, physbase);
+ native_load_gdt(&startup_gdt_descr);
+
+ /* New GDT is live - reload data segment registers */
+ asm volatile("movl %%eax, %%ds\n"
+ "movl %%eax, %%ss\n"
+ "movl %%eax, %%es\n" : : "a"(__KERNEL_DS) : "memory");
+}
diff --git a/arch/x86/kernel/head_64.S b/arch/x86/kernel/head_64.S
index 16da4ac..2b2e916 100644
--- a/arch/x86/kernel/head_64.S
+++ b/arch/x86/kernel/head_64.S
@@ -73,6 +73,20 @@ SYM_CODE_START_NOALIGN(startup_64)
/* Set up the stack for verify_cpu(), similar to initial_stack below */
leaq (__end_init_task - SIZEOF_PTREGS)(%rip), %rsp
+ leaq _text(%rip), %rdi
+ pushq %rsi
+ call startup_64_setup_env
+ popq %rsi
+
+ /* Now switch to __KERNEL_CS so IRET works reliably */
+ pushq $__KERNEL_CS
+ leaq .Lon_kernel_cs(%rip), %rax
+ pushq %rax
+ lretq
+
+.Lon_kernel_cs:
+ UNWIND_HINT_EMPTY
+
/* Sanitize CPU configuration */
call verify_cpu
^ permalink raw reply related [flat|nested] 264+ messages in thread
* [tip: x86/seves] x86/fpu: Move xgetbv()/xsetbv() into a separate header
2020-09-07 13:15 ` Joerg Roedel
(?)
@ 2020-09-10 9:22 ` tip-bot2 for Joerg Roedel
-1 siblings, 0 replies; 264+ messages in thread
From: tip-bot2 for Joerg Roedel @ 2020-09-10 9:22 UTC (permalink / raw)
To: linux-tip-commits; +Cc: Joerg Roedel, Borislav Petkov, x86, LKML
The following commit has been merged into the x86/seves branch of tip:
Commit-ID: 1b4fb8545f2b00f2844c4b7619d64d98440a477c
Gitweb: https://git.kernel.org/tip/1b4fb8545f2b00f2844c4b7619d64d98440a477c
Author: Joerg Roedel <jroedel@suse.de>
AuthorDate: Mon, 07 Sep 2020 15:15:27 +02:00
Committer: Borislav Petkov <bp@suse.de>
CommitterDate: Mon, 07 Sep 2020 19:54:20 +02:00
x86/fpu: Move xgetbv()/xsetbv() into a separate header
The xgetbv() function is needed in the pre-decompression boot code,
but asm/fpu/internal.h can't be included there directly. Doing so
opens the door to include-hell due to various include-magic in
boot/compressed/misc.h.
Avoid that by moving xgetbv()/xsetbv() to a separate header file and
include it instead.
Signed-off-by: Joerg Roedel <jroedel@suse.de>
Signed-off-by: Borislav Petkov <bp@suse.de>
Link: https://lkml.kernel.org/r/20200907131613.12703-27-joro@8bytes.org
---
arch/x86/include/asm/fpu/internal.h | 30 +-------------------------
arch/x86/include/asm/fpu/xcr.h | 34 ++++++++++++++++++++++++++++-
2 files changed, 35 insertions(+), 29 deletions(-)
create mode 100644 arch/x86/include/asm/fpu/xcr.h
diff --git a/arch/x86/include/asm/fpu/internal.h b/arch/x86/include/asm/fpu/internal.h
index 21a8b52..ceeba9f 100644
--- a/arch/x86/include/asm/fpu/internal.h
+++ b/arch/x86/include/asm/fpu/internal.h
@@ -19,6 +19,7 @@
#include <asm/user.h>
#include <asm/fpu/api.h>
#include <asm/fpu/xstate.h>
+#include <asm/fpu/xcr.h>
#include <asm/cpufeature.h>
#include <asm/trace/fpu.h>
@@ -585,33 +586,4 @@ static inline void switch_fpu_finish(struct fpu *new_fpu)
__write_pkru(pkru_val);
}
-/*
- * MXCSR and XCR definitions:
- */
-
-static inline void ldmxcsr(u32 mxcsr)
-{
- asm volatile("ldmxcsr %0" :: "m" (mxcsr));
-}
-
-extern unsigned int mxcsr_feature_mask;
-
-#define XCR_XFEATURE_ENABLED_MASK 0x00000000
-
-static inline u64 xgetbv(u32 index)
-{
- u32 eax, edx;
-
- asm volatile("xgetbv" : "=a" (eax), "=d" (edx) : "c" (index));
- return eax + ((u64)edx << 32);
-}
-
-static inline void xsetbv(u32 index, u64 value)
-{
- u32 eax = value;
- u32 edx = value >> 32;
-
- asm volatile("xsetbv" :: "a" (eax), "d" (edx), "c" (index));
-}
-
#endif /* _ASM_X86_FPU_INTERNAL_H */
diff --git a/arch/x86/include/asm/fpu/xcr.h b/arch/x86/include/asm/fpu/xcr.h
new file mode 100644
index 0000000..1c7ab8d
--- /dev/null
+++ b/arch/x86/include/asm/fpu/xcr.h
@@ -0,0 +1,34 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+#ifndef _ASM_X86_FPU_XCR_H
+#define _ASM_X86_FPU_XCR_H
+
+/*
+ * MXCSR and XCR definitions:
+ */
+
+static inline void ldmxcsr(u32 mxcsr)
+{
+ asm volatile("ldmxcsr %0" :: "m" (mxcsr));
+}
+
+extern unsigned int mxcsr_feature_mask;
+
+#define XCR_XFEATURE_ENABLED_MASK 0x00000000
+
+static inline u64 xgetbv(u32 index)
+{
+ u32 eax, edx;
+
+ asm volatile("xgetbv" : "=a" (eax), "=d" (edx) : "c" (index));
+ return eax + ((u64)edx << 32);
+}
+
+static inline void xsetbv(u32 index, u64 value)
+{
+ u32 eax = value;
+ u32 edx = value >> 32;
+
+ asm volatile("xsetbv" :: "a" (eax), "d" (edx), "c" (index));
+}
+
+#endif /* _ASM_X86_FPU_XCR_H */
^ permalink raw reply related [flat|nested] 264+ messages in thread
* [tip: x86/seves] x86/sev-es: Add CPUID handling to #VC handler
2020-09-07 13:15 ` Joerg Roedel
(?)
@ 2020-09-10 9:22 ` tip-bot2 for Tom Lendacky
-1 siblings, 0 replies; 264+ messages in thread
From: tip-bot2 for Tom Lendacky @ 2020-09-10 9:22 UTC (permalink / raw)
To: linux-tip-commits; +Cc: Tom Lendacky, Joerg Roedel, Borislav Petkov, x86, LKML
The following commit has been merged into the x86/seves branch of tip:
Commit-ID: a7de15d489d956217b47671705ac2218ca50eaae
Gitweb: https://git.kernel.org/tip/a7de15d489d956217b47671705ac2218ca50eaae
Author: Tom Lendacky <thomas.lendacky@amd.com>
AuthorDate: Mon, 07 Sep 2020 15:15:28 +02:00
Committer: Borislav Petkov <bp@suse.de>
CommitterDate: Mon, 07 Sep 2020 20:15:51 +02:00
x86/sev-es: Add CPUID handling to #VC handler
Handle #VC exceptions caused by CPUID instructions. These happen in
early boot code when the KASLR code checks for RDTSC.
Signed-off-by: Tom Lendacky <thomas.lendacky@amd.com>
[ jroedel@suse.de: Adapt to #VC handling framework ]
Co-developed-by: Joerg Roedel <jroedel@suse.de>
Signed-off-by: Joerg Roedel <jroedel@suse.de>
Signed-off-by: Borislav Petkov <bp@suse.de>
Link: https://lkml.kernel.org/r/20200907131613.12703-28-joro@8bytes.org
---
arch/x86/boot/compressed/sev-es.c | 4 +++-
arch/x86/kernel/sev-es-shared.c | 35 ++++++++++++++++++++++++++++++-
2 files changed, 39 insertions(+)
diff --git a/arch/x86/boot/compressed/sev-es.c b/arch/x86/boot/compressed/sev-es.c
index 61504eb..b1790f4 100644
--- a/arch/x86/boot/compressed/sev-es.c
+++ b/arch/x86/boot/compressed/sev-es.c
@@ -16,6 +16,7 @@
#include <asm/trapnr.h>
#include <asm/trap_pf.h>
#include <asm/msr-index.h>
+#include <asm/fpu/xcr.h>
#include <asm/ptrace.h>
#include <asm/svm.h>
@@ -183,6 +184,9 @@ void do_boot_stage2_vc(struct pt_regs *regs, unsigned long exit_code)
case SVM_EXIT_IOIO:
result = vc_handle_ioio(boot_ghcb, &ctxt);
break;
+ case SVM_EXIT_CPUID:
+ result = vc_handle_cpuid(boot_ghcb, &ctxt);
+ break;
default:
result = ES_UNSUPPORTED;
break;
diff --git a/arch/x86/kernel/sev-es-shared.c b/arch/x86/kernel/sev-es-shared.c
index bae7cf2..a6b4191 100644
--- a/arch/x86/kernel/sev-es-shared.c
+++ b/arch/x86/kernel/sev-es-shared.c
@@ -432,3 +432,38 @@ static enum es_result vc_handle_ioio(struct ghcb *ghcb, struct es_em_ctxt *ctxt)
return ret;
}
+
+static enum es_result vc_handle_cpuid(struct ghcb *ghcb,
+ struct es_em_ctxt *ctxt)
+{
+ struct pt_regs *regs = ctxt->regs;
+ u32 cr4 = native_read_cr4();
+ enum es_result ret;
+
+ ghcb_set_rax(ghcb, regs->ax);
+ ghcb_set_rcx(ghcb, regs->cx);
+
+ if (cr4 & X86_CR4_OSXSAVE)
+ /* Safe to read xcr0 */
+ ghcb_set_xcr0(ghcb, xgetbv(XCR_XFEATURE_ENABLED_MASK));
+ else
+ /* xgetbv will cause #GP - use reset value for xcr0 */
+ ghcb_set_xcr0(ghcb, 1);
+
+ ret = sev_es_ghcb_hv_call(ghcb, ctxt, SVM_EXIT_CPUID, 0, 0);
+ if (ret != ES_OK)
+ return ret;
+
+ if (!(ghcb_rax_is_valid(ghcb) &&
+ ghcb_rbx_is_valid(ghcb) &&
+ ghcb_rcx_is_valid(ghcb) &&
+ ghcb_rdx_is_valid(ghcb)))
+ return ES_VMM_ERROR;
+
+ regs->ax = ghcb->save.rax;
+ regs->bx = ghcb->save.rbx;
+ regs->cx = ghcb->save.rcx;
+ regs->dx = ghcb->save.rdx;
+
+ return ES_OK;
+}
^ permalink raw reply related [flat|nested] 264+ messages in thread
* [tip: x86/seves] x86/boot/compressed/64: Unmap GHCB page before booting the kernel
2020-09-07 13:15 ` Joerg Roedel
(?)
@ 2020-09-10 9:22 ` tip-bot2 for Joerg Roedel
-1 siblings, 0 replies; 264+ messages in thread
From: tip-bot2 for Joerg Roedel @ 2020-09-10 9:22 UTC (permalink / raw)
To: linux-tip-commits; +Cc: Joerg Roedel, Borislav Petkov, x86, LKML
The following commit has been merged into the x86/seves branch of tip:
Commit-ID: 69add17a7c1992593a7cf775a66e0256ad4b3ef8
Gitweb: https://git.kernel.org/tip/69add17a7c1992593a7cf775a66e0256ad4b3ef8
Author: Joerg Roedel <jroedel@suse.de>
AuthorDate: Mon, 07 Sep 2020 15:15:25 +02:00
Committer: Borislav Petkov <bp@suse.de>
CommitterDate: Mon, 07 Sep 2020 19:45:26 +02:00
x86/boot/compressed/64: Unmap GHCB page before booting the kernel
Force a page-fault on any further accesses to the GHCB page when they
shouldn't happen anymore. This will catch any bugs where a #VC exception
is raised even though none is expected anymore.
Signed-off-by: Joerg Roedel <jroedel@suse.de>
Signed-off-by: Borislav Petkov <bp@suse.de>
Link: https://lkml.kernel.org/r/20200907131613.12703-25-joro@8bytes.org
---
arch/x86/boot/compressed/ident_map_64.c | 17 +++++++++++++++--
arch/x86/boot/compressed/misc.h | 6 ++++++
arch/x86/boot/compressed/sev-es.c | 14 ++++++++++++++
3 files changed, 35 insertions(+), 2 deletions(-)
diff --git a/arch/x86/boot/compressed/ident_map_64.c b/arch/x86/boot/compressed/ident_map_64.c
index 05742f6..063a60e 100644
--- a/arch/x86/boot/compressed/ident_map_64.c
+++ b/arch/x86/boot/compressed/ident_map_64.c
@@ -298,6 +298,11 @@ int set_page_encrypted(unsigned long address)
return set_clr_page_flags(&mapping_info, address, _PAGE_ENC, 0);
}
+int set_page_non_present(unsigned long address)
+{
+ return set_clr_page_flags(&mapping_info, address, 0, _PAGE_PRESENT);
+}
+
static void do_pf_error(const char *msg, unsigned long error_code,
unsigned long address, unsigned long ip)
{
@@ -316,8 +321,14 @@ static void do_pf_error(const char *msg, unsigned long error_code,
void do_boot_page_fault(struct pt_regs *regs, unsigned long error_code)
{
- unsigned long address = native_read_cr2() & PMD_MASK;
- unsigned long end = address + PMD_SIZE;
+ unsigned long address = native_read_cr2();
+ unsigned long end;
+ bool ghcb_fault;
+
+ ghcb_fault = sev_es_check_ghcb_fault(address);
+
+ address &= PMD_MASK;
+ end = address + PMD_SIZE;
/*
* Check for unexpected error codes. Unexpected are:
@@ -327,6 +338,8 @@ void do_boot_page_fault(struct pt_regs *regs, unsigned long error_code)
*/
if (error_code & (X86_PF_PROT | X86_PF_USER | X86_PF_RSVD))
do_pf_error("Unexpected page-fault:", error_code, address, regs->ip);
+ else if (ghcb_fault)
+ do_pf_error("Page-fault on GHCB page:", error_code, address, regs->ip);
/*
* Error code is sane - now identity map the 2M region around
diff --git a/arch/x86/boot/compressed/misc.h b/arch/x86/boot/compressed/misc.h
index 9995c70..c0e0ffe 100644
--- a/arch/x86/boot/compressed/misc.h
+++ b/arch/x86/boot/compressed/misc.h
@@ -100,6 +100,7 @@ static inline void choose_random_location(unsigned long input,
#ifdef CONFIG_X86_64
extern int set_page_decrypted(unsigned long address);
extern int set_page_encrypted(unsigned long address);
+extern int set_page_non_present(unsigned long address);
extern unsigned char _pgtable[];
#endif
@@ -117,8 +118,13 @@ void set_sev_encryption_mask(void);
#ifdef CONFIG_AMD_MEM_ENCRYPT
void sev_es_shutdown_ghcb(void);
+extern bool sev_es_check_ghcb_fault(unsigned long address);
#else
static inline void sev_es_shutdown_ghcb(void) { }
+static inline bool sev_es_check_ghcb_fault(unsigned long address)
+{
+ return false;
+}
#endif
/* acpi.c */
diff --git a/arch/x86/boot/compressed/sev-es.c b/arch/x86/boot/compressed/sev-es.c
index fa62af7..1e1fab5 100644
--- a/arch/x86/boot/compressed/sev-es.c
+++ b/arch/x86/boot/compressed/sev-es.c
@@ -121,6 +121,20 @@ void sev_es_shutdown_ghcb(void)
*/
if (set_page_encrypted((unsigned long)&boot_ghcb_page))
error("Can't map GHCB page encrypted");
+
+ /*
+ * GHCB page is mapped encrypted again and flushed from the cache.
+ * Mark it non-present now to catch bugs when #VC exceptions trigger
+ * after this point.
+ */
+ if (set_page_non_present((unsigned long)&boot_ghcb_page))
+ error("Can't unmap GHCB page");
+}
+
+bool sev_es_check_ghcb_fault(unsigned long address)
+{
+ /* Check whether the fault was on the GHCB page */
+ return ((address & PAGE_MASK) == (unsigned long)&boot_ghcb_page);
}
void do_boot_stage2_vc(struct pt_regs *regs, unsigned long exit_code)
^ permalink raw reply related [flat|nested] 264+ messages in thread
* [tip: x86/seves] x86/sev-es: Add support for handling IOIO exceptions
2020-09-07 13:15 ` Joerg Roedel
(?)
@ 2020-09-10 9:22 ` tip-bot2 for Tom Lendacky
-1 siblings, 0 replies; 264+ messages in thread
From: tip-bot2 for Tom Lendacky @ 2020-09-10 9:22 UTC (permalink / raw)
To: linux-tip-commits; +Cc: Tom Lendacky, Joerg Roedel, Borislav Petkov, x86, LKML
The following commit has been merged into the x86/seves branch of tip:
Commit-ID: 25189d08e5168c098c307a0eaae5b30c13a331ef
Gitweb: https://git.kernel.org/tip/25189d08e5168c098c307a0eaae5b30c13a331ef
Author: Tom Lendacky <thomas.lendacky@amd.com>
AuthorDate: Mon, 07 Sep 2020 15:15:26 +02:00
Committer: Borislav Petkov <bp@suse.de>
CommitterDate: Mon, 07 Sep 2020 19:45:26 +02:00
x86/sev-es: Add support for handling IOIO exceptions
Add support for decoding and handling #VC exceptions for IOIO events.
[ jroedel@suse.de: Adapted code to #VC handling framework ]
Co-developed-by: Joerg Roedel <jroedel@suse.de>
Signed-off-by: Tom Lendacky <thomas.lendacky@amd.com>
Signed-off-by: Joerg Roedel <jroedel@suse.de>
Signed-off-by: Borislav Petkov <bp@suse.de>
Link: https://lkml.kernel.org/r/20200907131613.12703-26-joro@8bytes.org
---
arch/x86/boot/compressed/sev-es.c | 32 ++++-
arch/x86/kernel/sev-es-shared.c | 214 +++++++++++++++++++++++++++++-
2 files changed, 246 insertions(+)
diff --git a/arch/x86/boot/compressed/sev-es.c b/arch/x86/boot/compressed/sev-es.c
index 1e1fab5..61504eb 100644
--- a/arch/x86/boot/compressed/sev-es.c
+++ b/arch/x86/boot/compressed/sev-es.c
@@ -24,6 +24,35 @@
struct ghcb boot_ghcb_page __aligned(PAGE_SIZE);
struct ghcb *boot_ghcb;
+/*
+ * Copy a version of this function here - insn-eval.c can't be used in
+ * pre-decompression code.
+ */
+static bool insn_has_rep_prefix(struct insn *insn)
+{
+ int i;
+
+ insn_get_prefixes(insn);
+
+ for (i = 0; i < insn->prefixes.nbytes; i++) {
+ insn_byte_t p = insn->prefixes.bytes[i];
+
+ if (p == 0xf2 || p == 0xf3)
+ return true;
+ }
+
+ return false;
+}
+
+/*
+ * Only a dummy for insn_get_seg_base() - Early boot-code is 64bit only and
+ * doesn't use segments.
+ */
+static unsigned long insn_get_seg_base(struct pt_regs *regs, int seg_reg_idx)
+{
+ return 0UL;
+}
+
static inline u64 sev_es_rd_ghcb_msr(void)
{
unsigned long low, high;
@@ -151,6 +180,9 @@ void do_boot_stage2_vc(struct pt_regs *regs, unsigned long exit_code)
goto finish;
switch (exit_code) {
+ case SVM_EXIT_IOIO:
+ result = vc_handle_ioio(boot_ghcb, &ctxt);
+ break;
default:
result = ES_UNSUPPORTED;
break;
diff --git a/arch/x86/kernel/sev-es-shared.c b/arch/x86/kernel/sev-es-shared.c
index 7ac6e6b..bae7cf2 100644
--- a/arch/x86/kernel/sev-es-shared.c
+++ b/arch/x86/kernel/sev-es-shared.c
@@ -218,3 +218,217 @@ static enum es_result vc_insn_string_write(struct es_em_ctxt *ctxt,
return ret;
}
+
+#define IOIO_TYPE_STR BIT(2)
+#define IOIO_TYPE_IN 1
+#define IOIO_TYPE_INS (IOIO_TYPE_IN | IOIO_TYPE_STR)
+#define IOIO_TYPE_OUT 0
+#define IOIO_TYPE_OUTS (IOIO_TYPE_OUT | IOIO_TYPE_STR)
+
+#define IOIO_REP BIT(3)
+
+#define IOIO_ADDR_64 BIT(9)
+#define IOIO_ADDR_32 BIT(8)
+#define IOIO_ADDR_16 BIT(7)
+
+#define IOIO_DATA_32 BIT(6)
+#define IOIO_DATA_16 BIT(5)
+#define IOIO_DATA_8 BIT(4)
+
+#define IOIO_SEG_ES (0 << 10)
+#define IOIO_SEG_DS (3 << 10)
+
+static enum es_result vc_ioio_exitinfo(struct es_em_ctxt *ctxt, u64 *exitinfo)
+{
+ struct insn *insn = &ctxt->insn;
+ *exitinfo = 0;
+
+ switch (insn->opcode.bytes[0]) {
+ /* INS opcodes */
+ case 0x6c:
+ case 0x6d:
+ *exitinfo |= IOIO_TYPE_INS;
+ *exitinfo |= IOIO_SEG_ES;
+ *exitinfo |= (ctxt->regs->dx & 0xffff) << 16;
+ break;
+
+ /* OUTS opcodes */
+ case 0x6e:
+ case 0x6f:
+ *exitinfo |= IOIO_TYPE_OUTS;
+ *exitinfo |= IOIO_SEG_DS;
+ *exitinfo |= (ctxt->regs->dx & 0xffff) << 16;
+ break;
+
+ /* IN immediate opcodes */
+ case 0xe4:
+ case 0xe5:
+ *exitinfo |= IOIO_TYPE_IN;
+ *exitinfo |= (u64)insn->immediate.value << 16;
+ break;
+
+ /* OUT immediate opcodes */
+ case 0xe6:
+ case 0xe7:
+ *exitinfo |= IOIO_TYPE_OUT;
+ *exitinfo |= (u64)insn->immediate.value << 16;
+ break;
+
+ /* IN register opcodes */
+ case 0xec:
+ case 0xed:
+ *exitinfo |= IOIO_TYPE_IN;
+ *exitinfo |= (ctxt->regs->dx & 0xffff) << 16;
+ break;
+
+ /* OUT register opcodes */
+ case 0xee:
+ case 0xef:
+ *exitinfo |= IOIO_TYPE_OUT;
+ *exitinfo |= (ctxt->regs->dx & 0xffff) << 16;
+ break;
+
+ default:
+ return ES_DECODE_FAILED;
+ }
+
+ switch (insn->opcode.bytes[0]) {
+ case 0x6c:
+ case 0x6e:
+ case 0xe4:
+ case 0xe6:
+ case 0xec:
+ case 0xee:
+ /* Single byte opcodes */
+ *exitinfo |= IOIO_DATA_8;
+ break;
+ default:
+ /* Length determined by instruction parsing */
+ *exitinfo |= (insn->opnd_bytes == 2) ? IOIO_DATA_16
+ : IOIO_DATA_32;
+ }
+ switch (insn->addr_bytes) {
+ case 2:
+ *exitinfo |= IOIO_ADDR_16;
+ break;
+ case 4:
+ *exitinfo |= IOIO_ADDR_32;
+ break;
+ case 8:
+ *exitinfo |= IOIO_ADDR_64;
+ break;
+ }
+
+ if (insn_has_rep_prefix(insn))
+ *exitinfo |= IOIO_REP;
+
+ return ES_OK;
+}
+
+static enum es_result vc_handle_ioio(struct ghcb *ghcb, struct es_em_ctxt *ctxt)
+{
+ struct pt_regs *regs = ctxt->regs;
+ u64 exit_info_1, exit_info_2;
+ enum es_result ret;
+
+ ret = vc_ioio_exitinfo(ctxt, &exit_info_1);
+ if (ret != ES_OK)
+ return ret;
+
+ if (exit_info_1 & IOIO_TYPE_STR) {
+
+ /* (REP) INS/OUTS */
+
+ bool df = ((regs->flags & X86_EFLAGS_DF) == X86_EFLAGS_DF);
+ unsigned int io_bytes, exit_bytes;
+ unsigned int ghcb_count, op_count;
+ unsigned long es_base;
+ u64 sw_scratch;
+
+ /*
+ * For the string variants with rep prefix the amount of in/out
+ * operations per #VC exception is limited so that the kernel
+ * has a chance to take interrupts and re-schedule while the
+ * instruction is emulated.
+ */
+ io_bytes = (exit_info_1 >> 4) & 0x7;
+ ghcb_count = sizeof(ghcb->shared_buffer) / io_bytes;
+
+ op_count = (exit_info_1 & IOIO_REP) ? regs->cx : 1;
+ exit_info_2 = min(op_count, ghcb_count);
+ exit_bytes = exit_info_2 * io_bytes;
+
+ es_base = insn_get_seg_base(ctxt->regs, INAT_SEG_REG_ES);
+
+ /* Read bytes of OUTS into the shared buffer */
+ if (!(exit_info_1 & IOIO_TYPE_IN)) {
+ ret = vc_insn_string_read(ctxt,
+ (void *)(es_base + regs->si),
+ ghcb->shared_buffer, io_bytes,
+ exit_info_2, df);
+ if (ret)
+ return ret;
+ }
+
+ /*
+ * Issue an VMGEXIT to the HV to consume the bytes from the
+ * shared buffer or to have it write them into the shared buffer
+ * depending on the instruction: OUTS or INS.
+ */
+ sw_scratch = __pa(ghcb) + offsetof(struct ghcb, shared_buffer);
+ ghcb_set_sw_scratch(ghcb, sw_scratch);
+ ret = sev_es_ghcb_hv_call(ghcb, ctxt, SVM_EXIT_IOIO,
+ exit_info_1, exit_info_2);
+ if (ret != ES_OK)
+ return ret;
+
+ /* Read bytes from shared buffer into the guest's destination. */
+ if (exit_info_1 & IOIO_TYPE_IN) {
+ ret = vc_insn_string_write(ctxt,
+ (void *)(es_base + regs->di),
+ ghcb->shared_buffer, io_bytes,
+ exit_info_2, df);
+ if (ret)
+ return ret;
+
+ if (df)
+ regs->di -= exit_bytes;
+ else
+ regs->di += exit_bytes;
+ } else {
+ if (df)
+ regs->si -= exit_bytes;
+ else
+ regs->si += exit_bytes;
+ }
+
+ if (exit_info_1 & IOIO_REP)
+ regs->cx -= exit_info_2;
+
+ ret = regs->cx ? ES_RETRY : ES_OK;
+
+ } else {
+
+ /* IN/OUT into/from rAX */
+
+ int bits = (exit_info_1 & 0x70) >> 1;
+ u64 rax = 0;
+
+ if (!(exit_info_1 & IOIO_TYPE_IN))
+ rax = lower_bits(regs->ax, bits);
+
+ ghcb_set_rax(ghcb, rax);
+
+ ret = sev_es_ghcb_hv_call(ghcb, ctxt, SVM_EXIT_IOIO, exit_info_1, 0);
+ if (ret != ES_OK)
+ return ret;
+
+ if (exit_info_1 & IOIO_TYPE_IN) {
+ if (!ghcb_rax_is_valid(ghcb))
+ return ES_VMM_ERROR;
+ regs->ax = lower_bits(ghcb->save.rax, bits);
+ }
+ }
+
+ return ret;
+}
^ permalink raw reply related [flat|nested] 264+ messages in thread
* [tip: x86/seves] x86/boot/compressed/64: Check return value of kernel_ident_mapping_init()
2020-09-07 13:15 ` Joerg Roedel
(?)
@ 2020-09-10 9:22 ` tip-bot2 for Joerg Roedel
-1 siblings, 0 replies; 264+ messages in thread
From: tip-bot2 for Joerg Roedel @ 2020-09-10 9:22 UTC (permalink / raw)
To: linux-tip-commits; +Cc: Joerg Roedel, Borislav Petkov, x86, LKML
The following commit has been merged into the x86/seves branch of tip:
Commit-ID: 4b3fdca64a7e8ad90c87cad1fbc6991471f48dc7
Gitweb: https://git.kernel.org/tip/4b3fdca64a7e8ad90c87cad1fbc6991471f48dc7
Author: Joerg Roedel <jroedel@suse.de>
AuthorDate: Mon, 07 Sep 2020 15:15:22 +02:00
Committer: Borislav Petkov <bp@suse.de>
CommitterDate: Mon, 07 Sep 2020 19:45:25 +02:00
x86/boot/compressed/64: Check return value of kernel_ident_mapping_init()
The function can fail to create an identity mapping, check for that
and bail out if it happens.
Signed-off-by: Joerg Roedel <jroedel@suse.de>
Signed-off-by: Borislav Petkov <bp@suse.de>
Link: https://lkml.kernel.org/r/20200907131613.12703-22-joro@8bytes.org
---
arch/x86/boot/compressed/ident_map_64.c | 7 +++++--
1 file changed, 5 insertions(+), 2 deletions(-)
diff --git a/arch/x86/boot/compressed/ident_map_64.c b/arch/x86/boot/compressed/ident_map_64.c
index b4f2a5f..aa91beb 100644
--- a/arch/x86/boot/compressed/ident_map_64.c
+++ b/arch/x86/boot/compressed/ident_map_64.c
@@ -91,6 +91,8 @@ static struct x86_mapping_info mapping_info;
*/
static void add_identity_map(unsigned long start, unsigned long end)
{
+ int ret;
+
/* Align boundary to 2M. */
start = round_down(start, PMD_SIZE);
end = round_up(end, PMD_SIZE);
@@ -98,8 +100,9 @@ static void add_identity_map(unsigned long start, unsigned long end)
return;
/* Build the mapping. */
- kernel_ident_mapping_init(&mapping_info, (pgd_t *)top_level_pgt,
- start, end);
+ ret = kernel_ident_mapping_init(&mapping_info, (pgd_t *)top_level_pgt, start, end);
+ if (ret)
+ error("Error: kernel_ident_mapping_init() failed\n");
}
/* Locates and clears a region for a new top level page table. */
^ permalink raw reply related [flat|nested] 264+ messages in thread
* [tip: x86/seves] x86/boot/compressed/64: Setup a GHCB-based VC Exception handler
2020-09-07 13:15 ` Joerg Roedel
(?)
@ 2020-09-10 9:22 ` tip-bot2 for Joerg Roedel
-1 siblings, 0 replies; 264+ messages in thread
From: tip-bot2 for Joerg Roedel @ 2020-09-10 9:22 UTC (permalink / raw)
To: linux-tip-commits; +Cc: Joerg Roedel, Borislav Petkov, x86, LKML
The following commit has been merged into the x86/seves branch of tip:
Commit-ID: 597cfe48212a3f110ab0f918bf59791f453e65b7
Gitweb: https://git.kernel.org/tip/597cfe48212a3f110ab0f918bf59791f453e65b7
Author: Joerg Roedel <jroedel@suse.de>
AuthorDate: Mon, 07 Sep 2020 15:15:24 +02:00
Committer: Borislav Petkov <bp@suse.de>
CommitterDate: Mon, 07 Sep 2020 19:45:25 +02:00
x86/boot/compressed/64: Setup a GHCB-based VC Exception handler
Install an exception handler for #VC exception that uses a GHCB. Also
add the infrastructure for handling different exit-codes by decoding
the instruction that caused the exception and error handling.
Signed-off-by: Joerg Roedel <jroedel@suse.de>
Signed-off-by: Borislav Petkov <bp@suse.de>
Link: https://lkml.kernel.org/r/20200907131613.12703-24-joro@8bytes.org
---
arch/x86/Kconfig | 1 +-
arch/x86/boot/compressed/Makefile | 5 +-
arch/x86/boot/compressed/idt_64.c | 4 +-
arch/x86/boot/compressed/idt_handlers_64.S | 3 +-
arch/x86/boot/compressed/misc.c | 7 +-
arch/x86/boot/compressed/misc.h | 7 +-
arch/x86/boot/compressed/sev-es.c | 111 ++++++++++++++-
arch/x86/include/asm/sev-es.h | 39 +++++-
arch/x86/include/uapi/asm/svm.h | 1 +-
arch/x86/kernel/sev-es-shared.c | 154 ++++++++++++++++++++-
10 files changed, 331 insertions(+), 1 deletion(-)
diff --git a/arch/x86/Kconfig b/arch/x86/Kconfig
index 7101ac6..8289dd4 100644
--- a/arch/x86/Kconfig
+++ b/arch/x86/Kconfig
@@ -1521,6 +1521,7 @@ config AMD_MEM_ENCRYPT
select DYNAMIC_PHYSICAL_MASK
select ARCH_USE_MEMREMAP_PROT
select ARCH_HAS_FORCE_DMA_UNENCRYPTED
+ select INSTRUCTION_DECODER
help
Say yes to enable support for the encryption of system memory.
This requires an AMD processor that supports Secure Memory
diff --git a/arch/x86/boot/compressed/Makefile b/arch/x86/boot/compressed/Makefile
index 38f4a52..c01236a 100644
--- a/arch/x86/boot/compressed/Makefile
+++ b/arch/x86/boot/compressed/Makefile
@@ -44,6 +44,11 @@ KBUILD_CFLAGS += $(call cc-option,-fmacro-prefix-map=$(srctree)/=)
KBUILD_CFLAGS += -fno-asynchronous-unwind-tables
KBUILD_CFLAGS += -D__DISABLE_EXPORTS
+# sev-es.c indirectly inludes inat-table.h which is generated during
+# compilation and stored in $(objtree). Add the directory to the includes so
+# that the compiler finds it even with out-of-tree builds (make O=/some/path).
+CFLAGS_sev-es.o += -I$(objtree)/arch/x86/lib/
+
KBUILD_AFLAGS := $(KBUILD_CFLAGS) -D__ASSEMBLY__
GCOV_PROFILE := n
UBSAN_SANITIZE :=n
diff --git a/arch/x86/boot/compressed/idt_64.c b/arch/x86/boot/compressed/idt_64.c
index f3ca732..804a502 100644
--- a/arch/x86/boot/compressed/idt_64.c
+++ b/arch/x86/boot/compressed/idt_64.c
@@ -46,5 +46,9 @@ void load_stage2_idt(void)
set_idt_entry(X86_TRAP_PF, boot_page_fault);
+#ifdef CONFIG_AMD_MEM_ENCRYPT
+ set_idt_entry(X86_TRAP_VC, boot_stage2_vc);
+#endif
+
load_boot_idt(&boot_idt_desc);
}
diff --git a/arch/x86/boot/compressed/idt_handlers_64.S b/arch/x86/boot/compressed/idt_handlers_64.S
index 92eb4df..22890e1 100644
--- a/arch/x86/boot/compressed/idt_handlers_64.S
+++ b/arch/x86/boot/compressed/idt_handlers_64.S
@@ -72,5 +72,6 @@ SYM_FUNC_END(\name)
EXCEPTION_HANDLER boot_page_fault do_boot_page_fault error_code=1
#ifdef CONFIG_AMD_MEM_ENCRYPT
-EXCEPTION_HANDLER boot_stage1_vc do_vc_no_ghcb error_code=1
+EXCEPTION_HANDLER boot_stage1_vc do_vc_no_ghcb error_code=1
+EXCEPTION_HANDLER boot_stage2_vc do_boot_stage2_vc error_code=1
#endif
diff --git a/arch/x86/boot/compressed/misc.c b/arch/x86/boot/compressed/misc.c
index e478e40..267e7f9 100644
--- a/arch/x86/boot/compressed/misc.c
+++ b/arch/x86/boot/compressed/misc.c
@@ -442,6 +442,13 @@ asmlinkage __visible void *extract_kernel(void *rmode, memptr heap,
parse_elf(output);
handle_relocations(output, output_len, virt_addr);
debug_putstr("done.\nBooting the kernel.\n");
+
+ /*
+ * Flush GHCB from cache and map it encrypted again when running as
+ * SEV-ES guest.
+ */
+ sev_es_shutdown_ghcb();
+
return output;
}
diff --git a/arch/x86/boot/compressed/misc.h b/arch/x86/boot/compressed/misc.h
index 01c0fb3..9995c70 100644
--- a/arch/x86/boot/compressed/misc.h
+++ b/arch/x86/boot/compressed/misc.h
@@ -115,6 +115,12 @@ static inline void console_init(void)
void set_sev_encryption_mask(void);
+#ifdef CONFIG_AMD_MEM_ENCRYPT
+void sev_es_shutdown_ghcb(void);
+#else
+static inline void sev_es_shutdown_ghcb(void) { }
+#endif
+
/* acpi.c */
#ifdef CONFIG_ACPI
acpi_physical_address get_rsdp_addr(void);
@@ -144,5 +150,6 @@ extern struct desc_ptr boot_idt_desc;
/* IDT Entry Points */
void boot_page_fault(void);
void boot_stage1_vc(void);
+void boot_stage2_vc(void);
#endif /* BOOT_COMPRESSED_MISC_H */
diff --git a/arch/x86/boot/compressed/sev-es.c b/arch/x86/boot/compressed/sev-es.c
index 99c3bcd..fa62af7 100644
--- a/arch/x86/boot/compressed/sev-es.c
+++ b/arch/x86/boot/compressed/sev-es.c
@@ -13,10 +13,17 @@
#include "misc.h"
#include <asm/sev-es.h>
+#include <asm/trapnr.h>
+#include <asm/trap_pf.h>
#include <asm/msr-index.h>
#include <asm/ptrace.h>
#include <asm/svm.h>
+#include "error.h"
+
+struct ghcb boot_ghcb_page __aligned(PAGE_SIZE);
+struct ghcb *boot_ghcb;
+
static inline u64 sev_es_rd_ghcb_msr(void)
{
unsigned long low, high;
@@ -38,8 +45,112 @@ static inline void sev_es_wr_ghcb_msr(u64 val)
"a"(low), "d" (high) : "memory");
}
+static enum es_result vc_decode_insn(struct es_em_ctxt *ctxt)
+{
+ char buffer[MAX_INSN_SIZE];
+ enum es_result ret;
+
+ memcpy(buffer, (unsigned char *)ctxt->regs->ip, MAX_INSN_SIZE);
+
+ insn_init(&ctxt->insn, buffer, MAX_INSN_SIZE, 1);
+ insn_get_length(&ctxt->insn);
+
+ ret = ctxt->insn.immediate.got ? ES_OK : ES_DECODE_FAILED;
+
+ return ret;
+}
+
+static enum es_result vc_write_mem(struct es_em_ctxt *ctxt,
+ void *dst, char *buf, size_t size)
+{
+ memcpy(dst, buf, size);
+
+ return ES_OK;
+}
+
+static enum es_result vc_read_mem(struct es_em_ctxt *ctxt,
+ void *src, char *buf, size_t size)
+{
+ memcpy(buf, src, size);
+
+ return ES_OK;
+}
+
#undef __init
+#undef __pa
#define __init
+#define __pa(x) ((unsigned long)(x))
+
+#define __BOOT_COMPRESSED
+
+/* Basic instruction decoding support needed */
+#include "../../lib/inat.c"
+#include "../../lib/insn.c"
/* Include code for early handlers */
#include "../../kernel/sev-es-shared.c"
+
+static bool early_setup_sev_es(void)
+{
+ if (!sev_es_negotiate_protocol())
+ sev_es_terminate(GHCB_SEV_ES_REASON_PROTOCOL_UNSUPPORTED);
+
+ if (set_page_decrypted((unsigned long)&boot_ghcb_page))
+ return false;
+
+ /* Page is now mapped decrypted, clear it */
+ memset(&boot_ghcb_page, 0, sizeof(boot_ghcb_page));
+
+ boot_ghcb = &boot_ghcb_page;
+
+ /* Initialize lookup tables for the instruction decoder */
+ inat_init_tables();
+
+ return true;
+}
+
+void sev_es_shutdown_ghcb(void)
+{
+ if (!boot_ghcb)
+ return;
+
+ /*
+ * GHCB Page must be flushed from the cache and mapped encrypted again.
+ * Otherwise the running kernel will see strange cache effects when
+ * trying to use that page.
+ */
+ if (set_page_encrypted((unsigned long)&boot_ghcb_page))
+ error("Can't map GHCB page encrypted");
+}
+
+void do_boot_stage2_vc(struct pt_regs *regs, unsigned long exit_code)
+{
+ struct es_em_ctxt ctxt;
+ enum es_result result;
+
+ if (!boot_ghcb && !early_setup_sev_es())
+ sev_es_terminate(GHCB_SEV_ES_REASON_GENERAL_REQUEST);
+
+ vc_ghcb_invalidate(boot_ghcb);
+ result = vc_init_em_ctxt(&ctxt, regs, exit_code);
+ if (result != ES_OK)
+ goto finish;
+
+ switch (exit_code) {
+ default:
+ result = ES_UNSUPPORTED;
+ break;
+ }
+
+finish:
+ if (result == ES_OK) {
+ vc_finish_insn(&ctxt);
+ } else if (result != ES_RETRY) {
+ /*
+ * For now, just halt the machine. That makes debugging easier,
+ * later we just call sev_es_terminate() here.
+ */
+ while (true)
+ asm volatile("hlt\n");
+ }
+}
diff --git a/arch/x86/include/asm/sev-es.h b/arch/x86/include/asm/sev-es.h
index 48a4403..6dc5244 100644
--- a/arch/x86/include/asm/sev-es.h
+++ b/arch/x86/include/asm/sev-es.h
@@ -9,7 +9,14 @@
#define __ASM_ENCRYPTED_STATE_H
#include <linux/types.h>
+#include <asm/insn.h>
+#define GHCB_SEV_INFO 0x001UL
+#define GHCB_SEV_INFO_REQ 0x002UL
+#define GHCB_INFO(v) ((v) & 0xfffUL)
+#define GHCB_PROTO_MAX(v) (((v) >> 48) & 0xffffUL)
+#define GHCB_PROTO_MIN(v) (((v) >> 32) & 0xffffUL)
+#define GHCB_PROTO_OUR 0x0001UL
#define GHCB_SEV_CPUID_REQ 0x004UL
#define GHCB_CPUID_REQ_EAX 0
#define GHCB_CPUID_REQ_EBX 1
@@ -19,12 +26,44 @@
(((unsigned long)reg & 3) << 30) | \
(((unsigned long)fn) << 32))
+#define GHCB_PROTOCOL_MAX 0x0001UL
+#define GHCB_DEFAULT_USAGE 0x0000UL
+
#define GHCB_SEV_CPUID_RESP 0x005UL
#define GHCB_SEV_TERMINATE 0x100UL
+#define GHCB_SEV_TERMINATE_REASON(reason_set, reason_val) \
+ (((((u64)reason_set) & 0x7) << 12) | \
+ ((((u64)reason_val) & 0xff) << 16))
+#define GHCB_SEV_ES_REASON_GENERAL_REQUEST 0
+#define GHCB_SEV_ES_REASON_PROTOCOL_UNSUPPORTED 1
#define GHCB_SEV_GHCB_RESP_CODE(v) ((v) & 0xfff)
#define VMGEXIT() { asm volatile("rep; vmmcall\n\r"); }
+enum es_result {
+ ES_OK, /* All good */
+ ES_UNSUPPORTED, /* Requested operation not supported */
+ ES_VMM_ERROR, /* Unexpected state from the VMM */
+ ES_DECODE_FAILED, /* Instruction decoding failed */
+ ES_EXCEPTION, /* Instruction caused exception */
+ ES_RETRY, /* Retry instruction emulation */
+};
+
+struct es_fault_info {
+ unsigned long vector;
+ unsigned long error_code;
+ unsigned long cr2;
+};
+
+struct pt_regs;
+
+/* ES instruction emulation context */
+struct es_em_ctxt {
+ struct pt_regs *regs;
+ struct insn insn;
+ struct es_fault_info fi;
+};
+
void do_vc_no_ghcb(struct pt_regs *regs, unsigned long exit_code);
static inline u64 lower_bits(u64 val, unsigned int bits)
diff --git a/arch/x86/include/uapi/asm/svm.h b/arch/x86/include/uapi/asm/svm.h
index 2e8a30f..c68d161 100644
--- a/arch/x86/include/uapi/asm/svm.h
+++ b/arch/x86/include/uapi/asm/svm.h
@@ -29,6 +29,7 @@
#define SVM_EXIT_WRITE_DR6 0x036
#define SVM_EXIT_WRITE_DR7 0x037
#define SVM_EXIT_EXCP_BASE 0x040
+#define SVM_EXIT_LAST_EXCP 0x05f
#define SVM_EXIT_INTR 0x060
#define SVM_EXIT_NMI 0x061
#define SVM_EXIT_SMI 0x062
diff --git a/arch/x86/kernel/sev-es-shared.c b/arch/x86/kernel/sev-es-shared.c
index 0bea323..7ac6e6b 100644
--- a/arch/x86/kernel/sev-es-shared.c
+++ b/arch/x86/kernel/sev-es-shared.c
@@ -9,6 +9,118 @@
* and is included directly into both code-bases.
*/
+static void sev_es_terminate(unsigned int reason)
+{
+ u64 val = GHCB_SEV_TERMINATE;
+
+ /*
+ * Tell the hypervisor what went wrong - only reason-set 0 is
+ * currently supported.
+ */
+ val |= GHCB_SEV_TERMINATE_REASON(0, reason);
+
+ /* Request Guest Termination from Hypvervisor */
+ sev_es_wr_ghcb_msr(val);
+ VMGEXIT();
+
+ while (true)
+ asm volatile("hlt\n" : : : "memory");
+}
+
+static bool sev_es_negotiate_protocol(void)
+{
+ u64 val;
+
+ /* Do the GHCB protocol version negotiation */
+ sev_es_wr_ghcb_msr(GHCB_SEV_INFO_REQ);
+ VMGEXIT();
+ val = sev_es_rd_ghcb_msr();
+
+ if (GHCB_INFO(val) != GHCB_SEV_INFO)
+ return false;
+
+ if (GHCB_PROTO_MAX(val) < GHCB_PROTO_OUR ||
+ GHCB_PROTO_MIN(val) > GHCB_PROTO_OUR)
+ return false;
+
+ return true;
+}
+
+static void vc_ghcb_invalidate(struct ghcb *ghcb)
+{
+ memset(ghcb->save.valid_bitmap, 0, sizeof(ghcb->save.valid_bitmap));
+}
+
+static bool vc_decoding_needed(unsigned long exit_code)
+{
+ /* Exceptions don't require to decode the instruction */
+ return !(exit_code >= SVM_EXIT_EXCP_BASE &&
+ exit_code <= SVM_EXIT_LAST_EXCP);
+}
+
+static enum es_result vc_init_em_ctxt(struct es_em_ctxt *ctxt,
+ struct pt_regs *regs,
+ unsigned long exit_code)
+{
+ enum es_result ret = ES_OK;
+
+ memset(ctxt, 0, sizeof(*ctxt));
+ ctxt->regs = regs;
+
+ if (vc_decoding_needed(exit_code))
+ ret = vc_decode_insn(ctxt);
+
+ return ret;
+}
+
+static void vc_finish_insn(struct es_em_ctxt *ctxt)
+{
+ ctxt->regs->ip += ctxt->insn.length;
+}
+
+static enum es_result sev_es_ghcb_hv_call(struct ghcb *ghcb,
+ struct es_em_ctxt *ctxt,
+ u64 exit_code, u64 exit_info_1,
+ u64 exit_info_2)
+{
+ enum es_result ret;
+
+ /* Fill in protocol and format specifiers */
+ ghcb->protocol_version = GHCB_PROTOCOL_MAX;
+ ghcb->ghcb_usage = GHCB_DEFAULT_USAGE;
+
+ ghcb_set_sw_exit_code(ghcb, exit_code);
+ ghcb_set_sw_exit_info_1(ghcb, exit_info_1);
+ ghcb_set_sw_exit_info_2(ghcb, exit_info_2);
+
+ sev_es_wr_ghcb_msr(__pa(ghcb));
+ VMGEXIT();
+
+ if ((ghcb->save.sw_exit_info_1 & 0xffffffff) == 1) {
+ u64 info = ghcb->save.sw_exit_info_2;
+ unsigned long v;
+
+ info = ghcb->save.sw_exit_info_2;
+ v = info & SVM_EVTINJ_VEC_MASK;
+
+ /* Check if exception information from hypervisor is sane. */
+ if ((info & SVM_EVTINJ_VALID) &&
+ ((v == X86_TRAP_GP) || (v == X86_TRAP_UD)) &&
+ ((info & SVM_EVTINJ_TYPE_MASK) == SVM_EVTINJ_TYPE_EXEPT)) {
+ ctxt->fi.vector = v;
+ if (info & SVM_EVTINJ_VALID_ERR)
+ ctxt->fi.error_code = info >> 32;
+ ret = ES_EXCEPTION;
+ } else {
+ ret = ES_VMM_ERROR;
+ }
+ } else {
+ ret = ES_OK;
+ }
+
+ return ret;
+}
+
/*
* Boot VC Handler - This is the first VC handler during boot, there is no GHCB
* page yet, so it only supports the MSR based communication with the
@@ -64,3 +176,45 @@ fail:
while (true)
asm volatile("hlt\n");
}
+
+static enum es_result vc_insn_string_read(struct es_em_ctxt *ctxt,
+ void *src, char *buf,
+ unsigned int data_size,
+ unsigned int count,
+ bool backwards)
+{
+ int i, b = backwards ? -1 : 1;
+ enum es_result ret = ES_OK;
+
+ for (i = 0; i < count; i++) {
+ void *s = src + (i * data_size * b);
+ char *d = buf + (i * data_size);
+
+ ret = vc_read_mem(ctxt, s, d, data_size);
+ if (ret != ES_OK)
+ break;
+ }
+
+ return ret;
+}
+
+static enum es_result vc_insn_string_write(struct es_em_ctxt *ctxt,
+ void *dst, char *buf,
+ unsigned int data_size,
+ unsigned int count,
+ bool backwards)
+{
+ int i, s = backwards ? -1 : 1;
+ enum es_result ret = ES_OK;
+
+ for (i = 0; i < count; i++) {
+ void *d = dst + (i * data_size * s);
+ char *b = buf + (i * data_size);
+
+ ret = vc_write_mem(ctxt, d, b, data_size);
+ if (ret != ES_OK)
+ break;
+ }
+
+ return ret;
+}
^ permalink raw reply related [flat|nested] 264+ messages in thread
* [tip: x86/seves] x86/boot/compressed/64: Add set_page_en/decrypted() helpers
2020-09-07 13:15 ` Joerg Roedel
(?)
@ 2020-09-10 9:22 ` tip-bot2 for Joerg Roedel
-1 siblings, 0 replies; 264+ messages in thread
From: tip-bot2 for Joerg Roedel @ 2020-09-10 9:22 UTC (permalink / raw)
To: linux-tip-commits; +Cc: Joerg Roedel, Borislav Petkov, x86, LKML
The following commit has been merged into the x86/seves branch of tip:
Commit-ID: c81d60029a1393183d2125fcb4b64831629b8864
Gitweb: https://git.kernel.org/tip/c81d60029a1393183d2125fcb4b64831629b8864
Author: Joerg Roedel <jroedel@suse.de>
AuthorDate: Mon, 07 Sep 2020 15:15:23 +02:00
Committer: Borislav Petkov <bp@suse.de>
CommitterDate: Mon, 07 Sep 2020 19:45:25 +02:00
x86/boot/compressed/64: Add set_page_en/decrypted() helpers
The functions are needed to map the GHCB for SEV-ES guests. The GHCB
is used for communication with the hypervisor, so its content must not
be encrypted. After the GHCB is not needed anymore it must be mapped
encrypted again so that the running kernel image can safely re-use the
memory.
Signed-off-by: Joerg Roedel <jroedel@suse.de>
Signed-off-by: Borislav Petkov <bp@suse.de>
Link: https://lkml.kernel.org/r/20200907131613.12703-23-joro@8bytes.org
---
arch/x86/boot/compressed/ident_map_64.c | 133 +++++++++++++++++++++++-
arch/x86/boot/compressed/misc.h | 2 +-
2 files changed, 135 insertions(+)
diff --git a/arch/x86/boot/compressed/ident_map_64.c b/arch/x86/boot/compressed/ident_map_64.c
index aa91beb..05742f6 100644
--- a/arch/x86/boot/compressed/ident_map_64.c
+++ b/arch/x86/boot/compressed/ident_map_64.c
@@ -24,6 +24,7 @@
/* These actually do the work of building the kernel identity maps. */
#include <linux/pgtable.h>
+#include <asm/cmpxchg.h>
#include <asm/trap_pf.h>
#include <asm/trapnr.h>
#include <asm/init.h>
@@ -165,6 +166,138 @@ void finalize_identity_maps(void)
write_cr3(top_level_pgt);
}
+static pte_t *split_large_pmd(struct x86_mapping_info *info,
+ pmd_t *pmdp, unsigned long __address)
+{
+ unsigned long page_flags;
+ unsigned long address;
+ pte_t *pte;
+ pmd_t pmd;
+ int i;
+
+ pte = (pte_t *)info->alloc_pgt_page(info->context);
+ if (!pte)
+ return NULL;
+
+ address = __address & PMD_MASK;
+ /* No large page - clear PSE flag */
+ page_flags = info->page_flag & ~_PAGE_PSE;
+
+ /* Populate the PTEs */
+ for (i = 0; i < PTRS_PER_PMD; i++) {
+ set_pte(&pte[i], __pte(address | page_flags));
+ address += PAGE_SIZE;
+ }
+
+ /*
+ * Ideally we need to clear the large PMD first and do a TLB
+ * flush before we write the new PMD. But the 2M range of the
+ * PMD might contain the code we execute and/or the stack
+ * we are on, so we can't do that. But that should be safe here
+ * because we are going from large to small mappings and we are
+ * also the only user of the page-table, so there is no chance
+ * of a TLB multihit.
+ */
+ pmd = __pmd((unsigned long)pte | info->kernpg_flag);
+ set_pmd(pmdp, pmd);
+ /* Flush TLB to establish the new PMD */
+ write_cr3(top_level_pgt);
+
+ return pte + pte_index(__address);
+}
+
+static void clflush_page(unsigned long address)
+{
+ unsigned int flush_size;
+ char *cl, *start, *end;
+
+ /*
+ * Hardcode cl-size to 64 - CPUID can't be used here because that might
+ * cause another #VC exception and the GHCB is not ready to use yet.
+ */
+ flush_size = 64;
+ start = (char *)(address & PAGE_MASK);
+ end = start + PAGE_SIZE;
+
+ /*
+ * First make sure there are no pending writes on the cache-lines to
+ * flush.
+ */
+ asm volatile("mfence" : : : "memory");
+
+ for (cl = start; cl != end; cl += flush_size)
+ clflush(cl);
+}
+
+static int set_clr_page_flags(struct x86_mapping_info *info,
+ unsigned long address,
+ pteval_t set, pteval_t clr)
+{
+ pgd_t *pgdp = (pgd_t *)top_level_pgt;
+ p4d_t *p4dp;
+ pud_t *pudp;
+ pmd_t *pmdp;
+ pte_t *ptep, pte;
+
+ /*
+ * First make sure there is a PMD mapping for 'address'.
+ * It should already exist, but keep things generic.
+ *
+ * To map the page just read from it and fault it in if there is no
+ * mapping yet. add_identity_map() can't be called here because that
+ * would unconditionally map the address on PMD level, destroying any
+ * PTE-level mappings that might already exist. Use assembly here so
+ * the access won't be optimized away.
+ */
+ asm volatile("mov %[address], %%r9"
+ :: [address] "g" (*(unsigned long *)address)
+ : "r9", "memory");
+
+ /*
+ * The page is mapped at least with PMD size - so skip checks and walk
+ * directly to the PMD.
+ */
+ p4dp = p4d_offset(pgdp, address);
+ pudp = pud_offset(p4dp, address);
+ pmdp = pmd_offset(pudp, address);
+
+ if (pmd_large(*pmdp))
+ ptep = split_large_pmd(info, pmdp, address);
+ else
+ ptep = pte_offset_kernel(pmdp, address);
+
+ if (!ptep)
+ return -ENOMEM;
+
+ /*
+ * Changing encryption attributes of a page requires to flush it from
+ * the caches.
+ */
+ if ((set | clr) & _PAGE_ENC)
+ clflush_page(address);
+
+ /* Update PTE */
+ pte = *ptep;
+ pte = pte_set_flags(pte, set);
+ pte = pte_clear_flags(pte, clr);
+ set_pte(ptep, pte);
+
+ /* Flush TLB after changing encryption attribute */
+ write_cr3(top_level_pgt);
+
+ return 0;
+}
+
+int set_page_decrypted(unsigned long address)
+{
+ return set_clr_page_flags(&mapping_info, address, 0, _PAGE_ENC);
+}
+
+int set_page_encrypted(unsigned long address)
+{
+ return set_clr_page_flags(&mapping_info, address, _PAGE_ENC, 0);
+}
+
static void do_pf_error(const char *msg, unsigned long error_code,
unsigned long address, unsigned long ip)
{
diff --git a/arch/x86/boot/compressed/misc.h b/arch/x86/boot/compressed/misc.h
index eaa8b45..01c0fb3 100644
--- a/arch/x86/boot/compressed/misc.h
+++ b/arch/x86/boot/compressed/misc.h
@@ -98,6 +98,8 @@ static inline void choose_random_location(unsigned long input,
#endif
#ifdef CONFIG_X86_64
+extern int set_page_decrypted(unsigned long address);
+extern int set_page_encrypted(unsigned long address);
extern unsigned char _pgtable[];
#endif
^ permalink raw reply related [flat|nested] 264+ messages in thread
* [tip: x86/seves] x86/boot/compressed/64: Add stage1 #VC handler
2020-09-07 13:15 ` Joerg Roedel
(?)
(?)
@ 2020-09-10 9:22 ` tip-bot2 for Joerg Roedel
-1 siblings, 0 replies; 264+ messages in thread
From: tip-bot2 for Joerg Roedel @ 2020-09-10 9:22 UTC (permalink / raw)
To: linux-tip-commits; +Cc: Joerg Roedel, Borislav Petkov, x86, LKML
The following commit has been merged into the x86/seves branch of tip:
Commit-ID: 29dcc60f6a19fb0aaee97bd1ae2ed8a7dc6f0cfe
Gitweb: https://git.kernel.org/tip/29dcc60f6a19fb0aaee97bd1ae2ed8a7dc6f0cfe
Author: Joerg Roedel <jroedel@suse.de>
AuthorDate: Mon, 07 Sep 2020 15:15:20 +02:00
Committer: Borislav Petkov <bp@suse.de>
CommitterDate: Mon, 07 Sep 2020 19:45:25 +02:00
x86/boot/compressed/64: Add stage1 #VC handler
Add the first handler for #VC exceptions. At stage 1 there is no GHCB
yet because the kernel might still be running on the EFI page table.
The stage 1 handler is limited to the MSR-based protocol to talk to the
hypervisor and can only support CPUID exit-codes, but that is enough to
get to stage 2.
[ bp: Zap superfluous newlines after rd/wrmsr instruction mnemonics. ]
Signed-off-by: Joerg Roedel <jroedel@suse.de>
Signed-off-by: Borislav Petkov <bp@suse.de>
Link: https://lkml.kernel.org/r/20200907131613.12703-20-joro@8bytes.org
---
arch/x86/boot/compressed/Makefile | 1 +-
arch/x86/boot/compressed/idt_64.c | 4 +-
arch/x86/boot/compressed/idt_handlers_64.S | 4 +-
arch/x86/boot/compressed/misc.h | 1 +-
arch/x86/boot/compressed/sev-es.c | 45 ++++++++++++++-
arch/x86/include/asm/msr-index.h | 1 +-
arch/x86/include/asm/sev-es.h | 37 ++++++++++++-
arch/x86/include/asm/trapnr.h | 1 +-
arch/x86/kernel/sev-es-shared.c | 66 +++++++++++++++++++++-
9 files changed, 160 insertions(+)
create mode 100644 arch/x86/boot/compressed/sev-es.c
create mode 100644 arch/x86/include/asm/sev-es.h
create mode 100644 arch/x86/kernel/sev-es-shared.c
diff --git a/arch/x86/boot/compressed/Makefile b/arch/x86/boot/compressed/Makefile
index e7f3eba..38f4a52 100644
--- a/arch/x86/boot/compressed/Makefile
+++ b/arch/x86/boot/compressed/Makefile
@@ -88,6 +88,7 @@ ifdef CONFIG_X86_64
vmlinux-objs-y += $(obj)/idt_64.o $(obj)/idt_handlers_64.o
vmlinux-objs-y += $(obj)/mem_encrypt.o
vmlinux-objs-y += $(obj)/pgtable_64.o
+ vmlinux-objs-$(CONFIG_AMD_MEM_ENCRYPT) += $(obj)/sev-es.o
endif
vmlinux-objs-$(CONFIG_ACPI) += $(obj)/acpi.o
diff --git a/arch/x86/boot/compressed/idt_64.c b/arch/x86/boot/compressed/idt_64.c
index 5f08309..f3ca732 100644
--- a/arch/x86/boot/compressed/idt_64.c
+++ b/arch/x86/boot/compressed/idt_64.c
@@ -32,6 +32,10 @@ void load_stage1_idt(void)
{
boot_idt_desc.address = (unsigned long)boot_idt;
+
+ if (IS_ENABLED(CONFIG_AMD_MEM_ENCRYPT))
+ set_idt_entry(X86_TRAP_VC, boot_stage1_vc);
+
load_boot_idt(&boot_idt_desc);
}
diff --git a/arch/x86/boot/compressed/idt_handlers_64.S b/arch/x86/boot/compressed/idt_handlers_64.S
index b20e575..92eb4df 100644
--- a/arch/x86/boot/compressed/idt_handlers_64.S
+++ b/arch/x86/boot/compressed/idt_handlers_64.S
@@ -70,3 +70,7 @@ SYM_FUNC_END(\name)
.code64
EXCEPTION_HANDLER boot_page_fault do_boot_page_fault error_code=1
+
+#ifdef CONFIG_AMD_MEM_ENCRYPT
+EXCEPTION_HANDLER boot_stage1_vc do_vc_no_ghcb error_code=1
+#endif
diff --git a/arch/x86/boot/compressed/misc.h b/arch/x86/boot/compressed/misc.h
index 9840c82..eaa8b45 100644
--- a/arch/x86/boot/compressed/misc.h
+++ b/arch/x86/boot/compressed/misc.h
@@ -141,5 +141,6 @@ extern struct desc_ptr boot_idt_desc;
/* IDT Entry Points */
void boot_page_fault(void);
+void boot_stage1_vc(void);
#endif /* BOOT_COMPRESSED_MISC_H */
diff --git a/arch/x86/boot/compressed/sev-es.c b/arch/x86/boot/compressed/sev-es.c
new file mode 100644
index 0000000..99c3bcd
--- /dev/null
+++ b/arch/x86/boot/compressed/sev-es.c
@@ -0,0 +1,45 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * AMD Encrypted Register State Support
+ *
+ * Author: Joerg Roedel <jroedel@suse.de>
+ */
+
+/*
+ * misc.h needs to be first because it knows how to include the other kernel
+ * headers in the pre-decompression code in a way that does not break
+ * compilation.
+ */
+#include "misc.h"
+
+#include <asm/sev-es.h>
+#include <asm/msr-index.h>
+#include <asm/ptrace.h>
+#include <asm/svm.h>
+
+static inline u64 sev_es_rd_ghcb_msr(void)
+{
+ unsigned long low, high;
+
+ asm volatile("rdmsr" : "=a" (low), "=d" (high) :
+ "c" (MSR_AMD64_SEV_ES_GHCB));
+
+ return ((high << 32) | low);
+}
+
+static inline void sev_es_wr_ghcb_msr(u64 val)
+{
+ u32 low, high;
+
+ low = val & 0xffffffffUL;
+ high = val >> 32;
+
+ asm volatile("wrmsr" : : "c" (MSR_AMD64_SEV_ES_GHCB),
+ "a"(low), "d" (high) : "memory");
+}
+
+#undef __init
+#define __init
+
+/* Include code for early handlers */
+#include "../../kernel/sev-es-shared.c"
diff --git a/arch/x86/include/asm/msr-index.h b/arch/x86/include/asm/msr-index.h
index 2859ee4..da34fdb 100644
--- a/arch/x86/include/asm/msr-index.h
+++ b/arch/x86/include/asm/msr-index.h
@@ -466,6 +466,7 @@
#define MSR_AMD64_IBSBRTARGET 0xc001103b
#define MSR_AMD64_IBSOPDATA4 0xc001103d
#define MSR_AMD64_IBS_REG_COUNT_MAX 8 /* includes MSR_AMD64_IBSBRTARGET */
+#define MSR_AMD64_SEV_ES_GHCB 0xc0010130
#define MSR_AMD64_SEV 0xc0010131
#define MSR_AMD64_SEV_ENABLED_BIT 0
#define MSR_AMD64_SEV_ENABLED BIT_ULL(MSR_AMD64_SEV_ENABLED_BIT)
diff --git a/arch/x86/include/asm/sev-es.h b/arch/x86/include/asm/sev-es.h
new file mode 100644
index 0000000..48a4403
--- /dev/null
+++ b/arch/x86/include/asm/sev-es.h
@@ -0,0 +1,37 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/*
+ * AMD Encrypted Register State Support
+ *
+ * Author: Joerg Roedel <jroedel@suse.de>
+ */
+
+#ifndef __ASM_ENCRYPTED_STATE_H
+#define __ASM_ENCRYPTED_STATE_H
+
+#include <linux/types.h>
+
+#define GHCB_SEV_CPUID_REQ 0x004UL
+#define GHCB_CPUID_REQ_EAX 0
+#define GHCB_CPUID_REQ_EBX 1
+#define GHCB_CPUID_REQ_ECX 2
+#define GHCB_CPUID_REQ_EDX 3
+#define GHCB_CPUID_REQ(fn, reg) (GHCB_SEV_CPUID_REQ | \
+ (((unsigned long)reg & 3) << 30) | \
+ (((unsigned long)fn) << 32))
+
+#define GHCB_SEV_CPUID_RESP 0x005UL
+#define GHCB_SEV_TERMINATE 0x100UL
+
+#define GHCB_SEV_GHCB_RESP_CODE(v) ((v) & 0xfff)
+#define VMGEXIT() { asm volatile("rep; vmmcall\n\r"); }
+
+void do_vc_no_ghcb(struct pt_regs *regs, unsigned long exit_code);
+
+static inline u64 lower_bits(u64 val, unsigned int bits)
+{
+ u64 mask = (1ULL << bits) - 1;
+
+ return (val & mask);
+}
+
+#endif
diff --git a/arch/x86/include/asm/trapnr.h b/arch/x86/include/asm/trapnr.h
index 082f456..f5d2325 100644
--- a/arch/x86/include/asm/trapnr.h
+++ b/arch/x86/include/asm/trapnr.h
@@ -26,6 +26,7 @@
#define X86_TRAP_XF 19 /* SIMD Floating-Point Exception */
#define X86_TRAP_VE 20 /* Virtualization Exception */
#define X86_TRAP_CP 21 /* Control Protection Exception */
+#define X86_TRAP_VC 29 /* VMM Communication Exception */
#define X86_TRAP_IRET 32 /* IRET Exception */
#endif
diff --git a/arch/x86/kernel/sev-es-shared.c b/arch/x86/kernel/sev-es-shared.c
new file mode 100644
index 0000000..0bea323
--- /dev/null
+++ b/arch/x86/kernel/sev-es-shared.c
@@ -0,0 +1,66 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * AMD Encrypted Register State Support
+ *
+ * Author: Joerg Roedel <jroedel@suse.de>
+ *
+ * This file is not compiled stand-alone. It contains code shared
+ * between the pre-decompression boot code and the running Linux kernel
+ * and is included directly into both code-bases.
+ */
+
+/*
+ * Boot VC Handler - This is the first VC handler during boot, there is no GHCB
+ * page yet, so it only supports the MSR based communication with the
+ * hypervisor and only the CPUID exit-code.
+ */
+void __init do_vc_no_ghcb(struct pt_regs *regs, unsigned long exit_code)
+{
+ unsigned int fn = lower_bits(regs->ax, 32);
+ unsigned long val;
+
+ /* Only CPUID is supported via MSR protocol */
+ if (exit_code != SVM_EXIT_CPUID)
+ goto fail;
+
+ sev_es_wr_ghcb_msr(GHCB_CPUID_REQ(fn, GHCB_CPUID_REQ_EAX));
+ VMGEXIT();
+ val = sev_es_rd_ghcb_msr();
+ if (GHCB_SEV_GHCB_RESP_CODE(val) != GHCB_SEV_CPUID_RESP)
+ goto fail;
+ regs->ax = val >> 32;
+
+ sev_es_wr_ghcb_msr(GHCB_CPUID_REQ(fn, GHCB_CPUID_REQ_EBX));
+ VMGEXIT();
+ val = sev_es_rd_ghcb_msr();
+ if (GHCB_SEV_GHCB_RESP_CODE(val) != GHCB_SEV_CPUID_RESP)
+ goto fail;
+ regs->bx = val >> 32;
+
+ sev_es_wr_ghcb_msr(GHCB_CPUID_REQ(fn, GHCB_CPUID_REQ_ECX));
+ VMGEXIT();
+ val = sev_es_rd_ghcb_msr();
+ if (GHCB_SEV_GHCB_RESP_CODE(val) != GHCB_SEV_CPUID_RESP)
+ goto fail;
+ regs->cx = val >> 32;
+
+ sev_es_wr_ghcb_msr(GHCB_CPUID_REQ(fn, GHCB_CPUID_REQ_EDX));
+ VMGEXIT();
+ val = sev_es_rd_ghcb_msr();
+ if (GHCB_SEV_GHCB_RESP_CODE(val) != GHCB_SEV_CPUID_RESP)
+ goto fail;
+ regs->dx = val >> 32;
+
+ /* Skip over the CPUID two-byte opcode */
+ regs->ip += 2;
+
+ return;
+
+fail:
+ sev_es_wr_ghcb_msr(GHCB_SEV_TERMINATE);
+ VMGEXIT();
+
+ /* Shouldn't get here - if we do halt the machine */
+ while (true)
+ asm volatile("hlt\n");
+}
^ permalink raw reply related [flat|nested] 264+ messages in thread
* [tip: x86/seves] x86/boot/compressed/64: Call set_sev_encryption_mask() earlier
2020-09-07 13:15 ` Joerg Roedel
(?)
@ 2020-09-10 9:22 ` tip-bot2 for Joerg Roedel
-1 siblings, 0 replies; 264+ messages in thread
From: tip-bot2 for Joerg Roedel @ 2020-09-10 9:22 UTC (permalink / raw)
To: linux-tip-commits; +Cc: Joerg Roedel, Borislav Petkov, x86, LKML
The following commit has been merged into the x86/seves branch of tip:
Commit-ID: c2a0304a286f386e45cea3f4b0617f0813de67fd
Gitweb: https://git.kernel.org/tip/c2a0304a286f386e45cea3f4b0617f0813de67fd
Author: Joerg Roedel <jroedel@suse.de>
AuthorDate: Mon, 07 Sep 2020 15:15:21 +02:00
Committer: Borislav Petkov <bp@suse.de>
CommitterDate: Mon, 07 Sep 2020 19:45:25 +02:00
x86/boot/compressed/64: Call set_sev_encryption_mask() earlier
Call set_sev_encryption_mask() while still on the stage 1 #VC-handler
because the stage 2 handler needs the kernel's own page tables to be
set up, to which calling set_sev_encryption_mask() is a prerequisite.
Signed-off-by: Joerg Roedel <jroedel@suse.de>
Signed-off-by: Borislav Petkov <bp@suse.de>
Link: https://lkml.kernel.org/r/20200907131613.12703-21-joro@8bytes.org
---
arch/x86/boot/compressed/head_64.S | 9 ++++++++-
arch/x86/boot/compressed/ident_map_64.c | 3 ---
2 files changed, 8 insertions(+), 4 deletions(-)
diff --git a/arch/x86/boot/compressed/head_64.S b/arch/x86/boot/compressed/head_64.S
index fb6c039..42190c0 100644
--- a/arch/x86/boot/compressed/head_64.S
+++ b/arch/x86/boot/compressed/head_64.S
@@ -543,9 +543,16 @@ SYM_FUNC_START_LOCAL_NOALIGN(.Lrelocated)
rep stosq
/*
- * Load stage2 IDT and switch to our own page-table
+ * If running as an SEV guest, the encryption mask is required in the
+ * page-table setup code below. When the guest also has SEV-ES enabled
+ * set_sev_encryption_mask() will cause #VC exceptions, but the stage2
+ * handler can't map its GHCB because the page-table is not set up yet.
+ * So set up the encryption mask here while still on the stage1 #VC
+ * handler. Then load stage2 IDT and switch to the kernel's own
+ * page-table.
*/
pushq %rsi
+ call set_sev_encryption_mask
call load_stage2_idt
call initialize_identity_maps
popq %rsi
diff --git a/arch/x86/boot/compressed/ident_map_64.c b/arch/x86/boot/compressed/ident_map_64.c
index 62e42c1..b4f2a5f 100644
--- a/arch/x86/boot/compressed/ident_map_64.c
+++ b/arch/x86/boot/compressed/ident_map_64.c
@@ -105,9 +105,6 @@ static void add_identity_map(unsigned long start, unsigned long end)
/* Locates and clears a region for a new top level page table. */
void initialize_identity_maps(void)
{
- /* If running as an SEV guest, the encryption mask is required. */
- set_sev_encryption_mask();
-
/* Exclude the encryption mask from __PHYSICAL_MASK */
physical_mask &= ~sme_me_mask;
^ permalink raw reply related [flat|nested] 264+ messages in thread
* [tip: x86/seves] x86/boot/compressed/64: Change add_identity_map() to take start and end
2020-09-07 13:15 ` Joerg Roedel
(?)
@ 2020-09-10 9:22 ` tip-bot2 for Joerg Roedel
-1 siblings, 0 replies; 264+ messages in thread
From: tip-bot2 for Joerg Roedel @ 2020-09-10 9:22 UTC (permalink / raw)
To: linux-tip-commits; +Cc: Joerg Roedel, Borislav Petkov, Kees Cook, x86, LKML
The following commit has been merged into the x86/seves branch of tip:
Commit-ID: 21cf2372618ef167d8c4ae04880fb873b55b2daa
Gitweb: https://git.kernel.org/tip/21cf2372618ef167d8c4ae04880fb873b55b2daa
Author: Joerg Roedel <jroedel@suse.de>
AuthorDate: Mon, 07 Sep 2020 15:15:19 +02:00
Committer: Borislav Petkov <bp@suse.de>
CommitterDate: Mon, 07 Sep 2020 19:45:25 +02:00
x86/boot/compressed/64: Change add_identity_map() to take start and end
Changing the function to take start and end as parameters instead of
start and size simplifies the callers which don't need to calculate the
size if they already have start and end.
Signed-off-by: Joerg Roedel <jroedel@suse.de>
Signed-off-by: Borislav Petkov <bp@suse.de>
Reviewed-by: Kees Cook <keescook@chromium.org>
Link: https://lkml.kernel.org/r/20200907131613.12703-19-joro@8bytes.org
---
arch/x86/boot/compressed/ident_map_64.c | 15 +++++----------
1 file changed, 5 insertions(+), 10 deletions(-)
diff --git a/arch/x86/boot/compressed/ident_map_64.c b/arch/x86/boot/compressed/ident_map_64.c
index c63257b..62e42c1 100644
--- a/arch/x86/boot/compressed/ident_map_64.c
+++ b/arch/x86/boot/compressed/ident_map_64.c
@@ -89,10 +89,8 @@ static struct x86_mapping_info mapping_info;
/*
* Adds the specified range to the identity mappings.
*/
-static void add_identity_map(unsigned long start, unsigned long size)
+static void add_identity_map(unsigned long start, unsigned long end)
{
- unsigned long end = start + size;
-
/* Align boundary to 2M. */
start = round_down(start, PMD_SIZE);
end = round_up(end, PMD_SIZE);
@@ -107,8 +105,6 @@ static void add_identity_map(unsigned long start, unsigned long size)
/* Locates and clears a region for a new top level page table. */
void initialize_identity_maps(void)
{
- unsigned long start, size;
-
/* If running as an SEV guest, the encryption mask is required. */
set_sev_encryption_mask();
@@ -155,9 +151,7 @@ void initialize_identity_maps(void)
* New page-table is set up - map the kernel image and load it
* into cr3.
*/
- start = (unsigned long)_head;
- size = _end - _head;
- add_identity_map(start, size);
+ add_identity_map((unsigned long)_head, (unsigned long)_end);
write_cr3(top_level_pgt);
}
@@ -189,7 +183,8 @@ static void do_pf_error(const char *msg, unsigned long error_code,
void do_boot_page_fault(struct pt_regs *regs, unsigned long error_code)
{
- unsigned long address = native_read_cr2();
+ unsigned long address = native_read_cr2() & PMD_MASK;
+ unsigned long end = address + PMD_SIZE;
/*
* Check for unexpected error codes. Unexpected are:
@@ -204,5 +199,5 @@ void do_boot_page_fault(struct pt_regs *regs, unsigned long error_code)
* Error code is sane - now identity map the 2M region around
* the faulting address.
*/
- add_identity_map(address & PMD_MASK, PMD_SIZE);
+ add_identity_map(address, end);
}
^ permalink raw reply related [flat|nested] 264+ messages in thread
* [tip: x86/seves] x86/boot/compressed/64: Don't pre-map memory in KASLR code
2020-09-07 13:15 ` Joerg Roedel
(?)
@ 2020-09-10 9:22 ` tip-bot2 for Joerg Roedel
-1 siblings, 0 replies; 264+ messages in thread
From: tip-bot2 for Joerg Roedel @ 2020-09-10 9:22 UTC (permalink / raw)
To: linux-tip-commits; +Cc: Joerg Roedel, Borislav Petkov, Kees Cook, x86, LKML
The following commit has been merged into the x86/seves branch of tip:
Commit-ID: 8570978ea030757839747aa9944ea576708be3d4
Gitweb: https://git.kernel.org/tip/8570978ea030757839747aa9944ea576708be3d4
Author: Joerg Roedel <jroedel@suse.de>
AuthorDate: Mon, 07 Sep 2020 15:15:18 +02:00
Committer: Borislav Petkov <bp@suse.de>
CommitterDate: Mon, 07 Sep 2020 19:45:25 +02:00
x86/boot/compressed/64: Don't pre-map memory in KASLR code
With the page-fault handler in place, he identity mapping can be built
on-demand. So remove the code which manually creates the mappings and
unexport/remove the functions used for it.
Signed-off-by: Joerg Roedel <jroedel@suse.de>
Signed-off-by: Borislav Petkov <bp@suse.de>
Reviewed-by: Kees Cook <keescook@chromium.org>
Link: https://lkml.kernel.org/r/20200907131613.12703-18-joro@8bytes.org
---
arch/x86/boot/compressed/ident_map_64.c | 6 ++----
arch/x86/boot/compressed/kaslr.c | 24 +-----------------------
arch/x86/boot/compressed/misc.h | 10 +----------
3 files changed, 3 insertions(+), 37 deletions(-)
diff --git a/arch/x86/boot/compressed/ident_map_64.c b/arch/x86/boot/compressed/ident_map_64.c
index ecf9353..c63257b 100644
--- a/arch/x86/boot/compressed/ident_map_64.c
+++ b/arch/x86/boot/compressed/ident_map_64.c
@@ -87,11 +87,9 @@ phys_addr_t physical_mask = (1ULL << __PHYSICAL_MASK_SHIFT) - 1;
static struct x86_mapping_info mapping_info;
/*
- * Adds the specified range to what will become the new identity mappings.
- * Once all ranges have been added, the new mapping is activated by calling
- * finalize_identity_maps() below.
+ * Adds the specified range to the identity mappings.
*/
-void add_identity_map(unsigned long start, unsigned long size)
+static void add_identity_map(unsigned long start, unsigned long size)
{
unsigned long end = start + size;
diff --git a/arch/x86/boot/compressed/kaslr.c b/arch/x86/boot/compressed/kaslr.c
index 8266286..b59547c 100644
--- a/arch/x86/boot/compressed/kaslr.c
+++ b/arch/x86/boot/compressed/kaslr.c
@@ -397,8 +397,6 @@ static void mem_avoid_init(unsigned long input, unsigned long input_size,
*/
mem_avoid[MEM_AVOID_ZO_RANGE].start = input;
mem_avoid[MEM_AVOID_ZO_RANGE].size = (output + init_size) - input;
- add_identity_map(mem_avoid[MEM_AVOID_ZO_RANGE].start,
- mem_avoid[MEM_AVOID_ZO_RANGE].size);
/* Avoid initrd. */
initrd_start = (u64)boot_params->ext_ramdisk_image << 32;
@@ -416,15 +414,11 @@ static void mem_avoid_init(unsigned long input, unsigned long input_size,
cmd_line_size = strnlen((char *)cmd_line, COMMAND_LINE_SIZE-1) + 1;
mem_avoid[MEM_AVOID_CMDLINE].start = cmd_line;
mem_avoid[MEM_AVOID_CMDLINE].size = cmd_line_size;
- add_identity_map(mem_avoid[MEM_AVOID_CMDLINE].start,
- mem_avoid[MEM_AVOID_CMDLINE].size);
}
/* Avoid boot parameters. */
mem_avoid[MEM_AVOID_BOOTPARAMS].start = (unsigned long)boot_params;
mem_avoid[MEM_AVOID_BOOTPARAMS].size = sizeof(*boot_params);
- add_identity_map(mem_avoid[MEM_AVOID_BOOTPARAMS].start,
- mem_avoid[MEM_AVOID_BOOTPARAMS].size);
/* We don't need to set a mapping for setup_data. */
@@ -433,11 +427,6 @@ static void mem_avoid_init(unsigned long input, unsigned long input_size,
/* Enumerate the immovable memory regions */
num_immovable_mem = count_immovable_mem_regions();
-
-#ifdef CONFIG_X86_VERBOSE_BOOTUP
- /* Make sure video RAM can be used. */
- add_identity_map(0, PMD_SIZE);
-#endif
}
/*
@@ -884,19 +873,8 @@ void choose_random_location(unsigned long input,
warn("Physical KASLR disabled: no suitable memory region!");
} else {
/* Update the new physical address location. */
- if (*output != random_addr) {
- add_identity_map(random_addr, output_size);
+ if (*output != random_addr)
*output = random_addr;
- }
-
- /*
- * This loads the identity mapping page table.
- * This should only be done if a new physical address
- * is found for the kernel, otherwise we should keep
- * the old page table to make it be like the "nokaslr"
- * case.
- */
- finalize_identity_maps();
}
diff --git a/arch/x86/boot/compressed/misc.h b/arch/x86/boot/compressed/misc.h
index f0e1991..9840c82 100644
--- a/arch/x86/boot/compressed/misc.h
+++ b/arch/x86/boot/compressed/misc.h
@@ -98,17 +98,7 @@ static inline void choose_random_location(unsigned long input,
#endif
#ifdef CONFIG_X86_64
-void initialize_identity_maps(void);
-void add_identity_map(unsigned long start, unsigned long size);
-void finalize_identity_maps(void);
extern unsigned char _pgtable[];
-#else
-static inline void initialize_identity_maps(void)
-{ }
-static inline void add_identity_map(unsigned long start, unsigned long size)
-{ }
-static inline void finalize_identity_maps(void)
-{ }
#endif
#ifdef CONFIG_EARLY_PRINTK
^ permalink raw reply related [flat|nested] 264+ messages in thread
* [tip: x86/seves] x86/boot/compressed/64: Always switch to own page table
2020-09-07 13:15 ` Joerg Roedel
(?)
@ 2020-09-10 9:22 ` tip-bot2 for Joerg Roedel
-1 siblings, 0 replies; 264+ messages in thread
From: tip-bot2 for Joerg Roedel @ 2020-09-10 9:22 UTC (permalink / raw)
To: linux-tip-commits; +Cc: Joerg Roedel, Borislav Petkov, Kees Cook, x86, LKML
The following commit has been merged into the x86/seves branch of tip:
Commit-ID: ca0e22d4f011a56e974fa3a712d76e86a791559d
Gitweb: https://git.kernel.org/tip/ca0e22d4f011a56e974fa3a712d76e86a791559d
Author: Joerg Roedel <jroedel@suse.de>
AuthorDate: Mon, 07 Sep 2020 15:15:17 +02:00
Committer: Borislav Petkov <bp@suse.de>
CommitterDate: Mon, 07 Sep 2020 19:45:25 +02:00
x86/boot/compressed/64: Always switch to own page table
When booted through startup_64(), the kernel keeps running on the EFI
page table until the KASLR code sets up its own page table. Without
KASLR, the pre-decompression boot code never switches off the EFI page
table. Change that by unconditionally switching to a kernel-controlled
page table after relocation.
This makes sure the kernel can make changes to the mapping when
necessary, for example map pages unencrypted in SEV and SEV-ES guests.
Also, remove the debug_putstr() calls in initialize_identity_maps()
because the function now runs before console_init() is called.
[ bp: Massage commit message. ]
Signed-off-by: Joerg Roedel <jroedel@suse.de>
Signed-off-by: Borislav Petkov <bp@suse.de>
Reviewed-by: Kees Cook <keescook@chromium.org>
Link: https://lkml.kernel.org/r/20200907131613.12703-17-joro@8bytes.org
---
arch/x86/boot/compressed/head_64.S | 3 +-
arch/x86/boot/compressed/ident_map_64.c | 51 ++++++++++++++----------
arch/x86/boot/compressed/kaslr.c | 3 +-
3 files changed, 32 insertions(+), 25 deletions(-)
diff --git a/arch/x86/boot/compressed/head_64.S b/arch/x86/boot/compressed/head_64.S
index c634ed8..fb6c039 100644
--- a/arch/x86/boot/compressed/head_64.S
+++ b/arch/x86/boot/compressed/head_64.S
@@ -543,10 +543,11 @@ SYM_FUNC_START_LOCAL_NOALIGN(.Lrelocated)
rep stosq
/*
- * Load stage2 IDT
+ * Load stage2 IDT and switch to our own page-table
*/
pushq %rsi
call load_stage2_idt
+ call initialize_identity_maps
popq %rsi
/*
diff --git a/arch/x86/boot/compressed/ident_map_64.c b/arch/x86/boot/compressed/ident_map_64.c
index e3d980a..ecf9353 100644
--- a/arch/x86/boot/compressed/ident_map_64.c
+++ b/arch/x86/boot/compressed/ident_map_64.c
@@ -86,9 +86,31 @@ phys_addr_t physical_mask = (1ULL << __PHYSICAL_MASK_SHIFT) - 1;
*/
static struct x86_mapping_info mapping_info;
+/*
+ * Adds the specified range to what will become the new identity mappings.
+ * Once all ranges have been added, the new mapping is activated by calling
+ * finalize_identity_maps() below.
+ */
+void add_identity_map(unsigned long start, unsigned long size)
+{
+ unsigned long end = start + size;
+
+ /* Align boundary to 2M. */
+ start = round_down(start, PMD_SIZE);
+ end = round_up(end, PMD_SIZE);
+ if (start >= end)
+ return;
+
+ /* Build the mapping. */
+ kernel_ident_mapping_init(&mapping_info, (pgd_t *)top_level_pgt,
+ start, end);
+}
+
/* Locates and clears a region for a new top level page table. */
void initialize_identity_maps(void)
{
+ unsigned long start, size;
+
/* If running as an SEV guest, the encryption mask is required. */
set_sev_encryption_mask();
@@ -121,37 +143,24 @@ void initialize_identity_maps(void)
*/
top_level_pgt = read_cr3_pa();
if (p4d_offset((pgd_t *)top_level_pgt, 0) == (p4d_t *)_pgtable) {
- debug_putstr("booted via startup_32()\n");
pgt_data.pgt_buf = _pgtable + BOOT_INIT_PGT_SIZE;
pgt_data.pgt_buf_size = BOOT_PGT_SIZE - BOOT_INIT_PGT_SIZE;
memset(pgt_data.pgt_buf, 0, pgt_data.pgt_buf_size);
} else {
- debug_putstr("booted via startup_64()\n");
pgt_data.pgt_buf = _pgtable;
pgt_data.pgt_buf_size = BOOT_PGT_SIZE;
memset(pgt_data.pgt_buf, 0, pgt_data.pgt_buf_size);
top_level_pgt = (unsigned long)alloc_pgt_page(&pgt_data);
}
-}
-/*
- * Adds the specified range to what will become the new identity mappings.
- * Once all ranges have been added, the new mapping is activated by calling
- * finalize_identity_maps() below.
- */
-void add_identity_map(unsigned long start, unsigned long size)
-{
- unsigned long end = start + size;
-
- /* Align boundary to 2M. */
- start = round_down(start, PMD_SIZE);
- end = round_up(end, PMD_SIZE);
- if (start >= end)
- return;
-
- /* Build the mapping. */
- kernel_ident_mapping_init(&mapping_info, (pgd_t *)top_level_pgt,
- start, end);
+ /*
+ * New page-table is set up - map the kernel image and load it
+ * into cr3.
+ */
+ start = (unsigned long)_head;
+ size = _end - _head;
+ add_identity_map(start, size);
+ write_cr3(top_level_pgt);
}
/*
diff --git a/arch/x86/boot/compressed/kaslr.c b/arch/x86/boot/compressed/kaslr.c
index e27de98..8266286 100644
--- a/arch/x86/boot/compressed/kaslr.c
+++ b/arch/x86/boot/compressed/kaslr.c
@@ -861,9 +861,6 @@ void choose_random_location(unsigned long input,
boot_params->hdr.loadflags |= KASLR_FLAG;
- /* Prepare to add new identity pagetables on demand. */
- initialize_identity_maps();
-
if (IS_ENABLED(CONFIG_X86_32))
mem_limit = KERNEL_IMAGE_SIZE;
else
^ permalink raw reply related [flat|nested] 264+ messages in thread
* [tip: x86/seves] x86/boot/compressed/64: Rename kaslr_64.c to ident_map_64.c
2020-09-07 13:15 ` Joerg Roedel
(?)
@ 2020-09-10 9:22 ` tip-bot2 for Joerg Roedel
-1 siblings, 0 replies; 264+ messages in thread
From: tip-bot2 for Joerg Roedel @ 2020-09-10 9:22 UTC (permalink / raw)
To: linux-tip-commits; +Cc: Joerg Roedel, Borislav Petkov, Kees Cook, x86, LKML
The following commit has been merged into the x86/seves branch of tip:
Commit-ID: 5f2bb01682b7b067783207994c7b8a3dbeb1cd83
Gitweb: https://git.kernel.org/tip/5f2bb01682b7b067783207994c7b8a3dbeb1cd83
Author: Joerg Roedel <jroedel@suse.de>
AuthorDate: Mon, 07 Sep 2020 15:15:15 +02:00
Committer: Borislav Petkov <bp@suse.de>
CommitterDate: Mon, 07 Sep 2020 19:45:25 +02:00
x86/boot/compressed/64: Rename kaslr_64.c to ident_map_64.c
The file contains only code related to identity-mapped page tables.
Rename the file and compile it always in.
Signed-off-by: Joerg Roedel <jroedel@suse.de>
Signed-off-by: Borislav Petkov <bp@suse.de>
Reviewed-by: Kees Cook <keescook@chromium.org>
Link: https://lkml.kernel.org/r/20200907131613.12703-15-joro@8bytes.org
---
arch/x86/boot/compressed/Makefile | 2 +-
arch/x86/boot/compressed/ident_map_64.c | 162 +++++++++++++++++++++++-
arch/x86/boot/compressed/kaslr.c | 9 +-
arch/x86/boot/compressed/kaslr_64.c | 153 +----------------------
arch/x86/boot/compressed/misc.h | 8 +-
5 files changed, 171 insertions(+), 163 deletions(-)
create mode 100644 arch/x86/boot/compressed/ident_map_64.c
delete mode 100644 arch/x86/boot/compressed/kaslr_64.c
diff --git a/arch/x86/boot/compressed/Makefile b/arch/x86/boot/compressed/Makefile
index c661dc5..e7f3eba 100644
--- a/arch/x86/boot/compressed/Makefile
+++ b/arch/x86/boot/compressed/Makefile
@@ -84,7 +84,7 @@ vmlinux-objs-y := $(obj)/vmlinux.lds $(obj)/kernel_info.o $(obj)/head_$(BITS).o
vmlinux-objs-$(CONFIG_EARLY_PRINTK) += $(obj)/early_serial_console.o
vmlinux-objs-$(CONFIG_RANDOMIZE_BASE) += $(obj)/kaslr.o
ifdef CONFIG_X86_64
- vmlinux-objs-$(CONFIG_RANDOMIZE_BASE) += $(obj)/kaslr_64.o
+ vmlinux-objs-y += $(obj)/ident_map_64.o
vmlinux-objs-y += $(obj)/idt_64.o $(obj)/idt_handlers_64.o
vmlinux-objs-y += $(obj)/mem_encrypt.o
vmlinux-objs-y += $(obj)/pgtable_64.o
diff --git a/arch/x86/boot/compressed/ident_map_64.c b/arch/x86/boot/compressed/ident_map_64.c
new file mode 100644
index 0000000..d9932a1
--- /dev/null
+++ b/arch/x86/boot/compressed/ident_map_64.c
@@ -0,0 +1,162 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * This code is used on x86_64 to create page table identity mappings on
+ * demand by building up a new set of page tables (or appending to the
+ * existing ones), and then switching over to them when ready.
+ *
+ * Copyright (C) 2015-2016 Yinghai Lu
+ * Copyright (C) 2016 Kees Cook
+ */
+
+/*
+ * Since we're dealing with identity mappings, physical and virtual
+ * addresses are the same, so override these defines which are ultimately
+ * used by the headers in misc.h.
+ */
+#define __pa(x) ((unsigned long)(x))
+#define __va(x) ((void *)((unsigned long)(x)))
+
+/* No PAGE_TABLE_ISOLATION support needed either: */
+#undef CONFIG_PAGE_TABLE_ISOLATION
+
+#include "misc.h"
+
+/* These actually do the work of building the kernel identity maps. */
+#include <linux/pgtable.h>
+#include <asm/init.h>
+/* Use the static base for this part of the boot process */
+#undef __PAGE_OFFSET
+#define __PAGE_OFFSET __PAGE_OFFSET_BASE
+#include "../../mm/ident_map.c"
+
+#ifdef CONFIG_X86_5LEVEL
+unsigned int __pgtable_l5_enabled;
+unsigned int pgdir_shift = 39;
+unsigned int ptrs_per_p4d = 1;
+#endif
+
+/* Used by PAGE_KERN* macros: */
+pteval_t __default_kernel_pte_mask __read_mostly = ~0;
+
+/* Used to track our page table allocation area. */
+struct alloc_pgt_data {
+ unsigned char *pgt_buf;
+ unsigned long pgt_buf_size;
+ unsigned long pgt_buf_offset;
+};
+
+/*
+ * Allocates space for a page table entry, using struct alloc_pgt_data
+ * above. Besides the local callers, this is used as the allocation
+ * callback in mapping_info below.
+ */
+static void *alloc_pgt_page(void *context)
+{
+ struct alloc_pgt_data *pages = (struct alloc_pgt_data *)context;
+ unsigned char *entry;
+
+ /* Validate there is space available for a new page. */
+ if (pages->pgt_buf_offset >= pages->pgt_buf_size) {
+ debug_putstr("out of pgt_buf in " __FILE__ "!?\n");
+ debug_putaddr(pages->pgt_buf_offset);
+ debug_putaddr(pages->pgt_buf_size);
+ return NULL;
+ }
+
+ entry = pages->pgt_buf + pages->pgt_buf_offset;
+ pages->pgt_buf_offset += PAGE_SIZE;
+
+ return entry;
+}
+
+/* Used to track our allocated page tables. */
+static struct alloc_pgt_data pgt_data;
+
+/* The top level page table entry pointer. */
+static unsigned long top_level_pgt;
+
+phys_addr_t physical_mask = (1ULL << __PHYSICAL_MASK_SHIFT) - 1;
+
+/*
+ * Mapping information structure passed to kernel_ident_mapping_init().
+ * Due to relocation, pointers must be assigned at run time not build time.
+ */
+static struct x86_mapping_info mapping_info;
+
+/* Locates and clears a region for a new top level page table. */
+void initialize_identity_maps(void)
+{
+ /* If running as an SEV guest, the encryption mask is required. */
+ set_sev_encryption_mask();
+
+ /* Exclude the encryption mask from __PHYSICAL_MASK */
+ physical_mask &= ~sme_me_mask;
+
+ /* Init mapping_info with run-time function/buffer pointers. */
+ mapping_info.alloc_pgt_page = alloc_pgt_page;
+ mapping_info.context = &pgt_data;
+ mapping_info.page_flag = __PAGE_KERNEL_LARGE_EXEC | sme_me_mask;
+ mapping_info.kernpg_flag = _KERNPG_TABLE;
+
+ /*
+ * It should be impossible for this not to already be true,
+ * but since calling this a second time would rewind the other
+ * counters, let's just make sure this is reset too.
+ */
+ pgt_data.pgt_buf_offset = 0;
+
+ /*
+ * If we came here via startup_32(), cr3 will be _pgtable already
+ * and we must append to the existing area instead of entirely
+ * overwriting it.
+ *
+ * With 5-level paging, we use '_pgtable' to allocate the p4d page table,
+ * the top-level page table is allocated separately.
+ *
+ * p4d_offset(top_level_pgt, 0) would cover both the 4- and 5-level
+ * cases. On 4-level paging it's equal to 'top_level_pgt'.
+ */
+ top_level_pgt = read_cr3_pa();
+ if (p4d_offset((pgd_t *)top_level_pgt, 0) == (p4d_t *)_pgtable) {
+ debug_putstr("booted via startup_32()\n");
+ pgt_data.pgt_buf = _pgtable + BOOT_INIT_PGT_SIZE;
+ pgt_data.pgt_buf_size = BOOT_PGT_SIZE - BOOT_INIT_PGT_SIZE;
+ memset(pgt_data.pgt_buf, 0, pgt_data.pgt_buf_size);
+ } else {
+ debug_putstr("booted via startup_64()\n");
+ pgt_data.pgt_buf = _pgtable;
+ pgt_data.pgt_buf_size = BOOT_PGT_SIZE;
+ memset(pgt_data.pgt_buf, 0, pgt_data.pgt_buf_size);
+ top_level_pgt = (unsigned long)alloc_pgt_page(&pgt_data);
+ }
+}
+
+/*
+ * Adds the specified range to what will become the new identity mappings.
+ * Once all ranges have been added, the new mapping is activated by calling
+ * finalize_identity_maps() below.
+ */
+void add_identity_map(unsigned long start, unsigned long size)
+{
+ unsigned long end = start + size;
+
+ /* Align boundary to 2M. */
+ start = round_down(start, PMD_SIZE);
+ end = round_up(end, PMD_SIZE);
+ if (start >= end)
+ return;
+
+ /* Build the mapping. */
+ kernel_ident_mapping_init(&mapping_info, (pgd_t *)top_level_pgt,
+ start, end);
+}
+
+/*
+ * This switches the page tables to the new level4 that has been built
+ * via calls to add_identity_map() above. If booted via startup_32(),
+ * this is effectively a no-op.
+ */
+void finalize_identity_maps(void)
+{
+ write_cr3(top_level_pgt);
+}
diff --git a/arch/x86/boot/compressed/kaslr.c b/arch/x86/boot/compressed/kaslr.c
index 877970d..e27de98 100644
--- a/arch/x86/boot/compressed/kaslr.c
+++ b/arch/x86/boot/compressed/kaslr.c
@@ -40,17 +40,8 @@
#include <asm/setup.h> /* For COMMAND_LINE_SIZE */
#undef _SETUP
-#ifdef CONFIG_X86_5LEVEL
-unsigned int __pgtable_l5_enabled;
-unsigned int pgdir_shift __ro_after_init = 39;
-unsigned int ptrs_per_p4d __ro_after_init = 1;
-#endif
-
extern unsigned long get_cmd_line_ptr(void);
-/* Used by PAGE_KERN* macros: */
-pteval_t __default_kernel_pte_mask __read_mostly = ~0;
-
/* Simplified build-specific string for starting entropy. */
static const char build_str[] = UTS_RELEASE " (" LINUX_COMPILE_BY "@"
LINUX_COMPILE_HOST ") (" LINUX_COMPILER ") " UTS_VERSION;
diff --git a/arch/x86/boot/compressed/kaslr_64.c b/arch/x86/boot/compressed/kaslr_64.c
deleted file mode 100644
index f9c5c13..0000000
--- a/arch/x86/boot/compressed/kaslr_64.c
+++ /dev/null
@@ -1,153 +0,0 @@
-// SPDX-License-Identifier: GPL-2.0
-/*
- * This code is used on x86_64 to create page table identity mappings on
- * demand by building up a new set of page tables (or appending to the
- * existing ones), and then switching over to them when ready.
- *
- * Copyright (C) 2015-2016 Yinghai Lu
- * Copyright (C) 2016 Kees Cook
- */
-
-/*
- * Since we're dealing with identity mappings, physical and virtual
- * addresses are the same, so override these defines which are ultimately
- * used by the headers in misc.h.
- */
-#define __pa(x) ((unsigned long)(x))
-#define __va(x) ((void *)((unsigned long)(x)))
-
-/* No PAGE_TABLE_ISOLATION support needed either: */
-#undef CONFIG_PAGE_TABLE_ISOLATION
-
-#include "misc.h"
-
-/* These actually do the work of building the kernel identity maps. */
-#include <linux/pgtable.h>
-#include <asm/init.h>
-/* Use the static base for this part of the boot process */
-#undef __PAGE_OFFSET
-#define __PAGE_OFFSET __PAGE_OFFSET_BASE
-#include "../../mm/ident_map.c"
-
-/* Used to track our page table allocation area. */
-struct alloc_pgt_data {
- unsigned char *pgt_buf;
- unsigned long pgt_buf_size;
- unsigned long pgt_buf_offset;
-};
-
-/*
- * Allocates space for a page table entry, using struct alloc_pgt_data
- * above. Besides the local callers, this is used as the allocation
- * callback in mapping_info below.
- */
-static void *alloc_pgt_page(void *context)
-{
- struct alloc_pgt_data *pages = (struct alloc_pgt_data *)context;
- unsigned char *entry;
-
- /* Validate there is space available for a new page. */
- if (pages->pgt_buf_offset >= pages->pgt_buf_size) {
- debug_putstr("out of pgt_buf in " __FILE__ "!?\n");
- debug_putaddr(pages->pgt_buf_offset);
- debug_putaddr(pages->pgt_buf_size);
- return NULL;
- }
-
- entry = pages->pgt_buf + pages->pgt_buf_offset;
- pages->pgt_buf_offset += PAGE_SIZE;
-
- return entry;
-}
-
-/* Used to track our allocated page tables. */
-static struct alloc_pgt_data pgt_data;
-
-/* The top level page table entry pointer. */
-static unsigned long top_level_pgt;
-
-phys_addr_t physical_mask = (1ULL << __PHYSICAL_MASK_SHIFT) - 1;
-
-/*
- * Mapping information structure passed to kernel_ident_mapping_init().
- * Due to relocation, pointers must be assigned at run time not build time.
- */
-static struct x86_mapping_info mapping_info;
-
-/* Locates and clears a region for a new top level page table. */
-void initialize_identity_maps(void)
-{
- /* If running as an SEV guest, the encryption mask is required. */
- set_sev_encryption_mask();
-
- /* Exclude the encryption mask from __PHYSICAL_MASK */
- physical_mask &= ~sme_me_mask;
-
- /* Init mapping_info with run-time function/buffer pointers. */
- mapping_info.alloc_pgt_page = alloc_pgt_page;
- mapping_info.context = &pgt_data;
- mapping_info.page_flag = __PAGE_KERNEL_LARGE_EXEC | sme_me_mask;
- mapping_info.kernpg_flag = _KERNPG_TABLE;
-
- /*
- * It should be impossible for this not to already be true,
- * but since calling this a second time would rewind the other
- * counters, let's just make sure this is reset too.
- */
- pgt_data.pgt_buf_offset = 0;
-
- /*
- * If we came here via startup_32(), cr3 will be _pgtable already
- * and we must append to the existing area instead of entirely
- * overwriting it.
- *
- * With 5-level paging, we use '_pgtable' to allocate the p4d page table,
- * the top-level page table is allocated separately.
- *
- * p4d_offset(top_level_pgt, 0) would cover both the 4- and 5-level
- * cases. On 4-level paging it's equal to 'top_level_pgt'.
- */
- top_level_pgt = read_cr3_pa();
- if (p4d_offset((pgd_t *)top_level_pgt, 0) == (p4d_t *)_pgtable) {
- debug_putstr("booted via startup_32()\n");
- pgt_data.pgt_buf = _pgtable + BOOT_INIT_PGT_SIZE;
- pgt_data.pgt_buf_size = BOOT_PGT_SIZE - BOOT_INIT_PGT_SIZE;
- memset(pgt_data.pgt_buf, 0, pgt_data.pgt_buf_size);
- } else {
- debug_putstr("booted via startup_64()\n");
- pgt_data.pgt_buf = _pgtable;
- pgt_data.pgt_buf_size = BOOT_PGT_SIZE;
- memset(pgt_data.pgt_buf, 0, pgt_data.pgt_buf_size);
- top_level_pgt = (unsigned long)alloc_pgt_page(&pgt_data);
- }
-}
-
-/*
- * Adds the specified range to what will become the new identity mappings.
- * Once all ranges have been added, the new mapping is activated by calling
- * finalize_identity_maps() below.
- */
-void add_identity_map(unsigned long start, unsigned long size)
-{
- unsigned long end = start + size;
-
- /* Align boundary to 2M. */
- start = round_down(start, PMD_SIZE);
- end = round_up(end, PMD_SIZE);
- if (start >= end)
- return;
-
- /* Build the mapping. */
- kernel_ident_mapping_init(&mapping_info, (pgd_t *)top_level_pgt,
- start, end);
-}
-
-/*
- * This switches the page tables to the new level4 that has been built
- * via calls to add_identity_map() above. If booted via startup_32(),
- * this is effectively a no-op.
- */
-void finalize_identity_maps(void)
-{
- write_cr3(top_level_pgt);
-}
diff --git a/arch/x86/boot/compressed/misc.h b/arch/x86/boot/compressed/misc.h
index 8feb5f6..98b7a1d 100644
--- a/arch/x86/boot/compressed/misc.h
+++ b/arch/x86/boot/compressed/misc.h
@@ -134,6 +134,14 @@ int count_immovable_mem_regions(void);
static inline int count_immovable_mem_regions(void) { return 0; }
#endif
+/* ident_map_64.c */
+#ifdef CONFIG_X86_5LEVEL
+extern unsigned int __pgtable_l5_enabled, pgdir_shift, ptrs_per_p4d;
+#endif
+
+/* Used by PAGE_KERN* macros: */
+extern pteval_t __default_kernel_pte_mask;
+
/* idt_64.c */
extern gate_desc boot_idt[BOOT_IDT_ENTRIES];
extern struct desc_ptr boot_idt_desc;
^ permalink raw reply related [flat|nested] 264+ messages in thread
* [tip: x86/seves] x86/boot/compressed/64: Add page-fault handler
2020-09-07 13:15 ` Joerg Roedel
(?)
@ 2020-09-10 9:22 ` tip-bot2 for Joerg Roedel
-1 siblings, 0 replies; 264+ messages in thread
From: tip-bot2 for Joerg Roedel @ 2020-09-10 9:22 UTC (permalink / raw)
To: linux-tip-commits; +Cc: Joerg Roedel, Borislav Petkov, Kees Cook, x86, LKML
The following commit has been merged into the x86/seves branch of tip:
Commit-ID: 8b0d3b3b41ab6f14f1ce6d4a6b1c5f60b825123f
Gitweb: https://git.kernel.org/tip/8b0d3b3b41ab6f14f1ce6d4a6b1c5f60b825123f
Author: Joerg Roedel <jroedel@suse.de>
AuthorDate: Mon, 07 Sep 2020 15:15:16 +02:00
Committer: Borislav Petkov <bp@suse.de>
CommitterDate: Mon, 07 Sep 2020 19:45:25 +02:00
x86/boot/compressed/64: Add page-fault handler
Install a page-fault handler to add an identity mapping to addresses
not yet mapped. Also do some checking whether the error code is sane.
This makes non SEV-ES machines use the exception handling
infrastructure in the pre-decompressions boot code too, making it less
likely to break in the future.
Signed-off-by: Joerg Roedel <jroedel@suse.de>
Signed-off-by: Borislav Petkov <bp@suse.de>
Reviewed-by: Kees Cook <keescook@chromium.org>
Link: https://lkml.kernel.org/r/20200907131613.12703-16-joro@8bytes.org
---
arch/x86/boot/compressed/ident_map_64.c | 39 +++++++++++++++++++++-
arch/x86/boot/compressed/idt_64.c | 2 +-
arch/x86/boot/compressed/idt_handlers_64.S | 2 +-
arch/x86/boot/compressed/misc.h | 6 +++-
4 files changed, 49 insertions(+)
diff --git a/arch/x86/boot/compressed/ident_map_64.c b/arch/x86/boot/compressed/ident_map_64.c
index d9932a1..e3d980a 100644
--- a/arch/x86/boot/compressed/ident_map_64.c
+++ b/arch/x86/boot/compressed/ident_map_64.c
@@ -19,10 +19,13 @@
/* No PAGE_TABLE_ISOLATION support needed either: */
#undef CONFIG_PAGE_TABLE_ISOLATION
+#include "error.h"
#include "misc.h"
/* These actually do the work of building the kernel identity maps. */
#include <linux/pgtable.h>
+#include <asm/trap_pf.h>
+#include <asm/trapnr.h>
#include <asm/init.h>
/* Use the static base for this part of the boot process */
#undef __PAGE_OFFSET
@@ -160,3 +163,39 @@ void finalize_identity_maps(void)
{
write_cr3(top_level_pgt);
}
+
+static void do_pf_error(const char *msg, unsigned long error_code,
+ unsigned long address, unsigned long ip)
+{
+ error_putstr(msg);
+
+ error_putstr("\nError Code: ");
+ error_puthex(error_code);
+ error_putstr("\nCR2: 0x");
+ error_puthex(address);
+ error_putstr("\nRIP relative to _head: 0x");
+ error_puthex(ip - (unsigned long)_head);
+ error_putstr("\n");
+
+ error("Stopping.\n");
+}
+
+void do_boot_page_fault(struct pt_regs *regs, unsigned long error_code)
+{
+ unsigned long address = native_read_cr2();
+
+ /*
+ * Check for unexpected error codes. Unexpected are:
+ * - Faults on present pages
+ * - User faults
+ * - Reserved bits set
+ */
+ if (error_code & (X86_PF_PROT | X86_PF_USER | X86_PF_RSVD))
+ do_pf_error("Unexpected page-fault:", error_code, address, regs->ip);
+
+ /*
+ * Error code is sane - now identity map the 2M region around
+ * the faulting address.
+ */
+ add_identity_map(address & PMD_MASK, PMD_SIZE);
+}
diff --git a/arch/x86/boot/compressed/idt_64.c b/arch/x86/boot/compressed/idt_64.c
index 082cd6b..5f08309 100644
--- a/arch/x86/boot/compressed/idt_64.c
+++ b/arch/x86/boot/compressed/idt_64.c
@@ -40,5 +40,7 @@ void load_stage2_idt(void)
{
boot_idt_desc.address = (unsigned long)boot_idt;
+ set_idt_entry(X86_TRAP_PF, boot_page_fault);
+
load_boot_idt(&boot_idt_desc);
}
diff --git a/arch/x86/boot/compressed/idt_handlers_64.S b/arch/x86/boot/compressed/idt_handlers_64.S
index 36dee2f..b20e575 100644
--- a/arch/x86/boot/compressed/idt_handlers_64.S
+++ b/arch/x86/boot/compressed/idt_handlers_64.S
@@ -68,3 +68,5 @@ SYM_FUNC_END(\name)
.text
.code64
+
+EXCEPTION_HANDLER boot_page_fault do_boot_page_fault error_code=1
diff --git a/arch/x86/boot/compressed/misc.h b/arch/x86/boot/compressed/misc.h
index 98b7a1d..f0e1991 100644
--- a/arch/x86/boot/compressed/misc.h
+++ b/arch/x86/boot/compressed/misc.h
@@ -37,6 +37,9 @@
#define memptr unsigned
#endif
+/* boot/compressed/vmlinux start and end markers */
+extern char _head[], _end[];
+
/* misc.c */
extern memptr free_mem_ptr;
extern memptr free_mem_end_ptr;
@@ -146,4 +149,7 @@ extern pteval_t __default_kernel_pte_mask;
extern gate_desc boot_idt[BOOT_IDT_ENTRIES];
extern struct desc_ptr boot_idt_desc;
+/* IDT Entry Points */
+void boot_page_fault(void);
+
#endif /* BOOT_COMPRESSED_MISC_H */
^ permalink raw reply related [flat|nested] 264+ messages in thread
* [tip: x86/seves] x86/boot/compressed/64: Add IDT Infrastructure
2020-09-07 13:15 ` Joerg Roedel
(?)
@ 2020-09-10 9:22 ` tip-bot2 for Joerg Roedel
-1 siblings, 0 replies; 264+ messages in thread
From: tip-bot2 for Joerg Roedel @ 2020-09-10 9:22 UTC (permalink / raw)
To: linux-tip-commits; +Cc: Joerg Roedel, Borislav Petkov, x86, LKML
The following commit has been merged into the x86/seves branch of tip:
Commit-ID: 64e682638eb51070ba6044535b250aad43c5564e
Gitweb: https://git.kernel.org/tip/64e682638eb51070ba6044535b250aad43c5564e
Author: Joerg Roedel <jroedel@suse.de>
AuthorDate: Mon, 07 Sep 2020 15:15:14 +02:00
Committer: Borislav Petkov <bp@suse.de>
CommitterDate: Mon, 07 Sep 2020 19:45:25 +02:00
x86/boot/compressed/64: Add IDT Infrastructure
Add code needed to setup an IDT in the early pre-decompression
boot-code. The IDT is loaded first in startup_64, which is after
EfiExitBootServices() has been called, and later reloaded when the
kernel image has been relocated to the end of the decompression area.
This allows to setup different IDT handlers before and after the
relocation.
Signed-off-by: Joerg Roedel <jroedel@suse.de>
Signed-off-by: Borislav Petkov <bp@suse.de>
Link: https://lkml.kernel.org/r/20200907131613.12703-14-joro@8bytes.org
---
arch/x86/boot/compressed/Makefile | 1 +-
arch/x86/boot/compressed/head_64.S | 25 +++++++-
arch/x86/boot/compressed/idt_64.c | 44 +++++++++++++-
arch/x86/boot/compressed/idt_handlers_64.S | 70 +++++++++++++++++++++-
arch/x86/boot/compressed/misc.h | 5 ++-
arch/x86/include/asm/desc_defs.h | 3 +-
6 files changed, 147 insertions(+), 1 deletion(-)
create mode 100644 arch/x86/boot/compressed/idt_64.c
create mode 100644 arch/x86/boot/compressed/idt_handlers_64.S
diff --git a/arch/x86/boot/compressed/Makefile b/arch/x86/boot/compressed/Makefile
index 5343079..c661dc5 100644
--- a/arch/x86/boot/compressed/Makefile
+++ b/arch/x86/boot/compressed/Makefile
@@ -85,6 +85,7 @@ vmlinux-objs-$(CONFIG_EARLY_PRINTK) += $(obj)/early_serial_console.o
vmlinux-objs-$(CONFIG_RANDOMIZE_BASE) += $(obj)/kaslr.o
ifdef CONFIG_X86_64
vmlinux-objs-$(CONFIG_RANDOMIZE_BASE) += $(obj)/kaslr_64.o
+ vmlinux-objs-y += $(obj)/idt_64.o $(obj)/idt_handlers_64.o
vmlinux-objs-y += $(obj)/mem_encrypt.o
vmlinux-objs-y += $(obj)/pgtable_64.o
endif
diff --git a/arch/x86/boot/compressed/head_64.S b/arch/x86/boot/compressed/head_64.S
index 97d37f0..c634ed8 100644
--- a/arch/x86/boot/compressed/head_64.S
+++ b/arch/x86/boot/compressed/head_64.S
@@ -33,6 +33,7 @@
#include <asm/processor-flags.h>
#include <asm/asm-offsets.h>
#include <asm/bootparam.h>
+#include <asm/desc_defs.h>
#include "pgtable.h"
/*
@@ -410,6 +411,10 @@ SYM_CODE_START(startup_64)
.Lon_kernel_cs:
+ pushq %rsi
+ call load_stage1_idt
+ popq %rsi
+
/*
* paging_prepare() sets up the trampoline and checks if we need to
* enable 5-level paging.
@@ -538,6 +543,13 @@ SYM_FUNC_START_LOCAL_NOALIGN(.Lrelocated)
rep stosq
/*
+ * Load stage2 IDT
+ */
+ pushq %rsi
+ call load_stage2_idt
+ popq %rsi
+
+/*
* Do the extraction, and jump to the new kernel..
*/
pushq %rsi /* Save the real mode argument */
@@ -690,10 +702,21 @@ SYM_DATA_START_LOCAL(gdt)
.quad 0x0000000000000000 /* TS continued */
SYM_DATA_END_LABEL(gdt, SYM_L_LOCAL, gdt_end)
+SYM_DATA_START(boot_idt_desc)
+ .word boot_idt_end - boot_idt - 1
+ .quad 0
+SYM_DATA_END(boot_idt_desc)
+ .balign 8
+SYM_DATA_START(boot_idt)
+ .rept BOOT_IDT_ENTRIES
+ .quad 0
+ .quad 0
+ .endr
+SYM_DATA_END_LABEL(boot_idt, SYM_L_GLOBAL, boot_idt_end)
+
#ifdef CONFIG_EFI_STUB
SYM_DATA(image_offset, .long 0)
#endif
-
#ifdef CONFIG_EFI_MIXED
SYM_DATA_LOCAL(efi32_boot_args, .long 0, 0, 0)
SYM_DATA(efi_is64, .byte 1)
diff --git a/arch/x86/boot/compressed/idt_64.c b/arch/x86/boot/compressed/idt_64.c
new file mode 100644
index 0000000..082cd6b
--- /dev/null
+++ b/arch/x86/boot/compressed/idt_64.c
@@ -0,0 +1,44 @@
+// SPDX-License-Identifier: GPL-2.0-only
+#include <asm/trap_pf.h>
+#include <asm/segment.h>
+#include <asm/trapnr.h>
+#include "misc.h"
+
+static void set_idt_entry(int vector, void (*handler)(void))
+{
+ unsigned long address = (unsigned long)handler;
+ gate_desc entry;
+
+ memset(&entry, 0, sizeof(entry));
+
+ entry.offset_low = (u16)(address & 0xffff);
+ entry.segment = __KERNEL_CS;
+ entry.bits.type = GATE_TRAP;
+ entry.bits.p = 1;
+ entry.offset_middle = (u16)((address >> 16) & 0xffff);
+ entry.offset_high = (u32)(address >> 32);
+
+ memcpy(&boot_idt[vector], &entry, sizeof(entry));
+}
+
+/* Have this here so we don't need to include <asm/desc.h> */
+static void load_boot_idt(const struct desc_ptr *dtr)
+{
+ asm volatile("lidt %0"::"m" (*dtr));
+}
+
+/* Setup IDT before kernel jumping to .Lrelocated */
+void load_stage1_idt(void)
+{
+ boot_idt_desc.address = (unsigned long)boot_idt;
+
+ load_boot_idt(&boot_idt_desc);
+}
+
+/* Setup IDT after kernel jumping to .Lrelocated */
+void load_stage2_idt(void)
+{
+ boot_idt_desc.address = (unsigned long)boot_idt;
+
+ load_boot_idt(&boot_idt_desc);
+}
diff --git a/arch/x86/boot/compressed/idt_handlers_64.S b/arch/x86/boot/compressed/idt_handlers_64.S
new file mode 100644
index 0000000..36dee2f
--- /dev/null
+++ b/arch/x86/boot/compressed/idt_handlers_64.S
@@ -0,0 +1,70 @@
+/* SPDX-License-Identifier: GPL-2.0-only */
+/*
+ * Early IDT handler entry points
+ *
+ * Copyright (C) 2019 SUSE
+ *
+ * Author: Joerg Roedel <jroedel@suse.de>
+ */
+
+#include <asm/segment.h>
+
+/* For ORIG_RAX */
+#include "../../entry/calling.h"
+
+.macro EXCEPTION_HANDLER name function error_code=0
+SYM_FUNC_START(\name)
+
+ /* Build pt_regs */
+ .if \error_code == 0
+ pushq $0
+ .endif
+
+ pushq %rdi
+ pushq %rsi
+ pushq %rdx
+ pushq %rcx
+ pushq %rax
+ pushq %r8
+ pushq %r9
+ pushq %r10
+ pushq %r11
+ pushq %rbx
+ pushq %rbp
+ pushq %r12
+ pushq %r13
+ pushq %r14
+ pushq %r15
+
+ /* Call handler with pt_regs */
+ movq %rsp, %rdi
+ /* Error code is second parameter */
+ movq ORIG_RAX(%rsp), %rsi
+ call \function
+
+ /* Restore regs */
+ popq %r15
+ popq %r14
+ popq %r13
+ popq %r12
+ popq %rbp
+ popq %rbx
+ popq %r11
+ popq %r10
+ popq %r9
+ popq %r8
+ popq %rax
+ popq %rcx
+ popq %rdx
+ popq %rsi
+ popq %rdi
+
+ /* Remove error code and return */
+ addq $8, %rsp
+
+ iretq
+SYM_FUNC_END(\name)
+ .endm
+
+ .text
+ .code64
diff --git a/arch/x86/boot/compressed/misc.h b/arch/x86/boot/compressed/misc.h
index 3efce27..8feb5f6 100644
--- a/arch/x86/boot/compressed/misc.h
+++ b/arch/x86/boot/compressed/misc.h
@@ -23,6 +23,7 @@
#include <asm/page.h>
#include <asm/boot.h>
#include <asm/bootparam.h>
+#include <asm/desc_defs.h>
#define BOOT_CTYPE_H
#include <linux/acpi.h>
@@ -133,4 +134,8 @@ int count_immovable_mem_regions(void);
static inline int count_immovable_mem_regions(void) { return 0; }
#endif
+/* idt_64.c */
+extern gate_desc boot_idt[BOOT_IDT_ENTRIES];
+extern struct desc_ptr boot_idt_desc;
+
#endif /* BOOT_COMPRESSED_MISC_H */
diff --git a/arch/x86/include/asm/desc_defs.h b/arch/x86/include/asm/desc_defs.h
index a91f3b6..5621fb3 100644
--- a/arch/x86/include/asm/desc_defs.h
+++ b/arch/x86/include/asm/desc_defs.h
@@ -109,6 +109,9 @@ struct desc_ptr {
#endif /* !__ASSEMBLY__ */
+/* Boot IDT definitions */
+#define BOOT_IDT_ENTRIES 32
+
/* Access rights as returned by LAR */
#define AR_TYPE_RODATA (0 * (1 << 9))
#define AR_TYPE_RWDATA (1 * (1 << 9))
^ permalink raw reply related [flat|nested] 264+ messages in thread
* [tip: x86/seves] x86/boot/compressed/64: Disable red-zone usage
2020-09-07 13:15 ` Joerg Roedel
(?)
@ 2020-09-10 9:22 ` tip-bot2 for Joerg Roedel
-1 siblings, 0 replies; 264+ messages in thread
From: tip-bot2 for Joerg Roedel @ 2020-09-10 9:22 UTC (permalink / raw)
To: linux-tip-commits; +Cc: Joerg Roedel, Borislav Petkov, Kees Cook, x86, LKML
The following commit has been merged into the x86/seves branch of tip:
Commit-ID: 6ba0efa46047936afa81460489cfd24bc95dd863
Gitweb: https://git.kernel.org/tip/6ba0efa46047936afa81460489cfd24bc95dd863
Author: Joerg Roedel <jroedel@suse.de>
AuthorDate: Mon, 07 Sep 2020 15:15:13 +02:00
Committer: Borislav Petkov <bp@suse.de>
CommitterDate: Mon, 07 Sep 2020 19:45:25 +02:00
x86/boot/compressed/64: Disable red-zone usage
The x86-64 ABI defines a red-zone on the stack:
The 128-byte area beyond the location pointed to by %rsp is considered
to be reserved and shall not be modified by signal or interrupt
handlers. Therefore, functions may use this area for temporary data
that is not needed across function calls. In particular, leaf
functions may use this area for their entire stack frame, rather than
adjusting the stack pointer in the prologue and epilogue. This area is
known as the red zone.
This is not compatible with exception handling, because the IRET frame
written by the hardware at the stack pointer and the functions to handle
the exception will overwrite the temporary variables of the interrupted
function, causing undefined behavior. So disable red-zones for the
pre-decompression boot code.
Signed-off-by: Joerg Roedel <jroedel@suse.de>
Signed-off-by: Borislav Petkov <bp@suse.de>
Reviewed-by: Kees Cook <keescook@chromium.org>
Link: https://lkml.kernel.org/r/20200907131613.12703-13-joro@8bytes.org
---
arch/x86/boot/compressed/Makefile | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/arch/x86/boot/compressed/Makefile b/arch/x86/boot/compressed/Makefile
index 3962f59..5343079 100644
--- a/arch/x86/boot/compressed/Makefile
+++ b/arch/x86/boot/compressed/Makefile
@@ -32,7 +32,7 @@ KBUILD_CFLAGS := -m$(BITS) -O2
KBUILD_CFLAGS += -fno-strict-aliasing $(call cc-option, -fPIE, -fPIC)
KBUILD_CFLAGS += -DDISABLE_BRANCH_PROFILING
cflags-$(CONFIG_X86_32) := -march=i386
-cflags-$(CONFIG_X86_64) := -mcmodel=small
+cflags-$(CONFIG_X86_64) := -mcmodel=small -mno-red-zone
KBUILD_CFLAGS += $(cflags-y)
KBUILD_CFLAGS += -mno-mmx -mno-sse
KBUILD_CFLAGS += -ffreestanding
^ permalink raw reply related [flat|nested] 264+ messages in thread
* [tip: x86/seves] x86/insn: Add insn_get_modrm_reg_off()
2020-09-07 13:15 ` Joerg Roedel
(?)
@ 2020-09-10 9:22 ` tip-bot2 for Joerg Roedel
-1 siblings, 0 replies; 264+ messages in thread
From: tip-bot2 for Joerg Roedel @ 2020-09-10 9:22 UTC (permalink / raw)
To: linux-tip-commits
Cc: Joerg Roedel, Borislav Petkov, Masami Hiramatsu, x86, LKML
The following commit has been merged into the x86/seves branch of tip:
Commit-ID: 7af1bd822dd45a669fc178a35cc8183922333d56
Gitweb: https://git.kernel.org/tip/7af1bd822dd45a669fc178a35cc8183922333d56
Author: Joerg Roedel <jroedel@suse.de>
AuthorDate: Mon, 07 Sep 2020 15:15:11 +02:00
Committer: Borislav Petkov <bp@suse.de>
CommitterDate: Mon, 07 Sep 2020 19:45:24 +02:00
x86/insn: Add insn_get_modrm_reg_off()
Add a function to the instruction decoder which returns the pt_regs
offset of the register specified in the reg field of the modrm byte.
Signed-off-by: Joerg Roedel <jroedel@suse.de>
Signed-off-by: Borislav Petkov <bp@suse.de>
Acked-by: Masami Hiramatsu <mhiramat@kernel.org>
Link: https://lkml.kernel.org/r/20200907131613.12703-11-joro@8bytes.org
---
arch/x86/include/asm/insn-eval.h | 1 +
arch/x86/lib/insn-eval.c | 23 +++++++++++++++++++++++
2 files changed, 24 insertions(+)
diff --git a/arch/x86/include/asm/insn-eval.h b/arch/x86/include/asm/insn-eval.h
index 392b4fe..f748f57 100644
--- a/arch/x86/include/asm/insn-eval.h
+++ b/arch/x86/include/asm/insn-eval.h
@@ -17,6 +17,7 @@
void __user *insn_get_addr_ref(struct insn *insn, struct pt_regs *regs);
int insn_get_modrm_rm_off(struct insn *insn, struct pt_regs *regs);
+int insn_get_modrm_reg_off(struct insn *insn, struct pt_regs *regs);
unsigned long insn_get_seg_base(struct pt_regs *regs, int seg_reg_idx);
int insn_get_code_seg_params(struct pt_regs *regs);
int insn_fetch_from_user(struct pt_regs *regs,
diff --git a/arch/x86/lib/insn-eval.c b/arch/x86/lib/insn-eval.c
index 2323c85..f20942c 100644
--- a/arch/x86/lib/insn-eval.c
+++ b/arch/x86/lib/insn-eval.c
@@ -20,6 +20,7 @@
enum reg_type {
REG_TYPE_RM = 0,
+ REG_TYPE_REG,
REG_TYPE_INDEX,
REG_TYPE_BASE,
};
@@ -439,6 +440,13 @@ static int get_reg_offset(struct insn *insn, struct pt_regs *regs,
regno += 8;
break;
+ case REG_TYPE_REG:
+ regno = X86_MODRM_REG(insn->modrm.value);
+
+ if (X86_REX_R(insn->rex_prefix.value))
+ regno += 8;
+ break;
+
case REG_TYPE_INDEX:
regno = X86_SIB_INDEX(insn->sib.value);
if (X86_REX_X(insn->rex_prefix.value))
@@ -808,6 +816,21 @@ int insn_get_modrm_rm_off(struct insn *insn, struct pt_regs *regs)
}
/**
+ * insn_get_modrm_reg_off() - Obtain register in reg part of the ModRM byte
+ * @insn: Instruction containing the ModRM byte
+ * @regs: Register values as seen when entering kernel mode
+ *
+ * Returns:
+ *
+ * The register indicated by the reg part of the ModRM byte. The
+ * register is obtained as an offset from the base of pt_regs.
+ */
+int insn_get_modrm_reg_off(struct insn *insn, struct pt_regs *regs)
+{
+ return get_reg_offset(insn, regs, REG_TYPE_REG);
+}
+
+/**
* get_seg_base_limit() - obtain base address and limit of a segment
* @insn: Instruction. Must be valid.
* @regs: Register values as seen when entering kernel mode
^ permalink raw reply related [flat|nested] 264+ messages in thread
* [tip: x86/seves] x86/insn: Add insn_has_rep_prefix() helper
2020-09-07 13:15 ` Joerg Roedel
(?)
@ 2020-09-10 9:22 ` tip-bot2 for Joerg Roedel
-1 siblings, 0 replies; 264+ messages in thread
From: tip-bot2 for Joerg Roedel @ 2020-09-10 9:22 UTC (permalink / raw)
To: linux-tip-commits
Cc: Joerg Roedel, Borislav Petkov, Masami Hiramatsu, x86, LKML
The following commit has been merged into the x86/seves branch of tip:
Commit-ID: 5901781a11175a5e5ee91746ec8627f18d47eebd
Gitweb: https://git.kernel.org/tip/5901781a11175a5e5ee91746ec8627f18d47eebd
Author: Joerg Roedel <jroedel@suse.de>
AuthorDate: Mon, 07 Sep 2020 15:15:12 +02:00
Committer: Borislav Petkov <bp@suse.de>
CommitterDate: Mon, 07 Sep 2020 19:45:25 +02:00
x86/insn: Add insn_has_rep_prefix() helper
Add a function to check whether an instruction has a REP prefix.
Signed-off-by: Joerg Roedel <jroedel@suse.de>
Signed-off-by: Borislav Petkov <bp@suse.de>
Reviewed-by: Masami Hiramatsu <mhiramat@kernel.org>
Link: https://lkml.kernel.org/r/20200907131613.12703-12-joro@8bytes.org
---
arch/x86/include/asm/insn-eval.h | 1 +
arch/x86/lib/insn-eval.c | 24 ++++++++++++++++++++++++
2 files changed, 25 insertions(+)
diff --git a/arch/x86/include/asm/insn-eval.h b/arch/x86/include/asm/insn-eval.h
index f748f57..a0f839a 100644
--- a/arch/x86/include/asm/insn-eval.h
+++ b/arch/x86/include/asm/insn-eval.h
@@ -15,6 +15,7 @@
#define INSN_CODE_SEG_OPND_SZ(params) (params & 0xf)
#define INSN_CODE_SEG_PARAMS(oper_sz, addr_sz) (oper_sz | (addr_sz << 4))
+bool insn_has_rep_prefix(struct insn *insn);
void __user *insn_get_addr_ref(struct insn *insn, struct pt_regs *regs);
int insn_get_modrm_rm_off(struct insn *insn, struct pt_regs *regs);
int insn_get_modrm_reg_off(struct insn *insn, struct pt_regs *regs);
diff --git a/arch/x86/lib/insn-eval.c b/arch/x86/lib/insn-eval.c
index f20942c..58f7fb9 100644
--- a/arch/x86/lib/insn-eval.c
+++ b/arch/x86/lib/insn-eval.c
@@ -54,6 +54,30 @@ static bool is_string_insn(struct insn *insn)
}
/**
+ * insn_has_rep_prefix() - Determine if instruction has a REP prefix
+ * @insn: Instruction containing the prefix to inspect
+ *
+ * Returns:
+ *
+ * true if the instruction has a REP prefix, false if not.
+ */
+bool insn_has_rep_prefix(struct insn *insn)
+{
+ int i;
+
+ insn_get_prefixes(insn);
+
+ for (i = 0; i < insn->prefixes.nbytes; i++) {
+ insn_byte_t p = insn->prefixes.bytes[i];
+
+ if (p == 0xf2 || p == 0xf3)
+ return true;
+ }
+
+ return false;
+}
+
+/**
* get_seg_reg_override_idx() - obtain segment register override index
* @insn: Valid instruction with segment override prefixes
*
^ permalink raw reply related [flat|nested] 264+ messages in thread
* [tip: x86/seves] x86/insn: Make inat-tables.c suitable for pre-decompression code
2020-09-07 13:15 ` Joerg Roedel
(?)
@ 2020-09-10 9:22 ` tip-bot2 for Joerg Roedel
-1 siblings, 0 replies; 264+ messages in thread
From: tip-bot2 for Joerg Roedel @ 2020-09-10 9:22 UTC (permalink / raw)
To: linux-tip-commits
Cc: Joerg Roedel, Borislav Petkov, Masami Hiramatsu, x86, LKML
The following commit has been merged into the x86/seves branch of tip:
Commit-ID: 05a2ae7c033ee30f25fbed3ceed549a5cac398a9
Gitweb: https://git.kernel.org/tip/05a2ae7c033ee30f25fbed3ceed549a5cac398a9
Author: Joerg Roedel <jroedel@suse.de>
AuthorDate: Mon, 07 Sep 2020 15:15:08 +02:00
Committer: Borislav Petkov <bp@suse.de>
CommitterDate: Mon, 07 Sep 2020 19:45:24 +02:00
x86/insn: Make inat-tables.c suitable for pre-decompression code
The inat-tables.c file has some arrays in it that contain pointers to
other arrays. These pointers need to be relocated when the kernel
image is moved to a different location.
The pre-decompression boot-code has no support for applying ELF
relocations, so initialize these arrays at runtime in the
pre-decompression code to make sure all pointers are correctly
initialized.
Signed-off-by: Joerg Roedel <jroedel@suse.de>
Signed-off-by: Borislav Petkov <bp@suse.de>
Acked-by: Masami Hiramatsu <mhiramat@kernel.org>
Link: https://lkml.kernel.org/r/20200907131613.12703-8-joro@8bytes.org
---
arch/x86/tools/gen-insn-attr-x86.awk | 50 ++++++++++++++++++++-
tools/arch/x86/tools/gen-insn-attr-x86.awk | 50 ++++++++++++++++++++-
2 files changed, 98 insertions(+), 2 deletions(-)
diff --git a/arch/x86/tools/gen-insn-attr-x86.awk b/arch/x86/tools/gen-insn-attr-x86.awk
index a42015b..af38469 100644
--- a/arch/x86/tools/gen-insn-attr-x86.awk
+++ b/arch/x86/tools/gen-insn-attr-x86.awk
@@ -362,6 +362,9 @@ function convert_operands(count,opnd, i,j,imm,mod)
END {
if (awkchecked != "")
exit 1
+
+ print "#ifndef __BOOT_COMPRESSED\n"
+
# print escape opcode map's array
print "/* Escape opcode map array */"
print "const insn_attr_t * const inat_escape_tables[INAT_ESC_MAX + 1]" \
@@ -388,6 +391,51 @@ END {
for (j = 0; j < max_lprefix; j++)
if (atable[i,j])
print " ["i"]["j"] = "atable[i,j]","
- print "};"
+ print "};\n"
+
+ print "#else /* !__BOOT_COMPRESSED */\n"
+
+ print "/* Escape opcode map array */"
+ print "static const insn_attr_t *inat_escape_tables[INAT_ESC_MAX + 1]" \
+ "[INAT_LSTPFX_MAX + 1];"
+ print ""
+
+ print "/* Group opcode map array */"
+ print "static const insn_attr_t *inat_group_tables[INAT_GRP_MAX + 1]"\
+ "[INAT_LSTPFX_MAX + 1];"
+ print ""
+
+ print "/* AVX opcode map array */"
+ print "static const insn_attr_t *inat_avx_tables[X86_VEX_M_MAX + 1]"\
+ "[INAT_LSTPFX_MAX + 1];"
+ print ""
+
+ print "static void inat_init_tables(void)"
+ print "{"
+
+ # print escape opcode map's array
+ print "\t/* Print Escape opcode map array */"
+ for (i = 0; i < geid; i++)
+ for (j = 0; j < max_lprefix; j++)
+ if (etable[i,j])
+ print "\tinat_escape_tables["i"]["j"] = "etable[i,j]";"
+ print ""
+
+ # print group opcode map's array
+ print "\t/* Print Group opcode map array */"
+ for (i = 0; i < ggid; i++)
+ for (j = 0; j < max_lprefix; j++)
+ if (gtable[i,j])
+ print "\tinat_group_tables["i"]["j"] = "gtable[i,j]";"
+ print ""
+ # print AVX opcode map's array
+ print "\t/* Print AVX opcode map array */"
+ for (i = 0; i < gaid; i++)
+ for (j = 0; j < max_lprefix; j++)
+ if (atable[i,j])
+ print "\tinat_avx_tables["i"]["j"] = "atable[i,j]";"
+
+ print "}"
+ print "#endif"
}
diff --git a/tools/arch/x86/tools/gen-insn-attr-x86.awk b/tools/arch/x86/tools/gen-insn-attr-x86.awk
index a42015b..af38469 100644
--- a/tools/arch/x86/tools/gen-insn-attr-x86.awk
+++ b/tools/arch/x86/tools/gen-insn-attr-x86.awk
@@ -362,6 +362,9 @@ function convert_operands(count,opnd, i,j,imm,mod)
END {
if (awkchecked != "")
exit 1
+
+ print "#ifndef __BOOT_COMPRESSED\n"
+
# print escape opcode map's array
print "/* Escape opcode map array */"
print "const insn_attr_t * const inat_escape_tables[INAT_ESC_MAX + 1]" \
@@ -388,6 +391,51 @@ END {
for (j = 0; j < max_lprefix; j++)
if (atable[i,j])
print " ["i"]["j"] = "atable[i,j]","
- print "};"
+ print "};\n"
+
+ print "#else /* !__BOOT_COMPRESSED */\n"
+
+ print "/* Escape opcode map array */"
+ print "static const insn_attr_t *inat_escape_tables[INAT_ESC_MAX + 1]" \
+ "[INAT_LSTPFX_MAX + 1];"
+ print ""
+
+ print "/* Group opcode map array */"
+ print "static const insn_attr_t *inat_group_tables[INAT_GRP_MAX + 1]"\
+ "[INAT_LSTPFX_MAX + 1];"
+ print ""
+
+ print "/* AVX opcode map array */"
+ print "static const insn_attr_t *inat_avx_tables[X86_VEX_M_MAX + 1]"\
+ "[INAT_LSTPFX_MAX + 1];"
+ print ""
+
+ print "static void inat_init_tables(void)"
+ print "{"
+
+ # print escape opcode map's array
+ print "\t/* Print Escape opcode map array */"
+ for (i = 0; i < geid; i++)
+ for (j = 0; j < max_lprefix; j++)
+ if (etable[i,j])
+ print "\tinat_escape_tables["i"]["j"] = "etable[i,j]";"
+ print ""
+
+ # print group opcode map's array
+ print "\t/* Print Group opcode map array */"
+ for (i = 0; i < ggid; i++)
+ for (j = 0; j < max_lprefix; j++)
+ if (gtable[i,j])
+ print "\tinat_group_tables["i"]["j"] = "gtable[i,j]";"
+ print ""
+ # print AVX opcode map's array
+ print "\t/* Print AVX opcode map array */"
+ for (i = 0; i < gaid; i++)
+ for (j = 0; j < max_lprefix; j++)
+ if (atable[i,j])
+ print "\tinat_avx_tables["i"]["j"] = "atable[i,j]";"
+
+ print "}"
+ print "#endif"
}
^ permalink raw reply related [flat|nested] 264+ messages in thread
* [tip: x86/seves] x86/umip: Factor out instruction decoding
2020-09-07 13:15 ` Joerg Roedel
(?)
@ 2020-09-10 9:22 ` tip-bot2 for Joerg Roedel
-1 siblings, 0 replies; 264+ messages in thread
From: tip-bot2 for Joerg Roedel @ 2020-09-10 9:22 UTC (permalink / raw)
To: linux-tip-commits; +Cc: Joerg Roedel, Borislav Petkov, x86, LKML
The following commit has been merged into the x86/seves branch of tip:
Commit-ID: 172639d79977ca7b5ce6f84f6606262f4081718f
Gitweb: https://git.kernel.org/tip/172639d79977ca7b5ce6f84f6606262f4081718f
Author: Joerg Roedel <jroedel@suse.de>
AuthorDate: Mon, 07 Sep 2020 15:15:10 +02:00
Committer: Borislav Petkov <bp@suse.de>
CommitterDate: Mon, 07 Sep 2020 19:45:24 +02:00
x86/umip: Factor out instruction decoding
Factor out the code used to decode an instruction with the correct
address and operand sizes to a helper function.
No functional changes.
Signed-off-by: Joerg Roedel <jroedel@suse.de>
Signed-off-by: Borislav Petkov <bp@suse.de>
Link: https://lkml.kernel.org/r/20200907131613.12703-10-joro@8bytes.org
---
arch/x86/include/asm/insn-eval.h | 2 +-
arch/x86/kernel/umip.c | 23 +----------------
arch/x86/lib/insn-eval.c | 45 +++++++++++++++++++++++++++++++-
3 files changed, 48 insertions(+), 22 deletions(-)
diff --git a/arch/x86/include/asm/insn-eval.h b/arch/x86/include/asm/insn-eval.h
index b8b9ef1..392b4fe 100644
--- a/arch/x86/include/asm/insn-eval.h
+++ b/arch/x86/include/asm/insn-eval.h
@@ -21,5 +21,7 @@ unsigned long insn_get_seg_base(struct pt_regs *regs, int seg_reg_idx);
int insn_get_code_seg_params(struct pt_regs *regs);
int insn_fetch_from_user(struct pt_regs *regs,
unsigned char buf[MAX_INSN_SIZE]);
+bool insn_decode(struct insn *insn, struct pt_regs *regs,
+ unsigned char buf[MAX_INSN_SIZE], int buf_size);
#endif /* _ASM_X86_INSN_EVAL_H */
diff --git a/arch/x86/kernel/umip.c b/arch/x86/kernel/umip.c
index ad135be..f6225bf 100644
--- a/arch/x86/kernel/umip.c
+++ b/arch/x86/kernel/umip.c
@@ -342,7 +342,6 @@ bool fixup_umip_exception(struct pt_regs *regs)
unsigned long *reg_addr;
void __user *uaddr;
struct insn insn;
- int seg_defs;
if (!regs)
return false;
@@ -357,27 +356,7 @@ bool fixup_umip_exception(struct pt_regs *regs)
if (!nr_copied)
return false;
- insn_init(&insn, buf, nr_copied, user_64bit_mode(regs));
-
- /*
- * Override the default operand and address sizes with what is specified
- * in the code segment descriptor. The instruction decoder only sets
- * the address size it to either 4 or 8 address bytes and does nothing
- * for the operand bytes. This OK for most of the cases, but we could
- * have special cases where, for instance, a 16-bit code segment
- * descriptor is used.
- * If there is an address override prefix, the instruction decoder
- * correctly updates these values, even for 16-bit defaults.
- */
- seg_defs = insn_get_code_seg_params(regs);
- if (seg_defs == -EINVAL)
- return false;
-
- insn.addr_bytes = INSN_CODE_SEG_ADDR_SZ(seg_defs);
- insn.opnd_bytes = INSN_CODE_SEG_OPND_SZ(seg_defs);
-
- insn_get_length(&insn);
- if (nr_copied < insn.length)
+ if (!insn_decode(&insn, regs, buf, nr_copied))
return false;
umip_inst = identify_insn(&insn);
diff --git a/arch/x86/lib/insn-eval.c b/arch/x86/lib/insn-eval.c
index 947b7f1..2323c85 100644
--- a/arch/x86/lib/insn-eval.c
+++ b/arch/x86/lib/insn-eval.c
@@ -1405,3 +1405,48 @@ int insn_fetch_from_user(struct pt_regs *regs, unsigned char buf[MAX_INSN_SIZE])
return MAX_INSN_SIZE - not_copied;
}
+
+/**
+ * insn_decode() - Decode an instruction
+ * @insn: Structure to store decoded instruction
+ * @regs: Structure with register values as seen when entering kernel mode
+ * @buf: Buffer containing the instruction bytes
+ * @buf_size: Number of instruction bytes available in buf
+ *
+ * Decodes the instruction provided in buf and stores the decoding results in
+ * insn. Also determines the correct address and operand sizes.
+ *
+ * Returns:
+ *
+ * True if instruction was decoded, False otherwise.
+ */
+bool insn_decode(struct insn *insn, struct pt_regs *regs,
+ unsigned char buf[MAX_INSN_SIZE], int buf_size)
+{
+ int seg_defs;
+
+ insn_init(insn, buf, buf_size, user_64bit_mode(regs));
+
+ /*
+ * Override the default operand and address sizes with what is specified
+ * in the code segment descriptor. The instruction decoder only sets
+ * the address size it to either 4 or 8 address bytes and does nothing
+ * for the operand bytes. This OK for most of the cases, but we could
+ * have special cases where, for instance, a 16-bit code segment
+ * descriptor is used.
+ * If there is an address override prefix, the instruction decoder
+ * correctly updates these values, even for 16-bit defaults.
+ */
+ seg_defs = insn_get_code_seg_params(regs);
+ if (seg_defs == -EINVAL)
+ return false;
+
+ insn->addr_bytes = INSN_CODE_SEG_ADDR_SZ(seg_defs);
+ insn->opnd_bytes = INSN_CODE_SEG_OPND_SZ(seg_defs);
+
+ insn_get_length(insn);
+ if (buf_size < insn->length)
+ return false;
+
+ return true;
+}
^ permalink raw reply related [flat|nested] 264+ messages in thread
* [tip: x86/seves] x86/umip: Factor out instruction fetch
2020-09-07 13:15 ` Joerg Roedel
(?)
@ 2020-09-10 9:22 ` tip-bot2 for Joerg Roedel
-1 siblings, 0 replies; 264+ messages in thread
From: tip-bot2 for Joerg Roedel @ 2020-09-10 9:22 UTC (permalink / raw)
To: linux-tip-commits; +Cc: Joerg Roedel, Borislav Petkov, x86, LKML
The following commit has been merged into the x86/seves branch of tip:
Commit-ID: 172b75e56b08846e6fb07a88e5685ce4e24f4620
Gitweb: https://git.kernel.org/tip/172b75e56b08846e6fb07a88e5685ce4e24f4620
Author: Joerg Roedel <jroedel@suse.de>
AuthorDate: Mon, 07 Sep 2020 15:15:09 +02:00
Committer: Borislav Petkov <bp@suse.de>
CommitterDate: Mon, 07 Sep 2020 19:45:24 +02:00
x86/umip: Factor out instruction fetch
Factor out the code to fetch the instruction from user-space to a helper
function.
No functional changes.
Signed-off-by: Joerg Roedel <jroedel@suse.de>
Signed-off-by: Borislav Petkov <bp@suse.de>
Link: https://lkml.kernel.org/r/20200907131613.12703-9-joro@8bytes.org
---
arch/x86/include/asm/insn-eval.h | 2 ++-
arch/x86/kernel/umip.c | 26 ++++-----------------
arch/x86/lib/insn-eval.c | 38 +++++++++++++++++++++++++++++++-
3 files changed, 46 insertions(+), 20 deletions(-)
diff --git a/arch/x86/include/asm/insn-eval.h b/arch/x86/include/asm/insn-eval.h
index 2b6ccf2..b8b9ef1 100644
--- a/arch/x86/include/asm/insn-eval.h
+++ b/arch/x86/include/asm/insn-eval.h
@@ -19,5 +19,7 @@ void __user *insn_get_addr_ref(struct insn *insn, struct pt_regs *regs);
int insn_get_modrm_rm_off(struct insn *insn, struct pt_regs *regs);
unsigned long insn_get_seg_base(struct pt_regs *regs, int seg_reg_idx);
int insn_get_code_seg_params(struct pt_regs *regs);
+int insn_fetch_from_user(struct pt_regs *regs,
+ unsigned char buf[MAX_INSN_SIZE]);
#endif /* _ASM_X86_INSN_EVAL_H */
diff --git a/arch/x86/kernel/umip.c b/arch/x86/kernel/umip.c
index 2c304fd..ad135be 100644
--- a/arch/x86/kernel/umip.c
+++ b/arch/x86/kernel/umip.c
@@ -335,11 +335,11 @@ static void force_sig_info_umip_fault(void __user *addr, struct pt_regs *regs)
*/
bool fixup_umip_exception(struct pt_regs *regs)
{
- int not_copied, nr_copied, reg_offset, dummy_data_size, umip_inst;
- unsigned long seg_base = 0, *reg_addr;
+ int nr_copied, reg_offset, dummy_data_size, umip_inst;
/* 10 bytes is the maximum size of the result of UMIP instructions */
unsigned char dummy_data[10] = { 0 };
unsigned char buf[MAX_INSN_SIZE];
+ unsigned long *reg_addr;
void __user *uaddr;
struct insn insn;
int seg_defs;
@@ -347,26 +347,12 @@ bool fixup_umip_exception(struct pt_regs *regs)
if (!regs)
return false;
- /*
- * If not in user-space long mode, a custom code segment could be in
- * use. This is true in protected mode (if the process defined a local
- * descriptor table), or virtual-8086 mode. In most of the cases
- * seg_base will be zero as in USER_CS.
- */
- if (!user_64bit_mode(regs))
- seg_base = insn_get_seg_base(regs, INAT_SEG_REG_CS);
-
- if (seg_base == -1L)
- return false;
-
- not_copied = copy_from_user(buf, (void __user *)(seg_base + regs->ip),
- sizeof(buf));
- nr_copied = sizeof(buf) - not_copied;
+ nr_copied = insn_fetch_from_user(regs, buf);
/*
- * The copy_from_user above could have failed if user code is protected
- * by a memory protection key. Give up on emulation in such a case.
- * Should we issue a page fault?
+ * The insn_fetch_from_user above could have failed if user code
+ * is protected by a memory protection key. Give up on emulation
+ * in such a case. Should we issue a page fault?
*/
if (!nr_copied)
return false;
diff --git a/arch/x86/lib/insn-eval.c b/arch/x86/lib/insn-eval.c
index 5e69603..947b7f1 100644
--- a/arch/x86/lib/insn-eval.c
+++ b/arch/x86/lib/insn-eval.c
@@ -1367,3 +1367,41 @@ void __user *insn_get_addr_ref(struct insn *insn, struct pt_regs *regs)
return (void __user *)-1L;
}
}
+
+/**
+ * insn_fetch_from_user() - Copy instruction bytes from user-space memory
+ * @regs: Structure with register values as seen when entering kernel mode
+ * @buf: Array to store the fetched instruction
+ *
+ * Gets the linear address of the instruction and copies the instruction bytes
+ * to the buf.
+ *
+ * Returns:
+ *
+ * Number of instruction bytes copied.
+ *
+ * 0 if nothing was copied.
+ */
+int insn_fetch_from_user(struct pt_regs *regs, unsigned char buf[MAX_INSN_SIZE])
+{
+ unsigned long seg_base = 0;
+ int not_copied;
+
+ /*
+ * If not in user-space long mode, a custom code segment could be in
+ * use. This is true in protected mode (if the process defined a local
+ * descriptor table), or virtual-8086 mode. In most of the cases
+ * seg_base will be zero as in USER_CS.
+ */
+ if (!user_64bit_mode(regs)) {
+ seg_base = insn_get_seg_base(regs, INAT_SEG_REG_CS);
+ if (seg_base == -1L)
+ return 0;
+ }
+
+
+ not_copied = copy_from_user(buf, (void __user *)(seg_base + regs->ip),
+ MAX_INSN_SIZE);
+
+ return MAX_INSN_SIZE - not_copied;
+}
^ permalink raw reply related [flat|nested] 264+ messages in thread
* [tip: x86/seves] x86/cpufeatures: Add SEV-ES CPU feature
2020-09-07 13:15 ` Joerg Roedel
(?)
@ 2020-09-10 9:22 ` tip-bot2 for Tom Lendacky
-1 siblings, 0 replies; 264+ messages in thread
From: tip-bot2 for Tom Lendacky @ 2020-09-10 9:22 UTC (permalink / raw)
To: linux-tip-commits; +Cc: Tom Lendacky, Joerg Roedel, Borislav Petkov, x86, LKML
The following commit has been merged into the x86/seves branch of tip:
Commit-ID: 360e7c5c4ca4fd8e627781ed42f95d58bc3bb732
Gitweb: https://git.kernel.org/tip/360e7c5c4ca4fd8e627781ed42f95d58bc3bb732
Author: Tom Lendacky <thomas.lendacky@amd.com>
AuthorDate: Mon, 07 Sep 2020 15:15:06 +02:00
Committer: Borislav Petkov <bp@suse.de>
CommitterDate: Mon, 07 Sep 2020 19:45:24 +02:00
x86/cpufeatures: Add SEV-ES CPU feature
Add CPU feature detection for Secure Encrypted Virtualization with
Encrypted State. This feature enhances SEV by also encrypting the
guest register state, making it in-accessible to the hypervisor.
Signed-off-by: Tom Lendacky <thomas.lendacky@amd.com>
Signed-off-by: Joerg Roedel <jroedel@suse.de>
Signed-off-by: Borislav Petkov <bp@suse.de>
Link: https://lkml.kernel.org/r/20200907131613.12703-6-joro@8bytes.org
---
arch/x86/include/asm/cpufeatures.h | 1 +
arch/x86/kernel/cpu/amd.c | 3 ++-
arch/x86/kernel/cpu/scattered.c | 1 +
3 files changed, 4 insertions(+), 1 deletion(-)
diff --git a/arch/x86/include/asm/cpufeatures.h b/arch/x86/include/asm/cpufeatures.h
index 83fc9d3..1205c1b 100644
--- a/arch/x86/include/asm/cpufeatures.h
+++ b/arch/x86/include/asm/cpufeatures.h
@@ -236,6 +236,7 @@
#define X86_FEATURE_EPT_AD ( 8*32+17) /* Intel Extended Page Table access-dirty bit */
#define X86_FEATURE_VMCALL ( 8*32+18) /* "" Hypervisor supports the VMCALL instruction */
#define X86_FEATURE_VMW_VMMCALL ( 8*32+19) /* "" VMware prefers VMMCALL hypercall instruction */
+#define X86_FEATURE_SEV_ES ( 8*32+20) /* AMD Secure Encrypted Virtualization - Encrypted State */
/* Intel-defined CPU features, CPUID level 0x00000007:0 (EBX), word 9 */
#define X86_FEATURE_FSGSBASE ( 9*32+ 0) /* RDFSBASE, WRFSBASE, RDGSBASE, WRGSBASE instructions*/
diff --git a/arch/x86/kernel/cpu/amd.c b/arch/x86/kernel/cpu/amd.c
index dcc3d94..6062ce5 100644
--- a/arch/x86/kernel/cpu/amd.c
+++ b/arch/x86/kernel/cpu/amd.c
@@ -614,7 +614,7 @@ static void early_detect_mem_encrypt(struct cpuinfo_x86 *c)
* If BIOS has not enabled SME then don't advertise the
* SME feature (set in scattered.c).
* For SEV: If BIOS has not enabled SEV then don't advertise the
- * SEV feature (set in scattered.c).
+ * SEV and SEV_ES feature (set in scattered.c).
*
* In all cases, since support for SME and SEV requires long mode,
* don't advertise the feature under CONFIG_X86_32.
@@ -645,6 +645,7 @@ clear_all:
setup_clear_cpu_cap(X86_FEATURE_SME);
clear_sev:
setup_clear_cpu_cap(X86_FEATURE_SEV);
+ setup_clear_cpu_cap(X86_FEATURE_SEV_ES);
}
}
diff --git a/arch/x86/kernel/cpu/scattered.c b/arch/x86/kernel/cpu/scattered.c
index 62b137c..30f3549 100644
--- a/arch/x86/kernel/cpu/scattered.c
+++ b/arch/x86/kernel/cpu/scattered.c
@@ -41,6 +41,7 @@ static const struct cpuid_bit cpuid_bits[] = {
{ X86_FEATURE_MBA, CPUID_EBX, 6, 0x80000008, 0 },
{ X86_FEATURE_SME, CPUID_EAX, 0, 0x8000001f, 0 },
{ X86_FEATURE_SEV, CPUID_EAX, 1, 0x8000001f, 0 },
+ { X86_FEATURE_SEV_ES, CPUID_EAX, 3, 0x8000001f, 0 },
{ 0, 0, 0, 0, 0 }
};
^ permalink raw reply related [flat|nested] 264+ messages in thread
* [tip: x86/seves] x86/traps: Move pf error codes to <asm/trap_pf.h>
2020-09-07 13:15 ` Joerg Roedel
(?)
@ 2020-09-10 9:22 ` tip-bot2 for Joerg Roedel
-1 siblings, 0 replies; 264+ messages in thread
From: tip-bot2 for Joerg Roedel @ 2020-09-10 9:22 UTC (permalink / raw)
To: linux-tip-commits; +Cc: Joerg Roedel, Borislav Petkov, x86, LKML
The following commit has been merged into the x86/seves branch of tip:
Commit-ID: 05a2fdf3230306daee1def019b8f52cd06bd2e48
Gitweb: https://git.kernel.org/tip/05a2fdf3230306daee1def019b8f52cd06bd2e48
Author: Joerg Roedel <jroedel@suse.de>
AuthorDate: Mon, 07 Sep 2020 15:15:07 +02:00
Committer: Borislav Petkov <bp@suse.de>
CommitterDate: Mon, 07 Sep 2020 19:45:24 +02:00
x86/traps: Move pf error codes to <asm/trap_pf.h>
Move the definition of the x86 page-fault error code bits to a new
header file asm/trap_pf.h. This makes it easier to include them into
pre-decompression boot code. No functional changes.
Signed-off-by: Joerg Roedel <jroedel@suse.de>
Signed-off-by: Borislav Petkov <bp@suse.de>
Link: https://lkml.kernel.org/r/20200907131613.12703-7-joro@8bytes.org
---
arch/x86/include/asm/trap_pf.h | 24 ++++++++++++++++++++++++
arch/x86/include/asm/traps.h | 19 +------------------
2 files changed, 25 insertions(+), 18 deletions(-)
create mode 100644 arch/x86/include/asm/trap_pf.h
diff --git a/arch/x86/include/asm/trap_pf.h b/arch/x86/include/asm/trap_pf.h
new file mode 100644
index 0000000..305bc12
--- /dev/null
+++ b/arch/x86/include/asm/trap_pf.h
@@ -0,0 +1,24 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+#ifndef _ASM_X86_TRAP_PF_H
+#define _ASM_X86_TRAP_PF_H
+
+/*
+ * Page fault error code bits:
+ *
+ * bit 0 == 0: no page found 1: protection fault
+ * bit 1 == 0: read access 1: write access
+ * bit 2 == 0: kernel-mode access 1: user-mode access
+ * bit 3 == 1: use of reserved bit detected
+ * bit 4 == 1: fault was an instruction fetch
+ * bit 5 == 1: protection keys block access
+ */
+enum x86_pf_error_code {
+ X86_PF_PROT = 1 << 0,
+ X86_PF_WRITE = 1 << 1,
+ X86_PF_USER = 1 << 2,
+ X86_PF_RSVD = 1 << 3,
+ X86_PF_INSTR = 1 << 4,
+ X86_PF_PK = 1 << 5,
+};
+
+#endif /* _ASM_X86_TRAP_PF_H */
diff --git a/arch/x86/include/asm/traps.h b/arch/x86/include/asm/traps.h
index 714b1a3..6a30835 100644
--- a/arch/x86/include/asm/traps.h
+++ b/arch/x86/include/asm/traps.h
@@ -8,6 +8,7 @@
#include <asm/debugreg.h>
#include <asm/idtentry.h>
#include <asm/siginfo.h> /* TRAP_TRACE, ... */
+#include <asm/trap_pf.h>
#ifdef CONFIG_X86_64
asmlinkage __visible notrace struct pt_regs *sync_regs(struct pt_regs *eregs);
@@ -41,22 +42,4 @@ void __noreturn handle_stack_overflow(const char *message,
unsigned long fault_address);
#endif
-/*
- * Page fault error code bits:
- *
- * bit 0 == 0: no page found 1: protection fault
- * bit 1 == 0: read access 1: write access
- * bit 2 == 0: kernel-mode access 1: user-mode access
- * bit 3 == 1: use of reserved bit detected
- * bit 4 == 1: fault was an instruction fetch
- * bit 5 == 1: protection keys block access
- */
-enum x86_pf_error_code {
- X86_PF_PROT = 1 << 0,
- X86_PF_WRITE = 1 << 1,
- X86_PF_USER = 1 << 2,
- X86_PF_RSVD = 1 << 3,
- X86_PF_INSTR = 1 << 4,
- X86_PF_PK = 1 << 5,
-};
#endif /* _ASM_X86_TRAPS_H */
^ permalink raw reply related [flat|nested] 264+ messages in thread
* [tip: x86/seves] KVM: SVM: Use __packed shorthand
2020-09-07 13:15 ` Joerg Roedel
(?)
@ 2020-09-10 9:22 ` tip-bot2 for Borislav Petkov
-1 siblings, 0 replies; 264+ messages in thread
From: tip-bot2 for Borislav Petkov @ 2020-09-10 9:22 UTC (permalink / raw)
To: linux-tip-commits
Cc: Borislav Petkov, Joerg Roedel, Borislav Petkov, x86, LKML
The following commit has been merged into the x86/seves branch of tip:
Commit-ID: 976bc5e2aceedef13e0ba1f0e6e372a22164aa0c
Gitweb: https://git.kernel.org/tip/976bc5e2aceedef13e0ba1f0e6e372a22164aa0c
Author: Borislav Petkov <bp@alien8.de>
AuthorDate: Mon, 07 Sep 2020 15:15:05 +02:00
Committer: Borislav Petkov <bp@suse.de>
CommitterDate: Mon, 07 Sep 2020 19:45:24 +02:00
KVM: SVM: Use __packed shorthand
Use the shorthand to make it more readable.
No functional changes.
Signed-off-by: Borislav Petkov <bp@alien8.de>
Signed-off-by: Joerg Roedel <jroedel@suse.de>
Signed-off-by: Borislav Petkov <bp@suse.de>
Link: https://lkml.kernel.org/r/20200907131613.12703-5-joro@8bytes.org
---
arch/x86/include/asm/svm.h | 12 ++++++------
1 file changed, 6 insertions(+), 6 deletions(-)
diff --git a/arch/x86/include/asm/svm.h b/arch/x86/include/asm/svm.h
index 06e5258..cf13f9e 100644
--- a/arch/x86/include/asm/svm.h
+++ b/arch/x86/include/asm/svm.h
@@ -150,14 +150,14 @@ struct __attribute__ ((__packed__)) vmcb_control_area {
#define SVM_NESTED_CTL_NP_ENABLE BIT(0)
#define SVM_NESTED_CTL_SEV_ENABLE BIT(1)
-struct __attribute__ ((__packed__)) vmcb_seg {
+struct vmcb_seg {
u16 selector;
u16 attrib;
u32 limit;
u64 base;
-};
+} __packed;
-struct __attribute__ ((__packed__)) vmcb_save_area {
+struct vmcb_save_area {
struct vmcb_seg es;
struct vmcb_seg cs;
struct vmcb_seg ss;
@@ -231,7 +231,7 @@ struct __attribute__ ((__packed__)) vmcb_save_area {
u64 xcr0;
u8 valid_bitmap[16];
u64 x87_state_gpa;
-};
+} __packed;
struct ghcb {
struct vmcb_save_area save;
@@ -256,11 +256,11 @@ static inline void __unused_size_checks(void)
BUILD_BUG_ON(sizeof(struct ghcb) != EXPECTED_GHCB_SIZE);
}
-struct __attribute__ ((__packed__)) vmcb {
+struct vmcb {
struct vmcb_control_area control;
u8 reserved_control[1024 - sizeof(struct vmcb_control_area)];
struct vmcb_save_area save;
-};
+} __packed;
#define SVM_CPUID_FUNC 0x8000000a
^ permalink raw reply related [flat|nested] 264+ messages in thread
* [tip: x86/seves] KVM: SVM: Add GHCB definitions
2020-09-07 13:15 ` Joerg Roedel
(?)
@ 2020-09-10 9:22 ` tip-bot2 for Tom Lendacky
-1 siblings, 0 replies; 264+ messages in thread
From: tip-bot2 for Tom Lendacky @ 2020-09-10 9:22 UTC (permalink / raw)
To: linux-tip-commits; +Cc: Tom Lendacky, Joerg Roedel, Borislav Petkov, x86, LKML
The following commit has been merged into the x86/seves branch of tip:
Commit-ID: d07f46f9f51afc7fc9f021eae19eba3c2e7870ac
Gitweb: https://git.kernel.org/tip/d07f46f9f51afc7fc9f021eae19eba3c2e7870ac
Author: Tom Lendacky <thomas.lendacky@amd.com>
AuthorDate: Mon, 07 Sep 2020 15:15:03 +02:00
Committer: Borislav Petkov <bp@suse.de>
CommitterDate: Mon, 07 Sep 2020 19:45:24 +02:00
KVM: SVM: Add GHCB definitions
Extend the vmcb_safe_area with SEV-ES fields and add a new
'struct ghcb' which will be used for guest-hypervisor communication.
Signed-off-by: Tom Lendacky <thomas.lendacky@amd.com>
Signed-off-by: Joerg Roedel <jroedel@suse.de>
Signed-off-by: Borislav Petkov <bp@suse.de>
Link: https://lkml.kernel.org/r/20200907131613.12703-3-joro@8bytes.org
---
arch/x86/include/asm/svm.h | 51 +++++++++++++++++++++++++++++++++++--
arch/x86/kvm/svm/svm.c | 2 +-
2 files changed, 51 insertions(+), 2 deletions(-)
diff --git a/arch/x86/include/asm/svm.h b/arch/x86/include/asm/svm.h
index 8a1f538..acac55d 100644
--- a/arch/x86/include/asm/svm.h
+++ b/arch/x86/include/asm/svm.h
@@ -200,13 +200,60 @@ struct __attribute__ ((__packed__)) vmcb_save_area {
u64 br_to;
u64 last_excp_from;
u64 last_excp_to;
+
+ /*
+ * The following part of the save area is valid only for
+ * SEV-ES guests when referenced through the GHCB.
+ */
+ u8 reserved_7[104];
+ u64 reserved_8; /* rax already available at 0x01f8 */
+ u64 rcx;
+ u64 rdx;
+ u64 rbx;
+ u64 reserved_9; /* rsp already available at 0x01d8 */
+ u64 rbp;
+ u64 rsi;
+ u64 rdi;
+ u64 r8;
+ u64 r9;
+ u64 r10;
+ u64 r11;
+ u64 r12;
+ u64 r13;
+ u64 r14;
+ u64 r15;
+ u8 reserved_10[16];
+ u64 sw_exit_code;
+ u64 sw_exit_info_1;
+ u64 sw_exit_info_2;
+ u64 sw_scratch;
+ u8 reserved_11[56];
+ u64 xcr0;
+ u8 valid_bitmap[16];
+ u64 x87_state_gpa;
};
+struct ghcb {
+ struct vmcb_save_area save;
+ u8 reserved_save[2048 - sizeof(struct vmcb_save_area)];
+
+ u8 shared_buffer[2032];
+
+ u8 reserved_1[10];
+ u16 protocol_version; /* negotiated SEV-ES/GHCB protocol version */
+ u32 ghcb_usage;
+} __packed;
+
+
+#define EXPECTED_VMCB_SAVE_AREA_SIZE 1032
+#define EXPECTED_VMCB_CONTROL_AREA_SIZE 256
+#define EXPECTED_GHCB_SIZE PAGE_SIZE
static inline void __unused_size_checks(void)
{
- BUILD_BUG_ON(sizeof(struct vmcb_save_area) != 0x298);
- BUILD_BUG_ON(sizeof(struct vmcb_control_area) != 256);
+ BUILD_BUG_ON(sizeof(struct vmcb_save_area) != EXPECTED_VMCB_SAVE_AREA_SIZE);
+ BUILD_BUG_ON(sizeof(struct vmcb_control_area) != EXPECTED_VMCB_CONTROL_AREA_SIZE);
+ BUILD_BUG_ON(sizeof(struct ghcb) != EXPECTED_GHCB_SIZE);
}
struct __attribute__ ((__packed__)) vmcb {
diff --git a/arch/x86/kvm/svm/svm.c b/arch/x86/kvm/svm/svm.c
index 0194336..1db4fdc 100644
--- a/arch/x86/kvm/svm/svm.c
+++ b/arch/x86/kvm/svm/svm.c
@@ -4164,6 +4164,8 @@ static struct kvm_x86_init_ops svm_init_ops __initdata = {
static int __init svm_init(void)
{
+ __unused_size_checks();
+
return kvm_init(&svm_init_ops, sizeof(struct vcpu_svm),
__alignof__(struct vcpu_svm), THIS_MODULE);
}
^ permalink raw reply related [flat|nested] 264+ messages in thread
* [tip: x86/seves] KVM: SVM: Add GHCB Accessor functions
2020-09-07 13:15 ` Joerg Roedel
(?)
@ 2020-09-10 9:22 ` tip-bot2 for Joerg Roedel
-1 siblings, 0 replies; 264+ messages in thread
From: tip-bot2 for Joerg Roedel @ 2020-09-10 9:22 UTC (permalink / raw)
To: linux-tip-commits; +Cc: Joerg Roedel, Borislav Petkov, x86, LKML
The following commit has been merged into the x86/seves branch of tip:
Commit-ID: 3702c2f4eed2188440f65ecdfc89165106fe565d
Gitweb: https://git.kernel.org/tip/3702c2f4eed2188440f65ecdfc89165106fe565d
Author: Joerg Roedel <jroedel@suse.de>
AuthorDate: Mon, 07 Sep 2020 15:15:04 +02:00
Committer: Borislav Petkov <bp@suse.de>
CommitterDate: Mon, 07 Sep 2020 19:45:24 +02:00
KVM: SVM: Add GHCB Accessor functions
Building a correct GHCB for the hypervisor requires setting valid bits
in the GHCB. Simplify that process by providing accessor functions to
set values and to update the valid bitmap and to check the valid bitmap
in KVM.
Signed-off-by: Joerg Roedel <jroedel@suse.de>
Signed-off-by: Borislav Petkov <bp@suse.de>
Link: https://lkml.kernel.org/r/20200907131613.12703-4-joro@8bytes.org
---
arch/x86/include/asm/svm.h | 43 +++++++++++++++++++++++++++++++++++++-
1 file changed, 43 insertions(+)
diff --git a/arch/x86/include/asm/svm.h b/arch/x86/include/asm/svm.h
index acac55d..06e5258 100644
--- a/arch/x86/include/asm/svm.h
+++ b/arch/x86/include/asm/svm.h
@@ -345,4 +345,47 @@ struct __attribute__ ((__packed__)) vmcb {
#define SVM_CR0_SELECTIVE_MASK (X86_CR0_TS | X86_CR0_MP)
+/* GHCB Accessor functions */
+
+#define GHCB_BITMAP_IDX(field) \
+ (offsetof(struct vmcb_save_area, field) / sizeof(u64))
+
+#define DEFINE_GHCB_ACCESSORS(field) \
+ static inline bool ghcb_##field##_is_valid(const struct ghcb *ghcb) \
+ { \
+ return test_bit(GHCB_BITMAP_IDX(field), \
+ (unsigned long *)&ghcb->save.valid_bitmap); \
+ } \
+ \
+ static inline void ghcb_set_##field(struct ghcb *ghcb, u64 value) \
+ { \
+ __set_bit(GHCB_BITMAP_IDX(field), \
+ (unsigned long *)&ghcb->save.valid_bitmap); \
+ ghcb->save.field = value; \
+ }
+
+DEFINE_GHCB_ACCESSORS(cpl)
+DEFINE_GHCB_ACCESSORS(rip)
+DEFINE_GHCB_ACCESSORS(rsp)
+DEFINE_GHCB_ACCESSORS(rax)
+DEFINE_GHCB_ACCESSORS(rcx)
+DEFINE_GHCB_ACCESSORS(rdx)
+DEFINE_GHCB_ACCESSORS(rbx)
+DEFINE_GHCB_ACCESSORS(rbp)
+DEFINE_GHCB_ACCESSORS(rsi)
+DEFINE_GHCB_ACCESSORS(rdi)
+DEFINE_GHCB_ACCESSORS(r8)
+DEFINE_GHCB_ACCESSORS(r9)
+DEFINE_GHCB_ACCESSORS(r10)
+DEFINE_GHCB_ACCESSORS(r11)
+DEFINE_GHCB_ACCESSORS(r12)
+DEFINE_GHCB_ACCESSORS(r13)
+DEFINE_GHCB_ACCESSORS(r14)
+DEFINE_GHCB_ACCESSORS(r15)
+DEFINE_GHCB_ACCESSORS(sw_exit_code)
+DEFINE_GHCB_ACCESSORS(sw_exit_info_1)
+DEFINE_GHCB_ACCESSORS(sw_exit_info_2)
+DEFINE_GHCB_ACCESSORS(sw_scratch)
+DEFINE_GHCB_ACCESSORS(xcr0)
+
#endif
^ permalink raw reply related [flat|nested] 264+ messages in thread
* [tip: x86/seves] KVM: SVM: nested: Don't allocate VMCB structures on stack
2020-09-07 13:15 ` Joerg Roedel
(?)
@ 2020-09-10 9:22 ` tip-bot2 for Joerg Roedel
-1 siblings, 0 replies; 264+ messages in thread
From: tip-bot2 for Joerg Roedel @ 2020-09-10 9:22 UTC (permalink / raw)
To: linux-tip-commits; +Cc: Joerg Roedel, Borislav Petkov, x86, LKML
The following commit has been merged into the x86/seves branch of tip:
Commit-ID: 6ccbd29ade0d159ee1be398dc9defaae567c253d
Gitweb: https://git.kernel.org/tip/6ccbd29ade0d159ee1be398dc9defaae567c253d
Author: Joerg Roedel <jroedel@suse.de>
AuthorDate: Mon, 07 Sep 2020 15:15:02 +02:00
Committer: Borislav Petkov <bp@suse.de>
CommitterDate: Mon, 07 Sep 2020 19:45:24 +02:00
KVM: SVM: nested: Don't allocate VMCB structures on stack
Do not allocate a vmcb_control_area and a vmcb_save_area on the stack,
as these structures will become larger with future extenstions of
SVM and thus the svm_set_nested_state() function will become a too large
stack frame.
Signed-off-by: Joerg Roedel <jroedel@suse.de>
Signed-off-by: Borislav Petkov <bp@suse.de>
Link: https://lkml.kernel.org/r/20200907131613.12703-2-joro@8bytes.org
---
arch/x86/kvm/svm/nested.c | 47 ++++++++++++++++++++++++++------------
1 file changed, 33 insertions(+), 14 deletions(-)
diff --git a/arch/x86/kvm/svm/nested.c b/arch/x86/kvm/svm/nested.c
index fb68467..2803662 100644
--- a/arch/x86/kvm/svm/nested.c
+++ b/arch/x86/kvm/svm/nested.c
@@ -1060,10 +1060,14 @@ static int svm_set_nested_state(struct kvm_vcpu *vcpu,
struct vmcb *hsave = svm->nested.hsave;
struct vmcb __user *user_vmcb = (struct vmcb __user *)
&user_kvm_nested_state->data.svm[0];
- struct vmcb_control_area ctl;
- struct vmcb_save_area save;
+ struct vmcb_control_area *ctl;
+ struct vmcb_save_area *save;
+ int ret;
u32 cr0;
+ BUILD_BUG_ON(sizeof(struct vmcb_control_area) + sizeof(struct vmcb_save_area) >
+ KVM_STATE_NESTED_SVM_VMCB_SIZE);
+
if (kvm_state->format != KVM_STATE_NESTED_FORMAT_SVM)
return -EINVAL;
@@ -1095,13 +1099,22 @@ static int svm_set_nested_state(struct kvm_vcpu *vcpu,
return -EINVAL;
if (kvm_state->size < sizeof(*kvm_state) + KVM_STATE_NESTED_SVM_VMCB_SIZE)
return -EINVAL;
- if (copy_from_user(&ctl, &user_vmcb->control, sizeof(ctl)))
- return -EFAULT;
- if (copy_from_user(&save, &user_vmcb->save, sizeof(save)))
- return -EFAULT;
- if (!nested_vmcb_check_controls(&ctl))
- return -EINVAL;
+ ret = -ENOMEM;
+ ctl = kzalloc(sizeof(*ctl), GFP_KERNEL);
+ save = kzalloc(sizeof(*save), GFP_KERNEL);
+ if (!ctl || !save)
+ goto out_free;
+
+ ret = -EFAULT;
+ if (copy_from_user(ctl, &user_vmcb->control, sizeof(*ctl)))
+ goto out_free;
+ if (copy_from_user(save, &user_vmcb->save, sizeof(*save)))
+ goto out_free;
+
+ ret = -EINVAL;
+ if (!nested_vmcb_check_controls(ctl))
+ goto out_free;
/*
* Processor state contains L2 state. Check that it is
@@ -1109,15 +1122,15 @@ static int svm_set_nested_state(struct kvm_vcpu *vcpu,
*/
cr0 = kvm_read_cr0(vcpu);
if (((cr0 & X86_CR0_CD) == 0) && (cr0 & X86_CR0_NW))
- return -EINVAL;
+ goto out_free;
/*
* Validate host state saved from before VMRUN (see
* nested_svm_check_permissions).
* TODO: validate reserved bits for all saved state.
*/
- if (!(save.cr0 & X86_CR0_PG))
- return -EINVAL;
+ if (!(save->cr0 & X86_CR0_PG))
+ goto out_free;
/*
* All checks done, we can enter guest mode. L1 control fields
@@ -1126,15 +1139,21 @@ static int svm_set_nested_state(struct kvm_vcpu *vcpu,
* contains saved L1 state.
*/
copy_vmcb_control_area(&hsave->control, &svm->vmcb->control);
- hsave->save = save;
+ hsave->save = *save;
svm->nested.vmcb = kvm_state->hdr.svm.vmcb_pa;
- load_nested_vmcb_control(svm, &ctl);
+ load_nested_vmcb_control(svm, ctl);
nested_prepare_vmcb_control(svm);
out_set_gif:
svm_set_gif(svm, !!(kvm_state->flags & KVM_STATE_NESTED_GIF_SET));
- return 0;
+
+ ret = 0;
+out_free:
+ kfree(save);
+ kfree(ctl);
+
+ return ret;
}
struct kvm_x86_nested_ops svm_nested_ops = {
^ permalink raw reply related [flat|nested] 264+ messages in thread
* Re: [PATCH v7 71/72] x86/efi: Add GHCB mappings when SEV-ES is active
2020-09-09 13:49 ` Tom Lendacky
(?)
@ 2020-09-10 12:37 ` Ard Biesheuvel
-1 siblings, 0 replies; 264+ messages in thread
From: Ard Biesheuvel @ 2020-09-10 12:37 UTC (permalink / raw)
To: Tom Lendacky
Cc: Laszlo Ersek, Borislav Petkov, brijesh.singh, Joerg Roedel,
X86 ML, Joerg Roedel, H. Peter Anvin, Andy Lutomirski,
Dave Hansen, Peter Zijlstra, Jiri Slaby, Dan Williams,
Juergen Gross, Kees Cook, David Rientjes, Cfir Cohen,
Erdem Aktas, Masami Hiramatsu, Mike Stunes, Sean Christopherson,
Martin Radev, Linux Kernel Mailing List, kvm, virtualization
On Wed, 9 Sep 2020 at 16:49, Tom Lendacky <thomas.lendacky@amd.com> wrote:
>
> On 9/9/20 7:44 AM, Laszlo Ersek wrote:
> > On 09/09/20 10:27, Ard Biesheuvel wrote:
> >> (adding Laszlo and Brijesh)
> >>
> >> On Tue, 8 Sep 2020 at 20:46, Borislav Petkov <bp@alien8.de> wrote:
> >>>
> >>> + Ard so that he can ack the efi bits.
> >>>
> >>> On Mon, Sep 07, 2020 at 03:16:12PM +0200, Joerg Roedel wrote:
> >>>> From: Tom Lendacky <thomas.lendacky@amd.com>
> >>>>
> >>>> Calling down to EFI runtime services can result in the firmware
> >>>> performing VMGEXIT calls. The firmware is likely to use the GHCB of
> >>>> the OS (e.g., for setting EFI variables),
> >
> > I've had to stare at this for a while.
> >
> > Because, normally a VMGEXIT is supposed to occur like this:
> >
> > - guest does something privileged
> > - resultant non-automatic exit (NAE) injects a #VC exception
> > - exception handler figures out what that privileged thing was
> > - exception handler submits request to hypervisor via GHCB contents plus
> > VMGEXIT instruction
> >
> > Point being, the agent that "owns" the exception handler is supposed to
> > pre-allocate or otherwise provide the GHCB too, for information passing.
> >
> > So... what is the particular NAE that occurs during the execution of
> > UEFI runtime services (at OS runtime)?
> >
> > And assuming it occurs, I'm unsure if the exception handler (IDT) at
> > that point is owned (temporarily?) by the firmware.
> >
> > - If the #VC handler comes from the firmware, then I don't know why it
> > would use the OS's GHCB.
> >
> > - If the #VC handler comes from the OS, then I don't understand why the
> > commit message says "firmware performing VMGEXIT", given that in this
> > case it would be the OS's #VC handler executing VMGEXIT.
> >
> > So, I think the above commit message implies a VMGEXIT *without* a NAE /
> > #VC context. (Because, I fail to interpret the commit message in a NAE /
> > #VC context in any way; see above.)
>
> Correct.
>
> >
> > OK, so let's see where the firmware performs a VMGEXIT *outside* of an
> > exception handler, *while* at OS runtime. There seems to be one, in file
> > "OvmfPkg/QemuFlashFvbServicesRuntimeDxe/QemuFlashDxe.c":
>
> Again, correct. Basically this is what is invoked when setting UEFI variables.
>
> >
> >> VOID
> >> QemuFlashPtrWrite (
> >> IN volatile UINT8 *Ptr,
> >> IN UINT8 Value
> >> )
> >> {
> >> if (MemEncryptSevEsIsEnabled ()) {
> >> MSR_SEV_ES_GHCB_REGISTER Msr;
> >> GHCB *Ghcb;
> >>
> >> Msr.GhcbPhysicalAddress = AsmReadMsr64 (MSR_SEV_ES_GHCB);
> >> Ghcb = Msr.Ghcb;
> >>
> >> //
> >> // Writing to flash is emulated by the hypervisor through the use of write
> >> // protection. This won't work for an SEV-ES guest because the write won't
> >> // be recognized as a true MMIO write, which would result in the required
> >> // #VC exception. Instead, use the the VMGEXIT MMIO write support directly
> >> // to perform the update.
> >> //
> >> VmgInit (Ghcb);
> >> Ghcb->SharedBuffer[0] = Value;
> >> Ghcb->SaveArea.SwScratch = (UINT64) (UINTN) Ghcb->SharedBuffer;
> >> VmgExit (Ghcb, SVM_EXIT_MMIO_WRITE, (UINT64) (UINTN) Ptr, 1);
> >> VmgDone (Ghcb);
> >> } else {
> >> *Ptr = Value;
> >> }
> >> }
> >
> > This function *does* run at OS runtime (as a part of non-volatile UEFI
> > variable writes).
> >
> > And note that, wherever MSR_SEV_ES_GHCB points to at the moment, is used
> > as GHCB.
> >
> > If the guest kernel allocates its own GHCB and writes the allocation
> > address to MSR_SEV_ES_GHCB, then indeed the firmware will use the GHCB
> > of the OS.
> >
> > I reviewed edk2 commit 437eb3f7a8db
> > ("OvmfPkg/QemuFlashFvbServicesRuntimeDxe: Bypass flash detection with
> > SEV-ES", 2020-08-17), but I admit I never thought of the guest OS
> > changing MSR_SEV_ES_GHCB. I'm sorry about that.
> >
> > As long as this driver is running before OS runtime (i.e., during the
> > DXE and BDS phases), MSR_SEV_ES_GHCB is supposed to carry the value we
> > set in "OvmfPkg/PlatformPei/AmdSev.c":
> >
> >> STATIC
> >> VOID
> >> AmdSevEsInitialize (
> >> VOID
> >> )
> >> {
> >> VOID *GhcbBase;
> >> PHYSICAL_ADDRESS GhcbBasePa;
> >> UINTN GhcbPageCount, PageCount;
> >> RETURN_STATUS PcdStatus, DecryptStatus;
> >> IA32_DESCRIPTOR Gdtr;
> >> VOID *Gdt;
> >>
> >> if (!MemEncryptSevEsIsEnabled ()) {
> >> return;
> >> }
> >>
> >> PcdStatus = PcdSetBoolS (PcdSevEsIsEnabled, TRUE);
> >> ASSERT_RETURN_ERROR (PcdStatus);
> >>
> >> //
> >> // Allocate GHCB and per-CPU variable pages.
> >> // Since the pages must survive across the UEFI to OS transition
> >> // make them reserved.
> >> //
> >> GhcbPageCount = mMaxCpuCount * 2;
> >> GhcbBase = AllocateReservedPages (GhcbPageCount);
> >> ASSERT (GhcbBase != NULL);
> >>
> >> GhcbBasePa = (PHYSICAL_ADDRESS)(UINTN) GhcbBase;
> >>
> >> //
> >> // Each vCPU gets two consecutive pages, the first is the GHCB and the
> >> // second is the per-CPU variable page. Loop through the allocation and
> >> // only clear the encryption mask for the GHCB pages.
> >> //
> >> for (PageCount = 0; PageCount < GhcbPageCount; PageCount += 2) {
> >> DecryptStatus = MemEncryptSevClearPageEncMask (
> >> 0,
> >> GhcbBasePa + EFI_PAGES_TO_SIZE (PageCount),
> >> 1,
> >> TRUE
> >> );
> >> ASSERT_RETURN_ERROR (DecryptStatus);
> >> }
> >>
> >> ZeroMem (GhcbBase, EFI_PAGES_TO_SIZE (GhcbPageCount));
> >>
> >> PcdStatus = PcdSet64S (PcdGhcbBase, GhcbBasePa);
> >> ASSERT_RETURN_ERROR (PcdStatus);
> >> PcdStatus = PcdSet64S (PcdGhcbSize, EFI_PAGES_TO_SIZE (GhcbPageCount));
> >> ASSERT_RETURN_ERROR (PcdStatus);
> >>
> >> DEBUG ((DEBUG_INFO,
> >> "SEV-ES is enabled, %lu GHCB pages allocated starting at 0x%p\n",
> >> (UINT64)GhcbPageCount, GhcbBase));
> >>
> >> AsmWriteMsr64 (MSR_SEV_ES_GHCB, GhcbBasePa);
> >
> > So what is the *actual* problem at OS runtime:
> >
> > - Is it that MSR_SEV_ES_GHCB still points at this PEI-phase *reserved*
> > memory allocation (and so when QemuFlashPtrWrite() tries to access it
> > during OS runtime, it doesn't have a runtime mapping for it)?
>
> At this point the GHCB MSR points to the OS GHCB, which isn't mapped by
> the page tables supplied by the OS and used by UEFI.
>
> >
> > - Or is it that the OS actively changes MSR_SEV_ES_GHCB, pointing to a
> > memory area that the OS owns -- and *that* area is what
> > QemuFlashPtrWrite() cannot access at OS runtime?
>
> Correct.
>
> >
> > The first problem statement does *not* seem to apply, given -- again --
> > that the commit message says, "firmware is likely to use the GHCB of the
> > OS".
> >
> > So I think the second problem statement must apply.
> >
> > (I think the "reserved allocation" above is "reserved" only because we
> > want to keep the OS out of it around the ExitBootServices() transition.)
> >
> > Back to the email:
> >
> > On 09/09/20 10:27, Ard Biesheuvel wrote:
> >> On Tue, 8 Sep 2020 at 20:46, Borislav Petkov <bp@alien8.de> wrote:
> >>> On Mon, Sep 07, 2020 at 03:16:12PM +0200, Joerg Roedel wrote:
> >>>> so each GHCB in the system needs to be identity
> >>>> mapped in the EFI page tables, as unencrypted, to avoid page faults.
> >
> > Not sure I agree about this, but at least it seems to confirm my
> > understanding -- apparently the idea is, for the OS, to satisfy
> > QemuFlashPtrWrite() in the firmware, by putting the "expected" mapping
> > -- for wherever MSR_SEV_ES_GHCB is going to point to -- in place.
> >
> >>>>
> >>>> Signed-off-by: Tom Lendacky <thomas.lendacky@amd.com>
> >>>> [ jroedel@suse.de: Moved GHCB mapping loop to sev-es.c ]
> >>>> Signed-off-by: Joerg Roedel <jroedel@suse.de>
> >>
> >>
> >> This looks like it is papering over a more fundamental issue: any
> >> memory region that the firmware itself needs to access during the
> >> execution of runtime services needs to be described in the UEFI memory
> >> map, with the appropriate annotations so that the OS knows it should
> >> include these in the EFI runtime page tables. So why has this been
> >> omitted in this case?
> >
> > So yeah, the issue seems to be that the QemuFlashFvbServicesRuntimeDxe
> > driver does not *own* the GHCB that it attempts to use at OS runtime. It
> > doesn't know where MSR_SEV_ES_GHCB is going to point.
> >
> > Is QemuFlashFvbServicesRuntimeDxe permitted to change MSR_SEV_ES_GHCB
> > *temporarily* at OS runtime?
> >
> > Because, in that case:
> >
> > - QemuFlashFvbServicesRuntimeDxe should allocate a Runtime Services Data
> > block for GHCB when it starts up (if SEV-ES is active),
> >
> > - QemuFlashFvbServicesRuntimeDxe should register a SetVirtualAddressMap
> > handler, and use EfiConvertPointer() from UefiRuntimeLib to convert
> > the "runtime GHCB" address to virtual address, in that handler,
> >
> > - QemuFlashPtrWrite() should call EfiAtRuntime() from UefiRuntimeLib,
> > and if the latter returns TRUE, then (a) use the runtime-converted
> > address for populating the GHCB, and (b) temporarily swap
> > MSR_SEV_ES_GHCB with the address of the self-allocated GHCB. (The MSR
> > needs a *physical* address, so QemuFlashFvbServicesRuntimeDxe would
> > have to remember / retain the original (physical) allocation address
> > too.)
> >
> > If QemuFlashFvbServicesRuntimeDxe is not permitted to change
> > MSR_SEV_ES_GHCB even temporarily (at OS runtime), then I think the
> > approach proposed in this (guest kernel) patch is valid.
> >
> > Let me skim the code below...
> >
> >>
> >>
> >>
> >>>> ---
> >>>> arch/x86/boot/compressed/sev-es.c | 1 +
> >>>> arch/x86/include/asm/sev-es.h | 2 ++
> >>>> arch/x86/kernel/sev-es.c | 30 ++++++++++++++++++++++++++++++
> >>>> arch/x86/platform/efi/efi_64.c | 10 ++++++++++
> >>>> 4 files changed, 43 insertions(+)
> >>>>
> >>>> diff --git a/arch/x86/boot/compressed/sev-es.c b/arch/x86/boot/compressed/sev-es.c
> >>>> index 45702b866c33..0a9a248ca33d 100644
> >>>> --- a/arch/x86/boot/compressed/sev-es.c
> >>>> +++ b/arch/x86/boot/compressed/sev-es.c
> >>>> @@ -12,6 +12,7 @@
> >>>> */
> >>>> #include "misc.h"
> >>>>
> >>>> +#include <asm/pgtable_types.h>
> >>>> #include <asm/sev-es.h>
> >>>> #include <asm/trapnr.h>
> >>>> #include <asm/trap_pf.h>
> >>>> diff --git a/arch/x86/include/asm/sev-es.h b/arch/x86/include/asm/sev-es.h
> >>>> index e919f09ae33c..cf1d957c7091 100644
> >>>> --- a/arch/x86/include/asm/sev-es.h
> >>>> +++ b/arch/x86/include/asm/sev-es.h
> >>>> @@ -102,11 +102,13 @@ static __always_inline void sev_es_nmi_complete(void)
> >>>> if (static_branch_unlikely(&sev_es_enable_key))
> >>>> __sev_es_nmi_complete();
> >>>> }
> >>>> +extern int __init sev_es_efi_map_ghcbs(pgd_t *pgd);
> >>>> #else
> >>>> static inline void sev_es_ist_enter(struct pt_regs *regs) { }
> >>>> static inline void sev_es_ist_exit(void) { }
> >>>> static inline int sev_es_setup_ap_jump_table(struct real_mode_header *rmh) { return 0; }
> >>>> static inline void sev_es_nmi_complete(void) { }
> >>>> +static inline int sev_es_efi_map_ghcbs(pgd_t *pgd) { return 0; }
> >>>> #endif
> >>>>
> >>>> #endif
> >>>> diff --git a/arch/x86/kernel/sev-es.c b/arch/x86/kernel/sev-es.c
> >>>> index 9ab3a4dfecd8..4e2b7e4d9b87 100644
> >>>> --- a/arch/x86/kernel/sev-es.c
> >>>> +++ b/arch/x86/kernel/sev-es.c
> >>>> @@ -491,6 +491,36 @@ int sev_es_setup_ap_jump_table(struct real_mode_header *rmh)
> >>>> return 0;
> >>>> }
> >>>>
> >>>> +/*
> >>>> + * This is needed by the OVMF UEFI firmware which will use whatever it finds in
> >>>> + * the GHCB MSR as its GHCB to talk to the hypervisor. So make sure the per-cpu
> >>>> + * runtime GHCBs used by the kernel are also mapped in the EFI page-table.
> >
> > Yup, this pretty much confirms my suspicion that QemuFlashPtrWrite() is
> > at the center of this.
> >
> > (BTW, I don't think that the runtime services data allocation, in
> > QemuFlashFvbServicesRuntimeDxe, for OS runtime GHCB purposes, would have
> > to be "per CPU". Refer to "Table 35. Rules for Reentry Into Runtime
> > Services" in the UEFI spec -- if one processor is executing
> > SetVariable(), then no other processor must enter SetVariable(). And so
> > we'll have *at most* one VCPU in QemuFlashPtrWrite(), at any time.)
> >
> >>>> + */
> >>>> +int __init sev_es_efi_map_ghcbs(pgd_t *pgd)
> >>>> +{
> >>>> + struct sev_es_runtime_data *data;
> >>>> + unsigned long address, pflags;
> >>>> + int cpu;
> >>>> + u64 pfn;
> >>>> +
> >>>> + if (!sev_es_active())
> >>>> + return 0;
> >>>> +
> >>>> + pflags = _PAGE_NX | _PAGE_RW;
> >>>> +
> >>>> + for_each_possible_cpu(cpu) {
> >>>> + data = per_cpu(runtime_data, cpu);
> >>>> +
> >>>> + address = __pa(&data->ghcb_page);
> >>>> + pfn = address >> PAGE_SHIFT;
> >>>> +
> >>>> + if (kernel_map_pages_in_pgd(pgd, pfn, address, 1, pflags))
> >>>> + return 1;
> >>>> + }
> >>>> +
> >>>> + return 0;
> >>>> +}
> >>>> +
> >>>> static enum es_result vc_handle_msr(struct ghcb *ghcb, struct es_em_ctxt *ctxt)
> >>>> {
> >>>> struct pt_regs *regs = ctxt->regs;
> >>>> diff --git a/arch/x86/platform/efi/efi_64.c b/arch/x86/platform/efi/efi_64.c
> >>>> index 6af4da1149ba..8f5759df7776 100644
> >>>> --- a/arch/x86/platform/efi/efi_64.c
> >>>> +++ b/arch/x86/platform/efi/efi_64.c
> >>>> @@ -47,6 +47,7 @@
> >>>> #include <asm/realmode.h>
> >>>> #include <asm/time.h>
> >>>> #include <asm/pgalloc.h>
> >>>> +#include <asm/sev-es.h>
> >>>>
> >>>> /*
> >>>> * We allocate runtime services regions top-down, starting from -4G, i.e.
> >>>> @@ -229,6 +230,15 @@ int __init efi_setup_page_tables(unsigned long pa_memmap, unsigned num_pages)
> >>>> return 1;
> >>>> }
> >>>>
> >>>> + /*
> >>>> + * When SEV-ES is active, the GHCB as set by the kernel will be used
> >>>> + * by firmware. Create a 1:1 unencrypted mapping for each GHCB.
> >>>> + */
> >>>> + if (sev_es_efi_map_ghcbs(pgd)) {
> >>>> + pr_err("Failed to create 1:1 mapping for the GHCBs!\n");
> >>>> + return 1;
> >>>> + }
> >>>> +
> >>>> /*
> >>>> * When making calls to the firmware everything needs to be 1:1
> >>>> * mapped and addressable with 32-bit pointers. Map the kernel
> >
> > Good point!
> >
> > And it even makes me wonder if the QemuFlashFvbServicesRuntimeDxe
> > approach, with the runtime services data type memory allocation, is
> > feasible at all. Namely, a page's encryption status, under SEV, is
> > controlled through the PTE.
> >
> > And for this particular UEFI runtime area, it would *not* suffice for
> > the OS to just virt-map it. The OS would also have to *decrypt* the area
> > (mark the PTE as "plaintext").
> >
> > In other words, it would be an "unprecedented" PTE for the OS to set up:
> > the PTE would not only map the GVA to GPA, but also mark the area as
> > "plaintext".
> >
> > Otherwise -- if the OS covers *just* the virt-mapping --,
> > QemuFlashFvbServicesRuntimeDxe would populate its own "runtime GHCB"
> > area just fine, but the actual data hitting the host RAM would be
> > encrypted. And so the hypervisor could not interpret the GHCB.
> >
> > *If* QemuFlashFvbServicesRuntimeDxe should not change the kernel-owned
> > PTE at runtime, even temporarily, for marking the GHCB as "plaintext",
> > then the problem is indeed only solvable in the guest kernel, in my
> > opinion.
> >
> > There simply isn't an "architected annotation" for telling the kernel,
> > "virt-map this runtime services data type memory range, *and* mark it as
> > plaintext at the same time".
> >
> > This would be necessary, as both actions affect the exact same PTE, and
> > the firmware is not really allowed to touch the PTE at runtime. But we
> > don't have such a hint.
> >
> >
> > To summarize: for QemuFlashFvbServicesRuntimeDxe to allocate UEFI
> > Runtime Services Data type memory, for its own runtime GHCB, two
> > permissions are necessary (together), at OS runtime:
> >
> > - QemuFlashFvbServicesRuntimeDxe must be allowed to swap MSR_SEV_ES_GHCB
> > temporarily (before executing VMGEXIT),
> >
> > - QemuFlashFvbServicesRuntimeDxe must be allowed to change the OS-owned
> > PTE temporarily (for remapping the GHCB as plaintext, before writing
> > to it).
> >
>
> Amazing summarization Laszlo!
>
Agreed. And thanks a lot for taking the time to do the analysis.
Based on the above,
Acked-by: Ard Biesheuvel <ardb@kernel.org>
Thanks,
^ permalink raw reply [flat|nested] 264+ messages in thread
* [tip: x86/seves] x86/sev-es: Check required CPU features for SEV-ES
2020-09-07 13:16 ` Joerg Roedel
(?)
(?)
@ 2020-09-10 19:52 ` tip-bot2 for Martin Radev
-1 siblings, 0 replies; 264+ messages in thread
From: tip-bot2 for Martin Radev @ 2020-09-10 19:52 UTC (permalink / raw)
To: linux-tip-commits
Cc: Martin Radev, Joerg Roedel, Borislav Petkov, Kees Cook, x86, LKML
The following commit has been merged into the x86/seves branch of tip:
Commit-ID: f5ed777586e08e09c4b6f1e87161a145ee1431cf
Gitweb: https://git.kernel.org/tip/f5ed777586e08e09c4b6f1e87161a145ee1431cf
Author: Martin Radev <martin.b.radev@gmail.com>
AuthorDate: Mon, 07 Sep 2020 15:16:13 +02:00
Committer: Borislav Petkov <bp@suse.de>
CommitterDate: Thu, 10 Sep 2020 21:49:25 +02:00
x86/sev-es: Check required CPU features for SEV-ES
Make sure the machine supports RDRAND, otherwise there is no trusted
source of randomness in the system.
To also check this in the pre-decompression stage, make has_cpuflag()
not depend on CONFIG_RANDOMIZE_BASE anymore.
Signed-off-by: Martin Radev <martin.b.radev@gmail.com>
Signed-off-by: Joerg Roedel <jroedel@suse.de>
Signed-off-by: Borislav Petkov <bp@suse.de>
Reviewed-by: Kees Cook <keescook@chromium.org>
Link: https://lkml.kernel.org/r/20200907131613.12703-73-joro@8bytes.org
---
arch/x86/boot/compressed/cpuflags.c | 4 ----
arch/x86/boot/compressed/misc.h | 5 +++--
arch/x86/boot/compressed/sev-es.c | 3 +++
arch/x86/kernel/sev-es-shared.c | 15 +++++++++++++++
arch/x86/kernel/sev-es.c | 3 +++
5 files changed, 24 insertions(+), 6 deletions(-)
diff --git a/arch/x86/boot/compressed/cpuflags.c b/arch/x86/boot/compressed/cpuflags.c
index 6448a81..0cc1323 100644
--- a/arch/x86/boot/compressed/cpuflags.c
+++ b/arch/x86/boot/compressed/cpuflags.c
@@ -1,6 +1,4 @@
// SPDX-License-Identifier: GPL-2.0
-#ifdef CONFIG_RANDOMIZE_BASE
-
#include "../cpuflags.c"
bool has_cpuflag(int flag)
@@ -9,5 +7,3 @@ bool has_cpuflag(int flag)
return test_bit(flag, cpu.flags);
}
-
-#endif
diff --git a/arch/x86/boot/compressed/misc.h b/arch/x86/boot/compressed/misc.h
index c0e0ffe..6d31f1b 100644
--- a/arch/x86/boot/compressed/misc.h
+++ b/arch/x86/boot/compressed/misc.h
@@ -85,8 +85,6 @@ void choose_random_location(unsigned long input,
unsigned long *output,
unsigned long output_size,
unsigned long *virt_addr);
-/* cpuflags.c */
-bool has_cpuflag(int flag);
#else
static inline void choose_random_location(unsigned long input,
unsigned long input_size,
@@ -97,6 +95,9 @@ static inline void choose_random_location(unsigned long input,
}
#endif
+/* cpuflags.c */
+bool has_cpuflag(int flag);
+
#ifdef CONFIG_X86_64
extern int set_page_decrypted(unsigned long address);
extern int set_page_encrypted(unsigned long address);
diff --git a/arch/x86/boot/compressed/sev-es.c b/arch/x86/boot/compressed/sev-es.c
index 2a6c7c3..954cb27 100644
--- a/arch/x86/boot/compressed/sev-es.c
+++ b/arch/x86/boot/compressed/sev-es.c
@@ -145,6 +145,9 @@ void sev_es_shutdown_ghcb(void)
if (!boot_ghcb)
return;
+ if (!sev_es_check_cpu_features())
+ error("SEV-ES CPU Features missing.");
+
/*
* GHCB Page must be flushed from the cache and mapped encrypted again.
* Otherwise the running kernel will see strange cache effects when
diff --git a/arch/x86/kernel/sev-es-shared.c b/arch/x86/kernel/sev-es-shared.c
index 4be8af2..5f83cca 100644
--- a/arch/x86/kernel/sev-es-shared.c
+++ b/arch/x86/kernel/sev-es-shared.c
@@ -9,6 +9,21 @@
* and is included directly into both code-bases.
*/
+#ifndef __BOOT_COMPRESSED
+#define error(v) pr_err(v)
+#define has_cpuflag(f) boot_cpu_has(f)
+#endif
+
+static bool __init sev_es_check_cpu_features(void)
+{
+ if (!has_cpuflag(X86_FEATURE_RDRAND)) {
+ error("RDRAND instruction not supported - no trusted source of randomness available\n");
+ return false;
+ }
+
+ return true;
+}
+
static void sev_es_terminate(unsigned int reason)
{
u64 val = GHCB_SEV_TERMINATE;
diff --git a/arch/x86/kernel/sev-es.c b/arch/x86/kernel/sev-es.c
index 8cac9f8..6fcfdd3 100644
--- a/arch/x86/kernel/sev-es.c
+++ b/arch/x86/kernel/sev-es.c
@@ -665,6 +665,9 @@ void __init sev_es_init_vc_handling(void)
if (!sev_es_active())
return;
+ if (!sev_es_check_cpu_features())
+ panic("SEV-ES CPU Features missing");
+
/* Enable SEV-ES special handling */
static_branch_enable(&sev_es_enable_key);
^ permalink raw reply related [flat|nested] 264+ messages in thread
* [tip: x86/seves] x86/efi: Add GHCB mappings when SEV-ES is active
2020-09-07 13:16 ` Joerg Roedel
` (2 preceding siblings ...)
(?)
@ 2020-09-10 19:52 ` tip-bot2 for Tom Lendacky
-1 siblings, 0 replies; 264+ messages in thread
From: tip-bot2 for Tom Lendacky @ 2020-09-10 19:52 UTC (permalink / raw)
To: linux-tip-commits
Cc: Tom Lendacky, Joerg Roedel, Borislav Petkov, Ard Biesheuvel, x86, LKML
The following commit has been merged into the x86/seves branch of tip:
Commit-ID: 39336f4ffb2478ad384075cf4ba7ef2e5db2bbd7
Gitweb: https://git.kernel.org/tip/39336f4ffb2478ad384075cf4ba7ef2e5db2bbd7
Author: Tom Lendacky <thomas.lendacky@amd.com>
AuthorDate: Mon, 07 Sep 2020 15:16:12 +02:00
Committer: Borislav Petkov <bp@suse.de>
CommitterDate: Thu, 10 Sep 2020 21:48:50 +02:00
x86/efi: Add GHCB mappings when SEV-ES is active
Calling down to EFI runtime services can result in the firmware
performing VMGEXIT calls. The firmware is likely to use the GHCB of the
OS (e.g., for setting EFI variables), so each GHCB in the system needs
to be identity-mapped in the EFI page tables, as unencrypted, to avoid
page faults.
Signed-off-by: Tom Lendacky <thomas.lendacky@amd.com>
[ jroedel@suse.de: Moved GHCB mapping loop to sev-es.c ]
Signed-off-by: Joerg Roedel <jroedel@suse.de>
Signed-off-by: Borislav Petkov <bp@suse.de>
Acked-by: Ard Biesheuvel <ardb@kernel.org>
Link: https://lkml.kernel.org/r/20200907131613.12703-72-joro@8bytes.org
---
arch/x86/boot/compressed/sev-es.c | 1 +-
arch/x86/include/asm/sev-es.h | 2 ++-
arch/x86/kernel/sev-es.c | 30 ++++++++++++++++++++++++++++++-
arch/x86/platform/efi/efi_64.c | 10 ++++++++++-
4 files changed, 43 insertions(+)
diff --git a/arch/x86/boot/compressed/sev-es.c b/arch/x86/boot/compressed/sev-es.c
index 5f15e58..2a6c7c3 100644
--- a/arch/x86/boot/compressed/sev-es.c
+++ b/arch/x86/boot/compressed/sev-es.c
@@ -12,6 +12,7 @@
*/
#include "misc.h"
+#include <asm/pgtable_types.h>
#include <asm/sev-es.h>
#include <asm/trapnr.h>
#include <asm/trap_pf.h>
diff --git a/arch/x86/include/asm/sev-es.h b/arch/x86/include/asm/sev-es.h
index e919f09..cf1d957 100644
--- a/arch/x86/include/asm/sev-es.h
+++ b/arch/x86/include/asm/sev-es.h
@@ -102,11 +102,13 @@ static __always_inline void sev_es_nmi_complete(void)
if (static_branch_unlikely(&sev_es_enable_key))
__sev_es_nmi_complete();
}
+extern int __init sev_es_efi_map_ghcbs(pgd_t *pgd);
#else
static inline void sev_es_ist_enter(struct pt_regs *regs) { }
static inline void sev_es_ist_exit(void) { }
static inline int sev_es_setup_ap_jump_table(struct real_mode_header *rmh) { return 0; }
static inline void sev_es_nmi_complete(void) { }
+static inline int sev_es_efi_map_ghcbs(pgd_t *pgd) { return 0; }
#endif
#endif
diff --git a/arch/x86/kernel/sev-es.c b/arch/x86/kernel/sev-es.c
index b6518e9..8cac9f8 100644
--- a/arch/x86/kernel/sev-es.c
+++ b/arch/x86/kernel/sev-es.c
@@ -491,6 +491,36 @@ int sev_es_setup_ap_jump_table(struct real_mode_header *rmh)
return 0;
}
+/*
+ * This is needed by the OVMF UEFI firmware which will use whatever it finds in
+ * the GHCB MSR as its GHCB to talk to the hypervisor. So make sure the per-cpu
+ * runtime GHCBs used by the kernel are also mapped in the EFI page-table.
+ */
+int __init sev_es_efi_map_ghcbs(pgd_t *pgd)
+{
+ struct sev_es_runtime_data *data;
+ unsigned long address, pflags;
+ int cpu;
+ u64 pfn;
+
+ if (!sev_es_active())
+ return 0;
+
+ pflags = _PAGE_NX | _PAGE_RW;
+
+ for_each_possible_cpu(cpu) {
+ data = per_cpu(runtime_data, cpu);
+
+ address = __pa(&data->ghcb_page);
+ pfn = address >> PAGE_SHIFT;
+
+ if (kernel_map_pages_in_pgd(pgd, pfn, address, 1, pflags))
+ return 1;
+ }
+
+ return 0;
+}
+
static enum es_result vc_handle_msr(struct ghcb *ghcb, struct es_em_ctxt *ctxt)
{
struct pt_regs *regs = ctxt->regs;
diff --git a/arch/x86/platform/efi/efi_64.c b/arch/x86/platform/efi/efi_64.c
index 6af4da1..8f5759d 100644
--- a/arch/x86/platform/efi/efi_64.c
+++ b/arch/x86/platform/efi/efi_64.c
@@ -47,6 +47,7 @@
#include <asm/realmode.h>
#include <asm/time.h>
#include <asm/pgalloc.h>
+#include <asm/sev-es.h>
/*
* We allocate runtime services regions top-down, starting from -4G, i.e.
@@ -230,6 +231,15 @@ int __init efi_setup_page_tables(unsigned long pa_memmap, unsigned num_pages)
}
/*
+ * When SEV-ES is active, the GHCB as set by the kernel will be used
+ * by firmware. Create a 1:1 unencrypted mapping for each GHCB.
+ */
+ if (sev_es_efi_map_ghcbs(pgd)) {
+ pr_err("Failed to create 1:1 mapping for the GHCBs!\n");
+ return 1;
+ }
+
+ /*
* When making calls to the firmware everything needs to be 1:1
* mapped and addressable with 32-bit pointers. Map the kernel
* text and allocate a new stack because we can't rely on the
^ permalink raw reply related [flat|nested] 264+ messages in thread
* Re: [tip: x86/seves] x86/kvm: Add KVM-specific VMMCALL handling under SEV-ES
[not found] ` <CAAYXXYx=Eq4gYfUqdO7u37VRD_GpPYFQgN=GZySmAMcDc2AM=g@mail.gmail.com>
@ 2020-10-27 23:14 ` Erdem Aktas
2020-10-28 9:49 ` Joerg Roedel
0 siblings, 1 reply; 264+ messages in thread
From: Erdem Aktas @ 2020-10-27 23:14 UTC (permalink / raw)
To: linux-kernel, Lendacky, Thomas
Cc: linux-tip-commits, Joerg Roedel, Borislav Petkov, x86
[resending in plain/text, sorry for double sending]
It seems to me that the kvm_sev_es_hcall_prepare is leaking more
information than it is needed. Is this an expected behavior?
-Erdem
>
> On Thu, Sep 10, 2020 at 2:39 AM tip-bot2 for Tom Lendacky <tip-bot2@linutronix.de> wrote:
>>
>> The following commit has been merged into the x86/seves branch of tip:
>>
>> Commit-ID: 99419b251e5427b89dbfae103d8a2f469efaa4b2
>> Gitweb: https://git.kernel.org/tip/99419b251e5427b89dbfae103d8a2f469efaa4b2
>> Author: Tom Lendacky <thomas.lendacky@amd.com>
>> AuthorDate: Mon, 07 Sep 2020 15:16:04 +02:00
>> Committer: Borislav Petkov <bp@suse.de>
>> CommitterDate: Wed, 09 Sep 2020 11:33:20 +02:00
>>
>> x86/kvm: Add KVM-specific VMMCALL handling under SEV-ES
>>
>> Implement the callbacks to copy the processor state required by KVM to
>> the GHCB.
>>
>> Signed-off-by: Tom Lendacky <thomas.lendacky@amd.com>
>> [ jroedel@suse.de: - Split out of a larger patch
>> - Adapt to different callback functions ]
>> Co-developed-by: Joerg Roedel <jroedel@suse.de>
>> Signed-off-by: Joerg Roedel <jroedel@suse.de>
>> Signed-off-by: Borislav Petkov <bp@suse.de>
>> Link: https://lkml.kernel.org/r/20200907131613.12703-64-joro@8bytes.org
>> ---
>> arch/x86/kernel/kvm.c | 35 +++++++++++++++++++++++++++++------
>> 1 file changed, 29 insertions(+), 6 deletions(-)
>>
>> diff --git a/arch/x86/kernel/kvm.c b/arch/x86/kernel/kvm.c
>> index 08320b0..0f95972 100644
>> --- a/arch/x86/kernel/kvm.c
>> +++ b/arch/x86/kernel/kvm.c
>> @@ -36,6 +36,8 @@
>> #include <asm/hypervisor.h>
>> #include <asm/tlb.h>
>> #include <asm/cpuidle_haltpoll.h>
>> +#include <asm/ptrace.h>
>> +#include <asm/svm.h>
>>
>> DEFINE_STATIC_KEY_FALSE(kvm_async_pf_enabled);
>>
>> @@ -746,13 +748,34 @@ static void __init kvm_init_platform(void)
>> x86_platform.apic_post_init = kvm_apic_init;
>> }
>>
>> +#if defined(CONFIG_AMD_MEM_ENCRYPT)
>> +static void kvm_sev_es_hcall_prepare(struct ghcb *ghcb, struct pt_regs *regs)
>> +{
>> + /* RAX and CPL are already in the GHCB */
>> + ghcb_set_rbx(ghcb, regs->bx);
>> + ghcb_set_rcx(ghcb, regs->cx);
>> + ghcb_set_rdx(ghcb, regs->dx);
>> + ghcb_set_rsi(ghcb, regs->si);
>> +}
>> +
>> +static bool kvm_sev_es_hcall_finish(struct ghcb *ghcb, struct pt_regs *regs)
>> +{
>> + /* No checking of the return state needed */
>> + return true;
>> +}
>> +#endif
>> +
>> const __initconst struct hypervisor_x86 x86_hyper_kvm = {
>> - .name = "KVM",
>> - .detect = kvm_detect,
>> - .type = X86_HYPER_KVM,
>> - .init.guest_late_init = kvm_guest_init,
>> - .init.x2apic_available = kvm_para_available,
>> - .init.init_platform = kvm_init_platform,
>> + .name = "KVM",
>> + .detect = kvm_detect,
>> + .type = X86_HYPER_KVM,
>> + .init.guest_late_init = kvm_guest_init,
>> + .init.x2apic_available = kvm_para_available,
>> + .init.init_platform = kvm_init_platform,
>> +#if defined(CONFIG_AMD_MEM_ENCRYPT)
>> + .runtime.sev_es_hcall_prepare = kvm_sev_es_hcall_prepare,
>> + .runtime.sev_es_hcall_finish = kvm_sev_es_hcall_finish,
>> +#endif
>> };
>>
>> static __init int activate_jump_labels(void)
^ permalink raw reply [flat|nested] 264+ messages in thread
* Re: [tip: x86/seves] x86/vmware: Add VMware-specific handling for VMMCALL under SEV-ES
2020-09-10 9:22 ` [tip: x86/seves] x86/vmware: Add VMware-specific " tip-bot2 for Doug Covelli
@ 2020-10-27 23:19 ` Erdem Aktas
2020-10-28 9:54 ` Joerg Roedel
0 siblings, 1 reply; 264+ messages in thread
From: Erdem Aktas @ 2020-10-27 23:19 UTC (permalink / raw)
To: linux-kernel, Lendacky, Thomas, dcovelli
Cc: linux-tip-commits, Joerg Roedel, Borislav Petkov, x86
Looking at the VMWARE_VMCALL(cmd, eax, ebx, ecx, edx) definition, it
seems to me only 4 registers are required to be shared with
hypervisor. I don't know much about vmware but is not
vmware_sev_es_hcall_prepare expose more registers than needed and also
vmware_sev_es_hcall_finish might let the hypvervisor to modify
additional registers which are not used?
Just checking if this is intentional and what I am missing here.
Thanks
-Erdem
On Thu, Sep 10, 2020 at 2:23 AM tip-bot2 for Doug Covelli
<tip-bot2@linutronix.de> wrote:
>
> The following commit has been merged into the x86/seves branch of tip:
>
> Commit-ID: 1a222de8dcfb903d039810b0823570ee0be4e6c6
> Gitweb: https://git.kernel.org/tip/1a222de8dcfb903d039810b0823570ee0be4e6c6
> Author: Doug Covelli <dcovelli@vmware.com>
> AuthorDate: Mon, 07 Sep 2020 15:16:05 +02:00
> Committer: Borislav Petkov <bp@suse.de>
> CommitterDate: Wed, 09 Sep 2020 11:33:20 +02:00
>
> x86/vmware: Add VMware-specific handling for VMMCALL under SEV-ES
>
> Add VMware-specific handling for #VC faults caused by VMMCALL
> instructions.
>
> Signed-off-by: Doug Covelli <dcovelli@vmware.com>
> Signed-off-by: Tom Lendacky <thomas.lendacky@amd.com>
> [ jroedel@suse.de: - Adapt to different paravirt interface ]
> Co-developed-by: Joerg Roedel <jroedel@suse.de>
> Signed-off-by: Joerg Roedel <jroedel@suse.de>
> Signed-off-by: Borislav Petkov <bp@suse.de>
> Link: https://lkml.kernel.org/r/20200907131613.12703-65-joro@8bytes.org
> ---
> arch/x86/kernel/cpu/vmware.c | 50 +++++++++++++++++++++++++++++++----
> 1 file changed, 45 insertions(+), 5 deletions(-)
>
> diff --git a/arch/x86/kernel/cpu/vmware.c b/arch/x86/kernel/cpu/vmware.c
> index 9b6fafa..924571f 100644
> --- a/arch/x86/kernel/cpu/vmware.c
> +++ b/arch/x86/kernel/cpu/vmware.c
> @@ -33,6 +33,7 @@
> #include <asm/timer.h>
> #include <asm/apic.h>
> #include <asm/vmware.h>
> +#include <asm/svm.h>
>
> #undef pr_fmt
> #define pr_fmt(fmt) "vmware: " fmt
> @@ -476,10 +477,49 @@ static bool __init vmware_legacy_x2apic_available(void)
> (eax & (1 << VMWARE_CMD_LEGACY_X2APIC)) != 0;
> }
>
> +#ifdef CONFIG_AMD_MEM_ENCRYPT
> +static void vmware_sev_es_hcall_prepare(struct ghcb *ghcb,
> + struct pt_regs *regs)
> +{
> + /* Copy VMWARE specific Hypercall parameters to the GHCB */
> + ghcb_set_rip(ghcb, regs->ip);
> + ghcb_set_rbx(ghcb, regs->bx);
> + ghcb_set_rcx(ghcb, regs->cx);
> + ghcb_set_rdx(ghcb, regs->dx);
> + ghcb_set_rsi(ghcb, regs->si);
> + ghcb_set_rdi(ghcb, regs->di);
> + ghcb_set_rbp(ghcb, regs->bp);
> +}
> +
> +static bool vmware_sev_es_hcall_finish(struct ghcb *ghcb, struct pt_regs *regs)
> +{
> + if (!(ghcb_rbx_is_valid(ghcb) &&
> + ghcb_rcx_is_valid(ghcb) &&
> + ghcb_rdx_is_valid(ghcb) &&
> + ghcb_rsi_is_valid(ghcb) &&
> + ghcb_rdi_is_valid(ghcb) &&
> + ghcb_rbp_is_valid(ghcb)))
> + return false;
> +
> + regs->bx = ghcb->save.rbx;
> + regs->cx = ghcb->save.rcx;
> + regs->dx = ghcb->save.rdx;
> + regs->si = ghcb->save.rsi;
> + regs->di = ghcb->save.rdi;
> + regs->bp = ghcb->save.rbp;
> +
> + return true;
> +}
> +#endif
> +
> const __initconst struct hypervisor_x86 x86_hyper_vmware = {
> - .name = "VMware",
> - .detect = vmware_platform,
> - .type = X86_HYPER_VMWARE,
> - .init.init_platform = vmware_platform_setup,
> - .init.x2apic_available = vmware_legacy_x2apic_available,
> + .name = "VMware",
> + .detect = vmware_platform,
> + .type = X86_HYPER_VMWARE,
> + .init.init_platform = vmware_platform_setup,
> + .init.x2apic_available = vmware_legacy_x2apic_available,
> +#ifdef CONFIG_AMD_MEM_ENCRYPT
> + .runtime.sev_es_hcall_prepare = vmware_sev_es_hcall_prepare,
> + .runtime.sev_es_hcall_finish = vmware_sev_es_hcall_finish,
> +#endif
> };
^ permalink raw reply [flat|nested] 264+ messages in thread
* Re: [tip: x86/seves] x86/kvm: Add KVM-specific VMMCALL handling under SEV-ES
2020-10-27 23:14 ` Erdem Aktas
@ 2020-10-28 9:49 ` Joerg Roedel
2020-10-28 18:03 ` Erdem Aktas
0 siblings, 1 reply; 264+ messages in thread
From: Joerg Roedel @ 2020-10-28 9:49 UTC (permalink / raw)
To: Erdem Aktas
Cc: linux-kernel, Lendacky, Thomas, linux-tip-commits, Borislav Petkov, x86
On Tue, Oct 27, 2020 at 04:14:15PM -0700, Erdem Aktas wrote:
> It seems to me that the kvm_sev_es_hcall_prepare is leaking more
> information than it is needed. Is this an expected behavior?
What exactly is leaked? The kvm hypercall uses RAX, RBX, RCX, RDX and
RSI for parameters.
Regards,
Joerg
^ permalink raw reply [flat|nested] 264+ messages in thread
* Re: [tip: x86/seves] x86/vmware: Add VMware-specific handling for VMMCALL under SEV-ES
2020-10-27 23:19 ` Erdem Aktas
@ 2020-10-28 9:54 ` Joerg Roedel
0 siblings, 0 replies; 264+ messages in thread
From: Joerg Roedel @ 2020-10-28 9:54 UTC (permalink / raw)
To: Erdem Aktas, dcovelli
Cc: linux-kernel, Lendacky, Thomas, linux-tip-commits, Borislav Petkov, x86
On Tue, Oct 27, 2020 at 04:19:45PM -0700, Erdem Aktas wrote:
> Looking at the VMWARE_VMCALL(cmd, eax, ebx, ecx, edx) definition, it
> seems to me only 4 registers are required to be shared with
> hypervisor. I don't know much about vmware but is not
> vmware_sev_es_hcall_prepare expose more registers than needed and also
> vmware_sev_es_hcall_finish might let the hypvervisor to modify
> additional registers which are not used?
>
> Just checking if this is intentional and what I am missing here.
Original patch is from Doug, maybe he can clarify what needs to be
shared. I only adapted it to the call-backs.
Doug?
Regards,
Joerg
^ permalink raw reply [flat|nested] 264+ messages in thread
* Re: [tip: x86/seves] x86/kvm: Add KVM-specific VMMCALL handling under SEV-ES
2020-10-28 9:49 ` Joerg Roedel
@ 2020-10-28 18:03 ` Erdem Aktas
2020-10-30 10:23 ` Joerg Roedel
0 siblings, 1 reply; 264+ messages in thread
From: Erdem Aktas @ 2020-10-28 18:03 UTC (permalink / raw)
To: Joerg Roedel
Cc: linux-kernel, Lendacky, Thomas, linux-tip-commits, Borislav Petkov, x86
I might be missing something here but I think what you say is only
correct for the kvm_hypercall4 cases. All other functions use a
smaller number of registers. #VC blindly assumes that all those
registers are used in the vmcall and exposes them. Here are some
examples:
For example in the kvm_hypercall2 only rax, rbx, and rcx should be
exposed. apic_id address is leaked with rdx when this hypercall is
used in kvm_kick_cpu function. RSI is never used. I am not sure what
value will be exposed to VMM in this case:
54 static inline long kvm_hypercall2(unsigned int nr, unsigned long p1,
55 unsigned long p2)
56 {
57 long ret;
58 asm volatile(KVM_HYPERCALL
59 : "=a"(ret)
60 : "a"(nr), "b"(p1), "c"(p2)
61 : "memory");
62 return ret;
63 }
And this function is called in :
820 static void kvm_kick_cpu(int cpu)
821 {
822 int apicid;
823 unsigned long flags = 0;
824
825 apicid = per_cpu(x86_cpu_to_apicid, cpu);
826 kvm_hypercall2(KVM_HC_KICK_CPU, flags, apicid);
827 }
looking to what it is compiled in my machine :
151215 ffffffff8105def0 <kvm_kick_cpu>:
151216 {
151217 ffffffff8105def0: e8 fb 9e ff ff callq
ffffffff81057df0 <__fentry__>
151218 apicid = per_cpu(x86_cpu_to_apicid, cpu);
151219 ffffffff8105def5: 48 63 ff movslq %edi,%rdi
151220 {
151221 ffffffff8105def8: 53 push %rbx
151222 apicid = per_cpu(x86_cpu_to_apicid, cpu);
151223 ffffffff8105def9: 48 c7 c0 58 16 01 00 mov $0x11658,%rax
151224
151225 static inline long kvm_hypercall2(unsigned int nr, unsigned long p1,
151226 unsigned long p2)
151227 {
151228 long ret;
151229 asm volatile(KVM_HYPERCALL
151230 ffffffff8105df00: 31 db xor %ebx,%ebx
151231 ffffffff8105df02: 48 8b 14 fd 00 19 cb mov
-0x7e34e700(,%rdi,8),%rdx
151232 ffffffff8105df09: 81
151233 kvm_hypercall2(KVM_HC_KICK_CPU, flags, apicid);
151234 ffffffff8105df0a: 0f b7 0c 02 movzwl
(%rdx,%rax,1),%ecx
151235 ffffffff8105df0e: b8 05 00 00 00 mov $0x5,%eax
151236 ffffffff8105df13: 0f 01 c1 vmcall
151237 }
151238 ffffffff8105df16: 5b pop %rbx
151239 ffffffff8105df17: c3 retq
Similarly kvm_hypercall1 only need 2 registers to expose:
44 static inline long kvm_hypercall1(unsigned int nr, unsigned long p1)
45 {
46 long ret;
47 asm volatile(KVM_HYPERCALL
48 : "=a"(ret)
49 : "a"(nr), "b"(p1)
50 : "memory");
51 return ret;
52 }
And an example where it is used:
562 static void kvm_smp_send_call_func_ipi(const struct cpumask *mask)
563 {
564 int cpu;
565
566 native_send_call_func_ipi(mask);
567
568 /* Make sure other vCPUs get a chance to run if they need to. */
569 for_each_cpu(cpu, mask) {
570 if (vcpu_is_preempted(cpu)) {
571 kvm_hypercall1(KVM_HC_SCHED_YIELD,
per_cpu(x86_cpu_to_apicid, cpu));
572 break;
573 }
574 }
575 }
If we look at the function decompiled in my platform, here
x86_cpu_to_apicid address is leaked in rdx. RSI also leaks some
information from kvm_smp_send_call_function_ipi function. RCX is not
used so it might include something from a higher caller.
151243 ffffffff8105df20 <kvm_smp_send_call_func_ipi>:
151244 {
151245 ffffffff8105df20: e8 cb 9e ff ff callq
ffffffff81057df0 <__fentry__>
151246 ffffffff8105df25: 53 push %rbx
151247 ffffffff8105df26: 48 89 fb mov %rdi,%rbx
151248 native_send_call_func_ipi(mask);
151249 ffffffff8105df29: e8 a2 45 ff ff callq
ffffffff810524d0 <native_send_call_func_ipi>
151250 for_each_cpu(cpu, mask) {
151251 ffffffff8105df2e: 41 b8 ff ff ff ff mov $0xffffffff,%r8d
151252 ffffffff8105df34: eb 0e jmp
ffffffff8105df44 <kvm_smp_send_call_func_ipi+0x24>
151253 return PVOP_CALLEE1(bool, lock.vcpu_is_preempted, cpu);
151254 ffffffff8105df36: 49 63 f8 movslq %r8d,%rdi
151255 ffffffff8105df39: ff 14 25 90 93 02 82 callq
*0xffffffff82029390
151256 if (vcpu_is_preempted(cpu)) {
151257 ffffffff8105df40: 84 c0 test %al,%al
151258 ffffffff8105df42: 75 18 jne
ffffffff8105df5c <kvm_smp_send_call_func_ipi+0x3c>
151259 for_each_cpu(cpu, mask) {
151260 ffffffff8105df44: 44 89 c7 mov %r8d,%edi
151261 ffffffff8105df47: 48 89 de mov %rbx,%rsi
151262 ffffffff8105df4a: e8 61 44 39 00 callq
ffffffff813f23b0 <cpumask_next>
151263 ffffffff8105df4f: 3b 05 2f 7e 12 01 cmp
0x1127e2f(%rip),%eax # ffffffff82185d84 <nr_cpu_ids>
151264 ffffffff8105df55: 41 89 c0 mov %eax,%r8d
151265 ffffffff8105df58: 72 dc jb
ffffffff8105df36 <kvm_smp_send_call_func_ipi+0x16>
151266 }
151267 ffffffff8105df5a: 5b pop %rbx
151268 ffffffff8105df5b: c3 retq
151269 kvm_hypercall1(KVM_HC_SCHED_YIELD,
per_cpu(x86_cpu_to_apicid, cpu));
151270 ffffffff8105df5c: 48 8b 14 fd 00 19 cb mov
-0x7e34e700(,%rdi,8),%rdx
151271 ffffffff8105df63: 81
151272 ffffffff8105df64: 48 c7 c0 58 16 01 00 mov $0x11658,%rax
151273 ffffffff8105df6b: 0f b7 1c 02 movzwl
(%rdx,%rax,1),%ebx
151274 asm volatile(KVM_HYPERCALL
151275 ffffffff8105df6f: b8 0b 00 00 00 mov $0xb,%eax
151276 ffffffff8105df74: 0f 01 c1 vmcall
151277 }
I am not sure how those leaked registers can be used, but depending on
which function call hypercall[0-3], there will be some leak.
-Erdem
On Wed, Oct 28, 2020 at 2:49 AM Joerg Roedel <jroedel@suse.de> wrote:
>
> On Tue, Oct 27, 2020 at 04:14:15PM -0700, Erdem Aktas wrote:
> > It seems to me that the kvm_sev_es_hcall_prepare is leaking more
> > information than it is needed. Is this an expected behavior?
>
> What exactly is leaked? The kvm hypercall uses RAX, RBX, RCX, RDX and
> RSI for parameters.
>
> Regards,
>
> Joerg
^ permalink raw reply [flat|nested] 264+ messages in thread
* Re: [tip: x86/seves] x86/kvm: Add KVM-specific VMMCALL handling under SEV-ES
2020-10-28 18:03 ` Erdem Aktas
@ 2020-10-30 10:23 ` Joerg Roedel
0 siblings, 0 replies; 264+ messages in thread
From: Joerg Roedel @ 2020-10-30 10:23 UTC (permalink / raw)
To: Erdem Aktas
Cc: linux-kernel, Lendacky, Thomas, linux-tip-commits, Borislav Petkov, x86
On Wed, Oct 28, 2020 at 11:03:05AM -0700, Erdem Aktas wrote:
> I might be missing something here but I think what you say is only
> correct for the kvm_hypercall4 cases. All other functions use a
> smaller number of registers. #VC blindly assumes that all those
> registers are used in the vmcall and exposes them.
Right, I think we should fix that in the guest and zero out the unused
registers. VMMCALL can come from userspace after all, and the #VC
handler does not look at the hypercall numbers.
Further, on the host side KVM will unconditionally read out all 4
registers too, which requires us to set them valid in the GHCB.
Regards,
Joerg
^ permalink raw reply [flat|nested] 264+ messages in thread
* Re: [PATCH v7 01/72] KVM: SVM: nested: Don't allocate VMCB structures on stack
2020-09-07 13:15 ` Joerg Roedel
(?)
(?)
@ 2020-11-05 16:24 ` Michael Roth
2020-11-05 16:38 ` Borislav Petkov
2020-11-05 17:46 ` Michael Roth
-1 siblings, 2 replies; 264+ messages in thread
From: Michael Roth @ 2020-11-05 16:24 UTC (permalink / raw)
To: Joerg Roedel, x86
Cc: Juergen Gross, Tom Lendacky, Joerg Roedel, Mike Stunes,
Kees Cook, kvm, Peter Zijlstra, Cfir Cohen, Joerg Roedel,
Dave Hansen, linux-kernel, Sean Christopherson, virtualization,
Martin Radev, Masami Hiramatsu, Andy Lutomirski, hpa,
Erdem Aktas, David Rientjes, Dan Williams, Jiri Slaby
Quoting Joerg Roedel (2020-09-07 08:15:02)
> From: Joerg Roedel <jroedel@suse.de>
>
> Do not allocate a vmcb_control_area and a vmcb_save_area on the stack,
> as these structures will become larger with future extenstions of
> SVM and thus the svm_set_nested_state() function will become a too large
> stack frame.
>
> Signed-off-by: Joerg Roedel <jroedel@suse.de>
> ---
> arch/x86/kvm/svm/nested.c | 47 +++++++++++++++++++++++++++------------
> 1 file changed, 33 insertions(+), 14 deletions(-)
>
> diff --git a/arch/x86/kvm/svm/nested.c b/arch/x86/kvm/svm/nested.c
> index fb68467e6049..28036629abf8 100644
> --- a/arch/x86/kvm/svm/nested.c
> +++ b/arch/x86/kvm/svm/nested.c
> @@ -1060,10 +1060,14 @@ static int svm_set_nested_state(struct kvm_vcpu *vcpu,
> struct vmcb *hsave = svm->nested.hsave;
> struct vmcb __user *user_vmcb = (struct vmcb __user *)
> &user_kvm_nested_state->data.svm[0];
> - struct vmcb_control_area ctl;
> - struct vmcb_save_area save;
> + struct vmcb_control_area *ctl;
> + struct vmcb_save_area *save;
> + int ret;
> u32 cr0;
>
> + BUILD_BUG_ON(sizeof(struct vmcb_control_area) + sizeof(struct vmcb_save_area) >
> + KVM_STATE_NESTED_SVM_VMCB_SIZE);
> +
> if (kvm_state->format != KVM_STATE_NESTED_FORMAT_SVM)
> return -EINVAL;
>
> @@ -1095,13 +1099,22 @@ static int svm_set_nested_state(struct kvm_vcpu *vcpu,
> return -EINVAL;
> if (kvm_state->size < sizeof(*kvm_state) + KVM_STATE_NESTED_SVM_VMCB_SIZE)
> return -EINVAL;
> - if (copy_from_user(&ctl, &user_vmcb->control, sizeof(ctl)))
> - return -EFAULT;
> - if (copy_from_user(&save, &user_vmcb->save, sizeof(save)))
> - return -EFAULT;
>
> - if (!nested_vmcb_check_controls(&ctl))
> - return -EINVAL;
> + ret = -ENOMEM;
> + ctl = kzalloc(sizeof(*ctl), GFP_KERNEL);
> + save = kzalloc(sizeof(*save), GFP_KERNEL);
> + if (!ctl || !save)
> + goto out_free;
> +
> + ret = -EFAULT;
> + if (copy_from_user(ctl, &user_vmcb->control, sizeof(*ctl)))
> + goto out_free;
> + if (copy_from_user(save, &user_vmcb->save, sizeof(*save)))
> + goto out_free;
> +
> + ret = -EINVAL;
> + if (!nested_vmcb_check_controls(ctl))
> + goto out_free;
>
> /*
> * Processor state contains L2 state. Check that it is
> @@ -1109,15 +1122,15 @@ static int svm_set_nested_state(struct kvm_vcpu *vcpu,
> */
> cr0 = kvm_read_cr0(vcpu);
> if (((cr0 & X86_CR0_CD) == 0) && (cr0 & X86_CR0_NW))
> - return -EINVAL;
> + goto out_free;
>
> /*
> * Validate host state saved from before VMRUN (see
> * nested_svm_check_permissions).
> * TODO: validate reserved bits for all saved state.
> */
> - if (!(save.cr0 & X86_CR0_PG))
> - return -EINVAL;
> + if (!(save->cr0 & X86_CR0_PG))
> + goto out_free;
>
> /*
> * All checks done, we can enter guest mode. L1 control fields
> @@ -1126,15 +1139,21 @@ static int svm_set_nested_state(struct kvm_vcpu *vcpu,
> * contains saved L1 state.
> */
> copy_vmcb_control_area(&hsave->control, &svm->vmcb->control);
> - hsave->save = save;
> + hsave->save = *save;
>
> svm->nested.vmcb = kvm_state->hdr.svm.vmcb_pa;
> - load_nested_vmcb_control(svm, &ctl);
> + load_nested_vmcb_control(svm, ctl);
> nested_prepare_vmcb_control(svm);
>
> out_set_gif:
> svm_set_gif(svm, !!(kvm_state->flags & KVM_STATE_NESTED_GIF_SET));
> - return 0;
> +
> + ret = 0;
> +out_free:
> + kfree(save);
> + kfree(ctl);
This change seems to trigger a crash via smm-test.c (and state-test.c) KVM
selftest when we call vcpu_load_state->KVM_SET_NESTED_STATE. I think what's
happening is we are hitting the 'goto out_set_gif;' and then attempting to
free save and ctl, which are still uninitialized at that point:
[ 1999.801176] APIC base relocation is unsupported by KVM
[ 1999.828562] BUG: unable to handle page fault for address:
fffff12379020288
[ 1999.841968] #PF: supervisor read access in kernel mode
[ 1999.847693] #PF: error_code(0x0000) - not-present page
[ 1999.853426] PGD 0 P4D 0
[ 1999.856252] Oops: 0000 [#1] SMP NOPTI
[ 1999.861366] CPU: 112 PID: 10162 Comm: smm_test Tainted: G
E 5.9.1-amdsos-build31t0+ #1
[ 1999.871655] Hardware name: AMD Corporation ETHANOL_X/ETHANOL_X, BIOS
RXM0092B 10/27/2020
[ 1999.880694] RIP: 0010:kfree+0x5b/0x3c0
[ 1999.884876] Code: 80 49 01 dc 0f 82 70 03 00 00 48 c7 c0 00 00 00 80
48 2b 05 97 4a 1c 01 49 01 c4 49 c1 ec 0c 49 c1 e4 06 4c 03 25 75 4a 1c
01 <49> 8b 44 24 08 48 8d 50 ff a8 01 4c 0f 45 e2 49 8b 54 24 08 48 8d
[ 1999.906674] RSP: 0018:ffffb7004d9d3cf8 EFLAGS: 00010282
[ 1999.912502] RAX: 0000723d80000000 RBX: 000000004080aebf RCX:
0000000002000000
[ 1999.920464] RDX: 0000000000000000 RSI: 0000000000000000 RDI:
000000004080aebf
[ 1999.929335] RBP: ffffb7004d9d3d28 R08: ffff8de1c2a43628 R09:
0000000000000000
[ 1999.937298] R10: 0000000000000000 R11: 0000000000000000 R12:
fffff12379020280
[ 1999.945258] R13: ffffffffc0f6626a R14: 0000000000000000 R15:
ffffb7004d9d3db0
[ 1999.953221] FS: 00007f231b4c1740(0000) GS:ffff8de20e800000(0000)
knlGS:0000000000000000
[ 1999.963255] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 1999.969667] CR2: fffff12379020288 CR3: 0000001faf5aa006 CR4:
0000000000770ee0
[ 1999.977627] PKRU: 55555554
[ 1999.980644] Call Trace:
[ 1999.983384] svm_leave_nested+0xea/0x2f0 [kvm_amd]
[ 1999.988743] kvm_arch_vcpu_ioctl+0x6fd/0x1260 [kvm]
[ 1999.995026] ? avic_vcpu_load+0x20/0x130 [kvm_amd]
[ 2000.000372] kvm_vcpu_kick+0x705/0xae0 [kvm]
[ 2000.005216] __x64_sys_ioctl+0x91/0xc0
[ 2000.009470] do_syscall_64+0x38/0x90
[ 2000.013461] entry_SYSCALL_64_after_hwframe+0x44/0xa9
[ 2000.019090] RIP: 0033:0x7f231b5db50b
[ 2000.024052] Code: 0f 1e fa 48 8b 05 85 39 0d 00 64 c7 00 26 00 00 00
48 c7 c0 ff ff ff ff c3 66 0f 1f 44 00 00 f3 0f 1e fa b8 10 00 00 00 0f
05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 55 39 0d 00 f7 d8 64 89 01 48
[ 2000.045005] RSP: 002b:00007ffc01de6918 EFLAGS: 00000246 ORIG_RAX:
0000000000000010
[ 2000.053453] RAX: ffffffffffffffda RBX: 0000000002147880 RCX:
00007f231b5db50b
[ 2000.062375] RDX: 0000000002148c98 RSI: 000000004080aebf RDI:
0000000000000009
[ 2000.070335] RBP: 000000000214d0c0 R08: 00000000004103f8 R09:
0000000000000000
[ 2000.078296] R10: 0000000000000000 R11: 0000000000000246 R12:
00007ffc01de6960
[ 2000.087278] R13: 0000000000000002 R14: 00007f231b711000 R15:
0000000002147880
[ 2000.095244] Modules linked in: nls_iso8859_1(E) dm_multipath(E)
scsi_dh_rdac(E) scsi_dh_emc(E) scsi_dh_alua(E) intel_rapl_msr(E)
intel_rapl_common(E) amd64_edac_mod(E) kvm_amd(E) kvm(E) joydev(E)
input_leds(E) crct10dif_pclmul(E) crc32_pclmul(E) ghash_clmulni_intel(E)
aesni_intel(E) crypto_simd(E) cryptd(E) glue_helper(E) hid_generic(E)
efi_pstore(E) rapl(E) ipmi_si(E) ipmi_devintf(E) ipmi_msghandler(E)
usbhid(E) hid(E) ast(E) drm_vram_helper(E) drm_ttm_helper(E) ttm(E)
drm_kms_helper(E) cec(E) i2c_algo_bit(E) fb_sys_fops(E) syscopyarea(E)
sysfillrect(E) sysimgblt(E) wmi_bmof(E) e1000e(E) i2c_piix4(E) ccp(E)
wmi(E) mac_hid(E) sch_fq_codel(E) drm(E) ip_tables(E) x_tables(E)
autofs4(E)
[ 2000.164824] CR2: fffff12379020288
[ 2000.168522] ---[ end trace c5975ced3c660340 ]---
> +
> + return ret;
> }
>
> struct kvm_x86_nested_ops svm_nested_ops = {
> --
> 2.28.0
>
> _______________________________________________
> Virtualization mailing list
> Virtualization@lists.linux-foundation.org
> https://lists.linuxfoundation.org/mailman/listinfo/virtualization
^ permalink raw reply [flat|nested] 264+ messages in thread
* Re: [PATCH v7 01/72] KVM: SVM: nested: Don't allocate VMCB structures on stack
2020-11-05 16:24 ` [PATCH v7 01/72] " Michael Roth
@ 2020-11-05 16:38 ` Borislav Petkov
2020-11-05 17:46 ` Michael Roth
1 sibling, 0 replies; 264+ messages in thread
From: Borislav Petkov @ 2020-11-05 16:38 UTC (permalink / raw)
To: Michael Roth
Cc: Joerg Roedel, x86, Juergen Gross, Tom Lendacky, Joerg Roedel,
Mike Stunes, Kees Cook, kvm, Peter Zijlstra, Cfir Cohen,
Dave Hansen, linux-kernel, Sean Christopherson, virtualization,
Martin Radev, Masami Hiramatsu, Andy Lutomirski, hpa,
Erdem Aktas, David Rientjes, Dan Williams, Jiri Slaby
On Thu, Nov 05, 2020 at 10:24:37AM -0600, Michael Roth wrote:
> > out_set_gif:
> > svm_set_gif(svm, !!(kvm_state->flags & KVM_STATE_NESTED_GIF_SET));
> > - return 0;
> > +
> > + ret = 0;
> > +out_free:
> > + kfree(save);
> > + kfree(ctl);
>
> This change seems to trigger a crash via smm-test.c (and state-test.c) KVM
> selftest when we call vcpu_load_state->KVM_SET_NESTED_STATE. I think what's
> happening is we are hitting the 'goto out_set_gif;'
No out_set_gif upstream anymore after
d5cd6f340145 ("KVM: nSVM: Avoid freeing uninitialized pointers in svm_set_nested_state()")
and it looks like you hit the issue this patch is fixing.
Can you test with the above commit cherrypicked ontop of your what looks
like 5.9.1-ish tree?
If that fixes it, we should route this patch to stable.
Thx.
--
Regards/Gruss,
Boris.
https://people.kernel.org/tglx/notes-about-netiquette
^ permalink raw reply [flat|nested] 264+ messages in thread
* Re: [PATCH v7 01/72] KVM: SVM: nested: Don't allocate VMCB structures on stack
@ 2020-11-05 16:38 ` Borislav Petkov
0 siblings, 0 replies; 264+ messages in thread
From: Borislav Petkov @ 2020-11-05 16:38 UTC (permalink / raw)
To: Michael Roth
Cc: kvm, Peter Zijlstra, Dave Hansen, virtualization, hpa,
Jiri Slaby, Joerg Roedel, x86, David Rientjes, Martin Radev,
Tom Lendacky, Joerg Roedel, Kees Cook, Cfir Cohen,
Andy Lutomirski, Dan Williams, Juergen Gross, Mike Stunes,
linux-kernel, Sean Christopherson, Masami Hiramatsu, Erdem Aktas
On Thu, Nov 05, 2020 at 10:24:37AM -0600, Michael Roth wrote:
> > out_set_gif:
> > svm_set_gif(svm, !!(kvm_state->flags & KVM_STATE_NESTED_GIF_SET));
> > - return 0;
> > +
> > + ret = 0;
> > +out_free:
> > + kfree(save);
> > + kfree(ctl);
>
> This change seems to trigger a crash via smm-test.c (and state-test.c) KVM
> selftest when we call vcpu_load_state->KVM_SET_NESTED_STATE. I think what's
> happening is we are hitting the 'goto out_set_gif;'
No out_set_gif upstream anymore after
d5cd6f340145 ("KVM: nSVM: Avoid freeing uninitialized pointers in svm_set_nested_state()")
and it looks like you hit the issue this patch is fixing.
Can you test with the above commit cherrypicked ontop of your what looks
like 5.9.1-ish tree?
If that fixes it, we should route this patch to stable.
Thx.
--
Regards/Gruss,
Boris.
https://people.kernel.org/tglx/notes-about-netiquette
_______________________________________________
Virtualization mailing list
Virtualization@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/virtualization
^ permalink raw reply [flat|nested] 264+ messages in thread
* Re: [PATCH v7 01/72] KVM: SVM: nested: Don't allocate VMCB structures on stack
2020-11-05 16:24 ` [PATCH v7 01/72] " Michael Roth
2020-11-05 16:38 ` Borislav Petkov
@ 2020-11-05 17:46 ` Michael Roth
1 sibling, 0 replies; 264+ messages in thread
From: Michael Roth @ 2020-11-05 17:46 UTC (permalink / raw)
To: Joerg Roedel, x86
Cc: Juergen Gross, Tom Lendacky, Joerg Roedel, Mike Stunes,
Kees Cook, kvm, Peter Zijlstra, Cfir Cohen, Joerg Roedel,
Dave Hansen, linux-kernel, Sean Christopherson, virtualization,
Martin Radev, Masami Hiramatsu, Andy Lutomirski, hpa,
Erdem Aktas, David Rientjes, Dan Williams, Jiri Slaby
Quoting Michael Roth (2020-11-05 10:24:37)
> Quoting Joerg Roedel (2020-09-07 08:15:02)
> > From: Joerg Roedel <jroedel@suse.de>
> >
> > Do not allocate a vmcb_control_area and a vmcb_save_area on the stack,
> > as these structures will become larger with future extenstions of
> > SVM and thus the svm_set_nested_state() function will become a too large
> > stack frame.
> >
> > Signed-off-by: Joerg Roedel <jroedel@suse.de>
> > ---
> > arch/x86/kvm/svm/nested.c | 47 +++++++++++++++++++++++++++------------
> > 1 file changed, 33 insertions(+), 14 deletions(-)
> >
> > diff --git a/arch/x86/kvm/svm/nested.c b/arch/x86/kvm/svm/nested.c
> > index fb68467e6049..28036629abf8 100644
> > --- a/arch/x86/kvm/svm/nested.c
> > +++ b/arch/x86/kvm/svm/nested.c
> > @@ -1060,10 +1060,14 @@ static int svm_set_nested_state(struct kvm_vcpu *vcpu,
> > struct vmcb *hsave = svm->nested.hsave;
> > struct vmcb __user *user_vmcb = (struct vmcb __user *)
> > &user_kvm_nested_state->data.svm[0];
> > - struct vmcb_control_area ctl;
> > - struct vmcb_save_area save;
> > + struct vmcb_control_area *ctl;
> > + struct vmcb_save_area *save;
> > + int ret;
> > u32 cr0;
> >
> > + BUILD_BUG_ON(sizeof(struct vmcb_control_area) + sizeof(struct vmcb_save_area) >
> > + KVM_STATE_NESTED_SVM_VMCB_SIZE);
> > +
> > if (kvm_state->format != KVM_STATE_NESTED_FORMAT_SVM)
> > return -EINVAL;
> >
> > @@ -1095,13 +1099,22 @@ static int svm_set_nested_state(struct kvm_vcpu *vcpu,
> > return -EINVAL;
> > if (kvm_state->size < sizeof(*kvm_state) + KVM_STATE_NESTED_SVM_VMCB_SIZE)
> > return -EINVAL;
> > - if (copy_from_user(&ctl, &user_vmcb->control, sizeof(ctl)))
> > - return -EFAULT;
> > - if (copy_from_user(&save, &user_vmcb->save, sizeof(save)))
> > - return -EFAULT;
> >
> > - if (!nested_vmcb_check_controls(&ctl))
> > - return -EINVAL;
> > + ret = -ENOMEM;
> > + ctl = kzalloc(sizeof(*ctl), GFP_KERNEL);
> > + save = kzalloc(sizeof(*save), GFP_KERNEL);
> > + if (!ctl || !save)
> > + goto out_free;
> > +
> > + ret = -EFAULT;
> > + if (copy_from_user(ctl, &user_vmcb->control, sizeof(*ctl)))
> > + goto out_free;
> > + if (copy_from_user(save, &user_vmcb->save, sizeof(*save)))
> > + goto out_free;
> > +
> > + ret = -EINVAL;
> > + if (!nested_vmcb_check_controls(ctl))
> > + goto out_free;
> >
> > /*
> > * Processor state contains L2 state. Check that it is
> > @@ -1109,15 +1122,15 @@ static int svm_set_nested_state(struct kvm_vcpu *vcpu,
> > */
> > cr0 = kvm_read_cr0(vcpu);
> > if (((cr0 & X86_CR0_CD) == 0) && (cr0 & X86_CR0_NW))
> > - return -EINVAL;
> > + goto out_free;
> >
> > /*
> > * Validate host state saved from before VMRUN (see
> > * nested_svm_check_permissions).
> > * TODO: validate reserved bits for all saved state.
> > */
> > - if (!(save.cr0 & X86_CR0_PG))
> > - return -EINVAL;
> > + if (!(save->cr0 & X86_CR0_PG))
> > + goto out_free;
> >
> > /*
> > * All checks done, we can enter guest mode. L1 control fields
> > @@ -1126,15 +1139,21 @@ static int svm_set_nested_state(struct kvm_vcpu *vcpu,
> > * contains saved L1 state.
> > */
> > copy_vmcb_control_area(&hsave->control, &svm->vmcb->control);
> > - hsave->save = save;
> > + hsave->save = *save;
> >
> > svm->nested.vmcb = kvm_state->hdr.svm.vmcb_pa;
> > - load_nested_vmcb_control(svm, &ctl);
> > + load_nested_vmcb_control(svm, ctl);
> > nested_prepare_vmcb_control(svm);
> >
> > out_set_gif:
> > svm_set_gif(svm, !!(kvm_state->flags & KVM_STATE_NESTED_GIF_SET));
> > - return 0;
> > +
> > + ret = 0;
> > +out_free:
> > + kfree(save);
> > + kfree(ctl);
>
> This change seems to trigger a crash via smm-test.c (and state-test.c) KVM
> selftest when we call vcpu_load_state->KVM_SET_NESTED_STATE. I think what's
> happening is we are hitting the 'goto out_set_gif;' and then attempting to
> free save and ctl, which are still uninitialized at that point:
Sorry, looks like this one was already fixed upstream by
d5cd6f34014592a232ce79dc25e295778bd43c22
Please ignore.
>
> [ 1999.801176] APIC base relocation is unsupported by KVM
> [ 1999.828562] BUG: unable to handle page fault for address:
> fffff12379020288
> [ 1999.841968] #PF: supervisor read access in kernel mode
> [ 1999.847693] #PF: error_code(0x0000) - not-present page
> [ 1999.853426] PGD 0 P4D 0
> [ 1999.856252] Oops: 0000 [#1] SMP NOPTI
> [ 1999.861366] CPU: 112 PID: 10162 Comm: smm_test Tainted: G
> E 5.9.1-amdsos-build31t0+ #1
> [ 1999.871655] Hardware name: AMD Corporation ETHANOL_X/ETHANOL_X, BIOS
> RXM0092B 10/27/2020
> [ 1999.880694] RIP: 0010:kfree+0x5b/0x3c0
> [ 1999.884876] Code: 80 49 01 dc 0f 82 70 03 00 00 48 c7 c0 00 00 00 80
> 48 2b 05 97 4a 1c 01 49 01 c4 49 c1 ec 0c 49 c1 e4 06 4c 03 25 75 4a 1c
> 01 <49> 8b 44 24 08 48 8d 50 ff a8 01 4c 0f 45 e2 49 8b 54 24 08 48 8d
> [ 1999.906674] RSP: 0018:ffffb7004d9d3cf8 EFLAGS: 00010282
> [ 1999.912502] RAX: 0000723d80000000 RBX: 000000004080aebf RCX:
> 0000000002000000
> [ 1999.920464] RDX: 0000000000000000 RSI: 0000000000000000 RDI:
> 000000004080aebf
> [ 1999.929335] RBP: ffffb7004d9d3d28 R08: ffff8de1c2a43628 R09:
> 0000000000000000
> [ 1999.937298] R10: 0000000000000000 R11: 0000000000000000 R12:
> fffff12379020280
> [ 1999.945258] R13: ffffffffc0f6626a R14: 0000000000000000 R15:
> ffffb7004d9d3db0
> [ 1999.953221] FS: 00007f231b4c1740(0000) GS:ffff8de20e800000(0000)
> knlGS:0000000000000000
> [ 1999.963255] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> [ 1999.969667] CR2: fffff12379020288 CR3: 0000001faf5aa006 CR4:
> 0000000000770ee0
> [ 1999.977627] PKRU: 55555554
> [ 1999.980644] Call Trace:
> [ 1999.983384] svm_leave_nested+0xea/0x2f0 [kvm_amd]
> [ 1999.988743] kvm_arch_vcpu_ioctl+0x6fd/0x1260 [kvm]
> [ 1999.995026] ? avic_vcpu_load+0x20/0x130 [kvm_amd]
> [ 2000.000372] kvm_vcpu_kick+0x705/0xae0 [kvm]
> [ 2000.005216] __x64_sys_ioctl+0x91/0xc0
> [ 2000.009470] do_syscall_64+0x38/0x90
> [ 2000.013461] entry_SYSCALL_64_after_hwframe+0x44/0xa9
> [ 2000.019090] RIP: 0033:0x7f231b5db50b
> [ 2000.024052] Code: 0f 1e fa 48 8b 05 85 39 0d 00 64 c7 00 26 00 00 00
> 48 c7 c0 ff ff ff ff c3 66 0f 1f 44 00 00 f3 0f 1e fa b8 10 00 00 00 0f
> 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 55 39 0d 00 f7 d8 64 89 01 48
> [ 2000.045005] RSP: 002b:00007ffc01de6918 EFLAGS: 00000246 ORIG_RAX:
> 0000000000000010
> [ 2000.053453] RAX: ffffffffffffffda RBX: 0000000002147880 RCX:
> 00007f231b5db50b
> [ 2000.062375] RDX: 0000000002148c98 RSI: 000000004080aebf RDI:
> 0000000000000009
> [ 2000.070335] RBP: 000000000214d0c0 R08: 00000000004103f8 R09:
> 0000000000000000
> [ 2000.078296] R10: 0000000000000000 R11: 0000000000000246 R12:
> 00007ffc01de6960
> [ 2000.087278] R13: 0000000000000002 R14: 00007f231b711000 R15:
> 0000000002147880
> [ 2000.095244] Modules linked in: nls_iso8859_1(E) dm_multipath(E)
> scsi_dh_rdac(E) scsi_dh_emc(E) scsi_dh_alua(E) intel_rapl_msr(E)
> intel_rapl_common(E) amd64_edac_mod(E) kvm_amd(E) kvm(E) joydev(E)
> input_leds(E) crct10dif_pclmul(E) crc32_pclmul(E) ghash_clmulni_intel(E)
> aesni_intel(E) crypto_simd(E) cryptd(E) glue_helper(E) hid_generic(E)
> efi_pstore(E) rapl(E) ipmi_si(E) ipmi_devintf(E) ipmi_msghandler(E)
> usbhid(E) hid(E) ast(E) drm_vram_helper(E) drm_ttm_helper(E) ttm(E)
> drm_kms_helper(E) cec(E) i2c_algo_bit(E) fb_sys_fops(E) syscopyarea(E)
> sysfillrect(E) sysimgblt(E) wmi_bmof(E) e1000e(E) i2c_piix4(E) ccp(E)
> wmi(E) mac_hid(E) sch_fq_codel(E) drm(E) ip_tables(E) x_tables(E)
> autofs4(E)
> [ 2000.164824] CR2: fffff12379020288
> [ 2000.168522] ---[ end trace c5975ced3c660340 ]---
>
> > +
> > + return ret;
> > }
> >
> > struct kvm_x86_nested_ops svm_nested_ops = {
> > --
> > 2.28.0
> >
> > _______________________________________________
> > Virtualization mailing list
> > Virtualization@lists.linux-foundation.org
> > https://lists.linuxfoundation.org/mailman/listinfo/virtualization
^ permalink raw reply [flat|nested] 264+ messages in thread
* Re: [PATCH v7 01/72] KVM: SVM: nested: Don't allocate VMCB structures on stack
2020-11-05 16:38 ` Borislav Petkov
(?)
@ 2020-11-06 0:31 ` Michael Roth
2020-11-06 0:39 ` Borislav Petkov
-1 siblings, 1 reply; 264+ messages in thread
From: Michael Roth @ 2020-11-06 0:31 UTC (permalink / raw)
To: Borislav Petkov
Cc: Joerg Roedel, x86, Juergen Gross, Tom Lendacky, Joerg Roedel,
Mike Stunes, Kees Cook, kvm, Peter Zijlstra, Cfir Cohen,
Dave Hansen, linux-kernel, Sean Christopherson, virtualization,
Martin Radev, Masami Hiramatsu, Andy Lutomirski, hpa,
Erdem Aktas, David Rientjes, Dan Williams, Jiri Slaby
On Thu, Nov 05, 2020 at 05:38:12PM +0100, Borislav Petkov wrote:
> On Thu, Nov 05, 2020 at 10:24:37AM -0600, Michael Roth wrote:
> > > out_set_gif:
> > > svm_set_gif(svm, !!(kvm_state->flags & KVM_STATE_NESTED_GIF_SET));
> > > - return 0;
> > > +
> > > + ret = 0;
> > > +out_free:
> > > + kfree(save);
> > > + kfree(ctl);
> >
> > This change seems to trigger a crash via smm-test.c (and state-test.c) KVM
> > selftest when we call vcpu_load_state->KVM_SET_NESTED_STATE. I think what's
> > happening is we are hitting the 'goto out_set_gif;'
>
> No out_set_gif upstream anymore after
>
> d5cd6f340145 ("KVM: nSVM: Avoid freeing uninitialized pointers in svm_set_nested_state()")
>
> and it looks like you hit the issue this patch is fixing.
>
> Can you test with the above commit cherrypicked ontop of your what looks
> like 5.9.1-ish tree?
>
> If that fixes it, we should route this patch to stable.
Hi Boris,
I can confirm that patch fixes the issue. It is indeed a 5.9.1 tree, but
looks like the SEV-ES patches didn't go in until v5.10-rc1 (this tree
had a backport of them), so stable trees shouldn't be affected.
Thanks!
-Mike
>
> Thx.
>
> --
> Regards/Gruss,
> Boris.
>
> https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fpeople.kernel.org%2Ftglx%2Fnotes-about-netiquette&data=04%7C01%7Cmichael.roth%40amd.com%7Cfdf6f0dd23ed48449e1e08d881a93909%7C3dd8961fe4884e608e11a82d994e183d%7C0%7C0%7C637401911116171669%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=yvG6Gtb%2FVJnjMTXsOBApU83DCuKx3%2FRAID6f3TpEy7w%3D&reserved=0
^ permalink raw reply [flat|nested] 264+ messages in thread
* Re: [PATCH v7 01/72] KVM: SVM: nested: Don't allocate VMCB structures on stack
2020-11-06 0:31 ` Michael Roth
@ 2020-11-06 0:39 ` Borislav Petkov
0 siblings, 0 replies; 264+ messages in thread
From: Borislav Petkov @ 2020-11-06 0:39 UTC (permalink / raw)
To: Michael Roth
Cc: Joerg Roedel, x86, Juergen Gross, Tom Lendacky, Joerg Roedel,
Mike Stunes, Kees Cook, kvm, Peter Zijlstra, Cfir Cohen,
Dave Hansen, linux-kernel, Sean Christopherson, virtualization,
Martin Radev, Masami Hiramatsu, Andy Lutomirski, hpa,
Erdem Aktas, David Rientjes, Dan Williams, Jiri Slaby
On Thu, Nov 05, 2020 at 06:31:53PM -0600, Michael Roth wrote:
> I can confirm that patch fixes the issue. It is indeed a 5.9.1 tree, but
> looks like the SEV-ES patches didn't go in until v5.10-rc1
Yes, they went into 5.10-rc1 during the merge window.
> (this tree had a backport of them), so stable trees shouldn't be
> affected.
Ah, ok, that makes sense.
Thanks for checking!
--
Regards/Gruss,
Boris.
https://people.kernel.org/tglx/notes-about-netiquette
^ permalink raw reply [flat|nested] 264+ messages in thread
* Re: [PATCH v7 01/72] KVM: SVM: nested: Don't allocate VMCB structures on stack
@ 2020-11-06 0:39 ` Borislav Petkov
0 siblings, 0 replies; 264+ messages in thread
From: Borislav Petkov @ 2020-11-06 0:39 UTC (permalink / raw)
To: Michael Roth
Cc: kvm, Peter Zijlstra, Dave Hansen, virtualization, hpa,
Jiri Slaby, Joerg Roedel, x86, David Rientjes, Martin Radev,
Tom Lendacky, Joerg Roedel, Kees Cook, Cfir Cohen,
Andy Lutomirski, Dan Williams, Juergen Gross, Mike Stunes,
linux-kernel, Sean Christopherson, Masami Hiramatsu, Erdem Aktas
On Thu, Nov 05, 2020 at 06:31:53PM -0600, Michael Roth wrote:
> I can confirm that patch fixes the issue. It is indeed a 5.9.1 tree, but
> looks like the SEV-ES patches didn't go in until v5.10-rc1
Yes, they went into 5.10-rc1 during the merge window.
> (this tree had a backport of them), so stable trees shouldn't be
> affected.
Ah, ok, that makes sense.
Thanks for checking!
--
Regards/Gruss,
Boris.
https://people.kernel.org/tglx/notes-about-netiquette
_______________________________________________
Virtualization mailing list
Virtualization@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/virtualization
^ permalink raw reply [flat|nested] 264+ messages in thread
* Re: [PATCH v7 45/72] x86/entry/64: Add entry code for #VC handler
2020-09-07 13:15 ` Joerg Roedel
(?)
(?)
@ 2021-01-24 14:11 ` Lai Jiangshan
2021-01-28 13:18 ` Joerg Roedel
-1 siblings, 1 reply; 264+ messages in thread
From: Lai Jiangshan @ 2021-01-24 14:11 UTC (permalink / raw)
To: Joerg Roedel
Cc: X86 ML, Joerg Roedel, H. Peter Anvin, Andy Lutomirski,
Dave Hansen, Peter Zijlstra, Jiri Slaby, Dan Williams,
Tom Lendacky, Juergen Gross, Kees Cook, David Rientjes,
Cfir Cohen, Erdem Aktas, Masami Hiramatsu, Mike Stunes,
Sean Christopherson, Martin Radev, LKML, kvm, virtualization
> +
> + /*
> + * No need to switch back to the IST stack. The current stack is either
> + * identical to the stack in the IRET frame or the VC fall-back stack,
> + * so it is definitly mapped even with PTI enabled.
> + */
> + jmp paranoid_exit
> +
>
Hello
I know we don't enable PTI on AMD, but the above comment doesn't align to the
next code.
We assume PTI is enabled as the comments said "even with PTI enabled".
When #VC happens after entry_SYSCALL_64 but before it switches to the
kernel CR3. vc_switch_off_ist() will switch the stack to the kernel stack
and paranoid_exit can't work when it switches to user CR3 on the kernel stack.
The comment above lost information that the current stack is possible to be
the kernel stack which is mapped not user CR3.
Maybe I missed something.
Thanks
Lai
> +#ifdef CONFIG_AMD_MEM_ENCRYPT
> +asmlinkage __visible noinstr struct pt_regs *vc_switch_off_ist(struct pt_regs *regs)
> +{
> + unsigned long sp, *stack;
> + struct stack_info info;
> + struct pt_regs *regs_ret;
> +
> + /*
> + * In the SYSCALL entry path the RSP value comes from user-space - don't
> + * trust it and switch to the current kernel stack
> + */
> + if (regs->ip >= (unsigned long)entry_SYSCALL_64 &&
> + regs->ip < (unsigned long)entry_SYSCALL_64_safe_stack) {
> + sp = this_cpu_read(cpu_current_top_of_stack);
> + goto sync;
> + }
^ permalink raw reply [flat|nested] 264+ messages in thread
* Re: [PATCH v7 45/72] x86/entry/64: Add entry code for #VC handler
2021-01-24 14:11 ` [PATCH v7 45/72] " Lai Jiangshan
@ 2021-01-28 13:18 ` Joerg Roedel
0 siblings, 0 replies; 264+ messages in thread
From: Joerg Roedel @ 2021-01-28 13:18 UTC (permalink / raw)
To: Lai Jiangshan
Cc: X86 ML, Joerg Roedel, H. Peter Anvin, Andy Lutomirski,
Dave Hansen, Peter Zijlstra, Jiri Slaby, Dan Williams,
Tom Lendacky, Juergen Gross, Kees Cook, David Rientjes,
Cfir Cohen, Erdem Aktas, Masami Hiramatsu, Mike Stunes,
Sean Christopherson, Martin Radev, LKML, kvm, virtualization
Hello Lai,
On Sun, Jan 24, 2021 at 10:11:14PM +0800, Lai Jiangshan wrote:
> > +
> > + /*
> > + * No need to switch back to the IST stack. The current stack is either
> > + * identical to the stack in the IRET frame or the VC fall-back stack,
> > + * so it is definitly mapped even with PTI enabled.
> > + */
> > + jmp paranoid_exit
> > +
> >
>
> Hello
>
> I know we don't enable PTI on AMD, but the above comment doesn't align to the
> next code.
>
> We assume PTI is enabled as the comments said "even with PTI enabled".
>
> When #VC happens after entry_SYSCALL_64 but before it switches to the
> kernel CR3. vc_switch_off_ist() will switch the stack to the kernel stack
> and paranoid_exit can't work when it switches to user CR3 on the kernel stack.
>
> The comment above lost information that the current stack is possible to be
> the kernel stack which is mapped not user CR3.
>
> Maybe I missed something.
You are right, the scenario above would cause problems for the current
#VC entry code. With SEV-ES an #VC exception can't happen in the early
syscall entry code, so I think its the best to update the comment
reflecting this.
In the future this might change and then the #VC entry code needs to
take care of this case too. Thanks for pointing it out.
Regards,
Joerg
^ permalink raw reply [flat|nested] 264+ messages in thread
* Re: [PATCH v7 45/72] x86/entry/64: Add entry code for #VC handler
@ 2021-01-28 13:18 ` Joerg Roedel
0 siblings, 0 replies; 264+ messages in thread
From: Joerg Roedel @ 2021-01-28 13:18 UTC (permalink / raw)
To: Lai Jiangshan
Cc: Juergen Gross, Tom Lendacky, Dave Hansen, Mike Stunes, Kees Cook,
kvm, Peter Zijlstra, Cfir Cohen, X86 ML, LKML,
Sean Christopherson, virtualization, Martin Radev, Joerg Roedel,
Masami Hiramatsu, Andy Lutomirski, H. Peter Anvin, Erdem Aktas,
David Rientjes, Dan Williams, Jiri Slaby
Hello Lai,
On Sun, Jan 24, 2021 at 10:11:14PM +0800, Lai Jiangshan wrote:
> > +
> > + /*
> > + * No need to switch back to the IST stack. The current stack is either
> > + * identical to the stack in the IRET frame or the VC fall-back stack,
> > + * so it is definitly mapped even with PTI enabled.
> > + */
> > + jmp paranoid_exit
> > +
> >
>
> Hello
>
> I know we don't enable PTI on AMD, but the above comment doesn't align to the
> next code.
>
> We assume PTI is enabled as the comments said "even with PTI enabled".
>
> When #VC happens after entry_SYSCALL_64 but before it switches to the
> kernel CR3. vc_switch_off_ist() will switch the stack to the kernel stack
> and paranoid_exit can't work when it switches to user CR3 on the kernel stack.
>
> The comment above lost information that the current stack is possible to be
> the kernel stack which is mapped not user CR3.
>
> Maybe I missed something.
You are right, the scenario above would cause problems for the current
#VC entry code. With SEV-ES an #VC exception can't happen in the early
syscall entry code, so I think its the best to update the comment
reflecting this.
In the future this might change and then the #VC entry code needs to
take care of this case too. Thanks for pointing it out.
Regards,
Joerg
_______________________________________________
Virtualization mailing list
Virtualization@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/virtualization
^ permalink raw reply [flat|nested] 264+ messages in thread
end of thread, other threads:[~2021-01-28 13:19 UTC | newest]
Thread overview: 264+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2020-09-07 13:15 [PATCH v7 00/72] x86: SEV-ES Guest Support Joerg Roedel
2020-09-07 13:15 ` Joerg Roedel
2020-09-07 13:15 ` [PATCH v7 01/72] KVM: SVM: nested: Don't allocate VMCB structures on stack Joerg Roedel
2020-09-07 13:15 ` Joerg Roedel
2020-09-10 9:22 ` [tip: x86/seves] " tip-bot2 for Joerg Roedel
2020-11-05 16:24 ` [PATCH v7 01/72] " Michael Roth
2020-11-05 16:38 ` Borislav Petkov
2020-11-05 16:38 ` Borislav Petkov
2020-11-06 0:31 ` Michael Roth
2020-11-06 0:39 ` Borislav Petkov
2020-11-06 0:39 ` Borislav Petkov
2020-11-05 17:46 ` Michael Roth
2020-09-07 13:15 ` [PATCH v7 02/72] KVM: SVM: Add GHCB definitions Joerg Roedel
2020-09-07 13:15 ` Joerg Roedel
2020-09-10 9:22 ` [tip: x86/seves] " tip-bot2 for Tom Lendacky
2020-09-07 13:15 ` [PATCH v7 03/72] KVM: SVM: Add GHCB Accessor functions Joerg Roedel
2020-09-07 13:15 ` Joerg Roedel
2020-09-10 9:22 ` [tip: x86/seves] " tip-bot2 for Joerg Roedel
2020-09-07 13:15 ` [PATCH v7 04/72] KVM: SVM: Use __packed shorthand Joerg Roedel
2020-09-07 13:15 ` Joerg Roedel
2020-09-10 9:22 ` [tip: x86/seves] " tip-bot2 for Borislav Petkov
2020-09-07 13:15 ` [PATCH v7 05/72] x86/cpufeatures: Add SEV-ES CPU feature Joerg Roedel
2020-09-07 13:15 ` Joerg Roedel
2020-09-10 9:22 ` [tip: x86/seves] " tip-bot2 for Tom Lendacky
2020-09-07 13:15 ` [PATCH v7 06/72] x86/traps: Move pf error codes to <asm/trap_pf.h> Joerg Roedel
2020-09-07 13:15 ` Joerg Roedel
2020-09-10 9:22 ` [tip: x86/seves] " tip-bot2 for Joerg Roedel
2020-09-07 13:15 ` [PATCH v7 07/72] x86/insn: Make inat-tables.c suitable for pre-decompression code Joerg Roedel
2020-09-07 13:15 ` Joerg Roedel
2020-09-10 9:22 ` [tip: x86/seves] " tip-bot2 for Joerg Roedel
2020-09-07 13:15 ` [PATCH v7 08/72] x86/umip: Factor out instruction fetch Joerg Roedel
2020-09-07 13:15 ` Joerg Roedel
2020-09-10 9:22 ` [tip: x86/seves] " tip-bot2 for Joerg Roedel
2020-09-07 13:15 ` [PATCH v7 09/72] x86/umip: Factor out instruction decoding Joerg Roedel
2020-09-07 13:15 ` Joerg Roedel
2020-09-10 9:22 ` [tip: x86/seves] " tip-bot2 for Joerg Roedel
2020-09-07 13:15 ` [PATCH v7 10/72] x86/insn: Add insn_get_modrm_reg_off() Joerg Roedel
2020-09-07 13:15 ` Joerg Roedel
2020-09-10 9:22 ` [tip: x86/seves] " tip-bot2 for Joerg Roedel
2020-09-07 13:15 ` [PATCH v7 11/72] x86/insn: Add insn_has_rep_prefix() helper Joerg Roedel
2020-09-07 13:15 ` Joerg Roedel
2020-09-10 9:22 ` [tip: x86/seves] " tip-bot2 for Joerg Roedel
2020-09-07 13:15 ` [PATCH v7 12/72] x86/boot/compressed/64: Disable red-zone usage Joerg Roedel
2020-09-07 13:15 ` Joerg Roedel
2020-09-10 9:22 ` [tip: x86/seves] " tip-bot2 for Joerg Roedel
2020-09-07 13:15 ` [PATCH v7 13/72] x86/boot/compressed/64: Add IDT Infrastructure Joerg Roedel
2020-09-07 13:15 ` Joerg Roedel
2020-09-10 9:22 ` [tip: x86/seves] " tip-bot2 for Joerg Roedel
2020-09-07 13:15 ` [PATCH v7 14/72] x86/boot/compressed/64: Rename kaslr_64.c to ident_map_64.c Joerg Roedel
2020-09-07 13:15 ` Joerg Roedel
2020-09-10 9:22 ` [tip: x86/seves] " tip-bot2 for Joerg Roedel
2020-09-07 13:15 ` [PATCH v7 15/72] x86/boot/compressed/64: Add page-fault handler Joerg Roedel
2020-09-07 13:15 ` Joerg Roedel
2020-09-10 9:22 ` [tip: x86/seves] " tip-bot2 for Joerg Roedel
2020-09-07 13:15 ` [PATCH v7 16/72] x86/boot/compressed/64: Always switch to own page-table Joerg Roedel
2020-09-07 13:15 ` Joerg Roedel
2020-09-10 9:22 ` [tip: x86/seves] x86/boot/compressed/64: Always switch to own page table tip-bot2 for Joerg Roedel
2020-09-07 13:15 ` [PATCH v7 17/72] x86/boot/compressed/64: Don't pre-map memory in KASLR code Joerg Roedel
2020-09-07 13:15 ` Joerg Roedel
2020-09-10 9:22 ` [tip: x86/seves] " tip-bot2 for Joerg Roedel
2020-09-07 13:15 ` [PATCH v7 18/72] x86/boot/compressed/64: Change add_identity_map() to take start and end Joerg Roedel
2020-09-07 13:15 ` Joerg Roedel
2020-09-10 9:22 ` [tip: x86/seves] " tip-bot2 for Joerg Roedel
2020-09-07 13:15 ` [PATCH v7 19/72] x86/boot/compressed/64: Add stage1 #VC handler Joerg Roedel
2020-09-07 13:15 ` Joerg Roedel
2020-09-07 16:58 ` Borislav Petkov
2020-09-07 16:58 ` Borislav Petkov
2020-09-10 9:22 ` [tip: x86/seves] " tip-bot2 for Joerg Roedel
2020-09-07 13:15 ` [PATCH v7 20/72] x86/boot/compressed/64: Call set_sev_encryption_mask() earlier Joerg Roedel
2020-09-07 13:15 ` Joerg Roedel
2020-09-10 9:22 ` [tip: x86/seves] " tip-bot2 for Joerg Roedel
2020-09-07 13:15 ` [PATCH v7 21/72] x86/boot/compressed/64: Check return value of kernel_ident_mapping_init() Joerg Roedel
2020-09-07 13:15 ` Joerg Roedel
2020-09-10 9:22 ` [tip: x86/seves] " tip-bot2 for Joerg Roedel
2020-09-07 13:15 ` [PATCH v7 22/72] x86/boot/compressed/64: Add set_page_en/decrypted() helpers Joerg Roedel
2020-09-07 13:15 ` Joerg Roedel
2020-09-10 9:22 ` [tip: x86/seves] " tip-bot2 for Joerg Roedel
2020-09-07 13:15 ` [PATCH v7 23/72] x86/boot/compressed/64: Setup GHCB Based VC Exception handler Joerg Roedel
2020-09-07 13:15 ` Joerg Roedel
2020-09-10 9:22 ` [tip: x86/seves] x86/boot/compressed/64: Setup a GHCB-based " tip-bot2 for Joerg Roedel
2020-09-07 13:15 ` [PATCH v7 24/72] x86/boot/compressed/64: Unmap GHCB page before booting the kernel Joerg Roedel
2020-09-07 13:15 ` Joerg Roedel
2020-09-10 9:22 ` [tip: x86/seves] " tip-bot2 for Joerg Roedel
2020-09-07 13:15 ` [PATCH v7 25/72] x86/sev-es: Add support for handling IOIO exceptions Joerg Roedel
2020-09-07 13:15 ` Joerg Roedel
2020-09-10 9:22 ` [tip: x86/seves] " tip-bot2 for Tom Lendacky
2020-09-07 13:15 ` [PATCH v7 26/72] x86/fpu: Move xgetbv()/xsetbv() into separate header Joerg Roedel
2020-09-07 13:15 ` Joerg Roedel
2020-09-10 9:22 ` [tip: x86/seves] x86/fpu: Move xgetbv()/xsetbv() into a " tip-bot2 for Joerg Roedel
2020-09-07 13:15 ` [PATCH v7 27/72] x86/sev-es: Add CPUID handling to #VC handler Joerg Roedel
2020-09-07 13:15 ` Joerg Roedel
2020-09-10 9:22 ` [tip: x86/seves] " tip-bot2 for Tom Lendacky
2020-09-07 13:15 ` [PATCH v7 28/72] x86/idt: Split idt_data setup out of set_intr_gate() Joerg Roedel
2020-09-07 13:15 ` Joerg Roedel
2020-09-10 9:22 ` [tip: x86/seves] " tip-bot2 for Joerg Roedel
2020-09-07 13:15 ` [PATCH v7 29/72] x86/head/64: Install startup GDT Joerg Roedel
2020-09-07 13:15 ` Joerg Roedel
2020-09-10 9:22 ` [tip: x86/seves] " tip-bot2 for Joerg Roedel
2020-09-07 13:15 ` [PATCH v7 30/72] x86/head/64: Load GDT after switch to virtual addresses Joerg Roedel
2020-09-07 13:15 ` Joerg Roedel
2020-09-10 9:22 ` [tip: x86/seves] " tip-bot2 for Joerg Roedel
2020-09-07 13:15 ` [PATCH v7 31/72] x86/head/64: Load segment registers earlier Joerg Roedel
2020-09-07 13:15 ` Joerg Roedel
2020-09-10 9:22 ` [tip: x86/seves] " tip-bot2 for Joerg Roedel
2020-09-07 13:15 ` [PATCH v7 32/72] x86/head/64: Switch to initial stack earlier Joerg Roedel
2020-09-07 13:15 ` Joerg Roedel
2020-09-10 9:22 ` [tip: x86/seves] " tip-bot2 for Joerg Roedel
2020-09-07 13:15 ` [PATCH v7 33/72] x86/head/64: Install a CPU bringup IDT Joerg Roedel
2020-09-07 13:15 ` Joerg Roedel
2020-09-10 9:22 ` [tip: x86/seves] " tip-bot2 for Joerg Roedel
2020-09-07 13:15 ` [PATCH v7 34/72] x86/idt: Move two function from k/idt.c to i/a/desc.h Joerg Roedel
2020-09-07 13:15 ` Joerg Roedel
2020-09-10 9:22 ` [tip: x86/seves] x86/idt: Make IDT init functions static inlines tip-bot2 for Joerg Roedel
2020-09-07 13:15 ` [PATCH v7 35/72] x86/head/64: Move early exception dispatch to C code Joerg Roedel
2020-09-07 13:15 ` Joerg Roedel
2020-09-10 9:22 ` [tip: x86/seves] " tip-bot2 for Joerg Roedel
2020-09-07 13:15 ` [PATCH v7 36/72] x86/sev-es: Add SEV-ES Feature Detection Joerg Roedel
2020-09-07 13:15 ` Joerg Roedel
2020-09-07 21:02 ` Borislav Petkov
2020-09-07 21:02 ` Borislav Petkov
2020-09-10 9:22 ` [tip: x86/seves] " tip-bot2 for Joerg Roedel
2020-09-07 13:15 ` [PATCH v7 37/72] x86/sev-es: Print SEV-ES info into kernel log Joerg Roedel
2020-09-07 13:15 ` Joerg Roedel
2020-09-10 9:22 ` [tip: x86/seves] x86/sev-es: Print SEV-ES info into the " tip-bot2 for Joerg Roedel
2020-09-07 13:15 ` [PATCH v7 38/72] x86/sev-es: Compile early handler code into kernel image Joerg Roedel
2020-09-07 13:15 ` Joerg Roedel
2020-09-10 9:22 ` [tip: x86/seves] " tip-bot2 for Joerg Roedel
2020-09-07 13:15 ` [PATCH v7 39/72] x86/sev-es: Setup early #VC handler Joerg Roedel
2020-09-07 13:15 ` Joerg Roedel
2020-09-08 10:22 ` [PATCH v7.1 " Joerg Roedel
2020-09-08 10:22 ` Joerg Roedel
2020-09-08 12:35 ` [PATCH v7.2 39/74] " Joerg Roedel
2020-09-08 12:35 ` Joerg Roedel
2020-09-10 9:22 ` [tip: x86/seves] x86/sev-es: Setup an " tip-bot2 for Joerg Roedel
2020-09-07 13:15 ` [PATCH v7 40/72] x86/sev-es: Setup GHCB based boot " Joerg Roedel
2020-09-07 13:15 ` Joerg Roedel
2020-09-08 10:24 ` [PATCH v7.1 " Joerg Roedel
2020-09-08 10:24 ` Joerg Roedel
2020-09-08 12:38 ` [PATCH v7.2 40/74] " Joerg Roedel
2020-09-08 12:38 ` Joerg Roedel
2020-09-10 9:22 ` [tip: x86/seves] x86/sev-es: Setup GHCB-based " tip-bot2 for Joerg Roedel
2020-09-07 13:15 ` [PATCH v7 41/72] x86/sev-es: Setup per-cpu GHCBs for the runtime handler Joerg Roedel
2020-09-07 13:15 ` Joerg Roedel
2020-09-08 10:51 ` Borislav Petkov
2020-09-08 10:51 ` Borislav Petkov
2020-09-10 9:22 ` [tip: x86/seves] x86/sev-es: Setup per-CPU " tip-bot2 for Tom Lendacky
2020-09-07 13:15 ` [PATCH v7 42/72] x86/sev-es: Allocate and Map IST stack for #VC handler Joerg Roedel
2020-09-07 13:15 ` Joerg Roedel
2020-09-10 9:22 ` [tip: x86/seves] x86/sev-es: Allocate and map an " tip-bot2 for Joerg Roedel
2020-09-07 13:15 ` [PATCH v7 43/72] x86/sev-es: Adjust #VC IST Stack on entering NMI handler Joerg Roedel
2020-09-07 13:15 ` Joerg Roedel
2020-09-10 9:22 ` [tip: x86/seves] " tip-bot2 for Joerg Roedel
2020-09-07 13:15 ` [PATCH v7 44/72] x86/dumpstack/64: Add noinstr version of get_stack_info() Joerg Roedel
2020-09-07 13:15 ` Joerg Roedel
2020-09-10 9:22 ` [tip: x86/seves] " tip-bot2 for Joerg Roedel
2020-09-07 13:15 ` [PATCH v7 45/72] x86/entry/64: Add entry code for #VC handler Joerg Roedel
2020-09-07 13:15 ` Joerg Roedel
2020-09-10 9:22 ` [tip: x86/seves] " tip-bot2 for Joerg Roedel
2021-01-24 14:11 ` [PATCH v7 45/72] " Lai Jiangshan
2021-01-28 13:18 ` Joerg Roedel
2021-01-28 13:18 ` Joerg Roedel
2020-09-07 13:15 ` [PATCH v7 46/72] x86/sev-es: Add Runtime #VC Exception Handler Joerg Roedel
2020-09-07 13:15 ` Joerg Roedel
2020-09-10 9:22 ` [tip: x86/seves] x86/sev-es: Add a " tip-bot2 for Tom Lendacky
2020-09-07 13:15 ` [PATCH v7 47/72] x86/sev-es: Wire up existing #VC exit-code handlers Joerg Roedel
2020-09-07 13:15 ` Joerg Roedel
2020-09-10 9:22 ` [tip: x86/seves] " tip-bot2 for Joerg Roedel
2020-09-07 13:15 ` [PATCH v7 48/72] x86/sev-es: Handle instruction fetches from user-space Joerg Roedel
2020-09-07 13:15 ` Joerg Roedel
2020-09-10 9:22 ` [tip: x86/seves] " tip-bot2 for Joerg Roedel
2020-09-07 13:15 ` [PATCH v7 49/72] x86/sev-es: Handle MMIO events Joerg Roedel
2020-09-07 13:15 ` Joerg Roedel
2020-09-10 9:22 ` [tip: x86/seves] " tip-bot2 for Tom Lendacky
2020-09-07 13:15 ` [PATCH v7 50/72] x86/sev-es: Handle MMIO String Instructions Joerg Roedel
2020-09-07 13:15 ` Joerg Roedel
2020-09-10 9:22 ` [tip: x86/seves] " tip-bot2 for Joerg Roedel
2020-09-07 13:15 ` [PATCH v7 51/72] x86/sev-es: Handle MSR events Joerg Roedel
2020-09-07 13:15 ` Joerg Roedel
2020-09-10 9:22 ` [tip: x86/seves] " tip-bot2 for Tom Lendacky
2020-09-07 13:15 ` [PATCH v7 52/72] x86/sev-es: Handle DR7 read/write events Joerg Roedel
2020-09-07 13:15 ` Joerg Roedel
2020-09-10 9:22 ` [tip: x86/seves] " tip-bot2 for Tom Lendacky
2020-09-07 13:15 ` [PATCH v7 53/72] x86/sev-es: Handle WBINVD Events Joerg Roedel
2020-09-07 13:15 ` Joerg Roedel
2020-09-10 9:22 ` [tip: x86/seves] " tip-bot2 for Tom Lendacky
2020-09-07 13:15 ` [PATCH v7 54/72] x86/sev-es: Handle RDTSC(P) Events Joerg Roedel
2020-09-07 13:15 ` Joerg Roedel
2020-09-10 9:22 ` [tip: x86/seves] " tip-bot2 for Tom Lendacky
2020-09-07 13:15 ` [PATCH v7 55/72] x86/sev-es: Handle RDPMC Events Joerg Roedel
2020-09-07 13:15 ` Joerg Roedel
2020-09-10 9:22 ` [tip: x86/seves] " tip-bot2 for Tom Lendacky
2020-09-07 13:15 ` [PATCH v7 56/72] x86/sev-es: Handle INVD Events Joerg Roedel
2020-09-07 13:15 ` Joerg Roedel
2020-09-10 9:22 ` [tip: x86/seves] " tip-bot2 for Tom Lendacky
2020-09-07 13:15 ` [PATCH v7 57/72] x86/sev-es: Handle MONITOR/MONITORX Events Joerg Roedel
2020-09-07 13:15 ` Joerg Roedel
2020-09-10 9:22 ` [tip: x86/seves] " tip-bot2 for Tom Lendacky
2020-09-07 13:15 ` [PATCH v7 58/72] x86/sev-es: Handle MWAIT/MWAITX Events Joerg Roedel
2020-09-07 13:15 ` Joerg Roedel
2020-09-10 9:22 ` [tip: x86/seves] " tip-bot2 for Tom Lendacky
2020-09-07 13:16 ` [PATCH v7 59/72] x86/sev-es: Handle VMMCALL Events Joerg Roedel
2020-09-07 13:16 ` Joerg Roedel
2020-09-10 9:22 ` [tip: x86/seves] " tip-bot2 for Tom Lendacky
2020-09-07 13:16 ` [PATCH v7 60/72] x86/sev-es: Handle #AC Events Joerg Roedel
2020-09-07 13:16 ` Joerg Roedel
2020-09-10 9:22 ` [tip: x86/seves] " tip-bot2 for Joerg Roedel
2020-09-07 13:16 ` [PATCH v7 61/72] x86/sev-es: Handle #DB Events Joerg Roedel
2020-09-07 13:16 ` Joerg Roedel
2020-09-10 9:22 ` [tip: x86/seves] " tip-bot2 for Joerg Roedel
2020-09-07 13:16 ` [PATCH v7 62/72] x86/paravirt: Allow hypervisor specific VMMCALL handling under SEV-ES Joerg Roedel
2020-09-07 13:16 ` Joerg Roedel
2020-09-10 9:22 ` [tip: x86/seves] x86/paravirt: Allow hypervisor-specific " tip-bot2 for Joerg Roedel
2020-09-07 13:16 ` [PATCH v7 63/72] x86/kvm: Add KVM specific " Joerg Roedel
2020-09-07 13:16 ` Joerg Roedel
2020-09-10 9:22 ` [tip: x86/seves] x86/kvm: Add KVM-specific " tip-bot2 for Tom Lendacky
[not found] ` <CAAYXXYx=Eq4gYfUqdO7u37VRD_GpPYFQgN=GZySmAMcDc2AM=g@mail.gmail.com>
2020-10-27 23:14 ` Erdem Aktas
2020-10-28 9:49 ` Joerg Roedel
2020-10-28 18:03 ` Erdem Aktas
2020-10-30 10:23 ` Joerg Roedel
2020-09-07 13:16 ` [PATCH v7 64/72] x86/vmware: Add VMware specific handling for VMMCALL " Joerg Roedel
2020-09-07 13:16 ` Joerg Roedel
2020-09-10 9:22 ` [tip: x86/seves] x86/vmware: Add VMware-specific " tip-bot2 for Doug Covelli
2020-10-27 23:19 ` Erdem Aktas
2020-10-28 9:54 ` Joerg Roedel
2020-09-07 13:16 ` [PATCH v7 65/72] x86/realmode: Add SEV-ES specific trampoline entry point Joerg Roedel
2020-09-07 13:16 ` Joerg Roedel
2020-09-10 9:22 ` [tip: x86/seves] " tip-bot2 for Joerg Roedel
2020-09-07 13:16 ` [PATCH v7 66/72] x86/realmode: Setup AP jump table Joerg Roedel
2020-09-07 13:16 ` Joerg Roedel
2020-09-10 9:22 ` [tip: x86/seves] " tip-bot2 for Tom Lendacky
2020-09-07 13:16 ` [PATCH v7 67/72] x86/smpboot: Load TSS and getcpu GDT entry before loading IDT Joerg Roedel
2020-09-07 13:16 ` Joerg Roedel
2020-09-08 17:20 ` Borislav Petkov
2020-09-08 17:20 ` Borislav Petkov
2020-09-08 18:42 ` Joerg Roedel
2020-09-08 18:42 ` Joerg Roedel
2020-09-10 9:22 ` [tip: x86/seves] " tip-bot2 for Joerg Roedel
2020-09-07 13:16 ` [PATCH v7 68/72] x86/head/64: Don't call verify_cpu() on starting APs Joerg Roedel
2020-09-07 13:16 ` Joerg Roedel
2020-09-10 9:22 ` [tip: x86/seves] " tip-bot2 for Joerg Roedel
2020-09-07 13:16 ` [PATCH v7 69/72] x86/sev-es: Support CPU offline/online Joerg Roedel
2020-09-07 13:16 ` Joerg Roedel
2020-09-10 9:22 ` [tip: x86/seves] " tip-bot2 for Joerg Roedel
2020-09-07 13:16 ` [PATCH v7 70/72] x86/sev-es: Handle NMI State Joerg Roedel
2020-09-07 13:16 ` Joerg Roedel
2020-09-10 9:22 ` [tip: x86/seves] " tip-bot2 for Joerg Roedel
2020-09-07 13:16 ` [PATCH v7 71/72] x86/efi: Add GHCB mappings when SEV-ES is active Joerg Roedel
2020-09-07 13:16 ` Joerg Roedel
2020-09-08 17:46 ` Borislav Petkov
2020-09-08 17:46 ` Borislav Petkov
2020-09-09 8:27 ` Ard Biesheuvel
2020-09-09 12:44 ` Laszlo Ersek
2020-09-09 12:44 ` Laszlo Ersek
2020-09-09 13:24 ` Laszlo Ersek
2020-09-09 13:24 ` Laszlo Ersek
2020-09-09 13:49 ` Tom Lendacky
2020-09-09 13:49 ` Tom Lendacky
2020-09-10 12:37 ` Ard Biesheuvel
2020-09-10 9:22 ` [tip: x86/seves] " tip-bot2 for Tom Lendacky
2020-09-10 19:52 ` tip-bot2 for Tom Lendacky
2020-09-07 13:16 ` [PATCH v7 72/72] x86/sev-es: Check required CPU features for SEV-ES Joerg Roedel
2020-09-07 13:16 ` Joerg Roedel
2020-09-10 9:22 ` [tip: x86/seves] " tip-bot2 for Martin Radev
2020-09-10 19:52 ` tip-bot2 for Martin Radev
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.