linux-sgx.vger.kernel.org archive mirror
 help / color / mirror / Atom feed
From: Kai Huang <kai.huang@intel.com>
To: linux-sgx@vger.kernel.org, kvm@vger.kernel.org, x86@kernel.org
Cc: seanjc@google.com, jarkko@kernel.org, luto@kernel.org,
	dave.hansen@intel.com, haitao.huang@intel.com,
	pbonzini@redhat.com, bp@alien8.de, tglx@linutronix.de,
	mingo@redhat.com, hpa@zytor.com, jmattson@google.com,
	joro@8bytes.org, vkuznets@redhat.com, wanpengli@tencent.com,
	Kai Huang <kai.huang@intel.com>
Subject: [RFC PATCH v2 20/26] KVM: VMX: Add basic handling of VM-Exit from SGX enclave
Date: Mon, 18 Jan 2021 16:28:31 +1300	[thread overview]
Message-ID: <a2c5414361d7929f7aa5b3acca7ca7c2d7a83491.1610935432.git.kai.huang@intel.com> (raw)
In-Reply-To: <cover.1610935432.git.kai.huang@intel.com>

From: Sean Christopherson <sean.j.christopherson@intel.com>

Add support for handling VM-Exits that originate from a guest SGX
enclave.  In SGX, an "enclave" is a new CPL3-only execution environment,
wherein the CPU and memory state is protected by hardware to make the
state inaccesible to code running outside of the enclave.  When exiting
an enclave due to an asynchronous event (from the perspective of the
enclave), e.g. exceptions, interrupts, and VM-Exits, the enclave's state
is automatically saved and scrubbed (the CPU loads synthetic state), and
then reloaded when re-entering the enclave.  E.g. after an instruction
based VM-Exit from an enclave, vmcs.GUEST_RIP will not contain the RIP
of the enclave instruction that trigered VM-Exit, but will instead point
to a RIP in the enclave's untrusted runtime (the guest userspace code
that coordinates entry/exit to/from the enclave).

To help a VMM recognize and handle exits from enclaves, SGX adds bits to
existing VMCS fields, VM_EXIT_REASON.VMX_EXIT_REASON_FROM_ENCLAVE and
GUEST_INTERRUPTIBILITY_INFO.GUEST_INTR_STATE_ENCLAVE_INTR.  Define the
new architectural bits, and add a boolean to struct vcpu_vmx to cache
VMX_EXIT_REASON_FROM_ENCLAVE.  Clear the bit in exit_reason so that
checks against exit_reason do not need to account for SGX, e.g.
"if (exit_reason == EXIT_REASON_EXCEPTION_NMI)" continues to work.

KVM is a largely a passive observer of the new bits, e.g. KVM needs to
account for the bits when propagating information to a nested VMM, but
otherwise doesn't need to act differently for the majority of VM-Exits
from enclaves.

The one scenario that is directly impacted is emulation, which is for
all intents and purposes impossible[1] since KVM does not have access to
the RIP or instruction stream that triggered the VM-Exit.  The inability
to emulate is a non-issue for KVM, as most instructions that might
trigger VM-Exit unconditionally #UD in an enclave (before the VM-Exit
check.  For the few instruction that conditionally #UD, KVM either never
sets the exiting control, e.g. PAUSE_EXITING[2], or sets it if and only
if the feature is not exposed to the guest in order to inject a #UD,
e.g. RDRAND_EXITING.

But, because it is still possible for a guest to trigger emulation,
e.g. MMIO, inject a #UD if KVM ever attempts emulation after a VM-Exit
from an enclave.  This is architecturally accurate for instruction
VM-Exits, and for MMIO it's the least bad choice, e.g. it's preferable
to killing the VM.  In practice, only broken or particularly stupid
guests should ever encounter this behavior.

Add a WARN in skip_emulated_instruction to detect any attempt to
modify the guest's RIP during an SGX enclave VM-Exit as all such flows
should either be unreachable or must handle exits from enclaves before
getting to skip_emulated_instruction.

[1] Impossible for all practical purposes.  Not truly impossible
    since KVM could implement some form of para-virtualization scheme.

[2] PAUSE_LOOP_EXITING only affects CPL0 and enclaves exist only at
    CPL3, so we also don't need to worry about that interaction.

Signed-off-by: Sean Christopherson <sean.j.christopherson@intel.com>
 [Kai: Remove unlikely()s suggested by Dave Hansen.]
Signed-off-by: Kai Huang <kai.huang@intel.com>
---
 arch/x86/include/asm/vmx.h      |  1 +
 arch/x86/include/uapi/asm/vmx.h |  1 +
 arch/x86/kvm/vmx/nested.c       |  2 ++
 arch/x86/kvm/vmx/vmx.c          | 38 +++++++++++++++++++++++++++++++--
 4 files changed, 40 insertions(+), 2 deletions(-)

diff --git a/arch/x86/include/asm/vmx.h b/arch/x86/include/asm/vmx.h
index 38ca445a8429..e99021a00eb9 100644
--- a/arch/x86/include/asm/vmx.h
+++ b/arch/x86/include/asm/vmx.h
@@ -372,6 +372,7 @@ enum vmcs_field {
 #define GUEST_INTR_STATE_MOV_SS		0x00000002
 #define GUEST_INTR_STATE_SMI		0x00000004
 #define GUEST_INTR_STATE_NMI		0x00000008
+#define GUEST_INTR_STATE_ENCLAVE_INTR	0x00000010
 
 /* GUEST_ACTIVITY_STATE flags */
 #define GUEST_ACTIVITY_ACTIVE		0
diff --git a/arch/x86/include/uapi/asm/vmx.h b/arch/x86/include/uapi/asm/vmx.h
index ada955c5ebb6..c7a18eb2a074 100644
--- a/arch/x86/include/uapi/asm/vmx.h
+++ b/arch/x86/include/uapi/asm/vmx.h
@@ -27,6 +27,7 @@
 
 
 #define VMX_EXIT_REASONS_FAILED_VMENTRY         0x80000000
+#define VMX_EXIT_REASONS_SGX_ENCLAVE_MODE	0x08000000
 
 #define EXIT_REASON_EXCEPTION_NMI       0
 #define EXIT_REASON_EXTERNAL_INTERRUPT  1
diff --git a/arch/x86/kvm/vmx/nested.c b/arch/x86/kvm/vmx/nested.c
index f112c2482887..562eab7b0a51 100644
--- a/arch/x86/kvm/vmx/nested.c
+++ b/arch/x86/kvm/vmx/nested.c
@@ -4126,6 +4126,8 @@ static void prepare_vmcs12(struct kvm_vcpu *vcpu, struct vmcs12 *vmcs12,
 {
 	/* update exit information fields: */
 	vmcs12->vm_exit_reason = vm_exit_reason;
+	if (to_vmx(vcpu)->exit_reason.sgx_enclave_mode)
+		vmcs12->vm_exit_reason |= VMX_EXIT_REASONS_SGX_ENCLAVE_MODE;
 	vmcs12->exit_qualification = exit_qualification;
 	vmcs12->vm_exit_intr_info = exit_intr_info;
 
diff --git a/arch/x86/kvm/vmx/vmx.c b/arch/x86/kvm/vmx/vmx.c
index 746b87375aff..4cb8a3f1374c 100644
--- a/arch/x86/kvm/vmx/vmx.c
+++ b/arch/x86/kvm/vmx/vmx.c
@@ -1561,12 +1561,18 @@ static int vmx_rtit_ctl_check(struct kvm_vcpu *vcpu, u64 data)
 
 static bool vmx_can_emulate_instruction(struct kvm_vcpu *vcpu, void *insn, int insn_len)
 {
+	if (to_vmx(vcpu)->exit_reason.sgx_enclave_mode) {
+		kvm_queue_exception(vcpu, UD_VECTOR);
+		return false;
+	}
 	return true;
 }
 
 static int skip_emulated_instruction(struct kvm_vcpu *vcpu)
 {
+	union vmx_exit_reason exit_reason = to_vmx(vcpu)->exit_reason;
 	unsigned long rip, orig_rip;
+	u32 instr_len;
 
 	/*
 	 * Using VMCS.VM_EXIT_INSTRUCTION_LEN on EPT misconfig depends on
@@ -1577,9 +1583,33 @@ static int skip_emulated_instruction(struct kvm_vcpu *vcpu)
 	 * i.e. we end up advancing IP with some random value.
 	 */
 	if (!static_cpu_has(X86_FEATURE_HYPERVISOR) ||
-	    to_vmx(vcpu)->exit_reason.basic != EXIT_REASON_EPT_MISCONFIG) {
+	    exit_reason.basic != EXIT_REASON_EPT_MISCONFIG) {
+		instr_len = vmcs_read32(VM_EXIT_INSTRUCTION_LEN);
+
+		/*
+		 * Emulating an enclave's instructions isn't supported as KVM
+		 * cannot access the enclave's memory or its true RIP, e.g. the
+		 * vmcs.GUEST_RIP points at the exit point of the enclave, not
+		 * the RIP that actually triggered the VM-Exit.  But, because
+		 * most instructions that cause VM-Exit will #UD in an enclave,
+		 * most instruction-based VM-Exits simply do not occur.
+		 *
+		 * There are a few exceptions, notably the debug instructions
+		 * INT1ICEBRK and INT3, as they are allowed in debug enclaves
+		 * and generate #DB/#BP as expected, which KVM might intercept.
+		 * But again, the CPU does the dirty work and saves an instr
+		 * length of zero so VMMs don't shoot themselves in the foot.
+		 * WARN if KVM tries to skip a non-zero length instruction on
+		 * a VM-Exit from an enclave.
+		 */
+		if (!instr_len)
+			goto rip_updated;
+
+		WARN(exit_reason.sgx_enclave_mode,
+		     "KVM: skipping instruction after SGX enclave VM-Exit");
+
 		orig_rip = kvm_rip_read(vcpu);
-		rip = orig_rip + vmcs_read32(VM_EXIT_INSTRUCTION_LEN);
+		rip = orig_rip + instr_len;
 #ifdef CONFIG_X86_64
 		/*
 		 * We need to mask out the high 32 bits of RIP if not in 64-bit
@@ -1595,6 +1625,7 @@ static int skip_emulated_instruction(struct kvm_vcpu *vcpu)
 			return 0;
 	}
 
+rip_updated:
 	/* skipping an emulated instruction also counts */
 	vmx_set_interrupt_shadow(vcpu, 0);
 
@@ -5341,6 +5372,9 @@ static int handle_ept_misconfig(struct kvm_vcpu *vcpu)
 {
 	gpa_t gpa;
 
+	if (!vmx_can_emulate_instruction(vcpu, NULL, 0))
+		return 1;
+
 	/*
 	 * A nested guest cannot optimize MMIO vmexits, because we have an
 	 * nGPA here instead of the required GPA.
-- 
2.29.2


  parent reply	other threads:[~2021-01-18  3:30 UTC|newest]

Thread overview: 91+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2021-01-18  3:26 [RFC PATCH v2 00/26] KVM SGX virtualization support Kai Huang
2021-01-18  3:26 ` [RFC PATCH v2 01/26] x86/cpufeatures: Add SGX1 and SGX2 sub-features Kai Huang
2021-01-19 16:19   ` Borislav Petkov
2021-01-19 18:03     ` Sean Christopherson
2021-01-19 22:54       ` Kai Huang
2021-01-20 10:28       ` Borislav Petkov
2021-01-20 11:50   ` Jarkko Sakkinen
2021-01-20 23:23     ` Kai Huang
2021-01-21  1:01       ` Jarkko Sakkinen
2021-01-18  3:26 ` [RFC PATCH v2 02/26] x86/sgx: Remove a warn from sgx_free_epc_page() Kai Huang
2021-01-19  8:32   ` Jarkko Sakkinen
2021-01-20  0:42     ` Kai Huang
2021-01-18  3:26 ` [RFC PATCH v2 03/26] x86/sgx: Wipe out EREMOVE " Kai Huang
2021-01-18  3:26 ` [RFC PATCH v2 04/26] x86/sgx: Add SGX_CHILD_PRESENT hardware error code Kai Huang
2021-01-20 11:51   ` Jarkko Sakkinen
2021-01-18  3:26 ` [RFC PATCH v2 05/26] x86/sgx: Introduce virtual EPC for use by KVM guests Kai Huang
2021-01-20 11:54   ` Jarkko Sakkinen
2021-01-20 17:40     ` Sean Christopherson
2021-01-21  0:54       ` Jarkko Sakkinen
2021-01-21  0:55         ` Jarkko Sakkinen
2021-01-18  3:27 ` [RFC PATCH v2 06/26] x86/cpu/intel: Allow SGX virtualization without Launch Control support Kai Huang
2021-01-20 21:02   ` Dave Hansen
2021-01-20 22:36     ` Sean Christopherson
2021-01-20 23:27       ` Dave Hansen
2021-01-20 23:48         ` Kai Huang
2021-01-20 23:51           ` Dave Hansen
2021-01-21  1:53             ` Kai Huang
2021-01-21 14:35               ` Jarkko Sakkinen
2021-01-21  1:12         ` Jarkko Sakkinen
2021-01-20 23:50     ` Kai Huang
2021-01-21  1:11     ` Jarkko Sakkinen
2021-01-18  3:27 ` [RFC PATCH v2 07/26] x86/sgx: Initialize virtual EPC driver even when SGX driver is disabled Kai Huang
2021-01-18  3:27 ` [RFC PATCH v2 08/26] x86/sgx: Expose SGX architectural definitions to the kernel Kai Huang
2021-01-20 11:58   ` Jarkko Sakkinen
2021-01-20 23:53     ` Kai Huang
2021-01-18  3:27 ` [RFC PATCH v2 09/26] x86/sgx: Move ENCLS leaf definitions to sgx_arch.h Kai Huang
2021-01-20 11:59   ` Jarkko Sakkinen
2021-01-18  3:28 ` [RFC PATCH v2 10/26] x86/sgx: Add SGX2 ENCLS leaf definitions (EAUG, EMODPR and EMODT) Kai Huang
2021-01-20 11:59   ` Jarkko Sakkinen
2021-01-18  3:28 ` [RFC PATCH v2 11/26] x86/sgx: Add encls_faulted() helper Kai Huang
2021-01-20 12:03   ` Jarkko Sakkinen
2021-01-20 23:43     ` Kai Huang
2021-01-21  1:08       ` Jarkko Sakkinen
2021-01-21  1:12         ` Kai Huang
2021-01-21 14:38           ` Jarkko Sakkinen
2021-01-18  3:28 ` [RFC PATCH v2 12/26] x86/sgx: Add helper to update SGX_LEPUBKEYHASHn MSRs Kai Huang
2021-01-20 12:03   ` Jarkko Sakkinen
2021-01-20 18:36     ` Dave Hansen
2021-01-20 23:36       ` Kai Huang
2021-01-20 23:50         ` Dave Hansen
2021-01-21  1:06           ` Kai Huang
2021-01-21  1:15             ` Dave Hansen
2021-01-21  1:44               ` Kai Huang
2021-01-21 14:36                 ` Jarkko Sakkinen
2021-01-21  1:18             ` Kai Huang
2021-01-21  1:09         ` Jarkko Sakkinen
2021-01-21  1:08       ` Jarkko Sakkinen
2021-01-18  3:28 ` [RFC PATCH v2 13/26] x86/sgx: Add helpers to expose ECREATE and EINIT to KVM Kai Huang
2021-01-20 12:04   ` Jarkko Sakkinen
2021-01-20 23:29     ` Kai Huang
2021-01-18  3:28 ` [RFC PATCH v2 14/26] x86/sgx: Move provisioning device creation out of SGX driver Kai Huang
2021-01-20 14:09   ` Jarkko Sakkinen
2021-01-20 23:24     ` Kai Huang
2021-01-18  3:28 ` [RFC PATCH v2 15/26] KVM: VMX: Convert vcpu_vmx.exit_reason to a union Kai Huang
2021-01-20 14:18   ` Jarkko Sakkinen
2021-01-20 16:39     ` Sean Christopherson
2021-01-21  0:47       ` Jarkko Sakkinen
2021-01-21 16:33         ` Sean Christopherson
2021-01-22 17:29           ` Jarkko Sakkinen
2021-01-18  3:28 ` [RFC PATCH v2 16/26] KVM: x86: Export kvm_mmu_gva_to_gpa_{read,write}() for SGX (VMX) Kai Huang
2021-01-20 14:19   ` Jarkko Sakkinen
2021-01-18  3:28 ` [RFC PATCH v2 17/26] KVM: x86: Define new #PF SGX error code bit Kai Huang
2021-01-18  3:28 ` [RFC PATCH v2 18/26] KVM: x86: Add support for reverse CPUID lookup of scattered features Kai Huang
2021-01-18  3:28 ` [RFC PATCH v2 19/26] KVM: x86: Add reverse-CPUID lookup support for scattered SGX features Kai Huang
2021-01-18  3:28 ` Kai Huang [this message]
2021-01-18  3:28 ` [RFC PATCH v2 21/26] KVM: VMX: Frame in ENCLS handler for SGX virtualization Kai Huang
2021-01-18  3:28 ` [RFC PATCH v2 22/26] KVM: VMX: Add SGX ENCLS[ECREATE] handler to enforce CPUID restrictions Kai Huang
2021-01-18  3:28 ` [RFC PATCH v2 23/26] KVM: VMX: Add emulation of SGX Launch Control LE hash MSRs Kai Huang
2021-01-18  3:28 ` [RFC PATCH v2 24/26] KVM: VMX: Add ENCLS[EINIT] handler to support SGX Launch Control (LC) Kai Huang
2021-01-18  3:28 ` [RFC PATCH v2 25/26] KVM: VMX: Enable SGX virtualization for SGX1, SGX2 and LC Kai Huang
2021-01-18  3:29 ` [RFC PATCH v2 26/26] KVM: x86: Add capability to grant VM access to privileged SGX attribute Kai Huang
2021-01-19  8:23 ` [RFC PATCH v2 00/26] KVM SGX virtualization support Jarkko Sakkinen
2021-01-20  0:52   ` Kai Huang
2021-01-20 16:35     ` Sean Christopherson
2021-01-20 16:39       ` Paolo Bonzini
2021-01-21  1:28         ` Kai Huang
2021-01-20 23:43     ` Jarkko Sakkinen
2021-01-20 23:52       ` Kai Huang
2021-01-21  1:16         ` Jarkko Sakkinen
2021-01-21  1:27           ` Kai Huang
2021-01-21 14:34             ` Jarkko Sakkinen

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=a2c5414361d7929f7aa5b3acca7ca7c2d7a83491.1610935432.git.kai.huang@intel.com \
    --to=kai.huang@intel.com \
    --cc=bp@alien8.de \
    --cc=dave.hansen@intel.com \
    --cc=haitao.huang@intel.com \
    --cc=hpa@zytor.com \
    --cc=jarkko@kernel.org \
    --cc=jmattson@google.com \
    --cc=joro@8bytes.org \
    --cc=kvm@vger.kernel.org \
    --cc=linux-sgx@vger.kernel.org \
    --cc=luto@kernel.org \
    --cc=mingo@redhat.com \
    --cc=pbonzini@redhat.com \
    --cc=seanjc@google.com \
    --cc=tglx@linutronix.de \
    --cc=vkuznets@redhat.com \
    --cc=wanpengli@tencent.com \
    --cc=x86@kernel.org \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).