Re: [RFC PATCH 00/23] KVM SGX virtualization support

[Kai Huang <kai.huang@intel.com>]

Sorry that I made mistake when copy and paste Jim's email address :( 
Remove the wrong email address (mattson@google.com) and add the correct one
(jmattson@gmail.com). Really apologize for the noise.

Thanks,
-Kai

On Wed, 2021-01-06 at 14:55 +1300, Kai Huang wrote:
> --- Disclaimer ---
> 
> These patches were originally written by Sean Christopherson while at Intel.
> Now that Sean has left Intel, I (Kai) have taken over getting them upstream.
> This series needs more review before it can be merged.  It is being posted
> publicly and under RFC so Sean and others can review it. Maintainers are safe
> ignoring it for now.
> 
> ------------------
> 
> Hi all,
> 
> This series adds KVM SGX virtualization support. The first 12 patches starting
> with x86/sgx or x86/cpu.. are necessary changes to x86 and SGX core/driver to
> support KVM SGX virtualization, while the rest are patches to KVM subsystem.
> 
> Please help to review this series. Also I'd like to hear what is the proper
> way to merge this series, since it contains change to both x86/SGX and KVM
> subsystem. Any feedback is highly appreciated. And please let me know if I
> forgot to CC anyone, or anyone wants to be removed from CC. Thanks in advance!
> 
> This series is based against latest tip tree's x86/sgx branch. You can also get
> the code from tip branch of kvm-sgx repo on github:
> 
>         https://github.com/intel/kvm-sgx.git tip
> 
> It also requires Qemu changes to create VM with SGX support. You can find Qemu
> repo here:
> 
> 	https://github.com/intel/qemu-sgx.git next
> 
> Please refer to README.md of above qemu-sgx repo for detail on how to create
> guest with SGX support. At meantime, for your quick reference you can use below
> command to create SGX guest:
> 
> 	#qemu-system-x86_64 -smp 4 -m 2G -drive file=<your_vm_image>,if=virtio \
> 		-cpu host,+sgx_provisionkey \
> 		-sgx-epc id=epc1,memdev=mem1 \
> 		-object memory-backend-epc,id=mem1,size=64M,prealloc
> 
> Please note that the SGX relevant part is:
> 
> 		-cpu host,+sgx_provisionkey \
> 		-sgx-epc id=epc1,memdev=mem1 \
> 		-object memory-backend-epc,id=mem1,size=64M,prealloc
> 
> And you can change other parameters of your qemu command based on your needs.
> 
> =========
> KVM SGX virtualization Overview
> 
> - Virtual EPC
> 
> "Virtual EPC" is the EPC section exposed by KVM to guest so SGX software in
> guest can discover it and use it to create SGX enclaves. KVM exposes SGX to 
> guest via CPUID, and exposes one or more "virtual EPC" sections for guest.
> The size of "virtual EPC" is passed as Qemu parameter when creating the
> guest, and the base address is calcualted internally according to guest's
> configuration.
> 
> To support virtual EPC, add a new misc device /dev/sgx_virt_epc to SGX
> core/driver to allow userspace (Qemu) to allocate "raw" EPC, and use it as
> "virtual EPC" for guest. Obviously, unlike EPC allocated for host SGX driver,
> virtual EPC allocated via /dev/sgx_virt_epc doesn't have enclave associated,
> and how virtual EPC is used by guest is compeletely controlled by guest's SGX
> software.
> 
> Implement the "raw" EPC allocation in the x86 core-SGX subsystem via
> /dev/sgx_virt_epc rather than in KVM. Doing so has two major advantages:
> 
>   - Does not require changes to KVM's uAPI, e.g. EPC gets handled as
>     just another memory backend for guests.
> 
>   - EPC management is wholly contained in the SGX subsystem, e.g. SGX
>     does not have to export any symbols, changes to reclaim flows don't
>     need to be routed through KVM, SGX's dirty laundry doesn't have to
>     get aired out for the world to see, and so on and so forth.
> 
> The virtual EPC allocated to guests is currently not reclaimable, due to
> reclaiming EPC from KVM guests is not currently supported. Due to the
> complications of handling reclaim conflicts between guest and host, KVM
> EPC oversubscription, which allows total virtual EPC size greater than
> physical EPC by being able to reclaiming guests' EPC, is significantly more
> complex than basic support for SGX virtualization.
> 
> - Support SGX virtualization without SGX Launch Control unlocked mode
> 
> Although SGX driver requires SGX Launch Control unlocked mode to work, SGX
> virtualization doesn't, since how enclave is created is completely controlled
> by guest SGX software, which is not necessarily linux. Therefore, this series
> allows KVM to expose SGX to guest even SGX Launch Control is in locked mode,
> or is not present at all. The reason is the goal of SGX virtualization, or
> virtualization in general, is to expose hardware feature to guest, but not to
> make assumption how guest will use it. Therefore, KVM should support SGX guest
> as long as hardware is able to, to have chance to support more potential use
> cases in cloud environment.
> 
> - Support exposing SGX2
> 
> Due to the same reason above, SGX2 feature detection is added to core SGX code
> to allow KVM to expose SGX2 to guest, even currently SGX driver doesn't support
> SGX2, because SGX2 can work just fine in guest w/o any interaction to host SGX
> driver.
> 
> - Restricit SGX guest access to provisioning key
> 
> To grant guest being able to fully use SGX, guest needs to be able to create
> provisioning enclave. However provisioning key is sensitive and is restricted by
> /dev/sgx_provision in host SGX driver, therefore KVM SGX virtualization follows
> the same role: a new KVM_CAP_SGX_ATTRIBUTE is added to KVM uAPI, and only file
> descriptor of /dev/sgx_provision is passed to that CAP by usersppace hypervisor
> (Qemu) when creating the guest, it can access provisioning bit. This is done by
> making KVM trape ECREATE instruction from guest, and check the provisioning bit
> in ECREATE's attribute.
> 
> 
> Kai Huang (1):
>   x86/sgx: Add helper to update SGX_LEPUBKEYHASHn MSRs
> 
> Sean Christopherson (22):
>   x86/sgx: Split out adding EPC page to free list to separate helper
>   x86/sgx: Add enum for SGX_CHILD_PRESENT error code
>   x86/sgx: Introduce virtual EPC for use by KVM guests
>   x86/cpufeatures: Add SGX1 and SGX2 sub-features
>   x86/cpu/intel: Allow SGX virtualization without Launch Control support
>   x86/sgx: Expose SGX architectural definitions to the kernel
>   x86/sgx: Move ENCLS leaf definitions to sgx_arch.h
>   x86/sgx: Add SGX2 ENCLS leaf definitions (EAUG, EMODPR and EMODT)
>   x86/sgx: Add encls_faulted() helper
>   x86/sgx: Add helpers to expose ECREATE and EINIT to KVM
>   x86/sgx: Move provisioning device creation out of SGX driver
>   KVM: VMX: Convert vcpu_vmx.exit_reason to a union
>   KVM: x86: Export kvm_mmu_gva_to_gpa_{read,write}() for SGX (VMX)
>   KVM: x86: Define new #PF SGX error code bit
>   KVM: x86: Add SGX feature leaf to reverse CPUID lookup
>   KVM: VMX: Add basic handling of VM-Exit from SGX enclave
>   KVM: VMX: Frame in ENCLS handler for SGX virtualization
>   KVM: VMX: Add SGX ENCLS[ECREATE] handler to enforce CPUID restrictions
>   KVM: VMX: Add emulation of SGX Launch Control LE hash MSRs
>   KVM: VMX: Add ENCLS[EINIT] handler to support SGX Launch Control (LC)
>   KVM: VMX: Enable SGX virtualization for SGX1, SGX2 and LC
>   KVM: x86: Add capability to grant VM access to privileged SGX
>     attribute
> 
>  Documentation/virt/kvm/api.rst                |  23 +
>  arch/x86/Kconfig                              |  12 +
>  arch/x86/include/asm/cpufeature.h             |   5 +-
>  arch/x86/include/asm/cpufeatures.h            |   6 +-
>  arch/x86/include/asm/disabled-features.h      |   7 +-
>  arch/x86/include/asm/kvm_host.h               |   5 +
>  arch/x86/include/asm/required-features.h      |   2 +-
>  arch/x86/include/asm/sgx.h                    |  19 +
>  .../cpu/sgx/arch.h => include/asm/sgx_arch.h} |  20 +
>  arch/x86/include/asm/vmx.h                    |   1 +
>  arch/x86/include/uapi/asm/vmx.h               |   1 +
>  arch/x86/kernel/cpu/common.c                  |   4 +
>  arch/x86/kernel/cpu/feat_ctl.c                |  50 +-
>  arch/x86/kernel/cpu/sgx/Makefile              |   1 +
>  arch/x86/kernel/cpu/sgx/driver.c              |  17 -
>  arch/x86/kernel/cpu/sgx/encl.c                |   2 +-
>  arch/x86/kernel/cpu/sgx/encls.h               |  29 +-
>  arch/x86/kernel/cpu/sgx/ioctl.c               |  23 +-
>  arch/x86/kernel/cpu/sgx/main.c                |  79 ++-
>  arch/x86/kernel/cpu/sgx/sgx.h                 |   5 +-
>  arch/x86/kernel/cpu/sgx/virt.c                | 318 ++++++++++++
>  arch/x86/kernel/cpu/sgx/virt.h                |  14 +
>  arch/x86/kvm/Makefile                         |   2 +
>  arch/x86/kvm/cpuid.c                          |  58 ++-
>  arch/x86/kvm/cpuid.h                          |   1 +
>  arch/x86/kvm/vmx/nested.c                     |  70 ++-
>  arch/x86/kvm/vmx/nested.h                     |   5 +
>  arch/x86/kvm/vmx/sgx.c                        | 462 ++++++++++++++++++
>  arch/x86/kvm/vmx/sgx.h                        |  34 ++
>  arch/x86/kvm/vmx/vmcs12.c                     |   1 +
>  arch/x86/kvm/vmx/vmcs12.h                     |   4 +-
>  arch/x86/kvm/vmx/vmx.c                        | 171 +++++--
>  arch/x86/kvm/vmx/vmx.h                        |  27 +-
>  arch/x86/kvm/x86.c                            |  24 +
>  include/uapi/linux/kvm.h                      |   1 +
>  tools/testing/selftests/sgx/defines.h         |   2 +-
>  36 files changed, 1366 insertions(+), 139 deletions(-)
>  create mode 100644 arch/x86/include/asm/sgx.h
>  rename arch/x86/{kernel/cpu/sgx/arch.h => include/asm/sgx_arch.h} (96%)
>  create mode 100644 arch/x86/kernel/cpu/sgx/virt.c
>  create mode 100644 arch/x86/kernel/cpu/sgx/virt.h
>  create mode 100644 arch/x86/kvm/vmx/sgx.c
>  create mode 100644 arch/x86/kvm/vmx/sgx.h
>