All of lore.kernel.org
 help / color / mirror / Atom feed
* [PATCH v4 00/33] Qemu SGX virtualization
@ 2021-07-19 11:21 Yang Zhong
  2021-07-19 11:21 ` [PATCH v4 01/33] memory: Add RAM_PROTECTED flag to skip IOMMU mappings Yang Zhong
                   ` (35 more replies)
  0 siblings, 36 replies; 61+ messages in thread
From: Yang Zhong @ 2021-07-19 11:21 UTC (permalink / raw)
  To: qemu-devel; +Cc: yang.zhong, seanjc, kai.huang, jarkko, pbonzini, eblake

Since Sean Christopherson has left Intel and i am responsible for Qemu SGX
upstream work. His @intel.com address will be bouncing and his new email(
seanjc@google.com) is also in CC lists.

This series is Qemu SGX virtualization implementation rebased on latest
Qemu release. The numa support for SGX will be sent in another patchset
once this basic SGX patchset are merged.

You can find Qemu repo here:

    https://github.com/intel/qemu-sgx.git upstream

If you want to try SGX, you can directly install the linux release(at least 5.13.0-rc1+)
since kvm SGX has been merged into linux release.

    https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git

To simplify, you'd better install linux on host and guest, which can support
SGX on host and guest kernel. And to me, use below reference command to boot
SGX guest:

    #qemu-system-x86_64 \
        ...... \
        -cpu host,+sgx-provisionkey \
        -object memory-backend-epc,id=mem1,size=64M,prealloc=on \
        -object memory-backend-epc,id=mem2,size=28M \
        -M sgx-epc.0.memdev=mem1,sgx-epc.1.memdev=mem2

Overview
========

Intel Software Guard eXtensions (SGX) is a set of instructions and mechanisms
for memory accesses in order to provide security accesses for sensitive
applications and data. SGX allows an application to use it's pariticular
address space as an *enclave*, which is a protected area provides confidentiality
and integrity even in the presence of privileged malware. Accesses to the
enclave memory area from any software not resident in the enclave are prevented,
including those from privileged software.

SGX virtaulization
==================

The KVM SGX creates one new misc device, sgx_vepc, and Qemu will open '/dev/sgx_vepc'
device node to mmap() host EPC memory to guest. The Qemu also adds 'sgx-epc' device
to expose EPC sections to guest through CPUID and ACPI table.  The Qemu SGX also
supports multiple virtual EPC sections to guest, we just put them together physically
contiguous for the sake of simplicity. The kernel SGX NUMA has been merged into Linux
tip tree, we will support this function in the next phase.

Although the current host SGX subsystem can not support SGX2 feature, the KVM/Qemu
implementation still expose this feature to guest. Guest SGX2 support doesn't have
interaction with host kernel SGX driver, the SGX guest can normally use those new
instructions.

As for SGX virtualization detailed infomation, please reference docs/intel-sgx.txt
docuement(patch 33).

Changelog:
=========

(Changelog here is for global changes, please see each patch's changelog for changes
made to specific patch.)

v3-->v4:
   - Rebased the sgx patches into latest Qemu release.
   - Moved sgx compound property setter/getter from MachineState to X86MachineState(Paolo).
   - Re-defined struct SgxEPC, removed 'id' property and added struct SgxEPCList for
     sgx-epc.0.{memdev}(Paolo).
   - Removed g_malloc0(), and changed the 'SGXEPCState *sgx_epc' to 'SGXEPCState sgx_epc'
     in struct PCMachineState(Paolo).
   - Changed the SGX compound property cmdline from sgx-epc.{memdev}.0 to
     sgx-epc.0.{memdev}(Paolo).
   - Removed the signature from the 'git format-patch' command(Jarkko).

v2-->v3:
   - Rebased the sgx patches into latest Qemu release.
   - Implemented the compound property for SGX, ref patch5, the command from '-sgx-epc'
     to '-M'(Paolo).
   - Moved the sgx common code from sgx-epc.c to sgx.c. The sgx-epc.c is
     only responsible for virtual epc device.
   - Removed the previous patch13(linux-headers: Add placeholder for KVM_CAP_SGX_ATTRIBUTE)
     because ehabkost@redhat.com updated Linux headers to 5.13-rc4 with commit 278f064e452.
   - Updated the patch1 because ram_flags were changed by David Hildenbra.
   - Added one patch24, which avoid reset operation caused by bios reset.
   - Added one patch25, which make prealloc property consistent with Qemu cmdline during VM
     reset.

v1-->v2:
   - Rebased the sgx patches into latest Qemu release.
   - Unified the "share" and "protected" arguments with ram_flags in the
     memory_region_init_ram_from_fd()(Paolo).
   - Added the new MemoryBackendEpcProperties and related documents(Eric Blake).
   - Changed the KVM_CAP_SGX_ATTRIBUTE from 195 to 196(Kai).
   - Changed the version and some grammar issues(Eric Blake).

Sean Christopherson (21):
  memory: Add RAM_PROTECTED flag to skip IOMMU mappings
  hostmem: Add hostmem-epc as a backend for SGX EPC
  i386: Add 'sgx-epc' device to expose EPC sections to guest
  vl: Add sgx compound properties to expose SGX EPC sections to guest
  i386: Add primary SGX CPUID and MSR defines
  i386: Add SGX CPUID leaf FEAT_SGX_12_0_EAX
  i386: Add SGX CPUID leaf FEAT_SGX_12_0_EBX
  i386: Add SGX CPUID leaf FEAT_SGX_12_1_EAX
  i386: Add get/set/migrate support for SGX_LEPUBKEYHASH MSRs
  i386: Add feature control MSR dependency when SGX is enabled
  i386: Update SGX CPUID info according to hardware/KVM/user input
  i386: kvm: Add support for exposing PROVISIONKEY to guest
  i386: Propagate SGX CPUID sub-leafs to KVM
  Adjust min CPUID level to 0x12 when SGX is enabled
  hw/i386/fw_cfg: Set SGX bits in feature control fw_cfg accordingly
  hw/i386/pc: Account for SGX EPC sections when calculating device
    memory
  i386/pc: Add e820 entry for SGX EPC section(s)
  i386: acpi: Add SGX EPC entry to ACPI tables
  q35: Add support for SGX EPC
  i440fx: Add support for SGX EPC
  doc: Add the SGX doc

Yang Zhong (12):
  qom: Add memory-backend-epc ObjectOptions support
  hostmem-epc: Add the reset interface for EPC backend reset
  sgx-epc: Add the reset interface for sgx-epc virt device
  sgx-epc: Avoid bios reset during sgx epc initialization
  hostmem-epc: Make prealloc consistent with qemu cmdline during reset
  qmp: Add query-sgx command
  hmp: Add 'info sgx' command
  i386: Add sgx_get_info() interface
  bitops: Support 32 and 64 bit mask macro
  qmp: Add the qmp_query_sgx_capabilities()
  Kconfig: Add CONFIG_SGX support
  sgx-epc: Add the fill_device_info() callback support

 backends/hostmem-epc.c                   | 118 ++++++++++
 backends/meson.build                     |   1 +
 configs/devices/i386-softmmu/default.mak |   1 +
 docs/intel-sgx.txt                       | 167 ++++++++++++++
 hmp-commands-info.hx                     |  15 ++
 hw/i386/Kconfig                          |   5 +
 hw/i386/acpi-build.c                     |  22 ++
 hw/i386/fw_cfg.c                         |  10 +-
 hw/i386/meson.build                      |   2 +
 hw/i386/pc.c                             |  15 +-
 hw/i386/pc_piix.c                        |   4 +
 hw/i386/pc_q35.c                         |   3 +
 hw/i386/sgx-epc.c                        | 265 +++++++++++++++++++++++
 hw/i386/sgx-stub.c                       |  13 ++
 hw/i386/sgx.c                            | 170 +++++++++++++++
 hw/i386/x86.c                            |  29 +++
 hw/vfio/common.c                         |   1 +
 include/exec/memory.h                    |  15 +-
 include/hw/i386/pc.h                     |  10 +
 include/hw/i386/sgx-epc.h                |  68 ++++++
 include/hw/i386/x86.h                    |   1 +
 include/monitor/hmp.h                    |   1 +
 include/qemu/bitops.h                    |   7 +
 monitor/hmp-cmds.c                       |  32 +++
 monitor/qmp-cmds.c                       |  19 ++
 qapi/machine.json                        |  52 ++++-
 qapi/misc.json                           |  61 ++++++
 qapi/qom.json                            |  19 ++
 qemu-options.hx                          |  10 +-
 softmmu/memory.c                         |   5 +
 softmmu/physmem.c                        |   3 +-
 stubs/meson.build                        |   1 +
 stubs/sgx-stub.c                         |  12 +
 target/i386/cpu.c                        | 168 +++++++++++++-
 target/i386/cpu.h                        |  16 ++
 target/i386/kvm/kvm.c                    |  75 +++++++
 target/i386/kvm/kvm_i386.h               |   2 +
 target/i386/machine.c                    |  20 ++
 tests/qtest/qmp-cmd-test.c               |   2 +
 39 files changed, 1430 insertions(+), 10 deletions(-)
 create mode 100644 backends/hostmem-epc.c
 create mode 100644 docs/intel-sgx.txt
 create mode 100644 hw/i386/sgx-epc.c
 create mode 100644 hw/i386/sgx-stub.c
 create mode 100644 hw/i386/sgx.c
 create mode 100644 include/hw/i386/sgx-epc.h
 create mode 100644 stubs/sgx-stub.c



^ permalink raw reply	[flat|nested] 61+ messages in thread

* [PATCH v4 01/33] memory: Add RAM_PROTECTED flag to skip IOMMU mappings
  2021-07-19 11:21 [PATCH v4 00/33] Qemu SGX virtualization Yang Zhong
@ 2021-07-19 11:21 ` Yang Zhong
  2021-07-19 11:21 ` [PATCH v4 02/33] hostmem: Add hostmem-epc as a backend for SGX EPC Yang Zhong
                   ` (34 subsequent siblings)
  35 siblings, 0 replies; 61+ messages in thread
From: Yang Zhong @ 2021-07-19 11:21 UTC (permalink / raw)
  To: qemu-devel; +Cc: yang.zhong, seanjc, kai.huang, jarkko, pbonzini, eblake

From: Sean Christopherson <sean.j.christopherson@intel.com>

Add a new RAMBlock flag to denote "protected" memory, i.e. memory that
looks and acts like RAM but is inaccessible via normal mechanisms,
including DMA.  Use the flag to skip protected memory regions when
mapping RAM for DMA in VFIO.

Signed-off-by: Sean Christopherson <sean.j.christopherson@intel.com>
Signed-off-by: Yang Zhong <yang.zhong@intel.com>

v1-->v2:
   - Unified the "share" and "protected" arguments with ram_flags in the
     memory_region_init_ram_from_fd()(Paolo).
---
 hw/vfio/common.c      |  1 +
 include/exec/memory.h | 15 ++++++++++++++-
 softmmu/memory.c      |  5 +++++
 softmmu/physmem.c     |  3 ++-
 4 files changed, 22 insertions(+), 2 deletions(-)

diff --git a/hw/vfio/common.c b/hw/vfio/common.c
index 8728d4d5c2..1289cfa8be 100644
--- a/hw/vfio/common.c
+++ b/hw/vfio/common.c
@@ -562,6 +562,7 @@ static bool vfio_listener_skipped_section(MemoryRegionSection *section)
 {
     return (!memory_region_is_ram(section->mr) &&
             !memory_region_is_iommu(section->mr)) ||
+           memory_region_is_protected(section->mr) ||
            /*
             * Sizing an enabled 64-bit BAR can cause spurious mappings to
             * addresses in the upper part of the 64-bit address space.  These
diff --git a/include/exec/memory.h b/include/exec/memory.h
index c3d417d317..9446874d21 100644
--- a/include/exec/memory.h
+++ b/include/exec/memory.h
@@ -190,6 +190,9 @@ typedef struct IOMMUTLBEvent {
  */
 #define RAM_NORESERVE (1 << 7)
 
+/* RAM that isn't accessible through normal means. */
+#define RAM_PROTECTED (1 << 8)
+
 static inline void iommu_notifier_init(IOMMUNotifier *n, IOMMUNotify fn,
                                        IOMMUNotifierFlag flags,
                                        hwaddr start, hwaddr end,
@@ -1267,7 +1270,7 @@ void memory_region_init_ram_from_file(MemoryRegion *mr,
  * @name: the name of the region.
  * @size: size of the region.
  * @ram_flags: RamBlock flags. Supported flags: RAM_SHARED, RAM_PMEM,
- *             RAM_NORESERVE.
+ *             RAM_NORESERVE, RAM_PROTECTED.
  * @fd: the fd to mmap.
  * @offset: offset within the file referenced by fd
  * @errp: pointer to Error*, to store an error if it happens.
@@ -1568,6 +1571,16 @@ static inline bool memory_region_is_romd(MemoryRegion *mr)
     return mr->rom_device && mr->romd_mode;
 }
 
+/**
+ * memory_region_is_protected: check whether a memory region is protected
+ *
+ * Returns %true if a memory region is protected RAM and cannot be accessed
+ * via standard mechanisms, e.g. DMA.
+ *
+ * @mr: the memory region being queried
+ */
+bool memory_region_is_protected(MemoryRegion *mr);
+
 /**
  * memory_region_get_iommu: check whether a memory region is an iommu
  *
diff --git a/softmmu/memory.c b/softmmu/memory.c
index bfedaf9c4d..54cd0e9824 100644
--- a/softmmu/memory.c
+++ b/softmmu/memory.c
@@ -1811,6 +1811,11 @@ bool memory_region_is_ram_device(MemoryRegion *mr)
     return mr->ram_device;
 }
 
+bool memory_region_is_protected(MemoryRegion *mr)
+{
+    return mr->ram && (mr->ram_block->flags & RAM_PROTECTED);
+}
+
 uint8_t memory_region_get_dirty_log_mask(MemoryRegion *mr)
 {
     uint8_t mask = mr->dirty_log_mask;
diff --git a/softmmu/physmem.c b/softmmu/physmem.c
index 3c1912a1a0..f3b646658f 100644
--- a/softmmu/physmem.c
+++ b/softmmu/physmem.c
@@ -2052,7 +2052,8 @@ RAMBlock *qemu_ram_alloc_from_fd(ram_addr_t size, MemoryRegion *mr,
     int64_t file_size, file_align;
 
     /* Just support these ram flags by now. */
-    assert((ram_flags & ~(RAM_SHARED | RAM_PMEM | RAM_NORESERVE)) == 0);
+    assert((ram_flags & ~(RAM_SHARED | RAM_PMEM | RAM_NORESERVE |
+                          RAM_PROTECTED)) == 0);
 
     if (xen_enabled()) {
         error_setg(errp, "-mem-path not supported with Xen");


^ permalink raw reply related	[flat|nested] 61+ messages in thread

* [PATCH v4 02/33] hostmem: Add hostmem-epc as a backend for SGX EPC
  2021-07-19 11:21 [PATCH v4 00/33] Qemu SGX virtualization Yang Zhong
  2021-07-19 11:21 ` [PATCH v4 01/33] memory: Add RAM_PROTECTED flag to skip IOMMU mappings Yang Zhong
@ 2021-07-19 11:21 ` Yang Zhong
  2021-07-19 11:21 ` [PATCH v4 03/33] qom: Add memory-backend-epc ObjectOptions support Yang Zhong
                   ` (33 subsequent siblings)
  35 siblings, 0 replies; 61+ messages in thread
From: Yang Zhong @ 2021-07-19 11:21 UTC (permalink / raw)
  To: qemu-devel; +Cc: yang.zhong, seanjc, kai.huang, jarkko, pbonzini, eblake

From: Sean Christopherson <sean.j.christopherson@intel.com>

EPC (Enclave Page Cahe) is a specialized type of memory used by Intel
SGX (Software Guard Extensions).  The SDM desribes EPC as:

    The Enclave Page Cache (EPC) is the secure storage used to store
    enclave pages when they are a part of an executing enclave. For an
    EPC page, hardware performs additional access control checks to
    restrict access to the page. After the current page access checks
    and translations are performed, the hardware checks that the EPC
    page is accessible to the program currently executing. Generally an
    EPC page is only accessed by the owner of the executing enclave or
    an instruction which is setting up an EPC page.

Because of its unique requirements, Linux manages EPC separately from
normal memory.  Similar to memfd, the device /dev/sgx_vepc can be
opened to obtain a file descriptor which can in turn be used to mmap()
EPC memory.

Signed-off-by: Sean Christopherson <sean.j.christopherson@intel.com>
Signed-off-by: Yang Zhong <yang.zhong@intel.com>

v1-->v2:
   - Unified the "share" and "protected" arguments with ram_flags in the
     memory_region_init_ram_from_fd()(Paolo).
---
 backends/hostmem-epc.c | 92 ++++++++++++++++++++++++++++++++++++++++++
 backends/meson.build   |  1 +
 2 files changed, 93 insertions(+)
 create mode 100644 backends/hostmem-epc.c

diff --git a/backends/hostmem-epc.c b/backends/hostmem-epc.c
new file mode 100644
index 0000000000..b512a68cb0
--- /dev/null
+++ b/backends/hostmem-epc.c
@@ -0,0 +1,92 @@
+/*
+ * QEMU host SGX EPC memory backend
+ *
+ * Copyright (C) 2019 Intel Corporation
+ *
+ * Authors:
+ *   Sean Christopherson <sean.j.christopherson@intel.com>
+ *
+ * This work is licensed under the terms of the GNU GPL, version 2 or later.
+ * See the COPYING file in the top-level directory.
+ */
+#include <sys/ioctl.h>
+
+#include "qemu/osdep.h"
+#include "qemu-common.h"
+#include "qom/object_interfaces.h"
+#include "qapi/error.h"
+#include "sysemu/hostmem.h"
+
+#define TYPE_MEMORY_BACKEND_EPC "memory-backend-epc"
+
+#define MEMORY_BACKEND_EPC(obj)                                        \
+    OBJECT_CHECK(HostMemoryBackendEpc, (obj), TYPE_MEMORY_BACKEND_EPC)
+
+typedef struct HostMemoryBackendEpc HostMemoryBackendEpc;
+
+struct HostMemoryBackendEpc {
+    HostMemoryBackend parent_obj;
+};
+
+static void
+sgx_epc_backend_memory_alloc(HostMemoryBackend *backend, Error **errp)
+{
+    uint32_t ram_flags;
+    char *name;
+    int fd;
+
+    if (!backend->size) {
+        error_setg(errp, "can't create backend with size 0");
+        return;
+    }
+
+    fd = qemu_open_old("/dev/sgx_vepc", O_RDWR);
+    if (fd < 0) {
+        error_setg_errno(errp, errno,
+                         "failed to open /dev/sgx_vepc to alloc SGX EPC");
+        return;
+    }
+
+    name = object_get_canonical_path(OBJECT(backend));
+    ram_flags = (backend->share ? RAM_SHARED : 0) | RAM_PROTECTED;
+    memory_region_init_ram_from_fd(&backend->mr, OBJECT(backend),
+                                   name, backend->size, ram_flags,
+                                   fd, 0, errp);
+    g_free(name);
+}
+
+static void sgx_epc_backend_instance_init(Object *obj)
+{
+    HostMemoryBackend *m = MEMORY_BACKEND(obj);
+
+    m->share = true;
+    m->merge = false;
+    m->dump = false;
+}
+
+static void sgx_epc_backend_class_init(ObjectClass *oc, void *data)
+{
+    HostMemoryBackendClass *bc = MEMORY_BACKEND_CLASS(oc);
+
+    bc->alloc = sgx_epc_backend_memory_alloc;
+}
+
+static const TypeInfo sgx_epc_backed_info = {
+    .name = TYPE_MEMORY_BACKEND_EPC,
+    .parent = TYPE_MEMORY_BACKEND,
+    .instance_init = sgx_epc_backend_instance_init,
+    .class_init = sgx_epc_backend_class_init,
+    .instance_size = sizeof(HostMemoryBackendEpc),
+};
+
+static void register_types(void)
+{
+    int fd = qemu_open_old("/dev/sgx_vepc", O_RDWR);
+    if (fd >= 0) {
+        close(fd);
+
+        type_register_static(&sgx_epc_backed_info);
+    }
+}
+
+type_init(register_types);
diff --git a/backends/meson.build b/backends/meson.build
index d4221831fc..46fd16b269 100644
--- a/backends/meson.build
+++ b/backends/meson.build
@@ -16,5 +16,6 @@ softmmu_ss.add(when: ['CONFIG_VHOST_USER', 'CONFIG_VIRTIO'], if_true: files('vho
 softmmu_ss.add(when: 'CONFIG_VIRTIO_CRYPTO', if_true: files('cryptodev-vhost.c'))
 softmmu_ss.add(when: ['CONFIG_VIRTIO_CRYPTO', 'CONFIG_VHOST_CRYPTO'], if_true: files('cryptodev-vhost-user.c'))
 softmmu_ss.add(when: 'CONFIG_GIO', if_true: [files('dbus-vmstate.c'), gio])
+softmmu_ss.add(when: 'CONFIG_LINUX', if_true: files('hostmem-epc.c'))
 
 subdir('tpm')


^ permalink raw reply related	[flat|nested] 61+ messages in thread

* [PATCH v4 03/33] qom: Add memory-backend-epc ObjectOptions support
  2021-07-19 11:21 [PATCH v4 00/33] Qemu SGX virtualization Yang Zhong
  2021-07-19 11:21 ` [PATCH v4 01/33] memory: Add RAM_PROTECTED flag to skip IOMMU mappings Yang Zhong
  2021-07-19 11:21 ` [PATCH v4 02/33] hostmem: Add hostmem-epc as a backend for SGX EPC Yang Zhong
@ 2021-07-19 11:21 ` Yang Zhong
  2021-07-19 11:21 ` [PATCH v4 04/33] i386: Add 'sgx-epc' device to expose EPC sections to guest Yang Zhong
                   ` (32 subsequent siblings)
  35 siblings, 0 replies; 61+ messages in thread
From: Yang Zhong @ 2021-07-19 11:21 UTC (permalink / raw)
  To: qemu-devel; +Cc: yang.zhong, seanjc, kai.huang, jarkko, pbonzini, eblake

Add the new 'memory-backend-epc' user creatable QOM object in
the ObjectOptions to support SGX since v6.1, or the sgx backend
object cannot bootup.

Signed-off-by: Yang Zhong <yang.zhong@intel.com>

v1-->v2:
   - Added the new MemoryBackendEpcProperties and related documents,
     and updated the blurb(Eric Blake).
---
 qapi/qom.json | 19 +++++++++++++++++++
 1 file changed, 19 insertions(+)

diff --git a/qapi/qom.json b/qapi/qom.json
index 652be317b8..aba77aa97b 100644
--- a/qapi/qom.json
+++ b/qapi/qom.json
@@ -642,6 +642,23 @@
             '*hugetlbsize': 'size',
             '*seal': 'bool' } }
 
+##
+# @MemoryBackendEpcProperties:
+#
+# Properties for memory-backend-epc objects.
+#
+# The @share boolean option is true by default with epc
+#
+# The @merge boolean option is false by default with epc
+#
+# The @dump boolean option is false by default with epc
+#
+# Since: 6.1
+##
+{ 'struct': 'MemoryBackendEpcProperties',
+  'base': 'MemoryBackendProperties',
+  'data': {} }
+
 ##
 # @PrManagerHelperProperties:
 #
@@ -792,6 +809,7 @@
     { 'name': 'memory-backend-memfd',
       'if': 'defined(CONFIG_LINUX)' },
     'memory-backend-ram',
+    'memory-backend-epc',
     'pef-guest',
     'pr-manager-helper',
     'qtest',
@@ -850,6 +868,7 @@
       'memory-backend-memfd':       { 'type': 'MemoryBackendMemfdProperties',
                                       'if': 'defined(CONFIG_LINUX)' },
       'memory-backend-ram':         'MemoryBackendProperties',
+      'memory-backend-epc':         'MemoryBackendEpcProperties',
       'pr-manager-helper':          'PrManagerHelperProperties',
       'qtest':                      'QtestProperties',
       'rng-builtin':                'RngProperties',


^ permalink raw reply related	[flat|nested] 61+ messages in thread

* [PATCH v4 04/33] i386: Add 'sgx-epc' device to expose EPC sections to guest
  2021-07-19 11:21 [PATCH v4 00/33] Qemu SGX virtualization Yang Zhong
                   ` (2 preceding siblings ...)
  2021-07-19 11:21 ` [PATCH v4 03/33] qom: Add memory-backend-epc ObjectOptions support Yang Zhong
@ 2021-07-19 11:21 ` Yang Zhong
  2021-09-14  6:36   ` Philippe Mathieu-Daudé
  2021-07-19 11:21 ` [PATCH v4 05/33] vl: Add sgx compound properties to expose SGX " Yang Zhong
                   ` (31 subsequent siblings)
  35 siblings, 1 reply; 61+ messages in thread
From: Yang Zhong @ 2021-07-19 11:21 UTC (permalink / raw)
  To: qemu-devel; +Cc: yang.zhong, seanjc, kai.huang, jarkko, pbonzini, eblake

From: Sean Christopherson <sean.j.christopherson@intel.com>

SGX EPC is enumerated through CPUID, i.e. EPC "devices" need to be
realized prior to realizing the vCPUs themselves, which occurs long
before generic devices are parsed and realized.  Because of this,
do not allow 'sgx-epc' devices to be instantiated after vCPUS have
been created.

The 'sgx-epc' device is essentially a placholder at this time, it will
be fully implemented in a future patch along with a dedicated command
to create 'sgx-epc' devices.

Signed-off-by: Sean Christopherson <sean.j.christopherson@intel.com>
Signed-off-by: Yang Zhong <yang.zhong@intel.com>
---
 hw/i386/meson.build       |   1 +
 hw/i386/sgx-epc.c         | 161 ++++++++++++++++++++++++++++++++++++++
 include/hw/i386/sgx-epc.h |  44 +++++++++++
 3 files changed, 206 insertions(+)
 create mode 100644 hw/i386/sgx-epc.c
 create mode 100644 include/hw/i386/sgx-epc.h

diff --git a/hw/i386/meson.build b/hw/i386/meson.build
index 80dad29f2b..27476b36bb 100644
--- a/hw/i386/meson.build
+++ b/hw/i386/meson.build
@@ -5,6 +5,7 @@ i386_ss.add(files(
   'e820_memory_layout.c',
   'multiboot.c',
   'x86.c',
+  'sgx-epc.c',
 ))
 
 i386_ss.add(when: 'CONFIG_X86_IOMMU', if_true: files('x86-iommu.c'),
diff --git a/hw/i386/sgx-epc.c b/hw/i386/sgx-epc.c
new file mode 100644
index 0000000000..aa487dea79
--- /dev/null
+++ b/hw/i386/sgx-epc.c
@@ -0,0 +1,161 @@
+/*
+ * SGX EPC device
+ *
+ * Copyright (C) 2019 Intel Corporation
+ *
+ * Authors:
+ *   Sean Christopherson <sean.j.christopherson@intel.com>
+ *
+ * This work is licensed under the terms of the GNU GPL, version 2 or later.
+ * See the COPYING file in the top-level directory.
+ */
+#include "qemu/osdep.h"
+#include "hw/i386/pc.h"
+#include "hw/i386/sgx-epc.h"
+#include "hw/mem/memory-device.h"
+#include "hw/qdev-properties.h"
+#include "monitor/qdev.h"
+#include "qapi/error.h"
+#include "qapi/visitor.h"
+#include "qemu/config-file.h"
+#include "qemu/error-report.h"
+#include "qemu/option.h"
+#include "qemu/units.h"
+#include "target/i386/cpu.h"
+#include "exec/address-spaces.h"
+
+static Property sgx_epc_properties[] = {
+    DEFINE_PROP_UINT64(SGX_EPC_ADDR_PROP, SGXEPCDevice, addr, 0),
+    DEFINE_PROP_LINK(SGX_EPC_MEMDEV_PROP, SGXEPCDevice, hostmem,
+                     TYPE_MEMORY_BACKEND, HostMemoryBackend *),
+    DEFINE_PROP_END_OF_LIST(),
+};
+
+static void sgx_epc_get_size(Object *obj, Visitor *v, const char *name,
+                             void *opaque, Error **errp)
+{
+    Error *local_err = NULL;
+    uint64_t value;
+
+    value = memory_device_get_region_size(MEMORY_DEVICE(obj), &local_err);
+    if (local_err) {
+        error_propagate(errp, local_err);
+        return;
+    }
+
+    visit_type_uint64(v, name, &value, errp);
+}
+
+static void sgx_epc_init(Object *obj)
+{
+    object_property_add(obj, SGX_EPC_SIZE_PROP, "uint64", sgx_epc_get_size,
+                        NULL, NULL, NULL);
+}
+
+static void sgx_epc_realize(DeviceState *dev, Error **errp)
+{
+    PCMachineState *pcms = PC_MACHINE(qdev_get_machine());
+    X86MachineState *x86ms = X86_MACHINE(pcms);
+    SGXEPCDevice *epc = SGX_EPC(dev);
+    const char *path;
+
+    if (x86ms->boot_cpus != 0) {
+        error_setg(errp, "'" TYPE_SGX_EPC "' can't be created after vCPUs,"
+                         "e.g. via -device");
+        return;
+    }
+
+    if (!epc->hostmem) {
+        error_setg(errp, "'" SGX_EPC_MEMDEV_PROP "' property is not set");
+        return;
+    } else if (host_memory_backend_is_mapped(epc->hostmem)) {
+        path = object_get_canonical_path_component(OBJECT(epc->hostmem));
+        error_setg(errp, "can't use already busy memdev: %s", path);
+        return;
+    }
+
+    error_setg(errp, "'" TYPE_SGX_EPC "' not supported");
+}
+
+static void sgx_epc_unrealize(DeviceState *dev)
+{
+    SGXEPCDevice *epc = SGX_EPC(dev);
+
+    host_memory_backend_set_mapped(epc->hostmem, false);
+}
+
+static uint64_t sgx_epc_md_get_addr(const MemoryDeviceState *md)
+{
+    const SGXEPCDevice *epc = SGX_EPC(md);
+
+    return epc->addr;
+}
+
+static void sgx_epc_md_set_addr(MemoryDeviceState *md, uint64_t addr,
+                                Error **errp)
+{
+    object_property_set_uint(OBJECT(md), SGX_EPC_ADDR_PROP, addr, errp);
+}
+
+static uint64_t sgx_epc_md_get_plugged_size(const MemoryDeviceState *md,
+                                            Error **errp)
+{
+    return 0;
+}
+
+static MemoryRegion *sgx_epc_md_get_memory_region(MemoryDeviceState *md,
+                                                  Error **errp)
+{
+    SGXEPCDevice *epc = SGX_EPC(md);
+
+    if (!epc->hostmem) {
+        error_setg(errp, "'" SGX_EPC_MEMDEV_PROP "' property must be set");
+        return NULL;
+    }
+
+    return host_memory_backend_get_memory(epc->hostmem);
+}
+
+static void sgx_epc_md_fill_device_info(const MemoryDeviceState *md,
+                                        MemoryDeviceInfo *info)
+{
+    /* TODO */
+}
+
+static void sgx_epc_class_init(ObjectClass *oc, void *data)
+{
+    DeviceClass *dc = DEVICE_CLASS(oc);
+    MemoryDeviceClass *mdc = MEMORY_DEVICE_CLASS(oc);
+
+    dc->hotpluggable = false;
+    dc->realize = sgx_epc_realize;
+    dc->unrealize = sgx_epc_unrealize;
+    dc->desc = "SGX EPC section";
+    device_class_set_props(dc, sgx_epc_properties);
+
+    mdc->get_addr = sgx_epc_md_get_addr;
+    mdc->set_addr = sgx_epc_md_set_addr;
+    mdc->get_plugged_size = sgx_epc_md_get_plugged_size;
+    mdc->get_memory_region = sgx_epc_md_get_memory_region;
+    mdc->fill_device_info = sgx_epc_md_fill_device_info;
+}
+
+static TypeInfo sgx_epc_info = {
+    .name          = TYPE_SGX_EPC,
+    .parent        = TYPE_DEVICE,
+    .instance_size = sizeof(SGXEPCDevice),
+    .instance_init = sgx_epc_init,
+    .class_init    = sgx_epc_class_init,
+    .class_size    = sizeof(DeviceClass),
+    .interfaces = (InterfaceInfo[]) {
+        { TYPE_MEMORY_DEVICE },
+        { }
+    },
+};
+
+static void sgx_epc_register_types(void)
+{
+    type_register_static(&sgx_epc_info);
+}
+
+type_init(sgx_epc_register_types)
diff --git a/include/hw/i386/sgx-epc.h b/include/hw/i386/sgx-epc.h
new file mode 100644
index 0000000000..5fd9ae2d0c
--- /dev/null
+++ b/include/hw/i386/sgx-epc.h
@@ -0,0 +1,44 @@
+/*
+ * SGX EPC device
+ *
+ * Copyright (C) 2019 Intel Corporation
+ *
+ * Authors:
+ *   Sean Christopherson <sean.j.christopherson@intel.com>
+ *
+ * This work is licensed under the terms of the GNU GPL, version 2 or later.
+ * See the COPYING file in the top-level directory.
+ */
+#ifndef QEMU_SGX_EPC_H
+#define QEMU_SGX_EPC_H
+
+#include "sysemu/hostmem.h"
+
+#define TYPE_SGX_EPC "sgx-epc"
+#define SGX_EPC(obj) \
+    OBJECT_CHECK(SGXEPCDevice, (obj), TYPE_SGX_EPC)
+#define SGX_EPC_CLASS(oc) \
+    OBJECT_CLASS_CHECK(SGXEPCDeviceClass, (oc), TYPE_SGX_EPC)
+#define SGX_EPC_GET_CLASS(obj) \
+    OBJECT_GET_CLASS(SGXEPCDeviceClass, (obj), TYPE_SGX_EPC)
+
+#define SGX_EPC_ADDR_PROP "addr"
+#define SGX_EPC_SIZE_PROP "size"
+#define SGX_EPC_MEMDEV_PROP "memdev"
+
+/**
+ * SGXEPCDevice:
+ * @addr: starting guest physical address, where @SGXEPCDevice is mapped.
+ *         Default value: 0, means that address is auto-allocated.
+ * @hostmem: host memory backend providing memory for @SGXEPCDevice
+ */
+typedef struct SGXEPCDevice {
+    /* private */
+    DeviceState parent_obj;
+
+    /* public */
+    uint64_t addr;
+    HostMemoryBackend *hostmem;
+} SGXEPCDevice;
+
+#endif


^ permalink raw reply related	[flat|nested] 61+ messages in thread

* [PATCH v4 05/33] vl: Add sgx compound properties to expose SGX EPC sections to guest
  2021-07-19 11:21 [PATCH v4 00/33] Qemu SGX virtualization Yang Zhong
                   ` (3 preceding siblings ...)
  2021-07-19 11:21 ` [PATCH v4 04/33] i386: Add 'sgx-epc' device to expose EPC sections to guest Yang Zhong
@ 2021-07-19 11:21 ` Yang Zhong
  2021-07-19 11:21 ` [PATCH v4 06/33] i386: Add primary SGX CPUID and MSR defines Yang Zhong
                   ` (30 subsequent siblings)
  35 siblings, 0 replies; 61+ messages in thread
From: Yang Zhong @ 2021-07-19 11:21 UTC (permalink / raw)
  To: qemu-devel; +Cc: yang.zhong, seanjc, kai.huang, jarkko, pbonzini, eblake

From: Sean Christopherson <sean.j.christopherson@intel.com>

Because SGX EPC is enumerated through CPUID, EPC "devices" need to be
realized prior to realizing the vCPUs themselves, i.e. long before
generic devices are parsed and realized.  From a virtualization
perspective, the CPUID aspect also means that EPC sections cannot be
hotplugged without paravirtualizing the guest kernel (hardware does
not support hotplugging as EPC sections must be locked down during
pre-boot to provide EPC's security properties).

So even though EPC sections could be realized through the generic
-devices command, they need to be created much earlier for them to
actually be usable by the guest.  Place all EPC sections in a
contiguous block, somewhat arbitrarily starting after RAM above 4g.
Ensuring EPC is in a contiguous region simplifies calculations, e.g.
device memory base, PCI hole, etc..., allows dynamic calculation of the
total EPC size, e.g. exposing EPC to guests does not require -maxmem,
and last but not least allows all of EPC to be enumerated in a single
ACPI entry, which is expected by some kernels, e.g. Windows 7 and 8.

The new compound properties command for sgx like below:
 ......
 -object memory-backend-epc,id=mem1,size=28M,prealloc=on \
 -object memory-backend-epc,id=mem2,size=10M \
 -M sgx-epc.0.memdev=mem1,sgx-epc.1.memdev=mem2

Signed-off-by: Sean Christopherson <sean.j.christopherson@intel.com>
Signed-off-by: Yang Zhong <yang.zhong@intel.com>

v3-->v4:
  - Moved sgx compound property setter/getter from MachineState
    to X86MachineState(Paolo).
  - Re-defined struct SgxEPC, removed 'id' property and added struct
    SgxEPCList for sgx-epc.0.{memdev}(Paolo).
  - Removed g_malloc0(), and changed the 'SGXEPCState *sgx_epc' to
    'SGXEPCState sgx_epc' in struct PCMachineState(Paolo).
  - Changed the SGX compound property cmdline from sgx-epc.{memdev}.0
    to sgx-epc.0.{memdev}(Paolo).

v2-->v3:
   - Removed the QemuOptsList for sgx-epc virtual device and used the '-M' to
     replace '-sgx-epc' with compound properties(Paolo).
---
 hw/i386/meson.build       |  1 +
 hw/i386/sgx-epc.c         | 20 ++++++++----
 hw/i386/sgx.c             | 67 +++++++++++++++++++++++++++++++++++++++
 hw/i386/x86.c             | 29 +++++++++++++++++
 include/hw/i386/pc.h      |  6 ++++
 include/hw/i386/sgx-epc.h | 14 ++++++++
 include/hw/i386/x86.h     |  1 +
 qapi/machine.json         | 26 +++++++++++++++
 qemu-options.hx           | 10 ++++--
 9 files changed, 166 insertions(+), 8 deletions(-)
 create mode 100644 hw/i386/sgx.c

diff --git a/hw/i386/meson.build b/hw/i386/meson.build
index 27476b36bb..fefce9e4ba 100644
--- a/hw/i386/meson.build
+++ b/hw/i386/meson.build
@@ -6,6 +6,7 @@ i386_ss.add(files(
   'multiboot.c',
   'x86.c',
   'sgx-epc.c',
+  'sgx.c'
 ))
 
 i386_ss.add(when: 'CONFIG_X86_IOMMU', if_true: files('x86-iommu.c'),
diff --git a/hw/i386/sgx-epc.c b/hw/i386/sgx-epc.c
index aa487dea79..924dea22f0 100644
--- a/hw/i386/sgx-epc.c
+++ b/hw/i386/sgx-epc.c
@@ -14,13 +14,8 @@
 #include "hw/i386/sgx-epc.h"
 #include "hw/mem/memory-device.h"
 #include "hw/qdev-properties.h"
-#include "monitor/qdev.h"
 #include "qapi/error.h"
 #include "qapi/visitor.h"
-#include "qemu/config-file.h"
-#include "qemu/error-report.h"
-#include "qemu/option.h"
-#include "qemu/units.h"
 #include "target/i386/cpu.h"
 #include "exec/address-spaces.h"
 
@@ -56,6 +51,8 @@ static void sgx_epc_realize(DeviceState *dev, Error **errp)
 {
     PCMachineState *pcms = PC_MACHINE(qdev_get_machine());
     X86MachineState *x86ms = X86_MACHINE(pcms);
+    MemoryDeviceState *md = MEMORY_DEVICE(dev);
+    SGXEPCState *sgx_epc = &pcms->sgx_epc;
     SGXEPCDevice *epc = SGX_EPC(dev);
     const char *path;
 
@@ -74,7 +71,18 @@ static void sgx_epc_realize(DeviceState *dev, Error **errp)
         return;
     }
 
-    error_setg(errp, "'" TYPE_SGX_EPC "' not supported");
+    epc->addr = sgx_epc->base + sgx_epc->size;
+
+    memory_region_add_subregion(&sgx_epc->mr, epc->addr - sgx_epc->base,
+                                host_memory_backend_get_memory(epc->hostmem));
+
+    host_memory_backend_set_mapped(epc->hostmem, true);
+
+    sgx_epc->sections = g_renew(SGXEPCDevice *, sgx_epc->sections,
+                                sgx_epc->nr_sections + 1);
+    sgx_epc->sections[sgx_epc->nr_sections++] = epc;
+
+    sgx_epc->size += memory_device_get_region_size(md, errp);
 }
 
 static void sgx_epc_unrealize(DeviceState *dev)
diff --git a/hw/i386/sgx.c b/hw/i386/sgx.c
new file mode 100644
index 0000000000..e77deb0b00
--- /dev/null
+++ b/hw/i386/sgx.c
@@ -0,0 +1,67 @@
+/*
+ * SGX common code
+ *
+ * Copyright (C) 2021 Intel Corporation
+ *
+ * Authors:
+ *   Yang Zhong<yang.zhong@intel.com>
+ *   Sean Christopherson <sean.j.christopherson@intel.com>
+ *
+ * This work is licensed under the terms of the GNU GPL, version 2 or later.
+ * See the COPYING file in the top-level directory.
+ */
+#include "qemu/osdep.h"
+#include "hw/i386/pc.h"
+#include "hw/i386/sgx-epc.h"
+#include "hw/mem/memory-device.h"
+#include "monitor/qdev.h"
+#include "qapi/error.h"
+#include "exec/address-spaces.h"
+
+static int sgx_epc_set_property(void *opaque, const char *name,
+                                const char *value, Error **errp)
+{
+    Object *obj = opaque;
+    Error *err = NULL;
+
+    object_property_parse(obj, name, value, &err);
+    if (err != NULL) {
+        error_propagate(errp, err);
+        return -1;
+    }
+    return 0;
+}
+
+void pc_machine_init_sgx_epc(PCMachineState *pcms)
+{
+    SGXEPCState *sgx_epc = &pcms->sgx_epc;
+    X86MachineState *x86ms = X86_MACHINE(pcms);
+    Error *err = NULL;
+    SgxEPCList *list = NULL;
+    Object *obj;
+
+    memset(sgx_epc, 0, sizeof(SGXEPCState));
+    sgx_epc->base = 0x100000000ULL + x86ms->above_4g_mem_size;
+
+    memory_region_init(&sgx_epc->mr, OBJECT(pcms), "sgx-epc", UINT64_MAX);
+    memory_region_add_subregion(get_system_memory(), sgx_epc->base,
+                                &sgx_epc->mr);
+
+    for (list = x86ms->sgx_epc_list; list; list = list->next) {
+        obj = object_new("sgx-epc");
+
+        /* set the memdev link with memory backend */
+        sgx_epc_set_property(obj, SGX_EPC_MEMDEV_PROP, list->value->memdev,
+                             &err);
+        object_property_set_bool(obj, "realized", true, &err);
+        object_unref(obj);
+    }
+
+    if ((sgx_epc->base + sgx_epc->size) < sgx_epc->base) {
+        error_report("Size of all 'sgx-epc' =0x%"PRIu64" causes EPC to wrap",
+                     sgx_epc->size);
+        exit(EXIT_FAILURE);
+    }
+
+    memory_region_set_size(&sgx_epc->mr, sgx_epc->size);
+}
diff --git a/hw/i386/x86.c b/hw/i386/x86.c
index 00448ed55a..41ef9a84a9 100644
--- a/hw/i386/x86.c
+++ b/hw/i386/x86.c
@@ -30,6 +30,8 @@
 #include "qapi/error.h"
 #include "qapi/qmp/qerror.h"
 #include "qapi/qapi-visit-common.h"
+#include "qapi/clone-visitor.h"
+#include "qapi/qapi-visit-machine.h"
 #include "qapi/visitor.h"
 #include "sysemu/qtest.h"
 #include "sysemu/whpx.h"
@@ -1263,6 +1265,27 @@ static void x86_machine_set_bus_lock_ratelimit(Object *obj, Visitor *v,
     visit_type_uint64(v, name, &x86ms->bus_lock_ratelimit, errp);
 }
 
+static void machine_get_sgx_epc(Object *obj, Visitor *v, const char *name,
+                                void *opaque, Error **errp)
+{
+    X86MachineState *x86ms = X86_MACHINE(obj);
+    SgxEPCList *list = x86ms->sgx_epc_list;
+
+    visit_type_SgxEPCList(v, name, &list, errp);
+}
+
+static void machine_set_sgx_epc(Object *obj, Visitor *v, const char *name,
+                                void *opaque, Error **errp)
+{
+    X86MachineState *x86ms = X86_MACHINE(obj);
+    SgxEPCList *list;
+
+    list = x86ms->sgx_epc_list;
+    visit_type_SgxEPCList(v, name, &x86ms->sgx_epc_list, errp);
+
+    qapi_free_SgxEPCList(list);
+}
+
 static void x86_machine_initfn(Object *obj)
 {
     X86MachineState *x86ms = X86_MACHINE(obj);
@@ -1322,6 +1345,12 @@ static void x86_machine_class_init(ObjectClass *oc, void *data)
                                 x86_machine_set_bus_lock_ratelimit, NULL, NULL);
     object_class_property_set_description(oc, X86_MACHINE_BUS_LOCK_RATELIMIT,
             "Set the ratelimit for the bus locks acquired in VMs");
+
+    object_class_property_add(oc, "sgx-epc", "SgxEPC",
+        machine_get_sgx_epc, machine_set_sgx_epc,
+        NULL, NULL);
+    object_class_property_set_description(oc, "sgx-epc",
+        "SGX EPC device");
 }
 
 static const TypeInfo x86_machine_info = {
diff --git a/include/hw/i386/pc.h b/include/hw/i386/pc.h
index 88dffe7517..2fd6e6193b 100644
--- a/include/hw/i386/pc.h
+++ b/include/hw/i386/pc.h
@@ -12,6 +12,7 @@
 #include "hw/acpi/acpi_dev_interface.h"
 #include "hw/hotplug.h"
 #include "qom/object.h"
+#include "hw/i386/sgx-epc.h"
 
 #define HPET_INTCAP "hpet-intcap"
 
@@ -53,6 +54,8 @@ typedef struct PCMachineState {
 
     /* ACPI Memory hotplug IO base address */
     hwaddr memhp_io_base;
+
+    SGXEPCState sgx_epc;
 } PCMachineState;
 
 #define PC_MACHINE_ACPI_DEVICE_PROP "acpi-device"
@@ -199,6 +202,9 @@ void pc_madt_cpu_entry(AcpiDeviceIf *adev, int uid,
 extern GlobalProperty pc_compat_6_0[];
 extern const size_t pc_compat_6_0_len;
 
+/* sgx-epc.c */
+void pc_machine_init_sgx_epc(PCMachineState *pcms);
+
 extern GlobalProperty pc_compat_5_2[];
 extern const size_t pc_compat_5_2_len;
 
diff --git a/include/hw/i386/sgx-epc.h b/include/hw/i386/sgx-epc.h
index 5fd9ae2d0c..2b2490892b 100644
--- a/include/hw/i386/sgx-epc.h
+++ b/include/hw/i386/sgx-epc.h
@@ -41,4 +41,18 @@ typedef struct SGXEPCDevice {
     HostMemoryBackend *hostmem;
 } SGXEPCDevice;
 
+/*
+ * @base: address in guest physical address space where EPC regions start
+ * @mr: address space container for memory devices
+ */
+typedef struct SGXEPCState {
+    uint64_t base;
+    uint64_t size;
+
+    MemoryRegion mr;
+
+    struct SGXEPCDevice **sections;
+    int nr_sections;
+} SGXEPCState;
+
 #endif
diff --git a/include/hw/i386/x86.h b/include/hw/i386/x86.h
index 6e9244a82c..23267a3674 100644
--- a/include/hw/i386/x86.h
+++ b/include/hw/i386/x86.h
@@ -62,6 +62,7 @@ struct X86MachineState {
     unsigned pci_irq_mask;
     unsigned apic_id_limit;
     uint16_t boot_cpus;
+    SgxEPCList *sgx_epc_list;
 
     OnOffAuto smm;
     OnOffAuto acpi;
diff --git a/qapi/machine.json b/qapi/machine.json
index c3210ee1fb..3bbe5b590e 100644
--- a/qapi/machine.json
+++ b/qapi/machine.json
@@ -1194,6 +1194,32 @@
           }
 }
 
+##
+# @SgxEPC:
+#
+# Sgx EPC cmdline information
+#
+# @memdev: memory backend linked with device
+#
+# Since: 6.1
+##
+{ 'struct': 'SgxEPC',
+  'data': { 'memdev': 'str' }
+}
+
+##
+# @SgxEPCProperties:
+#
+# Properties for SgxEPC objects.
+#
+# @sgx-epc: sgx epc section properties.
+#
+# Since: 6.1
+##
+{ 'struct': 'SgxEPCProperties',
+  'data': { 'sgx-epc': ['SgxEPC'] }
+}
+
 ##
 # @MemoryDeviceInfo:
 #
diff --git a/qemu-options.hx b/qemu-options.hx
index 0c9ddc0274..dd05426f58 100644
--- a/qemu-options.hx
+++ b/qemu-options.hx
@@ -122,8 +122,14 @@ SRST
         -m 512M
 ERST
 
-HXCOMM Deprecated by -machine
-DEF("M", HAS_ARG, QEMU_OPTION_M, "", QEMU_ARCH_ALL)
+DEF("M", HAS_ARG, QEMU_OPTION_M,
+    "                sgx-epc.0.memdev=memid\n",
+    QEMU_ARCH_ALL)
+
+SRST
+``sgx-epc.0.memdev=@var{memid}``
+    Define an SGX EPC section.
+ERST
 
 DEF("cpu", HAS_ARG, QEMU_OPTION_cpu,
     "-cpu cpu        select CPU ('-cpu help' for list)\n", QEMU_ARCH_ALL)


^ permalink raw reply related	[flat|nested] 61+ messages in thread

* [PATCH v4 06/33] i386: Add primary SGX CPUID and MSR defines
  2021-07-19 11:21 [PATCH v4 00/33] Qemu SGX virtualization Yang Zhong
                   ` (4 preceding siblings ...)
  2021-07-19 11:21 ` [PATCH v4 05/33] vl: Add sgx compound properties to expose SGX " Yang Zhong
@ 2021-07-19 11:21 ` Yang Zhong
  2021-07-19 11:21 ` [PATCH v4 07/33] i386: Add SGX CPUID leaf FEAT_SGX_12_0_EAX Yang Zhong
                   ` (29 subsequent siblings)
  35 siblings, 0 replies; 61+ messages in thread
From: Yang Zhong @ 2021-07-19 11:21 UTC (permalink / raw)
  To: qemu-devel; +Cc: yang.zhong, seanjc, kai.huang, jarkko, pbonzini, eblake

From: Sean Christopherson <sean.j.christopherson@intel.com>

Add CPUID defines for SGX and SGX Launch Control (LC), as well as
defines for their associated FEATURE_CONTROL MSR bits.  Define the
Launch Enclave Public Key Hash MSRs (LE Hash MSRs), which exist
when SGX LC is present (in CPUID), and are writable when SGX LC is
enabled (in FEATURE_CONTROL).

Signed-off-by: Sean Christopherson <sean.j.christopherson@intel.com>
Signed-off-by: Yang Zhong <yang.zhong@intel.com>
---
 target/i386/cpu.c |  4 ++--
 target/i386/cpu.h | 12 ++++++++++++
 2 files changed, 14 insertions(+), 2 deletions(-)

diff --git a/target/i386/cpu.c b/target/i386/cpu.c
index 48b55ebd0a..7796d1caa6 100644
--- a/target/i386/cpu.c
+++ b/target/i386/cpu.c
@@ -794,7 +794,7 @@ FeatureWordInfo feature_word_info[FEATURE_WORDS] = {
     [FEAT_7_0_EBX] = {
         .type = CPUID_FEATURE_WORD,
         .feat_names = {
-            "fsgsbase", "tsc-adjust", NULL, "bmi1",
+            "fsgsbase", "tsc-adjust", "sgx", "bmi1",
             "hle", "avx2", NULL, "smep",
             "bmi2", "erms", "invpcid", "rtm",
             NULL, NULL, "mpx", NULL,
@@ -820,7 +820,7 @@ FeatureWordInfo feature_word_info[FEATURE_WORDS] = {
             "la57", NULL, NULL, NULL,
             NULL, NULL, "rdpid", NULL,
             "bus-lock-detect", "cldemote", NULL, "movdiri",
-            "movdir64b", NULL, NULL, "pks",
+            "movdir64b", NULL, "sgxlc", "pks",
         },
         .cpuid = {
             .eax = 7,
diff --git a/target/i386/cpu.h b/target/i386/cpu.h
index 5d98a4e7c0..af33f956fe 100644
--- a/target/i386/cpu.h
+++ b/target/i386/cpu.h
@@ -377,9 +377,17 @@ typedef enum X86Seg {
 #define MSR_IA32_PKRS                   0x6e1
 
 #define FEATURE_CONTROL_LOCKED                    (1<<0)
+#define FEATURE_CONTROL_VMXON_ENABLED_INSIDE_SMX  (1ULL << 1)
 #define FEATURE_CONTROL_VMXON_ENABLED_OUTSIDE_SMX (1<<2)
+#define FEATURE_CONTROL_SGX_LC                    (1ULL << 17)
+#define FEATURE_CONTROL_SGX                       (1ULL << 18)
 #define FEATURE_CONTROL_LMCE                      (1<<20)
 
+#define MSR_IA32_SGXLEPUBKEYHASH0       0x8c
+#define MSR_IA32_SGXLEPUBKEYHASH1       0x8d
+#define MSR_IA32_SGXLEPUBKEYHASH2       0x8e
+#define MSR_IA32_SGXLEPUBKEYHASH3       0x8f
+
 #define MSR_P6_PERFCTR0                 0xc1
 
 #define MSR_IA32_SMBASE                 0x9e
@@ -701,6 +709,8 @@ typedef uint64_t FeatureWordArray[FEATURE_WORDS];
 
 /* Support RDFSBASE/RDGSBASE/WRFSBASE/WRGSBASE */
 #define CPUID_7_0_EBX_FSGSBASE          (1U << 0)
+/* Support SGX */
+#define CPUID_7_0_EBX_SGX               (1U << 2)
 /* 1st Group of Advanced Bit Manipulation Extensions */
 #define CPUID_7_0_EBX_BMI1              (1U << 3)
 /* Hardware Lock Elision */
@@ -788,6 +798,8 @@ typedef uint64_t FeatureWordArray[FEATURE_WORDS];
 #define CPUID_7_0_ECX_MOVDIRI           (1U << 27)
 /* Move 64 Bytes as Direct Store Instruction */
 #define CPUID_7_0_ECX_MOVDIR64B         (1U << 28)
+/* Support SGX Launch Control */
+#define CPUID_7_0_ECX_SGX_LC            (1U << 30)
 /* Protection Keys for Supervisor-mode Pages */
 #define CPUID_7_0_ECX_PKS               (1U << 31)
 


^ permalink raw reply related	[flat|nested] 61+ messages in thread

* [PATCH v4 07/33] i386: Add SGX CPUID leaf FEAT_SGX_12_0_EAX
  2021-07-19 11:21 [PATCH v4 00/33] Qemu SGX virtualization Yang Zhong
                   ` (5 preceding siblings ...)
  2021-07-19 11:21 ` [PATCH v4 06/33] i386: Add primary SGX CPUID and MSR defines Yang Zhong
@ 2021-07-19 11:21 ` Yang Zhong
  2021-07-19 11:21 ` [PATCH v4 08/33] i386: Add SGX CPUID leaf FEAT_SGX_12_0_EBX Yang Zhong
                   ` (28 subsequent siblings)
  35 siblings, 0 replies; 61+ messages in thread
From: Yang Zhong @ 2021-07-19 11:21 UTC (permalink / raw)
  To: qemu-devel; +Cc: yang.zhong, seanjc, kai.huang, jarkko, pbonzini, eblake

From: Sean Christopherson <sean.j.christopherson@intel.com>

CPUID leaf 12_0_EAX is an Intel-defined feature bits leaf enumerating
the CPU's SGX capabilities, e.g. supported SGX instruction sets.
Currently there are four enumerated capabilities:

    - SGX1 instruction set, i.e. "base" SGX
    - SGX2 instruction set for dynamic EPC management
    - ENCLV instruction set for VMM oversubscription of EPC
    - ENCLS-C instruction set for thread safe variants of ENCLS

Signed-off-by: Sean Christopherson <sean.j.christopherson@intel.com>
Signed-off-by: Yang Zhong <yang.zhong@intel.com>
---
 target/i386/cpu.c | 20 ++++++++++++++++++++
 target/i386/cpu.h |  1 +
 2 files changed, 21 insertions(+)

diff --git a/target/i386/cpu.c b/target/i386/cpu.c
index 7796d1caa6..13289e0547 100644
--- a/target/i386/cpu.c
+++ b/target/i386/cpu.c
@@ -653,6 +653,7 @@ void x86_cpu_vendor_words2str(char *dst, uint32_t vendor1,
           /* missing:
           CPUID_XSAVE_XSAVEC, CPUID_XSAVE_XSAVES */
 #define TCG_14_0_ECX_FEATURES 0
+#define TCG_SGX_12_0_EAX_FEATURES 0
 
 FeatureWordInfo feature_word_info[FEATURE_WORDS] = {
     [FEAT_1_EDX] = {
@@ -1181,6 +1182,25 @@ FeatureWordInfo feature_word_info[FEATURE_WORDS] = {
         .tcg_features = TCG_14_0_ECX_FEATURES,
      },
 
+    [FEAT_SGX_12_0_EAX] = {
+        .type = CPUID_FEATURE_WORD,
+        .feat_names = {
+            "sgx1", "sgx2", NULL, NULL,
+            NULL, NULL, NULL, NULL,
+            NULL, NULL, NULL, NULL,
+            NULL, NULL, NULL, NULL,
+            NULL, NULL, NULL, NULL,
+            NULL, NULL, NULL, NULL,
+            NULL, NULL, NULL, NULL,
+            NULL, NULL, NULL, NULL,
+        },
+        .cpuid = {
+            .eax = 0x12,
+            .needs_ecx = true, .ecx = 0,
+            .reg = R_EAX,
+        },
+        .tcg_features = TCG_SGX_12_0_EAX_FEATURES,
+    },
 };
 
 typedef struct FeatureMask {
diff --git a/target/i386/cpu.h b/target/i386/cpu.h
index af33f956fe..21830b2d79 100644
--- a/target/i386/cpu.h
+++ b/target/i386/cpu.h
@@ -561,6 +561,7 @@ typedef enum FeatureWord {
     FEAT_VMX_BASIC,
     FEAT_VMX_VMFUNC,
     FEAT_14_0_ECX,
+    FEAT_SGX_12_0_EAX,  /* CPUID[EAX=0x12,ECX=0].EAX (SGX) */
     FEATURE_WORDS,
 } FeatureWord;
 


^ permalink raw reply related	[flat|nested] 61+ messages in thread

* [PATCH v4 08/33] i386: Add SGX CPUID leaf FEAT_SGX_12_0_EBX
  2021-07-19 11:21 [PATCH v4 00/33] Qemu SGX virtualization Yang Zhong
                   ` (6 preceding siblings ...)
  2021-07-19 11:21 ` [PATCH v4 07/33] i386: Add SGX CPUID leaf FEAT_SGX_12_0_EAX Yang Zhong
@ 2021-07-19 11:21 ` Yang Zhong
  2021-07-19 11:21 ` [PATCH v4 09/33] i386: Add SGX CPUID leaf FEAT_SGX_12_1_EAX Yang Zhong
                   ` (27 subsequent siblings)
  35 siblings, 0 replies; 61+ messages in thread
From: Yang Zhong @ 2021-07-19 11:21 UTC (permalink / raw)
  To: qemu-devel; +Cc: yang.zhong, seanjc, kai.huang, jarkko, pbonzini, eblake

From: Sean Christopherson <sean.j.christopherson@intel.com>

CPUID leaf 12_0_EBX is an Intel-defined feature bits leaf enumerating
the platform's SGX extended capabilities.  Currently there is a single
capabilitiy:

   - EXINFO: record information about #PFs and #GPs in the enclave's SSA

Signed-off-by: Sean Christopherson <sean.j.christopherson@intel.com>
Signed-off-by: Yang Zhong <yang.zhong@intel.com>
---
 target/i386/cpu.c | 21 +++++++++++++++++++++
 target/i386/cpu.h |  1 +
 2 files changed, 22 insertions(+)

diff --git a/target/i386/cpu.c b/target/i386/cpu.c
index 13289e0547..0525a4cf7f 100644
--- a/target/i386/cpu.c
+++ b/target/i386/cpu.c
@@ -654,6 +654,7 @@ void x86_cpu_vendor_words2str(char *dst, uint32_t vendor1,
           CPUID_XSAVE_XSAVEC, CPUID_XSAVE_XSAVES */
 #define TCG_14_0_ECX_FEATURES 0
 #define TCG_SGX_12_0_EAX_FEATURES 0
+#define TCG_SGX_12_0_EBX_FEATURES 0
 
 FeatureWordInfo feature_word_info[FEATURE_WORDS] = {
     [FEAT_1_EDX] = {
@@ -1201,6 +1202,26 @@ FeatureWordInfo feature_word_info[FEATURE_WORDS] = {
         },
         .tcg_features = TCG_SGX_12_0_EAX_FEATURES,
     },
+
+    [FEAT_SGX_12_0_EBX] = {
+        .type = CPUID_FEATURE_WORD,
+        .feat_names = {
+            "sgx-exinfo" , NULL, NULL, NULL,
+            NULL, NULL, NULL, NULL,
+            NULL, NULL, NULL, NULL,
+            NULL, NULL, NULL, NULL,
+            NULL, NULL, NULL, NULL,
+            NULL, NULL, NULL, NULL,
+            NULL, NULL, NULL, NULL,
+            NULL, NULL, NULL, NULL,
+        },
+        .cpuid = {
+            .eax = 0x12,
+            .needs_ecx = true, .ecx = 0,
+            .reg = R_EBX,
+        },
+        .tcg_features = TCG_SGX_12_0_EBX_FEATURES,
+    },
 };
 
 typedef struct FeatureMask {
diff --git a/target/i386/cpu.h b/target/i386/cpu.h
index 21830b2d79..a71d1c8fd0 100644
--- a/target/i386/cpu.h
+++ b/target/i386/cpu.h
@@ -562,6 +562,7 @@ typedef enum FeatureWord {
     FEAT_VMX_VMFUNC,
     FEAT_14_0_ECX,
     FEAT_SGX_12_0_EAX,  /* CPUID[EAX=0x12,ECX=0].EAX (SGX) */
+    FEAT_SGX_12_0_EBX,  /* CPUID[EAX=0x12,ECX=0].EBX (SGX MISCSELECT[31:0]) */
     FEATURE_WORDS,
 } FeatureWord;
 


^ permalink raw reply related	[flat|nested] 61+ messages in thread

* [PATCH v4 09/33] i386: Add SGX CPUID leaf FEAT_SGX_12_1_EAX
  2021-07-19 11:21 [PATCH v4 00/33] Qemu SGX virtualization Yang Zhong
                   ` (7 preceding siblings ...)
  2021-07-19 11:21 ` [PATCH v4 08/33] i386: Add SGX CPUID leaf FEAT_SGX_12_0_EBX Yang Zhong
@ 2021-07-19 11:21 ` Yang Zhong
  2021-07-19 11:21 ` [PATCH v4 10/33] i386: Add get/set/migrate support for SGX_LEPUBKEYHASH MSRs Yang Zhong
                   ` (26 subsequent siblings)
  35 siblings, 0 replies; 61+ messages in thread
From: Yang Zhong @ 2021-07-19 11:21 UTC (permalink / raw)
  To: qemu-devel; +Cc: yang.zhong, seanjc, kai.huang, jarkko, pbonzini, eblake

From: Sean Christopherson <sean.j.christopherson@intel.com>

CPUID leaf 12_1_EAX is an Intel-defined feature bits leaf enumerating
the platform's SGX capabilities that may be utilized by an enclave, e.g.
whether or not an enclave can gain access to the provision key.
Currently there are six capabilities:

   - INIT: set when the enclave has has been initialized by EINIT.  Cannot
           be set by software, i.e. forced to zero in CPUID.
   - DEBUG: permits a debugger to read/write into the enclave.
   - MODE64BIT: the enclave runs in 64-bit mode
   - PROVISIONKEY: grants has access to the provision key
   - EINITTOKENKEY: grants access to the EINIT token key, i.e. the
                    enclave can generate EINIT tokens
   - KSS: Key Separation and Sharing enabled for the enclave.

Note that the entirety of CPUID.0x12.0x1, i.e. all registers, enumerates
the allowed ATTRIBUTES (128 bits), but only bits 31:0 are directly
exposed to the user (via FEAT_12_1_EAX).  Bits 63:32 are currently all
reserved and bits 127:64 correspond to the allowed XSAVE Feature Request
Mask, which is calculated based on other CPU features, e.g. XSAVE, MPX,
AVX, etc... and is not exposed to the user.

Signed-off-by: Sean Christopherson <sean.j.christopherson@intel.com>
Signed-off-by: Yang Zhong <yang.zhong@intel.com>
---
 target/i386/cpu.c | 21 +++++++++++++++++++++
 target/i386/cpu.h |  1 +
 2 files changed, 22 insertions(+)

diff --git a/target/i386/cpu.c b/target/i386/cpu.c
index 0525a4cf7f..840f825431 100644
--- a/target/i386/cpu.c
+++ b/target/i386/cpu.c
@@ -655,6 +655,7 @@ void x86_cpu_vendor_words2str(char *dst, uint32_t vendor1,
 #define TCG_14_0_ECX_FEATURES 0
 #define TCG_SGX_12_0_EAX_FEATURES 0
 #define TCG_SGX_12_0_EBX_FEATURES 0
+#define TCG_SGX_12_1_EAX_FEATURES 0
 
 FeatureWordInfo feature_word_info[FEATURE_WORDS] = {
     [FEAT_1_EDX] = {
@@ -1222,6 +1223,26 @@ FeatureWordInfo feature_word_info[FEATURE_WORDS] = {
         },
         .tcg_features = TCG_SGX_12_0_EBX_FEATURES,
     },
+
+    [FEAT_SGX_12_1_EAX] = {
+        .type = CPUID_FEATURE_WORD,
+        .feat_names = {
+            NULL, "sgx-debug", "sgx-mode64", NULL,
+            "sgx-provisionkey", "sgx-tokenkey", NULL, "sgx-kss",
+            NULL, NULL, NULL, NULL,
+            NULL, NULL, NULL, NULL,
+            NULL, NULL, NULL, NULL,
+            NULL, NULL, NULL, NULL,
+            NULL, NULL, NULL, NULL,
+            NULL, NULL, NULL, NULL,
+        },
+        .cpuid = {
+            .eax = 0x12,
+            .needs_ecx = true, .ecx = 1,
+            .reg = R_EAX,
+        },
+        .tcg_features = TCG_SGX_12_1_EAX_FEATURES,
+    },
 };
 
 typedef struct FeatureMask {
diff --git a/target/i386/cpu.h b/target/i386/cpu.h
index a71d1c8fd0..e4d46cca80 100644
--- a/target/i386/cpu.h
+++ b/target/i386/cpu.h
@@ -563,6 +563,7 @@ typedef enum FeatureWord {
     FEAT_14_0_ECX,
     FEAT_SGX_12_0_EAX,  /* CPUID[EAX=0x12,ECX=0].EAX (SGX) */
     FEAT_SGX_12_0_EBX,  /* CPUID[EAX=0x12,ECX=0].EBX (SGX MISCSELECT[31:0]) */
+    FEAT_SGX_12_1_EAX,  /* CPUID[EAX=0x12,ECX=1].EAX (SGX ATTRIBUTES[31:0]) */
     FEATURE_WORDS,
 } FeatureWord;
 


^ permalink raw reply related	[flat|nested] 61+ messages in thread

* [PATCH v4 10/33] i386: Add get/set/migrate support for SGX_LEPUBKEYHASH MSRs
  2021-07-19 11:21 [PATCH v4 00/33] Qemu SGX virtualization Yang Zhong
                   ` (8 preceding siblings ...)
  2021-07-19 11:21 ` [PATCH v4 09/33] i386: Add SGX CPUID leaf FEAT_SGX_12_1_EAX Yang Zhong
@ 2021-07-19 11:21 ` Yang Zhong
  2021-09-14  6:38   ` Philippe Mathieu-Daudé
  2021-07-19 11:21 ` [PATCH v4 11/33] i386: Add feature control MSR dependency when SGX is enabled Yang Zhong
                   ` (25 subsequent siblings)
  35 siblings, 1 reply; 61+ messages in thread
From: Yang Zhong @ 2021-07-19 11:21 UTC (permalink / raw)
  To: qemu-devel; +Cc: yang.zhong, seanjc, kai.huang, jarkko, pbonzini, eblake

From: Sean Christopherson <sean.j.christopherson@intel.com>

On real hardware, on systems that supports SGX Launch Control, those
MSRs are initialized to digest of Intel's signing key; on systems that
don't support SGX Launch Control, those MSRs are not available but
hardware always uses digest of Intel's signing key in EINIT.

KVM advertises SGX LC via CPUID if and only if the MSRs are writable.
Unconditionally initialize those MSRs to digest of Intel's signing key
when CPU is realized and reset to reflect the fact. This avoids
potential bug in case kvm_arch_put_registers() is called before
kvm_arch_get_registers() is called, in which case guest's virtual
SGX_LEPUBKEYHASH MSRs will be set to 0, although KVM initializes those
to digest of Intel's signing key by default, since KVM allows those MSRs
to be updated by Qemu to support live migration.

Save/restore the SGX Launch Enclave Public Key Hash MSRs if SGX Launch
Control (LC) is exposed to the guest. Likewise, migrate the MSRs if they
are writable by the guest.

Signed-off-by: Sean Christopherson <sean.j.christopherson@intel.com>
Signed-off-by: Kai Huang <kai.huang@intel.com>
Signed-off-by: Yang Zhong <yang.zhong@intel.com>
---
 target/i386/cpu.c     | 17 ++++++++++++++++-
 target/i386/cpu.h     |  1 +
 target/i386/kvm/kvm.c | 22 ++++++++++++++++++++++
 target/i386/machine.c | 20 ++++++++++++++++++++
 4 files changed, 59 insertions(+), 1 deletion(-)

diff --git a/target/i386/cpu.c b/target/i386/cpu.c
index 840f825431..cea4307930 100644
--- a/target/i386/cpu.c
+++ b/target/i386/cpu.c
@@ -5673,6 +5673,16 @@ void cpu_x86_cpuid(CPUX86State *env, uint32_t index, uint32_t count,
     }
 }
 
+#ifndef CONFIG_USER_ONLY
+static void x86_cpu_set_sgxlepubkeyhash(CPUX86State *env)
+{
+    env->msr_ia32_sgxlepubkeyhash[0] = 0xa6053e051270b7acULL;
+    env->msr_ia32_sgxlepubkeyhash[1] = 0x6cfbe8ba8b3b413dULL;
+    env->msr_ia32_sgxlepubkeyhash[2] = 0xc4916d99f2b3735dULL;
+    env->msr_ia32_sgxlepubkeyhash[3] = 0xd4f8c05909f9bb3bULL;
+}
+#endif
+
 static void x86_cpu_reset(DeviceState *dev)
 {
     CPUState *s = CPU(dev);
@@ -5804,6 +5814,8 @@ static void x86_cpu_reset(DeviceState *dev)
     if (kvm_enabled()) {
         kvm_arch_reset_vcpu(cpu);
     }
+
+    x86_cpu_set_sgxlepubkeyhash(env);
 #endif
 }
 
@@ -6186,6 +6198,10 @@ static void x86_cpu_realizefn(DeviceState *dev, Error **errp)
            & CPUID_EXT2_AMD_ALIASES);
     }
 
+#ifndef CONFIG_USER_ONLY
+    x86_cpu_set_sgxlepubkeyhash(env);
+#endif
+
     /*
      * note: the call to the framework needs to happen after feature expansion,
      * but before the checks/modifications to ucode_rev, mwait, phys_bits.
@@ -6871,7 +6887,6 @@ static const TypeInfo x86_cpu_type_info = {
     .class_init = x86_cpu_common_class_init,
 };
 
-
 /* "base" CPU model, used by query-cpu-model-expansion */
 static void x86_cpu_base_class_init(ObjectClass *oc, void *data)
 {
diff --git a/target/i386/cpu.h b/target/i386/cpu.h
index e4d46cca80..892c0dfab4 100644
--- a/target/i386/cpu.h
+++ b/target/i386/cpu.h
@@ -1499,6 +1499,7 @@ typedef struct CPUX86State {
     uint64_t mcg_status;
     uint64_t msr_ia32_misc_enable;
     uint64_t msr_ia32_feature_control;
+    uint64_t msr_ia32_sgxlepubkeyhash[4];
 
     uint64_t msr_fixed_ctr_ctrl;
     uint64_t msr_global_ctrl;
diff --git a/target/i386/kvm/kvm.c b/target/i386/kvm/kvm.c
index 59ed8327ac..d4bf054ebe 100644
--- a/target/i386/kvm/kvm.c
+++ b/target/i386/kvm/kvm.c
@@ -3096,6 +3096,17 @@ static int kvm_put_msrs(X86CPU *cpu, int level)
             }
         }
 
+        if (env->features[FEAT_7_0_ECX] & CPUID_7_0_ECX_SGX_LC) {
+            kvm_msr_entry_add(cpu, MSR_IA32_SGXLEPUBKEYHASH0,
+                              env->msr_ia32_sgxlepubkeyhash[0]);
+            kvm_msr_entry_add(cpu, MSR_IA32_SGXLEPUBKEYHASH1,
+                              env->msr_ia32_sgxlepubkeyhash[1]);
+            kvm_msr_entry_add(cpu, MSR_IA32_SGXLEPUBKEYHASH2,
+                              env->msr_ia32_sgxlepubkeyhash[2]);
+            kvm_msr_entry_add(cpu, MSR_IA32_SGXLEPUBKEYHASH3,
+                              env->msr_ia32_sgxlepubkeyhash[3]);
+        }
+
         /* Note: MSR_IA32_FEATURE_CONTROL is written separately, see
          *       kvm_put_msr_feature_control. */
     }
@@ -3435,6 +3446,13 @@ static int kvm_get_msrs(X86CPU *cpu)
         }
     }
 
+    if (env->features[FEAT_7_0_ECX] & CPUID_7_0_ECX_SGX_LC) {
+        kvm_msr_entry_add(cpu, MSR_IA32_SGXLEPUBKEYHASH0, 0);
+        kvm_msr_entry_add(cpu, MSR_IA32_SGXLEPUBKEYHASH1, 0);
+        kvm_msr_entry_add(cpu, MSR_IA32_SGXLEPUBKEYHASH2, 0);
+        kvm_msr_entry_add(cpu, MSR_IA32_SGXLEPUBKEYHASH3, 0);
+    }
+
     ret = kvm_vcpu_ioctl(CPU(cpu), KVM_GET_MSRS, cpu->kvm_msr_buf);
     if (ret < 0) {
         return ret;
@@ -3724,6 +3742,10 @@ static int kvm_get_msrs(X86CPU *cpu)
         case MSR_IA32_RTIT_ADDR0_A ... MSR_IA32_RTIT_ADDR3_B:
             env->msr_rtit_addrs[index - MSR_IA32_RTIT_ADDR0_A] = msrs[i].data;
             break;
+        case MSR_IA32_SGXLEPUBKEYHASH0 ... MSR_IA32_SGXLEPUBKEYHASH3:
+            env->msr_ia32_sgxlepubkeyhash[index - MSR_IA32_SGXLEPUBKEYHASH0] =
+                           msrs[i].data;
+            break;
         }
     }
 
diff --git a/target/i386/machine.c b/target/i386/machine.c
index f6f094f1c9..099a4c36f7 100644
--- a/target/i386/machine.c
+++ b/target/i386/machine.c
@@ -1396,6 +1396,25 @@ static const VMStateDescription vmstate_msr_tsx_ctrl = {
     }
 };
 
+static bool intel_sgx_msrs_needed(void *opaque)
+{
+    X86CPU *cpu = opaque;
+    CPUX86State *env = &cpu->env;
+
+    return !!(env->features[FEAT_7_0_ECX] & CPUID_7_0_ECX_SGX_LC);
+}
+
+static const VMStateDescription vmstate_msr_intel_sgx = {
+    .name = "cpu/intel_sgx",
+    .version_id = 1,
+    .minimum_version_id = 1,
+    .needed = intel_sgx_msrs_needed,
+    .fields = (VMStateField[]) {
+        VMSTATE_UINT64_ARRAY(env.msr_ia32_sgxlepubkeyhash, X86CPU, 4),
+        VMSTATE_END_OF_LIST()
+    }
+};
+
 const VMStateDescription vmstate_x86_cpu = {
     .name = "cpu",
     .version_id = 12,
@@ -1531,6 +1550,7 @@ const VMStateDescription vmstate_x86_cpu = {
         &vmstate_nested_state,
 #endif
         &vmstate_msr_tsx_ctrl,
+        &vmstate_msr_intel_sgx,
         NULL
     }
 };


^ permalink raw reply related	[flat|nested] 61+ messages in thread

* [PATCH v4 11/33] i386: Add feature control MSR dependency when SGX is enabled
  2021-07-19 11:21 [PATCH v4 00/33] Qemu SGX virtualization Yang Zhong
                   ` (9 preceding siblings ...)
  2021-07-19 11:21 ` [PATCH v4 10/33] i386: Add get/set/migrate support for SGX_LEPUBKEYHASH MSRs Yang Zhong
@ 2021-07-19 11:21 ` Yang Zhong
  2021-07-19 11:21 ` [PATCH v4 12/33] i386: Update SGX CPUID info according to hardware/KVM/user input Yang Zhong
                   ` (24 subsequent siblings)
  35 siblings, 0 replies; 61+ messages in thread
From: Yang Zhong @ 2021-07-19 11:21 UTC (permalink / raw)
  To: qemu-devel; +Cc: yang.zhong, seanjc, kai.huang, jarkko, pbonzini, eblake

From: Sean Christopherson <sean.j.christopherson@intel.com>

SGX adds multiple flags to FEATURE_CONTROL to enable SGX and Flexible
Launch Control.

Signed-off-by: Sean Christopherson <sean.j.christopherson@intel.com>
Signed-off-by: Yang Zhong <yang.zhong@intel.com>
---
 target/i386/kvm/kvm.c | 5 +++++
 1 file changed, 5 insertions(+)

diff --git a/target/i386/kvm/kvm.c b/target/i386/kvm/kvm.c
index d4bf054ebe..8d017fb5a6 100644
--- a/target/i386/kvm/kvm.c
+++ b/target/i386/kvm/kvm.c
@@ -1865,6 +1865,11 @@ int kvm_arch_init_vcpu(CPUState *cs)
                                   !!(c->ecx & CPUID_EXT_SMX);
     }
 
+    c = cpuid_find_entry(&cpuid_data.cpuid, 7, 0);
+    if (c && (c->ebx & CPUID_7_0_EBX_SGX)) {
+        has_msr_feature_control = true;
+    }
+
     if (env->mcg_cap & MCG_LMCE_P) {
         has_msr_mcg_ext_ctl = has_msr_feature_control = true;
     }


^ permalink raw reply related	[flat|nested] 61+ messages in thread

* [PATCH v4 12/33] i386: Update SGX CPUID info according to hardware/KVM/user input
  2021-07-19 11:21 [PATCH v4 00/33] Qemu SGX virtualization Yang Zhong
                   ` (10 preceding siblings ...)
  2021-07-19 11:21 ` [PATCH v4 11/33] i386: Add feature control MSR dependency when SGX is enabled Yang Zhong
@ 2021-07-19 11:21 ` Yang Zhong
  2021-07-19 11:21 ` [PATCH v4 13/33] i386: kvm: Add support for exposing PROVISIONKEY to guest Yang Zhong
                   ` (23 subsequent siblings)
  35 siblings, 0 replies; 61+ messages in thread
From: Yang Zhong @ 2021-07-19 11:21 UTC (permalink / raw)
  To: qemu-devel; +Cc: yang.zhong, seanjc, kai.huang, jarkko, pbonzini, eblake

From: Sean Christopherson <sean.j.christopherson@intel.com>

Expose SGX to the guest if and only if KVM is enabled and supports
virtualization of SGX.  While the majority of ENCLS can be emulated to
some degree, because SGX uses a hardware-based root of trust, the
attestation aspects of SGX cannot be emulated in software, i.e.
ultimately emulation will fail as software cannot generate a valid
quote/report.  The complexity of partially emulating SGX in Qemu far
outweighs the value added, e.g. an SGX specific simulator for userspace
applications can emulate SGX for development and testing purposes.

Note, access to the PROVISIONKEY is not yet advertised to the guest as
KVM blocks access to the PROVISIONKEY by default and requires userspace
to provide additional credentials (via ioctl()) to expose PROVISIONKEY.

Signed-off-by: Sean Christopherson <sean.j.christopherson@intel.com>
Signed-off-by: Yang Zhong <yang.zhong@intel.com>

v3-->v4:
   - Replaced g_malloc0() with directly ....
---
 hw/i386/sgx.c             | 17 +++++++++
 include/hw/i386/sgx-epc.h |  2 +
 target/i386/cpu.c         | 77 +++++++++++++++++++++++++++++++++++++++
 3 files changed, 96 insertions(+)

diff --git a/hw/i386/sgx.c b/hw/i386/sgx.c
index e77deb0b00..5f988c6368 100644
--- a/hw/i386/sgx.c
+++ b/hw/i386/sgx.c
@@ -18,6 +18,23 @@
 #include "qapi/error.h"
 #include "exec/address-spaces.h"
 
+int sgx_epc_get_section(int section_nr, uint64_t *addr, uint64_t *size)
+{
+    PCMachineState *pcms = PC_MACHINE(qdev_get_machine());
+    SGXEPCDevice *epc;
+
+    if (pcms->sgx_epc.size == 0 || pcms->sgx_epc.nr_sections <= section_nr) {
+        return 1;
+    }
+
+    epc = pcms->sgx_epc.sections[section_nr];
+
+    *addr = epc->addr;
+    *size = memory_device_get_region_size(MEMORY_DEVICE(epc), &error_fatal);
+
+    return 0;
+}
+
 static int sgx_epc_set_property(void *opaque, const char *name,
                                 const char *value, Error **errp)
 {
diff --git a/include/hw/i386/sgx-epc.h b/include/hw/i386/sgx-epc.h
index 2b2490892b..f85fd2a4ca 100644
--- a/include/hw/i386/sgx-epc.h
+++ b/include/hw/i386/sgx-epc.h
@@ -55,4 +55,6 @@ typedef struct SGXEPCState {
     int nr_sections;
 } SGXEPCState;
 
+int sgx_epc_get_section(int section_nr, uint64_t *addr, uint64_t *size);
+
 #endif
diff --git a/target/i386/cpu.c b/target/i386/cpu.c
index cea4307930..8adfe80312 100644
--- a/target/i386/cpu.c
+++ b/target/i386/cpu.c
@@ -36,6 +36,7 @@
 #ifndef CONFIG_USER_ONLY
 #include "exec/address-spaces.h"
 #include "hw/boards.h"
+#include "hw/i386/sgx-epc.h"
 #endif
 
 #include "disas/capstone.h"
@@ -5301,6 +5302,25 @@ void cpu_x86_cpuid(CPUX86State *env, uint32_t index, uint32_t count,
                 *ecx |= CPUID_7_0_ECX_OSPKE;
             }
             *edx = env->features[FEAT_7_0_EDX]; /* Feature flags */
+
+            /*
+             * SGX cannot be emulated in software.  If hardware does not
+             * support enabling SGX and/or SGX flexible launch control,
+             * then we need to update the VM's CPUID values accordingly.
+             */
+            if ((*ebx & CPUID_7_0_EBX_SGX) &&
+                (!kvm_enabled() ||
+                 !(kvm_arch_get_supported_cpuid(cs->kvm_state, 0x7, 0, R_EBX) &
+                    CPUID_7_0_EBX_SGX))) {
+                *ebx &= ~CPUID_7_0_EBX_SGX;
+            }
+
+            if ((*ecx & CPUID_7_0_ECX_SGX_LC) &&
+                (!(*ebx & CPUID_7_0_EBX_SGX) || !kvm_enabled() ||
+                 !(kvm_arch_get_supported_cpuid(cs->kvm_state, 0x7, 0, R_ECX) &
+                    CPUID_7_0_ECX_SGX_LC))) {
+                *ecx &= ~CPUID_7_0_ECX_SGX_LC;
+            }
         } else if (count == 1) {
             *eax = env->features[FEAT_7_1_EAX];
             *ebx = 0;
@@ -5436,6 +5456,63 @@ void cpu_x86_cpuid(CPUX86State *env, uint32_t index, uint32_t count,
         }
         break;
     }
+    case 0x12:
+#ifndef CONFIG_USER_ONLY
+        if (!kvm_enabled() ||
+            !(env->features[FEAT_7_0_EBX] & CPUID_7_0_EBX_SGX)) {
+            *eax = *ebx = *ecx = *edx = 0;
+            break;
+        }
+
+        /*
+         * SGX sub-leafs CPUID.0x12.{0x2..N} enumerate EPC sections.  Retrieve
+         * the EPC properties, e.g. confidentiality and integrity, from the
+         * host's first EPC section, i.e. assume there is one EPC section or
+         * that all EPC sections have the same security properties.
+         */
+        if (count > 1) {
+            uint64_t epc_addr, epc_size;
+
+            if (sgx_epc_get_section(count - 2, &epc_addr, &epc_size)) {
+                *eax = *ebx = *ecx = *edx = 0;
+                break;
+            }
+            host_cpuid(index, 2, eax, ebx, ecx, edx);
+            *eax = (uint32_t)(epc_addr & 0xfffff000) | 0x1;
+            *ebx = (uint32_t)(epc_addr >> 32);
+            *ecx = (uint32_t)(epc_size & 0xfffff000) | (*ecx & 0xf);
+            *edx = (uint32_t)(epc_size >> 32);
+            break;
+        }
+
+        /*
+         * SGX sub-leafs CPUID.0x12.{0x0,0x1} are heavily dependent on hardware
+         * and KVM, i.e. QEMU cannot emulate features to override what KVM
+         * supports.  Features can be further restricted by userspace, but not
+         * made more permissive.
+         */
+        *eax = kvm_arch_get_supported_cpuid(cs->kvm_state, 0x12, count, R_EAX);
+        *ebx = kvm_arch_get_supported_cpuid(cs->kvm_state, 0x12, count, R_EBX);
+        *ecx = kvm_arch_get_supported_cpuid(cs->kvm_state, 0x12, count, R_ECX);
+        *edx = kvm_arch_get_supported_cpuid(cs->kvm_state, 0x12, count, R_EDX);
+
+        if (count == 0) {
+            *eax &= env->features[FEAT_SGX_12_0_EAX];
+            *ebx &= env->features[FEAT_SGX_12_0_EBX];
+        } else {
+            *eax &= env->features[FEAT_SGX_12_1_EAX];
+            *ebx &= 0; /* ebx reserve */
+            *ecx &= env->features[FEAT_XSAVE_COMP_LO];
+            *edx &= env->features[FEAT_XSAVE_COMP_HI];
+
+            /* FP and SSE are always allowed regardless of XSAVE/XCR0. */
+            *ecx |= XSTATE_FP_MASK | XSTATE_SSE_MASK;
+
+            /* Access to PROVISIONKEY requires additional credentials. */
+            *eax &= ~(1U << 4);
+        }
+#endif
+        break;
     case 0x14: {
         /* Intel Processor Trace Enumeration */
         *eax = 0;


^ permalink raw reply related	[flat|nested] 61+ messages in thread

* [PATCH v4 13/33] i386: kvm: Add support for exposing PROVISIONKEY to guest
  2021-07-19 11:21 [PATCH v4 00/33] Qemu SGX virtualization Yang Zhong
                   ` (11 preceding siblings ...)
  2021-07-19 11:21 ` [PATCH v4 12/33] i386: Update SGX CPUID info according to hardware/KVM/user input Yang Zhong
@ 2021-07-19 11:21 ` Yang Zhong
  2021-07-19 11:21 ` [PATCH v4 14/33] i386: Propagate SGX CPUID sub-leafs to KVM Yang Zhong
                   ` (22 subsequent siblings)
  35 siblings, 0 replies; 61+ messages in thread
From: Yang Zhong @ 2021-07-19 11:21 UTC (permalink / raw)
  To: qemu-devel; +Cc: yang.zhong, seanjc, kai.huang, jarkko, pbonzini, eblake

From: Sean Christopherson <sean.j.christopherson@intel.com>

If the guest want to fully use SGX, the guest needs to be able to
access provisioning key. Add a new KVM_CAP_SGX_ATTRIBUTE to KVM to
support provisioning key to KVM guests.

Signed-off-by: Sean Christopherson <sean.j.christopherson@intel.com>
Signed-off-by: Yang Zhong <yang.zhong@intel.com>
---
 target/i386/cpu.c          |  5 ++++-
 target/i386/kvm/kvm.c      | 29 +++++++++++++++++++++++++++++
 target/i386/kvm/kvm_i386.h |  2 ++
 3 files changed, 35 insertions(+), 1 deletion(-)

diff --git a/target/i386/cpu.c b/target/i386/cpu.c
index 8adfe80312..6ddeba7461 100644
--- a/target/i386/cpu.c
+++ b/target/i386/cpu.c
@@ -5509,7 +5509,10 @@ void cpu_x86_cpuid(CPUX86State *env, uint32_t index, uint32_t count,
             *ecx |= XSTATE_FP_MASK | XSTATE_SSE_MASK;
 
             /* Access to PROVISIONKEY requires additional credentials. */
-            *eax &= ~(1U << 4);
+            if ((*eax & (1U << 4)) &&
+                !kvm_enable_sgx_provisioning(cs->kvm_state)) {
+                *eax &= ~(1U << 4);
+            }
         }
 #endif
         break;
diff --git a/target/i386/kvm/kvm.c b/target/i386/kvm/kvm.c
index 8d017fb5a6..05a1c370c3 100644
--- a/target/i386/kvm/kvm.c
+++ b/target/i386/kvm/kvm.c
@@ -4633,6 +4633,35 @@ void kvm_arch_update_guest_debug(CPUState *cpu, struct kvm_guest_debug *dbg)
     }
 }
 
+static bool has_sgx_provisioning;
+
+static bool __kvm_enable_sgx_provisioning(KVMState *s)
+{
+    int fd, ret;
+
+    if (!kvm_vm_check_extension(s, KVM_CAP_SGX_ATTRIBUTE)) {
+        return false;
+    }
+
+    fd = qemu_open_old("/dev/sgx_provision", O_RDONLY);
+    if (fd < 0) {
+        return false;
+    }
+
+    ret = kvm_vm_enable_cap(s, KVM_CAP_SGX_ATTRIBUTE, 0, fd);
+    if (ret) {
+        error_report("Could not enable SGX PROVISIONKEY: %s", strerror(-ret));
+        exit(1);
+    }
+    close(fd);
+    return true;
+}
+
+bool kvm_enable_sgx_provisioning(KVMState *s)
+{
+    return MEMORIZE(__kvm_enable_sgx_provisioning(s), has_sgx_provisioning);
+}
+
 static bool host_supports_vmx(void)
 {
     uint32_t ecx, unused;
diff --git a/target/i386/kvm/kvm_i386.h b/target/i386/kvm/kvm_i386.h
index 54667b35f0..a978509d50 100644
--- a/target/i386/kvm/kvm_i386.h
+++ b/target/i386/kvm/kvm_i386.h
@@ -51,4 +51,6 @@ bool kvm_hyperv_expand_features(X86CPU *cpu, Error **errp);
 
 uint64_t kvm_swizzle_msi_ext_dest_id(uint64_t address);
 
+bool kvm_enable_sgx_provisioning(KVMState *s);
+
 #endif


^ permalink raw reply related	[flat|nested] 61+ messages in thread

* [PATCH v4 14/33] i386: Propagate SGX CPUID sub-leafs to KVM
  2021-07-19 11:21 [PATCH v4 00/33] Qemu SGX virtualization Yang Zhong
                   ` (12 preceding siblings ...)
  2021-07-19 11:21 ` [PATCH v4 13/33] i386: kvm: Add support for exposing PROVISIONKEY to guest Yang Zhong
@ 2021-07-19 11:21 ` Yang Zhong
  2021-07-19 11:21 ` [PATCH v4 15/33] Adjust min CPUID level to 0x12 when SGX is enabled Yang Zhong
                   ` (21 subsequent siblings)
  35 siblings, 0 replies; 61+ messages in thread
From: Yang Zhong @ 2021-07-19 11:21 UTC (permalink / raw)
  To: qemu-devel; +Cc: yang.zhong, seanjc, kai.huang, jarkko, pbonzini, eblake

From: Sean Christopherson <sean.j.christopherson@intel.com>

The SGX sub-leafs are enumerated at CPUID 0x12.  Indices 0 and 1 are
always present when SGX is supported, and enumerate SGX features and
capabilities.  Indices >=2 are directly correlated with the platform's
EPC sections.  Because the number of EPC sections is dynamic and user
defined, the number of SGX sub-leafs is "NULL" terminated.

Signed-off-by: Sean Christopherson <sean.j.christopherson@intel.com>
Signed-off-by: Yang Zhong <yang.zhong@intel.com>
---
 target/i386/kvm/kvm.c | 19 +++++++++++++++++++
 1 file changed, 19 insertions(+)

diff --git a/target/i386/kvm/kvm.c b/target/i386/kvm/kvm.c
index 05a1c370c3..c8179dcfb7 100644
--- a/target/i386/kvm/kvm.c
+++ b/target/i386/kvm/kvm.c
@@ -1691,6 +1691,25 @@ int kvm_arch_init_vcpu(CPUState *cs)
             }
             break;
         case 0x7:
+        case 0x12:
+            for (j = 0; ; j++) {
+                c->function = i;
+                c->flags = KVM_CPUID_FLAG_SIGNIFCANT_INDEX;
+                c->index = j;
+                cpu_x86_cpuid(env, i, j, &c->eax, &c->ebx, &c->ecx, &c->edx);
+
+                if (j > 1 && (c->eax & 0xf) != 1) {
+                    break;
+                }
+
+                if (cpuid_i == KVM_MAX_CPUID_ENTRIES) {
+                    fprintf(stderr, "cpuid_data is full, no space for "
+                                "cpuid(eax:0x12,ecx:0x%x)\n", j);
+                    abort();
+                }
+                c = &cpuid_data.entries[cpuid_i++];
+            }
+            break;
         case 0x14: {
             uint32_t times;
 


^ permalink raw reply related	[flat|nested] 61+ messages in thread

* [PATCH v4 15/33] Adjust min CPUID level to 0x12 when SGX is enabled
  2021-07-19 11:21 [PATCH v4 00/33] Qemu SGX virtualization Yang Zhong
                   ` (13 preceding siblings ...)
  2021-07-19 11:21 ` [PATCH v4 14/33] i386: Propagate SGX CPUID sub-leafs to KVM Yang Zhong
@ 2021-07-19 11:21 ` Yang Zhong
  2021-07-19 11:21 ` [PATCH v4 16/33] hw/i386/fw_cfg: Set SGX bits in feature control fw_cfg accordingly Yang Zhong
                   ` (20 subsequent siblings)
  35 siblings, 0 replies; 61+ messages in thread
From: Yang Zhong @ 2021-07-19 11:21 UTC (permalink / raw)
  To: qemu-devel; +Cc: yang.zhong, seanjc, kai.huang, jarkko, pbonzini, eblake

From: Sean Christopherson <sean.j.christopherson@intel.com>

SGX capabilities are enumerated through CPUID_0x12.

Signed-off-by: Sean Christopherson <sean.j.christopherson@intel.com>
Signed-off-by: Yang Zhong <yang.zhong@intel.com>
---
 target/i386/cpu.c | 5 +++++
 1 file changed, 5 insertions(+)

diff --git a/target/i386/cpu.c b/target/i386/cpu.c
index 6ddeba7461..1f2f6bdf2d 100644
--- a/target/i386/cpu.c
+++ b/target/i386/cpu.c
@@ -6125,6 +6125,11 @@ void x86_cpu_expand_features(X86CPU *cpu, Error **errp)
         if (sev_enabled()) {
             x86_cpu_adjust_level(cpu, &env->cpuid_min_xlevel, 0x8000001F);
         }
+
+        /* SGX requires CPUID[0x12] for EPC enumeration */
+        if (env->features[FEAT_7_0_EBX] & CPUID_7_0_EBX_SGX) {
+            x86_cpu_adjust_level(cpu, &env->cpuid_min_level, 0x12);
+        }
     }
 
     /* Set cpuid_*level* based on cpuid_min_*level, if not explicitly set */


^ permalink raw reply related	[flat|nested] 61+ messages in thread

* [PATCH v4 16/33] hw/i386/fw_cfg: Set SGX bits in feature control fw_cfg accordingly
  2021-07-19 11:21 [PATCH v4 00/33] Qemu SGX virtualization Yang Zhong
                   ` (14 preceding siblings ...)
  2021-07-19 11:21 ` [PATCH v4 15/33] Adjust min CPUID level to 0x12 when SGX is enabled Yang Zhong
@ 2021-07-19 11:21 ` Yang Zhong
  2021-07-19 11:21 ` [PATCH v4 17/33] hw/i386/pc: Account for SGX EPC sections when calculating device memory Yang Zhong
                   ` (19 subsequent siblings)
  35 siblings, 0 replies; 61+ messages in thread
From: Yang Zhong @ 2021-07-19 11:21 UTC (permalink / raw)
  To: qemu-devel; +Cc: yang.zhong, seanjc, kai.huang, jarkko, pbonzini, eblake

From: Sean Christopherson <sean.j.christopherson@intel.com>

Request SGX an SGX Launch Control to be enabled in FEATURE_CONTROL
when the features are exposed to the guest. Our design is the SGX
Launch Control bit will be unconditionally set in FEATURE_CONTROL,
which is unlike host bios.

Signed-off-by: Sean Christopherson <sean.j.christopherson@intel.com>
Signed-off-by: Yang Zhong <yang.zhong@intel.com>
---
 hw/i386/fw_cfg.c | 10 +++++++++-
 1 file changed, 9 insertions(+), 1 deletion(-)

diff --git a/hw/i386/fw_cfg.c b/hw/i386/fw_cfg.c
index 4e68d5dea4..f6d036dfbe 100644
--- a/hw/i386/fw_cfg.c
+++ b/hw/i386/fw_cfg.c
@@ -159,7 +159,7 @@ void fw_cfg_build_feature_control(MachineState *ms, FWCfgState *fw_cfg)
 {
     X86CPU *cpu = X86_CPU(ms->possible_cpus->cpus[0].cpu);
     CPUX86State *env = &cpu->env;
-    uint32_t unused, ecx, edx;
+    uint32_t unused, ebx, ecx, edx;
     uint64_t feature_control_bits = 0;
     uint64_t *val;
 
@@ -174,6 +174,14 @@ void fw_cfg_build_feature_control(MachineState *ms, FWCfgState *fw_cfg)
         feature_control_bits |= FEATURE_CONTROL_LMCE;
     }
 
+    cpu_x86_cpuid(env, 0x7, 0, &unused, &ebx, &ecx, &unused);
+    if (ebx & CPUID_7_0_EBX_SGX) {
+        feature_control_bits |= FEATURE_CONTROL_SGX;
+    }
+    if (ecx & CPUID_7_0_ECX_SGX_LC) {
+        feature_control_bits |= FEATURE_CONTROL_SGX_LC;
+    }
+
     if (!feature_control_bits) {
         return;
     }


^ permalink raw reply related	[flat|nested] 61+ messages in thread

* [PATCH v4 17/33] hw/i386/pc: Account for SGX EPC sections when calculating device memory
  2021-07-19 11:21 [PATCH v4 00/33] Qemu SGX virtualization Yang Zhong
                   ` (15 preceding siblings ...)
  2021-07-19 11:21 ` [PATCH v4 16/33] hw/i386/fw_cfg: Set SGX bits in feature control fw_cfg accordingly Yang Zhong
@ 2021-07-19 11:21 ` Yang Zhong
  2021-07-19 11:21 ` [PATCH v4 18/33] i386/pc: Add e820 entry for SGX EPC section(s) Yang Zhong
                   ` (18 subsequent siblings)
  35 siblings, 0 replies; 61+ messages in thread
From: Yang Zhong @ 2021-07-19 11:21 UTC (permalink / raw)
  To: qemu-devel; +Cc: yang.zhong, seanjc, kai.huang, jarkko, pbonzini, eblake

From: Sean Christopherson <sean.j.christopherson@intel.com>

Add helpers to detect if SGX EPC exists above 4g, and if so, where SGX
EPC above 4g ends.  Use the helpers to adjust the device memory range
if SGX EPC exists above 4g.

For multiple virtual EPC sections, we just put them together physically
contiguous for the simplicity because we don't support EPC NUMA affinity
now. Once the SGX EPC NUMA support in the kernel SGX driver, we will
support this in the future.

Note that SGX EPC is currently hardcoded to reside above 4g.

Signed-off-by: Sean Christopherson <sean.j.christopherson@intel.com>
Signed-off-by: Yang Zhong <yang.zhong@intel.com>
---
 hw/i386/pc.c              | 11 ++++++++++-
 include/hw/i386/sgx-epc.h |  7 +++++++
 2 files changed, 17 insertions(+), 1 deletion(-)

diff --git a/hw/i386/pc.c b/hw/i386/pc.c
index c2b9d62a35..109e835890 100644
--- a/hw/i386/pc.c
+++ b/hw/i386/pc.c
@@ -920,8 +920,15 @@ void pc_memory_init(PCMachineState *pcms,
             exit(EXIT_FAILURE);
         }
 
+        if (pcms->sgx_epc.size != 0) {
+            machine->device_memory->base = sgx_epc_above_4g_end(&pcms->sgx_epc);
+        } else {
+            machine->device_memory->base =
+                0x100000000ULL + x86ms->above_4g_mem_size;
+        }
+
         machine->device_memory->base =
-            ROUND_UP(0x100000000ULL + x86ms->above_4g_mem_size, 1 * GiB);
+            ROUND_UP(machine->device_memory->base, 1 * GiB);
 
         if (pcmc->enforce_aligned_dimm) {
             /* size device region assuming 1G page max alignment per slot */
@@ -1006,6 +1013,8 @@ uint64_t pc_pci_hole64_start(void)
         if (!pcmc->broken_reserved_end) {
             hole64_start += memory_region_size(&ms->device_memory->mr);
         }
+    } else if (pcms->sgx_epc.size != 0) {
+            hole64_start = sgx_epc_above_4g_end(&pcms->sgx_epc);
     } else {
         hole64_start = 0x100000000ULL + x86ms->above_4g_mem_size;
     }
diff --git a/include/hw/i386/sgx-epc.h b/include/hw/i386/sgx-epc.h
index f85fd2a4ca..155e8fad3e 100644
--- a/include/hw/i386/sgx-epc.h
+++ b/include/hw/i386/sgx-epc.h
@@ -57,4 +57,11 @@ typedef struct SGXEPCState {
 
 int sgx_epc_get_section(int section_nr, uint64_t *addr, uint64_t *size);
 
+static inline uint64_t sgx_epc_above_4g_end(SGXEPCState *sgx_epc)
+{
+    assert(sgx_epc != NULL && sgx_epc->base >= 0x100000000ULL);
+
+    return sgx_epc->base + sgx_epc->size;
+}
+
 #endif


^ permalink raw reply related	[flat|nested] 61+ messages in thread

* [PATCH v4 18/33] i386/pc: Add e820 entry for SGX EPC section(s)
  2021-07-19 11:21 [PATCH v4 00/33] Qemu SGX virtualization Yang Zhong
                   ` (16 preceding siblings ...)
  2021-07-19 11:21 ` [PATCH v4 17/33] hw/i386/pc: Account for SGX EPC sections when calculating device memory Yang Zhong
@ 2021-07-19 11:21 ` Yang Zhong
  2021-07-19 11:21 ` [PATCH v4 19/33] i386: acpi: Add SGX EPC entry to ACPI tables Yang Zhong
                   ` (17 subsequent siblings)
  35 siblings, 0 replies; 61+ messages in thread
From: Yang Zhong @ 2021-07-19 11:21 UTC (permalink / raw)
  To: qemu-devel; +Cc: yang.zhong, seanjc, kai.huang, jarkko, pbonzini, eblake

From: Sean Christopherson <sean.j.christopherson@intel.com>

Note that SGX EPC is currently guaranteed to reside in a single
contiguous chunk of memory regardless of the number of EPC sections.

Signed-off-by: Sean Christopherson <sean.j.christopherson@intel.com>
Signed-off-by: Yang Zhong <yang.zhong@intel.com>
---
 hw/i386/pc.c | 4 ++++
 1 file changed, 4 insertions(+)

diff --git a/hw/i386/pc.c b/hw/i386/pc.c
index 109e835890..80d3b5e62d 100644
--- a/hw/i386/pc.c
+++ b/hw/i386/pc.c
@@ -890,6 +890,10 @@ void pc_memory_init(PCMachineState *pcms,
         e820_add_entry(0x100000000ULL, x86ms->above_4g_mem_size, E820_RAM);
     }
 
+    if (pcms->sgx_epc.size != 0) {
+        e820_add_entry(pcms->sgx_epc.base, pcms->sgx_epc.size, E820_RESERVED);
+    }
+
     if (!pcmc->has_reserved_memory &&
         (machine->ram_slots ||
          (machine->maxram_size > machine->ram_size))) {


^ permalink raw reply related	[flat|nested] 61+ messages in thread

* [PATCH v4 19/33] i386: acpi: Add SGX EPC entry to ACPI tables
  2021-07-19 11:21 [PATCH v4 00/33] Qemu SGX virtualization Yang Zhong
                   ` (17 preceding siblings ...)
  2021-07-19 11:21 ` [PATCH v4 18/33] i386/pc: Add e820 entry for SGX EPC section(s) Yang Zhong
@ 2021-07-19 11:21 ` Yang Zhong
  2021-07-19 11:21 ` [PATCH v4 20/33] q35: Add support for SGX EPC Yang Zhong
                   ` (16 subsequent siblings)
  35 siblings, 0 replies; 61+ messages in thread
From: Yang Zhong @ 2021-07-19 11:21 UTC (permalink / raw)
  To: qemu-devel; +Cc: yang.zhong, seanjc, kai.huang, jarkko, pbonzini, eblake

From: Sean Christopherson <sean.j.christopherson@intel.com>

The ACPI Device entry for SGX EPC is essentially a hack whose primary
purpose is to provide software with a way to autoprobe SGX support,
e.g. to allow software to implement SGX support as a driver.  Details
on the individual EPC sections are not enumerated through ACPI tables,
i.e. software must enumerate the EPC sections via CPUID.  Furthermore,
software expects to see only a single EPC Device in the ACPI tables
regardless of the number of EPC sections in the system.

However, several versions of Windows do rely on the ACPI tables to
enumerate the address and size of the EPC.  So, regardless of the number
of EPC sections exposed to the guest, create exactly *one* EPC device
with a _CRS entry that spans the entirety of all EPC sections (which are
guaranteed to be contiguous in Qemu).

Note, NUMA support for EPC memory is intentionally not considered as
enumerating EPC NUMA information is not yet defined for bare metal.

Signed-off-by: Sean Christopherson <sean.j.christopherson@intel.com>
Signed-off-by: Yang Zhong <yang.zhong@intel.com>
---
 hw/i386/acpi-build.c | 22 ++++++++++++++++++++++
 1 file changed, 22 insertions(+)

diff --git a/hw/i386/acpi-build.c b/hw/i386/acpi-build.c
index 17836149fe..efebe640ce 100644
--- a/hw/i386/acpi-build.c
+++ b/hw/i386/acpi-build.c
@@ -1829,6 +1829,28 @@ build_dsdt(GArray *table_data, BIOSLinker *linker,
     }
 #endif
 
+    if (pcms->sgx_epc.size != 0) {
+        uint64_t epc_base = pcms->sgx_epc.base;
+        uint64_t epc_size = pcms->sgx_epc.size;
+
+        dev = aml_device("EPC");
+        aml_append(dev, aml_name_decl("_HID", aml_eisaid("INT0E0C")));
+        aml_append(dev, aml_name_decl("_STR",
+                                      aml_unicode("Enclave Page Cache 1.0")));
+        crs = aml_resource_template();
+        aml_append(crs,
+                   aml_qword_memory(AML_POS_DECODE, AML_MIN_FIXED,
+                                    AML_MAX_FIXED, AML_NON_CACHEABLE,
+                                    AML_READ_WRITE, 0, epc_base,
+                                    epc_base + epc_size - 1, 0, epc_size));
+        aml_append(dev, aml_name_decl("_CRS", crs));
+
+        method = aml_method("_STA", 0, AML_NOTSERIALIZED);
+        aml_append(method, aml_return(aml_int(0x0f)));
+        aml_append(dev, method);
+
+        aml_append(sb_scope, dev);
+    }
     aml_append(dsdt, sb_scope);
 
     /* copy AML table into ACPI tables blob and patch header there */


^ permalink raw reply related	[flat|nested] 61+ messages in thread

* [PATCH v4 20/33] q35: Add support for SGX EPC
  2021-07-19 11:21 [PATCH v4 00/33] Qemu SGX virtualization Yang Zhong
                   ` (18 preceding siblings ...)
  2021-07-19 11:21 ` [PATCH v4 19/33] i386: acpi: Add SGX EPC entry to ACPI tables Yang Zhong
@ 2021-07-19 11:21 ` Yang Zhong
  2021-07-19 11:21 ` [PATCH v4 21/33] i440fx: " Yang Zhong
                   ` (15 subsequent siblings)
  35 siblings, 0 replies; 61+ messages in thread
From: Yang Zhong @ 2021-07-19 11:21 UTC (permalink / raw)
  To: qemu-devel; +Cc: yang.zhong, seanjc, kai.huang, jarkko, pbonzini, eblake

From: Sean Christopherson <sean.j.christopherson@intel.com>

Enable SGX EPC virtualization, which is currently only support by KVM.

Signed-off-by: Sean Christopherson <sean.j.christopherson@intel.com>
Signed-off-by: Yang Zhong <yang.zhong@intel.com>
---
 hw/i386/pc_q35.c | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/hw/i386/pc_q35.c b/hw/i386/pc_q35.c
index 04b4a4788d..799c4fc61c 100644
--- a/hw/i386/pc_q35.c
+++ b/hw/i386/pc_q35.c
@@ -178,6 +178,9 @@ static void pc_q35_init(MachineState *machine)
         x86ms->below_4g_mem_size = machine->ram_size;
     }
 
+    if (x86ms->sgx_epc_list) {
+        pc_machine_init_sgx_epc(pcms);
+    }
     x86_cpus_init(x86ms, pcmc->default_cpu_version);
 
     kvmclock_create(pcmc->kvmclock_create_always);


^ permalink raw reply related	[flat|nested] 61+ messages in thread

* [PATCH v4 21/33] i440fx: Add support for SGX EPC
  2021-07-19 11:21 [PATCH v4 00/33] Qemu SGX virtualization Yang Zhong
                   ` (19 preceding siblings ...)
  2021-07-19 11:21 ` [PATCH v4 20/33] q35: Add support for SGX EPC Yang Zhong
@ 2021-07-19 11:21 ` Yang Zhong
  2021-07-19 11:21 ` [PATCH v4 22/33] hostmem-epc: Add the reset interface for EPC backend reset Yang Zhong
                   ` (14 subsequent siblings)
  35 siblings, 0 replies; 61+ messages in thread
From: Yang Zhong @ 2021-07-19 11:21 UTC (permalink / raw)
  To: qemu-devel; +Cc: yang.zhong, seanjc, kai.huang, jarkko, pbonzini, eblake

From: Sean Christopherson <sean.j.christopherson@intel.com>

Enable SGX EPC virtualization, which is currently only support by KVM.

Signed-off-by: Sean Christopherson <sean.j.christopherson@intel.com>
Signed-off-by: Yang Zhong <yang.zhong@intel.com>
---
 hw/i386/pc_piix.c | 4 ++++
 1 file changed, 4 insertions(+)

diff --git a/hw/i386/pc_piix.c b/hw/i386/pc_piix.c
index 30b8bd6ea9..c0a7552b88 100644
--- a/hw/i386/pc_piix.c
+++ b/hw/i386/pc_piix.c
@@ -154,6 +154,10 @@ static void pc_init1(MachineState *machine,
         }
     }
 
+    if (x86ms->sgx_epc_list) {
+        pc_machine_init_sgx_epc(pcms);
+    }
+
     x86_cpus_init(x86ms, pcmc->default_cpu_version);
 
     if (pcmc->kvmclock_enabled) {


^ permalink raw reply related	[flat|nested] 61+ messages in thread

* [PATCH v4 22/33] hostmem-epc: Add the reset interface for EPC backend reset
  2021-07-19 11:21 [PATCH v4 00/33] Qemu SGX virtualization Yang Zhong
                   ` (20 preceding siblings ...)
  2021-07-19 11:21 ` [PATCH v4 21/33] i440fx: " Yang Zhong
@ 2021-07-19 11:21 ` Yang Zhong
  2021-09-10 15:10   ` Paolo Bonzini
  2021-07-19 11:21 ` [PATCH v4 23/33] sgx-epc: Add the reset interface for sgx-epc virt device Yang Zhong
                   ` (13 subsequent siblings)
  35 siblings, 1 reply; 61+ messages in thread
From: Yang Zhong @ 2021-07-19 11:21 UTC (permalink / raw)
  To: qemu-devel; +Cc: yang.zhong, seanjc, kai.huang, jarkko, pbonzini, eblake

Add the sgx_memory_backend_reset() interface to handle EPC backend
reset when VM is reset. This reset function will destroy previous
backend memory region and re-mmap the EPC section for guest.

Signed-off-by: Yang Zhong <yang.zhong@intel.com>
---
 backends/hostmem-epc.c | 16 ++++++++++++++++
 include/hw/i386/pc.h   |  2 ++
 2 files changed, 18 insertions(+)

diff --git a/backends/hostmem-epc.c b/backends/hostmem-epc.c
index b512a68cb0..3bd1535d82 100644
--- a/backends/hostmem-epc.c
+++ b/backends/hostmem-epc.c
@@ -16,6 +16,7 @@
 #include "qom/object_interfaces.h"
 #include "qapi/error.h"
 #include "sysemu/hostmem.h"
+#include "hw/i386/pc.h"
 
 #define TYPE_MEMORY_BACKEND_EPC "memory-backend-epc"
 
@@ -55,6 +56,21 @@ sgx_epc_backend_memory_alloc(HostMemoryBackend *backend, Error **errp)
     g_free(name);
 }
 
+void sgx_memory_backend_reset(HostMemoryBackend *backend, int fd,
+                              Error **errp)
+{
+    MemoryRegion *mr = &backend->mr;
+
+    mr->enabled = false;
+
+    /* destroy the old memory region if it exist */
+    if (fd > 0 && mr->destructor) {
+        mr->destructor(mr);
+    }
+
+    sgx_epc_backend_memory_alloc(backend, errp);
+}
+
 static void sgx_epc_backend_instance_init(Object *obj)
 {
     HostMemoryBackend *m = MEMORY_BACKEND(obj);
diff --git a/include/hw/i386/pc.h b/include/hw/i386/pc.h
index 2fd6e6193b..c725dfef5b 100644
--- a/include/hw/i386/pc.h
+++ b/include/hw/i386/pc.h
@@ -204,6 +204,8 @@ extern const size_t pc_compat_6_0_len;
 
 /* sgx-epc.c */
 void pc_machine_init_sgx_epc(PCMachineState *pcms);
+void sgx_memory_backend_reset(HostMemoryBackend *backend, int fd,
+                              Error **errp);
 
 extern GlobalProperty pc_compat_5_2[];
 extern const size_t pc_compat_5_2_len;


^ permalink raw reply related	[flat|nested] 61+ messages in thread

* [PATCH v4 23/33] sgx-epc: Add the reset interface for sgx-epc virt device
  2021-07-19 11:21 [PATCH v4 00/33] Qemu SGX virtualization Yang Zhong
                   ` (21 preceding siblings ...)
  2021-07-19 11:21 ` [PATCH v4 22/33] hostmem-epc: Add the reset interface for EPC backend reset Yang Zhong
@ 2021-07-19 11:21 ` Yang Zhong
  2021-09-10 15:13   ` Paolo Bonzini
  2021-09-14  6:53   ` Philippe Mathieu-Daudé
  2021-07-19 11:21 ` [PATCH v4 24/33] sgx-epc: Avoid bios reset during sgx epc initialization Yang Zhong
                   ` (12 subsequent siblings)
  35 siblings, 2 replies; 61+ messages in thread
From: Yang Zhong @ 2021-07-19 11:21 UTC (permalink / raw)
  To: qemu-devel; +Cc: yang.zhong, seanjc, kai.huang, jarkko, pbonzini, eblake

If the VM is reset, we need make sure sgx virt epc in clean status.
Once the VM is reset, and sgx epc virt device will be reseted by
reset callback registered by qemu_register_reset(). Since this epc
virt device depend on backend, this reset will call backend reset
interface to re-mmap epc to guest.

Signed-off-by: Yang Zhong <yang.zhong@intel.com>
---
 hw/i386/sgx-epc.c | 94 ++++++++++++++++++++++++++++++++++++++++-------
 1 file changed, 81 insertions(+), 13 deletions(-)

diff --git a/hw/i386/sgx-epc.c b/hw/i386/sgx-epc.c
index 924dea22f0..9880d832d5 100644
--- a/hw/i386/sgx-epc.c
+++ b/hw/i386/sgx-epc.c
@@ -18,6 +18,9 @@
 #include "qapi/visitor.h"
 #include "target/i386/cpu.h"
 #include "exec/address-spaces.h"
+#include "sysemu/reset.h"
+
+uint32_t epc_num;
 
 static Property sgx_epc_properties[] = {
     DEFINE_PROP_UINT64(SGX_EPC_ADDR_PROP, SGXEPCDevice, addr, 0),
@@ -47,12 +50,84 @@ static void sgx_epc_init(Object *obj)
                         NULL, NULL, NULL);
 }
 
+static void sgx_epc_del_subregion(DeviceState *dev)
+{
+    PCMachineState *pcms = PC_MACHINE(qdev_get_machine());
+    SGXEPCState *sgx_epc = &pcms->sgx_epc;
+    SGXEPCDevice *epc = SGX_EPC(dev);
+
+    /* del subregion and related operations */
+    memory_region_del_subregion(&sgx_epc->mr,
+                                host_memory_backend_get_memory(epc->hostmem));
+    host_memory_backend_set_mapped(epc->hostmem, false);
+    g_free(sgx_epc->sections);
+    sgx_epc->sections = NULL;
+
+    /* multiple epc devices, only zero the first time */
+    if (epc_num == sgx_epc->nr_sections) {
+        sgx_epc->size = 0;
+        sgx_epc->nr_sections = 0;
+    }
+}
+
+static void sgx_epc_initialization(DeviceState *dev)
+{
+    PCMachineState *pcms = PC_MACHINE(qdev_get_machine());
+    SGXEPCState *sgx_epc = &pcms->sgx_epc;
+    MemoryDeviceState *md = MEMORY_DEVICE(dev);
+    SGXEPCDevice *epc = SGX_EPC(dev);
+    Error *errp = NULL;
+
+    if (!epc->hostmem) {
+        error_setg(&errp, "'" SGX_EPC_MEMDEV_PROP "' property is not set");
+        return;
+    }
+
+    epc->addr = sgx_epc->base + sgx_epc->size;
+
+    memory_region_add_subregion(&sgx_epc->mr, epc->addr - sgx_epc->base,
+                                host_memory_backend_get_memory(epc->hostmem));
+
+    host_memory_backend_set_mapped(epc->hostmem, true);
+
+    sgx_epc->sections = g_renew(SGXEPCDevice *, sgx_epc->sections,
+                                sgx_epc->nr_sections + 1);
+    sgx_epc->sections[sgx_epc->nr_sections++] = epc;
+
+    sgx_epc->size += memory_device_get_region_size(md, &errp);
+}
+
+static void sgx_epc_reset(void *opaque)
+{
+    DeviceState *dev = opaque;
+    SGXEPCDevice *epc = SGX_EPC(dev);
+    Error *errp = NULL;
+    int fd;
+
+    if (!epc->hostmem) {
+        error_setg(&errp, "'" SGX_EPC_MEMDEV_PROP "' property is not set");
+        return;
+    }
+
+    /* delete subregion and related operations */
+    sgx_epc_del_subregion(dev);
+
+    /* reset sgx backend */
+    fd = memory_region_get_fd(host_memory_backend_get_memory(epc->hostmem));
+    sgx_memory_backend_reset(epc->hostmem, fd, &errp);
+    if (errp) {
+        error_setg(&errp, "failed to call sgx_memory_backend_reset");
+        return;
+    }
+
+    /* re-add subregion and related operations */
+    sgx_epc_initialization(dev);
+}
+
 static void sgx_epc_realize(DeviceState *dev, Error **errp)
 {
     PCMachineState *pcms = PC_MACHINE(qdev_get_machine());
     X86MachineState *x86ms = X86_MACHINE(pcms);
-    MemoryDeviceState *md = MEMORY_DEVICE(dev);
-    SGXEPCState *sgx_epc = &pcms->sgx_epc;
     SGXEPCDevice *epc = SGX_EPC(dev);
     const char *path;
 
@@ -71,18 +146,11 @@ static void sgx_epc_realize(DeviceState *dev, Error **errp)
         return;
     }
 
-    epc->addr = sgx_epc->base + sgx_epc->size;
-
-    memory_region_add_subregion(&sgx_epc->mr, epc->addr - sgx_epc->base,
-                                host_memory_backend_get_memory(epc->hostmem));
-
-    host_memory_backend_set_mapped(epc->hostmem, true);
-
-    sgx_epc->sections = g_renew(SGXEPCDevice *, sgx_epc->sections,
-                                sgx_epc->nr_sections + 1);
-    sgx_epc->sections[sgx_epc->nr_sections++] = epc;
+    sgx_epc_initialization(dev);
+    epc_num++;
 
-    sgx_epc->size += memory_device_get_region_size(md, errp);
+    /* register the reset callback for sgx reset */
+    qemu_register_reset(sgx_epc_reset, dev);
 }
 
 static void sgx_epc_unrealize(DeviceState *dev)


^ permalink raw reply related	[flat|nested] 61+ messages in thread

* [PATCH v4 24/33] sgx-epc: Avoid bios reset during sgx epc initialization
  2021-07-19 11:21 [PATCH v4 00/33] Qemu SGX virtualization Yang Zhong
                   ` (22 preceding siblings ...)
  2021-07-19 11:21 ` [PATCH v4 23/33] sgx-epc: Add the reset interface for sgx-epc virt device Yang Zhong
@ 2021-07-19 11:21 ` Yang Zhong
  2021-07-19 11:21 ` [PATCH v4 25/33] hostmem-epc: Make prealloc consistent with qemu cmdline during reset Yang Zhong
                   ` (11 subsequent siblings)
  35 siblings, 0 replies; 61+ messages in thread
From: Yang Zhong @ 2021-07-19 11:21 UTC (permalink / raw)
  To: qemu-devel; +Cc: yang.zhong, seanjc, kai.huang, jarkko, pbonzini, eblake

Since bios do the reset when qemu boot up, and sgx epc will be
reset by the registered reset callback function. Like this, the
sgx epc will do two times initialization. This patch will check
protected mode from cr0 register, and will bypass reset operation
from bios. The reset callback will only accept reset operation
from guest.

Signed-off-by: Yang Zhong <yang.zhong@intel.com>
---
 hw/i386/sgx-epc.c | 19 +++++++++++++++++++
 1 file changed, 19 insertions(+)

diff --git a/hw/i386/sgx-epc.c b/hw/i386/sgx-epc.c
index 9880d832d5..70075db37c 100644
--- a/hw/i386/sgx-epc.c
+++ b/hw/i386/sgx-epc.c
@@ -19,6 +19,7 @@
 #include "target/i386/cpu.h"
 #include "exec/address-spaces.h"
 #include "sysemu/reset.h"
+#include "sysemu/hw_accel.h"
 
 uint32_t epc_num;
 
@@ -97,6 +98,21 @@ static void sgx_epc_initialization(DeviceState *dev)
     sgx_epc->size += memory_device_get_region_size(md, &errp);
 }
 
+static bool check_reset_from_guest(void)
+{
+    CPUState *cs = first_cpu;
+    X86CPU *cpu = X86_CPU(cs);
+    CPUX86State *env = &cpu->env;
+
+    cpu_synchronize_state(cs);
+
+    if (env->cr[0] & CR0_PE_MASK) {
+        return true;
+    }
+
+    return false;
+}
+
 static void sgx_epc_reset(void *opaque)
 {
     DeviceState *dev = opaque;
@@ -104,6 +120,9 @@ static void sgx_epc_reset(void *opaque)
     Error *errp = NULL;
     int fd;
 
+    if (!check_reset_from_guest())
+        return;
+
     if (!epc->hostmem) {
         error_setg(&errp, "'" SGX_EPC_MEMDEV_PROP "' property is not set");
         return;


^ permalink raw reply related	[flat|nested] 61+ messages in thread

* [PATCH v4 25/33] hostmem-epc: Make prealloc consistent with qemu cmdline during reset
  2021-07-19 11:21 [PATCH v4 00/33] Qemu SGX virtualization Yang Zhong
                   ` (23 preceding siblings ...)
  2021-07-19 11:21 ` [PATCH v4 24/33] sgx-epc: Avoid bios reset during sgx epc initialization Yang Zhong
@ 2021-07-19 11:21 ` Yang Zhong
  2021-07-19 11:21 ` [PATCH v4 26/33] qmp: Add query-sgx command Yang Zhong
                   ` (10 subsequent siblings)
  35 siblings, 0 replies; 61+ messages in thread
From: Yang Zhong @ 2021-07-19 11:21 UTC (permalink / raw)
  To: qemu-devel; +Cc: yang.zhong, seanjc, kai.huang, jarkko, pbonzini, eblake

If qemu cmdline set the prealloc property for sgx epc and VM do the
reset the prealloc property will be different with cmdline settings.
This patch can make sure same prealloc property setting with cmdline.

Signed-off-by: Yang Zhong <yang.zhong@intel.com>
---
 backends/hostmem-epc.c | 10 ++++++++++
 1 file changed, 10 insertions(+)

diff --git a/backends/hostmem-epc.c b/backends/hostmem-epc.c
index 3bd1535d82..9836358841 100644
--- a/backends/hostmem-epc.c
+++ b/backends/hostmem-epc.c
@@ -60,6 +60,8 @@ void sgx_memory_backend_reset(HostMemoryBackend *backend, int fd,
                               Error **errp)
 {
     MemoryRegion *mr = &backend->mr;
+    void *ptr;
+    uint64_t sz;
 
     mr->enabled = false;
 
@@ -69,6 +71,14 @@ void sgx_memory_backend_reset(HostMemoryBackend *backend, int fd,
     }
 
     sgx_epc_backend_memory_alloc(backend, errp);
+
+    ptr = memory_region_get_ram_ptr(&backend->mr);
+    sz = memory_region_size(&backend->mr);
+
+    if (backend->prealloc) {
+        os_mem_prealloc(memory_region_get_fd(&backend->mr), ptr, sz,
+                        backend->prealloc_threads, errp);
+    }
 }
 
 static void sgx_epc_backend_instance_init(Object *obj)


^ permalink raw reply related	[flat|nested] 61+ messages in thread

* [PATCH v4 26/33] qmp: Add query-sgx command
  2021-07-19 11:21 [PATCH v4 00/33] Qemu SGX virtualization Yang Zhong
                   ` (24 preceding siblings ...)
  2021-07-19 11:21 ` [PATCH v4 25/33] hostmem-epc: Make prealloc consistent with qemu cmdline during reset Yang Zhong
@ 2021-07-19 11:21 ` Yang Zhong
  2021-07-19 11:21 ` [PATCH v4 27/33] hmp: Add 'info sgx' command Yang Zhong
                   ` (9 subsequent siblings)
  35 siblings, 0 replies; 61+ messages in thread
From: Yang Zhong @ 2021-07-19 11:21 UTC (permalink / raw)
  To: qemu-devel; +Cc: yang.zhong, seanjc, kai.huang, jarkko, pbonzini, eblake

This QMP query command can be used by some userspaces to retrieve
the SGX information when SGX is enabled on Intel platform.

Signed-off-by: Yang Zhong <yang.zhong@intel.com>

v1-->v2:
   - "Since: 5.1" to "Since: 6.1", and grammar error(Eric Blake).
---
 monitor/qmp-cmds.c         |  6 ++++++
 qapi/misc.json             | 42 ++++++++++++++++++++++++++++++++++++++
 tests/qtest/qmp-cmd-test.c |  1 +
 3 files changed, 49 insertions(+)

diff --git a/monitor/qmp-cmds.c b/monitor/qmp-cmds.c
index f7d64a6457..d63d59149f 100644
--- a/monitor/qmp-cmds.c
+++ b/monitor/qmp-cmds.c
@@ -351,3 +351,9 @@ void qmp_display_reload(DisplayReloadOptions *arg, Error **errp)
         abort();
     }
 }
+
+SGXInfo *qmp_query_sgx(Error **errp)
+{
+    error_setg(errp, QERR_FEATURE_DISABLED, "query-sgx");
+    return NULL;
+}
diff --git a/qapi/misc.json b/qapi/misc.json
index 156f98203e..83b45a1460 100644
--- a/qapi/misc.json
+++ b/qapi/misc.json
@@ -519,3 +519,45 @@
  'data': { '*option': 'str' },
  'returns': ['CommandLineOptionInfo'],
  'allow-preconfig': true }
+
+##
+# @SGXInfo:
+#
+# Information about intel Safe Guard eXtension (SGX) support
+#
+# @sgx: true if SGX is supported
+#
+# @sgx1: true if SGX1 is supported
+#
+# @sgx2: true if SGX2 is supported
+#
+# @flc: true if FLC is supported
+#
+# @section-size: The EPC section size for guest
+#
+# Since: 6.1
+##
+{ 'struct': 'SGXInfo',
+  'data': { 'sgx': 'bool',
+            'sgx1': 'bool',
+            'sgx2': 'bool',
+            'flc': 'bool',
+            'section-size': 'uint64'}}
+
+##
+# @query-sgx:
+#
+# Returns information about SGX
+#
+# Returns: @SGXInfo
+#
+# Since: 6.1
+#
+# Example:
+#
+# -> { "execute": "query-sgx" }
+# <- { "return": { "sgx": true, "sgx1" : true, "sgx2" : true,
+#                  "flc": true, "section-size" : 0 } }
+#
+##
+{ 'command': 'query-sgx', 'returns': 'SGXInfo' }
diff --git a/tests/qtest/qmp-cmd-test.c b/tests/qtest/qmp-cmd-test.c
index c98b78d033..b75f3364f3 100644
--- a/tests/qtest/qmp-cmd-test.c
+++ b/tests/qtest/qmp-cmd-test.c
@@ -100,6 +100,7 @@ static bool query_is_ignored(const char *cmd)
         /* Success depends on Host or Hypervisor SEV support */
         "query-sev",
         "query-sev-capabilities",
+        "query-sgx",
         NULL
     };
     int i;


^ permalink raw reply related	[flat|nested] 61+ messages in thread

* [PATCH v4 27/33] hmp: Add 'info sgx' command
  2021-07-19 11:21 [PATCH v4 00/33] Qemu SGX virtualization Yang Zhong
                   ` (25 preceding siblings ...)
  2021-07-19 11:21 ` [PATCH v4 26/33] qmp: Add query-sgx command Yang Zhong
@ 2021-07-19 11:21 ` Yang Zhong
  2021-07-19 11:21 ` [PATCH v4 28/33] i386: Add sgx_get_info() interface Yang Zhong
                   ` (8 subsequent siblings)
  35 siblings, 0 replies; 61+ messages in thread
From: Yang Zhong @ 2021-07-19 11:21 UTC (permalink / raw)
  To: qemu-devel; +Cc: yang.zhong, seanjc, kai.huang, jarkko, pbonzini, eblake

The command can be used to show the SGX information in the monitor
when SGX is enabled on intel platform.

Signed-off-by: Yang Zhong <yang.zhong@intel.com>
---
 hmp-commands-info.hx  | 15 +++++++++++++++
 include/monitor/hmp.h |  1 +
 monitor/hmp-cmds.c    |  6 ++++++
 3 files changed, 22 insertions(+)

diff --git a/hmp-commands-info.hx b/hmp-commands-info.hx
index 27206ac049..4c966e8a6b 100644
--- a/hmp-commands-info.hx
+++ b/hmp-commands-info.hx
@@ -877,3 +877,18 @@ SRST
   ``info dirty_rate``
     Display the vcpu dirty rate information.
 ERST
+
+#if defined(TARGET_I386)
+    {
+        .name       = "sgx",
+        .args_type  = "",
+        .params     = "",
+        .help       = "show intel SGX information",
+        .cmd        = hmp_info_sgx,
+    },
+#endif
+
+SRST
+  ``info sgx``
+    Show intel SGX information.
+ERST
diff --git a/include/monitor/hmp.h b/include/monitor/hmp.h
index 3baa1058e2..eba5eda996 100644
--- a/include/monitor/hmp.h
+++ b/include/monitor/hmp.h
@@ -131,5 +131,6 @@ void hmp_replay_delete_break(Monitor *mon, const QDict *qdict);
 void hmp_replay_seek(Monitor *mon, const QDict *qdict);
 void hmp_info_dirty_rate(Monitor *mon, const QDict *qdict);
 void hmp_calc_dirty_rate(Monitor *mon, const QDict *qdict);
+void hmp_info_sgx(Monitor *mon, const QDict *qdict);
 
 #endif
diff --git a/monitor/hmp-cmds.c b/monitor/hmp-cmds.c
index 0942027208..9ebc86190f 100644
--- a/monitor/hmp-cmds.c
+++ b/monitor/hmp-cmds.c
@@ -2230,3 +2230,9 @@ void hmp_info_memory_size_summary(Monitor *mon, const QDict *qdict)
     }
     hmp_handle_error(mon, err);
 }
+
+void hmp_info_sgx(Monitor *mon, const QDict *qdict)
+{
+    error_setg(errp, QERR_FEATURE_DISABLED, "query-sgx");
+    return NULL;
+}


^ permalink raw reply related	[flat|nested] 61+ messages in thread

* [PATCH v4 28/33] i386: Add sgx_get_info() interface
  2021-07-19 11:21 [PATCH v4 00/33] Qemu SGX virtualization Yang Zhong
                   ` (26 preceding siblings ...)
  2021-07-19 11:21 ` [PATCH v4 27/33] hmp: Add 'info sgx' command Yang Zhong
@ 2021-07-19 11:21 ` Yang Zhong
  2021-07-19 11:21 ` [PATCH v4 29/33] bitops: Support 32 and 64 bit mask macro Yang Zhong
                   ` (7 subsequent siblings)
  35 siblings, 0 replies; 61+ messages in thread
From: Yang Zhong @ 2021-07-19 11:21 UTC (permalink / raw)
  To: qemu-devel; +Cc: yang.zhong, seanjc, kai.huang, jarkko, pbonzini, eblake

Add the sgx_get_info() interface for hmp and QMP usage, which
will get the SGX info from this API.

Signed-off-by: Yang Zhong <yang.zhong@intel.com>
---
 hw/i386/sgx.c             | 20 ++++++++++++++++++++
 include/hw/i386/pc.h      |  1 +
 include/hw/i386/sgx-epc.h |  1 +
 monitor/hmp-cmds.c        | 20 ++++++++++++++++++--
 monitor/qmp-cmds.c        | 12 ++++++++++--
 stubs/meson.build         |  1 +
 stubs/sgx-stub.c          |  7 +++++++
 7 files changed, 58 insertions(+), 4 deletions(-)
 create mode 100644 stubs/sgx-stub.c

diff --git a/hw/i386/sgx.c b/hw/i386/sgx.c
index 5f988c6368..beff7e3cf0 100644
--- a/hw/i386/sgx.c
+++ b/hw/i386/sgx.c
@@ -18,6 +18,26 @@
 #include "qapi/error.h"
 #include "exec/address-spaces.h"
 
+SGXInfo *sgx_get_info(void)
+{
+    SGXInfo *info = NULL;
+    MachineState *ms = MACHINE(qdev_get_machine());
+    X86MachineState *x86ms = X86_MACHINE(qdev_get_machine());
+
+    if (x86ms->sgx_epc_list) {
+        PCMachineState *pcms = PC_MACHINE(ms);
+        SGXEPCState *sgx_epc = &pcms->sgx_epc;
+        info = g_new0(SGXInfo, 1);
+
+        info->sgx = true;
+        info->sgx1 = true;
+        info->sgx2 = true;
+        info->flc = true;
+        info->section_size = sgx_epc->size;
+    }
+    return info;
+}
+
 int sgx_epc_get_section(int section_nr, uint64_t *addr, uint64_t *size)
 {
     PCMachineState *pcms = PC_MACHINE(qdev_get_machine());
diff --git a/include/hw/i386/pc.h b/include/hw/i386/pc.h
index c725dfef5b..de351ec48a 100644
--- a/include/hw/i386/pc.h
+++ b/include/hw/i386/pc.h
@@ -206,6 +206,7 @@ extern const size_t pc_compat_6_0_len;
 void pc_machine_init_sgx_epc(PCMachineState *pcms);
 void sgx_memory_backend_reset(HostMemoryBackend *backend, int fd,
                               Error **errp);
+SGXInfo *sgx_get_info(void);
 
 extern GlobalProperty pc_compat_5_2[];
 extern const size_t pc_compat_5_2_len;
diff --git a/include/hw/i386/sgx-epc.h b/include/hw/i386/sgx-epc.h
index 155e8fad3e..8f9c7becb9 100644
--- a/include/hw/i386/sgx-epc.h
+++ b/include/hw/i386/sgx-epc.h
@@ -13,6 +13,7 @@
 #define QEMU_SGX_EPC_H
 
 #include "sysemu/hostmem.h"
+#include "qapi/qapi-types-misc.h"
 
 #define TYPE_SGX_EPC "sgx-epc"
 #define SGX_EPC(obj) \
diff --git a/monitor/hmp-cmds.c b/monitor/hmp-cmds.c
index 9ebc86190f..053d210535 100644
--- a/monitor/hmp-cmds.c
+++ b/monitor/hmp-cmds.c
@@ -2233,6 +2233,22 @@ void hmp_info_memory_size_summary(Monitor *mon, const QDict *qdict)
 
 void hmp_info_sgx(Monitor *mon, const QDict *qdict)
 {
-    error_setg(errp, QERR_FEATURE_DISABLED, "query-sgx");
-    return NULL;
+    SGXInfo *info = qmp_query_sgx(NULL);
+
+    if (info && info->sgx) {
+        monitor_printf(mon, "SGX support: %s\n",
+                       info->sgx ? "enabled" : "disabled");
+        monitor_printf(mon, "SGX1 support: %s\n",
+                       info->sgx1 ? "enabled" : "disabled");
+        monitor_printf(mon, "SGX2 support: %s\n",
+                       info->sgx2 ? "enabled" : "disabled");
+        monitor_printf(mon, "FLC support: %s\n",
+                       info->flc ? "enabled" : "disabled");
+        monitor_printf(mon, "size: %" PRIu64 "\n",
+                       info->section_size);
+    } else {
+        monitor_printf(mon, "SGX is not enabled\n");
+    }
+
+    qapi_free_SGXInfo(info);
 }
diff --git a/monitor/qmp-cmds.c b/monitor/qmp-cmds.c
index d63d59149f..48f7708ffe 100644
--- a/monitor/qmp-cmds.c
+++ b/monitor/qmp-cmds.c
@@ -40,6 +40,7 @@
 #include "qapi/qmp/qerror.h"
 #include "hw/mem/memory-device.h"
 #include "hw/acpi/acpi_dev_interface.h"
+#include "hw/i386/pc.h"
 
 NameInfo *qmp_query_name(Error **errp)
 {
@@ -354,6 +355,13 @@ void qmp_display_reload(DisplayReloadOptions *arg, Error **errp)
 
 SGXInfo *qmp_query_sgx(Error **errp)
 {
-    error_setg(errp, QERR_FEATURE_DISABLED, "query-sgx");
-    return NULL;
+    SGXInfo *info;
+
+    info = sgx_get_info();
+    if (!info) {
+        error_setg(errp, "SGX features are not available");
+        return NULL;
+    }
+
+    return info;
 }
diff --git a/stubs/meson.build b/stubs/meson.build
index d3fa8646b3..d6788c3614 100644
--- a/stubs/meson.build
+++ b/stubs/meson.build
@@ -57,3 +57,4 @@ if have_system
 else
   stub_ss.add(files('qdev.c'))
 endif
+stub_ss.add(files('sgx-stub.c'))
diff --git a/stubs/sgx-stub.c b/stubs/sgx-stub.c
new file mode 100644
index 0000000000..c2b59a88fd
--- /dev/null
+++ b/stubs/sgx-stub.c
@@ -0,0 +1,7 @@
+#include "qemu/osdep.h"
+#include "hw/i386/pc.h"
+
+SGXInfo *sgx_get_info(void)
+{
+    return NULL;
+}


^ permalink raw reply related	[flat|nested] 61+ messages in thread

* [PATCH v4 29/33] bitops: Support 32 and 64 bit mask macro
  2021-07-19 11:21 [PATCH v4 00/33] Qemu SGX virtualization Yang Zhong
                   ` (27 preceding siblings ...)
  2021-07-19 11:21 ` [PATCH v4 28/33] i386: Add sgx_get_info() interface Yang Zhong
@ 2021-07-19 11:21 ` Yang Zhong
  2021-07-19 11:21 ` [PATCH v4 30/33] qmp: Add the qmp_query_sgx_capabilities() Yang Zhong
                   ` (6 subsequent siblings)
  35 siblings, 0 replies; 61+ messages in thread
From: Yang Zhong @ 2021-07-19 11:21 UTC (permalink / raw)
  To: qemu-devel; +Cc: yang.zhong, seanjc, kai.huang, jarkko, pbonzini, eblake

The Qemu should enable bit mask macro like Linux did in the
kernel, the GENMASK(h, l) and GENMASK_ULL(h, l) will set the
bit to 1 from l to h bit in the 32 bit or 64 bit long type.

Signed-off-by: Yang Zhong <yang.zhong@intel.com>
---
 include/qemu/bitops.h | 7 +++++++
 1 file changed, 7 insertions(+)

diff --git a/include/qemu/bitops.h b/include/qemu/bitops.h
index 110c56e099..130d746ba3 100644
--- a/include/qemu/bitops.h
+++ b/include/qemu/bitops.h
@@ -18,6 +18,7 @@
 
 #define BITS_PER_BYTE           CHAR_BIT
 #define BITS_PER_LONG           (sizeof (unsigned long) * BITS_PER_BYTE)
+#define BITS_PER_LONG_LONG       64
 
 #define BIT(nr)                 (1UL << (nr))
 #define BIT_ULL(nr)             (1ULL << (nr))
@@ -28,6 +29,12 @@
 #define MAKE_64BIT_MASK(shift, length) \
     (((~0ULL) >> (64 - (length))) << (shift))
 
+#define GENMASK(h, l) \
+    (((~0UL) << (l)) & (~0UL >> (BITS_PER_LONG - 1 - (h))))
+
+#define GENMASK_ULL(h, l) \
+    (((~0ULL) << (l)) & (~0ULL >> (BITS_PER_LONG_LONG - 1 - (h))))
+
 /**
  * set_bit - Set a bit in memory
  * @nr: the bit to set


^ permalink raw reply related	[flat|nested] 61+ messages in thread

* [PATCH v4 30/33] qmp: Add the qmp_query_sgx_capabilities()
  2021-07-19 11:21 [PATCH v4 00/33] Qemu SGX virtualization Yang Zhong
                   ` (28 preceding siblings ...)
  2021-07-19 11:21 ` [PATCH v4 29/33] bitops: Support 32 and 64 bit mask macro Yang Zhong
@ 2021-07-19 11:21 ` Yang Zhong
  2021-07-19 11:21 ` [PATCH v4 31/33] Kconfig: Add CONFIG_SGX support Yang Zhong
                   ` (5 subsequent siblings)
  35 siblings, 0 replies; 61+ messages in thread
From: Yang Zhong @ 2021-07-19 11:21 UTC (permalink / raw)
  To: qemu-devel; +Cc: yang.zhong, seanjc, kai.huang, jarkko, pbonzini, eblake

Libvirt can use qmp_query_sgx_capabilities() to get the host
sgx capabilities.

Signed-off-by: Yang Zhong <yang.zhong@intel.com>

v1-->v2:
   - Changed the blurb error and "Since: 5.1" to "Since: 6.1"(Eric Blake).
---
 hw/i386/sgx.c              | 66 ++++++++++++++++++++++++++++++++++++++
 include/hw/i386/pc.h       |  1 +
 monitor/qmp-cmds.c         |  5 +++
 qapi/misc.json             | 19 +++++++++++
 stubs/sgx-stub.c           |  5 +++
 tests/qtest/qmp-cmd-test.c |  1 +
 6 files changed, 97 insertions(+)

diff --git a/hw/i386/sgx.c b/hw/i386/sgx.c
index beff7e3cf0..993e885980 100644
--- a/hw/i386/sgx.c
+++ b/hw/i386/sgx.c
@@ -17,6 +17,72 @@
 #include "monitor/qdev.h"
 #include "qapi/error.h"
 #include "exec/address-spaces.h"
+#include "sysemu/hw_accel.h"
+
+#define SGX_MAX_EPC_SECTIONS            8
+#define SGX_CPUID_EPC_INVALID           0x0
+
+/* A valid EPC section. */
+#define SGX_CPUID_EPC_SECTION           0x1
+#define SGX_CPUID_EPC_MASK              GENMASK(3, 0)
+
+static uint64_t sgx_calc_section_metric(uint64_t low, uint64_t high)
+{
+    return (low & GENMASK_ULL(31, 12)) +
+           ((high & GENMASK_ULL(19, 0)) << 32);
+}
+
+static uint64_t sgx_calc_host_epc_section_size(void)
+{
+    uint32_t i, type;
+    uint32_t eax, ebx, ecx, edx;
+    uint64_t size = 0;
+
+    for (i = 0; i < SGX_MAX_EPC_SECTIONS; i++) {
+        host_cpuid(0x12, i + 2, &eax, &ebx, &ecx, &edx);
+
+        type = eax & SGX_CPUID_EPC_MASK;
+        if (type == SGX_CPUID_EPC_INVALID) {
+            break;
+        }
+
+        if (type != SGX_CPUID_EPC_SECTION) {
+            break;
+        }
+
+        size += sgx_calc_section_metric(ecx, edx);
+    }
+
+    return size;
+}
+
+SGXInfo *sgx_get_capabilities(Error **errp)
+{
+    SGXInfo *info = NULL;
+    uint32_t eax, ebx, ecx, edx;
+
+    int fd = qemu_open_old("/dev/sgx_vepc", O_RDWR);
+    if (fd < 0) {
+        error_setg(errp, "SGX is not enabled in KVM");
+        return NULL;
+    }
+
+    info = g_new0(SGXInfo, 1);
+    host_cpuid(0x7, 0, &eax, &ebx, &ecx, &edx);
+
+    info->sgx = ebx & (1U << 2) ? true : false;
+    info->flc = ecx & (1U << 30) ? true : false;
+
+    host_cpuid(0x12, 0, &eax, &ebx, &ecx, &edx);
+    info->sgx1 = eax & (1U << 0) ? true : false;
+    info->sgx2 = eax & (1U << 1) ? true : false;
+
+    info->section_size = sgx_calc_host_epc_section_size();
+
+    close(fd);
+
+    return info;
+}
 
 SGXInfo *sgx_get_info(void)
 {
diff --git a/include/hw/i386/pc.h b/include/hw/i386/pc.h
index de351ec48a..6f687e62bc 100644
--- a/include/hw/i386/pc.h
+++ b/include/hw/i386/pc.h
@@ -207,6 +207,7 @@ void pc_machine_init_sgx_epc(PCMachineState *pcms);
 void sgx_memory_backend_reset(HostMemoryBackend *backend, int fd,
                               Error **errp);
 SGXInfo *sgx_get_info(void);
+SGXInfo *sgx_get_capabilities(Error **errp);
 
 extern GlobalProperty pc_compat_5_2[];
 extern const size_t pc_compat_5_2_len;
diff --git a/monitor/qmp-cmds.c b/monitor/qmp-cmds.c
index 48f7708ffe..f1360e9f4e 100644
--- a/monitor/qmp-cmds.c
+++ b/monitor/qmp-cmds.c
@@ -365,3 +365,8 @@ SGXInfo *qmp_query_sgx(Error **errp)
 
     return info;
 }
+
+SGXInfo *qmp_query_sgx_capabilities(Error **errp)
+{
+    return sgx_get_capabilities(errp);
+}
diff --git a/qapi/misc.json b/qapi/misc.json
index 83b45a1460..8d1a78d138 100644
--- a/qapi/misc.json
+++ b/qapi/misc.json
@@ -561,3 +561,22 @@
 #
 ##
 { 'command': 'query-sgx', 'returns': 'SGXInfo' }
+
+
+##
+# @query-sgx-capabilities:
+#
+# Returns information from host SGX capabilities
+#
+# Returns: @SGXInfo
+#
+# Since: 6.1
+#
+# Example:
+#
+# -> { "execute": "query-sgx-capabilities" }
+# <- { "return": { "sgx": true, "sgx1" : true, "sgx2" : true,
+#                  "flc": true, "section-size" : 0 } }
+#
+##
+{ 'command': 'query-sgx-capabilities', 'returns': 'SGXInfo' }
diff --git a/stubs/sgx-stub.c b/stubs/sgx-stub.c
index c2b59a88fd..1dedf3f3db 100644
--- a/stubs/sgx-stub.c
+++ b/stubs/sgx-stub.c
@@ -5,3 +5,8 @@ SGXInfo *sgx_get_info(void)
 {
     return NULL;
 }
+
+SGXInfo *sgx_get_capabilities(Error **errp)
+{
+    return NULL;
+}
diff --git a/tests/qtest/qmp-cmd-test.c b/tests/qtest/qmp-cmd-test.c
index b75f3364f3..1af2f74c28 100644
--- a/tests/qtest/qmp-cmd-test.c
+++ b/tests/qtest/qmp-cmd-test.c
@@ -101,6 +101,7 @@ static bool query_is_ignored(const char *cmd)
         "query-sev",
         "query-sev-capabilities",
         "query-sgx",
+        "query-sgx-capabilities",
         NULL
     };
     int i;


^ permalink raw reply related	[flat|nested] 61+ messages in thread

* [PATCH v4 31/33] Kconfig: Add CONFIG_SGX support
  2021-07-19 11:21 [PATCH v4 00/33] Qemu SGX virtualization Yang Zhong
                   ` (29 preceding siblings ...)
  2021-07-19 11:21 ` [PATCH v4 30/33] qmp: Add the qmp_query_sgx_capabilities() Yang Zhong
@ 2021-07-19 11:21 ` Yang Zhong
  2021-07-19 11:21 ` [PATCH v4 32/33] sgx-epc: Add the fill_device_info() callback support Yang Zhong
                   ` (4 subsequent siblings)
  35 siblings, 0 replies; 61+ messages in thread
From: Yang Zhong @ 2021-07-19 11:21 UTC (permalink / raw)
  To: qemu-devel; +Cc: yang.zhong, seanjc, kai.huang, jarkko, pbonzini, eblake

Add new CONFIG_SGX for sgx support in the Qemu, and the Kconfig
default enable sgx in the i386 platform.

Signed-off-by: Yang Zhong <yang.zhong@intel.com>
---
 backends/meson.build                     |  2 +-
 configs/devices/i386-softmmu/default.mak |  1 +
 hw/i386/Kconfig                          |  5 +++++
 hw/i386/meson.build                      |  4 ++--
 hw/i386/sgx-stub.c                       | 13 +++++++++++++
 5 files changed, 22 insertions(+), 3 deletions(-)
 create mode 100644 hw/i386/sgx-stub.c

diff --git a/backends/meson.build b/backends/meson.build
index 46fd16b269..6e68945528 100644
--- a/backends/meson.build
+++ b/backends/meson.build
@@ -16,6 +16,6 @@ softmmu_ss.add(when: ['CONFIG_VHOST_USER', 'CONFIG_VIRTIO'], if_true: files('vho
 softmmu_ss.add(when: 'CONFIG_VIRTIO_CRYPTO', if_true: files('cryptodev-vhost.c'))
 softmmu_ss.add(when: ['CONFIG_VIRTIO_CRYPTO', 'CONFIG_VHOST_CRYPTO'], if_true: files('cryptodev-vhost-user.c'))
 softmmu_ss.add(when: 'CONFIG_GIO', if_true: [files('dbus-vmstate.c'), gio])
-softmmu_ss.add(when: 'CONFIG_LINUX', if_true: files('hostmem-epc.c'))
+softmmu_ss.add(when: 'CONFIG_SGX', if_true: files('hostmem-epc.c'))
 
 subdir('tpm')
diff --git a/configs/devices/i386-softmmu/default.mak b/configs/devices/i386-softmmu/default.mak
index 84d1a2487c..598c6646df 100644
--- a/configs/devices/i386-softmmu/default.mak
+++ b/configs/devices/i386-softmmu/default.mak
@@ -22,6 +22,7 @@
 #CONFIG_TPM_CRB=n
 #CONFIG_TPM_TIS_ISA=n
 #CONFIG_VTD=n
+#CONFIG_SGX=n
 
 # Boards:
 #
diff --git a/hw/i386/Kconfig b/hw/i386/Kconfig
index bad6cf5b4e..9072f9d030 100644
--- a/hw/i386/Kconfig
+++ b/hw/i386/Kconfig
@@ -6,6 +6,10 @@ config SEV
     select X86_FW_OVMF
     depends on KVM
 
+config SGX
+    bool
+    depends on KVM
+
 config PC
     bool
     imply APPLESMC
@@ -21,6 +25,7 @@ config PC
     imply PVPANIC_ISA
     imply QXL
     imply SEV
+    imply SGX
     imply SGA
     imply TEST_DEVICES
     imply TPM_CRB
diff --git a/hw/i386/meson.build b/hw/i386/meson.build
index fefce9e4ba..c502965219 100644
--- a/hw/i386/meson.build
+++ b/hw/i386/meson.build
@@ -5,8 +5,6 @@ i386_ss.add(files(
   'e820_memory_layout.c',
   'multiboot.c',
   'x86.c',
-  'sgx-epc.c',
-  'sgx.c'
 ))
 
 i386_ss.add(when: 'CONFIG_X86_IOMMU', if_true: files('x86-iommu.c'),
@@ -18,6 +16,8 @@ i386_ss.add(when: 'CONFIG_Q35', if_true: files('pc_q35.c'))
 i386_ss.add(when: 'CONFIG_VMMOUSE', if_true: files('vmmouse.c'))
 i386_ss.add(when: 'CONFIG_VMPORT', if_true: files('vmport.c'))
 i386_ss.add(when: 'CONFIG_VTD', if_true: files('intel_iommu.c'))
+i386_ss.add(when: 'CONFIG_SGX', if_true: files('sgx-epc.c','sgx.c'),
+                                if_false: files('sgx-stub.c'))
 
 i386_ss.add(when: 'CONFIG_ACPI', if_true: files('acpi-common.c'))
 i386_ss.add(when: 'CONFIG_ACPI_HW_REDUCED', if_true: files('generic_event_device_x86.c'))
diff --git a/hw/i386/sgx-stub.c b/hw/i386/sgx-stub.c
new file mode 100644
index 0000000000..edf17c3309
--- /dev/null
+++ b/hw/i386/sgx-stub.c
@@ -0,0 +1,13 @@
+#include "qemu/osdep.h"
+#include "hw/i386/pc.h"
+#include "hw/i386/sgx-epc.h"
+
+void pc_machine_init_sgx_epc(PCMachineState *pcms)
+{
+    return;
+}
+
+int sgx_epc_get_section(int section_nr, uint64_t *addr, uint64_t *size)
+{
+    return 1;
+}


^ permalink raw reply related	[flat|nested] 61+ messages in thread

* [PATCH v4 32/33] sgx-epc: Add the fill_device_info() callback support
  2021-07-19 11:21 [PATCH v4 00/33] Qemu SGX virtualization Yang Zhong
                   ` (30 preceding siblings ...)
  2021-07-19 11:21 ` [PATCH v4 31/33] Kconfig: Add CONFIG_SGX support Yang Zhong
@ 2021-07-19 11:21 ` Yang Zhong
  2021-07-19 11:21 ` [PATCH v4 33/33] doc: Add the SGX doc Yang Zhong
                   ` (3 subsequent siblings)
  35 siblings, 0 replies; 61+ messages in thread
From: Yang Zhong @ 2021-07-19 11:21 UTC (permalink / raw)
  To: qemu-devel; +Cc: yang.zhong, seanjc, kai.huang, jarkko, pbonzini, eblake

Since there is no fill_device_info() callback support, and when we
execute "info memory-devices" command in the monitor, the segfault
will be found.

This patch will add this callback support and "info memory-devices"
will show sgx epc memory exposed to guest. The result as below:

qemu) info memory-devices
Memory device [sgx-epc]: ""
  memaddr: 0x180000000
  size: 29360128
  memdev: /objects/mem1
Memory device [sgx-epc]: ""
  memaddr: 0x181c00000
  size: 10485760
  memdev: /objects/mem2

Signed-off-by: Yang Zhong <yang.zhong@intel.com>

v3-->v4:
    - Removed the epc device 'id' from compound property(Paolo).

v1-->v2:
    - "Since: 5.1" to "Since: 6.1"(Eric Blake).
---
 hw/i386/sgx-epc.c  | 11 ++++++++++-
 monitor/hmp-cmds.c | 10 ++++++++++
 qapi/machine.json  | 26 +++++++++++++++++++++++++-
 3 files changed, 45 insertions(+), 2 deletions(-)

diff --git a/hw/i386/sgx-epc.c b/hw/i386/sgx-epc.c
index 70075db37c..7fc3dd81b9 100644
--- a/hw/i386/sgx-epc.c
+++ b/hw/i386/sgx-epc.c
@@ -214,7 +214,16 @@ static MemoryRegion *sgx_epc_md_get_memory_region(MemoryDeviceState *md,
 static void sgx_epc_md_fill_device_info(const MemoryDeviceState *md,
                                         MemoryDeviceInfo *info)
 {
-    /* TODO */
+    SgxEPCDeviceInfo *se = g_new0(SgxEPCDeviceInfo, 1);
+    SGXEPCDevice *epc = SGX_EPC(md);
+
+    se->memaddr = epc->addr;
+    se->size = object_property_get_uint(OBJECT(epc), SGX_EPC_SIZE_PROP,
+                                        NULL);
+    se->memdev = object_get_canonical_path(OBJECT(epc->hostmem));
+
+    info->u.sgx_epc.data = se;
+    info->type = MEMORY_DEVICE_INFO_KIND_SGX_EPC;
 }
 
 static void sgx_epc_class_init(ObjectClass *oc, void *data)
diff --git a/monitor/hmp-cmds.c b/monitor/hmp-cmds.c
index 053d210535..7dfc98c380 100644
--- a/monitor/hmp-cmds.c
+++ b/monitor/hmp-cmds.c
@@ -1823,6 +1823,7 @@ void hmp_info_memory_devices(Monitor *mon, const QDict *qdict)
     VirtioMEMDeviceInfo *vmi;
     MemoryDeviceInfo *value;
     PCDIMMDeviceInfo *di;
+    SgxEPCDeviceInfo *se;
 
     for (info = info_list; info; info = info->next) {
         value = info->value;
@@ -1870,6 +1871,15 @@ void hmp_info_memory_devices(Monitor *mon, const QDict *qdict)
                                vmi->block_size);
                 monitor_printf(mon, "  memdev: %s\n", vmi->memdev);
                 break;
+            case MEMORY_DEVICE_INFO_KIND_SGX_EPC:
+                se = value->u.sgx_epc.data;
+                monitor_printf(mon, "Memory device [%s]: \"%s\"\n",
+                               MemoryDeviceInfoKind_str(value->type),
+                               se->id ? se->id : "");
+                monitor_printf(mon, "  memaddr: 0x%" PRIx64 "\n", se->memaddr);
+                monitor_printf(mon, "  size: %" PRIu64 "\n", se->size);
+                monitor_printf(mon, "  memdev: %s\n", se->memdev);
+                break;
             default:
                 g_assert_not_reached();
             }
diff --git a/qapi/machine.json b/qapi/machine.json
index 3bbe5b590e..b060635942 100644
--- a/qapi/machine.json
+++ b/qapi/machine.json
@@ -1220,6 +1220,29 @@
   'data': { 'sgx-epc': ['SgxEPC'] }
 }
 
+##
+# @SgxEPCDeviceInfo:
+#
+# Sgx EPC state information
+#
+# @id: device's ID
+#
+# @memaddr: physical address in memory, where device is mapped
+#
+# @size: size of memory that the device provides
+#
+# @memdev: memory backend linked with device
+#
+# Since: 6.1
+##
+{ 'struct': 'SgxEPCDeviceInfo',
+  'data': { '*id': 'str',
+            'memaddr': 'size',
+            'size': 'size',
+            'memdev': 'str'
+          }
+}
+
 ##
 # @MemoryDeviceInfo:
 #
@@ -1234,7 +1257,8 @@
   'data': { 'dimm': 'PCDIMMDeviceInfo',
             'nvdimm': 'PCDIMMDeviceInfo',
             'virtio-pmem': 'VirtioPMEMDeviceInfo',
-            'virtio-mem': 'VirtioMEMDeviceInfo'
+            'virtio-mem': 'VirtioMEMDeviceInfo',
+            'sgx-epc': 'SgxEPCDeviceInfo'
           }
 }
 


^ permalink raw reply related	[flat|nested] 61+ messages in thread

* [PATCH v4 33/33] doc: Add the SGX doc
  2021-07-19 11:21 [PATCH v4 00/33] Qemu SGX virtualization Yang Zhong
                   ` (31 preceding siblings ...)
  2021-07-19 11:21 ` [PATCH v4 32/33] sgx-epc: Add the fill_device_info() callback support Yang Zhong
@ 2021-07-19 11:21 ` Yang Zhong
  2021-07-28 15:57 ` [PATCH v4 00/33] Qemu SGX virtualization Paolo Bonzini
                   ` (2 subsequent siblings)
  35 siblings, 0 replies; 61+ messages in thread
From: Yang Zhong @ 2021-07-19 11:21 UTC (permalink / raw)
  To: qemu-devel; +Cc: yang.zhong, seanjc, kai.huang, jarkko, pbonzini, eblake

From: Sean Christopherson <sean.j.christopherson@intel.com>

Signed-off-by: Sean Christopherson <sean.j.christopherson@intel.com>
Signed-off-by: Yang Zhong <yang.zhong@intel.com>
---
 docs/intel-sgx.txt | 167 +++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 167 insertions(+)
 create mode 100644 docs/intel-sgx.txt

diff --git a/docs/intel-sgx.txt b/docs/intel-sgx.txt
new file mode 100644
index 0000000000..f934d92f37
--- /dev/null
+++ b/docs/intel-sgx.txt
@@ -0,0 +1,167 @@
+===============================
+Software Guard eXtensions (SGX)
+===============================
+
+Overview
+========
+
+Intel Software Guard eXtensions (SGX) is a set of instructions and mechanisms
+for memory accesses in order to provide security accesses for sensitive
+applications and data. SGX allows an application to use it's pariticular
+address space as an *enclave*, which is a protected area provides confidentiality
+and integrity even in the presence of privileged malware. Accesses to the
+enclave memory area from any software not resident in the enclave are prevented,
+including those from privileged software.
+
+Virtual SGX
+===========
+
+SGX feature is exposed to guest via SGX CPUID. Looking at SGX CPUID, we can
+report the same CPUID info to guest as on host for most of SGX CPUID. With
+reporting the same CPUID guest is able to use full capacity of SGX, and KVM
+doesn't need to emulate those info.
+
+The guest's EPC base and size are determined by Qemu, and KVM needs Qemu to
+notify such info to it before it can initialize SGX for guest.
+
+Virtual EPC
+-----------
+
+By default, Qemu does not assign EPC to a VM, i.e. fully enabling SGX in a VM
+requires explicit allocation of EPC to the VM. Similar to other specialized
+memory types, e.g. hugetlbfs, EPC is exposed as a memory backend.
+
+SGX EPC is enumerated through CPUID, i.e. EPC "devices" need to be realized
+prior to realizing the vCPUs themselves, which occurs long before generic
+devices are parsed and realized.  This limitation means that EPC does not
+require -maxmem as EPC is not treated as {cold,hot}plugged memory.
+
+Qemu does not artificially restrict the number of EPC sections exposed to a
+guest, e.g. Qemu will happily allow you to create 64 1M EPC sections. Be aware
+that some kernels may not recognize all EPC sections, e.g. the Linux SGX driver
+is hardwired to support only 8 EPC sections.
+
+The following Qemu snippet creates two EPC sections, with 64M pre-allocated
+to the VM and an additional 28M mapped but not allocated:
+
+ -object memory-backend-epc,id=mem1,size=64M,prealloc=on \
+ -object memory-backend-epc,id=mem2,size=28M \
+ -M sgx-epc.0.memdev=mem1,sgx-epc.1.memdev=mem2
+
+Note:
+
+The size and location of the virtual EPC are far less restricted compared
+to physical EPC. Because physical EPC is protected via range registers,
+the size of the physical EPC must be a power of two (though software sees
+a subset of the full EPC, e.g. 92M or 128M) and the EPC must be naturally
+aligned.  KVM SGX's virtual EPC is purely a software construct and only
+requires the size and location to be page aligned. Qemu enforces the EPC
+size is a multiple of 4k and will ensure the base of the EPC is 4k aligned.
+To simplify the implementation, EPC is always located above 4g in the guest
+physical address space.
+
+Migration
+---------
+
+Qemu/KVM doesn't prevent live migrating SGX VMs, although from hardware's
+perspective, SGX doesn't support live migration, since both EPC and the SGX
+key hierarchy are bound to the physical platform. However live migration
+can be supported in the sense if guest software stack can support recreating
+enclaves when it suffers sudden lose of EPC; and if guest enclaves can detect
+SGX keys being changed, and handle gracefully. For instance, when ERESUME fails
+with #PF.SGX, guest software can gracefully detect it and recreate enclaves;
+and when enclave fails to unseal sensitive information from outside, it can
+detect such error and sensitive information can be provisioned to it again.
+
+CPUID
+-----
+
+Due to its myriad dependencies, SGX is currently not listed as supported
+in any of Qemu's built-in CPU configuration. To expose SGX (and SGX Launch
+Control) to a guest, you must either use `-cpu host` to pass-through the
+host CPU model, or explicitly enable SGX when using a built-in CPU model,
+e.g. via `-cpu <model>,+sgx` or `-cpu <model>,+sgx,+sgxlc`.
+
+All SGX sub-features enumerated through CPUID, e.g. SGX2, MISCSELECT,
+ATTRIBUTES, etc... can be restricted via CPUID flags. Be aware that enforcing
+restriction of MISCSELECT, ATTRIBUTES and XFRM requires intercepting ECREATE,
+i.e. may marginally reduce SGX performance in the guest. All SGX sub-features
+controlled via -cpu are prefixed with "sgx", e.g.:
+
+$ qemu-system-x86_64 -cpu help | xargs printf "%s\n" | grep sgx
+  sgx
+  sgx-debug
+  sgx-encls-c
+  sgx-enclv
+  sgx-exinfo
+  sgx-kss
+  sgx-mode64
+  sgx-provisionkey
+  sgx-tokenkey
+  sgx1
+  sgx2
+  sgxlc
+
+The following Qemu snippet passes through the host CPU (and host physical
+address width) but restricts access to the provision and EINIT token keys:
+
+ -cpu host,host-phys-bits,-sgx-provisionkey,-sgx-tokenkey
+
+Note:
+
+SGX sub-features cannot be emulated, i.e. sub-features that are not present
+in hardware cannot be forced on via '-cpu'.
+
+Virtualize SGX Launch Control
+-----------------------------
+
+Qemu SGX support for Launch Control (LC) is passive, in the sense that it
+does not actively change the LC configuration.  Qemu SGX provides the user
+the ability to set/clear the CPUID flag (and by extension the associated
+IA32_FEATURE_CONTROL MSR bit in fw_cfg) and saves/restores the LE Hash MSRs
+when getting/putting guest state, but Qemu does not add new controls to
+directly modify the LC configuration.  Similar to hardware behavior, locking
+the LC configuration to a non-Intel value is left to guest firmware.  Unlike
+host bios setting for SGX launch control(LC), there is no special bios setting
+for SGX guest by our design. If host is in locked mode, we can still allow
+creating VM with SGX.
+
+Feature Control
+---------------
+
+Qemu SGX updates the `etc/msr_feature_control` fw_cfg entry to set the SGX
+(bit 18) and SGX LC (bit 17) flags based on their respective CPUID support,
+i.e. existing guest firmware will automatically set SGX and SGX LC accordingly,
+assuming said firmware supports fw_cfg.msr_feature_control.
+
+Launch a guest
+==============
+
+To launch a SGX guest
+${QEMU} \
+   -cpu host,+sgx-provisionkey \
+   -object memory-backend-epc,id=mem1,size=64M,prealloc=on \
+   -object memory-backend-epc,id=mem2,size=28M \
+   -M sgx-epc.0.memdev=mem1,sgx-epc.1.memdev=mem2
+
+Utilizing SGX in the guest requires a kernel/OS with SGX support.
+
+The support can be determined in guest by:
+$ grep sgx /proc/cpuinfo
+
+Check the SGX epc info in the Guest:
+$ dmesg | grep sgx
+[    1.242142] sgx: EPC section 0x180000000-0x181bfffff
+[    1.242319] sgx: EPC section 0x181c00000-0x1837fffff
+
+References
+==========
+
+SGX Homepage:
+https://software.intel.com/sgx
+
+SGX SDK:
+https://github.com/intel/linux-sgx.git
+
+SGX SPEC:
+Intel SDM Volume 3


^ permalink raw reply related	[flat|nested] 61+ messages in thread

* Re: [PATCH v4 00/33] Qemu SGX virtualization
  2021-07-19 11:21 [PATCH v4 00/33] Qemu SGX virtualization Yang Zhong
                   ` (32 preceding siblings ...)
  2021-07-19 11:21 ` [PATCH v4 33/33] doc: Add the SGX doc Yang Zhong
@ 2021-07-28 15:57 ` Paolo Bonzini
  2021-07-29 12:27   ` Yang Zhong
  2021-09-06 13:13 ` Paolo Bonzini
  2021-09-14  6:51 ` Philippe Mathieu-Daudé
  35 siblings, 1 reply; 61+ messages in thread
From: Paolo Bonzini @ 2021-07-28 15:57 UTC (permalink / raw)
  To: Yang Zhong, qemu-devel; +Cc: seanjc, kai.huang, eblake, jarkko

On 19/07/21 13:21, Yang Zhong wrote:
> Since Sean Christopherson has left Intel and i am responsible for Qemu SGX
> upstream work. His @intel.com address will be bouncing and his new email(
> seanjc@google.com) is also in CC lists.
> 
> This series is Qemu SGX virtualization implementation rebased on latest
> Qemu release. The numa support for SGX will be sent in another patchset
> once this basic SGX patchset are merged.
> 
> You can find Qemu repo here:
> 
>      https://github.com/intel/qemu-sgx.git upstream
> 
> If you want to try SGX, you can directly install the linux release(at least 5.13.0-rc1+)
> since kvm SGX has been merged into linux release.
> 
>      https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git
> 
> To simplify, you'd better install linux on host and guest, which can support
> SGX on host and guest kernel. And to me, use below reference command to boot
> SGX guest:
> 
>      #qemu-system-x86_64 \
>          ...... \
>          -cpu host,+sgx-provisionkey \
>          -object memory-backend-epc,id=mem1,size=64M,prealloc=on \
>          -object memory-backend-epc,id=mem2,size=28M \
>          -M sgx-epc.0.memdev=mem1,sgx-epc.1.memdev=mem2
> 
> Overview
> ========
> 
> Intel Software Guard eXtensions (SGX) is a set of instructions and mechanisms
> for memory accesses in order to provide security accesses for sensitive
> applications and data. SGX allows an application to use it's pariticular
> address space as an *enclave*, which is a protected area provides confidentiality
> and integrity even in the presence of privileged malware. Accesses to the
> enclave memory area from any software not resident in the enclave are prevented,
> including those from privileged software.
> 
> SGX virtaulization
> ==================
> 
> The KVM SGX creates one new misc device, sgx_vepc, and Qemu will open '/dev/sgx_vepc'
> device node to mmap() host EPC memory to guest. The Qemu also adds 'sgx-epc' device
> to expose EPC sections to guest through CPUID and ACPI table.  The Qemu SGX also
> supports multiple virtual EPC sections to guest, we just put them together physically
> contiguous for the sake of simplicity. The kernel SGX NUMA has been merged into Linux
> tip tree, we will support this function in the next phase.
> 
> Although the current host SGX subsystem can not support SGX2 feature, the KVM/Qemu
> implementation still expose this feature to guest. Guest SGX2 support doesn't have
> interaction with host kernel SGX driver, the SGX guest can normally use those new
> instructions.
> 
> As for SGX virtualization detailed infomation, please reference docs/intel-sgx.txt
> docuement(patch 33).
> 
> Changelog:
> =========
> 
> (Changelog here is for global changes, please see each patch's changelog for changes
> made to specific patch.)
> 
> v3-->v4:
>     - Rebased the sgx patches into latest Qemu release.
>     - Moved sgx compound property setter/getter from MachineState to X86MachineState(Paolo).
>     - Re-defined struct SgxEPC, removed 'id' property and added struct SgxEPCList for
>       sgx-epc.0.{memdev}(Paolo).
>     - Removed g_malloc0(), and changed the 'SGXEPCState *sgx_epc' to 'SGXEPCState sgx_epc'
>       in struct PCMachineState(Paolo).
>     - Changed the SGX compound property cmdline from sgx-epc.{memdev}.0 to
>       sgx-epc.0.{memdev}(Paolo).
>     - Removed the signature from the 'git format-patch' command(Jarkko).
> 
> v2-->v3:
>     - Rebased the sgx patches into latest Qemu release.
>     - Implemented the compound property for SGX, ref patch5, the command from '-sgx-epc'
>       to '-M'(Paolo).
>     - Moved the sgx common code from sgx-epc.c to sgx.c. The sgx-epc.c is
>       only responsible for virtual epc device.
>     - Removed the previous patch13(linux-headers: Add placeholder for KVM_CAP_SGX_ATTRIBUTE)
>       because ehabkost@redhat.com updated Linux headers to 5.13-rc4 with commit 278f064e452.
>     - Updated the patch1 because ram_flags were changed by David Hildenbra.
>     - Added one patch24, which avoid reset operation caused by bios reset.
>     - Added one patch25, which make prealloc property consistent with Qemu cmdline during VM
>       reset.
> 
> v1-->v2:
>     - Rebased the sgx patches into latest Qemu release.
>     - Unified the "share" and "protected" arguments with ram_flags in the
>       memory_region_init_ram_from_fd()(Paolo).
>     - Added the new MemoryBackendEpcProperties and related documents(Eric Blake).
>     - Changed the KVM_CAP_SGX_ATTRIBUTE from 195 to 196(Kai).
>     - Changed the version and some grammar issues(Eric Blake).

Looks good, I will queue it for 6.2.

Thanks for your patience with the compound machine properties support.

Paolo

> Sean Christopherson (21):
>    memory: Add RAM_PROTECTED flag to skip IOMMU mappings
>    hostmem: Add hostmem-epc as a backend for SGX EPC
>    i386: Add 'sgx-epc' device to expose EPC sections to guest
>    vl: Add sgx compound properties to expose SGX EPC sections to guest
>    i386: Add primary SGX CPUID and MSR defines
>    i386: Add SGX CPUID leaf FEAT_SGX_12_0_EAX
>    i386: Add SGX CPUID leaf FEAT_SGX_12_0_EBX
>    i386: Add SGX CPUID leaf FEAT_SGX_12_1_EAX
>    i386: Add get/set/migrate support for SGX_LEPUBKEYHASH MSRs
>    i386: Add feature control MSR dependency when SGX is enabled
>    i386: Update SGX CPUID info according to hardware/KVM/user input
>    i386: kvm: Add support for exposing PROVISIONKEY to guest
>    i386: Propagate SGX CPUID sub-leafs to KVM
>    Adjust min CPUID level to 0x12 when SGX is enabled
>    hw/i386/fw_cfg: Set SGX bits in feature control fw_cfg accordingly
>    hw/i386/pc: Account for SGX EPC sections when calculating device
>      memory
>    i386/pc: Add e820 entry for SGX EPC section(s)
>    i386: acpi: Add SGX EPC entry to ACPI tables
>    q35: Add support for SGX EPC
>    i440fx: Add support for SGX EPC
>    doc: Add the SGX doc
> 
> Yang Zhong (12):
>    qom: Add memory-backend-epc ObjectOptions support
>    hostmem-epc: Add the reset interface for EPC backend reset
>    sgx-epc: Add the reset interface for sgx-epc virt device
>    sgx-epc: Avoid bios reset during sgx epc initialization
>    hostmem-epc: Make prealloc consistent with qemu cmdline during reset
>    qmp: Add query-sgx command
>    hmp: Add 'info sgx' command
>    i386: Add sgx_get_info() interface
>    bitops: Support 32 and 64 bit mask macro
>    qmp: Add the qmp_query_sgx_capabilities()
>    Kconfig: Add CONFIG_SGX support
>    sgx-epc: Add the fill_device_info() callback support
> 
>   backends/hostmem-epc.c                   | 118 ++++++++++
>   backends/meson.build                     |   1 +
>   configs/devices/i386-softmmu/default.mak |   1 +
>   docs/intel-sgx.txt                       | 167 ++++++++++++++
>   hmp-commands-info.hx                     |  15 ++
>   hw/i386/Kconfig                          |   5 +
>   hw/i386/acpi-build.c                     |  22 ++
>   hw/i386/fw_cfg.c                         |  10 +-
>   hw/i386/meson.build                      |   2 +
>   hw/i386/pc.c                             |  15 +-
>   hw/i386/pc_piix.c                        |   4 +
>   hw/i386/pc_q35.c                         |   3 +
>   hw/i386/sgx-epc.c                        | 265 +++++++++++++++++++++++
>   hw/i386/sgx-stub.c                       |  13 ++
>   hw/i386/sgx.c                            | 170 +++++++++++++++
>   hw/i386/x86.c                            |  29 +++
>   hw/vfio/common.c                         |   1 +
>   include/exec/memory.h                    |  15 +-
>   include/hw/i386/pc.h                     |  10 +
>   include/hw/i386/sgx-epc.h                |  68 ++++++
>   include/hw/i386/x86.h                    |   1 +
>   include/monitor/hmp.h                    |   1 +
>   include/qemu/bitops.h                    |   7 +
>   monitor/hmp-cmds.c                       |  32 +++
>   monitor/qmp-cmds.c                       |  19 ++
>   qapi/machine.json                        |  52 ++++-
>   qapi/misc.json                           |  61 ++++++
>   qapi/qom.json                            |  19 ++
>   qemu-options.hx                          |  10 +-
>   softmmu/memory.c                         |   5 +
>   softmmu/physmem.c                        |   3 +-
>   stubs/meson.build                        |   1 +
>   stubs/sgx-stub.c                         |  12 +
>   target/i386/cpu.c                        | 168 +++++++++++++-
>   target/i386/cpu.h                        |  16 ++
>   target/i386/kvm/kvm.c                    |  75 +++++++
>   target/i386/kvm/kvm_i386.h               |   2 +
>   target/i386/machine.c                    |  20 ++
>   tests/qtest/qmp-cmd-test.c               |   2 +
>   39 files changed, 1430 insertions(+), 10 deletions(-)
>   create mode 100644 backends/hostmem-epc.c
>   create mode 100644 docs/intel-sgx.txt
>   create mode 100644 hw/i386/sgx-epc.c
>   create mode 100644 hw/i386/sgx-stub.c
>   create mode 100644 hw/i386/sgx.c
>   create mode 100644 include/hw/i386/sgx-epc.h
>   create mode 100644 stubs/sgx-stub.c
> 
> 



^ permalink raw reply	[flat|nested] 61+ messages in thread

* Re: [PATCH v4 00/33] Qemu SGX virtualization
  2021-07-28 15:57 ` [PATCH v4 00/33] Qemu SGX virtualization Paolo Bonzini
@ 2021-07-29 12:27   ` Yang Zhong
  0 siblings, 0 replies; 61+ messages in thread
From: Yang Zhong @ 2021-07-29 12:27 UTC (permalink / raw)
  To: Paolo Bonzini; +Cc: yang.zhong, seanjc, kai.huang, qemu-devel, jarkko, eblake

On Wed, Jul 28, 2021 at 05:57:51PM +0200, Paolo Bonzini wrote:
> On 19/07/21 13:21, Yang Zhong wrote:
> >Since Sean Christopherson has left Intel and i am responsible for Qemu SGX
> >upstream work. His @intel.com address will be bouncing and his new email(
> >seanjc@google.com) is also in CC lists.
> >
> >This series is Qemu SGX virtualization implementation rebased on latest
> >Qemu release. The numa support for SGX will be sent in another patchset
> >once this basic SGX patchset are merged.
> >
> >You can find Qemu repo here:
> >
> >     https://github.com/intel/qemu-sgx.git upstream
> >
> >If you want to try SGX, you can directly install the linux release(at least 5.13.0-rc1+)
> >since kvm SGX has been merged into linux release.
> >
> >     https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git
> >
> >To simplify, you'd better install linux on host and guest, which can support
> >SGX on host and guest kernel. And to me, use below reference command to boot
> >SGX guest:
> >
> >     #qemu-system-x86_64 \
> >         ...... \
> >         -cpu host,+sgx-provisionkey \
> >         -object memory-backend-epc,id=mem1,size=64M,prealloc=on \
> >         -object memory-backend-epc,id=mem2,size=28M \
> >         -M sgx-epc.0.memdev=mem1,sgx-epc.1.memdev=mem2
> >
> >Overview
> >========
> >
> >Intel Software Guard eXtensions (SGX) is a set of instructions and mechanisms
> >for memory accesses in order to provide security accesses for sensitive
> >applications and data. SGX allows an application to use it's pariticular
> >address space as an *enclave*, which is a protected area provides confidentiality
> >and integrity even in the presence of privileged malware. Accesses to the
> >enclave memory area from any software not resident in the enclave are prevented,
> >including those from privileged software.
> >
> >SGX virtaulization
> >==================
> >
> >The KVM SGX creates one new misc device, sgx_vepc, and Qemu will open '/dev/sgx_vepc'
> >device node to mmap() host EPC memory to guest. The Qemu also adds 'sgx-epc' device
> >to expose EPC sections to guest through CPUID and ACPI table.  The Qemu SGX also
> >supports multiple virtual EPC sections to guest, we just put them together physically
> >contiguous for the sake of simplicity. The kernel SGX NUMA has been merged into Linux
> >tip tree, we will support this function in the next phase.
> >
> >Although the current host SGX subsystem can not support SGX2 feature, the KVM/Qemu
> >implementation still expose this feature to guest. Guest SGX2 support doesn't have
> >interaction with host kernel SGX driver, the SGX guest can normally use those new
> >instructions.
> >
> >As for SGX virtualization detailed infomation, please reference docs/intel-sgx.txt
> >docuement(patch 33).
> >
> >Changelog:
> >=========
> >
> >(Changelog here is for global changes, please see each patch's changelog for changes
> >made to specific patch.)
> >
> >v3-->v4:
> >    - Rebased the sgx patches into latest Qemu release.
> >    - Moved sgx compound property setter/getter from MachineState to X86MachineState(Paolo).
> >    - Re-defined struct SgxEPC, removed 'id' property and added struct SgxEPCList for
> >      sgx-epc.0.{memdev}(Paolo).
> >    - Removed g_malloc0(), and changed the 'SGXEPCState *sgx_epc' to 'SGXEPCState sgx_epc'
> >      in struct PCMachineState(Paolo).
> >    - Changed the SGX compound property cmdline from sgx-epc.{memdev}.0 to
> >      sgx-epc.0.{memdev}(Paolo).
> >    - Removed the signature from the 'git format-patch' command(Jarkko).
> >
> >v2-->v3:
> >    - Rebased the sgx patches into latest Qemu release.
> >    - Implemented the compound property for SGX, ref patch5, the command from '-sgx-epc'
> >      to '-M'(Paolo).
> >    - Moved the sgx common code from sgx-epc.c to sgx.c. The sgx-epc.c is
> >      only responsible for virtual epc device.
> >    - Removed the previous patch13(linux-headers: Add placeholder for KVM_CAP_SGX_ATTRIBUTE)
> >      because ehabkost@redhat.com updated Linux headers to 5.13-rc4 with commit 278f064e452.
> >    - Updated the patch1 because ram_flags were changed by David Hildenbra.
> >    - Added one patch24, which avoid reset operation caused by bios reset.
> >    - Added one patch25, which make prealloc property consistent with Qemu cmdline during VM
> >      reset.
> >
> >v1-->v2:
> >    - Rebased the sgx patches into latest Qemu release.
> >    - Unified the "share" and "protected" arguments with ram_flags in the
> >      memory_region_init_ram_from_fd()(Paolo).
> >    - Added the new MemoryBackendEpcProperties and related documents(Eric Blake).
> >    - Changed the KVM_CAP_SGX_ATTRIBUTE from 195 to 196(Kai).
> >    - Changed the version and some grammar issues(Eric Blake).
> 
> Looks good, I will queue it for 6.2.
> 
> Thanks for your patience with the compound machine properties support.
>

  Paolo, thanks! Once this patchset is merged, i will send SGX NUMA patchset, thanks!

  Regards,

  Yang


 
> Paolo
> 
> >Sean Christopherson (21):
> >   memory: Add RAM_PROTECTED flag to skip IOMMU mappings
> >   hostmem: Add hostmem-epc as a backend for SGX EPC
> >   i386: Add 'sgx-epc' device to expose EPC sections to guest
> >   vl: Add sgx compound properties to expose SGX EPC sections to guest
> >   i386: Add primary SGX CPUID and MSR defines
> >   i386: Add SGX CPUID leaf FEAT_SGX_12_0_EAX
> >   i386: Add SGX CPUID leaf FEAT_SGX_12_0_EBX
> >   i386: Add SGX CPUID leaf FEAT_SGX_12_1_EAX
> >   i386: Add get/set/migrate support for SGX_LEPUBKEYHASH MSRs
> >   i386: Add feature control MSR dependency when SGX is enabled
> >   i386: Update SGX CPUID info according to hardware/KVM/user input
> >   i386: kvm: Add support for exposing PROVISIONKEY to guest
> >   i386: Propagate SGX CPUID sub-leafs to KVM
> >   Adjust min CPUID level to 0x12 when SGX is enabled
> >   hw/i386/fw_cfg: Set SGX bits in feature control fw_cfg accordingly
> >   hw/i386/pc: Account for SGX EPC sections when calculating device
> >     memory
> >   i386/pc: Add e820 entry for SGX EPC section(s)
> >   i386: acpi: Add SGX EPC entry to ACPI tables
> >   q35: Add support for SGX EPC
> >   i440fx: Add support for SGX EPC
> >   doc: Add the SGX doc
> >
> >Yang Zhong (12):
> >   qom: Add memory-backend-epc ObjectOptions support
> >   hostmem-epc: Add the reset interface for EPC backend reset
> >   sgx-epc: Add the reset interface for sgx-epc virt device
> >   sgx-epc: Avoid bios reset during sgx epc initialization
> >   hostmem-epc: Make prealloc consistent with qemu cmdline during reset
> >   qmp: Add query-sgx command
> >   hmp: Add 'info sgx' command
> >   i386: Add sgx_get_info() interface
> >   bitops: Support 32 and 64 bit mask macro
> >   qmp: Add the qmp_query_sgx_capabilities()
> >   Kconfig: Add CONFIG_SGX support
> >   sgx-epc: Add the fill_device_info() callback support
> >
> >  backends/hostmem-epc.c                   | 118 ++++++++++
> >  backends/meson.build                     |   1 +
> >  configs/devices/i386-softmmu/default.mak |   1 +
> >  docs/intel-sgx.txt                       | 167 ++++++++++++++
> >  hmp-commands-info.hx                     |  15 ++
> >  hw/i386/Kconfig                          |   5 +
> >  hw/i386/acpi-build.c                     |  22 ++
> >  hw/i386/fw_cfg.c                         |  10 +-
> >  hw/i386/meson.build                      |   2 +
> >  hw/i386/pc.c                             |  15 +-
> >  hw/i386/pc_piix.c                        |   4 +
> >  hw/i386/pc_q35.c                         |   3 +
> >  hw/i386/sgx-epc.c                        | 265 +++++++++++++++++++++++
> >  hw/i386/sgx-stub.c                       |  13 ++
> >  hw/i386/sgx.c                            | 170 +++++++++++++++
> >  hw/i386/x86.c                            |  29 +++
> >  hw/vfio/common.c                         |   1 +
> >  include/exec/memory.h                    |  15 +-
> >  include/hw/i386/pc.h                     |  10 +
> >  include/hw/i386/sgx-epc.h                |  68 ++++++
> >  include/hw/i386/x86.h                    |   1 +
> >  include/monitor/hmp.h                    |   1 +
> >  include/qemu/bitops.h                    |   7 +
> >  monitor/hmp-cmds.c                       |  32 +++
> >  monitor/qmp-cmds.c                       |  19 ++
> >  qapi/machine.json                        |  52 ++++-
> >  qapi/misc.json                           |  61 ++++++
> >  qapi/qom.json                            |  19 ++
> >  qemu-options.hx                          |  10 +-
> >  softmmu/memory.c                         |   5 +
> >  softmmu/physmem.c                        |   3 +-
> >  stubs/meson.build                        |   1 +
> >  stubs/sgx-stub.c                         |  12 +
> >  target/i386/cpu.c                        | 168 +++++++++++++-
> >  target/i386/cpu.h                        |  16 ++
> >  target/i386/kvm/kvm.c                    |  75 +++++++
> >  target/i386/kvm/kvm_i386.h               |   2 +
> >  target/i386/machine.c                    |  20 ++
> >  tests/qtest/qmp-cmd-test.c               |   2 +
> >  39 files changed, 1430 insertions(+), 10 deletions(-)
> >  create mode 100644 backends/hostmem-epc.c
> >  create mode 100644 docs/intel-sgx.txt
> >  create mode 100644 hw/i386/sgx-epc.c
> >  create mode 100644 hw/i386/sgx-stub.c
> >  create mode 100644 hw/i386/sgx.c
> >  create mode 100644 include/hw/i386/sgx-epc.h
> >  create mode 100644 stubs/sgx-stub.c
> >
> >


^ permalink raw reply	[flat|nested] 61+ messages in thread

* Re: [PATCH v4 00/33] Qemu SGX virtualization
  2021-07-19 11:21 [PATCH v4 00/33] Qemu SGX virtualization Yang Zhong
                   ` (33 preceding siblings ...)
  2021-07-28 15:57 ` [PATCH v4 00/33] Qemu SGX virtualization Paolo Bonzini
@ 2021-09-06 13:13 ` Paolo Bonzini
  2021-09-07  2:24   ` Yang Zhong
  2021-09-07  9:51   ` Yang Zhong
  2021-09-14  6:51 ` Philippe Mathieu-Daudé
  35 siblings, 2 replies; 61+ messages in thread
From: Paolo Bonzini @ 2021-09-06 13:13 UTC (permalink / raw)
  To: Yang Zhong
  Cc: Sean Christopherson, Jarkko Sakkinen, Blake, Eric, qemu-devel,
	Huang, Kai

Hi,

the monitor patches did not pass the test-hmp qtest, and also they
should be in target/i386/monitor.c (see other commands that were
implemented there for SEV).  However, I've sent a pull request with
the rest.

Thanks,

Paolo

On Mon, Jul 19, 2021 at 1:27 PM Yang Zhong <yang.zhong@intel.com> wrote:
>
> Since Sean Christopherson has left Intel and i am responsible for Qemu SGX
> upstream work. His @intel.com address will be bouncing and his new email(
> seanjc@google.com) is also in CC lists.
>
> This series is Qemu SGX virtualization implementation rebased on latest
> Qemu release. The numa support for SGX will be sent in another patchset
> once this basic SGX patchset are merged.
>
> You can find Qemu repo here:
>
>     https://github.com/intel/qemu-sgx.git upstream
>
> If you want to try SGX, you can directly install the linux release(at least 5.13.0-rc1+)
> since kvm SGX has been merged into linux release.
>
>     https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git
>
> To simplify, you'd better install linux on host and guest, which can support
> SGX on host and guest kernel. And to me, use below reference command to boot
> SGX guest:
>
>     #qemu-system-x86_64 \
>         ...... \
>         -cpu host,+sgx-provisionkey \
>         -object memory-backend-epc,id=mem1,size=64M,prealloc=on \
>         -object memory-backend-epc,id=mem2,size=28M \
>         -M sgx-epc.0.memdev=mem1,sgx-epc.1.memdev=mem2
>
> Overview
> ========
>
> Intel Software Guard eXtensions (SGX) is a set of instructions and mechanisms
> for memory accesses in order to provide security accesses for sensitive
> applications and data. SGX allows an application to use it's pariticular
> address space as an *enclave*, which is a protected area provides confidentiality
> and integrity even in the presence of privileged malware. Accesses to the
> enclave memory area from any software not resident in the enclave are prevented,
> including those from privileged software.
>
> SGX virtaulization
> ==================
>
> The KVM SGX creates one new misc device, sgx_vepc, and Qemu will open '/dev/sgx_vepc'
> device node to mmap() host EPC memory to guest. The Qemu also adds 'sgx-epc' device
> to expose EPC sections to guest through CPUID and ACPI table.  The Qemu SGX also
> supports multiple virtual EPC sections to guest, we just put them together physically
> contiguous for the sake of simplicity. The kernel SGX NUMA has been merged into Linux
> tip tree, we will support this function in the next phase.
>
> Although the current host SGX subsystem can not support SGX2 feature, the KVM/Qemu
> implementation still expose this feature to guest. Guest SGX2 support doesn't have
> interaction with host kernel SGX driver, the SGX guest can normally use those new
> instructions.
>
> As for SGX virtualization detailed infomation, please reference docs/intel-sgx.txt
> docuement(patch 33).
>
> Changelog:
> =========
>
> (Changelog here is for global changes, please see each patch's changelog for changes
> made to specific patch.)
>
> v3-->v4:
>    - Rebased the sgx patches into latest Qemu release.
>    - Moved sgx compound property setter/getter from MachineState to X86MachineState(Paolo).
>    - Re-defined struct SgxEPC, removed 'id' property and added struct SgxEPCList for
>      sgx-epc.0.{memdev}(Paolo).
>    - Removed g_malloc0(), and changed the 'SGXEPCState *sgx_epc' to 'SGXEPCState sgx_epc'
>      in struct PCMachineState(Paolo).
>    - Changed the SGX compound property cmdline from sgx-epc.{memdev}.0 to
>      sgx-epc.0.{memdev}(Paolo).
>    - Removed the signature from the 'git format-patch' command(Jarkko).
>
> v2-->v3:
>    - Rebased the sgx patches into latest Qemu release.
>    - Implemented the compound property for SGX, ref patch5, the command from '-sgx-epc'
>      to '-M'(Paolo).
>    - Moved the sgx common code from sgx-epc.c to sgx.c. The sgx-epc.c is
>      only responsible for virtual epc device.
>    - Removed the previous patch13(linux-headers: Add placeholder for KVM_CAP_SGX_ATTRIBUTE)
>      because ehabkost@redhat.com updated Linux headers to 5.13-rc4 with commit 278f064e452.
>    - Updated the patch1 because ram_flags were changed by David Hildenbra.
>    - Added one patch24, which avoid reset operation caused by bios reset.
>    - Added one patch25, which make prealloc property consistent with Qemu cmdline during VM
>      reset.
>
> v1-->v2:
>    - Rebased the sgx patches into latest Qemu release.
>    - Unified the "share" and "protected" arguments with ram_flags in the
>      memory_region_init_ram_from_fd()(Paolo).
>    - Added the new MemoryBackendEpcProperties and related documents(Eric Blake).
>    - Changed the KVM_CAP_SGX_ATTRIBUTE from 195 to 196(Kai).
>    - Changed the version and some grammar issues(Eric Blake).
>
> Sean Christopherson (21):
>   memory: Add RAM_PROTECTED flag to skip IOMMU mappings
>   hostmem: Add hostmem-epc as a backend for SGX EPC
>   i386: Add 'sgx-epc' device to expose EPC sections to guest
>   vl: Add sgx compound properties to expose SGX EPC sections to guest
>   i386: Add primary SGX CPUID and MSR defines
>   i386: Add SGX CPUID leaf FEAT_SGX_12_0_EAX
>   i386: Add SGX CPUID leaf FEAT_SGX_12_0_EBX
>   i386: Add SGX CPUID leaf FEAT_SGX_12_1_EAX
>   i386: Add get/set/migrate support for SGX_LEPUBKEYHASH MSRs
>   i386: Add feature control MSR dependency when SGX is enabled
>   i386: Update SGX CPUID info according to hardware/KVM/user input
>   i386: kvm: Add support for exposing PROVISIONKEY to guest
>   i386: Propagate SGX CPUID sub-leafs to KVM
>   Adjust min CPUID level to 0x12 when SGX is enabled
>   hw/i386/fw_cfg: Set SGX bits in feature control fw_cfg accordingly
>   hw/i386/pc: Account for SGX EPC sections when calculating device
>     memory
>   i386/pc: Add e820 entry for SGX EPC section(s)
>   i386: acpi: Add SGX EPC entry to ACPI tables
>   q35: Add support for SGX EPC
>   i440fx: Add support for SGX EPC
>   doc: Add the SGX doc
>
> Yang Zhong (12):
>   qom: Add memory-backend-epc ObjectOptions support
>   hostmem-epc: Add the reset interface for EPC backend reset
>   sgx-epc: Add the reset interface for sgx-epc virt device
>   sgx-epc: Avoid bios reset during sgx epc initialization
>   hostmem-epc: Make prealloc consistent with qemu cmdline during reset
>   qmp: Add query-sgx command
>   hmp: Add 'info sgx' command
>   i386: Add sgx_get_info() interface
>   bitops: Support 32 and 64 bit mask macro
>   qmp: Add the qmp_query_sgx_capabilities()
>   Kconfig: Add CONFIG_SGX support
>   sgx-epc: Add the fill_device_info() callback support
>
>  backends/hostmem-epc.c                   | 118 ++++++++++
>  backends/meson.build                     |   1 +
>  configs/devices/i386-softmmu/default.mak |   1 +
>  docs/intel-sgx.txt                       | 167 ++++++++++++++
>  hmp-commands-info.hx                     |  15 ++
>  hw/i386/Kconfig                          |   5 +
>  hw/i386/acpi-build.c                     |  22 ++
>  hw/i386/fw_cfg.c                         |  10 +-
>  hw/i386/meson.build                      |   2 +
>  hw/i386/pc.c                             |  15 +-
>  hw/i386/pc_piix.c                        |   4 +
>  hw/i386/pc_q35.c                         |   3 +
>  hw/i386/sgx-epc.c                        | 265 +++++++++++++++++++++++
>  hw/i386/sgx-stub.c                       |  13 ++
>  hw/i386/sgx.c                            | 170 +++++++++++++++
>  hw/i386/x86.c                            |  29 +++
>  hw/vfio/common.c                         |   1 +
>  include/exec/memory.h                    |  15 +-
>  include/hw/i386/pc.h                     |  10 +
>  include/hw/i386/sgx-epc.h                |  68 ++++++
>  include/hw/i386/x86.h                    |   1 +
>  include/monitor/hmp.h                    |   1 +
>  include/qemu/bitops.h                    |   7 +
>  monitor/hmp-cmds.c                       |  32 +++
>  monitor/qmp-cmds.c                       |  19 ++
>  qapi/machine.json                        |  52 ++++-
>  qapi/misc.json                           |  61 ++++++
>  qapi/qom.json                            |  19 ++
>  qemu-options.hx                          |  10 +-
>  softmmu/memory.c                         |   5 +
>  softmmu/physmem.c                        |   3 +-
>  stubs/meson.build                        |   1 +
>  stubs/sgx-stub.c                         |  12 +
>  target/i386/cpu.c                        | 168 +++++++++++++-
>  target/i386/cpu.h                        |  16 ++
>  target/i386/kvm/kvm.c                    |  75 +++++++
>  target/i386/kvm/kvm_i386.h               |   2 +
>  target/i386/machine.c                    |  20 ++
>  tests/qtest/qmp-cmd-test.c               |   2 +
>  39 files changed, 1430 insertions(+), 10 deletions(-)
>  create mode 100644 backends/hostmem-epc.c
>  create mode 100644 docs/intel-sgx.txt
>  create mode 100644 hw/i386/sgx-epc.c
>  create mode 100644 hw/i386/sgx-stub.c
>  create mode 100644 hw/i386/sgx.c
>  create mode 100644 include/hw/i386/sgx-epc.h
>  create mode 100644 stubs/sgx-stub.c
>



^ permalink raw reply	[flat|nested] 61+ messages in thread

* Re: [PATCH v4 00/33] Qemu SGX virtualization
  2021-09-06 13:13 ` Paolo Bonzini
@ 2021-09-07  2:24   ` Yang Zhong
  2021-09-07  9:51   ` Yang Zhong
  1 sibling, 0 replies; 61+ messages in thread
From: Yang Zhong @ 2021-09-07  2:24 UTC (permalink / raw)
  To: Paolo Bonzini; +Cc: yang.zhong, seanjc, qemu-devel, jarkko, kai.huang, eblake

On Mon, Sep 06, 2021 at 03:13:08PM +0200, Paolo Bonzini wrote:
> Hi,
> 
> the monitor patches did not pass the test-hmp qtest, and also they
> should be in target/i386/monitor.c (see other commands that were
> implemented there for SEV).  However, I've sent a pull request with
> the rest.
>

  Paolo, thanks for pulling those patches! In fact, the first POC implemented the 
  monitor to target/i386/monitor.c, but in that time, there were lots of 'pragma GCC poison'
  errors during the build, so i had to move to common monitor. Let me check this again, thanks!

  Yang
 
> Thanks,
> 
> Paolo
> 
> On Mon, Jul 19, 2021 at 1:27 PM Yang Zhong <yang.zhong@intel.com> wrote:
> >
> > Since Sean Christopherson has left Intel and i am responsible for Qemu SGX
> > upstream work. His @intel.com address will be bouncing and his new email(
> > seanjc@google.com) is also in CC lists.
> >
> > This series is Qemu SGX virtualization implementation rebased on latest
> > Qemu release. The numa support for SGX will be sent in another patchset
> > once this basic SGX patchset are merged.
> >
> > You can find Qemu repo here:
> >
> >     https://github.com/intel/qemu-sgx.git upstream
> >
> > If you want to try SGX, you can directly install the linux release(at least 5.13.0-rc1+)
> > since kvm SGX has been merged into linux release.
> >
> >     https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git
> >
> > To simplify, you'd better install linux on host and guest, which can support
> > SGX on host and guest kernel. And to me, use below reference command to boot
> > SGX guest:
> >
> >     #qemu-system-x86_64 \
> >         ...... \
> >         -cpu host,+sgx-provisionkey \
> >         -object memory-backend-epc,id=mem1,size=64M,prealloc=on \
> >         -object memory-backend-epc,id=mem2,size=28M \
> >         -M sgx-epc.0.memdev=mem1,sgx-epc.1.memdev=mem2
> >
> > Overview
> > ========
> >
> > Intel Software Guard eXtensions (SGX) is a set of instructions and mechanisms
> > for memory accesses in order to provide security accesses for sensitive
> > applications and data. SGX allows an application to use it's pariticular
> > address space as an *enclave*, which is a protected area provides confidentiality
> > and integrity even in the presence of privileged malware. Accesses to the
> > enclave memory area from any software not resident in the enclave are prevented,
> > including those from privileged software.
> >
> > SGX virtaulization
> > ==================
> >
> > The KVM SGX creates one new misc device, sgx_vepc, and Qemu will open '/dev/sgx_vepc'
> > device node to mmap() host EPC memory to guest. The Qemu also adds 'sgx-epc' device
> > to expose EPC sections to guest through CPUID and ACPI table.  The Qemu SGX also
> > supports multiple virtual EPC sections to guest, we just put them together physically
> > contiguous for the sake of simplicity. The kernel SGX NUMA has been merged into Linux
> > tip tree, we will support this function in the next phase.
> >
> > Although the current host SGX subsystem can not support SGX2 feature, the KVM/Qemu
> > implementation still expose this feature to guest. Guest SGX2 support doesn't have
> > interaction with host kernel SGX driver, the SGX guest can normally use those new
> > instructions.
> >
> > As for SGX virtualization detailed infomation, please reference docs/intel-sgx.txt
> > docuement(patch 33).
> >
> > Changelog:
> > =========
> >
> > (Changelog here is for global changes, please see each patch's changelog for changes
> > made to specific patch.)
> >
> > v3-->v4:
> >    - Rebased the sgx patches into latest Qemu release.
> >    - Moved sgx compound property setter/getter from MachineState to X86MachineState(Paolo).
> >    - Re-defined struct SgxEPC, removed 'id' property and added struct SgxEPCList for
> >      sgx-epc.0.{memdev}(Paolo).
> >    - Removed g_malloc0(), and changed the 'SGXEPCState *sgx_epc' to 'SGXEPCState sgx_epc'
> >      in struct PCMachineState(Paolo).
> >    - Changed the SGX compound property cmdline from sgx-epc.{memdev}.0 to
> >      sgx-epc.0.{memdev}(Paolo).
> >    - Removed the signature from the 'git format-patch' command(Jarkko).
> >
> > v2-->v3:
> >    - Rebased the sgx patches into latest Qemu release.
> >    - Implemented the compound property for SGX, ref patch5, the command from '-sgx-epc'
> >      to '-M'(Paolo).
> >    - Moved the sgx common code from sgx-epc.c to sgx.c. The sgx-epc.c is
> >      only responsible for virtual epc device.
> >    - Removed the previous patch13(linux-headers: Add placeholder for KVM_CAP_SGX_ATTRIBUTE)
> >      because ehabkost@redhat.com updated Linux headers to 5.13-rc4 with commit 278f064e452.
> >    - Updated the patch1 because ram_flags were changed by David Hildenbra.
> >    - Added one patch24, which avoid reset operation caused by bios reset.
> >    - Added one patch25, which make prealloc property consistent with Qemu cmdline during VM
> >      reset.
> >
> > v1-->v2:
> >    - Rebased the sgx patches into latest Qemu release.
> >    - Unified the "share" and "protected" arguments with ram_flags in the
> >      memory_region_init_ram_from_fd()(Paolo).
> >    - Added the new MemoryBackendEpcProperties and related documents(Eric Blake).
> >    - Changed the KVM_CAP_SGX_ATTRIBUTE from 195 to 196(Kai).
> >    - Changed the version and some grammar issues(Eric Blake).
> >
> > Sean Christopherson (21):
> >   memory: Add RAM_PROTECTED flag to skip IOMMU mappings
> >   hostmem: Add hostmem-epc as a backend for SGX EPC
> >   i386: Add 'sgx-epc' device to expose EPC sections to guest
> >   vl: Add sgx compound properties to expose SGX EPC sections to guest
> >   i386: Add primary SGX CPUID and MSR defines
> >   i386: Add SGX CPUID leaf FEAT_SGX_12_0_EAX
> >   i386: Add SGX CPUID leaf FEAT_SGX_12_0_EBX
> >   i386: Add SGX CPUID leaf FEAT_SGX_12_1_EAX
> >   i386: Add get/set/migrate support for SGX_LEPUBKEYHASH MSRs
> >   i386: Add feature control MSR dependency when SGX is enabled
> >   i386: Update SGX CPUID info according to hardware/KVM/user input
> >   i386: kvm: Add support for exposing PROVISIONKEY to guest
> >   i386: Propagate SGX CPUID sub-leafs to KVM
> >   Adjust min CPUID level to 0x12 when SGX is enabled
> >   hw/i386/fw_cfg: Set SGX bits in feature control fw_cfg accordingly
> >   hw/i386/pc: Account for SGX EPC sections when calculating device
> >     memory
> >   i386/pc: Add e820 entry for SGX EPC section(s)
> >   i386: acpi: Add SGX EPC entry to ACPI tables
> >   q35: Add support for SGX EPC
> >   i440fx: Add support for SGX EPC
> >   doc: Add the SGX doc
> >
> > Yang Zhong (12):
> >   qom: Add memory-backend-epc ObjectOptions support
> >   hostmem-epc: Add the reset interface for EPC backend reset
> >   sgx-epc: Add the reset interface for sgx-epc virt device
> >   sgx-epc: Avoid bios reset during sgx epc initialization
> >   hostmem-epc: Make prealloc consistent with qemu cmdline during reset
> >   qmp: Add query-sgx command
> >   hmp: Add 'info sgx' command
> >   i386: Add sgx_get_info() interface
> >   bitops: Support 32 and 64 bit mask macro
> >   qmp: Add the qmp_query_sgx_capabilities()
> >   Kconfig: Add CONFIG_SGX support
> >   sgx-epc: Add the fill_device_info() callback support
> >
> >  backends/hostmem-epc.c                   | 118 ++++++++++
> >  backends/meson.build                     |   1 +
> >  configs/devices/i386-softmmu/default.mak |   1 +
> >  docs/intel-sgx.txt                       | 167 ++++++++++++++
> >  hmp-commands-info.hx                     |  15 ++
> >  hw/i386/Kconfig                          |   5 +
> >  hw/i386/acpi-build.c                     |  22 ++
> >  hw/i386/fw_cfg.c                         |  10 +-
> >  hw/i386/meson.build                      |   2 +
> >  hw/i386/pc.c                             |  15 +-
> >  hw/i386/pc_piix.c                        |   4 +
> >  hw/i386/pc_q35.c                         |   3 +
> >  hw/i386/sgx-epc.c                        | 265 +++++++++++++++++++++++
> >  hw/i386/sgx-stub.c                       |  13 ++
> >  hw/i386/sgx.c                            | 170 +++++++++++++++
> >  hw/i386/x86.c                            |  29 +++
> >  hw/vfio/common.c                         |   1 +
> >  include/exec/memory.h                    |  15 +-
> >  include/hw/i386/pc.h                     |  10 +
> >  include/hw/i386/sgx-epc.h                |  68 ++++++
> >  include/hw/i386/x86.h                    |   1 +
> >  include/monitor/hmp.h                    |   1 +
> >  include/qemu/bitops.h                    |   7 +
> >  monitor/hmp-cmds.c                       |  32 +++
> >  monitor/qmp-cmds.c                       |  19 ++
> >  qapi/machine.json                        |  52 ++++-
> >  qapi/misc.json                           |  61 ++++++
> >  qapi/qom.json                            |  19 ++
> >  qemu-options.hx                          |  10 +-
> >  softmmu/memory.c                         |   5 +
> >  softmmu/physmem.c                        |   3 +-
> >  stubs/meson.build                        |   1 +
> >  stubs/sgx-stub.c                         |  12 +
> >  target/i386/cpu.c                        | 168 +++++++++++++-
> >  target/i386/cpu.h                        |  16 ++
> >  target/i386/kvm/kvm.c                    |  75 +++++++
> >  target/i386/kvm/kvm_i386.h               |   2 +
> >  target/i386/machine.c                    |  20 ++
> >  tests/qtest/qmp-cmd-test.c               |   2 +
> >  39 files changed, 1430 insertions(+), 10 deletions(-)
> >  create mode 100644 backends/hostmem-epc.c
> >  create mode 100644 docs/intel-sgx.txt
> >  create mode 100644 hw/i386/sgx-epc.c
> >  create mode 100644 hw/i386/sgx-stub.c
> >  create mode 100644 hw/i386/sgx.c
> >  create mode 100644 include/hw/i386/sgx-epc.h
> >  create mode 100644 stubs/sgx-stub.c
> >


^ permalink raw reply	[flat|nested] 61+ messages in thread

* Re: [PATCH v4 00/33] Qemu SGX virtualization
  2021-09-06 13:13 ` Paolo Bonzini
  2021-09-07  2:24   ` Yang Zhong
@ 2021-09-07  9:51   ` Yang Zhong
  2021-09-07 13:35     ` Jarkko Sakkinen
  2021-09-08  6:00     ` Paolo Bonzini
  1 sibling, 2 replies; 61+ messages in thread
From: Yang Zhong @ 2021-09-07  9:51 UTC (permalink / raw)
  To: Paolo Bonzini; +Cc: yang.zhong, seanjc, qemu-devel, jarkko, kai.huang, eblake

On Mon, Sep 06, 2021 at 03:13:08PM +0200, Paolo Bonzini wrote:
> Hi,
> 
> the monitor patches did not pass the test-hmp qtest, and also they
> should be in target/i386/monitor.c (see other commands that were
> implemented there for SEV).  However, I've sent a pull request with
> the rest.
>
  
  Paolo, I have moved hmp and qmp codes to target/i386/monitor.c and also
  fixed the issue with test tool(tests/qtest/test-hmp). This issue is caused
  by 'machine none' test in the hmp, the previous patches only covered qmp
  test in 'machine none' with Libvirt.

  So, the next issue:
  1) re-send all sgx basic patches(including monitors patches) to you?
  2) only send monitor patches in the next phase when the basic sgx patches
     are merged? 

  Regards,

  Yang

 
> Thanks,
> 
> Paolo
> 
> On Mon, Jul 19, 2021 at 1:27 PM Yang Zhong <yang.zhong@intel.com> wrote:
> >
> > Since Sean Christopherson has left Intel and i am responsible for Qemu SGX
> > upstream work. His @intel.com address will be bouncing and his new email(
> > seanjc@google.com) is also in CC lists.
> >
> > This series is Qemu SGX virtualization implementation rebased on latest
> > Qemu release. The numa support for SGX will be sent in another patchset
> > once this basic SGX patchset are merged.
> >
> > You can find Qemu repo here:
> >
> >     https://github.com/intel/qemu-sgx.git upstream
> >
> > If you want to try SGX, you can directly install the linux release(at least 5.13.0-rc1+)
> > since kvm SGX has been merged into linux release.
> >
> >     https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git
> >
> > To simplify, you'd better install linux on host and guest, which can support
> > SGX on host and guest kernel. And to me, use below reference command to boot
> > SGX guest:
> >
> >     #qemu-system-x86_64 \
> >         ...... \
> >         -cpu host,+sgx-provisionkey \
> >         -object memory-backend-epc,id=mem1,size=64M,prealloc=on \
> >         -object memory-backend-epc,id=mem2,size=28M \
> >         -M sgx-epc.0.memdev=mem1,sgx-epc.1.memdev=mem2
> >
> > Overview
> > ========
> >
> > Intel Software Guard eXtensions (SGX) is a set of instructions and mechanisms
> > for memory accesses in order to provide security accesses for sensitive
> > applications and data. SGX allows an application to use it's pariticular
> > address space as an *enclave*, which is a protected area provides confidentiality
> > and integrity even in the presence of privileged malware. Accesses to the
> > enclave memory area from any software not resident in the enclave are prevented,
> > including those from privileged software.
> >
> > SGX virtaulization
> > ==================
> >
> > The KVM SGX creates one new misc device, sgx_vepc, and Qemu will open '/dev/sgx_vepc'
> > device node to mmap() host EPC memory to guest. The Qemu also adds 'sgx-epc' device
> > to expose EPC sections to guest through CPUID and ACPI table.  The Qemu SGX also
> > supports multiple virtual EPC sections to guest, we just put them together physically
> > contiguous for the sake of simplicity. The kernel SGX NUMA has been merged into Linux
> > tip tree, we will support this function in the next phase.
> >
> > Although the current host SGX subsystem can not support SGX2 feature, the KVM/Qemu
> > implementation still expose this feature to guest. Guest SGX2 support doesn't have
> > interaction with host kernel SGX driver, the SGX guest can normally use those new
> > instructions.
> >
> > As for SGX virtualization detailed infomation, please reference docs/intel-sgx.txt
> > docuement(patch 33).
> >
> > Changelog:
> > =========
> >
> > (Changelog here is for global changes, please see each patch's changelog for changes
> > made to specific patch.)
> >
> > v3-->v4:
> >    - Rebased the sgx patches into latest Qemu release.
> >    - Moved sgx compound property setter/getter from MachineState to X86MachineState(Paolo).
> >    - Re-defined struct SgxEPC, removed 'id' property and added struct SgxEPCList for
> >      sgx-epc.0.{memdev}(Paolo).
> >    - Removed g_malloc0(), and changed the 'SGXEPCState *sgx_epc' to 'SGXEPCState sgx_epc'
> >      in struct PCMachineState(Paolo).
> >    - Changed the SGX compound property cmdline from sgx-epc.{memdev}.0 to
> >      sgx-epc.0.{memdev}(Paolo).
> >    - Removed the signature from the 'git format-patch' command(Jarkko).
> >
> > v2-->v3:
> >    - Rebased the sgx patches into latest Qemu release.
> >    - Implemented the compound property for SGX, ref patch5, the command from '-sgx-epc'
> >      to '-M'(Paolo).
> >    - Moved the sgx common code from sgx-epc.c to sgx.c. The sgx-epc.c is
> >      only responsible for virtual epc device.
> >    - Removed the previous patch13(linux-headers: Add placeholder for KVM_CAP_SGX_ATTRIBUTE)
> >      because ehabkost@redhat.com updated Linux headers to 5.13-rc4 with commit 278f064e452.
> >    - Updated the patch1 because ram_flags were changed by David Hildenbra.
> >    - Added one patch24, which avoid reset operation caused by bios reset.
> >    - Added one patch25, which make prealloc property consistent with Qemu cmdline during VM
> >      reset.
> >
> > v1-->v2:
> >    - Rebased the sgx patches into latest Qemu release.
> >    - Unified the "share" and "protected" arguments with ram_flags in the
> >      memory_region_init_ram_from_fd()(Paolo).
> >    - Added the new MemoryBackendEpcProperties and related documents(Eric Blake).
> >    - Changed the KVM_CAP_SGX_ATTRIBUTE from 195 to 196(Kai).
> >    - Changed the version and some grammar issues(Eric Blake).
> >
> > Sean Christopherson (21):
> >   memory: Add RAM_PROTECTED flag to skip IOMMU mappings
> >   hostmem: Add hostmem-epc as a backend for SGX EPC
> >   i386: Add 'sgx-epc' device to expose EPC sections to guest
> >   vl: Add sgx compound properties to expose SGX EPC sections to guest
> >   i386: Add primary SGX CPUID and MSR defines
> >   i386: Add SGX CPUID leaf FEAT_SGX_12_0_EAX
> >   i386: Add SGX CPUID leaf FEAT_SGX_12_0_EBX
> >   i386: Add SGX CPUID leaf FEAT_SGX_12_1_EAX
> >   i386: Add get/set/migrate support for SGX_LEPUBKEYHASH MSRs
> >   i386: Add feature control MSR dependency when SGX is enabled
> >   i386: Update SGX CPUID info according to hardware/KVM/user input
> >   i386: kvm: Add support for exposing PROVISIONKEY to guest
> >   i386: Propagate SGX CPUID sub-leafs to KVM
> >   Adjust min CPUID level to 0x12 when SGX is enabled
> >   hw/i386/fw_cfg: Set SGX bits in feature control fw_cfg accordingly
> >   hw/i386/pc: Account for SGX EPC sections when calculating device
> >     memory
> >   i386/pc: Add e820 entry for SGX EPC section(s)
> >   i386: acpi: Add SGX EPC entry to ACPI tables
> >   q35: Add support for SGX EPC
> >   i440fx: Add support for SGX EPC
> >   doc: Add the SGX doc
> >
> > Yang Zhong (12):
> >   qom: Add memory-backend-epc ObjectOptions support
> >   hostmem-epc: Add the reset interface for EPC backend reset
> >   sgx-epc: Add the reset interface for sgx-epc virt device
> >   sgx-epc: Avoid bios reset during sgx epc initialization
> >   hostmem-epc: Make prealloc consistent with qemu cmdline during reset
> >   qmp: Add query-sgx command
> >   hmp: Add 'info sgx' command
> >   i386: Add sgx_get_info() interface
> >   bitops: Support 32 and 64 bit mask macro
> >   qmp: Add the qmp_query_sgx_capabilities()
> >   Kconfig: Add CONFIG_SGX support
> >   sgx-epc: Add the fill_device_info() callback support
> >
> >  backends/hostmem-epc.c                   | 118 ++++++++++
> >  backends/meson.build                     |   1 +
> >  configs/devices/i386-softmmu/default.mak |   1 +
> >  docs/intel-sgx.txt                       | 167 ++++++++++++++
> >  hmp-commands-info.hx                     |  15 ++
> >  hw/i386/Kconfig                          |   5 +
> >  hw/i386/acpi-build.c                     |  22 ++
> >  hw/i386/fw_cfg.c                         |  10 +-
> >  hw/i386/meson.build                      |   2 +
> >  hw/i386/pc.c                             |  15 +-
> >  hw/i386/pc_piix.c                        |   4 +
> >  hw/i386/pc_q35.c                         |   3 +
> >  hw/i386/sgx-epc.c                        | 265 +++++++++++++++++++++++
> >  hw/i386/sgx-stub.c                       |  13 ++
> >  hw/i386/sgx.c                            | 170 +++++++++++++++
> >  hw/i386/x86.c                            |  29 +++
> >  hw/vfio/common.c                         |   1 +
> >  include/exec/memory.h                    |  15 +-
> >  include/hw/i386/pc.h                     |  10 +
> >  include/hw/i386/sgx-epc.h                |  68 ++++++
> >  include/hw/i386/x86.h                    |   1 +
> >  include/monitor/hmp.h                    |   1 +
> >  include/qemu/bitops.h                    |   7 +
> >  monitor/hmp-cmds.c                       |  32 +++
> >  monitor/qmp-cmds.c                       |  19 ++
> >  qapi/machine.json                        |  52 ++++-
> >  qapi/misc.json                           |  61 ++++++
> >  qapi/qom.json                            |  19 ++
> >  qemu-options.hx                          |  10 +-
> >  softmmu/memory.c                         |   5 +
> >  softmmu/physmem.c                        |   3 +-
> >  stubs/meson.build                        |   1 +
> >  stubs/sgx-stub.c                         |  12 +
> >  target/i386/cpu.c                        | 168 +++++++++++++-
> >  target/i386/cpu.h                        |  16 ++
> >  target/i386/kvm/kvm.c                    |  75 +++++++
> >  target/i386/kvm/kvm_i386.h               |   2 +
> >  target/i386/machine.c                    |  20 ++
> >  tests/qtest/qmp-cmd-test.c               |   2 +
> >  39 files changed, 1430 insertions(+), 10 deletions(-)
> >  create mode 100644 backends/hostmem-epc.c
> >  create mode 100644 docs/intel-sgx.txt
> >  create mode 100644 hw/i386/sgx-epc.c
> >  create mode 100644 hw/i386/sgx-stub.c
> >  create mode 100644 hw/i386/sgx.c
> >  create mode 100644 include/hw/i386/sgx-epc.h
> >  create mode 100644 stubs/sgx-stub.c
> >


^ permalink raw reply	[flat|nested] 61+ messages in thread

* Re: [PATCH v4 00/33] Qemu SGX virtualization
  2021-09-07  9:51   ` Yang Zhong
@ 2021-09-07 13:35     ` Jarkko Sakkinen
  2021-09-08  6:00     ` Paolo Bonzini
  1 sibling, 0 replies; 61+ messages in thread
From: Jarkko Sakkinen @ 2021-09-07 13:35 UTC (permalink / raw)
  To: Yang Zhong, Paolo Bonzini; +Cc: seanjc, kai.huang, eblake, qemu-devel

On Tue, 2021-09-07 at 17:51 +0800, Yang Zhong wrote:
> On Mon, Sep 06, 2021 at 03:13:08PM +0200, Paolo Bonzini wrote:
> > Hi,
> > 
> > the monitor patches did not pass the test-hmp qtest, and also they
> > should be in target/i386/monitor.c (see other commands that were
> > implemented there for SEV).  However, I've sent a pull request with
> > the rest.
> > 
>   
>   Paolo, I have moved hmp and qmp codes to target/i386/monitor.c and also
>   fixed the issue with test tool(tests/qtest/test-hmp). This issue is caused
>   by 'machine none' test in the hmp, the previous patches only covered qmp
>   test in 'machine none' with Libvirt.
> 
>   So, the next issue:
>   1) re-send all sgx basic patches(including monitors patches) to you?
>   2) only send monitor patches in the next phase when the basic sgx patches
>      are merged? 
> 
>   Regards,
> 
>   Yang
> 
>  
> > Thanks,
> > 
> > Paolo
> > 
> > On Mon, Jul 19, 2021 at 1:27 PM Yang Zhong <yang.zhong@intel.com> wrote:
> > > Since Sean Christopherson has left Intel and i am responsible for Qemu SGX
> > > upstream work. His @intel.com address will be bouncing and his new email(
> > > seanjc@google.com) is also in CC lists.
> > > 
> > > This series is Qemu SGX virtualization implementation rebased on latest
> > > Qemu release. The numa support for SGX will be sent in another patchset
> > > once this basic SGX patchset are merged.
> > > 
> > > You can find Qemu repo here:
> > > 
> > >     https://github.com/intel/qemu-sgx.git upstream
> > > 
> > > If you want to try SGX, you can directly install the linux release(at least 5.13.0-rc1+)
> > > since kvm SGX has been merged into linux release.
> > > 
> > >     https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git
> > > 
> > > To simplify, you'd better install linux on host and guest, which can support
> > > SGX on host and guest kernel. And to me, use below reference command to boot
> > > SGX guest:
> > > 
> > >     #qemu-system-x86_64 \
> > >         ...... \
> > >         -cpu host,+sgx-provisionkey \
> > >         -object memory-backend-epc,id=mem1,size=64M,prealloc=on \
> > >         -object memory-backend-epc,id=mem2,size=28M \

I would call these just "memory-backend-sgx".

> > >         -M sgx-epc.0.memdev=mem1,sgx-epc.1.memdev=mem2
> > > 
> > > Overview
> > > ========
> > > 
> > > Intel Software Guard eXtensions (SGX) is a set of instructions and mechanisms
> > > for memory accesses in order to provide security accesses for sensitive
> > > applications and data. SGX allows an application to use it's pariticular
> > > address space as an *enclave*, which is a protected area provides confidentiality
> > > and integrity even in the presence of privileged malware. Accesses to the
> > > enclave memory area from any software not resident in the enclave are prevented,
> > > including those from privileged software.
> > > 
> > > SGX virtaulization
          ~~~~~~~~~~~~~~
	  virtualization

I'm using these patches now instead of "real" hardwave. It seems that
9th gen cores are now Linux compatible SGX. Maybe some ucode update
has changed this because I don't recall this working before.


/Jarkko



^ permalink raw reply	[flat|nested] 61+ messages in thread

* Re: [PATCH v4 00/33] Qemu SGX virtualization
  2021-09-07  9:51   ` Yang Zhong
  2021-09-07 13:35     ` Jarkko Sakkinen
@ 2021-09-08  6:00     ` Paolo Bonzini
  1 sibling, 0 replies; 61+ messages in thread
From: Paolo Bonzini @ 2021-09-08  6:00 UTC (permalink / raw)
  To: Yang Zhong; +Cc: seanjc, jarkko, eblake, qemu-devel, kai.huang

On 07/09/21 11:51, Yang Zhong wrote:
>    
>    Paolo, I have moved hmp and qmp codes to target/i386/monitor.c and also
>    fixed the issue with test tool(tests/qtest/test-hmp). This issue is caused
>    by 'machine none' test in the hmp, the previous patches only covered qmp
>    test in 'machine none' with Libvirt.
> 
>    So, the next issue:
>    1) re-send all sgx basic patches(including monitors patches) to you?
>    2) only send monitor patches in the next phase when the basic sgx patches
>       are merged?

You can post them now, including

Based-on: <20210907071349.116329-1-pbonzini@redhat.com>

in the cover letter.

Paolo



^ permalink raw reply	[flat|nested] 61+ messages in thread

* Re: [PATCH v4 22/33] hostmem-epc: Add the reset interface for EPC backend reset
  2021-07-19 11:21 ` [PATCH v4 22/33] hostmem-epc: Add the reset interface for EPC backend reset Yang Zhong
@ 2021-09-10 15:10   ` Paolo Bonzini
  2021-09-10 15:34     ` Sean Christopherson
  2021-09-13 20:17     ` Jarkko Sakkinen
  0 siblings, 2 replies; 61+ messages in thread
From: Paolo Bonzini @ 2021-09-10 15:10 UTC (permalink / raw)
  To: qemu-devel, jarkko, seanjc, kai.huang; +Cc: Yang Zhong, eblake

On 19/07/21 13:21, Yang Zhong wrote:
> +void sgx_memory_backend_reset(HostMemoryBackend *backend, int fd,
> +                              Error **errp)
> +{
> +    MemoryRegion *mr = &backend->mr;
> +
> +    mr->enabled = false;
> +
> +    /* destroy the old memory region if it exist */
> +    if (fd > 0 && mr->destructor) {
> +        mr->destructor(mr);
> +    }
> +
> +    sgx_epc_backend_memory_alloc(backend, errp);
> +}
> +

Jarkko, Sean, Kai,

this I think is problematic because it has a race window while 
/dev/sgx_vepc is closed and then reopened.  First, the vEPC space could 
be exhausted by somebody doing another mmap in the meanwhile.  Second, 
somebody might (for whatever reason) remove /dev/sgx_vepc while QEMU runs.

Yang explained to me (offlist) that this is needed because Windows fails 
to reboot without it.  We would need a way to ask Linux to reinitialize 
the vEPC, that doesn't involve munmap/mmap; this could be for example 
fallocate(FALLOC_FL_ZERO_RANGE).

What do you all think?

Paolo



^ permalink raw reply	[flat|nested] 61+ messages in thread

* Re: [PATCH v4 23/33] sgx-epc: Add the reset interface for sgx-epc virt device
  2021-07-19 11:21 ` [PATCH v4 23/33] sgx-epc: Add the reset interface for sgx-epc virt device Yang Zhong
@ 2021-09-10 15:13   ` Paolo Bonzini
  2021-09-14  6:53   ` Philippe Mathieu-Daudé
  1 sibling, 0 replies; 61+ messages in thread
From: Paolo Bonzini @ 2021-09-10 15:13 UTC (permalink / raw)
  To: Yang Zhong, qemu-devel; +Cc: seanjc, kai.huang, eblake, jarkko

On 19/07/21 13:21, Yang Zhong wrote:
> +static void sgx_epc_del_subregion(DeviceState *dev)
> +{
> +    PCMachineState *pcms = PC_MACHINE(qdev_get_machine());
> +    SGXEPCState *sgx_epc = &pcms->sgx_epc;
> +    SGXEPCDevice *epc = SGX_EPC(dev);
> +
> +    /* del subregion and related operations */
> +    memory_region_del_subregion(&sgx_epc->mr,
> +                                host_memory_backend_get_memory(epc->hostmem));
> +    host_memory_backend_set_mapped(epc->hostmem, false);
> +    g_free(sgx_epc->sections);
> +    sgx_epc->sections = NULL;
> +
> +    /* multiple epc devices, only zero the first time */
> +    if (epc_num == sgx_epc->nr_sections) {
> +        sgx_epc->size = 0;
> +        sgx_epc->nr_sections = 0;
> +    }
> +}

Can it just invoke a method on the memory devices, without reusing the 
code in sgx_epc_realize?  In particular why is it necessary to update 
sgx_epc->size and sgx_epc->nr_sections?

Paolo



^ permalink raw reply	[flat|nested] 61+ messages in thread

* Re: [PATCH v4 22/33] hostmem-epc: Add the reset interface for EPC backend reset
  2021-09-10 15:10   ` Paolo Bonzini
@ 2021-09-10 15:34     ` Sean Christopherson
  2021-09-10 17:09       ` Paolo Bonzini
  2021-09-13 20:17     ` Jarkko Sakkinen
  1 sibling, 1 reply; 61+ messages in thread
From: Sean Christopherson @ 2021-09-10 15:34 UTC (permalink / raw)
  To: Paolo Bonzini; +Cc: qemu-devel, jarkko, kai.huang, eblake, Yang Zhong

On Fri, Sep 10, 2021, Paolo Bonzini wrote:
> On 19/07/21 13:21, Yang Zhong wrote:
> > +void sgx_memory_backend_reset(HostMemoryBackend *backend, int fd,
> > +                              Error **errp)
> > +{
> > +    MemoryRegion *mr = &backend->mr;
> > +
> > +    mr->enabled = false;
> > +
> > +    /* destroy the old memory region if it exist */
> > +    if (fd > 0 && mr->destructor) {
> > +        mr->destructor(mr);
> > +    }
> > +
> > +    sgx_epc_backend_memory_alloc(backend, errp);
> > +}
> > +
> 
> Jarkko, Sean, Kai,
> 
> this I think is problematic because it has a race window while /dev/sgx_vepc
> is closed and then reopened.  First, the vEPC space could be exhausted by
> somebody doing another mmap in the meanwhile.  Second, somebody might (for
> whatever reason) remove /dev/sgx_vepc while QEMU runs.

Yep, both error cases are possible.

> Yang explained to me (offlist) that this is needed because Windows fails to
> reboot without it.  We would need a way to ask Linux to reinitialize the
> vEPC, that doesn't involve munmap/mmap; this could be for example
> fallocate(FALLOC_FL_ZERO_RANGE).
> 
> What do you all think?

Mechanically, FALLOC_FL_ZERO_RANGE could work, but it's definitely a stretch of
the semantics as the underlying memory would not actually be zeroed.

The only other option that comes to mind is a dedicated ioctl().


^ permalink raw reply	[flat|nested] 61+ messages in thread

* Re: [PATCH v4 22/33] hostmem-epc: Add the reset interface for EPC backend reset
  2021-09-10 15:34     ` Sean Christopherson
@ 2021-09-10 17:09       ` Paolo Bonzini
  2021-09-10 17:34         ` Sean Christopherson
  0 siblings, 1 reply; 61+ messages in thread
From: Paolo Bonzini @ 2021-09-10 17:09 UTC (permalink / raw)
  To: Sean Christopherson; +Cc: Yang Zhong, jarkko, eblake, qemu-devel, kai.huang

On 10/09/21 17:34, Sean Christopherson wrote:
>> Yang explained to me (offlist) that this is needed because Windows fails to
>> reboot without it.  We would need a way to ask Linux to reinitialize the
>> vEPC, that doesn't involve munmap/mmap; this could be for example
>> fallocate(FALLOC_FL_ZERO_RANGE).
>>
>> What do you all think?
>
> Mechanically, FALLOC_FL_ZERO_RANGE could work, but it's definitely a stretch of
> the semantics as the underlying memory would not actually be zeroed.

The contents are not visible to anyone, so they might well be zero
(not entirely serious, but also not entirely unserious).

> The only other option that comes to mind is a dedicated ioctl().

If it is not too restrictive to do it for the whole mapping at once,
that would be fine.  If it makes sense to do it for a range, however,
the effort of defining a ioctl() API is probably not worth it when
fallocate() is available.

Either way, I suppose it would be just something like

	/* for fallocate; otherwise just use xa_for_each */
	size = min_t(unsigned long, size, -start);
	end = (start + size - 1) >> PAGE_SHIFT;
	start >>= PAGE_SHIFT;

	/*
	 * Removing in two passes lets us remove SECS pages as well,
	 * since they can only be EREMOVE'd after all their child pages.
	 * An EREMOVE failure on the second pass means that the SECS
	 * page still has children on another instance.  Since it is
	 * still in the xarray, bury our head in the sand and ignore
	 * it; sgx_vepc_release() will deal with it.
	 */
	LIST_HEAD(secs_pages);
         xa_for_each_range(&vepc->page_array, index, entry, start, end) {
                 if (!sgx_vepc_free_page(entry))
                         list_add_tail(&epc_page->list, &secs_pages);
         }

         list_for_each_entry_safe(epc_page, tmp, &secs_pages, list) {
		list_del(&epc_page->list);
                 sgx_vepc_free_page(epc_page);
         }
  
So another advantage of the ioctl would be e.g. being able to return the
number of successfully EREMOVEd pages.  But since QEMU would not do
anything with the return value and _also_ bury its head in the sand,
that would only be useful if you guys have other uses in mind.

Paolo



^ permalink raw reply	[flat|nested] 61+ messages in thread

* Re: [PATCH v4 22/33] hostmem-epc: Add the reset interface for EPC backend reset
  2021-09-10 17:09       ` Paolo Bonzini
@ 2021-09-10 17:34         ` Sean Christopherson
  2021-09-10 19:51           ` Paolo Bonzini
  0 siblings, 1 reply; 61+ messages in thread
From: Sean Christopherson @ 2021-09-10 17:34 UTC (permalink / raw)
  To: Paolo Bonzini; +Cc: qemu-devel, jarkko, kai.huang, eblake, Yang Zhong

On Fri, Sep 10, 2021, Paolo Bonzini wrote:
> On 10/09/21 17:34, Sean Christopherson wrote:
> > > Yang explained to me (offlist) that this is needed because Windows fails to
> > > reboot without it.  We would need a way to ask Linux to reinitialize the
> > > vEPC, that doesn't involve munmap/mmap; this could be for example
> > > fallocate(FALLOC_FL_ZERO_RANGE).
> > > 
> > > What do you all think?
> > 
> > Mechanically, FALLOC_FL_ZERO_RANGE could work, but it's definitely a stretch of
> > the semantics as the underlying memory would not actually be zeroed.
> 
> The contents are not visible to anyone, so they might well be zero
> (not entirely serious, but also not entirely unserious).

Yeah, it wouldn't be a sticking point, just odd.

> > The only other option that comes to mind is a dedicated ioctl().
> 
> If it is not too restrictive to do it for the whole mapping at once,
> that would be fine.

Oooh, rats.  That reminds me of a complication.  If QEMU creates multiple EPC
sections, e.g. for a vNUMA setup, resetting each section individually will fail
if the guest did an unclean RESET and a given enclaves has EPC pages from multiple
sections.  E.g. an SECS in vEPC[X] can have children in vEPC[0..N-1], and all
those children need to be removed before the SECS can be removed.  Yay SGX!

> If it makes sense to do it for a range, however,
> the effort of defining a ioctl() API is probably not worth it when
> fallocate() is available.
> 
> Either way, I suppose it would be just something like
> 
> 	/* for fallocate; otherwise just use xa_for_each */
> 	size = min_t(unsigned long, size, -start);
> 	end = (start + size - 1) >> PAGE_SHIFT;
> 	start >>= PAGE_SHIFT;
> 
> 	/*
> 	 * Removing in two passes lets us remove SECS pages as well,
> 	 * since they can only be EREMOVE'd after all their child pages.
> 	 * An EREMOVE failure on the second pass means that the SECS
> 	 * page still has children on another instance.  Since it is
> 	 * still in the xarray, bury our head in the sand and ignore
> 	 * it; sgx_vepc_release() will deal with it.
> 	 */
> 	LIST_HEAD(secs_pages);
>         xa_for_each_range(&vepc->page_array, index, entry, start, end) {
>                 if (!sgx_vepc_free_page(entry))
>                         list_add_tail(&epc_page->list, &secs_pages);
>         }
> 
>         list_for_each_entry_safe(epc_page, tmp, &secs_pages, list) {
> 		list_del(&epc_page->list);
>                 sgx_vepc_free_page(epc_page);
>         }
> So another advantage of the ioctl would be e.g. being able to return the
> number of successfully EREMOVEd pages.  But since QEMU would not do
> anything with the return value and _also_ bury its head in the sand,
> that would only be useful if you guys have other uses in mind.

There are two options: 1) QEMU has to handle "failure", or 2) the kernel provides
an ioctl() that takes multiple vEPC fds and handles the SECS dependencies.  #1 is
probably the least awful option.  For #2, in addition to the kernel having to deal
with multiple fds, it would also need a second list_head object in each page so
that it could track which pages failed to be removed.  Using the existing list_head
would work for now, but it won't work if/when an EPC cgroup is added.

Note, for #1, QEMU would have to potentially do three passes.

  1. Remove child pages for a given vEPC.
  2. Remove SECS for a given vEPC that were pinned by children in the same vEPC.
  3. Remove SECS for all vEPC that were pinned by children in different vEPC.


^ permalink raw reply	[flat|nested] 61+ messages in thread

* Re: [PATCH v4 22/33] hostmem-epc: Add the reset interface for EPC backend reset
  2021-09-10 17:34         ` Sean Christopherson
@ 2021-09-10 19:51           ` Paolo Bonzini
  2021-09-10 20:21             ` Sean Christopherson
  0 siblings, 1 reply; 61+ messages in thread
From: Paolo Bonzini @ 2021-09-10 19:51 UTC (permalink / raw)
  To: Sean Christopherson; +Cc: Yang Zhong, jarkko, eblake, qemu-devel, kai.huang

On 10/09/21 19:34, Sean Christopherson wrote:
> On Fri, Sep 10, 2021, Paolo Bonzini wrote:
>> On 10/09/21 17:34, Sean Christopherson wrote:
>>> The only other option that comes to mind is a dedicated ioctl().
>>
>> If it is not too restrictive to do it for the whole mapping at once,
>> that would be fine.
> 
> Oooh, rats.  That reminds me of a complication.  If QEMU creates multiple EPC
> sections, e.g. for a vNUMA setup, resetting each section individually will fail
> if the guest did an unclean RESET and a given enclaves has EPC pages from multiple
> sections.  E.g. an SECS in vEPC[X] can have children in vEPC[0..N-1], and all
> those children need to be removed before the SECS can be removed.  Yay SGX!
> 
> There are two options: 1) QEMU has to handle "failure", or 2) the kernel provides
> an ioctl() that takes multiple vEPC fds and handles the SECS dependencies.  #1 is
> probably the least awful option.  For #2, in addition to the kernel having to deal
> with multiple fds, it would also need a second list_head object in each page so
> that it could track which pages failed to be removed.  Using the existing list_head
> would work for now, but it won't work if/when an EPC cgroup is added.
> 
> Note, for #1, QEMU would have to potentially do three passes.
> 
>    1. Remove child pages for a given vEPC.
>    2. Remove SECS for a given vEPC that were pinned by children in the same vEPC.
>    3. Remove SECS for all vEPC that were pinned by children in different vEPC.

It's also possible that QEMU handles failure, but the kernel does two 
passes; then QEMU can just do two passes.  The kernel will overall do 
four passes, but:

1) the second (SECS pinned by children in the same vEPC) would be 
cheaper than a full second pass

2) the fourth would actually do nothing, because there would be no pages 
failing the EREMOV'al.

A hypothetical other SGX client that only uses one vEPC will do the 
right thing with a single pass.

Paolo



^ permalink raw reply	[flat|nested] 61+ messages in thread

* Re: [PATCH v4 22/33] hostmem-epc: Add the reset interface for EPC backend reset
  2021-09-10 19:51           ` Paolo Bonzini
@ 2021-09-10 20:21             ` Sean Christopherson
  2021-09-10 20:57               ` Paolo Bonzini
  0 siblings, 1 reply; 61+ messages in thread
From: Sean Christopherson @ 2021-09-10 20:21 UTC (permalink / raw)
  To: Paolo Bonzini; +Cc: qemu-devel, jarkko, kai.huang, eblake, Yang Zhong

On Fri, Sep 10, 2021, Paolo Bonzini wrote:
> On 10/09/21 19:34, Sean Christopherson wrote:
> > On Fri, Sep 10, 2021, Paolo Bonzini wrote:
> > > On 10/09/21 17:34, Sean Christopherson wrote:
> > > > The only other option that comes to mind is a dedicated ioctl().
> > > 
> > > If it is not too restrictive to do it for the whole mapping at once,
> > > that would be fine.
> > 
> > Oooh, rats.  That reminds me of a complication.  If QEMU creates multiple EPC
> > sections, e.g. for a vNUMA setup, resetting each section individually will fail
> > if the guest did an unclean RESET and a given enclaves has EPC pages from multiple
> > sections.  E.g. an SECS in vEPC[X] can have children in vEPC[0..N-1], and all
> > those children need to be removed before the SECS can be removed.  Yay SGX!
> > 
> > There are two options: 1) QEMU has to handle "failure", or 2) the kernel provides
> > an ioctl() that takes multiple vEPC fds and handles the SECS dependencies.  #1 is
> > probably the least awful option.  For #2, in addition to the kernel having to deal
> > with multiple fds, it would also need a second list_head object in each page so
> > that it could track which pages failed to be removed.  Using the existing list_head
> > would work for now, but it won't work if/when an EPC cgroup is added.
> > 
> > Note, for #1, QEMU would have to potentially do three passes.
> > 
> >    1. Remove child pages for a given vEPC.
> >    2. Remove SECS for a given vEPC that were pinned by children in the same vEPC.
> >    3. Remove SECS for all vEPC that were pinned by children in different vEPC.
> 
> It's also possible that QEMU handles failure, but the kernel does two
> passes; then QEMU can just do two passes.  The kernel will overall do four
> passes, but:
> 
> 1) the second (SECS pinned by children in the same vEPC) would be cheaper
> than a full second pass

The problem is that this would require a list_head (or temp allocations) to track
the SECS pages that failed the first time 'round.  For vEPC destruction, the kernel
can use sgx_epc_page.list because it can take the pages off the active/allocated
list, but that's not an option in this case because the presumably-upcoming EPC
cgroup needs to keep pages on the list to handle OOM.

The kernel's ioctl/syscall/whatever could return the number of pages that were
not freed, or maybe just -EAGAIN, and userspace could use that to know it needs
to do another reset to free everything.

My thought for QEMU was to do (bad pseudocode):

	/* Retry to EREMOVE pinned SECS pages if necessary. */
	ret = ioctl(SGX_VEPC_RESET, ...);
	if (ret)
		ret = ioctl(SGX_VEPC_RESET, ...);

	/*
	 * Tag the VM as needed yet another round of resets to ERMOVE SECS pages
	 * that were pinned across vEPC sections.
	 */
	vm->sgx_epc_final_reset_needed = !!ret;

> 2) the fourth would actually do nothing, because there would be no pages
> failing the EREMOV'al.
> 
> A hypothetical other SGX client that only uses one vEPC will do the right
> thing with a single pass.


^ permalink raw reply	[flat|nested] 61+ messages in thread

* Re: [PATCH v4 22/33] hostmem-epc: Add the reset interface for EPC backend reset
  2021-09-10 20:21             ` Sean Christopherson
@ 2021-09-10 20:57               ` Paolo Bonzini
  0 siblings, 0 replies; 61+ messages in thread
From: Paolo Bonzini @ 2021-09-10 20:57 UTC (permalink / raw)
  To: Sean Christopherson
  Cc: Yang Zhong, Jarkko Sakkinen, Blake, Eric, qemu-devel, Huang, Kai

[-- Attachment #1: Type: text/plain, Size: 1849 bytes --]

Il ven 10 set 2021, 22:21 Sean Christopherson <seanjc@google.com> ha
scritto:

> > It's also possible that QEMU handles failure, but the kernel does two
> > passes; then QEMU can just do two passes.  The kernel will overall do
> four
> > passes, but:
> >
> > 1) the second (SECS pinned by children in the same vEPC) would be cheaper
> > than a full second pass
>
> The problem is that this would require a list_head (or temp allocations)
> to track
> the SECS pages that failed the first time 'round.  For vEPC destruction,
> the kernel
> can use sgx_epc_page.list because it can take the pages off the
> active/allocated
> list, but that's not an option in this case because the
> presumably-upcoming EPC
> cgroup needs to keep pages on the list to handle OOM.
>

Good point, so yeah: let's go for a ioctl that does full removal, returning
the number of failures. I will try and cobble up a patch unless Kai beats
me to it.

Thanks for the quick discussion!

Paolo


> The kernel's ioctl/syscall/whatever could return the number of pages that
> were
> not freed, or maybe just -EAGAIN, and userspace could use that to know it
> needs
> to do another reset to free everything.
>
> My thought for QEMU was to do (bad pseudocode):
>
>         /* Retry to EREMOVE pinned SECS pages if necessary. */
>         ret = ioctl(SGX_VEPC_RESET, ...);
>         if (ret)
>                 ret = ioctl(SGX_VEPC_RESET, ...);
>
>         /*
>          * Tag the VM as needed yet another round of resets to ERMOVE SECS
> pages
>          * that were pinned across vEPC sections.
>          */
>         vm->sgx_epc_final_reset_needed = !!ret;
>
> > 2) the fourth would actually do nothing, because there would be no pages
> > failing the EREMOV'al.
> >
> > A hypothetical other SGX client that only uses one vEPC will do the right
> > thing with a single pass.
>
>

[-- Attachment #2: Type: text/html, Size: 2692 bytes --]

^ permalink raw reply	[flat|nested] 61+ messages in thread

* Re: [PATCH v4 22/33] hostmem-epc: Add the reset interface for EPC backend reset
  2021-09-10 15:10   ` Paolo Bonzini
  2021-09-10 15:34     ` Sean Christopherson
@ 2021-09-13 20:17     ` Jarkko Sakkinen
  2021-09-13 20:37       ` Sean Christopherson
  1 sibling, 1 reply; 61+ messages in thread
From: Jarkko Sakkinen @ 2021-09-13 20:17 UTC (permalink / raw)
  To: Paolo Bonzini, qemu-devel, seanjc, kai.huang; +Cc: Yang Zhong, eblake

On Fri, 2021-09-10 at 17:10 +0200, Paolo Bonzini wrote:
> On 19/07/21 13:21, Yang Zhong wrote:
> > +void sgx_memory_backend_reset(HostMemoryBackend *backend, int fd,
> > +                              Error **errp)
> > +{
> > +    MemoryRegion *mr = &backend->mr;
> > +
> > +    mr->enabled = false;
> > +
> > +    /* destroy the old memory region if it exist */
> > +    if (fd > 0 && mr->destructor) {
> > +        mr->destructor(mr);
> > +    }
> > +
> > +    sgx_epc_backend_memory_alloc(backend, errp);
> > +}
> > +
> 
> Jarkko, Sean, Kai,
> 
> this I think is problematic because it has a race window while 
> /dev/sgx_vepc is closed and then reopened.  First, the vEPC space could 
> be exhausted by somebody doing another mmap in the meanwhile.  Second, 
> somebody might (for whatever reason) remove /dev/sgx_vepc while QEMU runs.

1: Why is it a problem that mmap() could fail?

2: Are you speaking about removing device node? If you have succesfully
   mapped /dev/sgx_vepc, that should not have much effect (file refcount).

/Jarkko



^ permalink raw reply	[flat|nested] 61+ messages in thread

* Re: [PATCH v4 22/33] hostmem-epc: Add the reset interface for EPC backend reset
  2021-09-13 20:17     ` Jarkko Sakkinen
@ 2021-09-13 20:37       ` Sean Christopherson
  2021-09-13 21:23         ` Jarkko Sakkinen
  0 siblings, 1 reply; 61+ messages in thread
From: Sean Christopherson @ 2021-09-13 20:37 UTC (permalink / raw)
  To: Jarkko Sakkinen; +Cc: Paolo Bonzini, qemu-devel, kai.huang, eblake, Yang Zhong

On Mon, Sep 13, 2021, Jarkko Sakkinen wrote:
> On Fri, 2021-09-10 at 17:10 +0200, Paolo Bonzini wrote:
> > On 19/07/21 13:21, Yang Zhong wrote:
> > > +void sgx_memory_backend_reset(HostMemoryBackend *backend, int fd,
> > > +                              Error **errp)
> > > +{
> > > +    MemoryRegion *mr = &backend->mr;
> > > +
> > > +    mr->enabled = false;
> > > +
> > > +    /* destroy the old memory region if it exist */
> > > +    if (fd > 0 && mr->destructor) {
> > > +        mr->destructor(mr);
> > > +    }
> > > +
> > > +    sgx_epc_backend_memory_alloc(backend, errp);
> > > +}
> > > +
> > 
> > Jarkko, Sean, Kai,
> > 
> > this I think is problematic because it has a race window while 
> > /dev/sgx_vepc is closed and then reopened.  First, the vEPC space could 
> > be exhausted by somebody doing another mmap in the meanwhile.  Second, 
> > somebody might (for whatever reason) remove /dev/sgx_vepc while QEMU runs.
> 
> 1: Why is it a problem that mmap() could fail?

The flow in question is QEMU's emulation of a guest RESET.  If mmap() fails, QEMU
either has to kill the VM or disable SGX.  In either case, it's fatal to a running
workload/VM.

> 2: Are you speaking about removing device node? If you have succesfully
>    mapped /dev/sgx_vepc, that should not have much effect (file refcount).

Paolo was calling out that doing munmap() before mmap() would allow /dev/sgx_vepc
to be removed because QEMU would no longer hold a reference to /dev/sgx_vepc.  That
would again be fatal to the VM as QEMU would not be able to re-mmap() guest EPC.


^ permalink raw reply	[flat|nested] 61+ messages in thread

* Re: [PATCH v4 22/33] hostmem-epc: Add the reset interface for EPC backend reset
  2021-09-13 20:37       ` Sean Christopherson
@ 2021-09-13 21:23         ` Jarkko Sakkinen
  0 siblings, 0 replies; 61+ messages in thread
From: Jarkko Sakkinen @ 2021-09-13 21:23 UTC (permalink / raw)
  To: Sean Christopherson
  Cc: Yang Zhong, Paolo Bonzini, kai.huang, eblake, qemu-devel

On Mon, 2021-09-13 at 20:37 +0000, Sean Christopherson wrote:
> On Mon, Sep 13, 2021, Jarkko Sakkinen wrote:
> > On Fri, 2021-09-10 at 17:10 +0200, Paolo Bonzini wrote:
> > > On 19/07/21 13:21, Yang Zhong wrote:
> > > > +void sgx_memory_backend_reset(HostMemoryBackend *backend, int fd,
> > > > +                              Error **errp)
> > > > +{
> > > > +    MemoryRegion *mr = &backend->mr;
> > > > +
> > > > +    mr->enabled = false;
> > > > +
> > > > +    /* destroy the old memory region if it exist */
> > > > +    if (fd > 0 && mr->destructor) {
> > > > +        mr->destructor(mr);
> > > > +    }
> > > > +
> > > > +    sgx_epc_backend_memory_alloc(backend, errp);
> > > > +}
> > > > +
> > > 
> > > Jarkko, Sean, Kai,
> > > 
> > > this I think is problematic because it has a race window while 
> > > /dev/sgx_vepc is closed and then reopened.  First, the vEPC space could 
> > > be exhausted by somebody doing another mmap in the meanwhile.  Second, 
> > > somebody might (for whatever reason) remove /dev/sgx_vepc while QEMU runs.
> > 
> > 1: Why is it a problem that mmap() could fail?
> 
> The flow in question is QEMU's emulation of a guest RESET.  If mmap() fails, QEMU
> either has to kill the VM or disable SGX.  In either case, it's fatal to a running
> workload/VM.

Thanks for the explanations.

Isn't this more about badly configured system/workloads? That's
at least for me the existential question.

I'm interested of legit workloads where this behaviour could still
cause any issues.

I'd guess than in e.g. data center environment, you'd have firly
strict orchestration for this type of resource so that you know
that workloads have an appropriate bandwidth.

/Jarkko


^ permalink raw reply	[flat|nested] 61+ messages in thread

* Re: [PATCH v4 04/33] i386: Add 'sgx-epc' device to expose EPC sections to guest
  2021-07-19 11:21 ` [PATCH v4 04/33] i386: Add 'sgx-epc' device to expose EPC sections to guest Yang Zhong
@ 2021-09-14  6:36   ` Philippe Mathieu-Daudé
  2021-09-16  1:29     ` Yang Zhong
  0 siblings, 1 reply; 61+ messages in thread
From: Philippe Mathieu-Daudé @ 2021-09-14  6:36 UTC (permalink / raw)
  To: Yang Zhong, qemu-devel; +Cc: seanjc, kai.huang, eblake, jarkko, pbonzini

On 7/19/21 1:21 PM, Yang Zhong wrote:
> From: Sean Christopherson <sean.j.christopherson@intel.com>
> 
> SGX EPC is enumerated through CPUID, i.e. EPC "devices" need to be
> realized prior to realizing the vCPUs themselves, which occurs long
> before generic devices are parsed and realized.  Because of this,
> do not allow 'sgx-epc' devices to be instantiated after vCPUS have
> been created.
> 
> The 'sgx-epc' device is essentially a placholder at this time, it will
> be fully implemented in a future patch along with a dedicated command
> to create 'sgx-epc' devices.
> 
> Signed-off-by: Sean Christopherson <sean.j.christopherson@intel.com>
> Signed-off-by: Yang Zhong <yang.zhong@intel.com>
> ---
>  hw/i386/meson.build       |   1 +
>  hw/i386/sgx-epc.c         | 161 ++++++++++++++++++++++++++++++++++++++
>  include/hw/i386/sgx-epc.h |  44 +++++++++++
>  3 files changed, 206 insertions(+)
>  create mode 100644 hw/i386/sgx-epc.c
>  create mode 100644 include/hw/i386/sgx-epc.h
> 
> diff --git a/hw/i386/meson.build b/hw/i386/meson.build
> index 80dad29f2b..27476b36bb 100644
> --- a/hw/i386/meson.build
> +++ b/hw/i386/meson.build
> @@ -5,6 +5,7 @@ i386_ss.add(files(
>    'e820_memory_layout.c',
>    'multiboot.c',
>    'x86.c',
> +  'sgx-epc.c',
>  ))
>  
>  i386_ss.add(when: 'CONFIG_X86_IOMMU', if_true: files('x86-iommu.c'),
> diff --git a/hw/i386/sgx-epc.c b/hw/i386/sgx-epc.c
> new file mode 100644
> index 0000000000..aa487dea79
> --- /dev/null
> +++ b/hw/i386/sgx-epc.c
> @@ -0,0 +1,161 @@
> +/*
> + * SGX EPC device
> + *
> + * Copyright (C) 2019 Intel Corporation
> + *
> + * Authors:
> + *   Sean Christopherson <sean.j.christopherson@intel.com>
> + *
> + * This work is licensed under the terms of the GNU GPL, version 2 or later.
> + * See the COPYING file in the top-level directory.
> + */
> +#include "qemu/osdep.h"
> +#include "hw/i386/pc.h"
> +#include "hw/i386/sgx-epc.h"
> +#include "hw/mem/memory-device.h"
> +#include "hw/qdev-properties.h"
> +#include "monitor/qdev.h"

Is that include used?

> +#include "qapi/error.h"
> +#include "qapi/visitor.h"
> +#include "qemu/config-file.h"

Ditto.

> +#include "qemu/error-report.h"

Ditto.

> +#include "qemu/option.h"

Ditto.

> +#include "qemu/units.h"

Ditto.

> +#include "target/i386/cpu.h"
> +#include "exec/address-spaces.h"



^ permalink raw reply	[flat|nested] 61+ messages in thread

* Re: [PATCH v4 10/33] i386: Add get/set/migrate support for SGX_LEPUBKEYHASH MSRs
  2021-07-19 11:21 ` [PATCH v4 10/33] i386: Add get/set/migrate support for SGX_LEPUBKEYHASH MSRs Yang Zhong
@ 2021-09-14  6:38   ` Philippe Mathieu-Daudé
  2021-09-16  6:08     ` Yang Zhong
  0 siblings, 1 reply; 61+ messages in thread
From: Philippe Mathieu-Daudé @ 2021-09-14  6:38 UTC (permalink / raw)
  To: Yang Zhong, qemu-devel; +Cc: seanjc, kai.huang, eblake, jarkko, pbonzini

On 7/19/21 1:21 PM, Yang Zhong wrote:
> From: Sean Christopherson <sean.j.christopherson@intel.com>
> 
> On real hardware, on systems that supports SGX Launch Control, those
> MSRs are initialized to digest of Intel's signing key; on systems that
> don't support SGX Launch Control, those MSRs are not available but
> hardware always uses digest of Intel's signing key in EINIT.
> 
> KVM advertises SGX LC via CPUID if and only if the MSRs are writable.
> Unconditionally initialize those MSRs to digest of Intel's signing key
> when CPU is realized and reset to reflect the fact. This avoids
> potential bug in case kvm_arch_put_registers() is called before
> kvm_arch_get_registers() is called, in which case guest's virtual
> SGX_LEPUBKEYHASH MSRs will be set to 0, although KVM initializes those
> to digest of Intel's signing key by default, since KVM allows those MSRs
> to be updated by Qemu to support live migration.
> 
> Save/restore the SGX Launch Enclave Public Key Hash MSRs if SGX Launch
> Control (LC) is exposed to the guest. Likewise, migrate the MSRs if they
> are writable by the guest.
> 
> Signed-off-by: Sean Christopherson <sean.j.christopherson@intel.com>
> Signed-off-by: Kai Huang <kai.huang@intel.com>
> Signed-off-by: Yang Zhong <yang.zhong@intel.com>
> ---
>  target/i386/cpu.c     | 17 ++++++++++++++++-
>  target/i386/cpu.h     |  1 +
>  target/i386/kvm/kvm.c | 22 ++++++++++++++++++++++
>  target/i386/machine.c | 20 ++++++++++++++++++++
>  4 files changed, 59 insertions(+), 1 deletion(-)
> 
> diff --git a/target/i386/cpu.c b/target/i386/cpu.c
> index 840f825431..cea4307930 100644
> --- a/target/i386/cpu.c
> +++ b/target/i386/cpu.c
> @@ -5673,6 +5673,16 @@ void cpu_x86_cpuid(CPUX86State *env, uint32_t index, uint32_t count,
>      }
>  }
>  
> +#ifndef CONFIG_USER_ONLY
> +static void x86_cpu_set_sgxlepubkeyhash(CPUX86State *env)
> +{
> +    env->msr_ia32_sgxlepubkeyhash[0] = 0xa6053e051270b7acULL;
> +    env->msr_ia32_sgxlepubkeyhash[1] = 0x6cfbe8ba8b3b413dULL;
> +    env->msr_ia32_sgxlepubkeyhash[2] = 0xc4916d99f2b3735dULL;
> +    env->msr_ia32_sgxlepubkeyhash[3] = 0xd4f8c05909f9bb3bULL;
> +}
> +#endif

Maybe easier to move the #ifdef'ry inside the function.

Where these values come from btw?

> @@ -6186,6 +6198,10 @@ static void x86_cpu_realizefn(DeviceState *dev, Error **errp)
>             & CPUID_EXT2_AMD_ALIASES);
>      }
>  
> +#ifndef CONFIG_USER_ONLY
> +    x86_cpu_set_sgxlepubkeyhash(env);
> +#endif
> +



^ permalink raw reply	[flat|nested] 61+ messages in thread

* Re: [PATCH v4 00/33] Qemu SGX virtualization
  2021-07-19 11:21 [PATCH v4 00/33] Qemu SGX virtualization Yang Zhong
                   ` (34 preceding siblings ...)
  2021-09-06 13:13 ` Paolo Bonzini
@ 2021-09-14  6:51 ` Philippe Mathieu-Daudé
  2021-09-15 12:24   ` Yang Zhong
  35 siblings, 1 reply; 61+ messages in thread
From: Philippe Mathieu-Daudé @ 2021-09-14  6:51 UTC (permalink / raw)
  To: Yang Zhong, qemu-devel; +Cc: seanjc, kai.huang, eblake, jarkko, pbonzini

On 7/19/21 1:21 PM, Yang Zhong wrote:

> This series is Qemu SGX virtualization implementation rebased on latest
> Qemu release. The numa support for SGX will be sent in another patchset
> once this basic SGX patchset are merged.

>  backends/hostmem-epc.c                   | 118 ++++++++++
>  backends/meson.build                     |   1 +
>  configs/devices/i386-softmmu/default.mak |   1 +
>  docs/intel-sgx.txt                       | 167 ++++++++++++++
>  hmp-commands-info.hx                     |  15 ++
>  hw/i386/Kconfig                          |   5 +
>  hw/i386/acpi-build.c                     |  22 ++
>  hw/i386/fw_cfg.c                         |  10 +-
>  hw/i386/meson.build                      |   2 +
>  hw/i386/pc.c                             |  15 +-
>  hw/i386/pc_piix.c                        |   4 +
>  hw/i386/pc_q35.c                         |   3 +
>  hw/i386/sgx-epc.c                        | 265 +++++++++++++++++++++++
>  hw/i386/sgx-stub.c                       |  13 ++
>  hw/i386/sgx.c                            | 170 +++++++++++++++
>  hw/i386/x86.c                            |  29 +++
>  hw/vfio/common.c                         |   1 +
>  include/exec/memory.h                    |  15 +-
>  include/hw/i386/pc.h                     |  10 +
>  include/hw/i386/sgx-epc.h                |  68 ++++++
>  include/hw/i386/x86.h                    |   1 +
>  include/monitor/hmp.h                    |   1 +
>  include/qemu/bitops.h                    |   7 +

Consider using scripts/git.orderfile to ease reviewers workflow.



^ permalink raw reply	[flat|nested] 61+ messages in thread

* Re: [PATCH v4 23/33] sgx-epc: Add the reset interface for sgx-epc virt device
  2021-07-19 11:21 ` [PATCH v4 23/33] sgx-epc: Add the reset interface for sgx-epc virt device Yang Zhong
  2021-09-10 15:13   ` Paolo Bonzini
@ 2021-09-14  6:53   ` Philippe Mathieu-Daudé
  2021-09-15 11:33     ` Yang Zhong
  1 sibling, 1 reply; 61+ messages in thread
From: Philippe Mathieu-Daudé @ 2021-09-14  6:53 UTC (permalink / raw)
  To: Yang Zhong, qemu-devel; +Cc: seanjc, kai.huang, eblake, jarkko, pbonzini

On 7/19/21 1:21 PM, Yang Zhong wrote:
> If the VM is reset, we need make sure sgx virt epc in clean status.
> Once the VM is reset, and sgx epc virt device will be reseted by
> reset callback registered by qemu_register_reset(). Since this epc
> virt device depend on backend, this reset will call backend reset
> interface to re-mmap epc to guest.
> 
> Signed-off-by: Yang Zhong <yang.zhong@intel.com>
> ---
>  hw/i386/sgx-epc.c | 94 ++++++++++++++++++++++++++++++++++++++++-------
>  1 file changed, 81 insertions(+), 13 deletions(-)
> 
> diff --git a/hw/i386/sgx-epc.c b/hw/i386/sgx-epc.c
> index 924dea22f0..9880d832d5 100644
> --- a/hw/i386/sgx-epc.c
> +++ b/hw/i386/sgx-epc.c
> @@ -18,6 +18,9 @@
>  #include "qapi/visitor.h"
>  #include "target/i386/cpu.h"
>  #include "exec/address-spaces.h"
> +#include "sysemu/reset.h"
> +
> +uint32_t epc_num;

Missing 'static' qualifier.



^ permalink raw reply	[flat|nested] 61+ messages in thread

* Re: [PATCH v4 23/33] sgx-epc: Add the reset interface for sgx-epc virt device
  2021-09-14  6:53   ` Philippe Mathieu-Daudé
@ 2021-09-15 11:33     ` Yang Zhong
  0 siblings, 0 replies; 61+ messages in thread
From: Yang Zhong @ 2021-09-15 11:33 UTC (permalink / raw)
  To: Philippe Mathieu-Daudé
  Cc: yang.zhong, seanjc, kai.huang, qemu-devel, jarkko, pbonzini, eblake

On Tue, Sep 14, 2021 at 08:53:48AM +0200, Philippe Mathieu-Daudé wrote:
> On 7/19/21 1:21 PM, Yang Zhong wrote:
> > If the VM is reset, we need make sure sgx virt epc in clean status.
> > Once the VM is reset, and sgx epc virt device will be reseted by
> > reset callback registered by qemu_register_reset(). Since this epc
> > virt device depend on backend, this reset will call backend reset
> > interface to re-mmap epc to guest.
> > 
> > Signed-off-by: Yang Zhong <yang.zhong@intel.com>
> > ---
> >  hw/i386/sgx-epc.c | 94 ++++++++++++++++++++++++++++++++++++++++-------
> >  1 file changed, 81 insertions(+), 13 deletions(-)
> > 
> > diff --git a/hw/i386/sgx-epc.c b/hw/i386/sgx-epc.c
> > index 924dea22f0..9880d832d5 100644
> > --- a/hw/i386/sgx-epc.c
> > +++ b/hw/i386/sgx-epc.c
> > @@ -18,6 +18,9 @@
> >  #include "qapi/visitor.h"
> >  #include "target/i386/cpu.h"
> >  #include "exec/address-spaces.h"
> > +#include "sysemu/reset.h"
> > +
> > +uint32_t epc_num;
> 
> Missing 'static' qualifier.

  Philippe, thanks! This Qemu SGX reset solution will be replaced by kernel
  patchset made by Paolo(https://www.spinics.net/lists/kvm/msg253930.html).

  Yang
 


^ permalink raw reply	[flat|nested] 61+ messages in thread

* Re: [PATCH v4 00/33] Qemu SGX virtualization
  2021-09-14  6:51 ` Philippe Mathieu-Daudé
@ 2021-09-15 12:24   ` Yang Zhong
  0 siblings, 0 replies; 61+ messages in thread
From: Yang Zhong @ 2021-09-15 12:24 UTC (permalink / raw)
  To: Philippe Mathieu-Daudé
  Cc: yang.zhong, seanjc, kai.huang, qemu-devel, jarkko, pbonzini, eblake

On Tue, Sep 14, 2021 at 08:51:47AM +0200, Philippe Mathieu-Daudé wrote:
> On 7/19/21 1:21 PM, Yang Zhong wrote:
> 
> > This series is Qemu SGX virtualization implementation rebased on latest
> > Qemu release. The numa support for SGX will be sent in another patchset
> > once this basic SGX patchset are merged.
> 
> >  backends/hostmem-epc.c                   | 118 ++++++++++
> >  backends/meson.build                     |   1 +
> >  configs/devices/i386-softmmu/default.mak |   1 +
> >  docs/intel-sgx.txt                       | 167 ++++++++++++++
> >  hmp-commands-info.hx                     |  15 ++
> >  hw/i386/Kconfig                          |   5 +
> >  hw/i386/acpi-build.c                     |  22 ++
> >  hw/i386/fw_cfg.c                         |  10 +-
> >  hw/i386/meson.build                      |   2 +
> >  hw/i386/pc.c                             |  15 +-
> >  hw/i386/pc_piix.c                        |   4 +
> >  hw/i386/pc_q35.c                         |   3 +
> >  hw/i386/sgx-epc.c                        | 265 +++++++++++++++++++++++
> >  hw/i386/sgx-stub.c                       |  13 ++
> >  hw/i386/sgx.c                            | 170 +++++++++++++++
> >  hw/i386/x86.c                            |  29 +++
> >  hw/vfio/common.c                         |   1 +
> >  include/exec/memory.h                    |  15 +-
> >  include/hw/i386/pc.h                     |  10 +
> >  include/hw/i386/sgx-epc.h                |  68 ++++++
> >  include/hw/i386/x86.h                    |   1 +
> >  include/monitor/hmp.h                    |   1 +
> >  include/qemu/bitops.h                    |   7 +
> 
> Consider using scripts/git.orderfile to ease reviewers workflow.

  Thanks! I never noticed this issue before, thanks!

  My ref command:
  git format-patch -32 --no-signature --cover-letter -O scripts/git.orderfile --subject-prefix="PATCH"

  Yang


^ permalink raw reply	[flat|nested] 61+ messages in thread

* Re: [PATCH v4 04/33] i386: Add 'sgx-epc' device to expose EPC sections to guest
  2021-09-14  6:36   ` Philippe Mathieu-Daudé
@ 2021-09-16  1:29     ` Yang Zhong
  0 siblings, 0 replies; 61+ messages in thread
From: Yang Zhong @ 2021-09-16  1:29 UTC (permalink / raw)
  To: Philippe Mathieu-Daudé
  Cc: yang.zhong, seanjc, kai.huang, qemu-devel, jarkko, pbonzini, eblake

On Tue, Sep 14, 2021 at 08:36:24AM +0200, Philippe Mathieu-Daudé wrote:
> On 7/19/21 1:21 PM, Yang Zhong wrote:
> > From: Sean Christopherson <sean.j.christopherson@intel.com>
> > 
> > SGX EPC is enumerated through CPUID, i.e. EPC "devices" need to be
> > realized prior to realizing the vCPUs themselves, which occurs long
> > before generic devices are parsed and realized.  Because of this,
> > do not allow 'sgx-epc' devices to be instantiated after vCPUS have
> > been created.
> > 
> > The 'sgx-epc' device is essentially a placholder at this time, it will
> > be fully implemented in a future patch along with a dedicated command
> > to create 'sgx-epc' devices.
> > 
> > Signed-off-by: Sean Christopherson <sean.j.christopherson@intel.com>
> > Signed-off-by: Yang Zhong <yang.zhong@intel.com>
> > ---
> >  hw/i386/meson.build       |   1 +
> >  hw/i386/sgx-epc.c         | 161 ++++++++++++++++++++++++++++++++++++++
> >  include/hw/i386/sgx-epc.h |  44 +++++++++++
> >  3 files changed, 206 insertions(+)
> >  create mode 100644 hw/i386/sgx-epc.c
> >  create mode 100644 include/hw/i386/sgx-epc.h
> > 
> > diff --git a/hw/i386/meson.build b/hw/i386/meson.build
> > index 80dad29f2b..27476b36bb 100644
> > --- a/hw/i386/meson.build
> > +++ b/hw/i386/meson.build
> > @@ -5,6 +5,7 @@ i386_ss.add(files(
> >    'e820_memory_layout.c',
> >    'multiboot.c',
> >    'x86.c',
> > +  'sgx-epc.c',
> >  ))
> >  
> >  i386_ss.add(when: 'CONFIG_X86_IOMMU', if_true: files('x86-iommu.c'),
> > diff --git a/hw/i386/sgx-epc.c b/hw/i386/sgx-epc.c
> > new file mode 100644
> > index 0000000000..aa487dea79
> > --- /dev/null
> > +++ b/hw/i386/sgx-epc.c
> > @@ -0,0 +1,161 @@
> > +/*
> > + * SGX EPC device
> > + *
> > + * Copyright (C) 2019 Intel Corporation
> > + *
> > + * Authors:
> > + *   Sean Christopherson <sean.j.christopherson@intel.com>
> > + *
> > + * This work is licensed under the terms of the GNU GPL, version 2 or later.
> > + * See the COPYING file in the top-level directory.
> > + */
> > +#include "qemu/osdep.h"
> > +#include "hw/i386/pc.h"
> > +#include "hw/i386/sgx-epc.h"
> > +#include "hw/mem/memory-device.h"
> > +#include "hw/qdev-properties.h"
> > +#include "monitor/qdev.h"
> 
> Is that include used?
> 
> > +#include "qapi/error.h"
> > +#include "qapi/visitor.h"
> > +#include "qemu/config-file.h"
> 
> Ditto.
> 
> > +#include "qemu/error-report.h"
> 
> Ditto.
> 
> > +#include "qemu/option.h"
> 
> Ditto.
> 
> > +#include "qemu/units.h"
> 
> Ditto.
>
 
  Thanks Philippe, those header files have been removed in the Paolo's
  gitlab(sgx branch), thanks!

  Yang

 
> > +#include "target/i386/cpu.h"
> > +#include "exec/address-spaces.h"


^ permalink raw reply	[flat|nested] 61+ messages in thread

* Re: [PATCH v4 10/33] i386: Add get/set/migrate support for SGX_LEPUBKEYHASH MSRs
  2021-09-14  6:38   ` Philippe Mathieu-Daudé
@ 2021-09-16  6:08     ` Yang Zhong
  2021-09-16  6:35       ` Philippe Mathieu-Daudé
  0 siblings, 1 reply; 61+ messages in thread
From: Yang Zhong @ 2021-09-16  6:08 UTC (permalink / raw)
  To: Philippe Mathieu-Daudé
  Cc: yang.zhong, seanjc, kai.huang, qemu-devel, jarkko, pbonzini, eblake

On Tue, Sep 14, 2021 at 08:38:59AM +0200, Philippe Mathieu-Daudé wrote:
> On 7/19/21 1:21 PM, Yang Zhong wrote:
> > From: Sean Christopherson <sean.j.christopherson@intel.com>
> > 
> > On real hardware, on systems that supports SGX Launch Control, those
> > MSRs are initialized to digest of Intel's signing key; on systems that
> > don't support SGX Launch Control, those MSRs are not available but
> > hardware always uses digest of Intel's signing key in EINIT.
> > 
> > KVM advertises SGX LC via CPUID if and only if the MSRs are writable.
> > Unconditionally initialize those MSRs to digest of Intel's signing key
> > when CPU is realized and reset to reflect the fact. This avoids
> > potential bug in case kvm_arch_put_registers() is called before
> > kvm_arch_get_registers() is called, in which case guest's virtual
> > SGX_LEPUBKEYHASH MSRs will be set to 0, although KVM initializes those
> > to digest of Intel's signing key by default, since KVM allows those MSRs
> > to be updated by Qemu to support live migration.
> > 
> > Save/restore the SGX Launch Enclave Public Key Hash MSRs if SGX Launch
> > Control (LC) is exposed to the guest. Likewise, migrate the MSRs if they
> > are writable by the guest.
> > 
> > Signed-off-by: Sean Christopherson <sean.j.christopherson@intel.com>
> > Signed-off-by: Kai Huang <kai.huang@intel.com>
> > Signed-off-by: Yang Zhong <yang.zhong@intel.com>
> > ---
> >  target/i386/cpu.c     | 17 ++++++++++++++++-
> >  target/i386/cpu.h     |  1 +
> >  target/i386/kvm/kvm.c | 22 ++++++++++++++++++++++
> >  target/i386/machine.c | 20 ++++++++++++++++++++
> >  4 files changed, 59 insertions(+), 1 deletion(-)
> > 
> > diff --git a/target/i386/cpu.c b/target/i386/cpu.c
> > index 840f825431..cea4307930 100644
> > --- a/target/i386/cpu.c
> > +++ b/target/i386/cpu.c
> > @@ -5673,6 +5673,16 @@ void cpu_x86_cpuid(CPUX86State *env, uint32_t index, uint32_t count,
> >      }
> >  }
> >  
> > +#ifndef CONFIG_USER_ONLY
> > +static void x86_cpu_set_sgxlepubkeyhash(CPUX86State *env)
> > +{
> > +    env->msr_ia32_sgxlepubkeyhash[0] = 0xa6053e051270b7acULL;
> > +    env->msr_ia32_sgxlepubkeyhash[1] = 0x6cfbe8ba8b3b413dULL;
> > +    env->msr_ia32_sgxlepubkeyhash[2] = 0xc4916d99f2b3735dULL;
> > +    env->msr_ia32_sgxlepubkeyhash[3] = 0xd4f8c05909f9bb3bULL;
> > +}
> > +#endif
> 
> Maybe easier to move the #ifdef'ry inside the function.
>

  Thanks for comments, since this function is pure void function, we can move this #ifdef
  into function.
 
> Where these values come from btw?

  Those MSR values are intel default values, which were defined in Skylake platform.

  Yang

> 
> > @@ -6186,6 +6198,10 @@ static void x86_cpu_realizefn(DeviceState *dev, Error **errp)
> >             & CPUID_EXT2_AMD_ALIASES);
> >      }
> >  
> > +#ifndef CONFIG_USER_ONLY
> > +    x86_cpu_set_sgxlepubkeyhash(env);
> > +#endif
> > +


^ permalink raw reply	[flat|nested] 61+ messages in thread

* Re: [PATCH v4 10/33] i386: Add get/set/migrate support for SGX_LEPUBKEYHASH MSRs
  2021-09-16  6:08     ` Yang Zhong
@ 2021-09-16  6:35       ` Philippe Mathieu-Daudé
  0 siblings, 0 replies; 61+ messages in thread
From: Philippe Mathieu-Daudé @ 2021-09-16  6:35 UTC (permalink / raw)
  To: Yang Zhong; +Cc: seanjc, kai.huang, qemu-devel, jarkko, pbonzini, eblake

On 9/16/21 8:08 AM, Yang Zhong wrote:
> On Tue, Sep 14, 2021 at 08:38:59AM +0200, Philippe Mathieu-Daudé wrote:
>> On 7/19/21 1:21 PM, Yang Zhong wrote:
>>> From: Sean Christopherson <sean.j.christopherson@intel.com>
>>>
>>> On real hardware, on systems that supports SGX Launch Control, those
>>> MSRs are initialized to digest of Intel's signing key; on systems that
>>> don't support SGX Launch Control, those MSRs are not available but
>>> hardware always uses digest of Intel's signing key in EINIT.
>>>
>>> KVM advertises SGX LC via CPUID if and only if the MSRs are writable.
>>> Unconditionally initialize those MSRs to digest of Intel's signing key
>>> when CPU is realized and reset to reflect the fact. This avoids
>>> potential bug in case kvm_arch_put_registers() is called before
>>> kvm_arch_get_registers() is called, in which case guest's virtual
>>> SGX_LEPUBKEYHASH MSRs will be set to 0, although KVM initializes those
>>> to digest of Intel's signing key by default, since KVM allows those MSRs
>>> to be updated by Qemu to support live migration.
>>>
>>> Save/restore the SGX Launch Enclave Public Key Hash MSRs if SGX Launch
>>> Control (LC) is exposed to the guest. Likewise, migrate the MSRs if they
>>> are writable by the guest.
>>>
>>> Signed-off-by: Sean Christopherson <sean.j.christopherson@intel.com>
>>> Signed-off-by: Kai Huang <kai.huang@intel.com>
>>> Signed-off-by: Yang Zhong <yang.zhong@intel.com>
>>> ---
>>>  target/i386/cpu.c     | 17 ++++++++++++++++-
>>>  target/i386/cpu.h     |  1 +
>>>  target/i386/kvm/kvm.c | 22 ++++++++++++++++++++++
>>>  target/i386/machine.c | 20 ++++++++++++++++++++
>>>  4 files changed, 59 insertions(+), 1 deletion(-)
>>>
>>> diff --git a/target/i386/cpu.c b/target/i386/cpu.c
>>> index 840f825431..cea4307930 100644
>>> --- a/target/i386/cpu.c
>>> +++ b/target/i386/cpu.c
>>> @@ -5673,6 +5673,16 @@ void cpu_x86_cpuid(CPUX86State *env, uint32_t index, uint32_t count,
>>>      }
>>>  }
>>>  
>>> +#ifndef CONFIG_USER_ONLY
>>> +static void x86_cpu_set_sgxlepubkeyhash(CPUX86State *env)
>>> +{
>>> +    env->msr_ia32_sgxlepubkeyhash[0] = 0xa6053e051270b7acULL;
>>> +    env->msr_ia32_sgxlepubkeyhash[1] = 0x6cfbe8ba8b3b413dULL;
>>> +    env->msr_ia32_sgxlepubkeyhash[2] = 0xc4916d99f2b3735dULL;
>>> +    env->msr_ia32_sgxlepubkeyhash[3] = 0xd4f8c05909f9bb3bULL;
>>> +}
>>> +#endif
>>
>> Maybe easier to move the #ifdef'ry inside the function.
>>
> 
>   Thanks for comments, since this function is pure void function, we can move this #ifdef
>   into function.
>  
>> Where these values come from btw?
> 
>   Those MSR values are intel default values, which were defined in Skylake platform.

Could you add a comment (and reference if possible) about them please?

Thanks,

Phil.



^ permalink raw reply	[flat|nested] 61+ messages in thread

end of thread, other threads:[~2021-09-16  6:36 UTC | newest]

Thread overview: 61+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2021-07-19 11:21 [PATCH v4 00/33] Qemu SGX virtualization Yang Zhong
2021-07-19 11:21 ` [PATCH v4 01/33] memory: Add RAM_PROTECTED flag to skip IOMMU mappings Yang Zhong
2021-07-19 11:21 ` [PATCH v4 02/33] hostmem: Add hostmem-epc as a backend for SGX EPC Yang Zhong
2021-07-19 11:21 ` [PATCH v4 03/33] qom: Add memory-backend-epc ObjectOptions support Yang Zhong
2021-07-19 11:21 ` [PATCH v4 04/33] i386: Add 'sgx-epc' device to expose EPC sections to guest Yang Zhong
2021-09-14  6:36   ` Philippe Mathieu-Daudé
2021-09-16  1:29     ` Yang Zhong
2021-07-19 11:21 ` [PATCH v4 05/33] vl: Add sgx compound properties to expose SGX " Yang Zhong
2021-07-19 11:21 ` [PATCH v4 06/33] i386: Add primary SGX CPUID and MSR defines Yang Zhong
2021-07-19 11:21 ` [PATCH v4 07/33] i386: Add SGX CPUID leaf FEAT_SGX_12_0_EAX Yang Zhong
2021-07-19 11:21 ` [PATCH v4 08/33] i386: Add SGX CPUID leaf FEAT_SGX_12_0_EBX Yang Zhong
2021-07-19 11:21 ` [PATCH v4 09/33] i386: Add SGX CPUID leaf FEAT_SGX_12_1_EAX Yang Zhong
2021-07-19 11:21 ` [PATCH v4 10/33] i386: Add get/set/migrate support for SGX_LEPUBKEYHASH MSRs Yang Zhong
2021-09-14  6:38   ` Philippe Mathieu-Daudé
2021-09-16  6:08     ` Yang Zhong
2021-09-16  6:35       ` Philippe Mathieu-Daudé
2021-07-19 11:21 ` [PATCH v4 11/33] i386: Add feature control MSR dependency when SGX is enabled Yang Zhong
2021-07-19 11:21 ` [PATCH v4 12/33] i386: Update SGX CPUID info according to hardware/KVM/user input Yang Zhong
2021-07-19 11:21 ` [PATCH v4 13/33] i386: kvm: Add support for exposing PROVISIONKEY to guest Yang Zhong
2021-07-19 11:21 ` [PATCH v4 14/33] i386: Propagate SGX CPUID sub-leafs to KVM Yang Zhong
2021-07-19 11:21 ` [PATCH v4 15/33] Adjust min CPUID level to 0x12 when SGX is enabled Yang Zhong
2021-07-19 11:21 ` [PATCH v4 16/33] hw/i386/fw_cfg: Set SGX bits in feature control fw_cfg accordingly Yang Zhong
2021-07-19 11:21 ` [PATCH v4 17/33] hw/i386/pc: Account for SGX EPC sections when calculating device memory Yang Zhong
2021-07-19 11:21 ` [PATCH v4 18/33] i386/pc: Add e820 entry for SGX EPC section(s) Yang Zhong
2021-07-19 11:21 ` [PATCH v4 19/33] i386: acpi: Add SGX EPC entry to ACPI tables Yang Zhong
2021-07-19 11:21 ` [PATCH v4 20/33] q35: Add support for SGX EPC Yang Zhong
2021-07-19 11:21 ` [PATCH v4 21/33] i440fx: " Yang Zhong
2021-07-19 11:21 ` [PATCH v4 22/33] hostmem-epc: Add the reset interface for EPC backend reset Yang Zhong
2021-09-10 15:10   ` Paolo Bonzini
2021-09-10 15:34     ` Sean Christopherson
2021-09-10 17:09       ` Paolo Bonzini
2021-09-10 17:34         ` Sean Christopherson
2021-09-10 19:51           ` Paolo Bonzini
2021-09-10 20:21             ` Sean Christopherson
2021-09-10 20:57               ` Paolo Bonzini
2021-09-13 20:17     ` Jarkko Sakkinen
2021-09-13 20:37       ` Sean Christopherson
2021-09-13 21:23         ` Jarkko Sakkinen
2021-07-19 11:21 ` [PATCH v4 23/33] sgx-epc: Add the reset interface for sgx-epc virt device Yang Zhong
2021-09-10 15:13   ` Paolo Bonzini
2021-09-14  6:53   ` Philippe Mathieu-Daudé
2021-09-15 11:33     ` Yang Zhong
2021-07-19 11:21 ` [PATCH v4 24/33] sgx-epc: Avoid bios reset during sgx epc initialization Yang Zhong
2021-07-19 11:21 ` [PATCH v4 25/33] hostmem-epc: Make prealloc consistent with qemu cmdline during reset Yang Zhong
2021-07-19 11:21 ` [PATCH v4 26/33] qmp: Add query-sgx command Yang Zhong
2021-07-19 11:21 ` [PATCH v4 27/33] hmp: Add 'info sgx' command Yang Zhong
2021-07-19 11:21 ` [PATCH v4 28/33] i386: Add sgx_get_info() interface Yang Zhong
2021-07-19 11:21 ` [PATCH v4 29/33] bitops: Support 32 and 64 bit mask macro Yang Zhong
2021-07-19 11:21 ` [PATCH v4 30/33] qmp: Add the qmp_query_sgx_capabilities() Yang Zhong
2021-07-19 11:21 ` [PATCH v4 31/33] Kconfig: Add CONFIG_SGX support Yang Zhong
2021-07-19 11:21 ` [PATCH v4 32/33] sgx-epc: Add the fill_device_info() callback support Yang Zhong
2021-07-19 11:21 ` [PATCH v4 33/33] doc: Add the SGX doc Yang Zhong
2021-07-28 15:57 ` [PATCH v4 00/33] Qemu SGX virtualization Paolo Bonzini
2021-07-29 12:27   ` Yang Zhong
2021-09-06 13:13 ` Paolo Bonzini
2021-09-07  2:24   ` Yang Zhong
2021-09-07  9:51   ` Yang Zhong
2021-09-07 13:35     ` Jarkko Sakkinen
2021-09-08  6:00     ` Paolo Bonzini
2021-09-14  6:51 ` Philippe Mathieu-Daudé
2021-09-15 12:24   ` Yang Zhong

This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.