Linux-Sgx Archive on
 help / color / Atom feed
From: Jarkko Sakkinen <>
	Jarkko Sakkinen <>,
	Jonathan Corbet <>, Ingo Molnar <>,
	Borislav Petkov <>, "H. Peter Anvin" <>, (open list:DOCUMENTATION), (open list)
Subject: [PATCH v16 22/22] x86/sgx: SGX documentation
Date: Tue,  6 Nov 2018 15:46:01 +0200
Message-ID: <> (raw)
Message-ID: <20181106134601.6anRugpB-AxxK_QlPmH5K05drLhSkbkERul7WCFTbgk@z> (raw)
In-Reply-To: <>

Documentation of the features of the Software Guard eXtensions used
by the Linux kernel and basic design choices for the core and driver
and functionality.

Signed-off-by: Jarkko Sakkinen <>
 Documentation/index.rst         |   1 +
 Documentation/x86/index.rst     |   8 ++
 Documentation/x86/intel_sgx.rst | 201 ++++++++++++++++++++++++++++++++
 3 files changed, 210 insertions(+)
 create mode 100644 Documentation/x86/index.rst
 create mode 100644 Documentation/x86/intel_sgx.rst

diff --git a/Documentation/index.rst b/Documentation/index.rst
index 5db7e87c7cb1..1cdc139adb40 100644
--- a/Documentation/index.rst
+++ b/Documentation/index.rst
@@ -104,6 +104,7 @@ implementation.
    :maxdepth: 2
+   x86/index
 Filesystem Documentation
diff --git a/Documentation/x86/index.rst b/Documentation/x86/index.rst
new file mode 100644
index 000000000000..11d5b18d9537
--- /dev/null
+++ b/Documentation/x86/index.rst
@@ -0,0 +1,8 @@
+x86 Architecture Guide
+.. toctree::
+   :maxdepth: 2
+   intel_sgx
diff --git a/Documentation/x86/intel_sgx.rst b/Documentation/x86/intel_sgx.rst
new file mode 100644
index 000000000000..096756c56070
--- /dev/null
+++ b/Documentation/x86/intel_sgx.rst
@@ -0,0 +1,201 @@
+Intel(R) SGX driver
+Intel(R) SGX is a set of CPU instructions that can be used by applications to
+set aside private regions of code and data. The code outside the enclave is
+disallowed to access the memory inside the enclave by the CPU access control.
+In a way you can think that SGX provides inverted sandbox. It protects the
+application from a malicious host.
+You can tell if your CPU supports SGX by looking into ``/proc/cpuinfo``:
+	``cat /proc/cpuinfo  | grep sgx``
+Overview of SGX
+SGX has a set of data structures to maintain information about the enclaves and
+their security properties. BIOS reserves a fixed size region of physical memory
+for these structures by setting Processor Reserved Memory Range Registers
+This memory range is protected from outside access by the CPU and all the data
+coming in and out of the CPU package is encrypted by a key that is generated for
+each boot cycle.
+Enclaves execute in ring-3 in a special enclave submode using pages from the
+reserved memory range. A fixed logical address range for the enclave is reserved
+by ENCLS(ECREATE), a leaf instruction used to create enclaves. It is referred in
+the documentation commonly as the ELRANGE.
+Every memory access to the ELRANGE is asserted by the CPU. If the CPU is not
+executing in the enclave mode inside the enclave, #GP is raised. On the other
+hand, enclave code can make memory accesses both inside and outside of the
+Enclave can only execute code inside the ELRANGE. Instructions that may cause
+VMEXIT, IO instructions and instructions that require a privilege change are
+prohibited inside the enclave. Interrupts and exceptions always cause enclave
+to exit and jump to an address outside the enclave given when the enclave is
+entered by using the leaf instruction ENCLS(EENTER).
+Protected memory
+Enclave Page Cache (EPC)
+    Physical pages used with enclaves that are protected by the CPU from
+    unauthorized access.
+Enclave Page Cache Map (EPCM)
+    A database that describes the properties and state of the pages e.g. their
+    permissions or to which enclave they belong to.
+Memory Encryption Engine (MEE) integrity tree
+    Autonomously updated integrity tree. The root of the tree located in on-die
+    SRAM.
+EPC data types
+SGX Enclave Control Structure (SECS)
+    Describes the global properties of an enclave. Will not be mapped to the
+Regular (REG)
+    These pages contain code and data.
+Thread Control Structure (TCS)
+    The pages that define the entry points inside an enclave. An enclave can
+    only be entered through these entry points and each can host a single
+    hardware thread at a time.
+Version Array (VA)
+   The pages contain 64-bit version numbers for pages that have been swapped
+   outside the enclave. Each page has the capacity of 512 version numbers.
+Launch control
+To launch an enclave, two structures must be provided for ENCLS(EINIT):
+1. **SIGSTRUCT:** signed measurement of the enclave binary.
+2. **EINITTOKEN:** a cryptographic token CMAC-signed with a AES256-key called
+   *launch key*, which is re-generated for each boot cycle.
+The CPU holds a SHA256 hash of a 3072-bit RSA public key inside
+IA32_SGXLEPUBKEYHASHn MSRs. Enclaves with a SIGSTRUCT that is signed with this
+key do not require a valid EINITTOKEN and can be authorized with special
+privileges. One of those privileges is ability to acquire the launch key with
+**IA32_FEATURE_CONTROL[17]** is used by the BIOS configure whether
+IA32_SGXLEPUBKEYHASH MSRs are read-only or read-write before locking the
+feature control register and handing over control to the operating system.
+Enclave construction
+The construction is started by filling out the SECS that contains enclave
+address range, privileged attributes and measurement of TCS and REG pages (pages
+that will be mapped to the address range) among the other things. This structure
+is passed out to the ENCLS(ECREATE) together with a physical address of a page
+in EPC that will hold the SECS.
+The pages are added with ENCLS(EADD) and measured with ENCLS(EEXTEND) i.e.
+SHA256 hash MRENCLAVE residing in the SECS is extended with the page data.
+After all of the pages have been added, the enclave is initialized with
+ENCLS(EINIT). It will check that the SIGSTRUCT is signed with the contained
+public key. If the given EINITTOKEN has the valid bit set, the CPU checks that
+the token is valid (CMAC'd with the launch key). If the token is not valid,
+the CPU will check whether the enclave is signed with a key matching to the
+Swapping pages
+Enclave pages can be swapped out with ENCLS(EWB) to the unprotected memory. In
+addition to the EPC page, ENCLS(EWB) takes in a VA page and address for PCMD
+structure (Page Crypto MetaData) as input. The VA page will seal a version
+number for the page. PCMD is 128 byte structure that contains tracking
+information for the page, most importantly its MAC. With these structures the
+enclave is sealed and rollback protected while it resides in the unprotected
+Before the page can be swapped out it must not have any active TLB references.
+ENCLS(EBLOCK) instruction moves a page to the *blocked* state, which means
+that no new TLB entries can be created to it by the hardware threads.
+After this a shootdown sequence is started with ENCLS(ETRACK), which sets an
+increased counter value to the entering hardware threads. ENCLS(EWB) will
+return SGX_NOT_TRACKED error while there are still threads with the earlier
+couner value because that means that there might be hardware thread inside
+the enclave with TLB entries to pages that are to be swapped.
+Kernel internals
+Because SGX has an ever evolving and expanding feature set, it's possible for
+a BIOS or VMM to configure a system in such a way that not all CPUs are equal,
+e.g. where Launch Control is only enabled on a subset of CPUs.  Linux does
+*not* support such a heterogeneous system configuration, nor does it even
+attempt to play nice in the face of a misconfigured system.  With the exception
+of Launch Control's hash MSRs, which can vary per CPU, Linux assumes that all
+CPUs have a configuration that is identical to the boot CPU.
+Roles and responsibilities
+SGX introduces system resources, e.g. EPC memory, that must be accessible to
+multiple entities, e.g. the native kernel driver (to expose SGX to userspace)
+and KVM (to expose SGX to VMs), ideally without introducing any dependencies
+between each SGX entity.  To that end, the kernel owns and manages the shared
+system resources, i.e. the EPC and Launch Control MSRs, and defines functions
+that provide appropriate access to the shared resources.  SGX support for
+user space and VMs is left to the SGX platform driver and KVM respectively.
+Launching enclaves
+The current kernel implementation supports only unlocked MSRs i.e.
+FEATURE_CONTROL_SGX_LE_WR must be set. The launch is performed by setting the
+MSRs to the hash of the public key modulus of the enclave signer, which is one
+of the fields in the SIGSTRUCT.
+EPC management
+Due to the unique requirements for swapping EPC pages, and because EPC pages
+(currently) do not have associated page structures, management of the EPC is
+not handled by the standard Linux swapper.  SGX directly handles swapping
+of EPC pages, including a kthread to initiate reclaim and a rudimentary LRU
+mechanism. The consumers of EPC pages, e.g. the SGX driver, are required to
+implement function callbacks that can be invoked by the kernel to age,
+swap, and/or forcefully reclaim a target EPC page.  In effect, the kernel
+controls what happens and when, while the consumers (driver, KVM, etc..) do
+the actual work.
+SGX uapi
+.. kernel-doc:: drivers/platform/x86/intel_sgx/sgx_ioctl.c
+   :functions: sgx_ioc_enclave_create
+               sgx_ioc_enclave_add_page
+               sgx_ioc_enclave_init
+.. kernel-doc:: arch/x86/include/uapi/asm/sgx.h
+* A Memory Encryption Engine Suitable for General Purpose Processors
+  <>
+* System Programming Manual: 39.1.4 Intel® SGX Launch Control Configuration

  parent reply index

Thread overview: 61+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2018-11-06 13:45 [PATCH v16 00/22] Intel SGX1 support Jarkko Sakkinen
2018-11-06 13:45 ` Jarkko Sakkinen
2018-11-06 13:45 ` [PATCH v16 01/22] x86/sgx: Update MAINTAINERS Jarkko Sakkinen
2018-11-06 13:45   ` Jarkko Sakkinen
2018-11-06 13:45 ` [PATCH v16 02/22] x86/cpufeatures: Add Intel-defined SGX feature bit Jarkko Sakkinen
2018-11-06 13:45   ` Jarkko Sakkinen
2018-11-06 13:45 ` [PATCH v16 03/22] x86/cpufeatures: Add SGX sub-features (as Linux-defined bits) Jarkko Sakkinen
2018-11-06 13:45   ` Jarkko Sakkinen
2018-11-06 13:45 ` [PATCH v16 04/22] x86/msr: Add IA32_FEATURE_CONTROL.SGX_ENABLE definition Jarkko Sakkinen
2018-11-06 13:45   ` Jarkko Sakkinen
2018-11-06 13:45 ` [PATCH v16 05/22] x86/cpufeatures: Add Intel-defined SGX_LC feature bit Jarkko Sakkinen
2018-11-06 13:45   ` Jarkko Sakkinen
2018-11-06 13:45 ` [PATCH v16 06/22] x86/cpu/intel: Detect SGX support and update caps appropriately Jarkko Sakkinen
2018-11-06 13:45   ` Jarkko Sakkinen
2018-11-06 13:58   ` Sean Christopherson
2018-11-06 13:58     ` Sean Christopherson
2018-11-07 15:58     ` Jarkko Sakkinen
2018-11-06 13:45 ` [PATCH v16 07/22] x86/mm: x86/sgx: Add new 'PF_SGX' page fault error code bit Jarkko Sakkinen
2018-11-06 13:45   ` Jarkko Sakkinen
2018-11-06 13:45 ` [PATCH v16 08/22] x86/mm: x86/sgx: Signal SIGSEGV for userspace #PFs w/ PF_SGX Jarkko Sakkinen
2018-11-06 13:45   ` Jarkko Sakkinen
2018-11-06 13:45 ` [PATCH v16 09/22] x86/sgx: Define SGX1 and SGX2 ENCLS leafs Jarkko Sakkinen
2018-11-06 13:45   ` Jarkko Sakkinen
2018-11-06 13:45 ` [PATCH v16 10/22] x86/sgx: Add ENCLS architectural error codes Jarkko Sakkinen
2018-11-06 13:45   ` Jarkko Sakkinen
2018-11-06 13:45 ` [PATCH v16 11/22] x86/sgx: Add SGX1 and SGX2 architectural data structures Jarkko Sakkinen
2018-11-06 13:45   ` Jarkko Sakkinen
2018-11-06 13:45 ` [PATCH v16 12/22] x86/sgx: Add definitions for SGX's CPUID leaf and variable sub-leafs Jarkko Sakkinen
2018-11-06 13:45   ` Jarkko Sakkinen
2018-11-06 13:45 ` [PATCH v16 13/22] x86/msr: Add SGX Launch Control MSR definitions Jarkko Sakkinen
2018-11-06 13:45   ` Jarkko Sakkinen
2018-11-06 13:45 ` [PATCH v16 14/22] x86/sgx: Add wrappers for ENCLS leaf functions Jarkko Sakkinen
2018-11-06 13:45   ` Jarkko Sakkinen
2018-11-06 13:45 ` [PATCH v16 15/22] x86/sgx: Enumerate and track EPC sections Jarkko Sakkinen
2018-11-06 13:45   ` Jarkko Sakkinen
2018-11-06 13:45 ` [PATCH v16 16/22] x86/sgx: Add functions to allocate and free EPC pages Jarkko Sakkinen
2018-11-06 13:45   ` Jarkko Sakkinen
2018-11-06 13:45 ` [PATCH v16 17/22] x86/sgx: Add sgx_einit() for initializing enclaves Jarkko Sakkinen
2018-11-06 13:45   ` Jarkko Sakkinen
2018-11-06 13:45 ` [PATCH v16 18/22] platform/x86: Intel SGX driver Jarkko Sakkinen
2018-11-06 13:45   ` Jarkko Sakkinen
2018-11-06 16:40   ` Sean Christopherson
2018-11-06 16:57     ` Dave Hansen
2018-11-07 16:37     ` Jarkko Sakkinen
2018-11-07 18:00       ` Sean Christopherson
2018-11-07 18:00         ` Sean Christopherson
2018-11-08 14:46         ` Jarkko Sakkinen
2018-11-15 20:00           ` Jarkko Sakkinen
2018-11-15 20:04             ` Jarkko Sakkinen
2018-11-15 20:16               ` Jarkko Sakkinen
2018-11-21 11:46                 ` Jarkko Sakkinen
2018-11-07 10:29   ` David Laight
2018-11-06 13:45 ` [PATCH v16 19/22] platform/x86: sgx: Add swapping functionality to the " Jarkko Sakkinen
2018-11-06 13:45   ` Jarkko Sakkinen
2018-11-06 13:45 ` [PATCH v16 20/22] x86/sgx: Add a simple swapper for the EPC memory manager Jarkko Sakkinen
2018-11-06 13:45   ` Jarkko Sakkinen
2018-11-06 13:46 ` [PATCH v16 21/22] platform/x86: ptrace() support for the SGX driver Jarkko Sakkinen
2018-11-06 13:46   ` Jarkko Sakkinen
2018-11-06 13:46 ` Jarkko Sakkinen [this message]
2018-11-06 13:46   ` [PATCH v16 22/22] x86/sgx: SGX documentation Jarkko Sakkinen
2018-11-27 20:13   ` Pavel Machek

Reply instructions:

You may reply publically to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link

Linux-Sgx Archive on

Archives are clonable:
	git clone --mirror linux-sgx/git/0.git

	# If you have public-inbox 1.1+ installed, you may
	# initialize and index your mirror using the following commands:
	public-inbox-init -V2 linux-sgx linux-sgx/ \
	public-inbox-index linux-sgx

Newsgroup available over NNTP:

AGPL code for this site: git clone public-inbox