Linux-Sgx Archive on
 help / color / Atom feed
From: Jarkko Sakkinen <>
	Jarkko Sakkinen <>,, Randy Dunlap <>
Subject: [PATCH v29 20/20] docs: x86/sgx: Document SGX micro architecture and kernel internals
Date: Wed, 22 Apr 2020 00:53:16 +0300
Message-ID: <> (raw)
In-Reply-To: <>

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #0: Type: text/plain; charset=a, Size: 9760 bytes --]

Document the Intel SGX kernel architecture. The fine-grained micro
architecture details can be looked up from Intel SDM Volume 3D.

Co-developed-by: Sean Christopherson <>
Signed-off-by: Sean Christopherson <>
Acked-by: Randy Dunlap <>
Signed-off-by: Jarkko Sakkinen <>
 Documentation/x86/index.rst |   1 +
 Documentation/x86/sgx.rst   | 206 ++++++++++++++++++++++++++++++++++++
 2 files changed, 207 insertions(+)
 create mode 100644 Documentation/x86/sgx.rst

diff --git a/Documentation/x86/index.rst b/Documentation/x86/index.rst
index a8de2fbc1caa..971f30a7d166 100644
--- a/Documentation/x86/index.rst
+++ b/Documentation/x86/index.rst
@@ -31,3 +31,4 @@ x86-specific Documentation
+   sgx
diff --git a/Documentation/x86/sgx.rst b/Documentation/x86/sgx.rst
new file mode 100644
index 000000000000..9609a3409ad1
--- /dev/null
+++ b/Documentation/x86/sgx.rst
@@ -0,0 +1,206 @@
+.. SPDX-License-Identifier: GPL-2.0
+*Software Guard eXtensions (SGX)* is a set of instructions that enable ring-3
+applications to set aside private regions of code and data. These regions are
+called enclaves. An enclave can be entered to a fixed set of entry points. Only
+a CPU running inside the enclave can access its code and data.
+The support can be determined by
+	``grep sgx /proc/cpuinfo``
+Enclave Page Cache
+SGX utilizes an *Enclave Page Cache (EPC)* to store pages that are associated
+with an enclave. It is contained in a BIOS reserved region of physical memory.
+Unlike pages used for regular memory, pages can only be accessed outside the
+enclave for different purposes with the instructions **ENCLS**, **ENCLV** and
+Direct memory accesses to an enclave can be only done by a CPU executing inside
+the enclave. An enclave can be entered with **ENCLU[EENTER]** to a fixed set of
+entry points. However, a CPU executing inside the enclave can do outside memory
+Page Types
+**SGX Enclave Control Structure (SECS)**
+   Enclave's address range, attributes and other global data are defined
+   by this structure.
+**Regular (REG)**
+   Regular EPC pages contain the code and data of an enclave.
+**Thread Control Structure (TCS)**
+   Thread Control Structure pages define the entry points to an enclave and
+   track the execution state of an enclave thread.
+**Version Array (VA)**
+   Version Array pages contain 512 slots, each of which can contain a version
+   number for a page evicted from the EPC.
+Enclave Page Cache Map
+The processor tracks EPC pages via the *Enclave Page Cache Map (EPCM)*.  EPCM
+contains an entry for each EPC page, which describes the owning enclave, access
+rights and page type among the other things.
+The permissions from EPCM is consulted if and only if walking the kernel page
+tables succeeds. The total permissions are thus a conjunction between page table
+and EPCM permissions.
+For all intents and purposes the SGX architecture allows the processor to
+invalidate all EPCM entries at will, i.e. requires that software be prepared to
+handle an EPCM fault at any time. The contents of EPC are encrypted with an
+ephemeral key, which is lost on power transitions.
+EPC management
+EPC pages do not have ``struct page`` instances. They are IO memory from kernel
+perspective. The consequence is that they are always mapped as shared memory.
+Kernel defines ``/dev/sgx/enclave`` that can be mapped as ``MAP_SHARED`` to
+define the address range for an enclave.
+EPC Over-subscription
+When the amount of free EPC pages goes below a low watermark the swapping thread
+starts reclaiming pages. The pages that do not have the **A** bit set are
+selected as victim pages.
+Launch Control
+SGX provides a launch control mechanism. After all enclave pages have been
+copied, kernel executes **ENCLS[EINIT]**, which initializes the enclave. Only
+after this the CPU can execute inside the enclave.
+This leaf function takes an RSA-3072 signature of the enclave measurement and an
+optional cryptographic token. Linux does not take advantage of launch tokens.
+The instruction checks that the signature is signed with the key defined in
+**IA32_SGXLEPUBKEYHASH?** MSRs and the measurement is correct. If so, the
+enclave is allowed to be executed.
+MSRs can be configured by the BIOS to be either readable or writable. Linux
+supports only writable configuration in order to give full control to the kernel
+on launch control policy. Readable configuration requires the use of previously
+mentioned launch tokens.
+The current kernel implementation supports only writable MSRs. The launch is
+performed by setting the MSRs to the hash of the enclave signer's public key.
+The alternative would be to have *a launch enclave* that would be signed with
+the key set into MSRs, which would then generate launch tokens for other
+enclaves. This would only make sense with read-only MSRs, and thus the option
+has been discarded.
+Local Attestation
+In local attestation an enclave creates a **REPORT** data structure with
+**ENCLS[EREPORT]**, which describes the origin of an enclave. In particular, it
+contains a AES-CMAC of the enclave contents signed with a report key unique to
+each processor. All enclaves have access to this key.
+This mechanism can also be used in addition as a communication channel as the
+**REPORT** data structure includes a 64-byte field for variable information.
+Remote Attestation
+For remote attestation (or provisioning) there are schemes available:
+* EPID scheme, which requires the use of Intel managed attestation service.
+* ECDSA scheme, which allows a 3rd party to act as an attestation service.
+Intel has released an open source *Quoting Enclave (QE)* and *Provisioning
+Certification Enclave (PCE)* for the ECDSA based scheme. A PCE is used to
+certify the locally used QE's.
+Intel also provides a proprietary of the PCE. This is a necessary when one
+needs to be able to prove that an enclave is running on real hardware. To
+achieve this the enclave needs to be rooted to the Intel's PKI, which obviously
+cannot be exposed to 3rd parties.
+Both schemes require **ATTRIBUTES.PROVISIONKEY** but only EPID scheme uses the
+on-die provisioning key. This privilege should be under normal conditions given
+only to QE's because uncontrolled use of attestation could be used by malware
+for benefit.
+Encryption engines
+In order to conceal the enclave data while it is out of the CPU package,
+memory controller has to be extended with an encryption engine. MC can then
+route incoming requests coming from CPU cores running in enclave mode to the
+encryption engine.
+In CPUs prior to Icelake, Memory Encryption Engine (MEE) is used to
+encrypt pages leaving the CPU caches. MEE uses a n-ary Merkle tree with root in
+SRAM to maintain integrity of the encrypted data. This provides integrity and
+anti-replay protection but does not scale to large memory sizes because the time
+required to update the Merkle tree grows logarithmically in relation to the
+memory size.
+CPUs starting from Icelake use Total Memory Encryption (TME) in the place of
+MEE. TME throws away the Merkle tree, which means losing integrity and
+anti-replay protection but also enables variable size memory pools for EPC.
+Using this attack for benefit would require an interposer on the system bus.
+Backing storage
+Backing storage is shared and not accounted. It is implemented as a private
+shmem file. Providing a backing storage in some form from user space is not
+possible - accounting would go to invalid state as reclaimed pages would get
+accounted to the processes of which behalf the kernel happened to be acting on.
+Access control
+`mmap()` permissions are capped by the enclave permissions. A direct
+consequence of this is that all the pages for an address range must be added
+before `mmap()` can be applied. Effectively an enclave page with minimum
+permission in the address range sets the permission cap for the mapping
+Usage Models
+Shared Library
+Sensitive data and the code that acts on it is partitioned from the application
+into a separate library. The library is then linked as a DSO which can be loaded
+into an enclave. The application can then make individual function calls into
+the enclave through special SGX instructions. A run-time within the enclave is
+configured to marshal function parameters into and out of the enclave and to
+call the correct library function.
+Application Container
+An application may be loaded into a container enclave which is specially
+configured with a library OS and run-time which permits the application to run.
+The enclave run-time and library OS work together to execute the application
+when a thread enters the enclave.
+"Intel® Software Guard Extensions: EPID Provisioning and Attestation Services"
+"Supporting Third Party Attestation for Intel® SGX with Intel® Data Center
+Attestation Primitives"

  parent reply index

Thread overview: 83+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2020-04-21 21:52 [PATCH v29 00/20] Intel SGX foundations Jarkko Sakkinen
2020-04-21 21:52 ` [PATCH v29 01/20] x86/sgx: Update MAINTAINERS Jarkko Sakkinen
2020-04-21 21:52 ` [PATCH v29 02/20] x86/cpufeatures: x86/msr: Add Intel SGX hardware bits Jarkko Sakkinen
2020-04-21 21:52 ` [PATCH v29 03/20] x86/cpufeatures: x86/msr: Intel SGX Launch Control " Jarkko Sakkinen
2020-04-21 21:53 ` [PATCH v29 04/20] x86/mm: x86/sgx: Signal SIGSEGV with PF_SGX Jarkko Sakkinen
2020-04-21 21:53 ` [PATCH v29 05/20] x86/sgx: Add SGX microarchitectural data structures Jarkko Sakkinen
2020-04-21 21:53 ` [PATCH v29 06/20] x86/sgx: Add wrappers for ENCLS leaf functions Jarkko Sakkinen
2020-04-21 21:53 ` [PATCH v29 07/20] x86/cpu/intel: Detect SGX support Jarkko Sakkinen
2020-04-21 21:53 ` [PATCH v29 08/20] x86/sgx: Enumerate and track EPC sections Jarkko Sakkinen
2020-04-21 21:53 ` [PATCH v29 09/20] x86/sgx: Add functions to allocate and free EPC pages Jarkko Sakkinen
2020-04-21 21:53 ` [PATCH v29 10/20] mm: Introduce vm_ops->may_mprotect() Jarkko Sakkinen
2020-04-21 21:53 ` [PATCH v29 11/20] x86/sgx: Linux Enclave Driver Jarkko Sakkinen
2020-05-08 19:09   ` Sean Christopherson
2020-04-21 21:53 ` [PATCH v29 12/20] x86/sgx: Add provisioning Jarkko Sakkinen
2020-04-21 21:53 ` [PATCH v29 13/20] x86/sgx: Add a page reclaimer Jarkko Sakkinen
2020-04-21 21:53 ` [PATCH v29 14/20] x86/sgx: ptrace() support for the SGX driver Jarkko Sakkinen
2020-05-06 21:50   ` Sean Christopherson
2020-05-13 21:40     ` Jarkko Sakkinen
2020-04-21 21:53 ` [PATCH v29 15/20] x86/vdso: Add support for exception fixup in vDSO functions Jarkko Sakkinen
2020-04-21 21:53 ` [PATCH v29 16/20] x86/fault: Add helper function to sanitize error code Jarkko Sakkinen
2020-04-21 21:53 ` [PATCH v29 17/20] x86/traps: Attempt to fixup exceptions in vDSO before signaling Jarkko Sakkinen
2020-04-21 21:53 ` [PATCH v29 18/20] x86/vdso: Implement a vDSO for Intel SGX enclave call Jarkko Sakkinen
2020-04-21 21:53 ` [PATCH v29 19/20] selftests/x86: Add a selftest for SGX Jarkko Sakkinen
2020-04-21 21:53 ` Jarkko Sakkinen [this message]
2020-04-22 16:48 ` [PATCH v29 00/20] Intel SGX foundations Connor Kuehl
2020-04-29  5:22   ` Jarkko Sakkinen
2020-04-26 16:57 ` Dr. Greg
2020-04-29  5:23   ` Jarkko Sakkinen
2020-04-29 15:14     ` Sean Christopherson
2020-04-30  3:59       ` Jarkko Sakkinen
2020-05-02 23:05         ` Dr. Greg
2020-05-03  0:48           ` Andy Lutomirski
2020-05-04  9:34             ` Dr. Greg
2020-04-29 15:30   ` Sean Christopherson
2020-05-08  0:40     ` Andy Lutomirski
2020-05-07  0:41   ` Thomas Gleixner
2020-05-08 19:02     ` Dr. Greg
2020-05-08 19:56       ` Sean Christopherson
2020-05-14  9:16         ` Dr. Greg
2020-05-14 16:15           ` Sean Christopherson
2020-05-14 16:20             ` Borislav Petkov
2020-05-14 19:29               ` Thomas Gleixner
2020-05-15  9:28               ` Jarkko Sakkinen
2020-05-15  9:42                 ` Borislav Petkov
2020-05-15 16:24                   ` Jarkko Sakkinen
2020-05-15  0:09             ` Jarkko Sakkinen
2020-05-15 19:54           ` Nathaniel McCallum
2020-05-16  9:58             ` Jarkko Sakkinen
2020-05-24 21:27           ` Pavel Machek
2020-05-26  8:16             ` David Laight
2020-04-26 17:03 ` Dr. Greg
2020-04-29 15:27 ` Jethro Beekman
2020-04-30  3:46   ` Jarkko Sakkinen
2020-04-30  7:19     ` Jethro Beekman
2020-04-30  8:23       ` Jarkko Sakkinen
2020-04-30 14:12         ` Jethro Beekman
2020-05-06 12:16           ` Jarkko Sakkinen
2020-05-06 16:39 ` Jordan Hand
2020-05-07 18:06   ` Dr. Greg
2020-05-08 16:16     ` Jordan Hand
2020-05-13 23:09   ` Jarkko Sakkinen
2020-05-06 21:42 ` Nathaniel McCallum
2020-05-06 22:14   ` Sean Christopherson
2020-05-07  5:02     ` Haitao Huang
2020-05-07 16:49       ` Nathaniel McCallum
2020-05-07 19:34         ` Sean Christopherson
2020-05-07 22:35           ` Haitao Huang
2020-05-08  0:25             ` Sean Christopherson
2020-05-28 11:15               ` Jarkko Sakkinen
2020-05-28 11:19                 ` Jarkko Sakkinen
2020-05-07 22:31         ` Haitao Huang
2020-05-13 22:14   ` Jarkko Sakkinen
2020-05-13 22:18     ` Jarkko Sakkinen
2020-05-14  6:31     ` Jethro Beekman
2020-05-14 19:30     ` Thomas Gleixner
2020-05-15  0:22       ` Jarkko Sakkinen
2020-05-08 19:08 ` Sean Christopherson
2020-05-14 19:05   ` Seth Moore
2020-05-15  0:23     ` Jarkko Sakkinen
2020-05-12 11:55 ` Hui, Chunyang
2020-05-12 16:51   ` Dr. Greg
2020-05-14 10:49   ` Jarkko Sakkinen
2020-05-16  8:53 ` [PATCH] x86/cpu/intel: Add nosgx kernel parameter Jarkko Sakkinen

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link

Linux-Sgx Archive on

Archives are clonable:
	git clone --mirror linux-sgx/git/0.git

	# If you have public-inbox 1.1+ installed, you may
	# initialize and index your mirror using the following commands:
	public-inbox-init -V2 linux-sgx linux-sgx/ \
	public-inbox-index linux-sgx

Example config snippet for mirrors

Newsgroup available over NNTP:

AGPL code for this site: git clone