[PATCH] sgx.7: New page with overview of Software Guard eXtensions (SGX)

[PATCH] sgx.7: New page with overview of Software Guard eXtensions (SGX)

[Jarkko Sakkinen]

Signed-off-by: Jarkko Sakkinen <jarkko@kernel.org>
---
 man7/sgx.7 | 198 +++++++++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 198 insertions(+)
 create mode 100644 man7/sgx.7

diff --git a/man7/sgx.7 b/man7/sgx.7
new file mode 100644
index 000000000..429c9b64d
--- /dev/null
+++ b/man7/sgx.7
@@ -0,0 +1,198 @@
+.\" Copyright (C) 2020 Intel Corporation
+.\"
+.\" %%%LICENSE_START(VERBATIM)
+.\" Permission is granted to make and distribute verbatim copies of this
+.\" manual provided the copyright notice and this permission notice are
+.\" preserved on all copies.
+.\"
+.\" Permission is granted to copy and distribute modified versions of this
+.\" manual under the conditions for verbatim copying, provided that the
+.\" entire resulting derived work is distributed under the terms of a
+.\" permission notice identical to this one.
+.\"
+.\" Since the Linux kernel and libraries are constantly changing, this
+.\" manual page may be incorrect or out-of-date.  The author(s) assume no
+.\" responsibility for errors or omissions, or for damages resulting from
+.\" the use of the information contained herein.  The author(s) may not
+.\" have taken the same level of care in the production of this manual,
+.\" which is licensed free of charge, as they might when working
+.\" professionally.
+.\"
+.\" Formatted or processed versions of this manual, if unaccompanied by
+.\" the source, must acknowledge the copyright and authors of this work.
+.\" %%%LICENSE_END
+.\"
+.TH SGX 7 2020-12-02 "Linux" "Linux Programmer's Manual"
+.PP
+sgx - overview of Software Guard eXtensions
+.SH DESCRIPTION
+.PP
+Intel Software Guard eXtensions (SGX) allow user space applications to
+set aside private memory regions of code and data.
+These memory regions are called as enclaves.
+.PP
+SGX must be enabled by the BIOS.
+If SGX appears to be unsupported on a system having the hardware
+support, ensure that SGX is enabled in the BIOS.
+If a BIOS presents a choice between \[lq]Enabled\[rq] and \[lq]Software
+Enabled\[rq] modes for SGX, choose \[lq]Enabled\[rq].
+.PP
+An enclave can be only entered at a fixed set of entry points.
+Each of them can hold a single hardware thread at a time.
+While the enclave is loaded from a regular binary file, only the threads
+inside the enclave can access its memory.
+.PP
+Although carved out of normal DRAM, enclave memory is marked in the
+system memory map as reserved and is not managed by the Linux memory
+manager.
+There may be several regions spread across the system.
+Each contiguous region is called an Enclave Page Cache (EPC) section.
+EPC sections are enumerated via CPUID.
+These regions are encrypted when they leave the LLC.
+.PP
+SGX is available only if the kernel was configured and built with the
+\f[B]CONFIG_X86_SGX\f[R] option.
+The hardware support for SGX can be observed from
+\f[I]/proc/cpuinfo\f[R] with the \[lq]flags\[rq] field containing
+\[lq]sgx\[rq].
+.SS Enclave management
+.PP
+Enclave\[cq]s life-cycle starts by opening \f[I]/dev/sgx_enclave\f[R],
+and ends once all the references to the opened file have been closed.
+After opening the enclave, its contents must be populated with the ioctl
+interface provided by \f[I]<asm/sgx.h>\f[R].
+ENCLS is a privileged (ring-0) instruction with the functionality for
+managing enclave memory, and the ioctl inteface provides a wrapper for
+it.
+.PP
+Enclave construction starts by calling \f[B]SGX_IOC_ENCLAVE_CREATE\f[R],
+which takes in the initial version of SGX Enclave Control Structure
+(SECS).
+SGX Enclave Control Structure (SECS) contains the description of the
+enclave.
+The ioctl calls ENCLS[ECREATE] function, which will copy SECS to the SGX
+reserved memory.
+SECS is never mapped to the process address space, and thus cannot be
+directly referenced.
+.PP
+Among other things, SECS contains the base address and size of the
+enclave, meaning that the addresss space must be carved out before the
+creation.
+There is also a hardware constrain that the size must be a power of two,
+and the base address must be also naturally aligned relative to the
+size.
+.PP
+Some of the SECS fields must be initialized to zero because their values
+are determined dynamically after the enclave has been created.
+Most importantly SECS contains two SHA256 hashes: MRENCLAVE and
+MRSIGNER.
+Each enclave invocation, during the enclave construction, hashes its
+defining parameters MRENCLAVE.
+Arbitrary data can be also hashed into MRENCLAVE via the ENCLS[EEXTEND]
+function.
+MRSIGNER contains the hash of the enclave signer key, which is used to
+sign the SIGSTRUCT structure passed to the ENCLS[EINIT] function.
+A running enclave can use both of these fields as material for keys
+acquired with the ENCLU[EGETKEY]unction.
+.PP
+After the enclave has been created, a series of
+\f[B]SGX_IOC_ENCLAVE_ADD_PAGES\f[R]calls are issued.
+This ioctl copies the provided data to the enclave memory by invoking
+the ENCLS[EADD] function, and optionally hashes it with help of the
+ENCLS[EEXTEND] function.
+Hashing is optional because it is a slow operation and sometimes not
+required.
+For instance, one might only care to measure the code pages.
+.PP
+Finally, when all the data has been copied to the enclave memory,
+\f[B]SGX_IOC_ENCLAVE_INIT\f[R] is called, which invokes the ENCLS[EINIT]
+function.
+This function inspects that the accumulated MRENCLAVE matches the
+MRENCLAVE inside the caller provided SIGSTRUCT, andlocks down the
+enclave from further build operations.
+After this, the enclave can be invoked.
+.SS Enclave memory mapping
+.PP
+The processor tracks EPC pages in a hardware metadata structure called
+the \f[I]Enclave Page Cache Map (EPCM)\f[R].
+The EPCM contains an entry for each EPC page which describes the owning
+enclave, access rights and page type among the other things.
+EPCM permissions are separate from the normal page tables.
+This prevents the kernel from, for instance, allowing writes to data
+which an enclave wishes to remain read-only.
+EPCM permissions may only impose additional restrictions on top of
+normal x86 page permissions.
+.PP
+For all intents and purposes, the SGX architecture allows the processor
+to invalidate all EPCM entries at will.
+This requires that software be prepared to handle an EPCM fault at any
+time.
+In practice, this can happen on events like power transitions when the
+ephemeral key that encrypts enclave memory is lost.
+.PP
+Kernel records EPCM permissions when \f[B]SGX_IOC_ENCLAVE_ADD_PAGES\f[R]
+is called.
+When the pages are mapped to memory via mmap() or mprotect(), the EPCM
+permissions are compared against the declared permissions.
+If the declared permissions have bits set that are not part of the EPCM
+permissions, the operation fails with \f[B]-EACCES\f[R].
+.SS Enclave invocation
+.PP
+Enclaves encounter exceptions for lots of reasons: everything from
+enclave page faults to NULL pointer de-references, to system calls that
+must be called by a delegate from outside the enclave.
+Also, the enclave memory can be invalidated at any point of time when
+system state changes.
+For instance, when system wakes up from a sleep state, all EPC regions
+get invalidated.
+An SGX run-time must be prepared to all of this, and must be ready to
+even rebuild the enclave when the memory is invalidated.
+.PP
+This type of exception handling has been traditionally been handled in
+SIGSEGV handlers, registered by the library.
+But, being process-wide, shared state, signal handling and shared
+libraries do not mix well.
+.PP
+In order to assist the run-time, kernel provides a vDSO,
+\f[B]vsgx_enter_enclave\f[R], which wraps enclave entry functions
+ENCLU[EENTER] and EENTER[ERESUME], and returns exceptions to the caller
+at the point they happen.
+The exception information is filled in RDI, RSI and RDX.
+The kernel-provided user space portion of the vDSO handler will place
+this information in a user-provided buffer, or optionally trigger a
+user-provided callback at the time of the exception.
+.PP
+The vDSO function calling convention uses the standard RDI RSI, RDX,
+RCX, R8 and R9 registers.
+This makes it possible to declare the vDSO as a C prototype, but other
+than that there is no specific support for SystemV ABI.
+Things like storing XSAVE are the responsibility of the enclave and the
+runtime.
+.SS Page reclaimer
+.PP
+Just like normal RAM, there is a limited amount of enclave memory
+available and over-committing it is a very valuable tool to reduce
+resource use.
+Introduce a simple reclaim mechanism for enclave pages.
+.PP
+In contrast to normal page reclaim, the kernel cannot directly access
+enclave memory.
+To get around this, the SGX architecture provides a set of functions to
+help.
+Among other things, these functions copy enclave memory to and from
+normal memory, encrypting it and protecting its integrity in the
+process.
+.PP
+Kernel provides a page reclaimer by using these functions.
+It picks victim pages in LRU fashion from all the enclaves running in
+the system.
+A new kernel thread (ksgxd) reclaims pages in the background based on
+watermarks, similar to normal kswapd.
+.PP
+All enclave pages can be reclaimed, architecturally.
+But, there are some limits to this, such as the special SECS metadata
+page which must be reclaimed last.
+The page version array (used to mitigate replaying old reclaimed pages)
+is also architecturally reclaimable, but not yet implemented.
+The end result is that the vast majority of enclave pages are currently
+reclaimable.
-- 
2.27.0

Re: [PATCH] sgx.7: New page with overview of Software Guard eXtensions (SGX)

[Michael Kerrisk (man-pages)]

Hi Jarkko,

Thanks for the page. I'll have some more comments later, most likely.
But to begin with, are there any other manual pages that should be
listed in a SEE ALSO section for this manual page, and are there any
pages in man-pages that you think should refer to this page in their
SEE ALSO sections?

Cheers,

Michael

On Wed, 2 Dec 2020 at 08:16, Jarkko Sakkinen <jarkko@kernel.org> wrote:
>
> Signed-off-by: Jarkko Sakkinen <jarkko@kernel.org>
> ---
>  man7/sgx.7 | 198 +++++++++++++++++++++++++++++++++++++++++++++++++++++
>  1 file changed, 198 insertions(+)
>  create mode 100644 man7/sgx.7
>
> diff --git a/man7/sgx.7 b/man7/sgx.7
> new file mode 100644
> index 000000000..429c9b64d
> --- /dev/null
> +++ b/man7/sgx.7
> @@ -0,0 +1,198 @@
> +.\" Copyright (C) 2020 Intel Corporation
> +.\"
> +.\" %%%LICENSE_START(VERBATIM)
> +.\" Permission is granted to make and distribute verbatim copies of this
> +.\" manual provided the copyright notice and this permission notice are
> +.\" preserved on all copies.
> +.\"
> +.\" Permission is granted to copy and distribute modified versions of this
> +.\" manual under the conditions for verbatim copying, provided that the
> +.\" entire resulting derived work is distributed under the terms of a
> +.\" permission notice identical to this one.
> +.\"
> +.\" Since the Linux kernel and libraries are constantly changing, this
> +.\" manual page may be incorrect or out-of-date.  The author(s) assume no
> +.\" responsibility for errors or omissions, or for damages resulting from
> +.\" the use of the information contained herein.  The author(s) may not
> +.\" have taken the same level of care in the production of this manual,
> +.\" which is licensed free of charge, as they might when working
> +.\" professionally.
> +.\"
> +.\" Formatted or processed versions of this manual, if unaccompanied by
> +.\" the source, must acknowledge the copyright and authors of this work.
> +.\" %%%LICENSE_END
> +.\"
> +.TH SGX 7 2020-12-02 "Linux" "Linux Programmer's Manual"
> +.PP
> +sgx - overview of Software Guard eXtensions
> +.SH DESCRIPTION
> +.PP
> +Intel Software Guard eXtensions (SGX) allow user space applications to
> +set aside private memory regions of code and data.
> +These memory regions are called as enclaves.
> +.PP
> +SGX must be enabled by the BIOS.
> +If SGX appears to be unsupported on a system having the hardware
> +support, ensure that SGX is enabled in the BIOS.
> +If a BIOS presents a choice between \[lq]Enabled\[rq] and \[lq]Software
> +Enabled\[rq] modes for SGX, choose \[lq]Enabled\[rq].
> +.PP
> +An enclave can be only entered at a fixed set of entry points.
> +Each of them can hold a single hardware thread at a time.
> +While the enclave is loaded from a regular binary file, only the threads
> +inside the enclave can access its memory.
> +.PP
> +Although carved out of normal DRAM, enclave memory is marked in the
> +system memory map as reserved and is not managed by the Linux memory
> +manager.
> +There may be several regions spread across the system.
> +Each contiguous region is called an Enclave Page Cache (EPC) section.
> +EPC sections are enumerated via CPUID.
> +These regions are encrypted when they leave the LLC.
> +.PP
> +SGX is available only if the kernel was configured and built with the
> +\f[B]CONFIG_X86_SGX\f[R] option.
> +The hardware support for SGX can be observed from
> +\f[I]/proc/cpuinfo\f[R] with the \[lq]flags\[rq] field containing
> +\[lq]sgx\[rq].
> +.SS Enclave management
> +.PP
> +Enclave\[cq]s life-cycle starts by opening \f[I]/dev/sgx_enclave\f[R],
> +and ends once all the references to the opened file have been closed.
> +After opening the enclave, its contents must be populated with the ioctl
> +interface provided by \f[I]<asm/sgx.h>\f[R].
> +ENCLS is a privileged (ring-0) instruction with the functionality for
> +managing enclave memory, and the ioctl inteface provides a wrapper for
> +it.
> +.PP
> +Enclave construction starts by calling \f[B]SGX_IOC_ENCLAVE_CREATE\f[R],
> +which takes in the initial version of SGX Enclave Control Structure
> +(SECS).
> +SGX Enclave Control Structure (SECS) contains the description of the
> +enclave.
> +The ioctl calls ENCLS[ECREATE] function, which will copy SECS to the SGX
> +reserved memory.
> +SECS is never mapped to the process address space, and thus cannot be
> +directly referenced.
> +.PP
> +Among other things, SECS contains the base address and size of the
> +enclave, meaning that the addresss space must be carved out before the
> +creation.
> +There is also a hardware constrain that the size must be a power of two,
> +and the base address must be also naturally aligned relative to the
> +size.
> +.PP
> +Some of the SECS fields must be initialized to zero because their values
> +are determined dynamically after the enclave has been created.
> +Most importantly SECS contains two SHA256 hashes: MRENCLAVE and
> +MRSIGNER.
> +Each enclave invocation, during the enclave construction, hashes its
> +defining parameters MRENCLAVE.
> +Arbitrary data can be also hashed into MRENCLAVE via the ENCLS[EEXTEND]
> +function.
> +MRSIGNER contains the hash of the enclave signer key, which is used to
> +sign the SIGSTRUCT structure passed to the ENCLS[EINIT] function.
> +A running enclave can use both of these fields as material for keys
> +acquired with the ENCLU[EGETKEY]unction.
> +.PP
> +After the enclave has been created, a series of
> +\f[B]SGX_IOC_ENCLAVE_ADD_PAGES\f[R]calls are issued.
> +This ioctl copies the provided data to the enclave memory by invoking
> +the ENCLS[EADD] function, and optionally hashes it with help of the
> +ENCLS[EEXTEND] function.
> +Hashing is optional because it is a slow operation and sometimes not
> +required.
> +For instance, one might only care to measure the code pages.
> +.PP
> +Finally, when all the data has been copied to the enclave memory,
> +\f[B]SGX_IOC_ENCLAVE_INIT\f[R] is called, which invokes the ENCLS[EINIT]
> +function.
> +This function inspects that the accumulated MRENCLAVE matches the
> +MRENCLAVE inside the caller provided SIGSTRUCT, andlocks down the
> +enclave from further build operations.
> +After this, the enclave can be invoked.
> +.SS Enclave memory mapping
> +.PP
> +The processor tracks EPC pages in a hardware metadata structure called
> +the \f[I]Enclave Page Cache Map (EPCM)\f[R].
> +The EPCM contains an entry for each EPC page which describes the owning
> +enclave, access rights and page type among the other things.
> +EPCM permissions are separate from the normal page tables.
> +This prevents the kernel from, for instance, allowing writes to data
> +which an enclave wishes to remain read-only.
> +EPCM permissions may only impose additional restrictions on top of
> +normal x86 page permissions.
> +.PP
> +For all intents and purposes, the SGX architecture allows the processor
> +to invalidate all EPCM entries at will.
> +This requires that software be prepared to handle an EPCM fault at any
> +time.
> +In practice, this can happen on events like power transitions when the
> +ephemeral key that encrypts enclave memory is lost.
> +.PP
> +Kernel records EPCM permissions when \f[B]SGX_IOC_ENCLAVE_ADD_PAGES\f[R]
> +is called.
> +When the pages are mapped to memory via mmap() or mprotect(), the EPCM
> +permissions are compared against the declared permissions.
> +If the declared permissions have bits set that are not part of the EPCM
> +permissions, the operation fails with \f[B]-EACCES\f[R].
> +.SS Enclave invocation
> +.PP
> +Enclaves encounter exceptions for lots of reasons: everything from
> +enclave page faults to NULL pointer de-references, to system calls that
> +must be called by a delegate from outside the enclave.
> +Also, the enclave memory can be invalidated at any point of time when
> +system state changes.
> +For instance, when system wakes up from a sleep state, all EPC regions
> +get invalidated.
> +An SGX run-time must be prepared to all of this, and must be ready to
> +even rebuild the enclave when the memory is invalidated.
> +.PP
> +This type of exception handling has been traditionally been handled in
> +SIGSEGV handlers, registered by the library.
> +But, being process-wide, shared state, signal handling and shared
> +libraries do not mix well.
> +.PP
> +In order to assist the run-time, kernel provides a vDSO,
> +\f[B]vsgx_enter_enclave\f[R], which wraps enclave entry functions
> +ENCLU[EENTER] and EENTER[ERESUME], and returns exceptions to the caller
> +at the point they happen.
> +The exception information is filled in RDI, RSI and RDX.
> +The kernel-provided user space portion of the vDSO handler will place
> +this information in a user-provided buffer, or optionally trigger a
> +user-provided callback at the time of the exception.
> +.PP
> +The vDSO function calling convention uses the standard RDI RSI, RDX,
> +RCX, R8 and R9 registers.
> +This makes it possible to declare the vDSO as a C prototype, but other
> +than that there is no specific support for SystemV ABI.
> +Things like storing XSAVE are the responsibility of the enclave and the
> +runtime.
> +.SS Page reclaimer
> +.PP
> +Just like normal RAM, there is a limited amount of enclave memory
> +available and over-committing it is a very valuable tool to reduce
> +resource use.
> +Introduce a simple reclaim mechanism for enclave pages.
> +.PP
> +In contrast to normal page reclaim, the kernel cannot directly access
> +enclave memory.
> +To get around this, the SGX architecture provides a set of functions to
> +help.
> +Among other things, these functions copy enclave memory to and from
> +normal memory, encrypting it and protecting its integrity in the
> +process.
> +.PP
> +Kernel provides a page reclaimer by using these functions.
> +It picks victim pages in LRU fashion from all the enclaves running in
> +the system.
> +A new kernel thread (ksgxd) reclaims pages in the background based on
> +watermarks, similar to normal kswapd.
> +.PP
> +All enclave pages can be reclaimed, architecturally.
> +But, there are some limits to this, such as the special SECS metadata
> +page which must be reclaimed last.
> +The page version array (used to mitigate replaying old reclaimed pages)
> +is also architecturally reclaimable, but not yet implemented.
> +The end result is that the vast majority of enclave pages are currently
> +reclaimable.
> --
> 2.27.0
>


-- 
Michael Kerrisk
Linux man-pages maintainer; http://www.kernel.org/doc/man-pages/
Linux/UNIX System Programming Training: http://man7.org/training/

Re: [PATCH] sgx.7: New page with overview of Software Guard eXtensions (SGX)

[Alejandro Colomar (mailing lists; readonly)]

Hi Jarkko,

Thanks for the page.

Adding to Michael's comment,
here are a few things to fix (see below).

Michael, there's also a question for you (grep mtk).

Thanks,

Alex

On 12/2/20 8:15 AM, Jarkko Sakkinen wrote:
> Signed-off-by: Jarkko Sakkinen <jarkko@kernel.org>
> ---
>  man7/sgx.7 | 198 +++++++++++++++++++++++++++++++++++++++++++++++++++++
>  1 file changed, 198 insertions(+)
>  create mode 100644 man7/sgx.7
> 
> diff --git a/man7/sgx.7 b/man7/sgx.7
> new file mode 100644
> index 000000000..429c9b64d
> --- /dev/null
> +++ b/man7/sgx.7
> @@ -0,0 +1,198 @@
> +.\" Copyright (C) 2020 Intel Corporation
> +.\"
> +.\" %%%LICENSE_START(VERBATIM)
> +.\" Permission is granted to make and distribute verbatim copies of this
> +.\" manual provided the copyright notice and this permission notice are
> +.\" preserved on all copies.
> +.\"
> +.\" Permission is granted to copy and distribute modified versions of this
> +.\" manual under the conditions for verbatim copying, provided that the
> +.\" entire resulting derived work is distributed under the terms of a
> +.\" permission notice identical to this one.
> +.\"
> +.\" Since the Linux kernel and libraries are constantly changing, this
> +.\" manual page may be incorrect or out-of-date.  The author(s) assume no
> +.\" responsibility for errors or omissions, or for damages resulting from
> +.\" the use of the information contained herein.  The author(s) may not
> +.\" have taken the same level of care in the production of this manual,
> +.\" which is licensed free of charge, as they might when working
> +.\" professionally.
> +.\"
> +.\" Formatted or processed versions of this manual, if unaccompanied by
> +.\" the source, must acknowledge the copyright and authors of this work.
> +.\" %%%LICENSE_END
> +.\"
> +.TH SGX 7 2020-12-02 "Linux" "Linux Programmer's Manual"
> +.PP
> +sgx - overview of Software Guard eXtensions
> +.SH DESCRIPTION
> +.PP
> +Intel Software Guard eXtensions (SGX) allow user space applications to
> +set aside private memory regions of code and data.
> +These memory regions are called as enclaves.

wfix:

These memory regions are called enclaves.

> +.PP
> +SGX must be enabled by the BIOS.
> +If SGX appears to be unsupported on a system having the hardware

1)
s/having the hardware/having hardware/

2)
Please, use semantic newlines.

To understand 'semantic newlines',
please have a look at
man-pages(7)::STYLE GUIDE::Use semantic newlines

Basically, split lines at the most natural separation point,
instead of just when the line gets over the margin.

> +support, ensure that SGX is enabled in the BIOS.
> +If a BIOS presents a choice between \[lq]Enabled\[rq] and \[lq]Software

s/\\[lq]/\\(dq/g
s/\\[rq]/\\(dq/g

Basically, we use \(dq for any double quotes, and we don't care about
left or right.

Michael (mtk):

I was searching to see if it was documented in man-pages(7),
but I didn't find it.
Should we add something under man-pages(7)::STYLE GUIDE::Generating
optimal glyphs?


> +Enabled\[rq] modes for SGX, choose \[lq]Enabled\[rq].
> +.PP
> +An enclave can be only entered at a fixed set of entry points.
> +Each of them can hold a single hardware thread at a time.
> +While the enclave is loaded from a regular binary file, only the threads
> +inside the enclave can access its memory.
> +.PP
> +Although carved out of normal DRAM, enclave memory is marked in the
> +system memory map as reserved and is not managed by the Linux memory
> +manager.
> +There may be several regions spread across the system.
> +Each contiguous region is called an Enclave Page Cache (EPC) section.
> +EPC sections are enumerated via CPUID.
> +These regions are encrypted when they leave the LLC.
> +.PP
> +SGX is available only if the kernel was configured and built with the
> +\f[B]CONFIG_X86_SGX\f[R] option.

Replace by:

[
.B CONFIG_X86_SGX
option.
]

> +The hardware support for SGX can be observed from
> +\f[I]/proc/cpuinfo\f[R] with the \[lq]flags\[rq] field containing
> +\[lq]sgx\[rq].

[
.I /proc/cpuinfo
with the \(dqflags\(dq field containing \(dqsgx\(dq.
]

> +.SS Enclave management
> +.PP
> +Enclave\[cq]s life-cycle starts by opening \f[I]/dev/sgx_enclave\f[R],

For single quotes (or apostrophe), please use '\(aq':

[
Enclave\(aqs life ...
]

See man-pages(7)::STYLE GUIDE::Generating optimal glyphs

> +and ends once all the references to the opened file have been closed.
> +After opening the enclave, its contents must be populated with the ioctl
> +interface provided by \f[I]<asm/sgx.h>\f[R].
> +ENCLS is a privileged (ring-0) instruction with the functionality for
> +managing enclave memory, and the ioctl inteface provides a wrapper for
> +it.
> +.PP
> +Enclave construction starts by calling \f[B]SGX_IOC_ENCLAVE_CREATE\f[R],
> +which takes in the initial version of SGX Enclave Control Structure
> +(SECS).
> +SGX Enclave Control Structure (SECS) contains the description of the
> +enclave.
> +The ioctl calls ENCLS[ECREATE] function, which will copy SECS to the SGX
> +reserved memory.
> +SECS is never mapped to the process address space, and thus cannot be
> +directly referenced.
> +.PP
> +Among other things, SECS contains the base address and size of the
> +enclave, meaning that the addresss space must be carved out before the
> +creation.
> +There is also a hardware constrain that the size must be a power of two,
> +and the base address must be also naturally aligned relative to the
> +size.
> +.PP
> +Some of the SECS fields must be initialized to zero because their values
> +are determined dynamically after the enclave has been created.
> +Most importantly SECS contains two SHA256 hashes: MRENCLAVE and
> +MRSIGNER.
> +Each enclave invocation, during the enclave construction, hashes its
> +defining parameters MRENCLAVE.
> +Arbitrary data can be also hashed into MRENCLAVE via the ENCLS[EEXTEND]
> +function.
> +MRSIGNER contains the hash of the enclave signer key, which is used to
> +sign the SIGSTRUCT structure passed to the ENCLS[EINIT] function.
> +A running enclave can use both of these fields as material for keys
> +acquired with the ENCLU[EGETKEY]unction.
> +.PP
> +After the enclave has been created, a series of
> +\f[B]SGX_IOC_ENCLAVE_ADD_PAGES\f[R]calls are issued.
> +This ioctl copies the provided data to the enclave memory by invoking
> +the ENCLS[EADD] function, and optionally hashes it with help of the
> +ENCLS[EEXTEND] function.
> +Hashing is optional because it is a slow operation and sometimes not
> +required.
> +For instance, one might only care to measure the code pages.
> +.PP
> +Finally, when all the data has been copied to the enclave memory,
> +\f[B]SGX_IOC_ENCLAVE_INIT\f[R] is called, which invokes the ENCLS[EINIT]
> +function.
> +This function inspects that the accumulated MRENCLAVE matches the
> +MRENCLAVE inside the caller provided SIGSTRUCT, andlocks down the
> +enclave from further build operations.
> +After this, the enclave can be invoked.
> +.SS Enclave memory mapping
> +.PP
> +The processor tracks EPC pages in a hardware metadata structure called
> +the \f[I]Enclave Page Cache Map (EPCM)\f[R].
> +The EPCM contains an entry for each EPC page which describes the owning
> +enclave, access rights and page type among the other things.
> +EPCM permissions are separate from the normal page tables.
> +This prevents the kernel from, for instance, allowing writes to data
> +which an enclave wishes to remain read-only.
> +EPCM permissions may only impose additional restrictions on top of
> +normal x86 page permissions.
> +.PP
> +For all intents and purposes, the SGX architecture allows the processor
> +to invalidate all EPCM entries at will.
> +This requires that software be prepared to handle an EPCM fault at any
> +time.
> +In practice, this can happen on events like power transitions when the
> +ephemeral key that encrypts enclave memory is lost.
> +.PP
> +Kernel records EPCM permissions when \f[B]SGX_IOC_ENCLAVE_ADD_PAGES\f[R]
> +is called.
> +When the pages are mapped to memory via mmap() or mprotect(), the EPCM
> +permissions are compared against the declared permissions.
> +If the declared permissions have bits set that are not part of the EPCM
> +permissions, the operation fails with \f[B]-EACCES\f[R].
> +.SS Enclave invocation
> +.PP
> +Enclaves encounter exceptions for lots of reasons: everything from
> +enclave page faults to NULL pointer de-references, to system calls that
> +must be called by a delegate from outside the enclave.
> +Also, the enclave memory can be invalidated at any point of time when
> +system state changes.
> +For instance, when system wakes up from a sleep state, all EPC regions
> +get invalidated.
> +An SGX run-time must be prepared to all of this, and must be ready to
> +even rebuild the enclave when the memory is invalidated.
> +.PP
> +This type of exception handling has been traditionally been handled in
> +SIGSEGV handlers, registered by the library.
> +But, being process-wide, shared state, signal handling and shared
> +libraries do not mix well.
> +.PP
> +In order to assist the run-time, kernel provides a vDSO,
> +\f[B]vsgx_enter_enclave\f[R], which wraps enclave entry functions
> +ENCLU[EENTER] and EENTER[ERESUME], and returns exceptions to the caller
> +at the point they happen.
> +The exception information is filled in RDI, RSI and RDX.
> +The kernel-provided user space portion of the vDSO handler will place
> +this information in a user-provided buffer, or optionally trigger a
> +user-provided callback at the time of the exception.
> +.PP
> +The vDSO function calling convention uses the standard RDI RSI, RDX,
> +RCX, R8 and R9 registers.
> +This makes it possible to declare the vDSO as a C prototype, but other
> +than that there is no specific support for SystemV ABI.
> +Things like storing XSAVE are the responsibility of the enclave and the
> +runtime.
> +.SS Page reclaimer
> +.PP
> +Just like normal RAM, there is a limited amount of enclave memory
> +available and over-committing it is a very valuable tool to reduce
> +resource use.
> +Introduce a simple reclaim mechanism for enclave pages.
> +.PP
> +In contrast to normal page reclaim, the kernel cannot directly access
> +enclave memory.
> +To get around this, the SGX architecture provides a set of functions to
> +help.
> +Among other things, these functions copy enclave memory to and from
> +normal memory, encrypting it and protecting its integrity in the
> +process.
> +.PP
> +Kernel provides a page reclaimer by using these functions.
> +It picks victim pages in LRU fashion from all the enclaves running in
> +the system.
> +A new kernel thread (ksgxd) reclaims pages in the background based on
> +watermarks, similar to normal kswapd.
> +.PP
> +All enclave pages can be reclaimed, architecturally.
> +But, there are some limits to this, such as the special SECS metadata
> +page which must be reclaimed last.
> +The page version array (used to mitigate replaying old reclaimed pages)
> +is also architecturally reclaimable, but not yet implemented.
> +The end result is that the vast majority of enclave pages are currently
> +reclaimable.
> 

Re: [PATCH] sgx.7: New page with overview of Software Guard eXtensions (SGX)

[Michael Kerrisk (man-pages)]

On 12/2/20 12:50 PM, Alejandro Colomar (mailing lists; readonly) wrote:
> Hi Jarkko,
> 
> Thanks for the page.
> 
> Adding to Michael's comment,
> here are a few things to fix (see below).
> 
> Michael, there's also a question for you (grep mtk).
> 
> Thanks,
> 
> Alex
> 
> On 12/2/20 8:15 AM, Jarkko Sakkinen wrote:
>> Signed-off-by: Jarkko Sakkinen <jarkko@kernel.org>
>> ---
>>  man7/sgx.7 | 198 +++++++++++++++++++++++++++++++++++++++++++++++++++++
>>  1 file changed, 198 insertions(+)
>>  create mode 100644 man7/sgx.7
>>
>> diff --git a/man7/sgx.7 b/man7/sgx.7
>> new file mode 100644
>> index 000000000..429c9b64d
>> --- /dev/null
>> +++ b/man7/sgx.7
>> @@ -0,0 +1,198 @@

[...]

>> +support, ensure that SGX is enabled in the BIOS.
>> +If a BIOS presents a choice between \[lq]Enabled\[rq] and \[lq]Software
> 
> s/\\[lq]/\\(dq/g
> s/\\[rq]/\\(dq/g
> 
> Basically, we use \(dq for any double quotes, and we don't care about
> left or right.

(Yes.)

> Michael (mtk):
> 
> I was searching to see if it was documented in man-pages(7),
> but I didn't find it.
> Should we add something under man-pages(7)::STYLE GUIDE::Generating
> optimal glyphs?

I'm not sure. (I worry about overloading the page with too much
detail.) I think I'll just duck this question for now.

[...]

Thanks,

Michael


-- 
Michael Kerrisk
Linux man-pages maintainer; http://www.kernel.org/doc/man-pages/
Linux/UNIX System Programming Training: http://man7.org/training/

Re: [PATCH] sgx.7: New page with overview of Software Guard eXtensions (SGX)

[Michael Kerrisk (man-pages)]

Hi Jarkko,

By the way, is this feature already merged into the kernel? (It appears not.)

On 12/2/20 12:50 PM, Alejandro Colomar (mailing lists; readonly) wrote:
> Hi Jarkko,
> 
> Thanks for the page.
> 
> Adding to Michael's comment,
> here are a few things to fix (see below).
> 
> Michael, there's also a question for you (grep mtk).
> 
> Thanks,
> 
> Alex
> 
> On 12/2/20 8:15 AM, Jarkko Sakkinen wrote:
>> Signed-off-by: Jarkko Sakkinen <jarkko@kernel.org>
>> ---
>>  man7/sgx.7 | 198 +++++++++++++++++++++++++++++++++++++++++++++++++++++
>>  1 file changed, 198 insertions(+)
>>  create mode 100644 man7/sgx.7
>>
>> diff --git a/man7/sgx.7 b/man7/sgx.7
>> new file mode 100644
>> index 000000000..429c9b64d
>> --- /dev/null
>> +++ b/man7/sgx.7
>> @@ -0,0 +1,198 @@
>> +.\" Copyright (C) 2020 Intel Corporation
>> +.\"
>> +.\" %%%LICENSE_START(VERBATIM)
>> +.\" Permission is granted to make and distribute verbatim copies of this
>> +.\" manual provided the copyright notice and this permission notice are
>> +.\" preserved on all copies.
>> +.\"
>> +.\" Permission is granted to copy and distribute modified versions of this
>> +.\" manual under the conditions for verbatim copying, provided that the
>> +.\" entire resulting derived work is distributed under the terms of a
>> +.\" permission notice identical to this one.
>> +.\"
>> +.\" Since the Linux kernel and libraries are constantly changing, this
>> +.\" manual page may be incorrect or out-of-date.  The author(s) assume no
>> +.\" responsibility for errors or omissions, or for damages resulting from
>> +.\" the use of the information contained herein.  The author(s) may not
>> +.\" have taken the same level of care in the production of this manual,
>> +.\" which is licensed free of charge, as they might when working
>> +.\" professionally.
>> +.\"
>> +.\" Formatted or processed versions of this manual, if unaccompanied by
>> +.\" the source, must acknowledge the copyright and authors of this work.
>> +.\" %%%LICENSE_END
>> +.\"
>> +.TH SGX 7 2020-12-02 "Linux" "Linux Programmer's Manual"
>> +.PP
>> +sgx - overview of Software Guard eXtensions
>> +.SH DESCRIPTION
>> +.PP
>> +Intel Software Guard eXtensions (SGX) allow user space applications to
>> +set aside private memory regions of code and data.
>> +These memory regions are called as enclaves.
> 
> wfix:
> 
> These memory regions are called enclaves.
> 
>> +.PP
>> +SGX must be enabled by the BIOS.
>> +If SGX appears to be unsupported on a system having the hardware
> 
> 1)
> s/having the hardware/having hardware/
> 
> 2)
> Please, use semantic newlines.
> 
> To understand 'semantic newlines',
> please have a look at
> man-pages(7)::STYLE GUIDE::Use semantic newlines
> 
> Basically, split lines at the most natural separation point,
> instead of just when the line gets over the margin.
> 
>> +support, ensure that SGX is enabled in the BIOS.
>> +If a BIOS presents a choice between \[lq]Enabled\[rq] and \[lq]Software
> 
> s/\\[lq]/\\(dq/g
> s/\\[rq]/\\(dq/g
> 
> Basically, we use \(dq for any double quotes, and we don't care about
> left or right.
> 
> Michael (mtk):
> 
> I was searching to see if it was documented in man-pages(7),
> but I didn't find it.
> Should we add something under man-pages(7)::STYLE GUIDE::Generating
> optimal glyphs?
> 
> 
>> +Enabled\[rq] modes for SGX, choose \[lq]Enabled\[rq].
>> +.PP
>> +An enclave can be only entered at a fixed set of entry points.
>> +Each of them can hold a single hardware thread at a time.
>> +While the enclave is loaded from a regular binary file, only the threads
>> +inside the enclave can access its memory.
>> +.PP
>> +Although carved out of normal DRAM, enclave memory is marked in the
>> +system memory map as reserved and is not managed by the Linux memory
>> +manager.
>> +There may be several regions spread across the system.
>> +Each contiguous region is called an Enclave Page Cache (EPC) section.
>> +EPC sections are enumerated via CPUID.

Do you mean here the mechanism documented in the cpuid(4) manual page?
If yes, I'd recommend writing

.BR cpuid (4)

here.

>> +These regions are encrypted when they leave the LLC.

Please define LLC (expand the abbreviation).

>> +.PP
>> +SGX is available only if the kernel was configured and built with the
>> +\f[B]CONFIG_X86_SGX\f[R] option.
> 
> Replace by:
> 
> [
> .B CONFIG_X86_SGX
> option.
> ]
> 
>> +The hardware support for SGX can be observed from
>> +\f[I]/proc/cpuinfo\f[R] with the \[lq]flags\[rq] field containing
>> +\[lq]sgx\[rq].
> 
> [
> .I /proc/cpuinfo
> with the \(dqflags\(dq field containing \(dqsgx\(dq.
> ]
> 
>> +.SS Enclave management
>> +.PP
>> +Enclave\[cq]s life-cycle starts by opening \f[I]/dev/sgx_enclave\f[R],

S/Enclave/An enclave's/


> 
> For single quotes (or apostrophe), please use '\(aq':

Actually, '\(aq' is really only needed in code blocks. In normal running text,
writing just an actual single quote (') suffices and is preferred.

> [
> Enclave\(aqs life ...
> ]
> 
> See man-pages(7)::STYLE GUIDE::Generating optimal glyphs
> 
>> +and ends once all the references to the opened file have been closed.
>> +After opening the enclave, its contents must be populated with the ioctl
>> +interface provided by \f[I]<asm/sgx.h>\f[R].
>> +ENCLS is a privileged (ring-0) instruction with the functionality for
>> +managing enclave memory, and the ioctl inteface provides a wrapper for

s/inteface/interface/

>> +it.
>> +.PP
>> +Enclave construction starts by calling \f[B]SGX_IOC_ENCLAVE_CREATE\f[R],
>> +which takes in the initial version of SGX Enclave Control Structure
>> +(SECS).
>> +SGX Enclave Control Structure (SECS) contains the description of the
>> +enclave.
>> +The ioctl calls ENCLS[ECREATE] function, which will copy SECS to the SGX
>> +reserved memory.
>> +SECS is never mapped to the process address space, and thus cannot be
>> +directly referenced.
>> +.PP
>> +Among other things, SECS contains the base address and size of the
>> +enclave, meaning that the addresss space must be carved out before the
>> +creation.
>> +There is also a hardware constrain that the size must be a power of two,

s/constrain/constrain/

>> +and the base address must be also naturally aligned relative to the
>> +size.
>> +.PP
>> +Some of the SECS fields must be initialized to zero because their values
>> +are determined dynamically after the enclave has been created.
>> +Most importantly SECS contains two SHA256 hashes: MRENCLAVE and
>> +MRSIGNER.
>> +Each enclave invocation, during the enclave construction, hashes its
>> +defining parameters MRENCLAVE.
>> +Arbitrary data can be also hashed into MRENCLAVE via the ENCLS[EEXTEND]
>> +function.
>> +MRSIGNER contains the hash of the enclave signer key, which is used to
>> +sign the SIGSTRUCT structure passed to the ENCLS[EINIT] function.
>> +A running enclave can use both of these fields as material for keys
>> +acquired with the ENCLU[EGETKEY]unction.
>> +.PP
>> +After the enclave has been created, a series of
>> +\f[B]SGX_IOC_ENCLAVE_ADD_PAGES\f[R]calls are issued.
>> +This ioctl copies the provided data to the enclave memory by invoking
>> +the ENCLS[EADD] function, and optionally hashes it with help of the
>> +ENCLS[EEXTEND] function.
>> +Hashing is optional because it is a slow operation and sometimes not
>> +required.
>> +For instance, one might only care to measure the code pages.
>> +.PP
>> +Finally, when all the data has been copied to the enclave memory,
>> +\f[B]SGX_IOC_ENCLAVE_INIT\f[R] is called, which invokes the ENCLS[EINIT]
>> +function.
>> +This function inspects that the accumulated MRENCLAVE matches the
>> +MRENCLAVE inside the caller provided SIGSTRUCT, andlocks down the
>> +enclave from further build operations.
>> +After this, the enclave can be invoked.
>> +.SS Enclave memory mapping
>> +.PP
>> +The processor tracks EPC pages in a hardware metadata structure called
>> +the \f[I]Enclave Page Cache Map (EPCM)\f[R].
>> +The EPCM contains an entry for each EPC page which describes the owning
>> +enclave, access rights and page type among the other things.
>> +EPCM permissions are separate from the normal page tables.
>> +This prevents the kernel from, for instance, allowing writes to data
>> +which an enclave wishes to remain read-only.
>> +EPCM permissions may only impose additional restrictions on top of
>> +normal x86 page permissions.
>> +.PP
>> +For all intents and purposes, the SGX architecture allows the processor
>> +to invalidate all EPCM entries at will.
>> +This requires that software be prepared to handle an EPCM fault at any
>> +time.
>> +In practice, this can happen on events like power transitions when the
>> +ephemeral key that encrypts enclave memory is lost.
>> +.PP
>> +Kernel records EPCM permissions when \f[B]SGX_IOC_ENCLAVE_ADD_PAGES\f[R]
>> +is called.
>> +When the pages are mapped to memory via mmap() or mprotect(), the EPCM
>> +permissions are compared against the declared permissions.
>> +If the declared permissions have bits set that are not part of the EPCM
>> +permissions, the operation fails with \f[B]-EACCES\f[R].
>> +.SS Enclave invocation
>> +.PP
>> +Enclaves encounter exceptions for lots of reasons: everything from
>> +enclave page faults to NULL pointer de-references, to system calls that
>> +must be called by a delegate from outside the enclave.
>> +Also, the enclave memory can be invalidated at any point of time when
>> +system state changes.
>> +For instance, when system wakes up from a sleep state, all EPC regions
>> +get invalidated.
>> +An SGX run-time must be prepared to all of this, and must be ready to

Wording problem around "to all of this". What did you mean here?


>> +even rebuild the enclave when the memory is invalidated.
>> +.PP
>> +This type of exception handling has been traditionally been handled in
>> +SIGSEGV handlers, registered by the library.
>> +But, being process-wide, shared state, signal handling and shared

I think "shared" (as in "shared libraries") can be removed here. This is
a general issue for any kind of library, shared or static.

>> +libraries do not mix well.
>> +.PP
>> +In order to assist the run-time, kernel provides a vDSO,

By "a vDSO", do you mean "a vDSO entry point". If yes, it would be 
better to write that.

>> +\f[B]vsgx_enter_enclave\f[R], which wraps enclave entry functions
>> +ENCLU[EENTER] and EENTER[ERESUME], and returns exceptions to the caller
>> +at the point they happen.
>> +The exception information is filled in RDI, RSI and RDX.
>> +The kernel-provided user space portion of the vDSO handler will place
>> +this information in a user-provided buffer, or optionally trigger a
>> +user-provided callback at the time of the exception.
>> +.PP
>> +The vDSO function calling convention uses the standard RDI RSI, RDX,

s/RDI/RDI,/

>> +RCX, R8 and R9 registers.
>> +This makes it possible to declare the vDSO as a C prototype, but other
>> +than that there is no specific support for SystemV ABI.
>> +Things like storing XSAVE are the responsibility of the enclave and the
>> +runtime.
>> +.SS Page reclaimer
>> +.PP
>> +Just like normal RAM, there is a limited amount of enclave memory
>> +available and over-committing it is a very valuable tool to reduce
>> +resource use.
>> +Introduce a simple reclaim mechanism for enclave pages.

The previous sentence seems to be out of place. What did you mean 
here?

>> +.PP
>> +In contrast to normal page reclaim, the kernel cannot directly access
>> +enclave memory.
>> +To get around this, the SGX architecture provides a set of functions to
>> +help.
>> +Among other things, these functions copy enclave memory to and from
>> +normal memory, encrypting it and protecting its integrity in the
>> +process.
>> +.PP
>> +Kernel provides a page reclaimer by using these functions.
>> +It picks victim pages in LRU fashion from all the enclaves running in
>> +the system.
>> +A new kernel thread (ksgxd) reclaims pages in the background based on
>> +watermarks, similar to normal kswapd.
>> +.PP
>> +All enclave pages can be reclaimed, architecturally.
>> +But, there are some limits to this, such as the special SECS metadata
>> +page which must be reclaimed last.
>> +The page version array (used to mitigate replaying old reclaimed pages)
>> +is also architecturally reclaimable, but not yet implemented.
>> +The end result is that the vast majority of enclave pages are currently
>> +reclaimable.

Thanks,

Michael

-- 
Michael Kerrisk
Linux man-pages maintainer; http://www.kernel.org/doc/man-pages/
Linux/UNIX System Programming Training: http://man7.org/training/

Re: [PATCH] sgx.7: New page with overview of Software Guard eXtensions (SGX)

[Alejandro Colomar (man-pages)]

On 12/2/20 3:20 PM, Michael Kerrisk (man-pages) wrote:
>>> +There is also a hardware constrain that the size must be a power of two,
> 
> s/constrain/constrain/

?

Re: [PATCH] sgx.7: New page with overview of Software Guard eXtensions (SGX)

[Michael Kerrisk (man-pages)]

On 12/2/20 3:45 PM, Alejandro Colomar (man-pages) wrote:
> 
> 
> On 12/2/20 3:20 PM, Michael Kerrisk (man-pages) wrote:
>>>> +There is also a hardware constrain that the size must be a power of two,
>>
>> s/constrain/constrain/
> 
> ?

Oops. s/constrian/constraint/



-- 
Michael Kerrisk
Linux man-pages maintainer; http://www.kernel.org/doc/man-pages/
Linux/UNIX System Programming Training: http://man7.org/training/

Re: [PATCH] sgx.7: New page with overview of Software Guard eXtensions (SGX)

[Jarkko Sakkinen]

On Wed, Dec 02, 2020 at 11:37:01AM +0100, Michael Kerrisk (man-pages) wrote:
> Hi Jarkko,
> 
> Thanks for the page. I'll have some more comments later, most likely.
> But to begin with, are there any other manual pages that should be
> listed in a SEE ALSO section for this manual page, and are there any
> pages in man-pages that you think should refer to this page in their
> SEE ALSO sections?

Thanks for quick response.

SGX does not declare any syscalls but it does have an ioctl API and
a vDSO.

I think one thing that the man page is missing is SIGSEGV handling.
When user space directly invokes ENCLU[EENTER], then the exceptions
inside are recognized through SIGSEV. This should be probably added,
and then "SEE ALSO" should point out to sigaction().

> Cheers,
> 
> Michael

/Jarkko

Re: [PATCH] sgx.7: New page with overview of Software Guard eXtensions (SGX)

[Jarkko Sakkinen]

On Wed, Dec 02, 2020 at 12:50:20PM +0100, Alejandro Colomar (mailing lists; readonly) wrote:
> Hi Jarkko,
> 
> Thanks for the page.
> 
> Adding to Michael's comment,
> here are a few things to fix (see below).
> 
> Michael, there's also a question for you (grep mtk).
> 
> Thanks,
> 
> Alex
> 
> On 12/2/20 8:15 AM, Jarkko Sakkinen wrote:
> > Signed-off-by: Jarkko Sakkinen <jarkko@kernel.org>
> > ---
> >  man7/sgx.7 | 198 +++++++++++++++++++++++++++++++++++++++++++++++++++++
> >  1 file changed, 198 insertions(+)
> >  create mode 100644 man7/sgx.7
> > 
> > diff --git a/man7/sgx.7 b/man7/sgx.7
> > new file mode 100644
> > index 000000000..429c9b64d
> > --- /dev/null
> > +++ b/man7/sgx.7
> > @@ -0,0 +1,198 @@
> > +.\" Copyright (C) 2020 Intel Corporation
> > +.\"
> > +.\" %%%LICENSE_START(VERBATIM)
> > +.\" Permission is granted to make and distribute verbatim copies of this
> > +.\" manual provided the copyright notice and this permission notice are
> > +.\" preserved on all copies.
> > +.\"
> > +.\" Permission is granted to copy and distribute modified versions of this
> > +.\" manual under the conditions for verbatim copying, provided that the
> > +.\" entire resulting derived work is distributed under the terms of a
> > +.\" permission notice identical to this one.
> > +.\"
> > +.\" Since the Linux kernel and libraries are constantly changing, this
> > +.\" manual page may be incorrect or out-of-date.  The author(s) assume no
> > +.\" responsibility for errors or omissions, or for damages resulting from
> > +.\" the use of the information contained herein.  The author(s) may not
> > +.\" have taken the same level of care in the production of this manual,
> > +.\" which is licensed free of charge, as they might when working
> > +.\" professionally.
> > +.\"
> > +.\" Formatted or processed versions of this manual, if unaccompanied by
> > +.\" the source, must acknowledge the copyright and authors of this work.
> > +.\" %%%LICENSE_END
> > +.\"
> > +.TH SGX 7 2020-12-02 "Linux" "Linux Programmer's Manual"
> > +.PP
> > +sgx - overview of Software Guard eXtensions
> > +.SH DESCRIPTION
> > +.PP
> > +Intel Software Guard eXtensions (SGX) allow user space applications to
> > +set aside private memory regions of code and data.
> > +These memory regions are called as enclaves.
> 
> wfix:
> 
> These memory regions are called enclaves.
> 
> > +.PP
> > +SGX must be enabled by the BIOS.
> > +If SGX appears to be unsupported on a system having the hardware
> 
> 1)
> s/having the hardware/having hardware/
> 
> 2)
> Please, use semantic newlines.
> 
> To understand 'semantic newlines',
> please have a look at
> man-pages(7)::STYLE GUIDE::Use semantic newlines
> 
> Basically, split lines at the most natural separation point,
> instead of just when the line gets over the margin.

OK, I'll look into that, thanks.

> > +support, ensure that SGX is enabled in the BIOS.
> > +If a BIOS presents a choice between \[lq]Enabled\[rq] and \[lq]Software
> 
> s/\\[lq]/\\(dq/g
> s/\\[rq]/\\(dq/g
> 
> Basically, we use \(dq for any double quotes, and we don't care about
> left or right.

OK.

> Michael (mtk):
> 
> I was searching to see if it was documented in man-pages(7),
> but I didn't find it.
> Should we add something under man-pages(7)::STYLE GUIDE::Generating
> optimal glyphs?
> 
> 
> > +Enabled\[rq] modes for SGX, choose \[lq]Enabled\[rq].
> > +.PP
> > +An enclave can be only entered at a fixed set of entry points.
> > +Each of them can hold a single hardware thread at a time.
> > +While the enclave is loaded from a regular binary file, only the threads
> > +inside the enclave can access its memory.
> > +.PP
> > +Although carved out of normal DRAM, enclave memory is marked in the
> > +system memory map as reserved and is not managed by the Linux memory
> > +manager.
> > +There may be several regions spread across the system.
> > +Each contiguous region is called an Enclave Page Cache (EPC) section.
> > +EPC sections are enumerated via CPUID.
> > +These regions are encrypted when they leave the LLC.
> > +.PP
> > +SGX is available only if the kernel was configured and built with the
> > +\f[B]CONFIG_X86_SGX\f[R] option.
> 
> Replace by:
> 
> [
> .B CONFIG_X86_SGX
> option.
> ]
> 
> > +The hardware support for SGX can be observed from
> > +\f[I]/proc/cpuinfo\f[R] with the \[lq]flags\[rq] field containing
> > +\[lq]sgx\[rq].
> 
> [
> .I /proc/cpuinfo
> with the \(dqflags\(dq field containing \(dqsgx\(dq.
> ]
> 
> > +.SS Enclave management
> > +.PP
> > +Enclave\[cq]s life-cycle starts by opening \f[I]/dev/sgx_enclave\f[R],
> 
> For single quotes (or apostrophe), please use '\(aq':
> 
> [
> Enclave\(aqs life ...
> ]
> 
> See man-pages(7)::STYLE GUIDE::Generating optimal glyphs

Thank you for the detailed feedback.

/Jarkko

Re: [PATCH] sgx.7: New page with overview of Software Guard eXtensions (SGX)

[Jarkko Sakkinen]

On Wed, Dec 02, 2020 at 03:20:09PM +0100, Michael Kerrisk (man-pages) wrote:
> Hi Jarkko,
> 
> By the way, is this feature already merged into the kernel? (It
> appears not.)

No, it's not yet. Right now it's sitting in x86 tip master branch. I
thought that this is a good time to start preparing a man page given
that it's nearing completion, and also because writing man pages is new
territory for me.

I should have remarked this in the commit message. Sorry about that.

/Jarkko

Re: [PATCH] sgx.7: New page with overview of Software Guard eXtensions (SGX)

[Jarkko Sakkinen]

On Wed, Dec 02, 2020 at 07:17:42PM +0200, Jarkko Sakkinen wrote:
> On Wed, Dec 02, 2020 at 11:37:01AM +0100, Michael Kerrisk (man-pages) wrote:
> > Hi Jarkko,
> > 
> > Thanks for the page. I'll have some more comments later, most likely.
> > But to begin with, are there any other manual pages that should be
> > listed in a SEE ALSO section for this manual page, and are there any
> > pages in man-pages that you think should refer to this page in their
> > SEE ALSO sections?
> 
> Thanks for quick response.
> 
> SGX does not declare any syscalls but it does have an ioctl API and
> a vDSO.
> 
> I think one thing that the man page is missing is SIGSEGV handling.
> When user space directly invokes ENCLU[EENTER], then the exceptions
> inside are recognized through SIGSEV. This should be probably added,
> and then "SEE ALSO" should point out to sigaction().

Obviously I need to have references to mmap() and mprotect() too.

Ignoring everything else related to this feature and simplify the
concept, in SGX you build an enclave yet to be mapped memory, then
mmap() it and possibly mprotect() it. When you build an enclave, you
assign permissions to each page, and there's an invariant in the kernel
implementation that mmap/mprotect() permissions are not allowed to
surpass the intended permissions. The implementation is multi-process
by nature, i.e. one process can build an enclave and other process can
just map it (e.g. getting fd through fork or SCM_RIGHTS).

Hmm... that would be a good paragraph for my man page (with editing of
course) before any other details, when describing enclave construction
:-)

/Jarkko

Re: [PATCH] sgx.7: New page with overview of Software Guard eXtensions (SGX)

[Alejandro Colomar (man-pages)]

Hi Jarkko,

Linux 5.10 has been recently released.
Do you have any updates for this patch?

Thanks,

Alex

On 12/2/20 3:47 PM, Michael Kerrisk (man-pages) wrote:
> On 12/2/20 3:45 PM, Alejandro Colomar (man-pages) wrote:
>>
>>
>> On 12/2/20 3:20 PM, Michael Kerrisk (man-pages) wrote:
>>>>> +There is also a hardware constrain that the size must be a power of two,
>>>
>>> s/constrain/constrain/
>>
>> ?
> 
> Oops. s/constrian/constraint/
> 
> 
> 

-- 
Alejandro Colomar
Linux man-pages comaintainer; https://www.kernel.org/doc/man-pages/
http://www.alejandro-colomar.es/

Re: [PATCH] sgx.7: New page with overview of Software Guard eXtensions (SGX)

[Jarkko Sakkinen]

On Fri, Dec 18, 2020 at 11:24:38AM +0100, Alejandro Colomar (man-pages) wrote:
> Hi Jarkko,
> 
> Linux 5.10 has been recently released.
> Do you have any updates for this patch?
> 
> Thanks,
> 
> Alex

Hi, I've been on a vacation this week. I'm planning to update during the
upcoming week.

/Jarkko

[PATCH] sgx.7: New page with overview of Software Guard eXtensions (SGX)

[Jarkko Sakkinen]

Cc: linux-man@vger.kernel.org
Cc: linux-sgx@vger.kernel.org
Cc: Dave Hansen <dave.hansen@linux.intel.com>
Cc: Reinette Chatre <reinette.chatre@intel.com>
Signed-off-by: Jarkko Sakkinen <jarkko@kernel.org>
---

v8:
* Fix errors reported for the previous version.
v7:
* Added more meat about the address space and API.
* Reorganized the text to have focus more on developer to have a big
  picture of kernel provided interfaces.
v6:
* Small fixes based on Dave's and Reinette's feedback.
* Extended the "Permissions" section to cover mmap()
v5:
* Taking away hardware concepts and focusing more on the interface.
v4:
* Did a heavy edit trying to streamline the story a bit and focus on
  stuff important to the user (e.g. lighten up x86 details).
v3:
* Overhaul based on Michael's comments. Most likely needs to be refined
  in various places but this is at least a small step forward for sure.
v2:
* Fixed the semantic newlines convention and various style errors etc.
  that were reported by Alenjandro and Michael.
* SGX was merged to v5.

 man7/sgx.7 | 170 +++++++++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 170 insertions(+)
 create mode 100644 man7/sgx.7

diff --git a/man7/sgx.7 b/man7/sgx.7
new file mode 100644
index 000000000..85afea912
--- /dev/null
+++ b/man7/sgx.7
@@ -0,0 +1,170 @@
+.\" Copyright (C) 2021 Intel Corporation
+.\"
+.\" %%%LICENSE_START(VERBATIM)
+.\" Permission is granted to make and distribute verbatim copies of this
+.\" manual provided the copyright notice and this permission notice are
+.\" preserved on all copies.
+.\"
+.\" Permission is granted to copy and distribute modified versions of this
+.\" manual under the conditions for verbatim copying, provided that the
+.\" entire resulting derived work is distributed under the terms of a
+.\" permission notice identical to this one.
+.\"
+.\" Since the Linux kernel and libraries are constantly changing, this
+.\" manual page may be incorrect or out-of-date.  The author(s) assume no
+.\" responsibility for errors or omissions, or for damages resulting from
+.\" the use of the information contained herein.  The author(s) may not
+.\" have taken the same level of care in the production of this manual,
+.\" which is licensed free of charge, as they might when working
+.\" professionally.
+.\"
+.\" Formatted or processed versions of this manual, if unaccompanied by
+.\" the source, must acknowledge the copyright and authors of this work.
+.\" %%%LICENSE_END
+.\"
+.TH SGX 7 2021\-02\-02 "Linux" "Linux Programmer's Manual"
+.PP
+sgx - overview of Software Guard eXtensions
+.SH SYNOPSIS
+.EX
+.B #include <asm/sgx.h>
+.PP
+.IB enclave " = open(""/dev/sgx_enclave", " O_RDWR);"
+.EE
+.SH DESCRIPTION
+Intel Software Guard eXtensions (SGX) allow applications to host
+enclaves,
+protected executable objects in memory.
+.PP
+Enclaves are blobs of executable code,
+running inside a CPU enforced container,
+which is mapped to the process address space.
+They have a fixed set of entry points,
+defined when the enclave is built with the ioctls,
+provided by
+.I /dev/sgx_enclave.
+.PP
+SGX is available only if the kernel was configured and built with the
+.B CONFIG_X86_SGX
+option.
+You can verify that both the kernel and hardware have SGX enabled by
+checking that "sgx" appears in the
+.I flags
+field in
+.IR /proc/cpuinfo .
+.PP
+SGX must be enabled in BIOS.
+If SGX appears to be unsupported,
+ensure that SGX is enabled in the BIOS.
+If a BIOS presents a choice between
+.I Enabled
+and
+.I Software Enabled
+modes for SGX,
+choose
+.I Enabled.
+.PP
+Enclaves are shared objects, meaning that
+they can be shared with a
+.BR cmsg (3),
+and inherited in a fork.
+.SS Address space
+The address range for an enclave must be reserved with
+.BR mmap (2).
+This must happen before the enclave construction can begin,
+because the enclave page addresses are fixed during its build time.
+.PP
+The CPU requires the size of the enclave to be power of two,
+at least size of a one page,
+and the base address to be naturally aligned with the size.
+An appropriate address range can be found by an anonymous mapping:
+.PP
+.EX
+void *area = mmap(NULL, size * 2, PROT_NONE, MAP_PRIVATE | MAP_ANONYMOUS,
+                  -1, 0);
+
+/* Find the first address aligned to the size within the range. */
+void *base = ((uint64_t)area + size - 1) & ~(size - 1);
+.EE
+.PP
+.SS Ioctls
+An enclave life-cycle is started by opening
+.I /dev/sgx_enclave.
+They are  managed with an
+.BR ioctl (2)
+interface,
+.IR <asm/sgx.h>:
+.TP
+.IB SGX_IOC_ENCLAVE_CREATE
+Create SGX Enclave Control Structure (SECS) for an enclave.
+SECS is a hardware defined structure,
+which contains the properties of an enclave,
+such as its base address and size.
+The ioctl argument has the type
+.IR "struct\ *sgx_enclave_create" .
+.TP
+.IB SGX_IOC_ENCLAVE_ADD_PAGES
+Populate a range of enclave pages with the page data provided by the caller.
+The ioctl argument has the type
+.IR "struct\ *sgx_enclave_add_pages" .
+.TP
+.IB SGX_IOC_ENCLAVE_INIT
+Tell the CPU to prepare the enclave for use.
+After a successful initialization,
+no new pages can be added to the enclave.
+The ioctl argument has the type
+.IR "struct\ *sgx_enclave_init" .
+.PP
+The details of what these operations actually mean in the hardware level can be
+found at the Intel Software Developers Manual.
+.SS vDSO
+A process can access an enclave by entering into its address space through
+a set of entry points,
+which must be defined during the construction process.
+This requires a complex sequence of CPU instructions,
+and kernel assisted exception handling.
+For these reasons,
+it is encapsulated into
+a vDSO interface,
+provided by
+.BR vdso_sgx_enter_enclave_t ,
+which is declared in
+.IR <asm/sgx.h>.
+.SS Permissions
+In order to build an enclave, a process must be able to call
+.IR mmap (2)
+with
+.IR PROT_EXEC
+set,
+because, as with any other type of executable,
+the page table protections must be set appropriately.
+Therefore,
+.I /dev/sgx_enclave
+must reside in a partition,
+which is not mounted with
+.B noexec
+set in the mount options.
+.PP
+During the build process each enclave page is assigned protection bits,
+as part of
+.BR ioctl(SGX_IOC_ENCLAVE_ADD_PAGES).
+These protections are also the maximum protections to which the page can be be mapped.
+If
+.BR mmap (2)
+is called with higher protections than those defined during the build,
+it will return
+.B -EACCES.
+If
+.BR ioctl(SGX_IOC_ENCLAVE_ADD_PAGES)
+is called after
+.BR mmap (2)
+with lower protections,
+the caller receives
+.BR SIGBUS,
+once it accesses the page for the first time.
+.SH VERSIONS
+The SGX feature was added in Linux 5.11.
+.SH SEE ALSO
+.BR ioctl (2),
+.BR mmap (2),
+.BR mprotect (2)
-- 
2.25.1