LKML Archive on lore.kernel.org
 help / color / Atom feed
From: "Xing, Cedric" <cedric.xing@intel.com>
To: Jarkko Sakkinen <jarkko.sakkinen@linux.intel.com>,
	linux-kernel@vger.kernel.org, x86@kernel.org,
	linux-sgx@vger.kernel.org
Cc: akpm@linux-foundation.org, dave.hansen@intel.com,
	sean.j.christopherson@intel.com, nhorman@redhat.com,
	npmccallum@redhat.com, serge.ayoun@intel.com,
	shay.katz-zamir@intel.com, haitao.huang@intel.com,
	andriy.shevchenko@linux.intel.com, tglx@linutronix.de,
	kai.svahn@intel.com, bp@alien8.de, josh@joshtriplett.org,
	luto@kernel.org, kai.huang@intel.com, rientjes@google.com,
	Andy Lutomirski <luto@amacapital.net>,
	Dave Hansen <dave.hansen@linux.intel.com>,
	Haitao Huang <haitao.huang@linux.intel.com>,
	Jethro Beekman <jethro@fortanix.com>,
	"Dr . Greg Wettstein" <greg@enjellic.com>
Subject: Re: [PATCH v21 23/28] x86/vdso: Add __vdso_sgx_enter_enclave() to wrap SGX enclave transitions
Date: Wed, 17 Jul 2019 15:07:06 -0700
Message-ID: <d6a6865f-d7a9-dbe9-66b4-3f5cf62aaf6f@intel.com> (raw)
In-Reply-To: <20190713170804.2340-24-jarkko.sakkinen@linux.intel.com>

On 7/13/2019 10:07 AM, Jarkko Sakkinen wrote:
> From: Sean Christopherson <sean.j.christopherson@intel.com>
> 
> Intel Software Guard Extensions (SGX) introduces a new CPL3-only enclave
> mode that runs as a sort of black box shared object that is hosted by an
> untrusted normal CPL3 process.
> 
> Skipping over a great deal of gory architecture details[1], SGX was
> designed in such a way that the host process can utilize a library to
> build, launch and run an enclave.  This is roughly analogous to how
> e.g. libc implementations are used by most applications so that the
> application can focus on its business logic.
> 
> The big gotcha is that because enclaves can generate *and* handle
> exceptions, any SGX library must be prepared to handle nearly any
> exception at any time (well, any time a thread is executing in an
> enclave).  In Linux, this means the SGX library must register a
> signal handler in order to intercept relevant exceptions and forward
> them to the enclave (or in some cases, take action on behalf of the
> enclave).  Unfortunately, Linux's signal mechanism doesn't mesh well
> with libraries, e.g. signal handlers are process wide, are difficult
> to chain, etc...  This becomes particularly nasty when using multiple
> levels of libraries that register signal handlers, e.g. running an
> enclave via cgo inside of the Go runtime.
> 
> In comes vDSO to save the day.  Now that vDSO can fixup exceptions,
> add a function, __vdso_sgx_enter_enclave(), to wrap enclave transitions
> and intercept any exceptions that occur when running the enclave.
> 
> __vdso_sgx_enter_enclave() does NOT adhere to the x86-64 ABI and instead
> uses a custom calling convention.  The primary motivation is to avoid
> issues that arise due to asynchronous enclave exits.  The x86-64 ABI
> requires that EFLAGS.DF, MXCSR and FCW be preserved by the callee, and
> unfortunately for the vDSO, the aformentioned registers/bits are not
> restored after an asynchronous exit, e.g. EFLAGS.DF is in an unknown
> state while MXCSR and FCW are reset to their init values.  So the vDSO
> cannot simply pass the buck by requiring enclaves to adhere to the
> x86-64 ABI.  That leaves three somewhat reasonable options:
> 
>    1) Save/restore non-volatile GPRs, MXCSR and FCW, and clear EFLAGS.DF
> 
>       + 100% compliant with the x86-64 ABI
>       + Callable from any code
>       + Minimal documentation required
>       - Restoring MXCSR/FCW is likely unnecessary 99% of the time
>       - Slow
> 
>    2) Save/restore non-volatile GPRs and clear EFLAGS.DF
> 
>       + Mostly compliant with the x86-64 ABI
>       + Callable from any code that doesn't use SIMD registers
>       - Need to document deviations from x86-64 ABI, i.e. MXCSR and FCW
> 
>    3) Require the caller to save/restore everything.
> 
>       + Fast
>       + Userspace can pass all GPRs to the enclave (minus EAX, RBX and RCX)
>       - Custom ABI
>       - For all intents and purposes must be called from an assembly wrapper
> 
> __vdso_sgx_enter_enclave() implements option (3).  The custom ABI is
> mostly a documentation issue, and even that is offset by the fact that
> being more similar to hardware's ENCLU[EENTER/ERESUME] ABI reduces the
> amount of documentation needed for the vDSO, e.g. options (2) and (3)
> would need to document which registers are marshalled to/from enclaves.
> Requiring an assembly wrapper imparts minimal pain on userspace as SGX
> libraries and/or applications need a healthy chunk of assembly, e.g. in
> the enclave, regardless of the vDSO's implementation.
> 
> Note, the C-like pseudocode describing the assembly routine is wrapped
> in a non-existent macro instead of in a comment to trick kernel-doc into
> auto-parsing the documentation and function prototype.  This is a double
> win as the pseudocode is intended to aid kernel developers, not userland
> enclave developers.
> 
> [1] Documentation/x86/sgx/1.Architecture.rst
> 
> Suggested-by: Andy Lutomirski <luto@amacapital.net>
> Cc: Andy Lutomirski <luto@amacapital.net>
> Cc: Jarkko Sakkinen <jarkko.sakkinen@linux.intel.com>
> Cc: Dave Hansen <dave.hansen@linux.intel.com>
> Cc: Josh Triplett <josh@joshtriplett.org>
> Cc: Haitao Huang <haitao.huang@linux.intel.com>
> Cc: Jethro Beekman <jethro@fortanix.com>
> Cc: Dr. Greg Wettstein <greg@enjellic.com>
> Signed-off-by: Sean Christopherson <sean.j.christopherson@intel.com>
> Co-developed-by: Cedric Xing <cedric.xing@intel.com>
> Signed-off-by: Cedric Xing <cedric.xing@intel.com>
> ---
>   arch/x86/entry/vdso/Makefile             |   2 +
>   arch/x86/entry/vdso/vdso.lds.S           |   1 +
>   arch/x86/entry/vdso/vsgx_enter_enclave.S | 169 +++++++++++++++++++++++
>   arch/x86/include/uapi/asm/sgx.h          |  18 +++
>   4 files changed, 190 insertions(+)
>   create mode 100644 arch/x86/entry/vdso/vsgx_enter_enclave.S
> 
> diff --git a/arch/x86/entry/vdso/Makefile b/arch/x86/entry/vdso/Makefile
> index 715106395c71..1ae23e7d54a9 100644
> --- a/arch/x86/entry/vdso/Makefile
> +++ b/arch/x86/entry/vdso/Makefile
> @@ -24,6 +24,7 @@ VDSO32-$(CONFIG_IA32_EMULATION)	:= y
>   
>   # files to link into the vdso
>   vobjs-y := vdso-note.o vclock_gettime.o vgetcpu.o
> +vobjs-$(VDSO64-y)		+= vsgx_enter_enclave.o
>   
>   # files to link into kernel
>   obj-y				+= vma.o extable.o
> @@ -92,6 +93,7 @@ CFLAGS_REMOVE_vdso-note.o = -pg
>   CFLAGS_REMOVE_vclock_gettime.o = -pg
>   CFLAGS_REMOVE_vgetcpu.o = -pg
>   CFLAGS_REMOVE_vvar.o = -pg
> +CFLAGS_REMOVE_vsgx_enter_enclave.o = -pg
>   
>   #
>   # X32 processes use x32 vDSO to access 64bit kernel data.
> diff --git a/arch/x86/entry/vdso/vdso.lds.S b/arch/x86/entry/vdso/vdso.lds.S
> index 36b644e16272..4bf48462fca7 100644
> --- a/arch/x86/entry/vdso/vdso.lds.S
> +++ b/arch/x86/entry/vdso/vdso.lds.S
> @@ -27,6 +27,7 @@ VERSION {
>   		__vdso_time;
>   		clock_getres;
>   		__vdso_clock_getres;
> +		__vdso_sgx_enter_enclave;
>   	local: *;
>   	};
>   }
> diff --git a/arch/x86/entry/vdso/vsgx_enter_enclave.S b/arch/x86/entry/vdso/vsgx_enter_enclave.S
> new file mode 100644
> index 000000000000..9331279b8fa6
> --- /dev/null
> +++ b/arch/x86/entry/vdso/vsgx_enter_enclave.S
> @@ -0,0 +1,169 @@
> +/* SPDX-License-Identifier: GPL-2.0 */
> +
> +#include <linux/linkage.h>
> +#include <asm/export.h>
> +#include <asm/errno.h>
> +
> +#include "extable.h"
> +
> +#define EX_LEAF		0*8
> +#define EX_TRAPNR	0*8+4
> +#define EX_ERROR_CODE	0*8+6
> +#define EX_ADDRESS	1*8
> +
> +.code64
> +.section .text, "ax"
> +
> +#ifdef SGX_KERNEL_DOC
> +/**
> + * __vdso_sgx_enter_enclave() - Enter an SGX enclave
> + * @leaf:	ENCLU leaf, must be EENTER or ERESUME
> + * @tcs:	TCS, must be non-NULL
> + * @ex_info:	Optional struct sgx_enclave_exception instance
> + * @callback:	Optional callback function to be called on enclave exit or
> + *		exception
> + *
> + * **Important!**  __vdso_sgx_enter_enclave() is **NOT** compliant with the
> + * x86-64 ABI, i.e. cannot be called from standard C code. As noted above,
> + * input parameters must be passed via ``%eax``, ``8(%rsp)``, ``0x10(%rsp)`` and
> + * ``0x18(%rsp)``, with the return value passed via ``%eax``. All other
> + * registers will be passed through to the enclave as is. All registers except
> + * ``%rbp`` must be treated as volatile from the caller's perspective, including
> + * but not limited to GPRs, EFLAGS.DF, MXCSR, FCW, etc... Conversely, the
> + * enclave being run **must** preserve the untrusted ``%rbp``.
> + *
> + * ``callback`` has the following signature:
> + * int callback(long rdi, long rsi, long rdx,
> + *		struct sgx_enclave_exinfo *exinfo, long r8, long r9,
> + *		void *tcs, long ursp);
> + * ``callback`` **shall** follow x86_64 ABI. All GPRs **except** ``%rax``,
> + * ``%rbx`` and ``rcx`` are passed through to ``callback``. ``%rdi``, ``%rsi``,
> + * ``%rdx``, ``%r8``, ``%r9``, along with the value of ``%rsp`` when the enclave
> + * exited/excepted, can be accessed directly as input parameters, while other
> + * GPRs can be accessed in assembly if needed.  A positive value returned from
> + * ``callback`` will be treated as an ENCLU leaf (e.g. EENTER/ERESUME) to
> + * reenter the enclave (without popping the extra data pushed by the enclave off
> + * the stack), while 0 (zero) or a negative return value will be passed back to
> + * the caller of __vdso_sgx_enter_enclave(). It is also safe to leave
> + * ``callback`` via ``longjmp()`` or by throwing a C++ exception.
> + *
> + * Return:
> + *    0 on success,
> + *    -EINVAL if ENCLU leaf is not allowed,
> + *    -EFAULT if ENCL or the enclave faults or non-positive value is returned
> + *     from the callback.
> + */
> +typedef int (*sgx_callback)(long rdi, long rsi, long rdx,
> +			    struct sgx_enclave_exinfo *exinfo, long r8,
> +			    long r9, void *tcs, long ursp);
> +int __vdso_sgx_enter_enclave(int leaf, void *tcs,
> +			     struct sgx_enclave_exinfo *exinfo,
> +			     sgx_callback callback)

I may not have invoked kernel-doc properly but it seems kernel-doc isn't 
able to pick up the parameters correctly.

  reply index

Thread overview: 51+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2019-07-13 17:07 [PATCH v21 00/28] Intel SGX foundations Jarkko Sakkinen
2019-07-13 17:07 ` [PATCH v21 01/28] x86/cpufeatures: Add Intel-defined SGX feature bit Jarkko Sakkinen
2019-07-13 17:07 ` [PATCH v21 02/28] x86/cpufeatures: Add SGX sub-features (as Linux-defined bits) Jarkko Sakkinen
2019-07-13 17:07 ` [PATCH v21 03/28] x86/msr: Add IA32_FEATURE_CONTROL.SGX_ENABLE definition Jarkko Sakkinen
2019-07-13 17:07 ` [PATCH v21 04/28] x86/cpufeatures: Add Intel-defined SGX_LC feature bit Jarkko Sakkinen
2019-07-13 17:07 ` [PATCH v21 05/28] x86/msr: Add SGX Launch Control MSR definitions Jarkko Sakkinen
2019-07-13 17:07 ` [PATCH v21 06/28] x86/mm: x86/sgx: Add new 'PF_SGX' page fault error code bit Jarkko Sakkinen
2019-07-13 17:07 ` [PATCH v21 07/28] x86/mm: x86/sgx: Signal SIGSEGV for userspace #PFs w/ PF_SGX Jarkko Sakkinen
2019-07-13 17:07 ` [PATCH v21 08/28] x86/cpu/intel: Detect SGX support and update caps appropriately Jarkko Sakkinen
2019-07-24 19:35   ` Sean Christopherson
2019-08-02 20:48     ` Jarkko Sakkinen
2019-08-07 15:17     ` Jarkko Sakkinen
2019-07-13 17:07 ` [PATCH v21 09/28] x86/sgx: Add ENCLS architectural error codes Jarkko Sakkinen
2019-07-13 17:07 ` [PATCH v21 10/28] x86/sgx: Add SGX1 and SGX2 architectural data structures Jarkko Sakkinen
2019-07-13 17:07 ` [PATCH v21 11/28] x86/sgx: Add wrappers for ENCLS leaf functions Jarkko Sakkinen
2019-07-13 17:07 ` [PATCH v21 12/28] x86/sgx: Enumerate and track EPC sections Jarkko Sakkinen
2019-07-13 17:07 ` [PATCH v21 13/28] x86/sgx: Add functions to allocate and free EPC pages Jarkko Sakkinen
2019-07-13 17:07 ` [PATCH v21 14/28] x86/sgx: Add sgx_einit() for initializing enclaves Jarkko Sakkinen
2019-07-13 17:07 ` [PATCH v21 15/28] mm: Introduce vm_ops->may_mprotect() Jarkko Sakkinen
2019-07-13 17:07 ` [PATCH v21 16/28] x86/sgx: Add the Linux SGX Enclave Driver Jarkko Sakkinen
2019-07-29 11:17   ` Ayoun, Serge
2019-08-07 15:15     ` Jarkko Sakkinen
2019-08-07 15:17       ` Jarkko Sakkinen
2019-08-07 16:45         ` Jethro Beekman
2019-08-08 15:40       ` Sean Christopherson
2019-08-09 15:02         ` Jarkko Sakkinen
2019-08-09 15:24           ` Sean Christopherson
2019-08-05 16:16   ` Sean Christopherson
2019-08-05 21:39     ` Jarkko Sakkinen
2019-07-13 17:07 ` [PATCH v21 17/28] x86/sgx: Add provisioning Jarkko Sakkinen
2019-07-13 17:07 ` [PATCH v21 18/28] x86/sgx: Add swapping code to the core and SGX driver Jarkko Sakkinen
2019-08-07  6:33   ` Jethro Beekman
2019-08-07 19:12     ` Jarkko Sakkinen
2019-07-13 17:07 ` [PATCH v21 19/28] x86/sgx: ptrace() support for the " Jarkko Sakkinen
2019-07-13 17:07 ` [PATCH v21 20/28] x86/vdso: Add support for exception fixup in vDSO functions Jarkko Sakkinen
2019-07-13 17:07 ` [PATCH v21 21/28] x86/fault: Add helper function to sanitize error code Jarkko Sakkinen
2019-07-13 17:07 ` [PATCH v21 22/28] x86/traps: Attempt to fixup exceptions in vDSO before signaling Jarkko Sakkinen
2019-07-13 17:07 ` [PATCH v21 23/28] x86/vdso: Add __vdso_sgx_enter_enclave() to wrap SGX enclave transitions Jarkko Sakkinen
2019-07-17 22:07   ` Xing, Cedric [this message]
2019-07-13 17:08 ` [PATCH v21 24/28] selftests/x86: Add a selftest for SGX Jarkko Sakkinen
2019-07-17 22:37   ` Xing, Cedric
2019-08-02 20:46     ` Jarkko Sakkinen
2019-08-16 15:43     ` Jarkko Sakkinen
2019-08-16 15:51       ` Jarkko Sakkinen
2019-08-16 16:56     ` Jarkko Sakkinen
2019-07-13 17:08 ` [PATCH v21 25/28] x86/sgx: Update MAINTAINERS Jarkko Sakkinen
2019-07-13 17:08 ` [PATCH v21 26/28] docs: x86/sgx: Add Architecture documentation Jarkko Sakkinen
2019-07-13 17:08 ` [PATCH v21 27/28] docs: x86/sgx: Document kernel internals Jarkko Sakkinen
2019-07-13 17:08 ` [PATCH v21 28/28] docs: x86/sgx: Document the enclave API Jarkko Sakkinen
2019-07-14 14:36 ` [PATCH v21 00/28] Intel SGX foundations Jarkko Sakkinen
2019-08-07  6:40   ` Jethro Beekman

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=d6a6865f-d7a9-dbe9-66b4-3f5cf62aaf6f@intel.com \
    --to=cedric.xing@intel.com \
    --cc=akpm@linux-foundation.org \
    --cc=andriy.shevchenko@linux.intel.com \
    --cc=bp@alien8.de \
    --cc=dave.hansen@intel.com \
    --cc=dave.hansen@linux.intel.com \
    --cc=greg@enjellic.com \
    --cc=haitao.huang@intel.com \
    --cc=haitao.huang@linux.intel.com \
    --cc=jarkko.sakkinen@linux.intel.com \
    --cc=jethro@fortanix.com \
    --cc=josh@joshtriplett.org \
    --cc=kai.huang@intel.com \
    --cc=kai.svahn@intel.com \
    --cc=linux-kernel@vger.kernel.org \
    --cc=linux-sgx@vger.kernel.org \
    --cc=luto@amacapital.net \
    --cc=luto@kernel.org \
    --cc=nhorman@redhat.com \
    --cc=npmccallum@redhat.com \
    --cc=rientjes@google.com \
    --cc=sean.j.christopherson@intel.com \
    --cc=serge.ayoun@intel.com \
    --cc=shay.katz-zamir@intel.com \
    --cc=tglx@linutronix.de \
    --cc=x86@kernel.org \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link

LKML Archive on lore.kernel.org

Archives are clonable:
	git clone --mirror https://lore.kernel.org/lkml/0 lkml/git/0.git
	git clone --mirror https://lore.kernel.org/lkml/1 lkml/git/1.git
	git clone --mirror https://lore.kernel.org/lkml/2 lkml/git/2.git
	git clone --mirror https://lore.kernel.org/lkml/3 lkml/git/3.git
	git clone --mirror https://lore.kernel.org/lkml/4 lkml/git/4.git
	git clone --mirror https://lore.kernel.org/lkml/5 lkml/git/5.git
	git clone --mirror https://lore.kernel.org/lkml/6 lkml/git/6.git
	git clone --mirror https://lore.kernel.org/lkml/7 lkml/git/7.git
	git clone --mirror https://lore.kernel.org/lkml/8 lkml/git/8.git
	git clone --mirror https://lore.kernel.org/lkml/9 lkml/git/9.git

	# If you have public-inbox 1.1+ installed, you may
	# initialize and index your mirror using the following commands:
	public-inbox-init -V2 lkml lkml/ https://lore.kernel.org/lkml \
		linux-kernel@vger.kernel.org
	public-inbox-index lkml

Example config snippet for mirrors

Newsgroup available over NNTP:
	nntp://nntp.lore.kernel.org/org.kernel.vger.linux-kernel


AGPL code for this site: git clone https://public-inbox.org/public-inbox.git