[PATCH] x86/sgx: Allow enclaves to use Asynchrounous Exit Notification

[PATCH] x86/sgx: Allow enclaves to use Asynchrounous Exit Notification

[Dave Hansen]

Short Version:

Allow enclaves to use the new Asynchronous EXit (AEX)
notification mechanism.  This mechanism lets enclaves run a
handler after an AEX event.  These handlers can run mitigations
for things like SGX-Step[1].

AEX Notify will be made available both on upcoming processors and
on some older processors through microcode updates.

Long Version:

== SGX Attribute Background ==

The SGX architecture includes a list of SGX "attributes".  These
attributes ensure consistency and transparency around specific
enclave features.

As a simple example, the "DEBUG" attribute allows an enclave to
be debugged, but also destroys virtually all of SGX security.
Using attributes, enclaves can know that they are being debugged.
Attributes also affect enclave attestation so an enclave can, for
instance, be denied access to secrets while it is being debugged.

The kernel keeps a list of known attributes and will only
initialize enclaves that use a known set of attributes.  This
kernel policy eliminates the chance that a new SGX attribute
could cause undesired effects.

For example, imagine a new attribute was added called
"PROVISIONKEY2" that provided similar functionality to
"PROVISIIONKEY".  A kernel policy that allowed indiscriminate use
of unknown attributes and thus PROVISIONKEY2 would undermine the
existing kernel policy which limits use of PROVISIONKEY enclaves.

== AEX Notify Background ==

"Intel Architecture Instruction Set Extensions and Future
Features - Version 45" is out[2].  There is a new chapter:

	Asynchronous Enclave Exit Notify and the EDECCSSA User Leaf Function.

Enclaves exit can be either synchronous and consensual (EEXIT for
instance) or asynchronous (on an interrupt or fault).  The
asynchronous ones can evidently be exploited to single step
enclaves[1], on top of which other naughty things can be built.

AEX Notify will be made available both on upcoming processors and
on some older processors through microcode updates.

== The Problem ==

These attacks are currently entirely opaque to the enclave since
the hardware does the save/restore under the covers. The
Asynchronous Enclave Exit Notify (AEX Notify) mechanism provides
enclaves an ability to detect and mitigate potential exposure to
these kinds of attacks.

== The Solution ==

Define the new attribute value for AEX Notification.  Ensure the
attribute is cleared from the list reserved attributes which
allows it to be used in enclaves.

I just built this and ran it to make sure there were no obvious
regressions since I do not have the hardware to test it handy.
Tested-by's would be much appreciated.

1. https://github.com/jovanbulck/sgx-step
2. https://cdrdv2.intel.com/v1/dl/getContent/671368?explicitVersion=true

Signed-off-by: Dave Hansen <dave.hansen@linux.intel.com>
Cc: Jarkko Sakkinen <jarkko@kernel.org>
Cc: Andy Lutomirski <luto@kernel.org>
Cc: Thomas Gleixner <tglx@linutronix.de>
Cc: Ingo Molnar <mingo@redhat.com>
Cc: Borislav Petkov <bp@alien8.de>
Cc: x86@kernel.org
Cc: "H. Peter Anvin" <hpa@zytor.com>
Cc: linux-sgx@vger.kernel.org
Cc: linux-kernel@vger.kernel.org
---
 arch/x86/include/asm/sgx.h | 25 ++++++++++++++++++-------
 1 file changed, 18 insertions(+), 7 deletions(-)

diff --git a/arch/x86/include/asm/sgx.h b/arch/x86/include/asm/sgx.h
index 3f9334ef67cd..f7328d8efd83 100644
--- a/arch/x86/include/asm/sgx.h
+++ b/arch/x86/include/asm/sgx.h
@@ -110,17 +110,28 @@ enum sgx_miscselect {
  * %SGX_ATTR_EINITTOKENKEY:	Allow to use token signing key that is used to
  *				sign cryptographic tokens that can be passed to
  *				EINIT as an authorization to run an enclave.
+ * %SGX_ATTR_ASYNC_EXIT_NOTIFY:	Allow enclaves to be notified after an
+ *				asynchronous exit has occurred.
  */
 enum sgx_attribute {
-	SGX_ATTR_INIT		= BIT(0),
-	SGX_ATTR_DEBUG		= BIT(1),
-	SGX_ATTR_MODE64BIT	= BIT(2),
-	SGX_ATTR_PROVISIONKEY	= BIT(4),
-	SGX_ATTR_EINITTOKENKEY	= BIT(5),
-	SGX_ATTR_KSS		= BIT(7),
+	SGX_ATTR_INIT		   = BIT(0),
+	SGX_ATTR_DEBUG		   = BIT(1),
+	SGX_ATTR_MODE64BIT	   = BIT(2),
+				  /* BIT(3) is reserved */
+	SGX_ATTR_PROVISIONKEY	   = BIT(4),
+	SGX_ATTR_EINITTOKENKEY	   = BIT(5),
+				  /* BIT(6) is for CET */
+	SGX_ATTR_KSS		   = BIT(7),
+				  /* BIT(8) is reserved */
+				  /* BIT(9) is reserved */
+	SGX_ATTR_ASYNC_EXIT_NOTIFY = BIT(10),
 };
 
-#define SGX_ATTR_RESERVED_MASK	(BIT_ULL(3) | BIT_ULL(6) | GENMASK_ULL(63, 8))
+#define SGX_ATTR_RESERVED_MASK	(BIT_ULL(3) | \
+				 BIT_ULL(6) | \
+				 BIT_ULL(8) | \
+				 BIT_ULL(9) | \
+				 GENMASK_ULL(63, 11))
 
 /**
  * struct sgx_secs - SGX Enclave Control Structure (SECS)
-- 
2.34.1

Re: [PATCH] x86/sgx: Allow enclaves to use Asynchrounous Exit Notification

[Jarkko Sakkinen]

On Tue, Jul 05, 2022 at 11:36:48AM -0700, Dave Hansen wrote:
> Short Version:
> 
> Allow enclaves to use the new Asynchronous EXit (AEX)
> notification mechanism.  This mechanism lets enclaves run a
> handler after an AEX event.  These handlers can run mitigations
> for things like SGX-Step[1].
> 
> AEX Notify will be made available both on upcoming processors and
> on some older processors through microcode updates.
> 
> Long Version:
> 
> == SGX Attribute Background ==
> 
> The SGX architecture includes a list of SGX "attributes".  These
> attributes ensure consistency and transparency around specific
> enclave features.
> 
> As a simple example, the "DEBUG" attribute allows an enclave to
> be debugged, but also destroys virtually all of SGX security.
> Using attributes, enclaves can know that they are being debugged.
> Attributes also affect enclave attestation so an enclave can, for
> instance, be denied access to secrets while it is being debugged.
> 
> The kernel keeps a list of known attributes and will only
> initialize enclaves that use a known set of attributes.  This
> kernel policy eliminates the chance that a new SGX attribute
> could cause undesired effects.
> 
> For example, imagine a new attribute was added called
> "PROVISIONKEY2" that provided similar functionality to
> "PROVISIIONKEY".  A kernel policy that allowed indiscriminate use
> of unknown attributes and thus PROVISIONKEY2 would undermine the
> existing kernel policy which limits use of PROVISIONKEY enclaves.
> 
> == AEX Notify Background ==
> 
> "Intel Architecture Instruction Set Extensions and Future
> Features - Version 45" is out[2].  There is a new chapter:
> 
> 	Asynchronous Enclave Exit Notify and the EDECCSSA User Leaf Function.
> 
> Enclaves exit can be either synchronous and consensual (EEXIT for
> instance) or asynchronous (on an interrupt or fault).  The
> asynchronous ones can evidently be exploited to single step
> enclaves[1], on top of which other naughty things can be built.
> 
> AEX Notify will be made available both on upcoming processors and
> on some older processors through microcode updates.
> 
> == The Problem ==
> 
> These attacks are currently entirely opaque to the enclave since
> the hardware does the save/restore under the covers. The
> Asynchronous Enclave Exit Notify (AEX Notify) mechanism provides
> enclaves an ability to detect and mitigate potential exposure to
> these kinds of attacks.
> 
> == The Solution ==
> 
> Define the new attribute value for AEX Notification.  Ensure the
> attribute is cleared from the list reserved attributes which
> allows it to be used in enclaves.
> 
> I just built this and ran it to make sure there were no obvious
> regressions since I do not have the hardware to test it handy.
> Tested-by's would be much appreciated.

Is this available on recent ucode updates e.g. for Icelake
or Geminilake?

> 
> 1. https://github.com/jovanbulck/sgx-step
> 2. https://cdrdv2.intel.com/v1/dl/getContent/671368?explicitVersion=true
> 
> Signed-off-by: Dave Hansen <dave.hansen@linux.intel.com>
> Cc: Jarkko Sakkinen <jarkko@kernel.org>
> Cc: Andy Lutomirski <luto@kernel.org>
> Cc: Thomas Gleixner <tglx@linutronix.de>
> Cc: Ingo Molnar <mingo@redhat.com>
> Cc: Borislav Petkov <bp@alien8.de>
> Cc: x86@kernel.org
> Cc: "H. Peter Anvin" <hpa@zytor.com>
> Cc: linux-sgx@vger.kernel.org
> Cc: linux-kernel@vger.kernel.org
> ---
>  arch/x86/include/asm/sgx.h | 25 ++++++++++++++++++-------
>  1 file changed, 18 insertions(+), 7 deletions(-)
> 
> diff --git a/arch/x86/include/asm/sgx.h b/arch/x86/include/asm/sgx.h
> index 3f9334ef67cd..f7328d8efd83 100644
> --- a/arch/x86/include/asm/sgx.h
> +++ b/arch/x86/include/asm/sgx.h
> @@ -110,17 +110,28 @@ enum sgx_miscselect {
>   * %SGX_ATTR_EINITTOKENKEY:	Allow to use token signing key that is used to
>   *				sign cryptographic tokens that can be passed to
>   *				EINIT as an authorization to run an enclave.
> + * %SGX_ATTR_ASYNC_EXIT_NOTIFY:	Allow enclaves to be notified after an
> + *				asynchronous exit has occurred.
>   */
>  enum sgx_attribute {
> -	SGX_ATTR_INIT		= BIT(0),
> -	SGX_ATTR_DEBUG		= BIT(1),
> -	SGX_ATTR_MODE64BIT	= BIT(2),
> -	SGX_ATTR_PROVISIONKEY	= BIT(4),
> -	SGX_ATTR_EINITTOKENKEY	= BIT(5),
> -	SGX_ATTR_KSS		= BIT(7),
> +	SGX_ATTR_INIT		   = BIT(0),
> +	SGX_ATTR_DEBUG		   = BIT(1),
> +	SGX_ATTR_MODE64BIT	   = BIT(2),
> +				  /* BIT(3) is reserved */
> +	SGX_ATTR_PROVISIONKEY	   = BIT(4),
> +	SGX_ATTR_EINITTOKENKEY	   = BIT(5),
> +				  /* BIT(6) is for CET */
> +	SGX_ATTR_KSS		   = BIT(7),
> +				  /* BIT(8) is reserved */
> +				  /* BIT(9) is reserved */
> +	SGX_ATTR_ASYNC_EXIT_NOTIFY = BIT(10),
>  };
>  
> -#define SGX_ATTR_RESERVED_MASK	(BIT_ULL(3) | BIT_ULL(6) | GENMASK_ULL(63, 8))
> +#define SGX_ATTR_RESERVED_MASK	(BIT_ULL(3) | \
> +				 BIT_ULL(6) | \
> +				 BIT_ULL(8) | \
> +				 BIT_ULL(9) | \
> +				 GENMASK_ULL(63, 11))
>  
>  /**
>   * struct sgx_secs - SGX Enclave Control Structure (SECS)
> -- 
> 2.34.1
> 

BR, Jarkko

Re: [PATCH] x86/sgx: Allow enclaves to use Asynchrounous Exit Notification

[Jarkko Sakkinen]

On Mon, Jul 11, 2022 at 05:34:48AM +0300, Jarkko Sakkinen wrote:
> On Tue, Jul 05, 2022 at 11:36:48AM -0700, Dave Hansen wrote:
> > Short Version:
> > 
> > Allow enclaves to use the new Asynchronous EXit (AEX)
> > notification mechanism.  This mechanism lets enclaves run a
> > handler after an AEX event.  These handlers can run mitigations
> > for things like SGX-Step[1].
> > 
> > AEX Notify will be made available both on upcoming processors and
> > on some older processors through microcode updates.
> > 
> > Long Version:
> > 
> > == SGX Attribute Background ==
> > 
> > The SGX architecture includes a list of SGX "attributes".  These
> > attributes ensure consistency and transparency around specific
> > enclave features.
> > 
> > As a simple example, the "DEBUG" attribute allows an enclave to
> > be debugged, but also destroys virtually all of SGX security.
> > Using attributes, enclaves can know that they are being debugged.
> > Attributes also affect enclave attestation so an enclave can, for
> > instance, be denied access to secrets while it is being debugged.
> > 
> > The kernel keeps a list of known attributes and will only
> > initialize enclaves that use a known set of attributes.  This
> > kernel policy eliminates the chance that a new SGX attribute
> > could cause undesired effects.
> > 
> > For example, imagine a new attribute was added called
> > "PROVISIONKEY2" that provided similar functionality to
> > "PROVISIIONKEY".  A kernel policy that allowed indiscriminate use
> > of unknown attributes and thus PROVISIONKEY2 would undermine the
> > existing kernel policy which limits use of PROVISIONKEY enclaves.
> > 
> > == AEX Notify Background ==
> > 
> > "Intel Architecture Instruction Set Extensions and Future
> > Features - Version 45" is out[2].  There is a new chapter:
> > 
> > 	Asynchronous Enclave Exit Notify and the EDECCSSA User Leaf Function.
> > 
> > Enclaves exit can be either synchronous and consensual (EEXIT for
> > instance) or asynchronous (on an interrupt or fault).  The
> > asynchronous ones can evidently be exploited to single step
> > enclaves[1], on top of which other naughty things can be built.
> > 
> > AEX Notify will be made available both on upcoming processors and
> > on some older processors through microcode updates.
> > 
> > == The Problem ==
> > 
> > These attacks are currently entirely opaque to the enclave since
> > the hardware does the save/restore under the covers. The
> > Asynchronous Enclave Exit Notify (AEX Notify) mechanism provides
> > enclaves an ability to detect and mitigate potential exposure to
> > these kinds of attacks.
> > 
> > == The Solution ==
> > 
> > Define the new attribute value for AEX Notification.  Ensure the
> > attribute is cleared from the list reserved attributes which
> > allows it to be used in enclaves.
> > 
> > I just built this and ran it to make sure there were no obvious
> > regressions since I do not have the hardware to test it handy.
> > Tested-by's would be much appreciated.
> 
> Is this available on recent ucode updates e.g. for Icelake
> or Geminilake?

I mean it would not take me long to upgrade our exception handling flow
to this. Then I can run our full test suite on it. But this will of
course require ucode update for Icelake.

AEX Notify will actually just simplify everything. We kind of simulate
"AEX Notify" already with EENTER to execute in-enclave exception handler
before doing actual ERESUME.

BR, Jarkko

Re: [PATCH] x86/sgx: Allow enclaves to use Asynchrounous Exit Notification

[Haitao Huang]

Hi Dave

I think you need add this change in arch/x86/kernel/cpu/sgx/ioctl.c,  
inside sgx_encl_create function:

-	encl->attributes_mask = SGX_ATTR_DEBUG | SGX_ATTR_MODE64BIT |  
SGX_ATTR_KSS;
+	encl->attributes_mask = SGX_ATTR_DEBUG | SGX_ATTR_MODE64BIT |  
SGX_ATTR_KSS | SGX_ATTR_ASYNC_EXIT_NOTIFY;

Otherwise, EINIT ioctl fails with this new attribute set in enclave secs.


Thanks
Haitao
On Tue, 05 Jul 2022 13:36:48 -0500, Dave Hansen  
<dave.hansen@linux.intel.com> wrote:

> Short Version:
>
> Allow enclaves to use the new Asynchronous EXit (AEX)
> notification mechanism.  This mechanism lets enclaves run a
> handler after an AEX event.  These handlers can run mitigations
> for things like SGX-Step[1].
>
> AEX Notify will be made available both on upcoming processors and
> on some older processors through microcode updates.
>
> Long Version:
>
> == SGX Attribute Background ==
>
> The SGX architecture includes a list of SGX "attributes".  These
> attributes ensure consistency and transparency around specific
> enclave features.
>
> As a simple example, the "DEBUG" attribute allows an enclave to
> be debugged, but also destroys virtually all of SGX security.
> Using attributes, enclaves can know that they are being debugged.
> Attributes also affect enclave attestation so an enclave can, for
> instance, be denied access to secrets while it is being debugged.
>
> The kernel keeps a list of known attributes and will only
> initialize enclaves that use a known set of attributes.  This
> kernel policy eliminates the chance that a new SGX attribute
> could cause undesired effects.
>
> For example, imagine a new attribute was added called
> "PROVISIONKEY2" that provided similar functionality to
> "PROVISIIONKEY".  A kernel policy that allowed indiscriminate use
> of unknown attributes and thus PROVISIONKEY2 would undermine the
> existing kernel policy which limits use of PROVISIONKEY enclaves.
>
> == AEX Notify Background ==
>
> "Intel Architecture Instruction Set Extensions and Future
> Features - Version 45" is out[2].  There is a new chapter:
>
> 	Asynchronous Enclave Exit Notify and the EDECCSSA User Leaf Function.
>
> Enclaves exit can be either synchronous and consensual (EEXIT for
> instance) or asynchronous (on an interrupt or fault).  The
> asynchronous ones can evidently be exploited to single step
> enclaves[1], on top of which other naughty things can be built.
>
> AEX Notify will be made available both on upcoming processors and
> on some older processors through microcode updates.
>
> == The Problem ==
>
> These attacks are currently entirely opaque to the enclave since
> the hardware does the save/restore under the covers. The
> Asynchronous Enclave Exit Notify (AEX Notify) mechanism provides
> enclaves an ability to detect and mitigate potential exposure to
> these kinds of attacks.
>
> == The Solution ==
>
> Define the new attribute value for AEX Notification.  Ensure the
> attribute is cleared from the list reserved attributes which
> allows it to be used in enclaves.
>
> I just built this and ran it to make sure there were no obvious
> regressions since I do not have the hardware to test it handy.
> Tested-by's would be much appreciated.
>
> 1. https://github.com/jovanbulck/sgx-step
> 2. https://cdrdv2.intel.com/v1/dl/getContent/671368?explicitVersion=true
>
> Signed-off-by: Dave Hansen <dave.hansen@linux.intel.com>
> Cc: Jarkko Sakkinen <jarkko@kernel.org>
> Cc: Andy Lutomirski <luto@kernel.org>
> Cc: Thomas Gleixner <tglx@linutronix.de>
> Cc: Ingo Molnar <mingo@redhat.com>
> Cc: Borislav Petkov <bp@alien8.de>
> Cc: x86@kernel.org
> Cc: "H. Peter Anvin" <hpa@zytor.com>
> Cc: linux-sgx@vger.kernel.org
> Cc: linux-kernel@vger.kernel.org
> ---
>  arch/x86/include/asm/sgx.h | 25 ++++++++++++++++++-------
>  1 file changed, 18 insertions(+), 7 deletions(-)
>
> diff --git a/arch/x86/include/asm/sgx.h b/arch/x86/include/asm/sgx.h
> index 3f9334ef67cd..f7328d8efd83 100644
> --- a/arch/x86/include/asm/sgx.h
> +++ b/arch/x86/include/asm/sgx.h
> @@ -110,17 +110,28 @@ enum sgx_miscselect {
>   * %SGX_ATTR_EINITTOKENKEY:	Allow to use token signing key that is used  
> to
>   *				sign cryptographic tokens that can be passed to
>   *				EINIT as an authorization to run an enclave.
> + * %SGX_ATTR_ASYNC_EXIT_NOTIFY:	Allow enclaves to be notified after an
> + *				asynchronous exit has occurred.
>   */
>  enum sgx_attribute {
> -	SGX_ATTR_INIT		= BIT(0),
> -	SGX_ATTR_DEBUG		= BIT(1),
> -	SGX_ATTR_MODE64BIT	= BIT(2),
> -	SGX_ATTR_PROVISIONKEY	= BIT(4),
> -	SGX_ATTR_EINITTOKENKEY	= BIT(5),
> -	SGX_ATTR_KSS		= BIT(7),
> +	SGX_ATTR_INIT		   = BIT(0),
> +	SGX_ATTR_DEBUG		   = BIT(1),
> +	SGX_ATTR_MODE64BIT	   = BIT(2),
> +				  /* BIT(3) is reserved */
> +	SGX_ATTR_PROVISIONKEY	   = BIT(4),
> +	SGX_ATTR_EINITTOKENKEY	   = BIT(5),
> +				  /* BIT(6) is for CET */
> +	SGX_ATTR_KSS		   = BIT(7),
> +				  /* BIT(8) is reserved */
> +				  /* BIT(9) is reserved */
> +	SGX_ATTR_ASYNC_EXIT_NOTIFY = BIT(10),
>  };
> -#define SGX_ATTR_RESERVED_MASK	(BIT_ULL(3) | BIT_ULL(6) |  
> GENMASK_ULL(63, 8))
> +#define SGX_ATTR_RESERVED_MASK	(BIT_ULL(3) | \
> +				 BIT_ULL(6) | \
> +				 BIT_ULL(8) | \
> +				 BIT_ULL(9) | \
> +				 GENMASK_ULL(63, 11))
> /**
>   * struct sgx_secs - SGX Enclave Control Structure (SECS)

Re: [PATCH] x86/sgx: Allow enclaves to use Asynchrounous Exit Notification

[Dave Hansen]

On 7/10/22 19:34, Jarkko Sakkinen wrote:
> Is this available on recent ucode updates e.g. for Icelake
> or Geminilake?

I do not believe it is currently publicly available.

It *really* needs some @intel.com Tested-by's.

Re: [PATCH] x86/sgx: Allow enclaves to use Asynchrounous Exit Notification

[Dave Hansen]

On 7/14/22 06:19, Haitao Huang wrote:
> 
> I think you need add this change in arch/x86/kernel/cpu/sgx/ioctl.c,
> inside sgx_encl_create function:
> 
> -    encl->attributes_mask = SGX_ATTR_DEBUG | SGX_ATTR_MODE64BIT |
> SGX_ATTR_KSS;
> +    encl->attributes_mask = SGX_ATTR_DEBUG | SGX_ATTR_MODE64BIT |
> SGX_ATTR_KSS | SGX_ATTR_ASYNC_EXIT_NOTIFY;
> 
> Otherwise, EINIT ioctl fails with this new attribute set in enclave secs.

Makes sense.  I'll add it there.

I also just realized we need to update arch/x86/kvm/cpuid.c as well.  It
keeps a complete list of guest-available SGX attributes.

Re: [PATCH] x86/sgx: Allow enclaves to use Asynchrounous Exit Notification

[Dave Hansen]

Oh, and, btw...  I don't have any hardware that can actually test this
patch.  It can't be applied until I have an actual tested-by from
someone with the hardware.

Re: [PATCH] x86/sgx: Allow enclaves to use Asynchrounous Exit Notification

[Haitao Huang]

On Tue, 19 Jul 2022 12:54:41 -0500, Dave Hansen <dave.hansen@intel.com>  
wrote:

> Oh, and, btw...  I don't have any hardware that can actually test this
> patch.  It can't be applied until I have an actual tested-by from
> someone with the hardware.

I can ask someone in our team to test and report back results with  
Tested-by when you post updated patch.
Thanks
Haitao

Re: [PATCH] x86/sgx: Allow enclaves to use Asynchrounous Exit Notification

[Jarkko Sakkinen]

On Tue, Jul 19, 2022 at 10:54:41AM -0700, Dave Hansen wrote:
> Oh, and, btw...  I don't have any hardware that can actually test this
> patch.  It can't be applied until I have an actual tested-by from
> someone with the hardware.

... I bet that NUC7's will never get a patch

BR, Jarkko