SGX notes from KS/LPC

SGX notes from KS/LPC

[Andy Lutomirski]

Hi-

Here are my notes on SGX issues from KS/LPC.  It seems that I never
emailed it out to a public list -- oops.  It may contain any number of
typos or outright errors.

+++ cut here +++

=== Background and terminology ===

An enclave is an SGX enclave.  Once launched, unless the enclave is explicitly
debuggable, the enclave's data cannot be seen by the OS.

An "ordinary enclave" is an enclave without the PROVISIONKEY, EINITTOKENKEY,
or any reserved attributes set.  I expect the vast majority of enclaves
developed by ISVs to be ordinary enclaves.

A "special enclave" is any enclave that isn't an ordinary enclave.  A
"launch enclave" is a special enclave that has EINITTOKENKEY set.  Launch
enclaves are unusual in that a launch enclave can *only* be run if it is
signed by the private key associated with SGXLEPUBKEYHASH -- a launch
enclave cannot chain to another launch enclave by creating an EINITTOKEN
authorizing it.  (I'm not sure why this is the case, but that's what
the SDM says.)

"Flexible launch control" is the Intel feature that adds SGXLEPUBKEYHASH.

=== Goals for upstream Linux SGX support ===

I consider the following to be fairly strong requirements:

 * Linux SGX support should be useful to normal people.  This means that
   it should be convenient for developers to create, run, and distribute
   enclaves and to recompile and use other developers' enclaves.  Doing
   so should not require paying a fee or signing a contract in the general
   case.  There must not be any conditions incompatible with the use of
   open source enclaves.

 * System owners, admins, distributions, and designers of embedded systems
   should be able to apply reasonable policies that restrict what enclaves
   may be run on their systems.

 * Software should be portable across systems to a reasonable extent.
   This means that software should run on any machine with the
   appropriate hardware and kernel as long as that machine's policy
   permits the software to work.  In the context of SGX with flexible
   launch control, it should be at least inconvenient and maybe even
   impossible for an ISV to ship software signed by a particular
   launch enclave such that an end user cannot run the software
   without trusting the launch enclave.  See below.

=== Recommendations ===

Linux should not support SGX on non-flexible-launch-control systems except as
outlined below.  As it stands, it is impossible to write a Linux SGX driver
that can be used in any meaningful way by open source developers.  I doubt
that any upstream maintainers are willing to participate in the existing
licensing process.  I am certainly not willing to participate on my own
behalf.

On flexible launch control systems, SGX should be supported, but I think that
the existing proposed API workflow for loading an enclave should change.

I'll call the current proposal "user-provided EINITTOKEN".  It works like
this: A user program bundles a launch enclave and asks the kernel to launch
it.  Then the user program asks the launch enclave to generate an EINITTOKEN
for the normal enclave it wants to use.  If the launch enclave approves the
request, it gives the user program an EINITTOKEN.  Then the user program asks
the kernel to launch the normal enclave and supplies the EINITTOKEN to the
kernel.  The kernel launches the enclave.

I suggest an alternative that I'll call "kernel-provided EINITTOKEN".  In this
model, a user program supplies a normal enclave to the kernel and does *not*
supply EINITTOKEN.  The kernel applies its own policy, if any, to decide whether
to accept the enclave.

Here are some reasons that I think kernel-provided EINITTOKEN will work much
better for Linux:

 * Portability: Suppose an ISV ships a program (e.g. Chrome) that bundles a
   normal enclave.  That ISV will sign the enclave.  If they want the enclave
   to run on a system using, say, Red Hat's launch policy, they'll arrange for
   Red Hat to accept their signing key.  Under the user-provided EINITTOKEN
   model, they will ship a Red Hat-provided launch enclave that accepts their
   key.  They'll use this launch enclave at run tiem to generate their
   EINITTOKEN.  This has major problems:

    * It may be incompatible with fully open-source software.  If the ISV
      doesn't have the source to Red Hat's launch enclave, they are forced to
      bundle a binary blob and to give that blob access to their address
      space.  (In the Red Hat example, I imagine that source would be
      available, but this nonetheless imposes an unpleasant auditing
      requirement on the ISV.)

    * It is non-portable.  If a Debian user wants to run Chrome, they'll need
      to ensure that their system policy accepts Chrome's enclave.  This is
      fine, but even if they do so, Chrome still won't work unless Debian or
      the user also accepts the Red Hat launch enclave that Chrome bundles.
      The user may not want to do this.  I expect that distributions like
      Debian are unwilling to grant blanket trust to other vendors' launch
      enclaves.

    In the kernel-provided EINITTOKEN model, this is a non-issue: the launch
    enclave will be provided by the system and non-Red Hat users will simply
    provide a different launch enclave.

 * Performance: Imagine that a program like Chrome bundles an enclave.  The
   first time any user runs Chrome, an EINITTOKEN will need to be generated.
   Once this happens, in the user-provided EINITTOKEN model, if a different
   user runs Chrome, they'll have to generate their own EINITTOKEN.  In
   the kernel-provided EINITTOKEN model, the kernel can maintain a simple
   cache of EINITTOKENs and the second user's launch of Chrome can be
   faster.

 * Simplicity of SGXLEPUBKEYHASH handling: with user-provided EINITTOKEN,
   the kernel has to check that it considers the launch enclave's signer
   to be acceptable and update SGXLEPUBKEYHASH accordingly.  Then, when
   the normal enclave is launched, the kernel needs to make sure that
   the correct SGXLEPUBKEYHASH is set so that the CPU will accept the
   EINITTOKEN.

 * Security: there is no hardware mechanism that guarantees EINITTOKEN
   freshness.  This means that, if the kernel policy is changed by the
   system owner, old EINITTOKENs are still accepted by the hardware.
   As a result, for kernel policy to work properly, the kernel cannot trust
   a user-provided EINITTOKEN as an indication that an enclave is acceptable.

 * Flexibility: the kernel may want to enforce policies that are not directly
   enforceable by launch enclaves.  For example, different users may be
   granted permission to use different sets of enclaves.  Alternatively, the
   kernel may want to revoke permission to use certain enclaves.  As a
   practical matter, this means that the kernel may need to enforce its own
   policy when the user asks it to do EINIT, which means that the kernel
   needs to do most of the work anyway.

I'm inclined to suggest that normal users not be permitted to run launch
enclaves directly, mainly because, in the kernel-provided EINITTOKEN
model, I don't see the point.  If users can do this, they might do it
by rote even if they don't need to, this breaking portability.

I think that, in the first version at least, only root should be able
to run provisioning enclaves.  This is because I don't fully
understand the provisioning workflow and because there are potential
privacy implications to allowing unprivileged users to obtain a
provisioning key.

I think that the kernel should reject enclaves that have reserved attributes
set.  I don't see how the driver could handle such enclaves correctly without
knowing their semantics.  (For example, an enclave with the EINITTOKEN
attribute set is launched differently from other enclaves.)

Since it's currently quite awkward for the kernel to run CPL3 code, which it
would need to do to easily invoke a launch enclave, I hereby volunteer
to make it possible for the kernel to do this.

=== Virtualization considerations ===

At some point, someone will need to decide what restrictions if any
KVM should impose on its guests' use of SGX.

For example, should KVM limit the set of SGXLEPUBKEYHASH values that
its guests can use?  Should KVM intercept EINIT and filter enclaves
directly?

=== Support for non-flexible-launch-control CPUs (e.g. Skylake) ===

If there's a way to make Skylake work similarly to flexible launch control
systems under a kernel-provided EINITTOKEN model, I think that such a mechanism
could be acceptable upstream.  For example, if Intel provided a signed enclave
that could be distributed in linux-firmware.git that would allow the kernel to
enforce its own launch policy and generate the requisite EINITTOKEN
values in a comparably flexible and unrestrictive manner, I can see
this being acceptable upstream.

I'm not sure that upstream would want to support this on CPUs that
are capable of flexible launch control but have it locked by firmware.
I think we want to very strongly discourage firmware from locking
the SGXLEPUBKEYHASH registers.

=== Launch policies that the kernel should provide ===

The kernel's launch policy should be configurable by some reasonable
means.  One of the options should be "launch any normal enclave".  Another
should probably be "accept launch enclaves provided by root (e.g. in
/lib/firmware) that match one of the following SGXLEPUBKEYHASH
values).

I can imagine use cases for another policy like "launch the following
enclaves, listed by hash".

=== A security flaw that Intel should seriously consider fixing ===

SGX, as presently designed, has what I consider to be a significant design flaw
that is relevant in multi-tenant installations.  Specifically, suppose a
single cloud CPU hosts VMs from two unrelated tenants, A and B.  A uses
SGX to protect its data.  If B is able to obtain a dump of A's storage, B
may obtain an enclave that belongs to A along with data sealed by that
enclave.  While B cannot directly unseal A's data, there is no existing
mechanism that can usefully prevent B from running A's enclave and using
that enclave's API to manipulate A's data.

This could be mitigated in future SGX revisions by adding an additional
personalization parameter to EINIT.  This parameter would be mixed in
to the SGX key derivation process such that enclaves initialized using
different personalization keys would obtain different outputs from EGETKEY.
A hypervisor would trap EINIT, replace the personalization key with
KDF(guest's requested personalization key, tenant-specific personalization
secret).  The result would be that one tenant on a machine would be unable
to use another tenant's enclaves unless the hypervisor were compromised.

Re: SGX notes from KS/LPC

[Andy Lutomirski]

On Wed, Mar 8, 2017 at 10:48 AM, Andy Lutomirski <luto@kernel.org> wrote:
> Hi-
>
> Here are my notes on SGX issues from KS/LPC.  It seems that I never
> emailed it out to a public list -- oops.  It may contain any number of
> typos or outright errors.

Willy Tarreau points out that I didn't define SGX at all.  For those
who haven't buried their heads in the SDM or Intel's whitepapers, SGX
is "Software Guard eXtensions".  It's an instruction set extension
that allows you to execute a special kind of user code (packaged up
into an "enclave") on supported CPUs that is protected (to a certain
extent) from software attack from other user code, the kernel, a
hypervisor, and firmware (ME, SMM, etc) and from certain classes of
hardware attacks.  SGX is available on Skylake CPUs.  Imagine
something like Apple's "secure element" except that it runs
unprivileged on the host CPU.

Alas, it has a "feature" called Launch Control that has the effect
that (a) only code that is approved by Intel can use SGX and (b) as
implemented on Windows, user packages need to not only ship their SGX
code but also the Intel-signed blob that approves their code.  In
order to get Intel's approval, you need to go through a process, and
to learn what the process is, you need to sign an NDA (as far as I
know).  I haven't signed that NDA, so I don't know what the process
is.  I don't think I'm willing to sign the NDA, either.

I've heard various people say that Launch Control functions as a root
of trust.  This is simply incorrect -- it's just licensing.  The trust
model would be fully functional without any Launch Control at all.  It
may or may not be useful to reduce the risk that malware uses SGX.

The upstream kernel has a policy that DRM kernel code can't ship
unless open source userspace for it exists.  Since open source
userspace *can't* exist for SGX as it currently stands, I think the
upstream kernel should not support it.  There's a change coming to SGX
in future CPUs called "Flexible Launch Control" (marketing speak) and
IA32_PUBKEYHASH (in the SDM) that should make it workable for upstream
Linux under certain circumstances.  The email I'm replying to here
describes how that might be done.

The technical details are described (somewhat arcanely as usual) in
the Intel SDM volume 3.  Also as usual, there is no description at all
of when IA32_PUBKEYHASH will show up.

--Andy

Re: SGX notes from KS/LPC

[Willy Tarreau]

On Wed, Mar 08, 2017 at 12:19:22PM -0800, Andy Lutomirski wrote:
> On Wed, Mar 8, 2017 at 10:48 AM, Andy Lutomirski <luto@kernel.org> wrote:
> > Hi-
> >
> > Here are my notes on SGX issues from KS/LPC.  It seems that I never
> > emailed it out to a public list -- oops.  It may contain any number of
> > typos or outright errors.
> 
> Willy Tarreau points out that I didn't define SGX at all.  For those
> who haven't buried their heads in the SDM or Intel's whitepapers, SGX
> is "Software Guard eXtensions".  It's an instruction set extension
> that allows you to execute a special kind of user code (packaged up
> into an "enclave") on supported CPUs that is protected (to a certain
> extent) from software attack from other user code, the kernel, a
> hypervisor, and firmware (ME, SMM, etc) and from certain classes of
> hardware attacks.  SGX is available on Skylake CPUs.  Imagine
> something like Apple's "secure element" except that it runs
> unprivileged on the host CPU.
> 
> Alas, it has a "feature" called Launch Control that has the effect
> that (a) only code that is approved by Intel can use SGX and (b) as
> implemented on Windows, user packages need to not only ship their SGX
> code but also the Intel-signed blob that approves their code.  In
> order to get Intel's approval, you need to go through a process, and
> to learn what the process is, you need to sign an NDA (as far as I
> know).  I haven't signed that NDA, so I don't know what the process
> is.  I don't think I'm willing to sign the NDA, either.
> 
> I've heard various people say that Launch Control functions as a root
> of trust.  This is simply incorrect -- it's just licensing.  The trust
> model would be fully functional without any Launch Control at all.  It
> may or may not be useful to reduce the risk that malware uses SGX.
> 
> The upstream kernel has a policy that DRM kernel code can't ship
> unless open source userspace for it exists.  Since open source
> userspace *can't* exist for SGX as it currently stands, I think the
> upstream kernel should not support it.  There's a change coming to SGX
> in future CPUs called "Flexible Launch Control" (marketing speak) and
> IA32_PUBKEYHASH (in the SDM) that should make it workable for upstream
> Linux under certain circumstances.  The email I'm replying to here
> describes how that might be done.
> 
> The technical details are described (somewhat arcanely as usual) in
> the Intel SDM volume 3.  Also as usual, there is no description at all
> of when IA32_PUBKEYHASH will show up.

Thank you very much for the detailed explanation Andy, that was very
clear and informative!

Cheers,
Willy

Re: SGX notes from KS/LPC

[Andy Lutomirski]

On Wed, Mar 8, 2017 at 12:19 PM, Andy Lutomirski <luto@kernel.org> wrote:
> There's a change coming to SGX
> in future CPUs called "Flexible Launch Control" (marketing speak) and
> IA32_PUBKEYHASH (in the SDM)

And if you try to look this up, you'll notice that I typed it wrong.
It's IA32_SGXLEPUBKEYHASH.  Whoops.

Re: [intel-sgx-kernel-dev] SGX notes from KS/LPC

[Andy Lutomirski]

On Wed, Mar 8, 2017 at 10:03 PM, Kai Huang <kaih.linux@gmail.com> wrote:
> === Virtualization considerations === At some point, someone will need to
> decide what restrictions if any KVM should impose on its guests' use of SGX.
> For example, should KVM limit the set of SGXLEPUBKEYHASH values that its
> guests can use? Should KVM intercept EINIT and filter enclaves directly?

You must be using one of those newfangled gmail apps that totally
screws up quoting.  Sigh.  I need to switch away from gmail myself --
they keep regressing basic text editing functionality :(

>
>
> Hi Andy,
>
> Thank you very much for the informative notes. It is very helpful!
>
> For virtualization considerations, would you elaborate why we should impose
> restrictions on the guests use of SGX? I think in general virtualization
> only *virtualize* the hardware behavior but doesn't impose restrictions from
> software's perspective. For example, we cannot restrict a physical machine
> to only run launch enclave provided by RedHat (am I right?), so if we
> virtualize a machine with SGX supported, we should not restrict that the
> virtual machine to be only able to run launch enclave from RedHat.

Good question.  In general, I agree with you, but there are a couple
of SGX-specific considerations here.

1. Assuming that Launch Control serves a purpose at all, the best
legitimate purpose is to enforce some kind of policy on what enclaves
may run.  On Linux, this policy will primarily be configured by the
kernel.  If the kernel were to be configured so as to only allow some
subset of enclaves but KVM didn't restrict what guests could do, then
KVM could be used to do an end run around the kernel policy.  We don't
know what kinds of kernel policies will be useful, so I doubt it's
clear to anyone as to whether this matters.

2. Since Intel didn't provide a way for a hypervisor to make keys
derived by EGETKEY in guests different from keys derived in the host,
enclaves in a guest are effectively running at full host privilege and
with the host's identity.  This means, for example, that a
provisioning enclave running in the guest will be able to convince a
third part that it represents the *host*.  This may be a bad thing and
I can see some cause for preventing provisioning enclaves from running
in guests unless explicitly opted-in by the host admin.  Similarly,
normal enclaves in the guest get the same sealing key that they would
get in other guests on the same host, which could break certain
security assumptions.  I'm not sure how to address this without
Intel's help, but constraining what guests can do would be a start.