Linux-Sgx Archive on lore.kernel.org
 help / color / Atom feed
* x86/sgx: v23-rc2
@ 2019-10-10 11:37 Jarkko Sakkinen
  2019-10-10 13:37 ` Jarkko Sakkinen
  2019-10-11 16:37 ` Jethro Beekman
  0 siblings, 2 replies; 22+ messages in thread
From: Jarkko Sakkinen @ 2019-10-10 11:37 UTC (permalink / raw)
  To: linux-sgx; +Cc: sean.j.christopherson, serge.ayoun, shay.katz-zamir

tag v23-rc2
Tagger: Jarkko Sakkinen <jarkko.sakkinen@linux.intel.com>
Date:   Thu Oct 10 14:33:07 2019 +0300

x86/sgx: v23-rc1 patch set

* Return -EIO instead of -ECANCELED when ptrace() fails to read a TCS page.
* In the reclaimer, pin page before ENCLS[EBLOCK] because pinning can fail
  (because of OOM) even in legit behaviour and after EBLOCK the reclaiming
  flow can be only reverted by killing the whole enclave.
* Fixed SGX_ATTR_RESERVED_MASK. Bit 7 was marked as reserved while in fact
  it should have been bit 6 (Table 37-3 in the SDM).
* Return -EPERM from SGX_IOC_ENCLAVE_INIT when ENCLS[EINIT] returns an SGX
  error code.
* In v22 __uaccess_begin() was used to pin the source page in
  __sgx_encl_add_page(). Switch to get_user_pages() in order to avoid
  deadlock (mmap_sem might get locked twice in the same thread).
-----BEGIN PGP SIGNATURE-----

iJYEABYIAD4WIQRE6pSOnaBC00OEHEIaerohdGur0gUCXZ8XTyAcamFya2tvLnNh
a2tpbmVuQGxpbnV4LmludGVsLmNvbQAKCRAaerohdGur0phXAP9QPYcpyUTSO9hk
sG/pV7vsIjS4lO6pxBCgWCtg3/ZkvAEApLCu7mFvyZs3rDcbOlQA+nQiVv+rUwzu
tsYmW2YsgQ4=
=VeL3
-----END PGP SIGNATURE-----

/Jarkko

^ permalink raw reply	[flat|nested] 22+ messages in thread

* Re: x86/sgx: v23-rc2
  2019-10-10 11:37 x86/sgx: v23-rc2 Jarkko Sakkinen
@ 2019-10-10 13:37 ` Jarkko Sakkinen
  2019-10-10 17:09   ` Sean Christopherson
  2019-10-11 16:37 ` Jethro Beekman
  1 sibling, 1 reply; 22+ messages in thread
From: Jarkko Sakkinen @ 2019-10-10 13:37 UTC (permalink / raw)
  To: linux-sgx; +Cc: sean.j.christopherson, serge.ayoun, shay.katz-zamir

On Thu, Oct 10, 2019 at 02:37:45PM +0300, Jarkko Sakkinen wrote:
> tag v23-rc2
> Tagger: Jarkko Sakkinen <jarkko.sakkinen@linux.intel.com>
> Date:   Thu Oct 10 14:33:07 2019 +0300
> 
> x86/sgx: v23-rc1 patch set
> 
> * Return -EIO instead of -ECANCELED when ptrace() fails to read a TCS page.
> * In the reclaimer, pin page before ENCLS[EBLOCK] because pinning can fail
>   (because of OOM) even in legit behaviour and after EBLOCK the reclaiming
>   flow can be only reverted by killing the whole enclave.
> * Fixed SGX_ATTR_RESERVED_MASK. Bit 7 was marked as reserved while in fact
>   it should have been bit 6 (Table 37-3 in the SDM).
> * Return -EPERM from SGX_IOC_ENCLAVE_INIT when ENCLS[EINIT] returns an SGX
>   error code.
> * In v22 __uaccess_begin() was used to pin the source page in
>   __sgx_encl_add_page(). Switch to get_user_pages() in order to avoid
>   deadlock (mmap_sem might get locked twice in the same thread).

__uaccess_begin() is also needed to performan access checks the legit
user space address. What we can do is to use get_user_pages() just to
make sure that the page is faulted while we perform ENCLS[EADD].

I updated the master branch with the fix for this. Now the access
pattern is:

	ret = get_user_pages(src, 1, 0, &src_page, NULL);
	if (ret < 1)
		return ret;

	__uaccess_begin();

	pginfo.secs = (unsigned long)sgx_epc_addr(encl->secs.epc_page);
	pginfo.addr = SGX_ENCL_PAGE_ADDR(encl_page);
	pginfo.metadata = (unsigned long)secinfo;
	pginfo.contents = (unsigned long)src;

	ret = __eadd(&pginfo, sgx_epc_addr(epc_page));

	__uaccess_end();
	put_page(src_page);

/Jarkko

^ permalink raw reply	[flat|nested] 22+ messages in thread

* Re: x86/sgx: v23-rc2
  2019-10-10 13:37 ` Jarkko Sakkinen
@ 2019-10-10 17:09   ` Sean Christopherson
  2019-10-10 17:39     ` Sean Christopherson
  0 siblings, 1 reply; 22+ messages in thread
From: Sean Christopherson @ 2019-10-10 17:09 UTC (permalink / raw)
  To: Jarkko Sakkinen; +Cc: linux-sgx, serge.ayoun, shay.katz-zamir

On Thu, Oct 10, 2019 at 04:37:53PM +0300, Jarkko Sakkinen wrote:
> On Thu, Oct 10, 2019 at 02:37:45PM +0300, Jarkko Sakkinen wrote:
> > tag v23-rc2
> > Tagger: Jarkko Sakkinen <jarkko.sakkinen@linux.intel.com>
> > Date:   Thu Oct 10 14:33:07 2019 +0300
> > 
> > x86/sgx: v23-rc1 patch set
> > 
> > * Return -EIO instead of -ECANCELED when ptrace() fails to read a TCS page.
> > * In the reclaimer, pin page before ENCLS[EBLOCK] because pinning can fail
> >   (because of OOM) even in legit behaviour and after EBLOCK the reclaiming
> >   flow can be only reverted by killing the whole enclave.
> > * Fixed SGX_ATTR_RESERVED_MASK. Bit 7 was marked as reserved while in fact
> >   it should have been bit 6 (Table 37-3 in the SDM).
> > * Return -EPERM from SGX_IOC_ENCLAVE_INIT when ENCLS[EINIT] returns an SGX
> >   error code.
> > * In v22 __uaccess_begin() was used to pin the source page in
> >   __sgx_encl_add_page(). Switch to get_user_pages() in order to avoid
> >   deadlock (mmap_sem might get locked twice in the same thread).
> 
> __uaccess_begin() is also needed to performan access checks the legit

__uaccess_begin() doesn't check the address space, it temporarily disables
SMAP/SMEP so that the kernel can access a user mapping.  An explicit
access_ok() call should be added as well.

> user space address. What we can do is to use get_user_pages() just to
> make sure that the page is faulted while we perform ENCLS[EADD].


> I updated the master branch with the fix for this. Now the access
> pattern is:
> 
> 	ret = get_user_pages(src, 1, 0, &src_page, NULL);
> 	if (ret < 1)
> 		return ret;
> 
> 	__uaccess_begin();

This should be immediately before __eadd().  I also think it'd be a good
idea to disable page faults around __eadd() so that an unexpected #PF
manifests as an __eadd() failure and not a kernel hang.

> 	pginfo.secs = (unsigned long)sgx_epc_addr(encl->secs.epc_page);
> 	pginfo.addr = SGX_ENCL_PAGE_ADDR(encl_page);
> 	pginfo.metadata = (unsigned long)secinfo;
> 	pginfo.contents = (unsigned long)src;

> 	ret = __eadd(&pginfo, sgx_epc_addr(epc_page));
> 
> 	__uaccess_end();
> 	put_page(src_page);

Not shown here, but mmap_sem doesn't need to be held through EEXTEND. The
lock issue is that down_read() will block if there is a pending
down_write(), e.g. if userspace is doing mprotect() at the same time as
EADD, then deadlock will occur if EADD faults.  Holding encl->lock without
mmap_sem is perfectly ok.

I'll send a small series with the above changes.

^ permalink raw reply	[flat|nested] 22+ messages in thread

* Re: x86/sgx: v23-rc2
  2019-10-10 17:09   ` Sean Christopherson
@ 2019-10-10 17:39     ` Sean Christopherson
  0 siblings, 0 replies; 22+ messages in thread
From: Sean Christopherson @ 2019-10-10 17:39 UTC (permalink / raw)
  To: Jarkko Sakkinen; +Cc: linux-sgx, serge.ayoun, shay.katz-zamir

On Thu, Oct 10, 2019 at 10:09:21AM -0700, Sean Christopherson wrote:
> On Thu, Oct 10, 2019 at 04:37:53PM +0300, Jarkko Sakkinen wrote:
> > On Thu, Oct 10, 2019 at 02:37:45PM +0300, Jarkko Sakkinen wrote:
> > > tag v23-rc2
> > > Tagger: Jarkko Sakkinen <jarkko.sakkinen@linux.intel.com>
> > > Date:   Thu Oct 10 14:33:07 2019 +0300
> > > 
> > > x86/sgx: v23-rc1 patch set
> > > 
> > > * Return -EIO instead of -ECANCELED when ptrace() fails to read a TCS page.
> > > * In the reclaimer, pin page before ENCLS[EBLOCK] because pinning can fail
> > >   (because of OOM) even in legit behaviour and after EBLOCK the reclaiming
> > >   flow can be only reverted by killing the whole enclave.
> > > * Fixed SGX_ATTR_RESERVED_MASK. Bit 7 was marked as reserved while in fact
> > >   it should have been bit 6 (Table 37-3 in the SDM).
> > > * Return -EPERM from SGX_IOC_ENCLAVE_INIT when ENCLS[EINIT] returns an SGX
> > >   error code.
> > > * In v22 __uaccess_begin() was used to pin the source page in
> > >   __sgx_encl_add_page(). Switch to get_user_pages() in order to avoid
> > >   deadlock (mmap_sem might get locked twice in the same thread).
> > 
> > __uaccess_begin() is also needed to performan access checks the legit
> 
> __uaccess_begin() doesn't check the address space, it temporarily disables
> SMAP/SMEP so that the kernel can access a user mapping.  An explicit
> access_ok() call should be added as well.
> 
> > user space address. What we can do is to use get_user_pages() just to
> > make sure that the page is faulted while we perform ENCLS[EADD].
> 
> 
> > I updated the master branch with the fix for this. Now the access
> > pattern is:
> > 
> > 	ret = get_user_pages(src, 1, 0, &src_page, NULL);
> > 	if (ret < 1)
> > 		return ret;
> > 
> > 	__uaccess_begin();
> 
> This should be immediately before __eadd().  I also think it'd be a good
> idea to disable page faults around __eadd() so that an unexpected #PF
> manifests as an __eadd() failure and not a kernel hang.

Scratch that, using the userspace is address is flat out wrong.  gup()
doesn't guarantee the userspace mapping is valid, i.e. EADD can still get
a #PF and deadlock.  AIUI, the correct/bulletproof approach is to kmap()
after gup() and use the kernel translation.

^ permalink raw reply	[flat|nested] 22+ messages in thread

* Re: x86/sgx: v23-rc2
  2019-10-10 11:37 x86/sgx: v23-rc2 Jarkko Sakkinen
  2019-10-10 13:37 ` Jarkko Sakkinen
@ 2019-10-11 16:37 ` Jethro Beekman
  2019-10-11 18:15   ` Sean Christopherson
  1 sibling, 1 reply; 22+ messages in thread
From: Jethro Beekman @ 2019-10-11 16:37 UTC (permalink / raw)
  To: Jarkko Sakkinen, linux-sgx
  Cc: sean.j.christopherson, serge.ayoun, shay.katz-zamir

[-- Attachment #1: Type: text/plain, Size: 3239 bytes --]

Hi all,

I wrote a preliminary patch integrating the latest kernel patches with the Rust EDP, see https://github.com/fortanix/rust-sgx/pull/181 . I tested against (I think) v23-rc2 with the 3 patch sets from Sean applied. In particular, I started from https://github.com/jsakkine-intel/linux-sgx 8915aef. It would be nice if you'd use git tags so everyone can be on the same page.

I haven't done a whole lot of testing, so I can't speak to the stability of the driver. But I do have some comments regarding the API.



UAPI:

This got a whole lot more complex for userspace compared to the out-of-tree driver.

1. Manually needing to mmap a naturally-aligned memory region by allocating too much memory and then unmapping parts is quite annoying. Why was the auto-aligning removed? I think this will need to be handled the same for every consumer of SGX, so I don't see why this is not handled in the kernel. It never seems wrong to align if NULL is passed as the requested address. Alternatively, is there room in the flags for a MAP_ALIGNED bit?

2. Having to re-open the device for every enclave is also annoying. This means you need a filesystem available throughout the process lifetime. I tried dup, but that doesn't work. Can we make dup work?

3. Needing to mprotect every page with the precise permissions needed after EINIT is really bad. This means I have to remember this data for every page between EADD and EINIT. I don't care about SELinux, I trust the ECPM will do its job for me. Can we make it so that I can protect the whole range at once, or protect the individual pages at EADD time?



VDSO:

It is *difficult* to link to weakly link to a symbol in the VDSO. Anyway, I figured it out.

1. What if I don't want to automatically ERESUME after kernel interrupt?

2. I normally do a sanity check after ENCLU[EENTER] that EAX = EEXIT. The current implementation just clears EAX instead without looking at it.

--
Jethro Beekman | Fortanix

On 2019-10-10 13:37, Jarkko Sakkinen wrote:
> tag v23-rc2
> Tagger: Jarkko Sakkinen <jarkko.sakkinen@linux.intel.com>
> Date:   Thu Oct 10 14:33:07 2019 +0300
> 
> x86/sgx: v23-rc1 patch set
> 
> * Return -EIO instead of -ECANCELED when ptrace() fails to read a TCS page.
> * In the reclaimer, pin page before ENCLS[EBLOCK] because pinning can fail
>   (because of OOM) even in legit behaviour and after EBLOCK the reclaiming
>   flow can be only reverted by killing the whole enclave.
> * Fixed SGX_ATTR_RESERVED_MASK. Bit 7 was marked as reserved while in fact
>   it should have been bit 6 (Table 37-3 in the SDM).
> * Return -EPERM from SGX_IOC_ENCLAVE_INIT when ENCLS[EINIT] returns an SGX
>   error code.
> * In v22 __uaccess_begin() was used to pin the source page in
>   __sgx_encl_add_page(). Switch to get_user_pages() in order to avoid
>   deadlock (mmap_sem might get locked twice in the same thread).
> -----BEGIN PGP SIGNATURE-----
> 
> iJYEABYIAD4WIQRE6pSOnaBC00OEHEIaerohdGur0gUCXZ8XTyAcamFya2tvLnNh
> a2tpbmVuQGxpbnV4LmludGVsLmNvbQAKCRAaerohdGur0phXAP9QPYcpyUTSO9hk
> sG/pV7vsIjS4lO6pxBCgWCtg3/ZkvAEApLCu7mFvyZs3rDcbOlQA+nQiVv+rUwzu
> tsYmW2YsgQ4=
> =VeL3
> -----END PGP SIGNATURE-----
> 
> /Jarkko
> 


[-- Attachment #2: S/MIME Cryptographic Signature --]
[-- Type: application/pkcs7-signature, Size: 4054 bytes --]

^ permalink raw reply	[flat|nested] 22+ messages in thread

* Re: x86/sgx: v23-rc2
  2019-10-11 16:37 ` Jethro Beekman
@ 2019-10-11 18:15   ` Sean Christopherson
  2019-10-14  8:43     ` Jethro Beekman
  0 siblings, 1 reply; 22+ messages in thread
From: Sean Christopherson @ 2019-10-11 18:15 UTC (permalink / raw)
  To: Jethro Beekman; +Cc: Jarkko Sakkinen, linux-sgx, serge.ayoun, shay.katz-zamir

On Fri, Oct 11, 2019 at 04:37:25PM +0000, Jethro Beekman wrote:
> UAPI:
> 
> This got a whole lot more complex for userspace compared to the out-of-tree
> driver.
> 
> 1. Manually needing to mmap a naturally-aligned memory region by allocating
> too much memory and then unmapping parts is quite annoying. Why was the
> auto-aligning removed? I think this will need to be handled the same for
> every consumer of SGX, so I don't see why this is not handled in the kernel.
> It never seems wrong to align if NULL is passed as the requested address.
> Alternatively, is there room in the flags for a MAP_ALIGNED bit?

I'm pretty sure everyone agrees it's annoying.  The short of it is that
the SGX driver is the wrong place to do the alignment.  The driver could
key off addr=0, but we don't want to take on that implicit behavior.

A MAP_ALIGNED flag to have the allocation be naturally aligned is the
ideal solution.  It's definitely something we should pursue, but that can
and probably should be done in parallel to the SGX series.

> 2. Having to re-open the device for every enclave is also annoying. This
> means you need a filesystem available throughout the process lifetime. I
> tried dup, but that doesn't work. Can we make dup work?

The semantics of dup() won't get you what want, as dup() just creates a
new descriptor pointing at the same file.

An alternative solution that was proposed was to have an ioctl() for
creating an enclave.  But that means using an anonymous inode, which runs
afoul of SELinux permissions, e.g. every _process_ that runs enclaves
would require EXECMEM.  Linus was quite clear that SGX wouldn't be merged
if using it required users to degrade existing security.

I'm open to other ideas.  I wasn't aware this was a pain point and file
stuff isn't exactly my area of expertise, so I haven't put much/any
thought into alternatives.

> 3. Needing to mprotect every page with the precise permissions needed after
> EINIT is really bad. This means I have to remember this data for every page
> between EADD and EINIT. I don't care about SELinux, I trust the ECPM will do
> its job for me. Can we make it so that I can protect the whole range at once,
> or protect the individual pages at EADD time?

You can mprotect() or mmap(..., MAP_FIXED) an enclave range once all
pages covered by the specified range have been added to the enclave, i.e.
at EADD.  I double checked this with the selftest.  Holler if you're
seeing different behavior.


> VDSO:
> 
> It is *difficult* to link to weakly link to a symbol in the VDSO. Anyway, I
> figured it out.
> 
> 1. What if I don't want to automatically ERESUME after kernel interrupt?

Do EENTER/ERESUME directly instead of going through the vDSO.

> 2. I normally do a sanity check after ENCLU[EENTER] that EAX = EEXIT. The
> current implementation just clears EAX instead without looking at it.

Hmm, the only reason I can think of for checking EAX would be to support
userspace mucking with EAX in a #DB/#BP signal handler.  At that point, I
would expect the signal handler to modify RIP as well.  Reaching the XOR
via any other non-EEXIT path would require a kernel bug.

Was there a specific scenario or use case you had in mind?  I'm not
against adding a check, I just don't see what value it would provide.

^ permalink raw reply	[flat|nested] 22+ messages in thread

* Re: x86/sgx: v23-rc2
  2019-10-11 18:15   ` Sean Christopherson
@ 2019-10-14  8:43     ` Jethro Beekman
  2019-10-17 17:57       ` Sean Christopherson
  0 siblings, 1 reply; 22+ messages in thread
From: Jethro Beekman @ 2019-10-14  8:43 UTC (permalink / raw)
  To: Sean Christopherson
  Cc: Jarkko Sakkinen, linux-sgx, serge.ayoun, shay.katz-zamir

[-- Attachment #1: Type: text/plain, Size: 4949 bytes --]

On 2019-10-11 20:15, Sean Christopherson wrote:
> On Fri, Oct 11, 2019 at 04:37:25PM +0000, Jethro Beekman wrote:
>> UAPI:
>>
>> This got a whole lot more complex for userspace compared to the out-of-tree
>> driver.
>>
>> 1. Manually needing to mmap a naturally-aligned memory region by allocating
>> too much memory and then unmapping parts is quite annoying. Why was the
>> auto-aligning removed? I think this will need to be handled the same for
>> every consumer of SGX, so I don't see why this is not handled in the kernel.
>> It never seems wrong to align if NULL is passed as the requested address.
>> Alternatively, is there room in the flags for a MAP_ALIGNED bit?
> 
> I'm pretty sure everyone agrees it's annoying.  The short of it is that
> the SGX driver is the wrong place to do the alignment.  The driver could
> key off addr=0, but we don't want to take on that implicit behavior.

Why not?

> A MAP_ALIGNED flag to have the allocation be naturally aligned is the
> ideal solution.  It's definitely something we should pursue, but that can
> and probably should be done in parallel to the SGX series.
> 
>> 2. Having to re-open the device for every enclave is also annoying. This
>> means you need a filesystem available throughout the process lifetime. I
>> tried dup, but that doesn't work. Can we make dup work?
> 
> The semantics of dup() won't get you what want, as dup() just creates a
> new descriptor pointing at the same file.
> 
> An alternative solution that was proposed was to have an ioctl() for
> creating an enclave.  But that means using an anonymous inode, which runs
> afoul of SELinux permissions, e.g. every _process_ that runs enclaves
> would require EXECMEM.  Linus was quite clear that SGX wouldn't be merged
> if using it required users to degrade existing security.

It's ok if it's the same inode, it just needs to be a different struct file.

> I'm open to other ideas.  I wasn't aware this was a pain point and file
> stuff isn't exactly my area of expertise, so I haven't put much/any
> thought into alternatives.

The default permissions for /dev/sgx/enclave are root-only. This means you want to be able to do the same thing as network servers: initialize some resources as root, then drop privileges. This used to mean opening /dev/sgx and keeping the fd around which meant you could launch enclaves at will. With the new API, this is no longer possible, you can only launch one enclave per fd. Is there a different type of operation that doesn't just duplicate the fd but also the struct file? If not, can we add an ioctl for that?

There are other scenarios where it's not just the permissions on /dev/sgx/enclave that are the problem but using the filesystem in general that is. Maybe you've used seccomp to disable file operations, etc.

>> 3. Needing to mprotect every page with the precise permissions needed after
>> EINIT is really bad. This means I have to remember this data for every page
>> between EADD and EINIT. I don't care about SELinux, I trust the ECPM will do
>> its job for me. Can we make it so that I can protect the whole range at once,
>> or protect the individual pages at EADD time?
> 
> You can mprotect() or mmap(..., MAP_FIXED) an enclave range once all
> pages covered by the specified range have been added to the enclave, i.e.
> at EADD.  I double checked this with the selftest.  Holler if you're
> seeing different behavior.

I'd swear I tried this flow and I was getting EACCES. But I implemented it again now, and it works fine. So this is not an issue.

I also saw mmap(..., MAP_FIXED) being used in the selftest. Is there a reason to use this over mprotect?

>> VDSO:
>>
>> It is *difficult* to link to weakly link to a symbol in the VDSO. Anyway, I
>> figured it out.
>>
>> 1. What if I don't want to automatically ERESUME after kernel interrupt?
> 
> Do EENTER/ERESUME directly instead of going through the vDSO.

That kind of defeats the point.

>> 2. I normally do a sanity check after ENCLU[EENTER] that EAX = EEXIT. The
>> current implementation just clears EAX instead without looking at it.
> 
> Hmm, the only reason I can think of for checking EAX would be to support
> userspace mucking with EAX in a #DB/#BP signal handler.  At that point, I
> would expect the signal handler to modify RIP as well.  Reaching the XOR
> via any other non-EEXIT path would require a kernel bug.

Or a CPU bug.

> Was there a specific scenario or use case you had in mind?  I'm not
> against adding a check, I just don't see what value it would provide.

Nothing specific. It just seems like a prudent thing to do when messing with control flow in unexpected ways. Alternatively, just remove the xor, and change the API of the function so that 3 is the normal return value? Then the user can decide themselves if they think the check is worth it.

--
Jethro Beekman | Fortanix


[-- Attachment #2: S/MIME Cryptographic Signature --]
[-- Type: application/pkcs7-signature, Size: 4054 bytes --]

^ permalink raw reply	[flat|nested] 22+ messages in thread

* Re: x86/sgx: v23-rc2
  2019-10-14  8:43     ` Jethro Beekman
@ 2019-10-17 17:57       ` Sean Christopherson
  2020-02-13 14:10         ` Jethro Beekman
  0 siblings, 1 reply; 22+ messages in thread
From: Sean Christopherson @ 2019-10-17 17:57 UTC (permalink / raw)
  To: Jethro Beekman
  Cc: Andy Lutomirski, Jarkko Sakkinen, linux-sgx, serge.ayoun,
	shay.katz-zamir

+Cc Andy

On Mon, Oct 14, 2019 at 08:43:09AM +0000, Jethro Beekman wrote:
> On 2019-10-11 20:15, Sean Christopherson wrote:
> > On Fri, Oct 11, 2019 at 04:37:25PM +0000, Jethro Beekman wrote:
> >> UAPI:
> >>
> >> This got a whole lot more complex for userspace compared to the out-of-tree
> >> driver.
> >>
> >> 1. Manually needing to mmap a naturally-aligned memory region by allocating
> >> too much memory and then unmapping parts is quite annoying. Why was the
> >> auto-aligning removed? I think this will need to be handled the same for
> >> every consumer of SGX, so I don't see why this is not handled in the kernel.
> >> It never seems wrong to align if NULL is passed as the requested address.
> >> Alternatively, is there room in the flags for a MAP_ALIGNED bit?
> > 
> > I'm pretty sure everyone agrees it's annoying.  The short of it is that
> > the SGX driver is the wrong place to do the alignment.  The driver could
> > key off addr=0, but we don't want to take on that implicit behavior.
> 
> Why not?

Because it's a hack.  If a MAP_ALIGNED flag is added then SGX is stuck
with kludgy code that serves no purpose.  And userspace needs to manually
align the result if it provides an actual hint.  Regardless of whether
there are use cases for providing a hint for ELRANGE, having divergent
logic is ugly.

> > A MAP_ALIGNED flag to have the allocation be naturally aligned is the
> > ideal solution.  It's definitely something we should pursue, but that can
> > and probably should be done in parallel to the SGX series.
> > 
> >> 2. Having to re-open the device for every enclave is also annoying. This
> >> means you need a filesystem available throughout the process lifetime. I
> >> tried dup, but that doesn't work. Can we make dup work?
> > 
> > The semantics of dup() won't get you what want, as dup() just creates a
> > new descriptor pointing at the same file.
> > 
> > An alternative solution that was proposed was to have an ioctl() for
> > creating an enclave.  But that means using an anonymous inode, which runs
> > afoul of SELinux permissions, e.g. every _process_ that runs enclaves
> > would require EXECMEM.  Linus was quite clear that SGX wouldn't be merged
> > if using it required users to degrade existing security.
> 
> It's ok if it's the same inode, it just needs to be a different struct file.
> 
> > I'm open to other ideas.  I wasn't aware this was a pain point and file
> > stuff isn't exactly my area of expertise, so I haven't put much/any
> > thought into alternatives.
> 
> The default permissions for /dev/sgx/enclave are root-only. This means you
> want to be able to do the same thing as network servers: initialize some
> resources as root, then drop privileges. This used to mean opening /dev/sgx
> and keeping the fd around which meant you could launch enclaves at will. With
> the new API, this is no longer possible, you can only launch one enclave per
> fd. Is there a different type of operation that doesn't just duplicate the fd
> but also the struct file? If not, can we add an ioctl for that?

My approach to this would be to chown /dev/sgx/enclave so that it's owned
by root but accessible to users belonging to an sgx-specific group, e.g.
via a udev rule.

> There are other scenarios where it's not just the permissions on
> /dev/sgx/enclave that are the problem but using the filesystem in general
> that is. Maybe you've used seccomp to disable file operations, etc.

Andy and Jarkko, thoughts?

> >> 3. Needing to mprotect every page with the precise permissions needed after
> >> EINIT is really bad. This means I have to remember this data for every page
> >> between EADD and EINIT. I don't care about SELinux, I trust the ECPM will do
> >> its job for me. Can we make it so that I can protect the whole range at once,
> >> or protect the individual pages at EADD time?
> > 
> > You can mprotect() or mmap(..., MAP_FIXED) an enclave range once all
> > pages covered by the specified range have been added to the enclave, i.e.
> > at EADD.  I double checked this with the selftest.  Holler if you're
> > seeing different behavior.
> 
> I'd swear I tried this flow and I was getting EACCES. But I implemented it
> again now, and it works fine. So this is not an issue.

I wouldn't be surprised if it was broken at some point.

> I also saw mmap(..., MAP_FIXED) being used in the selftest. Is there a reason
> to use this over mprotect?

To test that SGX handles MAP_FIXED correctly.  At one point the driver was
ignoring it.

> >> VDSO:
> >>
> >> It is *difficult* to link to weakly link to a symbol in the VDSO. Anyway, I
> >> figured it out.
> >>
> >> 1. What if I don't want to automatically ERESUME after kernel interrupt?
> > 
> > Do EENTER/ERESUME directly instead of going through the vDSO.
> 
> That kind of defeats the point.

Of providing the vDSO?  My stance on not allowing userspace fine-grained
control over asynchronous enclave exists hasn't changed since we last
discussed the issue[1].  Preventing userspace from hooking interrupts can
help mitigate certain attacks, and it also aligns with normal userspace,
which doesn't get to hook interrupts either.

[1] https://lkml.kernel.org/r/20181102171251.GE7393@linux.intel.com

> >> 2. I normally do a sanity check after ENCLU[EENTER] that EAX = EEXIT. The
> >> current implementation just clears EAX instead without looking at it.
> > 
> > Hmm, the only reason I can think of for checking EAX would be to support
> > userspace mucking with EAX in a #DB/#BP signal handler.  At that point, I
> > would expect the signal handler to modify RIP as well.  Reaching the XOR
> > via any other non-EEXIT path would require a kernel bug.
> 
> Or a CPU bug.

Eh, it's possible, but odds are very high that a CPU bug in this area
would have other side effects, i.e. would be noticed quite quickly even
without an explicit check.

> > Was there a specific scenario or use case you had in mind?  I'm not
> > against adding a check, I just don't see what value it would provide.
> 
> Nothing specific. It just seems like a prudent thing to do when messing with
> control flow in unexpected ways. Alternatively, just remove the xor, and
> change the API of the function so that 3 is the normal return value? Then the
> user can decide themselves if they think the check is worth it.

Andy, thoughts?  We also never bottomed out what should be returned when
an exception is reported to the caller[2], i.e. if there's a better option
than -EFAULT.

[2] https://lkml.kernel.org/r/90D05734-1583-4306-A9A4-18E4A1390F3B@amacapital.net

^ permalink raw reply	[flat|nested] 22+ messages in thread

* Re: x86/sgx: v23-rc2
  2019-10-17 17:57       ` Sean Christopherson
@ 2020-02-13 14:10         ` Jethro Beekman
  2020-02-15  7:24           ` Jarkko Sakkinen
  0 siblings, 1 reply; 22+ messages in thread
From: Jethro Beekman @ 2020-02-13 14:10 UTC (permalink / raw)
  To: Sean Christopherson
  Cc: Andy Lutomirski, Jarkko Sakkinen, linux-sgx, serge.ayoun,
	shay.katz-zamir

[-- Attachment #1: Type: text/plain, Size: 3765 bytes --]

On 2019-10-17 19:57, Sean Christopherson wrote:
> +Cc Andy
> 
> On Mon, Oct 14, 2019 at 08:43:09AM +0000, Jethro Beekman wrote:
>> On 2019-10-11 20:15, Sean Christopherson wrote:
>>> On Fri, Oct 11, 2019 at 04:37:25PM +0000, Jethro Beekman wrote:
>>>> UAPI:
>>>>
>>>> This got a whole lot more complex for userspace compared to the out-of-tree
>>>> driver.
>>>>
>>>> 1. Manually needing to mmap a naturally-aligned memory region by allocating
>>>> too much memory and then unmapping parts is quite annoying. Why was the
>>>> auto-aligning removed? I think this will need to be handled the same for
>>>> every consumer of SGX, so I don't see why this is not handled in the kernel.
>>>> It never seems wrong to align if NULL is passed as the requested address.
>>>> Alternatively, is there room in the flags for a MAP_ALIGNED bit?
>>>
>>> I'm pretty sure everyone agrees it's annoying.  The short of it is that
>>> the SGX driver is the wrong place to do the alignment.  The driver could
>>> key off addr=0, but we don't want to take on that implicit behavior.
>>
>> Why not?
> 
> Because it's a hack.  If a MAP_ALIGNED flag is added then SGX is stuck
> with kludgy code that serves no purpose.  And userspace needs to manually
> align the result if it provides an actual hint.  Regardless of whether
> there are use cases for providing a hint for ELRANGE, having divergent
> logic is ugly.
> 
>>> A MAP_ALIGNED flag to have the allocation be naturally aligned is the
>>> ideal solution.  It's definitely something we should pursue, but that can
>>> and probably should be done in parallel to the SGX series.
>>>
>>>> 2. Having to re-open the device for every enclave is also annoying. This
>>>> means you need a filesystem available throughout the process lifetime. I
>>>> tried dup, but that doesn't work. Can we make dup work?
>>>
>>> The semantics of dup() won't get you what want, as dup() just creates a
>>> new descriptor pointing at the same file.
>>>
>>> An alternative solution that was proposed was to have an ioctl() for
>>> creating an enclave.  But that means using an anonymous inode, which runs
>>> afoul of SELinux permissions, e.g. every _process_ that runs enclaves
>>> would require EXECMEM.  Linus was quite clear that SGX wouldn't be merged
>>> if using it required users to degrade existing security.
>>
>> It's ok if it's the same inode, it just needs to be a different struct file.
>>
>>> I'm open to other ideas.  I wasn't aware this was a pain point and file
>>> stuff isn't exactly my area of expertise, so I haven't put much/any
>>> thought into alternatives.
>>
>> The default permissions for /dev/sgx/enclave are root-only. This means you
>> want to be able to do the same thing as network servers: initialize some
>> resources as root, then drop privileges. This used to mean opening /dev/sgx
>> and keeping the fd around which meant you could launch enclaves at will. With
>> the new API, this is no longer possible, you can only launch one enclave per
>> fd. Is there a different type of operation that doesn't just duplicate the fd
>> but also the struct file? If not, can we add an ioctl for that?
> 
> My approach to this would be to chown /dev/sgx/enclave so that it's owned
> by root but accessible to users belonging to an sgx-specific group, e.g.
> via a udev rule.
> 
>> There are other scenarios where it's not just the permissions on
>> /dev/sgx/enclave that are the problem but using the filesystem in general
>> that is. Maybe you've used seccomp to disable file operations, etc.
> 
> Andy and Jarkko, thoughts?

Folks, any more thoughts on how to resolve the issue that you need to call open() for every enclave?

--
Jethro Beekman | Fortanix


[-- Attachment #2: S/MIME Cryptographic Signature --]
[-- Type: application/pkcs7-signature, Size: 4054 bytes --]

^ permalink raw reply	[flat|nested] 22+ messages in thread

* Re: x86/sgx: v23-rc2
  2020-02-13 14:10         ` Jethro Beekman
@ 2020-02-15  7:24           ` Jarkko Sakkinen
  2020-02-17  8:52             ` Jethro Beekman
  0 siblings, 1 reply; 22+ messages in thread
From: Jarkko Sakkinen @ 2020-02-15  7:24 UTC (permalink / raw)
  To: Jethro Beekman
  Cc: Sean Christopherson, Andy Lutomirski, linux-sgx, serge.ayoun,
	shay.katz-zamir

On Thu, Feb 13, 2020 at 03:10:24PM +0100, Jethro Beekman wrote:
> >> There are other scenarios where it's not just the permissions on
> >> /dev/sgx/enclave that are the problem but using the filesystem in general
> >> that is. Maybe you've used seccomp to disable file operations, etc.
> > 
> > Andy and Jarkko, thoughts?
> 
> Folks, any more thoughts on how to resolve the issue that you need to
> call open() for every enclave?

Why is it an issue?

/Jarkko

^ permalink raw reply	[flat|nested] 22+ messages in thread

* Re: x86/sgx: v23-rc2
  2020-02-15  7:24           ` Jarkko Sakkinen
@ 2020-02-17  8:52             ` Jethro Beekman
  2020-02-17 18:55               ` Jarkko Sakkinen
  0 siblings, 1 reply; 22+ messages in thread
From: Jethro Beekman @ 2020-02-17  8:52 UTC (permalink / raw)
  To: Jarkko Sakkinen
  Cc: Sean Christopherson, Andy Lutomirski, linux-sgx, serge.ayoun,
	shay.katz-zamir

On 2020-02-15 08:24, Jarkko Sakkinen wrote:
> On Thu, Feb 13, 2020 at 03:10:24PM +0100, Jethro Beekman wrote:
>>>> There are other scenarios where it's not just the permissions on
>>>> /dev/sgx/enclave that are the problem but using the filesystem in general
>>>> that is. Maybe you've used seccomp to disable file operations, etc.
>>>
>>> Andy and Jarkko, thoughts?
>>
>> Folks, any more thoughts on how to resolve the issue that you need to
>> call open() for every enclave?
> 
> Why is it an issue?

Already discussed in https://www.spinics.net/lists/linux-sgx/msg02075.html

--
Jethro Beekman | Fortanix

^ permalink raw reply	[flat|nested] 22+ messages in thread

* Re: x86/sgx: v23-rc2
  2020-02-17  8:52             ` Jethro Beekman
@ 2020-02-17 18:55               ` Jarkko Sakkinen
  2020-02-17 18:56                 ` Jarkko Sakkinen
  2020-02-18 10:42                 ` Dr. Greg Wettstein
  0 siblings, 2 replies; 22+ messages in thread
From: Jarkko Sakkinen @ 2020-02-17 18:55 UTC (permalink / raw)
  To: Jethro Beekman
  Cc: Sean Christopherson, Andy Lutomirski, linux-sgx, serge.ayoun,
	shay.katz-zamir

On Mon, Feb 17, 2020 at 09:52:25AM +0100, Jethro Beekman wrote:
> On 2020-02-15 08:24, Jarkko Sakkinen wrote:
> > On Thu, Feb 13, 2020 at 03:10:24PM +0100, Jethro Beekman wrote:
> >>>> There are other scenarios where it's not just the permissions on
> >>>> /dev/sgx/enclave that are the problem but using the filesystem in general
> >>>> that is. Maybe you've used seccomp to disable file operations, etc.
> >>>
> >>> Andy and Jarkko, thoughts?
> >>
> >> Folks, any more thoughts on how to resolve the issue that you need to
> >> call open() for every enclave?
> > 
> > Why is it an issue?
> 
> Already discussed in https://www.spinics.net/lists/linux-sgx/msg02075.html

Not anyone has to have access to open /dev/sgx/enclave in order to use
enclaves.

It is as much a problem as for practically any driver that provides
devices for some use.

/Jarkko

^ permalink raw reply	[flat|nested] 22+ messages in thread

* Re: x86/sgx: v23-rc2
  2020-02-17 18:55               ` Jarkko Sakkinen
@ 2020-02-17 18:56                 ` Jarkko Sakkinen
  2020-02-18 10:42                 ` Dr. Greg Wettstein
  1 sibling, 0 replies; 22+ messages in thread
From: Jarkko Sakkinen @ 2020-02-17 18:56 UTC (permalink / raw)
  To: Jethro Beekman
  Cc: Sean Christopherson, Andy Lutomirski, linux-sgx, serge.ayoun,
	shay.katz-zamir

On Mon, Feb 17, 2020 at 08:55:26PM +0200, Jarkko Sakkinen wrote:
> On Mon, Feb 17, 2020 at 09:52:25AM +0100, Jethro Beekman wrote:
> > On 2020-02-15 08:24, Jarkko Sakkinen wrote:
> > > On Thu, Feb 13, 2020 at 03:10:24PM +0100, Jethro Beekman wrote:
> > >>>> There are other scenarios where it's not just the permissions on
> > >>>> /dev/sgx/enclave that are the problem but using the filesystem in general
> > >>>> that is. Maybe you've used seccomp to disable file operations, etc.
> > >>>
> > >>> Andy and Jarkko, thoughts?
> > >>
> > >> Folks, any more thoughts on how to resolve the issue that you need to
> > >> call open() for every enclave?
> > > 
> > > Why is it an issue?
> > 
> > Already discussed in https://www.spinics.net/lists/linux-sgx/msg02075.html
> 
> Not anyone has to have access to open /dev/sgx/enclave in order to use
      ~~~~~~
      everyone

/Jarkko

^ permalink raw reply	[flat|nested] 22+ messages in thread

* Re: x86/sgx: v23-rc2
  2020-02-17 18:55               ` Jarkko Sakkinen
  2020-02-17 18:56                 ` Jarkko Sakkinen
@ 2020-02-18 10:42                 ` Dr. Greg Wettstein
  2020-02-18 15:00                   ` Andy Lutomirski
  2020-02-18 15:52                   ` Jarkko Sakkinen
  1 sibling, 2 replies; 22+ messages in thread
From: Dr. Greg Wettstein @ 2020-02-18 10:42 UTC (permalink / raw)
  To: Jarkko Sakkinen
  Cc: Jethro Beekman, Sean Christopherson, Andy Lutomirski, linux-sgx,
	serge.ayoun, shay.katz-zamir

On Mon, Feb 17, 2020 at 08:55:26PM +0200, Jarkko Sakkinen wrote:

Good morning, I hope the week is starting well for everyone.

> On Mon, Feb 17, 2020 at 09:52:25AM +0100, Jethro Beekman wrote:
> > On 2020-02-15 08:24, Jarkko Sakkinen wrote:
> > > On Thu, Feb 13, 2020 at 03:10:24PM +0100, Jethro Beekman wrote:
> > >>>> There are other scenarios where it's not just the permissions on
> > >>>> /dev/sgx/enclave that are the problem but using the filesystem in general
> > >>>> that is. Maybe you've used seccomp to disable file operations, etc.
> > >>>
> > >>> Andy and Jarkko, thoughts?
> > >>
> > >> Folks, any more thoughts on how to resolve the issue that you need to
> > >> call open() for every enclave?
> > > 
> > > Why is it an issue?
> > 
> > Already discussed in https://www.spinics.net/lists/linux-sgx/msg02075.html

I believe an accurate summary of Dr. Beekman's concerns are as
follows:

1.) He envisions a need for an enclave orchestrator that uses root
privileges to open the SGX driver device and then drop privileges,
presumably in a permanent fashion.  The orchestrator would then use
the filehandle to load and initialize multiple enclaves on request.

2.) The enclave orchestrator may be run in an environment that has
SECCOMP limitations on the ability to conduct filesystem operations.

> Not everyone has to have access to open /dev/sgx/enclave in order to
> use enclaves.

That depends on the architecture of the runtime.

The Intel PSW has a model where the AESM daemon orchestrates
activities for applications that wish to use enclaves.

Our Secure Runtime Development Environment (SRDE), on the other hand,
uses a model where each application independently manages its use of
enclaves.  In our model, the SGX driver needs to be accessible by
whatever privilege level an application chooses to run at.

As a result of this model, our SRDE has around an order of magnitude
less complexity then the AESM model and virtually no external
dependencies.  At last count the AESM depends on about 35 separate
shared library systems.

Our model allows us to build applications that are statically linked
and completely self-contained.  A concept of reasonable utility for
containerized environments and embedded systems.  We also avoid C++
that allows us to build MUSL based security stacks.

> It is as much a problem as for practically any driver that provides
> devices for some use.

The only customers of this driver are the runtime developers and that
is currently an extremely limited spectrum of users.

I will concede that I don't understand the SECCOMP issue.  Jethro may
or may not be able to discuss an architecture that would not have
basic filesystem operations available to it.

The fundamental issue, that there appears to be a reluctance to
discuss, is that DAC/MAC controls are not relevant to SGX,
particularly SGX2 based systems, which is what most of this stuff
will be running on.

> /Jarkko

Have a good day.

Dr. Greg

As always,
Dr. Greg Wettstein, Ph.D    Worker / Principal Engineer
IDfusion, LLC
4206 19th Ave N.            Specialists in SGX secured infrastructure.
Fargo, ND  58102
PH: 701-281-1686            CELL: 701-361-2319
EMAIL: gw@idfusion.org
------------------------------------------------------------------------------
"... remember that innovation is saying 'no' to 1000 things."
                                -- Moxie Marlinspike

^ permalink raw reply	[flat|nested] 22+ messages in thread

* Re: x86/sgx: v23-rc2
  2020-02-18 10:42                 ` Dr. Greg Wettstein
@ 2020-02-18 15:00                   ` Andy Lutomirski
  2020-02-22  3:16                     ` Dr. Greg
  2020-02-18 15:52                   ` Jarkko Sakkinen
  1 sibling, 1 reply; 22+ messages in thread
From: Andy Lutomirski @ 2020-02-18 15:00 UTC (permalink / raw)
  To: Dr. Greg Wettstein
  Cc: Jarkko Sakkinen, Jethro Beekman, Sean Christopherson,
	Andy Lutomirski, linux-sgx, serge.ayoun, shay.katz-zamir



> On Feb 18, 2020, at 2:43 AM, Dr. Greg Wettstein <gw@idfusion.org> wrote:
> 
> On Mon, Feb 17, 2020 at 08:55:26PM +0200, Jarkko Sakkinen wrote:
> 
> Good morning, I hope the week is starting well for everyone.
> 
>>> On Mon, Feb 17, 2020 at 09:52:25AM +0100, Jethro Beekman wrote:
>>> On 2020-02-15 08:24, Jarkko Sakkinen wrote:
>>>> On Thu, Feb 13, 2020 at 03:10:24PM +0100, Jethro Beekman wrote:
>>>>>>> There are other scenarios where it's not just the permissions on
>>>>>>> /dev/sgx/enclave that are the problem but using the filesystem in general
>>>>>>> that is. Maybe you've used seccomp to disable file operations, etc.
>>>>>> 
>>>>>> Andy and Jarkko, thoughts?
>>>>> 
>>>>> Folks, any more thoughts on how to resolve the issue that you need to
>>>>> call open() for every enclave?
>>>> 
>>>> Why is it an issue?
>>> 
>>> Already discussed in https://www.spinics.net/lists/linux-sgx/msg02075.html
> 
> I believe an accurate summary of Dr. Beekman's concerns are as
> follows:
> 
> 1.) He envisions a need for an enclave orchestrator that uses root
> privileges to open the SGX driver device and then drop privileges,
> presumably in a permanent fashion.  The orchestrator would then use
> the filehandle to load and initialize multiple enclaves on request.
> 
> 2.) The enclave orchestrator may be run in an environment that has
> SECCOMP limitations on the ability to conduct filesystem operations.
> 
>> Not everyone has to have access to open /dev/sgx/enclave in order to
>> use enclaves.
> 
> That depends on the architecture of the runtime.
> 
> The Intel PSW has a model where the AESM daemon orchestrates
> activities for applications that wish to use enclaves.
> 
> Our Secure Runtime Development Environment (SRDE), on the other hand,
> uses a model where each application independently manages its use of
> enclaves.  In our model, the SGX driver needs to be accessible by
> whatever privilege level an application chooses to run at.
> 
> As a result of this model, our SRDE has around an order of magnitude
> less complexity then the AESM model and virtually no external
> dependencies.  At last count the AESM depends on about 35 separate
> shared library systems.
> 
> Our model allows us to build applications that are statically linked
> and completely self-contained.  A concept of reasonable utility for
> containerized environments and embedded systems.  We also avoid C++
> that allows us to build MUSL based security stacks.
> 
>> It is as much a problem as for practically any driver that provides
>> devices for some use.
> 
> The only customers of this driver are the runtime developers and that
> is currently an extremely limited spectrum of users.
> 
> I will concede that I don't understand the SECCOMP issue.  Jethro may
> or may not be able to discuss an architecture that would not have
> basic filesystem operations available to it.

Jethro, is there a way you could use SCM_RIGHTS to solve your problem?  In principle, the driver could allow an enclave fd that isn’t ECREATEd yet to be forked with an ioctl, but ISTM this is extra complexity and there should be a well understood use case before something like this is added.

> 
> The fundamental issue, that there appears to be a reluctance to
> discuss, is that DAC/MAC controls are not relevant to SGX,
> particularly SGX2 based systems, which is what most of this stuff
> will be running on.
> 

No, this isn’t the fundamental issue, because DAC/MAC is just as relevant to systems with SGX and SGX2.  I’m sure that there will be systems that work with SGX and with everything running as root (or even SGX on a kernel without access control at all), but these will be a minority. Even if you have absolutely perfect protection of sensitive data, you sacrifice any sort of defense in depth against an attacker DoSing you in any number of amazing ways.  Or, for that matter, leaking keys derived from the provisioning key.

(Don’t forget that, no matter how carefully your system might lock down the SGXPUBKEYHASH MSRs and control the associated keys, that lockdown is being done by non-enclave *software*, and a bad guy who gets root is quite likely to be able to unlock the registers by upgrading their root access to control the firmware.  Or get a launch token that can’t be revoked because SGX doesn’t have a token expiration mechanism.)

No one wants to discuss your DAC-less model because, as far as I can tell, no one agrees with you.

>> /Jarkko
> 
> Have a good day.
> 
> Dr. Greg
> 
> As always,
> Dr. Greg Wettstein, Ph.D    Worker / Principal Engineer
> IDfusion, LLC
> 4206 19th Ave N.            Specialists in SGX secured infrastructure.
> Fargo, ND  58102
> PH: 701-281-1686            CELL: 701-361-2319
> EMAIL: gw@idfusion.org
> ------------------------------------------------------------------------------
> "... remember that innovation is saying 'no' to 1000 things."
>                                -- Moxie Marlinspike

^ permalink raw reply	[flat|nested] 22+ messages in thread

* Re: x86/sgx: v23-rc2
  2020-02-18 10:42                 ` Dr. Greg Wettstein
  2020-02-18 15:00                   ` Andy Lutomirski
@ 2020-02-18 15:52                   ` Jarkko Sakkinen
  2020-02-19 16:26                     ` Dr. Greg
  1 sibling, 1 reply; 22+ messages in thread
From: Jarkko Sakkinen @ 2020-02-18 15:52 UTC (permalink / raw)
  To: Dr. Greg Wettstein
  Cc: Jethro Beekman, Sean Christopherson, Andy Lutomirski, linux-sgx,
	serge.ayoun, shay.katz-zamir

On Tue, Feb 18, 2020 at 04:42:43AM -0600, Dr. Greg Wettstein wrote:
> I believe an accurate summary of Dr. Beekman's concerns are as
> follows:
> 
> 1.) He envisions a need for an enclave orchestrator that uses root
> privileges to open the SGX driver device and then drop privileges,
> presumably in a permanent fashion.  The orchestrator would then use
> the filehandle to load and initialize multiple enclaves on request.
> 
> 2.) The enclave orchestrator may be run in an environment that has
> SECCOMP limitations on the ability to conduct filesystem operations.

Also UDS sockets with SCM_RIGHTS should work.

/Jarkko

^ permalink raw reply	[flat|nested] 22+ messages in thread

* Re: x86/sgx: v23-rc2
  2020-02-18 15:52                   ` Jarkko Sakkinen
@ 2020-02-19 16:26                     ` Dr. Greg
  2020-02-20 19:57                       ` Jarkko Sakkinen
  0 siblings, 1 reply; 22+ messages in thread
From: Dr. Greg @ 2020-02-19 16:26 UTC (permalink / raw)
  To: Jarkko Sakkinen
  Cc: Jethro Beekman, Sean Christopherson, Andy Lutomirski, linux-sgx,
	serge.ayoun, shay.katz-zamir

On Tue, Feb 18, 2020 at 05:52:47PM +0200, Jarkko Sakkinen wrote:

Good morning, I hope the day is going well for everyone.

> On Tue, Feb 18, 2020 at 04:42:43AM -0600, Dr. Greg Wettstein wrote:
> > I believe an accurate summary of Dr. Beekman's concerns are as
> > follows:
> > 
> > 1.) He envisions a need for an enclave orchestrator that uses root
> > privileges to open the SGX driver device and then drop privileges,
> > presumably in a permanent fashion.  The orchestrator would then use
> > the filehandle to load and initialize multiple enclaves on request.
> > 
> > 2.) The enclave orchestrator may be run in an environment that has
> > SECCOMP limitations on the ability to conduct filesystem operations.

> Also UDS sockets with SCM_RIGHTS should work.

The first clarification, I'm assuming you mean passing SCM_RIGHTS over
AF_UNIX/UNIX-domain sockets, not UDP sockets?

Secondly, it was my understanding that the reason major renovations to
the driver were initiated, almost a year ago now, was to create a
situation where there was a filehandle per enclave in order for the
LSM to get a 'look' at the executable TEXT that was being deployed to
the enclave.

It appears that the sgx_open() function, that is installed as the
->open method for the /dev/sgx/enclave device node, allocates the
master enclave definition structure, that is installed, and
subsequently released, as private data for the file instance object.
This master enclave structure has the reference to the virtual memory
definition for the enclave that is opened.

This would seem to imply that the driver is rather firmly architected
on the notion of one open() per enclave, a concept that Jethro seems
to have issues with.

> /Jarkko

Have a good day.

Dr. Greg

As always,
Dr. Greg Wettstein, Ph.D, Worker
IDfusion, LLC               SGX secured infrastructure and
4206 N. 19th Ave.           autonomously self-defensive platforms.
Fargo, ND  58102
PH: 701-281-1686            EMAIL: greg@idfusion.net
------------------------------------------------------------------------------
"Artifical Intelligence stands no chance against Natural Stupidity."
                                -- John Henders

^ permalink raw reply	[flat|nested] 22+ messages in thread

* Re: x86/sgx: v23-rc2
  2020-02-19 16:26                     ` Dr. Greg
@ 2020-02-20 19:57                       ` Jarkko Sakkinen
  2020-02-21  1:19                         ` Dr. Greg
  0 siblings, 1 reply; 22+ messages in thread
From: Jarkko Sakkinen @ 2020-02-20 19:57 UTC (permalink / raw)
  To: Dr. Greg
  Cc: Jethro Beekman, Sean Christopherson, Andy Lutomirski, linux-sgx,
	serge.ayoun, shay.katz-zamir

On Wed, Feb 19, 2020 at 10:26:40AM -0600, Dr. Greg wrote:
> On Tue, Feb 18, 2020 at 05:52:47PM +0200, Jarkko Sakkinen wrote:
> 
> Good morning, I hope the day is going well for everyone.
> 
> > On Tue, Feb 18, 2020 at 04:42:43AM -0600, Dr. Greg Wettstein wrote:
> > > I believe an accurate summary of Dr. Beekman's concerns are as
> > > follows:
> > > 
> > > 1.) He envisions a need for an enclave orchestrator that uses root
> > > privileges to open the SGX driver device and then drop privileges,
> > > presumably in a permanent fashion.  The orchestrator would then use
> > > the filehandle to load and initialize multiple enclaves on request.
> > > 
> > > 2.) The enclave orchestrator may be run in an environment that has
> > > SECCOMP limitations on the ability to conduct filesystem operations.
> 
> > Also UDS sockets with SCM_RIGHTS should work.
> 
> The first clarification, I'm assuming you mean passing SCM_RIGHTS over
> AF_UNIX/UNIX-domain sockets, not UDP sockets?

UNIX domain sockets.

> Secondly, it was my understanding that the reason major renovations to
> the driver were initiated, almost a year ago now, was to create a
> situation where there was a filehandle per enclave in order for the
> LSM to get a 'look' at the executable TEXT that was being deployed to
> the enclave.

File handle per enclave was driven by many things, e.g. ability to
access enclave from multiple processes either through forking or UDS
sockets.

LSM specifc changes are not part of the patch set but file handle based
model is adaptable to LSM's later on.

> It appears that the sgx_open() function, that is installed as the
> ->open method for the /dev/sgx/enclave device node, allocates the
> master enclave definition structure, that is installed, and
> subsequently released, as private data for the file instance object.
> This master enclave structure has the reference to the virtual memory
> definition for the enclave that is opened.

You can have multiple references to an enclave by using formentioned
tools.

> This would seem to imply that the driver is rather firmly architected
> on the notion of one open() per enclave, a concept that Jethro seems
> to have issues with.

I don't understand what concept you are talking about.

/Jarkko

^ permalink raw reply	[flat|nested] 22+ messages in thread

* Re: x86/sgx: v23-rc2
  2020-02-20 19:57                       ` Jarkko Sakkinen
@ 2020-02-21  1:19                         ` Dr. Greg
  2020-02-21 13:00                           ` Jarkko Sakkinen
  0 siblings, 1 reply; 22+ messages in thread
From: Dr. Greg @ 2020-02-21  1:19 UTC (permalink / raw)
  To: Jarkko Sakkinen
  Cc: Jethro Beekman, Sean Christopherson, Andy Lutomirski, linux-sgx,
	serge.ayoun, shay.katz-zamir

On Thu, Feb 20, 2020 at 09:57:11PM +0200, Jarkko Sakkinen wrote:

Good evening to everyone.

> On Wed, Feb 19, 2020 at 10:26:40AM -0600, Dr. Greg wrote:
> > On Tue, Feb 18, 2020 at 05:52:47PM +0200, Jarkko Sakkinen wrote:
> > 
> > Good morning, I hope the day is going well for everyone.
> > 
> > > On Tue, Feb 18, 2020 at 04:42:43AM -0600, Dr. Greg Wettstein wrote:
> > > > I believe an accurate summary of Dr. Beekman's concerns are as
> > > > follows:
> > > > 
> > > > 1.) He envisions a need for an enclave orchestrator that uses root
> > > > privileges to open the SGX driver device and then drop privileges,
> > > > presumably in a permanent fashion.  The orchestrator would then use
> > > > the filehandle to load and initialize multiple enclaves on request.
> > > > 
> > > > 2.) The enclave orchestrator may be run in an environment that has
> > > > SECCOMP limitations on the ability to conduct filesystem operations.
> > 
> > > Also UDS sockets with SCM_RIGHTS should work.
> > 
> > The first clarification, I'm assuming you mean passing SCM_RIGHTS over
> > AF_UNIX/UNIX-domain sockets, not UDP sockets?

> UNIX domain sockets.

That is what I thought, I wanted to make sure that UDS wasn't a typo.

> > It appears that the sgx_open() function, that is installed as the
> > ->open method for the /dev/sgx/enclave device node, allocates the
> > master enclave definition structure, that is installed, and
> > subsequently released, as private data for the file instance object.
> > This master enclave structure has the reference to the virtual memory
> > definition for the enclave that is opened.

> You can have multiple references to an enclave by using formentioned
> tools.

Yes, I understand multiple references to a file descriptor/enclave,
but see below.

> > This would seem to imply that the driver is rather firmly architected
> > on the notion of one open() per enclave, a concept that Jethro seems
> > to have issues with.

> I don't understand what concept you are talking about.

If memory serves me correctly, Jethro envisioned a model where a
single open of the SGX driver node would return a file descriptor that
could then be used to create/load/initialize multiple enclaves.  Your
clarifications indicate that a separate open will be needed for each
and every enclave instance that will be orchestrated.

Jethro, if I'm mistating your position on this, please jump in and
clarify.

> /Jarkko

Have a good end of the week.

Dr. Greg

As always,
Dr. Greg Wettstein, Ph.D, Worker
IDfusion, LLC               SGX secured infrastructure and
4206 N. 19th Ave.           autonomously self-defensive platforms.
Fargo, ND  58102
PH: 701-281-1686            EMAIL: greg@idfusion.net
------------------------------------------------------------------------------
"Real Programmers consider "what you see is what you get" to be just as
 bad a concept in Text Editors as it is in women.  No, the Real
 Programmer wants a "you asked for it, you got it" text editor --
 complicated, cryptic, powerful, unforgiving, dangerous."
                                -- Matthias Schniedermeyer

^ permalink raw reply	[flat|nested] 22+ messages in thread

* Re: x86/sgx: v23-rc2
  2020-02-21  1:19                         ` Dr. Greg
@ 2020-02-21 13:00                           ` Jarkko Sakkinen
  0 siblings, 0 replies; 22+ messages in thread
From: Jarkko Sakkinen @ 2020-02-21 13:00 UTC (permalink / raw)
  To: Dr. Greg
  Cc: Jethro Beekman, Sean Christopherson, Andy Lutomirski, linux-sgx,
	serge.ayoun, shay.katz-zamir

On Thu, Feb 20, 2020 at 07:19:13PM -0600, Dr. Greg wrote:
> > > This would seem to imply that the driver is rather firmly architected
> > > on the notion of one open() per enclave, a concept that Jethro seems
> > > to have issues with.
> 
> > I don't understand what concept you are talking about.
> 
> If memory serves me correctly, Jethro envisioned a model where a
> single open of the SGX driver node would return a file descriptor that
> could then be used to create/load/initialize multiple enclaves.  Your
> clarifications indicate that a separate open will be needed for each
> and every enclave instance that will be orchestrated.
> 
> Jethro, if I'm mistating your position on this, please jump in and
> clarify.

Ah.

You are speaking about having a factory to create enclaves and a
management interface. I.e. you'd have ioctl to create enclave that gives
you a file descriptor to access its management interface.

Out of top of my head I cannot recall why this was not favored in the
end but generally speaking added complexity should be justified by some
considerably strong measures.

/Jarkko

^ permalink raw reply	[flat|nested] 22+ messages in thread

* Re: x86/sgx: v23-rc2
  2020-02-18 15:00                   ` Andy Lutomirski
@ 2020-02-22  3:16                     ` Dr. Greg
  2020-02-22  5:41                       ` Andy Lutomirski
  0 siblings, 1 reply; 22+ messages in thread
From: Dr. Greg @ 2020-02-22  3:16 UTC (permalink / raw)
  To: Andy Lutomirski
  Cc: Jarkko Sakkinen, Jethro Beekman, Sean Christopherson, linux-sgx,
	serge.ayoun, shay.katz-zamir

On Tue, Feb 18, 2020 at 07:00:23AM -0800, Andy Lutomirski wrote:

Good evening Andy, always good to hear from you.

> > On Feb 18, 2020, at 2:43 AM, Dr. Greg Wettstein <gw@idfusion.org> wrote:
> > The fundamental issue, that there appears to be a reluctance to
> > discuss, is that DAC/MAC controls are not relevant to SGX,
> > particularly SGX2 based systems, which is what most of this stuff
> > will be running on.

> No, this isn't the fundamental issue, because DAC/MAC is just as
> relevant to systems with SGX and SGX2.  I'm sure that there will
> be systems that work with SGX and with everything running as root
> (or even SGX on a kernel without access control at all), but these
> will be a minority. Even if you have absolutely perfect protection
> of sensitive data, you sacrifice any sort of defense in depth
> against an attacker DoSing you in any number of amazing ways.  Or,
> for that matter, leaking keys derived from the provisioning key.

I should have been more precise, my apologies.

When I was suggesting that DAC/MAC controls were not relevant, I was
referring to their relevance with respect to protecting the platform
from running code that does not have defined provenance or origin.

The last time we had this conversation, it spurred you to make issue
of the fact that the current driver allows writable memory to be
executable, without the contents of the memory being properly vetted
by the LSM, without doubt an important issue.  LWN even ran an article
on the issue.

I'm certainly not argueing against defense in depth or the merits of
DAC/MAC controls.  I'm simply stating that they are irrelevant, with
respect to the concerns that you raised about code provenance and
origin, on flexible launch control platforms that support SGX2 and
EDMM.  Platforms that by and large will be the primary target for this
driver.

I have no illusions of convincing you or anyone else involved in the
development of this driver, of the merits of our concern.  However,
anyone who has worked on this technology at length, and is
intellectually honest, knows it is a legitimate security issue.

Bruce Schneier even weighed in on the issue, although from a somewhat
different perspective:

https://www.schneier.com/blog/archives/2017/03/using_intels_sg.html

I only bring up the issue in the interests of intellectual honesty and
the fact that I'm sure our exchanges will go down in the annals of
security history as being as important as the Lincoln/Douglas debates.

It is one of my jobs to follow the security literature closely.
Everyone who does so can easily envision a future abstract reading
something like the following:

"Using an unmodified and stock DIST_OS kernel we were able to
demonstrate the download and execution of malicious code into an
enclave that bypasses all security controls in the Linux operating
system allowing us to recover XXXX information from the platform with
subsequent exfiltration of the information from the platform without
detection."

:-)

FWIW, from your statement above, it is unclear how DAC/MAC controls of
any type or depth would prevent leaking keys derived from the
provisioning key.  One prevents leakage of provisioning derived keys
by implementing cryptographic provenance of code execution so that you
have some modicum of trust that enclave code that has access to the
provisioning key can be trusted to 'do the right thing'.

> (Don't forget that, no matter how carefully your system might lock
> down the SGXPUBKEYHASH MSRs and control the associated keys, that
> lockdown is being done by non-enclave *software*, and a bad guy who
> gets root is quite likely to be able to unlock the registers by
> upgrading their root access to control the firmware.  Or get a launch
> token that can't be revoked because SGX doesn't have a token
> expiration mechanism.)

Dispassionate observers would note that you make the case for locked
launch control registers.... :-)

At an SGX engineering meeting in Israel last summer, we made the case
for the fact that locked vs. unlocked platforms should be a BIOS
configurable option.  With an additional option that specifying locked
also allows specification of what the identity modulus signature
should be.  That would seem to be the best of all worlds, we will see
what happens.

One of the pushbacks we received, is that SGX is supposed to be immune
from firmware manipulation, which our suggested approach would open
the door for, which we noted was irrelevant given the trajectory that
the Linux kernel driver is on, ie. no cryptographic controls over code
origin and provenance.

Just to provide a frame of reference, our interest in SGX is with
respect to its guarantee of integrity of execution, for the purposes
of verifying that the kernel could not have executed code that was
outside a desired behavioral definition for the platform.

We namespaced a modified implementation of Linux IMA and our container
launch system pairs each behavioral canister with an SGX enclave that
introspects the namespace's behavior.  The enclave labels a process
attempting an extra-dimensional Turing event as a 'bad actor', subject
to appropriate discipline by a modified LSM.

So known cryptographic integrity and provenance of executed code
positions us to make a strong attestation statement regarding the
integrity of the platform as a whole.  There is a significant body of
interest in the ability to support such statements.

We also addressed the problem of perpetual launch tokens but that is
an issue for another time.

> No one wants to discuss your DAC-less model because, as far as I can
> tell, no one agrees with you.

Stating that no one, in a world of 7.7 billion people, agrees with us
on these issues would seem to make that contention vulnerable to any
finding of a defect in absolutism.

The number of downloads of our patch set and GIT pulls of our modified
version of Jarkko's driver speaks otherwise, and has convinced us to
expend the cycles needed to continue maintaining and making available
an approach that provides some notion of cryptographic based
provenance and origin of SGX based execution on FLC platforms.

We tend to take solace and energy from the fact that Henry Ford's
contemporaries criticized him for not spending his time breeding
faster horses that ate and drank less.

Best wishes for an enjoyable and productive weekend to everyone.

Dr. Greg

As always,
Dr. Greg Wettstein, Ph.D, Worker
IDfusion, LLC               SGX secured infrastructure and
4206 N. 19th Ave.           autonomously self-defensive platforms.
Fargo, ND  58102
PH: 701-281-1686            EMAIL: greg@idfusion.net
------------------------------------------------------------------------------
"Now Ed, is there a need to be concerned about edging?"
                                -- Alan Biddison - Snowmass 2019
                                   Resurrection.

^ permalink raw reply	[flat|nested] 22+ messages in thread

* Re: x86/sgx: v23-rc2
  2020-02-22  3:16                     ` Dr. Greg
@ 2020-02-22  5:41                       ` Andy Lutomirski
  0 siblings, 0 replies; 22+ messages in thread
From: Andy Lutomirski @ 2020-02-22  5:41 UTC (permalink / raw)
  To: Dr. Greg
  Cc: Jarkko Sakkinen, Jethro Beekman, Sean Christopherson, linux-sgx,
	serge.ayoun, shay.katz-zamir

On Fri, Feb 21, 2020 at 7:16 PM Dr. Greg <greg@enjellic.com> wrote:
>
> On Tue, Feb 18, 2020 at 07:00:23AM -0800, Andy Lutomirski wrote:
>
> > > On Feb 18, 2020, at 2:43 AM, Dr. Greg Wettstein <gw@idfusion.org> wrote:

> > No, this isn't the fundamental issue, because DAC/MAC is just as
> > relevant to systems with SGX and SGX2.  I'm sure that there will
> > be systems that work with SGX and with everything running as root
> > (or even SGX on a kernel without access control at all), but these
> > will be a minority. Even if you have absolutely perfect protection
> > of sensitive data, you sacrifice any sort of defense in depth
> > against an attacker DoSing you in any number of amazing ways.  Or,
> > for that matter, leaking keys derived from the provisioning key.
>
> I should have been more precise, my apologies.
>
> When I was suggesting that DAC/MAC controls were not relevant, I was
> referring to their relevance with respect to protecting the platform
> from running code that does not have defined provenance or origin.

Windows systems quite regularly get owned by code with "defined
provenance or origin" in the sense that it was signed by a valid
Authenticode certificate. :)

> I'm certainly not argueing against defense in depth or the merits of
> DAC/MAC controls.  I'm simply stating that they are irrelevant, with
> respect to the concerns that you raised about code provenance and
> origin, on flexible launch control platforms that support SGX2 and
> EDMM.  Platforms that by and large will be the primary target for this
> driver.

I have a little FLC platform on my desk.  What I don't have is an FLC
platform with anything resembling a policy that helps here at all.

> I only bring up the issue in the interests of intellectual honesty and
> the fact that I'm sure our exchanges will go down in the annals of
> security history as being as important as the Lincoln/Douglas debates.

That would be amazing!

>
> It is one of my jobs to follow the security literature closely.
> Everyone who does so can easily envision a future abstract reading
> something like the following:
>
> "Using an unmodified and stock DIST_OS kernel we were able to
> demonstrate the download and execution of malicious code into an
> enclave that bypasses all security controls in the Linux operating
> system allowing us to recover XXXX information from the platform with
> subsequent exfiltration of the information from the platform without
> detection."
>

Alas, the significance of this seems to depend on what XXXX is.  If
XXXX is a detail of the specific machine (a derived provisioning key,
a launch token, etc.), then the paper won't be terribly interesting.
As root (or someone who can escalate to root or better) on a given
system, I can extract *tons* of identifying details of the platform.
If XXXX is a secret created by the user (a sealing key, something
sealed by SGX, a private key or secret that allows one to pretend to
be an enclave with MRSIGNER=YYY, etc), then launch control and DAC are
essentially irrelevant to the discussion.  What actually happened is
that some code signed by YYY screwed up.  And this is because launch
control is not part of the SGX root of trust, nor is launch control
even a necessary part of the SGX security model.  SGX with launch
control deleted entirely would be just as useful, if potentially a bit
more dangerous in terms of being able to make malware hard to analyze.

> > (Don't forget that, no matter how carefully your system might lock
> > down the SGXPUBKEYHASH MSRs and control the associated keys, that
> > lockdown is being done by non-enclave *software*, and a bad guy who
> > gets root is quite likely to be able to unlock the registers by
> > upgrading their root access to control the firmware.  Or get a launch
> > token that can't be revoked because SGX doesn't have a token
> > expiration mechanism.)
>
> Dispassionate observers would note that you make the case for locked
> launch control registers.... :-)
>
> At an SGX engineering meeting in Israel last summer, we made the case
> for the fact that locked vs. unlocked platforms should be a BIOS
> configurable option.  With an additional option that specifying locked
> also allows specification of what the identity modulus signature
> should be.  That would seem to be the best of all worlds, we will see
> what happens.
>
> One of the pushbacks we received, is that SGX is supposed to be immune
> from firmware manipulation, which our suggested approach would open
> the door for, which we noted was irrelevant given the trajectory that
> the Linux kernel driver is on, ie. no cryptographic controls over code
> origin and provenance.

I don't believe I have ever said that Linux shouldn't support
cryptographic verification of SGX code provenance or that Linux
shouldn't eventually support locked launch control MSRs.  What I've
said is that the initial upstream version of the driver doesn't need
this, and that adding support for locked MSRs is tricky and needs some
design work.

FWIW, I didn't expect upstreaming to take anywhere near this long.

>
> Just to provide a frame of reference, our interest in SGX is with
> respect to its guarantee of integrity of execution, for the purposes
> of verifying that the kernel could not have executed code that was
> outside a desired behavioral definition for the platform.

SGX cannot and will never prevent the kernel from executing normal
CPL0 code of any sort.  I'm not really sure what you're getting at
here.  If you have magically trustworthy firmware and locked launch
control registers, you can prevent the kernel from executing
unapproved *enclave* code.  It's not entirely clear to me why you
consider unapproved enclave code to be worse than any other sort of
unapproved code.

^ permalink raw reply	[flat|nested] 22+ messages in thread

end of thread, back to index

Thread overview: 22+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2019-10-10 11:37 x86/sgx: v23-rc2 Jarkko Sakkinen
2019-10-10 13:37 ` Jarkko Sakkinen
2019-10-10 17:09   ` Sean Christopherson
2019-10-10 17:39     ` Sean Christopherson
2019-10-11 16:37 ` Jethro Beekman
2019-10-11 18:15   ` Sean Christopherson
2019-10-14  8:43     ` Jethro Beekman
2019-10-17 17:57       ` Sean Christopherson
2020-02-13 14:10         ` Jethro Beekman
2020-02-15  7:24           ` Jarkko Sakkinen
2020-02-17  8:52             ` Jethro Beekman
2020-02-17 18:55               ` Jarkko Sakkinen
2020-02-17 18:56                 ` Jarkko Sakkinen
2020-02-18 10:42                 ` Dr. Greg Wettstein
2020-02-18 15:00                   ` Andy Lutomirski
2020-02-22  3:16                     ` Dr. Greg
2020-02-22  5:41                       ` Andy Lutomirski
2020-02-18 15:52                   ` Jarkko Sakkinen
2020-02-19 16:26                     ` Dr. Greg
2020-02-20 19:57                       ` Jarkko Sakkinen
2020-02-21  1:19                         ` Dr. Greg
2020-02-21 13:00                           ` Jarkko Sakkinen

Linux-Sgx Archive on lore.kernel.org

Archives are clonable:
	git clone --mirror https://lore.kernel.org/linux-sgx/0 linux-sgx/git/0.git

	# If you have public-inbox 1.1+ installed, you may
	# initialize and index your mirror using the following commands:
	public-inbox-init -V2 linux-sgx linux-sgx/ https://lore.kernel.org/linux-sgx \
		linux-sgx@vger.kernel.org
	public-inbox-index linux-sgx

Example config snippet for mirrors

Newsgroup available over NNTP:
	nntp://nntp.lore.kernel.org/org.kernel.vger.linux-sgx


AGPL code for this site: git clone https://public-inbox.org/public-inbox.git