Linux-Sgx Archive on lore.kernel.org
 help / color / Atom feed
From: Sean Christopherson <sean.j.christopherson@intel.com>
To: Andy Lutomirski <luto@kernel.org>
Cc: Dave Hansen <dave.hansen@intel.com>,
	Jarkko Sakkinen <jarkko.sakkinen@linux.intel.com>,
	X86 ML <x86@kernel.org>,
	Platform Driver <platform-driver-x86@vger.kernel.org>,
	linux-sgx@vger.kernel.org, nhorman@redhat.com,
	npmccallum@redhat.com, "Ayoun, Serge" <serge.ayoun@intel.com>,
	shay.katz-zamir@intel.com,
	Haitao Huang <haitao.huang@linux.intel.com>,
	Andy Shevchenko <andriy.shevchenko@linux.intel.com>,
	Thomas Gleixner <tglx@linutronix.de>,
	"Svahn, Kai" <kai.svahn@intel.com>,
	mark.shanahan@intel.com,
	Suresh Siddha <suresh.b.siddha@intel.com>,
	Ingo Molnar <mingo@redhat.com>, Borislav Petkov <bp@alien8.de>,
	"H. Peter Anvin" <hpa@zytor.com>,
	Darren Hart <dvhart@infradead.org>,
	Andy Shevchenko <andy@infradead.org>,
	"open list:X86 ARCHITECTURE (32-BIT AND 64-BIT)" 
	<linux-kernel@vger.kernel.org>
Subject: Re: [PATCH v17 18/23] platform/x86: Intel SGX driver
Date: Mon, 17 Dec 2018 14:20:48 -0800
Message-ID: <20181217222047.GG12491@linux.intel.com> (raw)
In-Reply-To: <CALCETrX0W4dhFJBHEbDbKed_QZxmw6ZNMajfsoVgqGqhMB0nDg@mail.gmail.com>

On Mon, Dec 17, 2018 at 11:12:21AM -0800, Andy Lutomirski wrote:
> On Mon, Dec 17, 2018 at 10:47 AM Dave Hansen <dave.hansen@intel.com> wrote:
> >
> > On 12/17/18 10:43 AM, Jarkko Sakkinen wrote:
> > > On Mon, Dec 17, 2018 at 10:36:13AM -0800, Sean Christopherson wrote:
> > >> I'm pretty sure doing mmget() would result in circular dependencies and
> > >> a zombie enclave.  In the do_exit() case where a task is abruptly killed:
> > >>
> > >>   - __mmput() is never called because the enclave holds a ref
> > >>   - sgx_encl_release() is never be called because its VMAs hold refs
> > >>   - sgx_vma_close() is never called because __mmput()->exit_mmap() is
> > >>     blocked and the process itself is dead, i.e. won't unmap anything.
> > > Right, it does, you are absolutely right. Tried it and removed the
> > > commit already.
> > >
> > > Well, what we came up from your suggestion i.e. setting mm to NULL
> > > and checking that is very subtle change and does not have any such
> > > circular dependencies. We'll go with that.
> >
> > This all screams that you need to break out this code from the massive
> > "18" patch and get the mm interactions reviewed more thoroughly.
> >
> > Also, no matter what method you go with, you have a bunch of commenting
> > and changelogging to do here.
> 
> I'm going to ask an obnoxious high-level question: why does an enclave
> even refer to a specific mm?

Primarily because that's what the code has "always" done.  I can't
speak for Jarkko, but I got involved with this joyful project long after
the code was originally written.

> If I were designing this thing, and if I hadn't started trying to
> implement it, my first thought would be that an enclave tracks its
> linear address range, which is just a pair of numbers, and also keeps
> track of a whole bunch of physical EPC pages, data structures, etc.
> And that mmap() gets rejected unless the requested virtual address
> matches the linear address range that the enclave wants and, aside
> from that, just creates a VMA that keeps a reference to the enclave.
> (And, for convenience, I suppose that the first mmap() call done
> before any actual enclave setup happens could choose any address and
> then cause the enclave to lock itself to that address, although a
> regular anonymous PROT_NONE MAP_NORESERVE mapping would do just fine,
> too.)  And the driver would explicitly allow multiple different mms to
> have the same enclave mapped.  More importantly, a daemon could set up
> an enclave without necessarily mapping it at all and then SCM_RIGHTS
> the enclave over to the process that plans to run it.

Hmm, this could work, the obvious weirdness would be ensuring the linear
range is available in the destination mm, but that'd be userspace's
problem.

I don't think we'd need to keep a reference to the enclave in the VMA.
The enclave's ref could be held by the fd.  Assuming the kernel is using
its private mapping to access the enclave, that's all we'd need to be
able to manipulate the enclave, e.g. reclaim EPC pages.  Userspace would
need to keep the fd alive in order to use the VMA, but that sort of goes
without saying.  The mm/VMA juggling today is for zapping/testing the
correct PTEs, but as you pointed out in a different email we can use
unmap_mapping_range(), with the enclave's fd being the source of the
address space passed to unmap_mapping_range().  Removing a VMA simply
means we don't need to zap it or test its age.
 
> Now I'm sure this has all kinds of problems, such as the ISA possibly
> making it rather obnoxious to add pages to the enclave without having
> it mapped.  But these operations could, in principle, be done by
> having the enclave own a private mm that's used just for setup.  While
> this would be vaguely annoying, Nadav's still-pending-but-nearly-done
> text_poke series adds all the infrastructure that's needed for the
> kernel to manage little private mms.  But some things get simpler --
> invalidating the enclave can presumably use the regular rmap APIs to
> zap all the PTEs in all VMAs pointing into the enclave.

We don't even need a private mm, we can (and already do) use the kernel's
translations for ENCLS instructions.  Hardware only enforces the linear
address stuff when it's actually in enclave mode, i.e. executing the
enclave.  ENCLS instructions aren't subject to the ELRANGE checks and
can use any VA->PA combination.

> So I'm not saying that you shouldn't do it the way you are now, but I
> do think that the changelog or at least some emails should explain
> *why* the enclave needs to keep a pointer to the creating process's
> mm.  And, if you do keep the current model, it would be nice to
> understand what happens if you do something awful like mremap()ing an
> enclave, or calling madvise on it, or otherwise abusing the vma.  Or
> doing fork(), for that matter.
> 
> I also find it suspicious that the various ioctl handlers
> systematically ignore their "filep" parameters and instead use
> find_vma() to find the relevant mm data structures.  That seems
> backwards.

My brain is still sorting out the details, but I generally like the idea
of allocating an anon inode when creating an enclave, and exposing the
other ioctls() via the returned fd.  This is essentially the approach
used by KVM to manage multiple "layers" of ioctls across KVM itself, VMs
and vCPUS.  There are even similarities to accessing physical memory via
multiple disparate domains, e.g. host kernel, host userspace and guest.

The only potential hiccup I can see is the build flow.  Currently,
EADD+EEXTEND is done via a work queue to avoid major performance issues
(10x regression) when userspace is building multiple enclaves in parallel
using goroutines to wrap Cgo (the issue might apply to any M:N scheduler,
but I've only confirmed the Golang case).  The issue is that allocating
an EPC page acts like a blocking syscall when the EPC is under pressure,
i.e. an EPC page isn't immediately available.  This causes Go's scheduler
to thrash and tank performance[1].

That being said, I think we could simply do mmgrab()/mmdrop() for each
page to be added, and then do mmget_not_zero()/mmput() when actually
inserting into the mm's page tables.  Conceptually that seems cleaner
than implicitly relying on the mmu_notifier to guarantee the lifecycle
of the mm.

Alternatively, we could change the EADD+EEXTEND flow to not insert the
added page's PFN into the owner's process space, i.e. force userspace to
fault when it runs the enclave.  But that only delays the issue because
eventually we'll want to account EPC pages, i.e. add a cgroup, at which
point we'll likely need current->mm anyways.

[1] https://github.com/golang/go/issues/19574

  parent reply index

Thread overview: 161+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
     [not found] <20181116010412.23967-1-jarkko.sakkinen@linux.intel.com>
2018-11-16  1:01 ` [PATCH v17 01/23] x86/sgx: Update MAINTAINERS Jarkko Sakkinen
2018-11-16 14:22   ` Borislav Petkov
2018-11-16 15:07     ` Jarkko Sakkinen
2018-11-16 20:24       ` Borislav Petkov
2018-11-18  8:20         ` Jarkko Sakkinen
2018-11-16  1:01 ` [PATCH v17 02/23] x86/cpufeatures: Add Intel-defined SGX feature bit Jarkko Sakkinen
2018-11-16 14:28   ` Borislav Petkov
2018-11-16 15:13     ` Jarkko Sakkinen
2018-11-16 15:18       ` Jarkko Sakkinen
2018-11-16 20:53         ` Borislav Petkov
2018-11-16  1:01 ` [PATCH v17 03/23] x86/cpufeatures: Add SGX sub-features (as Linux-defined bits) Jarkko Sakkinen
2018-11-16 14:37   ` Borislav Petkov
2018-11-16 15:38     ` Sean Christopherson
2018-11-16 23:31   ` Dave Hansen
2018-11-18  8:36     ` Jarkko Sakkinen
2018-11-16  1:01 ` [PATCH v17 04/23] x86/msr: Add IA32_FEATURE_CONTROL.SGX_ENABLE definition Jarkko Sakkinen
2018-11-16  1:01 ` [PATCH v17 05/23] x86/cpufeatures: Add Intel-defined SGX_LC feature bit Jarkko Sakkinen
2018-11-16  1:01 ` [PATCH v17 06/23] x86/cpu/intel: Detect SGX support and update caps appropriately Jarkko Sakkinen
2018-11-16 23:32   ` Dave Hansen
2018-11-18  8:37     ` Jarkko Sakkinen
2018-11-21 18:17   ` Borislav Petkov
2018-11-24 13:54     ` Jarkko Sakkinen
2018-11-16  1:01 ` [PATCH v17 07/23] x86/mm: x86/sgx: Add new 'PF_SGX' page fault error code bit Jarkko Sakkinen
2018-11-16 23:33   ` Dave Hansen
2018-11-18  8:38     ` Jarkko Sakkinen
2018-11-16  1:01 ` [PATCH v17 08/23] x86/mm: x86/sgx: Signal SIGSEGV for userspace #PFs w/ PF_SGX Jarkko Sakkinen
2018-11-16  1:01 ` [PATCH v17 09/23] x86/sgx: Define SGX1 and SGX2 ENCLS leafs Jarkko Sakkinen
2018-11-16  1:01 ` [PATCH v17 10/23] x86/sgx: Add ENCLS architectural error codes Jarkko Sakkinen
2018-11-16  1:01 ` [PATCH v17 11/23] x86/sgx: Add SGX1 and SGX2 architectural data structures Jarkko Sakkinen
2018-11-16  1:01 ` [PATCH v17 12/23] x86/sgx: Add definitions for SGX's CPUID leaf and variable sub-leafs Jarkko Sakkinen
2018-11-16  1:01 ` [PATCH v17 13/23] x86/msr: Add SGX Launch Control MSR definitions Jarkko Sakkinen
2018-11-16 17:29   ` Sean Christopherson
2018-11-18  8:19     ` Jarkko Sakkinen
2018-11-16  1:01 ` [PATCH v17 14/23] x86/sgx: Add wrappers for ENCLS leaf functions Jarkko Sakkinen
2018-11-16  1:01 ` [PATCH v17 15/23] x86/sgx: Enumerate and track EPC sections Jarkko Sakkinen
2018-11-16  1:01 ` [PATCH v17 16/23] x86/sgx: Add functions to allocate and free EPC pages Jarkko Sakkinen
2018-11-16  1:01 ` [PATCH v17 17/23] x86/sgx: Add sgx_einit() for initializing enclaves Jarkko Sakkinen
2018-11-16  1:01 ` [PATCH v17 18/23] platform/x86: Intel SGX driver Jarkko Sakkinen
2018-11-16  1:37   ` Randy Dunlap
2018-11-16 11:23     ` Jarkko Sakkinen
2018-11-19 15:06   ` Jarkko Sakkinen
2018-11-19 16:22     ` Jethro Beekman
2018-11-19 17:19       ` Jarkko Sakkinen
2018-11-19 17:39         ` Jethro Beekman
2018-11-20 10:58           ` Jarkko Sakkinen
2018-11-21 15:24             ` Jarkko Sakkinen
2018-11-19 18:18         ` Andy Lutomirski
2018-11-20 11:00           ` Jarkko Sakkinen
2018-11-19 15:29   ` Andy Lutomirski
2018-11-19 16:19     ` Jarkko Sakkinen
2018-11-19 16:59       ` Andy Lutomirski
2018-11-20 12:04         ` Jarkko Sakkinen
2018-11-22 11:12           ` Dr. Greg
2018-11-22 15:21             ` Andy Lutomirski
2018-11-24 17:21               ` Jarkko Sakkinen
2018-11-24 20:13                 ` Dr. Greg
2018-11-26 21:15                   ` Jarkko Sakkinen
2018-11-25 14:53                 ` Jarkko Sakkinen
2018-11-25 16:22                   ` Andy Lutomirski
2018-11-25 18:55                     ` Dr. Greg
2018-11-25 23:51                       ` Jarkko Sakkinen
     [not found]                       ` <D45BC005-5064-4C75-B486-4E43C454E2F6@amacapital.net>
2018-11-26  0:37                         ` Andy Lutomirski
2018-11-26 11:00                           ` Dr. Greg
2018-11-26 18:22                             ` Andy Lutomirski
2018-11-26 22:16                             ` Jarkko Sakkinen
2018-11-26 21:51                     ` Jarkko Sakkinen
2018-11-26 23:04                       ` Jarkko Sakkinen
2018-11-27  8:55                         ` Dr. Greg
2018-11-27 16:41                           ` Jarkko Sakkinen
2018-11-27 17:55                             ` Andy Lutomirski
2018-11-28 10:49                               ` Dr. Greg
2018-11-28 19:22                                 ` Jarkko Sakkinen
2018-12-10 10:49                                   ` Dr. Greg
2018-12-12 18:00                                     ` Jarkko Sakkinen
2018-12-14 23:59                                       ` Dr. Greg
2018-12-15  0:06                                         ` Sean Christopherson
2018-12-15 23:22                                           ` Dr. Greg
2018-12-17 14:27                                             ` Sean Christopherson
2018-12-17 13:28                                           ` Jarkko Sakkinen
2018-12-17 13:39                                             ` Jarkko Sakkinen
2018-12-17 14:08                                               ` Jarkko Sakkinen
2018-12-17 14:13                                                 ` Jarkko Sakkinen
2018-12-17 16:34                                                   ` Dr. Greg
2018-12-17 17:31                                                 ` Sean Christopherson
2018-12-17 17:49                                                   ` Jarkko Sakkinen
2018-12-17 18:09                                                     ` Sean Christopherson
2018-12-17 18:23                                                       ` Jarkko Sakkinen
2018-12-17 18:46                                                         ` Sean Christopherson
2018-12-17 19:36                                                           ` Jarkko Sakkinen
2018-11-27 16:46                           ` Jarkko Sakkinen
2018-11-28 21:52                           ` Andy Lutomirski
2018-11-27  7:46                       ` Jethro Beekman
2018-11-27 16:36                         ` Jarkko Sakkinen
2018-11-22 20:56             ` Andy Lutomirski
2018-11-23 10:39               ` Dr. Greg
2018-11-24 16:45                 ` Jarkko Sakkinen
2018-11-28  5:08                   ` Jarkko Sakkinen
2018-11-28  5:38                     ` Jethro Beekman
2018-12-09 17:01         ` Pavel Machek
2018-11-20 11:15     ` Dr. Greg
2018-11-24 16:15       ` Jarkko Sakkinen
2018-11-24 19:24         ` Dr. Greg
2018-11-26 19:39           ` Jarkko Sakkinen
2018-12-09 17:01     ` Pavel Machek
2018-12-10 14:46       ` Dr. Greg
2018-12-17 17:45   ` Dave Hansen
2018-12-17 18:01     ` Jarkko Sakkinen
2018-12-17 18:07       ` Dave Hansen
2018-12-17 18:31         ` Jarkko Sakkinen
2018-12-17 18:36       ` Sean Christopherson
2018-12-17 18:43         ` Jarkko Sakkinen
2018-12-17 18:47           ` Dave Hansen
2018-12-17 19:12             ` Andy Lutomirski
2018-12-17 19:17               ` Dave Hansen
2018-12-17 19:25                 ` Andy Lutomirski
2018-12-17 19:54                   ` Jarkko Sakkinen
2018-12-17 19:49                 ` Jarkko Sakkinen
2018-12-17 19:53                   ` Dave Hansen
2018-12-17 19:55                     ` Andy Lutomirski
2018-12-17 20:03                       ` Dave Hansen
2018-12-17 20:10                         ` Andy Lutomirski
2018-12-17 20:15                           ` Dave Hansen
2018-12-17 22:36                             ` Sean Christopherson
2018-12-18  1:40                           ` Jarkko Sakkinen
2018-12-17 22:20               ` Sean Christopherson [this message]
2018-12-18  1:39                 ` Jarkko Sakkinen
2018-12-18  3:27                   ` Jarkko Sakkinen
2018-12-18  5:02                     ` Andy Lutomirski
2018-12-18 13:27                       ` Jarkko Sakkinen
2018-12-18  4:55                   ` Andy Lutomirski
2018-12-18 13:18                     ` Jarkko Sakkinen
2018-12-18  4:59                 ` Andy Lutomirski
2018-12-18 13:11                   ` Jarkko Sakkinen
2018-12-18 15:44                   ` Sean Christopherson
2018-12-18 18:53                     ` Sean Christopherson
2018-12-19  5:00                       ` Jarkko Sakkinen
2018-12-19  5:13                         ` Jarkko Sakkinen
2018-12-21 18:28                         ` Sean Christopherson
2018-12-22  0:01                           ` Jarkko Sakkinen
2018-12-19  4:47                     ` Jarkko Sakkinen
2018-12-19  5:24                       ` Jarkko Sakkinen
2018-12-18  1:17               ` Jarkko Sakkinen
2018-12-18  1:31                 ` Jarkko Sakkinen
2018-12-17 18:48           ` Sean Christopherson
2018-12-17 19:09             ` Dave Hansen
2018-12-17 19:37               ` Jarkko Sakkinen
2018-12-17 19:40                 ` Dave Hansen
2018-12-17 19:33             ` Jarkko Sakkinen
2018-12-17 20:21               ` Jarkko Sakkinen
2018-12-18 13:13                 ` Jarkko Sakkinen
2018-12-18 15:46                   ` Sean Christopherson
2018-12-18  5:55   ` Andy Lutomirski
2018-12-19  5:22     ` Jarkko Sakkinen
2018-11-16  1:01 ` [PATCH v17 19/23] platform/x86: sgx: Add swapping functionality to the " Jarkko Sakkinen
2018-11-16  1:01 ` [PATCH v17 20/23] x86/sgx: Add a simple swapper for the EPC memory manager Jarkko Sakkinen
2018-11-16  1:01 ` [PATCH v17 21/23] platform/x86: ptrace() support for the SGX driver Jarkko Sakkinen
2018-11-16  1:01 ` [PATCH v17 22/23] x86/sgx: SGX documentation Jarkko Sakkinen
2018-12-03  3:28   ` Randy Dunlap
2018-12-03  9:32     ` Jarkko Sakkinen
2018-11-16  1:01 ` [PATCH v17 23/23] selftests/x86: Add a selftest for SGX Jarkko Sakkinen
2018-11-16 11:17 ` [PATCH v17 00/23] Intel SGX1 support Jarkko Sakkinen

Reply instructions:

You may reply publically to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20181217222047.GG12491@linux.intel.com \
    --to=sean.j.christopherson@intel.com \
    --cc=andriy.shevchenko@linux.intel.com \
    --cc=andy@infradead.org \
    --cc=bp@alien8.de \
    --cc=dave.hansen@intel.com \
    --cc=dvhart@infradead.org \
    --cc=haitao.huang@linux.intel.com \
    --cc=hpa@zytor.com \
    --cc=jarkko.sakkinen@linux.intel.com \
    --cc=kai.svahn@intel.com \
    --cc=linux-kernel@vger.kernel.org \
    --cc=linux-sgx@vger.kernel.org \
    --cc=luto@kernel.org \
    --cc=mark.shanahan@intel.com \
    --cc=mingo@redhat.com \
    --cc=nhorman@redhat.com \
    --cc=npmccallum@redhat.com \
    --cc=platform-driver-x86@vger.kernel.org \
    --cc=serge.ayoun@intel.com \
    --cc=shay.katz-zamir@intel.com \
    --cc=suresh.b.siddha@intel.com \
    --cc=tglx@linutronix.de \
    --cc=x86@kernel.org \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link

Linux-Sgx Archive on lore.kernel.org

Archives are clonable:
	git clone --mirror https://lore.kernel.org/linux-sgx/0 linux-sgx/git/0.git

	# If you have public-inbox 1.1+ installed, you may
	# initialize and index your mirror using the following commands:
	public-inbox-init -V2 linux-sgx linux-sgx/ https://lore.kernel.org/linux-sgx \
		linux-sgx@vger.kernel.org linux-sgx@archiver.kernel.org
	public-inbox-index linux-sgx


Newsgroup available over NNTP:
	nntp://nntp.lore.kernel.org/org.kernel.vger.linux-sgx


AGPL code for this site: git clone https://public-inbox.org/ public-inbox