From: Andy Lutomirski <email@example.com> To: Sean Christopherson <firstname.lastname@example.org> Cc: Jethro Beekman <email@example.com>, Andy Lutomirski <firstname.lastname@example.org>, Jarkko Sakkinen <email@example.com>, "firstname.lastname@example.org" <email@example.com>, Dave Hansen <firstname.lastname@example.org>, Cedric Xing <email@example.com>, "Dr . Greg Wettstein" <firstname.lastname@example.org> Subject: Re: [PATCH 5/7] x86/sgx: Add flag to zero added region instead of copying from source Date: Thu, 13 Jun 2019 09:16:46 -0700 Message-ID: <CALCETrXmB=fFQSUdj0Zt8yRP7nbat=m3oyvhbDxQDCR52Gbw5A@mail.gmail.com> (raw) In-Reply-To: <20190613134603.GA5850@linux.intel.com> On Thu, Jun 13, 2019 at 6:46 AM Sean Christopherson <email@example.com> wrote: > > On Thu, Jun 13, 2019 at 12:38:02AM +0000, Jethro Beekman wrote: > > On 2019-06-10 11:53, Sean Christopherson wrote: > > >On Fri, Jun 07, 2019 at 12:32:23PM -0700, Andy Lutomirski wrote: > > >> > > >>>On Jun 6, 2019, at 10:32 AM, Sean Christopherson <firstname.lastname@example.org> wrote: > > >>> > > >>>>On Thu, Jun 06, 2019 at 10:20:38AM -0700, Andy Lutomirski wrote: > > >>>>On Wed, Jun 5, 2019 at 12:49 PM Sean Christopherson > > >>>><email@example.com> wrote: > > >>>>> > > >>>>>For some enclaves, e.g. an enclave with a small code footprint and a > > >>>>>large working set, the vast majority of pages added to the enclave are > > >>>>>zero pages. Introduce a flag to denote such zero pages. The major > > >>>>>benefit of the flag will be realized in a future patch to use Linux's > > >>>>>actual zero page as the source, as opposed to explicitly zeroing the > > >>>>>enclave's backing memory. > > >>>>> > > >>>> > > >>>>I feel like I probably asked this at some point, but why is there a > > >>>>workqueue here at all? > > >>> > > >>>Performance. A while back I wrote a patch set to remove the worker queue > > >>>and discovered that it tanks enclave build time when the enclave is being > > >>>hosted by a Golang application. Here's a snippet from a mail discussing > > >>>the code. > > >>> > > >>> The bad news is that I don't think we can remove the add page worker > > >>> as applications with userspace schedulers, e.g. Go's M:N scheduler, > > >>> can see a 10x or more throughput improvement when using the worker > > >>> queue. I did a bit of digging for the Golang case to make sure I > > >>> wasn't doing something horribly stupid/naive and found that it's a > > >>> generic issue in Golang with blocking (or just long-running) system > > >>> calls. Because Golang multiplexes Goroutines on top of OS threads, > > >>> blocking syscalls introduce latency and context switching overhead, > > >>> e.g. Go's scheduler will spin up a new OS thread to service other > > >>> Goroutines after it realizes the syscall has blocked, and will later > > >>> destroy one of the OS threads so that it doesn't build up too many > > >>> unused. > > >>> > > >>>IIRC, the scenario is spinning up several goroutines, each building an > > >>>enclave. I played around with adding a flag to do a synchronous EADD > > >>>but didn't see a meaningful change in performance for the simple case. > > >>>Supporting both the worker queue and direct paths was complex enough > > >>>that I decided it wasn't worth the trouble for initial upstreaming. > > >> > > >>Sigh. > > >> > > >>It seems silly to add a workaround for a language that has trouble calling > > >>somewhat-but-not-too-slow syscalls or ioctls. > > >> > > >>How about fixing this in Go directly? Either convince the golang people to > > >>add a way to allocate a real thread for a particular region of code or have > > >>the Go SGX folks write a bit of C code to do a whole bunch of ioctls and > > >>have Go call *that*. Then the mess stays in Go where it belongs. > > > > > >Actually, I'm pretty sure changing the ioctl() from ADD_PAGE to ADD_REGION > > >would eliminate the worst of the golang slowdown without requiring > > >userspace to get super fancy. I'm in favor of eliminating the work queue, > > >especially if the UAPI is changed to allow adding multiple pages in a > > >single syscall. > > > > > > > I don't know if this is going to matter a whole lot, but have you considered > > the performance impact of needing to the EPC paging while doing the EADD > > ioctl and how this interacts with having a workqueue? > > Yep, other than the goroutine case, eliminating the workqueue doesn't > substantially affect performance in either direction, regardless of the > pressure on the EPC. It should get rid of some extra copies and allocations, no?
next prev parent reply index Thread overview: 33+ messages / expand[flat|nested] mbox.gz Atom feed top 2019-06-05 19:48 [PATCH 0/7] x86/sgx: Clean up and enhance add pages ioctl Sean Christopherson 2019-06-05 19:48 ` [PATCH 1/7] x86/sgx: Remove dead code to handle non-existent IOR ioctl Sean Christopherson 2019-06-05 19:48 ` [PATCH 2/7] x86/sgx: Remove unnecessary @cmd parameter from ioctl helpers Sean Christopherson 2019-06-05 19:48 ` [PATCH 3/7] x86/sgx: Let ioctl helpers do copy to/from user Sean Christopherson 2019-06-05 19:48 ` [PATCH 4/7] x86/sgx: Allow userspace to add multiple pages in single ioctl() Sean Christopherson 2019-06-06 15:47 ` Jarkko Sakkinen 2019-06-13 0:43 ` Jethro Beekman 2019-06-13 16:51 ` Sean Christopherson 2019-06-13 19:05 ` Andy Lutomirski 2019-06-13 19:15 ` Sean Christopherson 2019-06-13 19:45 ` Xing, Cedric 2019-06-05 19:48 ` [PATCH 5/7] x86/sgx: Add flag to zero added region instead of copying from source Sean Christopherson 2019-06-06 17:20 ` Andy Lutomirski 2019-06-06 17:32 ` Sean Christopherson 2019-06-07 19:32 ` Andy Lutomirski 2019-06-10 17:06 ` Jarkko Sakkinen 2019-06-10 18:09 ` Xing, Cedric 2019-06-10 18:41 ` Sean Christopherson 2019-06-10 18:53 ` Sean Christopherson 2019-06-13 0:38 ` Jethro Beekman 2019-06-13 13:46 ` Sean Christopherson 2019-06-13 16:16 ` Andy Lutomirski [this message] 2019-06-13 16:54 ` Sean Christopherson 2019-06-05 19:48 ` [PATCH 6/7] x86/sgx: Use the actual zero page as the source when adding zero pages Sean Christopherson 2019-06-05 19:48 ` [PATCH 7/7] x86/sgx: Add a reserved field to sgx_enclave_add_region to drop 'packed' Sean Christopherson 2019-06-05 19:59 ` Dave Hansen 2019-06-05 20:00 ` Andy Lutomirski 2019-06-12 15:14 ` Jarkko Sakkinen 2019-06-12 15:23 ` Sean Christopherson 2019-06-13 0:44 ` Jethro Beekman 2019-06-13 15:38 ` Jarkko Sakkinen 2019-06-12 15:16 ` [PATCH 0/7] x86/sgx: Clean up and enhance add pages ioctl Jarkko Sakkinen 2019-06-12 18:14 ` Jarkko Sakkinen
Reply instructions: You may reply publically to this message via plain-text email using any one of the following methods: * Save the following mbox file, import it into your mail client, and reply-to-all from there: mbox Avoid top-posting and favor interleaved quoting: https://en.wikipedia.org/wiki/Posting_style#Interleaved_style * Reply using the --to, --cc, and --in-reply-to switches of git-send-email(1): git send-email \ --in-reply-to='CALCETrXmB=fFQSUdj0Zt8yRP7nbat=m3oyvhbDxQDCR52Gbw5A@mail.gmail.com' \ --firstname.lastname@example.org \ --email@example.com \ --firstname.lastname@example.org \ --email@example.com \ --firstname.lastname@example.org \ --email@example.com \ --firstname.lastname@example.org \ --email@example.com \ /path/to/YOUR_REPLY https://kernel.org/pub/software/scm/git/docs/git-send-email.html * If your mail client supports setting the In-Reply-To header via mailto: links, try the mailto: link
Linux-Sgx Archive on lore.kernel.org Archives are clonable: git clone --mirror https://lore.kernel.org/linux-sgx/0 linux-sgx/git/0.git # If you have public-inbox 1.1+ installed, you may # initialize and index your mirror using the following commands: public-inbox-init -V2 linux-sgx linux-sgx/ https://lore.kernel.org/linux-sgx \ firstname.lastname@example.org email@example.com public-inbox-index linux-sgx Example config snippet for mirrors Newsgroup available over NNTP: nntp://nntp.lore.kernel.org/org.kernel.vger.linux-sgx AGPL code for this site: git clone https://public-inbox.org/ public-inbox