From: Sean Christopherson <sean.j.christopherson@intel.com> To: Jann Horn <jannh@google.com> Cc: Andy Lutomirski <luto@kernel.org>, Dave Hansen <dave.hansen@intel.com>, Linus Torvalds <torvalds@linux-foundation.org>, <dalias@libc.org>, "Dave Hansen" <dave.hansen@linux.intel.com>, <jethro@fortanix.com>, <jarkko.sakkinen@linux.intel.com>, Florian Weimer <fweimer@redhat.com>, Linux API <linux-api@vger.kernel.org>, the arch/x86 maintainers <x86@kernel.org>, linux-arch <linux-arch@vger.kernel.org>, kernel list <linux-kernel@vger.kernel.org>, Peter Zijlstra <peterz@infradead.org>, <nhorman@redhat.com>, <npmccallum@redhat.com>, <serge.ayoun@intel.com>, <shay.katz-zamir@intel.com>, <linux-sgx@vger.kernel.org>, <andriy.shevchenko@linux.intel.com>, Thomas Gleixner <tglx@linutronix.de>, Ingo Molnar <mingo@redhat.com>, Borislav Petkov <bp@alien8.de>, <carlos@redhat.com>, <adhemerval.zanella@linaro.org> Subject: Re: RFC: userspace exception fixups Date: Fri, 2 Nov 2018 15:04:37 -0700 [thread overview] Message-ID: <20181102220437.GI7393@linux.intel.com> (raw) In-Reply-To: <CAG48ez0x7ZPcKvLt01WS-VrLm59WfvYE3nE1xbnyn3Qsp5T2rA@mail.gmail.com> On Fri, Nov 02, 2018 at 08:02:23PM +0100, Jann Horn wrote: > On Fri, Nov 2, 2018 at 7:27 PM Sean Christopherson > <sean.j.christopherson@intel.com> wrote: > > On Fri, Nov 02, 2018 at 10:48:38AM -0700, Andy Lutomirski wrote: > > > This whole mechanism seems very complicated, and it's not clear > > > exactly what behavior user code wants. > > > > No argument there. That's why I like the approach of dumping the > > exception to userspace without trying to do anything intelligent in > > the kernel. Userspace can then do whatever it wants AND we don't > > have to worry about mucking with stacks. > > > > One of the hiccups with the VDSO approach is that the enclave may > > want to use the untrusted stack, i.e. the stack that has the VDSO's > > stack frame. For example, Intel's SDK uses the untrusted stack to > > pass parameters for EEXIT, which means an AEX might occur with what > > is effectively a bad stack from the VDSO's perspective. > > What exactly does "uses the untrusted stack to pass parameters for > EEXIT" mean? I guess you're saying that the enclave is writing to > RSP+[0...some_positive_offset], and the written data needs to be > visible to the code outside the enclave afterwards? As is, they actually do it the other way around, i.e. negative offsets relative to the untrusted %RSP. Going into the enclave there is no reserved space on the stack. The SDK uses EEXIT like a function call, i.e. pushing parameters on the stack and making an call outside of the enclave, hence the name out-call. This allows the SDK to handle any reasonable out-call without a priori knowledge of the application's maximum out-call "size". Rough outline of what happens in a non-faulting case. 1: Userspace executes EENTER -------------------- | userspace stack | -------------------- <-- %RSP at EENTER 2: Enclave does EEXIT to invoke out-call function -------------------- | userspace stack | -------------------- <-- %RSP at EENTER | out-call func ID | | param1 | | ... | | paramN | -------------------- <-- %RSP at EEXIT 3: Userspace re-EENTERs enclave after handling EEXIT request -------------------- | userspace stack | -------------------- <-- %RSP at original EENTER | out-call func ID | | param1 | | ... | | paramN | -------------------- <-- %RSP at post-EEXIT EENTER 4: Enclave cleans up the stack -------------------- | userspace stack | -------------------- <-- %RSP back at original EENTER In the faulting case, an AEX can occur while the enclave is pushing parameters onto the stack for EEXIT. 1: Userspace executes EENTER -------------------- | userspace stack | -------------------- <-- %RSP at EENTER 2: AEX occurs during enclave prep for EEXIT -------------------- | userspace stack | -------------------- <-- %RSP at EENTER | out-call func ID | | param1 | | ... | -------------------- <-- %RSP at AEX 3: Userspace re-EENTERs enclave to invoke enclave fault handler -------------------- | userspace stack | -------------------- <-- %RSP at original EENTER | out-call func ID | | param1 | | ... | -------------------- <-- %RSP at AEX | userspace stack | -------------------- <-- %RSP at EENTER to fault handler 4: Enclave handles the fault, EEXITs back to userspace -------------------- | userspace stack | -------------------- <-- %RSP at original EENTER | out-call func ID | | param1 | | ... | -------------------- <-- %RSP at AEX | userspace stack | -------------------- <-- %RSP at EEXIT from fault handler 5: Userspace pops its stack and ERESUMEs back to the enclave -------------------- | userspace stack | -------------------- <-- %RSP at original EENTER | out-call func ID | | param1 | | ... | -------------------- <-- %RSP at ERESUME 6: Enclave finishes its EEXIT to invoke out-call function -------------------- | userspace stuff | -------------------- <-- %RSP at original EENTER | out-call func ID | | param1 | | ... | | paramN | -------------------- <-- %RSP at EEXIT > In other words, the vDSO helper would have to not touch the stack > pointer (only using the 128-byte redzone to store spilled data, at > least across the enclave entry), and return by decrementing the stack > pointer by 8 immediately before returning (storing the return pointer > in the redzone)? > > So you'd call the vDSO helper with a normal "call > vdso_helper_address", then the vDSO helper does "add rsp, 8", then the > vDSO helper does its magic, and then it returns with "sub rsp, 8" and > "ret"? That way you don't touch anything on the high-address side of > RSP while still avoiding running into CET problems. (I'm assuming that > you can use CET in a process that is hosting SGX enclaves?)
WARNING: multiple messages have this Message-ID (diff)
From: Sean Christopherson <sean.j.christopherson@intel.com> To: Jann Horn <jannh@google.com> Cc: Andy Lutomirski <luto@kernel.org>, Dave Hansen <dave.hansen@intel.com>, Linus Torvalds <torvalds@linux-foundation.org>, dalias@libc.org, Dave Hansen <dave.hansen@linux.intel.com>, jethro@fortanix.com, jarkko.sakkinen@linux.intel.com, Florian Weimer <fweimer@redhat.com>, Linux API <linux-api@vger.kernel.org>, the arch/x86 maintainers <x86@kernel.org>, linux-arch <linux-arch@vger.kernel.org>, kernel list <linux-kernel@vger.kernel.org>, Peter Zijlstra <peterz@infradead.org>, nhorman@redhat.com, npmccallum@redhat.com, serge.ayoun@intel.com, shay.katz-zamir@intel.com, linux-sgx@vger.kernel.org, andriy.shevchenko@linux.intel.com, Thomas Gleixner <tglx@linutronix.de>, Ingo Molnar <mingo@redhat.com>, Borislav Petkov <bp@alien8.de>, carlos@redhat.com, adhemerval.zanella@linaro.org Subject: Re: RFC: userspace exception fixups Date: Fri, 2 Nov 2018 15:04:37 -0700 [thread overview] Message-ID: <20181102220437.GI7393@linux.intel.com> (raw) Message-ID: <20181102220437.cMk3cjkxlidycIjLOkgDSnqL9rhgWFJj64pXTyn8gAE@z> (raw) In-Reply-To: <CAG48ez0x7ZPcKvLt01WS-VrLm59WfvYE3nE1xbnyn3Qsp5T2rA@mail.gmail.com> On Fri, Nov 02, 2018 at 08:02:23PM +0100, Jann Horn wrote: > On Fri, Nov 2, 2018 at 7:27 PM Sean Christopherson > <sean.j.christopherson@intel.com> wrote: > > On Fri, Nov 02, 2018 at 10:48:38AM -0700, Andy Lutomirski wrote: > > > This whole mechanism seems very complicated, and it's not clear > > > exactly what behavior user code wants. > > > > No argument there. That's why I like the approach of dumping the > > exception to userspace without trying to do anything intelligent in > > the kernel. Userspace can then do whatever it wants AND we don't > > have to worry about mucking with stacks. > > > > One of the hiccups with the VDSO approach is that the enclave may > > want to use the untrusted stack, i.e. the stack that has the VDSO's > > stack frame. For example, Intel's SDK uses the untrusted stack to > > pass parameters for EEXIT, which means an AEX might occur with what > > is effectively a bad stack from the VDSO's perspective. > > What exactly does "uses the untrusted stack to pass parameters for > EEXIT" mean? I guess you're saying that the enclave is writing to > RSP+[0...some_positive_offset], and the written data needs to be > visible to the code outside the enclave afterwards? As is, they actually do it the other way around, i.e. negative offsets relative to the untrusted %RSP. Going into the enclave there is no reserved space on the stack. The SDK uses EEXIT like a function call, i.e. pushing parameters on the stack and making an call outside of the enclave, hence the name out-call. This allows the SDK to handle any reasonable out-call without a priori knowledge of the application's maximum out-call "size". Rough outline of what happens in a non-faulting case. 1: Userspace executes EENTER -------------------- | userspace stack | -------------------- <-- %RSP at EENTER 2: Enclave does EEXIT to invoke out-call function -------------------- | userspace stack | -------------------- <-- %RSP at EENTER | out-call func ID | | param1 | | ... | | paramN | -------------------- <-- %RSP at EEXIT 3: Userspace re-EENTERs enclave after handling EEXIT request -------------------- | userspace stack | -------------------- <-- %RSP at original EENTER | out-call func ID | | param1 | | ... | | paramN | -------------------- <-- %RSP at post-EEXIT EENTER 4: Enclave cleans up the stack -------------------- | userspace stack | -------------------- <-- %RSP back at original EENTER In the faulting case, an AEX can occur while the enclave is pushing parameters onto the stack for EEXIT. 1: Userspace executes EENTER -------------------- | userspace stack | -------------------- <-- %RSP at EENTER 2: AEX occurs during enclave prep for EEXIT -------------------- | userspace stack | -------------------- <-- %RSP at EENTER | out-call func ID | | param1 | | ... | -------------------- <-- %RSP at AEX 3: Userspace re-EENTERs enclave to invoke enclave fault handler -------------------- | userspace stack | -------------------- <-- %RSP at original EENTER | out-call func ID | | param1 | | ... | -------------------- <-- %RSP at AEX | userspace stack | -------------------- <-- %RSP at EENTER to fault handler 4: Enclave handles the fault, EEXITs back to userspace -------------------- | userspace stack | -------------------- <-- %RSP at original EENTER | out-call func ID | | param1 | | ... | -------------------- <-- %RSP at AEX | userspace stack | -------------------- <-- %RSP at EEXIT from fault handler 5: Userspace pops its stack and ERESUMEs back to the enclave -------------------- | userspace stack | -------------------- <-- %RSP at original EENTER | out-call func ID | | param1 | | ... | -------------------- <-- %RSP at ERESUME 6: Enclave finishes its EEXIT to invoke out-call function -------------------- | userspace stuff | -------------------- <-- %RSP at original EENTER | out-call func ID | | param1 | | ... | | paramN | -------------------- <-- %RSP at EEXIT > In other words, the vDSO helper would have to not touch the stack > pointer (only using the 128-byte redzone to store spilled data, at > least across the enclave entry), and return by decrementing the stack > pointer by 8 immediately before returning (storing the return pointer > in the redzone)? > > So you'd call the vDSO helper with a normal "call > vdso_helper_address", then the vDSO helper does "add rsp, 8", then the > vDSO helper does its magic, and then it returns with "sub rsp, 8" and > "ret"? That way you don't touch anything on the high-address side of > RSP while still avoiding running into CET problems. (I'm assuming that > you can use CET in a process that is hosting SGX enclaves?)
next prev parent reply other threads:[~2018-11-03 7:13 UTC|newest] Thread overview: 163+ messages / expand[flat|nested] mbox.gz Atom feed top 2018-11-01 17:53 RFC: userspace exception fixups Andy Lutomirski 2018-11-01 17:53 ` Andy Lutomirski 2018-11-01 18:09 ` Florian Weimer 2018-11-01 18:09 ` Florian Weimer 2018-11-01 18:30 ` Rich Felker 2018-11-01 18:30 ` Rich Felker 2018-11-01 19:00 ` Jarkko Sakkinen 2018-11-01 19:00 ` Jarkko Sakkinen 2018-11-01 18:27 ` Rich Felker 2018-11-01 18:27 ` Rich Felker 2018-11-01 18:33 ` Jann Horn 2018-11-01 18:33 ` Jann Horn 2018-11-01 18:52 ` Rich Felker 2018-11-01 18:52 ` Rich Felker 2018-11-01 19:10 ` Linus Torvalds 2018-11-01 19:10 ` Linus Torvalds 2018-11-01 19:31 ` Rich Felker 2018-11-01 19:31 ` Rich Felker 2018-11-01 21:24 ` Linus Torvalds 2018-11-01 21:24 ` Linus Torvalds 2018-11-01 23:22 ` Andy Lutomirski 2018-11-01 23:22 ` Andy Lutomirski 2018-11-02 16:30 ` Sean Christopherson 2018-11-02 16:30 ` Sean Christopherson 2018-11-02 16:37 ` Jethro Beekman 2018-11-02 16:37 ` Jethro Beekman 2018-11-02 16:52 ` Sean Christopherson 2018-11-02 16:52 ` Sean Christopherson 2018-11-02 16:56 ` Jethro Beekman 2018-11-02 16:56 ` Jethro Beekman 2018-11-02 17:01 ` Andy Lutomirski 2018-11-02 17:01 ` Andy Lutomirski 2018-11-02 17:05 ` Jethro Beekman 2018-11-02 17:05 ` Jethro Beekman 2018-11-02 17:16 ` Andy Lutomirski 2018-11-02 17:16 ` Andy Lutomirski 2018-11-02 17:32 ` Rich Felker 2018-11-02 17:32 ` Rich Felker 2018-11-02 17:12 ` Sean Christopherson 2018-11-02 17:12 ` Sean Christopherson 2018-11-02 22:42 ` Jarkko Sakkinen 2018-11-02 22:42 ` Jarkko Sakkinen 2018-11-02 16:56 ` Dave Hansen 2018-11-02 16:56 ` Dave Hansen 2018-11-02 17:06 ` Sean Christopherson 2018-11-02 17:06 ` Sean Christopherson 2018-11-02 17:13 ` Dave Hansen 2018-11-02 17:13 ` Dave Hansen 2018-11-02 17:33 ` Sean Christopherson 2018-11-02 17:33 ` Sean Christopherson 2018-11-02 17:48 ` Andy Lutomirski 2018-11-02 17:48 ` Andy Lutomirski 2018-11-02 18:27 ` Sean Christopherson 2018-11-02 18:27 ` Sean Christopherson 2018-11-02 19:02 ` Jann Horn 2018-11-02 19:02 ` Jann Horn 2018-11-02 22:04 ` Sean Christopherson [this message] 2018-11-02 22:04 ` Sean Christopherson 2018-11-02 23:27 ` Jann Horn 2018-11-02 23:27 ` Jann Horn 2018-11-02 23:32 ` Andy Lutomirski 2018-11-02 23:32 ` Andy Lutomirski 2018-11-02 23:36 ` Jann Horn 2018-11-02 23:36 ` Jann Horn 2018-11-06 15:37 ` Sean Christopherson 2018-11-06 15:37 ` Sean Christopherson 2018-11-06 16:57 ` Andy Lutomirski 2018-11-06 16:57 ` Andy Lutomirski 2018-11-06 17:03 ` Dave Hansen 2018-11-06 17:03 ` Dave Hansen 2018-11-06 17:19 ` Sean Christopherson 2018-11-06 17:19 ` Sean Christopherson 2018-11-06 18:20 ` Andy Lutomirski 2018-11-06 18:20 ` Andy Lutomirski 2018-11-06 18:41 ` Dave Hansen 2018-11-06 18:41 ` Dave Hansen 2018-11-06 19:02 ` Andy Lutomirski 2018-11-06 19:02 ` Andy Lutomirski 2018-11-06 19:22 ` Dave Hansen 2018-11-06 19:22 ` Dave Hansen 2018-11-06 20:12 ` Andy Lutomirski 2018-11-06 20:12 ` Andy Lutomirski 2018-11-06 21:00 ` Dave Hansen 2018-11-06 21:00 ` Dave Hansen 2018-11-06 21:07 ` Andy Lutomirski 2018-11-06 21:07 ` Andy Lutomirski 2018-11-06 21:41 ` Andy Lutomirski 2018-11-06 21:41 ` Andy Lutomirski 2018-11-06 21:59 ` Sean Christopherson 2018-11-06 21:59 ` Sean Christopherson 2018-11-06 23:00 ` Andy Lutomirski 2018-11-06 23:00 ` Andy Lutomirski 2018-11-06 23:35 ` Sean Christopherson 2018-11-06 23:35 ` Sean Christopherson 2018-11-06 23:39 ` Andy Lutomirski 2018-11-06 23:39 ` Andy Lutomirski 2018-11-07 0:02 ` Sean Christopherson 2018-11-07 0:02 ` Sean Christopherson 2018-11-07 1:17 ` Andy Lutomirski 2018-11-07 1:17 ` Andy Lutomirski 2018-11-07 6:47 ` Jethro Beekman 2018-11-07 6:47 ` Jethro Beekman 2018-11-07 15:34 ` Sean Christopherson 2018-11-07 15:34 ` Sean Christopherson 2018-11-07 19:01 ` Sean Christopherson 2018-11-07 19:01 ` Sean Christopherson 2018-11-07 20:56 ` Dave Hansen 2018-11-07 20:56 ` Dave Hansen 2018-11-08 15:04 ` Jarkko Sakkinen 2018-11-08 15:04 ` Jarkko Sakkinen 2018-11-08 19:54 ` Sean Christopherson 2018-11-08 19:54 ` Sean Christopherson 2018-11-08 20:05 ` Andy Lutomirski 2018-11-08 20:05 ` Andy Lutomirski 2018-11-08 20:10 ` Dave Hansen 2018-11-08 20:10 ` Dave Hansen 2018-11-08 21:16 ` Sean Christopherson 2018-11-08 21:16 ` Sean Christopherson 2018-11-08 21:50 ` Dave Hansen 2018-11-08 21:50 ` Dave Hansen 2018-11-08 22:04 ` Sean Christopherson 2018-11-08 22:04 ` Sean Christopherson 2018-11-09 7:12 ` Christoph Hellwig 2018-11-09 7:12 ` Christoph Hellwig 2018-11-06 23:17 ` Rich Felker 2018-11-06 23:17 ` Rich Felker 2018-11-06 23:26 ` Sean Christopherson 2018-11-06 23:26 ` Sean Christopherson 2018-11-07 21:27 ` Rich Felker 2018-11-07 21:27 ` Rich Felker 2018-11-07 21:33 ` Andy Lutomirski 2018-11-07 21:33 ` Andy Lutomirski 2018-11-07 21:40 ` Sean Christopherson 2018-11-07 21:40 ` Sean Christopherson 2018-11-08 15:11 ` Jarkko Sakkinen 2018-11-08 15:11 ` Jarkko Sakkinen 2018-11-06 17:00 ` Dave Hansen 2018-11-06 17:00 ` Dave Hansen 2018-11-02 22:37 ` Jarkko Sakkinen 2018-11-02 22:37 ` Jarkko Sakkinen 2018-11-01 19:06 ` Linus Torvalds 2018-11-01 19:06 ` Linus Torvalds 2018-11-02 22:07 ` Jarkko Sakkinen 2018-11-02 22:07 ` Jarkko Sakkinen 2018-11-18 7:15 ` Jarkko Sakkinen 2018-11-18 7:18 ` Jarkko Sakkinen 2018-11-18 13:02 ` Jarkko Sakkinen 2018-11-19 5:17 ` Jethro Beekman 2018-11-19 14:05 ` Jarkko Sakkinen 2018-11-19 14:59 ` Jarkko Sakkinen 2018-11-19 15:29 ` Andy Lutomirski 2018-11-19 16:02 ` Jarkko Sakkinen 2018-11-19 17:00 ` Andy Lutomirski 2018-11-20 10:11 ` Jarkko Sakkinen 2018-11-20 15:19 ` Andy Lutomirski 2018-11-20 22:55 ` Jarkko Sakkinen 2018-11-21 5:17 ` Jethro Beekman 2018-11-21 15:17 ` Jarkko Sakkinen 2018-11-24 17:07 ` Jarkko Sakkinen 2018-11-26 14:35 ` Sean Christopherson 2018-11-26 22:06 ` Jarkko Sakkinen 2018-11-20 18:09 ` Sean Christopherson 2018-11-20 22:46 ` Jarkko Sakkinen
Reply instructions: You may reply publicly to this message via plain-text email using any one of the following methods: * Save the following mbox file, import it into your mail client, and reply-to-all from there: mbox Avoid top-posting and favor interleaved quoting: https://en.wikipedia.org/wiki/Posting_style#Interleaved_style * Reply using the --to, --cc, and --in-reply-to switches of git-send-email(1): git send-email \ --in-reply-to=20181102220437.GI7393@linux.intel.com \ --to=sean.j.christopherson@intel.com \ --cc=adhemerval.zanella@linaro.org \ --cc=andriy.shevchenko@linux.intel.com \ --cc=bp@alien8.de \ --cc=carlos@redhat.com \ --cc=dalias@libc.org \ --cc=dave.hansen@intel.com \ --cc=dave.hansen@linux.intel.com \ --cc=fweimer@redhat.com \ --cc=jannh@google.com \ --cc=jarkko.sakkinen@linux.intel.com \ --cc=jethro@fortanix.com \ --cc=linux-api@vger.kernel.org \ --cc=linux-arch@vger.kernel.org \ --cc=linux-kernel@vger.kernel.org \ --cc=linux-sgx@vger.kernel.org \ --cc=luto@kernel.org \ --cc=mingo@redhat.com \ --cc=nhorman@redhat.com \ --cc=npmccallum@redhat.com \ --cc=peterz@infradead.org \ --cc=serge.ayoun@intel.com \ --cc=shay.katz-zamir@intel.com \ --cc=tglx@linutronix.de \ --cc=torvalds@linux-foundation.org \ --cc=x86@kernel.org \ /path/to/YOUR_REPLY https://kernel.org/pub/software/scm/git/docs/git-send-email.html * If your mail client supports setting the In-Reply-To header via mailto: links, try the mailto: linkBe sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s).