Linux-Sgx Archive on lore.kernel.org
 help / color / Atom feed
From: Jarkko Sakkinen <jarkko.sakkinen@linux.intel.com>
To: Andy Lutomirski <luto@amacapital.net>
Cc: X86 ML <x86@kernel.org>,
	linux-sgx@vger.kernel.org, LKML <linux-kernel@vger.kernel.org>,
	Sean Christopherson <sean.j.christopherson@intel.com>,
	Jethro Beekman <jethro@fortanix.com>,
	Cedric Xing <cedric.xing@intel.com>,
	Andrew Morton <akpm@linux-foundation.org>,
	Andy Shevchenko <andriy.shevchenko@linux.intel.com>,
	asapek@google.com, Borislav Petkov <bp@alien8.de>,
	chenalexchen@google.com, Conrad Parker <conradparker@google.com>,
	cyhanish@google.com, Dave Hansen <dave.hansen@intel.com>,
	"Huang, Haitao" <haitao.huang@intel.com>,
	"Huang, Kai" <kai.huang@intel.com>,
	"Svahn, Kai" <kai.svahn@intel.com>, Keith Moyer <kmoy@google.com>,
	Christian Ludloff <ludloff@google.com>,
	Andrew Lutomirski <luto@kernel.org>,
	Neil Horman <nhorman@redhat.com>,
	Nathaniel McCallum <npmccallum@redhat.com>,
	Patrick Uiterwijk <puiterwijk@redhat.com>,
	David Rientjes <rientjes@google.com>,
	Thomas Gleixner <tglx@linutronix.de>,
	yaozhangx@google.com, mikko.ylinen@intel.com
Subject: Re: [PATCH v39 21/24] x86/vdso: Implement a vDSO for Intel SGX enclave call
Date: Sun, 18 Oct 2020 00:02:04 +0300
Message-ID: <20201017210204.GA64894@linux.intel.com> (raw)
In-Reply-To: <CALCETrUJG4LLk4SkBdufUYAx=ec=NyY2wv=Js=72=am6nFQfiw@mail.gmail.com>

On Fri, Oct 16, 2020 at 06:48:53PM -0700, Andy Lutomirski wrote:
> On Fri, Oct 2, 2020 at 9:51 PM Jarkko Sakkinen
> <jarkko.sakkinen@linux.intel.com> wrote:
> >
> > From: Sean Christopherson <sean.j.christopherson@intel.com>
> >
> > An SGX runtime must be aware of the exceptions, which happen inside an
> > enclave. Introduce a vDSO call that wraps EENTER/ERESUME cycle and returns
> > the CPU exception back to the caller exactly when it happens.
> >
> > Kernel fixups the exception information to RDI, RSI and RDX. The SGX call
> > vDSO handler fills this information to the user provided buffer or
> > alternatively trigger user provided callback at the time of the exception.
> >
> > The calling convention supports providing the parameters in standard RDI
> > RSI, RDX, RCX, R8 and R9 registers, i.e. it is possible to declare the vDSO
> > as a C prototype, but other than that there is no specific support for
> > SystemV ABI. Storing XSAVE etc. is all responsibility of the enclave and
> > the associated run-time.
> >
> > Suggested-by: Andy Lutomirski <luto@amacapital.net>
> > Acked-by: Jethro Beekman <jethro@fortanix.com>
> > Tested-by: Jethro Beekman <jethro@fortanix.com>
> > Signed-off-by: Sean Christopherson <sean.j.christopherson@intel.com>
> > Co-developed-by: Cedric Xing <cedric.xing@intel.com>
> > Signed-off-by: Cedric Xing <cedric.xing@intel.com>
> > Co-developed-by: Jarkko Sakkinen <jarkko.sakkinen@linux.intel.com>
> > Signed-off-by: Jarkko Sakkinen <jarkko.sakkinen@linux.intel.com>
> 
> > +SYM_FUNC_START(__vdso_sgx_enter_enclave)
> > +       /* Prolog */
> > +       .cfi_startproc
> > +       push    %rbp
> > +       .cfi_adjust_cfa_offset  8
> > +       .cfi_rel_offset         %rbp, 0
> > +       mov     %rsp, %rbp
> > +       .cfi_def_cfa_register   %rbp
> > +       push    %rbx
> > +       .cfi_rel_offset         %rbx, -8
> 
> This *looks* right, but I'm not really an expert.

I did not change this from earlier versions.

> > +
> > +       mov     %ecx, %eax
> > +.Lenter_enclave:
> > +       /* EENTER <= leaf <= ERESUME */
> > +       cmp     $EENTER, %eax
> > +       jb      .Linvalid_input
> > +       cmp     $ERESUME, %eax
> > +       ja      .Linvalid_input
> > +
> > +       mov     SGX_ENCLAVE_OFFSET_OF_RUN(%rbp), %rcx
> > +
> > +       /* Validate that the reserved area contains only zeros. */
> > +       push    %rax
> > +       push    %rbx
> 
> This could use a .cfi_register_something_or_other for rbx

Sean pointed out that saving %rbx is not necessary here:

https://lore.kernel.org/linux-sgx/20201006025703.GG15803@linux.intel.com/

> > +       mov     $SGX_ENCLAVE_RUN_RESERVED_START, %rbx
> > +1:
> > +       mov     (%rcx, %rbx), %rax
> > +       cmpq    $0, %rax
> > +       jne     .Linvalid_input
> > +
> > +       add     $8, %rbx
> > +       cmpq    $SGX_ENCLAVE_RUN_RESERVED_END, %rbx
> > +       jne     1b
> > +       pop     %rbx
> 
> This should undo it.

Given private feedback from Sean, I'm replacing this with:

        mov     $SGX_ENCLAVE_RUN_RESERVED_START, %rbx
 1:
        cmpq    $0, (%rcx, %rbx)
        jne     .Linvalid_input

        add     $8, %rbx
        cmpq    $SGX_ENCLAVE_RUN_RESERVED_END, %rbx
        jne     1b

There was bug in the error path, %rax was not popped. I did negative
testing (testing both branches) for this but it went clean.

I guess if I fix this, that will deal with all of your comments?

> > +       pop     %rax
> > +
> > +       /* Load TCS and AEP */
> > +       mov     SGX_ENCLAVE_RUN_TCS(%rcx), %rbx
> > +       lea     .Lasync_exit_pointer(%rip), %rcx
> > +
> > +       /* Single ENCLU serving as both EENTER and AEP (ERESUME) */
> > +.Lasync_exit_pointer:
> > +.Lenclu_eenter_eresume:
> > +       enclu
> > +
> > +       /* EEXIT jumps here unless the enclave is doing something fancy. */
> > +       mov     SGX_ENCLAVE_OFFSET_OF_RUN(%rbp), %rbx
> > +
> > +       /* Set exit_reason. */
> > +       movl    $EEXIT, SGX_ENCLAVE_RUN_LEAF(%rbx)
> > +
> > +       /* Invoke userspace's exit handler if one was provided. */
> > +.Lhandle_exit:
> > +       cmpq    $0, SGX_ENCLAVE_RUN_USER_HANDLER(%rbx)
> > +       jne     .Linvoke_userspace_handler
> > +
> > +       /* Success, in the sense that ENCLU was attempted. */
> > +       xor     %eax, %eax
> > +
> > +.Lout:
> > +       pop     %rbx
> 
> and this should undo the .cfi_register.
> 
> > +       leave
> > +       .cfi_def_cfa            %rsp, 8
> > +       ret
> > +
> > +       /* The out-of-line code runs with the pre-leave stack frame. */
> > +       .cfi_def_cfa            %rbp, 16
> > +
> > +.Linvalid_input:
> 
> Here rbx and rax are pushed, and I guess pop rbx and leave fixes that
> up, so okay.
> 
> > +       mov     $(-EINVAL), %eax
> > +       jmp     .Lout
> > +
> > +.Lhandle_exception:
> > +       mov     SGX_ENCLAVE_OFFSET_OF_RUN(%rbp), %rbx
> > +
> > +       /* Set the exception info. */
> > +       mov     %eax, (SGX_ENCLAVE_RUN_LEAF)(%rbx)
> > +       mov     %di,  (SGX_ENCLAVE_RUN_EXCEPTION_VECTOR)(%rbx)
> > +       mov     %si,  (SGX_ENCLAVE_RUN_EXCEPTION_ERROR_CODE)(%rbx)
> > +       mov     %rdx, (SGX_ENCLAVE_RUN_EXCEPTION_ADDR)(%rbx)
> > +       jmp     .Lhandle_exit
> > +
> > +.Linvoke_userspace_handler:
> > +       /* Pass the untrusted RSP (at exit) to the callback via %rcx. */
> > +       mov     %rsp, %rcx
> > +
> > +       /* Save struct sgx_enclave_exception %rbx is about to be clobbered. */
> > +       mov     %rbx, %rax
> > +
> > +       /* Save the untrusted RSP offset in %rbx (non-volatile register). */
> > +       mov     %rsp, %rbx
> > +       and     $0xf, %rbx
> > +
> > +       /*
> > +        * Align stack per x86_64 ABI. Note, %rsp needs to be 16-byte aligned
> > +        * _after_ pushing the parameters on the stack, hence the bonus push.
> > +        */
> > +       and     $-0x10, %rsp
> > +       push    %rax
> > +
> > +       /* Push struct sgx_enclave_exception as a param to the callback. */
> > +       push    %rax
> > +
> > +       /* Clear RFLAGS.DF per x86_64 ABI */
> > +       cld
> > +
> > +       /*
> > +        * Load the callback pointer to %rax and lfence for LVI (load value
> > +        * injection) protection before making the call.
> > +        */
> > +       mov     SGX_ENCLAVE_RUN_USER_HANDLER(%rax), %rax
> > +       lfence
> > +       call    *%rax
> > +
> > +       /* Undo the post-exit %rsp adjustment. */
> > +       lea     0x10(%rsp, %rbx), %rsp
> > +
> > +       /*
> > +        * If the return from callback is zero or negative, return immediately,
> > +        * else re-execute ENCLU with the postive return value interpreted as
> > +        * the requested ENCLU leaf.
> > +        */
> > +       cmp     $0, %eax
> > +       jle     .Lout
> > +       jmp     .Lenter_enclave
> > +
> > +       .cfi_endproc
> > +
> > +_ASM_VDSO_EXTABLE_HANDLE(.Lenclu_eenter_eresume, .Lhandle_exception)

/Jarkko

  reply index

Thread overview: 117+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2020-10-03  4:50 [PATCH v39 00/24] Intel SGX foundations Jarkko Sakkinen
2020-10-03  4:50 ` [PATCH v39 01/24] x86/cpufeatures: x86/msr: Add Intel SGX hardware bits Jarkko Sakkinen
2020-10-19 14:10   ` Dave Hansen
2020-10-19 17:49     ` Sean Christopherson
2020-10-03  4:50 ` [PATCH v39 02/24] x86/cpufeatures: x86/msr: Add Intel SGX Launch Control " Jarkko Sakkinen
2020-10-03  4:50 ` [PATCH v39 03/24] x86/mm: x86/sgx: Signal SIGSEGV with PF_SGX Jarkko Sakkinen
2020-10-03  4:50 ` [PATCH v39 04/24] x86/sgx: Add SGX microarchitectural data structures Jarkko Sakkinen
2020-10-03  4:50 ` [PATCH v39 05/24] x86/sgx: Add wrappers for ENCLS leaf functions Jarkko Sakkinen
2020-10-19 14:30   ` Dave Hansen
2020-10-19 17:38     ` Sean Christopherson
2020-10-19 17:48       ` Dave Hansen
2020-10-19 17:53         ` Sean Christopherson
2020-10-19 17:58           ` Dave Hansen
2020-10-03  4:50 ` [PATCH v39 06/24] x86/cpu/intel: Detect SGX support Jarkko Sakkinen
2020-10-03  4:50 ` [PATCH v39 07/24] x86/cpu/intel: Add nosgx kernel parameter Jarkko Sakkinen
2020-10-03  4:50 ` [PATCH v39 08/24] x86/sgx: Initialize metadata for Enclave Page Cache (EPC) sections Jarkko Sakkinen
2020-10-19  8:45   ` Jarkko Sakkinen
2020-10-19 12:39     ` Borislav Petkov
2020-10-23  9:01       ` Jarkko Sakkinen
2020-10-19 13:40     ` Dave Hansen
2020-10-23  9:03       ` Jarkko Sakkinen
2020-10-03  4:50 ` [PATCH v39 09/24] x86/sgx: Add __sgx_alloc_epc_page() and sgx_free_epc_page() Jarkko Sakkinen
2020-10-03  4:50 ` [PATCH v39 10/24] mm: Add 'mprotect' hook to struct vm_operations_struct Jarkko Sakkinen
2020-10-03  4:50 ` [PATCH v39 11/24] x86/sgx: Add SGX enclave driver Jarkko Sakkinen
2020-10-03 14:39   ` Greg KH
2020-10-04 14:32     ` Jarkko Sakkinen
2020-10-04 15:01       ` Jarkko Sakkinen
2020-10-05  9:42       ` Greg KH
2020-10-05 12:42         ` Jarkko Sakkinen
2020-10-07 18:09           ` Haitao Huang
2020-10-07 19:26             ` Greg KH
2020-10-09  6:44               ` Jarkko Sakkinen
2020-10-14 20:16                 ` Dave Hansen
2020-10-05  8:45     ` Christoph Hellwig
2020-10-05 11:42       ` Jarkko Sakkinen
2020-10-05 11:50         ` Greg KH
2020-10-05 14:23           ` Jarkko Sakkinen
2020-10-05 15:02             ` Greg KH
2020-10-05 16:40               ` Dave Hansen
2020-10-05 20:02                 ` Jarkko Sakkinen
2020-10-09  7:10     ` Pavel Machek
2020-10-09  7:21       ` Greg KH
2020-10-09  8:21         ` Pavel Machek
2020-10-03 19:54   ` Matthew Wilcox
2020-10-04 21:50     ` Jarkko Sakkinen
2020-10-04 22:02       ` Jarkko Sakkinen
2020-10-04 22:27       ` Matthew Wilcox
2020-10-04 23:41         ` Jarkko Sakkinen
2020-10-05  1:30           ` Matthew Wilcox
2020-10-05  3:06             ` Jarkko Sakkinen
2020-10-03  4:50 ` [PATCH v39 12/24] x86/sgx: Add SGX_IOC_ENCLAVE_CREATE Jarkko Sakkinen
2020-10-16 17:07   ` Dave Hansen
2020-10-18  4:26     ` Jarkko Sakkinen
2020-10-19 20:21       ` Dave Hansen
2020-10-19 20:48         ` Sean Christopherson
2020-10-03  4:50 ` [PATCH v39 13/24] x86/sgx: Add SGX_IOC_ENCLAVE_ADD_PAGES Jarkko Sakkinen
2020-10-16 21:25   ` Dave Hansen
2020-10-18  5:03     ` Jarkko Sakkinen
2020-10-19  7:03       ` Jarkko Sakkinen
2020-10-19 20:48       ` Dave Hansen
2020-10-19 21:15         ` Sean Christopherson
2020-10-19 21:44           ` Dave Hansen
2020-10-23 10:11             ` Jarkko Sakkinen
2020-10-03  4:50 ` [PATCH v39 14/24] x86/sgx: Add SGX_IOC_ENCLAVE_INIT Jarkko Sakkinen
2020-10-03  4:50 ` [PATCH v39 15/24] x86/sgx: Add SGX_IOC_ENCLAVE_PROVISION Jarkko Sakkinen
2020-10-20 15:48   ` Dave Hansen
2020-10-23 10:14     ` Jarkko Sakkinen
2020-10-20 21:19   ` Dave Hansen
2020-10-23 10:17     ` Jarkko Sakkinen
2020-10-23 14:19       ` Dave Hansen
2020-10-24 11:34         ` Jarkko Sakkinen
2020-10-24 15:47           ` Andy Lutomirski
2020-10-24 20:23             ` Jarkko Sakkinen
2020-10-27 10:38               ` Dr. Greg
2020-10-23 14:23       ` Jethro Beekman
2020-10-24 11:40         ` Jarkko Sakkinen
2020-10-03  4:50 ` [PATCH v39 16/24] x86/sgx: Add a page reclaimer Jarkko Sakkinen
2020-10-03  5:22   ` Haitao Huang
2020-10-03 13:32     ` Jarkko Sakkinen
2020-10-03 18:23       ` Haitao Huang
2020-10-04 22:39         ` Jarkko Sakkinen
2020-10-07 17:25           ` Jarkko Sakkinen
2020-10-03  4:50 ` [PATCH v39 17/24] x86/sgx: Add ptrace() support for the SGX driver Jarkko Sakkinen
2020-10-03  4:50 ` [PATCH v39 18/24] x86/vdso: Add support for exception fixup in vDSO functions Jarkko Sakkinen
2020-10-03  4:50 ` [PATCH v39 19/24] x86/fault: Add helper function to sanitize error code Jarkko Sakkinen
2020-10-03  4:50 ` [PATCH v39 20/24] x86/traps: Attempt to fixup exceptions in vDSO before signaling Jarkko Sakkinen
2020-10-03  4:50 ` [PATCH v39 21/24] x86/vdso: Implement a vDSO for Intel SGX enclave call Jarkko Sakkinen
2020-10-06  2:57   ` Sean Christopherson
2020-10-06  8:30     ` Jethro Beekman
2020-10-06 15:15       ` Sean Christopherson
2020-10-06 17:28         ` Jarkko Sakkinen
2020-10-06 23:21           ` Sean Christopherson
2020-10-07  0:22             ` Jarkko Sakkinen
2020-10-07  1:17               ` Sean Christopherson
2020-10-07  3:14                 ` Jarkko Sakkinen
2020-10-07  4:34                   ` Sean Christopherson
2020-10-07  7:39                     ` Jarkko Sakkinen
2020-10-07  8:04                       ` Jarkko Sakkinen
2020-10-07 15:25                       ` Sean Christopherson
2020-10-07 17:08                         ` Jarkko Sakkinen
2020-10-07 17:13                           ` Jarkko Sakkinen
2020-10-06 15:49       ` Jarkko Sakkinen
2020-10-06 15:36     ` Jarkko Sakkinen
2020-10-06 21:39     ` Jarkko Sakkinen
2020-10-07  0:23       ` Jarkko Sakkinen
2020-10-17  1:48   ` Andy Lutomirski
2020-10-17 21:02     ` Jarkko Sakkinen [this message]
2020-10-03  4:50 ` [PATCH v39 22/24] selftests/x86: Add a selftest for SGX Jarkko Sakkinen
2020-10-12 16:50   ` Jarkko Sakkinen
2020-10-03  4:50 ` [PATCH v39 23/24] docs: x86/sgx: Document SGX micro architecture and kernel internals Jarkko Sakkinen
2020-10-03  4:50 ` [PATCH v39 24/24] x86/sgx: Update MAINTAINERS Jarkko Sakkinen
2020-10-16 21:04   ` Dave Hansen
2020-10-18  4:27     ` Jarkko Sakkinen
2020-10-03 14:32 ` [PATCH v39 00/24] Intel SGX foundations Greg KH
2020-10-03 14:53   ` Jarkko Sakkinen
2020-10-15 19:06 ` Dave Hansen
2020-10-17 20:43   ` Jarkko Sakkinen

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20201017210204.GA64894@linux.intel.com \
    --to=jarkko.sakkinen@linux.intel.com \
    --cc=akpm@linux-foundation.org \
    --cc=andriy.shevchenko@linux.intel.com \
    --cc=asapek@google.com \
    --cc=bp@alien8.de \
    --cc=cedric.xing@intel.com \
    --cc=chenalexchen@google.com \
    --cc=conradparker@google.com \
    --cc=cyhanish@google.com \
    --cc=dave.hansen@intel.com \
    --cc=haitao.huang@intel.com \
    --cc=jethro@fortanix.com \
    --cc=kai.huang@intel.com \
    --cc=kai.svahn@intel.com \
    --cc=kmoy@google.com \
    --cc=linux-kernel@vger.kernel.org \
    --cc=linux-sgx@vger.kernel.org \
    --cc=ludloff@google.com \
    --cc=luto@amacapital.net \
    --cc=luto@kernel.org \
    --cc=mikko.ylinen@intel.com \
    --cc=nhorman@redhat.com \
    --cc=npmccallum@redhat.com \
    --cc=puiterwijk@redhat.com \
    --cc=rientjes@google.com \
    --cc=sean.j.christopherson@intel.com \
    --cc=tglx@linutronix.de \
    --cc=x86@kernel.org \
    --cc=yaozhangx@google.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link

Linux-Sgx Archive on lore.kernel.org

Archives are clonable:
	git clone --mirror https://lore.kernel.org/linux-sgx/0 linux-sgx/git/0.git

	# If you have public-inbox 1.1+ installed, you may
	# initialize and index your mirror using the following commands:
	public-inbox-init -V2 linux-sgx linux-sgx/ https://lore.kernel.org/linux-sgx \
		linux-sgx@vger.kernel.org
	public-inbox-index linux-sgx

Example config snippet for mirrors

Newsgroup available over NNTP:
	nntp://nntp.lore.kernel.org/org.kernel.vger.linux-sgx


AGPL code for this site: git clone https://public-inbox.org/public-inbox.git