From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-5.5 required=3.0 tests=HEADER_FROM_DIFFERENT_DOMAINS, MAILING_LIST_MULTI,SIGNED_OFF_BY,SPF_PASS,URIBL_BLOCKED,USER_AGENT_MUTT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id E7DEFC43381 for ; Thu, 21 Mar 2019 16:24:33 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id B7736218E2 for ; Thu, 21 Mar 2019 16:24:33 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727921AbfCUQYd (ORCPT ); Thu, 21 Mar 2019 12:24:33 -0400 Received: from mga18.intel.com ([134.134.136.126]:18212 "EHLO mga18.intel.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726787AbfCUQYd (ORCPT ); Thu, 21 Mar 2019 12:24:33 -0400 X-Amp-Result: UNSCANNABLE X-Amp-File-Uploaded: False Received: from orsmga001.jf.intel.com ([10.7.209.18]) by orsmga106.jf.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 21 Mar 2019 09:24:32 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.60,253,1549958400"; d="scan'208";a="216267145" Received: from dilu-mobl2.ccr.corp.intel.com (HELO localhost) ([10.249.254.184]) by orsmga001.jf.intel.com with ESMTP; 21 Mar 2019 09:24:23 -0700 Date: Thu, 21 Mar 2019 18:24:22 +0200 From: Jarkko Sakkinen To: Sean Christopherson Cc: x86@kernel.org, linux-sgx@vger.kernel.org, akpm@linux-foundation.org, dave.hansen@intel.com, nhorman@redhat.com, npmccallum@redhat.com, serge.ayoun@intel.com, shay.katz-zamir@intel.com, haitao.huang@intel.com, andriy.shevchenko@linux.intel.com, tglx@linutronix.de, kai.svahn@intel.com, bp@alien8.de, josh@joshtriplett.org, luto@kernel.org, kai.huang@intel.com, rientjes@google.com Subject: Re: [PATCH v19 25/27] x86/sgx: SGX documentation Message-ID: <20190321162422.GV4603@linux.intel.com> References: <20190317211456.13927-1-jarkko.sakkinen@linux.intel.com> <20190317211456.13927-26-jarkko.sakkinen@linux.intel.com> <20190320171422.GD30469@linux.intel.com> MIME-Version: 1.0 Content-Type: text/plain; charset=iso-8859-1 Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: <20190320171422.GD30469@linux.intel.com> Organization: Intel Finland Oy - BIC 0357606-4 - Westendinkatu 7, 02160 Espoo User-Agent: Mutt/1.10.1 (2018-07-13) Sender: linux-sgx-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-sgx@vger.kernel.org On Wed, Mar 20, 2019 at 10:14:22AM -0700, Sean Christopherson wrote: > On Sun, Mar 17, 2019 at 11:14:54PM +0200, Jarkko Sakkinen wrote: > > Documentation of the features of the Software Guard eXtensions (SGX), > > the basic design choices for the core and driver functionality and > > the UAPI. > > > > Signed-off-by: Jarkko Sakkinen > > Co-developed-by: Sean Christopherson > > Signed-off-by: Sean Christopherson > > --- > > +Intel(R) SGX is a set of CPU instructions that can be used by applications to > > +set aside private regions of code and data. The code outside the enclave is > > +disallowed to access the memory inside the enclave by the CPU access control. > > +In a way you can think that SGX provides an inverted sandbox. It protects the > > +application from a malicious host. > > + > > +You can tell if your CPU supports SGX by looking into ``/proc/cpuinfo``: > > + > > + ``cat /proc/cpuinfo | grep sgx`` > > + > > +Overview of SGX > > +=============== > > + > > +SGX has a set of data structures to maintain information about the enclaves and > > +their security properties. BIOS reserves a fixed size region of physical memory > > +for these structures by setting Processor Reserved Memory Range Registers > > +(PRMRR). > > + > > +This memory range is protected from outside access by the CPU and all the data > > +coming in and out of the CPU package is encrypted by a key that is generated for > > +each boot cycle. > > The overview should probably be limited to architectural features, e.g. the > ISA, *architectural* EPC and EPCM behavior, etc... Then add a dedicated > section on EPC implementation(s), which are micro-architecture specific. > > > + > > +Enclaves execute in ring 3 in a special enclave submode using pages from the > > +reserved memory range. A fixed logical address range for the enclave is reserved > ^^^^^^^ > > I'm 99.99999% certain the above should be "linear address range". The SDM > SDM first introduces ELRANGE as working with logical addresses: > > Address space allocation is the specification of the range of logical > addresses that the enclave may use. This range is called the ELRANGE. > > but I think that's a documentation bug. Subsequent references in the SDM > all refer to linear addresses, and the ELRANGE checks definitely work with > linear addresses. > > > +by ENCLS(ECREATE), a leaf instruction used to create enclaves. It is referred to > > +in the documentation commonly as the *ELRANGE*. > > + > > +Every memory access to the ELRANGE is asserted by the CPU. If the CPU is not > > +executing in the enclave mode inside the enclave, #GP is raised. On the other > > This isn't correct. ELRANGE protections are activated when only executing > inside an enclave. An enclave's memory is protected by the EPC{M} when the > CPU is operating outside the enclave or in a different enclave. > > > +hand, enclave code can make memory accesses both inside and outside of the > > +ELRANGE. > > + > > +An enclave can only execute code inside the ELRANGE. Instructions that may cause > > +VMEXIT, IO instructions and instructions that require a privilege change are > > The VM-Exit blurb is technically no longer accurate, e.g. a VMM can choose > to intercept RDTSC, but RDTSC is also allowed in enclaves. > > > +prohibited inside the enclave. Interrupts and exceptions always cause an enclave > > +to exit and jump to an address outside the enclave given when the enclave is > > +entered by using the leaf instruction ENCLS(EENTER). > > + > > +Protected memory > > +---------------- > > + > > +Enclave Page Cache (EPC) > > + Physical pages used with enclaves that are protected by the CPU from > > + unauthorized access. > > + > > +Enclave Page Cache Map (EPCM) > > + A database that describes the properties and state of the pages e.g. their > > + permissions or which enclave they belong to. > > + > > +Memory Encryption Engine (MEE) integrity tree > > + Autonomously updated integrity tree. The root of the tree located in on-die > > + SRAM. > > + > > +EPC data types > > +-------------- > > + > > +SGX Enclave Control Structure (SECS) > > + Describes the global properties of an enclave. Will not be mapped to the > > + ELRANGE. > > + > > +Regular (REG) > > + These pages contain code and data. > > + > > +Thread Control Structure (TCS) > > + The pages that define the entry points inside an enclave. An enclave can > > + only be entered through these entry points and each can host a single > > + hardware thread at a time. > > + > > +Version Array (VA) > > + The pages contain 64-bit version numbers for pages that have been swapped > > + outside the enclave. Each page has the capacity of 512 version numbers. > > The page type descriptions could use a bit of improvement. Tangentially > related, would it make sense to add a "Terminology" section given the > absurd number of acronyms used by SGX? > > > + > > +Launch control > > +-------------- > > + > > +To launch an enclave, two structures must be provided for ENCLS(EINIT): > > + > > +1. **SIGSTRUCT:** signed measurement of the enclave binary. > > +2. **EINITTOKEN:** a cryptographic token CMAC-signed with a AES256-key called > > + *launch key*, which is regenerated for each boot cycle. > > + > > +The CPU holds a SHA256 hash of a 3072-bit RSA public key inside > > +IA32_SGXLEPUBKEYHASHn MSRs. Enclaves with a SIGSTRUCT that is signed with this > > +key do not require a valid EINITTOKEN and can be authorized with special > > +privileges. One of those privileges is ability to acquire the launch key with > > +ENCLS(EGETKEY). > > This section should also call out that the IA32_SGXLEPUBKEYHASHn MSRs also > affect EINIT tokens. And simply being signed with the key referenced by > IA32_SGXLEPUBKEYHASHn doesn't grant access to the EINITTOKEN_KEY, the > enclave still needs to explicit request access, e.g. the kernel could deny > access to the EINITTOKEN_KEY by refusing to create enclaves that request > said key. > > > +**IA32_FEATURE_CONTROL[SGX_LE_WR]** is used by the BIOS configure whether > > +IA32_SGXLEPUBKEYHASH MSRs are read-only or read-write before locking the feature > > +control register and handing over control to the operating system. > > ... > > > +The PF_SGX bit is set if and only if the #PF is detected by the SGX Enclave Page > > The ordering here is a bit backwards, e.g. the EPCM and its protections > should come first, and then describe the faults that can be signaled by > the EPCM. > > > +Cache Map (EPCM). The EPCM is a hardware-managed table that enforces accesses to > > +an enclave's EPC pages in addition to the software-managed kernel page tables, > > +i.e. the effective permissions for an EPC page are a logical AND of the kernel's > > +page tables and the corresponding EPCM entry. > > + > > +The EPCM is consulted only after an access walks the kernel's page tables, i.e.: > > + > > +1. the access was allowed by the kernel > > +2. the kernel's tables have become less restrictive than the EPCM > > +3. the kernel cannot fixup the cause of the fault > > + > > +Notably, (2) implies that either the kernel has botched the EPC mappings or the > > +EPCM has been invalidated (see below). Regardless of why the fault occurred, > > +userspace needs to be alerted so that it can take appropriate action, e.g. > > +restart the enclave. This is reinforced by (3) as the kernel doesn't really > > +have any other reasonable option, i.e. signalling SIGSEGV is actually the least > > +severe action possible. > > + > > +Although the primary purpose of the EPCM is to prevent a malicious or > > +compromised kernel from attacking an enclave, e.g. by modifying the enclave's > > +page tables, do not WARN on a #PF with PF_SGX set. The SGX architecture > > +effectively allows the CPU to invalidate all EPCM entries at will and requires > > +that software be prepared to handle an EPCM fault at any time. The architecture > > +defines this behavior because the EPCM is encrypted with an ephemeral key that > > +isn't exposed to software. As such, the EPCM entries cannot be preserved across > > +transitions that result in a new key being used, e.g. CPU power down as part of > > +an S3 transition or when a VM is live migrated to a new physical system. > > + > > +SGX UAPI > > +======== > > + > > +.. kernel-doc:: drivers/platform/x86/intel_sgx/sgx_ioctl.c > > Stale file reference. > > > + :functions: sgx_ioc_enclave_create > > + sgx_ioc_enclave_add_page > > + sgx_ioc_enclave_init > > + > > +.. kernel-doc:: arch/x86/include/uapi/asm/sgx.h > > + > > +References > > +========== > > + > > +* A Memory Encryption Engine Suitable for General Purpose Processors > > + > > +* System Programming Manual: 39.1.4 Intel® SGX Launch Control Configuration > > Probably should leave off the exact section, e.g. "39.1.4" is already out > of date. > > IMO this whole document could use an overhaul to uplevel the overview > and {micro-}architecture descriptions to make it a useful standalone doc > instead of a poor man's regurgitation of the SDM. If there are no > objections, I'll start typing and reply with a fresh patch at some point. > > Oh, and the vDSO interface needs to be added, I'll do that too. Shortly: agree with the comments. /Jarkko