From: Jarkko Sakkinen <jarkko@kernel.org> To: Dave Hansen <dave.hansen@intel.com> Cc: akpm@linux-foundation.org, andriy.shevchenko@linux.intel.com, asapek@google.com, bp@alien8.de, cedric.xing@intel.com, chenalexchen@google.com, conradparker@google.com, cyhanish@google.com, haitao.huang@intel.com, jethro@fortanix.com, kai.huang@intel.com, kai.svahn@intel.com, kmoy@google.com, linux-kernel@vger.kernel.org, linux-sgx@vger.kernel.org, ludloff@google.com, luto@kernel.org, mikko.ylinen@intel.com, nhorman@redhat.com, npmccallum@redhat.com, puiterwijk@redhat.com, rientjes@google.com, sean.j.christopherson@intel.com, serge.ayoun@intel.com, tglx@linutronix.de, x86@kernel.org, yaozhangx@google.com, Dave Hansen <dave.hansen@linux.intel.com> Subject: Re: [PATCH] x86/sgx: clarify 'laundry_list' locking Date: Tue, 17 Nov 2020 21:29:12 +0200 Message-ID: <20201117192912.GD10393@kernel.org> (raw) In-Reply-To: <20201116222531.4834-1-dave.hansen@intel.com> On Mon, Nov 16, 2020 at 02:25:31PM -0800, Dave Hansen wrote: > From: Dave Hansen <dave.hansen@linux.intel.com> > > Short Version: > > The SGX section->laundry_list structure is effectively thread-local, > but declared next to some shared structures. Its semantics are clear > as mud. Fix that. No functional changes. Compile tested only. > > Long Version: > > The SGX hardware keeps per-page metadata. This can provide things like > permissions, integrity and replay protection. It also prevents things > like having an enclave page mapped multiple times or shared between > enclaves. > > But, that presents a problem for kexec()'d kernels (or any other kernel > that does not run immediately after a hardware reset). This is because > the last kernel may have been rude and forgotten to reset pages, which > would trigger the the "shared page" sanity check. > > To fix this, the SGX code "launders" the pages by running the EREMOVE > instruction on all pages at boot. This is slow and can take a long > time, so it is performed off in the SGX-specific ksgxd instead of being > synchronous at boot. The init code hands the list of pages to launder > in a per-SGX-section list: ->laundry_list. The only code to touch this > list is the init code and ksgxd. This means that no locking is > necessary for ->laundry_list. > > However, a lock is required for section->page_list, which is accessed > while creating enclaves and by ksgxd. This lock (section->lock is > acquired by ksgxd while also processing ->laundry_list. It is easy > to confuse the purpose of the locking as being for ->laundry_list > and ->page_list. > > Rename ->laundry_list to ->init_laundry_list to make it clear that > this is not normally used at runtime. Also add some comments > clarifying the locking, and reorganize 'sgx_epc_section' to put 'lock' > near the things it protects. > > Note: init_laundry_list is 128 bytes of wasted space at runtime. It > could theoretically be dynamically allocated and then freed after the > laundering process. But, I suspect it would take nearly 128 bytes > of extra instructions to do that. > > Cc: Jethro Beekman <jethro@fortanix.com> > Cc: Serge Ayoun <serge.ayoun@intel.com> > Cc: Jarkko Sakkinen <jarkko@kernel.org> > Signed-off-by: Jarkko Sakkinen <jarkko@kernel.org> > Signed-off-by: Dave Hansen <dave.hansen@linux.intel.com> Thansk, this does make sense to me. I'll squash it. /Jarkko > --- > > b/arch/x86/kernel/cpu/sgx/main.c | 14 ++++++++------ > b/arch/x86/kernel/cpu/sgx/sgx.h | 15 ++++++++++++--- > 2 files changed, 20 insertions(+), 9 deletions(-) > > diff -puN arch/x86/kernel/cpu/sgx/main.c~sgx-laundry-comments arch/x86/kernel/cpu/sgx/main.c > --- a/arch/x86/kernel/cpu/sgx/main.c~sgx-laundry-comments 2020-11-16 13:55:42.624972349 -0800 > +++ b/arch/x86/kernel/cpu/sgx/main.c 2020-11-16 13:58:10.652971980 -0800 > @@ -36,13 +36,15 @@ static void sgx_sanitize_section(struct > LIST_HEAD(dirty); > int ret; > > - while (!list_empty(§ion->laundry_list)) { > + /* init_laundry_list is thread-local, no need for a lock: */ > + while (!list_empty(§ion->init_laundry_list)) { > if (kthread_should_stop()) > return; > > + /* needed for access to ->page_list: */ > spin_lock(§ion->lock); > > - page = list_first_entry(§ion->laundry_list, > + page = list_first_entry(§ion->init_laundry_list, > struct sgx_epc_page, list); > > ret = __eremove(sgx_get_epc_virt_addr(page)); > @@ -56,7 +58,7 @@ static void sgx_sanitize_section(struct > cond_resched(); > } > > - list_splice(&dirty, §ion->laundry_list); > + list_splice(&dirty, §ion->init_laundry_list); > } > > static bool sgx_reclaimer_age(struct sgx_epc_page *epc_page) > @@ -418,7 +420,7 @@ static int ksgxswapd(void *p) > sgx_sanitize_section(&sgx_epc_sections[i]); > > /* Should never happen. */ > - if (!list_empty(&sgx_epc_sections[i].laundry_list)) > + if (!list_empty(&sgx_epc_sections[i].init_laundry_list)) > WARN(1, "EPC section %d has unsanitized pages.\n", i); > } > > @@ -632,13 +634,13 @@ static bool __init sgx_setup_epc_section > section->phys_addr = phys_addr; > spin_lock_init(§ion->lock); > INIT_LIST_HEAD(§ion->page_list); > - INIT_LIST_HEAD(§ion->laundry_list); > + INIT_LIST_HEAD(§ion->init_laundry_list); > > for (i = 0; i < nr_pages; i++) { > section->pages[i].section = index; > section->pages[i].flags = 0; > section->pages[i].owner = NULL; > - list_add_tail(§ion->pages[i].list, §ion->laundry_list); > + list_add_tail(§ion->pages[i].list, §ion->init_laundry_list); > } > > section->free_cnt = nr_pages; > diff -puN arch/x86/kernel/cpu/sgx/sgx.h~sgx-laundry-comments arch/x86/kernel/cpu/sgx/sgx.h > --- a/arch/x86/kernel/cpu/sgx/sgx.h~sgx-laundry-comments 2020-11-16 13:55:42.627972349 -0800 > +++ b/arch/x86/kernel/cpu/sgx/sgx.h 2020-11-16 13:55:42.631972349 -0800 > @@ -34,15 +34,24 @@ struct sgx_epc_page { > * physical memory e.g. for memory areas of the each node. This structure is > * used to store EPC pages for one EPC section and virtual memory area where > * the pages have been mapped. > + * > + * 'lock' must be held before accessing 'page_list' or 'free_cnt'. > */ > struct sgx_epc_section { > unsigned long phys_addr; > void *virt_addr; > - struct list_head page_list; > - struct list_head laundry_list; > struct sgx_epc_page *pages; > - unsigned long free_cnt; > + > spinlock_t lock; > + struct list_head page_list; > + unsigned long free_cnt; > + > + /* > + * Pages which need EREMOVE run on them before they can be > + * used. Only safe to be accessed in ksgxd and init code. > + * Not protected by locks. > + */ > + struct list_head init_laundry_list; > }; > > extern struct sgx_epc_section sgx_epc_sections[SGX_MAX_EPC_SECTIONS]; > _ >
next prev parent reply index Thread overview: 55+ messages / expand[flat|nested] mbox.gz Atom feed top 2020-11-12 22:01 [PATCH v41 00/24] Intel SGX foundations Jarkko Sakkinen 2020-11-12 22:01 ` [PATCH v41 01/24] x86/sgx: Add SGX architectural data structures Jarkko Sakkinen 2020-11-12 22:01 ` [PATCH v41 02/24] x86/sgx: Add wrappers for ENCLS functions Jarkko Sakkinen 2020-11-12 22:01 ` [PATCH v41 03/24] x86/cpufeatures: x86/msr: Add Intel SGX hardware bits Jarkko Sakkinen 2020-11-12 22:01 ` [PATCH v41 04/24] x86/cpufeatures: x86/msr: Add Intel SGX Launch Control " Jarkko Sakkinen 2020-11-12 22:01 ` [PATCH v41 05/24] x86/sgx: Initialize metadata for Enclave Page Cache (EPC) sections Jarkko Sakkinen 2020-11-16 22:25 ` [PATCH] x86/sgx: clarify 'laundry_list' locking Dave Hansen 2020-11-17 19:29 ` Jarkko Sakkinen [this message] 2020-11-12 22:01 ` [PATCH v41 06/24] x86/mm: x86/sgx: Signal SIGSEGV with PF_SGX Jarkko Sakkinen 2020-11-12 22:01 ` [PATCH v41 07/24] x86/cpu/intel: Detect SGX support Jarkko Sakkinen 2020-11-12 22:01 ` [PATCH v41 08/24] x86/cpu/intel: Add nosgx kernel parameter Jarkko Sakkinen 2020-11-12 22:01 ` [PATCH v41 09/24] x86/sgx: Add SGX page allocator functions Jarkko Sakkinen 2020-11-12 22:01 ` [PATCH v41 10/24] mm: Add 'mprotect' hook to struct vm_operations_struct Jarkko Sakkinen 2020-11-13 10:25 ` Mel Gorman 2020-11-17 18:16 ` Jarkko Sakkinen 2020-11-15 17:08 ` Dr. Greg 2020-11-15 17:32 ` Matthew Wilcox 2020-11-15 18:36 ` Dave Hansen 2020-11-16 10:09 ` Mel Gorman 2020-11-17 19:15 ` Jarkko Sakkinen 2020-11-12 22:01 ` [PATCH v41 11/24] x86/sgx: Add SGX misc driver interface Jarkko Sakkinen 2020-11-12 22:01 ` [PATCH v41 12/24] x86/sgx: Add SGX_IOC_ENCLAVE_CREATE Jarkko Sakkinen [not found] ` <20201115044044.11040-1-hdanton@sina.com> 2020-11-17 17:35 ` Jarkko Sakkinen 2020-11-12 22:01 ` [PATCH v41 13/24] x86/sgx: Add SGX_IOC_ENCLAVE_ADD_PAGES Jarkko Sakkinen 2020-11-12 22:01 ` [PATCH v41 14/24] x86/sgx: Add SGX_IOC_ENCLAVE_INIT Jarkko Sakkinen 2020-11-12 22:01 ` [PATCH v41 15/24] x86/sgx: Add SGX_IOC_ENCLAVE_PROVISION Jarkko Sakkinen 2020-11-12 22:01 ` [PATCH v41 16/24] x86/vdso: Add support for exception fixup in vDSO functions Jarkko Sakkinen 2020-11-12 22:01 ` [PATCH v41 17/24] x86/fault: Add helper function to sanitize error code Jarkko Sakkinen 2020-11-12 22:01 ` [PATCH v41 18/24] x86/traps: Attempt to fixup exceptions in vDSO before signaling Jarkko Sakkinen 2020-11-12 22:01 ` [PATCH v41 19/24] x86/vdso: Implement a vDSO for Intel SGX enclave call Jarkko Sakkinen 2020-11-17 13:14 ` Borislav Petkov 2020-11-17 19:41 ` Jarkko Sakkinen 2020-11-12 22:01 ` [PATCH v41 20/24] selftests/x86: Add a selftest for SGX Jarkko Sakkinen 2020-11-16 18:19 ` Shuah Khan 2020-11-17 13:22 ` Borislav Petkov 2020-11-17 19:42 ` Jarkko Sakkinen 2020-11-17 17:26 ` Borislav Petkov 2020-11-17 21:27 ` Jarkko Sakkinen 2020-11-17 21:38 ` Jarkko Sakkinen 2020-11-12 22:01 ` [PATCH v41 21/24] x86/sgx: Add a page reclaimer Jarkko Sakkinen 2020-11-12 22:01 ` [PATCH v41 22/24] x86/sgx: Add ptrace() support for the SGX driver Jarkko Sakkinen 2020-11-12 22:01 ` [PATCH v41 23/24] docs: x86/sgx: Document SGX kernel architecture Jarkko Sakkinen 2020-11-12 22:01 ` [PATCH v41 24/24] x86/sgx: Update MAINTAINERS Jarkko Sakkinen 2020-11-16 16:55 ` [PATCH v41 00/24] Intel SGX foundations Borislav Petkov 2020-11-16 17:21 ` Dave Hansen 2020-11-16 17:28 ` Borislav Petkov 2020-11-17 19:20 ` Jarkko Sakkinen [not found] ` <20201114084211.5284-1-hdanton@sina.com> 2020-11-16 18:33 ` [PATCH v41 05/24] x86/sgx: Initialize metadata for Enclave Page Cache (EPC) sections Dave Hansen [not found] ` <20201115040127.7804-1-hdanton@sina.com> 2020-11-16 21:11 ` [PATCH v41 11/24] x86/sgx: Add SGX misc driver interface Dave Hansen [not found] ` <20201114090708.8684-1-hdanton@sina.com> 2020-11-17 18:12 ` [PATCH v41 06/24] x86/mm: x86/sgx: Signal SIGSEGV with PF_SGX Jarkko Sakkinen [not found] ` <20201114093256.7800-1-hdanton@sina.com> 2020-11-17 18:14 ` [PATCH v41 09/24] x86/sgx: Add SGX page allocator functions Jarkko Sakkinen [not found] ` <20201115030548.1572-1-hdanton@sina.com> 2020-11-17 18:22 ` [PATCH v41 10/24] mm: Add 'mprotect' hook to struct vm_operations_struct Jarkko Sakkinen 2020-12-15 5:38 ` [PATCH v41 00/24] Intel SGX foundations Hui, Chunyang 2020-12-15 5:43 ` Hui, Chunyang 2020-12-15 15:58 ` Jarkko Sakkinen
Reply instructions: You may reply publicly to this message via plain-text email using any one of the following methods: * Save the following mbox file, import it into your mail client, and reply-to-all from there: mbox Avoid top-posting and favor interleaved quoting: https://en.wikipedia.org/wiki/Posting_style#Interleaved_style * Reply using the --to, --cc, and --in-reply-to switches of git-send-email(1): git send-email \ --in-reply-to=20201117192912.GD10393@kernel.org \ --to=jarkko@kernel.org \ --cc=akpm@linux-foundation.org \ --cc=andriy.shevchenko@linux.intel.com \ --cc=asapek@google.com \ --cc=bp@alien8.de \ --cc=cedric.xing@intel.com \ --cc=chenalexchen@google.com \ --cc=conradparker@google.com \ --cc=cyhanish@google.com \ --cc=dave.hansen@intel.com \ --cc=dave.hansen@linux.intel.com \ --cc=haitao.huang@intel.com \ --cc=jethro@fortanix.com \ --cc=kai.huang@intel.com \ --cc=kai.svahn@intel.com \ --cc=kmoy@google.com \ --cc=linux-kernel@vger.kernel.org \ --cc=linux-sgx@vger.kernel.org \ --cc=ludloff@google.com \ --cc=luto@kernel.org \ --cc=mikko.ylinen@intel.com \ --cc=nhorman@redhat.com \ --cc=npmccallum@redhat.com \ --cc=puiterwijk@redhat.com \ --cc=rientjes@google.com \ --cc=sean.j.christopherson@intel.com \ --cc=serge.ayoun@intel.com \ --cc=tglx@linutronix.de \ --cc=x86@kernel.org \ --cc=yaozhangx@google.com \ /path/to/YOUR_REPLY https://kernel.org/pub/software/scm/git/docs/git-send-email.html * If your mail client supports setting the In-Reply-To header via mailto: links, try the mailto: link
Linux-Sgx Archive on lore.kernel.org Archives are clonable: git clone --mirror https://lore.kernel.org/linux-sgx/0 linux-sgx/git/0.git # If you have public-inbox 1.1+ installed, you may # initialize and index your mirror using the following commands: public-inbox-init -V2 linux-sgx linux-sgx/ https://lore.kernel.org/linux-sgx \ linux-sgx@vger.kernel.org public-inbox-index linux-sgx Example config snippet for mirrors Newsgroup available over NNTP: nntp://nntp.lore.kernel.org/org.kernel.vger.linux-sgx AGPL code for this site: git clone https://public-inbox.org/public-inbox.git