Linux-Sgx Archive on lore.kernel.org
 help / color / Atom feed
From: Sean Christopherson <sean.j.christopherson@intel.com>
To: Jarkko Sakkinen <jarkko.sakkinen@linux.intel.com>
Cc: linux-kernel@vger.kernel.org, x86@kernel.org,
	linux-sgx@vger.kernel.org, akpm@linux-foundation.org,
	dave.hansen@intel.com, nhorman@redhat.com, npmccallum@redhat.com,
	haitao.huang@intel.com, andriy.shevchenko@linux.intel.com,
	tglx@linutronix.de, kai.svahn@intel.com, bp@alien8.de,
	josh@joshtriplett.org, luto@kernel.org, kai.huang@intel.com,
	rientjes@google.com, cedric.xing@intel.com,
	puiterwijk@redhat.com, Serge Ayoun <serge.ayoun@intel.com>
Subject: Re: [PATCH v25 07/21] x86/sgx: Enumerate and track EPC sections
Date: Wed, 5 Feb 2020 11:57:00 -0800
Message-ID: <20200205195700.GJ4877@linux.intel.com> (raw)
In-Reply-To: <20200204060545.31729-8-jarkko.sakkinen@linux.intel.com>

On Tue, Feb 04, 2020 at 08:05:31AM +0200, Jarkko Sakkinen wrote:
> From: Sean Christopherson <sean.j.christopherson@intel.com>
> 
> Enumerate Enclave Page Cache (EPC) sections via CPUID and add the data
> structures necessary to track EPC pages so that they can be allocated,
> freed and managed. As a system may have multiple EPC sections, invoke CPUID
> on SGX sub-leafs until an invalid leaf is encountered.
> 
> For simplicity, support a maximum of eight EPC sections. Existing client
> hardware supports only a single section, while upcoming server hardware
> will support at most eight sections. Bounding the number of sections also
> allows the section ID to be embedded along with a page's offset in a single
> unsigned long, enabling easy retrieval of both the VA and PA for a given
> page.

...

> +++ b/arch/x86/kernel/cpu/sgx/reclaim.c
> @@ -0,0 +1,87 @@
> +// SPDX-License-Identifier: (GPL-2.0 OR BSD-3-Clause)
> +// Copyright(c) 2016-19 Intel Corporation.
> +
> +#include <linux/freezer.h>
> +#include <linux/highmem.h>
> +#include <linux/kthread.h>
> +#include <linux/pagemap.h>
> +#include <linux/ratelimit.h>
> +#include <linux/slab.h>
> +#include <linux/sched/mm.h>
> +#include <linux/sched/signal.h>
> +#include "encls.h"
> +
> +struct task_struct *ksgxswapd_tsk;
> +
> +/*
> + * Reset all pages to uninitialized state. Pages could be in initialized on
> + * kmemexec.
> + */
> +static void sgx_sanitize_section(struct sgx_epc_section *section)
> +{
> +	struct sgx_epc_page *page, *tmp;
> +	LIST_HEAD(secs_list);
> +	int ret;
> +
> +	while (!list_empty(&section->unsanitized_page_list)) {
> +		if (kthread_should_stop())
> +			return;
> +
> +		spin_lock(&section->lock);
> +
> +		page = list_first_entry(&section->unsanitized_page_list,
> +					struct sgx_epc_page, list);
> +
> +		ret = __eremove(sgx_epc_addr(page));
> +		if (!ret)
> +			list_move(&page->list, &section->page_list);
> +		else
> +			list_move_tail(&page->list, &secs_list);
> +
> +		spin_unlock(&section->lock);
> +
> +		cond_resched();
> +	}
> +
> +	list_for_each_entry_safe(page, tmp, &secs_list, list) {
> +		if (kthread_should_stop())
> +			return;
> +
> +		ret = __eremove(sgx_epc_addr(page));
> +		if (!WARN_ON_ONCE(ret)) {

Sadly, this WARN can fire after kexec() on systems with multiple EPC
sections if the SECS has child pages in another section.

Virtual EPC (KVM) has a similar issue, which I'm handling by collecting all
pages that fail a second EREMOVE in a global list, and then retrying every
page in the global list every time a virtual EPC is destroyed, i.e. might
have freed pages.

The same approach should work here, e.g. retry all SECS pages a third time
once all sections have been sanitized and WARN if EREMOVE fails a third
time on a page.

> +			spin_lock(&section->lock);
> +			list_move(&page->list, &section->page_list);
> +			spin_unlock(&section->lock);
> +		} else {
> +			list_del(&page->list);
> +			kfree(page);
> +		}
> +
> +		cond_resched();
> +	}
> +}
> +
> +static int ksgxswapd(void *p)
> +{
> +	int i;
> +
> +	set_freezable();
> +
> +	for (i = 0; i < sgx_nr_epc_sections; i++)
> +		sgx_sanitize_section(&sgx_epc_sections[i]);

I'm having second thoughts about proactively sanitizing the EPC sections.
I think it'd be better to do EREMOVE the first time a page is allocated,
e.g. add a SGX_EPC_PAGE_UNSANITIZED flag.

  1. Sanitizing EPC that's never used is a waste of cycles, especially on
     platforms with 64gb+ of EPC.

  2. Deferring to allocation time automatically scales with the number of
     tasks that are allocating EPC.  And, the CPU time will also be
     accounted to those tasks.

  3. Breaks on-demand paging when running in a VM, e.g. if the VMM chooses
     to allocate a physical EPC page when it's actually accessed by the
     VM.  I don't expect this to be a problem any time soon, as all VMMs
     will likely preallocate EPC pages until KVM (or any other hypervisor)
     gains EPC oversusbscription support, which may or may not ever happen.
     But, I'd prefer to simply not have the problem in the first place.

The logic will be a bit more complicated, but not terribly so.  

> +
> +	return 0;
> +}

  reply index

Thread overview: 39+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2020-02-04  6:05 [PATCH v25 00/21] Intel SGX foundations Jarkko Sakkinen
2020-02-04  6:05 ` [PATCH v25 01/21] x86/cpufeatures: x86/msr: Add Intel SGX hardware bits Jarkko Sakkinen
2020-02-04  6:05 ` [PATCH v25 02/21] x86/cpufeatures: x86/msr: Intel SGX Launch Control " Jarkko Sakkinen
2020-02-04  6:05 ` [PATCH v25 03/21] x86/mm: x86/sgx: Signal SIGSEGV with PF_SGX Jarkko Sakkinen
2020-02-04  6:05 ` [PATCH v25 04/21] x86/sgx: Add SGX microarchitectural data structures Jarkko Sakkinen
2020-02-04  6:05 ` [PATCH v25 05/21] x86/sgx: Add wrappers for ENCLS leaf functions Jarkko Sakkinen
2020-02-04  6:05 ` [PATCH v25 06/21] x86/cpu/intel: Detect SGX supprt Jarkko Sakkinen
2020-02-04  6:05 ` [PATCH v25 07/21] x86/sgx: Enumerate and track EPC sections Jarkko Sakkinen
2020-02-05 19:57   ` Sean Christopherson [this message]
2020-02-05 23:11     ` Jarkko Sakkinen
2020-02-06 15:34       ` Jarkko Sakkinen
2020-02-06 15:35     ` Jarkko Sakkinen
2020-02-06 15:55       ` Sean Christopherson
2020-02-04  6:05 ` [PATCH v25 08/21] x86/sgx: Add functions to allocate and free EPC pages Jarkko Sakkinen
2020-02-04  6:05 ` [PATCH v25 09/21] mm: Introduce vm_ops->may_mprotect() Jarkko Sakkinen
2020-02-04  6:05 ` [PATCH v25 10/21] x86/sgx: Linux Enclave Driver Jarkko Sakkinen
2020-02-04  6:18   ` Jarkko Sakkinen
2020-02-05 15:58   ` Haitao Huang
2020-02-07 17:03   ` Haitao Huang
2020-02-04  6:05 ` [PATCH v25 11/21] selftests/x86: Recurse into subdirectories Jarkko Sakkinen
2020-02-04  6:05 ` [PATCH v25 12/21] selftests/x86: Add a selftest for SGX Jarkko Sakkinen
2020-02-04  6:05 ` [PATCH v25 13/21] x86/sgx: Add provisioning Jarkko Sakkinen
2020-02-04  6:05 ` [PATCH v25 14/21] x86/sgx: Add a page reclaimer Jarkko Sakkinen
2020-02-04  6:05 ` [PATCH v25 15/21] x86/sgx: ptrace() support for the SGX driver Jarkko Sakkinen
2020-02-04  6:05 ` [PATCH v25 16/21] x86/vdso: Add support for exception fixup in vDSO functions Jarkko Sakkinen
2020-02-04  6:05 ` [PATCH v25 17/21] x86/fault: Add helper function to sanitize error code Jarkko Sakkinen
2020-02-04  6:05 ` [PATCH v25 18/21] x86/traps: Attempt to fixup exceptions in vDSO before signaling Jarkko Sakkinen
2020-02-04  6:05 ` [PATCH v25 19/21] x86/vdso: Add __vdso_sgx_enter_enclave() to wrap SGX enclave transitions Jarkko Sakkinen
2020-02-04  6:05 ` [PATCH v25 20/21] selftests/x86: Add vDSO selftest for SGX Jarkko Sakkinen
2020-02-04  6:05 ` [PATCH v25 21/21] docs: x86/sgx: Document SGX micro architecture and kernel internals Jarkko Sakkinen
2020-02-05 17:54   ` Randy Dunlap
2020-02-05 23:07     ` Jarkko Sakkinen
2020-02-05 23:10       ` Randy Dunlap
2020-02-06 14:50         ` Jarkko Sakkinen
2020-02-04 15:11 ` [PATCH v25 00/21] Intel SGX foundations Sean Christopherson
2020-02-05 21:59   ` Jarkko Sakkinen
2020-02-05 23:09     ` Jarkko Sakkinen
2020-02-05 16:01 ` Haitao Huang
2020-02-05 22:01   ` Jarkko Sakkinen

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20200205195700.GJ4877@linux.intel.com \
    --to=sean.j.christopherson@intel.com \
    --cc=akpm@linux-foundation.org \
    --cc=andriy.shevchenko@linux.intel.com \
    --cc=bp@alien8.de \
    --cc=cedric.xing@intel.com \
    --cc=dave.hansen@intel.com \
    --cc=haitao.huang@intel.com \
    --cc=jarkko.sakkinen@linux.intel.com \
    --cc=josh@joshtriplett.org \
    --cc=kai.huang@intel.com \
    --cc=kai.svahn@intel.com \
    --cc=linux-kernel@vger.kernel.org \
    --cc=linux-sgx@vger.kernel.org \
    --cc=luto@kernel.org \
    --cc=nhorman@redhat.com \
    --cc=npmccallum@redhat.com \
    --cc=puiterwijk@redhat.com \
    --cc=rientjes@google.com \
    --cc=serge.ayoun@intel.com \
    --cc=tglx@linutronix.de \
    --cc=x86@kernel.org \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link

Linux-Sgx Archive on lore.kernel.org

Archives are clonable:
	git clone --mirror https://lore.kernel.org/linux-sgx/0 linux-sgx/git/0.git

	# If you have public-inbox 1.1+ installed, you may
	# initialize and index your mirror using the following commands:
	public-inbox-init -V2 linux-sgx linux-sgx/ https://lore.kernel.org/linux-sgx \
		linux-sgx@vger.kernel.org
	public-inbox-index linux-sgx

Example config snippet for mirrors

Newsgroup available over NNTP:
	nntp://nntp.lore.kernel.org/org.kernel.vger.linux-sgx


AGPL code for this site: git clone https://public-inbox.org/public-inbox.git