From: Jarkko Sakkinen <jarkko@kernel.org>
To: Reinette Chatre <reinette.chatre@intel.com>
Cc: dave.hansen@linux.intel.com, linux-sgx@vger.kernel.org,
haitao.huang@intel.com
Subject: Re: [RFC PATCH 3/4] x86/sgx: Obtain backing storage page with enclave mutex held
Date: Sat, 7 May 2022 01:43:53 +0300 [thread overview]
Message-ID: <YnWkqTR/fPsuzD+E@kernel.org> (raw)
In-Reply-To: <24fd9203331d11918b785c6a67f85d799d100be8.1651171455.git.reinette.chatre@intel.com>
On Thu, Apr 28, 2022 at 01:11:26PM -0700, Reinette Chatre wrote:
> The SGX backing storage is accessed on two paths: when there
> are insufficient enclave pages in the EPC the reclaimer works
> to move enclave pages to the backing storage and as enclaves
> access pages that have been moved to the backing storage
> they are retrieved from there as part of page fault handling.
>
> An oversubscribed SGX system will often run the reclaimer and
> page fault handler concurrently and needs to ensure that the
> backing store is accessed safely between the reclaimer and
> the page fault handler. The scenarios to consider here are:
> (a) faulting a page right after it was reclaimed,
> (b) faulting a page and reclaiming another page that are
> sharing a PCMD page.
>
> The reclaimer obtains pages from the backing storage without
> holding the enclave mutex and runs the risk of concurrently
> accessing the backing storage with the page fault handler that
> does access the backing storage with the enclave mutex held.
>
> In the scenario below a page is written to the backing store
> by the reclaimer and then immediately faulted back, before
> the reclaimer is able to set the dirty bit of the page:
>
> sgx_reclaim_pages() { sgx_vma_fault() {
> ... ...
> /* write data to backing store */
> sgx_reclaimer_write();
> mutex_lock(&encl->lock);
> __sgx_encl_eldu() {
> ...
> /* page not dirty -
> * contents may not be
> * up to date
> */
> sgx_encl_get_backing();
> ...
> }
> ...
> /* set page dirty */
> sgx_encl_put_backing();
> ...
> mutex_unlock(&encl->lock);
> } }
>
> While it is not possible to concurrently reclaim and fault the same
> enclave page the PCMD pages are shared between enclave pages
> in the enclave and enclave pages in the backing store.
> In the below scenario a PCMD page is truncated from the backing
> store after all its pages have been loaded in to the enclave
> at the same time the PCMD page is loaded from the backing store
> when one of its pages are reclaimed:
>
> sgx_reclaim_pages() { sgx_vma_fault() {
> ...
> mutex_lock(&encl->lock);
> ...
> __sgx_encl_eldu() {
> ...
> if (pcmd_page_empty) {
> /*
> * EPC page being reclaimed /*
> * shares a PCMD page with an * PCMD page truncated
> * enclave page that is being * while requested from
> * faulted in. * reclaimer.
> */ */
> sgx_encl_get_backing() <----------> sgx_encl_truncate_backing_page()
> }
> } }
>
> Protect the reclaimer's backing store access with the enclave's mutex
> to ensure that it can safely run concurrently with the page fault handler.
>
> Signed-off-by: Reinette Chatre <reinette.chatre@intel.com>
> ---
> arch/x86/kernel/cpu/sgx/main.c | 9 ++++++---
> 1 file changed, 6 insertions(+), 3 deletions(-)
>
> diff --git a/arch/x86/kernel/cpu/sgx/main.c b/arch/x86/kernel/cpu/sgx/main.c
> index 0e8741a80cf3..ae79b8d6f645 100644
> --- a/arch/x86/kernel/cpu/sgx/main.c
> +++ b/arch/x86/kernel/cpu/sgx/main.c
> @@ -252,6 +252,7 @@ static void sgx_reclaimer_write(struct sgx_epc_page *epc_page,
> sgx_encl_ewb(epc_page, backing);
> encl_page->epc_page = NULL;
> encl->secs_child_cnt--;
> + sgx_encl_put_backing(backing, true);
>
> if (!encl->secs_child_cnt && test_bit(SGX_ENCL_INITIALIZED, &encl->flags)) {
> ret = sgx_encl_get_backing(encl, PFN_DOWN(encl->size),
> @@ -323,11 +324,14 @@ static void sgx_reclaim_pages(void)
> goto skip;
>
> page_index = PFN_DOWN(encl_page->desc - encl_page->encl->base);
> +
> + mutex_lock(&encl_page->encl->lock);
> ret = sgx_encl_get_backing(encl_page->encl, page_index, &backing[i]);
> - if (ret)
> + if (ret) {
> + mutex_unlock(&encl_page->encl->lock);
> goto skip;
> + }
>
> - mutex_lock(&encl_page->encl->lock);
> encl_page->desc |= SGX_ENCL_PAGE_BEING_RECLAIMED;
> mutex_unlock(&encl_page->encl->lock);
> continue;
> @@ -355,7 +359,6 @@ static void sgx_reclaim_pages(void)
>
> encl_page = epc_page->owner;
> sgx_reclaimer_write(epc_page, &backing[i]);
> - sgx_encl_put_backing(&backing[i], true);
>
> kref_put(&encl_page->encl->refcount, sgx_encl_release);
> epc_page->flags &= ~SGX_EPC_PAGE_RECLAIMER_TRACKED;
> --
> 2.25.1
>
I fully agree with your fix but also there I would open code the required
statements from sgx_encL_put_backing(). Let's render that out overtime.
It masks more than helps.
BR, Jarkko
next prev parent reply other threads:[~2022-05-06 22:42 UTC|newest]
Thread overview: 50+ messages / expand[flat|nested] mbox.gz Atom feed top
2022-04-28 20:11 [RFC PATCH 0/4] SGX shmem backing store issue Reinette Chatre
2022-04-28 20:11 ` [RFC PATCH 1/4] x86/sgx: Do not free backing memory on ENCLS[ELDU] failure Reinette Chatre
2022-04-28 21:30 ` Dave Hansen
2022-04-28 22:20 ` Reinette Chatre
2022-04-28 22:53 ` Dave Hansen
2022-04-28 23:49 ` Reinette Chatre
2022-05-03 2:01 ` Kai Huang
2022-05-07 17:25 ` Jarkko Sakkinen
2022-05-09 17:17 ` Reinette Chatre
2022-05-10 0:36 ` Kai Huang
2022-05-11 10:26 ` Jarkko Sakkinen
2022-05-11 18:29 ` Haitao Huang
2022-05-11 22:00 ` Kai Huang
2022-05-12 21:14 ` Jarkko Sakkinen
2022-05-06 22:09 ` Jarkko Sakkinen
2022-04-28 20:11 ` [RFC PATCH 2/4] x86/sgx: Set dirty bit after modifying page contents Reinette Chatre
2022-04-28 21:40 ` Dave Hansen
2022-04-28 22:41 ` Reinette Chatre
2022-05-06 22:27 ` Jarkko Sakkinen
2022-05-06 22:40 ` Reinette Chatre
2022-05-07 18:01 ` Jarkko Sakkinen
2022-04-28 20:11 ` [RFC PATCH 3/4] x86/sgx: Obtain backing storage page with enclave mutex held Reinette Chatre
2022-04-28 21:58 ` Dave Hansen
2022-04-28 22:44 ` Reinette Chatre
2022-05-06 22:43 ` Jarkko Sakkinen [this message]
2022-04-28 20:11 ` [RFC PATCH 4/4] x86/sgx: Do not allocate backing pages when loading from backing store Reinette Chatre
2022-04-28 21:12 ` [RFC PATCH 0/4] SGX shmem backing store issue Dave Hansen
2022-04-29 18:50 ` Reinette Chatre
2022-04-29 19:45 ` Dave Hansen
2022-04-30 3:22 ` Reinette Chatre
2022-04-30 15:52 ` Reinette Chatre
2022-05-02 14:36 ` Dave Hansen
2022-05-02 17:11 ` Reinette Chatre
2022-05-02 21:33 ` Dave Hansen
2022-05-04 22:13 ` Reinette Chatre
2022-05-04 22:58 ` Dave Hansen
2022-05-04 23:36 ` Reinette Chatre
2022-05-04 23:50 ` Dave Hansen
2022-05-05 0:08 ` Reinette Chatre
2022-05-04 23:05 ` Dave Hansen
2022-05-07 17:46 ` Jarkko Sakkinen
2022-05-07 17:48 ` Jarkko Sakkinen
2022-05-09 17:09 ` Reinette Chatre
2022-05-10 22:28 ` Jarkko Sakkinen
2022-05-11 17:23 ` Reinette Chatre
2022-05-12 14:10 ` Jarkko Sakkinen
2022-04-28 21:29 ` Dave Hansen
2022-04-28 22:20 ` Reinette Chatre
2022-05-04 6:40 ` Jarkko Sakkinen
2022-05-05 6:09 ` Jarkko Sakkinen
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=YnWkqTR/fPsuzD+E@kernel.org \
--to=jarkko@kernel.org \
--cc=dave.hansen@linux.intel.com \
--cc=haitao.huang@intel.com \
--cc=linux-sgx@vger.kernel.org \
--cc=reinette.chatre@intel.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).