[PATCH v2 0/2] Take away EREMOVE from sgx_free_epc_page()

[PATCH v2 0/2] Take away EREMOVE from sgx_free_epc_page()

[jarkko]

From: "jarkko@kernel.org" <jarkko@kernel.org>

Refactor EREMOVE out of sgx_free_epc_page() since it is only used in
one function.

I re-organized my Git a bit in order to keep track of things:

- 'master' bleading edge and 'next' ready PR ready branch
- 'numa' has nearly finished stuff, including the patches I'm
  posting now.

jarkko@kernel.org (2):
  x86/sgx: Remove a warn from sgx_free_epc_page()
  x86/sgx: Wipe out EREMOVE from sgx_free_epc_page()

 arch/x86/kernel/cpu/sgx/encl.c | 15 ++++++++++++++-
 arch/x86/kernel/cpu/sgx/main.c | 12 ++++--------
 2 files changed, 18 insertions(+), 9 deletions(-)

-- 
2.29.2

[PATCH v2 1/2] x86/sgx: Remove a warn from sgx_free_epc_page()

[jarkko]

From: "jarkko@kernel.org" <jarkko@kernel.org>

Remove SGX_EPC_PAGE_RECLAIMER_TRACKED check and warning. This cannot
happen, as enclave pages are freed only at the time when encl->refcount
triggers, i.e. when both VFS and the page reclaimer have given up on
their references.

Signed-off-by: jarkko@kernel.org <jarkko@kernel.org>
---
 arch/x86/kernel/cpu/sgx/main.c | 2 --
 1 file changed, 2 deletions(-)

diff --git a/arch/x86/kernel/cpu/sgx/main.c b/arch/x86/kernel/cpu/sgx/main.c
index 0da510763c47..b2996cb63a8a 100644
--- a/arch/x86/kernel/cpu/sgx/main.c
+++ b/arch/x86/kernel/cpu/sgx/main.c
@@ -636,8 +636,6 @@ void sgx_free_epc_page(struct sgx_epc_page *page)
 	struct sgx_epc_section *section = &sgx_epc_sections[page->section];
 	int ret;
 
-	WARN_ON_ONCE(page->flags & SGX_EPC_PAGE_RECLAIMER_TRACKED);
-
 	ret = __eremove(sgx_get_epc_virt_addr(page));
 	if (WARN_ONCE(ret, "EREMOVE returned %d (0x%x)", ret, ret))
 		return;
-- 
2.29.2

[PATCH v2 2/2] x86/sgx: Wipe out EREMOVE from sgx_free_epc_page()

[jarkko]

From: "jarkko@kernel.org" <jarkko@kernel.org>

Encapsulate the snippet in sgx_free_epc_page() concerning EREMOVE to
sgx_reset_epc_page(), which is a static helper function for
sgx_encl_release(). It's the only function existing, which deals with
initialized pages.

Signed-off-by: jarkko@kernel.org <jarkko@kernel.org>
---
v2: Fixed a compilation error s/page/epc_page/ change not amended
    to the patches.
 arch/x86/kernel/cpu/sgx/encl.c | 15 ++++++++++++++-
 arch/x86/kernel/cpu/sgx/main.c | 10 ++++------
 2 files changed, 18 insertions(+), 7 deletions(-)

diff --git a/arch/x86/kernel/cpu/sgx/encl.c b/arch/x86/kernel/cpu/sgx/encl.c
index fe7256db6e73..511f4bc70699 100644
--- a/arch/x86/kernel/cpu/sgx/encl.c
+++ b/arch/x86/kernel/cpu/sgx/encl.c
@@ -389,6 +389,16 @@ const struct vm_operations_struct sgx_vm_ops = {
 	.access = sgx_vma_access,
 };
 
+
+static void sgx_reset_epc_page(struct sgx_epc_page *epc_page)
+{
+	int ret;
+
+	ret = __eremove(sgx_get_epc_virt_addr(epc_page));
+	if (WARN_ONCE(ret, "EREMOVE returned %d (0x%x)", ret, ret))
+		return;
+}
+
 /**
  * sgx_encl_release - Destroy an enclave instance
  * @kref:	address of a kref inside &sgx_encl
@@ -412,6 +422,7 @@ void sgx_encl_release(struct kref *ref)
 			if (sgx_unmark_page_reclaimable(entry->epc_page))
 				continue;
 
+			sgx_reset_epc_page(entry->epc_page);
 			sgx_free_epc_page(entry->epc_page);
 			encl->secs_child_cnt--;
 			entry->epc_page = NULL;
@@ -423,6 +434,7 @@ void sgx_encl_release(struct kref *ref)
 	xa_destroy(&encl->page_array);
 
 	if (!encl->secs_child_cnt && encl->secs.epc_page) {
+		sgx_reset_epc_page(encl->secs.epc_page);
 		sgx_free_epc_page(encl->secs.epc_page);
 		encl->secs.epc_page = NULL;
 	}
@@ -431,7 +443,8 @@ void sgx_encl_release(struct kref *ref)
 		va_page = list_first_entry(&encl->va_pages, struct sgx_va_page,
 					   list);
 		list_del(&va_page->list);
-		sgx_free_epc_page(va_page->epc_page);
+		sgx_reset_epc_page(entry->epc_page);
+		sgx_free_epc_page(entry->epc_page);
 		kfree(va_page);
 	}
 
diff --git a/arch/x86/kernel/cpu/sgx/main.c b/arch/x86/kernel/cpu/sgx/main.c
index b2996cb63a8a..89d52fb56d4b 100644
--- a/arch/x86/kernel/cpu/sgx/main.c
+++ b/arch/x86/kernel/cpu/sgx/main.c
@@ -629,16 +629,14 @@ struct sgx_epc_page *sgx_alloc_epc_page(void *owner, bool reclaim)
  * sgx_free_epc_page() - Free an EPC page
  * @page:	an EPC page
  *
- * Call EREMOVE for an EPC page and insert it back to the list of free pages.
+ * Put the EPC page back to the list of free pages. It's the callers
+ * responsibility to make sure that the page is in uninitialized state In other
+ * words, do EREMOVE, EWB or whatever operation is necessary before calling
+ * this function.
  */
 void sgx_free_epc_page(struct sgx_epc_page *page)
 {
 	struct sgx_epc_section *section = &sgx_epc_sections[page->section];
-	int ret;
-
-	ret = __eremove(sgx_get_epc_virt_addr(page));
-	if (WARN_ONCE(ret, "EREMOVE returned %d (0x%x)", ret, ret))
-		return;
 
 	spin_lock(&section->lock);
 	list_add_tail(&page->list, &section->page_list);
-- 
2.29.2

Re: [PATCH v2 2/2] x86/sgx: Wipe out EREMOVE from sgx_free_epc_page()

[Kai Huang]

On Wed, 13 Jan 2021 01:33:51 +0200 jarkko@kernel.org wrote:
> From: "jarkko@kernel.org" <jarkko@kernel.org>
> 
> Encapsulate the snippet in sgx_free_epc_page() concerning EREMOVE to
> sgx_reset_epc_page(), which is a static helper function for
> sgx_encl_release(). It's the only function existing, which deals with
> initialized pages.
> 
> Signed-off-by: jarkko@kernel.org <jarkko@kernel.org>
> ---
> v2: Fixed a compilation error s/page/epc_page/ change not amended
>     to the patches.
>  arch/x86/kernel/cpu/sgx/encl.c | 15 ++++++++++++++-
>  arch/x86/kernel/cpu/sgx/main.c | 10 ++++------
>  2 files changed, 18 insertions(+), 7 deletions(-)
> 
> diff --git a/arch/x86/kernel/cpu/sgx/encl.c b/arch/x86/kernel/cpu/sgx/encl.c
> index fe7256db6e73..511f4bc70699 100644
> --- a/arch/x86/kernel/cpu/sgx/encl.c
> +++ b/arch/x86/kernel/cpu/sgx/encl.c
> @@ -389,6 +389,16 @@ const struct vm_operations_struct sgx_vm_ops = {
>  	.access = sgx_vma_access,
>  };
>  
> +
> +static void sgx_reset_epc_page(struct sgx_epc_page *epc_page)
> +{
> +	int ret;
> +
> +	ret = __eremove(sgx_get_epc_virt_addr(epc_page));
> +	if (WARN_ONCE(ret, "EREMOVE returned %d (0x%x)", ret, ret))
> +		return;
> +}
> +
>  /**
>   * sgx_encl_release - Destroy an enclave instance
>   * @kref:	address of a kref inside &sgx_encl
> @@ -412,6 +422,7 @@ void sgx_encl_release(struct kref *ref)
>  			if (sgx_unmark_page_reclaimable(entry->epc_page))
>  				continue;
>  
> +			sgx_reset_epc_page(entry->epc_page);
>  			sgx_free_epc_page(entry->epc_page);
>  			encl->secs_child_cnt--;
>  			entry->epc_page = NULL;
> @@ -423,6 +434,7 @@ void sgx_encl_release(struct kref *ref)
>  	xa_destroy(&encl->page_array);
>  
>  	if (!encl->secs_child_cnt && encl->secs.epc_page) {
> +		sgx_reset_epc_page(encl->secs.epc_page);
>  		sgx_free_epc_page(encl->secs.epc_page);
>  		encl->secs.epc_page = NULL;
>  	}
> @@ -431,7 +443,8 @@ void sgx_encl_release(struct kref *ref)
>  		va_page = list_first_entry(&encl->va_pages, struct sgx_va_page,
>  					   list);
>  		list_del(&va_page->list);
> -		sgx_free_epc_page(va_page->epc_page);
> +		sgx_reset_epc_page(entry->epc_page);
> +		sgx_free_epc_page(entry->epc_page);

Reply so that Sean can see, since he just subscribed to linux-sgx list.

As Sean pointed out in KVM SGX virtualization thread, entry should be va_page.

>  		kfree(va_page);
>  	}
>  
> diff --git a/arch/x86/kernel/cpu/sgx/main.c b/arch/x86/kernel/cpu/sgx/main.c
> index b2996cb63a8a..89d52fb56d4b 100644
> --- a/arch/x86/kernel/cpu/sgx/main.c
> +++ b/arch/x86/kernel/cpu/sgx/main.c
> @@ -629,16 +629,14 @@ struct sgx_epc_page *sgx_alloc_epc_page(void *owner, bool reclaim)
>   * sgx_free_epc_page() - Free an EPC page
>   * @page:	an EPC page
>   *
> - * Call EREMOVE for an EPC page and insert it back to the list of free pages.
> + * Put the EPC page back to the list of free pages. It's the callers
> + * responsibility to make sure that the page is in uninitialized state In other
> + * words, do EREMOVE, EWB or whatever operation is necessary before calling
> + * this function.
>   */
>  void sgx_free_epc_page(struct sgx_epc_page *page)
>  {
>  	struct sgx_epc_section *section = &sgx_epc_sections[page->section];
> -	int ret;
> -
> -	ret = __eremove(sgx_get_epc_virt_addr(page));
> -	if (WARN_ONCE(ret, "EREMOVE returned %d (0x%x)", ret, ret))
> -		return;
>  
>  	spin_lock(&section->lock);
>  	list_add_tail(&page->list, &section->page_list);
> -- 
> 2.29.2
> 

Re: [PATCH v2 2/2] x86/sgx: Wipe out EREMOVE from sgx_free_epc_page()

[Jarkko Sakkinen]

On Wed, Jan 13, 2021 at 02:09:17PM +1300, Kai Huang wrote:
> On Wed, 13 Jan 2021 01:33:51 +0200 jarkko@kernel.org wrote:
> > From: "jarkko@kernel.org" <jarkko@kernel.org>
> > 
> > Encapsulate the snippet in sgx_free_epc_page() concerning EREMOVE to
> > sgx_reset_epc_page(), which is a static helper function for
> > sgx_encl_release(). It's the only function existing, which deals with
> > initialized pages.
> > 
> > Signed-off-by: jarkko@kernel.org <jarkko@kernel.org>
> > ---
> > v2: Fixed a compilation error s/page/epc_page/ change not amended
> >     to the patches.
> >  arch/x86/kernel/cpu/sgx/encl.c | 15 ++++++++++++++-
> >  arch/x86/kernel/cpu/sgx/main.c | 10 ++++------
> >  2 files changed, 18 insertions(+), 7 deletions(-)
> > 
> > diff --git a/arch/x86/kernel/cpu/sgx/encl.c b/arch/x86/kernel/cpu/sgx/encl.c
> > index fe7256db6e73..511f4bc70699 100644
> > --- a/arch/x86/kernel/cpu/sgx/encl.c
> > +++ b/arch/x86/kernel/cpu/sgx/encl.c
> > @@ -389,6 +389,16 @@ const struct vm_operations_struct sgx_vm_ops = {
> >  	.access = sgx_vma_access,
> >  };
> >  
> > +
> > +static void sgx_reset_epc_page(struct sgx_epc_page *epc_page)
> > +{
> > +	int ret;
> > +
> > +	ret = __eremove(sgx_get_epc_virt_addr(epc_page));
> > +	if (WARN_ONCE(ret, "EREMOVE returned %d (0x%x)", ret, ret))
> > +		return;
> > +}
> > +
> >  /**
> >   * sgx_encl_release - Destroy an enclave instance
> >   * @kref:	address of a kref inside &sgx_encl
> > @@ -412,6 +422,7 @@ void sgx_encl_release(struct kref *ref)
> >  			if (sgx_unmark_page_reclaimable(entry->epc_page))
> >  				continue;
> >  
> > +			sgx_reset_epc_page(entry->epc_page);
> >  			sgx_free_epc_page(entry->epc_page);
> >  			encl->secs_child_cnt--;
> >  			entry->epc_page = NULL;
> > @@ -423,6 +434,7 @@ void sgx_encl_release(struct kref *ref)
> >  	xa_destroy(&encl->page_array);
> >  
> >  	if (!encl->secs_child_cnt && encl->secs.epc_page) {
> > +		sgx_reset_epc_page(encl->secs.epc_page);
> >  		sgx_free_epc_page(encl->secs.epc_page);
> >  		encl->secs.epc_page = NULL;
> >  	}
> > @@ -431,7 +443,8 @@ void sgx_encl_release(struct kref *ref)
> >  		va_page = list_first_entry(&encl->va_pages, struct sgx_va_page,
> >  					   list);
> >  		list_del(&va_page->list);
> > -		sgx_free_epc_page(va_page->epc_page);
> > +		sgx_reset_epc_page(entry->epc_page);
> > +		sgx_free_epc_page(entry->epc_page);
> 
> Reply so that Sean can see, since he just subscribed to linux-sgx list.
> 
> As Sean pointed out in KVM SGX virtualization thread, entry should be va_page.

Thank, noticed, and responded. I'm sending v3. I think it would be best to
then include that version to KVM patch set, as there is no such a rush to
get my patches to upstream. I already dropped them from my bleeding edge
numa branch.

/Jarkko

Re: [PATCH v2 2/2] x86/sgx: Wipe out EREMOVE from sgx_free_epc_page()

[Kai Huang]

On Wed, 13 Jan 2021 23:08:08 +0200 Jarkko Sakkinen wrote:
> On Wed, Jan 13, 2021 at 02:09:17PM +1300, Kai Huang wrote:
> > On Wed, 13 Jan 2021 01:33:51 +0200 jarkko@kernel.org wrote:
> > > From: "jarkko@kernel.org" <jarkko@kernel.org>
> > > 
> > > Encapsulate the snippet in sgx_free_epc_page() concerning EREMOVE to
> > > sgx_reset_epc_page(), which is a static helper function for
> > > sgx_encl_release(). It's the only function existing, which deals with
> > > initialized pages.
> > > 
> > > Signed-off-by: jarkko@kernel.org <jarkko@kernel.org>
> > > ---
> > > v2: Fixed a compilation error s/page/epc_page/ change not amended
> > >     to the patches.
> > >  arch/x86/kernel/cpu/sgx/encl.c | 15 ++++++++++++++-
> > >  arch/x86/kernel/cpu/sgx/main.c | 10 ++++------
> > >  2 files changed, 18 insertions(+), 7 deletions(-)
> > > 
> > > diff --git a/arch/x86/kernel/cpu/sgx/encl.c b/arch/x86/kernel/cpu/sgx/encl.c
> > > index fe7256db6e73..511f4bc70699 100644
> > > --- a/arch/x86/kernel/cpu/sgx/encl.c
> > > +++ b/arch/x86/kernel/cpu/sgx/encl.c
> > > @@ -389,6 +389,16 @@ const struct vm_operations_struct sgx_vm_ops = {
> > >  	.access = sgx_vma_access,
> > >  };
> > >  
> > > +
> > > +static void sgx_reset_epc_page(struct sgx_epc_page *epc_page)
> > > +{
> > > +	int ret;
> > > +
> > > +	ret = __eremove(sgx_get_epc_virt_addr(epc_page));
> > > +	if (WARN_ONCE(ret, "EREMOVE returned %d (0x%x)", ret, ret))
> > > +		return;
> > > +}
> > > +
> > >  /**
> > >   * sgx_encl_release - Destroy an enclave instance
> > >   * @kref:	address of a kref inside &sgx_encl
> > > @@ -412,6 +422,7 @@ void sgx_encl_release(struct kref *ref)
> > >  			if (sgx_unmark_page_reclaimable(entry->epc_page))
> > >  				continue;
> > >  
> > > +			sgx_reset_epc_page(entry->epc_page);
> > >  			sgx_free_epc_page(entry->epc_page);
> > >  			encl->secs_child_cnt--;
> > >  			entry->epc_page = NULL;
> > > @@ -423,6 +434,7 @@ void sgx_encl_release(struct kref *ref)
> > >  	xa_destroy(&encl->page_array);
> > >  
> > >  	if (!encl->secs_child_cnt && encl->secs.epc_page) {
> > > +		sgx_reset_epc_page(encl->secs.epc_page);
> > >  		sgx_free_epc_page(encl->secs.epc_page);
> > >  		encl->secs.epc_page = NULL;
> > >  	}
> > > @@ -431,7 +443,8 @@ void sgx_encl_release(struct kref *ref)
> > >  		va_page = list_first_entry(&encl->va_pages, struct sgx_va_page,
> > >  					   list);
> > >  		list_del(&va_page->list);
> > > -		sgx_free_epc_page(va_page->epc_page);
> > > +		sgx_reset_epc_page(entry->epc_page);
> > > +		sgx_free_epc_page(entry->epc_page);
> > 
> > Reply so that Sean can see, since he just subscribed to linux-sgx list.
> > 
> > As Sean pointed out in KVM SGX virtualization thread, entry should be va_page.
> 
> Thank, noticed, and responded. I'm sending v3. I think it would be best to
> then include that version to KVM patch set, as there is no such a rush to
> get my patches to upstream. I already dropped them from my bleeding edge
> numa branch.

Sure. I'll take your patch once I see it. Thanks Jarkko.