Re: [PATCH v5 09/18] x86/sgx: Store struct sgx_encl when allocating new VA pages

From: "Haitao Huang" <haitao.huang@linux.intel.com>
To: "hpa@zytor.com" <hpa@zytor.com>,
	"linux-sgx@vger.kernel.org" <linux-sgx@vger.kernel.org>,
	"x86@kernel.org" <x86@kernel.org>,
	"dave.hansen@linux.intel.com" <dave.hansen@linux.intel.com>,
	"cgroups@vger.kernel.org" <cgroups@vger.kernel.org>,
	"bp@alien8.de" <bp@alien8.de>,
	"linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>,
	"jarkko@kernel.org" <jarkko@kernel.org>,
	"tglx@linutronix.de" <tglx@linutronix.de>,
	"Mehta, Sohil" <sohil.mehta@intel.com>,
	"tj@kernel.org" <tj@kernel.org>,
	"mingo@redhat.com" <mingo@redhat.com>,
	"Huang, Kai" <kai.huang@intel.com>
Cc: "kristen@linux.intel.com" <kristen@linux.intel.com>,
	"yangjie@microsoft.com" <yangjie@microsoft.com>,
	"Li, Zhiquan1" <zhiquan1.li@intel.com>, "Christopherson,,
	Sean" <seanjc@google.com>,
	"mikko.ylinen@linux.intel.com" <mikko.ylinen@linux.intel.com>,
	"Zhang, Bo" <zhanb@microsoft.com>,
	"anakrish@microsoft.com" <anakrish@microsoft.com>
Subject: Re: [PATCH v5 09/18] x86/sgx: Store struct sgx_encl when allocating new VA pages
Date: Tue, 03 Oct 2023 01:45:15 -0500	[thread overview]
Message-ID: <op.2b77pp0wwjvjmi@hhuan26-mobl.amr.corp.intel.com> (raw)
In-Reply-To: <ac66cb2b6e057f5e5e78345bcddea13a3e72ee5e.camel@intel.com>

On Wed, 27 Sep 2023 06:35:57 -0500, Huang, Kai <kai.huang@intel.com> wrote:

> On Fri, 2023-09-22 at 20:06 -0700, Haitao Huang wrote:
>> From: Sean Christopherson <sean.j.christopherson@intel.com>
>>
>> In a later patch, when a cgroup has exceeded the max capacity for EPC
>> pages, it may need to identify and OOM kill a less active enclave to
>> make room for other enclaves within the same group. Such a victim
>> enclave would have no active pages other than the unreclaimable Version
>> Array (VA) and SECS pages.
>
> What does "no active pages" mean?
>

EPC pages in use.

> A "less active enclave" doesn't necessarily mean it has "no active  
> pages"?
>

I'll rephrase the above sentences

>
>> Therefore, the cgroup needs examine its
> 			^
> 			needs to
>
>> unreclaimable page list, and finding an enclave given a SECS page or a
> 				^
> 				find
>
>> VA page. This will require a backpointer from a page to an enclave,
>> which is not available for VA pages.
>>
>> Because struct sgx_epc_page instances of VA pages are not owned by an
>> sgx_encl_page instance, mark their owner as sgx_encl: pass the struct
>> sgx_encl of the enclave allocating the VA page to sgx_alloc_epc_page(),
>> which will store this value in the owner field of the struct
>> sgx_epc_page.
>
> IMHO this paragraph is hard to understand and can be more concise:
>
> One VA page can be shared by multiple enclave pages thus cannot be  
> associated
> with any 'struct sgx_encl_page' instance.  Set the owner of VA page to  
> the
> enclave instead.
>
>

Agreed

>> In a later patch, VA pages will be placed in an
>> unreclaimable queue that can be examined by the cgroup to select the OOM
>> killed enclave.
>
> The code to "place the VA page to unreclaimable queue" has been done in  
> earlier
> patch ("x86/sgx: Introduce EPC page states").  Just the unreclaimable  
> list isn't
> introduced yet.  I think you should just introduce it first then you can  
> get rid
> of those "in a later patch" staff.
>

I hope I was able to clarify to you in other threads that VA pages are not  
placed in any queue/list until [PATCH v5 11/18] x86/sgx: store  
unreclaimable pages in LRU lists.

This patch is the first one to implement tracking for unreclaimable pages.  
I'll add that as a transition hint.

> And nit: please use "unreclaimable list" consistently (not queue).
>

Yes will do

>
> Btw, probably a dumb question:
>
> Theoretically if you only need to find a victim enclave you don't need  
> to put VA
> pages to the unreclaimable list, because those VA pages will be freed  
> anyway
> when enclave is killed.  So keeping VA pages in the list is for  
> accounting all
> the pages that the cgroup is having?

Yes basically tracking them in cgroups as they are allocated.

VAs and SECS may also come and go as swapping/unswapping happens. But if a  
cgroup is OOM, and all reclaimables are gone (swapped out), it'd have to  
reclaim VAs/SECs in the same cgroup starting from the front of the LRU  
list. To reclaim a VA/SECS, it identifies the enclave from the owner of  
the VA/SECS page and kills it, as killing enclave is the only way to  
reclaim VA/SECS pages.