Re: [PATCH v5 09/18] x86/sgx: Store struct sgx_encl when allocating new VA pages

From: "Huang, Kai" <kai.huang@intel.com>
To: "Mehta, Sohil" <sohil.mehta@intel.com>,
	"linux-sgx@vger.kernel.org" <linux-sgx@vger.kernel.org>,
	"x86@kernel.org" <x86@kernel.org>,
	"dave.hansen@linux.intel.com" <dave.hansen@linux.intel.com>,
	"cgroups@vger.kernel.org" <cgroups@vger.kernel.org>,
	"hpa@zytor.com" <hpa@zytor.com>,
	"mingo@redhat.com" <mingo@redhat.com>,
	"tj@kernel.org" <tj@kernel.org>, "bp@alien8.de" <bp@alien8.de>,
	"haitao.huang@linux.intel.com" <haitao.huang@linux.intel.com>,
	"tglx@linutronix.de" <tglx@linutronix.de>,
	"jarkko@kernel.org" <jarkko@kernel.org>,
	"linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>
Cc: "kristen@linux.intel.com" <kristen@linux.intel.com>,
	"yangjie@microsoft.com" <yangjie@microsoft.com>,
	"Li, Zhiquan1" <zhiquan1.li@intel.com>, "Christopherson,,
	Sean" <seanjc@google.com>,
	"mikko.ylinen@linux.intel.com" <mikko.ylinen@linux.intel.com>,
	"Zhang, Bo" <zhanb@microsoft.com>,
	"anakrish@microsoft.com" <anakrish@microsoft.com>
Subject: Re: [PATCH v5 09/18] x86/sgx: Store struct sgx_encl when allocating new VA pages
Date: Thu, 5 Oct 2023 06:49:13 +0000	[thread overview]
Message-ID: <d49edf96a0ed2a4bcbf2c7f192c89b800c6791c5.camel@intel.com> (raw)
In-Reply-To: <op.2cbqf0nkwjvjmi@hhuan26-mobl.amr.corp.intel.com>

On Wed, 2023-10-04 at 23:22 -0500, Haitao Huang wrote:
> On Wed, 04 Oct 2023 16:13:41 -0500, Huang, Kai <kai.huang@intel.com> wrote:
> 
> > On Wed, 2023-10-04 at 10:03 -0500, Haitao Huang wrote:
> > > On Tue, 03 Oct 2023 15:07:42 -0500, Huang, Kai <kai.huang@intel.com>  
> > > wrote:
> > > 
> > > > On Tue, 2023-10-03 at 01:45 -0500, Haitao Huang wrote:
> > > > > > 
> > > > > > Btw, probably a dumb question:
> > > > > > 
> > > > > > Theoretically if you only need to find a victim enclave you don't  
> > > need
> > > > > > to put VA
> > > > > > pages to the unreclaimable list, because those VA pages will be  
> > > freed
> > > > > > anyway
> > > > > > when enclave is killed.  So keeping VA pages in the list is for>
> > > > > accounting all
> > > > > > the pages that the cgroup is having?
> > > > > 
> > > > > Yes basically tracking them in cgroups as they are allocated.
> > > > > 
> > > > > VAs and SECS may also come and go as swapping/unswapping happens.  
> > > But
> > > > > if acgroup is OOM, and all reclaimables are gone (swapped out), it'd
> > > > > have toreclaim VAs/SECs in the same cgroup starting from the front  
> > > of
> > > > > the LRUlist. To reclaim a VA/SECS, it identifies the enclave from  
> > > the
> > > > > owner ofthe VA/SECS page and kills it, as killing enclave is the  
> > > only
> > > > > way toreclaim VA/SECS pages.
> > > > 
> > > > To kill enclave you just need to track SECS in  the unreclaimable  
> > > list.
> > > > Only when you want to account the total EPC pages via some list you
> > > > _probably_
> > > > need to track VA as well.  But I am not quite sure about this either.
> > > 
> > > There is a case where even SECS is paged out for an enclave with all
> > > reclaimables out.
> > 
> > Yes.  But this essentially means these enclaves are not active, thus  
> > shouldn't
> > be the victim of OOM?
> > 
> 
> But there are VA pages for the enclave at that moment. So it can be  
> candidate for OOM victim.

Yes.  I am not familiar with how does OOM choose victim, but it seems choosing
inactive enclaves seems more reasonable.

[...]

> > > There were some discussion on paging out VAs without killing enclaves  
> > > but
> > > it'd be complicated and not implemented yet.
> > 
> > No we don't involve swapping VA pages now.  It's a separate topic.
> > 
> Only mentioned it as a kind of constraints impacting current design.
> 
> Another potential alternative: we don't reclaim SECS either until OOM and  
> only track SECS pages for cgroups. But that would change current behavior.  
> And I'm not sure about other consequences, e.g., enclaves theoretically  
> can allocate pages (including VA pages) in different cgroups/processes, so  
> we may still end up tracking all VA pages for cgroups or we track SECS  
> page in all cgroups in which enclave allocated any pages. Let me know your  
> thoughts.

Let's not change current behaviour.  I seriously doubt that is needed.

So it seems to me that what we need is just some way to let the OOM find some
victim enclave.  I am not sure whether "tracking EPC pages in some lists" has
anything to do with cgroup accounting EPC pages, so will take a look the rest of
the patches.