Re: [RFC PATCH v4 2/4] x86/sgx: Implement support for MADV_WILLNEED

["Huang, Kai" <kai.huang@intel.com>]

On Tue, 2023-02-14 at 15:42 -0600, Haitao Huang wrote:
> On Tue, 14 Feb 2023 14:54:53 -0600, Huang, Kai <kai.huang@intel.com> wrote:
> 
> > On Tue, 2023-02-14 at 13:18 -0600, Haitao Huang wrote:
> > > Hi Kai
> > > 
> > > On Tue, 14 Feb 2023 03:47:24 -0600, Huang, Kai <kai.huang@intel.com>  
> > > wrote:
> > > 
> > > > On Fri, 2023-01-27 at 20:55 -0800, Haitao Huang wrote:
> > > > > @@ -97,10 +99,81 @@ static int sgx_mmap(struct file *file, struct
> > > > > vm_area_struct *vma)
> > > > >  	vma->vm_ops = &sgx_vm_ops;
> > > > >  	vma->vm_flags |= VM_PFNMAP | VM_DONTEXPAND | VM_DONTDUMP | VM_IO;
> > > > >  	vma->vm_private_data = encl;
> > > > > +	vma->vm_pgoff = PFN_DOWN(vma->vm_start - encl->base);
> > > > >   	return 0;
> > > > >  }
> > > > 
> > > > Perhaps I am missing something, but above change looks weird.
> > > > Conceptually, it doesn't/shouldn't belong to this series, which
> > > > essentially
> > > > preallocates and does EAUG EPC pages for a (or part of) given enclave.
> > > > The EAUG
> > > > logic should already be working for the normal fault path, which means
> > > > the code
> > > > change above either: 1) has been done at other place; 2) isn't needed.
> > > > 
> > > > I have kinda forgotten the userspace sequence to create an enclave.   
> > > If
> > > > I recall
> > > > correctly, you do below to create an enclave:
> > > > 
> > > > 	1) encl_fd = open("/dev/sgx_enclave");
> > > > 	2) encl_addr = mmap(encl_fd, encl_size, 0 /* pgoff */);
> > > > 	3) IOCTL(ECREATE, encl_addr, encl_size);
> > > > 
> > > > Would the above code change break the "mmap()" in above step 2?
> > > > 	
> > > 
> > > No, vm_pgoff was not used previously for enclave VMAs. I had to add this
> > > because the offset passed to sgx_fadvise is relative to file base and
> > > calculated in mm/madvise.c like this:
> > > 
> > >          offset = (loff_t)(start - vma->vm_start)
> > >                          + ((loff_t)vma->vm_pgoff << PAGE_SHIFT);
> > 
> > But shouldn't 'offset is relative to the file base' be conceptually  
> > correct from
> > the fadvice()'s point of view?
> > 
> > I think you should do:
> > 
> > 	encl_offset = offset + encl->base;
> > 
> > inside sgx_fadvice()?
> > 
> > > 
> If we don't set vma->vm_pgoff (default to zero), then offset will be  
> calculated as (start - vma->vm_start). Then the above calculation is wrong  
> if we have multiple VMAs for the same enclave, which is usually the case.

do_mmap() -> mmap_region() itself sets vma->vm_pgoff:

	vma = vm_area_alloc();
	...
	vma->vm_pgoff = pgoff;

	if (file)
		call_mmap(file, vma);	<- sgx_mmap()

I think you will always call mmap() against enclave's fd with 'pgoff' being set
to the offset relative to the file?

> 
> > > I had a comment in first version but removed it based on Jarkko's
> > > suggestion here:  
> > > https://lore.kernel.org/all/Y2B0jBsG6HE4KVk7@kernel.org/
> > > 
> > > The original comments probably seemed redundant to the definitions of  
> > > the
> > > vm_pgoff field and the fadvise interface. But let me know if we need  
> > > add a
> > > more helpful version of comments or any suggestion on the comments.
> > 
> > I still think this code change is wrong.
> > 
> > For instance, IIUC, it at least breaks the case where enclave hasn't been
> > created/initialized, where encl->base == 0 (although normal code path  
> > doesn't
> > use vm_pgoff, conceptually it's still wrong IIUC).
> > 
> > Maybe I am missing something?
> 
> The fadvise interface is only usable for an initialized enclave,  
> sgx_fadvise will return error otherwise. 
> 

True.  But that code change is unconditionally called for all mmap(), even when
enclave hasn't been created.

> Conceptually I view enclave base  
> as "file base", it's just that we don't ever need handle the zero case  
> caused by uninitialized enclave (kind of like a file never mapped). If an  
> initialized enclave happens to have zero base, it would also work.

A little bit confused about what does "enclave base" here.

To me, A file is an enclave, meaning the "file offset" equals to "enclave
offset".  "enclave base" is the base linear address of the enclave, it doesn't
matter whether it is 0 or not.  You get an "enclave address" from "enclave base"
plus "enclave offset" (or "file offset"):

	enclave_addr = enclave_base + enclave_offset/file_offset;

And such calculation is only valid after enclave has been created (enclave_base
is valid -- can be 0 or whatever).

Since sgx_mmap() can happen before enclave is created, calculating the vm_pgoff
from enclave_base is conceptually wrong.  Even if you really want to do it, it
should be:

	if (enclave_has_initialized())
		vma->vm_pgoff = ...;

But again I am not convinced why you cannot get the enclave_addr inside
sgx_fadvice().