Re: [RFC PATCH 00/23] KVM SGX virtualization support

From: Kai Huang <kai.huang@intel.com>
To: Sean Christopherson <seanjc@google.com>
Cc: Dave Hansen <dave.hansen@intel.com>,
	linux-sgx@vger.kernel.org, kvm@vger.kernel.org, x86@kernel.org,
	jarkko@kernel.org, luto@kernel.org, haitao.huang@intel.com,
	pbonzini@redhat.com, bp@alien8.de, tglx@linutronix.de,
	mingo@redhat.com, hpa@zytor.com, jethro@fortanix.com,
	b.thiel@posteo.de, jmattson@google.com, joro@8bytes.org,
	vkuznets@redhat.com, wanpengli@tencent.com, corbet@lwn.net
Subject: Re: [RFC PATCH 00/23] KVM SGX virtualization support
Date: Fri, 8 Jan 2021 15:16:45 +1300	[thread overview]
Message-ID: <20210108151645.6c1d624f6bf383c7de0d526a@intel.com> (raw)
In-Reply-To: <X/czdL6TmbuBdZ/b@google.com>

On Thu, 7 Jan 2021 08:14:44 -0800 Sean Christopherson wrote:
> On Thu, Jan 07, 2021, Kai Huang wrote:
> > On Wed, 6 Jan 2021 16:48:58 -0800 Dave Hansen wrote:
> > > Your series adds some new interfaces, including /dev/sgx_virt_epc.  If
> > > the kernel wants to add oversubscription in the future, will old binary
> > > application users of /dev/sgx_virt_epc be able to support
> > > oversubscription?  Or, would users of /dev/sgx_virt_epc need to change
> > > to support oversubscription?
> > 
> > Oversubscription will be completely done in kernel/kvm, and will be
> > transparent to userspace, so it will not impact ABI.
> 
> It's not transparent to userpsace, odds are very good that userspace would want
> to opt in/out of EPC reclaim for its VMs.  E.g. for cases where it would be
> preferable to fail to launch a VM than degrade performance.

It seems reasonable use case, but I don't have immediate picture how it
requires new ABI related to virtualization. For instance, SGX driver should
expose sysfs saying how frequent the EPC swapping is (with KVM
oversubscription, host SGX code should provide such info in whole I think),
and cloud admin can determine whether to launch new VM.

Another argument is, theoretically, cloud admin may not know how EPC will be
used in guest, so potentially guest will only use very little EPC, thus
creating new VM won't hurt a lot, so I am not sure that, if we want to
upstream KVM oversubscription one day, do we need to consider such case.

> 
> That being said, there are no anticipated /dev/sgx_virt_epc ABI changes to
> support reclaim, as the ABI changes will be in KVM.  In the KVM oversubscription
> POC, I added a KVM ioctl to allow enabling EPC reclaim/oversubscription.  That
> ioctl took a fd for a /dev/sgx_virt_epc instance.

Adding IOCTL to enable/disable oversubscription for particular VM seems
user-case dependent, and I am not sure whether we need to support that if we
want to upstream oversubscription one day. To me, it makes sense to upstream
*basic* oversubscription (which just supports reclaiming EPC from VM) first,
and then we can extend if needed according to use cases.

Anyway, oversubscription won't break existing ABI as you mentioned. 

> 
> The reason for routing through KVM was to solve two dependencies issues:
> 
>   - KVM needs a reference to the virt_epc instance to handle SGX_CONFLICT VM-Exits
> 
>   - The SGX subsystem needs to be able to translate GPAs to HVAs to retrieve the
>     SECS for a page it is reclaiming.  That requires a KVM instance and helper
>     function.
> 
> Routing the ioctl through KVM allows KVM to hand over a pointer of itself along
> with a GPA->HVA helper, and the SGX subsystem in turn can hand back the virt_epc
> instance resolved from the fd.
> 
> It would be possible to invert the flow, e.g. pass in a KVM fd to a new
> /dev/sgx_virt_epc ioctl, but I suspect that would be kludgier, and in any case
> it would be a new ioctl and so would not break existing users.