Re: [intel-sgx-kernel-dev] [PATCH 08/10] kvm: vmx: add guest's IA32_SGXLEPUBKEYHASHn runtime switch support

["Huang, Kai" <kai.huang@linux.intel.com>]

On 5/21/2017 9:55 AM, Andy Lutomirski wrote:
> On Thu, May 18, 2017 at 1:14 AM, Huang, Kai <kai.huang@linux.intel.com> wrote:
>> You are making assumption that KVM will run ENCLS on behalf of guest. :)
>>
>> If we don't need to look into guest's SIGSTRUCT, EINITTOKEN, etc, then I
>> actually prefer to using MTF, as with MTF we don't have to do all the
>> remapping guest's virtual address to KVM's virtual address thing, if we
>> don't need to look into guest's ENCLS parameter. But if we need to look into
>> guest's ENCLS parameters, for example, to locate physical SECS page, or to
>> update physical EPC page's info (that KVM needs to maintain), maybe we can
>> choose running ENCLS on behalf of guest.
>
> After thinking about this a bit, I don't see how MTF helps.
> Currently, KVM works kind of like this:
>
> local_irq_disable();
> set up stuff;
> VMRESUME;
> restore some host state;
> local_irq_enable();
>
> If the guest is going to run with the EINIT-exiting bit clear, the
> only way I see this working is to modify KVM along the lines of:
>
> local_irq_disable();
> set up stuff;
> if (condition here) {
>   WRMSR to SGXLEPUBKEYHASH;
>   update percpu shadow copyl
>   clear EINIT-exiting bit;
> } else {
>   set EINIT-exiting bit;
> }
> VMRESUME;
> restore some host state;
> local_irq_enable();
>
> where "condition here" might be something like "the last VMRESUME
> exited due to EINIT".
>
> I don't see how MTF helps much.  And if I were the KVM maintainer, I
> would probably prefer to trap EINIT instead of adding a special case
> to the main vm entry code.
>

Hi Andy,

Thanks for your comments. However I didn't intend to use MTF in your 
way. The idea of using MTF (along with ENCLS VMEXIT) is, by turning on 
MTF VMEXIT upon ENCLS VMEXIT, we are able to mark a single step VMEXIT 
after ENCLS so that ENCLS can run in guest as single step.

Let me explain how the two approaches work below in general, so that we 
can decide which is better. Only trapping EINIT in order to update 
IA32_SGXLEPUBKEYHASHn is relatively simpler but I'd compare the two in 
more general way, assuming we may want to trap more ENCLS in order to, 
ex, track EPC/Enclave status/info, in the future to support, ex, EPC 
oversubscription between KVM guests.

Below diagram shows the basic idea of the two approaches.


	--------------------------------------------------------------
			|	ENCLS		|
	--------------------------------------------------------------
			|	   	       /|\
	ENCLS VMEXIT	|			| VMENTRY
			|			|
		       \|/			|
		
     		1) identify which ENCLS leaf (RAX)
     		2) reconstruct/remap guest's ENCLS parameters, ex:
			- remap any guest VA (virtual address) to KVM VA
                 	- reconstruct PAGEINFO
         	3) do whatever needed before ENCLS, ex:
			- updating MSRs before EINIT
     		4) run ENCLS on behalf of guest, and skip ENCLS
		5) emulate ENCLS result (succeeded or not)
			- update guest's RAX-RDX.
			- and/or inject #GP (or #UD).
		6) do whatever needed after ENCLS, ex:
			- updating EPC/Enclave status/info

		   	1) Run ENCLS on behalf of guest


	--------------------------------------------------------------
			 |	ENCLS		   |
	--------------------------------------------------------------
			|/|\		          |/|\
	ENCLS VMEXIT	| | VMENTRY    MTF VMEXIT | | VMENTRY
		        | |	                  | |
		       \|/|		         \|/|
	1) Turn off EMCLS VMEXIT      1) Turn off MTF VMEXIT
	2) turn on MTF VMEXIT	      2) Turn on ENCLS VMEXIT
	3) cache ENCLS parameters     3) check whether ENCLS has run
	   (ENCLS changes RAX)        4) check whether ENCLS succeeded
	4) do whatever needed before     or not.
            ENCLS                      5) do whatever needed after ENCLS

			2) Using MTF

The concern of running ENCLS on behalf of guest is emulating ENCLS 
error. KVM needs to *correctly* emulate ENCLS error to guest so that the 
error we inject to guest can reflect the right behavior as if ENCLS run 
in guest. Running ENCLS in root-mode may be potentially different 
running ENCLS in non-root mode, therefore we have to go through all 
possible error codes to make sure we can emulate. And for some error 
code, ex, SGX_LOCKFAIL, we can handle it in KVM and don't have to inject 
error to guest. So the point is we have to go through all error code to 
make sure KVM can emulate ENCLS error code correctly for guest.
Another argument is Intel may add new error codes in the future when 
more SGX functionalities are introduced, so emulating error code may be 
a burden.

Using MTF is also a little bit tricky, as when we turn on MTF VMEXIT 
upon ENCLS VMEXIT, the MTF won't be absolutely pending at end of that 
ENCLS. For example, MTF may be pending at end of interrupt (cannot 
recall exactly) if event is pending during VMENTRY from ENCLS VMEXIT. 
Therefore we have to do additional thing to check whether this MTF 
VMEXIT really happens after ENCLS run (step 3 above). And depending on 
what we need to do, we may need to check whether ENCLS succeeded or not 
in guest, which is also tricky, as ENCLS can fail in either setting 
error code in RAX, or generating #GP or #UD (step 4 above). We may still 
need to do gva->gpa->hpa, ex, in order to locate EPC/SECS page and 
update status, depending on the purpose of trapping ENCLS.

But by using MTF, we don't have to worry about ENCLS error emulation, as 
ENCLS runs in guest, thus we don't need to worry about this root-mode 
and non-root mode difference. I think this is the major reason that we 
want to use MTF.