[RFC PATCH 0/1] KVM: VMX: Always sync CR3 to VMCS in nested_vmx_load_cr3

[RFC PATCH 0/1] KVM: VMX: Always sync CR3 to VMCS in nested_vmx_load_cr3

[Reto Buerki]

Muen [1] is an Open Source separation kernel (SK) for x86/64. It is
written in SPARK/Ada and uses VMX non-root mode to isolate both its
'native' (IA32-e paging) and VM guests (EPT).

Native guests are written in SPARK/Ada as well. They run directly on the
vCPU with no surrounding OS, page tables are compile-time static, guest
CR3 is initially setup in the associated VMCS by the SK and never
changes.

We are trying to use nested KVM/QEMU for faster deployment and testing
of Muen-based systems but encountered a problem with how KVM handles L2
guest CR3 in our setup. As soon as guests start to idle executing HLT,
things go sideways (#GP, unexpected page faults in native subjects,
triple faults in VMs).

We assume the failure is related to the "HLT Exiting" control in
Processor-Based VM-Execution Controls, as Muen has this flag disabled
due to the static, round-robin nature of its scheduler. This might be
uncommon.

We instrumented the SK to check if a known CR3 value changes during
runtime, which produced the following output. CR3 of guests 8 and 9 are
initially zero because these are VM guests with EPT.

Booting Muen kernel v0.9.1-262-g71ffcad4b-UNCLEAN (GNAT Community 2019 (20190517-83))
[..snip..]
PML4 address of subject 16#0000#: 16#00000000018c9000#
PML4 address of subject 16#0001#: 16#000000000189f000#
PML4 address of subject 16#0002#: 16#0000000001834000#
PML4 address of subject 16#0003#: 16#000000000183d000#
PML4 address of subject 16#0005#: 16#0000000001630000#
PML4 address of subject 16#0004#: 16#0000000001846000#
PML4 address of subject 16#0007#: 16#000000000186f000#
PML4 address of subject 16#0006#: 16#0000000001818000#
PML4 address of subject 16#0008#: 16#0000000000000000#
PML4 address of subject 16#0009#: 16#0000000000000000#
[..snip..]
ERROR CR3 of subject 1 changed: 16#0000000010ff9000# exit 16#0034#
ERROR Halting CPU

The CR3 value of guest 1 changed from 16#000000000189f000# to
16#0000000010ff9000#, which is the CR3 value of a VM guest.

A second run shows:
ERROR CR3 of subject 1 changed: 16#000000000183d000# exit 16#0034#,
which is the CR3 value of guest 3.

The problem does not only show for native guests, but also CR3 values of
VM guests with EPT get mixed up.

Tested KVM version is fd3edd4a9066.

A Muen system image to reproduce the problem is provided:

$ wget https://www.codelabs.ch/~reet/files/muen.iso.tar.xz
$ tar xfvJ muen.iso.tar.xz
$ qemu-system-x86_64 -cdrom muen.iso \
  -serial file:serial.out \
  -cpu host,+invtsc \
  --enable-kvm \
  -m 5120 \
  -smp cores=2,threads=2,sockets=1

This patch fixes the issue for Muen. I'm aware that this might not be
the proper fix, please advise how to update it if the problem is
reproducible.

[1] - https://muen.sk

Reto Buerki (1):
  KVM: VMX: Always sync CR3 to VMCS in nested_vmx_load_cr3

 arch/x86/kvm/vmx/nested.c | 2 ++
 1 file changed, 2 insertions(+)

-- 
2.20.1

[RFC PATCH] KVM: VMX: Always sync CR3 to VMCS in nested_vmx_load_cr3

[Reto Buerki]

Required to make a Muen system work on KVM nested.

Signed-off-by: Reto Buerki <reet@codelabs.ch>
---
 arch/x86/kvm/vmx/nested.c | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/arch/x86/kvm/vmx/nested.c b/arch/x86/kvm/vmx/nested.c
index 41abc62c9a8a..101b2c0c8480 100644
--- a/arch/x86/kvm/vmx/nested.c
+++ b/arch/x86/kvm/vmx/nested.c
@@ -1008,6 +1008,8 @@ static int nested_vmx_load_cr3(struct kvm_vcpu *vcpu, unsigned long cr3, bool ne
 		}
 	}
 
+	vmcs_writel(GUEST_CR3, cr3);
+
 	if (!nested_ept)
 		kvm_mmu_new_cr3(vcpu, cr3, false);
 
-- 
2.20.1

Re: [RFC PATCH] KVM: VMX: Always sync CR3 to VMCS in nested_vmx_load_cr3

[Sean Christopherson]

On Thu, Sep 26, 2019 at 04:05:41PM +0200, Reto Buerki wrote:
> Required to make a Muen system work on KVM nested.
> 
> Signed-off-by: Reto Buerki <reet@codelabs.ch>
> ---
>  arch/x86/kvm/vmx/nested.c | 2 ++
>  1 file changed, 2 insertions(+)
> 
> diff --git a/arch/x86/kvm/vmx/nested.c b/arch/x86/kvm/vmx/nested.c
> index 41abc62c9a8a..101b2c0c8480 100644
> --- a/arch/x86/kvm/vmx/nested.c
> +++ b/arch/x86/kvm/vmx/nested.c
> @@ -1008,6 +1008,8 @@ static int nested_vmx_load_cr3(struct kvm_vcpu *vcpu, unsigned long cr3, bool ne
>  		}
>  	}
>  
> +	vmcs_writel(GUEST_CR3, cr3);

This isn't wrong, but it's not the most precise fix.  I've figured out
what's going awry, in the process of determining how best to fix the issue.

> +
>  	if (!nested_ept)
>  		kvm_mmu_new_cr3(vcpu, cr3, false);
>  
> -- 
> 2.20.1
>