Re: Nested paging in nested SVM setup

From: Valentine Sinitsyn <valentine.sinitsyn@gmail.com>
To: Paolo Bonzini <pbonzini@redhat.com>,
	Jan Kiszka <jan.kiszka@siemens.com>,
	kvm@vger.kernel.org
Subject: Re: Nested paging in nested SVM setup
Date: Thu, 21 Aug 2014 14:48:16 +0600	[thread overview]
Message-ID: <53F5B250.3000206@gmail.com> (raw)
In-Reply-To: <53F59192.3070500@gmail.com>

On 21.08.2014 12:28, Valentine Sinitsyn wrote:
> KVM seems to work properly (no weird exceptions injected), although my
> code now freezes (quick look on the trace suggests it's looping reading
> APIC). Not sure whose bug is it, will look further.
Looks like the problem is that if nested page tables maps some GPA to 
0xfee00000 HPA, it's really mapped to this HPA, and not intercepted with 
KVM's virtual LAPIC implementation. Consider the following trace:

>  qemu-system-x86-344   [000]   644.974072: kvm_entry:            vcpu 0
>  qemu-system-x86-344   [000]   644.974075: kvm_exit:             reason npf rip 0xffffffff8104e883 info 10000000d fee000f0
>  qemu-system-x86-344   [000]   644.974075: kvm_page_fault:       address fee000f0 error_code d
>  qemu-system-x86-344   [000]   644.974077: kvm_emulate_insn:     0:ffffffff8104e883:8b 87 00 b0 5f ff (prot64)
>  qemu-system-x86-344   [000]   644.974078: kvm_apic:             apic_read APIC_SPIV = 0xf
>  qemu-system-x86-344   [000]   644.974079: kvm_mmio:             mmio read len 4 gpa 0xfee000f0 val 0x72007200000000f
>  qemu-system-x86-344   [000]   644.974081: kvm_entry:            vcpu 0
Here, I set up NPT so that any access to 0xfee00000 nested guest 
physical address cause VM exit. Then, my code writes or reads register 
that is mapped to 0xfee00000 KVM's GPA. kvm_apic is called, and 
everything works as expected.

However, if I set up NTP to make 0xfee00000 nested guest physical 
address reads don't cause nested VM exit (by simply clearing U/S flag in 
the NPTE), I get:

>  qemu-system-x86-1066  [003]  1105.864286: kvm_exit:             reason npf rip 0xffffffff8104eaa4 info 10000000f fee00310
>  qemu-system-x86-1066  [003]  1105.864287: kvm_nested_vmexit:    rip: 0xffffffff8104eaa4 reason: npf ext_inf1: 0x000000010000000f ext_inf2: 0x00000000fee00310 ext_int: 0x00000000 ext_int_err: 0x00000000
>  qemu-system-x86-1066  [003]  1105.864287: kvm_page_fault:       address fee00310 error_code f
>  qemu-system-x86-1064  [001]  1105.864288: kvm_exit:             reason npf rip 0xffffffff8104e876 info 10000000f fee000b0
>  qemu-system-x86-1066  [003]  1105.864289: kvm_emulate_insn:     0:ffffffff8104eaa4:89 14 25 10 b3 5f ff (prot64)
>  qemu-system-x86-1064  [001]  1105.864289: kvm_nested_vmexit:    rip: 0xffffffff8104e876 reason: npf ext_inf1: 0x000000010000000f ext_inf2: 0x00000000fee000b0 ext_int: 0x00000000 ext_int_err: 0x00000000
>  qemu-system-x86-1064  [001]  1105.864289: kvm_page_fault:       address fee000b0 error_code f
>  qemu-system-x86-1064  [001]  1105.864291: kvm_emulate_insn:     0:ffffffff8104e876:89 b7 00 b0 5f ff (prot64)
>  qemu-system-x86-1066  [003]  1105.864292: kvm_inj_exception:    e (0x2)
>  qemu-system-x86-1066  [003]  1105.864293: kvm_entry:            vcpu 3
>  qemu-system-x86-1064  [001]  1105.864294: kvm_inj_exception:    e (0x2)
>  qemu-system-x86-1064  [001]  1105.864295: kvm_entry:            vcpu 1

No kvm_apic: after NPTs are set up, no page faults caused by register 
read (error_code: d), to trap and emulate APIC access.

So I'm returning to my original question: is this an intended behavior 
of KVM that APIC access on nested page tables level are not trapped, or 
is this a bug?

Valentine