Nested paging in nested SVM setup

Nested paging in nested SVM setup

[Valentine Sinitsyn]

Hi all,

I'm using a KVM/Qemu nested SVM setup to debug another hypervisor 
(Jailhouse) I contribute to. IOW, the scheme is: AMD64 Linux host 
running [paravirtualized] AMD64 Linux guest (the same kernel as the 
host) running Jailhouse.

Jailhouse, in turn, uses Nested Paging to virtualize xAPIC: APIC page 
(0xfee00000, no APIC remapping) is mapped read-only into Jailhouse's 
guests. This of course implies that APIC page appears to Jailhouse 
guests as uncacheable (UC).

Is it achievable in the setup I described, or do I need to run my code 
on a real hardware to make the APIC page accesses in Jailhouse guests 
uncacheable?

Thanks in advance.

--
Best regards,
Valentine Sinitsyn

Re: Nested paging in nested SVM setup

[Jan Kiszka]

On 2014-06-18 13:36, Valentine Sinitsyn wrote:
> Hi all,
> 
> I'm using a KVM/Qemu nested SVM setup to debug another hypervisor
> (Jailhouse) I contribute to. IOW, the scheme is: AMD64 Linux host
> running [paravirtualized] AMD64 Linux guest (the same kernel as the
> host) running Jailhouse.
> 
> Jailhouse, in turn, uses Nested Paging to virtualize xAPIC: APIC page
> (0xfee00000, no APIC remapping) is mapped read-only into Jailhouse's
> guests. This of course implies that APIC page appears to Jailhouse
> guests as uncacheable (UC).
> 
> Is it achievable in the setup I described, or do I need to run my code
> on a real hardware to make the APIC page accesses in Jailhouse guests
> uncacheable?

If we want to provide useful nested SVM support, this must be feasible.
If there is a bug, it has to be fixed.

Maybe you can describe how you configured the involved units (NPT
structures, guest / host PAR, MTRR etc.).

Even better would be a test case based on kvm-unit-tests (see [1],
x86/svm.c) that replicates the observed behavior. If it reveals a bug,
this test would be very valuable for making sure it remains fixed (once
that is done).

Jan

[1] https://git.kernel.org/cgit/virt/kvm/kvm-unit-tests.git

-- 
Siemens AG, Corporate Technology, CT RTC ITP SES-DE
Corporate Competence Center Embedded Linux

Re: Nested paging in nested SVM setup

[Valentine Sinitsyn]

Hi Jan,

> If we want to provide useful nested SVM support, this must be feasible.
> If there is a bug, it has to be fixed.
I was more concerned about if it is supported (and it means I do 
something wrong), or if it is not supported (at least, now).

> Maybe you can describe how you configured the involved units (NPT
> structures, guest / host PAR, MTRR etc.).
I've tried different combinations, but to be specific:
- NPT: four-level long-mode page tables; all PTEs except terminal have 
U,R,P bits set (0x07), as per APMv2 15.25.5
- APIC page pte; physical address 0xfee00000, flags: PAT, PWT, PCD, U, P 
(0x9D)
- guest PAT and host PAT are the same, 0x7010600070106 (as set by the 
Linux kernel). Guest PAT is stored in VMCB; host PAT is restored at each 
#VMEXIT.
- MTRRs. No changes to what Linux use prior to VM entry here; #0 (base 
0x80000000, mask 0xFF80000000) uncacheable, others are disabled.

I also noticed that setting PAT MSR from the nested hypervisor leaves 
high word unassigned, i.e. the code like this:

   mov $0x70106, %rax
   mov %rax, %rdx
   mov $0x0277, %rcx
   wrmsr
   rdmsr

yields %rax = 0, %rdx = 0x70106.

> Even better would be a test case based on kvm-unit-tests (see [1],
Will have a look at it, thanks.

--
Best regards,
Valentine Sinitsyn

Re: Nested paging in nested SVM setup

[Valentine Sinitsyn]

Hi all,

> If we want to provide useful nested SVM support, this must be feasible.
> If there is a bug, it has to be fixed.
I did a quick look on KVM sources this morning, and although I can be 
wrong, this really looks like a bug.

The reason is nested_svm_vmrun() doesn't do anything to host or guest 
PATs (thus no easy way to set memory type if NPT is used and guest PAT 
has non-default value). And since svm_set_cr0() explicitly clears CD 
(for a good reason, I suppose), there seem to be no way to control 
caching from the nested setup altogether.

--
Regards,
Valentine Sinitsyn

Re: Nested paging in nested SVM setup

[Paolo Bonzini]

Il 18/06/2014 18:59, Valentine Sinitsyn ha scritto:
>
> I also noticed that setting PAT MSR from the nested hypervisor leaves
> high word unassigned, i.e. the code like this:
>
>   mov $0x70106, %rax
>   mov %rax, %rdx
>   mov $0x0277, %rcx
>   wrmsr
>   rdmsr
>
> yields %rax = 0, %rdx = 0x70106.

This should be the trivial fix:

diff --git a/arch/x86/include/asm/kvm_host.h 
b/arch/x86/include/asm/kvm_host.h
index 0b140dc65bee..8a1cdc0f8fe7 100644
--- a/arch/x86/include/asm/kvm_host.h
+++ b/arch/x86/include/asm/kvm_host.h
@@ -461,7 +461,7 @@ struct kvm_vcpu_arch {
  	bool nmi_injected;    /* Trying to inject an NMI this entry */

  	struct mtrr_state_type mtrr_state;
-	u32 pat;
+	u64 pat;

  	unsigned switch_db_regs;
  	unsigned long db[KVM_NR_DB_REGS];

Paolo

Re: Nested paging in nested SVM setup

[Valentine Sinitsyn]

Hi all,

Please excuse me for bringing alive a two-month old thread, but I had 
time to investigate the issue a bit only recently.

On 18.06.2014 18:47, Jan Kiszka wrote:
> On 2014-06-18 13:36, Valentine Sinitsyn wrote:
> If we want to provide useful nested SVM support, this must be feasible.
> If there is a bug, it has to be fixed.
Looks like it is a bug in KVM. I had a chance to run the same code 
bare-metal ([1], line 310 is uncommented for bare-metal case but present 
for nested SVM), and it seems to work as expected. However, When I trace 
it in nested SVM setup, after some successful APIC reads and writes, I 
get the following:

>  qemu-system-x86-1968  [001] 220417.681261: kvm_nested_vmexit:    rip: 0xffffffff8104f5b8 reason: npf ext_inf1: 0x000000010000000f ext_inf2: 0x00000000fee00300 ext_int:
>  0x00000000 ext_int_err: 0x00000000
>  qemu-system-x86-1968  [001] 220417.681261: kvm_page_fault:       address fee00300 error_code f
>  qemu-system-x86-1968  [001] 220417.681263: kvm_emulate_insn:     0:ffffffff8104f5b8:89 04 25 00 93 5f ff (prot64)
>  qemu-system-x86-1968  [001] 220417.681268: kvm_inj_exception:     (0x23c)
>  qemu-system-x86-1968  [001] 220417.681269: kvm_entry:            vcpu 0
>  qemu-system-x86-1968  [001] 220417.681271: kvm_exit:             reason  rip 0xffffffff8104f5b8 info 0 0

You can see the problem here: the code tries to access APIC MMIO 
register, which is trapped by KVM's MMU code (at nested page table 
walk). During MMIO access emulation, KVM decides to inject 0x23c 
exception (which looks wrong, as there is no exception with this number 
defined). After that things become flawed (pay attention to empty reason 
in the last line; the VMCB is certainly not in the state KVM 
expects/supports).

I'm no KVM expert, and will be grateful for debugging suggestions (or 
maybe even assistance).

Many thanks for the help.

1. 
https://github.com/vsinitsyn/jailhouse/blob/amd-v/hypervisor/arch/x86/svm.c#L301

--
Regards,
Valentine Sinitsyn

Re: Nested paging in nested SVM setup

[Paolo Bonzini]

Il 20/08/2014 08:46, Valentine Sinitsyn ha scritto:
> 
> You can see the problem here: the code tries to access APIC MMIO
> register, which is trapped by KVM's MMU code (at nested page table
> walk). During MMIO access emulation, KVM decides to inject 0x23c
> exception (which looks wrong, as there is no exception with this number
> defined). After that things become flawed (pay attention to empty reason
> in the last line; the VMCB is certainly not in the state KVM
> expects/supports).
> 
> I'm no KVM expert, and will be grateful for debugging suggestions (or
> maybe even assistance).

Is the 0x23c always the same?  Can you try this patch?

diff --git a/arch/x86/kvm/x86.c b/arch/x86/kvm/x86.c
index 204422de3fed..194e9300a31b 100644
--- a/arch/x86/kvm/x86.c
+++ b/arch/x86/kvm/x86.c
@@ -346,6 +346,7 @@ static void kvm_multiple_exception(struct kvm_vcpu *vcpu,
 
 	kvm_make_request(KVM_REQ_EVENT, vcpu);
 
+	WARN_ON(nr > 0x1f);
 	if (!vcpu->arch.exception.pending) {
 	queue:
 		vcpu->arch.exception.pending = true;

Paolo

Re: Nested paging in nested SVM setup

[Valentine Sinitsyn]

Hi Paolo,

On 20.08.2014 12:55, Paolo Bonzini wrote:
> Is the 0x23c always the same?
No, it's just a garbage - I've seen other values as well (0x80 last time).

>  Can you try this patch?
Sure. It does print a warning:

[ 2176.722098] ------------[ cut here ]------------
[ 2176.722118] WARNING: CPU: 0 PID: 1488 at 
/home/val/kvm-kmod/x86/x86.c:368 kvm_multiple_exception+0x121/0x130 [kvm]()
[ 2176.722121] Modules linked in: kvm_amd(O) kvm(O) amd_freq_sensitivity 
snd_hda_codec_realtek snd_hda_codec_hdmi snd_hda_codec_generic 
crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel 
snd_hda_intel aesni_intel snd_hda_controller radeon snd_hda_codec 
ipmi_si aes_x86_64 ipmi_msghandler snd_hwdep ttm r8169 ppdev mii lrw 
gf128mul snd_pcm glue_helper drm_kms_helper snd_timer fam15h_power evdev 
drm shpchp snd ablk_helper cryptd microcode mac_hid soundcore serio_raw 
pcspkr i2c_algo_bit k10temp i2c_piix4 i2c_core parport_pc parport hwmon 
edac_core tpm_tis edac_mce_amd tpm video button acpi_cpufreq processor 
ext4 crc16 mbcache jbd2 sd_mod crc_t10dif crct10dif_common atkbd libps2 
ahci libahci ohci_pci ohci_hcd ehci_pci xhci_hcd libata ehci_hcd usbcore 
scsi_mod usb_common i8042 serio [last unloaded: kvm]

[ 2176.722217] CPU: 0 PID: 1488 Comm: qemu-system-x86 Tainted: G 
W  O  3.16.1-1-ARCH #1
[ 2176.722221] Hardware name: To Be Filled By O.E.M. To Be Filled By 
O.E.M./IMB-A180, BIOS L0.17 05/24/2013
[ 2176.722224]  0000000000000000 0000000025350f51 ffff8800919fbbc0 
ffffffff8152ae6c
[ 2176.722229]  0000000000000000 ffff8800919fbbf8 ffffffff8106e45d 
ffff880037f68000
[ 2176.722234]  0000000000000080 0000000000000001 00000000000081a4 
0000000000000000
[ 2176.722239] Call Trace:
[ 2176.722250]  [<ffffffff8152ae6c>] dump_stack+0x4d/0x6f
[ 2176.722257]  [<ffffffff8106e45d>] warn_slowpath_common+0x7d/0xa0
[ 2176.722262]  [<ffffffff8106e58a>] warn_slowpath_null+0x1a/0x20
[ 2176.722275]  [<ffffffffa0651e41>] kvm_multiple_exception+0x121/0x130 
[kvm]
[ 2176.722288]  [<ffffffffa06594f8>] x86_emulate_instruction+0x548/0x640 
[kvm]
[ 2176.722303]  [<ffffffffa06653e1>] kvm_mmu_page_fault+0x91/0xf0 [kvm]
[ 2176.722310]  [<ffffffffa04eb6a7>] pf_interception+0xd7/0x180 [kvm_amd]
[ 2176.722317]  [<ffffffff8104e876>] ? native_apic_mem_write+0x6/0x10
[ 2176.722323]  [<ffffffffa04ef261>] handle_exit+0x141/0x9d0 [kvm_amd]
[ 2176.722335]  [<ffffffffa065512c>] ? kvm_set_cr8+0x1c/0x20 [kvm]
[ 2176.722341]  [<ffffffffa04ea3e0>] ? nested_svm_get_tdp_cr3+0x20/0x20 
[kvm_amd]
[ 2176.722355]  [<ffffffffa065adc7>] 
kvm_arch_vcpu_ioctl_run+0x597/0x1210 [kvm]
[ 2176.722368]  [<ffffffffa065705b>] ? kvm_arch_vcpu_load+0xbb/0x200 [kvm]
[ 2176.722378]  [<ffffffffa064a152>] kvm_vcpu_ioctl+0x2b2/0x5c0 [kvm]
[ 2176.722384]  [<ffffffff810b66b4>] ? __wake_up+0x44/0x50
[ 2176.722390]  [<ffffffff81200dcc>] ? fsnotify+0x28c/0x370
[ 2176.722397]  [<ffffffff811d4a70>] do_vfs_ioctl+0x2d0/0x4b0
[ 2176.722403]  [<ffffffff811df18e>] ? __fget+0x6e/0xb0
[ 2176.722408]  [<ffffffff811d4cd1>] SyS_ioctl+0x81/0xa0
[ 2176.722414]  [<ffffffff81530be9>] system_call_fastpath+0x16/0x1b
[ 2176.722418] ---[ end trace b0f81744c5a5ea4a ]---

Thanks,
Valentine

Re: Nested paging in nested SVM setup

[Paolo Bonzini]

Il 20/08/2014 09:37, Valentine Sinitsyn ha scritto:
> Hi Paolo,
> 
> On 20.08.2014 12:55, Paolo Bonzini wrote:
>> Is the 0x23c always the same?
> No, it's just a garbage - I've seen other values as well (0x80 last time).
> 
>>  Can you try this patch?
> Sure. It does print a warning:
> 
> [ 2176.722098] ------------[ cut here ]------------
> [ 2176.722118] WARNING: CPU: 0 PID: 1488 at
> /home/val/kvm-kmod/x86/x86.c:368 kvm_multiple_exception+0x121/0x130 [kvm]()
> [ 2176.722121] Modules linked in: kvm_amd(O) kvm(O) amd_freq_sensitivity
> snd_hda_codec_realtek snd_hda_codec_hdmi snd_hda_codec_generic
> crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel
> snd_hda_intel aesni_intel snd_hda_controller radeon snd_hda_codec
> ipmi_si aes_x86_64 ipmi_msghandler snd_hwdep ttm r8169 ppdev mii lrw
> gf128mul snd_pcm glue_helper drm_kms_helper snd_timer fam15h_power evdev
> drm shpchp snd ablk_helper cryptd microcode mac_hid soundcore serio_raw
> pcspkr i2c_algo_bit k10temp i2c_piix4 i2c_core parport_pc parport hwmon
> edac_core tpm_tis edac_mce_amd tpm video button acpi_cpufreq processor
> ext4 crc16 mbcache jbd2 sd_mod crc_t10dif crct10dif_common atkbd libps2
> ahci libahci ohci_pci ohci_hcd ehci_pci xhci_hcd libata ehci_hcd usbcore
> scsi_mod usb_common i8042 serio [last unloaded: kvm]
> 
> [ 2176.722217] CPU: 0 PID: 1488 Comm: qemu-system-x86 Tainted: G W  O 
> 3.16.1-1-ARCH #1
> [ 2176.722221] Hardware name: To Be Filled By O.E.M. To Be Filled By
> O.E.M./IMB-A180, BIOS L0.17 05/24/2013
> [ 2176.722224]  0000000000000000 0000000025350f51 ffff8800919fbbc0
> ffffffff8152ae6c
> [ 2176.722229]  0000000000000000 ffff8800919fbbf8 ffffffff8106e45d
> ffff880037f68000
> [ 2176.722234]  0000000000000080 0000000000000001 00000000000081a4
> 0000000000000000
> [ 2176.722239] Call Trace:
> [ 2176.722250]  [<ffffffff8152ae6c>] dump_stack+0x4d/0x6f
> [ 2176.722257]  [<ffffffff8106e45d>] warn_slowpath_common+0x7d/0xa0
> [ 2176.722262]  [<ffffffff8106e58a>] warn_slowpath_null+0x1a/0x20
> [ 2176.722275]  [<ffffffffa0651e41>] kvm_multiple_exception+0x121/0x130
> [kvm]
> [ 2176.722288]  [<ffffffffa06594f8>] x86_emulate_instruction+0x548/0x640
> [kvm]
> [ 2176.722303]  [<ffffffffa06653e1>] kvm_mmu_page_fault+0x91/0xf0 [kvm]
> [ 2176.722310]  [<ffffffffa04eb6a7>] pf_interception+0xd7/0x180 [kvm_amd]
> [ 2176.722317]  [<ffffffff8104e876>] ? native_apic_mem_write+0x6/0x10
> [ 2176.722323]  [<ffffffffa04ef261>] handle_exit+0x141/0x9d0 [kvm_amd]
> [ 2176.722335]  [<ffffffffa065512c>] ? kvm_set_cr8+0x1c/0x20 [kvm]
> [ 2176.722341]  [<ffffffffa04ea3e0>] ? nested_svm_get_tdp_cr3+0x20/0x20
> [kvm_amd]
> [ 2176.722355]  [<ffffffffa065adc7>]
> kvm_arch_vcpu_ioctl_run+0x597/0x1210 [kvm]
> [ 2176.722368]  [<ffffffffa065705b>] ? kvm_arch_vcpu_load+0xbb/0x200 [kvm]
> [ 2176.722378]  [<ffffffffa064a152>] kvm_vcpu_ioctl+0x2b2/0x5c0 [kvm]
> [ 2176.722384]  [<ffffffff810b66b4>] ? __wake_up+0x44/0x50
> [ 2176.722390]  [<ffffffff81200dcc>] ? fsnotify+0x28c/0x370
> [ 2176.722397]  [<ffffffff811d4a70>] do_vfs_ioctl+0x2d0/0x4b0
> [ 2176.722403]  [<ffffffff811df18e>] ? __fget+0x6e/0xb0
> [ 2176.722408]  [<ffffffff811d4cd1>] SyS_ioctl+0x81/0xa0
> [ 2176.722414]  [<ffffffff81530be9>] system_call_fastpath+0x16/0x1b
> [ 2176.722418] ---[ end trace b0f81744c5a5ea4a ]---
> 
> Thanks,
> Valentine
> -- 
> To unsubscribe from this list: send the line "unsubscribe kvm" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 

I audited the various places that return X86EMUl_PROPAGATE_FAULT and
I think the culprit is this code in paging_tmpl.h.

 	real_gpa = mmu->translate_gpa(vcpu, gfn_to_gpa(gfn), access);
	if (real_gpa == UNMAPPED_GVA)
 		return 0;

It returns zero without setting fault.vector.

Another patch...  I will post parts of it separately, if I am right
you should get 0xfe as the vector and a WARN from the gva_to_gpa function.

diff --git a/arch/x86/kvm/emulate.c b/arch/x86/kvm/emulate.c
index ef297919a691..e5bf13003cd2 100644
--- a/arch/x86/kvm/emulate.c
+++ b/arch/x86/kvm/emulate.c
@@ -527,6 +527,7 @@ static unsigned long seg_base(struct x86_emulate_ctxt *ctxt, int seg)
 static int emulate_exception(struct x86_emulate_ctxt *ctxt, int vec,
 			     u32 error, bool valid)
 {
+	WARN_ON(vec > 0x1f);
 	ctxt->exception.vector = vec;
 	ctxt->exception.error_code = error;
 	ctxt->exception.error_code_valid = valid;
@@ -3016,7 +3015,7 @@ static int em_movbe(struct x86_emulate_ctxt *ctxt)
 		ctxt->dst.val = swab64(ctxt->src.val);
 		break;
 	default:
-		return X86EMUL_PROPAGATE_FAULT;
+		BUG();
 	}
 	return X86EMUL_CONTINUE;
 }
@@ -4829,8 +4828,10 @@ writeback:
 	ctxt->eip = ctxt->_eip;
 
 done:
-	if (rc == X86EMUL_PROPAGATE_FAULT)
+	if (rc == X86EMUL_PROPAGATE_FAULT) {
+		WARN_ON(ctxt->exception.vector > 0x1f);
 		ctxt->have_exception = true;
+	}
 	if (rc == X86EMUL_INTERCEPTED)
 		return EMULATION_INTERCEPTED;
 
diff --git a/arch/x86/kvm/paging_tmpl.h b/arch/x86/kvm/paging_tmpl.h
index 410776528265..cd91d03c9320 100644
--- a/arch/x86/kvm/paging_tmpl.h
+++ b/arch/x86/kvm/paging_tmpl.h
@@ -365,8 +365,10 @@ retry_walk:
 		gfn += pse36_gfn_delta(pte);
 
 	real_gpa = mmu->translate_gpa(vcpu, gfn_to_gpa(gfn), access);
-	if (real_gpa == UNMAPPED_GVA)
+	if (real_gpa == UNMAPPED_GVA) {
+		walker->fault.vector = 0xfe;
 		return 0;
+	}
 
 	walker->gfn = real_gpa >> PAGE_SHIFT;
 
@@ -875,8 +877,10 @@ static gpa_t FNAME(gva_to_gpa)(struct kvm_vcpu *vcpu, gva_t vaddr, u32 access,
 	if (r) {
 		gpa = gfn_to_gpa(walker.gfn);
 		gpa |= vaddr & ~PAGE_MASK;
-	} else if (exception)
+	} else if (exception) {
+		WARN_ON(walker.fault.vector > 0x1f);
 		*exception = walker.fault;
+	}
 
 	return gpa;
 }
diff --git a/arch/x86/kvm/x86.c b/arch/x86/kvm/x86.c
index 737b4bdac41c..71f05585894e 100644
--- a/arch/x86/kvm/x86.c
+++ b/arch/x86/kvm/x86.c
@@ -5248,6 +5249,7 @@ int x86_emulate_instruction(struct kvm_vcpu *vcpu,
 
 		ctxt->interruptibility = 0;
 		ctxt->have_exception = false;
+		ctxt->exception.vector = 0xff;
 		ctxt->perm_ok = false;
 
 		ctxt->ud = emulation_type & EMULTYPE_TRAP_UD;

Re: Nested paging in nested SVM setup

[Valentine Sinitsyn]

On 20.08.2014 14:11, Paolo Bonzini wrote:
> Another patch...  I will post parts of it separately, if I am right
> you should get 0xfe as the vector and a WARN from the gva_to_gpa function.
I confirm the vector is 0xfe, however I see no warnings from 
gva_to_gpa() - only from emulate_exception():

> [ 3417.251967] ------------[ cut here ]------------
> [ 3417.251983] WARNING: CPU: 1 PID: 1584 at /home/val/kvm-kmod/x86/emulate.c:4839 x86_emulate_insn+0xb33/0xb70 [kvm]()

I can see both warnings, if I move 'WARN(walker.fault.vector > 0x1f)' 
from gva_to_gpa() to gva_to_gpa_nested(), however:

> [ 3841.420019] WARNING: CPU: 0 PID: 1945 at /home/val/kvm-kmod/x86/paging_tmpl.h:903 paging64_gva_to_gpa_nested+0xd1/0xe0 [kvm]()
> [ 3841.420457] WARNING: CPU: 0 PID: 1945 at /home/val/kvm-kmod/x86/emulate.c:4839 x86_emulate_insn+0xb33/0xb70 [kvm]()

Thanks,
Valentine

Re: Nested paging in nested SVM setup

[Valentine Sinitsyn]

Hi all,

On 20.08.2014 14:11, Paolo Bonzini wrote:
> Another patch...  I will post parts of it separately, if I am right
> you should get 0xfe as the vector and a WARN from the gva_to_gpa function.

With the patch like this:

> diff --git a/arch/x86/kvm/paging_tmpl.h b/arch/x86/kvm/paging_tmpl.h
> index 410776528265..cd91d03c9320 100644
> --- a/arch/x86/kvm/paging_tmpl.h
> +++ b/arch/x86/kvm/paging_tmpl.h
> @@ -365,8 +365,10 @@ retry_walk:
>  		gfn += pse36_gfn_delta(pte);
>
>  	real_gpa = mmu->translate_gpa(vcpu, gfn_to_gpa(gfn), access);
> 	if (real_gpa == UNMAPPED_GVA)
 > - 		return 0;
 > + 		goto error;
>
>  	walker->gfn = real_gpa >> PAGE_SHIFT;

KVM seems to work properly (no weird exceptions injected), although my 
code now freezes (quick look on the trace suggests it's looping reading 
APIC). Not sure whose bug is it, will look further.

Thanks for the help.

Valentine

Re: Nested paging in nested SVM setup

[Valentine Sinitsyn]

On 21.08.2014 12:28, Valentine Sinitsyn wrote:
> KVM seems to work properly (no weird exceptions injected), although my
> code now freezes (quick look on the trace suggests it's looping reading
> APIC). Not sure whose bug is it, will look further.
Looks like the problem is that if nested page tables maps some GPA to 
0xfee00000 HPA, it's really mapped to this HPA, and not intercepted with 
KVM's virtual LAPIC implementation. Consider the following trace:

>  qemu-system-x86-344   [000]   644.974072: kvm_entry:            vcpu 0
>  qemu-system-x86-344   [000]   644.974075: kvm_exit:             reason npf rip 0xffffffff8104e883 info 10000000d fee000f0
>  qemu-system-x86-344   [000]   644.974075: kvm_page_fault:       address fee000f0 error_code d
>  qemu-system-x86-344   [000]   644.974077: kvm_emulate_insn:     0:ffffffff8104e883:8b 87 00 b0 5f ff (prot64)
>  qemu-system-x86-344   [000]   644.974078: kvm_apic:             apic_read APIC_SPIV = 0xf
>  qemu-system-x86-344   [000]   644.974079: kvm_mmio:             mmio read len 4 gpa 0xfee000f0 val 0x72007200000000f
>  qemu-system-x86-344   [000]   644.974081: kvm_entry:            vcpu 0
Here, I set up NPT so that any access to 0xfee00000 nested guest 
physical address cause VM exit. Then, my code writes or reads register 
that is mapped to 0xfee00000 KVM's GPA. kvm_apic is called, and 
everything works as expected.

However, if I set up NTP to make 0xfee00000 nested guest physical 
address reads don't cause nested VM exit (by simply clearing U/S flag in 
the NPTE), I get:

>  qemu-system-x86-1066  [003]  1105.864286: kvm_exit:             reason npf rip 0xffffffff8104eaa4 info 10000000f fee00310
>  qemu-system-x86-1066  [003]  1105.864287: kvm_nested_vmexit:    rip: 0xffffffff8104eaa4 reason: npf ext_inf1: 0x000000010000000f ext_inf2: 0x00000000fee00310 ext_int: 0x00000000 ext_int_err: 0x00000000
>  qemu-system-x86-1066  [003]  1105.864287: kvm_page_fault:       address fee00310 error_code f
>  qemu-system-x86-1064  [001]  1105.864288: kvm_exit:             reason npf rip 0xffffffff8104e876 info 10000000f fee000b0
>  qemu-system-x86-1066  [003]  1105.864289: kvm_emulate_insn:     0:ffffffff8104eaa4:89 14 25 10 b3 5f ff (prot64)
>  qemu-system-x86-1064  [001]  1105.864289: kvm_nested_vmexit:    rip: 0xffffffff8104e876 reason: npf ext_inf1: 0x000000010000000f ext_inf2: 0x00000000fee000b0 ext_int: 0x00000000 ext_int_err: 0x00000000
>  qemu-system-x86-1064  [001]  1105.864289: kvm_page_fault:       address fee000b0 error_code f
>  qemu-system-x86-1064  [001]  1105.864291: kvm_emulate_insn:     0:ffffffff8104e876:89 b7 00 b0 5f ff (prot64)
>  qemu-system-x86-1066  [003]  1105.864292: kvm_inj_exception:    e (0x2)
>  qemu-system-x86-1066  [003]  1105.864293: kvm_entry:            vcpu 3
>  qemu-system-x86-1064  [001]  1105.864294: kvm_inj_exception:    e (0x2)
>  qemu-system-x86-1064  [001]  1105.864295: kvm_entry:            vcpu 1

No kvm_apic: after NPTs are set up, no page faults caused by register 
read (error_code: d), to trap and emulate APIC access.

So I'm returning to my original question: is this an intended behavior 
of KVM that APIC access on nested page tables level are not trapped, or 
is this a bug?

Valentine

Re: Nested paging in nested SVM setup

[Paolo Bonzini]

Il 21/08/2014 10:48, Valentine Sinitsyn ha scritto:
> So I'm returning to my original question: is this an intended behavior
> of KVM that APIC access on nested page tables level are not trapped, or
> is this a bug?

I think it's just a bug.  Nobody thought that you'd let L2 access L1's
APIC via NPT.  Let me think of how to fix it.

Paolo

Re: Nested paging in nested SVM setup

[Jan Kiszka]

On 2014-08-21 13:04, Paolo Bonzini wrote:
> Il 21/08/2014 10:48, Valentine Sinitsyn ha scritto:
>> So I'm returning to my original question: is this an intended behavior
>> of KVM that APIC access on nested page tables level are not trapped, or
>> is this a bug?
> 
> I think it's just a bug.  Nobody thought that you'd let L2 access L1's
> APIC via NPT.  Let me think of how to fix it.

Do you think it would only affect the APIC, or could it cause troubles
with other pass-through devices as well (some PCI BAR e.g.)?

Jan

-- 
Siemens AG, Corporate Technology, CT RTC ITP SES-DE
Corporate Competence Center Embedded Linux

Re: Nested paging in nested SVM setup

[Valentine Sinitsyn]

On 21.08.2014 17:06, Jan Kiszka wrote:
> Do you think it would only affect the APIC, or could it cause troubles
> with other pass-through devices as well (some PCI BAR e.g.)?
I've skimmed the KVM sources only quickly, but if feel there is nothing 
APIC-specific in nested paging code. I.e. access to any MMIO range 
mapped by nested page tables the way I did will not be trapped by KVM.

Valentine

Re: Nested paging in nested SVM setup

[Valentine Sinitsyn]

On 21.08.2014 17:04, Paolo Bonzini wrote:
> I think it's just a bug.  Nobody thought that you'd let L2 access L1's
Sure, this is by no means a common use case. However can be seen as a 
flaw that lets the malicious guest to affects others by mapping and 
reprogramming APICs or other.

Valentine

Re: Nested paging in nested SVM setup

[Paolo Bonzini]

Il 21/08/2014 10:48, Valentine Sinitsyn ha scritto:
> 
> No kvm_apic: after NPTs are set up, no page faults caused by register
> read (error_code: d), to trap and emulate APIC access.

It seems to work for VMX (see the testcase I just sent).  For SVM, can you
check if this test works for you, so that we can work on a simple testcase?

The patch applies to git://git.kernel.org/pub/scm/virt/kvm/kvm-unit-tests.git
and you can run the test like this (64-bit host):

   ./configure
   make
   ./x86-run x86/svm.flat -cpu host

Paolo

diff --git a/x86/svm.c b/x86/svm.c
index a9b29b1..aff00da 100644
--- a/x86/svm.c
+++ b/x86/svm.c
@@ -797,6 +797,27 @@ static bool npt_pfwalk_check(struct test *test)
 	   && (test->vmcb->control.exit_info_2 == read_cr3());
 }
 
+static void npt_l1mmio_prepare(struct test *test)
+{
+    vmcb_ident(test->vmcb);
+}
+
+u32 nested_apic_version;
+
+static void npt_l1mmio_test(struct test *test)
+{
+    u64 *data = (void*)(0xfee00030UL);
+
+    nested_apic_version = *data;
+}
+
+static bool npt_l1mmio_check(struct test *test)
+{
+    u64 *data = (void*)(0xfee00030);
+
+    return (nested_apic_version == *data);
+}
+
 static void latency_prepare(struct test *test)
 {
     default_prepare(test);
@@ -962,6 +983,8 @@ static struct test tests[] = {
 	    default_finished, npt_rw_check },
     { "npt_pfwalk", npt_supported, npt_pfwalk_prepare, null_test,
 	    default_finished, npt_pfwalk_check },
+    { "npt_l1mmio", npt_supported, npt_l1mmio_prepare, npt_l1mmio_test,
+	    default_finished, npt_l1mmio_check },
     { "latency_run_exit", default_supported, latency_prepare, latency_test,
       latency_finished, latency_check },
     { "latency_svm_insn", default_supported, lat_svm_insn_prepare, null_test,

Re: Nested paging in nested SVM setup

[Valentine Sinitsyn]

On 21.08.2014 17:24, Paolo Bonzini wrote:
> It seems to work for VMX (see the testcase I just sent).  For SVM, can you
> check if this test works for you, so that we can work on a simple testcase?
It passes for SVM, too.

However, npt_rsvd seems to be broken - maybe that is the reason?

Also, I tried to use different register values for npt_l1mmio_test() and 
npt_l1mmio_check() (like 0xfee00030 and 0xfee00400), but got test passed 
as well. Could it be a false positive then?

> qemu-system-x86_64 -enable-kvm -device pc-testdev -device isa-debug-exit,iobase=0xf4,iosize=0x4 -display none -serial stdio -device pci-testdev -kernel x86/svm.flat -cpu host
> enabling apic
> paging enabled
> cr0 = 80010011
> cr3 = 7fff000
> cr4 = 20
> NPT detected - running all tests with NPT enabled
> null: PASS
> vmrun: PASS
> ioio: PASS
> vmrun intercept check: PASS
> cr3 read intercept: PASS
> cr3 read nointercept: PASS
> next_rip: PASS
> mode_switch: PASS
> asid_zero: PASS
> sel_cr0_bug: PASS
> npt_nx: PASS
> npt_us: PASS
> npt_rsvd: FAIL
> npt_rw: PASS
> npt_pfwalk: PASS
> npt_l1mmio: PASS
>     Latency VMRUN : max: 93973 min: 22447 avg: 22766
>     Latency VMEXIT: max: 428760 min: 23039 avg: 23832
> latency_run_exit: PASS
>     Latency VMLOAD: max: 35697 min: 3828 avg: 3937
>     Latency VMSAVE: max: 42953 min: 3889 avg: 4012
>     Latency STGI:   max: 42961 min: 3517 avg: 3595
>     Latency CLGI:   max: 41177 min: 2859 avg: 2924
> latency_svm_insn: PASS
>
> SUMMARY: 18 TESTS, 1 FAILURES
> Return value from qemu: 3

Valentine

Re: Nested paging in nested SVM setup

[Valentine Sinitsyn]

On 21.08.2014 18:28, Valentine Sinitsyn wrote:
> Also, I tried to use different register values for npt_l1mmio_test() and
> npt_l1mmio_check() (like 0xfee00030 and 0xfee00400), but got test passed
Just a small clarification: I made npt_l1mmio_test() to read 0xfee00030 
and npt_l1mmio_check() to compare against 0xfee00020 or 0xfee00400. No 
reason, just arbitrary values to check if anything compares non-equal in 
the check.

Valentine

Re: Nested paging in nested SVM setup

[Valentine Sinitsyn]

Sorry for the chain letters.

On 21.08.2014 18:28, Valentine Sinitsyn wrote:
> It passes for SVM, too.
I also looked at SVM tests more closely, and found out that NPT maps the 
whole memory-range as cached memory. This can also be a reason for a 
false positive in the test (if there is one). Will look into it later today.

Valentine

Re: Nested paging in nested SVM setup

[Valentine Sinitsyn]

On 21.08.2014 17:24, Paolo Bonzini wrote:
> It seems to work for VMX (see the testcase I just sent).  For SVM, can you
> check if this test works for you, so that we can work on a simple testcase?
I was able to reproduce the bug with your testcase when I changed APIC 
register access size (see below). Please check if it fails on VMX as 
well now.

On a side note, npt_rsvd also seem to be broken as I mentioned previously.

HTH,
Valentine

diff --git a/x86/svm.c b/x86/svm.c
index a9b29b1..d0ddff7 100644
--- a/x86/svm.c
+++ b/x86/svm.c
@@ -797,6 +797,27 @@ static bool npt_pfwalk_check(struct test *test)
  	   && (test->vmcb->control.exit_info_2 == read_cr3());
  }

+static void npt_l1mmio_prepare(struct test *test)
+{
+    vmcb_ident(test->vmcb);
+}
+
+u32 nested_apic_version;
+
+static void npt_l1mmio_test(struct test *test)
+{
+    u32 *data = (u32*)(0xfee00030UL);
+
+    nested_apic_version = *data;
+}
+
+static bool npt_l1mmio_check(struct test *test)
+{
+    u32 *data = (u32*)(0xfee00030);
+
+    return (nested_apic_version == *data);
+}
+
  static void latency_prepare(struct test *test)
  {
      default_prepare(test);
@@ -962,6 +983,8 @@ static struct test tests[] = {
  	    default_finished, npt_rw_check },
      { "npt_pfwalk", npt_supported, npt_pfwalk_prepare, null_test,
  	    default_finished, npt_pfwalk_check },
+    { "npt_l1mmio", npt_supported, npt_l1mmio_prepare, npt_l1mmio_test,
+	    default_finished, npt_l1mmio_check },
      { "latency_run_exit", default_supported, latency_prepare, 
latency_test,
        latency_finished, latency_check },
      { "latency_svm_insn", default_supported, lat_svm_insn_prepare, 
null_test,

Re: Nested paging in nested SVM setup

[Paolo Bonzini]

Il 21/08/2014 19:35, Valentine Sinitsyn ha scritto:
>>
> I was able to reproduce the bug with your testcase when I changed APIC
> register access size (see below). Please check if it fails on VMX as
> well now.

VMX used the right access size already, the tests are separate for VMX
and SVM.

> On a side note, npt_rsvd also seem to be broken as I mentioned previously.

Yup, thanks.  I think I noticed a month ago or so.

Paolo

Re: Nested paging in nested SVM setup

[Valentine Sinitsyn]

On 22.08.2014 02:31, Paolo Bonzini wrote:
> VMX used the right access size already, the tests are separate for VMX
> and SVM.
Sure. So the bug is NPT-specific?

BTW I was likely wrong stating:

> if nested page tables maps some GPA to 0xfee00000 HPA, it's really mapped to this HPA,

Looks more like it gets mapped wherever 0xfee00000 GPA is translated to. 
No flaw then, just a minor glitch (for a general public).

Valentine

Re: Nested paging in nested SVM setup

[Paolo Bonzini]

Il 22/08/2014 06:33, Valentine Sinitsyn ha scritto:
> On 22.08.2014 02:31, Paolo Bonzini wrote:
>> VMX used the right access size already, the tests are separate for VMX
>> and SVM.
> Sure. So the bug is NPT-specific?

Looks like that, yes.

Paolo

Re: Nested paging in nested SVM setup

[Paolo Bonzini]

Il 22/08/2014 06:33, Valentine Sinitsyn ha scritto:
> On 22.08.2014 02:31, Paolo Bonzini wrote:
>> VMX used the right access size already, the tests are separate for VMX
>> and SVM.
> Sure. So the bug is NPT-specific?

Hmm, unfortunately the test cannot reproduce the bug, at least with 3.16.
It only failed due to a (somewhat unbelievable...) typo:

diff --git a/x86/svm.c b/x86/svm.c
index 54d804b..ca1e64e 100644
--- a/x86/svm.c
+++ b/x86/svm.c
@@ -87,7 +87,7 @@ static void setup_svm(void)
         page = alloc_page();
 
         for (j = 0; j < 512; ++j)
-            page[j] = (u64)pte[(i * 514) + j] | 0x027ULL;
+            page[j] = (u64)pte[(i * 512) + j] | 0x027ULL;
 
         pde[i] = page;
     }

The trace correctly points at APIC_LVR for both the guest read:

 qemu-system-x86-23749 [019]  6718.397998: kvm_exit:             reason npf rip 0x4003ba info 100000004 fee00030
 qemu-system-x86-23749 [019]  6718.397998: kvm_nested_vmexit:    rip: 0x00000000004003ba reason: npf ext_inf1: 0x0000000100000004 ext_inf2: 0x00000000fee00030 ext_int: 0x00000000 ext_int_err: 0x00000000
 qemu-system-x86-23749 [019]  6718.397999: kvm_page_fault:       address fee00030 error_code 4
 qemu-system-x86-23749 [019]  6718.398009: kvm_emulate_insn:     0:4003ba:a1 30 00 e0 fe 00 00 00 00 (prot64)
 qemu-system-x86-23749 [019]  6718.398013: kvm_apic:             apic_read APIC_LVR = 0x1050014
 qemu-system-x86-23749 [019]  6718.398014: kvm_mmio:             mmio read len 4 gpa 0xfee00030 val 0x1050014
 qemu-system-x86-23749 [019]  6718.398015: kvm_entry:            vcpu 0

and the host read:

 qemu-system-x86-23749 [019]  6718.398035: kvm_entry:            vcpu 0
 qemu-system-x86-23749 [019]  6718.398036: kvm_exit:             reason npf rip 0x4003ca info 10000000d fee00030
 qemu-system-x86-23749 [019]  6718.398037: kvm_page_fault:       address fee00030 error_code d
 qemu-system-x86-23749 [019]  6718.398039: kvm_emulate_insn:     0:4003ca:a1 30 00 e0 fe 00 00 00 00 (prot64)
 qemu-system-x86-23749 [019]  6718.398040: kvm_apic:             apic_read APIC_LVR = 0x1050014
 qemu-system-x86-23749 [019]  6718.398040: kvm_mmio:             mmio read len 4 gpa 0xfee00030 val 0x1050014

The different error codes are because the first read will install the shadow
page.  If I change the test to do two reads, the error codes match.  I will
look at this more closely tomorrow.

Paolo

Re: Nested paging in nested SVM setup

[Paolo Bonzini]

Il 20/08/2014 08:46, Valentine Sinitsyn ha scritto:
> Looks like it is a bug in KVM. I had a chance to run the same code
> bare-metal ([1], line 310 is uncommented for bare-metal case but present
> for nested SVM), and it seems to work as expected. However, When I trace
> it in nested SVM setup, after some successful APIC reads and writes, I
> get the following:

Valentine, can you produce another trace, this time with both kvm and
kvmmmu events enabled?

Thanks,

Paolo

Re: Nested paging in nested SVM setup

[Paolo Bonzini]

Il 21/08/2014 14:28, Valentine Sinitsyn ha scritto:
>> It seems to work for VMX (see the testcase I just sent).  For SVM, can
>> you check if this test works for you, so that we can work on a simple
>> testcase?
> 
> However, npt_rsvd seems to be broken - maybe that is the reason?

BTW npt_rsvd does *not* fail on the machine I've been testing on today.

Can you retry running the tests with the latest kvm-unit-tests (branch
"master"), gather a trace of kvm and kvmmmu events, and send the
compressed trace.dat my way?

Thanks,

Paolo

Re: Nested paging in nested SVM setup

[Valentine Sinitsyn]

Hi Paolo,

On 01.09.2014 23:41, Paolo Bonzini wrote:
> Il 21/08/2014 14:28, Valentine Sinitsyn ha scritto:
> BTW npt_rsvd does *not* fail on the machine I've been testing on today.
I can confirm l1mmio test doesn't fail in kvm-unit-test's master 
anymore. npt_rsvd still does. I also needed to disable ioio test, or it 
was hanging for a long time (this doesn't happen if I use Jan's patched 
KVM that have IOPM bugs fixed). However, l1mmio test passes regardless I 
use stock kvm 3.16.1 or a patched version.

> Can you retry running the tests with the latest kvm-unit-tests (branch
> "master"), gather a trace of kvm and kvmmmu events, and send the
> compressed trace.dat my way?
You mean the trace when the problem reveal itself (not from running 
tests), I assume? It's around 2G uncompressed (probably I'm enabling 
tracing to early or doing anything else wrong). Will look into it 
tomorrow, hopefully, I can reduce the size (e.g. by switching to 
uniprocessor mode). Below is a trace snippet similar to the one I've 
sent earlier.

----------------------------------------------------------------------
qemu-system-x86-2728  [002]  1726.426225: kvm_exit:             reason 
npf rip 0xffffffff8104e876 info 10000000f fee000b0
  qemu-system-x86-2728  [002]  1726.426226: kvm_nested_vmexit:    rip: 
0xffffffff8104e876 reason: npf ext_inf1: 0x000000010000000f ext_inf2: 
0x00000000fee000b0 ext_int: 0x00000000 ext_int_err: 0x00000000
  qemu-system-x86-2728  [002]  1726.426227: kvm_page_fault: 
address fee000b0 error_code f
  qemu-system-x86-2725  [000]  1726.426227: kvm_exit:             reason 
npf rip 0xffffffff8104e876 info 10000000f fee000b0
  qemu-system-x86-2725  [000]  1726.426228: kvm_nested_vmexit:    rip: 
0xffffffff8104e876 reason: npf ext_inf1: 0x000000010000000f ext_inf2: 
0x00000000fee000b0 ext_int: 0x00000000 ext_int_err: 0x00000000
  qemu-system-x86-2725  [000]  1726.426229: kvm_page_fault: 
address fee000b0 error_code f
  qemu-system-x86-2728  [002]  1726.426229: kvm_emulate_insn: 
0:ffffffff8104e876:89 b7 00 b0 5f ff (prot64)
  qemu-system-x86-2725  [000]  1726.426230: kvm_emulate_insn: 
0:ffffffff8104e876:89 b7 00 b0 5f ff (prot64)
  qemu-system-x86-2728  [002]  1726.426231: kvm_mmu_pagetable_walk: addr 
ffffffffff5fb0b0 pferr 2 W
  qemu-system-x86-2725  [000]  1726.426231: kvm_mmu_pagetable_walk: addr 
ffffffffff5fb0b0 pferr 2 W
  qemu-system-x86-2728  [002]  1726.426231: kvm_mmu_pagetable_walk: addr 
1811000 pferr 6 W|U
  qemu-system-x86-2725  [000]  1726.426232: kvm_mmu_pagetable_walk: addr 
36c49000 pferr 6 W|U
  qemu-system-x86-2728  [002]  1726.426232: kvm_mmu_paging_element: pte 
3c03a027 level 4
  qemu-system-x86-2725  [000]  1726.426232: kvm_mmu_paging_element: pte 
3c03a027 level 4
  qemu-system-x86-2728  [002]  1726.426232: kvm_mmu_paging_element: pte 
3c03d027 level 3
  qemu-system-x86-2725  [000]  1726.426233: kvm_mmu_paging_element: pte 
3c03d027 level 3
  qemu-system-x86-2728  [002]  1726.426233: kvm_mmu_paging_element: pte 
18000e7 level 2
  qemu-system-x86-2725  [000]  1726.426233: kvm_mmu_paging_element: pte 
36c000e7 level 2
  qemu-system-x86-2728  [002]  1726.426233: kvm_mmu_paging_element: pte 
1814067 level 4
  qemu-system-x86-2725  [000]  1726.426233: kvm_mmu_paging_element: pte 
1814067 level 4
  qemu-system-x86-2728  [002]  1726.426233: kvm_mmu_pagetable_walk: addr 
1814000 pferr 6 W|U
  qemu-system-x86-2725  [000]  1726.426234: kvm_mmu_pagetable_walk: addr 
1814000 pferr 6 W|U
  qemu-system-x86-2728  [002]  1726.426234: kvm_mmu_paging_element: pte 
3c03a027 level 4
  qemu-system-x86-2725  [000]  1726.426234: kvm_mmu_paging_element: pte 
3c03a027 level 4
  qemu-system-x86-2728  [002]  1726.426234: kvm_mmu_paging_element: pte 
3c03d027 level 3
  qemu-system-x86-2725  [000]  1726.426235: kvm_mmu_paging_element: pte 
3c03d027 level 3
  qemu-system-x86-2728  [002]  1726.426235: kvm_mmu_paging_element: pte 
18000e7 level 2
  qemu-system-x86-2725  [000]  1726.426235: kvm_mmu_paging_element: pte 
18000e7 level 2
  qemu-system-x86-2728  [002]  1726.426235: kvm_mmu_paging_element: pte 
1816067 level 3
  qemu-system-x86-2725  [000]  1726.426235: kvm_mmu_paging_element: pte 
1816067 level 3
  qemu-system-x86-2728  [002]  1726.426235: kvm_mmu_pagetable_walk: addr 
1816000 pferr 6 W|U
  qemu-system-x86-2725  [000]  1726.426236: kvm_mmu_pagetable_walk: addr 
1816000 pferr 6 W|U
  qemu-system-x86-2728  [002]  1726.426236: kvm_mmu_paging_element: pte 
3c03a027 level 4
  qemu-system-x86-2725  [000]  1726.426236: kvm_mmu_paging_element: pte 
3c03a027 level 4
  qemu-system-x86-2728  [002]  1726.426236: kvm_mmu_paging_element: pte 
3c03d027 level 3
  qemu-system-x86-2725  [000]  1726.426236: kvm_mmu_paging_element: pte 
3c03d027 level 3
  qemu-system-x86-2728  [002]  1726.426236: kvm_mmu_paging_element: pte 
18000e7 level 2
  qemu-system-x86-2725  [000]  1726.426237: kvm_mmu_paging_element: pte 
18000e7 level 2
  qemu-system-x86-2728  [002]  1726.426237: kvm_mmu_paging_element: pte 
1a06067 level 2
  qemu-system-x86-2725  [000]  1726.426237: kvm_mmu_paging_element: pte 
1a06067 level 2
  qemu-system-x86-2725  [000]  1726.426238: kvm_mmu_pagetable_walk: addr 
1a06000 pferr 6 W|U
  qemu-system-x86-2728  [002]  1726.426238: kvm_mmu_pagetable_walk: addr 
1a06000 pferr 6 W|U
  qemu-system-x86-2725  [000]  1726.426238: kvm_mmu_paging_element: pte 
3c03a027 level 4
  qemu-system-x86-2728  [002]  1726.426238: kvm_mmu_paging_element: pte 
3c03a027 level 4
  qemu-system-x86-2725  [000]  1726.426238: kvm_mmu_paging_element: pte 
3c03d027 level 3
  qemu-system-x86-2725  [000]  1726.426239: kvm_mmu_paging_element: pte 
1a000e7 level 2
  qemu-system-x86-2728  [002]  1726.426239: kvm_mmu_paging_element: pte 
3c03d027 level 3
  qemu-system-x86-2725  [000]  1726.426239: kvm_mmu_paging_element: pte 
80000000fee0017b level 1
  qemu-system-x86-2728  [002]  1726.426239: kvm_mmu_paging_element: pte 
1a000e7 level 2
  qemu-system-x86-2725  [000]  1726.426239: kvm_mmu_pagetable_walk: addr 
fee00000 pferr 6 W|U
  qemu-system-x86-2728  [002]  1726.426239: kvm_mmu_paging_element: pte 
80000000fee0017b level 1
  qemu-system-x86-2725  [000]  1726.426240: kvm_mmu_paging_element: pte 
3c03a027 level 4
  qemu-system-x86-2728  [002]  1726.426240: kvm_mmu_pagetable_walk: addr 
fee00000 pferr 6 W|U
  qemu-system-x86-2725  [000]  1726.426240: kvm_mmu_paging_element: pte 
3c03b027 level 3
  qemu-system-x86-2728  [002]  1726.426240: kvm_mmu_paging_element: pte 
3c03a027 level 4
  qemu-system-x86-2725  [000]  1726.426240: kvm_mmu_paging_element: pte 
3c03c027 level 2
  qemu-system-x86-2728  [002]  1726.426241: kvm_mmu_paging_element: pte 
3c03b027 level 3
  qemu-system-x86-2725  [000]  1726.426241: kvm_mmu_paging_element: pte 
fee0003d level 1
  qemu-system-x86-2728  [002]  1726.426241: kvm_mmu_paging_element: pte 
3c03c027 level 2
  qemu-system-x86-2725  [000]  1726.426241: kvm_mmu_walker_error: pferr 
7 P|W|U
  qemu-system-x86-2728  [002]  1726.426241: kvm_mmu_paging_element: pte 
fee0003d level 1
  qemu-system-x86-2725  [000]  1726.426241: kvm_mmu_walker_error: pferr 2 W
  qemu-system-x86-2728  [002]  1726.426242: kvm_mmu_walker_error: pferr 
7 P|W|U
  qemu-system-x86-2728  [002]  1726.426242: kvm_mmu_walker_error: pferr 2 W
  qemu-system-x86-2725  [000]  1726.426243: kvm_inj_exception:    e (0x2)
  qemu-system-x86-2728  [002]  1726.426243: kvm_inj_exception:    e (0x2)
  qemu-system-x86-2725  [000]  1726.426244: kvm_entry:            vcpu 0

Thanks,
Valentine

Re: Nested paging in nested SVM setup

[Valentine Sinitsyn]

Hi Paolo,

On 01.09.2014 23:04, Paolo Bonzini wrote:
> Valentine, can you produce another trace, this time with both kvm and
> kvmmmu events enabled?
I was able to make the trace shorter by grepping only what's happening 
on a single CPU core (#0):

https://www.dropbox.com/s/slbxmxyg74wh9hv/l1mmio-cpu0.txt.gz?dl=0

It was taken with kernel 3.16.1 modules with your paging-tmpl.h patch 
applied.

This time, the trace looks somewhat different, however my code still 
hangs in nested KVM (and doesn't on real HW).

Thanks,
Valentine

Re: Nested paging in nested SVM setup

[Valentine Sinitsyn]

On 02.09.2014 12:09, Valentine Sinitsyn wrote:
> https://www.dropbox.com/s/slbxmxyg74wh9hv/l1mmio-cpu0.txt.gz?dl=0
Forgot to say: the user space is vanilla QEMU 2.1.0 here.

Valentine

Re: Nested paging in nested SVM setup

[Paolo Bonzini]

Il 01/09/2014 21:21, Valentine Sinitsyn ha scritto:
> 
>> Can you retry running the tests with the latest kvm-unit-tests (branch
>> "master"), gather a trace of kvm and kvmmmu events, and send the
>> compressed trace.dat my way?
> You mean the trace when the problem reveal itself (not from running
> tests), I assume? It's around 2G uncompressed (probably I'm enabling
> tracing to early or doing anything else wrong). Will look into it
> tomorrow, hopefully, I can reduce the size (e.g. by switching to
> uniprocessor mode). Below is a trace snippet similar to the one I've
> sent earlier.

I actually meant kvm-unit-tests in order to understand the npt_rsvd
failure.  (I had sent a separate message for Jailhouse).

For kvm-unit-tests, you can comment out tests that do not fail to reduce
the trace size.

Paolo

Re: Nested paging in nested SVM setup

[Valentine Sinitsyn]

On 02.09.2014 14:25, Paolo Bonzini wrote:
> I actually meant kvm-unit-tests in order to understand the npt_rsvd
> failure.  (I had sent a separate message for Jailhouse).
Oops, sorry for misunderstanding. Uploaded it here:
https://www.dropbox.com/s/jp6ohb0ul3d6v4u/npt_rsvd.txt.bz2?dl=0

The environment is QEMU 2.1.0 + Linux 3.16.1 with paging_tmpl.h patch, 
and the only test enabled was npt_rsvd (others do pass now).

> For kvm-unit-tests, you can comment out tests that do not fail to reduce
> the trace size.
Yes, I've sent that trace earlier today.

Valentine

Re: Nested paging in nested SVM setup

[Paolo Bonzini]

Il 02/09/2014 08:09, Valentine Sinitsyn ha scritto:
> 
> https://www.dropbox.com/s/slbxmxyg74wh9hv/l1mmio-cpu0.txt.gz?dl=0
> 
> It was taken with kernel 3.16.1 modules with your paging-tmpl.h patch
> applied.
> 
> This time, the trace looks somewhat different, however my code still
> hangs in nested KVM (and doesn't on real HW).

This *is* different though.  I don't see any kvm_inj_exception at all
(with my patch it should be for vector 0xfe).

In any case, the problem seems specific to _writes_ to the APIC page.
I'm going to write a testcase for that and see if I can reproduce it now.

Paolo

Re: Nested paging in nested SVM setup

[Valentine Sinitsyn]

On 02.09.2014 15:45, Paolo Bonzini wrote:
> This *is* different though.  I don't see any kvm_inj_exception at all
> (with my patch it should be for vector 0xfe).
I've applied the part of your patch, that fixes the uninitialized 
exception vector problem, otherwise the lockup will trigger before my 
code will have chance to hang on APIC. Namely, I did the following change:

--- a/arch/x86/kvm/paging_tmpl.h	2014-09-02 21:53:26.035112557 +0600
+++ b/arch/x86/kvm/paging_tmpl.h	2014-09-02 21:53:46.145110721 +0600
@@ -366,7 +366,7 @@

  	real_gpa = mmu->translate_gpa(vcpu, gfn_to_gpa(gfn), access);
  	if (real_gpa == UNMAPPED_GVA)
-		return 0;
+		goto error;

  	walker->gfn = real_gpa >> PAGE_SHIFT;

So they should look like regular page faults (as they ought to, I guess) 
now.

Thanks,
Valentine

Re: Nested paging in nested SVM setup

[Valentine Sinitsyn]

On 02.09.2014 15:45, Paolo Bonzini wrote:
> In any case, the problem seems specific to _writes_ to the APIC page.
> I'm going to write a testcase for that and see if I can reproduce it now.
If you'll need a complete trace, not only CPU 0, please let me know - 
I'll upload it as well. It's about 17M compressed.

Valentine

Re: Nested paging in nested SVM setup

[Paolo Bonzini]

Il 02/09/2014 11:16, Valentine Sinitsyn ha scritto:
> On 02.09.2014 14:25, Paolo Bonzini wrote:
>> I actually meant kvm-unit-tests in order to understand the npt_rsvd
>> failure.  (I had sent a separate message for Jailhouse).
> Oops, sorry for misunderstanding. Uploaded it here:
> https://www.dropbox.com/s/jp6ohb0ul3d6v4u/npt_rsvd.txt.bz2?dl=0

Ugh, there are many bugs and the test is even wrong because the actual
error code should be 0x200000006 (error while visiting page tables).

Paolo

> The environment is QEMU 2.1.0 + Linux 3.16.1 with paging_tmpl.h patch,
> and the only test enabled was npt_rsvd (others do pass now).
> 
>> For kvm-unit-tests, you can comment out tests that do not fail to reduce
>> the trace size.
> Yes, I've sent that trace earlier today.
> 
> Valentine
> -- 
> To unsubscribe from this list: send the line "unsubscribe kvm" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 

Re: Nested paging in nested SVM setup

[Valentine Sinitsyn]

On 02.09.2014 17:21, Paolo Bonzini wrote:
> Ugh, there are many bugs and the test is even wrong because the actual
> error code should be 0x200000006 (error while visiting page tables).
Well, good they were spotted. :-) Haven't looked at the test code 
actually, just saw it fails for some reason.

Valentine

Re: Nested paging in nested SVM setup

[Paolo Bonzini]

Il 02/09/2014 11:53, Valentine Sinitsyn ha scritto:
> 
>      real_gpa = mmu->translate_gpa(vcpu, gfn_to_gpa(gfn), access);
>      if (real_gpa == UNMAPPED_GVA)
> -        return 0;
> +        goto error;
> 
>      walker->gfn = real_gpa >> PAGE_SHIFT;
> 
> So they should look like regular page faults (as they ought to, I guess)
> now.

Yes, they do look like a regular page fault with this patch.  However,
they actually should look like nested page faults...  I'm starting to
clean up this stuff.

Paolo