Re: [Xen-devel] [BUG]Nested virtualization, Xen on KVM, Xen cannot boot up as a guest of KVM

From: Chao Gao <chao.gao@intel.com>
To: "Chen, Farrah" <farrah.chen@intel.com>
Cc: "xen-devel@lists.xenproject.org" <xen-devel@lists.xenproject.org>,
	"Hao, Xudong" <xudong.hao@intel.com>
Subject: Re: [Xen-devel] [BUG]Nested virtualization, Xen on KVM, Xen cannot boot up as a guest of KVM
Date: Wed, 26 Feb 2020 15:37:50 +0800	[thread overview]
Message-ID: <20200226073747.GA1329@gao-cwp> (raw)
In-Reply-To: <1B925CC0CD9F3341B32D442251E7E1DC3E3DBF54@SHSMSX104.ccr.corp.intel.com>

On Wed, Feb 26, 2020 at 02:21:25PM +0800, Chen, Farrah wrote:
>Description:
>
>Nested virtualization, take KVM host as L0, create guest on it, install Xen on
>guest, then guest cannot boot up from Xen and keep rebooting.
>
> 
>
>Reproduce steps:
>1. Enable KVM nested on host(L0)
>rmmod kvm_intel
>modprobe kvm_intel nested=y
>cat /sys/module/kvm_intel/parameters/nested
>Y
>
>2.Create L1 guest via qemu:
>qemu-system-x86_64 -accel kvm -cpu host -drive file=rhel8.img,if=none,id=
>virtio-disk0 -device virtio-blk-pci,drive=virtio-disk0 -m 7168 -smp 8 -monitor
>pty -cpu host -device virtio-net-pci,netdev=nic0,mac=00:16:3e:72:5e:0a -netdev
>tap,id=nic0,br=virbr0,helper=/usr/libexec/qemu-bridge-helper,vhost=on -serial
>stdio
>
>3. Build and install Xen on L1 guest
>
>4. Reboot L1 and make it boot from Xen
>
> 
>
>Then L1 keep rebooting, full log attached.
>
>……
>
>(XEN) Running stub recovery selftests...
>
>(XEN) traps.c:1590: GPF (0000): ffff82d0bfffe041 [ffff82d0bfffe041] ->
>ffff82d08038e40c
>
>(XEN) traps.c:785: Trap 12: ffff82d0bfffe040 [ffff82d0bfffe040] ->
>ffff82d08038e40c
>
>(XEN) traps.c:1124: Trap 3: ffff82d0bfffe041 [ffff82d0bfffe041] ->
>ffff82d08038e40c
>
>(XEN) ----[ Xen-4.14-unstable  x86_64  debug=y   Tainted:  C   ]----
>
>(XEN) CPU:    0
>
>(XEN) RIP:    e008:[<ffff82d08043a8bc>] core2_vpmu_init+0xa5/0x221
>
>(XEN) RFLAGS: 0000000000010202   CONTEXT: hypervisor
>
>(XEN) rax: 0000000008300802   rbx: 0000000000000000   rcx: 0000000000000345
>
>(XEN) rdx: 0000000000000004   rsi: 000000000000000a   rdi: 0000000000000063
>
>(XEN) rbp: ffff82d0804b7d68   rsp: ffff82d0804b7d58   r8:  0000000000000004
>
>(XEN) r9:  0000000000000008   r10: ffff82d0805effe0   r11: 0000000000000032
>
>(XEN) r12: 0000000000000002   r13: 0000000000000008   r14: ffff82d0805dd0c0
>
>(XEN) r15: ffff82d0805de300   cr0: 000000008005003b   cr4: 00000000003526e0
>
>(XEN) cr3: 00000000bfca4000   cr2: 0000000000000000
>
>(XEN) fsb: 0000000000000000   gsb: 0000000000000000   gss: 0000000000000000
>
>(XEN) ds: 0000   es: 0000   fs: 0000   gs: 0000   ss: 0000   cs: e008
>
>(XEN) Xen code around <ffff82d08043a8bc> (core2_vpmu_init+0xa5/0x221):
>
>(XEN)  00 06 00 b9 45 03 00 00 <0f> 32 48 89 c1 48 c1 e9 0d 83 e1 01 88 0d 32
>00

The machine code above shows that #GP is generated when xen is reading
MSR_IA32_PERF_CAPABILITIES. In a KVM guest without Xen, cpuid tells that
perfmon isn't supported.

# ./cpuid -1 |grep "perfmon and debug"
 PDCM: perfmon and debug = false

So, it looks core2_vpmu_init() lacks a check to ensure the MSR is supported.

Thanks
Chao

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel