* [Xen-devel] [BUG]Nested virtualization, Xen on KVM, Xen cannot boot up as a guest of KVM
@ 2020-02-26 6:21 Chen, Farrah
2020-02-26 7:37 ` Chao Gao
2020-02-26 9:19 ` [Xen-devel] [PATCH] x86/vPMU: don't blindly assume IA32_PERF_CAPABILITIES MSR exists Jan Beulich
0 siblings, 2 replies; 8+ messages in thread
From: Chen, Farrah @ 2020-02-26 6:21 UTC (permalink / raw)
To: xen-devel; +Cc: Hao, Xudong, Gao, Chao
[-- Attachment #1.1: Type: text/plain, Size: 4826 bytes --]
Description:
Nested virtualization, take KVM host as L0, create guest on it, install Xen on guest, then guest cannot boot up from Xen and keep rebooting.
Reproduce steps:
1. Enable KVM nested on host(L0)
rmmod kvm_intel
modprobe kvm_intel nested=y
cat /sys/module/kvm_intel/parameters/nested
Y
2.Create L1 guest via qemu:
qemu-system-x86_64 -accel kvm -cpu host -drive file=rhel8.img,if=none,id=virtio-disk0 -device virtio-blk-pci,drive=virtio-disk0 -m 7168 -smp 8 -monitor pty -cpu host -device virtio-net-pci,netdev=nic0,mac=00:16:3e:72:5e:0a -netdev tap,id=nic0,br=virbr0,helper=/usr/libexec/qemu-bridge-helper,vhost=on -serial stdio
3. Build and install Xen on L1 guest
4. Reboot L1 and make it boot from Xen
Then L1 keep rebooting, full log attached.
......
(XEN) Running stub recovery selftests...
(XEN) traps.c:1590: GPF (0000): ffff82d0bfffe041 [ffff82d0bfffe041] -> ffff82d08038e40c
(XEN) traps.c:785: Trap 12: ffff82d0bfffe040 [ffff82d0bfffe040] -> ffff82d08038e40c
(XEN) traps.c:1124: Trap 3: ffff82d0bfffe041 [ffff82d0bfffe041] -> ffff82d08038e40c
(XEN) ----[ Xen-4.14-unstable x86_64 debug=y Tainted: C ]----
(XEN) CPU: 0
(XEN) RIP: e008:[<ffff82d08043a8bc>] core2_vpmu_init+0xa5/0x221
(XEN) RFLAGS: 0000000000010202 CONTEXT: hypervisor
(XEN) rax: 0000000008300802 rbx: 0000000000000000 rcx: 0000000000000345
(XEN) rdx: 0000000000000004 rsi: 000000000000000a rdi: 0000000000000063
(XEN) rbp: ffff82d0804b7d68 rsp: ffff82d0804b7d58 r8: 0000000000000004
(XEN) r9: 0000000000000008 r10: ffff82d0805effe0 r11: 0000000000000032
(XEN) r12: 0000000000000002 r13: 0000000000000008 r14: ffff82d0805dd0c0
(XEN) r15: ffff82d0805de300 cr0: 000000008005003b cr4: 00000000003526e0
(XEN) cr3: 00000000bfca4000 cr2: 0000000000000000
(XEN) fsb: 0000000000000000 gsb: 0000000000000000 gss: 0000000000000000
(XEN) ds: 0000 es: 0000 fs: 0000 gs: 0000 ss: 0000 cs: e008
(XEN) Xen code around <ffff82d08043a8bc> (core2_vpmu_init+0xa5/0x221):
(XEN) 00 06 00 b9 45 03 00 00 <0f> 32 48 89 c1 48 c1 e9 0d 83 e1 01 88 0d 32 00
(XEN) Xen stack trace from rsp=ffff82d0804b7d58:
(XEN) ffff82d080452168 ffff82d080452248 ffff82d0804b7d78 ffff82d08043a622
(XEN) ffff82d0804b7d98 ffff82d08040dcb0 0000000000000008 0000000000000000
(XEN) ffff82d0804b7ee8 ffff82d0804339db 0000000001fb4fff 00000000000001f6
(XEN) ffff83000009de80 ffffffff00000000 000ffff82d0807f6 0000000000000002
(XEN) 0000000000000002 0000000000000002 0000000000000002 0000000000000002
(XEN) 0000000000000001 0000000000000001 0000000000100000 ffff82d08047ac00
(XEN) 0000000000200000 ffff82d08047acdc 0000000100000008 0000000000000007
(XEN) 0000000001fb4000 0000000000000000 0000000000000007 ffff83000009dd20
(XEN) ffff83000009de80 ffff83000009dfb0 0000000000000000 0000000000000000
(XEN) 0000000400000000 0000000000000000 ffffffffffffffff 00000000ffffffff
(XEN) 0000000800000000 000000010000006e 0000000000000003 00000000000002f8
(XEN) 0000000000000002 0000000000000000 0000000000000000 0000000000000000
(XEN) 0000000000000000 0000000000000000 0000000000000000 ffff82d0802000ec
(XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000
(XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000
(XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000
(XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000
(XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000
(XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000
(XEN) 0000000000000000 0000e01000000000 ffff8301f136e000 0000000000000000
(XEN) Xen call trace:
(XEN) [<ffff82d08043a8bc>] R core2_vpmu_init+0xa5/0x221
(XEN) [<ffff82d08043a622>] F arch/x86/cpu/vpmu.c#vpmu_init+0xc4/0x100
(XEN) [<ffff82d08040dcb0>] F do_initcalls+0x35/0x44
(XEN) [<ffff82d0804339db>] F __start_xen+0x22d7/0x2808
(XEN) [<ffff82d0802000ec>] F __high_start+0x4c/0x4e
(XEN)
(XEN)
(XEN) ****************************************
(XEN) Panic on CPU 0:
(XEN) GENERAL PROTECTION FAULT
(XEN) [error_code=0000]
(XEN) ****************************************
(XEN)
(XEN) Reboot in five seconds...
Environment:
Host(L0):
KVM: https://git.kernel.org/pub/scm/virt/kvm/kvm.git/
Branch: next
Commit: ead68df94d248c80fdbae220ae5425eb5af2e753
Kernel version: 5.6.0_rc1
Qemu: https://git.qemu.org/git/qemu.git
Branch: master
Commit: 88e2b97aa3e369a454c9d8360afddc348070c708
Guest(L1):
Xen: https://xenbits.xen.org/git-http/xen.git
Branch: master
Commit: c47984aabead53918e5ba6d43cdb3f1467452739
Xen version: 4.14-unstable
Host OS(L0): Red Hat Enterprise Linux 8.1
Guest OS(L1): Red Hat Enterprise Linux 8.1
Thanks,
Fan
[-- Attachment #1.2: Type: text/html, Size: 16918 bytes --]
[-- Attachment #2: Boot_Xen_log.txt --]
[-- Type: text/plain, Size: 10491 bytes --]
[ 39.853943] kvm: exiting hardware virtualization
[ 39.856259] reboot: Restarting system
[ 39.856706] reboot: machine restart
Xen 4.14-unstable
(XEN) Xen version 4.14-unstable (vmm@sh.intel.com) (gcc (GCC) 8.3.1 20190507 (Red Hat 8.3.1-4)) debug=y Mon Feb 24 01:33:32 CST 2020
(XEN) Latest ChangeSet: Tue Feb 18 16:27:07 2020 +0100 git:c47984aabe-dirty
(XEN) build-id: f7c69059140e4f60be2c02fed5f98b80d87ad642
(XEN) Console output is synchronous.
(XEN) Bootloader: GRUB 2.03
(XEN) Command line: dom0_mem=8192M dom0_max_vcpus=8 loglvl=all guest_loglvl=all unrestricted_guest=1 msi=1 conring_size=4M console=com1 com1=115200,8n1 sync_console vpid=1 vpmu=1 altp2m=1 psr=cmt psr=cat psr=cdp ept=pml iommu=on,intpost hvm_fep=true extra_guest_irqs=1024,1024
(XEN) Xen image load base address: 0
(XEN) Video information:
(XEN) VGA is text mode 80x25, font 8x16
(XEN) VBE/DDC methods: V1 V2; EDID transfer time: 1 seconds
(XEN) Disc information:
(XEN) Found 1 MBR signatures
(XEN) Found 1 EDD information structures
(XEN) CPU Vendor: Intel, Family 6 (0x6), Model 106 (0x6a), Stepping 0 (raw 000606a0)
(XEN) Xen-e820 RAM map:
(XEN) [0000000000000000, 000000000009fbff] (usable)
(XEN) [000000000009fc00, 000000000009ffff] (reserved)
(XEN) [00000000000f0000, 00000000000fffff] (reserved)
(XEN) [0000000000100000, 00000000bffdcfff] (usable)
(XEN) [00000000bffdd000, 00000000bfffffff] (reserved)
(XEN) [00000000feffc000, 00000000feffffff] (reserved)
(XEN) [00000000fffc0000, 00000000ffffffff] (reserved)
(XEN) [0000000100000000, 00000001ffffffff] (usable)
(XEN) New Xen image base address: 0xbf800000
(XEN) ACPI: RSDP 000F59D0, 0014 (r0 BOCHS )
(XEN) ACPI: RSDT BFFE143E, 0030 (r1 BOCHS BXPCRSDT 1 BXPC 1)
(XEN) ACPI: FACP BFFE12E2, 0074 (r1 BOCHS BXPCFACP 1 BXPC 1)
(XEN) ACPI: DSDT BFFDFC80, 1662 (r1 BOCHS BXPCDSDT 1 BXPC 1)
(XEN) ACPI: FACS BFFDFC40, 0040
(XEN) ACPI: APIC BFFE1356, 00B0 (r1 BOCHS BXPCAPIC 1 BXPC 1)
(XEN) ACPI: HPET BFFE1406, 0038 (r1 BOCHS BXPCHPET 1 BXPC 1)
(XEN) System RAM: 7167MB (7339504kB)
(XEN) No NUMA configuration found
(XEN) Faking a node at 0000000000000000-0000000200000000
(XEN) Domain heap initialised
(XEN) Allocated console ring of 4096 KiB.
(XEN) found SMP MP-table at 000f5a10
(XEN) DMI 2.8 present.
(XEN) Using APIC driver default
(XEN) ACPI: PM-Timer IO Port: 0x608 (24 bits)
(XEN) ACPI: SLEEP INFO: pm1x_cnt[1:604,1:0], pm1x_evt[1:600,1:0]
(XEN) ACPI: wakeup_vec[bffdfc4c], vec_size[20]
(XEN) ACPI: Local APIC address 0xfee00000
(XEN) ACPI: LAPIC (acpi_id[0x00] lapic_id[0x00] enabled)
(XEN) ACPI: LAPIC (acpi_id[0x01] lapic_id[0x01] enabled)
(XEN) ACPI: LAPIC (acpi_id[0x02] lapic_id[0x02] enabled)
(XEN) ACPI: LAPIC (acpi_id[0x03] lapic_id[0x03] enabled)
(XEN) ACPI: LAPIC (acpi_id[0x04] lapic_id[0x04] enabled)
(XEN) ACPI: LAPIC (acpi_id[0x05] lapic_id[0x05] enabled)
(XEN) ACPI: LAPIC (acpi_id[0x06] lapic_id[0x06] enabled)
(XEN) ACPI: LAPIC (acpi_id[0x07] lapic_id[0x07] enabled)
(XEN) ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1])
(XEN) ACPI: IOAPIC (id[0x00] address[0xfec00000] gsi_base[0])
(XEN) IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23
(XEN) ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl)
(XEN) ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level)
(XEN) ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level)
(XEN) ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level)
(XEN) ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level)
(XEN) ACPI: IRQ0 used by override.
(XEN) ACPI: IRQ2 used by override.
(XEN) ACPI: IRQ5 used by override.
(XEN) ACPI: IRQ9 used by override.
(XEN) ACPI: IRQ10 used by override.
(XEN) ACPI: IRQ11 used by override.
(XEN) Enabling APIC mode: Flat. Using 1 I/O APICs
(XEN) ACPI: HPET id: 0x8086a201 base: 0xfed00000
(XEN) Using ACPI (MADT) for SMP configuration information
(XEN) SMP: Allowing 8 CPUs (0 hotplug CPUs)
(XEN) IRQ limits: 24 GSI, 1640 MSI/MSI-X
(XEN) Switched to APIC driver x2apic_phys
(XEN) xstate: size: 0xa88 and states: 0x2e7
(XEN) mce_intel.c:779: MCA Capability: firstbank 0, extended MCE MSR 0, BCAST, SER, LMCE
(XEN) CPU0: Intel machine check reporting enabled
(XEN) Unrecognised CPU model 0x6a - assuming vulnerable to LazyFPU
(XEN) Speculative mitigation facilities:
(XEN) Hardware features: IBRS/IBPB STIBP SSBD MD_CLEAR IBPB IBRS_ALL RDCL_NO SKIP_L1DFL MDS_NO
(XEN) Compiled-in support: INDIRECT_THUNK SHADOW_PAGING
(XEN) Xen settings: BTI-Thunk JMP, SPEC_CTRL: IBRS+ SSBD-, Other: IBPB BRANCH_HARDEN
(XEN) Support for HVM VMs: MSR_SPEC_CTRL RSB EAGER_FPU MD_CLEAR
(XEN) Support for PV VMs: MSR_SPEC_CTRL RSB EAGER_FPU MD_CLEAR
(XEN) XPTI (64-bit PV only): Dom0 disabled, DomU disabled (with PCID)
(XEN) PV L1TF shadowing: Dom0 disabled, DomU disabled
(XEN) Using scheduler: SMP Credit Scheduler rev2 (credit2)
(XEN) Initializing Credit2 scheduler
(XEN) load_precision_shift: 18
(XEN) load_window_shift: 30
(XEN) underload_balance_tolerance: 0
(XEN) overload_balance_tolerance: -3
(XEN) runqueues arrangement: socket
(XEN) cap enforcement granularity: 10ms
(XEN) load tracking window length 1073741824 ns
(XEN) Platform timer is 100.000MHz HPET
(XEN) Detected 2000.020 MHz processor.
(XEN) alt table ffff82d080481150 -> ffff82d08048f276
(XEN) I/O virtualisation disabled
(XEN) nr_sockets: 8
(XEN) ENABLING IO-APIC IRQs
(XEN) -> Using new ACK method
(XEN) ..TIMER: vector=0xF0 apic1=0 pin1=2 apic2=-1 pin2=-1
(XEN) TSC deadline timer enabled
(XEN) mwait-idle: does not run on family 6 model 106
(XEN) VMX: Supported advanced features:
(XEN) - APIC MMIO access virtualisation
(XEN) - APIC TPR shadow
(XEN) - Extended Page Tables (EPT)
(XEN) - Virtual-Processor Identifiers (VPID)
(XEN) - Virtual NMI
(XEN) - MSR direct-access bitmap
(XEN) - Unrestricted Guest
(XEN) - APIC Register Virtualization
(XEN) - Virtual Interrupt Delivery
(XEN) - Posted Interrupt Processing
(XEN) - VMCS shadowing
(XEN) - VM Functions
(XEN) - Page Modification Logging
(XEN) HVM: ASIDs enabled.
(XEN) HVM: VMX enabled
(XEN) HVM: Hardware Assisted Paging (HAP) detected
(XEN) HVM: HAP page sizes: 4kB, 2MB, 1GB
(XEN) alt table ffff82d080481150 -> ffff82d08048f276
(XEN) Brought up 8 CPUs
(XEN) Adding cpu 0 to runqueue 0
(XEN) First cpu on runqueue, activating
(XEN) Adding cpu 1 to runqueue 1
(XEN) First cpu on runqueue, activating
(XEN) Adding cpu 2 to runqueue 2
(XEN) First cpu on runqueue, activating
(XEN) Adding cpu 3 to runqueue 3
(XEN) First cpu on runqueue, activating
(XEN) Adding cpu 4 to runqueue 4
(XEN) First cpu on runqueue, activating
(XEN) Adding cpu 5 to runqueue 5
(XEN) First cpu on runqueue, activating
(XEN) Adding cpu 6 to runqueue 6
(XEN) First cpu on runqueue, activating
(XEN) Adding cpu 7 to runqueue 7
(XEN) First cpu on runqueue, activating
(XEN) Running stub recovery selftests...
(XEN) traps.c:1590: GPF (0000): ffff82d0bfffe041 [ffff82d0bfffe041] -> ffff82d08038e40c
(XEN) traps.c:785: Trap 12: ffff82d0bfffe040 [ffff82d0bfffe040] -> ffff82d08038e40c
(XEN) traps.c:1124: Trap 3: ffff82d0bfffe041 [ffff82d0bfffe041] -> ffff82d08038e40c
(XEN) ----[ Xen-4.14-unstable x86_64 debug=y Tainted: C ]----
(XEN) CPU: 0
(XEN) RIP: e008:[<ffff82d08043a8bc>] core2_vpmu_init+0xa5/0x221
(XEN) RFLAGS: 0000000000010202 CONTEXT: hypervisor
(XEN) rax: 0000000008300802 rbx: 0000000000000000 rcx: 0000000000000345
(XEN) rdx: 0000000000000004 rsi: 000000000000000a rdi: 0000000000000063
(XEN) rbp: ffff82d0804b7d68 rsp: ffff82d0804b7d58 r8: 0000000000000004
(XEN) r9: 0000000000000008 r10: ffff82d0805effe0 r11: 0000000000000032
(XEN) r12: 0000000000000002 r13: 0000000000000008 r14: ffff82d0805dd0c0
(XEN) r15: ffff82d0805de300 cr0: 000000008005003b cr4: 00000000003526e0
(XEN) cr3: 00000000bfca4000 cr2: 0000000000000000
(XEN) fsb: 0000000000000000 gsb: 0000000000000000 gss: 0000000000000000
(XEN) ds: 0000 es: 0000 fs: 0000 gs: 0000 ss: 0000 cs: e008
(XEN) Xen code around <ffff82d08043a8bc> (core2_vpmu_init+0xa5/0x221):
(XEN) 00 06 00 b9 45 03 00 00 <0f> 32 48 89 c1 48 c1 e9 0d 83 e1 01 88 0d 32 00
(XEN) Xen stack trace from rsp=ffff82d0804b7d58:
(XEN) ffff82d080452168 ffff82d080452248 ffff82d0804b7d78 ffff82d08043a622
(XEN) ffff82d0804b7d98 ffff82d08040dcb0 0000000000000008 0000000000000000
(XEN) ffff82d0804b7ee8 ffff82d0804339db 0000000001fb4fff 00000000000001f6
(XEN) ffff83000009de80 ffffffff00000000 000ffff82d0807f6 0000000000000002
(XEN) 0000000000000002 0000000000000002 0000000000000002 0000000000000002
(XEN) 0000000000000001 0000000000000001 0000000000100000 ffff82d08047ac00
(XEN) 0000000000200000 ffff82d08047acdc 0000000100000008 0000000000000007
(XEN) 0000000001fb4000 0000000000000000 0000000000000007 ffff83000009dd20
(XEN) ffff83000009de80 ffff83000009dfb0 0000000000000000 0000000000000000
(XEN) 0000000400000000 0000000000000000 ffffffffffffffff 00000000ffffffff
(XEN) 0000000800000000 000000010000006e 0000000000000003 00000000000002f8
(XEN) 0000000000000002 0000000000000000 0000000000000000 0000000000000000
(XEN) 0000000000000000 0000000000000000 0000000000000000 ffff82d0802000ec
(XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000
(XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000
(XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000
(XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000
(XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000
(XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000
(XEN) 0000000000000000 0000e01000000000 ffff8301f136e000 0000000000000000
(XEN) Xen call trace:
(XEN) [<ffff82d08043a8bc>] R core2_vpmu_init+0xa5/0x221
(XEN) [<ffff82d08043a622>] F arch/x86/cpu/vpmu.c#vpmu_init+0xc4/0x100
(XEN) [<ffff82d08040dcb0>] F do_initcalls+0x35/0x44
(XEN) [<ffff82d0804339db>] F __start_xen+0x22d7/0x2808
(XEN) [<ffff82d0802000ec>] F __high_start+0x4c/0x4e
(XEN)
(XEN)
(XEN) ****************************************
(XEN) Panic on CPU 0:
(XEN) GENERAL PROTECTION FAULT
(XEN) [error_code=0000]
(XEN) ****************************************
(XEN)
(XEN) Reboot in five seconds...
[-- Attachment #3: Type: text/plain, Size: 157 bytes --]
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: [Xen-devel] [BUG]Nested virtualization, Xen on KVM, Xen cannot boot up as a guest of KVM
2020-02-26 6:21 [Xen-devel] [BUG]Nested virtualization, Xen on KVM, Xen cannot boot up as a guest of KVM Chen, Farrah
@ 2020-02-26 7:37 ` Chao Gao
2020-02-26 9:19 ` [Xen-devel] [PATCH] x86/vPMU: don't blindly assume IA32_PERF_CAPABILITIES MSR exists Jan Beulich
1 sibling, 0 replies; 8+ messages in thread
From: Chao Gao @ 2020-02-26 7:37 UTC (permalink / raw)
To: Chen, Farrah; +Cc: xen-devel, Hao, Xudong
On Wed, Feb 26, 2020 at 02:21:25PM +0800, Chen, Farrah wrote:
>Description:
>
>Nested virtualization, take KVM host as L0, create guest on it, install Xen on
>guest, then guest cannot boot up from Xen and keep rebooting.
>
>
>
>Reproduce steps:
>1. Enable KVM nested on host(L0)
>rmmod kvm_intel
>modprobe kvm_intel nested=y
>cat /sys/module/kvm_intel/parameters/nested
>Y
>
>2.Create L1 guest via qemu:
>qemu-system-x86_64 -accel kvm -cpu host -drive file=rhel8.img,if=none,id=
>virtio-disk0 -device virtio-blk-pci,drive=virtio-disk0 -m 7168 -smp 8 -monitor
>pty -cpu host -device virtio-net-pci,netdev=nic0,mac=00:16:3e:72:5e:0a -netdev
>tap,id=nic0,br=virbr0,helper=/usr/libexec/qemu-bridge-helper,vhost=on -serial
>stdio
>
>3. Build and install Xen on L1 guest
>
>4. Reboot L1 and make it boot from Xen
>
>
>
>Then L1 keep rebooting, full log attached.
>
>……
>
>(XEN) Running stub recovery selftests...
>
>(XEN) traps.c:1590: GPF (0000): ffff82d0bfffe041 [ffff82d0bfffe041] ->
>ffff82d08038e40c
>
>(XEN) traps.c:785: Trap 12: ffff82d0bfffe040 [ffff82d0bfffe040] ->
>ffff82d08038e40c
>
>(XEN) traps.c:1124: Trap 3: ffff82d0bfffe041 [ffff82d0bfffe041] ->
>ffff82d08038e40c
>
>(XEN) ----[ Xen-4.14-unstable x86_64 debug=y Tainted: C ]----
>
>(XEN) CPU: 0
>
>(XEN) RIP: e008:[<ffff82d08043a8bc>] core2_vpmu_init+0xa5/0x221
>
>(XEN) RFLAGS: 0000000000010202 CONTEXT: hypervisor
>
>(XEN) rax: 0000000008300802 rbx: 0000000000000000 rcx: 0000000000000345
>
>(XEN) rdx: 0000000000000004 rsi: 000000000000000a rdi: 0000000000000063
>
>(XEN) rbp: ffff82d0804b7d68 rsp: ffff82d0804b7d58 r8: 0000000000000004
>
>(XEN) r9: 0000000000000008 r10: ffff82d0805effe0 r11: 0000000000000032
>
>(XEN) r12: 0000000000000002 r13: 0000000000000008 r14: ffff82d0805dd0c0
>
>(XEN) r15: ffff82d0805de300 cr0: 000000008005003b cr4: 00000000003526e0
>
>(XEN) cr3: 00000000bfca4000 cr2: 0000000000000000
>
>(XEN) fsb: 0000000000000000 gsb: 0000000000000000 gss: 0000000000000000
>
>(XEN) ds: 0000 es: 0000 fs: 0000 gs: 0000 ss: 0000 cs: e008
>
>(XEN) Xen code around <ffff82d08043a8bc> (core2_vpmu_init+0xa5/0x221):
>
>(XEN) 00 06 00 b9 45 03 00 00 <0f> 32 48 89 c1 48 c1 e9 0d 83 e1 01 88 0d 32
>00
The machine code above shows that #GP is generated when xen is reading
MSR_IA32_PERF_CAPABILITIES. In a KVM guest without Xen, cpuid tells that
perfmon isn't supported.
# ./cpuid -1 |grep "perfmon and debug"
PDCM: perfmon and debug = false
So, it looks core2_vpmu_init() lacks a check to ensure the MSR is supported.
Thanks
Chao
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel
^ permalink raw reply [flat|nested] 8+ messages in thread
* [Xen-devel] [PATCH] x86/vPMU: don't blindly assume IA32_PERF_CAPABILITIES MSR exists
2020-02-26 6:21 [Xen-devel] [BUG]Nested virtualization, Xen on KVM, Xen cannot boot up as a guest of KVM Chen, Farrah
2020-02-26 7:37 ` Chao Gao
@ 2020-02-26 9:19 ` Jan Beulich
2020-02-26 10:09 ` Andrew Cooper
2020-02-26 10:09 ` Roger Pau Monné
1 sibling, 2 replies; 8+ messages in thread
From: Jan Beulich @ 2020-02-26 9:19 UTC (permalink / raw)
To: xen-devel
Cc: Wei Liu, Andrew Cooper, Hao, Xudong, Chen, Farrah, Gao, Chao,
Roger Pau Monné
Just like VMX'es lbr_tsx_fixup_check() the respective CPUID bit should
be consulted first.
Reported-by: Farrah Chen <farrah.chen@intel.com>
Signed-off-by: Jan Beulich <jbeulich@suse.com>
--- a/xen/arch/x86/cpu/vpmu_intel.c
+++ b/xen/arch/x86/cpu/vpmu_intel.c
@@ -900,7 +900,6 @@ int vmx_vpmu_initialise(struct vcpu *v)
int __init core2_vpmu_init(void)
{
- u64 caps;
unsigned int version = 0;
unsigned int i;
@@ -932,8 +931,14 @@ int __init core2_vpmu_init(void)
arch_pmc_cnt = core2_get_arch_pmc_count();
fixed_pmc_cnt = core2_get_fixed_pmc_count();
- rdmsrl(MSR_IA32_PERF_CAPABILITIES, caps);
- full_width_write = (caps >> 13) & 1;
+
+ if ( cpu_has_pdcm )
+ {
+ uint64_t caps;
+
+ rdmsrl(MSR_IA32_PERF_CAPABILITIES, caps);
+ full_width_write = (caps >> 13) & 1;
+ }
fixed_ctrl_mask = ~((1ull << (fixed_pmc_cnt * FIXED_CTR_CTRL_BITS)) - 1);
/* mask .AnyThread bits for all fixed counters */
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: [Xen-devel] [PATCH] x86/vPMU: don't blindly assume IA32_PERF_CAPABILITIES MSR exists
2020-02-26 9:19 ` [Xen-devel] [PATCH] x86/vPMU: don't blindly assume IA32_PERF_CAPABILITIES MSR exists Jan Beulich
@ 2020-02-26 10:09 ` Andrew Cooper
2020-02-26 10:09 ` Roger Pau Monné
1 sibling, 0 replies; 8+ messages in thread
From: Andrew Cooper @ 2020-02-26 10:09 UTC (permalink / raw)
To: Jan Beulich, xen-devel
Cc: Roger Pau Monné, Hao, Xudong, Wei Liu, Chen, Farrah, Gao, Chao
On 26/02/2020 09:19, Jan Beulich wrote:
> Just like VMX'es lbr_tsx_fixup_check() the respective CPUID bit should
> be consulted first.
>
> Reported-by: Farrah Chen <farrah.chen@intel.com>
> Signed-off-by: Jan Beulich <jbeulich@suse.com>
Acked-by: Andrew Cooper <andrew.cooper3@citrix.com>
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: [Xen-devel] [PATCH] x86/vPMU: don't blindly assume IA32_PERF_CAPABILITIES MSR exists
2020-02-26 9:19 ` [Xen-devel] [PATCH] x86/vPMU: don't blindly assume IA32_PERF_CAPABILITIES MSR exists Jan Beulich
2020-02-26 10:09 ` Andrew Cooper
@ 2020-02-26 10:09 ` Roger Pau Monné
2020-02-26 10:39 ` Jan Beulich
1 sibling, 1 reply; 8+ messages in thread
From: Roger Pau Monné @ 2020-02-26 10:09 UTC (permalink / raw)
To: Jan Beulich
Cc: Wei Liu, Andrew Cooper, Hao, Xudong, Chen, Farrah, xen-devel, Gao, Chao
On Wed, Feb 26, 2020 at 10:19:19AM +0100, Jan Beulich wrote:
> Just like VMX'es lbr_tsx_fixup_check() the respective CPUID bit should
> be consulted first.
>
> Reported-by: Farrah Chen <farrah.chen@intel.com>
> Signed-off-by: Jan Beulich <jbeulich@suse.com>
>
> --- a/xen/arch/x86/cpu/vpmu_intel.c
> +++ b/xen/arch/x86/cpu/vpmu_intel.c
> @@ -900,7 +900,6 @@ int vmx_vpmu_initialise(struct vcpu *v)
>
> int __init core2_vpmu_init(void)
> {
> - u64 caps;
> unsigned int version = 0;
> unsigned int i;
>
> @@ -932,8 +931,14 @@ int __init core2_vpmu_init(void)
>
> arch_pmc_cnt = core2_get_arch_pmc_count();
> fixed_pmc_cnt = core2_get_fixed_pmc_count();
> - rdmsrl(MSR_IA32_PERF_CAPABILITIES, caps);
> - full_width_write = (caps >> 13) & 1;
> +
> + if ( cpu_has_pdcm )
> + {
> + uint64_t caps;
> +
> + rdmsrl(MSR_IA32_PERF_CAPABILITIES, caps);
> + full_width_write = (caps >> 13) & 1;
Will PMU work without PDCM?
I've been grepping the Intel SDMs, but the only mention is that PDCM
signal the availability of MSR_IA32_PERF_CAPABILITIES.
Thanks, Roger.
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: [Xen-devel] [PATCH] x86/vPMU: don't blindly assume IA32_PERF_CAPABILITIES MSR exists
2020-02-26 10:09 ` Roger Pau Monné
@ 2020-02-26 10:39 ` Jan Beulich
2020-02-26 10:56 ` Andrew Cooper
0 siblings, 1 reply; 8+ messages in thread
From: Jan Beulich @ 2020-02-26 10:39 UTC (permalink / raw)
To: Roger Pau Monné
Cc: Wei Liu, Andrew Cooper, Hao, Xudong, Chen, Farrah, xen-devel, Gao, Chao
On 26.02.2020 11:09, Roger Pau Monné wrote:
> On Wed, Feb 26, 2020 at 10:19:19AM +0100, Jan Beulich wrote:
>> Just like VMX'es lbr_tsx_fixup_check() the respective CPUID bit should
>> be consulted first.
>>
>> Reported-by: Farrah Chen <farrah.chen@intel.com>
>> Signed-off-by: Jan Beulich <jbeulich@suse.com>
>>
>> --- a/xen/arch/x86/cpu/vpmu_intel.c
>> +++ b/xen/arch/x86/cpu/vpmu_intel.c
>> @@ -900,7 +900,6 @@ int vmx_vpmu_initialise(struct vcpu *v)
>>
>> int __init core2_vpmu_init(void)
>> {
>> - u64 caps;
>> unsigned int version = 0;
>> unsigned int i;
>>
>> @@ -932,8 +931,14 @@ int __init core2_vpmu_init(void)
>>
>> arch_pmc_cnt = core2_get_arch_pmc_count();
>> fixed_pmc_cnt = core2_get_fixed_pmc_count();
>> - rdmsrl(MSR_IA32_PERF_CAPABILITIES, caps);
>> - full_width_write = (caps >> 13) & 1;
>> +
>> + if ( cpu_has_pdcm )
>> + {
>> + uint64_t caps;
>> +
>> + rdmsrl(MSR_IA32_PERF_CAPABILITIES, caps);
>> + full_width_write = (caps >> 13) & 1;
>
> Will PMU work without PDCM?
>
> I've been grepping the Intel SDMs, but the only mention is that PDCM
> signal the availability of MSR_IA32_PERF_CAPABILITIES.
Well, there's no other use of the MSR afaics except for getting
the one bit here, so I assume it'll work.
Jan
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: [Xen-devel] [PATCH] x86/vPMU: don't blindly assume IA32_PERF_CAPABILITIES MSR exists
2020-02-26 10:39 ` Jan Beulich
@ 2020-02-26 10:56 ` Andrew Cooper
2020-02-26 12:47 ` Chen, Farrah
0 siblings, 1 reply; 8+ messages in thread
From: Andrew Cooper @ 2020-02-26 10:56 UTC (permalink / raw)
To: Jan Beulich, Roger Pau Monné
Cc: xen-devel, Hao, Xudong, Wei Liu, Chen, Farrah, Gao, Chao
On 26/02/2020 10:39, Jan Beulich wrote:
> On 26.02.2020 11:09, Roger Pau Monné wrote:
>> On Wed, Feb 26, 2020 at 10:19:19AM +0100, Jan Beulich wrote:
>>> Just like VMX'es lbr_tsx_fixup_check() the respective CPUID bit should
>>> be consulted first.
>>>
>>> Reported-by: Farrah Chen <farrah.chen@intel.com>
>>> Signed-off-by: Jan Beulich <jbeulich@suse.com>
>>>
>>> --- a/xen/arch/x86/cpu/vpmu_intel.c
>>> +++ b/xen/arch/x86/cpu/vpmu_intel.c
>>> @@ -900,7 +900,6 @@ int vmx_vpmu_initialise(struct vcpu *v)
>>>
>>> int __init core2_vpmu_init(void)
>>> {
>>> - u64 caps;
>>> unsigned int version = 0;
>>> unsigned int i;
>>>
>>> @@ -932,8 +931,14 @@ int __init core2_vpmu_init(void)
>>>
>>> arch_pmc_cnt = core2_get_arch_pmc_count();
>>> fixed_pmc_cnt = core2_get_fixed_pmc_count();
>>> - rdmsrl(MSR_IA32_PERF_CAPABILITIES, caps);
>>> - full_width_write = (caps >> 13) & 1;
>>> +
>>> + if ( cpu_has_pdcm )
>>> + {
>>> + uint64_t caps;
>>> +
>>> + rdmsrl(MSR_IA32_PERF_CAPABILITIES, caps);
>>> + full_width_write = (caps >> 13) & 1;
>> Will PMU work without PDCM?
The performance counter interface in CPUs predate the introduction of
PERF_CAPS.
>> I've been grepping the Intel SDMs, but the only mention is that PDCM
>> signal the availability of MSR_IA32_PERF_CAPABILITIES.
> Well, there's no other use of the MSR afaics except for getting
> the one bit here, so I assume it'll work.
It is an off-by-default, outside security support area of functionality
with known functional bugs outstanding against it.
"not crash" is a fine improvement on the status quo.
~Andrew
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: [Xen-devel] [PATCH] x86/vPMU: don't blindly assume IA32_PERF_CAPABILITIES MSR exists
2020-02-26 10:56 ` Andrew Cooper
@ 2020-02-26 12:47 ` Chen, Farrah
0 siblings, 0 replies; 8+ messages in thread
From: Chen, Farrah @ 2020-02-26 12:47 UTC (permalink / raw)
To: Andrew Cooper, Jan Beulich, Roger Pau Monné
Cc: xen-devel, Hao, Xudong, Wei Liu, Gao, Chao
I applied this patch to Xen and retested, Xen on KVM booted up successfully, thanks a lot.
Thanks,
Fan
-----Original Message-----
From: Andrew Cooper <andrew.cooper3@citrix.com>
Sent: Wednesday, February 26, 2020 6:56 PM
To: Jan Beulich <jbeulich@suse.com>; Roger Pau Monné <roger.pau@citrix.com>
Cc: xen-devel@lists.xenproject.org; Chen, Farrah <farrah.chen@intel.com>; Hao, Xudong <xudong.hao@intel.com>; Gao, Chao <chao.gao@intel.com>; Wei Liu <wl@xen.org>
Subject: Re: [PATCH] x86/vPMU: don't blindly assume IA32_PERF_CAPABILITIES MSR exists
On 26/02/2020 10:39, Jan Beulich wrote:
> On 26.02.2020 11:09, Roger Pau Monné wrote:
>> On Wed, Feb 26, 2020 at 10:19:19AM +0100, Jan Beulich wrote:
>>> Just like VMX'es lbr_tsx_fixup_check() the respective CPUID bit
>>> should be consulted first.
>>>
>>> Reported-by: Farrah Chen <farrah.chen@intel.com>
>>> Signed-off-by: Jan Beulich <jbeulich@suse.com>
>>>
>>> --- a/xen/arch/x86/cpu/vpmu_intel.c
>>> +++ b/xen/arch/x86/cpu/vpmu_intel.c
>>> @@ -900,7 +900,6 @@ int vmx_vpmu_initialise(struct vcpu *v)
>>>
>>> int __init core2_vpmu_init(void)
>>> {
>>> - u64 caps;
>>> unsigned int version = 0;
>>> unsigned int i;
>>>
>>> @@ -932,8 +931,14 @@ int __init core2_vpmu_init(void)
>>>
>>> arch_pmc_cnt = core2_get_arch_pmc_count();
>>> fixed_pmc_cnt = core2_get_fixed_pmc_count();
>>> - rdmsrl(MSR_IA32_PERF_CAPABILITIES, caps);
>>> - full_width_write = (caps >> 13) & 1;
>>> +
>>> + if ( cpu_has_pdcm )
>>> + {
>>> + uint64_t caps;
>>> +
>>> + rdmsrl(MSR_IA32_PERF_CAPABILITIES, caps);
>>> + full_width_write = (caps >> 13) & 1;
>> Will PMU work without PDCM?
The performance counter interface in CPUs predate the introduction of PERF_CAPS.
>> I've been grepping the Intel SDMs, but the only mention is that PDCM
>> signal the availability of MSR_IA32_PERF_CAPABILITIES.
> Well, there's no other use of the MSR afaics except for getting the
> one bit here, so I assume it'll work.
It is an off-by-default, outside security support area of functionality with known functional bugs outstanding against it.
"not crash" is a fine improvement on the status quo.
~Andrew
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel
^ permalink raw reply [flat|nested] 8+ messages in thread
end of thread, other threads:[~2020-02-26 12:48 UTC | newest]
Thread overview: 8+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2020-02-26 6:21 [Xen-devel] [BUG]Nested virtualization, Xen on KVM, Xen cannot boot up as a guest of KVM Chen, Farrah
2020-02-26 7:37 ` Chao Gao
2020-02-26 9:19 ` [Xen-devel] [PATCH] x86/vPMU: don't blindly assume IA32_PERF_CAPABILITIES MSR exists Jan Beulich
2020-02-26 10:09 ` Andrew Cooper
2020-02-26 10:09 ` Roger Pau Monné
2020-02-26 10:39 ` Jan Beulich
2020-02-26 10:56 ` Andrew Cooper
2020-02-26 12:47 ` Chen, Farrah
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.