perf uncore & lkvm woes

perf uncore & lkvm woes

[Pekka Enberg]

Hello,
[    0.248962] Pid: 0, comm: swapper/0 Not tainted 3.6.0-rc1+ #24
[penberg@tux ~]$ cat perf-kvmtool-issue
Hello,

Has anyone seen this? It's kvmtool/next with 3.6.0-rc1. Looks like we
are doing uncore_init() on virtualized CPU which breaks boot.

			Pekka

[penberg@tux kvm]$ ./vm run
  # lkvm run -k ../../arch/x86/boot/bzImage -m 448 -c 4 --name guest-30425
early console in decompress_kernel

Decompressing Linux... Parsing ELF... done.
Booting the kernel.
[    0.000000] Initializing cgroup subsys cpuset
[    0.000000] Initializing cgroup subsys cpu
[    0.000000] Linux version 3.6.0-rc1+ (penberg@tux) (gcc version
4.6.3 20120306 (Red Hat 4.6.3-2) (GCC) ) #24 SMP Thu Aug 16 09:55:41
EEST 2012
[    0.000000] Command line: noapic noacpi pci=conf1 reboot=k panic=1
i8042.direct=1 i8042.dumbkbd=1 i8042.nopnp=1 console=ttyS0
earlyprintk=serial i8042.noaux=1  root=/dev/root rw
rootflags=rw,trans=virtio,version=9p2000.L rootfstype=9p
init=/virt/init  ip=dhcp
[    0.000000] e820: BIOS-provided physical RAM map:
[    0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable
[    0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved
[    0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000ffffe] reserved
[    0.000000] BIOS-e820: [mem 0x0000000000100000-0x000000001bffffff] usable
[    0.000000] bootconsole [earlyser0] enabled
[    0.000000] NX (Execute Disable) protection: active
[    0.000000] DMI not present or invalid.
[    0.000000] No AGP bridge found
[    0.000000] e820: last_pfn = 0x1c000 max_arch_pfn = 0x400000000
[    0.000000] x86 PAT enabled: cpu 0, old 0x70106, new 0x7010600070106
[    0.000000] CPU MTRRs all blank - virtualized system.
[    0.000000] found SMP MP-table at [mem 0x000f0370-0x000f037f]
mapped at [ffff8800000f0370]
[    0.000000] init_memory_mapping: [mem 0x00000000-0x1bffffff]
[    0.000000] ACPI BIOS Bug: Error: A valid RSDP was not found
(20120711/tbxfroot-219)
[    0.000000] No NUMA configuration found
[    0.000000] Faking a node at [mem 0x0000000000000000-0x000000001bffffff]
[    0.000000] Initmem setup node 0 [mem 0x00000000-0x1bffffff]
[    0.000000]   NODE_DATA [mem 0x1bffc000-0x1bffffff]
[    0.000000] Zone ranges:
[    0.000000]   DMA      [mem 0x00010000-0x00ffffff]
[    0.000000]   DMA32    [mem 0x01000000-0xffffffff]
[    0.000000]   Normal   empty
[    0.000000] Movable zone start for each node
[    0.000000] Early memory node ranges
[    0.000000]   node   0: [mem 0x00010000-0x0009efff]
[    0.000000]   node   0: [mem 0x00100000-0x1bffffff]
[    0.000000] Intel MultiProcessor Specification v1.4
[    0.000000] MPTABLE: OEM ID: KVMCPU00
[    0.000000] MPTABLE: Product ID: 0.1
[    0.000000] MPTABLE: APIC at: 0xFEE00000
[    0.000000] Processor #0 (Bootup-CPU)
[    0.000000] Processor #1
[    0.000000] Processor #2
[    0.000000] Processor #3
[    0.000000] IOAPIC[0]: apic_id 5, version 17, address 0xfec00000, GSI 0-23
[    0.000000] Processors: 4
[    0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs
[    0.000000] PM: Registered nosave memory: 000000000009f000 - 00000000000a0000
[    0.000000] PM: Registered nosave memory: 00000000000a0000 - 00000000000f0000
[    0.000000] PM: Registered nosave memory: 00000000000f0000 - 00000000000ff000
[    0.000000] PM: Registered nosave memory: 00000000000ff000 - 0000000000100000
[    0.000000] e820: [mem 0x1c000000-0xffffffff] available for PCI devices
[    0.000000] setup_percpu: NR_CPUS:64 nr_cpumask_bits:64
nr_cpu_ids:4 nr_node_ids:1
[    0.000000] PERCPU: Embedded 27 pages/cpu @ffff88001bc00000 s78272
r8192 d24128 u524288
[    0.000000] Built 1 zonelists in Node order, mobility grouping on.
Total pages: 112777
[    0.000000] Policy zone: DMA32
[    0.000000] Kernel command line: noapic noacpi pci=conf1 reboot=k
panic=1 i8042.direct=1 i8042.dumbkbd=1 i8042.nopnp=1 console=ttyS0
earlyprintk=serial i8042.noaux=1  root=/dev/root rw
rootflags=rw,trans=virtio,version=9p2000.L rootfstype=9p
init=/virt/init  ip=dhcp
[    0.000000] PID hash table entries: 2048 (order: 2, 16384 bytes)
[    0.000000] __ex_table already sorted, skipping sort
[    0.000000] xsave: enabled xstate_bv 0x7, cntxt size 0x340
[    0.000000] Checking aperture...
[    0.000000] No AGP bridge found
[    0.000000] Memory: 434972k/458752k available (7288k kernel code,
452k absent, 23328k reserved, 5691k data, 600k init)
[    0.000000] SLUB: Genslabs=15, HWalign=64, Order=0-3, MinObjects=0,
CPUs=4, Nodes=1
[    0.000000] Hierarchical RCU implementation.
[    0.000000] 	RCU restricting CPUs from NR_CPUS=64 to nr_cpu_ids=4.
[    0.000000] NR_IRQS:4352 nr_irqs:712 16
[    0.000000] Console: colour *CGA 80x25
[    0.000000] console [ttyS0] enabled, bootconsole disabled
[    0.000000] console [ttyS0] enabled, bootconsole disabled
[    0.000000] tsc: Fast TSC calibration using PIT
[    0.000000] tsc: Detected 2691.428 MHz processor
[    0.003002] Calibrating delay loop (skipped), value calculated
using timer frequency.. 5382.85 BogoMIPS (lpj=2691428)
[    0.004003] pid_max: default: 32768 minimum: 301
[    0.004451] Security Framework initialized
[    0.004835] SELinux:  Initializing.
[    0.005054] Dentry cache hash table entries: 65536 (order: 7, 524288 bytes)
[    0.006215] Inode-cache hash table entries: 32768 (order: 6, 262144 bytes)
[    0.007007] Mount-cache hash table entries: 256
[    0.007613] Initializing cgroup subsys cpuacct
[    0.008003] Initializing cgroup subsys freezer
[    0.008471] CPU: Physical Processor ID: 0
[    0.008845] CPU: Processor Core ID: 0
[    0.009010] mce: CPU supports 32 MCE banks
[    0.009475] Last level iTLB entries: 4KB 512, 2MB 0, 4MB 0
[    0.009475] Last level dTLB entries: 4KB 512, 2MB 32, 4MB 32
[    0.009475] tlb_flushall_shift is 0x5
[    0.011340] smpboot: CPU0: Intel 06/2a stepping 07
[    0.113066] Performance Events: unsupported p6 CPU model 42 no PMU
driver, software events only.
[    0.114236] smpboot: Booting Node   0, Processors  #1 #2 #3 OK
[    0.154994] Brought up 4 CPUs
[    0.155430] smpboot: Total of 4 processors activated (21531.42 BogoMIPS)
[    0.160325] kworker/u:0 (25) used greatest stack depth: 5496 bytes left
[    0.160305] RTC time:  6:57:03, date: 08/16/12
[    0.160305] NET: Registered protocol family 16
[    0.162117] PCI: Using configuration type 1 for base access
[    0.203066] bio: create slab <bio-0> at 0
[    0.204033] ACPI: Interpreter disabled.
[    0.205033] vgaarb: loaded
[    0.206011] SCSI subsystem initialized
[    0.208018] usbcore: registered new interface driver usbfs
[    0.209001] usbcore: registered new interface driver hub
[    0.209491] usbcore: registered new device driver usb
[    0.211051] Advanced Linux Sound Architecture Driver Version 1.0.25.
[    0.211972] PCI: Probing PCI hardware
[    0.213020] PCI host bridge to bus 0000:00
[    0.213976] pci_bus 0000:00: root bus resource [io  0x0000-0xffff]
[    0.214974] pci_bus 0000:00: root bus resource [mem 0x00000000-0xfffffffff]
[    0.216973] pci_bus 0000:00: No busn resource found for root bus,
will use [bus 00-ff]
[    0.222975] cfg80211: Calling CRDA to update world regulatory domain
[    0.223020] NetLabel: Initializing
[    0.223021] NetLabel:  domain hash size = 128
[    0.223021] NetLabel:  protocols = UNLABELED CIPSOv4
[    0.223044] NetLabel:  unlabeled traffic allowed by default
[    0.227100] pnp: PnP ACPI: disabled
[    0.238040] NET: Registered protocol family 2
[    0.239072] TCP established hash table entries: 16384 (order: 6,
262144 bytes)
[    0.240352] TCP bind hash table entries: 16384 (order: 6, 262144 bytes)
[    0.241060] TCP: Hash tables configured (established 16384 bind 16384)
[    0.242001] TCP: reno registered
[    0.242970] UDP hash table entries: 256 (order: 1, 8192 bytes)
[    0.243625] UDP-Lite hash table entries: 256 (order: 1, 8192 bytes)
[    0.244028] NET: Registered protocol family 1
[    0.245063] RPC: Registered named UNIX socket transport module.
[    0.245663] RPC: Registered udp transport module.
[    0.245965] RPC: Registered tcp transport module.
[    0.246410] RPC: Registered tcp NFSv4.1 backchannel transport module.
[    0.247141] platform rtc_cmos: registered platform RTC device (no
PNP device found)
[    0.247977] general protection fault: 0000 [#1] SMP
[    0.248736] Modules linked in:
[    0.248962] CPU 0
[    0.248962] Pid: 0, comm: swapper/0 Not tainted 3.6.0-rc1+ #24
[    0.248962] RIP: 0010:[<ffffffff810175d1>]  [<ffffffff810175d1>]
snb_uncore_msr_init_box+0x21/0x30
[    0.248962] RSP: 0018:ffff88001bc03ed8  EFLAGS: 00000046
[    0.248962] RAX: 000000002000000f RBX: ffff88001ab96800 RCX: 0000000000000391
[    0.248962] RDX: 0000000000000000 RSI: ffffffffffffeb9f RDI: ffff88001ab96a00
[    0.248962] RBP: ffff88001bc03ed8 R08: 0000000000000000 R09: 0000000000000000
[    0.248962] R10: 0000000000000000 R11: 0000000000000000 R12: ffffffff81ca3010
[    0.248962] R13: 0000000000000000 R14: ffff88001ab96a00 R15: 0000000000000000
[    0.248962] FS:  0000000000000000(0000) GS:ffff88001bc00000(0000)
knlGS:0000000000000000
[    0.248962] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[    0.248962] CR2: 0000000000000000 CR3: 0000000001c0b000 CR4: 00000000000406f0
[    0.248962] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[    0.248962] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[    0.248962] Process swapper/0 (pid: 0, threadinfo ffffffff81c00000,
task ffffffff81c13440)
[    0.248962] Stack:
[    0.248962]  ffff88001bc03ee8 ffffffff8170659d ffff88001bc03f48
ffffffff817008a1
[    0.248962]  ffff88001bc0d840 0000000000000000 0000000000000000
ffffffff81c172a0
[    0.248962]  ffff88001bc03f38 ffff88001bc93000 0000000000000000
00000000ffffffff
[    0.248962] Call Trace:
[    0.248962]  <IRQ>
[    0.248962]  [<ffffffff8170659d>] uncore_box_init+0x2c/0x2e
[    0.248962]  [<ffffffff817008a1>] uncore_cpu_starting+0xeb/0x12b
[    0.248962]  [<ffffffff81cca71e>] ? uncore_types_init+0x187/0x187
[    0.248962]  [<ffffffff81cca72f>] uncore_cpu_setup+0x11/0x13
[    0.248962]  [<ffffffff8108d00d>]
generic_smp_call_function_interrupt+0x8d/0x190
[    0.248962]  [<ffffffff81022e72>] smp_call_function_interrupt+0x22/0x40
[    0.248962]  [<ffffffff8171ab77>] call_function_interrupt+0x67/0x70
[    0.248962]  <EOI>
[    0.248962]  [<ffffffff8100a4e9>] ? default_idle+0x59/0x1f0
[    0.248962]  [<ffffffff8100aee6>] cpu_idle+0x86/0xd0
[    0.248962]  [<ffffffff816e7219>] rest_init+0x6d/0x74
[    0.248962]  [<ffffffff81cc2b50>] start_kernel+0x332/0x33f
[    0.248962]  [<ffffffff81cc261e>] ? repair_env_string+0x5a/0x5a
[    0.248962]  [<ffffffff81cc232d>] x86_64_start_reservations+0x131/0x135
[    0.248962]  [<ffffffff81cc2421>] x86_64_start_kernel+0xf0/0xf7
[    0.248962] Code: 66 2e 0f 1f 84 00 00 00 00 00 48 8b 87 08 01 00
00 55 48 89 e5 8b 80 d0 00 00 00 85 c0 75 0e 31 d2 b8 0f 00 00 20 b9
91 03 00 00 <0f> 30 5d c3 66 66 2e 0f 1f 84 00 00 00 00 00 55 31 d2 31
c0 b9
[    0.248962] RIP  [<ffffffff810175d1>] snb_uncore_msr_init_box+0x21/0x30
[    0.248962]  RSP <ffff88001bc03ed8>
[    0.248962] ---[ end trace 07d1ad15f2b29779 ]---
[    0.248962] Kernel panic - not syncing: Fatal exception in interrupt
[    0.248962] Rebooting in 1 seconds..
  # KVM compatibility warning.
	virtio-9p device was not detected.
	While you have requested a virtio-9p device, the guest kernel did not
initialize it.
	Please make sure that the guest kernel was compiled with
CONFIG_NET_9P_VIRTIO=y enabled in .config.

  # KVM compatibility warning.
	virtio-net device was not detected.
	While you have requested a virtio-net device, the guest kernel did
not initialize it.
	Please make sure that the guest kernel was compiled with
CONFIG_VIRTIO_NET=y enabled in .config.

  # KVM session ended normally.

Re: perf uncore & lkvm woes

[Cyrill Gorcunov]

On Thu, Aug 16, 2012 at 10:01:58AM +0300, Pekka Enberg wrote:
> Hello,
> [    0.248962] Pid: 0, comm: swapper/0 Not tainted 3.6.0-rc1+ #24
> [penberg@tux ~]$ cat perf-kvmtool-issue
> Hello,
> 
> Has anyone seen this? It's kvmtool/next with 3.6.0-rc1. Looks like we
> are doing uncore_init() on virtualized CPU which breaks boot.

Hi, I guess some cpuid/msr bit is not cleared again ;) I'll take a look
once time permit.

	Cyrill

Re: perf uncore & lkvm woes

[Cyrill Gorcunov]

On Thu, Aug 16, 2012 at 11:07:43AM +0400, Cyrill Gorcunov wrote:
> On Thu, Aug 16, 2012 at 10:01:58AM +0300, Pekka Enberg wrote:
> > Hello,
> > [    0.248962] Pid: 0, comm: swapper/0 Not tainted 3.6.0-rc1+ #24
> > [penberg@tux ~]$ cat perf-kvmtool-issue
> > Hello,
> > 
> > Has anyone seen this? It's kvmtool/next with 3.6.0-rc1. Looks like we
> > are doing uncore_init() on virtualized CPU which breaks boot.
> 
> Hi, I guess some cpuid/msr bit is not cleared again ;) I'll take a look
> once time permit.

If only I'm not missing something we've two options 1) either tune up
cpu model via cpuid interception in lkvm (which is bad I think)
2) provide some new kernel boot line option to not use unboxed pmu.

	Cyrill

Re: perf uncore & lkvm woes

[Pekka Enberg]

On Thu, Aug 16, 2012 at 10:01:58AM +0300, Pekka Enberg wrote:
> > Hello,
> > [    0.248962] Pid: 0, comm: swapper/0 Not tainted 3.6.0-rc1+ #24
> > [penberg@tux ~]$ cat perf-kvmtool-issue
> > Hello,
> > 
> > Has anyone seen this? It's kvmtool/next with 3.6.0-rc1. Looks like we
> > are doing uncore_init() on virtualized CPU which breaks boot.

On Thu, 16 Aug 2012, Cyrill Gorcunov wrote:
> Hi, I guess some cpuid/msr bit is not cleared again ;) I'll take a look
> once time permit.

Alternative fix would be to change our CPUID name to "KVMKVMKVM" or 
something to avoid going through these code paths.

Ingo?

			Pekka

Re: perf uncore & lkvm woes

[Peter Zijlstra]

On Thu, 2012-08-16 at 10:01 +0300, Pekka Enberg wrote:
> Has anyone seen this? It's kvmtool/next with 3.6.0-rc1. Looks like we
> are doing uncore_init() on virtualized CPU which breaks boot. 

I think you're the first.. I don't normally use kvm if I can at all
avoid it.

But I think its a 'simple' matter of kvm not emulating the entire
hardware. Afaik the uncore isn't enumerated and we simply assume MSR
presence based on cpu model.

Added Zheng Yan who wrote most of that stuff.

Re: perf uncore & lkvm woes

[Yan, Zheng]

On 08/16/2012 03:19 PM, Peter Zijlstra wrote:
> On Thu, 2012-08-16 at 10:01 +0300, Pekka Enberg wrote:
>> Has anyone seen this? It's kvmtool/next with 3.6.0-rc1. Looks like we
>> are doing uncore_init() on virtualized CPU which breaks boot. 
> 
> I think you're the first.. I don't normally use kvm if I can at all
> avoid it.
> 
> But I think its a 'simple' matter of kvm not emulating the entire
> hardware. Afaik the uncore isn't enumerated and we simply assume MSR
> presence based on cpu model.

The Intel uncore doc does not specify how to check if uncore exist.
How about disabling uncore on virtualized CPU?

Regards
Yan, Zheng

Re: perf uncore & lkvm woes

[Pekka Enberg]

On 08/16/2012 03:19 PM, Peter Zijlstra wrote:
>> On Thu, 2012-08-16 at 10:01 +0300, Pekka Enberg wrote:
>>> Has anyone seen this? It's kvmtool/next with 3.6.0-rc1. Looks like we
>>> are doing uncore_init() on virtualized CPU which breaks boot.
>>
>> I think you're the first.. I don't normally use kvm if I can at all
>> avoid it.
>>
>> But I think its a 'simple' matter of kvm not emulating the entire
>> hardware. Afaik the uncore isn't enumerated and we simply assume MSR
>> presence based on cpu model.

On Thu, Aug 16, 2012 at 10:38 AM, Yan, Zheng
<zheng.z.yan@linux.intel.com> wrote:
> The Intel uncore doc does not specify how to check if uncore exist.
> How about disabling uncore on virtualized CPU?

(CC'ing Avi.)

Re: perf uncore & lkvm woes

[Cyrill Gorcunov]

On Thu, Aug 16, 2012 at 10:41:53AM +0300, Pekka Enberg wrote:
> On 08/16/2012 03:19 PM, Peter Zijlstra wrote:
> >> On Thu, 2012-08-16 at 10:01 +0300, Pekka Enberg wrote:
> >>> Has anyone seen this? It's kvmtool/next with 3.6.0-rc1. Looks like we
> >>> are doing uncore_init() on virtualized CPU which breaks boot.
> >>
> >> I think you're the first.. I don't normally use kvm if I can at all
> >> avoid it.
> >>
> >> But I think its a 'simple' matter of kvm not emulating the entire
> >> hardware. Afaik the uncore isn't enumerated and we simply assume MSR
> >> presence based on cpu model.
> 
> On Thu, Aug 16, 2012 at 10:38 AM, Yan, Zheng
> <zheng.z.yan@linux.intel.com> wrote:
> > The Intel uncore doc does not specify how to check if uncore exist.
> > How about disabling uncore on virtualized CPU?
> 
> (CC'ing Avi.)

Why not simply add bootline option for that? Would it be acceptible?

	Cyrill

Re: perf uncore & lkvm woes

[Avi Kivity]

On 08/16/2012 10:41 AM, Pekka Enberg wrote:
> On 08/16/2012 03:19 PM, Peter Zijlstra wrote:
>>> On Thu, 2012-08-16 at 10:01 +0300, Pekka Enberg wrote:
>>>> Has anyone seen this? It's kvmtool/next with 3.6.0-rc1. Looks like we
>>>> are doing uncore_init() on virtualized CPU which breaks boot.
>>>
>>> I think you're the first.. I don't normally use kvm if I can at all
>>> avoid it.
>>>
>>> But I think its a 'simple' matter of kvm not emulating the entire
>>> hardware. Afaik the uncore isn't enumerated and we simply assume MSR
>>> presence based on cpu model.
> 
> On Thu, Aug 16, 2012 at 10:38 AM, Yan, Zheng
> <zheng.z.yan@linux.intel.com> wrote:
>> The Intel uncore doc does not specify how to check if uncore exist.
>> How about disabling uncore on virtualized CPU?
> 
> (CC'ing Avi.)
> 

Seems reasonable, if unfortunate.  It's pretty easy to check for.

If those are separate MSRs, we can also trap the #GP and exit gracefully.

-- 
error compiling committee.c: too many arguments to function

Re: perf uncore & lkvm woes

[Avi Kivity]

On 08/16/2012 10:46 AM, Cyrill Gorcunov wrote:
> On Thu, Aug 16, 2012 at 10:41:53AM +0300, Pekka Enberg wrote:
>> On 08/16/2012 03:19 PM, Peter Zijlstra wrote:
>> >> On Thu, 2012-08-16 at 10:01 +0300, Pekka Enberg wrote:
>> >>> Has anyone seen this? It's kvmtool/next with 3.6.0-rc1. Looks like we
>> >>> are doing uncore_init() on virtualized CPU which breaks boot.
>> >>
>> >> I think you're the first.. I don't normally use kvm if I can at all
>> >> avoid it.
>> >>
>> >> But I think its a 'simple' matter of kvm not emulating the entire
>> >> hardware. Afaik the uncore isn't enumerated and we simply assume MSR
>> >> presence based on cpu model.
>> 
>> On Thu, Aug 16, 2012 at 10:38 AM, Yan, Zheng
>> <zheng.z.yan@linux.intel.com> wrote:
>> > The Intel uncore doc does not specify how to check if uncore exist.
>> > How about disabling uncore on virtualized CPU?
>> 
>> (CC'ing Avi.)
> 
> Why not simply add bootline option for that? Would it be acceptible?
> 

Most users just install a distro, they don't mess with kernel command lines.


-- 
error compiling committee.c: too many arguments to function

Re: perf uncore & lkvm woes

[Cyrill Gorcunov]

On Thu, Aug 16, 2012 at 11:45:52AM +0300, Avi Kivity wrote:
> >> On Thu, Aug 16, 2012 at 10:38 AM, Yan, Zheng
> >> <zheng.z.yan@linux.intel.com> wrote:
> >> > The Intel uncore doc does not specify how to check if uncore exist.
> >> > How about disabling uncore on virtualized CPU?
> >> 
> >> (CC'ing Avi.)
> > 
> > Why not simply add bootline option for that? Would it be acceptible?
> 
> Most users just install a distro, they don't mess with kernel command lines.

The command line option might be added implicitly in qemu/lkvm.

Re: perf uncore & lkvm woes

[Pekka Enberg]

On Thu, Aug 16, 2012 at 12:06 PM, Cyrill Gorcunov <gorcunov@openvz.org> wrote:
>> Most users just install a distro, they don't mess with kernel command lines.
>
> The command line option might be added implicitly in qemu/lkvm.

That does not make sense for QEMU and we want less mandatory command line
options for LKVM too.

                        Pekka

Re: perf uncore & lkvm woes

[Avi Kivity]

On 08/16/2012 11:40 AM, Avi Kivity wrote:
> On 08/16/2012 10:41 AM, Pekka Enberg wrote:
>> On 08/16/2012 03:19 PM, Peter Zijlstra wrote:
>>>> On Thu, 2012-08-16 at 10:01 +0300, Pekka Enberg wrote:
>>>>> Has anyone seen this? It's kvmtool/next with 3.6.0-rc1. Looks like we
>>>>> are doing uncore_init() on virtualized CPU which breaks boot.
>>>>
>>>> I think you're the first.. I don't normally use kvm if I can at all
>>>> avoid it.
>>>>
>>>> But I think its a 'simple' matter of kvm not emulating the entire
>>>> hardware. Afaik the uncore isn't enumerated and we simply assume MSR
>>>> presence based on cpu model.
>> 
>> On Thu, Aug 16, 2012 at 10:38 AM, Yan, Zheng
>> <zheng.z.yan@linux.intel.com> wrote:
>>> The Intel uncore doc does not specify how to check if uncore exist.
>>> How about disabling uncore on virtualized CPU?
>> 
>> (CC'ing Avi.)
>> 
> 
> Seems reasonable, if unfortunate.  It's pretty easy to check for.
> 
> If those are separate MSRs, we can also trap the #GP and exit gracefully.

Another option is to deal with them on the host side.  That has the
benefit of working with non-Linux guests too.

We can just ignore the MSR and print some warning.

-- 
error compiling committee.c: too many arguments to function

Re: perf uncore & lkvm woes

[Peter Zijlstra]

On Thu, 2012-08-16 at 14:06 +0300, Avi Kivity wrote:

> Another option is to deal with them on the host side.  That has the
> benefit of working with non-Linux guests too.

Right, its an insane amount of MSRs though, but it could be done if
someone takes the time to enumerate them all.

If KVM then simply ignores all writes and returns all 0 on read we can
do the same we do for the regular PMU in check_hw_exists().

> We can just ignore the MSR and print some warning.

If you don't mind printing a warning every time a Linux guest boots ;-)

Re: perf uncore & lkvm woes

[Avi Kivity]

On 08/16/2012 02:17 PM, Peter Zijlstra wrote:
> On Thu, 2012-08-16 at 14:06 +0300, Avi Kivity wrote:
> 
>> Another option is to deal with them on the host side.  That has the
>> benefit of working with non-Linux guests too.
> 
> Right, its an insane amount of MSRs though, but it could be done if
> someone takes the time to enumerate them all.

It's tedious but that's life.

> 
> If KVM then simply ignores all writes and returns all 0 on read we can
> do the same we do for the regular PMU in check_hw_exists().
> 
>> We can just ignore the MSR and print some warning.
> 
> If you don't mind printing a warning every time a Linux guest boots ;-)

We can printk_once() it, or only under debug, now that we have dynamic
debug.  We also need to print the warning only if the counter is
enabled.  Will Linux configure uncore counters by default (why and which?)

-- 
error compiling committee.c: too many arguments to function

Re: perf uncore & lkvm woes

[David Ahern]

On 8/16/12 5:17 AM, Peter Zijlstra wrote:
> If you don't mind printing a warning every time a Linux guest boots ;-)

That feature already exists today for perf related probing. e.g.,

[585929.678746] kvm [10752]: vcpu0 unhandled rdmsr: 0x345
[585929.709870] kvm_set_msr_common: 54 callbacks suppressed
[585929.709986] kvm [10752]: vcpu0 unhandled wrmsr: 0x680 data 0
[585929.710104] kvm [10752]: vcpu0 unhandled wrmsr: 0x6c0 data 0
[585929.710221] kvm [10752]: vcpu0 unhandled wrmsr: 0x681 data 0
[585929.710352] kvm [10752]: vcpu0 unhandled wrmsr: 0x6c1 data 0
[585929.710467] kvm [10752]: vcpu0 unhandled wrmsr: 0x682 data 0
[585929.710581] kvm [10752]: vcpu0 unhandled wrmsr: 0x6c2 data 0
[585929.710707] kvm [10752]: vcpu0 unhandled wrmsr: 0x683 data 0
[585929.710822] kvm [10752]: vcpu0 unhandled wrmsr: 0x6c3 data 0
[585929.710937] kvm [10752]: vcpu0 unhandled wrmsr: 0x684 data 0
[585929.711052] kvm [10752]: vcpu0 unhandled wrmsr: 0x6c4 data 0


David

Re: perf uncore & lkvm woes

[Yan, Zheng]

On 08/16/2012 07:25 PM, Avi Kivity wrote:
> On 08/16/2012 02:17 PM, Peter Zijlstra wrote:
>> On Thu, 2012-08-16 at 14:06 +0300, Avi Kivity wrote:
>>
>>> Another option is to deal with them on the host side.  That has the
>>> benefit of working with non-Linux guests too.
>>
>> Right, its an insane amount of MSRs though, but it could be done if
>> someone takes the time to enumerate them all.
> 
> It's tedious but that's life.
> 
>>
>> If KVM then simply ignores all writes and returns all 0 on read we can
>> do the same we do for the regular PMU in check_hw_exists().
>>
>>> We can just ignore the MSR and print some warning.
>>
>> If you don't mind printing a warning every time a Linux guest boots ;-)
> 
> We can printk_once() it, or only under debug, now that we have dynamic
> debug.  We also need to print the warning only if the counter is
> enabled.  Will Linux configure uncore counters by default (why and which?)
> 

The uncore driver may write some global control MSRs during initialization.
I did it because of its simpleness.

Peter, do I need to submit a patch disables uncore on virtualized CPU?

Regards
Yan, Zheng

Re: perf uncore & lkvm woes

[Peter Zijlstra]

On Fri, 2012-08-17 at 09:40 +0800, Yan, Zheng wrote:
> 
> Peter, do I need to submit a patch disables uncore on virtualized CPU?
> 
I think Avi prefers the method where KVM 'fakes' the MSRs and we have to
detect if the MSRs actually work or not.

If you're willing to have a go at that, please do so. If you're not sure
how to do the KVM part, I'm sure Avi and/or Gleb can help you out.

Re: perf uncore & lkvm woes

[Avi Kivity]

On 08/17/2012 09:56 AM, Peter Zijlstra wrote:
> On Fri, 2012-08-17 at 09:40 +0800, Yan, Zheng wrote:
>> 
>> Peter, do I need to submit a patch disables uncore on virtualized CPU?
>> 
> I think Avi prefers the method where KVM 'fakes' the MSRs and we have to
> detect if the MSRs actually work or not.

s/we have/we don't have/.

> 
> If you're willing to have a go at that, please do so. If you're not sure
> how to do the KVM part, I'm sure Avi and/or Gleb can help you out.

Certainly, please see kvm_pmu_get_msr() and kvm_pmu_set_msr().

The approach is that if an msr write can be emulated correctly (for
example, it disables a counter) then we let it proceed; if it cannot be
emulated correctly (for example it enables a counter that we cannot
emulate), then we ignore it, but print out a message that tells the user
that we're faking something that may cause the guest to malfunction.


-- 
error compiling committee.c: too many arguments to function

Re: perf uncore & lkvm woes

[Yan, Zheng]

On 08/19/2012 05:55 PM, Avi Kivity wrote:
> On 08/17/2012 09:56 AM, Peter Zijlstra wrote:
>> On Fri, 2012-08-17 at 09:40 +0800, Yan, Zheng wrote:
>>>
>>> Peter, do I need to submit a patch disables uncore on virtualized CPU?
>>>
>> I think Avi prefers the method where KVM 'fakes' the MSRs and we have to
>> detect if the MSRs actually work or not.
> 
> s/we have/we don't have/.
> 
>>
>> If you're willing to have a go at that, please do so. If you're not sure
>> how to do the KVM part, I'm sure Avi and/or Gleb can help you out.
> 
> Certainly, please see kvm_pmu_get_msr() and kvm_pmu_set_msr().
> 
> The approach is that if an msr write can be emulated correctly (for
> example, it disables a counter) then we let it proceed; if it cannot be
> emulated correctly (for example it enables a counter that we cannot
> emulate), then we ignore it, but print out a message that tells the user
> that we're faking something that may cause the guest to malfunction.
> 

There is only one kvm_pmu structure in struct kvm_vcpu_arch, but the uncore
driver may define dozens of PMUs. Besides the uncore PMUs make extensive use
of extra registers, I don't think we can store these information in kvm_pmu
structure.

The uncore pmu collects system-wide events on a given socket, it may not be
possible to be simulated by virtualized CPU. I think it's better to just
disable uncore on virtualized CPU.

Regards
Yan, Zheng

Re: perf uncore & lkvm woes

[Yan, Zheng]

On 08/19/2012 05:55 PM, Avi Kivity wrote:
> On 08/17/2012 09:56 AM, Peter Zijlstra wrote:
>> On Fri, 2012-08-17 at 09:40 +0800, Yan, Zheng wrote:
>>>
>>> Peter, do I need to submit a patch disables uncore on virtualized CPU?
>>>
>> I think Avi prefers the method where KVM 'fakes' the MSRs and we have to
>> detect if the MSRs actually work or not.
> 
> s/we have/we don't have/.
> 
>>
>> If you're willing to have a go at that, please do so. If you're not sure
>> how to do the KVM part, I'm sure Avi and/or Gleb can help you out.
> 
> Certainly, please see kvm_pmu_get_msr() and kvm_pmu_set_msr().
> 
> The approach is that if an msr write can be emulated correctly (for
> example, it disables a counter) then we let it proceed; if it cannot be
> emulated correctly (for example it enables a counter that we cannot
> emulate), then we ignore it, but print out a message that tells the user
> that we're faking something that may cause the guest to malfunction.
> 
> 

Anyone knows how to detect if the kernel is running on virtualized CPU?

Regards
Yan, Zheng

Re: perf uncore & lkvm woes

[Avi Kivity]

On 08/20/2012 08:30 AM, Yan, Zheng wrote:
> On 08/19/2012 05:55 PM, Avi Kivity wrote:
>> On 08/17/2012 09:56 AM, Peter Zijlstra wrote:
>>> On Fri, 2012-08-17 at 09:40 +0800, Yan, Zheng wrote:
>>>>
>>>> Peter, do I need to submit a patch disables uncore on virtualized CPU?
>>>>
>>> I think Avi prefers the method where KVM 'fakes' the MSRs and we have to
>>> detect if the MSRs actually work or not.
>> 
>> s/we have/we don't have/.
>> 
>>>
>>> If you're willing to have a go at that, please do so. If you're not sure
>>> how to do the KVM part, I'm sure Avi and/or Gleb can help you out.
>> 
>> Certainly, please see kvm_pmu_get_msr() and kvm_pmu_set_msr().
>> 
>> The approach is that if an msr write can be emulated correctly (for
>> example, it disables a counter) then we let it proceed; if it cannot be
>> emulated correctly (for example it enables a counter that we cannot
>> emulate), then we ignore it, but print out a message that tells the user
>> that we're faking something that may cause the guest to malfunction.
>> 
>> 
> 
> Anyone knows how to detect if the kernel is running on virtualized CPU?

boot_cpu_has(X86_FEATURE_HYPERVISOR)


-- 
error compiling committee.c: too many arguments to function

Re: perf uncore & lkvm woes

[Avi Kivity]

On 08/20/2012 07:15 AM, Yan, Zheng wrote:
> On 08/19/2012 05:55 PM, Avi Kivity wrote:
>> On 08/17/2012 09:56 AM, Peter Zijlstra wrote:
>>> On Fri, 2012-08-17 at 09:40 +0800, Yan, Zheng wrote:
>>>>
>>>> Peter, do I need to submit a patch disables uncore on virtualized CPU?
>>>>
>>> I think Avi prefers the method where KVM 'fakes' the MSRs and we have to
>>> detect if the MSRs actually work or not.
>> 
>> s/we have/we don't have/.
>> 
>>>
>>> If you're willing to have a go at that, please do so. If you're not sure
>>> how to do the KVM part, I'm sure Avi and/or Gleb can help you out.
>> 
>> Certainly, please see kvm_pmu_get_msr() and kvm_pmu_set_msr().
>> 
>> The approach is that if an msr write can be emulated correctly (for
>> example, it disables a counter) then we let it proceed; if it cannot be
>> emulated correctly (for example it enables a counter that we cannot
>> emulate), then we ignore it, but print out a message that tells the user
>> that we're faking something that may cause the guest to malfunction.
>> 
> 
> There is only one kvm_pmu structure in struct kvm_vcpu_arch, but the uncore
> driver may define dozens of PMUs. Besides the uncore PMUs make extensive use
> of extra registers, I don't think we can store these information in kvm_pmu
> structure.

We don't need to store any information, just respond to those MSRs (and
ignore them).

> The uncore pmu collects system-wide events on a given socket, it may not be
> possible to be simulated by virtualized CPU. I think it's better to just
> disable uncore on virtualized CPU.

That only works for Linux guests.

-- 
error compiling committee.c: too many arguments to function

Re: perf uncore & lkvm woes

[Yan, Zheng]

On 08/20/2012 04:49 PM, Avi Kivity wrote:
> On 08/20/2012 07:15 AM, Yan, Zheng wrote:
>> On 08/19/2012 05:55 PM, Avi Kivity wrote:
>>> On 08/17/2012 09:56 AM, Peter Zijlstra wrote:
>>>> On Fri, 2012-08-17 at 09:40 +0800, Yan, Zheng wrote:
>>>>>
>>>>> Peter, do I need to submit a patch disables uncore on virtualized CPU?
>>>>>
>>>> I think Avi prefers the method where KVM 'fakes' the MSRs and we have to
>>>> detect if the MSRs actually work or not.
>>>
>>> s/we have/we don't have/.
>>>
>>>>
>>>> If you're willing to have a go at that, please do so. If you're not sure
>>>> how to do the KVM part, I'm sure Avi and/or Gleb can help you out.
>>>
>>> Certainly, please see kvm_pmu_get_msr() and kvm_pmu_set_msr().
>>>
>>> The approach is that if an msr write can be emulated correctly (for
>>> example, it disables a counter) then we let it proceed; if it cannot be
>>> emulated correctly (for example it enables a counter that we cannot
>>> emulate), then we ignore it, but print out a message that tells the user
>>> that we're faking something that may cause the guest to malfunction.
>>>
>>
>> There is only one kvm_pmu structure in struct kvm_vcpu_arch, but the uncore
>> driver may define dozens of PMUs. Besides the uncore PMUs make extensive use
>> of extra registers, I don't think we can store these information in kvm_pmu
>> structure.
> 
> We don't need to store any information, just respond to those MSRs (and
> ignore them).

The problem is that there are hundreds of MSRs, but only a few of them are
defined in the header file of uncore driver. The uncore driver computes
addresses of the rest MSRs by using uncore box index and counter index.

Regards
Yan, Zheng
> 
>> The uncore pmu collects system-wide events on a given socket, it may not be
>> possible to be simulated by virtualized CPU. I think it's better to just
>> disable uncore on virtualized CPU.
> 
> That only works for Linux guests.
> 

Re: perf uncore & lkvm woes

[Peter Zijlstra]

On Sun, 2012-08-19 at 12:55 +0300, Avi Kivity wrote:
> > I think Avi prefers the method where KVM 'fakes' the MSRs and we have to
> > detect if the MSRs actually work or not.
> 
> s/we have/we don't have/. 

So for the 'normal' PMU we actually do check to see if the MSRs are
being faked and bail if they are.

Re: perf uncore & lkvm woes

[Avi Kivity]

On 08/21/2012 04:11 AM, Yan, Zheng wrote:

>>> There is only one kvm_pmu structure in struct kvm_vcpu_arch, but the uncore
>>> driver may define dozens of PMUs. Besides the uncore PMUs make extensive use
>>> of extra registers, I don't think we can store these information in kvm_pmu
>>> structure.
>> 
>> We don't need to store any information, just respond to those MSRs (and
>> ignore them).
> 
> The problem is that there are hundreds of MSRs, but only a few of them are
> defined in the header file of uncore driver. The uncore driver computes
> addresses of the rest MSRs by using uncore box index and counter index.

You can do the same in kvm.  Maybe even move the functions that compute
the addresses to a header, so that they can be shared.

Can you point me at the code?  Maybe I'm misunderstanding something here.


-- 
error compiling committee.c: too many arguments to function

Re: perf uncore & lkvm woes

[Avi Kivity]

On 08/21/2012 10:11 AM, Peter Zijlstra wrote:
> On Sun, 2012-08-19 at 12:55 +0300, Avi Kivity wrote:
>> > I think Avi prefers the method where KVM 'fakes' the MSRs and we have to
>> > detect if the MSRs actually work or not.
>> 
>> s/we have/we don't have/. 
> 
> So for the 'normal' PMU we actually do check to see if the MSRs are
> being faked and bail if they are.

That was because earlier versions of kvm did not virtualize the pmu.

The approaches are not mutually exclusive.  We can check in the guest,
and fake it in the host.

The problem with faking it in the host is if someone actually relies on
the pmu for something, not just instrumentation.  We do that for the
watchdog, but I don't see it happening with the uncore pmu.


-- 
error compiling committee.c: too many arguments to function

Re: perf uncore & lkvm woes

[Yan, Zheng]

On 08/21/2012 04:32 PM, Avi Kivity wrote:
> On 08/21/2012 04:11 AM, Yan, Zheng wrote:
> 
>>>> There is only one kvm_pmu structure in struct kvm_vcpu_arch, but the uncore
>>>> driver may define dozens of PMUs. Besides the uncore PMUs make extensive use
>>>> of extra registers, I don't think we can store these information in kvm_pmu
>>>> structure.
>>>
>>> We don't need to store any information, just respond to those MSRs (and
>>> ignore them).
>>
>> The problem is that there are hundreds of MSRs, but only a few of them are
>> defined in the header file of uncore driver. The uncore driver computes
>> addresses of the rest MSRs by using uncore box index and counter index.
> 
> You can do the same in kvm.  Maybe even move the functions that compute
> the addresses to a header, so that they can be shared.
> 
> Can you point me at the code?  Maybe I'm misunderstanding something here.
> 

The code are located at linux/arch/x86/kernel/cpu/perf_event_intel_uncore.{h,c}.

Regards
Yan, Zheng

Re: perf uncore & lkvm woes

[Peter Zijlstra]

On Tue, 2012-08-21 at 11:34 +0300, Avi Kivity wrote:
> On 08/21/2012 10:11 AM, Peter Zijlstra wrote:
> > On Sun, 2012-08-19 at 12:55 +0300, Avi Kivity wrote:
> >> > I think Avi prefers the method where KVM 'fakes' the MSRs and we have to
> >> > detect if the MSRs actually work or not.
> >> 
> >> s/we have/we don't have/. 
> > 
> > So for the 'normal' PMU we actually do check to see if the MSRs are
> > being faked and bail if they are.
> 
> That was because earlier versions of kvm did not virtualize the pmu.
> 
> The approaches are not mutually exclusive.  We can check in the guest,
> and fake it in the host.

This is actually what I proposed.

> The problem with faking it in the host is if someone actually relies on
> the pmu for something, not just instrumentation.  We do that for the
> watchdog, but I don't see it happening with the uncore pmu.

Agreed, although from a usability POV its nicer to refuse the
device/events than to pretend it works while it doesn't.

Anyway, for now I've taken Zheng Yan's cpu_has_hypervisor patch, we can
always revisit this.

Re: perf uncore & lkvm woes

[Avi Kivity]

On 08/21/2012 01:35 PM, Peter Zijlstra wrote:
> On Tue, 2012-08-21 at 11:34 +0300, Avi Kivity wrote:
>> On 08/21/2012 10:11 AM, Peter Zijlstra wrote:
>> > On Sun, 2012-08-19 at 12:55 +0300, Avi Kivity wrote:
>> >> > I think Avi prefers the method where KVM 'fakes' the MSRs and we have to
>> >> > detect if the MSRs actually work or not.
>> >> 
>> >> s/we have/we don't have/. 
>> > 
>> > So for the 'normal' PMU we actually do check to see if the MSRs are
>> > being faked and bail if they are.
>> 
>> That was because earlier versions of kvm did not virtualize the pmu.
>> 
>> The approaches are not mutually exclusive.  We can check in the guest,
>> and fake it in the host.
> 
> This is actually what I proposed.

Ah, I misunderstood you.

> 
>> The problem with faking it in the host is if someone actually relies on
>> the pmu for something, not just instrumentation.  We do that for the
>> watchdog, but I don't see it happening with the uncore pmu.
> 
> Agreed, although from a usability POV its nicer to refuse the
> device/events than to pretend it works while it doesn't.

If/when we virtualize this pmu we can expose a flag that says "the pmu
actually works even though this is a virtual machine".

> Anyway, for now I've taken Zheng Yan's cpu_has_hypervisor patch, we can
> always revisit this.

Yup.

-- 
error compiling committee.c: too many arguments to function