Question about Perf's handling of in-use performance counters

[Taylor Andrews <andrewst@vmware.com>]

Hi all,

From what I have seen, this list seems full of perf experts, so I am hoping one or more experts will be able to help me understand some perf behavior we are seeing.

First some background:

VMware's virtual x86 performance counter implementation aims to expose in-use (unavailable) performance counters to the guest operating system in the hopes that software agents will recognize it as an "in-use" resource and follow the PMU sharing guidelines outlined in Intel's Performance Monitoring Unit Sharing Guide (https://software.intel.com/en-us/articles/performance-monitoring-unit-guidelines/).  There is also a VMware-based mechanism to force virtual performance counters to be exposed to the guest operating system as in-use.  "In-use" is defined in the sharing guidelines as the enable bits being found to be set, either in the general purpose PMC's event select MSR, or in the case of fixed function counters, in the Fixed Counter Control MSR.

The Linux PMU driver looks like it currently complains about the BIOS being "broken" if it finds any counters are in-use by it, but it still successfully initializes: https://github.com/torvalds/linux/blob/a5ebe0ba3dff658c5286e8d5f20e4328f719d5a3/arch/x86/kernel/cpu/perf_event.c#L181


The odd behavior:

When running perf in a VMware VM with virtualized x86 performance counters, we sometimes see messages like the following in the log file:

2016-10-10T15:22:46.434-04:00| vcpu-1| I125: VPMC: The guest wrote to the event selector of in-use virtual performance counter 0, which is disallowed.
2016-10-10T15:22:46.435-04:00| vcpu-1| I125: VPMC: The guest wrote to the event selector of in-use virtual performance counter 0, which is disallowed.
2016-10-10T15:22:46.436-04:00| vcpu-1| I125: VPMC: The guest wrote to the event selector of in-use virtual performance counter 0, which is disallowed.

The VMware virtual hardware platform software drops writes to virtual counters it exposes as in-use, and logs warnings similar to above.

We have tried perf 3.13.11-ckt39 on Linux kernel 3.13.0-92-generic, and perf 4.8.1 on Linux kernel  4.8.1.  In both cases, when we forcibly mark counters as in-use, we see warnings similar to above. When this happens, we also see perf sometimes report counts of 0 for the events it was attempting to count.  

For example, rotate1 is a simple c++ binary that loops and should advance cycles and instructions counts.  The following perf 3.13.11-ckt39 run was performed in a VMware VM x86 that has VPMC enabled on Linux kernel 3.13.0-92-generic, and all of the generic counters were forcibly marked as in-use. 

$ perf stat -e cycles -e instructions ./rotate1 10000
Took 9.519990 seconds

 Performance counter stats for './rotate1 10000':

                 0 cycles
                 0 instructions

      17.213091320 seconds time elapsed

By looking at these warnings, naively, it would appear perf is trying to use counters that are marked as in-use.  I would like to investigate if this is expected perf behavior, or unexpected perf behavior.  I have no knowledge about what perf's approach is to handle in-use counters, but I am hoping someone here might.  For example, if perf finds the Linux PMU driver has successfully initialized, does it assume it has authoritative access to all the counters?  If anyone has specific knowledge about how perf treats x86 performance counters that it finds as "in-use" (such as counters that are being used by BIOS, or host operating system), could you please reach out to me?

Thanks,
Taylor