Re: Instruction/Cycle Counting in Guest Using the Kvm PMU

[James Morse <james.morse@arm.com>]

Hi Jan,

On 23/11/2018 12:29, James Morse wrote:
> On 23/11/2018 09:36, Jan Bolke wrote:
>> I am using the Kvm Api and try to integrate it as an instruction set simulator
>> in a SystemC environment.
> 
> 
>> I need some mechanism to count executed instructions in the guest (or cycles).
>>
>> Currently I am trying to use the emulated PMU cycle counter in the guest to get
>> the number of executed cycles in the guest.
>>
>> I am working on Arm64 and use Linux Kernel 4.14.33.
>>
>> I create the PMU device without creating a in-kernel vgic.
> 
>> I configure the counter, then start the counter, execute 3 or 4 dummy
>> instructions and read the counter again and then exit the guest with an exit_mmio.
>>
>> I assumed the value should be a very small number, as the guest only executed a
>> few instructions.
> 
> (some of which are system register writes, which can take a long time)
> 
>> The thing is as I read the counter, the value is something like 2970 or 0
>> (changes in each run).

> You are missing some barriers in your assembly snippet. 0 is a good indication
> that the code you wanted to measure escaped the measurement-window!

This would be true on bare-metal, but for KVM I'm talking rubbish. Turns out KVM
traps all these registers, and emulates the lot.

kvm_arm_setup_debug() sets MDCR_EL2.TPM so almost all these accesses trap,
effectively putting an ISB after each one.


>> So to me it looks like the counter is also counting the cycles for instruction
>> emulation in the host, am I right?
> 
> I'd assume not, but I don't know anything about the PMU.
> Andrew Murray posted a series[0] that did some stuff with starting/stopping the
> the counters around the guest, but I think that was just for the host making
> measurements of itself, or the guest.

> It might be easier to count instructions from outside the guest using perf. I
> think Andrew's series is making that more reliable.

>> Is it possible to just count the cycles in the guest from the guests’s point of
>> view?

This is what KVM is doing, by emulating the PMU with perf.
kvm_pmu_get_counter_value() looks like it should be doing the right thing...

As a sanity check, does the cycle counter work on your test machine?:
| perf stat -e cycles -- ls



Thanks,

James