Getting started with perf; a couple of questions

Getting started with perf; a couple of questions

[Andrew Davidoff]

Hi,

I am getting started with perf and have a couple of questions.

* Does the version of perf being used need to be built from the source
for the running kernel, or is it supported to use a build from a newer
kernel source? I am currently running perf built from 3.9.4 kernel
sources on a machine running kernel 2.6.32.

* I am getting event output with a specific `perf stat` invocation,
but when I run the same invocation under `perf record`, `perf report`
complains "The perf.data file has no samples". Why would that be? The
invocation looks like this:

perf record -e cache-misses,page-faults,minor-faults,major-faults,cpu-migrations,L1-dcache-load-misses,L1-dcache-store-misses,L1-icache-load-misses,LLC-load-misses,LLC-store-misses,dTLB-load-misses,dTLB-store-misses,iTLB-load-misses
-- {program} {arg} {arg} ... {arg}


Any info/advice is appreciated.

Thanks.
Andy

Re: Getting started with perf; a couple of questions

[David Ahern]

On 5/30/13 11:26 PM, Andrew Davidoff wrote:
> Hi,
>
> I am getting started with perf and have a couple of questions.
>
> * Does the version of perf being used need to be built from the source
> for the running kernel, or is it supported to use a build from a newer
> kernel source? I am currently running perf built from 3.9.4 kernel
> sources on a machine running kernel 2.6.32.

In general, perf userspace from upstream can work with any kernel 
version. The only problem I can recall is 2.6.31 -- ie., upstream does 
not work with 2.6.31 and prior.

>
> * I am getting event output with a specific `perf stat` invocation,
> but when I run the same invocation under `perf record`, `perf report`
> complains "The perf.data file has no samples". Why would that be? The
> invocation looks like this:
>
> perf record -e cache-misses,page-faults,minor-faults,major-faults,cpu-migrations,L1-dcache-load-misses,L1-dcache-store-misses,L1-icache-load-misses,LLC-load-misses,LLC-store-misses,dTLB-load-misses,dTLB-store-misses,iTLB-load-misses
> -- {program} {arg} {arg} ... {arg}

add a -v and see if more info comes out.

perf script can be used to dump events.

David

Re: Getting started with perf; a couple of questions

[Andrew Davidoff]

Hi,

Thanks for the quick response.

> add a -v and see if more info comes out.
>
> perf script can be used to dump events.

I tried again with -v on record, but I am still not getting any event
samples in perf.data. Looking at an ASCII dump with report -D, I see
43 PERF_RECORD_MMAP entries, each of which appears to be related to a
kernel module, but nothing for any of the events I have selected to
sample (each is simply listed like "page-faults stats:").

The run ends like this every time:

[ perf record: Woken up 1 times to write data ]
[ perf record: Captured and wrote 0.004 MB perf.data (~194 samples) ]

"Woken up 1 times" seemed suspicious, if record should be sampling at
1000Hz. I tried using -F 1000 and adding cycles to my events list both
in an effort to get 1000Hz samples, but as far as I could tell, there
was no change in the recorded data.

Also note that when using -v with record, I get the message "failed to
write feature 2" at the very end of the run output. I do not get that
message without -v. I don't know what it's trying to tell me and
haven't looked at the source code yet.

The command I am running is a threaded python script (spawns workers).
As I understand the documentation, I shouldn't need any additional
flags for perf to collect data on all the threads, since it reads like
counter inheritance is on by default. In other words, I believe perf
record should be working in what is documented as "per-process" mode
by default. I tried using -i with record in case I wasn't
understanding correctly, but that didn't change anything.

I will play with this more tomorrow but any further advice is
appreciated. I feel like I'm probably just missing a switch or
something that's not entirely obvious from the scant documentation I
have found.

Thanks again.
Andy

Re: Getting started with perf; a couple of questions

[Andrew Davidoff]

On Fri, May 31, 2013 at 1:09 AM, Andrew Davidoff <davidoff@qedmf.net> wrote:
> I will play with this more tomorrow but any further advice is
> appreciated. I feel like I'm probably just missing a switch or
> something that's not entirely obvious from the scant documentation I
> have found.

After struggling a bit more with this on the original host:

Ubuntu 10.04.4 LTS
kernel 2.6.32-47-server
perf versions 3.9.4 and 3.8.13

with no change in results, I gave it a shot on a CentOS 6 virtual
machine on my Mac:

CentOS release 6.4 (Final)
kernel 3.9.4 AND kernel 2.6.32-358.2.1.el6.x86_64
perf version 3.9.4

And perf record is working as I'd expect it to. To be clear about the
description above, under CentOS,  perf record 3.9.4 is recording
samples whether I'm running kernel 3.9.4 or 2.6.32-358.2.1.el6.x86_64.

So...Ubuntu thing? Or maybe just a 10.04 thing (yeah, I know it's old).

Maybe this'll help someone.

Thanks all.
Andy

Re: Getting started with perf; a couple of questions

[David Ahern]

On 5/31/13 2:35 PM, Andrew Davidoff wrote:
> On Fri, May 31, 2013 at 1:09 AM, Andrew Davidoff <davidoff@qedmf.net> wrote:
>> I will play with this more tomorrow but any further advice is
>> appreciated. I feel like I'm probably just missing a switch or
>> something that's not entirely obvious from the scant documentation I
>> have found.
>
> After struggling a bit more with this on the original host:
>
> Ubuntu 10.04.4 LTS
> kernel 2.6.32-47-server
> perf versions 3.9.4 and 3.8.13
>
> with no change in results, I gave it a shot on a CentOS 6 virtual
> machine on my Mac:
>
> CentOS release 6.4 (Final)
> kernel 3.9.4 AND kernel 2.6.32-358.2.1.el6.x86_64
> perf version 3.9.4
>
> And perf record is working as I'd expect it to. To be clear about the
> description above, under CentOS,  perf record 3.9.4 is recording
> samples whether I'm running kernel 3.9.4 or 2.6.32-358.2.1.el6.x86_64.

It's a VM and not running on KVM so there is no PMU which means the -e 
cycles is most likely falling back to -e cpu-clock. You should see this 
in a message with the -v flag on record. perf-script should dump out the 
events and show cpu-clock versus cycles.

Based on that I suggest trying -e cpu-clock on your original host. If 
that works then something is messed up with the pmu config on the host.

David
>
> So...Ubuntu thing? Or maybe just a 10.04 thing (yeah, I know it's old).
>
> Maybe this'll help someone.
>
> Thanks all.
> Andy
>

Re: Getting started with perf; a couple of questions

[Andrew Davidoff]

On Thu, Jun 6, 2013 at 8:01 AM, David Ahern <dsahern@gmail.com> wrote:
>
> It's a VM and not running on KVM so there is no PMU which means the -e
> cycles is most likely falling back to -e cpu-clock. You should see this in a
> message with the -v flag on record. perf-script should dump out the events
> and show cpu-clock versus cycles.
>
> Based on that I suggest trying -e cpu-clock on your original host. If that
> works then something is messed up with the pmu config on the host.

Thanks. It wasn't clear to me if a VM had the same access to the PMU
that a non-virtualized host would. But I guess that might explain why
a bunch of events I'd expect samples for didn't have any (ex:
iTLB-load-misses), even on the VM that did record some samples?

I re-ran the record on the VM with -v and did see the message about
falling back to cpu-clock-ticks, but only after I added the cycles
even to my events list. Which actually leads into a question regarding
event sampling.

The documentation I have found mentions a "default sampling event"
(cycles) but doesn't explain the concept of a sampling event in
detail. Does each event listed with -e have its own sample period/rate
that can be adjusted? Or do -c or -F apply to all events sampled? Even
I don't sample cycles, is the default sampling rate still 1000Hz for
all events sampled in the absence of -c or -F?

Regarding using -e cpu-clock on the original host, that doesn't appear
to have changed the perf record behavior. Neither does using -e
cycles. And even with -v on record, I didn't see any additional
messaging (like fall back to cpu-clock). But I do still get sample
data if I run with -a. *shrug*. Not a big deal at this point - running
perf on this particular host/OS might not be something I need to do,
so for now I can drop it. But if you or anyone else is interested in
continuing to troubleshoot it, I'd be glad to assist.

Andy

>
> David
>
>>
>> So...Ubuntu thing? Or maybe just a 10.04 thing (yeah, I know it's old).
>>
>> Maybe this'll help someone.
>>
>> Thanks all.
>> Andy
>>
>

Re: Getting started with perf; a couple of questions

[David Ahern]

On 6/6/13 12:26 PM, Andrew Davidoff wrote:
> On Thu, Jun 6, 2013 at 8:01 AM, David Ahern <dsahern@gmail.com> wrote:
>>
>> It's a VM and not running on KVM so there is no PMU which means the -e
>> cycles is most likely falling back to -e cpu-clock. You should see this in a
>> message with the -v flag on record. perf-script should dump out the events
>> and show cpu-clock versus cycles.
>>
>> Based on that I suggest trying -e cpu-clock on your original host. If that
>> works then something is messed up with the pmu config on the host.
>
> Thanks. It wasn't clear to me if a VM had the same access to the PMU
> that a non-virtualized host would. But I guess that might explain why
> a bunch of events I'd expect samples for didn't have any (ex:
> iTLB-load-misses), even on the VM that did record some samples?
>

The hypervisor needs to expose a PMU to the guest. AFAIK KVM is the only 
one that does that. Without the PMU you do not get H/W based events.

> I re-ran the record on the VM with -v and did see the message about
> falling back to cpu-clock-ticks, but only after I added the cycles
> even to my events list. Which actually leads into a question regarding
> event sampling.

So perf record -v does not show the fallback? The default event has been 
'cycles' from the beginning I believe. Only if cycles is not supported 
does it fallback to cpu-clock. You should not have to add '-e cycles'. 
Well, at least that is the way it is for the code in Linus' tree.

>
> The documentation I have found mentions a "default sampling event"
> (cycles) but doesn't explain the concept of a sampling event in
> detail. Does each event listed with -e have its own sample period/rate
> that can be adjusted? Or do -c or -F apply to all events sampled? Even
> I don't sample cycles, is the default sampling rate still 1000Hz for
> all events sampled in the absence of -c or -F?

Unless it crept it with some changes Jiri made (I have not been using 
the fancy -e declarations) I believe there is a single -c/-F argument 
and it applies to all events. ie., all events have the same sampling 
interval.

The default is 1000 Hz (equivalent to -F 1000). -c specifies the period 
-- which is the similar to specifying -F.

     OPT_U64('c', "count", &record.opts.user_interval, "event period to 
sample"),
...
     OPT_UINTEGER('F', "freq", &record.opts.user_freq, "profile at this 
frequency"),


And then configuring the attributes for the event only 1 of the 2 is 
used with frequency taking precedent:

    if (!attr->sample_period || (opts->user_freq != UINT_MAX &&
                      opts->user_interval != ULLONG_MAX)) {
         if (opts->freq) {
             perf_evsel__set_sample_bit(evsel, PERIOD);
             attr->freq      = 1;
             attr->sample_freq   = opts->freq;
         } else {
             attr->sample_period = opts->default_interval;
         }
     }

>
> Regarding using -e cpu-clock on the original host, that doesn't appear
> to have changed the perf record behavior. Neither does using -e
> cycles. And even with -v on record, I didn't see any additional
> messaging (like fall back to cpu-clock). But I do still get sample
> data if I run with -a. *shrug*.

What command lines are you running? I seem to recall this is a 2.6.32 
kernel and basic perf record commands should work fine -- perf record -a 
or perf record -p <pid>. Perhaps something snuck in recently that broke 
compat with 2.6.32. You can determine that by running the perf command 
that comes with that version of Ubuntu and then newer ones -- 3.0, 3.4, 
3.8 would be a good distribution of versions to start with if you want 
to get to the bottom of it.

David

Re: Getting started with perf; a couple of questions

[Andrew Davidoff]

On Fri, Jun 7, 2013 at 9:05 AM, David Ahern <dsahern@gmail.com> wrote:
>
> So perf record -v does not show the fallback? The default event has been
> 'cycles' from the beginning I believe. Only if cycles is not supported does
> it fallback to cpu-clock. You should not have to add '-e cycles'. Well, at
> least that is the way it is for the code in Linus' tree.

The perf record invocation I am using in the CentOS 6.4 VM is in the
form of `perf record -v -e event0,...eventN <command-to-trace>`, and
perf is built from 3.9.4 sources from kernel.org, and the running
kernel is built from the same sources. In this environment If I pass
events to -e without including cycles, I do not get the message about
falling back to cpu-clock. If I do include cycles in the event list,
or do not provide an event list at all, I do get the message about
falling back to cpu-clock.

Thank you for the clarification regarding -c and -F / sampling rate/period.

> What command lines are you running? I seem to recall this is a 2.6.32 kernel
> and basic perf record commands should work fine -- perf record -a or perf
> record -p <pid>. Perhaps something snuck in recently that broke compat with
> 2.6.32. You can determine that by running the perf command that comes with
> that version of Ubuntu and then newer ones -- 3.0, 3.4, 3.8 would be a good
> distribution of versions to start with if you want to get to the bottom of
> it.

You are correct that the Ubuntu 10.04 host that doesn't seem to be
recording properly is running a 2.6.32 kernel (2.6.32-47-server from
Ubuntu). The invocation of perf record is the same as described for
the CentOS VM (i.e. I am not using -p, and would like to not use -a,
though -a is the only way I'm getting samples recorded for
<command-to-trace>).

I had moved to building a newer version of perf than the one that came
from the Ubuntu 10.04 repo (linux-tools-common) because that one
appeared ancient based on the output it was generating, and reports
version as 0.0.2.PERF (though the binary is installed as
/usr/bin/perf_2.6.32-47). perf record does record data with that
version, but it looks like (via perf report) that it's data only for
one event even when multiple are passed, and that event isn't labeled
in the report, so at a glance I am not sure which it is.

Anyway, for now I'm not worrying about getting this working on this
particular version of Ubuntu, but I appreciate all your help. If I
decide to dig further I'll start iterating through the versions you
suggested and see where things get extra broken.

Andy

Re: Getting started with perf; a couple of questions

[David Ahern]

On 6/7/13 5:49 PM, Andrew Davidoff wrote:

> You are correct that the Ubuntu 10.04 host that doesn't seem to be
> recording properly is running a 2.6.32 kernel (2.6.32-47-server from
> Ubuntu). The invocation of perf record is the same as described for
> the CentOS VM (i.e. I am not using -p, and would like to not use -a,
> though -a is the only way I'm getting samples recorded for
> <command-to-trace>).

CentOS 6 has newer perf code than its kernel version suggests -- 
compliments of the RHEL backporting Jiri does.

>
> I had moved to building a newer version of perf than the one that came
> from the Ubuntu 10.04 repo (linux-tools-common) because that one
> appeared ancient based on the output it was generating, and reports
> version as 0.0.2.PERF (though the binary is installed as
> /usr/bin/perf_2.6.32-47). perf record does record data with that
> version, but it looks like (via perf report) that it's data only for
> one event even when multiple are passed, and that event isn't labeled
> in the report, so at a glance I am not sure which it is.

If the original perf command works but newer versions do not then the 
compatibility wheels have fallen off somewhere along the line.

Most likely CentOS 6 works b/c of the backports.

David

>
> Anyway, for now I'm not worrying about getting this working on this
> particular version of Ubuntu, but I appreciate all your help. If I
> decide to dig further I'll start iterating through the versions you
> suggested and see where things get extra broken.
>
> Andy
>