[PATCH] perf, x86: Enabled PEBS event to be exported in a raw format

[PATCH] perf, x86: Enabled PEBS event to be exported in a raw format

[Feng Tang]

From: Andi Kleen <ak@linux.intel.com>

Add support for reporting PEBS records in a raw format that can
be then parsed by perf script.

This is roughly similar to the existing AMD IBS support.

This is mainly for advanced users and to experiment with
new usages. Widespread PEBS usages are expected to get higher
level interfaces over time, like the existing "precise ip" support.

[Small fix of adding "data.raw = &raw;", thanks to Andi, with this
patch I can test my perf script with pebs samples. - Feng]

Signed-off-by: Andi Kleen <ak@linux.intel.com>
Signed-off-by: Feng Tang <feng.tang@intel.com>
---
 arch/x86/kernel/cpu/perf_event_intel_ds.c |    7 +++++++
 1 files changed, 7 insertions(+), 0 deletions(-)

diff --git a/arch/x86/kernel/cpu/perf_event_intel_ds.c b/arch/x86/kernel/cpu/perf_event_intel_ds.c
index 026373e..04d3ba9 100644
--- a/arch/x86/kernel/cpu/perf_event_intel_ds.c
+++ b/arch/x86/kernel/cpu/perf_event_intel_ds.c
@@ -552,6 +552,7 @@ static void __intel_pmu_pebs_event(struct perf_event *event,
 	struct pebs_record_core *pebs = __pebs;
 	struct perf_sample_data data;
 	struct pt_regs regs;
+	struct perf_raw_record raw;
 
 	if (!intel_pmu_save_and_restart(event))
 		return;
@@ -578,6 +579,12 @@ static void __intel_pmu_pebs_event(struct perf_event *event,
 	else
 		regs.flags &= ~PERF_EFLAGS_EXACT;
 
+	if (event->attr.sample_type & PERF_SAMPLE_RAW) {
+		raw.size = x86_pmu.pebs_record_size;
+		raw.data = __pebs;
+		data.raw = &raw;
+	}
+
 	if (has_branch_stack(event))
 		data.br_stack = &cpuc->lbr_stack;
 
-- 
1.7.1

Re: [PATCH] perf, x86: Enabled PEBS event to be exported in a raw format

[Peter Zijlstra]

On Fri, 2012-06-29 at 00:00 +0800, Feng Tang wrote:
> From: Andi Kleen <ak@linux.intel.com>
> 
> Add support for reporting PEBS records in a raw format that can
> be then parsed by perf script.
> 
> This is roughly similar to the existing AMD IBS support.
> 
> This is mainly for advanced users and to experiment with
> new usages. Widespread PEBS usages are expected to get higher
> level interfaces over time, like the existing "precise ip" support.
> 
> [Small fix of adding "data.raw = &raw;", thanks to Andi, with this
> patch I can test my perf script with pebs samples. - Feng]
> 
> Signed-off-by: Andi Kleen <ak@linux.intel.com>
> Signed-off-by: Feng Tang <feng.tang@intel.com> 

Not without a useful userspace part. Preferably not a script.

Re: [PATCH] perf, x86: Enabled PEBS event to be exported in a raw format

[Stephane Eranian]

I will cover some useful use cases with my PEBS-LL patchset + perf support.

On Thu, Jun 28, 2012 at 6:12 PM, Peter Zijlstra <peterz@infradead.org> wrote:
> On Fri, 2012-06-29 at 00:00 +0800, Feng Tang wrote:
>> From: Andi Kleen <ak@linux.intel.com>
>>
>> Add support for reporting PEBS records in a raw format that can
>> be then parsed by perf script.
>>
>> This is roughly similar to the existing AMD IBS support.
>>
>> This is mainly for advanced users and to experiment with
>> new usages. Widespread PEBS usages are expected to get higher
>> level interfaces over time, like the existing "precise ip" support.
>>
>> [Small fix of adding "data.raw = &raw;", thanks to Andi, with this
>> patch I can test my perf script with pebs samples. - Feng]
>>
>> Signed-off-by: Andi Kleen <ak@linux.intel.com>
>> Signed-off-by: Feng Tang <feng.tang@intel.com>
>
> Not without a useful userspace part. Preferably not a script.

Re: [PATCH] perf, x86: Enabled PEBS event to be exported in a raw format

[Peter Zijlstra]

On Thu, 2012-06-28 at 18:15 +0200, Stephane Eranian wrote:
> I will cover some useful use cases with my PEBS-LL patchset + perf
> support.
> 
OK, I'll wait for those then.

Re: [PATCH] perf, x86: Enabled PEBS event to be exported in a raw format

[Stephane Eranian]

My patchset does not cover the general PEBS case (not LL). However, I
think we can leverage
the machine state captured by PEBS by adding a simple patch on top of
Jiri's sample regs
patchset. I have experimented with that last week. One interesting
measurement you can make
on SNB once you have that is : value profiling. That means sampling
the values of function
arguments. I think we can fit this into perf report.

My problem with the script approach is that you may end up
re-inventing a lot of the infrastructure
already in perf in terms of histograming and sorting. I think scripts
are good for dumping
the raw data or very simply analysis. Now I recognize that they are
handy to implement
mahcine specific decoding of the samples.

On Thu, Jun 28, 2012 at 6:19 PM, Peter Zijlstra <peterz@infradead.org> wrote:
> On Thu, 2012-06-28 at 18:15 +0200, Stephane Eranian wrote:
>> I will cover some useful use cases with my PEBS-LL patchset + perf
>> support.
>>
> OK, I'll wait for those then.

Re: [PATCH] perf, x86: Enabled PEBS event to be exported in a raw format

[Peter Zijlstra]

On Thu, 2012-06-28 at 18:27 +0200, Stephane Eranian wrote:
> 
> My problem with the script approach is that you may end up
> re-inventing a lot of the infrastructure
> already in perf in terms of histograming and sorting. I think scripts
> are good for dumping
> the raw data or very simply analysis. Now I recognize that they are
> handy to implement
> mahcine specific decoding of the samples. 

Yeah, I'm no fan of all that script nonsense either. If its worth doing
its worth doing properly.

Re: [PATCH] perf, x86: Enabled PEBS event to be exported in a raw format

[Andi Kleen]

> My problem with the script approach is that you may end up
> re-inventing a lot of the infrastructure
> already in perf in terms of histograming and sorting. I think scripts

All built into modern scripting languages. Only in C you have to reinvent
it.

> are good for dumping
> the raw data or very simply analysis. Now I recognize that they are
> handy to implement
> mahcine specific decoding of the samples.

The main use case is really custom analysis, so that people can do
whatever custom data mining they want.

But I expect once you get users engaged with a simple interface
you'll get much more. 

But even the existing script is quite useful and gives you
a lot of information. It's also very easy to extend to everyone
knowing SQLite.

Can you do all these things the simple script does?

-Andi

Re: [PATCH] perf, x86: Enabled PEBS event to be exported in a raw format

[Andi Kleen]

> Not without a useful userspace part. 

http://permalink.gmane.org/gmane.linux.kernel/1319962

> Preferably not a script.

You don't need to prefer it.

-Andi


-- 
ak@linux.intel.com -- Speaking for myself only

Re: [PATCH] perf, x86: Enabled PEBS event to be exported in a raw format

[Peter Zijlstra]

On Thu, 2012-06-28 at 11:20 -0700, Andi Kleen wrote:
> > Not without a useful userspace part. 
> 
> http://permalink.gmane.org/gmane.linux.kernel/1319962
> 
> > Preferably not a script.
> 
> You don't need to prefer it.

+# will be very time consuming (about 10+ minutes for 10000 samples) if the

Muhahaha... I'll wait for something useful.

Re: [PATCH] perf, x86: Enabled PEBS event to be exported in a raw format

[Feng Tang]

On Thu, 28 Jun 2012 22:51:37 +0200
Peter Zijlstra <peterz@infradead.org> wrote:

> On Thu, 2012-06-28 at 11:20 -0700, Andi Kleen wrote:
> > > Not without a useful userspace part. 
> > 
> > http://permalink.gmane.org/gmane.linux.kernel/1319962
> > 
> > > Preferably not a script.
> > 
> > You don't need to prefer it.
> 
> +# will be very time consuming (about 10+ minutes for 10000 samples) if the
> 
> Muhahaha... I'll wait for something useful.

Yeah, that number scared me too when I first saw it, as the time will make
the script totally useless. Then after migrating to the RAM based database
file, the database handling time could be cut down to 1 second.

I guess the sqlite itself realize the shortcoming too, it also provide a
native way to use in-memory database by specifying ":memory":
	con = sqlite3.connect(":memory:")

Thanks,
Feng

Re: [PATCH] perf, x86: Enabled PEBS event to be exported in a raw format

[Andi Kleen]

> Yeah, that number scared me too when I first saw it, as the time will make
> the script totally useless. Then after migrating to the RAM based database
> file, the database handling time could be cut down to 1 second.

The problem is just that sqlite does a fdatasync for every insert.
Just need to figure out how to tell it to stop doing that.

Shouldn't be too hard ...

But yes /dev/shm should work well enough for now.

> 
> I guess the sqlite itself realize the shortcoming too, it also provide a
> native way to use in-memory database by specifying ":memory":
> 	con = sqlite3.connect(":memory:")

I think it's better to have a file on disk, then you can easily query
more properties just from the command line.

I can see all kinds of interesting data mining on the events
to be enabled by this. They have all the register values and various
other information.

For example if you look for string operations you could do
a histogram over the ECX values and so do a histogram of
memcpy/memset sizes. 

[Would be easier if perf could pass in the assembler opcode,
but could be done without)

Or when sampling spinlocks look at the addresses.
Or lots of other things people could imagine.

-Andi
-- 
ak@linux.intel.com -- Speaking for myself only

Re: [PATCH] perf, x86: Enabled PEBS event to be exported in a raw format

[Ingo Molnar]

* Andi Kleen <ak@linux.intel.com> wrote:

> > Not without a useful userspace part. 
> 
> http://permalink.gmane.org/gmane.linux.kernel/1319962
> 
> > Preferably not a script.
> 
> You don't need to prefer it.

Kernel development 101, Chapter 1.:

  " If you want your patches to be applied by the maintainer 
    then you'll have to address the maintainer's review 
    requests. "

Thanks,

	Ingo

Re: [PATCH] perf, x86: Enabled PEBS event to be exported in a raw format

[Michael Ellerman]

On Fri, 2012-06-29 at 00:00 +0800, Feng Tang wrote:
> From: Andi Kleen <ak@linux.intel.com>
> 
> Add support for reporting PEBS records in a raw format that can
> be then parsed by perf script.
> 
> This is roughly similar to the existing AMD IBS support.
> 
> This is mainly for advanced users and to experiment with
> new usages. Widespread PEBS usages are expected to get higher
> level interfaces over time, like the existing "precise ip" support.

Hi folks,

Pardon if I slightly hijack this thread, but we'd like to do something
similar to this on powerpc.

I've been telling people not to use PERF_SAMPLE_RAW because it provides
no way for the consumer to know what data they're getting, and it's not
extensible in any way.

So is shovelling things out using PERF_SAMPLE_RAW generally acceptable
for "advanced usage" scenarios?

cheers

Re: [PATCH] perf, x86: Enabled PEBS event to be exported in a raw format

[Peter Zijlstra]

On Tue, 2012-07-03 at 17:36 +1000, Michael Ellerman wrote:
> On Fri, 2012-06-29 at 00:00 +0800, Feng Tang wrote:
> > From: Andi Kleen <ak@linux.intel.com>
> > 
> > Add support for reporting PEBS records in a raw format that can
> > be then parsed by perf script.
> > 
> > This is roughly similar to the existing AMD IBS support.
> > 
> > This is mainly for advanced users and to experiment with
> > new usages. Widespread PEBS usages are expected to get higher
> > level interfaces over time, like the existing "precise ip" support.
> 
> Hi folks,
> 
> Pardon if I slightly hijack this thread, but we'd like to do something
> similar to this on powerpc.

Do expand. What does exact information does it provide?

> I've been telling people not to use PERF_SAMPLE_RAW because it provides
> no way for the consumer to know what data they're getting, and it's not
> extensible in any way.
> 
> So is shovelling things out using PERF_SAMPLE_RAW generally acceptable
> for "advanced usage" scenarios?

No.

Re: [PATCH] perf, x86: Enabled PEBS event to be exported in a raw format

[Michael Ellerman]

On Tue, 2012-07-03 at 11:15 +0200, Peter Zijlstra wrote:
> On Tue, 2012-07-03 at 17:36 +1000, Michael Ellerman wrote:
> > On Fri, 2012-06-29 at 00:00 +0800, Feng Tang wrote:
> > > From: Andi Kleen <ak@linux.intel.com>
> > > 
> > > Add support for reporting PEBS records in a raw format that can
> > > be then parsed by perf script.
> > > 
> > > This is roughly similar to the existing AMD IBS support.
> > > 
> > > This is mainly for advanced users and to experiment with
> > > new usages. Widespread PEBS usages are expected to get higher
> > > level interfaces over time, like the existing "precise ip" support.
> > 
> > Hi folks,
> > 
> > Pardon if I slightly hijack this thread, but we'd like to do something
> > similar to this on powerpc.
> 
> Do expand. What does exact information does it provide?

Basically we have three registers which provide various information about
the instruction which caused an event to overflow.

The first reports the address of the instruction, the second the data
address associated with the instruction, and the third reports "other
stuff".

We use the first for SAMPLE_IP, and the second for SAMPLE_ADDR, but
currently we have no way of spitting out the third register.

There's a whole bunch of info in there. I think the most interesting are
stuff like whether and where in the pipeline the instruction stalled, or
where a load was serviced from. I'm not sure to be honest, so I've asked
the HW guys what they're most interested in.

So I can imagine us coming up with a generic sample type for "sample
other info at event overflow", but I'm struggling to see how we'd make
the content of the sample non architecture specific, or even processor
version specific.

> > I've been telling people not to use PERF_SAMPLE_RAW because it provides
> > no way for the consumer to know what data they're getting, and it's not
> > extensible in any way.
> > 
> > So is shovelling things out using PERF_SAMPLE_RAW generally acceptable
> > for "advanced usage" scenarios?
> 
> No.

OK good, at least I was right about that :)

cheers

Re: [PATCH] perf, x86: Enabled PEBS event to be exported in a raw format

[Peter Zijlstra]

On Wed, 2012-07-04 at 18:07 +1000, Michael Ellerman wrote:
> Basically we have three registers which provide various information about
> the instruction which caused an event to overflow.
> 
> The first reports the address of the instruction, the second the data
> address associated with the instruction, and the third reports "other
> stuff".
> 
> We use the first for SAMPLE_IP, and the second for SAMPLE_ADDR, but
> currently we have no way of spitting out the third register.
> 
> There's a whole bunch of info in there. I think the most interesting are
> stuff like whether and where in the pipeline the instruction stalled, or
> where a load was serviced from. I'm not sure to be honest, so I've asked
> the HW guys what they're most interested in.
> 
> So I can imagine us coming up with a generic sample type for "sample
> other info at event overflow", but I'm struggling to see how we'd make
> the content of the sample non architecture specific, or even processor
> version specific. 

Right so we're working on a data-source like thing, the last attempt at
adding something like that is in this thread:

 http://marc.info/?l=linux-kernel&m=130987553915170&w=2
 http://marc.info/?l=linux-kernel&m=131000375226044

I think Stephane is currently trying to revive that, although he hasn't
posted yet. Please have a look if that captures the possible PPC states
as well. Stephane could you share your current stuff so Michael can have
a look?


Not quite sure what to do for your pipeline stage thing, I'm not sure
I've seen that elsewhere.


Anyway, yes PERF_SAMPLE_RAW is the 'other info at overflow' which is
specific to the current cpu and phase of moon type field, and it does
get (ab)used in various places, but I still cringe every time I see it
used.

The problem with that thing is that it gives people the data they need
for their special tools and thus removes them from the incentive pool to
help create more generic,useful tools.

For AMD IBS we've given in since that thing is completely weird, IIRC
Robert was going to grow perf-ibs to record and report the data it
captures so we've at least got a useful open-source tool for it.

Re: [PATCH] perf, x86: Enabled PEBS event to be exported in a raw format

[Stephane Eranian]

On Wed, Jul 4, 2012 at 11:13 AM, Peter Zijlstra <peterz@infradead.org> wrote:
> On Wed, 2012-07-04 at 18:07 +1000, Michael Ellerman wrote:
>> Basically we have three registers which provide various information about
>> the instruction which caused an event to overflow.
>>
>> The first reports the address of the instruction, the second the data
>> address associated with the instruction, and the third reports "other
>> stuff".
>>
>> We use the first for SAMPLE_IP, and the second for SAMPLE_ADDR, but
>> currently we have no way of spitting out the third register.
>>
>> There's a whole bunch of info in there. I think the most interesting are
>> stuff like whether and where in the pipeline the instruction stalled, or
>> where a load was serviced from. I'm not sure to be honest, so I've asked
>> the HW guys what they're most interested in.
>>
>> So I can imagine us coming up with a generic sample type for "sample
>> other info at event overflow", but I'm struggling to see how we'd make
>> the content of the sample non architecture specific, or even processor
>> version specific.
>
> Right so we're working on a data-source like thing, the last attempt at
> adding something like that is in this thread:
>
>  http://marc.info/?l=linux-kernel&m=130987553915170&w=2
>  http://marc.info/?l=linux-kernel&m=131000375226044
>
> I think Stephane is currently trying to revive that, although he hasn't
> posted yet. Please have a look if that captures the possible PPC states
> as well. Stephane could you share your current stuff so Michael can have
> a look?
>
I am currently on vacation and with limited internet access.

But yes, the PEBS-LL patchset I have been working on
does abstract PEBS-LL and offers an abstracted data source
field. It uses:
PERF_SAMPLE_IP: for instr address
PERF_SAMPLE_ADDR: for the data address

It introduces:
PERF_SAMPLE_LATENCY: to capture the access latency
PERF_SAMPLE_DSRC: to encode the data source.

That latter value is a structured bitmask which covers:
- type of access (load, store, prefetch, instr)
- mem level: hit or miss in L1, LFB, L2, L3, LOC_RAM, REM_RAM, uncached, IO
- snoop access: hit, miss, none
- TLB access: hit or miss in L1, L2, HW walker, OS fault

Hopefully those are generic enough to cover all possible cases across
architectures. PPC would be a good test case.
Note that multiple bits per category may be set in case the HW cannot
disambiguate like this is the case on with PEBS-LL sometimes.

>
> Not quite sure what to do for your pipeline stage thing, I'm not sure
> I've seen that elsewhere.
>
>
> Anyway, yes PERF_SAMPLE_RAW is the 'other info at overflow' which is
> specific to the current cpu and phase of moon type field, and it does
> get (ab)used in various places, but I still cringe every time I see it
> used.
>
> The problem with that thing is that it gives people the data they need
> for their special tools and thus removes them from the incentive pool to
> help create more generic,useful tools.
>
> For AMD IBS we've given in since that thing is completely weird, IIRC
> Robert was going to grow perf-ibs to record and report the data it
> captures so we've at least got a useful open-source tool for it.

Re: [PATCH] perf, x86: Enabled PEBS event to be exported in a raw format

[Michael Ellerman]

On Thu, 2012-07-05 at 17:57 +0200, Stephane Eranian wrote:
> On Wed, Jul 4, 2012 at 11:13 AM, Peter Zijlstra <peterz@infradead.org> wrote:
> >
> > I think Stephane is currently trying to revive that, although he hasn't
> > posted yet. Please have a look if that captures the possible PPC states
> > as well. Stephane could you share your current stuff so Michael can have
> > a look?

> I am currently on vacation and with limited internet access.

No stress, I am currently busy doing other things :)

> But yes, the PEBS-LL patchset I have been working on
> does abstract PEBS-LL and offers an abstracted data source
> field. It uses:
> PERF_SAMPLE_IP: for instr address
> PERF_SAMPLE_ADDR: for the data address
> 
> It introduces:
> PERF_SAMPLE_LATENCY: to capture the access latency
> PERF_SAMPLE_DSRC: to encode the data source.
> 
> That latter value is a structured bitmask which covers:
> - type of access (load, store, prefetch, instr)
> - mem level: hit or miss in L1, LFB, L2, L3, LOC_RAM, REM_RAM, uncached, IO
> - snoop access: hit, miss, none
> - TLB access: hit or miss in L1, L2, HW walker, OS fault
> 
> Hopefully those are generic enough to cover all possible cases across
> architectures. PPC would be a good test case.
> Note that multiple bits per category may be set in case the HW cannot
> disambiguate like this is the case on with PEBS-LL sometimes.

OK that sounds promising. And I like that it's a bitmask.

>From a preliminary look I think the list you have above will cover most
of the information we can report. And we should have some bits left over
for extensibility.

cheers