RE: [perfmon] Re: quick overview of the perfmon2 interface

RE: [perfmon] Re: quick overview of the perfmon2 interface

[Truong, Dan]

Would you want Stephane to guard the extended
functionalities with tunables or something to
Disable their regular use and herd enterprise
Tools into a standard mold... yet allow R&D to
Move on by enabling the extentions?



Just crippling flexibility/cutting functionality
is like removing words out of a dictionary to
prevent people from thinking different.

It would restrict the R&D mindset, and new ideas.
The field hasn't grown yet to a stable mature form.
It is just beginning: profiling, monitoring, tuning,
compilers, JIT...

Flexibility is/was needed because:
- Tools need to port to Perfmon with min cost.
- Ability to support novel R&D ideas.
- Ability to support growth beyond just PMU data
- Allows early data aggregation
- Allow OS data correlated to PMU

What standardization adds:
- Coordinated access to PMU rssources from all tools
- All tools/formats etc all plug into the same OS framework.
- The interface gets ported across multiple platforms.
- The functionality is rich for all (fast data transfers,
  multiplexing, system vs thead, etc.)

Dan-

> -----Original Message-----
> From: perfmon-bounces@napali.hpl.hp.com [mailto:perfmon-
> bounces@napali.hpl.hp.com] On Behalf Of Andrew Morton
> Sent: Thursday, December 22, 2005 5:47 AM
> To: Truong, Dan
> Cc: Eranian, Stephane; perfmon@napali.hpl.hp.com; linux-
> ia64@vger.kernel.org; linux-kernel@vger.kernel.org; perfctr-
> devel@lists.sourceforge.net
> Subject: Re: [perfmon] Re: quick overview of the perfmon2 interface
> 
> "Truong, Dan" <dan.truong@hp.com> wrote:
> >
> > The PMU is becoming a standard commodity. Once Perfmon is
> > "the" Linux interface, all the tools can align on it and
> > coexist, push their R&D forward, and more importantly become
> > fully productized for businesses usage.
> >
> 
> The apparently-extreme flexibility of the perfmon interfaces would
tend to
> militate against that, actually.  It'd become better productised if it
had
> one interface and stuck to it.
> 
> (I haven't processed Stephane's reply yet - will get there)
> 
> _______________________________________________
> perfmon mailing list
> perfmon@linux.hpl.hp.com
> http://www.hpl.hp.com/hosted/linux/mail-archives/perfmon/

Re: [perfmon] Re: quick overview of the perfmon2 interface

[Andrew Morton]

"Truong, Dan" <dan.truong@hp.com> wrote:
>
> Would you want Stephane to guard the extended
> functionalities with tunables or something to
> Disable their regular use and herd enterprise
> Tools into a standard mold... yet allow R&D to
> Move on by enabling the extentions?

argh.  I'd prefer to avoid one-month gaps in the conversation, so we don't
all forget what we were talking about.

Look, we just need to get these patches on the wire so we can all look at
them, see what they do, understand what decisions were taken and why.

The conciseness and completeness of those patches' covering descriptions
will be key to helping this process along.

RE: [perfmon] Re: quick overview of the perfmon2 interface

[Bryan O'Sullivan]

On Fri, 2006-01-20 at 10:37 -0800, Truong, Dan wrote:
> Would you want Stephane to guard the extended
> functionalities with tunables or something to
> Disable their regular use and herd enterprise
> Tools into a standard mold... yet allow R&D to
> Move on by enabling the extentions?

I'd prefer to see all of the extended stuff left out entirely for now.
The mainline kernel has no PMU support for any popular architecture,
even though external patches have existed in stable form for years.
Filling that gap ought to be the priority; the interface can be extended
when actual users of new features show up and ask for them.

> It would restrict the R&D mindset, and new ideas.
> The field hasn't grown yet to a stable mature form.

The place for flailing around with uncooked ideas is arguably not the
mainline kernel.

> Flexibility is/was needed because:
> - Tools need to port to Perfmon with min cost.
> - Ability to support novel R&D ideas.
> - Ability to support growth beyond just PMU data
> - Allows early data aggregation
> - Allow OS data correlated to PMU

Speculatively adding complicated and unused interfaces to the kernel in
the hope that some wild-eyed visionary might eventually up and use them
helps nobody.

	<b

Re: [Perfctr-devel] RE: [perfmon] Re: quick overview of the perfmon2 interface

[Stephane Eranian]

Bryan,

On Wed, Jan 25, 2006 at 12:33:32PM -0800, Bryan O'Sullivan wrote:
> On Fri, 2006-01-20 at 10:37 -0800, Truong, Dan wrote:
> > Would you want Stephane to guard the extended
> > functionalities with tunables or something to
> > Disable their regular use and herd enterprise
> > Tools into a standard mold... yet allow R&D to
> > Move on by enabling the extentions?
> 
> I'd prefer to see all of the extended stuff left out entirely for now.

I usually don't add things to the interface just because they are cool
ideas but rather because there is a need expressed by some tool
developer or system person. So it would help if you could
name the extended features you referring to. 

The problem with an incremental approach is to maintained backward compatibility
for existing applications. I have had to deal with this on IA-64. For instance
moving from a single syscall to multiple syscall. Similarly, when passing
data structures, you have to provision some reserved fields for potential
extensions. You don't really want to add more system call if you need to
to add a feature.

> The mainline kernel has no PMU support for any popular architecture,
> even though external patches have existed in stable form for years.

You do not count Oprofile. I think this is a fine tool. And perfmon
does allow it to continue working using almost all of its kernel code.
This is leveraging the custom sampling buffer format support in perfmon.
So you can say this is an extended feature that adds complexity.
But OTOH, this is one elegant way of supporting an existing interface
without breaking all the tools.

Take another example, suppose some tool comes along and say: "I would
like to add in each recorded sample the kernel call stack at the point
of the counter overflow". How would you do this without having to hack
kernel code? With the buffer format, you simply insert of module that
does what you want. There are hundreds of things you can include in your
samples. I don't think that we can come up with a very generic sampling
buffer format.

Sometimes, it is not so much what is recorded but how it is recorded.
Some tool may prefer to have samples aggreagated in the kernel, other
would like to use a double-buffer approach to minimize blind spots.
All are valid requests. Our infrastructure allows this without modification
to the core interface nor core kernel code. I believe this is a very strong
value-add.

Without this infrastructure, it would have been pretty difficult to add
support for the P4 Precise Event Based Sampling (PEBS) which by the way,
nobody was able to offer so far. We were able to proide this support
with a few hundred lines of code without hacking the regular sampling
format. Instead we simply created a dedicated PEBS format as a kernel module.


> Filling that gap ought to be the priority; the interface can be extended
> when actual users of new features show up and ask for them.
> 
Again that is fine as long as you can keep backward complexity and a clean
interface.

> > It would restrict the R&D mindset, and new ideas.
> > The field hasn't grown yet to a stable mature form.
> 
I would agree with you, that people have not yet realized the potential
of those performance counters. But this maybe in part a chicken and egg
problem.  People cannot take full advantage because they don't have
a generic interface on any platform.

Designing a generic perfmon interface is hard because:
	- the hardware is extremely diverse
	- there are so many things you can measure
-- 
-Stephane

Re: [Perfctr-devel] RE: [perfmon] Re: quick overview of the perfmon2 interface

[Bryan O'Sullivan]

On Wed, 2006-01-25 at 14:28 -0800, Stephane Eranian wrote:

> So it would help if you could
> name the extended features you referring to. 

I'm dubious about the hands-off buffer format in general.  Does this
mean that userspace needs to modprobe a specific set of modules in order
to do normal sampling?  If so, how do you work around the need for users
to be root in order to use these interfaces?

> And perfmon
> does allow it to continue working using almost all of its kernel code.
> This is leveraging the custom sampling buffer format support in perfmon.
> So you can say this is an extended feature that adds complexity.
> But OTOH, this is one elegant way of supporting an existing interface
> without breaking all the tools.

So are you saying that part of the existing oprofile code can be deleted
if perfmon is merged, and that userspace won't notice?

> We were able to proide this support
> with a few hundred lines of code without hacking the regular sampling
> format. Instead we simply created a dedicated PEBS format as a kernel module.

Does this mean I can't sample the PMCs on a P4 if I don't have the
special PEBS module loaded?  Do I need to be root to do that?

	<b

Re: [Perfctr-devel] RE: [perfmon] Re: quick overview of the perfmon2 interface

[Stephane Eranian]

Bryan,

On Wed, Jan 25, 2006 at 02:46:43PM -0800, Bryan O'Sullivan wrote:
> On Wed, 2006-01-25 at 14:28 -0800, Stephane Eranian wrote:
> 
> > So it would help if you could
> > name the extended features you referring to. 
> 
> I'm dubious about the hands-off buffer format in general.  Does this
> mean that userspace needs to modprobe a specific set of modules in order
> to do normal sampling?  If so, how do you work around the need for users
> to be root in order to use these interfaces?

As I said, there is a builtin default format that is fairly generic. It does
work for HP Caliper, pfmon, q-tools. I suspect it is good enough for VTUNE.

You need to be root to insert the module. But I believe that for many user
environments, this is more practical than having to recompile a custom kernel.
You can imagine the format being shipped with the tool, when the sysadmin
installs the tool it also installs the module.

> 
> > And perfmon
> > does allow it to continue working using almost all of its kernel code.
> > This is leveraging the custom sampling buffer format support in perfmon.
> > So you can say this is an extended feature that adds complexity.
> > But OTOH, this is one elegant way of supporting an existing interface
> > without breaking all the tools.
> 
> So are you saying that part of the existing oprofile code can be deleted
> if perfmon is merged, and that userspace won't notice?
> 
The part of Oprofile that does actual programming of the PMU can be removed.
The part that stays is the one that deals with recording samples, exporting
samples,  and collecting OS events such as exit, mmap, exec. As the user
level, they need to migrated from the Oprofile way of programming counters
to the perfmon way. This has been done many years ago on Itanium and did
not cause any major problems.

> > We were able to proide this support
> > with a few hundred lines of code without hacking the regular sampling
> > format. Instead we simply created a dedicated PEBS format as a kernel module.
> 
> Does this mean I can't sample the PMCs on a P4 if I don't have the
> special PEBS module loaded?  Do I need to be root to do that?

PEBS is a P4 feature that has two advantages:
	- record the exact IP of where a counter overflows (no skid)
	- the CPU directly record the samples into a memory area designated
	  by the kernel. As such, you only get a PMU when that area fills up.

There are some limitations:
	- you cannot sample on any event
	- the format of a sample is fixed, it does not contain extra PMDs, just
	  IP and some general registers. The process id is not recorded
	  so it is not well suited for system-wide monitoring.
	- it appears to broken for HyperThreading setups.

So, it all depends on what you are after. Some people do care about avoiding
the skid of regular sampling and they want they like PEBS just for that. Others
would like to record a set of extra PMDs (PERFCTR) and they are willing to
compromise a bit on the skid of IP, so they can live with the default format.

-- 
-Stephane

Re: [Perfctr-devel] RE: [perfmon] Re: quick overview of the perfmon2 interface

[Bryan O'Sullivan]

On Wed, 2006-01-25 at 23:48 -0800, Stephane Eranian wrote:

> You need to be root to insert the module. But I believe that for many user
> environments, this is more practical than having to recompile a custom kernel.

Clearly.

> You can imagine the format being shipped with the tool, when the sysadmin
> installs the tool it also installs the module.

In that case, you need some kind of per-distro cruft to make sure the
module gets loaded at every boot, or a setuid program that can install
the module, right?.  Neither of these approaches works well in a cluster
environment where you're running your tools from a shared directory.

I'd really like the default mode of operation for users to not require
root privileges to get at normal functionality.  This is something
perfctr makes possible, for example.

	<b

perfmon2 code review: 32-bit ABI on 64-bit OS

[Stephane Eranian]

Heelo,

Here is another topic of the interface I'd like to see discussed on this list
because it was raised in the past and I am not necessarily satisfied with
the current solution.

Tools can program the PMU by passing structures to read/write the PMC/PMD
registers. Some of those structures contain bitmasks. For instance, 
When sampling, one can load a PMD (counter) value and indicate which 
other PMD registers must be included in the samples using the bitmask.

The interface supports the same fixed number of PMD registers on
all platforms. As such the number of bits in the bitmask is fixed
and we have arrange for it to be multiple of 4 and 8. The structure
looks as follows:

typedef struct {
        u16 reg_num;            /* which register */
        u16 reg_set;            /* event set for this register */
        pfm_flags_t reg_flags;  /* input: flags, return: reg error */
        u64 reg_value;          /* initial pmc/pmd value */
        u64 reg_long_reset;     /* value to reload after notification */
        u64 reg_short_reset;    /* reset after counter overflow */
        u64 reg_last_reset_val; /* return: PMD last reset value */
        u64 reg_ovfl_switch_cnt;/* #overflows before switch */
        unsigned long reg_reset_pmds[PFM_PMD_BV]; /* reset on overflow */
        unsigned long reg_smpl_pmds[PFM_PMD_BV];  /* record in sample */
        u64 reg_smpl_eventid;   /* opaque event identifier */
        u64 reg_random_mask;    /* bitmask used to limit random value */
        u32 reg_random_seed;    /* seed for randomization */
        u32 reg_reserved2[7];   /* for future use */
} pfarg_pmd_t;

We use unsigned long for bitmask because we can leverage the
kernel bitmap.h interface which is nice. All the others fields
in the struct have fixed size. If it was not for the bitmask
the structure would be identical in 32-bit or 64-bit mode. 

Why does it matter?

Many 64-bit Linux kernel do support running 32-bit native applications.
That is the case on PPC64, MIPS64K, X86-64, for instance. One could well
write a 32-bit monitoring tool on top of a 64-bit OS. If we want to avoid
the emulation trampoline in the kernel, we need to ensure that the 32-bit
applications use the same structure layout as the 64-bit OS.

In our particular case we rely on the fact that the number of bits is fixed.
We use 320 bits, which is either 10x32 bits or 5x64 bits. Either way,
the sizeof() is the same. As such the struct is identical. Internally,
the kernel will take the bitmask as u64. The only issue would be the
alignment of the bitmasks. The 32-bit version must be aligned on 64-bit
boundary. Structures are aligned on the size of their largest member, 
here 64-bit. The layout is such that the bitmask are always aligned.
We have verified that this does indeed work on MIPS with Phil Mucci.

The alternative approach would be to hardcode bitmask to be u64.
But that would require extra casting in the kernel to get to the
bitmap interface and may cause extra overhead in the 32-bit user programs.
Yet I don't anticipate that programming those bitmask be in the critical
path of monitoring tools.

The second alternative looks cleaner and safer in a sense and I am certainly
willing to make the change but I would like to get everybody else's opinion
as well.

Note that there are similar issues with the remapped sampling buffer.
There, you need to explicitly compile your tool with a special option
to force certain types to be 64-bit (size_t, void *). Unless someone
tell me how to tell the compiler we're compiling 32-bit to execute
on an 64-bit OS ABI.

Thanks.

Re: perfmon2 code review: 32-bit ABI on 64-bit OS

[Bryan O'Sullivan]

On Fri, 2006-02-10 at 07:36 -0800, Stephane Eranian wrote:

> Many 64-bit Linux kernel do support running 32-bit native applications.
> That is the case on PPC64, MIPS64K, X86-64, for instance.

And sparc64 and s390.

>  One could well
> write a 32-bit monitoring tool on top of a 64-bit OS.

On some 64-bit arches (e.g. x86_64), most userspace code is 64-bit,
while on others (e.g. powerpc), most is 32-bit.  Reducing the number of
things that a userspace tool or library writer can trip over seems like
a good thing here, even if it slightly complicates perfmon's internals.

> Note that there are similar issues with the remapped sampling buffer.
> There, you need to explicitly compile your tool with a special option
> to force certain types to be 64-bit (size_t, void *).

It's pretty normal to just use 64-bit quantities in these cases, and
cast appropriately.

	<b

Re: [perfmon] perfmon2 code review: 32-bit ABI on 64-bit OS

[Stephane Eranian]

Hello,

On Sun, Feb 12, 2006 at 12:16:25AM +0600, Philip Mucci wrote:
> > 
> > On some 64-bit arches (e.g. x86_64), most userspace code is 64-bit,
> > while on others (e.g. powerpc), most is 32-bit.  Reducing the number of
> > things that a userspace tool or library writer can trip over seems like
> > a good thing here, even if it slightly complicates perfmon's internals.
> > 
> > > Note that there are similar issues with the remapped sampling buffer.
> > > There, you need to explicitly compile your tool with a special option
> > > to force certain types to be 64-bit (size_t, void *).
> > 
> > It's pretty normal to just use 64-bit quantities in these cases, and
> > cast appropriately.
> 
> I agree with Bryan. Stephane, do you have any quantitative data for how
> much more expensive going to 64 bit quantities would be? Which
> performance critical operations access this structure? AFAIK, any
> performance monitoring system call is already slow by nature...and thus
> an additional dozen cycles isn't going to make a difference. Of course,
> if this structure needs to be read/written by get_pmd, including the
> userspace version (+ mmap offset), then the extra overhead should be
> considered. 
> 
I think I can easily convert the bitmasks to be u64 on all platforms.
I don't think it will negatively impact performance on 32-bit applications.

The sampling buffer is another matter. It is directly remapped. The default
format, exposes size_t and void *. The size_t is not on the critical
path, it is used to specify the buffer size. If we expose as 64-bit,
we need to check on 32-bit system that the value is below 4GB and cast
to size_t.

The most challenging piece is the IP (program pointer) that is in every
sample. Today it is defined as unsigned long because this is fairly
natural for a code address. The 64bit OS captures addresses as 64-bit,
the 32-bit monitoring tool running on top has to consume them as 64-bit
addresses, so u64 would be fine. 

But not on a 32-bit kernel with a 32-bit tool, addresses exported as u64
would certainly work but consume double to buffer space, and that is a
more serious issue in my mind.

What we need is:
	1/ 32-bit OS: IP is 32-bit in the sampling buffer
	2/ 64-bit OS: IP is 64-bit in the sampling buffer

Because of 32-bit ABI tool running on 2/, the IP would have
to be defined as u64. But then it would be overkill on 1/.

The problem is in the user level header file for the sampling buffer.
We would need a data type that is 64-bit for IP if the host OS is 64-bit
(regardless of the ABI used by the tool, i.e., the compiler). And a data
type that is 32-bit on 32-bit OS. The problem is that there is no compiler
flag or header flag somewhere that could guide the compiler. In the case
of MIPS, we have defined a libpfm compile flags that indicates we want
the 64-bit OS definition when compiling for a 32-bit application.

-- 
-Stephane

Re: [Perfctr-devel] Re: [perfmon] perfmon2 code review: 32-bit ABI on 64-bit OS

[David Gibson]

On Sat, Feb 11, 2006 at 02:33:54PM -0800, Stephane Eranian wrote:
> Hello,
> 
> On Sun, Feb 12, 2006 at 12:16:25AM +0600, Philip Mucci wrote:
> > > 
> > > On some 64-bit arches (e.g. x86_64), most userspace code is 64-bit,
> > > while on others (e.g. powerpc), most is 32-bit.  Reducing the number of
> > > things that a userspace tool or library writer can trip over seems like
> > > a good thing here, even if it slightly complicates perfmon's internals.
> > > 
> > > > Note that there are similar issues with the remapped sampling buffer.
> > > > There, you need to explicitly compile your tool with a special option
> > > > to force certain types to be 64-bit (size_t, void *).
> > > 
> > > It's pretty normal to just use 64-bit quantities in these cases, and
> > > cast appropriately.
> > 
> > I agree with Bryan. Stephane, do you have any quantitative data for how
> > much more expensive going to 64 bit quantities would be? Which
> > performance critical operations access this structure? AFAIK, any
> > performance monitoring system call is already slow by nature...and thus
> > an additional dozen cycles isn't going to make a difference. Of course,
> > if this structure needs to be read/written by get_pmd, including the
> > userspace version (+ mmap offset), then the extra overhead should be
> > considered. 
> > 
> I think I can easily convert the bitmasks to be u64 on all platforms.
> I don't think it will negatively impact performance on 32-bit applications.
> 
> The sampling buffer is another matter. It is directly remapped. The default
> format, exposes size_t and void *. The size_t is not on the critical
> path, it is used to specify the buffer size. If we expose as 64-bit,
> we need to check on 32-bit system that the value is below 4GB and cast
> to size_t.
> 
> The most challenging piece is the IP (program pointer) that is in every
> sample. Today it is defined as unsigned long because this is fairly
> natural for a code address. The 64bit OS captures addresses as 64-bit,
> the 32-bit monitoring tool running on top has to consume them as 64-bit
> addresses, so u64 would be fine. 
> 
> But not on a 32-bit kernel with a 32-bit tool, addresses exported as u64
> would certainly work but consume double to buffer space, and that is a
> more serious issue in my mind.

Hmm.. does the sampling buffer collect on userspace PC values, or
kernel ones as well?

-- 
David Gibson			| I'll have my music baroque, and my code
david AT gibson.dropbear.id.au	| minimalist, thank you.  NOT _the_ _other_
				| _way_ _around_!
http://www.ozlabs.org/~dgibson

Re: [Perfctr-devel] Re: [perfmon] perfmon2 code review: 32-bit ABI on 64-bit OS

[Eric Gouriou]

David Gibson wrote:
> On Sat, Feb 11, 2006 at 02:33:54PM -0800, Stephane Eranian wrote:
[...]
>> The most challenging piece is the IP (program pointer) that is in every
>> sample. Today it is defined as unsigned long because this is fairly
>> natural for a code address. The 64bit OS captures addresses as 64-bit,
>> the 32-bit monitoring tool running on top has to consume them as 64-bit
>> addresses, so u64 would be fine. 
>>
>> But not on a 32-bit kernel with a 32-bit tool, addresses exported as u64
>> would certainly work but consume double to buffer space, and that is a
>> more serious issue in my mind.
> 
> Hmm.. does the sampling buffer collect on userspace PC values, or
> kernel ones as well?

  Either, or both, depending on the measurement settings.

  I live in a 64-bit world, so my take on this issue would be to expose
the PC as a uint64_t, always. There is already so much overhead in the
default per-sample header that I wouldn't worry about it.

  Now 64 bit might not always be enough. E.g., on PA-RISC. But _I_ do
not care much about Linux on PA.

   Eric

-- 
Eric Gouriou                                         eric.gouriou@hp.com

Re: [perfmon] perfmon2 code review: 32-bit ABI on 64-bit OS

[Stephane Eranian]

David,

On Sun, Feb 12, 2006 at 04:03:44PM -0800, Eric Gouriou wrote:
> David Gibson wrote:
> >On Sat, Feb 11, 2006 at 02:33:54PM -0800, Stephane Eranian wrote:
> [...]
> >>The most challenging piece is the IP (program pointer) that is in every
> >>sample. Today it is defined as unsigned long because this is fairly
> >>natural for a code address. The 64bit OS captures addresses as 64-bit,
> >>the 32-bit monitoring tool running on top has to consume them as 64-bit
> >>addresses, so u64 would be fine. 
> >>
> >>But not on a 32-bit kernel with a 32-bit tool, addresses exported as u64
> >>would certainly work but consume double to buffer space, and that is a
> >>more serious issue in my mind.
> >
> >Hmm.. does the sampling buffer collect on userspace PC values, or
> >kernel ones as well?
> 
>  Either, or both, depending on the measurement settings.
> 
>  I live in a 64-bit world, so my take on this issue would be to expose
> the PC as a uint64_t, always. There is already so much overhead in the
> default per-sample header that I wouldn't worry about it.
> 
Eric is right, on many architectures, incl. PPC64 I am sure, you can easily
configure a counter to measure at any priv levels including at the kernel level.
As such a 32-bt monitoring tool could see 64-bit generated samples. Similarly,
I don't think it would be unreasonable to have a 32-bit tool monitor 64-bit
applications.

The question is whether hardcoding the IP to always be u64 is a valid choice.
Eric's comment about overhead is based on the current default sampling format
which systematically adds a fixed size header to each sample. That header
contains the IP. So adding 4 bytes to this header is not a big deal.

However, because we can define virtual PMDs that map to software resources, it
is likely that the default format will evolve to allow an application to specify
everything it needs for each sample. For instance, you can have a PMD
that maps to the current PID, another one that maps to the interrupt IP. Then
you can chose to include those into the sample and you would nto need a fixed size
header anymore.

-- 
-Stephane

Re: [Perfctr-devel] Re: [perfmon] perfmon2 code review: 32-bit ABI on 64-bit OS

[Stephane Eranian]

Phil,

On Tue, Feb 14, 2006 at 12:57:55AM +0600, Philip Mucci wrote:
> Stefane,
> 
> I know this is ugly, but what about in the user code checking the arch?
> Is it not true that if a kernel is running 64 bit, it has the 64 tagged
> on the the end of the arch? i.e. mips64, ppc64, x86_64?
> 
> The other solution would be to add am information call to the API, like
> perfctr has. This would export the processor type along with other
> feature bits, including the number of bits of the IP. 
> 

The problem is at compile time and not so much at runtime.
Take a kernel struct that is shared with user (i.e., passed through syscall)
that has the following layout:

	struct foo {
		unsigned long bar;
		int dummy;
	};

For a 64-bit app on a 64-bit OS OR a 32-bit app on a 32-bit OS, this works
perfectly.

For a 32-bit on a 64-bit OS, there is a problem, the 32-bit app must be compiled
with the following definition instead:
	struct foo {
		unsigned long long bar;
		int dummy;
	};

to share the struct with the 64-bit OS. An application compile with the above
struct, would not work when run on a 32-bit OS.

So I think that we need to replace all unsigned long, size_t, void * by uint64_t
to make sure this works either way. It is overkill on pure 32-bit but ensures
that the application can be migrated over to a 64-bit OS without the need for
special recompilation.

-- 
-Stephane

Re: perfmon2 code review: 32-bit ABI on 64-bit OS

[Stephane Eranian]

Hello,

So I worked some more on these 32bit vs 64bit ABI issues. I came
to the conclusion that the only way to make this work cleanly
without any special compilation flags or runtime support is
to simply ensure that all data structures exchanged/shared between
the monitoring tool  and the kernel use only fixed size types.
This way, the sizeof struct  and fields offset remains constant
in ILP32 or LP64 mode. As a consequence I now have:
	- all bitmasks use u64
	- all size_t are replaced with u64
	- the default format IP (instruction pointer) is u64

For the IP, the most significant bits contain all zeroes in
ILP32 mode. With the default sampling buffer format, you add
4 bytes to the sample header but then you have nothing special
to do with user level code. I think this is also inline with
the fact that 64-bit computing is becoming widely available
for desktop and laptop computers.

As an example, I can now compile a tool for P4 in 32-bit and
run it on my 32-bit Xeon. But I can also run the same *binary*
on my EM64T and it works as expected including for the sampling
buffer. I believe the same would be possible with Opteron, although
I have not tried.

I hope this closes another sticking point about the interface.


On Tue, Feb 14, 2006 at 12:57:55AM +0600, Philip Mucci wrote:
> 
> I know this is ugly, but what about in the user code checking the arch?
> Is it not true that if a kernel is running 64 bit, it has the 64 tagged
> on the the end of the arch? i.e. mips64, ppc64, x86_64?
> 
> The other solution would be to add am information call to the API, like
> perfctr has. This would export the processor type along with other
> feature bits, including the number of bits of the IP. 
> 
> Either of these combined with compile time ABI/bitness constants should
> cover the cases no?
> 
> Phil
> 
> 
> > The problem is in the user level header file for the sampling buffer.
> > We would need a data type that is 64-bit for IP if the host OS is 64-bit
> > (regardless of the ABI used by the tool, i.e., the compiler). And a data
> > type that is 32-bit on 32-bit OS. The problem is that there is no compiler
> > flag or header flag somewhere that could guide the compiler. In the case
> > of MIPS, we have defined a libpfm compile flags that indicates we want
> > the 64-bit OS definition when compiling for a 32-bit application.
> > 
> 
> 
> 
> -------------------------------------------------------
> This SF.net email is sponsored by: Splunk Inc. Do you grep through log files
> for problems?  Stop!  Download the new AJAX search engine that makes
> searching your log files as easy as surfing the  web.  DOWNLOAD SPLUNK!
> http://sel.as-us.falkag.net/sel?cmd=lnk&kid=103432&bid=230486&dat=121642
> _______________________________________________
> Perfctr-devel mailing list
> Perfctr-devel@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/perfctr-devel

-- 

-Stephane

Re: perfmon2 code review: 32-bit ABI on 64-bit OS

[Stephane Eranian]

Bryan,

On Fri, Feb 10, 2006 at 10:27:02AM -0800, Bryan O'Sullivan wrote:
> 
> On some 64-bit arches (e.g. x86_64), most userspace code is 64-bit,
> while on others (e.g. powerpc), most is 32-bit.  Reducing the number of
> things that a userspace tool or library writer can trip over seems like
> a good thing here, even if it slightly complicates perfmon's internals.
> 
> > Note that there are similar issues with the remapped sampling buffer.
> > There, you need to explicitly compile your tool with a special option
> > to force certain types to be 64-bit (size_t, void *).
> 
> It's pretty normal to just use 64-bit quantities in these cases, and
> cast appropriately.
> 

So if I understand you correctly, you are saying it is best to have bitmasks
hardcoded to u64 and have the kernel cast to match the bitmap_*() interface.
This would not cause any alignment problems on neither 32-bit nor 64-bit system.


-- 

-Stephane

RE: [perfmon] Re: quick overview of the perfmon2 interface

[Truong, Dan]

> Thanks to David, Dan and Phil for their comments.

Another note on the urgency of standardizing Perfmon:

Anarchy is not a good breeding ground for tools that need a
stable infrastructure to mature. Being "there" is what made
PAPI and perfctr popular and somewhat standard infrastructure.

Compilers, tools, JVMs... -you name it- are all moving
fast towards using hardware counters to get feedback,
tune, monitor or measure application behavior.

The PMU is becoming a standard commodity. Once Perfmon is
"the" Linux interface, all the tools can align on it and
coexist, push their R&D forward, and more importantly become
fully productized for businesses usage. Hopefully Perfmon's
interface is powerful enough to support future needs.

Good luck Stephane :)

Cheers,

Dan-

Re: [perfmon] Re: quick overview of the perfmon2 interface

[Andrew Morton]

"Truong, Dan" <dan.truong@hp.com> wrote:
>
> The PMU is becoming a standard commodity. Once Perfmon is
> "the" Linux interface, all the tools can align on it and
> coexist, push their R&D forward, and more importantly become
> fully productized for businesses usage.
>

The apparently-extreme flexibility of the perfmon interfaces would tend to
militate against that, actually.  It'd become better productised if it had
one interface and stuck to it.

(I haven't processed Stephane's reply yet - will get there)

RE: [perfmon] Re: quick overview of the perfmon2 interface

[Truong, Dan]

Just a couple of cents here to support Stephane's design :)


> Why would one want to change the format of the sampling buffer?

The idea is to allow user custom formats for one, and allow Exotic
architectures for second.

Say you want to reduce the volume of data passed to the application And
stored in the buffers, you can then do some pre-processing Inside the
kernel via a custom module.

You can also return more complex data than just PMU registers, think
Call stacks or other OS related information that can complement the The
PMU data.

Some data can be returned vis pseudo PMU registers (i.e. extentions),
Like the interval timer, PID/TID, etc. but for more complex and less
Synchronous data you may end up needed a more powerfull buffer format
With headers etc.

Another issue can be if hardware buffer support is provided. The
hardware Buffer support will not allow collection of pseudo-counters
which are Supported by software, so again the packaging may not be as
linear as A repeated sequence of counters...

With Stephane we had discussed this buffer format, and came to the
Conclusion that flexibility there will avoid hitting the wall.

You don't know what tomorrow is made of (yet)...



> > 	The PMU register description is implemented by a kernel
pluggable
> 
> Is that option important, or likely to be useful?  Are you sure there 
> isn't some overdesign here?

It will allow bringup of new PMUs on new architectures more easily, and
simpler distribution of support. It will also allow CPU designers to
create custom drivers that support non-public features to debug the
CPUs.



> hm.  I'm surprised at such a CPU-centric approach.  I'd have expected 
> to see a more task-centric model.

Both per-thread and system-wide measurments are useful.

Systemwide is used to evaluate the beavior of the whole system when
tuning a large load (think TPC-C, SpecWeb, SpecJapp...) Per thread is
used for specific application/thread tuning and self monitoring. Also
per-thread monitoring is not always adviseable, for example when there
Are a large number of threads loading the system, adding that many
monitors will impact the system performance, so you will want to measure
per CPU.

> So the kernel buffers these messages for the read()er.  How does it 
> handle the case of a process which requests the messages but never 
> gets around to read()ing them?

Stephane, I would assume that the monitoring session attached to
The buffer returning the message just stalls. If there is multiplexing,
Will coming back to that stalled buffer stall all the multiplexed
Sessions? I would assume so.



> Why would one want to randomise the PMD after an overflow?

Everybody does it :) Helps generate an un-biased picture.



> I think the usefulness of this needs justification.  CPUs are updated 
> all the time, and we release new kernels all the time to exploit the 
> new CPU features.  What's so special about performance counters that 
> they need such special treatment?

The PMU is not fully architected usually. Nothing prevents a
Model to be shipped with PMU upgrades.
Also the PMU can be used by architects for validation of the designs.
Easier early access to the functionalities helps.

The PMU is a direct evolution of the debug counters that were used
To debug CPUs but not available for general use. They are still used
In that fashion too, and a main reason they exist.


Cheers,

Dan-

quick overview of the perfmon2 interface

[Stephane Eranian]

Hello,

As suggested by Andrew, I wrote a quick overview of the perfmon2 interface
that we have implemented on several architectures now. The goal of this
introduction is to give you an idea of the key features of the interface.


------------------------------------------------------------------------------

	      ===========================================================

		  ----------------------------------------------------
		  A quick overview of the perfmon2 interface for Linux
		  ----------------------------------------------------

	       Copyright (c) 2005 Hewlett-Packard Development Company, L.P.
		   Contributed by Stephane Eranian <eranian@hpl.hp.com>

	      ===========================================================
	
I/ INTRODUCTION
   ------------

	The goal of the perfmon2 interface is to provide access to the hardware
	performance counters present in all modern processors.

	The interface is designed to be builtin, very generic, flexible and
	extensible. It is not designed to support a single application or a
	small class of monitoring tools. The goal is to avoid fragmentation
	where you have one tool using one interface. Because we want the
	interface to be an integral part of the kernel, special care is taken
	to make it robust and secure. The interface is uniform across all
	hardware platforms, i.e., it offers the same level of software
	functionalities on each platform. The nature of the captured data
	depends solely on the capabilities of the underlying hardware.

	Although, by nature the Performance Monitoring Unit (PMU) of each
	processor architecture can be quite different, it is possible to
	extrapolate a common hardware interface on which we can build a
	powerful kernel interface. All modern PMUs are implemented using a
	register interface. Two types of registers are typically present:
	configuration registers (PMC) and data registers (PMD). As such, the
	interface is simply exporting read/write operations on those registers.
	A minimal set of software abstractions is added, such as the notion of
	a perfmon context which is used to encapsulate the PMU state.

	The same interface provides support for per-thread AND system-wide
	measurements. For each mode, it is possible to collect simple counts
	or create full sampling measurements.

	Sampling is supported at the user level and also at the kernel level with
	the ability to use a kernel-level sampling buffer. The format of the kernel
	sampling buffer is implemented by a kernel pluggable module. As such it is
	very easy to develop a custom format without any modification to the
	interface nor its implementation.

	To compensate for limitations of many PMU, such as a small number of
	counters, the interface also exports the notion of event sets and allows
	applications to multiplex sets, thereby allowing more events to be
	measured in a single run than there are actual counters.

	The PMU register description is implemented by a kernel pluggable module
	thereby allowing new hardware to be supported without the need to wait
	for the next release of the kernel. The description table supports virtual
	PMD registers which can be tied to any kernel or hardware resource.

II/ BASE INTERFACE
    ---------------

	PMU registers are accessed by reading/writing PMC and PMD registers.
	The interface exposes a logical view of the PMU. The logical PMD/PMC
	registers are mapped onto the actual PMU registers (be them PMD/PMC or
	actual MSR) by the kernel. The mapping table is implemented by a kernel
	module and can thus easily be updated without a kernel recompile. This
	mechanism also makes it easy to add new PMU support inside a processor
	family. The mapping is exposed to users via a file in /proc
	(/proc/perfmon_map).


	The interface is implemented using a system call interface rather than
	a device driver. There are several reasons for this choice, the most
	important being that we do want to support per-thread monitoring and
	that requires access to the context switch code of the kernel to
	save/restore the PMU state. Another reason is to reinforce the fact
	that the interface must be an integral part of the kernel. Lastly, we
	think it give us more flexibility in terms of how arguments can be
	passed to/from the kernel.

	Whenever possible, the interface leverages existing kernel mechanisms.
	As such, we use a file descriptor to identify a perfmon context.

	The interface defines the following set of system calls:

	- int pfm_create_context(pfarg_ctx_t *ctx, void *smpl_arg, size_t smpl_size)

	    creates  a per-thread or system-wide perfmon context. It returns a
	    file descriptor that uniquely identifies the context. The regular
	    file descriptor semantics w.r.t. to access control, sharing are
	    supported.

	- pfm_write_pmds(int fd, pfarg_pmd_t *pmds, int n)

	    Write one or more PMD registers.

	- pfm_read_pmds(int fd, pfarg_pmd_t *pmds, int n)

	    Read the one or more PMD registers.

	- pfm_write_pmcs(int fd, pfarg_pmc_t*pmcs, int n)

	    Write the one or more PMC registers.

	- pfm_load_context(int fd, pfarg_load_t *load)

	   Attach a perfmon context to either a thread or a processor. In the
	   case of a thread, the thread id is passed. In the case of a
	   processor, the context is bound to the CPU on which the call is
	   performed.
	  
	- pfm_start(int fd, pfarg_start_t *start)

	   Start active monitoring.

	- pfm_stop(int fd)

	   Stop active monitoring.

	- pfm_restart(int fd)

	   Resume monitoring after a user level overflow notification. This
	   call is used in conjunction with kernel-level sampling.

	- pfm_create_evtsets(int fd, pfarg_setdesc_t *sets, int n)

	    Create/Modify one or more PMU event set. Each set encapsulates the
	    full PMU sets.

	- pfm_delete_evtsets(int fd, pfarg_setdesc_t *sets, int n)

	    Delete a PMU event set. It is possible to delete one or more sets
	    in a single call.

	- pfm_getinfo_evtsets(int fd, pfarg_setinfo_t *infos, int n):

	    Return information about an event set. It is possible to get
	    information about one or more sets in a single call. The call
	    returns, for instance, the number of times a set has been activated,
	    i.e., loaded onto the actual PMU.

	- pfm_unload_context(int fd)

	    Detach a context from a thread or a processor.

	By default, all counters are exported as 64-bit registers even when the
	underlying hardware implements less. This makes it much easier for
	applications that are doing event-based sampling because they don't
	need to worry about the width of counters. It is possible to turn the
	"virtualization" off.

	A system-wide context allows a tool to monitor all activities on one
	processor, i.e, across all threads. Full System-wide monitoring in an
	SMP system is achieved by creating and binding a perfmon context on
	each processor. By construction, a perfmon context can only be bound
	to one processor at a time.  This design choice is motivated by the
	desire to enforce locality and to simplify the kernel implementation.
	
	Multiple per-thread contexts can coexist at the same time on a system.
	Multiple system-wide can co-exist as long as they do not monitor the
	same set of processors. The existing implementation does not allow
	per-thread and system-wide context to exist at the same time. The
	restriction is not inherent to the interface but come from the
	existing implementation.
	
3/ SAMPLING SUPPORT
   ----------------

	The interface supports event-based sampling (EBS), where the sampling
	period is determined by the number of occurrences of an event rather
	than by time. Note that time-based sampling (TBS) can be emulated by
	using an event with some correlation to time.

	The EBS is based on the ability for PMU to generate an interrupt when
	a counter overflows. All modern PMU support this mode.

	Because counters are virtualized to 64 bits. A sampling period p,
	is setup by writing a PMD to 2^{64}-p -1 = -p.

	The interface does have the notion of a sampling period, it only
	manipulates PMD values. When a counter overflows, it is possible
	for a tool to request a notification. By construction, the interface
	supports as many sampling periods as there are counters on the host
	PMU making it possible to overlap distinct sampling measurements in
	one run. The notification can be requested per counter.

	The notification is sent as a message and can be extracted by invoking
	read() on the file descriptor of the context. The overflow message
	contains information about the overflow, such as the index of the
	overflowed PMD.

	The interface supports using select/poll on contexts file descriptors.
	Similarly, it is possible to get an asynchronous notification via SIGIO
	using the regular sequence of fcntl().

	By default, during a notification, monitoring is masked, i.e., nothing
	is captured. A tool uses the pfm_restart() call to resume monitoring.

	It is possible to request that on overflow notification, the monitoring
	thread be blocked. By default, it keeps on running with monitoring
	masked. Blocking is not supported in system-wide mode nor when a thread
	is self-monitoring.

4/ SAMPLING BUFFER SUPPORT
   -----------------------

	User level sampling works but it is quite expensive especially when for
	non self-monitoring threads. To minimize the overhead, the interface also
	supports a kernel level sampling buffer. The idea is simple: on overflow
	the kernel record a sample, and only when the buffer becomes full is the
	user level notification generated. Thus, we amortize the cost of the
	notification by simply calling the user when lots of samples are available.

	This is not such a new idea, it is present in OProfile or perfctr.
	However, the interface needs to remains generic and flexible. If
	the sampling buffer is in kernel, its format and what gets recorded
	becomes somehow locked by the implementation. Every tool has different
	needs. For instance, a tool such as Oprofile may want to aggregate
	samples in the kernel, others such as VTUNE want to record all samples
	sequentially. Furthermore, some tools may want to combine in each sample
	PMU information with other kernel level information, such as a kernel
	call stack for instance. It is hard to design a generic buffer format
	that can handle all possible request. Instead, the interface provides
	an infrastructure  in which the buffer format is implemented by a kernel
	module. Each module controls, what gets recorded, how it is recorded,
	how the information is exported to user, when a 'buffer full'
	notification must be sent. The perfmon core has an interface to
	dynamically register new formats. Each format is uniquely identified by
	a 128-bit UUID which is passed by the tool when the context is created.
	Arguments for the buffer format are also passed during this call.

	As part of the infrastructure, the interface provides a buffer allocation
	and remapping service to the buffer format modules. Format may use this
	service when the context is created. The kernel memory will be reserved
	and the tool will be able to get access to the buffer via remapping
	using the mmap() system call. This provides an efficient way of
	exporting samples without the need for copying large amount of data
	between the kernel and user space. But a format may choose to export
	its samples via another way, such as a device driver interface for
	instance.

	The current implementation comes with a simple default format builtin.
	This format records samples in a sequential order. Each sample has a
	fixed sized header which include the interrupted instruction pointer,
	which PMD overflowed, the PID and TID of the thread among other things.
	The header is followed by an optional variable size body where
	additional PMD values can be recorded.
	
	We have successfully hooked the Oprofile kernel infrastructure to
	our interface using a simple buffer format module on Linux/ia64.
	We have released a buffer format that implements n-way buffering to
	show how blind spots may be minimized. both modules required absolutely
	no change to the interface nor perfmon core implementation. We have also
	developed a buffer format module to support P4/Xeon Precise Event-Based
	Sampling (PEBS).

	Because sampling can happen in the kernel without user intervention,
	the kernel must have all the information to figure out what to record,
	how to restart the sampling period. This information is passed when
	the PMD used as the sampling period is programmed. For each such PMD,
	it is possible to indicate using bitvector, which PMDs to record on
	overflow, which PMDs to reset on overflow.

	For each PMD, the interface provides three possible values which are
	used when sampling. The initial value is that is first loaded into the
	PMD, i.e., the first sampling period. On overflow which does not
	trigger a user level notification, a so-called short reset value is
	used by the kernel to reload the PMD. After an overflow with a user
	level notification, the kernel uses the so-called long reset value.
	This mechanism can be exploited to hide the noise induced by the
	recovery after a user notification.

	The interface also supports automatic randomization of the reset value
	for a PMD after an overflow. Randomization is indicated per PMD and is
	controlled by a seed value. The range of variation is specified by a
	bitmask per PMD.

5/ EVENT SETS AND MULTIPLEXING:
   ----------------------------

	For many PMU models, the number of counters is fairly limited which
	makes it sometimes difficult to collect certain metric in a single run.
	But this is not always because you have a large number of counters that
	they all can measure any events at the same time. Such constraints can
	be alleviated by creating the notion of an event set. Each set
	encapsulates the full PMU state. At any one time only one set is loaded
	onto the actual PMU. Sets are then multiplexed. The counts collected
	by counters in each set can then be scaled to approximate what they
	would have been, had they run for the entire duration of the
	measurement. It is very important to keep in mind that this is an
	approximation. Its quality depends on the frequency at which sets can
	be switched and also the overhead involved.

	Event sets and multiplexing can be fully implemented at the user level
	but this is prohibitively expensive especially for non-self-monitoring
	threads.

	The interface exports the notion of event set. Each PMD/PMC can be
	assigned to a set when read/written.

	By default any perfmon context has a single set, namely set. Tools can
	create additional sets using pfm_create_evtsets(). A set is identified
	by a number between 0-65535. The number indicate the order in which
	sets are switched to and from. The kernel uses an ordered list managed
	in a round-robin fashion to determine the switch order.

	Switching can either be triggered by a timeout or by a counter overflow.
	The switch mode is set per set and it is possible to mix and match.

	The timeout is specified when the set is created (or modified for set0).
	It is limited by the granularity of the timer tick. The user timeout is
	rounded up to the nearest multiple of the timer tick frequency. The
	actual timeout is returned to the tool.

	For overflow switching, the interface does not require a dedicated
	counter. Each PMD has an overflow switch counter. On overflow the
	switch counter is decremented. When it reaches zero, switching occurs.
	There can be more than on "trigger" PMD per set.

	Overflow-based set switching can be used to implement counter
	cascading, where certain counters start measuring only when
	a certain threshold is reached on another event.

6/ PMU DESCRIPTION MODULES
   -----------------------

	The logical PMU is driven by a PMU description table. The table
	is implemented by a kernel pluggable module. As such, it can be
	updated at will without recompiling the kernel, waiting for the next
	release of a Linux kernel or distribution, and without rebooting the
	machine as long as the PMU model belongs to the same PMU family. For
	instance, for the Itanium Processor Family, the architecture specifies
	the framework for the PMU. Thus the Itanium PMU specific code is common
	across all processor implementations. This is not the case for IA-32.

	The interface and PMU description module support the notion of virtual
	PMU registers, i.e., register not necessarily tied to an actual PMU
	register. For instance, it may be interesting, especially when sampling,
	to export a kernel resource or a non-PMU hardware registers as a PMD.

	The actual read/write function for those virtual PMD is implemented by
	the PMU description module, allowing for maximum flexibility.

	It is important to understand that the interface, including the PMU
	description modules, do not know anything about PMU events. All event
	specific information, including event names and encodings has to be
	implemented at the user level.

7/ IMPLEMENTATION
   ---------------

	We have developed an implementation for the 2.6.x kernel series.
	We do have support for the following processors:

		- All Itanium processors (Itanium, McKinley/Madison, Montecito)
		- Intel EM64T/Xeon. Includes support for PEBS and HyperThreading (produced by Intel)
		- Intel P4/Xeon (32-bit). Includes support for PEBS and HyperThreading
		- Intel Pentium M and P6 processors
		- AMD 64-bit Opteron
		- preliminary support for IBM Power 5 (produced by IBM)
		- preliminary support for MIPS R5000 (produced by Phil Mucci)

	The so-called "new perfmon code base" incorporates all the features we describe here.
	At this point, this is a standalone patch for the 2.6.x kernels. The full patch can be
	downloaded from our project web site at:

		http://www.sf.net/projects/perfmon2

	On Linux/ia64, there is a older version (v2.0) of this interface that is currently
	provided on all 2.6.x based kernels. The "new code base" (v2.2) perfmon maintains
	backward compatibility with this version.

8/ EXISTING TOOLS
   --------------

	Several commercial products as well as open-source tools already exists for this
	interface (or previous incarnation) for Linux on Itanium where such interface has
	been available for quite some time:

		- HP Caliper for RHEL4, SLES9.
		- BEA JRockit with Dynamic Optimization
		- pfmon/libpfm by HPLabs
		- qtools/qprof by HPLabs
		- PerfSuite from NCSA
		- all PAPI-based tools
		- OProfile

	We think that once the interface is part of the mainline kernel, we will see even more
	tools being released and developed for the benefits of ALL users across ALL major
	hardware platforms.
-- 

-Stephane

Re: quick overview of the perfmon2 interface

[Andrew Morton]

Stephane Eranian <eranian@hpl.hp.com> wrote:
> 	or create full sampling measurements.
> ...
> 
> 	Sampling is supported at the user level and also at the kernel level with
> 	the ability to use a kernel-level sampling buffer. The format of the kernel
> 	sampling buffer is implemented by a kernel pluggable module. As such it is
> 	very easy to develop a custom format without any modification to the
> 	interface nor its implementation.

Why would one want to change the format of the sampling buffer?

Would much simplification be realised if we were to remove this option?

> 	To compensate for limitations of many PMU, such as a small number of
> 	counters, the interface also exports the notion of event sets and allows
> 	applications to multiplex sets, thereby allowing more events to be
> 	measured in a single run than there are actual counters.
> 
> 	The PMU register description is implemented by a kernel pluggable module
> 	thereby allowing new hardware to be supported without the need to wait
> 	for the next release of the kernel.

Is that option important, or likely to be useful?  Are you sure there isn't
some overdesign here?


> II/ BASE INTERFACE
>     ---------------
> 
> 	PMU registers are accessed by reading/writing PMC and PMD registers.
> 	The interface exposes a logical view of the PMU. The logical PMD/PMC
> 	registers are mapped onto the actual PMU registers (be them PMD/PMC or
> 	actual MSR) by the kernel. The mapping table is implemented by a kernel
> 	module and can thus easily be updated without a kernel recompile. This
> 	mechanism also makes it easy to add new PMU support inside a processor
> 	family.

Ditto.

> 	The interface is implemented using a system call interface rather than
> 	a device driver. There are several reasons for this choice, the most
> 	important being that we do want to support per-thread monitoring and
> 	that requires access to the context switch code of the kernel to
> 	save/restore the PMU state. Another reason is to reinforce the fact
> 	that the interface must be an integral part of the kernel. Lastly, we
> 	think it give us more flexibility in terms of how arguments can be
> 	passed to/from the kernel.
> 
> 	Whenever possible, the interface leverages existing kernel mechanisms.
> 	As such, we use a file descriptor to identify a perfmon context.
> 
> 	The interface defines the following set of system calls:
> 
> 	- int pfm_create_context(pfarg_ctx_t *ctx, void *smpl_arg, size_t smpl_size)

pfarg_ctx_t __user *ctx, I assume?

void __user *smpl_arg, I assume?

What is at *smpl_arg?  Anonymous pointers to userspace like this aren't
very popular - strongly typed interfaces are preferred.

> 
> 	    creates  a per-thread or system-wide perfmon context. It returns a
> 	    file descriptor that uniquely identifies the context. The regular
> 	    file descriptor semantics w.r.t. to access control, sharing are
> 	    supported.
> 
> 	- pfm_write_pmds(int fd, pfarg_pmd_t *pmds, int n)
> 
> 	    Write one or more PMD registers.
> 
> 	- pfm_read_pmds(int fd, pfarg_pmd_t *pmds, int n)
> 
> 	    Read the one or more PMD registers.
> 
> 	- pfm_write_pmcs(int fd, pfarg_pmc_t*pmcs, int n)
> 
> 	    Write the one or more PMC registers.
> 
> 	- pfm_load_context(int fd, pfarg_load_t *load)
> 
> 	   Attach a perfmon context to either a thread or a processor. In the
> 	   case of a thread, the thread id is passed. In the case of a
> 	   processor, the context is bound to the CPU on which the call is
> 	   performed.

Why should userspace concern itself with a particular CPU?  That's really
only valid if the process has bound itself to a single CPU?  If the CPU is
fully virtuialised by perfmon (it is) then why do we care about individual
CPU instances?


> 	- pfm_start(int fd, pfarg_start_t *start)
> 
> 	   Start active monitoring.
> 
> 	- pfm_stop(int fd)
> 
> 	   Stop active monitoring.
> 
> 	- pfm_restart(int fd)
> 
> 	   Resume monitoring after a user level overflow notification. This
> 	   call is used in conjunction with kernel-level sampling.
> 
> 	- pfm_create_evtsets(int fd, pfarg_setdesc_t *sets, int n)
> 
> 	    Create/Modify one or more PMU event set. Each set encapsulates the
> 	    full PMU sets.
> 
> 	- pfm_delete_evtsets(int fd, pfarg_setdesc_t *sets, int n)
> 
> 	    Delete a PMU event set. It is possible to delete one or more sets
> 	    in a single call.
> 
> 	- pfm_getinfo_evtsets(int fd, pfarg_setinfo_t *infos, int n):
> 
> 	    Return information about an event set. It is possible to get
> 	    information about one or more sets in a single call. The call
> 	    returns, for instance, the number of times a set has been activated,
> 	    i.e., loaded onto the actual PMU.
> 
> 	- pfm_unload_context(int fd)
> 
> 	    Detach a context from a thread or a processor.
> 
> 	By default, all counters are exported as 64-bit registers even when the
> 	underlying hardware implements less. This makes it much easier for
> 	applications that are doing event-based sampling because they don't
> 	need to worry about the width of counters. It is possible to turn the
> 	"virtualization" off.
> 
> 	A system-wide context allows a tool to monitor all activities on one
> 	processor, i.e, across all threads. Full System-wide monitoring in an
> 	SMP system is achieved by creating and binding a perfmon context on
> 	each processor. By construction, a perfmon context can only be bound
> 	to one processor at a time.  This design choice is motivated by the
> 	desire to enforce locality and to simplify the kernel implementation.

hm.  I'm surprised at such a CPU-centric approach.  I'd have expected to
see a more task-centric model.
	
> 	Multiple per-thread contexts can coexist at the same time on a system.
> 	Multiple system-wide can co-exist as long as they do not monitor the
> 	same set of processors. The existing implementation does not allow
> 	per-thread and system-wide context to exist at the same time. The
> 	restriction is not inherent to the interface but come from the
> 	existing implementation.
> 	
> 3/ SAMPLING SUPPORT
>    ----------------
> 
> 	The interface supports event-based sampling (EBS), where the sampling
> 	period is determined by the number of occurrences of an event rather
> 	than by time. Note that time-based sampling (TBS) can be emulated by
> 	using an event with some correlation to time.
> 
> 	The EBS is based on the ability for PMU to generate an interrupt when
> 	a counter overflows. All modern PMU support this mode.
> 
> 	Because counters are virtualized to 64 bits. A sampling period p,
> 	is setup by writing a PMD to 2^{64}-p -1 = -p.
> 
> 	The interface does have the notion of a sampling period, it only
> 	manipulates PMD values. When a counter overflows, it is possible
> 	for a tool to request a notification. By construction, the interface
> 	supports as many sampling periods as there are counters on the host
> 	PMU making it possible to overlap distinct sampling measurements in
> 	one run. The notification can be requested per counter.
> 
> 	The notification is sent as a message and can be extracted by invoking
> 	read() on the file descriptor of the context. The overflow message
> 	contains information about the overflow, such as the index of the
> 	overflowed PMD.
> 
> 	The interface supports using select/poll on contexts file descriptors.
> 	Similarly, it is possible to get an asynchronous notification via SIGIO
> 	using the regular sequence of fcntl().

So the kernel buffers these messages for the read()er.  How does it handle
the case of a process which requests the messages but never gets around to
read()ing them?

> 	By default, during a notification, monitoring is masked, i.e., nothing
> 	is captured. A tool uses the pfm_restart() call to resume monitoring.
> 
> 	It is possible to request that on overflow notification, the monitoring
> 	thread be blocked. By default, it keeps on running with monitoring
> 	masked. Blocking is not supported in system-wide mode nor when a thread
> 	is self-monitoring.
> 
> 4/ SAMPLING BUFFER SUPPORT
>    -----------------------
> 
> 	User level sampling works but it is quite expensive especially when for
> 	non self-monitoring threads. To minimize the overhead, the interface also
> 	supports a kernel level sampling buffer. The idea is simple: on overflow
> 	the kernel record a sample, and only when the buffer becomes full is the
> 	user level notification generated. Thus, we amortize the cost of the
> 	notification by simply calling the user when lots of samples are available.
> 
> 	This is not such a new idea, it is present in OProfile or perfctr.
> 	However, the interface needs to remains generic and flexible. If
> 	the sampling buffer is in kernel, its format and what gets recorded
> 	becomes somehow locked by the implementation. Every tool has different
> 	needs. For instance, a tool such as Oprofile may want to aggregate
> 	samples in the kernel, others such as VTUNE want to record all samples
> 	sequentially. Furthermore, some tools may want to combine in each sample
> 	PMU information with other kernel level information, such as a kernel
> 	call stack for instance. It is hard to design a generic buffer format
> 	that can handle all possible request. Instead, the interface provides
> 	an infrastructure  in which the buffer format is implemented by a kernel
> 	module. Each module controls, what gets recorded, how it is recorded,
> 	how the information is exported to user, when a 'buffer full'
> 	notification must be sent. The perfmon core has an interface to
> 	dynamically register new formats. Each format is uniquely identified by
> 	a 128-bit UUID which is passed by the tool when the context is created.
> 	Arguments for the buffer format are also passed during this call.

Well that addresses my earlier questions I guess.

Is this actually useful?  oprofile is there and works OK.  Again, is there
overdesign here?

And why is it necessary to make the presentation of the samples to
userspace pluggable?  I'd have thought that a single relayfs-based
implementation would suit all sampling buffer formats.

> 	As part of the infrastructure, the interface provides a buffer allocation
> 	and remapping service to the buffer format modules. Format may use this
> 	service when the context is created. The kernel memory will be reserved
> 	and the tool will be able to get access to the buffer via remapping
> 	using the mmap() system call. This provides an efficient way of
> 	exporting samples without the need for copying large amount of data
> 	between the kernel and user space. But a format may choose to export
> 	its samples via another way, such as a device driver interface for
> 	instance.

It doesn't sound like perfmon is using relayfs for the sampling buffer. 
Why not?

> 	The current implementation comes with a simple default format builtin.
> 	This format records samples in a sequential order. Each sample has a
> 	fixed sized header which include the interrupted instruction pointer,
> 	which PMD overflowed, the PID and TID of the thread among other things.
> 	The header is followed by an optional variable size body where
> 	additional PMD values can be recorded.
> 	
> 	We have successfully hooked the Oprofile kernel infrastructure to
> 	our interface using a simple buffer format module on Linux/ia64.

Neat, but do we actually *need* this?

> 	We have released a buffer format that implements n-way buffering to
> 	show how blind spots may be minimized. both modules required absolutely
> 	no change to the interface nor perfmon core implementation. We have also
> 	developed a buffer format module to support P4/Xeon Precise Event-Based
> 	Sampling (PEBS).
> 
> 	Because sampling can happen in the kernel without user intervention,
> 	the kernel must have all the information to figure out what to record,
> 	how to restart the sampling period. This information is passed when
> 	the PMD used as the sampling period is programmed. For each such PMD,
> 	it is possible to indicate using bitvector, which PMDs to record on
> 	overflow, which PMDs to reset on overflow.
> 
> 	For each PMD, the interface provides three possible values which are
> 	used when sampling. The initial value is that is first loaded into the
> 	PMD, i.e., the first sampling period. On overflow which does not
> 	trigger a user level notification, a so-called short reset value is
> 	used by the kernel to reload the PMD. After an overflow with a user
> 	level notification, the kernel uses the so-called long reset value.
> 	This mechanism can be exploited to hide the noise induced by the
> 	recovery after a user notification.
> 
> 	The interface also supports automatic randomization of the reset value
> 	for a PMD after an overflow.

Why would one want to randomise the PMD after an overflow?

> Randomization is indicated per PMD and is
> 	controlled by a seed value. The range of variation is specified by a
> 	bitmask per PMD.
> 
> 5/ EVENT SETS AND MULTIPLEXING:
>    ----------------------------
>
> ... 
> 
> 6/ PMU DESCRIPTION MODULES
>    -----------------------
> 
> 	The logical PMU is driven by a PMU description table. The table
> 	is implemented by a kernel pluggable module. As such, it can be
> 	updated at will without recompiling the kernel, waiting for the next
> 	release of a Linux kernel or distribution, and without rebooting the
> 	machine as long as the PMU model belongs to the same PMU family. For
> 	instance, for the Itanium Processor Family, the architecture specifies
> 	the framework for the PMU. Thus the Itanium PMU specific code is common
> 	across all processor implementations. This is not the case for IA-32.

I think the usefulness of this needs justification.  CPUs are updated all
the time, and we release new kernels all the time to exploit the new CPU
features.  What's so special about performance counters that they need such
special treatment?

>
> ...
> 
> 7/ IMPLEMENTATION
>    ---------------
> 
> 	We have developed an implementation for the 2.6.x kernel series.
> 	We do have support for the following processors:
> 
> 		- All Itanium processors (Itanium, McKinley/Madison, Montecito)
> 		- Intel EM64T/Xeon. Includes support for PEBS and HyperThreading (produced by Intel)
> 		- Intel P4/Xeon (32-bit). Includes support for PEBS and HyperThreading
> 		- Intel Pentium M and P6 processors
> 		- AMD 64-bit Opteron
> 		- preliminary support for IBM Power 5 (produced by IBM)
> 		- preliminary support for MIPS R5000 (produced by Phil Mucci)

Which achitectures does perfctr support?   More, I think?

> -Stephane

Thanks for putting this together.  It helps.

Overall: I worry about excessive configurability, excessive features.

Re: [perfmon] Re: quick overview of the perfmon2 interface

[Stephane Eranian]

Andrew,

> > 6/ PMU DESCRIPTION MODULES
> >    -----------------------
> > 
> > 	The logical PMU is driven by a PMU description table. The table
> > 	is implemented by a kernel pluggable module. As such, it can be
> > 	updated at will without recompiling the kernel, waiting for the next
> > 	release of a Linux kernel or distribution, and without rebooting the
> > 	machine as long as the PMU model belongs to the same PMU family. For
> > 	instance, for the Itanium Processor Family, the architecture specifies
> > 	the framework for the PMU. Thus the Itanium PMU specific code is common
> > 	across all processor implementations. This is not the case for IA-32.
> 
> I think the usefulness of this needs justification.  CPUs are updated all
> the time, and we release new kernels all the time to exploit the new CPU
> features.  What's so special about performance counters that they need such
> special treatment?
> 
Given the discussion we are having, I thought it would be useful to take
a concrete example to try and clarify what I am talking about here. I chose
to use the PMU description module/table of the Pentium M because this is
a very common platform supported by all interfaces. The actual module contains
the following (arch/i386/perfmon/perfmon_pm.c) information:

	- desciption of the PMU register: where they are, their type
	- a callback for an option PMC write checker.
	- a probe routine (not shown)
	- an module_init/module_exit (not shown)

Let's look at the informaiton in more details:

The first information is architecture specific structure
used by the architecture specific code (arch/i386/perfmon/perfmon.c).
It contains the information about the MSR addresses for each register
that we want to access. Let's look at PMC0:

	{{MSR_P6_EVNTSEL0, 0}, 0, PFM_REGT_PERFSEL},

   - field 0=MSR_P6_EVNTSEL0: PMC0 is mapped onto MSR EVENTSEL0 (for thread 0)
   - field 1=0: unused Pentium M does not support Hyperthreading (no thread 1)
   - field 2=0: PMC0 is controlling PMD 0
   - field 3=PFM_REGT_PERFSEL: this is a PMU control register

The business about HT is due to the fact that the i386 code is shared
with P4/Xeon.

struct pfm_arch_pmu_info pfm_pm_pmu_info={
	.pmc_addrs = {
		{{MSR_P6_EVNTSEL0, 0}, 0, PFM_REGT_PERFSEL},
                    
		{{MSR_P6_EVNTSEL1, 0}, 1, PFM_REGT_PERFSEL}
	},
	.pmd_addrs = {
		{{MSR_P6_PERFCTR0, 0}, 0, PFM_REGT_CTR},
		{{MSR_P6_PERFCTR1, 0}, 0, PFM_REGT_CTR}
	},
	.pmu_style = PFM_I386_PMU_P6,
	.lps_per_core = 1
};

Now let's look at the mapping table. It contains the following information:
	- attribute of the register
	- logical name
	- default value
	- reserved bitfield

The mapping table describes the very basic and generic properties of a register and
is using the same structure for all PMU models. In contrast the first structure
is totally architecture specific.

static struct pfm_reg_desc pfm_pm_pmc_desc[PFM_MAX_PMCS+1]={
/* pmc0  */ { PFM_REG_W, "PERFSEL0", PFM_PM_PMC_VAL, PFM_PM_PMC_RSVD},
/* pmc1  */ { PFM_REG_W, "PERFSEL1", PFM_PM_PMC_VAL, PFM_PM_PMC_RSVD},
	    { PFM_REG_END} /* end marker */
};

static struct pfm_reg_desc pfm_pm_pmd_desc[PFM_MAX_PMDS+1]={
/* pmd0  */ { PFM_REG_C  , "PERFCTR0", 0x0, -1},
/* pmd1  */ { PFM_REG_C  , "PERFCTR1", 0x0, -1},
	    { PFM_REG_END} /* end marker */
};

Now the write checker. It is used to intervene on the value passed by 
the user when it programs a PMC register. The role of the function is
to ensure that the reserved bitfields retains their default value.
It can be used to verify that a PMC value is actually authorized and
sane. PMU may disallowd certain combination of values. The checker is
optional. On Pentium M we simply enforce resreved bitfields.

static int pfm_pm_pmc_check(struct pfm_context *ctx, struct pfm_event_set *set,
			    u16 cnum, u32 flags, u64 *val)
{
	u64 tmpval, tmp1, tmp2;
	u64 rsvd_mask, dfl_value;

	tmpval = *val;
	rsvd_mask = pfm_pm_pmc_desc[cnum].reserved_mask;
	dfl_value = pfm_pm_pmc_desc[cnum].default_value;

	if (flags & PFM_REGFL_NO_EMUL64)
		dfl_value &= ~(1ULL << 20);

	/* remove reserved areas from user value */
	tmp1 = tmpval & rsvd_mask;

	/* get reserved fields values */
	tmp2 = dfl_value & ~rsvd_mask;
	*val = tmp1 | tmp2;

	return 0;
}

And finally the structure that we register with the core of perfmon.
It includes among other things the actual width of the counters as this
is useful for sampling and 64-bit virtualization of counters.

static struct pfm_pmu_config pfm_pm_pmu_conf={
	.pmu_name = "Intel Pentium M Processor",
	.counter_width = 31,
	.pmd_desc = pfm_pm_pmd_desc,
	.pmc_desc = pfm_pm_pmc_desc,
	.pmc_write_check = pfm_pm_pmc_check,
	.probe_pmu = pfm_pm_probe_pmu,
	.version = "1.0",
	.flags = PMU_FLAGS,
	.owner = THIS_MODULE,
	.arch_info = &pfm_pm_pmu_info
};

This is not much information.

If this is not implemented as a kernel module, it would have to be integrated into
the kernel no matter what. This is very basic information that perfmon needs to operate
on the PMU registers. I prefer the table driven approach to the hardcoding and checking
everywhere. I hope you agree with me here.

The PMU description module is simply a way to separate this information from the
core. Note that the modules can, of course, be compiled in.