RE: [perfmon] Re: quick overview of the perfmon2 interface

RE: [perfmon] Re: quick overview of the perfmon2 interface

[Truong, Dan]

> Thanks to David, Dan and Phil for their comments.

Another note on the urgency of standardizing Perfmon:

Anarchy is not a good breeding ground for tools that need a
stable infrastructure to mature. Being "there" is what made
PAPI and perfctr popular and somewhat standard infrastructure.

Compilers, tools, JVMs... -you name it- are all moving
fast towards using hardware counters to get feedback,
tune, monitor or measure application behavior.

The PMU is becoming a standard commodity. Once Perfmon is
"the" Linux interface, all the tools can align on it and
coexist, push their R&D forward, and more importantly become
fully productized for businesses usage. Hopefully Perfmon's
interface is powerful enough to support future needs.

Good luck Stephane :)

Cheers,

Dan-

Re: [perfmon] Re: quick overview of the perfmon2 interface

[Andrew Morton]

"Truong, Dan" <dan.truong@hp.com> wrote:
>
> The PMU is becoming a standard commodity. Once Perfmon is
> "the" Linux interface, all the tools can align on it and
> coexist, push their R&D forward, and more importantly become
> fully productized for businesses usage.
>

The apparently-extreme flexibility of the perfmon interfaces would tend to
militate against that, actually.  It'd become better productised if it had
one interface and stuck to it.

(I haven't processed Stephane's reply yet - will get there)

RE: [perfmon] Re: quick overview of the perfmon2 interface

[Bryan O'Sullivan]

On Fri, 2006-01-20 at 10:37 -0800, Truong, Dan wrote:
> Would you want Stephane to guard the extended
> functionalities with tunables or something to
> Disable their regular use and herd enterprise
> Tools into a standard mold... yet allow R&D to
> Move on by enabling the extentions?

I'd prefer to see all of the extended stuff left out entirely for now.
The mainline kernel has no PMU support for any popular architecture,
even though external patches have existed in stable form for years.
Filling that gap ought to be the priority; the interface can be extended
when actual users of new features show up and ask for them.

> It would restrict the R&D mindset, and new ideas.
> The field hasn't grown yet to a stable mature form.

The place for flailing around with uncooked ideas is arguably not the
mainline kernel.

> Flexibility is/was needed because:
> - Tools need to port to Perfmon with min cost.
> - Ability to support novel R&D ideas.
> - Ability to support growth beyond just PMU data
> - Allows early data aggregation
> - Allow OS data correlated to PMU

Speculatively adding complicated and unused interfaces to the kernel in
the hope that some wild-eyed visionary might eventually up and use them
helps nobody.

	<b

Re: [perfmon] Re: quick overview of the perfmon2 interface

[Andrew Morton]

"Truong, Dan" <dan.truong@hp.com> wrote:
>
> Would you want Stephane to guard the extended
> functionalities with tunables or something to
> Disable their regular use and herd enterprise
> Tools into a standard mold... yet allow R&D to
> Move on by enabling the extentions?

argh.  I'd prefer to avoid one-month gaps in the conversation, so we don't
all forget what we were talking about.

Look, we just need to get these patches on the wire so we can all look at
them, see what they do, understand what decisions were taken and why.

The conciseness and completeness of those patches' covering descriptions
will be key to helping this process along.

RE: [perfmon] Re: quick overview of the perfmon2 interface

[Truong, Dan]

Would you want Stephane to guard the extended
functionalities with tunables or something to
Disable their regular use and herd enterprise
Tools into a standard mold... yet allow R&D to
Move on by enabling the extentions?



Just crippling flexibility/cutting functionality
is like removing words out of a dictionary to
prevent people from thinking different.

It would restrict the R&D mindset, and new ideas.
The field hasn't grown yet to a stable mature form.
It is just beginning: profiling, monitoring, tuning,
compilers, JIT...

Flexibility is/was needed because:
- Tools need to port to Perfmon with min cost.
- Ability to support novel R&D ideas.
- Ability to support growth beyond just PMU data
- Allows early data aggregation
- Allow OS data correlated to PMU

What standardization adds:
- Coordinated access to PMU rssources from all tools
- All tools/formats etc all plug into the same OS framework.
- The interface gets ported across multiple platforms.
- The functionality is rich for all (fast data transfers,
  multiplexing, system vs thead, etc.)

Dan-

> -----Original Message-----
> From: perfmon-bounces@napali.hpl.hp.com [mailto:perfmon-
> bounces@napali.hpl.hp.com] On Behalf Of Andrew Morton
> Sent: Thursday, December 22, 2005 5:47 AM
> To: Truong, Dan
> Cc: Eranian, Stephane; perfmon@napali.hpl.hp.com; linux-
> ia64@vger.kernel.org; linux-kernel@vger.kernel.org; perfctr-
> devel@lists.sourceforge.net
> Subject: Re: [perfmon] Re: quick overview of the perfmon2 interface
> 
> "Truong, Dan" <dan.truong@hp.com> wrote:
> >
> > The PMU is becoming a standard commodity. Once Perfmon is
> > "the" Linux interface, all the tools can align on it and
> > coexist, push their R&D forward, and more importantly become
> > fully productized for businesses usage.
> >
> 
> The apparently-extreme flexibility of the perfmon interfaces would
tend to
> militate against that, actually.  It'd become better productised if it
had
> one interface and stuck to it.
> 
> (I haven't processed Stephane's reply yet - will get there)
> 
> _______________________________________________
> perfmon mailing list
> perfmon@linux.hpl.hp.com
> http://www.hpl.hp.com/hosted/linux/mail-archives/perfmon/

Re: [perfmon] Re: quick overview of the perfmon2 interface

[Stephane Eranian]

Andrew,

> > 6/ PMU DESCRIPTION MODULES
> >    -----------------------
> > 
> > 	The logical PMU is driven by a PMU description table. The table
> > 	is implemented by a kernel pluggable module. As such, it can be
> > 	updated at will without recompiling the kernel, waiting for the next
> > 	release of a Linux kernel or distribution, and without rebooting the
> > 	machine as long as the PMU model belongs to the same PMU family. For
> > 	instance, for the Itanium Processor Family, the architecture specifies
> > 	the framework for the PMU. Thus the Itanium PMU specific code is common
> > 	across all processor implementations. This is not the case for IA-32.
> 
> I think the usefulness of this needs justification.  CPUs are updated all
> the time, and we release new kernels all the time to exploit the new CPU
> features.  What's so special about performance counters that they need such
> special treatment?
> 
Given the discussion we are having, I thought it would be useful to take
a concrete example to try and clarify what I am talking about here. I chose
to use the PMU description module/table of the Pentium M because this is
a very common platform supported by all interfaces. The actual module contains
the following (arch/i386/perfmon/perfmon_pm.c) information:

	- desciption of the PMU register: where they are, their type
	- a callback for an option PMC write checker.
	- a probe routine (not shown)
	- an module_init/module_exit (not shown)

Let's look at the informaiton in more details:

The first information is architecture specific structure
used by the architecture specific code (arch/i386/perfmon/perfmon.c).
It contains the information about the MSR addresses for each register
that we want to access. Let's look at PMC0:

	{{MSR_P6_EVNTSEL0, 0}, 0, PFM_REGT_PERFSEL},

   - field 0=MSR_P6_EVNTSEL0: PMC0 is mapped onto MSR EVENTSEL0 (for thread 0)
   - field 1=0: unused Pentium M does not support Hyperthreading (no thread 1)
   - field 2=0: PMC0 is controlling PMD 0
   - field 3=PFM_REGT_PERFSEL: this is a PMU control register

The business about HT is due to the fact that the i386 code is shared
with P4/Xeon.

struct pfm_arch_pmu_info pfm_pm_pmu_info={
	.pmc_addrs = {
		{{MSR_P6_EVNTSEL0, 0}, 0, PFM_REGT_PERFSEL},
                    
		{{MSR_P6_EVNTSEL1, 0}, 1, PFM_REGT_PERFSEL}
	},
	.pmd_addrs = {
		{{MSR_P6_PERFCTR0, 0}, 0, PFM_REGT_CTR},
		{{MSR_P6_PERFCTR1, 0}, 0, PFM_REGT_CTR}
	},
	.pmu_style = PFM_I386_PMU_P6,
	.lps_per_core = 1
};

Now let's look at the mapping table. It contains the following information:
	- attribute of the register
	- logical name
	- default value
	- reserved bitfield

The mapping table describes the very basic and generic properties of a register and
is using the same structure for all PMU models. In contrast the first structure
is totally architecture specific.

static struct pfm_reg_desc pfm_pm_pmc_desc[PFM_MAX_PMCS+1]={
/* pmc0  */ { PFM_REG_W, "PERFSEL0", PFM_PM_PMC_VAL, PFM_PM_PMC_RSVD},
/* pmc1  */ { PFM_REG_W, "PERFSEL1", PFM_PM_PMC_VAL, PFM_PM_PMC_RSVD},
	    { PFM_REG_END} /* end marker */
};

static struct pfm_reg_desc pfm_pm_pmd_desc[PFM_MAX_PMDS+1]={
/* pmd0  */ { PFM_REG_C  , "PERFCTR0", 0x0, -1},
/* pmd1  */ { PFM_REG_C  , "PERFCTR1", 0x0, -1},
	    { PFM_REG_END} /* end marker */
};

Now the write checker. It is used to intervene on the value passed by 
the user when it programs a PMC register. The role of the function is
to ensure that the reserved bitfields retains their default value.
It can be used to verify that a PMC value is actually authorized and
sane. PMU may disallowd certain combination of values. The checker is
optional. On Pentium M we simply enforce resreved bitfields.

static int pfm_pm_pmc_check(struct pfm_context *ctx, struct pfm_event_set *set,
			    u16 cnum, u32 flags, u64 *val)
{
	u64 tmpval, tmp1, tmp2;
	u64 rsvd_mask, dfl_value;

	tmpval = *val;
	rsvd_mask = pfm_pm_pmc_desc[cnum].reserved_mask;
	dfl_value = pfm_pm_pmc_desc[cnum].default_value;

	if (flags & PFM_REGFL_NO_EMUL64)
		dfl_value &= ~(1ULL << 20);

	/* remove reserved areas from user value */
	tmp1 = tmpval & rsvd_mask;

	/* get reserved fields values */
	tmp2 = dfl_value & ~rsvd_mask;
	*val = tmp1 | tmp2;

	return 0;
}

And finally the structure that we register with the core of perfmon.
It includes among other things the actual width of the counters as this
is useful for sampling and 64-bit virtualization of counters.

static struct pfm_pmu_config pfm_pm_pmu_conf={
	.pmu_name = "Intel Pentium M Processor",
	.counter_width = 31,
	.pmd_desc = pfm_pm_pmd_desc,
	.pmc_desc = pfm_pm_pmc_desc,
	.pmc_write_check = pfm_pm_pmc_check,
	.probe_pmu = pfm_pm_probe_pmu,
	.version = "1.0",
	.flags = PMU_FLAGS,
	.owner = THIS_MODULE,
	.arch_info = &pfm_pm_pmu_info
};

This is not much information.

If this is not implemented as a kernel module, it would have to be integrated into
the kernel no matter what. This is very basic information that perfmon needs to operate
on the PMU registers. I prefer the table driven approach to the hardcoding and checking
everywhere. I hope you agree with me here.

The PMU description module is simply a way to separate this information from the
core. Note that the modules can, of course, be compiled in.

RE: [perfmon] Re: quick overview of the perfmon2 interface

[Truong, Dan]

Just a couple of cents here to support Stephane's design :)


> Why would one want to change the format of the sampling buffer?

The idea is to allow user custom formats for one, and allow Exotic
architectures for second.

Say you want to reduce the volume of data passed to the application And
stored in the buffers, you can then do some pre-processing Inside the
kernel via a custom module.

You can also return more complex data than just PMU registers, think
Call stacks or other OS related information that can complement the The
PMU data.

Some data can be returned vis pseudo PMU registers (i.e. extentions),
Like the interval timer, PID/TID, etc. but for more complex and less
Synchronous data you may end up needed a more powerfull buffer format
With headers etc.

Another issue can be if hardware buffer support is provided. The
hardware Buffer support will not allow collection of pseudo-counters
which are Supported by software, so again the packaging may not be as
linear as A repeated sequence of counters...

With Stephane we had discussed this buffer format, and came to the
Conclusion that flexibility there will avoid hitting the wall.

You don't know what tomorrow is made of (yet)...



> > 	The PMU register description is implemented by a kernel
pluggable
> 
> Is that option important, or likely to be useful?  Are you sure there 
> isn't some overdesign here?

It will allow bringup of new PMUs on new architectures more easily, and
simpler distribution of support. It will also allow CPU designers to
create custom drivers that support non-public features to debug the
CPUs.



> hm.  I'm surprised at such a CPU-centric approach.  I'd have expected 
> to see a more task-centric model.

Both per-thread and system-wide measurments are useful.

Systemwide is used to evaluate the beavior of the whole system when
tuning a large load (think TPC-C, SpecWeb, SpecJapp...) Per thread is
used for specific application/thread tuning and self monitoring. Also
per-thread monitoring is not always adviseable, for example when there
Are a large number of threads loading the system, adding that many
monitors will impact the system performance, so you will want to measure
per CPU.

> So the kernel buffers these messages for the read()er.  How does it 
> handle the case of a process which requests the messages but never 
> gets around to read()ing them?

Stephane, I would assume that the monitoring session attached to
The buffer returning the message just stalls. If there is multiplexing,
Will coming back to that stalled buffer stall all the multiplexed
Sessions? I would assume so.



> Why would one want to randomise the PMD after an overflow?

Everybody does it :) Helps generate an un-biased picture.



> I think the usefulness of this needs justification.  CPUs are updated 
> all the time, and we release new kernels all the time to exploit the 
> new CPU features.  What's so special about performance counters that 
> they need such special treatment?

The PMU is not fully architected usually. Nothing prevents a
Model to be shipped with PMU upgrades.
Also the PMU can be used by architects for validation of the designs.
Easier early access to the functionalities helps.

The PMU is a direct evolution of the debug counters that were used
To debug CPUs but not available for general use. They are still used
In that fashion too, and a main reason they exist.


Cheers,

Dan-