Re: kernel api for application profiling

Re: kernel api for application profiling

[Mikael Pettersson]

On Sun, 13 Oct 2002 16:56:01 +0100, John Levon wrote:
>On Sun, Oct 13, 2002 at 03:28:44PM +0200, Mikael Pettersson wrote:
>
>> And before people ask me: perfctr is not yet in official kernels,
>> I'm working on a cleaned up version for 2.5 submission RSN,
>
>This is going to clash head-on with oprofile. However, the two systems
>are both useful in their own rights: oprofile isn't useful for
>interstitial timings as described above, whereas perfctr isn't really
>designed for system profiling.
>
>So I suspect the simplest solution would be to make the config options
>for them exclusive. Comments ?

I've thought about this problem before, and I think there is
a simple & less drastic solution to it.

The HW resources in question are the local APIC LVTPC entry
and the performance counter MSRs. Agreed?

The HW is either free or owned in full by a single client
(the NMI watchdog, oprofile, perfctr, etc). Splitting the
HW resources gets too weird and doesn't actually work.
(Consider the P6s asymmetric EVNTSEL Enable flag for instance.)

A client needs to reserve the HW before it is allowed to touch it.
Normally, a client would release the HW automatically when it no
longer needs it. perfctr already works that way, and I imagine
oprofile could do the same.

Ignoring the NMI watchdog, the resource manager becomes trivial.

The NMI watchdog is a problem. This is a client which normally
doesn't release the HW; however, it should release it if some
other specialised client (like oprofile or perfctr) needs it.
Furthermore, when that client is done, the NMI watchdog should
ideally be activated again.

The NMI watchdog can either be special-cased so that the resource
manager knows that it is a low-priority default owner of the HW,
or we can try to encode this in the interface to the manager, using
callbacks like "are you willing to release the HW?" and results
like "yes, but please call this FUNC when you're done with the HW".

/Mikael

Re: kernel api for application profiling

[John Levon]

On Mon, Oct 14, 2002 at 12:17:23AM +0200, Mikael Pettersson wrote:

> The HW resources in question are the local APIC LVTPC entry
> and the performance counter MSRs. Agreed?

Right.

> The NMI watchdog can either be special-cased so that the resource
> manager knows that it is a low-priority default owner of the HW,
> or we can try to encode this in the interface to the manager, using
> callbacks like "are you willing to release the HW?" and results
> like "yes, but please call this FUNC when you're done with the HW".

I've been thinking along the exact same lines. I even started to
implement something like this originally, but ended up doing a simpler
save/restore thing in oprofile. It would be fairly easy to implement,
the biggest difficulty being the hand-off of the power management
routines and the NMI handler where appropriate.

I agree that it doesn't make sense to split up the resources (though at
some point I'd like to maintain the watchdog functionality even with
oprofile running). In fact, for now, I think the simple exclusive CONFIG
solution is the simplest - the things don't get on together, after all.

regards
john
-- 
"That's just kitten-eating wrong."
	- Richard Henderson

Re: kernel api for application profiling

[James Courtier-Dutton]

John Levon wrote:

>I agree that it doesn't make sense to split up the resources (though at
>some point I'd like to maintain the watchdog functionality even with
>oprofile running). In fact, for now, I think the simple exclusive CONFIG
>solution is the simplest - the things don't get on together, after all.
>
>regards
>john
>  
>
Speaking as the potential user of these tools, having to run a different 
kernel (My translation of exclusive CONFIG) to swich these features on 
and off would be annoying. I would prefer the simple loading/unloading 
of a kernel module to do the job. Having looked at a fair bit of kernel 
module code, I at first think that the kernel module api cannot achieve 
this, but other people on this list might think/know otherwise.
 From the user point of view it is adding/removing a kernel feature so a 
module would seem appropriate, but from a programmers point of view, the 
profiling code would have to be spread about all the kernel code in 
order to accurately catch the profiling information.

Summary: -
Please try to create a profiling kernel runtime loadable module instead 
of an exclusive CONFIG item.

Cheers
James

Re: kernel api for application profiling

[John Levon]

On Mon, Oct 14, 2002 at 12:45:53PM +1000, James Courtier-Dutton wrote:

> >oprofile running). In fact, for now, I think the simple exclusive CONFIG
> > 
> >
> Speaking as the potential user of these tools, having to run a different 
> kernel (My translation of exclusive CONFIG) to swich these features on 

Perhaps you didn't see the "for now" part of my post.

regards
john

-- 
"That's just kitten-eating wrong."
	- Richard Henderson

Re: kernel api for application profiling

[John Levon]

On Sun, Oct 13, 2002 at 03:28:44PM +0200, Mikael Pettersson wrote:

> And before people ask me: perfctr is not yet in official kernels,
> I'm working on a cleaned up version for 2.5 submission RSN,

This is going to clash head-on with oprofile. However, the two systems
are both useful in their own rights: oprofile isn't useful for
interstitial timings as described above, whereas perfctr isn't really
designed for system profiling.

So I suspect the simplest solution would be to make the config options
for them exclusive. Comments ?

regards
john

-- 
"That's just kitten-eating wrong."
	- Richard Henderson

Re: kernel api for application profiling

[Mikael Pettersson]

On Sun, 13 Oct 2002 21:08:30 +1000, James Courtier-Dutton wrote:
>I wish to be able to make a function call to start a timer, and then 
>later on in the program stop the timer and therefore gather information 
>about how long a particular routine took.
>The problem I have is that between start of timer and stop of timer, the 
>kernel might task switch to another thread or process. I would like this 
>kernel task switch to automatically stop the timer, and then restart it 
>when I get CPU time returned. I cannot find any such API availiable 
>currently in the linux kernel.

perfctr & PAPI support this. perfctr virtualises the TSC which
gives you high-res process times, and sampling is extremely
light-weight (no syscalls involved) so measuring small blocks
of code is feasible & accurate.
perfctr is a low-level Linux/x86-specific framework. PAPI is
a higher-level framework which supports a number of platforms.
PAPI uses perfctr on Linux/x86.

Both perfctr and PAPI also let you set up CPU performance
counters for counting other things than clock cycles.

See:
<http://www.csd.uu.se/~mikpe/linux/perfctr/>
<http://icl.cs.utk.edu/projects/papi/>

>Also, as an extension to this, it would be nice to know which other 
>tasks happened during the task switch. The reason I would like this, is 
>so that I can tell if the time was taken reading data off the hard disk 
>or instead time spent by X displaying/transfering to screen the next 
>video frame. If too much time is being spent doing a task like reading 
>from the hard disk, I would need to look into reducing the latency of 
>that task. This might also highlight buggy modules that take up too much 
>of a time slice. I have seen some task switches take 800ms away from my 
>audio out thread, and therefore causing underruns on the sound hardware, 
>that then causes glitches in the perceived audio coming from the speakers.

This could perhaps be measured by programming a TSC-like event into
a performance counter and programming it to count in kernel-mode only.
P6, K7, and P4 should all support this.

And before people ask me: perfctr is not yet in official kernels,
I'm working on a cleaned up version for 2.5 submission RSN,
perfctr isn't easy for newbies to use due to HW-specific details,
but PAPI gives you a nice cosy high-level API.

/Mikael

kernel api for application profiling

[James Courtier-Dutton]

Hello

I am helping to write a multi-media application that plays lots of 
different video and audio formats.
I would like to improve the performance of the program.
I wish to be able to make a function call to start a timer, and then 
later on in the program stop the timer and therefore gather information 
about how long a particular routine took.
The problem I have is that between start of timer and stop of timer, the 
kernel might task switch to another thread or process. I would like this 
kernel task switch to automatically stop the timer, and then restart it 
when I get CPU time returned. I cannot find any such API availiable 
currently in the linux kernel.

Also, as an extension to this, it would be nice to know which other 
tasks happened during the task switch. The reason I would like this, is 
so that I can tell if the time was taken reading data off the hard disk 
or instead time spent by X displaying/transfering to screen the next 
video frame. If too much time is being spent doing a task like reading 
from the hard disk, I would need to look into reducing the latency of 
that task. This might also highlight buggy modules that take up too much 
of a time slice. I have seen some task switches take 800ms away from my 
audio out thread, and therefore causing underruns on the sound hardware, 
that then causes glitches in the perceived audio coming from the speakers.

Also, the reason for the above requests is so that I can gather useful 
performance info without having to trawl through hugh global performance 
traces.

Can anyone help me?

Cheers
James