perf: prctl(PR_TASK_PERF_EVENTS_DISABLE) has no effect

perf: prctl(PR_TASK_PERF_EVENTS_DISABLE) has no effect

[Andrew Steets]

prctl(PR_TASK_PERF_EVENTS_DISABLE) doesn't appear to disable perf event
counters.  Here is a demonstration program:

#include <linux/prctl.h>

void loop_events_disabled() {
     volatile int x;
     int i;
     for (i = 0; i < 1000000000; i++)
         x = i;
     return;
}

void loop_events_enabled() {
     volatile int x;
     int i;
     for (i = 0; i < 1000000000; i++)
         x = i;
     return;
}

int main(int argc, char **argv) {
     prctl(PR_TASK_PERF_EVENTS_ENABLE);
     loop_events_enabled();
     prctl(PR_TASK_PERF_EVENTS_DISABLE);
     loop_events_disabled();
}

I would not expect to see loop_events_disabled() show up in the profile
as reported by perf report, but it does.

$ perf record ./a.out
$ perf report -n --stdio
# Events: 3K cycles
#
# Overhead  Samples    Command      Shared Object                Symbol
# ........ ..........  .......  .................  ....................
#
     51.80%       1679    a.out  a.out              [.] loop_events_enabled
     48.07%       1578    a.out  a.out              [.] loop_events_disabled
      0.03%          5    a.out  [kernel.kallsyms]  [k] intel_pmu_enable_all
      0.03%          1    a.out  [kernel.kallsyms]  [k] sched_clock
      0.03%          1    a.out  [kernel.kallsyms]  [k] ktime_get
      0.03%          1    a.out  [kernel.kallsyms]  [k] update_cpu_load
      0.01%          1    a.out  [sunrpc]           [k] generic_match

I have tested this on several kernels including 3.3rc1.  Can anyone tell
me if I'm using this wrong or if this is a bug?

-Andrew

-- 

---------------------------------------------------------------
This email, along with any attachments, is confidential. If you 
believe you received this message in error, please contact the 
sender immediately and delete all copies of the message.  
Thank you.

Re: perf: prctl(PR_TASK_PERF_EVENTS_DISABLE) has no effect

[Peter Zijlstra]

On Fri, 2012-01-27 at 11:03 -0600, Andrew Steets wrote:
> Can anyone tell
> me if I'm using this wrong or if this is a bug? 

You're using it wrong, it will disable events you own (created) not
events that monitor you.

Re: perf: prctl(PR_TASK_PERF_EVENTS_DISABLE) has no effect

[Andrew Steets]

On 1/27/12 11:12 AM, Peter Zijlstra wrote:
> On Fri, 2012-01-27 at 11:03 -0600, Andrew Steets wrote:
>> Can anyone tell
>> me if I'm using this wrong or if this is a bug?
>
> You're using it wrong, it will disable events you own (created) not
> events that monitor you.

Is there an alternate way of disabling events that monitor the current
process?  I ask because I came across the following description in
tools/perf/design.txt:

     A process can enable or disable all the counter groups that are
     attached to it, using prctl:

         prctl(PR_TASK_PERF_EVENTS_ENABLE);

         prctl(PR_TASK_PERF_EVENTS_DISABLE);

     This applies to all counters on the current process, whether created
     by this process or by another, and doesn't affect any counters that
     this process has created on other processes.  It only enables or
     disables the group leaders, not any other members in the groups.

-Andrew

-- 

---------------------------------------------------------------
This email, along with any attachments, is confidential. If you 
believe you received this message in error, please contact the 
sender immediately and delete all copies of the message.  
Thank you.

Re: perf: prctl(PR_TASK_PERF_EVENTS_DISABLE) has no effect

[Peter Zijlstra]

On Fri, 2012-01-27 at 14:06 -0600, Andrew Steets wrote:
> On 1/27/12 11:12 AM, Peter Zijlstra wrote:
> > On Fri, 2012-01-27 at 11:03 -0600, Andrew Steets wrote:
> >> Can anyone tell
> >> me if I'm using this wrong or if this is a bug?
> >
> > You're using it wrong, it will disable events you own (created) not
> > events that monitor you.
> 
> Is there an alternate way of disabling events that monitor the current
> process? 

Nope, nor will there ever be.

>  I ask because I came across the following description in
> tools/perf/design.txt:
> 
>      A process can enable or disable all the counter groups that are
>      attached to it, using prctl:
> 
>          prctl(PR_TASK_PERF_EVENTS_ENABLE);
> 
>          prctl(PR_TASK_PERF_EVENTS_DISABLE);
> 
>      This applies to all counters on the current process, whether created
>      by this process or by another, and doesn't affect any counters that
>      this process has created on other processes.  It only enables or
>      disables the group leaders, not any other members in the groups.


That's wrong.. also I wouldn't ever allow such a 'feature', that's just
asking for trouble.

Re: perf: prctl(PR_TASK_PERF_EVENTS_DISABLE) has no effect

[Ingo Molnar]

* Andrew Steets <asteets@rgmadvisors.com> wrote:

> prctl(PR_TASK_PERF_EVENTS_DISABLE) doesn't appear to disable 
> perf event counters.  Here is a demonstration program:

btw., what's your usecase?

Thanks,

	Ingo

Re: perf: prctl(PR_TASK_PERF_EVENTS_DISABLE) has no effect

[Andrew Steets]

On 1/28/12 6:01 AM, Ingo Molnar wrote:

>> prctl(PR_TASK_PERF_EVENTS_DISABLE) doesn't appear to disable
>> perf event counters.  Here is a demonstration program:
>
> btw., what's your usecase?
>

I'm trying to profile a small section of a long-running program.  I ran
into trouble using call-graph recording and I thought this might be an
alternative way of getting what I was after.

-Andrew

-- 

---------------------------------------------------------------
This email, along with any attachments, is confidential. If you 
believe you received this message in error, please contact the 
sender immediately and delete all copies of the message.  
Thank you.

Re: perf: prctl(PR_TASK_PERF_EVENTS_DISABLE) has no effect

[Ingo Molnar]

* Andrew Steets <asteets@rgmadvisors.com> wrote:

> On 1/28/12 6:01 AM, Ingo Molnar wrote:
> 
> >> prctl(PR_TASK_PERF_EVENTS_DISABLE) doesn't appear to 
> >> disable perf event counters.  Here is a demonstration 
> >> program:
> >
> > btw., what's your usecase?
> 
> I'm trying to profile a small section of a long-running 
> program.  I ran into trouble using call-graph recording and I 
> thought this might be an alternative way of getting what I was 
> after.

That usecase indeed makes sense. Peter, could we allow this for 
privileged tasks, depending on the perf_paranoia settings or 
such?

Thanks,

	Ingo

Re: perf: prctl(PR_TASK_PERF_EVENTS_DISABLE) has no effect

[Gleb Natapov]

On Sun, Jan 29, 2012 at 05:32:35PM +0100, Ingo Molnar wrote:
> 
> * Andrew Steets <asteets@rgmadvisors.com> wrote:
> 
> > On 1/28/12 6:01 AM, Ingo Molnar wrote:
> > 
> > >> prctl(PR_TASK_PERF_EVENTS_DISABLE) doesn't appear to 
> > >> disable perf event counters.  Here is a demonstration 
> > >> program:
> > >
> > > btw., what's your usecase?
> > 
> > I'm trying to profile a small section of a long-running 
> > program.  I ran into trouble using call-graph recording and I 
> > thought this might be an alternative way of getting what I was 
> > after.
> 
> That usecase indeed makes sense. Peter, could we allow this for 
> privileged tasks, depending on the perf_paranoia settings or 
> such?
> 
This sounds useful not only for privileged tasks. Why not make
it event attribute? If user wants PR_TASK_PERF_EVENTS_DISABLE be
respected for an event it can specify special argument like:

perf record -e cycles:n ./a.out

--
			Gleb.

Re: perf: prctl(PR_TASK_PERF_EVENTS_DISABLE) has no effect

[Peter Zijlstra]

On Sun, 2012-01-29 at 17:32 +0100, Ingo Molnar wrote:
> * Andrew Steets <asteets@rgmadvisors.com> wrote:
> 
> > On 1/28/12 6:01 AM, Ingo Molnar wrote:
> > 
> > >> prctl(PR_TASK_PERF_EVENTS_DISABLE) doesn't appear to 
> > >> disable perf event counters.  Here is a demonstration 
> > >> program:
> > >
> > > btw., what's your usecase?
> > 
> > I'm trying to profile a small section of a long-running 
> > program.  I ran into trouble using call-graph recording and I 
> > thought this might be an alternative way of getting what I was 
> > after.
> 
> That usecase indeed makes sense. Peter, could we allow this for 
> privileged tasks, depending on the perf_paranoia settings or 
> such?

I really dislike it. The sane way around this would be to allow easy
self-profiling instead of doing things arse about face like that.

Re: perf: prctl(PR_TASK_PERF_EVENTS_DISABLE) has no effect

[Ingo Molnar]

* Peter Zijlstra <a.p.zijlstra@chello.nl> wrote:

> On Sun, 2012-01-29 at 17:32 +0100, Ingo Molnar wrote:
> > * Andrew Steets <asteets@rgmadvisors.com> wrote:
> > 
> > > On 1/28/12 6:01 AM, Ingo Molnar wrote:
> > > 
> > > >> prctl(PR_TASK_PERF_EVENTS_DISABLE) doesn't appear to 
> > > >> disable perf event counters.  Here is a demonstration 
> > > >> program:
> > > >
> > > > btw., what's your usecase?
> > > 
> > > I'm trying to profile a small section of a long-running 
> > > program.  I ran into trouble using call-graph recording 
> > > and I thought this might be an alternative way of getting 
> > > what I was after.
> > 
> > That usecase indeed makes sense. Peter, could we allow this 
> > for privileged tasks, depending on the perf_paranoia 
> > settings or such?
> 
> I really dislike it. The sane way around this would be to 
> allow easy self-profiling instead of doing things arse about 
> face like that.

So, what workflow are you suggesting to Andrew?

I guess we are also hurting from the lack of dwarf stack 
backtrace decoding - that would allow the filtering by parent 
function without modifying the code. I think Frederic had a 
prototype working for 32-bit - any update on that?

Andrew could work that problem around right now by adding:

   -fno-omit-frame-pointer

to the build of the utility - that should activate -g and 
perf-report's --parent filter should also work fine.

Thanks,

	Ingo

Re: perf: prctl(PR_TASK_PERF_EVENTS_DISABLE) has no effect

[Peter Zijlstra]

On Mon, 2012-01-30 at 11:11 +0100, Ingo Molnar wrote:
> So, what workflow are you suggesting to Andrew?

Librarize perf record, then in your code do something like:

#include "perf_record.h"

  handle = perf_record_init(); /* creates perf events and creates
                                  a record thread that writes samples
                                  to perf.data, consumes env(PERF_*)
                                  for configuration, registers with
                                  at_exit() for cleanup */
  if (!handle)
    /* burn */

  /* do you other code */

  perf_record_start(handle);

  /* do the bit you want profiled */

  perf_record_stop(handle);

Then build with -lperfrecord or so. Not too hard, right?

Re: perf: prctl(PR_TASK_PERF_EVENTS_DISABLE) has no effect

[Ingo Molnar]

* Peter Zijlstra <a.p.zijlstra@chello.nl> wrote:

> On Mon, 2012-01-30 at 11:11 +0100, Ingo Molnar wrote:
> > So, what workflow are you suggesting to Andrew?
> 
> Librarize perf record, then in your code do something like:
> 
> #include "perf_record.h"

Maybe. (and then it shouldnt be limited to perf_record.h but 
should be events.h plus libevents.so or such)

> 
>   handle = perf_record_init(); /* creates perf events and creates
>                                   a record thread that writes samples
>                                   to perf.data, consumes env(PERF_*)
>                                   for configuration, registers with
>                                   at_exit() for cleanup */
>   if (!handle)
>     /* burn */
> 
>   /* do you other code */
> 
>   perf_record_start(handle);
> 
>   /* do the bit you want profiled */
> 
>   perf_record_stop(handle);
> 
> Then build with -lperfrecord or so. Not too hard, right?

Isnt a simple prctl() so much easier and faster?

What's your concern with the prctl()? This would arguably be the 
right kind of usage for prctl(): it's an established API/ABI for 
process/task-wide settings.

Thanks,

	Ingo

Re: perf: prctl(PR_TASK_PERF_EVENTS_DISABLE) has no effect

[Peter Zijlstra]

On Mon, 2012-01-30 at 12:31 +0100, Ingo Molnar wrote:
> * Peter Zijlstra <a.p.zijlstra@chello.nl> wrote:
> 
> > On Mon, 2012-01-30 at 11:11 +0100, Ingo Molnar wrote:
> > > So, what workflow are you suggesting to Andrew?
> > 
> > Librarize perf record, then in your code do something like:
> > 
> > #include "perf_record.h"
> 
> Maybe. (and then it shouldnt be limited to perf_record.h but 
> should be events.h plus libevents.so or such)

Yes it should be, you want to reserve the more generic name for less
narrow interfaces.

> > 
> >   handle = perf_record_init(); /* creates perf events and creates
> >                                   a record thread that writes samples
> >                                   to perf.data, consumes env(PERF_*)
> >                                   for configuration, registers with
> >                                   at_exit() for cleanup */
> >   if (!handle)
> >     /* burn */
> > 
> >   /* do you other code */
> > 
> >   perf_record_start(handle);
> > 
> >   /* do the bit you want profiled */
> > 
> >   perf_record_stop(handle);
> > 
> > Then build with -lperfrecord or so. Not too hard, right?
> 
> Isnt a simple prctl() so much easier and faster?

I really don't want to add another two prctl()s for this, ideally I'd
remove the ones we have now, but I've never done due to maintaining
backwards blah..

> What's your concern with the prctl()? This would arguably be the 
> right kind of usage for prctl(): it's an established API/ABI for 
> process/task-wide settings.

Its doing things backwards, also the whole concept of allowing people to
hide things from a profiler is so rotten I'm not willing to even
consider the notion.

Re: perf: prctl(PR_TASK_PERF_EVENTS_DISABLE) has no effect

[Ingo Molnar]

* Peter Zijlstra <a.p.zijlstra@chello.nl> wrote:

> > What's your concern with the prctl()? This would arguably be 
> > the right kind of usage for prctl(): it's an established 
> > API/ABI for process/task-wide settings.
> 
> Its doing things backwards, [...]

What does that mean?

> [...] also the whole concept of allowing people to hide things 
> from a profiler is so rotten I'm not willing to even consider 
> the notion.

But what we want here is not to hide things from the profiler, 
what we want to be able to is to *ask* the profiler to hide 
things for us, and we do that for a good reason.

This distinction (which I agree is important) could be 
implemented by adding a "allow self-profiling" bit (default 
disabled) in the perf_attr. That way such code would not be able 
to 'hide' from a simple:

  perf top
  perf record -a

session, but would be able to self-profile from such a session:

  perf record --allow-self-profile ...

where there could be some easy shortcut for 
--allow-self-profile, such as:

  perf record -S

that way no code could ever hide from a profiler, only if the 
profiler is specifically allowing self-profiling. (opt in)

The librarization you suggested might make sense too - but i 
think people will gravitate towards the easier to use variant, 
and prctl() is as easy and straightforward as it gets.

We can Cc: it to Linus with an explicit [RFC] and he can shoot 
it down if the API is ugly - but I don't think it's ugly.

Hm?

Thanks,

	Ingo

Re: perf: prctl(PR_TASK_PERF_EVENTS_DISABLE) has no effect

[Arnaldo Carvalho de Melo]

Em Mon, Jan 30, 2012 at 12:01:33PM +0100, Peter Zijlstra escreveu:
> On Mon, 2012-01-30 at 11:11 +0100, Ingo Molnar wrote:
> > So, what workflow are you suggesting to Andrew?
> 
> Librarize perf record, then in your code do something like:
> 
> #include "perf_record.h"
> 
>   handle = perf_record_init(); /* creates perf events and creates
>                                   a record thread that writes samples
>                                   to perf.data, consumes env(PERF_*)
>                                   for configuration, registers with
>                                   at_exit() for cleanup */
>   if (!handle)
>     /* burn */
> 
>   /* do you other code */
> 
>   perf_record_start(handle);
> 
>   /* do the bit you want profiled */
> 
>   perf_record_stop(handle);
> 
> Then build with -lperfrecord or so. Not too hard, right?

This looks like the test__PERF_RECORD function in
tools/perf/builtin-test.c, copying here just the equivalent parts to the
above:

	#include "util/evlist.h"

        struct perf_record_opts opts = {
                .target_pid = -1,
                .target_tid = -1,
                .no_delay   = true,
                .freq       = 10,
                .mmap_pages = 256,
                .sample_id_all_avail = true,
        };

        struct perf_evlist *handle = perf_evlist__new(NULL, NULL);

        /* 
         * We need at least one evsel in the evlist, use the default 
         * one: "cycles". 
         */ 
        err = perf_evlist__add_default(evlist);

	perf_evlist__config_attrs(evlist, &opts);

	err = perf_evlist__open(evlist, opts.group);

        perf_evlist__enable(evlist);

	/* do the bit you want profiled */

        perf_evlist__disable(evlist);


----


perf_evlist__config_attrs will, among other chores, do:

        if (opts->target_pid == -1 && opts->target_tid == -1 &&
            !opts->system_wide) {
                attr->disabled = 1;
                attr->enable_on_exec = 1;
        }

And you can fudge it a bit more to your liking before calling
perf_evlist__open().

	This 'perf test' use case was more for preparing for monitoring
a workload that a program would start, setting up the events, etc,
leaving all disabled and then letting the workload begin which would, by
means of enable_on_exec start the counters/events.

	But I guess you can just setup evlist->workload.pid to getpid()
and use perf_evlist__enable()/ perf_evlist_disable() pairs to your
liking.

	Writing a test case that would do exactly what you want in 'perf
test' would be awesome and would allow us to figure out if the
abstractions in place are enough for your use case. Then we could
librarize just this subset and go from there.

	Take a look as well at test__basic_mmap, it may be useful as
another example of using these evsel/evlist classes.

	What is needed to link you can find in tools/perf/util/setup.py,
remove util/python.c and you should be set.

- Arnaldo

Re: perf: prctl(PR_TASK_PERF_EVENTS_DISABLE) has no effect

[Arnaldo Carvalho de Melo]

Em Mon, Jan 30, 2012 at 02:45:07PM +0100, Peter Zijlstra escreveu:
> On Mon, 2012-01-30 at 12:31 +0100, Ingo Molnar wrote:
> > * Peter Zijlstra <a.p.zijlstra@chello.nl> wrote:
> > > On Mon, 2012-01-30 at 11:11 +0100, Ingo Molnar wrote:
> > > > So, what workflow are you suggesting to Andrew?
> > > Librarize perf record, then in your code do something like:
> > > #include "perf_record.h"

> > Maybe. (and then it shouldnt be limited to perf_record.h but 
> > should be events.h plus libevents.so or such)

> Yes it should be, you want to reserve the more generic name for less
> narrow interfaces.

Agreed.
 
- Arnaldo

Re: perf: prctl(PR_TASK_PERF_EVENTS_DISABLE) has no effect

[Frederic Weisbecker]

On Mon, Jan 30, 2012 at 11:11:21AM +0100, Ingo Molnar wrote:
> 
> * Peter Zijlstra <a.p.zijlstra@chello.nl> wrote:
> 
> > On Sun, 2012-01-29 at 17:32 +0100, Ingo Molnar wrote:
> > > * Andrew Steets <asteets@rgmadvisors.com> wrote:
> > > 
> > > > On 1/28/12 6:01 AM, Ingo Molnar wrote:
> > > > 
> > > > >> prctl(PR_TASK_PERF_EVENTS_DISABLE) doesn't appear to 
> > > > >> disable perf event counters.  Here is a demonstration 
> > > > >> program:
> > > > >
> > > > > btw., what's your usecase?
> > > > 
> > > > I'm trying to profile a small section of a long-running 
> > > > program.  I ran into trouble using call-graph recording 
> > > > and I thought this might be an alternative way of getting 
> > > > what I was after.
> > > 
> > > That usecase indeed makes sense. Peter, could we allow this 
> > > for privileged tasks, depending on the perf_paranoia 
> > > settings or such?
> > 
> > I really dislike it. The sane way around this would be to 
> > allow easy self-profiling instead of doing things arse about 
> > face like that.
> 
> So, what workflow are you suggesting to Andrew?
> 
> I guess we are also hurting from the lack of dwarf stack 
> backtrace decoding - that would allow the filtering by parent 
> function without modifying the code. I think Frederic had a 
> prototype working for 32-bit - any update on that?

I haven't touched that for a while. I would be glad if
somebody else could relay me on this work.

I think Jiri Olsa has been working on something about Dwarf cfi
unwinding with a different approach. Mine was about dumping chunks
of stack and regs and do the unwinding on post processing. I think
Jiri is doing the unwinding from the event overflow fast path.

> 
> Andrew could work that problem around right now by adding:
> 
>    -fno-omit-frame-pointer
> 
> to the build of the utility - that should activate -g and 
> perf-report's --parent filter should also work fine.
> 
> Thanks,
> 
> 	Ingo