* [RFC] perf/perf_events: misleading number of samples due to mmap()
@ 2010-06-16 14:40 Stephane Eranian
2010-06-16 14:52 ` Peter Zijlstra
2010-06-16 14:54 ` Peter Zijlstra
0 siblings, 2 replies; 6+ messages in thread
From: Stephane Eranian @ 2010-06-16 14:40 UTC (permalink / raw)
To: Arnaldo Carvalho de Melo
Cc: linux-kernel, peterz, mingo, paulus, davem, fweisbec,
perfmon2-devel, eranian
Hi,
I was using perf record to run various tests and I
realized perf output was rather misleading.
If you sample a noploop program wich runs for 10s:
$ perf record -F 1000 noploop 10
You expect a number of samples around: 10,000.
Now if you divide the rate by 4:
$ perf record -F 250 noploop 10
You expect around: 2500 samples
Well, it turns out the printed count depends on
the state of the whole system, not just noploop.
The reason is that perf reports an estimate based on the
number of bytes written to the buffer divided by the minimal
sample size of 24 bytes.
I think this is very confusing. It certainly got me.
I understand that perf does not parse the samples it gets from
the mmap'ed sampling buffer. Thus, it is not possible to get an
accurate average sample size nor actual number of samples.
What skews the estimate is the MMAP events (for the most part).
The sampling buffer records *all* mmap()s happening in the system
and this even if you are monitoring in per-thread mode. On a single-user
workstation that may be fine, but on a loaded server you get lots
of mmap events. And you don't care about most of them.
This leads me to another point. For per-thread sampling, why
do we need to record mmap() events happening *outside* of
the process? I can understand the exception of kernel modules.
Couldn't we restrict the event to those happening to the PID
the event is attached to?
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [RFC] perf/perf_events: misleading number of samples due to mmap()
2010-06-16 14:40 [RFC] perf/perf_events: misleading number of samples due to mmap() Stephane Eranian
@ 2010-06-16 14:52 ` Peter Zijlstra
2010-06-16 16:41 ` stephane eranian
2010-06-16 14:54 ` Peter Zijlstra
1 sibling, 1 reply; 6+ messages in thread
From: Peter Zijlstra @ 2010-06-16 14:52 UTC (permalink / raw)
To: Stephane Eranian
Cc: Arnaldo Carvalho de Melo, linux-kernel, mingo, paulus, davem,
fweisbec, perfmon2-devel, eranian
On Wed, 2010-06-16 at 16:40 +0200, Stephane Eranian wrote:
> This leads me to another point. For per-thread sampling, why
> do we need to record mmap() events happening *outside* of
> the process? I can understand the exception of kernel modules.
How does that happen? The per-thread events should be on the per-task
context, so another task's mmap() events should never end up there.
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [RFC] perf/perf_events: misleading number of samples due to mmap()
2010-06-16 14:40 [RFC] perf/perf_events: misleading number of samples due to mmap() Stephane Eranian
2010-06-16 14:52 ` Peter Zijlstra
@ 2010-06-16 14:54 ` Peter Zijlstra
1 sibling, 0 replies; 6+ messages in thread
From: Peter Zijlstra @ 2010-06-16 14:54 UTC (permalink / raw)
To: Stephane Eranian
Cc: Arnaldo Carvalho de Melo, linux-kernel, mingo, paulus, davem,
fweisbec, perfmon2-devel, eranian
On Wed, 2010-06-16 at 16:40 +0200, Stephane Eranian wrote:
> The reason is that perf reports an estimate based on the
> number of bytes written to the buffer divided by the minimal
> sample size of 24 bytes.
Right, we should change that based on the PERF_SAMPLE flags used. It
will remain an estimate due to the out of band events, but it should be
closer than assuming the minimal sample size.
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [RFC] perf/perf_events: misleading number of samples due to mmap()
2010-06-16 14:52 ` Peter Zijlstra
@ 2010-06-16 16:41 ` stephane eranian
2010-06-16 17:15 ` stephane eranian
2010-06-16 17:43 ` Peter Zijlstra
0 siblings, 2 replies; 6+ messages in thread
From: stephane eranian @ 2010-06-16 16:41 UTC (permalink / raw)
To: Peter Zijlstra
Cc: Stephane Eranian, Arnaldo Carvalho de Melo, linux-kernel, mingo,
paulus, davem, fweisbec, perfmon2-devel
On Wed, Jun 16, 2010 at 4:52 PM, Peter Zijlstra <peterz@infradead.org> wrote:
> On Wed, 2010-06-16 at 16:40 +0200, Stephane Eranian wrote:
>> This leads me to another point. For per-thread sampling, why
>> do we need to record mmap() events happening *outside* of
>> the process? I can understand the exception of kernel modules.
>
> How does that happen? The per-thread events should be on the per-task
> context, so another task's mmap() events should never end up there.
>
I don't see the test that says the vma does not belong to the current task.
I also don't see anything in perf_event_mmap_match().
It does seem to work as you said in recent kernels, though. So I am certainly
missing something here.
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [RFC] perf/perf_events: misleading number of samples due to mmap()
2010-06-16 16:41 ` stephane eranian
@ 2010-06-16 17:15 ` stephane eranian
2010-06-16 17:43 ` Peter Zijlstra
1 sibling, 0 replies; 6+ messages in thread
From: stephane eranian @ 2010-06-16 17:15 UTC (permalink / raw)
To: Peter Zijlstra
Cc: Stephane Eranian, Arnaldo Carvalho de Melo, linux-kernel, mingo,
paulus, davem, fweisbec, perfmon2-devel
On Wed, Jun 16, 2010 at 6:41 PM, stephane eranian
<eranian@googlemail.com> wrote:
> On Wed, Jun 16, 2010 at 4:52 PM, Peter Zijlstra <peterz@infradead.org> wrote:
>> On Wed, 2010-06-16 at 16:40 +0200, Stephane Eranian wrote:
>>> This leads me to another point. For per-thread sampling, why
>>> do we need to record mmap() events happening *outside* of
>>> the process? I can understand the exception of kernel modules.
>>
>> How does that happen? The per-thread events should be on the per-task
>> context, so another task's mmap() events should never end up there.
>>
>
> I don't see the test that says the vma does not belong to the current task.
> I also don't see anything in perf_event_mmap_match().
>
> It does seem to work as you said in recent kernels, though. So I am certainly
> missing something here.
>
I think I got it now. It's all based on the current->perf_event_ctxp.
So you only
look at the events (and thus sampling buffers) attached to current which issued
the mmap().
I found out that my problem is coming from perf creating synthetic
mmap() entries
for everything it finds in /proc/PID.
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [RFC] perf/perf_events: misleading number of samples due to mmap()
2010-06-16 16:41 ` stephane eranian
2010-06-16 17:15 ` stephane eranian
@ 2010-06-16 17:43 ` Peter Zijlstra
1 sibling, 0 replies; 6+ messages in thread
From: Peter Zijlstra @ 2010-06-16 17:43 UTC (permalink / raw)
To: eranian
Cc: Stephane Eranian, Arnaldo Carvalho de Melo, linux-kernel, mingo,
paulus, davem, fweisbec, perfmon2-devel
On Wed, 2010-06-16 at 18:41 +0200, stephane eranian wrote:
> On Wed, Jun 16, 2010 at 4:52 PM, Peter Zijlstra <peterz@infradead.org> wrote:
> > On Wed, 2010-06-16 at 16:40 +0200, Stephane Eranian wrote:
> >> This leads me to another point. For per-thread sampling, why
> >> do we need to record mmap() events happening *outside* of
> >> the process? I can understand the exception of kernel modules.
> >
> > How does that happen? The per-thread events should be on the per-task
> > context, so another task's mmap() events should never end up there.
> >
>
> I don't see the test that says the vma does not belong to the current task.
> I also don't see anything in perf_event_mmap_match().
>
> It does seem to work as you said in recent kernels, though. So I am certainly
> missing something here.
vma's are always part of the current task, its impossible to call mmap()
on another process's address space.
Look at the tail of perf_event_mmap_event(), it does:
rcu_read_lock();
cpuctx = &get_cpu_var(perf_cpu_context);
perf_event_mmap_ctx(&cpuctx->ctx, mmap_event, vma->vm_flags & VM_EXEC);
ctx = rcu_dereference(current->perf_event_ctxp);
if (ctx)
perf_event_mmap_ctx(ctx, mmap_event, vma->vm_flags & VM_EXEC);
put_cpu_var(perf_cpu_context);
rcu_read_unlock();
There it traverses the per-cpu context and the per-task context.
^ permalink raw reply [flat|nested] 6+ messages in thread
end of thread, other threads:[~2010-06-16 17:43 UTC | newest]
Thread overview: 6+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2010-06-16 14:40 [RFC] perf/perf_events: misleading number of samples due to mmap() Stephane Eranian
2010-06-16 14:52 ` Peter Zijlstra
2010-06-16 16:41 ` stephane eranian
2010-06-16 17:15 ` stephane eranian
2010-06-16 17:43 ` Peter Zijlstra
2010-06-16 14:54 ` Peter Zijlstra
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.