* zeros in frame pointer callchains
@ 2016-04-12 13:38 Arnaldo Carvalho de Melo
2016-04-13 1:06 ` Namhyung Kim
0 siblings, 1 reply; 2+ messages in thread
From: Arnaldo Carvalho de Melo @ 2016-04-12 13:38 UTC (permalink / raw)
To: Namhyung Kim
Cc: Jiri Olsa, David Ahern, Peter Zijlstra, Milian Wolff, Wang Nan,
Linux Kernel Mailing List
Hi,
Namhyung, while working on supporting callchains in 'perf trace' I
noticed something I think we discussed at some point, zeros in frame pointer
callchains...
Oh well figured it out, its probably userspace not having frame
pointers, grrr anyway, keeping the rest of the message, for reference,
but now I think we need to have some expressiveness to say that for that
particular event we want to use perf_event_attr.exclude_callchain_user...
First add a special probe point that 'perf trace' uses, if present:
# perf probe 'vfs_getname=getname_flags:72 pathname=filename:string'
Then lets ask to see just when this thing hits, no syscall
printing whatsoever ('trace' == 'perf trace'), sometimes just a few
zeros at the top, sometimes a long series, need to investigate how to
chop those off in the kernel, not to use the ring buffer needlessly,
will start chopping these in userspace (in perf trace):
# trace --no-sys --ev probe:vfs_getname/call-graph=fp/
34033.533 probe:vfs_getname:(ffffffff8124c06c) pathname="/usr/share/pixmaps")
getname_flags+0xfe20009d ([kernel.kallsyms])
user_path_at_empty+0xfe200023 ([kernel.kallsyms])
vfs_fstatat+0xfe200066 ([kernel.kallsyms])
SYSC_newstat+0xfe20002e ([kernel.kallsyms])
sys_newstat+0xfe20000e ([kernel.kallsyms])
entry_SYSCALL_64_fastpath+0xfe20001a ([kernel.kallsyms])
__xstat64+0xffff011e90b98015 (/usr/lib64/libc-2.22.so)
[0] ([unknown])
[0] ([unknown])
[0] ([unknown])
[0] ([unknown])
36001.681 probe:vfs_getname:(ffffffff8124c06c) pathname="/var/tmp")
getname_flags+0xfe20009d ([kernel.kallsyms])
user_path_at_empty+0xfe200023 ([kernel.kallsyms])
vfs_fstatat+0xfe200066 ([kernel.kallsyms])
SYSC_newstat+0xfe20002e ([kernel.kallsyms])
sys_newstat+0xfe20000e ([kernel.kallsyms])
entry_SYSCALL_64_fastpath+0xfe20001a ([kernel.kallsyms])
__xstat64+0xffff00b816a28015 (/usr/lib64/libc-2.22.so)
[0] ([unknown])
[0] ([unknown])
[0] ([unknown])
[0] ([unknown])
[0] ([unknown])
[0] ([unknown])
[0] ([unknown])
[0] ([unknown])
[0] ([unknown])
[0] ([unknown])
[0] ([unknown])
[0] ([unknown])
[0] ([unknown])
[0] ([unknown])
[0] ([unknown])
[0] ([unknown])
[0] ([unknown])
[0] ([unknown])
[0] ([unknown])
[0] ([unknown])
[0] ([unknown])
[0] ([unknown])
[0] ([unknown])
[0] ([unknown])
[0] ([unknown])
[0] ([unknown])
[0] ([unknown])
[0] ([unknown])
[0] ([unknown])
[0] ([unknown])
[0] ([unknown])
[0] ([unknown])
[0] ([unknown])
[0] ([unknown])
[0] ([unknown])
[0] ([unknown])
[0] ([unknown])
[0] ([unknown])
[0] ([unknown])
[0] ([unknown])
[0] ([unknown])
[0] ([unknown])
[0] ([unknown])
[0] ([unknown])
[0] ([unknown])
[0] ([unknown])
[0] ([unknown])
[0] ([unknown])
[0] ([unknown])
[0] ([unknown])
[0] ([unknown])
[0] ([unknown])
[0] ([unknown])
[0] ([unknown])
[0] ([unknown])
[0] ([unknown])
[0] ([unknown])
[0] ([unknown])
[0] ([unknown])
[0] ([unknown])
[0] ([unknown])
[0] ([unknown])
[0] ([unknown])
[0] ([unknown])
[0] ([unknown])
[0] ([unknown])
[0] ([unknown])
[0] ([unknown])
[0] ([unknown])
[0] ([unknown])
[0] ([unknown])
[0] ([unknown])
[0] ([unknown])
[0] ([unknown])
[0] ([unknown])
[0] ([unknown])
[0] ([unknown])
[0] ([unknown])
[0] ([unknown])
[0] ([unknown])
[0] ([unknown])
[0] ([unknown])
[0] ([unknown])
[0] ([unknown])
[0] ([unknown])
[0] ([unknown])
[0] ([unknown])
[0] ([unknown])
[0] ([unknown])
[0] ([unknown])
[0] ([unknown])
[0] ([unknown])
[0] ([unknown])
[0] ([unknown])
[0] ([unknown])
[0] ([unknown])
[0] ([unknown])
[0] ([unknown])
[0] ([unknown])
[0] ([unknown])
[0] ([unknown])
[0] ([unknown])
[0] ([unknown])
[0] ([unknown])
[0] ([unknown])
[0] ([unknown])
[0] ([unknown])
[0] ([unknown])
[0] ([unknown])
[0] ([unknown])
[0] ([unknown])
[0] ([unknown])
[0] ([unknown])
[0] ([unknown])
[0] ([unknown])
[0] ([unknown])
[0] ([unknown])
[0] ([unknown])
36001.696 probe:vfs_getname:(ffffffff8124c06c) pathname="/var/tmp")
getname_flags+0xfe20009d ([kernel.kallsyms])
user_path_at_empty+0xfe200023 ([kernel.kallsyms])
sys_access+0xfe2000b4 ([kernel.kallsyms])
entry_SYSCALL_64_fastpath+0xfe20001a ([kernel.kallsyms])
__GI___access+0xffff00b816a28007 (/usr/lib64/libc-2.22.so)
[0] ([unknown])
[0] ([unknown])
[0] ([unknown])
[0] ([unknown])
[0] ([unknown])
[0] ([unknown])
[0] ([unknown])
[0] ([unknown])
[0] ([unknown])
[0] ([unknown])
[0] ([unknown])
[0] ([unknown])
[0] ([unknown])
[0] ([unknown])
[0] ([unknown])
[0] ([unknown])
[0] ([unknown])
[0] ([unknown])
[0] ([unknown])
[0] ([unknown])
[0] ([unknown])
[0] ([unknown])
[0] ([unknown])
[0] ([unknown])
[0] ([unknown])
[0] ([unknown])
[0] ([unknown])
[0] ([unknown])
[0] ([unknown])
[0] ([unknown])
[0] ([unknown])
[0] ([unknown])
[0] ([unknown])
[0] ([unknown])
[0] ([unknown])
[0] ([unknown])
[0] ([unknown])
[0] ([unknown])
[0] ([unknown])
[0] ([unknown])
[0] ([unknown])
[0] ([unknown])
[0] ([unknown])
[0] ([unknown])
[0] ([unknown])
[0] ([unknown])
[0] ([unknown])
[0] ([unknown])
[0] ([unknown])
[0] ([unknown])
[0] ([unknown])
[0] ([unknown])
[0] ([unknown])
[0] ([unknown])
[0] ([unknown])
[0] ([unknown])
[0] ([unknown])
[0] ([unknown])
[0] ([unknown])
[0] ([unknown])
[0] ([unknown])
[0] ([unknown])
[0] ([unknown])
[0] ([unknown])
[0] ([unknown])
[0] ([unknown])
[0] ([unknown])
[0] ([unknown])
[0] ([unknown])
[0] ([unknown])
[0] ([unknown])
[0] ([unknown])
[0] ([unknown])
[0] ([unknown])
[0] ([unknown])
[0] ([unknown])
[0] ([unknown])
[0] ([unknown])
[0] ([unknown])
[0] ([unknown])
[0] ([unknown])
[0] ([unknown])
[0] ([unknown])
[0] ([unknown])
[0] ([unknown])
[0] ([unknown])
[0] ([unknown])
[0] ([unknown])
[0] ([unknown])
[0] ([unknown])
[0] ([unknown])
[0] ([unknown])
[0] ([unknown])
[0] ([unknown])
[0] ([unknown])
[0] ([unknown])
[0] ([unknown])
[0] ([unknown])
[0] ([unknown])
[0] ([unknown])
[0] ([unknown])
[0] ([unknown])
[0] ([unknown])
[0] ([unknown])
[0] ([unknown])
[0] ([unknown])
[0] ([unknown])
[0] ([unknown])
[0] ([unknown])
[0] ([unknown])
[0] ([unknown])
[0] ([unknown])
[0] ([unknown])
[0] ([unknown])
[0] ([unknown])
[0] ([unknown])
[0] ([unknown])
[0] ([unknown])
[0] ([unknown])
[0] ([unknown])
8254.597 probe:vfs_getname:(ffffffff8124c06c) pathname="/etc/localtime")
getname_flags+0xfe20009d ([kernel.kallsyms])
user_path_at_empty+0xfe200023 ([kernel.kallsyms])
vfs_fstatat+0xfe200066 ([kernel.kallsyms])
SYSC_newstat+0xfe20002e ([kernel.kallsyms])
sys_newstat+0xfe20000e ([kernel.kallsyms])
entry_SYSCALL_64_fastpath+0xfe20001a ([kernel.kallsyms])
__xstat64+0xffff0104f5696015 (/usr/lib64/libc-2.22.so)
[0] ([unknown])
[0] ([unknown])
9001.567 probe:vfs_getname:(ffffffff8124c06c) pathname="/etc/firewalld/ipsets")
getname_flags+0xfe20009d ([kernel.kallsyms])
user_path_at_empty+0xfe200023 ([kernel.kallsyms])
sys_inotify_add_watch+0xfe200098 ([kernel.kallsyms])
entry_SYSCALL_64_fastpath+0xfe20001a ([kernel.kallsyms])
inotify_add_watch+0xffff008bb9432007 (/usr/lib64/libc-2.22.so)
9450.181 probe:vfs_getname:(ffffffff8124c06c) pathname="/root")
getname_flags+0xfe20009d ([kernel.kallsyms])
user_path_at_empty+0xfe200023 ([kernel.kallsyms])
sys_inotify_add_watch+0xfe200098 ([kernel.kallsyms])
entry_SYSCALL_64_fastpath+0xfe20001a ([kernel.kallsyms])
inotify_add_watch+0xffff01b7aa4f2007 (/usr/lib64/libc-2.22.so)
^ permalink raw reply [flat|nested] 2+ messages in thread
* Re: zeros in frame pointer callchains
2016-04-12 13:38 zeros in frame pointer callchains Arnaldo Carvalho de Melo
@ 2016-04-13 1:06 ` Namhyung Kim
0 siblings, 0 replies; 2+ messages in thread
From: Namhyung Kim @ 2016-04-13 1:06 UTC (permalink / raw)
To: Arnaldo Carvalho de Melo
Cc: Jiri Olsa, David Ahern, Peter Zijlstra, Milian Wolff, Wang Nan,
Linux Kernel Mailing List
Hi Arnaldo,
On Tue, Apr 12, 2016 at 10:38:47AM -0300, Arnaldo Carvalho de Melo wrote:
> Hi,
>
> Namhyung, while working on supporting callchains in 'perf trace' I
> noticed something I think we discussed at some point, zeros in frame pointer
> callchains...
>
> Oh well figured it out, its probably userspace not having frame
> pointers, grrr anyway, keeping the rest of the message, for reference,
> but now I think we need to have some expressiveness to say that for that
> particular event we want to use perf_event_attr.exclude_callchain_user...
>
> First add a special probe point that 'perf trace' uses, if present:
>
> # perf probe 'vfs_getname=getname_flags:72 pathname=filename:string'
>
> Then lets ask to see just when this thing hits, no syscall
> printing whatsoever ('trace' == 'perf trace'), sometimes just a few
> zeros at the top, sometimes a long series, need to investigate how to
> chop those off in the kernel, not to use the ring buffer needlessly,
> will start chopping these in userspace (in perf trace):
My solution was to look at /proc/sys/vm/mmap_min_addr [1]. For kernel
it'd be simply discarding callchains after seeing an address below the
mmap_min_addr. For userspace, it requires to carry the mmap_min_addr
via feature bit or something. Or Maybe we can just discard addresses
under 4096 for simplicity?
[1] https://lkml.org/lkml/2015/11/26/40
Thanks,
Namhyung
>
> # trace --no-sys --ev probe:vfs_getname/call-graph=fp/
>
> 34033.533 probe:vfs_getname:(ffffffff8124c06c) pathname="/usr/share/pixmaps")
> getname_flags+0xfe20009d ([kernel.kallsyms])
> user_path_at_empty+0xfe200023 ([kernel.kallsyms])
> vfs_fstatat+0xfe200066 ([kernel.kallsyms])
> SYSC_newstat+0xfe20002e ([kernel.kallsyms])
> sys_newstat+0xfe20000e ([kernel.kallsyms])
> entry_SYSCALL_64_fastpath+0xfe20001a ([kernel.kallsyms])
> __xstat64+0xffff011e90b98015 (/usr/lib64/libc-2.22.so)
> [0] ([unknown])
> [0] ([unknown])
> [0] ([unknown])
> [0] ([unknown])
> 36001.681 probe:vfs_getname:(ffffffff8124c06c) pathname="/var/tmp")
> getname_flags+0xfe20009d ([kernel.kallsyms])
> user_path_at_empty+0xfe200023 ([kernel.kallsyms])
> vfs_fstatat+0xfe200066 ([kernel.kallsyms])
> SYSC_newstat+0xfe20002e ([kernel.kallsyms])
> sys_newstat+0xfe20000e ([kernel.kallsyms])
> entry_SYSCALL_64_fastpath+0xfe20001a ([kernel.kallsyms])
> __xstat64+0xffff00b816a28015 (/usr/lib64/libc-2.22.so)
> [0] ([unknown])
> [0] ([unknown])
> [0] ([unknown])
> [0] ([unknown])
> [0] ([unknown])
> [0] ([unknown])
> [0] ([unknown])
> [0] ([unknown])
> [0] ([unknown])
> [0] ([unknown])
> [0] ([unknown])
> [0] ([unknown])
> [0] ([unknown])
> [0] ([unknown])
> [0] ([unknown])
> [0] ([unknown])
> [0] ([unknown])
> [0] ([unknown])
> [0] ([unknown])
> [0] ([unknown])
> [0] ([unknown])
> [0] ([unknown])
> [0] ([unknown])
> [0] ([unknown])
> [0] ([unknown])
> [0] ([unknown])
> [0] ([unknown])
> [0] ([unknown])
> [0] ([unknown])
> [0] ([unknown])
> [0] ([unknown])
> [0] ([unknown])
> [0] ([unknown])
> [0] ([unknown])
> [0] ([unknown])
> [0] ([unknown])
> [0] ([unknown])
> [0] ([unknown])
> [0] ([unknown])
> [0] ([unknown])
> [0] ([unknown])
> [0] ([unknown])
> [0] ([unknown])
> [0] ([unknown])
> [0] ([unknown])
> [0] ([unknown])
> [0] ([unknown])
> [0] ([unknown])
> [0] ([unknown])
> [0] ([unknown])
> [0] ([unknown])
> [0] ([unknown])
> [0] ([unknown])
> [0] ([unknown])
> [0] ([unknown])
> [0] ([unknown])
> [0] ([unknown])
> [0] ([unknown])
> [0] ([unknown])
> [0] ([unknown])
> [0] ([unknown])
> [0] ([unknown])
> [0] ([unknown])
> [0] ([unknown])
> [0] ([unknown])
> [0] ([unknown])
> [0] ([unknown])
> [0] ([unknown])
> [0] ([unknown])
> [0] ([unknown])
> [0] ([unknown])
> [0] ([unknown])
> [0] ([unknown])
> [0] ([unknown])
> [0] ([unknown])
> [0] ([unknown])
> [0] ([unknown])
> [0] ([unknown])
> [0] ([unknown])
> [0] ([unknown])
> [0] ([unknown])
> [0] ([unknown])
> [0] ([unknown])
> [0] ([unknown])
> [0] ([unknown])
> [0] ([unknown])
> [0] ([unknown])
> [0] ([unknown])
> [0] ([unknown])
> [0] ([unknown])
> [0] ([unknown])
> [0] ([unknown])
> [0] ([unknown])
> [0] ([unknown])
> [0] ([unknown])
> [0] ([unknown])
> [0] ([unknown])
> [0] ([unknown])
> [0] ([unknown])
> [0] ([unknown])
> [0] ([unknown])
> [0] ([unknown])
> [0] ([unknown])
> [0] ([unknown])
> [0] ([unknown])
> [0] ([unknown])
> [0] ([unknown])
> [0] ([unknown])
> [0] ([unknown])
> [0] ([unknown])
> [0] ([unknown])
> [0] ([unknown])
> [0] ([unknown])
> [0] ([unknown])
> [0] ([unknown])
> [0] ([unknown])
> [0] ([unknown])
> [0] ([unknown])
> 36001.696 probe:vfs_getname:(ffffffff8124c06c) pathname="/var/tmp")
> getname_flags+0xfe20009d ([kernel.kallsyms])
> user_path_at_empty+0xfe200023 ([kernel.kallsyms])
> sys_access+0xfe2000b4 ([kernel.kallsyms])
> entry_SYSCALL_64_fastpath+0xfe20001a ([kernel.kallsyms])
> __GI___access+0xffff00b816a28007 (/usr/lib64/libc-2.22.so)
> [0] ([unknown])
> [0] ([unknown])
> [0] ([unknown])
> [0] ([unknown])
> [0] ([unknown])
> [0] ([unknown])
> [0] ([unknown])
> [0] ([unknown])
> [0] ([unknown])
> [0] ([unknown])
> [0] ([unknown])
> [0] ([unknown])
> [0] ([unknown])
> [0] ([unknown])
> [0] ([unknown])
> [0] ([unknown])
> [0] ([unknown])
> [0] ([unknown])
> [0] ([unknown])
> [0] ([unknown])
> [0] ([unknown])
> [0] ([unknown])
> [0] ([unknown])
> [0] ([unknown])
> [0] ([unknown])
> [0] ([unknown])
> [0] ([unknown])
> [0] ([unknown])
> [0] ([unknown])
> [0] ([unknown])
> [0] ([unknown])
> [0] ([unknown])
> [0] ([unknown])
> [0] ([unknown])
> [0] ([unknown])
> [0] ([unknown])
> [0] ([unknown])
> [0] ([unknown])
> [0] ([unknown])
> [0] ([unknown])
> [0] ([unknown])
> [0] ([unknown])
> [0] ([unknown])
> [0] ([unknown])
> [0] ([unknown])
> [0] ([unknown])
> [0] ([unknown])
> [0] ([unknown])
> [0] ([unknown])
> [0] ([unknown])
> [0] ([unknown])
> [0] ([unknown])
> [0] ([unknown])
> [0] ([unknown])
> [0] ([unknown])
> [0] ([unknown])
> [0] ([unknown])
> [0] ([unknown])
> [0] ([unknown])
> [0] ([unknown])
> [0] ([unknown])
> [0] ([unknown])
> [0] ([unknown])
> [0] ([unknown])
> [0] ([unknown])
> [0] ([unknown])
> [0] ([unknown])
> [0] ([unknown])
> [0] ([unknown])
> [0] ([unknown])
> [0] ([unknown])
> [0] ([unknown])
> [0] ([unknown])
> [0] ([unknown])
> [0] ([unknown])
> [0] ([unknown])
> [0] ([unknown])
> [0] ([unknown])
> [0] ([unknown])
> [0] ([unknown])
> [0] ([unknown])
> [0] ([unknown])
> [0] ([unknown])
> [0] ([unknown])
> [0] ([unknown])
> [0] ([unknown])
> [0] ([unknown])
> [0] ([unknown])
> [0] ([unknown])
> [0] ([unknown])
> [0] ([unknown])
> [0] ([unknown])
> [0] ([unknown])
> [0] ([unknown])
> [0] ([unknown])
> [0] ([unknown])
> [0] ([unknown])
> [0] ([unknown])
> [0] ([unknown])
> [0] ([unknown])
> [0] ([unknown])
> [0] ([unknown])
> [0] ([unknown])
> [0] ([unknown])
> [0] ([unknown])
> [0] ([unknown])
> [0] ([unknown])
> [0] ([unknown])
> [0] ([unknown])
> [0] ([unknown])
> [0] ([unknown])
> [0] ([unknown])
> [0] ([unknown])
> [0] ([unknown])
> [0] ([unknown])
> [0] ([unknown])
> [0] ([unknown])
> [0] ([unknown])
> [0] ([unknown])
> [0] ([unknown])
> 8254.597 probe:vfs_getname:(ffffffff8124c06c) pathname="/etc/localtime")
> getname_flags+0xfe20009d ([kernel.kallsyms])
> user_path_at_empty+0xfe200023 ([kernel.kallsyms])
> vfs_fstatat+0xfe200066 ([kernel.kallsyms])
> SYSC_newstat+0xfe20002e ([kernel.kallsyms])
> sys_newstat+0xfe20000e ([kernel.kallsyms])
> entry_SYSCALL_64_fastpath+0xfe20001a ([kernel.kallsyms])
> __xstat64+0xffff0104f5696015 (/usr/lib64/libc-2.22.so)
> [0] ([unknown])
> [0] ([unknown])
> 9001.567 probe:vfs_getname:(ffffffff8124c06c) pathname="/etc/firewalld/ipsets")
> getname_flags+0xfe20009d ([kernel.kallsyms])
> user_path_at_empty+0xfe200023 ([kernel.kallsyms])
> sys_inotify_add_watch+0xfe200098 ([kernel.kallsyms])
> entry_SYSCALL_64_fastpath+0xfe20001a ([kernel.kallsyms])
> inotify_add_watch+0xffff008bb9432007 (/usr/lib64/libc-2.22.so)
> 9450.181 probe:vfs_getname:(ffffffff8124c06c) pathname="/root")
> getname_flags+0xfe20009d ([kernel.kallsyms])
> user_path_at_empty+0xfe200023 ([kernel.kallsyms])
> sys_inotify_add_watch+0xfe200098 ([kernel.kallsyms])
> entry_SYSCALL_64_fastpath+0xfe20001a ([kernel.kallsyms])
> inotify_add_watch+0xffff01b7aa4f2007 (/usr/lib64/libc-2.22.so)
^ permalink raw reply [flat|nested] 2+ messages in thread
end of thread, other threads:[~2016-04-13 1:07 UTC | newest]
Thread overview: 2+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2016-04-12 13:38 zeros in frame pointer callchains Arnaldo Carvalho de Melo
2016-04-13 1:06 ` Namhyung Kim
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.