* problems with perf probe and system libraries
@ 2014-12-30 4:21 David Ahern
2014-12-30 22:18 ` Masami Hiramatsu
2014-12-31 6:27 ` [PATCH] [BUGFIX] perf-probe: Fix to fall back to find probe point in symbols Masami Hiramatsu
0 siblings, 2 replies; 6+ messages in thread
From: David Ahern @ 2014-12-30 4:21 UTC (permalink / raw)
To: Masami Hiramatsu; +Cc: Arnaldo Carvalho de Melo, LKML
Hi Masami:
I have been looking at perf-probe again and having a number of problems
with top of tree.
Here's the first one I have isolated:
$ perf probe -x /lib64/libc-2.18.so -a 'malloc size=%di'
Probe point 'malloc' not found.
Error: Failed to add events.
$ perf probe -x /lib64/libc-2.18.so -F | grep malloc
malloc
malloc@plt
malloc_atfork
malloc_check
malloc_consolidate
malloc_hook_ini
malloc_info
malloc_printerr
mallochook
ptmalloc_init
ptmalloc_init.part.8
ptmalloc_lock_all
ptmalloc_unlock_all
ptmalloc_unlock_all2
tr_mallochook
$ perf probe -x /lib64/libc-2.18.so -a 'malloc=malloc size=%di'
Probe point 'malloc' not found.
Error: Failed to add events.
A year ago (v3.12) this worked fine so I did a git bisect which points to:
commit fb7345bbf7fad9bf72ef63a19c707970b9685812
Author: Masami Hiramatsu <masami.hiramatsu.pt@hitachi.com>
Date: Thu Dec 26 05:41:53 2013 +0000
perf probe: Support basic dwarf-based operations on uprobe events
I have tried top of tree on Fedora 16, 18 and 20 with a variety of
kernels - and a variety of results. Reverting to
8a613d40e389b723fd5889ac8d4033ed4030be31 which is the commit before this
one and it works again.
David
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: problems with perf probe and system libraries
2014-12-30 4:21 problems with perf probe and system libraries David Ahern
@ 2014-12-30 22:18 ` Masami Hiramatsu
2014-12-31 6:27 ` [PATCH] [BUGFIX] perf-probe: Fix to fall back to find probe point in symbols Masami Hiramatsu
1 sibling, 0 replies; 6+ messages in thread
From: Masami Hiramatsu @ 2014-12-30 22:18 UTC (permalink / raw)
To: David Ahern; +Cc: Arnaldo Carvalho de Melo, LKML
Hi David,
(2014/12/30 13:21), David Ahern wrote:
> Hi Masami:
>
> I have been looking at perf-probe again and having a number of problems
> with top of tree.
>
> Here's the first one I have isolated:
>
> $ perf probe -x /lib64/libc-2.18.so -a 'malloc size=%di'
> Probe point 'malloc' not found.
> Error: Failed to add events.
It seems to fail when failed to open debuginfo.
I confirmed it on ubuntu 14.04 too.
I'll try to fix that.
Thank you for reporting!
>
> $ perf probe -x /lib64/libc-2.18.so -F | grep malloc
> malloc
> malloc@plt
> malloc_atfork
> malloc_check
> malloc_consolidate
> malloc_hook_ini
> malloc_info
> malloc_printerr
> mallochook
> ptmalloc_init
> ptmalloc_init.part.8
> ptmalloc_lock_all
> ptmalloc_unlock_all
> ptmalloc_unlock_all2
> tr_mallochook
>
> $ perf probe -x /lib64/libc-2.18.so -a 'malloc=malloc size=%di'
> Probe point 'malloc' not found.
> Error: Failed to add events.
>
> A year ago (v3.12) this worked fine so I did a git bisect which points to:
>
> commit fb7345bbf7fad9bf72ef63a19c707970b9685812
> Author: Masami Hiramatsu <masami.hiramatsu.pt@hitachi.com>
> Date: Thu Dec 26 05:41:53 2013 +0000
>
> perf probe: Support basic dwarf-based operations on uprobe events
>
> I have tried top of tree on Fedora 16, 18 and 20 with a variety of
> kernels - and a variety of results. Reverting to
> 8a613d40e389b723fd5889ac8d4033ed4030be31 which is the commit before this
> one and it works again.
>
> David
> --
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at http://www.tux.org/lkml/
>
--
Masami HIRAMATSU
Software Platform Research Dept. Linux Technology Research Center
Hitachi, Ltd., Yokohama Research Laboratory
E-mail: masami.hiramatsu.pt@hitachi.com
^ permalink raw reply [flat|nested] 6+ messages in thread
* [PATCH] [BUGFIX] perf-probe: Fix to fall back to find probe point in symbols
2014-12-30 4:21 problems with perf probe and system libraries David Ahern
2014-12-30 22:18 ` Masami Hiramatsu
@ 2014-12-31 6:27 ` Masami Hiramatsu
2014-12-31 8:02 ` Namhyung Kim
2015-01-08 9:52 ` [tip:perf/urgent] perf probe: " tip-bot for Masami Hiramatsu
1 sibling, 2 replies; 6+ messages in thread
From: Masami Hiramatsu @ 2014-12-31 6:27 UTC (permalink / raw)
To: Arnaldo Carvalho de Melo, David Ahern
Cc: namhyung, yrl.pp-manager.tt, Ingo Molnar, Linux Kernel Mailing List
Fix to fall back to find a probe point in symbols if perf fails
to find it in debuginfo.
This can happen when the target function is an alias of another
function. Such alias doesn't have an entry in debuginfo but in
symbols.
David Ahern reported this problem in https://lkml.org/lkml/2014/12/29/355
I ensured the problem and deeper investigation discovers it.
-----
eu-readelf --debug-dump=info /usr/lib/debug/lib/x86_64-linux-gnu/libc-2.19.so | grep \"malloc\" -A6
name (strp) "malloc"
decl_file (data1) 25
decl_line (data2) 466
prototyped (flag_present)
type (ref4) [ 81b5]
declaration (flag_present)
[ 8f58] formal_parameter
--
name (strp) "malloc"
decl_file (data1) 23
decl_line (data2) 466
prototyped (flag_present)
type (ref4) [ 9f4a]
declaration (flag_present)
sibling (ref4) [ bb29]
...
-----
All these entires have no instances (all of them are declarations)
This is why the perf probe failed to find it in debuginfo.
However, there are some malloc instances in symbols.
-----
eu-readelf --symbols /usr/lib/debug/lib/x86_64-linux-gnu/libc-2.19.so | grep malloc$
1181: 0000000000080700 5332 FUNC LOCAL DEFAULT 12 _int_malloc
4537: 00000000000831d0 339 FUNC LOCAL DEFAULT 12 __GI___libc_malloc
5545: 00000000000831d0 339 FUNC LOCAL DEFAULT 12 __malloc
6063: 00000000000831d0 339 FUNC GLOBAL DEFAULT 12 malloc
7302: 00000000000831d0 339 FUNC GLOBAL DEFAULT 12 __libc_malloc
-----
As you an see, malloc and __libc_malloc have same address, and actually
__libc_malloc has an entry in debuginfo. So you can set up a probe on
__libc_malloc.
To fix this problem shortly, perf probe simply falls back to find probe
point(malloc) in symbols if it is not found in debuginfo.
Signed-off-by: Masami Hiramatsu <masami.hiramatsu.pt@hitachi.com>
Reported-by: David Ahern <dsahern@gmail.com>
---
tools/perf/util/probe-event.c | 6 ++++--
1 file changed, 4 insertions(+), 2 deletions(-)
diff --git a/tools/perf/util/probe-event.c b/tools/perf/util/probe-event.c
index 28eb141..7f9b863 100644
--- a/tools/perf/util/probe-event.c
+++ b/tools/perf/util/probe-event.c
@@ -495,9 +495,11 @@ static int try_to_find_probe_trace_events(struct perf_probe_event *pev,
}
if (ntevs == 0) { /* No error but failed to find probe point. */
- pr_warning("Probe point '%s' not found.\n",
+ pr_warning("Probe point '%s' not found in debuginfo.\n",
synthesize_perf_probe_point(&pev->point));
- return -ENOENT;
+ if (need_dwarf)
+ return -ENOENT;
+ return 0;
}
/* Error path : ntevs < 0 */
pr_debug("An error occurred in debuginfo analysis (%d).\n", ntevs);
^ permalink raw reply related [flat|nested] 6+ messages in thread
* Re: [PATCH] [BUGFIX] perf-probe: Fix to fall back to find probe point in symbols
2014-12-31 6:27 ` [PATCH] [BUGFIX] perf-probe: Fix to fall back to find probe point in symbols Masami Hiramatsu
@ 2014-12-31 8:02 ` Namhyung Kim
2015-01-04 4:41 ` Masami Hiramatsu
2015-01-08 9:52 ` [tip:perf/urgent] perf probe: " tip-bot for Masami Hiramatsu
1 sibling, 1 reply; 6+ messages in thread
From: Namhyung Kim @ 2014-12-31 8:02 UTC (permalink / raw)
To: Masami Hiramatsu
Cc: Arnaldo Carvalho de Melo, David Ahern, yrl.pp-manager.tt,
Ingo Molnar, Linux Kernel Mailing List
Hi Masami,
On Wed, Dec 31, 2014 at 03:27:47PM +0900, Masami Hiramatsu wrote:
> Fix to fall back to find a probe point in symbols if perf fails
> to find it in debuginfo.
>
> This can happen when the target function is an alias of another
> function. Such alias doesn't have an entry in debuginfo but in
> symbols.
>
> David Ahern reported this problem in https://lkml.org/lkml/2014/12/29/355
>
> I ensured the problem and deeper investigation discovers it.
> -----
> eu-readelf --debug-dump=info /usr/lib/debug/lib/x86_64-linux-gnu/libc-2.19.so | grep \"malloc\" -A6
> name (strp) "malloc"
> decl_file (data1) 25
> decl_line (data2) 466
> prototyped (flag_present)
> type (ref4) [ 81b5]
> declaration (flag_present)
> [ 8f58] formal_parameter
> --
> name (strp) "malloc"
> decl_file (data1) 23
> decl_line (data2) 466
> prototyped (flag_present)
> type (ref4) [ 9f4a]
> declaration (flag_present)
> sibling (ref4) [ bb29]
> ...
> -----
> All these entires have no instances (all of them are declarations)
> This is why the perf probe failed to find it in debuginfo.
>
> However, there are some malloc instances in symbols.
> -----
> eu-readelf --symbols /usr/lib/debug/lib/x86_64-linux-gnu/libc-2.19.so | grep malloc$
> 1181: 0000000000080700 5332 FUNC LOCAL DEFAULT 12 _int_malloc
> 4537: 00000000000831d0 339 FUNC LOCAL DEFAULT 12 __GI___libc_malloc
> 5545: 00000000000831d0 339 FUNC LOCAL DEFAULT 12 __malloc
> 6063: 00000000000831d0 339 FUNC GLOBAL DEFAULT 12 malloc
> 7302: 00000000000831d0 339 FUNC GLOBAL DEFAULT 12 __libc_malloc
> -----
> As you an see, malloc and __libc_malloc have same address, and actually
> __libc_malloc has an entry in debuginfo. So you can set up a probe on
> __libc_malloc.
>
> To fix this problem shortly, perf probe simply falls back to find probe
> point(malloc) in symbols if it is not found in debuginfo.
Idealy, it'd be great if it can go to the definition. I think perf
only keeps a single symbol at a same address so if there're multiple
aliases at a specific address it might be missed even with this patch.
But that will be a rare case, so
Acked-by: Namhyung Kim <namhyung@kernel.org>
Thanks,
Namhyung
>
> Signed-off-by: Masami Hiramatsu <masami.hiramatsu.pt@hitachi.com>
> Reported-by: David Ahern <dsahern@gmail.com>
> ---
> tools/perf/util/probe-event.c | 6 ++++--
> 1 file changed, 4 insertions(+), 2 deletions(-)
>
> diff --git a/tools/perf/util/probe-event.c b/tools/perf/util/probe-event.c
> index 28eb141..7f9b863 100644
> --- a/tools/perf/util/probe-event.c
> +++ b/tools/perf/util/probe-event.c
> @@ -495,9 +495,11 @@ static int try_to_find_probe_trace_events(struct perf_probe_event *pev,
> }
>
> if (ntevs == 0) { /* No error but failed to find probe point. */
> - pr_warning("Probe point '%s' not found.\n",
> + pr_warning("Probe point '%s' not found in debuginfo.\n",
> synthesize_perf_probe_point(&pev->point));
> - return -ENOENT;
> + if (need_dwarf)
> + return -ENOENT;
> + return 0;
> }
> /* Error path : ntevs < 0 */
> pr_debug("An error occurred in debuginfo analysis (%d).\n", ntevs);
>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at http://www.tux.org/lkml/
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: Re: [PATCH] [BUGFIX] perf-probe: Fix to fall back to find probe point in symbols
2014-12-31 8:02 ` Namhyung Kim
@ 2015-01-04 4:41 ` Masami Hiramatsu
0 siblings, 0 replies; 6+ messages in thread
From: Masami Hiramatsu @ 2015-01-04 4:41 UTC (permalink / raw)
To: Namhyung Kim
Cc: Arnaldo Carvalho de Melo, David Ahern, yrl.pp-manager.tt,
Ingo Molnar, Linux Kernel Mailing List
(2014/12/31 17:02), Namhyung Kim wrote:
> Hi Masami,
>
> On Wed, Dec 31, 2014 at 03:27:47PM +0900, Masami Hiramatsu wrote:
>> Fix to fall back to find a probe point in symbols if perf fails
>> to find it in debuginfo.
>>
>> This can happen when the target function is an alias of another
>> function. Such alias doesn't have an entry in debuginfo but in
>> symbols.
>>
>> David Ahern reported this problem in https://lkml.org/lkml/2014/12/29/355
>>
>> I ensured the problem and deeper investigation discovers it.
>> -----
>> eu-readelf --debug-dump=info /usr/lib/debug/lib/x86_64-linux-gnu/libc-2.19.so | grep \"malloc\" -A6
>> name (strp) "malloc"
>> decl_file (data1) 25
>> decl_line (data2) 466
>> prototyped (flag_present)
>> type (ref4) [ 81b5]
>> declaration (flag_present)
>> [ 8f58] formal_parameter
>> --
>> name (strp) "malloc"
>> decl_file (data1) 23
>> decl_line (data2) 466
>> prototyped (flag_present)
>> type (ref4) [ 9f4a]
>> declaration (flag_present)
>> sibling (ref4) [ bb29]
>> ...
>> -----
>> All these entires have no instances (all of them are declarations)
>> This is why the perf probe failed to find it in debuginfo.
>>
>> However, there are some malloc instances in symbols.
>> -----
>> eu-readelf --symbols /usr/lib/debug/lib/x86_64-linux-gnu/libc-2.19.so | grep malloc$
>> 1181: 0000000000080700 5332 FUNC LOCAL DEFAULT 12 _int_malloc
>> 4537: 00000000000831d0 339 FUNC LOCAL DEFAULT 12 __GI___libc_malloc
>> 5545: 00000000000831d0 339 FUNC LOCAL DEFAULT 12 __malloc
>> 6063: 00000000000831d0 339 FUNC GLOBAL DEFAULT 12 malloc
>> 7302: 00000000000831d0 339 FUNC GLOBAL DEFAULT 12 __libc_malloc
>> -----
>> As you an see, malloc and __libc_malloc have same address, and actually
>> __libc_malloc has an entry in debuginfo. So you can set up a probe on
>> __libc_malloc.
>>
>> To fix this problem shortly, perf probe simply falls back to find probe
>> point(malloc) in symbols if it is not found in debuginfo.
>
> Idealy, it'd be great if it can go to the definition. I think perf
> only keeps a single symbol at a same address so if there're multiple
> aliases at a specific address it might be missed even with this patch.
Hmm, anyway, that should be fixed by improving perf's map, instead of
introducing new symbol finder.
> But that will be a rare case, so
>
> Acked-by: Namhyung Kim <namhyung@kernel.org>
Thank you!
>
> Thanks,
> Namhyung
>
>
>>
>> Signed-off-by: Masami Hiramatsu <masami.hiramatsu.pt@hitachi.com>
>> Reported-by: David Ahern <dsahern@gmail.com>
>> ---
>> tools/perf/util/probe-event.c | 6 ++++--
>> 1 file changed, 4 insertions(+), 2 deletions(-)
>>
>> diff --git a/tools/perf/util/probe-event.c b/tools/perf/util/probe-event.c
>> index 28eb141..7f9b863 100644
>> --- a/tools/perf/util/probe-event.c
>> +++ b/tools/perf/util/probe-event.c
>> @@ -495,9 +495,11 @@ static int try_to_find_probe_trace_events(struct perf_probe_event *pev,
>> }
>>
>> if (ntevs == 0) { /* No error but failed to find probe point. */
>> - pr_warning("Probe point '%s' not found.\n",
>> + pr_warning("Probe point '%s' not found in debuginfo.\n",
>> synthesize_perf_probe_point(&pev->point));
>> - return -ENOENT;
>> + if (need_dwarf)
>> + return -ENOENT;
>> + return 0;
>> }
>> /* Error path : ntevs < 0 */
>> pr_debug("An error occurred in debuginfo analysis (%d).\n", ntevs);
>>
>> --
>> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at http://vger.kernel.org/majordomo-info.html
>> Please read the FAQ at http://www.tux.org/lkml/
> --
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at http://www.tux.org/lkml/
>
--
Masami HIRAMATSU
Software Platform Research Dept. Linux Technology Research Center
Hitachi, Ltd., Yokohama Research Laboratory
E-mail: masami.hiramatsu.pt@hitachi.com
^ permalink raw reply [flat|nested] 6+ messages in thread
* [tip:perf/urgent] perf probe: Fix to fall back to find probe point in symbols
2014-12-31 6:27 ` [PATCH] [BUGFIX] perf-probe: Fix to fall back to find probe point in symbols Masami Hiramatsu
2014-12-31 8:02 ` Namhyung Kim
@ 2015-01-08 9:52 ` tip-bot for Masami Hiramatsu
1 sibling, 0 replies; 6+ messages in thread
From: tip-bot for Masami Hiramatsu @ 2015-01-08 9:52 UTC (permalink / raw)
To: linux-tip-commits
Cc: masami.hiramatsu.pt, hpa, acme, mingo, namhyung, dsahern, tglx,
linux-kernel
Commit-ID: 906451b98b6774042b707a5dcebb6a93dbd14c85
Gitweb: http://git.kernel.org/tip/906451b98b6774042b707a5dcebb6a93dbd14c85
Author: Masami Hiramatsu <masami.hiramatsu.pt@hitachi.com>
AuthorDate: Wed, 31 Dec 2014 15:27:47 +0900
Committer: Arnaldo Carvalho de Melo <acme@redhat.com>
CommitDate: Fri, 2 Jan 2015 12:42:17 -0300
perf probe: Fix to fall back to find probe point in symbols
Fix to fall back to find a probe point in symbols if perf fails to find
it in debuginfo.
This can happen when the target function is an alias of another
function. Such alias doesn't have an entry in debuginfo but in symbols.
David Ahern reported this problem in https://lkml.org/lkml/2014/12/29/355
I ensured the problem and deeper investigation discovers it.
-----
eu-readelf --debug-dump=info /usr/lib/debug/lib/x86_64-linux-gnu/libc-2.19.so | grep \"malloc\" -A6
name (strp) "malloc"
decl_file (data1) 25
decl_line (data2) 466
prototyped (flag_present)
type (ref4) [ 81b5]
declaration (flag_present)
[ 8f58] formal_parameter
--
name (strp) "malloc"
decl_file (data1) 23
decl_line (data2) 466
prototyped (flag_present)
type (ref4) [ 9f4a]
declaration (flag_present)
sibling (ref4) [ bb29]
...
-----
All these entires have no instances (all of them are declarations)
This is why the perf probe failed to find it in debuginfo.
However, there are some malloc instances in symbols.
-----
eu-readelf --symbols /usr/lib/debug/lib/x86_64-linux-gnu/libc-2.19.so | grep malloc$
1181: 0000000000080700 5332 FUNC LOCAL DEFAULT 12 _int_malloc
4537: 00000000000831d0 339 FUNC LOCAL DEFAULT 12 __GI___libc_malloc
5545: 00000000000831d0 339 FUNC LOCAL DEFAULT 12 __malloc
6063: 00000000000831d0 339 FUNC GLOBAL DEFAULT 12 malloc
7302: 00000000000831d0 339 FUNC GLOBAL DEFAULT 12 __libc_malloc
-----
As you an see, malloc and __libc_malloc have same address, and actually
__libc_malloc has an entry in debuginfo. So you can set up a probe on
__libc_malloc.
To fix this problem shortly, perf probe simply falls back to find probe
point(malloc) in symbols if it is not found in debuginfo.
Reported-by: David Ahern <dsahern@gmail.com>
Signed-off-by: Masami Hiramatsu <masami.hiramatsu.pt@hitachi.com>
Acked-by: Namhyung Kim <namhyung@kernel.org>
Cc: David Ahern <dsahern@gmail.com>
Cc: Ingo Molnar <mingo@kernel.org>
Cc: Namhyung Kim <namhyung@kernel.org>
Cc: yrl.pp-manager.tt@hitachi.com
Link: http://lkml.kernel.org/r/20141231062747.2087.80961.stgit@localhost.localdomain
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
---
tools/perf/util/probe-event.c | 6 ++++--
1 file changed, 4 insertions(+), 2 deletions(-)
diff --git a/tools/perf/util/probe-event.c b/tools/perf/util/probe-event.c
index 28eb141..7f9b863 100644
--- a/tools/perf/util/probe-event.c
+++ b/tools/perf/util/probe-event.c
@@ -495,9 +495,11 @@ static int try_to_find_probe_trace_events(struct perf_probe_event *pev,
}
if (ntevs == 0) { /* No error but failed to find probe point. */
- pr_warning("Probe point '%s' not found.\n",
+ pr_warning("Probe point '%s' not found in debuginfo.\n",
synthesize_perf_probe_point(&pev->point));
- return -ENOENT;
+ if (need_dwarf)
+ return -ENOENT;
+ return 0;
}
/* Error path : ntevs < 0 */
pr_debug("An error occurred in debuginfo analysis (%d).\n", ntevs);
^ permalink raw reply related [flat|nested] 6+ messages in thread
end of thread, other threads:[~2015-01-08 9:52 UTC | newest]
Thread overview: 6+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2014-12-30 4:21 problems with perf probe and system libraries David Ahern
2014-12-30 22:18 ` Masami Hiramatsu
2014-12-31 6:27 ` [PATCH] [BUGFIX] perf-probe: Fix to fall back to find probe point in symbols Masami Hiramatsu
2014-12-31 8:02 ` Namhyung Kim
2015-01-04 4:41 ` Masami Hiramatsu
2015-01-08 9:52 ` [tip:perf/urgent] perf probe: " tip-bot for Masami Hiramatsu
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).