* [PATCH perf/core 0/4] perf-probe: improve glibc support @ 2015-03-02 12:49 Masami Hiramatsu 2015-03-02 12:49 ` [PATCH perf/core 1/4] [RESEND][BUGFIX] perf-probe: Remove bias offset to find probe point by address Masami Hiramatsu ` (3 more replies) 0 siblings, 4 replies; 14+ messages in thread From: Masami Hiramatsu @ 2015-03-02 12:49 UTC (permalink / raw) To: Arnaldo Carvalho de Melo Cc: Naohiro Aota, Peter Zijlstra, Linux Kernel Mailing List, David Ahern, namhyung, Jiri Olsa, Ingo Molnar Hi, Here is a series of patches which improves perf-probe to handle glibc's aliased symbols more correctly. Longly, a major known issue of probing on glibc is that the some aliased symbols(e.g. malloc) can not find by perf-probe. Actually, glibc's malloc symbol is just an alias of __libc_malloc. Its debuginfo knows only __libc_malloc, and perf's symbol map knows only malloc. This difference always confuses users that they can see malloc by perf report or annotate, but they can not probe on it, nor find definitions by --line option. Previously, I've made a commit 906451b98b67 which solved this problem partly, but not completely fixed. So I decided to solve this issue completely by finding the symbols like malloc from perf's symbol map, and converting the symbol's address into debuginfo's location infomation. With this series, you can use --vars, --line and --add with the aliased symbols on glibc. Note that the 1/4 patch is just a resend patch which I sent last weekend (since this series depends on that) http://www.gossamer-threads.com/lists/linux/kernel/2115623 2/4 and 3/4 introduces above logic to solve this issue. And 4/4 reverts previous incomplete fix. Thank you, --- Masami Hiramatsu (4): [RESEND][BUGFIX] perf-probe: Remove bias offset to find probe point by address perf-probe: Fix to handle aliased symbols in glibc perf-probe: Fix --line to handle aliased symbols in glibc Revert "perf probe: Fix to fall back to find probe point in symbols" tools/perf/util/probe-event.c | 181 +++++++++++++++++++++++++++++++++++----- tools/perf/util/probe-finder.c | 5 - 2 files changed, 160 insertions(+), 26 deletions(-) -- Masami HIRAMATSU Software Platform Research Dpt. Linux Technology Center Hitachi, Ltd., Yokohama Research Laboratory E-mail: masami.hiramatsu.pt@hitachi.com ^ permalink raw reply [flat|nested] 14+ messages in thread
* [PATCH perf/core 1/4] [RESEND][BUGFIX] perf-probe: Remove bias offset to find probe point by address 2015-03-02 12:49 [PATCH perf/core 0/4] perf-probe: improve glibc support Masami Hiramatsu @ 2015-03-02 12:49 ` Masami Hiramatsu 2015-03-03 6:26 ` [tip:perf/core] perf probe: " tip-bot for Masami Hiramatsu 2015-03-02 12:49 ` [PATCH perf/core 2/4] perf-probe: Fix to handle aliased symbols in glibc Masami Hiramatsu ` (2 subsequent siblings) 3 siblings, 1 reply; 14+ messages in thread From: Masami Hiramatsu @ 2015-03-02 12:49 UTC (permalink / raw) To: Arnaldo Carvalho de Melo Cc: Naohiro Aota, Peter Zijlstra, Linux Kernel Mailing List, David Ahern, namhyung, Jiri Olsa, Ingo Molnar Remove bias offset to find probe point by address. Without this patch, probe points on kernel and executables are shown correctly, but not work with libraries. ----- # ./perf probe -l probe:do_fork (on do_fork@kernel/fork.c) probe_libc:malloc (on malloc in /usr/lib64/libc-2.17.so) probe_perf:strlist__new (on strlist__new@util/strlist.c in /home/mhiramat/ksrc/linux-3/tools/perf/perf) ----- Removing bias allows to show it as real place. ----- # ./perf probe -l probe:do_fork (on do_fork@kernel/fork.c) probe_libc:malloc (on __libc_malloc@malloc/malloc.c in /usr/lib64/libc-2.17.so) probe_perf:strlist__new (on strlist__new@util/strlist.c in /home/mhiramat/ksrc/linux-3/tools/perf/perf) ----- Signed-off-by: Masami Hiramatsu <masami.hiramatsu.pt@hitachi.com> --- tools/perf/util/probe-finder.c | 5 +---- 1 file changed, 1 insertion(+), 4 deletions(-) diff --git a/tools/perf/util/probe-finder.c b/tools/perf/util/probe-finder.c index d141935..46f009a 100644 --- a/tools/perf/util/probe-finder.c +++ b/tools/perf/util/probe-finder.c @@ -1345,11 +1345,8 @@ int debuginfo__find_probe_point(struct debuginfo *dbg, unsigned long addr, const char *fname = NULL, *func = NULL, *basefunc = NULL, *tmp; int baseline = 0, lineno = 0, ret = 0; - /* Adjust address with bias */ - addr += dbg->bias; - /* Find cu die */ - if (!dwarf_addrdie(dbg->dbg, (Dwarf_Addr)addr - dbg->bias, &cudie)) { + if (!dwarf_addrdie(dbg->dbg, (Dwarf_Addr)addr, &cudie)) { pr_warning("Failed to find debug information for address %lx\n", addr); ret = -EINVAL; ^ permalink raw reply related [flat|nested] 14+ messages in thread
* [tip:perf/core] perf probe: Remove bias offset to find probe point by address 2015-03-02 12:49 ` [PATCH perf/core 1/4] [RESEND][BUGFIX] perf-probe: Remove bias offset to find probe point by address Masami Hiramatsu @ 2015-03-03 6:26 ` tip-bot for Masami Hiramatsu 0 siblings, 0 replies; 14+ messages in thread From: tip-bot for Masami Hiramatsu @ 2015-03-03 6:26 UTC (permalink / raw) To: linux-tip-commits Cc: masami.hiramatsu.pt, tglx, peterz, mingo, dsahern, linux-kernel, hpa, jolsa, naota, acme, namhyung Commit-ID: 0104fe69e0287cf3635657b4c6b26a18e0091697 Gitweb: http://git.kernel.org/tip/0104fe69e0287cf3635657b4c6b26a18e0091697 Author: Masami Hiramatsu <masami.hiramatsu.pt@hitachi.com> AuthorDate: Mon, 2 Mar 2015 21:49:46 +0900 Committer: Arnaldo Carvalho de Melo <acme@redhat.com> CommitDate: Mon, 2 Mar 2015 12:34:38 -0300 perf probe: Remove bias offset to find probe point by address Remove bias offset to find probe point by address. Without this patch, probe points on kernel and executables are shown correctly, but do not work with libraries: # ./perf probe -l probe:do_fork (on do_fork@kernel/fork.c) probe_libc:malloc (on malloc in /usr/lib64/libc-2.17.so) probe_perf:strlist__new (on strlist__new@util/strlist.c in /home/mhiramat/ksrc/linux-3/tools/perf/perf) Removing bias allows it to show it as real place: # ./perf probe -l probe:do_fork (on do_fork@kernel/fork.c) probe_libc:malloc (on __libc_malloc@malloc/malloc.c in /usr/lib64/libc-2.17.so) probe_perf:strlist__new (on strlist__new@util/strlist.c in /home/mhiramat/ksrc/linux-3/tools/perf/perf) Signed-off-by: Masami Hiramatsu <masami.hiramatsu.pt@hitachi.com> Cc: David Ahern <dsahern@gmail.com> Cc: Jiri Olsa <jolsa@redhat.com> Cc: Namhyung Kim <namhyung@kernel.org> Cc: Naohiro Aota <naota@elisp.net> Cc: Peter Zijlstra <peterz@infradead.org> Link: http://lkml.kernel.org/r/20150302124946.9191.64085.stgit@localhost.localdomain Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com> --- tools/perf/util/probe-finder.c | 5 +---- 1 file changed, 1 insertion(+), 4 deletions(-) diff --git a/tools/perf/util/probe-finder.c b/tools/perf/util/probe-finder.c index d141935..46f009a 100644 --- a/tools/perf/util/probe-finder.c +++ b/tools/perf/util/probe-finder.c @@ -1345,11 +1345,8 @@ int debuginfo__find_probe_point(struct debuginfo *dbg, unsigned long addr, const char *fname = NULL, *func = NULL, *basefunc = NULL, *tmp; int baseline = 0, lineno = 0, ret = 0; - /* Adjust address with bias */ - addr += dbg->bias; - /* Find cu die */ - if (!dwarf_addrdie(dbg->dbg, (Dwarf_Addr)addr - dbg->bias, &cudie)) { + if (!dwarf_addrdie(dbg->dbg, (Dwarf_Addr)addr, &cudie)) { pr_warning("Failed to find debug information for address %lx\n", addr); ret = -EINVAL; ^ permalink raw reply related [flat|nested] 14+ messages in thread
* [PATCH perf/core 2/4] perf-probe: Fix to handle aliased symbols in glibc 2015-03-02 12:49 [PATCH perf/core 0/4] perf-probe: improve glibc support Masami Hiramatsu 2015-03-02 12:49 ` [PATCH perf/core 1/4] [RESEND][BUGFIX] perf-probe: Remove bias offset to find probe point by address Masami Hiramatsu @ 2015-03-02 12:49 ` Masami Hiramatsu 2015-03-02 15:46 ` Arnaldo Carvalho de Melo 2015-03-03 13:31 ` Namhyung Kim 2015-03-02 12:50 ` [PATCH perf/core 3/4] perf-probe: Fix --line " Masami Hiramatsu 2015-03-02 12:50 ` [PATCH perf/core 4/4] Revert "perf probe: Fix to fall back to find probe point in symbols" Masami Hiramatsu 3 siblings, 2 replies; 14+ messages in thread From: Masami Hiramatsu @ 2015-03-02 12:49 UTC (permalink / raw) To: Arnaldo Carvalho de Melo Cc: Naohiro Aota, Peter Zijlstra, Linux Kernel Mailing List, David Ahern, namhyung, Jiri Olsa, Ingo Molnar Fix perf probe to handle aliased symbols correctly in glibc. In the glibc, several symbols are defined as an alias of __libc_XXX, e.g. malloc is an alias of __libc_malloc. In such cases, dwarf has no subroutine instances of the alias functions (e.g. no "malloc" instance), but the map has that symbol and its address. Thus, if we search the alieased symbol in debuginfo, we always fail to find it, but it is in the map. To solve this problem, this fails back to address-based alternative search, which searches the symbol in the map, translates its address to alternative (correct) function name by using debuginfo, and retry to find the alternative function point from debuginfo. This adds fail-back process to --vars, --lines and --add options. So, now you can use those on malloc@libc :) Without this patch; ----- # ./perf probe -x /usr/lib64/libc-2.17.so -V malloc Failed to find the address of malloc Error: Failed to show vars. # ./perf probe -x /usr/lib64/libc-2.17.so -a "malloc bytes" Probe point 'malloc' not found in debuginfo. Error: Failed to add events. ----- With this patch; ----- # ./perf probe -x /usr/lib64/libc-2.17.so -V malloc Available variables at malloc @<__libc_malloc+0> size_t bytes # ./perf probe -x /usr/lib64/libc-2.17.so -a "malloc bytes" Added new event: probe_libc:malloc (on malloc in /usr/lib64/libc-2.17.so with bytes) You can now use it in all perf tools, such as: perf record -e probe_libc:malloc -aR sleep 1 ----- Reported-by: Arnaldo Carvalho de Melo <acme@kernel.org> Signed-off-by: Masami Hiramatsu <masami.hiramatsu.pt@hitachi.com> --- tools/perf/util/probe-event.c | 140 ++++++++++++++++++++++++++++++++++++----- 1 file changed, 124 insertions(+), 16 deletions(-) diff --git a/tools/perf/util/probe-event.c b/tools/perf/util/probe-event.c index 1c570c2fa7..b8f4578 100644 --- a/tools/perf/util/probe-event.c +++ b/tools/perf/util/probe-event.c @@ -178,6 +178,25 @@ static struct map *kernel_get_module_map(const char *module) return NULL; } +static struct map *get_target_map(const char *target, bool user) +{ + /* Init maps of given executable or kernel */ + if (user) + return dso__new_map(target); + else + return kernel_get_module_map(target); +} + +static void put_target_map(struct map *map, bool user) +{ + if (map && user) { + /* Only the user map needs to be released */ + dso__delete(map->dso); + map__delete(map); + } +} + + static struct dso *kernel_get_module_dso(const char *module) { struct dso *dso; @@ -249,6 +268,13 @@ out: return ret; } +static void clear_perf_probe_point(struct perf_probe_point *pp) +{ + free(pp->file); + free(pp->function); + free(pp->lazy_line); +} + static void clear_probe_trace_events(struct probe_trace_event *tevs, int ntevs) { int i; @@ -258,6 +284,74 @@ static void clear_probe_trace_events(struct probe_trace_event *tevs, int ntevs) } #ifdef HAVE_DWARF_SUPPORT +/* + * Some binaries like glibc have special symbols which are on the symbol + * table, but not in the debuginfo. If we can find the address of the + * symbol from map, we can translate the address back to the probe point. + */ +static int find_alternative_probe_point(struct debuginfo *dinfo, + struct perf_probe_point *pp, + struct perf_probe_point *result, + const char *target, bool uprobes) +{ + struct map *map = NULL; + struct symbol *sym; + u64 address = 0; + int ret = -ENOENT; + + /* This can work only for function-name based one */ + if (!pp->function || pp->file) + return -ENOTSUP; + + map = get_target_map(target, uprobes); + if (!map) + return -EINVAL; + + /* Find the address of given function */ + map__for_each_symbol_by_name(map, pp->function, sym) { + if (sym->binding == STB_GLOBAL || sym->binding == STB_LOCAL) { + address = sym->start; + break; + } + } + if (!address) { + ret = -ENOENT; + goto out; + } + pr_debug("Symbol %s address found : %lx\n", pp->function, address); + + ret = debuginfo__find_probe_point(dinfo, (unsigned long)address, + result); + if (ret <= 0) + ret = (!ret) ? -ENOENT : ret; + else { + result->offset += pp->offset; + result->line += pp->line; + ret = 0; + } + +out: + put_target_map(map, uprobes); + return ret; + +} + +static int get_alternative_probe_event(struct debuginfo *dinfo, + struct perf_probe_event *pev, + struct perf_probe_point *tmp, + const char *target) +{ + int ret; + + memcpy(tmp, &pev->point, sizeof(*tmp)); + memset(&pev->point, 0, sizeof(pev->point)); + ret = find_alternative_probe_point(dinfo, tmp, &pev->point, + target, pev->uprobes); + if (ret < 0) + memcpy(&pev->point, tmp, sizeof(*tmp)); + + return ret; +} /* Open new debuginfo of given module */ static struct debuginfo *open_debuginfo(const char *module, bool silent) @@ -466,6 +560,7 @@ static int try_to_find_probe_trace_events(struct perf_probe_event *pev, int max_tevs, const char *target) { bool need_dwarf = perf_probe_event_need_dwarf(pev); + struct perf_probe_point tmp; struct debuginfo *dinfo; int ntevs, ret = 0; @@ -482,6 +577,20 @@ static int try_to_find_probe_trace_events(struct perf_probe_event *pev, /* Searching trace events corresponding to a probe event */ ntevs = debuginfo__find_trace_events(dinfo, pev, tevs, max_tevs); + if (ntevs == 0) { /* Not found, retry with an alternative */ + ret = get_alternative_probe_event(dinfo, pev, &tmp, target); + if (!ret) { + ntevs = debuginfo__find_trace_events(dinfo, pev, + tevs, max_tevs); + /* + * Write back to the original probe_event for + * setting appropriate (user given) event name + */ + clear_perf_probe_point(&pev->point); + memcpy(&pev->point, &tmp, sizeof(tmp)); + } + } + debuginfo__delete(dinfo); if (ntevs > 0) { /* Succeeded to find trace events */ @@ -719,12 +828,13 @@ int show_line_range(struct line_range *lr, const char *module, bool user) static int show_available_vars_at(struct debuginfo *dinfo, struct perf_probe_event *pev, int max_vls, struct strfilter *_filter, - bool externs) + bool externs, const char *target) { char *buf; int ret, i, nvars; struct str_node *node; struct variable_list *vls = NULL, *vl; + struct perf_probe_point tmp; const char *var; buf = synthesize_perf_probe_point(&pev->point); @@ -734,6 +844,15 @@ static int show_available_vars_at(struct debuginfo *dinfo, ret = debuginfo__find_available_vars_at(dinfo, pev, &vls, max_vls, externs); + if (!ret) { /* Not found, retry with an alternative */ + ret = get_alternative_probe_event(dinfo, pev, &tmp, target); + if (!ret) { + ret = debuginfo__find_available_vars_at(dinfo, pev, + &vls, max_vls, externs); + /* Release the old probe_point */ + clear_perf_probe_point(&tmp); + } + } if (ret <= 0) { if (ret == 0 || ret == -ENOENT) { pr_err("Failed to find the address of %s\n", buf); @@ -796,7 +915,7 @@ int show_available_vars(struct perf_probe_event *pevs, int npevs, for (i = 0; i < npevs && ret >= 0; i++) ret = show_available_vars_at(dinfo, &pevs[i], max_vls, _filter, - externs); + externs, module); debuginfo__delete(dinfo); out: @@ -1742,15 +1861,12 @@ static int convert_to_perf_probe_event(struct probe_trace_event *tev, void clear_perf_probe_event(struct perf_probe_event *pev) { - struct perf_probe_point *pp = &pev->point; struct perf_probe_arg_field *field, *next; int i; free(pev->event); free(pev->group); - free(pp->file); - free(pp->function); - free(pp->lazy_line); + clear_perf_probe_point(&pev->point); for (i = 0; i < pev->nargs; i++) { free(pev->args[i].name); @@ -2367,11 +2483,7 @@ static int find_probe_trace_events_from_map(struct perf_probe_event *pev, int num_matched_functions; int ret, i; - /* Init maps of given executable or kernel */ - if (pev->uprobes) - map = dso__new_map(target); - else - map = kernel_get_module_map(target); + map = get_target_map(target, pev->uprobes); if (!map) { ret = -EINVAL; goto out; @@ -2464,11 +2576,7 @@ static int find_probe_trace_events_from_map(struct perf_probe_event *pev, } out: - if (map && pev->uprobes) { - /* Only when using uprobe(exec) map needs to be released */ - dso__delete(map->dso); - map__delete(map); - } + put_target_map(map, pev->uprobes); return ret; nomem_out: ^ permalink raw reply related [flat|nested] 14+ messages in thread
* Re: [PATCH perf/core 2/4] perf-probe: Fix to handle aliased symbols in glibc 2015-03-02 12:49 ` [PATCH perf/core 2/4] perf-probe: Fix to handle aliased symbols in glibc Masami Hiramatsu @ 2015-03-02 15:46 ` Arnaldo Carvalho de Melo 2015-03-03 2:39 ` Masami Hiramatsu 2015-03-03 13:31 ` Namhyung Kim 1 sibling, 1 reply; 14+ messages in thread From: Arnaldo Carvalho de Melo @ 2015-03-02 15:46 UTC (permalink / raw) To: Masami Hiramatsu Cc: Naohiro Aota, Peter Zijlstra, Linux Kernel Mailing List, David Ahern, namhyung, Jiri Olsa, Ingo Molnar Em Mon, Mar 02, 2015 at 09:49:53PM +0900, Masami Hiramatsu escreveu: > With this patch; > ----- > # ./perf probe -x /usr/lib64/libc-2.17.so -V malloc > Available variables at malloc > @<__libc_malloc+0> > size_t bytes > # ./perf probe -x /usr/lib64/libc-2.17.so -a "malloc bytes" > Added new event: > probe_libc:malloc (on malloc in /usr/lib64/libc-2.17.so with bytes) > > You can now use it in all perf tools, such as: > > perf record -e probe_libc:malloc -aR sleep 1 > Reported-by: Arnaldo Carvalho de Melo <acme@kernel.org> Humm, not working for me, after the patch: [root@ssdandy ~]# perf probe -x /usr/lib64/libc-2.17.so -V malloc Available variables at malloc @<__malloc_check_init+96> (No matched variables) [root@ssdandy ~]# And then the one asking for 'bytes' to be collectd fails. After processing the other patches I'll try to debug this... [root@ssdandy ~]# cat /etc/redhat-release Red Hat Enterprise Linux Server release 7.0 (Maipo) [root@ssdandy ~]# rpm -q glibc glibc-debuginfo glibc-2.17-55.el7_0.3.x86_64 glibc-debuginfo-2.17-55.el7_0.1.x86_64 [root@ssdandy ~]# [acme@ssdandy linux]$ readelf -Ws /usr/lib64/libc-2.17.so| grep malloc 438: 00000000000800c0 245 FUNC GLOBAL DEFAULT 12 __libc_malloc@@GLIBC_2.2.5 545: 0000000000082320 239 FUNC GLOBAL DEFAULT 12 malloc_info@@GLIBC_2.10 810: 00000000000820c0 490 FUNC WEAK DEFAULT 12 malloc_stats@@GLIBC_2.2.5 981: 00000000000802e0 507 FUNC WEAK DEFAULT 12 malloc_get_state@@GLIBC_2.2.5 1077: 00000000003ba740 8 OBJECT WEAK DEFAULT 32 __malloc_hook@@GLIBC_2.2.5 1170: 00000000000800c0 245 FUNC GLOBAL DEFAULT 12 malloc@@GLIBC_2.2.5 1204: 0000000000080d30 222 FUNC WEAK DEFAULT 12 malloc_usable_size@@GLIBC_2.2.5 1450: 0000000000081d50 604 FUNC WEAK DEFAULT 12 malloc_trim@@GLIBC_2.2.5 1767: 00000000003bca60 8 OBJECT WEAK DEFAULT 33 __malloc_initialize_hook@@GLIBC_2.2.5 2061: 00000000000814f0 1286 FUNC WEAK DEFAULT 12 malloc_set_state@@GLIBC_2.2.5 95: 00000000003bbaa0 4 OBJECT LOCAL DEFAULT 33 cache_malloced 1004: 0000000000000000 0 FILE LOCAL DEFAULT ABS malloc.c 1005: 000000000007b060 275 FUNC LOCAL DEFAULT 12 ptmalloc_lock_all 1011: 00000000003bcb30 8 OBJECT LOCAL DEFAULT 33 save_malloc_hook 1013: 00000000000801c0 285 FUNC LOCAL DEFAULT 12 malloc_atfork 1017: 000000000007b180 158 FUNC LOCAL DEFAULT 12 ptmalloc_unlock_all2 1030: 000000000007ba90 174 FUNC LOCAL DEFAULT 12 ptmalloc_unlock_all 1033: 000000000007bb40 232 FUNC LOCAL DEFAULT 12 malloc_printerr 1057: 000000000007c3d0 1518 FUNC LOCAL DEFAULT 12 malloc_consolidate 1089: 000000000007dee0 5195 FUNC LOCAL DEFAULT 12 _int_malloc 1100: 000000000007f330 282 FUNC LOCAL DEFAULT 12 malloc_check 1117: 00000000003bca90 4 OBJECT LOCAL DEFAULT 33 disallow_malloc_check 1118: 00000000003bcaa0 4 OBJECT LOCAL DEFAULT 33 using_malloc_checking 1136: 0000000000080fe0 1015 FUNC LOCAL DEFAULT 12 ptmalloc_init.part.8 1138: 00000000000813e0 21 FUNC LOCAL DEFAULT 12 ptmalloc_init 1139: 0000000000081400 60 FUNC LOCAL DEFAULT 12 malloc_hook_ini 1160: 0000000000082fc0 495 FUNC LOCAL DEFAULT 12 mallochook 1162: 00000000003bcbf0 8 OBJECT LOCAL DEFAULT 33 old_malloc_hook 1181: 00000000003bcc40 8 OBJECT LOCAL DEFAULT 33 tr_old_malloc_hook 1182: 0000000000083fc0 189 FUNC LOCAL DEFAULT 12 tr_mallochook 1194: 00000000003bcc60 8 OBJECT LOCAL DEFAULT 33 malloc_trace_buffer 3673: 00000000003ba170 4 OBJECT LOCAL DEFAULT 32 __libc_malloc_initialized 3734: 00000000000814f0 1286 FUNC LOCAL DEFAULT 12 __malloc_set_state 4047: 0000000000080d30 222 FUNC LOCAL DEFAULT 12 __malloc_usable_size 4101: 0000000000081d50 604 FUNC LOCAL DEFAULT 12 __malloc_trim 4338: 00000000000800c0 245 FUNC LOCAL DEFAULT 12 __GI___libc_malloc 4531: 00000000000802e0 507 FUNC LOCAL DEFAULT 12 __malloc_get_state 4569: 00000000000820c0 490 FUNC LOCAL DEFAULT 12 __malloc_stats 4849: 0000000000080050 107 FUNC LOCAL DEFAULT 12 __malloc_check_init 5351: 00000000000800c0 245 FUNC LOCAL DEFAULT 12 __malloc 5490: 00000000003bca60 8 OBJECT WEAK DEFAULT 33 __malloc_initialize_hook 5571: 00000000000814f0 1286 FUNC WEAK DEFAULT 12 malloc_set_state 5868: 00000000000800c0 245 FUNC GLOBAL DEFAULT 12 malloc 5878: 0000000000082320 239 FUNC GLOBAL DEFAULT 12 malloc_info 5988: 0000000000081d50 604 FUNC WEAK DEFAULT 12 malloc_trim 6526: 00000000003ba740 8 OBJECT WEAK DEFAULT 32 __malloc_hook 6615: 0000000000080d30 222 FUNC WEAK DEFAULT 12 malloc_usable_size 7087: 00000000000802e0 507 FUNC WEAK DEFAULT 12 malloc_get_state 7104: 00000000000800c0 245 FUNC GLOBAL DEFAULT 12 __libc_malloc 7271: 00000000000820c0 490 FUNC WEAK DEFAULT 12 malloc_stats [acme@ssdandy linux]$ - Arnaldo ^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: [PATCH perf/core 2/4] perf-probe: Fix to handle aliased symbols in glibc 2015-03-02 15:46 ` Arnaldo Carvalho de Melo @ 2015-03-03 2:39 ` Masami Hiramatsu 2015-03-03 2:45 ` Arnaldo Carvalho de Melo 0 siblings, 1 reply; 14+ messages in thread From: Masami Hiramatsu @ 2015-03-03 2:39 UTC (permalink / raw) To: Arnaldo Carvalho de Melo Cc: Naohiro Aota, Peter Zijlstra, Linux Kernel Mailing List, David Ahern, namhyung, Jiri Olsa, Ingo Molnar (2015/03/03 0:46), Arnaldo Carvalho de Melo wrote: > Em Mon, Mar 02, 2015 at 09:49:53PM +0900, Masami Hiramatsu escreveu: >> With this patch; >> ----- >> # ./perf probe -x /usr/lib64/libc-2.17.so -V malloc >> Available variables at malloc >> @<__libc_malloc+0> >> size_t bytes >> # ./perf probe -x /usr/lib64/libc-2.17.so -a "malloc bytes" >> Added new event: >> probe_libc:malloc (on malloc in /usr/lib64/libc-2.17.so with bytes) >> >> You can now use it in all perf tools, such as: >> >> perf record -e probe_libc:malloc -aR sleep 1 > >> Reported-by: Arnaldo Carvalho de Melo <acme@kernel.org> > > Humm, not working for me, after the patch: > > [root@ssdandy ~]# perf probe -x /usr/lib64/libc-2.17.so -V malloc > Available variables at malloc > @<__malloc_check_init+96> > (No matched variables) Could you run it with -v (verbose) option? > [root@ssdandy ~]# > > And then the one asking for 'bytes' to be collectd fails. > > After processing the other patches I'll try to debug this... > > [root@ssdandy ~]# cat /etc/redhat-release > Red Hat Enterprise Linux Server release 7.0 (Maipo) > [root@ssdandy ~]# rpm -q glibc glibc-debuginfo > glibc-2.17-55.el7_0.3.x86_64 > glibc-debuginfo-2.17-55.el7_0.1.x86_64 ^^^ why is this different from the glibc version?? > [root@ssdandy ~]# > [acme@ssdandy linux]$ readelf -Ws /usr/lib64/libc-2.17.so| grep malloc [...] > 4849: 0000000000080050 107 FUNC LOCAL DEFAULT 12 __malloc_check_init __malloc_check_init+96(0x60) becomes 0x80050 + 0x60 = 0x800b0 > 5351: 00000000000800c0 245 FUNC LOCAL DEFAULT 12 __malloc > 1170: 00000000000800c0 245 FUNC GLOBAL DEFAULT 12 malloc@@GLIBC_2.2.5 that is not 0x800c0, so something goes wrong when translating the address to probe point. Could you check the patch 1/4 was applied? Actually there is a bug in the routine which gets the probe point from address. 1/4 fixes it. Thank you, -- Masami HIRAMATSU Software Platform Research Dept. Linux Technology Research Center Hitachi, Ltd., Yokohama Research Laboratory E-mail: masami.hiramatsu.pt@hitachi.com ^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: [PATCH perf/core 2/4] perf-probe: Fix to handle aliased symbols in glibc 2015-03-03 2:39 ` Masami Hiramatsu @ 2015-03-03 2:45 ` Arnaldo Carvalho de Melo 2015-03-03 3:05 ` Arnaldo Carvalho de Melo 0 siblings, 1 reply; 14+ messages in thread From: Arnaldo Carvalho de Melo @ 2015-03-03 2:45 UTC (permalink / raw) To: Masami Hiramatsu Cc: Naohiro Aota, Peter Zijlstra, Linux Kernel Mailing List, David Ahern, namhyung, Jiri Olsa, Ingo Molnar Em Tue, Mar 03, 2015 at 11:39:02AM +0900, Masami Hiramatsu escreveu: > (2015/03/03 0:46), Arnaldo Carvalho de Melo wrote: > > Em Mon, Mar 02, 2015 at 09:49:53PM +0900, Masami Hiramatsu escreveu: > >> With this patch; > >> ----- > >> # ./perf probe -x /usr/lib64/libc-2.17.so -V malloc > >> Available variables at malloc > >> @<__libc_malloc+0> > >> size_t bytes > >> # ./perf probe -x /usr/lib64/libc-2.17.so -a "malloc bytes" > >> Added new event: > >> probe_libc:malloc (on malloc in /usr/lib64/libc-2.17.so with bytes) > >> > >> You can now use it in all perf tools, such as: > >> > >> perf record -e probe_libc:malloc -aR sleep 1 > > > >> Reported-by: Arnaldo Carvalho de Melo <acme@kernel.org> > > > > Humm, not working for me, after the patch: > > > > [root@ssdandy ~]# perf probe -x /usr/lib64/libc-2.17.so -V malloc > > Available variables at malloc > > @<__malloc_check_init+96> > > (No matched variables) > Will try after a 'make build-test' finishes for the current batch > > [root@ssdandy ~]# > > > > And then the one asking for 'bytes' to be collectd fails. > > > > After processing the other patches I'll try to debug this... > > > > [root@ssdandy ~]# cat /etc/redhat-release > > Red Hat Enterprise Linux Server release 7.0 (Maipo) > > [root@ssdandy ~]# rpm -q glibc glibc-debuginfo > > glibc-2.17-55.el7_0.3.x86_64 > > glibc-debuginfo-2.17-55.el7_0.1.x86_64 > ^^^ why is this different from the glibc version?? > > > [root@ssdandy ~]# > > [acme@ssdandy linux]$ readelf -Ws /usr/lib64/libc-2.17.so| grep malloc > [...] > > 4849: 0000000000080050 107 FUNC LOCAL DEFAULT 12 __malloc_check_init > > __malloc_check_init+96(0x60) becomes 0x80050 + 0x60 = 0x800b0 > > > 5351: 00000000000800c0 245 FUNC LOCAL DEFAULT 12 __malloc > > 1170: 00000000000800c0 245 FUNC GLOBAL DEFAULT 12 malloc@@GLIBC_2.2.5 > > that is not 0x800c0, so something goes wrong when translating the address > to probe point. Could you check the patch 1/4 was applied? > Actually there is a bug in the routine which gets the probe point from > address. 1/4 fixes it. This one? commit 0104fe69e0287cf3635657b4c6b26a18e0091697 Author: Masami Hiramatsu <masami.hiramatsu.pt@hitachi.com> Date: Mon Mar 2 21:49:46 2015 +0900 perf probe: Remove bias offset to find probe point by address Remove bias offset to find probe point by address. ---------------- Yes, it is applied. - Arnaldo ^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: [PATCH perf/core 2/4] perf-probe: Fix to handle aliased symbols in glibc 2015-03-03 2:45 ` Arnaldo Carvalho de Melo @ 2015-03-03 3:05 ` Arnaldo Carvalho de Melo 2015-03-03 4:11 ` Masami Hiramatsu 0 siblings, 1 reply; 14+ messages in thread From: Arnaldo Carvalho de Melo @ 2015-03-03 3:05 UTC (permalink / raw) To: Masami Hiramatsu Cc: Naohiro Aota, Peter Zijlstra, Linux Kernel Mailing List, David Ahern, namhyung, Jiri Olsa, Ingo Molnar Em Mon, Mar 02, 2015 at 11:45:12PM -0300, Arnaldo Carvalho de Melo escreveu: > Em Tue, Mar 03, 2015 at 11:39:02AM +0900, Masami Hiramatsu escreveu: > > (2015/03/03 0:46), Arnaldo Carvalho de Melo wrote: > > > Em Mon, Mar 02, 2015 at 09:49:53PM +0900, Masami Hiramatsu escreveu: > > >> With this patch; > > >> ----- > > >> # ./perf probe -x /usr/lib64/libc-2.17.so -V malloc > > >> Available variables at malloc > > >> @<__libc_malloc+0> > > >> size_t bytes > > >> # ./perf probe -x /usr/lib64/libc-2.17.so -a "malloc bytes" > > >> Added new event: > > >> probe_libc:malloc (on malloc in /usr/lib64/libc-2.17.so with bytes) > > >> > > >> You can now use it in all perf tools, such as: > > >> > > >> perf record -e probe_libc:malloc -aR sleep 1 > > > > > >> Reported-by: Arnaldo Carvalho de Melo <acme@kernel.org> > > > > > > Humm, not working for me, after the patch: > > > > > > [root@ssdandy ~]# perf probe -x /usr/lib64/libc-2.17.so -V malloc > > > Available variables at malloc > > > @<__malloc_check_init+96> > > > (No matched variables) > > > > Will try after a 'make build-test' finishes for the current batch [root@ssdandy ~]# perf probe -vvv -x /usr/lib64/libc-2.17.so -V malloc probe-definition(0): malloc symbol:malloc file:(null) line:0 offset:0 return:0 lazy:(null) 0 arguments Open Debuginfo file: /usr/lib/debug/usr/lib64/libc-2.17.so.debug Searching variables at malloc Symbol malloc address found : 800c0 Get 2611 lines from this CU Probe point found: __malloc_check_init+96 Available variables at malloc @<__malloc_check_init+96> (No matched variables) [root@ssdandy ~]# If I add one more 'v' I get the symtabs as read by symbol-elf.c and this is what is there for the malloc routines: [root@ssdandy ~]# grep malloc /tmp/4 probe-definition(0): malloc symbol:malloc file:(null) line:0 offset:0 return:0 lazy:(null) Searching variables at malloc symbol__new: ptmalloc_lock_all 0x7b060-0x7b173 symbol__new: malloc_atfork 0x801c0-0x802dd symbol__new: ptmalloc_unlock_all2 0x7b180-0x7b21e symbol__new: ptmalloc_unlock_all 0x7ba90-0x7bb3e symbol__new: malloc_printerr 0x7bb40-0x7bc28 symbol__new: malloc_consolidate 0x7c3d0-0x7c9be symbol__new: _int_malloc 0x7dee0-0x7f32b symbol__new: malloc_check 0x7f330-0x7f44a symbol__new: ptmalloc_init.part.8 0x80fe0-0x813d7 symbol__new: ptmalloc_init 0x813e0-0x813f5 symbol__new: malloc_hook_ini 0x81400-0x8143c symbol__new: mallochook 0x82fc0-0x831af symbol__new: tr_mallochook 0x83fc0-0x8407d symbol__new: __malloc_set_state 0x814f0-0x819f6 symbol__new: __malloc_usable_size 0x80d30-0x80e0e symbol__new: __malloc_trim 0x81d50-0x81fac symbol__new: __GI___libc_malloc 0x800c0-0x801b5 symbol__new: __malloc_get_state 0x802e0-0x804db symbol__new: __malloc_stats 0x820c0-0x822aa symbol__new: __malloc_check_init 0x80050-0x800bb symbol__new: __malloc 0x800c0-0x801b5 symbol__new: malloc_set_state 0x814f0-0x819f6 symbol__new: malloc 0x800c0-0x801b5 symbol__new: malloc_info 0x82320-0x8240f symbol__new: malloc_trim 0x81d50-0x81fac symbol__new: malloc_usable_size 0x80d30-0x80e0e symbol__new: malloc_get_state 0x802e0-0x804db symbol__new: __libc_malloc 0x800c0-0x801b5 symbol__new: malloc_stats 0x820c0-0x822aa symbol__new: malloc@plt 0x1f300-0x1f310 Symbol malloc address found : 800c0 Probe point found: __malloc_check_init+96 [root@ssdandy ~]# > > > [root@ssdandy ~]# > > > > > > And then the one asking for 'bytes' to be collectd fails. > > > > > > After processing the other patches I'll try to debug this... > > > > > > [root@ssdandy ~]# cat /etc/redhat-release > > > Red Hat Enterprise Linux Server release 7.0 (Maipo) > > > [root@ssdandy ~]# rpm -q glibc glibc-debuginfo > > > glibc-2.17-55.el7_0.3.x86_64 > > > glibc-debuginfo-2.17-55.el7_0.1.x86_64 > > ^^^ why is this different from the glibc version?? > > > > > [root@ssdandy ~]# > > > [acme@ssdandy linux]$ readelf -Ws /usr/lib64/libc-2.17.so| grep malloc > > [...] > > > 4849: 0000000000080050 107 FUNC LOCAL DEFAULT 12 __malloc_check_init > > > > __malloc_check_init+96(0x60) becomes 0x80050 + 0x60 = 0x800b0 > > > > > 5351: 00000000000800c0 245 FUNC LOCAL DEFAULT 12 __malloc > > > 1170: 00000000000800c0 245 FUNC GLOBAL DEFAULT 12 malloc@@GLIBC_2.2.5 > > > > that is not 0x800c0, so something goes wrong when translating the address > > to probe point. Could you check the patch 1/4 was applied? > > Actually there is a bug in the routine which gets the probe point from > > address. 1/4 fixes it. > > This one? > > commit 0104fe69e0287cf3635657b4c6b26a18e0091697 > Author: Masami Hiramatsu <masami.hiramatsu.pt@hitachi.com> > Date: Mon Mar 2 21:49:46 2015 +0900 > > perf probe: Remove bias offset to find probe point by address > > Remove bias offset to find probe point by address. > > ---------------- > > Yes, it is applied. > > - Arnaldo ^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: Re: [PATCH perf/core 2/4] perf-probe: Fix to handle aliased symbols in glibc 2015-03-03 3:05 ` Arnaldo Carvalho de Melo @ 2015-03-03 4:11 ` Masami Hiramatsu 2015-03-03 4:24 ` Arnaldo Carvalho de Melo 0 siblings, 1 reply; 14+ messages in thread From: Masami Hiramatsu @ 2015-03-03 4:11 UTC (permalink / raw) To: Arnaldo Carvalho de Melo Cc: Naohiro Aota, Peter Zijlstra, Linux Kernel Mailing List, David Ahern, namhyung, Jiri Olsa, Ingo Molnar (2015/03/03 12:05), Arnaldo Carvalho de Melo wrote: > Em Mon, Mar 02, 2015 at 11:45:12PM -0300, Arnaldo Carvalho de Melo escreveu: >> Em Tue, Mar 03, 2015 at 11:39:02AM +0900, Masami Hiramatsu escreveu: >>> (2015/03/03 0:46), Arnaldo Carvalho de Melo wrote: >>>> Em Mon, Mar 02, 2015 at 09:49:53PM +0900, Masami Hiramatsu escreveu: >>>>> With this patch; >>>>> ----- >>>>> # ./perf probe -x /usr/lib64/libc-2.17.so -V malloc >>>>> Available variables at malloc >>>>> @<__libc_malloc+0> >>>>> size_t bytes >>>>> # ./perf probe -x /usr/lib64/libc-2.17.so -a "malloc bytes" >>>>> Added new event: >>>>> probe_libc:malloc (on malloc in /usr/lib64/libc-2.17.so with bytes) >>>>> >>>>> You can now use it in all perf tools, such as: >>>>> >>>>> perf record -e probe_libc:malloc -aR sleep 1 >>>> >>>>> Reported-by: Arnaldo Carvalho de Melo <acme@kernel.org> >>>> >>>> Humm, not working for me, after the patch: >>>> >>>> [root@ssdandy ~]# perf probe -x /usr/lib64/libc-2.17.so -V malloc >>>> Available variables at malloc >>>> @<__malloc_check_init+96> >>>> (No matched variables) >>> >> >> Will try after a 'make build-test' finishes for the current batch > Thank you for checking this, > [root@ssdandy ~]# perf probe -vvv -x /usr/lib64/libc-2.17.so -V malloc > probe-definition(0): malloc > symbol:malloc file:(null) line:0 offset:0 return:0 lazy:(null) > 0 arguments > Open Debuginfo file: /usr/lib/debug/usr/lib64/libc-2.17.so.debug > Searching variables at malloc > Symbol malloc address found : 800c0 > Get 2611 lines from this CU Hmm, something wrong at here. > Probe point found: __malloc_check_init+96 This seems that the debuginfo is a bit odd. Could you also run eu-addr2line for 0x800c0 and 0x800b0? I'm using CentOS7 with a bit newer glibc rpms. # rpm -q glibc glibc-debuginfo glibc-2.17-55.el7_0.5.x86_64 glibc-debuginfo-2.17-55.el7_0.5.x86_64 And I got following results. [mhiramat@localhost perf]$ eu-addr2line -fi -e /usr/lib64/libc-2.17.so 0x800c0 __libc_malloc /usr/src/debug/glibc-2.17-c758a686/malloc/malloc.c:2855 [mhiramat@localhost perf]$ eu-addr2line -fi -e /usr/lib64/libc-2.17.so 0x800b0 __malloc_check_init /usr/src/debug/glibc-2.17-c758a686/malloc/hooks.c:75 So, 0x800b0 correctly points __malloc_check_init+96 but its address is not 0x800c0. Thank you, > Available variables at malloc > @<__malloc_check_init+96> > (No matched variables) > [root@ssdandy ~]# > > If I add one more 'v' I get the symtabs as read by symbol-elf.c and this is > what is there for the malloc routines: Yeah, it seems symbol maps works fine. Debuginfo analysis failed. > > [root@ssdandy ~]# grep malloc /tmp/4 > probe-definition(0): malloc > symbol:malloc file:(null) line:0 offset:0 return:0 lazy:(null) > Searching variables at malloc > symbol__new: ptmalloc_lock_all 0x7b060-0x7b173 > symbol__new: malloc_atfork 0x801c0-0x802dd > symbol__new: ptmalloc_unlock_all2 0x7b180-0x7b21e > symbol__new: ptmalloc_unlock_all 0x7ba90-0x7bb3e > symbol__new: malloc_printerr 0x7bb40-0x7bc28 > symbol__new: malloc_consolidate 0x7c3d0-0x7c9be > symbol__new: _int_malloc 0x7dee0-0x7f32b > symbol__new: malloc_check 0x7f330-0x7f44a > symbol__new: ptmalloc_init.part.8 0x80fe0-0x813d7 > symbol__new: ptmalloc_init 0x813e0-0x813f5 > symbol__new: malloc_hook_ini 0x81400-0x8143c > symbol__new: mallochook 0x82fc0-0x831af > symbol__new: tr_mallochook 0x83fc0-0x8407d > symbol__new: __malloc_set_state 0x814f0-0x819f6 > symbol__new: __malloc_usable_size 0x80d30-0x80e0e > symbol__new: __malloc_trim 0x81d50-0x81fac > symbol__new: __GI___libc_malloc 0x800c0-0x801b5 > symbol__new: __malloc_get_state 0x802e0-0x804db > symbol__new: __malloc_stats 0x820c0-0x822aa > symbol__new: __malloc_check_init 0x80050-0x800bb > symbol__new: __malloc 0x800c0-0x801b5 > symbol__new: malloc_set_state 0x814f0-0x819f6 > symbol__new: malloc 0x800c0-0x801b5 > symbol__new: malloc_info 0x82320-0x8240f > symbol__new: malloc_trim 0x81d50-0x81fac > symbol__new: malloc_usable_size 0x80d30-0x80e0e > symbol__new: malloc_get_state 0x802e0-0x804db > symbol__new: __libc_malloc 0x800c0-0x801b5 > symbol__new: malloc_stats 0x820c0-0x822aa > symbol__new: malloc@plt 0x1f300-0x1f310 > Symbol malloc address found : 800c0 > Probe point found: __malloc_check_init+96 > [root@ssdandy ~]# > >>>> [root@ssdandy ~]# >>>> >>>> And then the one asking for 'bytes' to be collectd fails. >>>> >>>> After processing the other patches I'll try to debug this... >>>> >>>> [root@ssdandy ~]# cat /etc/redhat-release >>>> Red Hat Enterprise Linux Server release 7.0 (Maipo) >>>> [root@ssdandy ~]# rpm -q glibc glibc-debuginfo >>>> glibc-2.17-55.el7_0.3.x86_64 >>>> glibc-debuginfo-2.17-55.el7_0.1.x86_64 >>> ^^^ why is this different from the glibc version?? >>> >>>> [root@ssdandy ~]# >>>> [acme@ssdandy linux]$ readelf -Ws /usr/lib64/libc-2.17.so| grep malloc >>> [...] >>>> 4849: 0000000000080050 107 FUNC LOCAL DEFAULT 12 __malloc_check_init >>> >>> __malloc_check_init+96(0x60) becomes 0x80050 + 0x60 = 0x800b0 >>> >>>> 5351: 00000000000800c0 245 FUNC LOCAL DEFAULT 12 __malloc >>>> 1170: 00000000000800c0 245 FUNC GLOBAL DEFAULT 12 malloc@@GLIBC_2.2.5 >>> >>> that is not 0x800c0, so something goes wrong when translating the address >>> to probe point. Could you check the patch 1/4 was applied? >>> Actually there is a bug in the routine which gets the probe point from >>> address. 1/4 fixes it. >> >> This one? >> >> commit 0104fe69e0287cf3635657b4c6b26a18e0091697 >> Author: Masami Hiramatsu <masami.hiramatsu.pt@hitachi.com> >> Date: Mon Mar 2 21:49:46 2015 +0900 >> >> perf probe: Remove bias offset to find probe point by address >> >> Remove bias offset to find probe point by address. >> >> ---------------- >> >> Yes, it is applied. >> >> - Arnaldo > -- > To unsubscribe from this list: send the line "unsubscribe linux-kernel" in > the body of a message to majordomo@vger.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html > Please read the FAQ at http://www.tux.org/lkml/ > -- Masami HIRAMATSU Software Platform Research Dept. Linux Technology Research Center Hitachi, Ltd., Yokohama Research Laboratory E-mail: masami.hiramatsu.pt@hitachi.com ^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: Re: [PATCH perf/core 2/4] perf-probe: Fix to handle aliased symbols in glibc 2015-03-03 4:11 ` Masami Hiramatsu @ 2015-03-03 4:24 ` Arnaldo Carvalho de Melo 0 siblings, 0 replies; 14+ messages in thread From: Arnaldo Carvalho de Melo @ 2015-03-03 4:24 UTC (permalink / raw) To: Masami Hiramatsu Cc: Naohiro Aota, Peter Zijlstra, Linux Kernel Mailing List, David Ahern, namhyung, Jiri Olsa, Ingo Molnar Em Tue, Mar 03, 2015 at 01:11:17PM +0900, Masami Hiramatsu escreveu: > (2015/03/03 12:05), Arnaldo Carvalho de Melo wrote: > > Em Mon, Mar 02, 2015 at 11:45:12PM -0300, Arnaldo Carvalho de Melo escreveu: > >> Em Tue, Mar 03, 2015 at 11:39:02AM +0900, Masami Hiramatsu escreveu: > >>> (2015/03/03 0:46), Arnaldo Carvalho de Melo wrote: > >>>> Em Mon, Mar 02, 2015 at 09:49:53PM +0900, Masami Hiramatsu escreveu: > >>>>> With this patch; > >>>>> ----- > >>>>> # ./perf probe -x /usr/lib64/libc-2.17.so -V malloc > >>>>> Available variables at malloc > >>>>> @<__libc_malloc+0> > >>>>> size_t bytes > >>>>> # ./perf probe -x /usr/lib64/libc-2.17.so -a "malloc bytes" > >>>>> Added new event: > >>>>> probe_libc:malloc (on malloc in /usr/lib64/libc-2.17.so with bytes) > >>>>> > >>>>> You can now use it in all perf tools, such as: > >>>>> > >>>>> perf record -e probe_libc:malloc -aR sleep 1 > >>>> > >>>>> Reported-by: Arnaldo Carvalho de Melo <acme@kernel.org> > >>>> > >>>> Humm, not working for me, after the patch: > >>>> > >>>> [root@ssdandy ~]# perf probe -x /usr/lib64/libc-2.17.so -V malloc > >>>> Available variables at malloc > >>>> @<__malloc_check_init+96> > >>>> (No matched variables) > >>> > >> > >> Will try after a 'make build-test' finishes for the current batch > > > > Thank you for checking this, > > > [root@ssdandy ~]# perf probe -vvv -x /usr/lib64/libc-2.17.so -V malloc > > probe-definition(0): malloc > > symbol:malloc file:(null) line:0 offset:0 return:0 lazy:(null) > > 0 arguments > > Open Debuginfo file: /usr/lib/debug/usr/lib64/libc-2.17.so.debug > > Searching variables at malloc > > Symbol malloc address found : 800c0 > > Get 2611 lines from this CU > > Hmm, something wrong at here. > > > Probe point found: __malloc_check_init+96 > > This seems that the debuginfo is a bit odd. > Could you also run eu-addr2line for 0x800c0 and 0x800b0? > > I'm using CentOS7 with a bit newer glibc rpms. > > # rpm -q glibc glibc-debuginfo > glibc-2.17-55.el7_0.5.x86_64 > glibc-debuginfo-2.17-55.el7_0.5.x86_64 > > And I got following results. > > [mhiramat@localhost perf]$ eu-addr2line -fi -e /usr/lib64/libc-2.17.so 0x800c0 > __libc_malloc > /usr/src/debug/glibc-2.17-c758a686/malloc/malloc.c:2855 > [mhiramat@localhost perf]$ eu-addr2line -fi -e /usr/lib64/libc-2.17.so 0x800b0 > __malloc_check_init > /usr/src/debug/glibc-2.17-c758a686/malloc/hooks.c:75 > > So, 0x800b0 correctly points __malloc_check_init+96 but its address is not 0x800c0. [acme@ssdandy linux]$ eu-addr2line -fi -e /usr/lib64/libc-2.17.so 0x800c0 malloc ??:0 [acme@ssdandy linux]$ eu-addr2line -fi -e /usr/lib64/libc-2.17.so 0x800b0 __malloc_check_init ??:0 [acme@ssdandy linux]$ If I do it over the debuginfo files: [acme@ssdandy linux]$ addr2line -fi -e /usr/lib/debug/usr/lib64/libc-2.17.so.debug 0x800c0 __malloc_check_init /usr/src/debug/glibc-2.17-c758a686/malloc/hooks.c:75 [acme@ssdandy linux]$ addr2line -fi -e /usr/lib/debug/usr/lib64/libc-2.17.so.debug 0x800b0 __malloc_check_init /usr/src/debug/glibc-2.17-c758a686/malloc/hooks.c:82 [acme@ssdandy linux]$ Tomorrow I'll check for newer packages or any reports about problems with debuginfo files, please let me know if you need any further info, Off to bed, zzZzz - Arnaldo > Thank you, > > > > Available variables at malloc > > @<__malloc_check_init+96> > > (No matched variables) > > [root@ssdandy ~]# > > > > If I add one more 'v' I get the symtabs as read by symbol-elf.c and this is > > what is there for the malloc routines: > > Yeah, it seems symbol maps works fine. Debuginfo analysis failed. > > > > > [root@ssdandy ~]# grep malloc /tmp/4 > > probe-definition(0): malloc > > symbol:malloc file:(null) line:0 offset:0 return:0 lazy:(null) > > Searching variables at malloc > > symbol__new: ptmalloc_lock_all 0x7b060-0x7b173 > > symbol__new: malloc_atfork 0x801c0-0x802dd > > symbol__new: ptmalloc_unlock_all2 0x7b180-0x7b21e > > symbol__new: ptmalloc_unlock_all 0x7ba90-0x7bb3e > > symbol__new: malloc_printerr 0x7bb40-0x7bc28 > > symbol__new: malloc_consolidate 0x7c3d0-0x7c9be > > symbol__new: _int_malloc 0x7dee0-0x7f32b > > symbol__new: malloc_check 0x7f330-0x7f44a > > symbol__new: ptmalloc_init.part.8 0x80fe0-0x813d7 > > symbol__new: ptmalloc_init 0x813e0-0x813f5 > > symbol__new: malloc_hook_ini 0x81400-0x8143c > > symbol__new: mallochook 0x82fc0-0x831af > > symbol__new: tr_mallochook 0x83fc0-0x8407d > > symbol__new: __malloc_set_state 0x814f0-0x819f6 > > symbol__new: __malloc_usable_size 0x80d30-0x80e0e > > symbol__new: __malloc_trim 0x81d50-0x81fac > > symbol__new: __GI___libc_malloc 0x800c0-0x801b5 > > symbol__new: __malloc_get_state 0x802e0-0x804db > > symbol__new: __malloc_stats 0x820c0-0x822aa > > symbol__new: __malloc_check_init 0x80050-0x800bb > > symbol__new: __malloc 0x800c0-0x801b5 > > symbol__new: malloc_set_state 0x814f0-0x819f6 > > symbol__new: malloc 0x800c0-0x801b5 > > symbol__new: malloc_info 0x82320-0x8240f > > symbol__new: malloc_trim 0x81d50-0x81fac > > symbol__new: malloc_usable_size 0x80d30-0x80e0e > > symbol__new: malloc_get_state 0x802e0-0x804db > > symbol__new: __libc_malloc 0x800c0-0x801b5 > > symbol__new: malloc_stats 0x820c0-0x822aa > > symbol__new: malloc@plt 0x1f300-0x1f310 > > Symbol malloc address found : 800c0 > > Probe point found: __malloc_check_init+96 > > [root@ssdandy ~]# > > > >>>> [root@ssdandy ~]# > >>>> > >>>> And then the one asking for 'bytes' to be collectd fails. > >>>> > >>>> After processing the other patches I'll try to debug this... > >>>> > >>>> [root@ssdandy ~]# cat /etc/redhat-release > >>>> Red Hat Enterprise Linux Server release 7.0 (Maipo) > >>>> [root@ssdandy ~]# rpm -q glibc glibc-debuginfo > >>>> glibc-2.17-55.el7_0.3.x86_64 > >>>> glibc-debuginfo-2.17-55.el7_0.1.x86_64 > >>> ^^^ why is this different from the glibc version?? > >>> > >>>> [root@ssdandy ~]# > >>>> [acme@ssdandy linux]$ readelf -Ws /usr/lib64/libc-2.17.so| grep malloc > >>> [...] > >>>> 4849: 0000000000080050 107 FUNC LOCAL DEFAULT 12 __malloc_check_init > >>> > >>> __malloc_check_init+96(0x60) becomes 0x80050 + 0x60 = 0x800b0 > >>> > >>>> 5351: 00000000000800c0 245 FUNC LOCAL DEFAULT 12 __malloc > >>>> 1170: 00000000000800c0 245 FUNC GLOBAL DEFAULT 12 malloc@@GLIBC_2.2.5 > >>> > >>> that is not 0x800c0, so something goes wrong when translating the address > >>> to probe point. Could you check the patch 1/4 was applied? > >>> Actually there is a bug in the routine which gets the probe point from > >>> address. 1/4 fixes it. > >> > >> This one? > >> > >> commit 0104fe69e0287cf3635657b4c6b26a18e0091697 > >> Author: Masami Hiramatsu <masami.hiramatsu.pt@hitachi.com> > >> Date: Mon Mar 2 21:49:46 2015 +0900 > >> > >> perf probe: Remove bias offset to find probe point by address > >> > >> Remove bias offset to find probe point by address. > >> > >> ---------------- > >> > >> Yes, it is applied. > >> > >> - Arnaldo > > -- > > To unsubscribe from this list: send the line "unsubscribe linux-kernel" in > > the body of a message to majordomo@vger.kernel.org > > More majordomo info at http://vger.kernel.org/majordomo-info.html > > Please read the FAQ at http://www.tux.org/lkml/ > > > > > -- > Masami HIRAMATSU > Software Platform Research Dept. Linux Technology Research Center > Hitachi, Ltd., Yokohama Research Laboratory > E-mail: masami.hiramatsu.pt@hitachi.com > ^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: [PATCH perf/core 2/4] perf-probe: Fix to handle aliased symbols in glibc 2015-03-02 12:49 ` [PATCH perf/core 2/4] perf-probe: Fix to handle aliased symbols in glibc Masami Hiramatsu 2015-03-02 15:46 ` Arnaldo Carvalho de Melo @ 2015-03-03 13:31 ` Namhyung Kim 2015-03-04 5:47 ` Masami Hiramatsu 1 sibling, 1 reply; 14+ messages in thread From: Namhyung Kim @ 2015-03-03 13:31 UTC (permalink / raw) To: Masami Hiramatsu Cc: Arnaldo Carvalho de Melo, Naohiro Aota, Peter Zijlstra, Linux Kernel Mailing List, David Ahern, Jiri Olsa, Ingo Molnar Hi Masami, On Mon, Mar 02, 2015 at 09:49:53PM +0900, Masami Hiramatsu wrote: > Fix perf probe to handle aliased symbols correctly in glibc. > In the glibc, several symbols are defined as an alias of > __libc_XXX, e.g. malloc is an alias of __libc_malloc. > In such cases, dwarf has no subroutine instances of the > alias functions (e.g. no "malloc" instance), but the map > has that symbol and its address. > Thus, if we search the alieased symbol in debuginfo, we > always fail to find it, but it is in the map. > > To solve this problem, this fails back to address-based > alternative search, which searches the symbol in the map, > translates its address to alternative (correct) function > name by using debuginfo, and retry to find the alternative > function point from debuginfo. > > This adds fail-back process to --vars, --lines and --add > options. So, now you can use those on malloc@libc :) So this is only for binaries that have debuginfo, right? I have a similar issue with no debuginfo. $ perf probe -x /usr/lib/libc.so.6 -V calloc The /usr/lib/libc-2.21.so file has no debug information. Rebuild with -g, or install an appropriate debuginfo package. Error: Failed to show vars. But it also failed to add a probe to calloc: $ perf probe -x /usr/lib/libc.so.6 -a calloc Failed to find symbol calloc in /usr/lib/libc-2.21.so Error: Failed to add events. Of course there's calloc in the libc binary. $ nm /usr/lib/libc.so.6 | grep calloc 000000000007b1f0 t __calloc 000000000007b1f0 T __libc_calloc 000000000007b1f0 W calloc I think the problem is that calloc is a weak symbol so it'll be discarded during the symbol loading. It's because to avoid multiple symbols (or aliases) at a same address so we choose a better symbol using heuristics. But for this case I think we can allow aliases since it's used only for finding probe points. Thanks, Namhyung ^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: Re: [PATCH perf/core 2/4] perf-probe: Fix to handle aliased symbols in glibc 2015-03-03 13:31 ` Namhyung Kim @ 2015-03-04 5:47 ` Masami Hiramatsu 0 siblings, 0 replies; 14+ messages in thread From: Masami Hiramatsu @ 2015-03-04 5:47 UTC (permalink / raw) To: Namhyung Kim Cc: Arnaldo Carvalho de Melo, Naohiro Aota, Peter Zijlstra, Linux Kernel Mailing List, David Ahern, Jiri Olsa, Ingo Molnar (2015/03/03 22:31), Namhyung Kim wrote: > Hi Masami, > > On Mon, Mar 02, 2015 at 09:49:53PM +0900, Masami Hiramatsu wrote: >> Fix perf probe to handle aliased symbols correctly in glibc. >> In the glibc, several symbols are defined as an alias of >> __libc_XXX, e.g. malloc is an alias of __libc_malloc. >> In such cases, dwarf has no subroutine instances of the >> alias functions (e.g. no "malloc" instance), but the map >> has that symbol and its address. >> Thus, if we search the alieased symbol in debuginfo, we >> always fail to find it, but it is in the map. >> >> To solve this problem, this fails back to address-based >> alternative search, which searches the symbol in the map, >> translates its address to alternative (correct) function >> name by using debuginfo, and retry to find the alternative >> function point from debuginfo. >> >> This adds fail-back process to --vars, --lines and --add >> options. So, now you can use those on malloc@libc :) > > So this is only for binaries that have debuginfo, right? > > I have a similar issue with no debuginfo. > > $ perf probe -x /usr/lib/libc.so.6 -V calloc > The /usr/lib/libc-2.21.so file has no debug information. > Rebuild with -g, or install an appropriate debuginfo package. > Error: Failed to show vars. > > > But it also failed to add a probe to calloc: > > $ perf probe -x /usr/lib/libc.so.6 -a calloc > Failed to find symbol calloc in /usr/lib/libc-2.21.so > Error: Failed to add events. > > > Of course there's calloc in the libc binary. > > $ nm /usr/lib/libc.so.6 | grep calloc > 000000000007b1f0 t __calloc > 000000000007b1f0 T __libc_calloc > 000000000007b1f0 W calloc > > > I think the problem is that calloc is a weak symbol so it'll be > discarded during the symbol loading. Right, I also hit same problem on waitpid, which is also an weak symbol. > It's because to avoid multiple > symbols (or aliases) at a same address so we choose a better symbol > using heuristics. But for this case I think we can allow aliases > since it's used only for finding probe points. I'm just using struct map and dso, so those should be improved. Thank you, -- Masami HIRAMATSU Software Platform Research Dept. Linux Technology Research Center Hitachi, Ltd., Yokohama Research Laboratory E-mail: masami.hiramatsu.pt@hitachi.com ^ permalink raw reply [flat|nested] 14+ messages in thread
* [PATCH perf/core 3/4] perf-probe: Fix --line to handle aliased symbols in glibc 2015-03-02 12:49 [PATCH perf/core 0/4] perf-probe: improve glibc support Masami Hiramatsu 2015-03-02 12:49 ` [PATCH perf/core 1/4] [RESEND][BUGFIX] perf-probe: Remove bias offset to find probe point by address Masami Hiramatsu 2015-03-02 12:49 ` [PATCH perf/core 2/4] perf-probe: Fix to handle aliased symbols in glibc Masami Hiramatsu @ 2015-03-02 12:50 ` Masami Hiramatsu 2015-03-02 12:50 ` [PATCH perf/core 4/4] Revert "perf probe: Fix to fall back to find probe point in symbols" Masami Hiramatsu 3 siblings, 0 replies; 14+ messages in thread From: Masami Hiramatsu @ 2015-03-02 12:50 UTC (permalink / raw) To: Arnaldo Carvalho de Melo Cc: Naohiro Aota, Peter Zijlstra, Linux Kernel Mailing List, David Ahern, namhyung, Jiri Olsa, Ingo Molnar Fix perf probe --line to handle aliased symbols correctly in glibc. This makes line_range search failing back to address-based alternative search as same as --add and --vars. Without this patch; ----- # ./perf probe -x /usr/lib64/libc-2.17.so -L malloc Specified source line is not found. Error: Failed to show lines. ----- With this patch; ----- # ./perf probe -x /usr/lib64/libc-2.17.so -L malloc <__libc_malloc@/usr/src/debug/glibc-2.17-c758a686/malloc/malloc.c:0> 0 __libc_malloc(size_t bytes) 1 { mstate ar_ptr; void *victim; __malloc_ptr_t (*hook) (size_t, const __malloc_ptr_t) 6 = force_reg (__malloc_hook); 7 if (__builtin_expect (hook != NULL, 0)) 8 return (*hook)(bytes, RETURN_ADDRESS (0)); 10 arena_lookup(ar_ptr); 12 arena_lock(ar_ptr, bytes); ----- Note that this actually shows __libc_malloc, since it is the real instance of malloc. User can use both __libc_malloc and malloc for --line. Signed-off-by: Masami Hiramatsu <masami.hiramatsu.pt@hitachi.com> --- tools/perf/util/probe-event.c | 35 +++++++++++++++++++++++++++++++++-- 1 file changed, 33 insertions(+), 2 deletions(-) diff --git a/tools/perf/util/probe-event.c b/tools/perf/util/probe-event.c index b8f4578..4cfd121 100644 --- a/tools/perf/util/probe-event.c +++ b/tools/perf/util/probe-event.c @@ -353,6 +353,31 @@ static int get_alternative_probe_event(struct debuginfo *dinfo, return ret; } +static int get_alternative_line_range(struct debuginfo *dinfo, + struct line_range *lr, + const char *target, bool user) +{ + struct perf_probe_point pp = { 0 }, result = { 0 }; + int ret, len = 0; + + pp.function = lr->function; + pp.file = lr->file; + pp.line = lr->start; + if (lr->end != INT_MAX) + len = lr->end - lr->start; + ret = find_alternative_probe_point(dinfo, &pp, &result, + target, user); + if (!ret) { + lr->function = result.function; + lr->file = result.file; + lr->start = result.line; + if (lr->end != INT_MAX) + lr->end = lr->start + len; + clear_perf_probe_point(&pp); + } + return ret; +} + /* Open new debuginfo of given module */ static struct debuginfo *open_debuginfo(const char *module, bool silent) { @@ -734,7 +759,8 @@ static int _show_one_line(FILE *fp, int l, bool skip, bool show_num) * Show line-range always requires debuginfo to find source file and * line number. */ -static int __show_line_range(struct line_range *lr, const char *module) +static int __show_line_range(struct line_range *lr, const char *module, + bool user) { int l = 1; struct int_node *ln; @@ -750,6 +776,11 @@ static int __show_line_range(struct line_range *lr, const char *module) return -ENOENT; ret = debuginfo__find_line_range(dinfo, lr); + if (!ret) { /* Not found, retry with an alternative */ + ret = get_alternative_line_range(dinfo, lr, module, user); + if (!ret) + ret = debuginfo__find_line_range(dinfo, lr); + } debuginfo__delete(dinfo); if (ret == 0 || ret == -ENOENT) { pr_warning("Specified source line is not found.\n"); @@ -819,7 +850,7 @@ int show_line_range(struct line_range *lr, const char *module, bool user) ret = init_symbol_maps(user); if (ret < 0) return ret; - ret = __show_line_range(lr, module); + ret = __show_line_range(lr, module, user); exit_symbol_maps(); return ret; ^ permalink raw reply related [flat|nested] 14+ messages in thread
* [PATCH perf/core 4/4] Revert "perf probe: Fix to fall back to find probe point in symbols" 2015-03-02 12:49 [PATCH perf/core 0/4] perf-probe: improve glibc support Masami Hiramatsu ` (2 preceding siblings ...) 2015-03-02 12:50 ` [PATCH perf/core 3/4] perf-probe: Fix --line " Masami Hiramatsu @ 2015-03-02 12:50 ` Masami Hiramatsu 3 siblings, 0 replies; 14+ messages in thread From: Masami Hiramatsu @ 2015-03-02 12:50 UTC (permalink / raw) To: Arnaldo Carvalho de Melo Cc: Naohiro Aota, Peter Zijlstra, Linux Kernel Mailing List, David Ahern, namhyung, Jiri Olsa, Ingo Molnar This reverts commit 906451b98b67 ("perf probe: Fix to fall back to find probe point in symbols"). Since perf-probe retries with the address of given symbol searched from map before this path, this fall back routine doesn't need anymore. Signed-off-by: Masami Hiramatsu <masami.hiramatsu.pt@hitachi.com> --- tools/perf/util/probe-event.c | 6 ++---- 1 file changed, 2 insertions(+), 4 deletions(-) diff --git a/tools/perf/util/probe-event.c b/tools/perf/util/probe-event.c index 4cfd121..c379ea0 100644 --- a/tools/perf/util/probe-event.c +++ b/tools/perf/util/probe-event.c @@ -630,11 +630,9 @@ static int try_to_find_probe_trace_events(struct perf_probe_event *pev, } if (ntevs == 0) { /* No error but failed to find probe point. */ - pr_warning("Probe point '%s' not found in debuginfo.\n", + pr_warning("Probe point '%s' not found.\n", synthesize_perf_probe_point(&pev->point)); - if (need_dwarf) - return -ENOENT; - return 0; + return -ENOENT; } /* Error path : ntevs < 0 */ pr_debug("An error occurred in debuginfo analysis (%d).\n", ntevs); ^ permalink raw reply related [flat|nested] 14+ messages in thread
end of thread, other threads:[~2015-03-04 5:48 UTC | newest] Thread overview: 14+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2015-03-02 12:49 [PATCH perf/core 0/4] perf-probe: improve glibc support Masami Hiramatsu 2015-03-02 12:49 ` [PATCH perf/core 1/4] [RESEND][BUGFIX] perf-probe: Remove bias offset to find probe point by address Masami Hiramatsu 2015-03-03 6:26 ` [tip:perf/core] perf probe: " tip-bot for Masami Hiramatsu 2015-03-02 12:49 ` [PATCH perf/core 2/4] perf-probe: Fix to handle aliased symbols in glibc Masami Hiramatsu 2015-03-02 15:46 ` Arnaldo Carvalho de Melo 2015-03-03 2:39 ` Masami Hiramatsu 2015-03-03 2:45 ` Arnaldo Carvalho de Melo 2015-03-03 3:05 ` Arnaldo Carvalho de Melo 2015-03-03 4:11 ` Masami Hiramatsu 2015-03-03 4:24 ` Arnaldo Carvalho de Melo 2015-03-03 13:31 ` Namhyung Kim 2015-03-04 5:47 ` Masami Hiramatsu 2015-03-02 12:50 ` [PATCH perf/core 3/4] perf-probe: Fix --line " Masami Hiramatsu 2015-03-02 12:50 ` [PATCH perf/core 4/4] Revert "perf probe: Fix to fall back to find probe point in symbols" Masami Hiramatsu
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s).