[PATCH perf/core 0/4] perf-probe: improve glibc support

[PATCH perf/core 0/4] perf-probe: improve glibc support

[Masami Hiramatsu]

Hi,

Here is a series of patches which improves perf-probe to
handle glibc's aliased symbols more correctly. Longly,
a major known issue of probing on glibc is that the
some aliased symbols(e.g. malloc) can not find by
perf-probe.

Actually, glibc's malloc symbol is just an alias of
__libc_malloc. Its debuginfo knows only __libc_malloc,
and perf's symbol map knows only malloc. This difference
always confuses users that they can see malloc by perf
report or annotate, but they can not probe on it, nor
find definitions by --line option.

Previously, I've made a commit 906451b98b67 which solved
this problem partly, but not completely fixed.
So I decided to solve this issue completely by finding
the symbols like malloc from perf's symbol map, and 
converting the symbol's address into debuginfo's
location infomation.

With this series, you can use --vars, --line and --add
with the aliased symbols on glibc.

Note that the 1/4 patch is just a resend patch which I
sent last weekend (since this series depends on that)
http://www.gossamer-threads.com/lists/linux/kernel/2115623

2/4 and 3/4 introduces above logic to solve this issue.
And 4/4 reverts previous incomplete fix.

Thank you,


---

Masami Hiramatsu (4):
      [RESEND][BUGFIX] perf-probe: Remove bias offset to find probe point by address
      perf-probe: Fix to handle aliased symbols in glibc
      perf-probe: Fix --line to handle aliased symbols in glibc
      Revert "perf probe: Fix to fall back to find probe point in symbols"


 tools/perf/util/probe-event.c  |  181 +++++++++++++++++++++++++++++++++++-----
 tools/perf/util/probe-finder.c |    5 -
 2 files changed, 160 insertions(+), 26 deletions(-)

--
Masami HIRAMATSU
Software Platform Research Dpt. Linux Technology Center
Hitachi, Ltd., Yokohama Research Laboratory
E-mail: masami.hiramatsu.pt@hitachi.com

[PATCH perf/core 1/4] [RESEND][BUGFIX] perf-probe: Remove bias offset to find probe point by address

[Masami Hiramatsu]

Remove bias offset to find probe point by address.

Without this patch, probe points on kernel and executables
are shown correctly, but not work with libraries.
  -----
  # ./perf probe -l
    probe:do_fork        (on do_fork@kernel/fork.c)
    probe_libc:malloc    (on malloc in /usr/lib64/libc-2.17.so)
    probe_perf:strlist__new (on strlist__new@util/strlist.c in /home/mhiramat/ksrc/linux-3/tools/perf/perf)
  -----

Removing bias allows to show it as real place.
  -----
  # ./perf probe -l
    probe:do_fork        (on do_fork@kernel/fork.c)
    probe_libc:malloc    (on __libc_malloc@malloc/malloc.c in /usr/lib64/libc-2.17.so)
    probe_perf:strlist__new (on strlist__new@util/strlist.c in /home/mhiramat/ksrc/linux-3/tools/perf/perf)
  -----

Signed-off-by: Masami Hiramatsu <masami.hiramatsu.pt@hitachi.com>
---
 tools/perf/util/probe-finder.c |    5 +----
 1 file changed, 1 insertion(+), 4 deletions(-)

diff --git a/tools/perf/util/probe-finder.c b/tools/perf/util/probe-finder.c
index d141935..46f009a 100644
--- a/tools/perf/util/probe-finder.c
+++ b/tools/perf/util/probe-finder.c
@@ -1345,11 +1345,8 @@ int debuginfo__find_probe_point(struct debuginfo *dbg, unsigned long addr,
 	const char *fname = NULL, *func = NULL, *basefunc = NULL, *tmp;
 	int baseline = 0, lineno = 0, ret = 0;
 
-	/* Adjust address with bias */
-	addr += dbg->bias;
-
 	/* Find cu die */
-	if (!dwarf_addrdie(dbg->dbg, (Dwarf_Addr)addr - dbg->bias, &cudie)) {
+	if (!dwarf_addrdie(dbg->dbg, (Dwarf_Addr)addr, &cudie)) {
 		pr_warning("Failed to find debug information for address %lx\n",
 			   addr);
 		ret = -EINVAL;

[tip:perf/core] perf probe: Remove bias offset to find probe point by address

[tip-bot for Masami Hiramatsu]

Commit-ID:  0104fe69e0287cf3635657b4c6b26a18e0091697
Gitweb:     http://git.kernel.org/tip/0104fe69e0287cf3635657b4c6b26a18e0091697
Author:     Masami Hiramatsu <masami.hiramatsu.pt@hitachi.com>
AuthorDate: Mon, 2 Mar 2015 21:49:46 +0900
Committer:  Arnaldo Carvalho de Melo <acme@redhat.com>
CommitDate: Mon, 2 Mar 2015 12:34:38 -0300

perf probe: Remove bias offset to find probe point by address

Remove bias offset to find probe point by address.

Without this patch, probe points on kernel and executables are shown
correctly, but do not work with libraries:

  # ./perf probe -l
    probe:do_fork        (on do_fork@kernel/fork.c)
    probe_libc:malloc    (on malloc in /usr/lib64/libc-2.17.so)
    probe_perf:strlist__new (on strlist__new@util/strlist.c in /home/mhiramat/ksrc/linux-3/tools/perf/perf)

Removing bias allows it to show it as real place:

  # ./perf probe -l
    probe:do_fork        (on do_fork@kernel/fork.c)
    probe_libc:malloc    (on __libc_malloc@malloc/malloc.c in /usr/lib64/libc-2.17.so)
    probe_perf:strlist__new (on strlist__new@util/strlist.c in /home/mhiramat/ksrc/linux-3/tools/perf/perf)

Signed-off-by: Masami Hiramatsu <masami.hiramatsu.pt@hitachi.com>
Cc: David Ahern <dsahern@gmail.com>
Cc: Jiri Olsa <jolsa@redhat.com>
Cc: Namhyung Kim <namhyung@kernel.org>
Cc: Naohiro Aota <naota@elisp.net>
Cc: Peter Zijlstra <peterz@infradead.org>
Link: http://lkml.kernel.org/r/20150302124946.9191.64085.stgit@localhost.localdomain
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
---
 tools/perf/util/probe-finder.c | 5 +----
 1 file changed, 1 insertion(+), 4 deletions(-)

diff --git a/tools/perf/util/probe-finder.c b/tools/perf/util/probe-finder.c
index d141935..46f009a 100644
--- a/tools/perf/util/probe-finder.c
+++ b/tools/perf/util/probe-finder.c
@@ -1345,11 +1345,8 @@ int debuginfo__find_probe_point(struct debuginfo *dbg, unsigned long addr,
 	const char *fname = NULL, *func = NULL, *basefunc = NULL, *tmp;
 	int baseline = 0, lineno = 0, ret = 0;
 
-	/* Adjust address with bias */
-	addr += dbg->bias;
-
 	/* Find cu die */
-	if (!dwarf_addrdie(dbg->dbg, (Dwarf_Addr)addr - dbg->bias, &cudie)) {
+	if (!dwarf_addrdie(dbg->dbg, (Dwarf_Addr)addr, &cudie)) {
 		pr_warning("Failed to find debug information for address %lx\n",
 			   addr);
 		ret = -EINVAL;

[PATCH perf/core 2/4] perf-probe: Fix to handle aliased symbols in glibc

[Masami Hiramatsu]

Fix perf probe to handle aliased symbols correctly in glibc.
In the glibc, several symbols are defined as an alias of
__libc_XXX, e.g. malloc is an alias of __libc_malloc.
In such cases, dwarf has no subroutine instances of the
alias functions (e.g. no "malloc" instance), but the map
has that symbol and its address.
Thus, if we search the alieased symbol in debuginfo, we
always fail to find it, but it is in the map.

To solve this problem, this fails back to address-based
alternative search, which searches the symbol in the map,
translates its address to alternative (correct) function
name by using debuginfo, and retry to find the alternative
function point from debuginfo.

This adds fail-back process to --vars, --lines and --add
options. So, now you can use those on malloc@libc :)

Without this patch;
  -----
  # ./perf probe -x /usr/lib64/libc-2.17.so -V malloc
  Failed to find the address of malloc
    Error: Failed to show vars.
  # ./perf probe -x /usr/lib64/libc-2.17.so -a "malloc bytes"
  Probe point 'malloc' not found in debuginfo.
    Error: Failed to add events.
  -----

With this patch;
  -----
  # ./perf probe -x /usr/lib64/libc-2.17.so -V malloc
  Available variables at malloc
          @<__libc_malloc+0>
                  size_t  bytes
  # ./perf probe -x /usr/lib64/libc-2.17.so -a "malloc bytes"
  Added new event:
    probe_libc:malloc    (on malloc in /usr/lib64/libc-2.17.so with bytes)

  You can now use it in all perf tools, such as:

          perf record -e probe_libc:malloc -aR sleep 1
  -----

Reported-by: Arnaldo Carvalho de Melo <acme@kernel.org>
Signed-off-by: Masami Hiramatsu <masami.hiramatsu.pt@hitachi.com>
---
 tools/perf/util/probe-event.c |  140 ++++++++++++++++++++++++++++++++++++-----
 1 file changed, 124 insertions(+), 16 deletions(-)

diff --git a/tools/perf/util/probe-event.c b/tools/perf/util/probe-event.c
index 1c570c2fa7..b8f4578 100644
--- a/tools/perf/util/probe-event.c
+++ b/tools/perf/util/probe-event.c
@@ -178,6 +178,25 @@ static struct map *kernel_get_module_map(const char *module)
 	return NULL;
 }
 
+static struct map *get_target_map(const char *target, bool user)
+{
+	/* Init maps of given executable or kernel */
+	if (user)
+		return dso__new_map(target);
+	else
+		return kernel_get_module_map(target);
+}
+
+static void put_target_map(struct map *map, bool user)
+{
+	if (map && user) {
+		/* Only the user map needs to be released */
+		dso__delete(map->dso);
+		map__delete(map);
+	}
+}
+
+
 static struct dso *kernel_get_module_dso(const char *module)
 {
 	struct dso *dso;
@@ -249,6 +268,13 @@ out:
 	return ret;
 }
 
+static void clear_perf_probe_point(struct perf_probe_point *pp)
+{
+	free(pp->file);
+	free(pp->function);
+	free(pp->lazy_line);
+}
+
 static void clear_probe_trace_events(struct probe_trace_event *tevs, int ntevs)
 {
 	int i;
@@ -258,6 +284,74 @@ static void clear_probe_trace_events(struct probe_trace_event *tevs, int ntevs)
 }
 
 #ifdef HAVE_DWARF_SUPPORT
+/*
+ * Some binaries like glibc have special symbols which are on the symbol
+ * table, but not in the debuginfo. If we can find the address of the
+ * symbol from map, we can translate the address back to the probe point.
+ */
+static int find_alternative_probe_point(struct debuginfo *dinfo,
+					struct perf_probe_point *pp,
+					struct perf_probe_point *result,
+					const char *target, bool uprobes)
+{
+	struct map *map = NULL;
+	struct symbol *sym;
+	u64 address = 0;
+	int ret = -ENOENT;
+
+	/* This can work only for function-name based one */
+	if (!pp->function || pp->file)
+		return -ENOTSUP;
+
+	map = get_target_map(target, uprobes);
+	if (!map)
+		return -EINVAL;
+
+	/* Find the address of given function */
+	map__for_each_symbol_by_name(map, pp->function, sym) {
+		if (sym->binding == STB_GLOBAL || sym->binding == STB_LOCAL) {
+			address = sym->start;
+			break;
+		}
+	}
+	if (!address) {
+		ret = -ENOENT;
+		goto out;
+	}
+	pr_debug("Symbol %s address found : %lx\n", pp->function, address);
+
+	ret = debuginfo__find_probe_point(dinfo, (unsigned long)address,
+					  result);
+	if (ret <= 0)
+		ret = (!ret) ? -ENOENT : ret;
+	else {
+		result->offset += pp->offset;
+		result->line += pp->line;
+		ret = 0;
+	}
+
+out:
+	put_target_map(map, uprobes);
+	return ret;
+
+}
+
+static int get_alternative_probe_event(struct debuginfo *dinfo,
+				       struct perf_probe_event *pev,
+				       struct perf_probe_point *tmp,
+				       const char *target)
+{
+	int ret;
+
+	memcpy(tmp, &pev->point, sizeof(*tmp));
+	memset(&pev->point, 0, sizeof(pev->point));
+	ret = find_alternative_probe_point(dinfo, tmp, &pev->point,
+					   target, pev->uprobes);
+	if (ret < 0)
+		memcpy(&pev->point, tmp, sizeof(*tmp));
+
+	return ret;
+}
 
 /* Open new debuginfo of given module */
 static struct debuginfo *open_debuginfo(const char *module, bool silent)
@@ -466,6 +560,7 @@ static int try_to_find_probe_trace_events(struct perf_probe_event *pev,
 					  int max_tevs, const char *target)
 {
 	bool need_dwarf = perf_probe_event_need_dwarf(pev);
+	struct perf_probe_point tmp;
 	struct debuginfo *dinfo;
 	int ntevs, ret = 0;
 
@@ -482,6 +577,20 @@ static int try_to_find_probe_trace_events(struct perf_probe_event *pev,
 	/* Searching trace events corresponding to a probe event */
 	ntevs = debuginfo__find_trace_events(dinfo, pev, tevs, max_tevs);
 
+	if (ntevs == 0)	{  /* Not found, retry with an alternative */
+		ret = get_alternative_probe_event(dinfo, pev, &tmp, target);
+		if (!ret) {
+			ntevs = debuginfo__find_trace_events(dinfo, pev,
+							     tevs, max_tevs);
+			/*
+			 * Write back to the original probe_event for
+			 * setting appropriate (user given) event name
+			 */
+			clear_perf_probe_point(&pev->point);
+			memcpy(&pev->point, &tmp, sizeof(tmp));
+		}
+	}
+
 	debuginfo__delete(dinfo);
 
 	if (ntevs > 0) {	/* Succeeded to find trace events */
@@ -719,12 +828,13 @@ int show_line_range(struct line_range *lr, const char *module, bool user)
 static int show_available_vars_at(struct debuginfo *dinfo,
 				  struct perf_probe_event *pev,
 				  int max_vls, struct strfilter *_filter,
-				  bool externs)
+				  bool externs, const char *target)
 {
 	char *buf;
 	int ret, i, nvars;
 	struct str_node *node;
 	struct variable_list *vls = NULL, *vl;
+	struct perf_probe_point tmp;
 	const char *var;
 
 	buf = synthesize_perf_probe_point(&pev->point);
@@ -734,6 +844,15 @@ static int show_available_vars_at(struct debuginfo *dinfo,
 
 	ret = debuginfo__find_available_vars_at(dinfo, pev, &vls,
 						max_vls, externs);
+	if (!ret) {  /* Not found, retry with an alternative */
+		ret = get_alternative_probe_event(dinfo, pev, &tmp, target);
+		if (!ret) {
+			ret = debuginfo__find_available_vars_at(dinfo, pev,
+						&vls, max_vls, externs);
+			/* Release the old probe_point */
+			clear_perf_probe_point(&tmp);
+		}
+	}
 	if (ret <= 0) {
 		if (ret == 0 || ret == -ENOENT) {
 			pr_err("Failed to find the address of %s\n", buf);
@@ -796,7 +915,7 @@ int show_available_vars(struct perf_probe_event *pevs, int npevs,
 
 	for (i = 0; i < npevs && ret >= 0; i++)
 		ret = show_available_vars_at(dinfo, &pevs[i], max_vls, _filter,
-					     externs);
+					     externs, module);
 
 	debuginfo__delete(dinfo);
 out:
@@ -1742,15 +1861,12 @@ static int convert_to_perf_probe_event(struct probe_trace_event *tev,
 
 void clear_perf_probe_event(struct perf_probe_event *pev)
 {
-	struct perf_probe_point *pp = &pev->point;
 	struct perf_probe_arg_field *field, *next;
 	int i;
 
 	free(pev->event);
 	free(pev->group);
-	free(pp->file);
-	free(pp->function);
-	free(pp->lazy_line);
+	clear_perf_probe_point(&pev->point);
 
 	for (i = 0; i < pev->nargs; i++) {
 		free(pev->args[i].name);
@@ -2367,11 +2483,7 @@ static int find_probe_trace_events_from_map(struct perf_probe_event *pev,
 	int num_matched_functions;
 	int ret, i;
 
-	/* Init maps of given executable or kernel */
-	if (pev->uprobes)
-		map = dso__new_map(target);
-	else
-		map = kernel_get_module_map(target);
+	map = get_target_map(target, pev->uprobes);
 	if (!map) {
 		ret = -EINVAL;
 		goto out;
@@ -2464,11 +2576,7 @@ static int find_probe_trace_events_from_map(struct perf_probe_event *pev,
 	}
 
 out:
-	if (map && pev->uprobes) {
-		/* Only when using uprobe(exec) map needs to be released */
-		dso__delete(map->dso);
-		map__delete(map);
-	}
+	put_target_map(map, pev->uprobes);
 	return ret;
 
 nomem_out:

Re: [PATCH perf/core 2/4] perf-probe: Fix to handle aliased symbols in glibc

[Arnaldo Carvalho de Melo]

Em Mon, Mar 02, 2015 at 09:49:53PM +0900, Masami Hiramatsu escreveu:
> With this patch;
>   -----
>   # ./perf probe -x /usr/lib64/libc-2.17.so -V malloc
>   Available variables at malloc
>           @<__libc_malloc+0>
>                   size_t  bytes
>   # ./perf probe -x /usr/lib64/libc-2.17.so -a "malloc bytes"
>   Added new event:
>     probe_libc:malloc    (on malloc in /usr/lib64/libc-2.17.so with bytes)
> 
>   You can now use it in all perf tools, such as:
> 
>           perf record -e probe_libc:malloc -aR sleep 1

> Reported-by: Arnaldo Carvalho de Melo <acme@kernel.org>

Humm, not working for me, after the patch:

[root@ssdandy ~]# perf probe -x /usr/lib64/libc-2.17.so -V malloc
Available variables at malloc
        @<__malloc_check_init+96>
                (No matched variables)
[root@ssdandy ~]#

And then the one asking for 'bytes' to be collectd fails.

After processing the other patches I'll try to debug this...

[root@ssdandy ~]# cat /etc/redhat-release 
Red Hat Enterprise Linux Server release 7.0 (Maipo)
[root@ssdandy ~]# rpm -q glibc glibc-debuginfo
glibc-2.17-55.el7_0.3.x86_64
glibc-debuginfo-2.17-55.el7_0.1.x86_64
[root@ssdandy ~]#
[acme@ssdandy linux]$ readelf -Ws /usr/lib64/libc-2.17.so| grep malloc
   438: 00000000000800c0   245 FUNC    GLOBAL DEFAULT   12 __libc_malloc@@GLIBC_2.2.5
   545: 0000000000082320   239 FUNC    GLOBAL DEFAULT   12 malloc_info@@GLIBC_2.10
   810: 00000000000820c0   490 FUNC    WEAK   DEFAULT   12 malloc_stats@@GLIBC_2.2.5
   981: 00000000000802e0   507 FUNC    WEAK   DEFAULT   12 malloc_get_state@@GLIBC_2.2.5
  1077: 00000000003ba740     8 OBJECT  WEAK   DEFAULT   32 __malloc_hook@@GLIBC_2.2.5
  1170: 00000000000800c0   245 FUNC    GLOBAL DEFAULT   12 malloc@@GLIBC_2.2.5
  1204: 0000000000080d30   222 FUNC    WEAK   DEFAULT   12 malloc_usable_size@@GLIBC_2.2.5
  1450: 0000000000081d50   604 FUNC    WEAK   DEFAULT   12 malloc_trim@@GLIBC_2.2.5
  1767: 00000000003bca60     8 OBJECT  WEAK   DEFAULT   33 __malloc_initialize_hook@@GLIBC_2.2.5
  2061: 00000000000814f0  1286 FUNC    WEAK   DEFAULT   12 malloc_set_state@@GLIBC_2.2.5
    95: 00000000003bbaa0     4 OBJECT  LOCAL  DEFAULT   33 cache_malloced
  1004: 0000000000000000     0 FILE    LOCAL  DEFAULT  ABS malloc.c
  1005: 000000000007b060   275 FUNC    LOCAL  DEFAULT   12 ptmalloc_lock_all
  1011: 00000000003bcb30     8 OBJECT  LOCAL  DEFAULT   33 save_malloc_hook
  1013: 00000000000801c0   285 FUNC    LOCAL  DEFAULT   12 malloc_atfork
  1017: 000000000007b180   158 FUNC    LOCAL  DEFAULT   12 ptmalloc_unlock_all2
  1030: 000000000007ba90   174 FUNC    LOCAL  DEFAULT   12 ptmalloc_unlock_all
  1033: 000000000007bb40   232 FUNC    LOCAL  DEFAULT   12 malloc_printerr
  1057: 000000000007c3d0  1518 FUNC    LOCAL  DEFAULT   12 malloc_consolidate
  1089: 000000000007dee0  5195 FUNC    LOCAL  DEFAULT   12 _int_malloc
  1100: 000000000007f330   282 FUNC    LOCAL  DEFAULT   12 malloc_check
  1117: 00000000003bca90     4 OBJECT  LOCAL  DEFAULT   33 disallow_malloc_check
  1118: 00000000003bcaa0     4 OBJECT  LOCAL  DEFAULT   33 using_malloc_checking
  1136: 0000000000080fe0  1015 FUNC    LOCAL  DEFAULT   12 ptmalloc_init.part.8
  1138: 00000000000813e0    21 FUNC    LOCAL  DEFAULT   12 ptmalloc_init
  1139: 0000000000081400    60 FUNC    LOCAL  DEFAULT   12 malloc_hook_ini
  1160: 0000000000082fc0   495 FUNC    LOCAL  DEFAULT   12 mallochook
  1162: 00000000003bcbf0     8 OBJECT  LOCAL  DEFAULT   33 old_malloc_hook
  1181: 00000000003bcc40     8 OBJECT  LOCAL  DEFAULT   33 tr_old_malloc_hook
  1182: 0000000000083fc0   189 FUNC    LOCAL  DEFAULT   12 tr_mallochook
  1194: 00000000003bcc60     8 OBJECT  LOCAL  DEFAULT   33 malloc_trace_buffer
  3673: 00000000003ba170     4 OBJECT  LOCAL  DEFAULT   32 __libc_malloc_initialized
  3734: 00000000000814f0  1286 FUNC    LOCAL  DEFAULT   12 __malloc_set_state
  4047: 0000000000080d30   222 FUNC    LOCAL  DEFAULT   12 __malloc_usable_size
  4101: 0000000000081d50   604 FUNC    LOCAL  DEFAULT   12 __malloc_trim
  4338: 00000000000800c0   245 FUNC    LOCAL  DEFAULT   12 __GI___libc_malloc
  4531: 00000000000802e0   507 FUNC    LOCAL  DEFAULT   12 __malloc_get_state
  4569: 00000000000820c0   490 FUNC    LOCAL  DEFAULT   12 __malloc_stats
  4849: 0000000000080050   107 FUNC    LOCAL  DEFAULT   12 __malloc_check_init
  5351: 00000000000800c0   245 FUNC    LOCAL  DEFAULT   12 __malloc
  5490: 00000000003bca60     8 OBJECT  WEAK   DEFAULT   33 __malloc_initialize_hook
  5571: 00000000000814f0  1286 FUNC    WEAK   DEFAULT   12 malloc_set_state
  5868: 00000000000800c0   245 FUNC    GLOBAL DEFAULT   12 malloc
  5878: 0000000000082320   239 FUNC    GLOBAL DEFAULT   12 malloc_info
  5988: 0000000000081d50   604 FUNC    WEAK   DEFAULT   12 malloc_trim
  6526: 00000000003ba740     8 OBJECT  WEAK   DEFAULT   32 __malloc_hook
  6615: 0000000000080d30   222 FUNC    WEAK   DEFAULT   12 malloc_usable_size
  7087: 00000000000802e0   507 FUNC    WEAK   DEFAULT   12 malloc_get_state
  7104: 00000000000800c0   245 FUNC    GLOBAL DEFAULT   12 __libc_malloc
  7271: 00000000000820c0   490 FUNC    WEAK   DEFAULT   12 malloc_stats
[acme@ssdandy linux]$

- Arnaldo

Re: [PATCH perf/core 2/4] perf-probe: Fix to handle aliased symbols in glibc

[Masami Hiramatsu]

(2015/03/03 0:46), Arnaldo Carvalho de Melo wrote:
> Em Mon, Mar 02, 2015 at 09:49:53PM +0900, Masami Hiramatsu escreveu:
>> With this patch;
>>   -----
>>   # ./perf probe -x /usr/lib64/libc-2.17.so -V malloc
>>   Available variables at malloc
>>           @<__libc_malloc+0>
>>                   size_t  bytes
>>   # ./perf probe -x /usr/lib64/libc-2.17.so -a "malloc bytes"
>>   Added new event:
>>     probe_libc:malloc    (on malloc in /usr/lib64/libc-2.17.so with bytes)
>>
>>   You can now use it in all perf tools, such as:
>>
>>           perf record -e probe_libc:malloc -aR sleep 1
> 
>> Reported-by: Arnaldo Carvalho de Melo <acme@kernel.org>
> 
> Humm, not working for me, after the patch:
> 
> [root@ssdandy ~]# perf probe -x /usr/lib64/libc-2.17.so -V malloc
> Available variables at malloc
>         @<__malloc_check_init+96>
>                 (No matched variables)

Could you run it with -v (verbose) option?

> [root@ssdandy ~]#
> 
> And then the one asking for 'bytes' to be collectd fails.
> 
> After processing the other patches I'll try to debug this...
> 
> [root@ssdandy ~]# cat /etc/redhat-release 
> Red Hat Enterprise Linux Server release 7.0 (Maipo)
> [root@ssdandy ~]# rpm -q glibc glibc-debuginfo
> glibc-2.17-55.el7_0.3.x86_64
> glibc-debuginfo-2.17-55.el7_0.1.x86_64
                              ^^^ why is this different from the glibc version??

> [root@ssdandy ~]#
> [acme@ssdandy linux]$ readelf -Ws /usr/lib64/libc-2.17.so| grep malloc
[...]
>   4849: 0000000000080050   107 FUNC    LOCAL  DEFAULT   12 __malloc_check_init

__malloc_check_init+96(0x60) becomes 0x80050 + 0x60 = 0x800b0

>   5351: 00000000000800c0   245 FUNC    LOCAL  DEFAULT   12 __malloc
>   1170: 00000000000800c0   245 FUNC    GLOBAL DEFAULT   12 malloc@@GLIBC_2.2.5

that is not 0x800c0, so something goes wrong when translating the address
to probe point. Could you check the patch 1/4 was applied?
Actually there is a bug in the routine which gets the probe point from
address. 1/4 fixes it.

Thank you,

-- 
Masami HIRAMATSU
Software Platform Research Dept. Linux Technology Research Center
Hitachi, Ltd., Yokohama Research Laboratory
E-mail: masami.hiramatsu.pt@hitachi.com

Re: [PATCH perf/core 2/4] perf-probe: Fix to handle aliased symbols in glibc

[Arnaldo Carvalho de Melo]

Em Tue, Mar 03, 2015 at 11:39:02AM +0900, Masami Hiramatsu escreveu:
> (2015/03/03 0:46), Arnaldo Carvalho de Melo wrote:
> > Em Mon, Mar 02, 2015 at 09:49:53PM +0900, Masami Hiramatsu escreveu:
> >> With this patch;
> >>   -----
> >>   # ./perf probe -x /usr/lib64/libc-2.17.so -V malloc
> >>   Available variables at malloc
> >>           @<__libc_malloc+0>
> >>                   size_t  bytes
> >>   # ./perf probe -x /usr/lib64/libc-2.17.so -a "malloc bytes"
> >>   Added new event:
> >>     probe_libc:malloc    (on malloc in /usr/lib64/libc-2.17.so with bytes)
> >>
> >>   You can now use it in all perf tools, such as:
> >>
> >>           perf record -e probe_libc:malloc -aR sleep 1
> > 
> >> Reported-by: Arnaldo Carvalho de Melo <acme@kernel.org>
> > 
> > Humm, not working for me, after the patch:
> > 
> > [root@ssdandy ~]# perf probe -x /usr/lib64/libc-2.17.so -V malloc
> > Available variables at malloc
> >         @<__malloc_check_init+96>
> >                 (No matched variables)
> 

Will try after a 'make build-test' finishes for the current batch

> > [root@ssdandy ~]#
> > 
> > And then the one asking for 'bytes' to be collectd fails.
> > 
> > After processing the other patches I'll try to debug this...
> > 
> > [root@ssdandy ~]# cat /etc/redhat-release 
> > Red Hat Enterprise Linux Server release 7.0 (Maipo)
> > [root@ssdandy ~]# rpm -q glibc glibc-debuginfo
> > glibc-2.17-55.el7_0.3.x86_64
> > glibc-debuginfo-2.17-55.el7_0.1.x86_64
>                               ^^^ why is this different from the glibc version??
> 
> > [root@ssdandy ~]#
> > [acme@ssdandy linux]$ readelf -Ws /usr/lib64/libc-2.17.so| grep malloc
> [...]
> >   4849: 0000000000080050   107 FUNC    LOCAL  DEFAULT   12 __malloc_check_init
> 
> __malloc_check_init+96(0x60) becomes 0x80050 + 0x60 = 0x800b0
> 
> >   5351: 00000000000800c0   245 FUNC    LOCAL  DEFAULT   12 __malloc
> >   1170: 00000000000800c0   245 FUNC    GLOBAL DEFAULT   12 malloc@@GLIBC_2.2.5
> 
> that is not 0x800c0, so something goes wrong when translating the address
> to probe point. Could you check the patch 1/4 was applied?
> Actually there is a bug in the routine which gets the probe point from
> address. 1/4 fixes it.

This one?

commit 0104fe69e0287cf3635657b4c6b26a18e0091697
Author: Masami Hiramatsu <masami.hiramatsu.pt@hitachi.com>
Date:   Mon Mar 2 21:49:46 2015 +0900

    perf probe: Remove bias offset to find probe point by address
    
    Remove bias offset to find probe point by address.

----------------

Yes, it is applied.

- Arnaldo

Re: [PATCH perf/core 2/4] perf-probe: Fix to handle aliased symbols in glibc

[Arnaldo Carvalho de Melo]

Em Mon, Mar 02, 2015 at 11:45:12PM -0300, Arnaldo Carvalho de Melo escreveu:
> Em Tue, Mar 03, 2015 at 11:39:02AM +0900, Masami Hiramatsu escreveu:
> > (2015/03/03 0:46), Arnaldo Carvalho de Melo wrote:
> > > Em Mon, Mar 02, 2015 at 09:49:53PM +0900, Masami Hiramatsu escreveu:
> > >> With this patch;
> > >>   -----
> > >>   # ./perf probe -x /usr/lib64/libc-2.17.so -V malloc
> > >>   Available variables at malloc
> > >>           @<__libc_malloc+0>
> > >>                   size_t  bytes
> > >>   # ./perf probe -x /usr/lib64/libc-2.17.so -a "malloc bytes"
> > >>   Added new event:
> > >>     probe_libc:malloc    (on malloc in /usr/lib64/libc-2.17.so with bytes)
> > >>
> > >>   You can now use it in all perf tools, such as:
> > >>
> > >>           perf record -e probe_libc:malloc -aR sleep 1
> > > 
> > >> Reported-by: Arnaldo Carvalho de Melo <acme@kernel.org>
> > > 
> > > Humm, not working for me, after the patch:
> > > 
> > > [root@ssdandy ~]# perf probe -x /usr/lib64/libc-2.17.so -V malloc
> > > Available variables at malloc
> > >         @<__malloc_check_init+96>
> > >                 (No matched variables)
> > 
> 
> Will try after a 'make build-test' finishes for the current batch

[root@ssdandy ~]# perf probe -vvv -x /usr/lib64/libc-2.17.so -V malloc
probe-definition(0): malloc
symbol:malloc file:(null) line:0 offset:0 return:0 lazy:(null)
0 arguments
Open Debuginfo file: /usr/lib/debug/usr/lib64/libc-2.17.so.debug
Searching variables at malloc
Symbol malloc address found : 800c0
Get 2611 lines from this CU
Probe point found: __malloc_check_init+96
Available variables at malloc
        @<__malloc_check_init+96>
                (No matched variables)
[root@ssdandy ~]#

If I add one more 'v' I get the symtabs as read by symbol-elf.c and this is
what is there for the malloc routines:

[root@ssdandy ~]# grep malloc /tmp/4
probe-definition(0): malloc
symbol:malloc file:(null) line:0 offset:0 return:0 lazy:(null)
Searching variables at malloc
symbol__new: ptmalloc_lock_all 0x7b060-0x7b173
symbol__new: malloc_atfork 0x801c0-0x802dd
symbol__new: ptmalloc_unlock_all2 0x7b180-0x7b21e
symbol__new: ptmalloc_unlock_all 0x7ba90-0x7bb3e
symbol__new: malloc_printerr 0x7bb40-0x7bc28
symbol__new: malloc_consolidate 0x7c3d0-0x7c9be
symbol__new: _int_malloc 0x7dee0-0x7f32b
symbol__new: malloc_check 0x7f330-0x7f44a
symbol__new: ptmalloc_init.part.8 0x80fe0-0x813d7
symbol__new: ptmalloc_init 0x813e0-0x813f5
symbol__new: malloc_hook_ini 0x81400-0x8143c
symbol__new: mallochook 0x82fc0-0x831af
symbol__new: tr_mallochook 0x83fc0-0x8407d
symbol__new: __malloc_set_state 0x814f0-0x819f6
symbol__new: __malloc_usable_size 0x80d30-0x80e0e
symbol__new: __malloc_trim 0x81d50-0x81fac
symbol__new: __GI___libc_malloc 0x800c0-0x801b5
symbol__new: __malloc_get_state 0x802e0-0x804db
symbol__new: __malloc_stats 0x820c0-0x822aa
symbol__new: __malloc_check_init 0x80050-0x800bb
symbol__new: __malloc 0x800c0-0x801b5
symbol__new: malloc_set_state 0x814f0-0x819f6
symbol__new: malloc 0x800c0-0x801b5
symbol__new: malloc_info 0x82320-0x8240f
symbol__new: malloc_trim 0x81d50-0x81fac
symbol__new: malloc_usable_size 0x80d30-0x80e0e
symbol__new: malloc_get_state 0x802e0-0x804db
symbol__new: __libc_malloc 0x800c0-0x801b5
symbol__new: malloc_stats 0x820c0-0x822aa
symbol__new: malloc@plt 0x1f300-0x1f310
Symbol malloc address found : 800c0
Probe point found: __malloc_check_init+96
[root@ssdandy ~]#
 
> > > [root@ssdandy ~]#
> > > 
> > > And then the one asking for 'bytes' to be collectd fails.
> > > 
> > > After processing the other patches I'll try to debug this...
> > > 
> > > [root@ssdandy ~]# cat /etc/redhat-release 
> > > Red Hat Enterprise Linux Server release 7.0 (Maipo)
> > > [root@ssdandy ~]# rpm -q glibc glibc-debuginfo
> > > glibc-2.17-55.el7_0.3.x86_64
> > > glibc-debuginfo-2.17-55.el7_0.1.x86_64
> >                               ^^^ why is this different from the glibc version??
> > 
> > > [root@ssdandy ~]#
> > > [acme@ssdandy linux]$ readelf -Ws /usr/lib64/libc-2.17.so| grep malloc
> > [...]
> > >   4849: 0000000000080050   107 FUNC    LOCAL  DEFAULT   12 __malloc_check_init
> > 
> > __malloc_check_init+96(0x60) becomes 0x80050 + 0x60 = 0x800b0
> > 
> > >   5351: 00000000000800c0   245 FUNC    LOCAL  DEFAULT   12 __malloc
> > >   1170: 00000000000800c0   245 FUNC    GLOBAL DEFAULT   12 malloc@@GLIBC_2.2.5
> > 
> > that is not 0x800c0, so something goes wrong when translating the address
> > to probe point. Could you check the patch 1/4 was applied?
> > Actually there is a bug in the routine which gets the probe point from
> > address. 1/4 fixes it.
> 
> This one?
> 
> commit 0104fe69e0287cf3635657b4c6b26a18e0091697
> Author: Masami Hiramatsu <masami.hiramatsu.pt@hitachi.com>
> Date:   Mon Mar 2 21:49:46 2015 +0900
> 
>     perf probe: Remove bias offset to find probe point by address
>     
>     Remove bias offset to find probe point by address.
> 
> ----------------
> 
> Yes, it is applied.
> 
> - Arnaldo

Re: Re: [PATCH perf/core 2/4] perf-probe: Fix to handle aliased symbols in glibc

[Masami Hiramatsu]

(2015/03/03 12:05), Arnaldo Carvalho de Melo wrote:
> Em Mon, Mar 02, 2015 at 11:45:12PM -0300, Arnaldo Carvalho de Melo escreveu:
>> Em Tue, Mar 03, 2015 at 11:39:02AM +0900, Masami Hiramatsu escreveu:
>>> (2015/03/03 0:46), Arnaldo Carvalho de Melo wrote:
>>>> Em Mon, Mar 02, 2015 at 09:49:53PM +0900, Masami Hiramatsu escreveu:
>>>>> With this patch;
>>>>>   -----
>>>>>   # ./perf probe -x /usr/lib64/libc-2.17.so -V malloc
>>>>>   Available variables at malloc
>>>>>           @<__libc_malloc+0>
>>>>>                   size_t  bytes
>>>>>   # ./perf probe -x /usr/lib64/libc-2.17.so -a "malloc bytes"
>>>>>   Added new event:
>>>>>     probe_libc:malloc    (on malloc in /usr/lib64/libc-2.17.so with bytes)
>>>>>
>>>>>   You can now use it in all perf tools, such as:
>>>>>
>>>>>           perf record -e probe_libc:malloc -aR sleep 1
>>>>
>>>>> Reported-by: Arnaldo Carvalho de Melo <acme@kernel.org>
>>>>
>>>> Humm, not working for me, after the patch:
>>>>
>>>> [root@ssdandy ~]# perf probe -x /usr/lib64/libc-2.17.so -V malloc
>>>> Available variables at malloc
>>>>         @<__malloc_check_init+96>
>>>>                 (No matched variables)
>>>
>>
>> Will try after a 'make build-test' finishes for the current batch
> 

Thank you for checking this,

> [root@ssdandy ~]# perf probe -vvv -x /usr/lib64/libc-2.17.so -V malloc
> probe-definition(0): malloc
> symbol:malloc file:(null) line:0 offset:0 return:0 lazy:(null)
> 0 arguments
> Open Debuginfo file: /usr/lib/debug/usr/lib64/libc-2.17.so.debug
> Searching variables at malloc
> Symbol malloc address found : 800c0
> Get 2611 lines from this CU

Hmm, something wrong at here.

> Probe point found: __malloc_check_init+96

This seems that the debuginfo is a bit odd.
Could you also run eu-addr2line for 0x800c0 and 0x800b0?

I'm using CentOS7 with a bit newer glibc rpms.

# rpm -q glibc glibc-debuginfo
glibc-2.17-55.el7_0.5.x86_64
glibc-debuginfo-2.17-55.el7_0.5.x86_64

And I got following results.

[mhiramat@localhost perf]$ eu-addr2line -fi -e /usr/lib64/libc-2.17.so 0x800c0
__libc_malloc
/usr/src/debug/glibc-2.17-c758a686/malloc/malloc.c:2855
[mhiramat@localhost perf]$ eu-addr2line -fi -e /usr/lib64/libc-2.17.so 0x800b0
__malloc_check_init
/usr/src/debug/glibc-2.17-c758a686/malloc/hooks.c:75

So, 0x800b0 correctly points __malloc_check_init+96 but its address is not 0x800c0.

Thank you,


> Available variables at malloc
>         @<__malloc_check_init+96>
>                 (No matched variables)
> [root@ssdandy ~]#
> 
> If I add one more 'v' I get the symtabs as read by symbol-elf.c and this is
> what is there for the malloc routines:

Yeah, it seems symbol maps works fine. Debuginfo analysis failed.

> 
> [root@ssdandy ~]# grep malloc /tmp/4
> probe-definition(0): malloc
> symbol:malloc file:(null) line:0 offset:0 return:0 lazy:(null)
> Searching variables at malloc
> symbol__new: ptmalloc_lock_all 0x7b060-0x7b173
> symbol__new: malloc_atfork 0x801c0-0x802dd
> symbol__new: ptmalloc_unlock_all2 0x7b180-0x7b21e
> symbol__new: ptmalloc_unlock_all 0x7ba90-0x7bb3e
> symbol__new: malloc_printerr 0x7bb40-0x7bc28
> symbol__new: malloc_consolidate 0x7c3d0-0x7c9be
> symbol__new: _int_malloc 0x7dee0-0x7f32b
> symbol__new: malloc_check 0x7f330-0x7f44a
> symbol__new: ptmalloc_init.part.8 0x80fe0-0x813d7
> symbol__new: ptmalloc_init 0x813e0-0x813f5
> symbol__new: malloc_hook_ini 0x81400-0x8143c
> symbol__new: mallochook 0x82fc0-0x831af
> symbol__new: tr_mallochook 0x83fc0-0x8407d
> symbol__new: __malloc_set_state 0x814f0-0x819f6
> symbol__new: __malloc_usable_size 0x80d30-0x80e0e
> symbol__new: __malloc_trim 0x81d50-0x81fac
> symbol__new: __GI___libc_malloc 0x800c0-0x801b5
> symbol__new: __malloc_get_state 0x802e0-0x804db
> symbol__new: __malloc_stats 0x820c0-0x822aa
> symbol__new: __malloc_check_init 0x80050-0x800bb
> symbol__new: __malloc 0x800c0-0x801b5
> symbol__new: malloc_set_state 0x814f0-0x819f6
> symbol__new: malloc 0x800c0-0x801b5
> symbol__new: malloc_info 0x82320-0x8240f
> symbol__new: malloc_trim 0x81d50-0x81fac
> symbol__new: malloc_usable_size 0x80d30-0x80e0e
> symbol__new: malloc_get_state 0x802e0-0x804db
> symbol__new: __libc_malloc 0x800c0-0x801b5
> symbol__new: malloc_stats 0x820c0-0x822aa
> symbol__new: malloc@plt 0x1f300-0x1f310
> Symbol malloc address found : 800c0
> Probe point found: __malloc_check_init+96
> [root@ssdandy ~]#
>  
>>>> [root@ssdandy ~]#
>>>>
>>>> And then the one asking for 'bytes' to be collectd fails.
>>>>
>>>> After processing the other patches I'll try to debug this...
>>>>
>>>> [root@ssdandy ~]# cat /etc/redhat-release 
>>>> Red Hat Enterprise Linux Server release 7.0 (Maipo)
>>>> [root@ssdandy ~]# rpm -q glibc glibc-debuginfo
>>>> glibc-2.17-55.el7_0.3.x86_64
>>>> glibc-debuginfo-2.17-55.el7_0.1.x86_64
>>>                               ^^^ why is this different from the glibc version??
>>>
>>>> [root@ssdandy ~]#
>>>> [acme@ssdandy linux]$ readelf -Ws /usr/lib64/libc-2.17.so| grep malloc
>>> [...]
>>>>   4849: 0000000000080050   107 FUNC    LOCAL  DEFAULT   12 __malloc_check_init
>>>
>>> __malloc_check_init+96(0x60) becomes 0x80050 + 0x60 = 0x800b0
>>>
>>>>   5351: 00000000000800c0   245 FUNC    LOCAL  DEFAULT   12 __malloc
>>>>   1170: 00000000000800c0   245 FUNC    GLOBAL DEFAULT   12 malloc@@GLIBC_2.2.5
>>>
>>> that is not 0x800c0, so something goes wrong when translating the address
>>> to probe point. Could you check the patch 1/4 was applied?
>>> Actually there is a bug in the routine which gets the probe point from
>>> address. 1/4 fixes it.
>>
>> This one?
>>
>> commit 0104fe69e0287cf3635657b4c6b26a18e0091697
>> Author: Masami Hiramatsu <masami.hiramatsu.pt@hitachi.com>
>> Date:   Mon Mar 2 21:49:46 2015 +0900
>>
>>     perf probe: Remove bias offset to find probe point by address
>>     
>>     Remove bias offset to find probe point by address.
>>
>> ----------------
>>
>> Yes, it is applied.
>>
>> - Arnaldo
> --
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/
> 


-- 
Masami HIRAMATSU
Software Platform Research Dept. Linux Technology Research Center
Hitachi, Ltd., Yokohama Research Laboratory
E-mail: masami.hiramatsu.pt@hitachi.com

Re: Re: [PATCH perf/core 2/4] perf-probe: Fix to handle aliased symbols in glibc

[Arnaldo Carvalho de Melo]

Em Tue, Mar 03, 2015 at 01:11:17PM +0900, Masami Hiramatsu escreveu:
> (2015/03/03 12:05), Arnaldo Carvalho de Melo wrote:
> > Em Mon, Mar 02, 2015 at 11:45:12PM -0300, Arnaldo Carvalho de Melo escreveu:
> >> Em Tue, Mar 03, 2015 at 11:39:02AM +0900, Masami Hiramatsu escreveu:
> >>> (2015/03/03 0:46), Arnaldo Carvalho de Melo wrote:
> >>>> Em Mon, Mar 02, 2015 at 09:49:53PM +0900, Masami Hiramatsu escreveu:
> >>>>> With this patch;
> >>>>>   -----
> >>>>>   # ./perf probe -x /usr/lib64/libc-2.17.so -V malloc
> >>>>>   Available variables at malloc
> >>>>>           @<__libc_malloc+0>
> >>>>>                   size_t  bytes
> >>>>>   # ./perf probe -x /usr/lib64/libc-2.17.so -a "malloc bytes"
> >>>>>   Added new event:
> >>>>>     probe_libc:malloc    (on malloc in /usr/lib64/libc-2.17.so with bytes)
> >>>>>
> >>>>>   You can now use it in all perf tools, such as:
> >>>>>
> >>>>>           perf record -e probe_libc:malloc -aR sleep 1
> >>>>
> >>>>> Reported-by: Arnaldo Carvalho de Melo <acme@kernel.org>
> >>>>
> >>>> Humm, not working for me, after the patch:
> >>>>
> >>>> [root@ssdandy ~]# perf probe -x /usr/lib64/libc-2.17.so -V malloc
> >>>> Available variables at malloc
> >>>>         @<__malloc_check_init+96>
> >>>>                 (No matched variables)
> >>>
> >>
> >> Will try after a 'make build-test' finishes for the current batch
> > 
> 
> Thank you for checking this,
> 
> > [root@ssdandy ~]# perf probe -vvv -x /usr/lib64/libc-2.17.so -V malloc
> > probe-definition(0): malloc
> > symbol:malloc file:(null) line:0 offset:0 return:0 lazy:(null)
> > 0 arguments
> > Open Debuginfo file: /usr/lib/debug/usr/lib64/libc-2.17.so.debug
> > Searching variables at malloc
> > Symbol malloc address found : 800c0
> > Get 2611 lines from this CU
> 
> Hmm, something wrong at here.
> 
> > Probe point found: __malloc_check_init+96
> 
> This seems that the debuginfo is a bit odd.
> Could you also run eu-addr2line for 0x800c0 and 0x800b0?
> 
> I'm using CentOS7 with a bit newer glibc rpms.
> 
> # rpm -q glibc glibc-debuginfo
> glibc-2.17-55.el7_0.5.x86_64
> glibc-debuginfo-2.17-55.el7_0.5.x86_64
> 
> And I got following results.
> 
> [mhiramat@localhost perf]$ eu-addr2line -fi -e /usr/lib64/libc-2.17.so 0x800c0
> __libc_malloc
> /usr/src/debug/glibc-2.17-c758a686/malloc/malloc.c:2855
> [mhiramat@localhost perf]$ eu-addr2line -fi -e /usr/lib64/libc-2.17.so 0x800b0
> __malloc_check_init
> /usr/src/debug/glibc-2.17-c758a686/malloc/hooks.c:75
> 
> So, 0x800b0 correctly points __malloc_check_init+96 but its address is not 0x800c0.

[acme@ssdandy linux]$ eu-addr2line -fi -e /usr/lib64/libc-2.17.so 0x800c0
malloc
??:0
[acme@ssdandy linux]$ eu-addr2line -fi -e /usr/lib64/libc-2.17.so 0x800b0
__malloc_check_init
??:0
[acme@ssdandy linux]$ 

If I do it over the debuginfo files:

[acme@ssdandy linux]$ addr2line -fi -e /usr/lib/debug/usr/lib64/libc-2.17.so.debug  0x800c0
__malloc_check_init
/usr/src/debug/glibc-2.17-c758a686/malloc/hooks.c:75
[acme@ssdandy linux]$ addr2line -fi -e /usr/lib/debug/usr/lib64/libc-2.17.so.debug  0x800b0
__malloc_check_init
/usr/src/debug/glibc-2.17-c758a686/malloc/hooks.c:82
[acme@ssdandy linux]$ 

Tomorrow I'll check for newer packages or any reports about problems
with debuginfo files, please let me know if you need any further info,

Off to bed, zzZzz

- Arnaldo
 
> Thank you,
> 
> 
> > Available variables at malloc
> >         @<__malloc_check_init+96>
> >                 (No matched variables)
> > [root@ssdandy ~]#
> > 
> > If I add one more 'v' I get the symtabs as read by symbol-elf.c and this is
> > what is there for the malloc routines:
> 
> Yeah, it seems symbol maps works fine. Debuginfo analysis failed.
> 
> > 
> > [root@ssdandy ~]# grep malloc /tmp/4
> > probe-definition(0): malloc
> > symbol:malloc file:(null) line:0 offset:0 return:0 lazy:(null)
> > Searching variables at malloc
> > symbol__new: ptmalloc_lock_all 0x7b060-0x7b173
> > symbol__new: malloc_atfork 0x801c0-0x802dd
> > symbol__new: ptmalloc_unlock_all2 0x7b180-0x7b21e
> > symbol__new: ptmalloc_unlock_all 0x7ba90-0x7bb3e
> > symbol__new: malloc_printerr 0x7bb40-0x7bc28
> > symbol__new: malloc_consolidate 0x7c3d0-0x7c9be
> > symbol__new: _int_malloc 0x7dee0-0x7f32b
> > symbol__new: malloc_check 0x7f330-0x7f44a
> > symbol__new: ptmalloc_init.part.8 0x80fe0-0x813d7
> > symbol__new: ptmalloc_init 0x813e0-0x813f5
> > symbol__new: malloc_hook_ini 0x81400-0x8143c
> > symbol__new: mallochook 0x82fc0-0x831af
> > symbol__new: tr_mallochook 0x83fc0-0x8407d
> > symbol__new: __malloc_set_state 0x814f0-0x819f6
> > symbol__new: __malloc_usable_size 0x80d30-0x80e0e
> > symbol__new: __malloc_trim 0x81d50-0x81fac
> > symbol__new: __GI___libc_malloc 0x800c0-0x801b5
> > symbol__new: __malloc_get_state 0x802e0-0x804db
> > symbol__new: __malloc_stats 0x820c0-0x822aa
> > symbol__new: __malloc_check_init 0x80050-0x800bb
> > symbol__new: __malloc 0x800c0-0x801b5
> > symbol__new: malloc_set_state 0x814f0-0x819f6
> > symbol__new: malloc 0x800c0-0x801b5
> > symbol__new: malloc_info 0x82320-0x8240f
> > symbol__new: malloc_trim 0x81d50-0x81fac
> > symbol__new: malloc_usable_size 0x80d30-0x80e0e
> > symbol__new: malloc_get_state 0x802e0-0x804db
> > symbol__new: __libc_malloc 0x800c0-0x801b5
> > symbol__new: malloc_stats 0x820c0-0x822aa
> > symbol__new: malloc@plt 0x1f300-0x1f310
> > Symbol malloc address found : 800c0
> > Probe point found: __malloc_check_init+96
> > [root@ssdandy ~]#
> >  
> >>>> [root@ssdandy ~]#
> >>>>
> >>>> And then the one asking for 'bytes' to be collectd fails.
> >>>>
> >>>> After processing the other patches I'll try to debug this...
> >>>>
> >>>> [root@ssdandy ~]# cat /etc/redhat-release 
> >>>> Red Hat Enterprise Linux Server release 7.0 (Maipo)
> >>>> [root@ssdandy ~]# rpm -q glibc glibc-debuginfo
> >>>> glibc-2.17-55.el7_0.3.x86_64
> >>>> glibc-debuginfo-2.17-55.el7_0.1.x86_64
> >>>                               ^^^ why is this different from the glibc version??
> >>>
> >>>> [root@ssdandy ~]#
> >>>> [acme@ssdandy linux]$ readelf -Ws /usr/lib64/libc-2.17.so| grep malloc
> >>> [...]
> >>>>   4849: 0000000000080050   107 FUNC    LOCAL  DEFAULT   12 __malloc_check_init
> >>>
> >>> __malloc_check_init+96(0x60) becomes 0x80050 + 0x60 = 0x800b0
> >>>
> >>>>   5351: 00000000000800c0   245 FUNC    LOCAL  DEFAULT   12 __malloc
> >>>>   1170: 00000000000800c0   245 FUNC    GLOBAL DEFAULT   12 malloc@@GLIBC_2.2.5
> >>>
> >>> that is not 0x800c0, so something goes wrong when translating the address
> >>> to probe point. Could you check the patch 1/4 was applied?
> >>> Actually there is a bug in the routine which gets the probe point from
> >>> address. 1/4 fixes it.
> >>
> >> This one?
> >>
> >> commit 0104fe69e0287cf3635657b4c6b26a18e0091697
> >> Author: Masami Hiramatsu <masami.hiramatsu.pt@hitachi.com>
> >> Date:   Mon Mar 2 21:49:46 2015 +0900
> >>
> >>     perf probe: Remove bias offset to find probe point by address
> >>     
> >>     Remove bias offset to find probe point by address.
> >>
> >> ----------------
> >>
> >> Yes, it is applied.
> >>
> >> - Arnaldo
> > --
> > To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> > the body of a message to majordomo@vger.kernel.org
> > More majordomo info at  http://vger.kernel.org/majordomo-info.html
> > Please read the FAQ at  http://www.tux.org/lkml/
> > 
> 
> 
> -- 
> Masami HIRAMATSU
> Software Platform Research Dept. Linux Technology Research Center
> Hitachi, Ltd., Yokohama Research Laboratory
> E-mail: masami.hiramatsu.pt@hitachi.com
> 

Re: [PATCH perf/core 2/4] perf-probe: Fix to handle aliased symbols in glibc

[Namhyung Kim]

Hi Masami,

On Mon, Mar 02, 2015 at 09:49:53PM +0900, Masami Hiramatsu wrote:
> Fix perf probe to handle aliased symbols correctly in glibc.
> In the glibc, several symbols are defined as an alias of
> __libc_XXX, e.g. malloc is an alias of __libc_malloc.
> In such cases, dwarf has no subroutine instances of the
> alias functions (e.g. no "malloc" instance), but the map
> has that symbol and its address.
> Thus, if we search the alieased symbol in debuginfo, we
> always fail to find it, but it is in the map.
> 
> To solve this problem, this fails back to address-based
> alternative search, which searches the symbol in the map,
> translates its address to alternative (correct) function
> name by using debuginfo, and retry to find the alternative
> function point from debuginfo.
> 
> This adds fail-back process to --vars, --lines and --add
> options. So, now you can use those on malloc@libc :)

So this is only for binaries that have debuginfo, right?

I have a similar issue with no debuginfo.

  $ perf probe -x /usr/lib/libc.so.6 -V calloc
  The /usr/lib/libc-2.21.so file has no debug information.
  Rebuild with -g, or install an appropriate debuginfo package.
    Error: Failed to show vars.


But it also failed to add a probe to calloc:

  $ perf probe -x /usr/lib/libc.so.6 -a calloc
  Failed to find symbol calloc in /usr/lib/libc-2.21.so
    Error: Failed to add events.
    

Of course there's calloc in the libc binary.

  $ nm /usr/lib/libc.so.6 | grep calloc
  000000000007b1f0 t __calloc
  000000000007b1f0 T __libc_calloc
  000000000007b1f0 W calloc


I think the problem is that calloc is a weak symbol so it'll be
discarded during the symbol loading.  It's because to avoid multiple
symbols (or aliases) at a same address so we choose a better symbol
using heuristics.  But for this case I think we can allow aliases
since it's used only for finding probe points.

Thanks,
Namhyung

Re: Re: [PATCH perf/core 2/4] perf-probe: Fix to handle aliased symbols in glibc

[Masami Hiramatsu]

(2015/03/03 22:31), Namhyung Kim wrote:
> Hi Masami,
> 
> On Mon, Mar 02, 2015 at 09:49:53PM +0900, Masami Hiramatsu wrote:
>> Fix perf probe to handle aliased symbols correctly in glibc.
>> In the glibc, several symbols are defined as an alias of
>> __libc_XXX, e.g. malloc is an alias of __libc_malloc.
>> In such cases, dwarf has no subroutine instances of the
>> alias functions (e.g. no "malloc" instance), but the map
>> has that symbol and its address.
>> Thus, if we search the alieased symbol in debuginfo, we
>> always fail to find it, but it is in the map.
>>
>> To solve this problem, this fails back to address-based
>> alternative search, which searches the symbol in the map,
>> translates its address to alternative (correct) function
>> name by using debuginfo, and retry to find the alternative
>> function point from debuginfo.
>>
>> This adds fail-back process to --vars, --lines and --add
>> options. So, now you can use those on malloc@libc :)
> 
> So this is only for binaries that have debuginfo, right?
> 
> I have a similar issue with no debuginfo.
> 
>   $ perf probe -x /usr/lib/libc.so.6 -V calloc
>   The /usr/lib/libc-2.21.so file has no debug information.
>   Rebuild with -g, or install an appropriate debuginfo package.
>     Error: Failed to show vars.
> 
> 
> But it also failed to add a probe to calloc:
> 
>   $ perf probe -x /usr/lib/libc.so.6 -a calloc
>   Failed to find symbol calloc in /usr/lib/libc-2.21.so
>     Error: Failed to add events.
>     
> 
> Of course there's calloc in the libc binary.
> 
>   $ nm /usr/lib/libc.so.6 | grep calloc
>   000000000007b1f0 t __calloc
>   000000000007b1f0 T __libc_calloc
>   000000000007b1f0 W calloc
> 
> 
> I think the problem is that calloc is a weak symbol so it'll be
> discarded during the symbol loading.

Right, I also hit same problem on waitpid, which is also an weak symbol.

>  It's because to avoid multiple
> symbols (or aliases) at a same address so we choose a better symbol
> using heuristics.  But for this case I think we can allow aliases
> since it's used only for finding probe points.

I'm just using struct map and dso, so those should be improved.

Thank you,


-- 
Masami HIRAMATSU
Software Platform Research Dept. Linux Technology Research Center
Hitachi, Ltd., Yokohama Research Laboratory
E-mail: masami.hiramatsu.pt@hitachi.com

[PATCH perf/core 3/4] perf-probe: Fix --line to handle aliased symbols in glibc

[Masami Hiramatsu]

Fix perf probe --line to handle aliased symbols correctly
in glibc.

This makes line_range search failing back to address-based
alternative search as same as --add and --vars.

Without this patch;
  -----
  # ./perf probe -x /usr/lib64/libc-2.17.so -L malloc
  Specified source line is not found.
    Error: Failed to show lines.
  -----

With this patch;
  -----
  # ./perf probe -x /usr/lib64/libc-2.17.so -L malloc
  <__libc_malloc@/usr/src/debug/glibc-2.17-c758a686/malloc/malloc.c:0>
        0  __libc_malloc(size_t bytes)
        1  {
             mstate ar_ptr;
             void *victim;

             __malloc_ptr_t (*hook) (size_t, const __malloc_ptr_t)
        6      = force_reg (__malloc_hook);
        7    if (__builtin_expect (hook != NULL, 0))
        8      return (*hook)(bytes, RETURN_ADDRESS (0));

       10    arena_lookup(ar_ptr);

       12    arena_lock(ar_ptr, bytes);
  -----

Note that this actually shows __libc_malloc, since it is
the real instance of malloc. User can use both __libc_malloc
and malloc for --line.

Signed-off-by: Masami Hiramatsu <masami.hiramatsu.pt@hitachi.com>
---
 tools/perf/util/probe-event.c |   35 +++++++++++++++++++++++++++++++++--
 1 file changed, 33 insertions(+), 2 deletions(-)

diff --git a/tools/perf/util/probe-event.c b/tools/perf/util/probe-event.c
index b8f4578..4cfd121 100644
--- a/tools/perf/util/probe-event.c
+++ b/tools/perf/util/probe-event.c
@@ -353,6 +353,31 @@ static int get_alternative_probe_event(struct debuginfo *dinfo,
 	return ret;
 }
 
+static int get_alternative_line_range(struct debuginfo *dinfo,
+				      struct line_range *lr,
+				      const char *target, bool user)
+{
+	struct perf_probe_point pp = { 0 }, result = { 0 };
+	int ret, len = 0;
+
+	pp.function = lr->function;
+	pp.file = lr->file;
+	pp.line = lr->start;
+	if (lr->end != INT_MAX)
+		len = lr->end - lr->start;
+	ret = find_alternative_probe_point(dinfo, &pp, &result,
+					   target, user);
+	if (!ret) {
+		lr->function = result.function;
+		lr->file = result.file;
+		lr->start = result.line;
+		if (lr->end != INT_MAX)
+			lr->end = lr->start + len;
+		clear_perf_probe_point(&pp);
+	}
+	return ret;
+}
+
 /* Open new debuginfo of given module */
 static struct debuginfo *open_debuginfo(const char *module, bool silent)
 {
@@ -734,7 +759,8 @@ static int _show_one_line(FILE *fp, int l, bool skip, bool show_num)
  * Show line-range always requires debuginfo to find source file and
  * line number.
  */
-static int __show_line_range(struct line_range *lr, const char *module)
+static int __show_line_range(struct line_range *lr, const char *module,
+			     bool user)
 {
 	int l = 1;
 	struct int_node *ln;
@@ -750,6 +776,11 @@ static int __show_line_range(struct line_range *lr, const char *module)
 		return -ENOENT;
 
 	ret = debuginfo__find_line_range(dinfo, lr);
+	if (!ret) {	/* Not found, retry with an alternative */
+		ret = get_alternative_line_range(dinfo, lr, module, user);
+		if (!ret)
+			ret = debuginfo__find_line_range(dinfo, lr);
+	}
 	debuginfo__delete(dinfo);
 	if (ret == 0 || ret == -ENOENT) {
 		pr_warning("Specified source line is not found.\n");
@@ -819,7 +850,7 @@ int show_line_range(struct line_range *lr, const char *module, bool user)
 	ret = init_symbol_maps(user);
 	if (ret < 0)
 		return ret;
-	ret = __show_line_range(lr, module);
+	ret = __show_line_range(lr, module, user);
 	exit_symbol_maps();
 
 	return ret;

[PATCH perf/core 4/4] Revert "perf probe: Fix to fall back to find probe point in symbols"

[Masami Hiramatsu]

This reverts commit 906451b98b67 ("perf probe: Fix to fall back to find probe point in symbols").

Since perf-probe retries with the address of given symbol
searched from map before this path, this fall back routine
doesn't need anymore.

Signed-off-by: Masami Hiramatsu <masami.hiramatsu.pt@hitachi.com>
---
 tools/perf/util/probe-event.c |    6 ++----
 1 file changed, 2 insertions(+), 4 deletions(-)

diff --git a/tools/perf/util/probe-event.c b/tools/perf/util/probe-event.c
index 4cfd121..c379ea0 100644
--- a/tools/perf/util/probe-event.c
+++ b/tools/perf/util/probe-event.c
@@ -630,11 +630,9 @@ static int try_to_find_probe_trace_events(struct perf_probe_event *pev,
 	}
 
 	if (ntevs == 0)	{	/* No error but failed to find probe point. */
-		pr_warning("Probe point '%s' not found in debuginfo.\n",
+		pr_warning("Probe point '%s' not found.\n",
 			   synthesize_perf_probe_point(&pev->point));
-		if (need_dwarf)
-			return -ENOENT;
-		return 0;
+		return -ENOENT;
 	}
 	/* Error path : ntevs < 0 */
 	pr_debug("An error occurred in debuginfo analysis (%d).\n", ntevs);