[PATCH 1/3] perf probe: Set retprobe flag when probe in address-based alternative mode

[PATCH 1/3] perf probe: Set retprobe flag when probe in address-based alternative mode

[He Kuang]

Perf probe misses to set retprobe flag back when falling back to
address-based alternative mode.

Can be reproduced as following:

  $ perf probe -v -k vmlinux --add='sys_write%return'
  ...
  Added new event:
  Writing event: p:probe/sys_write _stext+1584952
    probe:sys_write      (on sys_write%return)

  $ cat /sys/kernel/debug/tracing/kprobe_events
  p:probe/sys_write _stext+1584952

After this patch:

  $ perf probe -v -k vmlinux --add='sys_write%return'
  Added new event:
  Writing event: r:probe/sys_write SyS_write+0
    probe:sys_write      (on sys_write%return)

  $ cat /sys/kernel/debug/tracing/kprobe_events
  r:probe/sys_write SyS_write

Signed-off-by: He Kuang <hekuang@huawei.com>
---
 tools/perf/util/probe-event.c | 1 +
 1 file changed, 1 insertion(+)

diff --git a/tools/perf/util/probe-event.c b/tools/perf/util/probe-event.c
index 30545ce..5483d98 100644
--- a/tools/perf/util/probe-event.c
+++ b/tools/perf/util/probe-event.c
@@ -332,6 +332,7 @@ static int find_alternative_probe_point(struct debuginfo *dinfo,
 	else {
 		result->offset += pp->offset;
 		result->line += pp->line;
+		result->retprobe = pp->retprobe;
 		ret = 0;
 	}
 
-- 
2.3.3.220.g9ab698f

[PATCH 2/3] perf probe: Make --source avaiable when probe with lazy_line

[He Kuang]

Use get_real_path() to enable --source option when probe with lazy_line
pattern.

Before this patch:

  $ perf probe -s ./kernel_src/ -k ./vmlinux --add='fs/super.c;s->s_count=1;'
  Failed to open fs/super.c: No such file or directory
    Error: Failed to add events.

After this patch:

  $ perf probe -s ./kernel_src/ -k ./vmlinux  --add='fs/super.c;s->s_count=1;'
  Added new events:
    probe:_stext         (on @fs/super.c)
    probe:_stext_1       (on @fs/super.c)
  ...

Signed-off-by: He Kuang <hekuang@huawei.com>
---
 tools/perf/util/probe-event.c  |  2 +-
 tools/perf/util/probe-event.h  |  2 ++
 tools/perf/util/probe-finder.c | 18 +++++++++++++++---
 3 files changed, 18 insertions(+), 4 deletions(-)

diff --git a/tools/perf/util/probe-event.c b/tools/perf/util/probe-event.c
index 5483d98..35ee51a 100644
--- a/tools/perf/util/probe-event.c
+++ b/tools/perf/util/probe-event.c
@@ -661,7 +661,7 @@ static int try_to_find_probe_trace_events(struct perf_probe_event *pev,
  * a newly allocated path on success.
  * Return 0 if file was found and readable, -errno otherwise.
  */
-static int get_real_path(const char *raw_path, const char *comp_dir,
+int get_real_path(const char *raw_path, const char *comp_dir,
 			 char **new_path)
 {
 	const char *prefix = symbol_conf.source_prefix;
diff --git a/tools/perf/util/probe-event.h b/tools/perf/util/probe-event.h
index d6b7834..21809ea 100644
--- a/tools/perf/util/probe-event.h
+++ b/tools/perf/util/probe-event.h
@@ -135,6 +135,8 @@ extern int show_available_vars(struct perf_probe_event *pevs, int npevs,
 			       struct strfilter *filter, bool externs);
 extern int show_available_funcs(const char *module, struct strfilter *filter,
 				bool user);
+extern int get_real_path(const char *raw_path, const char *comp_dir,
+			char **new_path);
 
 /* Maximum index number of event-name postfix */
 #define MAX_EVENT_INDEX	1024
diff --git a/tools/perf/util/probe-finder.c b/tools/perf/util/probe-finder.c
index 7831e2d..431c12d 100644
--- a/tools/perf/util/probe-finder.c
+++ b/tools/perf/util/probe-finder.c
@@ -791,11 +791,20 @@ static int find_lazy_match_lines(struct intlist *list,
 	ssize_t len;
 	int count = 0, linenum = 1;
 	char sbuf[STRERR_BUFSIZE];
+	char *realname = NULL;
+	int ret;
 
-	fp = fopen(fname, "r");
+	ret = get_real_path(fname, NULL, &realname);
+	if (ret < 0) {
+		pr_warning("Failed to find source file %s.\n", fname);
+		return ret;
+	}
+
+	fp = fopen(realname, "r");
 	if (!fp) {
-		pr_warning("Failed to open %s: %s\n", fname,
+		pr_warning("Failed to open %s: %s\n", realname,
 			   strerror_r(errno, sbuf, sizeof(sbuf)));
+		free(realname);
 		return -errno;
 	}
 
@@ -817,7 +826,10 @@ static int find_lazy_match_lines(struct intlist *list,
 	fclose(fp);
 
 	if (count == 0)
-		pr_debug("No matched lines found in %s.\n", fname);
+		pr_debug("No matched lines found in %s.\n", realname);
+
+	free(realname);
+
 	return count;
 }
 
-- 
2.3.3.220.g9ab698f

[PATCH 3/3] perf probe: Fix segfault when probe with lazy_line to file

[He Kuang]

The first argument passed to find_probe_point_lazy() should be CU die,
which will be passed to die_walk_lines() when lazy_line
matches. Currently, when we probe with lazy_line pattern to file without
function name, NULL pointer is passed and causes a segment fault.

Can be repoduced as following:

  $ perf probe -k vmlinux --add='fs/super.c;s->s_count=1;'
  [ 1958.984658] perf[1020]: segfault at 10 ip 00007fc6e10d8c71 sp
  00007ffcbfaaf900 error 4 in libdw-0.161.so[7fc6e10ce000+34000]
  Segmentation fault

After this patch:

  $ perf probe -k vmlinux --add='fs/super.c;s->s_count=1;'
  Added new event:
  probe:_stext         (on @fs/super.c)

  You can now use it in all perf tools, such as:
    perf record -e probe:_stext -aR sleep 1

Signed-off-by: He Kuang <hekuang@huawei.com>
---
 tools/perf/util/probe-finder.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/tools/perf/util/probe-finder.c b/tools/perf/util/probe-finder.c
index 431c12d..e91101b 100644
--- a/tools/perf/util/probe-finder.c
+++ b/tools/perf/util/probe-finder.c
@@ -1067,7 +1067,7 @@ static int debuginfo__find_probes(struct debuginfo *dbg,
 			if (pp->function)
 				ret = find_probe_point_by_func(pf);
 			else if (pp->lazy_line)
-				ret = find_probe_point_lazy(NULL, pf);
+				ret = find_probe_point_lazy(&pf->cu_die, pf);
 			else {
 				pf->lno = pp->line;
 				ret = find_probe_point_by_line(pf);
-- 
2.3.3.220.g9ab698f

Re: [PATCH 1/3] perf probe: Set retprobe flag when probe in address-based alternative mode

[Arnaldo Carvalho de Melo]

Em Mon, Apr 13, 2015 at 07:41:28PM +0800, He Kuang escreveu:
> Perf probe misses to set retprobe flag back when falling back to
> address-based alternative mode.

And when is that happens, can you explain? Because I tried to follow your
instructions, but in my case it worked in the same way before and after your
patch.

Can you please provide more context for reviewing your patch?

Here are my results:

Before:

  [root@ssdandy ~]# perf probe -v --add='sys_write%return'
  probe-definition(0): sys_write%return
  symbol:sys_write file:(null) line:0 offset:0 return:1 lazy:(null)
  0 arguments
  Using /root/.debug/.build-id/dd/32e51921ede0fd46f034091b7f6a0f2e01ebda for symbols
  Could not open debuginfo. Try to use symbols.
  Opening /sys/kernel/debug/tracing/kprobe_events write=1
  Added new event:
  Writing event: r:probe/sys_write sys_write+0
      probe:sys_write      (on sys_write%return)

  You can now use it in all perf tools, such as:

	  perf record -e probe:sys_write -aR sleep 1
  [root@ssdandy ~]# cat /sys/kernel/debug/tracing/kprobe_events
  r:probe/sys_write sys_write

Remove it:

  [root@ssdandy ~]# perf probe --del *:*
  Removed event: probe:sys_write

After:

  [root@ssdandy ~]# perf probe -v --add='sys_write%return'
  probe-definition(0): sys_write%return
  symbol:sys_write file:(null) line:0 offset:0 return:1 lazy:(null)
  0 arguments
  Using /root/.debug/.build-id/dd/32e51921ede0fd46f034091b7f6a0f2e01ebda for symbols
  Could not open debuginfo. Try to use symbols.
  Opening /sys/kernel/debug/tracing/kprobe_events write=1
  Added new event:
  Writing event: r:probe/sys_write sys_write+0
    probe:sys_write      (on sys_write%return)
  
  You can now use it in all perf tools, such as:

	  perf record -e probe:sys_write -aR sleep 1

  [root@ssdandy ~]# 
  [root@ssdandy ~]# cat /sys/kernel/debug/tracing/kprobe_events
  r:probe/sys_write sys_write

Humm, noticed one other problem, but not with your patch, about this message:

"Could not open debuginfo. Try to use symbols."

That is really not clear, specially that "try to use symbols" :-)

[root@ssdandy ~]# ls -la /root/.debug/.build-id/dd/32e51921ede0fd46f034091b7f6a0f2e01ebda
lrwxrwxrwx. 1 root root 86 Apr 10 18:02 /root/.debug/.build-id/dd/32e51921ede0fd46f034091b7f6a0f2e01ebda -> ../../home/acme/git/build/v4.0.0-rc6+/vmlinux/dd32e51921ede0fd46f034091b7f6a0f2e01ebda
[root@ssdandy ~]# ls -la /root/.debug/.build-id/dd/../../home/acme/git/build/v4.0.0-rc6+/vmlinux/dd32e51921ede0fd46f034091b7f6a0f2e01ebda 
-rwxr-xr-x. 1 root root 22698661 Apr 10 18:02 /root/.debug/.build-id/dd/../../home/acme/git/build/v4.0.0-rc6+/vmlinux/dd32e51921ede0fd46f034091b7f6a0f2e01ebda
[root@ssdandy ~]# ls -la /root/.debug/.build-id/dd/../../home/acme/git/build/v4.0.0-rc6+/vmlinux/dd32e51921ede0fd46f034091b7f6a0f2e01ebda

I.e. it managed to read the debuginfo, its just that it has no symbols in it :-)

Anyway, digression ended.

- Arnaldo
 
> Can be reproduced as following:
> 
>   $ perf probe -v -k vmlinux --add='sys_write%return'
>   ...
>   Added new event:
>   Writing event: p:probe/sys_write _stext+1584952
>     probe:sys_write      (on sys_write%return)
> 
>   $ cat /sys/kernel/debug/tracing/kprobe_events
>   p:probe/sys_write _stext+1584952
> 
> After this patch:
> 
>   $ perf probe -v -k vmlinux --add='sys_write%return'
>   Added new event:
>   Writing event: r:probe/sys_write SyS_write+0
>     probe:sys_write      (on sys_write%return)
> 
>   $ cat /sys/kernel/debug/tracing/kprobe_events
>   r:probe/sys_write SyS_write
> 
> Signed-off-by: He Kuang <hekuang@huawei.com>
> ---
>  tools/perf/util/probe-event.c | 1 +
>  1 file changed, 1 insertion(+)
> 
> diff --git a/tools/perf/util/probe-event.c b/tools/perf/util/probe-event.c
> index 30545ce..5483d98 100644
> --- a/tools/perf/util/probe-event.c
> +++ b/tools/perf/util/probe-event.c
> @@ -332,6 +332,7 @@ static int find_alternative_probe_point(struct debuginfo *dinfo,
>  	else {
>  		result->offset += pp->offset;
>  		result->line += pp->line;
> +		result->retprobe = pp->retprobe;
>  		ret = 0;
>  	}
>  
> -- 
> 2.3.3.220.g9ab698f

Re: [PATCH 1/3] perf probe: Set retprobe flag when probe in address-based alternative mode

[Arnaldo Carvalho de Melo]

Em Mon, Apr 13, 2015 at 11:39:03AM -0300, Arnaldo Carvalho de Melo escreveu:
> Em Mon, Apr 13, 2015 at 07:41:28PM +0800, He Kuang escreveu:
> > Perf probe misses to set retprobe flag back when falling back to
> > address-based alternative mode.
 
> Humm, noticed one other problem, but not with your patch, about this message:
 
> "Could not open debuginfo. Try to use symbols."
 
> That is really not clear, specially that "try to use symbols" :-)
 
> [root@ssdandy ~]# ls -la /root/.debug/.build-id/dd/32e51921ede0fd46f034091b7f6a0f2e01ebda
> lrwxrwxrwx. 1 root root 86 Apr 10 18:02 /root/.debug/.build-id/dd/32e51921ede0fd46f034091b7f6a0f2e01ebda -> ../../home/acme/git/build/v4.0.0-rc6+/vmlinux/dd32e51921ede0fd46f034091b7f6a0f2e01ebda
> [root@ssdandy ~]# ls -la /root/.debug/.build-id/dd/../../home/acme/git/build/v4.0.0-rc6+/vmlinux/dd32e51921ede0fd46f034091b7f6a0f2e01ebda 
> -rwxr-xr-x. 1 root root 22698661 Apr 10 18:02 /root/.debug/.build-id/dd/../../home/acme/git/build/v4.0.0-rc6+/vmlinux/dd32e51921ede0fd46f034091b7f6a0f2e01ebda
> [root@ssdandy ~]# ls -la /root/.debug/.build-id/dd/../../home/acme/git/build/v4.0.0-rc6+/vmlinux/dd32e51921ede0fd46f034091b7f6a0f2e01ebda
 
> I.e. it managed to read the debuginfo, its just that it has no symbols in it :-)
 
> Anyway, digression ended.

Interesting is that when testing your next patch I see:

  [root@ssdandy linux]#  perf probe -s ./kernel_src/ --add='fs/super.c;s->s_count=1;'
  The /root/.debug/.build-id/dd/32e51921ede0fd46f034091b7f6a0f2e01ebda file has no debug information.
  Rebuild with CONFIG_DEBUG_INFO=y, or install an appropriate debuginfo package.
    Error: Failed to add events.
  [root@ssdandy linux]# 

Much, much clear message about that debuginfo file :-)

But then, to test your [2/3] patch I'll have to figure out how to setup the
environment so that I can match your results, shouldn't be hard, but would
save reviewing time if you stated it in the commit log message.

- Arnaldo

Re: [PATCH 1/3] perf probe: Set retprobe flag when probe in address-based alternative mode

[He Kuang]

On 04/13/2015 10:39 PM, Arnaldo Carvalho de Melo wrote:
> Em Mon, Apr 13, 2015 at 07:41:28PM +0800, He Kuang escreveu:
>> Perf probe misses to set retprobe flag back when falling back to
>> address-based alternative mode.
>
> And when is that happens, can you explain? Because I tried to follow your
> instructions, but in my case it worked in the same way before and after your
> patch.
>
> Can you please provide more context for reviewing your patch?
>

More details:

   [root]# perf probe -v -k vmlinux --add='sys_write%return'
   probe-definition(0): sys_write%return
   symbol:sys_write file:(null) line:0 offset:0 return:1 lazy:(null)
   0 arguments
   Use vmlinux: vmlinux
   map_groups__set_modules_path_dir: cannot open /lib/modules/4.0.0-rc6+ dir
   Problems setting modules path maps, continuing anyway...
   Using vmlinux for symbols

   Open Debuginfo file: vmlinux
   >>>==========================

   Try to find probe point from debuginfo.
   Symbol sys_write address found : ffffffff811a8cf0
   Probe point found: SyS_write+0
   Found 1 probe_trace_events.
   Opening /sys/kernel/debug/tracing/kprobe_events write=1
   Added new event:
   Writing event: p:probe/sys_write _stext+1739560
     probe:sys_write      (on sys_write%return)

   You can now use it in all perf tools, such as:

           perf record -e probe:sys_write -aR sleep 1

   [root@buildroot tmp]# cat /sys/kernel/debug/tracing/kprobe_events
   p:probe/sys_write _stext+1739560


The line I marked is different from your result, which is:
   """Could not open debuginfo. Try to use symbols."""


When perf probe searched in debuginfo and failed, it tried with 
alternative, in function get_alternative_probe_event():

         memcpy(tmp, &pev->point, sizeof(*tmp));
	memset(&pev->point, 0, sizeof(pev->point));

In this case, it drops the retprobe flag and forgets to set it back in 
find_alternative_probe_point(), so the problem occurred.



> Here are my results:
>
> Before:
>
>    [root@ssdandy ~]# perf probe -v --add='sys_write%return'
>    probe-definition(0): sys_write%return
>    symbol:sys_write file:(null) line:0 offset:0 return:1 lazy:(null)
>    0 arguments
>    Using /root/.debug/.build-id/dd/32e51921ede0fd46f034091b7f6a0f2e01ebda for symbols
>    Could not open debuginfo. Try to use symbols.
>    Opening /sys/kernel/debug/tracing/kprobe_events write=1
>    Added new event:
>    Writing event: r:probe/sys_write sys_write+0
>        probe:sys_write      (on sys_write%return)
>
>    You can now use it in all perf tools, such as:
>
> 	  perf record -e probe:sys_write -aR sleep 1
>    [root@ssdandy ~]# cat /sys/kernel/debug/tracing/kprobe_events
>    r:probe/sys_write sys_write
>
> Remove it:
>
>    [root@ssdandy ~]# perf probe --del *:*
>    Removed event: probe:sys_write
>
> After:
>
>    [root@ssdandy ~]# perf probe -v --add='sys_write%return'
>    probe-definition(0): sys_write%return
>    symbol:sys_write file:(null) line:0 offset:0 return:1 lazy:(null)
>    0 arguments
>    Using /root/.debug/.build-id/dd/32e51921ede0fd46f034091b7f6a0f2e01ebda for symbols
>    Could not open debuginfo. Try to use symbols.
>    Opening /sys/kernel/debug/tracing/kprobe_events write=1
>    Added new event:
>    Writing event: r:probe/sys_write sys_write+0
>      probe:sys_write      (on sys_write%return)
>
>    You can now use it in all perf tools, such as:
>
> 	  perf record -e probe:sys_write -aR sleep 1
>
>    [root@ssdandy ~]#
>    [root@ssdandy ~]# cat /sys/kernel/debug/tracing/kprobe_events
>    r:probe/sys_write sys_write
>
> Humm, noticed one other problem, but not with your patch, about this message:
>
> "Could not open debuginfo. Try to use symbols."
>
> That is really not clear, specially that "try to use symbols" :-)
>
> [root@ssdandy ~]# ls -la /root/.debug/.build-id/dd/32e51921ede0fd46f034091b7f6a0f2e01ebda
> lrwxrwxrwx. 1 root root 86 Apr 10 18:02 /root/.debug/.build-id/dd/32e51921ede0fd46f034091b7f6a0f2e01ebda -> ../../home/acme/git/build/v4.0.0-rc6+/vmlinux/dd32e51921ede0fd46f034091b7f6a0f2e01ebda
> [root@ssdandy ~]# ls -la /root/.debug/.build-id/dd/../../home/acme/git/build/v4.0.0-rc6+/vmlinux/dd32e51921ede0fd46f034091b7f6a0f2e01ebda
> -rwxr-xr-x. 1 root root 22698661 Apr 10 18:02 /root/.debug/.build-id/dd/../../home/acme/git/build/v4.0.0-rc6+/vmlinux/dd32e51921ede0fd46f034091b7f6a0f2e01ebda
> [root@ssdandy ~]# ls -la /root/.debug/.build-id/dd/../../home/acme/git/build/v4.0.0-rc6+/vmlinux/dd32e51921ede0fd46f034091b7f6a0f2e01ebda
>
> I.e. it managed to read the debuginfo, its just that it has no symbols in it :-)
>
> Anyway, digression ended.
>
> - Arnaldo
>
>> Can be reproduced as following:
>>
>>    $ perf probe -v -k vmlinux --add='sys_write%return'
>>    ...
>>    Added new event:
>>    Writing event: p:probe/sys_write _stext+1584952
>>      probe:sys_write      (on sys_write%return)
>>
>>    $ cat /sys/kernel/debug/tracing/kprobe_events
>>    p:probe/sys_write _stext+1584952
>>
>> After this patch:
>>
>>    $ perf probe -v -k vmlinux --add='sys_write%return'
>>    Added new event:
>>    Writing event: r:probe/sys_write SyS_write+0
>>      probe:sys_write      (on sys_write%return)
>>
>>    $ cat /sys/kernel/debug/tracing/kprobe_events
>>    r:probe/sys_write SyS_write
>>
>> Signed-off-by: He Kuang <hekuang@huawei.com>
>> ---
>>   tools/perf/util/probe-event.c | 1 +
>>   1 file changed, 1 insertion(+)
>>
>> diff --git a/tools/perf/util/probe-event.c b/tools/perf/util/probe-event.c
>> index 30545ce..5483d98 100644
>> --- a/tools/perf/util/probe-event.c
>> +++ b/tools/perf/util/probe-event.c
>> @@ -332,6 +332,7 @@ static int find_alternative_probe_point(struct debuginfo *dinfo,
>>   	else {
>>   		result->offset += pp->offset;
>>   		result->line += pp->line;
>> +		result->retprobe = pp->retprobe;
>>   		ret = 0;
>>   	}
>>
>> --
>> 2.3.3.220.g9ab698f
> --
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/
>

Re: [PATCH 1/3] perf probe: Set retprobe flag when probe in address-based alternative mode

[He Kuang]

Hi, Arnaldo

On 04/13/2015 10:42 PM, Arnaldo Carvalho de Melo wrote:
> Em Mon, Apr 13, 2015 at 11:39:03AM -0300, Arnaldo Carvalho de Melo escreveu:
>> Em Mon, Apr 13, 2015 at 07:41:28PM +0800, He Kuang escreveu:
>>> Perf probe misses to set retprobe flag back when falling back to
>>> address-based alternative mode.
>
>> Humm, noticed one other problem, but not with your patch, about this message:
>
>> "Could not open debuginfo. Try to use symbols."
>
>> That is really not clear, specially that "try to use symbols" :-)
>
>> [root@ssdandy ~]# ls -la /root/.debug/.build-id/dd/32e51921ede0fd46f034091b7f6a0f2e01ebda
>> lrwxrwxrwx. 1 root root 86 Apr 10 18:02 /root/.debug/.build-id/dd/32e51921ede0fd46f034091b7f6a0f2e01ebda -> ../../home/acme/git/build/v4.0.0-rc6+/vmlinux/dd32e51921ede0fd46f034091b7f6a0f2e01ebda
>> [root@ssdandy ~]# ls -la /root/.debug/.build-id/dd/../../home/acme/git/build/v4.0.0-rc6+/vmlinux/dd32e51921ede0fd46f034091b7f6a0f2e01ebda
>> -rwxr-xr-x. 1 root root 22698661 Apr 10 18:02 /root/.debug/.build-id/dd/../../home/acme/git/build/v4.0.0-rc6+/vmlinux/dd32e51921ede0fd46f034091b7f6a0f2e01ebda
>> [root@ssdandy ~]# ls -la /root/.debug/.build-id/dd/../../home/acme/git/build/v4.0.0-rc6+/vmlinux/dd32e51921ede0fd46f034091b7f6a0f2e01ebda
>
>> I.e. it managed to read the debuginfo, its just that it has no symbols in it :-)
>
>> Anyway, digression ended.
>
> Interesting is that when testing your next patch I see:
>
>    [root@ssdandy linux]#  perf probe -s ./kernel_src/ --add='fs/super.c;s->s_count=1;'
>    The /root/.debug/.build-id/dd/32e51921ede0fd46f034091b7f6a0f2e01ebda file has no debug information.
>    Rebuild with CONFIG_DEBUG_INFO=y, or install an appropriate debuginfo package.
>      Error: Failed to add events.
>    [root@ssdandy linux]#
>
> Much, much clear message about that debuginfo file :-)
>
> But then, to test your [2/3] patch I'll have to figure out how to setup the
> environment so that I can match your results, shouldn't be hard, but would
> save reviewing time if you stated it in the commit log message.

Sorry for not providing enough information.

-s ./kernel_src/

The kernel_src dir is a kernel source tree path, it can be
anywhere, for test you can only put fs/super.c in it:

kernel_src/
   -- fs
      -- super.c

-k vmlinux

This is necessary and should be matched to your running kernel.


>
> - Arnaldo
> --
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/
>

Re: [PATCH 2/3] perf probe: Make --source avaiable when probe with lazy_line

[Masami Hiramatsu]

Hi,

This one should already be fixed with Naohiro's patch.

 https://lkml.org/lkml/2015/3/13/16

Arnaldo, I've already given my ack for that.

https://lkml.org/lkml/2015/3/13/298

Should I resend that?


Thank you,

(2015/04/13 20:41), He Kuang wrote:
> Use get_real_path() to enable --source option when probe with lazy_line
> pattern.
> 
> Before this patch:
> 
>   $ perf probe -s ./kernel_src/ -k ./vmlinux --add='fs/super.c;s->s_count=1;'
>   Failed to open fs/super.c: No such file or directory
>     Error: Failed to add events.
> 
> After this patch:
> 
>   $ perf probe -s ./kernel_src/ -k ./vmlinux  --add='fs/super.c;s->s_count=1;'
>   Added new events:
>     probe:_stext         (on @fs/super.c)
>     probe:_stext_1       (on @fs/super.c)
>   ...
> 
> Signed-off-by: He Kuang <hekuang@huawei.com>
> ---
>  tools/perf/util/probe-event.c  |  2 +-
>  tools/perf/util/probe-event.h  |  2 ++
>  tools/perf/util/probe-finder.c | 18 +++++++++++++++---
>  3 files changed, 18 insertions(+), 4 deletions(-)
> 
> diff --git a/tools/perf/util/probe-event.c b/tools/perf/util/probe-event.c
> index 5483d98..35ee51a 100644
> --- a/tools/perf/util/probe-event.c
> +++ b/tools/perf/util/probe-event.c
> @@ -661,7 +661,7 @@ static int try_to_find_probe_trace_events(struct perf_probe_event *pev,
>   * a newly allocated path on success.
>   * Return 0 if file was found and readable, -errno otherwise.
>   */
> -static int get_real_path(const char *raw_path, const char *comp_dir,
> +int get_real_path(const char *raw_path, const char *comp_dir,
>  			 char **new_path)
>  {
>  	const char *prefix = symbol_conf.source_prefix;
> diff --git a/tools/perf/util/probe-event.h b/tools/perf/util/probe-event.h
> index d6b7834..21809ea 100644
> --- a/tools/perf/util/probe-event.h
> +++ b/tools/perf/util/probe-event.h
> @@ -135,6 +135,8 @@ extern int show_available_vars(struct perf_probe_event *pevs, int npevs,
>  			       struct strfilter *filter, bool externs);
>  extern int show_available_funcs(const char *module, struct strfilter *filter,
>  				bool user);
> +extern int get_real_path(const char *raw_path, const char *comp_dir,
> +			char **new_path);
>  
>  /* Maximum index number of event-name postfix */
>  #define MAX_EVENT_INDEX	1024
> diff --git a/tools/perf/util/probe-finder.c b/tools/perf/util/probe-finder.c
> index 7831e2d..431c12d 100644
> --- a/tools/perf/util/probe-finder.c
> +++ b/tools/perf/util/probe-finder.c
> @@ -791,11 +791,20 @@ static int find_lazy_match_lines(struct intlist *list,
>  	ssize_t len;
>  	int count = 0, linenum = 1;
>  	char sbuf[STRERR_BUFSIZE];
> +	char *realname = NULL;
> +	int ret;
>  
> -	fp = fopen(fname, "r");
> +	ret = get_real_path(fname, NULL, &realname);
> +	if (ret < 0) {
> +		pr_warning("Failed to find source file %s.\n", fname);
> +		return ret;
> +	}
> +
> +	fp = fopen(realname, "r");
>  	if (!fp) {
> -		pr_warning("Failed to open %s: %s\n", fname,
> +		pr_warning("Failed to open %s: %s\n", realname,
>  			   strerror_r(errno, sbuf, sizeof(sbuf)));
> +		free(realname);
>  		return -errno;
>  	}
>  
> @@ -817,7 +826,10 @@ static int find_lazy_match_lines(struct intlist *list,
>  	fclose(fp);
>  
>  	if (count == 0)
> -		pr_debug("No matched lines found in %s.\n", fname);
> +		pr_debug("No matched lines found in %s.\n", realname);
> +
> +	free(realname);
> +
>  	return count;
>  }
>  
> 


-- 
Masami HIRAMATSU
Linux Technology Research Center, System Productivity Research Dept.
Center for Technology Innovation - Systems Engineering
Hitachi, Ltd., Research & Development Group
E-mail: masami.hiramatsu.pt@hitachi.com

Re: [PATCH 1/3] perf probe: Set retprobe flag when probe in address-based alternative mode

[Masami Hiramatsu]

(2015/04/13 20:41), He Kuang wrote:
> Perf probe misses to set retprobe flag back when falling back to
> address-based alternative mode.
> 
> Can be reproduced as following:
> 
>   $ perf probe -v -k vmlinux --add='sys_write%return'
>   ...
>   Added new event:
>   Writing event: p:probe/sys_write _stext+1584952
>     probe:sys_write      (on sys_write%return)
> 
>   $ cat /sys/kernel/debug/tracing/kprobe_events
>   p:probe/sys_write _stext+1584952
> 
> After this patch:
> 
>   $ perf probe -v -k vmlinux --add='sys_write%return'
>   Added new event:
>   Writing event: r:probe/sys_write SyS_write+0
>     probe:sys_write      (on sys_write%return)
> 
>   $ cat /sys/kernel/debug/tracing/kprobe_events
>   r:probe/sys_write SyS_write
> 
> Signed-off-by: He Kuang <hekuang@huawei.com>

Oops, I missed that!

Acked-by: Masami Hiramatsu <masami.hiramatsu.pt@hitachi.com>

Thank you!

> ---
>  tools/perf/util/probe-event.c | 1 +
>  1 file changed, 1 insertion(+)
> 
> diff --git a/tools/perf/util/probe-event.c b/tools/perf/util/probe-event.c
> index 30545ce..5483d98 100644
> --- a/tools/perf/util/probe-event.c
> +++ b/tools/perf/util/probe-event.c
> @@ -332,6 +332,7 @@ static int find_alternative_probe_point(struct debuginfo *dinfo,
>  	else {
>  		result->offset += pp->offset;
>  		result->line += pp->line;
> +		result->retprobe = pp->retprobe;
>  		ret = 0;
>  	}
>  
> 


-- 
Masami HIRAMATSU
Linux Technology Research Center, System Productivity Research Dept.
Center for Technology Innovation - Systems Engineering
Hitachi, Ltd., Research & Development Group
E-mail: masami.hiramatsu.pt@hitachi.com

Re: [PATCH 3/3] perf probe: Fix segfault when probe with lazy_line to file

[Masami Hiramatsu]

(2015/04/13 20:41), He Kuang wrote:
> The first argument passed to find_probe_point_lazy() should be CU die,
> which will be passed to die_walk_lines() when lazy_line
> matches. Currently, when we probe with lazy_line pattern to file without
> function name, NULL pointer is passed and causes a segment fault.
> 
> Can be repoduced as following:
> 
>   $ perf probe -k vmlinux --add='fs/super.c;s->s_count=1;'
>   [ 1958.984658] perf[1020]: segfault at 10 ip 00007fc6e10d8c71 sp
>   00007ffcbfaaf900 error 4 in libdw-0.161.so[7fc6e10ce000+34000]
>   Segmentation fault
> 
> After this patch:
> 
>   $ perf probe -k vmlinux --add='fs/super.c;s->s_count=1;'
>   Added new event:
>   probe:_stext         (on @fs/super.c)
> 
>   You can now use it in all perf tools, such as:
>     perf record -e probe:_stext -aR sleep 1
> 

Acked-by: Masami Hiramatsu <masami.hiramatsu.pt@hitachi.com>

Thanks!

> Signed-off-by: He Kuang <hekuang@huawei.com>
> ---
>  tools/perf/util/probe-finder.c | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
> 
> diff --git a/tools/perf/util/probe-finder.c b/tools/perf/util/probe-finder.c
> index 431c12d..e91101b 100644
> --- a/tools/perf/util/probe-finder.c
> +++ b/tools/perf/util/probe-finder.c
> @@ -1067,7 +1067,7 @@ static int debuginfo__find_probes(struct debuginfo *dbg,
>  			if (pp->function)
>  				ret = find_probe_point_by_func(pf);
>  			else if (pp->lazy_line)
> -				ret = find_probe_point_lazy(NULL, pf);
> +				ret = find_probe_point_lazy(&pf->cu_die, pf);
>  			else {
>  				pf->lno = pp->line;
>  				ret = find_probe_point_by_line(pf);
> 


-- 
Masami HIRAMATSU
Linux Technology Research Center, System Productivity Research Dept.
Center for Technology Innovation - Systems Engineering
Hitachi, Ltd., Research & Development Group
E-mail: masami.hiramatsu.pt@hitachi.com

Re: [PATCH 2/3] perf probe: Make --source avaiable when probe with lazy_line

[Arnaldo Carvalho de Melo]

Em Tue, Apr 14, 2015 at 12:41:03AM +0900, Masami Hiramatsu escreveu:
> Hi,
> 
> This one should already be fixed with Naohiro's patch.
> 
>  https://lkml.org/lkml/2015/3/13/16
> 
> Arnaldo, I've already given my ack for that.
> 
> https://lkml.org/lkml/2015/3/13/298
> 
> Should I resend that?

I will check that and apply, thanks!

- Arnaldo
 
> 
> Thank you,
> 
> (2015/04/13 20:41), He Kuang wrote:
> > Use get_real_path() to enable --source option when probe with lazy_line
> > pattern.
> > 
> > Before this patch:
> > 
> >   $ perf probe -s ./kernel_src/ -k ./vmlinux --add='fs/super.c;s->s_count=1;'
> >   Failed to open fs/super.c: No such file or directory
> >     Error: Failed to add events.
> > 
> > After this patch:
> > 
> >   $ perf probe -s ./kernel_src/ -k ./vmlinux  --add='fs/super.c;s->s_count=1;'
> >   Added new events:
> >     probe:_stext         (on @fs/super.c)
> >     probe:_stext_1       (on @fs/super.c)
> >   ...
> > 
> > Signed-off-by: He Kuang <hekuang@huawei.com>
> > ---
> >  tools/perf/util/probe-event.c  |  2 +-
> >  tools/perf/util/probe-event.h  |  2 ++
> >  tools/perf/util/probe-finder.c | 18 +++++++++++++++---
> >  3 files changed, 18 insertions(+), 4 deletions(-)
> > 
> > diff --git a/tools/perf/util/probe-event.c b/tools/perf/util/probe-event.c
> > index 5483d98..35ee51a 100644
> > --- a/tools/perf/util/probe-event.c
> > +++ b/tools/perf/util/probe-event.c
> > @@ -661,7 +661,7 @@ static int try_to_find_probe_trace_events(struct perf_probe_event *pev,
> >   * a newly allocated path on success.
> >   * Return 0 if file was found and readable, -errno otherwise.
> >   */
> > -static int get_real_path(const char *raw_path, const char *comp_dir,
> > +int get_real_path(const char *raw_path, const char *comp_dir,
> >  			 char **new_path)
> >  {
> >  	const char *prefix = symbol_conf.source_prefix;
> > diff --git a/tools/perf/util/probe-event.h b/tools/perf/util/probe-event.h
> > index d6b7834..21809ea 100644
> > --- a/tools/perf/util/probe-event.h
> > +++ b/tools/perf/util/probe-event.h
> > @@ -135,6 +135,8 @@ extern int show_available_vars(struct perf_probe_event *pevs, int npevs,
> >  			       struct strfilter *filter, bool externs);
> >  extern int show_available_funcs(const char *module, struct strfilter *filter,
> >  				bool user);
> > +extern int get_real_path(const char *raw_path, const char *comp_dir,
> > +			char **new_path);
> >  
> >  /* Maximum index number of event-name postfix */
> >  #define MAX_EVENT_INDEX	1024
> > diff --git a/tools/perf/util/probe-finder.c b/tools/perf/util/probe-finder.c
> > index 7831e2d..431c12d 100644
> > --- a/tools/perf/util/probe-finder.c
> > +++ b/tools/perf/util/probe-finder.c
> > @@ -791,11 +791,20 @@ static int find_lazy_match_lines(struct intlist *list,
> >  	ssize_t len;
> >  	int count = 0, linenum = 1;
> >  	char sbuf[STRERR_BUFSIZE];
> > +	char *realname = NULL;
> > +	int ret;
> >  
> > -	fp = fopen(fname, "r");
> > +	ret = get_real_path(fname, NULL, &realname);
> > +	if (ret < 0) {
> > +		pr_warning("Failed to find source file %s.\n", fname);
> > +		return ret;
> > +	}
> > +
> > +	fp = fopen(realname, "r");
> >  	if (!fp) {
> > -		pr_warning("Failed to open %s: %s\n", fname,
> > +		pr_warning("Failed to open %s: %s\n", realname,
> >  			   strerror_r(errno, sbuf, sizeof(sbuf)));
> > +		free(realname);
> >  		return -errno;
> >  	}
> >  
> > @@ -817,7 +826,10 @@ static int find_lazy_match_lines(struct intlist *list,
> >  	fclose(fp);
> >  
> >  	if (count == 0)
> > -		pr_debug("No matched lines found in %s.\n", fname);
> > +		pr_debug("No matched lines found in %s.\n", realname);
> > +
> > +	free(realname);
> > +
> >  	return count;
> >  }
> >  
> > 
> 
> 
> -- 
> Masami HIRAMATSU
> Linux Technology Research Center, System Productivity Research Dept.
> Center for Technology Innovation - Systems Engineering
> Hitachi, Ltd., Research & Development Group
> E-mail: masami.hiramatsu.pt@hitachi.com
> 

Re: [PATCH 1/3] perf probe: Set retprobe flag when probe in address-based alternative mode

[Arnaldo Carvalho de Melo]

Em Tue, Apr 14, 2015 at 12:42:52AM +0900, Masami Hiramatsu escreveu:
> (2015/04/13 20:41), He Kuang wrote:
> > Perf probe misses to set retprobe flag back when falling back to
> > address-based alternative mode.
> > 
> > Can be reproduced as following:
> > 
> >   $ perf probe -v -k vmlinux --add='sys_write%return'
> >   ...
> >   Added new event:
> >   Writing event: p:probe/sys_write _stext+1584952
> >     probe:sys_write      (on sys_write%return)
> > 
> >   $ cat /sys/kernel/debug/tracing/kprobe_events
> >   p:probe/sys_write _stext+1584952
> > 
> > After this patch:
> > 
> >   $ perf probe -v -k vmlinux --add='sys_write%return'
> >   Added new event:
> >   Writing event: r:probe/sys_write SyS_write+0
> >     probe:sys_write      (on sys_write%return)
> > 
> >   $ cat /sys/kernel/debug/tracing/kprobe_events
> >   r:probe/sys_write SyS_write
> > 
> > Signed-off-by: He Kuang <hekuang@huawei.com>
> 
> Oops, I missed that!
> 
> Acked-by: Masami Hiramatsu <masami.hiramatsu.pt@hitachi.com>
> 
> Thank you!

Thanks to both of you, will add the extra comments from He and apply
after testing with a freshly build kernel with CONFIG_DEBUG_INFO=y.

- Arnaldo
 
> > ---
> >  tools/perf/util/probe-event.c | 1 +
> >  1 file changed, 1 insertion(+)
> > 
> > diff --git a/tools/perf/util/probe-event.c b/tools/perf/util/probe-event.c
> > index 30545ce..5483d98 100644
> > --- a/tools/perf/util/probe-event.c
> > +++ b/tools/perf/util/probe-event.c
> > @@ -332,6 +332,7 @@ static int find_alternative_probe_point(struct debuginfo *dinfo,
> >  	else {
> >  		result->offset += pp->offset;
> >  		result->line += pp->line;
> > +		result->retprobe = pp->retprobe;
> >  		ret = 0;
> >  	}
> >  
> > 
> 
> 
> -- 
> Masami HIRAMATSU
> Linux Technology Research Center, System Productivity Research Dept.
> Center for Technology Innovation - Systems Engineering
> Hitachi, Ltd., Research & Development Group
> E-mail: masami.hiramatsu.pt@hitachi.com
> 

Re: [PATCH 1/3] perf probe: Set retprobe flag when probe in address-based alternative mode

[Arnaldo Carvalho de Melo]

Em Mon, Apr 13, 2015 at 11:23:12PM +0800, He Kuang escreveu:
> 
> 
> On 04/13/2015 10:39 PM, Arnaldo Carvalho de Melo wrote:
> >Em Mon, Apr 13, 2015 at 07:41:28PM +0800, He Kuang escreveu:
> >>Perf probe misses to set retprobe flag back when falling back to
> >>address-based alternative mode.
> >
> >And when is that happens, can you explain? Because I tried to follow your
> >instructions, but in my case it worked in the same way before and after your
> >patch.
> >
> >Can you please provide more context for reviewing your patch?
> >
> 
> More details:
> 
>   [root]# perf probe -v -k vmlinux --add='sys_write%return'
>   probe-definition(0): sys_write%return
>   symbol:sys_write file:(null) line:0 offset:0 return:1 lazy:(null)
>   0 arguments
>   Use vmlinux: vmlinux
>   map_groups__set_modules_path_dir: cannot open /lib/modules/4.0.0-rc6+ dir
>   Problems setting modules path maps, continuing anyway...
>   Using vmlinux for symbols
> 
>   Open Debuginfo file: vmlinux
>   >>>==========================
> 
>   Try to find probe point from debuginfo.
>   Symbol sys_write address found : ffffffff811a8cf0
>   Probe point found: SyS_write+0
>   Found 1 probe_trace_events.
>   Opening /sys/kernel/debug/tracing/kprobe_events write=1
>   Added new event:
>   Writing event: p:probe/sys_write _stext+1739560
>     probe:sys_write      (on sys_write%return)
> 
>   You can now use it in all perf tools, such as:
> 
>           perf record -e probe:sys_write -aR sleep 1
> 
>   [root@buildroot tmp]# cat /sys/kernel/debug/tracing/kprobe_events
>   p:probe/sys_write _stext+1739560
> 
> 
> The line I marked is different from your result, which is:
>   """Could not open debuginfo. Try to use symbols."""
> 
> 
> When perf probe searched in debuginfo and failed, it tried with alternative,
> in function get_alternative_probe_event():
> 
>         memcpy(tmp, &pev->point, sizeof(*tmp));
> 	memset(&pev->point, 0, sizeof(pev->point));
> 
> In this case, it drops the retprobe flag and forgets to set it back in
> find_alternative_probe_point(), so the problem occurred.

Thanks for the additional information! I will add it to the commit log.

I'm still not being able to reproduce your results, but this time was
because I tried using those separate .dwo debuginfo files, to check if
'perf' would support it already because elfutils would somehow hide that
from us, but that doesn't seem to be the case:

[root@ssdandy ~]# perf probe -v --add='sys_write%return'
probe-definition(0): sys_write%return
symbol:sys_write file:(null) line:0 offset:0 return:1 lazy:(null)
0 arguments
Looking at the vmlinux_path (7 entries long)
Using /lib/modules/4.0.0-rc6+/build/vmlinux for symbols
Open Debuginfo file: /lib/modules/4.0.0-rc6+/build/vmlinux
Try to find probe point from debuginfo.
Symbol sys_write address found : ffffffff811e5f10
Failed to find scope of probe point.
An error occurred in debuginfo analysis (-2).
  Error: Failed to add events. Reason: No such file or directory (Code: -2)
[root@ssdandy ~]# cat /sys/kernel/debug/tracing/kprobe_events
[root@ssdandy ~]#

[root@ssdandy linux]# grep CONFIG_DEBUG_INFO ../build/v4.0.0-rc6+/.config
CONFIG_DEBUG_INFO=y
# CONFIG_DEBUG_INFO_REDUCED is not set
CONFIG_DEBUG_INFO_SPLIT=y
# CONFIG_DEBUG_INFO_DWARF4 is not set
[root@ssdandy linux]# 

Masami, this is about using CONFIG_DEBUG_INFO_SPLIT=y, that eventually we
should support, for now I will disable it and test again,

- Arnaldo

Re: [PATCH 1/3] perf probe: Set retprobe flag when probe in address-based alternative mode

[Arnaldo Carvalho de Melo]

Em Mon, Apr 13, 2015 at 11:38:19PM +0800, He Kuang escreveu:
> Hi, Arnaldo
> 
> On 04/13/2015 10:42 PM, Arnaldo Carvalho de Melo wrote:
> >Em Mon, Apr 13, 2015 at 11:39:03AM -0300, Arnaldo Carvalho de Melo escreveu:
> >>Em Mon, Apr 13, 2015 at 07:41:28PM +0800, He Kuang escreveu:
> >>>Perf probe misses to set retprobe flag back when falling back to
> >>>address-based alternative mode.
> >
> >>Humm, noticed one other problem, but not with your patch, about this message:
> >
> >>"Could not open debuginfo. Try to use symbols."
> >
> >>That is really not clear, specially that "try to use symbols" :-)
> >
> >>[root@ssdandy ~]# ls -la /root/.debug/.build-id/dd/32e51921ede0fd46f034091b7f6a0f2e01ebda
> >>lrwxrwxrwx. 1 root root 86 Apr 10 18:02 /root/.debug/.build-id/dd/32e51921ede0fd46f034091b7f6a0f2e01ebda -> ../../home/acme/git/build/v4.0.0-rc6+/vmlinux/dd32e51921ede0fd46f034091b7f6a0f2e01ebda
> >>[root@ssdandy ~]# ls -la /root/.debug/.build-id/dd/../../home/acme/git/build/v4.0.0-rc6+/vmlinux/dd32e51921ede0fd46f034091b7f6a0f2e01ebda
> >>-rwxr-xr-x. 1 root root 22698661 Apr 10 18:02 /root/.debug/.build-id/dd/../../home/acme/git/build/v4.0.0-rc6+/vmlinux/dd32e51921ede0fd46f034091b7f6a0f2e01ebda
> >>[root@ssdandy ~]# ls -la /root/.debug/.build-id/dd/../../home/acme/git/build/v4.0.0-rc6+/vmlinux/dd32e51921ede0fd46f034091b7f6a0f2e01ebda
> >
> >>I.e. it managed to read the debuginfo, its just that it has no symbols in it :-)
> >
> >>Anyway, digression ended.
> >
> >Interesting is that when testing your next patch I see:
> >
> >   [root@ssdandy linux]#  perf probe -s ./kernel_src/ --add='fs/super.c;s->s_count=1;'
> >   The /root/.debug/.build-id/dd/32e51921ede0fd46f034091b7f6a0f2e01ebda file has no debug information.
> >   Rebuild with CONFIG_DEBUG_INFO=y, or install an appropriate debuginfo package.
> >     Error: Failed to add events.
> >   [root@ssdandy linux]#
> >
> >Much, much clear message about that debuginfo file :-)
> >
> >But then, to test your [2/3] patch I'll have to figure out how to setup the
> >environment so that I can match your results, shouldn't be hard, but would
> >save reviewing time if you stated it in the commit log message.
> 
> Sorry for not providing enough information.
> 
> -s ./kernel_src/
> 
> The kernel_src dir is a kernel source tree path, it can be
> anywhere, for test you can only put fs/super.c in it:
> 
> kernel_src/
>   -- fs
>      -- super.c
> 
> -k vmlinux
> 
> This is necessary and should be matched to your running kernel.

Normally I don't use this, as 'make install' will set it to a place that
the perf symbol code will look, check its build id, see that it matches
the what is running, i.e. this will hold true:

[root@ssdandy ~]# perf buildid-list --hell 2>&1 | grep -- --kernel
    -k, --kernel          Show current kernel build id
[root@ssdandy ~]# perf buildid-list --kernel
cd1d2cf9f473d0cac668e3afee32866da4540bd4
[root@ssdandy ~]#
[root@ssdandy ~]# perf buildid-list -i
/lib/modules/4.0.0-rc6+/build/vmlinux
cd1d2cf9f473d0cac668e3afee32866da4540bd4

I.e. no need to explicitely pass '-k vmlinux':

[root@ssdandy ~]# perf probe -v --add='sys_write%return'
probe-definition(0): sys_write%return
symbol:sys_write file:(null) line:0 offset:0 return:1 lazy:(null)
0 arguments
Looking at the vmlinux_path (7 entries long)
Using /lib/modules/4.0.0-rc6+/build/vmlinux for symbols
Open Debuginfo file: /lib/modules/4.0.0-rc6+/build/vmlinux
Try to find probe point from debuginfo.
Symbol sys_write address found : ffffffff811e5f10
Probe point found: SyS_write+0
Found 1 probe_trace_events.
Opening /sys/kernel/debug/tracing/kprobe_events write=1
Added new event:
Writing event: r:probe/sys_write SyS_write+0
  probe:sys_write      (on sys_write%return)

You can now use it in all perf tools, such as:

	perf record -e probe:sys_write -aR sleep 1

[root@ssdandy ~]#

- Arnaldo

Re: [PATCH 2/3] perf probe: Make --source avaiable when probe with lazy_line

[Arnaldo Carvalho de Melo]

Em Mon, Apr 13, 2015 at 07:41:29PM +0800, He Kuang escreveu:
> Use get_real_path() to enable --source option when probe with lazy_line
> pattern.
> 
> Before this patch:
> 
>   $ perf probe -s ./kernel_src/ -k ./vmlinux --add='fs/super.c;s->s_count=1;'
>   Failed to open fs/super.c: No such file or directory
>     Error: Failed to add events.
> 
> After this patch:
> 
>   $ perf probe -s ./kernel_src/ -k ./vmlinux  --add='fs/super.c;s->s_count=1;'
>   Added new events:
>     probe:_stext         (on @fs/super.c)
>     probe:_stext_1       (on @fs/super.c)
>   ...

Tried just doing:

[root@ssdandy ~]# ulimit -c 100000000
[root@ssdandy ~]# perf probe --add='fs/super.c;s->s_count=1;'
Segmentation fault (core dumped)

I.e. not specifying vmlinux, as it finds the right one looking at the vmlinux path,
and as well not specifying -s, as it should find that as well from the vmlinux file,
but here I get a segfault:

[root@ssdandy ~]# gdb -c core.26128 ~/bin/perf
<SNIP>
Core was generated by `perf probe --add=fs/super.c;s->s_count=1;'.
Program terminated with signal 11, Segmentation fault.
#0  0x00007f2a021301c1 in dwarf_tag () from /lib64/libdw.so.1
Missing separate debuginfos, use: debuginfo-install audit-libs-2.4.1-5.el7.x86_64 bzip2-libs-1.0.6-12.el7.x86_64 elfutils-libelf-0.160-1.el7.x86_64 elfutils-libs-0.160-1.el7.x86_64 glibc-2.17-78.el7.x86_64 libgcc-4.8.3-9.el7.x86_64 libunwind-1.1-3.el7.x86_64 nss-softokn-freebl-3.16.2.3-9.el7.x86_64 numactl-libs-2.0.9-4.el7.x86_64 perl-libs-5.16.3-285.el7.x86_64 python-libs-2.7.5-16.el7.x86_64 slang-2.2.4-11.el7.x86_64 xz-libs-5.1.2-9alpha.el7.x86_64 zlib-1.2.7-13.el7.x86_64
(gdb) bt
#0  0x00007f2a021301c1 in dwarf_tag () from /lib64/libdw.so.1

#1  0x00000000004e3b78 in die_walk_lines (rt_die=rt_die@entry=0x0, callback=callback@entry=0x4e0b10 <probe_point_lazy_walker>, data=data@entry=0x7ffff2da1750) at util/dwarf-aux.c:690
#2  0x00000000004dff19 in find_probe_point_lazy (sp_die=sp_die@entry=0x0, pf=pf@entry=0x7ffff2da1750) at util/probe-finder.c:867
#3  0x00000000004e06e2 in debuginfo__find_probes (pf=pf@entry=0x7ffff2da1750, dbg=0x1767c50) at util/probe-finder.c:1058
#4  0x00000000004e13b2 in debuginfo__find_trace_events (dbg=dbg@entry=0x1767c50, pev=pev@entry=0x8fcd90 <params+16>, tevs=tevs@entry=0x1763978, max_tevs=max_tevs@entry=128) at util/probe-finder.c:1218
#5  0x00000000004dcbea in try_to_find_probe_trace_events (target=0x0, max_tevs=128, tevs=0x1763978, pev=0x8fcd90 <params+16>) at util/probe-event.c:611
#6  convert_to_probe_trace_events (target=0x0, max_tevs=128, tevs=0x1763978, pev=0x8fcd90 <params+16>) at util/probe-event.c:2645
#7  add_perf_probe_events (pevs=pevs@entry=0x8fcd90 <params+16>, npevs=<optimized out>, max_tevs=128, force_add=false) at util/probe-event.c:2680
#8  0x000000000044fa7b in __cmd_probe (argc=<optimized out>, argv=argv@entry=0x7ffff2da2600, prefix=<optimized out>) at builtin-probe.c:504
#9  0x000000000044fc34 in cmd_probe (argc=<optimized out>, argv=0x7ffff2da2600, prefix=<optimized out>) at builtin-probe.c:521
#10 0x0000000000470b35 in run_builtin (p=p@entry=0x838420 <commands+384>, argc=argc@entry=2, argv=argv@entry=0x7ffff2da2600) at perf.c:370
#11 0x000000000041d220 in handle_internal_command (argv=0x7ffff2da2600, argc=2) at perf.c:429
#12 run_argv (argv=0x7ffff2da2380, argcp=0x7ffff2da238c) at perf.c:473
#13 main (argc=2, argv=0x7ffff2da2600) at perf.c:588
(gdb) 

Now with your patch, another coredump:

[root@ssdandy ~]# rm -f core.26128 
[root@ssdandy ~]# perf probe --add='fs/super.c;s->s_count=1;'
Segmentation fault (core dumped)
[root@ssdandy ~]#
[root@ssdandy ~]# gdb -c core.26712 ~/bin/perf
<SNIP>
Core was generated by `perf probe --add=fs/super.c;s->s_count=1;'.
Program terminated with signal 11, Segmentation fault.
#0  0x00007f25cf4e61c1 in dwarf_tag () from /lib64/libdw.so.1
Missing separate debuginfos, use: debuginfo-install audit-libs-2.4.1-5.el7.x86_64 bzip2-libs-1.0.6-12.el7.x86_64 elfutils-libelf-0.160-1.el7.x86_64 elfutils-libs-0.160-1.el7.x86_64 glibc-2.17-78.el7.x86_64 libgcc-4.8.3-9.el7.x86_64 libunwind-1.1-3.el7.x86_64 nss-softokn-freebl-3.16.2.3-9.el7.x86_64 numactl-libs-2.0.9-4.el7.x86_64 perl-libs-5.16.3-285.el7.x86_64 python-libs-2.7.5-16.el7.x86_64 slang-2.2.4-11.el7.x86_64 xz-libs-5.1.2-9alpha.el7.x86_64 zlib-1.2.7-13.el7.x86_64
(gdb) bt
#0  0x00007f25cf4e61c1 in dwarf_tag () from /lib64/libdw.so.1
#1  0x00000000004e3c18 in die_walk_lines (rt_die=rt_die@entry=0x0, callback=callback@entry=0x4e0bb0 <probe_point_lazy_walker>, data=data@entry=0x7ffdf9244df0) at util/dwarf-aux.c:690
#2  0x00000000004dff9b in find_probe_point_lazy (sp_die=sp_die@entry=0x0, pf=pf@entry=0x7ffdf9244df0) at util/probe-finder.c:879
#3  0x00000000004e0782 in debuginfo__find_probes (pf=pf@entry=0x7ffdf9244df0, dbg=0xe3ec50) at util/probe-finder.c:1070
#4  0x00000000004e1452 in debuginfo__find_trace_events (dbg=dbg@entry=0xe3ec50, pev=pev@entry=0x8fcd90 <params+16>, tevs=tevs@entry=0xe3a978, max_tevs=max_tevs@entry=128) at util/probe-finder.c:1230
#5  0x00000000004dcc2a in try_to_find_probe_trace_events (target=0x0, max_tevs=128, tevs=0xe3a978, pev=0x8fcd90 <params+16>) at util/probe-event.c:611
#6  convert_to_probe_trace_events (target=0x0, max_tevs=128, tevs=0xe3a978, pev=0x8fcd90 <params+16>) at util/probe-event.c:2645
#7  add_perf_probe_events (pevs=pevs@entry=0x8fcd90 <params+16>, npevs=<optimized out>, max_tevs=128, force_add=false) at util/probe-event.c:2680
#8  0x000000000044faab in __cmd_probe (argc=<optimized out>, argv=argv@entry=0x7ffdf9245ca0, prefix=<optimized out>) at builtin-probe.c:504
#9  0x000000000044fc64 in cmd_probe (argc=<optimized out>, argv=0x7ffdf9245ca0, prefix=<optimized out>) at builtin-probe.c:521
#10 0x0000000000470b65 in run_builtin (p=p@entry=0x838420 <commands+384>, argc=argc@entry=2, argv=argv@entry=0x7ffdf9245ca0) at perf.c:370
#11 0x000000000041d250 in handle_internal_command (argv=0x7ffdf9245ca0, argc=2) at perf.c:429
#12 run_argv (argv=0x7ffdf9245a20, argcp=0x7ffdf9245a2c) at perf.c:473
#13 main (argc=2, argv=0x7ffdf9245ca0) at perf.c:588
(gdb)

Trying exactly as you did, first without the patch:

[root@ssdandy linux]# perf probe -s /home/acme/git/linux/ -k ../build/v4.0.0-rc6+/vmlinux  --add='fs/super.c;s->s_count=1;'
Segmentation fault (core dumped)
[root@ssdandy linux]# 

Now with it:

[root@ssdandy linux]# perf probe -s /home/acme/git/linux/ -k ../build/v4.0.0-rc6+/vmlinux  --add='fs/super.c;s->s_count=1;'
Segmentation fault (core dumped)
[root@ssdandy linux]#

Core was generated by `perf probe -s /home/acme/git/linux/ -k ../build/v4.0.0-rc6+/vmlinux --add=fs/su'.
Program terminated with signal 11, Segmentation fault.
#0  0x00007f76795c01c1 in dwarf_tag () from /lib64/libdw.so.1
Missing separate debuginfos, use: debuginfo-install audit-libs-2.4.1-5.el7.x86_64 bzip2-libs-1.0.6-12.el7.x86_64 elfutils-libelf-0.160-1.el7.x86_64 elfutils-libs-0.160-1.el7.x86_64 glibc-2.17-78.el7.x86_64 libgcc-4.8.3-9.el7.x86_64 libunwind-1.1-3.el7.x86_64 nss-softokn-freebl-3.16.2.3-9.el7.x86_64 numactl-libs-2.0.9-4.el7.x86_64 perl-libs-5.16.3-285.el7.x86_64 python-libs-2.7.5-16.el7.x86_64 slang-2.2.4-11.el7.x86_64 xz-libs-5.1.2-9alpha.el7.x86_64 zlib-1.2.7-13.el7.x86_64
(gdb) bt
#0  0x00007f76795c01c1 in dwarf_tag () from /lib64/libdw.so.1
#1  0x00000000004e3c18 in die_walk_lines (rt_die=rt_die@entry=0x0, callback=callback@entry=0x4e0bb0 <probe_point_lazy_walker>, data=data@entry=0x7ffcf282f280) at util/dwarf-aux.c:690
#2  0x00000000004dff9b in find_probe_point_lazy (sp_die=sp_die@entry=0x0, pf=pf@entry=0x7ffcf282f280) at util/probe-finder.c:879
#3  0x00000000004e0782 in debuginfo__find_probes (pf=pf@entry=0x7ffcf282f280, dbg=0x19c1b30) at util/probe-finder.c:1070
#4  0x00000000004e1452 in debuginfo__find_trace_events (dbg=dbg@entry=0x19c1b30, pev=pev@entry=0x8fcd90 <params+16>, tevs=tevs@entry=0x19be778, max_tevs=max_tevs@entry=128) at util/probe-finder.c:1230
#5  0x00000000004dcc2a in try_to_find_probe_trace_events (target=0x0, max_tevs=128, tevs=0x19be778, pev=0x8fcd90 <params+16>) at util/probe-event.c:611
#6  convert_to_probe_trace_events (target=0x0, max_tevs=128, tevs=0x19be778, pev=0x8fcd90 <params+16>) at util/probe-event.c:2645
#7  add_perf_probe_events (pevs=pevs@entry=0x8fcd90 <params+16>, npevs=<optimized out>, max_tevs=128, force_add=false) at util/probe-event.c:2680
#8  0x000000000044faab in __cmd_probe (argc=<optimized out>, argv=argv@entry=0x7ffcf2830130, prefix=<optimized out>) at builtin-probe.c:504
#9  0x000000000044fc64 in cmd_probe (argc=<optimized out>, argv=0x7ffcf2830130, prefix=<optimized out>) at builtin-probe.c:521
#10 0x0000000000470b65 in run_builtin (p=p@entry=0x838420 <commands+384>, argc=argc@entry=6, argv=argv@entry=0x7ffcf2830130) at perf.c:370
#11 0x000000000041d250 in handle_internal_command (argv=0x7ffcf2830130, argc=6) at perf.c:429
#12 run_argv (argv=0x7ffcf282feb0, argcp=0x7ffcf282febc) at perf.c:473
#13 main (argc=6, argv=0x7ffcf2830130) at perf.c:588
(gdb)

And:

[acme@ssdandy linux]$ ls -la /home/acme/git/linux/fs/super.c
-rw-rw-r--. 1 acme acme 36192 Abr  7 17:59 /home/acme/git/linux/fs/super.c

Processed [1/3], looking if [3/3] can be applied independently of this one.

- Arnaldo

Re: [PATCH 2/3] perf probe: Make --source avaiable when probe with lazy_line

[Arnaldo Carvalho de Melo]

Em Mon, Apr 13, 2015 at 05:56:59PM -0300, Arnaldo Carvalho de Melo escreveu:
> Em Mon, Apr 13, 2015 at 07:41:29PM +0800, He Kuang escreveu:
> 
> [root@ssdandy linux]# perf probe -s /home/acme/git/linux/ -k ../build/v4.0.0-rc6+/vmlinux  --add='fs/super.c;s->s_count=1;'
> Segmentation fault (core dumped)
> [root@ssdandy linux]#

[3/3] fixes a segfault, ok, probably this one, checking :-)

- Arnaldo

Re: [PATCH 2/3] perf probe: Make --source avaiable when probe with lazy_line

[Arnaldo Carvalho de Melo]

Em Mon, Apr 13, 2015 at 05:57:51PM -0300, Arnaldo Carvalho de Melo escreveu:
> Em Mon, Apr 13, 2015 at 05:56:59PM -0300, Arnaldo Carvalho de Melo escreveu:
> > Em Mon, Apr 13, 2015 at 07:41:29PM +0800, He Kuang escreveu:
> > 
> > [root@ssdandy linux]# perf probe -s /home/acme/git/linux/ -k ../build/v4.0.0-rc6+/vmlinux  --add='fs/super.c;s->s_count=1;'
> > Segmentation fault (core dumped)
> > [root@ssdandy linux]#
> 
> [3/3] fixes a segfault, ok, probably this one, checking :-)

Ok, all three applied, thanks!

- Arnaldo

[tip:perf/urgent] perf probe: Set retprobe flag when probe in address-based alternative mode

[tip-bot for He Kuang]

Commit-ID:  9d7b45c5721dfd524666bc1f07cf7d0855edb4d5
Gitweb:     http://git.kernel.org/tip/9d7b45c5721dfd524666bc1f07cf7d0855edb4d5
Author:     He Kuang <hekuang@huawei.com>
AuthorDate: Mon, 13 Apr 2015 19:41:28 +0800
Committer:  Arnaldo Carvalho de Melo <acme@redhat.com>
CommitDate: Mon, 13 Apr 2015 17:25:22 -0300

perf probe: Set retprobe flag when probe in address-based alternative mode

When perf probe searched in a debuginfo file and failed, it tried with
an alternative, in function get_alternative_probe_event():

        memcpy(tmp, &pev->point, sizeof(*tmp));
        memset(&pev->point, 0, sizeof(pev->point));

In this case, it drops the retprobe flag and forgets to set it back in
find_alternative_probe_point(), so the problem occurs.

Can be reproduced as following:

  $ perf probe -v -k vmlinux --add='sys_write%return'
  ...
  Added new event:
  Writing event: p:probe/sys_write _stext+1584952
    probe:sys_write      (on sys_write%return)

  $ cat /sys/kernel/debug/tracing/kprobe_events
  p:probe/sys_write _stext+1584952

After this patch:

  $ perf probe -v -k vmlinux --add='sys_write%return'
  Added new event:
  Writing event: r:probe/sys_write SyS_write+0
    probe:sys_write      (on sys_write%return)

  $ cat /sys/kernel/debug/tracing/kprobe_events
  r:probe/sys_write SyS_write

Signed-off-by: He Kuang <hekuang@huawei.com>
Tested-by: Arnaldo Carvalho de Melo <acme@redhat.com>
Acked-by: Masami Hiramatsu <masami.hiramatsu.pt@hitachi.com>
Cc: Namhyung Kim <namhyung@kernel.org>
Cc: Peter Zijlstra <a.p.zijlstra@chello.nl>
Cc: Wang Nan <wangnan0@huawei.com>
Link: http://lkml.kernel.org/r/1428925290-5623-1-git-send-email-hekuang@huawei.com
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
---
 tools/perf/util/probe-event.c | 1 +
 1 file changed, 1 insertion(+)

diff --git a/tools/perf/util/probe-event.c b/tools/perf/util/probe-event.c
index 30545ce..5483d98 100644
--- a/tools/perf/util/probe-event.c
+++ b/tools/perf/util/probe-event.c
@@ -332,6 +332,7 @@ static int find_alternative_probe_point(struct debuginfo *dinfo,
 	else {
 		result->offset += pp->offset;
 		result->line += pp->line;
+		result->retprobe = pp->retprobe;
 		ret = 0;
 	}
 

[tip:perf/urgent] perf probe: Fix segfault when probe with lazy_line to file

[tip-bot for He Kuang]

Commit-ID:  f19e80c640d58ddfd70f2454ee597f81ba966690
Gitweb:     http://git.kernel.org/tip/f19e80c640d58ddfd70f2454ee597f81ba966690
Author:     He Kuang <hekuang@huawei.com>
AuthorDate: Mon, 13 Apr 2015 19:41:30 +0800
Committer:  Arnaldo Carvalho de Melo <acme@redhat.com>
CommitDate: Mon, 13 Apr 2015 20:12:21 -0300

perf probe: Fix segfault when probe with lazy_line to file

The first argument passed to find_probe_point_lazy() should be CU die,
which will be passed to die_walk_lines() when lazy_line matches.
Currently, when we probe with lazy_line pattern to file without function
name, NULL pointer is passed and causes a segment fault.

Can be reproduced as following:

  $ perf probe -k vmlinux --add='fs/super.c;s->s_count=1;'
  [ 1958.984658] perf[1020]: segfault at 10 ip 00007fc6e10d8c71 sp
  00007ffcbfaaf900 error 4 in libdw-0.161.so[7fc6e10ce000+34000]
  Segmentation fault

After this patch:

  $ perf probe -k vmlinux --add='fs/super.c;s->s_count=1;'
  Added new event:
  probe:_stext         (on @fs/super.c)

  You can now use it in all perf tools, such as:
    perf record -e probe:_stext -aR sleep 1

Signed-off-by: He Kuang <hekuang@huawei.com>
Tested-by: Arnaldo Carvalho de Melo <acme@redhat.com>
Acked-by: Masami Hiramatsu <masami.hiramatsu.pt@hitachi.com>
Cc: Namhyung Kim <namhyung@kernel.org>
Cc: Peter Zijlstra <a.p.zijlstra@chello.nl>
Cc: Wang Nan <wangnan0@huawei.com>
Link: http://lkml.kernel.org/r/1428925290-5623-3-git-send-email-hekuang@huawei.com
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
---
 tools/perf/util/probe-finder.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/tools/perf/util/probe-finder.c b/tools/perf/util/probe-finder.c
index ff7865c..44554c3 100644
--- a/tools/perf/util/probe-finder.c
+++ b/tools/perf/util/probe-finder.c
@@ -1066,7 +1066,7 @@ static int debuginfo__find_probes(struct debuginfo *dbg,
 			if (pp->function)
 				ret = find_probe_point_by_func(pf);
 			else if (pp->lazy_line)
-				ret = find_probe_point_lazy(NULL, pf);
+				ret = find_probe_point_lazy(&pf->cu_die, pf);
 			else {
 				pf->lno = pp->line;
 				ret = find_probe_point_by_line(pf);