linux-kernel.vger.kernel.org archive mirror
 help / color / mirror / Atom feed
* [PATCH] perf script: Add stackcollapse.py script
@ 2016-04-12 13:26 Paolo Bonzini
  2016-04-15  5:20 ` ping " Paolo Bonzini
                   ` (2 more replies)
  0 siblings, 3 replies; 10+ messages in thread
From: Paolo Bonzini @ 2016-04-12 13:26 UTC (permalink / raw)
  To: linux-kernel; +Cc: Jiri Olsa, Arnaldo Carvalho de Melo, Brendan Gregg

Add stackcollapse.py script as an example of parsing call chains, and
also of using optparse to access command line options.

The flame graph tools include a set of scripts that parse output from
various tools (including "perf script"), remove the offsets in the
function and collapse each stack to a single line.  The website also says
"perf report could have a report style [...] that output folded stacks
directly, obviating the need for stackcollapse-perf.pl", so here it is.

This script is a Python rewrite of stackcollapse-perf.pl, using the perf
scripting interface to access the perf data directly from Python.

Cc: Jiri Olsa <jolsa@kernel.org>
Cc: Arnaldo Carvalho de Melo <acme@redhat.com>
Cc: Brendan Gregg <bgregg@netflix.com>
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
---
 tools/perf/scripts/python/bin/stackcollapse-record |   8 ++
 tools/perf/scripts/python/bin/stackcollapse-report |   3 +
 tools/perf/scripts/python/stackcollapse.py         | 127 +++++++++++++++++++++
 3 files changed, 138 insertions(+)
 create mode 100755 tools/perf/scripts/python/bin/stackcollapse-record
 create mode 100755 tools/perf/scripts/python/bin/stackcollapse-report
 create mode 100755 tools/perf/scripts/python/stackcollapse.py

diff --git a/tools/perf/scripts/python/bin/stackcollapse-record b/tools/perf/scripts/python/bin/stackcollapse-record
new file mode 100755
index 000000000000..9d8f9f0f3a17
--- /dev/null
+++ b/tools/perf/scripts/python/bin/stackcollapse-record
@@ -0,0 +1,8 @@
+#!/bin/sh
+
+#
+# stackcollapse.py can cover all type of perf samples including
+# the tracepoints, so no special record requirements, just record what
+# you want to analyze.
+#
+perf record "$@"
diff --git a/tools/perf/scripts/python/bin/stackcollapse-report b/tools/perf/scripts/python/bin/stackcollapse-report
new file mode 100755
index 000000000000..356b9656393d
--- /dev/null
+++ b/tools/perf/scripts/python/bin/stackcollapse-report
@@ -0,0 +1,3 @@
+#!/bin/sh
+# description: produce callgraphs in short form for scripting use
+perf script -s "$PERF_EXEC_PATH"/scripts/python/stackcollapse.py -- "$@"
diff --git a/tools/perf/scripts/python/stackcollapse.py b/tools/perf/scripts/python/stackcollapse.py
new file mode 100755
index 000000000000..a2dfcda41ae6
--- /dev/null
+++ b/tools/perf/scripts/python/stackcollapse.py
@@ -0,0 +1,127 @@
+#!/usr/bin/perl -w
+#
+# stackcollapse.py - format perf samples with one line per distinct call stack
+#
+# This script's output has two space-separated fields.  The first is a semicolon
+# separated stack including the program name (from the "comm" field) and the
+# function names from the call stack.  The second is a count:
+#
+#  swapper;start_kernel;rest_init;cpu_idle;default_idle;native_safe_halt 2
+#
+# The file is sorted according to the first field.
+#
+# Input may be created and processed using:
+#
+#  perf record -a -g -F 99 sleep 60
+#  perf script report stackcollapse > out.stacks-folded
+#
+# (perf script record stackcollapse works too).
+#
+# Written by Paolo Bonzini <pbonzini@redhat.com>
+# Based on Brendan Gregg's stackcollapse-perf.pl script.
+
+import os
+import sys
+from collections import defaultdict
+from optparse import OptionParser, make_option
+
+sys.path.append(os.environ['PERF_EXEC_PATH'] + \
+                '/scripts/python/Perf-Trace-Util/lib/Perf/Trace')
+
+from perf_trace_context import *
+from Core import *
+from EventClass import *
+
+# command line parsing
+
+option_list = [
+    # formatting options for the bottom entry of the stack
+    make_option("--include-tid", dest="include_tid",
+                 action="store_true", default=False,
+                 help="include thread id in stack"),
+    make_option("--include-pid", dest="include_pid",
+                 action="store_true", default=False,
+                 help="include process id in stack"),
+    make_option("--no-comm", dest="include_comm",
+                 action="store_false", default=True,
+                 help="do not separate stacks according to comm"),
+    make_option("--tidy-java", dest="tidy_java",
+                 action="store_true", default=False,
+                 help="beautify Java signatures"),
+    make_option("--kernel", dest="annotate_kernel",
+                 action="store_true", default=False,
+                 help="annotate kernel functions with _[k]")
+]
+
+parser = OptionParser(option_list=option_list)
+(opts, args) = parser.parse_args()
+
+if len(args) != 0:
+    parser.error("unexpected command line argument")
+if opts.include_tid and not opts.include_comm:
+    parser.error("requesting tid but not comm is invalid")
+if opts.include_pid and not opts.include_comm:
+    parser.error("requesting pid but not comm is invalid")
+
+# event handlers
+
+lines = defaultdict(lambda: 0)
+
+def process_event(param_dict):
+    def tidy_function_name(sym, dso):
+        if sym is None:
+            sym = '[unknown]'
+
+        sym = sym.replace(';', ':')
+        if opts.tidy_java:
+            # the original stackcollapse-perf.pl script gives the
+            # example of converting this:
+            #    Lorg/mozilla/javascript/MemberBox;.<init>(Ljava/lang/reflect/Method;)V
+            # to this:
+            #    org/mozilla/javascript/MemberBox:.init
+            sym = sym.replace('<', '')
+            sym = sym.replace('>', '')
+            if sym[0] == 'L' and sym.find('/'):
+                sym = sym[1:]
+            try:
+                sym = sym[:sym.index('(')]
+            except ValueError:
+                pass
+
+        if opts.annotate_kernel and dso == '[kernel.kallsyms]':
+            return sym + '_[k]'
+        else:
+            return sym
+
+    stack = list()
+    if 'callchain' in param_dict:
+        for entry in param_dict['callchain']:
+            entry.setdefault('sym', dict())
+            entry['sym'].setdefault('name', None)
+            entry.setdefault('dso', None)
+            stack.append(tidy_function_name(entry['sym']['name'],
+                                            entry['dso']))
+    else:
+        param_dict.setdefault('symbol', None)
+        param_dict.setdefault('dso', None)
+        stack.append(tidy_function_name(param_dict['symbol'],
+                                        param_dict['dso']))
+
+    if opts.include_comm:
+        comm = param_dict["comm"].replace(' ', '_')
+        sep = "-"
+        if opts.include_pid:
+            comm = comm + sep + str(param_dict['sample']['pid'])
+            sep = "/"
+        if opts.include_tid:
+            comm = comm + sep + str(param_dict['sample']['tid'])
+        stack.append(comm)
+
+    stack_string = ';'.join(reversed(stack))
+    lines[stack_string] = lines[stack_string] + 1
+
+def trace_end():
+    list = lines.keys()
+    list.sort()
+    for stack in list:
+        print "%s %d" % (stack, lines[stack])
-- 
2.5.5

^ permalink raw reply	[flat|nested] 10+ messages in thread

* ping Re: [PATCH] perf script: Add stackcollapse.py script
  2016-04-12 13:26 [PATCH] perf script: Add stackcollapse.py script Paolo Bonzini
@ 2016-04-15  5:20 ` Paolo Bonzini
  2016-04-15 10:57   ` Jiri Olsa
  2016-06-16  7:22 ` Paolo Bonzini
  2016-06-22  7:44 ` [tip:perf/core] " tip-bot for Paolo Bonzini
  2 siblings, 1 reply; 10+ messages in thread
From: Paolo Bonzini @ 2016-04-15  5:20 UTC (permalink / raw)
  To: linux-kernel; +Cc: Jiri Olsa, Arnaldo Carvalho de Melo, Brendan Gregg

On 12/04/2016 15:26, Paolo Bonzini wrote:
> Add stackcollapse.py script as an example of parsing call chains, and
> also of using optparse to access command line options.
> 
> The flame graph tools include a set of scripts that parse output from
> various tools (including "perf script"), remove the offsets in the
> function and collapse each stack to a single line.  The website also says
> "perf report could have a report style [...] that output folded stacks
> directly, obviating the need for stackcollapse-perf.pl", so here it is.
> 
> This script is a Python rewrite of stackcollapse-perf.pl, using the perf
> scripting interface to access the perf data directly from Python.
> 
> Cc: Jiri Olsa <jolsa@kernel.org>
> Cc: Arnaldo Carvalho de Melo <acme@redhat.com>
> Cc: Brendan Gregg <bgregg@netflix.com>
> Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>

Sorry for the very early ping, I'm going on vacation and I'm afraid the
next time I'd be able to ping would be too late for 4.7. :)

Paolo

> ---
>  tools/perf/scripts/python/bin/stackcollapse-record |   8 ++
>  tools/perf/scripts/python/bin/stackcollapse-report |   3 +
>  tools/perf/scripts/python/stackcollapse.py         | 127 +++++++++++++++++++++
>  3 files changed, 138 insertions(+)
>  create mode 100755 tools/perf/scripts/python/bin/stackcollapse-record
>  create mode 100755 tools/perf/scripts/python/bin/stackcollapse-report
>  create mode 100755 tools/perf/scripts/python/stackcollapse.py
> 
> diff --git a/tools/perf/scripts/python/bin/stackcollapse-record b/tools/perf/scripts/python/bin/stackcollapse-record
> new file mode 100755
> index 000000000000..9d8f9f0f3a17
> --- /dev/null
> +++ b/tools/perf/scripts/python/bin/stackcollapse-record
> @@ -0,0 +1,8 @@
> +#!/bin/sh
> +
> +#
> +# stackcollapse.py can cover all type of perf samples including
> +# the tracepoints, so no special record requirements, just record what
> +# you want to analyze.
> +#
> +perf record "$@"
> diff --git a/tools/perf/scripts/python/bin/stackcollapse-report b/tools/perf/scripts/python/bin/stackcollapse-report
> new file mode 100755
> index 000000000000..356b9656393d
> --- /dev/null
> +++ b/tools/perf/scripts/python/bin/stackcollapse-report
> @@ -0,0 +1,3 @@
> +#!/bin/sh
> +# description: produce callgraphs in short form for scripting use
> +perf script -s "$PERF_EXEC_PATH"/scripts/python/stackcollapse.py -- "$@"
> diff --git a/tools/perf/scripts/python/stackcollapse.py b/tools/perf/scripts/python/stackcollapse.py
> new file mode 100755
> index 000000000000..a2dfcda41ae6
> --- /dev/null
> +++ b/tools/perf/scripts/python/stackcollapse.py
> @@ -0,0 +1,127 @@
> +#!/usr/bin/perl -w
> +#
> +# stackcollapse.py - format perf samples with one line per distinct call stack
> +#
> +# This script's output has two space-separated fields.  The first is a semicolon
> +# separated stack including the program name (from the "comm" field) and the
> +# function names from the call stack.  The second is a count:
> +#
> +#  swapper;start_kernel;rest_init;cpu_idle;default_idle;native_safe_halt 2
> +#
> +# The file is sorted according to the first field.
> +#
> +# Input may be created and processed using:
> +#
> +#  perf record -a -g -F 99 sleep 60
> +#  perf script report stackcollapse > out.stacks-folded
> +#
> +# (perf script record stackcollapse works too).
> +#
> +# Written by Paolo Bonzini <pbonzini@redhat.com>
> +# Based on Brendan Gregg's stackcollapse-perf.pl script.
> +
> +import os
> +import sys
> +from collections import defaultdict
> +from optparse import OptionParser, make_option
> +
> +sys.path.append(os.environ['PERF_EXEC_PATH'] + \
> +                '/scripts/python/Perf-Trace-Util/lib/Perf/Trace')
> +
> +from perf_trace_context import *
> +from Core import *
> +from EventClass import *
> +
> +# command line parsing
> +
> +option_list = [
> +    # formatting options for the bottom entry of the stack
> +    make_option("--include-tid", dest="include_tid",
> +                 action="store_true", default=False,
> +                 help="include thread id in stack"),
> +    make_option("--include-pid", dest="include_pid",
> +                 action="store_true", default=False,
> +                 help="include process id in stack"),
> +    make_option("--no-comm", dest="include_comm",
> +                 action="store_false", default=True,
> +                 help="do not separate stacks according to comm"),
> +    make_option("--tidy-java", dest="tidy_java",
> +                 action="store_true", default=False,
> +                 help="beautify Java signatures"),
> +    make_option("--kernel", dest="annotate_kernel",
> +                 action="store_true", default=False,
> +                 help="annotate kernel functions with _[k]")
> +]
> +
> +parser = OptionParser(option_list=option_list)
> +(opts, args) = parser.parse_args()
> +
> +if len(args) != 0:
> +    parser.error("unexpected command line argument")
> +if opts.include_tid and not opts.include_comm:
> +    parser.error("requesting tid but not comm is invalid")
> +if opts.include_pid and not opts.include_comm:
> +    parser.error("requesting pid but not comm is invalid")
> +
> +# event handlers
> +
> +lines = defaultdict(lambda: 0)
> +
> +def process_event(param_dict):
> +    def tidy_function_name(sym, dso):
> +        if sym is None:
> +            sym = '[unknown]'
> +
> +        sym = sym.replace(';', ':')
> +        if opts.tidy_java:
> +            # the original stackcollapse-perf.pl script gives the
> +            # example of converting this:
> +            #    Lorg/mozilla/javascript/MemberBox;.<init>(Ljava/lang/reflect/Method;)V
> +            # to this:
> +            #    org/mozilla/javascript/MemberBox:.init
> +            sym = sym.replace('<', '')
> +            sym = sym.replace('>', '')
> +            if sym[0] == 'L' and sym.find('/'):
> +                sym = sym[1:]
> +            try:
> +                sym = sym[:sym.index('(')]
> +            except ValueError:
> +                pass
> +
> +        if opts.annotate_kernel and dso == '[kernel.kallsyms]':
> +            return sym + '_[k]'
> +        else:
> +            return sym
> +
> +    stack = list()
> +    if 'callchain' in param_dict:
> +        for entry in param_dict['callchain']:
> +            entry.setdefault('sym', dict())
> +            entry['sym'].setdefault('name', None)
> +            entry.setdefault('dso', None)
> +            stack.append(tidy_function_name(entry['sym']['name'],
> +                                            entry['dso']))
> +    else:
> +        param_dict.setdefault('symbol', None)
> +        param_dict.setdefault('dso', None)
> +        stack.append(tidy_function_name(param_dict['symbol'],
> +                                        param_dict['dso']))
> +
> +    if opts.include_comm:
> +        comm = param_dict["comm"].replace(' ', '_')
> +        sep = "-"
> +        if opts.include_pid:
> +            comm = comm + sep + str(param_dict['sample']['pid'])
> +            sep = "/"
> +        if opts.include_tid:
> +            comm = comm + sep + str(param_dict['sample']['tid'])
> +        stack.append(comm)
> +
> +    stack_string = ';'.join(reversed(stack))
> +    lines[stack_string] = lines[stack_string] + 1
> +
> +def trace_end():
> +    list = lines.keys()
> +    list.sort()
> +    for stack in list:
> +        print "%s %d" % (stack, lines[stack])
> 

^ permalink raw reply	[flat|nested] 10+ messages in thread

* Re: ping Re: [PATCH] perf script: Add stackcollapse.py script
  2016-04-15  5:20 ` ping " Paolo Bonzini
@ 2016-04-15 10:57   ` Jiri Olsa
  2016-04-15 11:09     ` Paolo Bonzini
  2016-05-20 11:01     ` Paolo Bonzini
  0 siblings, 2 replies; 10+ messages in thread
From: Jiri Olsa @ 2016-04-15 10:57 UTC (permalink / raw)
  To: Paolo Bonzini
  Cc: linux-kernel, Jiri Olsa, Arnaldo Carvalho de Melo, Brendan Gregg

On Fri, Apr 15, 2016 at 07:20:48AM +0200, Paolo Bonzini wrote:
> On 12/04/2016 15:26, Paolo Bonzini wrote:
> > Add stackcollapse.py script as an example of parsing call chains, and
> > also of using optparse to access command line options.
> > 
> > The flame graph tools include a set of scripts that parse output from
> > various tools (including "perf script"), remove the offsets in the
> > function and collapse each stack to a single line.  The website also says
> > "perf report could have a report style [...] that output folded stacks
> > directly, obviating the need for stackcollapse-perf.pl", so here it is.
> > 
> > This script is a Python rewrite of stackcollapse-perf.pl, using the perf
> > scripting interface to access the perf data directly from Python.
> > 
> > Cc: Jiri Olsa <jolsa@kernel.org>
> > Cc: Arnaldo Carvalho de Melo <acme@redhat.com>
> > Cc: Brendan Gregg <bgregg@netflix.com>
> > Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
> 
> Sorry for the very early ping, I'm going on vacation and I'm afraid the
> next time I'd be able to ping would be too late for 4.7. :)
> 
> Paolo
> 
> > ---
> >  tools/perf/scripts/python/bin/stackcollapse-record |   8 ++
> >  tools/perf/scripts/python/bin/stackcollapse-report |   3 +
> >  tools/perf/scripts/python/stackcollapse.py         | 127 +++++++++++++++++++++
> >  3 files changed, 138 insertions(+)
> >  create mode 100755 tools/perf/scripts/python/bin/stackcollapse-record
> >  create mode 100755 tools/perf/scripts/python/bin/stackcollapse-report
> >  create mode 100755 tools/perf/scripts/python/stackcollapse.py
> > 
> > diff --git a/tools/perf/scripts/python/bin/stackcollapse-record b/tools/perf/scripts/python/bin/stackcollapse-record
> > new file mode 100755
> > index 000000000000..9d8f9f0f3a17
> > --- /dev/null
> > +++ b/tools/perf/scripts/python/bin/stackcollapse-record
> > @@ -0,0 +1,8 @@
> > +#!/bin/sh
> > +
> > +#
> > +# stackcollapse.py can cover all type of perf samples including
> > +# the tracepoints, so no special record requirements, just record what
> > +# you want to analyze.
> > +#
> > +perf record "$@"
> > diff --git a/tools/perf/scripts/python/bin/stackcollapse-report b/tools/perf/scripts/python/bin/stackcollapse-report
> > new file mode 100755
> > index 000000000000..356b9656393d
> > --- /dev/null
> > +++ b/tools/perf/scripts/python/bin/stackcollapse-report
> > @@ -0,0 +1,3 @@
> > +#!/bin/sh
> > +# description: produce callgraphs in short form for scripting use
> > +perf script -s "$PERF_EXEC_PATH"/scripts/python/stackcollapse.py -- "$@"
> > diff --git a/tools/perf/scripts/python/stackcollapse.py b/tools/perf/scripts/python/stackcollapse.py
> > new file mode 100755
> > index 000000000000..a2dfcda41ae6
> > --- /dev/null
> > +++ b/tools/perf/scripts/python/stackcollapse.py
> > @@ -0,0 +1,127 @@
> > +#!/usr/bin/perl -w
> > +#
> > +# stackcollapse.py - format perf samples with one line per distinct call stack
> > +#
> > +# This script's output has two space-separated fields.  The first is a semicolon
> > +# separated stack including the program name (from the "comm" field) and the
> > +# function names from the call stack.  The second is a count:
> > +#
> > +#  swapper;start_kernel;rest_init;cpu_idle;default_idle;native_safe_halt 2
> > +#
> > +# The file is sorted according to the first field.
> > +#
> > +# Input may be created and processed using:
> > +#
> > +#  perf record -a -g -F 99 sleep 60
> > +#  perf script report stackcollapse > out.stacks-folded
> > +#
> > +# (perf script record stackcollapse works too).

IIRC Namhyung added -g folded option recently for report
so you could do:

perf report -g folded --stdio

however we dont seem to have it for perf script, so this might
be useful until we add the --call-graph support into perf script

jirka

> > +#
> > +# Written by Paolo Bonzini <pbonzini@redhat.com>
> > +# Based on Brendan Gregg's stackcollapse-perf.pl script.
> > +
> > +import os
> > +import sys
> > +from collections import defaultdict
> > +from optparse import OptionParser, make_option
> > +
> > +sys.path.append(os.environ['PERF_EXEC_PATH'] + \
> > +                '/scripts/python/Perf-Trace-Util/lib/Perf/Trace')
> > +
> > +from perf_trace_context import *
> > +from Core import *
> > +from EventClass import *
> > +
> > +# command line parsing
> > +
> > +option_list = [
> > +    # formatting options for the bottom entry of the stack
> > +    make_option("--include-tid", dest="include_tid",
> > +                 action="store_true", default=False,
> > +                 help="include thread id in stack"),
> > +    make_option("--include-pid", dest="include_pid",
> > +                 action="store_true", default=False,
> > +                 help="include process id in stack"),
> > +    make_option("--no-comm", dest="include_comm",
> > +                 action="store_false", default=True,
> > +                 help="do not separate stacks according to comm"),
> > +    make_option("--tidy-java", dest="tidy_java",
> > +                 action="store_true", default=False,
> > +                 help="beautify Java signatures"),
> > +    make_option("--kernel", dest="annotate_kernel",
> > +                 action="store_true", default=False,
> > +                 help="annotate kernel functions with _[k]")
> > +]
> > +
> > +parser = OptionParser(option_list=option_list)
> > +(opts, args) = parser.parse_args()
> > +
> > +if len(args) != 0:
> > +    parser.error("unexpected command line argument")
> > +if opts.include_tid and not opts.include_comm:
> > +    parser.error("requesting tid but not comm is invalid")
> > +if opts.include_pid and not opts.include_comm:
> > +    parser.error("requesting pid but not comm is invalid")
> > +
> > +# event handlers
> > +
> > +lines = defaultdict(lambda: 0)
> > +
> > +def process_event(param_dict):
> > +    def tidy_function_name(sym, dso):
> > +        if sym is None:
> > +            sym = '[unknown]'
> > +
> > +        sym = sym.replace(';', ':')
> > +        if opts.tidy_java:
> > +            # the original stackcollapse-perf.pl script gives the
> > +            # example of converting this:
> > +            #    Lorg/mozilla/javascript/MemberBox;.<init>(Ljava/lang/reflect/Method;)V
> > +            # to this:
> > +            #    org/mozilla/javascript/MemberBox:.init
> > +            sym = sym.replace('<', '')
> > +            sym = sym.replace('>', '')
> > +            if sym[0] == 'L' and sym.find('/'):
> > +                sym = sym[1:]
> > +            try:
> > +                sym = sym[:sym.index('(')]
> > +            except ValueError:
> > +                pass
> > +
> > +        if opts.annotate_kernel and dso == '[kernel.kallsyms]':
> > +            return sym + '_[k]'
> > +        else:
> > +            return sym
> > +
> > +    stack = list()
> > +    if 'callchain' in param_dict:
> > +        for entry in param_dict['callchain']:
> > +            entry.setdefault('sym', dict())
> > +            entry['sym'].setdefault('name', None)
> > +            entry.setdefault('dso', None)
> > +            stack.append(tidy_function_name(entry['sym']['name'],
> > +                                            entry['dso']))
> > +    else:
> > +        param_dict.setdefault('symbol', None)
> > +        param_dict.setdefault('dso', None)
> > +        stack.append(tidy_function_name(param_dict['symbol'],
> > +                                        param_dict['dso']))
> > +
> > +    if opts.include_comm:
> > +        comm = param_dict["comm"].replace(' ', '_')
> > +        sep = "-"
> > +        if opts.include_pid:
> > +            comm = comm + sep + str(param_dict['sample']['pid'])
> > +            sep = "/"
> > +        if opts.include_tid:
> > +            comm = comm + sep + str(param_dict['sample']['tid'])
> > +        stack.append(comm)
> > +
> > +    stack_string = ';'.join(reversed(stack))
> > +    lines[stack_string] = lines[stack_string] + 1
> > +
> > +def trace_end():
> > +    list = lines.keys()
> > +    list.sort()
> > +    for stack in list:
> > +        print "%s %d" % (stack, lines[stack])
> > 

^ permalink raw reply	[flat|nested] 10+ messages in thread

* Re: ping Re: [PATCH] perf script: Add stackcollapse.py script
  2016-04-15 10:57   ` Jiri Olsa
@ 2016-04-15 11:09     ` Paolo Bonzini
  2016-05-20 11:01     ` Paolo Bonzini
  1 sibling, 0 replies; 10+ messages in thread
From: Paolo Bonzini @ 2016-04-15 11:09 UTC (permalink / raw)
  To: Jiri Olsa
  Cc: linux-kernel, Jiri Olsa, Arnaldo Carvalho de Melo, Brendan Gregg



On 15/04/2016 12:57, Jiri Olsa wrote:
>>> > > +# Input may be created and processed using:
>>> > > +#
>>> > > +#  perf record -a -g -F 99 sleep 60
>>> > > +#  perf script report stackcollapse > out.stacks-folded
>>> > > +#
>>> > > +# (perf script record stackcollapse works too).
> IIRC Namhyung added -g folded option recently for report
> so you could do:
> 
> perf report -g folded --stdio
> 
> however we dont seem to have it for perf script, so this might
> be useful until we add the --call-graph support into perf script

Thanks for the pointer.  I think the output of "perf script -g folded" 
would still be a bit different from what the flamegraph tool expects.  
Something like

perf 13675 51165.817380:       1921 cycles:pp: native_write_cr2; do_nmi; end_repeat_nmi; ...

rather than this

perf;native_write_cr2;do_nmi;end_repeat_nmi;... 1921

which this script produces.  The format that flamegraph expects is very
simple so that they can have converters for perf, stap, dtrace and others.

I'd be just as happy if perf looked for scripts into $HOME/.local. :)  But
since it doesn't, this script can also be useful to demonstrate script access
to callchains.

Thanks,

Paolo

^ permalink raw reply	[flat|nested] 10+ messages in thread

* Re: ping Re: [PATCH] perf script: Add stackcollapse.py script
  2016-04-15 10:57   ` Jiri Olsa
  2016-04-15 11:09     ` Paolo Bonzini
@ 2016-05-20 11:01     ` Paolo Bonzini
  2016-06-16  7:49       ` Jiri Olsa
  1 sibling, 1 reply; 10+ messages in thread
From: Paolo Bonzini @ 2016-05-20 11:01 UTC (permalink / raw)
  To: Jiri Olsa
  Cc: linux-kernel, Jiri Olsa, Arnaldo Carvalho de Melo, Brendan Gregg



On 15/04/2016 12:57, Jiri Olsa wrote:
> On Fri, Apr 15, 2016 at 07:20:48AM +0200, Paolo Bonzini wrote:
>> On 12/04/2016 15:26, Paolo Bonzini wrote:
>>> Add stackcollapse.py script as an example of parsing call chains, and
>>> also of using optparse to access command line options.
>>>
>>> The flame graph tools include a set of scripts that parse output from
>>> various tools (including "perf script"), remove the offsets in the
>>> function and collapse each stack to a single line.  The website also says
>>> "perf report could have a report style [...] that output folded stacks
>>> directly, obviating the need for stackcollapse-perf.pl", so here it is.
>>>
>>> This script is a Python rewrite of stackcollapse-perf.pl, using the perf
>>> scripting interface to access the perf data directly from Python.
>>>
>>> Cc: Jiri Olsa <jolsa@kernel.org>
>>> Cc: Arnaldo Carvalho de Melo <acme@redhat.com>
>>> Cc: Brendan Gregg <bgregg@netflix.com>
>>> Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
>>
>> Sorry for the very early ping, I'm going on vacation and I'm afraid the
>> next time I'd be able to ping would be too late for 4.7. :)
>>
>> Paolo
>>
>>> ---
>>>  tools/perf/scripts/python/bin/stackcollapse-record |   8 ++
>>>  tools/perf/scripts/python/bin/stackcollapse-report |   3 +
>>>  tools/perf/scripts/python/stackcollapse.py         | 127 +++++++++++++++++++++
>>>  3 files changed, 138 insertions(+)
>>>  create mode 100755 tools/perf/scripts/python/bin/stackcollapse-record
>>>  create mode 100755 tools/perf/scripts/python/bin/stackcollapse-report
>>>  create mode 100755 tools/perf/scripts/python/stackcollapse.py
>>>
>>> diff --git a/tools/perf/scripts/python/bin/stackcollapse-record b/tools/perf/scripts/python/bin/stackcollapse-record
>>> new file mode 100755
>>> index 000000000000..9d8f9f0f3a17
>>> --- /dev/null
>>> +++ b/tools/perf/scripts/python/bin/stackcollapse-record
>>> @@ -0,0 +1,8 @@
>>> +#!/bin/sh
>>> +
>>> +#
>>> +# stackcollapse.py can cover all type of perf samples including
>>> +# the tracepoints, so no special record requirements, just record what
>>> +# you want to analyze.
>>> +#
>>> +perf record "$@"
>>> diff --git a/tools/perf/scripts/python/bin/stackcollapse-report b/tools/perf/scripts/python/bin/stackcollapse-report
>>> new file mode 100755
>>> index 000000000000..356b9656393d
>>> --- /dev/null
>>> +++ b/tools/perf/scripts/python/bin/stackcollapse-report
>>> @@ -0,0 +1,3 @@
>>> +#!/bin/sh
>>> +# description: produce callgraphs in short form for scripting use
>>> +perf script -s "$PERF_EXEC_PATH"/scripts/python/stackcollapse.py -- "$@"
>>> diff --git a/tools/perf/scripts/python/stackcollapse.py b/tools/perf/scripts/python/stackcollapse.py
>>> new file mode 100755
>>> index 000000000000..a2dfcda41ae6
>>> --- /dev/null
>>> +++ b/tools/perf/scripts/python/stackcollapse.py
>>> @@ -0,0 +1,127 @@
>>> +#!/usr/bin/perl -w
>>> +#
>>> +# stackcollapse.py - format perf samples with one line per distinct call stack
>>> +#
>>> +# This script's output has two space-separated fields.  The first is a semicolon
>>> +# separated stack including the program name (from the "comm" field) and the
>>> +# function names from the call stack.  The second is a count:
>>> +#
>>> +#  swapper;start_kernel;rest_init;cpu_idle;default_idle;native_safe_halt 2
>>> +#
>>> +# The file is sorted according to the first field.
>>> +#
>>> +# Input may be created and processed using:
>>> +#
>>> +#  perf record -a -g -F 99 sleep 60
>>> +#  perf script report stackcollapse > out.stacks-folded
>>> +#
>>> +# (perf script record stackcollapse works too).
> 
> IIRC Namhyung added -g folded option recently for report
> so you could do:
> 
> perf report -g folded --stdio
> 
> however we dont seem to have it for perf script, so this might
> be useful until we add the --call-graph support into perf script

While "perf report -g folded" is indeed similar in spirit, it doesn't
provide exactly the same output as expected by the flame graph tools.
The point of this patch is to talk directly to them, and to provide an
example of looking at call stacks from Python.

Thanks,

Paolo


> jirka
> 
>>> +#
>>> +# Written by Paolo Bonzini <pbonzini@redhat.com>
>>> +# Based on Brendan Gregg's stackcollapse-perf.pl script.
>>> +
>>> +import os
>>> +import sys
>>> +from collections import defaultdict
>>> +from optparse import OptionParser, make_option
>>> +
>>> +sys.path.append(os.environ['PERF_EXEC_PATH'] + \
>>> +                '/scripts/python/Perf-Trace-Util/lib/Perf/Trace')
>>> +
>>> +from perf_trace_context import *
>>> +from Core import *
>>> +from EventClass import *
>>> +
>>> +# command line parsing
>>> +
>>> +option_list = [
>>> +    # formatting options for the bottom entry of the stack
>>> +    make_option("--include-tid", dest="include_tid",
>>> +                 action="store_true", default=False,
>>> +                 help="include thread id in stack"),
>>> +    make_option("--include-pid", dest="include_pid",
>>> +                 action="store_true", default=False,
>>> +                 help="include process id in stack"),
>>> +    make_option("--no-comm", dest="include_comm",
>>> +                 action="store_false", default=True,
>>> +                 help="do not separate stacks according to comm"),
>>> +    make_option("--tidy-java", dest="tidy_java",
>>> +                 action="store_true", default=False,
>>> +                 help="beautify Java signatures"),
>>> +    make_option("--kernel", dest="annotate_kernel",
>>> +                 action="store_true", default=False,
>>> +                 help="annotate kernel functions with _[k]")
>>> +]
>>> +
>>> +parser = OptionParser(option_list=option_list)
>>> +(opts, args) = parser.parse_args()
>>> +
>>> +if len(args) != 0:
>>> +    parser.error("unexpected command line argument")
>>> +if opts.include_tid and not opts.include_comm:
>>> +    parser.error("requesting tid but not comm is invalid")
>>> +if opts.include_pid and not opts.include_comm:
>>> +    parser.error("requesting pid but not comm is invalid")
>>> +
>>> +# event handlers
>>> +
>>> +lines = defaultdict(lambda: 0)
>>> +
>>> +def process_event(param_dict):
>>> +    def tidy_function_name(sym, dso):
>>> +        if sym is None:
>>> +            sym = '[unknown]'
>>> +
>>> +        sym = sym.replace(';', ':')
>>> +        if opts.tidy_java:
>>> +            # the original stackcollapse-perf.pl script gives the
>>> +            # example of converting this:
>>> +            #    Lorg/mozilla/javascript/MemberBox;.<init>(Ljava/lang/reflect/Method;)V
>>> +            # to this:
>>> +            #    org/mozilla/javascript/MemberBox:.init
>>> +            sym = sym.replace('<', '')
>>> +            sym = sym.replace('>', '')
>>> +            if sym[0] == 'L' and sym.find('/'):
>>> +                sym = sym[1:]
>>> +            try:
>>> +                sym = sym[:sym.index('(')]
>>> +            except ValueError:
>>> +                pass
>>> +
>>> +        if opts.annotate_kernel and dso == '[kernel.kallsyms]':
>>> +            return sym + '_[k]'
>>> +        else:
>>> +            return sym
>>> +
>>> +    stack = list()
>>> +    if 'callchain' in param_dict:
>>> +        for entry in param_dict['callchain']:
>>> +            entry.setdefault('sym', dict())
>>> +            entry['sym'].setdefault('name', None)
>>> +            entry.setdefault('dso', None)
>>> +            stack.append(tidy_function_name(entry['sym']['name'],
>>> +                                            entry['dso']))
>>> +    else:
>>> +        param_dict.setdefault('symbol', None)
>>> +        param_dict.setdefault('dso', None)
>>> +        stack.append(tidy_function_name(param_dict['symbol'],
>>> +                                        param_dict['dso']))
>>> +
>>> +    if opts.include_comm:
>>> +        comm = param_dict["comm"].replace(' ', '_')
>>> +        sep = "-"
>>> +        if opts.include_pid:
>>> +            comm = comm + sep + str(param_dict['sample']['pid'])
>>> +            sep = "/"
>>> +        if opts.include_tid:
>>> +            comm = comm + sep + str(param_dict['sample']['tid'])
>>> +        stack.append(comm)
>>> +
>>> +    stack_string = ';'.join(reversed(stack))
>>> +    lines[stack_string] = lines[stack_string] + 1
>>> +
>>> +def trace_end():
>>> +    list = lines.keys()
>>> +    list.sort()
>>> +    for stack in list:
>>> +        print "%s %d" % (stack, lines[stack])
>>>
> 

^ permalink raw reply	[flat|nested] 10+ messages in thread

* Re: [PATCH] perf script: Add stackcollapse.py script
  2016-04-12 13:26 [PATCH] perf script: Add stackcollapse.py script Paolo Bonzini
  2016-04-15  5:20 ` ping " Paolo Bonzini
@ 2016-06-16  7:22 ` Paolo Bonzini
  2016-06-16  7:49   ` Jiri Olsa
  2016-06-22  7:44 ` [tip:perf/core] " tip-bot for Paolo Bonzini
  2 siblings, 1 reply; 10+ messages in thread
From: Paolo Bonzini @ 2016-06-16  7:22 UTC (permalink / raw)
  To: linux-kernel; +Cc: Jiri Olsa, Arnaldo Carvalho de Melo, Brendan Gregg

On 12/04/2016 15:26, Paolo Bonzini wrote:
> Add stackcollapse.py script as an example of parsing call chains, and
> also of using optparse to access command line options.
> 
> The flame graph tools include a set of scripts that parse output from
> various tools (including "perf script"), remove the offsets in the
> function and collapse each stack to a single line.  The website also says
> "perf report could have a report style [...] that output folded stacks
> directly, obviating the need for stackcollapse-perf.pl", so here it is.
> 
> This script is a Python rewrite of stackcollapse-perf.pl, using the perf
> scripting interface to access the perf data directly from Python.
> 
> Cc: Jiri Olsa <jolsa@kernel.org>
> Cc: Arnaldo Carvalho de Melo <acme@redhat.com>
> Cc: Brendan Gregg <bgregg@netflix.com>
> Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
> ---
>  tools/perf/scripts/python/bin/stackcollapse-record |   8 ++
>  tools/perf/scripts/python/bin/stackcollapse-report |   3 +
>  tools/perf/scripts/python/stackcollapse.py         | 127 +++++++++++++++++++++
>  3 files changed, 138 insertions(+)
>  create mode 100755 tools/perf/scripts/python/bin/stackcollapse-record
>  create mode 100755 tools/perf/scripts/python/bin/stackcollapse-report
>  create mode 100755 tools/perf/scripts/python/stackcollapse.py

Ping^2?

Paolo

> diff --git a/tools/perf/scripts/python/bin/stackcollapse-record b/tools/perf/scripts/python/bin/stackcollapse-record
> new file mode 100755
> index 000000000000..9d8f9f0f3a17
> --- /dev/null
> +++ b/tools/perf/scripts/python/bin/stackcollapse-record
> @@ -0,0 +1,8 @@
> +#!/bin/sh
> +
> +#
> +# stackcollapse.py can cover all type of perf samples including
> +# the tracepoints, so no special record requirements, just record what
> +# you want to analyze.
> +#
> +perf record "$@"
> diff --git a/tools/perf/scripts/python/bin/stackcollapse-report b/tools/perf/scripts/python/bin/stackcollapse-report
> new file mode 100755
> index 000000000000..356b9656393d
> --- /dev/null
> +++ b/tools/perf/scripts/python/bin/stackcollapse-report
> @@ -0,0 +1,3 @@
> +#!/bin/sh
> +# description: produce callgraphs in short form for scripting use
> +perf script -s "$PERF_EXEC_PATH"/scripts/python/stackcollapse.py -- "$@"
> diff --git a/tools/perf/scripts/python/stackcollapse.py b/tools/perf/scripts/python/stackcollapse.py
> new file mode 100755
> index 000000000000..a2dfcda41ae6
> --- /dev/null
> +++ b/tools/perf/scripts/python/stackcollapse.py
> @@ -0,0 +1,127 @@
> +#!/usr/bin/perl -w
> +#
> +# stackcollapse.py - format perf samples with one line per distinct call stack
> +#
> +# This script's output has two space-separated fields.  The first is a semicolon
> +# separated stack including the program name (from the "comm" field) and the
> +# function names from the call stack.  The second is a count:
> +#
> +#  swapper;start_kernel;rest_init;cpu_idle;default_idle;native_safe_halt 2
> +#
> +# The file is sorted according to the first field.
> +#
> +# Input may be created and processed using:
> +#
> +#  perf record -a -g -F 99 sleep 60
> +#  perf script report stackcollapse > out.stacks-folded
> +#
> +# (perf script record stackcollapse works too).
> +#
> +# Written by Paolo Bonzini <pbonzini@redhat.com>
> +# Based on Brendan Gregg's stackcollapse-perf.pl script.
> +
> +import os
> +import sys
> +from collections import defaultdict
> +from optparse import OptionParser, make_option
> +
> +sys.path.append(os.environ['PERF_EXEC_PATH'] + \
> +                '/scripts/python/Perf-Trace-Util/lib/Perf/Trace')
> +
> +from perf_trace_context import *
> +from Core import *
> +from EventClass import *
> +
> +# command line parsing
> +
> +option_list = [
> +    # formatting options for the bottom entry of the stack
> +    make_option("--include-tid", dest="include_tid",
> +                 action="store_true", default=False,
> +                 help="include thread id in stack"),
> +    make_option("--include-pid", dest="include_pid",
> +                 action="store_true", default=False,
> +                 help="include process id in stack"),
> +    make_option("--no-comm", dest="include_comm",
> +                 action="store_false", default=True,
> +                 help="do not separate stacks according to comm"),
> +    make_option("--tidy-java", dest="tidy_java",
> +                 action="store_true", default=False,
> +                 help="beautify Java signatures"),
> +    make_option("--kernel", dest="annotate_kernel",
> +                 action="store_true", default=False,
> +                 help="annotate kernel functions with _[k]")
> +]
> +
> +parser = OptionParser(option_list=option_list)
> +(opts, args) = parser.parse_args()
> +
> +if len(args) != 0:
> +    parser.error("unexpected command line argument")
> +if opts.include_tid and not opts.include_comm:
> +    parser.error("requesting tid but not comm is invalid")
> +if opts.include_pid and not opts.include_comm:
> +    parser.error("requesting pid but not comm is invalid")
> +
> +# event handlers
> +
> +lines = defaultdict(lambda: 0)
> +
> +def process_event(param_dict):
> +    def tidy_function_name(sym, dso):
> +        if sym is None:
> +            sym = '[unknown]'
> +
> +        sym = sym.replace(';', ':')
> +        if opts.tidy_java:
> +            # the original stackcollapse-perf.pl script gives the
> +            # example of converting this:
> +            #    Lorg/mozilla/javascript/MemberBox;.<init>(Ljava/lang/reflect/Method;)V
> +            # to this:
> +            #    org/mozilla/javascript/MemberBox:.init
> +            sym = sym.replace('<', '')
> +            sym = sym.replace('>', '')
> +            if sym[0] == 'L' and sym.find('/'):
> +                sym = sym[1:]
> +            try:
> +                sym = sym[:sym.index('(')]
> +            except ValueError:
> +                pass
> +
> +        if opts.annotate_kernel and dso == '[kernel.kallsyms]':
> +            return sym + '_[k]'
> +        else:
> +            return sym
> +
> +    stack = list()
> +    if 'callchain' in param_dict:
> +        for entry in param_dict['callchain']:
> +            entry.setdefault('sym', dict())
> +            entry['sym'].setdefault('name', None)
> +            entry.setdefault('dso', None)
> +            stack.append(tidy_function_name(entry['sym']['name'],
> +                                            entry['dso']))
> +    else:
> +        param_dict.setdefault('symbol', None)
> +        param_dict.setdefault('dso', None)
> +        stack.append(tidy_function_name(param_dict['symbol'],
> +                                        param_dict['dso']))
> +
> +    if opts.include_comm:
> +        comm = param_dict["comm"].replace(' ', '_')
> +        sep = "-"
> +        if opts.include_pid:
> +            comm = comm + sep + str(param_dict['sample']['pid'])
> +            sep = "/"
> +        if opts.include_tid:
> +            comm = comm + sep + str(param_dict['sample']['tid'])
> +        stack.append(comm)
> +
> +    stack_string = ';'.join(reversed(stack))
> +    lines[stack_string] = lines[stack_string] + 1
> +
> +def trace_end():
> +    list = lines.keys()
> +    list.sort()
> +    for stack in list:
> +        print "%s %d" % (stack, lines[stack])
> 

^ permalink raw reply	[flat|nested] 10+ messages in thread

* Re: ping Re: [PATCH] perf script: Add stackcollapse.py script
  2016-05-20 11:01     ` Paolo Bonzini
@ 2016-06-16  7:49       ` Jiri Olsa
  0 siblings, 0 replies; 10+ messages in thread
From: Jiri Olsa @ 2016-06-16  7:49 UTC (permalink / raw)
  To: Paolo Bonzini
  Cc: linux-kernel, Jiri Olsa, Arnaldo Carvalho de Melo, Brendan Gregg

On Fri, May 20, 2016 at 01:01:41PM +0200, Paolo Bonzini wrote:
> 
> 

SNIP

> >>> new file mode 100755
> >>> index 000000000000..356b9656393d
> >>> --- /dev/null
> >>> +++ b/tools/perf/scripts/python/bin/stackcollapse-report
> >>> @@ -0,0 +1,3 @@
> >>> +#!/bin/sh
> >>> +# description: produce callgraphs in short form for scripting use
> >>> +perf script -s "$PERF_EXEC_PATH"/scripts/python/stackcollapse.py -- "$@"
> >>> diff --git a/tools/perf/scripts/python/stackcollapse.py b/tools/perf/scripts/python/stackcollapse.py
> >>> new file mode 100755
> >>> index 000000000000..a2dfcda41ae6
> >>> --- /dev/null
> >>> +++ b/tools/perf/scripts/python/stackcollapse.py
> >>> @@ -0,0 +1,127 @@
> >>> +#!/usr/bin/perl -w
> >>> +#
> >>> +# stackcollapse.py - format perf samples with one line per distinct call stack
> >>> +#
> >>> +# This script's output has two space-separated fields.  The first is a semicolon
> >>> +# separated stack including the program name (from the "comm" field) and the
> >>> +# function names from the call stack.  The second is a count:
> >>> +#
> >>> +#  swapper;start_kernel;rest_init;cpu_idle;default_idle;native_safe_halt 2
> >>> +#
> >>> +# The file is sorted according to the first field.
> >>> +#
> >>> +# Input may be created and processed using:
> >>> +#
> >>> +#  perf record -a -g -F 99 sleep 60
> >>> +#  perf script report stackcollapse > out.stacks-folded
> >>> +#
> >>> +# (perf script record stackcollapse works too).
> > 
> > IIRC Namhyung added -g folded option recently for report
> > so you could do:
> > 
> > perf report -g folded --stdio
> > 
> > however we dont seem to have it for perf script, so this might
> > be useful until we add the --call-graph support into perf script
> 
> While "perf report -g folded" is indeed similar in spirit, it doesn't
> provide exactly the same output as expected by the flame graph tools.
> The point of this patch is to talk directly to them, and to provide an
> example of looking at call stacks from Python.

as I said, while I rather see the --call-graph option support,
there's no harm in having this done by the script meanwhile

Acked-by: Jiri Olsa <jolsa@kernel.org>

jirka

^ permalink raw reply	[flat|nested] 10+ messages in thread

* Re: [PATCH] perf script: Add stackcollapse.py script
  2016-06-16  7:22 ` Paolo Bonzini
@ 2016-06-16  7:49   ` Jiri Olsa
  2016-06-16  9:26     ` Paolo Bonzini
  0 siblings, 1 reply; 10+ messages in thread
From: Jiri Olsa @ 2016-06-16  7:49 UTC (permalink / raw)
  To: Paolo Bonzini
  Cc: linux-kernel, Jiri Olsa, Arnaldo Carvalho de Melo, Brendan Gregg

On Thu, Jun 16, 2016 at 09:22:59AM +0200, Paolo Bonzini wrote:
> On 12/04/2016 15:26, Paolo Bonzini wrote:
> > Add stackcollapse.py script as an example of parsing call chains, and
> > also of using optparse to access command line options.
> > 
> > The flame graph tools include a set of scripts that parse output from
> > various tools (including "perf script"), remove the offsets in the
> > function and collapse each stack to a single line.  The website also says
> > "perf report could have a report style [...] that output folded stacks
> > directly, obviating the need for stackcollapse-perf.pl", so here it is.
> > 
> > This script is a Python rewrite of stackcollapse-perf.pl, using the perf
> > scripting interface to access the perf data directly from Python.
> > 
> > Cc: Jiri Olsa <jolsa@kernel.org>
> > Cc: Arnaldo Carvalho de Melo <acme@redhat.com>
> > Cc: Brendan Gregg <bgregg@netflix.com>
> > Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
> > ---
> >  tools/perf/scripts/python/bin/stackcollapse-record |   8 ++
> >  tools/perf/scripts/python/bin/stackcollapse-report |   3 +
> >  tools/perf/scripts/python/stackcollapse.py         | 127 +++++++++++++++++++++
> >  3 files changed, 138 insertions(+)
> >  create mode 100755 tools/perf/scripts/python/bin/stackcollapse-record
> >  create mode 100755 tools/perf/scripts/python/bin/stackcollapse-report
> >  create mode 100755 tools/perf/scripts/python/stackcollapse.py
> 
> Ping^2?

sorry for delay.. I just answered the other email

jirka

^ permalink raw reply	[flat|nested] 10+ messages in thread

* Re: [PATCH] perf script: Add stackcollapse.py script
  2016-06-16  7:49   ` Jiri Olsa
@ 2016-06-16  9:26     ` Paolo Bonzini
  0 siblings, 0 replies; 10+ messages in thread
From: Paolo Bonzini @ 2016-06-16  9:26 UTC (permalink / raw)
  To: Jiri Olsa
  Cc: linux-kernel, Jiri Olsa, Arnaldo Carvalho de Melo, Brendan Gregg



On 16/06/2016 09:49, Jiri Olsa wrote:
> On Thu, Jun 16, 2016 at 09:22:59AM +0200, Paolo Bonzini wrote:
>> On 12/04/2016 15:26, Paolo Bonzini wrote:
>>> Add stackcollapse.py script as an example of parsing call chains, and
>>> also of using optparse to access command line options.
>>>
>>> The flame graph tools include a set of scripts that parse output from
>>> various tools (including "perf script"), remove the offsets in the
>>> function and collapse each stack to a single line.  The website also says
>>> "perf report could have a report style [...] that output folded stacks
>>> directly, obviating the need for stackcollapse-perf.pl", so here it is.
>>>
>>> This script is a Python rewrite of stackcollapse-perf.pl, using the perf
>>> scripting interface to access the perf data directly from Python.
>>>
>>> Cc: Jiri Olsa <jolsa@kernel.org>
>>> Cc: Arnaldo Carvalho de Melo <acme@redhat.com>
>>> Cc: Brendan Gregg <bgregg@netflix.com>
>>> Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
>>> ---
>>>  tools/perf/scripts/python/bin/stackcollapse-record |   8 ++
>>>  tools/perf/scripts/python/bin/stackcollapse-report |   3 +
>>>  tools/perf/scripts/python/stackcollapse.py         | 127 +++++++++++++++++++++
>>>  3 files changed, 138 insertions(+)
>>>  create mode 100755 tools/perf/scripts/python/bin/stackcollapse-record
>>>  create mode 100755 tools/perf/scripts/python/bin/stackcollapse-report
>>>  create mode 100755 tools/perf/scripts/python/stackcollapse.py
>>
>> Ping^2?
> 
> sorry for delay.. I just answered the other email

No problem, thank you very much!

Paolo

^ permalink raw reply	[flat|nested] 10+ messages in thread

* [tip:perf/core] perf script: Add stackcollapse.py script
  2016-04-12 13:26 [PATCH] perf script: Add stackcollapse.py script Paolo Bonzini
  2016-04-15  5:20 ` ping " Paolo Bonzini
  2016-06-16  7:22 ` Paolo Bonzini
@ 2016-06-22  7:44 ` tip-bot for Paolo Bonzini
  2 siblings, 0 replies; 10+ messages in thread
From: tip-bot for Paolo Bonzini @ 2016-06-22  7:44 UTC (permalink / raw)
  To: linux-tip-commits
  Cc: acme, linux-kernel, pbonzini, hpa, jolsa, tglx, bgregg, mingo

Commit-ID:  6745d8ea825966b0956c691cf7fccc13debedc39
Gitweb:     http://git.kernel.org/tip/6745d8ea825966b0956c691cf7fccc13debedc39
Author:     Paolo Bonzini <pbonzini@redhat.com>
AuthorDate: Tue, 12 Apr 2016 15:26:13 +0200
Committer:  Arnaldo Carvalho de Melo <acme@redhat.com>
CommitDate: Tue, 21 Jun 2016 13:18:35 -0300

perf script: Add stackcollapse.py script

Add stackcollapse.py script as an example of parsing call chains, and
also of using optparse to access command line options.

The flame graph tools include a set of scripts that parse output from
various tools (including "perf script"), remove the offsets in the
function and collapse each stack to a single line.  The website also
says "perf report could have a report style [...] that output folded
stacks directly, obviating the need for stackcollapse-perf.pl", so here
it is.

This script is a Python rewrite of stackcollapse-perf.pl, using the perf
scripting interface to access the perf data directly from Python.

Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
Acked-by: Jiri Olsa <jolsa@kernel.org>
Cc: Brendan Gregg <bgregg@netflix.com>
Link: http://lkml.kernel.org/r/1460467573-22989-1-git-send-email-pbonzini@redhat.com
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
---
 tools/perf/scripts/python/bin/stackcollapse-record |   8 ++
 tools/perf/scripts/python/bin/stackcollapse-report |   3 +
 tools/perf/scripts/python/stackcollapse.py         | 127 +++++++++++++++++++++
 3 files changed, 138 insertions(+)

diff --git a/tools/perf/scripts/python/bin/stackcollapse-record b/tools/perf/scripts/python/bin/stackcollapse-record
new file mode 100755
index 0000000..9d8f9f0
--- /dev/null
+++ b/tools/perf/scripts/python/bin/stackcollapse-record
@@ -0,0 +1,8 @@
+#!/bin/sh
+
+#
+# stackcollapse.py can cover all type of perf samples including
+# the tracepoints, so no special record requirements, just record what
+# you want to analyze.
+#
+perf record "$@"
diff --git a/tools/perf/scripts/python/bin/stackcollapse-report b/tools/perf/scripts/python/bin/stackcollapse-report
new file mode 100755
index 0000000..356b965
--- /dev/null
+++ b/tools/perf/scripts/python/bin/stackcollapse-report
@@ -0,0 +1,3 @@
+#!/bin/sh
+# description: produce callgraphs in short form for scripting use
+perf script -s "$PERF_EXEC_PATH"/scripts/python/stackcollapse.py -- "$@"
diff --git a/tools/perf/scripts/python/stackcollapse.py b/tools/perf/scripts/python/stackcollapse.py
new file mode 100755
index 0000000..a2dfcda
--- /dev/null
+++ b/tools/perf/scripts/python/stackcollapse.py
@@ -0,0 +1,127 @@
+#!/usr/bin/perl -w
+#
+# stackcollapse.py - format perf samples with one line per distinct call stack
+#
+# This script's output has two space-separated fields.  The first is a semicolon
+# separated stack including the program name (from the "comm" field) and the
+# function names from the call stack.  The second is a count:
+#
+#  swapper;start_kernel;rest_init;cpu_idle;default_idle;native_safe_halt 2
+#
+# The file is sorted according to the first field.
+#
+# Input may be created and processed using:
+#
+#  perf record -a -g -F 99 sleep 60
+#  perf script report stackcollapse > out.stacks-folded
+#
+# (perf script record stackcollapse works too).
+#
+# Written by Paolo Bonzini <pbonzini@redhat.com>
+# Based on Brendan Gregg's stackcollapse-perf.pl script.
+
+import os
+import sys
+from collections import defaultdict
+from optparse import OptionParser, make_option
+
+sys.path.append(os.environ['PERF_EXEC_PATH'] + \
+                '/scripts/python/Perf-Trace-Util/lib/Perf/Trace')
+
+from perf_trace_context import *
+from Core import *
+from EventClass import *
+
+# command line parsing
+
+option_list = [
+    # formatting options for the bottom entry of the stack
+    make_option("--include-tid", dest="include_tid",
+                 action="store_true", default=False,
+                 help="include thread id in stack"),
+    make_option("--include-pid", dest="include_pid",
+                 action="store_true", default=False,
+                 help="include process id in stack"),
+    make_option("--no-comm", dest="include_comm",
+                 action="store_false", default=True,
+                 help="do not separate stacks according to comm"),
+    make_option("--tidy-java", dest="tidy_java",
+                 action="store_true", default=False,
+                 help="beautify Java signatures"),
+    make_option("--kernel", dest="annotate_kernel",
+                 action="store_true", default=False,
+                 help="annotate kernel functions with _[k]")
+]
+
+parser = OptionParser(option_list=option_list)
+(opts, args) = parser.parse_args()
+
+if len(args) != 0:
+    parser.error("unexpected command line argument")
+if opts.include_tid and not opts.include_comm:
+    parser.error("requesting tid but not comm is invalid")
+if opts.include_pid and not opts.include_comm:
+    parser.error("requesting pid but not comm is invalid")
+
+# event handlers
+
+lines = defaultdict(lambda: 0)
+
+def process_event(param_dict):
+    def tidy_function_name(sym, dso):
+        if sym is None:
+            sym = '[unknown]'
+
+        sym = sym.replace(';', ':')
+        if opts.tidy_java:
+            # the original stackcollapse-perf.pl script gives the
+            # example of converting this:
+            #    Lorg/mozilla/javascript/MemberBox;.<init>(Ljava/lang/reflect/Method;)V
+            # to this:
+            #    org/mozilla/javascript/MemberBox:.init
+            sym = sym.replace('<', '')
+            sym = sym.replace('>', '')
+            if sym[0] == 'L' and sym.find('/'):
+                sym = sym[1:]
+            try:
+                sym = sym[:sym.index('(')]
+            except ValueError:
+                pass
+
+        if opts.annotate_kernel and dso == '[kernel.kallsyms]':
+            return sym + '_[k]'
+        else:
+            return sym
+
+    stack = list()
+    if 'callchain' in param_dict:
+        for entry in param_dict['callchain']:
+            entry.setdefault('sym', dict())
+            entry['sym'].setdefault('name', None)
+            entry.setdefault('dso', None)
+            stack.append(tidy_function_name(entry['sym']['name'],
+                                            entry['dso']))
+    else:
+        param_dict.setdefault('symbol', None)
+        param_dict.setdefault('dso', None)
+        stack.append(tidy_function_name(param_dict['symbol'],
+                                        param_dict['dso']))
+
+    if opts.include_comm:
+        comm = param_dict["comm"].replace(' ', '_')
+        sep = "-"
+        if opts.include_pid:
+            comm = comm + sep + str(param_dict['sample']['pid'])
+            sep = "/"
+        if opts.include_tid:
+            comm = comm + sep + str(param_dict['sample']['tid'])
+        stack.append(comm)
+
+    stack_string = ';'.join(reversed(stack))
+    lines[stack_string] = lines[stack_string] + 1
+
+def trace_end():
+    list = lines.keys()
+    list.sort()
+    for stack in list:
+        print "%s %d" % (stack, lines[stack])

^ permalink raw reply	[flat|nested] 10+ messages in thread

end of thread, other threads:[~2016-06-22  7:44 UTC | newest]

Thread overview: 10+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2016-04-12 13:26 [PATCH] perf script: Add stackcollapse.py script Paolo Bonzini
2016-04-15  5:20 ` ping " Paolo Bonzini
2016-04-15 10:57   ` Jiri Olsa
2016-04-15 11:09     ` Paolo Bonzini
2016-05-20 11:01     ` Paolo Bonzini
2016-06-16  7:49       ` Jiri Olsa
2016-06-16  7:22 ` Paolo Bonzini
2016-06-16  7:49   ` Jiri Olsa
2016-06-16  9:26     ` Paolo Bonzini
2016-06-22  7:44 ` [tip:perf/core] " tip-bot for Paolo Bonzini

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).