All of lore.kernel.org
 help / color / mirror / Atom feed
* [RFC 00/47] perf stat: Add scripting support
@ 2015-07-21 12:31 Jiri Olsa
  2015-07-21 12:31 ` [PATCH 01/47] perf test: Check for refcnt in thread_map test Jiri Olsa
                   ` (48 more replies)
  0 siblings, 49 replies; 72+ messages in thread
From: Jiri Olsa @ 2015-07-21 12:31 UTC (permalink / raw)
  To: Arnaldo Carvalho de Melo
  Cc: Andi Kleen, Ulrich Drepper, Will Deacon, Stephane Eranian, lkml,
	David Ahern, Ingo Molnar, Namhyung Kim, Peter Zijlstra

hi,
sending RFC on another attempt for stat scripting.

The initial attempt defined its own formula lang and allowed
triggering user's script on the end of the stat command:
  http://marc.info/?l=linux-kernel&m=136742146322273&w=2

This patchset abandons the idea of new formula language
and rather adds support to:
  - store stat data into perf.data file
  - add python support to process stat events

Basically it allows to store stat data into perf.data and
post process it with python scripts in a similar way we
do for sampling data.

Examples:

 To record data for command stat workload:
  $ perf stat record kill
  ...

   Performance counter stats for 'kill':

            0.372007      task-clock (msec)         #    0.613 CPUs utilized          
                   3      context-switches          #    0.008 M/sec                  
                   0      cpu-migrations            #    0.000 K/sec                  
                  62      page-faults               #    0.167 M/sec                  
           1,129,973      cycles                    #    3.038 GHz                    
     <not supported>      stalled-cycles-frontend  
     <not supported>      stalled-cycles-backend   
             813,313      instructions              #    0.72  insns per cycle        
             166,161      branches                  #  446.661 M/sec                  
               8,747      branch-misses             #    5.26% of all branches        

         0.000607287 seconds time elapsed

 To report perf stat data:
  $ perf stat report

   Performance counter stats for '/home/jolsa/bin/perf stat record kill':

            0.372007      task-clock (msec)         #      inf CPUs utilized          
                   3      context-switches          #    0.008 M/sec                  
                   0      cpu-migrations            #    0.000 K/sec                  
                  62      page-faults               #    0.167 M/sec                  
           1,129,973      cycles                    #    3.038 GHz                    
     <not supported>      stalled-cycles-frontend  
     <not supported>      stalled-cycles-backend   
             813,313      instructions              #    0.72  insns per cycle        
             166,161      branches                  #  446.661 M/sec                  
               8,747      branch-misses             #    5.26% of all branches        

         0.000000000 seconds time elapsed

 To store system-wide period stat data:
  $ perf stat -e cycles:u,instructions:u -a -I 1000 record
  #           time             counts unit events
       1.000265471        462,311,482      cycles:u                   (100.00%)
       1.000265471        590,037,440      instructions:u           
       2.000483453        722,532,336      cycles:u                   (100.00%)
       2.000483453        848,678,197      instructions:u           
       3.000759876         75,990,880      cycles:u                   (100.00%)
       3.000759876         86,187,813      instructions:u           
  ^C     3.213960893         85,329,533      cycles:u                   (100.00%)
       3.213960893        135,954,296      instructions:u           

 To report perf stat data:
  $ perf stat report
  #           time             counts unit events
       1.000265471        462,311,482      cycles:u                   (100.00%)
       1.000265471        590,037,440      instructions:u           
       2.000483453        722,532,336      cycles:u                   (100.00%)
       2.000483453        848,678,197      instructions:u           
       3.000759876         75,990,880      cycles:u                   (100.00%)
       3.000759876         86,187,813      instructions:u           
       3.213960893         85,329,533      cycles:u                   (100.00%)
       3.213960893        135,954,296      instructions:u           

 To run stat-cpi.py script over perf.data:
  $ perf script -s scripts/python/stat-cpi.py 
         1.000265: cpu -1, thread -1 -> cpi 0.783529 (462311482/590037440)
         2.000483: cpu -1, thread -1 -> cpi 0.851362 (722532336/848678197)
         3.000760: cpu -1, thread -1 -> cpi 0.881689 (75990880/86187813)
         3.213961: cpu -1, thread -1 -> cpi 0.627634 (85329533/135954296)

 To pipe data from stat to stat-cpi script:
  $ perf stat -e cycles:u,instructions:u -A -C 0 -I 1000 record | perf script -s scripts/python/stat-cpi.py 
         1.000192: cpu 0, thread -1 -> cpi 0.739535 (23921908/32347236)
         2.000376: cpu 0, thread -1 -> cpi 1.663482 (2519340/1514498)
         3.000621: cpu 0, thread -1 -> cpi 1.396308 (16162767/11575362)
         4.000700: cpu 0, thread -1 -> cpi 1.092246 (20077258/18381624)
         5.000867: cpu 0, thread -1 -> cpi 0.473816 (45157586/95306156)
         6.001034: cpu 0, thread -1 -> cpi 0.532792 (43701668/82023818)
         7.001195: cpu 0, thread -1 -> cpi 1.122059 (29890042/26638561)

 Raw script stat data output:
  $ perf stat -e cycles:u,instructions:u -A -C 0 -I 1000 record | perf --no-pager script
  CPU   THREAD             VAL             ENA             RUN            TIME EVENT
    0       -1        12302059      1000811347      1000810712      1000198821 cycles:u
    0       -1         2565362      1000823218      1000823218      1000198821 instructions:u
    0       -1        14453353      1000812704      1000812704      2000382283 cycles:u
    0       -1         4600932      1000799342      1000799342      2000382283 instructions:u
    0       -1        15245106      1000774425      1000774425      3000538255 cycles:u
    0       -1         2624324      1000769310      1000769310      3000538255 instructions:u


The stat data are stored in new stat, stat-round, stat-config user events.
  stat        - stored for each read syscall of the counter
  stat round  - stored for each interval or end of the command invocation
  stat config - stores all the config information needed to process data
                so report tool could restore the same output as record

The python script can now define 'stat__<eventname>_<modifier>' functions
to get stat events data and 'stat__interval' to get stat-round data. See
CPI script example in scripts/python/stat-cpi.py.

Also available in:
  git://git.kernel.org/pub/scm/linux/kernel/git/jolsa/perf.git
  perf/stat_script


This patchset still has MANY rough edges and loose ends,
but I think it's better approach than to define our own
formula scripting language.

The patchset is growing and there's still a long way to go,
so I'd like to hear some opinions before I go further ;-)


thanks for comments,
jirka


Cc: Andi Kleen <andi@firstfloor.org>
Cc: Ulrich Drepper <drepper@gmail.com>
Cc: Will Deacon <will.deacon@arm.com>
Cc: Stephane Eranian <eranian@google.com>
---
Jiri Olsa (47):
      perf test: Check for refcnt in thread_map test
      perf stat: Introduce struct perf_stat_config
      perf stat: Move scale into struct perf_stat_config
      perf stat: Move output into struct perf_stat_config
      perf stat: Move interval into struct perf_stat_config
      perf stat: Pass struct perf_stat_config into process_counter
      perf stat: Move counter processing code into stat object
      perf tools: Use bool instead of target argument in perf_evlist__propagate_maps
      perf tools: Tolerate NULL maps in perf_evlist__propagate_maps
      perf tools: Force perf_evlist__set_maps to propagate maps through events
      perf tools: Use argv style storage for cmdline feature data
      perf tools: Add thread_map event
      perf tools: Add thread_map event sythesize function
      perf tools: Add thread_map__new_event function
      perf tools: Add cpu_map event
      perf tools: Add cpu_map event synthesize function
      perf tools: Add cpu_map__new_event function
      perf tools: Add stat config event
      perf tools: Add stat config event synthesize function
      perf tools: Add stat config event read function
      perf tools: Add stat event
      perf tools: Add stat event synthesize function
      perf tools: Add stat event read function
      perf tools: Add stat round event
      perf tools: Add stat round event synthesize function
      perf tools: Introduce stat feature
      perf tools: Move id_offset out of struct perf_evsel union
      perf stat record: Add record command
      perf stat record: Initialize record features
      perf stat record: Synthesize stat record data
      perf stat record: Store events IDs in perf data file
      perf stat record: Add pipe support for record command
      perf stat record: Write stat events on record
      perf stat record: Write stat round events on record
      perf stat report: Add report command
      perf stat report: Process cpu/threads maps
      perf stat report: Process stat config event
      perf stat report: Process stat and stat round events
      perf stat report: Move csv_sep initialization before report command
      perf script: Check output fields only for samples
      perf script: Process cpu/threads maps
      perf script: Process stat config event
      perf script: Add process_stat/process_stat_interval scripting interface
      perf script: Add stat default handlers
      perf script: Display stat events by default
      perf script: Add python support for stat events
      perf script: Add stat-cpi.py script

 tools/perf/Documentation/perf-stat.txt             |  24 +
 tools/perf/builtin-record.c                        |   2 +
 tools/perf/builtin-script.c                        | 143 ++++-
 tools/perf/builtin-stat.c                          | 579 +++++++++++++++------
 tools/perf/scripts/python/stat-cpi.py              |  74 +++
 tools/perf/tests/Build                             |   2 +
 tools/perf/tests/builtin-test.c                    |  20 +
 tools/perf/tests/cpumap.c                          |  39 ++
 tools/perf/tests/stat.c                            | 108 ++++
 tools/perf/tests/tests.h                           |   6 +
 tools/perf/tests/thread-map.c                      |  47 ++
 tools/perf/util/cpumap.c                           |  27 +
 tools/perf/util/cpumap.h                           |   3 +
 tools/perf/util/event.c                            | 170 ++++++
 tools/perf/util/event.h                            |  94 +++-
 tools/perf/util/evlist.c                           |  34 +-
 tools/perf/util/evlist.h                           |  14 +-
 tools/perf/util/evsel.h                            |   2 +-
 tools/perf/util/header.c                           |  49 +-
 tools/perf/util/header.h                           |   2 +
 .../util/scripting-engines/trace-event-python.c    | 113 +++-
 tools/perf/util/session.c                          | 123 +++++
 tools/perf/util/stat.c                             | 162 ++++++
 tools/perf/util/stat.h                             |  16 +
 tools/perf/util/thread_map.c                       |  27 +
 tools/perf/util/thread_map.h                       |   3 +
 tools/perf/util/tool.h                             |   7 +-
 tools/perf/util/trace-event.h                      |   4 +
 28 files changed, 1686 insertions(+), 208 deletions(-)
 create mode 100644 tools/perf/scripts/python/stat-cpi.py
 create mode 100644 tools/perf/tests/cpumap.c
 create mode 100644 tools/perf/tests/stat.c

^ permalink raw reply	[flat|nested] 72+ messages in thread

end of thread, other threads:[~2015-08-07 13:10 UTC | newest]

Thread overview: 72+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2015-07-21 12:31 [RFC 00/47] perf stat: Add scripting support Jiri Olsa
2015-07-21 12:31 ` [PATCH 01/47] perf test: Check for refcnt in thread_map test Jiri Olsa
2015-07-21 17:26   ` Arnaldo Carvalho de Melo
2015-07-29  8:10   ` [tip:perf/core] " tip-bot for Jiri Olsa
2015-07-21 12:31 ` [PATCH 02/47] perf stat: Introduce struct perf_stat_config Jiri Olsa
2015-07-21 17:24   ` Arnaldo Carvalho de Melo
2015-07-21 17:37     ` Arnaldo Carvalho de Melo
2015-07-22  8:38       ` Jiri Olsa
2015-08-07  7:17   ` [tip:perf/core] " tip-bot for Jiri Olsa
2015-07-21 12:31 ` [PATCH 03/47] perf stat: Move scale into " Jiri Olsa
2015-08-07  7:17   ` [tip:perf/core] perf stat: Move 'scale' " tip-bot for Jiri Olsa
2015-07-21 12:31 ` [PATCH 04/47] perf stat: Move output " Jiri Olsa
2015-08-07  7:17   ` [tip:perf/core] perf stat: Move 'output' " tip-bot for Jiri Olsa
2015-07-21 12:31 ` [PATCH 05/47] perf stat: Move interval " Jiri Olsa
2015-08-07  7:18   ` [tip:perf/core] perf stat: Move 'interval' " tip-bot for Jiri Olsa
2015-07-21 12:31 ` [PATCH 06/47] perf stat: Pass struct perf_stat_config into process_counter Jiri Olsa
2015-08-07  7:18   ` [tip:perf/core] perf stat: Pass 'struct perf_stat_config' into process_counter() tip-bot for Jiri Olsa
2015-07-21 12:31 ` [PATCH 07/47] perf stat: Move counter processing code into stat object Jiri Olsa
2015-08-07  7:18   ` [tip:perf/core] " tip-bot for Jiri Olsa
2015-07-21 12:31 ` [PATCH 08/47] perf tools: Use bool instead of target argument in perf_evlist__propagate_maps Jiri Olsa
2015-07-29  8:10   ` [tip:perf/core] perf evlist: Use bool instead of target argument in propagate_maps() tip-bot for Jiri Olsa
2015-07-21 12:31 ` [PATCH 09/47] perf tools: Tolerate NULL maps in perf_evlist__propagate_maps Jiri Olsa
2015-07-21 17:31   ` Arnaldo Carvalho de Melo
2015-07-29  8:11   ` [tip:perf/core] perf evlist: Tolerate NULL maps in propagate_maps tip-bot for Jiri Olsa
2015-07-21 12:31 ` [PATCH 10/47] perf tools: Force perf_evlist__set_maps to propagate maps through events Jiri Olsa
2015-07-21 17:25   ` Arnaldo Carvalho de Melo
2015-07-29  8:10   ` [tip:perf/core] perf evlist: " tip-bot for Jiri Olsa
2015-07-21 12:31 ` [PATCH 11/47] perf tools: Use argv style storage for cmdline feature data Jiri Olsa
2015-07-21 17:29   ` Arnaldo Carvalho de Melo
2015-07-29  8:11   ` [tip:perf/core] perf header: " tip-bot for Jiri Olsa
2015-07-21 12:31 ` [PATCH 12/47] perf tools: Add thread_map event Jiri Olsa
2015-07-21 12:31 ` [PATCH 13/47] perf tools: Add thread_map event sythesize function Jiri Olsa
2015-07-21 12:31 ` [PATCH 14/47] perf tools: Add thread_map__new_event function Jiri Olsa
2015-07-21 12:31 ` [PATCH 15/47] perf tools: Add cpu_map event Jiri Olsa
2015-07-21 12:31 ` [PATCH 16/47] perf tools: Add cpu_map event synthesize function Jiri Olsa
2015-07-21 12:31 ` [PATCH 17/47] perf tools: Add cpu_map__new_event function Jiri Olsa
2015-07-21 12:31 ` [PATCH 18/47] perf tools: Add stat config event Jiri Olsa
2015-07-21 12:31 ` [PATCH 19/47] perf tools: Add stat config event synthesize function Jiri Olsa
2015-07-21 12:31 ` [PATCH 20/47] perf tools: Add stat config event read function Jiri Olsa
2015-07-21 12:31 ` [PATCH 21/47] perf tools: Add stat event Jiri Olsa
2015-07-21 12:31 ` [PATCH 22/47] perf tools: Add stat event synthesize function Jiri Olsa
2015-07-21 12:31 ` [PATCH 23/47] perf tools: Add stat event read function Jiri Olsa
2015-07-21 12:31 ` [PATCH 24/47] perf tools: Add stat round event Jiri Olsa
2015-07-21 12:31 ` [PATCH 25/47] perf tools: Add stat round event synthesize function Jiri Olsa
2015-07-21 12:31 ` [PATCH 26/47] perf tools: Introduce stat feature Jiri Olsa
2015-07-21 12:31 ` [PATCH 27/47] perf tools: Move id_offset out of struct perf_evsel union Jiri Olsa
2015-07-21 12:31 ` [PATCH 28/47] perf stat record: Add record command Jiri Olsa
2015-07-21 12:31 ` [PATCH 29/47] perf stat record: Initialize record features Jiri Olsa
2015-07-21 12:31 ` [PATCH 30/47] perf stat record: Synthesize stat record data Jiri Olsa
2015-07-21 12:31 ` [PATCH 31/47] perf stat record: Store events IDs in perf data file Jiri Olsa
2015-07-21 12:31 ` [PATCH 32/47] perf stat record: Add pipe support for record command Jiri Olsa
2015-07-21 12:31 ` [PATCH 33/47] perf stat record: Write stat events on record Jiri Olsa
2015-07-21 12:31 ` [PATCH 34/47] perf stat record: Write stat round " Jiri Olsa
2015-07-21 12:31 ` [PATCH 35/47] perf stat report: Add report command Jiri Olsa
2015-07-21 12:31 ` [PATCH 36/47] perf stat report: Process cpu/threads maps Jiri Olsa
2015-07-21 12:31 ` [PATCH 37/47] perf stat report: Process stat config event Jiri Olsa
2015-07-21 12:31 ` [PATCH 38/47] perf stat report: Process stat and stat round events Jiri Olsa
2015-07-21 12:31 ` [PATCH 39/47] perf stat report: Move csv_sep initialization before report command Jiri Olsa
2015-07-21 12:32 ` [PATCH 40/47] perf script: Check output fields only for samples Jiri Olsa
2015-07-21 12:32 ` [PATCH 41/47] perf script: Process cpu/threads maps Jiri Olsa
2015-07-21 12:32 ` [PATCH 42/47] perf script: Process stat config event Jiri Olsa
2015-07-21 12:32 ` [PATCH 43/47] perf script: Add process_stat/process_stat_interval scripting interface Jiri Olsa
2015-07-21 12:32 ` [PATCH 44/47] perf script: Add stat default handlers Jiri Olsa
2015-07-21 12:32 ` [PATCH 45/47] perf script: Display stat events by default Jiri Olsa
2015-07-21 12:32 ` [PATCH 46/47] perf script: Add python support for stat events Jiri Olsa
2015-07-21 12:32 ` [PATCH 47/47] perf script: Add stat-cpi.py script Jiri Olsa
2015-07-21 14:43 ` [RFC 00/47] perf stat: Add scripting support Andi Kleen
2015-07-26 13:51   ` Jiri Olsa
2015-07-27 12:46     ` Liang, Kan
2015-08-06 19:27 ` Arnaldo Carvalho de Melo
2015-08-07  7:45   ` Jiri Olsa
2015-08-07 13:10     ` Arnaldo Carvalho de Melo

This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.