From: Jiri Olsa <jolsa@kernel.org>
To: Arnaldo Carvalho de Melo <acme@kernel.org>
Cc: Andi Kleen <andi@firstfloor.org>,
Ulrich Drepper <drepper@gmail.com>,
Will Deacon <will.deacon@arm.com>,
Stephane Eranian <eranian@google.com>,
Don Zickus <dzickus@redhat.com>,
lkml <linux-kernel@vger.kernel.org>,
David Ahern <dsahern@gmail.com>, Ingo Molnar <mingo@kernel.org>,
Namhyung Kim <namhyung@kernel.org>,
Peter Zijlstra <a.p.zijlstra@chello.nl>,
"Liang, Kan" <kan.liang@intel.com>
Subject: [PATCHv5 00/52] perf stat: Add scripting support
Date: Sun, 25 Oct 2015 15:51:16 +0100 [thread overview]
Message-ID: <1445784728-21732-1-git-send-email-jolsa@kernel.org> (raw)
hi,
sending another version of stat scripting.
v5 changes:
- several patches from v4 already taken
- using u16 for cpu number in cpu_map_event
- renamed PERF_RECORD_HEADER_ATTR_UPDATE to PERF_RECORD_EVENT_UPDATE
- moved low hanging fuits patches to the start of the patchset
- patchset tested by Kan Liang, thanks!
v4 changes:
- added attr update event for event's cpumask
- forbig aggregation on task workloads
- some minor reorders and changelog fixes
v3 changes:
- added attr update event to handle unit,scale,name for event
it fixed the uncore_imc_1/cas_count_read/ record/report
- perf report -D now displays stat related events
- some minor and changelog fixes
v2 changes:
- rebased to latest Arnaldo's perf/core
- patches 1 to 11 already merged in
- added --per-core/--per-socket/-A options for perf stat report
command to allow custom aggregation in stat report, please
check new examples below
- couple changelogs changes
The initial attempt defined its own formula lang and allowed
triggering user's script on the end of the stat command:
http://marc.info/?l=linux-kernel&m=136742146322273&w=2
This patchset abandons the idea of new formula language
and rather adds support to:
- store stat data into perf.data file
- add python support to process stat events
Basically it allows to store stat data into perf.data and
post process it with python scripts in a similar way we
do for sampling data.
The stat data are stored in new stat, stat-round, stat-config user events.
stat - stored for each read syscall of the counter
stat round - stored for each interval or end of the command invocation
stat config - stores all the config information needed to process data
so report tool could restore the same output as record
The python script can now define 'stat__<eventname>_<modifier>' functions
to get stat events data and 'stat__interval' to get stat-round data.
See CPI script example in scripts/python/stat-cpi.py.
Also available in:
git://git.kernel.org/pub/scm/linux/kernel/git/jolsa/perf.git
perf/stat_script
thanks,
jirka
Examples:
- To record data for command stat workload:
$ perf stat record kill
...
Performance counter stats for 'kill':
0.372007 task-clock (msec) # 0.613 CPUs utilized
3 context-switches # 0.008 M/sec
0 cpu-migrations # 0.000 K/sec
62 page-faults # 0.167 M/sec
1,129,973 cycles # 3.038 GHz
<not supported> stalled-cycles-frontend
<not supported> stalled-cycles-backend
813,313 instructions # 0.72 insns per cycle
166,161 branches # 446.661 M/sec
8,747 branch-misses # 5.26% of all branches
0.000607287 seconds time elapsed
- To report perf stat data:
$ perf stat report
Performance counter stats for '/home/jolsa/bin/perf stat record kill':
0.372007 task-clock (msec) # inf CPUs utilized
3 context-switches # 0.008 M/sec
0 cpu-migrations # 0.000 K/sec
62 page-faults # 0.167 M/sec
1,129,973 cycles # 3.038 GHz
<not supported> stalled-cycles-frontend
<not supported> stalled-cycles-backend
813,313 instructions # 0.72 insns per cycle
166,161 branches # 446.661 M/sec
8,747 branch-misses # 5.26% of all branches
0.000000000 seconds time elapsed
- To store system-wide period stat data:
$ perf stat -e cycles:u,instructions:u -a -I 1000 record
# time counts unit events
1.000265471 462,311,482 cycles:u (100.00%)
1.000265471 590,037,440 instructions:u
2.000483453 722,532,336 cycles:u (100.00%)
2.000483453 848,678,197 instructions:u
3.000759876 75,990,880 cycles:u (100.00%)
3.000759876 86,187,813 instructions:u
^C 3.213960893 85,329,533 cycles:u (100.00%)
3.213960893 135,954,296 instructions:u
- To report perf stat data:
$ perf stat report
# time counts unit events
1.000265471 462,311,482 cycles:u (100.00%)
1.000265471 590,037,440 instructions:u
2.000483453 722,532,336 cycles:u (100.00%)
2.000483453 848,678,197 instructions:u
3.000759876 75,990,880 cycles:u (100.00%)
3.000759876 86,187,813 instructions:u
3.213960893 85,329,533 cycles:u (100.00%)
3.213960893 135,954,296 instructions:u
- To run stat-cpi.py script over perf.data:
$ perf script -s scripts/python/stat-cpi.py
1.000265: cpu -1, thread -1 -> cpi 0.783529 (462311482/590037440)
2.000483: cpu -1, thread -1 -> cpi 0.851362 (722532336/848678197)
3.000760: cpu -1, thread -1 -> cpi 0.881689 (75990880/86187813)
3.213961: cpu -1, thread -1 -> cpi 0.627634 (85329533/135954296)
- To pipe data from stat to stat-cpi script:
$ perf stat -e cycles:u,instructions:u -A -C 0 -I 1000 record | perf script -s scripts/python/stat-cpi.py
1.000192: cpu 0, thread -1 -> cpi 0.739535 (23921908/32347236)
2.000376: cpu 0, thread -1 -> cpi 1.663482 (2519340/1514498)
3.000621: cpu 0, thread -1 -> cpi 1.396308 (16162767/11575362)
4.000700: cpu 0, thread -1 -> cpi 1.092246 (20077258/18381624)
5.000867: cpu 0, thread -1 -> cpi 0.473816 (45157586/95306156)
6.001034: cpu 0, thread -1 -> cpi 0.532792 (43701668/82023818)
7.001195: cpu 0, thread -1 -> cpi 1.122059 (29890042/26638561)
- Raw script stat data output:
$ perf stat -e cycles:u,instructions:u -A -C 0 -I 1000 record | perf --no-pager script
CPU THREAD VAL ENA RUN TIME EVENT
0 -1 12302059 1000811347 1000810712 1000198821 cycles:u
0 -1 2565362 1000823218 1000823218 1000198821 instructions:u
0 -1 14453353 1000812704 1000812704 2000382283 cycles:u
0 -1 4600932 1000799342 1000799342 2000382283 instructions:u
0 -1 15245106 1000774425 1000774425 3000538255 cycles:u
0 -1 2624324 1000769310 1000769310 3000538255 instructions:u
- To display different aggregation in report:
$ perf stat -e cycles -a -I 1000 record sleep 3
# time counts unit events
1.000223609 703,427,617 cycles
2.000443651 609,975,307 cycles
3.000569616 668,479,597 cycles
3.000735323 1,155,816 cycles
$ perf stat report
# time counts unit events
1.000223609 703,427,617 cycles
2.000443651 609,975,307 cycles
3.000569616 668,479,597 cycles
3.000735323 1,155,816 cycles
$ perf stat report --per-core
# time core cpus counts unit events
1.000223609 S0-C0 2 327,612,412 cycles
1.000223609 S0-C1 2 375,815,205 cycles
2.000443651 S0-C0 2 287,462,177 cycles
2.000443651 S0-C1 2 322,513,130 cycles
3.000569616 S0-C0 2 271,571,908 cycles
3.000569616 S0-C1 2 396,907,689 cycles
3.000735323 S0-C0 2 694,977 cycles
3.000735323 S0-C1 2 460,839 cycles
$ perf stat report --per-socket
# time socket cpus counts unit events
1.000223609 S0 4 703,427,617 cycles
2.000443651 S0 4 609,975,307 cycles
3.000569616 S0 4 668,479,597 cycles
3.000735323 S0 4 1,155,816 cycles
$ perf stat report -A
# time CPU counts unit events
1.000223609 CPU0 205,431,505 cycles
1.000223609 CPU1 122,180,907 cycles
1.000223609 CPU2 176,649,682 cycles
1.000223609 CPU3 199,165,523 cycles
2.000443651 CPU0 148,447,922 cycles
2.000443651 CPU1 139,014,255 cycles
2.000443651 CPU2 204,436,559 cycles
2.000443651 CPU3 118,076,571 cycles
3.000569616 CPU0 149,788,954 cycles
3.000569616 CPU1 121,782,954 cycles
3.000569616 CPU2 247,277,700 cycles
3.000569616 CPU3 149,629,989 cycles
3.000735323 CPU0 269,675 cycles
3.000735323 CPU1 425,302 cycles
3.000735323 CPU2 364,169 cycles
3.000735323 CPU3 96,670 cycles
Cc: Andi Kleen <andi@firstfloor.org>
Cc: Ulrich Drepper <drepper@gmail.com>
Cc: Will Deacon <will.deacon@arm.com>
Cc: Stephane Eranian <eranian@google.com>
Cc: Don Zickus <dzickus@redhat.com>
Tested-by: Kan Liang <kan.liang@intel.com>
---
Jiri Olsa (52):
perf cpu_map: Add cpu_map__empty_new function
perf stat: Cache aggregated map entries in extra cpumap
perf tools: Add thread_map event
perf tools: Add thread_map event sythesize function
perf tools: Add thread_map__new_event function
perf tools: Add perf_event__fprintf_thread_map function
perf tools: Add cpu_map event
perf tools: Add cpu_map event synthesize function
perf tools: Add cpu_map__new_event function
perf tools: Add perf_event__fprintf_cpu_map function
perf tools: Add stat config event
perf tools: Add stat config event synthesize function
perf tools: Add stat config event read function
perf tools: Add stat event
perf tools: Add stat event synthesize function
perf tools: Add stat event read function
perf tools: Add stat round event
perf tools: Add stat round event synthesize function
perf tools: Add stat events fprintf functions
perf tools: Add event_update event
perf tools: Add event_update event unit type
perf tools: Add event_update event scale type
perf tools: Add event_update event name type
perf tools: Add event_update event cpus type
perf tools: Add perf_event__fprintf_event_update function
perf report: Display newly added events in raw dump
perf tools: Introduce stat feature
perf tools: Move id_offset out of struct perf_evsel union
perf stat record: Add record command
perf stat record: Initialize record features
perf stat record: Synthesize stat record data
perf stat record: Store events IDs in perf data file
perf stat record: Add pipe support for record command
perf stat record: Write stat events on record
perf stat record: Write stat round events on record
perf stat record: Do not allow record with multiple runs mode
perf stat record: Synthesize event update events
perf stat report: Add report command
perf stat report: Process cpu/threads maps
perf stat report: Process stat config event
perf stat report: Add support to initialize aggr_map from file
perf stat report: Process stat and stat round events
perf stat report: Process event update events
perf stat report: Move csv_sep initialization before report command
perf stat report: Allow to override aggr_mode
perf script: Process cpu/threads maps
perf script: Process stat config event
perf script: Add process_stat/process_stat_interval scripting interface
perf script: Add stat default handlers
perf script: Display stat events by default
perf script: Add python support for stat events
perf script: Add stat-cpi.py script
tools/perf/Documentation/perf-stat.txt | 34 ++
tools/perf/builtin-record.c | 2 +
tools/perf/builtin-script.c | 139 +++++
tools/perf/builtin-stat.c | 643 ++++++++++++++++++++-
tools/perf/scripts/python/stat-cpi.py | 74 +++
tools/perf/tests/Build | 3 +
tools/perf/tests/builtin-test.c | 24 +
tools/perf/tests/cpumap.c | 88 +++
tools/perf/tests/event_update.c | 117 ++++
tools/perf/tests/stat.c | 111 ++++
tools/perf/tests/tests.h | 7 +
tools/perf/tests/thread-map.c | 43 ++
tools/perf/util/cpumap.c | 59 ++
tools/perf/util/cpumap.h | 2 +
tools/perf/util/event.c | 308 ++++++++++
tools/perf/util/event.h | 150 ++++-
tools/perf/util/evlist.c | 6 +-
tools/perf/util/evlist.h | 3 +
tools/perf/util/evsel.h | 2 +-
tools/perf/util/header.c | 205 +++++++
tools/perf/util/header.h | 17 +
.../util/scripting-engines/trace-event-python.c | 114 +++-
tools/perf/util/session.c | 189 ++++++
tools/perf/util/stat.c | 62 ++
tools/perf/util/stat.h | 10 +
tools/perf/util/thread_map.c | 27 +
tools/perf/util/thread_map.h | 3 +
tools/perf/util/tool.h | 8 +-
tools/perf/util/trace-event.h | 4 +
29 files changed, 2427 insertions(+), 27 deletions(-)
create mode 100644 tools/perf/scripts/python/stat-cpi.py
create mode 100644 tools/perf/tests/cpumap.c
create mode 100644 tools/perf/tests/event_update.c
create mode 100644 tools/perf/tests/stat.c
next reply other threads:[~2015-10-25 14:52 UTC|newest]
Thread overview: 99+ messages / expand[flat|nested] mbox.gz Atom feed top
2015-10-25 14:51 Jiri Olsa [this message]
2015-10-25 14:51 ` [PATCH 01/52] perf cpu_map: Add cpu_map__empty_new function Jiri Olsa
2015-10-29 9:41 ` [tip:perf/core] " tip-bot for Jiri Olsa
2015-10-25 14:51 ` [PATCH 02/52] perf stat: Cache aggregated map entries in extra cpumap Jiri Olsa
2015-10-29 9:41 ` [tip:perf/core] " tip-bot for Jiri Olsa
2015-10-25 14:51 ` [PATCH 03/52] perf tools: Add thread_map event Jiri Olsa
2015-10-26 17:42 ` Arnaldo Carvalho de Melo
2015-10-26 17:45 ` Jiri Olsa
2015-10-26 17:48 ` Arnaldo Carvalho de Melo
2015-10-26 17:55 ` Jiri Olsa
2015-10-26 17:57 ` Arnaldo Carvalho de Melo
2015-12-18 9:00 ` [tip:perf/core] perf thread_map: Add thread_map user level event tip-bot for Jiri Olsa
2015-10-25 14:51 ` [PATCH 04/52] perf tools: Add thread_map event sythesize function Jiri Olsa
2015-12-18 9:00 ` [tip:perf/core] perf thread_map: " tip-bot for Jiri Olsa
2015-10-25 14:51 ` [PATCH 05/52] perf tools: Add thread_map__new_event function Jiri Olsa
2015-12-18 9:01 ` [tip:perf/core] perf thread_map: " tip-bot for Jiri Olsa
2015-10-25 14:51 ` [PATCH 06/52] perf tools: Add perf_event__fprintf_thread_map function Jiri Olsa
2015-12-18 9:01 ` [tip:perf/core] perf thread_map: " tip-bot for Jiri Olsa
2015-10-25 14:51 ` [PATCH 07/52] perf tools: Add cpu_map event Jiri Olsa
2015-12-18 9:01 ` [tip:perf/core] perf cpu_map: Add cpu_map user level event tip-bot for Jiri Olsa
2015-10-25 14:51 ` [PATCH 08/52] perf tools: Add cpu_map event synthesize function Jiri Olsa
2015-12-18 9:02 ` [tip:perf/core] perf cpu_map: " tip-bot for Jiri Olsa
2015-10-25 14:51 ` [PATCH 09/52] perf tools: Add cpu_map__new_event function Jiri Olsa
2015-12-18 9:02 ` [tip:perf/core] perf cpu_map: " tip-bot for Jiri Olsa
2015-10-25 14:51 ` [PATCH 10/52] perf tools: Add perf_event__fprintf_cpu_map function Jiri Olsa
2015-12-18 9:02 ` [tip:perf/core] perf cpu_map: " tip-bot for Jiri Olsa
2015-10-25 14:51 ` [PATCH 11/52] perf tools: Add stat config event Jiri Olsa
2015-10-27 13:16 ` Arnaldo Carvalho de Melo
2015-10-27 13:30 ` Jiri Olsa
2015-10-27 13:44 ` Arnaldo Carvalho de Melo
2015-10-27 13:51 ` Jiri Olsa
2015-10-27 14:13 ` Arnaldo Carvalho de Melo
2015-12-18 9:03 ` [tip:perf/core] perf tools: Add stat config user level event tip-bot for Jiri Olsa
2015-10-25 14:51 ` [PATCH 12/52] perf tools: Add stat config event synthesize function Jiri Olsa
2015-12-18 9:03 ` [tip:perf/core] " tip-bot for Jiri Olsa
2015-10-25 14:51 ` [PATCH 13/52] perf tools: Add stat config event read function Jiri Olsa
2015-12-18 9:03 ` [tip:perf/core] " tip-bot for Jiri Olsa
2015-10-25 14:51 ` [PATCH 14/52] perf tools: Add stat event Jiri Olsa
2015-12-18 9:04 ` [tip:perf/core] perf tools: Add stat user level event tip-bot for Jiri Olsa
2015-10-25 14:51 ` [PATCH 15/52] perf tools: Add stat event synthesize function Jiri Olsa
2015-12-18 9:04 ` [tip:perf/core] " tip-bot for Jiri Olsa
2015-10-25 14:51 ` [PATCH 16/52] perf tools: Add stat event read function Jiri Olsa
2015-12-18 9:04 ` [tip:perf/core] " tip-bot for Jiri Olsa
2015-10-25 14:51 ` [PATCH 17/52] perf tools: Add stat round event Jiri Olsa
2015-12-18 9:05 ` [tip:perf/core] perf tools: Add stat round user level event tip-bot for Jiri Olsa
2015-10-25 14:51 ` [PATCH 18/52] perf tools: Add stat round event synthesize function Jiri Olsa
2015-12-18 9:05 ` [tip:perf/core] " tip-bot for Jiri Olsa
2015-10-25 14:51 ` [PATCH 19/52] perf tools: Add stat events fprintf functions Jiri Olsa
2015-12-18 9:05 ` [tip:perf/core] " tip-bot for Jiri Olsa
2015-10-25 14:51 ` [PATCH 20/52] perf tools: Add event_update event Jiri Olsa
2015-12-18 9:06 ` [tip:perf/core] perf tools: Add event_update user level event tip-bot for Jiri Olsa
2015-10-25 14:51 ` [PATCH 21/52] perf tools: Add event_update event unit type Jiri Olsa
2015-12-18 9:06 ` [tip:perf/core] " tip-bot for Jiri Olsa
2015-10-25 14:51 ` [PATCH 22/52] perf tools: Add event_update event scale type Jiri Olsa
2015-12-18 9:06 ` [tip:perf/core] " tip-bot for Jiri Olsa
2015-10-25 14:51 ` [PATCH 23/52] perf tools: Add event_update event name type Jiri Olsa
2015-12-18 9:07 ` [tip:perf/core] " tip-bot for Jiri Olsa
2015-10-25 14:51 ` [PATCH 24/52] perf tools: Add event_update event cpus type Jiri Olsa
2015-12-18 9:07 ` [tip:perf/core] " tip-bot for Jiri Olsa
2015-10-25 14:51 ` [PATCH 25/52] perf tools: Add perf_event__fprintf_event_update function Jiri Olsa
2015-12-18 9:07 ` [tip:perf/core] " tip-bot for Jiri Olsa
2015-10-25 14:51 ` [PATCH 26/52] perf report: Display newly added events in raw dump Jiri Olsa
2015-12-18 9:07 ` [tip:perf/core] " tip-bot for Jiri Olsa
2015-10-25 14:51 ` [PATCH 27/52] perf tools: Introduce stat feature Jiri Olsa
2015-12-18 9:08 ` [tip:perf/core] perf tools: Introduce stat perf.data header feature tip-bot for Jiri Olsa
2015-10-25 14:51 ` [PATCH 28/52] perf tools: Move id_offset out of struct perf_evsel union Jiri Olsa
2015-10-29 9:40 ` [tip:perf/core] perf evsel: Move id_offset out of struct perf_evsel union member tip-bot for Jiri Olsa
2015-10-25 14:51 ` [PATCH 29/52] perf stat record: Add record command Jiri Olsa
2015-10-27 14:26 ` Arnaldo Carvalho de Melo
2015-10-25 14:51 ` [PATCH 30/52] perf stat record: Initialize record features Jiri Olsa
2015-10-25 14:51 ` [PATCH 31/52] perf stat record: Synthesize stat record data Jiri Olsa
2015-10-27 14:42 ` Arnaldo Carvalho de Melo
2015-10-28 12:15 ` Jiri Olsa
2015-10-28 13:03 ` Jiri Olsa
2015-10-28 13:04 ` Jiri Olsa
2015-10-25 14:51 ` [PATCH 32/52] perf stat record: Store events IDs in perf data file Jiri Olsa
2015-10-25 14:51 ` [PATCH 33/52] perf stat record: Add pipe support for record command Jiri Olsa
2015-10-25 14:51 ` [PATCH 34/52] perf stat record: Write stat events on record Jiri Olsa
2015-10-25 14:51 ` [PATCH 35/52] perf stat record: Write stat round " Jiri Olsa
2015-10-25 14:51 ` [PATCH 36/52] perf stat record: Do not allow record with multiple runs mode Jiri Olsa
2015-10-25 14:51 ` [PATCH 37/52] perf stat record: Synthesize event update events Jiri Olsa
2015-10-25 14:51 ` [PATCH 38/52] perf stat report: Add report command Jiri Olsa
2015-10-25 14:51 ` [PATCH 39/52] perf stat report: Process cpu/threads maps Jiri Olsa
2015-10-25 14:51 ` [PATCH 40/52] perf stat report: Process stat config event Jiri Olsa
2015-10-25 14:51 ` [PATCH 41/52] perf stat report: Add support to initialize aggr_map from file Jiri Olsa
2015-10-25 14:51 ` [PATCH 42/52] perf stat report: Process stat and stat round events Jiri Olsa
2015-10-25 14:51 ` [PATCH 43/52] perf stat report: Process event update events Jiri Olsa
2015-10-25 14:52 ` [PATCH 44/52] perf stat report: Move csv_sep initialization before report command Jiri Olsa
2015-10-25 14:52 ` [PATCH 45/52] perf stat report: Allow to override aggr_mode Jiri Olsa
2015-10-25 14:52 ` [PATCH 46/52] perf script: Process cpu/threads maps Jiri Olsa
2015-10-25 14:52 ` [PATCH 47/52] perf script: Process stat config event Jiri Olsa
2015-10-25 14:52 ` [PATCH 48/52] perf script: Add process_stat/process_stat_interval scripting interface Jiri Olsa
2015-10-25 14:52 ` [PATCH 49/52] perf script: Add stat default handlers Jiri Olsa
2015-10-25 14:52 ` [PATCH 50/52] perf script: Display stat events by default Jiri Olsa
2015-10-25 14:52 ` [PATCH 51/52] perf script: Add python support for stat events Jiri Olsa
2015-10-25 14:52 ` [PATCH 52/52] perf script: Add stat-cpi.py script Jiri Olsa
2015-10-26 17:34 ` [PATCHv5 00/52] perf stat: Add scripting support Arnaldo Carvalho de Melo
2015-10-27 14:24 ` Liang, Kan
2015-10-27 14:48 ` Arnaldo Carvalho de Melo
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=1445784728-21732-1-git-send-email-jolsa@kernel.org \
--to=jolsa@kernel.org \
--cc=a.p.zijlstra@chello.nl \
--cc=acme@kernel.org \
--cc=andi@firstfloor.org \
--cc=drepper@gmail.com \
--cc=dsahern@gmail.com \
--cc=dzickus@redhat.com \
--cc=eranian@google.com \
--cc=kan.liang@intel.com \
--cc=linux-kernel@vger.kernel.org \
--cc=mingo@kernel.org \
--cc=namhyung@kernel.org \
--cc=will.deacon@arm.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).