Add fine grained sampled metrics for perf script

Add fine grained sampled metrics for perf script

[Andi Kleen]

This patch kit adds perf script support for computing metrics for
sampled groups. This allows much more fine grained metrics
measurement than perf stat allows.

The patchkit contains one extra bug fix (add missing documentation)

This allows things like fine grained IPC tracking.

Note that the metric is still averaged over the sampling period,
it is not just for the sampling point.

For example to sample IPC:

$ perf record -e '{ref-cycles,cycles,instructions}:S' -a sleep 1
$ perf script -F metric,ip,sym,time,cpu,comm
...
 alsa-sink-ALC32 [000] 42815.856074:      7fd65937d6cc [unknown]
 alsa-sink-ALC32 [000] 42815.856074:      7fd65937d6cc [unknown]
 alsa-sink-ALC32 [000] 42815.856074:      7fd65937d6cc [unknown]
 alsa-sink-ALC32 [000] 42815.856074:    metric:    0.13  insn per cycle
         swapper [000] 42815.857961:  ffffffff81655df0 __schedule
         swapper [000] 42815.857961:  ffffffff81655df0 __schedule
 :1
        swapper [000] 42815.857961:  ffffffff81655df0 __schedule
         swapper [000] 42815.857961:    metric:    0.23  insn per cycle
 qemu-system-x86 [000] 42815.858130:  ffffffff8165ad0e
_raw_spin_unlock_irqrestore
 qemu-system-x86 [000] 42815.858130:  ffffffff8165ad0e
_raw_spin_unlock_irqrestore
 qemu-system-x86 [000] 42815.858130:  ffffffff8165ad0e
_raw_spin_unlock_irqrestore
 qemu-system-x86 [000] 42815.858130:    metric:    0.46  insn per cycle
           :4972 [000] 42815.858312:  ffffffffa080e5f2 vmx_vcpu_run
           :4972 [000] 42815.858312:  ffffffffa080e5f2 vmx_vcpu_run
           :4972 [000] 42815.858312:  ffffffffa080e5f2 vmx_vcpu_run
           :4972 [000] 42815.858312:    metric:    0.45  insn per cycle

TopDown:

Note this requires disabling SMT if you have it enabled (e.g. by offlining
the extra CPUs), because SMT would require sampling per core, which is not supported.

$ perf record -e '{ref-cycles,topdown-fetch-bubbles,topdown-recovery-bubbles,\
topdown-slots-retired,topdown-total-slots,topdown-slots-issued}:S' -a sleep 1
$ perf script --header -I -F cpu,ip,sym,event,metric,period
...
[000]     121108               ref-cycles:  ffffffff8165222e copy_user_enhanced_fast_string
[000]     190350    topdown-fetch-bubbles:  ffffffff8165222e copy_user_enhanced_fast_string
[000]       2055 topdown-recovery-bubbles:  ffffffff8165222e copy_user_enhanced_fast_string
[000]     148729    topdown-slots-retired:  ffffffff8165222e copy_user_enhanced_fast_string
[000]     144324      topdown-total-slots:  ffffffff8165222e copy_user_enhanced_fast_string
[000]     160852     topdown-slots-issued:  ffffffff8165222e copy_user_enhanced_fast_string
[000]   metric:     33.0% frontend bound
[000]   metric:      3.5% bad speculation
[000]   metric:     25.8% retiring
[000]   metric:     37.7% backend bound
[000]     112112               ref-cycles:  ffffffff8165aec8 _raw_spin_lock_irqsave
[000]     357222    topdown-fetch-bubbles:  ffffffff8165aec8 _raw_spin_lock_irqsave
[000]       3325 topdown-recovery-bubbles:  ffffffff8165aec8 _raw_spin_lock_irqsave
[000]     323553    topdown-slots-retired:  ffffffff8165aec8 _raw_spin_lock_irqsave
[000]     270507      topdown-total-slots:  ffffffff8165aec8 _raw_spin_lock_irqsave
[000]     341226     topdown-slots-issued:  ffffffff8165aec8 _raw_spin_lock_irqsave
[000]   metric:     33.0% frontend bound
[000]   metric:      2.9% bad speculation
[000]   metric:     29.9% retiring
[000]   metric:     34.2% backend bound


Git tree:
git://git.kernel.org/pub/scm/limux/kernel/git/ak/linux-misc.git perf/script-metric-1


v1: Initial post

[PATCH v1 1/5] perf, tools: Document some missing perf.data headers

[Andi Kleen]

From: Andi Kleen <ak@linux.intel.com>

Document STAT and CACHE header entries.

Signed-off-by: Andi Kleen <ak@linux.intel.com>
---
 tools/perf/Documentation/perf.data-file-format.txt | 23 ++++++++++++++++++++++
 1 file changed, 23 insertions(+)

diff --git a/tools/perf/Documentation/perf.data-file-format.txt b/tools/perf/Documentation/perf.data-file-format.txt
index e90c59c6d815..15e8b48077ba 100644
--- a/tools/perf/Documentation/perf.data-file-format.txt
+++ b/tools/perf/Documentation/perf.data-file-format.txt
@@ -238,6 +238,29 @@ struct auxtrace_index {
 	struct auxtrace_index_entry entries[PERF_AUXTRACE_INDEX_ENTRY_COUNT];
 };
 
+	HEADER_STAT = 19,
+
+This is merely a flag signifying that the data section contains data
+recorded from perf stat record.
+
+	HEADER_CACHE = 20,
+
+Description of the cache hierarchy. Based on the Linux sysfs format
+in /sys/devices/system/cpu/cpu*/cache/
+
+	u32 version	Currently always 1
+	u32 number_of_cache_levels
+
+struct {
+	u32	level;
+	u32	line_size;
+	u32	sets;
+	u32	ways;
+	struct perf_header_string type;
+	struct perf_header_string size;
+	struct perf_header_string map;
+}[number_of_cache_levels];
+
 	other bits are reserved and should ignored for now
 	HEADER_FEAT_BITS	= 256,
 
-- 
2.13.6

[PATCH v1 2/5] perf, tools: Save event scaling factors in perf.data

[Andi Kleen]

From: Andi Kleen <ak@linux.intel.com>

To process metrics, perf script needs to know the scaling
factors reported by sysfs for events. Save the scaling factors
in the perf.data metadata in a new SCALE header.

Signed-off-by: Andi Kleen <ak@linux.intel.com>
---
 tools/perf/Documentation/perf.data-file-format.txt |  8 +++
 tools/perf/util/header.c                           | 60 ++++++++++++++++++++++
 tools/perf/util/header.h                           |  1 +
 3 files changed, 69 insertions(+)

diff --git a/tools/perf/Documentation/perf.data-file-format.txt b/tools/perf/Documentation/perf.data-file-format.txt
index 15e8b48077ba..dc91d0578f46 100644
--- a/tools/perf/Documentation/perf.data-file-format.txt
+++ b/tools/perf/Documentation/perf.data-file-format.txt
@@ -261,6 +261,14 @@ struct {
 	struct perf_header_string map;
 }[number_of_cache_levels];
 
+	HEADER_SCALE = 21,
+
+Save scaling factor of events. One entry for each event in the same
+order as other events in the header.
+
+	u32 	nr;
+	double	scale[nr];
+
 	other bits are reserved and should ignored for now
 	HEADER_FEAT_BITS	= 256,
 
diff --git a/tools/perf/util/header.c b/tools/perf/util/header.c
index 605bbd5404fb..4f1d23cfb9b0 100644
--- a/tools/perf/util/header.c
+++ b/tools/perf/util/header.c
@@ -2147,6 +2147,65 @@ static int process_cache(struct feat_fd *ff, void *data __maybe_unused)
 	return -1;
 }
 
+static int write_scale(struct feat_fd *ff, struct perf_evlist *evlist)
+{
+	u32 num;
+	struct perf_evsel *evsel;
+
+	num = 0;
+	evlist__for_each_entry (evlist, evsel)
+		num++;
+
+	if (do_write(ff, &num, sizeof num) < 0)
+		return -1;
+	evlist__for_each_entry (evlist, evsel) {
+		if (do_write(ff, &evsel->scale, sizeof(double)) < 0)
+			return -1;
+	}
+	return 0;
+}
+
+static void print_scale(struct feat_fd *ff, FILE *fp)
+{
+	struct perf_session *session;
+	struct perf_evsel *evsel;
+	int num = 0;
+
+	session = container_of(ff->ph, struct perf_session, header);
+
+	num = 0;
+	evlist__for_each_entry(session->evlist, evsel) {
+		fprintf(fp, "# event %d %s scale %f\n",
+			num++,
+			evsel->name,
+			evsel->scale);
+	}
+}
+
+static int process_scale(struct feat_fd *ff, void *data __maybe_unused)
+{
+	u32 cnt;
+	struct perf_evsel *evsel;
+	struct perf_session *session;
+
+	session = container_of(ff->ph, struct perf_session, header);
+	if (do_read_u32(ff, &cnt))
+		return -1;
+	evlist__for_each_entry(session->evlist, evsel) {
+		if (__do_read(ff, &evsel->scale, sizeof(double)))
+			return -1;
+		if (ff->ph->needs_swap)
+			mem_bswap_64(&evsel->scale, sizeof(double));
+		if (--cnt <= 0)
+			break;
+	}
+	if (cnt) {
+		pr_debug("missing scale descriptors\n");
+		return 1;
+	}
+	return 0;
+}
+
 struct feature_ops {
 	int (*write)(struct feat_fd *ff, struct perf_evlist *evlist);
 	void (*print)(struct feat_fd *ff, FILE *fp);
@@ -2204,6 +2263,7 @@ static const struct feature_ops feat_ops[HEADER_LAST_FEATURE] = {
 	FEAT_OPN(AUXTRACE,	auxtrace,	false),
 	FEAT_OPN(STAT,		stat,		false),
 	FEAT_OPN(CACHE,		cache,		true),
+	FEAT_OPN(SCALE,		scale,		true),
 };
 
 struct header_print_data {
diff --git a/tools/perf/util/header.h b/tools/perf/util/header.h
index f7a16ee527b8..d13d9520191f 100644
--- a/tools/perf/util/header.h
+++ b/tools/perf/util/header.h
@@ -33,6 +33,7 @@ enum {
 	HEADER_AUXTRACE,
 	HEADER_STAT,
 	HEADER_CACHE,
+	HEADER_SCALE,
 	HEADER_LAST_FEATURE,
 	HEADER_FEAT_BITS	= 256,
 };
-- 
2.13.6

[PATCH v1 3/5] perf, tools, script: Allow printing period for non freq mode groups

[Andi Kleen]

From: Andi Kleen <ak@linux.intel.com>

When using leader sampling the values of the not sampled but counted
events are shown by perf script in "period".

Currently printing period is only allowed when the main event
has a period, that is it is in frequency mode.

This implies that we cannot dump the values of counted events
when the leader event is not in frequency mode.

Just remove the check that the period must be set on all
events. It will just be printed as 0 instead if it's not
available.

This fixes the following:

$ perf record -c 100000 -e '{cycles,branches}:S'
$ perf script -F event,period

Signed-off-by: Andi Kleen <ak@linux.intel.com>
---
 tools/perf/builtin-script.c | 5 -----
 1 file changed, 5 deletions(-)

diff --git a/tools/perf/builtin-script.c b/tools/perf/builtin-script.c
index 9092de0f7238..34d8b766e518 100644
--- a/tools/perf/builtin-script.c
+++ b/tools/perf/builtin-script.c
@@ -377,11 +377,6 @@ static int perf_evsel__check_attr(struct perf_evsel *evsel,
 					   PERF_OUTPUT_CPU, allow_user_set))
 		return -EINVAL;
 
-	if (PRINT_FIELD(PERIOD) &&
-		perf_evsel__check_stype(evsel, PERF_SAMPLE_PERIOD, "PERIOD",
-					PERF_OUTPUT_PERIOD))
-		return -EINVAL;
-
 	if (PRINT_FIELD(IREGS) &&
 		perf_evsel__check_stype(evsel, PERF_SAMPLE_REGS_INTR, "IREGS",
 					PERF_OUTPUT_IREGS))
-- 
2.13.6

[PATCH v1 4/5] perf, tools: Add fallback in perf_evsel__nr_cpus for no map

[Andi Kleen]

From: Andi Kleen <ak@linux.intel.com>

Support the case of the event having no cpumap in perf_evsel__nr_cpus.
Just return 1 in this case.  This can happen in perf script
when it uses the perf stat shadow functions.

Signed-off-by: Andi Kleen <ak@linux.intel.com>
---
 tools/perf/util/evsel.h | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/tools/perf/util/evsel.h b/tools/perf/util/evsel.h
index db658785d828..25f21cde8826 100644
--- a/tools/perf/util/evsel.h
+++ b/tools/perf/util/evsel.h
@@ -157,7 +157,8 @@ static inline struct cpu_map *perf_evsel__cpus(struct perf_evsel *evsel)
 
 static inline int perf_evsel__nr_cpus(struct perf_evsel *evsel)
 {
-	return perf_evsel__cpus(evsel)->nr;
+	struct cpu_map *map = perf_evsel__cpus(evsel);
+	return map ? map->nr : 1;
 }
 
 void perf_counts_values__scale(struct perf_counts_values *count,
-- 
2.13.6

[PATCH v1 5/5] perf, tools, script: Allow computing metrics in perf script

[Andi Kleen]

From: Andi Kleen <ak@linux.intel.com>

Add support for computing perf stat style metrics in perf script.

When using leader sampling we can get metrics for each sampling
period by computing formulas over the values of the different
group members.

This allows things like fine grained IPC tracking through sampling,
much more fine grained than with perf stat.

Note that the metric is still averaged over the sampling period,
it is not just for the sampling point.

This patch adds a new metric output field for perf script
that uses the existing perf stat metrics infrastructure to compute
any metrics supported by perf stat.

For example to sample IPC:

$ perf record -e '{ref-cycles,cycles,instructions}:S' -a sleep 1
$ perf script -F metric,ip,sym,time,cpu,comm
...
 alsa-sink-ALC32 [000] 42815.856074:      7fd65937d6cc [unknown]
 alsa-sink-ALC32 [000] 42815.856074:      7fd65937d6cc [unknown]
 alsa-sink-ALC32 [000] 42815.856074:      7fd65937d6cc [unknown]
 alsa-sink-ALC32 [000] 42815.856074:    metric:    0.13  insn per cycle
         swapper [000] 42815.857961:  ffffffff81655df0 __schedule
         swapper [000] 42815.857961:  ffffffff81655df0 __schedule
         swapper [000] 42815.857961:  ffffffff81655df0 __schedule
         swapper [000] 42815.857961:    metric:    0.23  insn per cycle
 qemu-system-x86 [000] 42815.858130:  ffffffff8165ad0e
_raw_spin_unlock_irqrestore
 qemu-system-x86 [000] 42815.858130:  ffffffff8165ad0e
_raw_spin_unlock_irqrestore
 qemu-system-x86 [000] 42815.858130:  ffffffff8165ad0e
_raw_spin_unlock_irqrestore
 qemu-system-x86 [000] 42815.858130:    metric:    0.46  insn per cycle
           :4972 [000] 42815.858312:  ffffffffa080e5f2 vmx_vcpu_run
           :4972 [000] 42815.858312:  ffffffffa080e5f2 vmx_vcpu_run
           :4972 [000] 42815.858312:  ffffffffa080e5f2 vmx_vcpu_run
           :4972 [000] 42815.858312:    metric:    0.45  insn per cycle

TopDown:

Note this requires disabling SMT if you have it enabled, because
SMT would require sampling per core, which is not supported.

$ perf record -e '{ref-cycles,topdown-fetch-bubbles,topdown-recovery-bubbles,\
topdown-slots-retired,topdown-total-slots,topdown-slots-issued}:S' -a sleep 1
$ perf script --header -I -F cpu,ip,sym,event,metric,period
...
[000]     121108               ref-cycles:  ffffffff8165222e copy_user_enhanced_fast_string
[000]     190350    topdown-fetch-bubbles:  ffffffff8165222e copy_user_enhanced_fast_string
[000]       2055 topdown-recovery-bubbles:  ffffffff8165222e copy_user_enhanced_fast_string
[000]     148729    topdown-slots-retired:  ffffffff8165222e copy_user_enhanced_fast_string
[000]     144324      topdown-total-slots:  ffffffff8165222e copy_user_enhanced_fast_string
[000]     160852     topdown-slots-issued:  ffffffff8165222e copy_user_enhanced_fast_string
[000]   metric:     33.0% frontend bound
[000]   metric:      3.5% bad speculation
[000]   metric:     25.8% retiring
[000]   metric:     37.7% backend bound
[000]     112112               ref-cycles:  ffffffff8165aec8 _raw_spin_lock_irqsave
[000]     357222    topdown-fetch-bubbles:  ffffffff8165aec8 _raw_spin_lock_irqsave
[000]       3325 topdown-recovery-bubbles:  ffffffff8165aec8 _raw_spin_lock_irqsave
[000]     323553    topdown-slots-retired:  ffffffff8165aec8 _raw_spin_lock_irqsave
[000]     270507      topdown-total-slots:  ffffffff8165aec8 _raw_spin_lock_irqsave
[000]     341226     topdown-slots-issued:  ffffffff8165aec8 _raw_spin_lock_irqsave
[000]   metric:     33.0% frontend bound
[000]   metric:      2.9% bad speculation
[000]   metric:     29.9% retiring
[000]   metric:     34.2% backend bound
...

Signed-off-by: Andi Kleen <ak@linux.intel.com>
---
 tools/perf/Documentation/perf-script.txt | 10 +++-
 tools/perf/builtin-script.c              | 82 ++++++++++++++++++++++++++++++++
 tools/perf/util/evsel.h                  |  2 +
 tools/perf/util/metricgroup.c            |  4 ++
 4 files changed, 97 insertions(+), 1 deletion(-)

diff --git a/tools/perf/Documentation/perf-script.txt b/tools/perf/Documentation/perf-script.txt
index bcc1ba35a2d8..62eb1acae8f0 100644
--- a/tools/perf/Documentation/perf-script.txt
+++ b/tools/perf/Documentation/perf-script.txt
@@ -117,7 +117,7 @@ OPTIONS
         Comma separated list of fields to print. Options are:
         comm, tid, pid, time, cpu, event, trace, ip, sym, dso, addr, symoff,
         srcline, period, iregs, uregs, brstack, brstacksym, flags, bpf-output, brstackinsn,
-        brstackoff, callindent, insn, insnlen, synth, phys_addr.
+	brstackoff, callindent, insn, insnlen, synth, phys_addr, metric.
         Field list can be prepended with the type, trace, sw or hw,
         to indicate to which event type the field list applies.
         e.g., -F sw:comm,tid,time,ip,sym  and -F trace:time,cpu,trace
@@ -217,6 +217,14 @@ OPTIONS
 
 	The brstackoff field will print an offset into a specific dso/binary.
 
+	With the metric option perf script can compute metrics for
+	sampling periods, similar to perf stat. This requires
+	specifying a group with multiple metrics with the :S option
+	for perf record. perf will sample on the first event, and
+	compute metrics for all the events in the group. Please note
+	that the metric computed is averaged over the whole sampling
+	period, not just for the sample point.
+
 -k::
 --vmlinux=<file>::
         vmlinux pathname
diff --git a/tools/perf/builtin-script.c b/tools/perf/builtin-script.c
index 34d8b766e518..9b989c584ce6 100644
--- a/tools/perf/builtin-script.c
+++ b/tools/perf/builtin-script.c
@@ -21,6 +21,7 @@
 #include "util/cpumap.h"
 #include "util/thread_map.h"
 #include "util/stat.h"
+#include "util/color.h"
 #include "util/string2.h"
 #include "util/thread-stack.h"
 #include "util/time-utils.h"
@@ -89,6 +90,7 @@ enum perf_output_field {
 	PERF_OUTPUT_SYNTH           = 1U << 25,
 	PERF_OUTPUT_PHYS_ADDR       = 1U << 26,
 	PERF_OUTPUT_UREGS	    = 1U << 27,
+	PERF_OUTPUT_METRIC	    = 1U << 28,
 };
 
 struct output_option {
@@ -123,6 +125,7 @@ struct output_option {
 	{.str = "brstackoff", .field = PERF_OUTPUT_BRSTACKOFF},
 	{.str = "synth", .field = PERF_OUTPUT_SYNTH},
 	{.str = "phys_addr", .field = PERF_OUTPUT_PHYS_ADDR},
+	{.str = "metric", .field = PERF_OUTPUT_METRIC},
 };
 
 enum {
@@ -1398,6 +1401,82 @@ static size_t data_src__printf(u64 data_src)
 	return printf("%-*s", maxlen, out);
 }
 
+struct metric_ctx {
+	struct perf_sample	*sample;
+	struct thread		*thread;
+	struct perf_evsel	*evsel;
+};
+
+static void script_print_metric(void *ctx,
+				const char *color,
+				const char *fmt,
+				const char *unit, double val)
+{
+	struct metric_ctx *mctx = ctx;
+
+	if (!fmt)
+		return;
+	print_sample_start(mctx->sample, mctx->thread, mctx->evsel);
+	fputs("\tmetric: ", stdout);
+	if (color)
+		color_fprintf(stdout, color, fmt, val);
+	else
+		printf(fmt, val);
+	printf(" %s\n", unit);
+}
+
+static void script_new_line(void *ctx)
+{
+	struct metric_ctx *mctx = ctx;
+
+	print_sample_start(mctx->sample, mctx->thread, mctx->evsel);
+	fputs("\tmetric: ", stdout);
+}
+
+static void print_metric(struct perf_script *script,
+			 struct thread *thread,
+			 struct perf_evsel *evsel,
+			 struct perf_sample *sample)
+{
+	struct perf_stat_output_ctx ctx = {
+		.print_metric = script_print_metric,
+		.new_line = script_new_line,
+		.ctx = &(struct metric_ctx) {
+				.sample = sample,
+				.thread = thread,
+				.evsel = evsel
+			 },
+		.force_header = false,
+	};
+	struct perf_evsel *ev2;
+	static bool init;
+	u64 val;
+
+	if (!init) {
+		perf_stat__init_shadow_stats();
+		init = true;
+	}
+	if (!evsel->priv)
+		perf_evlist__alloc_stats(script->session->evlist, false);
+	if (evsel->leader->gnum++ == 0)
+		perf_stat__reset_shadow_stats();
+	val = sample->period * evsel->scale;
+	perf_stat__update_shadow_stats(evsel,
+				       &val,
+				       sample->cpu);
+	evsel->val = sample->period;
+	if (evsel->leader->gnum == evsel->leader->nr_members) {
+		for_each_group_member (ev2, evsel->leader) {
+			perf_stat__print_shadow_stats(ev2,
+						      ev2->val * ev2->scale,
+						      sample->cpu,
+						      &ctx,
+						      NULL);
+		}
+		evsel->leader->gnum = 0;
+	}
+}
+
 static void process_event(struct perf_script *script,
 			  struct perf_sample *sample, struct perf_evsel *evsel,
 			  struct addr_location *al,
@@ -1481,6 +1560,9 @@ static void process_event(struct perf_script *script,
 	if (PRINT_FIELD(PHYS_ADDR))
 		printf("%16" PRIx64, sample->phys_addr);
 	printf("\n");
+
+	if (PRINT_FIELD(METRIC))
+		print_metric(script, thread, evsel, sample);
 }
 
 static struct scripting_ops	*scripting_ops;
diff --git a/tools/perf/util/evsel.h b/tools/perf/util/evsel.h
index 25f21cde8826..27a72b883645 100644
--- a/tools/perf/util/evsel.h
+++ b/tools/perf/util/evsel.h
@@ -108,6 +108,8 @@ struct perf_evsel {
 	struct cpu_map		*cpus;
 	struct cpu_map		*own_cpus;
 	struct thread_map	*threads;
+	u64			val;
+	int			gnum;
 	unsigned int		sample_size;
 	int			id_pos;
 	int			is_pos;
diff --git a/tools/perf/util/metricgroup.c b/tools/perf/util/metricgroup.c
index 0ddd9c199227..6fd709017bbc 100644
--- a/tools/perf/util/metricgroup.c
+++ b/tools/perf/util/metricgroup.c
@@ -38,6 +38,10 @@ struct metric_event *metricgroup__lookup(struct rblist *metric_events,
 	struct metric_event me = {
 		.evsel = evsel
 	};
+
+	if (!metric_events)
+		return NULL;
+
 	nd = rblist__find(metric_events, &me);
 	if (nd)
 		return container_of(nd, struct metric_event, nd);
-- 
2.13.6

Re: [PATCH v1 1/5] perf, tools: Document some missing perf.data headers

[Jiri Olsa]

On Thu, Nov 09, 2017 at 06:55:24AM -0800, Andi Kleen wrote:
> From: Andi Kleen <ak@linux.intel.com>
> 
> Document STAT and CACHE header entries.
> 
> Signed-off-by: Andi Kleen <ak@linux.intel.com>

Acked-by: Jiri Olsa <jolsa@kernel.org>

thanks,
jirka

> ---
>  tools/perf/Documentation/perf.data-file-format.txt | 23 ++++++++++++++++++++++
>  1 file changed, 23 insertions(+)
> 
> diff --git a/tools/perf/Documentation/perf.data-file-format.txt b/tools/perf/Documentation/perf.data-file-format.txt
> index e90c59c6d815..15e8b48077ba 100644
> --- a/tools/perf/Documentation/perf.data-file-format.txt
> +++ b/tools/perf/Documentation/perf.data-file-format.txt
> @@ -238,6 +238,29 @@ struct auxtrace_index {
>  	struct auxtrace_index_entry entries[PERF_AUXTRACE_INDEX_ENTRY_COUNT];
>  };
>  
> +	HEADER_STAT = 19,
> +
> +This is merely a flag signifying that the data section contains data
> +recorded from perf stat record.
> +
> +	HEADER_CACHE = 20,
> +
> +Description of the cache hierarchy. Based on the Linux sysfs format
> +in /sys/devices/system/cpu/cpu*/cache/
> +
> +	u32 version	Currently always 1
> +	u32 number_of_cache_levels
> +
> +struct {
> +	u32	level;
> +	u32	line_size;
> +	u32	sets;
> +	u32	ways;
> +	struct perf_header_string type;
> +	struct perf_header_string size;
> +	struct perf_header_string map;
> +}[number_of_cache_levels];
> +
>  	other bits are reserved and should ignored for now
>  	HEADER_FEAT_BITS	= 256,
>  
> -- 
> 2.13.6
> 

Re: [PATCH v1 2/5] perf, tools: Save event scaling factors in perf.data

[Jiri Olsa]

On Thu, Nov 09, 2017 at 06:55:25AM -0800, Andi Kleen wrote:
> From: Andi Kleen <ak@linux.intel.com>
> 
> To process metrics, perf script needs to know the scaling
> factors reported by sysfs for events. Save the scaling factors
> in the perf.data metadata in a new SCALE header.
> 
> Signed-off-by: Andi Kleen <ak@linux.intel.com>

there's event_update event for scale and some other evsel stuff
that was before we were able to put header through pipe

it might be less code changes.. sythesize it via:

       if (has_scale(counter)) {
               err = perf_event__synthesize_event_update_scale(NULL, counter, process_synthesized_event);
               if (err < 0) {
                       pr_err("Couldn't synthesize evsel scale.\n");
                       return err;
               }
       }

and new tool's event_update callback, which is already implemented
in perf_event__process_event_update

having event instead of header feature might have
other drawbacks.. not sure it will fit here

jirka

Re: [PATCH v1 3/5] perf, tools, script: Allow printing period for non freq mode groups

[Jiri Olsa]

On Thu, Nov 09, 2017 at 06:55:26AM -0800, Andi Kleen wrote:
> From: Andi Kleen <ak@linux.intel.com>
> 
> When using leader sampling the values of the not sampled but counted
> events are shown by perf script in "period".
> 
> Currently printing period is only allowed when the main event
> has a period, that is it is in frequency mode.
> 
> This implies that we cannot dump the values of counted events
> when the leader event is not in frequency mode.
> 
> Just remove the check that the period must be set on all
> events. It will just be printed as 0 instead if it's not
> available.

it'll be the value of configured period, not 0

int perf_evsel__parse_sample(struct ...
  ...
  data->period = evsel->attr.sample_period;

  $ perf record -c 100000 
  $ perf script -F event,period | head -3
  Failed to open /tmp/perf-2048.map, continuing without symbols
      100000 cycles:ppp: 
      100000 cycles:ppp: 


other than that I think we can remove that check,
because we will have always sane number in period

Acked-by: Jiri Olsa <jolsa@kernel.org>

thanks,
jirka

Re: [PATCH v1 4/5] perf, tools: Add fallback in perf_evsel__nr_cpus for no map

[Jiri Olsa]

On Thu, Nov 09, 2017 at 06:55:27AM -0800, Andi Kleen wrote:
> From: Andi Kleen <ak@linux.intel.com>
> 
> Support the case of the event having no cpumap in perf_evsel__nr_cpus.
> Just return 1 in this case.  This can happen in perf script
> when it uses the perf stat shadow functions.

why 1, where in shadow code? you can synthesize cpus for event
via event_update event

jirka

> 
> Signed-off-by: Andi Kleen <ak@linux.intel.com>
> ---
>  tools/perf/util/evsel.h | 3 ++-
>  1 file changed, 2 insertions(+), 1 deletion(-)
> 
> diff --git a/tools/perf/util/evsel.h b/tools/perf/util/evsel.h
> index db658785d828..25f21cde8826 100644
> --- a/tools/perf/util/evsel.h
> +++ b/tools/perf/util/evsel.h
> @@ -157,7 +157,8 @@ static inline struct cpu_map *perf_evsel__cpus(struct perf_evsel *evsel)
>  
>  static inline int perf_evsel__nr_cpus(struct perf_evsel *evsel)
>  {
> -	return perf_evsel__cpus(evsel)->nr;
> +	struct cpu_map *map = perf_evsel__cpus(evsel);
> +	return map ? map->nr : 1;
>  }
>  
>  void perf_counts_values__scale(struct perf_counts_values *count,
> -- 
> 2.13.6
> 

Re: [PATCH v1 5/5] perf, tools, script: Allow computing metrics in perf script

[Jiri Olsa]

On Thu, Nov 09, 2017 at 06:55:28AM -0800, Andi Kleen wrote:

SNIP

>  static void process_event(struct perf_script *script,
>  			  struct perf_sample *sample, struct perf_evsel *evsel,
>  			  struct addr_location *al,
> @@ -1481,6 +1560,9 @@ static void process_event(struct perf_script *script,
>  	if (PRINT_FIELD(PHYS_ADDR))
>  		printf("%16" PRIx64, sample->phys_addr);
>  	printf("\n");
> +
> +	if (PRINT_FIELD(METRIC))
> +		print_metric(script, thread, evsel, sample);
>  }
>  
>  static struct scripting_ops	*scripting_ops;
> diff --git a/tools/perf/util/evsel.h b/tools/perf/util/evsel.h
> index 25f21cde8826..27a72b883645 100644
> --- a/tools/perf/util/evsel.h
> +++ b/tools/perf/util/evsel.h
> @@ -108,6 +108,8 @@ struct perf_evsel {
>  	struct cpu_map		*cpus;
>  	struct cpu_map		*own_cpus;
>  	struct thread_map	*threads;
> +	u64			val;
> +	int			gnum;

please put this to the priv (struct perf_stat_evsel)
with some comments that it's used in script only

also there's per event dump feature that uses evsel->priv
(please rebase to latest Arnaldo's perf/core) in script,
so make sure you are mutualy exclusive

thanks,
jirka

Re: [PATCH v1 5/5] perf, tools, script: Allow computing metrics in perf script

[Arnaldo Carvalho de Melo]

Em Mon, Nov 13, 2017 at 10:30:11AM +0100, Jiri Olsa escreveu:
> On Thu, Nov 09, 2017 at 06:55:28AM -0800, Andi Kleen wrote:
> 
> SNIP
> 
> >  static void process_event(struct perf_script *script,
> >  			  struct perf_sample *sample, struct perf_evsel *evsel,
> >  			  struct addr_location *al,
> > @@ -1481,6 +1560,9 @@ static void process_event(struct perf_script *script,
> >  	if (PRINT_FIELD(PHYS_ADDR))
> >  		printf("%16" PRIx64, sample->phys_addr);
> >  	printf("\n");
> > +
> > +	if (PRINT_FIELD(METRIC))
> > +		print_metric(script, thread, evsel, sample);
> >  }
> >  
> >  static struct scripting_ops	*scripting_ops;
> > diff --git a/tools/perf/util/evsel.h b/tools/perf/util/evsel.h
> > index 25f21cde8826..27a72b883645 100644
> > --- a/tools/perf/util/evsel.h
> > +++ b/tools/perf/util/evsel.h
> > @@ -108,6 +108,8 @@ struct perf_evsel {
> >  	struct cpu_map		*cpus;
> >  	struct cpu_map		*own_cpus;
> >  	struct thread_map	*threads;
> > +	u64			val;
> > +	int			gnum;
> 
> please put this to the priv (struct perf_stat_evsel)
> with some comments that it's used in script only
> 
> also there's per event dump feature that uses evsel->priv
> (please rebase to latest Arnaldo's perf/core) in script,
> so make sure you are mutualy exclusive

No need, priv is supposed to be used by one tool only, i.e. its a too
private area, that perf_evsel_stats thing started to be used by multiple
tools, so I moved it to perf_evsel->stats and made perf_evsel->priv be
a tool priv area again.

I.e. move the above values to perf_stat_evsel and access it via
evsel->stats.

- Arnaldo
 
> thanks,
> jirka

Re: [PATCH v1 1/5] perf, tools: Document some missing perf.data headers

[Arnaldo Carvalho de Melo]

Em Mon, Nov 13, 2017 at 09:54:03AM +0100, Jiri Olsa escreveu:
> On Thu, Nov 09, 2017 at 06:55:24AM -0800, Andi Kleen wrote:
> > From: Andi Kleen <ak@linux.intel.com>
> > 
> > Document STAT and CACHE header entries.
> > 
> > Signed-off-by: Andi Kleen <ak@linux.intel.com>
> 
> Acked-by: Jiri Olsa <jolsa@kernel.org>

Applied.

- Arnaldo

Re: [PATCH v1 3/5] perf, tools, script: Allow printing period for non freq mode groups

[Arnaldo Carvalho de Melo]

Em Mon, Nov 13, 2017 at 10:11:58AM +0100, Jiri Olsa escreveu:
> On Thu, Nov 09, 2017 at 06:55:26AM -0800, Andi Kleen wrote:
> > From: Andi Kleen <ak@linux.intel.com>
> > 
> > When using leader sampling the values of the not sampled but counted
> > events are shown by perf script in "period".
> > 
> > Currently printing period is only allowed when the main event
> > has a period, that is it is in frequency mode.
> > 
> > This implies that we cannot dump the values of counted events
> > when the leader event is not in frequency mode.
> > 
> > Just remove the check that the period must be set on all
> > events. It will just be printed as 0 instead if it's not
> > available.
> 
> it'll be the value of configured period, not 0
> 
> int perf_evsel__parse_sample(struct ...
>   ...
>   data->period = evsel->attr.sample_period;
> 
>   $ perf record -c 100000 
>   $ perf script -F event,period | head -3
>   Failed to open /tmp/perf-2048.map, continuing without symbols
>       100000 cycles:ppp: 
>       100000 cycles:ppp: 
> 
> 
> other than that I think we can remove that check,
> because we will have always sane number in period
> 
> Acked-by: Jiri Olsa <jolsa@kernel.org>

Applied, after adding your comments.

- Arnaldo

Re: [PATCH v1 4/5] perf, tools: Add fallback in perf_evsel__nr_cpus for no map

[Andi Kleen]

On Mon, Nov 13, 2017 at 10:22:30AM +0100, Jiri Olsa wrote:
> On Thu, Nov 09, 2017 at 06:55:27AM -0800, Andi Kleen wrote:
> > From: Andi Kleen <ak@linux.intel.com>
> > 
> > Support the case of the event having no cpumap in perf_evsel__nr_cpus.
> > Just return 1 in this case.  This can happen in perf script
> > when it uses the perf stat shadow functions.
> 
> why 1, where in shadow code? you can synthesize cpus for event
> via event_update event

For sampling it should be always 1, right?

Where:

#0  0x0000000000570e03 in __perf_evsel_stat__is (evsel=0x2690ce0,
id=PERF_STAT_EVSEL_ID__CYCLES_IN_TX) at util/stat.c:75
#1  0x0000000000572375 in perf_stat__update_shadow_stats
(counter=0x2690ce0, count=3744, cpu=0) at util/stat-shadow.c:194

-Andi

[tip:perf/core] perf tools: Document some missing perf.data headers

[tip-bot for Andi Kleen]

Commit-ID:  35c0a81a97692cc0afe3d005c9a737bbde06e784
Gitweb:     https://git.kernel.org/tip/35c0a81a97692cc0afe3d005c9a737bbde06e784
Author:     Andi Kleen <ak@linux.intel.com>
AuthorDate: Thu, 9 Nov 2017 06:55:24 -0800
Committer:  Arnaldo Carvalho de Melo <acme@redhat.com>
CommitDate: Thu, 16 Nov 2017 14:49:55 -0300

perf tools: Document some missing perf.data headers

Document STAT and CACHE header entries.

Signed-off-by: Andi Kleen <ak@linux.intel.com>
Acked-by: Jiri Olsa <jolsa@kernel.org>
Link: http://lkml.kernel.org/r/20171109145528.23371-2-andi@firstfloor.org
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
---
 tools/perf/Documentation/perf.data-file-format.txt | 23 ++++++++++++++++++++++
 1 file changed, 23 insertions(+)

diff --git a/tools/perf/Documentation/perf.data-file-format.txt b/tools/perf/Documentation/perf.data-file-format.txt
index e90c59c..15e8b48 100644
--- a/tools/perf/Documentation/perf.data-file-format.txt
+++ b/tools/perf/Documentation/perf.data-file-format.txt
@@ -238,6 +238,29 @@ struct auxtrace_index {
 	struct auxtrace_index_entry entries[PERF_AUXTRACE_INDEX_ENTRY_COUNT];
 };
 
+	HEADER_STAT = 19,
+
+This is merely a flag signifying that the data section contains data
+recorded from perf stat record.
+
+	HEADER_CACHE = 20,
+
+Description of the cache hierarchy. Based on the Linux sysfs format
+in /sys/devices/system/cpu/cpu*/cache/
+
+	u32 version	Currently always 1
+	u32 number_of_cache_levels
+
+struct {
+	u32	level;
+	u32	line_size;
+	u32	sets;
+	u32	ways;
+	struct perf_header_string type;
+	struct perf_header_string size;
+	struct perf_header_string map;
+}[number_of_cache_levels];
+
 	other bits are reserved and should ignored for now
 	HEADER_FEAT_BITS	= 256,
 

[tip:perf/core] perf script: Allow printing period for non freq mode groups

[tip-bot for Andi Kleen]

Commit-ID:  5039c8a28fa97b8dce7b363a5ecd4bee2b87bf03
Gitweb:     https://git.kernel.org/tip/5039c8a28fa97b8dce7b363a5ecd4bee2b87bf03
Author:     Andi Kleen <ak@linux.intel.com>
AuthorDate: Thu, 9 Nov 2017 06:55:26 -0800
Committer:  Arnaldo Carvalho de Melo <acme@redhat.com>
CommitDate: Thu, 16 Nov 2017 14:49:55 -0300

perf script: Allow printing period for non freq mode groups

When using leader sampling the values of the not sampled but counted
events are shown by perf script in "period".

Currently printing period is only allowed when the main event has a
period, that is it is in frequency mode.

This implies that we cannot dump the values of counted events when the
leader event is not in frequency mode.

Just remove the check that the period must be set on all events. It will
just be printed as 0 instead if it's not available.

This fixes the following:

  $ perf record -c 100000 -e '{cycles,branches}:S'
  $ perf script -F event,period

Further commentary by Jiri Olsa:

The period will be the value of configured period, not 0:

int perf_evsel__parse_sample(struct ...
  ...
  data->period = evsel->attr.sample_period;

  $ perf record -c 100000
  $ perf script -F event,period | head -3
  Failed to open /tmp/perf-2048.map, continuing without symbols
      100000 cycles:ppp:
      100000 cycles:ppp:

other than that I think we can remove that check, because we will have
always sane number in period

Signed-off-by: Andi Kleen <ak@linux.intel.com>
Acked-by: Jiri Olsa <jolsa@kernel.org>
Link: http://lkml.kernel.org/r/20171109145528.23371-4-andi@firstfloor.org
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
---
 tools/perf/builtin-script.c | 5 -----
 1 file changed, 5 deletions(-)

diff --git a/tools/perf/builtin-script.c b/tools/perf/builtin-script.c
index 9b43bda..ee7c7aa 100644
--- a/tools/perf/builtin-script.c
+++ b/tools/perf/builtin-script.c
@@ -423,11 +423,6 @@ static int perf_evsel__check_attr(struct perf_evsel *evsel,
 					   PERF_OUTPUT_CPU, allow_user_set))
 		return -EINVAL;
 
-	if (PRINT_FIELD(PERIOD) &&
-		perf_evsel__check_stype(evsel, PERF_SAMPLE_PERIOD, "PERIOD",
-					PERF_OUTPUT_PERIOD))
-		return -EINVAL;
-
 	if (PRINT_FIELD(IREGS) &&
 		perf_evsel__check_stype(evsel, PERF_SAMPLE_REGS_INTR, "IREGS",
 					PERF_OUTPUT_IREGS))