[PATCH v2 0/6] perf tools: Use SIGUSR2 control data dumpping

[PATCH v2 0/6] perf tools: Use SIGUSR2 control data dumpping

[Wang Nan]

This patch set is based on perf/core.

v1 -> v2: Fix a bug which triggers output switching without
          '--switch-output' provided.

Patch 5/10 in v1 has a bug: with following cmdline:
 # perf record -e intel_bts// --per-thread --snapshot -p 13588

Sending 'SIGUSR2' to perf triggers output switching, which us
undesirable. The reason is in signal handler perf doesn't consider if
user provided --switch-output explicitly.

With this patchset applied:

 # /perf record --switch-output -e intel_bts// --per-thread --snapshot -p 13588 &
 [1] 15116
 # kill -s SIGUSR2 15116
 # [ perf record: dump data: Woken up 5 times ]
 [ perf record: Dump perf.data.2016041506170000 ]
 
 # kill -s SIGUSR2 15116
 # [ perf record: dump data: Woken up 3 times ]
 [ perf record: Dump perf.data.2016041506170447 ]
 
 # fg
 /perf record --switch-output -e intel_bts// --per-thread --snapshot -p 13588
 ^C[ perf record: Woken up 3 times to write data ]
 [ perf record: Dump perf.data.2016041506170785 ]
 [ perf record: Captured and wrote 0.012 MB perf.data.<timestamp> ]
 
 # ls -l ./perf.data*
 -rw------- 1 root root 4212888 Apr 15 06:17 ./perf.data.2016041506170000
 -rw------- 1 root root 4212704 Apr 15 06:17 ./perf.data.2016041506170447
 -rw------- 1 root root   18368 Apr 15 06:17 ./perf.data.2016041506170785

The first 2 perf.data works fine. The final one has no samples.

And:
 # rm ./perf.data*
 # /perf record -e intel_bts// --per-thread --snapshot -p 13588 &
 [1] 15396
 # kill -s SIGUSR2 15396
 # kill -s SIGUSR2 15396
 # fg
 /perf record -e intel_bts// --per-thread --snapshot -p 13588
 ^C[ perf record: Woken up 9 times to write data ]
 [ perf record: Captured and wrote 8.012 MB perf.data ]
 
 # ls -l ./perf.data*
 -rw------- 1 root root 8417552 Apr 15 06:20 ./perf.data

(In v1, the second test trigger output switching incorrectly)

Patch 1-2/6 in this patchset makes switch output trigger similar to
auxtrace snapshot trigger. Patch 1/6 introduces a 'trigger' class for
them. It would be better to merge patch 1/6 with commit c0bdc1c ("perf
record: Turns auxtrace_snapshot_enable into 3 states"). However, Ingo
has collected that commit, so make it a separated patch.

Signed-off-by: Wang Nan <wangnan0@huawei.com>
Cc: Arnaldo Carvalho de Melo <acme@redhat.com>
Cc: Adrian Hunter <adrian.hunter@intel.com>
Cc: Jiri Olsa <jolsa@kernel.org>
Cc: Masami Hiramatsu <mhiramat@kernel.org>
Cc: Namhyung Kim <namhyung@kernel.org>
Cc: Zefan Li <lizefan@huawei.com>
Cc: pi3orama@163.com
Cc: He Kuang <hekuang@huawei.com>

Wang Nan (6):
  perf tools: Extrace trigger class from auxtrace_snapshot
  perf record: Split output into multiple files via '--switch-output'
  perf record: Force enable --timestamp-filename when --switch-output is
    provided
  perf record: Disable buildid cache options by default in switch output
    mode
  perf record: Re-synthesize tracking events after output switching
  perf record: Generate tracking events for process forked by perf

 tools/perf/builtin-record.c | 147 +++++++++++++++++++++++++++++++-------------
 tools/perf/util/util.h      |  60 ++++++++++++++++++
 2 files changed, 163 insertions(+), 44 deletions(-)

-- 
1.8.3.4

[PATCH v2 1/6] perf tools: Extrace trigger class from auxtrace_snapshot

[Wang Nan]

Create a new 'class' named 'trigger' to model state of a trigger
and implement auxtrace_snapshot with it.

'trigger' defines 3 state transfering functions ('on', 'enable',
'disable') and 1 state query function ('is_enabled'). The state
'ON' and 'OFF' take higher priority than 'ENABLED' and 'DISABLED'.
A trigger must be 'on' before 'enable' and 'disable' take effect.

Signed-off-by: Wang Nan <wangnan0@huawei.com>
Cc: Arnaldo Carvalho de Melo <acme@redhat.com>
Cc: Adrian Hunter <adrian.hunter@intel.com>
Cc: Jiri Olsa <jolsa@kernel.org>
Cc: Masami Hiramatsu <mhiramat@kernel.org>
Cc: Namhyung Kim <namhyung@kernel.org>
Cc: Zefan Li <lizefan@huawei.com>
Cc: pi3orama@163.com
Cc: He Kuang <hekuang@huawei.com>
---
 tools/perf/builtin-record.c | 37 +-------------------------------
 tools/perf/util/util.h      | 51 +++++++++++++++++++++++++++++++++++++++++++++
 2 files changed, 52 insertions(+), 36 deletions(-)

diff --git a/tools/perf/builtin-record.c b/tools/perf/builtin-record.c
index 3239a6e..13fe9a6 100644
--- a/tools/perf/builtin-record.c
+++ b/tools/perf/builtin-record.c
@@ -127,42 +127,7 @@ static volatile int done;
 static volatile int signr = -1;
 static volatile int child_finished;
 
-static volatile enum {
-	AUXTRACE_SNAPSHOT_OFF = -1,
-	AUXTRACE_SNAPSHOT_DISABLED = 0,
-	AUXTRACE_SNAPSHOT_ENABLED = 1,
-} auxtrace_snapshot_state = AUXTRACE_SNAPSHOT_OFF;
-
-static inline void
-auxtrace_snapshot_on(void)
-{
-	auxtrace_snapshot_state = AUXTRACE_SNAPSHOT_DISABLED;
-}
-
-static inline void
-auxtrace_snapshot_enable(void)
-{
-	if (auxtrace_snapshot_state == AUXTRACE_SNAPSHOT_OFF)
-		return;
-	auxtrace_snapshot_state = AUXTRACE_SNAPSHOT_ENABLED;
-}
-
-static inline void
-auxtrace_snapshot_disable(void)
-{
-	if (auxtrace_snapshot_state == AUXTRACE_SNAPSHOT_OFF)
-		return;
-	auxtrace_snapshot_state = AUXTRACE_SNAPSHOT_DISABLED;
-}
-
-static inline bool
-auxtrace_snapshot_is_enabled(void)
-{
-	if (auxtrace_snapshot_state == AUXTRACE_SNAPSHOT_OFF)
-		return false;
-	return auxtrace_snapshot_state == AUXTRACE_SNAPSHOT_ENABLED;
-}
-
+static DEFINE_TRIGGER(auxtrace_snapshot, OFF);
 static volatile int auxtrace_snapshot_err;
 static volatile int auxtrace_record__snapshot_started;
 
diff --git a/tools/perf/util/util.h b/tools/perf/util/util.h
index 3bf3de8..8121029 100644
--- a/tools/perf/util/util.h
+++ b/tools/perf/util/util.h
@@ -361,4 +361,55 @@ typedef void (*print_binary_t)(enum binary_printer_ops,
 void print_binary(unsigned char *data, size_t len,
 		  size_t bytes_per_line, print_binary_t printer,
 		  void *extra);
+
+struct trigger {
+	volatile enum {
+		TRIGGER_OFF = -1,
+		TRIGGER_DISABLED = 0,
+		TRIGGER_ENABLED = 1,
+	} state;
+};
+
+static inline void
+trigger_on(struct trigger *s)
+{
+	s->state = TRIGGER_DISABLED;
+}
+
+static inline void
+trigger_enable(struct trigger *s)
+{
+	if (s->state != TRIGGER_OFF)
+		s->state = TRIGGER_ENABLED;
+}
+
+static inline void
+trigger_disable(struct trigger *s)
+{
+	if (s->state != TRIGGER_OFF)
+		s->state = TRIGGER_DISABLED;
+}
+
+static inline bool
+trigger_is_enabled(struct trigger *s)
+{
+	if (s->state != TRIGGER_OFF)
+		return s->state == TRIGGER_ENABLED;
+	return false;
+}
+
+#define __TRIGGER_VAR(n) n##_state
+#define __DEF_TRIGGER_VOID_FUNC(n, op)	\
+static inline void n##_##op(void) {trigger_##op(&__TRIGGER_VAR(n)); }
+
+#define __DEF_TRIGGER_FUNC(n, type, op)	\
+static inline type n##_##op(void) {return trigger_##op(&__TRIGGER_VAR(n)); }
+
+#define DEFINE_TRIGGER(n, def)				\
+struct trigger n##_state = {.state = TRIGGER_##def};\
+__DEF_TRIGGER_VOID_FUNC(n, on)				\
+__DEF_TRIGGER_VOID_FUNC(n, enable)			\
+__DEF_TRIGGER_VOID_FUNC(n, disable)			\
+__DEF_TRIGGER_FUNC(n, bool, is_enabled)
+
 #endif /* GIT_COMPAT_UTIL_H */
-- 
1.8.3.4

[PATCH v2 2/6] perf record: Split output into multiple files via '--switch-output'

[Wang Nan]

Allow 'perf record' splits its output into multiple files.

For example:

 # ~/perf record -a --timestamp-filename --switch-output &
 [1] 10763
 # kill -s SIGUSR2 10763
 [ perf record: dump data: Woken up 1 times ]
 # [ perf record: Dump perf.data.2015122622314468 ]

 # kill -s SIGUSR2 10763
 [ perf record: dump data: Woken up 1 times ]
 # [ perf record: Dump perf.data.2015122622314762 ]

 # kill -s SIGUSR2 10763
 [ perf record: dump data: Woken up 1 times ]
 #[ perf record: Dump perf.data.2015122622315171 ]

 # fg
 perf record -a --timestamp-filename --switch-output
 ^C[ perf record: Woken up 1 times to write data ]
 [ perf record: Dump perf.data.2015122622315513 ]
 [ perf record: Captured and wrote 0.014 MB perf.data.<timestamp> (296 samples) ]

 # ls -l
 total 920
 -rw------- 1 root root 797692 Dec 26 22:31 perf.data.2015122622314468
 -rw------- 1 root root  59960 Dec 26 22:31 perf.data.2015122622314762
 -rw------- 1 root root  59912 Dec 26 22:31 perf.data.2015122622315171
 -rw------- 1 root root  19220 Dec 26 22:31 perf.data.2015122622315513

Signed-off-by: Wang Nan <wangnan0@huawei.com>
Signed-off-by: He Kuang <hekuang@huawei.com>
Cc: Arnaldo Carvalho de Melo <acme@redhat.com>
Cc: Adrian Hunter <adrian.hunter@intel.com>
Cc: Jiri Olsa <jolsa@kernel.org>
Cc: Masami Hiramatsu <masami.hiramatsu.pt@hitachi.com>
Cc: Namhyung Kim <namhyung@kernel.org>
Cc: Zefan Li <lizefan@huawei.com>
Cc: pi3orama@163.com
---
 tools/perf/builtin-record.c | 41 ++++++++++++++++++++++++++++++++++-------
 tools/perf/util/util.h      | 11 ++++++++++-
 2 files changed, 44 insertions(+), 8 deletions(-)

diff --git a/tools/perf/builtin-record.c b/tools/perf/builtin-record.c
index 13fe9a6..ca40726 100644
--- a/tools/perf/builtin-record.c
+++ b/tools/perf/builtin-record.c
@@ -57,6 +57,7 @@ struct record {
 	bool			no_buildid_cache_set;
 	bool			buildid_all;
 	bool			timestamp_filename;
+	bool			switch_output;
 	unsigned long long	samples;
 };
 
@@ -131,6 +132,8 @@ static DEFINE_TRIGGER(auxtrace_snapshot, OFF);
 static volatile int auxtrace_snapshot_err;
 static volatile int auxtrace_record__snapshot_started;
 
+static DEFINE_TRIGGER(switch_output, OFF);
+
 static void sig_handler(int sig)
 {
 	if (sig == SIGCHLD)
@@ -649,9 +652,12 @@ static int __cmd_record(struct record *rec, int argc, const char **argv)
 	signal(SIGINT, sig_handler);
 	signal(SIGTERM, sig_handler);
 
-	if (rec->opts.auxtrace_snapshot_mode) {
+	if (rec->opts.auxtrace_snapshot_mode || rec->switch_output) {
 		signal(SIGUSR2, snapshot_sig_handler);
-		auxtrace_snapshot_on();
+		if (rec->opts.auxtrace_snapshot_mode)
+			auxtrace_snapshot_on();
+		if (rec->switch_output)
+			switch_output_on();
 	} else {
 		signal(SIGUSR2, SIG_IGN);
 	}
@@ -781,6 +787,7 @@ static int __cmd_record(struct record *rec, int argc, const char **argv)
 	}
 
 	auxtrace_snapshot_enable();
+	switch_output_enable();
 	for (;;) {
 		unsigned long long hits = rec->samples;
 
@@ -801,6 +808,21 @@ static int __cmd_record(struct record *rec, int argc, const char **argv)
 			}
 		}
 
+		if (switch_output_is_disabled()) {
+			switch_output_enable();
+
+			if (!quiet)
+				fprintf(stderr, "[ perf record: dump data: Woken up %ld times ]\n",
+					waking);
+			waking = 0;
+			fd = record__switch_output(rec, false);
+			if (fd < 0) {
+				pr_err("Failed to switch to new file\n");
+				err = fd;
+				goto out_child;
+			}
+		}
+
 		if (hits == rec->samples) {
 			if (done || draining)
 				break;
@@ -1259,6 +1281,8 @@ struct option __record_options[] = {
 		    "Record build-id of all DSOs regardless of hits"),
 	OPT_BOOLEAN(0, "timestamp-filename", &record.timestamp_filename,
 		    "append timestamp to output filename"),
+	OPT_BOOLEAN(0, "switch-output", &record.switch_output,
+		    "Switch output when receive SIGUSR2"),
 	OPT_END()
 };
 
@@ -1407,9 +1431,12 @@ out_symbol_exit:
 
 static void snapshot_sig_handler(int sig __maybe_unused)
 {
-	if (!auxtrace_snapshot_is_enabled())
-		return;
-	auxtrace_snapshot_disable();
-	auxtrace_snapshot_err = auxtrace_record__snapshot_start(record.itr);
-	auxtrace_record__snapshot_started = 1;
+	if (auxtrace_snapshot_is_enabled()) {
+		auxtrace_snapshot_disable();
+		auxtrace_snapshot_err = auxtrace_record__snapshot_start(record.itr);
+		auxtrace_record__snapshot_started = 1;
+	}
+
+	if (switch_output_is_enabled())
+		switch_output_disable();
 }
diff --git a/tools/perf/util/util.h b/tools/perf/util/util.h
index 8121029..46bf35b 100644
--- a/tools/perf/util/util.h
+++ b/tools/perf/util/util.h
@@ -398,6 +398,14 @@ trigger_is_enabled(struct trigger *s)
 	return false;
 }
 
+static inline bool
+trigger_is_disabled(struct trigger *s)
+{
+	if (s->state != TRIGGER_OFF)
+		return s->state == TRIGGER_DISABLED;
+	return false;
+}
+
 #define __TRIGGER_VAR(n) n##_state
 #define __DEF_TRIGGER_VOID_FUNC(n, op)	\
 static inline void n##_##op(void) {trigger_##op(&__TRIGGER_VAR(n)); }
@@ -410,6 +418,7 @@ struct trigger n##_state = {.state = TRIGGER_##def};\
 __DEF_TRIGGER_VOID_FUNC(n, on)				\
 __DEF_TRIGGER_VOID_FUNC(n, enable)			\
 __DEF_TRIGGER_VOID_FUNC(n, disable)			\
-__DEF_TRIGGER_FUNC(n, bool, is_enabled)
+__DEF_TRIGGER_FUNC(n, bool, is_enabled)			\
+__DEF_TRIGGER_FUNC(n, bool, is_disabled)
 
 #endif /* GIT_COMPAT_UTIL_H */
-- 
1.8.3.4

Re: [PATCH v2 2/6] perf record: Split output into multiple files via '--switch-output'

[Arnaldo Carvalho de Melo]

Em Thu, Apr 14, 2016 at 02:22:01PM +0000, Wang Nan escreveu:
> Allow 'perf record' splits its output into multiple files.
> 
> For example:
> 
>  # ~/perf record -a --timestamp-filename --switch-output &
>  [1] 10763
>  # kill -s SIGUSR2 10763
>  [ perf record: dump data: Woken up 1 times ]
>  # [ perf record: Dump perf.data.2015122622314468 ]
> 
>  # kill -s SIGUSR2 10763
>  [ perf record: dump data: Woken up 1 times ]
>  # [ perf record: Dump perf.data.2015122622314762 ]
> 
>  # kill -s SIGUSR2 10763
>  [ perf record: dump data: Woken up 1 times ]
>  #[ perf record: Dump perf.data.2015122622315171 ]
> 
>  # fg
>  perf record -a --timestamp-filename --switch-output
>  ^C[ perf record: Woken up 1 times to write data ]
>  [ perf record: Dump perf.data.2015122622315513 ]
>  [ perf record: Captured and wrote 0.014 MB perf.data.<timestamp> (296 samples) ]

Added this as an initial man page entry:


diff --git a/tools/perf/Documentation/perf-record.txt b/tools/perf/Documentation/perf-record.txt
index 19aa17532a16..a77a431ca4ef 100644
--- a/tools/perf/Documentation/perf-record.txt
+++ b/tools/perf/Documentation/perf-record.txt
@@ -347,6 +347,14 @@ Configure all used events to run in kernel space.
 --all-user::
 Configure all used events to run in user space.
 
+--switch-output::
+Generate multiple perf.data files, timestamp prefixed, switching to a new one
+when receiving a SIGUSR2.
+
+A possible use case is to, given an external event, slice the perf.data file
+that gets then processed, possibly via a perf script, to decide if that
+particular perf.data snapshot should be kept or not.
+
 SEE ALSO
 --------
 linkperf:perf-stat[1], linkperf:perf-list[1]

[PATCH v2 3/6] perf record: Force enable --timestamp-filename when --switch-output is provided

[Wang Nan]

Without this patch, the last output doesn't have timestamp appended if
--timestamp-filename is not explicitly provided. For example:

 # perf record -a --switch-output &
 [1] 11224
 # kill -s SIGUSR2 11224
 [ perf record: dump data: Woken up 1 times ]
 # [ perf record: Dump perf.data.2015122622372823 ]

 # fg
 perf record -a --switch-output
 ^C[ perf record: Woken up 1 times to write data ]
 [ perf record: Captured and wrote 0.027 MB perf.data (540 samples) ]

 # ls -l
 total 836
 -rw------- 1 root root  33256 Dec 26 22:37 perf.data   <---- *Odd*
 -rw------- 1 root root 817156 Dec 26 22:37 perf.data.2015122622372823

Signed-off-by: Wang Nan <wangnan0@huawei.com>
Signed-off-by: He Kuang <hekuang@huawei.com>
Cc: Arnaldo Carvalho de Melo <acme@redhat.com>
Cc: Adrian Hunter <adrian.hunter@intel.com>
Cc: Jiri Olsa <jolsa@kernel.org>
Cc: Masami Hiramatsu <masami.hiramatsu.pt@hitachi.com>
Cc: Namhyung Kim <namhyung@kernel.org>
Cc: Zefan Li <lizefan@huawei.com>
Cc: pi3orama@163.com
---
 tools/perf/builtin-record.c | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/tools/perf/builtin-record.c b/tools/perf/builtin-record.c
index ca40726..32db7a7 100644
--- a/tools/perf/builtin-record.c
+++ b/tools/perf/builtin-record.c
@@ -1338,6 +1338,9 @@ int cmd_record(int argc, const char **argv, const char *prefix __maybe_unused)
 		return -EINVAL;
 	}
 
+	if (rec->switch_output)
+		rec->timestamp_filename = true;
+
 	if (!rec->itr) {
 		rec->itr = auxtrace_record__init(rec->evlist, &err);
 		if (err)
-- 
1.8.3.4

[PATCH v2 4/6] perf record: Disable buildid cache options by default in switch output mode

[Wang Nan]

Cost of buildid cache processing is high: reading all events in output
perf.data, opening each elf file to read buildids then copying them into
~/.debug directory. In switch output mode, these heavy works block perf
from receiving perf events for too long.

Enable no-buildid and no-buildid-cache by default if --switch-output
is provided. Still allow user use --no-no-buildid to explicitly enable
buildid in this case.

Signed-off-by: Wang Nan <wangnan0@huawei.com>
Signed-off-by: He Kuang <hekuang@huawei.com>
Cc: Arnaldo Carvalho de Melo <acme@redhat.com>
Cc: Adrian Hunter <adrian.hunter@intel.com>
Cc: Jiri Olsa <jolsa@kernel.org>
Cc: Masami Hiramatsu <masami.hiramatsu.pt@hitachi.com>
Cc: Namhyung Kim <namhyung@kernel.org>
Cc: Zefan Li <lizefan@huawei.com>
Cc: pi3orama@163.com
---
 tools/perf/builtin-record.c | 30 +++++++++++++++++++++++++++++-
 1 file changed, 29 insertions(+), 1 deletion(-)

diff --git a/tools/perf/builtin-record.c b/tools/perf/builtin-record.c
index 32db7a7..f561a1b 100644
--- a/tools/perf/builtin-record.c
+++ b/tools/perf/builtin-record.c
@@ -1374,8 +1374,36 @@ int cmd_record(int argc, const char **argv, const char *prefix __maybe_unused)
 "If some relocation was applied (e.g. kexec) symbols may be misresolved\n"
 "even with a suitable vmlinux or kallsyms file.\n\n");
 
-	if (rec->no_buildid_cache || rec->no_buildid)
+	if (rec->no_buildid_cache || rec->no_buildid) {
 		disable_buildid_cache();
+	} else if (rec->switch_output) {
+		/*
+		 * In 'perf record --switch-output', disable buildid
+		 * generation by default to reduce data file switching
+		 * overhead. Still generate buildid if they are required
+		 * explicitly using
+		 *
+		 *  perf record --signal-trigger --no-no-buildid \
+		 *              --no-no-buildid-cache
+		 *
+		 * Following code equals to:
+		 *
+		 * if ((rec->no_buildid || !rec->no_buildid_set) &&
+		 *     (rec->no_buildid_cache || !rec->no_buildid_cache_set))
+		 *         disable_buildid_cache();
+		 */
+		bool disable = true;
+
+		if (rec->no_buildid_set && !rec->no_buildid)
+			disable = false;
+		if (rec->no_buildid_cache_set && !rec->no_buildid_cache)
+			disable = false;
+		if (disable) {
+			rec->no_buildid = true;
+			rec->no_buildid_cache = true;
+			disable_buildid_cache();
+		}
+	}
 
 	if (rec->evlist->nr_entries == 0 &&
 	    perf_evlist__add_default(rec->evlist) < 0) {
-- 
1.8.3.4

[PATCH v2 5/6] perf record: Re-synthesize tracking events after output switching

[Wang Nan]

Tracking events describe kernel and threads. They are generated by
reading /proc/kallsyms, /proc/*/maps and /proc/*/task/* during
initialization of 'perf record', serialized into event sequences and put
at the head of 'perf.data'. In case of output switching, each output
file should contain those events.

This patch calls record__synthesize() during output switching, so the
event sequences described above can be collected again.

Signed-off-by: Wang Nan <wangnan0@huawei.com>
Signed-off-by: He Kuang <hekuang@huawei.com>
Cc: Arnaldo Carvalho de Melo <acme@redhat.com>
Cc: Adrian Hunter <adrian.hunter@intel.com>
Cc: Jiri Olsa <jolsa@kernel.org>
Cc: Masami Hiramatsu <masami.hiramatsu.pt@hitachi.com>
Cc: Namhyung Kim <namhyung@kernel.org>
Cc: Zefan Li <lizefan@huawei.com>
Cc: pi3orama@163.com
---
 tools/perf/builtin-record.c | 7 +++++++
 1 file changed, 7 insertions(+)

diff --git a/tools/perf/builtin-record.c b/tools/perf/builtin-record.c
index f561a1b..fd4a72e 100644
--- a/tools/perf/builtin-record.c
+++ b/tools/perf/builtin-record.c
@@ -500,6 +500,8 @@ record__finish_output(struct record *rec)
 	return;
 }
 
+static int record__synthesize(struct record *rec);
+
 static int
 record__switch_output(struct record *rec, bool at_exit)
 {
@@ -528,6 +530,11 @@ record__switch_output(struct record *rec, bool at_exit)
 	if (!quiet)
 		fprintf(stderr, "[ perf record: Dump %s.%s ]\n",
 			file->path, timestamp);
+
+	/* Output tracking events */
+	if (!at_exit)
+		record__synthesize(rec);
+
 	return fd;
 }
 
-- 
1.8.3.4

[PATCH v2 6/6] perf record: Generate tracking events for process forked by perf

[Wang Nan]

With 'perf record --switch-output' without -a, record__synthesize() in
record__switch_output() won't generate tracking events because there's
no thread_map in evlist. Which causes newly created perf.data doesn't
contain map and comm information.

This patch creates a fake thread_map and directly call
perf_event__synthesize_thread_map() for those events.

Signed-off-by: Wang Nan <wangnan0@huawei.com>
Signed-off-by: He Kuang <hekuang@huawei.com>
Cc: Arnaldo Carvalho de Melo <acme@redhat.com>
Cc: Adrian Hunter <adrian.hunter@intel.com>
Cc: Jiri Olsa <jolsa@kernel.org>
Cc: Masami Hiramatsu <masami.hiramatsu.pt@hitachi.com>
Cc: Namhyung Kim <namhyung@kernel.org>
Cc: Zefan Li <lizefan@huawei.com>
Cc: pi3orama@163.com
---
 tools/perf/builtin-record.c | 31 ++++++++++++++++++++++++++++++-
 1 file changed, 30 insertions(+), 1 deletion(-)

diff --git a/tools/perf/builtin-record.c b/tools/perf/builtin-record.c
index fd4a72e..568f6c7 100644
--- a/tools/perf/builtin-record.c
+++ b/tools/perf/builtin-record.c
@@ -500,6 +500,23 @@ record__finish_output(struct record *rec)
 	return;
 }
 
+static int record__synthesize_workload(struct record *rec)
+{
+	struct {
+		struct thread_map map;
+		struct thread_map_data map_data;
+	} thread_map;
+
+	thread_map.map.nr = 1;
+	thread_map.map.map[0].pid = rec->evlist->workload.pid;
+	thread_map.map.map[0].comm = NULL;
+	return perf_event__synthesize_thread_map(&rec->tool, &thread_map.map,
+						 process_synthesized_event,
+						 &rec->session->machines.host,
+						 rec->opts.sample_address,
+						 rec->opts.proc_map_timeout);
+}
+
 static int record__synthesize(struct record *rec);
 
 static int
@@ -532,9 +549,21 @@ record__switch_output(struct record *rec, bool at_exit)
 			file->path, timestamp);
 
 	/* Output tracking events */
-	if (!at_exit)
+	if (!at_exit) {
 		record__synthesize(rec);
 
+		/*
+		 * In 'perf record --switch-output' without -a,
+		 * record__synthesize() in record__switch_output() won't
+		 * generate tracking events because there's no thread_map
+		 * in evlist. Which causes newly created perf.data doesn't
+		 * contain map and comm information.
+		 * Create a fake thread_map and directly call
+		 * perf_event__synthesize_thread_map() for those events.
+		 */
+		if (target__none(&rec->opts.target))
+			record__synthesize_workload(rec);
+	}
 	return fd;
 }
 
-- 
1.8.3.4

Re: [PATCH v2 0/6] perf tools: Use SIGUSR2 control data dumpping

[Arnaldo Carvalho de Melo]

Em Thu, Apr 14, 2016 at 02:21:59PM +0000, Wang Nan escreveu:
> This patch set is based on perf/core.
> 
> v1 -> v2: Fix a bug which triggers output switching without
>           '--switch-output' provided.
> 
> Patch 5/10 in v1 has a bug: with following cmdline:
>  # perf record -e intel_bts// --per-thread --snapshot -p 13588

Ok, I moved the trigger stuff to a new file, util/trigger.h and added
missing man page entries, pushed to acme/perf/core, please take a look
if all is well before I push this to Ingo, tomorrow.

Adrian/others, if you can, please take a look.

- Arnaldo
 
> Sending 'SIGUSR2' to perf triggers output switching, which us
> undesirable. The reason is in signal handler perf doesn't consider if
> user provided --switch-output explicitly.
> 
> With this patchset applied:
> 
>  # /perf record --switch-output -e intel_bts// --per-thread --snapshot -p 13588 &
>  [1] 15116
>  # kill -s SIGUSR2 15116
>  # [ perf record: dump data: Woken up 5 times ]
>  [ perf record: Dump perf.data.2016041506170000 ]
>  
>  # kill -s SIGUSR2 15116
>  # [ perf record: dump data: Woken up 3 times ]
>  [ perf record: Dump perf.data.2016041506170447 ]
>  
>  # fg
>  /perf record --switch-output -e intel_bts// --per-thread --snapshot -p 13588
>  ^C[ perf record: Woken up 3 times to write data ]
>  [ perf record: Dump perf.data.2016041506170785 ]
>  [ perf record: Captured and wrote 0.012 MB perf.data.<timestamp> ]
>  
>  # ls -l ./perf.data*
>  -rw------- 1 root root 4212888 Apr 15 06:17 ./perf.data.2016041506170000
>  -rw------- 1 root root 4212704 Apr 15 06:17 ./perf.data.2016041506170447
>  -rw------- 1 root root   18368 Apr 15 06:17 ./perf.data.2016041506170785
> 
> The first 2 perf.data works fine. The final one has no samples.
> 
> And:
>  # rm ./perf.data*
>  # /perf record -e intel_bts// --per-thread --snapshot -p 13588 &
>  [1] 15396
>  # kill -s SIGUSR2 15396
>  # kill -s SIGUSR2 15396
>  # fg
>  /perf record -e intel_bts// --per-thread --snapshot -p 13588
>  ^C[ perf record: Woken up 9 times to write data ]
>  [ perf record: Captured and wrote 8.012 MB perf.data ]
>  
>  # ls -l ./perf.data*
>  -rw------- 1 root root 8417552 Apr 15 06:20 ./perf.data
> 
> (In v1, the second test trigger output switching incorrectly)
> 
> Patch 1-2/6 in this patchset makes switch output trigger similar to
> auxtrace snapshot trigger. Patch 1/6 introduces a 'trigger' class for
> them. It would be better to merge patch 1/6 with commit c0bdc1c ("perf
> record: Turns auxtrace_snapshot_enable into 3 states"). However, Ingo
> has collected that commit, so make it a separated patch.
> 
> Signed-off-by: Wang Nan <wangnan0@huawei.com>
> Cc: Arnaldo Carvalho de Melo <acme@redhat.com>
> Cc: Adrian Hunter <adrian.hunter@intel.com>
> Cc: Jiri Olsa <jolsa@kernel.org>
> Cc: Masami Hiramatsu <mhiramat@kernel.org>
> Cc: Namhyung Kim <namhyung@kernel.org>
> Cc: Zefan Li <lizefan@huawei.com>
> Cc: pi3orama@163.com
> Cc: He Kuang <hekuang@huawei.com>
> 
> Wang Nan (6):
>   perf tools: Extrace trigger class from auxtrace_snapshot
>   perf record: Split output into multiple files via '--switch-output'
>   perf record: Force enable --timestamp-filename when --switch-output is
>     provided
>   perf record: Disable buildid cache options by default in switch output
>     mode
>   perf record: Re-synthesize tracking events after output switching
>   perf record: Generate tracking events for process forked by perf
> 
>  tools/perf/builtin-record.c | 147 +++++++++++++++++++++++++++++++-------------
>  tools/perf/util/util.h      |  60 ++++++++++++++++++
>  2 files changed, 163 insertions(+), 44 deletions(-)
> 
> -- 
> 1.8.3.4

Re: [PATCH v2 0/6] perf tools: Use SIGUSR2 control data dumpping

[Adrian Hunter]

On 15/04/16 03:19, Arnaldo Carvalho de Melo wrote:
> Em Thu, Apr 14, 2016 at 02:21:59PM +0000, Wang Nan escreveu:
>> This patch set is based on perf/core.
>>
>> v1 -> v2: Fix a bug which triggers output switching without
>>           '--switch-output' provided.
>>
>> Patch 5/10 in v1 has a bug: with following cmdline:
>>  # perf record -e intel_bts// --per-thread --snapshot -p 13588
> 
> Ok, I moved the trigger stuff to a new file, util/trigger.h and added
> missing man page entries, pushed to acme/perf/core, please take a look
> if all is well before I push this to Ingo, tomorrow.
> 
> Adrian/others, if you can, please take a look.

Looks OK.  I gave intel_pt a quick test with snapshot mode which didn't seem
to be broken.

Using --switch-output with intel_pt gives data that does not decode in the
2nd and subsequent files because there is no continuity of switch events.
Also decoding the last file segfaults because the auxtrace index has not
been split between the files.

Probably need to prevent people using --switch-output with intel_pt, but
I'll look at it more next week.

> 
> - Arnaldo
>  
>> Sending 'SIGUSR2' to perf triggers output switching, which us
>> undesirable. The reason is in signal handler perf doesn't consider if
>> user provided --switch-output explicitly.
>>
>> With this patchset applied:
>>
>>  # /perf record --switch-output -e intel_bts// --per-thread --snapshot -p 13588 &
>>  [1] 15116
>>  # kill -s SIGUSR2 15116
>>  # [ perf record: dump data: Woken up 5 times ]
>>  [ perf record: Dump perf.data.2016041506170000 ]
>>  
>>  # kill -s SIGUSR2 15116
>>  # [ perf record: dump data: Woken up 3 times ]
>>  [ perf record: Dump perf.data.2016041506170447 ]
>>  
>>  # fg
>>  /perf record --switch-output -e intel_bts// --per-thread --snapshot -p 13588
>>  ^C[ perf record: Woken up 3 times to write data ]
>>  [ perf record: Dump perf.data.2016041506170785 ]
>>  [ perf record: Captured and wrote 0.012 MB perf.data.<timestamp> ]
>>  
>>  # ls -l ./perf.data*
>>  -rw------- 1 root root 4212888 Apr 15 06:17 ./perf.data.2016041506170000
>>  -rw------- 1 root root 4212704 Apr 15 06:17 ./perf.data.2016041506170447
>>  -rw------- 1 root root   18368 Apr 15 06:17 ./perf.data.2016041506170785
>>
>> The first 2 perf.data works fine. The final one has no samples.
>>
>> And:
>>  # rm ./perf.data*
>>  # /perf record -e intel_bts// --per-thread --snapshot -p 13588 &
>>  [1] 15396
>>  # kill -s SIGUSR2 15396
>>  # kill -s SIGUSR2 15396
>>  # fg
>>  /perf record -e intel_bts// --per-thread --snapshot -p 13588
>>  ^C[ perf record: Woken up 9 times to write data ]
>>  [ perf record: Captured and wrote 8.012 MB perf.data ]
>>  
>>  # ls -l ./perf.data*
>>  -rw------- 1 root root 8417552 Apr 15 06:20 ./perf.data
>>
>> (In v1, the second test trigger output switching incorrectly)
>>
>> Patch 1-2/6 in this patchset makes switch output trigger similar to
>> auxtrace snapshot trigger. Patch 1/6 introduces a 'trigger' class for
>> them. It would be better to merge patch 1/6 with commit c0bdc1c ("perf
>> record: Turns auxtrace_snapshot_enable into 3 states"). However, Ingo
>> has collected that commit, so make it a separated patch.
>>
>> Signed-off-by: Wang Nan <wangnan0@huawei.com>
>> Cc: Arnaldo Carvalho de Melo <acme@redhat.com>
>> Cc: Adrian Hunter <adrian.hunter@intel.com>
>> Cc: Jiri Olsa <jolsa@kernel.org>
>> Cc: Masami Hiramatsu <mhiramat@kernel.org>
>> Cc: Namhyung Kim <namhyung@kernel.org>
>> Cc: Zefan Li <lizefan@huawei.com>
>> Cc: pi3orama@163.com
>> Cc: He Kuang <hekuang@huawei.com>
>>
>> Wang Nan (6):
>>   perf tools: Extrace trigger class from auxtrace_snapshot
>>   perf record: Split output into multiple files via '--switch-output'
>>   perf record: Force enable --timestamp-filename when --switch-output is
>>     provided
>>   perf record: Disable buildid cache options by default in switch output
>>     mode
>>   perf record: Re-synthesize tracking events after output switching
>>   perf record: Generate tracking events for process forked by perf
>>
>>  tools/perf/builtin-record.c | 147 +++++++++++++++++++++++++++++++-------------
>>  tools/perf/util/util.h      |  60 ++++++++++++++++++
>>  2 files changed, 163 insertions(+), 44 deletions(-)
>>
>> -- 
>> 1.8.3.4
> 

Re: [PATCH v2 0/6] perf tools: Use SIGUSR2 control data dumpping

[Arnaldo Carvalho de Melo]

Em Fri, Apr 15, 2016 at 04:14:26PM +0300, Adrian Hunter escreveu:
> On 15/04/16 03:19, Arnaldo Carvalho de Melo wrote:
> > Em Thu, Apr 14, 2016 at 02:21:59PM +0000, Wang Nan escreveu:
> >> This patch set is based on perf/core.
> >>
> >> v1 -> v2: Fix a bug which triggers output switching without
> >>           '--switch-output' provided.
> >>
> >> Patch 5/10 in v1 has a bug: with following cmdline:
> >>  # perf record -e intel_bts// --per-thread --snapshot -p 13588
> > 
> > Ok, I moved the trigger stuff to a new file, util/trigger.h and added
> > missing man page entries, pushed to acme/perf/core, please take a look
> > if all is well before I push this to Ingo, tomorrow.
> > 
> > Adrian/others, if you can, please take a look.
> 
> Looks OK.  I gave intel_pt a quick test with snapshot mode which didn't seem
> to be broken.
> 
> Using --switch-output with intel_pt gives data that does not decode in the
> 2nd and subsequent files because there is no continuity of switch events.
> Also decoding the last file segfaults because the auxtrace index has not
> been split between the files.

> Probably need to prevent people using --switch-output with intel_pt, but
> I'll look at it more next week.

But the workflow of continuous tracing with regular analysis to go on
dumping uninteresting sections of the trace seems interesting? With the
massive volumes of data with Intel PT continuous tracing is deemded
impossible?

- Arnaldo
 
> > 
> > - Arnaldo
> >  
> >> Sending 'SIGUSR2' to perf triggers output switching, which us
> >> undesirable. The reason is in signal handler perf doesn't consider if
> >> user provided --switch-output explicitly.
> >>
> >> With this patchset applied:
> >>
> >>  # /perf record --switch-output -e intel_bts// --per-thread --snapshot -p 13588 &
> >>  [1] 15116
> >>  # kill -s SIGUSR2 15116
> >>  # [ perf record: dump data: Woken up 5 times ]
> >>  [ perf record: Dump perf.data.2016041506170000 ]
> >>  
> >>  # kill -s SIGUSR2 15116
> >>  # [ perf record: dump data: Woken up 3 times ]
> >>  [ perf record: Dump perf.data.2016041506170447 ]
> >>  
> >>  # fg
> >>  /perf record --switch-output -e intel_bts// --per-thread --snapshot -p 13588
> >>  ^C[ perf record: Woken up 3 times to write data ]
> >>  [ perf record: Dump perf.data.2016041506170785 ]
> >>  [ perf record: Captured and wrote 0.012 MB perf.data.<timestamp> ]
> >>  
> >>  # ls -l ./perf.data*
> >>  -rw------- 1 root root 4212888 Apr 15 06:17 ./perf.data.2016041506170000
> >>  -rw------- 1 root root 4212704 Apr 15 06:17 ./perf.data.2016041506170447
> >>  -rw------- 1 root root   18368 Apr 15 06:17 ./perf.data.2016041506170785
> >>
> >> The first 2 perf.data works fine. The final one has no samples.
> >>
> >> And:
> >>  # rm ./perf.data*
> >>  # /perf record -e intel_bts// --per-thread --snapshot -p 13588 &
> >>  [1] 15396
> >>  # kill -s SIGUSR2 15396
> >>  # kill -s SIGUSR2 15396
> >>  # fg
> >>  /perf record -e intel_bts// --per-thread --snapshot -p 13588
> >>  ^C[ perf record: Woken up 9 times to write data ]
> >>  [ perf record: Captured and wrote 8.012 MB perf.data ]
> >>  
> >>  # ls -l ./perf.data*
> >>  -rw------- 1 root root 8417552 Apr 15 06:20 ./perf.data
> >>
> >> (In v1, the second test trigger output switching incorrectly)
> >>
> >> Patch 1-2/6 in this patchset makes switch output trigger similar to
> >> auxtrace snapshot trigger. Patch 1/6 introduces a 'trigger' class for
> >> them. It would be better to merge patch 1/6 with commit c0bdc1c ("perf
> >> record: Turns auxtrace_snapshot_enable into 3 states"). However, Ingo
> >> has collected that commit, so make it a separated patch.
> >>
> >> Signed-off-by: Wang Nan <wangnan0@huawei.com>
> >> Cc: Arnaldo Carvalho de Melo <acme@redhat.com>
> >> Cc: Adrian Hunter <adrian.hunter@intel.com>
> >> Cc: Jiri Olsa <jolsa@kernel.org>
> >> Cc: Masami Hiramatsu <mhiramat@kernel.org>
> >> Cc: Namhyung Kim <namhyung@kernel.org>
> >> Cc: Zefan Li <lizefan@huawei.com>
> >> Cc: pi3orama@163.com
> >> Cc: He Kuang <hekuang@huawei.com>
> >>
> >> Wang Nan (6):
> >>   perf tools: Extrace trigger class from auxtrace_snapshot
> >>   perf record: Split output into multiple files via '--switch-output'
> >>   perf record: Force enable --timestamp-filename when --switch-output is
> >>     provided
> >>   perf record: Disable buildid cache options by default in switch output
> >>     mode
> >>   perf record: Re-synthesize tracking events after output switching
> >>   perf record: Generate tracking events for process forked by perf
> >>
> >>  tools/perf/builtin-record.c | 147 +++++++++++++++++++++++++++++++-------------
> >>  tools/perf/util/util.h      |  60 ++++++++++++++++++
> >>  2 files changed, 163 insertions(+), 44 deletions(-)
> >>
> >> -- 
> >> 1.8.3.4
> > 

Re: [PATCH v2 0/6] perf tools: Use SIGUSR2 control data dumpping

[Wangnan (F)]

On 2016/4/15 21:35, Arnaldo Carvalho de Melo wrote:
> Em Fri, Apr 15, 2016 at 04:14:26PM +0300, Adrian Hunter escreveu:
>> On 15/04/16 03:19, Arnaldo Carvalho de Melo wrote:
>>> Em Thu, Apr 14, 2016 at 02:21:59PM +0000, Wang Nan escreveu:
>>>> This patch set is based on perf/core.
>>>>
>>>> v1 -> v2: Fix a bug which triggers output switching without
>>>>            '--switch-output' provided.
>>>>
>>>> Patch 5/10 in v1 has a bug: with following cmdline:
>>>>   # perf record -e intel_bts// --per-thread --snapshot -p 13588
>>> Ok, I moved the trigger stuff to a new file, util/trigger.h and added
>>> missing man page entries, pushed to acme/perf/core, please take a look
>>> if all is well before I push this to Ingo, tomorrow.
>>>
>>> Adrian/others, if you can, please take a look.
>> Looks OK.  I gave intel_pt a quick test with snapshot mode which didn't seem
>> to be broken.
>>
>> Using --switch-output with intel_pt gives data that does not decode in the
>> 2nd and subsequent files because there is no continuity of switch events.
>> Also decoding the last file segfaults because the auxtrace index has not
>> been split between the files.
>> Probably need to prevent people using --switch-output with intel_pt, but
>> I'll look at it more next week.
> But the workflow of continuous tracing with regular analysis to go on
> dumping uninteresting sections of the trace seems interesting? With the
> massive volumes of data with Intel PT continuous tracing is deemded
> impossible?

Another question: is it possible to make auxtrace (intel_bts, intel_bt
and coresight) writing to overwritable ring buffer, so we can continuous
trace them and silently drop uninteresting data?

Thank you.

> - Arnaldo
>   
>>> - Arnaldo
>>>   
>>>> Sending 'SIGUSR2' to perf triggers output switching, which us
>>>> undesirable. The reason is in signal handler perf doesn't consider if
>>>> user provided --switch-output explicitly.
>>>>
>>>> With this patchset applied:
>>>>
>>>>   # /perf record --switch-output -e intel_bts// --per-thread --snapshot -p 13588 &
>>>>   [1] 15116
>>>>   # kill -s SIGUSR2 15116
>>>>   # [ perf record: dump data: Woken up 5 times ]
>>>>   [ perf record: Dump perf.data.2016041506170000 ]
>>>>   
>>>>   # kill -s SIGUSR2 15116
>>>>   # [ perf record: dump data: Woken up 3 times ]
>>>>   [ perf record: Dump perf.data.2016041506170447 ]
>>>>   
>>>>   # fg
>>>>   /perf record --switch-output -e intel_bts// --per-thread --snapshot -p 13588
>>>>   ^C[ perf record: Woken up 3 times to write data ]
>>>>   [ perf record: Dump perf.data.2016041506170785 ]
>>>>   [ perf record: Captured and wrote 0.012 MB perf.data.<timestamp> ]
>>>>   
>>>>   # ls -l ./perf.data*
>>>>   -rw------- 1 root root 4212888 Apr 15 06:17 ./perf.data.2016041506170000
>>>>   -rw------- 1 root root 4212704 Apr 15 06:17 ./perf.data.2016041506170447
>>>>   -rw------- 1 root root   18368 Apr 15 06:17 ./perf.data.2016041506170785
>>>>
>>>> The first 2 perf.data works fine. The final one has no samples.
>>>>
>>>> And:
>>>>   # rm ./perf.data*
>>>>   # /perf record -e intel_bts// --per-thread --snapshot -p 13588 &
>>>>   [1] 15396
>>>>   # kill -s SIGUSR2 15396
>>>>   # kill -s SIGUSR2 15396
>>>>   # fg
>>>>   /perf record -e intel_bts// --per-thread --snapshot -p 13588
>>>>   ^C[ perf record: Woken up 9 times to write data ]
>>>>   [ perf record: Captured and wrote 8.012 MB perf.data ]
>>>>   
>>>>   # ls -l ./perf.data*
>>>>   -rw------- 1 root root 8417552 Apr 15 06:20 ./perf.data
>>>>
>>>> (In v1, the second test trigger output switching incorrectly)
>>>>
>>>> Patch 1-2/6 in this patchset makes switch output trigger similar to
>>>> auxtrace snapshot trigger. Patch 1/6 introduces a 'trigger' class for
>>>> them. It would be better to merge patch 1/6 with commit c0bdc1c ("perf
>>>> record: Turns auxtrace_snapshot_enable into 3 states"). However, Ingo
>>>> has collected that commit, so make it a separated patch.
>>>>
>>>> Signed-off-by: Wang Nan <wangnan0@huawei.com>
>>>> Cc: Arnaldo Carvalho de Melo <acme@redhat.com>
>>>> Cc: Adrian Hunter <adrian.hunter@intel.com>
>>>> Cc: Jiri Olsa <jolsa@kernel.org>
>>>> Cc: Masami Hiramatsu <mhiramat@kernel.org>
>>>> Cc: Namhyung Kim <namhyung@kernel.org>
>>>> Cc: Zefan Li <lizefan@huawei.com>
>>>> Cc: pi3orama@163.com
>>>> Cc: He Kuang <hekuang@huawei.com>
>>>>
>>>> Wang Nan (6):
>>>>    perf tools: Extrace trigger class from auxtrace_snapshot
>>>>    perf record: Split output into multiple files via '--switch-output'
>>>>    perf record: Force enable --timestamp-filename when --switch-output is
>>>>      provided
>>>>    perf record: Disable buildid cache options by default in switch output
>>>>      mode
>>>>    perf record: Re-synthesize tracking events after output switching
>>>>    perf record: Generate tracking events for process forked by perf
>>>>
>>>>   tools/perf/builtin-record.c | 147 +++++++++++++++++++++++++++++++-------------
>>>>   tools/perf/util/util.h      |  60 ++++++++++++++++++
>>>>   2 files changed, 163 insertions(+), 44 deletions(-)
>>>>
>>>> -- 
>>>> 1.8.3.4