[PATCHv5 00/11] perf tools: Add perf data CTF conversion

[PATCHv5 00/11] perf tools: Add perf data CTF conversion

[Jiri Olsa]

hi,
this is follow up on original RFC patchset:
  http://marc.info/?t=140732735600004&r=1&w=2

Basically we are adding 'perf data convert' command to
allow conversion of perf data file into CTF [1] data.

v5 changes:
  - added struct field_format::alias instead of priv void*
  - rebased to latest acme's perf/core

v4 changes:
  - sorted out babeltrace new code regressions
  - rebased to latest acme's perf/core
  - includes fixes from Wang Nan, patches 9/10/11
  - patch 1/11 changelog updated with babeltrace
    compilation instructions [Arnaldo]

v3 changes:
  - rebased to latest acme's perf/core

v2 changes:
  - addressed comments from Namhyung
  - rebased to latest acme's perf/core

Changes from RFC:
  - able to generate CTF data, that are possible to be displayed under
    tracecompas GUI [3], please check several screenshots in here [4]
  - storing CTF data streams per cpu
  - several cleanups

Examples:
- Catch default perf data (cycles event):
  $ perf record ls
  [ perf record: Woken up 1 times to write data ]
  [ perf record: Captured and wrote 0.012 MB perf.data (~546 samples) ]

- To display converted CTF data run [2]:
  $ babeltrace ./ctf-data/
  [03:19:13.962125533] (+?.?????????) cycles: { }, { ip = 0xFFFFFFFF8105443A, tid = 20714, pid = 20714, period = 1 }
  [03:19:13.962130001] (+0.000004468) cycles: { }, { ip = 0xFFFFFFFF8105443A, tid = 20714, pid = 20714, period = 1 }
  [03:19:13.962131936] (+0.000001935) cycles: { }, { ip = 0xFFFFFFFF8105443A, tid = 20714, pid = 20714, period = 8 }
  [03:19:13.962133732] (+0.000001796) cycles: { }, { ip = 0xFFFFFFFF8105443A, tid = 20714, pid = 20714, period = 114 }
  [03:19:13.962135557] (+0.000001825) cycles: { }, { ip = 0xFFFFFFFF8105443A, tid = 20714, pid = 20714, period = 2087 }
  [03:19:13.962137627] (+0.000002070) cycles: { }, { ip = 0xFFFFFFFF81361938, tid = 20714, pid = 20714, period = 37582 }
  [03:19:13.962161091] (+0.000023464) cycles: { }, { ip = 0xFFFFFFFF8124218F, tid = 20714, pid = 20714, period = 600246 }
  [03:19:13.962517569] (+0.000356478) cycles: { }, { ip = 0xFFFFFFFF811A75DB, tid = 20714, pid = 20714, period = 1325731 }
  [03:19:13.969518008] (+0.007000439) cycles: { }, { ip = 0x34080917B2, tid = 20714, pid = 20714, period = 1144298 }

- To get some nice output in tracecompas GUI [3], please capture sched:*
  and syscall tracepoints like:
  # perf record -e 'sched:*,raw_syscalls:*' -a
  ^C[ perf record: Woken up 0 times to write data ]
  [ perf record: Captured and wrote 412.347 MB perf.data (~18015721 samples) ]

- To convert perf data file run:
  # perf data convert --to-ctf=./ctf 
  [ perf data convert: Converted 'perf.data' into CTF data './ctf' ]
  [ perf data convert: Converted and wrote 408.421 MB (3964792 samples) ]

- To display converted CTF data run [2]:
  # babeltrace ./ctf-data/
  [23:32:20.165354855] (+0.000000507) sched:sched_wakeup: { cpu_id = 0 }, { perf_ip = 0xFFFFFFFF810BCA72, perf_tid = 0, perf_pid = 0, perf_id = 462554, perf_period = 1, common_type = 265, ...
  [23:32:20.165359078] (+0.000001181) sched:sched_switch: { cpu_id = 0 }, { perf_ip = 0xFFFFFFFF8172A110, perf_tid = 0, perf_pid = 0, perf_id = 462562, perf_period = 1, common_type = 263, ...
  [23:32:20.165364686] (+0.000000328) sched:sched_stat_runtime: { cpu_id = 0 }, { perf_ip = 0xFFFFFFFF810C8AE5, perf_tid = 5326, perf_pid = 5326, perf_id = 462610, perf_period = 11380, ...
  [23:32:20.165366067] (+0.000001205) sched:sched_switch: { cpu_id = 0 }, { perf_ip = 0xFFFFFFFF8172A110, perf_tid = 5326, perf_pid = 5326, perf_id = 462562, perf_period = 1, common_type  ...
  [23:32:20.165723312] (+0.000001479) sched:sched_stat_runtime: { cpu_id = 2 }, { perf_ip = 0xFFFFFFFF810C8AE5, perf_tid = 11821, perf_pid = 11821, perf_id = 462612, perf_period = 1000265, ...
  [23:32:20.065282391] (+?.?????????) raw_syscalls:sys_enter: { cpu_id = 1 }, { perf_ip = 0xFFFFFFFF810230AF, perf_tid = 26155, perf_pid = 26155, perf_id = 462635, perf_period = 1, ...
  [23:32:20.065286422] (+0.000004031) raw_syscalls:sys_exit: { cpu_id = 1 }, { perf_ip = 0xFFFFFFFF810231D8, perf_tid = 26155, perf_pid = 26155, perf_id = 462639, perf_period = 1, ...

- Or run tracecompass and open the CTF data ;-)

Changes are also reachable in here:
  git://git.kernel.org/pub/scm/linux/kernel/git/jolsa/perf.git
  perf/core_ctf_convert

thanks,
jirka

[1] Common Trace Format - http://www.efficios.com/ctf
[2] babeltrace - http://www.efficios.com/babeltrace
[3] Trace compass - http://projects.eclipse.org/projects/tools.tracecompass
[4] screenshots - http://people.redhat.com/~jolsa/tracecompass-perf/

Cc: Arnaldo Carvalho de Melo <acme@redhat.com>
Cc: David Ahern <dsahern@gmail.com>
Cc: Dominique Toupin <dominique.toupin@ericsson.com>
Cc: Frederic Weisbecker <fweisbec@gmail.com>
Cc: Jeremie Galarneau <jgalar@efficios.com>
Cc: Jiri Olsa <jolsa@kernel.org>
Cc: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Cc: Namhyung Kim <namhyung@gmail.com>
Cc: Paul Mackerras <paulus@samba.org>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Sebastian Andrzej Siewior <bigeasy@linutronix.de>
Cc: Tom Zanussi <tzanussi@gmail.com>
Cc: Wang Nan <wangnan0@huawei.com>
---
Jiri Olsa (6):
      perf tools: Add feature check for libbabeltrace
      perf tools: Add new perf data command
      perf data: Add perf data to CTF conversion support
      perf data: Enable stream flush within processing
      perf data: Add support for setting ordered_events queue size
      tools lib traceevent: Add alias field to struct format_field

Sebastian Andrzej Siewior (3):
      perf data: Add a 'perf' prefix to the generic fields
      perf data: Add tracepoint events fields CTF conversion support
      perf data: Switch to multiple cpu stream files

Wang Nan (2):
      perf data: Fix duplicate field names and avoid reserved keywords
      perf data: Fix signess of value

 tools/lib/traceevent/event-parse.c                 |    1 +
 tools/lib/traceevent/event-parse.h                 |    1 +
 tools/perf/Build                                   |    1 +
 tools/perf/Documentation/perf-data.txt             |   40 +
 tools/perf/Documentation/perf.txt                  |    7 +-
 tools/perf/Makefile.perf                           |    4 +-
 tools/perf/builtin-data.c                          |  119 ++
 tools/perf/builtin.h                               |    1 +
 tools/perf/command-list.txt                        |    1 +
 tools/perf/config/Makefile                         |   25 +
 tools/perf/config/feature-checks/Makefile          |    8 +-
 tools/perf/config/feature-checks/test-all.c        |    5 +
 .../config/feature-checks/test-libbabeltrace.c     |    8 +
 tools/perf/perf.c                                  |    1 +
 tools/perf/util/Build                              |    2 +
 tools/perf/util/data-convert-bt.c                  | 1174 ++++++++++++++++++++
 tools/perf/util/data-convert-bt.h                  |    8 +
 tools/perf/util/debug.c                            |    2 +
 tools/perf/util/debug.h                            |    1 +
 19 files changed, 1405 insertions(+), 4 deletions(-)
 create mode 100644 tools/perf/Documentation/perf-data.txt
 create mode 100644 tools/perf/builtin-data.c
 create mode 100644 tools/perf/config/feature-checks/test-libbabeltrace.c
 create mode 100644 tools/perf/util/data-convert-bt.c
 create mode 100644 tools/perf/util/data-convert-bt.h

[PATCH 01/11] perf tools: Add feature check for libbabeltrace

[Jiri Olsa]

Adding feature check for babeltrace library [1], which will be
used for perf data file CTF [2] conversion in following patches.

The babeltrace library is now automatically detected as standard
feature. It's possible to specify LIBBABELTRACE_DIR make variable
to specify location of installed libbabeltrace, like:

  $ make LIBBABELTRACE_DIR=/opt/libbabeltrace/
    BUILD:   Doing 'make -j4' parallel build

  Auto-detecting system features:
  ...                         dwarf: [ on  ]
  ...                         glibc: [ on  ]
  ...                          gtk2: [ on  ]
  ...                      libaudit: [ on  ]
  ...                        libbfd: [ on  ]
  ...                        libelf: [ on  ]
  ...                       libnuma: [ on  ]
  ...                       libperl: [ on  ]
  ...                     libpython: [ on  ]
  ...                      libslang: [ on  ]
  ...                     libunwind: [ on  ]
  ...                 libbabeltrace: [ on  ]
  ...            libdw-dwarf-unwind: [ on  ]
  ...                          zlib: [ on  ]
  ...     DWARF post unwind library: libunwind

NOTE The installation of the [1] to to used by above make:
     $ ./configure --prefix=/opt/libbabeltrace
     $ make prefix=/opt/libbabeltrace
     $ sudo make install prefix=/opt/libbabeltrace

[1] babeltrace - http://www.efficios.com/babeltrace
[2] Common Trace Format - http://www.efficios.com/ctf

Cc: Arnaldo Carvalho de Melo <acme@redhat.com>
Cc: David Ahern <dsahern@gmail.com>
Cc: Frederic Weisbecker <fweisbec@gmail.com>
Cc: Jeremie Galarneau <jgalar@efficios.com>
Cc: Jiri Olsa <jolsa@kernel.org>
Cc: Namhyung Kim <namhyung@gmail.com>
Cc: Paul Mackerras <paulus@samba.org>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Sebastian Andrzej Siewior <bigeasy@linutronix.de>
Cc: Tom Zanussi <tzanussi@gmail.com>
Cc: Wang Nan <wangnan0@huawei.com>
Signed-off-by: Sebastian Andrzej Siewior <bigeasy@linutronix.de>
Signed-off-by: Jiri Olsa <jolsa@kernel.org>
---
 tools/perf/Makefile.perf                           |  4 +++-
 tools/perf/config/Makefile                         | 24 ++++++++++++++++++++++
 tools/perf/config/feature-checks/Makefile          |  8 ++++++--
 tools/perf/config/feature-checks/test-all.c        |  5 +++++
 .../config/feature-checks/test-libbabeltrace.c     |  8 ++++++++
 5 files changed, 46 insertions(+), 3 deletions(-)
 create mode 100644 tools/perf/config/feature-checks/test-libbabeltrace.c

diff --git a/tools/perf/Makefile.perf b/tools/perf/Makefile.perf
index efc5158738f4..ec4c063ed9f3 100644
--- a/tools/perf/Makefile.perf
+++ b/tools/perf/Makefile.perf
@@ -68,7 +68,9 @@ include config/utilities.mak
 # for reading the x32 mode 32-bit compatibility VDSO in 64-bit mode
 #
 # Define NO_ZLIB if you do not want to support compressed kernel modules
-
+#
+# Define NO_LIBBABELTRACE if you do not want libbabeltrace support
+# for CTF data format.
 
 ifeq ($(srctree),)
 srctree := $(patsubst %/,%,$(dir $(shell pwd)))
diff --git a/tools/perf/config/Makefile b/tools/perf/config/Makefile
index b97a7b903a23..6f129b0a9c64 100644
--- a/tools/perf/config/Makefile
+++ b/tools/perf/config/Makefile
@@ -96,6 +96,17 @@ ifndef NO_LIBELF
   FEATURE_CHECK_LDFLAGS-libdw-dwarf-unwind := $(LIBDW_LDFLAGS) -ldw
 endif
 
+ifndef NO_LIBBABELTRACE
+  # for linking with debug library, run like:
+  # make DEBUG=1 LIBBABELTRACE_DIR=/opt/libbabeltrace/
+  ifdef LIBBABELTRACE_DIR
+    LIBBABELTRACE_CFLAGS  := -I$(LIBBABELTRACE_DIR)/include
+    LIBBABELTRACE_LDFLAGS := -L$(LIBBABELTRACE_DIR)/lib
+  endif
+  FEATURE_CHECK_CFLAGS-libbabeltrace := $(LIBBABELTRACE_CFLAGS)
+  FEATURE_CHECK_LDFLAGS-libbabeltrace := $(LIBBABELTRACE_LDFLAGS) -lbabeltrace-ctf
+endif
+
 # include ARCH specific config
 -include $(src-perf)/arch/$(ARCH)/Makefile
 
@@ -216,6 +227,7 @@ CORE_FEATURE_TESTS =			\
 	stackprotector-all		\
 	timerfd				\
 	libdw-dwarf-unwind		\
+	libbabeltrace			\
 	zlib
 
 LIB_FEATURE_TESTS =			\
@@ -231,6 +243,7 @@ LIB_FEATURE_TESTS =			\
 	libslang			\
 	libunwind			\
 	libdw-dwarf-unwind		\
+	libbabeltrace			\
 	zlib
 
 VF_FEATURE_TESTS =			\
@@ -692,6 +705,17 @@ else
   NO_PERF_READ_VDSOX32 := 1
 endif
 
+ifndef NO_LIBBABELTRACE
+  ifeq ($(feature-libbabeltrace), 0)
+    msg := $(warning No libbabeltrace found, disables 'perf data' CTF format support, please install libbabeltrace-devel/libbabeltrace-ctf-dev);
+    NO_LIBBABELTRACE := 1
+  else
+    CFLAGS += -DHAVE_LIBBABELTRACE_SUPPORT $(LIBBABELTRACE_CFLAGS)
+    LDFLAGS += $(LIBBABELTRACE_LDFLAGS)
+    EXTLIBS += -lbabeltrace-ctf
+  endif
+endif
+
 # Among the variables below, these:
 #   perfexecdir
 #   template_dir
diff --git a/tools/perf/config/feature-checks/Makefile b/tools/perf/config/feature-checks/Makefile
index b32ff3372514..70c9aebe9da3 100644
--- a/tools/perf/config/feature-checks/Makefile
+++ b/tools/perf/config/feature-checks/Makefile
@@ -29,6 +29,7 @@ FILES=					\
 	test-stackprotector-all.bin	\
 	test-timerfd.bin		\
 	test-libdw-dwarf-unwind.bin	\
+	test-libbabeltrace.bin		\
 	test-compile-32.bin		\
 	test-compile-x32.bin		\
 	test-zlib.bin
@@ -43,7 +44,7 @@ BUILD = $(CC) $(CFLAGS) -o $(OUTPUT)$@ $(patsubst %.bin,%.c,$@) $(LDFLAGS)
 ###############################
 
 test-all.bin:
-	$(BUILD) -Werror -fstack-protector-all -O2 -Werror -D_FORTIFY_SOURCE=2 -ldw -lelf -lnuma -lelf -laudit -I/usr/include/slang -lslang $(shell $(PKG_CONFIG) --libs --cflags gtk+-2.0 2>/dev/null) $(FLAGS_PERL_EMBED) $(FLAGS_PYTHON_EMBED) -DPACKAGE='"perf"' -lbfd -ldl -lz
+	$(BUILD) -Werror -fstack-protector-all -O2 -Werror -D_FORTIFY_SOURCE=2 -ldw -lelf -lnuma -lelf -laudit -I/usr/include/slang -lslang $(shell $(PKG_CONFIG) --libs --cflags gtk+-2.0 2>/dev/null) $(FLAGS_PERL_EMBED) $(FLAGS_PYTHON_EMBED) -DPACKAGE='"perf"' -lbfd -ldl -lz -lbabeltrace
 
 test-hello.bin:
 	$(BUILD)
@@ -133,7 +134,10 @@ test-timerfd.bin:
 	$(BUILD)
 
 test-libdw-dwarf-unwind.bin:
-	$(BUILD)
+	$(BUILD) # -ldw provided by $(FEATURE_CHECK_LDFLAGS-libdw-dwarf-unwind)
+
+test-libbabeltrace.bin:
+	$(BUILD) # -lbabeltrace provided by $(FEATURE_CHECK_LDFLAGS-libbabeltrace)
 
 test-sync-compare-and-swap.bin:
 	$(BUILD) -Werror
diff --git a/tools/perf/config/feature-checks/test-all.c b/tools/perf/config/feature-checks/test-all.c
index 6d4d09323922..1ffc3da5ca10 100644
--- a/tools/perf/config/feature-checks/test-all.c
+++ b/tools/perf/config/feature-checks/test-all.c
@@ -101,6 +101,10 @@
 # include "test-pthread_attr_setaffinity_np.c"
 #undef main
 
+#define main main_test_libbabeltrace
+# include "test-libbabeltrace.c"
+#undef main
+
 int main(int argc, char *argv[])
 {
 	main_test_libpython();
@@ -126,6 +130,7 @@ int main(int argc, char *argv[])
 	main_test_sync_compare_and_swap(argc, argv);
 	main_test_zlib();
 	main_test_pthread_attr_setaffinity_np();
+	main_test_libbabeltrace();
 
 	return 0;
 }
diff --git a/tools/perf/config/feature-checks/test-libbabeltrace.c b/tools/perf/config/feature-checks/test-libbabeltrace.c
new file mode 100644
index 000000000000..3b7dd68a4d52
--- /dev/null
+++ b/tools/perf/config/feature-checks/test-libbabeltrace.c
@@ -0,0 +1,8 @@
+
+#include <babeltrace/ctf-writer/writer.h>
+
+int main(void)
+{
+	bt_ctf_stream_class_get_packet_context_type((void *) 0);
+	return 0;
+}
-- 
1.9.3

[PATCH 02/11] perf tools: Add new perf data command

[Jiri Olsa]

Adding new 'perf data' command to provide operations over
data files.

The 'perf data convert' sub command is coming in following
patch, but there's possibility for other useful commands
like 'perf data ls' (to display perf data file in directory
in ls style).

Cc: Arnaldo Carvalho de Melo <acme@redhat.com>
Cc: David Ahern <dsahern@gmail.com>
Cc: Frederic Weisbecker <fweisbec@gmail.com>
Cc: Jeremie Galarneau <jgalar@efficios.com>
Cc: Jiri Olsa <jolsa@kernel.org>
Cc: Namhyung Kim <namhyung@gmail.com>
Cc: Paul Mackerras <paulus@samba.org>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Sebastian Andrzej Siewior <bigeasy@linutronix.de>
Cc: Tom Zanussi <tzanussi@gmail.com>
Cc: Wang Nan <wangnan0@huawei.com>
Signed-off-by: Sebastian Andrzej Siewior <bigeasy@linutronix.de>
Signed-off-by: Jiri Olsa <jolsa@kernel.org>
---
 tools/perf/Build                       |  1 +
 tools/perf/Documentation/perf-data.txt | 15 +++++++
 tools/perf/builtin-data.c              | 75 ++++++++++++++++++++++++++++++++++
 tools/perf/builtin.h                   |  1 +
 tools/perf/command-list.txt            |  1 +
 tools/perf/perf.c                      |  1 +
 6 files changed, 94 insertions(+)
 create mode 100644 tools/perf/Documentation/perf-data.txt
 create mode 100644 tools/perf/builtin-data.c

diff --git a/tools/perf/Build b/tools/perf/Build
index 976e03849f6d..b77370ef7005 100644
--- a/tools/perf/Build
+++ b/tools/perf/Build
@@ -18,6 +18,7 @@ perf-y += builtin-lock.o
 perf-y += builtin-kvm.o
 perf-y += builtin-inject.o
 perf-y += builtin-mem.o
+perf-y += builtin-data.o
 
 perf-$(CONFIG_AUDIT) += builtin-trace.o
 perf-$(CONFIG_LIBELF) += builtin-probe.o
diff --git a/tools/perf/Documentation/perf-data.txt b/tools/perf/Documentation/perf-data.txt
new file mode 100644
index 000000000000..b8c83947715c
--- /dev/null
+++ b/tools/perf/Documentation/perf-data.txt
@@ -0,0 +1,15 @@
+perf-data(1)
+==============
+
+NAME
+----
+perf-data - Data file related processing
+
+SYNOPSIS
+--------
+[verse]
+'perf data' [<common options>] <command> [<options>]",
+
+DESCRIPTION
+-----------
+Data file related processing.
diff --git a/tools/perf/builtin-data.c b/tools/perf/builtin-data.c
new file mode 100644
index 000000000000..1eee97d020fa
--- /dev/null
+++ b/tools/perf/builtin-data.c
@@ -0,0 +1,75 @@
+#include <linux/compiler.h>
+#include "builtin.h"
+#include "perf.h"
+#include "debug.h"
+#include "parse-options.h"
+
+typedef int (*data_cmd_fn_t)(int argc, const char **argv, const char *prefix);
+
+struct data_cmd {
+	const char	*name;
+	const char	*summary;
+	data_cmd_fn_t	fn;
+};
+
+static struct data_cmd data_cmds[];
+
+#define for_each_cmd(cmd) \
+	for (cmd = data_cmds; cmd && cmd->name; cmd++)
+
+static const struct option data_options[] = {
+	OPT_END()
+};
+
+static const char * const data_usage[] = {
+	"perf data [<common options>] <command> [<options>]",
+	NULL
+};
+
+static void print_usage(void)
+{
+	struct data_cmd *cmd;
+
+	printf("Usage:\n");
+	printf("\t%s\n\n", data_usage[0]);
+	printf("\tAvailable commands:\n");
+
+	for_each_cmd(cmd) {
+		printf("\t %s\t- %s\n", cmd->name, cmd->summary);
+	}
+
+	printf("\n");
+}
+
+static struct data_cmd data_cmds[] = {
+	{ NULL },
+};
+
+int cmd_data(int argc, const char **argv, const char *prefix)
+{
+	struct data_cmd *cmd;
+	const char *cmdstr;
+
+	/* No command specified. */
+	if (argc < 2)
+		goto usage;
+
+	argc = parse_options(argc, argv, data_options, data_usage,
+			     PARSE_OPT_STOP_AT_NON_OPTION);
+	if (argc < 1)
+		goto usage;
+
+	cmdstr = argv[0];
+
+	for_each_cmd(cmd) {
+		if (strcmp(cmd->name, cmdstr))
+			continue;
+
+		return cmd->fn(argc, argv, prefix);
+	}
+
+	pr_err("Unknown command: %s\n", cmdstr);
+usage:
+	print_usage();
+	return -1;
+}
diff --git a/tools/perf/builtin.h b/tools/perf/builtin.h
index b210d62907e4..3688ad29085f 100644
--- a/tools/perf/builtin.h
+++ b/tools/perf/builtin.h
@@ -37,6 +37,7 @@ extern int cmd_test(int argc, const char **argv, const char *prefix);
 extern int cmd_trace(int argc, const char **argv, const char *prefix);
 extern int cmd_inject(int argc, const char **argv, const char *prefix);
 extern int cmd_mem(int argc, const char **argv, const char *prefix);
+extern int cmd_data(int argc, const char **argv, const char *prefix);
 
 extern int find_scripts(char **scripts_array, char **scripts_path_array);
 #endif
diff --git a/tools/perf/command-list.txt b/tools/perf/command-list.txt
index 0906fc401c52..00fcaf8a5b8d 100644
--- a/tools/perf/command-list.txt
+++ b/tools/perf/command-list.txt
@@ -7,6 +7,7 @@ perf-archive			mainporcelain common
 perf-bench			mainporcelain common
 perf-buildid-cache		mainporcelain common
 perf-buildid-list		mainporcelain common
+perf-data			mainporcelain common
 perf-diff			mainporcelain common
 perf-evlist			mainporcelain common
 perf-inject			mainporcelain common
diff --git a/tools/perf/perf.c b/tools/perf/perf.c
index 3700a7faca6c..f3c66b81c6be 100644
--- a/tools/perf/perf.c
+++ b/tools/perf/perf.c
@@ -62,6 +62,7 @@ static struct cmd_struct commands[] = {
 #endif
 	{ "inject",	cmd_inject,	0 },
 	{ "mem",	cmd_mem,	0 },
+	{ "data",	cmd_data,	0 },
 };
 
 struct pager_config {
-- 
1.9.3

[PATCH 03/11] perf data: Add perf data to CTF conversion support

[Jiri Olsa]

Adding 'perf data convert' to convert perf data file into
different format. This patch adds support for CTF format
conversion.

To convert perf.data into CTF run:
  $ perf data convert --to-ctf=./ctf-data/
  [ perf data convert: Converted 'perf.data' into CTF data './ctf-data/' ]
  [ perf data convert: Converted and wrote 11.268 MB (100230 samples) ]

The command will create CTF metadata out of perf.data file
(or one specified via -i option) and then convert all sample
events into single CTF stream.

Each sample_type bit is translated into separated CTF event
field apart from following exceptions:

  PERF_SAMPLE_RAW          - added in next patch
  PERF_SAMPLE_READ         - TODO
  PERF_SAMPLE_CALLCHAIN    - TODO
  PERF_SAMPLE_BRANCH_STACK - TODO
  PERF_SAMPLE_REGS_USER    - TODO
  PERF_SAMPLE_STACK_USER   - TODO

  $ perf --debug=data-convert=2 data convert ...

The converted CTF data could be analyzed by CTF tools, like
babletrace or tracecompass [1].

  $ babeltrace ./ctf-data/
  [03:19:13.962125533] (+?.?????????) cycles: { }, { ip = 0xFFFFFFFF8105443A, tid = 20714, pid = 20714, period = 1 }
  [03:19:13.962130001] (+0.000004468) cycles: { }, { ip = 0xFFFFFFFF8105443A, tid = 20714, pid = 20714, period = 1 }
  [03:19:13.962131936] (+0.000001935) cycles: { }, { ip = 0xFFFFFFFF8105443A, tid = 20714, pid = 20714, period = 8 }
  [03:19:13.962133732] (+0.000001796) cycles: { }, { ip = 0xFFFFFFFF8105443A, tid = 20714, pid = 20714, period = 114 }
  [03:19:13.962135557] (+0.000001825) cycles: { }, { ip = 0xFFFFFFFF8105443A, tid = 20714, pid = 20714, period = 2087 }
  [03:19:13.962137627] (+0.000002070) cycles: { }, { ip = 0xFFFFFFFF81361938, tid = 20714, pid = 20714, period = 37582 }
  [03:19:13.962161091] (+0.000023464) cycles: { }, { ip = 0xFFFFFFFF8124218F, tid = 20714, pid = 20714, period = 600246 }
  [03:19:13.962517569] (+0.000356478) cycles: { }, { ip = 0xFFFFFFFF811A75DB, tid = 20714, pid = 20714, period = 1325731 }
  [03:19:13.969518008] (+0.007000439) cycles: { }, { ip = 0x34080917B2, tid = 20714, pid = 20714, period = 1144298 }

The following members to the ctf-environment were decided
to be added to distinguish and specify perf CTF data:
  - domain
    It says "kernel" because it contains a kernel trace (not to be
    confused with a user space like lttng-ust does)
  - tracer_name
    It says perf. This can be used to distinguish between lttng and
    perf CTF based trace.
  - version
    The kernel version from stream. In addition to release, this is
    what it looks like on a Debian kernel:
      release = "3.14-1-amd64";
      version = "3.14.0";

[1] http://projects.eclipse.org/projects/tools.tracecompass

Cc: Arnaldo Carvalho de Melo <acme@redhat.com>
Cc: David Ahern <dsahern@gmail.com>
Cc: Frederic Weisbecker <fweisbec@gmail.com>
Cc: Jeremie Galarneau <jgalar@efficios.com>
Cc: Jiri Olsa <jolsa@kernel.org>
Cc: Namhyung Kim <namhyung@gmail.com>
Cc: Paul Mackerras <paulus@samba.org>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Sebastian Andrzej Siewior <bigeasy@linutronix.de>
Cc: Tom Zanussi <tzanussi@gmail.com>
Cc: Wang Nan <wangnan0@huawei.com>
Signed-off-by: Sebastian Andrzej Siewior <bigeasy@linutronix.de>
Signed-off-by: Jiri Olsa <jolsa@kernel.org>
---
 tools/perf/Documentation/perf-data.txt |  25 ++
 tools/perf/Documentation/perf.txt      |   7 +-
 tools/perf/builtin-data.c              |  44 +++
 tools/perf/config/Makefile             |   1 +
 tools/perf/util/Build                  |   2 +
 tools/perf/util/data-convert-bt.c      | 612 +++++++++++++++++++++++++++++++++
 tools/perf/util/data-convert-bt.h      |   8 +
 tools/perf/util/debug.c                |   2 +
 tools/perf/util/debug.h                |   1 +
 9 files changed, 701 insertions(+), 1 deletion(-)
 create mode 100644 tools/perf/util/data-convert-bt.c
 create mode 100644 tools/perf/util/data-convert-bt.h

diff --git a/tools/perf/Documentation/perf-data.txt b/tools/perf/Documentation/perf-data.txt
index b8c83947715c..be8fa1a0a97e 100644
--- a/tools/perf/Documentation/perf-data.txt
+++ b/tools/perf/Documentation/perf-data.txt
@@ -13,3 +13,28 @@ SYNOPSIS
 DESCRIPTION
 -----------
 Data file related processing.
+
+COMMANDS
+--------
+convert::
+	Converts perf data file into another format (only CTF [1] format is support by now).
+	It's possible to set data-convert debug variable to get debug messages from conversion,
+	like:
+	  perf --debug data-convert data convert ...
+
+OPTIONS for 'convert'
+---------------------
+--to-ctf::
+	Triggers the CTF conversion, specify the path of CTF data directory.
+
+-i::
+	Specify input perf data file path.
+
+-v::
+--verbose::
+        Be more verbose (show counter open errors, etc).
+
+SEE ALSO
+--------
+linkperf:perf[1]
+[1] Common Trace Format - http://www.efficios.com/ctf
diff --git a/tools/perf/Documentation/perf.txt b/tools/perf/Documentation/perf.txt
index 1e8e400b4493..2b131776363e 100644
--- a/tools/perf/Documentation/perf.txt
+++ b/tools/perf/Documentation/perf.txt
@@ -13,11 +13,16 @@ SYNOPSIS
 OPTIONS
 -------
 --debug::
-	Setup debug variable (just verbose for now) in value
+	Setup debug variable (see list below) in value
 	range (0, 10). Use like:
 	  --debug verbose   # sets verbose = 1
 	  --debug verbose=2 # sets verbose = 2
 
+	List of debug variables allowed to set:
+	  verbose          - general debug messages
+	  ordered-events   - ordered events object debug messages
+	  data-convert     - data convert command debug messages
+
 --buildid-dir::
 	Setup buildid cache directory. It has higher priority than
 	buildid.dir config file option.
diff --git a/tools/perf/builtin-data.c b/tools/perf/builtin-data.c
index 1eee97d020fa..9705ba7e4c16 100644
--- a/tools/perf/builtin-data.c
+++ b/tools/perf/builtin-data.c
@@ -3,6 +3,7 @@
 #include "perf.h"
 #include "debug.h"
 #include "parse-options.h"
+#include "data-convert-bt.h"
 
 typedef int (*data_cmd_fn_t)(int argc, const char **argv, const char *prefix);
 
@@ -41,7 +42,50 @@ static void print_usage(void)
 	printf("\n");
 }
 
+static const char * const data_convert_usage[] = {
+	"perf data convert [<options>]",
+	NULL
+};
+
+static int cmd_data_convert(int argc, const char **argv,
+			    const char *prefix __maybe_unused)
+{
+	const char *to_ctf     = NULL;
+	const struct option options[] = {
+		OPT_INCR('v', "verbose", &verbose, "be more verbose"),
+		OPT_STRING('i', "input", &input_name, "file", "input file name"),
+#ifdef HAVE_LIBBABELTRACE_SUPPORT
+		OPT_STRING(0, "to-ctf", &to_ctf, NULL, "Convert to CTF format"),
+#endif
+		OPT_END()
+	};
+
+#ifndef HAVE_LIBBABELTRACE_SUPPORT
+	pr_err("No conversion support compiled in.\n");
+	return -1;
+#endif
+
+	argc = parse_options(argc, argv, options,
+			     data_convert_usage, 0);
+	if (argc) {
+		usage_with_options(data_convert_usage, options);
+		return -1;
+	}
+
+	if (to_ctf) {
+#ifdef HAVE_LIBBABELTRACE_SUPPORT
+		return bt_convert__perf2ctf(input_name, to_ctf);
+#else
+		pr_err("The libbabeltrace support is not compiled in.\n");
+		return -1;
+#endif
+	}
+
+	return 0;
+}
+
 static struct data_cmd data_cmds[] = {
+	{ "convert", "converts data file between formats", cmd_data_convert },
 	{ NULL },
 };
 
diff --git a/tools/perf/config/Makefile b/tools/perf/config/Makefile
index 6f129b0a9c64..c3570b5f3bf3 100644
--- a/tools/perf/config/Makefile
+++ b/tools/perf/config/Makefile
@@ -713,6 +713,7 @@ ifndef NO_LIBBABELTRACE
     CFLAGS += -DHAVE_LIBBABELTRACE_SUPPORT $(LIBBABELTRACE_CFLAGS)
     LDFLAGS += $(LIBBABELTRACE_LDFLAGS)
     EXTLIBS += -lbabeltrace-ctf
+    $(call detected,CONFIG_LIBBABELTRACE)
   endif
 endif
 
diff --git a/tools/perf/util/Build b/tools/perf/util/Build
index 32f9327b1a97..a2c8047d25f7 100644
--- a/tools/perf/util/Build
+++ b/tools/perf/util/Build
@@ -88,6 +88,8 @@ libperf-$(CONFIG_DWARF) += dwarf-aux.o
 libperf-$(CONFIG_LIBDW_DWARF_UNWIND) += unwind-libdw.o
 libperf-$(CONFIG_LIBUNWIND)          += unwind-libunwind.o
 
+libperf-$(CONFIG_LIBBABELTRACE) += data-convert-bt.o
+
 libperf-y += scripting-engines/
 
 libperf-$(CONFIG_PERF_REGS) += perf_regs.o
diff --git a/tools/perf/util/data-convert-bt.c b/tools/perf/util/data-convert-bt.c
new file mode 100644
index 000000000000..ff4826c1745f
--- /dev/null
+++ b/tools/perf/util/data-convert-bt.c
@@ -0,0 +1,612 @@
+/*
+ * CTF writing support via babeltrace.
+ *
+ * Copyright (C) 2014, Jiri Olsa <jolsa@redhat.com>
+ * Copyright (C) 2014, Sebastian Andrzej Siewior <bigeasy@linutronix.de>
+ *
+ * Released under the GPL v2. (and only v2, not any later version)
+ */
+
+#include <linux/compiler.h>
+#include <babeltrace/ctf-writer/writer.h>
+#include <babeltrace/ctf-writer/clock.h>
+#include <babeltrace/ctf-writer/stream.h>
+#include <babeltrace/ctf-writer/event.h>
+#include <babeltrace/ctf-writer/event-types.h>
+#include <babeltrace/ctf-writer/event-fields.h>
+#include <babeltrace/ctf/events.h>
+#include <traceevent/event-parse.h>
+#include "asm/bug.h"
+#include "data-convert-bt.h"
+#include "session.h"
+#include "util.h"
+#include "debug.h"
+#include "tool.h"
+#include "evlist.h"
+#include "evsel.h"
+#include "machine.h"
+
+#define pr_N(n, fmt, ...) \
+	eprintf(n, debug_data_convert, fmt, ##__VA_ARGS__)
+
+#define pr(fmt, ...)  pr_N(1, pr_fmt(fmt), ##__VA_ARGS__)
+#define pr2(fmt, ...) pr_N(2, pr_fmt(fmt), ##__VA_ARGS__)
+
+#define pr_time2(t, fmt, ...) pr_time_N(2, debug_data_convert, t, pr_fmt(fmt), ##__VA_ARGS__)
+
+struct evsel_priv {
+	struct bt_ctf_event_class *event_class;
+};
+
+struct ctf_writer {
+	/* writer primitives */
+	struct bt_ctf_writer		*writer;
+	struct bt_ctf_stream		*stream;
+	struct bt_ctf_stream_class	*stream_class;
+	struct bt_ctf_clock		*clock;
+
+	/* data types */
+	union {
+		struct {
+			struct bt_ctf_field_type	*s64;
+			struct bt_ctf_field_type	*u64;
+			struct bt_ctf_field_type	*s32;
+			struct bt_ctf_field_type	*u32;
+			struct bt_ctf_field_type	*string;
+			struct bt_ctf_field_type	*u64_hex;
+		};
+		struct bt_ctf_field_type *array[6];
+	} data;
+};
+
+struct convert {
+	struct perf_tool	tool;
+	struct ctf_writer	writer;
+
+	u64			events_size;
+	u64			events_count;
+};
+
+static int value_set(struct bt_ctf_field_type *type,
+		     struct bt_ctf_event *event,
+		     const char *name, u64 val)
+{
+	struct bt_ctf_field *field;
+	bool sign = bt_ctf_field_type_integer_get_signed(type);
+	int ret;
+
+	field = bt_ctf_field_create(type);
+	if (!field) {
+		pr_err("failed to create a field %s\n", name);
+		return -1;
+	}
+
+	if (sign) {
+		ret = bt_ctf_field_signed_integer_set_value(field, val);
+		if (ret) {
+			pr_err("failed to set field value %s\n", name);
+			goto err;
+		}
+	} else {
+		ret = bt_ctf_field_unsigned_integer_set_value(field, val);
+		if (ret) {
+			pr_err("failed to set field value %s\n", name);
+			goto err;
+		}
+	}
+
+	ret = bt_ctf_event_set_payload(event, name, field);
+	if (ret) {
+		pr_err("failed to set payload %s\n", name);
+		goto err;
+	}
+
+	pr2("  SET [%s = %" PRIu64 "]\n", name, val);
+
+err:
+	bt_ctf_field_put(field);
+	return ret;
+}
+
+#define __FUNC_VALUE_SET(_name, _val_type)				\
+static __maybe_unused int value_set_##_name(struct ctf_writer *cw,	\
+			     struct bt_ctf_event *event,		\
+			     const char *name,				\
+			     _val_type val)				\
+{									\
+	struct bt_ctf_field_type *type = cw->data._name;		\
+	return value_set(type, event, name, (u64) val);			\
+}
+
+#define FUNC_VALUE_SET(_name) __FUNC_VALUE_SET(_name, _name)
+
+FUNC_VALUE_SET(s32)
+FUNC_VALUE_SET(u32)
+FUNC_VALUE_SET(s64)
+FUNC_VALUE_SET(u64)
+__FUNC_VALUE_SET(u64_hex, u64)
+
+static int add_generic_values(struct ctf_writer *cw,
+			      struct bt_ctf_event *event,
+			      struct perf_evsel *evsel,
+			      struct perf_sample *sample)
+{
+	u64 type = evsel->attr.sample_type;
+	int ret;
+
+	/*
+	 * missing:
+	 *   PERF_SAMPLE_TIME         - not needed as we have it in
+	 *                              ctf event header
+	 *   PERF_SAMPLE_READ         - TODO
+	 *   PERF_SAMPLE_CALLCHAIN    - TODO
+	 *   PERF_SAMPLE_RAW          - tracepoint fields are handled separately
+	 *   PERF_SAMPLE_BRANCH_STACK - TODO
+	 *   PERF_SAMPLE_REGS_USER    - TODO
+	 *   PERF_SAMPLE_STACK_USER   - TODO
+	 */
+
+	if (type & PERF_SAMPLE_IP) {
+		ret = value_set_u64_hex(cw, event, "ip", sample->ip);
+		if (ret)
+			return -1;
+	}
+
+	if (type & PERF_SAMPLE_TID) {
+		ret = value_set_s32(cw, event, "tid", sample->tid);
+		if (ret)
+			return -1;
+
+		ret = value_set_s32(cw, event, "pid", sample->pid);
+		if (ret)
+			return -1;
+	}
+
+	if ((type & PERF_SAMPLE_ID) ||
+	    (type & PERF_SAMPLE_IDENTIFIER)) {
+		ret = value_set_u64(cw, event, "id", sample->id);
+		if (ret)
+			return -1;
+	}
+
+	if (type & PERF_SAMPLE_STREAM_ID) {
+		ret = value_set_u64(cw, event, "stream_id", sample->stream_id);
+		if (ret)
+			return -1;
+	}
+
+	if (type & PERF_SAMPLE_CPU) {
+		ret = value_set_u32(cw, event, "cpu", sample->cpu);
+		if (ret)
+			return -1;
+	}
+
+	if (type & PERF_SAMPLE_PERIOD) {
+		ret = value_set_u64(cw, event, "period", sample->period);
+		if (ret)
+			return -1;
+	}
+
+	if (type & PERF_SAMPLE_WEIGHT) {
+		ret = value_set_u64(cw, event, "weight", sample->weight);
+		if (ret)
+			return -1;
+	}
+
+	if (type & PERF_SAMPLE_DATA_SRC) {
+		ret = value_set_u64(cw, event, "data_src", sample->data_src);
+		if (ret)
+			return -1;
+	}
+
+	if (type & PERF_SAMPLE_TRANSACTION) {
+		ret = value_set_u64(cw, event, "transaction", sample->transaction);
+		if (ret)
+			return -1;
+	}
+
+	return 0;
+}
+
+static int process_sample_event(struct perf_tool *tool,
+				union perf_event *_event __maybe_unused,
+				struct perf_sample *sample,
+				struct perf_evsel *evsel,
+				struct machine *machine __maybe_unused)
+{
+	struct convert *c = container_of(tool, struct convert, tool);
+	struct evsel_priv *priv = evsel->priv;
+	struct ctf_writer *cw = &c->writer;
+	struct bt_ctf_event_class *event_class;
+	struct bt_ctf_event *event;
+	int ret;
+
+	if (WARN_ONCE(!priv, "Failed to setup all events.\n"))
+		return 0;
+
+	event_class = priv->event_class;
+
+	/* update stats */
+	c->events_count++;
+	c->events_size += _event->header.size;
+
+	pr_time2(sample->time, "sample %" PRIu64 "\n", c->events_count);
+
+	event = bt_ctf_event_create(event_class);
+	if (!event) {
+		pr_err("Failed to create an CTF event\n");
+		return -1;
+	}
+
+	bt_ctf_clock_set_time(cw->clock, sample->time);
+
+	ret = add_generic_values(cw, event, evsel, sample);
+	if (ret)
+		return -1;
+
+	bt_ctf_stream_append_event(cw->stream, event);
+	bt_ctf_event_put(event);
+	return 0;
+}
+
+static int add_generic_types(struct ctf_writer *cw, struct perf_evsel *evsel,
+			     struct bt_ctf_event_class *event_class)
+{
+	u64 type = evsel->attr.sample_type;
+
+	/*
+	 * missing:
+	 *   PERF_SAMPLE_TIME         - not needed as we have it in
+	 *                              ctf event header
+	 *   PERF_SAMPLE_READ         - TODO
+	 *   PERF_SAMPLE_CALLCHAIN    - TODO
+	 *   PERF_SAMPLE_RAW          - tracepoint fields are handled separately
+	 *   PERF_SAMPLE_BRANCH_STACK - TODO
+	 *   PERF_SAMPLE_REGS_USER    - TODO
+	 *   PERF_SAMPLE_STACK_USER   - TODO
+	 */
+
+#define ADD_FIELD(cl, t, n)						\
+	do {								\
+		pr2("  field '%s'\n", n);				\
+		if (bt_ctf_event_class_add_field(cl, t, n)) {		\
+			pr_err("Failed to add field '%s;\n", n);	\
+			return -1;					\
+		}							\
+	} while (0)
+
+	if (type & PERF_SAMPLE_IP)
+		ADD_FIELD(event_class, cw->data.u64_hex, "ip");
+
+	if (type & PERF_SAMPLE_TID) {
+		ADD_FIELD(event_class, cw->data.s32, "tid");
+		ADD_FIELD(event_class, cw->data.s32, "pid");
+	}
+
+	if ((type & PERF_SAMPLE_ID) ||
+	    (type & PERF_SAMPLE_IDENTIFIER))
+		ADD_FIELD(event_class, cw->data.u64, "id");
+
+	if (type & PERF_SAMPLE_STREAM_ID)
+		ADD_FIELD(event_class, cw->data.u64, "stream_id");
+
+	if (type & PERF_SAMPLE_CPU)
+		ADD_FIELD(event_class, cw->data.u32, "cpu");
+
+	if (type & PERF_SAMPLE_PERIOD)
+		ADD_FIELD(event_class, cw->data.u64, "period");
+
+	if (type & PERF_SAMPLE_WEIGHT)
+		ADD_FIELD(event_class, cw->data.u64, "weight");
+
+	if (type & PERF_SAMPLE_DATA_SRC)
+		ADD_FIELD(event_class, cw->data.u64, "data_src");
+
+	if (type & PERF_SAMPLE_TRANSACTION)
+		ADD_FIELD(event_class, cw->data.u64, "transaction");
+
+#undef ADD_FIELD
+	return 0;
+}
+
+static int add_event(struct ctf_writer *cw, struct perf_evsel *evsel)
+{
+	struct bt_ctf_event_class *event_class;
+	struct evsel_priv *priv;
+	const char *name = perf_evsel__name(evsel);
+	int ret;
+
+	pr("Adding event '%s' (type %d)\n", name, evsel->attr.type);
+
+	event_class = bt_ctf_event_class_create(name);
+	if (!event_class)
+		return -1;
+
+	ret = add_generic_types(cw, evsel, event_class);
+	if (ret)
+		goto err;
+
+	ret = bt_ctf_stream_class_add_event_class(cw->stream_class, event_class);
+	if (ret) {
+		pr("Failed to add event class into stream.\n");
+		goto err;
+	}
+
+	priv = malloc(sizeof(*priv));
+	if (!priv)
+		goto err;
+
+	priv->event_class = event_class;
+	evsel->priv       = priv;
+	return 0;
+
+err:
+	bt_ctf_event_class_put(event_class);
+	pr_err("Failed to add event '%s'.\n", name);
+	return -1;
+}
+
+static int setup_events(struct ctf_writer *cw, struct perf_session *session)
+{
+	struct perf_evlist *evlist = session->evlist;
+	struct perf_evsel *evsel;
+	int ret;
+
+	evlist__for_each(evlist, evsel) {
+		ret = add_event(cw, evsel);
+		if (ret)
+			return ret;
+	}
+	return 0;
+}
+
+static int ctf_writer__setup_env(struct ctf_writer *cw,
+				 struct perf_session *session)
+{
+	struct perf_header *header = &session->header;
+	struct bt_ctf_writer *writer = cw->writer;
+
+#define ADD(__n, __v)							\
+do {									\
+	if (bt_ctf_writer_add_environment_field(writer, __n, __v))	\
+		return -1;						\
+} while (0)
+
+	ADD("host",    header->env.hostname);
+	ADD("sysname", "Linux");
+	ADD("release", header->env.os_release);
+	ADD("version", header->env.version);
+	ADD("machine", header->env.arch);
+	ADD("domain", "kernel");
+	ADD("tracer_name", "perf");
+
+#undef ADD
+	return 0;
+}
+
+static int ctf_writer__setup_clock(struct ctf_writer *cw)
+{
+	struct bt_ctf_clock *clock = cw->clock;
+
+	bt_ctf_clock_set_description(clock, "perf clock");
+
+#define SET(__n, __v)				\
+do {						\
+	if (bt_ctf_clock_set_##__n(clock, __v))	\
+		return -1;			\
+} while (0)
+
+	SET(frequency,   1000000000);
+	SET(offset_s,    0);
+	SET(offset,      0);
+	SET(precision,   10);
+	SET(is_absolute, 0);
+
+#undef SET
+	return 0;
+}
+
+static struct bt_ctf_field_type *create_int_type(int size, bool sign, bool hex)
+{
+	struct bt_ctf_field_type *type;
+
+	type = bt_ctf_field_type_integer_create(size);
+	if (!type)
+		return NULL;
+
+	if (sign &&
+	    bt_ctf_field_type_integer_set_signed(type, 1))
+		goto err;
+
+	if (hex &&
+	    bt_ctf_field_type_integer_set_base(type, BT_CTF_INTEGER_BASE_HEXADECIMAL))
+		goto err;
+
+	pr2("Created type: INTEGER %d-bit %ssigned %s\n",
+	    size, sign ? "un" : "", hex ? "hex" : "");
+	return type;
+
+err:
+	bt_ctf_field_type_put(type);
+	return NULL;
+}
+
+static void ctf_writer__cleanup_data(struct ctf_writer *cw)
+{
+	unsigned int i;
+
+	for (i = 0; i < ARRAY_SIZE(cw->data.array); i++)
+		bt_ctf_field_type_put(cw->data.array[i]);
+}
+
+static int ctf_writer__init_data(struct ctf_writer *cw)
+{
+#define CREATE_INT_TYPE(type, size, sign, hex)		\
+do {							\
+	(type) = create_int_type(size, sign, hex);	\
+	if (!(type))					\
+		goto err;				\
+} while (0)
+
+	CREATE_INT_TYPE(cw->data.s64, 64, true,  false);
+	CREATE_INT_TYPE(cw->data.u64, 64, false, false);
+	CREATE_INT_TYPE(cw->data.s32, 32, true,  false);
+	CREATE_INT_TYPE(cw->data.u32, 32, false, false);
+	CREATE_INT_TYPE(cw->data.u64_hex, 64, false, true);
+
+	cw->data.string  = bt_ctf_field_type_string_create();
+	if (cw->data.string)
+		return 0;
+
+err:
+	ctf_writer__cleanup_data(cw);
+	pr_err("Failed to create data types.\n");
+	return -1;
+}
+
+static void ctf_writer__cleanup(struct ctf_writer *cw)
+{
+	ctf_writer__cleanup_data(cw);
+
+	bt_ctf_clock_put(cw->clock);
+	bt_ctf_stream_put(cw->stream);
+	bt_ctf_stream_class_put(cw->stream_class);
+	bt_ctf_writer_put(cw->writer);
+
+	/* and NULL all the pointers */
+	memset(cw, 0, sizeof(*cw));
+}
+
+static int ctf_writer__init(struct ctf_writer *cw, const char *path)
+{
+	struct bt_ctf_writer		*writer;
+	struct bt_ctf_stream_class	*stream_class;
+	struct bt_ctf_stream		*stream;
+	struct bt_ctf_clock		*clock;
+
+	/* CTF writer */
+	writer = bt_ctf_writer_create(path);
+	if (!writer)
+		goto err;
+
+	cw->writer = writer;
+
+	/* CTF clock */
+	clock = bt_ctf_clock_create("perf_clock");
+	if (!clock) {
+		pr("Failed to create CTF clock.\n");
+		goto err_cleanup;
+	}
+
+	cw->clock = clock;
+
+	if (ctf_writer__setup_clock(cw)) {
+		pr("Failed to setup CTF clock.\n");
+		goto err_cleanup;
+	}
+
+	/* CTF stream class */
+	stream_class = bt_ctf_stream_class_create("perf_stream");
+	if (!stream_class) {
+		pr("Failed to create CTF stream class.\n");
+		goto err_cleanup;
+	}
+
+	cw->stream_class = stream_class;
+
+	/* CTF clock stream setup */
+	if (bt_ctf_stream_class_set_clock(stream_class, clock)) {
+		pr("Failed to assign CTF clock to stream class.\n");
+		goto err_cleanup;
+	}
+
+	if (ctf_writer__init_data(cw))
+		goto err_cleanup;
+
+	/* CTF stream instance */
+	stream = bt_ctf_writer_create_stream(writer, stream_class);
+	if (!stream) {
+		pr("Failed to create CTF stream.\n");
+		goto err_cleanup;
+	}
+
+	cw->stream = stream;
+
+	/* CTF clock writer setup */
+	if (bt_ctf_writer_add_clock(writer, clock)) {
+		pr("Failed to assign CTF clock to writer.\n");
+		goto err_cleanup;
+	}
+
+	return 0;
+
+err_cleanup:
+	ctf_writer__cleanup(cw);
+err:
+	pr_err("Failed to setup CTF writer.\n");
+	return -1;
+}
+
+int bt_convert__perf2ctf(const char *input, const char *path)
+{
+	struct perf_session *session;
+	struct perf_data_file file = {
+		.path = input,
+		.mode = PERF_DATA_MODE_READ,
+	};
+	struct convert c = {
+		.tool = {
+			.sample          = process_sample_event,
+			.mmap            = perf_event__process_mmap,
+			.mmap2           = perf_event__process_mmap2,
+			.comm            = perf_event__process_comm,
+			.exit            = perf_event__process_exit,
+			.fork            = perf_event__process_fork,
+			.lost            = perf_event__process_lost,
+			.tracing_data    = perf_event__process_tracing_data,
+			.build_id        = perf_event__process_build_id,
+			.ordered_events  = true,
+			.ordering_requires_timestamps = true,
+		},
+	};
+	struct ctf_writer *cw = &c.writer;
+	int err = -1;
+
+	/* CTF writer */
+	if (ctf_writer__init(cw, path))
+		return -1;
+
+	/* perf.data session */
+	session = perf_session__new(&file, 0, NULL);
+	if (!session)
+		goto free_writer;
+
+	/* CTF writer env/clock setup  */
+	if (ctf_writer__setup_env(cw, session))
+		goto free_session;
+
+	/* CTF events setup */
+	if (setup_events(cw, session))
+		goto free_session;
+
+	err = perf_session__process_events(session, &c.tool);
+	if (!err)
+		err = bt_ctf_stream_flush(cw->stream);
+
+	fprintf(stderr,
+		"[ perf data convert: Converted '%s' into CTF data '%s' ]\n",
+		file.path, path);
+
+	fprintf(stderr,
+		"[ perf data convert: Converted and wrote %.3f MB (%" PRIu64 " samples) ]\n",
+		(double) c.events_size / 1024.0 / 1024.0,
+		c.events_count);
+
+	/* its all good */
+free_session:
+	perf_session__delete(session);
+
+free_writer:
+	ctf_writer__cleanup(cw);
+	return err;
+}
diff --git a/tools/perf/util/data-convert-bt.h b/tools/perf/util/data-convert-bt.h
new file mode 100644
index 000000000000..dda30c5d0792
--- /dev/null
+++ b/tools/perf/util/data-convert-bt.h
@@ -0,0 +1,8 @@
+#ifndef __DATA_CONVERT_BT_H
+#define __DATA_CONVERT_BT_H
+#ifdef HAVE_LIBBABELTRACE_SUPPORT
+
+int bt_convert__perf2ctf(const char *input_name, const char *to_ctf);
+
+#endif /* HAVE_LIBBABELTRACE_SUPPORT */
+#endif /* __DATA_CONVERT_BT_H */
diff --git a/tools/perf/util/debug.c b/tools/perf/util/debug.c
index ad60b2f20258..2da5581ec74d 100644
--- a/tools/perf/util/debug.c
+++ b/tools/perf/util/debug.c
@@ -20,6 +20,7 @@ int verbose;
 bool dump_trace = false, quiet = false;
 int debug_ordered_events;
 static int redirect_to_stderr;
+int debug_data_convert;
 
 static int _eprintf(int level, int var, const char *fmt, va_list args)
 {
@@ -147,6 +148,7 @@ static struct debug_variable {
 	{ .name = "verbose",		.ptr = &verbose },
 	{ .name = "ordered-events",	.ptr = &debug_ordered_events},
 	{ .name = "stderr",		.ptr = &redirect_to_stderr},
+	{ .name = "data-convert",	.ptr = &debug_data_convert },
 	{ .name = NULL, }
 };
 
diff --git a/tools/perf/util/debug.h b/tools/perf/util/debug.h
index be264d6f3b30..caac2fdc6105 100644
--- a/tools/perf/util/debug.h
+++ b/tools/perf/util/debug.h
@@ -12,6 +12,7 @@
 extern int verbose;
 extern bool quiet, dump_trace;
 extern int debug_ordered_events;
+extern int debug_data_convert;
 
 #ifndef pr_fmt
 #define pr_fmt(fmt) fmt
-- 
1.9.3

[PATCH 04/11] perf data: Add a 'perf' prefix to the generic fields

[Jiri Olsa]

From: Sebastian Andrzej Siewior <bigeasy@linutronix.de>

Some of the tracers bring their own id or pid fields and we can end up
having two of them. This patch adds a "perf_" prefix to the 'generic'
fields so we avoid a clash of the member names.

The change is visible in the babeltrace output:

Before:
  $ babeltrace ./ctf-data/
  [03:19:13.962131936] (+0.000001935) cycles: { }, { ip = 0xFFFFFFFF8105443A, tid = 20714, pid = 20714, period = 8 }
  [03:19:13.962133732] (+0.000001796) cycles: { }, { ip = 0xFFFFFFFF8105443A, tid = 20714, pid = 20714, period = 114 }
  ...

Now:
  $ babeltrace ./ctf-data/
  [03:19:13.962131936] (+0.000001935) cycles: { }, { perf_ip = 0xFFFFFFFF8105443A, perf_tid = 20714, perf_pid = 20714, perf_period = 8 }
  [03:19:13.962133732] (+0.000001796) cycles: { }, { perf_ip = 0xFFFFFFFF8105443A, perf_tid = 20714, perf_pid = 20714, perf_period = 114 }
  ...

Cc: Arnaldo Carvalho de Melo <acme@redhat.com>
Cc: David Ahern <dsahern@gmail.com>
Cc: Frederic Weisbecker <fweisbec@gmail.com>
Cc: Jeremie Galarneau <jgalar@efficios.com>
Cc: Jiri Olsa <jolsa@kernel.org>
Cc: Namhyung Kim <namhyung@gmail.com>
Cc: Paul Mackerras <paulus@samba.org>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Sebastian Andrzej Siewior <bigeasy@linutronix.de>
Cc: Tom Zanussi <tzanussi@gmail.com>
Cc: Wang Nan <wangnan0@huawei.com>
Signed-off-by: Jiri Olsa <jolsa@kernel.org>
Signed-off-by: Sebastian Andrzej Siewior <bigeasy@linutronix.de>
---
 tools/perf/util/data-convert-bt.c | 42 ++++++++++++++++++++-------------------
 1 file changed, 22 insertions(+), 20 deletions(-)

diff --git a/tools/perf/util/data-convert-bt.c b/tools/perf/util/data-convert-bt.c
index ff4826c1745f..e372e03ff480 100644
--- a/tools/perf/util/data-convert-bt.c
+++ b/tools/perf/util/data-convert-bt.c
@@ -147,60 +147,62 @@ static int add_generic_values(struct ctf_writer *cw,
 	 */
 
 	if (type & PERF_SAMPLE_IP) {
-		ret = value_set_u64_hex(cw, event, "ip", sample->ip);
+		ret = value_set_u64_hex(cw, event, "perf_ip", sample->ip);
 		if (ret)
 			return -1;
 	}
 
 	if (type & PERF_SAMPLE_TID) {
-		ret = value_set_s32(cw, event, "tid", sample->tid);
+		ret = value_set_s32(cw, event, "perf_tid", sample->tid);
 		if (ret)
 			return -1;
 
-		ret = value_set_s32(cw, event, "pid", sample->pid);
+		ret = value_set_s32(cw, event, "perf_pid", sample->pid);
 		if (ret)
 			return -1;
 	}
 
 	if ((type & PERF_SAMPLE_ID) ||
 	    (type & PERF_SAMPLE_IDENTIFIER)) {
-		ret = value_set_u64(cw, event, "id", sample->id);
+		ret = value_set_u64(cw, event, "perf_id", sample->id);
 		if (ret)
 			return -1;
 	}
 
 	if (type & PERF_SAMPLE_STREAM_ID) {
-		ret = value_set_u64(cw, event, "stream_id", sample->stream_id);
+		ret = value_set_u64(cw, event, "perf_stream_id", sample->stream_id);
 		if (ret)
 			return -1;
 	}
 
 	if (type & PERF_SAMPLE_CPU) {
-		ret = value_set_u32(cw, event, "cpu", sample->cpu);
+		ret = value_set_u32(cw, event, "perf_cpu", sample->cpu);
 		if (ret)
 			return -1;
 	}
 
 	if (type & PERF_SAMPLE_PERIOD) {
-		ret = value_set_u64(cw, event, "period", sample->period);
+		ret = value_set_u64(cw, event, "perf_period", sample->period);
 		if (ret)
 			return -1;
 	}
 
 	if (type & PERF_SAMPLE_WEIGHT) {
-		ret = value_set_u64(cw, event, "weight", sample->weight);
+		ret = value_set_u64(cw, event, "perf_weight", sample->weight);
 		if (ret)
 			return -1;
 	}
 
 	if (type & PERF_SAMPLE_DATA_SRC) {
-		ret = value_set_u64(cw, event, "data_src", sample->data_src);
+		ret = value_set_u64(cw, event, "perf_data_src",
+				sample->data_src);
 		if (ret)
 			return -1;
 	}
 
 	if (type & PERF_SAMPLE_TRANSACTION) {
-		ret = value_set_u64(cw, event, "transaction", sample->transaction);
+		ret = value_set_u64(cw, event, "perf_transaction",
+				sample->transaction);
 		if (ret)
 			return -1;
 	}
@@ -276,34 +278,34 @@ static int add_generic_types(struct ctf_writer *cw, struct perf_evsel *evsel,
 	} while (0)
 
 	if (type & PERF_SAMPLE_IP)
-		ADD_FIELD(event_class, cw->data.u64_hex, "ip");
+		ADD_FIELD(event_class, cw->data.u64_hex, "perf_ip");
 
 	if (type & PERF_SAMPLE_TID) {
-		ADD_FIELD(event_class, cw->data.s32, "tid");
-		ADD_FIELD(event_class, cw->data.s32, "pid");
+		ADD_FIELD(event_class, cw->data.s32, "perf_tid");
+		ADD_FIELD(event_class, cw->data.s32, "perf_pid");
 	}
 
 	if ((type & PERF_SAMPLE_ID) ||
 	    (type & PERF_SAMPLE_IDENTIFIER))
-		ADD_FIELD(event_class, cw->data.u64, "id");
+		ADD_FIELD(event_class, cw->data.u64, "perf_id");
 
 	if (type & PERF_SAMPLE_STREAM_ID)
-		ADD_FIELD(event_class, cw->data.u64, "stream_id");
+		ADD_FIELD(event_class, cw->data.u64, "perf_stream_id");
 
 	if (type & PERF_SAMPLE_CPU)
-		ADD_FIELD(event_class, cw->data.u32, "cpu");
+		ADD_FIELD(event_class, cw->data.u32, "perf_cpu");
 
 	if (type & PERF_SAMPLE_PERIOD)
-		ADD_FIELD(event_class, cw->data.u64, "period");
+		ADD_FIELD(event_class, cw->data.u64, "perf_period");
 
 	if (type & PERF_SAMPLE_WEIGHT)
-		ADD_FIELD(event_class, cw->data.u64, "weight");
+		ADD_FIELD(event_class, cw->data.u64, "perf_weight");
 
 	if (type & PERF_SAMPLE_DATA_SRC)
-		ADD_FIELD(event_class, cw->data.u64, "data_src");
+		ADD_FIELD(event_class, cw->data.u64, "perf_data_src");
 
 	if (type & PERF_SAMPLE_TRANSACTION)
-		ADD_FIELD(event_class, cw->data.u64, "transaction");
+		ADD_FIELD(event_class, cw->data.u64, "perf_transaction");
 
 #undef ADD_FIELD
 	return 0;
-- 
1.9.3

[PATCH 05/11] perf data: Add tracepoint events fields CTF conversion support

[Jiri Olsa]

From: Sebastian Andrzej Siewior <bigeasy@linutronix.de>

Adding support to convert tracepoint event fields into CTF
event fields.

We parse each tracepoint event for CTF conversion and add
tracepoint fields as regular CTF event fields, so they
appear in babeltrace output like:

  $ babeltrace ./ctf-data/
  ...
  [09:02:00.950703057] (+?.?????????) sched:sched_stat_runtime: { }, { perf_ip = ... SNIP ... common_type = 298, common_flags = 1, \
  common_preempt_count = 0, common_pid = 31813, comm = "perf", pid = 31813, runtime = 458800, vruntime = 52059858071 }
  ...

Cc: Arnaldo Carvalho de Melo <acme@redhat.com>
Cc: David Ahern <dsahern@gmail.com>
Cc: Frederic Weisbecker <fweisbec@gmail.com>
Cc: Jeremie Galarneau <jgalar@efficios.com>
Cc: Jiri Olsa <jolsa@kernel.org>
Cc: Namhyung Kim <namhyung@gmail.com>
Cc: Paul Mackerras <paulus@samba.org>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Sebastian Andrzej Siewior <bigeasy@linutronix.de>
Cc: Tom Zanussi <tzanussi@gmail.com>
Cc: Wang Nan <wangnan0@huawei.com>
Signed-off-by: Jiri Olsa <jolsa@kernel.org>
Signed-off-by: Sebastian Andrzej Siewior <bigeasy@linutronix.de>
---
 tools/perf/util/data-convert-bt.c | 242 ++++++++++++++++++++++++++++++++++++++
 1 file changed, 242 insertions(+)

diff --git a/tools/perf/util/data-convert-bt.c b/tools/perf/util/data-convert-bt.c
index e372e03ff480..6fa5c3ef336b 100644
--- a/tools/perf/util/data-convert-bt.c
+++ b/tools/perf/util/data-convert-bt.c
@@ -126,6 +126,177 @@ FUNC_VALUE_SET(s64)
 FUNC_VALUE_SET(u64)
 __FUNC_VALUE_SET(u64_hex, u64)
 
+static struct bt_ctf_field_type*
+get_tracepoint_field_type(struct ctf_writer *cw, struct format_field *field)
+{
+	unsigned long flags = field->flags;
+
+	if (flags & FIELD_IS_STRING)
+		return cw->data.string;
+
+	if (!(flags & FIELD_IS_SIGNED)) {
+		/* unsigned long are mostly pointers */
+		if (flags & FIELD_IS_LONG || flags & FIELD_IS_POINTER)
+			return cw->data.u64_hex;
+	}
+
+	if (flags & FIELD_IS_SIGNED) {
+		if (field->size == 8)
+			return cw->data.s64;
+		else
+			return cw->data.s32;
+	}
+
+	if (field->size == 8)
+		return cw->data.u64;
+	else
+		return cw->data.u32;
+}
+
+static int add_tracepoint_field_value(struct ctf_writer *cw,
+				      struct bt_ctf_event_class *event_class,
+				      struct bt_ctf_event *event,
+				      struct perf_sample *sample,
+				      struct format_field *fmtf)
+{
+	struct bt_ctf_field_type *type;
+	struct bt_ctf_field *array_field;
+	struct bt_ctf_field *field;
+	const char *name = fmtf->name;
+	void *data = sample->raw_data;
+	unsigned long long value_int;
+	unsigned long flags = fmtf->flags;
+	unsigned int n_items;
+	unsigned int i;
+	unsigned int offset;
+	unsigned int len;
+	int ret;
+
+	offset = fmtf->offset;
+	len = fmtf->size;
+	if (flags & FIELD_IS_STRING)
+		flags &= ~FIELD_IS_ARRAY;
+
+	if (flags & FIELD_IS_DYNAMIC) {
+		unsigned long long tmp_val;
+
+		tmp_val = pevent_read_number(fmtf->event->pevent,
+				data + offset, len);
+		offset = tmp_val;
+		len = offset >> 16;
+		offset &= 0xffff;
+	}
+
+	if (flags & FIELD_IS_ARRAY) {
+
+		type = bt_ctf_event_class_get_field_by_name(
+				event_class, name);
+		array_field = bt_ctf_field_create(type);
+		bt_ctf_field_type_put(type);
+		if (!array_field) {
+			pr_err("Failed to create array type %s\n", name);
+			return -1;
+		}
+
+		len = fmtf->size / fmtf->arraylen;
+		n_items = fmtf->arraylen;
+	} else {
+		n_items = 1;
+		array_field = NULL;
+	}
+
+	type = get_tracepoint_field_type(cw, fmtf);
+
+	for (i = 0; i < n_items; i++) {
+		if (!(flags & FIELD_IS_STRING))
+			value_int = pevent_read_number(
+					fmtf->event->pevent,
+					data + offset + i * len, len);
+
+		if (flags & FIELD_IS_ARRAY)
+			field = bt_ctf_field_array_get_field(array_field, i);
+		else
+			field = bt_ctf_field_create(type);
+
+		if (!field) {
+			pr_err("failed to create a field %s\n", name);
+			return -1;
+		}
+
+		if (flags & FIELD_IS_STRING)
+			ret = bt_ctf_field_string_set_value(field,
+					data + offset + i * len);
+		else if (!(flags & FIELD_IS_SIGNED))
+			ret = bt_ctf_field_unsigned_integer_set_value(
+					field, value_int);
+		else
+			ret = bt_ctf_field_signed_integer_set_value(
+					field, value_int);
+		if (ret) {
+			pr_err("failed to set file value %s\n", name);
+			goto err_put_field;
+		}
+		if (!(flags & FIELD_IS_ARRAY)) {
+			ret = bt_ctf_event_set_payload(event, name, field);
+			if (ret) {
+				pr_err("failed to set payload %s\n", name);
+				goto err_put_field;
+			}
+		}
+		bt_ctf_field_put(field);
+	}
+	if (flags & FIELD_IS_ARRAY) {
+		ret = bt_ctf_event_set_payload(event, name, array_field);
+		if (ret) {
+			pr_err("Failed add payload array %s\n", name);
+			return -1;
+		}
+		bt_ctf_field_put(array_field);
+	}
+	return 0;
+
+err_put_field:
+	bt_ctf_field_put(field);
+	return -1;
+}
+
+static int add_tracepoint_fields_values(struct ctf_writer *cw,
+					struct bt_ctf_event_class *event_class,
+					struct bt_ctf_event *event,
+					struct format_field *fields,
+					struct perf_sample *sample)
+{
+	struct format_field *field;
+	int ret;
+
+	for (field = fields; field; field = field->next) {
+		ret = add_tracepoint_field_value(cw, event_class, event, sample,
+				field);
+		if (ret)
+			return -1;
+	}
+	return 0;
+}
+
+static int add_tracepoint_values(struct ctf_writer *cw,
+				 struct bt_ctf_event_class *event_class,
+				 struct bt_ctf_event *event,
+				 struct perf_evsel *evsel,
+				 struct perf_sample *sample)
+{
+	struct format_field *common_fields = evsel->tp_format->format.common_fields;
+	struct format_field *fields        = evsel->tp_format->format.fields;
+	int ret;
+
+	ret = add_tracepoint_fields_values(cw, event_class, event,
+					   common_fields, sample);
+	if (!ret)
+		ret = add_tracepoint_fields_values(cw, event_class, event,
+						   fields, sample);
+
+	return ret;
+}
+
 static int add_generic_values(struct ctf_writer *cw,
 			      struct bt_ctf_event *event,
 			      struct perf_evsel *evsel,
@@ -246,11 +417,76 @@ static int process_sample_event(struct perf_tool *tool,
 	if (ret)
 		return -1;
 
+	if (evsel->attr.type == PERF_TYPE_TRACEPOINT) {
+		ret = add_tracepoint_values(cw, event_class, event,
+					    evsel, sample);
+		if (ret)
+			return -1;
+	}
+
 	bt_ctf_stream_append_event(cw->stream, event);
 	bt_ctf_event_put(event);
 	return 0;
 }
 
+static int add_tracepoint_fields_types(struct ctf_writer *cw,
+				       struct format_field *fields,
+				       struct bt_ctf_event_class *event_class)
+{
+	struct format_field *field;
+	int ret;
+
+	for (field = fields; field; field = field->next) {
+		struct bt_ctf_field_type *type;
+		unsigned long flags = field->flags;
+
+		pr2("  field '%s'\n", field->name);
+
+		type = get_tracepoint_field_type(cw, field);
+		if (!type)
+			return -1;
+
+		/*
+		 * A string is an array of chars. For this we use the string
+		 * type and don't care that it is an array. What we don't
+		 * support is an array of strings.
+		 */
+		if (flags & FIELD_IS_STRING)
+			flags &= ~FIELD_IS_ARRAY;
+
+		if (flags & FIELD_IS_ARRAY)
+			type = bt_ctf_field_type_array_create(type, field->arraylen);
+
+		ret = bt_ctf_event_class_add_field(event_class, type,
+				field->name);
+
+		if (flags & FIELD_IS_ARRAY)
+			bt_ctf_field_type_put(type);
+
+		if (ret) {
+			pr_err("Failed to add field '%s\n", field->name);
+			return -1;
+		}
+	}
+
+	return 0;
+}
+
+static int add_tracepoint_types(struct ctf_writer *cw,
+				struct perf_evsel *evsel,
+				struct bt_ctf_event_class *class)
+{
+	struct format_field *common_fields = evsel->tp_format->format.common_fields;
+	struct format_field *fields        = evsel->tp_format->format.fields;
+	int ret;
+
+	ret = add_tracepoint_fields_types(cw, common_fields, class);
+	if (!ret)
+		ret = add_tracepoint_fields_types(cw, fields, class);
+
+	return ret;
+}
+
 static int add_generic_types(struct ctf_writer *cw, struct perf_evsel *evsel,
 			     struct bt_ctf_event_class *event_class)
 {
@@ -328,6 +564,12 @@ static int add_event(struct ctf_writer *cw, struct perf_evsel *evsel)
 	if (ret)
 		goto err;
 
+	if (evsel->attr.type == PERF_TYPE_TRACEPOINT) {
+		ret = add_tracepoint_types(cw, evsel, event_class);
+		if (ret)
+			goto err;
+	}
+
 	ret = bt_ctf_stream_class_add_event_class(cw->stream_class, event_class);
 	if (ret) {
 		pr("Failed to add event class into stream.\n");
-- 
1.9.3

[PATCH 06/11] perf data: Switch to multiple cpu stream files

[Jiri Olsa]

From: Sebastian Andrzej Siewior <bigeasy@linutronix.de>

Currently we store the data into single data strea/file. The cpu
if data is stored within the event sample. The lttng puts the CPU
number that belongs to the event into the packet context instead
into the event.

This patch makes sure that the trace produce by perf does look the
same way. We now use one stream per-CPU. Having it all in one stream
increased the total size of the resulting file. The test went from
416KiB (with perf_cpu event member) to 24MiB due to the required
(and pointless) flush. With the per-cpu streams the total size went
up to 588KiB.

Cc: Arnaldo Carvalho de Melo <acme@redhat.com>
Cc: David Ahern <dsahern@gmail.com>
Cc: Frederic Weisbecker <fweisbec@gmail.com>
Cc: Jeremie Galarneau <jgalar@efficios.com>
Cc: Jiri Olsa <jolsa@kernel.org>
Cc: Namhyung Kim <namhyung@gmail.com>
Cc: Paul Mackerras <paulus@samba.org>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Sebastian Andrzej Siewior <bigeasy@linutronix.de>
Cc: Tom Zanussi <tzanussi@gmail.com>
Cc: Wang Nan <wangnan0@huawei.com>
Signed-off-by: Jiri Olsa <jolsa@kernel.org>
Signed-off-by: Sebastian Andrzej Siewior <bigeasy@linutronix.de>
---
 tools/perf/util/data-convert-bt.c | 205 +++++++++++++++++++++++++++++++++-----
 1 file changed, 181 insertions(+), 24 deletions(-)

diff --git a/tools/perf/util/data-convert-bt.c b/tools/perf/util/data-convert-bt.c
index 6fa5c3ef336b..4bb769e081a8 100644
--- a/tools/perf/util/data-convert-bt.c
+++ b/tools/perf/util/data-convert-bt.c
@@ -38,12 +38,20 @@ struct evsel_priv {
 	struct bt_ctf_event_class *event_class;
 };
 
+#define MAX_CPUS	4096
+
+struct ctf_stream {
+	struct bt_ctf_stream *stream;
+	int cpu;
+};
+
 struct ctf_writer {
 	/* writer primitives */
-	struct bt_ctf_writer		*writer;
-	struct bt_ctf_stream		*stream;
-	struct bt_ctf_stream_class	*stream_class;
-	struct bt_ctf_clock		*clock;
+	struct bt_ctf_writer		 *writer;
+	struct ctf_stream		**stream;
+	int				  stream_cnt;
+	struct bt_ctf_stream_class	 *stream_class;
+	struct bt_ctf_clock		 *clock;
 
 	/* data types */
 	union {
@@ -346,12 +354,6 @@ static int add_generic_values(struct ctf_writer *cw,
 			return -1;
 	}
 
-	if (type & PERF_SAMPLE_CPU) {
-		ret = value_set_u32(cw, event, "perf_cpu", sample->cpu);
-		if (ret)
-			return -1;
-	}
-
 	if (type & PERF_SAMPLE_PERIOD) {
 		ret = value_set_u64(cw, event, "perf_period", sample->period);
 		if (ret)
@@ -381,6 +383,113 @@ static int add_generic_values(struct ctf_writer *cw,
 	return 0;
 }
 
+static int ctf_stream__flush(struct ctf_stream *cs)
+{
+	int err = 0;
+
+	if (cs) {
+		err = bt_ctf_stream_flush(cs->stream);
+		if (err)
+			pr_err("CTF stream %d flush failed\n", cs->cpu);
+
+		pr("Flush stream for cpu %d\n", cs->cpu);
+	}
+
+	return err;
+}
+
+static struct ctf_stream *ctf_stream__create(struct ctf_writer *cw, int cpu)
+{
+	struct ctf_stream *cs;
+	struct bt_ctf_field *pkt_ctx   = NULL;
+	struct bt_ctf_field *cpu_field = NULL;
+	struct bt_ctf_stream *stream   = NULL;
+	int ret;
+
+	cs = zalloc(sizeof(*cs));
+	if (!cs) {
+		pr_err("Failed to allocate ctf stream\n");
+		return NULL;
+	}
+
+	stream = bt_ctf_writer_create_stream(cw->writer, cw->stream_class);
+	if (!stream) {
+		pr_err("Failed to create CTF stream\n");
+		goto out;
+	}
+
+	pkt_ctx = bt_ctf_stream_get_packet_context(stream);
+	if (!pkt_ctx) {
+		pr_err("Failed to obtain packet context\n");
+		goto out;
+	}
+
+	cpu_field = bt_ctf_field_structure_get_field(pkt_ctx, "cpu_id");
+	bt_ctf_field_put(pkt_ctx);
+	if (!cpu_field) {
+		pr_err("Failed to obtain cpu field\n");
+		goto out;
+	}
+
+	ret = bt_ctf_field_unsigned_integer_set_value(cpu_field, (u32) cpu);
+	if (ret) {
+		pr_err("Failed to update CPU number\n");
+		goto out;
+	}
+
+	bt_ctf_field_put(cpu_field);
+
+	cs->cpu    = cpu;
+	cs->stream = stream;
+	return cs;
+
+out:
+	if (cpu_field)
+		bt_ctf_field_put(cpu_field);
+	if (stream)
+		bt_ctf_stream_put(stream);
+
+	free(cs);
+	return NULL;
+}
+
+static void ctf_stream__delete(struct ctf_stream *cs)
+{
+	if (cs) {
+		bt_ctf_stream_put(cs->stream);
+		free(cs);
+	}
+}
+
+static struct ctf_stream *ctf_stream(struct ctf_writer *cw, int cpu)
+{
+	struct ctf_stream *cs = cw->stream[cpu];
+
+	if (!cs) {
+		cs = ctf_stream__create(cw, cpu);
+		cw->stream[cpu] = cs;
+	}
+
+	return cs;
+}
+
+static int get_sample_cpu(struct ctf_writer *cw, struct perf_sample *sample,
+			  struct perf_evsel *evsel)
+{
+	int cpu = 0;
+
+	if (evsel->attr.sample_type & PERF_SAMPLE_CPU)
+		cpu = sample->cpu;
+
+	if (cpu > cw->stream_cnt) {
+		pr_err("Event was recorded for CPU %d, limit is at %d.\n",
+			cpu, cw->stream_cnt);
+		cpu = 0;
+	}
+
+	return cpu;
+}
+
 static int process_sample_event(struct perf_tool *tool,
 				union perf_event *_event __maybe_unused,
 				struct perf_sample *sample,
@@ -390,6 +499,7 @@ static int process_sample_event(struct perf_tool *tool,
 	struct convert *c = container_of(tool, struct convert, tool);
 	struct evsel_priv *priv = evsel->priv;
 	struct ctf_writer *cw = &c->writer;
+	struct ctf_stream *cs;
 	struct bt_ctf_event_class *event_class;
 	struct bt_ctf_event *event;
 	int ret;
@@ -424,9 +534,12 @@ static int process_sample_event(struct perf_tool *tool,
 			return -1;
 	}
 
-	bt_ctf_stream_append_event(cw->stream, event);
+	cs = ctf_stream(cw, get_sample_cpu(cw, sample, evsel));
+	if (cs)
+		bt_ctf_stream_append_event(cs->stream, event);
+
 	bt_ctf_event_put(event);
-	return 0;
+	return cs ? 0 : -1;
 }
 
 static int add_tracepoint_fields_types(struct ctf_writer *cw,
@@ -528,9 +641,6 @@ static int add_generic_types(struct ctf_writer *cw, struct perf_evsel *evsel,
 	if (type & PERF_SAMPLE_STREAM_ID)
 		ADD_FIELD(event_class, cw->data.u64, "perf_stream_id");
 
-	if (type & PERF_SAMPLE_CPU)
-		ADD_FIELD(event_class, cw->data.u32, "perf_cpu");
-
 	if (type & PERF_SAMPLE_PERIOD)
 		ADD_FIELD(event_class, cw->data.u64, "perf_period");
 
@@ -604,6 +714,39 @@ static int setup_events(struct ctf_writer *cw, struct perf_session *session)
 	return 0;
 }
 
+static int setup_streams(struct ctf_writer *cw, struct perf_session *session)
+{
+	struct ctf_stream **stream;
+	struct perf_header *ph = &session->header;
+	int ncpus;
+
+	/*
+	 * Try to get the number of cpus used in the data file,
+	 * if not present fallback to the MAX_CPUS.
+	 */
+	ncpus = ph->env.nr_cpus_avail ?: MAX_CPUS;
+
+	stream = zalloc(sizeof(*stream) * ncpus);
+	if (!stream) {
+		pr_err("Failed to allocate streams.\n");
+		return -ENOMEM;
+	}
+
+	cw->stream     = stream;
+	cw->stream_cnt = ncpus;
+	return 0;
+}
+
+static void free_streams(struct ctf_writer *cw)
+{
+	int cpu;
+
+	for (cpu = 0; cpu < cw->stream_cnt; cpu++)
+		ctf_stream__delete(cw->stream[cpu]);
+
+	free(cw->stream);
+}
+
 static int ctf_writer__setup_env(struct ctf_writer *cw,
 				 struct perf_session *session)
 {
@@ -713,7 +856,7 @@ static void ctf_writer__cleanup(struct ctf_writer *cw)
 	ctf_writer__cleanup_data(cw);
 
 	bt_ctf_clock_put(cw->clock);
-	bt_ctf_stream_put(cw->stream);
+	free_streams(cw);
 	bt_ctf_stream_class_put(cw->stream_class);
 	bt_ctf_writer_put(cw->writer);
 
@@ -725,8 +868,9 @@ static int ctf_writer__init(struct ctf_writer *cw, const char *path)
 {
 	struct bt_ctf_writer		*writer;
 	struct bt_ctf_stream_class	*stream_class;
-	struct bt_ctf_stream		*stream;
 	struct bt_ctf_clock		*clock;
+	struct bt_ctf_field_type	*pkt_ctx_type;
+	int				ret;
 
 	/* CTF writer */
 	writer = bt_ctf_writer_create(path);
@@ -767,14 +911,15 @@ static int ctf_writer__init(struct ctf_writer *cw, const char *path)
 	if (ctf_writer__init_data(cw))
 		goto err_cleanup;
 
-	/* CTF stream instance */
-	stream = bt_ctf_writer_create_stream(writer, stream_class);
-	if (!stream) {
-		pr("Failed to create CTF stream.\n");
+	/* Add cpu_id for packet context */
+	pkt_ctx_type = bt_ctf_stream_class_get_packet_context_type(stream_class);
+	if (!pkt_ctx_type)
 		goto err_cleanup;
-	}
 
-	cw->stream = stream;
+	ret = bt_ctf_field_type_structure_add_field(pkt_ctx_type, cw->data.u32, "cpu_id");
+	bt_ctf_field_type_put(pkt_ctx_type);
+	if (ret)
+		goto err_cleanup;
 
 	/* CTF clock writer setup */
 	if (bt_ctf_writer_add_clock(writer, clock)) {
@@ -791,6 +936,14 @@ err:
 	return -1;
 }
 
+static void ctf_writer__flush_streams(struct ctf_writer *cw)
+{
+	int cpu;
+
+	for (cpu = 0; cpu < cw->stream_cnt; cpu++)
+		ctf_stream__flush(cw->stream[cpu]);
+}
+
 int bt_convert__perf2ctf(const char *input, const char *path)
 {
 	struct perf_session *session;
@@ -833,9 +986,13 @@ int bt_convert__perf2ctf(const char *input, const char *path)
 	if (setup_events(cw, session))
 		goto free_session;
 
+	if (setup_streams(cw, session))
+		goto free_session;
+
 	err = perf_session__process_events(session, &c.tool);
 	if (!err)
-		err = bt_ctf_stream_flush(cw->stream);
+		ctf_writer__flush_streams(cw);
+
 
 	fprintf(stderr,
 		"[ perf data convert: Converted '%s' into CTF data '%s' ]\n",
-- 
1.9.3

[PATCH 07/11] perf data: Enable stream flush within processing

[Jiri Olsa]

For big data files the size of data allocated for stream
instance could get really high. It's needed to flush the
data out of the stream once in a while.

Unfortunately there's no size indication in the stream
object, so we govern the flush based on the number of
stored events. Current flush limit is set ot 100000
events.

Cc: Arnaldo Carvalho de Melo <acme@redhat.com>
Cc: David Ahern <dsahern@gmail.com>
Cc: Frederic Weisbecker <fweisbec@gmail.com>
Cc: Jeremie Galarneau <jgalar@efficios.com>
Cc: Jiri Olsa <jolsa@kernel.org>
Cc: Namhyung Kim <namhyung@gmail.com>
Cc: Paul Mackerras <paulus@samba.org>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Sebastian Andrzej Siewior <bigeasy@linutronix.de>
Cc: Tom Zanussi <tzanussi@gmail.com>
Cc: Wang Nan <wangnan0@huawei.com>
Signed-off-by: Jiri Olsa <jolsa@kernel.org>
---
 tools/perf/util/data-convert-bt.c | 26 ++++++++++++++++++++++++--
 1 file changed, 24 insertions(+), 2 deletions(-)

diff --git a/tools/perf/util/data-convert-bt.c b/tools/perf/util/data-convert-bt.c
index 4bb769e081a8..3d73e51ee10d 100644
--- a/tools/perf/util/data-convert-bt.c
+++ b/tools/perf/util/data-convert-bt.c
@@ -43,6 +43,7 @@ struct evsel_priv {
 struct ctf_stream {
 	struct bt_ctf_stream *stream;
 	int cpu;
+	u32 count;
 };
 
 struct ctf_writer {
@@ -392,7 +393,10 @@ static int ctf_stream__flush(struct ctf_stream *cs)
 		if (err)
 			pr_err("CTF stream %d flush failed\n", cs->cpu);
 
-		pr("Flush stream for cpu %d\n", cs->cpu);
+		pr("Flush stream for cpu %d (%u samples)\n",
+		   cs->cpu, cs->count);
+
+		cs->count = 0;
 	}
 
 	return err;
@@ -490,6 +494,19 @@ static int get_sample_cpu(struct ctf_writer *cw, struct perf_sample *sample,
 	return cpu;
 }
 
+#define STREAM_FLUSH_COUNT 100000
+
+/*
+ * Currently we have no other way to determine the
+ * time for the stream flush other than keep track
+ * of the number of events and check it against
+ * threshold.
+ */
+static bool is_flush_needed(struct ctf_stream *cs)
+{
+	return cs->count >= STREAM_FLUSH_COUNT;
+}
+
 static int process_sample_event(struct perf_tool *tool,
 				union perf_event *_event __maybe_unused,
 				struct perf_sample *sample,
@@ -535,8 +552,13 @@ static int process_sample_event(struct perf_tool *tool,
 	}
 
 	cs = ctf_stream(cw, get_sample_cpu(cw, sample, evsel));
-	if (cs)
+	if (cs) {
+		if (is_flush_needed(cs))
+			ctf_stream__flush(cs);
+
+		cs->count++;
 		bt_ctf_stream_append_event(cs->stream, event);
+	}
 
 	bt_ctf_event_put(event);
 	return cs ? 0 : -1;
-- 
1.9.3

[PATCH 08/11] perf data: Add support for setting ordered_events queue size

[Jiri Olsa]

Adding support to limit the size of ordered_events queue,
so we could control allocation size of perf data files
without proper finished round events.

Cc: Arnaldo Carvalho de Melo <acme@redhat.com>
Cc: David Ahern <dsahern@gmail.com>
Cc: Frederic Weisbecker <fweisbec@gmail.com>
Cc: Jeremie Galarneau <jgalar@efficios.com>
Cc: Jiri Olsa <jolsa@kernel.org>
Cc: Namhyung Kim <namhyung@gmail.com>
Cc: Paul Mackerras <paulus@samba.org>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Sebastian Andrzej Siewior <bigeasy@linutronix.de>
Cc: Tom Zanussi <tzanussi@gmail.com>
Cc: Wang Nan <wangnan0@huawei.com>
Signed-off-by: Jiri Olsa <jolsa@kernel.org>
---
 tools/perf/util/data-convert-bt.c | 22 ++++++++++++++++++++++
 1 file changed, 22 insertions(+)

diff --git a/tools/perf/util/data-convert-bt.c b/tools/perf/util/data-convert-bt.c
index 3d73e51ee10d..ddecce88bf6d 100644
--- a/tools/perf/util/data-convert-bt.c
+++ b/tools/perf/util/data-convert-bt.c
@@ -74,6 +74,9 @@ struct convert {
 
 	u64			events_size;
 	u64			events_count;
+
+	/* Ordered events configured queue size. */
+	u64			queue_size;
 };
 
 static int value_set(struct bt_ctf_field_type *type,
@@ -966,6 +969,18 @@ static void ctf_writer__flush_streams(struct ctf_writer *cw)
 		ctf_stream__flush(cw->stream[cpu]);
 }
 
+static int convert__config(const char *var, const char *value, void *cb)
+{
+	struct convert *c = cb;
+
+	if (!strcmp(var, "convert.queue-size")) {
+		c->queue_size = perf_config_u64(var, value);
+		return 0;
+	}
+
+	return perf_default_config(var, value, cb);
+}
+
 int bt_convert__perf2ctf(const char *input, const char *path)
 {
 	struct perf_session *session;
@@ -991,6 +1006,8 @@ int bt_convert__perf2ctf(const char *input, const char *path)
 	struct ctf_writer *cw = &c.writer;
 	int err = -1;
 
+	perf_config(convert__config, &c);
+
 	/* CTF writer */
 	if (ctf_writer__init(cw, path))
 		return -1;
@@ -1000,6 +1017,11 @@ int bt_convert__perf2ctf(const char *input, const char *path)
 	if (!session)
 		goto free_writer;
 
+	if (c.queue_size) {
+		ordered_events__set_alloc_size(&session->ordered_events,
+					       c.queue_size);
+	}
+
 	/* CTF writer env/clock setup  */
 	if (ctf_writer__setup_env(cw, session))
 		goto free_session;
-- 
1.9.3

[PATCH 09/11] tools lib traceevent: Add alias field to struct format_field

[Jiri Olsa]

Introduce an alias field to 'struct format_field' to be able
to use alternative name for the field.

This will be handy when converting data into CTF, where each
field within event needs to have a unique name (while this
is not required for tracepoint). Converter can easily assign
unique name into the format_field struct.

Suggested-by: Steven Rostedt <rostedt@goodmis.org>
Link: http://marc.info/?l=linux-kernel&m=142263154901362&w=2
Cc: Wang Nan <wangnan0@huawei.com>
Cc: Arnaldo Carvalho de Melo <acme@redhat.com>
Cc: David Ahern <dsahern@gmail.com>
Cc: Frederic Weisbecker <fweisbec@gmail.com>
Cc: Jiri Olsa <jolsa@kernel.org>
Cc: Namhyung Kim <namhyung@gmail.com>
Cc: Paul Mackerras <paulus@samba.org>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Sebastian Andrzej Siewior <bigeasy@linutronix.de>
Cc: Tom Zanussi <tzanussi@gmail.com>
Signed-off-by: Jiri Olsa <jolsa@kernel.org>
---
 tools/lib/traceevent/event-parse.c | 1 +
 tools/lib/traceevent/event-parse.h | 1 +
 2 files changed, 2 insertions(+)

diff --git a/tools/lib/traceevent/event-parse.c b/tools/lib/traceevent/event-parse.c
index afe20ed9fac8..b35664cc1edc 100644
--- a/tools/lib/traceevent/event-parse.c
+++ b/tools/lib/traceevent/event-parse.c
@@ -6236,6 +6236,7 @@ static void free_format_fields(struct format_field *field)
 		next = field->next;
 		free(field->type);
 		free(field->name);
+		free(field->alias);
 		free(field);
 		field = next;
 	}
diff --git a/tools/lib/traceevent/event-parse.h b/tools/lib/traceevent/event-parse.h
index 5b4efc062320..8490a4afffa2 100644
--- a/tools/lib/traceevent/event-parse.h
+++ b/tools/lib/traceevent/event-parse.h
@@ -187,6 +187,7 @@ struct format_field {
 	struct event_format	*event;
 	char			*type;
 	char			*name;
+	char			*alias;
 	int			offset;
 	int			size;
 	unsigned int		arraylen;
-- 
1.9.3

[PATCH 10/11] perf data: Fix duplicate field names and avoid reserved keywords

[Jiri Olsa]

From: Wang Nan <wangnan0@huawei.com>

Some parameters of syscall tracepoints named as 'nr', 'event', etc.
When dealing with them, perf convert to ctf meets some problem:

 1. If a parameter with name 'nr', it will duplicate syscall's
    common field 'nr'. One such syscall is io_submit().

 2. If a parameter with name 'event', it is denied to be inserted
    because 'event' is a babeltrace keywork. One such syscall is
    epoll_ctl.

This patch appends '_dupl_X' suffix to avoid problem 1, prepend a '_'
prefix to avoid problem 2.

Signed-off-by: Wang Nan <wangnan0@huawei.com>
Cc: Arnaldo Carvalho de Melo <acme@redhat.com>
Cc: David Ahern <dsahern@gmail.com>
Cc: Frederic Weisbecker <fweisbec@gmail.com>
Cc: Jeremie Galarneau <jgalar@efficios.com>
Cc: Jiri Olsa <jolsa@kernel.org>
Cc: Namhyung Kim <namhyung@gmail.com>
Cc: Paul Mackerras <paulus@samba.org>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Sebastian Andrzej Siewior <bigeasy@linutronix.de>
Cc: Tom Zanussi <tzanussi@gmail.com>
[ changed to use format_file::alias ]
Signed-off-by: Jiri Olsa <jolsa@kernel.org>
---
 tools/perf/util/data-convert-bt.c | 85 +++++++++++++++++++++++++++++++++++++--
 1 file changed, 81 insertions(+), 4 deletions(-)

diff --git a/tools/perf/util/data-convert-bt.c b/tools/perf/util/data-convert-bt.c
index ddecce88bf6d..95de04c31a80 100644
--- a/tools/perf/util/data-convert-bt.c
+++ b/tools/perf/util/data-convert-bt.c
@@ -14,6 +14,7 @@
 #include <babeltrace/ctf-writer/event.h>
 #include <babeltrace/ctf-writer/event-types.h>
 #include <babeltrace/ctf-writer/event-fields.h>
+#include <babeltrace/ctf-ir/utils.h>
 #include <babeltrace/ctf/events.h>
 #include <traceevent/event-parse.h>
 #include "asm/bug.h"
@@ -184,6 +185,7 @@ static int add_tracepoint_field_value(struct ctf_writer *cw,
 	unsigned int len;
 	int ret;
 
+	name = fmtf->alias ? (const char *)fmtf->alias : fmtf->name;
 	offset = fmtf->offset;
 	len = fmtf->size;
 	if (flags & FIELD_IS_STRING)
@@ -567,6 +569,81 @@ static int process_sample_event(struct perf_tool *tool,
 	return cs ? 0 : -1;
 }
 
+/* If dup < 0, add a prefix. Else, add _dupl_X suffix. */
+static char *change_name(char *name, char *orig_name, int dup)
+{
+	char *new_name = NULL;
+	size_t len;
+
+	if (!name)
+		name = orig_name;
+
+	if (dup >= 10)
+		goto out;
+	/*
+	 * Add '_' prefix to potential keywork.  According to
+	 * Mathieu Desnoyers (https://lkml.org/lkml/2015/1/23/652),
+	 * futher CTF spec updating may require us to use '$'.
+	 */
+	if (dup < 0)
+		len = strlen(name) + sizeof("_");
+	else
+		len = strlen(orig_name) + sizeof("_dupl_X");
+
+	new_name = malloc(len);
+	if (!new_name)
+		goto out;
+
+	if (dup < 0)
+		snprintf(new_name, len, "_%s", name);
+	else
+		snprintf(new_name, len, "%s_dupl_%d", orig_name, dup);
+
+out:
+	if (name != orig_name)
+		free(name);
+	return new_name;
+}
+
+static int event_class_add_field(struct bt_ctf_event_class *event_class,
+		struct bt_ctf_field_type *type,
+		struct format_field *field)
+{
+	struct bt_ctf_field_type *t = NULL;
+	char *name;
+	int dup = 1;
+	int ret;
+
+	if (field->alias)
+		return bt_ctf_event_class_add_field(event_class, type,
+				(char *)field->alias);
+
+	name = field->name;
+
+	/* If 'name' is a keywork, add prefix. */
+	if (bt_ctf_validate_identifier(name))
+		name = change_name(name, field->name, -1);
+
+	if (!name) {
+		pr_err("Failed to fix invalid identifier.");
+		return -1;
+	}
+	while ((t = bt_ctf_event_class_get_field_by_name(event_class, name))) {
+		bt_ctf_field_type_put(t);
+		name = change_name(name, field->name, dup++);
+		if (!name) {
+			pr_err("Failed to create dup name for '%s'\n", field->name);
+			return -1;
+		}
+	}
+
+	ret = bt_ctf_event_class_add_field(event_class, type, name);
+	if (!ret)
+		field->alias = name;
+
+	return ret;
+}
+
 static int add_tracepoint_fields_types(struct ctf_writer *cw,
 				       struct format_field *fields,
 				       struct bt_ctf_event_class *event_class)
@@ -595,14 +672,14 @@ static int add_tracepoint_fields_types(struct ctf_writer *cw,
 		if (flags & FIELD_IS_ARRAY)
 			type = bt_ctf_field_type_array_create(type, field->arraylen);
 
-		ret = bt_ctf_event_class_add_field(event_class, type,
-				field->name);
+		ret = event_class_add_field(event_class, type, field);
 
 		if (flags & FIELD_IS_ARRAY)
 			bt_ctf_field_type_put(type);
 
 		if (ret) {
-			pr_err("Failed to add field '%s\n", field->name);
+			pr_err("Failed to add field '%s': %d\n",
+					field->name, ret);
 			return -1;
 		}
 	}
@@ -646,7 +723,7 @@ static int add_generic_types(struct ctf_writer *cw, struct perf_evsel *evsel,
 	do {								\
 		pr2("  field '%s'\n", n);				\
 		if (bt_ctf_event_class_add_field(cl, t, n)) {		\
-			pr_err("Failed to add field '%s;\n", n);	\
+			pr_err("Failed to add field '%s';\n", n);	\
 			return -1;					\
 		}							\
 	} while (0)
-- 
1.9.3

[PATCH 11/11] perf data: Fix signess of value

[Jiri Olsa]

From: Wang Nan <wangnan0@huawei.com>

When converting int values, perf first extractes it to a ulonglong, then
feeds it to babeltrace as a signed value. For negative 32 bit values
(for example, return values of failed syscalls), the extracted data
should be something like 0xfffffffe (-2). It becomes a large int64
value. Babeltrace denies to insert it with
bt_ctf_field_signed_integer_set_value() because it is larger than
0x7fffffff, the largest positive value a signed 32 bit int can be.

This patch introduces adjust_signess(), which fills high bits of
ulonglong with 1 if the value is negative.

Signed-off-by: Wang Nan <wangnan0@huawei.com>
Cc: Arnaldo Carvalho de Melo <acme@redhat.com>
Cc: David Ahern <dsahern@gmail.com>
Cc: Frederic Weisbecker <fweisbec@gmail.com>
Cc: Jeremie Galarneau <jgalar@efficios.com>
Cc: Jiri Olsa <jolsa@kernel.org>
Cc: Namhyung Kim <namhyung@gmail.com>
Cc: Paul Mackerras <paulus@samba.org>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Sebastian Andrzej Siewior <bigeasy@linutronix.de>
Cc: Tom Zanussi <tzanussi@gmail.com>
Signed-off-by: Jiri Olsa <jolsa@kernel.org>
---
 tools/perf/util/data-convert-bt.c | 64 +++++++++++++++++++++++++++++++--------
 1 file changed, 52 insertions(+), 12 deletions(-)

diff --git a/tools/perf/util/data-convert-bt.c b/tools/perf/util/data-convert-bt.c
index 95de04c31a80..827f20262f28 100644
--- a/tools/perf/util/data-convert-bt.c
+++ b/tools/perf/util/data-convert-bt.c
@@ -166,6 +166,43 @@ get_tracepoint_field_type(struct ctf_writer *cw, struct format_field *field)
 		return cw->data.u32;
 }
 
+static unsigned long long adjust_signess(unsigned long long value_int, int size)
+{
+	unsigned long long value_mask;
+
+	/*
+	 * value_mask = (1 << (size * 8 - 1)) - 1.
+	 * Directly set value_mask for code readers.
+	 */
+	switch (size) {
+	case 1:
+		value_mask = 0x7fULL;
+		break;
+	case 2:
+		value_mask = 0x7fffULL;
+		break;
+	case 4:
+		value_mask = 0x7fffffffULL;
+		break;
+	case 8:
+		/*
+		 * For 64 bit value, return it self. There is no need
+		 * to fill high bit.
+		 */
+		/* Fall through */
+	default:
+		/* BUG! */
+		return value_int;
+	}
+
+	/* If it is a positive value, don't adjust. */
+	if ((value_int & (~0ULL - value_mask)) == 0)
+		return value_int;
+
+	/* Fill upper part of value_int with 1 to make it a negative long long. */
+	return (value_int & value_mask) | ~value_mask;
+}
+
 static int add_tracepoint_field_value(struct ctf_writer *cw,
 				      struct bt_ctf_event_class *event_class,
 				      struct bt_ctf_event *event,
@@ -177,7 +214,6 @@ static int add_tracepoint_field_value(struct ctf_writer *cw,
 	struct bt_ctf_field *field;
 	const char *name = fmtf->name;
 	void *data = sample->raw_data;
-	unsigned long long value_int;
 	unsigned long flags = fmtf->flags;
 	unsigned int n_items;
 	unsigned int i;
@@ -222,11 +258,6 @@ static int add_tracepoint_field_value(struct ctf_writer *cw,
 	type = get_tracepoint_field_type(cw, fmtf);
 
 	for (i = 0; i < n_items; i++) {
-		if (!(flags & FIELD_IS_STRING))
-			value_int = pevent_read_number(
-					fmtf->event->pevent,
-					data + offset + i * len, len);
-
 		if (flags & FIELD_IS_ARRAY)
 			field = bt_ctf_field_array_get_field(array_field, i);
 		else
@@ -240,12 +271,21 @@ static int add_tracepoint_field_value(struct ctf_writer *cw,
 		if (flags & FIELD_IS_STRING)
 			ret = bt_ctf_field_string_set_value(field,
 					data + offset + i * len);
-		else if (!(flags & FIELD_IS_SIGNED))
-			ret = bt_ctf_field_unsigned_integer_set_value(
-					field, value_int);
-		else
-			ret = bt_ctf_field_signed_integer_set_value(
-					field, value_int);
+		else {
+			unsigned long long value_int;
+
+			value_int = pevent_read_number(
+					fmtf->event->pevent,
+					data + offset + i * len, len);
+
+			if (!(flags & FIELD_IS_SIGNED))
+				ret = bt_ctf_field_unsigned_integer_set_value(
+						field, value_int);
+			else
+				ret = bt_ctf_field_signed_integer_set_value(
+						field, adjust_signess(value_int, len));
+		}
+
 		if (ret) {
 			pr_err("failed to set file value %s\n", name);
 			goto err_put_field;
-- 
1.9.3

Re: [PATCH 09/11] tools lib traceevent: Add alias field to struct format_field

[Steven Rostedt]

On Fri, Feb 20, 2015 at 11:17:06PM +0100, Jiri Olsa wrote:
> ---
>  tools/lib/traceevent/event-parse.c | 1 +
>  tools/lib/traceevent/event-parse.h | 1 +
>  2 files changed, 2 insertions(+)
> 
> diff --git a/tools/lib/traceevent/event-parse.c b/tools/lib/traceevent/event-parse.c
> index afe20ed9fac8..b35664cc1edc 100644
> --- a/tools/lib/traceevent/event-parse.c
> +++ b/tools/lib/traceevent/event-parse.c
> @@ -6236,6 +6236,7 @@ static void free_format_fields(struct format_field *field)
>  		next = field->next;
>  		free(field->type);
>  		free(field->name);
> +		free(field->alias);

Hmm, didn't we say that if there's no need for the alias that the alias will
simply be a pointer to name? If that's the case we need:

	if (field->alias != field->name)
		free(field->alias);
	free(field->name);

Otherwise it will be a double free.

-- Steve

>  		free(field);
>  		field = next;
>  	}
> diff --git a/tools/lib/traceevent/event-parse.h b/tools/lib/traceevent/event-parse.h
> index 5b4efc062320..8490a4afffa2 100644
> --- a/tools/lib/traceevent/event-parse.h
> +++ b/tools/lib/traceevent/event-parse.h
> @@ -187,6 +187,7 @@ struct format_field {
>  	struct event_format	*event;
>  	char			*type;
>  	char			*name;
> +	char			*alias;
>  	int			offset;
>  	int			size;

Re: [PATCH 09/11] tools lib traceevent: Add alias field to struct format_field

[Jiri Olsa]

On Tue, Feb 24, 2015 at 05:54:58PM -0500, Steven Rostedt wrote:
> On Fri, Feb 20, 2015 at 11:17:06PM +0100, Jiri Olsa wrote:
> > ---
> >  tools/lib/traceevent/event-parse.c | 1 +
> >  tools/lib/traceevent/event-parse.h | 1 +
> >  2 files changed, 2 insertions(+)
> > 
> > diff --git a/tools/lib/traceevent/event-parse.c b/tools/lib/traceevent/event-parse.c
> > index afe20ed9fac8..b35664cc1edc 100644
> > --- a/tools/lib/traceevent/event-parse.c
> > +++ b/tools/lib/traceevent/event-parse.c
> > @@ -6236,6 +6236,7 @@ static void free_format_fields(struct format_field *field)
> >  		next = field->next;
> >  		free(field->type);
> >  		free(field->name);
> > +		free(field->alias);
> 
> Hmm, didn't we say that if there's no need for the alias that the alias will
> simply be a pointer to name? If that's the case we need:
> 
> 	if (field->alias != field->name)
> 		free(field->alias);
> 	free(field->name);
> 
> Otherwise it will be a double free.

i used it separately.. like if there's no alias it's set to NULL,
but I'll add the logic you described and repost, it seems ok

jirka

Re: [PATCH 05/11] perf data: Add tracepoint events fields CTF conversion support

[Arnaldo Carvalho de Melo]

Em Fri, Feb 20, 2015 at 11:17:02PM +0100, Jiri Olsa escreveu:
> From: Sebastian Andrzej Siewior <bigeasy@linutronix.de>
> Adding support to convert tracepoint event fields into CTF
> event fields.
 
> We parse each tracepoint event for CTF conversion and add
> tracepoint fields as regular CTF event fields, so they
> appear in babeltrace output like:
 
>   $ babeltrace ./ctf-data/
>   ...
>   [09:02:00.950703057] (+?.?????????) sched:sched_stat_runtime: { }, { perf_ip = ... SNIP ... common_type = 298, common_flags = 1, \
>   common_preempt_count = 0, common_pid = 31813, comm = "perf", pid = 31813, runtime = 458800, vruntime = 52059858071 }
>   ...

Processed the previous patches, everything ok:

[acme@ssdandy linux]$ ls -la perf.data
ls: cannot access perf.data: No such file or directory
[acme@ssdandy linux]$ trace record usleep 1
[ perf record: Woken up 1 times to write data ]
[ perf record: Captured and wrote 0.029 MB perf.data (88 samples) ]
[acme@ssdandy linux]$ ls -la perf.data
-rw-------. 1 acme acme 5399896 Fev 25 16:19 perf.data
[acme@ssdandy linux]$ perf evlist
raw_syscalls:sys_enter
raw_syscalls:sys_exit
[acme@ssdandy linux]$ perf evlist -v
raw_syscalls:sys_enter: sample_freq=1, type: 2, config: 75, size: 104, sample_type: IP|TID|TIME|ID|CPU|PERIOD|RAW, read_format: ID, disabled: 1, inherit: 1, mmap: 1, mmap2: 1, comm: 1, comm_exec: 1, enable_on_exec: 1, task: 1, sample_id_all: 1, exclude_guest: 1
raw_syscalls:sys_exit: sample_freq=1, type: 2, config: 74, size: 104, sample_type: IP|TID|TIME|ID|CPU|PERIOD|RAW, read_format: ID, disabled: 1, inherit: 1, enable_on_exec: 1, sample_id_all: 1, exclude_guest: 1
[acme@ssdandy linux]$ perf data convert --to-ctf=./ctf-data/
[ perf data convert: Converted 'perf.data' into CTF data './ctf-data/' ]
[ perf data convert: Converted and wrote 0.009 MB (88 samples) ]
[acme@ssdandy linux]$ babeltrace ./ctf-data/
[23:48:47.557933780] (+?.?????????) raw_syscalls:sys_exit: { }, { perf_ip = 0xFFFFFFFF81020FBA, perf_tid = 5093, perf_pid = 5093, perf_id = 1512, perf_cpu = 3, perf_period = 1 }
[23:48:47.557957461] (+0.000023681) raw_syscalls:sys_enter: { }, { perf_ip = 0xFFFFFFFF81020EA6, perf_tid = 5093, perf_pid = 5093, perf_id = 1504, perf_cpu = 3, perf_period = 1 }
[23:48:47.557958406] (+0.000000945) raw_syscalls:sys_exit: { }, { perf_ip = 0xFFFFFFFF81020FBA, perf_tid = 5093, perf_pid = 5093, perf_id = 1512, perf_cpu = 3, perf_period = 1 }
[23:48:47.557973567] (+0.000015161) raw_syscalls:sys_enter: { }, { perf_ip = 0xFFFFFFFF81020EA6, perf_tid = 5093, perf_pid = 5093, perf_id = 1504, perf_cpu = 3, perf_period = 1 }
[23:48:47.557976047] (+0.000002480) raw_syscalls:sys_exit: { }, { perf_ip = 0xFFFFFFFF81020FBA, perf_tid = 5093, perf_pid = 5093, perf_id = 1512, perf_cpu = 3, perf_period = 1 }
[23:48:47.557985774] (+0.000009727) raw_syscalls:sys_enter: { }, { perf_ip = 0xFFFFFFFF81020EA6, perf_tid = 5093, perf_pid = 5093, perf_id = 1504, perf_cpu = 3, perf_period = 1 }
[23:48:47.557990826] (+0.000005052) raw_syscalls:sys_exit: { }, { perf_ip = 0xFFFFFFFF81020FBA, perf_tid = 5093, perf_pid = 5093, perf_id = 1512, perf_cpu = 3, perf_period = 1 }
<SNIP>

But then I apply this patch (convert tracepoint events fields into CTF event fields) and:

[acme@ssdandy linux]$ perf data convert --to-ctf=./ctf-data/
[ perf data convert: Converted 'perf.data' into CTF data './ctf-data/' ]
[ perf data convert: Converted and wrote 0.009 MB (88 samples) ]
[acme@ssdandy linux]$ babeltrace ./ctf-data/
[error] Packet size (18446744073709551615 bits) is larger than remaining file size (262144 bits).
[error] Stream index creation error.
[error] Open file stream error.
[warning] [Context] Cannot open_trace of format ctf at path ./ctf-data.
[warning] [Context] cannot open trace "./ctf-data" from ./ctf-data/ for reading.
[error] Cannot open any trace for reading.

[error] opening trace "./ctf-data/" for reading.

[error] none of the specified trace paths could be opened.

[acme@ssdandy linux]$

It stops working.

[acme@ssdandy linux]$ ls -la ctf-data/
total 44
drwxrwx---.  2 acme acme    41 Fev 25 16:12 .
drwxrwxr-x. 28 acme acme  4096 Fev 25 16:19 ..
-rw-rw----.  1 acme acme  4666 Fev 25 16:21 metadata
-rw-rw----.  1 acme acme 32768 Fev 25 16:21 perf_stream_0
[acme@ssdandy linux]$

Can you try to reproduce this? The ctf-data/metadata file is below:

- Arnaldo

[acme@ssdandy linux]$ cat ctf-data/metadata 
/* CTF 1.8 */

trace {
	major = 1;
	minor = 8;
	uuid = "8b5ba0f6-f29f-4adc-bf12-1fc8e9afec5f";
	byte_order = le;
	packet.header := struct {
		integer { size = 32; align = 8; signed = false; encoding = none; base = decimal; byte_order = le; } magic;
		integer { size = 8; align = 8; signed = false; encoding = none; base = decimal; byte_order = le; } uuid[16];
		integer { size = 32; align = 8; signed = false; encoding = none; base = decimal; byte_order = le; } stream_id;
	} align(8);
};

env {
	host = "ssdandy.ghostprotocols.net";
	sysname = "Linux";
	release = "3.10.0-210.el7.x86_64";
	version = "3.19.rc7.gd15174";
	machine = "x86_64";
	domain = "kernel";
	tracer_name = "perf";
};

clock {
	name = perf_clock;
	uuid = "b2bfe13f-06ed-4ef6-a3cd-fb0f28c6c608";
	description = "perf clock";
	freq = 1000000000;
	precision = 10;
	offset_s = 0;
	offset = 0;
	absolute = FALSE;
};

stream {
	id = 0;
	event.header := struct {
		integer { size = 32; align = 8; signed = false; encoding = none; base = decimal; byte_order = le; } id;
		integer { size = 64; align = 8; signed = false; encoding = none; base = decimal; byte_order = le; map = clock.perf_clock.value; } timestamp;
	} align(8);

	packet.context := struct {
		integer { size = 64; align = 8; signed = false; encoding = none; base = decimal; byte_order = le; } timestamp_begin;
		integer { size = 64; align = 8; signed = false; encoding = none; base = decimal; byte_order = le; } timestamp_end;
		integer { size = 64; align = 8; signed = false; encoding = none; base = decimal; byte_order = le; } content_size;
		integer { size = 64; align = 8; signed = false; encoding = none; base = decimal; byte_order = le; } packet_size;
		integer { size = 64; align = 8; signed = false; encoding = none; base = decimal; byte_order = le; } events_discarded;
	} align(8);
};

event {
	name = "raw_syscalls:sys_enter";
	id = 0;
	stream_id = 0;
	fields := struct {
		integer { size = 64; align = 1; signed = false; encoding = none; base = hexadecimal; byte_order = le; } perf_ip;
		integer { size = 32; align = 1; signed = true; encoding = none; base = decimal; byte_order = le; } perf_tid;
		integer { size = 32; align = 1; signed = true; encoding = none; base = decimal; byte_order = le; } perf_pid;
		integer { size = 64; align = 1; signed = false; encoding = none; base = decimal; byte_order = le; } perf_id;
		integer { size = 32; align = 1; signed = false; encoding = none; base = decimal; byte_order = le; } perf_cpu;
		integer { size = 64; align = 1; signed = false; encoding = none; base = decimal; byte_order = le; } perf_period;
		integer { size = 32; align = 1; signed = false; encoding = none; base = decimal; byte_order = le; } common_type;
		integer { size = 32; align = 1; signed = false; encoding = none; base = decimal; byte_order = le; } common_flags;
		integer { size = 32; align = 1; signed = false; encoding = none; base = decimal; byte_order = le; } common_preempt_count;
		integer { size = 32; align = 1; signed = true; encoding = none; base = decimal; byte_order = le; } common_pid;
		integer { size = 64; align = 1; signed = true; encoding = none; base = decimal; byte_order = le; } id;
		string { encoding = UTF8; } args;
	} align(8);
};

event {
	name = "raw_syscalls:sys_exit";
	id = 1;
	stream_id = 0;
	fields := struct {
		integer { size = 64; align = 1; signed = false; encoding = none; base = hexadecimal; byte_order = le; } perf_ip;
		integer { size = 32; align = 1; signed = true; encoding = none; base = decimal; byte_order = le; } perf_tid;
		integer { size = 32; align = 1; signed = true; encoding = none; base = decimal; byte_order = le; } perf_pid;
		integer { size = 64; align = 1; signed = false; encoding = none; base = decimal; byte_order = le; } perf_id;
		integer { size = 32; align = 1; signed = false; encoding = none; base = decimal; byte_order = le; } perf_cpu;
		integer { size = 64; align = 1; signed = false; encoding = none; base = decimal; byte_order = le; } perf_period;
		integer { size = 32; align = 1; signed = false; encoding = none; base = decimal; byte_order = le; } common_type;
		integer { size = 32; align = 1; signed = false; encoding = none; base = decimal; byte_order = le; } common_flags;
		integer { size = 32; align = 1; signed = false; encoding = none; base = decimal; byte_order = le; } common_preempt_count;
		integer { size = 32; align = 1; signed = true; encoding = none; base = decimal; byte_order = le; } common_pid;
		integer { size = 64; align = 1; signed = true; encoding = none; base = decimal; byte_order = le; } id;
		integer { size = 64; align = 1; signed = true; encoding = none; base = decimal; byte_order = le; } ret;
	} align(1);
};

[acme@ssdandy linux]$

[tip:perf/core] perf tools: Add feature check for libbabeltrace

[tip-bot for Jiri Olsa]

Commit-ID:  53d0a57343949b2af9b27229db534b98e5a0c4d0
Gitweb:     http://git.kernel.org/tip/53d0a57343949b2af9b27229db534b98e5a0c4d0
Author:     Jiri Olsa <jolsa@kernel.org>
AuthorDate: Fri, 20 Feb 2015 23:16:58 +0100
Committer:  Arnaldo Carvalho de Melo <acme@redhat.com>
CommitDate: Wed, 25 Feb 2015 12:42:24 -0300

perf tools: Add feature check for libbabeltrace

Adding feature check for babeltrace library [1], which will be used for
perf data file CTF [2] conversion in following patches.

The babeltrace library is now automatically detected as standard
feature. It's possible to specify LIBBABELTRACE_DIR make variable to
specify location of installed libbabeltrace, like:

  $ make LIBBABELTRACE_DIR=/opt/libbabeltrace/
    BUILD:   Doing 'make -j4' parallel build

  Auto-detecting system features:
  ...                         dwarf: [ on  ]
  ...                         glibc: [ on  ]
  ...                          gtk2: [ on  ]
  ...                      libaudit: [ on  ]
  ...                        libbfd: [ on  ]
  ...                        libelf: [ on  ]
  ...                       libnuma: [ on  ]
  ...                       libperl: [ on  ]
  ...                     libpython: [ on  ]
  ...                      libslang: [ on  ]
  ...                     libunwind: [ on  ]
  ...                 libbabeltrace: [ on  ]
  ...            libdw-dwarf-unwind: [ on  ]
  ...                          zlib: [ on  ]
  ...     DWARF post unwind library: libunwind

NOTE The installation of the [1] to to used by above make:
     $ git clone git://git.efficios.com/babeltrace.git
     $ cd babeltrace
     $ vim README
     $ ./bootstrap
     $ ./configure --prefix=/opt/libbabeltrace
     $ make prefix=/opt/libbabeltrace
     $ sudo make install prefix=/opt/libbabeltrace

Please make sure that the /opt/libbabeltrace/lib directory is in your
LD_LIBRARY_PATH:

 $ export LD_LIBRARY_PATH=/opt/libbabeltrace/lib

[1] babeltrace - http://www.efficios.com/babeltrace
[2] Common Trace Format - http://www.efficios.com/ctf

Signed-off-by: Jiri Olsa <jolsa@kernel.org>
Acked-by: Namhyung Kim <namhyung@kernel.org>
Reviewed-by: David Ahern <dsahern@gmail.com>
Cc: Frederic Weisbecker <fweisbec@gmail.com>
Cc: Jeremie Galarneau <jgalar@efficios.com>
Cc: Paul Mackerras <paulus@samba.org>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Sebastian Andrzej Siewior <bigeasy@linutronix.de>
Cc: Tom Zanussi <tzanussi@gmail.com>
Cc: Wang Nan <wangnan0@huawei.com>
Link: http://lkml.kernel.org/r/1424470628-5969-2-git-send-email-jolsa@kernel.org
Signed-off-by: Sebastian Andrzej Siewior <bigeasy@linutronix.de>
[ Added missing babeltrace build instructions ]
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
---
 tools/perf/Makefile.perf                           |  4 +++-
 tools/perf/config/Makefile                         | 24 ++++++++++++++++++++++
 tools/perf/config/feature-checks/Makefile          |  8 ++++++--
 tools/perf/config/feature-checks/test-all.c        |  5 +++++
 .../config/feature-checks/test-libbabeltrace.c     |  8 ++++++++
 5 files changed, 46 insertions(+), 3 deletions(-)

diff --git a/tools/perf/Makefile.perf b/tools/perf/Makefile.perf
index efc5158..ec4c063 100644
--- a/tools/perf/Makefile.perf
+++ b/tools/perf/Makefile.perf
@@ -68,7 +68,9 @@ include config/utilities.mak
 # for reading the x32 mode 32-bit compatibility VDSO in 64-bit mode
 #
 # Define NO_ZLIB if you do not want to support compressed kernel modules
-
+#
+# Define NO_LIBBABELTRACE if you do not want libbabeltrace support
+# for CTF data format.
 
 ifeq ($(srctree),)
 srctree := $(patsubst %/,%,$(dir $(shell pwd)))
diff --git a/tools/perf/config/Makefile b/tools/perf/config/Makefile
index b97a7b9..6f129b0 100644
--- a/tools/perf/config/Makefile
+++ b/tools/perf/config/Makefile
@@ -96,6 +96,17 @@ ifndef NO_LIBELF
   FEATURE_CHECK_LDFLAGS-libdw-dwarf-unwind := $(LIBDW_LDFLAGS) -ldw
 endif
 
+ifndef NO_LIBBABELTRACE
+  # for linking with debug library, run like:
+  # make DEBUG=1 LIBBABELTRACE_DIR=/opt/libbabeltrace/
+  ifdef LIBBABELTRACE_DIR
+    LIBBABELTRACE_CFLAGS  := -I$(LIBBABELTRACE_DIR)/include
+    LIBBABELTRACE_LDFLAGS := -L$(LIBBABELTRACE_DIR)/lib
+  endif
+  FEATURE_CHECK_CFLAGS-libbabeltrace := $(LIBBABELTRACE_CFLAGS)
+  FEATURE_CHECK_LDFLAGS-libbabeltrace := $(LIBBABELTRACE_LDFLAGS) -lbabeltrace-ctf
+endif
+
 # include ARCH specific config
 -include $(src-perf)/arch/$(ARCH)/Makefile
 
@@ -216,6 +227,7 @@ CORE_FEATURE_TESTS =			\
 	stackprotector-all		\
 	timerfd				\
 	libdw-dwarf-unwind		\
+	libbabeltrace			\
 	zlib
 
 LIB_FEATURE_TESTS =			\
@@ -231,6 +243,7 @@ LIB_FEATURE_TESTS =			\
 	libslang			\
 	libunwind			\
 	libdw-dwarf-unwind		\
+	libbabeltrace			\
 	zlib
 
 VF_FEATURE_TESTS =			\
@@ -692,6 +705,17 @@ else
   NO_PERF_READ_VDSOX32 := 1
 endif
 
+ifndef NO_LIBBABELTRACE
+  ifeq ($(feature-libbabeltrace), 0)
+    msg := $(warning No libbabeltrace found, disables 'perf data' CTF format support, please install libbabeltrace-devel/libbabeltrace-ctf-dev);
+    NO_LIBBABELTRACE := 1
+  else
+    CFLAGS += -DHAVE_LIBBABELTRACE_SUPPORT $(LIBBABELTRACE_CFLAGS)
+    LDFLAGS += $(LIBBABELTRACE_LDFLAGS)
+    EXTLIBS += -lbabeltrace-ctf
+  endif
+endif
+
 # Among the variables below, these:
 #   perfexecdir
 #   template_dir
diff --git a/tools/perf/config/feature-checks/Makefile b/tools/perf/config/feature-checks/Makefile
index b32ff33..70c9aeb 100644
--- a/tools/perf/config/feature-checks/Makefile
+++ b/tools/perf/config/feature-checks/Makefile
@@ -29,6 +29,7 @@ FILES=					\
 	test-stackprotector-all.bin	\
 	test-timerfd.bin		\
 	test-libdw-dwarf-unwind.bin	\
+	test-libbabeltrace.bin		\
 	test-compile-32.bin		\
 	test-compile-x32.bin		\
 	test-zlib.bin
@@ -43,7 +44,7 @@ BUILD = $(CC) $(CFLAGS) -o $(OUTPUT)$@ $(patsubst %.bin,%.c,$@) $(LDFLAGS)
 ###############################
 
 test-all.bin:
-	$(BUILD) -Werror -fstack-protector-all -O2 -Werror -D_FORTIFY_SOURCE=2 -ldw -lelf -lnuma -lelf -laudit -I/usr/include/slang -lslang $(shell $(PKG_CONFIG) --libs --cflags gtk+-2.0 2>/dev/null) $(FLAGS_PERL_EMBED) $(FLAGS_PYTHON_EMBED) -DPACKAGE='"perf"' -lbfd -ldl -lz
+	$(BUILD) -Werror -fstack-protector-all -O2 -Werror -D_FORTIFY_SOURCE=2 -ldw -lelf -lnuma -lelf -laudit -I/usr/include/slang -lslang $(shell $(PKG_CONFIG) --libs --cflags gtk+-2.0 2>/dev/null) $(FLAGS_PERL_EMBED) $(FLAGS_PYTHON_EMBED) -DPACKAGE='"perf"' -lbfd -ldl -lz -lbabeltrace
 
 test-hello.bin:
 	$(BUILD)
@@ -133,7 +134,10 @@ test-timerfd.bin:
 	$(BUILD)
 
 test-libdw-dwarf-unwind.bin:
-	$(BUILD)
+	$(BUILD) # -ldw provided by $(FEATURE_CHECK_LDFLAGS-libdw-dwarf-unwind)
+
+test-libbabeltrace.bin:
+	$(BUILD) # -lbabeltrace provided by $(FEATURE_CHECK_LDFLAGS-libbabeltrace)
 
 test-sync-compare-and-swap.bin:
 	$(BUILD) -Werror
diff --git a/tools/perf/config/feature-checks/test-all.c b/tools/perf/config/feature-checks/test-all.c
index 6d4d093..1ffc3da 100644
--- a/tools/perf/config/feature-checks/test-all.c
+++ b/tools/perf/config/feature-checks/test-all.c
@@ -101,6 +101,10 @@
 # include "test-pthread_attr_setaffinity_np.c"
 #undef main
 
+#define main main_test_libbabeltrace
+# include "test-libbabeltrace.c"
+#undef main
+
 int main(int argc, char *argv[])
 {
 	main_test_libpython();
@@ -126,6 +130,7 @@ int main(int argc, char *argv[])
 	main_test_sync_compare_and_swap(argc, argv);
 	main_test_zlib();
 	main_test_pthread_attr_setaffinity_np();
+	main_test_libbabeltrace();
 
 	return 0;
 }
diff --git a/tools/perf/config/feature-checks/test-libbabeltrace.c b/tools/perf/config/feature-checks/test-libbabeltrace.c
new file mode 100644
index 0000000..3b7dd68
--- /dev/null
+++ b/tools/perf/config/feature-checks/test-libbabeltrace.c
@@ -0,0 +1,8 @@
+
+#include <babeltrace/ctf-writer/writer.h>
+
+int main(void)
+{
+	bt_ctf_stream_class_get_packet_context_type((void *) 0);
+	return 0;
+}

[tip:perf/core] perf tools: Add new 'perf data' command

[tip-bot for Jiri Olsa]

Commit-ID:  2245bf1410d2d719f3bfce729b07ab83fe6142f7
Gitweb:     http://git.kernel.org/tip/2245bf1410d2d719f3bfce729b07ab83fe6142f7
Author:     Jiri Olsa <jolsa@kernel.org>
AuthorDate: Fri, 20 Feb 2015 23:16:59 +0100
Committer:  Arnaldo Carvalho de Melo <acme@redhat.com>
CommitDate: Wed, 25 Feb 2015 12:42:25 -0300

perf tools: Add new 'perf data' command

Adding new 'perf data' command to provide operations over data files.

The 'perf data convert' sub command is coming in following patch, but
there's possibility for other useful commands like 'perf data ls' (to
display perf data file in directory in ls style).

Signed-off-by: Jiri Olsa <jolsa@kernel.org>
Acked-by: Namhyung Kim <namhyung@kernel.org>
Reviewed-by: David Ahern <dsahern@gmail.com>
Cc: Frederic Weisbecker <fweisbec@gmail.com>
Cc: Jeremie Galarneau <jgalar@efficios.com>
Cc: Paul Mackerras <paulus@samba.org>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Sebastian Andrzej Siewior <bigeasy@linutronix.de>
Cc: Tom Zanussi <tzanussi@gmail.com>
Cc: Wang Nan <wangnan0@huawei.com>
Link: http://lkml.kernel.org/r/1424470628-5969-3-git-send-email-jolsa@kernel.org
Signed-off-by: Sebastian Andrzej Siewior <bigeasy@linutronix.de>
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
---
 tools/perf/Build                       |  1 +
 tools/perf/Documentation/perf-data.txt | 15 +++++++
 tools/perf/builtin-data.c              | 75 ++++++++++++++++++++++++++++++++++
 tools/perf/builtin.h                   |  1 +
 tools/perf/command-list.txt            |  1 +
 tools/perf/perf.c                      |  1 +
 6 files changed, 94 insertions(+)

diff --git a/tools/perf/Build b/tools/perf/Build
index 976e038..b77370e 100644
--- a/tools/perf/Build
+++ b/tools/perf/Build
@@ -18,6 +18,7 @@ perf-y += builtin-lock.o
 perf-y += builtin-kvm.o
 perf-y += builtin-inject.o
 perf-y += builtin-mem.o
+perf-y += builtin-data.o
 
 perf-$(CONFIG_AUDIT) += builtin-trace.o
 perf-$(CONFIG_LIBELF) += builtin-probe.o
diff --git a/tools/perf/Documentation/perf-data.txt b/tools/perf/Documentation/perf-data.txt
new file mode 100644
index 0000000..b8c8394
--- /dev/null
+++ b/tools/perf/Documentation/perf-data.txt
@@ -0,0 +1,15 @@
+perf-data(1)
+==============
+
+NAME
+----
+perf-data - Data file related processing
+
+SYNOPSIS
+--------
+[verse]
+'perf data' [<common options>] <command> [<options>]",
+
+DESCRIPTION
+-----------
+Data file related processing.
diff --git a/tools/perf/builtin-data.c b/tools/perf/builtin-data.c
new file mode 100644
index 0000000..1eee97d
--- /dev/null
+++ b/tools/perf/builtin-data.c
@@ -0,0 +1,75 @@
+#include <linux/compiler.h>
+#include "builtin.h"
+#include "perf.h"
+#include "debug.h"
+#include "parse-options.h"
+
+typedef int (*data_cmd_fn_t)(int argc, const char **argv, const char *prefix);
+
+struct data_cmd {
+	const char	*name;
+	const char	*summary;
+	data_cmd_fn_t	fn;
+};
+
+static struct data_cmd data_cmds[];
+
+#define for_each_cmd(cmd) \
+	for (cmd = data_cmds; cmd && cmd->name; cmd++)
+
+static const struct option data_options[] = {
+	OPT_END()
+};
+
+static const char * const data_usage[] = {
+	"perf data [<common options>] <command> [<options>]",
+	NULL
+};
+
+static void print_usage(void)
+{
+	struct data_cmd *cmd;
+
+	printf("Usage:\n");
+	printf("\t%s\n\n", data_usage[0]);
+	printf("\tAvailable commands:\n");
+
+	for_each_cmd(cmd) {
+		printf("\t %s\t- %s\n", cmd->name, cmd->summary);
+	}
+
+	printf("\n");
+}
+
+static struct data_cmd data_cmds[] = {
+	{ NULL },
+};
+
+int cmd_data(int argc, const char **argv, const char *prefix)
+{
+	struct data_cmd *cmd;
+	const char *cmdstr;
+
+	/* No command specified. */
+	if (argc < 2)
+		goto usage;
+
+	argc = parse_options(argc, argv, data_options, data_usage,
+			     PARSE_OPT_STOP_AT_NON_OPTION);
+	if (argc < 1)
+		goto usage;
+
+	cmdstr = argv[0];
+
+	for_each_cmd(cmd) {
+		if (strcmp(cmd->name, cmdstr))
+			continue;
+
+		return cmd->fn(argc, argv, prefix);
+	}
+
+	pr_err("Unknown command: %s\n", cmdstr);
+usage:
+	print_usage();
+	return -1;
+}
diff --git a/tools/perf/builtin.h b/tools/perf/builtin.h
index b210d62..3688ad2 100644
--- a/tools/perf/builtin.h
+++ b/tools/perf/builtin.h
@@ -37,6 +37,7 @@ extern int cmd_test(int argc, const char **argv, const char *prefix);
 extern int cmd_trace(int argc, const char **argv, const char *prefix);
 extern int cmd_inject(int argc, const char **argv, const char *prefix);
 extern int cmd_mem(int argc, const char **argv, const char *prefix);
+extern int cmd_data(int argc, const char **argv, const char *prefix);
 
 extern int find_scripts(char **scripts_array, char **scripts_path_array);
 #endif
diff --git a/tools/perf/command-list.txt b/tools/perf/command-list.txt
index 0906fc4..00fcaf8 100644
--- a/tools/perf/command-list.txt
+++ b/tools/perf/command-list.txt
@@ -7,6 +7,7 @@ perf-archive			mainporcelain common
 perf-bench			mainporcelain common
 perf-buildid-cache		mainporcelain common
 perf-buildid-list		mainporcelain common
+perf-data			mainporcelain common
 perf-diff			mainporcelain common
 perf-evlist			mainporcelain common
 perf-inject			mainporcelain common
diff --git a/tools/perf/perf.c b/tools/perf/perf.c
index 3700a7f..f3c66b8 100644
--- a/tools/perf/perf.c
+++ b/tools/perf/perf.c
@@ -62,6 +62,7 @@ static struct cmd_struct commands[] = {
 #endif
 	{ "inject",	cmd_inject,	0 },
 	{ "mem",	cmd_mem,	0 },
+	{ "data",	cmd_data,	0 },
 };
 
 struct pager_config {

[tip:perf/core] perf data: Add perf data to CTF conversion support

[tip-bot for Jiri Olsa]

Commit-ID:  edbe9817aeb540aa1494aa20276a2bfc7f4ab816
Gitweb:     http://git.kernel.org/tip/edbe9817aeb540aa1494aa20276a2bfc7f4ab816
Author:     Jiri Olsa <jolsa@kernel.org>
AuthorDate: Fri, 20 Feb 2015 23:17:00 +0100
Committer:  Arnaldo Carvalho de Melo <acme@redhat.com>
CommitDate: Wed, 25 Feb 2015 16:13:12 -0300

perf data: Add perf data to CTF conversion support

Adding 'perf data convert' to convert perf data file into different
format. This patch adds support for CTF format conversion.

To convert perf.data into CTF run:
  $ perf data convert --to-ctf=./ctf-data/
  [ perf data convert: Converted 'perf.data' into CTF data './ctf-data/' ]
  [ perf data convert: Converted and wrote 11.268 MB (100230 samples) ]

The command will create CTF metadata out of perf.data file (or one
specified via -i option) and then convert all sample events into single
CTF stream.

Each sample_type bit is translated into separated CTF event field apart
from following exceptions:

  PERF_SAMPLE_RAW          - added in next patch
  PERF_SAMPLE_READ         - TODO
  PERF_SAMPLE_CALLCHAIN    - TODO
  PERF_SAMPLE_BRANCH_STACK - TODO
  PERF_SAMPLE_REGS_USER    - TODO
  PERF_SAMPLE_STACK_USER   - TODO

  $ perf --debug=data-convert=2 data convert ...

The converted CTF data could be analyzed by CTF tools, like babletrace
or tracecompass [1].

  $ babeltrace ./ctf-data/
  [03:19:13.962125533] (+?.?????????) cycles: { }, { ip = 0xFFFFFFFF8105443A, tid = 20714, pid = 20714, period = 1 }
  [03:19:13.962130001] (+0.000004468) cycles: { }, { ip = 0xFFFFFFFF8105443A, tid = 20714, pid = 20714, period = 1 }
  [03:19:13.962131936] (+0.000001935) cycles: { }, { ip = 0xFFFFFFFF8105443A, tid = 20714, pid = 20714, period = 8 }
  [03:19:13.962133732] (+0.000001796) cycles: { }, { ip = 0xFFFFFFFF8105443A, tid = 20714, pid = 20714, period = 114 }
  [03:19:13.962135557] (+0.000001825) cycles: { }, { ip = 0xFFFFFFFF8105443A, tid = 20714, pid = 20714, period = 2087 }
  [03:19:13.962137627] (+0.000002070) cycles: { }, { ip = 0xFFFFFFFF81361938, tid = 20714, pid = 20714, period = 37582 }
  [03:19:13.962161091] (+0.000023464) cycles: { }, { ip = 0xFFFFFFFF8124218F, tid = 20714, pid = 20714, period = 600246 }
  [03:19:13.962517569] (+0.000356478) cycles: { }, { ip = 0xFFFFFFFF811A75DB, tid = 20714, pid = 20714, period = 1325731 }
  [03:19:13.969518008] (+0.007000439) cycles: { }, { ip = 0x34080917B2, tid = 20714, pid = 20714, period = 1144298 }

The following members to the ctf-environment were decided to be added to
distinguish and specify perf CTF data:

  - domain

    It says "kernel" because it contains a kernel trace (not to be
    confused with a user space like lttng-ust does)

  - tracer_name

    It says perf. This can be used to distinguish between lttng and perf
    CTF based trace.

  - version

    The kernel version from stream. In addition to release, this is what
    it looks like on a Debian kernel:

      release = "3.14-1-amd64";
      version = "3.14.0";

[1] http://projects.eclipse.org/projects/tools.tracecompass

Signed-off-by: Jiri Olsa <jolsa@kernel.org>
Acked-by: Namhyung Kim <namhyung@kernel.org>
Reviewed-by: David Ahern <dsahern@gmail.com>
Tested-by: Arnaldo Carvalho de Melo <acme@redhat.com>
Cc: Frederic Weisbecker <fweisbec@gmail.com>
Cc: Jeremie Galarneau <jgalar@efficios.com>
Cc: Paul Mackerras <paulus@samba.org>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Sebastian Andrzej Siewior <bigeasy@linutronix.de>
Cc: Tom Zanussi <tzanussi@gmail.com>
Cc: Wang Nan <wangnan0@huawei.com>
Link: http://lkml.kernel.org/r/1424470628-5969-4-git-send-email-jolsa@kernel.org
Signed-off-by: Sebastian Andrzej Siewior <bigeasy@linutronix.de>
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
---
 tools/perf/Documentation/perf-data.txt |  25 ++
 tools/perf/Documentation/perf.txt      |   7 +-
 tools/perf/builtin-data.c              |  44 +++
 tools/perf/config/Makefile             |   1 +
 tools/perf/util/Build                  |   2 +
 tools/perf/util/data-convert-bt.c      | 612 +++++++++++++++++++++++++++++++++
 tools/perf/util/data-convert-bt.h      |   8 +
 tools/perf/util/debug.c                |   2 +
 tools/perf/util/debug.h                |   1 +
 9 files changed, 701 insertions(+), 1 deletion(-)

diff --git a/tools/perf/Documentation/perf-data.txt b/tools/perf/Documentation/perf-data.txt
index b8c8394..be8fa1a 100644
--- a/tools/perf/Documentation/perf-data.txt
+++ b/tools/perf/Documentation/perf-data.txt
@@ -13,3 +13,28 @@ SYNOPSIS
 DESCRIPTION
 -----------
 Data file related processing.
+
+COMMANDS
+--------
+convert::
+	Converts perf data file into another format (only CTF [1] format is support by now).
+	It's possible to set data-convert debug variable to get debug messages from conversion,
+	like:
+	  perf --debug data-convert data convert ...
+
+OPTIONS for 'convert'
+---------------------
+--to-ctf::
+	Triggers the CTF conversion, specify the path of CTF data directory.
+
+-i::
+	Specify input perf data file path.
+
+-v::
+--verbose::
+        Be more verbose (show counter open errors, etc).
+
+SEE ALSO
+--------
+linkperf:perf[1]
+[1] Common Trace Format - http://www.efficios.com/ctf
diff --git a/tools/perf/Documentation/perf.txt b/tools/perf/Documentation/perf.txt
index 1e8e400..2b13177 100644
--- a/tools/perf/Documentation/perf.txt
+++ b/tools/perf/Documentation/perf.txt
@@ -13,11 +13,16 @@ SYNOPSIS
 OPTIONS
 -------
 --debug::
-	Setup debug variable (just verbose for now) in value
+	Setup debug variable (see list below) in value
 	range (0, 10). Use like:
 	  --debug verbose   # sets verbose = 1
 	  --debug verbose=2 # sets verbose = 2
 
+	List of debug variables allowed to set:
+	  verbose          - general debug messages
+	  ordered-events   - ordered events object debug messages
+	  data-convert     - data convert command debug messages
+
 --buildid-dir::
 	Setup buildid cache directory. It has higher priority than
 	buildid.dir config file option.
diff --git a/tools/perf/builtin-data.c b/tools/perf/builtin-data.c
index 1eee97d..9705ba7 100644
--- a/tools/perf/builtin-data.c
+++ b/tools/perf/builtin-data.c
@@ -3,6 +3,7 @@
 #include "perf.h"
 #include "debug.h"
 #include "parse-options.h"
+#include "data-convert-bt.h"
 
 typedef int (*data_cmd_fn_t)(int argc, const char **argv, const char *prefix);
 
@@ -41,7 +42,50 @@ static void print_usage(void)
 	printf("\n");
 }
 
+static const char * const data_convert_usage[] = {
+	"perf data convert [<options>]",
+	NULL
+};
+
+static int cmd_data_convert(int argc, const char **argv,
+			    const char *prefix __maybe_unused)
+{
+	const char *to_ctf     = NULL;
+	const struct option options[] = {
+		OPT_INCR('v', "verbose", &verbose, "be more verbose"),
+		OPT_STRING('i', "input", &input_name, "file", "input file name"),
+#ifdef HAVE_LIBBABELTRACE_SUPPORT
+		OPT_STRING(0, "to-ctf", &to_ctf, NULL, "Convert to CTF format"),
+#endif
+		OPT_END()
+	};
+
+#ifndef HAVE_LIBBABELTRACE_SUPPORT
+	pr_err("No conversion support compiled in.\n");
+	return -1;
+#endif
+
+	argc = parse_options(argc, argv, options,
+			     data_convert_usage, 0);
+	if (argc) {
+		usage_with_options(data_convert_usage, options);
+		return -1;
+	}
+
+	if (to_ctf) {
+#ifdef HAVE_LIBBABELTRACE_SUPPORT
+		return bt_convert__perf2ctf(input_name, to_ctf);
+#else
+		pr_err("The libbabeltrace support is not compiled in.\n");
+		return -1;
+#endif
+	}
+
+	return 0;
+}
+
 static struct data_cmd data_cmds[] = {
+	{ "convert", "converts data file between formats", cmd_data_convert },
 	{ NULL },
 };
 
diff --git a/tools/perf/config/Makefile b/tools/perf/config/Makefile
index 6f129b0..c3570b5 100644
--- a/tools/perf/config/Makefile
+++ b/tools/perf/config/Makefile
@@ -713,6 +713,7 @@ ifndef NO_LIBBABELTRACE
     CFLAGS += -DHAVE_LIBBABELTRACE_SUPPORT $(LIBBABELTRACE_CFLAGS)
     LDFLAGS += $(LIBBABELTRACE_LDFLAGS)
     EXTLIBS += -lbabeltrace-ctf
+    $(call detected,CONFIG_LIBBABELTRACE)
   endif
 endif
 
diff --git a/tools/perf/util/Build b/tools/perf/util/Build
index 32f9327..a2c8047 100644
--- a/tools/perf/util/Build
+++ b/tools/perf/util/Build
@@ -88,6 +88,8 @@ libperf-$(CONFIG_DWARF) += dwarf-aux.o
 libperf-$(CONFIG_LIBDW_DWARF_UNWIND) += unwind-libdw.o
 libperf-$(CONFIG_LIBUNWIND)          += unwind-libunwind.o
 
+libperf-$(CONFIG_LIBBABELTRACE) += data-convert-bt.o
+
 libperf-y += scripting-engines/
 
 libperf-$(CONFIG_PERF_REGS) += perf_regs.o
diff --git a/tools/perf/util/data-convert-bt.c b/tools/perf/util/data-convert-bt.c
new file mode 100644
index 0000000..ff4826c
--- /dev/null
+++ b/tools/perf/util/data-convert-bt.c
@@ -0,0 +1,612 @@
+/*
+ * CTF writing support via babeltrace.
+ *
+ * Copyright (C) 2014, Jiri Olsa <jolsa@redhat.com>
+ * Copyright (C) 2014, Sebastian Andrzej Siewior <bigeasy@linutronix.de>
+ *
+ * Released under the GPL v2. (and only v2, not any later version)
+ */
+
+#include <linux/compiler.h>
+#include <babeltrace/ctf-writer/writer.h>
+#include <babeltrace/ctf-writer/clock.h>
+#include <babeltrace/ctf-writer/stream.h>
+#include <babeltrace/ctf-writer/event.h>
+#include <babeltrace/ctf-writer/event-types.h>
+#include <babeltrace/ctf-writer/event-fields.h>
+#include <babeltrace/ctf/events.h>
+#include <traceevent/event-parse.h>
+#include "asm/bug.h"
+#include "data-convert-bt.h"
+#include "session.h"
+#include "util.h"
+#include "debug.h"
+#include "tool.h"
+#include "evlist.h"
+#include "evsel.h"
+#include "machine.h"
+
+#define pr_N(n, fmt, ...) \
+	eprintf(n, debug_data_convert, fmt, ##__VA_ARGS__)
+
+#define pr(fmt, ...)  pr_N(1, pr_fmt(fmt), ##__VA_ARGS__)
+#define pr2(fmt, ...) pr_N(2, pr_fmt(fmt), ##__VA_ARGS__)
+
+#define pr_time2(t, fmt, ...) pr_time_N(2, debug_data_convert, t, pr_fmt(fmt), ##__VA_ARGS__)
+
+struct evsel_priv {
+	struct bt_ctf_event_class *event_class;
+};
+
+struct ctf_writer {
+	/* writer primitives */
+	struct bt_ctf_writer		*writer;
+	struct bt_ctf_stream		*stream;
+	struct bt_ctf_stream_class	*stream_class;
+	struct bt_ctf_clock		*clock;
+
+	/* data types */
+	union {
+		struct {
+			struct bt_ctf_field_type	*s64;
+			struct bt_ctf_field_type	*u64;
+			struct bt_ctf_field_type	*s32;
+			struct bt_ctf_field_type	*u32;
+			struct bt_ctf_field_type	*string;
+			struct bt_ctf_field_type	*u64_hex;
+		};
+		struct bt_ctf_field_type *array[6];
+	} data;
+};
+
+struct convert {
+	struct perf_tool	tool;
+	struct ctf_writer	writer;
+
+	u64			events_size;
+	u64			events_count;
+};
+
+static int value_set(struct bt_ctf_field_type *type,
+		     struct bt_ctf_event *event,
+		     const char *name, u64 val)
+{
+	struct bt_ctf_field *field;
+	bool sign = bt_ctf_field_type_integer_get_signed(type);
+	int ret;
+
+	field = bt_ctf_field_create(type);
+	if (!field) {
+		pr_err("failed to create a field %s\n", name);
+		return -1;
+	}
+
+	if (sign) {
+		ret = bt_ctf_field_signed_integer_set_value(field, val);
+		if (ret) {
+			pr_err("failed to set field value %s\n", name);
+			goto err;
+		}
+	} else {
+		ret = bt_ctf_field_unsigned_integer_set_value(field, val);
+		if (ret) {
+			pr_err("failed to set field value %s\n", name);
+			goto err;
+		}
+	}
+
+	ret = bt_ctf_event_set_payload(event, name, field);
+	if (ret) {
+		pr_err("failed to set payload %s\n", name);
+		goto err;
+	}
+
+	pr2("  SET [%s = %" PRIu64 "]\n", name, val);
+
+err:
+	bt_ctf_field_put(field);
+	return ret;
+}
+
+#define __FUNC_VALUE_SET(_name, _val_type)				\
+static __maybe_unused int value_set_##_name(struct ctf_writer *cw,	\
+			     struct bt_ctf_event *event,		\
+			     const char *name,				\
+			     _val_type val)				\
+{									\
+	struct bt_ctf_field_type *type = cw->data._name;		\
+	return value_set(type, event, name, (u64) val);			\
+}
+
+#define FUNC_VALUE_SET(_name) __FUNC_VALUE_SET(_name, _name)
+
+FUNC_VALUE_SET(s32)
+FUNC_VALUE_SET(u32)
+FUNC_VALUE_SET(s64)
+FUNC_VALUE_SET(u64)
+__FUNC_VALUE_SET(u64_hex, u64)
+
+static int add_generic_values(struct ctf_writer *cw,
+			      struct bt_ctf_event *event,
+			      struct perf_evsel *evsel,
+			      struct perf_sample *sample)
+{
+	u64 type = evsel->attr.sample_type;
+	int ret;
+
+	/*
+	 * missing:
+	 *   PERF_SAMPLE_TIME         - not needed as we have it in
+	 *                              ctf event header
+	 *   PERF_SAMPLE_READ         - TODO
+	 *   PERF_SAMPLE_CALLCHAIN    - TODO
+	 *   PERF_SAMPLE_RAW          - tracepoint fields are handled separately
+	 *   PERF_SAMPLE_BRANCH_STACK - TODO
+	 *   PERF_SAMPLE_REGS_USER    - TODO
+	 *   PERF_SAMPLE_STACK_USER   - TODO
+	 */
+
+	if (type & PERF_SAMPLE_IP) {
+		ret = value_set_u64_hex(cw, event, "ip", sample->ip);
+		if (ret)
+			return -1;
+	}
+
+	if (type & PERF_SAMPLE_TID) {
+		ret = value_set_s32(cw, event, "tid", sample->tid);
+		if (ret)
+			return -1;
+
+		ret = value_set_s32(cw, event, "pid", sample->pid);
+		if (ret)
+			return -1;
+	}
+
+	if ((type & PERF_SAMPLE_ID) ||
+	    (type & PERF_SAMPLE_IDENTIFIER)) {
+		ret = value_set_u64(cw, event, "id", sample->id);
+		if (ret)
+			return -1;
+	}
+
+	if (type & PERF_SAMPLE_STREAM_ID) {
+		ret = value_set_u64(cw, event, "stream_id", sample->stream_id);
+		if (ret)
+			return -1;
+	}
+
+	if (type & PERF_SAMPLE_CPU) {
+		ret = value_set_u32(cw, event, "cpu", sample->cpu);
+		if (ret)
+			return -1;
+	}
+
+	if (type & PERF_SAMPLE_PERIOD) {
+		ret = value_set_u64(cw, event, "period", sample->period);
+		if (ret)
+			return -1;
+	}
+
+	if (type & PERF_SAMPLE_WEIGHT) {
+		ret = value_set_u64(cw, event, "weight", sample->weight);
+		if (ret)
+			return -1;
+	}
+
+	if (type & PERF_SAMPLE_DATA_SRC) {
+		ret = value_set_u64(cw, event, "data_src", sample->data_src);
+		if (ret)
+			return -1;
+	}
+
+	if (type & PERF_SAMPLE_TRANSACTION) {
+		ret = value_set_u64(cw, event, "transaction", sample->transaction);
+		if (ret)
+			return -1;
+	}
+
+	return 0;
+}
+
+static int process_sample_event(struct perf_tool *tool,
+				union perf_event *_event __maybe_unused,
+				struct perf_sample *sample,
+				struct perf_evsel *evsel,
+				struct machine *machine __maybe_unused)
+{
+	struct convert *c = container_of(tool, struct convert, tool);
+	struct evsel_priv *priv = evsel->priv;
+	struct ctf_writer *cw = &c->writer;
+	struct bt_ctf_event_class *event_class;
+	struct bt_ctf_event *event;
+	int ret;
+
+	if (WARN_ONCE(!priv, "Failed to setup all events.\n"))
+		return 0;
+
+	event_class = priv->event_class;
+
+	/* update stats */
+	c->events_count++;
+	c->events_size += _event->header.size;
+
+	pr_time2(sample->time, "sample %" PRIu64 "\n", c->events_count);
+
+	event = bt_ctf_event_create(event_class);
+	if (!event) {
+		pr_err("Failed to create an CTF event\n");
+		return -1;
+	}
+
+	bt_ctf_clock_set_time(cw->clock, sample->time);
+
+	ret = add_generic_values(cw, event, evsel, sample);
+	if (ret)
+		return -1;
+
+	bt_ctf_stream_append_event(cw->stream, event);
+	bt_ctf_event_put(event);
+	return 0;
+}
+
+static int add_generic_types(struct ctf_writer *cw, struct perf_evsel *evsel,
+			     struct bt_ctf_event_class *event_class)
+{
+	u64 type = evsel->attr.sample_type;
+
+	/*
+	 * missing:
+	 *   PERF_SAMPLE_TIME         - not needed as we have it in
+	 *                              ctf event header
+	 *   PERF_SAMPLE_READ         - TODO
+	 *   PERF_SAMPLE_CALLCHAIN    - TODO
+	 *   PERF_SAMPLE_RAW          - tracepoint fields are handled separately
+	 *   PERF_SAMPLE_BRANCH_STACK - TODO
+	 *   PERF_SAMPLE_REGS_USER    - TODO
+	 *   PERF_SAMPLE_STACK_USER   - TODO
+	 */
+
+#define ADD_FIELD(cl, t, n)						\
+	do {								\
+		pr2("  field '%s'\n", n);				\
+		if (bt_ctf_event_class_add_field(cl, t, n)) {		\
+			pr_err("Failed to add field '%s;\n", n);	\
+			return -1;					\
+		}							\
+	} while (0)
+
+	if (type & PERF_SAMPLE_IP)
+		ADD_FIELD(event_class, cw->data.u64_hex, "ip");
+
+	if (type & PERF_SAMPLE_TID) {
+		ADD_FIELD(event_class, cw->data.s32, "tid");
+		ADD_FIELD(event_class, cw->data.s32, "pid");
+	}
+
+	if ((type & PERF_SAMPLE_ID) ||
+	    (type & PERF_SAMPLE_IDENTIFIER))
+		ADD_FIELD(event_class, cw->data.u64, "id");
+
+	if (type & PERF_SAMPLE_STREAM_ID)
+		ADD_FIELD(event_class, cw->data.u64, "stream_id");
+
+	if (type & PERF_SAMPLE_CPU)
+		ADD_FIELD(event_class, cw->data.u32, "cpu");
+
+	if (type & PERF_SAMPLE_PERIOD)
+		ADD_FIELD(event_class, cw->data.u64, "period");
+
+	if (type & PERF_SAMPLE_WEIGHT)
+		ADD_FIELD(event_class, cw->data.u64, "weight");
+
+	if (type & PERF_SAMPLE_DATA_SRC)
+		ADD_FIELD(event_class, cw->data.u64, "data_src");
+
+	if (type & PERF_SAMPLE_TRANSACTION)
+		ADD_FIELD(event_class, cw->data.u64, "transaction");
+
+#undef ADD_FIELD
+	return 0;
+}
+
+static int add_event(struct ctf_writer *cw, struct perf_evsel *evsel)
+{
+	struct bt_ctf_event_class *event_class;
+	struct evsel_priv *priv;
+	const char *name = perf_evsel__name(evsel);
+	int ret;
+
+	pr("Adding event '%s' (type %d)\n", name, evsel->attr.type);
+
+	event_class = bt_ctf_event_class_create(name);
+	if (!event_class)
+		return -1;
+
+	ret = add_generic_types(cw, evsel, event_class);
+	if (ret)
+		goto err;
+
+	ret = bt_ctf_stream_class_add_event_class(cw->stream_class, event_class);
+	if (ret) {
+		pr("Failed to add event class into stream.\n");
+		goto err;
+	}
+
+	priv = malloc(sizeof(*priv));
+	if (!priv)
+		goto err;
+
+	priv->event_class = event_class;
+	evsel->priv       = priv;
+	return 0;
+
+err:
+	bt_ctf_event_class_put(event_class);
+	pr_err("Failed to add event '%s'.\n", name);
+	return -1;
+}
+
+static int setup_events(struct ctf_writer *cw, struct perf_session *session)
+{
+	struct perf_evlist *evlist = session->evlist;
+	struct perf_evsel *evsel;
+	int ret;
+
+	evlist__for_each(evlist, evsel) {
+		ret = add_event(cw, evsel);
+		if (ret)
+			return ret;
+	}
+	return 0;
+}
+
+static int ctf_writer__setup_env(struct ctf_writer *cw,
+				 struct perf_session *session)
+{
+	struct perf_header *header = &session->header;
+	struct bt_ctf_writer *writer = cw->writer;
+
+#define ADD(__n, __v)							\
+do {									\
+	if (bt_ctf_writer_add_environment_field(writer, __n, __v))	\
+		return -1;						\
+} while (0)
+
+	ADD("host",    header->env.hostname);
+	ADD("sysname", "Linux");
+	ADD("release", header->env.os_release);
+	ADD("version", header->env.version);
+	ADD("machine", header->env.arch);
+	ADD("domain", "kernel");
+	ADD("tracer_name", "perf");
+
+#undef ADD
+	return 0;
+}
+
+static int ctf_writer__setup_clock(struct ctf_writer *cw)
+{
+	struct bt_ctf_clock *clock = cw->clock;
+
+	bt_ctf_clock_set_description(clock, "perf clock");
+
+#define SET(__n, __v)				\
+do {						\
+	if (bt_ctf_clock_set_##__n(clock, __v))	\
+		return -1;			\
+} while (0)
+
+	SET(frequency,   1000000000);
+	SET(offset_s,    0);
+	SET(offset,      0);
+	SET(precision,   10);
+	SET(is_absolute, 0);
+
+#undef SET
+	return 0;
+}
+
+static struct bt_ctf_field_type *create_int_type(int size, bool sign, bool hex)
+{
+	struct bt_ctf_field_type *type;
+
+	type = bt_ctf_field_type_integer_create(size);
+	if (!type)
+		return NULL;
+
+	if (sign &&
+	    bt_ctf_field_type_integer_set_signed(type, 1))
+		goto err;
+
+	if (hex &&
+	    bt_ctf_field_type_integer_set_base(type, BT_CTF_INTEGER_BASE_HEXADECIMAL))
+		goto err;
+
+	pr2("Created type: INTEGER %d-bit %ssigned %s\n",
+	    size, sign ? "un" : "", hex ? "hex" : "");
+	return type;
+
+err:
+	bt_ctf_field_type_put(type);
+	return NULL;
+}
+
+static void ctf_writer__cleanup_data(struct ctf_writer *cw)
+{
+	unsigned int i;
+
+	for (i = 0; i < ARRAY_SIZE(cw->data.array); i++)
+		bt_ctf_field_type_put(cw->data.array[i]);
+}
+
+static int ctf_writer__init_data(struct ctf_writer *cw)
+{
+#define CREATE_INT_TYPE(type, size, sign, hex)		\
+do {							\
+	(type) = create_int_type(size, sign, hex);	\
+	if (!(type))					\
+		goto err;				\
+} while (0)
+
+	CREATE_INT_TYPE(cw->data.s64, 64, true,  false);
+	CREATE_INT_TYPE(cw->data.u64, 64, false, false);
+	CREATE_INT_TYPE(cw->data.s32, 32, true,  false);
+	CREATE_INT_TYPE(cw->data.u32, 32, false, false);
+	CREATE_INT_TYPE(cw->data.u64_hex, 64, false, true);
+
+	cw->data.string  = bt_ctf_field_type_string_create();
+	if (cw->data.string)
+		return 0;
+
+err:
+	ctf_writer__cleanup_data(cw);
+	pr_err("Failed to create data types.\n");
+	return -1;
+}
+
+static void ctf_writer__cleanup(struct ctf_writer *cw)
+{
+	ctf_writer__cleanup_data(cw);
+
+	bt_ctf_clock_put(cw->clock);
+	bt_ctf_stream_put(cw->stream);
+	bt_ctf_stream_class_put(cw->stream_class);
+	bt_ctf_writer_put(cw->writer);
+
+	/* and NULL all the pointers */
+	memset(cw, 0, sizeof(*cw));
+}
+
+static int ctf_writer__init(struct ctf_writer *cw, const char *path)
+{
+	struct bt_ctf_writer		*writer;
+	struct bt_ctf_stream_class	*stream_class;
+	struct bt_ctf_stream		*stream;
+	struct bt_ctf_clock		*clock;
+
+	/* CTF writer */
+	writer = bt_ctf_writer_create(path);
+	if (!writer)
+		goto err;
+
+	cw->writer = writer;
+
+	/* CTF clock */
+	clock = bt_ctf_clock_create("perf_clock");
+	if (!clock) {
+		pr("Failed to create CTF clock.\n");
+		goto err_cleanup;
+	}
+
+	cw->clock = clock;
+
+	if (ctf_writer__setup_clock(cw)) {
+		pr("Failed to setup CTF clock.\n");
+		goto err_cleanup;
+	}
+
+	/* CTF stream class */
+	stream_class = bt_ctf_stream_class_create("perf_stream");
+	if (!stream_class) {
+		pr("Failed to create CTF stream class.\n");
+		goto err_cleanup;
+	}
+
+	cw->stream_class = stream_class;
+
+	/* CTF clock stream setup */
+	if (bt_ctf_stream_class_set_clock(stream_class, clock)) {
+		pr("Failed to assign CTF clock to stream class.\n");
+		goto err_cleanup;
+	}
+
+	if (ctf_writer__init_data(cw))
+		goto err_cleanup;
+
+	/* CTF stream instance */
+	stream = bt_ctf_writer_create_stream(writer, stream_class);
+	if (!stream) {
+		pr("Failed to create CTF stream.\n");
+		goto err_cleanup;
+	}
+
+	cw->stream = stream;
+
+	/* CTF clock writer setup */
+	if (bt_ctf_writer_add_clock(writer, clock)) {
+		pr("Failed to assign CTF clock to writer.\n");
+		goto err_cleanup;
+	}
+
+	return 0;
+
+err_cleanup:
+	ctf_writer__cleanup(cw);
+err:
+	pr_err("Failed to setup CTF writer.\n");
+	return -1;
+}
+
+int bt_convert__perf2ctf(const char *input, const char *path)
+{
+	struct perf_session *session;
+	struct perf_data_file file = {
+		.path = input,
+		.mode = PERF_DATA_MODE_READ,
+	};
+	struct convert c = {
+		.tool = {
+			.sample          = process_sample_event,
+			.mmap            = perf_event__process_mmap,
+			.mmap2           = perf_event__process_mmap2,
+			.comm            = perf_event__process_comm,
+			.exit            = perf_event__process_exit,
+			.fork            = perf_event__process_fork,
+			.lost            = perf_event__process_lost,
+			.tracing_data    = perf_event__process_tracing_data,
+			.build_id        = perf_event__process_build_id,
+			.ordered_events  = true,
+			.ordering_requires_timestamps = true,
+		},
+	};
+	struct ctf_writer *cw = &c.writer;
+	int err = -1;
+
+	/* CTF writer */
+	if (ctf_writer__init(cw, path))
+		return -1;
+
+	/* perf.data session */
+	session = perf_session__new(&file, 0, NULL);
+	if (!session)
+		goto free_writer;
+
+	/* CTF writer env/clock setup  */
+	if (ctf_writer__setup_env(cw, session))
+		goto free_session;
+
+	/* CTF events setup */
+	if (setup_events(cw, session))
+		goto free_session;
+
+	err = perf_session__process_events(session, &c.tool);
+	if (!err)
+		err = bt_ctf_stream_flush(cw->stream);
+
+	fprintf(stderr,
+		"[ perf data convert: Converted '%s' into CTF data '%s' ]\n",
+		file.path, path);
+
+	fprintf(stderr,
+		"[ perf data convert: Converted and wrote %.3f MB (%" PRIu64 " samples) ]\n",
+		(double) c.events_size / 1024.0 / 1024.0,
+		c.events_count);
+
+	/* its all good */
+free_session:
+	perf_session__delete(session);
+
+free_writer:
+	ctf_writer__cleanup(cw);
+	return err;
+}
diff --git a/tools/perf/util/data-convert-bt.h b/tools/perf/util/data-convert-bt.h
new file mode 100644
index 0000000..dda30c5
--- /dev/null
+++ b/tools/perf/util/data-convert-bt.h
@@ -0,0 +1,8 @@
+#ifndef __DATA_CONVERT_BT_H
+#define __DATA_CONVERT_BT_H
+#ifdef HAVE_LIBBABELTRACE_SUPPORT
+
+int bt_convert__perf2ctf(const char *input_name, const char *to_ctf);
+
+#endif /* HAVE_LIBBABELTRACE_SUPPORT */
+#endif /* __DATA_CONVERT_BT_H */
diff --git a/tools/perf/util/debug.c b/tools/perf/util/debug.c
index ad60b2f..2da5581 100644
--- a/tools/perf/util/debug.c
+++ b/tools/perf/util/debug.c
@@ -20,6 +20,7 @@ int verbose;
 bool dump_trace = false, quiet = false;
 int debug_ordered_events;
 static int redirect_to_stderr;
+int debug_data_convert;
 
 static int _eprintf(int level, int var, const char *fmt, va_list args)
 {
@@ -147,6 +148,7 @@ static struct debug_variable {
 	{ .name = "verbose",		.ptr = &verbose },
 	{ .name = "ordered-events",	.ptr = &debug_ordered_events},
 	{ .name = "stderr",		.ptr = &redirect_to_stderr},
+	{ .name = "data-convert",	.ptr = &debug_data_convert },
 	{ .name = NULL, }
 };
 
diff --git a/tools/perf/util/debug.h b/tools/perf/util/debug.h
index be264d6..caac2fd 100644
--- a/tools/perf/util/debug.h
+++ b/tools/perf/util/debug.h
@@ -12,6 +12,7 @@
 extern int verbose;
 extern bool quiet, dump_trace;
 extern int debug_ordered_events;
+extern int debug_data_convert;
 
 #ifndef pr_fmt
 #define pr_fmt(fmt) fmt

[tip:perf/core] perf data: Add a 'perf' prefix to the generic fields

[tip-bot for Sebastian Andrzej Siewior]

Commit-ID:  54cf776a9c5c2e6a91de31954bba4d3bad6c657c
Gitweb:     http://git.kernel.org/tip/54cf776a9c5c2e6a91de31954bba4d3bad6c657c
Author:     Sebastian Andrzej Siewior <bigeasy@linutronix.de>
AuthorDate: Fri, 20 Feb 2015 23:17:01 +0100
Committer:  Arnaldo Carvalho de Melo <acme@redhat.com>
CommitDate: Wed, 25 Feb 2015 16:14:33 -0300

perf data: Add a 'perf' prefix to the generic fields

Some of the tracers bring their own id or pid fields and we can end up
having two of them. This patch adds a "perf_" prefix to the 'generic'
fields so we avoid a clash of the member names.

The change is visible in the babeltrace output:

Before:
  $ babeltrace ./ctf-data/
  [03:19:13.962131936] (+0.000001935) cycles: { }, { ip = 0xFFFFFFFF8105443A, tid = 20714, pid = 20714, period = 8 }
  [03:19:13.962133732] (+0.000001796) cycles: { }, { ip = 0xFFFFFFFF8105443A, tid = 20714, pid = 20714, period = 114 }
  ...

Now:
  $ babeltrace ./ctf-data/
  [03:19:13.962131936] (+0.000001935) cycles: { }, { perf_ip = 0xFFFFFFFF8105443A, perf_tid = 20714, perf_pid = 20714, perf_period = 8 }
  [03:19:13.962133732] (+0.000001796) cycles: { }, { perf_ip = 0xFFFFFFFF8105443A, perf_tid = 20714, perf_pid = 20714, perf_period = 114 }
  ...

Signed-off-by: Sebastian Andrzej Siewior <bigeasy@linutronix.de>
Acked-by: Namhyung Kim <namhyung@kernel.org>
Reviewed-by: David Ahern <dsahern@gmail.com>
Tested-by: Arnaldo Carvalho de Melo <acme@redhat.com>
Cc: Frederic Weisbecker <fweisbec@gmail.com>
Cc: Jeremie Galarneau <jgalar@efficios.com>
Cc: Jiri Olsa <jolsa@kernel.org>
Cc: Paul Mackerras <paulus@samba.org>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Tom Zanussi <tzanussi@gmail.com>
Cc: Wang Nan <wangnan0@huawei.com>
Link: http://lkml.kernel.org/r/1424470628-5969-5-git-send-email-jolsa@kernel.org
Signed-off-by: Jiri Olsa <jolsa@kernel.org>
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
---
 tools/perf/util/data-convert-bt.c | 42 ++++++++++++++++++++-------------------
 1 file changed, 22 insertions(+), 20 deletions(-)

diff --git a/tools/perf/util/data-convert-bt.c b/tools/perf/util/data-convert-bt.c
index ff4826c..e372e03 100644
--- a/tools/perf/util/data-convert-bt.c
+++ b/tools/perf/util/data-convert-bt.c
@@ -147,60 +147,62 @@ static int add_generic_values(struct ctf_writer *cw,
 	 */
 
 	if (type & PERF_SAMPLE_IP) {
-		ret = value_set_u64_hex(cw, event, "ip", sample->ip);
+		ret = value_set_u64_hex(cw, event, "perf_ip", sample->ip);
 		if (ret)
 			return -1;
 	}
 
 	if (type & PERF_SAMPLE_TID) {
-		ret = value_set_s32(cw, event, "tid", sample->tid);
+		ret = value_set_s32(cw, event, "perf_tid", sample->tid);
 		if (ret)
 			return -1;
 
-		ret = value_set_s32(cw, event, "pid", sample->pid);
+		ret = value_set_s32(cw, event, "perf_pid", sample->pid);
 		if (ret)
 			return -1;
 	}
 
 	if ((type & PERF_SAMPLE_ID) ||
 	    (type & PERF_SAMPLE_IDENTIFIER)) {
-		ret = value_set_u64(cw, event, "id", sample->id);
+		ret = value_set_u64(cw, event, "perf_id", sample->id);
 		if (ret)
 			return -1;
 	}
 
 	if (type & PERF_SAMPLE_STREAM_ID) {
-		ret = value_set_u64(cw, event, "stream_id", sample->stream_id);
+		ret = value_set_u64(cw, event, "perf_stream_id", sample->stream_id);
 		if (ret)
 			return -1;
 	}
 
 	if (type & PERF_SAMPLE_CPU) {
-		ret = value_set_u32(cw, event, "cpu", sample->cpu);
+		ret = value_set_u32(cw, event, "perf_cpu", sample->cpu);
 		if (ret)
 			return -1;
 	}
 
 	if (type & PERF_SAMPLE_PERIOD) {
-		ret = value_set_u64(cw, event, "period", sample->period);
+		ret = value_set_u64(cw, event, "perf_period", sample->period);
 		if (ret)
 			return -1;
 	}
 
 	if (type & PERF_SAMPLE_WEIGHT) {
-		ret = value_set_u64(cw, event, "weight", sample->weight);
+		ret = value_set_u64(cw, event, "perf_weight", sample->weight);
 		if (ret)
 			return -1;
 	}
 
 	if (type & PERF_SAMPLE_DATA_SRC) {
-		ret = value_set_u64(cw, event, "data_src", sample->data_src);
+		ret = value_set_u64(cw, event, "perf_data_src",
+				sample->data_src);
 		if (ret)
 			return -1;
 	}
 
 	if (type & PERF_SAMPLE_TRANSACTION) {
-		ret = value_set_u64(cw, event, "transaction", sample->transaction);
+		ret = value_set_u64(cw, event, "perf_transaction",
+				sample->transaction);
 		if (ret)
 			return -1;
 	}
@@ -276,34 +278,34 @@ static int add_generic_types(struct ctf_writer *cw, struct perf_evsel *evsel,
 	} while (0)
 
 	if (type & PERF_SAMPLE_IP)
-		ADD_FIELD(event_class, cw->data.u64_hex, "ip");
+		ADD_FIELD(event_class, cw->data.u64_hex, "perf_ip");
 
 	if (type & PERF_SAMPLE_TID) {
-		ADD_FIELD(event_class, cw->data.s32, "tid");
-		ADD_FIELD(event_class, cw->data.s32, "pid");
+		ADD_FIELD(event_class, cw->data.s32, "perf_tid");
+		ADD_FIELD(event_class, cw->data.s32, "perf_pid");
 	}
 
 	if ((type & PERF_SAMPLE_ID) ||
 	    (type & PERF_SAMPLE_IDENTIFIER))
-		ADD_FIELD(event_class, cw->data.u64, "id");
+		ADD_FIELD(event_class, cw->data.u64, "perf_id");
 
 	if (type & PERF_SAMPLE_STREAM_ID)
-		ADD_FIELD(event_class, cw->data.u64, "stream_id");
+		ADD_FIELD(event_class, cw->data.u64, "perf_stream_id");
 
 	if (type & PERF_SAMPLE_CPU)
-		ADD_FIELD(event_class, cw->data.u32, "cpu");
+		ADD_FIELD(event_class, cw->data.u32, "perf_cpu");
 
 	if (type & PERF_SAMPLE_PERIOD)
-		ADD_FIELD(event_class, cw->data.u64, "period");
+		ADD_FIELD(event_class, cw->data.u64, "perf_period");
 
 	if (type & PERF_SAMPLE_WEIGHT)
-		ADD_FIELD(event_class, cw->data.u64, "weight");
+		ADD_FIELD(event_class, cw->data.u64, "perf_weight");
 
 	if (type & PERF_SAMPLE_DATA_SRC)
-		ADD_FIELD(event_class, cw->data.u64, "data_src");
+		ADD_FIELD(event_class, cw->data.u64, "perf_data_src");
 
 	if (type & PERF_SAMPLE_TRANSACTION)
-		ADD_FIELD(event_class, cw->data.u64, "transaction");
+		ADD_FIELD(event_class, cw->data.u64, "perf_transaction");
 
 #undef ADD_FIELD
 	return 0;

Re: [tip:perf/core] perf tools: Add feature check for libbabeltrace

[Ingo Molnar]

* tip-bot for Jiri Olsa <tipbot@zytor.com> wrote:

> Commit-ID:  53d0a57343949b2af9b27229db534b98e5a0c4d0
> Gitweb:     http://git.kernel.org/tip/53d0a57343949b2af9b27229db534b98e5a0c4d0
> Author:     Jiri Olsa <jolsa@kernel.org>
> AuthorDate: Fri, 20 Feb 2015 23:16:58 +0100
> Committer:  Arnaldo Carvalho de Melo <acme@redhat.com>
> CommitDate: Wed, 25 Feb 2015 12:42:24 -0300
> 
> perf tools: Add feature check for libbabeltrace
> 
> Adding feature check for babeltrace library [1], which will be used for
> perf data file CTF [2] conversion in following patches.

So this doesn't work on recent Ubuntu systems:

...                 libbabeltrace: [ OFF ]

triton:~/tip/tools/perf/config/feature-checks> make 
test-libbabeltrace.bin
gcc -MD  -Wall -Werror -o test-libbabeltrace.bin test-libbabeltrace.c  
# -lbabeltrace provided by 
test-libbabeltrace.c: In function ‘main’:test-libbabeltrace.c:6:2: error: implicit declaration of function ‘bt_ctf_stream_class_get_packet_context_type’ [-Werror=implicit-function-declaration]
  bt_ctf_stream_class_get_packet_context_type((void *) 0);
  ^
cc1: all warnings being treated as errors
Makefile:140: recipe for target 'test-libbabeltrace.bin' failed

I have all the requirements, libbabeltrace-ctf-dev and 
libbabeltrace-dev installed, yet the writer.h file:

-rw-r--r-- 1 root root 5283 Oct  7 14:44 /usr/include/babeltrace/ctf-writer/writer.h

does not include a prototype for 
bt_ctf_stream_class_get_packet_context_type().

If Ubuntu's writer.h is too old then either our API use should match 
the ones available:

triton:~/tip/tools/perf/config/feature-checks> grep extern 
/usr/include/babeltrace/ctf-writer/writer.h 
extern "C" {
extern struct bt_ctf_writer *bt_ctf_writer_create(const char *path);
extern struct bt_ctf_stream *bt_ctf_writer_create_stream(
extern int bt_ctf_writer_add_environment_field(struct bt_ctf_writer *writer,
extern int bt_ctf_writer_add_clock(struct bt_ctf_writer *writer,
extern char *bt_ctf_writer_get_metadata_string(struct bt_ctf_writer *writer);
extern void bt_ctf_writer_flush_metadata(struct bt_ctf_writer *writer);
extern int bt_ctf_writer_set_byte_order(struct bt_ctf_writer *writer,
extern void bt_ctf_writer_get(struct bt_ctf_writer *writer);
extern void bt_ctf_writer_put(struct bt_ctf_writer *writer);

or CTF support should be removed from the default set of perf 
features, because right now it blocks the 'all' build speedup.

Thanks,

	Ingo

Re: [tip:perf/core] perf tools: Add feature check for libbabeltrace

[Jiri Olsa]

On Sat, Feb 28, 2015 at 10:25:03AM +0100, Ingo Molnar wrote:
> 
> * tip-bot for Jiri Olsa <tipbot@zytor.com> wrote:
> 
> > Commit-ID:  53d0a57343949b2af9b27229db534b98e5a0c4d0
> > Gitweb:     http://git.kernel.org/tip/53d0a57343949b2af9b27229db534b98e5a0c4d0
> > Author:     Jiri Olsa <jolsa@kernel.org>
> > AuthorDate: Fri, 20 Feb 2015 23:16:58 +0100
> > Committer:  Arnaldo Carvalho de Melo <acme@redhat.com>
> > CommitDate: Wed, 25 Feb 2015 12:42:24 -0300
> > 
> > perf tools: Add feature check for libbabeltrace > > 
> > Adding feature check for babeltrace library [1], which will be used for
> > perf data file CTF [2] conversion in following patches.
> 
> So this doesn't work on recent Ubuntu systems:
> 
> ...                 libbabeltrace: [ OFF ]
> 
> triton:~/tip/tools/perf/config/feature-checks> make 
> test-libbabeltrace.bin
> gcc -MD  -Wall -Werror -o test-libbabeltrace.bin test-libbabeltrace.c  
> # -lbabeltrace provided by 
> test-libbabeltrace.c: In function ‘main’:test-libbabeltrace.c:6:2: error: implicit declaration of function ‘bt_ctf_stream_class_get_packet_context_type’ [-Werror=implicit-function-declaration]
>   bt_ctf_stream_class_get_packet_context_type((void *) 0);
>   ^
> cc1: all warnings being treated as errors
> Makefile:140: recipe for target 'test-libbabeltrace.bin' failed
> 
> I have all the requirements, libbabeltrace-ctf-dev and 
> libbabeltrace-dev installed, yet the writer.h file:
> 
> -rw-r--r-- 1 root root 5283 Oct  7 14:44 /usr/include/babeltrace/ctf-writer/writer.h
> 
> does not include a prototype for 
> bt_ctf_stream_class_get_packet_context_type().

hum, I have it in stream-class.h and it seems to be there since Jul 29 2014

What is the libbabeltrace package version on Ubuntu?

> 
> If Ubuntu's writer.h is too old then either our API use should match 
> the ones available:
> 
> triton:~/tip/tools/perf/config/feature-checks> grep extern 
> /usr/include/babeltrace/ctf-writer/writer.h 
> extern "C" {
> extern struct bt_ctf_writer *bt_ctf_writer_create(const char *path);
> extern struct bt_ctf_stream *bt_ctf_writer_create_stream(
> extern int bt_ctf_writer_add_environment_field(struct bt_ctf_writer *writer,
> extern int bt_ctf_writer_add_clock(struct bt_ctf_writer *writer,
> extern char *bt_ctf_writer_get_metadata_string(struct bt_ctf_writer *writer);
> extern void bt_ctf_writer_flush_metadata(struct bt_ctf_writer *writer);
> extern int bt_ctf_writer_set_byte_order(struct bt_ctf_writer *writer,
> extern void bt_ctf_writer_get(struct bt_ctf_writer *writer);
> extern void bt_ctf_writer_put(struct bt_ctf_writer *writer);
> 
> or CTF support should be removed from the default set of perf 
> features, because right now it blocks the 'all' build speedup.

I'll try to get Ubuntu system and check, but I'd be ok to put
it out of the default set until ditributions catch up..

I'll check next week and send patch or update

thanks,
jirka

Re: [PATCH 05/11] perf data: Add tracepoint events fields CTF conversion support

[Jiri Olsa]

On Wed, Feb 25, 2015 at 04:23:44PM -0300, Arnaldo Carvalho de Melo wrote:
> Em Fri, Feb 20, 2015 at 11:17:02PM +0100, Jiri Olsa escreveu:
> > From: Sebastian Andrzej Siewior <bigeasy@linutronix.de>
> > Adding support to convert tracepoint event fields into CTF
> > event fields.
>  
> > We parse each tracepoint event for CTF conversion and add
> > tracepoint fields as regular CTF event fields, so they
> > appear in babeltrace output like:
>  
> >   $ babeltrace ./ctf-data/
> >   ...
> >   [09:02:00.950703057] (+?.?????????) sched:sched_stat_runtime: { }, { perf_ip = ... SNIP ... common_type = 298, common_flags = 1, \
> >   common_preempt_count = 0, common_pid = 31813, comm = "perf", pid = 31813, runtime = 458800, vruntime = 52059858071 }
> >   ...
> 
> Processed the previous patches, everything ok:
> 
> [acme@ssdandy linux]$ ls -la perf.data
> ls: cannot access perf.data: No such file or directory
> [acme@ssdandy linux]$ trace record usleep 1
> [ perf record: Woken up 1 times to write data ]
> [ perf record: Captured and wrote 0.029 MB perf.data (88 samples) ]
> [acme@ssdandy linux]$ ls -la perf.data
> -rw-------. 1 acme acme 5399896 Fev 25 16:19 perf.data
> [acme@ssdandy linux]$ perf evlist
> raw_syscalls:sys_enter
> raw_syscalls:sys_exit
> [acme@ssdandy linux]$ perf evlist -v
> raw_syscalls:sys_enter: sample_freq=1, type: 2, config: 75, size: 104, sample_type: IP|TID|TIME|ID|CPU|PERIOD|RAW, read_format: ID, disabled: 1, inherit: 1, mmap: 1, mmap2: 1, comm: 1, comm_exec: 1, enable_on_exec: 1, task: 1, sample_id_all: 1, exclude_guest: 1
> raw_syscalls:sys_exit: sample_freq=1, type: 2, config: 74, size: 104, sample_type: IP|TID|TIME|ID|CPU|PERIOD|RAW, read_format: ID, disabled: 1, inherit: 1, enable_on_exec: 1, sample_id_all: 1, exclude_guest: 1
> [acme@ssdandy linux]$ perf data convert --to-ctf=./ctf-data/
> [ perf data convert: Converted 'perf.data' into CTF data './ctf-data/' ]
> [ perf data convert: Converted and wrote 0.009 MB (88 samples) ]
> [acme@ssdandy linux]$ babeltrace ./ctf-data/
> [23:48:47.557933780] (+?.?????????) raw_syscalls:sys_exit: { }, { perf_ip = 0xFFFFFFFF81020FBA, perf_tid = 5093, perf_pid = 5093, perf_id = 1512, perf_cpu = 3, perf_period = 1 }
> [23:48:47.557957461] (+0.000023681) raw_syscalls:sys_enter: { }, { perf_ip = 0xFFFFFFFF81020EA6, perf_tid = 5093, perf_pid = 5093, perf_id = 1504, perf_cpu = 3, perf_period = 1 }
> [23:48:47.557958406] (+0.000000945) raw_syscalls:sys_exit: { }, { perf_ip = 0xFFFFFFFF81020FBA, perf_tid = 5093, perf_pid = 5093, perf_id = 1512, perf_cpu = 3, perf_period = 1 }
> [23:48:47.557973567] (+0.000015161) raw_syscalls:sys_enter: { }, { perf_ip = 0xFFFFFFFF81020EA6, perf_tid = 5093, perf_pid = 5093, perf_id = 1504, perf_cpu = 3, perf_period = 1 }
> [23:48:47.557976047] (+0.000002480) raw_syscalls:sys_exit: { }, { perf_ip = 0xFFFFFFFF81020FBA, perf_tid = 5093, perf_pid = 5093, perf_id = 1512, perf_cpu = 3, perf_period = 1 }
> [23:48:47.557985774] (+0.000009727) raw_syscalls:sys_enter: { }, { perf_ip = 0xFFFFFFFF81020EA6, perf_tid = 5093, perf_pid = 5093, perf_id = 1504, perf_cpu = 3, perf_period = 1 }
> [23:48:47.557990826] (+0.000005052) raw_syscalls:sys_exit: { }, { perf_ip = 0xFFFFFFFF81020FBA, perf_tid = 5093, perf_pid = 5093, perf_id = 1512, perf_cpu = 3, perf_period = 1 }
> <SNIP>
> 
> But then I apply this patch (convert tracepoint events fields into CTF event fields) and:
> 
> [acme@ssdandy linux]$ perf data convert --to-ctf=./ctf-data/
> [ perf data convert: Converted 'perf.data' into CTF data './ctf-data/' ]
> [ perf data convert: Converted and wrote 0.009 MB (88 samples) ]
> [acme@ssdandy linux]$ babeltrace ./ctf-data/
> [error] Packet size (18446744073709551615 bits) is larger than remaining file size (262144 bits).
> [error] Stream index creation error.
> [error] Open file stream error.
> [warning] [Context] Cannot open_trace of format ctf at path ./ctf-data.
> [warning] [Context] cannot open trace "./ctf-data" from ./ctf-data/ for reading.
> [error] Cannot open any trace for reading.
> 
> [error] opening trace "./ctf-data/" for reading.
> 
> [error] none of the specified trace paths could be opened.
> 
> [acme@ssdandy linux]$
> 
> It stops working.
> 
> [acme@ssdandy linux]$ ls -la ctf-data/
> total 44
> drwxrwx---.  2 acme acme    41 Fev 25 16:12 .
> drwxrwxr-x. 28 acme acme  4096 Fev 25 16:19 ..
> -rw-rw----.  1 acme acme  4666 Fev 25 16:21 metadata
> -rw-rw----.  1 acme acme 32768 Fev 25 16:21 perf_stream_0
> [acme@ssdandy linux]$
> 
> Can you try to reproduce this? The ctf-data/metadata file is below:

hum, i just tried and can't reproduce this one.. anychance of mixed
babeltrace installations? How did you install babeltrace sources?

I assume this was some standard Fedora latest something system right?

jirka


git HEAD:

[root@krava perf]# git log --oneline | head -3
9534f89 perf data: Add tracepoint events fields CTF conversion support
bb848d1 perf stat: Report unsupported events properly
ffdfa43 perf tools: Compare JOBS to 0 after grep

test:

[root@krava perf]# LD_LIBRARY_PATH=/opt/libbabeltrace/lib/ ./perf trace record usleep 1
[ perf record: Woken up 1 times to write data ]
[ perf record: Captured and wrote 0.023 MB perf.data (88 samples) ]
[root@krava perf]# LD_LIBRARY_PATH=/opt/libbabeltrace/lib/ ./perf data convert --to-ctf=./ctf-data
[ perf data convert: Converted 'perf.data' into CTF data './ctf-data' ]
[ perf data convert: Converted and wrote 0.009 MB (88 samples) ]
[root@krava perf]# babeltrace ./ctf-data/
[10:04:03.320544407] (+?.?????????) raw_syscalls:sys_exit: { }, { perf_ip = 0xFFFFFFFF810235AC, perf_tid = 23899, perf_pid = 23899, perf_id = 82, perf_cpu = 3, perf_period = 1, common_type = 75, common_flags = 0, common_preempt_count = 0, common_pid = 23899, id = 59, ret = 0 }
[10:04:03.320568054] (+0.000023647) raw_syscalls:sys_enter: { }, { perf_ip = 0xFFFFFFFF81023420, perf_tid = 23899, perf_pid = 23899, perf_id = 78, perf_cpu = 3, perf_period = 1, common_type = 76, common_flags = 0, common_preempt_count = 0, common_pid = 23899, id = 12, args = [ [0] = 0x0, [1] = 0x0, [2] = 0x0, [3] = 0x0, [4] = 0x37F, [5] = 0x64 ] }
[10:04:03.320568993] (+0.000000939) raw_syscalls:sys_exit: { }, { perf_ip = 0xFFFFFFFF810235AC, perf_tid = 23899, perf_pid = 23899, perf_id = 82, perf_cpu = 3, perf_period = 1, common_type = 75, common_flags = 0, common_preempt_count = 0, common_pid = 23899, id = 12, ret = 24440832 }
...

Re: [PATCH 05/11] perf data: Add tracepoint events fields CTF conversion support

[Arnaldo Carvalho de Melo]

Em Sun, Mar 01, 2015 at 02:20:43PM +0100, Jiri Olsa escreveu:
> On Wed, Feb 25, 2015 at 04:23:44PM -0300, Arnaldo Carvalho de Melo wrote:
> > Em Fri, Feb 20, 2015 at 11:17:02PM +0100, Jiri Olsa escreveu:
> > > From: Sebastian Andrzej Siewior <bigeasy@linutronix.de>
> > > Adding support to convert tracepoint event fields into CTF
> > > event fields.
> >  
> > > We parse each tracepoint event for CTF conversion and add
> > > tracepoint fields as regular CTF event fields, so they
> > > appear in babeltrace output like:
> >  
> > >   $ babeltrace ./ctf-data/
> > >   ...
> > >   [09:02:00.950703057] (+?.?????????) sched:sched_stat_runtime: { }, { perf_ip = ... SNIP ... common_type = 298, common_flags = 1, \
> > >   common_preempt_count = 0, common_pid = 31813, comm = "perf", pid = 31813, runtime = 458800, vruntime = 52059858071 }
> > >   ...
> > 
> > Processed the previous patches, everything ok:
> > 
> > [acme@ssdandy linux]$ ls -la perf.data
> > ls: cannot access perf.data: No such file or directory
> > [acme@ssdandy linux]$ trace record usleep 1
> > [ perf record: Woken up 1 times to write data ]
> > [ perf record: Captured and wrote 0.029 MB perf.data (88 samples) ]
> > [acme@ssdandy linux]$ ls -la perf.data
> > -rw-------. 1 acme acme 5399896 Fev 25 16:19 perf.data
> > [acme@ssdandy linux]$ perf evlist
> > raw_syscalls:sys_enter
> > raw_syscalls:sys_exit
> > [acme@ssdandy linux]$ perf evlist -v
> > raw_syscalls:sys_enter: sample_freq=1, type: 2, config: 75, size: 104, sample_type: IP|TID|TIME|ID|CPU|PERIOD|RAW, read_format: ID, disabled: 1, inherit: 1, mmap: 1, mmap2: 1, comm: 1, comm_exec: 1, enable_on_exec: 1, task: 1, sample_id_all: 1, exclude_guest: 1
> > raw_syscalls:sys_exit: sample_freq=1, type: 2, config: 74, size: 104, sample_type: IP|TID|TIME|ID|CPU|PERIOD|RAW, read_format: ID, disabled: 1, inherit: 1, enable_on_exec: 1, sample_id_all: 1, exclude_guest: 1
> > [acme@ssdandy linux]$ perf data convert --to-ctf=./ctf-data/
> > [ perf data convert: Converted 'perf.data' into CTF data './ctf-data/' ]
> > [ perf data convert: Converted and wrote 0.009 MB (88 samples) ]
> > [acme@ssdandy linux]$ babeltrace ./ctf-data/
> > [23:48:47.557933780] (+?.?????????) raw_syscalls:sys_exit: { }, { perf_ip = 0xFFFFFFFF81020FBA, perf_tid = 5093, perf_pid = 5093, perf_id = 1512, perf_cpu = 3, perf_period = 1 }
> > [23:48:47.557957461] (+0.000023681) raw_syscalls:sys_enter: { }, { perf_ip = 0xFFFFFFFF81020EA6, perf_tid = 5093, perf_pid = 5093, perf_id = 1504, perf_cpu = 3, perf_period = 1 }
> > [23:48:47.557958406] (+0.000000945) raw_syscalls:sys_exit: { }, { perf_ip = 0xFFFFFFFF81020FBA, perf_tid = 5093, perf_pid = 5093, perf_id = 1512, perf_cpu = 3, perf_period = 1 }
> > [23:48:47.557973567] (+0.000015161) raw_syscalls:sys_enter: { }, { perf_ip = 0xFFFFFFFF81020EA6, perf_tid = 5093, perf_pid = 5093, perf_id = 1504, perf_cpu = 3, perf_period = 1 }
> > [23:48:47.557976047] (+0.000002480) raw_syscalls:sys_exit: { }, { perf_ip = 0xFFFFFFFF81020FBA, perf_tid = 5093, perf_pid = 5093, perf_id = 1512, perf_cpu = 3, perf_period = 1 }
> > [23:48:47.557985774] (+0.000009727) raw_syscalls:sys_enter: { }, { perf_ip = 0xFFFFFFFF81020EA6, perf_tid = 5093, perf_pid = 5093, perf_id = 1504, perf_cpu = 3, perf_period = 1 }
> > [23:48:47.557990826] (+0.000005052) raw_syscalls:sys_exit: { }, { perf_ip = 0xFFFFFFFF81020FBA, perf_tid = 5093, perf_pid = 5093, perf_id = 1512, perf_cpu = 3, perf_period = 1 }
> > <SNIP>
> > 
> > But then I apply this patch (convert tracepoint events fields into CTF event fields) and:
> > 
> > [acme@ssdandy linux]$ perf data convert --to-ctf=./ctf-data/
> > [ perf data convert: Converted 'perf.data' into CTF data './ctf-data/' ]
> > [ perf data convert: Converted and wrote 0.009 MB (88 samples) ]
> > [acme@ssdandy linux]$ babeltrace ./ctf-data/
> > [error] Packet size (18446744073709551615 bits) is larger than remaining file size (262144 bits).
> > [error] Stream index creation error.
> > [error] Open file stream error.
> > [warning] [Context] Cannot open_trace of format ctf at path ./ctf-data.
> > [warning] [Context] cannot open trace "./ctf-data" from ./ctf-data/ for reading.
> > [error] Cannot open any trace for reading.
> > 
> > [error] opening trace "./ctf-data/" for reading.
> > 
> > [error] none of the specified trace paths could be opened.
> > 
> > [acme@ssdandy linux]$
> > 
> > It stops working.
> > 
> > [acme@ssdandy linux]$ ls -la ctf-data/
> > total 44
> > drwxrwx---.  2 acme acme    41 Fev 25 16:12 .
> > drwxrwxr-x. 28 acme acme  4096 Fev 25 16:19 ..
> > -rw-rw----.  1 acme acme  4666 Fev 25 16:21 metadata
> > -rw-rw----.  1 acme acme 32768 Fev 25 16:21 perf_stream_0
> > [acme@ssdandy linux]$
> > 
> > Can you try to reproduce this? The ctf-data/metadata file is below:
> 
> hum, i just tried and can't reproduce this one.. anychance of mixed
> babeltrace installations? How did you install babeltrace sources?

I will recheck
 
> I assume this was some standard Fedora latest something system right?

This was on RHEL7
 
> jirka
> 
> 
> git HEAD:
> 
> [root@krava perf]# git log --oneline | head -3
> 9534f89 perf data: Add tracepoint events fields CTF conversion support
> bb848d1 perf stat: Report unsupported events properly
> ffdfa43 perf tools: Compare JOBS to 0 after grep
> 
> test:
> 
> [root@krava perf]# LD_LIBRARY_PATH=/opt/libbabeltrace/lib/ ./perf trace record usleep 1
> [ perf record: Woken up 1 times to write data ]
> [ perf record: Captured and wrote 0.023 MB perf.data (88 samples) ]
> [root@krava perf]# LD_LIBRARY_PATH=/opt/libbabeltrace/lib/ ./perf data convert --to-ctf=./ctf-data
> [ perf data convert: Converted 'perf.data' into CTF data './ctf-data' ]
> [ perf data convert: Converted and wrote 0.009 MB (88 samples) ]
> [root@krava perf]# babeltrace ./ctf-data/
> [10:04:03.320544407] (+?.?????????) raw_syscalls:sys_exit: { }, { perf_ip = 0xFFFFFFFF810235AC, perf_tid = 23899, perf_pid = 23899, perf_id = 82, perf_cpu = 3, perf_period = 1, common_type = 75, common_flags = 0, common_preempt_count = 0, common_pid = 23899, id = 59, ret = 0 }
> [10:04:03.320568054] (+0.000023647) raw_syscalls:sys_enter: { }, { perf_ip = 0xFFFFFFFF81023420, perf_tid = 23899, perf_pid = 23899, perf_id = 78, perf_cpu = 3, perf_period = 1, common_type = 76, common_flags = 0, common_preempt_count = 0, common_pid = 23899, id = 12, args = [ [0] = 0x0, [1] = 0x0, [2] = 0x0, [3] = 0x0, [4] = 0x37F, [5] = 0x64 ] }
> [10:04:03.320568993] (+0.000000939) raw_syscalls:sys_exit: { }, { perf_ip = 0xFFFFFFFF810235AC, perf_tid = 23899, perf_pid = 23899, perf_id = 82, perf_cpu = 3, perf_period = 1, common_type = 75, common_flags = 0, common_preempt_count = 0, common_pid = 23899, id = 12, ret = 24440832 }
> ...

Re: [PATCH 05/11] perf data: Add tracepoint events fields CTF conversion support

[Jiri Olsa]

On Mon, Mar 02, 2015 at 12:32:20PM -0300, Arnaldo Carvalho de Melo wrote:

SNIP

> > > 
> > > Can you try to reproduce this? The ctf-data/metadata file is below:
> > 
> > hum, i just tried and can't reproduce this one.. anychance of mixed
> > babeltrace installations? How did you install babeltrace sources?
> 
> I will recheck
>  
> > I assume this was some standard Fedora latest something system right?
> 
> This was on RHEL7
>  

hum, how did you compile babeltrace on RHEL7 im getting following error:

[root@intel-s3ea2-09 babeltrace]# ./bootstrap 
+ '[' '!' -e config ']'
+ libtoolize --force --copy
libtoolize: putting auxiliary files in AC_CONFIG_AUX_DIR, `config'.
libtoolize: copying file `config/ltmain.sh'
libtoolize: putting macros in AC_CONFIG_MACRO_DIR, `m4'.
libtoolize: copying file `m4/libtool.m4'
libtoolize: copying file `m4/ltoptions.m4'
libtoolize: copying file `m4/ltsugar.m4'
libtoolize: copying file `m4/ltversion.m4'
libtoolize: copying file `m4/lt~obsolete.m4'
+ autoreconf -i
configure.ac:48: warning: macro 'AM_PATH_GLIB_2_0' not found in library
configure.ac:48: warning: macro 'AM_PATH_GLIB_2_0' not found in library
configure.ac:48: error: possibly undefined macro: AM_PATH_GLIB_2_0
      If this token and others are legitimate, please use m4_pattern_allow.
      See the Autoconf documentation.
autoreconf: /usr/bin/autoconf failed with exit status: 1


btw I put all remaining (already acked) changes into 'perf/ctf' branch
(without those 3 patches that need redo)

jirka

Re: [PATCH 05/11] perf data: Add tracepoint events fields CTF conversion support

[Arnaldo Carvalho de Melo]

Em Mon, Mar 09, 2015 at 01:12:04PM +0100, Jiri Olsa escreveu:
> On Mon, Mar 02, 2015 at 12:32:20PM -0300, Arnaldo Carvalho de Melo wrote:
> 
> SNIP
> 
> > > > 
> > > > Can you try to reproduce this? The ctf-data/metadata file is below:
> > > 
> > > hum, i just tried and can't reproduce this one.. anychance of mixed
> > > babeltrace installations? How did you install babeltrace sources?
> > 
> > I will recheck
> >  
> > > I assume this was some standard Fedora latest something system right?
> > 
> > This was on RHEL7
> >  
> 
> hum, how did you compile babeltrace on RHEL7 im getting following error:
> 
> [root@intel-s3ea2-09 babeltrace]# ./bootstrap 
> + '[' '!' -e config ']'
> + libtoolize --force --copy
> libtoolize: putting auxiliary files in AC_CONFIG_AUX_DIR, `config'.
> libtoolize: copying file `config/ltmain.sh'
> libtoolize: putting macros in AC_CONFIG_MACRO_DIR, `m4'.
> libtoolize: copying file `m4/libtool.m4'
> libtoolize: copying file `m4/ltoptions.m4'
> libtoolize: copying file `m4/ltsugar.m4'
> libtoolize: copying file `m4/ltversion.m4'
> libtoolize: copying file `m4/lt~obsolete.m4'
> + autoreconf -i
> configure.ac:48: warning: macro 'AM_PATH_GLIB_2_0' not found in library
> configure.ac:48: warning: macro 'AM_PATH_GLIB_2_0' not found in library
> configure.ac:48: error: possibly undefined macro: AM_PATH_GLIB_2_0
>       If this token and others are legitimate, please use m4_pattern_allow.
>       See the Autoconf documentation.
> autoreconf: /usr/bin/autoconf failed with exit status: 1

Trying again, freshly cloned babeltrace:

[acme@ssdandy git]$ git clone git://git.efficios.com/babeltrace.git
Cloning into 'babeltrace'...
remote: Counting objects: 10295, done.
remote: Compressing objects: 100% (3900/3900), done.
remote: Total 10295 (delta 7295), reused 8784 (delta 6112)
Receiving objects: 100% (10295/10295), 2.35 MiB | 537.00 KiB/s, done.
Resolving deltas: 100% (7295/7295), done.
[acme@ssdandy git]$
[acme@ssdandy linux]$ cd ../babeltrace/
[acme@ssdandy babeltrace]$ ./bootstrap 
+ '[' '!' -e config ']'
+ mkdir config
+ libtoolize --force --copy
libtoolize: putting auxiliary files in AC_CONFIG_AUX_DIR, `config'.
libtoolize: copying file `config/ltmain.sh'
libtoolize: putting macros in AC_CONFIG_MACRO_DIR, `m4'.
libtoolize: copying file `m4/libtool.m4'
libtoolize: copying file `m4/ltoptions.m4'
libtoolize: copying file `m4/ltsugar.m4'
libtoolize: copying file `m4/ltversion.m4'
libtoolize: copying file `m4/lt~obsolete.m4'
+ autoreconf -i
configure.ac:11: installing 'config/config.guess'
configure.ac:11: installing 'config/config.sub'
configure.ac:13: installing 'config/install-sh'
configure.ac:13: installing 'config/missing'
bindings/python/Makefile.am: installing 'config/depcomp'
bindings/python/Makefile.am:9: installing 'config/py-compile'
configure.ac: installing 'config/ylwrap'
[acme@ssdandy babeltrace]$ cat /etc/redhat-release 
Red Hat Enterprise Linux Server release 7.1 (Maipo)
[acme@ssdandy babeltrace]$
[acme@ssdandy babeltrace]$ ./configure --prefix=/opt/libbabeltrace
checking build system type... x86_64-unknown-linux-gnu
checking host system type... x86_64-unknown-linux-gnu
checking target system type... x86_64-unknown-linux-gnu
checking for a BSD-compatible install... /usr/bin/install -c
checking whether build environment is sane... yes
checking for a thread-safe mkdir -p... /usr/bin/mkdir -p

<SNIP>

config.status: creating babeltrace-ctf.pc
config.status: creating config.h
config.status: executing depfiles commands
config.status: executing libtool commands

[acme@ssdandy babeltrace]$ make prefix=/opt/libbabeltrace
<SNIP>
[acme@dandy babeltrace]$ sudo make install prefix=/opt/libbabeltrace


All looks fine, trying to build...


[acme@ssdandy linux]$ rm -rf /tmp/build/perf ; mkdir -p /tmp/build/perf
; time make LIBBABELTRACE_DIR=/opt/libbabeltrace O=/tmp/build/perf -C
tools/perf install-bin
make: Entering directory `/home/acme/git/linux/tools/perf'
  BUILD:   Doing 'make -j8' parallel build
config/Makefile:425: No libunwind found. Please install
libunwind-dev[el] >= 1.1 and/or set LIBUNWIND_DIR
config/Makefile:709: No libbabeltrace found, disables 'perf data' CTF
format support, please install
libbabeltrace-dev[el]/libbabeltrace-ctf-dev

Auto-detecting system features:
...                         dwarf: [ on  ]
...                         glibc: [ on  ]
...                          gtk2: [ on  ]
...                      libaudit: [ on  ]
...                        libbfd: [ on  ]
...                        libelf: [ on  ]
...                       libnuma: [ on  ]
...                       libperl: [ on  ]
...                     libpython: [ on  ]
...                      libslang: [ on  ]
...                     libunwind: [ OFF ]
...            libdw-dwarf-unwind: [ on  ]
...                 libbabeltrace: [ OFF ]


Doesn't work... Will try not using O=:

[acme@ssdandy perf]$ time make LIBBABELTRACE_DIR=/opt/libbabeltrace
install-bin
  BUILD:   Doing 'make -j8' parallel build
config/Makefile:425: No libunwind found. Please install
libunwind-dev[el] >= 1.1 and/or set LIBUNWIND_DIR
config/Makefile:709: No libbabeltrace found, disables 'perf data' CTF
format support, please install
libbabeltrace-dev[el]/libbabeltrace-ctf-dev

Auto-detecting system features:
...                         dwarf: [ on  ]
...                         glibc: [ on  ]
...                          gtk2: [ on  ]
...                      libaudit: [ on  ]
...                        libbfd: [ on  ]
...                        libelf: [ on  ]
...                       libnuma: [ on  ]
...                       libperl: [ on  ]
...                     libpython: [ on  ]
...                      libslang: [ on  ]
...                     libunwind: [ OFF ]
...            libdw-dwarf-unwind: [ on  ]
...                 libbabeltrace: [ OFF ]
...                          zlib: [ on  ]
...     DWARF post unwind library: libdw


Doesn't work as well...

:-\

Will try debugging...

- Arnaldo

Re: [PATCH 05/11] perf data: Add tracepoint events fields CTF conversion support

[Arnaldo Carvalho de Melo]

Em Mon, Mar 09, 2015 at 06:51:21PM -0300, Arnaldo Carvalho de Melo escreveu:
> Doesn't work as well...
> 
> :-\
> 
> Will try debugging...

So I tried checkout out before Ingo's changes to libbabeltrace detection, i.e.:

[acme@zoo linux]$ git checkout -b ttmp 9a75606ca06d94aab1ed0dbe96935e3f89dfb81c

And it works:

[acme@zoo linux]$ git checkout ttmp
Switched to branch 'ttmp'
[acme@zoo linux]$ rm -rf /tmp/build/perf ; mkdir -p /tmp/build/perf ; time make LIBBABELTRACE_DIR=/opt/libbabeltrace/ -C tools/perf O=/tmp/build/perf install-bin
make: Entering directory `/home/git/linux/tools/perf'
  BUILD:   Doing 'make -j4' parallel build

Auto-detecting system features:
...                         dwarf: [ on  ]
...                         glibc: [ on  ]
...                          gtk2: [ on  ]
...                      libaudit: [ on  ]
...                        libbfd: [ on  ]
...                        libelf: [ on  ]
...                       libnuma: [ on  ]
...                       libperl: [ on  ]
...                     libpython: [ on  ]
...                      libslang: [ on  ]
...                     libunwind: [ on  ]
...            libdw-dwarf-unwind: [ on  ]
...                 libbabeltrace: [ on  ]
...                          zlib: [ on  ]
...     DWARF post unwind library: libunwind

  MKDIR    /tmp/build/perf/util/

But if I checkout tip/perf/core:

[acme@zoo linux]$ git checkout -b ttmp2 tip/perf/core
Branch ttmp2 set up to track remote branch perf/core from tip.
Switched to a new branch 'ttmp2'
[acme@zoo linux]$ rm -rf /tmp/build/perf ; mkdir -p /tmp/build/perf ; time make LIBBABELTRACE_DIR=/opt/libbabeltrace/ -C tools/perf O=/tmp/build/perf install-bin
make: Entering directory `/home/git/linux/tools/perf'
  BUILD:   Doing 'make -j4' parallel build
config/Makefile:709: No libbabeltrace found, disables 'perf data' CTF format support, please install libbabeltrace-dev[el]/libbabeltrace-ctf-dev

Auto-detecting system features:
...                         dwarf: [ on  ]
...                         glibc: [ on  ]
...                          gtk2: [ on  ]
...                      libaudit: [ on  ]
...                        libbfd: [ on  ]
...                        libelf: [ on  ]
...                       libnuma: [ on  ]
...                       libperl: [ on  ]
...                     libpython: [ on  ]
...                      libslang: [ on  ]
...                     libunwind: [ on  ]
...            libdw-dwarf-unwind: [ on  ]
...                 libbabeltrace: [ OFF ]
...                          zlib: [ on  ]
...     DWARF post unwind library: libunwind

  MKDIR    /tmp/build/perf/util/

It doesn't work, bisecting:

[acme@zoo linux]$ git bisect bad
b49f1a4be701c2386ccc7496dc8442cf26424d5c is the first bad commit
commit b49f1a4be701c2386ccc7496dc8442cf26424d5c
Author: Ingo Molnar <mingo@kernel.org>
Date:   Sat Feb 28 10:16:27 2015 +0100

    perf tools: Improve feature test debuggability
    
    Certain feature tests fail with link errors:

--------------------------------------------------------------

Ran out of time, will continue later/tomorrow.

- Arnaldo

Re: [PATCH 05/11] perf data: Add tracepoint events fields CTF conversion support

[Arnaldo Carvalho de Melo]

Em Mon, Mar 09, 2015 at 08:11:19PM -0300, Arnaldo Carvalho de Melo escreveu:
> Em Mon, Mar 09, 2015 at 06:51:21PM -0300, Arnaldo Carvalho de Melo escreveu:
> > Doesn't work as well...
> > 
> > :-\
> > 
> > Will try debugging...
> 
> So I tried checkout out before Ingo's changes to libbabeltrace detection, i.e.:
> 
> [acme@zoo linux]$ git checkout -b ttmp 9a75606ca06d94aab1ed0dbe96935e3f89dfb81c
> 
> And it works:

So it seems its due to the -Werror
 
> [acme@zoo linux]$ git bisect bad
> b49f1a4be701c2386ccc7496dc8442cf26424d5c is the first bad commit
> commit b49f1a4be701c2386ccc7496dc8442cf26424d5c
> Author: Ingo Molnar <mingo@kernel.org>
> Date:   Sat Feb 28 10:16:27 2015 +0100
> 
>     perf tools: Improve feature test debuggability
>     
>     Certain feature tests fail with link errors:
> 
> --------------------------------------------------------------
> 
> Ran out of time, will continue later/tomorrow.

[acme@zoo linux]$ git diff
diff --git a/tools/perf/config/Makefile b/tools/perf/config/Makefile
index e2350ada6ccd..d46e0ef45417 100644
--- a/tools/perf/config/Makefile
+++ b/tools/perf/config/Makefile
@@ -184,7 +184,7 @@ endif
 
 feature_check = $(eval $(feature_check_code))
 define feature_check_code
-  feature-$(1) := $(shell $(MAKE) OUTPUT=$(OUTPUT_FEATURES) CFLAGS="$(EXTRA_CFLAGS) $(FEATURE_CHECK_CFLAGS-$(1))" LDFLAGS="$(LDFLAGS) $(FEATURE_CHECK_LDFLAGS-$(1))" -C config/feature-checks test-$1.bin >/dev/null 2>/dev/null && echo 1 ||
+  feature-$(1) := $(shell $(MAKE) OUTPUT=$(OUTPUT_FEATURES) CFLAGS="$(EXTRA_CFLAGS) $(FEATURE_CHECK_CFLAGS-$(1))" LDFLAGS="$(LDFLAGS) $(FEATURE_CHECK_LDFLAGS-$(1))" -C config/feature-checks test-$1.bin >/tmp/output-$(1) 2>/dev/null && ec
 endef
 
 feature_set = $(eval $(feature_set_code))
[acme@zoo linux]$ cd /home/git/linux/tools/perf/config/feature-checks
[acme@zoo feature-checks]$ gcc -MD -I/opt/libbabeltrace//include -Wall -Werror -o /tmp/build/perf/config/feature-checks/test-libbabeltrace.bin test-libbabeltrace.c -Wl,-z,noexecstack -L/opt/libbabeltrace//lib -lbabeltrace-ctf
test-libbabeltrace.c: In function ‘main’:
test-libbabeltrace.c:6:2: error: implicit declaration of function ‘bt_ctf_stream_class_get_packet_context_type’ [-Werror=implicit-function-declaration]
  bt_ctf_stream_class_get_packet_context_type((void *) 0);
  ^
cc1: all warnings being treated as errors
[acme@zoo feature-checks]$

[root@zoo ~]# find /opt/libbabeltrace/include/babeltrace/ -type f | xargs grep bt_ctf_stream_class_get_packet_context_type
/opt/libbabeltrace/include/babeltrace/ctf-ir/stream-class.h: * bt_ctf_stream_class_get_packet_context_type: get the stream class' packet
/opt/libbabeltrace/include/babeltrace/ctf-ir/stream-class.h:extern struct bt_ctf_field_type *bt_ctf_stream_class_get_packet_context_type(
[root@zoo ~]#

That seems to be included from the file included in the feature test code :-\

Really ran out of time now...

- Arnaldo

[PATCH] perf build: Fix libbabeltrace detection

[Jiri Olsa]

On Mon, Mar 09, 2015 at 08:28:45PM -0300, Arnaldo Carvalho de Melo wrote:
> Em Mon, Mar 09, 2015 at 08:11:19PM -0300, Arnaldo Carvalho de Melo escreveu:
> > Em Mon, Mar 09, 2015 at 06:51:21PM -0300, Arnaldo Carvalho de Melo escreveu:

SNIP

> [root@zoo ~]# find /opt/libbabeltrace/include/babeltrace/ -type f | xargs grep bt_ctf_stream_class_get_packet_context_type
> /opt/libbabeltrace/include/babeltrace/ctf-ir/stream-class.h: * bt_ctf_stream_class_get_packet_context_type: get the stream class' packet
> /opt/libbabeltrace/include/babeltrace/ctf-ir/stream-class.h:extern struct bt_ctf_field_type *bt_ctf_stream_class_get_packet_context_type(
> [root@zoo ~]#
> 
> That seems to be included from the file included in the feature test code :-\
> 
> Really ran out of time now...

ouch, thanks a lot for debugging.. the reason is indeed the -Werror added by:

  b49f1a4be701 perf tools: Improve feature test debuggability

following patch seems to fix this for me


thanks,
jirka

---
Following patch added -Werror for feature builds:
  b49f1a4be701 perf tools: Improve feature test debuggability

and broke libbabeltrace feature build, because it was including
wrong header and gcc couldn't find the used symbol definition.

Adding proper header and keeping the old one as it is needed
also (libbabeltrace quirk).

Signed-off-by: Jiri Olsa <jolsa@kernel.org>
---
 tools/perf/config/feature-checks/test-libbabeltrace.c | 1 +
 1 file changed, 1 insertion(+)

diff --git a/tools/perf/config/feature-checks/test-libbabeltrace.c b/tools/perf/config/feature-checks/test-libbabeltrace.c
index 3b7dd68a4d52..9cf802a04885 100644
--- a/tools/perf/config/feature-checks/test-libbabeltrace.c
+++ b/tools/perf/config/feature-checks/test-libbabeltrace.c
@@ -1,5 +1,6 @@
 
 #include <babeltrace/ctf-writer/writer.h>
+#include <babeltrace/ctf-ir/stream-class.h>
 
 int main(void)
 {
-- 
1.9.3

Re: [PATCH] perf build: Fix libbabeltrace detection

[Arnaldo Carvalho de Melo]

Em Tue, Mar 10, 2015 at 01:00:35PM +0100, Jiri Olsa escreveu:
> On Mon, Mar 09, 2015 at 08:28:45PM -0300, Arnaldo Carvalho de Melo wrote:
> > Em Mon, Mar 09, 2015 at 08:11:19PM -0300, Arnaldo Carvalho de Melo escreveu:
> > > Em Mon, Mar 09, 2015 at 06:51:21PM -0300, Arnaldo Carvalho de Melo escreveu:
> > [root@zoo ~]# find /opt/libbabeltrace/include/babeltrace/ -type f | xargs grep bt_ctf_stream_class_get_packet_context_type
> > /opt/libbabeltrace/include/babeltrace/ctf-ir/stream-class.h:extern struct bt_ctf_field_type *bt_ctf_stream_class_get_packet_context_type(

> > That seems to be included from the file included in the feature test code :-\
> > Really ran out of time now...
 
> ouch, thanks a lot for debugging.. the reason is indeed the -Werror added by:
>   b49f1a4be701 perf tools: Improve feature test debuggability
 
> following patch seems to fix this for me
> ---
> Following patch added -Werror for feature builds:
>   b49f1a4be701 perf tools: Improve feature test debuggability
 
> and broke libbabeltrace feature build, because it was including
> wrong header and gcc couldn't find the used symbol definition.
 
> Adding proper header and keeping the old one as it is needed
> also (libbabeltrace quirk).

> +++ b/tools/perf/config/feature-checks/test-libbabeltrace.c
>  #include <babeltrace/ctf-writer/writer.h>
> +#include <babeltrace/ctf-ir/stream-class.h>

Right, that was the fix I found as well, well, I was trying to include
_just_ babeltrace/ctf-ir/stream-class.h, but that is buggy in that it
doesn't include the header file with the uint32_t and int64_t types:

[acme@zoo linux]$ cat /tmp/build/perf/config/feature-checks/make-libbabeltrace 
make[1]: Entering directory `/home/git/linux/tools/perf/config/feature-checks'
gcc -MD -I/opt/libbabeltrace//include -Wall -Werror -o /tmp/build/perf/config/feature-checks/test-libbabeltrace.bin test-libbabeltrace.c -Wl,-z,noexecstack -L/opt/libbabeltrace//lib -lbabeltrace-ctf # -lbabeltrace provided by 
In file included from test-libbabeltrace.c:2:0:
/opt/libbabeltrace//include/babeltrace/ctf-ir/stream-class.h:118:1: error: unknown type name ‘int64_t’
 extern int64_t bt_ctf_stream_class_get_id(
 ^
/opt/libbabeltrace//include/babeltrace/ctf-ir/stream-class.h:134:45: error: unknown type name ‘uint32_t’
   struct bt_ctf_stream_class *stream_class, uint32_t id);
                                             ^
make[1]: *** [test-libbabeltrace.bin] Error 1
make[1]: Leaving directory `/home/git/linux/tools/perf/config/feature-checks'
[acme@zoo linux]$ 

>  
>  int main(void)
>  {
> -- 
> 1.9.3

Ah, I found the following patch useful in debugging this, if you're ok with it,
I'll put it in my tree with your Acked-by, ok?

- Arnaldo

[acme@zoo linux]$ git diff tools/perf/config/Makefile
diff --git a/tools/perf/config/Makefile b/tools/perf/config/Makefile
index e2350ada6ccd..4992c39652e6 100644
--- a/tools/perf/config/Makefile
+++ b/tools/perf/config/Makefile
@@ -184,7 +184,7 @@ endif
 
 feature_check = $(eval $(feature_check_code))
 define feature_check_code
-  feature-$(1) := $(shell $(MAKE) OUTPUT=$(OUTPUT_FEATURES) CFLAGS="$(EXTRA_CFLAGS) $(FEATURE_CHECK_CFLAGS-$(1))" LDFLAGS="$(LDFLAGS) $(FEATURE_CHECK_LDFLAGS-$(1))" -C config/feature-checks test-$1.bin >/dev/null 2>/dev/null && echo 1 ||
+  feature-$(1) := $(shell $(MAKE) OUTPUT=$(OUTPUT_FEATURES) CFLAGS="$(EXTRA_CFLAGS) $(FEATURE_CHECK_CFLAGS-$(1))" LDFLAGS="$(LDFLAGS) $(FEATURE_CHECK_LDFLAGS-$(1))" -C config/feature-checks test-$1.bin > $(OUTPUT_FEATURES)make-$(1) 2>&1 
 endef
 
 feature_set = $(eval $(feature_set_code))
[acme@zoo linux]$ 

Re: [PATCH] perf build: Fix libbabeltrace detection

[Jiri Olsa]

On Tue, Mar 10, 2015 at 11:01:34AM -0300, Arnaldo Carvalho de Melo wrote:
> Em Tue, Mar 10, 2015 at 01:00:35PM +0100, Jiri Olsa escreveu:
> > On Mon, Mar 09, 2015 at 08:28:45PM -0300, Arnaldo Carvalho de Melo wrote:
> > > Em Mon, Mar 09, 2015 at 08:11:19PM -0300, Arnaldo Carvalho de Melo escreveu:
> > > > Em Mon, Mar 09, 2015 at 06:51:21PM -0300, Arnaldo Carvalho de Melo escreveu:
> > > [root@zoo ~]# find /opt/libbabeltrace/include/babeltrace/ -type f | xargs grep bt_ctf_stream_class_get_packet_context_type
> > > /opt/libbabeltrace/include/babeltrace/ctf-ir/stream-class.h:extern struct bt_ctf_field_type *bt_ctf_stream_class_get_packet_context_type(
> 
> > > That seems to be included from the file included in the feature test code :-\
> > > Really ran out of time now...
>  
> > ouch, thanks a lot for debugging.. the reason is indeed the -Werror added by:
> >   b49f1a4be701 perf tools: Improve feature test debuggability
>  
> > following patch seems to fix this for me
> > ---
> > Following patch added -Werror for feature builds:
> >   b49f1a4be701 perf tools: Improve feature test debuggability
>  
> > and broke libbabeltrace feature build, because it was including
> > wrong header and gcc couldn't find the used symbol definition.
>  
> > Adding proper header and keeping the old one as it is needed
> > also (libbabeltrace quirk).
> 
> > +++ b/tools/perf/config/feature-checks/test-libbabeltrace.c
> >  #include <babeltrace/ctf-writer/writer.h>
> > +#include <babeltrace/ctf-ir/stream-class.h>
> 
> Right, that was the fix I found as well, well, I was trying to include
> _just_ babeltrace/ctf-ir/stream-class.h, but that is buggy in that it
> doesn't include the header file with the uint32_t and int64_t types:

yep, thats the 'libbabeltrace quirk' I mentioned in the changelog ;-)

> 
> [acme@zoo linux]$ cat /tmp/build/perf/config/feature-checks/make-libbabeltrace 
> make[1]: Entering directory `/home/git/linux/tools/perf/config/feature-checks'
> gcc -MD -I/opt/libbabeltrace//include -Wall -Werror -o /tmp/build/perf/config/feature-checks/test-libbabeltrace.bin test-libbabeltrace.c -Wl,-z,noexecstack -L/opt/libbabeltrace//lib -lbabeltrace-ctf # -lbabeltrace provided by 
> In file included from test-libbabeltrace.c:2:0:
> /opt/libbabeltrace//include/babeltrace/ctf-ir/stream-class.h:118:1: error: unknown type name ‘int64_t’
>  extern int64_t bt_ctf_stream_class_get_id(
>  ^
> /opt/libbabeltrace//include/babeltrace/ctf-ir/stream-class.h:134:45: error: unknown type name ‘uint32_t’
>    struct bt_ctf_stream_class *stream_class, uint32_t id);
>                                              ^
> make[1]: *** [test-libbabeltrace.bin] Error 1
> make[1]: Leaving directory `/home/git/linux/tools/perf/config/feature-checks'
> [acme@zoo linux]$ 
> 
> >  
> >  int main(void)
> >  {
> > -- 
> > 1.9.3
> 
> Ah, I found the following patch useful in debugging this, if you're ok with it,
> I'll put it in my tree with your Acked-by, ok?
> 
> - Arnaldo
> 
> [acme@zoo linux]$ git diff tools/perf/config/Makefile
> diff --git a/tools/perf/config/Makefile b/tools/perf/config/Makefile
> index e2350ada6ccd..4992c39652e6 100644
> --- a/tools/perf/config/Makefile
> +++ b/tools/perf/config/Makefile
> @@ -184,7 +184,7 @@ endif
>  
>  feature_check = $(eval $(feature_check_code))
>  define feature_check_code
> -  feature-$(1) := $(shell $(MAKE) OUTPUT=$(OUTPUT_FEATURES) CFLAGS="$(EXTRA_CFLAGS) $(FEATURE_CHECK_CFLAGS-$(1))" LDFLAGS="$(LDFLAGS) $(FEATURE_CHECK_LDFLAGS-$(1))" -C config/feature-checks test-$1.bin >/dev/null 2>/dev/null && echo 1 ||
> +  feature-$(1) := $(shell $(MAKE) OUTPUT=$(OUTPUT_FEATURES) CFLAGS="$(EXTRA_CFLAGS) $(FEATURE_CHECK_CFLAGS-$(1))" LDFLAGS="$(LDFLAGS) $(FEATURE_CHECK_LDFLAGS-$(1))" -C config/feature-checks test-$1.bin > $(OUTPUT_FEATURES)make-$(1) 2>&1 
>  endef

ack, that seem usefull.. I always redirected this to '>> /tmp/krava'
never got enough will to make it nice and generic ;-)

thanks,
jirka

Re: [PATCH] perf build: Fix libbabeltrace detection

[Jérémie Galarneau]

On Tue, Mar 10, 2015 at 10:11 AM, Jiri Olsa <jolsa@redhat.com> wrote:
> On Tue, Mar 10, 2015 at 11:01:34AM -0300, Arnaldo Carvalho de Melo wrote:
>> Em Tue, Mar 10, 2015 at 01:00:35PM +0100, Jiri Olsa escreveu:
>> > On Mon, Mar 09, 2015 at 08:28:45PM -0300, Arnaldo Carvalho de Melo wrote:
>> > > Em Mon, Mar 09, 2015 at 08:11:19PM -0300, Arnaldo Carvalho de Melo escreveu:
>> > > > Em Mon, Mar 09, 2015 at 06:51:21PM -0300, Arnaldo Carvalho de Melo escreveu:
>> > > [root@zoo ~]# find /opt/libbabeltrace/include/babeltrace/ -type f | xargs grep bt_ctf_stream_class_get_packet_context_type
>> > > /opt/libbabeltrace/include/babeltrace/ctf-ir/stream-class.h:extern struct bt_ctf_field_type *bt_ctf_stream_class_get_packet_context_type(
>>
>> > > That seems to be included from the file included in the feature test code :-\
>> > > Really ran out of time now...
>>
>> > ouch, thanks a lot for debugging.. the reason is indeed the -Werror added by:
>> >   b49f1a4be701 perf tools: Improve feature test debuggability
>>
>> > following patch seems to fix this for me
>> > ---
>> > Following patch added -Werror for feature builds:
>> >   b49f1a4be701 perf tools: Improve feature test debuggability
>>
>> > and broke libbabeltrace feature build, because it was including
>> > wrong header and gcc couldn't find the used symbol definition.
>>
>> > Adding proper header and keeping the old one as it is needed
>> > also (libbabeltrace quirk).
>>
>> > +++ b/tools/perf/config/feature-checks/test-libbabeltrace.c
>> >  #include <babeltrace/ctf-writer/writer.h>
>> > +#include <babeltrace/ctf-ir/stream-class.h>
>>
>> Right, that was the fix I found as well, well, I was trying to include
>> _just_ babeltrace/ctf-ir/stream-class.h, but that is buggy in that it
>> doesn't include the header file with the uint32_t and int64_t types:
>
> yep, thats the 'libbabeltrace quirk' I mentioned in the changelog ;-)
>

Seems I missed that being discussed. Fixed in Babeltrace master as of 48d711aa2.

Jérémie

>>
>> [acme@zoo linux]$ cat /tmp/build/perf/config/feature-checks/make-libbabeltrace
>> make[1]: Entering directory `/home/git/linux/tools/perf/config/feature-checks'
>> gcc -MD -I/opt/libbabeltrace//include -Wall -Werror -o /tmp/build/perf/config/feature-checks/test-libbabeltrace.bin test-libbabeltrace.c -Wl,-z,noexecstack -L/opt/libbabeltrace//lib -lbabeltrace-ctf # -lbabeltrace provided by
>> In file included from test-libbabeltrace.c:2:0:
>> /opt/libbabeltrace//include/babeltrace/ctf-ir/stream-class.h:118:1: error: unknown type name ‘int64_t’
>>  extern int64_t bt_ctf_stream_class_get_id(
>>  ^
>> /opt/libbabeltrace//include/babeltrace/ctf-ir/stream-class.h:134:45: error: unknown type name ‘uint32_t’
>>    struct bt_ctf_stream_class *stream_class, uint32_t id);
>>                                              ^
>> make[1]: *** [test-libbabeltrace.bin] Error 1
>> make[1]: Leaving directory `/home/git/linux/tools/perf/config/feature-checks'
>> [acme@zoo linux]$
>>
>> >
>> >  int main(void)
>> >  {
>> > --
>> > 1.9.3
>>
>> Ah, I found the following patch useful in debugging this, if you're ok with it,
>> I'll put it in my tree with your Acked-by, ok?
>>
>> - Arnaldo
>>
>> [acme@zoo linux]$ git diff tools/perf/config/Makefile
>> diff --git a/tools/perf/config/Makefile b/tools/perf/config/Makefile
>> index e2350ada6ccd..4992c39652e6 100644
>> --- a/tools/perf/config/Makefile
>> +++ b/tools/perf/config/Makefile
>> @@ -184,7 +184,7 @@ endif
>>
>>  feature_check = $(eval $(feature_check_code))
>>  define feature_check_code
>> -  feature-$(1) := $(shell $(MAKE) OUTPUT=$(OUTPUT_FEATURES) CFLAGS="$(EXTRA_CFLAGS) $(FEATURE_CHECK_CFLAGS-$(1))" LDFLAGS="$(LDFLAGS) $(FEATURE_CHECK_LDFLAGS-$(1))" -C config/feature-checks test-$1.bin >/dev/null 2>/dev/null && echo 1 ||
>> +  feature-$(1) := $(shell $(MAKE) OUTPUT=$(OUTPUT_FEATURES) CFLAGS="$(EXTRA_CFLAGS) $(FEATURE_CHECK_CFLAGS-$(1))" LDFLAGS="$(LDFLAGS) $(FEATURE_CHECK_LDFLAGS-$(1))" -C config/feature-checks test-$1.bin > $(OUTPUT_FEATURES)make-$(1) 2>&1
>>  endef
>
> ack, that seem usefull.. I always redirected this to '>> /tmp/krava'
> never got enough will to make it nice and generic ;-)
>
> thanks,
> jirka



-- 
Jérémie Galarneau
EfficiOS Inc.
http://www.efficios.com

Re: [PATCH] perf build: Fix libbabeltrace detection

[Arnaldo Carvalho de Melo]

Em Tue, Mar 10, 2015 at 10:44:00AM -0400, Jérémie Galarneau escreveu:
> On Tue, Mar 10, 2015 at 10:11 AM, Jiri Olsa <jolsa@redhat.com> wrote:
> > On Tue, Mar 10, 2015 at 11:01:34AM -0300, Arnaldo Carvalho de Melo wrote:
> >> Em Tue, Mar 10, 2015 at 01:00:35PM +0100, Jiri Olsa escreveu:
> >> > Adding proper header and keeping the old one as it is needed
> >> > also (libbabeltrace quirk).
> >>
> >> > +#include <babeltrace/ctf-ir/stream-class.h>
> >>
> >> Right, that was the fix I found as well, well, I was trying to include
> >> _just_ babeltrace/ctf-ir/stream-class.h, but that is buggy in that it
> >> doesn't include the header file with the uint32_t and int64_t types:
> >
> > yep, thats the 'libbabeltrace quirk' I mentioned in the changelog ;-)

> Seems I missed that being discussed. Fixed in Babeltrace master as of 48d711aa2.

Thanks for doing it, just refreshed by babeltrace git repo and found it
there!

Will add Jiri's patch tho, so that it works with the older git repos.

Ah, since you guys adopted the Signed-off-by tag, consider adopting the
Reported-by as well 8-)

- Arnaldo

Re: [PATCH] perf build: Fix libbabeltrace detection

[Arnaldo Carvalho de Melo]

Em Tue, Mar 10, 2015 at 03:11:11PM +0100, Jiri Olsa escreveu:
> On Tue, Mar 10, 2015 at 11:01:34AM -0300, Arnaldo Carvalho de Melo wrote:
> > Em Tue, Mar 10, 2015 at 01:00:35PM +0100, Jiri Olsa escreveu:
> > > On Mon, Mar 09, 2015 at 08:28:45PM -0300, Arnaldo Carvalho de Melo wrote:
> > > > Em Mon, Mar 09, 2015 at 08:11:19PM -0300, Arnaldo Carvalho de Melo escreveu:
> > > +#include <babeltrace/ctf-ir/stream-class.h>

> > Right, that was the fix I found as well, well, I was trying to include
> > _just_ babeltrace/ctf-ir/stream-class.h, but that is buggy in that it
> > doesn't include the header file with the uint32_t and int64_t types:
> 
> yep, thats the 'libbabeltrace quirk' I mentioned in the changelog ;-)

:-)
 
> > +++ b/tools/perf/config/Makefile
> > @@ -184,7 +184,7 @@ endif
> >  
> >  feature_check = $(eval $(feature_check_code))
> >  define feature_check_code
> > -  feature-$(1) := $(shell $(MAKE) OUTPUT=$(OUTPUT_FEATURES) CFLAGS="$(EXTRA_CFLAGS) $(FEATURE_CHECK_CFLAGS-$(1))" LDFLAGS="$(LDFLAGS) $(FEATURE_CHECK_LDFLAGS-$(1))" -C config/feature-checks test-$1.bin >/dev/null 2>/dev/null && echo 1 ||
> > +  feature-$(1) := $(shell $(MAKE) OUTPUT=$(OUTPUT_FEATURES) CFLAGS="$(EXTRA_CFLAGS) $(FEATURE_CHECK_CFLAGS-$(1))" LDFLAGS="$(LDFLAGS) $(FEATURE_CHECK_LDFLAGS-$(1))" -C config/feature-checks test-$1.bin > $(OUTPUT_FEATURES)make-$(1) 2>&1 
> >  endef
> 
> ack, that seem usefull.. I always redirected this to '>> /tmp/krava'
> never got enough will to make it nice and generic ;-)

Ok, lemme do a stop-machine and collect all these fixes...

- Arnaldo

Re: [PATCH] perf build: Fix libbabeltrace detection

[Arnaldo Carvalho de Melo]

Em Tue, Mar 10, 2015 at 03:11:11PM +0100, Jiri Olsa escreveu:
> On Tue, Mar 10, 2015 at 11:01:34AM -0300, Arnaldo Carvalho de Melo wrote:
> > Ah, I found the following patch useful in debugging this, if you're ok with it,
> > I'll put it in my tree with your Acked-by, ok?
> > 
> > - Arnaldo
> > 
> > [acme@zoo linux]$ git diff tools/perf/config/Makefile
> > diff --git a/tools/perf/config/Makefile b/tools/perf/config/Makefile
> > index e2350ada6ccd..4992c39652e6 100644
> > --- a/tools/perf/config/Makefile
> > +++ b/tools/perf/config/Makefile
> > @@ -184,7 +184,7 @@ endif
> >  
> >  feature_check = $(eval $(feature_check_code))
> >  define feature_check_code
> > -  feature-$(1) := $(shell $(MAKE) OUTPUT=$(OUTPUT_FEATURES) CFLAGS="$(EXTRA_CFLAGS) $(FEATURE_CHECK_CFLAGS-$(1))" LDFLAGS="$(LDFLAGS) $(FEATURE_CHECK_LDFLAGS-$(1))" -C config/feature-checks test-$1.bin >/dev/null 2>/dev/null && echo 1 ||
> > +  feature-$(1) := $(shell $(MAKE) OUTPUT=$(OUTPUT_FEATURES) CFLAGS="$(EXTRA_CFLAGS) $(FEATURE_CHECK_CFLAGS-$(1))" LDFLAGS="$(LDFLAGS) $(FEATURE_CHECK_LDFLAGS-$(1))" -C config/feature-checks test-$1.bin > $(OUTPUT_FEATURES)make-$(1) 2>&1 
> >  endef
> 
> ack, that seem usefull.. I always redirected this to '>> /tmp/krava'
> never got enough will to make it nice and generic ;-)

Got stuck trying to get the 'clean' target done, argh, makefiles, trying
to use the OUTPUT_FEATURES to add a clean target on
tools/perf/config/Makefile, but then if I call it from the main
makefile... it will do feature detection again...

So I think I'll have to redefine it on the main Makefile to avoid having
to call tools/perf/config/Makefile again, right?

Anyway, after lunch I'll try again.

- Arnaldo


diff --git a/tools/perf/Makefile.perf b/tools/perf/Makefile.perf
index ec4c063ed9f3..8177a063ee0a 100644
--- a/tools/perf/Makefile.perf
+++ b/tools/perf/Makefile.perf
@@ -530,6 +530,7 @@ clean: $(LIBTRACEEVENT)-clean $(LIBAPI)-clean config-clean
 	$(call QUIET_CLEAN, core-progs) $(RM) $(ALL_PROGRAMS) perf perf-read-vdso32 perf-read-vdsox32
 	$(call QUIET_CLEAN, core-gen)   $(RM)  *.spec *.pyc *.pyo */*.pyc */*.pyo $(OUTPUT)common-cmds.h TAGS tags cscope* $(OUTPUT)PERF-VERSION-FILE $(OUTPUT)PERF-FEATURES $(OUTPUT)util/*-bison* $(OUTPUT)util/*-flex*
 	$(QUIET_SUBDIR0)Documentation $(QUIET_SUBDIR1) clean
+	$(QUIET_SUBDIR0)config $(QUIET_SUBDIR1) clean
 	$(python-clean)
 
 #
diff --git a/tools/perf/config/Makefile b/tools/perf/config/Makefile
index d44c64d64465..9a492b56984a 100644
--- a/tools/perf/config/Makefile
+++ b/tools/perf/config/Makefile
@@ -184,7 +184,7 @@ endif
 
 feature_check = $(eval $(feature_check_code))
 define feature_check_code
-  feature-$(1) := $(shell $(MAKE) OUTPUT=$(OUTPUT_FEATURES) CFLAGS="$(EXTRA_CFLAGS) $(FEATURE_CHECK_CFLAGS-$(1))" LDFLAGS="$(LDFLAGS) $(FEATURE_CHECK_LDFLAGS-$(1))" -C config/feature-checks test-$1.bin >/dev/null 2>/dev/null && echo 1 || echo 0)
+  feature-$(1) := $(shell $(MAKE) OUTPUT=$(OUTPUT_FEATURES) CFLAGS="$(EXTRA_CFLAGS) $(FEATURE_CHECK_CFLAGS-$(1))" LDFLAGS="$(LDFLAGS) $(FEATURE_CHECK_LDFLAGS-$(1))" -C config/feature-checks test-$1.bin > $(OUTPUT_FEATURES)make-$(1).feature-test.output 2>&1 && echo 1 || echo 0)
 endef
 
 feature_set = $(eval $(feature_set_code))
@@ -880,3 +880,9 @@ $(call detected_var,LIBDIR)
 $(call detected_var,GTK_CFLAGS)
 $(call detected_var,PERL_EMBED_CCOPTS)
 $(call detected_var,PYTHON_EMBED_CCOPTS)
+
+CLEAN_FILES = $(OUTPUT_FEATURES)make-*.feature-test.output
+
+clean:
+	$(call QUIET_CLEAN, config) $(RM) $(CLEAN_FILES)
+

Re: [PATCH] perf build: Fix libbabeltrace detection

[Jiri Olsa]

On Tue, Mar 10, 2015 at 01:04:26PM -0300, Arnaldo Carvalho de Melo wrote:

SNIP

>  endef
>  
>  feature_set = $(eval $(feature_set_code))
> @@ -880,3 +880,9 @@ $(call detected_var,LIBDIR)
>  $(call detected_var,GTK_CFLAGS)
>  $(call detected_var,PERL_EMBED_CCOPTS)
>  $(call detected_var,PYTHON_EMBED_CCOPTS)
> +
> +CLEAN_FILES = $(OUTPUT_FEATURES)make-*.feature-test.output
> +
> +clean:
> +	$(call QUIET_CLEAN, config) $(RM) $(CLEAN_FILES)
> +

hum, I think it'd be better if it stay in feature-checks dir,
I'll try to send patch for this as part of the v2 build patchset 

jirka

Re: [PATCH] perf build: Fix libbabeltrace detection

[Arnaldo Carvalho de Melo]

Em Wed, Mar 11, 2015 at 09:45:48AM +0100, Jiri Olsa escreveu:
> On Tue, Mar 10, 2015 at 01:04:26PM -0300, Arnaldo Carvalho de Melo wrote:
> 
> SNIP
> 
> >  endef
> >  
> >  feature_set = $(eval $(feature_set_code))
> > @@ -880,3 +880,9 @@ $(call detected_var,LIBDIR)
> >  $(call detected_var,GTK_CFLAGS)
> >  $(call detected_var,PERL_EMBED_CCOPTS)
> >  $(call detected_var,PYTHON_EMBED_CCOPTS)
> > +
> > +CLEAN_FILES = $(OUTPUT_FEATURES)make-*.feature-test.output
> > +
> > +clean:
> > +	$(call QUIET_CLEAN, config) $(RM) $(CLEAN_FILES)
> > +
> 
> hum, I think it'd be better if it stay in feature-checks dir,
> I'll try to send patch for this as part of the v2 build patchset 

I fixed it up already, nevermind.

- Arnaldo

Re: [PATCH 06/11] perf data: Switch to multiple cpu stream files

[Wang Nan]

Hi Jiri,

Have you noticed that this patch causes a endianess problem?

Without this patch:

$ perf  data convert --to-ctf ./out.ctf
[ perf data convert: Converted 'perf.data' into CTF data './out.ctf' ]
[ perf data convert: Converted and wrote 0.000 MB (11 samples) ]

With this patch:

$ perf  data convert --to-ctf ./out.ctf
perf: event-types.c:1855: bt_ctf_field_type_set_native_byte_order: Assertion `byte_order == 1234 || byte_order == 4321' failed.
Aborted

I'll look into this problem if you haven't solved it yet. Please let me know if you have already
have some solutions.

Thank you.

On 2015/2/21 6:17, Jiri Olsa wrote:
> From: Sebastian Andrzej Siewior <bigeasy@linutronix.de>
> 
> Currently we store the data into single data strea/file. The cpu
> if data is stored within the event sample. The lttng puts the CPU
> number that belongs to the event into the packet context instead
> into the event.
> 
> This patch makes sure that the trace produce by perf does look the
> same way. We now use one stream per-CPU. Having it all in one stream
> increased the total size of the resulting file. The test went from
> 416KiB (with perf_cpu event member) to 24MiB due to the required
> (and pointless) flush. With the per-cpu streams the total size went
> up to 588KiB.
> 
> Cc: Arnaldo Carvalho de Melo <acme@redhat.com>
> Cc: David Ahern <dsahern@gmail.com>
> Cc: Frederic Weisbecker <fweisbec@gmail.com>
> Cc: Jeremie Galarneau <jgalar@efficios.com>
> Cc: Jiri Olsa <jolsa@kernel.org>
> Cc: Namhyung Kim <namhyung@gmail.com>
> Cc: Paul Mackerras <paulus@samba.org>
> Cc: Peter Zijlstra <peterz@infradead.org>
> Cc: Sebastian Andrzej Siewior <bigeasy@linutronix.de>
> Cc: Tom Zanussi <tzanussi@gmail.com>
> Cc: Wang Nan <wangnan0@huawei.com>
> Signed-off-by: Jiri Olsa <jolsa@kernel.org>
> Signed-off-by: Sebastian Andrzej Siewior <bigeasy@linutronix.de>
> ---
>  tools/perf/util/data-convert-bt.c | 205 +++++++++++++++++++++++++++++++++-----
>  1 file changed, 181 insertions(+), 24 deletions(-)
> 
> diff --git a/tools/perf/util/data-convert-bt.c b/tools/perf/util/data-convert-bt.c
> index 6fa5c3ef336b..4bb769e081a8 100644
> --- a/tools/perf/util/data-convert-bt.c
> +++ b/tools/perf/util/data-convert-bt.c
> @@ -38,12 +38,20 @@ struct evsel_priv {
>  	struct bt_ctf_event_class *event_class;
>  };
>  
> +#define MAX_CPUS	4096
> +
> +struct ctf_stream {
> +	struct bt_ctf_stream *stream;
> +	int cpu;
> +};
> +
>  struct ctf_writer {
>  	/* writer primitives */
> -	struct bt_ctf_writer		*writer;
> -	struct bt_ctf_stream		*stream;
> -	struct bt_ctf_stream_class	*stream_class;
> -	struct bt_ctf_clock		*clock;
> +	struct bt_ctf_writer		 *writer;
> +	struct ctf_stream		**stream;
> +	int				  stream_cnt;
> +	struct bt_ctf_stream_class	 *stream_class;
> +	struct bt_ctf_clock		 *clock;
>  
>  	/* data types */
>  	union {
> @@ -346,12 +354,6 @@ static int add_generic_values(struct ctf_writer *cw,
>  			return -1;
>  	}
>  
> -	if (type & PERF_SAMPLE_CPU) {
> -		ret = value_set_u32(cw, event, "perf_cpu", sample->cpu);
> -		if (ret)
> -			return -1;
> -	}
> -
>  	if (type & PERF_SAMPLE_PERIOD) {
>  		ret = value_set_u64(cw, event, "perf_period", sample->period);
>  		if (ret)
> @@ -381,6 +383,113 @@ static int add_generic_values(struct ctf_writer *cw,
>  	return 0;
>  }
>  
> +static int ctf_stream__flush(struct ctf_stream *cs)
> +{
> +	int err = 0;
> +
> +	if (cs) {
> +		err = bt_ctf_stream_flush(cs->stream);
> +		if (err)
> +			pr_err("CTF stream %d flush failed\n", cs->cpu);
> +
> +		pr("Flush stream for cpu %d\n", cs->cpu);
> +	}
> +
> +	return err;
> +}
> +
> +static struct ctf_stream *ctf_stream__create(struct ctf_writer *cw, int cpu)
> +{
> +	struct ctf_stream *cs;
> +	struct bt_ctf_field *pkt_ctx   = NULL;
> +	struct bt_ctf_field *cpu_field = NULL;
> +	struct bt_ctf_stream *stream   = NULL;
> +	int ret;
> +
> +	cs = zalloc(sizeof(*cs));
> +	if (!cs) {
> +		pr_err("Failed to allocate ctf stream\n");
> +		return NULL;
> +	}
> +
> +	stream = bt_ctf_writer_create_stream(cw->writer, cw->stream_class);
> +	if (!stream) {
> +		pr_err("Failed to create CTF stream\n");
> +		goto out;
> +	}
> +
> +	pkt_ctx = bt_ctf_stream_get_packet_context(stream);
> +	if (!pkt_ctx) {
> +		pr_err("Failed to obtain packet context\n");
> +		goto out;
> +	}
> +
> +	cpu_field = bt_ctf_field_structure_get_field(pkt_ctx, "cpu_id");
> +	bt_ctf_field_put(pkt_ctx);
> +	if (!cpu_field) {
> +		pr_err("Failed to obtain cpu field\n");
> +		goto out;
> +	}
> +
> +	ret = bt_ctf_field_unsigned_integer_set_value(cpu_field, (u32) cpu);
> +	if (ret) {
> +		pr_err("Failed to update CPU number\n");
> +		goto out;
> +	}
> +
> +	bt_ctf_field_put(cpu_field);
> +
> +	cs->cpu    = cpu;
> +	cs->stream = stream;
> +	return cs;
> +
> +out:
> +	if (cpu_field)
> +		bt_ctf_field_put(cpu_field);
> +	if (stream)
> +		bt_ctf_stream_put(stream);
> +
> +	free(cs);
> +	return NULL;
> +}
> +
> +static void ctf_stream__delete(struct ctf_stream *cs)
> +{
> +	if (cs) {
> +		bt_ctf_stream_put(cs->stream);
> +		free(cs);
> +	}
> +}
> +
> +static struct ctf_stream *ctf_stream(struct ctf_writer *cw, int cpu)
> +{
> +	struct ctf_stream *cs = cw->stream[cpu];
> +
> +	if (!cs) {
> +		cs = ctf_stream__create(cw, cpu);
> +		cw->stream[cpu] = cs;
> +	}
> +
> +	return cs;
> +}
> +
> +static int get_sample_cpu(struct ctf_writer *cw, struct perf_sample *sample,
> +			  struct perf_evsel *evsel)
> +{
> +	int cpu = 0;
> +
> +	if (evsel->attr.sample_type & PERF_SAMPLE_CPU)
> +		cpu = sample->cpu;
> +
> +	if (cpu > cw->stream_cnt) {
> +		pr_err("Event was recorded for CPU %d, limit is at %d.\n",
> +			cpu, cw->stream_cnt);
> +		cpu = 0;
> +	}
> +
> +	return cpu;
> +}
> +
>  static int process_sample_event(struct perf_tool *tool,
>  				union perf_event *_event __maybe_unused,
>  				struct perf_sample *sample,
> @@ -390,6 +499,7 @@ static int process_sample_event(struct perf_tool *tool,
>  	struct convert *c = container_of(tool, struct convert, tool);
>  	struct evsel_priv *priv = evsel->priv;
>  	struct ctf_writer *cw = &c->writer;
> +	struct ctf_stream *cs;
>  	struct bt_ctf_event_class *event_class;
>  	struct bt_ctf_event *event;
>  	int ret;
> @@ -424,9 +534,12 @@ static int process_sample_event(struct perf_tool *tool,
>  			return -1;
>  	}
>  
> -	bt_ctf_stream_append_event(cw->stream, event);
> +	cs = ctf_stream(cw, get_sample_cpu(cw, sample, evsel));
> +	if (cs)
> +		bt_ctf_stream_append_event(cs->stream, event);
> +
>  	bt_ctf_event_put(event);
> -	return 0;
> +	return cs ? 0 : -1;
>  }
>  
>  static int add_tracepoint_fields_types(struct ctf_writer *cw,
> @@ -528,9 +641,6 @@ static int add_generic_types(struct ctf_writer *cw, struct perf_evsel *evsel,
>  	if (type & PERF_SAMPLE_STREAM_ID)
>  		ADD_FIELD(event_class, cw->data.u64, "perf_stream_id");
>  
> -	if (type & PERF_SAMPLE_CPU)
> -		ADD_FIELD(event_class, cw->data.u32, "perf_cpu");
> -
>  	if (type & PERF_SAMPLE_PERIOD)
>  		ADD_FIELD(event_class, cw->data.u64, "perf_period");
>  
> @@ -604,6 +714,39 @@ static int setup_events(struct ctf_writer *cw, struct perf_session *session)
>  	return 0;
>  }
>  
> +static int setup_streams(struct ctf_writer *cw, struct perf_session *session)
> +{
> +	struct ctf_stream **stream;
> +	struct perf_header *ph = &session->header;
> +	int ncpus;
> +
> +	/*
> +	 * Try to get the number of cpus used in the data file,
> +	 * if not present fallback to the MAX_CPUS.
> +	 */
> +	ncpus = ph->env.nr_cpus_avail ?: MAX_CPUS;
> +
> +	stream = zalloc(sizeof(*stream) * ncpus);
> +	if (!stream) {
> +		pr_err("Failed to allocate streams.\n");
> +		return -ENOMEM;
> +	}
> +
> +	cw->stream     = stream;
> +	cw->stream_cnt = ncpus;
> +	return 0;
> +}
> +
> +static void free_streams(struct ctf_writer *cw)
> +{
> +	int cpu;
> +
> +	for (cpu = 0; cpu < cw->stream_cnt; cpu++)
> +		ctf_stream__delete(cw->stream[cpu]);
> +
> +	free(cw->stream);
> +}
> +
>  static int ctf_writer__setup_env(struct ctf_writer *cw,
>  				 struct perf_session *session)
>  {
> @@ -713,7 +856,7 @@ static void ctf_writer__cleanup(struct ctf_writer *cw)
>  	ctf_writer__cleanup_data(cw);
>  
>  	bt_ctf_clock_put(cw->clock);
> -	bt_ctf_stream_put(cw->stream);
> +	free_streams(cw);
>  	bt_ctf_stream_class_put(cw->stream_class);
>  	bt_ctf_writer_put(cw->writer);
>  
> @@ -725,8 +868,9 @@ static int ctf_writer__init(struct ctf_writer *cw, const char *path)
>  {
>  	struct bt_ctf_writer		*writer;
>  	struct bt_ctf_stream_class	*stream_class;
> -	struct bt_ctf_stream		*stream;
>  	struct bt_ctf_clock		*clock;
> +	struct bt_ctf_field_type	*pkt_ctx_type;
> +	int				ret;
>  
>  	/* CTF writer */
>  	writer = bt_ctf_writer_create(path);
> @@ -767,14 +911,15 @@ static int ctf_writer__init(struct ctf_writer *cw, const char *path)
>  	if (ctf_writer__init_data(cw))
>  		goto err_cleanup;
>  
> -	/* CTF stream instance */
> -	stream = bt_ctf_writer_create_stream(writer, stream_class);
> -	if (!stream) {
> -		pr("Failed to create CTF stream.\n");
> +	/* Add cpu_id for packet context */
> +	pkt_ctx_type = bt_ctf_stream_class_get_packet_context_type(stream_class);
> +	if (!pkt_ctx_type)
>  		goto err_cleanup;
> -	}
>  
> -	cw->stream = stream;
> +	ret = bt_ctf_field_type_structure_add_field(pkt_ctx_type, cw->data.u32, "cpu_id");
> +	bt_ctf_field_type_put(pkt_ctx_type);
> +	if (ret)
> +		goto err_cleanup;
>  
>  	/* CTF clock writer setup */
>  	if (bt_ctf_writer_add_clock(writer, clock)) {
> @@ -791,6 +936,14 @@ err:
>  	return -1;
>  }
>  
> +static void ctf_writer__flush_streams(struct ctf_writer *cw)
> +{
> +	int cpu;
> +
> +	for (cpu = 0; cpu < cw->stream_cnt; cpu++)
> +		ctf_stream__flush(cw->stream[cpu]);
> +}
> +
>  int bt_convert__perf2ctf(const char *input, const char *path)
>  {
>  	struct perf_session *session;
> @@ -833,9 +986,13 @@ int bt_convert__perf2ctf(const char *input, const char *path)
>  	if (setup_events(cw, session))
>  		goto free_session;
>  
> +	if (setup_streams(cw, session))
> +		goto free_session;
> +
>  	err = perf_session__process_events(session, &c.tool);
>  	if (!err)
> -		err = bt_ctf_stream_flush(cw->stream);
> +		ctf_writer__flush_streams(cw);
> +
>  
>  	fprintf(stderr,
>  		"[ perf data convert: Converted '%s' into CTF data '%s' ]\n",
> 

Re: [PATCH 06/11] perf data: Switch to multiple cpu stream files

[Jiri Olsa]

On Thu, Mar 12, 2015 at 07:37:02PM +0800, Wang Nan wrote:
> Hi Jiri,
> 
> Have you noticed that this patch causes a endianess problem?
> 
> Without this patch:
> 
> $ perf  data convert --to-ctf ./out.ctf
> [ perf data convert: Converted 'perf.data' into CTF data './out.ctf' ]
> [ perf data convert: Converted and wrote 0.000 MB (11 samples) ]
> 
> With this patch:
> 
> $ perf  data convert --to-ctf ./out.ctf
> perf: event-types.c:1855: bt_ctf_field_type_set_native_byte_order: Assertion `byte_order == 1234 || byte_order == 4321' failed.
> Aborted
> 
> I'll look into this problem if you haven't solved it yet. Please let me know if you have already
> have some solutions.

hum, never seen this.. any other details? you moved perf.data
through different endian server or something like that?

jirka

Re: [PATCH 06/11] perf data: Switch to multiple cpu stream files

[Wang Nan]

On 2015/3/12 20:34, Jiri Olsa wrote:
> On Thu, Mar 12, 2015 at 07:37:02PM +0800, Wang Nan wrote:
>> Hi Jiri,
>>
>> Have you noticed that this patch causes a endianess problem?
>>
>> Without this patch:
>>
>> $ perf  data convert --to-ctf ./out.ctf
>> [ perf data convert: Converted 'perf.data' into CTF data './out.ctf' ]
>> [ perf data convert: Converted and wrote 0.000 MB (11 samples) ]
>>
>> With this patch:
>>
>> $ perf  data convert --to-ctf ./out.ctf
>> perf: event-types.c:1855: bt_ctf_field_type_set_native_byte_order: Assertion `byte_order == 1234 || byte_order == 4321' failed.
>> Aborted
>>
>> I'll look into this problem if you haven't solved it yet. Please let me know if you have already
>> have some solutions.
> 
> hum, never seen this.. any other details?



My perf is -tip tree (commit 4b52a8d), with patch 'perf data: Add tracepoint events fields CTF conversion support'
applied. Babeltrace is fresh new git clone (commit 48d711a).

you moved perf.data
> through different endian server or something like that?
> 

Never.

> jirka
> 
---
I think I find the problem, but I fill strange whether you and others never hit it.

The problem is that, in babeltrace, stream_class->byte_order is never setup until someone calls bt_ctf_trace_create_stream(),
which do it by bt_ctf_stream_class_set_byte_order(), set stream_class->byte_order according to trace->byte_order.

(However, according to the name of bt_ctf_trace_create_stream, people (like me) believe stream_class
should be a fully initialized structure, bt_ctf_trace_create_stream() is used to create a bt_ctf_stream,
it should not be a part of initialization of bt_ctf_stream_class().)

After applying patch 6/11, such call (bt_ctf_writer_create_stream()) is moved from ctf_writer__init()
to ctf_stream(), that is, later than bt_ctf_event_create(). However, bt_ctf_event_create() -> bt_ctf_stream_class_freeze()
requires stream_class->byte_order been set.

At the bottom of this mail is a workable workaround (an ugly workaround since I'm not sure which part is responsible
for the problem) and my SOB.

Thank you.

Signed-off-by: Wang Nan <wangnan0@huawei.com>

---
--- a/tools/perf/util/data-convert-bt.c
+++ b/tools/perf/util/data-convert-bt.c
@@ -557,6 +557,8 @@ static int process_sample_event(struct perf_tool *tool,

	pr_time2(sample->time, "sample %" PRIu64 "\n", c->events_count);

+	cs = ctf_stream(cw, get_sample_cpu(cw, sample, evsel));
+
	event = bt_ctf_event_create(event_class);
	if (!event) {
		pr_err("Failed to create an CTF event\n");
@@ -576,7 +578,6 @@ static int process_sample_event(struct perf_tool *tool,
			return -1;
	}

-	cs = ctf_stream(cw, get_sample_cpu(cw, sample, evsel));
	if (cs)
		bt_ctf_stream_append_event(cs->stream, event);

Re: [PATCH 06/11] perf data: Switch to multiple cpu stream files

[Arnaldo Carvalho de Melo]

Em Thu, Mar 12, 2015 at 09:40:09PM +0800, Wang Nan escreveu:
> On 2015/3/12 20:34, Jiri Olsa wrote:
> > On Thu, Mar 12, 2015 at 07:37:02PM +0800, Wang Nan wrote:
> >> Hi Jiri,
> >>
> >> Have you noticed that this patch causes a endianess problem?
> >>
> >> Without this patch:
> >>
> >> $ perf  data convert --to-ctf ./out.ctf
> >> [ perf data convert: Converted 'perf.data' into CTF data './out.ctf' ]
> >> [ perf data convert: Converted and wrote 0.000 MB (11 samples) ]
> >>
> >> With this patch:
> >>
> >> $ perf  data convert --to-ctf ./out.ctf
> >> perf: event-types.c:1855: bt_ctf_field_type_set_native_byte_order: Assertion `byte_order == 1234 || byte_order == 4321' failed.
> >> Aborted
> >>
> >> I'll look into this problem if you haven't solved it yet. Please let me know if you have already
> >> have some solutions.
> > 
> > hum, never seen this.. any other details?
> 
> 
> 
> My perf is -tip tree (commit 4b52a8d), with patch 'perf data: Add tracepoint events fields CTF conversion support'
> applied. Babeltrace is fresh new git clone (commit 48d711a).
> 
> you moved perf.data
> > through different endian server or something like that?
> > 
> 
> Never.

Just reproduced this:

[acme@ssdandy linux]$ perf data convert --to-ctf=./ctf-data/
perf: event-types.c:1855: bt_ctf_field_type_set_native_byte_order:
Assertion `byte_order == 1234 || byte_order == 4321' failed.
Aborted (core dumped)
[acme@ssdandy linux]$


perf: event-types.c:1855: bt_ctf_field_type_set_native_byte_order: Assertion `byte_order == 1234 || byte_order == 4321' failed.

Program received signal SIGABRT, Aborted.
0x00007ffff5b345d7 in raise () from /lib64/libc.so.6
Missing separate debuginfos, use: debuginfo-install audit-libs-2.4.1-5.el7.x86_64 bzip2-libs-1.0.6-12.el7.x86_64 elfutils-libelf-0.160-1.el7.x86_64 elfutils-libs-0.160-1.el7.x86_64 glib2-2.40.0-4.el7.x86_64 glibc-2.17-78.el7.x86_64 libgcc-4.8.3-9.el7.x86_64 libuuid-2.23.2-21.el7.x86_64 nss-softokn-freebl-3.16.2.3-9.el7.x86_64 numactl-libs-2.0.9-4.el7.x86_64 perl-libs-5.16.3-285.el7.x86_64 popt-1.13-16.el7.x86_64 python-libs-2.7.5-16.el7.x86_64 slang-2.2.4-11.el7.x86_64 xz-libs-5.1.2-9alpha.el7.x86_64 zlib-1.2.7-13.el7.x86_64
(gdb) bt
#0  0x00007ffff5b345d7 in raise () from /lib64/libc.so.6
#1  0x00007ffff5b35cc8 in abort () from /lib64/libc.so.6
#2  0x00007ffff5b2d546 in __assert_fail_base () from /lib64/libc.so.6
#3  0x00007ffff5b2d5f2 in __assert_fail () from /lib64/libc.so.6
#4  0x00007ffff510034a in bt_ctf_field_type_set_native_byte_order (type=<optimized out>, byte_order=<optimized out>) at event-types.c:1855
#5  0x00007ffff5101b59 in bt_ctf_stream_class_freeze (stream_class=0x8acc20) at stream-class.c:478
#6  0x00007ffff50fbdc5 in bt_ctf_event_create (event_class=event_class@entry=0x133d590) at event.c:324
#7  0x00000000004e158d in process_sample_event (tool=0x7fffffffdc40, _event=<optimized out>, sample=0x7fffffffd840, evsel=0x8b1040, machine=<optimized out>) at util/data-convert-bt.c:518
#8  0x00000000004b011d in __ordered_events__flush (oe=oe@entry=0x8b0210) at util/ordered-events.c:214
#9  0x00000000004b043a in ordered_events__flush (oe=oe@entry=0x8b0210, how=how@entry=OE_FLUSH__FINAL) at util/ordered-events.c:279
#10 0x00000000004aee18 in __perf_session__process_events (file_size=139704, data_size=<optimized out>, data_offset=<optimized out>, session=0x8b0020) at util/session.c:1394
#11 perf_session__process_events (session=session@entry=0x8b0020) at util/session.c:1412
#12 0x00000000004e270e in bt_convert__perf2ctf (input=<optimized out>, path=0x7fffffffe59d "./ctf-data/") at util/data-convert-bt.c:992
#13 0x00000000004479cf in cmd_data_convert (argc=<optimized out>, argv=<optimized out>, prefix=<optimized out>) at builtin-data.c:77
#14 0x000000000046f365 in run_builtin (p=p@entry=0x8345a0 <commands+576>, argc=argc@entry=3, argv=argv@entry=0x7fffffffe2d0) at perf.c:370
#15 0x000000000041def0 in handle_internal_command (argv=0x7fffffffe2d0, argc=3) at perf.c:429
#16 run_argv (argv=0x7fffffffe050, argcp=0x7fffffffe05c) at perf.c:473
#17 main (argc=3, argv=0x7fffffffe2d0) at perf.c:588
(gdb) 

Stopping at this to process other patches, will be back to this after reducing the queue,

- Arnaldo

Re: [PATCH 06/11] perf data: Switch to multiple cpu stream files

[Jérémie Galarneau]

On Thu, Mar 12, 2015 at 3:17 PM, Arnaldo Carvalho de Melo
<acme@redhat.com> wrote:
> Em Thu, Mar 12, 2015 at 09:40:09PM +0800, Wang Nan escreveu:
>> On 2015/3/12 20:34, Jiri Olsa wrote:
>> > On Thu, Mar 12, 2015 at 07:37:02PM +0800, Wang Nan wrote:
>> >> Hi Jiri,
>> >>
>> >> Have you noticed that this patch causes a endianess problem?
>> >>
>> >> Without this patch:
>> >>
>> >> $ perf  data convert --to-ctf ./out.ctf
>> >> [ perf data convert: Converted 'perf.data' into CTF data './out.ctf' ]
>> >> [ perf data convert: Converted and wrote 0.000 MB (11 samples) ]
>> >>
>> >> With this patch:
>> >>
>> >> $ perf  data convert --to-ctf ./out.ctf
>> >> perf: event-types.c:1855: bt_ctf_field_type_set_native_byte_order: Assertion `byte_order == 1234 || byte_order == 4321' failed.
>> >> Aborted
>> >>
>> >> I'll look into this problem if you haven't solved it yet. Please let me know if you have already
>> >> have some solutions.
>> >
>> > hum, never seen this.. any other details?
>>
>>
>>
>> My perf is -tip tree (commit 4b52a8d), with patch 'perf data: Add tracepoint events fields CTF conversion support'
>> applied. Babeltrace is fresh new git clone (commit 48d711a).
>>
>> you moved perf.data
>> > through different endian server or something like that?
>> >
>>
>> Never.
>
> Just reproduced this:
>
> [acme@ssdandy linux]$ perf data convert --to-ctf=./ctf-data/
> perf: event-types.c:1855: bt_ctf_field_type_set_native_byte_order:
> Assertion `byte_order == 1234 || byte_order == 4321' failed.
> Aborted (core dumped)
> [acme@ssdandy linux]$
>
>
> perf: event-types.c:1855: bt_ctf_field_type_set_native_byte_order: Assertion `byte_order == 1234 || byte_order == 4321' failed.
>
> Program received signal SIGABRT, Aborted.
> 0x00007ffff5b345d7 in raise () from /lib64/libc.so.6
> Missing separate debuginfos, use: debuginfo-install audit-libs-2.4.1-5.el7.x86_64 bzip2-libs-1.0.6-12.el7.x86_64 elfutils-libelf-0.160-1.el7.x86_64 elfutils-libs-0.160-1.el7.x86_64 glib2-2.40.0-4.el7.x86_64 glibc-2.17-78.el7.x86_64 libgcc-4.8.3-9.el7.x86_64 libuuid-2.23.2-21.el7.x86_64 nss-softokn-freebl-3.16.2.3-9.el7.x86_64 numactl-libs-2.0.9-4.el7.x86_64 perl-libs-5.16.3-285.el7.x86_64 popt-1.13-16.el7.x86_64 python-libs-2.7.5-16.el7.x86_64 slang-2.2.4-11.el7.x86_64 xz-libs-5.1.2-9alpha.el7.x86_64 zlib-1.2.7-13.el7.x86_64
> (gdb) bt
> #0  0x00007ffff5b345d7 in raise () from /lib64/libc.so.6
> #1  0x00007ffff5b35cc8 in abort () from /lib64/libc.so.6
> #2  0x00007ffff5b2d546 in __assert_fail_base () from /lib64/libc.so.6
> #3  0x00007ffff5b2d5f2 in __assert_fail () from /lib64/libc.so.6
> #4  0x00007ffff510034a in bt_ctf_field_type_set_native_byte_order (type=<optimized out>, byte_order=<optimized out>) at event-types.c:1855
> #5  0x00007ffff5101b59 in bt_ctf_stream_class_freeze (stream_class=0x8acc20) at stream-class.c:478
> #6  0x00007ffff50fbdc5 in bt_ctf_event_create (event_class=event_class@entry=0x133d590) at event.c:324
> #7  0x00000000004e158d in process_sample_event (tool=0x7fffffffdc40, _event=<optimized out>, sample=0x7fffffffd840, evsel=0x8b1040, machine=<optimized out>) at util/data-convert-bt.c:518
> #8  0x00000000004b011d in __ordered_events__flush (oe=oe@entry=0x8b0210) at util/ordered-events.c:214
> #9  0x00000000004b043a in ordered_events__flush (oe=oe@entry=0x8b0210, how=how@entry=OE_FLUSH__FINAL) at util/ordered-events.c:279
> #10 0x00000000004aee18 in __perf_session__process_events (file_size=139704, data_size=<optimized out>, data_offset=<optimized out>, session=0x8b0020) at util/session.c:1394
> #11 perf_session__process_events (session=session@entry=0x8b0020) at util/session.c:1412
> #12 0x00000000004e270e in bt_convert__perf2ctf (input=<optimized out>, path=0x7fffffffe59d "./ctf-data/") at util/data-convert-bt.c:992
> #13 0x00000000004479cf in cmd_data_convert (argc=<optimized out>, argv=<optimized out>, prefix=<optimized out>) at builtin-data.c:77
> #14 0x000000000046f365 in run_builtin (p=p@entry=0x8345a0 <commands+576>, argc=argc@entry=3, argv=argv@entry=0x7fffffffe2d0) at perf.c:370
> #15 0x000000000041def0 in handle_internal_command (argv=0x7fffffffe2d0, argc=3) at perf.c:429
> #16 run_argv (argv=0x7fffffffe050, argcp=0x7fffffffe05c) at perf.c:473
> #17 main (argc=3, argv=0x7fffffffe2d0) at perf.c:588
> (gdb)

Thanks for providing this info, I'm looking into it and will keep you posted.

Jérémie

>
> Stopping at this to process other patches, will be back to this after reducing the queue,
>
> - Arnaldo



-- 
Jérémie Galarneau
EfficiOS Inc.
http://www.efficios.com

Re: [PATCH 06/11] perf data: Switch to multiple cpu stream files

[Wang Nan]

On 2015/3/13 3:59, Jérémie Galarneau wrote:
> On Thu, Mar 12, 2015 at 3:17 PM, Arnaldo Carvalho de Melo
> <acme@redhat.com> wrote:
>> Em Thu, Mar 12, 2015 at 09:40:09PM +0800, Wang Nan escreveu:
>>> On 2015/3/12 20:34, Jiri Olsa wrote:
>>>> On Thu, Mar 12, 2015 at 07:37:02PM +0800, Wang Nan wrote:
>>>>> Hi Jiri,
>>>>>
>>>>> Have you noticed that this patch causes a endianess problem?
>>>>>
>>>>> Without this patch:
>>>>>
>>>>> $ perf  data convert --to-ctf ./out.ctf
>>>>> [ perf data convert: Converted 'perf.data' into CTF data './out.ctf' ]
>>>>> [ perf data convert: Converted and wrote 0.000 MB (11 samples) ]
>>>>>
>>>>> With this patch:
>>>>>
>>>>> $ perf  data convert --to-ctf ./out.ctf
>>>>> perf: event-types.c:1855: bt_ctf_field_type_set_native_byte_order: Assertion `byte_order == 1234 || byte_order == 4321' failed.
>>>>> Aborted
>>>>>
>>>>> I'll look into this problem if you haven't solved it yet. Please let me know if you have already
>>>>> have some solutions.
>>>>
>>>> hum, never seen this.. any other details?
>>>
>>>
>>>
>>> My perf is -tip tree (commit 4b52a8d), with patch 'perf data: Add tracepoint events fields CTF conversion support'
>>> applied. Babeltrace is fresh new git clone (commit 48d711a).
>>>
>>> you moved perf.data
>>>> through different endian server or something like that?
>>>>
>>>
>>> Never.
>>
>> Just reproduced this:
>>
>> [acme@ssdandy linux]$ perf data convert --to-ctf=./ctf-data/
>> perf: event-types.c:1855: bt_ctf_field_type_set_native_byte_order:
>> Assertion `byte_order == 1234 || byte_order == 4321' failed.
>> Aborted (core dumped)
>> [acme@ssdandy linux]$
>>
>>
>> perf: event-types.c:1855: bt_ctf_field_type_set_native_byte_order: Assertion `byte_order == 1234 || byte_order == 4321' failed.
>>
>> Program received signal SIGABRT, Aborted.
>> 0x00007ffff5b345d7 in raise () from /lib64/libc.so.6
>> Missing separate debuginfos, use: debuginfo-install audit-libs-2.4.1-5.el7.x86_64 bzip2-libs-1.0.6-12.el7.x86_64 elfutils-libelf-0.160-1.el7.x86_64 elfutils-libs-0.160-1.el7.x86_64 glib2-2.40.0-4.el7.x86_64 glibc-2.17-78.el7.x86_64 libgcc-4.8.3-9.el7.x86_64 libuuid-2.23.2-21.el7.x86_64 nss-softokn-freebl-3.16.2.3-9.el7.x86_64 numactl-libs-2.0.9-4.el7.x86_64 perl-libs-5.16.3-285.el7.x86_64 popt-1.13-16.el7.x86_64 python-libs-2.7.5-16.el7.x86_64 slang-2.2.4-11.el7.x86_64 xz-libs-5.1.2-9alpha.el7.x86_64 zlib-1.2.7-13.el7.x86_64
>> (gdb) bt
>> #0  0x00007ffff5b345d7 in raise () from /lib64/libc.so.6
>> #1  0x00007ffff5b35cc8 in abort () from /lib64/libc.so.6
>> #2  0x00007ffff5b2d546 in __assert_fail_base () from /lib64/libc.so.6
>> #3  0x00007ffff5b2d5f2 in __assert_fail () from /lib64/libc.so.6
>> #4  0x00007ffff510034a in bt_ctf_field_type_set_native_byte_order (type=<optimized out>, byte_order=<optimized out>) at event-types.c:1855
>> #5  0x00007ffff5101b59 in bt_ctf_stream_class_freeze (stream_class=0x8acc20) at stream-class.c:478
>> #6  0x00007ffff50fbdc5 in bt_ctf_event_create (event_class=event_class@entry=0x133d590) at event.c:324
>> #7  0x00000000004e158d in process_sample_event (tool=0x7fffffffdc40, _event=<optimized out>, sample=0x7fffffffd840, evsel=0x8b1040, machine=<optimized out>) at util/data-convert-bt.c:518
>> #8  0x00000000004b011d in __ordered_events__flush (oe=oe@entry=0x8b0210) at util/ordered-events.c:214
>> #9  0x00000000004b043a in ordered_events__flush (oe=oe@entry=0x8b0210, how=how@entry=OE_FLUSH__FINAL) at util/ordered-events.c:279
>> #10 0x00000000004aee18 in __perf_session__process_events (file_size=139704, data_size=<optimized out>, data_offset=<optimized out>, session=0x8b0020) at util/session.c:1394
>> #11 perf_session__process_events (session=session@entry=0x8b0020) at util/session.c:1412
>> #12 0x00000000004e270e in bt_convert__perf2ctf (input=<optimized out>, path=0x7fffffffe59d "./ctf-data/") at util/data-convert-bt.c:992
>> #13 0x00000000004479cf in cmd_data_convert (argc=<optimized out>, argv=<optimized out>, prefix=<optimized out>) at builtin-data.c:77
>> #14 0x000000000046f365 in run_builtin (p=p@entry=0x8345a0 <commands+576>, argc=argc@entry=3, argv=argv@entry=0x7fffffffe2d0) at perf.c:370
>> #15 0x000000000041def0 in handle_internal_command (argv=0x7fffffffe2d0, argc=3) at perf.c:429
>> #16 run_argv (argv=0x7fffffffe050, argcp=0x7fffffffe05c) at perf.c:473
>> #17 main (argc=3, argv=0x7fffffffe2d0) at perf.c:588
>> (gdb)
> 

I have identical call stack. Have you ever tried my workaround in my previous mail?

Thank you!

> Thanks for providing this info, I'm looking into it and will keep you posted.
> 
> Jérémie
> 
>>
>> Stopping at this to process other patches, will be back to this after reducing the queue,
>>
>> - Arnaldo
> 
> 
> 

Re: [PATCH 06/11] perf data: Switch to multiple cpu stream files

[Jérémie Galarneau]

On Thu, Mar 12, 2015 at 3:17 PM, Arnaldo Carvalho de Melo
<acme@redhat.com> wrote:
> Em Thu, Mar 12, 2015 at 09:40:09PM +0800, Wang Nan escreveu:
>> On 2015/3/12 20:34, Jiri Olsa wrote:
>> > On Thu, Mar 12, 2015 at 07:37:02PM +0800, Wang Nan wrote:
>> >> Hi Jiri,
>> >>
>> >> Have you noticed that this patch causes a endianess problem?
>> >>
>> >> Without this patch:
>> >>
>> >> $ perf  data convert --to-ctf ./out.ctf
>> >> [ perf data convert: Converted 'perf.data' into CTF data './out.ctf' ]
>> >> [ perf data convert: Converted and wrote 0.000 MB (11 samples) ]
>> >>
>> >> With this patch:
>> >>
>> >> $ perf  data convert --to-ctf ./out.ctf
>> >> perf: event-types.c:1855: bt_ctf_field_type_set_native_byte_order: Assertion `byte_order == 1234 || byte_order == 4321' failed.
>> >> Aborted
>> >>
>> >> I'll look into this problem if you haven't solved it yet. Please let me know if you have already
>> >> have some solutions.
>> >
>> > hum, never seen this.. any other details?
>>
>>
>>
>> My perf is -tip tree (commit 4b52a8d), with patch 'perf data: Add tracepoint events fields CTF conversion support'
>> applied. Babeltrace is fresh new git clone (commit 48d711a).
>>
>> you moved perf.data
>> > through different endian server or something like that?
>> >
>>
>> Never.
>
> Just reproduced this:
>
> [acme@ssdandy linux]$ perf data convert --to-ctf=./ctf-data/
> perf: event-types.c:1855: bt_ctf_field_type_set_native_byte_order:
> Assertion `byte_order == 1234 || byte_order == 4321' failed.
> Aborted (core dumped)
> [acme@ssdandy linux]$
>
>
> perf: event-types.c:1855: bt_ctf_field_type_set_native_byte_order: Assertion `byte_order == 1234 || byte_order == 4321' failed.
>
> Program received signal SIGABRT, Aborted.
> 0x00007ffff5b345d7 in raise () from /lib64/libc.so.6
> Missing separate debuginfos, use: debuginfo-install audit-libs-2.4.1-5.el7.x86_64 bzip2-libs-1.0.6-12.el7.x86_64 elfutils-libelf-0.160-1.el7.x86_64 elfutils-libs-0.160-1.el7.x86_64 glib2-2.40.0-4.el7.x86_64 glibc-2.17-78.el7.x86_64 libgcc-4.8.3-9.el7.x86_64 libuuid-2.23.2-21.el7.x86_64 nss-softokn-freebl-3.16.2.3-9.el7.x86_64 numactl-libs-2.0.9-4.el7.x86_64 perl-libs-5.16.3-285.el7.x86_64 popt-1.13-16.el7.x86_64 python-libs-2.7.5-16.el7.x86_64 slang-2.2.4-11.el7.x86_64 xz-libs-5.1.2-9alpha.el7.x86_64 zlib-1.2.7-13.el7.x86_64
> (gdb) bt
> #0  0x00007ffff5b345d7 in raise () from /lib64/libc.so.6
> #1  0x00007ffff5b35cc8 in abort () from /lib64/libc.so.6
> #2  0x00007ffff5b2d546 in __assert_fail_base () from /lib64/libc.so.6
> #3  0x00007ffff5b2d5f2 in __assert_fail () from /lib64/libc.so.6
> #4  0x00007ffff510034a in bt_ctf_field_type_set_native_byte_order (type=<optimized out>, byte_order=<optimized out>) at event-types.c:1855
> #5  0x00007ffff5101b59 in bt_ctf_stream_class_freeze (stream_class=0x8acc20) at stream-class.c:478
> #6  0x00007ffff50fbdc5 in bt_ctf_event_create (event_class=event_class@entry=0x133d590) at event.c:324
> #7  0x00000000004e158d in process_sample_event (tool=0x7fffffffdc40, _event=<optimized out>, sample=0x7fffffffd840, evsel=0x8b1040, machine=<optimized out>) at util/data-convert-bt.c:518
> #8  0x00000000004b011d in __ordered_events__flush (oe=oe@entry=0x8b0210) at util/ordered-events.c:214
> #9  0x00000000004b043a in ordered_events__flush (oe=oe@entry=0x8b0210, how=how@entry=OE_FLUSH__FINAL) at util/ordered-events.c:279
> #10 0x00000000004aee18 in __perf_session__process_events (file_size=139704, data_size=<optimized out>, data_offset=<optimized out>, session=0x8b0020) at util/session.c:1394
> #11 perf_session__process_events (session=session@entry=0x8b0020) at util/session.c:1412
> #12 0x00000000004e270e in bt_convert__perf2ctf (input=<optimized out>, path=0x7fffffffe59d "./ctf-data/") at util/data-convert-bt.c:992
> #13 0x00000000004479cf in cmd_data_convert (argc=<optimized out>, argv=<optimized out>, prefix=<optimized out>) at builtin-data.c:77
> #14 0x000000000046f365 in run_builtin (p=p@entry=0x8345a0 <commands+576>, argc=argc@entry=3, argv=argv@entry=0x7fffffffe2d0) at perf.c:370
> #15 0x000000000041def0 in handle_internal_command (argv=0x7fffffffe2d0, argc=3) at perf.c:429
> #16 run_argv (argv=0x7fffffffe050, argcp=0x7fffffffe05c) at perf.c:473
> #17 main (argc=3, argv=0x7fffffffe2d0) at perf.c:588
> (gdb)
>

I was able to reproduce this on my end as well. I pushed a fix
addressing this issue in Babeltrace master as of 5ca83563.
This scenario has also been added to Babeltrace's regression test suite.

Jérémie

> Stopping at this to process other patches, will be back to this after reducing the queue,
>
> - Arnaldo



-- 
Jérémie Galarneau
EfficiOS Inc.
http://www.efficios.com

[tip:perf/core] perf build: Fix libbabeltrace detection

[tip-bot for Jiri Olsa]

Commit-ID:  19a9df35fe9e8ffd60ce4b6f888b72e7c8422d31
Gitweb:     http://git.kernel.org/tip/19a9df35fe9e8ffd60ce4b6f888b72e7c8422d31
Author:     Jiri Olsa <jolsa@redhat.com>
AuthorDate: Tue, 10 Mar 2015 13:00:35 +0100
Committer:  Arnaldo Carvalho de Melo <acme@redhat.com>
CommitDate: Thu, 12 Mar 2015 12:39:56 -0300

perf build: Fix libbabeltrace detection

Following patch added -Werror for feature builds:

  b49f1a4be701 perf tools: Improve feature test debuggability

and exposed a problem in the libbabeltrace feature build, because it was
including wrong header and gcc couldn't find the used symbol definition.

Adding proper header and keeping the old one as it is needed also
(libbabeltrace quirk).

Reported-by: Arnaldo Carvalho de Melo <acme@redhat.com>
Tested-by: Arnaldo Carvalho de Melo <acme@redhat.com>
Signed-off-by: Jiri Olsa <jolsa@kernel.org>
Cc: David Ahern <dsahern@gmail.com>
Cc: Frederic Weisbecker <fweisbec@gmail.com>
Cc: Jeremie Galarneau <jgalar@efficios.com>
Cc: Namhyung Kim <namhyung@gmail.com>
Cc: Paul Mackerras <paulus@samba.org>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Sebastian Andrzej Siewior <bigeasy@linutronix.de>
Cc: Tom Zanussi <tzanussi@gmail.com>
Cc: Wang Nan <wangnan0@huawei.com>
Link: http://lkml.kernel.org/r/20150310120035.GA4333@krava.redhat.com
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
---
 tools/perf/config/feature-checks/test-libbabeltrace.c | 1 +
 1 file changed, 1 insertion(+)

diff --git a/tools/perf/config/feature-checks/test-libbabeltrace.c b/tools/perf/config/feature-checks/test-libbabeltrace.c
index 3b7dd68..9cf802a 100644
--- a/tools/perf/config/feature-checks/test-libbabeltrace.c
+++ b/tools/perf/config/feature-checks/test-libbabeltrace.c
@@ -1,5 +1,6 @@
 
 #include <babeltrace/ctf-writer/writer.h>
+#include <babeltrace/ctf-ir/stream-class.h>
 
 int main(void)
 {

[tip:perf/core] perf data: Add tracepoint events fields CTF conversion support

[tip-bot for Sebastian Andrzej Siewior]

Commit-ID:  69364727be2f3dc71a046771965c3c9d5ccce699
Gitweb:     http://git.kernel.org/tip/69364727be2f3dc71a046771965c3c9d5ccce699
Author:     Sebastian Andrzej Siewior <bigeasy@linutronix.de>
AuthorDate: Fri, 20 Feb 2015 23:17:02 +0100
Committer:  Arnaldo Carvalho de Melo <acme@redhat.com>
CommitDate: Fri, 13 Mar 2015 07:47:49 -0300

perf data: Add tracepoint events fields CTF conversion support

Adding support to convert tracepoint event fields into CTF
event fields.

We parse each tracepoint event for CTF conversion and add
tracepoint fields as regular CTF event fields, so they
appear in babeltrace output like:

  $ babeltrace ./ctf-data/
  ...
  [09:02:00.950703057] (+?.?????????) sched:sched_stat_runtime: { }, { perf_ip = ... SNIP ... common_type = 298, common_flags = 1, \
  common_preempt_count = 0, common_pid = 31813, comm = "perf", pid = 31813, runtime = 458800, vruntime = 52059858071 }
  ...

Signed-off-by: Sebastian Andrzej Siewior <bigeasy@linutronix.de>
Tested-by: Arnaldo Carvalho de Melo <acme@redhat.com>
Cc: David Ahern <dsahern@gmail.com>
Cc: Frederic Weisbecker <fweisbec@gmail.com>
Cc: Jeremie Galarneau <jgalar@efficios.com>
Cc: Namhyung Kim <namhyung@gmail.com>
Cc: Paul Mackerras <paulus@samba.org>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Tom Zanussi <tzanussi@gmail.com>
Cc: Wang Nan <wangnan0@huawei.com>
Link: http://lkml.kernel.org/r/1424470628-5969-6-git-send-email-jolsa@kernel.org
Signed-off-by: Jiri Olsa <jolsa@kernel.org>
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
---
 tools/perf/util/data-convert-bt.c | 242 ++++++++++++++++++++++++++++++++++++++
 1 file changed, 242 insertions(+)

diff --git a/tools/perf/util/data-convert-bt.c b/tools/perf/util/data-convert-bt.c
index 1afd381..c6d6226 100644
--- a/tools/perf/util/data-convert-bt.c
+++ b/tools/perf/util/data-convert-bt.c
@@ -126,6 +126,177 @@ FUNC_VALUE_SET(s64)
 FUNC_VALUE_SET(u64)
 __FUNC_VALUE_SET(u64_hex, u64)
 
+static struct bt_ctf_field_type*
+get_tracepoint_field_type(struct ctf_writer *cw, struct format_field *field)
+{
+	unsigned long flags = field->flags;
+
+	if (flags & FIELD_IS_STRING)
+		return cw->data.string;
+
+	if (!(flags & FIELD_IS_SIGNED)) {
+		/* unsigned long are mostly pointers */
+		if (flags & FIELD_IS_LONG || flags & FIELD_IS_POINTER)
+			return cw->data.u64_hex;
+	}
+
+	if (flags & FIELD_IS_SIGNED) {
+		if (field->size == 8)
+			return cw->data.s64;
+		else
+			return cw->data.s32;
+	}
+
+	if (field->size == 8)
+		return cw->data.u64;
+	else
+		return cw->data.u32;
+}
+
+static int add_tracepoint_field_value(struct ctf_writer *cw,
+				      struct bt_ctf_event_class *event_class,
+				      struct bt_ctf_event *event,
+				      struct perf_sample *sample,
+				      struct format_field *fmtf)
+{
+	struct bt_ctf_field_type *type;
+	struct bt_ctf_field *array_field;
+	struct bt_ctf_field *field;
+	const char *name = fmtf->name;
+	void *data = sample->raw_data;
+	unsigned long long value_int;
+	unsigned long flags = fmtf->flags;
+	unsigned int n_items;
+	unsigned int i;
+	unsigned int offset;
+	unsigned int len;
+	int ret;
+
+	offset = fmtf->offset;
+	len = fmtf->size;
+	if (flags & FIELD_IS_STRING)
+		flags &= ~FIELD_IS_ARRAY;
+
+	if (flags & FIELD_IS_DYNAMIC) {
+		unsigned long long tmp_val;
+
+		tmp_val = pevent_read_number(fmtf->event->pevent,
+				data + offset, len);
+		offset = tmp_val;
+		len = offset >> 16;
+		offset &= 0xffff;
+	}
+
+	if (flags & FIELD_IS_ARRAY) {
+
+		type = bt_ctf_event_class_get_field_by_name(
+				event_class, name);
+		array_field = bt_ctf_field_create(type);
+		bt_ctf_field_type_put(type);
+		if (!array_field) {
+			pr_err("Failed to create array type %s\n", name);
+			return -1;
+		}
+
+		len = fmtf->size / fmtf->arraylen;
+		n_items = fmtf->arraylen;
+	} else {
+		n_items = 1;
+		array_field = NULL;
+	}
+
+	type = get_tracepoint_field_type(cw, fmtf);
+
+	for (i = 0; i < n_items; i++) {
+		if (!(flags & FIELD_IS_STRING))
+			value_int = pevent_read_number(
+					fmtf->event->pevent,
+					data + offset + i * len, len);
+
+		if (flags & FIELD_IS_ARRAY)
+			field = bt_ctf_field_array_get_field(array_field, i);
+		else
+			field = bt_ctf_field_create(type);
+
+		if (!field) {
+			pr_err("failed to create a field %s\n", name);
+			return -1;
+		}
+
+		if (flags & FIELD_IS_STRING)
+			ret = bt_ctf_field_string_set_value(field,
+					data + offset + i * len);
+		else if (!(flags & FIELD_IS_SIGNED))
+			ret = bt_ctf_field_unsigned_integer_set_value(
+					field, value_int);
+		else
+			ret = bt_ctf_field_signed_integer_set_value(
+					field, value_int);
+		if (ret) {
+			pr_err("failed to set file value %s\n", name);
+			goto err_put_field;
+		}
+		if (!(flags & FIELD_IS_ARRAY)) {
+			ret = bt_ctf_event_set_payload(event, name, field);
+			if (ret) {
+				pr_err("failed to set payload %s\n", name);
+				goto err_put_field;
+			}
+		}
+		bt_ctf_field_put(field);
+	}
+	if (flags & FIELD_IS_ARRAY) {
+		ret = bt_ctf_event_set_payload(event, name, array_field);
+		if (ret) {
+			pr_err("Failed add payload array %s\n", name);
+			return -1;
+		}
+		bt_ctf_field_put(array_field);
+	}
+	return 0;
+
+err_put_field:
+	bt_ctf_field_put(field);
+	return -1;
+}
+
+static int add_tracepoint_fields_values(struct ctf_writer *cw,
+					struct bt_ctf_event_class *event_class,
+					struct bt_ctf_event *event,
+					struct format_field *fields,
+					struct perf_sample *sample)
+{
+	struct format_field *field;
+	int ret;
+
+	for (field = fields; field; field = field->next) {
+		ret = add_tracepoint_field_value(cw, event_class, event, sample,
+				field);
+		if (ret)
+			return -1;
+	}
+	return 0;
+}
+
+static int add_tracepoint_values(struct ctf_writer *cw,
+				 struct bt_ctf_event_class *event_class,
+				 struct bt_ctf_event *event,
+				 struct perf_evsel *evsel,
+				 struct perf_sample *sample)
+{
+	struct format_field *common_fields = evsel->tp_format->format.common_fields;
+	struct format_field *fields        = evsel->tp_format->format.fields;
+	int ret;
+
+	ret = add_tracepoint_fields_values(cw, event_class, event,
+					   common_fields, sample);
+	if (!ret)
+		ret = add_tracepoint_fields_values(cw, event_class, event,
+						   fields, sample);
+
+	return ret;
+}
+
 static int add_generic_values(struct ctf_writer *cw,
 			      struct bt_ctf_event *event,
 			      struct perf_evsel *evsel,
@@ -246,11 +417,76 @@ static int process_sample_event(struct perf_tool *tool,
 	if (ret)
 		return -1;
 
+	if (evsel->attr.type == PERF_TYPE_TRACEPOINT) {
+		ret = add_tracepoint_values(cw, event_class, event,
+					    evsel, sample);
+		if (ret)
+			return -1;
+	}
+
 	bt_ctf_stream_append_event(cw->stream, event);
 	bt_ctf_event_put(event);
 	return 0;
 }
 
+static int add_tracepoint_fields_types(struct ctf_writer *cw,
+				       struct format_field *fields,
+				       struct bt_ctf_event_class *event_class)
+{
+	struct format_field *field;
+	int ret;
+
+	for (field = fields; field; field = field->next) {
+		struct bt_ctf_field_type *type;
+		unsigned long flags = field->flags;
+
+		pr2("  field '%s'\n", field->name);
+
+		type = get_tracepoint_field_type(cw, field);
+		if (!type)
+			return -1;
+
+		/*
+		 * A string is an array of chars. For this we use the string
+		 * type and don't care that it is an array. What we don't
+		 * support is an array of strings.
+		 */
+		if (flags & FIELD_IS_STRING)
+			flags &= ~FIELD_IS_ARRAY;
+
+		if (flags & FIELD_IS_ARRAY)
+			type = bt_ctf_field_type_array_create(type, field->arraylen);
+
+		ret = bt_ctf_event_class_add_field(event_class, type,
+				field->name);
+
+		if (flags & FIELD_IS_ARRAY)
+			bt_ctf_field_type_put(type);
+
+		if (ret) {
+			pr_err("Failed to add field '%s\n", field->name);
+			return -1;
+		}
+	}
+
+	return 0;
+}
+
+static int add_tracepoint_types(struct ctf_writer *cw,
+				struct perf_evsel *evsel,
+				struct bt_ctf_event_class *class)
+{
+	struct format_field *common_fields = evsel->tp_format->format.common_fields;
+	struct format_field *fields        = evsel->tp_format->format.fields;
+	int ret;
+
+	ret = add_tracepoint_fields_types(cw, common_fields, class);
+	if (!ret)
+		ret = add_tracepoint_fields_types(cw, fields, class);
+
+	return ret;
+}
+
 static int add_generic_types(struct ctf_writer *cw, struct perf_evsel *evsel,
 			     struct bt_ctf_event_class *event_class)
 {
@@ -328,6 +564,12 @@ static int add_event(struct ctf_writer *cw, struct perf_evsel *evsel)
 	if (ret)
 		goto err;
 
+	if (evsel->attr.type == PERF_TYPE_TRACEPOINT) {
+		ret = add_tracepoint_types(cw, evsel, event_class);
+		if (ret)
+			goto err;
+	}
+
 	ret = bt_ctf_stream_class_add_event_class(cw->stream_class, event_class);
 	if (ret) {
 		pr("Failed to add event class into stream.\n");

Re: [PATCH] perf build: Fix libbabeltrace detection

[Jiri Olsa]

On Tue, Mar 10, 2015 at 10:44:00AM -0400, Jérémie Galarneau wrote:
> On Tue, Mar 10, 2015 at 10:11 AM, Jiri Olsa <jolsa@redhat.com> wrote:
> > On Tue, Mar 10, 2015 at 11:01:34AM -0300, Arnaldo Carvalho de Melo wrote:
> >> Em Tue, Mar 10, 2015 at 01:00:35PM +0100, Jiri Olsa escreveu:
> >> > On Mon, Mar 09, 2015 at 08:28:45PM -0300, Arnaldo Carvalho de Melo wrote:
> >> > > Em Mon, Mar 09, 2015 at 08:11:19PM -0300, Arnaldo Carvalho de Melo escreveu:
> >> > > > Em Mon, Mar 09, 2015 at 06:51:21PM -0300, Arnaldo Carvalho de Melo escreveu:
> >> > > [root@zoo ~]# find /opt/libbabeltrace/include/babeltrace/ -type f | xargs grep bt_ctf_stream_class_get_packet_context_type
> >> > > /opt/libbabeltrace/include/babeltrace/ctf-ir/stream-class.h:extern struct bt_ctf_field_type *bt_ctf_stream_class_get_packet_context_type(
> >>
> >> > > That seems to be included from the file included in the feature test code :-\
> >> > > Really ran out of time now...
> >>
> >> > ouch, thanks a lot for debugging.. the reason is indeed the -Werror added by:
> >> >   b49f1a4be701 perf tools: Improve feature test debuggability
> >>
> >> > following patch seems to fix this for me
> >> > ---
> >> > Following patch added -Werror for feature builds:
> >> >   b49f1a4be701 perf tools: Improve feature test debuggability
> >>
> >> > and broke libbabeltrace feature build, because it was including
> >> > wrong header and gcc couldn't find the used symbol definition.
> >>
> >> > Adding proper header and keeping the old one as it is needed
> >> > also (libbabeltrace quirk).
> >>
> >> > +++ b/tools/perf/config/feature-checks/test-libbabeltrace.c
> >> >  #include <babeltrace/ctf-writer/writer.h>
> >> > +#include <babeltrace/ctf-ir/stream-class.h>
> >>
> >> Right, that was the fix I found as well, well, I was trying to include
> >> _just_ babeltrace/ctf-ir/stream-class.h, but that is buggy in that it
> >> doesn't include the header file with the uint32_t and int64_t types:
> >
> > yep, thats the 'libbabeltrace quirk' I mentioned in the changelog ;-)
> >
> 
> Seems I missed that being discussed. Fixed in Babeltrace master as of 48d711aa2.
> 

hi,
that seems to work, but I could now load ctf trace data only in babeltrace.
I've got following error in tracecompas while trying to load the trace:

 "No trace types found to match location /tmp/ctf-data/metadata"

but I might have old version of tracecompas and Im little fuzzy
on how to upgrade to new one.. CC-ing tracecompas guys ;-)

for anyone interested, the latest perf ctf code is in:
  git://git.kernel.org/pub/scm/linux/kernel/git/jolsa/perf.git
  perf/ctf branch

thanks,
jirka

Re: [PATCH] perf build: Fix libbabeltrace detection

[Arnaldo Carvalho de Melo]

Em Thu, Mar 26, 2015 at 10:25:21AM +0100, Jiri Olsa escreveu:
> On Tue, Mar 10, 2015 at 10:44:00AM -0400, Jérémie Galarneau wrote:
> > On Tue, Mar 10, 2015 at 10:11 AM, Jiri Olsa <jolsa@redhat.com> wrote:
> > > On Tue, Mar 10, 2015 at 11:01:34AM -0300, Arnaldo Carvalho de Melo wrote:
> > > yep, thats the 'libbabeltrace quirk' I mentioned in the changelog ;-)

> > Seems I missed that being discussed. Fixed in Babeltrace master as of 48d711aa2.
 
> that seems to work, but I could now load ctf trace data only in babeltrace.
> I've got following error in tracecompas while trying to load the trace:
> 
>  "No trace types found to match location /tmp/ctf-data/metadata"
> 
> but I might have old version of tracecompas and Im little fuzzy
> on how to upgrade to new one.. CC-ing tracecompas guys ;-)
> 
> for anyone interested, the latest perf ctf code is in:
>   git://git.kernel.org/pub/scm/linux/kernel/git/jolsa/perf.git
>   perf/ctf branch

Please let me know when you think this should go and what the steps are
for me to properly test it (libbabeltrace git sha I need to have
installed to get this working).

- Arnaldo