[RFC 0/23] perf tool: Add support for multiple data file storage

[RFC 0/23] perf tool: Add support for multiple data file storage

[Jiri Olsa]

hi,
this *RFC* adds support for record command to split perf
data file output into multiple files. Each such data file
is fully-fledged and readable upon by any perf command.

This patchset is actually doing 3 essential things:
  1) adds new perf data format version (v3)
  2) adds class/object to handle perf data file
  3) adds record '-M size' option

ad 1) Along the way I realized it'd be much easier to have the
      format simplified. I ended up with just 'DATA' and 'FEATURES'
      sections making the file format more generic.

      The easier/generic format will be for example useful for the
      cpu hotplug notification (and maybe even for ftrace data files
      features?)

      The version 2 should be still readable by perf, while it
      produces data in version 3 from this patchset onwards.

ad 2) I needed to unify perf data file handling to make the -M
      implementation clear.

Loose ends/TODO/ideas.. like:
  - test/fix bugs
  - more automated tests for version 2 data reading so we're
    sure we don't break v2 compatibility.
  - record '-M time'
  - separate work thread to finish the data file (write header),
    while record command continues to poll/read/store to new file
  - perf daemon running on background storing continuously data 
  - perf top storing data on key press

It's reachable here:
git://git.kernel.org/pub/scm/linux/kernel/git/jolsa/perf.git
perf/file22

thanks for comments,
jirka


Example:
  $ ./perf record -M 100000 -e cycles:u yes > /dev/null
  ^C[ perf record: Woken up 25 times to write data ]
  [ perf record: Captured and wrote 0.630 MB perf-[0-7].data(~27531 samples) ]
  yes: Interrupt
  $ ls -l perf-0000*
  -rw------- 1 jolsa jolsa 111828 Jul 17 18:23 perf-00000.data
  -rw------- 1 jolsa jolsa 115240 Jul 17 18:23 perf-00001.data
  -rw------- 1 jolsa jolsa 112020 Jul 17 18:23 perf-00002.data
  -rw------- 1 jolsa jolsa 112020 Jul 17 18:23 perf-00003.data
  -rw------- 1 jolsa jolsa 112120 Jul 17 18:23 perf-00004.data
  -rw------- 1 jolsa jolsa 112120 Jul 17 18:23 perf-00005.data
  -rw------- 1 jolsa jolsa  18900 Jul 17 18:23 perf-00006.data
  $ ./perf diff perf-0000*
  # Event 'cycles:u'
  #
  # Data files:
  #  [0] perf-00000.data (Baseline)
  #  [1] perf-00001.data
  #  [2] perf-00002.data
  #  [3] perf-00003.data
  #  [4] perf-00004.data
  #  [5] perf-00005.data
  #  [6] perf-00006.data
  #
  # Baseline/0  Delta/1  Delta/2  Delta/3  Delta/4  Delta/5  Delta/6      Shared Object                              Symbol
  # ..........  .......  .......  .......  .......  .......  .......  .................  ..................................
  #
        37.03%   +1.23%   +0.12%   +0.61%   +1.12%   +0.93%   -7.53%  libc-2.15.so       [.] _IO_file_xsputn@@GLIBC_2.2.5
        31.40%   -0.55%   -0.79%   -1.00%   -0.66%   -0.59%   +4.09%  yes                [.] main
        16.36%   -0.18%   +1.39%   +1.07%   +0.66%   +0.23%   +1.72%  libc-2.15.so       [.] __strlen_sse2
        14.66%   -0.61%   -0.81%   -0.58%   -1.18%   -0.26%   +1.66%  libc-2.15.so       [.] fputs_unlocked
         0.24%   +0.34%   +0.27%   +0.08%   +0.24%   -0.12%   +0.35%  yes                [.] fputs_unlocked@plt
         0.12%                     -0.08%   -0.08%   -0.08%           libc-2.15.so       [.] _IO_file_write@@GLIBC_2.2.5
         0.08%                                                        libc-2.15.so       [.] new_do_write
         0.04%                                                        libc-2.15.so       [.] _IO_default_xsputn
  ...



Signed-off-by: Jiri Olsa <jolsa@redhat.com>
Cc: Corey Ashford <cjashfor@linux.vnet.ibm.com>
Cc: Frederic Weisbecker <fweisbec@gmail.com>
Cc: Ingo Molnar <mingo@elte.hu>
Cc: Namhyung Kim <namhyung@kernel.org>
Cc: Paul Mackerras <paulus@samba.org>
Cc: Peter Zijlstra <a.p.zijlstra@chello.nl>
Cc: Arnaldo Carvalho de Melo <acme@redhat.com>
Cc: Andi Kleen <ak@linux.intel.com>
Cc: David Ahern <dsahern@gmail.com>
---
Jiri Olsa (23):
      perf tools: Use session->fd instead of passing fd as argument
      perf tools: Remove data_offset seek as it's not needed
      perf tools: Remove attr_offset from perf_header
      perf tools: Introduce feat_offset into perf_header
      perf tests: Add simple session read/write test
      perf doc: Add perf data file documentation
      perf tools: Recognize version number for perf data file
      perf tools: Introduce perf data file version CHECK macro
      perf tools: Introduce swap_features function
      perf tools: Introduce swap_header function
      perf tools: Separate version 2 specific perf data header bits
      perf tools: Using evlist as a holder for event_desc feature
      perf tools: Introduce perf.data version 3 format
      perf tools: Add perf data version 3 header swap
      perf tools: Add perf data version 3 header read
      perf tools: Add perf.data version 3 header write
      perf tools: Get rid of post_processing_offset in record command
      perf tools: Move synthetizing into single function
      perf tools: Add data object to handle perf data file
      perf tools: Add perf_data_file__open interface to data object
      perf tools: Separating data file properties from session
      perf tools: Add multi file '-M' option for record command
      perf tools: Have the process properly sythesized in subsequent data files

 tools/perf/Documentation/perf-data-file-v2.txt | 256 ++++++++++++++++++++++++++++++++++++++++++++++++
 tools/perf/Documentation/perf-data-file-v3.txt |  60 ++++++++++++
 tools/perf/Makefile                            |   2 +
 tools/perf/builtin-annotate.c                  |  11 ++-
 tools/perf/builtin-buildid-cache.c             |   8 +-
 tools/perf/builtin-buildid-list.c              |  11 ++-
 tools/perf/builtin-diff.c                      |  19 ++--
 tools/perf/builtin-evlist.c                    |   7 +-
 tools/perf/builtin-inject.c                    |   9 +-
 tools/perf/builtin-kmem.c                      |   7 +-
 tools/perf/builtin-kvm.c                       |   8 +-
 tools/perf/builtin-lock.c                      |   8 +-
 tools/perf/builtin-mem.c                       |   9 +-
 tools/perf/builtin-record.c                    | 400 +++++++++++++++++++++++++++++++++++++++++++++++++-------------------------
 tools/perf/builtin-report.c                    |  18 +++-
 tools/perf/builtin-sched.c                     |   6 +-
 tools/perf/builtin-script.c                    |  16 ++-
 tools/perf/builtin-timechart.c                 |  10 +-
 tools/perf/builtin-top.c                       |   7 +-
 tools/perf/perf.h                              |   2 +-
 tools/perf/tests/builtin-test.c                |   4 +
 tools/perf/tests/session-simple.c              | 275 +++++++++++++++++++++++++++++++++++++++++++++++++++
 tools/perf/tests/tests.h                       |   1 +
 tools/perf/util/data.c                         | 120 +++++++++++++++++++++++
 tools/perf/util/data.h                         |  48 +++++++++
 tools/perf/util/evsel.c                        |   5 +
 tools/perf/util/header.c                       | 450 +++++++++++++++++++++++++++++++++++++++++++++++++-----------------------------------
 tools/perf/util/header.h                       |  49 +++++++---
 tools/perf/util/session.c                      | 116 ++++++++--------------
 tools/perf/util/session.h                      |  11 +--
 30 files changed, 1501 insertions(+), 452 deletions(-)
 create mode 100644 tools/perf/Documentation/perf-data-file-v2.txt
 create mode 100644 tools/perf/Documentation/perf-data-file-v3.txt
 create mode 100644 tools/perf/tests/session-simple.c
 create mode 100644 tools/perf/util/data.c
 create mode 100644 tools/perf/util/data.h

[PATCH 01/23] perf tools: Use session->fd instead of passing fd as argument

[Jiri Olsa]

Using session->fd instead of passing fd as argument because it's
always session->fd that's passed as fd argument.

Signed-off-by: Jiri Olsa <jolsa@redhat.com>
Cc: Corey Ashford <cjashfor@linux.vnet.ibm.com>
Cc: Frederic Weisbecker <fweisbec@gmail.com>
Cc: Ingo Molnar <mingo@elte.hu>
Cc: Namhyung Kim <namhyung@kernel.org>
Cc: Paul Mackerras <paulus@samba.org>
Cc: Peter Zijlstra <a.p.zijlstra@chello.nl>
Cc: Arnaldo Carvalho de Melo <acme@redhat.com>
Cc: Andi Kleen <ak@linux.intel.com>
Cc: David Ahern <dsahern@gmail.com>
---
 tools/perf/util/header.c  | 11 +++++------
 tools/perf/util/header.h  |  2 +-
 tools/perf/util/session.c |  4 ++--
 3 files changed, 8 insertions(+), 9 deletions(-)

diff --git a/tools/perf/util/header.c b/tools/perf/util/header.c
index b28a65e..b3f253f 100644
--- a/tools/perf/util/header.c
+++ b/tools/perf/util/header.c
@@ -2619,19 +2619,17 @@ static int perf_file_header__read_pipe(struct perf_pipe_file_header *header,
 	return 0;
 }
 
-static int perf_header__read_pipe(struct perf_session *session, int fd)
+static int perf_header__read_pipe(struct perf_session *session)
 {
 	struct perf_header *header = &session->header;
 	struct perf_pipe_file_header f_header;
 
-	if (perf_file_header__read_pipe(&f_header, header, fd,
+	if (perf_file_header__read_pipe(&f_header, header, session->fd,
 					session->repipe) < 0) {
 		pr_debug("incompatible file format\n");
 		return -EINVAL;
 	}
 
-	session->fd = fd;
-
 	return 0;
 }
 
@@ -2725,20 +2723,21 @@ static int perf_evlist__prepare_tracepoint_events(struct perf_evlist *evlist,
 	return 0;
 }
 
-int perf_session__read_header(struct perf_session *session, int fd)
+int perf_session__read_header(struct perf_session *session)
 {
 	struct perf_header *header = &session->header;
 	struct perf_file_header	f_header;
 	struct perf_file_attr	f_attr;
 	u64			f_id;
 	int nr_attrs, nr_ids, i, j;
+	int fd = session->fd;
 
 	session->evlist = perf_evlist__new();
 	if (session->evlist == NULL)
 		return -ENOMEM;
 
 	if (session->fd_pipe)
-		return perf_header__read_pipe(session, fd);
+		return perf_header__read_pipe(session);
 
 	if (perf_file_header__read(&f_header, header, fd) < 0)
 		return -EINVAL;
diff --git a/tools/perf/util/header.h b/tools/perf/util/header.h
index 669fda5..0bacb2d 100644
--- a/tools/perf/util/header.h
+++ b/tools/perf/util/header.h
@@ -96,7 +96,7 @@ struct perf_header {
 struct perf_evlist;
 struct perf_session;
 
-int perf_session__read_header(struct perf_session *session, int fd);
+int perf_session__read_header(struct perf_session *session);
 int perf_session__write_header(struct perf_session *session,
 			       struct perf_evlist *evlist,
 			       int fd, bool at_exit);
diff --git a/tools/perf/util/session.c b/tools/perf/util/session.c
index d0d9f94..272c9cf 100644
--- a/tools/perf/util/session.c
+++ b/tools/perf/util/session.c
@@ -24,7 +24,7 @@ static int perf_session__open(struct perf_session *self, bool force)
 		self->fd_pipe = true;
 		self->fd = STDIN_FILENO;
 
-		if (perf_session__read_header(self, self->fd) < 0)
+		if (perf_session__read_header(self) < 0)
 			pr_err("incompatible file format (rerun with -v to learn more)");
 
 		return 0;
@@ -56,7 +56,7 @@ static int perf_session__open(struct perf_session *self, bool force)
 		goto out_close;
 	}
 
-	if (perf_session__read_header(self, self->fd) < 0) {
+	if (perf_session__read_header(self) < 0) {
 		pr_err("incompatible file format (rerun with -v to learn more)");
 		goto out_close;
 	}
-- 
1.7.11.7

[PATCH 02/23] perf tools: Remove data_offset seek as it's not needed

[Jiri Olsa]

Removing data_offset seek as it's not needed, because
data are not read by syscall but mmaped instead.

Signed-off-by: Jiri Olsa <jolsa@redhat.com>
Cc: Corey Ashford <cjashfor@linux.vnet.ibm.com>
Cc: Frederic Weisbecker <fweisbec@gmail.com>
Cc: Ingo Molnar <mingo@elte.hu>
Cc: Namhyung Kim <namhyung@kernel.org>
Cc: Paul Mackerras <paulus@samba.org>
Cc: Peter Zijlstra <a.p.zijlstra@chello.nl>
Cc: Arnaldo Carvalho de Melo <acme@redhat.com>
Cc: Andi Kleen <ak@linux.intel.com>
Cc: David Ahern <dsahern@gmail.com>
---
 tools/perf/util/header.c | 2 --
 1 file changed, 2 deletions(-)

diff --git a/tools/perf/util/header.c b/tools/perf/util/header.c
index b3f253f..035efe7 100644
--- a/tools/perf/util/header.c
+++ b/tools/perf/util/header.c
@@ -2794,8 +2794,6 @@ int perf_session__read_header(struct perf_session *session)
 	perf_header__process_sections(header, fd, &session->pevent,
 				      perf_file_section__process);
 
-	lseek(fd, header->data_offset, SEEK_SET);
-
 	if (perf_evlist__prepare_tracepoint_events(session->evlist,
 						   session->pevent))
 		goto out_delete_evlist;
-- 
1.7.11.7

[PATCH 03/23] perf tools: Remove attr_offset from perf_header

[Jiri Olsa]

Removing attr_offset from perf_header as it's possible
to use it as a local variable.

Signed-off-by: Jiri Olsa <jolsa@redhat.com>
Cc: Corey Ashford <cjashfor@linux.vnet.ibm.com>
Cc: Frederic Weisbecker <fweisbec@gmail.com>
Cc: Ingo Molnar <mingo@elte.hu>
Cc: Namhyung Kim <namhyung@kernel.org>
Cc: Paul Mackerras <paulus@samba.org>
Cc: Peter Zijlstra <a.p.zijlstra@chello.nl>
Cc: Arnaldo Carvalho de Melo <acme@redhat.com>
Cc: Andi Kleen <ak@linux.intel.com>
Cc: David Ahern <dsahern@gmail.com>
---
 tools/perf/util/header.c | 5 +++--
 tools/perf/util/header.h | 1 -
 2 files changed, 3 insertions(+), 3 deletions(-)

diff --git a/tools/perf/util/header.c b/tools/perf/util/header.c
index 035efe7..dcba56a 100644
--- a/tools/perf/util/header.c
+++ b/tools/perf/util/header.c
@@ -2272,6 +2272,7 @@ int perf_session__write_header(struct perf_session *session,
 	struct perf_file_attr   f_attr;
 	struct perf_header *header = &session->header;
 	struct perf_evsel *evsel;
+	u64 attr_offset;
 	int err;
 
 	lseek(fd, sizeof(f_header), SEEK_SET);
@@ -2285,7 +2286,7 @@ int perf_session__write_header(struct perf_session *session,
 		}
 	}
 
-	header->attr_offset = lseek(fd, 0, SEEK_CUR);
+	attr_offset = lseek(fd, 0, SEEK_CUR);
 
 	list_for_each_entry(evsel, &evlist->entries, node) {
 		f_attr = (struct perf_file_attr){
@@ -2315,7 +2316,7 @@ int perf_session__write_header(struct perf_session *session,
 		.size	   = sizeof(f_header),
 		.attr_size = sizeof(f_attr),
 		.attrs = {
-			.offset = header->attr_offset,
+			.offset = attr_offset,
 			.size   = evlist->nr_entries * sizeof(f_attr),
 		},
 		.data = {
diff --git a/tools/perf/util/header.h b/tools/perf/util/header.h
index 0bacb2d..6fa80f9 100644
--- a/tools/perf/util/header.h
+++ b/tools/perf/util/header.h
@@ -86,7 +86,6 @@ struct perf_session_env {
 
 struct perf_header {
 	bool			needs_swap;
-	s64			attr_offset;
 	u64			data_offset;
 	u64			data_size;
 	DECLARE_BITMAP(adds_features, HEADER_FEAT_BITS);
-- 
1.7.11.7

[PATCH 04/23] perf tools: Introduce feat_offset into perf_header

[Jiri Olsa]

Introducing feat_offset into perf_header to make the
location of the features section clear.

Signed-off-by: Jiri Olsa <jolsa@redhat.com>
Cc: Corey Ashford <cjashfor@linux.vnet.ibm.com>
Cc: Frederic Weisbecker <fweisbec@gmail.com>
Cc: Ingo Molnar <mingo@elte.hu>
Cc: Namhyung Kim <namhyung@kernel.org>
Cc: Paul Mackerras <paulus@samba.org>
Cc: Peter Zijlstra <a.p.zijlstra@chello.nl>
Cc: Arnaldo Carvalho de Melo <acme@redhat.com>
Cc: Andi Kleen <ak@linux.intel.com>
Cc: David Ahern <dsahern@gmail.com>
---
 tools/perf/util/header.c | 6 ++++--
 tools/perf/util/header.h | 1 +
 2 files changed, 5 insertions(+), 2 deletions(-)

diff --git a/tools/perf/util/header.c b/tools/perf/util/header.c
index dcba56a..eefb052 100644
--- a/tools/perf/util/header.c
+++ b/tools/perf/util/header.c
@@ -2225,7 +2225,7 @@ static int perf_header__adds_write(struct perf_header *header,
 
 	sec_size = sizeof(*feat_sec) * nr_sections;
 
-	sec_start = header->data_offset + header->data_size;
+	sec_start = header->feat_offset;
 	lseek(fd, sec_start + sec_size, SEEK_SET);
 
 	for_each_set_bit(feat, header->adds_features, HEADER_FEAT_BITS) {
@@ -2304,6 +2304,7 @@ int perf_session__write_header(struct perf_session *session,
 	}
 
 	header->data_offset = lseek(fd, 0, SEEK_CUR);
+	header->feat_offset = header->data_offset + header->data_size;
 
 	if (at_exit) {
 		err = perf_header__adds_write(header, evlist, fd);
@@ -2373,7 +2374,7 @@ int perf_header__process_sections(struct perf_header *header, int fd,
 
 	sec_size = sizeof(*feat_sec) * nr_sections;
 
-	lseek(fd, header->data_offset + header->data_size, SEEK_SET);
+	lseek(fd, header->feat_offset, SEEK_SET);
 
 	err = perf_header__getbuffer64(header, fd, feat_sec, sec_size);
 	if (err < 0)
@@ -2572,6 +2573,7 @@ int perf_file_header__read(struct perf_file_header *header,
 
 	ph->data_offset  = header->data.offset;
 	ph->data_size	 = header->data.size;
+	ph->feat_offset  = header->data.offset + header->data.size;
 	return 0;
 }
 
diff --git a/tools/perf/util/header.h b/tools/perf/util/header.h
index 6fa80f9..e6387dc 100644
--- a/tools/perf/util/header.h
+++ b/tools/perf/util/header.h
@@ -88,6 +88,7 @@ struct perf_header {
 	bool			needs_swap;
 	u64			data_offset;
 	u64			data_size;
+	u64			feat_offset;
 	DECLARE_BITMAP(adds_features, HEADER_FEAT_BITS);
 	struct perf_session_env env;
 };
-- 
1.7.11.7

[PATCH 05/23] perf tests: Add simple session read/write test

[Jiri Olsa]

Adding simple session read/write test to keep up
with file format changes.

Signed-off-by: Jiri Olsa <jolsa@redhat.com>
Cc: Corey Ashford <cjashfor@linux.vnet.ibm.com>
Cc: Frederic Weisbecker <fweisbec@gmail.com>
Cc: Ingo Molnar <mingo@elte.hu>
Cc: Namhyung Kim <namhyung@kernel.org>
Cc: Paul Mackerras <paulus@samba.org>
Cc: Peter Zijlstra <a.p.zijlstra@chello.nl>
Cc: Arnaldo Carvalho de Melo <acme@redhat.com>
Cc: Andi Kleen <ak@linux.intel.com>
Cc: David Ahern <dsahern@gmail.com>
---
 tools/perf/Makefile               |   1 +
 tools/perf/tests/builtin-test.c   |   4 +
 tools/perf/tests/session-simple.c | 274 ++++++++++++++++++++++++++++++++++++++
 tools/perf/tests/tests.h          |   1 +
 4 files changed, 280 insertions(+)
 create mode 100644 tools/perf/tests/session-simple.c

diff --git a/tools/perf/Makefile b/tools/perf/Makefile
index 5b7c6db..ca3e17b 100644
--- a/tools/perf/Makefile
+++ b/tools/perf/Makefile
@@ -389,6 +389,7 @@ LIB_OBJS += $(OUTPUT)tests/bp_signal.o
 LIB_OBJS += $(OUTPUT)tests/bp_signal_overflow.o
 LIB_OBJS += $(OUTPUT)tests/task-exit.o
 LIB_OBJS += $(OUTPUT)tests/sw-clock.o
+LIB_OBJS += $(OUTPUT)tests/session-simple.o
 
 BUILTIN_OBJS += $(OUTPUT)builtin-annotate.o
 BUILTIN_OBJS += $(OUTPUT)builtin-bench.o
diff --git a/tools/perf/tests/builtin-test.c b/tools/perf/tests/builtin-test.c
index 35b45f1466..67fc0ea 100644
--- a/tools/perf/tests/builtin-test.c
+++ b/tools/perf/tests/builtin-test.c
@@ -94,6 +94,10 @@ static struct test {
 		.func = test__sw_clock_freq,
 	},
 	{
+		.desc = "Test session - simple read/write",
+		.func = test__session_simple,
+	},
+	{
 		.func = NULL,
 	},
 };
diff --git a/tools/perf/tests/session-simple.c b/tools/perf/tests/session-simple.c
new file mode 100644
index 0000000..3bcd7eb
--- /dev/null
+++ b/tools/perf/tests/session-simple.c
@@ -0,0 +1,274 @@
+
+#include <stdlib.h>
+#include <linux/kernel.h>
+
+#include "tests.h"
+#include "session.h"
+#include "header.h"
+#include "util.h"
+#include "evlist.h"
+
+
+static char *get_file(void)
+{
+	static char buf[PATH_MAX];
+	static char *file;
+	int fd;
+
+	if (file)
+		return file;
+
+	snprintf(buf, PATH_MAX, "/tmp/perf-test-session-simple-data-XXXXXX");
+	fd = mkstemp(buf);
+	if (fd < 0) {
+		pr_err("mkstemp failed");
+		return NULL;
+	}
+
+	close(fd);
+
+	file = buf;
+	pr_debug("data file %s\n", file);
+	return file;
+}
+
+static struct perf_evlist *get_evlist(void)
+{
+	struct perf_evlist *evlist;
+	int err;
+
+	evlist = perf_evlist__new();
+	if (evlist == NULL) {
+		pr_err("perf_evlist__new failed\n");
+		return NULL;
+	}
+
+	err = perf_evlist__add_default(evlist);
+	if (err < 0) {
+		pr_err("Not enough memory to create evsel\n");
+		perf_evlist__delete(evlist);
+		evlist = NULL;
+	}
+
+	return evlist;
+}
+
+static union perf_event *get_event_MMAP(void)
+{
+	static union perf_event event;
+	size_t size;
+
+	size = snprintf(event.mmap.filename, sizeof(event.mmap.filename),
+			"krava") + 1;
+	size = PERF_ALIGN(size, sizeof(u64));
+
+	event.header.type = PERF_RECORD_MMAP;
+	event.header.misc = PERF_RECORD_MISC_KERNEL;
+	event.header.size = sizeof(event.mmap) -
+			    (sizeof(event.mmap.filename) - size);
+
+	event.mmap.pgoff = 10;
+	event.mmap.start = 0;
+	event.mmap.len   = 10;
+	event.mmap.pid   = 123;
+
+	return &event;
+}
+
+static union perf_event *get_event_COMM(void)
+{
+	static union perf_event event;
+	size_t size;
+
+	size = snprintf(event.comm.comm, sizeof(event.comm.comm),
+			"krava") + 1;
+	size = PERF_ALIGN(size, sizeof(u64));
+
+	event.header.type = PERF_RECORD_COMM;
+	event.header.size = sizeof(event.comm) -
+			    (sizeof(event.comm.comm) - size);
+
+	event.comm.pid = 1234;
+	event.comm.tid = 4321;
+
+	return &event;
+}
+
+static int store_event(int fd, union perf_event *event, size_t *size)
+{
+	*size += event->header.size;
+	return write(fd, event, event->header.size) > 0 ? 0 : -1;
+}
+
+static int session_write(void)
+{
+	struct perf_session *session;
+	struct perf_evlist *evlist;
+	size_t size = 0;
+	char *file;
+	int feat, fd;
+
+	file = get_file();
+	TEST_ASSERT_VAL("failed to get temporary file", file);
+
+	fd = open(file, O_RDWR);
+	TEST_ASSERT_VAL("failed to open data file", fd >= 0);
+
+	evlist = get_evlist();
+	TEST_ASSERT_VAL("failed to get evlist", evlist);
+
+	pr_debug("session writing start\n");
+
+	session = perf_session__new(file, O_WRONLY, true, false, NULL);
+	TEST_ASSERT_VAL("failed to create session", session);
+
+	session->evlist = evlist;
+	session->fd     = fd;
+
+	for (feat = HEADER_FIRST_FEATURE; feat < HEADER_LAST_FEATURE; feat++)
+		perf_header__set_feat(&session->header, feat);
+
+	perf_header__clear_feat(&session->header, HEADER_BUILD_ID);
+	perf_header__clear_feat(&session->header, HEADER_TRACING_DATA);
+	perf_header__clear_feat(&session->header, HEADER_BRANCH_STACK);
+
+	TEST_ASSERT_VAL("failed to write header",
+		!perf_session__write_header(session, evlist, fd, false));
+
+	TEST_ASSERT_VAL("failed to store MMAP event",
+		!store_event(fd, get_event_MMAP(), &size));
+
+	TEST_ASSERT_VAL("failed to store COMM event",
+		!store_event(fd, get_event_COMM(), &size));
+
+	session->header.data_size += size;
+
+	TEST_ASSERT_VAL("failed to write header",
+		!perf_session__write_header(session, evlist, fd, true));
+
+	perf_session__delete(session);
+	perf_evlist__delete(evlist);
+
+	pr_debug("session writing stop\n");
+	return 0;
+}
+
+static int events_comm;
+static int events_mmap;
+
+static int process_comm(struct perf_tool *tool __maybe_unused,
+			union perf_event *event,
+			struct perf_sample *sample __maybe_unused,
+			struct machine *machine __maybe_unused)
+{
+	pr_debug("event COMM pid %d, tid %d, comm '%s'\n",
+		 event->comm.pid,
+		 event->comm.pid,
+		 event->comm.comm);
+
+	TEST_ASSERT_VAL("wrong COMM pid\n",
+		event->comm.pid == 1234);
+
+	TEST_ASSERT_VAL("wrong COMM tid\n",
+		event->comm.tid == 4321);
+
+	TEST_ASSERT_VAL("wrong COMM comm\n",
+		!strcmp(event->comm.comm,"krava"));
+
+	events_comm++;
+	return 0;
+}
+
+static int process_mmap(struct perf_tool *tool __maybe_unused,
+			union perf_event *event,
+			struct perf_sample *sample __maybe_unused,
+			struct machine *machine __maybe_unused)
+{
+	pr_debug("event MMAP misc %d, pgoff %lu, start %lu,"
+		 " len %lu, pid %d, filename '%s'\n",
+		 event->header.misc,
+		 event->mmap.pgoff,
+		 event->mmap.start,
+		 event->mmap.len,
+		 event->mmap.pid,
+		 event->mmap.filename);
+
+	TEST_ASSERT_VAL("wrong MMAP misc\n",
+		event->header.misc == PERF_RECORD_MISC_KERNEL);
+
+	TEST_ASSERT_VAL("wrong MMAP pgoff\n",
+		event->mmap.pgoff == 10);
+
+	TEST_ASSERT_VAL("wrong MMAP start\n",
+		event->mmap.start == 0);
+
+	TEST_ASSERT_VAL("wrong MMAP len\n",
+		event->mmap.len == 10);
+
+	TEST_ASSERT_VAL("wrong MMAP pid\n",
+		event->mmap.pid == 123);
+
+	TEST_ASSERT_VAL("wrong MMAP filename\n",
+		!strcmp(event->mmap.filename,"krava"));
+
+	TEST_ASSERT_VAL("wrong MMAP misc\n",
+		event->header.misc == PERF_RECORD_MISC_KERNEL);
+
+	TEST_ASSERT_VAL("wrong MMAP pgoff\n",
+		event->mmap.pgoff == 10);
+
+	TEST_ASSERT_VAL("wrong MMAP start\n",
+		event->mmap.start == 0);
+
+	TEST_ASSERT_VAL("wrong MMAP len\n",
+		event->mmap.len == 10);
+
+	TEST_ASSERT_VAL("wrong MMAP pid\n",
+		event->mmap.pid == 123);
+
+	TEST_ASSERT_VAL("wrong MMAP filename\n",
+		!strcmp(event->mmap.filename,"krava"));
+
+	events_mmap++;
+	return 0;
+}
+
+static int session_read(void)
+{
+	struct perf_session *session;
+	struct perf_tool tool = {
+		.mmap = process_mmap,
+		.comm = process_comm,
+	};
+	char *file;
+
+	file = get_file();
+	TEST_ASSERT_VAL("failed to get temporary file", file);
+
+	pr_debug("session reading start\n");
+
+	session = perf_session__new(file, O_RDONLY, false, false, &tool);
+	TEST_ASSERT_VAL("failed to create session", session);
+
+	TEST_ASSERT_VAL("failed to process events", session);
+		perf_session__process_events(session, &tool);
+
+	perf_session__delete(session);
+
+	pr_debug("session reading stop\n");
+
+	return 0;
+}
+
+int test__session_simple(void)
+{
+	TEST_ASSERT_VAL("failed to write data", !session_write());
+	TEST_ASSERT_VAL("failed to read data",  !session_read());
+
+	unlink(get_file());
+
+	TEST_ASSERT_VAL("wrong MMAP events count", events_mmap == 1);
+	TEST_ASSERT_VAL("wrong COMM events count", events_comm == 1);
+
+	return 0;
+}
diff --git a/tools/perf/tests/tests.h b/tools/perf/tests/tests.h
index 07a92f9..155b474 100644
--- a/tools/perf/tests/tests.h
+++ b/tools/perf/tests/tests.h
@@ -35,5 +35,6 @@ int test__bp_signal(void);
 int test__bp_signal_overflow(void);
 int test__task_exit(void);
 int test__sw_clock_freq(void);
+int test__session_simple(void);
 
 #endif /* TESTS_H */
-- 
1.7.11.7

[PATCH 06/23] perf doc: Add perf data file documentation

[Jiri Olsa]

Adding perf data file documentation.

TODO: Finish the FEATURES section

Signed-off-by: Jiri Olsa <jolsa@redhat.com>
Cc: Corey Ashford <cjashfor@linux.vnet.ibm.com>
Cc: Frederic Weisbecker <fweisbec@gmail.com>
Cc: Ingo Molnar <mingo@elte.hu>
Cc: Namhyung Kim <namhyung@kernel.org>
Cc: Paul Mackerras <paulus@samba.org>
Cc: Peter Zijlstra <a.p.zijlstra@chello.nl>
Cc: Arnaldo Carvalho de Melo <acme@redhat.com>
Cc: Andi Kleen <ak@linux.intel.com>
Cc: David Ahern <dsahern@gmail.com>
---
 tools/perf/Documentation/perf-data-file-v2.txt | 255 +++++++++++++++++++++++++
 1 file changed, 255 insertions(+)
 create mode 100644 tools/perf/Documentation/perf-data-file-v2.txt

diff --git a/tools/perf/Documentation/perf-data-file-v2.txt b/tools/perf/Documentation/perf-data-file-v2.txt
new file mode 100644
index 0000000..3931dca
--- /dev/null
+++ b/tools/perf/Documentation/perf-data-file-v2.txt
@@ -0,0 +1,255 @@
+perf-data-file-v2(1)
+====================
+Following text describes version 2 of the perf data file format,
+which is version that is currently used by perf tool.
+
+The perf data file format is composed of several sections
+describing monitored events and the data itself.
+
+High level view of the format:
+  FILE HEADER
+  EVENT IDS
+  EVENT ATTRIBUTES
+  EVENT TYPES
+  EVENT DATA
+  FEATURES
+
+
+COMMON PRIMITIVES
+-----------------
+Following definitions are used as primitives on multiple places
+within the perf data file.
+
+struct perf_file_section::
+  Used to determine location of data sections within the file.
+
+  struct perf_file_section {
+        u64 offset;
+        u64 size;
+  };
+
+  offset  section location (offset) within the file
+  size    section size in bytes
+
+
+FILE HEADER
+-----------
+Starting point of the data file with magic bytes and global
+section information.
+
+The section contains following data:
+  struct perf_file_header header
+
+struct perf_file_header::
+	struct perf_file_header {
+		u64                             magic;
+		u64                             size;
+		u64                             attr_size;
+		struct perf_file_section        attrs;
+		struct perf_file_section        data;
+		struct perf_file_section        event_types;
+		DECLARE_BITMAP(adds_features, HEADER_FEAT_BITS);
+	}
+
+	magic           ID 'PERFFILE|PERFILE2'
+	size            header size minus sizeof(struct perf_file_header)
+	attr_size       attribute size minus sizeof(struct perf_file_attr)
+	attrs           location of 'EVENT ATTRIBUTES'
+	data            location of 'EVENT DATA'
+	event_types     location of 'EVENT TYPES'
+	adds_features   'FEATURES' bitmask
+
+
+EVENT IDS
+---------
+This section contains CPU related IDs for each monitored event.
+It is used only if there was more than 1 events monitored. The
+reason is to be able to map event's data to event definition.
+
+The section contains following data:
+  u64  event1_ids[]
+  u64  event2_ids[]
+  ...
+  u64  eventX_ids[]
+
+eventX_ids::
+  per-cpu event X IDs array
+  this array is referenced in 'EVENT ATTRIBUTES'
+
+
+EVENT ATTRIBUTES
+----------------
+This section contains attribute information together with the
+location of the IDs array for each event.
+
+The section contains following data:
+  struct perf_file_attr  attrs[]
+
+struct perf_file_attr::
+  struct perf_file_attr {
+        struct perf_event_attr    attr;
+        struct perf_file_section  ids;
+  }
+
+  attr  perf event interface structure for each event
+  ids   IDs location for event in 'EVENT IDS' section
+
+
+EVENT TYPES
+-----------
+This section map event's config value (struct perf_event_attr::config)
+with the event name.
+
+The section contains following data:
+  struct perf_trace_event_type  types[]
+
+#define MAX_EVENT_NAME 64
+
+struct perf_trace_event_type::
+  struct perf_trace_event_type {
+        u64   event_id;
+        char  name[MAX_EVENT_NAME];
+  };
+
+  event_id  struct perf_event_attr::config value
+  name      event name
+
+NOTE This section got deprecated and its data are no longer stored.
+     The 'event_types' location in the header is zero-ed.
+
+
+EVENT DATA
+----------
+This section contains blob of all events' data - auxiliary events
+and samples.
+
+
+FEATURES
+--------
+This section contains various configuration data and its contents
+depends on the header's adds_features bitmask. Each bit in this
+bitmask represent a single feature (0 - NOT present, 1 present).
+
+The section contains following data:
+  struct perf_file_section feature_locations[nr_features];
+
+  nr_features        number of present features bitmap_weight(adds_features)
+  feature_locations  location of each feature data
+
+Available features (possible max 256):
+  enum {
+        HEADER_RESERVED         = 0,
+        HEADER_TRACING_DATA     = 1,
+        HEADER_BUILD_ID,
+        HEADER_HOSTNAME,
+        HEADER_OSRELEASE,
+        HEADER_VERSION,
+        HEADER_ARCH,
+        HEADER_NRCPUS,
+        HEADER_CPUDESC,
+        HEADER_CPUID,
+        HEADER_TOTAL_MEM,
+        HEADER_CMDLINE,
+        HEADER_EVENT_DESC,
+        HEADER_CPU_TOPOLOGY,
+        HEADER_NUMA_TOPOLOGY,
+        HEADER_BRANCH_STACK,
+        HEADER_PMU_MAPPINGS,
+        HEADER_GROUP_DESC,
+        HEADER_LAST_FEATURE,
+        HEADER_FEAT_BITS        = 256,
+  }
+
+
+TODO finish FEATURES::* sections descriptions.
+
+FEATURES::HEADER_RESERVED
+-------------------------
+allways zero bit
+
+
+FEATURES::HEADER_TRACING_DATA
+-----------------------------
+tracing data
+
+
+FEATURES::HEADER_BUILD_ID
+-------------------------
+build ids
+
+
+FEATURES::HEADER_HOSTNAME
+-------------------------
+uname hostname
+
+
+FEATURES::HEADER_OSRELEASE
+--------------------------
+uname osrelease
+
+
+FEATURES::HEADER_VERSION
+------------------------
+perf version
+
+
+FEATURES::HEADER_ARCH
+---------------------
+uname arch
+
+
+FEATURES::HEADER_NRCPUS
+-----------------------
+offset          size(B) description
+0               4       sysconf(_SC_NPROCESSORS_CONF)
+4               4       sysconf(_SC_NPROCESSORS_ONLN)
+
+
+FEATURES::HEADER_CPUDESC
+------------------------
+/proc/cpuinfo data for 'Model name'
+
+
+FEATURES::HEADER_CPUID
+----------------------
+'vendor,family,model,step'
+
+
+FEATURES::HEADER_TOTAL_MEM
+--------------------------
+/proc/meminfo data for 'MemTotal'
+
+
+FEATURES::HEADER_CMDLINE
+------------------------
+perf complete command line
+
+
+FEATURES::HEADER_EVENT_DESC
+---------------------------
+event attributes plus related info(ids)
+
+
+FEATURES::HEADER_CPU_TOPOLOGY
+-----------------------------
+cpu topology
+
+
+FEATURES::HEADER_NUMA_TOPOLOGY
+------------------------------
+numa topology
+
+
+FEATURES::HEADER_BRANCH_STACK
+-----------------------------
+nothing
+
+
+FEATURES::HEADER_PMU_MAPPINGS
+-----------------------------
+available pmus
+
+
+FEATURES::HEADER_GROUP_DESC
+---------------------------
+event groups info
-- 
1.7.11.7

[PATCH 07/23] perf tools: Recognize version number for perf data file

[Jiri Olsa]

Keep the recognized data file version within 'struct perf_header'.

Signed-off-by: Jiri Olsa <jolsa@redhat.com>
Cc: Corey Ashford <cjashfor@linux.vnet.ibm.com>
Cc: Frederic Weisbecker <fweisbec@gmail.com>
Cc: Ingo Molnar <mingo@elte.hu>
Cc: Namhyung Kim <namhyung@kernel.org>
Cc: Paul Mackerras <paulus@samba.org>
Cc: Peter Zijlstra <a.p.zijlstra@chello.nl>
Cc: Arnaldo Carvalho de Melo <acme@redhat.com>
Cc: Andi Kleen <ak@linux.intel.com>
Cc: David Ahern <dsahern@gmail.com>
---
 tools/perf/util/header.c |  2 ++
 tools/perf/util/header.h | 16 +++++++++++-----
 2 files changed, 13 insertions(+), 5 deletions(-)

diff --git a/tools/perf/util/header.c b/tools/perf/util/header.c
index eefb052..f558f83 100644
--- a/tools/perf/util/header.c
+++ b/tools/perf/util/header.c
@@ -2480,6 +2480,7 @@ static int check_magic_endian(u64 magic, uint64_t hdr_sz,
 	/* check for legacy format */
 	ret = memcmp(&magic, __perf_magic1, sizeof(magic));
 	if (ret == 0) {
+		ph->version = PERF_HEADER_VERSION_1;
 		pr_debug("legacy perf.data format\n");
 		if (is_pipe)
 			return try_all_pipe_abis(hdr_sz, ph);
@@ -2501,6 +2502,7 @@ static int check_magic_endian(u64 magic, uint64_t hdr_sz,
 		return -1;
 
 	ph->needs_swap = true;
+	ph->version = PERF_HEADER_VERSION_2;
 
 	return 0;
 }
diff --git a/tools/perf/util/header.h b/tools/perf/util/header.h
index e6387dc..307c9ae 100644
--- a/tools/perf/util/header.h
+++ b/tools/perf/util/header.h
@@ -34,6 +34,11 @@ enum {
 	HEADER_FEAT_BITS	= 256,
 };
 
+enum perf_header_version {
+	PERF_HEADER_VERSION_1,
+	PERF_HEADER_VERSION_2,
+};
+
 struct perf_file_section {
 	u64 offset;
 	u64 size;
@@ -85,12 +90,13 @@ struct perf_session_env {
 };
 
 struct perf_header {
-	bool			needs_swap;
-	u64			data_offset;
-	u64			data_size;
-	u64			feat_offset;
+	enum perf_header_version	version;
+	bool				needs_swap;
+	u64				data_offset;
+	u64				data_size;
+	u64				feat_offset;
 	DECLARE_BITMAP(adds_features, HEADER_FEAT_BITS);
-	struct perf_session_env env;
+	struct perf_session_env 	env;
 };
 
 struct perf_evlist;
-- 
1.7.11.7

[PATCH 08/23] perf tools: Introduce perf data file version CHECK macro

[Jiri Olsa]

Introducing perf data file version CHECK macro, so the check
is easily extensible for new versions.

Signed-off-by: Jiri Olsa <jolsa@redhat.com>
Cc: Corey Ashford <cjashfor@linux.vnet.ibm.com>
Cc: Frederic Weisbecker <fweisbec@gmail.com>
Cc: Ingo Molnar <mingo@elte.hu>
Cc: Namhyung Kim <namhyung@kernel.org>
Cc: Paul Mackerras <paulus@samba.org>
Cc: Peter Zijlstra <a.p.zijlstra@chello.nl>
Cc: Arnaldo Carvalho de Melo <acme@redhat.com>
Cc: Andi Kleen <ak@linux.intel.com>
Cc: David Ahern <dsahern@gmail.com>
---
 tools/perf/util/header.c | 27 ++++++++++++++++-----------
 1 file changed, 16 insertions(+), 11 deletions(-)

diff --git a/tools/perf/util/header.c b/tools/perf/util/header.c
index f558f83..4ebe4a1 100644
--- a/tools/perf/util/header.c
+++ b/tools/perf/util/header.c
@@ -2493,18 +2493,23 @@ static int check_magic_endian(u64 magic, uint64_t hdr_sz,
 	 * - encode endianness of file
 	 */
 
-	/* check magic number with one endianness */
-	if (magic == __perf_magic2)
-		return 0;
-
-	/* check magic number with opposite endianness */
-	if (magic != __perf_magic2_sw)
-		return -1;
+#define CHECK(__m, __v)						\
+do {								\
+	/* check magic number with one endianness */		\
+	if (magic == __m)					\
+		goto v ## __v;					\
+	/* check magic number with opposite endianness */	\
+	if (magic != __m ## _sw)				\
+		break;						\
+	ph->needs_swap = true;					\
+ v ## __v:							\
+	ph->version = __v;					\
+	return 0;						\
+} while (0)
+
+	CHECK(__perf_magic2, PERF_HEADER_VERSION_2);
 
-	ph->needs_swap = true;
-	ph->version = PERF_HEADER_VERSION_2;
-
-	return 0;
+	return -1;
 }
 
 int perf_file_header__read(struct perf_file_header *header,
-- 
1.7.11.7

[PATCH 09/23] perf tools: Introduce swap_features function

[Jiri Olsa]

Introducing swap_features function to make the swapping
code more clear and extensible.

Signed-off-by: Jiri Olsa <jolsa@redhat.com>
Cc: Corey Ashford <cjashfor@linux.vnet.ibm.com>
Cc: Frederic Weisbecker <fweisbec@gmail.com>
Cc: Ingo Molnar <mingo@elte.hu>
Cc: Namhyung Kim <namhyung@kernel.org>
Cc: Paul Mackerras <paulus@samba.org>
Cc: Peter Zijlstra <a.p.zijlstra@chello.nl>
Cc: Arnaldo Carvalho de Melo <acme@redhat.com>
Cc: Andi Kleen <ak@linux.intel.com>
Cc: David Ahern <dsahern@gmail.com>
---
 tools/perf/util/header.c | 69 ++++++++++++++++++++++++------------------------
 1 file changed, 35 insertions(+), 34 deletions(-)

diff --git a/tools/perf/util/header.c b/tools/perf/util/header.c
index 4ebe4a1..4d82dac 100644
--- a/tools/perf/util/header.c
+++ b/tools/perf/util/header.c
@@ -2512,6 +2512,39 @@ do {								\
 	return -1;
 }
 
+static void swap_features(unsigned long *adds_features)
+{
+	/*
+	 * feature bitmap is declared as an array of unsigned longs --
+	 * not good since its size can differ between the host that
+	 * generated the data file and the host analyzing the file.
+	 *
+	 * We need to handle endianness, but we don't know the size of
+	 * the unsigned long where the file was generated. Take a best
+	 * guess at determining it: try 64-bit swap first (ie., file
+	 * created on a 64-bit host), and check if the hostname feature
+	 * bit is set (this feature bit is forced on as of fbe96f2).
+	 * If the bit is not, undo the 64-bit swap and try a 32-bit
+	 * swap. If the hostname bit is still not set (e.g., older data
+	 * file), punt and fallback to the original behavior --
+	 * clearing all feature bits and setting buildid.
+	 */
+	mem_bswap_64(adds_features, BITS_TO_U64(HEADER_FEAT_BITS));
+
+	if (!test_bit(HEADER_HOSTNAME, adds_features)) {
+		/* unswap as u64 */
+		mem_bswap_64(adds_features, BITS_TO_U64(HEADER_FEAT_BITS));
+
+		/* unswap as u32 */
+		mem_bswap_32(adds_features, BITS_TO_U32(HEADER_FEAT_BITS));
+	}
+
+	if (!test_bit(HEADER_HOSTNAME, adds_features)) {
+		bitmap_zero(adds_features, HEADER_FEAT_BITS);
+		set_bit(HEADER_BUILD_ID, adds_features);
+	}
+}
+
 int perf_file_header__read(struct perf_file_header *header,
 			   struct perf_header *ph, int fd)
 {
@@ -2540,40 +2573,8 @@ int perf_file_header__read(struct perf_file_header *header,
 			bitmap_zero(header->adds_features, HEADER_FEAT_BITS);
 		else
 			return -1;
-	} else if (ph->needs_swap) {
-		/*
-		 * feature bitmap is declared as an array of unsigned longs --
-		 * not good since its size can differ between the host that
-		 * generated the data file and the host analyzing the file.
-		 *
-		 * We need to handle endianness, but we don't know the size of
-		 * the unsigned long where the file was generated. Take a best
-		 * guess at determining it: try 64-bit swap first (ie., file
-		 * created on a 64-bit host), and check if the hostname feature
-		 * bit is set (this feature bit is forced on as of fbe96f2).
-		 * If the bit is not, undo the 64-bit swap and try a 32-bit
-		 * swap. If the hostname bit is still not set (e.g., older data
-		 * file), punt and fallback to the original behavior --
-		 * clearing all feature bits and setting buildid.
-		 */
-		mem_bswap_64(&header->adds_features,
-			    BITS_TO_U64(HEADER_FEAT_BITS));
-
-		if (!test_bit(HEADER_HOSTNAME, header->adds_features)) {
-			/* unswap as u64 */
-			mem_bswap_64(&header->adds_features,
-				    BITS_TO_U64(HEADER_FEAT_BITS));
-
-			/* unswap as u32 */
-			mem_bswap_32(&header->adds_features,
-				    BITS_TO_U32(HEADER_FEAT_BITS));
-		}
-
-		if (!test_bit(HEADER_HOSTNAME, header->adds_features)) {
-			bitmap_zero(header->adds_features, HEADER_FEAT_BITS);
-			set_bit(HEADER_BUILD_ID, header->adds_features);
-		}
-	}
+	} else if (ph->needs_swap)
+		swap_features(header->adds_features);
 
 	memcpy(&ph->adds_features, &header->adds_features,
 	       sizeof(ph->adds_features));
-- 
1.7.11.7

[PATCH 10/23] perf tools: Introduce swap_header function

[Jiri Olsa]

Introducing swap_header function to make the swapping
code more clear and extensible.

Signed-off-by: Jiri Olsa <jolsa@redhat.com>
Cc: Corey Ashford <cjashfor@linux.vnet.ibm.com>
Cc: Frederic Weisbecker <fweisbec@gmail.com>
Cc: Ingo Molnar <mingo@elte.hu>
Cc: Namhyung Kim <namhyung@kernel.org>
Cc: Paul Mackerras <paulus@samba.org>
Cc: Peter Zijlstra <a.p.zijlstra@chello.nl>
Cc: Arnaldo Carvalho de Melo <acme@redhat.com>
Cc: Andi Kleen <ak@linux.intel.com>
Cc: David Ahern <dsahern@gmail.com>
---
 tools/perf/util/header.c | 31 ++++++++++++++++++-------------
 1 file changed, 18 insertions(+), 13 deletions(-)

diff --git a/tools/perf/util/header.c b/tools/perf/util/header.c
index 4d82dac..eb50de8 100644
--- a/tools/perf/util/header.c
+++ b/tools/perf/util/header.c
@@ -2545,6 +2545,22 @@ static void swap_features(unsigned long *adds_features)
 	}
 }
 
+static int swap_header(struct perf_file_header *header)
+{
+	mem_bswap_64(header, offsetof(struct perf_file_header, adds_features));
+
+	if (header->size != sizeof(*header)) {
+		/* Support the previous format */
+		if (header->size == offsetof(typeof(*header), adds_features))
+			bitmap_zero(header->adds_features, HEADER_FEAT_BITS);
+		else
+			return -1;
+	} else
+		swap_features(header->adds_features);
+
+	return 0;
+}
+
 int perf_file_header__read(struct perf_file_header *header,
 			   struct perf_header *ph, int fd)
 {
@@ -2562,19 +2578,8 @@ int perf_file_header__read(struct perf_file_header *header,
 		return -1;
 	}
 
-	if (ph->needs_swap) {
-		mem_bswap_64(header, offsetof(struct perf_file_header,
-			     adds_features));
-	}
-
-	if (header->size != sizeof(*header)) {
-		/* Support the previous format */
-		if (header->size == offsetof(typeof(*header), adds_features))
-			bitmap_zero(header->adds_features, HEADER_FEAT_BITS);
-		else
-			return -1;
-	} else if (ph->needs_swap)
-		swap_features(header->adds_features);
+	if (ph->needs_swap && swap_header(header))
+		return -1;
 
 	memcpy(&ph->adds_features, &header->adds_features,
 	       sizeof(ph->adds_features));
-- 
1.7.11.7

[PATCH 11/23] perf tools: Separate version 2 specific perf data header bits

[Jiri Olsa]

Separating version 2 specific perf data header bits,
so the code could be extented with new format version.

Signed-off-by: Jiri Olsa <jolsa@redhat.com>
Cc: Corey Ashford <cjashfor@linux.vnet.ibm.com>
Cc: Frederic Weisbecker <fweisbec@gmail.com>
Cc: Ingo Molnar <mingo@elte.hu>
Cc: Namhyung Kim <namhyung@kernel.org>
Cc: Paul Mackerras <paulus@samba.org>
Cc: Peter Zijlstra <a.p.zijlstra@chello.nl>
Cc: Arnaldo Carvalho de Melo <acme@redhat.com>
Cc: Andi Kleen <ak@linux.intel.com>
Cc: David Ahern <dsahern@gmail.com>
---
 tools/perf/util/header.c | 120 ++++++++++++++++++++++++++++++-----------------
 tools/perf/util/header.h |  16 +++++--
 2 files changed, 90 insertions(+), 46 deletions(-)

diff --git a/tools/perf/util/header.c b/tools/perf/util/header.c
index eb50de8..17084a1 100644
--- a/tools/perf/util/header.c
+++ b/tools/perf/util/header.c
@@ -2316,18 +2316,20 @@ int perf_session__write_header(struct perf_session *session,
 		.magic	   = PERF_MAGIC,
 		.size	   = sizeof(f_header),
 		.attr_size = sizeof(f_attr),
-		.attrs = {
-			.offset = attr_offset,
-			.size   = evlist->nr_entries * sizeof(f_attr),
-		},
-		.data = {
-			.offset = header->data_offset,
-			.size	= header->data_size,
-		},
+		.v2        = {
+			.attrs = {
+				.offset = attr_offset,
+				.size   = evlist->nr_entries * sizeof(f_attr),
+			},
+			.data = {
+				.offset = header->data_offset,
+				.size	= header->data_size,
+			},
 		/* event_types is ignored, store zeros */
+		},
 	};
 
-	memcpy(&f_header.adds_features, &header->adds_features, sizeof(header->adds_features));
+	memcpy(&f_header.v2.adds_features, &header->adds_features, sizeof(header->adds_features));
 
 	lseek(fd, 0, SEEK_SET);
 	err = do_write(fd, &f_header, sizeof(f_header));
@@ -2545,22 +2547,34 @@ static void swap_features(unsigned long *adds_features)
 	}
 }
 
-static int swap_header(struct perf_file_header *header)
+static int swap_header_v2(struct perf_file_header *header)
 {
-	mem_bswap_64(header, offsetof(struct perf_file_header, adds_features));
+	struct perf_file_header_v2 *v2 = &header->v2;
+
+	mem_bswap_64(v2, offsetof(struct perf_file_header_v2,
+		     adds_features));
 
 	if (header->size != sizeof(*header)) {
 		/* Support the previous format */
-		if (header->size == offsetof(typeof(*header), adds_features))
-			bitmap_zero(header->adds_features, HEADER_FEAT_BITS);
+		if (header->size == offsetof(typeof(*header), v2.adds_features))
+			bitmap_zero(v2->adds_features, HEADER_FEAT_BITS);
 		else
 			return -1;
 	} else
-		swap_features(header->adds_features);
+		swap_features(v2->adds_features);
 
 	return 0;
 }
 
+static int swap_header(struct perf_file_header *header)
+{
+	/* swap the generic part */
+	mem_bswap_64(header, offsetof(struct perf_file_header, v2));
+
+	/* version specific swap */
+	return swap_header_v2(header);
+}
+
 int perf_file_header__read(struct perf_file_header *header,
 			   struct perf_header *ph, int fd)
 {
@@ -2578,16 +2592,7 @@ int perf_file_header__read(struct perf_file_header *header,
 		return -1;
 	}
 
-	if (ph->needs_swap && swap_header(header))
-		return -1;
-
-	memcpy(&ph->adds_features, &header->adds_features,
-	       sizeof(ph->adds_features));
-
-	ph->data_offset  = header->data.offset;
-	ph->data_size	 = header->data.size;
-	ph->feat_offset  = header->data.offset + header->data.size;
-	return 0;
+	return ph->needs_swap ? swap_header(header) : 0;
 }
 
 static int perf_file_section__process(struct perf_file_section *section,
@@ -2739,12 +2744,13 @@ static int perf_evlist__prepare_tracepoint_events(struct perf_evlist *evlist,
 	return 0;
 }
 
-int perf_session__read_header(struct perf_session *session)
+static int __perf_session__read_header_v2(struct perf_session *session,
+					  struct perf_file_header *header)
 {
-	struct perf_header *header = &session->header;
-	struct perf_file_header	f_header;
-	struct perf_file_attr	f_attr;
-	u64			f_id;
+	struct perf_header		*ph = &session->header;
+	struct perf_file_header_v2	*v2 = &header->v2;
+	struct perf_file_attr		f_attr;
+	u64				f_id;
 	int nr_attrs, nr_ids, i, j;
 	int fd = session->fd;
 
@@ -2752,23 +2758,17 @@ int perf_session__read_header(struct perf_session *session)
 	if (session->evlist == NULL)
 		return -ENOMEM;
 
-	if (session->fd_pipe)
-		return perf_header__read_pipe(session);
-
-	if (perf_file_header__read(&f_header, header, fd) < 0)
-		return -EINVAL;
-
-	nr_attrs = f_header.attrs.size / f_header.attr_size;
-	lseek(fd, f_header.attrs.offset, SEEK_SET);
+	nr_attrs = v2->attrs.size / header->attr_size;
+	lseek(fd, v2->attrs.offset, SEEK_SET);
 
 	for (i = 0; i < nr_attrs; i++) {
 		struct perf_evsel *evsel;
 		off_t tmp;
 
-		if (read_attr(fd, header, &f_attr) < 0)
+		if (read_attr(fd, ph, &f_attr) < 0)
 			goto out_errno;
 
-		if (header->needs_swap)
+		if (ph->needs_swap)
 			perf_event__attr_swap(&f_attr.attr);
 
 		tmp = lseek(fd, 0, SEEK_CUR);
@@ -2777,7 +2777,7 @@ int perf_session__read_header(struct perf_session *session)
 		if (evsel == NULL)
 			goto out_delete_evlist;
 
-		evsel->needs_swap = header->needs_swap;
+		evsel->needs_swap = ph->needs_swap;
 		/*
 		 * Do it before so that if perf_evsel__alloc_id fails, this
 		 * entry gets purged too at perf_evlist__delete().
@@ -2796,7 +2796,7 @@ int perf_session__read_header(struct perf_session *session)
 		lseek(fd, f_attr.ids.offset, SEEK_SET);
 
 		for (j = 0; j < nr_ids; j++) {
-			if (perf_header__getbuffer64(header, fd, &f_id, sizeof(f_id)))
+			if (perf_header__getbuffer64(ph, fd, &f_id, sizeof(f_id)))
 				goto out_errno;
 
 			perf_evlist__id_add(session->evlist, evsel, 0, j, f_id);
@@ -2807,7 +2807,7 @@ int perf_session__read_header(struct perf_session *session)
 
 	symbol_conf.nr_events = nr_attrs;
 
-	perf_header__process_sections(header, fd, &session->pevent,
+	perf_header__process_sections(ph, fd, &session->pevent,
 				      perf_file_section__process);
 
 	if (perf_evlist__prepare_tracepoint_events(session->evlist,
@@ -2824,6 +2824,42 @@ out_delete_evlist:
 	return -ENOMEM;
 }
 
+
+static int perf_session__read_header_v2(struct perf_session *session,
+					struct perf_file_header *header)
+{
+	struct perf_header *ph = &session->header;
+	struct perf_file_header_v2 *v2 = &header->v2;
+
+	memcpy(&ph->adds_features, &v2->adds_features,
+	       sizeof(ph->adds_features));
+
+	ph->data_offset  = v2->data.offset;
+	ph->data_size	 = v2->data.size;
+	ph->feat_offset  = v2->data.offset + v2->data.size;
+
+	return __perf_session__read_header_v2(session, header);
+}
+
+static int perf_header_read_file(struct perf_session *session)
+{
+	struct perf_file_header header;
+
+	if (perf_file_header__read(&header, &session->header, session->fd))
+		return -1;
+
+	/* read v2 specific data */
+	return perf_session__read_header_v2(session, &header);
+}
+
+int perf_session__read_header(struct perf_session *session)
+{
+	if (session->fd_pipe)
+		return perf_header__read_pipe(session);
+
+	return perf_header_read_file(session) < 0 ? -EINVAL : 0;
+}
+
 int perf_event__synthesize_attr(struct perf_tool *tool,
 				struct perf_event_attr *attr, u32 ids, u64 *id,
 				perf_event__handler_t process)
diff --git a/tools/perf/util/header.h b/tools/perf/util/header.h
index 307c9ae..c6ea4603 100644
--- a/tools/perf/util/header.h
+++ b/tools/perf/util/header.h
@@ -44,10 +44,7 @@ struct perf_file_section {
 	u64 size;
 };
 
-struct perf_file_header {
-	u64				magic;
-	u64				size;
-	u64				attr_size;
+struct perf_file_header_v2 {
 	struct perf_file_section	attrs;
 	struct perf_file_section	data;
 	/* event_types is ignored */
@@ -55,6 +52,17 @@ struct perf_file_header {
 	DECLARE_BITMAP(adds_features, HEADER_FEAT_BITS);
 };
 
+struct perf_file_header {
+	u64	magic;
+	u64	size;
+	u64	attr_size;
+
+	/* version specific data */
+	union {
+		struct perf_file_header_v2 v2;
+	};
+};
+
 struct perf_pipe_file_header {
 	u64				magic;
 	u64				size;
-- 
1.7.11.7

[PATCH 12/23] perf tools: Using evlist as a holder for event_desc feature

[Jiri Olsa]

Using evlist as a holder for events stored via event_desc
feature. The event_desc events data is same as the one
stored via main header. We will use event_desc data to
replace the main header data source in next patches.

Signed-off-by: Jiri Olsa <jolsa@redhat.com>
Cc: Corey Ashford <cjashfor@linux.vnet.ibm.com>
Cc: Frederic Weisbecker <fweisbec@gmail.com>
Cc: Ingo Molnar <mingo@elte.hu>
Cc: Namhyung Kim <namhyung@kernel.org>
Cc: Paul Mackerras <paulus@samba.org>
Cc: Peter Zijlstra <a.p.zijlstra@chello.nl>
Cc: Arnaldo Carvalho de Melo <acme@redhat.com>
Cc: Andi Kleen <ak@linux.intel.com>
Cc: David Ahern <dsahern@gmail.com>
---
 tools/perf/util/header.c | 84 ++++++++++++++++++++++--------------------------
 1 file changed, 38 insertions(+), 46 deletions(-)

diff --git a/tools/perf/util/header.c b/tools/perf/util/header.c
index 17084a1..65e44cc 100644
--- a/tools/perf/util/header.c
+++ b/tools/perf/util/header.c
@@ -1205,28 +1205,11 @@ static void print_cpu_topology(struct perf_header *ph, int fd __maybe_unused,
 	}
 }
 
-static void free_event_desc(struct perf_evsel *events)
-{
-	struct perf_evsel *evsel;
-
-	if (!events)
-		return;
-
-	for (evsel = events; evsel->attr.size; evsel++) {
-		if (evsel->name)
-			free(evsel->name);
-		if (evsel->id)
-			free(evsel->id);
-	}
-
-	free(events);
-}
-
-static struct perf_evsel *
+static struct perf_evlist *
 read_event_desc(struct perf_header *ph, int fd)
 {
-	struct perf_evsel *evsel, *events = NULL;
-	u64 *id;
+	struct perf_evlist *evlist = NULL;
+	struct perf_evsel *evsel;
 	void *buf = NULL;
 	u32 nre, sz, nr, i, j;
 	ssize_t ret;
@@ -1252,18 +1235,15 @@ read_event_desc(struct perf_header *ph, int fd)
 	if (!buf)
 		goto error;
 
-	/* the last event terminates with evsel->attr.size == 0: */
-	events = calloc(nre + 1, sizeof(*events));
-	if (!events)
+	evlist = perf_evlist__new();
+	if (!evlist)
 		goto error;
 
 	msz = sizeof(evsel->attr);
 	if (sz < msz)
 		msz = sz;
 
-	for (i = 0, evsel = events; i < nre; evsel++, i++) {
-		evsel->idx = i;
-
+	for (i = 0; i < nre; i++) {
 		/*
 		 * must read entire on-file attr struct to
 		 * sync up with layout.
@@ -1272,6 +1252,10 @@ read_event_desc(struct perf_header *ph, int fd)
 		if (ret != (ssize_t)sz)
 			goto error;
 
+		evsel = perf_evsel__new(buf, i);
+		if (!evsel)
+			goto error;
+
 		if (ph->needs_swap)
 			perf_event__attr_swap(buf);
 
@@ -1286,49 +1270,55 @@ read_event_desc(struct perf_header *ph, int fd)
 			evsel->needs_swap = true;
 		}
 
+		perf_evlist__add(evlist, evsel);
+
 		evsel->name = do_read_string(fd, ph);
 
 		if (!nr)
 			continue;
 
-		id = calloc(nr, sizeof(*id));
-		if (!id)
+		if (perf_evsel__alloc_id(evsel, 1, nr))
 			goto error;
+
 		evsel->ids = nr;
-		evsel->id = id;
 
 		for (j = 0 ; j < nr; j++) {
-			ret = readn(fd, id, sizeof(*id));
-			if (ret != (ssize_t)sizeof(*id))
+			u64 id;
+
+			ret = readn(fd, &id, sizeof(id));
+			if (ret != (ssize_t)sizeof(id))
 				goto error;
 			if (ph->needs_swap)
-				*id = bswap_64(*id);
-			id++;
+				id = bswap_64(id);
+
+			perf_evlist__id_add(evlist, evsel, 0, j, id);
 		}
 	}
 out:
 	if (buf)
 		free(buf);
-	return events;
+	return evlist;
 error:
-	if (events)
-		free_event_desc(events);
-	events = NULL;
+	if (evlist)
+		perf_evlist__delete(evlist);
+	evlist = NULL;
 	goto out;
 }
 
 static void print_event_desc(struct perf_header *ph, int fd, FILE *fp)
 {
-	struct perf_evsel *evsel, *events = read_event_desc(ph, fd);
+	struct perf_evlist *evlist;
+	struct perf_evsel *evsel;
 	u32 j;
 	u64 *id;
 
-	if (!events) {
+	evlist = read_event_desc(ph, fd);
+	if (!evlist) {
 		fprintf(fp, "# event desc: not available or unable to read\n");
 		return;
 	}
 
-	for (evsel = events; evsel->attr.size; evsel++) {
+	list_for_each_entry(evsel, &evlist->entries, node) {
 		fprintf(fp, "# event : name = %s, ", evsel->name);
 
 		fprintf(fp, "type = %d, config = 0x%"PRIx64
@@ -1361,7 +1351,7 @@ static void print_event_desc(struct perf_header *ph, int fd, FILE *fp)
 		fputc('\n', fp);
 	}
 
-	free_event_desc(events);
+	perf_evlist__delete(evlist);
 }
 
 static void print_total_mem(struct perf_header *ph, int fd __maybe_unused,
@@ -1782,17 +1772,19 @@ process_event_desc(struct perf_file_section *section __maybe_unused,
 		   void *data __maybe_unused)
 {
 	struct perf_session *session;
-	struct perf_evsel *evsel, *events = read_event_desc(header, fd);
+	struct perf_evlist *evlist;
+	struct perf_evsel *evsel;
 
-	if (!events)
+	evlist = read_event_desc(header, fd);
+	if (!evlist)
 		return 0;
 
 	session = container_of(header, struct perf_session, header);
-	for (evsel = events; evsel->attr.size; evsel++)
-		perf_evlist__set_event_name(session->evlist, evsel);
 
-	free_event_desc(events);
+	list_for_each_entry(evsel, &evlist->entries, node)
+		perf_evlist__set_event_name(session->evlist, evsel);
 
+	perf_evlist__delete(evlist);
 	return 0;
 }
 
-- 
1.7.11.7

[PATCH 13/23] perf tools: Introduce perf.data version 3 format

[Jiri Olsa]

Introducing perf.data version 3 format. No functional change,
just introducing doc, magic bytes and the struct.

Signed-off-by: Jiri Olsa <jolsa@redhat.com>
Cc: Corey Ashford <cjashfor@linux.vnet.ibm.com>
Cc: Frederic Weisbecker <fweisbec@gmail.com>
Cc: Ingo Molnar <mingo@elte.hu>
Cc: Namhyung Kim <namhyung@kernel.org>
Cc: Paul Mackerras <paulus@samba.org>
Cc: Peter Zijlstra <a.p.zijlstra@chello.nl>
Cc: Arnaldo Carvalho de Melo <acme@redhat.com>
Cc: Andi Kleen <ak@linux.intel.com>
Cc: David Ahern <dsahern@gmail.com>
---
 tools/perf/Documentation/perf-data-file-v3.txt | 60 ++++++++++++++++++++++++++
 tools/perf/util/header.c                       |  8 +++-
 tools/perf/util/header.h                       |  8 ++++
 3 files changed, 75 insertions(+), 1 deletion(-)
 create mode 100644 tools/perf/Documentation/perf-data-file-v3.txt

diff --git a/tools/perf/Documentation/perf-data-file-v3.txt b/tools/perf/Documentation/perf-data-file-v3.txt
new file mode 100644
index 0000000..9073931
--- /dev/null
+++ b/tools/perf/Documentation/perf-data-file-v3.txt
@@ -0,0 +1,60 @@
+perf-data-file-v3(1)
+====================
+Following text describes version 3 of the perf data file format,
+which is version that is currently used by perf tool.
+
+In version 3 we got rid of following sections:
+  EVENT IDS
+  EVENT ATTRIBUTES
+  EVENT TYPES
+
+and keep only data and features sections. All the data
+from above listed sections are now read from specific
+FEATURES sections.
+
+benefits:
+  - siplified header storage at the time the file is closed
+  - allowing cpu hotpluged addition/removal event IDs
+    during the session due to 1)
+
+High level view of the format:
+  FILE HEADER
+  EVENT DATA
+  FEATURES
+
+
+FILE HEADER
+-----------
+Starting point of the data file with magic bytes and global
+section information.
+
+The section contains following data:
+  struct perf_file_header header
+
+struct perf_file_header::
+  struct perf_file_header {
+        u64  magic;
+        u64  size;
+        u64  attr_size;
+        struct perf_file_section  data;
+        struct perf_file_section  features;
+        DECLARE_BITMAP(adds_features, HEADER_FEAT_BITS);
+  }
+
+  magic           ID 'PERFILE3'
+  size            size minus sizeof(struct perf_file_header)
+  attr_size       attribute size minus sizeof(struct perf_file_attr)
+  data            location of 'EVENT DATA'
+  features        location of 'EVENT TYPES'
+  adds_features   'FEATURES' bitmask
+
+
+EVENT DATA
+----------
+This section contains blob of all events' data - auxiliary events
+and samples.
+
+
+FEATURES
+--------
+Same as in version 2.
diff --git a/tools/perf/util/header.c b/tools/perf/util/header.c
index 65e44cc..3f97ae1 100644
--- a/tools/perf/util/header.c
+++ b/tools/perf/util/header.c
@@ -30,6 +30,7 @@ static const char **header_argv;
 
 /*
  * magic2 = "PERFILE2"
+ * magic3 = "PERFILE3"
  * must be a numerical value to let the endianness
  * determine the memory layout. That way we are able
  * to detect endianness when reading the perf.data file
@@ -40,6 +41,8 @@ static const char **header_argv;
 static const char *__perf_magic1 = "PERFFILE";
 static const u64 __perf_magic2    = 0x32454c4946524550ULL;
 static const u64 __perf_magic2_sw = 0x50455246494c4532ULL;
+static const u64 __perf_magic3    = 0x33454c4946524550ULL;
+static const u64 __perf_magic3_sw = 0x50455246494c4533ULL;
 
 #define PERF_MAGIC	__perf_magic2
 
@@ -2460,7 +2463,9 @@ bool is_perf_magic(u64 magic)
 {
 	if (!memcmp(&magic, __perf_magic1, sizeof(magic))
 		|| magic == __perf_magic2
-		|| magic == __perf_magic2_sw)
+		|| magic == __perf_magic2_sw
+		|| magic == __perf_magic3
+		|| magic == __perf_magic3_sw)
 		return true;
 
 	return false;
@@ -2502,6 +2507,7 @@ do {								\
 } while (0)
 
 	CHECK(__perf_magic2, PERF_HEADER_VERSION_2);
+	CHECK(__perf_magic3, PERF_HEADER_VERSION_3);
 
 	return -1;
 }
diff --git a/tools/perf/util/header.h b/tools/perf/util/header.h
index c6ea4603..bcd3e64 100644
--- a/tools/perf/util/header.h
+++ b/tools/perf/util/header.h
@@ -37,6 +37,7 @@ enum {
 enum perf_header_version {
 	PERF_HEADER_VERSION_1,
 	PERF_HEADER_VERSION_2,
+	PERF_HEADER_VERSION_3,
 };
 
 struct perf_file_section {
@@ -52,6 +53,12 @@ struct perf_file_header_v2 {
 	DECLARE_BITMAP(adds_features, HEADER_FEAT_BITS);
 };
 
+struct perf_file_header_v3 {
+	struct perf_file_section	data;
+	struct perf_file_section	features;
+	DECLARE_BITMAP(adds_features, HEADER_FEAT_BITS);
+};
+
 struct perf_file_header {
 	u64	magic;
 	u64	size;
@@ -60,6 +67,7 @@ struct perf_file_header {
 	/* version specific data */
 	union {
 		struct perf_file_header_v2 v2;
+		struct perf_file_header_v3 v3;
 	};
 };
 
-- 
1.7.11.7

[PATCH 14/23] perf tools: Add perf data version 3 header swap

[Jiri Olsa]

Adding perf data version 3 header swap same way
as it's done for v2.

Signed-off-by: Jiri Olsa <jolsa@redhat.com>
Cc: Corey Ashford <cjashfor@linux.vnet.ibm.com>
Cc: Frederic Weisbecker <fweisbec@gmail.com>
Cc: Ingo Molnar <mingo@elte.hu>
Cc: Namhyung Kim <namhyung@kernel.org>
Cc: Paul Mackerras <paulus@samba.org>
Cc: Peter Zijlstra <a.p.zijlstra@chello.nl>
Cc: Arnaldo Carvalho de Melo <acme@redhat.com>
Cc: Andi Kleen <ak@linux.intel.com>
Cc: David Ahern <dsahern@gmail.com>
---
 tools/perf/util/header.c | 21 ++++++++++++++++++---
 1 file changed, 18 insertions(+), 3 deletions(-)

diff --git a/tools/perf/util/header.c b/tools/perf/util/header.c
index 3f97ae1..5df73da 100644
--- a/tools/perf/util/header.c
+++ b/tools/perf/util/header.c
@@ -2564,13 +2564,28 @@ static int swap_header_v2(struct perf_file_header *header)
 	return 0;
 }
 
-static int swap_header(struct perf_file_header *header)
+static int swap_header_v3(struct perf_file_header *header)
+{
+	struct perf_file_header_v3 *v3 = &header->v3;
+
+	mem_bswap_64(v3, offsetof(struct perf_file_header_v3,
+		     adds_features));
+
+	swap_features(v3->adds_features);
+	return 0;
+}
+
+static int swap_header(struct perf_header *ph,
+		       struct perf_file_header *header)
 {
 	/* swap the generic part */
 	mem_bswap_64(header, offsetof(struct perf_file_header, v2));
 
 	/* version specific swap */
-	return swap_header_v2(header);
+	if (ph->version <= PERF_HEADER_VERSION_2)
+		return swap_header_v2(header);
+
+	return swap_header_v3(header);
 }
 
 int perf_file_header__read(struct perf_file_header *header,
@@ -2590,7 +2605,7 @@ int perf_file_header__read(struct perf_file_header *header,
 		return -1;
 	}
 
-	return ph->needs_swap ? swap_header(header) : 0;
+	return ph->needs_swap ? swap_header(ph, header) : 0;
 }
 
 static int perf_file_section__process(struct perf_file_section *section,
-- 
1.7.11.7

[PATCH 15/23] perf tools: Add perf data version 3 header read

[Jiri Olsa]

Adding perf data version 3 header read code to load
data for v3 format.

Signed-off-by: Jiri Olsa <jolsa@redhat.com>
Cc: Corey Ashford <cjashfor@linux.vnet.ibm.com>
Cc: Frederic Weisbecker <fweisbec@gmail.com>
Cc: Ingo Molnar <mingo@elte.hu>
Cc: Namhyung Kim <namhyung@kernel.org>
Cc: Paul Mackerras <paulus@samba.org>
Cc: Peter Zijlstra <a.p.zijlstra@chello.nl>
Cc: Arnaldo Carvalho de Melo <acme@redhat.com>
Cc: Andi Kleen <ak@linux.intel.com>
Cc: David Ahern <dsahern@gmail.com>
---
 tools/perf/util/header.c | 48 +++++++++++++++++++++++++++++++++++++++++-------
 1 file changed, 41 insertions(+), 7 deletions(-)

diff --git a/tools/perf/util/header.c b/tools/perf/util/header.c
index 5df73da..50f0d9a 100644
--- a/tools/perf/util/header.c
+++ b/tools/perf/util/header.c
@@ -1777,17 +1777,28 @@ process_event_desc(struct perf_file_section *section __maybe_unused,
 	struct perf_session *session;
 	struct perf_evlist *evlist;
 	struct perf_evsel *evsel;
+	struct perf_header *ph;
+
+	session = container_of(header, struct perf_session, header);
+	if (session->evlist)
+		return 0;
 
 	evlist = read_event_desc(header, fd);
 	if (!evlist)
-		return 0;
+		return -1;
 
-	session = container_of(header, struct perf_session, header);
+	ph = &session->header;
 
-	list_for_each_entry(evsel, &evlist->entries, node)
-		perf_evlist__set_event_name(session->evlist, evsel);
+	if (ph->version <= PERF_HEADER_VERSION_2) {
+		list_for_each_entry(evsel, &evlist->entries, node)
+			perf_evlist__set_event_name(session->evlist, evsel);
+
+		perf_evlist__delete(evlist);
+	} else {
+		session->evlist = evlist;
+		symbol_conf.nr_events = evlist->nr_entries;
+	}
 
-	perf_evlist__delete(evlist);
 	return 0;
 }
 
@@ -2854,15 +2865,38 @@ static int perf_session__read_header_v2(struct perf_session *session,
 	return __perf_session__read_header_v2(session, header);
 }
 
+static int perf_session__read_header_v3(struct perf_session *session,
+					struct perf_file_header *header)
+{
+	struct perf_header *ph = &session->header;
+	struct perf_file_header_v3 *v3 = &header->v3;
+
+	memcpy(&ph->adds_features, &v3->adds_features,
+	       sizeof(ph->adds_features));
+
+	ph->data_offset  = v3->data.offset;
+	ph->data_size	 = v3->data.size;
+	ph->feat_offset  = v3->features.offset;
+
+	perf_header__process_sections(ph, session->fd, &session->pevent,
+				      perf_file_section__process);
+
+	return 0;
+}
+
 static int perf_header_read_file(struct perf_session *session)
 {
 	struct perf_file_header header;
+	struct perf_header *ph = &session->header;
 
 	if (perf_file_header__read(&header, &session->header, session->fd))
 		return -1;
 
-	/* read v2 specific data */
-	return perf_session__read_header_v2(session, &header);
+	/* read version specific data */
+	if (ph->version <= PERF_HEADER_VERSION_2)
+		return perf_session__read_header_v2(session, &header);
+
+	return perf_session__read_header_v3(session, &header);
 }
 
 int perf_session__read_header(struct perf_session *session)
-- 
1.7.11.7

[PATCH 16/23] perf tools: Add perf.data version 3 header write

[Jiri Olsa]

Adding perf data version 3 header write code and
switch perf tool storing to version 3.

Signed-off-by: Jiri Olsa <jolsa@redhat.com>
Cc: Corey Ashford <cjashfor@linux.vnet.ibm.com>
Cc: Frederic Weisbecker <fweisbec@gmail.com>
Cc: Ingo Molnar <mingo@elte.hu>
Cc: Namhyung Kim <namhyung@kernel.org>
Cc: Paul Mackerras <paulus@samba.org>
Cc: Peter Zijlstra <a.p.zijlstra@chello.nl>
Cc: Arnaldo Carvalho de Melo <acme@redhat.com>
Cc: Andi Kleen <ak@linux.intel.com>
Cc: David Ahern <dsahern@gmail.com>
---
 tools/perf/Documentation/perf-data-file-v2.txt |  5 +-
 tools/perf/builtin-inject.c                    |  2 +-
 tools/perf/builtin-record.c                    |  5 +-
 tools/perf/tests/session-simple.c              |  4 +-
 tools/perf/util/header.c                       | 70 ++++++++------------------
 tools/perf/util/header.h                       |  6 ++-
 6 files changed, 35 insertions(+), 57 deletions(-)

diff --git a/tools/perf/Documentation/perf-data-file-v2.txt b/tools/perf/Documentation/perf-data-file-v2.txt
index 3931dca..ae607ce 100644
--- a/tools/perf/Documentation/perf-data-file-v2.txt
+++ b/tools/perf/Documentation/perf-data-file-v2.txt
@@ -1,7 +1,8 @@
 perf-data-file-v2(1)
 ====================
-Following text describes version 2 of the perf data file format,
-which is version that is currently used by perf tool.
+Following text describes version 2 of the perf data file format.
+The perf tool supports this format for reading, but for writing
+it uses version 3 format perf-data-file-v3(1).
 
 The perf data file format is composed of several sections
 describing monitored events and the data itself.
diff --git a/tools/perf/builtin-inject.c b/tools/perf/builtin-inject.c
index 1d8de2e..8fcaf77 100644
--- a/tools/perf/builtin-inject.c
+++ b/tools/perf/builtin-inject.c
@@ -374,7 +374,7 @@ static int __cmd_inject(struct perf_inject *inject)
 
 	if (!inject->pipe_output) {
 		session->header.data_size = inject->bytes_written;
-		perf_session__write_header(session, session->evlist, inject->output, true);
+		perf_session__write_header(session, session->evlist, inject->output);
 	}
 
 	perf_session__delete(session);
diff --git a/tools/perf/builtin-record.c b/tools/perf/builtin-record.c
index a41ac415..fcd0baf 100644
--- a/tools/perf/builtin-record.c
+++ b/tools/perf/builtin-record.c
@@ -277,7 +277,7 @@ static void perf_record__exit(int status, void *arg)
 		if (!rec->no_buildid)
 			process_buildids(rec);
 		perf_session__write_header(rec->session, rec->evlist,
-					   rec->output, true);
+					   rec->output);
 		perf_session__delete(rec->session);
 		perf_evlist__delete(rec->evlist);
 		symbol__exit();
@@ -448,8 +448,7 @@ static int __cmd_record(struct perf_record *rec, int argc, const char **argv)
 		if (err < 0)
 			goto out_delete_session;
 	} else {
-		err = perf_session__write_header(session, evsel_list,
-						 output, false);
+		err = perf_session__prepare_header(output);
 		if (err < 0)
 			goto out_delete_session;
 	}
diff --git a/tools/perf/tests/session-simple.c b/tools/perf/tests/session-simple.c
index 3bcd7eb..3a34843 100644
--- a/tools/perf/tests/session-simple.c
+++ b/tools/perf/tests/session-simple.c
@@ -133,7 +133,7 @@ static int session_write(void)
 	perf_header__clear_feat(&session->header, HEADER_BRANCH_STACK);
 
 	TEST_ASSERT_VAL("failed to write header",
-		!perf_session__write_header(session, evlist, fd, false));
+		!perf_session__prepare_header(fd));
 
 	TEST_ASSERT_VAL("failed to store MMAP event",
 		!store_event(fd, get_event_MMAP(), &size));
@@ -144,7 +144,7 @@ static int session_write(void)
 	session->header.data_size += size;
 
 	TEST_ASSERT_VAL("failed to write header",
-		!perf_session__write_header(session, evlist, fd, true));
+		!perf_session__write_header(session, evlist, fd));
 
 	perf_session__delete(session);
 	perf_evlist__delete(evlist);
diff --git a/tools/perf/util/header.c b/tools/perf/util/header.c
index 50f0d9a..f5b7529 100644
--- a/tools/perf/util/header.c
+++ b/tools/perf/util/header.c
@@ -44,7 +44,7 @@ static const u64 __perf_magic2_sw = 0x50455246494c4532ULL;
 static const u64 __perf_magic3    = 0x33454c4946524550ULL;
 static const u64 __perf_magic3_sw = 0x50455246494c4533ULL;
 
-#define PERF_MAGIC	__perf_magic2
+#define PERF_MAGIC	__perf_magic3
 
 struct perf_file_attr {
 	struct perf_event_attr	attr;
@@ -2239,6 +2239,7 @@ static int perf_header__adds_write(struct perf_header *header,
 			perf_header__clear_feat(header, feat);
 	}
 
+	header->feat_size = lseek(fd, 0, SEEK_CUR) - sec_start;
 	lseek(fd, sec_start, SEEK_SET);
 	/*
 	 * may write more than needed due to dropped feature, but
@@ -2270,72 +2271,46 @@ int perf_header__write_pipe(int fd)
 	return 0;
 }
 
+int perf_session__prepare_header(int fd)
+{
+	off_t off = lseek(fd, PERF_FILE_HEADER__DATA_OFFSET, SEEK_SET);
+	return off == PERF_FILE_HEADER__DATA_OFFSET ? 0 : -1;
+}
+
 int perf_session__write_header(struct perf_session *session,
 			       struct perf_evlist *evlist,
-			       int fd, bool at_exit)
+			       int fd)
 {
 	struct perf_file_header f_header;
 	struct perf_file_attr   f_attr;
 	struct perf_header *header = &session->header;
-	struct perf_evsel *evsel;
-	u64 attr_offset;
 	int err;
 
-	lseek(fd, sizeof(f_header), SEEK_SET);
-
-	list_for_each_entry(evsel, &evlist->entries, node) {
-		evsel->id_offset = lseek(fd, 0, SEEK_CUR);
-		err = do_write(fd, evsel->id, evsel->ids * sizeof(u64));
-		if (err < 0) {
-			pr_debug("failed to write perf header\n");
-			return err;
-		}
-	}
-
-	attr_offset = lseek(fd, 0, SEEK_CUR);
-
-	list_for_each_entry(evsel, &evlist->entries, node) {
-		f_attr = (struct perf_file_attr){
-			.attr = evsel->attr,
-			.ids  = {
-				.offset = evsel->id_offset,
-				.size   = evsel->ids * sizeof(u64),
-			}
-		};
-		err = do_write(fd, &f_attr, sizeof(f_attr));
-		if (err < 0) {
-			pr_debug("failed to write perf header attribute\n");
-			return err;
-		}
-	}
-
-	header->data_offset = lseek(fd, 0, SEEK_CUR);
-	header->feat_offset = header->data_offset + header->data_size;
+	header->feat_offset = PERF_FILE_HEADER__DATA_OFFSET +
+			      header->data_size;
 
-	if (at_exit) {
-		err = perf_header__adds_write(header, evlist, fd);
-		if (err < 0)
-			return err;
-	}
+	err = perf_header__adds_write(header, evlist, fd);
+	if (err < 0)
+		return err;
 
 	f_header = (struct perf_file_header){
 		.magic	   = PERF_MAGIC,
 		.size	   = sizeof(f_header),
 		.attr_size = sizeof(f_attr),
-		.v2        = {
-			.attrs = {
-				.offset = attr_offset,
-				.size   = evlist->nr_entries * sizeof(f_attr),
-			},
+		.v3        = {
 			.data = {
-				.offset = header->data_offset,
+				.offset = PERF_FILE_HEADER__DATA_OFFSET,
 				.size	= header->data_size,
 			},
-		/* event_types is ignored, store zeros */
+			.features = {
+				.offset = header->feat_offset,
+				.size   = header->feat_size,
+			},
 		},
 	};
 
-	memcpy(&f_header.v2.adds_features, &header->adds_features, sizeof(header->adds_features));
+	memcpy(&f_header.v3.adds_features, &header->adds_features,
+	       sizeof(header->adds_features));
 
 	lseek(fd, 0, SEEK_SET);
 	err = do_write(fd, &f_header, sizeof(f_header));
@@ -2343,7 +2318,6 @@ int perf_session__write_header(struct perf_session *session,
 		pr_debug("failed to write perf header\n");
 		return err;
 	}
-	lseek(fd, header->data_offset + header->data_size, SEEK_SET);
 
 	return 0;
 }
diff --git a/tools/perf/util/header.h b/tools/perf/util/header.h
index bcd3e64..4982e04 100644
--- a/tools/perf/util/header.h
+++ b/tools/perf/util/header.h
@@ -53,6 +53,8 @@ struct perf_file_header_v2 {
 	DECLARE_BITMAP(adds_features, HEADER_FEAT_BITS);
 };
 
+#define PERF_FILE_HEADER__DATA_OFFSET (sizeof(struct perf_file_header))
+
 struct perf_file_header_v3 {
 	struct perf_file_section	data;
 	struct perf_file_section	features;
@@ -111,6 +113,7 @@ struct perf_header {
 	u64				data_offset;
 	u64				data_size;
 	u64				feat_offset;
+	u64				feat_size;
 	DECLARE_BITMAP(adds_features, HEADER_FEAT_BITS);
 	struct perf_session_env 	env;
 };
@@ -119,9 +122,10 @@ struct perf_evlist;
 struct perf_session;
 
 int perf_session__read_header(struct perf_session *session);
+int perf_session__prepare_header(int fd);
 int perf_session__write_header(struct perf_session *session,
 			       struct perf_evlist *evlist,
-			       int fd, bool at_exit);
+			       int fd);
 int perf_header__write_pipe(int fd);
 
 void perf_header__set_feat(struct perf_header *header, int feat);
-- 
1.7.11.7

[PATCH 17/23] perf tools: Get rid of post_processing_offset in record command

[Jiri Olsa]

No need to use post_processing_offset variable now when
we know where the data starts.

Signed-off-by: Jiri Olsa <jolsa@redhat.com>
Cc: Corey Ashford <cjashfor@linux.vnet.ibm.com>
Cc: Frederic Weisbecker <fweisbec@gmail.com>
Cc: Ingo Molnar <mingo@elte.hu>
Cc: Namhyung Kim <namhyung@kernel.org>
Cc: Paul Mackerras <paulus@samba.org>
Cc: Peter Zijlstra <a.p.zijlstra@chello.nl>
Cc: Arnaldo Carvalho de Melo <acme@redhat.com>
Cc: Andi Kleen <ak@linux.intel.com>
Cc: David Ahern <dsahern@gmail.com>
---
 tools/perf/builtin-record.c | 13 ++++++-------
 1 file changed, 6 insertions(+), 7 deletions(-)

diff --git a/tools/perf/builtin-record.c b/tools/perf/builtin-record.c
index fcd0baf..b67564c 100644
--- a/tools/perf/builtin-record.c
+++ b/tools/perf/builtin-record.c
@@ -75,7 +75,6 @@ struct perf_record {
 	bool			no_buildid;
 	bool			no_buildid_cache;
 	long			samples;
-	off_t			post_processing_offset;
 };
 
 static void advance_output(struct perf_record *rec, size_t size)
@@ -253,15 +252,17 @@ out:
 
 static int process_buildids(struct perf_record *rec)
 {
-	u64 size = lseek(rec->output, 0, SEEK_CUR);
+	struct perf_session *session = rec->session;
+	u64 data_offset              = PERF_FILE_HEADER__DATA_OFFSET;
+	u64 size                     = session->header.data_size;
 
 	if (size == 0)
 		return 0;
 
 	rec->session->fd = rec->output;
-	return __perf_session__process_events(rec->session, rec->post_processing_offset,
-					      size - rec->post_processing_offset,
-					      size, &build_id__mark_dso_hit_ops);
+	return __perf_session__process_events(session, data_offset,
+					      size - data_offset, size,
+					      &build_id__mark_dso_hit_ops);
 }
 
 static void perf_record__exit(int status, void *arg)
@@ -461,8 +462,6 @@ static int __cmd_record(struct perf_record *rec, int argc, const char **argv)
 		goto out_delete_session;
 	}
 
-	rec->post_processing_offset = lseek(output, 0, SEEK_CUR);
-
 	machine = &session->machines.host;
 
 	if (opts->pipe_output) {
-- 
1.7.11.7

[PATCH 18/23] perf tools: Move synthetizing into single function

[Jiri Olsa]

Moving synthetizing into single function, so it
could be reused.

Signed-off-by: Jiri Olsa <jolsa@redhat.com>
Cc: Corey Ashford <cjashfor@linux.vnet.ibm.com>
Cc: Frederic Weisbecker <fweisbec@gmail.com>
Cc: Ingo Molnar <mingo@elte.hu>
Cc: Namhyung Kim <namhyung@kernel.org>
Cc: Paul Mackerras <paulus@samba.org>
Cc: Peter Zijlstra <a.p.zijlstra@chello.nl>
Cc: Arnaldo Carvalho de Melo <acme@redhat.com>
Cc: Andi Kleen <ak@linux.intel.com>
Cc: David Ahern <dsahern@gmail.com>
---
 tools/perf/builtin-record.c | 155 ++++++++++++++++++++++++++------------------
 1 file changed, 91 insertions(+), 64 deletions(-)

diff --git a/tools/perf/builtin-record.c b/tools/perf/builtin-record.c
index b67564c..33a5bce 100644
--- a/tools/perf/builtin-record.c
+++ b/tools/perf/builtin-record.c
@@ -317,6 +317,95 @@ static void perf_event__synthesize_guest_os(struct machine *machine, void *data)
 		       " relocation symbol.\n", machine->pid);
 }
 
+static int synthesize_record_pipe(struct perf_record *rec)
+{
+	struct perf_session  *session = rec->session;
+	struct perf_tool        *tool = &rec->tool;
+	struct perf_evlist    *evlist = rec->evlist;
+	int err;
+
+	err = perf_event__synthesize_attrs(tool, session,
+					   process_synthesized_event);
+	if (err < 0) {
+		pr_err("Couldn't synthesize attrs.\n");
+		return err;
+	}
+
+	if (have_tracepoints(&evlist->entries)) {
+		/*
+		 * FIXME err <= 0 here actually means that
+		 * there were no tracepoints so its not really
+		 * an error, just that we don't need to
+		 * synthesize anything.  We really have to
+		 * return this more properly and also
+		 * propagate errors that now are calling die()
+		 */
+		err = perf_event__synthesize_tracing_data(tool, rec->output, evlist,
+							  process_synthesized_event);
+		if (err <= 0) {
+			pr_err("Couldn't record tracing data.\n");
+			return err;
+		}
+		advance_output(rec, err);
+	}
+
+	return 0;
+}
+
+static int synthesize_record_file(struct perf_record *rec)
+{
+	struct perf_session  *session = rec->session;
+	struct perf_tool        *tool = &rec->tool;
+	struct perf_record_opts *opts = &rec->opts;
+	struct machine       *machine = &session->machines.host;
+	struct perf_evlist    *evlist = rec->evlist;
+	int err;
+
+	err = perf_event__synthesize_kernel_mmap(tool, process_synthesized_event,
+						 machine, "_text");
+	if (err < 0)
+		err = perf_event__synthesize_kernel_mmap(tool, process_synthesized_event,
+							 machine, "_stext");
+	if (err < 0)
+		pr_err("Couldn't record kernel reference relocation symbol\n"
+		       "Symbol resolution may be skewed if relocation was used (e.g. kexec).\n"
+		       "Check /proc/kallsyms permission or run as root.\n");
+
+	err = perf_event__synthesize_modules(tool, process_synthesized_event,
+					     machine);
+	if (err < 0)
+		pr_err("Couldn't record kernel module information.\n"
+		       "Symbol resolution may be skewed if relocation was used (e.g. kexec).\n"
+		       "Check /proc/modules permission or run as root.\n");
+
+	if (perf_guest)
+		machines__process_guests(&session->machines,
+					 perf_event__synthesize_guest_os, tool);
+
+	if (perf_target__has_task(&opts->target))
+		err = perf_event__synthesize_thread_map(tool, evlist->threads,
+						  process_synthesized_event,
+						  machine);
+	else if (perf_target__has_cpu(&opts->target))
+		err = perf_event__synthesize_threads(tool, process_synthesized_event,
+					       machine);
+	else /* command specified */
+		err = 0;
+
+	return err;
+}
+
+static int synthesize_record(struct perf_record *rec)
+{
+	struct perf_record_opts *opts = &rec->opts;
+	int err = 0;
+
+	if (opts->pipe_output)
+		err = synthesize_record_pipe(rec);
+
+	return err ? err : synthesize_record_file(rec);
+}
+
 static struct perf_event_header finished_round_event = {
 	.size = sizeof(struct perf_event_header),
 	.type = PERF_RECORD_FINISHED_ROUND,
@@ -351,8 +440,6 @@ static int __cmd_record(struct perf_record *rec, int argc, const char **argv)
 	int err, output, feat;
 	unsigned long waking = 0;
 	const bool forks = argc > 0;
-	struct machine *machine;
-	struct perf_tool *tool = &rec->tool;
 	struct perf_record_opts *opts = &rec->opts;
 	struct perf_evlist *evsel_list = rec->evlist;
 	const char *output_name = rec->output_name;
@@ -462,68 +549,8 @@ static int __cmd_record(struct perf_record *rec, int argc, const char **argv)
 		goto out_delete_session;
 	}
 
-	machine = &session->machines.host;
-
-	if (opts->pipe_output) {
-		err = perf_event__synthesize_attrs(tool, session,
-						   process_synthesized_event);
-		if (err < 0) {
-			pr_err("Couldn't synthesize attrs.\n");
-			goto out_delete_session;
-		}
-
-		if (have_tracepoints(&evsel_list->entries)) {
-			/*
-			 * FIXME err <= 0 here actually means that
-			 * there were no tracepoints so its not really
-			 * an error, just that we don't need to
-			 * synthesize anything.  We really have to
-			 * return this more properly and also
-			 * propagate errors that now are calling die()
-			 */
-			err = perf_event__synthesize_tracing_data(tool, output, evsel_list,
-								  process_synthesized_event);
-			if (err <= 0) {
-				pr_err("Couldn't record tracing data.\n");
-				goto out_delete_session;
-			}
-			advance_output(rec, err);
-		}
-	}
-
-	err = perf_event__synthesize_kernel_mmap(tool, process_synthesized_event,
-						 machine, "_text");
-	if (err < 0)
-		err = perf_event__synthesize_kernel_mmap(tool, process_synthesized_event,
-							 machine, "_stext");
-	if (err < 0)
-		pr_err("Couldn't record kernel reference relocation symbol\n"
-		       "Symbol resolution may be skewed if relocation was used (e.g. kexec).\n"
-		       "Check /proc/kallsyms permission or run as root.\n");
-
-	err = perf_event__synthesize_modules(tool, process_synthesized_event,
-					     machine);
-	if (err < 0)
-		pr_err("Couldn't record kernel module information.\n"
-		       "Symbol resolution may be skewed if relocation was used (e.g. kexec).\n"
-		       "Check /proc/modules permission or run as root.\n");
-
-	if (perf_guest) {
-		machines__process_guests(&session->machines,
-					 perf_event__synthesize_guest_os, tool);
-	}
-
-	if (perf_target__has_task(&opts->target))
-		err = perf_event__synthesize_thread_map(tool, evsel_list->threads,
-						  process_synthesized_event,
-						  machine);
-	else if (perf_target__has_cpu(&opts->target))
-		err = perf_event__synthesize_threads(tool, process_synthesized_event,
-					       machine);
-	else /* command specified */
-		err = 0;
-
-	if (err != 0)
+	err = synthesize_record(rec);
+	if (err)
 		goto out_delete_session;
 
 	if (rec->realtime_prio) {
-- 
1.7.11.7

[PATCH 19/23] perf tools: Add data object to handle perf data file

[Jiri Olsa]

Adding data object to define 'struct perf_data_file', which
holds all attributes wrt to perf.data file.

This patch uses 'struct perf_data_file' as a placeholder for
all attributes regarding perf.data file handling. Changing
perf_session__new to take it as an argument.

The rest of the functionality will be added later to keep this
change simple enough, because all the places using perf_session
are changed now.

Signed-off-by: Jiri Olsa <jolsa@redhat.com>
Cc: Corey Ashford <cjashfor@linux.vnet.ibm.com>
Cc: Frederic Weisbecker <fweisbec@gmail.com>
Cc: Ingo Molnar <mingo@elte.hu>
Cc: Namhyung Kim <namhyung@kernel.org>
Cc: Paul Mackerras <paulus@samba.org>
Cc: Peter Zijlstra <a.p.zijlstra@chello.nl>
Cc: Arnaldo Carvalho de Melo <acme@redhat.com>
Cc: Andi Kleen <ak@linux.intel.com>
Cc: David Ahern <dsahern@gmail.com>
---
 tools/perf/builtin-annotate.c      |  9 ++++--
 tools/perf/builtin-buildid-cache.c |  8 ++++--
 tools/perf/builtin-buildid-list.c  |  9 ++++--
 tools/perf/builtin-diff.c          | 19 +++++++-----
 tools/perf/builtin-evlist.c        |  7 ++++-
 tools/perf/builtin-inject.c        |  7 ++++-
 tools/perf/builtin-kmem.c          |  7 ++++-
 tools/perf/builtin-kvm.c           |  8 ++++--
 tools/perf/builtin-lock.c          |  8 +++++-
 tools/perf/builtin-mem.c           |  9 ++++--
 tools/perf/builtin-record.c        | 59 +++++++++++++++++++++-----------------
 tools/perf/builtin-report.c        | 10 +++++--
 tools/perf/builtin-sched.c         |  6 +++-
 tools/perf/builtin-script.c        | 14 +++++++--
 tools/perf/builtin-timechart.c     | 10 +++++--
 tools/perf/builtin-top.c           |  6 +++-
 tools/perf/perf.h                  |  1 -
 tools/perf/tests/session-simple.c  | 36 ++++++++++++-----------
 tools/perf/util/data.h             | 29 +++++++++++++++++++
 tools/perf/util/session.c          | 12 ++++----
 tools/perf/util/session.h          |  6 ++--
 21 files changed, 197 insertions(+), 83 deletions(-)
 create mode 100644 tools/perf/util/data.h

diff --git a/tools/perf/builtin-annotate.c b/tools/perf/builtin-annotate.c
index db491e9..048576f 100644
--- a/tools/perf/builtin-annotate.c
+++ b/tools/perf/builtin-annotate.c
@@ -28,6 +28,7 @@
 #include "util/hist.h"
 #include "util/session.h"
 #include "util/tool.h"
+#include "util/data.h"
 #include "arch/common.h"
 
 #include <linux/bitmap.h>
@@ -189,9 +190,13 @@ static int __cmd_annotate(struct perf_annotate *ann)
 	struct perf_session *session;
 	struct perf_evsel *pos;
 	u64 total_nr_samples;
+	struct perf_data_file file = {
+		.path  = input_name,
+		.mode  = PERF_DATA_MODE_READ,
+		.force = ann->force,
+	};
 
-	session = perf_session__new(input_name, O_RDONLY,
-				    ann->force, false, &ann->tool);
+	session = perf_session__new(&file, false, &ann->tool);
 	if (session == NULL)
 		return -ENOMEM;
 
diff --git a/tools/perf/builtin-buildid-cache.c b/tools/perf/builtin-buildid-cache.c
index c96c8fa..581c62a 100644
--- a/tools/perf/builtin-buildid-cache.c
+++ b/tools/perf/builtin-buildid-cache.c
@@ -82,8 +82,12 @@ static bool dso__missing_buildid_cache(struct dso *dso, int parm __maybe_unused)
 
 static int build_id_cache__fprintf_missing(const char *filename, bool force, FILE *fp)
 {
-	struct perf_session *session = perf_session__new(filename, O_RDONLY,
-							 force, false, NULL);
+	struct perf_data_file file = {
+		.path  = filename,
+		.mode  = PERF_DATA_MODE_READ,
+		.force = force,
+	};
+	struct perf_session *session = perf_session__new(&file, false, NULL);
 	if (session == NULL)
 		return -1;
 
diff --git a/tools/perf/builtin-buildid-list.c b/tools/perf/builtin-buildid-list.c
index e74366a..0164c1c 100644
--- a/tools/perf/builtin-buildid-list.c
+++ b/tools/perf/builtin-buildid-list.c
@@ -15,6 +15,7 @@
 #include "util/parse-options.h"
 #include "util/session.h"
 #include "util/symbol.h"
+#include "util/data.h"
 
 static int sysfs__fprintf_build_id(FILE *fp)
 {
@@ -52,6 +53,11 @@ static bool dso__skip_buildid(struct dso *dso, int with_hits)
 static int perf_session__list_build_ids(bool force, bool with_hits)
 {
 	struct perf_session *session;
+	struct perf_data_file file = {
+		.path  = input_name,
+		.mode  = PERF_DATA_MODE_READ,
+		.force = force,
+	};
 
 	symbol__elf_init();
 	/*
@@ -60,8 +66,7 @@ static int perf_session__list_build_ids(bool force, bool with_hits)
 	if (filename__fprintf_build_id(input_name, stdout))
 		goto out;
 
-	session = perf_session__new(input_name, O_RDONLY, force, false,
-				    &build_id__mark_dso_hit_ops);
+	session = perf_session__new(&file, false, &build_id__mark_dso_hit_ops);
 	if (session == NULL)
 		return -1;
 	/*
diff --git a/tools/perf/builtin-diff.c b/tools/perf/builtin-diff.c
index 93de3ac..ee7430b 100644
--- a/tools/perf/builtin-diff.c
+++ b/tools/perf/builtin-diff.c
@@ -16,6 +16,7 @@
 #include "util/sort.h"
 #include "util/symbol.h"
 #include "util/util.h"
+#include "util/data.h"
 
 #include <stdlib.h>
 #include <math.h>
@@ -42,7 +43,7 @@ struct diff_hpp_fmt {
 
 struct data__file {
 	struct perf_session	*session;
-	const char		*file;
+	struct perf_data_file	file;
 	int			 idx;
 	struct hists		*hists;
 	struct diff_hpp_fmt	 fmt[PERF_HPP_DIFF__MAX_INDEX];
@@ -599,7 +600,7 @@ static void data__fprintf(void)
 
 	data__for_each_file(i, d)
 		fprintf(stdout, "#  [%d] %s %s\n",
-			d->idx, d->file,
+			d->idx, d->file.path,
 			!d->idx ? "(Baseline)" : "");
 
 	fprintf(stdout, "#\n");
@@ -661,17 +662,16 @@ static int __cmd_diff(void)
 	int ret = -EINVAL, i;
 
 	data__for_each_file(i, d) {
-		d->session = perf_session__new(d->file, O_RDONLY, force,
-					       false, &tool);
+		d->session = perf_session__new(&d->file, false, &tool);
 		if (!d->session) {
-			pr_err("Failed to open %s\n", d->file);
+			pr_err("Failed to open %s\n", d->file.path);
 			ret = -ENOMEM;
 			goto out_delete;
 		}
 
 		ret = perf_session__process_events(d->session, &tool);
 		if (ret) {
-			pr_err("Failed to process %s\n", d->file);
+			pr_err("Failed to process %s\n", d->file.path);
 			goto out_delete;
 		}
 
@@ -1014,7 +1014,12 @@ static int data_init(int argc, const char **argv)
 		return -ENOMEM;
 
 	data__for_each_file(i, d) {
-		d->file = use_default ? defaults[i] : argv[i];
+		struct perf_data_file *file = &d->file;
+
+		file->path  = use_default ? defaults[i] : argv[i];
+		file->mode  = PERF_DATA_MODE_READ,
+		file->force = force,
+
 		d->idx  = i;
 	}
 
diff --git a/tools/perf/builtin-evlist.c b/tools/perf/builtin-evlist.c
index 05bd9df..20b0f12 100644
--- a/tools/perf/builtin-evlist.c
+++ b/tools/perf/builtin-evlist.c
@@ -14,13 +14,18 @@
 #include "util/parse-events.h"
 #include "util/parse-options.h"
 #include "util/session.h"
+#include "util/data.h"
 
 static int __cmd_evlist(const char *file_name, struct perf_attr_details *details)
 {
 	struct perf_session *session;
 	struct perf_evsel *pos;
+	struct perf_data_file file = {
+		.path = file_name,
+		.mode = PERF_DATA_MODE_READ,
+	};
 
-	session = perf_session__new(file_name, O_RDONLY, 0, false, NULL);
+	session = perf_session__new(&file, 0, NULL);
 	if (session == NULL)
 		return -ENOMEM;
 
diff --git a/tools/perf/builtin-inject.c b/tools/perf/builtin-inject.c
index 8fcaf77..6305bdf 100644
--- a/tools/perf/builtin-inject.c
+++ b/tools/perf/builtin-inject.c
@@ -15,6 +15,7 @@
 #include "util/tool.h"
 #include "util/debug.h"
 #include "util/build-id.h"
+#include "util/data.h"
 
 #include "util/parse-options.h"
 
@@ -332,6 +333,10 @@ static int __cmd_inject(struct perf_inject *inject)
 {
 	struct perf_session *session;
 	int ret = -EINVAL;
+	struct perf_data_file file = {
+		.path = inject->input_name,
+		.mode = PERF_DATA_MODE_READ,
+	};
 
 	signal(SIGINT, sig_handler);
 
@@ -341,7 +346,7 @@ static int __cmd_inject(struct perf_inject *inject)
 		inject->tool.tracing_data = perf_event__repipe_tracing_data;
 	}
 
-	session = perf_session__new(inject->input_name, O_RDONLY, false, true, &inject->tool);
+	session = perf_session__new(&file, true, &inject->tool);
 	if (session == NULL)
 		return -ENOMEM;
 
diff --git a/tools/perf/builtin-kmem.c b/tools/perf/builtin-kmem.c
index b49f5c5..08ea212 100644
--- a/tools/perf/builtin-kmem.c
+++ b/tools/perf/builtin-kmem.c
@@ -13,6 +13,7 @@
 
 #include "util/parse-options.h"
 #include "util/trace-event.h"
+#include "util/data.h"
 
 #include "util/debug.h"
 
@@ -485,8 +486,12 @@ static int __cmd_kmem(void)
 		{ "kmem:kfree",			perf_evsel__process_free_event, },
     		{ "kmem:kmem_cache_free",	perf_evsel__process_free_event, },
 	};
+	struct perf_data_file file = {
+		.path = input_name,
+		.mode = PERF_DATA_MODE_READ,
+	};
 
-	session = perf_session__new(input_name, O_RDONLY, 0, false, &perf_kmem);
+	session = perf_session__new(&file, false, &perf_kmem);
 	if (session == NULL)
 		return -ENOMEM;
 
diff --git a/tools/perf/builtin-kvm.c b/tools/perf/builtin-kvm.c
index 24b78ae..88ddb03 100644
--- a/tools/perf/builtin-kvm.c
+++ b/tools/perf/builtin-kvm.c
@@ -15,6 +15,7 @@
 #include <lk/debugfs.h>
 #include "util/tool.h"
 #include "util/stat.h"
+#include "util/data.h"
 
 #include <sys/prctl.h>
 
@@ -733,10 +734,13 @@ static int read_events(struct perf_kvm_stat *kvm)
 		.comm			= perf_event__process_comm,
 		.ordered_samples	= true,
 	};
+	struct perf_data_file file = {
+		.path = input_name,
+		.mode = PERF_DATA_MODE_READ,
+	};
 
 	kvm->tool = eops;
-	kvm->session = perf_session__new(kvm->file_name, O_RDONLY, 0, false,
-					 &kvm->tool);
+	kvm->session = perf_session__new(&file, false, &kvm->tool);
 	if (!kvm->session) {
 		pr_err("Initializing perf session failed\n");
 		return -EINVAL;
diff --git a/tools/perf/builtin-lock.c b/tools/perf/builtin-lock.c
index 76543a4..65d6bec 100644
--- a/tools/perf/builtin-lock.c
+++ b/tools/perf/builtin-lock.c
@@ -15,6 +15,7 @@
 #include "util/debug.h"
 #include "util/session.h"
 #include "util/tool.h"
+#include "util/data.h"
 
 #include <sys/types.h>
 #include <sys/prctl.h>
@@ -835,7 +836,12 @@ static int read_events(void)
 		.comm		 = perf_event__process_comm,
 		.ordered_samples = true,
 	};
-	session = perf_session__new(input_name, O_RDONLY, 0, false, &eops);
+	struct perf_data_file file = {
+		.path = input_name,
+		.mode = PERF_DATA_MODE_READ,
+	};
+
+	session = perf_session__new(&file, false, &eops);
 	if (!session) {
 		pr_err("Initializing perf session failed\n");
 		return -1;
diff --git a/tools/perf/builtin-mem.c b/tools/perf/builtin-mem.c
index a8ff6d2..a7e8312 100644
--- a/tools/perf/builtin-mem.c
+++ b/tools/perf/builtin-mem.c
@@ -5,6 +5,7 @@
 #include "util/trace-event.h"
 #include "util/tool.h"
 #include "util/session.h"
+#include "util/data.h"
 
 #define MEM_OPERATION_LOAD	"load"
 #define MEM_OPERATION_STORE	"store"
@@ -121,10 +122,14 @@ static int process_sample_event(struct perf_tool *tool,
 
 static int report_raw_events(struct perf_mem *mem)
 {
+	struct perf_data_file file = {
+		.path = input_name,
+		.mode = PERF_DATA_MODE_READ,
+	};
 	int err = -EINVAL;
 	int ret;
-	struct perf_session *session = perf_session__new(input_name, O_RDONLY,
-							 0, false, &mem->tool);
+	struct perf_session *session = perf_session__new(&file, false,
+							 &mem->tool);
 
 	if (session == NULL)
 		return -ENOMEM;
diff --git a/tools/perf/builtin-record.c b/tools/perf/builtin-record.c
index 33a5bce..c4ec7eb 100644
--- a/tools/perf/builtin-record.c
+++ b/tools/perf/builtin-record.c
@@ -24,6 +24,7 @@
 #include "util/symbol.h"
 #include "util/cpumap.h"
 #include "util/thread_map.h"
+#include "util/data.h"
 
 #include <unistd.h>
 #include <sched.h>
@@ -65,11 +66,11 @@ struct perf_record {
 	struct perf_tool	tool;
 	struct perf_record_opts	opts;
 	u64			bytes_written;
-	const char		*output_name;
+	struct perf_data_file	file_base;
+	struct perf_data_file	*file;
 	struct perf_evlist	*evlist;
 	struct perf_session	*session;
 	const char		*progname;
-	int			output;
 	unsigned int		page_size;
 	int			realtime_prio;
 	bool			no_buildid;
@@ -84,8 +85,10 @@ static void advance_output(struct perf_record *rec, size_t size)
 
 static int write_output(struct perf_record *rec, void *buf, size_t size)
 {
+	struct perf_data_file *file = rec->file;
+
 	while (size) {
-		int ret = write(rec->output, buf, size);
+		int ret = write(file->fd, buf, size);
 
 		if (ret < 0) {
 			pr_err("failed to write\n");
@@ -252,6 +255,7 @@ out:
 
 static int process_buildids(struct perf_record *rec)
 {
+	struct perf_data_file *file  = rec->file;
 	struct perf_session *session = rec->session;
 	u64 data_offset              = PERF_FILE_HEADER__DATA_OFFSET;
 	u64 size                     = session->header.data_size;
@@ -259,7 +263,7 @@ static int process_buildids(struct perf_record *rec)
 	if (size == 0)
 		return 0;
 
-	rec->session->fd = rec->output;
+	rec->session->fd = file->fd;
 	return __perf_session__process_events(session, data_offset,
 					      size - data_offset, size,
 					      &build_id__mark_dso_hit_ops);
@@ -268,17 +272,18 @@ static int process_buildids(struct perf_record *rec)
 static void perf_record__exit(int status, void *arg)
 {
 	struct perf_record *rec = arg;
+	struct perf_data_file *file = rec->file;
 
 	if (status != 0)
 		return;
 
-	if (!rec->opts.pipe_output) {
+	if (!file->is_pipe) {
 		rec->session->header.data_size += rec->bytes_written;
 
 		if (!rec->no_buildid)
 			process_buildids(rec);
 		perf_session__write_header(rec->session, rec->evlist,
-					   rec->output);
+					   file->fd);
 		perf_session__delete(rec->session);
 		perf_evlist__delete(rec->evlist);
 		symbol__exit();
@@ -322,6 +327,7 @@ static int synthesize_record_pipe(struct perf_record *rec)
 	struct perf_session  *session = rec->session;
 	struct perf_tool        *tool = &rec->tool;
 	struct perf_evlist    *evlist = rec->evlist;
+	struct perf_data_file   *file = rec->file;
 	int err;
 
 	err = perf_event__synthesize_attrs(tool, session,
@@ -340,7 +346,7 @@ static int synthesize_record_pipe(struct perf_record *rec)
 		 * return this more properly and also
 		 * propagate errors that now are calling die()
 		 */
-		err = perf_event__synthesize_tracing_data(tool, rec->output, evlist,
+		err = perf_event__synthesize_tracing_data(tool, file->fd, evlist,
 							  process_synthesized_event);
 		if (err <= 0) {
 			pr_err("Couldn't record tracing data.\n");
@@ -397,10 +403,10 @@ static int synthesize_record_file(struct perf_record *rec)
 
 static int synthesize_record(struct perf_record *rec)
 {
-	struct perf_record_opts *opts = &rec->opts;
+	struct perf_data_file *file = rec->file;
 	int err = 0;
 
-	if (opts->pipe_output)
+	if (file->is_pipe)
 		err = synthesize_record_pipe(rec);
 
 	return err ? err : synthesize_record_file(rec);
@@ -437,12 +443,13 @@ static int __cmd_record(struct perf_record *rec, int argc, const char **argv)
 {
 	struct stat st;
 	int flags;
-	int err, output, feat;
+	int err, feat;
 	unsigned long waking = 0;
 	const bool forks = argc > 0;
 	struct perf_record_opts *opts = &rec->opts;
 	struct perf_evlist *evsel_list = rec->evlist;
-	const char *output_name = rec->output_name;
+	struct perf_data_file *file = rec->file;
+	const char *output_name = file->path;
 	struct perf_session *session;
 	bool disabled = false;
 
@@ -458,13 +465,13 @@ static int __cmd_record(struct perf_record *rec, int argc, const char **argv)
 
 	if (!output_name) {
 		if (!fstat(STDOUT_FILENO, &st) && S_ISFIFO(st.st_mode))
-			opts->pipe_output = true;
+			file->is_pipe = true;
 		else
-			rec->output_name = output_name = "perf.data";
+			file->path = output_name = "perf.data";
 	}
 	if (output_name) {
 		if (!strcmp(output_name, "-"))
-			opts->pipe_output = true;
+			file->is_pipe = true;
 		else if (!stat(output_name, &st) && st.st_size) {
 			char oldname[PATH_MAX];
 			snprintf(oldname, sizeof(oldname), "%s.old",
@@ -476,19 +483,16 @@ static int __cmd_record(struct perf_record *rec, int argc, const char **argv)
 
 	flags = O_CREAT|O_RDWR|O_TRUNC;
 
-	if (opts->pipe_output)
-		output = STDOUT_FILENO;
+	if (file->is_pipe)
+		file->fd = STDOUT_FILENO;
 	else
-		output = open(output_name, flags, S_IRUSR | S_IWUSR);
-	if (output < 0) {
+		file->fd = open(output_name, flags, S_IRUSR | S_IWUSR);
+	if (file->fd < 0) {
 		perror("failed to create output file");
 		return -1;
 	}
 
-	rec->output = output;
-
-	session = perf_session__new(output_name, O_WRONLY,
-				    true, false, NULL);
+	session = perf_session__new(file, false, NULL);
 	if (session == NULL) {
 		pr_err("Not enough memory for reading perf file header\n");
 		return -1;
@@ -510,7 +514,7 @@ static int __cmd_record(struct perf_record *rec, int argc, const char **argv)
 
 	if (forks) {
 		err = perf_evlist__prepare_workload(evsel_list, &opts->target,
-						    argv, opts->pipe_output,
+						    argv, file->is_pipe,
 						    true);
 		if (err < 0) {
 			pr_err("Couldn't run the workload!\n");
@@ -531,12 +535,12 @@ static int __cmd_record(struct perf_record *rec, int argc, const char **argv)
 	 */
 	on_exit(perf_record__exit, rec);
 
-	if (opts->pipe_output) {
-		err = perf_header__write_pipe(output);
+	if (file->is_pipe) {
+		err = perf_header__write_pipe(file->fd);
 		if (err < 0)
 			goto out_delete_session;
 	} else {
-		err = perf_session__prepare_header(output);
+		err = perf_session__prepare_header(file->fd);
 		if (err < 0)
 			goto out_delete_session;
 	}
@@ -836,6 +840,7 @@ static struct perf_record record = {
 			.uses_mmap   = true,
 		},
 	},
+	.file = &record.file_base,
 };
 
 #define CALLCHAIN_HELP "do call-graph (stack chain/backtrace) recording: "
@@ -874,7 +879,7 @@ const struct option record_options[] = {
 	OPT_STRING('C', "cpu", &record.opts.target.cpu_list, "cpu",
 		    "list of cpus to monitor"),
 	OPT_U64('c', "count", &record.opts.user_interval, "event period to sample"),
-	OPT_STRING('o', "output", &record.output_name, "file",
+	OPT_STRING('o', "output", &record.file_base.path, "file",
 		    "output file name"),
 	OPT_BOOLEAN('i', "no-inherit", &record.opts.no_inherit,
 		    "child tasks do not inherit counters"),
diff --git a/tools/perf/builtin-report.c b/tools/perf/builtin-report.c
index a34c587..7904173 100644
--- a/tools/perf/builtin-report.c
+++ b/tools/perf/builtin-report.c
@@ -33,6 +33,7 @@
 #include "util/thread.h"
 #include "util/sort.h"
 #include "util/hist.h"
+#include "util/data.h"
 #include "arch/common.h"
 
 #include <linux/bitmap.h>
@@ -834,6 +835,9 @@ int cmd_report(int argc, const char **argv, const char *prefix __maybe_unused)
 		     "Don't show entries under that percent", parse_percent_limit),
 	OPT_END()
 	};
+	struct perf_data_file file = {
+		.mode  = PERF_DATA_MODE_READ,
+	};
 
 	perf_config(perf_report_config, &report);
 
@@ -863,9 +867,11 @@ int cmd_report(int argc, const char **argv, const char *prefix __maybe_unused)
 		perf_hpp__init();
 	}
 
+	file.path  = input_name;
+	file.force = report.force;
+
 repeat:
-	session = perf_session__new(input_name, O_RDONLY,
-				    report.force, false, &report.tool);
+	session = perf_session__new(&file, false, &report.tool);
 	if (session == NULL)
 		return -ENOMEM;
 
diff --git a/tools/perf/builtin-sched.c b/tools/perf/builtin-sched.c
index 948183a..1e50899 100644
--- a/tools/perf/builtin-sched.c
+++ b/tools/perf/builtin-sched.c
@@ -1458,8 +1458,12 @@ static int perf_sched__read_events(struct perf_sched *sched, bool destroy,
 		{ "sched:sched_migrate_task", process_sched_migrate_task_event, },
 	};
 	struct perf_session *session;
+	struct perf_data_file file = {
+		.path = input_name,
+		.mode = PERF_DATA_MODE_READ,
+	};
 
-	session = perf_session__new(input_name, O_RDONLY, 0, false, &sched->tool);
+	session = perf_session__new(&file, false, &sched->tool);
 	if (session == NULL) {
 		pr_debug("No Memory for session\n");
 		return -1;
diff --git a/tools/perf/builtin-script.c b/tools/perf/builtin-script.c
index ecb6979..bbd524c 100644
--- a/tools/perf/builtin-script.c
+++ b/tools/perf/builtin-script.c
@@ -15,6 +15,7 @@
 #include "util/evlist.h"
 #include "util/evsel.h"
 #include "util/sort.h"
+#include "util/data.h"
 #include <linux/bitmap.h>
 
 static char const		*script_name;
@@ -1092,10 +1093,14 @@ int find_scripts(char **scripts_array, char **scripts_path_array)
 	char scripts_path[MAXPATHLEN], lang_path[MAXPATHLEN];
 	DIR *scripts_dir, *lang_dir;
 	struct perf_session *session;
+	struct perf_data_file file = {
+		.path = input_name,
+		.mode = PERF_DATA_MODE_READ,
+	};
 	char *temp;
 	int i = 0;
 
-	session = perf_session__new(input_name, O_RDONLY, 0, false, NULL);
+	session = perf_session__new(&file, false, NULL);
 	if (!session)
 		return -1;
 
@@ -1296,6 +1301,10 @@ int cmd_script(int argc, const char **argv, const char *prefix __maybe_unused)
 		"perf script [<options>] <top-script> [script-args]",
 		NULL
 	};
+	struct perf_data_file file = {
+		.path = input_name,
+		.mode = PERF_DATA_MODE_READ,
+	};
 
 	setup_scripting();
 
@@ -1465,8 +1474,7 @@ int cmd_script(int argc, const char **argv, const char *prefix __maybe_unused)
 	if (!script_name)
 		setup_pager();
 
-	session = perf_session__new(input_name, O_RDONLY, 0, false,
-				    &perf_script);
+	session = perf_session__new(&file, false, &perf_script);
 	if (session == NULL)
 		return -ENOMEM;
 
diff --git a/tools/perf/builtin-timechart.c b/tools/perf/builtin-timechart.c
index c2e0231..e11c61d 100644
--- a/tools/perf/builtin-timechart.c
+++ b/tools/perf/builtin-timechart.c
@@ -36,6 +36,7 @@
 #include "util/session.h"
 #include "util/svghelper.h"
 #include "util/tool.h"
+#include "util/data.h"
 
 #define SUPPORT_OLD_POWER_EVENTS 1
 #define PWR_EVENT_EXIT -1
@@ -990,8 +991,13 @@ static int __cmd_timechart(const char *output_name)
 		{ "power:power_frequency",	process_sample_power_frequency },
 #endif
 	};
-	struct perf_session *session = perf_session__new(input_name, O_RDONLY,
-							 0, false, &perf_timechart);
+	struct perf_data_file file = {
+		.path = input_name,
+		.mode = PERF_DATA_MODE_READ,
+	};
+
+	struct perf_session *session = perf_session__new(&file, false,
+							 &perf_timechart);
 	int ret = -EINVAL;
 
 	if (session == NULL)
diff --git a/tools/perf/builtin-top.c b/tools/perf/builtin-top.c
index bbf4635..3b01281 100644
--- a/tools/perf/builtin-top.c
+++ b/tools/perf/builtin-top.c
@@ -931,11 +931,15 @@ static int __cmd_top(struct perf_top *top)
 	struct perf_record_opts *opts = &top->record_opts;
 	pthread_t thread;
 	int ret;
+	struct perf_data_file file = {
+		.mode = PERF_DATA_MODE_WRITE,
+	};
+
 	/*
 	 * FIXME: perf_session__new should allow passing a O_MMAP, so that all this
 	 * mmap reading, etc is encapsulated in it. Use O_WRONLY for now.
 	 */
-	top->session = perf_session__new(NULL, O_WRONLY, false, false, NULL);
+	top->session = perf_session__new(&file, false, NULL);
 	if (top->session == NULL)
 		return -ENOMEM;
 
diff --git a/tools/perf/perf.h b/tools/perf/perf.h
index 32bd102..95b2903 100644
--- a/tools/perf/perf.h
+++ b/tools/perf/perf.h
@@ -215,7 +215,6 @@ struct perf_record_opts {
 	bool	     no_delay;
 	bool	     no_inherit;
 	bool	     no_samples;
-	bool	     pipe_output;
 	bool	     raw_samples;
 	bool	     sample_address;
 	bool	     sample_weight;
diff --git a/tools/perf/tests/session-simple.c b/tools/perf/tests/session-simple.c
index 3a34843..4038a8e 100644
--- a/tools/perf/tests/session-simple.c
+++ b/tools/perf/tests/session-simple.c
@@ -7,6 +7,7 @@
 #include "header.h"
 #include "util.h"
 #include "evlist.h"
+#include "data.h"
 
 
 static char *get_file(void)
@@ -102,28 +103,29 @@ static int store_event(int fd, union perf_event *event, size_t *size)
 
 static int session_write(void)
 {
+	struct perf_data_file file = {
+		.mode  = PERF_DATA_MODE_WRITE,
+	};
 	struct perf_session *session;
 	struct perf_evlist *evlist;
 	size_t size = 0;
-	char *file;
-	int feat, fd;
+	int feat;
 
-	file = get_file();
-	TEST_ASSERT_VAL("failed to get temporary file", file);
+	file.path = get_file();
+	TEST_ASSERT_VAL("failed to get temporary file", file.path);
 
-	fd = open(file, O_RDWR);
-	TEST_ASSERT_VAL("failed to open data file", fd >= 0);
+	file.fd = open(file.path, O_RDWR);
+	TEST_ASSERT_VAL("failed to open data file", file.fd >= 0);
 
 	evlist = get_evlist();
 	TEST_ASSERT_VAL("failed to get evlist", evlist);
 
 	pr_debug("session writing start\n");
 
-	session = perf_session__new(file, O_WRONLY, true, false, NULL);
+	session = perf_session__new(&file, false, NULL);
 	TEST_ASSERT_VAL("failed to create session", session);
 
 	session->evlist = evlist;
-	session->fd     = fd;
 
 	for (feat = HEADER_FIRST_FEATURE; feat < HEADER_LAST_FEATURE; feat++)
 		perf_header__set_feat(&session->header, feat);
@@ -133,18 +135,18 @@ static int session_write(void)
 	perf_header__clear_feat(&session->header, HEADER_BRANCH_STACK);
 
 	TEST_ASSERT_VAL("failed to write header",
-		!perf_session__prepare_header(fd));
+		!perf_session__prepare_header(file.fd));
 
 	TEST_ASSERT_VAL("failed to store MMAP event",
-		!store_event(fd, get_event_MMAP(), &size));
+		!store_event(file.fd, get_event_MMAP(), &size));
 
 	TEST_ASSERT_VAL("failed to store COMM event",
-		!store_event(fd, get_event_COMM(), &size));
+		!store_event(file.fd, get_event_COMM(), &size));
 
 	session->header.data_size += size;
 
 	TEST_ASSERT_VAL("failed to write header",
-		!perf_session__write_header(session, evlist, fd));
+		!perf_session__write_header(session, evlist, file.fd));
 
 	perf_session__delete(session);
 	perf_evlist__delete(evlist);
@@ -235,19 +237,21 @@ static int process_mmap(struct perf_tool *tool __maybe_unused,
 
 static int session_read(void)
 {
+	struct perf_data_file file = {
+		.mode  = PERF_DATA_MODE_READ,
+	};
 	struct perf_session *session;
 	struct perf_tool tool = {
 		.mmap = process_mmap,
 		.comm = process_comm,
 	};
-	char *file;
 
-	file = get_file();
-	TEST_ASSERT_VAL("failed to get temporary file", file);
+	file.path = get_file();
+	TEST_ASSERT_VAL("failed to get temporary file", file.path);
 
 	pr_debug("session reading start\n");
 
-	session = perf_session__new(file, O_RDONLY, false, false, &tool);
+	session = perf_session__new(&file, false, &tool);
 	TEST_ASSERT_VAL("failed to create session", session);
 
 	TEST_ASSERT_VAL("failed to process events", session);
diff --git a/tools/perf/util/data.h b/tools/perf/util/data.h
new file mode 100644
index 0000000..ffa0186
--- /dev/null
+++ b/tools/perf/util/data.h
@@ -0,0 +1,29 @@
+#ifndef __PERF_DATA_H
+#define __PERF_DATA_H
+
+#include <stdbool.h>
+
+enum perf_data_mode {
+	PERF_DATA_MODE_WRITE,
+	PERF_DATA_MODE_READ,
+};
+
+struct perf_data_file {
+	const char *path;
+	int fd;
+	bool is_pipe;
+	bool force;
+	enum perf_data_mode mode;
+};
+
+static inline bool perf_data_file__is_read(struct perf_data_file *file)
+{
+	return file->mode == PERF_DATA_MODE_READ;
+}
+
+static inline bool perf_data_file__is_write(struct perf_data_file *file)
+{
+	return file->mode == PERF_DATA_MODE_WRITE;
+}
+
+#endif /* __PERF_DATA_H */
diff --git a/tools/perf/util/session.c b/tools/perf/util/session.c
index 272c9cf..60d55ca 100644
--- a/tools/perf/util/session.c
+++ b/tools/perf/util/session.c
@@ -101,11 +101,11 @@ static void perf_session__destroy_kernel_maps(struct perf_session *self)
 	machines__destroy_kernel_maps(&self->machines);
 }
 
-struct perf_session *perf_session__new(const char *filename, int mode,
-				       bool force, bool repipe,
-				       struct perf_tool *tool)
+struct perf_session *perf_session__new(struct perf_data_file *file,
+				       bool repipe, struct perf_tool *tool)
 {
 	struct perf_session *self;
+	const char *filename = file->path;
 	struct stat st;
 	size_t len;
 
@@ -129,11 +129,11 @@ struct perf_session *perf_session__new(const char *filename, int mode,
 	INIT_LIST_HEAD(&self->ordered_samples.to_free);
 	machines__init(&self->machines);
 
-	if (mode == O_RDONLY) {
-		if (perf_session__open(self, force) < 0)
+	if (perf_data_file__is_read(file)) {
+		if (perf_session__open(self, file->force) < 0)
 			goto out_delete;
 		perf_session__set_id_hdr_size(self);
-	} else if (mode == O_WRONLY) {
+	} else if (perf_data_file__is_write(file)) {
 		/*
 		 * In O_RDONLY mode this will be performed when reading the
 		 * kernel MMAP event, in perf_event__process_mmap().
diff --git a/tools/perf/util/session.h b/tools/perf/util/session.h
index ad8d3d4..8054acd 100644
--- a/tools/perf/util/session.h
+++ b/tools/perf/util/session.h
@@ -7,6 +7,7 @@
 #include "machine.h"
 #include "symbol.h"
 #include "thread.h"
+#include "data.h"
 #include <linux/rbtree.h>
 #include <linux/perf_event.h>
 
@@ -43,9 +44,8 @@ struct perf_session {
 
 struct perf_tool;
 
-struct perf_session *perf_session__new(const char *filename, int mode,
-				       bool force, bool repipe,
-				       struct perf_tool *tool);
+struct perf_session *perf_session__new(struct perf_data_file *file,
+				       bool repipe, struct perf_tool *tool);
 void perf_session__delete(struct perf_session *session);
 
 void perf_event_header__bswap(struct perf_event_header *self);
-- 
1.7.11.7

[PATCH 20/23] perf tools: Add perf_data_file__open interface to data object

[Jiri Olsa]

Adding perf_data_file__open interface to data object
to open the perf.data file for both read and write.

Signed-off-by: Jiri Olsa <jolsa@redhat.com>
Cc: Corey Ashford <cjashfor@linux.vnet.ibm.com>
Cc: Frederic Weisbecker <fweisbec@gmail.com>
Cc: Ingo Molnar <mingo@elte.hu>
Cc: Namhyung Kim <namhyung@kernel.org>
Cc: Paul Mackerras <paulus@samba.org>
Cc: Peter Zijlstra <a.p.zijlstra@chello.nl>
Cc: Arnaldo Carvalho de Melo <acme@redhat.com>
Cc: Andi Kleen <ak@linux.intel.com>
Cc: David Ahern <dsahern@gmail.com>
---
 tools/perf/Makefile               |   1 +
 tools/perf/builtin-record.c       |  34 +----------
 tools/perf/builtin-top.c          |   9 +--
 tools/perf/tests/session-simple.c |   3 -
 tools/perf/util/data.c            | 120 ++++++++++++++++++++++++++++++++++++++
 tools/perf/util/data.h            |   4 ++
 tools/perf/util/session.c         |  84 +++++++++-----------------
 tools/perf/util/session.h         |   2 +-
 8 files changed, 154 insertions(+), 103 deletions(-)
 create mode 100644 tools/perf/util/data.c

diff --git a/tools/perf/Makefile b/tools/perf/Makefile
index ca3e17b..f86424c 100644
--- a/tools/perf/Makefile
+++ b/tools/perf/Makefile
@@ -360,6 +360,7 @@ LIB_OBJS += $(OUTPUT)util/rblist.o
 LIB_OBJS += $(OUTPUT)util/intlist.o
 LIB_OBJS += $(OUTPUT)util/vdso.o
 LIB_OBJS += $(OUTPUT)util/stat.o
+LIB_OBJS += $(OUTPUT)util/data.o
 
 LIB_OBJS += $(OUTPUT)ui/setup.o
 LIB_OBJS += $(OUTPUT)ui/helpline.o
diff --git a/tools/perf/builtin-record.c b/tools/perf/builtin-record.c
index c4ec7eb..4754ec2 100644
--- a/tools/perf/builtin-record.c
+++ b/tools/perf/builtin-record.c
@@ -441,15 +441,12 @@ out:
 
 static int __cmd_record(struct perf_record *rec, int argc, const char **argv)
 {
-	struct stat st;
-	int flags;
 	int err, feat;
 	unsigned long waking = 0;
 	const bool forks = argc > 0;
 	struct perf_record_opts *opts = &rec->opts;
 	struct perf_evlist *evsel_list = rec->evlist;
 	struct perf_data_file *file = rec->file;
-	const char *output_name = file->path;
 	struct perf_session *session;
 	bool disabled = false;
 
@@ -463,35 +460,6 @@ static int __cmd_record(struct perf_record *rec, int argc, const char **argv)
 	signal(SIGUSR1, sig_handler);
 	signal(SIGTERM, sig_handler);
 
-	if (!output_name) {
-		if (!fstat(STDOUT_FILENO, &st) && S_ISFIFO(st.st_mode))
-			file->is_pipe = true;
-		else
-			file->path = output_name = "perf.data";
-	}
-	if (output_name) {
-		if (!strcmp(output_name, "-"))
-			file->is_pipe = true;
-		else if (!stat(output_name, &st) && st.st_size) {
-			char oldname[PATH_MAX];
-			snprintf(oldname, sizeof(oldname), "%s.old",
-				 output_name);
-			unlink(oldname);
-			rename(output_name, oldname);
-		}
-	}
-
-	flags = O_CREAT|O_RDWR|O_TRUNC;
-
-	if (file->is_pipe)
-		file->fd = STDOUT_FILENO;
-	else
-		file->fd = open(output_name, flags, S_IRUSR | S_IWUSR);
-	if (file->fd < 0) {
-		perror("failed to create output file");
-		return -1;
-	}
-
 	session = perf_session__new(file, false, NULL);
 	if (session == NULL) {
 		pr_err("Not enough memory for reading perf file header\n");
@@ -619,7 +587,7 @@ static int __cmd_record(struct perf_record *rec, int argc, const char **argv)
 	fprintf(stderr,
 		"[ perf record: Captured and wrote %.3f MB %s (~%" PRIu64 " samples) ]\n",
 		(double)rec->bytes_written / 1024.0 / 1024.0,
-		output_name,
+		file->path,
 		rec->bytes_written / 24);
 
 	return 0;
diff --git a/tools/perf/builtin-top.c b/tools/perf/builtin-top.c
index 3b01281..26f80a9 100644
--- a/tools/perf/builtin-top.c
+++ b/tools/perf/builtin-top.c
@@ -931,15 +931,8 @@ static int __cmd_top(struct perf_top *top)
 	struct perf_record_opts *opts = &top->record_opts;
 	pthread_t thread;
 	int ret;
-	struct perf_data_file file = {
-		.mode = PERF_DATA_MODE_WRITE,
-	};
 
-	/*
-	 * FIXME: perf_session__new should allow passing a O_MMAP, so that all this
-	 * mmap reading, etc is encapsulated in it. Use O_WRONLY for now.
-	 */
-	top->session = perf_session__new(&file, false, NULL);
+	top->session = perf_session__new(NULL, false, NULL);
 	if (top->session == NULL)
 		return -ENOMEM;
 
diff --git a/tools/perf/tests/session-simple.c b/tools/perf/tests/session-simple.c
index 4038a8e..391671b 100644
--- a/tools/perf/tests/session-simple.c
+++ b/tools/perf/tests/session-simple.c
@@ -114,9 +114,6 @@ static int session_write(void)
 	file.path = get_file();
 	TEST_ASSERT_VAL("failed to get temporary file", file.path);
 
-	file.fd = open(file.path, O_RDWR);
-	TEST_ASSERT_VAL("failed to open data file", file.fd >= 0);
-
 	evlist = get_evlist();
 	TEST_ASSERT_VAL("failed to get evlist", evlist);
 
diff --git a/tools/perf/util/data.c b/tools/perf/util/data.c
new file mode 100644
index 0000000..7d09faf
--- /dev/null
+++ b/tools/perf/util/data.c
@@ -0,0 +1,120 @@
+#include <linux/compiler.h>
+#include <linux/kernel.h>
+#include <sys/types.h>
+#include <sys/stat.h>
+#include <unistd.h>
+#include <string.h>
+
+#include "data.h"
+#include "util.h"
+
+static bool check_pipe(struct perf_data_file *file)
+{
+	struct stat st;
+	bool is_pipe = false;
+	int fd = perf_data_file__is_read(file) ?
+		 STDIN_FILENO : STDOUT_FILENO;
+
+	if (!file->path) {
+		if (!fstat(fd, &st) && S_ISFIFO(st.st_mode))
+			is_pipe = true;
+	} else {
+		if (!strcmp(file->path, "-"))
+			is_pipe = true;
+	}
+
+	if (is_pipe)
+		file->fd = fd;
+
+	return file->is_pipe = is_pipe;
+}
+
+static int check_backup(struct perf_data_file *file)
+{
+	struct stat st;
+
+	if (!stat(file->path, &st) && st.st_size) {
+		/* TODO check errors properly */
+		char oldname[PATH_MAX];
+		snprintf(oldname, sizeof(oldname), "%s.old",
+			 file->path);
+		unlink(oldname);
+		rename(file->path, oldname);
+	}
+
+	return 0;
+}
+
+static int open_file_read(struct perf_data_file *file)
+{
+	struct stat st;
+	int fd;
+
+	fd = open(file->path, O_RDONLY);
+	if (fd < 0) {
+		int err = errno;
+
+		pr_err("failed to open %s: %s", file->path, strerror(err));
+		if (err == ENOENT && !strcmp(file->path, "perf.data"))
+			pr_err("  (try 'perf record' first)");
+		pr_err("\n");
+		return -err;
+	}
+
+	if (fstat(fd, &st) < 0)
+		goto out_close;
+
+	if (!file->force && st.st_uid && (st.st_uid != geteuid())) {
+		pr_err("file %s not owned by current user or root\n",
+		       file->path);
+		goto out_close;
+	}
+
+	if (!st.st_size) {
+		pr_info("zero-sized file (%s), nothing to do!\n",
+			file->path);
+		goto out_close;
+	}
+
+	file->size = st.st_size;
+	return fd;
+
+ out_close:
+	close(fd);
+	return -1;
+}
+
+static int open_file_write(struct perf_data_file *file)
+{
+	if (check_backup(file))
+		return -1;
+
+	return open(file->path, O_CREAT|O_RDWR|O_TRUNC, S_IRUSR|S_IWUSR);
+}
+
+static int open_file(struct perf_data_file *file)
+{
+	int fd;
+
+	fd = perf_data_file__is_read(file) ?
+	     open_file_read(file) : open_file_write(file);
+
+	file->fd = fd;
+	return fd < 0 ? -1 : 0;
+}
+
+int perf_data_file__open(struct perf_data_file *file)
+{
+	if (check_pipe(file))
+		return 0;
+
+	if (!file->path)
+		file->path = "perf.data";
+
+	return open_file(file);
+}
+
+void perf_data_file__close(struct perf_data_file *file)
+{
+	close(file->fd);
+}
diff --git a/tools/perf/util/data.h b/tools/perf/util/data.h
index ffa0186..d6c262e 100644
--- a/tools/perf/util/data.h
+++ b/tools/perf/util/data.h
@@ -13,6 +13,7 @@ struct perf_data_file {
 	int fd;
 	bool is_pipe;
 	bool force;
+	unsigned long size;
 	enum perf_data_mode mode;
 };
 
@@ -26,4 +27,7 @@ static inline bool perf_data_file__is_write(struct perf_data_file *file)
 	return file->mode == PERF_DATA_MODE_WRITE;
 }
 
+int perf_data_file__open(struct perf_data_file *file);
+void perf_data_file__close(struct perf_data_file *file);
+
 #endif /* __PERF_DATA_H */
diff --git a/tools/perf/util/session.c b/tools/perf/util/session.c
index 60d55ca..cfd9735 100644
--- a/tools/perf/util/session.c
+++ b/tools/perf/util/session.c
@@ -16,46 +16,14 @@
 #include "perf_regs.h"
 #include "vdso.h"
 
-static int perf_session__open(struct perf_session *self, bool force)
+static int perf_session__open(struct perf_session *self)
 {
-	struct stat input_stat;
-
-	if (!strcmp(self->filename, "-")) {
-		self->fd_pipe = true;
-		self->fd = STDIN_FILENO;
-
+	if (self->fd_pipe) {
 		if (perf_session__read_header(self) < 0)
 			pr_err("incompatible file format (rerun with -v to learn more)");
-
 		return 0;
 	}
 
-	self->fd = open(self->filename, O_RDONLY);
-	if (self->fd < 0) {
-		int err = errno;
-
-		pr_err("failed to open %s: %s", self->filename, strerror(err));
-		if (err == ENOENT && !strcmp(self->filename, "perf.data"))
-			pr_err("  (try 'perf record' first)");
-		pr_err("\n");
-		return -errno;
-	}
-
-	if (fstat(self->fd, &input_stat) < 0)
-		goto out_close;
-
-	if (!force && input_stat.st_uid && (input_stat.st_uid != geteuid())) {
-		pr_err("file %s not owned by current user or root\n",
-		       self->filename);
-		goto out_close;
-	}
-
-	if (!input_stat.st_size) {
-		pr_info("zero-sized file (%s), nothing to do!\n",
-			self->filename);
-		goto out_close;
-	}
-
 	if (perf_session__read_header(self) < 0) {
 		pr_err("incompatible file format (rerun with -v to learn more)");
 		goto out_close;
@@ -71,12 +39,9 @@ static int perf_session__open(struct perf_session *self, bool force)
 		goto out_close;
 	}
 
-	self->size = input_stat.st_size;
 	return 0;
 
 out_close:
-	close(self->fd);
-	self->fd = -1;
 	return -1;
 }
 
@@ -105,35 +70,35 @@ struct perf_session *perf_session__new(struct perf_data_file *file,
 				       bool repipe, struct perf_tool *tool)
 {
 	struct perf_session *self;
-	const char *filename = file->path;
-	struct stat st;
-	size_t len;
 
-	if (!filename || !strlen(filename)) {
-		if (!fstat(STDIN_FILENO, &st) && S_ISFIFO(st.st_mode))
-			filename = "-";
-		else
-			filename = "perf.data";
-	}
-
-	len = strlen(filename);
-	self = zalloc(sizeof(*self) + len);
-
-	if (self == NULL)
+	self = zalloc(sizeof(*self));
+	if (!self)
 		goto out;
 
-	memcpy(self->filename, filename, len);
 	self->repipe = repipe;
 	INIT_LIST_HEAD(&self->ordered_samples.samples);
 	INIT_LIST_HEAD(&self->ordered_samples.sample_cache);
 	INIT_LIST_HEAD(&self->ordered_samples.to_free);
 	machines__init(&self->machines);
 
-	if (perf_data_file__is_read(file)) {
-		if (perf_session__open(self, file->force) < 0)
+	if (file) {
+		if (perf_data_file__open(file))
 			goto out_delete;
-		perf_session__set_id_hdr_size(self);
-	} else if (perf_data_file__is_write(file)) {
+
+		self->fd       = file->fd;
+		self->fd_pipe  = file->is_pipe;
+		self->filename = file->path;
+		self->size     = file->size;
+
+		if (perf_data_file__is_read(file)) {
+			if (perf_session__open(self) < 0)
+				goto out_close;
+
+			perf_session__set_id_hdr_size(self);
+		}
+	}
+
+	if (!file || perf_data_file__is_write(file)) {
 		/*
 		 * In O_RDONLY mode this will be performed when reading the
 		 * kernel MMAP event, in perf_event__process_mmap().
@@ -148,10 +113,13 @@ struct perf_session *perf_session__new(struct perf_data_file *file,
 		tool->ordered_samples = false;
 	}
 
-out:
 	return self;
-out_delete:
+
+ out_close:
+	perf_data_file__close(file);
+ out_delete:
 	perf_session__delete(self);
+ out:
 	return NULL;
 }
 
diff --git a/tools/perf/util/session.h b/tools/perf/util/session.h
index 8054acd..aa8fc69 100644
--- a/tools/perf/util/session.h
+++ b/tools/perf/util/session.h
@@ -39,7 +39,7 @@ struct perf_session {
 	bool			fd_pipe;
 	bool			repipe;
 	struct ordered_samples	ordered_samples;
-	char			filename[1];
+	const char		*filename;
 };
 
 struct perf_tool;
-- 
1.7.11.7

[PATCH 21/23] perf tools: Separating data file properties from session

[Jiri Olsa]

Removing 'fd, fd_pipe, filename, size' from struct perf_session
and replacing it with struct perf_data_file.

Signed-off-by: Jiri Olsa <jolsa@redhat.com>
Cc: Corey Ashford <cjashfor@linux.vnet.ibm.com>
Cc: Frederic Weisbecker <fweisbec@gmail.com>
Cc: Ingo Molnar <mingo@elte.hu>
Cc: Namhyung Kim <namhyung@kernel.org>
Cc: Paul Mackerras <paulus@samba.org>
Cc: Peter Zijlstra <a.p.zijlstra@chello.nl>
Cc: Arnaldo Carvalho de Melo <acme@redhat.com>
Cc: Andi Kleen <ak@linux.intel.com>
Cc: David Ahern <dsahern@gmail.com>
---
 tools/perf/builtin-annotate.c     |  2 +-
 tools/perf/builtin-buildid-list.c |  2 +-
 tools/perf/builtin-record.c       |  2 --
 tools/perf/builtin-report.c       |  8 +++++---
 tools/perf/builtin-script.c       |  2 +-
 tools/perf/util/data.h            | 15 +++++++++++++++
 tools/perf/util/header.c          | 30 ++++++++++++++++++++----------
 tools/perf/util/header.h          |  1 +
 tools/perf/util/session.c         | 36 +++++++++++++++++++-----------------
 tools/perf/util/session.h         |  5 +----
 10 files changed, 64 insertions(+), 39 deletions(-)

diff --git a/tools/perf/builtin-annotate.c b/tools/perf/builtin-annotate.c
index 048576f..a032a28 100644
--- a/tools/perf/builtin-annotate.c
+++ b/tools/perf/builtin-annotate.c
@@ -247,7 +247,7 @@ static int __cmd_annotate(struct perf_annotate *ann)
 	}
 
 	if (total_nr_samples == 0) {
-		ui__error("The %s file has no samples!\n", session->filename);
+		ui__error("The %s file has no samples!\n", file.path);
 		goto out_delete;
 	}
 
diff --git a/tools/perf/builtin-buildid-list.c b/tools/perf/builtin-buildid-list.c
index 0164c1c..ed3873b 100644
--- a/tools/perf/builtin-buildid-list.c
+++ b/tools/perf/builtin-buildid-list.c
@@ -73,7 +73,7 @@ static int perf_session__list_build_ids(bool force, bool with_hits)
 	 * in pipe-mode, the only way to get the buildids is to parse
 	 * the record stream. Buildids are stored as RECORD_HEADER_BUILD_ID
 	 */
-	if (with_hits || session->fd_pipe)
+	if (with_hits || perf_data_file__is_pipe(&file))
 		perf_session__process_events(session, &build_id__mark_dso_hit_ops);
 
 	perf_session__fprintf_dsos_buildid(session, stdout, dso__skip_buildid, with_hits);
diff --git a/tools/perf/builtin-record.c b/tools/perf/builtin-record.c
index 4754ec2..960908da 100644
--- a/tools/perf/builtin-record.c
+++ b/tools/perf/builtin-record.c
@@ -255,7 +255,6 @@ out:
 
 static int process_buildids(struct perf_record *rec)
 {
-	struct perf_data_file *file  = rec->file;
 	struct perf_session *session = rec->session;
 	u64 data_offset              = PERF_FILE_HEADER__DATA_OFFSET;
 	u64 size                     = session->header.data_size;
@@ -263,7 +262,6 @@ static int process_buildids(struct perf_record *rec)
 	if (size == 0)
 		return 0;
 
-	rec->session->fd = file->fd;
 	return __perf_session__process_events(session, data_offset,
 					      size - data_offset, size,
 					      &build_id__mark_dso_hit_ops);
diff --git a/tools/perf/builtin-report.c b/tools/perf/builtin-report.c
index 7904173..f90f4d6 100644
--- a/tools/perf/builtin-report.c
+++ b/tools/perf/builtin-report.c
@@ -369,8 +369,9 @@ static int perf_report__setup_sample_type(struct perf_report *rep)
 {
 	struct perf_session *self = rep->session;
 	u64 sample_type = perf_evlist__sample_type(self->evlist);
+	bool is_pipe = perf_data_file__is_pipe(self->file);
 
-	if (!self->fd_pipe && !(sample_type & PERF_SAMPLE_CALLCHAIN)) {
+	if (!is_pipe && !(sample_type & PERF_SAMPLE_CALLCHAIN)) {
 		if (sort__has_parent) {
 			ui__error("Selected --sort parent, but no "
 				    "callchain data. Did you call "
@@ -393,7 +394,7 @@ static int perf_report__setup_sample_type(struct perf_report *rep)
 	}
 
 	if (sort__mode == SORT_MODE__BRANCH) {
-		if (!self->fd_pipe &&
+		if (!is_pipe &&
 		    !(sample_type & PERF_SAMPLE_BRANCH_STACK)) {
 			ui__error("Selected -b but no branch data. "
 				  "Did you call perf record without -b?\n");
@@ -491,6 +492,7 @@ static int __cmd_report(struct perf_report *rep)
 	struct map *kernel_map;
 	struct kmap *kernel_kmap;
 	const char *help = "For a higher level overview, try: perf report --sort comm,dso";
+	struct perf_data_file *file = session->file;
 
 	signal(SIGINT, sig_handler);
 
@@ -572,7 +574,7 @@ static int __cmd_report(struct perf_report *rep)
 	}
 
 	if (nr_samples == 0) {
-		ui__error("The %s file has no samples!\n", session->filename);
+		ui__error("The %s file has no samples!\n", file->path);
 		return 0;
 	}
 
diff --git a/tools/perf/builtin-script.c b/tools/perf/builtin-script.c
index bbd524c..e0269bf 100644
--- a/tools/perf/builtin-script.c
+++ b/tools/perf/builtin-script.c
@@ -1501,7 +1501,7 @@ int cmd_script(int argc, const char **argv, const char *prefix __maybe_unused)
 			return -1;
 		}
 
-		input = open(session->filename, O_RDONLY);	/* input_name */
+		input = open(file.path, O_RDONLY);	/* input_name */
 		if (input < 0) {
 			perror("failed to open file");
 			return -1;
diff --git a/tools/perf/util/data.h b/tools/perf/util/data.h
index d6c262e..8c2df80 100644
--- a/tools/perf/util/data.h
+++ b/tools/perf/util/data.h
@@ -27,6 +27,21 @@ static inline bool perf_data_file__is_write(struct perf_data_file *file)
 	return file->mode == PERF_DATA_MODE_WRITE;
 }
 
+static inline int perf_data_file__is_pipe(struct perf_data_file *file)
+{
+	return file->is_pipe;
+}
+
+static inline int perf_data_file__fd(struct perf_data_file *file)
+{
+	return file->fd;
+}
+
+static inline unsigned long perf_data_file__size(struct perf_data_file *file)
+{
+	return file->size;
+}
+
 int perf_data_file__open(struct perf_data_file *file);
 void perf_data_file__close(struct perf_data_file *file);
 
diff --git a/tools/perf/util/header.c b/tools/perf/util/header.c
index f5b7529..cb986b5 100644
--- a/tools/perf/util/header.c
+++ b/tools/perf/util/header.c
@@ -2174,7 +2174,7 @@ int perf_header__fprintf_info(struct perf_session *session, FILE *fp, bool full)
 {
 	struct header_print_data hd;
 	struct perf_header *header = &session->header;
-	int fd = session->fd;
+	int fd = perf_data_file__fd(session->file);
 	hd.fp = fp;
 	hd.full = full;
 
@@ -2642,8 +2642,10 @@ static int perf_header__read_pipe(struct perf_session *session)
 {
 	struct perf_header *header = &session->header;
 	struct perf_pipe_file_header f_header;
+	struct perf_data_file *file = session->file;
 
-	if (perf_file_header__read_pipe(&f_header, header, session->fd,
+	if (perf_file_header__read_pipe(&f_header, header,
+					perf_data_file__fd(file),
 					session->repipe) < 0) {
 		pr_debug("incompatible file format\n");
 		return -EINVAL;
@@ -2745,12 +2747,13 @@ static int perf_evlist__prepare_tracepoint_events(struct perf_evlist *evlist,
 static int __perf_session__read_header_v2(struct perf_session *session,
 					  struct perf_file_header *header)
 {
+	struct perf_data_file		*file = session->file;
 	struct perf_header		*ph = &session->header;
 	struct perf_file_header_v2	*v2 = &header->v2;
 	struct perf_file_attr		f_attr;
 	u64				f_id;
 	int nr_attrs, nr_ids, i, j;
-	int fd = session->fd;
+	int fd = perf_data_file__fd(file);
 
 	session->evlist = perf_evlist__new();
 	if (session->evlist == NULL)
@@ -2844,6 +2847,8 @@ static int perf_session__read_header_v3(struct perf_session *session,
 {
 	struct perf_header *ph = &session->header;
 	struct perf_file_header_v3 *v3 = &header->v3;
+	struct perf_data_file *file = session->file;
+	int fd = perf_data_file__fd(file);
 
 	memcpy(&ph->adds_features, &v3->adds_features,
 	       sizeof(ph->adds_features));
@@ -2852,7 +2857,7 @@ static int perf_session__read_header_v3(struct perf_session *session,
 	ph->data_size	 = v3->data.size;
 	ph->feat_offset  = v3->features.offset;
 
-	perf_header__process_sections(ph, session->fd, &session->pevent,
+	perf_header__process_sections(ph, fd, &session->pevent,
 				      perf_file_section__process);
 
 	return 0;
@@ -2862,8 +2867,10 @@ static int perf_header_read_file(struct perf_session *session)
 {
 	struct perf_file_header header;
 	struct perf_header *ph = &session->header;
+	struct perf_data_file *file = session->file;
 
-	if (perf_file_header__read(&header, &session->header, session->fd))
+	if (perf_file_header__read(&header, &session->header,
+				   perf_data_file__fd(file)))
 		return -1;
 
 	/* read version specific data */
@@ -2875,7 +2882,9 @@ static int perf_header_read_file(struct perf_session *session)
 
 int perf_session__read_header(struct perf_session *session)
 {
-	if (session->fd_pipe)
+	struct perf_data_file *file = session->file;
+
+	if (perf_data_file__is_pipe(file))
 		return perf_header__read_pipe(session);
 
 	return perf_header_read_file(session) < 0 ? -EINVAL : 0;
@@ -3025,18 +3034,19 @@ int perf_event__process_tracing_data(struct perf_tool *tool __maybe_unused,
 				     struct perf_session *session)
 {
 	ssize_t size_read, padding, size = event->tracing_data.size;
-	off_t offset = lseek(session->fd, 0, SEEK_CUR);
+	int fd = perf_data_file__fd(session->file);
+	off_t offset = lseek(fd, 0, SEEK_CUR);
 	char buf[BUFSIZ];
 
 	/* setup for reading amidst mmap */
-	lseek(session->fd, offset + sizeof(struct tracing_data_event),
+	lseek(fd, offset + sizeof(struct tracing_data_event),
 	      SEEK_SET);
 
-	size_read = trace_report(session->fd, &session->pevent,
+	size_read = trace_report(fd, &session->pevent,
 				 session->repipe);
 	padding = PERF_ALIGN(size_read, sizeof(u64)) - size_read;
 
-	if (readn(session->fd, buf, padding) < 0) {
+	if (readn(fd, buf, padding) < 0) {
 		pr_err("%s: reading input file", __func__);
 		return -1;
 	}
diff --git a/tools/perf/util/header.h b/tools/perf/util/header.h
index 4982e04..a5875f2 100644
--- a/tools/perf/util/header.h
+++ b/tools/perf/util/header.h
@@ -6,6 +6,7 @@
 #include <stdbool.h>
 #include "types.h"
 #include "event.h"
+#include "data.h"
 
 #include <linux/bitmap.h>
 
diff --git a/tools/perf/util/session.c b/tools/perf/util/session.c
index cfd9735..7a92450 100644
--- a/tools/perf/util/session.c
+++ b/tools/perf/util/session.c
@@ -18,17 +18,16 @@
 
 static int perf_session__open(struct perf_session *self)
 {
-	if (self->fd_pipe) {
-		if (perf_session__read_header(self) < 0)
-			pr_err("incompatible file format (rerun with -v to learn more)");
-		return 0;
-	}
+	struct perf_data_file *file = self->file;
 
 	if (perf_session__read_header(self) < 0) {
 		pr_err("incompatible file format (rerun with -v to learn more)");
 		goto out_close;
 	}
 
+	if (perf_data_file__is_pipe(file))
+		return 0;
+
 	if (!perf_evlist__valid_sample_type(self->evlist)) {
 		pr_err("non matching sample_type");
 		goto out_close;
@@ -85,10 +84,7 @@ struct perf_session *perf_session__new(struct perf_data_file *file,
 		if (perf_data_file__open(file))
 			goto out_delete;
 
-		self->fd       = file->fd;
-		self->fd_pipe  = file->is_pipe;
-		self->filename = file->path;
-		self->size     = file->size;
+		self->file = file;
 
 		if (perf_data_file__is_read(file)) {
 			if (perf_session__open(self) < 0)
@@ -156,7 +152,8 @@ void perf_session__delete(struct perf_session *self)
 	perf_session__delete_threads(self);
 	perf_session_env__delete(&self->header.env);
 	machines__exit(&self->machines);
-	close(self->fd);
+	if (self->file)
+		perf_data_file__close(self->file);
 	free(self);
 	vdso__exit();
 }
@@ -876,6 +873,7 @@ static int perf_session__preprocess_sample(struct perf_session *session,
 static int perf_session__process_user_event(struct perf_session *session, union perf_event *event,
 					    struct perf_tool *tool, u64 file_offset)
 {
+	int fd = perf_data_file__fd(session->file);
 	int err;
 
 	dump_event(session, event, file_offset, NULL);
@@ -889,7 +887,7 @@ static int perf_session__process_user_event(struct perf_session *session, union
 		return err;
 	case PERF_RECORD_HEADER_TRACING_DATA:
 		/* setup for reading amidst mmap */
-		lseek(session->fd, file_offset, SEEK_SET);
+		lseek(fd, file_offset, SEEK_SET);
 		return tool->tracing_data(tool, event, session);
 	case PERF_RECORD_HEADER_BUILD_ID:
 		return tool->build_id(tool, event, session);
@@ -1020,6 +1018,7 @@ volatile int session_done;
 static int __perf_session__process_pipe_events(struct perf_session *self,
 					       struct perf_tool *tool)
 {
+	int fd = perf_data_file__fd(self->file);
 	union perf_event *event;
 	uint32_t size, cur_size = 0;
 	void *buf = NULL;
@@ -1038,7 +1037,7 @@ static int __perf_session__process_pipe_events(struct perf_session *self,
 		return -errno;
 more:
 	event = buf;
-	err = readn(self->fd, event, sizeof(struct perf_event_header));
+	err = readn(fd, event, sizeof(struct perf_event_header));
 	if (err <= 0) {
 		if (err == 0)
 			goto done;
@@ -1070,7 +1069,7 @@ more:
 	p += sizeof(struct perf_event_header);
 
 	if (size - sizeof(struct perf_event_header)) {
-		err = readn(self->fd, p, size - sizeof(struct perf_event_header));
+		err = readn(fd, p, size - sizeof(struct perf_event_header));
 		if (err <= 0) {
 			if (err == 0) {
 				pr_err("unexpected end of event stream\n");
@@ -1149,6 +1148,7 @@ int __perf_session__process_events(struct perf_session *session,
 				   u64 data_offset, u64 data_size,
 				   u64 file_size, struct perf_tool *tool)
 {
+	int fd = perf_data_file__fd(session->file);
 	u64 head, page_offset, file_offset, file_pos, progress_next;
 	int err, mmap_prot, mmap_flags, map_idx = 0;
 	size_t	mmap_size;
@@ -1181,7 +1181,7 @@ int __perf_session__process_events(struct perf_session *session,
 		mmap_flags = MAP_PRIVATE;
 	}
 remap:
-	buf = mmap(NULL, mmap_size, mmap_prot, mmap_flags, session->fd,
+	buf = mmap(NULL, mmap_size, mmap_prot, mmap_flags, fd,
 		   file_offset);
 	if (buf == MAP_FAILED) {
 		pr_err("failed to mmap file\n");
@@ -1243,16 +1243,17 @@ out_err:
 int perf_session__process_events(struct perf_session *self,
 				 struct perf_tool *tool)
 {
+	u64 size = perf_data_file__size(self->file);
 	int err;
 
 	if (perf_session__register_idle_thread(self) == NULL)
 		return -ENOMEM;
 
-	if (!self->fd_pipe)
+	if (!perf_data_file__is_pipe(self->file))
 		err = __perf_session__process_events(self,
 						     self->header.data_offset,
 						     self->header.data_size,
-						     self->size, tool);
+						     size, tool);
 	else
 		err = __perf_session__process_pipe_events(self, tool);
 
@@ -1454,13 +1455,14 @@ int perf_session__cpu_bitmap(struct perf_session *session,
 void perf_session__fprintf_info(struct perf_session *session, FILE *fp,
 				bool full)
 {
+	int fd = perf_data_file__fd(session->file);
 	struct stat st;
 	int ret;
 
 	if (session == NULL || fp == NULL)
 		return;
 
-	ret = fstat(session->fd, &st);
+	ret = fstat(fd, &st);
 	if (ret == -1)
 		return;
 
diff --git a/tools/perf/util/session.h b/tools/perf/util/session.h
index aa8fc69..650ce35 100644
--- a/tools/perf/util/session.h
+++ b/tools/perf/util/session.h
@@ -30,16 +30,13 @@ struct ordered_samples {
 
 struct perf_session {
 	struct perf_header	header;
-	unsigned long		size;
 	struct machines		machines;
 	struct perf_evlist	*evlist;
 	struct pevent		*pevent;
 	struct events_stats	stats;
-	int			fd;
-	bool			fd_pipe;
 	bool			repipe;
 	struct ordered_samples	ordered_samples;
-	const char		*filename;
+	struct perf_data_file	*file;
 };
 
 struct perf_tool;
-- 
1.7.11.7

[PATCH 22/23] perf tools: Add multi file '-M' option for record command

[Jiri Olsa]

Adding multi file '-M' option for record command to store
split event data into multiple files based on the file
size specified as an argument to the option.

Also watermark/wakeup_watermark is set to get wake ups
more often so we could get close enough to the file size
promise.

Example:
  $ ./perf record -M 100000 -e cycles:u yes > /dev/null
  ^C[ perf record: Woken up 25 times to write data ]
  [ perf record: Captured and wrote 0.630 MB perf-[0-7].data(~27531 samples) ]
  yes: Interrupt
  $ ls -l perf-0000*
  -rw------- 1 jolsa jolsa 111828 Jul 17 18:23 perf-00000.data
  -rw------- 1 jolsa jolsa 115240 Jul 17 18:23 perf-00001.data
  -rw------- 1 jolsa jolsa 112020 Jul 17 18:23 perf-00002.data
  -rw------- 1 jolsa jolsa 112020 Jul 17 18:23 perf-00003.data
  -rw------- 1 jolsa jolsa 112120 Jul 17 18:23 perf-00004.data
  -rw------- 1 jolsa jolsa 112120 Jul 17 18:23 perf-00005.data
  -rw------- 1 jolsa jolsa  18900 Jul 17 18:23 perf-00006.data
  $ ./perf diff perf-0000*
  # Event 'cycles:u'
  #
  # Data files:
  #  [0] perf-00000.data (Baseline)
  #  [1] perf-00001.data
  #  [2] perf-00002.data
  #  [3] perf-00003.data
  #  [4] perf-00004.data
  #  [5] perf-00005.data
  #  [6] perf-00006.data
  #
  # Baseline/0  Delta/1  Delta/2  Delta/3  Delta/4  Delta/5  Delta/6      Shared Object                              Symbol
  # ..........  .......  .......  .......  .......  .......  .......  .................  ..................................
  #
        37.03%   +1.23%   +0.12%   +0.61%   +1.12%   +0.93%   -7.53%  libc-2.15.so       [.] _IO_file_xsputn@@GLIBC_2.2.5
        31.40%   -0.55%   -0.79%   -1.00%   -0.66%   -0.59%   +4.09%  yes                [.] main
        16.36%   -0.18%   +1.39%   +1.07%   +0.66%   +0.23%   +1.72%  libc-2.15.so       [.] __strlen_sse2
        14.66%   -0.61%   -0.81%   -0.58%   -1.18%   -0.26%   +1.66%  libc-2.15.so       [.] fputs_unlocked
         0.24%   +0.34%   +0.27%   +0.08%   +0.24%   -0.12%   +0.35%  yes                [.] fputs_unlocked@plt
         0.12%                     -0.08%   -0.08%   -0.08%           libc-2.15.so       [.] _IO_file_write@@GLIBC_2.2.5
         0.08%                                                        libc-2.15.so       [.] new_do_write
         0.04%                                                        libc-2.15.so       [.] _IO_default_xsputn
  ...

Signed-off-by: Jiri Olsa <jolsa@redhat.com>
Cc: Corey Ashford <cjashfor@linux.vnet.ibm.com>
Cc: Frederic Weisbecker <fweisbec@gmail.com>
Cc: Ingo Molnar <mingo@elte.hu>
Cc: Namhyung Kim <namhyung@kernel.org>
Cc: Paul Mackerras <paulus@samba.org>
Cc: Peter Zijlstra <a.p.zijlstra@chello.nl>
Cc: Arnaldo Carvalho de Melo <acme@redhat.com>
Cc: Andi Kleen <ak@linux.intel.com>
Cc: David Ahern <dsahern@gmail.com>
---
 tools/perf/builtin-record.c | 167 ++++++++++++++++++++++++++++++++++++++------
 tools/perf/perf.h           |   1 +
 tools/perf/util/evsel.c     |   5 ++
 3 files changed, 152 insertions(+), 21 deletions(-)

diff --git a/tools/perf/builtin-record.c b/tools/perf/builtin-record.c
index 960908da..920d69a 100644
--- a/tools/perf/builtin-record.c
+++ b/tools/perf/builtin-record.c
@@ -66,6 +66,8 @@ struct perf_record {
 	struct perf_tool	tool;
 	struct perf_record_opts	opts;
 	u64			bytes_written;
+	u64			multi_bytes_written;
+	unsigned int		multi_idx;
 	struct perf_data_file	file_base;
 	struct perf_data_file	*file;
 	struct perf_evlist	*evlist;
@@ -253,11 +255,10 @@ out:
 	return rc;
 }
 
-static int process_buildids(struct perf_record *rec)
+static int process_buildids(struct perf_session *session)
 {
-	struct perf_session *session = rec->session;
-	u64 data_offset              = PERF_FILE_HEADER__DATA_OFFSET;
-	u64 size                     = session->header.data_size;
+	u64 data_offset = PERF_FILE_HEADER__DATA_OFFSET;
+	u64 size        = session->header.data_size;
 
 	if (size == 0)
 		return 0;
@@ -267,6 +268,19 @@ static int process_buildids(struct perf_record *rec)
 					      &build_id__mark_dso_hit_ops);
 }
 
+static int file_finish(struct perf_record *rec,
+		       struct perf_data_file *file,
+		       struct perf_session *session,
+		       u64 bytes_written)
+{
+	session->header.data_size = bytes_written;
+
+	if (!rec->no_buildid)
+		process_buildids(session);
+
+	return perf_session__write_header(session, session->evlist, file->fd);
+}
+
 static void perf_record__exit(int status, void *arg)
 {
 	struct perf_record *rec = arg;
@@ -276,12 +290,8 @@ static void perf_record__exit(int status, void *arg)
 		return;
 
 	if (!file->is_pipe) {
-		rec->session->header.data_size += rec->bytes_written;
-
-		if (!rec->no_buildid)
-			process_buildids(rec);
-		perf_session__write_header(rec->session, rec->evlist,
-					   file->fd);
+		file_finish(rec, rec->file, rec->session,
+			    rec->bytes_written);
 		perf_session__delete(rec->session);
 		perf_evlist__delete(rec->evlist);
 		symbol__exit();
@@ -410,6 +420,98 @@ static int synthesize_record(struct perf_record *rec)
 	return err ? err : synthesize_record_file(rec);
 }
 
+static const char *multi_file_base(struct perf_data_file *file)
+{
+	static const char *base;
+
+	if (!base)
+		base = file->path;
+	if (!base)
+		base = "perf";
+
+	return base;
+}
+
+static int multi_file_name(struct perf_data_file *file, unsigned int idx)
+{
+	char path[PATH_MAX];
+
+	snprintf(path, PATH_MAX, "%s-%05u.data",
+		 multi_file_base(file), idx);
+	file->path = strdup(path);
+
+	return file->path ? 0 : -ENOMEM;
+}
+
+static int multi_file_finish(struct perf_record *rec)
+{
+	struct perf_data_file *file = rec->file;
+	struct perf_session *session;
+	int err;
+
+	/* TODO create perf_session__dup(session) */
+	session = perf_session__new(NULL, false, NULL);
+	if (!session)
+		return -ENOMEM;
+
+	session->evlist = rec->evlist;
+	session->file   = file;
+	session->header = rec->session->header;
+
+	err = file_finish(rec, file, session, rec->bytes_written);
+	if (!err)
+		pr_debug("multi: written file %s [%s]\n",
+			 file->path, err ? "failed" : "ok");
+
+	perf_session__delete(session);
+	return err;
+}
+
+static int multi_file_init(struct perf_record *rec)
+{
+	struct perf_data_file *file = rec->file;
+	int err;
+
+	if (multi_file_name(rec->file, rec->multi_idx++))
+		return -ENOMEM;
+
+	err = perf_data_file__open(file);
+	if (err)
+		return err;
+
+	err = perf_session__prepare_header(file->fd);
+	if (err)
+		goto out_close;
+
+	err = synthesize_record_file(rec);
+	if (err)
+		goto out_close;
+
+	return 0;
+
+ out_close:
+	perf_data_file__close(file);
+	return err;
+}
+
+static int multi_file_threshold(struct perf_record *rec)
+{
+	unsigned int limit = rec->opts.data_size_limit;
+	int err;
+
+	if (!limit || rec->bytes_written < limit)
+		return 0;
+
+	pr_debug("multi: file limit crossed %lu B\n", rec->bytes_written);
+
+	err = multi_file_finish(rec);
+
+	rec->multi_bytes_written += rec->bytes_written;
+	rec->bytes_written = 0;
+
+	return err ? err : multi_file_init(rec);
+}
+
 static struct perf_event_header finished_round_event = {
 	.size = sizeof(struct perf_event_header),
 	.type = PERF_RECORD_FINISHED_ROUND,
@@ -427,6 +529,9 @@ static int perf_record__mmap_read_all(struct perf_record *rec)
 				goto out;
 			}
 		}
+
+		if (multi_file_threshold(rec))
+			return -1;
 	}
 
 	if (perf_header__has_feat(&rec->session->header, HEADER_TRACING_DATA))
@@ -437,6 +542,28 @@ out:
 	return rc;
 }
 
+static void display_exit_msg(struct perf_record *rec, unsigned long waking)
+{
+	struct perf_data_file *file = rec->file;
+	bool multi = rec->opts.data_size_limit > 0;
+	char buf[PATH_MAX];
+	u64  bytes = multi ? rec->multi_bytes_written : rec->bytes_written;
+	char *path = multi ? buf : (char *) file->path;
+
+	if (multi)
+		snprintf(path, PATH_MAX, "%s-[0-%u].data",
+			 multi_file_base(file), rec->multi_idx);
+
+	fprintf(stderr, "[ perf record: Woken up %ld times to write data ]\n", waking);
+
+	/*
+	 * Approximate RIP event size: 24 bytes.
+	 */
+	fprintf(stderr,
+		"[ perf record: Captured and wrote %.3f MB %s(~%" PRIu64 " samples) ]\n",
+		(double) bytes / 1024.0 / 1024.0, path, bytes / 24);
+}
+
 static int __cmd_record(struct perf_record *rec, int argc, const char **argv)
 {
 	int err, feat;
@@ -458,6 +585,12 @@ static int __cmd_record(struct perf_record *rec, int argc, const char **argv)
 	signal(SIGUSR1, sig_handler);
 	signal(SIGTERM, sig_handler);
 
+	if (rec->opts.data_size_limit &&
+	    multi_file_name(file, rec->multi_idx++)) {
+		pr_err("Not enough memory\n");
+		return -1;
+	}
+
 	session = perf_session__new(file, false, NULL);
 	if (session == NULL) {
 		pr_err("Not enough memory for reading perf file header\n");
@@ -577,17 +710,7 @@ static int __cmd_record(struct perf_record *rec, int argc, const char **argv)
 	if (quiet || signr == SIGUSR1)
 		return 0;
 
-	fprintf(stderr, "[ perf record: Woken up %ld times to write data ]\n", waking);
-
-	/*
-	 * Approximate RIP event size: 24 bytes.
-	 */
-	fprintf(stderr,
-		"[ perf record: Captured and wrote %.3f MB %s (~%" PRIu64 " samples) ]\n",
-		(double)rec->bytes_written / 1024.0 / 1024.0,
-		file->path,
-		rec->bytes_written / 24);
-
+	display_exit_msg(rec, waking);
 	return 0;
 
 out_delete_session:
@@ -852,6 +975,8 @@ const struct option record_options[] = {
 	OPT_UINTEGER('F', "freq", &record.opts.user_freq, "profile at this frequency"),
 	OPT_UINTEGER('m', "mmap-pages", &record.opts.mmap_pages,
 		     "number of mmap data pages"),
+	OPT_UINTEGER('M', "multi", &record.opts.data_size_limit,
+		     "split data into more data files"),
 	OPT_BOOLEAN(0, "group", &record.opts.group,
 		    "put the counters into a counter group"),
 	OPT_CALLBACK_DEFAULT('g', "call-graph", &record.opts,
diff --git a/tools/perf/perf.h b/tools/perf/perf.h
index 95b2903..d06331f 100644
--- a/tools/perf/perf.h
+++ b/tools/perf/perf.h
@@ -222,6 +222,7 @@ struct perf_record_opts {
 	bool	     period;
 	unsigned int freq;
 	unsigned int mmap_pages;
+	unsigned int data_size_limit;
 	unsigned int user_freq;
 	u64          branch_stack;
 	u64	     default_interval;
diff --git a/tools/perf/util/evsel.c b/tools/perf/util/evsel.c
index a635461..05408e5 100644
--- a/tools/perf/util/evsel.c
+++ b/tools/perf/util/evsel.c
@@ -566,6 +566,11 @@ void perf_evsel__config(struct perf_evsel *evsel,
 		attr->branch_sample_type = opts->branch_stack;
 	}
 
+	if (opts->data_size_limit) {
+		attr->watermark = 1;
+		attr->wakeup_watermark = opts->data_size_limit / 4;
+	}
+
 	if (opts->sample_weight)
 		attr->sample_type	|= PERF_SAMPLE_WEIGHT;
 
-- 
1.7.11.7

[PATCH 23/23] perf tools: Have the process properly sythesized in subsequent data files

[Jiri Olsa]

Change the target so the sythesizing routine would think
it needs to synthesized whole process, because there's no
exec COMM event for subsequent data files.

Signed-off-by: Jiri Olsa <jolsa@redhat.com>
Cc: Corey Ashford <cjashfor@linux.vnet.ibm.com>
Cc: Frederic Weisbecker <fweisbec@gmail.com>
Cc: Ingo Molnar <mingo@elte.hu>
Cc: Namhyung Kim <namhyung@kernel.org>
Cc: Paul Mackerras <paulus@samba.org>
Cc: Peter Zijlstra <a.p.zijlstra@chello.nl>
Cc: Arnaldo Carvalho de Melo <acme@redhat.com>
Cc: Andi Kleen <ak@linux.intel.com>
Cc: David Ahern <dsahern@gmail.com>
---
 tools/perf/builtin-record.c | 11 +++++++++++
 1 file changed, 11 insertions(+)

diff --git a/tools/perf/builtin-record.c b/tools/perf/builtin-record.c
index 920d69a..3740c5e 100644
--- a/tools/perf/builtin-record.c
+++ b/tools/perf/builtin-record.c
@@ -470,6 +470,8 @@ static int multi_file_finish(struct perf_record *rec)
 static int multi_file_init(struct perf_record *rec)
 {
 	struct perf_data_file *file = rec->file;
+	struct perf_record_opts *opts = &rec->opts;
+	static bool target_change;
 	int err;
 
 	if (multi_file_name(rec->file, rec->multi_idx++))
@@ -483,6 +485,15 @@ static int multi_file_init(struct perf_record *rec)
 	if (err)
 		goto out_close;
 
+	/*
+	 * XXX HACK - Make the target looks like we have pid
+	 * so we get the process fully synthetised.
+	 */
+	if (perf_target__none(&opts->target) && !target_change) {
+		target_change = true;
+		opts->target.pid = "multi";
+	}
+
 	err = synthesize_record_file(rec);
 	if (err)
 		goto out_close;
-- 
1.7.11.7

Re: [PATCH 06/23] perf doc: Add perf data file documentation

[Arnaldo Carvalho de Melo]

Em Wed, Jul 17, 2013 at 07:49:46PM +0200, Jiri Olsa escreveu:
> Adding perf data file documentation.
> 
> TODO: Finish the FEATURES section
> 
> Signed-off-by: Jiri Olsa <jolsa@redhat.com>

    ASCIIDOC /tmp/build/perf/perf-evlist.xml
asciidoc: ERROR: perf-data-file-v2.txt: line 5: name section expected
asciidoc: FAILED: perf-data-file-v2.txt: line 5: section title expected
make[2]: *** [/tmp/build/perf/perf-data-file-v2.xml] Error 1
make[2]: *** Waiting for unfinished jobs....
make[2]: Leaving directory `/home/git/linux/tools/perf/Documentation'
make[1]: *** [try-install-man] Error 2
make[1]: Leaving directory `/home/git/linux/tools/perf/Documentation'
make: *** [try-install-man] Error 2
make: Leaving directory `/home/git/linux/tools/perf'
[acme@zoo linux]$ 

Re: [PATCH 02/23] perf tools: Remove data_offset seek as it's not needed

[Namhyung Kim]

On Wed, 17 Jul 2013 19:49:42 +0200, Jiri Olsa wrote:
> Removing data_offset seek as it's not needed, because
> data are not read by syscall but mmaped instead.

Yes we do, but it's not mandatory, right?  I'd rather expect the file
offset remains at the beginning of data after calling
perf_session__read_header() instead of a random point (EOF?).

Thanks,
Namhyung

Re: [PATCH 05/23] perf tests: Add simple session read/write test

[Namhyung Kim]

On Wed, 17 Jul 2013 19:49:45 +0200, Jiri Olsa wrote:
> Adding simple session read/write test to keep up
> with file format changes.
>
[SNIP]

> +	session = perf_session__new(file, O_RDONLY, false, false, &tool);
> +	TEST_ASSERT_VAL("failed to create session", session);
> +
> +	TEST_ASSERT_VAL("failed to process events", session);
> +		perf_session__process_events(session, &tool);

Did you mean

	TEST_ASSERT_VAL("failed to process events",
		perf_session__process_events(session, &tool));
?


Thanks,
Namhyung

Re: [PATCH 08/23] perf tools: Introduce perf data file version CHECK macro

[Jiri Olsa]

On Wed, Jul 17, 2013 at 04:42:50PM -0300, Arnaldo Carvalho de Melo wrote:
> Em Wed, Jul 17, 2013 at 07:49:48PM +0200, Jiri Olsa escreveu:
> > Introducing perf data file version CHECK macro, so the check
> > is easily extensible for new versions.
> > 
> > Signed-off-by: Jiri Olsa <jolsa@redhat.com>
> > Cc: Corey Ashford <cjashfor@linux.vnet.ibm.com>
> > Cc: Frederic Weisbecker <fweisbec@gmail.com>
> > Cc: Ingo Molnar <mingo@elte.hu>
> > Cc: Namhyung Kim <namhyung@kernel.org>
> > Cc: Paul Mackerras <paulus@samba.org>
> > Cc: Peter Zijlstra <a.p.zijlstra@chello.nl>
> > Cc: Arnaldo Carvalho de Melo <acme@redhat.com>
> > Cc: Andi Kleen <ak@linux.intel.com>
> > Cc: David Ahern <dsahern@gmail.com>
> > ---
> >  tools/perf/util/header.c | 27 ++++++++++++++++-----------
> >  1 file changed, 16 insertions(+), 11 deletions(-)
> > 
> > diff --git a/tools/perf/util/header.c b/tools/perf/util/header.c
> > index f558f83..4ebe4a1 100644
> > --- a/tools/perf/util/header.c
> > +++ b/tools/perf/util/header.c
> > @@ -2493,18 +2493,23 @@ static int check_magic_endian(u64 magic, uint64_t hdr_sz,
> >  	 * - encode endianness of file
> >  	 */
> >  
> > -	/* check magic number with one endianness */
> > -	if (magic == __perf_magic2)
> > -		return 0;
> > -
> > -	/* check magic number with opposite endianness */
> > -	if (magic != __perf_magic2_sw)
> > -		return -1;
> > +#define CHECK(__m, __v)						\
> > +do {								\
> > +	/* check magic number with one endianness */		\
> > +	if (magic == __m)					\
> > +		goto v ## __v;					\
> > +	/* check magic number with opposite endianness */	\
> > +	if (magic != __m ## _sw)				\
> > +		break;						\
> > +	ph->needs_swap = true;					\
> > + v ## __v:							\
> > +	ph->version = __v;					\
> > +	return 0;						\
> > +} while (0)
> 
> Ouch, why not:
> 
> #define PERF_CHECK_VERSION(__m, __v)
> 	/* check magic number with one endianness */		\
> 	if (magic == __m) {					\
> 		ph->version = __v;				\
> 		return 0;					\
> 	/* check magic number with opposite endianness */	\
> 	} else if (magic == __m ## _sw) {			\
> 		ph->needs_swap = true;				\
> 	}

ok, will change

thanks,
jirka

Re: [PATCH 06/23] perf doc: Add perf data file documentation

[Jiri Olsa]

On Wed, Jul 17, 2013 at 04:59:05PM -0300, Arnaldo Carvalho de Melo wrote:
> Em Wed, Jul 17, 2013 at 07:49:46PM +0200, Jiri Olsa escreveu:
> > Adding perf data file documentation.
> > 
> > TODO: Finish the FEATURES section
> > 
> > Signed-off-by: Jiri Olsa <jolsa@redhat.com>
> 
>     ASCIIDOC /tmp/build/perf/perf-evlist.xml
> asciidoc: ERROR: perf-data-file-v2.txt: line 5: name section expected
> asciidoc: FAILED: perf-data-file-v2.txt: line 5: section title expected

hum, I did not know there were some rules for doc :-) will fix

thanks,
jirka

> make[2]: *** [/tmp/build/perf/perf-data-file-v2.xml] Error 1
> make[2]: *** Waiting for unfinished jobs....
> make[2]: Leaving directory `/home/git/linux/tools/perf/Documentation'
> make[1]: *** [try-install-man] Error 2
> make[1]: Leaving directory `/home/git/linux/tools/perf/Documentation'
> make: *** [try-install-man] Error 2
> make: Leaving directory `/home/git/linux/tools/perf'
> [acme@zoo linux]$ 
> 

Re: [PATCH 02/23] perf tools: Remove data_offset seek as it's not needed

[Jiri Olsa]

On Thu, Jul 18, 2013 at 04:27:47PM +0900, Namhyung Kim wrote:
> On Wed, 17 Jul 2013 19:49:42 +0200, Jiri Olsa wrote:
> > Removing data_offset seek as it's not needed, because
> > data are not read by syscall but mmaped instead.
> 
> Yes we do, but it's not mandatory, right?  I'd rather expect the file
> offset remains at the beginning of data after calling
> perf_session__read_header() instead of a random point (EOF?).

well it's just useless, and I think it's better and more clear
to set it properly before the reading anyway. Also AFAICS there's
no other read after this seek even later.

jirka

Re: [PATCH 05/23] perf tests: Add simple session read/write test

[Jiri Olsa]

On Wed, Jul 17, 2013 at 04:33:13PM -0300, Arnaldo Carvalho de Melo wrote:
> Em Wed, Jul 17, 2013 at 07:49:45PM +0200, Jiri Olsa escreveu:
> > +static struct perf_evlist *get_evlist(void)
> > +{
> > +	int err;
> > +	struct perf_evlist *evlist = perf_evlist__new();
> > +	if (evlist == NULL) {
> > +		pr_err("perf_evlist__new failed\n");
> > +		return NULL;
> > +	}
> > +	err = perf_evlist__add_default(evlist);
> > +	if (err < 0) {
> > +		pr_err("Not enough memory to create evsel\n");
> > +		perf_evlist__delete(evlist);
> > +		evlist = NULL;
> > +	}
> > +	return evlist;
> > +}
> 
> I think the above can be such a common idiom that having a new
> constructor for it is doable, i.e.:
> 
> 	evlist = perf_evlist__new_default(void);
> 	if (evlist == NULL)
> 		pr_debug, etc

ook

> 
> > +
> > +static union perf_event *get_event_MMAP(void)
> > +{
> > +	static union perf_event event;
> > +	size_t size;
> > +
> > +	size = snprintf(event.mmap.filename, sizeof(event.mmap.filename),
> > +			"krava") + 1;
> > +	size = PERF_ALIGN(size, sizeof(u64));
> > +
> > +	event.header.type = PERF_RECORD_MMAP;
> > +	event.header.misc = PERF_RECORD_MISC_KERNEL;
> > +	event.header.size = sizeof(event.mmap) -
> > +			    (sizeof(event.mmap.filename) - size);
> > +
> > +	event.mmap.pgoff = 10;
> > +	event.mmap.start = 0;
> > +	event.mmap.len   = 10;
> > +	event.mmap.pid   = 123;
> > +
> > +	return &event;
> > +}
> 
> Don't use statics, here perhaps we can have another general purpose
> helper:
> 
> 	void perf_event__init_mmap(union perf_event *event)
> 
> We then could use it on the synthesizing routines, etc.

ook

> 

SNIP

> > +
> > +static int store_event(int fd, union perf_event *event, size_t *size)
> > +{
> > +	*size += event->header.size;
> > +	return write(fd, event, event->header.size) > 0 ? 0 : -1;
> > +}
> 
> 	I think we need a writen like we have readn and then do it like:
> 
> 	return writen(fd, event, event->header.size) == event->header.size ? 0 : -1;

hum, we could generalize write_output function from builtin-record

> 
> ?
> 

SNIP

> > +
> > +	TEST_ASSERT_VAL("failed to write header",
> > +		!perf_session__write_header(session, evlist, fd, true));
> 
> Humm, can't perf_session__write_header use session->fd? Just like you
> did for some other routines in previous patches in this series?

right, but need to check if the builtin-inject caller uses the same fd

thanks,
jirka

Re: [PATCH 05/23] perf tests: Add simple session read/write test

[Jiri Olsa]

On Thu, Jul 18, 2013 at 04:46:03PM +0900, Namhyung Kim wrote:
> On Wed, 17 Jul 2013 19:49:45 +0200, Jiri Olsa wrote:
> > Adding simple session read/write test to keep up
> > with file format changes.
> >
> [SNIP]
> 
> > +	session = perf_session__new(file, O_RDONLY, false, false, &tool);
> > +	TEST_ASSERT_VAL("failed to create session", session);
> > +
> > +	TEST_ASSERT_VAL("failed to process events", session);
> > +		perf_session__process_events(session, &tool);
> 
> Did you mean
> 
> 	TEST_ASSERT_VAL("failed to process events",
> 		perf_session__process_events(session, &tool));
> ?

ouch, yes I did ;-)

thanks,
jirka

Re: [PATCH 05/23] perf tests: Add simple session read/write test

[Jiri Olsa]

On Wed, Jul 17, 2013 at 04:33:13PM -0300, Arnaldo Carvalho de Melo wrote:

SNIP

> > +
> > +static union perf_event *get_event_MMAP(void)
> > +{
> > +	static union perf_event event;
> > +	size_t size;
> > +
> > +	size = snprintf(event.mmap.filename, sizeof(event.mmap.filename),
> > +			"krava") + 1;
> > +	size = PERF_ALIGN(size, sizeof(u64));
> > +
> > +	event.header.type = PERF_RECORD_MMAP;
> > +	event.header.misc = PERF_RECORD_MISC_KERNEL;
> > +	event.header.size = sizeof(event.mmap) -
> > +			    (sizeof(event.mmap.filename) - size);
> > +
> > +	event.mmap.pgoff = 10;
> > +	event.mmap.start = 0;
> > +	event.mmap.len   = 10;
> > +	event.mmap.pid   = 123;
> > +
> > +	return &event;
> > +}
> 
> Don't use statics, here perhaps we can have another general purpose
> helper:
> 
> 	void perf_event__init_mmap(union perf_event *event)

well, the thing is that it makes the test simple, I can use it like:

	TEST_ASSERT_VAL("failed to store MMAP event",
			!store_event(fd, get_event_MMAP(), &size));

no need to call extra functions, just get the single event

jirka

[tip:perf/core] perf session: Use session-> fd instead of passing fd as argument

[tip-bot for Jiri Olsa]

Commit-ID:  d4339569a6f8bb1e703337175dcd4fd4bcd5d891
Gitweb:     http://git.kernel.org/tip/d4339569a6f8bb1e703337175dcd4fd4bcd5d891
Author:     Jiri Olsa <jolsa@redhat.com>
AuthorDate: Wed, 17 Jul 2013 19:49:41 +0200
Committer:  Arnaldo Carvalho de Melo <acme@redhat.com>
CommitDate: Wed, 17 Jul 2013 16:46:07 -0300

perf session: Use session->fd instead of passing fd as argument

Using session->fd instead of passing fd as argument because it's always
session->fd that's passed as fd argument.

Signed-off-by: Jiri Olsa <jolsa@redhat.com>
Cc: Andi Kleen <ak@linux.intel.com>
Cc: Corey Ashford <cjashfor@linux.vnet.ibm.com>
Cc: David Ahern <dsahern@gmail.com>
Cc: Frederic Weisbecker <fweisbec@gmail.com>
Cc: Ingo Molnar <mingo@elte.hu>
Cc: Namhyung Kim <namhyung@kernel.org>
Cc: Paul Mackerras <paulus@samba.org>
Cc: Peter Zijlstra <a.p.zijlstra@chello.nl>
Link: http://lkml.kernel.org/r/1374083403-14591-2-git-send-email-jolsa@redhat.com
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
---
 tools/perf/util/header.c  | 11 +++++------
 tools/perf/util/header.h  |  2 +-
 tools/perf/util/session.c |  4 ++--
 3 files changed, 8 insertions(+), 9 deletions(-)

diff --git a/tools/perf/util/header.c b/tools/perf/util/header.c
index b28a65e..b3f253f 100644
--- a/tools/perf/util/header.c
+++ b/tools/perf/util/header.c
@@ -2619,19 +2619,17 @@ static int perf_file_header__read_pipe(struct perf_pipe_file_header *header,
 	return 0;
 }
 
-static int perf_header__read_pipe(struct perf_session *session, int fd)
+static int perf_header__read_pipe(struct perf_session *session)
 {
 	struct perf_header *header = &session->header;
 	struct perf_pipe_file_header f_header;
 
-	if (perf_file_header__read_pipe(&f_header, header, fd,
+	if (perf_file_header__read_pipe(&f_header, header, session->fd,
 					session->repipe) < 0) {
 		pr_debug("incompatible file format\n");
 		return -EINVAL;
 	}
 
-	session->fd = fd;
-
 	return 0;
 }
 
@@ -2725,20 +2723,21 @@ static int perf_evlist__prepare_tracepoint_events(struct perf_evlist *evlist,
 	return 0;
 }
 
-int perf_session__read_header(struct perf_session *session, int fd)
+int perf_session__read_header(struct perf_session *session)
 {
 	struct perf_header *header = &session->header;
 	struct perf_file_header	f_header;
 	struct perf_file_attr	f_attr;
 	u64			f_id;
 	int nr_attrs, nr_ids, i, j;
+	int fd = session->fd;
 
 	session->evlist = perf_evlist__new();
 	if (session->evlist == NULL)
 		return -ENOMEM;
 
 	if (session->fd_pipe)
-		return perf_header__read_pipe(session, fd);
+		return perf_header__read_pipe(session);
 
 	if (perf_file_header__read(&f_header, header, fd) < 0)
 		return -EINVAL;
diff --git a/tools/perf/util/header.h b/tools/perf/util/header.h
index 669fda5..0bacb2d 100644
--- a/tools/perf/util/header.h
+++ b/tools/perf/util/header.h
@@ -96,7 +96,7 @@ struct perf_header {
 struct perf_evlist;
 struct perf_session;
 
-int perf_session__read_header(struct perf_session *session, int fd);
+int perf_session__read_header(struct perf_session *session);
 int perf_session__write_header(struct perf_session *session,
 			       struct perf_evlist *evlist,
 			       int fd, bool at_exit);
diff --git a/tools/perf/util/session.c b/tools/perf/util/session.c
index d0d9f94..272c9cf 100644
--- a/tools/perf/util/session.c
+++ b/tools/perf/util/session.c
@@ -24,7 +24,7 @@ static int perf_session__open(struct perf_session *self, bool force)
 		self->fd_pipe = true;
 		self->fd = STDIN_FILENO;
 
-		if (perf_session__read_header(self, self->fd) < 0)
+		if (perf_session__read_header(self) < 0)
 			pr_err("incompatible file format (rerun with -v to learn more)");
 
 		return 0;
@@ -56,7 +56,7 @@ static int perf_session__open(struct perf_session *self, bool force)
 		goto out_close;
 	}
 
-	if (perf_session__read_header(self, self->fd) < 0) {
+	if (perf_session__read_header(self) < 0) {
 		pr_err("incompatible file format (rerun with -v to learn more)");
 		goto out_close;
 	}

[tip:perf/core] perf header: Remove data_offset seek as it' s not needed

[tip-bot for Jiri Olsa]

Commit-ID:  75211dbfabf1de3a42703f4d3cd6f6e047289d40
Gitweb:     http://git.kernel.org/tip/75211dbfabf1de3a42703f4d3cd6f6e047289d40
Author:     Jiri Olsa <jolsa@redhat.com>
AuthorDate: Wed, 17 Jul 2013 19:49:42 +0200
Committer:  Arnaldo Carvalho de Melo <acme@redhat.com>
CommitDate: Wed, 17 Jul 2013 16:46:17 -0300

perf header: Remove data_offset seek as it's not needed

Removing data_offset seek as it's not needed, because data are not read
by syscall but mmaped instead.

Signed-off-by: Jiri Olsa <jolsa@redhat.com>
Cc: Andi Kleen <ak@linux.intel.com>
Cc: Corey Ashford <cjashfor@linux.vnet.ibm.com>
Cc: David Ahern <dsahern@gmail.com>
Cc: Frederic Weisbecker <fweisbec@gmail.com>
Cc: Ingo Molnar <mingo@elte.hu>
Cc: Namhyung Kim <namhyung@kernel.org>
Cc: Paul Mackerras <paulus@samba.org>
Cc: Peter Zijlstra <a.p.zijlstra@chello.nl>
Link: http://lkml.kernel.org/r/1374083403-14591-3-git-send-email-jolsa@redhat.com
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
---
 tools/perf/util/header.c | 2 --
 1 file changed, 2 deletions(-)

diff --git a/tools/perf/util/header.c b/tools/perf/util/header.c
index b3f253f..035efe7 100644
--- a/tools/perf/util/header.c
+++ b/tools/perf/util/header.c
@@ -2794,8 +2794,6 @@ int perf_session__read_header(struct perf_session *session)
 	perf_header__process_sections(header, fd, &session->pevent,
 				      perf_file_section__process);
 
-	lseek(fd, header->data_offset, SEEK_SET);
-
 	if (perf_evlist__prepare_tracepoint_events(session->evlist,
 						   session->pevent))
 		goto out_delete_evlist;

[tip:perf/core] perf header: Remove attr_offset from perf_header

[tip-bot for Jiri Olsa]

Commit-ID:  944d62ba4c8e1c9bb2962a70e3d495106fae8cd9
Gitweb:     http://git.kernel.org/tip/944d62ba4c8e1c9bb2962a70e3d495106fae8cd9
Author:     Jiri Olsa <jolsa@redhat.com>
AuthorDate: Wed, 17 Jul 2013 19:49:43 +0200
Committer:  Arnaldo Carvalho de Melo <acme@redhat.com>
CommitDate: Wed, 17 Jul 2013 16:46:25 -0300

perf header: Remove attr_offset from perf_header

Removing attr_offset from perf_header as it's possible to use it as a
local variable.

Signed-off-by: Jiri Olsa <jolsa@redhat.com>
Cc: Andi Kleen <ak@linux.intel.com>
Cc: Corey Ashford <cjashfor@linux.vnet.ibm.com>
Cc: David Ahern <dsahern@gmail.com>
Cc: Frederic Weisbecker <fweisbec@gmail.com>
Cc: Ingo Molnar <mingo@elte.hu>
Cc: Namhyung Kim <namhyung@kernel.org>
Cc: Paul Mackerras <paulus@samba.org>
Cc: Peter Zijlstra <a.p.zijlstra@chello.nl>
Link: http://lkml.kernel.org/r/1374083403-14591-4-git-send-email-jolsa@redhat.com
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
---
 tools/perf/util/header.c | 5 +++--
 tools/perf/util/header.h | 1 -
 2 files changed, 3 insertions(+), 3 deletions(-)

diff --git a/tools/perf/util/header.c b/tools/perf/util/header.c
index 035efe7..dcba56a 100644
--- a/tools/perf/util/header.c
+++ b/tools/perf/util/header.c
@@ -2272,6 +2272,7 @@ int perf_session__write_header(struct perf_session *session,
 	struct perf_file_attr   f_attr;
 	struct perf_header *header = &session->header;
 	struct perf_evsel *evsel;
+	u64 attr_offset;
 	int err;
 
 	lseek(fd, sizeof(f_header), SEEK_SET);
@@ -2285,7 +2286,7 @@ int perf_session__write_header(struct perf_session *session,
 		}
 	}
 
-	header->attr_offset = lseek(fd, 0, SEEK_CUR);
+	attr_offset = lseek(fd, 0, SEEK_CUR);
 
 	list_for_each_entry(evsel, &evlist->entries, node) {
 		f_attr = (struct perf_file_attr){
@@ -2315,7 +2316,7 @@ int perf_session__write_header(struct perf_session *session,
 		.size	   = sizeof(f_header),
 		.attr_size = sizeof(f_attr),
 		.attrs = {
-			.offset = header->attr_offset,
+			.offset = attr_offset,
 			.size   = evlist->nr_entries * sizeof(f_attr),
 		},
 		.data = {
diff --git a/tools/perf/util/header.h b/tools/perf/util/header.h
index 0bacb2d..6fa80f9 100644
--- a/tools/perf/util/header.h
+++ b/tools/perf/util/header.h
@@ -86,7 +86,6 @@ struct perf_session_env {
 
 struct perf_header {
 	bool			needs_swap;
-	s64			attr_offset;
 	u64			data_offset;
 	u64			data_size;
 	DECLARE_BITMAP(adds_features, HEADER_FEAT_BITS);

[tip:perf/core] perf header: Introduce feat_offset into perf_header

[tip-bot for Jiri Olsa]

Commit-ID:  8d541e974f4b3ed5db7e278b4d64481af34910a9
Gitweb:     http://git.kernel.org/tip/8d541e974f4b3ed5db7e278b4d64481af34910a9
Author:     Jiri Olsa <jolsa@redhat.com>
AuthorDate: Wed, 17 Jul 2013 19:49:44 +0200
Committer:  Arnaldo Carvalho de Melo <acme@redhat.com>
CommitDate: Wed, 17 Jul 2013 16:46:32 -0300

perf header: Introduce feat_offset into perf_header

Introducing feat_offset into perf_header to make the location of the
features section clear.

Signed-off-by: Jiri Olsa <jolsa@redhat.com>
Cc: Andi Kleen <ak@linux.intel.com>
Cc: Corey Ashford <cjashfor@linux.vnet.ibm.com>
Cc: David Ahern <dsahern@gmail.com>
Cc: Frederic Weisbecker <fweisbec@gmail.com>
Cc: Ingo Molnar <mingo@elte.hu>
Cc: Namhyung Kim <namhyung@kernel.org>
Cc: Paul Mackerras <paulus@samba.org>
Cc: Peter Zijlstra <a.p.zijlstra@chello.nl>
Link: http://lkml.kernel.org/r/1374083403-14591-5-git-send-email-jolsa@redhat.com
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
---
 tools/perf/util/header.c | 6 ++++--
 tools/perf/util/header.h | 1 +
 2 files changed, 5 insertions(+), 2 deletions(-)

diff --git a/tools/perf/util/header.c b/tools/perf/util/header.c
index dcba56a..eefb052 100644
--- a/tools/perf/util/header.c
+++ b/tools/perf/util/header.c
@@ -2225,7 +2225,7 @@ static int perf_header__adds_write(struct perf_header *header,
 
 	sec_size = sizeof(*feat_sec) * nr_sections;
 
-	sec_start = header->data_offset + header->data_size;
+	sec_start = header->feat_offset;
 	lseek(fd, sec_start + sec_size, SEEK_SET);
 
 	for_each_set_bit(feat, header->adds_features, HEADER_FEAT_BITS) {
@@ -2304,6 +2304,7 @@ int perf_session__write_header(struct perf_session *session,
 	}
 
 	header->data_offset = lseek(fd, 0, SEEK_CUR);
+	header->feat_offset = header->data_offset + header->data_size;
 
 	if (at_exit) {
 		err = perf_header__adds_write(header, evlist, fd);
@@ -2373,7 +2374,7 @@ int perf_header__process_sections(struct perf_header *header, int fd,
 
 	sec_size = sizeof(*feat_sec) * nr_sections;
 
-	lseek(fd, header->data_offset + header->data_size, SEEK_SET);
+	lseek(fd, header->feat_offset, SEEK_SET);
 
 	err = perf_header__getbuffer64(header, fd, feat_sec, sec_size);
 	if (err < 0)
@@ -2572,6 +2573,7 @@ int perf_file_header__read(struct perf_file_header *header,
 
 	ph->data_offset  = header->data.offset;
 	ph->data_size	 = header->data.size;
+	ph->feat_offset  = header->data.offset + header->data.size;
 	return 0;
 }
 
diff --git a/tools/perf/util/header.h b/tools/perf/util/header.h
index 6fa80f9..e6387dc 100644
--- a/tools/perf/util/header.h
+++ b/tools/perf/util/header.h
@@ -88,6 +88,7 @@ struct perf_header {
 	bool			needs_swap;
 	u64			data_offset;
 	u64			data_size;
+	u64			feat_offset;
 	DECLARE_BITMAP(adds_features, HEADER_FEAT_BITS);
 	struct perf_session_env env;
 };

[tip:perf/core] perf header: Recognize version number for perf data file

[tip-bot for Jiri Olsa]

Commit-ID:  2a08c3ec4f7d6058a450d2d4bc6e366955872707
Gitweb:     http://git.kernel.org/tip/2a08c3ec4f7d6058a450d2d4bc6e366955872707
Author:     Jiri Olsa <jolsa@redhat.com>
AuthorDate: Wed, 17 Jul 2013 19:49:47 +0200
Committer:  Arnaldo Carvalho de Melo <acme@redhat.com>
CommitDate: Wed, 17 Jul 2013 17:04:00 -0300

perf header: Recognize version number for perf data file

Keep the recognized data file version within 'struct perf_header'.

Signed-off-by: Jiri Olsa <jolsa@redhat.com>
Cc: Andi Kleen <ak@linux.intel.com>
Cc: Corey Ashford <cjashfor@linux.vnet.ibm.com>
Cc: David Ahern <dsahern@gmail.com>
Cc: Frederic Weisbecker <fweisbec@gmail.com>
Cc: Ingo Molnar <mingo@elte.hu>
Cc: Namhyung Kim <namhyung@kernel.org>
Cc: Paul Mackerras <paulus@samba.org>
Cc: Peter Zijlstra <a.p.zijlstra@chello.nl>
Link: http://lkml.kernel.org/r/1374083403-14591-8-git-send-email-jolsa@redhat.com
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
---
 tools/perf/util/header.c |  2 ++
 tools/perf/util/header.h | 16 +++++++++++-----
 2 files changed, 13 insertions(+), 5 deletions(-)

diff --git a/tools/perf/util/header.c b/tools/perf/util/header.c
index eefb052..f558f83 100644
--- a/tools/perf/util/header.c
+++ b/tools/perf/util/header.c
@@ -2480,6 +2480,7 @@ static int check_magic_endian(u64 magic, uint64_t hdr_sz,
 	/* check for legacy format */
 	ret = memcmp(&magic, __perf_magic1, sizeof(magic));
 	if (ret == 0) {
+		ph->version = PERF_HEADER_VERSION_1;
 		pr_debug("legacy perf.data format\n");
 		if (is_pipe)
 			return try_all_pipe_abis(hdr_sz, ph);
@@ -2501,6 +2502,7 @@ static int check_magic_endian(u64 magic, uint64_t hdr_sz,
 		return -1;
 
 	ph->needs_swap = true;
+	ph->version = PERF_HEADER_VERSION_2;
 
 	return 0;
 }
diff --git a/tools/perf/util/header.h b/tools/perf/util/header.h
index e6387dc..307c9ae 100644
--- a/tools/perf/util/header.h
+++ b/tools/perf/util/header.h
@@ -34,6 +34,11 @@ enum {
 	HEADER_FEAT_BITS	= 256,
 };
 
+enum perf_header_version {
+	PERF_HEADER_VERSION_1,
+	PERF_HEADER_VERSION_2,
+};
+
 struct perf_file_section {
 	u64 offset;
 	u64 size;
@@ -85,12 +90,13 @@ struct perf_session_env {
 };
 
 struct perf_header {
-	bool			needs_swap;
-	u64			data_offset;
-	u64			data_size;
-	u64			feat_offset;
+	enum perf_header_version	version;
+	bool				needs_swap;
+	u64				data_offset;
+	u64				data_size;
+	u64				feat_offset;
 	DECLARE_BITMAP(adds_features, HEADER_FEAT_BITS);
-	struct perf_session_env env;
+	struct perf_session_env 	env;
 };
 
 struct perf_evlist;

Re: [PATCH 10/23] perf tools: Introduce swap_header function

[Namhyung Kim]

On Wed, 17 Jul 2013 19:49:50 +0200, Jiri Olsa wrote:
> Introducing swap_header function to make the swapping
> code more clear and extensible.

[SNIP]
>  
> +static int swap_header(struct perf_file_header *header)
> +{
> +	mem_bswap_64(header, offsetof(struct perf_file_header, adds_features));
> +
> +	if (header->size != sizeof(*header)) {
> +		/* Support the previous format */
> +		if (header->size == offsetof(typeof(*header), adds_features))
> +			bitmap_zero(header->adds_features, HEADER_FEAT_BITS);
> +		else
> +			return -1;
> +	} else
> +		swap_features(header->adds_features);
> +
> +	return 0;
> +}
> +
>  int perf_file_header__read(struct perf_file_header *header,
>  			   struct perf_header *ph, int fd)
>  {
> @@ -2562,19 +2578,8 @@ int perf_file_header__read(struct perf_file_header *header,
>  		return -1;
>  	}
>  
> -	if (ph->needs_swap) {
> -		mem_bswap_64(header, offsetof(struct perf_file_header,
> -			     adds_features));
> -	}
> -
> -	if (header->size != sizeof(*header)) {
> -		/* Support the previous format */
> -		if (header->size == offsetof(typeof(*header), adds_features))
> -			bitmap_zero(header->adds_features, HEADER_FEAT_BITS);
> -		else
> -			return -1;
> -	} else if (ph->needs_swap)
> -		swap_features(header->adds_features);
> +	if (ph->needs_swap && swap_header(header))
> +		return -1;

Doesn't it change existing behavior?  It looks like supporting previous
format on a same endian machines will be removed by this patch.

Thanks,
Namhyung

Re: [PATCH 13/23] perf tools: Introduce perf.data version 3 format

[Namhyung Kim]

On Wed, 17 Jul 2013 19:49:53 +0200, Jiri Olsa wrote:
> Introducing perf.data version 3 format. No functional change,
> just introducing doc, magic bytes and the struct.
>
[SNIP]

> +perf-data-file-v3(1)

s/(1)/(5)/

The same goes to v2 doc.


> +====================
> +Following text describes version 3 of the perf data file format,
> +which is version that is currently used by perf tool.
> +
> +In version 3 we got rid of following sections:
> +  EVENT IDS
> +  EVENT ATTRIBUTES
> +  EVENT TYPES
> +
> +and keep only data and features sections. All the data
> +from above listed sections are now read from specific
> +FEATURES sections.
> +
> +benefits:
> +  - siplified header storage at the time the file is closed

s/siplified/simplified/


> +  - allowing cpu hotpluged addition/removal event IDs
> +    during the session due to 1)

What is 1) ?

Thanks,
Namhyung

Re: [PATCH 15/23] perf tools: Add perf data version 3 header read

[Namhyung Kim]

On Wed, 17 Jul 2013 19:49:55 +0200, Jiri Olsa wrote:
> Adding perf data version 3 header read code to load
> data for v3 format.
>
[SNIP]
> +
> +	session = container_of(header, struct perf_session, header);
> +	if (session->evlist)
> +		return 0;

I guess we need version check here.  The v2 format still needs to
process it even though it's session already has a evlist, right?

Thanks,
Namhyung

>  
>  	evlist = read_event_desc(header, fd);
>  	if (!evlist)
> -		return 0;
> +		return -1;
>  
> -	session = container_of(header, struct perf_session, header);
> +	ph = &session->header;
>  
> -	list_for_each_entry(evsel, &evlist->entries, node)
> -		perf_evlist__set_event_name(session->evlist, evsel);
> +	if (ph->version <= PERF_HEADER_VERSION_2) {
> +		list_for_each_entry(evsel, &evlist->entries, node)
> +			perf_evlist__set_event_name(session->evlist, evsel);
> +
> +		perf_evlist__delete(evlist);
> +	} else {
> +		session->evlist = evlist;
> +		symbol_conf.nr_events = evlist->nr_entries;
> +	}
>  
> -	perf_evlist__delete(evlist);
>  	return 0;
>  }

Re: [PATCH 18/23] perf tools: Move synthetizing into single function

[Namhyung Kim]

On Wed, 17 Jul 2013 19:49:58 +0200, Jiri Olsa wrote:
> Moving synthetizing into single function, so it

s/synthetizing/synthesizing/

The same goes to the subject line too.


> could be reused.
>
> Signed-off-by: Jiri Olsa <jolsa@redhat.com>
> Cc: Corey Ashford <cjashfor@linux.vnet.ibm.com>
> Cc: Frederic Weisbecker <fweisbec@gmail.com>
> Cc: Ingo Molnar <mingo@elte.hu>
> Cc: Namhyung Kim <namhyung@kernel.org>
> Cc: Paul Mackerras <paulus@samba.org>
> Cc: Peter Zijlstra <a.p.zijlstra@chello.nl>
> Cc: Arnaldo Carvalho de Melo <acme@redhat.com>
> Cc: Andi Kleen <ak@linux.intel.com>
> Cc: David Ahern <dsahern@gmail.com>
> ---
>  tools/perf/builtin-record.c | 155 ++++++++++++++++++++++++++------------------
>  1 file changed, 91 insertions(+), 64 deletions(-)
>
> diff --git a/tools/perf/builtin-record.c b/tools/perf/builtin-record.c
> index b67564c..33a5bce 100644
> --- a/tools/perf/builtin-record.c
> +++ b/tools/perf/builtin-record.c
> @@ -317,6 +317,95 @@ static void perf_event__synthesize_guest_os(struct machine *machine, void *data)
>  		       " relocation symbol.\n", machine->pid);
>  }
>  
> +static int synthesize_record_pipe(struct perf_record *rec)
> +{
> +	struct perf_session  *session = rec->session;
> +	struct perf_tool        *tool = &rec->tool;
> +	struct perf_evlist    *evlist = rec->evlist;
> +	int err;
> +
> +	err = perf_event__synthesize_attrs(tool, session,
> +					   process_synthesized_event);
> +	if (err < 0) {
> +		pr_err("Couldn't synthesize attrs.\n");
> +		return err;
> +	}
> +
> +	if (have_tracepoints(&evlist->entries)) {
> +		/*
> +		 * FIXME err <= 0 here actually means that
> +		 * there were no tracepoints so its not really
> +		 * an error, just that we don't need to
> +		 * synthesize anything.  We really have to
> +		 * return this more properly and also
> +		 * propagate errors that now are calling die()
> +		 */

I believe that this comment is doubly wrong.  First we already check
whether evlist has tracepoints before calling the below function.
Secondly I think I got rid of all of the die() calls from the
tracing_data_get/put path.

So I would suggest simply removing this comment block.

Thanks,
Namhyung


> +		err = perf_event__synthesize_tracing_data(tool, rec->output, evlist,
> +							  process_synthesized_event);
> +		if (err <= 0) {
> +			pr_err("Couldn't record tracing data.\n");
> +			return err;
> +		}
> +		advance_output(rec, err);
> +	}
> +
> +	return 0;
> +}

Re: [PATCH 22/23] perf tools: Add multi file '-M' option for record command

[Namhyung Kim]

On Wed, 17 Jul 2013 19:50:02 +0200, Jiri Olsa wrote:
> Adding multi file '-M' option for record command to store
> split event data into multiple files based on the file
> size specified as an argument to the option.

It seems you need to update the doc too. :)

Thanks,
Namhyung

Re: [PATCH 06/23] perf doc: Add perf data file documentation

[Andi Kleen]

On Wed, Jul 17, 2013 at 07:49:46PM +0200, Jiri Olsa wrote:
> Adding perf data file documentation.

FWIW I wrote the following python "construct" description of the data format
some time ago. It's  more formal than yours. There were still a few things
missing though and one mysterious bug.

# A description of the perf.data file format in "construct"
#
from construct import *

def fork_exit(name): 
    return Struct(name,
                  SNInt32("pid"),
                  SNInt32("ppid"),
                  SNInt32("tid"),
                  SNInt32("ptid"),
                  UNInt64("time"))

def throttle(name):
    return Struct(name,
                  UNInt64("time"),
                  UNInt64("id"),
                  UNInt64("stream_id"))

def event(sample_type, read_format):
    return Embedded(
         Struct("event",
                If(lambda ctx: sample_type.ip,
                   UNInt64("ip")),
                If(lambda ctx: sample_type.tid,
                   Struct("tid",
                          SNInt64("pid"),
                          SNInt64("tid"))),
                If(lambda ctx: sample_type.time,
                   UNInt64("time")),
                If(lambda ctx: sample_type.addr,
                   UNInt64("addr")),
                If(lambda ctx: sample_type.stream_id,
                   UNInt64("stream_id")),
                If(lambda ctx: sample_type.cpu,
                   Struct("cpu",
                          UNInt32("cpu"),
                          UNInt32("res"))),
                If(lambda ctx: sample_type.period,
                   UNInt64("period")),
                # XXX
                #If(lambda ctx: sample_type.read,
                #   read_format(rformat)),
                If(lambda ctx: sample_type.callchain,
                   Struct("callchain",
                          UNInt64("nr"),
                          Array(lambda ctx: ctx.nr,
                                UNInt64("addr")))),
                If(lambda ctx: sample_type.raw,
                   Struct("raw",
                          UNInt32("size"),
                          Bytes("raw", lambda ctx: ctx.size))),
                If(lambda ctx: sample_type.branch_stack,
                   Struct("branch_stack",
                          UNInt64("nr"),
                          Array(lambda ctx: ctx.nr,
                                Struct(None,
                                       UNInt64("from"),
                                       UNInt64("to"),
                                       UNInt64("flags"))))), # XXX split
                # both need parameters passed in
                # sample reg
                # stack user
                If(lambda ctx: sample_type.weight,
                   UNInt64("weight")),
                If(lambda ctx: sample_type.data_src,
                   UNInt64("data_src")),
                Anchor("end_event"),
                Padding(lambda ctx: max(0, ctx.size - ctx.end_event))))

# XXX need to make OnDemand for large files

def perf_event(sample_type, read_format):         
    return Struct("perf_event",
                    Anchor("start"),
                    Enum(UNInt32("type"),
                         MMAP                   = 1,
                         LOST                   = 2,
                         COMM                   = 3,
                         EXIT                   = 4,
                         THROTTLE               = 5,
                         UNTHROTTLE             = 6,
                         FORK                   = 7,
                         READ                   = 8,
                         SAMPLE                 = 9),
                    UNInt16("misc"),
                    UNInt16("size"),
                    Switch("data",
                           lambda ctx: ctx.type,
                           {
                              "MMAP": Struct("mmap",
                                              SNInt32("pid"),
                                              SNInt32("tid"),
                                              UNInt64("addr"),
                                              UNInt64("len"),
                                              UNInt64("pgoff"),
                                                CString("filename")),
                              "LOST": Struct("lost",
                                              UNInt64("id"),
                                              UNInt64("lost")),
                              "COMM": Struct("comm",
                                             SNInt32("pid"),
                                             SNInt32("tid"),
                                             CString("comm")),
                              "EXIT": fork_exit("exit"),
                              "THROTTLE": throttle("thottle"),
                              "UNTHROTTLE": throttle("unthottle"),
                              "FORK": fork_exit("fork"),
                              #"READ": read_format(read_format),
                              "SAMPLE": event(sample_type, read_format),
                           }),
                        Anchor("end"),
                        Padding(lambda ctx: ctx.size - (ctx.end - ctx.start))
                    )

def perf_event_seq(sample_type, read_format):
    return GreedyRange(perf_event(sample_type, read_format))

perf_event_attr_sizes = (64, 72, 80, 96)

perf_event_attr = Struct("perf_event_attr",
                         Anchor("start"),
                         Enum(UNInt32("type"),
                              HARDWARE = 0,
                              SOFTWARE = 1,
                              TRACEPOINT = 2,
                              HW_CACHE = 3,
                              RAW = 4,
                              BREAKPOINT = 5),
                         UNInt32("size"),
                         UNInt64("config"),
                         UNInt64("sample_period_freq"),
                         BitStruct("sample_type",
                                            Flag("ip"),
                                            Flag("tid"),
                                            Flag("time"),
                                            Flag("addr"),
                                            Flag("read"),
                                            Flag("callchain"),
                                            Flag("id"),
                                            Flag("cpu"),
                                            Flag("period"),
                                            Flag("stream_id"),
                                            Flag("raw"),
                                            Flag("branch_stack"),
                                            Flag("regs_user"),
                                            Flag("stack_user"),
                                            Flag("weight"),
                                            Flag("data_src"),
                                            Padding(64 - 16)),
                         BitStruct("read_format",
                                            Flag("total_time_enabled"),
                                            Flag("total_time_running"),
                                            Flag("id"),
                                            Flag("group"),
                                            Padding(64 - 4)),
                         Embedded(BitStruct(None,
                                            Flag("disabled"),
                                            Flag("inherit"),
                                            Flag("pinned"),            
                                            Flag("exclusive"),      
                                            Flag("exclude_user"),   
                                            Flag("exclude_kernel"),
                                            Flag("exclude_hv"),
                                            Flag("exclude_idle"),
                                            Flag("mmap"),
                                            Flag("comm"),
                                            Flag("freq"),
                                            Flag("inherit_stat"),
                                            Flag("enable_on_exec"),
                                            Flag("task"),
                                            Flag("watermark"),
                                            BitField("precise_ip", 2),
                                            Flag("mmap_data"),
                                            Flag("sample_id_all"),
                                            Flag("exclude_host"),
                                            Flag("exclude_guest"),
                                            Flag("exclude_callchain_kernel"),
                                            Flag("exclude_callchain_user"),
                                            Padding(41))),
                         UNInt32("wakeup_events"),
                         UNInt32("bp_type"),
                         UNInt64("config1"),
                         If(lambda ctx: ctx.size >= perf_event_attr_sizes[1],
                            UNInt64("config2")),
                         If(lambda ctx: ctx.size >= perf_event_attr_sizes[2],
                            UNInt64("branch_sample_type")),
                         If(lambda ctx: ctx.size >= perf_event_attr_sizes[3],
                            Embedded(Struct(None,
                                            UNInt64("sample_regs_user"),
                                            UNInt32("sample_stack_user")))),
                         Anchor("end"),
                         Value("perf_event_attr_size", lambda ctx: ctx.end - ctx.start),
                         Padding(lambda ctx: ctx.size - ctx.perf_event_attr_size))

# assumes all attributes are the same size

perf_file_attr = Struct("perf_file_attr",
                        Peek(Embedded(Struct(None, UNInt32("type"), UNInt32("size")))),
                        Array(lambda ctx: ctx._.size / ctx.size, perf_event_attr))

perf_event_types = Struct("perf_file_attr",
                          Anchor("here"),
                          Padding(lambda ctx: ctx._.size))

perf_data = OnDemand(Bytes("perf_data", lambda ctx: ctx.size))
                             
def perf_file_section(name, target):
    return Struct(name,
                  UNInt64("offset"),
                  UNInt64("size"),
                  Pointer(lambda ctx: ctx.offset, target))

perf_file = Struct("perf_file_header",
                   UNInt64("magic"), # XXX
                   UNInt64("size"),
                   UNInt64("attr_size"),
                   perf_file_section("attrs", perf_file_attr),
                   perf_file_section("data", perf_data),
                   perf_file_section("event_types", perf_event_types),
                   BitStruct("adds_features",
                             Flag("tracing_data"),
                             Flag("build_id"),
                             Flag("hostname"),
                             Flag("osrelease"),
                             Flag("version"),
                             Flag("arch"),
                             Flag("nrcpus"),
                             Flag("cpudesc"),
                             Flag("cpuid"),
                             Flag("total_mem"),
                             Flag("cmdline"),
                             Flag("event_desc"),
                             Flag("cpu_topology"),
                             Flag("numa_topology"),
                             Flag("branch_stack"),
                             Flag("pmu_mappings"),
                             Flag("group_desc"),
                             Padding(64 - 17)),
                   Padding(3 * 8))

def get_events(h):
    data = h.data.perf_data.value
    # assumes event 0 attributes applies to all samples?
    # XXX
    ev0 = h.attrs.perf_file_attr.perf_event_attr[0]
    assert ev0.size in perf_event_attr_sizes
    return perf_event_seq(ev0.sample_type, ev0.read_format).parse(data)

if __name__ == '__main__':
    import sys
    
    with open(sys.argv[1], "rb") as f:
        h = perf_file.parse_stream(f)
        print h
        print get_events(h)

Re: [PATCH 13/23] perf tools: Introduce perf.data version 3 format

[Jiri Olsa]

On Fri, Jul 19, 2013 at 09:06:22PM +0900, Namhyung Kim wrote:
> On Wed, 17 Jul 2013 19:49:53 +0200, Jiri Olsa wrote:
> > Introducing perf.data version 3 format. No functional change,
> > just introducing doc, magic bytes and the struct.
> >
> [SNIP]
> 
> > +perf-data-file-v3(1)
> 
> s/(1)/(5)/
> 
> The same goes to v2 doc.
ok

> 
> 
> > +====================
> > +Following text describes version 3 of the perf data file format,
> > +which is version that is currently used by perf tool.
> > +
> > +In version 3 we got rid of following sections:
> > +  EVENT IDS
> > +  EVENT ATTRIBUTES
> > +  EVENT TYPES
> > +
> > +and keep only data and features sections. All the data
> > +from above listed sections are now read from specific
> > +FEATURES sections.
> > +
> > +benefits:
> > +  - siplified header storage at the time the file is closed
> 
> s/siplified/simplified/
ok

> 
> 
> > +  - allowing cpu hotpluged addition/removal event IDs
> > +    during the session due to 1)

it's the first '-' before I switched from numbering to '-' ;-) will fix

thanks,
jirka

Re: [PATCH 15/23] perf tools: Add perf data version 3 header read

[Jiri Olsa]

On Fri, Jul 19, 2013 at 09:11:39PM +0900, Namhyung Kim wrote:
> On Wed, 17 Jul 2013 19:49:55 +0200, Jiri Olsa wrote:
> > Adding perf data version 3 header read code to load
> > data for v3 format.
> >
> [SNIP]
> > +
> > +	session = container_of(header, struct perf_session, header);
> > +	if (session->evlist)
> > +		return 0;
> 
> I guess we need version check here.  The v2 format still needs to
> process it even though it's session already has a evlist, right?

right, we need it for event names.. will fix

thanks,
jirka

Re: [PATCH 18/23] perf tools: Move synthetizing into single function

[Jiri Olsa]

On Fri, Jul 19, 2013 at 09:30:50PM +0900, Namhyung Kim wrote:
> On Wed, 17 Jul 2013 19:49:58 +0200, Jiri Olsa wrote:
> > Moving synthetizing into single function, so it
> 
> s/synthetizing/synthesizing/
> 
> The same goes to the subject line too.

ok


> 
> 
> > could be reused.
> >
> > Signed-off-by: Jiri Olsa <jolsa@redhat.com>
> > Cc: Corey Ashford <cjashfor@linux.vnet.ibm.com>
> > Cc: Frederic Weisbecker <fweisbec@gmail.com>
> > Cc: Ingo Molnar <mingo@elte.hu>
> > Cc: Namhyung Kim <namhyung@kernel.org>
> > Cc: Paul Mackerras <paulus@samba.org>
> > Cc: Peter Zijlstra <a.p.zijlstra@chello.nl>
> > Cc: Arnaldo Carvalho de Melo <acme@redhat.com>
> > Cc: Andi Kleen <ak@linux.intel.com>
> > Cc: David Ahern <dsahern@gmail.com>
> > ---
> >  tools/perf/builtin-record.c | 155 ++++++++++++++++++++++++++------------------
> >  1 file changed, 91 insertions(+), 64 deletions(-)
> >
> > diff --git a/tools/perf/builtin-record.c b/tools/perf/builtin-record.c
> > index b67564c..33a5bce 100644
> > --- a/tools/perf/builtin-record.c
> > +++ b/tools/perf/builtin-record.c
> > @@ -317,6 +317,95 @@ static void perf_event__synthesize_guest_os(struct machine *machine, void *data)
> >  		       " relocation symbol.\n", machine->pid);
> >  }
> >  
> > +static int synthesize_record_pipe(struct perf_record *rec)
> > +{
> > +	struct perf_session  *session = rec->session;
> > +	struct perf_tool        *tool = &rec->tool;
> > +	struct perf_evlist    *evlist = rec->evlist;
> > +	int err;
> > +
> > +	err = perf_event__synthesize_attrs(tool, session,
> > +					   process_synthesized_event);
> > +	if (err < 0) {
> > +		pr_err("Couldn't synthesize attrs.\n");
> > +		return err;
> > +	}
> > +
> > +	if (have_tracepoints(&evlist->entries)) {
> > +		/*
> > +		 * FIXME err <= 0 here actually means that
> > +		 * there were no tracepoints so its not really
> > +		 * an error, just that we don't need to
> > +		 * synthesize anything.  We really have to
> > +		 * return this more properly and also
> > +		 * propagate errors that now are calling die()
> > +		 */
> 
> I believe that this comment is doubly wrong.  First we already check
> whether evlist has tracepoints before calling the below function.
> Secondly I think I got rid of all of the die() calls from the
> tracing_data_get/put path.
> 
> So I would suggest simply removing this comment block.
ook

thanks,
jirka

Re: [PATCH 22/23] perf tools: Add multi file '-M' option for record command

[Jiri Olsa]

On Fri, Jul 19, 2013 at 10:02:41PM +0900, Namhyung Kim wrote:
> On Wed, 17 Jul 2013 19:50:02 +0200, Jiri Olsa wrote:
> > Adding multi file '-M' option for record command to store
> > split event data into multiple files based on the file
> > size specified as an argument to the option.
> 
> It seems you need to update the doc too. :)

will do ;-)

thanks,
jirka

Re: [PATCH 10/23] perf tools: Introduce swap_header function

[Jiri Olsa]

On Fri, Jul 19, 2013 at 08:33:37PM +0900, Namhyung Kim wrote:
> On Wed, 17 Jul 2013 19:49:50 +0200, Jiri Olsa wrote:
> > Introducing swap_header function to make the swapping
> > code more clear and extensible.
> 
> [SNIP]
> >  
> > +static int swap_header(struct perf_file_header *header)
> > +{
> > +	mem_bswap_64(header, offsetof(struct perf_file_header, adds_features));
> > +
> > +	if (header->size != sizeof(*header)) {
> > +		/* Support the previous format */
> > +		if (header->size == offsetof(typeof(*header), adds_features))
> > +			bitmap_zero(header->adds_features, HEADER_FEAT_BITS);
> > +		else
> > +			return -1;
> > +	} else
> > +		swap_features(header->adds_features);
> > +
> > +	return 0;
> > +}
> > +
> >  int perf_file_header__read(struct perf_file_header *header,
> >  			   struct perf_header *ph, int fd)
> >  {
> > @@ -2562,19 +2578,8 @@ int perf_file_header__read(struct perf_file_header *header,
> >  		return -1;
> >  	}
> >  
> > -	if (ph->needs_swap) {
> > -		mem_bswap_64(header, offsetof(struct perf_file_header,
> > -			     adds_features));
> > -	}
> > -
> > -	if (header->size != sizeof(*header)) {
> > -		/* Support the previous format */
> > -		if (header->size == offsetof(typeof(*header), adds_features))
> > -			bitmap_zero(header->adds_features, HEADER_FEAT_BITS);
> > -		else
> > -			return -1;
> > -	} else if (ph->needs_swap)
> > -		swap_features(header->adds_features);
> > +	if (ph->needs_swap && swap_header(header))
> > +		return -1;
> 
> Doesn't it change existing behavior?  It looks like supporting previous
> format on a same endian machines will be removed by this patch.

I'll check and try to add v2 perf.data reading test for both endians

thanks,
jirka