All of lore.kernel.org
 help / color / mirror / Atom feed
* [PATCHv5 00/11] perf record: Add support to store data in directory
@ 2019-03-08 13:47 Jiri Olsa
  2019-03-08 13:47 ` [PATCH 01/11] perf data: Add directory support Jiri Olsa
                   ` (10 more replies)
  0 siblings, 11 replies; 27+ messages in thread
From: Jiri Olsa @ 2019-03-08 13:47 UTC (permalink / raw)
  To: Arnaldo Carvalho de Melo
  Cc: lkml, Ingo Molnar, Namhyung Kim, Alexander Shishkin,
	Peter Zijlstra, Adrian Hunter, Andi Kleen, Stephane Eranian,
	Alexey Budankov

hi,
this patchset adds the --dir option to record command (and all
the other record command that overload cmd_record) that allows
the data to be stored in directory with multiple data files.

It's next step for multiple threads implementation in record.
It's now possible to make directory data via --dir option, like:

  $ perf record --dir perf bench sched messaging
  $ ls -l perf.data
  total 344
  -rw-------. 1 jolsa jolsa 43864 Jan 20 22:26 data.0
  -rw-------. 1 jolsa jolsa 30464 Jan 20 22:26 data.1
  -rw-------. 1 jolsa jolsa 53816 Jan 20 22:26 data.2
  -rw-------. 1 jolsa jolsa 30368 Jan 20 22:26 data.3
  -rw-------. 1 jolsa jolsa 40088 Jan 20 22:26 data.4
  -rw-------. 1 jolsa jolsa 42592 Jan 20 22:26 data.5
  -rw-------. 1 jolsa jolsa 56136 Jan 20 22:26 data.6
  -rw-------. 1 jolsa jolsa 25992 Jan 20 22:26 data.7
  -rw-------. 1 jolsa jolsa  8832 Jan 20 22:26 header

There's a data file created for every cpu and it's storing
data for those cpu maps. The report command will read it
transparently, sort it and display as single file data.

There's new DIR_FORMAT feature to describe directory data
layout/format. In future we can describe different data files
layout according to special needs.

It's possible to transform directory data into standard
perf.data file via simple inject command:
    
  $ perf inject -o perf.data.file -i perf.data

The old perf fails over the directory data with following message:
  $ perf report
  incompatible file format (rerun with -v to learn more)

I'm now testing the record threads support, so I'd like to
have some agreement on the directory data support before.

v5 changes:
  - rebased to latest Arnaldo's perf/core

v4 changes:
  - some of the patches already taken
  - removing up perf.data directory on perf_data__open
    error path

v3 changes:
  - add rm_rf_perf_data to safely remove file/directory perf data
  - allocation fix in perf_data__create_dir

v2 changes:
  - rm_rf changes are already accepted with requested changes
  - updated doc/man plus adding perf.data-directory-format.txt
    to describe directory format/layout
  - the --switch-output options now works over directory data
  - data rollback is not part of this patchset, updated my TODO though ;-)
  - added --output-dir option to combine -o and --dir
  - added DIR_FORMAT feature to describe directory data
  - disabling directory output for aio for now

It's also available in here:
  git://git.kernel.org/pub/scm/linux/kernel/git/jolsa/perf.git
  perf/dir

thanks,
jirka


---
Jiri Olsa (11):
      perf data: Add directory support
      perf data: Don't store auxtrace index for directory data file
      perf data: Add perf_data__update_dir function
      perf data: Make perf_data__size to work over directory
      perf header: Add DIR_FORMAT feature to describe directory data
      perf session: Add process callback to reader object
      perf session: Add __perf_session__process_dir_events function
      perf session: Add path to reader object
      perf record: Add --dir option to store data in directory
      perf record: Add --output-dir option to store data in directory
      perf record: Describe perf.data directory format

 tools/lib/subcmd/parse-options.h                        |   4 ++++
 tools/perf/Documentation/perf-record.txt                |   6 ++++++
 tools/perf/Documentation/perf.data-directory-format.txt |  54 ++++++++++++++++++++++++++++++++++++++++++++++++++++++
 tools/perf/builtin-record.c                             |  95 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++-------
 tools/perf/util/data.c                                  |  96 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++---
 tools/perf/util/data.h                                  |  12 ++++++++----
 tools/perf/util/header.c                                |  44 +++++++++++++++++++++++++++++++++++++++++++-
 tools/perf/util/header.h                                |   5 +++++
 tools/perf/util/mmap.h                                  |  23 ++++++++++++-----------
 tools/perf/util/session.c                               | 125 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++--------
 10 files changed, 430 insertions(+), 34 deletions(-)
 create mode 100644 tools/perf/Documentation/perf.data-directory-format.txt

^ permalink raw reply	[flat|nested] 27+ messages in thread

* [PATCH 01/11] perf data: Add directory support
  2019-03-08 13:47 [PATCHv5 00/11] perf record: Add support to store data in directory Jiri Olsa
@ 2019-03-08 13:47 ` Jiri Olsa
  2019-03-08 17:26   ` Arnaldo Carvalho de Melo
  2019-03-22 22:03   ` [tip:perf/urgent] perf data: Support having perf.data stored as a directory tip-bot for Jiri Olsa
  2019-03-08 13:47 ` [PATCH 02/11] perf data: Don't store auxtrace index for directory data file Jiri Olsa
                   ` (9 subsequent siblings)
  10 siblings, 2 replies; 27+ messages in thread
From: Jiri Olsa @ 2019-03-08 13:47 UTC (permalink / raw)
  To: Arnaldo Carvalho de Melo
  Cc: lkml, Ingo Molnar, Namhyung Kim, Alexander Shishkin,
	Peter Zijlstra, Adrian Hunter, Andi Kleen, Stephane Eranian,
	Alexey Budankov

Adding support to have directory as perf.data.

The caller needs to set 'struct perf_data::is_dir flag
and the path will be treated as directory.

The 'struct perf_data::file' is initialized and open
as 'path/header' file.

Adding check to direcory interface functions to check
on is_dir flag.

Link: http://lkml.kernel.org/n/tip-pvot1aywiem9epgqpfi1agaj@git.kernel.org
Signed-off-by: Jiri Olsa <jolsa@kernel.org>
---
 tools/perf/util/data.c    | 49 ++++++++++++++++++++++++++++++++++++++-
 tools/perf/util/data.h    |  6 +++++
 tools/perf/util/session.c |  4 ++++
 3 files changed, 58 insertions(+), 1 deletion(-)

diff --git a/tools/perf/util/data.c b/tools/perf/util/data.c
index e098e189f93e..e28fd0fde5e0 100644
--- a/tools/perf/util/data.c
+++ b/tools/perf/util/data.c
@@ -34,6 +34,9 @@ int perf_data__create_dir(struct perf_data *data, int nr)
 	struct perf_data_file *files = NULL;
 	int i, ret = -1;
 
+	if (WARN_ON(!data->is_dir))
+		return -EINVAL;
+
 	files = zalloc(nr * sizeof(*files));
 	if (!files)
 		return -ENOMEM;
@@ -69,6 +72,9 @@ int perf_data__open_dir(struct perf_data *data)
 	DIR *dir;
 	int nr = 0;
 
+	if (WARN_ON(!data->is_dir))
+		return -EINVAL;
+
 	dir = opendir(data->path);
 	if (!dir)
 		return -EINVAL;
@@ -173,6 +179,16 @@ static int check_backup(struct perf_data *data)
 	return 0;
 }
 
+static bool is_dir(struct perf_data *data)
+{
+	struct stat st;
+
+	if (stat(data->path, &st))
+		return false;
+
+	return (st.st_mode & S_IFMT) == S_IFDIR;
+}
+
 static int open_file_read(struct perf_data *data)
 {
 	struct stat st;
@@ -254,6 +270,30 @@ static int open_file_dup(struct perf_data *data)
 	return open_file(data);
 }
 
+static int open_dir(struct perf_data *data)
+{
+	int ret;
+
+	/*
+	 * So far we open only the header, so we
+	 * can read the data version and layout.
+	 */
+	if (asprintf(&data->file.path, "%s/header", data->path) < 0)
+		return -ENOMEM;
+
+	if (perf_data__is_write(data) &&
+	    mkdir(data->path, S_IRWXU) < 0)
+		return -1;
+
+	ret = open_file(data);
+
+	/* Cleanup whatever we managed to create so far. */
+	if (ret && perf_data__is_write(data))
+		rm_rf_perf_data(data->path);
+
+	return ret;
+}
+
 int perf_data__open(struct perf_data *data)
 {
 	if (check_pipe(data))
@@ -265,11 +305,18 @@ int perf_data__open(struct perf_data *data)
 	if (check_backup(data))
 		return -1;
 
-	return open_file_dup(data);
+	if (perf_data__is_read(data))
+		data->is_dir = is_dir(data);
+
+	return perf_data__is_dir(data) ?
+	       open_dir(data) : open_file_dup(data);
 }
 
 void perf_data__close(struct perf_data *data)
 {
+	if (perf_data__is_dir(data))
+		perf_data__close_dir(data);
+
 	zfree(&data->file.path);
 	close(data->file.fd);
 }
diff --git a/tools/perf/util/data.h b/tools/perf/util/data.h
index 14b47be2bd69..06aefeda311f 100644
--- a/tools/perf/util/data.h
+++ b/tools/perf/util/data.h
@@ -19,6 +19,7 @@ struct perf_data {
 	const char		*path;
 	struct perf_data_file	 file;
 	bool			 is_pipe;
+	bool			 is_dir;
 	bool			 force;
 	enum perf_data_mode	 mode;
 
@@ -43,6 +44,11 @@ static inline int perf_data__is_pipe(struct perf_data *data)
 	return data->is_pipe;
 }
 
+static inline bool perf_data__is_dir(struct perf_data *data)
+{
+	return data->is_dir;
+}
+
 static inline int perf_data__fd(struct perf_data *data)
 {
 	return data->file.fd;
diff --git a/tools/perf/util/session.c b/tools/perf/util/session.c
index db643f3c2b95..de777bdc0ed3 100644
--- a/tools/perf/util/session.c
+++ b/tools/perf/util/session.c
@@ -152,6 +152,10 @@ struct perf_session *perf_session__new(struct perf_data *data,
 			}
 
 			perf_evlist__init_trace_event_sample_raw(session->evlist);
+
+			/* Open the directory data. */
+			if (data->is_dir && perf_data__open_dir(data))
+				goto out_delete;
 		}
 	} else  {
 		session->machines.host.env = &perf_env;
-- 
2.17.2


^ permalink raw reply	[flat|nested] 27+ messages in thread

* [PATCH 02/11] perf data: Don't store auxtrace index for directory data file
  2019-03-08 13:47 [PATCHv5 00/11] perf record: Add support to store data in directory Jiri Olsa
  2019-03-08 13:47 ` [PATCH 01/11] perf data: Add directory support Jiri Olsa
@ 2019-03-08 13:47 ` Jiri Olsa
  2019-03-22 22:03   ` [tip:perf/urgent] " tip-bot for Jiri Olsa
  2019-03-08 13:47 ` [PATCH 03/11] perf data: Add perf_data__update_dir function Jiri Olsa
                   ` (8 subsequent siblings)
  10 siblings, 1 reply; 27+ messages in thread
From: Jiri Olsa @ 2019-03-08 13:47 UTC (permalink / raw)
  To: Arnaldo Carvalho de Melo
  Cc: lkml, Ingo Molnar, Namhyung Kim, Alexander Shishkin,
	Peter Zijlstra, Adrian Hunter, Andi Kleen, Stephane Eranian,
	Alexey Budankov

We can't store auxtrace index when we store to multiple files,
because we keep only offset for it, not the file.

The auxtrace data will be processed correctly in the 'pipe' mode.

Link: http://lkml.kernel.org/n/tip-og11od5s6nfxuf0ftxyu8m6k@git.kernel.org
Signed-off-by: Jiri Olsa <jolsa@kernel.org>
---
 tools/perf/builtin-record.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/tools/perf/builtin-record.c b/tools/perf/builtin-record.c
index f3f7f3100336..e983c8d71a79 100644
--- a/tools/perf/builtin-record.c
+++ b/tools/perf/builtin-record.c
@@ -392,7 +392,7 @@ static int record__process_auxtrace(struct perf_tool *tool,
 	size_t padding;
 	u8 pad[8] = {0};
 
-	if (!perf_data__is_pipe(data)) {
+	if (!perf_data__is_pipe(data) && !perf_data__is_dir(data)) {
 		off_t file_offset;
 		int fd = perf_data__fd(data);
 		int err;
-- 
2.17.2


^ permalink raw reply	[flat|nested] 27+ messages in thread

* [PATCH 03/11] perf data: Add perf_data__update_dir function
  2019-03-08 13:47 [PATCHv5 00/11] perf record: Add support to store data in directory Jiri Olsa
  2019-03-08 13:47 ` [PATCH 01/11] perf data: Add directory support Jiri Olsa
  2019-03-08 13:47 ` [PATCH 02/11] perf data: Don't store auxtrace index for directory data file Jiri Olsa
@ 2019-03-08 13:47 ` Jiri Olsa
  2019-03-22 22:04   ` [tip:perf/urgent] perf data: Add perf_data__update_dir() function tip-bot for Jiri Olsa
  2019-03-08 13:47 ` [PATCH 04/11] perf data: Make perf_data__size to work over directory Jiri Olsa
                   ` (7 subsequent siblings)
  10 siblings, 1 reply; 27+ messages in thread
From: Jiri Olsa @ 2019-03-08 13:47 UTC (permalink / raw)
  To: Arnaldo Carvalho de Melo
  Cc: lkml, Ingo Molnar, Namhyung Kim, Alexander Shishkin,
	Peter Zijlstra, Adrian Hunter, Andi Kleen, Stephane Eranian,
	Alexey Budankov

Adding perf_data__update_dir function to update
size for every file within the perf.data directory.

Link: http://lkml.kernel.org/n/tip-3ii12l48u4tyvrzxo1797w7e@git.kernel.org
Signed-off-by: Jiri Olsa <jolsa@kernel.org>
---
 tools/perf/util/data.c | 20 ++++++++++++++++++++
 tools/perf/util/data.h |  1 +
 2 files changed, 21 insertions(+)

diff --git a/tools/perf/util/data.c b/tools/perf/util/data.c
index e28fd0fde5e0..d568975c5ee3 100644
--- a/tools/perf/util/data.c
+++ b/tools/perf/util/data.c
@@ -124,6 +124,26 @@ int perf_data__open_dir(struct perf_data *data)
 	return ret;
 }
 
+int perf_data__update_dir(struct perf_data *data)
+{
+	int i;
+
+	if (WARN_ON(!data->is_dir))
+		return -EINVAL;
+
+	for (i = 0; i < data->dir.nr; i++) {
+		struct perf_data_file *file = &data->dir.files[i];
+		struct stat st;
+
+		if (fstat(file->fd, &st))
+			return -1;
+
+		file->size = st.st_size;
+	}
+
+	return 0;
+}
+
 static bool check_pipe(struct perf_data *data)
 {
 	struct stat st;
diff --git a/tools/perf/util/data.h b/tools/perf/util/data.h
index 06aefeda311f..0deeb1af9f54 100644
--- a/tools/perf/util/data.h
+++ b/tools/perf/util/data.h
@@ -79,4 +79,5 @@ int perf_data__switch(struct perf_data *data,
 int perf_data__create_dir(struct perf_data *data, int nr);
 int perf_data__open_dir(struct perf_data *data);
 void perf_data__close_dir(struct perf_data *data);
+int perf_data__update_dir(struct perf_data *data);
 #endif /* __PERF_DATA_H */
-- 
2.17.2


^ permalink raw reply	[flat|nested] 27+ messages in thread

* [PATCH 04/11] perf data: Make perf_data__size to work over directory
  2019-03-08 13:47 [PATCHv5 00/11] perf record: Add support to store data in directory Jiri Olsa
                   ` (2 preceding siblings ...)
  2019-03-08 13:47 ` [PATCH 03/11] perf data: Add perf_data__update_dir function Jiri Olsa
@ 2019-03-08 13:47 ` Jiri Olsa
  2019-03-08 17:33   ` Arnaldo Carvalho de Melo
  2019-03-22 22:05   ` [tip:perf/urgent] perf data: Make perf_data__size() " tip-bot for Jiri Olsa
  2019-03-08 13:47 ` [PATCH 05/11] perf header: Add DIR_FORMAT feature to describe directory data Jiri Olsa
                   ` (6 subsequent siblings)
  10 siblings, 2 replies; 27+ messages in thread
From: Jiri Olsa @ 2019-03-08 13:47 UTC (permalink / raw)
  To: Arnaldo Carvalho de Melo
  Cc: lkml, Ingo Molnar, Namhyung Kim, Alexander Shishkin,
	Peter Zijlstra, Adrian Hunter, Andi Kleen, Stephane Eranian,
	Alexey Budankov

Making perf_data__size to return proper size
for directory data.

Link: http://lkml.kernel.org/n/tip-t4dm8cctat2ginmy2bb08xe8@git.kernel.org
Signed-off-by: Jiri Olsa <jolsa@kernel.org>
---
 tools/perf/util/data.c | 17 +++++++++++++++++
 tools/perf/util/data.h |  6 +-----
 2 files changed, 18 insertions(+), 5 deletions(-)

diff --git a/tools/perf/util/data.c b/tools/perf/util/data.c
index d568975c5ee3..498c531e2cb9 100644
--- a/tools/perf/util/data.c
+++ b/tools/perf/util/data.c
@@ -393,3 +393,20 @@ int perf_data__switch(struct perf_data *data,
 	free(new_filepath);
 	return ret;
 }
+
+unsigned long perf_data__size(struct perf_data *data)
+{
+	u64 size = data->file.size;
+	int i;
+
+	if (!data->is_dir)
+		return size;
+
+	for (i = 0; i < data->dir.nr; i++) {
+		struct perf_data_file *file = &data->dir.files[i];
+
+		size += file->size;
+	}
+
+	return size;
+}
diff --git a/tools/perf/util/data.h b/tools/perf/util/data.h
index 0deeb1af9f54..d342469bdfda 100644
--- a/tools/perf/util/data.h
+++ b/tools/perf/util/data.h
@@ -54,11 +54,6 @@ static inline int perf_data__fd(struct perf_data *data)
 	return data->file.fd;
 }
 
-static inline unsigned long perf_data__size(struct perf_data *data)
-{
-	return data->file.size;
-}
-
 int perf_data__open(struct perf_data *data);
 void perf_data__close(struct perf_data *data);
 ssize_t perf_data__write(struct perf_data *data,
@@ -80,4 +75,5 @@ int perf_data__create_dir(struct perf_data *data, int nr);
 int perf_data__open_dir(struct perf_data *data);
 void perf_data__close_dir(struct perf_data *data);
 int perf_data__update_dir(struct perf_data *data);
+unsigned long perf_data__size(struct perf_data *data);
 #endif /* __PERF_DATA_H */
-- 
2.17.2


^ permalink raw reply	[flat|nested] 27+ messages in thread

* [PATCH 05/11] perf header: Add DIR_FORMAT feature to describe directory data
  2019-03-08 13:47 [PATCHv5 00/11] perf record: Add support to store data in directory Jiri Olsa
                   ` (3 preceding siblings ...)
  2019-03-08 13:47 ` [PATCH 04/11] perf data: Make perf_data__size to work over directory Jiri Olsa
@ 2019-03-08 13:47 ` Jiri Olsa
  2019-03-08 18:24   ` Arnaldo Carvalho de Melo
  2019-03-22 22:05   ` [tip:perf/urgent] " tip-bot for Jiri Olsa
  2019-03-08 13:47 ` [PATCH 06/11] perf session: Add process callback to reader object Jiri Olsa
                   ` (5 subsequent siblings)
  10 siblings, 2 replies; 27+ messages in thread
From: Jiri Olsa @ 2019-03-08 13:47 UTC (permalink / raw)
  To: Arnaldo Carvalho de Melo
  Cc: lkml, Ingo Molnar, Namhyung Kim, Alexander Shishkin,
	Peter Zijlstra, Adrian Hunter, Andi Kleen, Stephane Eranian,
	Alexey Budankov

The data files layout is described by HEADER_DIR_FORMAT
feature.  Currently it holds only version number (1):

     uint64_t version;

The current version holds only version value (1) means
that data files:
  - follow the 'data.*' name format
  - contain raw events data in standard perf format as
    read from kernel (and need to be sorted)

Future versions are expected to describe different data
files layout according to special needs.

Link: http://lkml.kernel.org/n/tip-6l53s3oabwr1r4n6wke21d8g@git.kernel.org
Signed-off-by: Jiri Olsa <jolsa@kernel.org>
---
 tools/perf/builtin-record.c |  2 ++
 tools/perf/util/data.c      | 10 +++++++--
 tools/perf/util/data.h      |  1 +
 tools/perf/util/header.c    | 44 ++++++++++++++++++++++++++++++++++++-
 tools/perf/util/header.h    |  5 +++++
 5 files changed, 59 insertions(+), 3 deletions(-)

diff --git a/tools/perf/builtin-record.c b/tools/perf/builtin-record.c
index e983c8d71a79..a468d882e74f 100644
--- a/tools/perf/builtin-record.c
+++ b/tools/perf/builtin-record.c
@@ -837,6 +837,8 @@ static void record__init_features(struct record *rec)
 	if (!(rec->opts.use_clockid && rec->opts.clockid_res_ns))
 		perf_header__clear_feat(&session->header, HEADER_CLOCKID);
 
+	perf_header__clear_feat(&session->header, HEADER_DIR_FORMAT);
+
 	perf_header__clear_feat(&session->header, HEADER_STAT);
 }
 
diff --git a/tools/perf/util/data.c b/tools/perf/util/data.c
index 498c531e2cb9..41a9f478f0f0 100644
--- a/tools/perf/util/data.c
+++ b/tools/perf/util/data.c
@@ -14,6 +14,7 @@
 #include "data.h"
 #include "util.h"
 #include "debug.h"
+#include "header.h"
 
 static void close_dir(struct perf_data_file *files, int nr)
 {
@@ -41,8 +42,9 @@ int perf_data__create_dir(struct perf_data *data, int nr)
 	if (!files)
 		return -ENOMEM;
 
-	data->dir.files = files;
-	data->dir.nr    = nr;
+	data->dir.version = PERF_DIR_VERSION;
+	data->dir.files   = files;
+	data->dir.nr      = nr;
 
 	for (i = 0; i < nr; i++) {
 		struct perf_data_file *file = &files[i];
@@ -75,6 +77,10 @@ int perf_data__open_dir(struct perf_data *data)
 	if (WARN_ON(!data->is_dir))
 		return -EINVAL;
 
+	/* The version is provided by DIR_FORMAT feature. */
+	if (WARN_ON(data->dir.version != PERF_DIR_VERSION))
+		return -1;
+
 	dir = opendir(data->path);
 	if (!dir)
 		return -EINVAL;
diff --git a/tools/perf/util/data.h b/tools/perf/util/data.h
index d342469bdfda..6aef8746469f 100644
--- a/tools/perf/util/data.h
+++ b/tools/perf/util/data.h
@@ -24,6 +24,7 @@ struct perf_data {
 	enum perf_data_mode	 mode;
 
 	struct {
+		u64			 version;
 		struct perf_data_file	*files;
 		int			 nr;
 	} dir;
diff --git a/tools/perf/util/header.c b/tools/perf/util/header.c
index 01b324c275b9..b0683bf4d9f3 100644
--- a/tools/perf/util/header.c
+++ b/tools/perf/util/header.c
@@ -861,6 +861,21 @@ static int write_clockid(struct feat_fd *ff,
 			sizeof(ff->ph->env.clockid_res_ns));
 }
 
+static int write_dir_format(struct feat_fd *ff,
+			    struct perf_evlist *evlist __maybe_unused)
+{
+	struct perf_session *session;
+	struct perf_data *data;
+
+	session = container_of(ff->ph, struct perf_session, header);
+	data = session->data;
+
+	if (WARN_ON(!perf_data__is_dir(data)))
+		return -1;
+
+	return do_write(ff, &data->dir.version, sizeof(data->dir.version));
+}
+
 static int cpu_cache_level__sort(const void *a, const void *b)
 {
 	struct cpu_cache_level *cache_a = (struct cpu_cache_level *)a;
@@ -1341,6 +1356,17 @@ static void print_clockid(struct feat_fd *ff, FILE *fp)
 		ff->ph->env.clockid_res_ns * 1000);
 }
 
+static void print_dir_format(struct feat_fd *ff, FILE *fp)
+{
+	struct perf_session *session;
+	struct perf_data *data;
+
+	session = container_of(ff->ph, struct perf_session, header);
+	data = session->data;
+
+	fprintf(fp, "# directory data version : %"PRIu64"\n", data->dir.version);
+}
+
 static void free_event_desc(struct perf_evsel *events)
 {
 	struct perf_evsel *evsel;
@@ -2373,6 +2399,21 @@ static int process_clockid(struct feat_fd *ff,
 	return 0;
 }
 
+static int process_dir_format(struct feat_fd *ff,
+			      void *_data __maybe_unused)
+{
+	struct perf_session *session;
+	struct perf_data *data;
+
+	session = container_of(ff->ph, struct perf_session, header);
+	data = session->data;
+
+	if (WARN_ON(!perf_data__is_dir(data)))
+		return -1;
+
+	return do_read_u64(ff, &data->dir.version);
+}
+
 struct feature_ops {
 	int (*write)(struct feat_fd *ff, struct perf_evlist *evlist);
 	void (*print)(struct feat_fd *ff, FILE *fp);
@@ -2432,7 +2473,8 @@ static const struct feature_ops feat_ops[HEADER_LAST_FEATURE] = {
 	FEAT_OPN(CACHE,		cache,		true),
 	FEAT_OPR(SAMPLE_TIME,	sample_time,	false),
 	FEAT_OPR(MEM_TOPOLOGY,	mem_topology,	true),
-	FEAT_OPR(CLOCKID,       clockid,        false)
+	FEAT_OPR(CLOCKID,	clockid,	false),
+	FEAT_OPN(DIR_FORMAT,	dir_format,	false)
 };
 
 struct header_print_data {
diff --git a/tools/perf/util/header.h b/tools/perf/util/header.h
index 0d553ddca0a3..6a231340238d 100644
--- a/tools/perf/util/header.h
+++ b/tools/perf/util/header.h
@@ -39,6 +39,7 @@ enum {
 	HEADER_SAMPLE_TIME,
 	HEADER_MEM_TOPOLOGY,
 	HEADER_CLOCKID,
+	HEADER_DIR_FORMAT,
 	HEADER_LAST_FEATURE,
 	HEADER_FEAT_BITS	= 256,
 };
@@ -48,6 +49,10 @@ enum perf_header_version {
 	PERF_HEADER_VERSION_2,
 };
 
+enum perf_dir_version {
+	PERF_DIR_VERSION	= 1,
+};
+
 struct perf_file_section {
 	u64 offset;
 	u64 size;
-- 
2.17.2


^ permalink raw reply	[flat|nested] 27+ messages in thread

* [PATCH 06/11] perf session: Add process callback to reader object
  2019-03-08 13:47 [PATCHv5 00/11] perf record: Add support to store data in directory Jiri Olsa
                   ` (4 preceding siblings ...)
  2019-03-08 13:47 ` [PATCH 05/11] perf header: Add DIR_FORMAT feature to describe directory data Jiri Olsa
@ 2019-03-08 13:47 ` Jiri Olsa
  2019-03-22 22:06   ` [tip:perf/urgent] " tip-bot for Jiri Olsa
  2019-03-08 13:47 ` [PATCH 07/11] perf session: Add __perf_session__process_dir_events function Jiri Olsa
                   ` (4 subsequent siblings)
  10 siblings, 1 reply; 27+ messages in thread
From: Jiri Olsa @ 2019-03-08 13:47 UTC (permalink / raw)
  To: Arnaldo Carvalho de Melo
  Cc: lkml, Ingo Molnar, Namhyung Kim, Alexander Shishkin,
	Peter Zijlstra, Adrian Hunter, Andi Kleen, Stephane Eranian,
	Alexey Budankov

Adding callback function to reader object so
callers can process data in different ways.

Link: http://lkml.kernel.org/n/tip-8g1islzz6xkl36tz0z1nkuff@git.kernel.org
Signed-off-by: Jiri Olsa <jolsa@kernel.org>
---
 tools/perf/util/session.c | 23 +++++++++++++++++++----
 1 file changed, 19 insertions(+), 4 deletions(-)

diff --git a/tools/perf/util/session.c b/tools/perf/util/session.c
index de777bdc0ed3..0ec34227bd60 100644
--- a/tools/perf/util/session.c
+++ b/tools/perf/util/session.c
@@ -1847,10 +1847,17 @@ fetch_mmaped_event(struct perf_session *session,
 #define NUM_MMAPS 128
 #endif
 
+struct reader;
+
+typedef s64 (*reader_cb_t)(struct perf_session *session,
+			   union perf_event *event,
+			   u64 file_offset);
+
 struct reader {
-	int	fd;
-	u64	data_size;
-	u64	data_offset;
+	int		 fd;
+	u64		 data_size;
+	u64		 data_offset;
+	reader_cb_t	 process;
 };
 
 static int
@@ -1921,7 +1928,7 @@ reader__process_events(struct reader *rd, struct perf_session *session,
 	size = event->header.size;
 
 	if (size < sizeof(struct perf_event_header) ||
-	    (skip = perf_session__process_event(session, event, file_pos)) < 0) {
+	    (skip = rd->process(session, event, file_pos)) < 0) {
 		pr_err("%#" PRIx64 " [%#x]: failed to process type: %d\n",
 		       file_offset + head, event->header.size,
 		       event->header.type);
@@ -1947,12 +1954,20 @@ reader__process_events(struct reader *rd, struct perf_session *session,
 	return err;
 }
 
+static s64 process_simple(struct perf_session *session,
+			  union perf_event *event,
+			  u64 file_offset)
+{
+	return perf_session__process_event(session, event, file_offset);
+}
+
 static int __perf_session__process_events(struct perf_session *session)
 {
 	struct reader rd = {
 		.fd		= perf_data__fd(session->data),
 		.data_size	= session->header.data_size,
 		.data_offset	= session->header.data_offset,
+		.process	= process_simple,
 	};
 	struct ordered_events *oe = &session->ordered_events;
 	struct perf_tool *tool = session->tool;
-- 
2.17.2


^ permalink raw reply	[flat|nested] 27+ messages in thread

* [PATCH 07/11] perf session: Add __perf_session__process_dir_events function
  2019-03-08 13:47 [PATCHv5 00/11] perf record: Add support to store data in directory Jiri Olsa
                   ` (5 preceding siblings ...)
  2019-03-08 13:47 ` [PATCH 06/11] perf session: Add process callback to reader object Jiri Olsa
@ 2019-03-08 13:47 ` Jiri Olsa
  2019-03-08 18:38   ` Arnaldo Carvalho de Melo
  2019-03-08 13:47 ` [PATCH 08/11] perf session: Add path to reader object Jiri Olsa
                   ` (3 subsequent siblings)
  10 siblings, 1 reply; 27+ messages in thread
From: Jiri Olsa @ 2019-03-08 13:47 UTC (permalink / raw)
  To: Arnaldo Carvalho de Melo
  Cc: lkml, Ingo Molnar, Namhyung Kim, Alexander Shishkin,
	Peter Zijlstra, Adrian Hunter, Andi Kleen, Stephane Eranian,
	Alexey Budankov

Adding __perf_session__process_dir_events function
to process events over the directory data.

All directory events are pushed into sessions ordered
data and flushed for processing.

Link: http://lkml.kernel.org/n/tip-n3zl0wo3z18tatv5x7epmjrh@git.kernel.org
Signed-off-by: Jiri Olsa <jolsa@kernel.org>
---
 tools/perf/util/session.c | 88 ++++++++++++++++++++++++++++++++++++++-
 1 file changed, 86 insertions(+), 2 deletions(-)

diff --git a/tools/perf/util/session.c b/tools/perf/util/session.c
index 0ec34227bd60..b55f4281b1da 100644
--- a/tools/perf/util/session.c
+++ b/tools/perf/util/session.c
@@ -1876,8 +1876,6 @@ reader__process_events(struct reader *rd, struct perf_session *session,
 	file_offset = page_offset;
 	head = rd->data_offset - page_offset;
 
-	ui_progress__init_size(prog, data_size, "Processing events...");
-
 	data_size += rd->data_offset;
 
 	mmap_size = MMAP_SIZE;
@@ -2006,6 +2004,89 @@ static int __perf_session__process_events(struct perf_session *session)
 	return err;
 }
 
+static s64 process_index(struct perf_session *session,
+			 union perf_event *event,
+			 u64 file_offset)
+{
+	struct perf_evlist *evlist = session->evlist;
+	u64 timestamp;
+	s64 ret;
+
+	if (session->header.needs_swap)
+		event_swap(event, perf_evlist__sample_id_all(evlist));
+
+	if (event->header.type >= PERF_RECORD_HEADER_MAX)
+		return -EINVAL;
+
+	events_stats__inc(&evlist->stats, event->header.type);
+
+	if (event->header.type >= PERF_RECORD_USER_TYPE_START)
+		return perf_session__process_user_event(session, event, file_offset);
+
+	ret = perf_evlist__parse_sample_timestamp(evlist, event, &timestamp);
+	if (ret)
+		return ret;
+
+	return ordered_events__queue(&session->ordered_events, event,
+				     timestamp, file_offset);
+}
+
+static int __perf_session__process_dir_events(struct perf_session *session)
+{
+	struct perf_data *data = session->data;
+	struct perf_tool *tool = session->tool;
+	struct reader rd = {
+		.fd		= perf_data__fd(session->data),
+		.data_size	= session->header.data_size,
+		.data_offset	= session->header.data_offset,
+		.process	= process_simple,
+	};
+	int i, ret = 0;
+	struct ui_progress prog;
+	u64 total_size = perf_data__size(session->data);
+
+	perf_tool__fill_defaults(tool);
+
+	ui_progress__init_size(&prog, total_size, "Processing events...");
+
+	/* Read data from the header file.. */
+	ret = reader__process_events(&rd, session, &prog);
+	if (ret)
+		goto out_err;
+
+	/* ... and continue with data files. */
+	for (i = 0; i < data->dir.nr ; i++) {
+		struct perf_data_file *file = &data->dir.files[i];
+
+		if (file->size == 0)
+			continue;
+
+		rd = (struct reader) {
+			.fd		= file->fd,
+			.data_size	= file->size,
+			.data_offset	= 0,
+			.process	= process_index,
+		};
+
+		ret = reader__process_events(&rd, session, &prog);
+		if (ret)
+			goto out_err;
+	}
+
+	ret = ordered_events__flush(&session->ordered_events, OE_FLUSH__FINAL);
+
+out_err:
+	if (!tool->no_warn)
+		perf_session__warn_about_errors(session);
+
+	/*
+	 * We may switching perf.data output, make ordered_events
+	 * reusable.
+	 */
+	ordered_events__reinit(&session->ordered_events);
+	return ret;
+}
+
 int perf_session__process_events(struct perf_session *session)
 {
 	if (perf_session__register_idle_thread(session) < 0)
@@ -2014,6 +2095,9 @@ int perf_session__process_events(struct perf_session *session)
 	if (perf_data__is_pipe(session->data))
 		return __perf_session__process_pipe_events(session);
 
+	if (perf_data__is_dir(session->data))
+		return __perf_session__process_dir_events(session);
+
 	return __perf_session__process_events(session);
 }
 
-- 
2.17.2


^ permalink raw reply	[flat|nested] 27+ messages in thread

* [PATCH 08/11] perf session: Add path to reader object
  2019-03-08 13:47 [PATCHv5 00/11] perf record: Add support to store data in directory Jiri Olsa
                   ` (6 preceding siblings ...)
  2019-03-08 13:47 ` [PATCH 07/11] perf session: Add __perf_session__process_dir_events function Jiri Olsa
@ 2019-03-08 13:47 ` Jiri Olsa
  2019-03-08 13:47 ` [PATCH 09/11] perf record: Add --dir option to store data in directory Jiri Olsa
                   ` (2 subsequent siblings)
  10 siblings, 0 replies; 27+ messages in thread
From: Jiri Olsa @ 2019-03-08 13:47 UTC (permalink / raw)
  To: Arnaldo Carvalho de Melo
  Cc: lkml, Ingo Molnar, Namhyung Kim, Alexander Shishkin,
	Peter Zijlstra, Adrian Hunter, Andi Kleen, Stephane Eranian,
	Alexey Budankov

Adding path to reader object, so we can display file
the processing fails for (string in [] brackets).

  $ perf report --stdio
  0x5e0 [perf.data/data.3] [0xa200]: failed to process type: -1577027574

Link: http://lkml.kernel.org/n/tip-4bjnoy4sln7adqtd3505q29q@git.kernel.org
Signed-off-by: Jiri Olsa <jolsa@kernel.org>
---
 tools/perf/util/session.c | 10 ++++++++--
 1 file changed, 8 insertions(+), 2 deletions(-)

diff --git a/tools/perf/util/session.c b/tools/perf/util/session.c
index b55f4281b1da..33f363caaa7c 100644
--- a/tools/perf/util/session.c
+++ b/tools/perf/util/session.c
@@ -1858,6 +1858,7 @@ struct reader {
 	u64		 data_size;
 	u64		 data_offset;
 	reader_cb_t	 process;
+	char		*path;
 };
 
 static int
@@ -1872,6 +1873,8 @@ reader__process_events(struct reader *rd, struct perf_session *session,
 	union perf_event *event;
 	s64 skip;
 
+	pr_debug("reader processing %s\n", rd->path);
+
 	page_offset = page_size * (rd->data_offset / page_size);
 	file_offset = page_offset;
 	head = rd->data_offset - page_offset;
@@ -1927,8 +1930,8 @@ reader__process_events(struct reader *rd, struct perf_session *session,
 
 	if (size < sizeof(struct perf_event_header) ||
 	    (skip = rd->process(session, event, file_pos)) < 0) {
-		pr_err("%#" PRIx64 " [%#x]: failed to process type: %d\n",
-		       file_offset + head, event->header.size,
+		pr_err("%#" PRIx64 " [%s] [%#x]: failed to process type: %d\n",
+		       file_offset + head, rd->path, event->header.size,
 		       event->header.type);
 		err = -EINVAL;
 		goto out;
@@ -1966,6 +1969,7 @@ static int __perf_session__process_events(struct perf_session *session)
 		.data_size	= session->header.data_size,
 		.data_offset	= session->header.data_offset,
 		.process	= process_simple,
+		.path		= session->data->file.path,
 	};
 	struct ordered_events *oe = &session->ordered_events;
 	struct perf_tool *tool = session->tool;
@@ -2040,6 +2044,7 @@ static int __perf_session__process_dir_events(struct perf_session *session)
 		.data_size	= session->header.data_size,
 		.data_offset	= session->header.data_offset,
 		.process	= process_simple,
+		.path		= session->data->file.path,
 	};
 	int i, ret = 0;
 	struct ui_progress prog;
@@ -2066,6 +2071,7 @@ static int __perf_session__process_dir_events(struct perf_session *session)
 			.data_size	= file->size,
 			.data_offset	= 0,
 			.process	= process_index,
+			.path		= file->path,
 		};
 
 		ret = reader__process_events(&rd, session, &prog);
-- 
2.17.2


^ permalink raw reply	[flat|nested] 27+ messages in thread

* [PATCH 09/11] perf record: Add --dir option to store data in directory
  2019-03-08 13:47 [PATCHv5 00/11] perf record: Add support to store data in directory Jiri Olsa
                   ` (7 preceding siblings ...)
  2019-03-08 13:47 ` [PATCH 08/11] perf session: Add path to reader object Jiri Olsa
@ 2019-03-08 13:47 ` Jiri Olsa
  2019-03-08 13:47 ` [PATCH 10/11] perf record: Add --output-dir " Jiri Olsa
  2019-03-08 13:47 ` [PATCH 11/11] perf record: Describe perf.data directory format Jiri Olsa
  10 siblings, 0 replies; 27+ messages in thread
From: Jiri Olsa @ 2019-03-08 13:47 UTC (permalink / raw)
  To: Arnaldo Carvalho de Melo
  Cc: lkml, Ingo Molnar, Namhyung Kim, Alexander Shishkin,
	Peter Zijlstra, Adrian Hunter, Andi Kleen, Stephane Eranian,
	Alexey Budankov

Adding --dir option to store data in directory. It's next
step for multiple threads in record. It's now possible to
make directory data via --dir option, like:

  $ perf record --dir perf bench sched messaging
  $ ls -l perf.data
  total 344
  -rw-------. 1 jolsa jolsa 43864 Jan 20 22:26 data.0
  -rw-------. 1 jolsa jolsa 30464 Jan 20 22:26 data.1
  -rw-------. 1 jolsa jolsa 53816 Jan 20 22:26 data.2
  -rw-------. 1 jolsa jolsa 30368 Jan 20 22:26 data.3
  -rw-------. 1 jolsa jolsa 40088 Jan 20 22:26 data.4
  -rw-------. 1 jolsa jolsa 42592 Jan 20 22:26 data.5
  -rw-------. 1 jolsa jolsa 56136 Jan 20 22:26 data.6
  -rw-------. 1 jolsa jolsa 25992 Jan 20 22:26 data.7
  -rw-------. 1 jolsa jolsa  8832 Jan 20 22:26 header

There's a data file created for every cpu and it's storing
data for those cpu maps.

It's possible to transform directory data into standard
perf.data file via following inject command:

  $ perf inject -o perf.data.file -i perf.data

The --dir option enabled DIR_FORMAT feature to be stored
in header file to indicate the directory layout.

Don't allow to use --dir with --aio yet. It needs
to be investigated first.

Link: http://lkml.kernel.org/n/tip-0kjm8wpglzu2tm18tpagfm4d@git.kernel.org
Signed-off-by: Jiri Olsa <jolsa@kernel.org>
---
 tools/perf/Documentation/perf-record.txt |  3 +
 tools/perf/builtin-record.c              | 80 ++++++++++++++++++++++--
 tools/perf/util/mmap.h                   | 23 +++----
 3 files changed, 90 insertions(+), 16 deletions(-)

diff --git a/tools/perf/Documentation/perf-record.txt b/tools/perf/Documentation/perf-record.txt
index 8f0c2be34848..445b7a4eb130 100644
--- a/tools/perf/Documentation/perf-record.txt
+++ b/tools/perf/Documentation/perf-record.txt
@@ -524,6 +524,9 @@ config terms. For example: 'cycles/overwrite/' and 'instructions/no-overwrite/'.
 
 Implies --tail-synthesize.
 
+--dir::
+Store data into directory with one data file for cpu.
+
 SEE ALSO
 --------
 linkperf:perf-stat[1], linkperf:perf-list[1]
diff --git a/tools/perf/builtin-record.c b/tools/perf/builtin-record.c
index a468d882e74f..26981be13aa0 100644
--- a/tools/perf/builtin-record.c
+++ b/tools/perf/builtin-record.c
@@ -112,10 +112,13 @@ static bool switch_output_time(struct record *rec)
 	       trigger_is_ready(&switch_output_trigger);
 }
 
-static int record__write(struct record *rec, struct perf_mmap *map __maybe_unused,
+static int record__write(struct record *rec, struct perf_mmap *map,
 			 void *bf, size_t size)
 {
-	struct perf_data_file *file = &rec->session->data->file;
+	struct perf_data_file *file = &rec->data.file;
+
+	if (map && map->file)
+		file = map->file;
 
 	if (perf_data_file__write(file, bf, size) < 0) {
 		pr_err("failed to write perf data, error: %m\n");
@@ -124,6 +127,15 @@ static int record__write(struct record *rec, struct perf_mmap *map __maybe_unuse
 
 	rec->bytes_written += size;
 
+	/*
+	 * Update header file size manualy, data files size are
+	 * ok to be updated by stat command, but header files
+	 * contains more stuff, so we need to track data size
+	 * manualy.
+	 */
+	if (file == &rec->data.file)
+		rec->session->header.data_size += size;
+
 	if (switch_output_size(rec))
 		trigger_hit(&switch_output_trigger);
 
@@ -247,6 +259,7 @@ static int record__aio_pushfn(void *to, struct aiocb *cblock, void *bf, size_t s
 	ret = record__aio_write(cblock, trace_fd, bf, size, off);
 	if (!ret) {
 		rec->bytes_written += size;
+		rec->session->header.data_size += size;
 		if (switch_output_size(rec))
 			trigger_hit(&switch_output_trigger);
 	}
@@ -564,6 +577,25 @@ static int record__mmap_evlist(struct record *rec,
 	return 0;
 }
 
+static int record__mmap_dir_data(struct record *rec)
+{
+	struct perf_evlist *evlist = rec->evlist;
+	struct perf_data *data = &rec->data;
+	int i, ret, nr = evlist->nr_mmaps;
+
+	ret = perf_data__create_dir(data, nr);
+	if (ret)
+		return ret;
+
+	for (i = 0; i < nr; i++) {
+		struct perf_mmap *map = &evlist->mmap[i];
+
+		map->file = &data->dir.files[i];
+	}
+
+	return 0;
+}
+
 static int record__mmap(struct record *rec)
 {
 	return record__mmap_evlist(rec, rec->evlist);
@@ -793,8 +825,12 @@ static int record__mmap_read_evlist(struct record *rec, struct perf_evlist *evli
 	/*
 	 * Mark the round finished in case we wrote
 	 * at least one event.
+	 *
+	 * No need for round events in directory mode,
+	 * because per-cpu files/maps have sorted data
+	 * from kernel.
 	 */
-	if (bytes_written != rec->bytes_written)
+	if (!perf_data__is_dir(&rec->data) && bytes_written != rec->bytes_written)
 		rc = record__write(rec, NULL, &finished_round_event, sizeof(finished_round_event));
 
 	if (overwrite)
@@ -837,7 +873,8 @@ static void record__init_features(struct record *rec)
 	if (!(rec->opts.use_clockid && rec->opts.clockid_res_ns))
 		perf_header__clear_feat(&session->header, HEADER_CLOCKID);
 
-	perf_header__clear_feat(&session->header, HEADER_DIR_FORMAT);
+	if (!perf_data__is_dir(session->data))
+		perf_header__clear_feat(&session->header, HEADER_DIR_FORMAT);
 
 	perf_header__clear_feat(&session->header, HEADER_STAT);
 }
@@ -851,9 +888,11 @@ record__finish_output(struct record *rec)
 	if (data->is_pipe)
 		return;
 
-	rec->session->header.data_size += rec->bytes_written;
 	data->file.size = lseek(perf_data__fd(data), 0, SEEK_CUR);
 
+	if (perf_data__is_dir(data))
+		perf_data__update_dir(data);
+
 	if (!rec->no_buildid) {
 		process_buildids(rec);
 
@@ -924,6 +963,12 @@ record__switch_output(struct record *rec, bool at_exit)
 
 	/* Output tracking events */
 	if (!at_exit) {
+		if (perf_data__is_dir(data)) {
+			err = record__mmap_dir_data(rec);
+			if (err)
+				return -1;
+		}
+
 		record__synthesize(rec, false);
 
 		/*
@@ -1173,11 +1218,23 @@ static int __cmd_record(struct record *rec, int argc, const char **argv)
 	if (data->is_pipe && rec->evlist->nr_entries == 1)
 		rec->opts.sample_id = true;
 
+	if (data->is_pipe && perf_data__is_dir(data)) {
+		pr_err("Directory output is not allowed for pipe output\n");
+		err = -1;
+		goto out_child;
+	}
+
 	if (record__open(rec) != 0) {
 		err = -1;
 		goto out_child;
 	}
 
+	if (perf_data__is_dir(data)) {
+		err = record__mmap_dir_data(rec);
+		if (err)
+			goto out_child;
+	}
+
 	err = bpf__apply_obj_config();
 	if (err) {
 		char errbuf[BUFSIZ];
@@ -1983,6 +2040,8 @@ static struct option __record_options[] = {
 	OPT_CALLBACK(0, "affinity", &record.opts, "node|cpu",
 		     "Set affinity mask of trace reading thread to NUMA node cpu mask or cpu of processed mmap buffer",
 		     record__parse_affinity),
+	OPT_BOOLEAN(0, "dir", &record.data.is_dir,
+		    "Store data into directory perf.data"),
 	OPT_END()
 };
 
@@ -2134,6 +2193,17 @@ int cmd_record(int argc, const char **argv)
 		goto out;
 	}
 
+	if (perf_data__is_dir(&rec->data)) {
+		if (!rec->opts.sample_time) {
+			pr_err("Sample timestamp is required for indexing\n");
+			goto out;
+		}
+		if (record__aio_enabled(rec)) {
+			pr_err("Cannot use both --dir and --aio yet.\n");
+			goto out;
+		}
+	}
+
 	if (rec->opts.target.tid && !rec->opts.no_inherit_set)
 		rec->opts.no_inherit = true;
 
diff --git a/tools/perf/util/mmap.h b/tools/perf/util/mmap.h
index e566c19b242b..3e8595a8d6ce 100644
--- a/tools/perf/util/mmap.h
+++ b/tools/perf/util/mmap.h
@@ -19,17 +19,18 @@ struct aiocb;
  * @refcnt - e.g. code using PERF_EVENT_IOC_SET_OUTPUT to share this
  */
 struct perf_mmap {
-	void		 *base;
-	int		 mask;
-	int		 fd;
-	int		 cpu;
-	refcount_t	 refcnt;
-	u64		 prev;
-	u64		 start;
-	u64		 end;
-	bool		 overwrite;
-	struct auxtrace_mmap auxtrace_mmap;
-	char		 event_copy[PERF_SAMPLE_MAX_SIZE] __aligned(8);
+	void			 *base;
+	int			 mask;
+	int			 fd;
+	int			 cpu;
+	refcount_t		 refcnt;
+	u64			 prev;
+	u64			 start;
+	u64			 end;
+	bool			 overwrite;
+	struct auxtrace_mmap	 auxtrace_mmap;
+	struct perf_data_file	*file;
+	char			 event_copy[PERF_SAMPLE_MAX_SIZE] __aligned(8);
 #ifdef HAVE_AIO_SUPPORT
 	struct {
 		void		 **data;
-- 
2.17.2


^ permalink raw reply	[flat|nested] 27+ messages in thread

* [PATCH 10/11] perf record: Add --output-dir option to store data in directory
  2019-03-08 13:47 [PATCHv5 00/11] perf record: Add support to store data in directory Jiri Olsa
                   ` (8 preceding siblings ...)
  2019-03-08 13:47 ` [PATCH 09/11] perf record: Add --dir option to store data in directory Jiri Olsa
@ 2019-03-08 13:47 ` Jiri Olsa
  2019-03-08 13:47 ` [PATCH 11/11] perf record: Describe perf.data directory format Jiri Olsa
  10 siblings, 0 replies; 27+ messages in thread
From: Jiri Olsa @ 2019-03-08 13:47 UTC (permalink / raw)
  To: Arnaldo Carvalho de Melo
  Cc: lkml, Ingo Molnar, Namhyung Kim, Alexander Shishkin,
	Peter Zijlstra, Adrian Hunter, Andi Kleen, Stephane Eranian,
	Alexey Budankov

Adding --output-dir option to mimic -o and --dir options.
following commands do the same:

  $ perf record -o perf.dir.data --dir ...
  $ perf record --output-dir perf.dir.data ...

User cannot use both -o and output-dir together,
error is displayed.

Link: http://lkml.kernel.org/n/tip-76ldd2ss6vjvlnjgwy7wxfzt@git.kernel.org
Signed-off-by: Jiri Olsa <jolsa@kernel.org>
---
 tools/lib/subcmd/parse-options.h         |  4 ++++
 tools/perf/Documentation/perf-record.txt |  3 +++
 tools/perf/builtin-record.c              | 13 +++++++++++--
 3 files changed, 18 insertions(+), 2 deletions(-)

diff --git a/tools/lib/subcmd/parse-options.h b/tools/lib/subcmd/parse-options.h
index af9def589863..8a3be77a3346 100644
--- a/tools/lib/subcmd/parse-options.h
+++ b/tools/lib/subcmd/parse-options.h
@@ -146,6 +146,10 @@ struct option {
 	  .value = check_vtype(v, const char **), .argh = (a), .help = (h), \
 	  .flags = PARSE_OPT_OPTARG, .defval = (intptr_t)(d), \
 	  .set = check_vtype(os, bool *)}
+#define OPT_STRING_SET(s, l, v, os, a, h) \
+	{ .type = OPTION_STRING, .short_name = (s), .long_name = (l), \
+	  .value = check_vtype(v, const char **), .argh = (a), .help = (h), \
+	  .set = check_vtype(os, bool *)}
 #define OPT_STRING_NOEMPTY(s, l, v, a, h)   { .type = OPTION_STRING,  .short_name = (s), .long_name = (l), .value = check_vtype(v, const char **), .argh = (a), .help = (h), .flags = PARSE_OPT_NOEMPTY}
 #define OPT_DATE(s, l, v, h) \
 	{ .type = OPTION_CALLBACK, .short_name = (s), .long_name = (l), .value = (v), .argh = "time", .help = (h), .callback = parse_opt_approxidate_cb }
diff --git a/tools/perf/Documentation/perf-record.txt b/tools/perf/Documentation/perf-record.txt
index 445b7a4eb130..aac609887fb7 100644
--- a/tools/perf/Documentation/perf-record.txt
+++ b/tools/perf/Documentation/perf-record.txt
@@ -527,6 +527,9 @@ Implies --tail-synthesize.
 --dir::
 Store data into directory with one data file for cpu.
 
+--output-dir::
+Same as --dir option, can't be used together with -o option.
+
 SEE ALSO
 --------
 linkperf:perf-stat[1], linkperf:perf-list[1]
diff --git a/tools/perf/builtin-record.c b/tools/perf/builtin-record.c
index 26981be13aa0..115316e94b34 100644
--- a/tools/perf/builtin-record.c
+++ b/tools/perf/builtin-record.c
@@ -80,6 +80,7 @@ struct record {
 	bool			buildid_all;
 	bool			timestamp_filename;
 	bool			timestamp_boundary;
+	bool			output_is_file;
 	struct switch_output	switch_output;
 	unsigned long long	samples;
 	cpu_set_t		affinity_mask;
@@ -1921,8 +1922,10 @@ static struct option __record_options[] = {
 	OPT_STRING('C', "cpu", &record.opts.target.cpu_list, "cpu",
 		    "list of cpus to monitor"),
 	OPT_U64('c', "count", &record.opts.user_interval, "event period to sample"),
-	OPT_STRING('o', "output", &record.data.path, "file",
-		    "output file name"),
+	OPT_STRING_SET('o', "output", &record.data.path, &record.output_is_file,
+		       "file", "output file name"),
+	OPT_STRING_SET(0, "output-dir", &record.data.path, &record.data.is_dir,
+		       "file", "output directory name"),
 	OPT_BOOLEAN_SET('i', "no-inherit", &record.opts.no_inherit,
 			&record.opts.no_inherit_set,
 			"child tasks do not inherit counters"),
@@ -2101,6 +2104,12 @@ int cmd_record(int argc, const char **argv)
 			"cgroup monitoring only available in system-wide mode");
 
 	}
+
+	if (perf_data__is_dir(&rec->data) && record.output_is_file) {
+		ui__error("cannot use both -o and --output-dir\n");
+		return -EINVAL;
+	}
+
 	if (rec->opts.record_switch_events &&
 	    !perf_can_record_switch_events()) {
 		ui__error("kernel does not support recording context switch events\n");
-- 
2.17.2


^ permalink raw reply	[flat|nested] 27+ messages in thread

* [PATCH 11/11] perf record: Describe perf.data directory format
  2019-03-08 13:47 [PATCHv5 00/11] perf record: Add support to store data in directory Jiri Olsa
                   ` (9 preceding siblings ...)
  2019-03-08 13:47 ` [PATCH 10/11] perf record: Add --output-dir " Jiri Olsa
@ 2019-03-08 13:47 ` Jiri Olsa
  10 siblings, 0 replies; 27+ messages in thread
From: Jiri Olsa @ 2019-03-08 13:47 UTC (permalink / raw)
  To: Arnaldo Carvalho de Melo
  Cc: lkml, Ingo Molnar, Namhyung Kim, Alexander Shishkin,
	Peter Zijlstra, Adrian Hunter, Andi Kleen, Stephane Eranian,
	Alexey Budankov

Adding perf.data-directory-format.txt to describe the
directory data layout.

Link: http://lkml.kernel.org/n/tip-1c8u1thx63v2ldwfdas4xc5d@git.kernel.org
Signed-off-by: Jiri Olsa <jolsa@kernel.org>
---
 .../perf.data-directory-format.txt            | 54 +++++++++++++++++++
 1 file changed, 54 insertions(+)
 create mode 100644 tools/perf/Documentation/perf.data-directory-format.txt

diff --git a/tools/perf/Documentation/perf.data-directory-format.txt b/tools/perf/Documentation/perf.data-directory-format.txt
new file mode 100644
index 000000000000..bbd6d31b10c8
--- /dev/null
+++ b/tools/perf/Documentation/perf.data-directory-format.txt
@@ -0,0 +1,54 @@
+perf.data directory format
+
+DISCLAIMER This is not ABI yet and is subject to possible change
+           in following versions of perf. We will remove this
+           disclaimer once the directory format soaks in.
+
+
+This document describes the on-disk perf.data format, generated
+by perf record with --dir option and consumed by the other perf
+tools.
+
+The directory perf.data is created by perf record command by
+using the --dir option:
+
+  $ perf record --dir perf bench sched messaging
+  $ ls -l perf.data
+      total 344
+      -rw-------. 1 jolsa jolsa 43864 Jan 20 22:26 data.0
+      -rw-------. 1 jolsa jolsa 30464 Jan 20 22:26 data.1
+      -rw-------. 1 jolsa jolsa 53816 Jan 20 22:26 data.2
+      -rw-------. 1 jolsa jolsa 30368 Jan 20 22:26 data.3
+      -rw-------. 1 jolsa jolsa 40088 Jan 20 22:26 data.4
+      -rw-------. 1 jolsa jolsa 42592 Jan 20 22:26 data.5
+      -rw-------. 1 jolsa jolsa 56136 Jan 20 22:26 data.6
+      -rw-------. 1 jolsa jolsa 25992 Jan 20 22:26 data.7
+      -rw-------. 1 jolsa jolsa  8832 Jan 20 22:26 header
+
+The header file keeps the standard perf.data file header,
+and the data.* files keep data.
+
+header file
+-----------
+The header file following the standard format describe in
+Documentation/perf.data-file-format doc. Including its data
+portion that is used to store manually synthesized events.
+
+data file
+---------
+The data files layout is described by HEADER_DIR_FORMAT feature.
+Currently it holds only version number (1):
+
+  HEADER_DIR_FORMAT = 24
+
+  struct {
+     uint64_t version;
+  }
+
+The current only only version value 1 means that data files:
+  - follow the 'data.*' format
+  - contain raw events data in standard perf format as read
+    from kernel (and need to be sorted)
+
+Future versions are expected to describe different data files
+layout according to special needs.
-- 
2.17.2


^ permalink raw reply	[flat|nested] 27+ messages in thread

* Re: [PATCH 01/11] perf data: Add directory support
  2019-03-08 13:47 ` [PATCH 01/11] perf data: Add directory support Jiri Olsa
@ 2019-03-08 17:26   ` Arnaldo Carvalho de Melo
  2019-03-22 22:03   ` [tip:perf/urgent] perf data: Support having perf.data stored as a directory tip-bot for Jiri Olsa
  1 sibling, 0 replies; 27+ messages in thread
From: Arnaldo Carvalho de Melo @ 2019-03-08 17:26 UTC (permalink / raw)
  To: Jiri Olsa
  Cc: lkml, Ingo Molnar, Namhyung Kim, Alexander Shishkin,
	Peter Zijlstra, Adrian Hunter, Andi Kleen, Stephane Eranian,
	Alexey Budankov

Em Fri, Mar 08, 2019 at 02:47:35PM +0100, Jiri Olsa escreveu:
> Adding support to have directory as perf.data.
> 
> The caller needs to set 'struct perf_data::is_dir flag
> and the path will be treated as directory.
> 
> The 'struct perf_data::file' is initialized and open
> as 'path/header' file.
> 
> Adding check to direcory interface functions to check
> on is_dir flag.
> 
> Link: http://lkml.kernel.org/n/tip-pvot1aywiem9epgqpfi1agaj@git.kernel.org
> Signed-off-by: Jiri Olsa <jolsa@kernel.org>
> ---
>  tools/perf/util/data.c    | 49 ++++++++++++++++++++++++++++++++++++++-
>  tools/perf/util/data.h    |  6 +++++
>  tools/perf/util/session.c |  4 ++++
>  3 files changed, 58 insertions(+), 1 deletion(-)
> 
> diff --git a/tools/perf/util/data.c b/tools/perf/util/data.c
> index e098e189f93e..e28fd0fde5e0 100644
> --- a/tools/perf/util/data.c
> +++ b/tools/perf/util/data.c
> @@ -34,6 +34,9 @@ int perf_data__create_dir(struct perf_data *data, int nr)
>  	struct perf_data_file *files = NULL;
>  	int i, ret = -1;
>  
> +	if (WARN_ON(!data->is_dir))
> +		return -EINVAL;
> +
>  	files = zalloc(nr * sizeof(*files));
>  	if (!files)
>  		return -ENOMEM;
> @@ -69,6 +72,9 @@ int perf_data__open_dir(struct perf_data *data)
>  	DIR *dir;
>  	int nr = 0;
>  
> +	if (WARN_ON(!data->is_dir))
> +		return -EINVAL;
> +
>  	dir = opendir(data->path);
>  	if (!dir)
>  		return -EINVAL;
> @@ -173,6 +179,16 @@ static int check_backup(struct perf_data *data)
>  	return 0;
>  }
>  
> +static bool is_dir(struct perf_data *data)
> +{
> +	struct stat st;
> +
> +	if (stat(data->path, &st))
> +		return false;
> +
> +	return (st.st_mode & S_IFMT) == S_IFDIR;
> +}
> +
>  static int open_file_read(struct perf_data *data)
>  {
>  	struct stat st;
> @@ -254,6 +270,30 @@ static int open_file_dup(struct perf_data *data)
>  	return open_file(data);
>  }
>  
> +static int open_dir(struct perf_data *data)
> +{
> +	int ret;
> +
> +	/*
> +	 * So far we open only the header, so we
> +	 * can read the data version and layout.
> +	 */
> +	if (asprintf(&data->file.path, "%s/header", data->path) < 0)
> +		return -ENOMEM;
> +
> +	if (perf_data__is_write(data) &&
> +	    mkdir(data->path, S_IRWXU) < 0)
> +		return -1;

Please be consistent, either -ESOMETHING or -1, I think returning -1 and
letting the caller look at errno should be enough, as asprintf() returns
straight away if the malloc it does fail, leaving errno set to ENOMEM.

I'm doing this change here.

> +
> +	ret = open_file(data);
> +
> +	/* Cleanup whatever we managed to create so far. */
> +	if (ret && perf_data__is_write(data))
> +		rm_rf_perf_data(data->path);
> +
> +	return ret;
> +}
> +
>  int perf_data__open(struct perf_data *data)
>  {
>  	if (check_pipe(data))
> @@ -265,11 +305,18 @@ int perf_data__open(struct perf_data *data)
>  	if (check_backup(data))
>  		return -1;
>  
> -	return open_file_dup(data);
> +	if (perf_data__is_read(data))
> +		data->is_dir = is_dir(data);
> +
> +	return perf_data__is_dir(data) ?
> +	       open_dir(data) : open_file_dup(data);
>  }
>  
>  void perf_data__close(struct perf_data *data)
>  {
> +	if (perf_data__is_dir(data))
> +		perf_data__close_dir(data);
> +
>  	zfree(&data->file.path);
>  	close(data->file.fd);
>  }
> diff --git a/tools/perf/util/data.h b/tools/perf/util/data.h
> index 14b47be2bd69..06aefeda311f 100644
> --- a/tools/perf/util/data.h
> +++ b/tools/perf/util/data.h
> @@ -19,6 +19,7 @@ struct perf_data {
>  	const char		*path;
>  	struct perf_data_file	 file;
>  	bool			 is_pipe;
> +	bool			 is_dir;
>  	bool			 force;
>  	enum perf_data_mode	 mode;
>  
> @@ -43,6 +44,11 @@ static inline int perf_data__is_pipe(struct perf_data *data)
>  	return data->is_pipe;
>  }
>  
> +static inline bool perf_data__is_dir(struct perf_data *data)
> +{
> +	return data->is_dir;
> +}
> +
>  static inline int perf_data__fd(struct perf_data *data)
>  {
>  	return data->file.fd;
> diff --git a/tools/perf/util/session.c b/tools/perf/util/session.c
> index db643f3c2b95..de777bdc0ed3 100644
> --- a/tools/perf/util/session.c
> +++ b/tools/perf/util/session.c
> @@ -152,6 +152,10 @@ struct perf_session *perf_session__new(struct perf_data *data,
>  			}
>  
>  			perf_evlist__init_trace_event_sample_raw(session->evlist);
> +
> +			/* Open the directory data. */
> +			if (data->is_dir && perf_data__open_dir(data))
> +				goto out_delete;
>  		}
>  	} else  {
>  		session->machines.host.env = &perf_env;
> -- 
> 2.17.2

-- 

- Arnaldo

^ permalink raw reply	[flat|nested] 27+ messages in thread

* Re: [PATCH 04/11] perf data: Make perf_data__size to work over directory
  2019-03-08 13:47 ` [PATCH 04/11] perf data: Make perf_data__size to work over directory Jiri Olsa
@ 2019-03-08 17:33   ` Arnaldo Carvalho de Melo
  2019-03-09 17:21     ` Jiri Olsa
  2019-03-22 22:05   ` [tip:perf/urgent] perf data: Make perf_data__size() " tip-bot for Jiri Olsa
  1 sibling, 1 reply; 27+ messages in thread
From: Arnaldo Carvalho de Melo @ 2019-03-08 17:33 UTC (permalink / raw)
  To: Jiri Olsa
  Cc: lkml, Ingo Molnar, Namhyung Kim, Alexander Shishkin,
	Peter Zijlstra, Adrian Hunter, Andi Kleen, Stephane Eranian,
	Alexey Budankov

Em Fri, Mar 08, 2019 at 02:47:38PM +0100, Jiri Olsa escreveu:
> Making perf_data__size to return proper size
> for directory data.
> 
> Link: http://lkml.kernel.org/n/tip-t4dm8cctat2ginmy2bb08xe8@git.kernel.org
> Signed-off-by: Jiri Olsa <jolsa@kernel.org>
> ---
>  tools/perf/util/data.c | 17 +++++++++++++++++
>  tools/perf/util/data.h |  6 +-----
>  2 files changed, 18 insertions(+), 5 deletions(-)
> 
> diff --git a/tools/perf/util/data.c b/tools/perf/util/data.c
> index d568975c5ee3..498c531e2cb9 100644
> --- a/tools/perf/util/data.c
> +++ b/tools/perf/util/data.c
> @@ -393,3 +393,20 @@ int perf_data__switch(struct perf_data *data,
>  	free(new_filepath);
>  	return ret;
>  }
> +
> +unsigned long perf_data__size(struct perf_data *data)
> +{
> +	u64 size = data->file.size;
> +	int i;
> +
> +	if (!data->is_dir)
> +		return size;

So you do the sum over dir.files[i] _plus_ what is in data->file.size,
what is in data->file.size? Zero for the dir case? Or some non-per-cpu
metadata?

- Arnaldo

> +
> +	for (i = 0; i < data->dir.nr; i++) {
> +		struct perf_data_file *file = &data->dir.files[i];
> +
> +		size += file->size;
> +	}
> +
> +	return size;
> +}
> diff --git a/tools/perf/util/data.h b/tools/perf/util/data.h
> index 0deeb1af9f54..d342469bdfda 100644
> --- a/tools/perf/util/data.h
> +++ b/tools/perf/util/data.h
> @@ -54,11 +54,6 @@ static inline int perf_data__fd(struct perf_data *data)
>  	return data->file.fd;
>  }
>  
> -static inline unsigned long perf_data__size(struct perf_data *data)
> -{
> -	return data->file.size;
> -}
> -
>  int perf_data__open(struct perf_data *data);
>  void perf_data__close(struct perf_data *data);
>  ssize_t perf_data__write(struct perf_data *data,
> @@ -80,4 +75,5 @@ int perf_data__create_dir(struct perf_data *data, int nr);
>  int perf_data__open_dir(struct perf_data *data);
>  void perf_data__close_dir(struct perf_data *data);
>  int perf_data__update_dir(struct perf_data *data);
> +unsigned long perf_data__size(struct perf_data *data);
>  #endif /* __PERF_DATA_H */
> -- 
> 2.17.2

-- 

- Arnaldo

^ permalink raw reply	[flat|nested] 27+ messages in thread

* Re: [PATCH 05/11] perf header: Add DIR_FORMAT feature to describe directory data
  2019-03-08 13:47 ` [PATCH 05/11] perf header: Add DIR_FORMAT feature to describe directory data Jiri Olsa
@ 2019-03-08 18:24   ` Arnaldo Carvalho de Melo
  2019-03-09 17:21     ` Jiri Olsa
  2019-03-22 22:05   ` [tip:perf/urgent] " tip-bot for Jiri Olsa
  1 sibling, 1 reply; 27+ messages in thread
From: Arnaldo Carvalho de Melo @ 2019-03-08 18:24 UTC (permalink / raw)
  To: Jiri Olsa
  Cc: lkml, Ingo Molnar, Namhyung Kim, Alexander Shishkin,
	Peter Zijlstra, Adrian Hunter, Andi Kleen, Stephane Eranian,
	Alexey Budankov

Em Fri, Mar 08, 2019 at 02:47:39PM +0100, Jiri Olsa escreveu:
> The data files layout is described by HEADER_DIR_FORMAT
> feature.  Currently it holds only version number (1):
> 
>      uint64_t version;
> 
> The current version holds only version value (1) means
> that data files:
>   - follow the 'data.*' name format
>   - contain raw events data in standard perf format as
>     read from kernel (and need to be sorted)

and need to be sorted when multiple files are processed, right?

- Arnaldo
 
> Future versions are expected to describe different data
> files layout according to special needs.
> 
> Link: http://lkml.kernel.org/n/tip-6l53s3oabwr1r4n6wke21d8g@git.kernel.org
> Signed-off-by: Jiri Olsa <jolsa@kernel.org>
> ---
>  tools/perf/builtin-record.c |  2 ++
>  tools/perf/util/data.c      | 10 +++++++--
>  tools/perf/util/data.h      |  1 +
>  tools/perf/util/header.c    | 44 ++++++++++++++++++++++++++++++++++++-
>  tools/perf/util/header.h    |  5 +++++
>  5 files changed, 59 insertions(+), 3 deletions(-)
> 
> diff --git a/tools/perf/builtin-record.c b/tools/perf/builtin-record.c
> index e983c8d71a79..a468d882e74f 100644
> --- a/tools/perf/builtin-record.c
> +++ b/tools/perf/builtin-record.c
> @@ -837,6 +837,8 @@ static void record__init_features(struct record *rec)
>  	if (!(rec->opts.use_clockid && rec->opts.clockid_res_ns))
>  		perf_header__clear_feat(&session->header, HEADER_CLOCKID);
>  
> +	perf_header__clear_feat(&session->header, HEADER_DIR_FORMAT);
> +
>  	perf_header__clear_feat(&session->header, HEADER_STAT);
>  }
>  
> diff --git a/tools/perf/util/data.c b/tools/perf/util/data.c
> index 498c531e2cb9..41a9f478f0f0 100644
> --- a/tools/perf/util/data.c
> +++ b/tools/perf/util/data.c
> @@ -14,6 +14,7 @@
>  #include "data.h"
>  #include "util.h"
>  #include "debug.h"
> +#include "header.h"
>  
>  static void close_dir(struct perf_data_file *files, int nr)
>  {
> @@ -41,8 +42,9 @@ int perf_data__create_dir(struct perf_data *data, int nr)
>  	if (!files)
>  		return -ENOMEM;
>  
> -	data->dir.files = files;
> -	data->dir.nr    = nr;
> +	data->dir.version = PERF_DIR_VERSION;
> +	data->dir.files   = files;
> +	data->dir.nr      = nr;
>  
>  	for (i = 0; i < nr; i++) {
>  		struct perf_data_file *file = &files[i];
> @@ -75,6 +77,10 @@ int perf_data__open_dir(struct perf_data *data)
>  	if (WARN_ON(!data->is_dir))
>  		return -EINVAL;
>  
> +	/* The version is provided by DIR_FORMAT feature. */
> +	if (WARN_ON(data->dir.version != PERF_DIR_VERSION))
> +		return -1;
> +
>  	dir = opendir(data->path);
>  	if (!dir)
>  		return -EINVAL;
> diff --git a/tools/perf/util/data.h b/tools/perf/util/data.h
> index d342469bdfda..6aef8746469f 100644
> --- a/tools/perf/util/data.h
> +++ b/tools/perf/util/data.h
> @@ -24,6 +24,7 @@ struct perf_data {
>  	enum perf_data_mode	 mode;
>  
>  	struct {
> +		u64			 version;
>  		struct perf_data_file	*files;
>  		int			 nr;
>  	} dir;
> diff --git a/tools/perf/util/header.c b/tools/perf/util/header.c
> index 01b324c275b9..b0683bf4d9f3 100644
> --- a/tools/perf/util/header.c
> +++ b/tools/perf/util/header.c
> @@ -861,6 +861,21 @@ static int write_clockid(struct feat_fd *ff,
>  			sizeof(ff->ph->env.clockid_res_ns));
>  }
>  
> +static int write_dir_format(struct feat_fd *ff,
> +			    struct perf_evlist *evlist __maybe_unused)
> +{
> +	struct perf_session *session;
> +	struct perf_data *data;
> +
> +	session = container_of(ff->ph, struct perf_session, header);
> +	data = session->data;
> +
> +	if (WARN_ON(!perf_data__is_dir(data)))
> +		return -1;
> +
> +	return do_write(ff, &data->dir.version, sizeof(data->dir.version));
> +}
> +
>  static int cpu_cache_level__sort(const void *a, const void *b)
>  {
>  	struct cpu_cache_level *cache_a = (struct cpu_cache_level *)a;
> @@ -1341,6 +1356,17 @@ static void print_clockid(struct feat_fd *ff, FILE *fp)
>  		ff->ph->env.clockid_res_ns * 1000);
>  }
>  
> +static void print_dir_format(struct feat_fd *ff, FILE *fp)
> +{
> +	struct perf_session *session;
> +	struct perf_data *data;
> +
> +	session = container_of(ff->ph, struct perf_session, header);
> +	data = session->data;
> +
> +	fprintf(fp, "# directory data version : %"PRIu64"\n", data->dir.version);
> +}
> +
>  static void free_event_desc(struct perf_evsel *events)
>  {
>  	struct perf_evsel *evsel;
> @@ -2373,6 +2399,21 @@ static int process_clockid(struct feat_fd *ff,
>  	return 0;
>  }
>  
> +static int process_dir_format(struct feat_fd *ff,
> +			      void *_data __maybe_unused)
> +{
> +	struct perf_session *session;
> +	struct perf_data *data;
> +
> +	session = container_of(ff->ph, struct perf_session, header);
> +	data = session->data;
> +
> +	if (WARN_ON(!perf_data__is_dir(data)))
> +		return -1;
> +
> +	return do_read_u64(ff, &data->dir.version);
> +}
> +
>  struct feature_ops {
>  	int (*write)(struct feat_fd *ff, struct perf_evlist *evlist);
>  	void (*print)(struct feat_fd *ff, FILE *fp);
> @@ -2432,7 +2473,8 @@ static const struct feature_ops feat_ops[HEADER_LAST_FEATURE] = {
>  	FEAT_OPN(CACHE,		cache,		true),
>  	FEAT_OPR(SAMPLE_TIME,	sample_time,	false),
>  	FEAT_OPR(MEM_TOPOLOGY,	mem_topology,	true),
> -	FEAT_OPR(CLOCKID,       clockid,        false)
> +	FEAT_OPR(CLOCKID,	clockid,	false),
> +	FEAT_OPN(DIR_FORMAT,	dir_format,	false)
>  };
>  
>  struct header_print_data {
> diff --git a/tools/perf/util/header.h b/tools/perf/util/header.h
> index 0d553ddca0a3..6a231340238d 100644
> --- a/tools/perf/util/header.h
> +++ b/tools/perf/util/header.h
> @@ -39,6 +39,7 @@ enum {
>  	HEADER_SAMPLE_TIME,
>  	HEADER_MEM_TOPOLOGY,
>  	HEADER_CLOCKID,
> +	HEADER_DIR_FORMAT,
>  	HEADER_LAST_FEATURE,
>  	HEADER_FEAT_BITS	= 256,
>  };
> @@ -48,6 +49,10 @@ enum perf_header_version {
>  	PERF_HEADER_VERSION_2,
>  };
>  
> +enum perf_dir_version {
> +	PERF_DIR_VERSION	= 1,
> +};
> +
>  struct perf_file_section {
>  	u64 offset;
>  	u64 size;
> -- 
> 2.17.2

-- 

- Arnaldo

^ permalink raw reply	[flat|nested] 27+ messages in thread

* Re: [PATCH 07/11] perf session: Add __perf_session__process_dir_events function
  2019-03-08 13:47 ` [PATCH 07/11] perf session: Add __perf_session__process_dir_events function Jiri Olsa
@ 2019-03-08 18:38   ` Arnaldo Carvalho de Melo
  2019-03-08 18:39     ` Arnaldo Carvalho de Melo
  2019-03-09 17:21     ` Jiri Olsa
  0 siblings, 2 replies; 27+ messages in thread
From: Arnaldo Carvalho de Melo @ 2019-03-08 18:38 UTC (permalink / raw)
  To: Jiri Olsa
  Cc: lkml, Ingo Molnar, Namhyung Kim, Alexander Shishkin,
	Peter Zijlstra, Adrian Hunter, Andi Kleen, Stephane Eranian,
	Alexey Budankov

Em Fri, Mar 08, 2019 at 02:47:41PM +0100, Jiri Olsa escreveu:
> Adding __perf_session__process_dir_events function
> to process events over the directory data.
> 
> All directory events are pushed into sessions ordered
> data and flushed for processing.
> 
> Link: http://lkml.kernel.org/n/tip-n3zl0wo3z18tatv5x7epmjrh@git.kernel.org
> Signed-off-by: Jiri Olsa <jolsa@kernel.org>
> ---
>  tools/perf/util/session.c | 88 ++++++++++++++++++++++++++++++++++++++-
>  1 file changed, 86 insertions(+), 2 deletions(-)
> 
> diff --git a/tools/perf/util/session.c b/tools/perf/util/session.c
> index 0ec34227bd60..b55f4281b1da 100644
> --- a/tools/perf/util/session.c
> +++ b/tools/perf/util/session.c
> @@ -1876,8 +1876,6 @@ reader__process_events(struct reader *rd, struct perf_session *session,
>  	file_offset = page_offset;
>  	head = rd->data_offset - page_offset;
>  
> -	ui_progress__init_size(prog, data_size, "Processing events...");
> -
>  	data_size += rd->data_offset;
>  
>  	mmap_size = MMAP_SIZE;
> @@ -2006,6 +2004,89 @@ static int __perf_session__process_events(struct perf_session *session)
>  	return err;
>  }
>  
> +static s64 process_index(struct perf_session *session,
> +			 union perf_event *event,
> +			 u64 file_offset)
> +{
> +	struct perf_evlist *evlist = session->evlist;
> +	u64 timestamp;
> +	s64 ret;
> +
> +	if (session->header.needs_swap)
> +		event_swap(event, perf_evlist__sample_id_all(evlist));
> +
> +	if (event->header.type >= PERF_RECORD_HEADER_MAX)
> +		return -EINVAL;
> +
> +	events_stats__inc(&evlist->stats, event->header.type);
> +
> +	if (event->header.type >= PERF_RECORD_USER_TYPE_START)
> +		return perf_session__process_user_event(session, event, file_offset);
> +
> +	ret = perf_evlist__parse_sample_timestamp(evlist, event, &timestamp);
> +	if (ret)
> +		return ret;
> +
> +	return ordered_events__queue(&session->ordered_events, event,
> +				     timestamp, file_offset);
> +}
> +
> +static int __perf_session__process_dir_events(struct perf_session *session)
> +{
> +	struct perf_data *data = session->data;
> +	struct perf_tool *tool = session->tool;
> +	struct reader rd = {
> +		.fd		= perf_data__fd(session->data),
> +		.data_size	= session->header.data_size,
> +		.data_offset	= session->header.data_offset,
> +		.process	= process_simple,
> +	};
> +	int i, ret = 0;
> +	struct ui_progress prog;
> +	u64 total_size = perf_data__size(session->data);
> +
> +	perf_tool__fill_defaults(tool);
> +
> +	ui_progress__init_size(&prog, total_size, "Processing events...");
> +
> +	/* Read data from the header file.. */
> +	ret = reader__process_events(&rd, session, &prog);
> +	if (ret)
> +		goto out_err;
> +
> +	/* ... and continue with data files. */
> +	for (i = 0; i < data->dir.nr ; i++) {
> +		struct perf_data_file *file = &data->dir.files[i];
> +
> +		if (file->size == 0)
> +			continue;
> +
> +		rd = (struct reader) {
> +			.fd		= file->fd,
> +			.data_size	= file->size,
> +			.data_offset	= 0,
> +			.process	= process_index,
> +		};
> +
> +		ret = reader__process_events(&rd, session, &prog);
> +		if (ret)
> +			goto out_err;

Don't we have to have some handling of PERF_RECORD_FINISHED_ROUND here?
I.e. what happens if we fill th ordered events with just the contents
of, say, the first CPU and then have those events flushed and processed
before we start even looking at the events in the other CPUs?

I think some detailed explanation of what happens here is in need, no?

- Arnaldo

> +	}
> +
> +	ret = ordered_events__flush(&session->ordered_events, OE_FLUSH__FINAL);
> +
> +out_err:
> +	if (!tool->no_warn)
> +		perf_session__warn_about_errors(session);
> +
> +	/*
> +	 * We may switching perf.data output, make ordered_events
> +	 * reusable.
> +	 */
> +	ordered_events__reinit(&session->ordered_events);
> +	return ret;
> +}
> +
>  int perf_session__process_events(struct perf_session *session)
>  {
>  	if (perf_session__register_idle_thread(session) < 0)
> @@ -2014,6 +2095,9 @@ int perf_session__process_events(struct perf_session *session)
>  	if (perf_data__is_pipe(session->data))
>  		return __perf_session__process_pipe_events(session);
>  
> +	if (perf_data__is_dir(session->data))
> +		return __perf_session__process_dir_events(session);
> +
>  	return __perf_session__process_events(session);
>  }
>  
> -- 
> 2.17.2

-- 

- Arnaldo

^ permalink raw reply	[flat|nested] 27+ messages in thread

* Re: [PATCH 07/11] perf session: Add __perf_session__process_dir_events function
  2019-03-08 18:38   ` Arnaldo Carvalho de Melo
@ 2019-03-08 18:39     ` Arnaldo Carvalho de Melo
  2019-03-09 17:21     ` Jiri Olsa
  1 sibling, 0 replies; 27+ messages in thread
From: Arnaldo Carvalho de Melo @ 2019-03-08 18:39 UTC (permalink / raw)
  To: Jiri Olsa
  Cc: lkml, Ingo Molnar, Namhyung Kim, Alexander Shishkin,
	Peter Zijlstra, Adrian Hunter, Andi Kleen, Stephane Eranian,
	Alexey Budankov

Em Fri, Mar 08, 2019 at 03:38:05PM -0300, Arnaldo Carvalho de Melo escreveu:
> Em Fri, Mar 08, 2019 at 02:47:41PM +0100, Jiri Olsa escreveu:
> > +static int __perf_session__process_dir_events(struct perf_session *session)
> > +{
> > +	struct perf_data *data = session->data;
> > +	struct perf_tool *tool = session->tool;
> > +	struct reader rd = {
> > +		.fd		= perf_data__fd(session->data),
> > +		.data_size	= session->header.data_size,
> > +		.data_offset	= session->header.data_offset,
> > +		.process	= process_simple,
> > +	};
> > +	int i, ret = 0;
> > +	struct ui_progress prog;
> > +	u64 total_size = perf_data__size(session->data);
> > +
> > +	perf_tool__fill_defaults(tool);
> > +
> > +	ui_progress__init_size(&prog, total_size, "Processing events...");
> > +
> > +	/* Read data from the header file.. */
> > +	ret = reader__process_events(&rd, session, &prog);
> > +	if (ret)
> > +		goto out_err;
> > +
> > +	/* ... and continue with data files. */
> > +	for (i = 0; i < data->dir.nr ; i++) {
> > +		struct perf_data_file *file = &data->dir.files[i];
> > +
> > +		if (file->size == 0)
> > +			continue;
> > +
> > +		rd = (struct reader) {
> > +			.fd		= file->fd,
> > +			.data_size	= file->size,
> > +			.data_offset	= 0,
> > +			.process	= process_index,
> > +		};
> > +
> > +		ret = reader__process_events(&rd, session, &prog);
> > +		if (ret)
> > +			goto out_err;
 
> Don't we have to have some handling of PERF_RECORD_FINISHED_ROUND here?
> I.e. what happens if we fill th ordered events with just the contents
> of, say, the first CPU and then have those events flushed and processed
> before we start even looking at the events in the other CPUs?
 
> I think some detailed explanation of what happens here is in need, no?

Ah, I've pushed what I have merged so far to acme/perf/core on
git.kernel.org, please continue from there.

- Arnaldo

^ permalink raw reply	[flat|nested] 27+ messages in thread

* Re: [PATCH 07/11] perf session: Add __perf_session__process_dir_events function
  2019-03-08 18:38   ` Arnaldo Carvalho de Melo
  2019-03-08 18:39     ` Arnaldo Carvalho de Melo
@ 2019-03-09 17:21     ` Jiri Olsa
  1 sibling, 0 replies; 27+ messages in thread
From: Jiri Olsa @ 2019-03-09 17:21 UTC (permalink / raw)
  To: Arnaldo Carvalho de Melo
  Cc: Jiri Olsa, lkml, Ingo Molnar, Namhyung Kim, Alexander Shishkin,
	Peter Zijlstra, Adrian Hunter, Andi Kleen, Stephane Eranian,
	Alexey Budankov

On Fri, Mar 08, 2019 at 03:38:05PM -0300, Arnaldo Carvalho de Melo wrote:

SNIP

> > +	/* Read data from the header file.. */
> > +	ret = reader__process_events(&rd, session, &prog);
> > +	if (ret)
> > +		goto out_err;
> > +
> > +	/* ... and continue with data files. */
> > +	for (i = 0; i < data->dir.nr ; i++) {
> > +		struct perf_data_file *file = &data->dir.files[i];
> > +
> > +		if (file->size == 0)
> > +			continue;
> > +
> > +		rd = (struct reader) {
> > +			.fd		= file->fd,
> > +			.data_size	= file->size,
> > +			.data_offset	= 0,
> > +			.process	= process_index,
> > +		};
> > +
> > +		ret = reader__process_events(&rd, session, &prog);
> > +		if (ret)
> > +			goto out_err;
> 
> Don't we have to have some handling of PERF_RECORD_FINISHED_ROUND here?
> I.e. what happens if we fill th ordered events with just the contents
> of, say, the first CPU and then have those events flushed and processed
> before we start even looking at the events in the other CPUs?

PERF_RECORD_FINISHED_ROUND is disabled for directory data,
because each file contains sorted cpu data, so we only
merge them together

> 
> I think some detailed explanation of what happens here is in need, no?

right, I'll put the comment for the function

jirka

^ permalink raw reply	[flat|nested] 27+ messages in thread

* Re: [PATCH 04/11] perf data: Make perf_data__size to work over directory
  2019-03-08 17:33   ` Arnaldo Carvalho de Melo
@ 2019-03-09 17:21     ` Jiri Olsa
  0 siblings, 0 replies; 27+ messages in thread
From: Jiri Olsa @ 2019-03-09 17:21 UTC (permalink / raw)
  To: Arnaldo Carvalho de Melo
  Cc: Jiri Olsa, lkml, Ingo Molnar, Namhyung Kim, Alexander Shishkin,
	Peter Zijlstra, Adrian Hunter, Andi Kleen, Stephane Eranian,
	Alexey Budankov

On Fri, Mar 08, 2019 at 02:33:05PM -0300, Arnaldo Carvalho de Melo wrote:
> Em Fri, Mar 08, 2019 at 02:47:38PM +0100, Jiri Olsa escreveu:
> > Making perf_data__size to return proper size
> > for directory data.
> > 
> > Link: http://lkml.kernel.org/n/tip-t4dm8cctat2ginmy2bb08xe8@git.kernel.org
> > Signed-off-by: Jiri Olsa <jolsa@kernel.org>
> > ---
> >  tools/perf/util/data.c | 17 +++++++++++++++++
> >  tools/perf/util/data.h |  6 +-----
> >  2 files changed, 18 insertions(+), 5 deletions(-)
> > 
> > diff --git a/tools/perf/util/data.c b/tools/perf/util/data.c
> > index d568975c5ee3..498c531e2cb9 100644
> > --- a/tools/perf/util/data.c
> > +++ b/tools/perf/util/data.c
> > @@ -393,3 +393,20 @@ int perf_data__switch(struct perf_data *data,
> >  	free(new_filepath);
> >  	return ret;
> >  }
> > +
> > +unsigned long perf_data__size(struct perf_data *data)
> > +{
> > +	u64 size = data->file.size;
> > +	int i;
> > +
> > +	if (!data->is_dir)
> > +		return size;
> 
> So you do the sum over dir.files[i] _plus_ what is in data->file.size,
> what is in data->file.size? Zero for the dir case? Or some non-per-cpu
> metadata?

yes, saw the updated changelong, thanks

jirka

^ permalink raw reply	[flat|nested] 27+ messages in thread

* Re: [PATCH 05/11] perf header: Add DIR_FORMAT feature to describe directory data
  2019-03-08 18:24   ` Arnaldo Carvalho de Melo
@ 2019-03-09 17:21     ` Jiri Olsa
  0 siblings, 0 replies; 27+ messages in thread
From: Jiri Olsa @ 2019-03-09 17:21 UTC (permalink / raw)
  To: Arnaldo Carvalho de Melo
  Cc: Jiri Olsa, lkml, Ingo Molnar, Namhyung Kim, Alexander Shishkin,
	Peter Zijlstra, Adrian Hunter, Andi Kleen, Stephane Eranian,
	Alexey Budankov

On Fri, Mar 08, 2019 at 03:24:35PM -0300, Arnaldo Carvalho de Melo wrote:
> Em Fri, Mar 08, 2019 at 02:47:39PM +0100, Jiri Olsa escreveu:
> > The data files layout is described by HEADER_DIR_FORMAT
> > feature.  Currently it holds only version number (1):
> > 
> >      uint64_t version;
> > 
> > The current version holds only version value (1) means
> > that data files:
> >   - follow the 'data.*' name format
> >   - contain raw events data in standard perf format as
> >     read from kernel (and need to be sorted)
> 
> and need to be sorted when multiple files are processed, right?

correct

jirka

> 
> - Arnaldo
>  
> > Future versions are expected to describe different data
> > files layout according to special needs.
> > 
> > Link: http://lkml.kernel.org/n/tip-6l53s3oabwr1r4n6wke21d8g@git.kernel.org
> > Signed-off-by: Jiri Olsa <jolsa@kernel.org>
> > ---
> >  tools/perf/builtin-record.c |  2 ++
> >  tools/perf/util/data.c      | 10 +++++++--
> >  tools/perf/util/data.h      |  1 +
> >  tools/perf/util/header.c    | 44 ++++++++++++++++++++++++++++++++++++-
> >  tools/perf/util/header.h    |  5 +++++
> >  5 files changed, 59 insertions(+), 3 deletions(-)
> > 
> > diff --git a/tools/perf/builtin-record.c b/tools/perf/builtin-record.c
> > index e983c8d71a79..a468d882e74f 100644
> > --- a/tools/perf/builtin-record.c
> > +++ b/tools/perf/builtin-record.c
> > @@ -837,6 +837,8 @@ static void record__init_features(struct record *rec)
> >  	if (!(rec->opts.use_clockid && rec->opts.clockid_res_ns))
> >  		perf_header__clear_feat(&session->header, HEADER_CLOCKID);
> >  
> > +	perf_header__clear_feat(&session->header, HEADER_DIR_FORMAT);
> > +
> >  	perf_header__clear_feat(&session->header, HEADER_STAT);
> >  }
> >  
> > diff --git a/tools/perf/util/data.c b/tools/perf/util/data.c
> > index 498c531e2cb9..41a9f478f0f0 100644
> > --- a/tools/perf/util/data.c
> > +++ b/tools/perf/util/data.c
> > @@ -14,6 +14,7 @@
> >  #include "data.h"
> >  #include "util.h"
> >  #include "debug.h"
> > +#include "header.h"
> >  
> >  static void close_dir(struct perf_data_file *files, int nr)
> >  {
> > @@ -41,8 +42,9 @@ int perf_data__create_dir(struct perf_data *data, int nr)
> >  	if (!files)
> >  		return -ENOMEM;
> >  
> > -	data->dir.files = files;
> > -	data->dir.nr    = nr;
> > +	data->dir.version = PERF_DIR_VERSION;
> > +	data->dir.files   = files;
> > +	data->dir.nr      = nr;
> >  
> >  	for (i = 0; i < nr; i++) {
> >  		struct perf_data_file *file = &files[i];
> > @@ -75,6 +77,10 @@ int perf_data__open_dir(struct perf_data *data)
> >  	if (WARN_ON(!data->is_dir))
> >  		return -EINVAL;
> >  
> > +	/* The version is provided by DIR_FORMAT feature. */
> > +	if (WARN_ON(data->dir.version != PERF_DIR_VERSION))
> > +		return -1;
> > +
> >  	dir = opendir(data->path);
> >  	if (!dir)
> >  		return -EINVAL;
> > diff --git a/tools/perf/util/data.h b/tools/perf/util/data.h
> > index d342469bdfda..6aef8746469f 100644
> > --- a/tools/perf/util/data.h
> > +++ b/tools/perf/util/data.h
> > @@ -24,6 +24,7 @@ struct perf_data {
> >  	enum perf_data_mode	 mode;
> >  
> >  	struct {
> > +		u64			 version;
> >  		struct perf_data_file	*files;
> >  		int			 nr;
> >  	} dir;
> > diff --git a/tools/perf/util/header.c b/tools/perf/util/header.c
> > index 01b324c275b9..b0683bf4d9f3 100644
> > --- a/tools/perf/util/header.c
> > +++ b/tools/perf/util/header.c
> > @@ -861,6 +861,21 @@ static int write_clockid(struct feat_fd *ff,
> >  			sizeof(ff->ph->env.clockid_res_ns));
> >  }
> >  
> > +static int write_dir_format(struct feat_fd *ff,
> > +			    struct perf_evlist *evlist __maybe_unused)
> > +{
> > +	struct perf_session *session;
> > +	struct perf_data *data;
> > +
> > +	session = container_of(ff->ph, struct perf_session, header);
> > +	data = session->data;
> > +
> > +	if (WARN_ON(!perf_data__is_dir(data)))
> > +		return -1;
> > +
> > +	return do_write(ff, &data->dir.version, sizeof(data->dir.version));
> > +}
> > +
> >  static int cpu_cache_level__sort(const void *a, const void *b)
> >  {
> >  	struct cpu_cache_level *cache_a = (struct cpu_cache_level *)a;
> > @@ -1341,6 +1356,17 @@ static void print_clockid(struct feat_fd *ff, FILE *fp)
> >  		ff->ph->env.clockid_res_ns * 1000);
> >  }
> >  
> > +static void print_dir_format(struct feat_fd *ff, FILE *fp)
> > +{
> > +	struct perf_session *session;
> > +	struct perf_data *data;
> > +
> > +	session = container_of(ff->ph, struct perf_session, header);
> > +	data = session->data;
> > +
> > +	fprintf(fp, "# directory data version : %"PRIu64"\n", data->dir.version);
> > +}
> > +
> >  static void free_event_desc(struct perf_evsel *events)
> >  {
> >  	struct perf_evsel *evsel;
> > @@ -2373,6 +2399,21 @@ static int process_clockid(struct feat_fd *ff,
> >  	return 0;
> >  }
> >  
> > +static int process_dir_format(struct feat_fd *ff,
> > +			      void *_data __maybe_unused)
> > +{
> > +	struct perf_session *session;
> > +	struct perf_data *data;
> > +
> > +	session = container_of(ff->ph, struct perf_session, header);
> > +	data = session->data;
> > +
> > +	if (WARN_ON(!perf_data__is_dir(data)))
> > +		return -1;
> > +
> > +	return do_read_u64(ff, &data->dir.version);
> > +}
> > +
> >  struct feature_ops {
> >  	int (*write)(struct feat_fd *ff, struct perf_evlist *evlist);
> >  	void (*print)(struct feat_fd *ff, FILE *fp);
> > @@ -2432,7 +2473,8 @@ static const struct feature_ops feat_ops[HEADER_LAST_FEATURE] = {
> >  	FEAT_OPN(CACHE,		cache,		true),
> >  	FEAT_OPR(SAMPLE_TIME,	sample_time,	false),
> >  	FEAT_OPR(MEM_TOPOLOGY,	mem_topology,	true),
> > -	FEAT_OPR(CLOCKID,       clockid,        false)
> > +	FEAT_OPR(CLOCKID,	clockid,	false),
> > +	FEAT_OPN(DIR_FORMAT,	dir_format,	false)
> >  };
> >  
> >  struct header_print_data {
> > diff --git a/tools/perf/util/header.h b/tools/perf/util/header.h
> > index 0d553ddca0a3..6a231340238d 100644
> > --- a/tools/perf/util/header.h
> > +++ b/tools/perf/util/header.h
> > @@ -39,6 +39,7 @@ enum {
> >  	HEADER_SAMPLE_TIME,
> >  	HEADER_MEM_TOPOLOGY,
> >  	HEADER_CLOCKID,
> > +	HEADER_DIR_FORMAT,
> >  	HEADER_LAST_FEATURE,
> >  	HEADER_FEAT_BITS	= 256,
> >  };
> > @@ -48,6 +49,10 @@ enum perf_header_version {
> >  	PERF_HEADER_VERSION_2,
> >  };
> >  
> > +enum perf_dir_version {
> > +	PERF_DIR_VERSION	= 1,
> > +};
> > +
> >  struct perf_file_section {
> >  	u64 offset;
> >  	u64 size;
> > -- 
> > 2.17.2
> 
> -- 
> 
> - Arnaldo

^ permalink raw reply	[flat|nested] 27+ messages in thread

* [tip:perf/urgent] perf data: Support having perf.data stored as a directory
  2019-03-08 13:47 ` [PATCH 01/11] perf data: Add directory support Jiri Olsa
  2019-03-08 17:26   ` Arnaldo Carvalho de Melo
@ 2019-03-22 22:03   ` tip-bot for Jiri Olsa
  1 sibling, 0 replies; 27+ messages in thread
From: tip-bot for Jiri Olsa @ 2019-03-22 22:03 UTC (permalink / raw)
  To: linux-tip-commits
  Cc: mingo, tglx, ak, jolsa, eranian, alexander.shishkin, namhyung,
	adrian.hunter, alexey.budankov, acme, peterz, linux-kernel, hpa

Commit-ID:  ec65def1045e4c7817b7f741a86dadae82877a93
Gitweb:     https://git.kernel.org/tip/ec65def1045e4c7817b7f741a86dadae82877a93
Author:     Jiri Olsa <jolsa@kernel.org>
AuthorDate: Fri, 8 Mar 2019 14:47:35 +0100
Committer:  Arnaldo Carvalho de Melo <acme@redhat.com>
CommitDate: Mon, 11 Mar 2019 11:56:03 -0300

perf data: Support having perf.data stored as a directory

The caller needs to set 'struct perf_data::is_dir flag and the path will
be treated as a directory.

The 'struct perf_data::file' is initialized and open as 'path/header'
file.

Add a check to the direcory interface functions to check the is_dir flag.

Signed-off-by: Jiri Olsa <jolsa@kernel.org>
Cc: Adrian Hunter <adrian.hunter@intel.com>
Cc: Alexander Shishkin <alexander.shishkin@linux.intel.com>
Cc: Alexey Budankov <alexey.budankov@linux.intel.com>
Cc: Andi Kleen <ak@linux.intel.com>
Cc: Namhyung Kim <namhyung@kernel.org>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Stephane Eranian <eranian@google.com>
Link: http://lkml.kernel.org/r/20190308134745.5057-2-jolsa@kernel.org
[ Be consistent on how to signal failure, i.e. use -1 and let users check errno ]
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
---
 tools/perf/util/data.c    | 49 ++++++++++++++++++++++++++++++++++++++++++++++-
 tools/perf/util/data.h    |  6 ++++++
 tools/perf/util/session.c |  4 ++++
 3 files changed, 58 insertions(+), 1 deletion(-)

diff --git a/tools/perf/util/data.c b/tools/perf/util/data.c
index e098e189f93e..18fa8d4614eb 100644
--- a/tools/perf/util/data.c
+++ b/tools/perf/util/data.c
@@ -34,6 +34,9 @@ int perf_data__create_dir(struct perf_data *data, int nr)
 	struct perf_data_file *files = NULL;
 	int i, ret = -1;
 
+	if (WARN_ON(!data->is_dir))
+		return -EINVAL;
+
 	files = zalloc(nr * sizeof(*files));
 	if (!files)
 		return -ENOMEM;
@@ -69,6 +72,9 @@ int perf_data__open_dir(struct perf_data *data)
 	DIR *dir;
 	int nr = 0;
 
+	if (WARN_ON(!data->is_dir))
+		return -EINVAL;
+
 	dir = opendir(data->path);
 	if (!dir)
 		return -EINVAL;
@@ -173,6 +179,16 @@ static int check_backup(struct perf_data *data)
 	return 0;
 }
 
+static bool is_dir(struct perf_data *data)
+{
+	struct stat st;
+
+	if (stat(data->path, &st))
+		return false;
+
+	return (st.st_mode & S_IFMT) == S_IFDIR;
+}
+
 static int open_file_read(struct perf_data *data)
 {
 	struct stat st;
@@ -254,6 +270,30 @@ static int open_file_dup(struct perf_data *data)
 	return open_file(data);
 }
 
+static int open_dir(struct perf_data *data)
+{
+	int ret;
+
+	/*
+	 * So far we open only the header, so we can read the data version and
+	 * layout.
+	 */
+	if (asprintf(&data->file.path, "%s/header", data->path) < 0)
+		return -1;
+
+	if (perf_data__is_write(data) &&
+	    mkdir(data->path, S_IRWXU) < 0)
+		return -1;
+
+	ret = open_file(data);
+
+	/* Cleanup whatever we managed to create so far. */
+	if (ret && perf_data__is_write(data))
+		rm_rf_perf_data(data->path);
+
+	return ret;
+}
+
 int perf_data__open(struct perf_data *data)
 {
 	if (check_pipe(data))
@@ -265,11 +305,18 @@ int perf_data__open(struct perf_data *data)
 	if (check_backup(data))
 		return -1;
 
-	return open_file_dup(data);
+	if (perf_data__is_read(data))
+		data->is_dir = is_dir(data);
+
+	return perf_data__is_dir(data) ?
+	       open_dir(data) : open_file_dup(data);
 }
 
 void perf_data__close(struct perf_data *data)
 {
+	if (perf_data__is_dir(data))
+		perf_data__close_dir(data);
+
 	zfree(&data->file.path);
 	close(data->file.fd);
 }
diff --git a/tools/perf/util/data.h b/tools/perf/util/data.h
index 14b47be2bd69..06aefeda311f 100644
--- a/tools/perf/util/data.h
+++ b/tools/perf/util/data.h
@@ -19,6 +19,7 @@ struct perf_data {
 	const char		*path;
 	struct perf_data_file	 file;
 	bool			 is_pipe;
+	bool			 is_dir;
 	bool			 force;
 	enum perf_data_mode	 mode;
 
@@ -43,6 +44,11 @@ static inline int perf_data__is_pipe(struct perf_data *data)
 	return data->is_pipe;
 }
 
+static inline bool perf_data__is_dir(struct perf_data *data)
+{
+	return data->is_dir;
+}
+
 static inline int perf_data__fd(struct perf_data *data)
 {
 	return data->file.fd;
diff --git a/tools/perf/util/session.c b/tools/perf/util/session.c
index db643f3c2b95..de777bdc0ed3 100644
--- a/tools/perf/util/session.c
+++ b/tools/perf/util/session.c
@@ -152,6 +152,10 @@ struct perf_session *perf_session__new(struct perf_data *data,
 			}
 
 			perf_evlist__init_trace_event_sample_raw(session->evlist);
+
+			/* Open the directory data. */
+			if (data->is_dir && perf_data__open_dir(data))
+				goto out_delete;
 		}
 	} else  {
 		session->machines.host.env = &perf_env;

^ permalink raw reply	[flat|nested] 27+ messages in thread

* [tip:perf/urgent] perf data: Don't store auxtrace index for directory data file
  2019-03-08 13:47 ` [PATCH 02/11] perf data: Don't store auxtrace index for directory data file Jiri Olsa
@ 2019-03-22 22:03   ` tip-bot for Jiri Olsa
  0 siblings, 0 replies; 27+ messages in thread
From: tip-bot for Jiri Olsa @ 2019-03-22 22:03 UTC (permalink / raw)
  To: linux-tip-commits
  Cc: alexander.shishkin, tglx, hpa, mingo, jolsa, eranian,
	alexey.budankov, adrian.hunter, linux-kernel, peterz, acme, ak,
	namhyung

Commit-ID:  cd3dd8dd8ff62374d90cb3f2e54b8c94106c7810
Gitweb:     https://git.kernel.org/tip/cd3dd8dd8ff62374d90cb3f2e54b8c94106c7810
Author:     Jiri Olsa <jolsa@kernel.org>
AuthorDate: Fri, 8 Mar 2019 14:47:36 +0100
Committer:  Arnaldo Carvalho de Melo <acme@redhat.com>
CommitDate: Mon, 11 Mar 2019 11:56:03 -0300

perf data: Don't store auxtrace index for directory data file

We can't store the auxtrace index when we store into multiple files,
because we keep only offset for it, not the file.

The auxtrace data will be processed correctly in the 'pipe' mode.

Signed-off-by: Jiri Olsa <jolsa@kernel.org>
Cc: Adrian Hunter <adrian.hunter@intel.com>
Cc: Alexander Shishkin <alexander.shishkin@linux.intel.com>
Cc: Alexey Budankov <alexey.budankov@linux.intel.com>
Cc: Andi Kleen <ak@linux.intel.com>
Cc: Namhyung Kim <namhyung@kernel.org>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Stephane Eranian <eranian@google.com>
Link: http://lkml.kernel.org/r/20190308134745.5057-3-jolsa@kernel.org
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
---
 tools/perf/builtin-record.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/tools/perf/builtin-record.c b/tools/perf/builtin-record.c
index f3f7f3100336..e983c8d71a79 100644
--- a/tools/perf/builtin-record.c
+++ b/tools/perf/builtin-record.c
@@ -392,7 +392,7 @@ static int record__process_auxtrace(struct perf_tool *tool,
 	size_t padding;
 	u8 pad[8] = {0};
 
-	if (!perf_data__is_pipe(data)) {
+	if (!perf_data__is_pipe(data) && !perf_data__is_dir(data)) {
 		off_t file_offset;
 		int fd = perf_data__fd(data);
 		int err;

^ permalink raw reply	[flat|nested] 27+ messages in thread

* [tip:perf/urgent] perf data: Add perf_data__update_dir() function
  2019-03-08 13:47 ` [PATCH 03/11] perf data: Add perf_data__update_dir function Jiri Olsa
@ 2019-03-22 22:04   ` tip-bot for Jiri Olsa
  0 siblings, 0 replies; 27+ messages in thread
From: tip-bot for Jiri Olsa @ 2019-03-22 22:04 UTC (permalink / raw)
  To: linux-tip-commits
  Cc: peterz, adrian.hunter, linux-kernel, namhyung, mingo,
	alexey.budankov, alexander.shishkin, jolsa, acme, hpa, eranian,
	tglx, ak

Commit-ID:  e8be135751f26aa5de63e517d375ecf69e9b20c3
Gitweb:     https://git.kernel.org/tip/e8be135751f26aa5de63e517d375ecf69e9b20c3
Author:     Jiri Olsa <jolsa@kernel.org>
AuthorDate: Fri, 8 Mar 2019 14:47:37 +0100
Committer:  Arnaldo Carvalho de Melo <acme@redhat.com>
CommitDate: Mon, 11 Mar 2019 11:56:03 -0300

perf data: Add perf_data__update_dir() function

Add perf_data__update_dir() to update the size for every file within the
perf.data directory.

Signed-off-by: Jiri Olsa <jolsa@kernel.org>
Cc: Adrian Hunter <adrian.hunter@intel.com>
Cc: Alexander Shishkin <alexander.shishkin@linux.intel.com>
Cc: Alexey Budankov <alexey.budankov@linux.intel.com>
Cc: Andi Kleen <ak@linux.intel.com>
Cc: Namhyung Kim <namhyung@kernel.org>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Stephane Eranian <eranian@google.com>
Link: http://lkml.kernel.org/r/20190308134745.5057-4-jolsa@kernel.org
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
---
 tools/perf/util/data.c | 20 ++++++++++++++++++++
 tools/perf/util/data.h |  1 +
 2 files changed, 21 insertions(+)

diff --git a/tools/perf/util/data.c b/tools/perf/util/data.c
index 18fa8d4614eb..5d0f26aef77f 100644
--- a/tools/perf/util/data.c
+++ b/tools/perf/util/data.c
@@ -124,6 +124,26 @@ out_err:
 	return ret;
 }
 
+int perf_data__update_dir(struct perf_data *data)
+{
+	int i;
+
+	if (WARN_ON(!data->is_dir))
+		return -EINVAL;
+
+	for (i = 0; i < data->dir.nr; i++) {
+		struct perf_data_file *file = &data->dir.files[i];
+		struct stat st;
+
+		if (fstat(file->fd, &st))
+			return -1;
+
+		file->size = st.st_size;
+	}
+
+	return 0;
+}
+
 static bool check_pipe(struct perf_data *data)
 {
 	struct stat st;
diff --git a/tools/perf/util/data.h b/tools/perf/util/data.h
index 06aefeda311f..0deeb1af9f54 100644
--- a/tools/perf/util/data.h
+++ b/tools/perf/util/data.h
@@ -79,4 +79,5 @@ int perf_data__switch(struct perf_data *data,
 int perf_data__create_dir(struct perf_data *data, int nr);
 int perf_data__open_dir(struct perf_data *data);
 void perf_data__close_dir(struct perf_data *data);
+int perf_data__update_dir(struct perf_data *data);
 #endif /* __PERF_DATA_H */

^ permalink raw reply	[flat|nested] 27+ messages in thread

* [tip:perf/urgent] perf data: Make perf_data__size() work over directory
  2019-03-08 13:47 ` [PATCH 04/11] perf data: Make perf_data__size to work over directory Jiri Olsa
  2019-03-08 17:33   ` Arnaldo Carvalho de Melo
@ 2019-03-22 22:05   ` tip-bot for Jiri Olsa
  1 sibling, 0 replies; 27+ messages in thread
From: tip-bot for Jiri Olsa @ 2019-03-22 22:05 UTC (permalink / raw)
  To: linux-tip-commits
  Cc: eranian, alexander.shishkin, acme, jolsa, namhyung, ak,
	adrian.hunter, mingo, hpa, alexey.budankov, linux-kernel, tglx,
	peterz

Commit-ID:  29583c17b5ce0bc17fdd80da939f8199a03d9668
Gitweb:     https://git.kernel.org/tip/29583c17b5ce0bc17fdd80da939f8199a03d9668
Author:     Jiri Olsa <jolsa@kernel.org>
AuthorDate: Fri, 8 Mar 2019 14:47:38 +0100
Committer:  Arnaldo Carvalho de Melo <acme@redhat.com>
CommitDate: Mon, 11 Mar 2019 11:56:03 -0300

perf data: Make perf_data__size() work over directory

Make perf_data__size() return proper size for directory data, summing up
all the individual file sizes.

Signed-off-by: Jiri Olsa <jolsa@kernel.org>
Cc: Adrian Hunter <adrian.hunter@intel.com>
Cc: Alexander Shishkin <alexander.shishkin@linux.intel.com>
Cc: Alexey Budankov <alexey.budankov@linux.intel.com>
Cc: Andi Kleen <ak@linux.intel.com>
Cc: Namhyung Kim <namhyung@kernel.org>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Stephane Eranian <eranian@google.com>
Link: http://lkml.kernel.org/r/20190308134745.5057-5-jolsa@kernel.org
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
---
 tools/perf/util/data.c | 17 +++++++++++++++++
 tools/perf/util/data.h |  6 +-----
 2 files changed, 18 insertions(+), 5 deletions(-)

diff --git a/tools/perf/util/data.c b/tools/perf/util/data.c
index 5d0f26aef77f..b71c441cafbb 100644
--- a/tools/perf/util/data.c
+++ b/tools/perf/util/data.c
@@ -393,3 +393,20 @@ out:
 	free(new_filepath);
 	return ret;
 }
+
+unsigned long perf_data__size(struct perf_data *data)
+{
+	u64 size = data->file.size;
+	int i;
+
+	if (!data->is_dir)
+		return size;
+
+	for (i = 0; i < data->dir.nr; i++) {
+		struct perf_data_file *file = &data->dir.files[i];
+
+		size += file->size;
+	}
+
+	return size;
+}
diff --git a/tools/perf/util/data.h b/tools/perf/util/data.h
index 0deeb1af9f54..d342469bdfda 100644
--- a/tools/perf/util/data.h
+++ b/tools/perf/util/data.h
@@ -54,11 +54,6 @@ static inline int perf_data__fd(struct perf_data *data)
 	return data->file.fd;
 }
 
-static inline unsigned long perf_data__size(struct perf_data *data)
-{
-	return data->file.size;
-}
-
 int perf_data__open(struct perf_data *data);
 void perf_data__close(struct perf_data *data);
 ssize_t perf_data__write(struct perf_data *data,
@@ -80,4 +75,5 @@ int perf_data__create_dir(struct perf_data *data, int nr);
 int perf_data__open_dir(struct perf_data *data);
 void perf_data__close_dir(struct perf_data *data);
 int perf_data__update_dir(struct perf_data *data);
+unsigned long perf_data__size(struct perf_data *data);
 #endif /* __PERF_DATA_H */

^ permalink raw reply	[flat|nested] 27+ messages in thread

* [tip:perf/urgent] perf header: Add DIR_FORMAT feature to describe directory data
  2019-03-08 13:47 ` [PATCH 05/11] perf header: Add DIR_FORMAT feature to describe directory data Jiri Olsa
  2019-03-08 18:24   ` Arnaldo Carvalho de Melo
@ 2019-03-22 22:05   ` tip-bot for Jiri Olsa
  1 sibling, 0 replies; 27+ messages in thread
From: tip-bot for Jiri Olsa @ 2019-03-22 22:05 UTC (permalink / raw)
  To: linux-tip-commits
  Cc: acme, ak, adrian.hunter, jolsa, mingo, namhyung, alexey.budankov,
	peterz, hpa, eranian, linux-kernel, tglx, alexander.shishkin

Commit-ID:  258031c017c353e899902342a25579fc81a34cc1
Gitweb:     https://git.kernel.org/tip/258031c017c353e899902342a25579fc81a34cc1
Author:     Jiri Olsa <jolsa@kernel.org>
AuthorDate: Fri, 8 Mar 2019 14:47:39 +0100
Committer:  Arnaldo Carvalho de Melo <acme@redhat.com>
CommitDate: Mon, 11 Mar 2019 11:56:03 -0300

perf header: Add DIR_FORMAT feature to describe directory data

The data files layout is described by HEADER_DIR_FORMAT feature.
Currently it holds only version number (1):

     uint64_t version;

The current version holds only version value (1) means that data files:

  - Follow the 'data.*' name format.

  - Contain raw events data in standard perf format as read from kernel
    (and need to be sorted)

Future versions are expected to describe different data files layout
according to special needs.

Signed-off-by: Jiri Olsa <jolsa@kernel.org>
Cc: Adrian Hunter <adrian.hunter@intel.com>
Cc: Alexander Shishkin <alexander.shishkin@linux.intel.com>
Cc: Alexey Budankov <alexey.budankov@linux.intel.com>
Cc: Andi Kleen <ak@linux.intel.com>
Cc: Namhyung Kim <namhyung@kernel.org>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Stephane Eranian <eranian@google.com>
Link: http://lkml.kernel.org/r/20190308134745.5057-6-jolsa@kernel.org
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
---
 tools/perf/builtin-record.c |  2 ++
 tools/perf/util/data.c      | 10 ++++++++--
 tools/perf/util/data.h      |  1 +
 tools/perf/util/header.c    | 44 +++++++++++++++++++++++++++++++++++++++++++-
 tools/perf/util/header.h    |  5 +++++
 5 files changed, 59 insertions(+), 3 deletions(-)

diff --git a/tools/perf/builtin-record.c b/tools/perf/builtin-record.c
index e983c8d71a79..a468d882e74f 100644
--- a/tools/perf/builtin-record.c
+++ b/tools/perf/builtin-record.c
@@ -837,6 +837,8 @@ static void record__init_features(struct record *rec)
 	if (!(rec->opts.use_clockid && rec->opts.clockid_res_ns))
 		perf_header__clear_feat(&session->header, HEADER_CLOCKID);
 
+	perf_header__clear_feat(&session->header, HEADER_DIR_FORMAT);
+
 	perf_header__clear_feat(&session->header, HEADER_STAT);
 }
 
diff --git a/tools/perf/util/data.c b/tools/perf/util/data.c
index b71c441cafbb..c6b67efea11a 100644
--- a/tools/perf/util/data.c
+++ b/tools/perf/util/data.c
@@ -14,6 +14,7 @@
 #include "data.h"
 #include "util.h"
 #include "debug.h"
+#include "header.h"
 
 static void close_dir(struct perf_data_file *files, int nr)
 {
@@ -41,8 +42,9 @@ int perf_data__create_dir(struct perf_data *data, int nr)
 	if (!files)
 		return -ENOMEM;
 
-	data->dir.files = files;
-	data->dir.nr    = nr;
+	data->dir.version = PERF_DIR_VERSION;
+	data->dir.files   = files;
+	data->dir.nr      = nr;
 
 	for (i = 0; i < nr; i++) {
 		struct perf_data_file *file = &files[i];
@@ -75,6 +77,10 @@ int perf_data__open_dir(struct perf_data *data)
 	if (WARN_ON(!data->is_dir))
 		return -EINVAL;
 
+	/* The version is provided by DIR_FORMAT feature. */
+	if (WARN_ON(data->dir.version != PERF_DIR_VERSION))
+		return -1;
+
 	dir = opendir(data->path);
 	if (!dir)
 		return -EINVAL;
diff --git a/tools/perf/util/data.h b/tools/perf/util/data.h
index d342469bdfda..6aef8746469f 100644
--- a/tools/perf/util/data.h
+++ b/tools/perf/util/data.h
@@ -24,6 +24,7 @@ struct perf_data {
 	enum perf_data_mode	 mode;
 
 	struct {
+		u64			 version;
 		struct perf_data_file	*files;
 		int			 nr;
 	} dir;
diff --git a/tools/perf/util/header.c b/tools/perf/util/header.c
index 01b324c275b9..b0683bf4d9f3 100644
--- a/tools/perf/util/header.c
+++ b/tools/perf/util/header.c
@@ -861,6 +861,21 @@ static int write_clockid(struct feat_fd *ff,
 			sizeof(ff->ph->env.clockid_res_ns));
 }
 
+static int write_dir_format(struct feat_fd *ff,
+			    struct perf_evlist *evlist __maybe_unused)
+{
+	struct perf_session *session;
+	struct perf_data *data;
+
+	session = container_of(ff->ph, struct perf_session, header);
+	data = session->data;
+
+	if (WARN_ON(!perf_data__is_dir(data)))
+		return -1;
+
+	return do_write(ff, &data->dir.version, sizeof(data->dir.version));
+}
+
 static int cpu_cache_level__sort(const void *a, const void *b)
 {
 	struct cpu_cache_level *cache_a = (struct cpu_cache_level *)a;
@@ -1341,6 +1356,17 @@ static void print_clockid(struct feat_fd *ff, FILE *fp)
 		ff->ph->env.clockid_res_ns * 1000);
 }
 
+static void print_dir_format(struct feat_fd *ff, FILE *fp)
+{
+	struct perf_session *session;
+	struct perf_data *data;
+
+	session = container_of(ff->ph, struct perf_session, header);
+	data = session->data;
+
+	fprintf(fp, "# directory data version : %"PRIu64"\n", data->dir.version);
+}
+
 static void free_event_desc(struct perf_evsel *events)
 {
 	struct perf_evsel *evsel;
@@ -2373,6 +2399,21 @@ static int process_clockid(struct feat_fd *ff,
 	return 0;
 }
 
+static int process_dir_format(struct feat_fd *ff,
+			      void *_data __maybe_unused)
+{
+	struct perf_session *session;
+	struct perf_data *data;
+
+	session = container_of(ff->ph, struct perf_session, header);
+	data = session->data;
+
+	if (WARN_ON(!perf_data__is_dir(data)))
+		return -1;
+
+	return do_read_u64(ff, &data->dir.version);
+}
+
 struct feature_ops {
 	int (*write)(struct feat_fd *ff, struct perf_evlist *evlist);
 	void (*print)(struct feat_fd *ff, FILE *fp);
@@ -2432,7 +2473,8 @@ static const struct feature_ops feat_ops[HEADER_LAST_FEATURE] = {
 	FEAT_OPN(CACHE,		cache,		true),
 	FEAT_OPR(SAMPLE_TIME,	sample_time,	false),
 	FEAT_OPR(MEM_TOPOLOGY,	mem_topology,	true),
-	FEAT_OPR(CLOCKID,       clockid,        false)
+	FEAT_OPR(CLOCKID,	clockid,	false),
+	FEAT_OPN(DIR_FORMAT,	dir_format,	false)
 };
 
 struct header_print_data {
diff --git a/tools/perf/util/header.h b/tools/perf/util/header.h
index 0d553ddca0a3..6a231340238d 100644
--- a/tools/perf/util/header.h
+++ b/tools/perf/util/header.h
@@ -39,6 +39,7 @@ enum {
 	HEADER_SAMPLE_TIME,
 	HEADER_MEM_TOPOLOGY,
 	HEADER_CLOCKID,
+	HEADER_DIR_FORMAT,
 	HEADER_LAST_FEATURE,
 	HEADER_FEAT_BITS	= 256,
 };
@@ -48,6 +49,10 @@ enum perf_header_version {
 	PERF_HEADER_VERSION_2,
 };
 
+enum perf_dir_version {
+	PERF_DIR_VERSION	= 1,
+};
+
 struct perf_file_section {
 	u64 offset;
 	u64 size;

^ permalink raw reply	[flat|nested] 27+ messages in thread

* [tip:perf/urgent] perf session: Add process callback to reader object
  2019-03-08 13:47 ` [PATCH 06/11] perf session: Add process callback to reader object Jiri Olsa
@ 2019-03-22 22:06   ` tip-bot for Jiri Olsa
  0 siblings, 0 replies; 27+ messages in thread
From: tip-bot for Jiri Olsa @ 2019-03-22 22:06 UTC (permalink / raw)
  To: linux-tip-commits
  Cc: peterz, tglx, hpa, alexey.budankov, adrian.hunter, jolsa, mingo,
	alexander.shishkin, linux-kernel, eranian, namhyung, ak, acme

Commit-ID:  e51f806198306a8ad7ae6e34d1af0716ef73da80
Gitweb:     https://git.kernel.org/tip/e51f806198306a8ad7ae6e34d1af0716ef73da80
Author:     Jiri Olsa <jolsa@kernel.org>
AuthorDate: Fri, 8 Mar 2019 14:47:40 +0100
Committer:  Arnaldo Carvalho de Melo <acme@redhat.com>
CommitDate: Mon, 11 Mar 2019 11:56:03 -0300

perf session: Add process callback to reader object

Adding callback function to reader object so callers can process data in
different ways.

Signed-off-by: Jiri Olsa <jolsa@kernel.org>
Cc: Adrian Hunter <adrian.hunter@intel.com>
Cc: Alexander Shishkin <alexander.shishkin@linux.intel.com>
Cc: Alexey Budankov <alexey.budankov@linux.intel.com>
Cc: Andi Kleen <ak@linux.intel.com>
Cc: Namhyung Kim <namhyung@kernel.org>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Stephane Eranian <eranian@google.com>
Link: http://lkml.kernel.org/r/20190308134745.5057-7-jolsa@kernel.org
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
---
 tools/perf/util/session.c | 23 +++++++++++++++++++----
 1 file changed, 19 insertions(+), 4 deletions(-)

diff --git a/tools/perf/util/session.c b/tools/perf/util/session.c
index de777bdc0ed3..0ec34227bd60 100644
--- a/tools/perf/util/session.c
+++ b/tools/perf/util/session.c
@@ -1847,10 +1847,17 @@ fetch_mmaped_event(struct perf_session *session,
 #define NUM_MMAPS 128
 #endif
 
+struct reader;
+
+typedef s64 (*reader_cb_t)(struct perf_session *session,
+			   union perf_event *event,
+			   u64 file_offset);
+
 struct reader {
-	int	fd;
-	u64	data_size;
-	u64	data_offset;
+	int		 fd;
+	u64		 data_size;
+	u64		 data_offset;
+	reader_cb_t	 process;
 };
 
 static int
@@ -1921,7 +1928,7 @@ more:
 	size = event->header.size;
 
 	if (size < sizeof(struct perf_event_header) ||
-	    (skip = perf_session__process_event(session, event, file_pos)) < 0) {
+	    (skip = rd->process(session, event, file_pos)) < 0) {
 		pr_err("%#" PRIx64 " [%#x]: failed to process type: %d\n",
 		       file_offset + head, event->header.size,
 		       event->header.type);
@@ -1947,12 +1954,20 @@ out:
 	return err;
 }
 
+static s64 process_simple(struct perf_session *session,
+			  union perf_event *event,
+			  u64 file_offset)
+{
+	return perf_session__process_event(session, event, file_offset);
+}
+
 static int __perf_session__process_events(struct perf_session *session)
 {
 	struct reader rd = {
 		.fd		= perf_data__fd(session->data),
 		.data_size	= session->header.data_size,
 		.data_offset	= session->header.data_offset,
+		.process	= process_simple,
 	};
 	struct ordered_events *oe = &session->ordered_events;
 	struct perf_tool *tool = session->tool;

^ permalink raw reply	[flat|nested] 27+ messages in thread

* [PATCH 02/11] perf data: Don't store auxtrace index for directory data file
  2019-02-26  8:48 [PATCHv4 00/11] perf record: Add support to store data in directory Jiri Olsa
@ 2019-02-26  8:48 ` Jiri Olsa
  0 siblings, 0 replies; 27+ messages in thread
From: Jiri Olsa @ 2019-02-26  8:48 UTC (permalink / raw)
  To: Arnaldo Carvalho de Melo
  Cc: lkml, Ingo Molnar, Namhyung Kim, Alexander Shishkin,
	Peter Zijlstra, Adrian Hunter, Andi Kleen, Stephane Eranian,
	Alexey Budankov

We can't store auxtrace index when we store to multiple files,
because we keep only offset for it, not the file.

The auxtrace data will be processed correctly in the 'pipe' mode.

Link: http://lkml.kernel.org/n/tip-og11od5s6nfxuf0ftxyu8m6k@git.kernel.org
Signed-off-by: Jiri Olsa <jolsa@kernel.org>
---
 tools/perf/builtin-record.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/tools/perf/builtin-record.c b/tools/perf/builtin-record.c
index f3f7f3100336..e983c8d71a79 100644
--- a/tools/perf/builtin-record.c
+++ b/tools/perf/builtin-record.c
@@ -392,7 +392,7 @@ static int record__process_auxtrace(struct perf_tool *tool,
 	size_t padding;
 	u8 pad[8] = {0};
 
-	if (!perf_data__is_pipe(data)) {
+	if (!perf_data__is_pipe(data) && !perf_data__is_dir(data)) {
 		off_t file_offset;
 		int fd = perf_data__fd(data);
 		int err;
-- 
2.17.2


^ permalink raw reply	[flat|nested] 27+ messages in thread

end of thread, other threads:[~2019-03-22 22:06 UTC | newest]

Thread overview: 27+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2019-03-08 13:47 [PATCHv5 00/11] perf record: Add support to store data in directory Jiri Olsa
2019-03-08 13:47 ` [PATCH 01/11] perf data: Add directory support Jiri Olsa
2019-03-08 17:26   ` Arnaldo Carvalho de Melo
2019-03-22 22:03   ` [tip:perf/urgent] perf data: Support having perf.data stored as a directory tip-bot for Jiri Olsa
2019-03-08 13:47 ` [PATCH 02/11] perf data: Don't store auxtrace index for directory data file Jiri Olsa
2019-03-22 22:03   ` [tip:perf/urgent] " tip-bot for Jiri Olsa
2019-03-08 13:47 ` [PATCH 03/11] perf data: Add perf_data__update_dir function Jiri Olsa
2019-03-22 22:04   ` [tip:perf/urgent] perf data: Add perf_data__update_dir() function tip-bot for Jiri Olsa
2019-03-08 13:47 ` [PATCH 04/11] perf data: Make perf_data__size to work over directory Jiri Olsa
2019-03-08 17:33   ` Arnaldo Carvalho de Melo
2019-03-09 17:21     ` Jiri Olsa
2019-03-22 22:05   ` [tip:perf/urgent] perf data: Make perf_data__size() " tip-bot for Jiri Olsa
2019-03-08 13:47 ` [PATCH 05/11] perf header: Add DIR_FORMAT feature to describe directory data Jiri Olsa
2019-03-08 18:24   ` Arnaldo Carvalho de Melo
2019-03-09 17:21     ` Jiri Olsa
2019-03-22 22:05   ` [tip:perf/urgent] " tip-bot for Jiri Olsa
2019-03-08 13:47 ` [PATCH 06/11] perf session: Add process callback to reader object Jiri Olsa
2019-03-22 22:06   ` [tip:perf/urgent] " tip-bot for Jiri Olsa
2019-03-08 13:47 ` [PATCH 07/11] perf session: Add __perf_session__process_dir_events function Jiri Olsa
2019-03-08 18:38   ` Arnaldo Carvalho de Melo
2019-03-08 18:39     ` Arnaldo Carvalho de Melo
2019-03-09 17:21     ` Jiri Olsa
2019-03-08 13:47 ` [PATCH 08/11] perf session: Add path to reader object Jiri Olsa
2019-03-08 13:47 ` [PATCH 09/11] perf record: Add --dir option to store data in directory Jiri Olsa
2019-03-08 13:47 ` [PATCH 10/11] perf record: Add --output-dir " Jiri Olsa
2019-03-08 13:47 ` [PATCH 11/11] perf record: Describe perf.data directory format Jiri Olsa
  -- strict thread matches above, loose matches on Subject: below --
2019-02-26  8:48 [PATCHv4 00/11] perf record: Add support to store data in directory Jiri Olsa
2019-02-26  8:48 ` [PATCH 02/11] perf data: Don't store auxtrace index for directory data file Jiri Olsa

This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.