* [PATCHv6 0/5] perf record: Add support to store data in directory
@ 2019-03-10 15:30 Jiri Olsa
2019-03-10 15:30 ` [PATCH 1/5] perf session: Add __perf_session__process_dir_events function Jiri Olsa
` (5 more replies)
0 siblings, 6 replies; 8+ messages in thread
From: Jiri Olsa @ 2019-03-10 15:30 UTC (permalink / raw)
To: Arnaldo Carvalho de Melo
Cc: lkml, Ingo Molnar, Namhyung Kim, Alexander Shishkin,
Peter Zijlstra, Adrian Hunter, Andi Kleen, Stephane Eranian,
Alexey Budankov
hi,
this patchset adds the --dir option to record command (and all
the other record command that overload cmd_record) that allows
the data to be stored in directory with multiple data files.
It's next step for multiple threads implementation in record.
It's now possible to make directory data via --dir option, like:
$ perf record --dir perf bench sched messaging
$ ls -l perf.data
total 344
-rw-------. 1 jolsa jolsa 43864 Jan 20 22:26 data.0
-rw-------. 1 jolsa jolsa 30464 Jan 20 22:26 data.1
-rw-------. 1 jolsa jolsa 53816 Jan 20 22:26 data.2
-rw-------. 1 jolsa jolsa 30368 Jan 20 22:26 data.3
-rw-------. 1 jolsa jolsa 40088 Jan 20 22:26 data.4
-rw-------. 1 jolsa jolsa 42592 Jan 20 22:26 data.5
-rw-------. 1 jolsa jolsa 56136 Jan 20 22:26 data.6
-rw-------. 1 jolsa jolsa 25992 Jan 20 22:26 data.7
-rw-------. 1 jolsa jolsa 8832 Jan 20 22:26 header
There's a data file created for every cpu and it's storing
data for those cpu maps. The report command will read it
transparently, sort it and display as single file data.
There's new DIR_FORMAT feature to describe directory data
layout/format. In future we can describe different data files
layout according to special needs.
It's possible to transform directory data into standard
perf.data file via simple inject command:
$ perf inject -o perf.data.file -i perf.data
The old perf fails over the directory data with following message:
$ perf report
incompatible file format (rerun with -v to learn more)
I'm now testing the record threads support, so I'd like to
have some agreement on the directory data support before.
v6 changes:
- rebased to latest Arnaldo's perf/core
- some of the patches already taken
- adding comments to __perf_session__process_dir_events
v5 changes:
- rebased to latest Arnaldo's perf/core
v4 changes:
- some of the patches already taken
- removing up perf.data directory on perf_data__open
error path
v3 changes:
- add rm_rf_perf_data to safely remove file/directory perf data
- allocation fix in perf_data__create_dir
v2 changes:
- rm_rf changes are already accepted with requested changes
- updated doc/man plus adding perf.data-directory-format.txt
to describe directory format/layout
- the --switch-output options now works over directory data
- data rollback is not part of this patchset, updated my TODO though ;-)
- added --output-dir option to combine -o and --dir
- added DIR_FORMAT feature to describe directory data
- disabling directory output for aio for now
It's also available in here:
git://git.kernel.org/pub/scm/linux/kernel/git/jolsa/perf.git
perf/dir
thanks,
jirka
---
Jiri Olsa (5):
perf session: Add __perf_session__process_dir_events function
perf session: Add path to reader object
perf record: Add --dir option to store data in directory
perf record: Add --output-dir option to store data in directory
perf record: Describe perf.data directory format
tools/lib/subcmd/parse-options.h | 4 ++++
tools/perf/Documentation/perf-record.txt | 6 ++++++
tools/perf/Documentation/perf.data-directory-format.txt | 54 ++++++++++++++++++++++++++++++++++++++++++++++++++++++
tools/perf/builtin-record.c | 93 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++-------
tools/perf/util/mmap.h | 23 ++++++++++++-----------
tools/perf/util/session.c | 114 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++----
6 files changed, 272 insertions(+), 22 deletions(-)
create mode 100644 tools/perf/Documentation/perf.data-directory-format.txt
^ permalink raw reply [flat|nested] 8+ messages in thread
* [PATCH 1/5] perf session: Add __perf_session__process_dir_events function
2019-03-10 15:30 [PATCHv6 0/5] perf record: Add support to store data in directory Jiri Olsa
@ 2019-03-10 15:30 ` Jiri Olsa
2019-03-10 15:30 ` [PATCH 2/5] perf session: Add path to reader object Jiri Olsa
` (4 subsequent siblings)
5 siblings, 0 replies; 8+ messages in thread
From: Jiri Olsa @ 2019-03-10 15:30 UTC (permalink / raw)
To: Arnaldo Carvalho de Melo
Cc: lkml, Ingo Molnar, Namhyung Kim, Alexander Shishkin,
Peter Zijlstra, Adrian Hunter, Andi Kleen, Stephane Eranian,
Alexey Budankov
Adding __perf_session__process_dir_events function
to process events over the directory data.
All directory events are pushed into sessions ordered
data and flushed for processing.
Link: http://lkml.kernel.org/n/tip-n3zl0wo3z18tatv5x7epmjrh@git.kernel.org
Signed-off-by: Jiri Olsa <jolsa@kernel.org>
---
tools/perf/util/session.c | 104 +++++++++++++++++++++++++++++++++++++-
1 file changed, 102 insertions(+), 2 deletions(-)
diff --git a/tools/perf/util/session.c b/tools/perf/util/session.c
index 0ec34227bd60..b0c29a19f189 100644
--- a/tools/perf/util/session.c
+++ b/tools/perf/util/session.c
@@ -1876,8 +1876,6 @@ reader__process_events(struct reader *rd, struct perf_session *session,
file_offset = page_offset;
head = rd->data_offset - page_offset;
- ui_progress__init_size(prog, data_size, "Processing events...");
-
data_size += rd->data_offset;
mmap_size = MMAP_SIZE;
@@ -2006,6 +2004,105 @@ static int __perf_session__process_events(struct perf_session *session)
return err;
}
+static s64 process_index(struct perf_session *session,
+ union perf_event *event,
+ u64 file_offset)
+{
+ struct perf_evlist *evlist = session->evlist;
+ u64 timestamp;
+ s64 ret;
+
+ if (session->header.needs_swap)
+ event_swap(event, perf_evlist__sample_id_all(evlist));
+
+ if (event->header.type >= PERF_RECORD_HEADER_MAX)
+ return -EINVAL;
+
+ events_stats__inc(&evlist->stats, event->header.type);
+
+ if (event->header.type >= PERF_RECORD_USER_TYPE_START)
+ return perf_session__process_user_event(session, event, file_offset);
+
+ ret = perf_evlist__parse_sample_timestamp(evlist, event, ×tamp);
+ if (ret)
+ return ret;
+
+ return ordered_events__queue(&session->ordered_events, event,
+ timestamp, file_offset);
+}
+
+/*
+ * This function reads, merge and process directory data.
+ * It assumens the version 1 of directory data, where each
+ * data file holds per-cpu data, already sorted by kernel.
+ */
+static int __perf_session__process_dir_events(struct perf_session *session)
+{
+ struct perf_data *data = session->data;
+ struct perf_tool *tool = session->tool;
+ struct reader rd = {
+ .fd = perf_data__fd(session->data),
+ .data_size = session->header.data_size,
+ .data_offset = session->header.data_offset,
+ .process = process_simple,
+ };
+ int i, ret = 0;
+ struct ui_progress prog;
+ u64 total_size = perf_data__size(session->data);
+
+ perf_tool__fill_defaults(tool);
+
+ ui_progress__init_size(&prog, total_size, "Processing events...");
+
+ /*
+ * The data are sequentially read into session->ordered_events
+ * queue, without no flushing during the reading. When all of
+ * data is merged in, we flush the whole queue.
+ *
+ * Note, that we don't store PERF_RECORD_FINISHED_ROUND events
+ * for directory data, so there are no 'middle' flushes of the
+ * queue, just the final, when all is in.
+ *
+ * Read first data from the header file..
+ */
+ ret = reader__process_events(&rd, session, &prog);
+ if (ret)
+ goto out_err;
+
+ /* ... and continue with data files.. */
+ for (i = 0; i < data->dir.nr ; i++) {
+ struct perf_data_file *file = &data->dir.files[i];
+
+ if (file->size == 0)
+ continue;
+
+ rd = (struct reader) {
+ .fd = file->fd,
+ .data_size = file->size,
+ .data_offset = 0,
+ .process = process_index,
+ };
+
+ ret = reader__process_events(&rd, session, &prog);
+ if (ret)
+ goto out_err;
+ }
+
+ /* ... and flush everything out. */
+ ret = ordered_events__flush(&session->ordered_events, OE_FLUSH__FINAL);
+
+out_err:
+ if (!tool->no_warn)
+ perf_session__warn_about_errors(session);
+
+ /*
+ * We may switching perf.data output, make ordered_events
+ * reusable.
+ */
+ ordered_events__reinit(&session->ordered_events);
+ return ret;
+}
+
int perf_session__process_events(struct perf_session *session)
{
if (perf_session__register_idle_thread(session) < 0)
@@ -2014,6 +2111,9 @@ int perf_session__process_events(struct perf_session *session)
if (perf_data__is_pipe(session->data))
return __perf_session__process_pipe_events(session);
+ if (perf_data__is_dir(session->data))
+ return __perf_session__process_dir_events(session);
+
return __perf_session__process_events(session);
}
--
2.17.2
^ permalink raw reply related [flat|nested] 8+ messages in thread
* [PATCH 2/5] perf session: Add path to reader object
2019-03-10 15:30 [PATCHv6 0/5] perf record: Add support to store data in directory Jiri Olsa
2019-03-10 15:30 ` [PATCH 1/5] perf session: Add __perf_session__process_dir_events function Jiri Olsa
@ 2019-03-10 15:30 ` Jiri Olsa
2019-03-10 15:30 ` [PATCH 3/5] perf record: Add --dir option to store data in directory Jiri Olsa
` (3 subsequent siblings)
5 siblings, 0 replies; 8+ messages in thread
From: Jiri Olsa @ 2019-03-10 15:30 UTC (permalink / raw)
To: Arnaldo Carvalho de Melo
Cc: lkml, Ingo Molnar, Namhyung Kim, Alexander Shishkin,
Peter Zijlstra, Adrian Hunter, Andi Kleen, Stephane Eranian,
Alexey Budankov
Adding path to reader object, so we can display file
the processing fails for (string in [] brackets).
$ perf report --stdio
0x5e0 [perf.data/data.3] [0xa200]: failed to process type: -1577027574
Link: http://lkml.kernel.org/n/tip-4bjnoy4sln7adqtd3505q29q@git.kernel.org
Signed-off-by: Jiri Olsa <jolsa@kernel.org>
---
tools/perf/util/session.c | 10 ++++++++--
1 file changed, 8 insertions(+), 2 deletions(-)
diff --git a/tools/perf/util/session.c b/tools/perf/util/session.c
index b0c29a19f189..942067d7d853 100644
--- a/tools/perf/util/session.c
+++ b/tools/perf/util/session.c
@@ -1858,6 +1858,7 @@ struct reader {
u64 data_size;
u64 data_offset;
reader_cb_t process;
+ char *path;
};
static int
@@ -1872,6 +1873,8 @@ reader__process_events(struct reader *rd, struct perf_session *session,
union perf_event *event;
s64 skip;
+ pr_debug("reader processing %s\n", rd->path);
+
page_offset = page_size * (rd->data_offset / page_size);
file_offset = page_offset;
head = rd->data_offset - page_offset;
@@ -1927,8 +1930,8 @@ reader__process_events(struct reader *rd, struct perf_session *session,
if (size < sizeof(struct perf_event_header) ||
(skip = rd->process(session, event, file_pos)) < 0) {
- pr_err("%#" PRIx64 " [%#x]: failed to process type: %d\n",
- file_offset + head, event->header.size,
+ pr_err("%#" PRIx64 " [%s] [%#x]: failed to process type: %d\n",
+ file_offset + head, rd->path, event->header.size,
event->header.type);
err = -EINVAL;
goto out;
@@ -1966,6 +1969,7 @@ static int __perf_session__process_events(struct perf_session *session)
.data_size = session->header.data_size,
.data_offset = session->header.data_offset,
.process = process_simple,
+ .path = session->data->file.path,
};
struct ordered_events *oe = &session->ordered_events;
struct perf_tool *tool = session->tool;
@@ -2045,6 +2049,7 @@ static int __perf_session__process_dir_events(struct perf_session *session)
.data_size = session->header.data_size,
.data_offset = session->header.data_offset,
.process = process_simple,
+ .path = session->data->file.path,
};
int i, ret = 0;
struct ui_progress prog;
@@ -2081,6 +2086,7 @@ static int __perf_session__process_dir_events(struct perf_session *session)
.data_size = file->size,
.data_offset = 0,
.process = process_index,
+ .path = file->path,
};
ret = reader__process_events(&rd, session, &prog);
--
2.17.2
^ permalink raw reply related [flat|nested] 8+ messages in thread
* [PATCH 3/5] perf record: Add --dir option to store data in directory
2019-03-10 15:30 [PATCHv6 0/5] perf record: Add support to store data in directory Jiri Olsa
2019-03-10 15:30 ` [PATCH 1/5] perf session: Add __perf_session__process_dir_events function Jiri Olsa
2019-03-10 15:30 ` [PATCH 2/5] perf session: Add path to reader object Jiri Olsa
@ 2019-03-10 15:30 ` Jiri Olsa
2019-03-13 14:07 ` Namhyung Kim
2019-03-10 15:30 ` [PATCH 4/5] perf record: Add --output-dir " Jiri Olsa
` (2 subsequent siblings)
5 siblings, 1 reply; 8+ messages in thread
From: Jiri Olsa @ 2019-03-10 15:30 UTC (permalink / raw)
To: Arnaldo Carvalho de Melo
Cc: lkml, Ingo Molnar, Namhyung Kim, Alexander Shishkin,
Peter Zijlstra, Adrian Hunter, Andi Kleen, Stephane Eranian,
Alexey Budankov
Adding --dir option to store data in directory. It's next
step for multiple threads in record. It's now possible to
make directory data via --dir option, like:
$ perf record --dir perf bench sched messaging
$ ls -l perf.data
total 344
-rw-------. 1 jolsa jolsa 43864 Jan 20 22:26 data.0
-rw-------. 1 jolsa jolsa 30464 Jan 20 22:26 data.1
-rw-------. 1 jolsa jolsa 53816 Jan 20 22:26 data.2
-rw-------. 1 jolsa jolsa 30368 Jan 20 22:26 data.3
-rw-------. 1 jolsa jolsa 40088 Jan 20 22:26 data.4
-rw-------. 1 jolsa jolsa 42592 Jan 20 22:26 data.5
-rw-------. 1 jolsa jolsa 56136 Jan 20 22:26 data.6
-rw-------. 1 jolsa jolsa 25992 Jan 20 22:26 data.7
-rw-------. 1 jolsa jolsa 8832 Jan 20 22:26 header
There's a data file created for every cpu and it's storing
data for those cpu maps.
It's possible to transform directory data into standard
perf.data file via following inject command:
$ perf inject -o perf.data.file -i perf.data
The --dir option enabled DIR_FORMAT feature to be stored
in header file to indicate the directory layout.
Don't allow to use --dir with --aio yet. It needs
to be investigated first.
Link: http://lkml.kernel.org/n/tip-0kjm8wpglzu2tm18tpagfm4d@git.kernel.org
Signed-off-by: Jiri Olsa <jolsa@kernel.org>
---
tools/perf/Documentation/perf-record.txt | 3 +
tools/perf/builtin-record.c | 80 ++++++++++++++++++++++--
tools/perf/util/mmap.h | 23 +++----
3 files changed, 90 insertions(+), 16 deletions(-)
diff --git a/tools/perf/Documentation/perf-record.txt b/tools/perf/Documentation/perf-record.txt
index 8f0c2be34848..445b7a4eb130 100644
--- a/tools/perf/Documentation/perf-record.txt
+++ b/tools/perf/Documentation/perf-record.txt
@@ -524,6 +524,9 @@ config terms. For example: 'cycles/overwrite/' and 'instructions/no-overwrite/'.
Implies --tail-synthesize.
+--dir::
+Store data into directory with one data file for cpu.
+
SEE ALSO
--------
linkperf:perf-stat[1], linkperf:perf-list[1]
diff --git a/tools/perf/builtin-record.c b/tools/perf/builtin-record.c
index a468d882e74f..26981be13aa0 100644
--- a/tools/perf/builtin-record.c
+++ b/tools/perf/builtin-record.c
@@ -112,10 +112,13 @@ static bool switch_output_time(struct record *rec)
trigger_is_ready(&switch_output_trigger);
}
-static int record__write(struct record *rec, struct perf_mmap *map __maybe_unused,
+static int record__write(struct record *rec, struct perf_mmap *map,
void *bf, size_t size)
{
- struct perf_data_file *file = &rec->session->data->file;
+ struct perf_data_file *file = &rec->data.file;
+
+ if (map && map->file)
+ file = map->file;
if (perf_data_file__write(file, bf, size) < 0) {
pr_err("failed to write perf data, error: %m\n");
@@ -124,6 +127,15 @@ static int record__write(struct record *rec, struct perf_mmap *map __maybe_unuse
rec->bytes_written += size;
+ /*
+ * Update header file size manualy, data files size are
+ * ok to be updated by stat command, but header files
+ * contains more stuff, so we need to track data size
+ * manualy.
+ */
+ if (file == &rec->data.file)
+ rec->session->header.data_size += size;
+
if (switch_output_size(rec))
trigger_hit(&switch_output_trigger);
@@ -247,6 +259,7 @@ static int record__aio_pushfn(void *to, struct aiocb *cblock, void *bf, size_t s
ret = record__aio_write(cblock, trace_fd, bf, size, off);
if (!ret) {
rec->bytes_written += size;
+ rec->session->header.data_size += size;
if (switch_output_size(rec))
trigger_hit(&switch_output_trigger);
}
@@ -564,6 +577,25 @@ static int record__mmap_evlist(struct record *rec,
return 0;
}
+static int record__mmap_dir_data(struct record *rec)
+{
+ struct perf_evlist *evlist = rec->evlist;
+ struct perf_data *data = &rec->data;
+ int i, ret, nr = evlist->nr_mmaps;
+
+ ret = perf_data__create_dir(data, nr);
+ if (ret)
+ return ret;
+
+ for (i = 0; i < nr; i++) {
+ struct perf_mmap *map = &evlist->mmap[i];
+
+ map->file = &data->dir.files[i];
+ }
+
+ return 0;
+}
+
static int record__mmap(struct record *rec)
{
return record__mmap_evlist(rec, rec->evlist);
@@ -793,8 +825,12 @@ static int record__mmap_read_evlist(struct record *rec, struct perf_evlist *evli
/*
* Mark the round finished in case we wrote
* at least one event.
+ *
+ * No need for round events in directory mode,
+ * because per-cpu files/maps have sorted data
+ * from kernel.
*/
- if (bytes_written != rec->bytes_written)
+ if (!perf_data__is_dir(&rec->data) && bytes_written != rec->bytes_written)
rc = record__write(rec, NULL, &finished_round_event, sizeof(finished_round_event));
if (overwrite)
@@ -837,7 +873,8 @@ static void record__init_features(struct record *rec)
if (!(rec->opts.use_clockid && rec->opts.clockid_res_ns))
perf_header__clear_feat(&session->header, HEADER_CLOCKID);
- perf_header__clear_feat(&session->header, HEADER_DIR_FORMAT);
+ if (!perf_data__is_dir(session->data))
+ perf_header__clear_feat(&session->header, HEADER_DIR_FORMAT);
perf_header__clear_feat(&session->header, HEADER_STAT);
}
@@ -851,9 +888,11 @@ record__finish_output(struct record *rec)
if (data->is_pipe)
return;
- rec->session->header.data_size += rec->bytes_written;
data->file.size = lseek(perf_data__fd(data), 0, SEEK_CUR);
+ if (perf_data__is_dir(data))
+ perf_data__update_dir(data);
+
if (!rec->no_buildid) {
process_buildids(rec);
@@ -924,6 +963,12 @@ record__switch_output(struct record *rec, bool at_exit)
/* Output tracking events */
if (!at_exit) {
+ if (perf_data__is_dir(data)) {
+ err = record__mmap_dir_data(rec);
+ if (err)
+ return -1;
+ }
+
record__synthesize(rec, false);
/*
@@ -1173,11 +1218,23 @@ static int __cmd_record(struct record *rec, int argc, const char **argv)
if (data->is_pipe && rec->evlist->nr_entries == 1)
rec->opts.sample_id = true;
+ if (data->is_pipe && perf_data__is_dir(data)) {
+ pr_err("Directory output is not allowed for pipe output\n");
+ err = -1;
+ goto out_child;
+ }
+
if (record__open(rec) != 0) {
err = -1;
goto out_child;
}
+ if (perf_data__is_dir(data)) {
+ err = record__mmap_dir_data(rec);
+ if (err)
+ goto out_child;
+ }
+
err = bpf__apply_obj_config();
if (err) {
char errbuf[BUFSIZ];
@@ -1983,6 +2040,8 @@ static struct option __record_options[] = {
OPT_CALLBACK(0, "affinity", &record.opts, "node|cpu",
"Set affinity mask of trace reading thread to NUMA node cpu mask or cpu of processed mmap buffer",
record__parse_affinity),
+ OPT_BOOLEAN(0, "dir", &record.data.is_dir,
+ "Store data into directory perf.data"),
OPT_END()
};
@@ -2134,6 +2193,17 @@ int cmd_record(int argc, const char **argv)
goto out;
}
+ if (perf_data__is_dir(&rec->data)) {
+ if (!rec->opts.sample_time) {
+ pr_err("Sample timestamp is required for indexing\n");
+ goto out;
+ }
+ if (record__aio_enabled(rec)) {
+ pr_err("Cannot use both --dir and --aio yet.\n");
+ goto out;
+ }
+ }
+
if (rec->opts.target.tid && !rec->opts.no_inherit_set)
rec->opts.no_inherit = true;
diff --git a/tools/perf/util/mmap.h b/tools/perf/util/mmap.h
index e566c19b242b..3e8595a8d6ce 100644
--- a/tools/perf/util/mmap.h
+++ b/tools/perf/util/mmap.h
@@ -19,17 +19,18 @@ struct aiocb;
* @refcnt - e.g. code using PERF_EVENT_IOC_SET_OUTPUT to share this
*/
struct perf_mmap {
- void *base;
- int mask;
- int fd;
- int cpu;
- refcount_t refcnt;
- u64 prev;
- u64 start;
- u64 end;
- bool overwrite;
- struct auxtrace_mmap auxtrace_mmap;
- char event_copy[PERF_SAMPLE_MAX_SIZE] __aligned(8);
+ void *base;
+ int mask;
+ int fd;
+ int cpu;
+ refcount_t refcnt;
+ u64 prev;
+ u64 start;
+ u64 end;
+ bool overwrite;
+ struct auxtrace_mmap auxtrace_mmap;
+ struct perf_data_file *file;
+ char event_copy[PERF_SAMPLE_MAX_SIZE] __aligned(8);
#ifdef HAVE_AIO_SUPPORT
struct {
void **data;
--
2.17.2
^ permalink raw reply related [flat|nested] 8+ messages in thread
* [PATCH 4/5] perf record: Add --output-dir option to store data in directory
2019-03-10 15:30 [PATCHv6 0/5] perf record: Add support to store data in directory Jiri Olsa
` (2 preceding siblings ...)
2019-03-10 15:30 ` [PATCH 3/5] perf record: Add --dir option to store data in directory Jiri Olsa
@ 2019-03-10 15:30 ` Jiri Olsa
2019-03-10 15:30 ` [PATCH 5/5] perf record: Describe perf.data directory format Jiri Olsa
2019-03-13 14:29 ` [PATCHv6 0/5] perf record: Add support to store data in directory Namhyung Kim
5 siblings, 0 replies; 8+ messages in thread
From: Jiri Olsa @ 2019-03-10 15:30 UTC (permalink / raw)
To: Arnaldo Carvalho de Melo
Cc: lkml, Ingo Molnar, Namhyung Kim, Alexander Shishkin,
Peter Zijlstra, Adrian Hunter, Andi Kleen, Stephane Eranian,
Alexey Budankov
Adding --output-dir option to mimic -o and --dir options.
following commands do the same:
$ perf record -o perf.dir.data --dir ...
$ perf record --output-dir perf.dir.data ...
User cannot use both -o and output-dir together,
error is displayed.
Link: http://lkml.kernel.org/n/tip-76ldd2ss6vjvlnjgwy7wxfzt@git.kernel.org
Signed-off-by: Jiri Olsa <jolsa@kernel.org>
---
tools/lib/subcmd/parse-options.h | 4 ++++
tools/perf/Documentation/perf-record.txt | 3 +++
tools/perf/builtin-record.c | 13 +++++++++++--
3 files changed, 18 insertions(+), 2 deletions(-)
diff --git a/tools/lib/subcmd/parse-options.h b/tools/lib/subcmd/parse-options.h
index af9def589863..8a3be77a3346 100644
--- a/tools/lib/subcmd/parse-options.h
+++ b/tools/lib/subcmd/parse-options.h
@@ -146,6 +146,10 @@ struct option {
.value = check_vtype(v, const char **), .argh = (a), .help = (h), \
.flags = PARSE_OPT_OPTARG, .defval = (intptr_t)(d), \
.set = check_vtype(os, bool *)}
+#define OPT_STRING_SET(s, l, v, os, a, h) \
+ { .type = OPTION_STRING, .short_name = (s), .long_name = (l), \
+ .value = check_vtype(v, const char **), .argh = (a), .help = (h), \
+ .set = check_vtype(os, bool *)}
#define OPT_STRING_NOEMPTY(s, l, v, a, h) { .type = OPTION_STRING, .short_name = (s), .long_name = (l), .value = check_vtype(v, const char **), .argh = (a), .help = (h), .flags = PARSE_OPT_NOEMPTY}
#define OPT_DATE(s, l, v, h) \
{ .type = OPTION_CALLBACK, .short_name = (s), .long_name = (l), .value = (v), .argh = "time", .help = (h), .callback = parse_opt_approxidate_cb }
diff --git a/tools/perf/Documentation/perf-record.txt b/tools/perf/Documentation/perf-record.txt
index 445b7a4eb130..aac609887fb7 100644
--- a/tools/perf/Documentation/perf-record.txt
+++ b/tools/perf/Documentation/perf-record.txt
@@ -527,6 +527,9 @@ Implies --tail-synthesize.
--dir::
Store data into directory with one data file for cpu.
+--output-dir::
+Same as --dir option, can't be used together with -o option.
+
SEE ALSO
--------
linkperf:perf-stat[1], linkperf:perf-list[1]
diff --git a/tools/perf/builtin-record.c b/tools/perf/builtin-record.c
index 26981be13aa0..115316e94b34 100644
--- a/tools/perf/builtin-record.c
+++ b/tools/perf/builtin-record.c
@@ -80,6 +80,7 @@ struct record {
bool buildid_all;
bool timestamp_filename;
bool timestamp_boundary;
+ bool output_is_file;
struct switch_output switch_output;
unsigned long long samples;
cpu_set_t affinity_mask;
@@ -1921,8 +1922,10 @@ static struct option __record_options[] = {
OPT_STRING('C', "cpu", &record.opts.target.cpu_list, "cpu",
"list of cpus to monitor"),
OPT_U64('c', "count", &record.opts.user_interval, "event period to sample"),
- OPT_STRING('o', "output", &record.data.path, "file",
- "output file name"),
+ OPT_STRING_SET('o', "output", &record.data.path, &record.output_is_file,
+ "file", "output file name"),
+ OPT_STRING_SET(0, "output-dir", &record.data.path, &record.data.is_dir,
+ "file", "output directory name"),
OPT_BOOLEAN_SET('i', "no-inherit", &record.opts.no_inherit,
&record.opts.no_inherit_set,
"child tasks do not inherit counters"),
@@ -2101,6 +2104,12 @@ int cmd_record(int argc, const char **argv)
"cgroup monitoring only available in system-wide mode");
}
+
+ if (perf_data__is_dir(&rec->data) && record.output_is_file) {
+ ui__error("cannot use both -o and --output-dir\n");
+ return -EINVAL;
+ }
+
if (rec->opts.record_switch_events &&
!perf_can_record_switch_events()) {
ui__error("kernel does not support recording context switch events\n");
--
2.17.2
^ permalink raw reply related [flat|nested] 8+ messages in thread
* [PATCH 5/5] perf record: Describe perf.data directory format
2019-03-10 15:30 [PATCHv6 0/5] perf record: Add support to store data in directory Jiri Olsa
` (3 preceding siblings ...)
2019-03-10 15:30 ` [PATCH 4/5] perf record: Add --output-dir " Jiri Olsa
@ 2019-03-10 15:30 ` Jiri Olsa
2019-03-13 14:29 ` [PATCHv6 0/5] perf record: Add support to store data in directory Namhyung Kim
5 siblings, 0 replies; 8+ messages in thread
From: Jiri Olsa @ 2019-03-10 15:30 UTC (permalink / raw)
To: Arnaldo Carvalho de Melo
Cc: lkml, Ingo Molnar, Namhyung Kim, Alexander Shishkin,
Peter Zijlstra, Adrian Hunter, Andi Kleen, Stephane Eranian,
Alexey Budankov
Adding perf.data-directory-format.txt to describe the
directory data layout.
Link: http://lkml.kernel.org/n/tip-1c8u1thx63v2ldwfdas4xc5d@git.kernel.org
Signed-off-by: Jiri Olsa <jolsa@kernel.org>
---
.../perf.data-directory-format.txt | 54 +++++++++++++++++++
1 file changed, 54 insertions(+)
create mode 100644 tools/perf/Documentation/perf.data-directory-format.txt
diff --git a/tools/perf/Documentation/perf.data-directory-format.txt b/tools/perf/Documentation/perf.data-directory-format.txt
new file mode 100644
index 000000000000..bbd6d31b10c8
--- /dev/null
+++ b/tools/perf/Documentation/perf.data-directory-format.txt
@@ -0,0 +1,54 @@
+perf.data directory format
+
+DISCLAIMER This is not ABI yet and is subject to possible change
+ in following versions of perf. We will remove this
+ disclaimer once the directory format soaks in.
+
+
+This document describes the on-disk perf.data format, generated
+by perf record with --dir option and consumed by the other perf
+tools.
+
+The directory perf.data is created by perf record command by
+using the --dir option:
+
+ $ perf record --dir perf bench sched messaging
+ $ ls -l perf.data
+ total 344
+ -rw-------. 1 jolsa jolsa 43864 Jan 20 22:26 data.0
+ -rw-------. 1 jolsa jolsa 30464 Jan 20 22:26 data.1
+ -rw-------. 1 jolsa jolsa 53816 Jan 20 22:26 data.2
+ -rw-------. 1 jolsa jolsa 30368 Jan 20 22:26 data.3
+ -rw-------. 1 jolsa jolsa 40088 Jan 20 22:26 data.4
+ -rw-------. 1 jolsa jolsa 42592 Jan 20 22:26 data.5
+ -rw-------. 1 jolsa jolsa 56136 Jan 20 22:26 data.6
+ -rw-------. 1 jolsa jolsa 25992 Jan 20 22:26 data.7
+ -rw-------. 1 jolsa jolsa 8832 Jan 20 22:26 header
+
+The header file keeps the standard perf.data file header,
+and the data.* files keep data.
+
+header file
+-----------
+The header file following the standard format describe in
+Documentation/perf.data-file-format doc. Including its data
+portion that is used to store manually synthesized events.
+
+data file
+---------
+The data files layout is described by HEADER_DIR_FORMAT feature.
+Currently it holds only version number (1):
+
+ HEADER_DIR_FORMAT = 24
+
+ struct {
+ uint64_t version;
+ }
+
+The current only only version value 1 means that data files:
+ - follow the 'data.*' format
+ - contain raw events data in standard perf format as read
+ from kernel (and need to be sorted)
+
+Future versions are expected to describe different data files
+layout according to special needs.
--
2.17.2
^ permalink raw reply related [flat|nested] 8+ messages in thread
* Re: [PATCH 3/5] perf record: Add --dir option to store data in directory
2019-03-10 15:30 ` [PATCH 3/5] perf record: Add --dir option to store data in directory Jiri Olsa
@ 2019-03-13 14:07 ` Namhyung Kim
0 siblings, 0 replies; 8+ messages in thread
From: Namhyung Kim @ 2019-03-13 14:07 UTC (permalink / raw)
To: Jiri Olsa
Cc: Arnaldo Carvalho de Melo, lkml, Ingo Molnar, Alexander Shishkin,
Peter Zijlstra, Adrian Hunter, Andi Kleen, Stephane Eranian,
Alexey Budankov
Hi Jirka,
On Mon, Mar 11, 2019 at 12:30 AM Jiri Olsa <jolsa@kernel.org> wrote:
>
> Adding --dir option to store data in directory. It's next
> step for multiple threads in record. It's now possible to
> make directory data via --dir option, like:
>
> $ perf record --dir perf bench sched messaging
> $ ls -l perf.data
> total 344
> -rw-------. 1 jolsa jolsa 43864 Jan 20 22:26 data.0
> -rw-------. 1 jolsa jolsa 30464 Jan 20 22:26 data.1
> -rw-------. 1 jolsa jolsa 53816 Jan 20 22:26 data.2
> -rw-------. 1 jolsa jolsa 30368 Jan 20 22:26 data.3
> -rw-------. 1 jolsa jolsa 40088 Jan 20 22:26 data.4
> -rw-------. 1 jolsa jolsa 42592 Jan 20 22:26 data.5
> -rw-------. 1 jolsa jolsa 56136 Jan 20 22:26 data.6
> -rw-------. 1 jolsa jolsa 25992 Jan 20 22:26 data.7
> -rw-------. 1 jolsa jolsa 8832 Jan 20 22:26 header
>
> There's a data file created for every cpu and it's storing
> data for those cpu maps.
>
> It's possible to transform directory data into standard
> perf.data file via following inject command:
>
> $ perf inject -o perf.data.file -i perf.data
>
> The --dir option enabled DIR_FORMAT feature to be stored
> in header file to indicate the directory layout.
>
> Don't allow to use --dir with --aio yet. It needs
> to be investigated first.
>
> Link: http://lkml.kernel.org/n/tip-0kjm8wpglzu2tm18tpagfm4d@git.kernel.org
> Signed-off-by: Jiri Olsa <jolsa@kernel.org>
> ---
[SNIP]
> @@ -1983,6 +2040,8 @@ static struct option __record_options[] = {
> OPT_CALLBACK(0, "affinity", &record.opts, "node|cpu",
> "Set affinity mask of trace reading thread to NUMA node cpu mask or cpu of processed mmap buffer",
> record__parse_affinity),
> + OPT_BOOLEAN(0, "dir", &record.data.is_dir,
> + "Store data into directory perf.data"),
> OPT_END()
> };
>
> @@ -2134,6 +2193,17 @@ int cmd_record(int argc, const char **argv)
> goto out;
> }
>
> + if (perf_data__is_dir(&rec->data)) {
> + if (!rec->opts.sample_time) {
> + pr_err("Sample timestamp is required for indexing\n");
s/indexing/directory output/ ?
Thanks
Namhyung
> + goto out;
> + }
> + if (record__aio_enabled(rec)) {
> + pr_err("Cannot use both --dir and --aio yet.\n");
> + goto out;
> + }
> + }
> +
> if (rec->opts.target.tid && !rec->opts.no_inherit_set)
> rec->opts.no_inherit = true;
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: [PATCHv6 0/5] perf record: Add support to store data in directory
2019-03-10 15:30 [PATCHv6 0/5] perf record: Add support to store data in directory Jiri Olsa
` (4 preceding siblings ...)
2019-03-10 15:30 ` [PATCH 5/5] perf record: Describe perf.data directory format Jiri Olsa
@ 2019-03-13 14:29 ` Namhyung Kim
5 siblings, 0 replies; 8+ messages in thread
From: Namhyung Kim @ 2019-03-13 14:29 UTC (permalink / raw)
To: Jiri Olsa
Cc: Arnaldo Carvalho de Melo, lkml, Ingo Molnar, Alexander Shishkin,
Peter Zijlstra, Adrian Hunter, Andi Kleen, Stephane Eranian,
Alexey Budankov
On Mon, Mar 11, 2019 at 12:30 AM Jiri Olsa <jolsa@kernel.org> wrote:
>
> hi,
> this patchset adds the --dir option to record command (and all
> the other record command that overload cmd_record) that allows
> the data to be stored in directory with multiple data files.
>
> It's next step for multiple threads implementation in record.
> It's now possible to make directory data via --dir option, like:
>
> $ perf record --dir perf bench sched messaging
> $ ls -l perf.data
> total 344
> -rw-------. 1 jolsa jolsa 43864 Jan 20 22:26 data.0
> -rw-------. 1 jolsa jolsa 30464 Jan 20 22:26 data.1
> -rw-------. 1 jolsa jolsa 53816 Jan 20 22:26 data.2
> -rw-------. 1 jolsa jolsa 30368 Jan 20 22:26 data.3
> -rw-------. 1 jolsa jolsa 40088 Jan 20 22:26 data.4
> -rw-------. 1 jolsa jolsa 42592 Jan 20 22:26 data.5
> -rw-------. 1 jolsa jolsa 56136 Jan 20 22:26 data.6
> -rw-------. 1 jolsa jolsa 25992 Jan 20 22:26 data.7
> -rw-------. 1 jolsa jolsa 8832 Jan 20 22:26 header
>
> There's a data file created for every cpu and it's storing
> data for those cpu maps. The report command will read it
> transparently, sort it and display as single file data.
>
> There's new DIR_FORMAT feature to describe directory data
> layout/format. In future we can describe different data files
> layout according to special needs.
>
> It's possible to transform directory data into standard
> perf.data file via simple inject command:
>
> $ perf inject -o perf.data.file -i perf.data
>
> The old perf fails over the directory data with following message:
> $ perf report
> incompatible file format (rerun with -v to learn more)
>
> I'm now testing the record threads support, so I'd like to
> have some agreement on the directory data support before.
>
> v6 changes:
> - rebased to latest Arnaldo's perf/core
> - some of the patches already taken
> - adding comments to __perf_session__process_dir_events
>
> v5 changes:
> - rebased to latest Arnaldo's perf/core
>
> v4 changes:
> - some of the patches already taken
> - removing up perf.data directory on perf_data__open
> error path
>
> v3 changes:
> - add rm_rf_perf_data to safely remove file/directory perf data
> - allocation fix in perf_data__create_dir
>
> v2 changes:
> - rm_rf changes are already accepted with requested changes
> - updated doc/man plus adding perf.data-directory-format.txt
> to describe directory format/layout
> - the --switch-output options now works over directory data
> - data rollback is not part of this patchset, updated my TODO though ;-)
> - added --output-dir option to combine -o and --dir
> - added DIR_FORMAT feature to describe directory data
> - disabling directory output for aio for now
>
> It's also available in here:
> git://git.kernel.org/pub/scm/linux/kernel/git/jolsa/perf.git
> perf/dir
>
It seems I'm too late, but anyway for the series
Acked-by: Namhyung Kim <namhyung@kernel.org>
Thanks for the work!
Namhyung
> Jiri Olsa (5):
> perf session: Add __perf_session__process_dir_events function
> perf session: Add path to reader object
> perf record: Add --dir option to store data in directory
> perf record: Add --output-dir option to store data in directory
> perf record: Describe perf.data directory format
>
> tools/lib/subcmd/parse-options.h | 4 ++++
> tools/perf/Documentation/perf-record.txt | 6 ++++++
> tools/perf/Documentation/perf.data-directory-format.txt | 54 ++++++++++++++++++++++++++++++++++++++++++++++++++++++
> tools/perf/builtin-record.c | 93 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++-------
> tools/perf/util/mmap.h | 23 ++++++++++++-----------
> tools/perf/util/session.c | 114 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++----
> 6 files changed, 272 insertions(+), 22 deletions(-)
> create mode 100644 tools/perf/Documentation/perf.data-directory-format.txt
^ permalink raw reply [flat|nested] 8+ messages in thread
end of thread, other threads:[~2019-03-13 14:30 UTC | newest]
Thread overview: 8+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2019-03-10 15:30 [PATCHv6 0/5] perf record: Add support to store data in directory Jiri Olsa
2019-03-10 15:30 ` [PATCH 1/5] perf session: Add __perf_session__process_dir_events function Jiri Olsa
2019-03-10 15:30 ` [PATCH 2/5] perf session: Add path to reader object Jiri Olsa
2019-03-10 15:30 ` [PATCH 3/5] perf record: Add --dir option to store data in directory Jiri Olsa
2019-03-13 14:07 ` Namhyung Kim
2019-03-10 15:30 ` [PATCH 4/5] perf record: Add --output-dir " Jiri Olsa
2019-03-10 15:30 ` [PATCH 5/5] perf record: Describe perf.data directory format Jiri Olsa
2019-03-13 14:29 ` [PATCHv6 0/5] perf record: Add support to store data in directory Namhyung Kim
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).