From: Jiri Olsa <jolsa@redhat.com>
To: Alexey Bayduraev <alexey.v.bayduraev@linux.intel.com>
Cc: Arnaldo Carvalho de Melo <acme@kernel.org>,
Namhyung Kim <namhyung@kernel.org>,
Alexander Shishkin <alexander.shishkin@linux.intel.com>,
Peter Zijlstra <peterz@infradead.org>,
Ingo Molnar <mingo@redhat.com>,
linux-kernel <linux-kernel@vger.kernel.org>,
Andi Kleen <ak@linux.intel.com>,
Adrian Hunter <adrian.hunter@intel.com>,
Alexander Antonov <alexander.antonov@linux.intel.com>,
Alexei Budankov <abudankov@huawei.com>,
Riccardo Mancini <rickyman7@gmail.com>
Subject: Re: [PATCH v11 24/24] perf session: Load data directory files for analysis
Date: Sun, 12 Sep 2021 22:45:11 +0200 [thread overview]
Message-ID: <YT5m18X9Mve2xgE3@krava> (raw)
In-Reply-To: <03e17b23b46647db7a71b9255ed2e3cd0cfd76e2.1629186429.git.alexey.v.bayduraev@linux.intel.com>
On Tue, Aug 17, 2021 at 11:23:27AM +0300, Alexey Bayduraev wrote:
> Load data directory files and provide basic raw dump and aggregated
> analysis support of data directories in report mode, still with no
> memory consumption optimizations.
>
> Design and implementation are based on the prototype [1], [2].
>
> [1] git clone https://git.kernel.org/pub/scm/linux/kernel/git/jolsa/perf.git -b perf/record_threads
> [2] https://lore.kernel.org/lkml/20180913125450.21342-1-jolsa@kernel.org/
>
> Suggested-by: Jiri Olsa <jolsa@kernel.org>
> Acked-by: Namhyung Kim <namhyung@gmail.com>
> Reviewed-by: Riccardo Mancini <rickyman7@gmail.com>
> Tested-by: Riccardo Mancini <rickyman7@gmail.com>
> Signed-off-by: Alexey Bayduraev <alexey.v.bayduraev@linux.intel.com>
> ---
> tools/perf/util/session.c | 131 ++++++++++++++++++++++++++++++++++++++
> 1 file changed, 131 insertions(+)
>
> diff --git a/tools/perf/util/session.c b/tools/perf/util/session.c
> index b49b52768681..f5a82f66e36b 100644
> --- a/tools/perf/util/session.c
> +++ b/tools/perf/util/session.c
> @@ -48,6 +48,13 @@
> #define NUM_MMAPS 128
> #endif
>
> +/*
> + * Processing 10MBs of data from each reader in sequence,
> + * because that's the way the ordered events sorting works
> + * most efficiently.
> + */
> +#define READER_MAX_SIZE (10 * 1024 * 1024)
why 10? ;-) I think we discussed this in your earlier iteration,
I put the 10M there to speedup the processing, because it worked
on my current data file for the test at that time
I think this needs some better justification and some setup based
on the env setup and data size perhaps?
jirka
> +
> struct reader;
>
> typedef s64 (*reader_cb_t)(struct perf_session *session,
> @@ -65,6 +72,7 @@ struct reader_state {
> u64 data_size;
> u64 head;
> bool eof;
> + u64 size;
> };
>
> enum {
> @@ -2328,6 +2336,7 @@ reader__read_event(struct reader *rd, struct perf_session *session,
> if (skip)
> size += skip;
>
> + st->size += size;
> st->head += size;
> st->file_pos += size;
>
> @@ -2427,6 +2436,125 @@ static int __perf_session__process_events(struct perf_session *session)
> return err;
> }
>
> +/*
> + * This function reads, merge and process directory data.
> + * It assumens the version 1 of directory data, where each
> + * data file holds per-cpu data, already sorted by kernel.
> + */
> +static int __perf_session__process_dir_events(struct perf_session *session)
> +{
> + struct perf_data *data = session->data;
> + struct perf_tool *tool = session->tool;
> + int i, ret = 0, readers = 1;
> + struct ui_progress prog;
> + u64 total_size = perf_data__size(session->data);
> + struct reader *rd;
> +
> + perf_tool__fill_defaults(tool);
> +
> + ui_progress__init_size(&prog, total_size, "Sorting events...");
> +
> + for (i = 0; i < data->dir.nr; i++) {
> + if (data->dir.files[i].size)
> + readers++;
> + }
> +
> + rd = session->readers = zalloc(readers * sizeof(struct reader));
> + if (!rd)
> + return -ENOMEM;
> + session->nr_readers = readers;
> + readers = 0;
> +
> + rd[readers] = (struct reader) {
> + .fd = perf_data__fd(session->data),
> + .path = session->data->file.path,
> + .data_size = session->header.data_size,
> + .data_offset = session->header.data_offset,
> + .in_place_update = session->data->in_place_update,
> + };
> + ret = reader__init(&rd[readers], NULL);
> + if (ret)
> + goto out_err;
> + ret = reader__mmap(&rd[readers], session);
> + if (ret != READER_OK) {
> + if (ret == READER_EOF)
> + ret = -EINVAL;
> + goto out_err;
> + }
> + readers++;
> +
> + for (i = 0; i < data->dir.nr; i++) {
> + if (!data->dir.files[i].size)
> + continue;
> + rd[readers] = (struct reader) {
> + .fd = data->dir.files[i].fd,
> + .path = data->dir.files[i].path,
> + .data_size = data->dir.files[i].size,
> + .data_offset = 0,
> + .in_place_update = session->data->in_place_update,
> + };
> + ret = reader__init(&rd[readers], NULL);
> + if (ret)
> + goto out_err;
> + ret = reader__mmap(&rd[readers], session);
> + if (ret != READER_OK) {
> + if (ret == READER_EOF)
> + ret = -EINVAL;
> + goto out_err;
> + }
> + readers++;
> + }
> +
> + i = 0;
> +
> + while ((ret >= 0) && readers) {
> + if (session_done())
> + return 0;
> +
> + if (rd[i].state.eof) {
> + i = (i + 1) % session->nr_readers;
> + continue;
> + }
> +
> + ret = reader__read_event(&rd[i], session, &prog);
> + if (ret < 0)
> + break;
> + if (ret == READER_NODATA) {
> + ret = reader__mmap(&rd[i], session);
> + if (ret < 0)
> + goto out_err;
> + if (ret == READER_EOF)
> + readers--;
> + }
> +
> + if (rd[i].state.size >= READER_MAX_SIZE) {
> + rd[i].state.size = 0;
> + i = (i + 1) % session->nr_readers;
> + }
> + }
> +
> + ret = ordered_events__flush(&session->ordered_events, OE_FLUSH__FINAL);
> + if (ret)
> + goto out_err;
> +
> + ret = perf_session__flush_thread_stacks(session);
> +out_err:
> + ui_progress__finish();
> +
> + if (!tool->no_warn)
> + perf_session__warn_about_errors(session);
> +
> + /*
> + * We may switching perf.data output, make ordered_events
> + * reusable.
> + */
> + ordered_events__reinit(&session->ordered_events);
> +
> + session->one_mmap = false;
> +
> + return ret;
> +}
> +
> int perf_session__process_events(struct perf_session *session)
> {
> if (perf_session__register_idle_thread(session) < 0)
> @@ -2435,6 +2563,9 @@ int perf_session__process_events(struct perf_session *session)
> if (perf_data__is_pipe(session->data))
> return __perf_session__process_pipe_events(session);
>
> + if (perf_data__is_dir(session->data))
> + return __perf_session__process_dir_events(session);
> +
> return __perf_session__process_events(session);
> }
>
> --
> 2.19.0
>
next prev parent reply other threads:[~2021-09-12 20:45 UTC|newest]
Thread overview: 35+ messages / expand[flat|nested] mbox.gz Atom feed top
2021-08-17 8:23 [PATCH v11 00/24] Introduce threaded trace streaming for basic perf record operation Alexey Bayduraev
2021-08-17 8:23 ` [PATCH v11 01/24] perf record: Introduce thread affinity and mmap masks Alexey Bayduraev
2021-09-12 20:45 ` Jiri Olsa
2021-08-17 8:23 ` [PATCH v11 02/24] tools lib: Introduce fdarray duplicate function Alexey Bayduraev
2021-08-17 8:23 ` [PATCH v11 03/24] perf record: Introduce thread specific data array Alexey Bayduraev
2021-08-17 8:23 ` [PATCH v11 04/24] perf record: Introduce function to propagate control commands Alexey Bayduraev
2021-08-17 8:23 ` [PATCH v11 05/24] perf record: Introduce thread local variable Alexey Bayduraev
2021-08-17 8:23 ` [PATCH v11 06/24] perf record: Stop threads in the end of trace streaming Alexey Bayduraev
2021-08-17 8:23 ` [PATCH v11 07/24] perf record: Start threads in the beginning " Alexey Bayduraev
2021-09-12 20:46 ` Jiri Olsa
2021-08-17 8:23 ` [PATCH v11 08/24] perf record: Introduce data file at mmap buffer object Alexey Bayduraev
2021-09-12 20:46 ` Jiri Olsa
2021-08-17 8:23 ` [PATCH v11 09/24] perf record: Introduce bytes written stats to support --max-size option Alexey Bayduraev
2021-09-12 20:46 ` Jiri Olsa
2021-09-20 12:54 ` Bayduraev, Alexey V
2021-09-12 20:46 ` Jiri Olsa
2021-08-17 8:23 ` [PATCH v11 10/24] perf record: Introduce data transferred and compressed stats Alexey Bayduraev
2021-08-17 8:23 ` [PATCH v11 11/24] perf record: Init data file at mmap buffer object Alexey Bayduraev
2021-08-17 8:23 ` [PATCH v11 12/24] perf record: Introduce --threads command line option Alexey Bayduraev
2021-08-17 8:23 ` [PATCH v11 13/24] perf record: Extend " Alexey Bayduraev
2021-09-12 21:01 ` Jiri Olsa
2021-08-17 8:23 ` [PATCH v11 14/24] perf record: Implement compatibility checks Alexey Bayduraev
2021-08-17 8:23 ` [PATCH v11 15/24] perf report: Output non-zero offset for decompressed records Alexey Bayduraev
2021-08-17 8:23 ` [PATCH v11 16/24] perf report: Output data file name in raw trace dump Alexey Bayduraev
2021-08-17 8:23 ` [PATCH v11 17/24] perf session: Move reader structure to the top Alexey Bayduraev
2021-08-17 8:23 ` [PATCH v11 18/24] perf session: Introduce reader_state in reader object Alexey Bayduraev
2021-08-17 8:23 ` [PATCH v11 19/24] perf session: Introduce reader objects in session object Alexey Bayduraev
2021-09-12 20:44 ` Jiri Olsa
2021-08-17 8:23 ` [PATCH v11 20/24] perf session: Introduce decompressor into trace reader object Alexey Bayduraev
2021-08-17 8:23 ` [PATCH v11 21/24] perf session: Move init into reader__init function Alexey Bayduraev
2021-08-17 8:23 ` [PATCH v11 22/24] perf session: Move map/unmap into reader__mmap function Alexey Bayduraev
2021-08-17 8:23 ` [PATCH v11 23/24] perf session: Load single file for analysis Alexey Bayduraev
2021-08-17 8:23 ` [PATCH v11 24/24] perf session: Load data directory files " Alexey Bayduraev
2021-09-12 20:45 ` Jiri Olsa [this message]
2021-09-12 20:44 ` [PATCH v11 00/24] Introduce threaded trace streaming for basic perf record operation Jiri Olsa
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=YT5m18X9Mve2xgE3@krava \
--to=jolsa@redhat.com \
--cc=abudankov@huawei.com \
--cc=acme@kernel.org \
--cc=adrian.hunter@intel.com \
--cc=ak@linux.intel.com \
--cc=alexander.antonov@linux.intel.com \
--cc=alexander.shishkin@linux.intel.com \
--cc=alexey.v.bayduraev@linux.intel.com \
--cc=linux-kernel@vger.kernel.org \
--cc=mingo@redhat.com \
--cc=namhyung@kernel.org \
--cc=peterz@infradead.org \
--cc=rickyman7@gmail.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).