All of lore.kernel.org
 help / color / mirror / Atom feed
* [RFC/PATCHSET 00/42] perf tools: Speed-up perf report by using multi thread (v2)
@ 2015-01-29  8:06 Namhyung Kim
  2015-01-29  8:06 ` [PATCH 01/42] perf tools: Support to read compressed module from build-id cache Namhyung Kim
                   ` (42 more replies)
  0 siblings, 43 replies; 221+ messages in thread
From: Namhyung Kim @ 2015-01-29  8:06 UTC (permalink / raw)
  To: Arnaldo Carvalho de Melo
  Cc: Ingo Molnar, Peter Zijlstra, Jiri Olsa, LKML, David Ahern,
	Adrian Hunter, Andi Kleen, Stephane Eranian, Frederic Weisbecker

Hello,

This patchset converts perf report to use multiple threads in order to
speed up the processing on large data files.  I can see a minimum ~30%
of speedup with this change.  The code is still experimental and
contains many rough edges.  But I'd like to share and give some
feedbacks.

The main change in this version is using single data file with an
index table rather than using multiple files.  It seems that single
thread performance was improved by this than previous version but multi
thread performance remains almost same.

The perf report processes (sample) events like below:

  1. preprocess sample to get matching thread/dso/symbol info
  2. insert it to hists rbtree (with callchain tree) based on the info
  3. optionally collapse hist entries that match given sort key(s)
  4. resort hist entries (by overhead) for output
  5. display the hist entries

The stage 1 is a preprocessing and mostly act like a read-only
operation in that it doesn't change a machine state during the sample
processing.  Meta events like fork, comm and mmap can change the
machine/thread state but symbols can be loaded during the processing
(stage 2).

The stage 2 consumes most of the time especially with callchains and
 --children option is enabled.  And this work can be easily patitioned
as each sample is independent to others.  But the resulting hists must
be combined/collapsed to a single global hists before going to further
steps.

The stage 3 is optional and only needed by certain sort keys - but
with stage 2 paralellized, it needs to be done always.

The stage 4 and 5 works on whole hists so must be done serially.

So my approach is like this:

Partially do stage 1 first - but only for meta events that changes
machine state.  To do this I add a dummy tracking event to perf record
and make it collect such meta events only.  They are saved as normal
data and processed before sample events at perf report time.

This also requires to handle multiple sample data concurrently and to
find a corresponding machine state when processing samples.  On a
large profiling session, many tasks were created and exited so pid
might be recycled (even more than once!).  To deal with it, I managed
to have thread, map_groups and comm in time sorted.  The only
remaining thing is symbol loading as it's done lazily when sample
requires it.

With that being done, the stage 2 can be done by multiple threads.  I
also save each sample data (per-cpu or per-thread) in separate files
during record and then merge them into a single data file with an
index table.  On perf report time, each region of sample data will be
processed by each thread.  And symbol loading is protected by a mutex
lock.

For DWARF post-unwinding, dso cache data also needs to be protected by
a lock and this caused a huge contention.  I made it to search the
rbtree speculatively first and then, if it didn't find one, search it
again under the dso lock.  Please take a look at it if it's acceptable.

The patch 1-4 are independent fixes and cleans.  The patch 5-14 are to
support indexing for data file.  With --index option, perf record will
create a intermediate directory and then save meta events and sample
data to separate files.  And finally it'll build an index table and
concatenate the data files.

The patch 15-26 are to manage machine and thread state using timestamp
so that it can be searched when processing samples.  The patch 27-40
are to implement parallel report.  And finally I implemented 'perf
data index' command to build an index table for a given data file.

This patchset didn't change perf record to use multi-thread.  But I
think it can be easily done later if needed.

Note that output has a slight difference to original version when
compared using indexed data file.  But they're mostly unresolved
symbols for callchains.

Here is the result:

This is just elapsed time measured by 'perf stat -r 5'.

The data file was recorded during kernel build with fp callchain and
size is 2.1GB.  The machine has 6 core with hyper-threading enabled
and I got a similar result on my laptop too.

 perf report          --children  --no-children  + --call-graph none
 		   -------------  -------------  -------------------
 current           285.708340593   94.317412961      36.707232978  
 with index        253.322717665   77.079748639      24.892021523
 + --multi-thread  174.037760271   44.717308080       8.300466711


This result is with 7.7GB data file using libunwind for callchain.

 perf report          --children  --no-children  + --call-graph none
 		   -------------  -------------  -------------------
 current           247.070444039  196.393820003       5.068489333
 with index        149.456483830  108.917644447       3.642109876
 + --multi-thread   43.990095636   28.342798882       1.829218561

I guess the speedup of indexed data file came from skipping ordered
event layer.

This result is with same file but using libdw for callchain unwind.

 perf report          --children  --no-children  + --call-graph none
 		   -------------  -------------  -------------------
 current           465.661321115  496.153153039       4.629841428
 with index        445.712762188  462.146612217       3.535147499
 + --multi-thread  215.264706814   29.279996335       1.938137940

On my archlinux system, callchain unwind using libdw is much slower
than libunwind.  I'm using elfutils version 0.160.  Also I don't know
why --children takes less time than --no-children.  Anyway we can see
the --multi-thread performance is much better for each case.


You can get it from 'perf/threaded-v2' branch on my tree at:

  git://git.kernel.org/pub/scm/linux/kernel/git/namhyung/linux-perf.git

Please take a look and play with it.  Any comments are welcome! :)

Thanks,
Namhyung


Jiri Olsa (1):
  perf tools: Add new perf data command

Namhyung Kim (41):
  perf tools: Support to read compressed module from build-id cache
  perf tools: Do not use __perf_session__process_events() directly
  perf record: Show precise number of samples
  perf header: Set header version correctly
  perf tools: Set attr.task bit for a tracking event
  perf tools: Use a software dummy event to track task/mmap events
  perf tools: Use perf_data_file__fd() consistently
  perf tools: Add rm_rf() utility function
  perf tools: Introduce copyfile_offset() function
  perf tools: Create separate mmap for dummy tracking event
  perf tools: Introduce perf_evlist__mmap_track()
  perf tools: Add HEADER_DATA_INDEX feature
  perf tools: Handle indexed data file properly
  perf record: Add --index option for building index table
  perf report: Skip dummy tracking event
  perf tools: Pass session arg to perf_event__preprocess_sample()
  perf script: Pass session arg to ->process_event callback
  perf tools: Introduce thread__comm_time() helpers
  perf tools: Add a test case for thread comm handling
  perf tools: Use thread__comm_time() when adding hist entries
  perf tools: Convert dead thread list into rbtree
  perf tools: Introduce machine__find*_thread_time()
  perf tools: Add a test case for timed thread handling
  perf tools: Maintain map groups list in a leader thread
  perf tools: Introduce thread__find_addr_location_time() and friends
  perf tools: Add a test case for timed map groups handling
  perf tools: Protect dso symbol loading using a mutex
  perf tools: Protect dso cache tree using dso->lock
  perf tools: Protect dso cache fd with a mutex
  perf session: Pass struct events stats to event processing functions
  perf hists: Pass hists struct to hist_entry_iter functions
  perf tools: Move BUILD_ID_SIZE definition to perf.h
  perf report: Parallelize perf report using multi-thread
  perf tools: Add missing_threads rb tree
  perf record: Synthesize COMM event for a command line workload
  perf tools: Fix progress ui to support multi thread
  perf report: Add --multi-thread option and config item
  perf session: Handle index files generally
  perf tools: Convert lseek + read to pread
  perf callchain: Save eh/debug frame offset for dwarf unwind
  perf data: Implement 'index' subcommand

 tools/perf/Documentation/perf-data.txt             |  44 +++
 tools/perf/Documentation/perf-record.txt           |   4 +
 tools/perf/Documentation/perf-report.txt           |   3 +
 tools/perf/Makefile.perf                           |   4 +
 tools/perf/builtin-annotate.c                      |   8 +-
 tools/perf/builtin-data.c                          | 428 +++++++++++++++++++++
 tools/perf/builtin-diff.c                          |  21 +-
 tools/perf/builtin-inject.c                        |   5 +-
 tools/perf/builtin-mem.c                           |   6 +-
 tools/perf/builtin-record.c                        | 261 +++++++++++--
 tools/perf/builtin-report.c                        |  74 +++-
 tools/perf/builtin-script.c                        |  54 ++-
 tools/perf/builtin-timechart.c                     |  10 +-
 tools/perf/builtin-top.c                           |   7 +-
 tools/perf/builtin.h                               |   1 +
 tools/perf/command-list.txt                        |   1 +
 tools/perf/perf.c                                  |   1 +
 tools/perf/perf.h                                  |   2 +
 tools/perf/tests/builtin-test.c                    |  12 +
 tools/perf/tests/dso-data.c                        |   5 +
 tools/perf/tests/dwarf-unwind.c                    |   8 +-
 tools/perf/tests/hists_common.c                    |   3 +-
 tools/perf/tests/hists_cumulate.c                  |   6 +-
 tools/perf/tests/hists_filter.c                    |   5 +-
 tools/perf/tests/hists_link.c                      |  10 +-
 tools/perf/tests/hists_output.c                    |   6 +-
 tools/perf/tests/tests.h                           |   3 +
 tools/perf/tests/thread-comm.c                     |  47 +++
 tools/perf/tests/thread-lookup-time.c              | 180 +++++++++
 tools/perf/tests/thread-mg-share.c                 |   7 +-
 tools/perf/tests/thread-mg-time.c                  |  88 +++++
 tools/perf/ui/browsers/hists.c                     |  30 +-
 tools/perf/ui/gtk/hists.c                          |   3 +
 tools/perf/util/build-id.c                         |   9 +-
 tools/perf/util/build-id.h                         |   2 -
 tools/perf/util/db-export.c                        |   6 +-
 tools/perf/util/db-export.h                        |   4 +-
 tools/perf/util/dso.c                              | 159 +++++---
 tools/perf/util/dso.h                              |   3 +
 tools/perf/util/event.c                            | 106 ++++-
 tools/perf/util/event.h                            |  13 +-
 tools/perf/util/evlist.c                           | 161 ++++++--
 tools/perf/util/evlist.h                           |  22 +-
 tools/perf/util/evsel.c                            |   1 +
 tools/perf/util/evsel.h                            |  15 +
 tools/perf/util/header.c                           |  63 ++-
 tools/perf/util/header.h                           |   3 +
 tools/perf/util/hist.c                             | 121 ++++--
 tools/perf/util/hist.h                             |  12 +-
 tools/perf/util/machine.c                          | 258 +++++++++++--
 tools/perf/util/machine.h                          |  12 +-
 tools/perf/util/map.c                              |   1 +
 tools/perf/util/map.h                              |   2 +
 tools/perf/util/ordered-events.c                   |   4 +-
 .../perf/util/scripting-engines/trace-event-perl.c |   3 +-
 .../util/scripting-engines/trace-event-python.c    |   5 +-
 tools/perf/util/session.c                          | 356 ++++++++++++++---
 tools/perf/util/session.h                          |  11 +-
 tools/perf/util/symbol-elf.c                       |  13 +-
 tools/perf/util/symbol.c                           |  34 +-
 tools/perf/util/thread.c                           | 139 ++++++-
 tools/perf/util/thread.h                           |  28 +-
 tools/perf/util/tool.h                             |  14 +
 tools/perf/util/trace-event-scripting.c            |   3 +-
 tools/perf/util/trace-event.h                      |   3 +-
 tools/perf/util/unwind-libdw.c                     |  11 +-
 tools/perf/util/unwind-libunwind.c                 |  49 ++-
 tools/perf/util/util.c                             |  81 +++-
 tools/perf/util/util.h                             |   2 +
 69 files changed, 2662 insertions(+), 414 deletions(-)
 create mode 100644 tools/perf/Documentation/perf-data.txt
 create mode 100644 tools/perf/builtin-data.c
 create mode 100644 tools/perf/tests/thread-comm.c
 create mode 100644 tools/perf/tests/thread-lookup-time.c
 create mode 100644 tools/perf/tests/thread-mg-time.c

-- 
2.2.2


^ permalink raw reply	[flat|nested] 221+ messages in thread

* [PATCH 01/42] perf tools: Support to read compressed module from build-id cache
  2015-01-29  8:06 [RFC/PATCHSET 00/42] perf tools: Speed-up perf report by using multi thread (v2) Namhyung Kim
@ 2015-01-29  8:06 ` Namhyung Kim
  2015-01-30 14:32   ` Jiri Olsa
  2015-01-30 18:33   ` [tip:perf/core] perf symbols: " tip-bot for Namhyung Kim
  2015-01-29  8:06 ` [PATCH 02/42] perf tools: Do not use __perf_session__process_events() directly Namhyung Kim
                   ` (41 subsequent siblings)
  42 siblings, 2 replies; 221+ messages in thread
From: Namhyung Kim @ 2015-01-29  8:06 UTC (permalink / raw)
  To: Arnaldo Carvalho de Melo
  Cc: Ingo Molnar, Peter Zijlstra, Jiri Olsa, LKML, David Ahern,
	Adrian Hunter, Andi Kleen, Stephane Eranian, Frederic Weisbecker

The commit c00c48fc6e6e ("perf symbols: Preparation for compressed
kernel module support") added support for compressed kernel modules
but it only supports system path DSOs.  When a dso is read from
build-id cache, its filename doesn't end with ".gz" but has build-id.
In this case, we should fallback to the original dso->name.

Signed-off-by: Namhyung Kim <namhyung@kernel.org>
---
 tools/perf/util/symbol-elf.c | 13 ++++++++-----
 1 file changed, 8 insertions(+), 5 deletions(-)

diff --git a/tools/perf/util/symbol-elf.c b/tools/perf/util/symbol-elf.c
index 06fcd1bf98b6..b24f9d8727a8 100644
--- a/tools/perf/util/symbol-elf.c
+++ b/tools/perf/util/symbol-elf.c
@@ -574,13 +574,16 @@ static int decompress_kmodule(struct dso *dso, const char *name,
 	const char *ext = strrchr(name, '.');
 	char tmpbuf[] = "/tmp/perf-kmod-XXXXXX";
 
-	if ((type != DSO_BINARY_TYPE__SYSTEM_PATH_KMODULE_COMP &&
-	     type != DSO_BINARY_TYPE__GUEST_KMODULE_COMP) ||
-	    type != dso->symtab_type)
+	if (type != DSO_BINARY_TYPE__SYSTEM_PATH_KMODULE_COMP &&
+	    type != DSO_BINARY_TYPE__GUEST_KMODULE_COMP &&
+	    type != DSO_BINARY_TYPE__BUILD_ID_CACHE)
 		return -1;
 
-	if (!ext || !is_supported_compression(ext + 1))
-		return -1;
+	if (!ext || !is_supported_compression(ext + 1)) {
+		ext = strrchr(dso->name, '.');
+		if (!ext || !is_supported_compression(ext + 1))
+			return -1;
+	}
 
 	fd = mkstemp(tmpbuf);
 	if (fd < 0)
-- 
2.2.2


^ permalink raw reply related	[flat|nested] 221+ messages in thread

* [PATCH 02/42] perf tools: Do not use __perf_session__process_events() directly
  2015-01-29  8:06 [RFC/PATCHSET 00/42] perf tools: Speed-up perf report by using multi thread (v2) Namhyung Kim
  2015-01-29  8:06 ` [PATCH 01/42] perf tools: Support to read compressed module from build-id cache Namhyung Kim
@ 2015-01-29  8:06 ` Namhyung Kim
  2015-01-30 18:32   ` [tip:perf/core] " tip-bot for Namhyung Kim
  2015-01-29  8:06 ` [PATCH 03/42] perf record: Show precise number of samples Namhyung Kim
                   ` (40 subsequent siblings)
  42 siblings, 1 reply; 221+ messages in thread
From: Namhyung Kim @ 2015-01-29  8:06 UTC (permalink / raw)
  To: Arnaldo Carvalho de Melo
  Cc: Ingo Molnar, Peter Zijlstra, Jiri Olsa, LKML, David Ahern,
	Adrian Hunter, Andi Kleen, Stephane Eranian, Frederic Weisbecker

It's only used for perf record to process build-id because its file
size it's not fixed at this time due to remaining header features.
However data offset and size is available so that we can use the
perf_session__process_events() once we set the file size as the
current offset like for now.

Signed-off-by: Namhyung Kim <namhyung@kernel.org>
---
 tools/perf/builtin-record.c | 7 +++----
 tools/perf/util/session.c   | 6 +++---
 tools/perf/util/session.h   | 3 ---
 3 files changed, 6 insertions(+), 10 deletions(-)

diff --git a/tools/perf/builtin-record.c b/tools/perf/builtin-record.c
index 8648c6d3003d..1134de22979e 100644
--- a/tools/perf/builtin-record.c
+++ b/tools/perf/builtin-record.c
@@ -194,12 +194,13 @@ static int process_buildids(struct record *rec)
 {
 	struct perf_data_file *file  = &rec->file;
 	struct perf_session *session = rec->session;
-	u64 start = session->header.data_offset;
 
 	u64 size = lseek(file->fd, 0, SEEK_CUR);
 	if (size == 0)
 		return 0;
 
+	file->size = size;
+
 	/*
 	 * During this process, it'll load kernel map and replace the
 	 * dso->long_name to a real pathname it found.  In this case
@@ -211,9 +212,7 @@ static int process_buildids(struct record *rec)
 	 */
 	symbol_conf.ignore_vmlinux_buildid = true;
 
-	return __perf_session__process_events(session, start,
-					      size - start,
-					      size, &build_id__mark_dso_hit_ops);
+	return perf_session__process_events(session, &build_id__mark_dso_hit_ops);
 }
 
 static void perf_event__synthesize_guest_os(struct machine *machine, void *data)
diff --git a/tools/perf/util/session.c b/tools/perf/util/session.c
index b0ce3d6e6231..0baf75f12b7c 100644
--- a/tools/perf/util/session.c
+++ b/tools/perf/util/session.c
@@ -1251,9 +1251,9 @@ fetch_mmaped_event(struct perf_session *session,
 #define NUM_MMAPS 128
 #endif
 
-int __perf_session__process_events(struct perf_session *session,
-				   u64 data_offset, u64 data_size,
-				   u64 file_size, struct perf_tool *tool)
+static int __perf_session__process_events(struct perf_session *session,
+					  u64 data_offset, u64 data_size,
+					  u64 file_size, struct perf_tool *tool)
 {
 	int fd = perf_data_file__fd(session->file);
 	u64 head, page_offset, file_offset, file_pos, size;
diff --git a/tools/perf/util/session.h b/tools/perf/util/session.h
index dc26ebf60fe4..6d663dc76404 100644
--- a/tools/perf/util/session.h
+++ b/tools/perf/util/session.h
@@ -49,9 +49,6 @@ int perf_session__peek_event(struct perf_session *session, off_t file_offset,
 			     union perf_event **event_ptr,
 			     struct perf_sample *sample);
 
-int __perf_session__process_events(struct perf_session *session,
-				   u64 data_offset, u64 data_size, u64 size,
-				   struct perf_tool *tool);
 int perf_session__process_events(struct perf_session *session,
 				 struct perf_tool *tool);
 
-- 
2.2.2


^ permalink raw reply related	[flat|nested] 221+ messages in thread

* [PATCH 03/42] perf record: Show precise number of samples
  2015-01-29  8:06 [RFC/PATCHSET 00/42] perf tools: Speed-up perf report by using multi thread (v2) Namhyung Kim
  2015-01-29  8:06 ` [PATCH 01/42] perf tools: Support to read compressed module from build-id cache Namhyung Kim
  2015-01-29  8:06 ` [PATCH 02/42] perf tools: Do not use __perf_session__process_events() directly Namhyung Kim
@ 2015-01-29  8:06 ` Namhyung Kim
  2015-01-30 18:32   ` [tip:perf/core] " tip-bot for Namhyung Kim
  2015-01-29  8:06 ` [PATCH 04/42] perf header: Set header version correctly Namhyung Kim
                   ` (39 subsequent siblings)
  42 siblings, 1 reply; 221+ messages in thread
From: Namhyung Kim @ 2015-01-29  8:06 UTC (permalink / raw)
  To: Arnaldo Carvalho de Melo
  Cc: Ingo Molnar, Peter Zijlstra, Jiri Olsa, LKML, David Ahern,
	Adrian Hunter, Andi Kleen, Stephane Eranian, Frederic Weisbecker,
	Milian Wolff, Jiri Olsa

After perf record finishes, it prints file size and number of samples
in the file but this info is wrong since it assumes typical sample
size of 24 bytes and divides file size by the value.

However as we post-process recorded samples for build-id, it can show
correct number like below.  If build-id post-processing is not requested
just omit the wrong number of samples.

  $ perf record noploop 1
    [ perf record: Woken up 1 times to write data ]
    [ perf record: Captured and wrote 0.159 MB perf.data (3989 samples) ]

  $ perf report --stdio -n
  # To display the perf.data header info, please use --header/--header-only options.
  #
  # Samples: 3K of event 'cycles'
  # Event count (approx.): 3771330663
  #
  # Overhead       Samples  Command  Shared Object     Symbol
  # ........  ............  .......  ................  ..........................
  #
      99.90%          3982  noploop  noploop           [.] main
       0.09%             1  noploop  ld-2.17.so        [.] _dl_check_map_versions
       0.01%             1  noploop  [kernel.vmlinux]  [k] setup_arg_pages
       0.00%             5  noploop  [kernel.vmlinux]  [k] intel_pmu_enable_all

Reported-by: Milian Wolff <mail@milianw.de>
Reviewed-by: Jiri Olsa <jolsa@kernel.org>
Signed-off-by: Namhyung Kim <namhyung@kernel.org>
---
 tools/perf/builtin-record.c | 51 ++++++++++++++++++++++++++++++++++-----------
 1 file changed, 39 insertions(+), 12 deletions(-)

diff --git a/tools/perf/builtin-record.c b/tools/perf/builtin-record.c
index 1134de22979e..9900b433e861 100644
--- a/tools/perf/builtin-record.c
+++ b/tools/perf/builtin-record.c
@@ -190,6 +190,19 @@ static int record__open(struct record *rec)
 	return rc;
 }
 
+static int process_sample_event(struct perf_tool *tool,
+				union perf_event *event,
+				struct perf_sample *sample,
+				struct perf_evsel *evsel,
+				struct machine *machine)
+{
+	struct record *rec = container_of(tool, struct record, tool);
+
+	rec->samples++;
+
+	return build_id__mark_dso_hit(tool, event, sample, evsel, machine);
+}
+
 static int process_buildids(struct record *rec)
 {
 	struct perf_data_file *file  = &rec->file;
@@ -212,7 +225,7 @@ static int process_buildids(struct record *rec)
 	 */
 	symbol_conf.ignore_vmlinux_buildid = true;
 
-	return perf_session__process_events(session, &build_id__mark_dso_hit_ops);
+	return perf_session__process_events(session, &rec->tool);
 }
 
 static void perf_event__synthesize_guest_os(struct machine *machine, void *data)
@@ -503,19 +516,9 @@ static int __cmd_record(struct record *rec, int argc, const char **argv)
 		goto out_child;
 	}
 
-	if (!quiet) {
+	if (!quiet)
 		fprintf(stderr, "[ perf record: Woken up %ld times to write data ]\n", waking);
 
-		/*
-		 * Approximate RIP event size: 24 bytes.
-		 */
-		fprintf(stderr,
-			"[ perf record: Captured and wrote %.3f MB %s (~%" PRIu64 " samples) ]\n",
-			(double)rec->bytes_written / 1024.0 / 1024.0,
-			file->path,
-			rec->bytes_written / 24);
-	}
-
 out_child:
 	if (forks) {
 		int exit_status;
@@ -534,6 +537,9 @@ static int __cmd_record(struct record *rec, int argc, const char **argv)
 	} else
 		status = err;
 
+	/* this will be recalculated during process_buildids() */
+	rec->samples = 0;
+
 	if (!err && !file->is_pipe) {
 		rec->session->header.data_size += rec->bytes_written;
 
@@ -543,6 +549,20 @@ static int __cmd_record(struct record *rec, int argc, const char **argv)
 					   file->fd, true);
 	}
 
+	if (!err && !quiet) {
+		char samples[128];
+
+		if (rec->samples)
+			scnprintf(samples, sizeof(samples),
+				  " (%" PRIu64 " samples)", rec->samples);
+		else
+			samples[0] = '\0';
+
+		fprintf(stderr,	"[ perf record: Captured and wrote %.3f MB %s%s ]\n",
+			perf_data_file__size(file) / 1024.0 / 1024.0,
+			file->path, samples);
+	}
+
 out_delete_session:
 	perf_session__delete(session);
 	return status;
@@ -719,6 +739,13 @@ static struct record record = {
 			.default_per_cpu = true,
 		},
 	},
+	.tool = {
+		.sample		= process_sample_event,
+		.fork		= perf_event__process_fork,
+		.comm		= perf_event__process_comm,
+		.mmap		= perf_event__process_mmap,
+		.mmap2		= perf_event__process_mmap2,
+	},
 };
 
 #define CALLCHAIN_HELP "setup and enables call-graph (stack chain/backtrace) recording: "
-- 
2.2.2


^ permalink raw reply related	[flat|nested] 221+ messages in thread

* [PATCH 04/42] perf header: Set header version correctly
  2015-01-29  8:06 [RFC/PATCHSET 00/42] perf tools: Speed-up perf report by using multi thread (v2) Namhyung Kim
                   ` (2 preceding siblings ...)
  2015-01-29  8:06 ` [PATCH 03/42] perf record: Show precise number of samples Namhyung Kim
@ 2015-01-29  8:06 ` Namhyung Kim
  2015-01-30 18:33   ` [tip:perf/core] " tip-bot for Namhyung Kim
  2015-01-29  8:06 ` [PATCH 05/42] perf tools: Set attr.task bit for a tracking event Namhyung Kim
                   ` (38 subsequent siblings)
  42 siblings, 1 reply; 221+ messages in thread
From: Namhyung Kim @ 2015-01-29  8:06 UTC (permalink / raw)
  To: Arnaldo Carvalho de Melo
  Cc: Ingo Molnar, Peter Zijlstra, Jiri Olsa, LKML, David Ahern,
	Adrian Hunter, Andi Kleen, Stephane Eranian, Frederic Weisbecker

When check_magic_endian() is called, it checks the magic number in the
perf data file to determine version and endianness.  But if it uses a
same endian the verison number wasn't updated and makes confusion.

Signed-off-by: Namhyung Kim <namhyung@kernel.org>
---
 tools/perf/util/header.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/tools/perf/util/header.c b/tools/perf/util/header.c
index b20e40c74468..1f407f7352a7 100644
--- a/tools/perf/util/header.c
+++ b/tools/perf/util/header.c
@@ -2237,6 +2237,7 @@ static int check_magic_endian(u64 magic, uint64_t hdr_sz,
 	 * - unique number to identify actual perf.data files
 	 * - encode endianness of file
 	 */
+	ph->version = PERF_HEADER_VERSION_2;
 
 	/* check magic number with one endianness */
 	if (magic == __perf_magic2)
@@ -2247,7 +2248,6 @@ static int check_magic_endian(u64 magic, uint64_t hdr_sz,
 		return -1;
 
 	ph->needs_swap = true;
-	ph->version = PERF_HEADER_VERSION_2;
 
 	return 0;
 }
-- 
2.2.2


^ permalink raw reply related	[flat|nested] 221+ messages in thread

* [PATCH 05/42] perf tools: Set attr.task bit for a tracking event
  2015-01-29  8:06 [RFC/PATCHSET 00/42] perf tools: Speed-up perf report by using multi thread (v2) Namhyung Kim
                   ` (3 preceding siblings ...)
  2015-01-29  8:06 ` [PATCH 04/42] perf header: Set header version correctly Namhyung Kim
@ 2015-01-29  8:06 ` Namhyung Kim
  2015-01-30 18:33   ` [tip:perf/core] perf evsel: " tip-bot for Namhyung Kim
  2015-01-29  8:06 ` [PATCH 06/42] perf tools: Use a software dummy event to track task/mmap events Namhyung Kim
                   ` (37 subsequent siblings)
  42 siblings, 1 reply; 221+ messages in thread
From: Namhyung Kim @ 2015-01-29  8:06 UTC (permalink / raw)
  To: Arnaldo Carvalho de Melo
  Cc: Ingo Molnar, Peter Zijlstra, Jiri Olsa, LKML, David Ahern,
	Adrian Hunter, Andi Kleen, Stephane Eranian, Frederic Weisbecker

The perf_event_attr.task bit is to track task (fork and exit) events
but it missed to be set by perf_evsel__config().  While it was not a
problem in practice since setting other bits (comm/mmap) ended up
being in same result, it'd be good to set it explicitly anyway.

The attr->task is to track task related events (fork/exit) only but
other meta events like comm and mmap[2] also needs the task events.
So setting attr->comm and/or attr->mmap causes the kernel emits the
task events anyway.  So the attr->task is only meaningful when other
bits are off but I'd like to set it for completeness.

Signed-off-by: Namhyung Kim <namhyung@kernel.org>
---
 tools/perf/util/evsel.c | 1 +
 1 file changed, 1 insertion(+)

diff --git a/tools/perf/util/evsel.c b/tools/perf/util/evsel.c
index 1d826d63bc20..ea51a90e20a0 100644
--- a/tools/perf/util/evsel.c
+++ b/tools/perf/util/evsel.c
@@ -709,6 +709,7 @@ void perf_evsel__config(struct perf_evsel *evsel, struct record_opts *opts)
 	if (opts->sample_weight)
 		perf_evsel__set_sample_bit(evsel, WEIGHT);
 
+	attr->task  = track;
 	attr->mmap  = track;
 	attr->mmap2 = track && !perf_missing_features.mmap2;
 	attr->comm  = track;
-- 
2.2.2


^ permalink raw reply related	[flat|nested] 221+ messages in thread

* [PATCH 06/42] perf tools: Use a software dummy event to track task/mmap events
  2015-01-29  8:06 [RFC/PATCHSET 00/42] perf tools: Speed-up perf report by using multi thread (v2) Namhyung Kim
                   ` (4 preceding siblings ...)
  2015-01-29  8:06 ` [PATCH 05/42] perf tools: Set attr.task bit for a tracking event Namhyung Kim
@ 2015-01-29  8:06 ` Namhyung Kim
  2015-01-29  8:06 ` [PATCH 07/42] perf tools: Use perf_data_file__fd() consistently Namhyung Kim
                   ` (36 subsequent siblings)
  42 siblings, 0 replies; 221+ messages in thread
From: Namhyung Kim @ 2015-01-29  8:06 UTC (permalink / raw)
  To: Arnaldo Carvalho de Melo
  Cc: Ingo Molnar, Peter Zijlstra, Jiri Olsa, LKML, David Ahern,
	Adrian Hunter, Andi Kleen, Stephane Eranian, Frederic Weisbecker

Add APIs for software dummy event to track task/comm/mmap events
separately.  The perf record will use them to save such events in a
separate mmap buffer to make it easy to index.  This is a preparation of
multi-thread support which will come later.

Cc: Adrian Hunter <adrian.hunter@intel.com>
Signed-off-by: Namhyung Kim <namhyung@kernel.org>
---
 tools/perf/util/evlist.c | 30 ++++++++++++++++++++++++++++++
 tools/perf/util/evlist.h |  1 +
 tools/perf/util/evsel.h  | 15 +++++++++++++++
 3 files changed, 46 insertions(+)

diff --git a/tools/perf/util/evlist.c b/tools/perf/util/evlist.c
index 28b8ce86bf12..2d81b4d154f4 100644
--- a/tools/perf/util/evlist.c
+++ b/tools/perf/util/evlist.c
@@ -195,6 +195,36 @@ int perf_evlist__add_default(struct perf_evlist *evlist)
 	return -ENOMEM;
 }
 
+int perf_evlist__add_dummy_tracking(struct perf_evlist *evlist)
+{
+	struct perf_event_attr attr = {
+		.type = PERF_TYPE_SOFTWARE,
+		.config = PERF_COUNT_SW_DUMMY,
+		.exclude_kernel = 1,
+	};
+	struct perf_evsel *evsel;
+
+	event_attr_init(&attr);
+
+	evsel = perf_evsel__new(&attr);
+	if (evsel == NULL)
+		goto error;
+
+	/* use strdup() because free(evsel) assumes name is allocated */
+	evsel->name = strdup("dummy");
+	if (!evsel->name)
+		goto error_free;
+
+	perf_evlist__add(evlist, evsel);
+	perf_evlist__set_tracking_event(evlist, evsel);
+
+	return 0;
+error_free:
+	perf_evsel__delete(evsel);
+error:
+	return -ENOMEM;
+}
+
 static int perf_evlist__add_attrs(struct perf_evlist *evlist,
 				  struct perf_event_attr *attrs, size_t nr_attrs)
 {
diff --git a/tools/perf/util/evlist.h b/tools/perf/util/evlist.h
index c94a9e03ecf1..771175e70d2f 100644
--- a/tools/perf/util/evlist.h
+++ b/tools/perf/util/evlist.h
@@ -67,6 +67,7 @@ void perf_evlist__delete(struct perf_evlist *evlist);
 
 void perf_evlist__add(struct perf_evlist *evlist, struct perf_evsel *entry);
 int perf_evlist__add_default(struct perf_evlist *evlist);
+int perf_evlist__add_dummy_tracking(struct perf_evlist *evlist);
 int __perf_evlist__add_default_attrs(struct perf_evlist *evlist,
 				     struct perf_event_attr *attrs, size_t nr_attrs);
 
diff --git a/tools/perf/util/evsel.h b/tools/perf/util/evsel.h
index 38622747d130..9764e9456546 100644
--- a/tools/perf/util/evsel.h
+++ b/tools/perf/util/evsel.h
@@ -331,6 +331,21 @@ static inline bool perf_evsel__is_function_event(struct perf_evsel *evsel)
 #undef FUNCTION_EVENT
 }
 
+/**
+ * perf_evsel__is_dummy_tracking - Return whether given evsel is a dummy
+ * event for tracking meta events only
+ *
+ * @evsel - evsel selector to be tested
+ *
+ * Return %true if event is a dummy tracking event
+ */
+static inline bool perf_evsel__is_dummy_tracking(struct perf_evsel *evsel)
+{
+	return evsel->attr.type == PERF_TYPE_SOFTWARE &&
+		evsel->attr.config == PERF_COUNT_SW_DUMMY &&
+		evsel->attr.task == 1 && evsel->attr.mmap == 1;
+}
+
 struct perf_attr_details {
 	bool freq;
 	bool verbose;
-- 
2.2.2


^ permalink raw reply related	[flat|nested] 221+ messages in thread

* [PATCH 07/42] perf tools: Use perf_data_file__fd() consistently
  2015-01-29  8:06 [RFC/PATCHSET 00/42] perf tools: Speed-up perf report by using multi thread (v2) Namhyung Kim
                   ` (5 preceding siblings ...)
  2015-01-29  8:06 ` [PATCH 06/42] perf tools: Use a software dummy event to track task/mmap events Namhyung Kim
@ 2015-01-29  8:06 ` Namhyung Kim
  2015-01-30 18:33   ` [tip:perf/core] " tip-bot for Namhyung Kim
  2015-01-29  8:06 ` [PATCH 08/42] perf tools: Add rm_rf() utility function Namhyung Kim
                   ` (35 subsequent siblings)
  42 siblings, 1 reply; 221+ messages in thread
From: Namhyung Kim @ 2015-01-29  8:06 UTC (permalink / raw)
  To: Arnaldo Carvalho de Melo
  Cc: Ingo Molnar, Peter Zijlstra, Jiri Olsa, LKML, David Ahern,
	Adrian Hunter, Andi Kleen, Stephane Eranian, Frederic Weisbecker

Do not reference file->fd directly since we want hide the
implementation details from outside for possible future changes.

Signed-off-by: Namhyung Kim <namhyung@kernel.org>
---
 tools/perf/builtin-inject.c |  5 +++--
 tools/perf/builtin-record.c | 14 +++++++-------
 2 files changed, 10 insertions(+), 9 deletions(-)

diff --git a/tools/perf/builtin-inject.c b/tools/perf/builtin-inject.c
index 84df2deed988..a13641e066f5 100644
--- a/tools/perf/builtin-inject.c
+++ b/tools/perf/builtin-inject.c
@@ -343,6 +343,7 @@ static int __cmd_inject(struct perf_inject *inject)
 	int ret = -EINVAL;
 	struct perf_session *session = inject->session;
 	struct perf_data_file *file_out = &inject->output;
+	int fd = perf_data_file__fd(file_out);
 
 	signal(SIGINT, sig_handler);
 
@@ -376,7 +377,7 @@ static int __cmd_inject(struct perf_inject *inject)
 	}
 
 	if (!file_out->is_pipe)
-		lseek(file_out->fd, session->header.data_offset, SEEK_SET);
+		lseek(fd, session->header.data_offset, SEEK_SET);
 
 	ret = perf_session__process_events(session, &inject->tool);
 
@@ -385,7 +386,7 @@ static int __cmd_inject(struct perf_inject *inject)
 			perf_header__set_feat(&session->header,
 					      HEADER_BUILD_ID);
 		session->header.data_size = inject->bytes_written;
-		perf_session__write_header(session, session->evlist, file_out->fd, true);
+		perf_session__write_header(session, session->evlist, fd, true);
 	}
 
 	return ret;
diff --git a/tools/perf/builtin-record.c b/tools/perf/builtin-record.c
index 9900b433e861..404ab3434052 100644
--- a/tools/perf/builtin-record.c
+++ b/tools/perf/builtin-record.c
@@ -208,7 +208,7 @@ static int process_buildids(struct record *rec)
 	struct perf_data_file *file  = &rec->file;
 	struct perf_session *session = rec->session;
 
-	u64 size = lseek(file->fd, 0, SEEK_CUR);
+	u64 size = lseek(perf_data_file__fd(file), 0, SEEK_CUR);
 	if (size == 0)
 		return 0;
 
@@ -334,6 +334,7 @@ static int __cmd_record(struct record *rec, int argc, const char **argv)
 	struct perf_data_file *file = &rec->file;
 	struct perf_session *session;
 	bool disabled = false, draining = false;
+	int fd;
 
 	rec->progname = argv[0];
 
@@ -348,6 +349,7 @@ static int __cmd_record(struct record *rec, int argc, const char **argv)
 		return -1;
 	}
 
+	fd = perf_data_file__fd(file);
 	rec->session = session;
 
 	record__init_features(rec);
@@ -372,12 +374,11 @@ static int __cmd_record(struct record *rec, int argc, const char **argv)
 		perf_header__clear_feat(&session->header, HEADER_GROUP_DESC);
 
 	if (file->is_pipe) {
-		err = perf_header__write_pipe(file->fd);
+		err = perf_header__write_pipe(fd);
 		if (err < 0)
 			goto out_child;
 	} else {
-		err = perf_session__write_header(session, rec->evlist,
-						 file->fd, false);
+		err = perf_session__write_header(session, rec->evlist, fd, false);
 		if (err < 0)
 			goto out_child;
 	}
@@ -409,7 +410,7 @@ static int __cmd_record(struct record *rec, int argc, const char **argv)
 			 * return this more properly and also
 			 * propagate errors that now are calling die()
 			 */
-			err = perf_event__synthesize_tracing_data(tool, file->fd, rec->evlist,
+			err = perf_event__synthesize_tracing_data(tool,	fd, rec->evlist,
 								  process_synthesized_event);
 			if (err <= 0) {
 				pr_err("Couldn't record tracing data.\n");
@@ -545,8 +546,7 @@ static int __cmd_record(struct record *rec, int argc, const char **argv)
 
 		if (!rec->no_buildid)
 			process_buildids(rec);
-		perf_session__write_header(rec->session, rec->evlist,
-					   file->fd, true);
+		perf_session__write_header(rec->session, rec->evlist, fd, true);
 	}
 
 	if (!err && !quiet) {
-- 
2.2.2


^ permalink raw reply related	[flat|nested] 221+ messages in thread

* [PATCH 08/42] perf tools: Add rm_rf() utility function
  2015-01-29  8:06 [RFC/PATCHSET 00/42] perf tools: Speed-up perf report by using multi thread (v2) Namhyung Kim
                   ` (6 preceding siblings ...)
  2015-01-29  8:06 ` [PATCH 07/42] perf tools: Use perf_data_file__fd() consistently Namhyung Kim
@ 2015-01-29  8:06 ` Namhyung Kim
  2015-01-30 15:02   ` Jiri Olsa
  2015-01-29  8:06 ` [PATCH 09/42] perf tools: Introduce copyfile_offset() function Namhyung Kim
                   ` (34 subsequent siblings)
  42 siblings, 1 reply; 221+ messages in thread
From: Namhyung Kim @ 2015-01-29  8:06 UTC (permalink / raw)
  To: Arnaldo Carvalho de Melo
  Cc: Ingo Molnar, Peter Zijlstra, Jiri Olsa, LKML, David Ahern,
	Adrian Hunter, Andi Kleen, Stephane Eranian, Frederic Weisbecker

The rm_rf() function does same as the shell command 'rm -rf' which
removes all directory entries recursively.

Signed-off-by: Namhyung Kim <namhyung@kernel.org>
---
 tools/perf/util/util.c | 43 +++++++++++++++++++++++++++++++++++++++++++
 tools/perf/util/util.h |  1 +
 2 files changed, 44 insertions(+)

diff --git a/tools/perf/util/util.c b/tools/perf/util/util.c
index b86744f29eef..de1307a6ff4b 100644
--- a/tools/perf/util/util.c
+++ b/tools/perf/util/util.c
@@ -72,6 +72,49 @@ int mkdir_p(char *path, mode_t mode)
 	return (stat(path, &st) && mkdir(path, mode)) ? -1 : 0;
 }
 
+int rm_rf(char *path)
+{
+	DIR *dir;
+	int ret = 0;
+	struct dirent *d;
+	char namebuf[PATH_MAX];
+
+	dir = opendir(path);
+	if (dir == NULL)
+		return 0;
+
+	while ((d = readdir(dir)) != NULL && !ret) {
+		struct stat statbuf;
+
+		if (!strcmp(d->d_name, ".") || !strcmp(d->d_name, ".."))
+			continue;
+
+		scnprintf(namebuf, sizeof(namebuf), "%s/%s",
+			  path, d->d_name);
+
+		ret = stat(namebuf, &statbuf);
+		if (ret < 0) {
+			pr_debug("stat failed: %s\n", namebuf);
+			break;
+		}
+
+		if (S_ISREG(statbuf.st_mode))
+			ret = unlink(namebuf);
+		else if (S_ISDIR(statbuf.st_mode))
+			ret = rm_rf(namebuf);
+		else {
+			pr_debug("unknown file: %s\n", namebuf);
+			ret = -1;
+		}
+	}
+	closedir(dir);
+
+	if (ret < 0)
+		return ret;
+
+	return rmdir(path);
+}
+
 static int slow_copyfile(const char *from, const char *to, mode_t mode)
 {
 	int err = -1;
diff --git a/tools/perf/util/util.h b/tools/perf/util/util.h
index 027a5153495c..7b71ad87cdc3 100644
--- a/tools/perf/util/util.h
+++ b/tools/perf/util/util.h
@@ -248,6 +248,7 @@ static inline int sane_case(int x, int high)
 }
 
 int mkdir_p(char *path, mode_t mode);
+int rm_rf(char *path);
 int copyfile(const char *from, const char *to);
 int copyfile_mode(const char *from, const char *to, mode_t mode);
 
-- 
2.2.2


^ permalink raw reply related	[flat|nested] 221+ messages in thread

* [PATCH 09/42] perf tools: Introduce copyfile_offset() function
  2015-01-29  8:06 [RFC/PATCHSET 00/42] perf tools: Speed-up perf report by using multi thread (v2) Namhyung Kim
                   ` (7 preceding siblings ...)
  2015-01-29  8:06 ` [PATCH 08/42] perf tools: Add rm_rf() utility function Namhyung Kim
@ 2015-01-29  8:06 ` Namhyung Kim
  2015-01-29  8:06 ` [PATCH 10/42] perf tools: Create separate mmap for dummy tracking event Namhyung Kim
                   ` (33 subsequent siblings)
  42 siblings, 0 replies; 221+ messages in thread
From: Namhyung Kim @ 2015-01-29  8:06 UTC (permalink / raw)
  To: Arnaldo Carvalho de Melo
  Cc: Ingo Molnar, Peter Zijlstra, Jiri Olsa, LKML, David Ahern,
	Adrian Hunter, Andi Kleen, Stephane Eranian, Frederic Weisbecker

The copyfile_offset() function is to copy source data from given
offset to a destination file with an offset.  It'll be used to build
an indexed data file.

Signed-off-by: Namhyung Kim <namhyung@kernel.org>
---
 tools/perf/util/util.c | 38 +++++++++++++++++++++++++++++---------
 tools/perf/util/util.h |  1 +
 2 files changed, 30 insertions(+), 9 deletions(-)

diff --git a/tools/perf/util/util.c b/tools/perf/util/util.c
index de1307a6ff4b..cc1aa309d643 100644
--- a/tools/perf/util/util.c
+++ b/tools/perf/util/util.c
@@ -145,11 +145,38 @@ static int slow_copyfile(const char *from, const char *to, mode_t mode)
 	return err;
 }
 
+int copyfile_offset(int ifd, loff_t off_in, int ofd, loff_t off_out, u64 size)
+{
+	void *ptr;
+	loff_t pgoff;
+
+	pgoff = off_in & ~(page_size - 1);
+	off_in -= pgoff;
+
+	ptr = mmap(NULL, off_in + size, PROT_READ, MAP_PRIVATE, ifd, pgoff);
+	if (ptr == MAP_FAILED)
+		return -1;
+
+	while (size) {
+		ssize_t ret = pwrite(ofd, ptr + off_in, size, off_out);
+		if (ret < 0 && errno == EINTR)
+			continue;
+		if (ret <= 0)
+			break;
+
+		size -= ret;
+		off_in += ret;
+		off_out -= ret;
+	}
+	munmap(ptr, off_in + size);
+
+	return size ? -1 : 0;
+}
+
 int copyfile_mode(const char *from, const char *to, mode_t mode)
 {
 	int fromfd, tofd;
 	struct stat st;
-	void *addr;
 	int err = -1;
 
 	if (stat(from, &st))
@@ -166,15 +193,8 @@ int copyfile_mode(const char *from, const char *to, mode_t mode)
 	if (tofd < 0)
 		goto out_close_from;
 
-	addr = mmap(NULL, st.st_size, PROT_READ, MAP_PRIVATE, fromfd, 0);
-	if (addr == MAP_FAILED)
-		goto out_close_to;
-
-	if (write(tofd, addr, st.st_size) == st.st_size)
-		err = 0;
+	err = copyfile_offset(fromfd, 0, tofd, 0, st.st_size);
 
-	munmap(addr, st.st_size);
-out_close_to:
 	close(tofd);
 	if (err)
 		unlink(to);
diff --git a/tools/perf/util/util.h b/tools/perf/util/util.h
index 7b71ad87cdc3..2291f08f11fe 100644
--- a/tools/perf/util/util.h
+++ b/tools/perf/util/util.h
@@ -251,6 +251,7 @@ int mkdir_p(char *path, mode_t mode);
 int rm_rf(char *path);
 int copyfile(const char *from, const char *to);
 int copyfile_mode(const char *from, const char *to, mode_t mode);
+int copyfile_offset(int fromfd, loff_t from_ofs, int tofd, loff_t to_ofs, u64 size);
 
 s64 perf_atoll(const char *str);
 char **argv_split(const char *str, int *argcp);
-- 
2.2.2


^ permalink raw reply related	[flat|nested] 221+ messages in thread

* [PATCH 10/42] perf tools: Create separate mmap for dummy tracking event
  2015-01-29  8:06 [RFC/PATCHSET 00/42] perf tools: Speed-up perf report by using multi thread (v2) Namhyung Kim
                   ` (8 preceding siblings ...)
  2015-01-29  8:06 ` [PATCH 09/42] perf tools: Introduce copyfile_offset() function Namhyung Kim
@ 2015-01-29  8:06 ` Namhyung Kim
  2015-01-29  8:06 ` [PATCH 11/42] perf tools: Introduce perf_evlist__mmap_track() Namhyung Kim
                   ` (32 subsequent siblings)
  42 siblings, 0 replies; 221+ messages in thread
From: Namhyung Kim @ 2015-01-29  8:06 UTC (permalink / raw)
  To: Arnaldo Carvalho de Melo
  Cc: Ingo Molnar, Peter Zijlstra, Jiri Olsa, LKML, David Ahern,
	Adrian Hunter, Andi Kleen, Stephane Eranian, Frederic Weisbecker

When indexed data file support is enabled, a dummy tracking event will
be used to track metadata (like task, comm and mmap events) for a
session and actual samples will be recorded in separate (intermediate)
files and then merged (with index table).

Provide separate mmap to the dummy tracking event.  The size is fixed
to 128KiB (+ 1 page) as the event rate will be lower than samples.  I
originally wanted to use a single mmap for this but cross-cpu sharing
is prohibited so it's per-cpu (or per-task) like normal mmaps.

Cc: Adrian Hunter <adrian.hunter@intel.com>
Signed-off-by: Namhyung Kim <namhyung@kernel.org>
---
 tools/perf/builtin-record.c |   9 +++-
 tools/perf/util/evlist.c    | 122 +++++++++++++++++++++++++++++++++++---------
 tools/perf/util/evlist.h    |  11 +++-
 3 files changed, 117 insertions(+), 25 deletions(-)

diff --git a/tools/perf/builtin-record.c b/tools/perf/builtin-record.c
index 404ab3434052..adb3eefb51ed 100644
--- a/tools/perf/builtin-record.c
+++ b/tools/perf/builtin-record.c
@@ -69,7 +69,7 @@ static int process_synthesized_event(struct perf_tool *tool,
 
 static int record__mmap_read(struct record *rec, int idx)
 {
-	struct perf_mmap *md = &rec->evlist->mmap[idx];
+	struct perf_mmap *md = perf_evlist__mmap_desc(rec->evlist, idx);
 	unsigned int head = perf_mmap__read_head(md);
 	unsigned int old = md->prev;
 	unsigned char *data = md->base + page_size;
@@ -105,6 +105,7 @@ static int record__mmap_read(struct record *rec, int idx)
 	}
 
 	md->prev = old;
+
 	perf_evlist__mmap_consume(rec->evlist, idx);
 out:
 	return rc;
@@ -275,6 +276,12 @@ static int record__mmap_read_all(struct record *rec)
 				goto out;
 			}
 		}
+		if (rec->evlist->track_mmap) {
+			if (record__mmap_read(rec, track_mmap_idx(i)) != 0) {
+				rc = -1;
+				goto out;
+			}
+		}
 	}
 
 	/*
diff --git a/tools/perf/util/evlist.c b/tools/perf/util/evlist.c
index 2d81b4d154f4..ac31edecffaf 100644
--- a/tools/perf/util/evlist.c
+++ b/tools/perf/util/evlist.c
@@ -29,6 +29,7 @@
 
 static void perf_evlist__mmap_put(struct perf_evlist *evlist, int idx);
 static void __perf_evlist__munmap(struct perf_evlist *evlist, int idx);
+static void __perf_evlist__munmap_track(struct perf_evlist *evlist, int idx);
 
 #define FD(e, x, y) (*(int *)xyarray__entry(e->fd, x, y))
 #define SID(e, x, y) xyarray__entry(e->sample_id, x, y)
@@ -729,22 +730,39 @@ static bool perf_mmap__empty(struct perf_mmap *md)
 	return perf_mmap__read_head(md) != md->prev;
 }
 
+struct perf_mmap *perf_evlist__mmap_desc(struct perf_evlist *evlist, int idx)
+{
+	if (idx >= 0)
+		return &evlist->mmap[idx];
+	else
+		return &evlist->track_mmap[track_mmap_idx(idx)];
+}
+
 static void perf_evlist__mmap_get(struct perf_evlist *evlist, int idx)
 {
-	++evlist->mmap[idx].refcnt;
+	struct perf_mmap *md = perf_evlist__mmap_desc(evlist, idx);
+
+	++md->refcnt;
 }
 
 static void perf_evlist__mmap_put(struct perf_evlist *evlist, int idx)
 {
-	BUG_ON(evlist->mmap[idx].refcnt == 0);
+	struct perf_mmap *md = perf_evlist__mmap_desc(evlist, idx);
+
+	BUG_ON(md->refcnt == 0);
+
+	if (--md->refcnt != 0)
+		return;
 
-	if (--evlist->mmap[idx].refcnt == 0)
+	if (idx >= 0)
 		__perf_evlist__munmap(evlist, idx);
+	else
+		__perf_evlist__munmap_track(evlist, track_mmap_idx(idx));
 }
 
 void perf_evlist__mmap_consume(struct perf_evlist *evlist, int idx)
 {
-	struct perf_mmap *md = &evlist->mmap[idx];
+	struct perf_mmap *md = perf_evlist__mmap_desc(evlist, idx);
 
 	if (!evlist->overwrite) {
 		unsigned int old = md->prev;
@@ -765,6 +783,15 @@ static void __perf_evlist__munmap(struct perf_evlist *evlist, int idx)
 	}
 }
 
+static void __perf_evlist__munmap_track(struct perf_evlist *evlist, int idx)
+{
+	if (evlist->track_mmap[idx].base != NULL) {
+		munmap(evlist->track_mmap[idx].base, TRACK_MMAP_SIZE);
+		evlist->track_mmap[idx].base = NULL;
+		evlist->track_mmap[idx].refcnt = 0;
+	}
+}
+
 void perf_evlist__munmap(struct perf_evlist *evlist)
 {
 	int i;
@@ -776,23 +803,43 @@ void perf_evlist__munmap(struct perf_evlist *evlist)
 		__perf_evlist__munmap(evlist, i);
 
 	zfree(&evlist->mmap);
+
+	if (evlist->track_mmap == NULL)
+		return;
+
+	for (i = 0; i < evlist->nr_mmaps; i++)
+		__perf_evlist__munmap_track(evlist, i);
+
+	zfree(&evlist->track_mmap);
 }
 
-static int perf_evlist__alloc_mmap(struct perf_evlist *evlist)
+static int perf_evlist__alloc_mmap(struct perf_evlist *evlist, bool track_mmap)
 {
 	evlist->nr_mmaps = cpu_map__nr(evlist->cpus);
 	if (cpu_map__empty(evlist->cpus))
 		evlist->nr_mmaps = thread_map__nr(evlist->threads);
 	evlist->mmap = zalloc(evlist->nr_mmaps * sizeof(struct perf_mmap));
-	return evlist->mmap != NULL ? 0 : -ENOMEM;
+	if (evlist->mmap == NULL)
+		return -ENOMEM;
+
+	if (track_mmap) {
+		evlist->track_mmap = calloc(evlist->nr_mmaps,
+					    sizeof(struct perf_mmap));
+		if (evlist->track_mmap == NULL) {
+			zfree(&evlist->mmap);
+			return -ENOMEM;
+		}
+	}
+	return 0;
 }
 
 struct mmap_params {
-	int prot;
-	int mask;
+	int	prot;
+	size_t	len;
 };
 
-static int __perf_evlist__mmap(struct perf_evlist *evlist, int idx,
+static int __perf_evlist__mmap(struct perf_evlist *evlist __maybe_unused,
+			       struct perf_mmap *pmmap,
 			       struct mmap_params *mp, int fd)
 {
 	/*
@@ -808,15 +855,14 @@ static int __perf_evlist__mmap(struct perf_evlist *evlist, int idx,
 	 * evlist layer can't just drop it when filtering events in
 	 * perf_evlist__filter_pollfd().
 	 */
-	evlist->mmap[idx].refcnt = 2;
-	evlist->mmap[idx].prev = 0;
-	evlist->mmap[idx].mask = mp->mask;
-	evlist->mmap[idx].base = mmap(NULL, evlist->mmap_len, mp->prot,
-				      MAP_SHARED, fd, 0);
-	if (evlist->mmap[idx].base == MAP_FAILED) {
+	pmmap->refcnt = 2;
+	pmmap->prev = 0;
+	pmmap->mask = mp->len - page_size - 1;
+	pmmap->base = mmap(NULL, mp->len, mp->prot, MAP_SHARED, fd, 0);
+	if (pmmap->base == MAP_FAILED) {
 		pr_debug2("failed to mmap perf event ring buffer, error %d\n",
 			  errno);
-		evlist->mmap[idx].base = NULL;
+		pmmap->base = NULL;
 		return -1;
 	}
 
@@ -825,7 +871,8 @@ static int __perf_evlist__mmap(struct perf_evlist *evlist, int idx,
 
 static int perf_evlist__mmap_per_evsel(struct perf_evlist *evlist, int idx,
 				       struct mmap_params *mp, int cpu,
-				       int thread, int *output)
+				       int thread, int *output,
+				       int *track_output)
 {
 	struct perf_evsel *evsel;
 
@@ -837,9 +884,30 @@ static int perf_evlist__mmap_per_evsel(struct perf_evlist *evlist, int idx,
 
 		fd = FD(evsel, cpu, thread);
 
-		if (*output == -1) {
+		if (perf_evsel__is_dummy_tracking(evsel)) {
+			struct mmap_params track_mp = {
+				.prot	= mp->prot,
+				.len	= TRACK_MMAP_SIZE,
+			};
+
+			if (*track_output == -1) {
+				*track_output = fd;
+				if (__perf_evlist__mmap(evlist,
+							&evlist->track_mmap[idx],
+							&track_mp, fd) < 0)
+					return -1;
+			} else {
+				if (ioctl(fd, PERF_EVENT_IOC_SET_OUTPUT,
+					  *track_output) != 0)
+					return -1;
+			}
+
+			/* mark idx as track mmap idx (negative) */
+			idx = track_mmap_idx(idx);
+		} else if (*output == -1) {
 			*output = fd;
-			if (__perf_evlist__mmap(evlist, idx, mp, *output) < 0)
+			if (__perf_evlist__mmap(evlist, &evlist->mmap[idx],
+						mp, *output) < 0)
 				return -1;
 		} else {
 			if (ioctl(fd, PERF_EVENT_IOC_SET_OUTPUT, *output) != 0)
@@ -868,6 +936,11 @@ static int perf_evlist__mmap_per_evsel(struct perf_evlist *evlist, int idx,
 			perf_evlist__set_sid_idx(evlist, evsel, idx, cpu,
 						 thread);
 		}
+
+		if (perf_evsel__is_dummy_tracking(evsel)) {
+			/* restore idx as normal idx (positive) */
+			idx = track_mmap_idx(idx);
+		}
 	}
 
 	return 0;
@@ -883,10 +956,12 @@ static int perf_evlist__mmap_per_cpu(struct perf_evlist *evlist,
 	pr_debug2("perf event ring buffer mmapped per cpu\n");
 	for (cpu = 0; cpu < nr_cpus; cpu++) {
 		int output = -1;
+		int track_output = -1;
 
 		for (thread = 0; thread < nr_threads; thread++) {
 			if (perf_evlist__mmap_per_evsel(evlist, cpu, mp, cpu,
-							thread, &output))
+							thread, &output,
+							&track_output))
 				goto out_unmap;
 		}
 	}
@@ -908,9 +983,10 @@ static int perf_evlist__mmap_per_thread(struct perf_evlist *evlist,
 	pr_debug2("perf event ring buffer mmapped per thread\n");
 	for (thread = 0; thread < nr_threads; thread++) {
 		int output = -1;
+		int track_output = -1;
 
 		if (perf_evlist__mmap_per_evsel(evlist, thread, mp, 0, thread,
-						&output))
+						&output, &track_output))
 			goto out_unmap;
 	}
 
@@ -1033,7 +1109,7 @@ int perf_evlist__mmap(struct perf_evlist *evlist, unsigned int pages,
 		.prot = PROT_READ | (overwrite ? 0 : PROT_WRITE),
 	};
 
-	if (evlist->mmap == NULL && perf_evlist__alloc_mmap(evlist) < 0)
+	if (evlist->mmap == NULL && perf_evlist__alloc_mmap(evlist, true) < 0)
 		return -ENOMEM;
 
 	if (evlist->pollfd.entries == NULL && perf_evlist__alloc_pollfd(evlist) < 0)
@@ -1042,7 +1118,7 @@ int perf_evlist__mmap(struct perf_evlist *evlist, unsigned int pages,
 	evlist->overwrite = overwrite;
 	evlist->mmap_len = perf_evlist__mmap_size(pages);
 	pr_debug("mmap size %zuB\n", evlist->mmap_len);
-	mp.mask = evlist->mmap_len - page_size - 1;
+	mp.len = evlist->mmap_len;
 
 	evlist__for_each(evlist, evsel) {
 		if ((evsel->attr.read_format & PERF_FORMAT_ID) &&
diff --git a/tools/perf/util/evlist.h b/tools/perf/util/evlist.h
index 771175e70d2f..bf697632458d 100644
--- a/tools/perf/util/evlist.h
+++ b/tools/perf/util/evlist.h
@@ -48,11 +48,14 @@ struct perf_evlist {
 	bool		 overwrite;
 	struct fdarray	 pollfd;
 	struct perf_mmap *mmap;
+	struct perf_mmap *track_mmap;
 	struct thread_map *threads;
 	struct cpu_map	  *cpus;
 	struct perf_evsel *selected;
 };
 
+#define TRACK_MMAP_SIZE  (((128 * 1024 / page_size) + 1) * page_size)
+
 struct perf_evsel_str_handler {
 	const char *name;
 	void	   *handler;
@@ -100,8 +103,8 @@ struct perf_evsel *perf_evlist__id2evsel(struct perf_evlist *evlist, u64 id);
 struct perf_sample_id *perf_evlist__id2sid(struct perf_evlist *evlist, u64 id);
 
 union perf_event *perf_evlist__mmap_read(struct perf_evlist *evlist, int idx);
-
 void perf_evlist__mmap_consume(struct perf_evlist *evlist, int idx);
+struct perf_mmap *perf_evlist__mmap_desc(struct perf_evlist *evlist, int idx);
 
 int perf_evlist__open(struct perf_evlist *evlist);
 void perf_evlist__close(struct perf_evlist *evlist);
@@ -211,6 +214,12 @@ bool perf_evlist__can_select_event(struct perf_evlist *evlist, const char *str);
 void perf_evlist__to_front(struct perf_evlist *evlist,
 			   struct perf_evsel *move_evsel);
 
+/* convert from/to negative idx for track mmaps */
+static inline int track_mmap_idx(int idx)
+{
+	return -idx - 1;
+}
+
 /**
  * __evlist__for_each - iterate thru all the evsels
  * @list: list_head instance to iterate
-- 
2.2.2


^ permalink raw reply related	[flat|nested] 221+ messages in thread

* [PATCH 11/42] perf tools: Introduce perf_evlist__mmap_track()
  2015-01-29  8:06 [RFC/PATCHSET 00/42] perf tools: Speed-up perf report by using multi thread (v2) Namhyung Kim
                   ` (9 preceding siblings ...)
  2015-01-29  8:06 ` [PATCH 10/42] perf tools: Create separate mmap for dummy tracking event Namhyung Kim
@ 2015-01-29  8:06 ` Namhyung Kim
  2015-01-29  8:06 ` [PATCH 12/42] perf tools: Add HEADER_DATA_INDEX feature Namhyung Kim
                   ` (31 subsequent siblings)
  42 siblings, 0 replies; 221+ messages in thread
From: Namhyung Kim @ 2015-01-29  8:06 UTC (permalink / raw)
  To: Arnaldo Carvalho de Melo
  Cc: Ingo Molnar, Peter Zijlstra, Jiri Olsa, LKML, David Ahern,
	Adrian Hunter, Andi Kleen, Stephane Eranian, Frederic Weisbecker

The perf_evlist__mmap_track function creates data mmaps and optionally
tracking mmaps for events.  It'll be used for perf record to save events
in a separate files and build an index table.  Checking dummy tracking
event in perf_evlist__mmap() alone is not enough as users can specify a
dummy event (like in keep tracking testcase) without the index option.

Cc: Adrian Hunter <adrian.hunter@intel.com>
Signed-off-by: Namhyung Kim <namhyung@kernel.org>
---
 tools/perf/builtin-record.c |  3 ++-
 tools/perf/util/evlist.c    | 15 +++++++++------
 tools/perf/util/evlist.h    | 10 ++++++++--
 3 files changed, 19 insertions(+), 9 deletions(-)

diff --git a/tools/perf/builtin-record.c b/tools/perf/builtin-record.c
index adb3eefb51ed..56118d8cf74a 100644
--- a/tools/perf/builtin-record.c
+++ b/tools/perf/builtin-record.c
@@ -169,7 +169,8 @@ static int record__open(struct record *rec)
 		goto out;
 	}
 
-	if (perf_evlist__mmap(evlist, opts->mmap_pages, false) < 0) {
+	if (perf_evlist__mmap_track(evlist, opts->mmap_pages, false,
+				    false) < 0) {
 		if (errno == EPERM) {
 			pr_err("Permission error mapping pages.\n"
 			       "Consider increasing "
diff --git a/tools/perf/util/evlist.c b/tools/perf/util/evlist.c
index ac31edecffaf..fb27b468427d 100644
--- a/tools/perf/util/evlist.c
+++ b/tools/perf/util/evlist.c
@@ -835,6 +835,7 @@ static int perf_evlist__alloc_mmap(struct perf_evlist *evlist, bool track_mmap)
 
 struct mmap_params {
 	int	prot;
+	bool	track;
 	size_t	len;
 };
 
@@ -884,7 +885,7 @@ static int perf_evlist__mmap_per_evsel(struct perf_evlist *evlist, int idx,
 
 		fd = FD(evsel, cpu, thread);
 
-		if (perf_evsel__is_dummy_tracking(evsel)) {
+		if (mp->track && perf_evsel__is_dummy_tracking(evsel)) {
 			struct mmap_params track_mp = {
 				.prot	= mp->prot,
 				.len	= TRACK_MMAP_SIZE,
@@ -937,7 +938,7 @@ static int perf_evlist__mmap_per_evsel(struct perf_evlist *evlist, int idx,
 						 thread);
 		}
 
-		if (perf_evsel__is_dummy_tracking(evsel)) {
+		if (mp->track && perf_evsel__is_dummy_tracking(evsel)) {
 			/* restore idx as normal idx (positive) */
 			idx = track_mmap_idx(idx);
 		}
@@ -1088,10 +1089,11 @@ int perf_evlist__parse_mmap_pages(const struct option *opt, const char *str,
 }
 
 /**
- * perf_evlist__mmap - Create mmaps to receive events.
+ * perf_evlist__mmap_track - Create mmaps to receive events.
  * @evlist: list of events
  * @pages: map length in pages
  * @overwrite: overwrite older events?
+ * @use_track_mmap: use another mmaps to track meta events
  *
  * If @overwrite is %false the user needs to signal event consumption using
  * perf_mmap__write_tail().  Using perf_evlist__mmap_read() does this
@@ -1099,17 +1101,18 @@ int perf_evlist__parse_mmap_pages(const struct option *opt, const char *str,
  *
  * Return: %0 on success, negative error code otherwise.
  */
-int perf_evlist__mmap(struct perf_evlist *evlist, unsigned int pages,
-		      bool overwrite)
+int perf_evlist__mmap_track(struct perf_evlist *evlist, unsigned int pages,
+			    bool overwrite, bool use_track_mmap)
 {
 	struct perf_evsel *evsel;
 	const struct cpu_map *cpus = evlist->cpus;
 	const struct thread_map *threads = evlist->threads;
 	struct mmap_params mp = {
 		.prot = PROT_READ | (overwrite ? 0 : PROT_WRITE),
+		.track = use_track_mmap,
 	};
 
-	if (evlist->mmap == NULL && perf_evlist__alloc_mmap(evlist, true) < 0)
+	if (evlist->mmap == NULL && perf_evlist__alloc_mmap(evlist, mp.track) < 0)
 		return -ENOMEM;
 
 	if (evlist->pollfd.entries == NULL && perf_evlist__alloc_pollfd(evlist) < 0)
diff --git a/tools/perf/util/evlist.h b/tools/perf/util/evlist.h
index bf697632458d..c3b7b3428cbb 100644
--- a/tools/perf/util/evlist.h
+++ b/tools/perf/util/evlist.h
@@ -127,10 +127,16 @@ int perf_evlist__parse_mmap_pages(const struct option *opt,
 				  const char *str,
 				  int unset);
 
-int perf_evlist__mmap(struct perf_evlist *evlist, unsigned int pages,
-		      bool overwrite);
+int perf_evlist__mmap_track(struct perf_evlist *evlist, unsigned int pages,
+			    bool overwrite, bool use_track_mmap);
 void perf_evlist__munmap(struct perf_evlist *evlist);
 
+static inline int perf_evlist__mmap(struct perf_evlist *evlist,
+				    unsigned int pages, bool overwrite)
+{
+	return perf_evlist__mmap_track(evlist, pages, overwrite, false);
+}
+
 void perf_evlist__disable(struct perf_evlist *evlist);
 void perf_evlist__enable(struct perf_evlist *evlist);
 
-- 
2.2.2


^ permalink raw reply related	[flat|nested] 221+ messages in thread

* [PATCH 12/42] perf tools: Add HEADER_DATA_INDEX feature
  2015-01-29  8:06 [RFC/PATCHSET 00/42] perf tools: Speed-up perf report by using multi thread (v2) Namhyung Kim
                   ` (10 preceding siblings ...)
  2015-01-29  8:06 ` [PATCH 11/42] perf tools: Introduce perf_evlist__mmap_track() Namhyung Kim
@ 2015-01-29  8:06 ` Namhyung Kim
  2015-01-29  8:06 ` [PATCH 13/42] perf tools: Handle indexed data file properly Namhyung Kim
                   ` (30 subsequent siblings)
  42 siblings, 0 replies; 221+ messages in thread
From: Namhyung Kim @ 2015-01-29  8:06 UTC (permalink / raw)
  To: Arnaldo Carvalho de Melo
  Cc: Ingo Molnar, Peter Zijlstra, Jiri Olsa, LKML, David Ahern,
	Adrian Hunter, Andi Kleen, Stephane Eranian, Frederic Weisbecker

The HEADER_DATA_INDEX feature is to record index table for sample data
so that they can be processed by multiple thread concurrently.  Each
item is a struct perf_file_section which consists of an offset and size.

Signed-off-by: Namhyung Kim <namhyung@kernel.org>
---
 tools/perf/builtin-data.c   |  0
 tools/perf/builtin-record.c |  2 ++
 tools/perf/util/header.c    | 61 +++++++++++++++++++++++++++++++++++++++++++++
 tools/perf/util/header.h    |  3 +++
 4 files changed, 66 insertions(+)
 create mode 100644 tools/perf/builtin-data.c

diff --git a/tools/perf/builtin-data.c b/tools/perf/builtin-data.c
new file mode 100644
index 000000000000..e69de29bb2d1
diff --git a/tools/perf/builtin-record.c b/tools/perf/builtin-record.c
index 56118d8cf74a..b057e2caa5f1 100644
--- a/tools/perf/builtin-record.c
+++ b/tools/perf/builtin-record.c
@@ -312,6 +312,8 @@ static void record__init_features(struct record *rec)
 
 	if (!rec->opts.branch_stack)
 		perf_header__clear_feat(&session->header, HEADER_BRANCH_STACK);
+
+	perf_header__clear_feat(&session->header, HEADER_DATA_INDEX);
 }
 
 static volatile int workload_exec_errno;
diff --git a/tools/perf/util/header.c b/tools/perf/util/header.c
index 1f407f7352a7..77206a6cbf65 100644
--- a/tools/perf/util/header.c
+++ b/tools/perf/util/header.c
@@ -869,6 +869,24 @@ static int write_branch_stack(int fd __maybe_unused,
 	return 0;
 }
 
+static int write_data_index(int fd, struct perf_header *h,
+			    struct perf_evlist *evlist __maybe_unused)
+{
+	int ret;
+	unsigned i;
+
+	ret = do_write(fd, &h->nr_index, sizeof(h->nr_index));
+	if (ret < 0)
+		return ret;
+
+	for (i = 0; i < h->nr_index; i++) {
+		ret = do_write(fd, &h->index[i], sizeof(*h->index));
+		if (ret < 0)
+			return ret;
+	}
+	return 0;
+}
+
 static void print_hostname(struct perf_header *ph, int fd __maybe_unused,
 			   FILE *fp)
 {
@@ -1225,6 +1243,12 @@ static void print_group_desc(struct perf_header *ph, int fd __maybe_unused,
 	}
 }
 
+static void print_data_index(struct perf_header *ph __maybe_unused,
+			     int fd __maybe_unused, FILE *fp)
+{
+	fprintf(fp, "# contains data index for parallel processing\n");
+}
+
 static int __event_process_build_id(struct build_id_event *bev,
 				    char *filename,
 				    struct perf_session *session)
@@ -1833,6 +1857,42 @@ static int process_group_desc(struct perf_file_section *section __maybe_unused,
 	return ret;
 }
 
+static int process_data_index(struct perf_file_section *section __maybe_unused,
+			      struct perf_header *ph, int fd,
+			      void *data __maybe_unused)
+{
+	ssize_t ret;
+	u64 nr_index;
+	unsigned i;
+	struct perf_file_section *index;
+
+	ret = readn(fd, &nr_index, sizeof(nr_index));
+	if (ret != sizeof(nr_index))
+		return -1;
+
+	if (ph->needs_swap)
+		nr_index = bswap_64(nr_index);
+
+	index = calloc(nr_index, sizeof(*index));
+	if (index == NULL)
+		return -1;
+
+	for (i = 0; i < nr_index; i++) {
+		ret = readn(fd, &index[i], sizeof(*index));
+		if (ret != sizeof(*index))
+			return ret;
+
+		if (ph->needs_swap) {
+			index[i].offset = bswap_64(index[i].offset);
+			index[i].size   = bswap_64(index[i].size);
+		}
+	}
+
+	ph->index = index;
+	ph->nr_index = nr_index;
+	return 0;
+}
+
 struct feature_ops {
 	int (*write)(int fd, struct perf_header *h, struct perf_evlist *evlist);
 	void (*print)(struct perf_header *h, int fd, FILE *fp);
@@ -1873,6 +1933,7 @@ static const struct feature_ops feat_ops[HEADER_LAST_FEATURE] = {
 	FEAT_OPA(HEADER_BRANCH_STACK,	branch_stack),
 	FEAT_OPP(HEADER_PMU_MAPPINGS,	pmu_mappings),
 	FEAT_OPP(HEADER_GROUP_DESC,	group_desc),
+	FEAT_OPP(HEADER_DATA_INDEX,	data_index),
 };
 
 struct header_print_data {
diff --git a/tools/perf/util/header.h b/tools/perf/util/header.h
index 3bb90ac172a1..e5594f0d6dcd 100644
--- a/tools/perf/util/header.h
+++ b/tools/perf/util/header.h
@@ -30,6 +30,7 @@ enum {
 	HEADER_BRANCH_STACK,
 	HEADER_PMU_MAPPINGS,
 	HEADER_GROUP_DESC,
+	HEADER_DATA_INDEX,
 	HEADER_LAST_FEATURE,
 	HEADER_FEAT_BITS	= 256,
 };
@@ -94,6 +95,8 @@ struct perf_header {
 	bool				needs_swap;
 	u64				data_offset;
 	u64				data_size;
+	struct perf_file_section	*index;
+	u64				nr_index;
 	u64				feat_offset;
 	DECLARE_BITMAP(adds_features, HEADER_FEAT_BITS);
 	struct perf_session_env 	env;
-- 
2.2.2


^ permalink raw reply related	[flat|nested] 221+ messages in thread

* [PATCH 13/42] perf tools: Handle indexed data file properly
  2015-01-29  8:06 [RFC/PATCHSET 00/42] perf tools: Speed-up perf report by using multi thread (v2) Namhyung Kim
                   ` (11 preceding siblings ...)
  2015-01-29  8:06 ` [PATCH 12/42] perf tools: Add HEADER_DATA_INDEX feature Namhyung Kim
@ 2015-01-29  8:06 ` Namhyung Kim
  2015-01-29  8:06 ` [PATCH 14/42] perf record: Add --index option for building index table Namhyung Kim
                   ` (29 subsequent siblings)
  42 siblings, 0 replies; 221+ messages in thread
From: Namhyung Kim @ 2015-01-29  8:06 UTC (permalink / raw)
  To: Arnaldo Carvalho de Melo
  Cc: Ingo Molnar, Peter Zijlstra, Jiri Olsa, LKML, David Ahern,
	Adrian Hunter, Andi Kleen, Stephane Eranian, Frederic Weisbecker

When perf detects data file has index table, process header file first
and then rest data files in a row.  Note that the indexed data is
recorded for each cpu/thread separately, it's already ordered with
respect to themselves so no need to use the ordered event queue
interface.

Signed-off-by: Namhyung Kim <namhyung@kernel.org>
---
 tools/perf/util/session.c | 47 ++++++++++++++++++++++++++++++++++++-----------
 tools/perf/util/session.h |  5 +++++
 2 files changed, 41 insertions(+), 11 deletions(-)

diff --git a/tools/perf/util/session.c b/tools/perf/util/session.c
index 0baf75f12b7c..ff4d5913220c 100644
--- a/tools/perf/util/session.c
+++ b/tools/perf/util/session.c
@@ -1251,11 +1251,10 @@ fetch_mmaped_event(struct perf_session *session,
 #define NUM_MMAPS 128
 #endif
 
-static int __perf_session__process_events(struct perf_session *session,
+static int __perf_session__process_events(struct perf_session *session, int fd,
 					  u64 data_offset, u64 data_size,
 					  u64 file_size, struct perf_tool *tool)
 {
-	int fd = perf_data_file__fd(session->file);
 	u64 head, page_offset, file_offset, file_pos, size;
 	int err, mmap_prot, mmap_flags, map_idx = 0;
 	size_t	mmap_size;
@@ -1278,7 +1277,9 @@ static int __perf_session__process_events(struct perf_session *session,
 	mmap_size = MMAP_SIZE;
 	if (mmap_size > file_size) {
 		mmap_size = file_size;
-		session->one_mmap = true;
+
+		if (!perf_session__has_index(session))
+			session->one_mmap = true;
 	}
 
 	memset(mmaps, 0, sizeof(mmaps));
@@ -1360,19 +1361,43 @@ static int __perf_session__process_events(struct perf_session *session,
 int perf_session__process_events(struct perf_session *session,
 				 struct perf_tool *tool)
 {
-	u64 size = perf_data_file__size(session->file);
-	int err;
+	struct perf_data_file *file = session->file;
+	u64 size = perf_data_file__size(file);
+	int err, i;
 
 	if (perf_session__register_idle_thread(session) == NULL)
 		return -ENOMEM;
 
-	if (!perf_data_file__is_pipe(session->file))
+	if (perf_data_file__is_pipe(file))
+		return __perf_session__process_pipe_events(session, tool);
+
+	err = __perf_session__process_events(session,
+					     perf_data_file__fd(file),
+					     session->header.data_offset,
+					     session->header.data_size,
+					     size, tool);
+
+	if (err < 0 || !perf_session__has_index(session))
+		return err;
+
+	/*
+	 * For indexed data file, events are processed for
+	 * each cpu/thread so it's already ordered.
+	 */
+	tool->ordered_events = false;
+
+	for (i = 0; i < (int)session->header.nr_index; i++) {
+		if (!session->header.index[i].size)
+			continue;
+
 		err = __perf_session__process_events(session,
-						     session->header.data_offset,
-						     session->header.data_size,
-						     size, tool);
-	else
-		err = __perf_session__process_pipe_events(session, tool);
+						perf_data_file__fd(file),
+						session->header.index[i].offset,
+						session->header.index[i].size,
+						size, tool);
+		if (err < 0)
+			break;
+	}
 
 	return err;
 }
diff --git a/tools/perf/util/session.h b/tools/perf/util/session.h
index 6d663dc76404..419976d74b51 100644
--- a/tools/perf/util/session.h
+++ b/tools/perf/util/session.h
@@ -138,4 +138,9 @@ int perf_event__synthesize_id_index(struct perf_tool *tool,
 				    struct perf_evlist *evlist,
 				    struct machine *machine);
 
+static inline bool perf_session__has_index(struct perf_session *session)
+{
+	return perf_header__has_feat(&session->header, HEADER_DATA_INDEX);
+}
+
 #endif /* __PERF_SESSION_H */
-- 
2.2.2


^ permalink raw reply related	[flat|nested] 221+ messages in thread

* [PATCH 14/42] perf record: Add --index option for building index table
  2015-01-29  8:06 [RFC/PATCHSET 00/42] perf tools: Speed-up perf report by using multi thread (v2) Namhyung Kim
                   ` (12 preceding siblings ...)
  2015-01-29  8:06 ` [PATCH 13/42] perf tools: Handle indexed data file properly Namhyung Kim
@ 2015-01-29  8:06 ` Namhyung Kim
  2015-02-01 18:06   ` Jiri Olsa
  2015-01-29  8:06 ` [PATCH 15/42] perf report: Skip dummy tracking event Namhyung Kim
                   ` (28 subsequent siblings)
  42 siblings, 1 reply; 221+ messages in thread
From: Namhyung Kim @ 2015-01-29  8:06 UTC (permalink / raw)
  To: Arnaldo Carvalho de Melo
  Cc: Ingo Molnar, Peter Zijlstra, Jiri Olsa, LKML, David Ahern,
	Adrian Hunter, Andi Kleen, Stephane Eranian, Frederic Weisbecker

The new --index option will create indexed data file which can be
processed by multiple threads parallelly.  It saves meta event and
sample data in separate files and merges them with an index table.

To build an index table, it needs to know exact offsets and sizes for
each sample data.  However the offset only can be calculated after the
feature data is fixed, and to save feature data it needs to access to
the sample data because it needs to mark used DSOs for build-id table.

So I ended up with reserving 1MB hole for the feature data area and then
put sample data and calculated offsets.  Now an indexed perf data file
will look like below:

        +---------------------+
        |     file header     |
        |---------------------|
        |                     |
        |     meta events     |
        |                     |
        |---------------------|
        |     feature data    |
        |   (contains index) -+--+
        |---------------------|  |
        |      ~1MB hole      |  |
        |---------------------|  |
        |                     |  |
        |    sample data[1] <-+--+
        |                     |  |
        |---------------------|  |
        |                     |  |
        |    sample data[2] <-|--+
        |                     |  |
        |---------------------|  |
        |         ...         | ...
        +---------------------+

Signed-off-by: Namhyung Kim <namhyung@kernel.org>
---
 tools/perf/Documentation/perf-record.txt |   4 +
 tools/perf/builtin-data.c                |   0
 tools/perf/builtin-record.c              | 165 +++++++++++++++++++++++++++++--
 tools/perf/perf.h                        |   1 +
 tools/perf/util/session.c                |   1 +
 5 files changed, 161 insertions(+), 10 deletions(-)
 delete mode 100644 tools/perf/builtin-data.c

diff --git a/tools/perf/Documentation/perf-record.txt b/tools/perf/Documentation/perf-record.txt
index 31e977459c51..1fe8736cc0ff 100644
--- a/tools/perf/Documentation/perf-record.txt
+++ b/tools/perf/Documentation/perf-record.txt
@@ -235,6 +235,10 @@ Capture machine state (registers) at interrupt, i.e., on counter overflows for
 each sample. List of captured registers depends on the architecture. This option
 is off by default.
 
+--index::
+Build an index table for sample data.  This will speed up perf report by
+parallel processing.
+
 SEE ALSO
 --------
 linkperf:perf-stat[1], linkperf:perf-list[1]
diff --git a/tools/perf/builtin-data.c b/tools/perf/builtin-data.c
deleted file mode 100644
index e69de29bb2d1..000000000000
diff --git a/tools/perf/builtin-record.c b/tools/perf/builtin-record.c
index b057e2caa5f1..0db47c97446b 100644
--- a/tools/perf/builtin-record.c
+++ b/tools/perf/builtin-record.c
@@ -38,6 +38,7 @@ struct record {
 	struct record_opts	opts;
 	u64			bytes_written;
 	struct perf_data_file	file;
+	int			*fds;
 	struct perf_evlist	*evlist;
 	struct perf_session	*session;
 	const char		*progname;
@@ -47,14 +48,23 @@ struct record {
 	long			samples;
 };
 
-static int record__write(struct record *rec, void *bf, size_t size)
+static int record__write(struct record *rec, void *bf, size_t size, int idx)
 {
-	if (perf_data_file__write(rec->session->file, bf, size) < 0) {
+	int fd;
+
+	if (rec->fds && idx >= 0) {
+		fd = rec->fds[idx];
+		/* do not update data size for index files */
+	} else {
+		fd = perf_data_file__fd(rec->session->file);
+		rec->bytes_written += size;
+	}
+
+	if (writen(fd, bf, size) < 0) {
 		pr_err("failed to write perf data, error: %m\n");
 		return -1;
 	}
 
-	rec->bytes_written += size;
 	return 0;
 }
 
@@ -64,7 +74,7 @@ static int process_synthesized_event(struct perf_tool *tool,
 				     struct machine *machine __maybe_unused)
 {
 	struct record *rec = container_of(tool, struct record, tool);
-	return record__write(rec, event, event->header.size);
+	return record__write(rec, event, event->header.size, -1);
 }
 
 static int record__mmap_read(struct record *rec, int idx)
@@ -89,7 +99,7 @@ static int record__mmap_read(struct record *rec, int idx)
 		size = md->mask + 1 - (old & md->mask);
 		old += size;
 
-		if (record__write(rec, buf, size) < 0) {
+		if (record__write(rec, buf, size, idx) < 0) {
 			rc = -1;
 			goto out;
 		}
@@ -99,7 +109,7 @@ static int record__mmap_read(struct record *rec, int idx)
 	size = head - old;
 	old += size;
 
-	if (record__write(rec, buf, size) < 0) {
+	if (record__write(rec, buf, size, idx) < 0) {
 		rc = -1;
 		goto out;
 	}
@@ -111,6 +121,113 @@ static int record__mmap_read(struct record *rec, int idx)
 	return rc;
 }
 
+#define INDEX_FILE_FMT  "%s.dir/perf.data.%d"
+
+static int record__create_index_files(struct record *rec, int nr_index)
+{
+	int i = 0;
+	int ret = -1;
+	char path[PATH_MAX];
+	struct perf_data_file *file = &rec->file;
+
+	rec->fds = malloc(nr_index * sizeof(int));
+	if (rec->fds == NULL)
+		return -ENOMEM;
+
+	scnprintf(path, sizeof(path), "%s.dir", file->path);
+	if (mkdir(path, S_IRWXU) < 0)
+		goto out_err;
+
+	for (i = 0; i < nr_index; i++) {
+		scnprintf(path, sizeof(path), INDEX_FILE_FMT, file->path, i);
+		ret = open(path, O_RDWR|O_CREAT|O_TRUNC, S_IRUSR|S_IWUSR);
+		if (ret < 0)
+			goto out_err;
+
+		rec->fds[i] = ret;
+	}
+	return 0;
+
+out_err:
+	while (--i >= 0)
+		close(rec->fds[i]);
+	zfree(&rec->fds);
+
+	scnprintf(path, sizeof(path), "%s.dir", file->path);
+	rm_rf(path);
+
+	return ret;
+}
+
+static int record__merge_index_files(struct record *rec, int nr_index)
+{
+	int i;
+	int ret = -1;
+	u64 offset;
+	char path[PATH_MAX];
+	struct perf_file_section *idx;
+	struct perf_data_file *file = &rec->file;
+	struct perf_session *session = rec->session;
+	int output_fd = perf_data_file__fd(file);
+
+	idx = calloc(nr_index, sizeof(*idx));
+	if (idx == NULL)
+		goto out_close;
+
+	/* index data will be placed after header */
+	offset = lseek(output_fd, 0, SEEK_END);
+	if (offset == (u64)(loff_t) -1)
+		goto out_close;
+
+	/*
+	 * increase the offset for header features (including index).
+	 * which set later.  we cannot know exact size at this stage,
+	 * but I guess 1MB should be enough..
+	 */
+	offset += 1024 * 1024;
+	offset = PERF_ALIGN(offset, page_size);
+
+	for (i = 0; i < nr_index; i++) {
+		struct stat stbuf;
+		int fd = rec->fds[i];
+
+		if (fstat(fd, &stbuf) < 0)
+			goto out_close;
+
+		idx[i].offset = offset;
+		idx[i].size   = stbuf.st_size;
+
+		offset += PERF_ALIGN(stbuf.st_size, page_size);
+	}
+
+	session->header.index = idx;
+	session->header.nr_index = nr_index;
+
+	/* copy sample events */
+	for (i = 0; i < nr_index; i++) {
+		int fd = rec->fds[i];
+
+		if (idx[i].size == 0)
+			continue;
+
+		if (copyfile_offset(fd, 0, output_fd, idx[i].offset,
+				    idx[i].size) < 0)
+			goto out_close;
+	}
+
+	ret = 0;
+
+out_close:
+	for (i = 0; i < nr_index; i++)
+		close(rec->fds[i]);
+
+	scnprintf(path, sizeof(path), "%s.dir", file->path);
+	rm_rf(path);
+
+	zfree(&rec->fds);
+	return ret;
+}
+
 static volatile int done = 0;
 static volatile int signr = -1;
 static volatile int child_finished = 0;
@@ -170,7 +287,7 @@ static int record__open(struct record *rec)
 	}
 
 	if (perf_evlist__mmap_track(evlist, opts->mmap_pages, false,
-				    false) < 0) {
+				    opts->index) < 0) {
 		if (errno == EPERM) {
 			pr_err("Permission error mapping pages.\n"
 			       "Consider increasing "
@@ -186,6 +303,12 @@ static int record__open(struct record *rec)
 		goto out;
 	}
 
+	if (opts->index) {
+		rc = record__create_index_files(rec, evlist->nr_mmaps);
+		if (rc < 0)
+			goto out;
+	}
+
 	session->evlist = evlist;
 	perf_session__set_id_hdr_size(session);
 out:
@@ -210,7 +333,8 @@ static int process_buildids(struct record *rec)
 	struct perf_data_file *file  = &rec->file;
 	struct perf_session *session = rec->session;
 
-	u64 size = lseek(perf_data_file__fd(file), 0, SEEK_CUR);
+	/* update file size after merging sample files with index */
+	u64 size = lseek(perf_data_file__fd(file), 0, SEEK_END);
 	if (size == 0)
 		return 0;
 
@@ -290,7 +414,8 @@ static int record__mmap_read_all(struct record *rec)
 	 * at least one event.
 	 */
 	if (bytes_written != rec->bytes_written)
-		rc = record__write(rec, &finished_round_event, sizeof(finished_round_event));
+		rc = record__write(rec, &finished_round_event,
+				   sizeof(finished_round_event), -1);
 
 out:
 	return rc;
@@ -313,7 +438,8 @@ static void record__init_features(struct record *rec)
 	if (!rec->opts.branch_stack)
 		perf_header__clear_feat(&session->header, HEADER_BRANCH_STACK);
 
-	perf_header__clear_feat(&session->header, HEADER_DATA_INDEX);
+	if (!rec->opts.index)
+		perf_header__clear_feat(&session->header, HEADER_DATA_INDEX);
 }
 
 static volatile int workload_exec_errno;
@@ -375,6 +501,11 @@ static int __cmd_record(struct record *rec, int argc, const char **argv)
 		}
 	}
 
+	if (file->is_pipe && opts->index) {
+		pr_warning("Indexing is disabled for pipe output\n");
+		opts->index = false;
+	}
+
 	if (record__open(rec) != 0) {
 		err = -1;
 		goto out_child;
@@ -554,6 +685,9 @@ static int __cmd_record(struct record *rec, int argc, const char **argv)
 	if (!err && !file->is_pipe) {
 		rec->session->header.data_size += rec->bytes_written;
 
+		if (rec->opts.index)
+			record__merge_index_files(rec, rec->evlist->nr_mmaps);
+
 		if (!rec->no_buildid)
 			process_buildids(rec);
 		perf_session__write_header(rec->session, rec->evlist, fd, true);
@@ -849,6 +983,8 @@ struct option __record_options[] = {
 		    "use per-thread mmaps"),
 	OPT_BOOLEAN('I', "intr-regs", &record.opts.sample_intr_regs,
 		    "Sample machine registers on interrupt"),
+	OPT_BOOLEAN(0, "index", &record.opts.index,
+		    "make index for sample data to speed-up processing"),
 	OPT_END()
 };
 
@@ -898,6 +1034,15 @@ int cmd_record(int argc, const char **argv, const char *prefix __maybe_unused)
 		goto out_symbol_exit;
 	}
 
+	if (rec->opts.index) {
+		if (!rec->opts.sample_time) {
+			pr_err("Sample timestamp is required for indexing\n");
+			goto out_symbol_exit;
+		}
+
+		perf_evlist__add_dummy_tracking(rec->evlist);
+	}
+
 	if (rec->opts.target.tid && !rec->opts.no_inherit_set)
 		rec->opts.no_inherit = true;
 
diff --git a/tools/perf/perf.h b/tools/perf/perf.h
index 1dabb8553499..b0fad99c9252 100644
--- a/tools/perf/perf.h
+++ b/tools/perf/perf.h
@@ -53,6 +53,7 @@ struct record_opts {
 	bool	     sample_time;
 	bool	     period;
 	bool	     sample_intr_regs;
+	bool	     index;
 	unsigned int freq;
 	unsigned int mmap_pages;
 	unsigned int user_freq;
diff --git a/tools/perf/util/session.c b/tools/perf/util/session.c
index ff4d5913220c..e7b59fbebbc4 100644
--- a/tools/perf/util/session.c
+++ b/tools/perf/util/session.c
@@ -173,6 +173,7 @@ void perf_session__delete(struct perf_session *session)
 	machines__exit(&session->machines);
 	if (session->file)
 		perf_data_file__close(session->file);
+	free(session->header.index);
 	free(session);
 }
 
-- 
2.2.2


^ permalink raw reply related	[flat|nested] 221+ messages in thread

* [PATCH 15/42] perf report: Skip dummy tracking event
  2015-01-29  8:06 [RFC/PATCHSET 00/42] perf tools: Speed-up perf report by using multi thread (v2) Namhyung Kim
                   ` (13 preceding siblings ...)
  2015-01-29  8:06 ` [PATCH 14/42] perf record: Add --index option for building index table Namhyung Kim
@ 2015-01-29  8:06 ` Namhyung Kim
  2015-01-29  8:06 ` [PATCH 16/42] perf tools: Pass session arg to perf_event__preprocess_sample() Namhyung Kim
                   ` (27 subsequent siblings)
  42 siblings, 0 replies; 221+ messages in thread
From: Namhyung Kim @ 2015-01-29  8:06 UTC (permalink / raw)
  To: Arnaldo Carvalho de Melo
  Cc: Ingo Molnar, Peter Zijlstra, Jiri Olsa, LKML, David Ahern,
	Adrian Hunter, Andi Kleen, Stephane Eranian, Frederic Weisbecker

The dummy tracking event is only for tracking task/comom/mmap events
and has no sample data for itself.  So no need to report, just skip it.

Signed-off-by: Namhyung Kim <namhyung@kernel.org>
---
 tools/perf/builtin-report.c    |  3 +++
 tools/perf/ui/browsers/hists.c | 30 ++++++++++++++++++++++++------
 tools/perf/ui/gtk/hists.c      |  3 +++
 3 files changed, 30 insertions(+), 6 deletions(-)

diff --git a/tools/perf/builtin-report.c b/tools/perf/builtin-report.c
index 2f91094e228b..4cac79ad3085 100644
--- a/tools/perf/builtin-report.c
+++ b/tools/perf/builtin-report.c
@@ -318,6 +318,9 @@ static int perf_evlist__tty_browse_hists(struct perf_evlist *evlist,
 		struct hists *hists = evsel__hists(pos);
 		const char *evname = perf_evsel__name(pos);
 
+		if (perf_evsel__is_dummy_tracking(pos))
+			continue;
+
 		if (symbol_conf.event_group &&
 		    !perf_evsel__is_group_leader(pos))
 			continue;
diff --git a/tools/perf/ui/browsers/hists.c b/tools/perf/ui/browsers/hists.c
index 788506eef567..7d33d7dc0824 100644
--- a/tools/perf/ui/browsers/hists.c
+++ b/tools/perf/ui/browsers/hists.c
@@ -1947,14 +1947,17 @@ static int perf_evsel_menu__run(struct perf_evsel_menu *menu,
 	return key;
 }
 
-static bool filter_group_entries(struct ui_browser *browser __maybe_unused,
-				 void *entry)
+static bool filter_entries(struct ui_browser *browser __maybe_unused,
+			   void *entry)
 {
 	struct perf_evsel *evsel = list_entry(entry, struct perf_evsel, node);
 
 	if (symbol_conf.event_group && !perf_evsel__is_group_leader(evsel))
 		return true;
 
+	if (perf_evsel__is_dummy_tracking(evsel))
+		return true;
+
 	return false;
 }
 
@@ -1971,7 +1974,7 @@ static int __perf_evlist__tui_browse_hists(struct perf_evlist *evlist,
 			.refresh    = ui_browser__list_head_refresh,
 			.seek	    = ui_browser__list_head_seek,
 			.write	    = perf_evsel_menu__write,
-			.filter	    = filter_group_entries,
+			.filter	    = filter_entries,
 			.nr_entries = nr_entries,
 			.priv	    = evlist,
 		},
@@ -1998,21 +2001,22 @@ int perf_evlist__tui_browse_hists(struct perf_evlist *evlist, const char *help,
 				  struct perf_session_env *env)
 {
 	int nr_entries = evlist->nr_entries;
+	struct perf_evsel *first = perf_evlist__first(evlist);
+	struct perf_evsel *pos;
 
 single_entry:
 	if (nr_entries == 1) {
-		struct perf_evsel *first = perf_evlist__first(evlist);
-
 		return perf_evsel__hists_browse(first, nr_entries, help,
 						false, hbt, min_pcnt,
 						env);
 	}
 
 	if (symbol_conf.event_group) {
-		struct perf_evsel *pos;
 
 		nr_entries = 0;
 		evlist__for_each(evlist, pos) {
+			if (perf_evsel__is_dummy_tracking(pos))
+				continue;
 			if (perf_evsel__is_group_leader(pos))
 				nr_entries++;
 		}
@@ -2021,6 +2025,20 @@ int perf_evlist__tui_browse_hists(struct perf_evlist *evlist, const char *help,
 			goto single_entry;
 	}
 
+	evlist__for_each(evlist, pos) {
+		if (perf_evsel__is_dummy_tracking(pos))
+			nr_entries--;
+	}
+
+	if (nr_entries == 1) {
+		evlist__for_each(evlist, pos) {
+			if (!perf_evsel__is_dummy_tracking(pos)) {
+				first = pos;
+				goto single_entry;
+			}
+		}
+	}
+
 	return __perf_evlist__tui_browse_hists(evlist, nr_entries, help,
 					       hbt, min_pcnt, env);
 }
diff --git a/tools/perf/ui/gtk/hists.c b/tools/perf/ui/gtk/hists.c
index 4b3585eed1e8..83a7ecd5cda8 100644
--- a/tools/perf/ui/gtk/hists.c
+++ b/tools/perf/ui/gtk/hists.c
@@ -317,6 +317,9 @@ int perf_evlist__gtk_browse_hists(struct perf_evlist *evlist,
 		char buf[512];
 		size_t size = sizeof(buf);
 
+		if (perf_evsel__is_dummy_tracking(pos))
+			continue;
+
 		if (symbol_conf.event_group) {
 			if (!perf_evsel__is_group_leader(pos))
 				continue;
-- 
2.2.2


^ permalink raw reply related	[flat|nested] 221+ messages in thread

* [PATCH 16/42] perf tools: Pass session arg to perf_event__preprocess_sample()
  2015-01-29  8:06 [RFC/PATCHSET 00/42] perf tools: Speed-up perf report by using multi thread (v2) Namhyung Kim
                   ` (14 preceding siblings ...)
  2015-01-29  8:06 ` [PATCH 15/42] perf report: Skip dummy tracking event Namhyung Kim
@ 2015-01-29  8:06 ` Namhyung Kim
  2015-01-29  8:06 ` [PATCH 17/42] perf script: Pass session arg to ->process_event callback Namhyung Kim
                   ` (26 subsequent siblings)
  42 siblings, 0 replies; 221+ messages in thread
From: Namhyung Kim @ 2015-01-29  8:06 UTC (permalink / raw)
  To: Arnaldo Carvalho de Melo
  Cc: Ingo Molnar, Peter Zijlstra, Jiri Olsa, LKML, David Ahern,
	Adrian Hunter, Andi Kleen, Stephane Eranian, Frederic Weisbecker

The perf_event__preprocess_sample() translates a given ip into a
matching symbol.  To do that, it first finds a corresponding thread
and map in the current thread tree.  But for indexed data files, it
needs to find a thread (and map) with slightly different APIs using
timestamp.  So it needs a way to know whether this session deals with
an indexed data file or not.

Signed-off-by: Namhyung Kim <namhyung@kernel.org>
---
 tools/perf/builtin-annotate.c     |  3 ++-
 tools/perf/builtin-diff.c         | 13 +++++++++----
 tools/perf/builtin-mem.c          |  6 +++++-
 tools/perf/builtin-report.c       |  3 ++-
 tools/perf/builtin-script.c       | 20 +++++++++++---------
 tools/perf/builtin-timechart.c    | 10 +++++++---
 tools/perf/builtin-top.c          |  3 ++-
 tools/perf/tests/hists_cumulate.c |  2 +-
 tools/perf/tests/hists_filter.c   |  2 +-
 tools/perf/tests/hists_link.c     |  4 ++--
 tools/perf/tests/hists_output.c   |  2 +-
 tools/perf/util/event.c           |  3 ++-
 tools/perf/util/event.h           |  4 +++-
 13 files changed, 48 insertions(+), 27 deletions(-)

diff --git a/tools/perf/builtin-annotate.c b/tools/perf/builtin-annotate.c
index 747f86103599..b89e4c6ed488 100644
--- a/tools/perf/builtin-annotate.c
+++ b/tools/perf/builtin-annotate.c
@@ -85,7 +85,8 @@ static int process_sample_event(struct perf_tool *tool,
 	struct perf_annotate *ann = container_of(tool, struct perf_annotate, tool);
 	struct addr_location al;
 
-	if (perf_event__preprocess_sample(event, machine, &al, sample) < 0) {
+	if (perf_event__preprocess_sample(event, machine, &al, sample,
+					  ann->session) < 0) {
 		pr_warning("problem processing %d event, skipping it.\n",
 			   event->header.type);
 		return -1;
diff --git a/tools/perf/builtin-diff.c b/tools/perf/builtin-diff.c
index 74aada554b12..3e2229227062 100644
--- a/tools/perf/builtin-diff.c
+++ b/tools/perf/builtin-diff.c
@@ -42,6 +42,7 @@ struct diff_hpp_fmt {
 };
 
 struct data__file {
+	struct perf_tool	tool;
 	struct perf_session	*session;
 	struct perf_data_file	file;
 	int			 idx;
@@ -320,16 +321,18 @@ static int hists__add_entry(struct hists *hists,
 	return -ENOMEM;
 }
 
-static int diff__process_sample_event(struct perf_tool *tool __maybe_unused,
+static int diff__process_sample_event(struct perf_tool *tool,
 				      union perf_event *event,
 				      struct perf_sample *sample,
 				      struct perf_evsel *evsel,
 				      struct machine *machine)
 {
 	struct addr_location al;
+	struct data__file *d = container_of(tool, struct data__file, tool);
 	struct hists *hists = evsel__hists(evsel);
 
-	if (perf_event__preprocess_sample(event, machine, &al, sample) < 0) {
+	if (perf_event__preprocess_sample(event, machine, &al, sample,
+					  d->session) < 0) {
 		pr_warning("problem processing %d event, skipping it.\n",
 			   event->header.type);
 		return -1;
@@ -740,14 +743,16 @@ static int __cmd_diff(void)
 	int ret = -EINVAL, i;
 
 	data__for_each_file(i, d) {
-		d->session = perf_session__new(&d->file, false, &tool);
+		memcpy(&d->tool, &tool, sizeof(tool));
+
+		d->session = perf_session__new(&d->file, false, &d->tool);
 		if (!d->session) {
 			pr_err("Failed to open %s\n", d->file.path);
 			ret = -1;
 			goto out_delete;
 		}
 
-		ret = perf_session__process_events(d->session, &tool);
+		ret = perf_session__process_events(d->session, &d->tool);
 		if (ret) {
 			pr_err("Failed to process %s\n", d->file.path);
 			goto out_delete;
diff --git a/tools/perf/builtin-mem.c b/tools/perf/builtin-mem.c
index 9b5663950a4d..21d46918860e 100644
--- a/tools/perf/builtin-mem.c
+++ b/tools/perf/builtin-mem.c
@@ -12,6 +12,7 @@
 
 struct perf_mem {
 	struct perf_tool	tool;
+	struct perf_session	*session;
 	char const		*input_name;
 	bool			hide_unresolved;
 	bool			dump_raw;
@@ -66,7 +67,8 @@ dump_raw_samples(struct perf_tool *tool,
 	struct addr_location al;
 	const char *fmt;
 
-	if (perf_event__preprocess_sample(event, machine, &al, sample) < 0) {
+	if (perf_event__preprocess_sample(event, machine, &al, sample,
+					  mem->session) < 0) {
 		fprintf(stderr, "problem processing %d event, skipping it.\n",
 				event->header.type);
 		return -1;
@@ -129,6 +131,8 @@ static int report_raw_events(struct perf_mem *mem)
 	if (session == NULL)
 		return -1;
 
+	mem->session = session;
+
 	if (mem->cpu_list) {
 		ret = perf_session__cpu_bitmap(session, mem->cpu_list,
 					       mem->cpu_bitmap);
diff --git a/tools/perf/builtin-report.c b/tools/perf/builtin-report.c
index 4cac79ad3085..68d06bc02266 100644
--- a/tools/perf/builtin-report.c
+++ b/tools/perf/builtin-report.c
@@ -142,7 +142,8 @@ static int process_sample_event(struct perf_tool *tool,
 	};
 	int ret;
 
-	if (perf_event__preprocess_sample(event, machine, &al, sample) < 0) {
+	if (perf_event__preprocess_sample(event, machine, &al, sample,
+					  rep->session) < 0) {
 		pr_debug("problem processing %d event, skipping it.\n",
 			 event->header.type);
 		return -1;
diff --git a/tools/perf/builtin-script.c b/tools/perf/builtin-script.c
index ce304dfd962a..ab920f8cded6 100644
--- a/tools/perf/builtin-script.c
+++ b/tools/perf/builtin-script.c
@@ -542,13 +542,21 @@ static int cleanup_scripting(void)
 	return scripting_ops->stop_script();
 }
 
-static int process_sample_event(struct perf_tool *tool __maybe_unused,
+struct perf_script {
+	struct perf_tool	tool;
+	struct perf_session	*session;
+	bool			show_task_events;
+	bool			show_mmap_events;
+};
+
+static int process_sample_event(struct perf_tool *tool,
 				union perf_event *event,
 				struct perf_sample *sample,
 				struct perf_evsel *evsel,
 				struct machine *machine)
 {
 	struct addr_location al;
+	struct perf_script *script = container_of(tool, struct perf_script, tool);
 	struct thread *thread = machine__findnew_thread(machine, sample->pid,
 							sample->tid);
 
@@ -569,7 +577,8 @@ static int process_sample_event(struct perf_tool *tool __maybe_unused,
 		return 0;
 	}
 
-	if (perf_event__preprocess_sample(event, machine, &al, sample) < 0) {
+	if (perf_event__preprocess_sample(event, machine, &al, sample,
+					  script->session) < 0) {
 		pr_err("problem processing %d event, skipping it.\n",
 		       event->header.type);
 		return -1;
@@ -586,13 +595,6 @@ static int process_sample_event(struct perf_tool *tool __maybe_unused,
 	return 0;
 }
 
-struct perf_script {
-	struct perf_tool	tool;
-	struct perf_session	*session;
-	bool			show_task_events;
-	bool			show_mmap_events;
-};
-
 static int process_attr(struct perf_tool *tool, union perf_event *event,
 			struct perf_evlist **pevlist)
 {
diff --git a/tools/perf/builtin-timechart.c b/tools/perf/builtin-timechart.c
index f3bb1a4bf060..4178727be12c 100644
--- a/tools/perf/builtin-timechart.c
+++ b/tools/perf/builtin-timechart.c
@@ -48,6 +48,7 @@ struct wake_event;
 
 struct timechart {
 	struct perf_tool	tool;
+	struct perf_session	*session;
 	struct per_pid		*all_data;
 	struct power_event	*power_events;
 	struct wake_event	*wake_events;
@@ -469,7 +470,8 @@ static void sched_switch(struct timechart *tchart, int cpu, u64 timestamp,
 
 static const char *cat_backtrace(union perf_event *event,
 				 struct perf_sample *sample,
-				 struct machine *machine)
+				 struct machine *machine,
+				 struct perf_session *session)
 {
 	struct addr_location al;
 	unsigned int i;
@@ -488,7 +490,8 @@ static const char *cat_backtrace(union perf_event *event,
 	if (!chain)
 		goto exit;
 
-	if (perf_event__preprocess_sample(event, machine, &al, sample) < 0) {
+	if (perf_event__preprocess_sample(event, machine, &al, sample,
+					  session) < 0) {
 		fprintf(stderr, "problem processing %d event, skipping it.\n",
 			event->header.type);
 		goto exit;
@@ -567,7 +570,7 @@ static int process_sample_event(struct perf_tool *tool,
 	if (evsel->handler != NULL) {
 		tracepoint_handler f = evsel->handler;
 		return f(tchart, evsel, sample,
-			 cat_backtrace(event, sample, machine));
+			 cat_backtrace(event, sample, machine, tchart->session));
 	}
 
 	return 0;
@@ -1623,6 +1626,7 @@ static int __cmd_timechart(struct timechart *tchart, const char *output_name)
 		goto out_delete;
 	}
 
+	tchart->session = session;
 	ret = perf_session__process_events(session, &tchart->tool);
 	if (ret)
 		goto out_delete;
diff --git a/tools/perf/builtin-top.c b/tools/perf/builtin-top.c
index c4c7eac69de4..69a0badfb745 100644
--- a/tools/perf/builtin-top.c
+++ b/tools/perf/builtin-top.c
@@ -723,7 +723,8 @@ static void perf_event__process_sample(struct perf_tool *tool,
 	if (event->header.misc & PERF_RECORD_MISC_EXACT_IP)
 		top->exact_samples++;
 
-	if (perf_event__preprocess_sample(event, machine, &al, sample) < 0)
+	if (perf_event__preprocess_sample(event, machine, &al, sample,
+					  top->session) < 0)
 		return;
 
 	if (!top->kptr_restrict_warned &&
diff --git a/tools/perf/tests/hists_cumulate.c b/tools/perf/tests/hists_cumulate.c
index 18619966454c..60682e62d9de 100644
--- a/tools/perf/tests/hists_cumulate.c
+++ b/tools/perf/tests/hists_cumulate.c
@@ -101,7 +101,7 @@ static int add_hist_entries(struct hists *hists, struct machine *machine)
 		sample.callchain = (struct ip_callchain *)fake_callchains[i];
 
 		if (perf_event__preprocess_sample(&event, machine, &al,
-						  &sample) < 0)
+						  &sample, NULL) < 0)
 			goto out;
 
 		if (hist_entry_iter__add(&iter, &al, evsel, &sample,
diff --git a/tools/perf/tests/hists_filter.c b/tools/perf/tests/hists_filter.c
index 59e53db7914c..1c4e495d5137 100644
--- a/tools/perf/tests/hists_filter.c
+++ b/tools/perf/tests/hists_filter.c
@@ -78,7 +78,7 @@ static int add_hist_entries(struct perf_evlist *evlist,
 			sample.ip = fake_samples[i].ip;
 
 			if (perf_event__preprocess_sample(&event, machine, &al,
-							  &sample) < 0)
+							  &sample, NULL) < 0)
 				goto out;
 
 			if (hist_entry_iter__add(&iter, &al, evsel, &sample,
diff --git a/tools/perf/tests/hists_link.c b/tools/perf/tests/hists_link.c
index 278ba8344c23..a731a531a3e2 100644
--- a/tools/perf/tests/hists_link.c
+++ b/tools/perf/tests/hists_link.c
@@ -86,7 +86,7 @@ static int add_hist_entries(struct perf_evlist *evlist, struct machine *machine)
 			sample.tid = fake_common_samples[k].pid;
 			sample.ip = fake_common_samples[k].ip;
 			if (perf_event__preprocess_sample(&event, machine, &al,
-							  &sample) < 0)
+							  &sample, NULL) < 0)
 				goto out;
 
 			he = __hists__add_entry(hists, &al, NULL,
@@ -110,7 +110,7 @@ static int add_hist_entries(struct perf_evlist *evlist, struct machine *machine)
 			sample.tid = fake_samples[i][k].pid;
 			sample.ip = fake_samples[i][k].ip;
 			if (perf_event__preprocess_sample(&event, machine, &al,
-							  &sample) < 0)
+							  &sample, NULL) < 0)
 				goto out;
 
 			he = __hists__add_entry(hists, &al, NULL,
diff --git a/tools/perf/tests/hists_output.c b/tools/perf/tests/hists_output.c
index b52c9faea224..f4e3286cd496 100644
--- a/tools/perf/tests/hists_output.c
+++ b/tools/perf/tests/hists_output.c
@@ -67,7 +67,7 @@ static int add_hist_entries(struct hists *hists, struct machine *machine)
 		sample.ip = fake_samples[i].ip;
 
 		if (perf_event__preprocess_sample(&event, machine, &al,
-						  &sample) < 0)
+						  &sample, NULL) < 0)
 			goto out;
 
 		if (hist_entry_iter__add(&iter, &al, evsel, &sample,
diff --git a/tools/perf/util/event.c b/tools/perf/util/event.c
index 6c6d044e959a..7a90c62ad07a 100644
--- a/tools/perf/util/event.c
+++ b/tools/perf/util/event.c
@@ -822,7 +822,8 @@ void thread__find_addr_location(struct thread *thread,
 int perf_event__preprocess_sample(const union perf_event *event,
 				  struct machine *machine,
 				  struct addr_location *al,
-				  struct perf_sample *sample)
+				  struct perf_sample *sample,
+				  struct perf_session *session __maybe_unused)
 {
 	u8 cpumode = event->header.misc & PERF_RECORD_MISC_CPUMODE_MASK;
 	struct thread *thread = machine__findnew_thread(machine, sample->pid,
diff --git a/tools/perf/util/event.h b/tools/perf/util/event.h
index c4ffe2bd0738..19814f70292b 100644
--- a/tools/perf/util/event.h
+++ b/tools/perf/util/event.h
@@ -353,11 +353,13 @@ int perf_event__process(struct perf_tool *tool,
 			struct machine *machine);
 
 struct addr_location;
+struct perf_session;
 
 int perf_event__preprocess_sample(const union perf_event *event,
 				  struct machine *machine,
 				  struct addr_location *al,
-				  struct perf_sample *sample);
+				  struct perf_sample *sample,
+				  struct perf_session *session);
 
 struct thread;
 
-- 
2.2.2


^ permalink raw reply related	[flat|nested] 221+ messages in thread

* [PATCH 17/42] perf script: Pass session arg to ->process_event callback
  2015-01-29  8:06 [RFC/PATCHSET 00/42] perf tools: Speed-up perf report by using multi thread (v2) Namhyung Kim
                   ` (15 preceding siblings ...)
  2015-01-29  8:06 ` [PATCH 16/42] perf tools: Pass session arg to perf_event__preprocess_sample() Namhyung Kim
@ 2015-01-29  8:06 ` Namhyung Kim
  2015-01-29  8:06 ` [PATCH 18/42] perf tools: Introduce thread__comm_time() helpers Namhyung Kim
                   ` (25 subsequent siblings)
  42 siblings, 0 replies; 221+ messages in thread
From: Namhyung Kim @ 2015-01-29  8:06 UTC (permalink / raw)
  To: Arnaldo Carvalho de Melo
  Cc: Ingo Molnar, Peter Zijlstra, Jiri Olsa, LKML, David Ahern,
	Adrian Hunter, Andi Kleen, Stephane Eranian, Frederic Weisbecker

Sometimes it needs to retrieve symbol info inside a script engine so
we need to pass the session pointer to find the symbol correctly as
with previous patch.

Signed-off-by: Namhyung Kim <namhyung@kernel.org>
---
 tools/perf/builtin-script.c                        | 23 ++++++++++++----------
 tools/perf/util/db-export.c                        |  6 ++++--
 tools/perf/util/db-export.h                        |  4 +++-
 tools/perf/util/event.c                            |  3 ++-
 tools/perf/util/event.h                            |  3 ++-
 .../perf/util/scripting-engines/trace-event-perl.c |  3 ++-
 .../util/scripting-engines/trace-event-python.c    |  5 +++--
 tools/perf/util/trace-event-scripting.c            |  3 ++-
 tools/perf/util/trace-event.h                      |  3 ++-
 9 files changed, 33 insertions(+), 20 deletions(-)

diff --git a/tools/perf/builtin-script.c b/tools/perf/builtin-script.c
index ab920f8cded6..4a007110d2f7 100644
--- a/tools/perf/builtin-script.c
+++ b/tools/perf/builtin-script.c
@@ -377,9 +377,10 @@ static void print_sample_start(struct perf_sample *sample,
 }
 
 static void print_sample_addr(union perf_event *event,
-			  struct perf_sample *sample,
-			  struct thread *thread,
-			  struct perf_event_attr *attr)
+			      struct perf_sample *sample,
+			      struct thread *thread,
+			      struct perf_event_attr *attr,
+			      struct perf_session *session)
 {
 	struct addr_location al;
 
@@ -388,7 +389,7 @@ static void print_sample_addr(union perf_event *event,
 	if (!sample_addr_correlates_sym(attr))
 		return;
 
-	perf_event__preprocess_sample_addr(event, sample, thread, &al);
+	perf_event__preprocess_sample_addr(event, sample, thread, &al, session);
 
 	if (PRINT_FIELD(SYM)) {
 		printf(" ");
@@ -409,7 +410,8 @@ static void print_sample_bts(union perf_event *event,
 			     struct perf_sample *sample,
 			     struct perf_evsel *evsel,
 			     struct thread *thread,
-			     struct addr_location *al)
+			     struct addr_location *al,
+			     struct perf_session *session)
 {
 	struct perf_event_attr *attr = &evsel->attr;
 	bool print_srcline_last = false;
@@ -436,7 +438,7 @@ static void print_sample_bts(union perf_event *event,
 	    ((evsel->attr.sample_type & PERF_SAMPLE_ADDR) &&
 	     !output[attr->type].user_set)) {
 		printf(" => ");
-		print_sample_addr(event, sample, thread, attr);
+		print_sample_addr(event, sample, thread, attr, session);
 	}
 
 	if (print_srcline_last)
@@ -447,7 +449,7 @@ static void print_sample_bts(union perf_event *event,
 
 static void process_event(union perf_event *event, struct perf_sample *sample,
 			  struct perf_evsel *evsel, struct thread *thread,
-			  struct addr_location *al)
+			  struct addr_location *al, struct perf_session *session)
 {
 	struct perf_event_attr *attr = &evsel->attr;
 
@@ -465,7 +467,7 @@ static void process_event(union perf_event *event, struct perf_sample *sample,
 	}
 
 	if (is_bts_event(attr)) {
-		print_sample_bts(event, sample, evsel, thread, al);
+		print_sample_bts(event, sample, evsel, thread, al, session);
 		return;
 	}
 
@@ -473,7 +475,7 @@ static void process_event(union perf_event *event, struct perf_sample *sample,
 		event_format__print(evsel->tp_format, sample->cpu,
 				    sample->raw_data, sample->raw_size);
 	if (PRINT_FIELD(ADDR))
-		print_sample_addr(event, sample, thread, attr);
+		print_sample_addr(event, sample, thread, attr, session);
 
 	if (PRINT_FIELD(IP)) {
 		if (!symbol_conf.use_callchain)
@@ -590,7 +592,8 @@ static int process_sample_event(struct perf_tool *tool,
 	if (cpu_list && !test_bit(sample->cpu, cpu_bitmap))
 		return 0;
 
-	scripting_ops->process_event(event, sample, evsel, thread, &al);
+	scripting_ops->process_event(event, sample, evsel, thread, &al,
+				     script->session);
 
 	return 0;
 }
diff --git a/tools/perf/util/db-export.c b/tools/perf/util/db-export.c
index c81dae399763..e9ad11fe2e16 100644
--- a/tools/perf/util/db-export.c
+++ b/tools/perf/util/db-export.c
@@ -282,7 +282,8 @@ int db_export__branch_type(struct db_export *dbe, u32 branch_type,
 
 int db_export__sample(struct db_export *dbe, union perf_event *event,
 		      struct perf_sample *sample, struct perf_evsel *evsel,
-		      struct thread *thread, struct addr_location *al)
+		      struct thread *thread, struct addr_location *al,
+		      struct perf_session *session)
 {
 	struct export_sample es = {
 		.event = event,
@@ -328,7 +329,8 @@ int db_export__sample(struct db_export *dbe, union perf_event *event,
 	    sample_addr_correlates_sym(&evsel->attr)) {
 		struct addr_location addr_al;
 
-		perf_event__preprocess_sample_addr(event, sample, thread, &addr_al);
+		perf_event__preprocess_sample_addr(event, sample, thread,
+						   &addr_al, session);
 		err = db_ids_from_al(dbe, &addr_al, &es.addr_dso_db_id,
 				     &es.addr_sym_db_id, &es.addr_offset);
 		if (err)
diff --git a/tools/perf/util/db-export.h b/tools/perf/util/db-export.h
index adbd22d66798..b994f1041d19 100644
--- a/tools/perf/util/db-export.h
+++ b/tools/perf/util/db-export.h
@@ -29,6 +29,7 @@ struct addr_location;
 struct call_return_processor;
 struct call_path;
 struct call_return;
+struct perf_session;
 
 struct export_sample {
 	union perf_event	*event;
@@ -97,7 +98,8 @@ int db_export__branch_type(struct db_export *dbe, u32 branch_type,
 			   const char *name);
 int db_export__sample(struct db_export *dbe, union perf_event *event,
 		      struct perf_sample *sample, struct perf_evsel *evsel,
-		      struct thread *thread, struct addr_location *al);
+		      struct thread *thread, struct addr_location *al,
+		      struct perf_session *session);
 
 int db_export__branch_types(struct db_export *dbe);
 
diff --git a/tools/perf/util/event.c b/tools/perf/util/event.c
index 7a90c62ad07a..186960a09024 100644
--- a/tools/perf/util/event.c
+++ b/tools/perf/util/event.c
@@ -904,7 +904,8 @@ bool sample_addr_correlates_sym(struct perf_event_attr *attr)
 void perf_event__preprocess_sample_addr(union perf_event *event,
 					struct perf_sample *sample,
 					struct thread *thread,
-					struct addr_location *al)
+					struct addr_location *al,
+					struct perf_session *session __maybe_unused)
 {
 	u8 cpumode = event->header.misc & PERF_RECORD_MISC_CPUMODE_MASK;
 
diff --git a/tools/perf/util/event.h b/tools/perf/util/event.h
index 19814f70292b..27261320249a 100644
--- a/tools/perf/util/event.h
+++ b/tools/perf/util/event.h
@@ -368,7 +368,8 @@ bool sample_addr_correlates_sym(struct perf_event_attr *attr);
 void perf_event__preprocess_sample_addr(union perf_event *event,
 					struct perf_sample *sample,
 					struct thread *thread,
-					struct addr_location *al);
+					struct addr_location *al,
+					struct perf_session *session);
 
 const char *perf_event__name(unsigned int id);
 
diff --git a/tools/perf/util/scripting-engines/trace-event-perl.c b/tools/perf/util/scripting-engines/trace-event-perl.c
index 22ebc46226e7..dd69fbaf03b8 100644
--- a/tools/perf/util/scripting-engines/trace-event-perl.c
+++ b/tools/perf/util/scripting-engines/trace-event-perl.c
@@ -356,7 +356,8 @@ static void perl_process_event(union perf_event *event,
 			       struct perf_sample *sample,
 			       struct perf_evsel *evsel,
 			       struct thread *thread,
-			       struct addr_location *al __maybe_unused)
+			       struct addr_location *al __maybe_unused,
+			       struct perf_session *session __maybe_unused)
 {
 	perl_process_tracepoint(sample, evsel, thread);
 	perl_process_event_generic(event, sample, evsel);
diff --git a/tools/perf/util/scripting-engines/trace-event-python.c b/tools/perf/util/scripting-engines/trace-event-python.c
index 0c815a40a6e8..802def46af7b 100644
--- a/tools/perf/util/scripting-engines/trace-event-python.c
+++ b/tools/perf/util/scripting-engines/trace-event-python.c
@@ -839,7 +839,8 @@ static void python_process_event(union perf_event *event,
 				 struct perf_sample *sample,
 				 struct perf_evsel *evsel,
 				 struct thread *thread,
-				 struct addr_location *al)
+				 struct addr_location *al,
+				 struct perf_session *session)
 {
 	struct tables *tables = &tables_global;
 
@@ -851,7 +852,7 @@ static void python_process_event(union perf_event *event,
 	default:
 		if (tables->db_export_mode)
 			db_export__sample(&tables->dbe, event, sample, evsel,
-					  thread, al);
+					  thread, al, session);
 		else
 			python_process_general_event(sample, evsel, thread, al);
 	}
diff --git a/tools/perf/util/trace-event-scripting.c b/tools/perf/util/trace-event-scripting.c
index 5c9bdd1591a9..36ed50d71171 100644
--- a/tools/perf/util/trace-event-scripting.c
+++ b/tools/perf/util/trace-event-scripting.c
@@ -44,7 +44,8 @@ static void process_event_unsupported(union perf_event *event __maybe_unused,
 				      struct perf_sample *sample __maybe_unused,
 				      struct perf_evsel *evsel __maybe_unused,
 				      struct thread *thread __maybe_unused,
-				      struct addr_location *al __maybe_unused)
+				      struct addr_location *al __maybe_unused,
+				      struct perf_session *session __maybe_unused)
 {
 }
 
diff --git a/tools/perf/util/trace-event.h b/tools/perf/util/trace-event.h
index 52aaa19e1eb1..c5870e57eee9 100644
--- a/tools/perf/util/trace-event.h
+++ b/tools/perf/util/trace-event.h
@@ -70,7 +70,8 @@ struct scripting_ops {
 			       struct perf_sample *sample,
 			       struct perf_evsel *evsel,
 			       struct thread *thread,
-				   struct addr_location *al);
+			       struct addr_location *al,
+			       struct perf_session *session);
 	int (*generate_script) (struct pevent *pevent, const char *outfile);
 };
 
-- 
2.2.2


^ permalink raw reply related	[flat|nested] 221+ messages in thread

* [PATCH 18/42] perf tools: Introduce thread__comm_time() helpers
  2015-01-29  8:06 [RFC/PATCHSET 00/42] perf tools: Speed-up perf report by using multi thread (v2) Namhyung Kim
                   ` (16 preceding siblings ...)
  2015-01-29  8:06 ` [PATCH 17/42] perf script: Pass session arg to ->process_event callback Namhyung Kim
@ 2015-01-29  8:06 ` Namhyung Kim
  2015-01-29  8:07 ` [PATCH 19/42] perf tools: Add a test case for thread comm handling Namhyung Kim
                   ` (24 subsequent siblings)
  42 siblings, 0 replies; 221+ messages in thread
From: Namhyung Kim @ 2015-01-29  8:06 UTC (permalink / raw)
  To: Arnaldo Carvalho de Melo
  Cc: Ingo Molnar, Peter Zijlstra, Jiri Olsa, LKML, David Ahern,
	Adrian Hunter, Andi Kleen, Stephane Eranian, Frederic Weisbecker

When data file indexing is enabled, it processes all task, comm and mmap
events first and then goes to the sample events.  So all it sees is the
last comm of a thread although it has information at the time of sample.

Sort thread's comm by time so that it can find appropriate comm at the
sample time.  The thread__comm_time() will mostly work even if
PERF_SAMPLE_TIME bit is off since in that case, sample->time will be
-1 so it'll take the last comm anyway.

Cc: Frederic Weisbecker <fweisbec@gmail.com>
Signed-off-by: Namhyung Kim <namhyung@kernel.org>
---
 tools/perf/util/thread.c | 33 ++++++++++++++++++++++++++++++++-
 tools/perf/util/thread.h |  2 ++
 2 files changed, 34 insertions(+), 1 deletion(-)

diff --git a/tools/perf/util/thread.c b/tools/perf/util/thread.c
index 9ebc8b1f9be5..ad96725105c2 100644
--- a/tools/perf/util/thread.c
+++ b/tools/perf/util/thread.c
@@ -103,6 +103,21 @@ struct comm *thread__exec_comm(const struct thread *thread)
 	return last;
 }
 
+struct comm *thread__comm_time(const struct thread *thread, u64 timestamp)
+{
+	struct comm *comm;
+
+	list_for_each_entry(comm, &thread->comm_list, list) {
+		if (timestamp >= comm->start)
+			return comm;
+	}
+
+	if (list_empty(&thread->comm_list))
+		return NULL;
+
+	return list_last_entry(&thread->comm_list, struct comm, list);
+}
+
 int __thread__set_comm(struct thread *thread, const char *str, u64 timestamp,
 		       bool exec)
 {
@@ -118,7 +133,13 @@ int __thread__set_comm(struct thread *thread, const char *str, u64 timestamp,
 		new = comm__new(str, timestamp, exec);
 		if (!new)
 			return -ENOMEM;
-		list_add(&new->list, &thread->comm_list);
+
+		/* sort by time */
+		list_for_each_entry(curr, &thread->comm_list, list) {
+			if (timestamp >= curr->start)
+				break;
+		}
+		list_add_tail(&new->list, &curr->list);
 
 		if (exec)
 			unwind__flush_access(thread);
@@ -139,6 +160,16 @@ const char *thread__comm_str(const struct thread *thread)
 	return comm__str(comm);
 }
 
+const char *thread__comm_str_time(const struct thread *thread, u64 timestamp)
+{
+	const struct comm *comm = thread__comm_time(thread, timestamp);
+
+	if (!comm)
+		return NULL;
+
+	return comm__str(comm);
+}
+
 /* CHECKME: it should probably better return the max comm len from its comm list */
 int thread__comm_len(struct thread *thread)
 {
diff --git a/tools/perf/util/thread.h b/tools/perf/util/thread.h
index 160fd066a7d1..be67c3bad5e7 100644
--- a/tools/perf/util/thread.h
+++ b/tools/perf/util/thread.h
@@ -53,7 +53,9 @@ static inline int thread__set_comm(struct thread *thread, const char *comm,
 int thread__comm_len(struct thread *thread);
 struct comm *thread__comm(const struct thread *thread);
 struct comm *thread__exec_comm(const struct thread *thread);
+struct comm *thread__comm_time(const struct thread *thread, u64 timestamp);
 const char *thread__comm_str(const struct thread *thread);
+const char *thread__comm_str_time(const struct thread *thread, u64 timestamp);
 void thread__insert_map(struct thread *thread, struct map *map);
 int thread__fork(struct thread *thread, struct thread *parent, u64 timestamp);
 size_t thread__fprintf(struct thread *thread, FILE *fp);
-- 
2.2.2


^ permalink raw reply related	[flat|nested] 221+ messages in thread

* [PATCH 19/42] perf tools: Add a test case for thread comm handling
  2015-01-29  8:06 [RFC/PATCHSET 00/42] perf tools: Speed-up perf report by using multi thread (v2) Namhyung Kim
                   ` (17 preceding siblings ...)
  2015-01-29  8:06 ` [PATCH 18/42] perf tools: Introduce thread__comm_time() helpers Namhyung Kim
@ 2015-01-29  8:07 ` Namhyung Kim
  2015-01-29  8:07 ` [PATCH 20/42] perf tools: Use thread__comm_time() when adding hist entries Namhyung Kim
                   ` (23 subsequent siblings)
  42 siblings, 0 replies; 221+ messages in thread
From: Namhyung Kim @ 2015-01-29  8:07 UTC (permalink / raw)
  To: Arnaldo Carvalho de Melo
  Cc: Ingo Molnar, Peter Zijlstra, Jiri Olsa, LKML, David Ahern,
	Adrian Hunter, Andi Kleen, Stephane Eranian, Frederic Weisbecker

The new test case checks various thread comm handling like overridding
and time sorting.

Cc: Frederic Weisbecker <fweisbec@gmail.com>
Signed-off-by: Namhyung Kim <namhyung@kernel.org>
---
 tools/perf/Makefile.perf        |  1 +
 tools/perf/tests/builtin-test.c |  4 ++++
 tools/perf/tests/tests.h        |  1 +
 tools/perf/tests/thread-comm.c  | 47 +++++++++++++++++++++++++++++++++++++++++
 4 files changed, 53 insertions(+)
 create mode 100644 tools/perf/tests/thread-comm.c

diff --git a/tools/perf/Makefile.perf b/tools/perf/Makefile.perf
index aa6a50447c32..8507891db69d 100644
--- a/tools/perf/Makefile.perf
+++ b/tools/perf/Makefile.perf
@@ -458,6 +458,7 @@ endif
 LIB_OBJS += $(OUTPUT)tests/mmap-thread-lookup.o
 LIB_OBJS += $(OUTPUT)tests/thread-mg-share.o
 LIB_OBJS += $(OUTPUT)tests/switch-tracking.o
+LIB_OBJS += $(OUTPUT)tests/thread-comm.o
 
 BUILTIN_OBJS += $(OUTPUT)builtin-annotate.o
 BUILTIN_OBJS += $(OUTPUT)builtin-bench.o
diff --git a/tools/perf/tests/builtin-test.c b/tools/perf/tests/builtin-test.c
index 4b7d9ab0f049..1b463d82a71a 100644
--- a/tools/perf/tests/builtin-test.c
+++ b/tools/perf/tests/builtin-test.c
@@ -167,6 +167,10 @@ static struct test {
 		.func = test__fdarray__add,
 	},
 	{
+		.desc = "Test thread comm handling",
+		.func = test__thread_comm,
+	},
+	{
 		.func = NULL,
 	},
 };
diff --git a/tools/perf/tests/tests.h b/tools/perf/tests/tests.h
index 00e776a87a9c..43ac17780629 100644
--- a/tools/perf/tests/tests.h
+++ b/tools/perf/tests/tests.h
@@ -51,6 +51,7 @@ int test__hists_cumulate(void);
 int test__switch_tracking(void);
 int test__fdarray__filter(void);
 int test__fdarray__add(void);
+int test__thread_comm(void);
 
 #if defined(__x86_64__) || defined(__i386__) || defined(__arm__)
 #ifdef HAVE_DWARF_UNWIND_SUPPORT
diff --git a/tools/perf/tests/thread-comm.c b/tools/perf/tests/thread-comm.c
new file mode 100644
index 000000000000..44ee85d71581
--- /dev/null
+++ b/tools/perf/tests/thread-comm.c
@@ -0,0 +1,47 @@
+#include "tests.h"
+#include "machine.h"
+#include "thread.h"
+#include "debug.h"
+
+int test__thread_comm(void)
+{
+	struct machines machines;
+	struct machine *machine;
+	struct thread *t;
+
+	/*
+	 * This test is to check whether it can retrieve a correct
+	 * comm for a given time.  When multi-file data storage is
+	 * enabled, those task/comm events are processed first so the
+	 * later sample should find a matching comm properly.
+	 */
+	machines__init(&machines);
+	machine = &machines.host;
+
+	t = machine__findnew_thread(machine, 100, 100);
+	TEST_ASSERT_VAL("wrong init thread comm",
+			!strcmp(thread__comm_str(t), ":100"));
+
+	thread__set_comm(t, "perf-test1", 10000);
+	TEST_ASSERT_VAL("failed to override thread comm",
+			!strcmp(thread__comm_str(t), "perf-test1"));
+
+	thread__set_comm(t, "perf-test2", 20000);
+	thread__set_comm(t, "perf-test3", 30000);
+	thread__set_comm(t, "perf-test4", 40000);
+
+	TEST_ASSERT_VAL("failed to find timed comm",
+			!strcmp(thread__comm_str_time(t, 20000), "perf-test2"));
+	TEST_ASSERT_VAL("failed to find timed comm",
+			!strcmp(thread__comm_str_time(t, 35000), "perf-test3"));
+	TEST_ASSERT_VAL("failed to find timed comm",
+			!strcmp(thread__comm_str_time(t, 50000), "perf-test4"));
+
+	thread__set_comm(t, "perf-test1.5", 15000);
+	TEST_ASSERT_VAL("failed to sort timed comm",
+			!strcmp(thread__comm_str_time(t, 15000), "perf-test1.5"));
+
+	machine__delete_threads(machine);
+	machines__exit(&machines);
+	return 0;
+}
-- 
2.2.2


^ permalink raw reply related	[flat|nested] 221+ messages in thread

* [PATCH 20/42] perf tools: Use thread__comm_time() when adding hist entries
  2015-01-29  8:06 [RFC/PATCHSET 00/42] perf tools: Speed-up perf report by using multi thread (v2) Namhyung Kim
                   ` (18 preceding siblings ...)
  2015-01-29  8:07 ` [PATCH 19/42] perf tools: Add a test case for thread comm handling Namhyung Kim
@ 2015-01-29  8:07 ` Namhyung Kim
  2015-01-29  8:07 ` [PATCH 21/42] perf tools: Convert dead thread list into rbtree Namhyung Kim
                   ` (22 subsequent siblings)
  42 siblings, 0 replies; 221+ messages in thread
From: Namhyung Kim @ 2015-01-29  8:07 UTC (permalink / raw)
  To: Arnaldo Carvalho de Melo
  Cc: Ingo Molnar, Peter Zijlstra, Jiri Olsa, LKML, David Ahern,
	Adrian Hunter, Andi Kleen, Stephane Eranian, Frederic Weisbecker

Now thread->comm can be handled with time properly, use it to find
correct comm when adding hist entries.

Cc: Frederic Weisbecker <fweisbec@gmail.com>
Signed-off-by: Namhyung Kim <namhyung@kernel.org>
---
 tools/perf/builtin-annotate.c |  5 +++--
 tools/perf/builtin-diff.c     |  8 ++++----
 tools/perf/tests/hists_link.c |  4 ++--
 tools/perf/util/hist.c        | 19 ++++++++++---------
 tools/perf/util/hist.h        |  2 +-
 5 files changed, 20 insertions(+), 18 deletions(-)

diff --git a/tools/perf/builtin-annotate.c b/tools/perf/builtin-annotate.c
index b89e4c6ed488..50628900f9fa 100644
--- a/tools/perf/builtin-annotate.c
+++ b/tools/perf/builtin-annotate.c
@@ -47,7 +47,7 @@ struct perf_annotate {
 };
 
 static int perf_evsel__add_sample(struct perf_evsel *evsel,
-				  struct perf_sample *sample __maybe_unused,
+				  struct perf_sample *sample,
 				  struct addr_location *al,
 				  struct perf_annotate *ann)
 {
@@ -67,7 +67,8 @@ static int perf_evsel__add_sample(struct perf_evsel *evsel,
 		return 0;
 	}
 
-	he = __hists__add_entry(hists, al, NULL, NULL, NULL, 1, 1, 0, true);
+	he = __hists__add_entry(hists, al, NULL, NULL, NULL, 1, 1, 0,
+				sample->time, true);
 	if (he == NULL)
 		return -ENOMEM;
 
diff --git a/tools/perf/builtin-diff.c b/tools/perf/builtin-diff.c
index 3e2229227062..ddf6f0999838 100644
--- a/tools/perf/builtin-diff.c
+++ b/tools/perf/builtin-diff.c
@@ -313,10 +313,10 @@ static int formula_fprintf(struct hist_entry *he, struct hist_entry *pair,
 
 static int hists__add_entry(struct hists *hists,
 			    struct addr_location *al, u64 period,
-			    u64 weight, u64 transaction)
+			    u64 weight, u64 transaction, u64 timestamp)
 {
 	if (__hists__add_entry(hists, al, NULL, NULL, NULL, period, weight,
-			       transaction, true) != NULL)
+			       transaction, timestamp, true) != NULL)
 		return 0;
 	return -ENOMEM;
 }
@@ -338,8 +338,8 @@ static int diff__process_sample_event(struct perf_tool *tool,
 		return -1;
 	}
 
-	if (hists__add_entry(hists, &al, sample->period,
-			     sample->weight, sample->transaction)) {
+	if (hists__add_entry(hists, &al, sample->period, sample->weight,
+			     sample->transaction, sample->time)) {
 		pr_warning("problem incrementing symbol period, skipping event\n");
 		return -1;
 	}
diff --git a/tools/perf/tests/hists_link.c b/tools/perf/tests/hists_link.c
index a731a531a3e2..4f3d45692acb 100644
--- a/tools/perf/tests/hists_link.c
+++ b/tools/perf/tests/hists_link.c
@@ -90,7 +90,7 @@ static int add_hist_entries(struct perf_evlist *evlist, struct machine *machine)
 				goto out;
 
 			he = __hists__add_entry(hists, &al, NULL,
-						NULL, NULL, 1, 1, 0, true);
+						NULL, NULL, 1, 1, 0, -1, true);
 			if (he == NULL)
 				goto out;
 
@@ -114,7 +114,7 @@ static int add_hist_entries(struct perf_evlist *evlist, struct machine *machine)
 				goto out;
 
 			he = __hists__add_entry(hists, &al, NULL,
-						NULL, NULL, 1, 1, 0, true);
+						NULL, NULL, 1, 1, 0, -1, true);
 			if (he == NULL)
 				goto out;
 
diff --git a/tools/perf/util/hist.c b/tools/perf/util/hist.c
index 70b48a65064c..4badf2491fbf 100644
--- a/tools/perf/util/hist.c
+++ b/tools/perf/util/hist.c
@@ -447,11 +447,11 @@ struct hist_entry *__hists__add_entry(struct hists *hists,
 				      struct branch_info *bi,
 				      struct mem_info *mi,
 				      u64 period, u64 weight, u64 transaction,
-				      bool sample_self)
+				      u64 timestamp, bool sample_self)
 {
 	struct hist_entry entry = {
 		.thread	= al->thread,
-		.comm = thread__comm(al->thread),
+		.comm = thread__comm_time(al->thread, timestamp),
 		.ms = {
 			.map	= al->map,
 			.sym	= al->sym,
@@ -509,13 +509,14 @@ iter_add_single_mem_entry(struct hist_entry_iter *iter, struct addr_location *al
 {
 	u64 cost;
 	struct mem_info *mi = iter->priv;
+	struct perf_sample *sample = iter->sample;
 	struct hists *hists = evsel__hists(iter->evsel);
 	struct hist_entry *he;
 
 	if (mi == NULL)
 		return -EINVAL;
 
-	cost = iter->sample->weight;
+	cost = sample->weight;
 	if (!cost)
 		cost = 1;
 
@@ -527,7 +528,7 @@ iter_add_single_mem_entry(struct hist_entry_iter *iter, struct addr_location *al
 	 * and the he_stat__add_period() function.
 	 */
 	he = __hists__add_entry(hists, al, iter->parent, NULL, mi,
-				cost, cost, 0, true);
+				cost, cost, 0, sample->time, true);
 	if (!he)
 		return -ENOMEM;
 
@@ -628,7 +629,7 @@ iter_add_next_branch_entry(struct hist_entry_iter *iter, struct addr_location *a
 	 * and not events sampled. Thus we use a pseudo period of 1.
 	 */
 	he = __hists__add_entry(hists, al, iter->parent, &bi[i], NULL,
-				1, 1, 0, true);
+				1, 1, 0, iter->sample->time, true);
 	if (he == NULL)
 		return -ENOMEM;
 
@@ -666,7 +667,7 @@ iter_add_single_normal_entry(struct hist_entry_iter *iter, struct addr_location
 
 	he = __hists__add_entry(evsel__hists(evsel), al, iter->parent, NULL, NULL,
 				sample->period, sample->weight,
-				sample->transaction, true);
+				sample->transaction, sample->time, true);
 	if (he == NULL)
 		return -ENOMEM;
 
@@ -728,7 +729,7 @@ iter_add_single_cumulative_entry(struct hist_entry_iter *iter,
 
 	he = __hists__add_entry(hists, al, iter->parent, NULL, NULL,
 				sample->period, sample->weight,
-				sample->transaction, true);
+				sample->transaction, sample->time, true);
 	if (he == NULL)
 		return -ENOMEM;
 
@@ -772,7 +773,7 @@ iter_add_next_cumulative_entry(struct hist_entry_iter *iter,
 	struct hist_entry he_tmp = {
 		.cpu = al->cpu,
 		.thread = al->thread,
-		.comm = thread__comm(al->thread),
+		.comm = thread__comm_time(al->thread, sample->time),
 		.ip = al->addr,
 		.ms = {
 			.map = al->map,
@@ -801,7 +802,7 @@ iter_add_next_cumulative_entry(struct hist_entry_iter *iter,
 
 	he = __hists__add_entry(evsel__hists(evsel), al, iter->parent, NULL, NULL,
 				sample->period, sample->weight,
-				sample->transaction, false);
+				sample->transaction, sample->time, false);
 	if (he == NULL)
 		return -ENOMEM;
 
diff --git a/tools/perf/util/hist.h b/tools/perf/util/hist.h
index 2b690d028907..0eed50a5b1f0 100644
--- a/tools/perf/util/hist.h
+++ b/tools/perf/util/hist.h
@@ -109,7 +109,7 @@ struct hist_entry *__hists__add_entry(struct hists *hists,
 				      struct branch_info *bi,
 				      struct mem_info *mi, u64 period,
 				      u64 weight, u64 transaction,
-				      bool sample_self);
+				      u64 timestamp, bool sample_self);
 int hist_entry_iter__add(struct hist_entry_iter *iter, struct addr_location *al,
 			 struct perf_evsel *evsel, struct perf_sample *sample,
 			 int max_stack_depth, void *arg);
-- 
2.2.2


^ permalink raw reply related	[flat|nested] 221+ messages in thread

* [PATCH 21/42] perf tools: Convert dead thread list into rbtree
  2015-01-29  8:06 [RFC/PATCHSET 00/42] perf tools: Speed-up perf report by using multi thread (v2) Namhyung Kim
                   ` (19 preceding siblings ...)
  2015-01-29  8:07 ` [PATCH 20/42] perf tools: Use thread__comm_time() when adding hist entries Namhyung Kim
@ 2015-01-29  8:07 ` Namhyung Kim
  2015-01-29  8:07 ` [PATCH 22/42] perf tools: Introduce machine__find*_thread_time() Namhyung Kim
                   ` (21 subsequent siblings)
  42 siblings, 0 replies; 221+ messages in thread
From: Namhyung Kim @ 2015-01-29  8:07 UTC (permalink / raw)
  To: Arnaldo Carvalho de Melo
  Cc: Ingo Molnar, Peter Zijlstra, Jiri Olsa, LKML, David Ahern,
	Adrian Hunter, Andi Kleen, Stephane Eranian, Frederic Weisbecker

Currently perf maintains dead threads in a linked list but this can be
a problem if someone needs to search from it especially in a large
session which might have many dead threads.  Convert it to a rbtree
like normal threads and it'll be used later with multi-file changes.

The list node is now used for chaining dead threads of same tid since
it's easier to handle such threads in time order.

Cc: Frederic Weisbecker <fweisbec@gmail.com>
Signed-off-by: Namhyung Kim <namhyung@kernel.org>
---
 tools/perf/util/machine.c | 70 +++++++++++++++++++++++++++++++++++++++++------
 tools/perf/util/machine.h |  2 +-
 tools/perf/util/thread.c  |  1 +
 tools/perf/util/thread.h  | 11 ++++----
 4 files changed, 68 insertions(+), 16 deletions(-)

diff --git a/tools/perf/util/machine.c b/tools/perf/util/machine.c
index 1bca3a9f2b16..d4050fcba851 100644
--- a/tools/perf/util/machine.c
+++ b/tools/perf/util/machine.c
@@ -28,7 +28,7 @@ int machine__init(struct machine *machine, const char *root_dir, pid_t pid)
 	dsos__init(&machine->kernel_dsos);
 
 	machine->threads = RB_ROOT;
-	INIT_LIST_HEAD(&machine->dead_threads);
+	machine->dead_threads = RB_ROOT;
 	machine->last_match = NULL;
 
 	machine->vdso_info = NULL;
@@ -91,10 +91,22 @@ static void dsos__delete(struct dsos *dsos)
 
 void machine__delete_dead_threads(struct machine *machine)
 {
-	struct thread *n, *t;
+	struct rb_node *nd = rb_first(&machine->dead_threads);
+
+	while (nd) {
+		struct thread *t = rb_entry(nd, struct thread, rb_node);
+		struct thread *pos;
+
+		nd = rb_next(nd);
+		rb_erase(&t->rb_node, &machine->dead_threads);
+
+		while (!list_empty(&t->tid_node)) {
+			pos = list_first_entry(&t->tid_node,
+					       struct thread, tid_node);
+			list_del(&pos->tid_node);
+			thread__delete(pos);
+		}
 
-	list_for_each_entry_safe(t, n, &machine->dead_threads, node) {
-		list_del(&t->node);
 		thread__delete(t);
 	}
 }
@@ -106,8 +118,8 @@ void machine__delete_threads(struct machine *machine)
 	while (nd) {
 		struct thread *t = rb_entry(nd, struct thread, rb_node);
 
-		rb_erase(&t->rb_node, &machine->threads);
 		nd = rb_next(nd);
+		rb_erase(&t->rb_node, &machine->threads);
 		thread__delete(t);
 	}
 }
@@ -1238,13 +1250,46 @@ int machine__process_mmap_event(struct machine *machine, union perf_event *event
 
 static void machine__remove_thread(struct machine *machine, struct thread *th)
 {
+	struct rb_node **p = &machine->dead_threads.rb_node;
+	struct rb_node *parent = NULL;
+	struct thread *pos;
+
 	machine->last_match = NULL;
 	rb_erase(&th->rb_node, &machine->threads);
+
+	th->dead = true;
+
 	/*
 	 * We may have references to this thread, for instance in some hist_entry
-	 * instances, so just move them to a separate list.
+	 * instances, so just move them to a separate list in rbtree.
 	 */
-	list_add_tail(&th->node, &machine->dead_threads);
+	while (*p != NULL) {
+		parent = *p;
+		pos = rb_entry(parent, struct thread, rb_node);
+
+		if (pos->tid == th->tid) {
+			struct thread *old;
+
+			/* sort by time */
+			list_for_each_entry(old, &pos->tid_node, tid_node) {
+				if (th->start_time >= old->start_time) {
+					pos = old;
+					break;
+				}
+			}
+
+			list_add_tail(&th->tid_node, &pos->tid_node);
+			return;
+		}
+
+		if (th->tid < pos->tid)
+			p = &(*p)->rb_left;
+		else
+			p = &(*p)->rb_right;
+	}
+
+	rb_link_node(&th->rb_node, parent, p);
+	rb_insert_color(&th->rb_node, &machine->dead_threads);
 }
 
 int machine__process_fork_event(struct machine *machine, union perf_event *event,
@@ -1649,7 +1694,7 @@ int machine__for_each_thread(struct machine *machine,
 			     void *priv)
 {
 	struct rb_node *nd;
-	struct thread *thread;
+	struct thread *thread, *pos;
 	int rc = 0;
 
 	for (nd = rb_first(&machine->threads); nd; nd = rb_next(nd)) {
@@ -1659,10 +1704,17 @@ int machine__for_each_thread(struct machine *machine,
 			return rc;
 	}
 
-	list_for_each_entry(thread, &machine->dead_threads, node) {
+	for (nd = rb_first(&machine->dead_threads); nd; nd = rb_next(nd)) {
+		thread = rb_entry(nd, struct thread, rb_node);
 		rc = fn(thread, priv);
 		if (rc != 0)
 			return rc;
+
+		list_for_each_entry(pos, &thread->tid_node, tid_node) {
+			rc = fn(pos, priv);
+			if (rc != 0)
+				return rc;
+		}
 	}
 	return rc;
 }
diff --git a/tools/perf/util/machine.h b/tools/perf/util/machine.h
index e8b7779a0a3f..4349946a38ff 100644
--- a/tools/perf/util/machine.h
+++ b/tools/perf/util/machine.h
@@ -30,7 +30,7 @@ struct machine {
 	bool		  comm_exec;
 	char		  *root_dir;
 	struct rb_root	  threads;
-	struct list_head  dead_threads;
+	struct rb_root	  dead_threads;
 	struct thread	  *last_match;
 	struct vdso_info  *vdso_info;
 	struct dsos	  user_dsos;
diff --git a/tools/perf/util/thread.c b/tools/perf/util/thread.c
index ad96725105c2..c9ae0e1599da 100644
--- a/tools/perf/util/thread.c
+++ b/tools/perf/util/thread.c
@@ -38,6 +38,7 @@ struct thread *thread__new(pid_t pid, pid_t tid)
 		thread->ppid = -1;
 		thread->cpu = -1;
 		INIT_LIST_HEAD(&thread->comm_list);
+		INIT_LIST_HEAD(&thread->tid_node);
 
 		if (unwind__prepare_access(thread) < 0)
 			goto err_thread;
diff --git a/tools/perf/util/thread.h b/tools/perf/util/thread.h
index be67c3bad5e7..21268e66b2ad 100644
--- a/tools/perf/util/thread.h
+++ b/tools/perf/util/thread.h
@@ -11,10 +11,8 @@
 struct thread_stack;
 
 struct thread {
-	union {
-		struct rb_node	 rb_node;
-		struct list_head node;
-	};
+	struct rb_node	 	rb_node;
+	struct list_head 	tid_node;
 	struct map_groups	*mg;
 	pid_t			pid_; /* Not all tools update this */
 	pid_t			tid;
@@ -22,7 +20,8 @@ struct thread {
 	int			cpu;
 	char			shortname[3];
 	bool			comm_set;
-	bool			dead; /* if set thread has exited */
+	bool			exited; /* if set thread has exited */
+	bool			dead; /* thread is in dead_threads list */
 	struct list_head	comm_list;
 	int			comm_len;
 	u64			db_id;
@@ -39,7 +38,7 @@ int thread__init_map_groups(struct thread *thread, struct machine *machine);
 void thread__delete(struct thread *thread);
 static inline void thread__exited(struct thread *thread)
 {
-	thread->dead = true;
+	thread->exited = true;
 }
 
 int __thread__set_comm(struct thread *thread, const char *comm, u64 timestamp,
-- 
2.2.2


^ permalink raw reply related	[flat|nested] 221+ messages in thread

* [PATCH 22/42] perf tools: Introduce machine__find*_thread_time()
  2015-01-29  8:06 [RFC/PATCHSET 00/42] perf tools: Speed-up perf report by using multi thread (v2) Namhyung Kim
                   ` (20 preceding siblings ...)
  2015-01-29  8:07 ` [PATCH 21/42] perf tools: Convert dead thread list into rbtree Namhyung Kim
@ 2015-01-29  8:07 ` Namhyung Kim
  2015-01-29  8:07 ` [PATCH 23/42] perf tools: Add a test case for timed thread handling Namhyung Kim
                   ` (20 subsequent siblings)
  42 siblings, 0 replies; 221+ messages in thread
From: Namhyung Kim @ 2015-01-29  8:07 UTC (permalink / raw)
  To: Arnaldo Carvalho de Melo
  Cc: Ingo Molnar, Peter Zijlstra, Jiri Olsa, LKML, David Ahern,
	Adrian Hunter, Andi Kleen, Stephane Eranian, Frederic Weisbecker

With data file indexing is enabled, it needs to search thread based on
sample time since sample processing is done after other (task, comm and
mmap) events are processed.  This can be a problem if a session is very
long and pid is recycled - in that case it'll only see the last one.

So keep thread start time in it, and search thread based on the time.
This patch introduces machine__find{,new}_thread_time() function for
this.  It'll first search current thread rbtree and then dead thread
tree and list.  If it couldn't find anyone, it'll create a new thread.

The sample timestamp of 0 means that this is called from synthesized
event so just use current rbtree.  The timestamp will be -1 if sample
didn't record the timestamp so will see current threads automatically.

Cc: Frederic Weisbecker <fweisbec@gmail.com>
Signed-off-by: Namhyung Kim <namhyung@kernel.org>
---
 tools/perf/builtin-script.c     |  11 ++++-
 tools/perf/tests/dwarf-unwind.c |   8 ++--
 tools/perf/tests/hists_common.c |   3 +-
 tools/perf/tests/hists_link.c   |   2 +-
 tools/perf/util/event.c         |  14 ++++--
 tools/perf/util/machine.c       | 102 +++++++++++++++++++++++++++++++++++++++-
 tools/perf/util/machine.h       |   8 +++-
 tools/perf/util/thread.c        |   4 ++
 tools/perf/util/thread.h        |   1 +
 9 files changed, 138 insertions(+), 15 deletions(-)

diff --git a/tools/perf/builtin-script.c b/tools/perf/builtin-script.c
index 4a007110d2f7..65b3a07be2bf 100644
--- a/tools/perf/builtin-script.c
+++ b/tools/perf/builtin-script.c
@@ -559,8 +559,15 @@ static int process_sample_event(struct perf_tool *tool,
 {
 	struct addr_location al;
 	struct perf_script *script = container_of(tool, struct perf_script, tool);
-	struct thread *thread = machine__findnew_thread(machine, sample->pid,
-							sample->tid);
+	struct thread *thread;
+
+	if (perf_session__has_index(script->session))
+		thread = machine__findnew_thread_time(machine, sample->pid,
+						      sample->tid,
+						      sample->time);
+	else
+		thread = machine__findnew_thread(machine, sample->pid,
+						 sample->tid);
 
 	if (thread == NULL) {
 		pr_debug("problem processing %d event, skipping it.\n",
diff --git a/tools/perf/tests/dwarf-unwind.c b/tools/perf/tests/dwarf-unwind.c
index 0bf06bec68c7..7e04feb431cb 100644
--- a/tools/perf/tests/dwarf-unwind.c
+++ b/tools/perf/tests/dwarf-unwind.c
@@ -16,10 +16,10 @@
 
 static int mmap_handler(struct perf_tool *tool __maybe_unused,
 			union perf_event *event,
-			struct perf_sample *sample __maybe_unused,
+			struct perf_sample *sample,
 			struct machine *machine)
 {
-	return machine__process_mmap2_event(machine, event, NULL);
+	return machine__process_mmap2_event(machine, event, sample);
 }
 
 static int init_live_machine(struct machine *machine)
@@ -66,12 +66,10 @@ static int unwind_entry(struct unwind_entry *entry, void *arg)
 __attribute__ ((noinline))
 static int unwind_thread(struct thread *thread)
 {
-	struct perf_sample sample;
+	struct perf_sample sample = { .time = -1ULL, };
 	unsigned long cnt = 0;
 	int err = -1;
 
-	memset(&sample, 0, sizeof(sample));
-
 	if (test__arch_unwind_sample(&sample, thread)) {
 		pr_debug("failed to get unwind sample\n");
 		goto out;
diff --git a/tools/perf/tests/hists_common.c b/tools/perf/tests/hists_common.c
index a62c09134516..86a8fdb41804 100644
--- a/tools/perf/tests/hists_common.c
+++ b/tools/perf/tests/hists_common.c
@@ -80,6 +80,7 @@ static struct {
 struct machine *setup_fake_machine(struct machines *machines)
 {
 	struct machine *machine = machines__find(machines, HOST_KERNEL_ID);
+	struct perf_sample sample = { .time = -1ULL, };
 	size_t i;
 
 	if (machine == NULL) {
@@ -113,7 +114,7 @@ struct machine *setup_fake_machine(struct machines *machines)
 		strcpy(fake_mmap_event.mmap.filename,
 		       fake_mmap_info[i].filename);
 
-		machine__process_mmap_event(machine, &fake_mmap_event, NULL);
+		machine__process_mmap_event(machine, &fake_mmap_event, &sample);
 	}
 
 	for (i = 0; i < ARRAY_SIZE(fake_symbols); i++) {
diff --git a/tools/perf/tests/hists_link.c b/tools/perf/tests/hists_link.c
index 4f3d45692acb..1237cc87e8d5 100644
--- a/tools/perf/tests/hists_link.c
+++ b/tools/perf/tests/hists_link.c
@@ -64,7 +64,7 @@ static int add_hist_entries(struct perf_evlist *evlist, struct machine *machine)
 	struct perf_evsel *evsel;
 	struct addr_location al;
 	struct hist_entry *he;
-	struct perf_sample sample = { .period = 1, };
+	struct perf_sample sample = { .period = 1, .time = -1ULL, };
 	size_t i = 0, k;
 
 	/*
diff --git a/tools/perf/util/event.c b/tools/perf/util/event.c
index 186960a09024..8b9fe0a908e8 100644
--- a/tools/perf/util/event.c
+++ b/tools/perf/util/event.c
@@ -9,6 +9,7 @@
 #include "strlist.h"
 #include "thread.h"
 #include "thread_map.h"
+#include "session.h"
 #include "symbol/kallsyms.h"
 
 static const char *perf_event__names[] = {
@@ -823,11 +824,18 @@ int perf_event__preprocess_sample(const union perf_event *event,
 				  struct machine *machine,
 				  struct addr_location *al,
 				  struct perf_sample *sample,
-				  struct perf_session *session __maybe_unused)
+				  struct perf_session *session)
 {
 	u8 cpumode = event->header.misc & PERF_RECORD_MISC_CPUMODE_MASK;
-	struct thread *thread = machine__findnew_thread(machine, sample->pid,
-							sample->tid);
+	struct thread *thread;
+
+	if (session && perf_session__has_index(session))
+		thread = machine__findnew_thread_time(machine, sample->pid,
+						      sample->tid,
+						      sample->time);
+	else
+		thread = machine__findnew_thread(machine, sample->pid,
+						 sample->tid);
 
 	if (thread == NULL)
 		return -1;
diff --git a/tools/perf/util/machine.c b/tools/perf/util/machine.c
index d4050fcba851..f8bc2f67b515 100644
--- a/tools/perf/util/machine.c
+++ b/tools/perf/util/machine.c
@@ -434,6 +434,106 @@ struct thread *machine__find_thread(struct machine *machine, pid_t pid,
 	return __machine__findnew_thread(machine, pid, tid, false);
 }
 
+static struct thread *__machine__findnew_thread_time(struct machine *machine,
+						     pid_t pid, pid_t tid,
+						     u64 timestamp, bool create)
+{
+	struct thread *curr, *pos, *new;
+	struct thread *th = NULL;
+	struct rb_node **p;
+	struct rb_node *parent = NULL;
+
+	curr = __machine__findnew_thread(machine, pid, tid, false);
+	if (curr && timestamp >= curr->start_time)
+		return curr;
+
+	p = &machine->dead_threads.rb_node;
+	while (*p != NULL) {
+		parent = *p;
+		th = rb_entry(parent, struct thread, rb_node);
+
+		if (th->tid == tid) {
+			list_for_each_entry(pos, &th->tid_node, tid_node) {
+				if (timestamp >= pos->start_time &&
+				    pos->start_time > th->start_time) {
+					th = pos;
+					break;
+				}
+			}
+
+			if (timestamp >= th->start_time) {
+				machine__update_thread_pid(machine, th, pid);
+				return th;
+			}
+			break;
+		}
+
+		if (tid < th->tid)
+			p = &(*p)->rb_left;
+		else
+			p = &(*p)->rb_right;
+	}
+
+	if (!create)
+		return NULL;
+
+	if (!curr && !*p)
+		return __machine__findnew_thread(machine, pid, tid, true);
+
+	new = thread__new(pid, tid);
+	if (new == NULL)
+		return NULL;
+
+	new->dead = true;
+	new->start_time = timestamp;
+
+	if (*p) {
+		list_for_each_entry(pos, &th->tid_node, tid_node) {
+			/* sort by time */
+			if (timestamp >= pos->start_time) {
+				th = pos;
+				break;
+			}
+		}
+		list_add_tail(&new->tid_node, &th->tid_node);
+	} else {
+		rb_link_node(&new->rb_node, parent, p);
+		rb_insert_color(&new->rb_node, &machine->dead_threads);
+	}
+
+	/*
+	 * We have to initialize map_groups separately
+	 * after rb tree is updated.
+	 *
+	 * The reason is that we call machine__findnew_thread
+	 * within thread__init_map_groups to find the thread
+	 * leader and that would screwed the rb tree.
+	 */
+	if (thread__init_map_groups(new, machine)) {
+		if (!list_empty(&new->tid_node))
+			list_del(&new->tid_node);
+		else
+			rb_erase(&new->rb_node, &machine->dead_threads);
+
+		thread__delete(new);
+		return NULL;
+	}
+
+	return new;
+}
+
+struct thread *machine__find_thread_time(struct machine *machine, pid_t pid,
+					 pid_t tid, u64 timestamp)
+{
+	return __machine__findnew_thread_time(machine, pid, tid, timestamp, false);
+}
+
+struct thread *machine__findnew_thread_time(struct machine *machine, pid_t pid,
+					    pid_t tid, u64 timestamp)
+{
+	return __machine__findnew_thread_time(machine, pid, tid, timestamp, true);
+}
+
 struct comm *machine__thread_exec_comm(struct machine *machine,
 				       struct thread *thread)
 {
@@ -1172,7 +1272,7 @@ int machine__process_mmap2_event(struct machine *machine,
 	}
 
 	thread = machine__findnew_thread(machine, event->mmap2.pid,
-					event->mmap2.tid);
+					 event->mmap2.tid);
 	if (thread == NULL)
 		goto out_problem;
 
diff --git a/tools/perf/util/machine.h b/tools/perf/util/machine.h
index 4349946a38ff..9571b6b1c5b5 100644
--- a/tools/perf/util/machine.h
+++ b/tools/perf/util/machine.h
@@ -68,8 +68,6 @@ static inline bool machine__kernel_ip(struct machine *machine, u64 ip)
 	return ip >= kernel_start;
 }
 
-struct thread *machine__find_thread(struct machine *machine, pid_t pid,
-				    pid_t tid);
 struct comm *machine__thread_exec_comm(struct machine *machine,
 				       struct thread *thread);
 
@@ -149,6 +147,12 @@ static inline bool machine__is_host(struct machine *machine)
 
 struct thread *machine__findnew_thread(struct machine *machine, pid_t pid,
 				       pid_t tid);
+struct thread *machine__find_thread(struct machine *machine, pid_t pid,
+				    pid_t tid);
+struct thread *machine__findnew_thread_time(struct machine *machine, pid_t pid,
+					    pid_t tid, u64 timestamp);
+struct thread *machine__find_thread_time(struct machine *machine, pid_t pid,
+					 pid_t tid, u64 timestamp);
 
 size_t machine__fprintf(struct machine *machine, FILE *fp);
 
diff --git a/tools/perf/util/thread.c b/tools/perf/util/thread.c
index c9ae0e1599da..306bdaede019 100644
--- a/tools/perf/util/thread.c
+++ b/tools/perf/util/thread.c
@@ -127,6 +127,9 @@ int __thread__set_comm(struct thread *thread, const char *str, u64 timestamp,
 
 	/* Override the default :tid entry */
 	if (!thread->comm_set) {
+		if (!thread->start_time)
+			thread->start_time = timestamp;
+
 		err = comm__override(curr, str, timestamp, exec);
 		if (err)
 			return err;
@@ -228,6 +231,7 @@ int thread__fork(struct thread *thread, struct thread *parent, u64 timestamp)
 	}
 
 	thread->ppid = parent->tid;
+	thread->start_time = timestamp;
 	return thread__clone_map_groups(thread, parent);
 }
 
diff --git a/tools/perf/util/thread.h b/tools/perf/util/thread.h
index 21268e66b2ad..e5d7abd255ea 100644
--- a/tools/perf/util/thread.h
+++ b/tools/perf/util/thread.h
@@ -25,6 +25,7 @@ struct thread {
 	struct list_head	comm_list;
 	int			comm_len;
 	u64			db_id;
+	u64			start_time;
 
 	void			*priv;
 	struct thread_stack	*ts;
-- 
2.2.2


^ permalink raw reply related	[flat|nested] 221+ messages in thread

* [PATCH 23/42] perf tools: Add a test case for timed thread handling
  2015-01-29  8:06 [RFC/PATCHSET 00/42] perf tools: Speed-up perf report by using multi thread (v2) Namhyung Kim
                   ` (21 preceding siblings ...)
  2015-01-29  8:07 ` [PATCH 22/42] perf tools: Introduce machine__find*_thread_time() Namhyung Kim
@ 2015-01-29  8:07 ` Namhyung Kim
  2015-01-29  8:07 ` [PATCH 24/42] perf tools: Maintain map groups list in a leader thread Namhyung Kim
                   ` (19 subsequent siblings)
  42 siblings, 0 replies; 221+ messages in thread
From: Namhyung Kim @ 2015-01-29  8:07 UTC (permalink / raw)
  To: Arnaldo Carvalho de Melo
  Cc: Ingo Molnar, Peter Zijlstra, Jiri Olsa, LKML, David Ahern,
	Adrian Hunter, Andi Kleen, Stephane Eranian, Frederic Weisbecker

A test case for verifying live and dead thread tree management during
time change and new machine__find{,new}_thread_time().

Cc: Frederic Weisbecker <fweisbec@gmail.com>
Signed-off-by: Namhyung Kim <namhyung@kernel.org>
---
 tools/perf/Makefile.perf              |   1 +
 tools/perf/tests/builtin-test.c       |   4 +
 tools/perf/tests/tests.h              |   1 +
 tools/perf/tests/thread-lookup-time.c | 174 ++++++++++++++++++++++++++++++++++
 4 files changed, 180 insertions(+)
 create mode 100644 tools/perf/tests/thread-lookup-time.c

diff --git a/tools/perf/Makefile.perf b/tools/perf/Makefile.perf
index 8507891db69d..8bb35ee91fd5 100644
--- a/tools/perf/Makefile.perf
+++ b/tools/perf/Makefile.perf
@@ -459,6 +459,7 @@ LIB_OBJS += $(OUTPUT)tests/mmap-thread-lookup.o
 LIB_OBJS += $(OUTPUT)tests/thread-mg-share.o
 LIB_OBJS += $(OUTPUT)tests/switch-tracking.o
 LIB_OBJS += $(OUTPUT)tests/thread-comm.o
+LIB_OBJS += $(OUTPUT)tests/thread-lookup-time.o
 
 BUILTIN_OBJS += $(OUTPUT)builtin-annotate.o
 BUILTIN_OBJS += $(OUTPUT)builtin-bench.o
diff --git a/tools/perf/tests/builtin-test.c b/tools/perf/tests/builtin-test.c
index 1b463d82a71a..e4d335de19ea 100644
--- a/tools/perf/tests/builtin-test.c
+++ b/tools/perf/tests/builtin-test.c
@@ -171,6 +171,10 @@ static struct test {
 		.func = test__thread_comm,
 	},
 	{
+		.desc = "Test thread lookup with time",
+		.func = test__thread_lookup_time,
+	},
+	{
 		.func = NULL,
 	},
 };
diff --git a/tools/perf/tests/tests.h b/tools/perf/tests/tests.h
index 43ac17780629..1090337f63e5 100644
--- a/tools/perf/tests/tests.h
+++ b/tools/perf/tests/tests.h
@@ -52,6 +52,7 @@ int test__switch_tracking(void);
 int test__fdarray__filter(void);
 int test__fdarray__add(void);
 int test__thread_comm(void);
+int test__thread_lookup_time(void);
 
 #if defined(__x86_64__) || defined(__i386__) || defined(__arm__)
 #ifdef HAVE_DWARF_UNWIND_SUPPORT
diff --git a/tools/perf/tests/thread-lookup-time.c b/tools/perf/tests/thread-lookup-time.c
new file mode 100644
index 000000000000..6237ecf8caae
--- /dev/null
+++ b/tools/perf/tests/thread-lookup-time.c
@@ -0,0 +1,174 @@
+#include "tests.h"
+#include "machine.h"
+#include "thread.h"
+#include "map.h"
+#include "debug.h"
+
+static int thread__print_cb(struct thread *th, void *arg __maybe_unused)
+{
+	printf("thread: %d, start time: %"PRIu64" %s\n",
+	       th->tid, th->start_time, th->dead ? "(dead)" : "");
+	return 0;
+}
+
+static int lookup_with_timestamp(struct machine *machine)
+{
+	struct thread *t1, *t2, *t3;
+	union perf_event fork = {
+		.fork = {
+			.pid = 0,
+			.tid = 0,
+			.ppid = 1,
+			.ptid = 1,
+		},
+	};
+	struct perf_sample sample = {
+		.time = 50000,
+	};
+
+	/* start_time is set to 0 */
+	t1 = machine__findnew_thread(machine, 0, 0);
+
+	if (verbose > 1) {
+		printf("========= after t1 created ==========\n");
+		machine__for_each_thread(machine, thread__print_cb, NULL);
+	}
+
+	TEST_ASSERT_VAL("wrong start time of old thread", t1->start_time == 0);
+
+	TEST_ASSERT_VAL("cannot find current thread",
+			machine__find_thread(machine, 0, 0) == t1);
+
+	TEST_ASSERT_VAL("cannot find current thread with time",
+			machine__findnew_thread_time(machine, 0, 0, 10000) == t1);
+
+	/* start_time is overwritten to new value */
+	thread__set_comm(t1, "/usr/bin/perf", 20000);
+
+	if (verbose > 1) {
+		printf("========= after t1 set comm ==========\n");
+		machine__for_each_thread(machine, thread__print_cb, NULL);
+	}
+
+	TEST_ASSERT_VAL("failed to update start time", t1->start_time == 20000);
+
+	TEST_ASSERT_VAL("should not find passed thread",
+			/* this will create yet another dead thread */
+			machine__findnew_thread_time(machine, 0, 0, 10000) != t1);
+
+	TEST_ASSERT_VAL("cannot find overwritten thread with time",
+			machine__find_thread_time(machine, 0, 0, 20000) == t1);
+
+	/* now t1 goes to dead thread tree, and create t2 */
+	machine__process_fork_event(machine, &fork, &sample);
+
+	if (verbose > 1) {
+		printf("========= after t2 forked ==========\n");
+		machine__for_each_thread(machine, thread__print_cb, NULL);
+	}
+
+	t2 = machine__find_thread(machine, 0, 0);
+	TEST_ASSERT_VAL("cannot find current thread", t2 != NULL);
+
+	TEST_ASSERT_VAL("wrong start time of new thread", t2->start_time == 50000);
+
+	TEST_ASSERT_VAL("dead thread cannot be found",
+			machine__find_thread_time(machine, 0, 0, 10000) != t1);
+
+	TEST_ASSERT_VAL("cannot find dead thread after new thread",
+			machine__find_thread_time(machine, 0, 0, 30000) == t1);
+
+	TEST_ASSERT_VAL("cannot find current thread after new thread",
+			machine__find_thread_time(machine, 0, 0, 50000) == t2);
+
+	/* now t2 goes to dead thread tree, and create t3 */
+	sample.time = 60000;
+	machine__process_fork_event(machine, &fork, &sample);
+
+	if (verbose > 1) {
+		printf("========= after t3 forked ==========\n");
+		machine__for_each_thread(machine, thread__print_cb, NULL);
+	}
+
+	t3 = machine__find_thread(machine, 0, 0);
+	TEST_ASSERT_VAL("cannot find current thread", t3 != NULL);
+
+	TEST_ASSERT_VAL("wrong start time of new thread", t3->start_time == 60000);
+
+	TEST_ASSERT_VAL("cannot find dead thread after new thread",
+			machine__findnew_thread_time(machine, 0, 0, 30000) == t1);
+
+	TEST_ASSERT_VAL("cannot find dead thread after new thread",
+			machine__findnew_thread_time(machine, 0, 0, 50000) == t2);
+
+	TEST_ASSERT_VAL("cannot find current thread after new thread",
+			machine__findnew_thread_time(machine, 0, 0, 70000) == t3);
+
+	machine__delete_threads(machine);
+	return 0;
+}
+
+static int lookup_without_timestamp(struct machine *machine)
+{
+	struct thread *t1, *t2, *t3;
+	union perf_event fork = {
+		.fork = {
+			.pid = 0,
+			.tid = 0,
+			.ppid = 1,
+			.ptid = 1,
+		},
+	};
+	struct perf_sample sample = {
+		.time = -1ULL,
+	};
+
+	t1 = machine__findnew_thread(machine, 0, 0);
+	TEST_ASSERT_VAL("cannot find current thread", t1 != NULL);
+
+	TEST_ASSERT_VAL("cannot find new thread with time",
+			machine__findnew_thread_time(machine, 0, 0, -1ULL) == t1);
+
+	machine__process_fork_event(machine, &fork, &sample);
+
+	t2 = machine__find_thread(machine, 0, 0);
+	TEST_ASSERT_VAL("cannot find current thread", t2 != NULL);
+
+	TEST_ASSERT_VAL("cannot find new thread with time",
+			machine__find_thread_time(machine, 0, 0, -1ULL) == t2);
+
+	machine__process_fork_event(machine, &fork, &sample);
+
+	t3 = machine__find_thread(machine, 0, 0);
+	TEST_ASSERT_VAL("cannot find current thread", t3 != NULL);
+
+	TEST_ASSERT_VAL("cannot find new thread with time",
+			machine__findnew_thread_time(machine, 0, 0, -1ULL) == t3);
+
+	machine__delete_threads(machine);
+	return 0;
+}
+
+int test__thread_lookup_time(void)
+{
+	struct machines machines;
+	struct machine *machine;
+
+	/*
+	 * This test is to check whether it can retrieve a correct
+	 * thread for a given time.  When multi-file data storage is
+	 * enabled, those task/comm/mmap events are processed first so
+	 * the later sample should find a matching thread properly.
+	 */
+	machines__init(&machines);
+	machine = &machines.host;
+
+	if (lookup_with_timestamp(machine) < 0)
+		return -1;
+
+	if (lookup_without_timestamp(machine) < 0)
+		return -1;
+
+	machines__exit(&machines);
+	return 0;
+}
-- 
2.2.2


^ permalink raw reply related	[flat|nested] 221+ messages in thread

* [PATCH 24/42] perf tools: Maintain map groups list in a leader thread
  2015-01-29  8:06 [RFC/PATCHSET 00/42] perf tools: Speed-up perf report by using multi thread (v2) Namhyung Kim
                   ` (22 preceding siblings ...)
  2015-01-29  8:07 ` [PATCH 23/42] perf tools: Add a test case for timed thread handling Namhyung Kim
@ 2015-01-29  8:07 ` Namhyung Kim
  2015-01-29  8:07 ` [PATCH 25/42] perf tools: Introduce thread__find_addr_location_time() and friends Namhyung Kim
                   ` (18 subsequent siblings)
  42 siblings, 0 replies; 221+ messages in thread
From: Namhyung Kim @ 2015-01-29  8:07 UTC (permalink / raw)
  To: Arnaldo Carvalho de Melo
  Cc: Ingo Molnar, Peter Zijlstra, Jiri Olsa, LKML, David Ahern,
	Adrian Hunter, Andi Kleen, Stephane Eranian, Frederic Weisbecker

To support multi-threaded perf report, we need to maintain time-sorted
map groups.  Add ->mg_list member to struct thread and sort the list
by time.  Now leader threads have one more refcnt for map groups in
the list so also update the thread-mg-share test case.

Currently only add a new map groups when an exec (comm) event is
received.

Cc: Frederic Weisbecker <fweisbec@gmail.com>
Signed-off-by: Namhyung Kim <namhyung@kernel.org>
---
 tools/perf/tests/thread-mg-share.c |  7 +++-
 tools/perf/util/event.c            |  2 +
 tools/perf/util/machine.c          |  2 +-
 tools/perf/util/map.c              |  1 +
 tools/perf/util/map.h              |  2 +
 tools/perf/util/thread.c           | 80 +++++++++++++++++++++++++++++++++++++-
 tools/perf/util/thread.h           |  3 ++
 7 files changed, 93 insertions(+), 4 deletions(-)

diff --git a/tools/perf/tests/thread-mg-share.c b/tools/perf/tests/thread-mg-share.c
index b028499dd3cf..8933e01d0549 100644
--- a/tools/perf/tests/thread-mg-share.c
+++ b/tools/perf/tests/thread-mg-share.c
@@ -23,6 +23,9 @@ int test__thread_mg_share(void)
 	 * with several threads and checks they properly share and
 	 * maintain map groups info (struct map_groups).
 	 *
+	 * Note that a leader thread has one more refcnt for its
+	 * (current) map groups.
+	 *
 	 * thread group (pid: 0, tids: 0, 1, 2, 3)
 	 * other  group (pid: 4, tids: 4, 5)
 	*/
@@ -43,7 +46,7 @@ int test__thread_mg_share(void)
 			leader && t1 && t2 && t3 && other);
 
 	mg = leader->mg;
-	TEST_ASSERT_VAL("wrong refcnt", mg->refcnt == 4);
+	TEST_ASSERT_VAL("wrong refcnt", mg->refcnt == 5);
 
 	/* test the map groups pointer is shared */
 	TEST_ASSERT_VAL("map groups don't match", mg == t1->mg);
@@ -59,7 +62,7 @@ int test__thread_mg_share(void)
 	TEST_ASSERT_VAL("failed to find other leader", other_leader);
 
 	other_mg = other->mg;
-	TEST_ASSERT_VAL("wrong refcnt", other_mg->refcnt == 2);
+	TEST_ASSERT_VAL("wrong refcnt", other_mg->refcnt == 3);
 
 	TEST_ASSERT_VAL("map groups don't match", other_mg == other_leader->mg);
 
diff --git a/tools/perf/util/event.c b/tools/perf/util/event.c
index 8b9fe0a908e8..1558a7085c7f 100644
--- a/tools/perf/util/event.c
+++ b/tools/perf/util/event.c
@@ -751,6 +751,8 @@ void thread__find_addr_map(struct thread *thread, u8 cpumode,
 		return;
 	}
 
+	BUG_ON(mg == NULL);
+
 	if (cpumode == PERF_RECORD_MISC_KERNEL && perf_host) {
 		al->level = 'k';
 		mg = &machine->kmaps;
diff --git a/tools/perf/util/machine.c b/tools/perf/util/machine.c
index f8bc2f67b515..09c2edccccd9 100644
--- a/tools/perf/util/machine.c
+++ b/tools/perf/util/machine.c
@@ -331,7 +331,7 @@ static void machine__update_thread_pid(struct machine *machine,
 		goto out_err;
 
 	if (!leader->mg)
-		leader->mg = map_groups__new(machine);
+		thread__set_map_groups(leader, map_groups__new(machine), 0);
 
 	if (!leader->mg)
 		goto out_err;
diff --git a/tools/perf/util/map.c b/tools/perf/util/map.c
index 62ca9f2607d5..f0c1e2a24fee 100644
--- a/tools/perf/util/map.c
+++ b/tools/perf/util/map.c
@@ -422,6 +422,7 @@ void map_groups__init(struct map_groups *mg, struct machine *machine)
 	}
 	mg->machine = machine;
 	mg->refcnt = 1;
+	mg->timestamp = 0;
 }
 
 static void maps__delete(struct rb_root *maps)
diff --git a/tools/perf/util/map.h b/tools/perf/util/map.h
index 0e42438b1e59..f33d49029ac0 100644
--- a/tools/perf/util/map.h
+++ b/tools/perf/util/map.h
@@ -61,7 +61,9 @@ struct map_groups {
 	struct rb_root	 maps[MAP__NR_TYPES];
 	struct list_head removed_maps[MAP__NR_TYPES];
 	struct machine	 *machine;
+	u64		 timestamp;
 	int		 refcnt;
+	struct list_head list;
 };
 
 struct map_groups *map_groups__new(struct machine *machine);
diff --git a/tools/perf/util/thread.c b/tools/perf/util/thread.c
index 306bdaede019..895c74683c81 100644
--- a/tools/perf/util/thread.c
+++ b/tools/perf/util/thread.c
@@ -10,13 +10,64 @@
 #include "comm.h"
 #include "unwind.h"
 
+struct map_groups *thread__get_map_groups(struct thread *thread, u64 timestamp)
+{
+	struct map_groups *mg;
+
+	list_for_each_entry(mg, &thread->mg_list, list)
+		if (timestamp >= mg->timestamp)
+			return mg;
+
+	return thread->mg;
+}
+
+int thread__set_map_groups(struct thread *thread, struct map_groups *mg,
+			   u64 timestamp)
+{
+	struct list_head *pos;
+	struct map_groups *old;
+
+	if (mg == NULL)
+		return -ENOMEM;
+
+	/*
+	 * Only a leader thread can have map groups list - others
+	 * reference it through map_groups__get.  This means the
+	 * leader thread will have one more refcnt than others.
+	 */
+	if (thread->tid != thread->pid_)
+		return -EINVAL;
+
+	if (thread->mg) {
+		BUG_ON(thread->mg->refcnt <= 1);
+		map_groups__put(thread->mg);
+	}
+
+	/* sort by time */
+	list_for_each(pos, &thread->mg_list) {
+		old = list_entry(pos, struct map_groups, list);
+		if (timestamp > old->timestamp)
+			break;
+	}
+
+	list_add_tail(&mg->list, pos);
+	mg->timestamp = timestamp;
+
+	/* set current ->mg to most recent one */
+	thread->mg = list_first_entry(&thread->mg_list, struct map_groups, list);
+	/* increase one more refcnt for current */
+	map_groups__get(thread->mg);
+
+	return 0;
+}
+
 int thread__init_map_groups(struct thread *thread, struct machine *machine)
 {
 	struct thread *leader;
 	pid_t pid = thread->pid_;
 
 	if (pid == thread->tid || pid == -1) {
-		thread->mg = map_groups__new(machine);
+		thread__set_map_groups(thread, map_groups__new(machine), 0);
 	} else {
 		leader = machine__findnew_thread(machine, pid, pid);
 		if (leader)
@@ -39,6 +90,7 @@ struct thread *thread__new(pid_t pid, pid_t tid)
 		thread->cpu = -1;
 		INIT_LIST_HEAD(&thread->comm_list);
 		INIT_LIST_HEAD(&thread->tid_node);
+		INIT_LIST_HEAD(&thread->mg_list);
 
 		if (unwind__prepare_access(thread) < 0)
 			goto err_thread;
@@ -67,6 +119,7 @@ struct thread *thread__new(pid_t pid, pid_t tid)
 void thread__delete(struct thread *thread)
 {
 	struct comm *comm, *tmp;
+	struct map_groups *mg, *tmp_mg;
 
 	thread_stack__free(thread);
 
@@ -74,6 +127,11 @@ void thread__delete(struct thread *thread)
 		map_groups__put(thread->mg);
 		thread->mg = NULL;
 	}
+	/* only leader threads have mg list */
+	list_for_each_entry_safe(mg, tmp_mg, &thread->mg_list, list) {
+		list_del(&mg->list);
+		map_groups__put(mg);
+	}
 	list_for_each_entry_safe(comm, tmp, &thread->comm_list, list) {
 		list_del(&comm->list);
 		comm__free(comm);
@@ -149,6 +207,26 @@ int __thread__set_comm(struct thread *thread, const char *str, u64 timestamp,
 			unwind__flush_access(thread);
 	}
 
+	if (exec) {
+		struct machine *machine;
+
+		BUG_ON(thread->mg == NULL || thread->mg->machine == NULL);
+
+		if (thread->tid != thread->pid_) {
+			/* now it'll be a new leader */
+			thread->pid_ = thread->tid;
+
+			/* current mg of leader thread needs one more refcnt */
+			map_groups__get(thread->mg);
+
+			thread__set_map_groups(thread, thread->mg,
+					       thread->mg->timestamp);
+		}
+
+		machine = thread->mg->machine;
+		thread__set_map_groups(thread, map_groups__new(machine), timestamp);
+	}
+
 	thread->comm_set = true;
 
 	return 0;
diff --git a/tools/perf/util/thread.h b/tools/perf/util/thread.h
index e5d7abd255ea..08cafa2d97f9 100644
--- a/tools/perf/util/thread.h
+++ b/tools/perf/util/thread.h
@@ -14,6 +14,7 @@ struct thread {
 	struct rb_node	 	rb_node;
 	struct list_head 	tid_node;
 	struct map_groups	*mg;
+	struct list_head	mg_list;
 	pid_t			pid_; /* Not all tools update this */
 	pid_t			tid;
 	pid_t			ppid;
@@ -56,6 +57,8 @@ struct comm *thread__exec_comm(const struct thread *thread);
 struct comm *thread__comm_time(const struct thread *thread, u64 timestamp);
 const char *thread__comm_str(const struct thread *thread);
 const char *thread__comm_str_time(const struct thread *thread, u64 timestamp);
+struct map_groups *thread__get_map_groups(struct thread *thread, u64 timestamp);
+int thread__set_map_groups(struct thread *thread, struct map_groups *mg, u64 timestamp);
 void thread__insert_map(struct thread *thread, struct map *map);
 int thread__fork(struct thread *thread, struct thread *parent, u64 timestamp);
 size_t thread__fprintf(struct thread *thread, FILE *fp);
-- 
2.2.2


^ permalink raw reply related	[flat|nested] 221+ messages in thread

* [PATCH 25/42] perf tools: Introduce thread__find_addr_location_time() and friends
  2015-01-29  8:06 [RFC/PATCHSET 00/42] perf tools: Speed-up perf report by using multi thread (v2) Namhyung Kim
                   ` (23 preceding siblings ...)
  2015-01-29  8:07 ` [PATCH 24/42] perf tools: Maintain map groups list in a leader thread Namhyung Kim
@ 2015-01-29  8:07 ` Namhyung Kim
  2015-01-29  8:07 ` [PATCH 26/42] perf tools: Add a test case for timed map groups handling Namhyung Kim
                   ` (17 subsequent siblings)
  42 siblings, 0 replies; 221+ messages in thread
From: Namhyung Kim @ 2015-01-29  8:07 UTC (permalink / raw)
  To: Arnaldo Carvalho de Melo
  Cc: Ingo Molnar, Peter Zijlstra, Jiri Olsa, LKML, David Ahern,
	Adrian Hunter, Andi Kleen, Stephane Eranian, Frederic Weisbecker

The *_time() variants are for find appropriate map (and symbol) at the
given time.  This is based on the fact that map_groups list is sorted
by time in the previous patch.

Cc: Frederic Weisbecker <fweisbec@gmail.com>
Signed-off-by: Namhyung Kim <namhyung@kernel.org>
---
 tools/perf/util/event.c            | 80 +++++++++++++++++++++++++++++++++-----
 tools/perf/util/machine.c          | 51 ++++++++++++++----------
 tools/perf/util/thread.c           | 21 ++++++++++
 tools/perf/util/thread.h           | 10 +++++
 tools/perf/util/unwind-libdw.c     | 11 +++---
 tools/perf/util/unwind-libunwind.c | 18 ++++-----
 6 files changed, 146 insertions(+), 45 deletions(-)

diff --git a/tools/perf/util/event.c b/tools/perf/util/event.c
index 1558a7085c7f..e7152a6e3043 100644
--- a/tools/perf/util/event.c
+++ b/tools/perf/util/event.c
@@ -732,16 +732,14 @@ int perf_event__process(struct perf_tool *tool __maybe_unused,
 	return machine__process_event(machine, event, sample);
 }
 
-void thread__find_addr_map(struct thread *thread, u8 cpumode,
-			   enum map_type type, u64 addr,
-			   struct addr_location *al)
+static void map_groups__find_addr_map(struct map_groups *mg, u8 cpumode,
+				      enum map_type type, u64 addr,
+				      struct addr_location *al)
 {
-	struct map_groups *mg = thread->mg;
 	struct machine *machine = mg->machine;
 	bool load_map = false;
 
 	al->machine = machine;
-	al->thread = thread;
 	al->addr = addr;
 	al->cpumode = cpumode;
 	al->filtered = 0;
@@ -810,6 +808,36 @@ void thread__find_addr_map(struct thread *thread, u8 cpumode,
 	}
 }
 
+void thread__find_addr_map(struct thread *thread, u8 cpumode,
+			   enum map_type type, u64 addr,
+			   struct addr_location *al)
+{
+	al->thread = thread;
+	map_groups__find_addr_map(thread->mg, cpumode, type, addr, al);
+}
+
+void thread__find_addr_map_time(struct thread *thread, u8 cpumode,
+				enum map_type type, u64 addr,
+				struct addr_location *al, u64 timestamp)
+{
+	struct map_groups *mg;
+	struct thread *leader;
+
+	if (thread->tid == thread->pid_)
+		leader = thread;
+	else
+		leader = machine__findnew_thread_time(thread->mg->machine,
+						      thread->pid_,
+						      thread->pid_,
+						      timestamp);
+	BUG_ON(leader == NULL);
+
+	mg = thread__get_map_groups(leader, timestamp);
+
+	al->thread = thread;
+	map_groups__find_addr_map(mg, cpumode, type, addr, al);
+}
+
 void thread__find_addr_location(struct thread *thread,
 				u8 cpumode, enum map_type type, u64 addr,
 				struct addr_location *al)
@@ -822,6 +850,21 @@ void thread__find_addr_location(struct thread *thread,
 		al->sym = NULL;
 }
 
+void thread__find_addr_location_time(struct thread *thread, u8 cpumode,
+				     enum map_type type, u64 addr,
+				     struct addr_location *al, u64 timestamp)
+{
+	struct map_groups *mg;
+
+	mg = thread__get_map_groups(thread, timestamp);
+	map_groups__find_addr_map(mg, cpumode, type, addr, al);
+	if (al->map != NULL)
+		al->sym = map__find_symbol(al->map, al->addr,
+					   mg->machine->symbol_filter);
+	else
+		al->sym = NULL;
+}
+
 int perf_event__preprocess_sample(const union perf_event *event,
 				  struct machine *machine,
 				  struct addr_location *al,
@@ -854,7 +897,13 @@ int perf_event__preprocess_sample(const union perf_event *event,
 	    machine->vmlinux_maps[MAP__FUNCTION] == NULL)
 		machine__create_kernel_maps(machine);
 
-	thread__find_addr_map(thread, cpumode, MAP__FUNCTION, sample->ip, al);
+	if (session && perf_session__has_index(session))
+		thread__find_addr_map_time(thread, cpumode, MAP__FUNCTION,
+					   sample->ip, al, sample->time);
+	else
+		thread__find_addr_map(thread, cpumode, MAP__FUNCTION,
+				      sample->ip, al);
+
 	dump_printf(" ...... dso: %s\n",
 		    al->map ? al->map->dso->long_name :
 			al->level == 'H' ? "[hypervisor]" : "<not found>");
@@ -915,15 +964,26 @@ void perf_event__preprocess_sample_addr(union perf_event *event,
 					struct perf_sample *sample,
 					struct thread *thread,
 					struct addr_location *al,
-					struct perf_session *session __maybe_unused)
+					struct perf_session *session)
 {
 	u8 cpumode = event->header.misc & PERF_RECORD_MISC_CPUMODE_MASK;
 
-	thread__find_addr_map(thread, cpumode, MAP__FUNCTION, sample->addr, al);
-	if (!al->map)
-		thread__find_addr_map(thread, cpumode, MAP__VARIABLE,
+	if (session && perf_session__has_index(session))
+		thread__find_addr_map_time(thread, cpumode, MAP__FUNCTION,
+					   sample->addr, al, sample->time);
+	else
+		thread__find_addr_map(thread, cpumode, MAP__FUNCTION,
 				      sample->addr, al);
 
+	if (!al->map) {
+		if (session && perf_session__has_index(session))
+			thread__find_addr_map_time(thread, cpumode, MAP__VARIABLE,
+						   sample->addr, al, sample->time);
+		else
+			thread__find_addr_map(thread, cpumode, MAP__VARIABLE,
+					      sample->addr, al);
+	}
+
 	al->cpu = sample->cpu;
 	al->sym = NULL;
 
diff --git a/tools/perf/util/machine.c b/tools/perf/util/machine.c
index 09c2edccccd9..7dc044b93cf8 100644
--- a/tools/perf/util/machine.c
+++ b/tools/perf/util/machine.c
@@ -1471,7 +1471,7 @@ static bool symbol__match_regex(struct symbol *sym, regex_t *regex)
 
 static void ip__resolve_ams(struct thread *thread,
 			    struct addr_map_symbol *ams,
-			    u64 ip)
+			    u64 ip, u64 timestamp)
 {
 	struct addr_location al;
 
@@ -1483,7 +1483,8 @@ static void ip__resolve_ams(struct thread *thread,
 	 * Thus, we have to try consecutively until we find a match
 	 * or else, the symbol is unknown
 	 */
-	thread__find_cpumode_addr_location(thread, MAP__FUNCTION, ip, &al);
+	thread__find_cpumode_addr_location_time(thread, MAP__FUNCTION, ip, &al,
+						timestamp);
 
 	ams->addr = ip;
 	ams->al_addr = al.addr;
@@ -1491,21 +1492,24 @@ static void ip__resolve_ams(struct thread *thread,
 	ams->map = al.map;
 }
 
-static void ip__resolve_data(struct thread *thread,
-			     u8 m, struct addr_map_symbol *ams, u64 addr)
+static void ip__resolve_data(struct thread *thread, u8 m,
+			     struct addr_map_symbol *ams,
+			     u64 addr, u64 timestamp)
 {
 	struct addr_location al;
 
 	memset(&al, 0, sizeof(al));
 
-	thread__find_addr_location(thread, m, MAP__VARIABLE, addr, &al);
+	thread__find_addr_location_time(thread, m, MAP__VARIABLE, addr,
+					&al, timestamp);
 	if (al.map == NULL) {
 		/*
 		 * some shared data regions have execute bit set which puts
 		 * their mapping in the MAP__FUNCTION type array.
 		 * Check there as a fallback option before dropping the sample.
 		 */
-		thread__find_addr_location(thread, m, MAP__FUNCTION, addr, &al);
+		thread__find_addr_location_time(thread, m, MAP__FUNCTION, addr,
+						&al, timestamp);
 	}
 
 	ams->addr = addr;
@@ -1522,8 +1526,9 @@ struct mem_info *sample__resolve_mem(struct perf_sample *sample,
 	if (!mi)
 		return NULL;
 
-	ip__resolve_ams(al->thread, &mi->iaddr, sample->ip);
-	ip__resolve_data(al->thread, al->cpumode, &mi->daddr, sample->addr);
+	ip__resolve_ams(al->thread, &mi->iaddr, sample->ip, sample->time);
+	ip__resolve_data(al->thread, al->cpumode, &mi->daddr, sample->addr,
+			 sample->time);
 	mi->data_src.val = sample->data_src;
 
 	return mi;
@@ -1533,15 +1538,16 @@ static int add_callchain_ip(struct thread *thread,
 			    struct symbol **parent,
 			    struct addr_location *root_al,
 			    bool branch_history,
-			    u64 ip)
+			    u64 ip, u64 timestamp)
 {
 	struct addr_location al;
 
 	al.filtered = 0;
 	al.sym = NULL;
+
 	if (branch_history)
-		thread__find_cpumode_addr_location(thread, MAP__FUNCTION,
-						   ip, &al);
+		thread__find_cpumode_addr_location_time(thread, MAP__FUNCTION,
+							ip, &al, timestamp);
 	else {
 		u8 cpumode = PERF_RECORD_MISC_USER;
 
@@ -1568,8 +1574,8 @@ static int add_callchain_ip(struct thread *thread,
 			}
 			return 0;
 		}
-		thread__find_addr_location(thread, cpumode, MAP__FUNCTION,
-				   ip, &al);
+		thread__find_addr_location_time(thread, cpumode, MAP__FUNCTION,
+						ip, &al, timestamp);
 	}
 
 	if (al.sym != NULL) {
@@ -1599,8 +1605,10 @@ struct branch_info *sample__resolve_bstack(struct perf_sample *sample,
 		return NULL;
 
 	for (i = 0; i < bs->nr; i++) {
-		ip__resolve_ams(al->thread, &bi[i].to, bs->entries[i].to);
-		ip__resolve_ams(al->thread, &bi[i].from, bs->entries[i].from);
+		ip__resolve_ams(al->thread, &bi[i].to, bs->entries[i].to,
+				sample->time);
+		ip__resolve_ams(al->thread, &bi[i].from, bs->entries[i].from,
+				sample->time);
 		bi[i].flags = bs->entries[i].flags;
 	}
 	return bi;
@@ -1652,7 +1660,7 @@ static int thread__resolve_callchain_sample(struct thread *thread,
 					     struct branch_stack *branch,
 					     struct symbol **parent,
 					     struct addr_location *root_al,
-					     int max_stack)
+					     int max_stack, u64 timestamp)
 {
 	int chain_nr = min(max_stack, (int)chain->nr);
 	int i, j, err;
@@ -1713,10 +1721,10 @@ static int thread__resolve_callchain_sample(struct thread *thread,
 
 		for (i = 0; i < nr; i++) {
 			err = add_callchain_ip(thread, parent, root_al,
-					       true, be[i].to);
+					       true, be[i].to, timestamp);
 			if (!err)
 				err = add_callchain_ip(thread, parent, root_al,
-						       true, be[i].from);
+						true, be[i].from, timestamp);
 			if (err == -EINVAL)
 				break;
 			if (err)
@@ -1745,8 +1753,8 @@ static int thread__resolve_callchain_sample(struct thread *thread,
 #endif
 		ip = chain->ips[j];
 
-		err = add_callchain_ip(thread, parent, root_al, false, ip);
-
+		err = add_callchain_ip(thread, parent, root_al, false, ip,
+				       timestamp);
 		if (err)
 			return (err < 0) ? err : 0;
 	}
@@ -1770,7 +1778,8 @@ int thread__resolve_callchain(struct thread *thread,
 {
 	int ret = thread__resolve_callchain_sample(thread, sample->callchain,
 						   sample->branch_stack,
-						   parent, root_al, max_stack);
+						   parent, root_al, max_stack,
+						   sample->time);
 	if (ret)
 		return ret;
 
diff --git a/tools/perf/util/thread.c b/tools/perf/util/thread.c
index 895c74683c81..293157dafd2c 100644
--- a/tools/perf/util/thread.c
+++ b/tools/perf/util/thread.c
@@ -331,3 +331,24 @@ void thread__find_cpumode_addr_location(struct thread *thread,
 			break;
 	}
 }
+
+void thread__find_cpumode_addr_location_time(struct thread *thread,
+					     enum map_type type, u64 addr,
+					     struct addr_location *al,
+					     u64 timestamp)
+{
+	size_t i;
+	const u8 const cpumodes[] = {
+		PERF_RECORD_MISC_USER,
+		PERF_RECORD_MISC_KERNEL,
+		PERF_RECORD_MISC_GUEST_USER,
+		PERF_RECORD_MISC_GUEST_KERNEL
+	};
+
+	for (i = 0; i < ARRAY_SIZE(cpumodes); i++) {
+		thread__find_addr_location_time(thread, cpumodes[i], type,
+						addr, al, timestamp);
+		if (al->map)
+			break;
+	}
+}
diff --git a/tools/perf/util/thread.h b/tools/perf/util/thread.h
index 08cafa2d97f9..5209ad5adadf 100644
--- a/tools/perf/util/thread.h
+++ b/tools/perf/util/thread.h
@@ -66,14 +66,24 @@ size_t thread__fprintf(struct thread *thread, FILE *fp);
 void thread__find_addr_map(struct thread *thread,
 			   u8 cpumode, enum map_type type, u64 addr,
 			   struct addr_location *al);
+void thread__find_addr_map_time(struct thread *thread, u8 cpumode,
+				enum map_type type, u64 addr,
+				struct addr_location *al, u64 timestamp);
 
 void thread__find_addr_location(struct thread *thread,
 				u8 cpumode, enum map_type type, u64 addr,
 				struct addr_location *al);
+void thread__find_addr_location_time(struct thread *thread, u8 cpumode,
+				     enum map_type type, u64 addr,
+				     struct addr_location *al, u64 timestamp);
 
 void thread__find_cpumode_addr_location(struct thread *thread,
 					enum map_type type, u64 addr,
 					struct addr_location *al);
+void thread__find_cpumode_addr_location_time(struct thread *thread,
+					     enum map_type type, u64 addr,
+					     struct addr_location *al,
+					     u64 timestamp);
 
 static inline void *thread__priv(struct thread *thread)
 {
diff --git a/tools/perf/util/unwind-libdw.c b/tools/perf/util/unwind-libdw.c
index 2dcfe9a7c8d0..ba8d8e41d680 100644
--- a/tools/perf/util/unwind-libdw.c
+++ b/tools/perf/util/unwind-libdw.c
@@ -26,9 +26,10 @@ static int __report_module(struct addr_location *al, u64 ip,
 	Dwfl_Module *mod;
 	struct dso *dso = NULL;
 
-	thread__find_addr_location(ui->thread,
-				   PERF_RECORD_MISC_USER,
-				   MAP__FUNCTION, ip, al);
+	thread__find_addr_location_time(ui->thread,
+					PERF_RECORD_MISC_USER,
+					MAP__FUNCTION, ip, al,
+					ui->sample->time);
 
 	if (al->map)
 		dso = al->map->dso;
@@ -89,8 +90,8 @@ static int access_dso_mem(struct unwind_info *ui, Dwarf_Addr addr,
 	struct addr_location al;
 	ssize_t size;
 
-	thread__find_addr_map(ui->thread, PERF_RECORD_MISC_USER,
-			      MAP__FUNCTION, addr, &al);
+	thread__find_addr_map_time(ui->thread, PERF_RECORD_MISC_USER,
+				   MAP__FUNCTION, addr, &al, ui->sample->time);
 	if (!al.map) {
 		pr_debug("unwind: no map for %lx\n", (unsigned long)addr);
 		return -1;
diff --git a/tools/perf/util/unwind-libunwind.c b/tools/perf/util/unwind-libunwind.c
index 6edf535f65c2..7ed6eaf232b6 100644
--- a/tools/perf/util/unwind-libunwind.c
+++ b/tools/perf/util/unwind-libunwind.c
@@ -306,8 +306,8 @@ static struct map *find_map(unw_word_t ip, struct unwind_info *ui)
 {
 	struct addr_location al;
 
-	thread__find_addr_map(ui->thread, PERF_RECORD_MISC_USER,
-			      MAP__FUNCTION, ip, &al);
+	thread__find_addr_map_time(ui->thread, PERF_RECORD_MISC_USER,
+				   MAP__FUNCTION, ip, &al, ui->sample->time);
 	return al.map;
 }
 
@@ -400,8 +400,8 @@ static int access_dso_mem(struct unwind_info *ui, unw_word_t addr,
 	struct addr_location al;
 	ssize_t size;
 
-	thread__find_addr_map(ui->thread, PERF_RECORD_MISC_USER,
-			      MAP__FUNCTION, addr, &al);
+	thread__find_addr_map_time(ui->thread, PERF_RECORD_MISC_USER,
+				   MAP__FUNCTION, addr, &al, ui->sample->time);
 	if (!al.map) {
 		pr_debug("unwind: no map for %lx\n", (unsigned long)addr);
 		return -1;
@@ -502,14 +502,14 @@ static void put_unwind_info(unw_addr_space_t __maybe_unused as,
 	pr_debug("unwind: put_unwind_info called\n");
 }
 
-static int entry(u64 ip, struct thread *thread,
+static int entry(u64 ip, struct thread *thread, u64 timestamp,
 		 unwind_entry_cb_t cb, void *arg)
 {
 	struct unwind_entry e;
 	struct addr_location al;
 
-	thread__find_addr_location(thread, PERF_RECORD_MISC_USER,
-				   MAP__FUNCTION, ip, &al);
+	thread__find_addr_location_time(thread, PERF_RECORD_MISC_USER,
+					MAP__FUNCTION, ip, &al, timestamp);
 
 	e.ip = ip;
 	e.map = al.map;
@@ -611,7 +611,7 @@ static int get_entries(struct unwind_info *ui, unwind_entry_cb_t cb,
 		unw_word_t ip;
 
 		unw_get_reg(&c, UNW_REG_IP, &ip);
-		ret = ip ? entry(ip, ui->thread, cb, arg) : 0;
+		ret = ip ? entry(ip, ui->thread, ui->sample->time, cb, arg) : 0;
 	}
 
 	return ret;
@@ -636,7 +636,7 @@ int unwind__get_entries(unwind_entry_cb_t cb, void *arg,
 	if (ret)
 		return ret;
 
-	ret = entry(ip, thread, cb, arg);
+	ret = entry(ip, thread, data->time, cb, arg);
 	if (ret)
 		return -ENOMEM;
 
-- 
2.2.2


^ permalink raw reply related	[flat|nested] 221+ messages in thread

* [PATCH 26/42] perf tools: Add a test case for timed map groups handling
  2015-01-29  8:06 [RFC/PATCHSET 00/42] perf tools: Speed-up perf report by using multi thread (v2) Namhyung Kim
                   ` (24 preceding siblings ...)
  2015-01-29  8:07 ` [PATCH 25/42] perf tools: Introduce thread__find_addr_location_time() and friends Namhyung Kim
@ 2015-01-29  8:07 ` Namhyung Kim
  2015-01-29  8:07 ` [PATCH 27/42] perf tools: Protect dso symbol loading using a mutex Namhyung Kim
                   ` (16 subsequent siblings)
  42 siblings, 0 replies; 221+ messages in thread
From: Namhyung Kim @ 2015-01-29  8:07 UTC (permalink / raw)
  To: Arnaldo Carvalho de Melo
  Cc: Ingo Molnar, Peter Zijlstra, Jiri Olsa, LKML, David Ahern,
	Adrian Hunter, Andi Kleen, Stephane Eranian, Frederic Weisbecker

A test case for verifying thread->mg and ->mg_list handling during
time change and new thread__find_addr_map_time() and friends.

Cc: Frederic Weisbecker <fweisbec@gmail.com>
Signed-off-by: Namhyung Kim <namhyung@kernel.org>
---
 tools/perf/Makefile.perf          |  1 +
 tools/perf/tests/builtin-test.c   |  4 ++
 tools/perf/tests/tests.h          |  1 +
 tools/perf/tests/thread-mg-time.c | 88 +++++++++++++++++++++++++++++++++++++++
 4 files changed, 94 insertions(+)
 create mode 100644 tools/perf/tests/thread-mg-time.c

diff --git a/tools/perf/Makefile.perf b/tools/perf/Makefile.perf
index 8bb35ee91fd5..2f8c8b918cac 100644
--- a/tools/perf/Makefile.perf
+++ b/tools/perf/Makefile.perf
@@ -460,6 +460,7 @@ LIB_OBJS += $(OUTPUT)tests/thread-mg-share.o
 LIB_OBJS += $(OUTPUT)tests/switch-tracking.o
 LIB_OBJS += $(OUTPUT)tests/thread-comm.o
 LIB_OBJS += $(OUTPUT)tests/thread-lookup-time.o
+LIB_OBJS += $(OUTPUT)tests/thread-mg-time.o
 
 BUILTIN_OBJS += $(OUTPUT)builtin-annotate.o
 BUILTIN_OBJS += $(OUTPUT)builtin-bench.o
diff --git a/tools/perf/tests/builtin-test.c b/tools/perf/tests/builtin-test.c
index e4d335de19ea..8f61a7e291ee 100644
--- a/tools/perf/tests/builtin-test.c
+++ b/tools/perf/tests/builtin-test.c
@@ -175,6 +175,10 @@ static struct test {
 		.func = test__thread_lookup_time,
 	},
 	{
+		.desc = "Test thread map group handling with time",
+		.func = test__thread_mg_time,
+	},
+	{
 		.func = NULL,
 	},
 };
diff --git a/tools/perf/tests/tests.h b/tools/perf/tests/tests.h
index 1090337f63e5..03557563f31d 100644
--- a/tools/perf/tests/tests.h
+++ b/tools/perf/tests/tests.h
@@ -53,6 +53,7 @@ int test__fdarray__filter(void);
 int test__fdarray__add(void);
 int test__thread_comm(void);
 int test__thread_lookup_time(void);
+int test__thread_mg_time(void);
 
 #if defined(__x86_64__) || defined(__i386__) || defined(__arm__)
 #ifdef HAVE_DWARF_UNWIND_SUPPORT
diff --git a/tools/perf/tests/thread-mg-time.c b/tools/perf/tests/thread-mg-time.c
new file mode 100644
index 000000000000..69fd13752c1d
--- /dev/null
+++ b/tools/perf/tests/thread-mg-time.c
@@ -0,0 +1,88 @@
+#include "tests.h"
+#include "machine.h"
+#include "thread.h"
+#include "map.h"
+#include "debug.h"
+
+#define PERF_MAP_START  0x40000
+
+int test__thread_mg_time(void)
+{
+	struct machines machines;
+	struct machine *machine;
+	struct thread *t;
+	struct map_groups *mg;
+	struct map *map;
+	struct addr_location al = { .map = NULL, };
+
+	/*
+	 * This test is to check whether it can retrieve a correct map
+	 * for a given time.  When multi-file data storage is enabled,
+	 * those task/comm/mmap events are processed first so the
+	 * later sample should find a matching comm properly.
+	 */
+	machines__init(&machines);
+	machine = &machines.host;
+
+	t = machine__findnew_thread(machine, 0, 0);
+	mg = t->mg;
+
+	map = dso__new_map("/usr/bin/perf");
+	map->start = PERF_MAP_START;
+	map->end = PERF_MAP_START + 0x1000;
+
+	thread__insert_map(t, map);
+
+	if (verbose > 1)
+		map_groups__fprintf(t->mg, stderr);
+
+	thread__find_addr_map(t, PERF_RECORD_MISC_USER, MAP__FUNCTION,
+			      PERF_MAP_START, &al);
+
+	TEST_ASSERT_VAL("cannot find mapping for perf", al.map != NULL);
+	TEST_ASSERT_VAL("non matched mapping found", al.map == map);
+	TEST_ASSERT_VAL("incorrect map groups", al.map->groups == mg);
+	TEST_ASSERT_VAL("incorrect map groups", al.map->groups == t->mg);
+
+	thread__find_addr_map_time(t, PERF_RECORD_MISC_USER,
+				   MAP__FUNCTION, PERF_MAP_START, &al, -1ULL);
+
+	TEST_ASSERT_VAL("cannot find timed mapping for perf", al.map != NULL);
+	TEST_ASSERT_VAL("non matched timed mapping", al.map == map);
+	TEST_ASSERT_VAL("incorrect timed map groups", al.map->groups == mg);
+	TEST_ASSERT_VAL("incorrect map groups", al.map->groups == t->mg);
+
+
+	pr_debug("simulate EXEC event (generate new mg)\n");
+	__thread__set_comm(t, "perf-test", 10000, true);
+
+	map = dso__new_map("/usr/bin/perf-test");
+	map->start = PERF_MAP_START;
+	map->end = PERF_MAP_START + 0x2000;
+
+	thread__insert_map(t, map);
+
+	if (verbose > 1)
+		map_groups__fprintf(t->mg, stderr);
+
+	thread__find_addr_map(t, PERF_RECORD_MISC_USER, MAP__FUNCTION,
+			      PERF_MAP_START + 4, &al);
+
+	TEST_ASSERT_VAL("cannot find mapping for perf-test", al.map != NULL);
+	TEST_ASSERT_VAL("invalid mapping found", al.map == map);
+	TEST_ASSERT_VAL("incorrect map groups", al.map->groups != mg);
+	TEST_ASSERT_VAL("incorrect map groups", al.map->groups == t->mg);
+
+	pr_debug("searching map in the old mag groups\n");
+	thread__find_addr_map_time(t, PERF_RECORD_MISC_USER,
+				   MAP__FUNCTION, PERF_MAP_START, &al, 5000);
+
+	TEST_ASSERT_VAL("cannot find timed mapping for perf-test", al.map != NULL);
+	TEST_ASSERT_VAL("non matched timed mapping", al.map != map);
+	TEST_ASSERT_VAL("incorrect timed map groups", al.map->groups == mg);
+	TEST_ASSERT_VAL("incorrect map groups", al.map->groups != t->mg);
+
+	machine__delete_threads(machine);
+	machines__exit(&machines);
+	return 0;
+}
-- 
2.2.2


^ permalink raw reply related	[flat|nested] 221+ messages in thread

* [PATCH 27/42] perf tools: Protect dso symbol loading using a mutex
  2015-01-29  8:06 [RFC/PATCHSET 00/42] perf tools: Speed-up perf report by using multi thread (v2) Namhyung Kim
                   ` (25 preceding siblings ...)
  2015-01-29  8:07 ` [PATCH 26/42] perf tools: Add a test case for timed map groups handling Namhyung Kim
@ 2015-01-29  8:07 ` Namhyung Kim
  2015-01-29 12:34   ` Arnaldo Carvalho de Melo
  2015-01-29  8:07 ` [PATCH 28/42] perf tools: Protect dso cache tree using dso->lock Namhyung Kim
                   ` (15 subsequent siblings)
  42 siblings, 1 reply; 221+ messages in thread
From: Namhyung Kim @ 2015-01-29  8:07 UTC (permalink / raw)
  To: Arnaldo Carvalho de Melo
  Cc: Ingo Molnar, Peter Zijlstra, Jiri Olsa, LKML, David Ahern,
	Adrian Hunter, Andi Kleen, Stephane Eranian, Frederic Weisbecker

When multi-thread support for perf report is enabled, it's possible to
access a dso concurrently.  Add a new pthread_mutex to protect it from
concurrent dso__load().

Signed-off-by: Namhyung Kim <namhyung@kernel.org>
---
 tools/perf/util/dso.c    |  2 ++
 tools/perf/util/dso.h    |  1 +
 tools/perf/util/symbol.c | 34 ++++++++++++++++++++++++----------
 3 files changed, 27 insertions(+), 10 deletions(-)

diff --git a/tools/perf/util/dso.c b/tools/perf/util/dso.c
index 45be944d450a..3da75816b8f8 100644
--- a/tools/perf/util/dso.c
+++ b/tools/perf/util/dso.c
@@ -888,6 +888,7 @@ struct dso *dso__new(const char *name)
 		RB_CLEAR_NODE(&dso->rb_node);
 		INIT_LIST_HEAD(&dso->node);
 		INIT_LIST_HEAD(&dso->data.open_entry);
+		pthread_mutex_init(&dso->lock, NULL);
 	}
 
 	return dso;
@@ -917,6 +918,7 @@ void dso__delete(struct dso *dso)
 	dso_cache__free(&dso->data.cache);
 	dso__free_a2l(dso);
 	zfree(&dso->symsrc_filename);
+	pthread_mutex_destroy(&dso->lock);
 	free(dso);
 }
 
diff --git a/tools/perf/util/dso.h b/tools/perf/util/dso.h
index 3782c82c6e44..ac753594a469 100644
--- a/tools/perf/util/dso.h
+++ b/tools/perf/util/dso.h
@@ -102,6 +102,7 @@ struct dsos {
 };
 
 struct dso {
+	pthread_mutex_t	 lock;
 	struct list_head node;
 	struct rb_node	 rb_node;	/* rbtree node sorted by long name */
 	struct rb_root	 symbols[MAP__NR_TYPES];
diff --git a/tools/perf/util/symbol.c b/tools/perf/util/symbol.c
index a69066865a55..714e20c99354 100644
--- a/tools/perf/util/symbol.c
+++ b/tools/perf/util/symbol.c
@@ -1357,12 +1357,22 @@ int dso__load(struct dso *dso, struct map *map, symbol_filter_t filter)
 	struct symsrc *syms_ss = NULL, *runtime_ss = NULL;
 	bool kmod;
 
-	dso__set_loaded(dso, map->type);
+	pthread_mutex_lock(&dso->lock);
+
+	/* check again under the dso->lock */
+	if (dso__loaded(dso, map->type)) {
+		ret = 1;
+		goto out;
+	}
+
+	if (dso->kernel) {
+		if (dso->kernel == DSO_TYPE_KERNEL)
+			ret = dso__load_kernel_sym(dso, map, filter);
+		else if (dso->kernel == DSO_TYPE_GUEST_KERNEL)
+			ret = dso__load_guest_kernel_sym(dso, map, filter);
 
-	if (dso->kernel == DSO_TYPE_KERNEL)
-		return dso__load_kernel_sym(dso, map, filter);
-	else if (dso->kernel == DSO_TYPE_GUEST_KERNEL)
-		return dso__load_guest_kernel_sym(dso, map, filter);
+		goto out;
+	}
 
 	if (map->groups && map->groups->machine)
 		machine = map->groups->machine;
@@ -1375,18 +1385,18 @@ int dso__load(struct dso *dso, struct map *map, symbol_filter_t filter)
 		struct stat st;
 
 		if (lstat(dso->name, &st) < 0)
-			return -1;
+			goto out;
 
 		if (st.st_uid && (st.st_uid != geteuid())) {
 			pr_warning("File %s not owned by current user or root, "
 				"ignoring it.\n", dso->name);
-			return -1;
+			goto out;
 		}
 
 		ret = dso__load_perf_map(dso, map, filter);
 		dso->symtab_type = ret > 0 ? DSO_BINARY_TYPE__JAVA_JIT :
 					     DSO_BINARY_TYPE__NOT_FOUND;
-		return ret;
+		goto out;
 	}
 
 	if (machine)
@@ -1394,7 +1404,7 @@ int dso__load(struct dso *dso, struct map *map, symbol_filter_t filter)
 
 	name = malloc(PATH_MAX);
 	if (!name)
-		return -1;
+		goto out;
 
 	kmod = dso->symtab_type == DSO_BINARY_TYPE__SYSTEM_PATH_KMODULE ||
 		dso->symtab_type == DSO_BINARY_TYPE__SYSTEM_PATH_KMODULE_COMP ||
@@ -1475,7 +1485,11 @@ int dso__load(struct dso *dso, struct map *map, symbol_filter_t filter)
 out_free:
 	free(name);
 	if (ret < 0 && strstr(dso->name, " (deleted)") != NULL)
-		return 0;
+		ret = 0;
+out:
+	dso__set_loaded(dso, map->type);
+	pthread_mutex_unlock(&dso->lock);
+
 	return ret;
 }
 
-- 
2.2.2


^ permalink raw reply related	[flat|nested] 221+ messages in thread

* [PATCH 28/42] perf tools: Protect dso cache tree using dso->lock
  2015-01-29  8:06 [RFC/PATCHSET 00/42] perf tools: Speed-up perf report by using multi thread (v2) Namhyung Kim
                   ` (26 preceding siblings ...)
  2015-01-29  8:07 ` [PATCH 27/42] perf tools: Protect dso symbol loading using a mutex Namhyung Kim
@ 2015-01-29  8:07 ` Namhyung Kim
  2015-01-29  8:07 ` [PATCH 29/42] perf tools: Protect dso cache fd with a mutex Namhyung Kim
                   ` (14 subsequent siblings)
  42 siblings, 0 replies; 221+ messages in thread
From: Namhyung Kim @ 2015-01-29  8:07 UTC (permalink / raw)
  To: Arnaldo Carvalho de Melo
  Cc: Ingo Molnar, Peter Zijlstra, Jiri Olsa, LKML, David Ahern,
	Adrian Hunter, Andi Kleen, Stephane Eranian, Frederic Weisbecker

The dso cache is accessed during dwarf callchain unwind and it might
be processed concurrently when multi-thread report is enabled.
Protect it under dso->lock.

Note that it doesn't protect dso_cache__find().  I think it's safe to
access to the cache tree without the lock since we don't delete nodes.
It it missed an existing node due to rotation, it'll find it during
dso_cache__insert() anyway.

Signed-off-by: Namhyung Kim <namhyung@kernel.org>
---
 tools/perf/util/dso.c | 34 +++++++++++++++++++++++++++-------
 1 file changed, 27 insertions(+), 7 deletions(-)

diff --git a/tools/perf/util/dso.c b/tools/perf/util/dso.c
index 3da75816b8f8..11ece224ef50 100644
--- a/tools/perf/util/dso.c
+++ b/tools/perf/util/dso.c
@@ -443,10 +443,12 @@ bool dso__data_status_seen(struct dso *dso, enum dso_data_status_seen by)
 }
 
 static void
-dso_cache__free(struct rb_root *root)
+dso_cache__free(struct dso *dso)
 {
+	struct rb_root *root = &dso->data.cache;
 	struct rb_node *next = rb_first(root);
 
+	pthread_mutex_lock(&dso->lock);
 	while (next) {
 		struct dso_cache *cache;
 
@@ -455,10 +457,12 @@ dso_cache__free(struct rb_root *root)
 		rb_erase(&cache->rb_node, root);
 		free(cache);
 	}
+	pthread_mutex_unlock(&dso->lock);
 }
 
-static struct dso_cache *dso_cache__find(const struct rb_root *root, u64 offset)
+static struct dso_cache *dso_cache__find(struct dso *dso, u64 offset)
 {
+	const struct rb_root *root = &dso->data.cache;
 	struct rb_node * const *p = &root->rb_node;
 	const struct rb_node *parent = NULL;
 	struct dso_cache *cache;
@@ -477,17 +481,20 @@ static struct dso_cache *dso_cache__find(const struct rb_root *root, u64 offset)
 		else
 			return cache;
 	}
+
 	return NULL;
 }
 
-static void
-dso_cache__insert(struct rb_root *root, struct dso_cache *new)
+static struct dso_cache *
+dso_cache__insert(struct dso *dso, struct dso_cache *new)
 {
+	struct rb_root *root = &dso->data.cache;
 	struct rb_node **p = &root->rb_node;
 	struct rb_node *parent = NULL;
 	struct dso_cache *cache;
 	u64 offset = new->offset;
 
+	pthread_mutex_lock(&dso->lock);
 	while (*p != NULL) {
 		u64 end;
 
@@ -499,10 +506,17 @@ dso_cache__insert(struct rb_root *root, struct dso_cache *new)
 			p = &(*p)->rb_left;
 		else if (offset >= end)
 			p = &(*p)->rb_right;
+		else
+			goto out;
 	}
 
 	rb_link_node(&new->rb_node, parent, p);
 	rb_insert_color(&new->rb_node, root);
+
+	cache = NULL;
+out:
+	pthread_mutex_unlock(&dso->lock);
+	return cache;
 }
 
 static ssize_t
@@ -520,6 +534,7 @@ static ssize_t
 dso_cache__read(struct dso *dso, u64 offset, u8 *data, ssize_t size)
 {
 	struct dso_cache *cache;
+	struct dso_cache *old;
 	ssize_t ret;
 
 	do {
@@ -543,7 +558,12 @@ dso_cache__read(struct dso *dso, u64 offset, u8 *data, ssize_t size)
 
 		cache->offset = cache_offset;
 		cache->size   = ret;
-		dso_cache__insert(&dso->data.cache, cache);
+		old = dso_cache__insert(dso, cache);
+		if (old) {
+			/* we lose the race */
+			free(cache);
+			cache = old;
+		}
 
 		ret = dso_cache__memcpy(cache, offset, data, size);
 
@@ -560,7 +580,7 @@ static ssize_t dso_cache_read(struct dso *dso, u64 offset,
 {
 	struct dso_cache *cache;
 
-	cache = dso_cache__find(&dso->data.cache, offset);
+	cache = dso_cache__find(dso, offset);
 	if (cache)
 		return dso_cache__memcpy(cache, offset, data, size);
 	else
@@ -915,7 +935,7 @@ void dso__delete(struct dso *dso)
 	}
 
 	dso__data_close(dso);
-	dso_cache__free(&dso->data.cache);
+	dso_cache__free(dso);
 	dso__free_a2l(dso);
 	zfree(&dso->symsrc_filename);
 	pthread_mutex_destroy(&dso->lock);
-- 
2.2.2


^ permalink raw reply related	[flat|nested] 221+ messages in thread

* [PATCH 29/42] perf tools: Protect dso cache fd with a mutex
  2015-01-29  8:06 [RFC/PATCHSET 00/42] perf tools: Speed-up perf report by using multi thread (v2) Namhyung Kim
                   ` (27 preceding siblings ...)
  2015-01-29  8:07 ` [PATCH 28/42] perf tools: Protect dso cache tree using dso->lock Namhyung Kim
@ 2015-01-29  8:07 ` Namhyung Kim
  2015-01-29 12:31   ` Arnaldo Carvalho de Melo
  2015-01-29  8:07 ` [PATCH 30/42] perf session: Pass struct events stats to event processing functions Namhyung Kim
                   ` (13 subsequent siblings)
  42 siblings, 1 reply; 221+ messages in thread
From: Namhyung Kim @ 2015-01-29  8:07 UTC (permalink / raw)
  To: Arnaldo Carvalho de Melo
  Cc: Ingo Molnar, Peter Zijlstra, Jiri Olsa, LKML, David Ahern,
	Adrian Hunter, Andi Kleen, Stephane Eranian, Frederic Weisbecker

When dso cache is accessed in multi-thread environment, it's possible
to close other dso->data.fd during operation due to open file limit.
Protect the file descriptors using a separate mutex.

Signed-off-by: Namhyung Kim <namhyung@kernel.org>
---
 tools/perf/tests/dso-data.c |   5 ++
 tools/perf/util/dso.c       | 136 +++++++++++++++++++++++++++++---------------
 2 files changed, 94 insertions(+), 47 deletions(-)

diff --git a/tools/perf/tests/dso-data.c b/tools/perf/tests/dso-data.c
index caaf37f079b1..0276e7d2d41b 100644
--- a/tools/perf/tests/dso-data.c
+++ b/tools/perf/tests/dso-data.c
@@ -111,6 +111,9 @@ int test__dso_data(void)
 	memset(&machine, 0, sizeof(machine));
 
 	dso = dso__new((const char *)file);
+	TEST_ASSERT_VAL("failed to get dso", dso);
+
+	dso->binary_type = DSO_BINARY_TYPE__SYSTEM_PATH_DSO;
 
 	/* Basic 10 bytes tests. */
 	for (i = 0; i < ARRAY_SIZE(offsets); i++) {
@@ -199,6 +202,8 @@ static int dsos__create(int cnt, int size)
 
 		dsos[i] = dso__new(file);
 		TEST_ASSERT_VAL("failed to get dso", dsos[i]);
+
+		dsos[i]->binary_type = DSO_BINARY_TYPE__SYSTEM_PATH_DSO;
 	}
 
 	return 0;
diff --git a/tools/perf/util/dso.c b/tools/perf/util/dso.c
index 11ece224ef50..ae92046ae2c8 100644
--- a/tools/perf/util/dso.c
+++ b/tools/perf/util/dso.c
@@ -213,6 +213,7 @@ bool dso__needs_decompress(struct dso *dso)
  */
 static LIST_HEAD(dso__data_open);
 static long dso__data_open_cnt;
+static pthread_mutex_t dso__data_open_lock = PTHREAD_MUTEX_INITIALIZER;
 
 static void dso__list_add(struct dso *dso)
 {
@@ -240,7 +241,7 @@ static int do_open(char *name)
 		if (fd >= 0)
 			return fd;
 
-		pr_debug("dso open failed, mmap: %s\n",
+		pr_debug("dso open failed: %s\n",
 			 strerror_r(errno, sbuf, sizeof(sbuf)));
 		if (!dso__data_open_cnt || errno != EMFILE)
 			break;
@@ -382,7 +383,9 @@ static void check_data_close(void)
  */
 void dso__data_close(struct dso *dso)
 {
+	pthread_mutex_lock(&dso__data_open_lock);
 	close_dso(dso);
+	pthread_mutex_unlock(&dso__data_open_lock);
 }
 
 /**
@@ -405,6 +408,8 @@ int dso__data_fd(struct dso *dso, struct machine *machine)
 	if (dso->data.status == DSO_DATA_STATUS_ERROR)
 		return -1;
 
+	pthread_mutex_lock(&dso__data_open_lock);
+
 	if (dso->data.fd >= 0)
 		goto out;
 
@@ -427,6 +432,7 @@ int dso__data_fd(struct dso *dso, struct machine *machine)
 	else
 		dso->data.status = DSO_DATA_STATUS_ERROR;
 
+	pthread_mutex_unlock(&dso__data_open_lock);
 	return dso->data.fd;
 }
 
@@ -531,52 +537,66 @@ dso_cache__memcpy(struct dso_cache *cache, u64 offset,
 }
 
 static ssize_t
-dso_cache__read(struct dso *dso, u64 offset, u8 *data, ssize_t size)
+dso_cache__read(struct dso *dso, struct machine *machine,
+		u64 offset, u8 *data, ssize_t size)
 {
 	struct dso_cache *cache;
 	struct dso_cache *old;
-	ssize_t ret;
-
-	do {
-		u64 cache_offset;
+	ssize_t ret = -EINVAL;
+	u64 cache_offset;
 
-		ret = -ENOMEM;
+	cache = zalloc(sizeof(*cache) + DSO__DATA_CACHE_SIZE);
+	if (!cache)
+		return -ENOMEM;
 
-		cache = zalloc(sizeof(*cache) + DSO__DATA_CACHE_SIZE);
-		if (!cache)
-			break;
+	cache_offset = offset & DSO__DATA_CACHE_MASK;
 
-		cache_offset = offset & DSO__DATA_CACHE_MASK;
-		ret = -EINVAL;
+	pthread_mutex_lock(&dso__data_open_lock);
 
-		if (-1 == lseek(dso->data.fd, cache_offset, SEEK_SET))
-			break;
+	/*
+	 * dso->data.fd might be closed if other thread opened another
+	 * file (dso) due to open file limit (RLIMIT_NOFILE).
+	 */
+	if (dso->data.fd < 0) {
+		dso->data.fd = open_dso(dso, machine);
+		if (dso->data.fd < 0) {
+			ret = -errno;
+			dso->data.status = DSO_DATA_STATUS_ERROR;
+			goto err_unlock;
+		}
+	}
 
-		ret = read(dso->data.fd, cache->data, DSO__DATA_CACHE_SIZE);
-		if (ret <= 0)
-			break;
+	if (-1 == lseek(dso->data.fd, cache_offset, SEEK_SET))
+		goto err_unlock;
 
-		cache->offset = cache_offset;
-		cache->size   = ret;
-		old = dso_cache__insert(dso, cache);
-		if (old) {
-			/* we lose the race */
-			free(cache);
-			cache = old;
-		}
+	ret = read(dso->data.fd, cache->data, DSO__DATA_CACHE_SIZE);
+	if (ret <= 0)
+		goto err_unlock;
 
-		ret = dso_cache__memcpy(cache, offset, data, size);
+	pthread_mutex_unlock(&dso__data_open_lock);
 
-	} while (0);
+	cache->offset = cache_offset;
+	cache->size   = ret;
+	old = dso_cache__insert(dso, cache);
+	if (old) {
+		/* we lose the race */
+		free(cache);
+		cache = old;
+	}
 
+	ret = dso_cache__memcpy(cache, offset, data, size);
 	if (ret <= 0)
 		free(cache);
 
 	return ret;
+
+err_unlock:
+	pthread_mutex_unlock(&dso__data_open_lock);
+	return ret;
 }
 
-static ssize_t dso_cache_read(struct dso *dso, u64 offset,
-			      u8 *data, ssize_t size)
+static ssize_t dso_cache_read(struct dso *dso, struct machine *machine,
+			      u64 offset, u8 *data, ssize_t size)
 {
 	struct dso_cache *cache;
 
@@ -584,7 +604,7 @@ static ssize_t dso_cache_read(struct dso *dso, u64 offset,
 	if (cache)
 		return dso_cache__memcpy(cache, offset, data, size);
 	else
-		return dso_cache__read(dso, offset, data, size);
+		return dso_cache__read(dso, machine, offset, data, size);
 }
 
 /*
@@ -592,7 +612,8 @@ static ssize_t dso_cache_read(struct dso *dso, u64 offset,
  * in the rb_tree. Any read to already cached data is served
  * by cached data.
  */
-static ssize_t cached_read(struct dso *dso, u64 offset, u8 *data, ssize_t size)
+static ssize_t cached_read(struct dso *dso, struct machine *machine,
+			   u64 offset, u8 *data, ssize_t size)
 {
 	ssize_t r = 0;
 	u8 *p = data;
@@ -600,7 +621,7 @@ static ssize_t cached_read(struct dso *dso, u64 offset, u8 *data, ssize_t size)
 	do {
 		ssize_t ret;
 
-		ret = dso_cache_read(dso, offset, p, size);
+		ret = dso_cache_read(dso, machine, offset, p, size);
 		if (ret < 0)
 			return ret;
 
@@ -620,21 +641,42 @@ static ssize_t cached_read(struct dso *dso, u64 offset, u8 *data, ssize_t size)
 	return r;
 }
 
-static int data_file_size(struct dso *dso)
+static int data_file_size(struct dso *dso, struct machine *machine)
 {
+	int ret = 0;
 	struct stat st;
 	char sbuf[STRERR_BUFSIZE];
 
-	if (!dso->data.file_size) {
-		if (fstat(dso->data.fd, &st)) {
-			pr_err("dso mmap failed, fstat: %s\n",
-				strerror_r(errno, sbuf, sizeof(sbuf)));
-			return -1;
+	if (dso->data.file_size)
+		return 0;
+
+	pthread_mutex_lock(&dso__data_open_lock);
+
+	/*
+	 * dso->data.fd might be closed if other thread opened another
+	 * file (dso) due to open file limit (RLIMIT_NOFILE).
+	 */
+	if (dso->data.fd < 0) {
+		dso->data.fd = open_dso(dso, machine);
+		if (dso->data.fd < 0) {
+			ret = -errno;
+			dso->data.status = DSO_DATA_STATUS_ERROR;
+			goto out;
 		}
-		dso->data.file_size = st.st_size;
 	}
 
-	return 0;
+	if (fstat(dso->data.fd, &st) < 0) {
+		ret = -errno;
+		pr_err("dso cache fstat failed: %s\n",
+		       strerror_r(errno, sbuf, sizeof(sbuf)));
+		dso->data.status = DSO_DATA_STATUS_ERROR;
+		goto out;
+	}
+	dso->data.file_size = st.st_size;
+
+out:
+	pthread_mutex_unlock(&dso__data_open_lock);
+	return ret;
 }
 
 /**
@@ -652,17 +694,17 @@ off_t dso__data_size(struct dso *dso, struct machine *machine)
 	if (fd < 0)
 		return fd;
 
-	if (data_file_size(dso))
+	if (data_file_size(dso, machine))
 		return -1;
 
 	/* For now just estimate dso data size is close to file size */
 	return dso->data.file_size;
 }
 
-static ssize_t data_read_offset(struct dso *dso, u64 offset,
-				u8 *data, ssize_t size)
+static ssize_t data_read_offset(struct dso *dso, struct machine *machine,
+				u64 offset, u8 *data, ssize_t size)
 {
-	if (data_file_size(dso))
+	if (data_file_size(dso, machine))
 		return -1;
 
 	/* Check the offset sanity. */
@@ -672,7 +714,7 @@ static ssize_t data_read_offset(struct dso *dso, u64 offset,
 	if (offset + size < offset)
 		return -1;
 
-	return cached_read(dso, offset, data, size);
+	return cached_read(dso, machine, offset, data, size);
 }
 
 /**
@@ -689,10 +731,10 @@ static ssize_t data_read_offset(struct dso *dso, u64 offset,
 ssize_t dso__data_read_offset(struct dso *dso, struct machine *machine,
 			      u64 offset, u8 *data, ssize_t size)
 {
-	if (dso__data_fd(dso, machine) < 0)
+	if (dso->data.status == DSO_DATA_STATUS_ERROR)
 		return -1;
 
-	return data_read_offset(dso, offset, data, size);
+	return data_read_offset(dso, machine, offset, data, size);
 }
 
 /**
-- 
2.2.2


^ permalink raw reply related	[flat|nested] 221+ messages in thread

* [PATCH 30/42] perf session: Pass struct events stats to event processing functions
  2015-01-29  8:06 [RFC/PATCHSET 00/42] perf tools: Speed-up perf report by using multi thread (v2) Namhyung Kim
                   ` (28 preceding siblings ...)
  2015-01-29  8:07 ` [PATCH 29/42] perf tools: Protect dso cache fd with a mutex Namhyung Kim
@ 2015-01-29  8:07 ` Namhyung Kim
  2015-01-29  8:07 ` [PATCH 31/42] perf hists: Pass hists struct to hist_entry_iter functions Namhyung Kim
                   ` (12 subsequent siblings)
  42 siblings, 0 replies; 221+ messages in thread
From: Namhyung Kim @ 2015-01-29  8:07 UTC (permalink / raw)
  To: Arnaldo Carvalho de Melo
  Cc: Ingo Molnar, Peter Zijlstra, Jiri Olsa, LKML, David Ahern,
	Adrian Hunter, Andi Kleen, Stephane Eranian, Frederic Weisbecker

Pass stats structure so that it can point separate object when used in
multi-thread environment.

Signed-off-by: Namhyung Kim <namhyung@kernel.org>
---
 tools/perf/util/ordered-events.c |  4 +-
 tools/perf/util/session.c        | 81 ++++++++++++++++++++++------------------
 tools/perf/util/session.h        |  1 +
 3 files changed, 49 insertions(+), 37 deletions(-)

diff --git a/tools/perf/util/ordered-events.c b/tools/perf/util/ordered-events.c
index fd4be94125fb..e933c51d7090 100644
--- a/tools/perf/util/ordered-events.c
+++ b/tools/perf/util/ordered-events.c
@@ -183,7 +183,9 @@ static int __ordered_events__flush(struct perf_session *s,
 		if (ret)
 			pr_err("Can't parse sample, err = %d\n", ret);
 		else {
-			ret = perf_session__deliver_event(s, iter->event, &sample, tool,
+			ret = perf_session__deliver_event(s, &s->stats,
+							  iter->event,
+							  &sample, tool,
 							  iter->file_offset);
 			if (ret)
 				return ret;
diff --git a/tools/perf/util/session.c b/tools/perf/util/session.c
index e7b59fbebbc4..7114427f3d0f 100644
--- a/tools/perf/util/session.c
+++ b/tools/perf/util/session.c
@@ -776,6 +776,7 @@ static struct machine *
 }
 
 static int deliver_sample_value(struct perf_session *session,
+				struct events_stats *stats,
 				struct perf_tool *tool,
 				union perf_event *event,
 				struct perf_sample *sample,
@@ -792,7 +793,7 @@ static int deliver_sample_value(struct perf_session *session,
 	}
 
 	if (!sid || sid->evsel == NULL) {
-		++session->stats.nr_unknown_id;
+		++stats->nr_unknown_id;
 		return 0;
 	}
 
@@ -800,6 +801,7 @@ static int deliver_sample_value(struct perf_session *session,
 }
 
 static int deliver_sample_group(struct perf_session *session,
+				struct events_stats *stats,
 				struct perf_tool *tool,
 				union  perf_event *event,
 				struct perf_sample *sample,
@@ -809,7 +811,7 @@ static int deliver_sample_group(struct perf_session *session,
 	u64 i;
 
 	for (i = 0; i < sample->read.group.nr; i++) {
-		ret = deliver_sample_value(session, tool, event, sample,
+		ret = deliver_sample_value(session, stats, tool, event, sample,
 					   &sample->read.group.values[i],
 					   machine);
 		if (ret)
@@ -821,6 +823,7 @@ static int deliver_sample_group(struct perf_session *session,
 
 static int
 perf_session__deliver_sample(struct perf_session *session,
+			     struct events_stats *stats,
 			     struct perf_tool *tool,
 			     union  perf_event *event,
 			     struct perf_sample *sample,
@@ -837,14 +840,15 @@ perf_session__deliver_sample(struct perf_session *session,
 
 	/* For PERF_SAMPLE_READ we have either single or group mode. */
 	if (read_format & PERF_FORMAT_GROUP)
-		return deliver_sample_group(session, tool, event, sample,
+		return deliver_sample_group(session, stats, tool, event, sample,
 					    machine);
 	else
-		return deliver_sample_value(session, tool, event, sample,
+		return deliver_sample_value(session, stats, tool, event, sample,
 					    &sample->read.one, machine);
 }
 
 int perf_session__deliver_event(struct perf_session *session,
+				struct events_stats *stats,
 				union perf_event *event,
 				struct perf_sample *sample,
 				struct perf_tool *tool, u64 file_offset)
@@ -863,14 +867,14 @@ int perf_session__deliver_event(struct perf_session *session,
 	case PERF_RECORD_SAMPLE:
 		dump_sample(evsel, event, sample);
 		if (evsel == NULL) {
-			++session->stats.nr_unknown_id;
+			++stats->nr_unknown_id;
 			return 0;
 		}
 		if (machine == NULL) {
-			++session->stats.nr_unprocessable_samples;
+			++stats->nr_unprocessable_samples;
 			return 0;
 		}
-		return perf_session__deliver_sample(session, tool, event,
+		return perf_session__deliver_sample(session, stats, tool, event,
 						    sample, evsel, machine);
 	case PERF_RECORD_MMAP:
 		return tool->mmap(tool, event, sample, machine);
@@ -884,7 +888,7 @@ int perf_session__deliver_event(struct perf_session *session,
 		return tool->exit(tool, event, sample, machine);
 	case PERF_RECORD_LOST:
 		if (tool->lost == perf_event__process_lost)
-			session->stats.total_lost += event->lost.lost;
+			stats->total_lost += event->lost.lost;
 		return tool->lost(tool, event, sample, machine);
 	case PERF_RECORD_READ:
 		return tool->read(tool, event, sample, evsel, machine);
@@ -893,7 +897,7 @@ int perf_session__deliver_event(struct perf_session *session,
 	case PERF_RECORD_UNTHROTTLE:
 		return tool->unthrottle(tool, event, sample, machine);
 	default:
-		++session->stats.nr_unknown_events;
+		++stats->nr_unknown_events;
 		return -1;
 	}
 }
@@ -948,7 +952,8 @@ int perf_session__deliver_synth_event(struct perf_session *session,
 	if (event->header.type >= PERF_RECORD_USER_TYPE_START)
 		return perf_session__process_user_event(session, event, tool, 0);
 
-	return perf_session__deliver_event(session, event, sample, tool, 0);
+	return perf_session__deliver_event(session, &session->stats,
+					   event, sample, tool, 0);
 }
 
 static void event_swap(union perf_event *event, bool sample_id_all)
@@ -1016,6 +1021,7 @@ int perf_session__peek_event(struct perf_session *session, off_t file_offset,
 }
 
 static s64 perf_session__process_event(struct perf_session *session,
+				       struct events_stats *stats,
 				       union perf_event *event,
 				       struct perf_tool *tool,
 				       u64 file_offset)
@@ -1029,7 +1035,7 @@ static s64 perf_session__process_event(struct perf_session *session,
 	if (event->header.type >= PERF_RECORD_HEADER_MAX)
 		return -EINVAL;
 
-	events_stats__inc(&session->stats, event->header.type);
+	events_stats__inc(stats, event->header.type);
 
 	if (event->header.type >= PERF_RECORD_USER_TYPE_START)
 		return perf_session__process_user_event(session, event, tool, file_offset);
@@ -1048,8 +1054,8 @@ static s64 perf_session__process_event(struct perf_session *session,
 			return ret;
 	}
 
-	return perf_session__deliver_event(session, event, &sample, tool,
-					   file_offset);
+	return perf_session__deliver_event(session, stats, event, &sample,
+					   tool, file_offset);
 }
 
 void perf_event_header__bswap(struct perf_event_header *hdr)
@@ -1077,47 +1083,49 @@ static struct thread *perf_session__register_idle_thread(struct perf_session *se
 	return thread;
 }
 
-static void perf_session__warn_about_errors(const struct perf_session *session,
+static void events_stats__warn_about_errors(const struct events_stats *stats,
 					    const struct perf_tool *tool)
 {
 	if (tool->lost == perf_event__process_lost &&
-	    session->stats.nr_events[PERF_RECORD_LOST] != 0) {
+	    stats->nr_events[PERF_RECORD_LOST] != 0) {
 		ui__warning("Processed %d events and lost %d chunks!\n\n"
 			    "Check IO/CPU overload!\n\n",
-			    session->stats.nr_events[0],
-			    session->stats.nr_events[PERF_RECORD_LOST]);
+			    stats->nr_events[0],
+			    stats->nr_events[PERF_RECORD_LOST]);
 	}
 
-	if (session->stats.nr_unknown_events != 0) {
+	if (stats->nr_unknown_events != 0) {
 		ui__warning("Found %u unknown events!\n\n"
 			    "Is this an older tool processing a perf.data "
 			    "file generated by a more recent tool?\n\n"
 			    "If that is not the case, consider "
 			    "reporting to linux-kernel@vger.kernel.org.\n\n",
-			    session->stats.nr_unknown_events);
+			    stats->nr_unknown_events);
 	}
 
-	if (session->stats.nr_unknown_id != 0) {
+	if (stats->nr_unknown_id != 0) {
 		ui__warning("%u samples with id not present in the header\n",
-			    session->stats.nr_unknown_id);
+			    stats->nr_unknown_id);
 	}
 
- 	if (session->stats.nr_invalid_chains != 0) {
+	if (stats->nr_invalid_chains != 0) {
  		ui__warning("Found invalid callchains!\n\n"
  			    "%u out of %u events were discarded for this reason.\n\n"
  			    "Consider reporting to linux-kernel@vger.kernel.org.\n\n",
- 			    session->stats.nr_invalid_chains,
- 			    session->stats.nr_events[PERF_RECORD_SAMPLE]);
+			    stats->nr_invalid_chains,
+			    stats->nr_events[PERF_RECORD_SAMPLE]);
  	}
 
-	if (session->stats.nr_unprocessable_samples != 0) {
+	if (stats->nr_unprocessable_samples != 0) {
 		ui__warning("%u unprocessable samples recorded.\n"
 			    "Do you have a KVM guest running and not using 'perf kvm'?\n",
-			    session->stats.nr_unprocessable_samples);
+			    stats->nr_unprocessable_samples);
 	}
 
-	if (session->stats.nr_unordered_events != 0)
-		ui__warning("%u out of order events recorded.\n", session->stats.nr_unordered_events);
+	if (stats->nr_unordered_events != 0) {
+		ui__warning("%u out of order events recorded.\n",
+			    stats->nr_unordered_events);
+	}
 }
 
 volatile int session_done;
@@ -1188,7 +1196,8 @@ static int __perf_session__process_pipe_events(struct perf_session *session,
 		}
 	}
 
-	if ((skip = perf_session__process_event(session, event, tool, head)) < 0) {
+	if ((skip = perf_session__process_event(session, &session->stats,
+						event, tool, head)) < 0) {
 		pr_err("%#" PRIx64 " [%#x]: failed to process type: %d\n",
 		       head, event->header.size, event->header.type);
 		err = -EINVAL;
@@ -1207,7 +1216,7 @@ static int __perf_session__process_pipe_events(struct perf_session *session,
 	err = ordered_events__flush(session, tool, OE_FLUSH__FINAL);
 out_err:
 	free(buf);
-	perf_session__warn_about_errors(session, tool);
+	events_stats__warn_about_errors(&session->stats, tool);
 	ordered_events__free(&session->ordered_events);
 	return err;
 }
@@ -1252,7 +1261,8 @@ fetch_mmaped_event(struct perf_session *session,
 #define NUM_MMAPS 128
 #endif
 
-static int __perf_session__process_events(struct perf_session *session, int fd,
+static int __perf_session__process_events(struct perf_session *session,
+					  struct events_stats *stats, int fd,
 					  u64 data_offset, u64 data_size,
 					  u64 file_size, struct perf_tool *tool)
 {
@@ -1325,8 +1335,8 @@ static int __perf_session__process_events(struct perf_session *session, int fd,
 	size = event->header.size;
 
 	if (size < sizeof(struct perf_event_header) ||
-	    (skip = perf_session__process_event(session, event, tool, file_pos))
-									< 0) {
+	    (skip = perf_session__process_event(session, stats, event,
+						tool, file_pos)) < 0) {
 		pr_err("%#" PRIx64 " [%#x]: failed to process type: %d\n",
 		       file_offset + head, event->header.size,
 		       event->header.type);
@@ -1353,7 +1363,6 @@ static int __perf_session__process_events(struct perf_session *session, int fd,
 	err = ordered_events__flush(session, tool, OE_FLUSH__FINAL);
 out_err:
 	ui_progress__finish();
-	perf_session__warn_about_errors(session, tool);
 	ordered_events__free(&session->ordered_events);
 	session->one_mmap = false;
 	return err;
@@ -1372,7 +1381,7 @@ int perf_session__process_events(struct perf_session *session,
 	if (perf_data_file__is_pipe(file))
 		return __perf_session__process_pipe_events(session, tool);
 
-	err = __perf_session__process_events(session,
+	err = __perf_session__process_events(session, &session->stats,
 					     perf_data_file__fd(file),
 					     session->header.data_offset,
 					     session->header.data_size,
@@ -1391,7 +1400,7 @@ int perf_session__process_events(struct perf_session *session,
 		if (!session->header.index[i].size)
 			continue;
 
-		err = __perf_session__process_events(session,
+		err = __perf_session__process_events(session, &session->stats,
 						perf_data_file__fd(file),
 						session->header.index[i].offset,
 						session->header.index[i].size,
diff --git a/tools/perf/util/session.h b/tools/perf/util/session.h
index 419976d74b51..33af571f9d08 100644
--- a/tools/perf/util/session.h
+++ b/tools/perf/util/session.h
@@ -59,6 +59,7 @@ int perf_session_queue_event(struct perf_session *s, union perf_event *event,
 void perf_tool__fill_defaults(struct perf_tool *tool);
 
 int perf_session__deliver_event(struct perf_session *session,
+				struct events_stats *stats,
 				union perf_event *event,
 				struct perf_sample *sample,
 				struct perf_tool *tool, u64 file_offset);
-- 
2.2.2


^ permalink raw reply related	[flat|nested] 221+ messages in thread

* [PATCH 31/42] perf hists: Pass hists struct to hist_entry_iter functions
  2015-01-29  8:06 [RFC/PATCHSET 00/42] perf tools: Speed-up perf report by using multi thread (v2) Namhyung Kim
                   ` (29 preceding siblings ...)
  2015-01-29  8:07 ` [PATCH 30/42] perf session: Pass struct events stats to event processing functions Namhyung Kim
@ 2015-01-29  8:07 ` Namhyung Kim
  2015-01-29  8:07 ` [PATCH 32/42] perf tools: Move BUILD_ID_SIZE definition to perf.h Namhyung Kim
                   ` (11 subsequent siblings)
  42 siblings, 0 replies; 221+ messages in thread
From: Namhyung Kim @ 2015-01-29  8:07 UTC (permalink / raw)
  To: Arnaldo Carvalho de Melo
  Cc: Ingo Molnar, Peter Zijlstra, Jiri Olsa, LKML, David Ahern,
	Adrian Hunter, Andi Kleen, Stephane Eranian, Frederic Weisbecker

This is a preparation for perf report multi-thread support.  When
multi-thread is enable, each thread will have its own hists during the
sample processing.

Signed-off-by: Namhyung Kim <namhyung@kernel.org>
---
 tools/perf/builtin-report.c       |  4 ++--
 tools/perf/builtin-top.c          |  4 ++--
 tools/perf/tests/hists_cumulate.c |  4 ++--
 tools/perf/tests/hists_filter.c   |  3 ++-
 tools/perf/tests/hists_output.c   |  4 ++--
 tools/perf/util/hist.c            | 26 +++++++++++---------------
 tools/perf/util/hist.h            |  6 ++++--
 7 files changed, 25 insertions(+), 26 deletions(-)

diff --git a/tools/perf/builtin-report.c b/tools/perf/builtin-report.c
index 68d06bc02266..8a40c79d9273 100644
--- a/tools/perf/builtin-report.c
+++ b/tools/perf/builtin-report.c
@@ -167,8 +167,8 @@ static int process_sample_event(struct perf_tool *tool,
 	if (al.map != NULL)
 		al.map->dso->hit = 1;
 
-	ret = hist_entry_iter__add(&iter, &al, evsel, sample, rep->max_stack,
-				   rep);
+	ret = hist_entry_iter__add(&iter, evsel__hists(evsel), evsel, &al,
+				   sample, rep->max_stack, rep);
 	if (ret < 0)
 		pr_debug("problem adding hist entry, skipping event\n");
 
diff --git a/tools/perf/builtin-top.c b/tools/perf/builtin-top.c
index 69a0badfb745..a49a34bcf791 100644
--- a/tools/perf/builtin-top.c
+++ b/tools/perf/builtin-top.c
@@ -784,8 +784,8 @@ static void perf_event__process_sample(struct perf_tool *tool,
 
 		pthread_mutex_lock(&hists->lock);
 
-		err = hist_entry_iter__add(&iter, &al, evsel, sample,
-					   top->max_stack, top);
+		err = hist_entry_iter__add(&iter, evsel__hists(evsel), evsel,
+					   &al, sample, top->max_stack, top);
 		if (err < 0)
 			pr_err("Problem incrementing symbol period, skipping event\n");
 
diff --git a/tools/perf/tests/hists_cumulate.c b/tools/perf/tests/hists_cumulate.c
index 60682e62d9de..71156c0d6ad5 100644
--- a/tools/perf/tests/hists_cumulate.c
+++ b/tools/perf/tests/hists_cumulate.c
@@ -104,8 +104,8 @@ static int add_hist_entries(struct hists *hists, struct machine *machine)
 						  &sample, NULL) < 0)
 			goto out;
 
-		if (hist_entry_iter__add(&iter, &al, evsel, &sample,
-					 PERF_MAX_STACK_DEPTH, NULL) < 0)
+		if (hist_entry_iter__add(&iter, evsel__hists(evsel), evsel, &al,
+					 &sample, PERF_MAX_STACK_DEPTH, NULL) < 0)
 			goto out;
 
 		fake_samples[i].thread = al.thread;
diff --git a/tools/perf/tests/hists_filter.c b/tools/perf/tests/hists_filter.c
index 1c4e495d5137..408ee7e48802 100644
--- a/tools/perf/tests/hists_filter.c
+++ b/tools/perf/tests/hists_filter.c
@@ -81,7 +81,8 @@ static int add_hist_entries(struct perf_evlist *evlist,
 							  &sample, NULL) < 0)
 				goto out;
 
-			if (hist_entry_iter__add(&iter, &al, evsel, &sample,
+			if (hist_entry_iter__add(&iter, evsel__hists(evsel),
+						 evsel, &al, &sample,
 						 PERF_MAX_STACK_DEPTH, NULL) < 0)
 				goto out;
 
diff --git a/tools/perf/tests/hists_output.c b/tools/perf/tests/hists_output.c
index f4e3286cd496..bffe8832d692 100644
--- a/tools/perf/tests/hists_output.c
+++ b/tools/perf/tests/hists_output.c
@@ -70,8 +70,8 @@ static int add_hist_entries(struct hists *hists, struct machine *machine)
 						  &sample, NULL) < 0)
 			goto out;
 
-		if (hist_entry_iter__add(&iter, &al, evsel, &sample,
-					 PERF_MAX_STACK_DEPTH, NULL) < 0)
+		if (hist_entry_iter__add(&iter, evsel__hists(evsel), evsel, &al,
+					 &sample, PERF_MAX_STACK_DEPTH, NULL) < 0)
 			goto out;
 
 		fake_samples[i].thread = al.thread;
diff --git a/tools/perf/util/hist.c b/tools/perf/util/hist.c
index 4badf2491fbf..c44565b382c5 100644
--- a/tools/perf/util/hist.c
+++ b/tools/perf/util/hist.c
@@ -510,7 +510,7 @@ iter_add_single_mem_entry(struct hist_entry_iter *iter, struct addr_location *al
 	u64 cost;
 	struct mem_info *mi = iter->priv;
 	struct perf_sample *sample = iter->sample;
-	struct hists *hists = evsel__hists(iter->evsel);
+	struct hists *hists = iter->hists;
 	struct hist_entry *he;
 
 	if (mi == NULL)
@@ -540,8 +540,7 @@ static int
 iter_finish_mem_entry(struct hist_entry_iter *iter,
 		      struct addr_location *al __maybe_unused)
 {
-	struct perf_evsel *evsel = iter->evsel;
-	struct hists *hists = evsel__hists(evsel);
+	struct hists *hists = iter->hists;
 	struct hist_entry *he = iter->he;
 	int err = -EINVAL;
 
@@ -613,8 +612,7 @@ static int
 iter_add_next_branch_entry(struct hist_entry_iter *iter, struct addr_location *al)
 {
 	struct branch_info *bi;
-	struct perf_evsel *evsel = iter->evsel;
-	struct hists *hists = evsel__hists(evsel);
+	struct hists *hists = iter->hists;
 	struct hist_entry *he = NULL;
 	int i = iter->curr;
 	int err = 0;
@@ -661,11 +659,10 @@ iter_prepare_normal_entry(struct hist_entry_iter *iter __maybe_unused,
 static int
 iter_add_single_normal_entry(struct hist_entry_iter *iter, struct addr_location *al)
 {
-	struct perf_evsel *evsel = iter->evsel;
 	struct perf_sample *sample = iter->sample;
 	struct hist_entry *he;
 
-	he = __hists__add_entry(evsel__hists(evsel), al, iter->parent, NULL, NULL,
+	he = __hists__add_entry(iter->hists, al, iter->parent, NULL, NULL,
 				sample->period, sample->weight,
 				sample->transaction, sample->time, true);
 	if (he == NULL)
@@ -680,7 +677,6 @@ iter_finish_normal_entry(struct hist_entry_iter *iter,
 			 struct addr_location *al __maybe_unused)
 {
 	struct hist_entry *he = iter->he;
-	struct perf_evsel *evsel = iter->evsel;
 	struct perf_sample *sample = iter->sample;
 
 	if (he == NULL)
@@ -688,7 +684,7 @@ iter_finish_normal_entry(struct hist_entry_iter *iter,
 
 	iter->he = NULL;
 
-	hists__inc_nr_samples(evsel__hists(evsel), he->filtered);
+	hists__inc_nr_samples(iter->hists, he->filtered);
 
 	return hist_entry__append_callchain(he, sample);
 }
@@ -720,8 +716,7 @@ static int
 iter_add_single_cumulative_entry(struct hist_entry_iter *iter,
 				 struct addr_location *al)
 {
-	struct perf_evsel *evsel = iter->evsel;
-	struct hists *hists = evsel__hists(evsel);
+	struct hists *hists = iter->hists;
 	struct perf_sample *sample = iter->sample;
 	struct hist_entry **he_cache = iter->priv;
 	struct hist_entry *he;
@@ -766,7 +761,6 @@ static int
 iter_add_next_cumulative_entry(struct hist_entry_iter *iter,
 			       struct addr_location *al)
 {
-	struct perf_evsel *evsel = iter->evsel;
 	struct perf_sample *sample = iter->sample;
 	struct hist_entry **he_cache = iter->priv;
 	struct hist_entry *he;
@@ -800,7 +794,7 @@ iter_add_next_cumulative_entry(struct hist_entry_iter *iter,
 		}
 	}
 
-	he = __hists__add_entry(evsel__hists(evsel), al, iter->parent, NULL, NULL,
+	he = __hists__add_entry(iter->hists, al, iter->parent, NULL, NULL,
 				sample->period, sample->weight,
 				sample->transaction, sample->time, false);
 	if (he == NULL)
@@ -856,8 +850,9 @@ const struct hist_iter_ops hist_iter_cumulative = {
 	.finish_entry 		= iter_finish_cumulative_entry,
 };
 
-int hist_entry_iter__add(struct hist_entry_iter *iter, struct addr_location *al,
-			 struct perf_evsel *evsel, struct perf_sample *sample,
+int hist_entry_iter__add(struct hist_entry_iter *iter, struct hists *hists,
+			 struct perf_evsel *evsel, struct addr_location *al,
+			 struct perf_sample *sample,
 			 int max_stack_depth, void *arg)
 {
 	int err, err2;
@@ -867,6 +862,7 @@ int hist_entry_iter__add(struct hist_entry_iter *iter, struct addr_location *al,
 	if (err)
 		return err;
 
+	iter->hists = hists;
 	iter->evsel = evsel;
 	iter->sample = sample;
 
diff --git a/tools/perf/util/hist.h b/tools/perf/util/hist.h
index 0eed50a5b1f0..991ca5504cbd 100644
--- a/tools/perf/util/hist.h
+++ b/tools/perf/util/hist.h
@@ -86,6 +86,7 @@ struct hist_entry_iter {
 
 	bool hide_unresolved;
 
+	struct hists *hists;
 	struct perf_evsel *evsel;
 	struct perf_sample *sample;
 	struct hist_entry *he;
@@ -110,8 +111,9 @@ struct hist_entry *__hists__add_entry(struct hists *hists,
 				      struct mem_info *mi, u64 period,
 				      u64 weight, u64 transaction,
 				      u64 timestamp, bool sample_self);
-int hist_entry_iter__add(struct hist_entry_iter *iter, struct addr_location *al,
-			 struct perf_evsel *evsel, struct perf_sample *sample,
+int hist_entry_iter__add(struct hist_entry_iter *iter, struct hists *hists,
+			 struct perf_evsel *evsel, struct addr_location *al,
+			 struct perf_sample *sample,
 			 int max_stack_depth, void *arg);
 
 int64_t hist_entry__cmp(struct hist_entry *left, struct hist_entry *right);
-- 
2.2.2


^ permalink raw reply related	[flat|nested] 221+ messages in thread

* [PATCH 32/42] perf tools: Move BUILD_ID_SIZE definition to perf.h
  2015-01-29  8:06 [RFC/PATCHSET 00/42] perf tools: Speed-up perf report by using multi thread (v2) Namhyung Kim
                   ` (30 preceding siblings ...)
  2015-01-29  8:07 ` [PATCH 31/42] perf hists: Pass hists struct to hist_entry_iter functions Namhyung Kim
@ 2015-01-29  8:07 ` Namhyung Kim
  2015-01-29  8:07 ` [PATCH 33/42] perf report: Parallelize perf report using multi-thread Namhyung Kim
                   ` (10 subsequent siblings)
  42 siblings, 0 replies; 221+ messages in thread
From: Namhyung Kim @ 2015-01-29  8:07 UTC (permalink / raw)
  To: Arnaldo Carvalho de Melo
  Cc: Ingo Molnar, Peter Zijlstra, Jiri Olsa, LKML, David Ahern,
	Adrian Hunter, Andi Kleen, Stephane Eranian, Frederic Weisbecker

The util/event.h includes util/build-id.h only for BUILD_ID_SIZE.
This is a problem when I include util/event.h from util/tool.h which
is also included by util/build-id.h since it now makes a circular
dependency resulting in incomplete type error.

Signed-off-by: Namhyung Kim <namhyung@kernel.org>
---
 tools/perf/perf.h          | 1 +
 tools/perf/util/build-id.h | 2 --
 tools/perf/util/dso.h      | 1 +
 tools/perf/util/event.h    | 1 -
 4 files changed, 2 insertions(+), 3 deletions(-)

diff --git a/tools/perf/perf.h b/tools/perf/perf.h
index b0fad99c9252..386de322f3a1 100644
--- a/tools/perf/perf.h
+++ b/tools/perf/perf.h
@@ -30,6 +30,7 @@ static inline unsigned long long rdclock(void)
 }
 
 #define MAX_NR_CPUS			256
+#define BUILD_ID_SIZE			20
 
 extern const char *input_name;
 extern bool perf_host, perf_guest;
diff --git a/tools/perf/util/build-id.h b/tools/perf/util/build-id.h
index 8236319514d5..8f31545edc5b 100644
--- a/tools/perf/util/build-id.h
+++ b/tools/perf/util/build-id.h
@@ -1,8 +1,6 @@
 #ifndef PERF_BUILD_ID_H_
 #define PERF_BUILD_ID_H_ 1
 
-#define BUILD_ID_SIZE 20
-
 #include "tool.h"
 #include <linux/types.h>
 
diff --git a/tools/perf/util/dso.h b/tools/perf/util/dso.h
index ac753594a469..c18fcc0e8081 100644
--- a/tools/perf/util/dso.h
+++ b/tools/perf/util/dso.h
@@ -7,6 +7,7 @@
 #include <linux/types.h>
 #include <linux/bitops.h>
 #include "map.h"
+#include "perf.h"
 #include "build-id.h"
 
 enum dso_binary_type {
diff --git a/tools/perf/util/event.h b/tools/perf/util/event.h
index 27261320249a..1f86c279520e 100644
--- a/tools/perf/util/event.h
+++ b/tools/perf/util/event.h
@@ -6,7 +6,6 @@
 
 #include "../perf.h"
 #include "map.h"
-#include "build-id.h"
 #include "perf_regs.h"
 
 struct mmap_event {
-- 
2.2.2


^ permalink raw reply related	[flat|nested] 221+ messages in thread

* [PATCH 33/42] perf report: Parallelize perf report using multi-thread
  2015-01-29  8:06 [RFC/PATCHSET 00/42] perf tools: Speed-up perf report by using multi thread (v2) Namhyung Kim
                   ` (31 preceding siblings ...)
  2015-01-29  8:07 ` [PATCH 32/42] perf tools: Move BUILD_ID_SIZE definition to perf.h Namhyung Kim
@ 2015-01-29  8:07 ` Namhyung Kim
  2015-01-29  8:07 ` [PATCH 34/42] perf tools: Add missing_threads rb tree Namhyung Kim
                   ` (9 subsequent siblings)
  42 siblings, 0 replies; 221+ messages in thread
From: Namhyung Kim @ 2015-01-29  8:07 UTC (permalink / raw)
  To: Arnaldo Carvalho de Melo
  Cc: Ingo Molnar, Peter Zijlstra, Jiri Olsa, LKML, David Ahern,
	Adrian Hunter, Andi Kleen, Stephane Eranian, Frederic Weisbecker

Introduce perf_session__process_events_mt() to enable multi-thread
sample processing.  It allocates a struct perf_tool_mt and fills
needed info in it.

The session and hists event stats are counted for each thread and
summed after finishing the processing.  Similarly hist entries are
added to per-thread hists first and then move to the original hists
using hists__mt_resort().  This function reuses hists__collapse_
resort() code so makes sort__need_collapse force to true and skips
the collapsing function.

Note that most of preprocessing stage is already done by processing
meta events in dummy tracking evsel first.  We can find corresponding
thread and map based on the sample time and symbol loading and dso
cache access is protected by pthread mutex.

Signed-off-by: Namhyung Kim <namhyung@kernel.org>
---
 tools/perf/util/hist.c    |  75 +++++++++++++++++++-----
 tools/perf/util/hist.h    |   3 +
 tools/perf/util/session.c | 141 ++++++++++++++++++++++++++++++++++++++++++++++
 tools/perf/util/session.h |   2 +
 tools/perf/util/tool.h    |  12 ++++
 5 files changed, 220 insertions(+), 13 deletions(-)

diff --git a/tools/perf/util/hist.c b/tools/perf/util/hist.c
index c44565b382c5..14d4b9358ac6 100644
--- a/tools/perf/util/hist.c
+++ b/tools/perf/util/hist.c
@@ -950,7 +950,7 @@ void hist_entry__delete(struct hist_entry *he)
  * collapse the histogram
  */
 
-static bool hists__collapse_insert_entry(struct hists *hists __maybe_unused,
+static bool hists__collapse_insert_entry(struct hists *hists,
 					 struct rb_root *root,
 					 struct hist_entry *he)
 {
@@ -987,6 +987,13 @@ static bool hists__collapse_insert_entry(struct hists *hists __maybe_unused,
 	}
 	hists->nr_entries++;
 
+	/*
+	 * For multi-threaded report, he->hists points to a dummy
+	 * hists in the struct perf_tool_mt.  Please see
+	 * perf_session__process_events_mt().
+	 */
+	he->hists = hists;
+
 	rb_link_node(&he->rb_node_in, parent, p);
 	rb_insert_color(&he->rb_node_in, root);
 	return true;
@@ -1014,19 +1021,12 @@ static void hists__apply_filters(struct hists *hists, struct hist_entry *he)
 	hists__filter_entry_by_symbol(hists, he);
 }
 
-void hists__collapse_resort(struct hists *hists, struct ui_progress *prog)
+static void __hists__collapse_resort(struct hists *hists, struct rb_root *root,
+				     struct ui_progress *prog)
 {
-	struct rb_root *root;
 	struct rb_node *next;
 	struct hist_entry *n;
 
-	if (!sort__need_collapse)
-		return;
-
-	hists->nr_entries = 0;
-
-	root = hists__get_rotate_entries_in(hists);
-
 	next = rb_first(root);
 
 	while (next) {
@@ -1049,6 +1049,27 @@ void hists__collapse_resort(struct hists *hists, struct ui_progress *prog)
 	}
 }
 
+void hists__collapse_resort(struct hists *hists, struct ui_progress *prog)
+{
+	struct rb_root *root;
+
+	if (!sort__need_collapse)
+		return;
+
+	hists->nr_entries = 0;
+
+	root = hists__get_rotate_entries_in(hists);
+	__hists__collapse_resort(hists, root, prog);
+}
+
+void hists__mt_resort(struct hists *dst, struct hists *src)
+{
+	struct rb_root *root = src->entries_in;
+
+	sort__need_collapse = 1;
+	__hists__collapse_resort(dst, root, NULL);
+}
+
 static int hist_entry__sort(struct hist_entry *a, struct hist_entry *b)
 {
 	struct perf_hpp_fmt *fmt;
@@ -1277,6 +1298,29 @@ void events_stats__inc(struct events_stats *stats, u32 type)
 	++stats->nr_events[type];
 }
 
+void events_stats__add(struct events_stats *dst, struct events_stats *src)
+{
+	int i;
+
+#define ADD(_field)  dst->_field += src->_field
+
+	ADD(total_period);
+	ADD(total_non_filtered_period);
+	ADD(total_lost);
+	ADD(total_invalid_chains);
+	ADD(nr_non_filtered_samples);
+	ADD(nr_lost_warned);
+	ADD(nr_unknown_events);
+	ADD(nr_invalid_chains);
+	ADD(nr_unknown_id);
+	ADD(nr_unprocessable_samples);
+
+	for (i = 0; i < PERF_RECORD_HEADER_MAX; i++)
+		ADD(nr_events[i]);
+
+#undef ADD
+}
+
 void hists__inc_nr_events(struct hists *hists, u32 type)
 {
 	events_stats__inc(&hists->stats, type);
@@ -1453,16 +1497,21 @@ int perf_hist_config(const char *var, const char *value)
 	return 0;
 }
 
-static int hists_evsel__init(struct perf_evsel *evsel)
+void __hists__init(struct hists *hists)
 {
-	struct hists *hists = evsel__hists(evsel);
-
 	memset(hists, 0, sizeof(*hists));
 	hists->entries_in_array[0] = hists->entries_in_array[1] = RB_ROOT;
 	hists->entries_in = &hists->entries_in_array[0];
 	hists->entries_collapsed = RB_ROOT;
 	hists->entries = RB_ROOT;
 	pthread_mutex_init(&hists->lock, NULL);
+}
+
+static int hists_evsel__init(struct perf_evsel *evsel)
+{
+	struct hists *hists = evsel__hists(evsel);
+
+	__hists__init(hists);
 	return 0;
 }
 
diff --git a/tools/perf/util/hist.h b/tools/perf/util/hist.h
index 991ca5504cbd..2c29d70b2cfe 100644
--- a/tools/perf/util/hist.h
+++ b/tools/perf/util/hist.h
@@ -124,6 +124,7 @@ int hist_entry__sort_snprintf(struct hist_entry *he, char *bf, size_t size,
 void hist_entry__delete(struct hist_entry *he);
 
 void hists__output_resort(struct hists *hists, struct ui_progress *prog);
+void hists__mt_resort(struct hists *dst, struct hists *src);
 void hists__collapse_resort(struct hists *hists, struct ui_progress *prog);
 
 void hists__decay_entries(struct hists *hists, bool zap_user, bool zap_kernel);
@@ -136,6 +137,7 @@ void hists__inc_stats(struct hists *hists, struct hist_entry *h);
 void hists__inc_nr_events(struct hists *hists, u32 type);
 void hists__inc_nr_samples(struct hists *hists, bool filtered);
 void events_stats__inc(struct events_stats *stats, u32 type);
+void events_stats__add(struct events_stats *dst, struct events_stats *src);
 size_t events_stats__fprintf(struct events_stats *stats, FILE *fp);
 
 size_t hists__fprintf(struct hists *hists, bool show_header, int max_rows,
@@ -179,6 +181,7 @@ static inline struct hists *evsel__hists(struct perf_evsel *evsel)
 }
 
 int hists__init(void);
+void __hists__init(struct hists *hists);
 
 struct perf_hpp {
 	char *buf;
diff --git a/tools/perf/util/session.c b/tools/perf/util/session.c
index 7114427f3d0f..d1d5e0b3a26e 100644
--- a/tools/perf/util/session.c
+++ b/tools/perf/util/session.c
@@ -1412,6 +1412,147 @@ int perf_session__process_events(struct perf_session *session,
 	return err;
 }
 
+static void *processing_thread_idx(void *arg)
+{
+	struct perf_tool_mt *mt_tool = arg;
+	struct perf_session *session = mt_tool->session;
+	int fd = perf_data_file__fd(session->file);
+	u64 offset = session->header.index[mt_tool->idx].offset;
+	u64 size = session->header.index[mt_tool->idx].size;
+	u64 file_size = perf_data_file__size(session->file);
+
+	pr_debug("processing samples using thread [%d]\n", mt_tool->idx);
+	if (__perf_session__process_events(session, &mt_tool->stats,
+					   fd, offset, size, file_size,
+					   &mt_tool->tool) < 0) {
+		pr_err("processing samples failed (thread [%d)\n", mt_tool->idx);
+		return NULL;
+	}
+
+	pr_debug("processing samples done for thread [%d]\n", mt_tool->idx);
+	return arg;
+}
+
+int perf_session__process_events_mt(struct perf_session *session,
+				    struct perf_tool *tool, void *arg)
+{
+	struct perf_data_file *file = session->file;
+	struct perf_evlist *evlist = session->evlist;
+	struct perf_evsel *evsel;
+	u64 nr_entries = 0;
+	struct perf_tool_mt *mt_tools = NULL;
+	struct perf_tool_mt *mt;
+	pthread_t *th_id;
+	int err, i, k;
+	int nr_index = session->header.nr_index;
+	u64 size = perf_data_file__size(file);
+
+	if (perf_session__register_idle_thread(session) == NULL)
+		return -ENOMEM;
+
+	if (perf_data_file__is_pipe(file) || !session->header.index) {
+		pr_err("data file doesn't contain the index table\n");
+		return -EINVAL;
+	}
+
+	err = __perf_session__process_events(session, &session->stats,
+					     perf_data_file__fd(file),
+					     session->header.data_offset,
+					     session->header.data_size,
+					     size, tool);
+	if (err)
+		return err;
+
+	th_id = calloc(nr_index, sizeof(*th_id));
+	if (th_id == NULL)
+		goto out;
+
+	mt_tools = calloc(nr_index, sizeof(*mt_tools));
+	if (mt_tools == NULL)
+		goto out;
+
+	for (i = 0; i < nr_index; i++) {
+		mt = &mt_tools[i];
+
+		memcpy(&mt->tool, tool, sizeof(*tool));
+
+		mt->hists = calloc(evlist->nr_entries, sizeof(*mt->hists));
+		if (mt->hists == NULL)
+			goto err;
+
+		for (k = 0; k < evlist->nr_entries; k++)
+			__hists__init(&mt->hists[k]);
+
+		mt->session = session;
+		mt->tool.ordered_events = false;
+		mt->idx = i;
+		mt->priv = arg;
+
+		pthread_create(&th_id[i], NULL, processing_thread_idx, mt);
+	}
+
+	for (i = 0; i < nr_index; i++) {
+		pthread_join(th_id[i], (void **)&mt);
+		if (mt == NULL) {
+			err = -EINVAL;
+			continue;
+		}
+
+		events_stats__add(&session->stats, &mt->stats);
+
+		evlist__for_each(evlist, evsel) {
+			struct hists *hists = evsel__hists(evsel);
+
+			events_stats__add(&hists->stats,
+					  &mt->hists[evsel->idx].stats);
+
+			nr_entries += mt->hists[evsel->idx].nr_entries;
+		}
+	}
+
+	for (i = 0; i < nr_index; i++) {
+		mt = &mt_tools[i];
+
+		evlist__for_each(evlist, evsel) {
+			struct hists *hists = evsel__hists(evsel);
+
+			if (perf_evsel__is_dummy_tracking(evsel))
+				continue;
+
+			hists__mt_resort(hists, &mt->hists[evsel->idx]);
+
+			/* Non-group events are considered as leader */
+			if (symbol_conf.event_group &&
+			    !perf_evsel__is_group_leader(evsel)) {
+				struct hists *leader_hists;
+
+				leader_hists = evsel__hists(evsel->leader);
+				hists__match(leader_hists, hists);
+				hists__link(leader_hists, hists);
+			}
+		}
+	}
+
+out:
+	events_stats__warn_about_errors(&session->stats, tool);
+
+	if (mt_tools) {
+		for (i = 0; i < nr_index; i++)
+			free(mt_tools[i].hists);
+		free(mt_tools);
+	}
+
+	free(th_id);
+	return err;
+
+err:
+	while (i-- > 0) {
+		pthread_cancel(th_id[i]);
+		pthread_join(th_id[i], NULL);
+	}
+
+	goto out;
+}
 bool perf_session__has_traces(struct perf_session *session, const char *msg)
 {
 	struct perf_evsel *evsel;
diff --git a/tools/perf/util/session.h b/tools/perf/util/session.h
index 33af571f9d08..8027d6aa5fe4 100644
--- a/tools/perf/util/session.h
+++ b/tools/perf/util/session.h
@@ -51,6 +51,8 @@ int perf_session__peek_event(struct perf_session *session, off_t file_offset,
 
 int perf_session__process_events(struct perf_session *session,
 				 struct perf_tool *tool);
+int perf_session__process_events_mt(struct perf_session *session,
+				    struct perf_tool *tool, void *arg);
 
 int perf_session_queue_event(struct perf_session *s, union perf_event *event,
 			     struct perf_tool *tool, struct perf_sample *sample,
diff --git a/tools/perf/util/tool.h b/tools/perf/util/tool.h
index bb2708bbfaca..a04826bbe991 100644
--- a/tools/perf/util/tool.h
+++ b/tools/perf/util/tool.h
@@ -2,6 +2,7 @@
 #define __PERF_TOOL_H
 
 #include <stdbool.h>
+#include "util/event.h"
 
 struct perf_session;
 union perf_event;
@@ -10,6 +11,7 @@ struct perf_evsel;
 struct perf_sample;
 struct perf_tool;
 struct machine;
+struct hists;
 
 typedef int (*event_sample)(struct perf_tool *tool, union perf_event *event,
 			    struct perf_sample *sample,
@@ -45,4 +47,14 @@ struct perf_tool {
 	bool		ordering_requires_timestamps;
 };
 
+struct perf_tool_mt {
+	struct perf_tool	tool;
+	struct events_stats	stats;
+	struct hists		*hists;
+	struct perf_session	*session;
+	int			idx;
+
+	void			*priv;
+};
+
 #endif /* __PERF_TOOL_H */
-- 
2.2.2


^ permalink raw reply related	[flat|nested] 221+ messages in thread

* [PATCH 34/42] perf tools: Add missing_threads rb tree
  2015-01-29  8:06 [RFC/PATCHSET 00/42] perf tools: Speed-up perf report by using multi thread (v2) Namhyung Kim
                   ` (32 preceding siblings ...)
  2015-01-29  8:07 ` [PATCH 33/42] perf report: Parallelize perf report using multi-thread Namhyung Kim
@ 2015-01-29  8:07 ` Namhyung Kim
  2015-01-29  8:07 ` [PATCH 35/42] perf record: Synthesize COMM event for a command line workload Namhyung Kim
                   ` (8 subsequent siblings)
  42 siblings, 0 replies; 221+ messages in thread
From: Namhyung Kim @ 2015-01-29  8:07 UTC (permalink / raw)
  To: Arnaldo Carvalho de Melo
  Cc: Ingo Molnar, Peter Zijlstra, Jiri Olsa, LKML, David Ahern,
	Adrian Hunter, Andi Kleen, Stephane Eranian, Frederic Weisbecker

Sometimes it's possible to miss certain meta events like fork/exit and
in this case it can fail to find such thread in the machine's rbtree.
But adding a thread to the tree is dangerous since it's now executed
in multi-thread environment otherwise it'll add an overhead in order
to grab a lock for every search.  So adds a separate missing_threads
tree and protect it with a mutex.  It's expected to be accessed only
if a thread is not found in a normal tree.

Signed-off-by: Namhyung Kim <namhyung@kernel.org>
---
 tools/perf/tests/thread-lookup-time.c |   8 ++-
 tools/perf/util/build-id.c            |   9 ++-
 tools/perf/util/machine.c             | 129 +++++++++++++++++++++-------------
 tools/perf/util/machine.h             |   2 +
 tools/perf/util/session.c             |   8 +--
 tools/perf/util/thread.h              |   1 +
 6 files changed, 101 insertions(+), 56 deletions(-)

diff --git a/tools/perf/tests/thread-lookup-time.c b/tools/perf/tests/thread-lookup-time.c
index 6237ecf8caae..04cdde9329d6 100644
--- a/tools/perf/tests/thread-lookup-time.c
+++ b/tools/perf/tests/thread-lookup-time.c
@@ -7,7 +7,9 @@
 static int thread__print_cb(struct thread *th, void *arg __maybe_unused)
 {
 	printf("thread: %d, start time: %"PRIu64" %s\n",
-	       th->tid, th->start_time, th->dead ? "(dead)" : "");
+	       th->tid, th->start_time,
+	       th->dead ? "(dead)" : th->exited ? "(exited)" :
+	       th->missing ? "(missing)" : "");
 	return 0;
 }
 
@@ -105,6 +107,8 @@ static int lookup_with_timestamp(struct machine *machine)
 			machine__findnew_thread_time(machine, 0, 0, 70000) == t3);
 
 	machine__delete_threads(machine);
+	machine__delete_dead_threads(machine);
+	machine__delete_missing_threads(machine);
 	return 0;
 }
 
@@ -146,6 +150,8 @@ static int lookup_without_timestamp(struct machine *machine)
 			machine__findnew_thread_time(machine, 0, 0, -1ULL) == t3);
 
 	machine__delete_threads(machine);
+	machine__delete_dead_threads(machine);
+	machine__delete_missing_threads(machine);
 	return 0;
 }
 
diff --git a/tools/perf/util/build-id.c b/tools/perf/util/build-id.c
index 0c72680a977f..1a37da34d852 100644
--- a/tools/perf/util/build-id.c
+++ b/tools/perf/util/build-id.c
@@ -60,7 +60,14 @@ static int perf_event__exit_del_thread(struct perf_tool *tool __maybe_unused,
 		    event->fork.ppid, event->fork.ptid);
 
 	if (thread) {
-		rb_erase(&thread->rb_node, &machine->threads);
+		if (thread->dead)
+			rb_erase(&thread->rb_node, &machine->dead_threads);
+		else if (thread->missing)
+			rb_erase(&thread->rb_node, &machine->missing_threads);
+		else
+			rb_erase(&thread->rb_node, &machine->threads);
+
+		list_del(&thread->tid_node);
 		machine->last_match = NULL;
 		thread__delete(thread);
 	}
diff --git a/tools/perf/util/machine.c b/tools/perf/util/machine.c
index 7dc044b93cf8..b55454d85f60 100644
--- a/tools/perf/util/machine.c
+++ b/tools/perf/util/machine.c
@@ -29,6 +29,7 @@ int machine__init(struct machine *machine, const char *root_dir, pid_t pid)
 
 	machine->threads = RB_ROOT;
 	machine->dead_threads = RB_ROOT;
+	machine->missing_threads = RB_ROOT;
 	machine->last_match = NULL;
 
 	machine->vdso_info = NULL;
@@ -89,6 +90,19 @@ static void dsos__delete(struct dsos *dsos)
 	}
 }
 
+void machine__delete_missing_threads(struct machine *machine)
+{
+	struct rb_node *nd = rb_first(&machine->missing_threads);
+
+	while (nd) {
+		struct thread *t = rb_entry(nd, struct thread, rb_node);
+
+		nd = rb_next(nd);
+		rb_erase(&t->rb_node, &machine->missing_threads);
+		thread__delete(t);
+	}
+}
+
 void machine__delete_dead_threads(struct machine *machine)
 {
 	struct rb_node *nd = rb_first(&machine->dead_threads);
@@ -434,20 +448,14 @@ struct thread *machine__find_thread(struct machine *machine, pid_t pid,
 	return __machine__findnew_thread(machine, pid, tid, false);
 }
 
-static struct thread *__machine__findnew_thread_time(struct machine *machine,
-						     pid_t pid, pid_t tid,
-						     u64 timestamp, bool create)
+static struct thread *machine__find_dead_thread_time(struct machine *machine,
+						     pid_t pid __maybe_unused,
+						     pid_t tid, u64 timestamp)
 {
-	struct thread *curr, *pos, *new;
-	struct thread *th = NULL;
-	struct rb_node **p;
+	struct thread *th, *pos;
+	struct rb_node **p = &machine->dead_threads.rb_node;
 	struct rb_node *parent = NULL;
 
-	curr = __machine__findnew_thread(machine, pid, tid, false);
-	if (curr && timestamp >= curr->start_time)
-		return curr;
-
-	p = &machine->dead_threads.rb_node;
 	while (*p != NULL) {
 		parent = *p;
 		th = rb_entry(parent, struct thread, rb_node);
@@ -461,10 +469,9 @@ static struct thread *__machine__findnew_thread_time(struct machine *machine,
 				}
 			}
 
-			if (timestamp >= th->start_time) {
-				machine__update_thread_pid(machine, th, pid);
+			if (timestamp >= th->start_time)
 				return th;
-			}
+
 			break;
 		}
 
@@ -474,50 +481,67 @@ static struct thread *__machine__findnew_thread_time(struct machine *machine,
 			p = &(*p)->rb_right;
 	}
 
-	if (!create)
-		return NULL;
+	return NULL;
+}
 
-	if (!curr && !*p)
-		return __machine__findnew_thread(machine, pid, tid, true);
+static struct thread *__machine__findnew_thread_time(struct machine *machine,
+						     pid_t pid, pid_t tid,
+						     u64 timestamp, bool create)
+{
+	struct thread *th, *new = NULL;
+	struct rb_node **p = &machine->missing_threads.rb_node;
+	struct rb_node *parent = NULL;
 
-	new = thread__new(pid, tid);
-	if (new == NULL)
-		return NULL;
+	static pthread_mutex_t missing_thread_lock = PTHREAD_MUTEX_INITIALIZER;
 
-	new->dead = true;
-	new->start_time = timestamp;
+	th = __machine__findnew_thread(machine, pid, tid, false);
+	if (th && timestamp >= th->start_time)
+		return th;
 
-	if (*p) {
-		list_for_each_entry(pos, &th->tid_node, tid_node) {
-			/* sort by time */
-			if (timestamp >= pos->start_time) {
-				th = pos;
-				break;
-			}
+	th = machine__find_dead_thread_time(machine, pid, tid, timestamp);
+	if (th)
+		return th;
+
+	pthread_mutex_lock(&missing_thread_lock);
+
+	while (*p != NULL) {
+		parent = *p;
+		th = rb_entry(parent, struct thread, rb_node);
+
+		if (th->tid == tid) {
+			pthread_mutex_unlock(&missing_thread_lock);
+			return th;
 		}
-		list_add_tail(&new->tid_node, &th->tid_node);
-	} else {
-		rb_link_node(&new->rb_node, parent, p);
-		rb_insert_color(&new->rb_node, &machine->dead_threads);
+
+		if (tid < th->tid)
+			p = &(*p)->rb_left;
+		else
+			p = &(*p)->rb_right;
 	}
 
+	if (!create)
+		goto out;
+
+	new = thread__new(pid, tid);
+	if (new == NULL)
+		goto out;
+
+	/* missing threads are not bothered with timestamp */
+	new->start_time = 0;
+	new->missing = true;
+
 	/*
-	 * We have to initialize map_groups separately
-	 * after rb tree is updated.
-	 *
-	 * The reason is that we call machine__findnew_thread
-	 * within thread__init_map_groups to find the thread
-	 * leader and that would screwed the rb tree.
+	 * missing threads have their own map groups regardless of
+	 * leader for the sake of simplicity.  it's okay since the map
+	 * groups has no map in it anyway.
 	 */
-	if (thread__init_map_groups(new, machine)) {
-		if (!list_empty(&new->tid_node))
-			list_del(&new->tid_node);
-		else
-			rb_erase(&new->rb_node, &machine->dead_threads);
+	new->mg = map_groups__new(machine);
 
-		thread__delete(new);
-		return NULL;
-	}
+	rb_link_node(&new->rb_node, parent, p);
+	rb_insert_color(&new->rb_node, &machine->missing_threads);
+
+out:
+	pthread_mutex_unlock(&missing_thread_lock);
 
 	return new;
 }
@@ -1356,6 +1380,7 @@ static void machine__remove_thread(struct machine *machine, struct thread *th)
 
 	machine->last_match = NULL;
 	rb_erase(&th->rb_node, &machine->threads);
+	RB_CLEAR_NODE(&th->rb_node);
 
 	th->dead = true;
 
@@ -1825,6 +1850,14 @@ int machine__for_each_thread(struct machine *machine,
 				return rc;
 		}
 	}
+
+	for (nd = rb_first(&machine->missing_threads); nd; nd = rb_next(nd)) {
+		thread = rb_entry(nd, struct thread, rb_node);
+		rc = fn(thread, priv);
+		if (rc != 0)
+			return rc;
+	}
+
 	return rc;
 }
 
diff --git a/tools/perf/util/machine.h b/tools/perf/util/machine.h
index 9571b6b1c5b5..40af1f59e360 100644
--- a/tools/perf/util/machine.h
+++ b/tools/perf/util/machine.h
@@ -31,6 +31,7 @@ struct machine {
 	char		  *root_dir;
 	struct rb_root	  threads;
 	struct rb_root	  dead_threads;
+	struct rb_root	  missing_threads;
 	struct thread	  *last_match;
 	struct vdso_info  *vdso_info;
 	struct dsos	  user_dsos;
@@ -116,6 +117,7 @@ void machines__set_comm_exec(struct machines *machines, bool comm_exec);
 struct machine *machine__new_host(void);
 int machine__init(struct machine *machine, const char *root_dir, pid_t pid);
 void machine__exit(struct machine *machine);
+void machine__delete_missing_threads(struct machine *machine);
 void machine__delete_dead_threads(struct machine *machine);
 void machine__delete_threads(struct machine *machine);
 void machine__delete(struct machine *machine);
diff --git a/tools/perf/util/session.c b/tools/perf/util/session.c
index d1d5e0b3a26e..507db51ccfea 100644
--- a/tools/perf/util/session.c
+++ b/tools/perf/util/session.c
@@ -138,14 +138,11 @@ struct perf_session *perf_session__new(struct perf_data_file *file,
 	return NULL;
 }
 
-static void perf_session__delete_dead_threads(struct perf_session *session)
-{
-	machine__delete_dead_threads(&session->machines.host);
-}
-
 static void perf_session__delete_threads(struct perf_session *session)
 {
 	machine__delete_threads(&session->machines.host);
+	machine__delete_dead_threads(&session->machines.host);
+	machine__delete_missing_threads(&session->machines.host);
 }
 
 static void perf_session_env__delete(struct perf_session_env *env)
@@ -167,7 +164,6 @@ static void perf_session_env__delete(struct perf_session_env *env)
 void perf_session__delete(struct perf_session *session)
 {
 	perf_session__destroy_kernel_maps(session);
-	perf_session__delete_dead_threads(session);
 	perf_session__delete_threads(session);
 	perf_session_env__delete(&session->header.env);
 	machines__exit(&session->machines);
diff --git a/tools/perf/util/thread.h b/tools/perf/util/thread.h
index 5209ad5adadf..88fee3d8c0dc 100644
--- a/tools/perf/util/thread.h
+++ b/tools/perf/util/thread.h
@@ -23,6 +23,7 @@ struct thread {
 	bool			comm_set;
 	bool			exited; /* if set thread has exited */
 	bool			dead; /* thread is in dead_threads list */
+	bool			missing; /* thread is in missing_threads list */
 	struct list_head	comm_list;
 	int			comm_len;
 	u64			db_id;
-- 
2.2.2


^ permalink raw reply related	[flat|nested] 221+ messages in thread

* [PATCH 35/42] perf record: Synthesize COMM event for a command line workload
  2015-01-29  8:06 [RFC/PATCHSET 00/42] perf tools: Speed-up perf report by using multi thread (v2) Namhyung Kim
                   ` (33 preceding siblings ...)
  2015-01-29  8:07 ` [PATCH 34/42] perf tools: Add missing_threads rb tree Namhyung Kim
@ 2015-01-29  8:07 ` Namhyung Kim
  2015-01-29  8:07 ` [PATCH 36/42] perf tools: Fix progress ui to support multi thread Namhyung Kim
                   ` (7 subsequent siblings)
  42 siblings, 0 replies; 221+ messages in thread
From: Namhyung Kim @ 2015-01-29  8:07 UTC (permalink / raw)
  To: Arnaldo Carvalho de Melo
  Cc: Ingo Molnar, Peter Zijlstra, Jiri Olsa, LKML, David Ahern,
	Adrian Hunter, Andi Kleen, Stephane Eranian, Frederic Weisbecker

When perf creates a new child to profile, the events are enabled on
exec().  And in this case, it doesn't synthesize any event for the
child since they'll be generated during exec().  But there's an window
between the enabling and the event generation.

It used to be overcome since samples are only in kernel (so we always
have the map) and the comm is overridden by a later COMM event.
However it won't work anymore since those samples will go to a missing
thread now but the COMM event will create a (current) thread.  This
leads to those early samples (like native_write_msr_safe) not having a
comm but pid (like ':15328').

So it needs to synthesize COMM event for the child explicitly before
enabling so that it can have a correct comm.  But at this time, the
comm will be "perf" since it's not exec-ed yet.

Signed-off-by: Namhyung Kim <namhyung@kernel.org>
---
 tools/perf/builtin-record.c | 18 +++++++++++++++++-
 tools/perf/util/event.c     |  8 ++++----
 tools/perf/util/event.h     |  5 +++++
 3 files changed, 26 insertions(+), 5 deletions(-)

diff --git a/tools/perf/builtin-record.c b/tools/perf/builtin-record.c
index 0db47c97446b..9500e350ca95 100644
--- a/tools/perf/builtin-record.c
+++ b/tools/perf/builtin-record.c
@@ -607,8 +607,24 @@ static int __cmd_record(struct record *rec, int argc, const char **argv)
 	/*
 	 * Let the child rip
 	 */
-	if (forks)
+	if (forks) {
+		union perf_event *comm_event;
+
+		comm_event = malloc(sizeof(*comm_event) + machine->id_hdr_size);
+		if (comm_event == NULL)
+			goto out_child;
+
+		err = perf_event__synthesize_comm(tool, comm_event,
+						  rec->evlist->threads->map[0],
+						  process_synthesized_event,
+						  machine);
+		free(comm_event);
+
+		if (err < 0)
+			goto out_child;
+
 		perf_evlist__start_workload(rec->evlist);
+	}
 
 	if (opts->initial_delay) {
 		usleep(opts->initial_delay * 1000);
diff --git a/tools/perf/util/event.c b/tools/perf/util/event.c
index e7152a6e3043..49452852b103 100644
--- a/tools/perf/util/event.c
+++ b/tools/perf/util/event.c
@@ -96,10 +96,10 @@ static pid_t perf_event__get_comm_tgid(pid_t pid, char *comm, size_t len)
 	return tgid;
 }
 
-static pid_t perf_event__synthesize_comm(struct perf_tool *tool,
-					 union perf_event *event, pid_t pid,
-					 perf_event__handler_t process,
-					 struct machine *machine)
+pid_t perf_event__synthesize_comm(struct perf_tool *tool,
+				  union perf_event *event, pid_t pid,
+				  perf_event__handler_t process,
+				  struct machine *machine)
 {
 	size_t size;
 	pid_t tgid;
diff --git a/tools/perf/util/event.h b/tools/perf/util/event.h
index 1f86c279520e..6df23199fea0 100644
--- a/tools/perf/util/event.h
+++ b/tools/perf/util/event.h
@@ -386,6 +386,11 @@ int perf_event__synthesize_mmap_events(struct perf_tool *tool,
 				       struct machine *machine,
 				       bool mmap_data);
 
+pid_t perf_event__synthesize_comm(struct perf_tool *tool,
+				  union perf_event *event, pid_t pid,
+				  perf_event__handler_t process,
+				  struct machine *machine);
+
 size_t perf_event__fprintf_comm(union perf_event *event, FILE *fp);
 size_t perf_event__fprintf_mmap(union perf_event *event, FILE *fp);
 size_t perf_event__fprintf_mmap2(union perf_event *event, FILE *fp);
-- 
2.2.2


^ permalink raw reply related	[flat|nested] 221+ messages in thread

* [PATCH 36/42] perf tools: Fix progress ui to support multi thread
  2015-01-29  8:06 [RFC/PATCHSET 00/42] perf tools: Speed-up perf report by using multi thread (v2) Namhyung Kim
                   ` (34 preceding siblings ...)
  2015-01-29  8:07 ` [PATCH 35/42] perf record: Synthesize COMM event for a command line workload Namhyung Kim
@ 2015-01-29  8:07 ` Namhyung Kim
  2015-01-29  8:07 ` [PATCH 37/42] perf report: Add --multi-thread option and config item Namhyung Kim
                   ` (6 subsequent siblings)
  42 siblings, 0 replies; 221+ messages in thread
From: Namhyung Kim @ 2015-01-29  8:07 UTC (permalink / raw)
  To: Arnaldo Carvalho de Melo
  Cc: Ingo Molnar, Peter Zijlstra, Jiri Olsa, LKML, David Ahern,
	Adrian Hunter, Andi Kleen, Stephane Eranian, Frederic Weisbecker

Split ui_progress struct into global and local one.  Each thread
updates local struct without lock and only updates global one if
meaningful progress is done (with lock).

To do that, pass struct ui_progress to __perf_session__process_event()
and set it for the total size of multi-file storage.

Signed-off-by: Namhyung Kim <namhyung@kernel.org>
---
 tools/perf/util/hist.c    |  5 ++--
 tools/perf/util/hist.h    |  3 ++-
 tools/perf/util/session.c | 63 +++++++++++++++++++++++++++++++++++++----------
 tools/perf/util/tool.h    |  3 +++
 4 files changed, 58 insertions(+), 16 deletions(-)

diff --git a/tools/perf/util/hist.c b/tools/perf/util/hist.c
index 14d4b9358ac6..dab3b8b3a06a 100644
--- a/tools/perf/util/hist.c
+++ b/tools/perf/util/hist.c
@@ -1062,12 +1062,13 @@ void hists__collapse_resort(struct hists *hists, struct ui_progress *prog)
 	__hists__collapse_resort(hists, root, prog);
 }
 
-void hists__mt_resort(struct hists *dst, struct hists *src)
+void hists__mt_resort(struct hists *dst, struct hists *src,
+		      struct ui_progress *prog)
 {
 	struct rb_root *root = src->entries_in;
 
 	sort__need_collapse = 1;
-	__hists__collapse_resort(dst, root, NULL);
+	__hists__collapse_resort(dst, root, prog);
 }
 
 static int hist_entry__sort(struct hist_entry *a, struct hist_entry *b)
diff --git a/tools/perf/util/hist.h b/tools/perf/util/hist.h
index 2c29d70b2cfe..94179107906a 100644
--- a/tools/perf/util/hist.h
+++ b/tools/perf/util/hist.h
@@ -124,7 +124,8 @@ int hist_entry__sort_snprintf(struct hist_entry *he, char *bf, size_t size,
 void hist_entry__delete(struct hist_entry *he);
 
 void hists__output_resort(struct hists *hists, struct ui_progress *prog);
-void hists__mt_resort(struct hists *dst, struct hists *src);
+void hists__mt_resort(struct hists *dst, struct hists *src,
+		      struct ui_progress *prog);
 void hists__collapse_resort(struct hists *hists, struct ui_progress *prog);
 
 void hists__decay_entries(struct hists *hists, bool zap_user, bool zap_kernel);
diff --git a/tools/perf/util/session.c b/tools/perf/util/session.c
index 507db51ccfea..3596bb608f3c 100644
--- a/tools/perf/util/session.c
+++ b/tools/perf/util/session.c
@@ -1260,14 +1260,14 @@ fetch_mmaped_event(struct perf_session *session,
 static int __perf_session__process_events(struct perf_session *session,
 					  struct events_stats *stats, int fd,
 					  u64 data_offset, u64 data_size,
-					  u64 file_size, struct perf_tool *tool)
+					  u64 file_size, struct perf_tool *tool,
+					  struct ui_progress *prog)
 {
 	u64 head, page_offset, file_offset, file_pos, size;
 	int err, mmap_prot, mmap_flags, map_idx = 0;
 	size_t	mmap_size;
 	char *buf, *mmaps[NUM_MMAPS];
 	union perf_event *event;
-	struct ui_progress prog;
 	s64 skip;
 
 	perf_tool__fill_defaults(tool);
@@ -1279,8 +1279,6 @@ static int __perf_session__process_events(struct perf_session *session,
 	if (data_size && (data_offset + data_size < file_size))
 		file_size = data_offset + data_size;
 
-	ui_progress__init(&prog, file_size, "Processing events...");
-
 	mmap_size = MMAP_SIZE;
 	if (mmap_size > file_size) {
 		mmap_size = file_size;
@@ -1346,7 +1344,7 @@ static int __perf_session__process_events(struct perf_session *session,
 	head += size;
 	file_pos += size;
 
-	ui_progress__update(&prog, size);
+	ui_progress__update(prog, size);
 
 	if (session_done())
 		goto out;
@@ -1358,7 +1356,6 @@ static int __perf_session__process_events(struct perf_session *session,
 	/* do the final flush for ordered samples */
 	err = ordered_events__flush(session, tool, OE_FLUSH__FINAL);
 out_err:
-	ui_progress__finish();
 	ordered_events__free(&session->ordered_events);
 	session->one_mmap = false;
 	return err;
@@ -1367,6 +1364,7 @@ static int __perf_session__process_events(struct perf_session *session,
 int perf_session__process_events(struct perf_session *session,
 				 struct perf_tool *tool)
 {
+	struct ui_progress prog;
 	struct perf_data_file *file = session->file;
 	u64 size = perf_data_file__size(file);
 	int err, i;
@@ -1377,11 +1375,13 @@ int perf_session__process_events(struct perf_session *session,
 	if (perf_data_file__is_pipe(file))
 		return __perf_session__process_pipe_events(session, tool);
 
+	ui_progress__init(&prog, size, "Processing events...");
+
 	err = __perf_session__process_events(session, &session->stats,
 					     perf_data_file__fd(file),
 					     session->header.data_offset,
 					     session->header.data_size,
-					     size, tool);
+					     size, tool, &prog);
 
 	if (err < 0 || !perf_session__has_index(session))
 		return err;
@@ -1400,7 +1400,7 @@ int perf_session__process_events(struct perf_session *session,
 						perf_data_file__fd(file),
 						session->header.index[i].offset,
 						session->header.index[i].size,
-						size, tool);
+						size, tool, &prog);
 		if (err < 0)
 			break;
 	}
@@ -1408,6 +1408,29 @@ int perf_session__process_events(struct perf_session *session,
 	return err;
 }
 
+struct ui_progress_ops *orig_progress__ops;
+
+static void mt_progress__update(struct ui_progress *p)
+{
+	struct perf_tool_mt *mt_tool = container_of(p, struct perf_tool_mt, prog);
+	struct ui_progress *gprog = mt_tool->global_prog;
+	static pthread_mutex_t prog_lock = PTHREAD_MUTEX_INITIALIZER;
+
+	pthread_mutex_lock(&prog_lock);
+
+	gprog->curr += p->step;
+	if (gprog->curr >= gprog->next) {
+		gprog->next += gprog->step;
+		orig_progress__ops->update(gprog);
+	}
+
+	pthread_mutex_unlock(&prog_lock);
+}
+
+static struct ui_progress_ops mt_progress__ops = {
+	.update = mt_progress__update,
+};
+
 static void *processing_thread_idx(void *arg)
 {
 	struct perf_tool_mt *mt_tool = arg;
@@ -1417,10 +1440,12 @@ static void *processing_thread_idx(void *arg)
 	u64 size = session->header.index[mt_tool->idx].size;
 	u64 file_size = perf_data_file__size(session->file);
 
+	ui_progress__init(&mt_tool->prog, size, "");
+
 	pr_debug("processing samples using thread [%d]\n", mt_tool->idx);
 	if (__perf_session__process_events(session, &mt_tool->stats,
 					   fd, offset, size, file_size,
-					   &mt_tool->tool) < 0) {
+					   &mt_tool->tool, &mt_tool->prog) < 0) {
 		pr_err("processing samples failed (thread [%d)\n", mt_tool->idx);
 		return NULL;
 	}
@@ -1438,7 +1463,8 @@ int perf_session__process_events_mt(struct perf_session *session,
 	u64 nr_entries = 0;
 	struct perf_tool_mt *mt_tools = NULL;
 	struct perf_tool_mt *mt;
-	pthread_t *th_id;
+	struct ui_progress prog;
+	pthread_t *th_id = NULL;
 	int err, i, k;
 	int nr_index = session->header.nr_index;
 	u64 size = perf_data_file__size(file);
@@ -1451,13 +1477,19 @@ int perf_session__process_events_mt(struct perf_session *session,
 		return -EINVAL;
 	}
 
+	ui_progress__init(&prog, size, "Processing events...");
+
 	err = __perf_session__process_events(session, &session->stats,
 					     perf_data_file__fd(file),
 					     session->header.data_offset,
 					     session->header.data_size,
-					     size, tool);
+					     size, tool, &prog);
 	if (err)
-		return err;
+		goto out;
+
+	orig_progress__ops = ui_progress__ops;
+	ui_progress__ops = &mt_progress__ops;
+	ui_progress__ops->finish = orig_progress__ops->finish;
 
 	th_id = calloc(nr_index, sizeof(*th_id));
 	if (th_id == NULL)
@@ -1483,6 +1515,7 @@ int perf_session__process_events_mt(struct perf_session *session,
 		mt->tool.ordered_events = false;
 		mt->idx = i;
 		mt->priv = arg;
+		mt->global_prog = &prog;
 
 		pthread_create(&th_id[i], NULL, processing_thread_idx, mt);
 	}
@@ -1506,6 +1539,9 @@ int perf_session__process_events_mt(struct perf_session *session,
 		}
 	}
 
+	ui_progress__ops = orig_progress__ops;
+	ui_progress__init(&prog, nr_entries, "Merging related events...");
+
 	for (i = 0; i < nr_index; i++) {
 		mt = &mt_tools[i];
 
@@ -1515,7 +1551,7 @@ int perf_session__process_events_mt(struct perf_session *session,
 			if (perf_evsel__is_dummy_tracking(evsel))
 				continue;
 
-			hists__mt_resort(hists, &mt->hists[evsel->idx]);
+			hists__mt_resort(hists, &mt->hists[evsel->idx], &prog);
 
 			/* Non-group events are considered as leader */
 			if (symbol_conf.event_group &&
@@ -1530,6 +1566,7 @@ int perf_session__process_events_mt(struct perf_session *session,
 	}
 
 out:
+	ui_progress__finish();
 	events_stats__warn_about_errors(&session->stats, tool);
 
 	if (mt_tools) {
diff --git a/tools/perf/util/tool.h b/tools/perf/util/tool.h
index a04826bbe991..aa7f110b9425 100644
--- a/tools/perf/util/tool.h
+++ b/tools/perf/util/tool.h
@@ -3,6 +3,7 @@
 
 #include <stdbool.h>
 #include "util/event.h"
+#include "ui/progress.h"
 
 struct perf_session;
 union perf_event;
@@ -52,6 +53,8 @@ struct perf_tool_mt {
 	struct events_stats	stats;
 	struct hists		*hists;
 	struct perf_session	*session;
+	struct ui_progress	prog;
+	struct ui_progress	*global_prog;
 	int			idx;
 
 	void			*priv;
-- 
2.2.2


^ permalink raw reply related	[flat|nested] 221+ messages in thread

* [PATCH 37/42] perf report: Add --multi-thread option and config item
  2015-01-29  8:06 [RFC/PATCHSET 00/42] perf tools: Speed-up perf report by using multi thread (v2) Namhyung Kim
                   ` (35 preceding siblings ...)
  2015-01-29  8:07 ` [PATCH 36/42] perf tools: Fix progress ui to support multi thread Namhyung Kim
@ 2015-01-29  8:07 ` Namhyung Kim
  2015-01-29  8:07 ` [PATCH 38/42] perf session: Handle index files generally Namhyung Kim
                   ` (5 subsequent siblings)
  42 siblings, 0 replies; 221+ messages in thread
From: Namhyung Kim @ 2015-01-29  8:07 UTC (permalink / raw)
  To: Arnaldo Carvalho de Melo
  Cc: Ingo Molnar, Peter Zijlstra, Jiri Olsa, LKML, David Ahern,
	Adrian Hunter, Andi Kleen, Stephane Eranian, Frederic Weisbecker

The --multi-thread option is to enable parallel processing so user can
force serial processing even for indexed data file.  It default to false
for now but users also can changes this by setting "report.multi_thread"
config option in ~/.perfconfig file.

Signed-off-by: Namhyung Kim <namhyung@kernel.org>
---
 tools/perf/Documentation/perf-report.txt |  3 ++
 tools/perf/builtin-report.c              | 66 +++++++++++++++++++++++++++-----
 tools/perf/util/session.c                |  1 +
 3 files changed, 61 insertions(+), 9 deletions(-)

diff --git a/tools/perf/Documentation/perf-report.txt b/tools/perf/Documentation/perf-report.txt
index dd7cccdde498..e00077a658c1 100644
--- a/tools/perf/Documentation/perf-report.txt
+++ b/tools/perf/Documentation/perf-report.txt
@@ -318,6 +318,9 @@ OPTIONS
 --header-only::
 	Show only perf.data header (forces --stdio).
 
+--multi-thread::
+	Speed up report by parallelizing sample processing using multi-thread.
+
 SEE ALSO
 --------
 linkperf:perf-stat[1], linkperf:perf-annotate[1]
diff --git a/tools/perf/builtin-report.c b/tools/perf/builtin-report.c
index 8a40c79d9273..b0539c017898 100644
--- a/tools/perf/builtin-report.c
+++ b/tools/perf/builtin-report.c
@@ -51,6 +51,7 @@ struct report {
 	bool			mem_mode;
 	bool			header;
 	bool			header_only;
+	bool			multi_thread;
 	int			max_stack;
 	struct perf_read_values	show_threads_values;
 	const char		*pretty_printing_style;
@@ -82,6 +83,10 @@ static int report__config(const char *var, const char *value, void *cb)
 		rep->queue_size = perf_config_u64(var, value);
 		return 0;
 	}
+	if (!strcmp(var, "report.multi-thread")) {
+		rep->multi_thread = perf_config_bool(var, value);
+		return 0;
+	}
 
 	return perf_default_config(var, value, cb);
 }
@@ -128,13 +133,14 @@ static int hist_iter__report_callback(struct hist_entry_iter *iter,
 	return err;
 }
 
-static int process_sample_event(struct perf_tool *tool,
-				union perf_event *event,
-				struct perf_sample *sample,
-				struct perf_evsel *evsel,
-				struct machine *machine)
+static int __process_sample_event(struct perf_tool *tool __maybe_unused,
+				  union perf_event *event,
+				  struct perf_sample *sample,
+				  struct perf_evsel *evsel,
+				  struct machine *machine,
+				  struct hists *hists,
+				  struct report *rep)
 {
-	struct report *rep = container_of(tool, struct report, tool);
 	struct addr_location al;
 	struct hist_entry_iter iter = {
 		.hide_unresolved = rep->hide_unresolved,
@@ -167,7 +173,7 @@ static int process_sample_event(struct perf_tool *tool,
 	if (al.map != NULL)
 		al.map->dso->hit = 1;
 
-	ret = hist_entry_iter__add(&iter, evsel__hists(evsel), evsel, &al,
+	ret = hist_entry_iter__add(&iter, hists, evsel, &al,
 				   sample, rep->max_stack, rep);
 	if (ret < 0)
 		pr_debug("problem adding hist entry, skipping event\n");
@@ -175,6 +181,31 @@ static int process_sample_event(struct perf_tool *tool,
 	return ret;
 }
 
+static int process_sample_event(struct perf_tool *tool,
+				union perf_event *event,
+				struct perf_sample *sample,
+				struct perf_evsel *evsel,
+				struct machine *machine)
+{
+	struct report *rep = container_of(tool, struct report, tool);
+
+	return __process_sample_event(tool, event, sample, evsel, machine,
+				      evsel__hists(evsel), rep);
+}
+
+static int process_sample_event_mt(struct perf_tool *tool,
+				   union perf_event *event,
+				   struct perf_sample *sample,
+				   struct perf_evsel *evsel,
+				   struct machine *machine)
+{
+	struct perf_tool_mt *mt = container_of(tool, struct perf_tool_mt, tool);
+	struct report *rep = mt->priv;
+
+	return __process_sample_event(tool, event, sample, evsel, machine,
+				      &mt->hists[evsel->idx], rep);
+}
+
 static int process_read_event(struct perf_tool *tool,
 			      union perf_event *event,
 			      struct perf_sample *sample __maybe_unused,
@@ -484,7 +515,12 @@ static int __cmd_report(struct report *rep)
 	if (ret)
 		return ret;
 
-	ret = perf_session__process_events(session, &rep->tool);
+	if (rep->multi_thread) {
+		rep->tool.sample = process_sample_event_mt;
+		ret = perf_session__process_events_mt(session, &rep->tool, rep);
+	} else {
+		ret = perf_session__process_events(session, &rep->tool);
+	}
 	if (ret)
 		return ret;
 
@@ -507,7 +543,12 @@ static int __cmd_report(struct report *rep)
 		}
 	}
 
-	report__collapse_hists(rep);
+	/*
+	 * For multi-thread report, it already calls hists__mt_resort()
+	 * so no need to collapse here.
+	 */
+	if (!rep->multi_thread)
+		report__collapse_hists(rep);
 
 	if (session_done())
 		return 0;
@@ -715,6 +756,8 @@ int cmd_report(int argc, const char **argv, const char *prefix __maybe_unused)
 		     "Don't show entries under that percent", parse_percent_limit),
 	OPT_CALLBACK(0, "percentage", NULL, "relative|absolute",
 		     "how to display percentage of filtered entries", parse_filter_percentage),
+	OPT_BOOLEAN(0, "multi-thread", &report.multi_thread,
+		    "Speed up sample processing using multi-thead"),
 	OPT_END()
 	};
 	struct perf_data_file file = {
@@ -759,6 +802,11 @@ int cmd_report(int argc, const char **argv, const char *prefix __maybe_unused)
 					       report.queue_size);
 	}
 
+	if (report.multi_thread && !perf_session__has_index(session)) {
+		pr_debug("fallback to single thread for normal data file.\n");
+		report.multi_thread = false;
+	}
+
 	report.session = session;
 
 	has_br_stack = perf_header__has_feat(&session->header,
diff --git a/tools/perf/util/session.c b/tools/perf/util/session.c
index 3596bb608f3c..6d34c880010f 100644
--- a/tools/perf/util/session.c
+++ b/tools/perf/util/session.c
@@ -1586,6 +1586,7 @@ int perf_session__process_events_mt(struct perf_session *session,
 
 	goto out;
 }
+
 bool perf_session__has_traces(struct perf_session *session, const char *msg)
 {
 	struct perf_evsel *evsel;
-- 
2.2.2


^ permalink raw reply related	[flat|nested] 221+ messages in thread

* [PATCH 38/42] perf session: Handle index files generally
  2015-01-29  8:06 [RFC/PATCHSET 00/42] perf tools: Speed-up perf report by using multi thread (v2) Namhyung Kim
                   ` (36 preceding siblings ...)
  2015-01-29  8:07 ` [PATCH 37/42] perf report: Add --multi-thread option and config item Namhyung Kim
@ 2015-01-29  8:07 ` Namhyung Kim
  2015-01-29  8:07 ` [PATCH 39/42] perf tools: Convert lseek + read to pread Namhyung Kim
                   ` (4 subsequent siblings)
  42 siblings, 0 replies; 221+ messages in thread
From: Namhyung Kim @ 2015-01-29  8:07 UTC (permalink / raw)
  To: Arnaldo Carvalho de Melo
  Cc: Ingo Molnar, Peter Zijlstra, Jiri Olsa, LKML, David Ahern,
	Adrian Hunter, Andi Kleen, Stephane Eranian, Frederic Weisbecker

The current code assumes that the number of index item and cpu are
matched so it creates that number of threads.  But it's not the case
of non-system-wide session or data came from different machine.

Just creates threads at most number of online cpus and process data.

Signed-off-by: Namhyung Kim <namhyung@kernel.org>
---
 tools/perf/util/session.c | 68 ++++++++++++++++++++++++++++++++++-------------
 tools/perf/util/tool.h    |  1 -
 2 files changed, 50 insertions(+), 19 deletions(-)

diff --git a/tools/perf/util/session.c b/tools/perf/util/session.c
index 6d34c880010f..ccf9371ef292 100644
--- a/tools/perf/util/session.c
+++ b/tools/perf/util/session.c
@@ -1431,26 +1431,51 @@ static struct ui_progress_ops mt_progress__ops = {
 	.update = mt_progress__update,
 };
 
+static int perf_session__get_index(struct perf_session *session)
+{
+	int ret;
+	static unsigned index;
+	static pthread_mutex_t idx_lock = PTHREAD_MUTEX_INITIALIZER;
+
+	pthread_mutex_lock(&idx_lock);
+	if (index < session->header.nr_index)
+		ret = index++;
+	else
+		ret = -1;
+	pthread_mutex_unlock(&idx_lock);
+
+	return ret;
+}
+
 static void *processing_thread_idx(void *arg)
 {
 	struct perf_tool_mt *mt_tool = arg;
 	struct perf_session *session = mt_tool->session;
 	int fd = perf_data_file__fd(session->file);
-	u64 offset = session->header.index[mt_tool->idx].offset;
-	u64 size = session->header.index[mt_tool->idx].size;
 	u64 file_size = perf_data_file__size(session->file);
+	int idx;
 
-	ui_progress__init(&mt_tool->prog, size, "");
+	while ((idx = perf_session__get_index(session)) >= 0) {
+		u64 offset = session->header.index[idx].offset;
+		u64 size = session->header.index[idx].size;
+		struct perf_tool_mt *mtt = &mt_tool[idx];
 
-	pr_debug("processing samples using thread [%d]\n", mt_tool->idx);
-	if (__perf_session__process_events(session, &mt_tool->stats,
-					   fd, offset, size, file_size,
-					   &mt_tool->tool, &mt_tool->prog) < 0) {
-		pr_err("processing samples failed (thread [%d)\n", mt_tool->idx);
-		return NULL;
+		if (size == 0)
+			continue;
+
+		pr_debug("processing samples [index %d]\n", idx);
+
+		ui_progress__init(&mtt->prog, size, "");
+
+		if (__perf_session__process_events(session, &mtt->stats,
+						   fd, offset, size, file_size,
+						   &mtt->tool, &mtt->prog) < 0) {
+			pr_err("processing samples failed [index %d]\n", idx);
+			return NULL;
+		}
+		pr_debug("processing samples done [index %d]\n", idx);
 	}
 
-	pr_debug("processing samples done for thread [%d]\n", mt_tool->idx);
 	return arg;
 }
 
@@ -1468,6 +1493,7 @@ int perf_session__process_events_mt(struct perf_session *session,
 	int err, i, k;
 	int nr_index = session->header.nr_index;
 	u64 size = perf_data_file__size(file);
+	int nr_thread = sysconf(_SC_NPROCESSORS_ONLN);
 
 	if (perf_session__register_idle_thread(session) == NULL)
 		return -ENOMEM;
@@ -1491,10 +1517,6 @@ int perf_session__process_events_mt(struct perf_session *session,
 	ui_progress__ops = &mt_progress__ops;
 	ui_progress__ops->finish = orig_progress__ops->finish;
 
-	th_id = calloc(nr_index, sizeof(*th_id));
-	if (th_id == NULL)
-		goto out;
-
 	mt_tools = calloc(nr_index, sizeof(*mt_tools));
 	if (mt_tools == NULL)
 		goto out;
@@ -1513,20 +1535,30 @@ int perf_session__process_events_mt(struct perf_session *session,
 
 		mt->session = session;
 		mt->tool.ordered_events = false;
-		mt->idx = i;
 		mt->priv = arg;
 		mt->global_prog = &prog;
-
-		pthread_create(&th_id[i], NULL, processing_thread_idx, mt);
 	}
 
-	for (i = 0; i < nr_index; i++) {
+	if (nr_thread > nr_index)
+		nr_thread = nr_index;
+
+	th_id = calloc(nr_thread, sizeof(*th_id));
+	if (th_id == NULL)
+		goto out;
+
+	for (i = 0; i < nr_thread; i++)
+		pthread_create(&th_id[i], NULL, processing_thread_idx, mt_tools);
+
+	for (i = 0; i < nr_thread; i++) {
 		pthread_join(th_id[i], (void **)&mt);
 		if (mt == NULL) {
 			err = -EINVAL;
 			continue;
 		}
+	}
 
+	for (i = 0; i < nr_index; i++) {
+		mt = &mt_tools[i];
 		events_stats__add(&session->stats, &mt->stats);
 
 		evlist__for_each(evlist, evsel) {
diff --git a/tools/perf/util/tool.h b/tools/perf/util/tool.h
index aa7f110b9425..e52c936d1b9e 100644
--- a/tools/perf/util/tool.h
+++ b/tools/perf/util/tool.h
@@ -55,7 +55,6 @@ struct perf_tool_mt {
 	struct perf_session	*session;
 	struct ui_progress	prog;
 	struct ui_progress	*global_prog;
-	int			idx;
 
 	void			*priv;
 };
-- 
2.2.2


^ permalink raw reply related	[flat|nested] 221+ messages in thread

* [PATCH 39/42] perf tools: Convert lseek + read to pread
  2015-01-29  8:06 [RFC/PATCHSET 00/42] perf tools: Speed-up perf report by using multi thread (v2) Namhyung Kim
                   ` (37 preceding siblings ...)
  2015-01-29  8:07 ` [PATCH 38/42] perf session: Handle index files generally Namhyung Kim
@ 2015-01-29  8:07 ` Namhyung Kim
  2015-01-30 18:34   ` [tip:perf/core] perf symbols: " tip-bot for Namhyung Kim
  2015-01-29  8:07 ` [PATCH 40/42] perf callchain: Save eh/debug frame offset for dwarf unwind Namhyung Kim
                   ` (3 subsequent siblings)
  42 siblings, 1 reply; 221+ messages in thread
From: Namhyung Kim @ 2015-01-29  8:07 UTC (permalink / raw)
  To: Arnaldo Carvalho de Melo
  Cc: Ingo Molnar, Peter Zijlstra, Jiri Olsa, LKML, David Ahern,
	Adrian Hunter, Andi Kleen, Stephane Eranian, Frederic Weisbecker

When dso_cache__read() is called, it reads data from the given offset
using lseek + normal read syscall.  It can be combined to a single
pread syscall.

Signed-off-by: Namhyung Kim <namhyung@kernel.org>
---
 tools/perf/util/dso.c | 5 +----
 1 file changed, 1 insertion(+), 4 deletions(-)

diff --git a/tools/perf/util/dso.c b/tools/perf/util/dso.c
index ae92046ae2c8..b6ad22b3c6f2 100644
--- a/tools/perf/util/dso.c
+++ b/tools/perf/util/dso.c
@@ -566,10 +566,7 @@ dso_cache__read(struct dso *dso, struct machine *machine,
 		}
 	}
 
-	if (-1 == lseek(dso->data.fd, cache_offset, SEEK_SET))
-		goto err_unlock;
-
-	ret = read(dso->data.fd, cache->data, DSO__DATA_CACHE_SIZE);
+	ret = pread(dso->data.fd, cache->data, DSO__DATA_CACHE_SIZE, cache_offset);
 	if (ret <= 0)
 		goto err_unlock;
 
-- 
2.2.2


^ permalink raw reply related	[flat|nested] 221+ messages in thread

* [PATCH 40/42] perf callchain: Save eh/debug frame offset for dwarf unwind
  2015-01-29  8:06 [RFC/PATCHSET 00/42] perf tools: Speed-up perf report by using multi thread (v2) Namhyung Kim
                   ` (38 preceding siblings ...)
  2015-01-29  8:07 ` [PATCH 39/42] perf tools: Convert lseek + read to pread Namhyung Kim
@ 2015-01-29  8:07 ` Namhyung Kim
  2015-01-29 12:38   ` Arnaldo Carvalho de Melo
                     ` (2 more replies)
  2015-01-29  8:07 ` [PATCH 41/42] perf tools: Add new perf data command Namhyung Kim
                   ` (2 subsequent siblings)
  42 siblings, 3 replies; 221+ messages in thread
From: Namhyung Kim @ 2015-01-29  8:07 UTC (permalink / raw)
  To: Arnaldo Carvalho de Melo
  Cc: Ingo Molnar, Peter Zijlstra, Jiri Olsa, LKML, David Ahern,
	Adrian Hunter, Andi Kleen, Stephane Eranian, Frederic Weisbecker

When libunwind tries to resolve callchains it needs to know the offset
of .eh_frame_hdr or .debug_frame to access the dso.  Since it calls
dso__data_fd(), it'll try to grab dso->lock everytime for same
information.  So save it to dso_data struct and reuse it.

Note that there's a window between dso__data_fd() and actual use of
the fd.  The fd could be closed by other threads to deal with the open
file limit in dso cache code.  But I think it's ok since in that case
elf_section_offset() will return 0 so it'll be tried in next acess.

Signed-off-by: Namhyung Kim <namhyung@kernel.org>
---
 tools/perf/util/dso.h              |  1 +
 tools/perf/util/unwind-libunwind.c | 31 ++++++++++++++++++++-----------
 2 files changed, 21 insertions(+), 11 deletions(-)

diff --git a/tools/perf/util/dso.h b/tools/perf/util/dso.h
index c18fcc0e8081..323ee08d56fc 100644
--- a/tools/perf/util/dso.h
+++ b/tools/perf/util/dso.h
@@ -141,6 +141,7 @@ struct dso {
 		u32		 status_seen;
 		size_t		 file_size;
 		struct list_head open_entry;
+		u64		 frame_offset;
 	} data;
 
 	union { /* Tool specific area */
diff --git a/tools/perf/util/unwind-libunwind.c b/tools/perf/util/unwind-libunwind.c
index 7ed6eaf232b6..3219b20837b5 100644
--- a/tools/perf/util/unwind-libunwind.c
+++ b/tools/perf/util/unwind-libunwind.c
@@ -266,14 +266,17 @@ static int read_unwind_spec_eh_frame(struct dso *dso, struct machine *machine,
 				     u64 *fde_count)
 {
 	int ret = -EINVAL, fd;
-	u64 offset;
+	u64 offset = dso->data.frame_offset;
 
-	fd = dso__data_fd(dso, machine);
-	if (fd < 0)
-		return -EINVAL;
+	if (offset == 0) {
+		fd = dso__data_fd(dso, machine);
+		if (fd < 0)
+			return -EINVAL;
 
-	/* Check the .eh_frame section for unwinding info */
-	offset = elf_section_offset(fd, ".eh_frame_hdr");
+		/* Check the .eh_frame section for unwinding info */
+		offset = elf_section_offset(fd, ".eh_frame_hdr");
+		dso->data.frame_offset = offset;
+	}
 
 	if (offset)
 		ret = unwind_spec_ehframe(dso, machine, offset,
@@ -287,14 +290,20 @@ static int read_unwind_spec_eh_frame(struct dso *dso, struct machine *machine,
 static int read_unwind_spec_debug_frame(struct dso *dso,
 					struct machine *machine, u64 *offset)
 {
-	int fd = dso__data_fd(dso, machine);
+	int fd;
+	u64 ofs = dso->data.frame_offset;
 
-	if (fd < 0)
-		return -EINVAL;
+	if (ofs == 0) {
+		fd = dso__data_fd(dso, machine);
+		if (fd < 0)
+			return -EINVAL;
 
-	/* Check the .debug_frame section for unwinding info */
-	*offset = elf_section_offset(fd, ".debug_frame");
+		/* Check the .debug_frame section for unwinding info */
+		ofs = elf_section_offset(fd, ".debug_frame");
+		dso->data.frame_offset = ofs;
+	}
 
+	*offset = ofs;
 	if (*offset)
 		return 0;
 
-- 
2.2.2


^ permalink raw reply related	[flat|nested] 221+ messages in thread

* [PATCH 41/42] perf tools: Add new perf data command
  2015-01-29  8:06 [RFC/PATCHSET 00/42] perf tools: Speed-up perf report by using multi thread (v2) Namhyung Kim
                   ` (39 preceding siblings ...)
  2015-01-29  8:07 ` [PATCH 40/42] perf callchain: Save eh/debug frame offset for dwarf unwind Namhyung Kim
@ 2015-01-29  8:07 ` Namhyung Kim
  2015-01-29  8:07 ` [PATCH 42/42] perf data: Implement 'index' subcommand Namhyung Kim
  2015-01-29 19:56 ` [RFC/PATCHSET 00/42] perf tools: Speed-up perf report by using multi thread (v2) Arnaldo Carvalho de Melo
  42 siblings, 0 replies; 221+ messages in thread
From: Namhyung Kim @ 2015-01-29  8:07 UTC (permalink / raw)
  To: Arnaldo Carvalho de Melo
  Cc: Ingo Molnar, Peter Zijlstra, Jiri Olsa, LKML, David Ahern,
	Adrian Hunter, Andi Kleen, Stephane Eranian, Frederic Weisbecker,
	Jiri Olsa, Sebastian Andrzej Siewior, Jiri Olsa

From: Jiri Olsa <namhyung@kernel.org>

Adding new 'perf data' command to provide operations over
data files.

The 'perf data convert' sub command is coming in following
patch, but there's possibility for other useful commands
like 'perf data ls' (to display perf data file in directory
in ls style).

Signed-off-by: Sebastian Andrzej Siewior <bigeasy@linutronix.de>
Signed-off-by: Jiri Olsa <jolsa@kernel.org>
Signed-off-by: Namhyung Kim <namhyung@kernel.org>
---
 tools/perf/Documentation/perf-data.txt | 15 +++++++
 tools/perf/Makefile.perf               |  1 +
 tools/perf/builtin-data.c              | 75 ++++++++++++++++++++++++++++++++++
 tools/perf/builtin.h                   |  1 +
 tools/perf/command-list.txt            |  1 +
 tools/perf/perf.c                      |  1 +
 6 files changed, 94 insertions(+)
 create mode 100644 tools/perf/Documentation/perf-data.txt
 create mode 100644 tools/perf/builtin-data.c

diff --git a/tools/perf/Documentation/perf-data.txt b/tools/perf/Documentation/perf-data.txt
new file mode 100644
index 000000000000..b8c83947715c
--- /dev/null
+++ b/tools/perf/Documentation/perf-data.txt
@@ -0,0 +1,15 @@
+perf-data(1)
+==============
+
+NAME
+----
+perf-data - Data file related processing
+
+SYNOPSIS
+--------
+[verse]
+'perf data' [<common options>] <command> [<options>]",
+
+DESCRIPTION
+-----------
+Data file related processing.
diff --git a/tools/perf/Makefile.perf b/tools/perf/Makefile.perf
index 2f8c8b918cac..16ff21fbde55 100644
--- a/tools/perf/Makefile.perf
+++ b/tools/perf/Makefile.perf
@@ -498,6 +498,7 @@ BUILTIN_OBJS += $(OUTPUT)builtin-kvm.o
 BUILTIN_OBJS += $(OUTPUT)builtin-inject.o
 BUILTIN_OBJS += $(OUTPUT)tests/builtin-test.o
 BUILTIN_OBJS += $(OUTPUT)builtin-mem.o
+BUILTIN_OBJS += $(OUTPUT)builtin-data.o
 
 PERFLIBS = $(LIB_FILE) $(LIBAPIKFS) $(LIBTRACEEVENT)
 
diff --git a/tools/perf/builtin-data.c b/tools/perf/builtin-data.c
new file mode 100644
index 000000000000..1eee97d020fa
--- /dev/null
+++ b/tools/perf/builtin-data.c
@@ -0,0 +1,75 @@
+#include <linux/compiler.h>
+#include "builtin.h"
+#include "perf.h"
+#include "debug.h"
+#include "parse-options.h"
+
+typedef int (*data_cmd_fn_t)(int argc, const char **argv, const char *prefix);
+
+struct data_cmd {
+	const char	*name;
+	const char	*summary;
+	data_cmd_fn_t	fn;
+};
+
+static struct data_cmd data_cmds[];
+
+#define for_each_cmd(cmd) \
+	for (cmd = data_cmds; cmd && cmd->name; cmd++)
+
+static const struct option data_options[] = {
+	OPT_END()
+};
+
+static const char * const data_usage[] = {
+	"perf data [<common options>] <command> [<options>]",
+	NULL
+};
+
+static void print_usage(void)
+{
+	struct data_cmd *cmd;
+
+	printf("Usage:\n");
+	printf("\t%s\n\n", data_usage[0]);
+	printf("\tAvailable commands:\n");
+
+	for_each_cmd(cmd) {
+		printf("\t %s\t- %s\n", cmd->name, cmd->summary);
+	}
+
+	printf("\n");
+}
+
+static struct data_cmd data_cmds[] = {
+	{ NULL },
+};
+
+int cmd_data(int argc, const char **argv, const char *prefix)
+{
+	struct data_cmd *cmd;
+	const char *cmdstr;
+
+	/* No command specified. */
+	if (argc < 2)
+		goto usage;
+
+	argc = parse_options(argc, argv, data_options, data_usage,
+			     PARSE_OPT_STOP_AT_NON_OPTION);
+	if (argc < 1)
+		goto usage;
+
+	cmdstr = argv[0];
+
+	for_each_cmd(cmd) {
+		if (strcmp(cmd->name, cmdstr))
+			continue;
+
+		return cmd->fn(argc, argv, prefix);
+	}
+
+	pr_err("Unknown command: %s\n", cmdstr);
+usage:
+	print_usage();
+	return -1;
+}
diff --git a/tools/perf/builtin.h b/tools/perf/builtin.h
index b210d62907e4..3688ad29085f 100644
--- a/tools/perf/builtin.h
+++ b/tools/perf/builtin.h
@@ -37,6 +37,7 @@ extern int cmd_test(int argc, const char **argv, const char *prefix);
 extern int cmd_trace(int argc, const char **argv, const char *prefix);
 extern int cmd_inject(int argc, const char **argv, const char *prefix);
 extern int cmd_mem(int argc, const char **argv, const char *prefix);
+extern int cmd_data(int argc, const char **argv, const char *prefix);
 
 extern int find_scripts(char **scripts_array, char **scripts_path_array);
 #endif
diff --git a/tools/perf/command-list.txt b/tools/perf/command-list.txt
index 0906fc401c52..00fcaf8a5b8d 100644
--- a/tools/perf/command-list.txt
+++ b/tools/perf/command-list.txt
@@ -7,6 +7,7 @@ perf-archive			mainporcelain common
 perf-bench			mainporcelain common
 perf-buildid-cache		mainporcelain common
 perf-buildid-list		mainporcelain common
+perf-data			mainporcelain common
 perf-diff			mainporcelain common
 perf-evlist			mainporcelain common
 perf-inject			mainporcelain common
diff --git a/tools/perf/perf.c b/tools/perf/perf.c
index 3700a7faca6c..f3c66b81c6be 100644
--- a/tools/perf/perf.c
+++ b/tools/perf/perf.c
@@ -62,6 +62,7 @@ static struct cmd_struct commands[] = {
 #endif
 	{ "inject",	cmd_inject,	0 },
 	{ "mem",	cmd_mem,	0 },
+	{ "data",	cmd_data,	0 },
 };
 
 struct pager_config {
-- 
2.2.2


^ permalink raw reply related	[flat|nested] 221+ messages in thread

* [PATCH 42/42] perf data: Implement 'index' subcommand
  2015-01-29  8:06 [RFC/PATCHSET 00/42] perf tools: Speed-up perf report by using multi thread (v2) Namhyung Kim
                   ` (40 preceding siblings ...)
  2015-01-29  8:07 ` [PATCH 41/42] perf tools: Add new perf data command Namhyung Kim
@ 2015-01-29  8:07 ` Namhyung Kim
  2015-01-29 19:56 ` [RFC/PATCHSET 00/42] perf tools: Speed-up perf report by using multi thread (v2) Arnaldo Carvalho de Melo
  42 siblings, 0 replies; 221+ messages in thread
From: Namhyung Kim @ 2015-01-29  8:07 UTC (permalink / raw)
  To: Arnaldo Carvalho de Melo
  Cc: Ingo Molnar, Peter Zijlstra, Jiri Olsa, LKML, David Ahern,
	Adrian Hunter, Andi Kleen, Stephane Eranian, Frederic Weisbecker

The index command first splits a given data file into intermediate
data files and merges them into a final data file with an index table
so that it can processed using multi threads.  The HEADER_DATA_INDEX
feature bit is added to distinguish data file that has an index table.

Signed-off-by: Namhyung Kim <namhyung@kernel.org>
---
 tools/perf/Documentation/perf-data.txt |  29 +++
 tools/perf/builtin-data.c              | 353 +++++++++++++++++++++++++++++++++
 2 files changed, 382 insertions(+)

diff --git a/tools/perf/Documentation/perf-data.txt b/tools/perf/Documentation/perf-data.txt
index b8c83947715c..468ef7eb53e7 100644
--- a/tools/perf/Documentation/perf-data.txt
+++ b/tools/perf/Documentation/perf-data.txt
@@ -13,3 +13,32 @@ SYNOPSIS
 DESCRIPTION
 -----------
 Data file related processing.
+
+COMMANDS
+--------
+index::
+	Build an index table for data file so that it can be processed
+	with multiple threads concurrently.
+
+
+OPTIONS for 'split'
+---------------------
+-i::
+--input::
+	Specify input perf data file path.
+
+-o::
+--output::
+	Specify output perf data directory path.
+
+-v::
+--verbose::
+        Be more verbose (show counter open errors, etc).
+
+-f::
+--force::
+        Don't complain, do it.
+
+SEE ALSO
+--------
+linkperf:perf[1], linkperf:perf-report[1]
diff --git a/tools/perf/builtin-data.c b/tools/perf/builtin-data.c
index 1eee97d020fa..be44215355e6 100644
--- a/tools/perf/builtin-data.c
+++ b/tools/perf/builtin-data.c
@@ -2,10 +2,15 @@
 #include "builtin.h"
 #include "perf.h"
 #include "debug.h"
+#include "session.h"
+#include "evlist.h"
 #include "parse-options.h"
+#include <sys/mman.h>
 
 typedef int (*data_cmd_fn_t)(int argc, const char **argv, const char *prefix);
 
+static const char *output_name;
+
 struct data_cmd {
 	const char	*name;
 	const char	*summary;
@@ -41,10 +46,358 @@ static void print_usage(void)
 	printf("\n");
 }
 
+static int data_cmd_index(int argc, const char **argv, const char *prefix);
+
 static struct data_cmd data_cmds[] = {
+	{ "index", "merge data file and add index", data_cmd_index },
 	{ NULL },
 };
 
+#define FD_HASH_BITS  7
+#define FD_HASH_SIZE  (1 << FD_HASH_BITS)
+#define FD_HASH_MASK  (FD_HASH_SIZE - 1)
+
+struct data_index {
+	struct perf_tool	tool;
+	struct perf_session	*session;
+	enum {
+		PER_CPU,
+		PER_THREAD,
+	} split_mode;
+	char			*tmpdir;
+	int 			header_fd;
+	u64			header_written;
+	struct hlist_head	fd_hash[FD_HASH_SIZE];
+	int			fd_hash_nr;
+	int			output_fd;
+};
+
+struct fdhash_node {
+	int			id;
+	int			fd;
+	struct hlist_node	list;
+};
+
+static struct hlist_head *get_hash(struct data_index *index, int id)
+{
+	return &index->fd_hash[id % FD_HASH_MASK];
+}
+
+static int perf_event__rewrite_header(struct perf_tool *tool,
+				      union perf_event *event)
+{
+	struct data_index *index = container_of(tool, struct data_index, tool);
+	ssize_t size;
+
+	size = writen(index->header_fd, event, event->header.size);
+	if (size < 0)
+		return -errno;
+
+	index->header_written += size;
+	return 0;
+}
+
+static int split_other_events(struct perf_tool *tool,
+				union perf_event *event,
+				struct perf_sample *sample __maybe_unused,
+				struct machine *machine __maybe_unused)
+{
+	return perf_event__rewrite_header(tool, event);
+}
+
+static int split_sample_event(struct perf_tool *tool,
+				union perf_event *event,
+				struct perf_sample *sample,
+				struct perf_evsel *evsel __maybe_unused,
+				struct machine *machine __maybe_unused)
+{
+	struct data_index *index = container_of(tool, struct data_index, tool);
+	int id = index->split_mode == PER_CPU ? sample->cpu : sample->tid;
+	int fd = -1;
+	char buf[PATH_MAX];
+	struct hlist_head *head;
+	struct fdhash_node *node;
+
+	head = get_hash(index, id);
+	hlist_for_each_entry(node, head, list) {
+		if (node->id == id) {
+			fd = node->fd;
+			break;
+		}
+	}
+
+	if (fd == -1) {
+		scnprintf(buf, sizeof(buf), "%s/perf.data.%d",
+			  index->tmpdir, index->fd_hash_nr++);
+
+		fd = open(buf, O_RDWR|O_CREAT|O_TRUNC, 0600);
+		if (fd < 0) {
+			pr_err("cannot open data file: %s: %m\n", buf);
+			return -1;
+		}
+
+		node = malloc(sizeof(*node));
+		if (node == NULL) {
+			pr_err("memory allocation failed\n");
+			return -1;
+		}
+
+		node->id = id;
+		node->fd = fd;
+
+		hlist_add_head(&node->list, head);
+	}
+
+	return writen(fd, event, event->header.size) > 0 ? 0 : -errno;
+}
+
+static int split_data_file(struct data_index *index)
+{
+	struct perf_session *session = index->session;
+	char buf[PATH_MAX];
+	u64 sample_type;
+	int header_fd;
+
+	if (asprintf(&index->tmpdir, "%s.dir", output_name) < 0) {
+		pr_err("memory allocation failed\n");
+		return -1;
+	}
+
+	if (mkdir(index->tmpdir, 0700) < 0) {
+		pr_err("cannot create intermediate directory\n");
+		return -1;
+	}
+
+	/*
+	 * This is necessary to write (copy) build-id table.  After
+	 * processing header, dsos list will only contain dso which
+	 * was on the original build-id table.
+	 */
+	dsos__hit_all(session);
+
+	scnprintf(buf, sizeof(buf), "%s/perf.header", index->tmpdir);
+	header_fd = open(buf, O_RDWR|O_CREAT|O_TRUNC, 0600);
+	if (header_fd < 0) {
+		pr_err("cannot open header file: %s: %m\n", buf);
+		return -1;
+	}
+
+	lseek(header_fd, session->header.data_offset, SEEK_SET);
+
+	sample_type = perf_evlist__combined_sample_type(session->evlist);
+	if (sample_type & PERF_SAMPLE_CPU)
+		index->split_mode = PER_CPU;
+	else
+		index->split_mode = PER_THREAD;
+
+	pr_debug("splitting data file for %s\n",
+		 index->split_mode == PER_CPU ? "CPUs" : "threads");
+
+	index->header_fd = header_fd;
+	if (perf_session__process_events(session, &index->tool) < 0) {
+		pr_err("failed to process events\n");
+		return -1;
+	}
+
+	session->header.data_size = index->header_written;
+	/* 
+	 * This is needed for index to determine current (header) file
+	 * size (including feature data).
+	 */
+	perf_session__write_header(session, session->evlist, header_fd, true);
+
+	return 0;
+}
+
+static int build_index_table(struct data_index *index)
+{
+	int i, n;
+	u64 offset;
+	u64 nr_index = index->fd_hash_nr;
+	struct perf_file_section *idx;
+	struct perf_session *session = index->session;
+
+	idx = calloc(nr_index, sizeof(*idx));
+	if (idx == NULL)
+		return -1;
+
+	/* index data will be placed after header file */
+	offset = lseek(index->header_fd, 0, SEEK_END);
+	if (offset == (u64)(loff_t) -1)
+		goto out;
+
+	/* increase the offset for added index data */
+	offset += sizeof(nr_index) + nr_index * sizeof(*index);
+	offset = PERF_ALIGN(offset, page_size);
+
+	for (i = n = 0; i < FD_HASH_SIZE; i++) {
+		struct fdhash_node *node;
+
+		hlist_for_each_entry(node, &index->fd_hash[i], list) {
+			struct stat stbuf;
+
+			if (fstat(node->fd, &stbuf) < 0)
+				goto out;
+
+			idx[n].offset = offset;
+			idx[n].size   = stbuf.st_size;
+			n++;
+
+			offset += PERF_ALIGN(stbuf.st_size, page_size);
+		}
+	}
+
+	BUG_ON(n != (int)nr_index);
+
+	session->header.index = idx;
+	session->header.nr_index = nr_index;
+	perf_header__set_feat(&session->header, HEADER_DATA_INDEX);
+
+	perf_session__write_header(session, session->evlist,
+				   index->output_fd, true);
+	return 0;
+
+out:
+	free(idx);
+	return -1;
+}
+
+static int cleanup_temp_files(struct data_index *index)
+{
+	int i;
+
+	for (i = 0; i < FD_HASH_SIZE; i++) {
+		struct fdhash_node *pos;
+		struct hlist_node *tmp;
+
+		hlist_for_each_entry_safe(pos, tmp, &index->fd_hash[i], list) {
+			hlist_del(&pos->list);
+			close(pos->fd);
+			free(pos);
+		}
+	}
+	close(index->header_fd);
+
+	rm_rf(index->tmpdir);
+	zfree(&index->tmpdir);
+	return 0;
+}
+
+static int __data_cmd_index(struct data_index *index)
+{
+	struct perf_session *session = index->session;
+	char *output = NULL;
+	int ret = -1;
+	int i, n;
+
+	if (!output_name) {
+		if (asprintf(&output, "%s.out", session->file->path) < 0) {
+			pr_err("memory allocation failed\n");
+			return -1;
+		}
+
+		output_name = output;
+	}
+
+	index->output_fd = open(output_name, O_RDWR|O_CREAT|O_TRUNC, 0600);
+	if (index->output_fd < 0) {
+		pr_err("cannot create output file: %s\n", output_name);
+		goto out;
+	}
+
+	/*
+	 * This is necessary to write (copy) build-id table.  After
+	 * processing header, dsos list will contain dso which was on
+	 * the original build-id table.
+	 */
+	dsos__hit_all(session);
+
+	if (split_data_file(index) < 0)
+		goto out_clean;
+
+	if (build_index_table(index) < 0)
+		goto out_clean;
+
+	/* copy meta-events */
+	if (copyfile_offset(index->header_fd, session->header.data_offset,
+			   index->output_fd, session->header.data_offset,
+			   session->header.data_size) < 0)
+		goto out_clean;
+
+	/* copy sample events */
+	for (i = n = 0; i < FD_HASH_SIZE; i++) {
+		struct fdhash_node *node;
+
+		hlist_for_each_entry(node, &index->fd_hash[i], list) {
+			if (copyfile_offset(node->fd, 0, index->output_fd,
+					    session->header.index[n].offset,
+					    session->header.index[n].size) < 0)
+				goto out_clean;
+			n++;
+		}
+	}
+	ret = 0;
+
+out_clean:
+	cleanup_temp_files(index);
+	close(index->output_fd);
+out:
+	free(output);
+	return ret;
+}
+
+int data_cmd_index(int argc, const char **argv, const char *prefix __maybe_unused)
+{
+	bool force = false;
+	struct perf_session *session;
+	struct perf_data_file file = {
+		.mode  = PERF_DATA_MODE_READ,
+	};
+	struct data_index index = {
+		.tool = {
+			.sample		= split_sample_event,
+			.fork		= split_other_events,
+			.comm		= split_other_events,
+			.exit		= split_other_events,
+			.mmap		= split_other_events,
+			.mmap2		= split_other_events,
+			.lost		= split_other_events,
+			.throttle	= split_other_events,
+			.unthrottle	= split_other_events,
+			.ordered_events = false,
+		},
+	};
+	const char * const index_usage[] = {
+		"perf data index [<options>]",
+		NULL
+	};
+	const struct option index_options[] = {
+	OPT_STRING('i', "input", &input_name, "file", "input file name"),
+	OPT_STRING('o', "output", &output_name, "file", "output directory name"),
+	OPT_BOOLEAN('f', "force", &force, "don't complain, do it"),
+	OPT_INCR('v', "verbose", &verbose, "be more verbose"),
+	OPT_END()
+	};
+
+	argc = parse_options(argc, argv, index_options, index_usage, 0);
+	if (argc)
+		usage_with_options(index_usage, index_options);
+
+	file.path = input_name;
+	file.force = force;
+	session = perf_session__new(&file, false, &index.tool);
+	if (session == NULL)
+		return -1;
+
+	index.session = session;
+	symbol__init(&session->header.env);
+
+	__data_cmd_index(&index);
+
+	perf_session__delete(session);
+	return 0;
+}
+
 int cmd_data(int argc, const char **argv, const char *prefix)
 {
 	struct data_cmd *cmd;
-- 
2.2.2


^ permalink raw reply related	[flat|nested] 221+ messages in thread

* Re: [PATCH 29/42] perf tools: Protect dso cache fd with a mutex
  2015-01-29  8:07 ` [PATCH 29/42] perf tools: Protect dso cache fd with a mutex Namhyung Kim
@ 2015-01-29 12:31   ` Arnaldo Carvalho de Melo
  2015-01-29 13:19     ` Namhyung Kim
  0 siblings, 1 reply; 221+ messages in thread
From: Arnaldo Carvalho de Melo @ 2015-01-29 12:31 UTC (permalink / raw)
  To: Namhyung Kim
  Cc: Ingo Molnar, Peter Zijlstra, Jiri Olsa, LKML, David Ahern,
	Adrian Hunter, Andi Kleen, Stephane Eranian, Frederic Weisbecker

Em Thu, Jan 29, 2015 at 05:07:10PM +0900, Namhyung Kim escreveu:
> When dso cache is accessed in multi-thread environment, it's possible
> to close other dso->data.fd during operation due to open file limit.
> Protect the file descriptors using a separate mutex.
> 
> Signed-off-by: Namhyung Kim <namhyung@kernel.org>
> ---
>  tools/perf/tests/dso-data.c |   5 ++
>  tools/perf/util/dso.c       | 136 +++++++++++++++++++++++++++++---------------
>  2 files changed, 94 insertions(+), 47 deletions(-)
> 
> diff --git a/tools/perf/tests/dso-data.c b/tools/perf/tests/dso-data.c
> index caaf37f079b1..0276e7d2d41b 100644
> --- a/tools/perf/tests/dso-data.c
> +++ b/tools/perf/tests/dso-data.c
> @@ -111,6 +111,9 @@ int test__dso_data(void)
>  	memset(&machine, 0, sizeof(machine));
>  
>  	dso = dso__new((const char *)file);
> +	TEST_ASSERT_VAL("failed to get dso", dso);
> +
> +	dso->binary_type = DSO_BINARY_TYPE__SYSTEM_PATH_DSO;
>  
>  	/* Basic 10 bytes tests. */
>  	for (i = 0; i < ARRAY_SIZE(offsets); i++) {
> @@ -199,6 +202,8 @@ static int dsos__create(int cnt, int size)
>  
>  		dsos[i] = dso__new(file);
>  		TEST_ASSERT_VAL("failed to get dso", dsos[i]);
> +
> +		dsos[i]->binary_type = DSO_BINARY_TYPE__SYSTEM_PATH_DSO;

Those two are unrelated, please put them in a separate patch, one that I
can even cherrypick ahead of the other patches.

>  	}
>  
>  	return 0;
> diff --git a/tools/perf/util/dso.c b/tools/perf/util/dso.c
> index 11ece224ef50..ae92046ae2c8 100644
> --- a/tools/perf/util/dso.c
> +++ b/tools/perf/util/dso.c
> @@ -213,6 +213,7 @@ bool dso__needs_decompress(struct dso *dso)
>   */
>  static LIST_HEAD(dso__data_open);
>  static long dso__data_open_cnt;
> +static pthread_mutex_t dso__data_open_lock = PTHREAD_MUTEX_INITIALIZER;
>  
>  static void dso__list_add(struct dso *dso)
>  {
> @@ -240,7 +241,7 @@ static int do_open(char *name)
>  		if (fd >= 0)
>  			return fd;
>  
> -		pr_debug("dso open failed, mmap: %s\n",
> +		pr_debug("dso open failed: %s\n",
>  			 strerror_r(errno, sbuf, sizeof(sbuf)));
>  		if (!dso__data_open_cnt || errno != EMFILE)

Ditto, another unrelated patch, please separate.

>  			break;
> @@ -382,7 +383,9 @@ static void check_data_close(void)
>   */
>  void dso__data_close(struct dso *dso)
>  {
> +	pthread_mutex_lock(&dso__data_open_lock);
>  	close_dso(dso);
> +	pthread_mutex_unlock(&dso__data_open_lock);
>  }
>  
>  /**
> @@ -405,6 +408,8 @@ int dso__data_fd(struct dso *dso, struct machine *machine)
>  	if (dso->data.status == DSO_DATA_STATUS_ERROR)
>  		return -1;
>  
> +	pthread_mutex_lock(&dso__data_open_lock);
> +
>  	if (dso->data.fd >= 0)
>  		goto out;
>  
> @@ -427,6 +432,7 @@ int dso__data_fd(struct dso *dso, struct machine *machine)
>  	else
>  		dso->data.status = DSO_DATA_STATUS_ERROR;
>  
> +	pthread_mutex_unlock(&dso__data_open_lock);
>  	return dso->data.fd;
>  }
>  
> @@ -531,52 +537,66 @@ dso_cache__memcpy(struct dso_cache *cache, u64 offset,
>  }
>  
>  static ssize_t
> -dso_cache__read(struct dso *dso, u64 offset, u8 *data, ssize_t size)
> +dso_cache__read(struct dso *dso, struct machine *machine,
> +		u64 offset, u8 *data, ssize_t size)
>  {
>  	struct dso_cache *cache;
>  	struct dso_cache *old;
> -	ssize_t ret;
> -
> -	do {
> -		u64 cache_offset;

While I understand that there was no need for this do { } while (0)
construct in the first place, removing it in this patch is not
interesting, as it is both unrelated to this patch and makes the it
harder to review by just looking at the patch :-\ Please refrain from
doing this in this patch.

A later patch that does _just_ that could be done, if you feel like
doing it.

> +	ssize_t ret = -EINVAL;
> +	u64 cache_offset;
>  
> -		ret = -ENOMEM;
> +	cache = zalloc(sizeof(*cache) + DSO__DATA_CACHE_SIZE);
> +	if (!cache)
> +		return -ENOMEM;
>  
> -		cache = zalloc(sizeof(*cache) + DSO__DATA_CACHE_SIZE);
> -		if (!cache)
> -			break;
> +	cache_offset = offset & DSO__DATA_CACHE_MASK;
>  
> -		cache_offset = offset & DSO__DATA_CACHE_MASK;
> -		ret = -EINVAL;
> +	pthread_mutex_lock(&dso__data_open_lock);
>  
> -		if (-1 == lseek(dso->data.fd, cache_offset, SEEK_SET))
> -			break;
> +	/*
> +	 * dso->data.fd might be closed if other thread opened another
> +	 * file (dso) due to open file limit (RLIMIT_NOFILE).
> +	 */
> +	if (dso->data.fd < 0) {
> +		dso->data.fd = open_dso(dso, machine);

Also please consider adding a backpointer to machine in the dso object,
since you need to reopen it, so that we don't have to go on passing
machine around to dso_cache__read(), etc.

This probably needs to be done in the patch that makes dso->data.fd to
be closed due to limit.

> +		if (dso->data.fd < 0) {
> +			ret = -errno;
> +			dso->data.status = DSO_DATA_STATUS_ERROR;
> +			goto err_unlock;
> +		}
> +	}
>  
> -		ret = read(dso->data.fd, cache->data, DSO__DATA_CACHE_SIZE);
> -		if (ret <= 0)
> -			break;
> +	if (-1 == lseek(dso->data.fd, cache_offset, SEEK_SET))
> +		goto err_unlock;
>  
> -		cache->offset = cache_offset;
> -		cache->size   = ret;
> -		old = dso_cache__insert(dso, cache);
> -		if (old) {
> -			/* we lose the race */
> -			free(cache);
> -			cache = old;
> -		}
> +	ret = read(dso->data.fd, cache->data, DSO__DATA_CACHE_SIZE);
> +	if (ret <= 0)
> +		goto err_unlock;
>  
> -		ret = dso_cache__memcpy(cache, offset, data, size);
> +	pthread_mutex_unlock(&dso__data_open_lock);
>  
> -	} while (0);
> +	cache->offset = cache_offset;
> +	cache->size   = ret;
> +	old = dso_cache__insert(dso, cache);
> +	if (old) {
> +		/* we lose the race */
> +		free(cache);
> +		cache = old;
> +	}
>  
> +	ret = dso_cache__memcpy(cache, offset, data, size);
>  	if (ret <= 0)
>  		free(cache);
>  
>  	return ret;
> +
> +err_unlock:
> +	pthread_mutex_unlock(&dso__data_open_lock);
> +	return ret;
>  }
>  
> -static ssize_t dso_cache_read(struct dso *dso, u64 offset,
> -			      u8 *data, ssize_t size)
> +static ssize_t dso_cache_read(struct dso *dso, struct machine *machine,
> +			      u64 offset, u8 *data, ssize_t size)
>  {
>  	struct dso_cache *cache;
>  
> @@ -584,7 +604,7 @@ static ssize_t dso_cache_read(struct dso *dso, u64 offset,
>  	if (cache)
>  		return dso_cache__memcpy(cache, offset, data, size);
>  	else
> -		return dso_cache__read(dso, offset, data, size);
> +		return dso_cache__read(dso, machine, offset, data, size);
>  }
>  
>  /*
> @@ -592,7 +612,8 @@ static ssize_t dso_cache_read(struct dso *dso, u64 offset,
>   * in the rb_tree. Any read to already cached data is served
>   * by cached data.
>   */
> -static ssize_t cached_read(struct dso *dso, u64 offset, u8 *data, ssize_t size)
> +static ssize_t cached_read(struct dso *dso, struct machine *machine,
> +			   u64 offset, u8 *data, ssize_t size)
>  {
>  	ssize_t r = 0;
>  	u8 *p = data;
> @@ -600,7 +621,7 @@ static ssize_t cached_read(struct dso *dso, u64 offset, u8 *data, ssize_t size)
>  	do {
>  		ssize_t ret;
>  
> -		ret = dso_cache_read(dso, offset, p, size);
> +		ret = dso_cache_read(dso, machine, offset, p, size);
>  		if (ret < 0)
>  			return ret;
>  
> @@ -620,21 +641,42 @@ static ssize_t cached_read(struct dso *dso, u64 offset, u8 *data, ssize_t size)
>  	return r;
>  }
>  
> -static int data_file_size(struct dso *dso)
> +static int data_file_size(struct dso *dso, struct machine *machine)
>  {
> +	int ret = 0;
>  	struct stat st;
>  	char sbuf[STRERR_BUFSIZE];
>  
> -	if (!dso->data.file_size) {
> -		if (fstat(dso->data.fd, &st)) {
> -			pr_err("dso mmap failed, fstat: %s\n",
> -				strerror_r(errno, sbuf, sizeof(sbuf)));
> -			return -1;
> +	if (dso->data.file_size)
> +		return 0;
> +
> +	pthread_mutex_lock(&dso__data_open_lock);
> +
> +	/*
> +	 * dso->data.fd might be closed if other thread opened another
> +	 * file (dso) due to open file limit (RLIMIT_NOFILE).
> +	 */
> +	if (dso->data.fd < 0) {
> +		dso->data.fd = open_dso(dso, machine);
> +		if (dso->data.fd < 0) {
> +			ret = -errno;
> +			dso->data.status = DSO_DATA_STATUS_ERROR;
> +			goto out;
>  		}
> -		dso->data.file_size = st.st_size;
>  	}
>  
> -	return 0;
> +	if (fstat(dso->data.fd, &st) < 0) {
> +		ret = -errno;
> +		pr_err("dso cache fstat failed: %s\n",
> +		       strerror_r(errno, sbuf, sizeof(sbuf)));
> +		dso->data.status = DSO_DATA_STATUS_ERROR;
> +		goto out;
> +	}
> +	dso->data.file_size = st.st_size;
> +
> +out:
> +	pthread_mutex_unlock(&dso__data_open_lock);
> +	return ret;
>  }
>  
>  /**
> @@ -652,17 +694,17 @@ off_t dso__data_size(struct dso *dso, struct machine *machine)
>  	if (fd < 0)
>  		return fd;
>  
> -	if (data_file_size(dso))
> +	if (data_file_size(dso, machine))
>  		return -1;
>  
>  	/* For now just estimate dso data size is close to file size */
>  	return dso->data.file_size;
>  }
>  
> -static ssize_t data_read_offset(struct dso *dso, u64 offset,
> -				u8 *data, ssize_t size)
> +static ssize_t data_read_offset(struct dso *dso, struct machine *machine,
> +				u64 offset, u8 *data, ssize_t size)
>  {
> -	if (data_file_size(dso))
> +	if (data_file_size(dso, machine))
>  		return -1;
>  
>  	/* Check the offset sanity. */
> @@ -672,7 +714,7 @@ static ssize_t data_read_offset(struct dso *dso, u64 offset,
>  	if (offset + size < offset)
>  		return -1;
>  
> -	return cached_read(dso, offset, data, size);
> +	return cached_read(dso, machine, offset, data, size);
>  }
>  
>  /**
> @@ -689,10 +731,10 @@ static ssize_t data_read_offset(struct dso *dso, u64 offset,
>  ssize_t dso__data_read_offset(struct dso *dso, struct machine *machine,
>  			      u64 offset, u8 *data, ssize_t size)
>  {
> -	if (dso__data_fd(dso, machine) < 0)
> +	if (dso->data.status == DSO_DATA_STATUS_ERROR)
>  		return -1;
>  
> -	return data_read_offset(dso, offset, data, size);
> +	return data_read_offset(dso, machine, offset, data, size);
>  }
>  
>  /**
> -- 
> 2.2.2

^ permalink raw reply	[flat|nested] 221+ messages in thread

* Re: [PATCH 27/42] perf tools: Protect dso symbol loading using a mutex
  2015-01-29  8:07 ` [PATCH 27/42] perf tools: Protect dso symbol loading using a mutex Namhyung Kim
@ 2015-01-29 12:34   ` Arnaldo Carvalho de Melo
  2015-01-29 12:48     ` Namhyung Kim
  0 siblings, 1 reply; 221+ messages in thread
From: Arnaldo Carvalho de Melo @ 2015-01-29 12:34 UTC (permalink / raw)
  To: Namhyung Kim
  Cc: Ingo Molnar, Peter Zijlstra, Jiri Olsa, LKML, David Ahern,
	Adrian Hunter, Andi Kleen, Stephane Eranian, Frederic Weisbecker

Em Thu, Jan 29, 2015 at 05:07:08PM +0900, Namhyung Kim escreveu:
> When multi-thread support for perf report is enabled, it's possible to
> access a dso concurrently.  Add a new pthread_mutex to protect it from
> concurrent dso__load().
> 
> Signed-off-by: Namhyung Kim <namhyung@kernel.org>
> ---
>  tools/perf/util/dso.c    |  2 ++
>  tools/perf/util/dso.h    |  1 +
>  tools/perf/util/symbol.c | 34 ++++++++++++++++++++++++----------
>  3 files changed, 27 insertions(+), 10 deletions(-)
> 
> diff --git a/tools/perf/util/dso.c b/tools/perf/util/dso.c
> index 45be944d450a..3da75816b8f8 100644
> --- a/tools/perf/util/dso.c
> +++ b/tools/perf/util/dso.c
> @@ -888,6 +888,7 @@ struct dso *dso__new(const char *name)
>  		RB_CLEAR_NODE(&dso->rb_node);
>  		INIT_LIST_HEAD(&dso->node);
>  		INIT_LIST_HEAD(&dso->data.open_entry);
> +		pthread_mutex_init(&dso->lock, NULL);
>  	}
>  
>  	return dso;
> @@ -917,6 +918,7 @@ void dso__delete(struct dso *dso)
>  	dso_cache__free(&dso->data.cache);
>  	dso__free_a2l(dso);
>  	zfree(&dso->symsrc_filename);
> +	pthread_mutex_destroy(&dso->lock);
>  	free(dso);
>  }
>  
> diff --git a/tools/perf/util/dso.h b/tools/perf/util/dso.h
> index 3782c82c6e44..ac753594a469 100644
> --- a/tools/perf/util/dso.h
> +++ b/tools/perf/util/dso.h
> @@ -102,6 +102,7 @@ struct dsos {
>  };
>  
>  struct dso {
> +	pthread_mutex_t	 lock;
>  	struct list_head node;
>  	struct rb_node	 rb_node;	/* rbtree node sorted by long name */
>  	struct rb_root	 symbols[MAP__NR_TYPES];
> diff --git a/tools/perf/util/symbol.c b/tools/perf/util/symbol.c
> index a69066865a55..714e20c99354 100644
> --- a/tools/perf/util/symbol.c
> +++ b/tools/perf/util/symbol.c
> @@ -1357,12 +1357,22 @@ int dso__load(struct dso *dso, struct map *map, symbol_filter_t filter)
>  	struct symsrc *syms_ss = NULL, *runtime_ss = NULL;
>  	bool kmod;
>  
> -	dso__set_loaded(dso, map->type);
> +	pthread_mutex_lock(&dso->lock);
> +
> +	/* check again under the dso->lock */

Again? Where was it first checked? Perhaps we should lock there, so that
we don't have to do two checks, one unlocked, the other locked?

> +	if (dso__loaded(dso, map->type)) {
> +		ret = 1;
> +		goto out;
> +	}
> +

- Arnaldo

^ permalink raw reply	[flat|nested] 221+ messages in thread

* Re: [PATCH 40/42] perf callchain: Save eh/debug frame offset for dwarf unwind
  2015-01-29  8:07 ` [PATCH 40/42] perf callchain: Save eh/debug frame offset for dwarf unwind Namhyung Kim
@ 2015-01-29 12:38   ` Arnaldo Carvalho de Melo
  2015-01-29 13:23     ` Namhyung Kim
  2015-01-29 19:22   ` Arnaldo Carvalho de Melo
  2015-01-30 18:32   ` [tip:perf/core] perf callchain: Cache eh/ debug " tip-bot for Namhyung Kim
  2 siblings, 1 reply; 221+ messages in thread
From: Arnaldo Carvalho de Melo @ 2015-01-29 12:38 UTC (permalink / raw)
  To: Namhyung Kim
  Cc: Ingo Molnar, Peter Zijlstra, Jiri Olsa, LKML, David Ahern,
	Adrian Hunter, Andi Kleen, Stephane Eranian, Frederic Weisbecker

Em Thu, Jan 29, 2015 at 05:07:21PM +0900, Namhyung Kim escreveu:
> When libunwind tries to resolve callchains it needs to know the offset
> of .eh_frame_hdr or .debug_frame to access the dso.  Since it calls
> dso__data_fd(), it'll try to grab dso->lock everytime for same
> information.  So save it to dso_data struct and reuse it.
> 
> Note that there's a window between dso__data_fd() and actual use of
> the fd.  The fd could be closed by other threads to deal with the open
> file limit in dso cache code.  But I think it's ok since in that case
> elf_section_offset() will return 0 so it'll be tried in next acess.

I know that you did this in the context of your multi threading
patchkit, but this seems useful even without that patckhit, i.e. this
can be cherry picked on the grounds that it speeds up things by caching
something that doesn't change, right?

I.e. I'll probably just rewrite the comment and apply it before
considering the other patches, so that other people can comment on the
other patches, etc.

- Arnaldo
 
> Signed-off-by: Namhyung Kim <namhyung@kernel.org>
> ---
>  tools/perf/util/dso.h              |  1 +
>  tools/perf/util/unwind-libunwind.c | 31 ++++++++++++++++++++-----------
>  2 files changed, 21 insertions(+), 11 deletions(-)
> 
> diff --git a/tools/perf/util/dso.h b/tools/perf/util/dso.h
> index c18fcc0e8081..323ee08d56fc 100644
> --- a/tools/perf/util/dso.h
> +++ b/tools/perf/util/dso.h
> @@ -141,6 +141,7 @@ struct dso {
>  		u32		 status_seen;
>  		size_t		 file_size;
>  		struct list_head open_entry;
> +		u64		 frame_offset;
>  	} data;
>  
>  	union { /* Tool specific area */
> diff --git a/tools/perf/util/unwind-libunwind.c b/tools/perf/util/unwind-libunwind.c
> index 7ed6eaf232b6..3219b20837b5 100644
> --- a/tools/perf/util/unwind-libunwind.c
> +++ b/tools/perf/util/unwind-libunwind.c
> @@ -266,14 +266,17 @@ static int read_unwind_spec_eh_frame(struct dso *dso, struct machine *machine,
>  				     u64 *fde_count)
>  {
>  	int ret = -EINVAL, fd;
> -	u64 offset;
> +	u64 offset = dso->data.frame_offset;
>  
> -	fd = dso__data_fd(dso, machine);
> -	if (fd < 0)
> -		return -EINVAL;
> +	if (offset == 0) {
> +		fd = dso__data_fd(dso, machine);
> +		if (fd < 0)
> +			return -EINVAL;
>  
> -	/* Check the .eh_frame section for unwinding info */
> -	offset = elf_section_offset(fd, ".eh_frame_hdr");
> +		/* Check the .eh_frame section for unwinding info */
> +		offset = elf_section_offset(fd, ".eh_frame_hdr");
> +		dso->data.frame_offset = offset;
> +	}
>  
>  	if (offset)
>  		ret = unwind_spec_ehframe(dso, machine, offset,
> @@ -287,14 +290,20 @@ static int read_unwind_spec_eh_frame(struct dso *dso, struct machine *machine,
>  static int read_unwind_spec_debug_frame(struct dso *dso,
>  					struct machine *machine, u64 *offset)
>  {
> -	int fd = dso__data_fd(dso, machine);
> +	int fd;
> +	u64 ofs = dso->data.frame_offset;
>  
> -	if (fd < 0)
> -		return -EINVAL;
> +	if (ofs == 0) {
> +		fd = dso__data_fd(dso, machine);
> +		if (fd < 0)
> +			return -EINVAL;
>  
> -	/* Check the .debug_frame section for unwinding info */
> -	*offset = elf_section_offset(fd, ".debug_frame");
> +		/* Check the .debug_frame section for unwinding info */
> +		ofs = elf_section_offset(fd, ".debug_frame");
> +		dso->data.frame_offset = ofs;
> +	}
>  
> +	*offset = ofs;
>  	if (*offset)
>  		return 0;
>  
> -- 
> 2.2.2

^ permalink raw reply	[flat|nested] 221+ messages in thread

* Re: [PATCH 27/42] perf tools: Protect dso symbol loading using a mutex
  2015-01-29 12:34   ` Arnaldo Carvalho de Melo
@ 2015-01-29 12:48     ` Namhyung Kim
  0 siblings, 0 replies; 221+ messages in thread
From: Namhyung Kim @ 2015-01-29 12:48 UTC (permalink / raw)
  To: Arnaldo Carvalho de Melo
  Cc: Ingo Molnar, Peter Zijlstra, Jiri Olsa, LKML, David Ahern,
	Adrian Hunter, Andi Kleen, Stephane Eranian, Frederic Weisbecker

Hi Arnaldo,

On Thu, Jan 29, 2015 at 9:34 PM, Arnaldo Carvalho de Melo
<acme@kernel.org> wrote:
> Em Thu, Jan 29, 2015 at 05:07:08PM +0900, Namhyung Kim escreveu:
>> When multi-thread support for perf report is enabled, it's possible to
>> access a dso concurrently.  Add a new pthread_mutex to protect it from
>> concurrent dso__load().
>>
>> Signed-off-by: Namhyung Kim <namhyung@kernel.org>
>> ---
>>  tools/perf/util/dso.c    |  2 ++
>>  tools/perf/util/dso.h    |  1 +
>>  tools/perf/util/symbol.c | 34 ++++++++++++++++++++++++----------
>>  3 files changed, 27 insertions(+), 10 deletions(-)
>>
>> diff --git a/tools/perf/util/dso.c b/tools/perf/util/dso.c
>> index 45be944d450a..3da75816b8f8 100644
>> --- a/tools/perf/util/dso.c
>> +++ b/tools/perf/util/dso.c
>> @@ -888,6 +888,7 @@ struct dso *dso__new(const char *name)
>>               RB_CLEAR_NODE(&dso->rb_node);
>>               INIT_LIST_HEAD(&dso->node);
>>               INIT_LIST_HEAD(&dso->data.open_entry);
>> +             pthread_mutex_init(&dso->lock, NULL);
>>       }
>>
>>       return dso;
>> @@ -917,6 +918,7 @@ void dso__delete(struct dso *dso)
>>       dso_cache__free(&dso->data.cache);
>>       dso__free_a2l(dso);
>>       zfree(&dso->symsrc_filename);
>> +     pthread_mutex_destroy(&dso->lock);
>>       free(dso);
>>  }
>>
>> diff --git a/tools/perf/util/dso.h b/tools/perf/util/dso.h
>> index 3782c82c6e44..ac753594a469 100644
>> --- a/tools/perf/util/dso.h
>> +++ b/tools/perf/util/dso.h
>> @@ -102,6 +102,7 @@ struct dsos {
>>  };
>>
>>  struct dso {
>> +     pthread_mutex_t  lock;
>>       struct list_head node;
>>       struct rb_node   rb_node;       /* rbtree node sorted by long name */
>>       struct rb_root   symbols[MAP__NR_TYPES];
>> diff --git a/tools/perf/util/symbol.c b/tools/perf/util/symbol.c
>> index a69066865a55..714e20c99354 100644
>> --- a/tools/perf/util/symbol.c
>> +++ b/tools/perf/util/symbol.c
>> @@ -1357,12 +1357,22 @@ int dso__load(struct dso *dso, struct map *map, symbol_filter_t filter)
>>       struct symsrc *syms_ss = NULL, *runtime_ss = NULL;
>>       bool kmod;
>>
>> -     dso__set_loaded(dso, map->type);
>> +     pthread_mutex_lock(&dso->lock);
>> +
>> +     /* check again under the dso->lock */
>
> Again? Where was it first checked?

Please see map__load().


> Perhaps we should lock there, so that
> we don't have to do two checks, one unlocked, the other locked?

Hmm.. maybe.  I just keep it to avoid locking overhead since it'll be
called whenever it searches symbols during preprocessing.  I didn't
measure the overhead but it could be huge IMHO.

Thanks,
Namhyung


>
>> +     if (dso__loaded(dso, map->type)) {
>> +             ret = 1;
>> +             goto out;
>> +     }
>> +
>
> - Arnaldo

^ permalink raw reply	[flat|nested] 221+ messages in thread

* Re: [PATCH 29/42] perf tools: Protect dso cache fd with a mutex
  2015-01-29 12:31   ` Arnaldo Carvalho de Melo
@ 2015-01-29 13:19     ` Namhyung Kim
  2015-01-29 16:23       ` Arnaldo Carvalho de Melo
  0 siblings, 1 reply; 221+ messages in thread
From: Namhyung Kim @ 2015-01-29 13:19 UTC (permalink / raw)
  To: Arnaldo Carvalho de Melo
  Cc: Ingo Molnar, Peter Zijlstra, Jiri Olsa, LKML, David Ahern,
	Adrian Hunter, Andi Kleen, Stephane Eranian, Frederic Weisbecker

On Thu, Jan 29, 2015 at 09:31:07AM -0300, Arnaldo Carvalho de Melo wrote:
> Em Thu, Jan 29, 2015 at 05:07:10PM +0900, Namhyung Kim escreveu:
> > When dso cache is accessed in multi-thread environment, it's possible
> > to close other dso->data.fd during operation due to open file limit.
> > Protect the file descriptors using a separate mutex.
> > 
> > Signed-off-by: Namhyung Kim <namhyung@kernel.org>
> > ---
> >  tools/perf/tests/dso-data.c |   5 ++
> >  tools/perf/util/dso.c       | 136 +++++++++++++++++++++++++++++---------------
> >  2 files changed, 94 insertions(+), 47 deletions(-)
> > 
> > diff --git a/tools/perf/tests/dso-data.c b/tools/perf/tests/dso-data.c
> > index caaf37f079b1..0276e7d2d41b 100644
> > --- a/tools/perf/tests/dso-data.c
> > +++ b/tools/perf/tests/dso-data.c
> > @@ -111,6 +111,9 @@ int test__dso_data(void)
> >  	memset(&machine, 0, sizeof(machine));
> >  
> >  	dso = dso__new((const char *)file);
> > +	TEST_ASSERT_VAL("failed to get dso", dso);
> > +
> > +	dso->binary_type = DSO_BINARY_TYPE__SYSTEM_PATH_DSO;
> >  
> >  	/* Basic 10 bytes tests. */
> >  	for (i = 0; i < ARRAY_SIZE(offsets); i++) {
> > @@ -199,6 +202,8 @@ static int dsos__create(int cnt, int size)
> >  
> >  		dsos[i] = dso__new(file);
> >  		TEST_ASSERT_VAL("failed to get dso", dsos[i]);
> > +
> > +		dsos[i]->binary_type = DSO_BINARY_TYPE__SYSTEM_PATH_DSO;
> 
> Those two are unrelated, please put them in a separate patch, one that I
> can even cherrypick ahead of the other patches.

It's a consequence of changing dso__data_read_offset() not to call
dso__data_fd() due to a performance reason.  The binary_type was
determined during the dso__data_fd() before, but now it needs to be
set explicitly for this test.

In the original code, it was called everytime we access to the dso
cache just to check an error, I guess.  But it's enough to check the
status field.


> 
> >  	}
> >  
> >  	return 0;
> > diff --git a/tools/perf/util/dso.c b/tools/perf/util/dso.c
> > index 11ece224ef50..ae92046ae2c8 100644
> > --- a/tools/perf/util/dso.c
> > +++ b/tools/perf/util/dso.c
> > @@ -213,6 +213,7 @@ bool dso__needs_decompress(struct dso *dso)
> >   */
> >  static LIST_HEAD(dso__data_open);
> >  static long dso__data_open_cnt;
> > +static pthread_mutex_t dso__data_open_lock = PTHREAD_MUTEX_INITIALIZER;
> >  
> >  static void dso__list_add(struct dso *dso)
> >  {
> > @@ -240,7 +241,7 @@ static int do_open(char *name)
> >  		if (fd >= 0)
> >  			return fd;
> >  
> > -		pr_debug("dso open failed, mmap: %s\n",
> > +		pr_debug("dso open failed: %s\n",
> >  			 strerror_r(errno, sbuf, sizeof(sbuf)));
> >  		if (!dso__data_open_cnt || errno != EMFILE)
> 
> Ditto, another unrelated patch, please separate.

Ah, okay.  I kept it since it's just a small change.  But I'd like to
separate if it helps reviewing.


> 
> >  			break;
> > @@ -382,7 +383,9 @@ static void check_data_close(void)
> >   */
> >  void dso__data_close(struct dso *dso)
> >  {
> > +	pthread_mutex_lock(&dso__data_open_lock);
> >  	close_dso(dso);
> > +	pthread_mutex_unlock(&dso__data_open_lock);
> >  }
> >  
> >  /**
> > @@ -405,6 +408,8 @@ int dso__data_fd(struct dso *dso, struct machine *machine)
> >  	if (dso->data.status == DSO_DATA_STATUS_ERROR)
> >  		return -1;
> >  
> > +	pthread_mutex_lock(&dso__data_open_lock);
> > +
> >  	if (dso->data.fd >= 0)
> >  		goto out;
> >  
> > @@ -427,6 +432,7 @@ int dso__data_fd(struct dso *dso, struct machine *machine)
> >  	else
> >  		dso->data.status = DSO_DATA_STATUS_ERROR;
> >  
> > +	pthread_mutex_unlock(&dso__data_open_lock);
> >  	return dso->data.fd;
> >  }
> >  
> > @@ -531,52 +537,66 @@ dso_cache__memcpy(struct dso_cache *cache, u64 offset,
> >  }
> >  
> >  static ssize_t
> > -dso_cache__read(struct dso *dso, u64 offset, u8 *data, ssize_t size)
> > +dso_cache__read(struct dso *dso, struct machine *machine,
> > +		u64 offset, u8 *data, ssize_t size)
> >  {
> >  	struct dso_cache *cache;
> >  	struct dso_cache *old;
> > -	ssize_t ret;
> > -
> > -	do {
> > -		u64 cache_offset;
> 
> While I understand that there was no need for this do { } while (0)
> construct in the first place, removing it in this patch is not
> interesting, as it is both unrelated to this patch and makes the it
> harder to review by just looking at the patch :-\ Please refrain from
> doing this in this patch.

Understood, sorry for bothering! :)


> 
> A later patch that does _just_ that could be done, if you feel like
> doing it.

Okay.


> 
> > +	ssize_t ret = -EINVAL;
> > +	u64 cache_offset;
> >  
> > -		ret = -ENOMEM;
> > +	cache = zalloc(sizeof(*cache) + DSO__DATA_CACHE_SIZE);
> > +	if (!cache)
> > +		return -ENOMEM;
> >  
> > -		cache = zalloc(sizeof(*cache) + DSO__DATA_CACHE_SIZE);
> > -		if (!cache)
> > -			break;
> > +	cache_offset = offset & DSO__DATA_CACHE_MASK;
> >  
> > -		cache_offset = offset & DSO__DATA_CACHE_MASK;
> > -		ret = -EINVAL;
> > +	pthread_mutex_lock(&dso__data_open_lock);
> >  
> > -		if (-1 == lseek(dso->data.fd, cache_offset, SEEK_SET))
> > -			break;
> > +	/*
> > +	 * dso->data.fd might be closed if other thread opened another
> > +	 * file (dso) due to open file limit (RLIMIT_NOFILE).
> > +	 */
> > +	if (dso->data.fd < 0) {
> > +		dso->data.fd = open_dso(dso, machine);
> 
> Also please consider adding a backpointer to machine in the dso object,
> since you need to reopen it, so that we don't have to go on passing
> machine around to dso_cache__read(), etc.

Yeah, it's a pain to passing a machine pointer.

> 
> This probably needs to be done in the patch that makes dso->data.fd to
> be closed due to limit.

I don't know which patch you are refering..  It already closes an fd
if it reaches the limit - what this patch does is protecting such
concurrent open and close when multi-thread is used.

Thanks,
Namhyung


> 
> > +		if (dso->data.fd < 0) {
> > +			ret = -errno;
> > +			dso->data.status = DSO_DATA_STATUS_ERROR;
> > +			goto err_unlock;
> > +		}
> > +	}
> >  
> > -		ret = read(dso->data.fd, cache->data, DSO__DATA_CACHE_SIZE);
> > -		if (ret <= 0)
> > -			break;
> > +	if (-1 == lseek(dso->data.fd, cache_offset, SEEK_SET))
> > +		goto err_unlock;
> >  
> > -		cache->offset = cache_offset;
> > -		cache->size   = ret;
> > -		old = dso_cache__insert(dso, cache);
> > -		if (old) {
> > -			/* we lose the race */
> > -			free(cache);
> > -			cache = old;
> > -		}
> > +	ret = read(dso->data.fd, cache->data, DSO__DATA_CACHE_SIZE);
> > +	if (ret <= 0)
> > +		goto err_unlock;
> >  
> > -		ret = dso_cache__memcpy(cache, offset, data, size);
> > +	pthread_mutex_unlock(&dso__data_open_lock);
> >  
> > -	} while (0);
> > +	cache->offset = cache_offset;
> > +	cache->size   = ret;
> > +	old = dso_cache__insert(dso, cache);
> > +	if (old) {
> > +		/* we lose the race */
> > +		free(cache);
> > +		cache = old;
> > +	}
> >  
> > +	ret = dso_cache__memcpy(cache, offset, data, size);
> >  	if (ret <= 0)
> >  		free(cache);
> >  
> >  	return ret;
> > +
> > +err_unlock:
> > +	pthread_mutex_unlock(&dso__data_open_lock);
> > +	return ret;
> >  }
> >  
> > -static ssize_t dso_cache_read(struct dso *dso, u64 offset,
> > -			      u8 *data, ssize_t size)
> > +static ssize_t dso_cache_read(struct dso *dso, struct machine *machine,
> > +			      u64 offset, u8 *data, ssize_t size)
> >  {
> >  	struct dso_cache *cache;
> >  
> > @@ -584,7 +604,7 @@ static ssize_t dso_cache_read(struct dso *dso, u64 offset,
> >  	if (cache)
> >  		return dso_cache__memcpy(cache, offset, data, size);
> >  	else
> > -		return dso_cache__read(dso, offset, data, size);
> > +		return dso_cache__read(dso, machine, offset, data, size);
> >  }
> >  
> >  /*
> > @@ -592,7 +612,8 @@ static ssize_t dso_cache_read(struct dso *dso, u64 offset,
> >   * in the rb_tree. Any read to already cached data is served
> >   * by cached data.
> >   */
> > -static ssize_t cached_read(struct dso *dso, u64 offset, u8 *data, ssize_t size)
> > +static ssize_t cached_read(struct dso *dso, struct machine *machine,
> > +			   u64 offset, u8 *data, ssize_t size)
> >  {
> >  	ssize_t r = 0;
> >  	u8 *p = data;
> > @@ -600,7 +621,7 @@ static ssize_t cached_read(struct dso *dso, u64 offset, u8 *data, ssize_t size)
> >  	do {
> >  		ssize_t ret;
> >  
> > -		ret = dso_cache_read(dso, offset, p, size);
> > +		ret = dso_cache_read(dso, machine, offset, p, size);
> >  		if (ret < 0)
> >  			return ret;
> >  
> > @@ -620,21 +641,42 @@ static ssize_t cached_read(struct dso *dso, u64 offset, u8 *data, ssize_t size)
> >  	return r;
> >  }
> >  
> > -static int data_file_size(struct dso *dso)
> > +static int data_file_size(struct dso *dso, struct machine *machine)
> >  {
> > +	int ret = 0;
> >  	struct stat st;
> >  	char sbuf[STRERR_BUFSIZE];
> >  
> > -	if (!dso->data.file_size) {
> > -		if (fstat(dso->data.fd, &st)) {
> > -			pr_err("dso mmap failed, fstat: %s\n",
> > -				strerror_r(errno, sbuf, sizeof(sbuf)));
> > -			return -1;
> > +	if (dso->data.file_size)
> > +		return 0;
> > +
> > +	pthread_mutex_lock(&dso__data_open_lock);
> > +
> > +	/*
> > +	 * dso->data.fd might be closed if other thread opened another
> > +	 * file (dso) due to open file limit (RLIMIT_NOFILE).
> > +	 */
> > +	if (dso->data.fd < 0) {
> > +		dso->data.fd = open_dso(dso, machine);
> > +		if (dso->data.fd < 0) {
> > +			ret = -errno;
> > +			dso->data.status = DSO_DATA_STATUS_ERROR;
> > +			goto out;
> >  		}
> > -		dso->data.file_size = st.st_size;
> >  	}
> >  
> > -	return 0;
> > +	if (fstat(dso->data.fd, &st) < 0) {
> > +		ret = -errno;
> > +		pr_err("dso cache fstat failed: %s\n",
> > +		       strerror_r(errno, sbuf, sizeof(sbuf)));
> > +		dso->data.status = DSO_DATA_STATUS_ERROR;
> > +		goto out;
> > +	}
> > +	dso->data.file_size = st.st_size;
> > +
> > +out:
> > +	pthread_mutex_unlock(&dso__data_open_lock);
> > +	return ret;
> >  }
> >  
> >  /**
> > @@ -652,17 +694,17 @@ off_t dso__data_size(struct dso *dso, struct machine *machine)
> >  	if (fd < 0)
> >  		return fd;
> >  
> > -	if (data_file_size(dso))
> > +	if (data_file_size(dso, machine))
> >  		return -1;
> >  
> >  	/* For now just estimate dso data size is close to file size */
> >  	return dso->data.file_size;
> >  }
> >  
> > -static ssize_t data_read_offset(struct dso *dso, u64 offset,
> > -				u8 *data, ssize_t size)
> > +static ssize_t data_read_offset(struct dso *dso, struct machine *machine,
> > +				u64 offset, u8 *data, ssize_t size)
> >  {
> > -	if (data_file_size(dso))
> > +	if (data_file_size(dso, machine))
> >  		return -1;
> >  
> >  	/* Check the offset sanity. */
> > @@ -672,7 +714,7 @@ static ssize_t data_read_offset(struct dso *dso, u64 offset,
> >  	if (offset + size < offset)
> >  		return -1;
> >  
> > -	return cached_read(dso, offset, data, size);
> > +	return cached_read(dso, machine, offset, data, size);
> >  }
> >  
> >  /**
> > @@ -689,10 +731,10 @@ static ssize_t data_read_offset(struct dso *dso, u64 offset,
> >  ssize_t dso__data_read_offset(struct dso *dso, struct machine *machine,
> >  			      u64 offset, u8 *data, ssize_t size)
> >  {
> > -	if (dso__data_fd(dso, machine) < 0)
> > +	if (dso->data.status == DSO_DATA_STATUS_ERROR)
> >  		return -1;
> >  
> > -	return data_read_offset(dso, offset, data, size);
> > +	return data_read_offset(dso, machine, offset, data, size);
> >  }
> >  
> >  /**
> > -- 
> > 2.2.2

^ permalink raw reply	[flat|nested] 221+ messages in thread

* Re: [PATCH 40/42] perf callchain: Save eh/debug frame offset for dwarf unwind
  2015-01-29 12:38   ` Arnaldo Carvalho de Melo
@ 2015-01-29 13:23     ` Namhyung Kim
  0 siblings, 0 replies; 221+ messages in thread
From: Namhyung Kim @ 2015-01-29 13:23 UTC (permalink / raw)
  To: Arnaldo Carvalho de Melo
  Cc: Ingo Molnar, Peter Zijlstra, Jiri Olsa, LKML, David Ahern,
	Adrian Hunter, Andi Kleen, Stephane Eranian, Frederic Weisbecker

On Thu, Jan 29, 2015 at 09:38:31AM -0300, Arnaldo Carvalho de Melo wrote:
> Em Thu, Jan 29, 2015 at 05:07:21PM +0900, Namhyung Kim escreveu:
> > When libunwind tries to resolve callchains it needs to know the offset
> > of .eh_frame_hdr or .debug_frame to access the dso.  Since it calls
> > dso__data_fd(), it'll try to grab dso->lock everytime for same
> > information.  So save it to dso_data struct and reuse it.
> > 
> > Note that there's a window between dso__data_fd() and actual use of
> > the fd.  The fd could be closed by other threads to deal with the open
> > file limit in dso cache code.  But I think it's ok since in that case
> > elf_section_offset() will return 0 so it'll be tried in next acess.
> 
> I know that you did this in the context of your multi threading
> patchkit, but this seems useful even without that patckhit, i.e. this
> can be cherry picked on the grounds that it speeds up things by caching
> something that doesn't change, right?

Right.

> 
> I.e. I'll probably just rewrite the comment and apply it before
> considering the other patches, so that other people can comment on the
> other patches, etc.

Thanks for doing that!
Namhyung


> 
> - Arnaldo
>  
> > Signed-off-by: Namhyung Kim <namhyung@kernel.org>
> > ---
> >  tools/perf/util/dso.h              |  1 +
> >  tools/perf/util/unwind-libunwind.c | 31 ++++++++++++++++++++-----------
> >  2 files changed, 21 insertions(+), 11 deletions(-)
> > 
> > diff --git a/tools/perf/util/dso.h b/tools/perf/util/dso.h
> > index c18fcc0e8081..323ee08d56fc 100644
> > --- a/tools/perf/util/dso.h
> > +++ b/tools/perf/util/dso.h
> > @@ -141,6 +141,7 @@ struct dso {
> >  		u32		 status_seen;
> >  		size_t		 file_size;
> >  		struct list_head open_entry;
> > +		u64		 frame_offset;
> >  	} data;
> >  
> >  	union { /* Tool specific area */
> > diff --git a/tools/perf/util/unwind-libunwind.c b/tools/perf/util/unwind-libunwind.c
> > index 7ed6eaf232b6..3219b20837b5 100644
> > --- a/tools/perf/util/unwind-libunwind.c
> > +++ b/tools/perf/util/unwind-libunwind.c
> > @@ -266,14 +266,17 @@ static int read_unwind_spec_eh_frame(struct dso *dso, struct machine *machine,
> >  				     u64 *fde_count)
> >  {
> >  	int ret = -EINVAL, fd;
> > -	u64 offset;
> > +	u64 offset = dso->data.frame_offset;
> >  
> > -	fd = dso__data_fd(dso, machine);
> > -	if (fd < 0)
> > -		return -EINVAL;
> > +	if (offset == 0) {
> > +		fd = dso__data_fd(dso, machine);
> > +		if (fd < 0)
> > +			return -EINVAL;
> >  
> > -	/* Check the .eh_frame section for unwinding info */
> > -	offset = elf_section_offset(fd, ".eh_frame_hdr");
> > +		/* Check the .eh_frame section for unwinding info */
> > +		offset = elf_section_offset(fd, ".eh_frame_hdr");
> > +		dso->data.frame_offset = offset;
> > +	}
> >  
> >  	if (offset)
> >  		ret = unwind_spec_ehframe(dso, machine, offset,
> > @@ -287,14 +290,20 @@ static int read_unwind_spec_eh_frame(struct dso *dso, struct machine *machine,
> >  static int read_unwind_spec_debug_frame(struct dso *dso,
> >  					struct machine *machine, u64 *offset)
> >  {
> > -	int fd = dso__data_fd(dso, machine);
> > +	int fd;
> > +	u64 ofs = dso->data.frame_offset;
> >  
> > -	if (fd < 0)
> > -		return -EINVAL;
> > +	if (ofs == 0) {
> > +		fd = dso__data_fd(dso, machine);
> > +		if (fd < 0)
> > +			return -EINVAL;
> >  
> > -	/* Check the .debug_frame section for unwinding info */
> > -	*offset = elf_section_offset(fd, ".debug_frame");
> > +		/* Check the .debug_frame section for unwinding info */
> > +		ofs = elf_section_offset(fd, ".debug_frame");
> > +		dso->data.frame_offset = ofs;
> > +	}
> >  
> > +	*offset = ofs;
> >  	if (*offset)
> >  		return 0;
> >  
> > -- 
> > 2.2.2

^ permalink raw reply	[flat|nested] 221+ messages in thread

* Re: [PATCH 29/42] perf tools: Protect dso cache fd with a mutex
  2015-01-29 13:19     ` Namhyung Kim
@ 2015-01-29 16:23       ` Arnaldo Carvalho de Melo
  2015-01-30  0:51         ` Namhyung Kim
  0 siblings, 1 reply; 221+ messages in thread
From: Arnaldo Carvalho de Melo @ 2015-01-29 16:23 UTC (permalink / raw)
  To: Namhyung Kim
  Cc: Ingo Molnar, Peter Zijlstra, Jiri Olsa, LKML, David Ahern,
	Adrian Hunter, Andi Kleen, Stephane Eranian, Frederic Weisbecker

Em Thu, Jan 29, 2015 at 10:19:38PM +0900, Namhyung Kim escreveu:
> On Thu, Jan 29, 2015 at 09:31:07AM -0300, Arnaldo Carvalho de Melo wrote:
> > Em Thu, Jan 29, 2015 at 05:07:10PM +0900, Namhyung Kim escreveu:
> > > When dso cache is accessed in multi-thread environment, it's possible
> > > to close other dso->data.fd during operation due to open file limit.
> > > Protect the file descriptors using a separate mutex.
> > > 
> > > Signed-off-by: Namhyung Kim <namhyung@kernel.org>
> > > ---
> > >  tools/perf/tests/dso-data.c |   5 ++
> > >  tools/perf/util/dso.c       | 136 +++++++++++++++++++++++++++++---------------
> > >  2 files changed, 94 insertions(+), 47 deletions(-)
> > > 
> > > diff --git a/tools/perf/tests/dso-data.c b/tools/perf/tests/dso-data.c
> > > index caaf37f079b1..0276e7d2d41b 100644
> > > --- a/tools/perf/tests/dso-data.c
> > > +++ b/tools/perf/tests/dso-data.c
> > > @@ -111,6 +111,9 @@ int test__dso_data(void)
> > >  	memset(&machine, 0, sizeof(machine));
> > >  
> > >  	dso = dso__new((const char *)file);
> > > +	TEST_ASSERT_VAL("failed to get dso", dso);
> > > +
> > > +	dso->binary_type = DSO_BINARY_TYPE__SYSTEM_PATH_DSO;
> > >  
> > >  	/* Basic 10 bytes tests. */
> > >  	for (i = 0; i < ARRAY_SIZE(offsets); i++) {
> > > @@ -199,6 +202,8 @@ static int dsos__create(int cnt, int size)
> > >  
> > >  		dsos[i] = dso__new(file);
> > >  		TEST_ASSERT_VAL("failed to get dso", dsos[i]);
> > > +
> > > +		dsos[i]->binary_type = DSO_BINARY_TYPE__SYSTEM_PATH_DSO;
> > 
> > Those two are unrelated, please put them in a separate patch, one that I
> > can even cherrypick ahead of the other patches.
> 
> It's a consequence of changing dso__data_read_offset() not to call
> dso__data_fd() due to a performance reason.  The binary_type was
> determined during the dso__data_fd() before, but now it needs to be
> set explicitly for this test.
> 
> In the original code, it was called everytime we access to the dso
> cache just to check an error, I guess.  But it's enough to check the
> status field.

Are you saying that this test should not rely on some function that is
called somewhere down the functions it uses and should instead do as you
do above?

I.e. if that is the case, then this stands out as a separate patch, if
not, if this is indeed really related to this patch (at first sight it
doesn't look like) then this explanation you give should be in the
patch comment log.

> > >  	return 0;
> > > diff --git a/tools/perf/util/dso.c b/tools/perf/util/dso.c
> > > index 11ece224ef50..ae92046ae2c8 100644
> > > --- a/tools/perf/util/dso.c
> > > +++ b/tools/perf/util/dso.c
> > > @@ -213,6 +213,7 @@ bool dso__needs_decompress(struct dso *dso)
> > >   */
> > >  static LIST_HEAD(dso__data_open);
> > >  static long dso__data_open_cnt;
> > > +static pthread_mutex_t dso__data_open_lock = PTHREAD_MUTEX_INITIALIZER;
> > >  
> > >  static void dso__list_add(struct dso *dso)
> > >  {
> > > @@ -240,7 +241,7 @@ static int do_open(char *name)
> > >  		if (fd >= 0)
> > >  			return fd;
> > >  
> > > -		pr_debug("dso open failed, mmap: %s\n",
> > > +		pr_debug("dso open failed: %s\n",
> > >  			 strerror_r(errno, sbuf, sizeof(sbuf)));
> > >  		if (!dso__data_open_cnt || errno != EMFILE)
> > 
> > Ditto, another unrelated patch, please separate.
> 
> Ah, okay.  I kept it since it's just a small change.  But I'd like to
> separate if it helps reviewing.

Thanks

> > >  			break;
> > > @@ -382,7 +383,9 @@ static void check_data_close(void)
> > >   */
> > >  void dso__data_close(struct dso *dso)
> > >  {
> > > +	pthread_mutex_lock(&dso__data_open_lock);
> > >  	close_dso(dso);
> > > +	pthread_mutex_unlock(&dso__data_open_lock);
> > >  }
> > >  
> > >  /**
> > > @@ -405,6 +408,8 @@ int dso__data_fd(struct dso *dso, struct machine *machine)
> > >  	if (dso->data.status == DSO_DATA_STATUS_ERROR)
> > >  		return -1;
> > >  
> > > +	pthread_mutex_lock(&dso__data_open_lock);
> > > +
> > >  	if (dso->data.fd >= 0)
> > >  		goto out;
> > >  
> > > @@ -427,6 +432,7 @@ int dso__data_fd(struct dso *dso, struct machine *machine)
> > >  	else
> > >  		dso->data.status = DSO_DATA_STATUS_ERROR;
> > >  
> > > +	pthread_mutex_unlock(&dso__data_open_lock);
> > >  	return dso->data.fd;
> > >  }
> > >  
> > > @@ -531,52 +537,66 @@ dso_cache__memcpy(struct dso_cache *cache, u64 offset,
> > >  }
> > >  
> > >  static ssize_t
> > > -dso_cache__read(struct dso *dso, u64 offset, u8 *data, ssize_t size)
> > > +dso_cache__read(struct dso *dso, struct machine *machine,
> > > +		u64 offset, u8 *data, ssize_t size)
> > >  {
> > >  	struct dso_cache *cache;
> > >  	struct dso_cache *old;
> > > -	ssize_t ret;
> > > -
> > > -	do {
> > > -		u64 cache_offset;
> > 
> > While I understand that there was no need for this do { } while (0)
> > construct in the first place, removing it in this patch is not
> > interesting, as it is both unrelated to this patch and makes the it
> > harder to review by just looking at the patch :-\ Please refrain from
> > doing this in this patch.
> 
> Understood, sorry for bothering! :)

:-)

> > A later patch that does _just_ that could be done, if you feel like
> > doing it.
> 
> Okay.
> 
> 
> > 
> > > +	ssize_t ret = -EINVAL;
> > > +	u64 cache_offset;
> > >  
> > > -		ret = -ENOMEM;
> > > +	cache = zalloc(sizeof(*cache) + DSO__DATA_CACHE_SIZE);
> > > +	if (!cache)
> > > +		return -ENOMEM;
> > >  
> > > -		cache = zalloc(sizeof(*cache) + DSO__DATA_CACHE_SIZE);
> > > -		if (!cache)
> > > -			break;
> > > +	cache_offset = offset & DSO__DATA_CACHE_MASK;
> > >  
> > > -		cache_offset = offset & DSO__DATA_CACHE_MASK;
> > > -		ret = -EINVAL;
> > > +	pthread_mutex_lock(&dso__data_open_lock);
> > >  
> > > -		if (-1 == lseek(dso->data.fd, cache_offset, SEEK_SET))
> > > -			break;
> > > +	/*
> > > +	 * dso->data.fd might be closed if other thread opened another
> > > +	 * file (dso) due to open file limit (RLIMIT_NOFILE).
> > > +	 */
> > > +	if (dso->data.fd < 0) {
> > > +		dso->data.fd = open_dso(dso, machine);
> > 
> > Also please consider adding a backpointer to machine in the dso object,
> > since you need to reopen it, so that we don't have to go on passing
> > machine around to dso_cache__read(), etc.
> 
> Yeah, it's a pain to passing a machine pointer.

Hey, so setting of dso->data.fd is protected in this function and we can
be sure that it will not be closed _again_ just before we do that lseek
and other operations, I guess so, just checking...
 
> > 
> > This probably needs to be done in the patch that makes dso->data.fd to
> > be closed due to limit.
> 
> I don't know which patch you are refering..  It already closes an fd
> if it reaches the limit - what this patch does is protecting such
> concurrent open and close when multi-thread is used.

Ok then, i.e.:

A) take the lock, close it if over the limit, drop the lock.

B) take the lock, check if it is closed, open if so, use it, drop the
lock.

Is that right?

- Arnaldo

^ permalink raw reply	[flat|nested] 221+ messages in thread

* Re: [PATCH 40/42] perf callchain: Save eh/debug frame offset for dwarf unwind
  2015-01-29  8:07 ` [PATCH 40/42] perf callchain: Save eh/debug frame offset for dwarf unwind Namhyung Kim
  2015-01-29 12:38   ` Arnaldo Carvalho de Melo
@ 2015-01-29 19:22   ` Arnaldo Carvalho de Melo
  2015-01-30 18:32   ` [tip:perf/core] perf callchain: Cache eh/ debug " tip-bot for Namhyung Kim
  2 siblings, 0 replies; 221+ messages in thread
From: Arnaldo Carvalho de Melo @ 2015-01-29 19:22 UTC (permalink / raw)
  To: Namhyung Kim
  Cc: Ingo Molnar, Peter Zijlstra, Jiri Olsa, LKML, David Ahern,
	Adrian Hunter, Andi Kleen, Stephane Eranian, Frederic Weisbecker

Em Thu, Jan 29, 2015 at 05:07:21PM +0900, Namhyung Kim escreveu:
> When libunwind tries to resolve callchains it needs to know the offset
> of .eh_frame_hdr or .debug_frame to access the dso.  Since it calls
> dso__data_fd(), it'll try to grab dso->lock everytime for same
> information.  So save it to dso_data struct and reuse it.
> 
> Note that there's a window between dso__data_fd() and actual use of
> the fd.  The fd could be closed by other threads to deal with the open
> file limit in dso cache code.  But I think it's ok since in that case
> elf_section_offset() will return 0 so it'll be tried in next acess.

Thanks, applied after rewriting the changelog to read as:

---
    perf callchain: Cache eh/debug frame offset for dwarf unwind
    
    When libunwind tries to resolve callchains it needs to know the
    offset of .eh_frame_hdr or .debug_frame to access the dso.
    
    Since it will always return the same result for a given DSO, just
    cache the result as an optimization.
---

- Arnaldo

^ permalink raw reply	[flat|nested] 221+ messages in thread

* Re: [RFC/PATCHSET 00/42] perf tools: Speed-up perf report by using multi thread (v2)
  2015-01-29  8:06 [RFC/PATCHSET 00/42] perf tools: Speed-up perf report by using multi thread (v2) Namhyung Kim
                   ` (41 preceding siblings ...)
  2015-01-29  8:07 ` [PATCH 42/42] perf data: Implement 'index' subcommand Namhyung Kim
@ 2015-01-29 19:56 ` Arnaldo Carvalho de Melo
  42 siblings, 0 replies; 221+ messages in thread
From: Arnaldo Carvalho de Melo @ 2015-01-29 19:56 UTC (permalink / raw)
  To: Namhyung Kim
  Cc: Ingo Molnar, Peter Zijlstra, Jiri Olsa, LKML, David Ahern,
	Adrian Hunter, Andi Kleen, Stephane Eranian, Frederic Weisbecker


Applied 1-5 and dwarf unwind caching one, will look at the others.

- Arnaldo

^ permalink raw reply	[flat|nested] 221+ messages in thread

* Re: [PATCH 29/42] perf tools: Protect dso cache fd with a mutex
  2015-01-29 16:23       ` Arnaldo Carvalho de Melo
@ 2015-01-30  0:51         ` Namhyung Kim
  0 siblings, 0 replies; 221+ messages in thread
From: Namhyung Kim @ 2015-01-30  0:51 UTC (permalink / raw)
  To: Arnaldo Carvalho de Melo
  Cc: Ingo Molnar, Peter Zijlstra, Jiri Olsa, LKML, David Ahern,
	Adrian Hunter, Andi Kleen, Stephane Eranian, Frederic Weisbecker

On Thu, Jan 29, 2015 at 01:23:33PM -0300, Arnaldo Carvalho de Melo wrote:
> Em Thu, Jan 29, 2015 at 10:19:38PM +0900, Namhyung Kim escreveu:
> > On Thu, Jan 29, 2015 at 09:31:07AM -0300, Arnaldo Carvalho de Melo wrote:
> > > Em Thu, Jan 29, 2015 at 05:07:10PM +0900, Namhyung Kim escreveu:
> > > > When dso cache is accessed in multi-thread environment, it's possible
> > > > to close other dso->data.fd during operation due to open file limit.
> > > > Protect the file descriptors using a separate mutex.
> > > > 
> > > > Signed-off-by: Namhyung Kim <namhyung@kernel.org>
> > > > ---
> > > >  tools/perf/tests/dso-data.c |   5 ++
> > > >  tools/perf/util/dso.c       | 136 +++++++++++++++++++++++++++++---------------
> > > >  2 files changed, 94 insertions(+), 47 deletions(-)
> > > > 
> > > > diff --git a/tools/perf/tests/dso-data.c b/tools/perf/tests/dso-data.c
> > > > index caaf37f079b1..0276e7d2d41b 100644
> > > > --- a/tools/perf/tests/dso-data.c
> > > > +++ b/tools/perf/tests/dso-data.c
> > > > @@ -111,6 +111,9 @@ int test__dso_data(void)
> > > >  	memset(&machine, 0, sizeof(machine));
> > > >  
> > > >  	dso = dso__new((const char *)file);
> > > > +	TEST_ASSERT_VAL("failed to get dso", dso);
> > > > +
> > > > +	dso->binary_type = DSO_BINARY_TYPE__SYSTEM_PATH_DSO;
> > > >  
> > > >  	/* Basic 10 bytes tests. */
> > > >  	for (i = 0; i < ARRAY_SIZE(offsets); i++) {
> > > > @@ -199,6 +202,8 @@ static int dsos__create(int cnt, int size)
> > > >  
> > > >  		dsos[i] = dso__new(file);
> > > >  		TEST_ASSERT_VAL("failed to get dso", dsos[i]);
> > > > +
> > > > +		dsos[i]->binary_type = DSO_BINARY_TYPE__SYSTEM_PATH_DSO;
> > > 
> > > Those two are unrelated, please put them in a separate patch, one that I
> > > can even cherrypick ahead of the other patches.
> > 
> > It's a consequence of changing dso__data_read_offset() not to call
> > dso__data_fd() due to a performance reason.  The binary_type was
> > determined during the dso__data_fd() before, but now it needs to be
> > set explicitly for this test.
> > 
> > In the original code, it was called everytime we access to the dso
> > cache just to check an error, I guess.  But it's enough to check the
> > status field.
> 
> Are you saying that this test should not rely on some function that is
> called somewhere down the functions it uses and should instead do as you
> do above?
> 
> I.e. if that is the case, then this stands out as a separate patch, if
> not, if this is indeed really related to this patch (at first sight it
> doesn't look like) then this explanation you give should be in the
> patch comment log.

I think it'd be better adding dso__data_fd() after dso__new() as an
extra validation step.  With that we don't need to specify binary type
manually and have a more consistent usage pattern.  I'll do it as a
separate patch.


> 
> > > >  	return 0;
> > > > diff --git a/tools/perf/util/dso.c b/tools/perf/util/dso.c
> > > > index 11ece224ef50..ae92046ae2c8 100644
> > > > --- a/tools/perf/util/dso.c
> > > > +++ b/tools/perf/util/dso.c
> > > > @@ -213,6 +213,7 @@ bool dso__needs_decompress(struct dso *dso)
> > > >   */
> > > >  static LIST_HEAD(dso__data_open);
> > > >  static long dso__data_open_cnt;
> > > > +static pthread_mutex_t dso__data_open_lock = PTHREAD_MUTEX_INITIALIZER;
> > > >  
> > > >  static void dso__list_add(struct dso *dso)
> > > >  {
> > > > @@ -240,7 +241,7 @@ static int do_open(char *name)
> > > >  		if (fd >= 0)
> > > >  			return fd;
> > > >  
> > > > -		pr_debug("dso open failed, mmap: %s\n",
> > > > +		pr_debug("dso open failed: %s\n",
> > > >  			 strerror_r(errno, sbuf, sizeof(sbuf)));
> > > >  		if (!dso__data_open_cnt || errno != EMFILE)
> > > 
> > > Ditto, another unrelated patch, please separate.
> > 
> > Ah, okay.  I kept it since it's just a small change.  But I'd like to
> > separate if it helps reviewing.
> 
> Thanks
> 
> > > >  			break;
> > > > @@ -382,7 +383,9 @@ static void check_data_close(void)
> > > >   */
> > > >  void dso__data_close(struct dso *dso)
> > > >  {
> > > > +	pthread_mutex_lock(&dso__data_open_lock);
> > > >  	close_dso(dso);
> > > > +	pthread_mutex_unlock(&dso__data_open_lock);
> > > >  }
> > > >  
> > > >  /**
> > > > @@ -405,6 +408,8 @@ int dso__data_fd(struct dso *dso, struct machine *machine)
> > > >  	if (dso->data.status == DSO_DATA_STATUS_ERROR)
> > > >  		return -1;
> > > >  
> > > > +	pthread_mutex_lock(&dso__data_open_lock);
> > > > +
> > > >  	if (dso->data.fd >= 0)
> > > >  		goto out;
> > > >  
> > > > @@ -427,6 +432,7 @@ int dso__data_fd(struct dso *dso, struct machine *machine)
> > > >  	else
> > > >  		dso->data.status = DSO_DATA_STATUS_ERROR;
> > > >  
> > > > +	pthread_mutex_unlock(&dso__data_open_lock);
> > > >  	return dso->data.fd;
> > > >  }
> > > >  
> > > > @@ -531,52 +537,66 @@ dso_cache__memcpy(struct dso_cache *cache, u64 offset,
> > > >  }
> > > >  
> > > >  static ssize_t
> > > > -dso_cache__read(struct dso *dso, u64 offset, u8 *data, ssize_t size)
> > > > +dso_cache__read(struct dso *dso, struct machine *machine,
> > > > +		u64 offset, u8 *data, ssize_t size)
> > > >  {
> > > >  	struct dso_cache *cache;
> > > >  	struct dso_cache *old;
> > > > -	ssize_t ret;
> > > > -
> > > > -	do {
> > > > -		u64 cache_offset;
> > > 
> > > While I understand that there was no need for this do { } while (0)
> > > construct in the first place, removing it in this patch is not
> > > interesting, as it is both unrelated to this patch and makes the it
> > > harder to review by just looking at the patch :-\ Please refrain from
> > > doing this in this patch.
> > 
> > Understood, sorry for bothering! :)
> 
> :-)
> 
> > > A later patch that does _just_ that could be done, if you feel like
> > > doing it.
> > 
> > Okay.
> > 
> > 
> > > 
> > > > +	ssize_t ret = -EINVAL;
> > > > +	u64 cache_offset;
> > > >  
> > > > -		ret = -ENOMEM;
> > > > +	cache = zalloc(sizeof(*cache) + DSO__DATA_CACHE_SIZE);
> > > > +	if (!cache)
> > > > +		return -ENOMEM;
> > > >  
> > > > -		cache = zalloc(sizeof(*cache) + DSO__DATA_CACHE_SIZE);
> > > > -		if (!cache)
> > > > -			break;
> > > > +	cache_offset = offset & DSO__DATA_CACHE_MASK;
> > > >  
> > > > -		cache_offset = offset & DSO__DATA_CACHE_MASK;
> > > > -		ret = -EINVAL;
> > > > +	pthread_mutex_lock(&dso__data_open_lock);
> > > >  
> > > > -		if (-1 == lseek(dso->data.fd, cache_offset, SEEK_SET))
> > > > -			break;
> > > > +	/*
> > > > +	 * dso->data.fd might be closed if other thread opened another
> > > > +	 * file (dso) due to open file limit (RLIMIT_NOFILE).
> > > > +	 */
> > > > +	if (dso->data.fd < 0) {
> > > > +		dso->data.fd = open_dso(dso, machine);
> > > 
> > > Also please consider adding a backpointer to machine in the dso object,
> > > since you need to reopen it, so that we don't have to go on passing
> > > machine around to dso_cache__read(), etc.
> > 
> > Yeah, it's a pain to passing a machine pointer.
> 
> Hey, so setting of dso->data.fd is protected in this function and we can
> be sure that it will not be closed _again_ just before we do that lseek
> and other operations, I guess so, just checking...

Right.  Accessing to dso->data.fd is safe only if it grabs the
dso__data_open_lock.


>  
> > > 
> > > This probably needs to be done in the patch that makes dso->data.fd to
> > > be closed due to limit.
> > 
> > I don't know which patch you are refering..  It already closes an fd
> > if it reaches the limit - what this patch does is protecting such
> > concurrent open and close when multi-thread is used.
> 
> Ok then, i.e.:
> 
> A) take the lock, close it if over the limit, drop the lock.
> 
> B) take the lock, check if it is closed, open if so, use it, drop the
> lock.
> 
> Is that right?

It's like this:

A) take the lock, check if it's still open, use it and drop the lock.

B) take the lock, it's closed, reopen it and check if it reaches the
limit.  Then close the first (kinda in LRU) dso to make next open()
succeeded.  Use it and drop the lock.

Thanks,
Namhyung

^ permalink raw reply	[flat|nested] 221+ messages in thread

* Re: [PATCH 01/42] perf tools: Support to read compressed module from build-id cache
  2015-01-29  8:06 ` [PATCH 01/42] perf tools: Support to read compressed module from build-id cache Namhyung Kim
@ 2015-01-30 14:32   ` Jiri Olsa
  2015-02-02 15:03     ` Namhyung Kim
  2015-01-30 18:33   ` [tip:perf/core] perf symbols: " tip-bot for Namhyung Kim
  1 sibling, 1 reply; 221+ messages in thread
From: Jiri Olsa @ 2015-01-30 14:32 UTC (permalink / raw)
  To: Namhyung Kim
  Cc: Arnaldo Carvalho de Melo, Ingo Molnar, Peter Zijlstra, LKML,
	David Ahern, Adrian Hunter, Andi Kleen, Stephane Eranian,
	Frederic Weisbecker

On Thu, Jan 29, 2015 at 05:06:42PM +0900, Namhyung Kim wrote:
> The commit c00c48fc6e6e ("perf symbols: Preparation for compressed
> kernel module support") added support for compressed kernel modules
> but it only supports system path DSOs.  When a dso is read from
> build-id cache, its filename doesn't end with ".gz" but has build-id.
> In this case, we should fallback to the original dso->name.
> 
> Signed-off-by: Namhyung Kim <namhyung@kernel.org>
> ---
>  tools/perf/util/symbol-elf.c | 13 ++++++++-----
>  1 file changed, 8 insertions(+), 5 deletions(-)
> 
> diff --git a/tools/perf/util/symbol-elf.c b/tools/perf/util/symbol-elf.c
> index 06fcd1bf98b6..b24f9d8727a8 100644
> --- a/tools/perf/util/symbol-elf.c
> +++ b/tools/perf/util/symbol-elf.c
> @@ -574,13 +574,16 @@ static int decompress_kmodule(struct dso *dso, const char *name,
>  	const char *ext = strrchr(name, '.');
>  	char tmpbuf[] = "/tmp/perf-kmod-XXXXXX";
>  
> -	if ((type != DSO_BINARY_TYPE__SYSTEM_PATH_KMODULE_COMP &&
> -	     type != DSO_BINARY_TYPE__GUEST_KMODULE_COMP) ||
> -	    type != dso->symtab_type)
> +	if (type != DSO_BINARY_TYPE__SYSTEM_PATH_KMODULE_COMP &&
> +	    type != DSO_BINARY_TYPE__GUEST_KMODULE_COMP &&
> +	    type != DSO_BINARY_TYPE__BUILD_ID_CACHE)
>  		return -1;

hum, is it possible the type == DSO_BINARY_TYPE__BUILD_ID_CACHE could get in here?


---
        for (i = 0; i < DSO_BINARY_TYPE__SYMTAB_CNT; i++) {
                struct symsrc *ss = &ss_[ss_pos];
                bool next_slot = false;

                enum dso_binary_type symtab_type = binary_type_symtab[i];

                if (!dso__is_compatible_symtab_type(dso, kmod, symtab_type))
                        continue;

---		^^^ this check should rule out buildid symtab_type for kmod dso?

		symsrc__init(
			

I wonder wether we should set special type from compressed binaries (as of now),
or instead try to decompress anything that looks like it's compressed ;-)
it seems more to be more generic and could simplify the code..

jirka

^ permalink raw reply	[flat|nested] 221+ messages in thread

* Re: [PATCH 08/42] perf tools: Add rm_rf() utility function
  2015-01-29  8:06 ` [PATCH 08/42] perf tools: Add rm_rf() utility function Namhyung Kim
@ 2015-01-30 15:02   ` Jiri Olsa
  2015-05-20 12:24     ` [tip:perf/core] " tip-bot for Namhyung Kim
  0 siblings, 1 reply; 221+ messages in thread
From: Jiri Olsa @ 2015-01-30 15:02 UTC (permalink / raw)
  To: Namhyung Kim
  Cc: Arnaldo Carvalho de Melo, Ingo Molnar, Peter Zijlstra, LKML,
	David Ahern, Adrian Hunter, Andi Kleen, Stephane Eranian,
	Frederic Weisbecker

On Thu, Jan 29, 2015 at 05:06:49PM +0900, Namhyung Kim wrote:
> The rm_rf() function does same as the shell command 'rm -rf' which
> removes all directory entries recursively.

Acked-by: Jiri Olsa <jolsa@kernel.org>

thanks,
jirka

^ permalink raw reply	[flat|nested] 221+ messages in thread

* [tip:perf/core] perf callchain: Cache eh/ debug frame offset for dwarf unwind
  2015-01-29  8:07 ` [PATCH 40/42] perf callchain: Save eh/debug frame offset for dwarf unwind Namhyung Kim
  2015-01-29 12:38   ` Arnaldo Carvalho de Melo
  2015-01-29 19:22   ` Arnaldo Carvalho de Melo
@ 2015-01-30 18:32   ` tip-bot for Namhyung Kim
  2 siblings, 0 replies; 221+ messages in thread
From: tip-bot for Namhyung Kim @ 2015-01-30 18:32 UTC (permalink / raw)
  To: linux-tip-commits
  Cc: a.p.zijlstra, fweisbec, hpa, andi, dsahern, eranian, jolsa,
	adrian.hunter, mingo, linux-kernel, tglx, acme, namhyung

Commit-ID:  f1f13af99a903ae873f5373e965508e0486c1c29
Gitweb:     http://git.kernel.org/tip/f1f13af99a903ae873f5373e965508e0486c1c29
Author:     Namhyung Kim <namhyung@kernel.org>
AuthorDate: Thu, 29 Jan 2015 17:07:21 +0900
Committer:  Arnaldo Carvalho de Melo <acme@redhat.com>
CommitDate: Thu, 29 Jan 2015 16:20:42 -0300

perf callchain: Cache eh/debug frame offset for dwarf unwind

When libunwind tries to resolve callchains it needs to know the offset
of .eh_frame_hdr or .debug_frame to access the dso.

Since it will always return the same result for a given DSO, just cache
the result as an optimization.

Signed-off-by: Namhyung Kim <namhyung@kernel.org>
Cc: Adrian Hunter <adrian.hunter@intel.com>
Cc: Andi Kleen <andi@firstfloor.org>
Cc: David Ahern <dsahern@gmail.com>
Cc: Frederic Weisbecker <fweisbec@gmail.com>
Cc: Jiri Olsa <jolsa@redhat.com>
Cc: Peter Zijlstra <a.p.zijlstra@chello.nl>
Cc: Stephane Eranian <eranian@google.com>
Link: http://lkml.kernel.org/r/1422518843-25818-41-git-send-email-namhyung@kernel.org
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
---
 tools/perf/util/dso.h              |  1 +
 tools/perf/util/unwind-libunwind.c | 31 ++++++++++++++++++++-----------
 2 files changed, 21 insertions(+), 11 deletions(-)

diff --git a/tools/perf/util/dso.h b/tools/perf/util/dso.h
index 3782c82..ced9284 100644
--- a/tools/perf/util/dso.h
+++ b/tools/perf/util/dso.h
@@ -139,6 +139,7 @@ struct dso {
 		u32		 status_seen;
 		size_t		 file_size;
 		struct list_head open_entry;
+		u64		 frame_offset;
 	} data;
 
 	union { /* Tool specific area */
diff --git a/tools/perf/util/unwind-libunwind.c b/tools/perf/util/unwind-libunwind.c
index 6edf535..e3c40a5 100644
--- a/tools/perf/util/unwind-libunwind.c
+++ b/tools/perf/util/unwind-libunwind.c
@@ -266,14 +266,17 @@ static int read_unwind_spec_eh_frame(struct dso *dso, struct machine *machine,
 				     u64 *fde_count)
 {
 	int ret = -EINVAL, fd;
-	u64 offset;
+	u64 offset = dso->data.frame_offset;
 
-	fd = dso__data_fd(dso, machine);
-	if (fd < 0)
-		return -EINVAL;
+	if (offset == 0) {
+		fd = dso__data_fd(dso, machine);
+		if (fd < 0)
+			return -EINVAL;
 
-	/* Check the .eh_frame section for unwinding info */
-	offset = elf_section_offset(fd, ".eh_frame_hdr");
+		/* Check the .eh_frame section for unwinding info */
+		offset = elf_section_offset(fd, ".eh_frame_hdr");
+		dso->data.frame_offset = offset;
+	}
 
 	if (offset)
 		ret = unwind_spec_ehframe(dso, machine, offset,
@@ -287,14 +290,20 @@ static int read_unwind_spec_eh_frame(struct dso *dso, struct machine *machine,
 static int read_unwind_spec_debug_frame(struct dso *dso,
 					struct machine *machine, u64 *offset)
 {
-	int fd = dso__data_fd(dso, machine);
+	int fd;
+	u64 ofs = dso->data.frame_offset;
 
-	if (fd < 0)
-		return -EINVAL;
+	if (ofs == 0) {
+		fd = dso__data_fd(dso, machine);
+		if (fd < 0)
+			return -EINVAL;
 
-	/* Check the .debug_frame section for unwinding info */
-	*offset = elf_section_offset(fd, ".debug_frame");
+		/* Check the .debug_frame section for unwinding info */
+		ofs = elf_section_offset(fd, ".debug_frame");
+		dso->data.frame_offset = ofs;
+	}
 
+	*offset = ofs;
 	if (*offset)
 		return 0;
 

^ permalink raw reply related	[flat|nested] 221+ messages in thread

* [tip:perf/core] perf tools: Do not use __perf_session__process_events() directly
  2015-01-29  8:06 ` [PATCH 02/42] perf tools: Do not use __perf_session__process_events() directly Namhyung Kim
@ 2015-01-30 18:32   ` tip-bot for Namhyung Kim
  0 siblings, 0 replies; 221+ messages in thread
From: tip-bot for Namhyung Kim @ 2015-01-30 18:32 UTC (permalink / raw)
  To: linux-tip-commits
  Cc: linux-kernel, namhyung, eranian, adrian.hunter, mingo, hpa, andi,
	dsahern, a.p.zijlstra, acme, fweisbec, tglx, jolsa

Commit-ID:  4ac30cf74b308fb01338e660d3471cd490a7958a
Gitweb:     http://git.kernel.org/tip/4ac30cf74b308fb01338e660d3471cd490a7958a
Author:     Namhyung Kim <namhyung@kernel.org>
AuthorDate: Thu, 29 Jan 2015 17:06:43 +0900
Committer:  Arnaldo Carvalho de Melo <acme@redhat.com>
CommitDate: Thu, 29 Jan 2015 16:36:32 -0300

perf tools: Do not use __perf_session__process_events() directly

It's only used for perf record to process build-id because its file size
it's not fixed at this time due to remaining header features.

However data offset and size is available so that we can use the
perf_session__process_events() once we set the file size as the current
offset like for now.

Signed-off-by: Namhyung Kim <namhyung@kernel.org>
Cc: Adrian Hunter <adrian.hunter@intel.com>
Cc: Andi Kleen <andi@firstfloor.org>
Cc: David Ahern <dsahern@gmail.com>
Cc: Frederic Weisbecker <fweisbec@gmail.com>
Cc: Jiri Olsa <jolsa@redhat.com>
Cc: Peter Zijlstra <a.p.zijlstra@chello.nl>
Cc: Stephane Eranian <eranian@google.com>
Link: http://lkml.kernel.org/r/1422518843-25818-3-git-send-email-namhyung@kernel.org
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
---
 tools/perf/builtin-record.c | 7 +++----
 tools/perf/util/session.c   | 6 +++---
 tools/perf/util/session.h   | 3 ---
 3 files changed, 6 insertions(+), 10 deletions(-)

diff --git a/tools/perf/builtin-record.c b/tools/perf/builtin-record.c
index 8648c6d..1134de2 100644
--- a/tools/perf/builtin-record.c
+++ b/tools/perf/builtin-record.c
@@ -194,12 +194,13 @@ static int process_buildids(struct record *rec)
 {
 	struct perf_data_file *file  = &rec->file;
 	struct perf_session *session = rec->session;
-	u64 start = session->header.data_offset;
 
 	u64 size = lseek(file->fd, 0, SEEK_CUR);
 	if (size == 0)
 		return 0;
 
+	file->size = size;
+
 	/*
 	 * During this process, it'll load kernel map and replace the
 	 * dso->long_name to a real pathname it found.  In this case
@@ -211,9 +212,7 @@ static int process_buildids(struct record *rec)
 	 */
 	symbol_conf.ignore_vmlinux_buildid = true;
 
-	return __perf_session__process_events(session, start,
-					      size - start,
-					      size, &build_id__mark_dso_hit_ops);
+	return perf_session__process_events(session, &build_id__mark_dso_hit_ops);
 }
 
 static void perf_event__synthesize_guest_os(struct machine *machine, void *data)
diff --git a/tools/perf/util/session.c b/tools/perf/util/session.c
index b0ce3d6..0baf75f 100644
--- a/tools/perf/util/session.c
+++ b/tools/perf/util/session.c
@@ -1251,9 +1251,9 @@ fetch_mmaped_event(struct perf_session *session,
 #define NUM_MMAPS 128
 #endif
 
-int __perf_session__process_events(struct perf_session *session,
-				   u64 data_offset, u64 data_size,
-				   u64 file_size, struct perf_tool *tool)
+static int __perf_session__process_events(struct perf_session *session,
+					  u64 data_offset, u64 data_size,
+					  u64 file_size, struct perf_tool *tool)
 {
 	int fd = perf_data_file__fd(session->file);
 	u64 head, page_offset, file_offset, file_pos, size;
diff --git a/tools/perf/util/session.h b/tools/perf/util/session.h
index dc26ebf..6d663dc 100644
--- a/tools/perf/util/session.h
+++ b/tools/perf/util/session.h
@@ -49,9 +49,6 @@ int perf_session__peek_event(struct perf_session *session, off_t file_offset,
 			     union perf_event **event_ptr,
 			     struct perf_sample *sample);
 
-int __perf_session__process_events(struct perf_session *session,
-				   u64 data_offset, u64 data_size, u64 size,
-				   struct perf_tool *tool);
 int perf_session__process_events(struct perf_session *session,
 				 struct perf_tool *tool);
 

^ permalink raw reply related	[flat|nested] 221+ messages in thread

* [tip:perf/core] perf record: Show precise number of samples
  2015-01-29  8:06 ` [PATCH 03/42] perf record: Show precise number of samples Namhyung Kim
@ 2015-01-30 18:32   ` tip-bot for Namhyung Kim
  0 siblings, 0 replies; 221+ messages in thread
From: tip-bot for Namhyung Kim @ 2015-01-30 18:32 UTC (permalink / raw)
  To: linux-tip-commits
  Cc: acme, andi, mail, eranian, adrian.hunter, namhyung, mingo,
	a.p.zijlstra, fweisbec, jolsa, tglx, dsahern, hpa, linux-kernel

Commit-ID:  e3d5911221f5cf71e1f0306256d4e42d34a365d2
Gitweb:     http://git.kernel.org/tip/e3d5911221f5cf71e1f0306256d4e42d34a365d2
Author:     Namhyung Kim <namhyung@kernel.org>
AuthorDate: Thu, 29 Jan 2015 17:06:44 +0900
Committer:  Arnaldo Carvalho de Melo <acme@redhat.com>
CommitDate: Thu, 29 Jan 2015 16:37:20 -0300

perf record: Show precise number of samples

After perf record finishes, it prints file size and number of samples in
the file but this info is wrong since it assumes typical sample size of
24 bytes and divides file size by the value.

However as we post-process recorded samples for build-id, it can show
correct number like below.  If build-id post-processing is not requested
just omit the wrong number of samples.

  $ perf record noploop 1
    [ perf record: Woken up 1 times to write data ]
    [ perf record: Captured and wrote 0.159 MB perf.data (3989 samples) ]

  $ perf report --stdio -n
  # To display the perf.data header info, please use --header/--header-only options.
  #
  # Samples: 3K of event 'cycles'
  # Event count (approx.): 3771330663
  #
  # Overhead       Samples  Command  Shared Object     Symbol
  # ........  ............  .......  ................  ..........................
  #
      99.90%          3982  noploop  noploop           [.] main
       0.09%             1  noploop  ld-2.17.so        [.] _dl_check_map_versions
       0.01%             1  noploop  [kernel.vmlinux]  [k] setup_arg_pages
       0.00%             5  noploop  [kernel.vmlinux]  [k] intel_pmu_enable_all

Reported-by: Milian Wolff <mail@milianw.de>
Signed-off-by: Namhyung Kim <namhyung@kernel.org>
Reviewed-by: Jiri Olsa <jolsa@kernel.org>
Cc: Adrian Hunter <adrian.hunter@intel.com>
Cc: Andi Kleen <andi@firstfloor.org>
Cc: David Ahern <dsahern@gmail.com>
Cc: Frederic Weisbecker <fweisbec@gmail.com>
Cc: Peter Zijlstra <a.p.zijlstra@chello.nl>
Cc: Stephane Eranian <eranian@google.com>
Link: http://lkml.kernel.org/r/1422518843-25818-4-git-send-email-namhyung@kernel.org
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
---
 tools/perf/builtin-record.c | 51 ++++++++++++++++++++++++++++++++++-----------
 1 file changed, 39 insertions(+), 12 deletions(-)

diff --git a/tools/perf/builtin-record.c b/tools/perf/builtin-record.c
index 1134de2..9900b43 100644
--- a/tools/perf/builtin-record.c
+++ b/tools/perf/builtin-record.c
@@ -190,6 +190,19 @@ out:
 	return rc;
 }
 
+static int process_sample_event(struct perf_tool *tool,
+				union perf_event *event,
+				struct perf_sample *sample,
+				struct perf_evsel *evsel,
+				struct machine *machine)
+{
+	struct record *rec = container_of(tool, struct record, tool);
+
+	rec->samples++;
+
+	return build_id__mark_dso_hit(tool, event, sample, evsel, machine);
+}
+
 static int process_buildids(struct record *rec)
 {
 	struct perf_data_file *file  = &rec->file;
@@ -212,7 +225,7 @@ static int process_buildids(struct record *rec)
 	 */
 	symbol_conf.ignore_vmlinux_buildid = true;
 
-	return perf_session__process_events(session, &build_id__mark_dso_hit_ops);
+	return perf_session__process_events(session, &rec->tool);
 }
 
 static void perf_event__synthesize_guest_os(struct machine *machine, void *data)
@@ -503,19 +516,9 @@ static int __cmd_record(struct record *rec, int argc, const char **argv)
 		goto out_child;
 	}
 
-	if (!quiet) {
+	if (!quiet)
 		fprintf(stderr, "[ perf record: Woken up %ld times to write data ]\n", waking);
 
-		/*
-		 * Approximate RIP event size: 24 bytes.
-		 */
-		fprintf(stderr,
-			"[ perf record: Captured and wrote %.3f MB %s (~%" PRIu64 " samples) ]\n",
-			(double)rec->bytes_written / 1024.0 / 1024.0,
-			file->path,
-			rec->bytes_written / 24);
-	}
-
 out_child:
 	if (forks) {
 		int exit_status;
@@ -534,6 +537,9 @@ out_child:
 	} else
 		status = err;
 
+	/* this will be recalculated during process_buildids() */
+	rec->samples = 0;
+
 	if (!err && !file->is_pipe) {
 		rec->session->header.data_size += rec->bytes_written;
 
@@ -543,6 +549,20 @@ out_child:
 					   file->fd, true);
 	}
 
+	if (!err && !quiet) {
+		char samples[128];
+
+		if (rec->samples)
+			scnprintf(samples, sizeof(samples),
+				  " (%" PRIu64 " samples)", rec->samples);
+		else
+			samples[0] = '\0';
+
+		fprintf(stderr,	"[ perf record: Captured and wrote %.3f MB %s%s ]\n",
+			perf_data_file__size(file) / 1024.0 / 1024.0,
+			file->path, samples);
+	}
+
 out_delete_session:
 	perf_session__delete(session);
 	return status;
@@ -719,6 +739,13 @@ static struct record record = {
 			.default_per_cpu = true,
 		},
 	},
+	.tool = {
+		.sample		= process_sample_event,
+		.fork		= perf_event__process_fork,
+		.comm		= perf_event__process_comm,
+		.mmap		= perf_event__process_mmap,
+		.mmap2		= perf_event__process_mmap2,
+	},
 };
 
 #define CALLCHAIN_HELP "setup and enables call-graph (stack chain/backtrace) recording: "

^ permalink raw reply related	[flat|nested] 221+ messages in thread

* [tip:perf/core] perf header: Set header version correctly
  2015-01-29  8:06 ` [PATCH 04/42] perf header: Set header version correctly Namhyung Kim
@ 2015-01-30 18:33   ` tip-bot for Namhyung Kim
  0 siblings, 0 replies; 221+ messages in thread
From: tip-bot for Namhyung Kim @ 2015-01-30 18:33 UTC (permalink / raw)
  To: linux-tip-commits
  Cc: dsahern, linux-kernel, namhyung, a.p.zijlstra, acme, mingo,
	eranian, fweisbec, andi, adrian.hunter, jolsa, hpa, tglx

Commit-ID:  f7913971bdad1a72c6158074786babed477d61e2
Gitweb:     http://git.kernel.org/tip/f7913971bdad1a72c6158074786babed477d61e2
Author:     Namhyung Kim <namhyung@kernel.org>
AuthorDate: Thu, 29 Jan 2015 17:06:45 +0900
Committer:  Arnaldo Carvalho de Melo <acme@redhat.com>
CommitDate: Thu, 29 Jan 2015 16:53:11 -0300

perf header: Set header version correctly

When check_magic_endian() is called, it checks the magic number in the
perf data file to determine version and endianness.  But if it uses a
same endian the verison number wasn't updated and makes confusion.

Signed-off-by: Namhyung Kim <namhyung@kernel.org>
Cc: Adrian Hunter <adrian.hunter@intel.com>
Cc: Andi Kleen <andi@firstfloor.org>
Cc: David Ahern <dsahern@gmail.com>
Cc: Frederic Weisbecker <fweisbec@gmail.com>
Cc: Jiri Olsa <jolsa@redhat.com>
Cc: Peter Zijlstra <a.p.zijlstra@chello.nl>
Cc: Stephane Eranian <eranian@google.com>
Link: http://lkml.kernel.org/r/1422518843-25818-5-git-send-email-namhyung@kernel.org
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
---
 tools/perf/util/header.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/tools/perf/util/header.c b/tools/perf/util/header.c
index b20e40c..1f407f7 100644
--- a/tools/perf/util/header.c
+++ b/tools/perf/util/header.c
@@ -2237,6 +2237,7 @@ static int check_magic_endian(u64 magic, uint64_t hdr_sz,
 	 * - unique number to identify actual perf.data files
 	 * - encode endianness of file
 	 */
+	ph->version = PERF_HEADER_VERSION_2;
 
 	/* check magic number with one endianness */
 	if (magic == __perf_magic2)
@@ -2247,7 +2248,6 @@ static int check_magic_endian(u64 magic, uint64_t hdr_sz,
 		return -1;
 
 	ph->needs_swap = true;
-	ph->version = PERF_HEADER_VERSION_2;
 
 	return 0;
 }

^ permalink raw reply related	[flat|nested] 221+ messages in thread

* [tip:perf/core] perf evsel: Set attr.task bit for a tracking event
  2015-01-29  8:06 ` [PATCH 05/42] perf tools: Set attr.task bit for a tracking event Namhyung Kim
@ 2015-01-30 18:33   ` tip-bot for Namhyung Kim
  0 siblings, 0 replies; 221+ messages in thread
From: tip-bot for Namhyung Kim @ 2015-01-30 18:33 UTC (permalink / raw)
  To: linux-tip-commits
  Cc: adrian.hunter, hpa, tglx, acme, namhyung, dsahern, andi, jolsa,
	fweisbec, mingo, a.p.zijlstra, eranian, linux-kernel

Commit-ID:  62e503b7ed98fcdf16308cda0b5378e7840f4339
Gitweb:     http://git.kernel.org/tip/62e503b7ed98fcdf16308cda0b5378e7840f4339
Author:     Namhyung Kim <namhyung@kernel.org>
AuthorDate: Thu, 29 Jan 2015 17:06:46 +0900
Committer:  Arnaldo Carvalho de Melo <acme@redhat.com>
CommitDate: Thu, 29 Jan 2015 16:54:59 -0300

perf evsel: Set attr.task bit for a tracking event

The perf_event_attr.task bit is to track task (fork and exit) events but
it missed to be set by perf_evsel__config().  While it was not a problem
in practice since setting other bits (comm/mmap) ended up being in same
result, it'd be good to set it explicitly anyway.

The attr->task is to track task related events (fork/exit) only but
other meta events like comm and mmap[2] also needs the task events.  So
setting attr->comm and/or attr->mmap causes the kernel emits the task
events anyway.  So the attr->task is only meaningful when other bits are
off but I'd like to set it for completeness.

Signed-off-by: Namhyung Kim <namhyung@kernel.org>
Cc: Adrian Hunter <adrian.hunter@intel.com>
Cc: Andi Kleen <andi@firstfloor.org>
Cc: David Ahern <dsahern@gmail.com>
Cc: Frederic Weisbecker <fweisbec@gmail.com>
Cc: Jiri Olsa <jolsa@redhat.com>
Cc: Peter Zijlstra <a.p.zijlstra@chello.nl>
Cc: Stephane Eranian <eranian@google.com>
Link: http://lkml.kernel.org/r/1422518843-25818-6-git-send-email-namhyung@kernel.org
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
---
 tools/perf/util/evsel.c | 1 +
 1 file changed, 1 insertion(+)

diff --git a/tools/perf/util/evsel.c b/tools/perf/util/evsel.c
index 1d826d6..ea51a90 100644
--- a/tools/perf/util/evsel.c
+++ b/tools/perf/util/evsel.c
@@ -709,6 +709,7 @@ void perf_evsel__config(struct perf_evsel *evsel, struct record_opts *opts)
 	if (opts->sample_weight)
 		perf_evsel__set_sample_bit(evsel, WEIGHT);
 
+	attr->task  = track;
 	attr->mmap  = track;
 	attr->mmap2 = track && !perf_missing_features.mmap2;
 	attr->comm  = track;

^ permalink raw reply related	[flat|nested] 221+ messages in thread

* [tip:perf/core] perf symbols: Support to read compressed module from build-id cache
  2015-01-29  8:06 ` [PATCH 01/42] perf tools: Support to read compressed module from build-id cache Namhyung Kim
  2015-01-30 14:32   ` Jiri Olsa
@ 2015-01-30 18:33   ` tip-bot for Namhyung Kim
  1 sibling, 0 replies; 221+ messages in thread
From: tip-bot for Namhyung Kim @ 2015-01-30 18:33 UTC (permalink / raw)
  To: linux-tip-commits
  Cc: adrian.hunter, namhyung, mingo, dsahern, hpa, eranian, acme,
	fweisbec, jolsa, linux-kernel, tglx, a.p.zijlstra, andi

Commit-ID:  0b064f43001fc2627ff8c3020647b85db040235f
Gitweb:     http://git.kernel.org/tip/0b064f43001fc2627ff8c3020647b85db040235f
Author:     Namhyung Kim <namhyung@kernel.org>
AuthorDate: Thu, 29 Jan 2015 17:06:42 +0900
Committer:  Arnaldo Carvalho de Melo <acme@redhat.com>
CommitDate: Thu, 29 Jan 2015 16:56:54 -0300

perf symbols: Support to read compressed module from build-id cache

The commit c00c48fc6e6e ("perf symbols: Preparation for compressed
kernel module support") added support for compressed kernel modules but
it only supports system path DSOs.  When a dso is read from build-id
cache, its filename doesn't end with ".gz" but has build-id.  In this
case, we should fallback to the original dso->name.

Signed-off-by: Namhyung Kim <namhyung@kernel.org>
Cc: Adrian Hunter <adrian.hunter@intel.com>
Cc: Andi Kleen <andi@firstfloor.org>
Cc: David Ahern <dsahern@gmail.com>
Cc: Frederic Weisbecker <fweisbec@gmail.com>
Cc: Jiri Olsa <jolsa@redhat.com>
Cc: Peter Zijlstra <a.p.zijlstra@chello.nl>
Cc: Stephane Eranian <eranian@google.com>
Link: http://lkml.kernel.org/r/1422518843-25818-2-git-send-email-namhyung@kernel.org
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
---
 tools/perf/util/symbol-elf.c | 13 ++++++++-----
 1 file changed, 8 insertions(+), 5 deletions(-)

diff --git a/tools/perf/util/symbol-elf.c b/tools/perf/util/symbol-elf.c
index 06fcd1b..b24f9d8 100644
--- a/tools/perf/util/symbol-elf.c
+++ b/tools/perf/util/symbol-elf.c
@@ -574,13 +574,16 @@ static int decompress_kmodule(struct dso *dso, const char *name,
 	const char *ext = strrchr(name, '.');
 	char tmpbuf[] = "/tmp/perf-kmod-XXXXXX";
 
-	if ((type != DSO_BINARY_TYPE__SYSTEM_PATH_KMODULE_COMP &&
-	     type != DSO_BINARY_TYPE__GUEST_KMODULE_COMP) ||
-	    type != dso->symtab_type)
+	if (type != DSO_BINARY_TYPE__SYSTEM_PATH_KMODULE_COMP &&
+	    type != DSO_BINARY_TYPE__GUEST_KMODULE_COMP &&
+	    type != DSO_BINARY_TYPE__BUILD_ID_CACHE)
 		return -1;
 
-	if (!ext || !is_supported_compression(ext + 1))
-		return -1;
+	if (!ext || !is_supported_compression(ext + 1)) {
+		ext = strrchr(dso->name, '.');
+		if (!ext || !is_supported_compression(ext + 1))
+			return -1;
+	}
 
 	fd = mkstemp(tmpbuf);
 	if (fd < 0)

^ permalink raw reply related	[flat|nested] 221+ messages in thread

* [tip:perf/core] perf tools: Use perf_data_file__fd() consistently
  2015-01-29  8:06 ` [PATCH 07/42] perf tools: Use perf_data_file__fd() consistently Namhyung Kim
@ 2015-01-30 18:33   ` tip-bot for Namhyung Kim
  0 siblings, 0 replies; 221+ messages in thread
From: tip-bot for Namhyung Kim @ 2015-01-30 18:33 UTC (permalink / raw)
  To: linux-tip-commits
  Cc: hpa, dsahern, acme, linux-kernel, eranian, fweisbec,
	a.p.zijlstra, andi, mingo, adrian.hunter, tglx, namhyung, jolsa

Commit-ID:  42aa276f40730211383e9a9923416f1fb9841d68
Gitweb:     http://git.kernel.org/tip/42aa276f40730211383e9a9923416f1fb9841d68
Author:     Namhyung Kim <namhyung@kernel.org>
AuthorDate: Thu, 29 Jan 2015 17:06:48 +0900
Committer:  Arnaldo Carvalho de Melo <acme@redhat.com>
CommitDate: Thu, 29 Jan 2015 16:58:24 -0300

perf tools: Use perf_data_file__fd() consistently

Do not reference file->fd directly since we want hide the
implementation details from outside for possible future changes.

Signed-off-by: Namhyung Kim <namhyung@kernel.org>
Cc: Adrian Hunter <adrian.hunter@intel.com>
Cc: Andi Kleen <andi@firstfloor.org>
Cc: David Ahern <dsahern@gmail.com>
Cc: Frederic Weisbecker <fweisbec@gmail.com>
Cc: Jiri Olsa <jolsa@redhat.com>
Cc: Peter Zijlstra <a.p.zijlstra@chello.nl>
Cc: Stephane Eranian <eranian@google.com>
Link: http://lkml.kernel.org/r/1422518843-25818-8-git-send-email-namhyung@kernel.org
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
---
 tools/perf/builtin-inject.c |  5 +++--
 tools/perf/builtin-record.c | 14 +++++++-------
 2 files changed, 10 insertions(+), 9 deletions(-)

diff --git a/tools/perf/builtin-inject.c b/tools/perf/builtin-inject.c
index 84df2de..a13641e 100644
--- a/tools/perf/builtin-inject.c
+++ b/tools/perf/builtin-inject.c
@@ -343,6 +343,7 @@ static int __cmd_inject(struct perf_inject *inject)
 	int ret = -EINVAL;
 	struct perf_session *session = inject->session;
 	struct perf_data_file *file_out = &inject->output;
+	int fd = perf_data_file__fd(file_out);
 
 	signal(SIGINT, sig_handler);
 
@@ -376,7 +377,7 @@ static int __cmd_inject(struct perf_inject *inject)
 	}
 
 	if (!file_out->is_pipe)
-		lseek(file_out->fd, session->header.data_offset, SEEK_SET);
+		lseek(fd, session->header.data_offset, SEEK_SET);
 
 	ret = perf_session__process_events(session, &inject->tool);
 
@@ -385,7 +386,7 @@ static int __cmd_inject(struct perf_inject *inject)
 			perf_header__set_feat(&session->header,
 					      HEADER_BUILD_ID);
 		session->header.data_size = inject->bytes_written;
-		perf_session__write_header(session, session->evlist, file_out->fd, true);
+		perf_session__write_header(session, session->evlist, fd, true);
 	}
 
 	return ret;
diff --git a/tools/perf/builtin-record.c b/tools/perf/builtin-record.c
index 9900b43..404ab34 100644
--- a/tools/perf/builtin-record.c
+++ b/tools/perf/builtin-record.c
@@ -208,7 +208,7 @@ static int process_buildids(struct record *rec)
 	struct perf_data_file *file  = &rec->file;
 	struct perf_session *session = rec->session;
 
-	u64 size = lseek(file->fd, 0, SEEK_CUR);
+	u64 size = lseek(perf_data_file__fd(file), 0, SEEK_CUR);
 	if (size == 0)
 		return 0;
 
@@ -334,6 +334,7 @@ static int __cmd_record(struct record *rec, int argc, const char **argv)
 	struct perf_data_file *file = &rec->file;
 	struct perf_session *session;
 	bool disabled = false, draining = false;
+	int fd;
 
 	rec->progname = argv[0];
 
@@ -348,6 +349,7 @@ static int __cmd_record(struct record *rec, int argc, const char **argv)
 		return -1;
 	}
 
+	fd = perf_data_file__fd(file);
 	rec->session = session;
 
 	record__init_features(rec);
@@ -372,12 +374,11 @@ static int __cmd_record(struct record *rec, int argc, const char **argv)
 		perf_header__clear_feat(&session->header, HEADER_GROUP_DESC);
 
 	if (file->is_pipe) {
-		err = perf_header__write_pipe(file->fd);
+		err = perf_header__write_pipe(fd);
 		if (err < 0)
 			goto out_child;
 	} else {
-		err = perf_session__write_header(session, rec->evlist,
-						 file->fd, false);
+		err = perf_session__write_header(session, rec->evlist, fd, false);
 		if (err < 0)
 			goto out_child;
 	}
@@ -409,7 +410,7 @@ static int __cmd_record(struct record *rec, int argc, const char **argv)
 			 * return this more properly and also
 			 * propagate errors that now are calling die()
 			 */
-			err = perf_event__synthesize_tracing_data(tool, file->fd, rec->evlist,
+			err = perf_event__synthesize_tracing_data(tool,	fd, rec->evlist,
 								  process_synthesized_event);
 			if (err <= 0) {
 				pr_err("Couldn't record tracing data.\n");
@@ -545,8 +546,7 @@ out_child:
 
 		if (!rec->no_buildid)
 			process_buildids(rec);
-		perf_session__write_header(rec->session, rec->evlist,
-					   file->fd, true);
+		perf_session__write_header(rec->session, rec->evlist, fd, true);
 	}
 
 	if (!err && !quiet) {

^ permalink raw reply related	[flat|nested] 221+ messages in thread

* [tip:perf/core] perf symbols: Convert lseek + read to pread
  2015-01-29  8:07 ` [PATCH 39/42] perf tools: Convert lseek + read to pread Namhyung Kim
@ 2015-01-30 18:34   ` tip-bot for Namhyung Kim
  0 siblings, 0 replies; 221+ messages in thread
From: tip-bot for Namhyung Kim @ 2015-01-30 18:34 UTC (permalink / raw)
  To: linux-tip-commits
  Cc: a.p.zijlstra, fweisbec, linux-kernel, eranian, mingo, tglx, acme,
	dsahern, jolsa, hpa, andi, adrian.hunter, namhyung

Commit-ID:  c52686f9f888d23ca72f1309e86af8e91d075697
Gitweb:     http://git.kernel.org/tip/c52686f9f888d23ca72f1309e86af8e91d075697
Author:     Namhyung Kim <namhyung@kernel.org>
AuthorDate: Thu, 29 Jan 2015 17:02:01 -0300
Committer:  Arnaldo Carvalho de Melo <acme@redhat.com>
CommitDate: Thu, 29 Jan 2015 17:02:01 -0300

perf symbols: Convert lseek + read to pread

When dso_cache__read() is called, it reads data from the given offset
using lseek + normal read syscall.  It can be combined to a single pread
syscall.

Signed-off-by: Namhyung Kim <namhyung@kernel.org>
Cc: Adrian Hunter <adrian.hunter@intel.com>
Cc: Andi Kleen <andi@firstfloor.org>
Cc: David Ahern <dsahern@gmail.com>
Cc: Frederic Weisbecker <fweisbec@gmail.com>
Cc: Jiri Olsa <jolsa@redhat.com>
Cc: Peter Zijlstra <a.p.zijlstra@chello.nl>
Cc: Stephane Eranian <eranian@google.com>
Link: http://lkml.kernel.org/r/1422518843-25818-40-git-send-email-namhyung@kernel.org
[ Fixed it up when cherry picking it from the multi threaded patchkit ]
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
---
 tools/perf/util/dso.c | 6 +-----
 1 file changed, 1 insertion(+), 5 deletions(-)

diff --git a/tools/perf/util/dso.c b/tools/perf/util/dso.c
index 45be944..c2f7d3b 100644
--- a/tools/perf/util/dso.c
+++ b/tools/perf/util/dso.c
@@ -532,12 +532,8 @@ dso_cache__read(struct dso *dso, u64 offset, u8 *data, ssize_t size)
 			break;
 
 		cache_offset = offset & DSO__DATA_CACHE_MASK;
-		ret = -EINVAL;
 
-		if (-1 == lseek(dso->data.fd, cache_offset, SEEK_SET))
-			break;
-
-		ret = read(dso->data.fd, cache->data, DSO__DATA_CACHE_SIZE);
+		ret = pread(dso->data.fd, cache->data, DSO__DATA_CACHE_SIZE, cache_offset);
 		if (ret <= 0)
 			break;
 

^ permalink raw reply related	[flat|nested] 221+ messages in thread

* Re: [PATCH 14/42] perf record: Add --index option for building index table
  2015-01-29  8:06 ` [PATCH 14/42] perf record: Add --index option for building index table Namhyung Kim
@ 2015-02-01 18:06   ` Jiri Olsa
  2015-02-02  8:34     ` Adrian Hunter
  0 siblings, 1 reply; 221+ messages in thread
From: Jiri Olsa @ 2015-02-01 18:06 UTC (permalink / raw)
  To: Namhyung Kim
  Cc: Arnaldo Carvalho de Melo, Ingo Molnar, Peter Zijlstra, LKML,
	David Ahern, Adrian Hunter, Andi Kleen, Stephane Eranian,
	Frederic Weisbecker

On Thu, Jan 29, 2015 at 05:06:55PM +0900, Namhyung Kim wrote:
> The new --index option will create indexed data file which can be
> processed by multiple threads parallelly.  It saves meta event and
> sample data in separate files and merges them with an index table.
> 
> To build an index table, it needs to know exact offsets and sizes for
> each sample data.  However the offset only can be calculated after the
> feature data is fixed, and to save feature data it needs to access to
> the sample data because it needs to mark used DSOs for build-id table.
> 
> So I ended up with reserving 1MB hole for the feature data area and then
> put sample data and calculated offsets.  Now an indexed perf data file
> will look like below:
> 
>         +---------------------+
>         |     file header     |
>         |---------------------|
>         |                     |
>         |     meta events     |
>         |                     |
>         |---------------------|
>         |     feature data    |
>         |   (contains index) -+--+
>         |---------------------|  |
>         |      ~1MB hole      |  |
>         |---------------------|  |
>         |                     |  |
>         |    sample data[1] <-+--+
>         |                     |  |
>         |---------------------|  |
>         |                     |  |
>         |    sample data[2] <-|--+
>         |                     |  |
>         |---------------------|  |
>         |         ...         | ...
>         +---------------------+

I also dont see how to store it in a nice way under current header layout,
but how about bump up the header version for this feature? ;-)

currently it's:

struct perf_file_header {
        u64                             magic;
        u64                             size;
        u64                             attr_size;
        struct perf_file_section        attrs;
        struct perf_file_section        data;
        /* event_types is ignored */
        struct perf_file_section        event_types;
        DECLARE_BITMAP(adds_features, HEADER_FEAT_BITS);
};


- we already store attrs as a FEATURE so we could omit that
- your patch stores only synthesized data into 'data' section (-1 idx)
  this could be stored into separate file and get merged with the rest
- new header version would have 'features' section, so the features
  position wouldnt depend on the 'data' end as of now and we could
  easily store after all data is merged:

struct perf_file_header {
        u64                             magic;
        u64                             size;
        u64                             attr_size;
        struct perf_file_section        features;
        DECLARE_BITMAP(adds_features, HEADER_FEAT_BITS);
};


thoughts?
jirka

^ permalink raw reply	[flat|nested] 221+ messages in thread

* Re: [PATCH 14/42] perf record: Add --index option for building index table
  2015-02-01 18:06   ` Jiri Olsa
@ 2015-02-02  8:34     ` Adrian Hunter
  2015-02-02  9:15       ` Jiri Olsa
  0 siblings, 1 reply; 221+ messages in thread
From: Adrian Hunter @ 2015-02-02  8:34 UTC (permalink / raw)
  To: Jiri Olsa, Namhyung Kim
  Cc: Arnaldo Carvalho de Melo, Ingo Molnar, Peter Zijlstra, LKML,
	David Ahern, Andi Kleen, Stephane Eranian, Frederic Weisbecker

On 01/02/15 20:06, Jiri Olsa wrote:
> On Thu, Jan 29, 2015 at 05:06:55PM +0900, Namhyung Kim wrote:
>> The new --index option will create indexed data file which can be
>> processed by multiple threads parallelly.  It saves meta event and
>> sample data in separate files and merges them with an index table.
>>
>> To build an index table, it needs to know exact offsets and sizes for
>> each sample data.  However the offset only can be calculated after the
>> feature data is fixed, and to save feature data it needs to access to
>> the sample data because it needs to mark used DSOs for build-id table.
>>
>> So I ended up with reserving 1MB hole for the feature data area and then
>> put sample data and calculated offsets.  Now an indexed perf data file
>> will look like below:
>>
>>         +---------------------+
>>         |     file header     |
>>         |---------------------|
>>         |                     |
>>         |     meta events     |
>>         |                     |
>>         |---------------------|
>>         |     feature data    |
>>         |   (contains index) -+--+
>>         |---------------------|  |
>>         |      ~1MB hole      |  |
>>         |---------------------|  |
>>         |                     |  |
>>         |    sample data[1] <-+--+
>>         |                     |  |
>>         |---------------------|  |
>>         |                     |  |
>>         |    sample data[2] <-|--+
>>         |                     |  |
>>         |---------------------|  |
>>         |         ...         | ...
>>         +---------------------+
> 
> I also dont see how to store it in a nice way under current header layout,
> but how about bump up the header version for this feature? ;-)
> 
> currently it's:
> 
> struct perf_file_header {
>         u64                             magic;
>         u64                             size;
>         u64                             attr_size;
>         struct perf_file_section        attrs;
>         struct perf_file_section        data;
>         /* event_types is ignored */
>         struct perf_file_section        event_types;
>         DECLARE_BITMAP(adds_features, HEADER_FEAT_BITS);
> };
> 
> 
> - we already store attrs as a FEATURE so we could omit that
> - your patch stores only synthesized data into 'data' section (-1 idx)
>   this could be stored into separate file and get merged with the rest
> - new header version would have 'features' section, so the features
>   position wouldnt depend on the 'data' end as of now and we could
>   easily store after all data is merged:
> 
> struct perf_file_header {
>         u64                             magic;
>         u64                             size;
>         u64                             attr_size;
>         struct perf_file_section        features;
>         DECLARE_BITMAP(adds_features, HEADER_FEAT_BITS);
> };
> 
> 
> thoughts?

How come the features are being written before the sample data anyway?
I would have expected:
	- write the data (update the index in memory)
	- write the features (including index)


^ permalink raw reply	[flat|nested] 221+ messages in thread

* Re: [PATCH 14/42] perf record: Add --index option for building index table
  2015-02-02  8:34     ` Adrian Hunter
@ 2015-02-02  9:15       ` Jiri Olsa
  2015-02-02  9:52         ` Adrian Hunter
  0 siblings, 1 reply; 221+ messages in thread
From: Jiri Olsa @ 2015-02-02  9:15 UTC (permalink / raw)
  To: Adrian Hunter
  Cc: Namhyung Kim, Arnaldo Carvalho de Melo, Ingo Molnar,
	Peter Zijlstra, LKML, David Ahern, Andi Kleen, Stephane Eranian,
	Frederic Weisbecker

On Mon, Feb 02, 2015 at 10:34:50AM +0200, Adrian Hunter wrote:

SNIP

> > but how about bump up the header version for this feature? ;-)
> > 
> > currently it's:
> > 
> > struct perf_file_header {
> >         u64                             magic;
> >         u64                             size;
> >         u64                             attr_size;
> >         struct perf_file_section        attrs;
> >         struct perf_file_section        data;
> >         /* event_types is ignored */
> >         struct perf_file_section        event_types;
> >         DECLARE_BITMAP(adds_features, HEADER_FEAT_BITS);
> > };
> > 
> > 
> > - we already store attrs as a FEATURE so we could omit that
> > - your patch stores only synthesized data into 'data' section (-1 idx)
> >   this could be stored into separate file and get merged with the rest
> > - new header version would have 'features' section, so the features
> >   position wouldnt depend on the 'data' end as of now and we could
> >   easily store after all data is merged:
> > 
> > struct perf_file_header {
> >         u64                             magic;
> >         u64                             size;
> >         u64                             attr_size;
> >         struct perf_file_section        features;
> >         DECLARE_BITMAP(adds_features, HEADER_FEAT_BITS);
> > };
> > 
> > 
> > thoughts?
> 
> How come the features are being written before the sample data anyway?
> I would have expected:
> 	- write the data (update the index in memory)
> 	- write the features (including index)
>

I think the problem is that the only way how to get features offset
right now is via perf_file_header::data.offset + perf_file_headerdata.size,
and we still use this section to carry 'sythesized' data, so it needs
to have correct size.

I guess we could workaround that by storing the 'perf_file_header::data'
as the last data section. That would require to treat it the same way as
all other data sections, but we could keep current header layout.

jirka

^ permalink raw reply	[flat|nested] 221+ messages in thread

* Re: [PATCH 14/42] perf record: Add --index option for building index table
  2015-02-02  9:15       ` Jiri Olsa
@ 2015-02-02  9:52         ` Adrian Hunter
  2015-02-02 10:05           ` Jiri Olsa
  0 siblings, 1 reply; 221+ messages in thread
From: Adrian Hunter @ 2015-02-02  9:52 UTC (permalink / raw)
  To: Jiri Olsa
  Cc: Namhyung Kim, Arnaldo Carvalho de Melo, Ingo Molnar,
	Peter Zijlstra, LKML, David Ahern, Andi Kleen, Stephane Eranian,
	Frederic Weisbecker

On 02/02/15 11:15, Jiri Olsa wrote:
> On Mon, Feb 02, 2015 at 10:34:50AM +0200, Adrian Hunter wrote:
> 
> SNIP
> 
>>> but how about bump up the header version for this feature? ;-)
>>>
>>> currently it's:
>>>
>>> struct perf_file_header {
>>>         u64                             magic;
>>>         u64                             size;
>>>         u64                             attr_size;
>>>         struct perf_file_section        attrs;
>>>         struct perf_file_section        data;
>>>         /* event_types is ignored */
>>>         struct perf_file_section        event_types;
>>>         DECLARE_BITMAP(adds_features, HEADER_FEAT_BITS);
>>> };
>>>
>>>
>>> - we already store attrs as a FEATURE so we could omit that
>>> - your patch stores only synthesized data into 'data' section (-1 idx)
>>>   this could be stored into separate file and get merged with the rest
>>> - new header version would have 'features' section, so the features
>>>   position wouldnt depend on the 'data' end as of now and we could
>>>   easily store after all data is merged:
>>>
>>> struct perf_file_header {
>>>         u64                             magic;
>>>         u64                             size;
>>>         u64                             attr_size;
>>>         struct perf_file_section        features;
>>>         DECLARE_BITMAP(adds_features, HEADER_FEAT_BITS);
>>> };
>>>
>>>
>>> thoughts?
>>
>> How come the features are being written before the sample data anyway?
>> I would have expected:
>> 	- write the data (update the index in memory)
>> 	- write the features (including index)
>>
> 
> I think the problem is that the only way how to get features offset
> right now is via perf_file_header::data.offset + perf_file_headerdata.size,
> and we still use this section to carry 'sythesized' data, so it needs
> to have correct size.

Why not make it the same as all the other data. i.e. find the start and size
via the index? And then just lump all the data together?

> I guess we could workaround that by storing the 'perf_file_header::data'
> as the last data section. That would require to treat it the same way as
> all other data sections, but we could keep current header layout.

Would it need to be last? Logically it should precede the data that depends
on it.


^ permalink raw reply	[flat|nested] 221+ messages in thread

* Re: [PATCH 14/42] perf record: Add --index option for building index table
  2015-02-02  9:52         ` Adrian Hunter
@ 2015-02-02 10:05           ` Jiri Olsa
  2015-02-02 12:07             ` Adrian Hunter
  0 siblings, 1 reply; 221+ messages in thread
From: Jiri Olsa @ 2015-02-02 10:05 UTC (permalink / raw)
  To: Adrian Hunter
  Cc: Namhyung Kim, Arnaldo Carvalho de Melo, Ingo Molnar,
	Peter Zijlstra, LKML, David Ahern, Andi Kleen, Stephane Eranian,
	Frederic Weisbecker

On Mon, Feb 02, 2015 at 11:52:26AM +0200, Adrian Hunter wrote:
> On 02/02/15 11:15, Jiri Olsa wrote:
> > On Mon, Feb 02, 2015 at 10:34:50AM +0200, Adrian Hunter wrote:
> > 
> > SNIP
> > 
> >>> but how about bump up the header version for this feature? ;-)
> >>>
> >>> currently it's:
> >>>
> >>> struct perf_file_header {
> >>>         u64                             magic;
> >>>         u64                             size;
> >>>         u64                             attr_size;
> >>>         struct perf_file_section        attrs;
> >>>         struct perf_file_section        data;
> >>>         /* event_types is ignored */
> >>>         struct perf_file_section        event_types;
> >>>         DECLARE_BITMAP(adds_features, HEADER_FEAT_BITS);
> >>> };
> >>>
> >>>
> >>> - we already store attrs as a FEATURE so we could omit that
> >>> - your patch stores only synthesized data into 'data' section (-1 idx)
> >>>   this could be stored into separate file and get merged with the rest
> >>> - new header version would have 'features' section, so the features
> >>>   position wouldnt depend on the 'data' end as of now and we could
> >>>   easily store after all data is merged:
> >>>
> >>> struct perf_file_header {
> >>>         u64                             magic;
> >>>         u64                             size;
> >>>         u64                             attr_size;
> >>>         struct perf_file_section        features;
> >>>         DECLARE_BITMAP(adds_features, HEADER_FEAT_BITS);
> >>> };
> >>>
> >>>
> >>> thoughts?
> >>
> >> How come the features are being written before the sample data anyway?
> >> I would have expected:
> >> 	- write the data (update the index in memory)
> >> 	- write the features (including index)
> >>
> > 
> > I think the problem is that the only way how to get features offset
> > right now is via perf_file_header::data.offset + perf_file_headerdata.size,
> > and we still use this section to carry 'sythesized' data, so it needs
> > to have correct size.
> 
> Why not make it the same as all the other data. i.e. find the start and size
> via the index? And then just lump all the data together?

thats what I suggested

> 
> > I guess we could workaround that by storing the 'perf_file_header::data'
> > as the last data section. That would require to treat it the same way as
> > all other data sections, but we could keep current header layout.
> 
> Would it need to be last? Logically it should precede the data that depends
> on it.

i suggested this as a workaround for having features at the end of the file
while keeping the current perf data header

jirka

^ permalink raw reply	[flat|nested] 221+ messages in thread

* Re: [PATCH 14/42] perf record: Add --index option for building index table
  2015-02-02 10:05           ` Jiri Olsa
@ 2015-02-02 12:07             ` Adrian Hunter
  2015-02-02 12:13               ` Jiri Olsa
  0 siblings, 1 reply; 221+ messages in thread
From: Adrian Hunter @ 2015-02-02 12:07 UTC (permalink / raw)
  To: Jiri Olsa
  Cc: Namhyung Kim, Arnaldo Carvalho de Melo, Ingo Molnar,
	Peter Zijlstra, LKML, David Ahern, Andi Kleen, Stephane Eranian,
	Frederic Weisbecker

On 02/02/15 12:05, Jiri Olsa wrote:
> On Mon, Feb 02, 2015 at 11:52:26AM +0200, Adrian Hunter wrote:
>> On 02/02/15 11:15, Jiri Olsa wrote:
>>> On Mon, Feb 02, 2015 at 10:34:50AM +0200, Adrian Hunter wrote:
>>>
>>> SNIP
>>>
>>>>> but how about bump up the header version for this feature? ;-)
>>>>>
>>>>> currently it's:
>>>>>
>>>>> struct perf_file_header {
>>>>>         u64                             magic;
>>>>>         u64                             size;
>>>>>         u64                             attr_size;
>>>>>         struct perf_file_section        attrs;
>>>>>         struct perf_file_section        data;
>>>>>         /* event_types is ignored */
>>>>>         struct perf_file_section        event_types;
>>>>>         DECLARE_BITMAP(adds_features, HEADER_FEAT_BITS);
>>>>> };
>>>>>
>>>>>
>>>>> - we already store attrs as a FEATURE so we could omit that
>>>>> - your patch stores only synthesized data into 'data' section (-1 idx)
>>>>>   this could be stored into separate file and get merged with the rest
>>>>> - new header version would have 'features' section, so the features
>>>>>   position wouldnt depend on the 'data' end as of now and we could
>>>>>   easily store after all data is merged:
>>>>>
>>>>> struct perf_file_header {
>>>>>         u64                             magic;
>>>>>         u64                             size;
>>>>>         u64                             attr_size;
>>>>>         struct perf_file_section        features;
>>>>>         DECLARE_BITMAP(adds_features, HEADER_FEAT_BITS);
>>>>> };
>>>>>
>>>>>
>>>>> thoughts?
>>>>
>>>> How come the features are being written before the sample data anyway?
>>>> I would have expected:
>>>> 	- write the data (update the index in memory)
>>>> 	- write the features (including index)
>>>>
>>>
>>> I think the problem is that the only way how to get features offset
>>> right now is via perf_file_header::data.offset + perf_file_headerdata.size,
>>> and we still use this section to carry 'sythesized' data, so it needs
>>> to have correct size.
>>
>> Why not make it the same as all the other data. i.e. find the start and size
>> via the index? And then just lump all the data together?
> 
> thats what I suggested

No, I meant really lump it all together. i.e. perf_file_header.data.size =
total data size

> 
>>
>>> I guess we could workaround that by storing the 'perf_file_header::data'
>>> as the last data section. That would require to treat it the same way as
>>> all other data sections, but we could keep current header layout.
>>
>> Would it need to be last? Logically it should precede the data that depends
>> on it.
> 
> i suggested this as a workaround for having features at the end of the file
> while keeping the current perf data header

Which wouldn't be necessary if you lump it all together?


^ permalink raw reply	[flat|nested] 221+ messages in thread

* Re: [PATCH 14/42] perf record: Add --index option for building index table
  2015-02-02 12:07             ` Adrian Hunter
@ 2015-02-02 12:13               ` Jiri Olsa
  2015-02-02 14:56                 ` Namhyung Kim
  0 siblings, 1 reply; 221+ messages in thread
From: Jiri Olsa @ 2015-02-02 12:13 UTC (permalink / raw)
  To: Adrian Hunter
  Cc: Namhyung Kim, Arnaldo Carvalho de Melo, Ingo Molnar,
	Peter Zijlstra, LKML, David Ahern, Andi Kleen, Stephane Eranian,
	Frederic Weisbecker

On Mon, Feb 02, 2015 at 02:07:27PM +0200, Adrian Hunter wrote:

SNIP

> >>
> >> Why not make it the same as all the other data. i.e. find the start and size
> >> via the index? And then just lump all the data together?
> > 
> > thats what I suggested
> 
> No, I meant really lump it all together. i.e. perf_file_header.data.size =
> total data size
> 
> > 
> >>
> >>> I guess we could workaround that by storing the 'perf_file_header::data'
> >>> as the last data section. That would require to treat it the same way as
> >>> all other data sections, but we could keep current header layout.
> >>
> >> Would it need to be last? Logically it should precede the data that depends
> >> on it.
> > 
> > i suggested this as a workaround for having features at the end of the file
> > while keeping the current perf data header
> 
> Which wouldn't be necessary if you lump it all together?

yep, that's also an option

jirka

^ permalink raw reply	[flat|nested] 221+ messages in thread

* Re: [PATCH 14/42] perf record: Add --index option for building index table
  2015-02-02 12:13               ` Jiri Olsa
@ 2015-02-02 14:56                 ` Namhyung Kim
  2015-02-02 17:30                   ` Jiri Olsa
  0 siblings, 1 reply; 221+ messages in thread
From: Namhyung Kim @ 2015-02-02 14:56 UTC (permalink / raw)
  To: Jiri Olsa
  Cc: Adrian Hunter, Arnaldo Carvalho de Melo, Ingo Molnar,
	Peter Zijlstra, LKML, David Ahern, Andi Kleen, Stephane Eranian,
	Frederic Weisbecker

Hi Jiri and Adrian,

On Mon, Feb 2, 2015 at 9:13 PM, Jiri Olsa <jolsa@redhat.com> wrote:
> On Mon, Feb 02, 2015 at 02:07:27PM +0200, Adrian Hunter wrote:
>
> SNIP
>
>> >>
>> >> Why not make it the same as all the other data. i.e. find the start and size
>> >> via the index? And then just lump all the data together?
>> >
>> > thats what I suggested
>>
>> No, I meant really lump it all together. i.e. perf_file_header.data.size =
>> total data size
>>
>> >
>> >>
>> >>> I guess we could workaround that by storing the 'perf_file_header::data'
>> >>> as the last data section. That would require to treat it the same way as
>> >>> all other data sections, but we could keep current header layout.
>> >>
>> >> Would it need to be last? Logically it should precede the data that depends
>> >> on it.
>> >
>> > i suggested this as a workaround for having features at the end of the file
>> > while keeping the current perf data header
>>
>> Which wouldn't be necessary if you lump it all together?
>
> yep, that's also an option

So we want a single section for the entire data area, right?

I also thought about it.  My concern was the holes between each data
due to page alignment.  If an old tool which doesn't know about the
index accesses to the data file, it'd just see a event type of 0 and
stop processing.

Maybe the page alignment is not necessary?

Thanks,
Namhyung

^ permalink raw reply	[flat|nested] 221+ messages in thread

* Re: [PATCH 01/42] perf tools: Support to read compressed module from build-id cache
  2015-01-30 14:32   ` Jiri Olsa
@ 2015-02-02 15:03     ` Namhyung Kim
  0 siblings, 0 replies; 221+ messages in thread
From: Namhyung Kim @ 2015-02-02 15:03 UTC (permalink / raw)
  To: Jiri Olsa
  Cc: Arnaldo Carvalho de Melo, Ingo Molnar, Peter Zijlstra, LKML,
	David Ahern, Adrian Hunter, Andi Kleen, Stephane Eranian,
	Frederic Weisbecker

On Fri, Jan 30, 2015 at 11:32 PM, Jiri Olsa <jolsa@redhat.com> wrote:
> On Thu, Jan 29, 2015 at 05:06:42PM +0900, Namhyung Kim wrote:
>> The commit c00c48fc6e6e ("perf symbols: Preparation for compressed
>> kernel module support") added support for compressed kernel modules
>> but it only supports system path DSOs.  When a dso is read from
>> build-id cache, its filename doesn't end with ".gz" but has build-id.
>> In this case, we should fallback to the original dso->name.
>>
>> Signed-off-by: Namhyung Kim <namhyung@kernel.org>
>> ---
>>  tools/perf/util/symbol-elf.c | 13 ++++++++-----
>>  1 file changed, 8 insertions(+), 5 deletions(-)
>>
>> diff --git a/tools/perf/util/symbol-elf.c b/tools/perf/util/symbol-elf.c
>> index 06fcd1bf98b6..b24f9d8727a8 100644
>> --- a/tools/perf/util/symbol-elf.c
>> +++ b/tools/perf/util/symbol-elf.c
>> @@ -574,13 +574,16 @@ static int decompress_kmodule(struct dso *dso, const char *name,
>>       const char *ext = strrchr(name, '.');
>>       char tmpbuf[] = "/tmp/perf-kmod-XXXXXX";
>>
>> -     if ((type != DSO_BINARY_TYPE__SYSTEM_PATH_KMODULE_COMP &&
>> -          type != DSO_BINARY_TYPE__GUEST_KMODULE_COMP) ||
>> -         type != dso->symtab_type)
>> +     if (type != DSO_BINARY_TYPE__SYSTEM_PATH_KMODULE_COMP &&
>> +         type != DSO_BINARY_TYPE__GUEST_KMODULE_COMP &&
>> +         type != DSO_BINARY_TYPE__BUILD_ID_CACHE)
>>               return -1;
>
> hum, is it possible the type == DSO_BINARY_TYPE__BUILD_ID_CACHE could get in here?
>
>
> ---
>         for (i = 0; i < DSO_BINARY_TYPE__SYMTAB_CNT; i++) {
>                 struct symsrc *ss = &ss_[ss_pos];
>                 bool next_slot = false;
>
>                 enum dso_binary_type symtab_type = binary_type_symtab[i];
>
>                 if (!dso__is_compatible_symtab_type(dso, kmod, symtab_type))
>                         continue;
>
> ---             ^^^ this check should rule out buildid symtab_type for kmod dso?

AFAICS symtab_type of BUILD_ID_CACHE always returns true in this function.


>
>                 symsrc__init(
>
>
> I wonder wether we should set special type from compressed binaries (as of now),
> or instead try to decompress anything that looks like it's compressed ;-)
> it seems more to be more generic and could simplify the code..

I don't know.  But it seems only kernel modules are compressed now.
If user-level dso also supports compression, we need to think about it
again..

Thanks,
Namhyung

^ permalink raw reply	[flat|nested] 221+ messages in thread

* Re: [PATCH 14/42] perf record: Add --index option for building index table
  2015-02-02 14:56                 ` Namhyung Kim
@ 2015-02-02 17:30                   ` Jiri Olsa
  2015-02-03  8:42                     ` Adrian Hunter
  0 siblings, 1 reply; 221+ messages in thread
From: Jiri Olsa @ 2015-02-02 17:30 UTC (permalink / raw)
  To: Namhyung Kim
  Cc: Adrian Hunter, Arnaldo Carvalho de Melo, Ingo Molnar,
	Peter Zijlstra, LKML, David Ahern, Andi Kleen, Stephane Eranian,
	Frederic Weisbecker

On Mon, Feb 02, 2015 at 11:56:09PM +0900, Namhyung Kim wrote:
> Hi Jiri and Adrian,
> 
> On Mon, Feb 2, 2015 at 9:13 PM, Jiri Olsa <jolsa@redhat.com> wrote:
> > On Mon, Feb 02, 2015 at 02:07:27PM +0200, Adrian Hunter wrote:
> >
> > SNIP
> >
> >> >>
> >> >> Why not make it the same as all the other data. i.e. find the start and size
> >> >> via the index? And then just lump all the data together?
> >> >
> >> > thats what I suggested
> >>
> >> No, I meant really lump it all together. i.e. perf_file_header.data.size =
> >> total data size
> >>
> >> >
> >> >>
> >> >>> I guess we could workaround that by storing the 'perf_file_header::data'
> >> >>> as the last data section. That would require to treat it the same way as
> >> >>> all other data sections, but we could keep current header layout.
> >> >>
> >> >> Would it need to be last? Logically it should precede the data that depends
> >> >> on it.
> >> >
> >> > i suggested this as a workaround for having features at the end of the file
> >> > while keeping the current perf data header
> >>
> >> Which wouldn't be necessary if you lump it all together?
> >
> > yep, that's also an option
> 
> So we want a single section for the entire data area, right?
> 
> I also thought about it.  My concern was the holes between each data
> due to page alignment.  If an old tool which doesn't know about the
> index accesses to the data file, it'd just see a event type of 0 and
> stop processing.
> 
> Maybe the page alignment is not necessary?

seems ok,  but how about time ordering.. every time you reach new
file data you'll hit 'out of order event' right?

hum, maybe it's not a big deal now when it's just incrementing counter ;-)

jirka

^ permalink raw reply	[flat|nested] 221+ messages in thread

* Re: [PATCH 14/42] perf record: Add --index option for building index table
  2015-02-02 17:30                   ` Jiri Olsa
@ 2015-02-03  8:42                     ` Adrian Hunter
  0 siblings, 0 replies; 221+ messages in thread
From: Adrian Hunter @ 2015-02-03  8:42 UTC (permalink / raw)
  To: Jiri Olsa, Namhyung Kim
  Cc: Arnaldo Carvalho de Melo, Ingo Molnar, Peter Zijlstra, LKML,
	David Ahern, Andi Kleen, Stephane Eranian, Frederic Weisbecker

On 02/02/15 19:30, Jiri Olsa wrote:
> On Mon, Feb 02, 2015 at 11:56:09PM +0900, Namhyung Kim wrote:
>> Hi Jiri and Adrian,
>>
>> On Mon, Feb 2, 2015 at 9:13 PM, Jiri Olsa <jolsa@redhat.com> wrote:
>>> On Mon, Feb 02, 2015 at 02:07:27PM +0200, Adrian Hunter wrote:
>>>
>>> SNIP
>>>
>>>>>>
>>>>>> Why not make it the same as all the other data. i.e. find the start and size
>>>>>> via the index? And then just lump all the data together?
>>>>>
>>>>> thats what I suggested
>>>>
>>>> No, I meant really lump it all together. i.e. perf_file_header.data.size =
>>>> total data size
>>>>
>>>>>
>>>>>>
>>>>>>> I guess we could workaround that by storing the 'perf_file_header::data'
>>>>>>> as the last data section. That would require to treat it the same way as
>>>>>>> all other data sections, but we could keep current header layout.
>>>>>>
>>>>>> Would it need to be last? Logically it should precede the data that depends
>>>>>> on it.
>>>>>
>>>>> i suggested this as a workaround for having features at the end of the file
>>>>> while keeping the current perf data header
>>>>
>>>> Which wouldn't be necessary if you lump it all together?
>>>
>>> yep, that's also an option
>>
>> So we want a single section for the entire data area, right?
>>
>> I also thought about it.  My concern was the holes between each data
>> due to page alignment.  If an old tool which doesn't know about the
>> index accesses to the data file, it'd just see a event type of 0 and
>> stop processing.

Please don't leave holes. Either fill them with a padding event or put the
data end-to-end.

>>
>> Maybe the page alignment is not necessary?
> 
> seems ok,  but how about time ordering.. every time you reach new
> file data you'll hit 'out of order event' right?
> 
> hum, maybe it's not a big deal now when it's just incrementing counter ;-)
> 
> jirka
> 
> 


^ permalink raw reply	[flat|nested] 221+ messages in thread

* [RFC/PATCHSET 00/38] perf tools: Speed-up perf report by using multi thread (v3)
@ 2015-03-03  3:07 Namhyung Kim
  2015-03-03  3:07 ` [PATCH 01/38] perf tools: Use a software dummy event to track task/mmap events Namhyung Kim
                   ` (37 more replies)
  0 siblings, 38 replies; 221+ messages in thread
From: Namhyung Kim @ 2015-03-03  3:07 UTC (permalink / raw)
  To: Arnaldo Carvalho de Melo
  Cc: Ingo Molnar, Peter Zijlstra, Jiri Olsa, LKML,
	Frederic Weisbecker, Adrian Hunter, Stephane Eranian, Andi Kleen,
	David Ahern

Hello,

This patchset converts perf report to use multiple threads in order to
speed up the processing on large data files.  I can see a minimum ~30%
of speedup with this change.  The code is still experimental and
contains many rough edges.  But I'd like to share and give some
feedbacks.

 * changes in v3)
  - handle header (metadata) same as sample data (at index 0)
  - maintain libunwind address space in map_groups instead of thread
  - use *_time API only for indexed data file
  - resolve callchain with the *_time API
  - use dso__data_get/put_fd() to protect access to fd
  - synthesize COMM event for command line workload

 * changes in v2)
  - rework with single indexed data file rather than multiple files in
    a directory

The perf report processes (sample) events like below:

  1. preprocess sample to get matching thread/dso/symbol info
  2. insert it to hists rbtree (with callchain tree) based on the info
  3. optionally collapse hist entries that match given sort key(s)
  4. resort hist entries (by overhead) for output
  5. display the hist entries

The stage 1 is a preprocessing and mostly act like a read-only
operation in that it doesn't change a machine state during the sample
processing.  Meta events like fork, comm and mmap can change the
machine/thread state but symbols can be loaded during the processing
(stage 2).

The stage 2 consumes most of the time especially with callchains and
 --children option is enabled.  And this work can be easily patitioned
as each sample is independent to others.  But the resulting hists must
be combined/collapsed to a single global hists before going to further
steps.

The stage 3 is optional and only needed by certain sort keys - but
with stage 2 paralellized, it needs to be done always.

The stage 4 and 5 works on whole hists so must be done serially.

So my approach is like this:

Partially do stage 1 first - but only for meta events that changes
machine state.  To do this I add a dummy tracking event to perf record
and make it collect such meta events only.  They are saved as normal
data and processed before sample events at perf report time.

This also requires to handle multiple sample data concurrently and to
find a corresponding machine state when processing samples.  On a
large profiling session, many tasks were created and exited so pid
might be recycled (even more than once!).  To deal with it, I managed
to have thread, map_groups and comm in time sorted.  The only
remaining thing is symbol loading as it's done lazily when sample
requires it.

With that being done, the stage 2 can be done by multiple threads.  I
also save each sample data (per-cpu or per-thread) in separate files
during record and then merge them into a single data file with an
index table.  On perf report time, each region of sample data will be
processed by each thread.  And symbol loading is protected by a mutex
lock.

For DWARF post-unwinding, dso cache data also needs to be protected by
a lock and this caused a huge contention.  I made it to search the
rbtree speculatively first and then, if it didn't find one, search it
again under the dso lock.  Please take a look at it if it's acceptable.

The patch 1-9 are to support indexing for data file.  With --index
option, perf record will create a intermediate directory and then save
meta events and sample data to separate files.  And finally it'll
build an index table and concatenate the data files (and also remove
the intermediate direcotry).

The patch 10-23 are to manage machine and thread state using timestamp
so that it can be searched when processing samples.  The patch 24-37
are to implement parallel report.  And finally I implemented 'perf
data index' command to build an index table for a given data file.

This patchset didn't change perf record to use multi-thread.  But I
think it can be easily done later if needed.

Note that output has a slight difference to original version when
compared using indexed data file.  But they're mostly unresolved
symbols for callchains.

Here is the result:

This is just elapsed time measured by 'perf stat -r 5'.

The data file was recorded during kernel build with fp callchain and
size is 2.1GB.  The machine has 6 core with hyper-threading enabled
and I got a similar result on my laptop too.

 perf report          --children  --no-children  + --call-graph none
 		   -------------  -------------  -------------------
 current           286.213349446   93.753958745      36.860880945  
 with index        270.158361549   87.963067415      32.896841653
 + --multi-thread  166.039011492   43.209152911       8.434560193


This result is with 7.7GB data file using libunwind for callchain.

 perf report          --children  --no-children  + --call-graph none
 		   -------------  -------------  -------------------
 current           150.714039134  111.420099831       5.035423803
 with index        152.438739157  112.691612534       3.642109876
 + --multi-thread   45.966048256   29.844907087       1.829218561

I guess the speedup of indexed data file came from skipping ordered
event layer.

This result is with same file but using libdw for callchain unwind.

 perf report          --children  --no-children  + --call-graph none
 		   -------------  -------------  -------------------
 current           457.507820205  491.520096816       4.831840810
 with index        441.140769287  461.993666236       3.767947395
 + --multi-thread  219.289176894  171.935294339       1.785351793

On my archlinux system, callchain unwind using libdw is much slower
than libunwind.  I'm using elfutils version 0.160.  Also I don't know
why --children takes less time than --no-children.  Anyway we can see
the --multi-thread performance is much better for each case.


You can get it from 'perf/threaded-v3' branch on my tree at:

  git://git.kernel.org/pub/scm/linux/kernel/git/namhyung/linux-perf.git

Please take a look and play with it.  Any comments are welcome! :)

Thanks,
Namhyung


Namhyung Kim (38):
  perf tools: Use a software dummy event to track task/mmap events
  perf tools: Add rm_rf() utility function
  perf tools: Introduce copyfile_offset() function
  perf tools: Create separate mmap for dummy tracking event
  perf tools: Introduce perf_evlist__mmap_track()
  perf tools: Add HEADER_DATA_INDEX feature
  perf tools: Handle indexed data file properly
  perf record: Add --index option for building index table
  perf report: Skip dummy tracking event
  perf tools: Pass session arg to perf_event__preprocess_sample()
  perf script: Pass session arg to ->process_event callback
  perf tools: Introduce thread__comm_time() helpers
  perf tools: Add a test case for thread comm handling
  perf tools: Use thread__comm_time() when adding hist entries
  perf tools: Convert dead thread list into rbtree
  perf tools: Introduce machine__find*_thread_time()
  perf tools: Add a test case for timed thread handling
  perf tools: Reducing arguments of hist_entry_iter__add()
  perf tools: Pass session to hist_entry_iter struct
  perf tools: Maintain map groups list in a leader thread
  perf tools: Introduce session__find_addr_location() and friends
  perf callchain: Use session__find_addr_location() and friends
  perf tools: Add a test case for timed map groups handling
  perf tools: Protect dso symbol loading using a mutex
  perf tools: Protect dso cache tree using dso->lock
  perf tools: Protect dso cache fd with a mutex
  perf callchain: Maintain libunwind's address space in map_groups
  perf tools: Add dso__data_get/put_fd()
  perf session: Pass struct events stats to event processing functions
  perf hists: Pass hists struct to hist_entry_iter struct
  perf tools: Move BUILD_ID_SIZE definition to perf.h
  perf report: Parallelize perf report using multi-thread
  perf tools: Add missing_threads rb tree
  perf record: Synthesize COMM event for a command line workload
  perf tools: Fix progress ui to support multi thread
  perf report: Add --multi-thread option and config item
  perf session: Handle index files generally
  perf data: Implement 'index' subcommand

 tools/perf/Documentation/perf-data.txt             |  25 +-
 tools/perf/Documentation/perf-record.txt           |   4 +
 tools/perf/Documentation/perf-report.txt           |   3 +
 tools/perf/builtin-annotate.c                      |   8 +-
 tools/perf/builtin-data.c                          | 349 ++++++++++++++++++++-
 tools/perf/builtin-diff.c                          |  21 +-
 tools/perf/builtin-mem.c                           |   6 +-
 tools/perf/builtin-record.c                        | 189 ++++++++++-
 tools/perf/builtin-report.c                        |  81 ++++-
 tools/perf/builtin-script.c                        |  58 ++--
 tools/perf/builtin-timechart.c                     |  10 +-
 tools/perf/builtin-top.c                           |  12 +-
 tools/perf/perf.h                                  |   2 +
 tools/perf/tests/Build                             |   3 +
 tools/perf/tests/builtin-test.c                    |  12 +
 tools/perf/tests/dwarf-unwind.c                    |  14 +-
 tools/perf/tests/hists_common.c                    |   3 +-
 tools/perf/tests/hists_cumulate.c                  |   9 +-
 tools/perf/tests/hists_filter.c                    |   7 +-
 tools/perf/tests/hists_link.c                      |  10 +-
 tools/perf/tests/hists_output.c                    |   9 +-
 tools/perf/tests/tests.h                           |   3 +
 tools/perf/tests/thread-comm.c                     |  47 +++
 tools/perf/tests/thread-lookup-time.c              | 180 +++++++++++
 tools/perf/tests/thread-mg-share.c                 |   7 +-
 tools/perf/tests/thread-mg-time.c                  |  88 ++++++
 tools/perf/ui/browsers/hists.c                     |  30 +-
 tools/perf/ui/gtk/hists.c                          |   3 +
 tools/perf/util/build-id.c                         |   9 +-
 tools/perf/util/build-id.h                         |   2 -
 tools/perf/util/callchain.c                        |   6 +-
 tools/perf/util/callchain.h                        |   4 +-
 tools/perf/util/db-export.c                        |   6 +-
 tools/perf/util/db-export.h                        |   4 +-
 tools/perf/util/dso.c                              | 172 +++++++---
 tools/perf/util/dso.h                              |  11 +-
 tools/perf/util/event.c                            |  77 ++++-
 tools/perf/util/event.h                            |  13 +-
 tools/perf/util/evlist.c                           | 161 ++++++++--
 tools/perf/util/evlist.h                           |  22 +-
 tools/perf/util/evsel.h                            |  15 +
 tools/perf/util/header.c                           |  61 ++++
 tools/perf/util/header.h                           |   3 +
 tools/perf/util/hist.c                             | 126 +++++---
 tools/perf/util/hist.h                             |   9 +-
 tools/perf/util/machine.c                          | 287 ++++++++++++++---
 tools/perf/util/machine.h                          |  15 +-
 tools/perf/util/map.c                              |   8 +
 tools/perf/util/map.h                              |   3 +
 tools/perf/util/ordered-events.c                   |   4 +-
 .../perf/util/scripting-engines/trace-event-perl.c |   3 +-
 .../util/scripting-engines/trace-event-python.c    |  32 +-
 tools/perf/util/session.c                          | 345 +++++++++++++++++---
 tools/perf/util/session.h                          |  48 ++-
 tools/perf/util/symbol.c                           |  34 +-
 tools/perf/util/thread.c                           | 173 +++++++++-
 tools/perf/util/thread.h                           |  28 +-
 tools/perf/util/tool.h                             |  14 +
 tools/perf/util/trace-event-scripting.c            |   3 +-
 tools/perf/util/trace-event.h                      |   3 +-
 tools/perf/util/unwind-libdw.c                     |  14 +-
 tools/perf/util/unwind-libdw.h                     |   1 +
 tools/perf/util/unwind-libunwind.c                 |  98 +++---
 tools/perf/util/unwind.h                           |  18 +-
 tools/perf/util/util.c                             |  81 ++++-
 tools/perf/util/util.h                             |   2 +
 66 files changed, 2670 insertions(+), 438 deletions(-)
 create mode 100644 tools/perf/tests/thread-comm.c
 create mode 100644 tools/perf/tests/thread-lookup-time.c
 create mode 100644 tools/perf/tests/thread-mg-time.c

-- 
2.2.2


^ permalink raw reply	[flat|nested] 221+ messages in thread

* [PATCH 01/38] perf tools: Use a software dummy event to track task/mmap events
  2015-03-03  3:07 [RFC/PATCHSET 00/38] perf tools: Speed-up perf report by using multi thread (v3) Namhyung Kim
@ 2015-03-03  3:07 ` Namhyung Kim
  2015-03-03  3:07 ` [PATCH 02/38] perf tools: Add rm_rf() utility function Namhyung Kim
                   ` (36 subsequent siblings)
  37 siblings, 0 replies; 221+ messages in thread
From: Namhyung Kim @ 2015-03-03  3:07 UTC (permalink / raw)
  To: Arnaldo Carvalho de Melo
  Cc: Ingo Molnar, Peter Zijlstra, Jiri Olsa, LKML,
	Frederic Weisbecker, Adrian Hunter, Stephane Eranian, Andi Kleen,
	David Ahern

Add APIs for software dummy event to track task/comm/mmap events
separately.  The perf record will use them to save such events in a
separate mmap buffer to make it easy to index.  This is a preparation of
multi-thread support which will come later.

Cc: Adrian Hunter <adrian.hunter@intel.com>
Signed-off-by: Namhyung Kim <namhyung@kernel.org>
---
 tools/perf/util/evlist.c | 30 ++++++++++++++++++++++++++++++
 tools/perf/util/evlist.h |  1 +
 tools/perf/util/evsel.h  | 15 +++++++++++++++
 3 files changed, 46 insertions(+)

diff --git a/tools/perf/util/evlist.c b/tools/perf/util/evlist.c
index 8d0b62361129..928a5750648d 100644
--- a/tools/perf/util/evlist.c
+++ b/tools/perf/util/evlist.c
@@ -194,6 +194,36 @@ int perf_evlist__add_default(struct perf_evlist *evlist)
 	return -ENOMEM;
 }
 
+int perf_evlist__add_dummy_tracking(struct perf_evlist *evlist)
+{
+	struct perf_event_attr attr = {
+		.type = PERF_TYPE_SOFTWARE,
+		.config = PERF_COUNT_SW_DUMMY,
+		.exclude_kernel = 1,
+	};
+	struct perf_evsel *evsel;
+
+	event_attr_init(&attr);
+
+	evsel = perf_evsel__new(&attr);
+	if (evsel == NULL)
+		goto error;
+
+	/* use strdup() because free(evsel) assumes name is allocated */
+	evsel->name = strdup("dummy");
+	if (!evsel->name)
+		goto error_free;
+
+	perf_evlist__add(evlist, evsel);
+	perf_evlist__set_tracking_event(evlist, evsel);
+
+	return 0;
+error_free:
+	perf_evsel__delete(evsel);
+error:
+	return -ENOMEM;
+}
+
 static int perf_evlist__add_attrs(struct perf_evlist *evlist,
 				  struct perf_event_attr *attrs, size_t nr_attrs)
 {
diff --git a/tools/perf/util/evlist.h b/tools/perf/util/evlist.h
index f07c984465f0..a278df8fbed3 100644
--- a/tools/perf/util/evlist.h
+++ b/tools/perf/util/evlist.h
@@ -68,6 +68,7 @@ void perf_evlist__delete(struct perf_evlist *evlist);
 
 void perf_evlist__add(struct perf_evlist *evlist, struct perf_evsel *entry);
 int perf_evlist__add_default(struct perf_evlist *evlist);
+int perf_evlist__add_dummy_tracking(struct perf_evlist *evlist);
 int __perf_evlist__add_default_attrs(struct perf_evlist *evlist,
 				     struct perf_event_attr *attrs, size_t nr_attrs);
 
diff --git a/tools/perf/util/evsel.h b/tools/perf/util/evsel.h
index dcf202aebe9f..80aeb3d84593 100644
--- a/tools/perf/util/evsel.h
+++ b/tools/perf/util/evsel.h
@@ -331,6 +331,21 @@ static inline bool perf_evsel__is_function_event(struct perf_evsel *evsel)
 #undef FUNCTION_EVENT
 }
 
+/**
+ * perf_evsel__is_dummy_tracking - Return whether given evsel is a dummy
+ * event for tracking meta events only
+ *
+ * @evsel - evsel selector to be tested
+ *
+ * Return %true if event is a dummy tracking event
+ */
+static inline bool perf_evsel__is_dummy_tracking(struct perf_evsel *evsel)
+{
+	return evsel->attr.type == PERF_TYPE_SOFTWARE &&
+		evsel->attr.config == PERF_COUNT_SW_DUMMY &&
+		evsel->attr.task == 1 && evsel->attr.mmap == 1;
+}
+
 struct perf_attr_details {
 	bool freq;
 	bool verbose;
-- 
2.2.2


^ permalink raw reply related	[flat|nested] 221+ messages in thread

* [PATCH 02/38] perf tools: Add rm_rf() utility function
  2015-03-03  3:07 [RFC/PATCHSET 00/38] perf tools: Speed-up perf report by using multi thread (v3) Namhyung Kim
  2015-03-03  3:07 ` [PATCH 01/38] perf tools: Use a software dummy event to track task/mmap events Namhyung Kim
@ 2015-03-03  3:07 ` Namhyung Kim
  2015-03-03  3:07 ` [PATCH 03/38] perf tools: Introduce copyfile_offset() function Namhyung Kim
                   ` (35 subsequent siblings)
  37 siblings, 0 replies; 221+ messages in thread
From: Namhyung Kim @ 2015-03-03  3:07 UTC (permalink / raw)
  To: Arnaldo Carvalho de Melo
  Cc: Ingo Molnar, Peter Zijlstra, Jiri Olsa, LKML,
	Frederic Weisbecker, Adrian Hunter, Stephane Eranian, Andi Kleen,
	David Ahern

The rm_rf() function does same as the shell command 'rm -rf' which
removes all directory entries recursively.

Signed-off-by: Namhyung Kim <namhyung@kernel.org>
---
 tools/perf/util/util.c | 43 +++++++++++++++++++++++++++++++++++++++++++
 tools/perf/util/util.h |  1 +
 2 files changed, 44 insertions(+)

diff --git a/tools/perf/util/util.c b/tools/perf/util/util.c
index 4ee6d0d4c993..6104afb7e1ef 100644
--- a/tools/perf/util/util.c
+++ b/tools/perf/util/util.c
@@ -72,6 +72,49 @@ int mkdir_p(char *path, mode_t mode)
 	return (stat(path, &st) && mkdir(path, mode)) ? -1 : 0;
 }
 
+int rm_rf(char *path)
+{
+	DIR *dir;
+	int ret = 0;
+	struct dirent *d;
+	char namebuf[PATH_MAX];
+
+	dir = opendir(path);
+	if (dir == NULL)
+		return 0;
+
+	while ((d = readdir(dir)) != NULL && !ret) {
+		struct stat statbuf;
+
+		if (!strcmp(d->d_name, ".") || !strcmp(d->d_name, ".."))
+			continue;
+
+		scnprintf(namebuf, sizeof(namebuf), "%s/%s",
+			  path, d->d_name);
+
+		ret = stat(namebuf, &statbuf);
+		if (ret < 0) {
+			pr_debug("stat failed: %s\n", namebuf);
+			break;
+		}
+
+		if (S_ISREG(statbuf.st_mode))
+			ret = unlink(namebuf);
+		else if (S_ISDIR(statbuf.st_mode))
+			ret = rm_rf(namebuf);
+		else {
+			pr_debug("unknown file: %s\n", namebuf);
+			ret = -1;
+		}
+	}
+	closedir(dir);
+
+	if (ret < 0)
+		return ret;
+
+	return rmdir(path);
+}
+
 static int slow_copyfile(const char *from, const char *to, mode_t mode)
 {
 	int err = -1;
diff --git a/tools/perf/util/util.h b/tools/perf/util/util.h
index fbd598afc606..ba31979fcdcc 100644
--- a/tools/perf/util/util.h
+++ b/tools/perf/util/util.h
@@ -249,6 +249,7 @@ static inline int sane_case(int x, int high)
 }
 
 int mkdir_p(char *path, mode_t mode);
+int rm_rf(char *path);
 int copyfile(const char *from, const char *to);
 int copyfile_mode(const char *from, const char *to, mode_t mode);
 
-- 
2.2.2


^ permalink raw reply related	[flat|nested] 221+ messages in thread

* [PATCH 03/38] perf tools: Introduce copyfile_offset() function
  2015-03-03  3:07 [RFC/PATCHSET 00/38] perf tools: Speed-up perf report by using multi thread (v3) Namhyung Kim
  2015-03-03  3:07 ` [PATCH 01/38] perf tools: Use a software dummy event to track task/mmap events Namhyung Kim
  2015-03-03  3:07 ` [PATCH 02/38] perf tools: Add rm_rf() utility function Namhyung Kim
@ 2015-03-03  3:07 ` Namhyung Kim
  2015-03-04 14:58   ` Jiri Olsa
  2015-03-03  3:07 ` [PATCH 04/38] perf tools: Create separate mmap for dummy tracking event Namhyung Kim
                   ` (34 subsequent siblings)
  37 siblings, 1 reply; 221+ messages in thread
From: Namhyung Kim @ 2015-03-03  3:07 UTC (permalink / raw)
  To: Arnaldo Carvalho de Melo
  Cc: Ingo Molnar, Peter Zijlstra, Jiri Olsa, LKML,
	Frederic Weisbecker, Adrian Hunter, Stephane Eranian, Andi Kleen,
	David Ahern

The copyfile_offset() function is to copy source data from given
offset to a destination file with an offset.  It'll be used to build
an indexed data file.

Signed-off-by: Namhyung Kim <namhyung@kernel.org>
---
 tools/perf/util/util.c | 38 +++++++++++++++++++++++++++++---------
 tools/perf/util/util.h |  1 +
 2 files changed, 30 insertions(+), 9 deletions(-)

diff --git a/tools/perf/util/util.c b/tools/perf/util/util.c
index 6104afb7e1ef..0c264bc685ac 100644
--- a/tools/perf/util/util.c
+++ b/tools/perf/util/util.c
@@ -145,11 +145,38 @@ static int slow_copyfile(const char *from, const char *to, mode_t mode)
 	return err;
 }
 
+int copyfile_offset(int ifd, loff_t off_in, int ofd, loff_t off_out, u64 size)
+{
+	void *ptr;
+	loff_t pgoff;
+
+	pgoff = off_in & ~(page_size - 1);
+	off_in -= pgoff;
+
+	ptr = mmap(NULL, off_in + size, PROT_READ, MAP_PRIVATE, ifd, pgoff);
+	if (ptr == MAP_FAILED)
+		return -1;
+
+	while (size) {
+		ssize_t ret = pwrite(ofd, ptr + off_in, size, off_out);
+		if (ret < 0 && errno == EINTR)
+			continue;
+		if (ret <= 0)
+			break;
+
+		size -= ret;
+		off_in += ret;
+		off_out -= ret;
+	}
+	munmap(ptr, off_in + size);
+
+	return size ? -1 : 0;
+}
+
 int copyfile_mode(const char *from, const char *to, mode_t mode)
 {
 	int fromfd, tofd;
 	struct stat st;
-	void *addr;
 	int err = -1;
 
 	if (stat(from, &st))
@@ -166,15 +193,8 @@ int copyfile_mode(const char *from, const char *to, mode_t mode)
 	if (tofd < 0)
 		goto out_close_from;
 
-	addr = mmap(NULL, st.st_size, PROT_READ, MAP_PRIVATE, fromfd, 0);
-	if (addr == MAP_FAILED)
-		goto out_close_to;
-
-	if (write(tofd, addr, st.st_size) == st.st_size)
-		err = 0;
+	err = copyfile_offset(fromfd, 0, tofd, 0, st.st_size);
 
-	munmap(addr, st.st_size);
-out_close_to:
 	close(tofd);
 	if (err)
 		unlink(to);
diff --git a/tools/perf/util/util.h b/tools/perf/util/util.h
index ba31979fcdcc..91535bceb1bf 100644
--- a/tools/perf/util/util.h
+++ b/tools/perf/util/util.h
@@ -252,6 +252,7 @@ int mkdir_p(char *path, mode_t mode);
 int rm_rf(char *path);
 int copyfile(const char *from, const char *to);
 int copyfile_mode(const char *from, const char *to, mode_t mode);
+int copyfile_offset(int fromfd, loff_t from_ofs, int tofd, loff_t to_ofs, u64 size);
 
 s64 perf_atoll(const char *str);
 char **argv_split(const char *str, int *argcp);
-- 
2.2.2


^ permalink raw reply related	[flat|nested] 221+ messages in thread

* [PATCH 04/38] perf tools: Create separate mmap for dummy tracking event
  2015-03-03  3:07 [RFC/PATCHSET 00/38] perf tools: Speed-up perf report by using multi thread (v3) Namhyung Kim
                   ` (2 preceding siblings ...)
  2015-03-03  3:07 ` [PATCH 03/38] perf tools: Introduce copyfile_offset() function Namhyung Kim
@ 2015-03-03  3:07 ` Namhyung Kim
  2015-03-03  3:07 ` [PATCH 05/38] perf tools: Introduce perf_evlist__mmap_track() Namhyung Kim
                   ` (33 subsequent siblings)
  37 siblings, 0 replies; 221+ messages in thread
From: Namhyung Kim @ 2015-03-03  3:07 UTC (permalink / raw)
  To: Arnaldo Carvalho de Melo
  Cc: Ingo Molnar, Peter Zijlstra, Jiri Olsa, LKML,
	Frederic Weisbecker, Adrian Hunter, Stephane Eranian, Andi Kleen,
	David Ahern

When indexed data file support is enabled, a dummy tracking event will
be used to track metadata (like task, comm and mmap events) for a
session and actual samples will be recorded in separate (intermediate)
files and then merged (with index table).

Provide separate mmap to the dummy tracking event.  The size is fixed
to 128KiB (+ 1 page) as the event rate will be lower than samples.  I
originally wanted to use a single mmap for this but cross-cpu sharing
is prohibited so it's per-cpu (or per-task) like normal mmaps.

Cc: Adrian Hunter <adrian.hunter@intel.com>
Signed-off-by: Namhyung Kim <namhyung@kernel.org>
---
 tools/perf/builtin-record.c |   9 +++-
 tools/perf/util/evlist.c    | 122 +++++++++++++++++++++++++++++++++++---------
 tools/perf/util/evlist.h    |  11 +++-
 3 files changed, 117 insertions(+), 25 deletions(-)

diff --git a/tools/perf/builtin-record.c b/tools/perf/builtin-record.c
index 4fdad06d37db..2bd724763e1d 100644
--- a/tools/perf/builtin-record.c
+++ b/tools/perf/builtin-record.c
@@ -69,7 +69,7 @@ static int process_synthesized_event(struct perf_tool *tool,
 
 static int record__mmap_read(struct record *rec, int idx)
 {
-	struct perf_mmap *md = &rec->evlist->mmap[idx];
+	struct perf_mmap *md = perf_evlist__mmap_desc(rec->evlist, idx);
 	unsigned int head = perf_mmap__read_head(md);
 	unsigned int old = md->prev;
 	unsigned char *data = md->base + page_size;
@@ -105,6 +105,7 @@ static int record__mmap_read(struct record *rec, int idx)
 	}
 
 	md->prev = old;
+
 	perf_evlist__mmap_consume(rec->evlist, idx);
 out:
 	return rc;
@@ -275,6 +276,12 @@ static int record__mmap_read_all(struct record *rec)
 				goto out;
 			}
 		}
+		if (rec->evlist->track_mmap) {
+			if (record__mmap_read(rec, track_mmap_idx(i)) != 0) {
+				rc = -1;
+				goto out;
+			}
+		}
 	}
 
 	/*
diff --git a/tools/perf/util/evlist.c b/tools/perf/util/evlist.c
index 928a5750648d..ebbec07843a2 100644
--- a/tools/perf/util/evlist.c
+++ b/tools/perf/util/evlist.c
@@ -28,6 +28,7 @@
 
 static void perf_evlist__mmap_put(struct perf_evlist *evlist, int idx);
 static void __perf_evlist__munmap(struct perf_evlist *evlist, int idx);
+static void __perf_evlist__munmap_track(struct perf_evlist *evlist, int idx);
 
 #define FD(e, x, y) (*(int *)xyarray__entry(e->fd, x, y))
 #define SID(e, x, y) xyarray__entry(e->sample_id, x, y)
@@ -728,22 +729,39 @@ static bool perf_mmap__empty(struct perf_mmap *md)
 	return perf_mmap__read_head(md) != md->prev;
 }
 
+struct perf_mmap *perf_evlist__mmap_desc(struct perf_evlist *evlist, int idx)
+{
+	if (idx >= 0)
+		return &evlist->mmap[idx];
+	else
+		return &evlist->track_mmap[track_mmap_idx(idx)];
+}
+
 static void perf_evlist__mmap_get(struct perf_evlist *evlist, int idx)
 {
-	++evlist->mmap[idx].refcnt;
+	struct perf_mmap *md = perf_evlist__mmap_desc(evlist, idx);
+
+	++md->refcnt;
 }
 
 static void perf_evlist__mmap_put(struct perf_evlist *evlist, int idx)
 {
-	BUG_ON(evlist->mmap[idx].refcnt == 0);
+	struct perf_mmap *md = perf_evlist__mmap_desc(evlist, idx);
+
+	BUG_ON(md->refcnt == 0);
+
+	if (--md->refcnt != 0)
+		return;
 
-	if (--evlist->mmap[idx].refcnt == 0)
+	if (idx >= 0)
 		__perf_evlist__munmap(evlist, idx);
+	else
+		__perf_evlist__munmap_track(evlist, track_mmap_idx(idx));
 }
 
 void perf_evlist__mmap_consume(struct perf_evlist *evlist, int idx)
 {
-	struct perf_mmap *md = &evlist->mmap[idx];
+	struct perf_mmap *md = perf_evlist__mmap_desc(evlist, idx);
 
 	if (!evlist->overwrite) {
 		unsigned int old = md->prev;
@@ -764,6 +782,15 @@ static void __perf_evlist__munmap(struct perf_evlist *evlist, int idx)
 	}
 }
 
+static void __perf_evlist__munmap_track(struct perf_evlist *evlist, int idx)
+{
+	if (evlist->track_mmap[idx].base != NULL) {
+		munmap(evlist->track_mmap[idx].base, TRACK_MMAP_SIZE);
+		evlist->track_mmap[idx].base = NULL;
+		evlist->track_mmap[idx].refcnt = 0;
+	}
+}
+
 void perf_evlist__munmap(struct perf_evlist *evlist)
 {
 	int i;
@@ -775,23 +802,43 @@ void perf_evlist__munmap(struct perf_evlist *evlist)
 		__perf_evlist__munmap(evlist, i);
 
 	zfree(&evlist->mmap);
+
+	if (evlist->track_mmap == NULL)
+		return;
+
+	for (i = 0; i < evlist->nr_mmaps; i++)
+		__perf_evlist__munmap_track(evlist, i);
+
+	zfree(&evlist->track_mmap);
 }
 
-static int perf_evlist__alloc_mmap(struct perf_evlist *evlist)
+static int perf_evlist__alloc_mmap(struct perf_evlist *evlist, bool track_mmap)
 {
 	evlist->nr_mmaps = cpu_map__nr(evlist->cpus);
 	if (cpu_map__empty(evlist->cpus))
 		evlist->nr_mmaps = thread_map__nr(evlist->threads);
 	evlist->mmap = zalloc(evlist->nr_mmaps * sizeof(struct perf_mmap));
-	return evlist->mmap != NULL ? 0 : -ENOMEM;
+	if (evlist->mmap == NULL)
+		return -ENOMEM;
+
+	if (track_mmap) {
+		evlist->track_mmap = calloc(evlist->nr_mmaps,
+					    sizeof(struct perf_mmap));
+		if (evlist->track_mmap == NULL) {
+			zfree(&evlist->mmap);
+			return -ENOMEM;
+		}
+	}
+	return 0;
 }
 
 struct mmap_params {
-	int prot;
-	int mask;
+	int	prot;
+	size_t	len;
 };
 
-static int __perf_evlist__mmap(struct perf_evlist *evlist, int idx,
+static int __perf_evlist__mmap(struct perf_evlist *evlist __maybe_unused,
+			       struct perf_mmap *pmmap,
 			       struct mmap_params *mp, int fd)
 {
 	/*
@@ -807,15 +854,14 @@ static int __perf_evlist__mmap(struct perf_evlist *evlist, int idx,
 	 * evlist layer can't just drop it when filtering events in
 	 * perf_evlist__filter_pollfd().
 	 */
-	evlist->mmap[idx].refcnt = 2;
-	evlist->mmap[idx].prev = 0;
-	evlist->mmap[idx].mask = mp->mask;
-	evlist->mmap[idx].base = mmap(NULL, evlist->mmap_len, mp->prot,
-				      MAP_SHARED, fd, 0);
-	if (evlist->mmap[idx].base == MAP_FAILED) {
+	pmmap->refcnt = 2;
+	pmmap->prev = 0;
+	pmmap->mask = mp->len - page_size - 1;
+	pmmap->base = mmap(NULL, mp->len, mp->prot, MAP_SHARED, fd, 0);
+	if (pmmap->base == MAP_FAILED) {
 		pr_debug2("failed to mmap perf event ring buffer, error %d\n",
 			  errno);
-		evlist->mmap[idx].base = NULL;
+		pmmap->base = NULL;
 		return -1;
 	}
 
@@ -824,7 +870,8 @@ static int __perf_evlist__mmap(struct perf_evlist *evlist, int idx,
 
 static int perf_evlist__mmap_per_evsel(struct perf_evlist *evlist, int idx,
 				       struct mmap_params *mp, int cpu,
-				       int thread, int *output)
+				       int thread, int *output,
+				       int *track_output)
 {
 	struct perf_evsel *evsel;
 
@@ -836,9 +883,30 @@ static int perf_evlist__mmap_per_evsel(struct perf_evlist *evlist, int idx,
 
 		fd = FD(evsel, cpu, thread);
 
-		if (*output == -1) {
+		if (perf_evsel__is_dummy_tracking(evsel)) {
+			struct mmap_params track_mp = {
+				.prot	= mp->prot,
+				.len	= TRACK_MMAP_SIZE,
+			};
+
+			if (*track_output == -1) {
+				*track_output = fd;
+				if (__perf_evlist__mmap(evlist,
+							&evlist->track_mmap[idx],
+							&track_mp, fd) < 0)
+					return -1;
+			} else {
+				if (ioctl(fd, PERF_EVENT_IOC_SET_OUTPUT,
+					  *track_output) != 0)
+					return -1;
+			}
+
+			/* mark idx as track mmap idx (negative) */
+			idx = track_mmap_idx(idx);
+		} else if (*output == -1) {
 			*output = fd;
-			if (__perf_evlist__mmap(evlist, idx, mp, *output) < 0)
+			if (__perf_evlist__mmap(evlist, &evlist->mmap[idx],
+						mp, *output) < 0)
 				return -1;
 		} else {
 			if (ioctl(fd, PERF_EVENT_IOC_SET_OUTPUT, *output) != 0)
@@ -867,6 +935,11 @@ static int perf_evlist__mmap_per_evsel(struct perf_evlist *evlist, int idx,
 			perf_evlist__set_sid_idx(evlist, evsel, idx, cpu,
 						 thread);
 		}
+
+		if (perf_evsel__is_dummy_tracking(evsel)) {
+			/* restore idx as normal idx (positive) */
+			idx = track_mmap_idx(idx);
+		}
 	}
 
 	return 0;
@@ -882,10 +955,12 @@ static int perf_evlist__mmap_per_cpu(struct perf_evlist *evlist,
 	pr_debug2("perf event ring buffer mmapped per cpu\n");
 	for (cpu = 0; cpu < nr_cpus; cpu++) {
 		int output = -1;
+		int track_output = -1;
 
 		for (thread = 0; thread < nr_threads; thread++) {
 			if (perf_evlist__mmap_per_evsel(evlist, cpu, mp, cpu,
-							thread, &output))
+							thread, &output,
+							&track_output))
 				goto out_unmap;
 		}
 	}
@@ -907,9 +982,10 @@ static int perf_evlist__mmap_per_thread(struct perf_evlist *evlist,
 	pr_debug2("perf event ring buffer mmapped per thread\n");
 	for (thread = 0; thread < nr_threads; thread++) {
 		int output = -1;
+		int track_output = -1;
 
 		if (perf_evlist__mmap_per_evsel(evlist, thread, mp, 0, thread,
-						&output))
+						&output, &track_output))
 			goto out_unmap;
 	}
 
@@ -1032,7 +1108,7 @@ int perf_evlist__mmap(struct perf_evlist *evlist, unsigned int pages,
 		.prot = PROT_READ | (overwrite ? 0 : PROT_WRITE),
 	};
 
-	if (evlist->mmap == NULL && perf_evlist__alloc_mmap(evlist) < 0)
+	if (evlist->mmap == NULL && perf_evlist__alloc_mmap(evlist, true) < 0)
 		return -ENOMEM;
 
 	if (evlist->pollfd.entries == NULL && perf_evlist__alloc_pollfd(evlist) < 0)
@@ -1041,7 +1117,7 @@ int perf_evlist__mmap(struct perf_evlist *evlist, unsigned int pages,
 	evlist->overwrite = overwrite;
 	evlist->mmap_len = perf_evlist__mmap_size(pages);
 	pr_debug("mmap size %zuB\n", evlist->mmap_len);
-	mp.mask = evlist->mmap_len - page_size - 1;
+	mp.len = evlist->mmap_len;
 
 	evlist__for_each(evlist, evsel) {
 		if ((evsel->attr.read_format & PERF_FORMAT_ID) &&
diff --git a/tools/perf/util/evlist.h b/tools/perf/util/evlist.h
index a278df8fbed3..3bd9747bb9aa 100644
--- a/tools/perf/util/evlist.h
+++ b/tools/perf/util/evlist.h
@@ -48,12 +48,15 @@ struct perf_evlist {
 	bool		 overwrite;
 	struct fdarray	 pollfd;
 	struct perf_mmap *mmap;
+	struct perf_mmap *track_mmap;
 	struct thread_map *threads;
 	struct cpu_map	  *cpus;
 	struct perf_evsel *selected;
 	struct events_stats stats;
 };
 
+#define TRACK_MMAP_SIZE  (((128 * 1024 / page_size) + 1) * page_size)
+
 struct perf_evsel_str_handler {
 	const char *name;
 	void	   *handler;
@@ -103,8 +106,8 @@ struct perf_evsel *perf_evlist__id2evsel(struct perf_evlist *evlist, u64 id);
 struct perf_sample_id *perf_evlist__id2sid(struct perf_evlist *evlist, u64 id);
 
 union perf_event *perf_evlist__mmap_read(struct perf_evlist *evlist, int idx);
-
 void perf_evlist__mmap_consume(struct perf_evlist *evlist, int idx);
+struct perf_mmap *perf_evlist__mmap_desc(struct perf_evlist *evlist, int idx);
 
 int perf_evlist__open(struct perf_evlist *evlist);
 void perf_evlist__close(struct perf_evlist *evlist);
@@ -214,6 +217,12 @@ bool perf_evlist__can_select_event(struct perf_evlist *evlist, const char *str);
 void perf_evlist__to_front(struct perf_evlist *evlist,
 			   struct perf_evsel *move_evsel);
 
+/* convert from/to negative idx for track mmaps */
+static inline int track_mmap_idx(int idx)
+{
+	return -idx - 1;
+}
+
 /**
  * __evlist__for_each - iterate thru all the evsels
  * @list: list_head instance to iterate
-- 
2.2.2


^ permalink raw reply related	[flat|nested] 221+ messages in thread

* [PATCH 05/38] perf tools: Introduce perf_evlist__mmap_track()
  2015-03-03  3:07 [RFC/PATCHSET 00/38] perf tools: Speed-up perf report by using multi thread (v3) Namhyung Kim
                   ` (3 preceding siblings ...)
  2015-03-03  3:07 ` [PATCH 04/38] perf tools: Create separate mmap for dummy tracking event Namhyung Kim
@ 2015-03-03  3:07 ` Namhyung Kim
  2015-03-03  3:07 ` [PATCH 06/38] perf tools: Add HEADER_DATA_INDEX feature Namhyung Kim
                   ` (32 subsequent siblings)
  37 siblings, 0 replies; 221+ messages in thread
From: Namhyung Kim @ 2015-03-03  3:07 UTC (permalink / raw)
  To: Arnaldo Carvalho de Melo
  Cc: Ingo Molnar, Peter Zijlstra, Jiri Olsa, LKML,
	Frederic Weisbecker, Adrian Hunter, Stephane Eranian, Andi Kleen,
	David Ahern

The perf_evlist__mmap_track function creates data mmaps and optionally
tracking mmaps for events.  It'll be used for perf record to save events
in a separate files and build an index table.  Checking dummy tracking
event in perf_evlist__mmap() alone is not enough as users can specify a
dummy event (like in keep tracking testcase) without the index option.

Cc: Adrian Hunter <adrian.hunter@intel.com>
Signed-off-by: Namhyung Kim <namhyung@kernel.org>
---
 tools/perf/builtin-record.c |  3 ++-
 tools/perf/util/evlist.c    | 15 +++++++++------
 tools/perf/util/evlist.h    | 10 ++++++++--
 3 files changed, 19 insertions(+), 9 deletions(-)

diff --git a/tools/perf/builtin-record.c b/tools/perf/builtin-record.c
index 2bd724763e1d..4568bc4117a1 100644
--- a/tools/perf/builtin-record.c
+++ b/tools/perf/builtin-record.c
@@ -169,7 +169,8 @@ static int record__open(struct record *rec)
 		goto out;
 	}
 
-	if (perf_evlist__mmap(evlist, opts->mmap_pages, false) < 0) {
+	if (perf_evlist__mmap_track(evlist, opts->mmap_pages, false,
+				    false) < 0) {
 		if (errno == EPERM) {
 			pr_err("Permission error mapping pages.\n"
 			       "Consider increasing "
diff --git a/tools/perf/util/evlist.c b/tools/perf/util/evlist.c
index ebbec07843a2..d264ba3602b1 100644
--- a/tools/perf/util/evlist.c
+++ b/tools/perf/util/evlist.c
@@ -834,6 +834,7 @@ static int perf_evlist__alloc_mmap(struct perf_evlist *evlist, bool track_mmap)
 
 struct mmap_params {
 	int	prot;
+	bool	track;
 	size_t	len;
 };
 
@@ -883,7 +884,7 @@ static int perf_evlist__mmap_per_evsel(struct perf_evlist *evlist, int idx,
 
 		fd = FD(evsel, cpu, thread);
 
-		if (perf_evsel__is_dummy_tracking(evsel)) {
+		if (mp->track && perf_evsel__is_dummy_tracking(evsel)) {
 			struct mmap_params track_mp = {
 				.prot	= mp->prot,
 				.len	= TRACK_MMAP_SIZE,
@@ -936,7 +937,7 @@ static int perf_evlist__mmap_per_evsel(struct perf_evlist *evlist, int idx,
 						 thread);
 		}
 
-		if (perf_evsel__is_dummy_tracking(evsel)) {
+		if (mp->track && perf_evsel__is_dummy_tracking(evsel)) {
 			/* restore idx as normal idx (positive) */
 			idx = track_mmap_idx(idx);
 		}
@@ -1087,10 +1088,11 @@ int perf_evlist__parse_mmap_pages(const struct option *opt, const char *str,
 }
 
 /**
- * perf_evlist__mmap - Create mmaps to receive events.
+ * perf_evlist__mmap_track - Create mmaps to receive events.
  * @evlist: list of events
  * @pages: map length in pages
  * @overwrite: overwrite older events?
+ * @use_track_mmap: use another mmaps to track meta events
  *
  * If @overwrite is %false the user needs to signal event consumption using
  * perf_mmap__write_tail().  Using perf_evlist__mmap_read() does this
@@ -1098,17 +1100,18 @@ int perf_evlist__parse_mmap_pages(const struct option *opt, const char *str,
  *
  * Return: %0 on success, negative error code otherwise.
  */
-int perf_evlist__mmap(struct perf_evlist *evlist, unsigned int pages,
-		      bool overwrite)
+int perf_evlist__mmap_track(struct perf_evlist *evlist, unsigned int pages,
+			    bool overwrite, bool use_track_mmap)
 {
 	struct perf_evsel *evsel;
 	const struct cpu_map *cpus = evlist->cpus;
 	const struct thread_map *threads = evlist->threads;
 	struct mmap_params mp = {
 		.prot = PROT_READ | (overwrite ? 0 : PROT_WRITE),
+		.track = use_track_mmap,
 	};
 
-	if (evlist->mmap == NULL && perf_evlist__alloc_mmap(evlist, true) < 0)
+	if (evlist->mmap == NULL && perf_evlist__alloc_mmap(evlist, mp.track) < 0)
 		return -ENOMEM;
 
 	if (evlist->pollfd.entries == NULL && perf_evlist__alloc_pollfd(evlist) < 0)
diff --git a/tools/perf/util/evlist.h b/tools/perf/util/evlist.h
index 3bd9747bb9aa..da45074ee500 100644
--- a/tools/perf/util/evlist.h
+++ b/tools/perf/util/evlist.h
@@ -130,10 +130,16 @@ int perf_evlist__parse_mmap_pages(const struct option *opt,
 				  const char *str,
 				  int unset);
 
-int perf_evlist__mmap(struct perf_evlist *evlist, unsigned int pages,
-		      bool overwrite);
+int perf_evlist__mmap_track(struct perf_evlist *evlist, unsigned int pages,
+			    bool overwrite, bool use_track_mmap);
 void perf_evlist__munmap(struct perf_evlist *evlist);
 
+static inline int perf_evlist__mmap(struct perf_evlist *evlist,
+				    unsigned int pages, bool overwrite)
+{
+	return perf_evlist__mmap_track(evlist, pages, overwrite, false);
+}
+
 void perf_evlist__disable(struct perf_evlist *evlist);
 void perf_evlist__enable(struct perf_evlist *evlist);
 
-- 
2.2.2


^ permalink raw reply related	[flat|nested] 221+ messages in thread

* [PATCH 06/38] perf tools: Add HEADER_DATA_INDEX feature
  2015-03-03  3:07 [RFC/PATCHSET 00/38] perf tools: Speed-up perf report by using multi thread (v3) Namhyung Kim
                   ` (4 preceding siblings ...)
  2015-03-03  3:07 ` [PATCH 05/38] perf tools: Introduce perf_evlist__mmap_track() Namhyung Kim
@ 2015-03-03  3:07 ` Namhyung Kim
  2015-03-03  3:07 ` [PATCH 07/38] perf tools: Handle indexed data file properly Namhyung Kim
                   ` (31 subsequent siblings)
  37 siblings, 0 replies; 221+ messages in thread
From: Namhyung Kim @ 2015-03-03  3:07 UTC (permalink / raw)
  To: Arnaldo Carvalho de Melo
  Cc: Ingo Molnar, Peter Zijlstra, Jiri Olsa, LKML,
	Frederic Weisbecker, Adrian Hunter, Stephane Eranian, Andi Kleen,
	David Ahern

The HEADER_DATA_INDEX feature is to record index table for sample data
so that they can be processed by multiple thread concurrently.  Each
item is a struct perf_file_section which consists of an offset and size.

Signed-off-by: Namhyung Kim <namhyung@kernel.org>
---
 tools/perf/builtin-record.c |  2 ++
 tools/perf/util/header.c    | 61 +++++++++++++++++++++++++++++++++++++++++++++
 tools/perf/util/header.h    |  3 +++
 3 files changed, 66 insertions(+)

diff --git a/tools/perf/builtin-record.c b/tools/perf/builtin-record.c
index 4568bc4117a1..1bdf7e4a0a6f 100644
--- a/tools/perf/builtin-record.c
+++ b/tools/perf/builtin-record.c
@@ -312,6 +312,8 @@ static void record__init_features(struct record *rec)
 
 	if (!rec->opts.branch_stack)
 		perf_header__clear_feat(&session->header, HEADER_BRANCH_STACK);
+
+	perf_header__clear_feat(&session->header, HEADER_DATA_INDEX);
 }
 
 static volatile int workload_exec_errno;
diff --git a/tools/perf/util/header.c b/tools/perf/util/header.c
index 1f407f7352a7..77206a6cbf65 100644
--- a/tools/perf/util/header.c
+++ b/tools/perf/util/header.c
@@ -869,6 +869,24 @@ static int write_branch_stack(int fd __maybe_unused,
 	return 0;
 }
 
+static int write_data_index(int fd, struct perf_header *h,
+			    struct perf_evlist *evlist __maybe_unused)
+{
+	int ret;
+	unsigned i;
+
+	ret = do_write(fd, &h->nr_index, sizeof(h->nr_index));
+	if (ret < 0)
+		return ret;
+
+	for (i = 0; i < h->nr_index; i++) {
+		ret = do_write(fd, &h->index[i], sizeof(*h->index));
+		if (ret < 0)
+			return ret;
+	}
+	return 0;
+}
+
 static void print_hostname(struct perf_header *ph, int fd __maybe_unused,
 			   FILE *fp)
 {
@@ -1225,6 +1243,12 @@ static void print_group_desc(struct perf_header *ph, int fd __maybe_unused,
 	}
 }
 
+static void print_data_index(struct perf_header *ph __maybe_unused,
+			     int fd __maybe_unused, FILE *fp)
+{
+	fprintf(fp, "# contains data index for parallel processing\n");
+}
+
 static int __event_process_build_id(struct build_id_event *bev,
 				    char *filename,
 				    struct perf_session *session)
@@ -1833,6 +1857,42 @@ static int process_group_desc(struct perf_file_section *section __maybe_unused,
 	return ret;
 }
 
+static int process_data_index(struct perf_file_section *section __maybe_unused,
+			      struct perf_header *ph, int fd,
+			      void *data __maybe_unused)
+{
+	ssize_t ret;
+	u64 nr_index;
+	unsigned i;
+	struct perf_file_section *index;
+
+	ret = readn(fd, &nr_index, sizeof(nr_index));
+	if (ret != sizeof(nr_index))
+		return -1;
+
+	if (ph->needs_swap)
+		nr_index = bswap_64(nr_index);
+
+	index = calloc(nr_index, sizeof(*index));
+	if (index == NULL)
+		return -1;
+
+	for (i = 0; i < nr_index; i++) {
+		ret = readn(fd, &index[i], sizeof(*index));
+		if (ret != sizeof(*index))
+			return ret;
+
+		if (ph->needs_swap) {
+			index[i].offset = bswap_64(index[i].offset);
+			index[i].size   = bswap_64(index[i].size);
+		}
+	}
+
+	ph->index = index;
+	ph->nr_index = nr_index;
+	return 0;
+}
+
 struct feature_ops {
 	int (*write)(int fd, struct perf_header *h, struct perf_evlist *evlist);
 	void (*print)(struct perf_header *h, int fd, FILE *fp);
@@ -1873,6 +1933,7 @@ static const struct feature_ops feat_ops[HEADER_LAST_FEATURE] = {
 	FEAT_OPA(HEADER_BRANCH_STACK,	branch_stack),
 	FEAT_OPP(HEADER_PMU_MAPPINGS,	pmu_mappings),
 	FEAT_OPP(HEADER_GROUP_DESC,	group_desc),
+	FEAT_OPP(HEADER_DATA_INDEX,	data_index),
 };
 
 struct header_print_data {
diff --git a/tools/perf/util/header.h b/tools/perf/util/header.h
index 3bb90ac172a1..e5594f0d6dcd 100644
--- a/tools/perf/util/header.h
+++ b/tools/perf/util/header.h
@@ -30,6 +30,7 @@ enum {
 	HEADER_BRANCH_STACK,
 	HEADER_PMU_MAPPINGS,
 	HEADER_GROUP_DESC,
+	HEADER_DATA_INDEX,
 	HEADER_LAST_FEATURE,
 	HEADER_FEAT_BITS	= 256,
 };
@@ -94,6 +95,8 @@ struct perf_header {
 	bool				needs_swap;
 	u64				data_offset;
 	u64				data_size;
+	struct perf_file_section	*index;
+	u64				nr_index;
 	u64				feat_offset;
 	DECLARE_BITMAP(adds_features, HEADER_FEAT_BITS);
 	struct perf_session_env 	env;
-- 
2.2.2


^ permalink raw reply related	[flat|nested] 221+ messages in thread

* [PATCH 07/38] perf tools: Handle indexed data file properly
  2015-03-03  3:07 [RFC/PATCHSET 00/38] perf tools: Speed-up perf report by using multi thread (v3) Namhyung Kim
                   ` (5 preceding siblings ...)
  2015-03-03  3:07 ` [PATCH 06/38] perf tools: Add HEADER_DATA_INDEX feature Namhyung Kim
@ 2015-03-03  3:07 ` Namhyung Kim
  2015-03-04 16:19   ` Jiri Olsa
  2015-03-03  3:07 ` [PATCH 08/38] perf record: Add --index option for building index table Namhyung Kim
                   ` (30 subsequent siblings)
  37 siblings, 1 reply; 221+ messages in thread
From: Namhyung Kim @ 2015-03-03  3:07 UTC (permalink / raw)
  To: Arnaldo Carvalho de Melo
  Cc: Ingo Molnar, Peter Zijlstra, Jiri Olsa, LKML,
	Frederic Weisbecker, Adrian Hunter, Stephane Eranian, Andi Kleen,
	David Ahern

When perf detects data file has index table, process header part first
and then rest data files in a row.  Note that the indexed sample data is
recorded for each cpu/thread separately, it's already ordered with
respect to themselves so no need to use the ordered event queue
interface.

Signed-off-by: Namhyung Kim <namhyung@kernel.org>
---
 tools/perf/util/session.c | 62 ++++++++++++++++++++++++++++++++++++++---------
 tools/perf/util/session.h |  5 ++++
 2 files changed, 55 insertions(+), 12 deletions(-)

diff --git a/tools/perf/util/session.c b/tools/perf/util/session.c
index e4f166981ff0..00cd1ad427be 100644
--- a/tools/perf/util/session.c
+++ b/tools/perf/util/session.c
@@ -1300,11 +1300,10 @@ fetch_mmaped_event(struct perf_session *session,
 #define NUM_MMAPS 128
 #endif
 
-static int __perf_session__process_events(struct perf_session *session,
+static int __perf_session__process_events(struct perf_session *session, int fd,
 					  u64 data_offset, u64 data_size,
 					  u64 file_size, struct perf_tool *tool)
 {
-	int fd = perf_data_file__fd(session->file);
 	u64 head, page_offset, file_offset, file_pos, size;
 	int err, mmap_prot, mmap_flags, map_idx = 0;
 	size_t	mmap_size;
@@ -1327,7 +1326,9 @@ static int __perf_session__process_events(struct perf_session *session,
 	mmap_size = MMAP_SIZE;
 	if (mmap_size > file_size) {
 		mmap_size = file_size;
-		session->one_mmap = true;
+
+		if (!perf_session__has_index(session))
+			session->one_mmap = true;
 	}
 
 	memset(mmaps, 0, sizeof(mmaps));
@@ -1400,29 +1401,66 @@ static int __perf_session__process_events(struct perf_session *session,
 	err = ordered_events__flush(session, tool, OE_FLUSH__FINAL);
 out_err:
 	ui_progress__finish();
-	perf_tool__warn_about_errors(tool, &session->evlist->stats);
 	ordered_events__free(&session->ordered_events);
 	session->one_mmap = false;
 	return err;
 }
 
+static int __perf_session__process_indexed_events(struct perf_session *session,
+						  struct perf_tool *tool)
+{
+	struct perf_data_file *file = session->file;
+	u64 size = perf_data_file__size(file);
+	int err = 0, i;
+
+	for (i = 0; i < (int)session->header.nr_index; i++) {
+		struct perf_file_section *index = &session->header.index[i];
+
+		if (!index->size)
+			continue;
+
+		/*
+		 * For indexed data file, samples are processed for
+		 * each cpu/thread so it's already ordered.  However
+		 * meta-events at index 0 should be processed in order.
+		 */
+		if (i > 0)
+			tool->ordered_events = false;
+
+		err = __perf_session__process_events(session,
+						     perf_data_file__fd(file),
+						     index->offset, index->size,
+						     size, tool);
+		if (err < 0)
+			break;
+	}
+
+	perf_tool__warn_about_errors(tool, &session->evlist->stats);
+	return err;
+}
+
 int perf_session__process_events(struct perf_session *session,
 				 struct perf_tool *tool)
 {
-	u64 size = perf_data_file__size(session->file);
+	struct perf_data_file *file = session->file;
+	u64 size = perf_data_file__size(file);
 	int err;
 
 	if (perf_session__register_idle_thread(session) == NULL)
 		return -ENOMEM;
 
-	if (!perf_data_file__is_pipe(session->file))
-		err = __perf_session__process_events(session,
-						     session->header.data_offset,
-						     session->header.data_size,
-						     size, tool);
-	else
-		err = __perf_session__process_pipe_events(session, tool);
+	if (perf_data_file__is_pipe(file))
+		return __perf_session__process_pipe_events(session, tool);
+	if (perf_session__has_index(session))
+		return __perf_session__process_indexed_events(session, tool);
+
+	err = __perf_session__process_events(session,
+					     perf_data_file__fd(file),
+					     session->header.data_offset,
+					     session->header.data_size,
+					     size, tool);
 
+	perf_tool__warn_about_errors(tool, &session->evlist->stats);
 	return err;
 }
 
diff --git a/tools/perf/util/session.h b/tools/perf/util/session.h
index fe859f379ca7..aff0d2b4cc0b 100644
--- a/tools/perf/util/session.h
+++ b/tools/perf/util/session.h
@@ -137,4 +137,9 @@ int perf_event__synthesize_id_index(struct perf_tool *tool,
 				    struct perf_evlist *evlist,
 				    struct machine *machine);
 
+static inline bool perf_session__has_index(struct perf_session *session)
+{
+	return perf_header__has_feat(&session->header, HEADER_DATA_INDEX);
+}
+
 #endif /* __PERF_SESSION_H */
-- 
2.2.2


^ permalink raw reply related	[flat|nested] 221+ messages in thread

* [PATCH 08/38] perf record: Add --index option for building index table
  2015-03-03  3:07 [RFC/PATCHSET 00/38] perf tools: Speed-up perf report by using multi thread (v3) Namhyung Kim
                   ` (6 preceding siblings ...)
  2015-03-03  3:07 ` [PATCH 07/38] perf tools: Handle indexed data file properly Namhyung Kim
@ 2015-03-03  3:07 ` Namhyung Kim
  2015-03-05  7:56   ` Jiri Olsa
  2015-03-03  3:07 ` [PATCH 09/38] perf report: Skip dummy tracking event Namhyung Kim
                   ` (29 subsequent siblings)
  37 siblings, 1 reply; 221+ messages in thread
From: Namhyung Kim @ 2015-03-03  3:07 UTC (permalink / raw)
  To: Arnaldo Carvalho de Melo
  Cc: Ingo Molnar, Peter Zijlstra, Jiri Olsa, LKML,
	Frederic Weisbecker, Adrian Hunter, Stephane Eranian, Andi Kleen,
	David Ahern

The new --index option will create indexed data file which can be
processed by multiple threads parallelly.  It saves meta event and
sample data in separate files and merges them with an index table.

If there's an index table in the data file, the HEADER_DATA_INDEX
feature bit is set and session->header.index[0] will point to the meta
event area, and rest are sample data.  It'd look like below:

        +---------------------+
        |     file header     |
        |---------------------|
        |                     |
        |    meta events[0] <-+--+
        |                     |  |
        |---------------------|  |
        |                     |  |
        |    sample data[1] <-+--+
        |                     |  |
        |---------------------|  |
        |                     |  |
        |    sample data[2] <-|--+
        |                     |  |
        |---------------------|  |
        |         ...         | ...
        |---------------------|  |
        |     feature data    |  |
        |   (contains index) -+--+
        +---------------------+

Signed-off-by: Namhyung Kim <namhyung@kernel.org>
---
 tools/perf/Documentation/perf-record.txt |   4 +
 tools/perf/builtin-record.c              | 161 +++++++++++++++++++++++++++++--
 tools/perf/perf.h                        |   1 +
 tools/perf/util/session.c                |   1 +
 4 files changed, 158 insertions(+), 9 deletions(-)

diff --git a/tools/perf/Documentation/perf-record.txt b/tools/perf/Documentation/perf-record.txt
index 355c4f5569b5..5476432c045f 100644
--- a/tools/perf/Documentation/perf-record.txt
+++ b/tools/perf/Documentation/perf-record.txt
@@ -250,6 +250,10 @@ is off by default.
 --running-time::
 Record running and enabled time for read events (:S)
 
+--index::
+Build an index table for sample data.  This will speed up perf report by
+parallel processing.
+
 SEE ALSO
 --------
 linkperf:perf-stat[1], linkperf:perf-list[1]
diff --git a/tools/perf/builtin-record.c b/tools/perf/builtin-record.c
index 1bdf7e4a0a6f..ecf8e7293015 100644
--- a/tools/perf/builtin-record.c
+++ b/tools/perf/builtin-record.c
@@ -38,6 +38,7 @@ struct record {
 	struct record_opts	opts;
 	u64			bytes_written;
 	struct perf_data_file	file;
+	int			*fds;
 	struct perf_evlist	*evlist;
 	struct perf_session	*session;
 	const char		*progname;
@@ -47,9 +48,16 @@ struct record {
 	long			samples;
 };
 
-static int record__write(struct record *rec, void *bf, size_t size)
+static int record__write(struct record *rec, void *bf, size_t size, int idx)
 {
-	if (perf_data_file__write(rec->session->file, bf, size) < 0) {
+	int fd;
+
+	if (rec->fds && idx >= 0)
+		fd = rec->fds[idx];
+	else
+		fd = perf_data_file__fd(rec->session->file);
+
+	if (writen(fd, bf, size) < 0) {
 		pr_err("failed to write perf data, error: %m\n");
 		return -1;
 	}
@@ -64,7 +72,7 @@ static int process_synthesized_event(struct perf_tool *tool,
 				     struct machine *machine __maybe_unused)
 {
 	struct record *rec = container_of(tool, struct record, tool);
-	return record__write(rec, event, event->header.size);
+	return record__write(rec, event, event->header.size, -1);
 }
 
 static int record__mmap_read(struct record *rec, int idx)
@@ -89,7 +97,7 @@ static int record__mmap_read(struct record *rec, int idx)
 		size = md->mask + 1 - (old & md->mask);
 		old += size;
 
-		if (record__write(rec, buf, size) < 0) {
+		if (record__write(rec, buf, size, idx) < 0) {
 			rc = -1;
 			goto out;
 		}
@@ -99,7 +107,7 @@ static int record__mmap_read(struct record *rec, int idx)
 	size = head - old;
 	old += size;
 
-	if (record__write(rec, buf, size) < 0) {
+	if (record__write(rec, buf, size, idx) < 0) {
 		rc = -1;
 		goto out;
 	}
@@ -111,6 +119,113 @@ static int record__mmap_read(struct record *rec, int idx)
 	return rc;
 }
 
+#define INDEX_FILE_FMT  "%s.dir/perf.data.%d"
+
+static int record__create_index_files(struct record *rec, int nr_index)
+{
+	int i = 0;
+	int ret = -1;
+	char path[PATH_MAX];
+	struct perf_data_file *file = &rec->file;
+
+	/* +1 for header file itself */
+	nr_index++;
+
+	rec->fds = malloc(nr_index * sizeof(int));
+	if (rec->fds == NULL)
+		return -ENOMEM;
+
+	scnprintf(path, sizeof(path), "%s.dir", file->path);
+	if (mkdir(path, S_IRWXU) < 0)
+		goto out_err;
+
+	rec->fds[0] = perf_data_file__fd(file);
+
+	for (i = 1; i < nr_index; i++) {
+		scnprintf(path, sizeof(path), INDEX_FILE_FMT, file->path, i);
+		ret = open(path, O_RDWR|O_CREAT|O_TRUNC, S_IRUSR|S_IWUSR);
+		if (ret < 0)
+			goto out_err;
+
+		rec->fds[i] = ret;
+	}
+	return 0;
+
+out_err:
+	while (--i >= 1)
+		close(rec->fds[i]);
+	zfree(&rec->fds);
+
+	scnprintf(path, sizeof(path), "%s.dir", file->path);
+	rm_rf(path);
+
+	return ret;
+}
+
+static int record__merge_index_files(struct record *rec, int nr_index)
+{
+	int i;
+	int ret = -1;
+	u64 offset;
+	char path[PATH_MAX];
+	struct perf_file_section *idx;
+	struct perf_data_file *file = &rec->file;
+	struct perf_session *session = rec->session;
+	int output_fd = perf_data_file__fd(file);
+
+	/* +1 for header file itself */
+	nr_index++;
+
+	idx = calloc(nr_index, sizeof(*idx));
+	if (idx == NULL)
+		goto out_close;
+
+	offset = lseek(output_fd, 0, SEEK_END);
+
+	idx[0].offset = session->header.data_offset;
+	idx[0].size   = offset - idx[0].offset;
+
+	for (i = 1; i < nr_index; i++) {
+		struct stat stbuf;
+		int fd = rec->fds[i];
+
+		if (fstat(fd, &stbuf) < 0)
+			goto out_close;
+
+		idx[i].offset = offset;
+		idx[i].size   = stbuf.st_size;
+
+		offset += stbuf.st_size;
+	}
+
+	/* copy sample events */
+	for (i = 1; i < nr_index; i++) {
+		int fd = rec->fds[i];
+
+		if (idx[i].size == 0)
+			continue;
+
+		if (copyfile_offset(fd, 0, output_fd, idx[i].offset,
+				    idx[i].size) < 0)
+			goto out_close;
+	}
+
+	session->header.index = idx;
+	session->header.nr_index = nr_index;
+
+	ret = 0;
+
+out_close:
+	for (i = 1; i < nr_index; i++)
+		close(rec->fds[i]);
+
+	scnprintf(path, sizeof(path), "%s.dir", file->path);
+	rm_rf(path);
+
+	zfree(&rec->fds);
+	return ret;
+}
+
 static volatile int done = 0;
 static volatile int signr = -1;
 static volatile int child_finished = 0;
@@ -170,7 +285,7 @@ static int record__open(struct record *rec)
 	}
 
 	if (perf_evlist__mmap_track(evlist, opts->mmap_pages, false,
-				    false) < 0) {
+				    opts->index) < 0) {
 		if (errno == EPERM) {
 			pr_err("Permission error mapping pages.\n"
 			       "Consider increasing "
@@ -186,6 +301,12 @@ static int record__open(struct record *rec)
 		goto out;
 	}
 
+	if (opts->index) {
+		rc = record__create_index_files(rec, evlist->nr_mmaps);
+		if (rc < 0)
+			goto out;
+	}
+
 	session->evlist = evlist;
 	perf_session__set_id_hdr_size(session);
 out:
@@ -210,7 +331,8 @@ static int process_buildids(struct record *rec)
 	struct perf_data_file *file  = &rec->file;
 	struct perf_session *session = rec->session;
 
-	u64 size = lseek(perf_data_file__fd(file), 0, SEEK_CUR);
+	/* update file size after merging sample files with index */
+	u64 size = lseek(perf_data_file__fd(file), 0, SEEK_END);
 	if (size == 0)
 		return 0;
 
@@ -290,7 +412,8 @@ static int record__mmap_read_all(struct record *rec)
 	 * at least one event.
 	 */
 	if (bytes_written != rec->bytes_written)
-		rc = record__write(rec, &finished_round_event, sizeof(finished_round_event));
+		rc = record__write(rec, &finished_round_event,
+				   sizeof(finished_round_event), -1);
 
 out:
 	return rc;
@@ -313,7 +436,8 @@ static void record__init_features(struct record *rec)
 	if (!rec->opts.branch_stack)
 		perf_header__clear_feat(&session->header, HEADER_BRANCH_STACK);
 
-	perf_header__clear_feat(&session->header, HEADER_DATA_INDEX);
+	if (!rec->opts.index)
+		perf_header__clear_feat(&session->header, HEADER_DATA_INDEX);
 }
 
 static volatile int workload_exec_errno;
@@ -375,6 +499,11 @@ static int __cmd_record(struct record *rec, int argc, const char **argv)
 		}
 	}
 
+	if (file->is_pipe && opts->index) {
+		pr_warning("Indexing is disabled for pipe output\n");
+		opts->index = false;
+	}
+
 	if (record__open(rec) != 0) {
 		err = -1;
 		goto out_child;
@@ -554,6 +683,9 @@ static int __cmd_record(struct record *rec, int argc, const char **argv)
 	if (!err && !file->is_pipe) {
 		rec->session->header.data_size += rec->bytes_written;
 
+		if (rec->opts.index)
+			record__merge_index_files(rec, rec->evlist->nr_mmaps);
+
 		if (!rec->no_buildid)
 			process_buildids(rec);
 		perf_session__write_header(rec->session, rec->evlist, fd, true);
@@ -851,6 +983,8 @@ struct option __record_options[] = {
 		    "Sample machine registers on interrupt"),
 	OPT_BOOLEAN(0, "running-time", &record.opts.running_time,
 		    "Record running/enabled time of read (:S) events"),
+	OPT_BOOLEAN(0, "index", &record.opts.index,
+		    "make index for sample data to speed-up processing"),
 	OPT_END()
 };
 
@@ -900,6 +1034,15 @@ int cmd_record(int argc, const char **argv, const char *prefix __maybe_unused)
 		goto out_symbol_exit;
 	}
 
+	if (rec->opts.index) {
+		if (!rec->opts.sample_time) {
+			pr_err("Sample timestamp is required for indexing\n");
+			goto out_symbol_exit;
+		}
+
+		perf_evlist__add_dummy_tracking(rec->evlist);
+	}
+
 	if (rec->opts.target.tid && !rec->opts.no_inherit_set)
 		rec->opts.no_inherit = true;
 
diff --git a/tools/perf/perf.h b/tools/perf/perf.h
index 1caa70a4a9e1..a03552849399 100644
--- a/tools/perf/perf.h
+++ b/tools/perf/perf.h
@@ -54,6 +54,7 @@ struct record_opts {
 	bool	     period;
 	bool	     sample_intr_regs;
 	bool	     running_time;
+	bool	     index;
 	unsigned int freq;
 	unsigned int mmap_pages;
 	unsigned int user_freq;
diff --git a/tools/perf/util/session.c b/tools/perf/util/session.c
index 00cd1ad427be..46761a39fbae 100644
--- a/tools/perf/util/session.c
+++ b/tools/perf/util/session.c
@@ -173,6 +173,7 @@ void perf_session__delete(struct perf_session *session)
 	machines__exit(&session->machines);
 	if (session->file)
 		perf_data_file__close(session->file);
+	free(session->header.index);
 	free(session);
 }
 
-- 
2.2.2


^ permalink raw reply related	[flat|nested] 221+ messages in thread

* [PATCH 09/38] perf report: Skip dummy tracking event
  2015-03-03  3:07 [RFC/PATCHSET 00/38] perf tools: Speed-up perf report by using multi thread (v3) Namhyung Kim
                   ` (7 preceding siblings ...)
  2015-03-03  3:07 ` [PATCH 08/38] perf record: Add --index option for building index table Namhyung Kim
@ 2015-03-03  3:07 ` Namhyung Kim
  2015-03-03  3:07 ` [PATCH 10/38] perf tools: Pass session arg to perf_event__preprocess_sample() Namhyung Kim
                   ` (28 subsequent siblings)
  37 siblings, 0 replies; 221+ messages in thread
From: Namhyung Kim @ 2015-03-03  3:07 UTC (permalink / raw)
  To: Arnaldo Carvalho de Melo
  Cc: Ingo Molnar, Peter Zijlstra, Jiri Olsa, LKML,
	Frederic Weisbecker, Adrian Hunter, Stephane Eranian, Andi Kleen,
	David Ahern

The dummy tracking event is only for tracking task/comom/mmap events
and has no sample data for itself.  So no need to report, just skip it.

Signed-off-by: Namhyung Kim <namhyung@kernel.org>
---
 tools/perf/builtin-report.c    |  3 +++
 tools/perf/ui/browsers/hists.c | 30 ++++++++++++++++++++++++------
 tools/perf/ui/gtk/hists.c      |  3 +++
 3 files changed, 30 insertions(+), 6 deletions(-)

diff --git a/tools/perf/builtin-report.c b/tools/perf/builtin-report.c
index fb350343b1d7..7d132e1e2af9 100644
--- a/tools/perf/builtin-report.c
+++ b/tools/perf/builtin-report.c
@@ -320,6 +320,9 @@ static int perf_evlist__tty_browse_hists(struct perf_evlist *evlist,
 		struct hists *hists = evsel__hists(pos);
 		const char *evname = perf_evsel__name(pos);
 
+		if (perf_evsel__is_dummy_tracking(pos))
+			continue;
+
 		if (symbol_conf.event_group &&
 		    !perf_evsel__is_group_leader(pos))
 			continue;
diff --git a/tools/perf/ui/browsers/hists.c b/tools/perf/ui/browsers/hists.c
index 788506eef567..7d33d7dc0824 100644
--- a/tools/perf/ui/browsers/hists.c
+++ b/tools/perf/ui/browsers/hists.c
@@ -1947,14 +1947,17 @@ static int perf_evsel_menu__run(struct perf_evsel_menu *menu,
 	return key;
 }
 
-static bool filter_group_entries(struct ui_browser *browser __maybe_unused,
-				 void *entry)
+static bool filter_entries(struct ui_browser *browser __maybe_unused,
+			   void *entry)
 {
 	struct perf_evsel *evsel = list_entry(entry, struct perf_evsel, node);
 
 	if (symbol_conf.event_group && !perf_evsel__is_group_leader(evsel))
 		return true;
 
+	if (perf_evsel__is_dummy_tracking(evsel))
+		return true;
+
 	return false;
 }
 
@@ -1971,7 +1974,7 @@ static int __perf_evlist__tui_browse_hists(struct perf_evlist *evlist,
 			.refresh    = ui_browser__list_head_refresh,
 			.seek	    = ui_browser__list_head_seek,
 			.write	    = perf_evsel_menu__write,
-			.filter	    = filter_group_entries,
+			.filter	    = filter_entries,
 			.nr_entries = nr_entries,
 			.priv	    = evlist,
 		},
@@ -1998,21 +2001,22 @@ int perf_evlist__tui_browse_hists(struct perf_evlist *evlist, const char *help,
 				  struct perf_session_env *env)
 {
 	int nr_entries = evlist->nr_entries;
+	struct perf_evsel *first = perf_evlist__first(evlist);
+	struct perf_evsel *pos;
 
 single_entry:
 	if (nr_entries == 1) {
-		struct perf_evsel *first = perf_evlist__first(evlist);
-
 		return perf_evsel__hists_browse(first, nr_entries, help,
 						false, hbt, min_pcnt,
 						env);
 	}
 
 	if (symbol_conf.event_group) {
-		struct perf_evsel *pos;
 
 		nr_entries = 0;
 		evlist__for_each(evlist, pos) {
+			if (perf_evsel__is_dummy_tracking(pos))
+				continue;
 			if (perf_evsel__is_group_leader(pos))
 				nr_entries++;
 		}
@@ -2021,6 +2025,20 @@ int perf_evlist__tui_browse_hists(struct perf_evlist *evlist, const char *help,
 			goto single_entry;
 	}
 
+	evlist__for_each(evlist, pos) {
+		if (perf_evsel__is_dummy_tracking(pos))
+			nr_entries--;
+	}
+
+	if (nr_entries == 1) {
+		evlist__for_each(evlist, pos) {
+			if (!perf_evsel__is_dummy_tracking(pos)) {
+				first = pos;
+				goto single_entry;
+			}
+		}
+	}
+
 	return __perf_evlist__tui_browse_hists(evlist, nr_entries, help,
 					       hbt, min_pcnt, env);
 }
diff --git a/tools/perf/ui/gtk/hists.c b/tools/perf/ui/gtk/hists.c
index 4b3585eed1e8..83a7ecd5cda8 100644
--- a/tools/perf/ui/gtk/hists.c
+++ b/tools/perf/ui/gtk/hists.c
@@ -317,6 +317,9 @@ int perf_evlist__gtk_browse_hists(struct perf_evlist *evlist,
 		char buf[512];
 		size_t size = sizeof(buf);
 
+		if (perf_evsel__is_dummy_tracking(pos))
+			continue;
+
 		if (symbol_conf.event_group) {
 			if (!perf_evsel__is_group_leader(pos))
 				continue;
-- 
2.2.2


^ permalink raw reply related	[flat|nested] 221+ messages in thread

* [PATCH 10/38] perf tools: Pass session arg to perf_event__preprocess_sample()
  2015-03-03  3:07 [RFC/PATCHSET 00/38] perf tools: Speed-up perf report by using multi thread (v3) Namhyung Kim
                   ` (8 preceding siblings ...)
  2015-03-03  3:07 ` [PATCH 09/38] perf report: Skip dummy tracking event Namhyung Kim
@ 2015-03-03  3:07 ` Namhyung Kim
  2015-03-03 13:59   ` Arnaldo Carvalho de Melo
  2015-03-03  3:07 ` [PATCH 11/38] perf script: Pass session arg to ->process_event callback Namhyung Kim
                   ` (27 subsequent siblings)
  37 siblings, 1 reply; 221+ messages in thread
From: Namhyung Kim @ 2015-03-03  3:07 UTC (permalink / raw)
  To: Arnaldo Carvalho de Melo
  Cc: Ingo Molnar, Peter Zijlstra, Jiri Olsa, LKML,
	Frederic Weisbecker, Adrian Hunter, Stephane Eranian, Andi Kleen,
	David Ahern

The perf_event__preprocess_sample() translates a given ip into a
matching symbol.  To do that, it first finds a corresponding thread
and map in the current thread tree.  But for indexed data files, it
needs to find a thread (and map) with slightly different APIs using
timestamp.  So it needs a way to know whether this session deals with
an indexed data file or not.

Signed-off-by: Namhyung Kim <namhyung@kernel.org>
---
 tools/perf/builtin-annotate.c     |  3 ++-
 tools/perf/builtin-diff.c         | 13 +++++++++----
 tools/perf/builtin-mem.c          |  6 +++++-
 tools/perf/builtin-report.c       |  3 ++-
 tools/perf/builtin-script.c       | 20 +++++++++++---------
 tools/perf/builtin-timechart.c    | 10 +++++++---
 tools/perf/builtin-top.c          |  3 ++-
 tools/perf/tests/hists_cumulate.c |  2 +-
 tools/perf/tests/hists_filter.c   |  2 +-
 tools/perf/tests/hists_link.c     |  4 ++--
 tools/perf/tests/hists_output.c   |  2 +-
 tools/perf/util/event.c           |  3 ++-
 tools/perf/util/event.h           |  4 +++-
 13 files changed, 48 insertions(+), 27 deletions(-)

diff --git a/tools/perf/builtin-annotate.c b/tools/perf/builtin-annotate.c
index 747f86103599..b89e4c6ed488 100644
--- a/tools/perf/builtin-annotate.c
+++ b/tools/perf/builtin-annotate.c
@@ -85,7 +85,8 @@ static int process_sample_event(struct perf_tool *tool,
 	struct perf_annotate *ann = container_of(tool, struct perf_annotate, tool);
 	struct addr_location al;
 
-	if (perf_event__preprocess_sample(event, machine, &al, sample) < 0) {
+	if (perf_event__preprocess_sample(event, machine, &al, sample,
+					  ann->session) < 0) {
 		pr_warning("problem processing %d event, skipping it.\n",
 			   event->header.type);
 		return -1;
diff --git a/tools/perf/builtin-diff.c b/tools/perf/builtin-diff.c
index 74aada554b12..3e2229227062 100644
--- a/tools/perf/builtin-diff.c
+++ b/tools/perf/builtin-diff.c
@@ -42,6 +42,7 @@ struct diff_hpp_fmt {
 };
 
 struct data__file {
+	struct perf_tool	tool;
 	struct perf_session	*session;
 	struct perf_data_file	file;
 	int			 idx;
@@ -320,16 +321,18 @@ static int hists__add_entry(struct hists *hists,
 	return -ENOMEM;
 }
 
-static int diff__process_sample_event(struct perf_tool *tool __maybe_unused,
+static int diff__process_sample_event(struct perf_tool *tool,
 				      union perf_event *event,
 				      struct perf_sample *sample,
 				      struct perf_evsel *evsel,
 				      struct machine *machine)
 {
 	struct addr_location al;
+	struct data__file *d = container_of(tool, struct data__file, tool);
 	struct hists *hists = evsel__hists(evsel);
 
-	if (perf_event__preprocess_sample(event, machine, &al, sample) < 0) {
+	if (perf_event__preprocess_sample(event, machine, &al, sample,
+					  d->session) < 0) {
 		pr_warning("problem processing %d event, skipping it.\n",
 			   event->header.type);
 		return -1;
@@ -740,14 +743,16 @@ static int __cmd_diff(void)
 	int ret = -EINVAL, i;
 
 	data__for_each_file(i, d) {
-		d->session = perf_session__new(&d->file, false, &tool);
+		memcpy(&d->tool, &tool, sizeof(tool));
+
+		d->session = perf_session__new(&d->file, false, &d->tool);
 		if (!d->session) {
 			pr_err("Failed to open %s\n", d->file.path);
 			ret = -1;
 			goto out_delete;
 		}
 
-		ret = perf_session__process_events(d->session, &tool);
+		ret = perf_session__process_events(d->session, &d->tool);
 		if (ret) {
 			pr_err("Failed to process %s\n", d->file.path);
 			goto out_delete;
diff --git a/tools/perf/builtin-mem.c b/tools/perf/builtin-mem.c
index 9b5663950a4d..21d46918860e 100644
--- a/tools/perf/builtin-mem.c
+++ b/tools/perf/builtin-mem.c
@@ -12,6 +12,7 @@
 
 struct perf_mem {
 	struct perf_tool	tool;
+	struct perf_session	*session;
 	char const		*input_name;
 	bool			hide_unresolved;
 	bool			dump_raw;
@@ -66,7 +67,8 @@ dump_raw_samples(struct perf_tool *tool,
 	struct addr_location al;
 	const char *fmt;
 
-	if (perf_event__preprocess_sample(event, machine, &al, sample) < 0) {
+	if (perf_event__preprocess_sample(event, machine, &al, sample,
+					  mem->session) < 0) {
 		fprintf(stderr, "problem processing %d event, skipping it.\n",
 				event->header.type);
 		return -1;
@@ -129,6 +131,8 @@ static int report_raw_events(struct perf_mem *mem)
 	if (session == NULL)
 		return -1;
 
+	mem->session = session;
+
 	if (mem->cpu_list) {
 		ret = perf_session__cpu_bitmap(session, mem->cpu_list,
 					       mem->cpu_bitmap);
diff --git a/tools/perf/builtin-report.c b/tools/perf/builtin-report.c
index 7d132e1e2af9..fe1f34c00c58 100644
--- a/tools/perf/builtin-report.c
+++ b/tools/perf/builtin-report.c
@@ -142,7 +142,8 @@ static int process_sample_event(struct perf_tool *tool,
 	};
 	int ret;
 
-	if (perf_event__preprocess_sample(event, machine, &al, sample) < 0) {
+	if (perf_event__preprocess_sample(event, machine, &al, sample,
+					  rep->session) < 0) {
 		pr_debug("problem processing %d event, skipping it.\n",
 			 event->header.type);
 		return -1;
diff --git a/tools/perf/builtin-script.c b/tools/perf/builtin-script.c
index ce304dfd962a..ab920f8cded6 100644
--- a/tools/perf/builtin-script.c
+++ b/tools/perf/builtin-script.c
@@ -542,13 +542,21 @@ static int cleanup_scripting(void)
 	return scripting_ops->stop_script();
 }
 
-static int process_sample_event(struct perf_tool *tool __maybe_unused,
+struct perf_script {
+	struct perf_tool	tool;
+	struct perf_session	*session;
+	bool			show_task_events;
+	bool			show_mmap_events;
+};
+
+static int process_sample_event(struct perf_tool *tool,
 				union perf_event *event,
 				struct perf_sample *sample,
 				struct perf_evsel *evsel,
 				struct machine *machine)
 {
 	struct addr_location al;
+	struct perf_script *script = container_of(tool, struct perf_script, tool);
 	struct thread *thread = machine__findnew_thread(machine, sample->pid,
 							sample->tid);
 
@@ -569,7 +577,8 @@ static int process_sample_event(struct perf_tool *tool __maybe_unused,
 		return 0;
 	}
 
-	if (perf_event__preprocess_sample(event, machine, &al, sample) < 0) {
+	if (perf_event__preprocess_sample(event, machine, &al, sample,
+					  script->session) < 0) {
 		pr_err("problem processing %d event, skipping it.\n",
 		       event->header.type);
 		return -1;
@@ -586,13 +595,6 @@ static int process_sample_event(struct perf_tool *tool __maybe_unused,
 	return 0;
 }
 
-struct perf_script {
-	struct perf_tool	tool;
-	struct perf_session	*session;
-	bool			show_task_events;
-	bool			show_mmap_events;
-};
-
 static int process_attr(struct perf_tool *tool, union perf_event *event,
 			struct perf_evlist **pevlist)
 {
diff --git a/tools/perf/builtin-timechart.c b/tools/perf/builtin-timechart.c
index f3bb1a4bf060..4178727be12c 100644
--- a/tools/perf/builtin-timechart.c
+++ b/tools/perf/builtin-timechart.c
@@ -48,6 +48,7 @@ struct wake_event;
 
 struct timechart {
 	struct perf_tool	tool;
+	struct perf_session	*session;
 	struct per_pid		*all_data;
 	struct power_event	*power_events;
 	struct wake_event	*wake_events;
@@ -469,7 +470,8 @@ static void sched_switch(struct timechart *tchart, int cpu, u64 timestamp,
 
 static const char *cat_backtrace(union perf_event *event,
 				 struct perf_sample *sample,
-				 struct machine *machine)
+				 struct machine *machine,
+				 struct perf_session *session)
 {
 	struct addr_location al;
 	unsigned int i;
@@ -488,7 +490,8 @@ static const char *cat_backtrace(union perf_event *event,
 	if (!chain)
 		goto exit;
 
-	if (perf_event__preprocess_sample(event, machine, &al, sample) < 0) {
+	if (perf_event__preprocess_sample(event, machine, &al, sample,
+					  session) < 0) {
 		fprintf(stderr, "problem processing %d event, skipping it.\n",
 			event->header.type);
 		goto exit;
@@ -567,7 +570,7 @@ static int process_sample_event(struct perf_tool *tool,
 	if (evsel->handler != NULL) {
 		tracepoint_handler f = evsel->handler;
 		return f(tchart, evsel, sample,
-			 cat_backtrace(event, sample, machine));
+			 cat_backtrace(event, sample, machine, tchart->session));
 	}
 
 	return 0;
@@ -1623,6 +1626,7 @@ static int __cmd_timechart(struct timechart *tchart, const char *output_name)
 		goto out_delete;
 	}
 
+	tchart->session = session;
 	ret = perf_session__process_events(session, &tchart->tool);
 	if (ret)
 		goto out_delete;
diff --git a/tools/perf/builtin-top.c b/tools/perf/builtin-top.c
index 5fb8723c7128..054c56206481 100644
--- a/tools/perf/builtin-top.c
+++ b/tools/perf/builtin-top.c
@@ -723,7 +723,8 @@ static void perf_event__process_sample(struct perf_tool *tool,
 	if (event->header.misc & PERF_RECORD_MISC_EXACT_IP)
 		top->exact_samples++;
 
-	if (perf_event__preprocess_sample(event, machine, &al, sample) < 0)
+	if (perf_event__preprocess_sample(event, machine, &al, sample,
+					  top->session) < 0)
 		return;
 
 	if (!top->kptr_restrict_warned &&
diff --git a/tools/perf/tests/hists_cumulate.c b/tools/perf/tests/hists_cumulate.c
index 18619966454c..60682e62d9de 100644
--- a/tools/perf/tests/hists_cumulate.c
+++ b/tools/perf/tests/hists_cumulate.c
@@ -101,7 +101,7 @@ static int add_hist_entries(struct hists *hists, struct machine *machine)
 		sample.callchain = (struct ip_callchain *)fake_callchains[i];
 
 		if (perf_event__preprocess_sample(&event, machine, &al,
-						  &sample) < 0)
+						  &sample, NULL) < 0)
 			goto out;
 
 		if (hist_entry_iter__add(&iter, &al, evsel, &sample,
diff --git a/tools/perf/tests/hists_filter.c b/tools/perf/tests/hists_filter.c
index 59e53db7914c..1c4e495d5137 100644
--- a/tools/perf/tests/hists_filter.c
+++ b/tools/perf/tests/hists_filter.c
@@ -78,7 +78,7 @@ static int add_hist_entries(struct perf_evlist *evlist,
 			sample.ip = fake_samples[i].ip;
 
 			if (perf_event__preprocess_sample(&event, machine, &al,
-							  &sample) < 0)
+							  &sample, NULL) < 0)
 				goto out;
 
 			if (hist_entry_iter__add(&iter, &al, evsel, &sample,
diff --git a/tools/perf/tests/hists_link.c b/tools/perf/tests/hists_link.c
index 278ba8344c23..a731a531a3e2 100644
--- a/tools/perf/tests/hists_link.c
+++ b/tools/perf/tests/hists_link.c
@@ -86,7 +86,7 @@ static int add_hist_entries(struct perf_evlist *evlist, struct machine *machine)
 			sample.tid = fake_common_samples[k].pid;
 			sample.ip = fake_common_samples[k].ip;
 			if (perf_event__preprocess_sample(&event, machine, &al,
-							  &sample) < 0)
+							  &sample, NULL) < 0)
 				goto out;
 
 			he = __hists__add_entry(hists, &al, NULL,
@@ -110,7 +110,7 @@ static int add_hist_entries(struct perf_evlist *evlist, struct machine *machine)
 			sample.tid = fake_samples[i][k].pid;
 			sample.ip = fake_samples[i][k].ip;
 			if (perf_event__preprocess_sample(&event, machine, &al,
-							  &sample) < 0)
+							  &sample, NULL) < 0)
 				goto out;
 
 			he = __hists__add_entry(hists, &al, NULL,
diff --git a/tools/perf/tests/hists_output.c b/tools/perf/tests/hists_output.c
index b52c9faea224..f4e3286cd496 100644
--- a/tools/perf/tests/hists_output.c
+++ b/tools/perf/tests/hists_output.c
@@ -67,7 +67,7 @@ static int add_hist_entries(struct hists *hists, struct machine *machine)
 		sample.ip = fake_samples[i].ip;
 
 		if (perf_event__preprocess_sample(&event, machine, &al,
-						  &sample) < 0)
+						  &sample, NULL) < 0)
 			goto out;
 
 		if (hist_entry_iter__add(&iter, &al, evsel, &sample,
diff --git a/tools/perf/util/event.c b/tools/perf/util/event.c
index d5efa5092ce6..704ef27cc7c8 100644
--- a/tools/perf/util/event.c
+++ b/tools/perf/util/event.c
@@ -836,7 +836,8 @@ void thread__find_addr_location(struct thread *thread,
 int perf_event__preprocess_sample(const union perf_event *event,
 				  struct machine *machine,
 				  struct addr_location *al,
-				  struct perf_sample *sample)
+				  struct perf_sample *sample,
+				  struct perf_session *session __maybe_unused)
 {
 	u8 cpumode = event->header.misc & PERF_RECORD_MISC_CPUMODE_MASK;
 	struct thread *thread = machine__findnew_thread(machine, sample->pid,
diff --git a/tools/perf/util/event.h b/tools/perf/util/event.h
index c4ffe2bd0738..19814f70292b 100644
--- a/tools/perf/util/event.h
+++ b/tools/perf/util/event.h
@@ -353,11 +353,13 @@ int perf_event__process(struct perf_tool *tool,
 			struct machine *machine);
 
 struct addr_location;
+struct perf_session;
 
 int perf_event__preprocess_sample(const union perf_event *event,
 				  struct machine *machine,
 				  struct addr_location *al,
-				  struct perf_sample *sample);
+				  struct perf_sample *sample,
+				  struct perf_session *session);
 
 struct thread;
 
-- 
2.2.2


^ permalink raw reply related	[flat|nested] 221+ messages in thread

* [PATCH 11/38] perf script: Pass session arg to ->process_event callback
  2015-03-03  3:07 [RFC/PATCHSET 00/38] perf tools: Speed-up perf report by using multi thread (v3) Namhyung Kim
                   ` (9 preceding siblings ...)
  2015-03-03  3:07 ` [PATCH 10/38] perf tools: Pass session arg to perf_event__preprocess_sample() Namhyung Kim
@ 2015-03-03  3:07 ` Namhyung Kim
  2015-03-03  3:07 ` [PATCH 12/38] perf tools: Introduce thread__comm_time() helpers Namhyung Kim
                   ` (26 subsequent siblings)
  37 siblings, 0 replies; 221+ messages in thread
From: Namhyung Kim @ 2015-03-03  3:07 UTC (permalink / raw)
  To: Arnaldo Carvalho de Melo
  Cc: Ingo Molnar, Peter Zijlstra, Jiri Olsa, LKML,
	Frederic Weisbecker, Adrian Hunter, Stephane Eranian, Andi Kleen,
	David Ahern

Sometimes it needs to retrieve symbol info inside a script engine so
we need to pass the session pointer to find the symbol correctly as
with previous patch.

Signed-off-by: Namhyung Kim <namhyung@kernel.org>
---
 tools/perf/builtin-script.c                        | 23 ++++++++++++----------
 tools/perf/util/db-export.c                        |  6 ++++--
 tools/perf/util/db-export.h                        |  4 +++-
 tools/perf/util/event.c                            |  3 ++-
 tools/perf/util/event.h                            |  3 ++-
 .../perf/util/scripting-engines/trace-event-perl.c |  3 ++-
 .../util/scripting-engines/trace-event-python.c    |  5 +++--
 tools/perf/util/trace-event-scripting.c            |  3 ++-
 tools/perf/util/trace-event.h                      |  3 ++-
 9 files changed, 33 insertions(+), 20 deletions(-)

diff --git a/tools/perf/builtin-script.c b/tools/perf/builtin-script.c
index ab920f8cded6..4a007110d2f7 100644
--- a/tools/perf/builtin-script.c
+++ b/tools/perf/builtin-script.c
@@ -377,9 +377,10 @@ static void print_sample_start(struct perf_sample *sample,
 }
 
 static void print_sample_addr(union perf_event *event,
-			  struct perf_sample *sample,
-			  struct thread *thread,
-			  struct perf_event_attr *attr)
+			      struct perf_sample *sample,
+			      struct thread *thread,
+			      struct perf_event_attr *attr,
+			      struct perf_session *session)
 {
 	struct addr_location al;
 
@@ -388,7 +389,7 @@ static void print_sample_addr(union perf_event *event,
 	if (!sample_addr_correlates_sym(attr))
 		return;
 
-	perf_event__preprocess_sample_addr(event, sample, thread, &al);
+	perf_event__preprocess_sample_addr(event, sample, thread, &al, session);
 
 	if (PRINT_FIELD(SYM)) {
 		printf(" ");
@@ -409,7 +410,8 @@ static void print_sample_bts(union perf_event *event,
 			     struct perf_sample *sample,
 			     struct perf_evsel *evsel,
 			     struct thread *thread,
-			     struct addr_location *al)
+			     struct addr_location *al,
+			     struct perf_session *session)
 {
 	struct perf_event_attr *attr = &evsel->attr;
 	bool print_srcline_last = false;
@@ -436,7 +438,7 @@ static void print_sample_bts(union perf_event *event,
 	    ((evsel->attr.sample_type & PERF_SAMPLE_ADDR) &&
 	     !output[attr->type].user_set)) {
 		printf(" => ");
-		print_sample_addr(event, sample, thread, attr);
+		print_sample_addr(event, sample, thread, attr, session);
 	}
 
 	if (print_srcline_last)
@@ -447,7 +449,7 @@ static void print_sample_bts(union perf_event *event,
 
 static void process_event(union perf_event *event, struct perf_sample *sample,
 			  struct perf_evsel *evsel, struct thread *thread,
-			  struct addr_location *al)
+			  struct addr_location *al, struct perf_session *session)
 {
 	struct perf_event_attr *attr = &evsel->attr;
 
@@ -465,7 +467,7 @@ static void process_event(union perf_event *event, struct perf_sample *sample,
 	}
 
 	if (is_bts_event(attr)) {
-		print_sample_bts(event, sample, evsel, thread, al);
+		print_sample_bts(event, sample, evsel, thread, al, session);
 		return;
 	}
 
@@ -473,7 +475,7 @@ static void process_event(union perf_event *event, struct perf_sample *sample,
 		event_format__print(evsel->tp_format, sample->cpu,
 				    sample->raw_data, sample->raw_size);
 	if (PRINT_FIELD(ADDR))
-		print_sample_addr(event, sample, thread, attr);
+		print_sample_addr(event, sample, thread, attr, session);
 
 	if (PRINT_FIELD(IP)) {
 		if (!symbol_conf.use_callchain)
@@ -590,7 +592,8 @@ static int process_sample_event(struct perf_tool *tool,
 	if (cpu_list && !test_bit(sample->cpu, cpu_bitmap))
 		return 0;
 
-	scripting_ops->process_event(event, sample, evsel, thread, &al);
+	scripting_ops->process_event(event, sample, evsel, thread, &al,
+				     script->session);
 
 	return 0;
 }
diff --git a/tools/perf/util/db-export.c b/tools/perf/util/db-export.c
index c81dae399763..e9ad11fe2e16 100644
--- a/tools/perf/util/db-export.c
+++ b/tools/perf/util/db-export.c
@@ -282,7 +282,8 @@ int db_export__branch_type(struct db_export *dbe, u32 branch_type,
 
 int db_export__sample(struct db_export *dbe, union perf_event *event,
 		      struct perf_sample *sample, struct perf_evsel *evsel,
-		      struct thread *thread, struct addr_location *al)
+		      struct thread *thread, struct addr_location *al,
+		      struct perf_session *session)
 {
 	struct export_sample es = {
 		.event = event,
@@ -328,7 +329,8 @@ int db_export__sample(struct db_export *dbe, union perf_event *event,
 	    sample_addr_correlates_sym(&evsel->attr)) {
 		struct addr_location addr_al;
 
-		perf_event__preprocess_sample_addr(event, sample, thread, &addr_al);
+		perf_event__preprocess_sample_addr(event, sample, thread,
+						   &addr_al, session);
 		err = db_ids_from_al(dbe, &addr_al, &es.addr_dso_db_id,
 				     &es.addr_sym_db_id, &es.addr_offset);
 		if (err)
diff --git a/tools/perf/util/db-export.h b/tools/perf/util/db-export.h
index adbd22d66798..b994f1041d19 100644
--- a/tools/perf/util/db-export.h
+++ b/tools/perf/util/db-export.h
@@ -29,6 +29,7 @@ struct addr_location;
 struct call_return_processor;
 struct call_path;
 struct call_return;
+struct perf_session;
 
 struct export_sample {
 	union perf_event	*event;
@@ -97,7 +98,8 @@ int db_export__branch_type(struct db_export *dbe, u32 branch_type,
 			   const char *name);
 int db_export__sample(struct db_export *dbe, union perf_event *event,
 		      struct perf_sample *sample, struct perf_evsel *evsel,
-		      struct thread *thread, struct addr_location *al);
+		      struct thread *thread, struct addr_location *al,
+		      struct perf_session *session);
 
 int db_export__branch_types(struct db_export *dbe);
 
diff --git a/tools/perf/util/event.c b/tools/perf/util/event.c
index 704ef27cc7c8..510a308c2158 100644
--- a/tools/perf/util/event.c
+++ b/tools/perf/util/event.c
@@ -918,7 +918,8 @@ bool sample_addr_correlates_sym(struct perf_event_attr *attr)
 void perf_event__preprocess_sample_addr(union perf_event *event,
 					struct perf_sample *sample,
 					struct thread *thread,
-					struct addr_location *al)
+					struct addr_location *al,
+					struct perf_session *session __maybe_unused)
 {
 	u8 cpumode = event->header.misc & PERF_RECORD_MISC_CPUMODE_MASK;
 
diff --git a/tools/perf/util/event.h b/tools/perf/util/event.h
index 19814f70292b..27261320249a 100644
--- a/tools/perf/util/event.h
+++ b/tools/perf/util/event.h
@@ -368,7 +368,8 @@ bool sample_addr_correlates_sym(struct perf_event_attr *attr);
 void perf_event__preprocess_sample_addr(union perf_event *event,
 					struct perf_sample *sample,
 					struct thread *thread,
-					struct addr_location *al);
+					struct addr_location *al,
+					struct perf_session *session);
 
 const char *perf_event__name(unsigned int id);
 
diff --git a/tools/perf/util/scripting-engines/trace-event-perl.c b/tools/perf/util/scripting-engines/trace-event-perl.c
index 22ebc46226e7..dd69fbaf03b8 100644
--- a/tools/perf/util/scripting-engines/trace-event-perl.c
+++ b/tools/perf/util/scripting-engines/trace-event-perl.c
@@ -356,7 +356,8 @@ static void perl_process_event(union perf_event *event,
 			       struct perf_sample *sample,
 			       struct perf_evsel *evsel,
 			       struct thread *thread,
-			       struct addr_location *al __maybe_unused)
+			       struct addr_location *al __maybe_unused,
+			       struct perf_session *session __maybe_unused)
 {
 	perl_process_tracepoint(sample, evsel, thread);
 	perl_process_event_generic(event, sample, evsel);
diff --git a/tools/perf/util/scripting-engines/trace-event-python.c b/tools/perf/util/scripting-engines/trace-event-python.c
index 0c815a40a6e8..802def46af7b 100644
--- a/tools/perf/util/scripting-engines/trace-event-python.c
+++ b/tools/perf/util/scripting-engines/trace-event-python.c
@@ -839,7 +839,8 @@ static void python_process_event(union perf_event *event,
 				 struct perf_sample *sample,
 				 struct perf_evsel *evsel,
 				 struct thread *thread,
-				 struct addr_location *al)
+				 struct addr_location *al,
+				 struct perf_session *session)
 {
 	struct tables *tables = &tables_global;
 
@@ -851,7 +852,7 @@ static void python_process_event(union perf_event *event,
 	default:
 		if (tables->db_export_mode)
 			db_export__sample(&tables->dbe, event, sample, evsel,
-					  thread, al);
+					  thread, al, session);
 		else
 			python_process_general_event(sample, evsel, thread, al);
 	}
diff --git a/tools/perf/util/trace-event-scripting.c b/tools/perf/util/trace-event-scripting.c
index 5c9bdd1591a9..36ed50d71171 100644
--- a/tools/perf/util/trace-event-scripting.c
+++ b/tools/perf/util/trace-event-scripting.c
@@ -44,7 +44,8 @@ static void process_event_unsupported(union perf_event *event __maybe_unused,
 				      struct perf_sample *sample __maybe_unused,
 				      struct perf_evsel *evsel __maybe_unused,
 				      struct thread *thread __maybe_unused,
-				      struct addr_location *al __maybe_unused)
+				      struct addr_location *al __maybe_unused,
+				      struct perf_session *session __maybe_unused)
 {
 }
 
diff --git a/tools/perf/util/trace-event.h b/tools/perf/util/trace-event.h
index 356629a30ca9..40e19c2af606 100644
--- a/tools/perf/util/trace-event.h
+++ b/tools/perf/util/trace-event.h
@@ -73,7 +73,8 @@ struct scripting_ops {
 			       struct perf_sample *sample,
 			       struct perf_evsel *evsel,
 			       struct thread *thread,
-				   struct addr_location *al);
+			       struct addr_location *al,
+			       struct perf_session *session);
 	int (*generate_script) (struct pevent *pevent, const char *outfile);
 };
 
-- 
2.2.2


^ permalink raw reply related	[flat|nested] 221+ messages in thread

* [PATCH 12/38] perf tools: Introduce thread__comm_time() helpers
  2015-03-03  3:07 [RFC/PATCHSET 00/38] perf tools: Speed-up perf report by using multi thread (v3) Namhyung Kim
                   ` (10 preceding siblings ...)
  2015-03-03  3:07 ` [PATCH 11/38] perf script: Pass session arg to ->process_event callback Namhyung Kim
@ 2015-03-03  3:07 ` Namhyung Kim
  2015-03-03 16:28   ` Frederic Weisbecker
  2015-03-03  3:07 ` [PATCH 13/38] perf tools: Add a test case for thread comm handling Namhyung Kim
                   ` (25 subsequent siblings)
  37 siblings, 1 reply; 221+ messages in thread
From: Namhyung Kim @ 2015-03-03  3:07 UTC (permalink / raw)
  To: Arnaldo Carvalho de Melo
  Cc: Ingo Molnar, Peter Zijlstra, Jiri Olsa, LKML,
	Frederic Weisbecker, Adrian Hunter, Stephane Eranian, Andi Kleen,
	David Ahern

When data file indexing is enabled, it processes all task, comm and mmap
events first and then goes to the sample events.  So all it sees is the
last comm of a thread although it has information at the time of sample.

Sort thread's comm by time so that it can find appropriate comm at the
sample time.  The thread__comm_time() will mostly work even if
PERF_SAMPLE_TIME bit is off since in that case, sample->time will be
-1 so it'll take the last comm anyway.

Cc: Frederic Weisbecker <fweisbec@gmail.com>
Signed-off-by: Namhyung Kim <namhyung@kernel.org>
---
 tools/perf/util/thread.c | 33 ++++++++++++++++++++++++++++++++-
 tools/perf/util/thread.h |  2 ++
 2 files changed, 34 insertions(+), 1 deletion(-)

diff --git a/tools/perf/util/thread.c b/tools/perf/util/thread.c
index 9ebc8b1f9be5..ad96725105c2 100644
--- a/tools/perf/util/thread.c
+++ b/tools/perf/util/thread.c
@@ -103,6 +103,21 @@ struct comm *thread__exec_comm(const struct thread *thread)
 	return last;
 }
 
+struct comm *thread__comm_time(const struct thread *thread, u64 timestamp)
+{
+	struct comm *comm;
+
+	list_for_each_entry(comm, &thread->comm_list, list) {
+		if (timestamp >= comm->start)
+			return comm;
+	}
+
+	if (list_empty(&thread->comm_list))
+		return NULL;
+
+	return list_last_entry(&thread->comm_list, struct comm, list);
+}
+
 int __thread__set_comm(struct thread *thread, const char *str, u64 timestamp,
 		       bool exec)
 {
@@ -118,7 +133,13 @@ int __thread__set_comm(struct thread *thread, const char *str, u64 timestamp,
 		new = comm__new(str, timestamp, exec);
 		if (!new)
 			return -ENOMEM;
-		list_add(&new->list, &thread->comm_list);
+
+		/* sort by time */
+		list_for_each_entry(curr, &thread->comm_list, list) {
+			if (timestamp >= curr->start)
+				break;
+		}
+		list_add_tail(&new->list, &curr->list);
 
 		if (exec)
 			unwind__flush_access(thread);
@@ -139,6 +160,16 @@ const char *thread__comm_str(const struct thread *thread)
 	return comm__str(comm);
 }
 
+const char *thread__comm_str_time(const struct thread *thread, u64 timestamp)
+{
+	const struct comm *comm = thread__comm_time(thread, timestamp);
+
+	if (!comm)
+		return NULL;
+
+	return comm__str(comm);
+}
+
 /* CHECKME: it should probably better return the max comm len from its comm list */
 int thread__comm_len(struct thread *thread)
 {
diff --git a/tools/perf/util/thread.h b/tools/perf/util/thread.h
index 160fd066a7d1..be67c3bad5e7 100644
--- a/tools/perf/util/thread.h
+++ b/tools/perf/util/thread.h
@@ -53,7 +53,9 @@ static inline int thread__set_comm(struct thread *thread, const char *comm,
 int thread__comm_len(struct thread *thread);
 struct comm *thread__comm(const struct thread *thread);
 struct comm *thread__exec_comm(const struct thread *thread);
+struct comm *thread__comm_time(const struct thread *thread, u64 timestamp);
 const char *thread__comm_str(const struct thread *thread);
+const char *thread__comm_str_time(const struct thread *thread, u64 timestamp);
 void thread__insert_map(struct thread *thread, struct map *map);
 int thread__fork(struct thread *thread, struct thread *parent, u64 timestamp);
 size_t thread__fprintf(struct thread *thread, FILE *fp);
-- 
2.2.2


^ permalink raw reply related	[flat|nested] 221+ messages in thread

* [PATCH 13/38] perf tools: Add a test case for thread comm handling
  2015-03-03  3:07 [RFC/PATCHSET 00/38] perf tools: Speed-up perf report by using multi thread (v3) Namhyung Kim
                   ` (11 preceding siblings ...)
  2015-03-03  3:07 ` [PATCH 12/38] perf tools: Introduce thread__comm_time() helpers Namhyung Kim
@ 2015-03-03  3:07 ` Namhyung Kim
  2015-03-03  3:07 ` [PATCH 14/38] perf tools: Use thread__comm_time() when adding hist entries Namhyung Kim
                   ` (24 subsequent siblings)
  37 siblings, 0 replies; 221+ messages in thread
From: Namhyung Kim @ 2015-03-03  3:07 UTC (permalink / raw)
  To: Arnaldo Carvalho de Melo
  Cc: Ingo Molnar, Peter Zijlstra, Jiri Olsa, LKML,
	Frederic Weisbecker, Adrian Hunter, Stephane Eranian, Andi Kleen,
	David Ahern

The new test case checks various thread comm handling like overridding
and time sorting.

Cc: Frederic Weisbecker <fweisbec@gmail.com>
Signed-off-by: Namhyung Kim <namhyung@kernel.org>
---
 tools/perf/tests/Build          |  1 +
 tools/perf/tests/builtin-test.c |  4 ++++
 tools/perf/tests/tests.h        |  1 +
 tools/perf/tests/thread-comm.c  | 47 +++++++++++++++++++++++++++++++++++++++++
 4 files changed, 53 insertions(+)
 create mode 100644 tools/perf/tests/thread-comm.c

diff --git a/tools/perf/tests/Build b/tools/perf/tests/Build
index 2de01a4b4084..af8f31a3b678 100644
--- a/tools/perf/tests/Build
+++ b/tools/perf/tests/Build
@@ -24,6 +24,7 @@ perf-y += bp_signal_overflow.o
 perf-y += task-exit.o
 perf-y += sw-clock.o
 perf-y += mmap-thread-lookup.o
+perf-y += thread-comm.o
 perf-y += thread-mg-share.o
 perf-y += switch-tracking.o
 perf-y += keep-tracking.o
diff --git a/tools/perf/tests/builtin-test.c b/tools/perf/tests/builtin-test.c
index 4b7d9ab0f049..1b463d82a71a 100644
--- a/tools/perf/tests/builtin-test.c
+++ b/tools/perf/tests/builtin-test.c
@@ -167,6 +167,10 @@ static struct test {
 		.func = test__fdarray__add,
 	},
 	{
+		.desc = "Test thread comm handling",
+		.func = test__thread_comm,
+	},
+	{
 		.func = NULL,
 	},
 };
diff --git a/tools/perf/tests/tests.h b/tools/perf/tests/tests.h
index 00e776a87a9c..43ac17780629 100644
--- a/tools/perf/tests/tests.h
+++ b/tools/perf/tests/tests.h
@@ -51,6 +51,7 @@ int test__hists_cumulate(void);
 int test__switch_tracking(void);
 int test__fdarray__filter(void);
 int test__fdarray__add(void);
+int test__thread_comm(void);
 
 #if defined(__x86_64__) || defined(__i386__) || defined(__arm__)
 #ifdef HAVE_DWARF_UNWIND_SUPPORT
diff --git a/tools/perf/tests/thread-comm.c b/tools/perf/tests/thread-comm.c
new file mode 100644
index 000000000000..44ee85d71581
--- /dev/null
+++ b/tools/perf/tests/thread-comm.c
@@ -0,0 +1,47 @@
+#include "tests.h"
+#include "machine.h"
+#include "thread.h"
+#include "debug.h"
+
+int test__thread_comm(void)
+{
+	struct machines machines;
+	struct machine *machine;
+	struct thread *t;
+
+	/*
+	 * This test is to check whether it can retrieve a correct
+	 * comm for a given time.  When multi-file data storage is
+	 * enabled, those task/comm events are processed first so the
+	 * later sample should find a matching comm properly.
+	 */
+	machines__init(&machines);
+	machine = &machines.host;
+
+	t = machine__findnew_thread(machine, 100, 100);
+	TEST_ASSERT_VAL("wrong init thread comm",
+			!strcmp(thread__comm_str(t), ":100"));
+
+	thread__set_comm(t, "perf-test1", 10000);
+	TEST_ASSERT_VAL("failed to override thread comm",
+			!strcmp(thread__comm_str(t), "perf-test1"));
+
+	thread__set_comm(t, "perf-test2", 20000);
+	thread__set_comm(t, "perf-test3", 30000);
+	thread__set_comm(t, "perf-test4", 40000);
+
+	TEST_ASSERT_VAL("failed to find timed comm",
+			!strcmp(thread__comm_str_time(t, 20000), "perf-test2"));
+	TEST_ASSERT_VAL("failed to find timed comm",
+			!strcmp(thread__comm_str_time(t, 35000), "perf-test3"));
+	TEST_ASSERT_VAL("failed to find timed comm",
+			!strcmp(thread__comm_str_time(t, 50000), "perf-test4"));
+
+	thread__set_comm(t, "perf-test1.5", 15000);
+	TEST_ASSERT_VAL("failed to sort timed comm",
+			!strcmp(thread__comm_str_time(t, 15000), "perf-test1.5"));
+
+	machine__delete_threads(machine);
+	machines__exit(&machines);
+	return 0;
+}
-- 
2.2.2


^ permalink raw reply related	[flat|nested] 221+ messages in thread

* [PATCH 14/38] perf tools: Use thread__comm_time() when adding hist entries
  2015-03-03  3:07 [RFC/PATCHSET 00/38] perf tools: Speed-up perf report by using multi thread (v3) Namhyung Kim
                   ` (12 preceding siblings ...)
  2015-03-03  3:07 ` [PATCH 13/38] perf tools: Add a test case for thread comm handling Namhyung Kim
@ 2015-03-03  3:07 ` Namhyung Kim
  2015-03-03  3:07 ` [PATCH 15/38] perf tools: Convert dead thread list into rbtree Namhyung Kim
                   ` (23 subsequent siblings)
  37 siblings, 0 replies; 221+ messages in thread
From: Namhyung Kim @ 2015-03-03  3:07 UTC (permalink / raw)
  To: Arnaldo Carvalho de Melo
  Cc: Ingo Molnar, Peter Zijlstra, Jiri Olsa, LKML,
	Frederic Weisbecker, Adrian Hunter, Stephane Eranian, Andi Kleen,
	David Ahern

Now thread->comm can be handled with time properly, use it to find
correct comm when adding hist entries.

Cc: Frederic Weisbecker <fweisbec@gmail.com>
Signed-off-by: Namhyung Kim <namhyung@kernel.org>
---
 tools/perf/builtin-annotate.c |  5 +++--
 tools/perf/builtin-diff.c     |  8 ++++----
 tools/perf/tests/hists_link.c |  4 ++--
 tools/perf/util/hist.c        | 19 ++++++++++---------
 tools/perf/util/hist.h        |  2 +-
 5 files changed, 20 insertions(+), 18 deletions(-)

diff --git a/tools/perf/builtin-annotate.c b/tools/perf/builtin-annotate.c
index b89e4c6ed488..50628900f9fa 100644
--- a/tools/perf/builtin-annotate.c
+++ b/tools/perf/builtin-annotate.c
@@ -47,7 +47,7 @@ struct perf_annotate {
 };
 
 static int perf_evsel__add_sample(struct perf_evsel *evsel,
-				  struct perf_sample *sample __maybe_unused,
+				  struct perf_sample *sample,
 				  struct addr_location *al,
 				  struct perf_annotate *ann)
 {
@@ -67,7 +67,8 @@ static int perf_evsel__add_sample(struct perf_evsel *evsel,
 		return 0;
 	}
 
-	he = __hists__add_entry(hists, al, NULL, NULL, NULL, 1, 1, 0, true);
+	he = __hists__add_entry(hists, al, NULL, NULL, NULL, 1, 1, 0,
+				sample->time, true);
 	if (he == NULL)
 		return -ENOMEM;
 
diff --git a/tools/perf/builtin-diff.c b/tools/perf/builtin-diff.c
index 3e2229227062..ddf6f0999838 100644
--- a/tools/perf/builtin-diff.c
+++ b/tools/perf/builtin-diff.c
@@ -313,10 +313,10 @@ static int formula_fprintf(struct hist_entry *he, struct hist_entry *pair,
 
 static int hists__add_entry(struct hists *hists,
 			    struct addr_location *al, u64 period,
-			    u64 weight, u64 transaction)
+			    u64 weight, u64 transaction, u64 timestamp)
 {
 	if (__hists__add_entry(hists, al, NULL, NULL, NULL, period, weight,
-			       transaction, true) != NULL)
+			       transaction, timestamp, true) != NULL)
 		return 0;
 	return -ENOMEM;
 }
@@ -338,8 +338,8 @@ static int diff__process_sample_event(struct perf_tool *tool,
 		return -1;
 	}
 
-	if (hists__add_entry(hists, &al, sample->period,
-			     sample->weight, sample->transaction)) {
+	if (hists__add_entry(hists, &al, sample->period, sample->weight,
+			     sample->transaction, sample->time)) {
 		pr_warning("problem incrementing symbol period, skipping event\n");
 		return -1;
 	}
diff --git a/tools/perf/tests/hists_link.c b/tools/perf/tests/hists_link.c
index a731a531a3e2..4f3d45692acb 100644
--- a/tools/perf/tests/hists_link.c
+++ b/tools/perf/tests/hists_link.c
@@ -90,7 +90,7 @@ static int add_hist_entries(struct perf_evlist *evlist, struct machine *machine)
 				goto out;
 
 			he = __hists__add_entry(hists, &al, NULL,
-						NULL, NULL, 1, 1, 0, true);
+						NULL, NULL, 1, 1, 0, -1, true);
 			if (he == NULL)
 				goto out;
 
@@ -114,7 +114,7 @@ static int add_hist_entries(struct perf_evlist *evlist, struct machine *machine)
 				goto out;
 
 			he = __hists__add_entry(hists, &al, NULL,
-						NULL, NULL, 1, 1, 0, true);
+						NULL, NULL, 1, 1, 0, -1, true);
 			if (he == NULL)
 				goto out;
 
diff --git a/tools/perf/util/hist.c b/tools/perf/util/hist.c
index 70b48a65064c..4badf2491fbf 100644
--- a/tools/perf/util/hist.c
+++ b/tools/perf/util/hist.c
@@ -447,11 +447,11 @@ struct hist_entry *__hists__add_entry(struct hists *hists,
 				      struct branch_info *bi,
 				      struct mem_info *mi,
 				      u64 period, u64 weight, u64 transaction,
-				      bool sample_self)
+				      u64 timestamp, bool sample_self)
 {
 	struct hist_entry entry = {
 		.thread	= al->thread,
-		.comm = thread__comm(al->thread),
+		.comm = thread__comm_time(al->thread, timestamp),
 		.ms = {
 			.map	= al->map,
 			.sym	= al->sym,
@@ -509,13 +509,14 @@ iter_add_single_mem_entry(struct hist_entry_iter *iter, struct addr_location *al
 {
 	u64 cost;
 	struct mem_info *mi = iter->priv;
+	struct perf_sample *sample = iter->sample;
 	struct hists *hists = evsel__hists(iter->evsel);
 	struct hist_entry *he;
 
 	if (mi == NULL)
 		return -EINVAL;
 
-	cost = iter->sample->weight;
+	cost = sample->weight;
 	if (!cost)
 		cost = 1;
 
@@ -527,7 +528,7 @@ iter_add_single_mem_entry(struct hist_entry_iter *iter, struct addr_location *al
 	 * and the he_stat__add_period() function.
 	 */
 	he = __hists__add_entry(hists, al, iter->parent, NULL, mi,
-				cost, cost, 0, true);
+				cost, cost, 0, sample->time, true);
 	if (!he)
 		return -ENOMEM;
 
@@ -628,7 +629,7 @@ iter_add_next_branch_entry(struct hist_entry_iter *iter, struct addr_location *a
 	 * and not events sampled. Thus we use a pseudo period of 1.
 	 */
 	he = __hists__add_entry(hists, al, iter->parent, &bi[i], NULL,
-				1, 1, 0, true);
+				1, 1, 0, iter->sample->time, true);
 	if (he == NULL)
 		return -ENOMEM;
 
@@ -666,7 +667,7 @@ iter_add_single_normal_entry(struct hist_entry_iter *iter, struct addr_location
 
 	he = __hists__add_entry(evsel__hists(evsel), al, iter->parent, NULL, NULL,
 				sample->period, sample->weight,
-				sample->transaction, true);
+				sample->transaction, sample->time, true);
 	if (he == NULL)
 		return -ENOMEM;
 
@@ -728,7 +729,7 @@ iter_add_single_cumulative_entry(struct hist_entry_iter *iter,
 
 	he = __hists__add_entry(hists, al, iter->parent, NULL, NULL,
 				sample->period, sample->weight,
-				sample->transaction, true);
+				sample->transaction, sample->time, true);
 	if (he == NULL)
 		return -ENOMEM;
 
@@ -772,7 +773,7 @@ iter_add_next_cumulative_entry(struct hist_entry_iter *iter,
 	struct hist_entry he_tmp = {
 		.cpu = al->cpu,
 		.thread = al->thread,
-		.comm = thread__comm(al->thread),
+		.comm = thread__comm_time(al->thread, sample->time),
 		.ip = al->addr,
 		.ms = {
 			.map = al->map,
@@ -801,7 +802,7 @@ iter_add_next_cumulative_entry(struct hist_entry_iter *iter,
 
 	he = __hists__add_entry(evsel__hists(evsel), al, iter->parent, NULL, NULL,
 				sample->period, sample->weight,
-				sample->transaction, false);
+				sample->transaction, sample->time, false);
 	if (he == NULL)
 		return -ENOMEM;
 
diff --git a/tools/perf/util/hist.h b/tools/perf/util/hist.h
index 2b690d028907..0eed50a5b1f0 100644
--- a/tools/perf/util/hist.h
+++ b/tools/perf/util/hist.h
@@ -109,7 +109,7 @@ struct hist_entry *__hists__add_entry(struct hists *hists,
 				      struct branch_info *bi,
 				      struct mem_info *mi, u64 period,
 				      u64 weight, u64 transaction,
-				      bool sample_self);
+				      u64 timestamp, bool sample_self);
 int hist_entry_iter__add(struct hist_entry_iter *iter, struct addr_location *al,
 			 struct perf_evsel *evsel, struct perf_sample *sample,
 			 int max_stack_depth, void *arg);
-- 
2.2.2


^ permalink raw reply related	[flat|nested] 221+ messages in thread

* [PATCH 15/38] perf tools: Convert dead thread list into rbtree
  2015-03-03  3:07 [RFC/PATCHSET 00/38] perf tools: Speed-up perf report by using multi thread (v3) Namhyung Kim
                   ` (13 preceding siblings ...)
  2015-03-03  3:07 ` [PATCH 14/38] perf tools: Use thread__comm_time() when adding hist entries Namhyung Kim
@ 2015-03-03  3:07 ` Namhyung Kim
  2015-03-03  3:07 ` [PATCH 16/38] perf tools: Introduce machine__find*_thread_time() Namhyung Kim
                   ` (22 subsequent siblings)
  37 siblings, 0 replies; 221+ messages in thread
From: Namhyung Kim @ 2015-03-03  3:07 UTC (permalink / raw)
  To: Arnaldo Carvalho de Melo
  Cc: Ingo Molnar, Peter Zijlstra, Jiri Olsa, LKML,
	Frederic Weisbecker, Adrian Hunter, Stephane Eranian, Andi Kleen,
	David Ahern

Currently perf maintains dead threads in a linked list but this can be
a problem if someone needs to search from it especially in a large
session which might have many dead threads.  Convert it to a rbtree
like normal threads and it'll be used later with multi-file changes.

The list node is now used for chaining dead threads of same tid since
it's easier to handle such threads in time order.

Cc: Frederic Weisbecker <fweisbec@gmail.com>
Signed-off-by: Namhyung Kim <namhyung@kernel.org>
---
 tools/perf/util/machine.c | 70 +++++++++++++++++++++++++++++++++++++++++------
 tools/perf/util/machine.h |  2 +-
 tools/perf/util/thread.c  |  1 +
 tools/perf/util/thread.h  | 11 ++++----
 4 files changed, 68 insertions(+), 16 deletions(-)

diff --git a/tools/perf/util/machine.c b/tools/perf/util/machine.c
index 9e0f60a7e7b3..6b8236dc4367 100644
--- a/tools/perf/util/machine.c
+++ b/tools/perf/util/machine.c
@@ -28,7 +28,7 @@ int machine__init(struct machine *machine, const char *root_dir, pid_t pid)
 	dsos__init(&machine->kernel_dsos);
 
 	machine->threads = RB_ROOT;
-	INIT_LIST_HEAD(&machine->dead_threads);
+	machine->dead_threads = RB_ROOT;
 	machine->last_match = NULL;
 
 	machine->vdso_info = NULL;
@@ -91,10 +91,22 @@ static void dsos__delete(struct dsos *dsos)
 
 void machine__delete_dead_threads(struct machine *machine)
 {
-	struct thread *n, *t;
+	struct rb_node *nd = rb_first(&machine->dead_threads);
+
+	while (nd) {
+		struct thread *t = rb_entry(nd, struct thread, rb_node);
+		struct thread *pos;
+
+		nd = rb_next(nd);
+		rb_erase(&t->rb_node, &machine->dead_threads);
+
+		while (!list_empty(&t->tid_node)) {
+			pos = list_first_entry(&t->tid_node,
+					       struct thread, tid_node);
+			list_del(&pos->tid_node);
+			thread__delete(pos);
+		}
 
-	list_for_each_entry_safe(t, n, &machine->dead_threads, node) {
-		list_del(&t->node);
 		thread__delete(t);
 	}
 }
@@ -106,8 +118,8 @@ void machine__delete_threads(struct machine *machine)
 	while (nd) {
 		struct thread *t = rb_entry(nd, struct thread, rb_node);
 
-		rb_erase(&t->rb_node, &machine->threads);
 		nd = rb_next(nd);
+		rb_erase(&t->rb_node, &machine->threads);
 		thread__delete(t);
 	}
 }
@@ -1238,13 +1250,46 @@ int machine__process_mmap_event(struct machine *machine, union perf_event *event
 
 static void machine__remove_thread(struct machine *machine, struct thread *th)
 {
+	struct rb_node **p = &machine->dead_threads.rb_node;
+	struct rb_node *parent = NULL;
+	struct thread *pos;
+
 	machine->last_match = NULL;
 	rb_erase(&th->rb_node, &machine->threads);
+
+	th->dead = true;
+
 	/*
 	 * We may have references to this thread, for instance in some hist_entry
-	 * instances, so just move them to a separate list.
+	 * instances, so just move them to a separate list in rbtree.
 	 */
-	list_add_tail(&th->node, &machine->dead_threads);
+	while (*p != NULL) {
+		parent = *p;
+		pos = rb_entry(parent, struct thread, rb_node);
+
+		if (pos->tid == th->tid) {
+			struct thread *old;
+
+			/* sort by time */
+			list_for_each_entry(old, &pos->tid_node, tid_node) {
+				if (th->start_time >= old->start_time) {
+					pos = old;
+					break;
+				}
+			}
+
+			list_add_tail(&th->tid_node, &pos->tid_node);
+			return;
+		}
+
+		if (th->tid < pos->tid)
+			p = &(*p)->rb_left;
+		else
+			p = &(*p)->rb_right;
+	}
+
+	rb_link_node(&th->rb_node, parent, p);
+	rb_insert_color(&th->rb_node, &machine->dead_threads);
 }
 
 int machine__process_fork_event(struct machine *machine, union perf_event *event,
@@ -1729,7 +1774,7 @@ int machine__for_each_thread(struct machine *machine,
 			     void *priv)
 {
 	struct rb_node *nd;
-	struct thread *thread;
+	struct thread *thread, *pos;
 	int rc = 0;
 
 	for (nd = rb_first(&machine->threads); nd; nd = rb_next(nd)) {
@@ -1739,10 +1784,17 @@ int machine__for_each_thread(struct machine *machine,
 			return rc;
 	}
 
-	list_for_each_entry(thread, &machine->dead_threads, node) {
+	for (nd = rb_first(&machine->dead_threads); nd; nd = rb_next(nd)) {
+		thread = rb_entry(nd, struct thread, rb_node);
 		rc = fn(thread, priv);
 		if (rc != 0)
 			return rc;
+
+		list_for_each_entry(pos, &thread->tid_node, tid_node) {
+			rc = fn(pos, priv);
+			if (rc != 0)
+				return rc;
+		}
 	}
 	return rc;
 }
diff --git a/tools/perf/util/machine.h b/tools/perf/util/machine.h
index e8b7779a0a3f..4349946a38ff 100644
--- a/tools/perf/util/machine.h
+++ b/tools/perf/util/machine.h
@@ -30,7 +30,7 @@ struct machine {
 	bool		  comm_exec;
 	char		  *root_dir;
 	struct rb_root	  threads;
-	struct list_head  dead_threads;
+	struct rb_root	  dead_threads;
 	struct thread	  *last_match;
 	struct vdso_info  *vdso_info;
 	struct dsos	  user_dsos;
diff --git a/tools/perf/util/thread.c b/tools/perf/util/thread.c
index ad96725105c2..c9ae0e1599da 100644
--- a/tools/perf/util/thread.c
+++ b/tools/perf/util/thread.c
@@ -38,6 +38,7 @@ struct thread *thread__new(pid_t pid, pid_t tid)
 		thread->ppid = -1;
 		thread->cpu = -1;
 		INIT_LIST_HEAD(&thread->comm_list);
+		INIT_LIST_HEAD(&thread->tid_node);
 
 		if (unwind__prepare_access(thread) < 0)
 			goto err_thread;
diff --git a/tools/perf/util/thread.h b/tools/perf/util/thread.h
index be67c3bad5e7..21268e66b2ad 100644
--- a/tools/perf/util/thread.h
+++ b/tools/perf/util/thread.h
@@ -11,10 +11,8 @@
 struct thread_stack;
 
 struct thread {
-	union {
-		struct rb_node	 rb_node;
-		struct list_head node;
-	};
+	struct rb_node	 	rb_node;
+	struct list_head 	tid_node;
 	struct map_groups	*mg;
 	pid_t			pid_; /* Not all tools update this */
 	pid_t			tid;
@@ -22,7 +20,8 @@ struct thread {
 	int			cpu;
 	char			shortname[3];
 	bool			comm_set;
-	bool			dead; /* if set thread has exited */
+	bool			exited; /* if set thread has exited */
+	bool			dead; /* thread is in dead_threads list */
 	struct list_head	comm_list;
 	int			comm_len;
 	u64			db_id;
@@ -39,7 +38,7 @@ int thread__init_map_groups(struct thread *thread, struct machine *machine);
 void thread__delete(struct thread *thread);
 static inline void thread__exited(struct thread *thread)
 {
-	thread->dead = true;
+	thread->exited = true;
 }
 
 int __thread__set_comm(struct thread *thread, const char *comm, u64 timestamp,
-- 
2.2.2


^ permalink raw reply related	[flat|nested] 221+ messages in thread

* [PATCH 16/38] perf tools: Introduce machine__find*_thread_time()
  2015-03-03  3:07 [RFC/PATCHSET 00/38] perf tools: Speed-up perf report by using multi thread (v3) Namhyung Kim
                   ` (14 preceding siblings ...)
  2015-03-03  3:07 ` [PATCH 15/38] perf tools: Convert dead thread list into rbtree Namhyung Kim
@ 2015-03-03  3:07 ` Namhyung Kim
  2015-03-03  3:07 ` [PATCH 17/38] perf tools: Add a test case for timed thread handling Namhyung Kim
                   ` (21 subsequent siblings)
  37 siblings, 0 replies; 221+ messages in thread
From: Namhyung Kim @ 2015-03-03  3:07 UTC (permalink / raw)
  To: Arnaldo Carvalho de Melo
  Cc: Ingo Molnar, Peter Zijlstra, Jiri Olsa, LKML,
	Frederic Weisbecker, Adrian Hunter, Stephane Eranian, Andi Kleen,
	David Ahern

With data file indexing is enabled, it needs to search thread based on
sample time since sample processing is done after other (task, comm and
mmap) events are processed.  This can be a problem if a session is very
long and pid is recycled - in that case it'll only see the last one.

So keep thread start time in it, and search thread based on the time.
This patch introduces machine__find{,new}_thread_time() function for
this.  It'll first search current thread rbtree and then dead thread
tree and list.  If it couldn't find anyone, it'll create a new thread.

The sample timestamp of 0 means that this is called from synthesized
event so just use current rbtree.  The timestamp will be -1 if sample
didn't record the timestamp so will see current threads automatically.

Cc: Frederic Weisbecker <fweisbec@gmail.com>
Signed-off-by: Namhyung Kim <namhyung@kernel.org>
---
 tools/perf/builtin-script.c     |  11 ++++-
 tools/perf/tests/dwarf-unwind.c |   8 ++--
 tools/perf/tests/hists_common.c |   3 +-
 tools/perf/tests/hists_link.c   |   2 +-
 tools/perf/util/event.c         |  14 ++++--
 tools/perf/util/machine.c       | 102 +++++++++++++++++++++++++++++++++++++++-
 tools/perf/util/machine.h       |   8 +++-
 tools/perf/util/thread.c        |   4 ++
 tools/perf/util/thread.h        |   1 +
 9 files changed, 138 insertions(+), 15 deletions(-)

diff --git a/tools/perf/builtin-script.c b/tools/perf/builtin-script.c
index 4a007110d2f7..65b3a07be2bf 100644
--- a/tools/perf/builtin-script.c
+++ b/tools/perf/builtin-script.c
@@ -559,8 +559,15 @@ static int process_sample_event(struct perf_tool *tool,
 {
 	struct addr_location al;
 	struct perf_script *script = container_of(tool, struct perf_script, tool);
-	struct thread *thread = machine__findnew_thread(machine, sample->pid,
-							sample->tid);
+	struct thread *thread;
+
+	if (perf_session__has_index(script->session))
+		thread = machine__findnew_thread_time(machine, sample->pid,
+						      sample->tid,
+						      sample->time);
+	else
+		thread = machine__findnew_thread(machine, sample->pid,
+						 sample->tid);
 
 	if (thread == NULL) {
 		pr_debug("problem processing %d event, skipping it.\n",
diff --git a/tools/perf/tests/dwarf-unwind.c b/tools/perf/tests/dwarf-unwind.c
index 0bf06bec68c7..7e04feb431cb 100644
--- a/tools/perf/tests/dwarf-unwind.c
+++ b/tools/perf/tests/dwarf-unwind.c
@@ -16,10 +16,10 @@
 
 static int mmap_handler(struct perf_tool *tool __maybe_unused,
 			union perf_event *event,
-			struct perf_sample *sample __maybe_unused,
+			struct perf_sample *sample,
 			struct machine *machine)
 {
-	return machine__process_mmap2_event(machine, event, NULL);
+	return machine__process_mmap2_event(machine, event, sample);
 }
 
 static int init_live_machine(struct machine *machine)
@@ -66,12 +66,10 @@ static int unwind_entry(struct unwind_entry *entry, void *arg)
 __attribute__ ((noinline))
 static int unwind_thread(struct thread *thread)
 {
-	struct perf_sample sample;
+	struct perf_sample sample = { .time = -1ULL, };
 	unsigned long cnt = 0;
 	int err = -1;
 
-	memset(&sample, 0, sizeof(sample));
-
 	if (test__arch_unwind_sample(&sample, thread)) {
 		pr_debug("failed to get unwind sample\n");
 		goto out;
diff --git a/tools/perf/tests/hists_common.c b/tools/perf/tests/hists_common.c
index a62c09134516..86a8fdb41804 100644
--- a/tools/perf/tests/hists_common.c
+++ b/tools/perf/tests/hists_common.c
@@ -80,6 +80,7 @@ static struct {
 struct machine *setup_fake_machine(struct machines *machines)
 {
 	struct machine *machine = machines__find(machines, HOST_KERNEL_ID);
+	struct perf_sample sample = { .time = -1ULL, };
 	size_t i;
 
 	if (machine == NULL) {
@@ -113,7 +114,7 @@ struct machine *setup_fake_machine(struct machines *machines)
 		strcpy(fake_mmap_event.mmap.filename,
 		       fake_mmap_info[i].filename);
 
-		machine__process_mmap_event(machine, &fake_mmap_event, NULL);
+		machine__process_mmap_event(machine, &fake_mmap_event, &sample);
 	}
 
 	for (i = 0; i < ARRAY_SIZE(fake_symbols); i++) {
diff --git a/tools/perf/tests/hists_link.c b/tools/perf/tests/hists_link.c
index 4f3d45692acb..1237cc87e8d5 100644
--- a/tools/perf/tests/hists_link.c
+++ b/tools/perf/tests/hists_link.c
@@ -64,7 +64,7 @@ static int add_hist_entries(struct perf_evlist *evlist, struct machine *machine)
 	struct perf_evsel *evsel;
 	struct addr_location al;
 	struct hist_entry *he;
-	struct perf_sample sample = { .period = 1, };
+	struct perf_sample sample = { .period = 1, .time = -1ULL, };
 	size_t i = 0, k;
 
 	/*
diff --git a/tools/perf/util/event.c b/tools/perf/util/event.c
index 510a308c2158..3bfe10fe0c69 100644
--- a/tools/perf/util/event.c
+++ b/tools/perf/util/event.c
@@ -9,6 +9,7 @@
 #include "strlist.h"
 #include "thread.h"
 #include "thread_map.h"
+#include "session.h"
 #include "symbol/kallsyms.h"
 
 static const char *perf_event__names[] = {
@@ -837,11 +838,18 @@ int perf_event__preprocess_sample(const union perf_event *event,
 				  struct machine *machine,
 				  struct addr_location *al,
 				  struct perf_sample *sample,
-				  struct perf_session *session __maybe_unused)
+				  struct perf_session *session)
 {
 	u8 cpumode = event->header.misc & PERF_RECORD_MISC_CPUMODE_MASK;
-	struct thread *thread = machine__findnew_thread(machine, sample->pid,
-							sample->tid);
+	struct thread *thread;
+
+	if (session && perf_session__has_index(session))
+		thread = machine__findnew_thread_time(machine, sample->pid,
+						      sample->tid,
+						      sample->time);
+	else
+		thread = machine__findnew_thread(machine, sample->pid,
+						 sample->tid);
 
 	if (thread == NULL)
 		return -1;
diff --git a/tools/perf/util/machine.c b/tools/perf/util/machine.c
index 6b8236dc4367..b4b97b5e1f1c 100644
--- a/tools/perf/util/machine.c
+++ b/tools/perf/util/machine.c
@@ -434,6 +434,106 @@ struct thread *machine__find_thread(struct machine *machine, pid_t pid,
 	return __machine__findnew_thread(machine, pid, tid, false);
 }
 
+static struct thread *__machine__findnew_thread_time(struct machine *machine,
+						     pid_t pid, pid_t tid,
+						     u64 timestamp, bool create)
+{
+	struct thread *curr, *pos, *new;
+	struct thread *th = NULL;
+	struct rb_node **p;
+	struct rb_node *parent = NULL;
+
+	curr = __machine__findnew_thread(machine, pid, tid, false);
+	if (curr && timestamp >= curr->start_time)
+		return curr;
+
+	p = &machine->dead_threads.rb_node;
+	while (*p != NULL) {
+		parent = *p;
+		th = rb_entry(parent, struct thread, rb_node);
+
+		if (th->tid == tid) {
+			list_for_each_entry(pos, &th->tid_node, tid_node) {
+				if (timestamp >= pos->start_time &&
+				    pos->start_time > th->start_time) {
+					th = pos;
+					break;
+				}
+			}
+
+			if (timestamp >= th->start_time) {
+				machine__update_thread_pid(machine, th, pid);
+				return th;
+			}
+			break;
+		}
+
+		if (tid < th->tid)
+			p = &(*p)->rb_left;
+		else
+			p = &(*p)->rb_right;
+	}
+
+	if (!create)
+		return NULL;
+
+	if (!curr && !*p)
+		return __machine__findnew_thread(machine, pid, tid, true);
+
+	new = thread__new(pid, tid);
+	if (new == NULL)
+		return NULL;
+
+	new->dead = true;
+	new->start_time = timestamp;
+
+	if (*p) {
+		list_for_each_entry(pos, &th->tid_node, tid_node) {
+			/* sort by time */
+			if (timestamp >= pos->start_time) {
+				th = pos;
+				break;
+			}
+		}
+		list_add_tail(&new->tid_node, &th->tid_node);
+	} else {
+		rb_link_node(&new->rb_node, parent, p);
+		rb_insert_color(&new->rb_node, &machine->dead_threads);
+	}
+
+	/*
+	 * We have to initialize map_groups separately
+	 * after rb tree is updated.
+	 *
+	 * The reason is that we call machine__findnew_thread
+	 * within thread__init_map_groups to find the thread
+	 * leader and that would screwed the rb tree.
+	 */
+	if (thread__init_map_groups(new, machine)) {
+		if (!list_empty(&new->tid_node))
+			list_del(&new->tid_node);
+		else
+			rb_erase(&new->rb_node, &machine->dead_threads);
+
+		thread__delete(new);
+		return NULL;
+	}
+
+	return new;
+}
+
+struct thread *machine__find_thread_time(struct machine *machine, pid_t pid,
+					 pid_t tid, u64 timestamp)
+{
+	return __machine__findnew_thread_time(machine, pid, tid, timestamp, false);
+}
+
+struct thread *machine__findnew_thread_time(struct machine *machine, pid_t pid,
+					    pid_t tid, u64 timestamp)
+{
+	return __machine__findnew_thread_time(machine, pid, tid, timestamp, true);
+}
+
 struct comm *machine__thread_exec_comm(struct machine *machine,
 				       struct thread *thread)
 {
@@ -1172,7 +1272,7 @@ int machine__process_mmap2_event(struct machine *machine,
 	}
 
 	thread = machine__findnew_thread(machine, event->mmap2.pid,
-					event->mmap2.tid);
+					 event->mmap2.tid);
 	if (thread == NULL)
 		goto out_problem;
 
diff --git a/tools/perf/util/machine.h b/tools/perf/util/machine.h
index 4349946a38ff..9571b6b1c5b5 100644
--- a/tools/perf/util/machine.h
+++ b/tools/perf/util/machine.h
@@ -68,8 +68,6 @@ static inline bool machine__kernel_ip(struct machine *machine, u64 ip)
 	return ip >= kernel_start;
 }
 
-struct thread *machine__find_thread(struct machine *machine, pid_t pid,
-				    pid_t tid);
 struct comm *machine__thread_exec_comm(struct machine *machine,
 				       struct thread *thread);
 
@@ -149,6 +147,12 @@ static inline bool machine__is_host(struct machine *machine)
 
 struct thread *machine__findnew_thread(struct machine *machine, pid_t pid,
 				       pid_t tid);
+struct thread *machine__find_thread(struct machine *machine, pid_t pid,
+				    pid_t tid);
+struct thread *machine__findnew_thread_time(struct machine *machine, pid_t pid,
+					    pid_t tid, u64 timestamp);
+struct thread *machine__find_thread_time(struct machine *machine, pid_t pid,
+					 pid_t tid, u64 timestamp);
 
 size_t machine__fprintf(struct machine *machine, FILE *fp);
 
diff --git a/tools/perf/util/thread.c b/tools/perf/util/thread.c
index c9ae0e1599da..306bdaede019 100644
--- a/tools/perf/util/thread.c
+++ b/tools/perf/util/thread.c
@@ -127,6 +127,9 @@ int __thread__set_comm(struct thread *thread, const char *str, u64 timestamp,
 
 	/* Override the default :tid entry */
 	if (!thread->comm_set) {
+		if (!thread->start_time)
+			thread->start_time = timestamp;
+
 		err = comm__override(curr, str, timestamp, exec);
 		if (err)
 			return err;
@@ -228,6 +231,7 @@ int thread__fork(struct thread *thread, struct thread *parent, u64 timestamp)
 	}
 
 	thread->ppid = parent->tid;
+	thread->start_time = timestamp;
 	return thread__clone_map_groups(thread, parent);
 }
 
diff --git a/tools/perf/util/thread.h b/tools/perf/util/thread.h
index 21268e66b2ad..e5d7abd255ea 100644
--- a/tools/perf/util/thread.h
+++ b/tools/perf/util/thread.h
@@ -25,6 +25,7 @@ struct thread {
 	struct list_head	comm_list;
 	int			comm_len;
 	u64			db_id;
+	u64			start_time;
 
 	void			*priv;
 	struct thread_stack	*ts;
-- 
2.2.2


^ permalink raw reply related	[flat|nested] 221+ messages in thread

* [PATCH 17/38] perf tools: Add a test case for timed thread handling
  2015-03-03  3:07 [RFC/PATCHSET 00/38] perf tools: Speed-up perf report by using multi thread (v3) Namhyung Kim
                   ` (15 preceding siblings ...)
  2015-03-03  3:07 ` [PATCH 16/38] perf tools: Introduce machine__find*_thread_time() Namhyung Kim
@ 2015-03-03  3:07 ` Namhyung Kim
  2015-03-03  3:07 ` [PATCH 18/38] perf tools: Reducing arguments of hist_entry_iter__add() Namhyung Kim
                   ` (20 subsequent siblings)
  37 siblings, 0 replies; 221+ messages in thread
From: Namhyung Kim @ 2015-03-03  3:07 UTC (permalink / raw)
  To: Arnaldo Carvalho de Melo
  Cc: Ingo Molnar, Peter Zijlstra, Jiri Olsa, LKML,
	Frederic Weisbecker, Adrian Hunter, Stephane Eranian, Andi Kleen,
	David Ahern

A test case for verifying live and dead thread tree management during
time change and new machine__find{,new}_thread_time().

Cc: Frederic Weisbecker <fweisbec@gmail.com>
Signed-off-by: Namhyung Kim <namhyung@kernel.org>
---
 tools/perf/tests/Build                |   1 +
 tools/perf/tests/builtin-test.c       |   4 +
 tools/perf/tests/tests.h              |   1 +
 tools/perf/tests/thread-lookup-time.c | 174 ++++++++++++++++++++++++++++++++++
 4 files changed, 180 insertions(+)
 create mode 100644 tools/perf/tests/thread-lookup-time.c

diff --git a/tools/perf/tests/Build b/tools/perf/tests/Build
index af8f31a3b678..bfa0aa35761f 100644
--- a/tools/perf/tests/Build
+++ b/tools/perf/tests/Build
@@ -26,6 +26,7 @@ perf-y += sw-clock.o
 perf-y += mmap-thread-lookup.o
 perf-y += thread-comm.o
 perf-y += thread-mg-share.o
+perf-y += thread-lookup-time.o
 perf-y += switch-tracking.o
 perf-y += keep-tracking.o
 perf-y += code-reading.o
diff --git a/tools/perf/tests/builtin-test.c b/tools/perf/tests/builtin-test.c
index 1b463d82a71a..e4d335de19ea 100644
--- a/tools/perf/tests/builtin-test.c
+++ b/tools/perf/tests/builtin-test.c
@@ -171,6 +171,10 @@ static struct test {
 		.func = test__thread_comm,
 	},
 	{
+		.desc = "Test thread lookup with time",
+		.func = test__thread_lookup_time,
+	},
+	{
 		.func = NULL,
 	},
 };
diff --git a/tools/perf/tests/tests.h b/tools/perf/tests/tests.h
index 43ac17780629..1090337f63e5 100644
--- a/tools/perf/tests/tests.h
+++ b/tools/perf/tests/tests.h
@@ -52,6 +52,7 @@ int test__switch_tracking(void);
 int test__fdarray__filter(void);
 int test__fdarray__add(void);
 int test__thread_comm(void);
+int test__thread_lookup_time(void);
 
 #if defined(__x86_64__) || defined(__i386__) || defined(__arm__)
 #ifdef HAVE_DWARF_UNWIND_SUPPORT
diff --git a/tools/perf/tests/thread-lookup-time.c b/tools/perf/tests/thread-lookup-time.c
new file mode 100644
index 000000000000..6237ecf8caae
--- /dev/null
+++ b/tools/perf/tests/thread-lookup-time.c
@@ -0,0 +1,174 @@
+#include "tests.h"
+#include "machine.h"
+#include "thread.h"
+#include "map.h"
+#include "debug.h"
+
+static int thread__print_cb(struct thread *th, void *arg __maybe_unused)
+{
+	printf("thread: %d, start time: %"PRIu64" %s\n",
+	       th->tid, th->start_time, th->dead ? "(dead)" : "");
+	return 0;
+}
+
+static int lookup_with_timestamp(struct machine *machine)
+{
+	struct thread *t1, *t2, *t3;
+	union perf_event fork = {
+		.fork = {
+			.pid = 0,
+			.tid = 0,
+			.ppid = 1,
+			.ptid = 1,
+		},
+	};
+	struct perf_sample sample = {
+		.time = 50000,
+	};
+
+	/* start_time is set to 0 */
+	t1 = machine__findnew_thread(machine, 0, 0);
+
+	if (verbose > 1) {
+		printf("========= after t1 created ==========\n");
+		machine__for_each_thread(machine, thread__print_cb, NULL);
+	}
+
+	TEST_ASSERT_VAL("wrong start time of old thread", t1->start_time == 0);
+
+	TEST_ASSERT_VAL("cannot find current thread",
+			machine__find_thread(machine, 0, 0) == t1);
+
+	TEST_ASSERT_VAL("cannot find current thread with time",
+			machine__findnew_thread_time(machine, 0, 0, 10000) == t1);
+
+	/* start_time is overwritten to new value */
+	thread__set_comm(t1, "/usr/bin/perf", 20000);
+
+	if (verbose > 1) {
+		printf("========= after t1 set comm ==========\n");
+		machine__for_each_thread(machine, thread__print_cb, NULL);
+	}
+
+	TEST_ASSERT_VAL("failed to update start time", t1->start_time == 20000);
+
+	TEST_ASSERT_VAL("should not find passed thread",
+			/* this will create yet another dead thread */
+			machine__findnew_thread_time(machine, 0, 0, 10000) != t1);
+
+	TEST_ASSERT_VAL("cannot find overwritten thread with time",
+			machine__find_thread_time(machine, 0, 0, 20000) == t1);
+
+	/* now t1 goes to dead thread tree, and create t2 */
+	machine__process_fork_event(machine, &fork, &sample);
+
+	if (verbose > 1) {
+		printf("========= after t2 forked ==========\n");
+		machine__for_each_thread(machine, thread__print_cb, NULL);
+	}
+
+	t2 = machine__find_thread(machine, 0, 0);
+	TEST_ASSERT_VAL("cannot find current thread", t2 != NULL);
+
+	TEST_ASSERT_VAL("wrong start time of new thread", t2->start_time == 50000);
+
+	TEST_ASSERT_VAL("dead thread cannot be found",
+			machine__find_thread_time(machine, 0, 0, 10000) != t1);
+
+	TEST_ASSERT_VAL("cannot find dead thread after new thread",
+			machine__find_thread_time(machine, 0, 0, 30000) == t1);
+
+	TEST_ASSERT_VAL("cannot find current thread after new thread",
+			machine__find_thread_time(machine, 0, 0, 50000) == t2);
+
+	/* now t2 goes to dead thread tree, and create t3 */
+	sample.time = 60000;
+	machine__process_fork_event(machine, &fork, &sample);
+
+	if (verbose > 1) {
+		printf("========= after t3 forked ==========\n");
+		machine__for_each_thread(machine, thread__print_cb, NULL);
+	}
+
+	t3 = machine__find_thread(machine, 0, 0);
+	TEST_ASSERT_VAL("cannot find current thread", t3 != NULL);
+
+	TEST_ASSERT_VAL("wrong start time of new thread", t3->start_time == 60000);
+
+	TEST_ASSERT_VAL("cannot find dead thread after new thread",
+			machine__findnew_thread_time(machine, 0, 0, 30000) == t1);
+
+	TEST_ASSERT_VAL("cannot find dead thread after new thread",
+			machine__findnew_thread_time(machine, 0, 0, 50000) == t2);
+
+	TEST_ASSERT_VAL("cannot find current thread after new thread",
+			machine__findnew_thread_time(machine, 0, 0, 70000) == t3);
+
+	machine__delete_threads(machine);
+	return 0;
+}
+
+static int lookup_without_timestamp(struct machine *machine)
+{
+	struct thread *t1, *t2, *t3;
+	union perf_event fork = {
+		.fork = {
+			.pid = 0,
+			.tid = 0,
+			.ppid = 1,
+			.ptid = 1,
+		},
+	};
+	struct perf_sample sample = {
+		.time = -1ULL,
+	};
+
+	t1 = machine__findnew_thread(machine, 0, 0);
+	TEST_ASSERT_VAL("cannot find current thread", t1 != NULL);
+
+	TEST_ASSERT_VAL("cannot find new thread with time",
+			machine__findnew_thread_time(machine, 0, 0, -1ULL) == t1);
+
+	machine__process_fork_event(machine, &fork, &sample);
+
+	t2 = machine__find_thread(machine, 0, 0);
+	TEST_ASSERT_VAL("cannot find current thread", t2 != NULL);
+
+	TEST_ASSERT_VAL("cannot find new thread with time",
+			machine__find_thread_time(machine, 0, 0, -1ULL) == t2);
+
+	machine__process_fork_event(machine, &fork, &sample);
+
+	t3 = machine__find_thread(machine, 0, 0);
+	TEST_ASSERT_VAL("cannot find current thread", t3 != NULL);
+
+	TEST_ASSERT_VAL("cannot find new thread with time",
+			machine__findnew_thread_time(machine, 0, 0, -1ULL) == t3);
+
+	machine__delete_threads(machine);
+	return 0;
+}
+
+int test__thread_lookup_time(void)
+{
+	struct machines machines;
+	struct machine *machine;
+
+	/*
+	 * This test is to check whether it can retrieve a correct
+	 * thread for a given time.  When multi-file data storage is
+	 * enabled, those task/comm/mmap events are processed first so
+	 * the later sample should find a matching thread properly.
+	 */
+	machines__init(&machines);
+	machine = &machines.host;
+
+	if (lookup_with_timestamp(machine) < 0)
+		return -1;
+
+	if (lookup_without_timestamp(machine) < 0)
+		return -1;
+
+	machines__exit(&machines);
+	return 0;
+}
-- 
2.2.2


^ permalink raw reply related	[flat|nested] 221+ messages in thread

* [PATCH 18/38] perf tools: Reducing arguments of hist_entry_iter__add()
  2015-03-03  3:07 [RFC/PATCHSET 00/38] perf tools: Speed-up perf report by using multi thread (v3) Namhyung Kim
                   ` (16 preceding siblings ...)
  2015-03-03  3:07 ` [PATCH 17/38] perf tools: Add a test case for timed thread handling Namhyung Kim
@ 2015-03-03  3:07 ` Namhyung Kim
  2015-03-03  3:07 ` [PATCH 19/38] perf tools: Pass session to hist_entry_iter struct Namhyung Kim
                   ` (19 subsequent siblings)
  37 siblings, 0 replies; 221+ messages in thread
From: Namhyung Kim @ 2015-03-03  3:07 UTC (permalink / raw)
  To: Arnaldo Carvalho de Melo
  Cc: Ingo Molnar, Peter Zijlstra, Jiri Olsa, LKML,
	Frederic Weisbecker, Adrian Hunter, Stephane Eranian, Andi Kleen,
	David Ahern

The evsel and sample arguments are to set iter for later use.  As it
also receives an iter as another argument, just set them before
calling the function.

Signed-off-by: Namhyung Kim <namhyung@kernel.org>
---
 tools/perf/builtin-report.c       | 9 +++++----
 tools/perf/builtin-top.c          | 7 ++++---
 tools/perf/tests/hists_cumulate.c | 6 ++++--
 tools/perf/tests/hists_filter.c   | 4 +++-
 tools/perf/tests/hists_output.c   | 6 ++++--
 tools/perf/util/hist.c            | 8 ++------
 tools/perf/util/hist.h            | 1 -
 7 files changed, 22 insertions(+), 19 deletions(-)

diff --git a/tools/perf/builtin-report.c b/tools/perf/builtin-report.c
index fe1f34c00c58..cff357522358 100644
--- a/tools/perf/builtin-report.c
+++ b/tools/perf/builtin-report.c
@@ -137,8 +137,10 @@ static int process_sample_event(struct perf_tool *tool,
 	struct report *rep = container_of(tool, struct report, tool);
 	struct addr_location al;
 	struct hist_entry_iter iter = {
-		.hide_unresolved = rep->hide_unresolved,
-		.add_entry_cb = hist_iter__report_callback,
+		.evsel 			= evsel,
+		.sample 		= sample,
+		.hide_unresolved 	= rep->hide_unresolved,
+		.add_entry_cb 		= hist_iter__report_callback,
 	};
 	int ret;
 
@@ -167,8 +169,7 @@ static int process_sample_event(struct perf_tool *tool,
 	if (al.map != NULL)
 		al.map->dso->hit = 1;
 
-	ret = hist_entry_iter__add(&iter, &al, evsel, sample, rep->max_stack,
-				   rep);
+	ret = hist_entry_iter__add(&iter, &al, rep->max_stack, rep);
 	if (ret < 0)
 		pr_debug("problem adding hist entry, skipping event\n");
 
diff --git a/tools/perf/builtin-top.c b/tools/perf/builtin-top.c
index 054c56206481..2c37bff901ba 100644
--- a/tools/perf/builtin-top.c
+++ b/tools/perf/builtin-top.c
@@ -774,7 +774,9 @@ static void perf_event__process_sample(struct perf_tool *tool,
 	if (al.sym == NULL || !al.sym->ignore) {
 		struct hists *hists = evsel__hists(evsel);
 		struct hist_entry_iter iter = {
-			.add_entry_cb = hist_iter__top_callback,
+			.evsel		= evsel,
+			.sample 	= sample,
+			.add_entry_cb 	= hist_iter__top_callback,
 		};
 
 		if (symbol_conf.cumulate_callchain)
@@ -784,8 +786,7 @@ static void perf_event__process_sample(struct perf_tool *tool,
 
 		pthread_mutex_lock(&hists->lock);
 
-		err = hist_entry_iter__add(&iter, &al, evsel, sample,
-					   top->max_stack, top);
+		err = hist_entry_iter__add(&iter, &al, top->max_stack, top);
 		if (err < 0)
 			pr_err("Problem incrementing symbol period, skipping event\n");
 
diff --git a/tools/perf/tests/hists_cumulate.c b/tools/perf/tests/hists_cumulate.c
index 60682e62d9de..da64acbd35b7 100644
--- a/tools/perf/tests/hists_cumulate.c
+++ b/tools/perf/tests/hists_cumulate.c
@@ -87,6 +87,8 @@ static int add_hist_entries(struct hists *hists, struct machine *machine)
 			},
 		};
 		struct hist_entry_iter iter = {
+			.evsel = evsel,
+			.sample	= &sample,
 			.hide_unresolved = false,
 		};
 
@@ -104,8 +106,8 @@ static int add_hist_entries(struct hists *hists, struct machine *machine)
 						  &sample, NULL) < 0)
 			goto out;
 
-		if (hist_entry_iter__add(&iter, &al, evsel, &sample,
-					 PERF_MAX_STACK_DEPTH, NULL) < 0)
+		if (hist_entry_iter__add(&iter, &al, PERF_MAX_STACK_DEPTH,
+					 NULL) < 0)
 			goto out;
 
 		fake_samples[i].thread = al.thread;
diff --git a/tools/perf/tests/hists_filter.c b/tools/perf/tests/hists_filter.c
index 1c4e495d5137..f5c0c69383dc 100644
--- a/tools/perf/tests/hists_filter.c
+++ b/tools/perf/tests/hists_filter.c
@@ -63,6 +63,8 @@ static int add_hist_entries(struct perf_evlist *evlist,
 				},
 			};
 			struct hist_entry_iter iter = {
+				.evsel = evsel,
+				.sample = &sample,
 				.ops = &hist_iter_normal,
 				.hide_unresolved = false,
 			};
@@ -81,7 +83,7 @@ static int add_hist_entries(struct perf_evlist *evlist,
 							  &sample, NULL) < 0)
 				goto out;
 
-			if (hist_entry_iter__add(&iter, &al, evsel, &sample,
+			if (hist_entry_iter__add(&iter, &al,
 						 PERF_MAX_STACK_DEPTH, NULL) < 0)
 				goto out;
 
diff --git a/tools/perf/tests/hists_output.c b/tools/perf/tests/hists_output.c
index f4e3286cd496..4e3cff568eaa 100644
--- a/tools/perf/tests/hists_output.c
+++ b/tools/perf/tests/hists_output.c
@@ -57,6 +57,8 @@ static int add_hist_entries(struct hists *hists, struct machine *machine)
 			},
 		};
 		struct hist_entry_iter iter = {
+			.evsel = evsel,
+			.sample = &sample,
 			.ops = &hist_iter_normal,
 			.hide_unresolved = false,
 		};
@@ -70,8 +72,8 @@ static int add_hist_entries(struct hists *hists, struct machine *machine)
 						  &sample, NULL) < 0)
 			goto out;
 
-		if (hist_entry_iter__add(&iter, &al, evsel, &sample,
-					 PERF_MAX_STACK_DEPTH, NULL) < 0)
+		if (hist_entry_iter__add(&iter, &al, PERF_MAX_STACK_DEPTH,
+					 NULL) < 0)
 			goto out;
 
 		fake_samples[i].thread = al.thread;
diff --git a/tools/perf/util/hist.c b/tools/perf/util/hist.c
index 4badf2491fbf..0553a14a80a4 100644
--- a/tools/perf/util/hist.c
+++ b/tools/perf/util/hist.c
@@ -857,19 +857,15 @@ const struct hist_iter_ops hist_iter_cumulative = {
 };
 
 int hist_entry_iter__add(struct hist_entry_iter *iter, struct addr_location *al,
-			 struct perf_evsel *evsel, struct perf_sample *sample,
 			 int max_stack_depth, void *arg)
 {
 	int err, err2;
 
-	err = sample__resolve_callchain(sample, &iter->parent, evsel, al,
-					max_stack_depth);
+	err = sample__resolve_callchain(iter->sample, &iter->parent,
+					iter->evsel, al, max_stack_depth);
 	if (err)
 		return err;
 
-	iter->evsel = evsel;
-	iter->sample = sample;
-
 	err = iter->ops->prepare_entry(iter, al);
 	if (err)
 		goto out;
diff --git a/tools/perf/util/hist.h b/tools/perf/util/hist.h
index 0eed50a5b1f0..0098aad4a23c 100644
--- a/tools/perf/util/hist.h
+++ b/tools/perf/util/hist.h
@@ -111,7 +111,6 @@ struct hist_entry *__hists__add_entry(struct hists *hists,
 				      u64 weight, u64 transaction,
 				      u64 timestamp, bool sample_self);
 int hist_entry_iter__add(struct hist_entry_iter *iter, struct addr_location *al,
-			 struct perf_evsel *evsel, struct perf_sample *sample,
 			 int max_stack_depth, void *arg);
 
 int64_t hist_entry__cmp(struct hist_entry *left, struct hist_entry *right);
-- 
2.2.2


^ permalink raw reply related	[flat|nested] 221+ messages in thread

* [PATCH 19/38] perf tools: Pass session to hist_entry_iter struct
  2015-03-03  3:07 [RFC/PATCHSET 00/38] perf tools: Speed-up perf report by using multi thread (v3) Namhyung Kim
                   ` (17 preceding siblings ...)
  2015-03-03  3:07 ` [PATCH 18/38] perf tools: Reducing arguments of hist_entry_iter__add() Namhyung Kim
@ 2015-03-03  3:07 ` Namhyung Kim
  2015-03-03  3:07 ` [PATCH 20/38] perf tools: Maintain map groups list in a leader thread Namhyung Kim
                   ` (18 subsequent siblings)
  37 siblings, 0 replies; 221+ messages in thread
From: Namhyung Kim @ 2015-03-03  3:07 UTC (permalink / raw)
  To: Arnaldo Carvalho de Melo
  Cc: Ingo Molnar, Peter Zijlstra, Jiri Olsa, LKML,
	Frederic Weisbecker, Adrian Hunter, Stephane Eranian, Andi Kleen,
	David Ahern

The session is necessary to determine whether this is an indexed data
so that it needs to use timestamp for searching threads/symbols.

Signed-off-by: Namhyung Kim <namhyung@kernel.org>
---
 tools/perf/builtin-report.c | 1 +
 tools/perf/builtin-top.c    | 1 +
 tools/perf/util/hist.h      | 1 +
 3 files changed, 3 insertions(+)

diff --git a/tools/perf/builtin-report.c b/tools/perf/builtin-report.c
index cff357522358..0d6e6bff7994 100644
--- a/tools/perf/builtin-report.c
+++ b/tools/perf/builtin-report.c
@@ -139,6 +139,7 @@ static int process_sample_event(struct perf_tool *tool,
 	struct hist_entry_iter iter = {
 		.evsel 			= evsel,
 		.sample 		= sample,
+		.session 		= rep->session,
 		.hide_unresolved 	= rep->hide_unresolved,
 		.add_entry_cb 		= hist_iter__report_callback,
 	};
diff --git a/tools/perf/builtin-top.c b/tools/perf/builtin-top.c
index 2c37bff901ba..f33cb0e2aa0d 100644
--- a/tools/perf/builtin-top.c
+++ b/tools/perf/builtin-top.c
@@ -776,6 +776,7 @@ static void perf_event__process_sample(struct perf_tool *tool,
 		struct hist_entry_iter iter = {
 			.evsel		= evsel,
 			.sample 	= sample,
+			.session 	= top->session,
 			.add_entry_cb 	= hist_iter__top_callback,
 		};
 
diff --git a/tools/perf/util/hist.h b/tools/perf/util/hist.h
index 0098aad4a23c..0afe15ba0277 100644
--- a/tools/perf/util/hist.h
+++ b/tools/perf/util/hist.h
@@ -88,6 +88,7 @@ struct hist_entry_iter {
 
 	struct perf_evsel *evsel;
 	struct perf_sample *sample;
+	struct perf_session *session;
 	struct hist_entry *he;
 	struct symbol *parent;
 	void *priv;
-- 
2.2.2


^ permalink raw reply related	[flat|nested] 221+ messages in thread

* [PATCH 20/38] perf tools: Maintain map groups list in a leader thread
  2015-03-03  3:07 [RFC/PATCHSET 00/38] perf tools: Speed-up perf report by using multi thread (v3) Namhyung Kim
                   ` (18 preceding siblings ...)
  2015-03-03  3:07 ` [PATCH 19/38] perf tools: Pass session to hist_entry_iter struct Namhyung Kim
@ 2015-03-03  3:07 ` Namhyung Kim
  2015-03-03  3:07 ` [PATCH 21/38] perf tools: Introduce session__find_addr_location() and friends Namhyung Kim
                   ` (17 subsequent siblings)
  37 siblings, 0 replies; 221+ messages in thread
From: Namhyung Kim @ 2015-03-03  3:07 UTC (permalink / raw)
  To: Arnaldo Carvalho de Melo
  Cc: Ingo Molnar, Peter Zijlstra, Jiri Olsa, LKML,
	Frederic Weisbecker, Adrian Hunter, Stephane Eranian, Andi Kleen,
	David Ahern

To support multi-threaded perf report, we need to maintain time-sorted
map groups.  Add ->mg_list member to struct thread and sort the list
by time.  Now leader threads have one more refcnt for map groups in
the list so also update the thread-mg-share test case.

Currently only add a new map groups when an exec (comm) event is
received.

Cc: Frederic Weisbecker <fweisbec@gmail.com>
Signed-off-by: Namhyung Kim <namhyung@kernel.org>
---
 tools/perf/tests/thread-mg-share.c |   7 ++-
 tools/perf/util/event.c            |   2 +
 tools/perf/util/machine.c          |   4 +-
 tools/perf/util/map.c              |   3 ++
 tools/perf/util/map.h              |   2 +
 tools/perf/util/thread.c           | 108 ++++++++++++++++++++++++++++++++++++-
 tools/perf/util/thread.h           |   3 ++
 7 files changed, 124 insertions(+), 5 deletions(-)

diff --git a/tools/perf/tests/thread-mg-share.c b/tools/perf/tests/thread-mg-share.c
index b028499dd3cf..8933e01d0549 100644
--- a/tools/perf/tests/thread-mg-share.c
+++ b/tools/perf/tests/thread-mg-share.c
@@ -23,6 +23,9 @@ int test__thread_mg_share(void)
 	 * with several threads and checks they properly share and
 	 * maintain map groups info (struct map_groups).
 	 *
+	 * Note that a leader thread has one more refcnt for its
+	 * (current) map groups.
+	 *
 	 * thread group (pid: 0, tids: 0, 1, 2, 3)
 	 * other  group (pid: 4, tids: 4, 5)
 	*/
@@ -43,7 +46,7 @@ int test__thread_mg_share(void)
 			leader && t1 && t2 && t3 && other);
 
 	mg = leader->mg;
-	TEST_ASSERT_VAL("wrong refcnt", mg->refcnt == 4);
+	TEST_ASSERT_VAL("wrong refcnt", mg->refcnt == 5);
 
 	/* test the map groups pointer is shared */
 	TEST_ASSERT_VAL("map groups don't match", mg == t1->mg);
@@ -59,7 +62,7 @@ int test__thread_mg_share(void)
 	TEST_ASSERT_VAL("failed to find other leader", other_leader);
 
 	other_mg = other->mg;
-	TEST_ASSERT_VAL("wrong refcnt", other_mg->refcnt == 2);
+	TEST_ASSERT_VAL("wrong refcnt", other_mg->refcnt == 3);
 
 	TEST_ASSERT_VAL("map groups don't match", other_mg == other_leader->mg);
 
diff --git a/tools/perf/util/event.c b/tools/perf/util/event.c
index 3bfe10fe0c69..8d4a5cb829d0 100644
--- a/tools/perf/util/event.c
+++ b/tools/perf/util/event.c
@@ -765,6 +765,8 @@ void thread__find_addr_map(struct thread *thread, u8 cpumode,
 		return;
 	}
 
+	BUG_ON(mg == NULL);
+
 	if (cpumode == PERF_RECORD_MISC_KERNEL && perf_host) {
 		al->level = 'k';
 		mg = &machine->kmaps;
diff --git a/tools/perf/util/machine.c b/tools/perf/util/machine.c
index b4b97b5e1f1c..b14e4dc5261d 100644
--- a/tools/perf/util/machine.c
+++ b/tools/perf/util/machine.c
@@ -331,7 +331,7 @@ static void machine__update_thread_pid(struct machine *machine,
 		goto out_err;
 
 	if (!leader->mg)
-		leader->mg = map_groups__new(machine);
+		thread__set_map_groups(leader, map_groups__new(machine), 0);
 
 	if (!leader->mg)
 		goto out_err;
@@ -348,7 +348,7 @@ static void machine__update_thread_pid(struct machine *machine,
 		if (!map_groups__empty(th->mg))
 			pr_err("Discarding thread maps for %d:%d\n",
 			       th->pid_, th->tid);
-		map_groups__delete(th->mg);
+		map_groups__put(th->mg);
 	}
 
 	th->mg = map_groups__get(leader->mg);
diff --git a/tools/perf/util/map.c b/tools/perf/util/map.c
index 62ca9f2607d5..85fbb1b3e69f 100644
--- a/tools/perf/util/map.c
+++ b/tools/perf/util/map.c
@@ -422,6 +422,8 @@ void map_groups__init(struct map_groups *mg, struct machine *machine)
 	}
 	mg->machine = machine;
 	mg->refcnt = 1;
+	mg->timestamp = 0;
+	INIT_LIST_HEAD(&mg->list);
 }
 
 static void maps__delete(struct rb_root *maps)
@@ -484,6 +486,7 @@ struct map_groups *map_groups__new(struct machine *machine)
 void map_groups__delete(struct map_groups *mg)
 {
 	map_groups__exit(mg);
+	list_del(&mg->list);
 	free(mg);
 }
 
diff --git a/tools/perf/util/map.h b/tools/perf/util/map.h
index 0e42438b1e59..f33d49029ac0 100644
--- a/tools/perf/util/map.h
+++ b/tools/perf/util/map.h
@@ -61,7 +61,9 @@ struct map_groups {
 	struct rb_root	 maps[MAP__NR_TYPES];
 	struct list_head removed_maps[MAP__NR_TYPES];
 	struct machine	 *machine;
+	u64		 timestamp;
 	int		 refcnt;
+	struct list_head list;
 };
 
 struct map_groups *map_groups__new(struct machine *machine);
diff --git a/tools/perf/util/thread.c b/tools/perf/util/thread.c
index 306bdaede019..7b01c171dcfa 100644
--- a/tools/perf/util/thread.c
+++ b/tools/perf/util/thread.c
@@ -10,13 +10,76 @@
 #include "comm.h"
 #include "unwind.h"
 
+struct map_groups *thread__get_map_groups(struct thread *thread, u64 timestamp)
+{
+	struct map_groups *mg;
+	struct thread *leader = thread;
+
+	BUG_ON(thread->mg == NULL);
+
+	if (thread->tid != thread->pid_) {
+		leader = machine__find_thread_time(thread->mg->machine,
+						   thread->pid_, thread->pid_,
+						   timestamp);
+		if (leader == NULL)
+			goto out;
+	}
+
+	list_for_each_entry(mg, &leader->mg_list, list)
+		if (timestamp >= mg->timestamp)
+			return mg;
+
+out:
+	return thread->mg;
+}
+
+int thread__set_map_groups(struct thread *thread, struct map_groups *mg,
+			   u64 timestamp)
+{
+	struct list_head *pos;
+	struct map_groups *old;
+
+	if (mg == NULL)
+		return -ENOMEM;
+
+	/*
+	 * Only a leader thread can have map groups list - others
+	 * reference it through map_groups__get.  This means the
+	 * leader thread will have one more refcnt than others.
+	 */
+	if (thread->tid != thread->pid_)
+		return -EINVAL;
+
+	if (thread->mg) {
+		BUG_ON(thread->mg->refcnt <= 1);
+		map_groups__put(thread->mg);
+	}
+
+	/* sort by time */
+	list_for_each(pos, &thread->mg_list) {
+		old = list_entry(pos, struct map_groups, list);
+		if (timestamp > old->timestamp)
+			break;
+	}
+
+	list_add_tail(&mg->list, pos);
+	mg->timestamp = timestamp;
+
+	/* set current ->mg to most recent one */
+	thread->mg = list_first_entry(&thread->mg_list, struct map_groups, list);
+	/* increase one more refcnt for current */
+	map_groups__get(thread->mg);
+
+	return 0;
+}
+
 int thread__init_map_groups(struct thread *thread, struct machine *machine)
 {
 	struct thread *leader;
 	pid_t pid = thread->pid_;
 
 	if (pid == thread->tid || pid == -1) {
-		thread->mg = map_groups__new(machine);
+		thread__set_map_groups(thread, map_groups__new(machine), 0);
 	} else {
 		leader = machine__findnew_thread(machine, pid, pid);
 		if (leader)
@@ -39,6 +102,7 @@ struct thread *thread__new(pid_t pid, pid_t tid)
 		thread->cpu = -1;
 		INIT_LIST_HEAD(&thread->comm_list);
 		INIT_LIST_HEAD(&thread->tid_node);
+		INIT_LIST_HEAD(&thread->mg_list);
 
 		if (unwind__prepare_access(thread) < 0)
 			goto err_thread;
@@ -67,6 +131,7 @@ struct thread *thread__new(pid_t pid, pid_t tid)
 void thread__delete(struct thread *thread)
 {
 	struct comm *comm, *tmp;
+	struct map_groups *mg, *tmp_mg;
 
 	thread_stack__free(thread);
 
@@ -74,6 +139,10 @@ void thread__delete(struct thread *thread)
 		map_groups__put(thread->mg);
 		thread->mg = NULL;
 	}
+	/* only leader threads have mg list */
+	list_for_each_entry_safe(mg, tmp_mg, &thread->mg_list, list)
+		map_groups__put(mg);
+
 	list_for_each_entry_safe(comm, tmp, &thread->comm_list, list) {
 		list_del(&comm->list);
 		comm__free(comm);
@@ -119,6 +188,9 @@ struct comm *thread__comm_time(const struct thread *thread, u64 timestamp)
 	return list_last_entry(&thread->comm_list, struct comm, list);
 }
 
+static int thread__clone_map_groups(struct thread *thread,
+				    struct thread *parent);
+
 int __thread__set_comm(struct thread *thread, const char *str, u64 timestamp,
 		       bool exec)
 {
@@ -149,6 +221,40 @@ int __thread__set_comm(struct thread *thread, const char *str, u64 timestamp,
 			unwind__flush_access(thread);
 	}
 
+	if (exec) {
+		struct machine *machine;
+
+		BUG_ON(thread->mg == NULL || thread->mg->machine == NULL);
+
+		machine = thread->mg->machine;
+
+		if (thread->tid != thread->pid_) {
+			struct map_groups *old = thread->mg;
+			struct thread *leader;
+
+			leader = machine__findnew_thread(machine, thread->pid_,
+							 thread->pid_);
+
+			/* now it'll be a new leader */
+			thread->pid_ = thread->tid;
+
+			thread->mg = map_groups__new(old->machine);
+			if (thread->mg == NULL)
+				return -ENOMEM;
+
+			/* save current mg in the new leader */
+			thread__clone_map_groups(thread, leader);
+
+			/* current mg of leader thread needs one more refcnt */
+			map_groups__get(thread->mg);
+
+			thread__set_map_groups(thread, thread->mg, old->timestamp);
+		}
+
+		/* create a new mg for newly executed binary */
+		thread__set_map_groups(thread, map_groups__new(machine), timestamp);
+	}
+
 	thread->comm_set = true;
 
 	return 0;
diff --git a/tools/perf/util/thread.h b/tools/perf/util/thread.h
index e5d7abd255ea..08cafa2d97f9 100644
--- a/tools/perf/util/thread.h
+++ b/tools/perf/util/thread.h
@@ -14,6 +14,7 @@ struct thread {
 	struct rb_node	 	rb_node;
 	struct list_head 	tid_node;
 	struct map_groups	*mg;
+	struct list_head	mg_list;
 	pid_t			pid_; /* Not all tools update this */
 	pid_t			tid;
 	pid_t			ppid;
@@ -56,6 +57,8 @@ struct comm *thread__exec_comm(const struct thread *thread);
 struct comm *thread__comm_time(const struct thread *thread, u64 timestamp);
 const char *thread__comm_str(const struct thread *thread);
 const char *thread__comm_str_time(const struct thread *thread, u64 timestamp);
+struct map_groups *thread__get_map_groups(struct thread *thread, u64 timestamp);
+int thread__set_map_groups(struct thread *thread, struct map_groups *mg, u64 timestamp);
 void thread__insert_map(struct thread *thread, struct map *map);
 int thread__fork(struct thread *thread, struct thread *parent, u64 timestamp);
 size_t thread__fprintf(struct thread *thread, FILE *fp);
-- 
2.2.2


^ permalink raw reply related	[flat|nested] 221+ messages in thread

* [PATCH 21/38] perf tools: Introduce session__find_addr_location() and friends
  2015-03-03  3:07 [RFC/PATCHSET 00/38] perf tools: Speed-up perf report by using multi thread (v3) Namhyung Kim
                   ` (19 preceding siblings ...)
  2015-03-03  3:07 ` [PATCH 20/38] perf tools: Maintain map groups list in a leader thread Namhyung Kim
@ 2015-03-03  3:07 ` Namhyung Kim
  2015-03-03  3:07 ` [PATCH 22/38] perf callchain: Use " Namhyung Kim
                   ` (16 subsequent siblings)
  37 siblings, 0 replies; 221+ messages in thread
From: Namhyung Kim @ 2015-03-03  3:07 UTC (permalink / raw)
  To: Arnaldo Carvalho de Melo
  Cc: Ingo Molnar, Peter Zijlstra, Jiri Olsa, LKML,
	Frederic Weisbecker, Adrian Hunter, Stephane Eranian, Andi Kleen,
	David Ahern

These new functions are for find appropriate map (and symbol) at the
given time when used with an indexed data file.  This is based on the
fact that map_groups list is sorted by time in the previous patch.

Cc: Frederic Weisbecker <fweisbec@gmail.com>
Signed-off-by: Namhyung Kim <namhyung@kernel.org>
---
 tools/perf/util/event.c   | 57 ++++++++++++++++++++++++++++++++++++++---------
 tools/perf/util/hist.c    |  4 ++--
 tools/perf/util/machine.c | 37 ++++++++++++++++++++----------
 tools/perf/util/machine.h |  2 ++
 tools/perf/util/session.h | 38 +++++++++++++++++++++++++++++++
 tools/perf/util/thread.c  | 21 +++++++++++++++++
 tools/perf/util/thread.h  | 10 +++++++++
 7 files changed, 145 insertions(+), 24 deletions(-)

diff --git a/tools/perf/util/event.c b/tools/perf/util/event.c
index 8d4a5cb829d0..5abf7086c97c 100644
--- a/tools/perf/util/event.c
+++ b/tools/perf/util/event.c
@@ -746,16 +746,14 @@ int perf_event__process(struct perf_tool *tool __maybe_unused,
 	return machine__process_event(machine, event, sample);
 }
 
-void thread__find_addr_map(struct thread *thread, u8 cpumode,
-			   enum map_type type, u64 addr,
-			   struct addr_location *al)
+static void map_groups__find_addr_map(struct map_groups *mg, u8 cpumode,
+				      enum map_type type, u64 addr,
+				      struct addr_location *al)
 {
-	struct map_groups *mg = thread->mg;
 	struct machine *machine = mg->machine;
 	bool load_map = false;
 
 	al->machine = machine;
-	al->thread = thread;
 	al->addr = addr;
 	al->cpumode = cpumode;
 	al->filtered = 0;
@@ -824,6 +822,26 @@ void thread__find_addr_map(struct thread *thread, u8 cpumode,
 	}
 }
 
+void thread__find_addr_map(struct thread *thread, u8 cpumode,
+			   enum map_type type, u64 addr,
+			   struct addr_location *al)
+{
+	al->thread = thread;
+	map_groups__find_addr_map(thread->mg, cpumode, type, addr, al);
+}
+
+void thread__find_addr_map_time(struct thread *thread, u8 cpumode,
+				enum map_type type, u64 addr,
+				struct addr_location *al, u64 timestamp)
+{
+	struct map_groups *mg;
+
+	mg = thread__get_map_groups(thread, timestamp);
+
+	al->thread = thread;
+	map_groups__find_addr_map(mg, cpumode, type, addr, al);
+}
+
 void thread__find_addr_location(struct thread *thread,
 				u8 cpumode, enum map_type type, u64 addr,
 				struct addr_location *al)
@@ -836,6 +854,21 @@ void thread__find_addr_location(struct thread *thread,
 		al->sym = NULL;
 }
 
+void thread__find_addr_location_time(struct thread *thread, u8 cpumode,
+				     enum map_type type, u64 addr,
+				     struct addr_location *al, u64 timestamp)
+{
+	struct map_groups *mg;
+
+	mg = thread__get_map_groups(thread, timestamp);
+	map_groups__find_addr_map(mg, cpumode, type, addr, al);
+	if (al->map != NULL)
+		al->sym = map__find_symbol(al->map, al->addr,
+					   mg->machine->symbol_filter);
+	else
+		al->sym = NULL;
+}
+
 int perf_event__preprocess_sample(const union perf_event *event,
 				  struct machine *machine,
 				  struct addr_location *al,
@@ -868,7 +901,9 @@ int perf_event__preprocess_sample(const union perf_event *event,
 	    machine->vmlinux_maps[MAP__FUNCTION] == NULL)
 		machine__create_kernel_maps(machine);
 
-	thread__find_addr_map(thread, cpumode, MAP__FUNCTION, sample->ip, al);
+	session__find_addr_map(session, thread, cpumode, MAP__FUNCTION,
+			       sample->ip, al, sample->time);
+
 	dump_printf(" ...... dso: %s\n",
 		    al->map ? al->map->dso->long_name :
 			al->level == 'H' ? "[hypervisor]" : "<not found>");
@@ -929,14 +964,16 @@ void perf_event__preprocess_sample_addr(union perf_event *event,
 					struct perf_sample *sample,
 					struct thread *thread,
 					struct addr_location *al,
-					struct perf_session *session __maybe_unused)
+					struct perf_session *session)
 {
 	u8 cpumode = event->header.misc & PERF_RECORD_MISC_CPUMODE_MASK;
 
-	thread__find_addr_map(thread, cpumode, MAP__FUNCTION, sample->addr, al);
+	session__find_addr_map(session, thread, cpumode, MAP__FUNCTION,
+			       sample->addr, al, sample->time);
+
 	if (!al->map)
-		thread__find_addr_map(thread, cpumode, MAP__VARIABLE,
-				      sample->addr, al);
+		session__find_addr_map(session, thread, cpumode, MAP__VARIABLE,
+				       sample->addr, al, sample->time);
 
 	al->cpu = sample->cpu;
 	al->sym = NULL;
diff --git a/tools/perf/util/hist.c b/tools/perf/util/hist.c
index 0553a14a80a4..0d189ae76922 100644
--- a/tools/perf/util/hist.c
+++ b/tools/perf/util/hist.c
@@ -496,7 +496,7 @@ iter_prepare_mem_entry(struct hist_entry_iter *iter, struct addr_location *al)
 	struct perf_sample *sample = iter->sample;
 	struct mem_info *mi;
 
-	mi = sample__resolve_mem(sample, al);
+	mi = sample__resolve_mem(sample, iter->session, al);
 	if (mi == NULL)
 		return -ENOMEM;
 
@@ -570,7 +570,7 @@ iter_prepare_branch_entry(struct hist_entry_iter *iter, struct addr_location *al
 	struct branch_info *bi;
 	struct perf_sample *sample = iter->sample;
 
-	bi = sample__resolve_bstack(sample, al);
+	bi = sample__resolve_bstack(sample, iter->session, al);
 	if (!bi)
 		return -ENOMEM;
 
diff --git a/tools/perf/util/machine.c b/tools/perf/util/machine.c
index b14e4dc5261d..4743718d4bf1 100644
--- a/tools/perf/util/machine.c
+++ b/tools/perf/util/machine.c
@@ -8,6 +8,7 @@
 #include "sort.h"
 #include "strlist.h"
 #include "thread.h"
+#include "session.h"
 #include "vdso.h"
 #include <stdbool.h>
 #include <symbol/kallsyms.h>
@@ -1469,9 +1470,10 @@ static bool symbol__match_regex(struct symbol *sym, regex_t *regex)
 	return 0;
 }
 
-static void ip__resolve_ams(struct thread *thread,
+static void ip__resolve_ams(struct perf_session *session,
+			    struct thread *thread,
 			    struct addr_map_symbol *ams,
-			    u64 ip)
+			    u64 ip, u64 timestamp)
 {
 	struct addr_location al;
 
@@ -1483,7 +1485,8 @@ static void ip__resolve_ams(struct thread *thread,
 	 * Thus, we have to try consecutively until we find a match
 	 * or else, the symbol is unknown
 	 */
-	thread__find_cpumode_addr_location(thread, MAP__FUNCTION, ip, &al);
+	session__find_cpumode_addr_location(session, thread, MAP__FUNCTION,
+					    ip, &al, timestamp);
 
 	ams->addr = ip;
 	ams->al_addr = al.addr;
@@ -1491,21 +1494,25 @@ static void ip__resolve_ams(struct thread *thread,
 	ams->map = al.map;
 }
 
-static void ip__resolve_data(struct thread *thread,
-			     u8 m, struct addr_map_symbol *ams, u64 addr)
+static void ip__resolve_data(struct perf_session *session,
+			     struct thread *thread, u8 m,
+			     struct addr_map_symbol *ams,
+			     u64 addr, u64 timestamp)
 {
 	struct addr_location al;
 
 	memset(&al, 0, sizeof(al));
 
-	thread__find_addr_location(thread, m, MAP__VARIABLE, addr, &al);
+	session__find_addr_location(session, thread, m, MAP__VARIABLE, addr,
+				    &al, timestamp);
 	if (al.map == NULL) {
 		/*
 		 * some shared data regions have execute bit set which puts
 		 * their mapping in the MAP__FUNCTION type array.
 		 * Check there as a fallback option before dropping the sample.
 		 */
-		thread__find_addr_location(thread, m, MAP__FUNCTION, addr, &al);
+		session__find_addr_location(session, thread, m, MAP__FUNCTION,
+					    addr, &al, timestamp);
 	}
 
 	ams->addr = addr;
@@ -1515,6 +1522,7 @@ static void ip__resolve_data(struct thread *thread,
 }
 
 struct mem_info *sample__resolve_mem(struct perf_sample *sample,
+				     struct perf_session *session,
 				     struct addr_location *al)
 {
 	struct mem_info *mi = zalloc(sizeof(*mi));
@@ -1522,8 +1530,10 @@ struct mem_info *sample__resolve_mem(struct perf_sample *sample,
 	if (!mi)
 		return NULL;
 
-	ip__resolve_ams(al->thread, &mi->iaddr, sample->ip);
-	ip__resolve_data(al->thread, al->cpumode, &mi->daddr, sample->addr);
+	ip__resolve_ams(session, al->thread, &mi->iaddr, sample->ip,
+			sample->time);
+	ip__resolve_data(session, al->thread, al->cpumode, &mi->daddr,
+			 sample->addr, sample->time);
 	mi->data_src.val = sample->data_src;
 
 	return mi;
@@ -1539,6 +1549,7 @@ static int add_callchain_ip(struct thread *thread,
 
 	al.filtered = 0;
 	al.sym = NULL;
+
 	if (branch_history)
 		thread__find_cpumode_addr_location(thread, MAP__FUNCTION,
 						   ip, &al);
@@ -1589,6 +1600,7 @@ static int add_callchain_ip(struct thread *thread,
 }
 
 struct branch_info *sample__resolve_bstack(struct perf_sample *sample,
+					   struct perf_session *session,
 					   struct addr_location *al)
 {
 	unsigned int i;
@@ -1599,8 +1611,10 @@ struct branch_info *sample__resolve_bstack(struct perf_sample *sample,
 		return NULL;
 
 	for (i = 0; i < bs->nr; i++) {
-		ip__resolve_ams(al->thread, &bi[i].to, bs->entries[i].to);
-		ip__resolve_ams(al->thread, &bi[i].from, bs->entries[i].from);
+		ip__resolve_ams(session, al->thread, &bi[i].to,
+				bs->entries[i].to, sample->time);
+		ip__resolve_ams(session, al->thread, &bi[i].from,
+				bs->entries[i].from, sample->time);
 		bi[i].flags = bs->entries[i].flags;
 	}
 	return bi;
@@ -1826,7 +1840,6 @@ static int thread__resolve_callchain_sample(struct thread *thread,
 		ip = chain->ips[j];
 
 		err = add_callchain_ip(thread, parent, root_al, false, ip);
-
 		if (err)
 			return (err < 0) ? err : 0;
 	}
diff --git a/tools/perf/util/machine.h b/tools/perf/util/machine.h
index 9571b6b1c5b5..45aee0c329ef 100644
--- a/tools/perf/util/machine.h
+++ b/tools/perf/util/machine.h
@@ -121,8 +121,10 @@ void machine__delete_threads(struct machine *machine);
 void machine__delete(struct machine *machine);
 
 struct branch_info *sample__resolve_bstack(struct perf_sample *sample,
+					   struct perf_session *session,
 					   struct addr_location *al);
 struct mem_info *sample__resolve_mem(struct perf_sample *sample,
+				     struct perf_session *session,
 				     struct addr_location *al);
 int thread__resolve_callchain(struct thread *thread,
 			      struct perf_evsel *evsel,
diff --git a/tools/perf/util/session.h b/tools/perf/util/session.h
index aff0d2b4cc0b..dbd21f8e7cf1 100644
--- a/tools/perf/util/session.h
+++ b/tools/perf/util/session.h
@@ -142,4 +142,42 @@ static inline bool perf_session__has_index(struct perf_session *session)
 	return perf_header__has_feat(&session->header, HEADER_DATA_INDEX);
 }
 
+static inline void
+session__find_addr_map(struct perf_session *session, struct thread *thread,
+		       u8 cpumode, enum map_type type, u64 addr,
+		       struct addr_location *al, u64 timestamp)
+{
+	if (session && perf_session__has_index(session))
+		thread__find_addr_map_time(thread, cpumode, type, addr, al,
+					   timestamp);
+	else
+		thread__find_addr_map(thread, cpumode, type, addr, al);
+}
+
+static inline void
+session__find_addr_location(struct perf_session *session, struct thread *thread,
+			    u8 cpumode, enum map_type type, u64 addr,
+			    struct addr_location *al, u64 timestamp)
+{
+	if (session && perf_session__has_index(session))
+		thread__find_addr_location_time(thread, cpumode, type, addr, al,
+						timestamp);
+	else
+		thread__find_addr_location(thread, cpumode, type, addr, al);
+}
+
+static inline void
+session__find_cpumode_addr_location(struct perf_session *session,
+				    struct thread *thread, enum map_type type,
+				    u64 addr, struct addr_location *al,
+				    u64 timestamp)
+{
+	if (session && perf_session__has_index(session))
+		thread__find_cpumode_addr_location_time(thread, type, addr, al,
+							timestamp);
+	else
+		thread__find_cpumode_addr_location(thread, type, addr, al);
+}
+
+
 #endif /* __PERF_SESSION_H */
diff --git a/tools/perf/util/thread.c b/tools/perf/util/thread.c
index 7b01c171dcfa..9ae1ce8606af 100644
--- a/tools/perf/util/thread.c
+++ b/tools/perf/util/thread.c
@@ -359,3 +359,24 @@ void thread__find_cpumode_addr_location(struct thread *thread,
 			break;
 	}
 }
+
+void thread__find_cpumode_addr_location_time(struct thread *thread,
+					     enum map_type type, u64 addr,
+					     struct addr_location *al,
+					     u64 timestamp)
+{
+	size_t i;
+	const u8 const cpumodes[] = {
+		PERF_RECORD_MISC_USER,
+		PERF_RECORD_MISC_KERNEL,
+		PERF_RECORD_MISC_GUEST_USER,
+		PERF_RECORD_MISC_GUEST_KERNEL
+	};
+
+	for (i = 0; i < ARRAY_SIZE(cpumodes); i++) {
+		thread__find_addr_location_time(thread, cpumodes[i], type,
+						addr, al, timestamp);
+		if (al->map)
+			break;
+	}
+}
diff --git a/tools/perf/util/thread.h b/tools/perf/util/thread.h
index 08cafa2d97f9..5209ad5adadf 100644
--- a/tools/perf/util/thread.h
+++ b/tools/perf/util/thread.h
@@ -66,14 +66,24 @@ size_t thread__fprintf(struct thread *thread, FILE *fp);
 void thread__find_addr_map(struct thread *thread,
 			   u8 cpumode, enum map_type type, u64 addr,
 			   struct addr_location *al);
+void thread__find_addr_map_time(struct thread *thread, u8 cpumode,
+				enum map_type type, u64 addr,
+				struct addr_location *al, u64 timestamp);
 
 void thread__find_addr_location(struct thread *thread,
 				u8 cpumode, enum map_type type, u64 addr,
 				struct addr_location *al);
+void thread__find_addr_location_time(struct thread *thread, u8 cpumode,
+				     enum map_type type, u64 addr,
+				     struct addr_location *al, u64 timestamp);
 
 void thread__find_cpumode_addr_location(struct thread *thread,
 					enum map_type type, u64 addr,
 					struct addr_location *al);
+void thread__find_cpumode_addr_location_time(struct thread *thread,
+					     enum map_type type, u64 addr,
+					     struct addr_location *al,
+					     u64 timestamp);
 
 static inline void *thread__priv(struct thread *thread)
 {
-- 
2.2.2


^ permalink raw reply related	[flat|nested] 221+ messages in thread

* [PATCH 22/38] perf callchain: Use session__find_addr_location() and friends
  2015-03-03  3:07 [RFC/PATCHSET 00/38] perf tools: Speed-up perf report by using multi thread (v3) Namhyung Kim
                   ` (20 preceding siblings ...)
  2015-03-03  3:07 ` [PATCH 21/38] perf tools: Introduce session__find_addr_location() and friends Namhyung Kim
@ 2015-03-03  3:07 ` Namhyung Kim
  2015-03-03 14:01   ` Arnaldo Carvalho de Melo
  2015-03-03  3:07 ` [PATCH 23/38] perf tools: Add a test case for timed map groups handling Namhyung Kim
                   ` (15 subsequent siblings)
  37 siblings, 1 reply; 221+ messages in thread
From: Namhyung Kim @ 2015-03-03  3:07 UTC (permalink / raw)
  To: Arnaldo Carvalho de Melo
  Cc: Ingo Molnar, Peter Zijlstra, Jiri Olsa, LKML,
	Frederic Weisbecker, Adrian Hunter, Stephane Eranian, Andi Kleen,
	David Ahern

Pass session struct to callchain resolve routines and find correct
thread/map/symbol using proper functions.

Cc: Frederic Weisbecker <fweisbec@gmail.com>
Signed-off-by: Namhyung Kim <namhyung@kernel.org>
---
 tools/perf/builtin-script.c                        |  4 +--
 tools/perf/tests/dwarf-unwind.c                    |  2 +-
 tools/perf/util/callchain.c                        |  6 ++--
 tools/perf/util/callchain.h                        |  4 ++-
 tools/perf/util/hist.c                             |  5 +--
 tools/perf/util/machine.c                          | 41 +++++++++++++---------
 tools/perf/util/machine.h                          |  1 +
 .../util/scripting-engines/trace-event-python.c    | 27 +++++++-------
 tools/perf/util/session.c                          |  6 ++--
 tools/perf/util/session.h                          |  2 +-
 tools/perf/util/unwind-libdw.c                     | 14 ++++----
 tools/perf/util/unwind-libdw.h                     |  1 +
 tools/perf/util/unwind-libunwind.c                 | 32 +++++++++--------
 tools/perf/util/unwind.h                           |  3 +-
 14 files changed, 86 insertions(+), 62 deletions(-)

diff --git a/tools/perf/builtin-script.c b/tools/perf/builtin-script.c
index 65b3a07be2bf..90a401a52868 100644
--- a/tools/perf/builtin-script.c
+++ b/tools/perf/builtin-script.c
@@ -429,7 +429,7 @@ static void print_sample_bts(union perf_event *event,
 				print_opts &= ~PRINT_IP_OPT_SRCLINE;
 			}
 		}
-		perf_evsel__print_ip(evsel, sample, al, print_opts,
+		perf_evsel__print_ip(evsel, sample, session, al, print_opts,
 				     PERF_MAX_STACK_DEPTH);
 	}
 
@@ -483,7 +483,7 @@ static void process_event(union perf_event *event, struct perf_sample *sample,
 		else
 			printf("\n");
 
-		perf_evsel__print_ip(evsel, sample, al,
+		perf_evsel__print_ip(evsel, sample, session, al,
 				     output[attr->type].print_ip_opts,
 				     PERF_MAX_STACK_DEPTH);
 	}
diff --git a/tools/perf/tests/dwarf-unwind.c b/tools/perf/tests/dwarf-unwind.c
index 7e04feb431cb..241270374e93 100644
--- a/tools/perf/tests/dwarf-unwind.c
+++ b/tools/perf/tests/dwarf-unwind.c
@@ -75,7 +75,7 @@ static int unwind_thread(struct thread *thread)
 		goto out;
 	}
 
-	err = unwind__get_entries(unwind_entry, &cnt, thread,
+	err = unwind__get_entries(unwind_entry, &cnt, thread, NULL,
 				  &sample, MAX_STACK);
 	if (err)
 		pr_debug("unwind failed\n");
diff --git a/tools/perf/util/callchain.c b/tools/perf/util/callchain.c
index 9f643ee77001..f95b27037dc8 100644
--- a/tools/perf/util/callchain.c
+++ b/tools/perf/util/callchain.c
@@ -757,7 +757,9 @@ int callchain_cursor_append(struct callchain_cursor *cursor,
 	return 0;
 }
 
-int sample__resolve_callchain(struct perf_sample *sample, struct symbol **parent,
+int sample__resolve_callchain(struct perf_sample *sample,
+			      struct perf_session *session,
+			      struct symbol **parent,
 			      struct perf_evsel *evsel, struct addr_location *al,
 			      int max_stack)
 {
@@ -767,7 +769,7 @@ int sample__resolve_callchain(struct perf_sample *sample, struct symbol **parent
 	if (symbol_conf.use_callchain || symbol_conf.cumulate_callchain ||
 	    sort__has_parent) {
 		return thread__resolve_callchain(al->thread, evsel, sample,
-						 parent, al, max_stack);
+						 session, parent, al, max_stack);
 	}
 	return 0;
 }
diff --git a/tools/perf/util/callchain.h b/tools/perf/util/callchain.h
index 6033a0a212ca..ca9048f84cb5 100644
--- a/tools/perf/util/callchain.h
+++ b/tools/perf/util/callchain.h
@@ -165,7 +165,9 @@ struct hist_entry;
 int record_parse_callchain_opt(const struct option *opt, const char *arg, int unset);
 int record_callchain_opt(const struct option *opt, const char *arg, int unset);
 
-int sample__resolve_callchain(struct perf_sample *sample, struct symbol **parent,
+int sample__resolve_callchain(struct perf_sample *sample,
+			      struct perf_session *session,
+			      struct symbol **parent,
 			      struct perf_evsel *evsel, struct addr_location *al,
 			      int max_stack);
 int hist_entry__append_callchain(struct hist_entry *he, struct perf_sample *sample);
diff --git a/tools/perf/util/hist.c b/tools/perf/util/hist.c
index 0d189ae76922..dbe7f3744bf1 100644
--- a/tools/perf/util/hist.c
+++ b/tools/perf/util/hist.c
@@ -861,8 +861,9 @@ int hist_entry_iter__add(struct hist_entry_iter *iter, struct addr_location *al,
 {
 	int err, err2;
 
-	err = sample__resolve_callchain(iter->sample, &iter->parent,
-					iter->evsel, al, max_stack_depth);
+	err = sample__resolve_callchain(iter->sample, iter->session,
+					&iter->parent, iter->evsel, al,
+					max_stack_depth);
 	if (err)
 		return err;
 
diff --git a/tools/perf/util/machine.c b/tools/perf/util/machine.c
index 4743718d4bf1..63d860dca74b 100644
--- a/tools/perf/util/machine.c
+++ b/tools/perf/util/machine.c
@@ -1540,10 +1540,11 @@ struct mem_info *sample__resolve_mem(struct perf_sample *sample,
 }
 
 static int add_callchain_ip(struct thread *thread,
+			    struct perf_session *session,
 			    struct symbol **parent,
 			    struct addr_location *root_al,
 			    bool branch_history,
-			    u64 ip)
+			    u64 ip, u64 timestamp)
 {
 	struct addr_location al;
 
@@ -1551,8 +1552,9 @@ static int add_callchain_ip(struct thread *thread,
 	al.sym = NULL;
 
 	if (branch_history)
-		thread__find_cpumode_addr_location(thread, MAP__FUNCTION,
-						   ip, &al);
+		session__find_cpumode_addr_location(session, thread,
+						    MAP__FUNCTION, ip, &al,
+						    timestamp);
 	else {
 		u8 cpumode = PERF_RECORD_MISC_USER;
 
@@ -1579,8 +1581,8 @@ static int add_callchain_ip(struct thread *thread,
 			}
 			return 0;
 		}
-		thread__find_addr_location(thread, cpumode, MAP__FUNCTION,
-				   ip, &al);
+		session__find_addr_location(session,thread, cpumode,
+					    MAP__FUNCTION, ip, &al, timestamp);
 	}
 
 	if (al.sym != NULL) {
@@ -1670,6 +1672,7 @@ static int remove_loops(struct branch_entry *l, int nr)
  */
 static int resolve_lbr_callchain_sample(struct thread *thread,
 					struct perf_sample *sample,
+					struct perf_session *session,
 					struct symbol **parent,
 					struct addr_location *root_al,
 					int max_stack)
@@ -1722,7 +1725,8 @@ static int resolve_lbr_callchain_sample(struct thread *thread,
 					ip = lbr_stack->entries[0].to;
 			}
 
-			err = add_callchain_ip(thread, parent, root_al, false, ip);
+			err = add_callchain_ip(thread, session, parent, root_al,
+					       false, ip, sample->time);
 			if (err)
 				return (err < 0) ? err : 0;
 		}
@@ -1735,6 +1739,7 @@ static int resolve_lbr_callchain_sample(struct thread *thread,
 static int thread__resolve_callchain_sample(struct thread *thread,
 					    struct perf_evsel *evsel,
 					    struct perf_sample *sample,
+					    struct perf_session *session,
 					    struct symbol **parent,
 					    struct addr_location *root_al,
 					    int max_stack)
@@ -1742,6 +1747,7 @@ static int thread__resolve_callchain_sample(struct thread *thread,
 	struct branch_stack *branch = sample->branch_stack;
 	struct ip_callchain *chain = sample->callchain;
 	int chain_nr = min(max_stack, (int)chain->nr);
+	u64 timestamp = sample->time;
 	int i, j, err;
 	int skip_idx = -1;
 	int first_call = 0;
@@ -1749,8 +1755,8 @@ static int thread__resolve_callchain_sample(struct thread *thread,
 	callchain_cursor_reset(&callchain_cursor);
 
 	if (has_branch_callstack(evsel)) {
-		err = resolve_lbr_callchain_sample(thread, sample, parent,
-						   root_al, max_stack);
+		err = resolve_lbr_callchain_sample(thread, sample, session,
+						   parent, root_al, max_stack);
 		if (err)
 			return (err < 0) ? err : 0;
 	}
@@ -1806,11 +1812,12 @@ static int thread__resolve_callchain_sample(struct thread *thread,
 		nr = remove_loops(be, nr);
 
 		for (i = 0; i < nr; i++) {
-			err = add_callchain_ip(thread, parent, root_al,
-					       true, be[i].to);
+			err = add_callchain_ip(thread, session, parent, root_al,
+					       true, be[i].to, timestamp);
 			if (!err)
-				err = add_callchain_ip(thread, parent, root_al,
-						       true, be[i].from);
+				err = add_callchain_ip(thread, session, parent,
+						       root_al, true,
+						       be[i].from, timestamp);
 			if (err == -EINVAL)
 				break;
 			if (err)
@@ -1839,7 +1846,8 @@ static int thread__resolve_callchain_sample(struct thread *thread,
 #endif
 		ip = chain->ips[j];
 
-		err = add_callchain_ip(thread, parent, root_al, false, ip);
+		err = add_callchain_ip(thread, session, parent, root_al, false,
+				       ip, timestamp);
 		if (err)
 			return (err < 0) ? err : 0;
 	}
@@ -1857,12 +1865,13 @@ static int unwind_entry(struct unwind_entry *entry, void *arg)
 int thread__resolve_callchain(struct thread *thread,
 			      struct perf_evsel *evsel,
 			      struct perf_sample *sample,
+			      struct perf_session *session,
 			      struct symbol **parent,
 			      struct addr_location *root_al,
 			      int max_stack)
 {
-	int ret = thread__resolve_callchain_sample(thread, evsel,
-						   sample, parent,
+	int ret = thread__resolve_callchain_sample(thread, evsel, sample,
+						   session, parent,
 						   root_al, max_stack);
 	if (ret)
 		return ret;
@@ -1878,7 +1887,7 @@ int thread__resolve_callchain(struct thread *thread,
 		return 0;
 
 	return unwind__get_entries(unwind_entry, &callchain_cursor,
-				   thread, sample, max_stack);
+				   thread, session, sample, max_stack);
 
 }
 
diff --git a/tools/perf/util/machine.h b/tools/perf/util/machine.h
index 45aee0c329ef..38ead24f0f47 100644
--- a/tools/perf/util/machine.h
+++ b/tools/perf/util/machine.h
@@ -129,6 +129,7 @@ struct mem_info *sample__resolve_mem(struct perf_sample *sample,
 int thread__resolve_callchain(struct thread *thread,
 			      struct perf_evsel *evsel,
 			      struct perf_sample *sample,
+			      struct perf_session *session,
 			      struct symbol **parent,
 			      struct addr_location *root_al,
 			      int max_stack);
diff --git a/tools/perf/util/scripting-engines/trace-event-python.c b/tools/perf/util/scripting-engines/trace-event-python.c
index 802def46af7b..e8c2896055c5 100644
--- a/tools/perf/util/scripting-engines/trace-event-python.c
+++ b/tools/perf/util/scripting-engines/trace-event-python.c
@@ -298,9 +298,10 @@ static PyObject *get_field_numeric_entry(struct event_format *event,
 }
 
 
-static PyObject *python_process_callchain(struct perf_sample *sample,
-					 struct perf_evsel *evsel,
-					 struct addr_location *al)
+static PyObject *python_process_callchain(struct perf_session *session,
+					  struct perf_sample *sample,
+					  struct perf_evsel *evsel,
+					  struct addr_location *al)
 {
 	PyObject *pylist;
 
@@ -311,9 +312,8 @@ static PyObject *python_process_callchain(struct perf_sample *sample,
 	if (!symbol_conf.use_callchain || !sample->callchain)
 		goto exit;
 
-	if (thread__resolve_callchain(al->thread, evsel,
-				      sample, NULL, NULL,
-				      PERF_MAX_STACK_DEPTH) != 0) {
+	if (thread__resolve_callchain(al->thread, evsel, sample, session,
+				      NULL, NULL, PERF_MAX_STACK_DEPTH) != 0) {
 		pr_err("Failed to resolve callchain. Skipping\n");
 		goto exit;
 	}
@@ -374,7 +374,8 @@ static PyObject *python_process_callchain(struct perf_sample *sample,
 }
 
 
-static void python_process_tracepoint(struct perf_sample *sample,
+static void python_process_tracepoint(struct perf_session *session,
+				      struct perf_sample *sample,
 				      struct perf_evsel *evsel,
 				      struct thread *thread,
 				      struct addr_location *al)
@@ -424,7 +425,7 @@ static void python_process_tracepoint(struct perf_sample *sample,
 	PyTuple_SetItem(t, n++, context);
 
 	/* ip unwinding */
-	callchain = python_process_callchain(sample, evsel, al);
+	callchain = python_process_callchain(session, sample, evsel, al);
 
 	if (handler) {
 		PyTuple_SetItem(t, n++, PyInt_FromLong(cpu));
@@ -759,7 +760,8 @@ static int python_process_call_return(struct call_return *cr, void *data)
 	return db_export__call_return(dbe, cr);
 }
 
-static void python_process_general_event(struct perf_sample *sample,
+static void python_process_general_event(struct perf_session *session,
+					 struct perf_sample *sample,
 					 struct perf_evsel *evsel,
 					 struct thread *thread,
 					 struct addr_location *al)
@@ -822,7 +824,7 @@ static void python_process_general_event(struct perf_sample *sample,
 	}
 
 	/* ip unwinding */
-	callchain = python_process_callchain(sample, evsel, al);
+	callchain = python_process_callchain(session, sample, evsel, al);
 	pydict_set_item_string_decref(dict, "callchain", callchain);
 
 	PyTuple_SetItem(t, n++, dict);
@@ -846,7 +848,7 @@ static void python_process_event(union perf_event *event,
 
 	switch (evsel->attr.type) {
 	case PERF_TYPE_TRACEPOINT:
-		python_process_tracepoint(sample, evsel, thread, al);
+		python_process_tracepoint(session, sample, evsel, thread, al);
 		break;
 	/* Reserve for future process_hw/sw/raw APIs */
 	default:
@@ -854,7 +856,8 @@ static void python_process_event(union perf_event *event,
 			db_export__sample(&tables->dbe, event, sample, evsel,
 					  thread, al, session);
 		else
-			python_process_general_event(sample, evsel, thread, al);
+			python_process_general_event(session, sample, evsel,
+						     thread, al);
 	}
 }
 
diff --git a/tools/perf/util/session.c b/tools/perf/util/session.c
index 46761a39fbae..d89dfa8592a9 100644
--- a/tools/perf/util/session.c
+++ b/tools/perf/util/session.c
@@ -1550,7 +1550,7 @@ struct perf_evsel *perf_session__find_first_evtype(struct perf_session *session,
 }
 
 void perf_evsel__print_ip(struct perf_evsel *evsel, struct perf_sample *sample,
-			  struct addr_location *al,
+			  struct perf_session *session, struct addr_location *al,
 			  unsigned int print_opts, unsigned int stack_depth)
 {
 	struct callchain_cursor_node *node;
@@ -1565,8 +1565,8 @@ void perf_evsel__print_ip(struct perf_evsel *evsel, struct perf_sample *sample,
 	if (symbol_conf.use_callchain && sample->callchain) {
 		struct addr_location node_al;
 
-		if (thread__resolve_callchain(al->thread, evsel,
-					      sample, NULL, NULL,
+		if (thread__resolve_callchain(al->thread, evsel, sample,
+					      session, NULL, NULL,
 					      PERF_MAX_STACK_DEPTH) != 0) {
 			if (verbose)
 				error("Failed to resolve callchain. Skipping\n");
diff --git a/tools/perf/util/session.h b/tools/perf/util/session.h
index dbd21f8e7cf1..4d264fef8968 100644
--- a/tools/perf/util/session.h
+++ b/tools/perf/util/session.h
@@ -102,7 +102,7 @@ struct perf_evsel *perf_session__find_first_evtype(struct perf_session *session,
 					    unsigned int type);
 
 void perf_evsel__print_ip(struct perf_evsel *evsel, struct perf_sample *sample,
-			  struct addr_location *al,
+			  struct perf_session *session, struct addr_location *al,
 			  unsigned int print_opts, unsigned int stack_depth);
 
 int perf_session__cpu_bitmap(struct perf_session *session,
diff --git a/tools/perf/util/unwind-libdw.c b/tools/perf/util/unwind-libdw.c
index 2dcfe9a7c8d0..ebaf51b58c92 100644
--- a/tools/perf/util/unwind-libdw.c
+++ b/tools/perf/util/unwind-libdw.c
@@ -8,6 +8,7 @@
 #include "unwind-libdw.h"
 #include "machine.h"
 #include "thread.h"
+#include "session.h"
 #include <linux/types.h>
 #include "event.h"
 #include "perf_regs.h"
@@ -26,10 +27,9 @@ static int __report_module(struct addr_location *al, u64 ip,
 	Dwfl_Module *mod;
 	struct dso *dso = NULL;
 
-	thread__find_addr_location(ui->thread,
-				   PERF_RECORD_MISC_USER,
-				   MAP__FUNCTION, ip, al);
-
+	session__find_addr_location(ui->session, ui->thread,
+				    PERF_RECORD_MISC_USER, MAP__FUNCTION,
+				    ip, al, ui->sample->time);
 	if (al->map)
 		dso = al->map->dso;
 
@@ -89,8 +89,8 @@ static int access_dso_mem(struct unwind_info *ui, Dwarf_Addr addr,
 	struct addr_location al;
 	ssize_t size;
 
-	thread__find_addr_map(ui->thread, PERF_RECORD_MISC_USER,
-			      MAP__FUNCTION, addr, &al);
+	session__find_addr_map(ui->session, ui->thread, PERF_RECORD_MISC_USER,
+			       MAP__FUNCTION, addr, &al, ui->sample->time);
 	if (!al.map) {
 		pr_debug("unwind: no map for %lx\n", (unsigned long)addr);
 		return -1;
@@ -165,12 +165,14 @@ frame_callback(Dwfl_Frame *state, void *arg)
 
 int unwind__get_entries(unwind_entry_cb_t cb, void *arg,
 			struct thread *thread,
+			struct perf_session *session,
 			struct perf_sample *data,
 			int max_stack)
 {
 	struct unwind_info ui = {
 		.sample		= data,
 		.thread		= thread,
+		.session 	= session,
 		.machine	= thread->mg->machine,
 		.cb		= cb,
 		.arg		= arg,
diff --git a/tools/perf/util/unwind-libdw.h b/tools/perf/util/unwind-libdw.h
index 417a1426f3ad..806e522713a2 100644
--- a/tools/perf/util/unwind-libdw.h
+++ b/tools/perf/util/unwind-libdw.h
@@ -11,6 +11,7 @@ bool libdw__arch_set_initial_registers(Dwfl_Thread *thread, void *arg);
 struct unwind_info {
 	Dwfl			*dwfl;
 	struct perf_sample      *sample;
+	struct perf_session     *session;
 	struct machine          *machine;
 	struct thread           *thread;
 	unwind_entry_cb_t	cb;
diff --git a/tools/perf/util/unwind-libunwind.c b/tools/perf/util/unwind-libunwind.c
index e3c40a520a25..9ee63179383e 100644
--- a/tools/perf/util/unwind-libunwind.c
+++ b/tools/perf/util/unwind-libunwind.c
@@ -86,6 +86,7 @@ UNW_OBJ(dwarf_find_debug_frame) (int found, unw_dyn_info_t *di_debug,
 
 struct unwind_info {
 	struct perf_sample	*sample;
+	struct perf_session	*session;
 	struct machine		*machine;
 	struct thread		*thread;
 };
@@ -315,8 +316,8 @@ static struct map *find_map(unw_word_t ip, struct unwind_info *ui)
 {
 	struct addr_location al;
 
-	thread__find_addr_map(ui->thread, PERF_RECORD_MISC_USER,
-			      MAP__FUNCTION, ip, &al);
+	session__find_addr_map(ui->session, ui->thread, PERF_RECORD_MISC_USER,
+			       MAP__FUNCTION, ip, &al, ui->sample->time);
 	return al.map;
 }
 
@@ -406,20 +407,19 @@ get_proc_name(unw_addr_space_t __maybe_unused as,
 static int access_dso_mem(struct unwind_info *ui, unw_word_t addr,
 			  unw_word_t *data)
 {
-	struct addr_location al;
+	struct map *map;
 	ssize_t size;
 
-	thread__find_addr_map(ui->thread, PERF_RECORD_MISC_USER,
-			      MAP__FUNCTION, addr, &al);
-	if (!al.map) {
+	map = find_map(addr, ui);
+	if (!map) {
 		pr_debug("unwind: no map for %lx\n", (unsigned long)addr);
 		return -1;
 	}
 
-	if (!al.map->dso)
+	if (!map->dso)
 		return -1;
 
-	size = dso__data_read_addr(al.map->dso, al.map, ui->machine,
+	size = dso__data_read_addr(map->dso, map, ui->machine,
 				   addr, (u8 *) data, sizeof(*data));
 
 	return !(size == sizeof(*data));
@@ -511,14 +511,14 @@ static void put_unwind_info(unw_addr_space_t __maybe_unused as,
 	pr_debug("unwind: put_unwind_info called\n");
 }
 
-static int entry(u64 ip, struct thread *thread,
-		 unwind_entry_cb_t cb, void *arg)
+static int entry(u64 ip, struct thread *thread, struct perf_session *session,
+		 u64 timestamp, unwind_entry_cb_t cb, void *arg)
 {
 	struct unwind_entry e;
 	struct addr_location al;
 
-	thread__find_addr_location(thread, PERF_RECORD_MISC_USER,
-				   MAP__FUNCTION, ip, &al);
+	session__find_addr_location(session, thread, PERF_RECORD_MISC_USER,
+				    MAP__FUNCTION, ip, &al, timestamp);
 
 	e.ip = ip;
 	e.map = al.map;
@@ -620,20 +620,22 @@ static int get_entries(struct unwind_info *ui, unwind_entry_cb_t cb,
 		unw_word_t ip;
 
 		unw_get_reg(&c, UNW_REG_IP, &ip);
-		ret = ip ? entry(ip, ui->thread, cb, arg) : 0;
+		ret = ip ? entry(ip, ui->thread, ui->session,
+				 ui->sample->time, cb, arg) : 0;
 	}
 
 	return ret;
 }
 
 int unwind__get_entries(unwind_entry_cb_t cb, void *arg,
-			struct thread *thread,
+			struct thread *thread, struct perf_session *session,
 			struct perf_sample *data, int max_stack)
 {
 	u64 ip;
 	struct unwind_info ui = {
 		.sample       = data,
 		.thread       = thread,
+		.session      = session,
 		.machine      = thread->mg->machine,
 	};
 	int ret;
@@ -645,7 +647,7 @@ int unwind__get_entries(unwind_entry_cb_t cb, void *arg,
 	if (ret)
 		return ret;
 
-	ret = entry(ip, thread, cb, arg);
+	ret = entry(ip, thread, session, data->time, cb, arg);
 	if (ret)
 		return -ENOMEM;
 
diff --git a/tools/perf/util/unwind.h b/tools/perf/util/unwind.h
index 12790cf94618..c619890e60ad 100644
--- a/tools/perf/util/unwind.h
+++ b/tools/perf/util/unwind.h
@@ -16,7 +16,7 @@ typedef int (*unwind_entry_cb_t)(struct unwind_entry *entry, void *arg);
 
 #ifdef HAVE_DWARF_UNWIND_SUPPORT
 int unwind__get_entries(unwind_entry_cb_t cb, void *arg,
-			struct thread *thread,
+			struct thread *thread, struct perf_session *session,
 			struct perf_sample *data, int max_stack);
 /* libunwind specific */
 #ifdef HAVE_LIBUNWIND_SUPPORT
@@ -38,6 +38,7 @@ static inline int
 unwind__get_entries(unwind_entry_cb_t cb __maybe_unused,
 		    void *arg __maybe_unused,
 		    struct thread *thread __maybe_unused,
+		    struct perf_session *session __maybe_unused,
 		    struct perf_sample *data __maybe_unused,
 		    int max_stack __maybe_unused)
 {
-- 
2.2.2


^ permalink raw reply related	[flat|nested] 221+ messages in thread

* [PATCH 23/38] perf tools: Add a test case for timed map groups handling
  2015-03-03  3:07 [RFC/PATCHSET 00/38] perf tools: Speed-up perf report by using multi thread (v3) Namhyung Kim
                   ` (21 preceding siblings ...)
  2015-03-03  3:07 ` [PATCH 22/38] perf callchain: Use " Namhyung Kim
@ 2015-03-03  3:07 ` Namhyung Kim
  2015-03-03  3:07 ` [PATCH 24/38] perf tools: Protect dso symbol loading using a mutex Namhyung Kim
                   ` (14 subsequent siblings)
  37 siblings, 0 replies; 221+ messages in thread
From: Namhyung Kim @ 2015-03-03  3:07 UTC (permalink / raw)
  To: Arnaldo Carvalho de Melo
  Cc: Ingo Molnar, Peter Zijlstra, Jiri Olsa, LKML,
	Frederic Weisbecker, Adrian Hunter, Stephane Eranian, Andi Kleen,
	David Ahern

A test case for verifying thread->mg and ->mg_list handling during
time change and new thread__find_addr_map_time() and friends.

Cc: Frederic Weisbecker <fweisbec@gmail.com>
Signed-off-by: Namhyung Kim <namhyung@kernel.org>
---
 tools/perf/tests/Build            |  1 +
 tools/perf/tests/builtin-test.c   |  4 ++
 tools/perf/tests/tests.h          |  1 +
 tools/perf/tests/thread-mg-time.c | 88 +++++++++++++++++++++++++++++++++++++++
 4 files changed, 94 insertions(+)
 create mode 100644 tools/perf/tests/thread-mg-time.c

diff --git a/tools/perf/tests/Build b/tools/perf/tests/Build
index bfa0aa35761f..b6f50e3e301f 100644
--- a/tools/perf/tests/Build
+++ b/tools/perf/tests/Build
@@ -27,6 +27,7 @@ perf-y += mmap-thread-lookup.o
 perf-y += thread-comm.o
 perf-y += thread-mg-share.o
 perf-y += thread-lookup-time.o
+perf-y += thread-mg-time.o
 perf-y += switch-tracking.o
 perf-y += keep-tracking.o
 perf-y += code-reading.o
diff --git a/tools/perf/tests/builtin-test.c b/tools/perf/tests/builtin-test.c
index e4d335de19ea..8f61a7e291ee 100644
--- a/tools/perf/tests/builtin-test.c
+++ b/tools/perf/tests/builtin-test.c
@@ -175,6 +175,10 @@ static struct test {
 		.func = test__thread_lookup_time,
 	},
 	{
+		.desc = "Test thread map group handling with time",
+		.func = test__thread_mg_time,
+	},
+	{
 		.func = NULL,
 	},
 };
diff --git a/tools/perf/tests/tests.h b/tools/perf/tests/tests.h
index 1090337f63e5..03557563f31d 100644
--- a/tools/perf/tests/tests.h
+++ b/tools/perf/tests/tests.h
@@ -53,6 +53,7 @@ int test__fdarray__filter(void);
 int test__fdarray__add(void);
 int test__thread_comm(void);
 int test__thread_lookup_time(void);
+int test__thread_mg_time(void);
 
 #if defined(__x86_64__) || defined(__i386__) || defined(__arm__)
 #ifdef HAVE_DWARF_UNWIND_SUPPORT
diff --git a/tools/perf/tests/thread-mg-time.c b/tools/perf/tests/thread-mg-time.c
new file mode 100644
index 000000000000..69fd13752c1d
--- /dev/null
+++ b/tools/perf/tests/thread-mg-time.c
@@ -0,0 +1,88 @@
+#include "tests.h"
+#include "machine.h"
+#include "thread.h"
+#include "map.h"
+#include "debug.h"
+
+#define PERF_MAP_START  0x40000
+
+int test__thread_mg_time(void)
+{
+	struct machines machines;
+	struct machine *machine;
+	struct thread *t;
+	struct map_groups *mg;
+	struct map *map;
+	struct addr_location al = { .map = NULL, };
+
+	/*
+	 * This test is to check whether it can retrieve a correct map
+	 * for a given time.  When multi-file data storage is enabled,
+	 * those task/comm/mmap events are processed first so the
+	 * later sample should find a matching comm properly.
+	 */
+	machines__init(&machines);
+	machine = &machines.host;
+
+	t = machine__findnew_thread(machine, 0, 0);
+	mg = t->mg;
+
+	map = dso__new_map("/usr/bin/perf");
+	map->start = PERF_MAP_START;
+	map->end = PERF_MAP_START + 0x1000;
+
+	thread__insert_map(t, map);
+
+	if (verbose > 1)
+		map_groups__fprintf(t->mg, stderr);
+
+	thread__find_addr_map(t, PERF_RECORD_MISC_USER, MAP__FUNCTION,
+			      PERF_MAP_START, &al);
+
+	TEST_ASSERT_VAL("cannot find mapping for perf", al.map != NULL);
+	TEST_ASSERT_VAL("non matched mapping found", al.map == map);
+	TEST_ASSERT_VAL("incorrect map groups", al.map->groups == mg);
+	TEST_ASSERT_VAL("incorrect map groups", al.map->groups == t->mg);
+
+	thread__find_addr_map_time(t, PERF_RECORD_MISC_USER,
+				   MAP__FUNCTION, PERF_MAP_START, &al, -1ULL);
+
+	TEST_ASSERT_VAL("cannot find timed mapping for perf", al.map != NULL);
+	TEST_ASSERT_VAL("non matched timed mapping", al.map == map);
+	TEST_ASSERT_VAL("incorrect timed map groups", al.map->groups == mg);
+	TEST_ASSERT_VAL("incorrect map groups", al.map->groups == t->mg);
+
+
+	pr_debug("simulate EXEC event (generate new mg)\n");
+	__thread__set_comm(t, "perf-test", 10000, true);
+
+	map = dso__new_map("/usr/bin/perf-test");
+	map->start = PERF_MAP_START;
+	map->end = PERF_MAP_START + 0x2000;
+
+	thread__insert_map(t, map);
+
+	if (verbose > 1)
+		map_groups__fprintf(t->mg, stderr);
+
+	thread__find_addr_map(t, PERF_RECORD_MISC_USER, MAP__FUNCTION,
+			      PERF_MAP_START + 4, &al);
+
+	TEST_ASSERT_VAL("cannot find mapping for perf-test", al.map != NULL);
+	TEST_ASSERT_VAL("invalid mapping found", al.map == map);
+	TEST_ASSERT_VAL("incorrect map groups", al.map->groups != mg);
+	TEST_ASSERT_VAL("incorrect map groups", al.map->groups == t->mg);
+
+	pr_debug("searching map in the old mag groups\n");
+	thread__find_addr_map_time(t, PERF_RECORD_MISC_USER,
+				   MAP__FUNCTION, PERF_MAP_START, &al, 5000);
+
+	TEST_ASSERT_VAL("cannot find timed mapping for perf-test", al.map != NULL);
+	TEST_ASSERT_VAL("non matched timed mapping", al.map != map);
+	TEST_ASSERT_VAL("incorrect timed map groups", al.map->groups == mg);
+	TEST_ASSERT_VAL("incorrect map groups", al.map->groups != t->mg);
+
+	machine__delete_threads(machine);
+	machines__exit(&machines);
+	return 0;
+}
-- 
2.2.2


^ permalink raw reply related	[flat|nested] 221+ messages in thread

* [PATCH 24/38] perf tools: Protect dso symbol loading using a mutex
  2015-03-03  3:07 [RFC/PATCHSET 00/38] perf tools: Speed-up perf report by using multi thread (v3) Namhyung Kim
                   ` (22 preceding siblings ...)
  2015-03-03  3:07 ` [PATCH 23/38] perf tools: Add a test case for timed map groups handling Namhyung Kim
@ 2015-03-03  3:07 ` Namhyung Kim
  2015-03-03  3:07 ` [PATCH 25/38] perf tools: Protect dso cache tree using dso->lock Namhyung Kim
                   ` (13 subsequent siblings)
  37 siblings, 0 replies; 221+ messages in thread
From: Namhyung Kim @ 2015-03-03  3:07 UTC (permalink / raw)
  To: Arnaldo Carvalho de Melo
  Cc: Ingo Molnar, Peter Zijlstra, Jiri Olsa, LKML,
	Frederic Weisbecker, Adrian Hunter, Stephane Eranian, Andi Kleen,
	David Ahern

When multi-thread support for perf report is enabled, it's possible to
access a dso concurrently.  Add a new pthread_mutex to protect it from
concurrent dso__load().

Signed-off-by: Namhyung Kim <namhyung@kernel.org>
---
 tools/perf/util/dso.c    |  2 ++
 tools/perf/util/dso.h    |  1 +
 tools/perf/util/symbol.c | 34 ++++++++++++++++++++++++----------
 3 files changed, 27 insertions(+), 10 deletions(-)

diff --git a/tools/perf/util/dso.c b/tools/perf/util/dso.c
index 814554d1b857..f10269c3fe2f 100644
--- a/tools/perf/util/dso.c
+++ b/tools/perf/util/dso.c
@@ -884,6 +884,7 @@ struct dso *dso__new(const char *name)
 		RB_CLEAR_NODE(&dso->rb_node);
 		INIT_LIST_HEAD(&dso->node);
 		INIT_LIST_HEAD(&dso->data.open_entry);
+		pthread_mutex_init(&dso->lock, NULL);
 	}
 
 	return dso;
@@ -913,6 +914,7 @@ void dso__delete(struct dso *dso)
 	dso_cache__free(&dso->data.cache);
 	dso__free_a2l(dso);
 	zfree(&dso->symsrc_filename);
+	pthread_mutex_destroy(&dso->lock);
 	free(dso);
 }
 
diff --git a/tools/perf/util/dso.h b/tools/perf/util/dso.h
index ced92841ff97..da188a73d034 100644
--- a/tools/perf/util/dso.h
+++ b/tools/perf/util/dso.h
@@ -102,6 +102,7 @@ struct dsos {
 };
 
 struct dso {
+	pthread_mutex_t	 lock;
 	struct list_head node;
 	struct rb_node	 rb_node;	/* rbtree node sorted by long name */
 	struct rb_root	 symbols[MAP__NR_TYPES];
diff --git a/tools/perf/util/symbol.c b/tools/perf/util/symbol.c
index a69066865a55..714e20c99354 100644
--- a/tools/perf/util/symbol.c
+++ b/tools/perf/util/symbol.c
@@ -1357,12 +1357,22 @@ int dso__load(struct dso *dso, struct map *map, symbol_filter_t filter)
 	struct symsrc *syms_ss = NULL, *runtime_ss = NULL;
 	bool kmod;
 
-	dso__set_loaded(dso, map->type);
+	pthread_mutex_lock(&dso->lock);
+
+	/* check again under the dso->lock */
+	if (dso__loaded(dso, map->type)) {
+		ret = 1;
+		goto out;
+	}
+
+	if (dso->kernel) {
+		if (dso->kernel == DSO_TYPE_KERNEL)
+			ret = dso__load_kernel_sym(dso, map, filter);
+		else if (dso->kernel == DSO_TYPE_GUEST_KERNEL)
+			ret = dso__load_guest_kernel_sym(dso, map, filter);
 
-	if (dso->kernel == DSO_TYPE_KERNEL)
-		return dso__load_kernel_sym(dso, map, filter);
-	else if (dso->kernel == DSO_TYPE_GUEST_KERNEL)
-		return dso__load_guest_kernel_sym(dso, map, filter);
+		goto out;
+	}
 
 	if (map->groups && map->groups->machine)
 		machine = map->groups->machine;
@@ -1375,18 +1385,18 @@ int dso__load(struct dso *dso, struct map *map, symbol_filter_t filter)
 		struct stat st;
 
 		if (lstat(dso->name, &st) < 0)
-			return -1;
+			goto out;
 
 		if (st.st_uid && (st.st_uid != geteuid())) {
 			pr_warning("File %s not owned by current user or root, "
 				"ignoring it.\n", dso->name);
-			return -1;
+			goto out;
 		}
 
 		ret = dso__load_perf_map(dso, map, filter);
 		dso->symtab_type = ret > 0 ? DSO_BINARY_TYPE__JAVA_JIT :
 					     DSO_BINARY_TYPE__NOT_FOUND;
-		return ret;
+		goto out;
 	}
 
 	if (machine)
@@ -1394,7 +1404,7 @@ int dso__load(struct dso *dso, struct map *map, symbol_filter_t filter)
 
 	name = malloc(PATH_MAX);
 	if (!name)
-		return -1;
+		goto out;
 
 	kmod = dso->symtab_type == DSO_BINARY_TYPE__SYSTEM_PATH_KMODULE ||
 		dso->symtab_type == DSO_BINARY_TYPE__SYSTEM_PATH_KMODULE_COMP ||
@@ -1475,7 +1485,11 @@ int dso__load(struct dso *dso, struct map *map, symbol_filter_t filter)
 out_free:
 	free(name);
 	if (ret < 0 && strstr(dso->name, " (deleted)") != NULL)
-		return 0;
+		ret = 0;
+out:
+	dso__set_loaded(dso, map->type);
+	pthread_mutex_unlock(&dso->lock);
+
 	return ret;
 }
 
-- 
2.2.2


^ permalink raw reply related	[flat|nested] 221+ messages in thread

* [PATCH 25/38] perf tools: Protect dso cache tree using dso->lock
  2015-03-03  3:07 [RFC/PATCHSET 00/38] perf tools: Speed-up perf report by using multi thread (v3) Namhyung Kim
                   ` (23 preceding siblings ...)
  2015-03-03  3:07 ` [PATCH 24/38] perf tools: Protect dso symbol loading using a mutex Namhyung Kim
@ 2015-03-03  3:07 ` Namhyung Kim
  2015-03-03  3:07 ` [PATCH 26/38] perf tools: Protect dso cache fd with a mutex Namhyung Kim
                   ` (12 subsequent siblings)
  37 siblings, 0 replies; 221+ messages in thread
From: Namhyung Kim @ 2015-03-03  3:07 UTC (permalink / raw)
  To: Arnaldo Carvalho de Melo
  Cc: Ingo Molnar, Peter Zijlstra, Jiri Olsa, LKML,
	Frederic Weisbecker, Adrian Hunter, Stephane Eranian, Andi Kleen,
	David Ahern

The dso cache is accessed during dwarf callchain unwind and it might
be processed concurrently when multi-thread report is enabled.
Protect it under dso->lock.

Note that it doesn't protect dso_cache__find().  I think it's safe to
access to the cache tree without the lock since we don't delete nodes.
It it missed an existing node due to rotation, it'll find it during
dso_cache__insert() anyway.

Signed-off-by: Namhyung Kim <namhyung@kernel.org>
---
 tools/perf/util/dso.c | 34 +++++++++++++++++++++++++++-------
 1 file changed, 27 insertions(+), 7 deletions(-)

diff --git a/tools/perf/util/dso.c b/tools/perf/util/dso.c
index f10269c3fe2f..3bfbe0e76e96 100644
--- a/tools/perf/util/dso.c
+++ b/tools/perf/util/dso.c
@@ -443,10 +443,12 @@ bool dso__data_status_seen(struct dso *dso, enum dso_data_status_seen by)
 }
 
 static void
-dso_cache__free(struct rb_root *root)
+dso_cache__free(struct dso *dso)
 {
+	struct rb_root *root = &dso->data.cache;
 	struct rb_node *next = rb_first(root);
 
+	pthread_mutex_lock(&dso->lock);
 	while (next) {
 		struct dso_cache *cache;
 
@@ -455,10 +457,12 @@ dso_cache__free(struct rb_root *root)
 		rb_erase(&cache->rb_node, root);
 		free(cache);
 	}
+	pthread_mutex_unlock(&dso->lock);
 }
 
-static struct dso_cache *dso_cache__find(const struct rb_root *root, u64 offset)
+static struct dso_cache *dso_cache__find(struct dso *dso, u64 offset)
 {
+	const struct rb_root *root = &dso->data.cache;
 	struct rb_node * const *p = &root->rb_node;
 	const struct rb_node *parent = NULL;
 	struct dso_cache *cache;
@@ -477,17 +481,20 @@ static struct dso_cache *dso_cache__find(const struct rb_root *root, u64 offset)
 		else
 			return cache;
 	}
+
 	return NULL;
 }
 
-static void
-dso_cache__insert(struct rb_root *root, struct dso_cache *new)
+static struct dso_cache *
+dso_cache__insert(struct dso *dso, struct dso_cache *new)
 {
+	struct rb_root *root = &dso->data.cache;
 	struct rb_node **p = &root->rb_node;
 	struct rb_node *parent = NULL;
 	struct dso_cache *cache;
 	u64 offset = new->offset;
 
+	pthread_mutex_lock(&dso->lock);
 	while (*p != NULL) {
 		u64 end;
 
@@ -499,10 +506,17 @@ dso_cache__insert(struct rb_root *root, struct dso_cache *new)
 			p = &(*p)->rb_left;
 		else if (offset >= end)
 			p = &(*p)->rb_right;
+		else
+			goto out;
 	}
 
 	rb_link_node(&new->rb_node, parent, p);
 	rb_insert_color(&new->rb_node, root);
+
+	cache = NULL;
+out:
+	pthread_mutex_unlock(&dso->lock);
+	return cache;
 }
 
 static ssize_t
@@ -520,6 +534,7 @@ static ssize_t
 dso_cache__read(struct dso *dso, u64 offset, u8 *data, ssize_t size)
 {
 	struct dso_cache *cache;
+	struct dso_cache *old;
 	ssize_t ret;
 
 	do {
@@ -539,7 +554,12 @@ dso_cache__read(struct dso *dso, u64 offset, u8 *data, ssize_t size)
 
 		cache->offset = cache_offset;
 		cache->size   = ret;
-		dso_cache__insert(&dso->data.cache, cache);
+		old = dso_cache__insert(dso, cache);
+		if (old) {
+			/* we lose the race */
+			free(cache);
+			cache = old;
+		}
 
 		ret = dso_cache__memcpy(cache, offset, data, size);
 
@@ -556,7 +576,7 @@ static ssize_t dso_cache_read(struct dso *dso, u64 offset,
 {
 	struct dso_cache *cache;
 
-	cache = dso_cache__find(&dso->data.cache, offset);
+	cache = dso_cache__find(dso, offset);
 	if (cache)
 		return dso_cache__memcpy(cache, offset, data, size);
 	else
@@ -911,7 +931,7 @@ void dso__delete(struct dso *dso)
 	}
 
 	dso__data_close(dso);
-	dso_cache__free(&dso->data.cache);
+	dso_cache__free(dso);
 	dso__free_a2l(dso);
 	zfree(&dso->symsrc_filename);
 	pthread_mutex_destroy(&dso->lock);
-- 
2.2.2


^ permalink raw reply related	[flat|nested] 221+ messages in thread

* [PATCH 26/38] perf tools: Protect dso cache fd with a mutex
  2015-03-03  3:07 [RFC/PATCHSET 00/38] perf tools: Speed-up perf report by using multi thread (v3) Namhyung Kim
                   ` (24 preceding siblings ...)
  2015-03-03  3:07 ` [PATCH 25/38] perf tools: Protect dso cache tree using dso->lock Namhyung Kim
@ 2015-03-03  3:07 ` Namhyung Kim
  2015-03-03  3:07 ` [PATCH 27/38] perf callchain: Maintain libunwind's address space in map_groups Namhyung Kim
                   ` (11 subsequent siblings)
  37 siblings, 0 replies; 221+ messages in thread
From: Namhyung Kim @ 2015-03-03  3:07 UTC (permalink / raw)
  To: Arnaldo Carvalho de Melo
  Cc: Ingo Molnar, Peter Zijlstra, Jiri Olsa, LKML,
	Frederic Weisbecker, Adrian Hunter, Stephane Eranian, Andi Kleen,
	David Ahern

When dso cache is accessed in multi-thread environment, it's possible
to close other dso->data.fd during operation due to open file limit.
Protect the file descriptors using a separate mutex.

Signed-off-by: Namhyung Kim <namhyung@kernel.org>
---
 tools/perf/util/dso.c | 98 +++++++++++++++++++++++++++++++++++++--------------
 1 file changed, 72 insertions(+), 26 deletions(-)

diff --git a/tools/perf/util/dso.c b/tools/perf/util/dso.c
index 3bfbe0e76e96..64aaa45dcdd7 100644
--- a/tools/perf/util/dso.c
+++ b/tools/perf/util/dso.c
@@ -213,6 +213,7 @@ bool dso__needs_decompress(struct dso *dso)
  */
 static LIST_HEAD(dso__data_open);
 static long dso__data_open_cnt;
+static pthread_mutex_t dso__data_open_lock = PTHREAD_MUTEX_INITIALIZER;
 
 static void dso__list_add(struct dso *dso)
 {
@@ -382,7 +383,9 @@ static void check_data_close(void)
  */
 void dso__data_close(struct dso *dso)
 {
+	pthread_mutex_lock(&dso__data_open_lock);
 	close_dso(dso);
+	pthread_mutex_unlock(&dso__data_open_lock);
 }
 
 /**
@@ -405,6 +408,8 @@ int dso__data_fd(struct dso *dso, struct machine *machine)
 	if (dso->data.status == DSO_DATA_STATUS_ERROR)
 		return -1;
 
+	pthread_mutex_lock(&dso__data_open_lock);
+
 	if (dso->data.fd >= 0)
 		goto out;
 
@@ -427,6 +432,7 @@ int dso__data_fd(struct dso *dso, struct machine *machine)
 	else
 		dso->data.status = DSO_DATA_STATUS_ERROR;
 
+	pthread_mutex_unlock(&dso__data_open_lock);
 	return dso->data.fd;
 }
 
@@ -531,7 +537,8 @@ dso_cache__memcpy(struct dso_cache *cache, u64 offset,
 }
 
 static ssize_t
-dso_cache__read(struct dso *dso, u64 offset, u8 *data, ssize_t size)
+dso_cache__read(struct dso *dso, struct machine *machine,
+		u64 offset, u8 *data, ssize_t size)
 {
 	struct dso_cache *cache;
 	struct dso_cache *old;
@@ -540,11 +547,24 @@ dso_cache__read(struct dso *dso, u64 offset, u8 *data, ssize_t size)
 	do {
 		u64 cache_offset;
 
-		ret = -ENOMEM;
-
 		cache = zalloc(sizeof(*cache) + DSO__DATA_CACHE_SIZE);
 		if (!cache)
-			break;
+			return -ENOMEM;
+
+		pthread_mutex_lock(&dso__data_open_lock);
+
+		/*
+		 * dso->data.fd might be closed if other thread opened another
+		 * file (dso) due to open file limit (RLIMIT_NOFILE).
+		 */
+		if (dso->data.fd < 0) {
+			dso->data.fd = open_dso(dso, machine);
+			if (dso->data.fd < 0) {
+				ret = -errno;
+				dso->data.status = DSO_DATA_STATUS_ERROR;
+				break;
+			}
+		}
 
 		cache_offset = offset & DSO__DATA_CACHE_MASK;
 
@@ -554,6 +574,11 @@ dso_cache__read(struct dso *dso, u64 offset, u8 *data, ssize_t size)
 
 		cache->offset = cache_offset;
 		cache->size   = ret;
+	} while (0);
+
+	pthread_mutex_unlock(&dso__data_open_lock);
+
+	if (ret > 0) {
 		old = dso_cache__insert(dso, cache);
 		if (old) {
 			/* we lose the race */
@@ -562,8 +587,7 @@ dso_cache__read(struct dso *dso, u64 offset, u8 *data, ssize_t size)
 		}
 
 		ret = dso_cache__memcpy(cache, offset, data, size);
-
-	} while (0);
+	}
 
 	if (ret <= 0)
 		free(cache);
@@ -571,8 +595,8 @@ dso_cache__read(struct dso *dso, u64 offset, u8 *data, ssize_t size)
 	return ret;
 }
 
-static ssize_t dso_cache_read(struct dso *dso, u64 offset,
-			      u8 *data, ssize_t size)
+static ssize_t dso_cache_read(struct dso *dso, struct machine *machine,
+			      u64 offset, u8 *data, ssize_t size)
 {
 	struct dso_cache *cache;
 
@@ -580,7 +604,7 @@ static ssize_t dso_cache_read(struct dso *dso, u64 offset,
 	if (cache)
 		return dso_cache__memcpy(cache, offset, data, size);
 	else
-		return dso_cache__read(dso, offset, data, size);
+		return dso_cache__read(dso, machine, offset, data, size);
 }
 
 /*
@@ -588,7 +612,8 @@ static ssize_t dso_cache_read(struct dso *dso, u64 offset,
  * in the rb_tree. Any read to already cached data is served
  * by cached data.
  */
-static ssize_t cached_read(struct dso *dso, u64 offset, u8 *data, ssize_t size)
+static ssize_t cached_read(struct dso *dso, struct machine *machine,
+			   u64 offset, u8 *data, ssize_t size)
 {
 	ssize_t r = 0;
 	u8 *p = data;
@@ -596,7 +621,7 @@ static ssize_t cached_read(struct dso *dso, u64 offset, u8 *data, ssize_t size)
 	do {
 		ssize_t ret;
 
-		ret = dso_cache_read(dso, offset, p, size);
+		ret = dso_cache_read(dso, machine, offset, p, size);
 		if (ret < 0)
 			return ret;
 
@@ -616,21 +641,42 @@ static ssize_t cached_read(struct dso *dso, u64 offset, u8 *data, ssize_t size)
 	return r;
 }
 
-static int data_file_size(struct dso *dso)
+static int data_file_size(struct dso *dso, struct machine *machine)
 {
+	int ret = 0;
 	struct stat st;
 	char sbuf[STRERR_BUFSIZE];
 
-	if (!dso->data.file_size) {
-		if (fstat(dso->data.fd, &st)) {
-			pr_err("dso mmap failed, fstat: %s\n",
-				strerror_r(errno, sbuf, sizeof(sbuf)));
-			return -1;
+	if (dso->data.file_size)
+		return 0;
+
+	pthread_mutex_lock(&dso__data_open_lock);
+
+	/*
+	 * dso->data.fd might be closed if other thread opened another
+	 * file (dso) due to open file limit (RLIMIT_NOFILE).
+	 */
+	if (dso->data.fd < 0) {
+		dso->data.fd = open_dso(dso, machine);
+		if (dso->data.fd < 0) {
+			ret = -errno;
+			dso->data.status = DSO_DATA_STATUS_ERROR;
+			goto out;
 		}
-		dso->data.file_size = st.st_size;
 	}
 
-	return 0;
+	if (fstat(dso->data.fd, &st) < 0) {
+		ret = -errno;
+		pr_err("dso cache fstat failed: %s\n",
+		       strerror_r(errno, sbuf, sizeof(sbuf)));
+		dso->data.status = DSO_DATA_STATUS_ERROR;
+		goto out;
+	}
+	dso->data.file_size = st.st_size;
+
+out:
+	pthread_mutex_unlock(&dso__data_open_lock);
+	return ret;
 }
 
 /**
@@ -648,17 +694,17 @@ off_t dso__data_size(struct dso *dso, struct machine *machine)
 	if (fd < 0)
 		return fd;
 
-	if (data_file_size(dso))
+	if (data_file_size(dso, machine))
 		return -1;
 
 	/* For now just estimate dso data size is close to file size */
 	return dso->data.file_size;
 }
 
-static ssize_t data_read_offset(struct dso *dso, u64 offset,
-				u8 *data, ssize_t size)
+static ssize_t data_read_offset(struct dso *dso, struct machine *machine,
+				u64 offset, u8 *data, ssize_t size)
 {
-	if (data_file_size(dso))
+	if (data_file_size(dso, machine))
 		return -1;
 
 	/* Check the offset sanity. */
@@ -668,7 +714,7 @@ static ssize_t data_read_offset(struct dso *dso, u64 offset,
 	if (offset + size < offset)
 		return -1;
 
-	return cached_read(dso, offset, data, size);
+	return cached_read(dso, machine, offset, data, size);
 }
 
 /**
@@ -685,10 +731,10 @@ static ssize_t data_read_offset(struct dso *dso, u64 offset,
 ssize_t dso__data_read_offset(struct dso *dso, struct machine *machine,
 			      u64 offset, u8 *data, ssize_t size)
 {
-	if (dso__data_fd(dso, machine) < 0)
+	if (dso->data.status == DSO_DATA_STATUS_ERROR)
 		return -1;
 
-	return data_read_offset(dso, offset, data, size);
+	return data_read_offset(dso, machine, offset, data, size);
 }
 
 /**
-- 
2.2.2


^ permalink raw reply related	[flat|nested] 221+ messages in thread

* [PATCH 27/38] perf callchain: Maintain libunwind's address space in map_groups
  2015-03-03  3:07 [RFC/PATCHSET 00/38] perf tools: Speed-up perf report by using multi thread (v3) Namhyung Kim
                   ` (25 preceding siblings ...)
  2015-03-03  3:07 ` [PATCH 26/38] perf tools: Protect dso cache fd with a mutex Namhyung Kim
@ 2015-03-03  3:07 ` Namhyung Kim
  2015-03-03  3:07 ` [PATCH 28/38] perf tools: Add dso__data_get/put_fd() Namhyung Kim
                   ` (10 subsequent siblings)
  37 siblings, 0 replies; 221+ messages in thread
From: Namhyung Kim @ 2015-03-03  3:07 UTC (permalink / raw)
  To: Arnaldo Carvalho de Melo
  Cc: Ingo Molnar, Peter Zijlstra, Jiri Olsa, LKML,
	Frederic Weisbecker, Adrian Hunter, Stephane Eranian, Andi Kleen,
	David Ahern

Currently the address_space was kept in thread struct but it's more
appropriate to keep it in map_groups as it's maintained with time.
Also we don't need to flush after exec since it still can be accessed
when used with an indexed data file.

Signed-off-by: Namhyung Kim <namhyung@kernel.org>
---
 tools/perf/tests/dwarf-unwind.c    |  4 ++--
 tools/perf/util/map.c              |  5 +++++
 tools/perf/util/map.h              |  1 +
 tools/perf/util/thread.c           |  8 --------
 tools/perf/util/unwind-libunwind.c | 28 +++++++++++++---------------
 tools/perf/util/unwind.h           | 15 ++++++---------
 6 files changed, 27 insertions(+), 34 deletions(-)

diff --git a/tools/perf/tests/dwarf-unwind.c b/tools/perf/tests/dwarf-unwind.c
index 241270374e93..0fa26e77e28a 100644
--- a/tools/perf/tests/dwarf-unwind.c
+++ b/tools/perf/tests/dwarf-unwind.c
@@ -143,6 +143,8 @@ int test__dwarf_unwind(void)
 	struct thread *thread;
 	int err = -1;
 
+	callchain_param.record_mode = CALLCHAIN_DWARF;
+
 	machines__init(&machines);
 
 	machine = machines__find(&machines, HOST_KERNEL_ID);
@@ -151,8 +153,6 @@ int test__dwarf_unwind(void)
 		return -1;
 	}
 
-	callchain_param.record_mode = CALLCHAIN_DWARF;
-
 	if (init_live_machine(machine)) {
 		pr_err("Could not init machine\n");
 		goto out;
diff --git a/tools/perf/util/map.c b/tools/perf/util/map.c
index 85fbb1b3e69f..c7eeabafa6c9 100644
--- a/tools/perf/util/map.c
+++ b/tools/perf/util/map.c
@@ -14,6 +14,7 @@
 #include "util.h"
 #include "debug.h"
 #include "machine.h"
+#include "unwind.h"
 #include <linux/string.h>
 
 const char *map_type__name[MAP__NR_TYPES] = {
@@ -424,6 +425,8 @@ void map_groups__init(struct map_groups *mg, struct machine *machine)
 	mg->refcnt = 1;
 	mg->timestamp = 0;
 	INIT_LIST_HEAD(&mg->list);
+
+	unwind__prepare_access(mg);
 }
 
 static void maps__delete(struct rb_root *maps)
@@ -457,6 +460,8 @@ void map_groups__exit(struct map_groups *mg)
 		maps__delete(&mg->maps[i]);
 		maps__delete_removed(&mg->removed_maps[i]);
 	}
+
+	unwind__finish_access(mg);
 }
 
 bool map_groups__empty(struct map_groups *mg)
diff --git a/tools/perf/util/map.h b/tools/perf/util/map.h
index f33d49029ac0..f7db4a010dc8 100644
--- a/tools/perf/util/map.h
+++ b/tools/perf/util/map.h
@@ -64,6 +64,7 @@ struct map_groups {
 	u64		 timestamp;
 	int		 refcnt;
 	struct list_head list;
+	void		 *priv;
 };
 
 struct map_groups *map_groups__new(struct machine *machine);
diff --git a/tools/perf/util/thread.c b/tools/perf/util/thread.c
index 9ae1ce8606af..552e1a56af6a 100644
--- a/tools/perf/util/thread.c
+++ b/tools/perf/util/thread.c
@@ -8,7 +8,6 @@
 #include "util.h"
 #include "debug.h"
 #include "comm.h"
-#include "unwind.h"
 
 struct map_groups *thread__get_map_groups(struct thread *thread, u64 timestamp)
 {
@@ -104,9 +103,6 @@ struct thread *thread__new(pid_t pid, pid_t tid)
 		INIT_LIST_HEAD(&thread->tid_node);
 		INIT_LIST_HEAD(&thread->mg_list);
 
-		if (unwind__prepare_access(thread) < 0)
-			goto err_thread;
-
 		comm_str = malloc(32);
 		if (!comm_str)
 			goto err_thread;
@@ -147,7 +143,6 @@ void thread__delete(struct thread *thread)
 		list_del(&comm->list);
 		comm__free(comm);
 	}
-	unwind__finish_access(thread);
 
 	free(thread);
 }
@@ -216,9 +211,6 @@ int __thread__set_comm(struct thread *thread, const char *str, u64 timestamp,
 				break;
 		}
 		list_add_tail(&new->list, &curr->list);
-
-		if (exec)
-			unwind__flush_access(thread);
 	}
 
 	if (exec) {
diff --git a/tools/perf/util/unwind-libunwind.c b/tools/perf/util/unwind-libunwind.c
index 9ee63179383e..9d7ecb26fde9 100644
--- a/tools/perf/util/unwind-libunwind.c
+++ b/tools/perf/util/unwind-libunwind.c
@@ -32,6 +32,7 @@
 #include "symbol.h"
 #include "util.h"
 #include "debug.h"
+#include "map.h"
 
 extern int
 UNW_OBJ(dwarf_search_unwind_table) (unw_addr_space_t as,
@@ -560,7 +561,7 @@ static unw_accessors_t accessors = {
 	.get_proc_name		= get_proc_name,
 };
 
-int unwind__prepare_access(struct thread *thread)
+int unwind__prepare_access(struct map_groups *mg)
 {
 	unw_addr_space_t addr_space;
 
@@ -574,41 +575,38 @@ int unwind__prepare_access(struct thread *thread)
 	}
 
 	unw_set_caching_policy(addr_space, UNW_CACHE_GLOBAL);
-	thread__set_priv(thread, addr_space);
+	mg->priv = addr_space;
 
 	return 0;
 }
 
-void unwind__flush_access(struct thread *thread)
+void unwind__finish_access(struct map_groups *mg)
 {
-	unw_addr_space_t addr_space;
+	unw_addr_space_t addr_space = mg->priv;
 
 	if (callchain_param.record_mode != CALLCHAIN_DWARF)
 		return;
 
-	addr_space = thread__priv(thread);
-	unw_flush_cache(addr_space, 0, 0);
-}
-
-void unwind__finish_access(struct thread *thread)
-{
-	unw_addr_space_t addr_space;
-
-	if (callchain_param.record_mode != CALLCHAIN_DWARF)
+	if (addr_space == NULL)
 		return;
 
-	addr_space = thread__priv(thread);
 	unw_destroy_addr_space(addr_space);
+	mg->priv = NULL;
 }
 
 static int get_entries(struct unwind_info *ui, unwind_entry_cb_t cb,
 		       void *arg, int max_stack)
 {
+	struct map_groups *mg;
 	unw_addr_space_t addr_space;
 	unw_cursor_t c;
 	int ret;
 
-	addr_space = thread__priv(ui->thread);
+	mg = thread__get_map_groups(ui->thread, ui->sample->time);
+	if (mg == NULL)
+		return -1;
+
+	addr_space = mg->priv;
 	if (addr_space == NULL)
 		return -1;
 
diff --git a/tools/perf/util/unwind.h b/tools/perf/util/unwind.h
index c619890e60ad..b3833eaf4c3b 100644
--- a/tools/perf/util/unwind.h
+++ b/tools/perf/util/unwind.h
@@ -21,17 +21,15 @@ int unwind__get_entries(unwind_entry_cb_t cb, void *arg,
 /* libunwind specific */
 #ifdef HAVE_LIBUNWIND_SUPPORT
 int libunwind__arch_reg_id(int regnum);
-int unwind__prepare_access(struct thread *thread);
-void unwind__flush_access(struct thread *thread);
-void unwind__finish_access(struct thread *thread);
+int unwind__prepare_access(struct map_groups *mg);
+void unwind__finish_access(struct map_groups *mg);
 #else
-static inline int unwind__prepare_access(struct thread *thread __maybe_unused)
+static inline int unwind__prepare_access(struct map_groups *mg __maybe_unused)
 {
 	return 0;
 }
 
-static inline void unwind__flush_access(struct thread *thread __maybe_unused) {}
-static inline void unwind__finish_access(struct thread *thread __maybe_unused) {}
+static inline void unwind__finish_access(struct map_groups *mg __maybe_unused) {}
 #endif
 #else
 static inline int
@@ -45,12 +43,11 @@ unwind__get_entries(unwind_entry_cb_t cb __maybe_unused,
 	return 0;
 }
 
-static inline int unwind__prepare_access(struct thread *thread __maybe_unused)
+static inline int unwind__prepare_access(struct map_groups *mg __maybe_unused)
 {
 	return 0;
 }
 
-static inline void unwind__flush_access(struct thread *thread __maybe_unused) {}
-static inline void unwind__finish_access(struct thread *thread __maybe_unused) {}
+static inline void unwind__finish_access(struct map_groups *mg __maybe_unused) {}
 #endif /* HAVE_DWARF_UNWIND_SUPPORT */
 #endif /* __UNWIND_H */
-- 
2.2.2


^ permalink raw reply related	[flat|nested] 221+ messages in thread

* [PATCH 28/38] perf tools: Add dso__data_get/put_fd()
  2015-03-03  3:07 [RFC/PATCHSET 00/38] perf tools: Speed-up perf report by using multi thread (v3) Namhyung Kim
                   ` (26 preceding siblings ...)
  2015-03-03  3:07 ` [PATCH 27/38] perf callchain: Maintain libunwind's address space in map_groups Namhyung Kim
@ 2015-03-03  3:07 ` Namhyung Kim
  2015-03-03  3:07 ` [PATCH 29/38] perf session: Pass struct events stats to event processing functions Namhyung Kim
                   ` (9 subsequent siblings)
  37 siblings, 0 replies; 221+ messages in thread
From: Namhyung Kim @ 2015-03-03  3:07 UTC (permalink / raw)
  To: Arnaldo Carvalho de Melo
  Cc: Ingo Molnar, Peter Zijlstra, Jiri Olsa, LKML,
	Frederic Weisbecker, Adrian Hunter, Stephane Eranian, Andi Kleen,
	David Ahern

Using dso__data_fd() in multi-thread environment is not safe since
returned fd can be closed and/or reused anytime.  So convert it to the
dso__data_get/put_fd() pair to protect the access with lock.

The original dso__data_fd() is deprecated and kept only for testing.

Signed-off-by: Namhyung Kim <namhyung@kernel.org>
---
 tools/perf/util/dso.c              | 44 +++++++++++++++++++++++++++++---------
 tools/perf/util/dso.h              |  9 ++++++--
 tools/perf/util/unwind-libunwind.c | 38 +++++++++++++++++++-------------
 3 files changed, 64 insertions(+), 27 deletions(-)

diff --git a/tools/perf/util/dso.c b/tools/perf/util/dso.c
index 64aaa45dcdd7..50bb2f93b7e9 100644
--- a/tools/perf/util/dso.c
+++ b/tools/perf/util/dso.c
@@ -389,14 +389,15 @@ void dso__data_close(struct dso *dso)
 }
 
 /**
- * dso__data_fd - Get dso's data file descriptor
+ * dso__data_get_fd - Get dso's data file descriptor
  * @dso: dso object
  * @machine: machine object
  *
  * External interface to find dso's file, open it and
- * returns file descriptor.
+ * returns file descriptor.  Should be paired with
+ * dso__data_put_fd().
  */
-int dso__data_fd(struct dso *dso, struct machine *machine)
+int dso__data_get_fd(struct dso *dso, struct machine *machine)
 {
 	enum dso_binary_type binary_type_data[] = {
 		DSO_BINARY_TYPE__BUILD_ID_CACHE,
@@ -405,11 +406,11 @@ int dso__data_fd(struct dso *dso, struct machine *machine)
 	};
 	int i = 0;
 
+	pthread_mutex_lock(&dso__data_open_lock);
+
 	if (dso->data.status == DSO_DATA_STATUS_ERROR)
 		return -1;
 
-	pthread_mutex_lock(&dso__data_open_lock);
-
 	if (dso->data.fd >= 0)
 		goto out;
 
@@ -432,10 +433,31 @@ int dso__data_fd(struct dso *dso, struct machine *machine)
 	else
 		dso->data.status = DSO_DATA_STATUS_ERROR;
 
-	pthread_mutex_unlock(&dso__data_open_lock);
 	return dso->data.fd;
 }
 
+void dso__data_put_fd(struct dso *dso __maybe_unused)
+{
+	pthread_mutex_unlock(&dso__data_open_lock);
+}
+
+/**
+ * dso__data_get_fd - Get dso's data file descriptor
+ * @dso: dso object
+ * @machine: machine object
+ *
+ * Obsolete interface to find dso's file, open it and
+ * returns file descriptor.  It's not thread-safe in that
+ * the returned fd may be reused for other file.
+ */
+int dso__data_fd(struct dso *dso, struct machine *machine)
+{
+	int fd = dso__data_get_fd(dso, machine);
+
+	dso__data_put_fd(dso);
+	return fd;
+}
+
 bool dso__data_status_seen(struct dso *dso, enum dso_data_status_seen by)
 {
 	u32 flag = 1 << by;
@@ -1144,10 +1166,12 @@ size_t dso__fprintf(struct dso *dso, enum map_type type, FILE *fp)
 enum dso_type dso__type(struct dso *dso, struct machine *machine)
 {
 	int fd;
+	enum dso_type type = DSO__TYPE_UNKNOWN;
 
-	fd = dso__data_fd(dso, machine);
-	if (fd < 0)
-		return DSO__TYPE_UNKNOWN;
+	fd = dso__data_get_fd(dso, machine);
+	if (fd >= 0)
+		type = dso__type_fd(fd);
+	dso__data_put_fd(dso);
 
-	return dso__type_fd(fd);
+	return type;
 }
diff --git a/tools/perf/util/dso.h b/tools/perf/util/dso.h
index da188a73d034..9f1e67da9c01 100644
--- a/tools/perf/util/dso.h
+++ b/tools/perf/util/dso.h
@@ -197,7 +197,9 @@ bool dso__needs_decompress(struct dso *dso);
 
 /*
  * The dso__data_* external interface provides following functions:
- *   dso__data_fd
+ *   dso__data_fd (obsolete)
+ *   dso__data_get_fd
+ *   dso__data_put_fd
  *   dso__data_close
  *   dso__data_size
  *   dso__data_read_offset
@@ -214,8 +216,9 @@ bool dso__needs_decompress(struct dso *dso);
  * The current usage of the dso__data_* interface is as follows:
  *
  * Get DSO's fd:
- *   int fd = dso__data_fd(dso, machine);
+ *   int fd = dso__data_get_fd(dso, machine);
  *   USE 'fd' SOMEHOW
+ *   dso__data_put_fd(dso)
  *
  * Read DSO's data:
  *   n = dso__data_read_offset(dso_0, &machine, 0, buf, BUFSIZE);
@@ -235,6 +238,8 @@ bool dso__needs_decompress(struct dso *dso);
  * TODO
 */
 int dso__data_fd(struct dso *dso, struct machine *machine);
+int dso__data_get_fd(struct dso *dso, struct machine *machine);
+void dso__data_put_fd(struct dso *dso);
 void dso__data_close(struct dso *dso);
 
 off_t dso__data_size(struct dso *dso, struct machine *machine);
diff --git a/tools/perf/util/unwind-libunwind.c b/tools/perf/util/unwind-libunwind.c
index 9d7ecb26fde9..e807ba9d375a 100644
--- a/tools/perf/util/unwind-libunwind.c
+++ b/tools/perf/util/unwind-libunwind.c
@@ -271,13 +271,13 @@ static int read_unwind_spec_eh_frame(struct dso *dso, struct machine *machine,
 	u64 offset = dso->data.frame_offset;
 
 	if (offset == 0) {
-		fd = dso__data_fd(dso, machine);
-		if (fd < 0)
-			return -EINVAL;
-
-		/* Check the .eh_frame section for unwinding info */
-		offset = elf_section_offset(fd, ".eh_frame_hdr");
-		dso->data.frame_offset = offset;
+		fd = dso__data_get_fd(dso, machine);
+		if (fd >= 0) {
+			/* Check the .eh_frame section for unwinding info */
+			offset = elf_section_offset(fd, ".eh_frame_hdr");
+			dso->data.frame_offset = offset;
+		}
+		dso__data_put_fd(dso);
 	}
 
 	if (offset)
@@ -296,13 +296,19 @@ static int read_unwind_spec_debug_frame(struct dso *dso,
 	u64 ofs = dso->data.frame_offset;
 
 	if (ofs == 0) {
-		fd = dso__data_fd(dso, machine);
-		if (fd < 0)
-			return -EINVAL;
-
-		/* Check the .debug_frame section for unwinding info */
-		ofs = elf_section_offset(fd, ".debug_frame");
-		dso->data.frame_offset = ofs;
+		int ret = 0;
+
+		fd = dso__data_get_fd(dso, machine);
+		if (fd >= 0) {
+			/* Check the .debug_frame section for unwinding info */
+			ofs = elf_section_offset(fd, ".debug_frame");
+			dso->data.frame_offset = ofs;
+		} else
+			ret = -EINVAL;
+
+		dso__data_put_fd(dso);
+		if (ret)
+			return ret;
 	}
 
 	*offset = ofs;
@@ -355,10 +361,12 @@ find_proc_info(unw_addr_space_t as, unw_word_t ip, unw_proc_info_t *pi,
 #ifndef NO_LIBUNWIND_DEBUG_FRAME
 	/* Check the .debug_frame section for unwinding info */
 	if (!read_unwind_spec_debug_frame(map->dso, ui->machine, &segbase)) {
-		int fd = dso__data_fd(map->dso, ui->machine);
+		int fd = dso__data_get_fd(map->dso, ui->machine);
 		int is_exec = elf_is_exec(fd, map->dso->name);
 		unw_word_t base = is_exec ? 0 : map->start;
 
+		dso__data_put_fd(map->dso);
+
 		memset(&di, 0, sizeof(di));
 		if (dwarf_find_debug_frame(0, &di, ip, base, map->dso->name,
 					   map->start, map->end))
-- 
2.2.2


^ permalink raw reply related	[flat|nested] 221+ messages in thread

* [PATCH 29/38] perf session: Pass struct events stats to event processing functions
  2015-03-03  3:07 [RFC/PATCHSET 00/38] perf tools: Speed-up perf report by using multi thread (v3) Namhyung Kim
                   ` (27 preceding siblings ...)
  2015-03-03  3:07 ` [PATCH 28/38] perf tools: Add dso__data_get/put_fd() Namhyung Kim
@ 2015-03-03  3:07 ` Namhyung Kim
  2015-03-03  3:07 ` [PATCH 30/38] perf hists: Pass hists struct to hist_entry_iter struct Namhyung Kim
                   ` (8 subsequent siblings)
  37 siblings, 0 replies; 221+ messages in thread
From: Namhyung Kim @ 2015-03-03  3:07 UTC (permalink / raw)
  To: Arnaldo Carvalho de Melo
  Cc: Ingo Molnar, Peter Zijlstra, Jiri Olsa, LKML,
	Frederic Weisbecker, Adrian Hunter, Stephane Eranian, Andi Kleen,
	David Ahern

Pass stats structure so that it can point separate object when used in
multi-thread environment.

Signed-off-by: Namhyung Kim <namhyung@kernel.org>
---
 tools/perf/util/ordered-events.c |  4 ++-
 tools/perf/util/session.c        | 53 ++++++++++++++++++++++++----------------
 tools/perf/util/session.h        |  1 +
 3 files changed, 36 insertions(+), 22 deletions(-)

diff --git a/tools/perf/util/ordered-events.c b/tools/perf/util/ordered-events.c
index 077ddd25189f..35b7c0fd103b 100644
--- a/tools/perf/util/ordered-events.c
+++ b/tools/perf/util/ordered-events.c
@@ -183,7 +183,9 @@ static int __ordered_events__flush(struct perf_session *s,
 		if (ret)
 			pr_err("Can't parse sample, err = %d\n", ret);
 		else {
-			ret = perf_session__deliver_event(s, iter->event, &sample, tool,
+			ret = perf_session__deliver_event(s, &s->evlist->stats,
+							  iter->event,
+							  &sample, tool,
 							  iter->file_offset);
 			if (ret)
 				return ret;
diff --git a/tools/perf/util/session.c b/tools/perf/util/session.c
index d89dfa8592a9..0090eb8c6974 100644
--- a/tools/perf/util/session.c
+++ b/tools/perf/util/session.c
@@ -826,6 +826,7 @@ static struct machine *machines__find_for_cpumode(struct machines *machines,
 }
 
 static int deliver_sample_value(struct perf_evlist *evlist,
+				struct events_stats *stats,
 				struct perf_tool *tool,
 				union perf_event *event,
 				struct perf_sample *sample,
@@ -841,7 +842,7 @@ static int deliver_sample_value(struct perf_evlist *evlist,
 	}
 
 	if (!sid || sid->evsel == NULL) {
-		++evlist->stats.nr_unknown_id;
+		++stats->nr_unknown_id;
 		return 0;
 	}
 
@@ -849,6 +850,7 @@ static int deliver_sample_value(struct perf_evlist *evlist,
 }
 
 static int deliver_sample_group(struct perf_evlist *evlist,
+				struct events_stats *stats,
 				struct perf_tool *tool,
 				union  perf_event *event,
 				struct perf_sample *sample,
@@ -858,7 +860,7 @@ static int deliver_sample_group(struct perf_evlist *evlist,
 	u64 i;
 
 	for (i = 0; i < sample->read.group.nr; i++) {
-		ret = deliver_sample_value(evlist, tool, event, sample,
+		ret = deliver_sample_value(evlist, stats, tool, event, sample,
 					   &sample->read.group.values[i],
 					   machine);
 		if (ret)
@@ -870,6 +872,7 @@ static int deliver_sample_group(struct perf_evlist *evlist,
 
 static int
  perf_evlist__deliver_sample(struct perf_evlist *evlist,
+			     struct events_stats *stats,
 			     struct perf_tool *tool,
 			     union  perf_event *event,
 			     struct perf_sample *sample,
@@ -886,14 +889,15 @@ static int
 
 	/* For PERF_SAMPLE_READ we have either single or group mode. */
 	if (read_format & PERF_FORMAT_GROUP)
-		return deliver_sample_group(evlist, tool, event, sample,
+		return deliver_sample_group(evlist, stats, tool, event, sample,
 					    machine);
 	else
-		return deliver_sample_value(evlist, tool, event, sample,
+		return deliver_sample_value(evlist, stats, tool, event, sample,
 					    &sample->read.one, machine);
 }
 
 int perf_session__deliver_event(struct perf_session *session,
+				struct events_stats *stats,
 				union perf_event *event,
 				struct perf_sample *sample,
 				struct perf_tool *tool, u64 file_offset)
@@ -912,14 +916,15 @@ int perf_session__deliver_event(struct perf_session *session,
 	case PERF_RECORD_SAMPLE:
 		dump_sample(evsel, event, sample);
 		if (evsel == NULL) {
-			++evlist->stats.nr_unknown_id;
+			++stats->nr_unknown_id;
 			return 0;
 		}
 		if (machine == NULL) {
-			++evlist->stats.nr_unprocessable_samples;
+			++stats->nr_unprocessable_samples;
 			return 0;
 		}
-		return perf_evlist__deliver_sample(evlist, tool, event, sample, evsel, machine);
+		return perf_evlist__deliver_sample(evlist, stats, tool, event,
+						   sample, evsel, machine);
 	case PERF_RECORD_MMAP:
 		return tool->mmap(tool, event, sample, machine);
 	case PERF_RECORD_MMAP2:
@@ -932,7 +937,7 @@ int perf_session__deliver_event(struct perf_session *session,
 		return tool->exit(tool, event, sample, machine);
 	case PERF_RECORD_LOST:
 		if (tool->lost == perf_event__process_lost)
-			evlist->stats.total_lost += event->lost.lost;
+			stats->total_lost += event->lost.lost;
 		return tool->lost(tool, event, sample, machine);
 	case PERF_RECORD_READ:
 		return tool->read(tool, event, sample, evsel, machine);
@@ -941,7 +946,7 @@ int perf_session__deliver_event(struct perf_session *session,
 	case PERF_RECORD_UNTHROTTLE:
 		return tool->unthrottle(tool, event, sample, machine);
 	default:
-		++evlist->stats.nr_unknown_events;
+		++stats->nr_unknown_events;
 		return -1;
 	}
 }
@@ -996,7 +1001,8 @@ int perf_session__deliver_synth_event(struct perf_session *session,
 	if (event->header.type >= PERF_RECORD_USER_TYPE_START)
 		return perf_session__process_user_event(session, event, tool, 0);
 
-	return perf_session__deliver_event(session, event, sample, tool, 0);
+	return perf_session__deliver_event(session, &session->evlist->stats,
+					   event, sample, tool, 0);
 }
 
 static void event_swap(union perf_event *event, bool sample_id_all)
@@ -1064,6 +1070,7 @@ int perf_session__peek_event(struct perf_session *session, off_t file_offset,
 }
 
 static s64 perf_session__process_event(struct perf_session *session,
+				       struct events_stats *stats,
 				       union perf_event *event,
 				       struct perf_tool *tool,
 				       u64 file_offset)
@@ -1078,7 +1085,7 @@ static s64 perf_session__process_event(struct perf_session *session,
 	if (event->header.type >= PERF_RECORD_HEADER_MAX)
 		return -EINVAL;
 
-	events_stats__inc(&evlist->stats, event->header.type);
+	events_stats__inc(stats, event->header.type);
 
 	if (event->header.type >= PERF_RECORD_USER_TYPE_START)
 		return perf_session__process_user_event(session, event, tool, file_offset);
@@ -1097,8 +1104,8 @@ static s64 perf_session__process_event(struct perf_session *session,
 			return ret;
 	}
 
-	return perf_session__deliver_event(session, event, &sample, tool,
-					   file_offset);
+	return perf_session__deliver_event(session, stats, event, &sample,
+					   tool, file_offset);
 }
 
 void perf_event_header__bswap(struct perf_event_header *hdr)
@@ -1237,7 +1244,8 @@ static int __perf_session__process_pipe_events(struct perf_session *session,
 		}
 	}
 
-	if ((skip = perf_session__process_event(session, event, tool, head)) < 0) {
+	if ((skip = perf_session__process_event(session, &session->evlist->stats,
+						event, tool, head)) < 0) {
 		pr_err("%#" PRIx64 " [%#x]: failed to process type: %d\n",
 		       head, event->header.size, event->header.type);
 		err = -EINVAL;
@@ -1301,7 +1309,8 @@ fetch_mmaped_event(struct perf_session *session,
 #define NUM_MMAPS 128
 #endif
 
-static int __perf_session__process_events(struct perf_session *session, int fd,
+static int __perf_session__process_events(struct perf_session *session,
+					  struct events_stats *stats, int fd,
 					  u64 data_offset, u64 data_size,
 					  u64 file_size, struct perf_tool *tool)
 {
@@ -1374,8 +1383,8 @@ static int __perf_session__process_events(struct perf_session *session, int fd,
 	size = event->header.size;
 
 	if (size < sizeof(struct perf_event_header) ||
-	    (skip = perf_session__process_event(session, event, tool, file_pos))
-									< 0) {
+	    (skip = perf_session__process_event(session, stats, event,
+						tool, file_pos)) < 0) {
 		pr_err("%#" PRIx64 " [%#x]: failed to process type: %d\n",
 		       file_offset + head, event->header.size,
 		       event->header.type);
@@ -1412,6 +1421,7 @@ static int __perf_session__process_indexed_events(struct perf_session *session,
 {
 	struct perf_data_file *file = session->file;
 	u64 size = perf_data_file__size(file);
+	struct events_stats *stats = &session->evlist->stats;
 	int err = 0, i;
 
 	for (i = 0; i < (int)session->header.nr_index; i++) {
@@ -1428,7 +1438,7 @@ static int __perf_session__process_indexed_events(struct perf_session *session,
 		if (i > 0)
 			tool->ordered_events = false;
 
-		err = __perf_session__process_events(session,
+		err = __perf_session__process_events(session, stats,
 						     perf_data_file__fd(file),
 						     index->offset, index->size,
 						     size, tool);
@@ -1436,7 +1446,7 @@ static int __perf_session__process_indexed_events(struct perf_session *session,
 			break;
 	}
 
-	perf_tool__warn_about_errors(tool, &session->evlist->stats);
+	perf_tool__warn_about_errors(tool, stats);
 	return err;
 }
 
@@ -1445,6 +1455,7 @@ int perf_session__process_events(struct perf_session *session,
 {
 	struct perf_data_file *file = session->file;
 	u64 size = perf_data_file__size(file);
+	struct events_stats *stats = &session->evlist->stats;
 	int err;
 
 	if (perf_session__register_idle_thread(session) == NULL)
@@ -1455,13 +1466,13 @@ int perf_session__process_events(struct perf_session *session,
 	if (perf_session__has_index(session))
 		return __perf_session__process_indexed_events(session, tool);
 
-	err = __perf_session__process_events(session,
+	err = __perf_session__process_events(session, stats,
 					     perf_data_file__fd(file),
 					     session->header.data_offset,
 					     session->header.data_size,
 					     size, tool);
 
-	perf_tool__warn_about_errors(tool, &session->evlist->stats);
+	perf_tool__warn_about_errors(tool, stats);
 	return err;
 }
 
diff --git a/tools/perf/util/session.h b/tools/perf/util/session.h
index 4d264fef8968..c9a53ecf658d 100644
--- a/tools/perf/util/session.h
+++ b/tools/perf/util/session.h
@@ -58,6 +58,7 @@ int perf_session_queue_event(struct perf_session *s, union perf_event *event,
 void perf_tool__fill_defaults(struct perf_tool *tool);
 
 int perf_session__deliver_event(struct perf_session *session,
+				struct events_stats *stats,
 				union perf_event *event,
 				struct perf_sample *sample,
 				struct perf_tool *tool, u64 file_offset);
-- 
2.2.2


^ permalink raw reply related	[flat|nested] 221+ messages in thread

* [PATCH 30/38] perf hists: Pass hists struct to hist_entry_iter struct
  2015-03-03  3:07 [RFC/PATCHSET 00/38] perf tools: Speed-up perf report by using multi thread (v3) Namhyung Kim
                   ` (28 preceding siblings ...)
  2015-03-03  3:07 ` [PATCH 29/38] perf session: Pass struct events stats to event processing functions Namhyung Kim
@ 2015-03-03  3:07 ` Namhyung Kim
  2015-03-03  3:07 ` [PATCH 31/38] perf tools: Move BUILD_ID_SIZE definition to perf.h Namhyung Kim
                   ` (7 subsequent siblings)
  37 siblings, 0 replies; 221+ messages in thread
From: Namhyung Kim @ 2015-03-03  3:07 UTC (permalink / raw)
  To: Arnaldo Carvalho de Melo
  Cc: Ingo Molnar, Peter Zijlstra, Jiri Olsa, LKML,
	Frederic Weisbecker, Adrian Hunter, Stephane Eranian, Andi Kleen,
	David Ahern

This is a preparation for perf report multi-thread support.  When
multi-thread is enable, each thread will have its own hists during the
sample processing.

Signed-off-by: Namhyung Kim <namhyung@kernel.org>
---
 tools/perf/builtin-report.c       |  1 +
 tools/perf/builtin-top.c          |  1 +
 tools/perf/tests/hists_cumulate.c |  1 +
 tools/perf/tests/hists_filter.c   |  1 +
 tools/perf/tests/hists_output.c   |  1 +
 tools/perf/util/hist.c            | 20 +++++++-------------
 tools/perf/util/hist.h            |  1 +
 7 files changed, 13 insertions(+), 13 deletions(-)

diff --git a/tools/perf/builtin-report.c b/tools/perf/builtin-report.c
index 0d6e6bff7994..5adf269b84a9 100644
--- a/tools/perf/builtin-report.c
+++ b/tools/perf/builtin-report.c
@@ -138,6 +138,7 @@ static int process_sample_event(struct perf_tool *tool,
 	struct addr_location al;
 	struct hist_entry_iter iter = {
 		.evsel 			= evsel,
+		.hists 			= evsel__hists(evsel),
 		.sample 		= sample,
 		.session 		= rep->session,
 		.hide_unresolved 	= rep->hide_unresolved,
diff --git a/tools/perf/builtin-top.c b/tools/perf/builtin-top.c
index f33cb0e2aa0d..52c6d5d16ecb 100644
--- a/tools/perf/builtin-top.c
+++ b/tools/perf/builtin-top.c
@@ -775,6 +775,7 @@ static void perf_event__process_sample(struct perf_tool *tool,
 		struct hists *hists = evsel__hists(evsel);
 		struct hist_entry_iter iter = {
 			.evsel		= evsel,
+			.hists 		= evsel__hists(evsel),
 			.sample 	= sample,
 			.session 	= top->session,
 			.add_entry_cb 	= hist_iter__top_callback,
diff --git a/tools/perf/tests/hists_cumulate.c b/tools/perf/tests/hists_cumulate.c
index da64acbd35b7..273182c7cc12 100644
--- a/tools/perf/tests/hists_cumulate.c
+++ b/tools/perf/tests/hists_cumulate.c
@@ -88,6 +88,7 @@ static int add_hist_entries(struct hists *hists, struct machine *machine)
 		};
 		struct hist_entry_iter iter = {
 			.evsel = evsel,
+			.hists = evsel__hists(evsel),
 			.sample	= &sample,
 			.hide_unresolved = false,
 		};
diff --git a/tools/perf/tests/hists_filter.c b/tools/perf/tests/hists_filter.c
index f5c0c69383dc..67b9d498d731 100644
--- a/tools/perf/tests/hists_filter.c
+++ b/tools/perf/tests/hists_filter.c
@@ -64,6 +64,7 @@ static int add_hist_entries(struct perf_evlist *evlist,
 			};
 			struct hist_entry_iter iter = {
 				.evsel = evsel,
+				.hists = evsel__hists(evsel),
 				.sample = &sample,
 				.ops = &hist_iter_normal,
 				.hide_unresolved = false,
diff --git a/tools/perf/tests/hists_output.c b/tools/perf/tests/hists_output.c
index 4e3cff568eaa..541cf09280c1 100644
--- a/tools/perf/tests/hists_output.c
+++ b/tools/perf/tests/hists_output.c
@@ -58,6 +58,7 @@ static int add_hist_entries(struct hists *hists, struct machine *machine)
 		};
 		struct hist_entry_iter iter = {
 			.evsel = evsel,
+			.hists = evsel__hists(evsel),
 			.sample = &sample,
 			.ops = &hist_iter_normal,
 			.hide_unresolved = false,
diff --git a/tools/perf/util/hist.c b/tools/perf/util/hist.c
index dbe7f3744bf1..cbcfda5f1eac 100644
--- a/tools/perf/util/hist.c
+++ b/tools/perf/util/hist.c
@@ -510,7 +510,7 @@ iter_add_single_mem_entry(struct hist_entry_iter *iter, struct addr_location *al
 	u64 cost;
 	struct mem_info *mi = iter->priv;
 	struct perf_sample *sample = iter->sample;
-	struct hists *hists = evsel__hists(iter->evsel);
+	struct hists *hists = iter->hists;
 	struct hist_entry *he;
 
 	if (mi == NULL)
@@ -540,8 +540,7 @@ static int
 iter_finish_mem_entry(struct hist_entry_iter *iter,
 		      struct addr_location *al __maybe_unused)
 {
-	struct perf_evsel *evsel = iter->evsel;
-	struct hists *hists = evsel__hists(evsel);
+	struct hists *hists = iter->hists;
 	struct hist_entry *he = iter->he;
 	int err = -EINVAL;
 
@@ -613,8 +612,7 @@ static int
 iter_add_next_branch_entry(struct hist_entry_iter *iter, struct addr_location *al)
 {
 	struct branch_info *bi;
-	struct perf_evsel *evsel = iter->evsel;
-	struct hists *hists = evsel__hists(evsel);
+	struct hists *hists = iter->hists;
 	struct hist_entry *he = NULL;
 	int i = iter->curr;
 	int err = 0;
@@ -661,11 +659,10 @@ iter_prepare_normal_entry(struct hist_entry_iter *iter __maybe_unused,
 static int
 iter_add_single_normal_entry(struct hist_entry_iter *iter, struct addr_location *al)
 {
-	struct perf_evsel *evsel = iter->evsel;
 	struct perf_sample *sample = iter->sample;
 	struct hist_entry *he;
 
-	he = __hists__add_entry(evsel__hists(evsel), al, iter->parent, NULL, NULL,
+	he = __hists__add_entry(iter->hists, al, iter->parent, NULL, NULL,
 				sample->period, sample->weight,
 				sample->transaction, sample->time, true);
 	if (he == NULL)
@@ -680,7 +677,6 @@ iter_finish_normal_entry(struct hist_entry_iter *iter,
 			 struct addr_location *al __maybe_unused)
 {
 	struct hist_entry *he = iter->he;
-	struct perf_evsel *evsel = iter->evsel;
 	struct perf_sample *sample = iter->sample;
 
 	if (he == NULL)
@@ -688,7 +684,7 @@ iter_finish_normal_entry(struct hist_entry_iter *iter,
 
 	iter->he = NULL;
 
-	hists__inc_nr_samples(evsel__hists(evsel), he->filtered);
+	hists__inc_nr_samples(iter->hists, he->filtered);
 
 	return hist_entry__append_callchain(he, sample);
 }
@@ -720,8 +716,7 @@ static int
 iter_add_single_cumulative_entry(struct hist_entry_iter *iter,
 				 struct addr_location *al)
 {
-	struct perf_evsel *evsel = iter->evsel;
-	struct hists *hists = evsel__hists(evsel);
+	struct hists *hists = iter->hists;
 	struct perf_sample *sample = iter->sample;
 	struct hist_entry **he_cache = iter->priv;
 	struct hist_entry *he;
@@ -766,7 +761,6 @@ static int
 iter_add_next_cumulative_entry(struct hist_entry_iter *iter,
 			       struct addr_location *al)
 {
-	struct perf_evsel *evsel = iter->evsel;
 	struct perf_sample *sample = iter->sample;
 	struct hist_entry **he_cache = iter->priv;
 	struct hist_entry *he;
@@ -800,7 +794,7 @@ iter_add_next_cumulative_entry(struct hist_entry_iter *iter,
 		}
 	}
 
-	he = __hists__add_entry(evsel__hists(evsel), al, iter->parent, NULL, NULL,
+	he = __hists__add_entry(iter->hists, al, iter->parent, NULL, NULL,
 				sample->period, sample->weight,
 				sample->transaction, sample->time, false);
 	if (he == NULL)
diff --git a/tools/perf/util/hist.h b/tools/perf/util/hist.h
index 0afe15ba0277..6db118613ff5 100644
--- a/tools/perf/util/hist.h
+++ b/tools/perf/util/hist.h
@@ -86,6 +86,7 @@ struct hist_entry_iter {
 
 	bool hide_unresolved;
 
+	struct hists *hists;
 	struct perf_evsel *evsel;
 	struct perf_sample *sample;
 	struct perf_session *session;
-- 
2.2.2


^ permalink raw reply related	[flat|nested] 221+ messages in thread

* [PATCH 31/38] perf tools: Move BUILD_ID_SIZE definition to perf.h
  2015-03-03  3:07 [RFC/PATCHSET 00/38] perf tools: Speed-up perf report by using multi thread (v3) Namhyung Kim
                   ` (29 preceding siblings ...)
  2015-03-03  3:07 ` [PATCH 30/38] perf hists: Pass hists struct to hist_entry_iter struct Namhyung Kim
@ 2015-03-03  3:07 ` Namhyung Kim
  2015-03-03  3:07 ` [PATCH 32/38] perf report: Parallelize perf report using multi-thread Namhyung Kim
                   ` (6 subsequent siblings)
  37 siblings, 0 replies; 221+ messages in thread
From: Namhyung Kim @ 2015-03-03  3:07 UTC (permalink / raw)
  To: Arnaldo Carvalho de Melo
  Cc: Ingo Molnar, Peter Zijlstra, Jiri Olsa, LKML,
	Frederic Weisbecker, Adrian Hunter, Stephane Eranian, Andi Kleen,
	David Ahern

The util/event.h includes util/build-id.h only for BUILD_ID_SIZE.
This is a problem when I include util/event.h from util/tool.h which
is also included by util/build-id.h since it now makes a circular
dependency resulting in incomplete type error.

Signed-off-by: Namhyung Kim <namhyung@kernel.org>
---
 tools/perf/perf.h          | 1 +
 tools/perf/util/build-id.h | 2 --
 tools/perf/util/dso.h      | 1 +
 tools/perf/util/event.h    | 1 -
 4 files changed, 2 insertions(+), 3 deletions(-)

diff --git a/tools/perf/perf.h b/tools/perf/perf.h
index a03552849399..c32bee696f41 100644
--- a/tools/perf/perf.h
+++ b/tools/perf/perf.h
@@ -30,6 +30,7 @@ static inline unsigned long long rdclock(void)
 }
 
 #define MAX_NR_CPUS			256
+#define BUILD_ID_SIZE			20
 
 extern const char *input_name;
 extern bool perf_host, perf_guest;
diff --git a/tools/perf/util/build-id.h b/tools/perf/util/build-id.h
index 85011222cc14..e71304c9c86f 100644
--- a/tools/perf/util/build-id.h
+++ b/tools/perf/util/build-id.h
@@ -1,8 +1,6 @@
 #ifndef PERF_BUILD_ID_H_
 #define PERF_BUILD_ID_H_ 1
 
-#define BUILD_ID_SIZE 20
-
 #include "tool.h"
 #include "strlist.h"
 #include <linux/types.h>
diff --git a/tools/perf/util/dso.h b/tools/perf/util/dso.h
index 9f1e67da9c01..0e5a9897d1bf 100644
--- a/tools/perf/util/dso.h
+++ b/tools/perf/util/dso.h
@@ -7,6 +7,7 @@
 #include <linux/types.h>
 #include <linux/bitops.h>
 #include "map.h"
+#include "perf.h"
 #include "build-id.h"
 
 enum dso_binary_type {
diff --git a/tools/perf/util/event.h b/tools/perf/util/event.h
index 27261320249a..1f86c279520e 100644
--- a/tools/perf/util/event.h
+++ b/tools/perf/util/event.h
@@ -6,7 +6,6 @@
 
 #include "../perf.h"
 #include "map.h"
-#include "build-id.h"
 #include "perf_regs.h"
 
 struct mmap_event {
-- 
2.2.2


^ permalink raw reply related	[flat|nested] 221+ messages in thread

* [PATCH 32/38] perf report: Parallelize perf report using multi-thread
  2015-03-03  3:07 [RFC/PATCHSET 00/38] perf tools: Speed-up perf report by using multi thread (v3) Namhyung Kim
                   ` (30 preceding siblings ...)
  2015-03-03  3:07 ` [PATCH 31/38] perf tools: Move BUILD_ID_SIZE definition to perf.h Namhyung Kim
@ 2015-03-03  3:07 ` Namhyung Kim
  2015-03-03  3:07 ` [PATCH 33/38] perf tools: Add missing_threads rb tree Namhyung Kim
                   ` (5 subsequent siblings)
  37 siblings, 0 replies; 221+ messages in thread
From: Namhyung Kim @ 2015-03-03  3:07 UTC (permalink / raw)
  To: Arnaldo Carvalho de Melo
  Cc: Ingo Molnar, Peter Zijlstra, Jiri Olsa, LKML,
	Frederic Weisbecker, Adrian Hunter, Stephane Eranian, Andi Kleen,
	David Ahern

Introduce perf_session__process_events_mt() to enable multi-thread
sample processing.  It allocates a struct perf_tool_mt and fills
needed info in it.

The session and hists event stats are counted for each thread and
summed after finishing the processing.  Similarly hist entries are
added to per-thread hists first and then move to the original hists
using hists__mt_resort().  This function reuses hists__collapse_
resort() code so makes sort__need_collapse force to true and skips
the collapsing function.

Note that most of preprocessing stage is already done by processing
meta events in dummy tracking evsel first.  We can find corresponding
thread and map based on the sample time and symbol loading and dso
cache access is protected by pthread mutex.

Signed-off-by: Namhyung Kim <namhyung@kernel.org>
---
 tools/perf/util/hist.c    |  75 +++++++++++++++++++-----
 tools/perf/util/hist.h    |   3 +
 tools/perf/util/session.c | 142 ++++++++++++++++++++++++++++++++++++++++++++++
 tools/perf/util/session.h |   2 +
 tools/perf/util/tool.h    |  12 ++++
 5 files changed, 221 insertions(+), 13 deletions(-)

diff --git a/tools/perf/util/hist.c b/tools/perf/util/hist.c
index cbcfda5f1eac..a6bbe730c4af 100644
--- a/tools/perf/util/hist.c
+++ b/tools/perf/util/hist.c
@@ -945,7 +945,7 @@ void hist_entry__delete(struct hist_entry *he)
  * collapse the histogram
  */
 
-static bool hists__collapse_insert_entry(struct hists *hists __maybe_unused,
+static bool hists__collapse_insert_entry(struct hists *hists,
 					 struct rb_root *root,
 					 struct hist_entry *he)
 {
@@ -982,6 +982,13 @@ static bool hists__collapse_insert_entry(struct hists *hists __maybe_unused,
 	}
 	hists->nr_entries++;
 
+	/*
+	 * For multi-threaded report, he->hists points to a dummy
+	 * hists in the struct perf_tool_mt.  Please see
+	 * perf_session__process_events_mt().
+	 */
+	he->hists = hists;
+
 	rb_link_node(&he->rb_node_in, parent, p);
 	rb_insert_color(&he->rb_node_in, root);
 	return true;
@@ -1009,19 +1016,12 @@ static void hists__apply_filters(struct hists *hists, struct hist_entry *he)
 	hists__filter_entry_by_symbol(hists, he);
 }
 
-void hists__collapse_resort(struct hists *hists, struct ui_progress *prog)
+static void __hists__collapse_resort(struct hists *hists, struct rb_root *root,
+				     struct ui_progress *prog)
 {
-	struct rb_root *root;
 	struct rb_node *next;
 	struct hist_entry *n;
 
-	if (!sort__need_collapse)
-		return;
-
-	hists->nr_entries = 0;
-
-	root = hists__get_rotate_entries_in(hists);
-
 	next = rb_first(root);
 
 	while (next) {
@@ -1044,6 +1044,27 @@ void hists__collapse_resort(struct hists *hists, struct ui_progress *prog)
 	}
 }
 
+void hists__collapse_resort(struct hists *hists, struct ui_progress *prog)
+{
+	struct rb_root *root;
+
+	if (!sort__need_collapse)
+		return;
+
+	hists->nr_entries = 0;
+
+	root = hists__get_rotate_entries_in(hists);
+	__hists__collapse_resort(hists, root, prog);
+}
+
+void hists__mt_resort(struct hists *dst, struct hists *src)
+{
+	struct rb_root *root = src->entries_in;
+
+	sort__need_collapse = 1;
+	__hists__collapse_resort(dst, root, NULL);
+}
+
 static int hist_entry__sort(struct hist_entry *a, struct hist_entry *b)
 {
 	struct perf_hpp_fmt *fmt;
@@ -1272,6 +1293,29 @@ void events_stats__inc(struct events_stats *stats, u32 type)
 	++stats->nr_events[type];
 }
 
+void events_stats__add(struct events_stats *dst, struct events_stats *src)
+{
+	int i;
+
+#define ADD(_field)  dst->_field += src->_field
+
+	ADD(total_period);
+	ADD(total_non_filtered_period);
+	ADD(total_lost);
+	ADD(total_invalid_chains);
+	ADD(nr_non_filtered_samples);
+	ADD(nr_lost_warned);
+	ADD(nr_unknown_events);
+	ADD(nr_invalid_chains);
+	ADD(nr_unknown_id);
+	ADD(nr_unprocessable_samples);
+
+	for (i = 0; i < PERF_RECORD_HEADER_MAX; i++)
+		ADD(nr_events[i]);
+
+#undef ADD
+}
+
 void hists__inc_nr_events(struct hists *hists, u32 type)
 {
 	events_stats__inc(&hists->stats, type);
@@ -1448,16 +1492,21 @@ int perf_hist_config(const char *var, const char *value)
 	return 0;
 }
 
-static int hists_evsel__init(struct perf_evsel *evsel)
+void __hists__init(struct hists *hists)
 {
-	struct hists *hists = evsel__hists(evsel);
-
 	memset(hists, 0, sizeof(*hists));
 	hists->entries_in_array[0] = hists->entries_in_array[1] = RB_ROOT;
 	hists->entries_in = &hists->entries_in_array[0];
 	hists->entries_collapsed = RB_ROOT;
 	hists->entries = RB_ROOT;
 	pthread_mutex_init(&hists->lock, NULL);
+}
+
+static int hists_evsel__init(struct perf_evsel *evsel)
+{
+	struct hists *hists = evsel__hists(evsel);
+
+	__hists__init(hists);
 	return 0;
 }
 
diff --git a/tools/perf/util/hist.h b/tools/perf/util/hist.h
index 6db118613ff5..3ef30f632948 100644
--- a/tools/perf/util/hist.h
+++ b/tools/perf/util/hist.h
@@ -123,6 +123,7 @@ int hist_entry__sort_snprintf(struct hist_entry *he, char *bf, size_t size,
 void hist_entry__delete(struct hist_entry *he);
 
 void hists__output_resort(struct hists *hists, struct ui_progress *prog);
+void hists__mt_resort(struct hists *dst, struct hists *src);
 void hists__collapse_resort(struct hists *hists, struct ui_progress *prog);
 
 void hists__decay_entries(struct hists *hists, bool zap_user, bool zap_kernel);
@@ -135,6 +136,7 @@ void hists__inc_stats(struct hists *hists, struct hist_entry *h);
 void hists__inc_nr_events(struct hists *hists, u32 type);
 void hists__inc_nr_samples(struct hists *hists, bool filtered);
 void events_stats__inc(struct events_stats *stats, u32 type);
+void events_stats__add(struct events_stats *dst, struct events_stats *src);
 size_t events_stats__fprintf(struct events_stats *stats, FILE *fp);
 
 size_t hists__fprintf(struct hists *hists, bool show_header, int max_rows,
@@ -178,6 +180,7 @@ static inline struct hists *evsel__hists(struct perf_evsel *evsel)
 }
 
 int hists__init(void);
+void __hists__init(struct hists *hists);
 
 struct perf_hpp {
 	char *buf;
diff --git a/tools/perf/util/session.c b/tools/perf/util/session.c
index 0090eb8c6974..566d62e58928 100644
--- a/tools/perf/util/session.c
+++ b/tools/perf/util/session.c
@@ -1476,6 +1476,148 @@ int perf_session__process_events(struct perf_session *session,
 	return err;
 }
 
+static void *processing_thread_idx(void *arg)
+{
+	struct perf_tool_mt *mt_tool = arg;
+	struct perf_session *session = mt_tool->session;
+	int fd = perf_data_file__fd(session->file);
+	u64 offset = session->header.index[mt_tool->idx].offset;
+	u64 size = session->header.index[mt_tool->idx].size;
+	u64 file_size = perf_data_file__size(session->file);
+
+	pr_debug("processing samples using thread [%d]\n", mt_tool->idx);
+	if (__perf_session__process_events(session, &mt_tool->stats,
+					   fd, offset, size, file_size,
+					   &mt_tool->tool) < 0) {
+		pr_err("processing samples failed (thread [%d])\n", mt_tool->idx);
+		return NULL;
+	}
+
+	pr_debug("processing samples done for thread [%d]\n", mt_tool->idx);
+	return arg;
+}
+
+int perf_session__process_events_mt(struct perf_session *session,
+				    struct perf_tool *tool, void *arg)
+{
+	struct perf_data_file *file = session->file;
+	struct perf_evlist *evlist = session->evlist;
+	struct perf_evsel *evsel;
+	u64 nr_entries = 0;
+	struct perf_tool_mt *mt_tools = NULL;
+	struct perf_tool_mt *mt;
+	pthread_t *th_id;
+	int err, i, k;
+	int nr_index = session->header.nr_index;
+	u64 size = perf_data_file__size(file);
+
+	if (perf_data_file__is_pipe(file) || !session->header.index) {
+		pr_err("data file doesn't contain the index table\n");
+		return -EINVAL;
+	}
+
+	if (perf_session__register_idle_thread(session) == NULL)
+		return -ENOMEM;
+
+	err = __perf_session__process_events(session, &evlist->stats,
+					     perf_data_file__fd(file),
+					     session->header.index[0].offset,
+					     session->header.index[0].size,
+					     size, tool);
+	if (err)
+		return err;
+
+	th_id = calloc(nr_index, sizeof(*th_id));
+	if (th_id == NULL)
+		goto out;
+
+	mt_tools = calloc(nr_index, sizeof(*mt_tools));
+	if (mt_tools == NULL)
+		goto out;
+
+	for (i = 1; i < nr_index; i++) {
+		mt = &mt_tools[i];
+
+		memcpy(&mt->tool, tool, sizeof(*tool));
+
+		mt->hists = calloc(evlist->nr_entries, sizeof(*mt->hists));
+		if (mt->hists == NULL)
+			goto err;
+
+		for (k = 0; k < evlist->nr_entries; k++)
+			__hists__init(&mt->hists[k]);
+
+		mt->session = session;
+		mt->tool.ordered_events = false;
+		mt->idx = i;
+		mt->priv = arg;
+
+		pthread_create(&th_id[i], NULL, processing_thread_idx, mt);
+	}
+
+	for (i = 1; i < nr_index; i++) {
+		pthread_join(th_id[i], (void **)&mt);
+		if (mt == NULL) {
+			err = -EINVAL;
+			continue;
+		}
+
+		events_stats__add(&evlist->stats, &mt->stats);
+
+		evlist__for_each(evlist, evsel) {
+			struct hists *hists = evsel__hists(evsel);
+
+			events_stats__add(&hists->stats,
+					  &mt->hists[evsel->idx].stats);
+
+			nr_entries += mt->hists[evsel->idx].nr_entries;
+		}
+	}
+
+	for (i = 1; i < nr_index; i++) {
+		mt = &mt_tools[i];
+
+		evlist__for_each(evlist, evsel) {
+			struct hists *hists = evsel__hists(evsel);
+
+			if (perf_evsel__is_dummy_tracking(evsel))
+				continue;
+
+			hists__mt_resort(hists, &mt->hists[evsel->idx]);
+
+			/* Non-group events are considered as leader */
+			if (symbol_conf.event_group &&
+			    !perf_evsel__is_group_leader(evsel)) {
+				struct hists *leader_hists;
+
+				leader_hists = evsel__hists(evsel->leader);
+				hists__match(leader_hists, hists);
+				hists__link(leader_hists, hists);
+			}
+		}
+	}
+
+out:
+	perf_tool__warn_about_errors(tool, &evlist->stats);
+
+	if (mt_tools) {
+		for (i = 1; i < nr_index; i++)
+			free(mt_tools[i].hists);
+		free(mt_tools);
+	}
+
+	free(th_id);
+	return err;
+
+err:
+	while (i-- > 1) {
+		pthread_cancel(th_id[i]);
+		pthread_join(th_id[i], NULL);
+	}
+
+	goto out;
+}
+
 bool perf_session__has_traces(struct perf_session *session, const char *msg)
 {
 	struct perf_evsel *evsel;
diff --git a/tools/perf/util/session.h b/tools/perf/util/session.h
index c9a53ecf658d..90b5dce9ea79 100644
--- a/tools/perf/util/session.h
+++ b/tools/perf/util/session.h
@@ -50,6 +50,8 @@ int perf_session__peek_event(struct perf_session *session, off_t file_offset,
 
 int perf_session__process_events(struct perf_session *session,
 				 struct perf_tool *tool);
+int perf_session__process_events_mt(struct perf_session *session,
+				    struct perf_tool *tool, void *arg);
 
 int perf_session_queue_event(struct perf_session *s, union perf_event *event,
 			     struct perf_tool *tool, struct perf_sample *sample,
diff --git a/tools/perf/util/tool.h b/tools/perf/util/tool.h
index bb2708bbfaca..a04826bbe991 100644
--- a/tools/perf/util/tool.h
+++ b/tools/perf/util/tool.h
@@ -2,6 +2,7 @@
 #define __PERF_TOOL_H
 
 #include <stdbool.h>
+#include "util/event.h"
 
 struct perf_session;
 union perf_event;
@@ -10,6 +11,7 @@ struct perf_evsel;
 struct perf_sample;
 struct perf_tool;
 struct machine;
+struct hists;
 
 typedef int (*event_sample)(struct perf_tool *tool, union perf_event *event,
 			    struct perf_sample *sample,
@@ -45,4 +47,14 @@ struct perf_tool {
 	bool		ordering_requires_timestamps;
 };
 
+struct perf_tool_mt {
+	struct perf_tool	tool;
+	struct events_stats	stats;
+	struct hists		*hists;
+	struct perf_session	*session;
+	int			idx;
+
+	void			*priv;
+};
+
 #endif /* __PERF_TOOL_H */
-- 
2.2.2


^ permalink raw reply related	[flat|nested] 221+ messages in thread

* [PATCH 33/38] perf tools: Add missing_threads rb tree
  2015-03-03  3:07 [RFC/PATCHSET 00/38] perf tools: Speed-up perf report by using multi thread (v3) Namhyung Kim
                   ` (31 preceding siblings ...)
  2015-03-03  3:07 ` [PATCH 32/38] perf report: Parallelize perf report using multi-thread Namhyung Kim
@ 2015-03-03  3:07 ` Namhyung Kim
  2015-03-03  3:07 ` [PATCH 34/38] perf record: Synthesize COMM event for a command line workload Namhyung Kim
                   ` (4 subsequent siblings)
  37 siblings, 0 replies; 221+ messages in thread
From: Namhyung Kim @ 2015-03-03  3:07 UTC (permalink / raw)
  To: Arnaldo Carvalho de Melo
  Cc: Ingo Molnar, Peter Zijlstra, Jiri Olsa, LKML,
	Frederic Weisbecker, Adrian Hunter, Stephane Eranian, Andi Kleen,
	David Ahern

Sometimes it's possible to miss certain meta events like fork/exit and
in this case it can fail to find such thread in the machine's rbtree.
But adding a thread to the tree is dangerous since it's now executed
in multi-thread environment otherwise it'll add an overhead in order
to grab a lock for every search.  So adds a separate missing_threads
tree and protect it with a mutex.  It's expected to be accessed only
if a thread is not found in a normal tree.

Signed-off-by: Namhyung Kim <namhyung@kernel.org>
---
 tools/perf/tests/thread-lookup-time.c |   8 ++-
 tools/perf/util/build-id.c            |   9 ++-
 tools/perf/util/machine.c             | 129 +++++++++++++++++++++-------------
 tools/perf/util/machine.h             |   2 +
 tools/perf/util/session.c             |   8 +--
 tools/perf/util/thread.h              |   1 +
 6 files changed, 101 insertions(+), 56 deletions(-)

diff --git a/tools/perf/tests/thread-lookup-time.c b/tools/perf/tests/thread-lookup-time.c
index 6237ecf8caae..04cdde9329d6 100644
--- a/tools/perf/tests/thread-lookup-time.c
+++ b/tools/perf/tests/thread-lookup-time.c
@@ -7,7 +7,9 @@
 static int thread__print_cb(struct thread *th, void *arg __maybe_unused)
 {
 	printf("thread: %d, start time: %"PRIu64" %s\n",
-	       th->tid, th->start_time, th->dead ? "(dead)" : "");
+	       th->tid, th->start_time,
+	       th->dead ? "(dead)" : th->exited ? "(exited)" :
+	       th->missing ? "(missing)" : "");
 	return 0;
 }
 
@@ -105,6 +107,8 @@ static int lookup_with_timestamp(struct machine *machine)
 			machine__findnew_thread_time(machine, 0, 0, 70000) == t3);
 
 	machine__delete_threads(machine);
+	machine__delete_dead_threads(machine);
+	machine__delete_missing_threads(machine);
 	return 0;
 }
 
@@ -146,6 +150,8 @@ static int lookup_without_timestamp(struct machine *machine)
 			machine__findnew_thread_time(machine, 0, 0, -1ULL) == t3);
 
 	machine__delete_threads(machine);
+	machine__delete_dead_threads(machine);
+	machine__delete_missing_threads(machine);
 	return 0;
 }
 
diff --git a/tools/perf/util/build-id.c b/tools/perf/util/build-id.c
index ffdc338df925..5b8974400422 100644
--- a/tools/perf/util/build-id.c
+++ b/tools/perf/util/build-id.c
@@ -60,7 +60,14 @@ static int perf_event__exit_del_thread(struct perf_tool *tool __maybe_unused,
 		    event->fork.ppid, event->fork.ptid);
 
 	if (thread) {
-		rb_erase(&thread->rb_node, &machine->threads);
+		if (thread->dead)
+			rb_erase(&thread->rb_node, &machine->dead_threads);
+		else if (thread->missing)
+			rb_erase(&thread->rb_node, &machine->missing_threads);
+		else
+			rb_erase(&thread->rb_node, &machine->threads);
+
+		list_del(&thread->tid_node);
 		machine->last_match = NULL;
 		thread__delete(thread);
 	}
diff --git a/tools/perf/util/machine.c b/tools/perf/util/machine.c
index 63d860dca74b..ec401f82efb3 100644
--- a/tools/perf/util/machine.c
+++ b/tools/perf/util/machine.c
@@ -30,6 +30,7 @@ int machine__init(struct machine *machine, const char *root_dir, pid_t pid)
 
 	machine->threads = RB_ROOT;
 	machine->dead_threads = RB_ROOT;
+	machine->missing_threads = RB_ROOT;
 	machine->last_match = NULL;
 
 	machine->vdso_info = NULL;
@@ -90,6 +91,19 @@ static void dsos__delete(struct dsos *dsos)
 	}
 }
 
+void machine__delete_missing_threads(struct machine *machine)
+{
+	struct rb_node *nd = rb_first(&machine->missing_threads);
+
+	while (nd) {
+		struct thread *t = rb_entry(nd, struct thread, rb_node);
+
+		nd = rb_next(nd);
+		rb_erase(&t->rb_node, &machine->missing_threads);
+		thread__delete(t);
+	}
+}
+
 void machine__delete_dead_threads(struct machine *machine)
 {
 	struct rb_node *nd = rb_first(&machine->dead_threads);
@@ -435,20 +449,14 @@ struct thread *machine__find_thread(struct machine *machine, pid_t pid,
 	return __machine__findnew_thread(machine, pid, tid, false);
 }
 
-static struct thread *__machine__findnew_thread_time(struct machine *machine,
-						     pid_t pid, pid_t tid,
-						     u64 timestamp, bool create)
+static struct thread *machine__find_dead_thread_time(struct machine *machine,
+						     pid_t pid __maybe_unused,
+						     pid_t tid, u64 timestamp)
 {
-	struct thread *curr, *pos, *new;
-	struct thread *th = NULL;
-	struct rb_node **p;
+	struct thread *th, *pos;
+	struct rb_node **p = &machine->dead_threads.rb_node;
 	struct rb_node *parent = NULL;
 
-	curr = __machine__findnew_thread(machine, pid, tid, false);
-	if (curr && timestamp >= curr->start_time)
-		return curr;
-
-	p = &machine->dead_threads.rb_node;
 	while (*p != NULL) {
 		parent = *p;
 		th = rb_entry(parent, struct thread, rb_node);
@@ -462,10 +470,9 @@ static struct thread *__machine__findnew_thread_time(struct machine *machine,
 				}
 			}
 
-			if (timestamp >= th->start_time) {
-				machine__update_thread_pid(machine, th, pid);
+			if (timestamp >= th->start_time)
 				return th;
-			}
+
 			break;
 		}
 
@@ -475,50 +482,67 @@ static struct thread *__machine__findnew_thread_time(struct machine *machine,
 			p = &(*p)->rb_right;
 	}
 
-	if (!create)
-		return NULL;
+	return NULL;
+}
 
-	if (!curr && !*p)
-		return __machine__findnew_thread(machine, pid, tid, true);
+static struct thread *__machine__findnew_thread_time(struct machine *machine,
+						     pid_t pid, pid_t tid,
+						     u64 timestamp, bool create)
+{
+	struct thread *th, *new = NULL;
+	struct rb_node **p = &machine->missing_threads.rb_node;
+	struct rb_node *parent = NULL;
 
-	new = thread__new(pid, tid);
-	if (new == NULL)
-		return NULL;
+	static pthread_mutex_t missing_thread_lock = PTHREAD_MUTEX_INITIALIZER;
 
-	new->dead = true;
-	new->start_time = timestamp;
+	th = __machine__findnew_thread(machine, pid, tid, false);
+	if (th && timestamp >= th->start_time)
+		return th;
 
-	if (*p) {
-		list_for_each_entry(pos, &th->tid_node, tid_node) {
-			/* sort by time */
-			if (timestamp >= pos->start_time) {
-				th = pos;
-				break;
-			}
+	th = machine__find_dead_thread_time(machine, pid, tid, timestamp);
+	if (th)
+		return th;
+
+	pthread_mutex_lock(&missing_thread_lock);
+
+	while (*p != NULL) {
+		parent = *p;
+		th = rb_entry(parent, struct thread, rb_node);
+
+		if (th->tid == tid) {
+			pthread_mutex_unlock(&missing_thread_lock);
+			return th;
 		}
-		list_add_tail(&new->tid_node, &th->tid_node);
-	} else {
-		rb_link_node(&new->rb_node, parent, p);
-		rb_insert_color(&new->rb_node, &machine->dead_threads);
+
+		if (tid < th->tid)
+			p = &(*p)->rb_left;
+		else
+			p = &(*p)->rb_right;
 	}
 
+	if (!create)
+		goto out;
+
+	new = thread__new(pid, tid);
+	if (new == NULL)
+		goto out;
+
+	/* missing threads are not bothered with timestamp */
+	new->start_time = 0;
+	new->missing = true;
+
 	/*
-	 * We have to initialize map_groups separately
-	 * after rb tree is updated.
-	 *
-	 * The reason is that we call machine__findnew_thread
-	 * within thread__init_map_groups to find the thread
-	 * leader and that would screwed the rb tree.
+	 * missing threads have their own map groups regardless of
+	 * leader for the sake of simplicity.  it's okay since the map
+	 * groups has no map in it anyway.
 	 */
-	if (thread__init_map_groups(new, machine)) {
-		if (!list_empty(&new->tid_node))
-			list_del(&new->tid_node);
-		else
-			rb_erase(&new->rb_node, &machine->dead_threads);
+	new->mg = map_groups__new(machine);
 
-		thread__delete(new);
-		return NULL;
-	}
+	rb_link_node(&new->rb_node, parent, p);
+	rb_insert_color(&new->rb_node, &machine->missing_threads);
+
+out:
+	pthread_mutex_unlock(&missing_thread_lock);
 
 	return new;
 }
@@ -1357,6 +1381,7 @@ static void machine__remove_thread(struct machine *machine, struct thread *th)
 
 	machine->last_match = NULL;
 	rb_erase(&th->rb_node, &machine->threads);
+	RB_CLEAR_NODE(&th->rb_node);
 
 	th->dead = true;
 
@@ -1918,6 +1943,14 @@ int machine__for_each_thread(struct machine *machine,
 				return rc;
 		}
 	}
+
+	for (nd = rb_first(&machine->missing_threads); nd; nd = rb_next(nd)) {
+		thread = rb_entry(nd, struct thread, rb_node);
+		rc = fn(thread, priv);
+		if (rc != 0)
+			return rc;
+	}
+
 	return rc;
 }
 
diff --git a/tools/perf/util/machine.h b/tools/perf/util/machine.h
index 38ead24f0f47..d43310d246c1 100644
--- a/tools/perf/util/machine.h
+++ b/tools/perf/util/machine.h
@@ -31,6 +31,7 @@ struct machine {
 	char		  *root_dir;
 	struct rb_root	  threads;
 	struct rb_root	  dead_threads;
+	struct rb_root	  missing_threads;
 	struct thread	  *last_match;
 	struct vdso_info  *vdso_info;
 	struct dsos	  user_dsos;
@@ -116,6 +117,7 @@ void machines__set_comm_exec(struct machines *machines, bool comm_exec);
 struct machine *machine__new_host(void);
 int machine__init(struct machine *machine, const char *root_dir, pid_t pid);
 void machine__exit(struct machine *machine);
+void machine__delete_missing_threads(struct machine *machine);
 void machine__delete_dead_threads(struct machine *machine);
 void machine__delete_threads(struct machine *machine);
 void machine__delete(struct machine *machine);
diff --git a/tools/perf/util/session.c b/tools/perf/util/session.c
index 566d62e58928..49ded46104dd 100644
--- a/tools/perf/util/session.c
+++ b/tools/perf/util/session.c
@@ -138,14 +138,11 @@ struct perf_session *perf_session__new(struct perf_data_file *file,
 	return NULL;
 }
 
-static void perf_session__delete_dead_threads(struct perf_session *session)
-{
-	machine__delete_dead_threads(&session->machines.host);
-}
-
 static void perf_session__delete_threads(struct perf_session *session)
 {
 	machine__delete_threads(&session->machines.host);
+	machine__delete_dead_threads(&session->machines.host);
+	machine__delete_missing_threads(&session->machines.host);
 }
 
 static void perf_session_env__delete(struct perf_session_env *env)
@@ -167,7 +164,6 @@ static void perf_session_env__delete(struct perf_session_env *env)
 void perf_session__delete(struct perf_session *session)
 {
 	perf_session__destroy_kernel_maps(session);
-	perf_session__delete_dead_threads(session);
 	perf_session__delete_threads(session);
 	perf_session_env__delete(&session->header.env);
 	machines__exit(&session->machines);
diff --git a/tools/perf/util/thread.h b/tools/perf/util/thread.h
index 5209ad5adadf..88fee3d8c0dc 100644
--- a/tools/perf/util/thread.h
+++ b/tools/perf/util/thread.h
@@ -23,6 +23,7 @@ struct thread {
 	bool			comm_set;
 	bool			exited; /* if set thread has exited */
 	bool			dead; /* thread is in dead_threads list */
+	bool			missing; /* thread is in missing_threads list */
 	struct list_head	comm_list;
 	int			comm_len;
 	u64			db_id;
-- 
2.2.2


^ permalink raw reply related	[flat|nested] 221+ messages in thread

* [PATCH 34/38] perf record: Synthesize COMM event for a command line workload
  2015-03-03  3:07 [RFC/PATCHSET 00/38] perf tools: Speed-up perf report by using multi thread (v3) Namhyung Kim
                   ` (32 preceding siblings ...)
  2015-03-03  3:07 ` [PATCH 33/38] perf tools: Add missing_threads rb tree Namhyung Kim
@ 2015-03-03  3:07 ` Namhyung Kim
  2015-03-03  3:07 ` [PATCH 35/38] perf tools: Fix progress ui to support multi thread Namhyung Kim
                   ` (3 subsequent siblings)
  37 siblings, 0 replies; 221+ messages in thread
From: Namhyung Kim @ 2015-03-03  3:07 UTC (permalink / raw)
  To: Arnaldo Carvalho de Melo
  Cc: Ingo Molnar, Peter Zijlstra, Jiri Olsa, LKML,
	Frederic Weisbecker, Adrian Hunter, Stephane Eranian, Andi Kleen,
	David Ahern

When perf creates a new child to profile, the events are enabled on
exec().  And in this case, it doesn't synthesize any event for the
child since they'll be generated during exec().  But there's an window
between the enabling and the event generation.

It used to be overcome since samples are only in kernel (so we always
have the map) and the comm is overridden by a later COMM event.
However it won't work anymore since those samples will go to a missing
thread now but the COMM event will create a (current) thread.  This
leads to those early samples (like native_write_msr_safe) not having a
comm but pid (like ':15328').

So it needs to synthesize COMM event for the child explicitly before
enabling so that it can have a correct comm.  But at this time, the
comm will be "perf" since it's not exec-ed yet.

Signed-off-by: Namhyung Kim <namhyung@kernel.org>
---
 tools/perf/builtin-record.c | 18 +++++++++++++++++-
 tools/perf/util/event.c     |  2 +-
 tools/perf/util/event.h     |  5 +++++
 3 files changed, 23 insertions(+), 2 deletions(-)

diff --git a/tools/perf/builtin-record.c b/tools/perf/builtin-record.c
index ecf8e7293015..6f141f17c4ba 100644
--- a/tools/perf/builtin-record.c
+++ b/tools/perf/builtin-record.c
@@ -605,8 +605,24 @@ static int __cmd_record(struct record *rec, int argc, const char **argv)
 	/*
 	 * Let the child rip
 	 */
-	if (forks)
+	if (forks) {
+		union perf_event *comm_event;
+
+		comm_event = malloc(sizeof(*comm_event) + machine->id_hdr_size);
+		if (comm_event == NULL)
+			goto out_child;
+
+		err = perf_event__synthesize_comm(tool, comm_event,
+						  rec->evlist->threads->map[0],
+						  process_synthesized_event,
+						  machine);
+		free(comm_event);
+
+		if (err < 0)
+			goto out_child;
+
 		perf_evlist__start_workload(rec->evlist);
+	}
 
 	if (opts->initial_delay) {
 		usleep(opts->initial_delay * 1000);
diff --git a/tools/perf/util/event.c b/tools/perf/util/event.c
index 5abf7086c97c..2ad7e2805400 100644
--- a/tools/perf/util/event.c
+++ b/tools/perf/util/event.c
@@ -127,7 +127,7 @@ static pid_t perf_event__prepare_comm(union perf_event *event, pid_t pid,
 	return tgid;
 }
 
-static pid_t perf_event__synthesize_comm(struct perf_tool *tool,
+pid_t perf_event__synthesize_comm(struct perf_tool *tool,
 					 union perf_event *event, pid_t pid,
 					 perf_event__handler_t process,
 					 struct machine *machine)
diff --git a/tools/perf/util/event.h b/tools/perf/util/event.h
index 1f86c279520e..6df23199fea0 100644
--- a/tools/perf/util/event.h
+++ b/tools/perf/util/event.h
@@ -386,6 +386,11 @@ int perf_event__synthesize_mmap_events(struct perf_tool *tool,
 				       struct machine *machine,
 				       bool mmap_data);
 
+pid_t perf_event__synthesize_comm(struct perf_tool *tool,
+				  union perf_event *event, pid_t pid,
+				  perf_event__handler_t process,
+				  struct machine *machine);
+
 size_t perf_event__fprintf_comm(union perf_event *event, FILE *fp);
 size_t perf_event__fprintf_mmap(union perf_event *event, FILE *fp);
 size_t perf_event__fprintf_mmap2(union perf_event *event, FILE *fp);
-- 
2.2.2


^ permalink raw reply related	[flat|nested] 221+ messages in thread

* [PATCH 35/38] perf tools: Fix progress ui to support multi thread
  2015-03-03  3:07 [RFC/PATCHSET 00/38] perf tools: Speed-up perf report by using multi thread (v3) Namhyung Kim
                   ` (33 preceding siblings ...)
  2015-03-03  3:07 ` [PATCH 34/38] perf record: Synthesize COMM event for a command line workload Namhyung Kim
@ 2015-03-03  3:07 ` Namhyung Kim
  2015-03-03  3:07 ` [PATCH 36/38] perf report: Add --multi-thread option and config item Namhyung Kim
                   ` (2 subsequent siblings)
  37 siblings, 0 replies; 221+ messages in thread
From: Namhyung Kim @ 2015-03-03  3:07 UTC (permalink / raw)
  To: Arnaldo Carvalho de Melo
  Cc: Ingo Molnar, Peter Zijlstra, Jiri Olsa, LKML,
	Frederic Weisbecker, Adrian Hunter, Stephane Eranian, Andi Kleen,
	David Ahern

Split ui_progress struct into global and local one.  Each thread
updates local struct without lock and only updates global one if
meaningful progress is done (with lock).

To do that, pass struct ui_progress to __perf_session__process_event()
and set it for the total size of multi-file storage.

Signed-off-by: Namhyung Kim <namhyung@kernel.org>
---
 tools/perf/util/hist.c    |  5 ++--
 tools/perf/util/hist.h    |  3 +-
 tools/perf/util/session.c | 71 +++++++++++++++++++++++++++++++++++++----------
 tools/perf/util/tool.h    |  3 ++
 4 files changed, 65 insertions(+), 17 deletions(-)

diff --git a/tools/perf/util/hist.c b/tools/perf/util/hist.c
index a6bbe730c4af..05f6a20bc0d4 100644
--- a/tools/perf/util/hist.c
+++ b/tools/perf/util/hist.c
@@ -1057,12 +1057,13 @@ void hists__collapse_resort(struct hists *hists, struct ui_progress *prog)
 	__hists__collapse_resort(hists, root, prog);
 }
 
-void hists__mt_resort(struct hists *dst, struct hists *src)
+void hists__mt_resort(struct hists *dst, struct hists *src,
+		      struct ui_progress *prog)
 {
 	struct rb_root *root = src->entries_in;
 
 	sort__need_collapse = 1;
-	__hists__collapse_resort(dst, root, NULL);
+	__hists__collapse_resort(dst, root, prog);
 }
 
 static int hist_entry__sort(struct hist_entry *a, struct hist_entry *b)
diff --git a/tools/perf/util/hist.h b/tools/perf/util/hist.h
index 3ef30f632948..d201fd86dbc9 100644
--- a/tools/perf/util/hist.h
+++ b/tools/perf/util/hist.h
@@ -123,7 +123,8 @@ int hist_entry__sort_snprintf(struct hist_entry *he, char *bf, size_t size,
 void hist_entry__delete(struct hist_entry *he);
 
 void hists__output_resort(struct hists *hists, struct ui_progress *prog);
-void hists__mt_resort(struct hists *dst, struct hists *src);
+void hists__mt_resort(struct hists *dst, struct hists *src,
+		      struct ui_progress *prog);
 void hists__collapse_resort(struct hists *hists, struct ui_progress *prog);
 
 void hists__decay_entries(struct hists *hists, bool zap_user, bool zap_kernel);
diff --git a/tools/perf/util/session.c b/tools/perf/util/session.c
index 49ded46104dd..a9e3dd309d4d 100644
--- a/tools/perf/util/session.c
+++ b/tools/perf/util/session.c
@@ -1308,14 +1308,14 @@ fetch_mmaped_event(struct perf_session *session,
 static int __perf_session__process_events(struct perf_session *session,
 					  struct events_stats *stats, int fd,
 					  u64 data_offset, u64 data_size,
-					  u64 file_size, struct perf_tool *tool)
+					  u64 file_size, struct perf_tool *tool,
+					  struct ui_progress *prog)
 {
 	u64 head, page_offset, file_offset, file_pos, size;
 	int err, mmap_prot, mmap_flags, map_idx = 0;
 	size_t	mmap_size;
 	char *buf, *mmaps[NUM_MMAPS];
 	union perf_event *event;
-	struct ui_progress prog;
 	s64 skip;
 
 	perf_tool__fill_defaults(tool);
@@ -1327,8 +1327,6 @@ static int __perf_session__process_events(struct perf_session *session,
 	if (data_size && (data_offset + data_size < file_size))
 		file_size = data_offset + data_size;
 
-	ui_progress__init(&prog, file_size, "Processing events...");
-
 	mmap_size = MMAP_SIZE;
 	if (mmap_size > file_size) {
 		mmap_size = file_size;
@@ -1394,7 +1392,7 @@ static int __perf_session__process_events(struct perf_session *session,
 	head += size;
 	file_pos += size;
 
-	ui_progress__update(&prog, size);
+	ui_progress__update(prog, size);
 
 	if (session_done())
 		goto out;
@@ -1406,7 +1404,6 @@ static int __perf_session__process_events(struct perf_session *session,
 	/* do the final flush for ordered samples */
 	err = ordered_events__flush(session, tool, OE_FLUSH__FINAL);
 out_err:
-	ui_progress__finish();
 	ordered_events__free(&session->ordered_events);
 	session->one_mmap = false;
 	return err;
@@ -1415,11 +1412,14 @@ static int __perf_session__process_events(struct perf_session *session,
 static int __perf_session__process_indexed_events(struct perf_session *session,
 						  struct perf_tool *tool)
 {
+	struct ui_progress prog;
 	struct perf_data_file *file = session->file;
 	u64 size = perf_data_file__size(file);
 	struct events_stats *stats = &session->evlist->stats;
 	int err = 0, i;
 
+	ui_progress__init(&prog, size, "Processing events...");
+
 	for (i = 0; i < (int)session->header.nr_index; i++) {
 		struct perf_file_section *index = &session->header.index[i];
 
@@ -1437,18 +1437,21 @@ static int __perf_session__process_indexed_events(struct perf_session *session,
 		err = __perf_session__process_events(session, stats,
 						     perf_data_file__fd(file),
 						     index->offset, index->size,
-						     size, tool);
+						     size, tool, &prog);
 		if (err < 0)
 			break;
 	}
 
+	ui_progress__finish();
 	perf_tool__warn_about_errors(tool, stats);
+
 	return err;
 }
 
 int perf_session__process_events(struct perf_session *session,
 				 struct perf_tool *tool)
 {
+	struct ui_progress prog;
 	struct perf_data_file *file = session->file;
 	u64 size = perf_data_file__size(file);
 	struct events_stats *stats = &session->evlist->stats;
@@ -1462,16 +1465,42 @@ int perf_session__process_events(struct perf_session *session,
 	if (perf_session__has_index(session))
 		return __perf_session__process_indexed_events(session, tool);
 
+	ui_progress__init(&prog, size, "Processing events...");
+
 	err = __perf_session__process_events(session, stats,
 					     perf_data_file__fd(file),
 					     session->header.data_offset,
 					     session->header.data_size,
-					     size, tool);
-
+					     size, tool, &prog);
+	ui_progress__finish();
 	perf_tool__warn_about_errors(tool, stats);
+
 	return err;
 }
 
+struct ui_progress_ops *orig_progress__ops;
+
+static void mt_progress__update(struct ui_progress *p)
+{
+	struct perf_tool_mt *mt_tool = container_of(p, struct perf_tool_mt, prog);
+	struct ui_progress *gprog = mt_tool->global_prog;
+	static pthread_mutex_t prog_lock = PTHREAD_MUTEX_INITIALIZER;
+
+	pthread_mutex_lock(&prog_lock);
+
+	gprog->curr += p->step;
+	if (gprog->curr >= gprog->next) {
+		gprog->next += gprog->step;
+		orig_progress__ops->update(gprog);
+	}
+
+	pthread_mutex_unlock(&prog_lock);
+}
+
+static struct ui_progress_ops mt_progress__ops = {
+	.update = mt_progress__update,
+};
+
 static void *processing_thread_idx(void *arg)
 {
 	struct perf_tool_mt *mt_tool = arg;
@@ -1481,10 +1510,12 @@ static void *processing_thread_idx(void *arg)
 	u64 size = session->header.index[mt_tool->idx].size;
 	u64 file_size = perf_data_file__size(session->file);
 
+	ui_progress__init(&mt_tool->prog, size, "");
+
 	pr_debug("processing samples using thread [%d]\n", mt_tool->idx);
 	if (__perf_session__process_events(session, &mt_tool->stats,
 					   fd, offset, size, file_size,
-					   &mt_tool->tool) < 0) {
+					   &mt_tool->tool, &mt_tool->prog) < 0) {
 		pr_err("processing samples failed (thread [%d])\n", mt_tool->idx);
 		return NULL;
 	}
@@ -1502,7 +1533,8 @@ int perf_session__process_events_mt(struct perf_session *session,
 	u64 nr_entries = 0;
 	struct perf_tool_mt *mt_tools = NULL;
 	struct perf_tool_mt *mt;
-	pthread_t *th_id;
+	struct ui_progress prog;
+	pthread_t *th_id = NULL;
 	int err, i, k;
 	int nr_index = session->header.nr_index;
 	u64 size = perf_data_file__size(file);
@@ -1515,13 +1547,19 @@ int perf_session__process_events_mt(struct perf_session *session,
 	if (perf_session__register_idle_thread(session) == NULL)
 		return -ENOMEM;
 
+	ui_progress__init(&prog, size, "Processing events...");
+
 	err = __perf_session__process_events(session, &evlist->stats,
 					     perf_data_file__fd(file),
 					     session->header.index[0].offset,
 					     session->header.index[0].size,
-					     size, tool);
+					     size, tool, &prog);
 	if (err)
-		return err;
+		goto out;
+
+	orig_progress__ops = ui_progress__ops;
+	ui_progress__ops = &mt_progress__ops;
+	ui_progress__ops->finish = orig_progress__ops->finish;
 
 	th_id = calloc(nr_index, sizeof(*th_id));
 	if (th_id == NULL)
@@ -1547,6 +1585,7 @@ int perf_session__process_events_mt(struct perf_session *session,
 		mt->tool.ordered_events = false;
 		mt->idx = i;
 		mt->priv = arg;
+		mt->global_prog = &prog;
 
 		pthread_create(&th_id[i], NULL, processing_thread_idx, mt);
 	}
@@ -1570,6 +1609,9 @@ int perf_session__process_events_mt(struct perf_session *session,
 		}
 	}
 
+	ui_progress__ops = orig_progress__ops;
+	ui_progress__init(&prog, nr_entries, "Merging related events...");
+
 	for (i = 1; i < nr_index; i++) {
 		mt = &mt_tools[i];
 
@@ -1579,7 +1621,7 @@ int perf_session__process_events_mt(struct perf_session *session,
 			if (perf_evsel__is_dummy_tracking(evsel))
 				continue;
 
-			hists__mt_resort(hists, &mt->hists[evsel->idx]);
+			hists__mt_resort(hists, &mt->hists[evsel->idx], &prog);
 
 			/* Non-group events are considered as leader */
 			if (symbol_conf.event_group &&
@@ -1594,6 +1636,7 @@ int perf_session__process_events_mt(struct perf_session *session,
 	}
 
 out:
+	ui_progress__finish();
 	perf_tool__warn_about_errors(tool, &evlist->stats);
 
 	if (mt_tools) {
diff --git a/tools/perf/util/tool.h b/tools/perf/util/tool.h
index a04826bbe991..aa7f110b9425 100644
--- a/tools/perf/util/tool.h
+++ b/tools/perf/util/tool.h
@@ -3,6 +3,7 @@
 
 #include <stdbool.h>
 #include "util/event.h"
+#include "ui/progress.h"
 
 struct perf_session;
 union perf_event;
@@ -52,6 +53,8 @@ struct perf_tool_mt {
 	struct events_stats	stats;
 	struct hists		*hists;
 	struct perf_session	*session;
+	struct ui_progress	prog;
+	struct ui_progress	*global_prog;
 	int			idx;
 
 	void			*priv;
-- 
2.2.2


^ permalink raw reply related	[flat|nested] 221+ messages in thread

* [PATCH 36/38] perf report: Add --multi-thread option and config item
  2015-03-03  3:07 [RFC/PATCHSET 00/38] perf tools: Speed-up perf report by using multi thread (v3) Namhyung Kim
                   ` (34 preceding siblings ...)
  2015-03-03  3:07 ` [PATCH 35/38] perf tools: Fix progress ui to support multi thread Namhyung Kim
@ 2015-03-03  3:07 ` Namhyung Kim
  2015-03-03  3:07 ` [PATCH 37/38] perf session: Handle index files generally Namhyung Kim
  2015-03-03  3:07 ` [PATCH 38/38] perf data: Implement 'index' subcommand Namhyung Kim
  37 siblings, 0 replies; 221+ messages in thread
From: Namhyung Kim @ 2015-03-03  3:07 UTC (permalink / raw)
  To: Arnaldo Carvalho de Melo
  Cc: Ingo Molnar, Peter Zijlstra, Jiri Olsa, LKML,
	Frederic Weisbecker, Adrian Hunter, Stephane Eranian, Andi Kleen,
	David Ahern

The --multi-thread option is to enable parallel processing so user can
force serial processing even for indexed data file.  It default to false
for now but users also can changes this by setting "report.multi_thread"
config option in ~/.perfconfig file.

Signed-off-by: Namhyung Kim <namhyung@kernel.org>
---
 tools/perf/Documentation/perf-report.txt |  3 ++
 tools/perf/builtin-report.c              | 66 +++++++++++++++++++++++++++-----
 2 files changed, 60 insertions(+), 9 deletions(-)

diff --git a/tools/perf/Documentation/perf-report.txt b/tools/perf/Documentation/perf-report.txt
index dd7cccdde498..e00077a658c1 100644
--- a/tools/perf/Documentation/perf-report.txt
+++ b/tools/perf/Documentation/perf-report.txt
@@ -318,6 +318,9 @@ OPTIONS
 --header-only::
 	Show only perf.data header (forces --stdio).
 
+--multi-thread::
+	Speed up report by parallelizing sample processing using multi-thread.
+
 SEE ALSO
 --------
 linkperf:perf-stat[1], linkperf:perf-annotate[1]
diff --git a/tools/perf/builtin-report.c b/tools/perf/builtin-report.c
index 5adf269b84a9..4db4cfbb8c75 100644
--- a/tools/perf/builtin-report.c
+++ b/tools/perf/builtin-report.c
@@ -51,6 +51,7 @@ struct report {
 	bool			mem_mode;
 	bool			header;
 	bool			header_only;
+	bool			multi_thread;
 	int			max_stack;
 	struct perf_read_values	show_threads_values;
 	const char		*pretty_printing_style;
@@ -82,6 +83,10 @@ static int report__config(const char *var, const char *value, void *cb)
 		rep->queue_size = perf_config_u64(var, value);
 		return 0;
 	}
+	if (!strcmp(var, "report.multi-thread")) {
+		rep->multi_thread = perf_config_bool(var, value);
+		return 0;
+	}
 
 	return perf_default_config(var, value, cb);
 }
@@ -128,17 +133,18 @@ static int hist_iter__report_callback(struct hist_entry_iter *iter,
 	return err;
 }
 
-static int process_sample_event(struct perf_tool *tool,
-				union perf_event *event,
-				struct perf_sample *sample,
-				struct perf_evsel *evsel,
-				struct machine *machine)
+static int __process_sample_event(struct perf_tool *tool __maybe_unused,
+				  union perf_event *event,
+				  struct perf_sample *sample,
+				  struct perf_evsel *evsel,
+				  struct machine *machine,
+				  struct hists *hists,
+				  struct report *rep)
 {
-	struct report *rep = container_of(tool, struct report, tool);
 	struct addr_location al;
 	struct hist_entry_iter iter = {
 		.evsel 			= evsel,
-		.hists 			= evsel__hists(evsel),
+		.hists 			= hists,
 		.sample 		= sample,
 		.session 		= rep->session,
 		.hide_unresolved 	= rep->hide_unresolved,
@@ -178,6 +184,31 @@ static int process_sample_event(struct perf_tool *tool,
 	return ret;
 }
 
+static int process_sample_event(struct perf_tool *tool,
+				union perf_event *event,
+				struct perf_sample *sample,
+				struct perf_evsel *evsel,
+				struct machine *machine)
+{
+	struct report *rep = container_of(tool, struct report, tool);
+
+	return __process_sample_event(tool, event, sample, evsel, machine,
+				      evsel__hists(evsel), rep);
+}
+
+static int process_sample_event_mt(struct perf_tool *tool,
+				   union perf_event *event,
+				   struct perf_sample *sample,
+				   struct perf_evsel *evsel,
+				   struct machine *machine)
+{
+	struct perf_tool_mt *mt = container_of(tool, struct perf_tool_mt, tool);
+	struct report *rep = mt->priv;
+
+	return __process_sample_event(tool, event, sample, evsel, machine,
+				      &mt->hists[evsel->idx], rep);
+}
+
 static int process_read_event(struct perf_tool *tool,
 			      union perf_event *event,
 			      struct perf_sample *sample __maybe_unused,
@@ -489,7 +520,12 @@ static int __cmd_report(struct report *rep)
 	if (ret)
 		return ret;
 
-	ret = perf_session__process_events(session, &rep->tool);
+	if (rep->multi_thread) {
+		rep->tool.sample = process_sample_event_mt;
+		ret = perf_session__process_events_mt(session, &rep->tool, rep);
+	} else {
+		ret = perf_session__process_events(session, &rep->tool);
+	}
 	if (ret)
 		return ret;
 
@@ -512,7 +548,12 @@ static int __cmd_report(struct report *rep)
 		}
 	}
 
-	report__collapse_hists(rep);
+	/*
+	 * For multi-thread report, it already calls hists__mt_resort()
+	 * so no need to collapse here.
+	 */
+	if (!rep->multi_thread)
+		report__collapse_hists(rep);
 
 	if (session_done())
 		return 0;
@@ -720,6 +761,8 @@ int cmd_report(int argc, const char **argv, const char *prefix __maybe_unused)
 		     "Don't show entries under that percent", parse_percent_limit),
 	OPT_CALLBACK(0, "percentage", NULL, "relative|absolute",
 		     "how to display percentage of filtered entries", parse_filter_percentage),
+	OPT_BOOLEAN(0, "multi-thread", &report.multi_thread,
+		    "Speed up sample processing using multi-thead"),
 	OPT_END()
 	};
 	struct perf_data_file file = {
@@ -764,6 +807,11 @@ int cmd_report(int argc, const char **argv, const char *prefix __maybe_unused)
 					       report.queue_size);
 	}
 
+	if (report.multi_thread && !perf_session__has_index(session)) {
+		pr_debug("fallback to single thread for normal data file.\n");
+		report.multi_thread = false;
+	}
+
 	report.session = session;
 
 	has_br_stack = perf_header__has_feat(&session->header,
-- 
2.2.2


^ permalink raw reply related	[flat|nested] 221+ messages in thread

* [PATCH 37/38] perf session: Handle index files generally
  2015-03-03  3:07 [RFC/PATCHSET 00/38] perf tools: Speed-up perf report by using multi thread (v3) Namhyung Kim
                   ` (35 preceding siblings ...)
  2015-03-03  3:07 ` [PATCH 36/38] perf report: Add --multi-thread option and config item Namhyung Kim
@ 2015-03-03  3:07 ` Namhyung Kim
  2015-03-03  3:07 ` [PATCH 38/38] perf data: Implement 'index' subcommand Namhyung Kim
  37 siblings, 0 replies; 221+ messages in thread
From: Namhyung Kim @ 2015-03-03  3:07 UTC (permalink / raw)
  To: Arnaldo Carvalho de Melo
  Cc: Ingo Molnar, Peter Zijlstra, Jiri Olsa, LKML,
	Frederic Weisbecker, Adrian Hunter, Stephane Eranian, Andi Kleen,
	David Ahern

The current code assumes that the number of index item and cpu are
matched so it creates that number of threads.  But it's not the case
of non-system-wide session or data came from different machine.

Just creates threads at most number of online cpus and process data.

Signed-off-by: Namhyung Kim <namhyung@kernel.org>
---
 tools/perf/util/session.c | 76 +++++++++++++++++++++++++++++++++--------------
 tools/perf/util/tool.h    |  1 -
 2 files changed, 54 insertions(+), 23 deletions(-)

diff --git a/tools/perf/util/session.c b/tools/perf/util/session.c
index a9e3dd309d4d..3c999bae8e52 100644
--- a/tools/perf/util/session.c
+++ b/tools/perf/util/session.c
@@ -1501,26 +1501,51 @@ static struct ui_progress_ops mt_progress__ops = {
 	.update = mt_progress__update,
 };
 
+static int perf_session__get_index(struct perf_session *session)
+{
+	int ret;
+	static unsigned index = 1;
+	static pthread_mutex_t idx_lock = PTHREAD_MUTEX_INITIALIZER;
+
+	pthread_mutex_lock(&idx_lock);
+	if (index < session->header.nr_index)
+		ret = index++;
+	else
+		ret = -1;
+	pthread_mutex_unlock(&idx_lock);
+
+	return ret;
+}
+
 static void *processing_thread_idx(void *arg)
 {
 	struct perf_tool_mt *mt_tool = arg;
 	struct perf_session *session = mt_tool->session;
 	int fd = perf_data_file__fd(session->file);
-	u64 offset = session->header.index[mt_tool->idx].offset;
-	u64 size = session->header.index[mt_tool->idx].size;
 	u64 file_size = perf_data_file__size(session->file);
+	int idx;
 
-	ui_progress__init(&mt_tool->prog, size, "");
+	while ((idx = perf_session__get_index(session)) >= 0) {
+		u64 offset = session->header.index[idx].offset;
+		u64 size = session->header.index[idx].size;
+		struct perf_tool_mt *mtt = &mt_tool[idx];
 
-	pr_debug("processing samples using thread [%d]\n", mt_tool->idx);
-	if (__perf_session__process_events(session, &mt_tool->stats,
-					   fd, offset, size, file_size,
-					   &mt_tool->tool, &mt_tool->prog) < 0) {
-		pr_err("processing samples failed (thread [%d])\n", mt_tool->idx);
-		return NULL;
+		if (size == 0)
+			continue;
+
+		pr_debug("processing samples [index %d]\n", idx);
+
+		ui_progress__init(&mtt->prog, size, "");
+
+		if (__perf_session__process_events(session, &mtt->stats,
+						   fd, offset, size, file_size,
+						   &mtt->tool, &mtt->prog) < 0) {
+			pr_err("processing samples failed [index %d]\n", idx);
+			return NULL;
+		}
+		pr_debug("processing samples done [index %d]\n", idx);
 	}
 
-	pr_debug("processing samples done for thread [%d]\n", mt_tool->idx);
 	return arg;
 }
 
@@ -1538,6 +1563,7 @@ int perf_session__process_events_mt(struct perf_session *session,
 	int err, i, k;
 	int nr_index = session->header.nr_index;
 	u64 size = perf_data_file__size(file);
+	int nr_thread = sysconf(_SC_NPROCESSORS_ONLN);
 
 	if (perf_data_file__is_pipe(file) || !session->header.index) {
 		pr_err("data file doesn't contain the index table\n");
@@ -1561,15 +1587,11 @@ int perf_session__process_events_mt(struct perf_session *session,
 	ui_progress__ops = &mt_progress__ops;
 	ui_progress__ops->finish = orig_progress__ops->finish;
 
-	th_id = calloc(nr_index, sizeof(*th_id));
-	if (th_id == NULL)
-		goto out;
-
 	mt_tools = calloc(nr_index, sizeof(*mt_tools));
 	if (mt_tools == NULL)
 		goto out;
 
-	for (i = 1; i < nr_index; i++) {
+	for (i = 0; i < nr_index; i++) {
 		mt = &mt_tools[i];
 
 		memcpy(&mt->tool, tool, sizeof(*tool));
@@ -1583,20 +1605,30 @@ int perf_session__process_events_mt(struct perf_session *session,
 
 		mt->session = session;
 		mt->tool.ordered_events = false;
-		mt->idx = i;
 		mt->priv = arg;
 		mt->global_prog = &prog;
-
-		pthread_create(&th_id[i], NULL, processing_thread_idx, mt);
 	}
 
-	for (i = 1; i < nr_index; i++) {
+	if (nr_thread > nr_index - 1)
+		nr_thread = nr_index - 1;
+
+	th_id = calloc(nr_thread, sizeof(*th_id));
+	if (th_id == NULL)
+		goto out;
+
+	for (i = 0; i < nr_thread; i++)
+		pthread_create(&th_id[i], NULL, processing_thread_idx, mt_tools);
+
+	for (i = 0; i < nr_thread; i++) {
 		pthread_join(th_id[i], (void **)&mt);
 		if (mt == NULL) {
 			err = -EINVAL;
 			continue;
 		}
+	}
 
+	for (i = 0; i < nr_index; i++) {
+		mt = &mt_tools[i];
 		events_stats__add(&evlist->stats, &mt->stats);
 
 		evlist__for_each(evlist, evsel) {
@@ -1612,7 +1644,7 @@ int perf_session__process_events_mt(struct perf_session *session,
 	ui_progress__ops = orig_progress__ops;
 	ui_progress__init(&prog, nr_entries, "Merging related events...");
 
-	for (i = 1; i < nr_index; i++) {
+	for (i = 0; i < nr_index; i++) {
 		mt = &mt_tools[i];
 
 		evlist__for_each(evlist, evsel) {
@@ -1640,7 +1672,7 @@ int perf_session__process_events_mt(struct perf_session *session,
 	perf_tool__warn_about_errors(tool, &evlist->stats);
 
 	if (mt_tools) {
-		for (i = 1; i < nr_index; i++)
+		for (i = 0; i < nr_index; i++)
 			free(mt_tools[i].hists);
 		free(mt_tools);
 	}
@@ -1649,7 +1681,7 @@ int perf_session__process_events_mt(struct perf_session *session,
 	return err;
 
 err:
-	while (i-- > 1) {
+	while (i-- > 0) {
 		pthread_cancel(th_id[i]);
 		pthread_join(th_id[i], NULL);
 	}
diff --git a/tools/perf/util/tool.h b/tools/perf/util/tool.h
index aa7f110b9425..e52c936d1b9e 100644
--- a/tools/perf/util/tool.h
+++ b/tools/perf/util/tool.h
@@ -55,7 +55,6 @@ struct perf_tool_mt {
 	struct perf_session	*session;
 	struct ui_progress	prog;
 	struct ui_progress	*global_prog;
-	int			idx;
 
 	void			*priv;
 };
-- 
2.2.2


^ permalink raw reply related	[flat|nested] 221+ messages in thread

* [PATCH 38/38] perf data: Implement 'index' subcommand
  2015-03-03  3:07 [RFC/PATCHSET 00/38] perf tools: Speed-up perf report by using multi thread (v3) Namhyung Kim
                   ` (36 preceding siblings ...)
  2015-03-03  3:07 ` [PATCH 37/38] perf session: Handle index files generally Namhyung Kim
@ 2015-03-03  3:07 ` Namhyung Kim
  37 siblings, 0 replies; 221+ messages in thread
From: Namhyung Kim @ 2015-03-03  3:07 UTC (permalink / raw)
  To: Arnaldo Carvalho de Melo
  Cc: Ingo Molnar, Peter Zijlstra, Jiri Olsa, LKML,
	Frederic Weisbecker, Adrian Hunter, Stephane Eranian, Andi Kleen,
	David Ahern

The index command first splits a given data file into intermediate
data files and merges them into a final data file with an index table
so that it can processed using multi threads.  The HEADER_DATA_INDEX
feature bit is added to distinguish data file that has an index table.

Signed-off-by: Namhyung Kim <namhyung@kernel.org>
---
 tools/perf/Documentation/perf-data.txt |  25 ++-
 tools/perf/builtin-data.c              | 349 ++++++++++++++++++++++++++++++++-
 2 files changed, 370 insertions(+), 4 deletions(-)

diff --git a/tools/perf/Documentation/perf-data.txt b/tools/perf/Documentation/perf-data.txt
index be8fa1a0a97e..fdac46ea6732 100644
--- a/tools/perf/Documentation/perf-data.txt
+++ b/tools/perf/Documentation/perf-data.txt
@@ -22,6 +22,11 @@ COMMANDS
 	like:
 	  perf --debug data-convert data convert ...
 
+index::
+	Build an index table for data file so that it can be processed
+	with multiple threads concurrently.
+
+
 OPTIONS for 'convert'
 ---------------------
 --to-ctf::
@@ -34,7 +39,25 @@ OPTIONS for 'convert'
 --verbose::
         Be more verbose (show counter open errors, etc).
 
+OPTIONS for 'index'
+-------------------
+-i::
+--input::
+	Specify input perf data file path.
+
+-o::
+--output::
+	Specify output perf data directory path.
+
+-v::
+--verbose::
+        Be more verbose (show counter open errors, etc).
+
+-f::
+--force::
+        Don't complain, do it.
+
 SEE ALSO
 --------
-linkperf:perf[1]
+linkperf:perf[1], linkperf:perf-report[1]
 [1] Common Trace Format - http://www.efficios.com/ctf
diff --git a/tools/perf/builtin-data.c b/tools/perf/builtin-data.c
index 155cf75b8199..8df7321434c0 100644
--- a/tools/perf/builtin-data.c
+++ b/tools/perf/builtin-data.c
@@ -2,11 +2,16 @@
 #include "builtin.h"
 #include "perf.h"
 #include "debug.h"
+#include "session.h"
+#include "evlist.h"
 #include "parse-options.h"
 #include "data-convert-bt.h"
+#include <sys/mman.h>
 
 typedef int (*data_cmd_fn_t)(int argc, const char **argv, const char *prefix);
 
+static const char *output_name;
+
 struct data_cmd {
 	const char	*name;
 	const char	*summary;
@@ -42,6 +47,15 @@ static void print_usage(void)
 	printf("\n");
 }
 
+static int cmd_data_convert(int argc, const char **argv, const char *prefix);
+static int data_cmd_index(int argc, const char **argv, const char *prefix);
+
+static struct data_cmd data_cmds[] = {
+	{ "convert", "converts data file between formats", cmd_data_convert },
+	{ "index", "merge data file and add index", data_cmd_index },
+	{ .name = NULL, },
+};
+
 static const char * const data_convert_usage[] = {
 	"perf data convert [<options>]",
 	NULL
@@ -84,11 +98,340 @@ static int cmd_data_convert(int argc, const char **argv,
 	return 0;
 }
 
-static struct data_cmd data_cmds[] = {
-	{ "convert", "converts data file between formats", cmd_data_convert },
-	{ .name = NULL, },
+#define FD_HASH_BITS  7
+#define FD_HASH_SIZE  (1 << FD_HASH_BITS)
+#define FD_HASH_MASK  (FD_HASH_SIZE - 1)
+
+struct data_index {
+	struct perf_tool	tool;
+	struct perf_session	*session;
+	enum {
+		PER_CPU,
+		PER_THREAD,
+	} split_mode;
+	char			*tmpdir;
+	int 			header_fd;
+	u64			header_written;
+	struct hlist_head	fd_hash[FD_HASH_SIZE];
+	int			fd_hash_nr;
+	int			output_fd;
 };
 
+struct fdhash_node {
+	int			id;
+	int			fd;
+	struct hlist_node	list;
+};
+
+static struct hlist_head *get_hash(struct data_index *index, int id)
+{
+	return &index->fd_hash[id % FD_HASH_MASK];
+}
+
+static int perf_event__rewrite_header(struct perf_tool *tool,
+				      union perf_event *event)
+{
+	struct data_index *index = container_of(tool, struct data_index, tool);
+	ssize_t size;
+
+	size = writen(index->header_fd, event, event->header.size);
+	if (size < 0)
+		return -errno;
+
+	index->header_written += size;
+	return 0;
+}
+
+static int split_other_events(struct perf_tool *tool,
+				union perf_event *event,
+				struct perf_sample *sample __maybe_unused,
+				struct machine *machine __maybe_unused)
+{
+	return perf_event__rewrite_header(tool, event);
+}
+
+static int split_sample_event(struct perf_tool *tool,
+				union perf_event *event,
+				struct perf_sample *sample,
+				struct perf_evsel *evsel __maybe_unused,
+				struct machine *machine __maybe_unused)
+{
+	struct data_index *index = container_of(tool, struct data_index, tool);
+	int id = index->split_mode == PER_CPU ? sample->cpu : sample->tid;
+	int fd = -1;
+	char buf[PATH_MAX];
+	struct hlist_head *head;
+	struct fdhash_node *node;
+
+	head = get_hash(index, id);
+	hlist_for_each_entry(node, head, list) {
+		if (node->id == id) {
+			fd = node->fd;
+			break;
+		}
+	}
+
+	if (fd == -1) {
+		scnprintf(buf, sizeof(buf), "%s/perf.data.%d",
+			  index->tmpdir, index->fd_hash_nr++);
+
+		fd = open(buf, O_RDWR|O_CREAT|O_TRUNC, 0600);
+		if (fd < 0) {
+			pr_err("cannot open data file: %s: %m\n", buf);
+			return -1;
+		}
+
+		node = malloc(sizeof(*node));
+		if (node == NULL) {
+			pr_err("memory allocation failed\n");
+			return -1;
+		}
+
+		node->id = id;
+		node->fd = fd;
+
+		hlist_add_head(&node->list, head);
+	}
+
+	return writen(fd, event, event->header.size) > 0 ? 0 : -errno;
+}
+
+static int split_data_file(struct data_index *index)
+{
+	struct perf_session *session = index->session;
+	char buf[PATH_MAX];
+	u64 sample_type;
+	int header_fd;
+
+	if (asprintf(&index->tmpdir, "%s.dir", output_name) < 0) {
+		pr_err("memory allocation failed\n");
+		return -1;
+	}
+
+	if (mkdir(index->tmpdir, 0700) < 0) {
+		pr_err("cannot create intermediate directory\n");
+		return -1;
+	}
+
+	/*
+	 * This is necessary to write (copy) build-id table.  After
+	 * processing header, dsos list will only contain dso which
+	 * was on the original build-id table.
+	 */
+	dsos__hit_all(session);
+
+	scnprintf(buf, sizeof(buf), "%s/perf.header", index->tmpdir);
+	header_fd = open(buf, O_RDWR|O_CREAT|O_TRUNC, 0600);
+	if (header_fd < 0) {
+		pr_err("cannot open header file: %s: %m\n", buf);
+		return -1;
+	}
+
+	lseek(header_fd, session->header.data_offset, SEEK_SET);
+
+	sample_type = perf_evlist__combined_sample_type(session->evlist);
+	if (sample_type & PERF_SAMPLE_CPU)
+		index->split_mode = PER_CPU;
+	else
+		index->split_mode = PER_THREAD;
+
+	pr_debug("splitting data file for %s\n",
+		 index->split_mode == PER_CPU ? "CPUs" : "threads");
+
+	index->header_fd = header_fd;
+	if (perf_session__process_events(session, &index->tool) < 0) {
+		pr_err("failed to process events\n");
+		return -1;
+	}
+
+	return 0;
+}
+
+static int build_index_table(struct data_index *index)
+{
+	int i, n;
+	u64 offset;
+	u64 nr_index = index->fd_hash_nr + 1;
+	struct perf_file_section *idx;
+	struct perf_session *session = index->session;
+
+	idx = calloc(nr_index, sizeof(*idx));
+	if (idx == NULL)
+		return -1;
+
+	idx[0].offset = session->header.data_offset;
+	idx[0].size   = index->header_written;
+
+	offset = idx[0].offset + idx[0].size;
+
+	for (i = 0, n = 1; i < FD_HASH_SIZE; i++) {
+		struct fdhash_node *node;
+
+		hlist_for_each_entry(node, &index->fd_hash[i], list) {
+			struct stat stbuf;
+
+			if (fstat(node->fd, &stbuf) < 0)
+				goto out;
+
+			idx[n].offset = offset;
+			idx[n].size   = stbuf.st_size;
+			n++;
+
+			offset += stbuf.st_size;
+		}
+	}
+
+	BUG_ON(n != (int)nr_index);
+
+	session->header.index = idx;
+	session->header.nr_index = nr_index;
+	perf_header__set_feat(&session->header, HEADER_DATA_INDEX);
+
+	perf_session__write_header(session, session->evlist,
+				   index->output_fd, true);
+	return 0;
+
+out:
+	free(idx);
+	return -1;
+}
+
+static int cleanup_temp_files(struct data_index *index)
+{
+	int i;
+
+	for (i = 0; i < FD_HASH_SIZE; i++) {
+		struct fdhash_node *pos;
+		struct hlist_node *tmp;
+
+		hlist_for_each_entry_safe(pos, tmp, &index->fd_hash[i], list) {
+			hlist_del(&pos->list);
+			close(pos->fd);
+			free(pos);
+		}
+	}
+	close(index->header_fd);
+
+	rm_rf(index->tmpdir);
+	zfree(&index->tmpdir);
+	return 0;
+}
+
+static int __data_cmd_index(struct data_index *index)
+{
+	struct perf_session *session = index->session;
+	char *output = NULL;
+	int ret = -1;
+	int i, n;
+
+	if (!output_name) {
+		if (asprintf(&output, "%s.out", session->file->path) < 0) {
+			pr_err("memory allocation failed\n");
+			return -1;
+		}
+
+		output_name = output;
+	}
+
+	index->output_fd = open(output_name, O_RDWR|O_CREAT|O_TRUNC, 0600);
+	if (index->output_fd < 0) {
+		pr_err("cannot create output file: %s\n", output_name);
+		goto out;
+	}
+
+	/*
+	 * This is necessary to write (copy) build-id table.  After
+	 * processing header, dsos list will contain dso which was on
+	 * the original build-id table.
+	 */
+	dsos__hit_all(session);
+
+	if (split_data_file(index) < 0)
+		goto out_clean;
+
+	if (build_index_table(index) < 0)
+		goto out_clean;
+
+	/* copy meta-events */
+	if (copyfile_offset(index->header_fd, session->header.data_offset,
+			   index->output_fd, session->header.data_offset,
+			   index->header_written) < 0)
+		goto out_clean;
+
+	/* copy sample events */
+	for (i = 0, n = 1; i < FD_HASH_SIZE; i++) {
+		struct fdhash_node *node;
+
+		hlist_for_each_entry(node, &index->fd_hash[i], list) {
+			if (copyfile_offset(node->fd, 0, index->output_fd,
+					    session->header.index[n].offset,
+					    session->header.index[n].size) < 0)
+				goto out_clean;
+			n++;
+		}
+	}
+	ret = 0;
+
+out_clean:
+	cleanup_temp_files(index);
+	close(index->output_fd);
+out:
+	free(output);
+	return ret;
+}
+
+int data_cmd_index(int argc, const char **argv, const char *prefix __maybe_unused)
+{
+	bool force = false;
+	struct perf_session *session;
+	struct perf_data_file file = {
+		.mode  = PERF_DATA_MODE_READ,
+	};
+	struct data_index index = {
+		.tool = {
+			.sample		= split_sample_event,
+			.fork		= split_other_events,
+			.comm		= split_other_events,
+			.exit		= split_other_events,
+			.mmap		= split_other_events,
+			.mmap2		= split_other_events,
+			.lost		= split_other_events,
+			.throttle	= split_other_events,
+			.unthrottle	= split_other_events,
+			.ordered_events = false,
+		},
+	};
+	const char * const index_usage[] = {
+		"perf data index [<options>]",
+		NULL
+	};
+	const struct option index_options[] = {
+	OPT_STRING('i', "input", &input_name, "file", "input file name"),
+	OPT_STRING('o', "output", &output_name, "file", "output directory name"),
+	OPT_BOOLEAN('f', "force", &force, "don't complain, do it"),
+	OPT_INCR('v', "verbose", &verbose, "be more verbose"),
+	OPT_END()
+	};
+
+	argc = parse_options(argc, argv, index_options, index_usage, 0);
+	if (argc)
+		usage_with_options(index_usage, index_options);
+
+	file.path = input_name;
+	file.force = force;
+	session = perf_session__new(&file, false, &index.tool);
+	if (session == NULL)
+		return -1;
+
+	index.session = session;
+	symbol__init(&session->header.env);
+
+	__data_cmd_index(&index);
+
+	perf_session__delete(session);
+	return 0;
+}
+
 int cmd_data(int argc, const char **argv, const char *prefix)
 {
 	struct data_cmd *cmd;
-- 
2.2.2


^ permalink raw reply related	[flat|nested] 221+ messages in thread

* Re: [PATCH 10/38] perf tools: Pass session arg to perf_event__preprocess_sample()
  2015-03-03  3:07 ` [PATCH 10/38] perf tools: Pass session arg to perf_event__preprocess_sample() Namhyung Kim
@ 2015-03-03 13:59   ` Arnaldo Carvalho de Melo
  2015-03-03 14:18     ` Namhyung Kim
  0 siblings, 1 reply; 221+ messages in thread
From: Arnaldo Carvalho de Melo @ 2015-03-03 13:59 UTC (permalink / raw)
  To: Namhyung Kim
  Cc: Ingo Molnar, Peter Zijlstra, Jiri Olsa, LKML,
	Frederic Weisbecker, Adrian Hunter, Stephane Eranian, Andi Kleen,
	David Ahern

Em Tue, Mar 03, 2015 at 12:07:22PM +0900, Namhyung Kim escreveu:
> The perf_event__preprocess_sample() translates a given ip into a
> matching symbol.  To do that, it first finds a corresponding thread
> and map in the current thread tree.  But for indexed data files, it
> needs to find a thread (and map) with slightly different APIs using
> timestamp.  So it needs a way to know whether this session deals with
> an indexed data file or not.

I need to look into this, but in general, I'm trying to _remove_
accesses to perf_session :-)

- Arnaldo

^ permalink raw reply	[flat|nested] 221+ messages in thread

* Re: [PATCH 22/38] perf callchain: Use session__find_addr_location() and friends
  2015-03-03  3:07 ` [PATCH 22/38] perf callchain: Use " Namhyung Kim
@ 2015-03-03 14:01   ` Arnaldo Carvalho de Melo
  0 siblings, 0 replies; 221+ messages in thread
From: Arnaldo Carvalho de Melo @ 2015-03-03 14:01 UTC (permalink / raw)
  To: Namhyung Kim
  Cc: Ingo Molnar, Peter Zijlstra, Jiri Olsa, LKML,
	Frederic Weisbecker, Adrian Hunter, Stephane Eranian, Andi Kleen,
	David Ahern

Em Tue, Mar 03, 2015 at 12:07:34PM +0900, Namhyung Kim escreveu:
> Pass session struct to callchain resolve routines and find correct
> thread/map/symbol using proper functions.

For instance, in this case it looks like a case to pass a 'struct
machines' and not a perf_session one.

- Arnaldo
 
> Cc: Frederic Weisbecker <fweisbec@gmail.com>
> Signed-off-by: Namhyung Kim <namhyung@kernel.org>
> ---
>  tools/perf/builtin-script.c                        |  4 +--
>  tools/perf/tests/dwarf-unwind.c                    |  2 +-
>  tools/perf/util/callchain.c                        |  6 ++--
>  tools/perf/util/callchain.h                        |  4 ++-
>  tools/perf/util/hist.c                             |  5 +--
>  tools/perf/util/machine.c                          | 41 +++++++++++++---------
>  tools/perf/util/machine.h                          |  1 +
>  .../util/scripting-engines/trace-event-python.c    | 27 +++++++-------
>  tools/perf/util/session.c                          |  6 ++--
>  tools/perf/util/session.h                          |  2 +-
>  tools/perf/util/unwind-libdw.c                     | 14 ++++----
>  tools/perf/util/unwind-libdw.h                     |  1 +
>  tools/perf/util/unwind-libunwind.c                 | 32 +++++++++--------
>  tools/perf/util/unwind.h                           |  3 +-
>  14 files changed, 86 insertions(+), 62 deletions(-)
> 
> diff --git a/tools/perf/builtin-script.c b/tools/perf/builtin-script.c
> index 65b3a07be2bf..90a401a52868 100644
> --- a/tools/perf/builtin-script.c
> +++ b/tools/perf/builtin-script.c
> @@ -429,7 +429,7 @@ static void print_sample_bts(union perf_event *event,
>  				print_opts &= ~PRINT_IP_OPT_SRCLINE;
>  			}
>  		}
> -		perf_evsel__print_ip(evsel, sample, al, print_opts,
> +		perf_evsel__print_ip(evsel, sample, session, al, print_opts,
>  				     PERF_MAX_STACK_DEPTH);
>  	}
>  
> @@ -483,7 +483,7 @@ static void process_event(union perf_event *event, struct perf_sample *sample,
>  		else
>  			printf("\n");
>  
> -		perf_evsel__print_ip(evsel, sample, al,
> +		perf_evsel__print_ip(evsel, sample, session, al,
>  				     output[attr->type].print_ip_opts,
>  				     PERF_MAX_STACK_DEPTH);
>  	}
> diff --git a/tools/perf/tests/dwarf-unwind.c b/tools/perf/tests/dwarf-unwind.c
> index 7e04feb431cb..241270374e93 100644
> --- a/tools/perf/tests/dwarf-unwind.c
> +++ b/tools/perf/tests/dwarf-unwind.c
> @@ -75,7 +75,7 @@ static int unwind_thread(struct thread *thread)
>  		goto out;
>  	}
>  
> -	err = unwind__get_entries(unwind_entry, &cnt, thread,
> +	err = unwind__get_entries(unwind_entry, &cnt, thread, NULL,
>  				  &sample, MAX_STACK);
>  	if (err)
>  		pr_debug("unwind failed\n");
> diff --git a/tools/perf/util/callchain.c b/tools/perf/util/callchain.c
> index 9f643ee77001..f95b27037dc8 100644
> --- a/tools/perf/util/callchain.c
> +++ b/tools/perf/util/callchain.c
> @@ -757,7 +757,9 @@ int callchain_cursor_append(struct callchain_cursor *cursor,
>  	return 0;
>  }
>  
> -int sample__resolve_callchain(struct perf_sample *sample, struct symbol **parent,
> +int sample__resolve_callchain(struct perf_sample *sample,
> +			      struct perf_session *session,
> +			      struct symbol **parent,
>  			      struct perf_evsel *evsel, struct addr_location *al,
>  			      int max_stack)
>  {
> @@ -767,7 +769,7 @@ int sample__resolve_callchain(struct perf_sample *sample, struct symbol **parent
>  	if (symbol_conf.use_callchain || symbol_conf.cumulate_callchain ||
>  	    sort__has_parent) {
>  		return thread__resolve_callchain(al->thread, evsel, sample,
> -						 parent, al, max_stack);
> +						 session, parent, al, max_stack);
>  	}
>  	return 0;
>  }
> diff --git a/tools/perf/util/callchain.h b/tools/perf/util/callchain.h
> index 6033a0a212ca..ca9048f84cb5 100644
> --- a/tools/perf/util/callchain.h
> +++ b/tools/perf/util/callchain.h
> @@ -165,7 +165,9 @@ struct hist_entry;
>  int record_parse_callchain_opt(const struct option *opt, const char *arg, int unset);
>  int record_callchain_opt(const struct option *opt, const char *arg, int unset);
>  
> -int sample__resolve_callchain(struct perf_sample *sample, struct symbol **parent,
> +int sample__resolve_callchain(struct perf_sample *sample,
> +			      struct perf_session *session,
> +			      struct symbol **parent,
>  			      struct perf_evsel *evsel, struct addr_location *al,
>  			      int max_stack);
>  int hist_entry__append_callchain(struct hist_entry *he, struct perf_sample *sample);
> diff --git a/tools/perf/util/hist.c b/tools/perf/util/hist.c
> index 0d189ae76922..dbe7f3744bf1 100644
> --- a/tools/perf/util/hist.c
> +++ b/tools/perf/util/hist.c
> @@ -861,8 +861,9 @@ int hist_entry_iter__add(struct hist_entry_iter *iter, struct addr_location *al,
>  {
>  	int err, err2;
>  
> -	err = sample__resolve_callchain(iter->sample, &iter->parent,
> -					iter->evsel, al, max_stack_depth);
> +	err = sample__resolve_callchain(iter->sample, iter->session,
> +					&iter->parent, iter->evsel, al,
> +					max_stack_depth);
>  	if (err)
>  		return err;
>  
> diff --git a/tools/perf/util/machine.c b/tools/perf/util/machine.c
> index 4743718d4bf1..63d860dca74b 100644
> --- a/tools/perf/util/machine.c
> +++ b/tools/perf/util/machine.c
> @@ -1540,10 +1540,11 @@ struct mem_info *sample__resolve_mem(struct perf_sample *sample,
>  }
>  
>  static int add_callchain_ip(struct thread *thread,
> +			    struct perf_session *session,
>  			    struct symbol **parent,
>  			    struct addr_location *root_al,
>  			    bool branch_history,
> -			    u64 ip)
> +			    u64 ip, u64 timestamp)
>  {
>  	struct addr_location al;
>  
> @@ -1551,8 +1552,9 @@ static int add_callchain_ip(struct thread *thread,
>  	al.sym = NULL;
>  
>  	if (branch_history)
> -		thread__find_cpumode_addr_location(thread, MAP__FUNCTION,
> -						   ip, &al);
> +		session__find_cpumode_addr_location(session, thread,
> +						    MAP__FUNCTION, ip, &al,
> +						    timestamp);
>  	else {
>  		u8 cpumode = PERF_RECORD_MISC_USER;
>  
> @@ -1579,8 +1581,8 @@ static int add_callchain_ip(struct thread *thread,
>  			}
>  			return 0;
>  		}
> -		thread__find_addr_location(thread, cpumode, MAP__FUNCTION,
> -				   ip, &al);
> +		session__find_addr_location(session,thread, cpumode,
> +					    MAP__FUNCTION, ip, &al, timestamp);
>  	}
>  
>  	if (al.sym != NULL) {
> @@ -1670,6 +1672,7 @@ static int remove_loops(struct branch_entry *l, int nr)
>   */
>  static int resolve_lbr_callchain_sample(struct thread *thread,
>  					struct perf_sample *sample,
> +					struct perf_session *session,
>  					struct symbol **parent,
>  					struct addr_location *root_al,
>  					int max_stack)
> @@ -1722,7 +1725,8 @@ static int resolve_lbr_callchain_sample(struct thread *thread,
>  					ip = lbr_stack->entries[0].to;
>  			}
>  
> -			err = add_callchain_ip(thread, parent, root_al, false, ip);
> +			err = add_callchain_ip(thread, session, parent, root_al,
> +					       false, ip, sample->time);
>  			if (err)
>  				return (err < 0) ? err : 0;
>  		}
> @@ -1735,6 +1739,7 @@ static int resolve_lbr_callchain_sample(struct thread *thread,
>  static int thread__resolve_callchain_sample(struct thread *thread,
>  					    struct perf_evsel *evsel,
>  					    struct perf_sample *sample,
> +					    struct perf_session *session,
>  					    struct symbol **parent,
>  					    struct addr_location *root_al,
>  					    int max_stack)
> @@ -1742,6 +1747,7 @@ static int thread__resolve_callchain_sample(struct thread *thread,
>  	struct branch_stack *branch = sample->branch_stack;
>  	struct ip_callchain *chain = sample->callchain;
>  	int chain_nr = min(max_stack, (int)chain->nr);
> +	u64 timestamp = sample->time;
>  	int i, j, err;
>  	int skip_idx = -1;
>  	int first_call = 0;
> @@ -1749,8 +1755,8 @@ static int thread__resolve_callchain_sample(struct thread *thread,
>  	callchain_cursor_reset(&callchain_cursor);
>  
>  	if (has_branch_callstack(evsel)) {
> -		err = resolve_lbr_callchain_sample(thread, sample, parent,
> -						   root_al, max_stack);
> +		err = resolve_lbr_callchain_sample(thread, sample, session,
> +						   parent, root_al, max_stack);
>  		if (err)
>  			return (err < 0) ? err : 0;
>  	}
> @@ -1806,11 +1812,12 @@ static int thread__resolve_callchain_sample(struct thread *thread,
>  		nr = remove_loops(be, nr);
>  
>  		for (i = 0; i < nr; i++) {
> -			err = add_callchain_ip(thread, parent, root_al,
> -					       true, be[i].to);
> +			err = add_callchain_ip(thread, session, parent, root_al,
> +					       true, be[i].to, timestamp);
>  			if (!err)
> -				err = add_callchain_ip(thread, parent, root_al,
> -						       true, be[i].from);
> +				err = add_callchain_ip(thread, session, parent,
> +						       root_al, true,
> +						       be[i].from, timestamp);
>  			if (err == -EINVAL)
>  				break;
>  			if (err)
> @@ -1839,7 +1846,8 @@ static int thread__resolve_callchain_sample(struct thread *thread,
>  #endif
>  		ip = chain->ips[j];
>  
> -		err = add_callchain_ip(thread, parent, root_al, false, ip);
> +		err = add_callchain_ip(thread, session, parent, root_al, false,
> +				       ip, timestamp);
>  		if (err)
>  			return (err < 0) ? err : 0;
>  	}
> @@ -1857,12 +1865,13 @@ static int unwind_entry(struct unwind_entry *entry, void *arg)
>  int thread__resolve_callchain(struct thread *thread,
>  			      struct perf_evsel *evsel,
>  			      struct perf_sample *sample,
> +			      struct perf_session *session,
>  			      struct symbol **parent,
>  			      struct addr_location *root_al,
>  			      int max_stack)
>  {
> -	int ret = thread__resolve_callchain_sample(thread, evsel,
> -						   sample, parent,
> +	int ret = thread__resolve_callchain_sample(thread, evsel, sample,
> +						   session, parent,
>  						   root_al, max_stack);
>  	if (ret)
>  		return ret;
> @@ -1878,7 +1887,7 @@ int thread__resolve_callchain(struct thread *thread,
>  		return 0;
>  
>  	return unwind__get_entries(unwind_entry, &callchain_cursor,
> -				   thread, sample, max_stack);
> +				   thread, session, sample, max_stack);
>  
>  }
>  
> diff --git a/tools/perf/util/machine.h b/tools/perf/util/machine.h
> index 45aee0c329ef..38ead24f0f47 100644
> --- a/tools/perf/util/machine.h
> +++ b/tools/perf/util/machine.h
> @@ -129,6 +129,7 @@ struct mem_info *sample__resolve_mem(struct perf_sample *sample,
>  int thread__resolve_callchain(struct thread *thread,
>  			      struct perf_evsel *evsel,
>  			      struct perf_sample *sample,
> +			      struct perf_session *session,
>  			      struct symbol **parent,
>  			      struct addr_location *root_al,
>  			      int max_stack);
> diff --git a/tools/perf/util/scripting-engines/trace-event-python.c b/tools/perf/util/scripting-engines/trace-event-python.c
> index 802def46af7b..e8c2896055c5 100644
> --- a/tools/perf/util/scripting-engines/trace-event-python.c
> +++ b/tools/perf/util/scripting-engines/trace-event-python.c
> @@ -298,9 +298,10 @@ static PyObject *get_field_numeric_entry(struct event_format *event,
>  }
>  
>  
> -static PyObject *python_process_callchain(struct perf_sample *sample,
> -					 struct perf_evsel *evsel,
> -					 struct addr_location *al)
> +static PyObject *python_process_callchain(struct perf_session *session,
> +					  struct perf_sample *sample,
> +					  struct perf_evsel *evsel,
> +					  struct addr_location *al)
>  {
>  	PyObject *pylist;
>  
> @@ -311,9 +312,8 @@ static PyObject *python_process_callchain(struct perf_sample *sample,
>  	if (!symbol_conf.use_callchain || !sample->callchain)
>  		goto exit;
>  
> -	if (thread__resolve_callchain(al->thread, evsel,
> -				      sample, NULL, NULL,
> -				      PERF_MAX_STACK_DEPTH) != 0) {
> +	if (thread__resolve_callchain(al->thread, evsel, sample, session,
> +				      NULL, NULL, PERF_MAX_STACK_DEPTH) != 0) {
>  		pr_err("Failed to resolve callchain. Skipping\n");
>  		goto exit;
>  	}
> @@ -374,7 +374,8 @@ static PyObject *python_process_callchain(struct perf_sample *sample,
>  }
>  
>  
> -static void python_process_tracepoint(struct perf_sample *sample,
> +static void python_process_tracepoint(struct perf_session *session,
> +				      struct perf_sample *sample,
>  				      struct perf_evsel *evsel,
>  				      struct thread *thread,
>  				      struct addr_location *al)
> @@ -424,7 +425,7 @@ static void python_process_tracepoint(struct perf_sample *sample,
>  	PyTuple_SetItem(t, n++, context);
>  
>  	/* ip unwinding */
> -	callchain = python_process_callchain(sample, evsel, al);
> +	callchain = python_process_callchain(session, sample, evsel, al);
>  
>  	if (handler) {
>  		PyTuple_SetItem(t, n++, PyInt_FromLong(cpu));
> @@ -759,7 +760,8 @@ static int python_process_call_return(struct call_return *cr, void *data)
>  	return db_export__call_return(dbe, cr);
>  }
>  
> -static void python_process_general_event(struct perf_sample *sample,
> +static void python_process_general_event(struct perf_session *session,
> +					 struct perf_sample *sample,
>  					 struct perf_evsel *evsel,
>  					 struct thread *thread,
>  					 struct addr_location *al)
> @@ -822,7 +824,7 @@ static void python_process_general_event(struct perf_sample *sample,
>  	}
>  
>  	/* ip unwinding */
> -	callchain = python_process_callchain(sample, evsel, al);
> +	callchain = python_process_callchain(session, sample, evsel, al);
>  	pydict_set_item_string_decref(dict, "callchain", callchain);
>  
>  	PyTuple_SetItem(t, n++, dict);
> @@ -846,7 +848,7 @@ static void python_process_event(union perf_event *event,
>  
>  	switch (evsel->attr.type) {
>  	case PERF_TYPE_TRACEPOINT:
> -		python_process_tracepoint(sample, evsel, thread, al);
> +		python_process_tracepoint(session, sample, evsel, thread, al);
>  		break;
>  	/* Reserve for future process_hw/sw/raw APIs */
>  	default:
> @@ -854,7 +856,8 @@ static void python_process_event(union perf_event *event,
>  			db_export__sample(&tables->dbe, event, sample, evsel,
>  					  thread, al, session);
>  		else
> -			python_process_general_event(sample, evsel, thread, al);
> +			python_process_general_event(session, sample, evsel,
> +						     thread, al);
>  	}
>  }
>  
> diff --git a/tools/perf/util/session.c b/tools/perf/util/session.c
> index 46761a39fbae..d89dfa8592a9 100644
> --- a/tools/perf/util/session.c
> +++ b/tools/perf/util/session.c
> @@ -1550,7 +1550,7 @@ struct perf_evsel *perf_session__find_first_evtype(struct perf_session *session,
>  }
>  
>  void perf_evsel__print_ip(struct perf_evsel *evsel, struct perf_sample *sample,
> -			  struct addr_location *al,
> +			  struct perf_session *session, struct addr_location *al,
>  			  unsigned int print_opts, unsigned int stack_depth)
>  {
>  	struct callchain_cursor_node *node;
> @@ -1565,8 +1565,8 @@ void perf_evsel__print_ip(struct perf_evsel *evsel, struct perf_sample *sample,
>  	if (symbol_conf.use_callchain && sample->callchain) {
>  		struct addr_location node_al;
>  
> -		if (thread__resolve_callchain(al->thread, evsel,
> -					      sample, NULL, NULL,
> +		if (thread__resolve_callchain(al->thread, evsel, sample,
> +					      session, NULL, NULL,
>  					      PERF_MAX_STACK_DEPTH) != 0) {
>  			if (verbose)
>  				error("Failed to resolve callchain. Skipping\n");
> diff --git a/tools/perf/util/session.h b/tools/perf/util/session.h
> index dbd21f8e7cf1..4d264fef8968 100644
> --- a/tools/perf/util/session.h
> +++ b/tools/perf/util/session.h
> @@ -102,7 +102,7 @@ struct perf_evsel *perf_session__find_first_evtype(struct perf_session *session,
>  					    unsigned int type);
>  
>  void perf_evsel__print_ip(struct perf_evsel *evsel, struct perf_sample *sample,
> -			  struct addr_location *al,
> +			  struct perf_session *session, struct addr_location *al,
>  			  unsigned int print_opts, unsigned int stack_depth);
>  
>  int perf_session__cpu_bitmap(struct perf_session *session,
> diff --git a/tools/perf/util/unwind-libdw.c b/tools/perf/util/unwind-libdw.c
> index 2dcfe9a7c8d0..ebaf51b58c92 100644
> --- a/tools/perf/util/unwind-libdw.c
> +++ b/tools/perf/util/unwind-libdw.c
> @@ -8,6 +8,7 @@
>  #include "unwind-libdw.h"
>  #include "machine.h"
>  #include "thread.h"
> +#include "session.h"
>  #include <linux/types.h>
>  #include "event.h"
>  #include "perf_regs.h"
> @@ -26,10 +27,9 @@ static int __report_module(struct addr_location *al, u64 ip,
>  	Dwfl_Module *mod;
>  	struct dso *dso = NULL;
>  
> -	thread__find_addr_location(ui->thread,
> -				   PERF_RECORD_MISC_USER,
> -				   MAP__FUNCTION, ip, al);
> -
> +	session__find_addr_location(ui->session, ui->thread,
> +				    PERF_RECORD_MISC_USER, MAP__FUNCTION,
> +				    ip, al, ui->sample->time);
>  	if (al->map)
>  		dso = al->map->dso;
>  
> @@ -89,8 +89,8 @@ static int access_dso_mem(struct unwind_info *ui, Dwarf_Addr addr,
>  	struct addr_location al;
>  	ssize_t size;
>  
> -	thread__find_addr_map(ui->thread, PERF_RECORD_MISC_USER,
> -			      MAP__FUNCTION, addr, &al);
> +	session__find_addr_map(ui->session, ui->thread, PERF_RECORD_MISC_USER,
> +			       MAP__FUNCTION, addr, &al, ui->sample->time);
>  	if (!al.map) {
>  		pr_debug("unwind: no map for %lx\n", (unsigned long)addr);
>  		return -1;
> @@ -165,12 +165,14 @@ frame_callback(Dwfl_Frame *state, void *arg)
>  
>  int unwind__get_entries(unwind_entry_cb_t cb, void *arg,
>  			struct thread *thread,
> +			struct perf_session *session,
>  			struct perf_sample *data,
>  			int max_stack)
>  {
>  	struct unwind_info ui = {
>  		.sample		= data,
>  		.thread		= thread,
> +		.session 	= session,
>  		.machine	= thread->mg->machine,
>  		.cb		= cb,
>  		.arg		= arg,
> diff --git a/tools/perf/util/unwind-libdw.h b/tools/perf/util/unwind-libdw.h
> index 417a1426f3ad..806e522713a2 100644
> --- a/tools/perf/util/unwind-libdw.h
> +++ b/tools/perf/util/unwind-libdw.h
> @@ -11,6 +11,7 @@ bool libdw__arch_set_initial_registers(Dwfl_Thread *thread, void *arg);
>  struct unwind_info {
>  	Dwfl			*dwfl;
>  	struct perf_sample      *sample;
> +	struct perf_session     *session;
>  	struct machine          *machine;
>  	struct thread           *thread;
>  	unwind_entry_cb_t	cb;
> diff --git a/tools/perf/util/unwind-libunwind.c b/tools/perf/util/unwind-libunwind.c
> index e3c40a520a25..9ee63179383e 100644
> --- a/tools/perf/util/unwind-libunwind.c
> +++ b/tools/perf/util/unwind-libunwind.c
> @@ -86,6 +86,7 @@ UNW_OBJ(dwarf_find_debug_frame) (int found, unw_dyn_info_t *di_debug,
>  
>  struct unwind_info {
>  	struct perf_sample	*sample;
> +	struct perf_session	*session;
>  	struct machine		*machine;
>  	struct thread		*thread;
>  };
> @@ -315,8 +316,8 @@ static struct map *find_map(unw_word_t ip, struct unwind_info *ui)
>  {
>  	struct addr_location al;
>  
> -	thread__find_addr_map(ui->thread, PERF_RECORD_MISC_USER,
> -			      MAP__FUNCTION, ip, &al);
> +	session__find_addr_map(ui->session, ui->thread, PERF_RECORD_MISC_USER,
> +			       MAP__FUNCTION, ip, &al, ui->sample->time);
>  	return al.map;
>  }
>  
> @@ -406,20 +407,19 @@ get_proc_name(unw_addr_space_t __maybe_unused as,
>  static int access_dso_mem(struct unwind_info *ui, unw_word_t addr,
>  			  unw_word_t *data)
>  {
> -	struct addr_location al;
> +	struct map *map;
>  	ssize_t size;
>  
> -	thread__find_addr_map(ui->thread, PERF_RECORD_MISC_USER,
> -			      MAP__FUNCTION, addr, &al);
> -	if (!al.map) {
> +	map = find_map(addr, ui);
> +	if (!map) {
>  		pr_debug("unwind: no map for %lx\n", (unsigned long)addr);
>  		return -1;
>  	}
>  
> -	if (!al.map->dso)
> +	if (!map->dso)
>  		return -1;
>  
> -	size = dso__data_read_addr(al.map->dso, al.map, ui->machine,
> +	size = dso__data_read_addr(map->dso, map, ui->machine,
>  				   addr, (u8 *) data, sizeof(*data));
>  
>  	return !(size == sizeof(*data));
> @@ -511,14 +511,14 @@ static void put_unwind_info(unw_addr_space_t __maybe_unused as,
>  	pr_debug("unwind: put_unwind_info called\n");
>  }
>  
> -static int entry(u64 ip, struct thread *thread,
> -		 unwind_entry_cb_t cb, void *arg)
> +static int entry(u64 ip, struct thread *thread, struct perf_session *session,
> +		 u64 timestamp, unwind_entry_cb_t cb, void *arg)
>  {
>  	struct unwind_entry e;
>  	struct addr_location al;
>  
> -	thread__find_addr_location(thread, PERF_RECORD_MISC_USER,
> -				   MAP__FUNCTION, ip, &al);
> +	session__find_addr_location(session, thread, PERF_RECORD_MISC_USER,
> +				    MAP__FUNCTION, ip, &al, timestamp);
>  
>  	e.ip = ip;
>  	e.map = al.map;
> @@ -620,20 +620,22 @@ static int get_entries(struct unwind_info *ui, unwind_entry_cb_t cb,
>  		unw_word_t ip;
>  
>  		unw_get_reg(&c, UNW_REG_IP, &ip);
> -		ret = ip ? entry(ip, ui->thread, cb, arg) : 0;
> +		ret = ip ? entry(ip, ui->thread, ui->session,
> +				 ui->sample->time, cb, arg) : 0;
>  	}
>  
>  	return ret;
>  }
>  
>  int unwind__get_entries(unwind_entry_cb_t cb, void *arg,
> -			struct thread *thread,
> +			struct thread *thread, struct perf_session *session,
>  			struct perf_sample *data, int max_stack)
>  {
>  	u64 ip;
>  	struct unwind_info ui = {
>  		.sample       = data,
>  		.thread       = thread,
> +		.session      = session,
>  		.machine      = thread->mg->machine,
>  	};
>  	int ret;
> @@ -645,7 +647,7 @@ int unwind__get_entries(unwind_entry_cb_t cb, void *arg,
>  	if (ret)
>  		return ret;
>  
> -	ret = entry(ip, thread, cb, arg);
> +	ret = entry(ip, thread, session, data->time, cb, arg);
>  	if (ret)
>  		return -ENOMEM;
>  
> diff --git a/tools/perf/util/unwind.h b/tools/perf/util/unwind.h
> index 12790cf94618..c619890e60ad 100644
> --- a/tools/perf/util/unwind.h
> +++ b/tools/perf/util/unwind.h
> @@ -16,7 +16,7 @@ typedef int (*unwind_entry_cb_t)(struct unwind_entry *entry, void *arg);
>  
>  #ifdef HAVE_DWARF_UNWIND_SUPPORT
>  int unwind__get_entries(unwind_entry_cb_t cb, void *arg,
> -			struct thread *thread,
> +			struct thread *thread, struct perf_session *session,
>  			struct perf_sample *data, int max_stack);
>  /* libunwind specific */
>  #ifdef HAVE_LIBUNWIND_SUPPORT
> @@ -38,6 +38,7 @@ static inline int
>  unwind__get_entries(unwind_entry_cb_t cb __maybe_unused,
>  		    void *arg __maybe_unused,
>  		    struct thread *thread __maybe_unused,
> +		    struct perf_session *session __maybe_unused,
>  		    struct perf_sample *data __maybe_unused,
>  		    int max_stack __maybe_unused)
>  {
> -- 
> 2.2.2

^ permalink raw reply	[flat|nested] 221+ messages in thread

* Re: [PATCH 10/38] perf tools: Pass session arg to perf_event__preprocess_sample()
  2015-03-03 13:59   ` Arnaldo Carvalho de Melo
@ 2015-03-03 14:18     ` Namhyung Kim
  0 siblings, 0 replies; 221+ messages in thread
From: Namhyung Kim @ 2015-03-03 14:18 UTC (permalink / raw)
  To: Arnaldo Carvalho de Melo
  Cc: Ingo Molnar, Peter Zijlstra, Jiri Olsa, LKML,
	Frederic Weisbecker, Adrian Hunter, Stephane Eranian, Andi Kleen,
	David Ahern

On Tue, Mar 03, 2015 at 10:59:30AM -0300, Arnaldo Carvalho de Melo wrote:
> Em Tue, Mar 03, 2015 at 12:07:22PM +0900, Namhyung Kim escreveu:
> > The perf_event__preprocess_sample() translates a given ip into a
> > matching symbol.  To do that, it first finds a corresponding thread
> > and map in the current thread tree.  But for indexed data files, it
> > needs to find a thread (and map) with slightly different APIs using
> > timestamp.  So it needs a way to know whether this session deals with
> > an indexed data file or not.
> 
> I need to look into this, but in general, I'm trying to _remove_
> accesses to perf_session :-)

The reason is that it needs to know whether the current session has
index so that it's accessing the samples in sequence or in random
order.  If we can save that info at some other place (like a global
variable?) it doesn't need to pass session.

Thanks,
Namhyung

^ permalink raw reply	[flat|nested] 221+ messages in thread

* Re: [PATCH 12/38] perf tools: Introduce thread__comm_time() helpers
  2015-03-03  3:07 ` [PATCH 12/38] perf tools: Introduce thread__comm_time() helpers Namhyung Kim
@ 2015-03-03 16:28   ` Frederic Weisbecker
  2015-03-04  0:02     ` Namhyung Kim
  0 siblings, 1 reply; 221+ messages in thread
From: Frederic Weisbecker @ 2015-03-03 16:28 UTC (permalink / raw)
  To: Namhyung Kim
  Cc: Arnaldo Carvalho de Melo, Ingo Molnar, Peter Zijlstra, Jiri Olsa,
	LKML, Adrian Hunter, Stephane Eranian, Andi Kleen, David Ahern

On Tue, Mar 03, 2015 at 12:07:24PM +0900, Namhyung Kim wrote:
> When data file indexing is enabled, it processes all task, comm and mmap
> events first and then goes to the sample events.  So all it sees is the
> last comm of a thread although it has information at the time of sample.
> 
> Sort thread's comm by time so that it can find appropriate comm at the
> sample time.  The thread__comm_time() will mostly work even if
> PERF_SAMPLE_TIME bit is off since in that case, sample->time will be
> -1 so it'll take the last comm anyway.
> 
> Cc: Frederic Weisbecker <fweisbec@gmail.com>
> Signed-off-by: Namhyung Kim <namhyung@kernel.org>
> ---
>  tools/perf/util/thread.c | 33 ++++++++++++++++++++++++++++++++-
>  tools/perf/util/thread.h |  2 ++
>  2 files changed, 34 insertions(+), 1 deletion(-)
> 
> diff --git a/tools/perf/util/thread.c b/tools/perf/util/thread.c
> index 9ebc8b1f9be5..ad96725105c2 100644
> --- a/tools/perf/util/thread.c
> +++ b/tools/perf/util/thread.c
> @@ -103,6 +103,21 @@ struct comm *thread__exec_comm(const struct thread *thread)
>  	return last;
>  }
>  
> +struct comm *thread__comm_time(const struct thread *thread, u64 timestamp)

Usually thread__comm_foo() would suggest that we return the "foo" from a thread comm.
For example thread__comm_len() returns the len of the last thread comm.
thread__comm_str() returns the string of the last thread comm.

So thread__comm_time() suggests that we return the timestamp of the default thread comm.
Ideally all thread__comm_foo() accessors should now take a timestamp as an argument. Now there
are quite some callers of such APIs, I'm not sure they will all turn into passing a precise timestamp,
but the current callers are interested in the last comm so perhaps those can be turned into thread__last_comm[_str]().
The call would gain some clarity.

> +{
> +	struct comm *comm;
> +
> +	list_for_each_entry(comm, &thread->comm_list, list) {
> +		if (timestamp >= comm->start)
> +			return comm;
> +	}
> +
> +	if (list_empty(&thread->comm_list))
> +		return NULL;
> +
> +	return list_last_entry(&thread->comm_list, struct comm, list);
> +}

Yes, handling the thread's comm lifecycle correctly with fetching the right comm at a given time is
the evolution I had in mind. I haven't looked at the rest of your patchset but this
change alone seem to go to the right direction.

Thanks!


> +
>  int __thread__set_comm(struct thread *thread, const char *str, u64 timestamp,
>  		       bool exec)
>  {
> @@ -118,7 +133,13 @@ int __thread__set_comm(struct thread *thread, const char *str, u64 timestamp,
>  		new = comm__new(str, timestamp, exec);
>  		if (!new)
>  			return -ENOMEM;
> -		list_add(&new->list, &thread->comm_list);
> +
> +		/* sort by time */
> +		list_for_each_entry(curr, &thread->comm_list, list) {
> +			if (timestamp >= curr->start)
> +				break;
> +		}
> +		list_add_tail(&new->list, &curr->list);
>  
>  		if (exec)
>  			unwind__flush_access(thread);
> @@ -139,6 +160,16 @@ const char *thread__comm_str(const struct thread *thread)
>  	return comm__str(comm);
>  }
>  
> +const char *thread__comm_str_time(const struct thread *thread, u64 timestamp)
> +{
> +	const struct comm *comm = thread__comm_time(thread, timestamp);
> +
> +	if (!comm)
> +		return NULL;
> +
> +	return comm__str(comm);
> +}
> +
>  /* CHECKME: it should probably better return the max comm len from its comm list */
>  int thread__comm_len(struct thread *thread)
>  {
> diff --git a/tools/perf/util/thread.h b/tools/perf/util/thread.h
> index 160fd066a7d1..be67c3bad5e7 100644
> --- a/tools/perf/util/thread.h
> +++ b/tools/perf/util/thread.h
> @@ -53,7 +53,9 @@ static inline int thread__set_comm(struct thread *thread, const char *comm,
>  int thread__comm_len(struct thread *thread);
>  struct comm *thread__comm(const struct thread *thread);
>  struct comm *thread__exec_comm(const struct thread *thread);
> +struct comm *thread__comm_time(const struct thread *thread, u64 timestamp);
>  const char *thread__comm_str(const struct thread *thread);
> +const char *thread__comm_str_time(const struct thread *thread, u64 timestamp);
>  void thread__insert_map(struct thread *thread, struct map *map);
>  int thread__fork(struct thread *thread, struct thread *parent, u64 timestamp);
>  size_t thread__fprintf(struct thread *thread, FILE *fp);
> -- 
> 2.2.2
> 

^ permalink raw reply	[flat|nested] 221+ messages in thread

* Re: [PATCH 12/38] perf tools: Introduce thread__comm_time() helpers
  2015-03-03 16:28   ` Frederic Weisbecker
@ 2015-03-04  0:02     ` Namhyung Kim
  2015-03-05 16:08       ` Frederic Weisbecker
  0 siblings, 1 reply; 221+ messages in thread
From: Namhyung Kim @ 2015-03-04  0:02 UTC (permalink / raw)
  To: Frederic Weisbecker
  Cc: Arnaldo Carvalho de Melo, Ingo Molnar, Peter Zijlstra, Jiri Olsa,
	LKML, Adrian Hunter, Stephane Eranian, Andi Kleen, David Ahern

Hi Frederic,

On Tue, Mar 03, 2015 at 05:28:40PM +0100, Frederic Weisbecker wrote:
> On Tue, Mar 03, 2015 at 12:07:24PM +0900, Namhyung Kim wrote:
> > When data file indexing is enabled, it processes all task, comm and mmap
> > events first and then goes to the sample events.  So all it sees is the
> > last comm of a thread although it has information at the time of sample.
> > 
> > Sort thread's comm by time so that it can find appropriate comm at the
> > sample time.  The thread__comm_time() will mostly work even if
> > PERF_SAMPLE_TIME bit is off since in that case, sample->time will be
> > -1 so it'll take the last comm anyway.
> > 
> > Cc: Frederic Weisbecker <fweisbec@gmail.com>
> > Signed-off-by: Namhyung Kim <namhyung@kernel.org>
> > ---
> >  tools/perf/util/thread.c | 33 ++++++++++++++++++++++++++++++++-
> >  tools/perf/util/thread.h |  2 ++
> >  2 files changed, 34 insertions(+), 1 deletion(-)
> > 
> > diff --git a/tools/perf/util/thread.c b/tools/perf/util/thread.c
> > index 9ebc8b1f9be5..ad96725105c2 100644
> > --- a/tools/perf/util/thread.c
> > +++ b/tools/perf/util/thread.c
> > @@ -103,6 +103,21 @@ struct comm *thread__exec_comm(const struct thread *thread)
> >  	return last;
> >  }
> >  
> > +struct comm *thread__comm_time(const struct thread *thread, u64 timestamp)
> 
> Usually thread__comm_foo() would suggest that we return the "foo" from a thread comm.
> For example thread__comm_len() returns the len of the last thread comm.
> thread__comm_str() returns the string of the last thread comm.

Ah, okay.

> 
> So thread__comm_time() suggests that we return the timestamp of the default thread comm.
> Ideally all thread__comm_foo() accessors should now take a timestamp as an argument. Now there
> are quite some callers of such APIs, I'm not sure they will all turn into passing a precise timestamp,
> but the current callers are interested in the last comm so perhaps those can be turned into thread__last_comm[_str]().
> The call would gain some clarity.

I'm fine with this change.  Actually I also don't like the _time
suffix but couldn't come up with a better name. ;-)

I also think the last comm also be thought as current comm.  So how
about thread__curr_comm[_str]() then?


> 
> > +{
> > +	struct comm *comm;
> > +
> > +	list_for_each_entry(comm, &thread->comm_list, list) {
> > +		if (timestamp >= comm->start)
> > +			return comm;
> > +	}
> > +
> > +	if (list_empty(&thread->comm_list))
> > +		return NULL;
> > +
> > +	return list_last_entry(&thread->comm_list, struct comm, list);
> > +}
> 
> Yes, handling the thread's comm lifecycle correctly with fetching the right comm at a given time is
> the evolution I had in mind. I haven't looked at the rest of your patchset but this
> change alone seem to go to the right direction.

The idea is extending it to find thread and maps so that we can
process samples parallelly.

Thanks for your review!
Namhyung

^ permalink raw reply	[flat|nested] 221+ messages in thread

* Re: [PATCH 03/38] perf tools: Introduce copyfile_offset() function
  2015-03-03  3:07 ` [PATCH 03/38] perf tools: Introduce copyfile_offset() function Namhyung Kim
@ 2015-03-04 14:58   ` Jiri Olsa
  0 siblings, 0 replies; 221+ messages in thread
From: Jiri Olsa @ 2015-03-04 14:58 UTC (permalink / raw)
  To: Namhyung Kim
  Cc: Arnaldo Carvalho de Melo, Ingo Molnar, Peter Zijlstra, LKML,
	Frederic Weisbecker, Adrian Hunter, Stephane Eranian, Andi Kleen,
	David Ahern

On Tue, Mar 03, 2015 at 12:07:15PM +0900, Namhyung Kim wrote:
> The copyfile_offset() function is to copy source data from given
> offset to a destination file with an offset.  It'll be used to build
> an indexed data file.
> 
> Signed-off-by: Namhyung Kim <namhyung@kernel.org>

Acked-by: Jiri Olsa <jolsa@kernel.org>

thanks,
jirka

^ permalink raw reply	[flat|nested] 221+ messages in thread

* Re: [PATCH 07/38] perf tools: Handle indexed data file properly
  2015-03-03  3:07 ` [PATCH 07/38] perf tools: Handle indexed data file properly Namhyung Kim
@ 2015-03-04 16:19   ` Jiri Olsa
  2015-03-06  4:17     ` Namhyung Kim
  0 siblings, 1 reply; 221+ messages in thread
From: Jiri Olsa @ 2015-03-04 16:19 UTC (permalink / raw)
  To: Namhyung Kim
  Cc: Arnaldo Carvalho de Melo, Ingo Molnar, Peter Zijlstra, LKML,
	Frederic Weisbecker, Adrian Hunter, Stephane Eranian, Andi Kleen,
	David Ahern

On Tue, Mar 03, 2015 at 12:07:19PM +0900, Namhyung Kim wrote:
> When perf detects data file has index table, process header part first
> and then rest data files in a row.  Note that the indexed sample data is
> recorded for each cpu/thread separately, it's already ordered with
> respect to themselves so no need to use the ordered event queue
> interface.
> 
> Signed-off-by: Namhyung Kim <namhyung@kernel.org>
> ---
>  tools/perf/util/session.c | 62 ++++++++++++++++++++++++++++++++++++++---------
>  tools/perf/util/session.h |  5 ++++
>  2 files changed, 55 insertions(+), 12 deletions(-)
> 
> diff --git a/tools/perf/util/session.c b/tools/perf/util/session.c
> index e4f166981ff0..00cd1ad427be 100644
> --- a/tools/perf/util/session.c
> +++ b/tools/perf/util/session.c
> @@ -1300,11 +1300,10 @@ fetch_mmaped_event(struct perf_session *session,
>  #define NUM_MMAPS 128
>  #endif
>  
> -static int __perf_session__process_events(struct perf_session *session,
> +static int __perf_session__process_events(struct perf_session *session, int fd,
>  					  u64 data_offset, u64 data_size,
>  					  u64 file_size, struct perf_tool *tool)
>  {
> -	int fd = perf_data_file__fd(session->file);

why is 'fd' passed separatelly here? we have single file now
and the only 'file::fd' we use is in session, no?

I might be missing some change in later patches ;-)

jirka

^ permalink raw reply	[flat|nested] 221+ messages in thread

* Re: [PATCH 08/38] perf record: Add --index option for building index table
  2015-03-03  3:07 ` [PATCH 08/38] perf record: Add --index option for building index table Namhyung Kim
@ 2015-03-05  7:56   ` Jiri Olsa
  2015-03-06  4:19     ` Namhyung Kim
  0 siblings, 1 reply; 221+ messages in thread
From: Jiri Olsa @ 2015-03-05  7:56 UTC (permalink / raw)
  To: Namhyung Kim
  Cc: Arnaldo Carvalho de Melo, Ingo Molnar, Peter Zijlstra, LKML,
	Frederic Weisbecker, Adrian Hunter, Stephane Eranian, Andi Kleen,
	David Ahern

On Tue, Mar 03, 2015 at 12:07:20PM +0900, Namhyung Kim wrote:

SNIP

> +static int record__merge_index_files(struct record *rec, int nr_index)
> +{
> +	int i;
> +	int ret = -1;
> +	u64 offset;
> +	char path[PATH_MAX];
> +	struct perf_file_section *idx;
> +	struct perf_data_file *file = &rec->file;
> +	struct perf_session *session = rec->session;
> +	int output_fd = perf_data_file__fd(file);
> +
> +	/* +1 for header file itself */
> +	nr_index++;
> +
> +	idx = calloc(nr_index, sizeof(*idx));
> +	if (idx == NULL)
> +		goto out_close;
> +
> +	offset = lseek(output_fd, 0, SEEK_END);
> +
> +	idx[0].offset = session->header.data_offset;
> +	idx[0].size   = offset - idx[0].offset;
> +
> +	for (i = 1; i < nr_index; i++) {
> +		struct stat stbuf;
> +		int fd = rec->fds[i];
> +
> +		if (fstat(fd, &stbuf) < 0)
> +			goto out_close;
> +
> +		idx[i].offset = offset;
> +		idx[i].size   = stbuf.st_size;
> +
> +		offset += stbuf.st_size;
> +	}
> +
> +	/* copy sample events */
> +	for (i = 1; i < nr_index; i++) {
> +		int fd = rec->fds[i];
> +
> +		if (idx[i].size == 0)
> +			continue;
> +
> +		if (copyfile_offset(fd, 0, output_fd, idx[i].offset,
> +				    idx[i].size) < 0)
> +			goto out_close;
> +	}

why not do the copy in previous loop as well?

jirka

^ permalink raw reply	[flat|nested] 221+ messages in thread

* Re: [PATCH 12/38] perf tools: Introduce thread__comm_time() helpers
  2015-03-04  0:02     ` Namhyung Kim
@ 2015-03-05 16:08       ` Frederic Weisbecker
  2015-03-06  4:38         ` Namhyung Kim
  0 siblings, 1 reply; 221+ messages in thread
From: Frederic Weisbecker @ 2015-03-05 16:08 UTC (permalink / raw)
  To: Namhyung Kim
  Cc: Arnaldo Carvalho de Melo, Ingo Molnar, Peter Zijlstra, Jiri Olsa,
	LKML, Adrian Hunter, Stephane Eranian, Andi Kleen, David Ahern

On Wed, Mar 04, 2015 at 09:02:55AM +0900, Namhyung Kim wrote:
> Hi Frederic,
> 
> On Tue, Mar 03, 2015 at 05:28:40PM +0100, Frederic Weisbecker wrote:
> > On Tue, Mar 03, 2015 at 12:07:24PM +0900, Namhyung Kim wrote:
> > > When data file indexing is enabled, it processes all task, comm and mmap
> > > events first and then goes to the sample events.  So all it sees is the
> > > last comm of a thread although it has information at the time of sample.
> > > 
> > > Sort thread's comm by time so that it can find appropriate comm at the
> > > sample time.  The thread__comm_time() will mostly work even if
> > > PERF_SAMPLE_TIME bit is off since in that case, sample->time will be
> > > -1 so it'll take the last comm anyway.
> > > 
> > > Cc: Frederic Weisbecker <fweisbec@gmail.com>
> > > Signed-off-by: Namhyung Kim <namhyung@kernel.org>
> > > ---
> > >  tools/perf/util/thread.c | 33 ++++++++++++++++++++++++++++++++-
> > >  tools/perf/util/thread.h |  2 ++
> > >  2 files changed, 34 insertions(+), 1 deletion(-)
> > > 
> > > diff --git a/tools/perf/util/thread.c b/tools/perf/util/thread.c
> > > index 9ebc8b1f9be5..ad96725105c2 100644
> > > --- a/tools/perf/util/thread.c
> > > +++ b/tools/perf/util/thread.c
> > > @@ -103,6 +103,21 @@ struct comm *thread__exec_comm(const struct thread *thread)
> > >  	return last;
> > >  }
> > >  
> > > +struct comm *thread__comm_time(const struct thread *thread, u64 timestamp)
> > 
> > Usually thread__comm_foo() would suggest that we return the "foo" from a thread comm.
> > For example thread__comm_len() returns the len of the last thread comm.
> > thread__comm_str() returns the string of the last thread comm.
> 
> Ah, okay.

I mean, that's just an impression, others may have a different one :o)

> 
> > 
> > So thread__comm_time() suggests that we return the timestamp of the default thread comm.
> > Ideally all thread__comm_foo() accessors should now take a timestamp as an argument. Now there
> > are quite some callers of such APIs, I'm not sure they will all turn into passing a precise timestamp,
> > but the current callers are interested in the last comm so perhaps those can be turned into thread__last_comm[_str]().
> > The call would gain some clarity.
> 
> I'm fine with this change.  Actually I also don't like the _time
> suffix but couldn't come up with a better name. ;-)
> 
> I also think the last comm also be thought as current comm.  So how
> about thread__curr_comm[_str]() then?

Yeah, curr works for me too.

Thanks!

> 
> 
> > 
> > > +{
> > > +	struct comm *comm;
> > > +
> > > +	list_for_each_entry(comm, &thread->comm_list, list) {
> > > +		if (timestamp >= comm->start)
> > > +			return comm;
> > > +	}
> > > +
> > > +	if (list_empty(&thread->comm_list))
> > > +		return NULL;
> > > +
> > > +	return list_last_entry(&thread->comm_list, struct comm, list);
> > > +}
> > 
> > Yes, handling the thread's comm lifecycle correctly with fetching the right comm at a given time is
> > the evolution I had in mind. I haven't looked at the rest of your patchset but this
> > change alone seem to go to the right direction.
> 
> The idea is extending it to find thread and maps so that we can
> process samples parallelly.

Ok!

Thanks!

> 
> Thanks for your review!
> Namhyung

^ permalink raw reply	[flat|nested] 221+ messages in thread

* Re: [PATCH 07/38] perf tools: Handle indexed data file properly
  2015-03-04 16:19   ` Jiri Olsa
@ 2015-03-06  4:17     ` Namhyung Kim
  0 siblings, 0 replies; 221+ messages in thread
From: Namhyung Kim @ 2015-03-06  4:17 UTC (permalink / raw)
  To: Jiri Olsa
  Cc: Arnaldo Carvalho de Melo, Ingo Molnar, Peter Zijlstra, LKML,
	Frederic Weisbecker, Adrian Hunter, Stephane Eranian, Andi Kleen,
	David Ahern

Hi Jiri,

On Wed, Mar 04, 2015 at 05:19:54PM +0100, Jiri Olsa wrote:
> On Tue, Mar 03, 2015 at 12:07:19PM +0900, Namhyung Kim wrote:
> > When perf detects data file has index table, process header part first
> > and then rest data files in a row.  Note that the indexed sample data is
> > recorded for each cpu/thread separately, it's already ordered with
> > respect to themselves so no need to use the ordered event queue
> > interface.
> > 
> > Signed-off-by: Namhyung Kim <namhyung@kernel.org>
> > ---
> >  tools/perf/util/session.c | 62 ++++++++++++++++++++++++++++++++++++++---------
> >  tools/perf/util/session.h |  5 ++++
> >  2 files changed, 55 insertions(+), 12 deletions(-)
> > 
> > diff --git a/tools/perf/util/session.c b/tools/perf/util/session.c
> > index e4f166981ff0..00cd1ad427be 100644
> > --- a/tools/perf/util/session.c
> > +++ b/tools/perf/util/session.c
> > @@ -1300,11 +1300,10 @@ fetch_mmaped_event(struct perf_session *session,
> >  #define NUM_MMAPS 128
> >  #endif
> >  
> > -static int __perf_session__process_events(struct perf_session *session,
> > +static int __perf_session__process_events(struct perf_session *session, int fd,
> >  					  u64 data_offset, u64 data_size,
> >  					  u64 file_size, struct perf_tool *tool)
> >  {
> > -	int fd = perf_data_file__fd(session->file);
> 
> why is 'fd' passed separatelly here? we have single file now
> and the only 'file::fd' we use is in session, no?

You're right, it's a leftover from the old code.  Will change.

Thanks,
Namhyung

^ permalink raw reply	[flat|nested] 221+ messages in thread

* Re: [PATCH 08/38] perf record: Add --index option for building index table
  2015-03-05  7:56   ` Jiri Olsa
@ 2015-03-06  4:19     ` Namhyung Kim
  0 siblings, 0 replies; 221+ messages in thread
From: Namhyung Kim @ 2015-03-06  4:19 UTC (permalink / raw)
  To: Jiri Olsa
  Cc: Arnaldo Carvalho de Melo, Ingo Molnar, Peter Zijlstra, LKML,
	Frederic Weisbecker, Adrian Hunter, Stephane Eranian, Andi Kleen,
	David Ahern

On Thu, Mar 05, 2015 at 08:56:44AM +0100, Jiri Olsa wrote:
> On Tue, Mar 03, 2015 at 12:07:20PM +0900, Namhyung Kim wrote:
> 
> SNIP
> 
> > +static int record__merge_index_files(struct record *rec, int nr_index)
> > +{
> > +	int i;
> > +	int ret = -1;
> > +	u64 offset;
> > +	char path[PATH_MAX];
> > +	struct perf_file_section *idx;
> > +	struct perf_data_file *file = &rec->file;
> > +	struct perf_session *session = rec->session;
> > +	int output_fd = perf_data_file__fd(file);
> > +
> > +	/* +1 for header file itself */
> > +	nr_index++;
> > +
> > +	idx = calloc(nr_index, sizeof(*idx));
> > +	if (idx == NULL)
> > +		goto out_close;
> > +
> > +	offset = lseek(output_fd, 0, SEEK_END);
> > +
> > +	idx[0].offset = session->header.data_offset;
> > +	idx[0].size   = offset - idx[0].offset;
> > +
> > +	for (i = 1; i < nr_index; i++) {
> > +		struct stat stbuf;
> > +		int fd = rec->fds[i];
> > +
> > +		if (fstat(fd, &stbuf) < 0)
> > +			goto out_close;
> > +
> > +		idx[i].offset = offset;
> > +		idx[i].size   = stbuf.st_size;
> > +
> > +		offset += stbuf.st_size;
> > +	}
> > +
> > +	/* copy sample events */
> > +	for (i = 1; i < nr_index; i++) {
> > +		int fd = rec->fds[i];
> > +
> > +		if (idx[i].size == 0)
> > +			continue;
> > +
> > +		if (copyfile_offset(fd, 0, output_fd, idx[i].offset,
> > +				    idx[i].size) < 0)
> > +			goto out_close;
> > +	}
> 
> why not do the copy in previous loop as well?

I will change it in the next version.

Thanks,
Namhyung

^ permalink raw reply	[flat|nested] 221+ messages in thread

* Re: [PATCH 12/38] perf tools: Introduce thread__comm_time() helpers
  2015-03-05 16:08       ` Frederic Weisbecker
@ 2015-03-06  4:38         ` Namhyung Kim
  2015-03-06 12:48           ` Arnaldo Carvalho de Melo
  0 siblings, 1 reply; 221+ messages in thread
From: Namhyung Kim @ 2015-03-06  4:38 UTC (permalink / raw)
  To: Frederic Weisbecker, Arnaldo Carvalho de Melo
  Cc: Ingo Molnar, Peter Zijlstra, Jiri Olsa, LKML, Adrian Hunter,
	Stephane Eranian, Andi Kleen, David Ahern

Hi Frederic and Arnaldo,

On Thu, Mar 05, 2015 at 05:08:56PM +0100, Frederic Weisbecker wrote:
> On Wed, Mar 04, 2015 at 09:02:55AM +0900, Namhyung Kim wrote:
> > Hi Frederic,
> > 
> > On Tue, Mar 03, 2015 at 05:28:40PM +0100, Frederic Weisbecker wrote:
> > > On Tue, Mar 03, 2015 at 12:07:24PM +0900, Namhyung Kim wrote:
> > > > When data file indexing is enabled, it processes all task, comm and mmap
> > > > events first and then goes to the sample events.  So all it sees is the
> > > > last comm of a thread although it has information at the time of sample.
> > > > 
> > > > Sort thread's comm by time so that it can find appropriate comm at the
> > > > sample time.  The thread__comm_time() will mostly work even if
> > > > PERF_SAMPLE_TIME bit is off since in that case, sample->time will be
> > > > -1 so it'll take the last comm anyway.
> > > > 
> > > > Cc: Frederic Weisbecker <fweisbec@gmail.com>
> > > > Signed-off-by: Namhyung Kim <namhyung@kernel.org>
> > > > ---
> > > >  tools/perf/util/thread.c | 33 ++++++++++++++++++++++++++++++++-
> > > >  tools/perf/util/thread.h |  2 ++
> > > >  2 files changed, 34 insertions(+), 1 deletion(-)
> > > > 
> > > > diff --git a/tools/perf/util/thread.c b/tools/perf/util/thread.c
> > > > index 9ebc8b1f9be5..ad96725105c2 100644
> > > > --- a/tools/perf/util/thread.c
> > > > +++ b/tools/perf/util/thread.c
> > > > @@ -103,6 +103,21 @@ struct comm *thread__exec_comm(const struct thread *thread)
> > > >  	return last;
> > > >  }
> > > >  
> > > > +struct comm *thread__comm_time(const struct thread *thread, u64 timestamp)
> > > 
> > > Usually thread__comm_foo() would suggest that we return the "foo" from a thread comm.
> > > For example thread__comm_len() returns the len of the last thread comm.
> > > thread__comm_str() returns the string of the last thread comm.
> > 
> > Ah, okay.
> 
> I mean, that's just an impression, others may have a different one :o)

Right.  Although I agree with your idea of function naming, I'm not
sure it's worth changing every function call site for this - and for
similar machine__find(new)_thread()_time.

Arnaldo, What do you think?

Thanks,
Namhyung

^ permalink raw reply	[flat|nested] 221+ messages in thread

* Re: [PATCH 12/38] perf tools: Introduce thread__comm_time() helpers
  2015-03-06  4:38         ` Namhyung Kim
@ 2015-03-06 12:48           ` Arnaldo Carvalho de Melo
  2015-03-10  6:55             ` Namhyung Kim
  0 siblings, 1 reply; 221+ messages in thread
From: Arnaldo Carvalho de Melo @ 2015-03-06 12:48 UTC (permalink / raw)
  To: Namhyung Kim
  Cc: Frederic Weisbecker, Ingo Molnar, Peter Zijlstra, Jiri Olsa,
	LKML, Adrian Hunter, Stephane Eranian, Andi Kleen, David Ahern

Em Fri, Mar 06, 2015 at 01:38:06PM +0900, Namhyung Kim escreveu:
> On Thu, Mar 05, 2015 at 05:08:56PM +0100, Frederic Weisbecker wrote:
> > On Wed, Mar 04, 2015 at 09:02:55AM +0900, Namhyung Kim wrote:
> > > On Tue, Mar 03, 2015 at 05:28:40PM +0100, Frederic Weisbecker wrote:
> > > > On Tue, Mar 03, 2015 at 12:07:24PM +0900, Namhyung Kim wrote:
> > > > > When data file indexing is enabled, it processes all task, comm and mmap
> > > > > events first and then goes to the sample events.  So all it sees is the
> > > > > last comm of a thread although it has information at the time of sample.

> > > > > Sort thread's comm by time so that it can find appropriate comm at the
> > > > > sample time.  The thread__comm_time() will mostly work even if
> > > > > PERF_SAMPLE_TIME bit is off since in that case, sample->time will be
> > > > > -1 so it'll take the last comm anyway.

> > > > > +++ b/tools/perf/util/thread.c
> > > > > @@ -103,6 +103,21 @@ struct comm *thread__exec_comm(const struct thread *thread)
> > > > >  	return last;
> > > > > +struct comm *thread__comm_time(const struct thread *thread, u64 timestamp)

> > > > Usually thread__comm_foo() would suggest that we return the "foo" from a thread comm.
> > > > For example thread__comm_len() returns the len of the last thread comm.
> > > > thread__comm_str() returns the string of the last thread comm.

> > > Ah, okay.

> > I mean, that's just an impression, others may have a different one :o)
> 
> Right.  Although I agree with your idea of function naming, I'm not
> sure it's worth changing every function call site for this - and for
> similar machine__find(new)_thread()_time.

So you need to find a thread using a new search key, namely (pid, tid,
time), is that right?

I.e. you process all the meta events so that you can have the lifetime
events for the whole session, right?

If that is the case, then I think we should keep the existing functions
as is, for tools not yet converted to this new way of processing
samples, be it just because there was no time yet to do it or because we
may find that it is not adequate for some tools, and introduce the new
API, that takes the time as part of the key, so, in addition to:


	machine__find_thread
	machine__findnew_thread

namely:

	machine__find_thread_by_time
	machine__findnew_thread_by_time

But thread__comm_time() for me is to get the time of most recent comm
for that thread. I.e. I agree with Frédéric, if what you want is to
_find_ the comm for a specific key, i.e. time, then I think it should
be:

        thread__find_comm_by_time(thread, tstamp)

Ah, how do you manage the thread rbtree? First a search by pid/tid and
then another by time? Or put it all together and then
machine__find_thread becomes just a:

struct thread *machine__find_thread(machine, pid, tid)
{
	return machine__find_thread_by_time(machine, pid, tid, -1);
}

And it returns the most recent thread for that pid/tid?

I.e.  machine__remove_thread() becomes a noop for such use cases and we
never delete a struct thread instance?

- Arnaldo

^ permalink raw reply	[flat|nested] 221+ messages in thread

* Re: [PATCH 12/38] perf tools: Introduce thread__comm_time() helpers
  2015-03-06 12:48           ` Arnaldo Carvalho de Melo
@ 2015-03-10  6:55             ` Namhyung Kim
  0 siblings, 0 replies; 221+ messages in thread
From: Namhyung Kim @ 2015-03-10  6:55 UTC (permalink / raw)
  To: Arnaldo Carvalho de Melo
  Cc: Frederic Weisbecker, Ingo Molnar, Peter Zijlstra, Jiri Olsa,
	LKML, Adrian Hunter, Stephane Eranian, Andi Kleen, David Ahern

Hi Arnaldo,

On Fri, Mar 06, 2015 at 09:48:05AM -0300, Arnaldo Carvalho de Melo wrote:
> Em Fri, Mar 06, 2015 at 01:38:06PM +0900, Namhyung Kim escreveu:
> > On Thu, Mar 05, 2015 at 05:08:56PM +0100, Frederic Weisbecker wrote:
> > > On Wed, Mar 04, 2015 at 09:02:55AM +0900, Namhyung Kim wrote:
> > > > On Tue, Mar 03, 2015 at 05:28:40PM +0100, Frederic Weisbecker wrote:
> > > > > On Tue, Mar 03, 2015 at 12:07:24PM +0900, Namhyung Kim wrote:
> > > > > > When data file indexing is enabled, it processes all task, comm and mmap
> > > > > > events first and then goes to the sample events.  So all it sees is the
> > > > > > last comm of a thread although it has information at the time of sample.
> 
> > > > > > Sort thread's comm by time so that it can find appropriate comm at the
> > > > > > sample time.  The thread__comm_time() will mostly work even if
> > > > > > PERF_SAMPLE_TIME bit is off since in that case, sample->time will be
> > > > > > -1 so it'll take the last comm anyway.
> 
> > > > > > +++ b/tools/perf/util/thread.c
> > > > > > @@ -103,6 +103,21 @@ struct comm *thread__exec_comm(const struct thread *thread)
> > > > > >  	return last;
> > > > > > +struct comm *thread__comm_time(const struct thread *thread, u64 timestamp)
> 
> > > > > Usually thread__comm_foo() would suggest that we return the "foo" from a thread comm.
> > > > > For example thread__comm_len() returns the len of the last thread comm.
> > > > > thread__comm_str() returns the string of the last thread comm.
> 
> > > > Ah, okay.
> 
> > > I mean, that's just an impression, others may have a different one :o)
> > 
> > Right.  Although I agree with your idea of function naming, I'm not
> > sure it's worth changing every function call site for this - and for
> > similar machine__find(new)_thread()_time.
> 
> So you need to find a thread using a new search key, namely (pid, tid,
> time), is that right?

Right.

> 
> I.e. you process all the meta events so that you can have the lifetime
> events for the whole session, right?

Exactly. :)


> 
> If that is the case, then I think we should keep the existing functions
> as is, for tools not yet converted to this new way of processing
> samples, be it just because there was no time yet to do it or because we
> may find that it is not adequate for some tools, and introduce the new
> API, that takes the time as part of the key, so, in addition to:
> 
> 
> 	machine__find_thread
> 	machine__findnew_thread
> 
> namely:
> 
> 	machine__find_thread_by_time
> 	machine__findnew_thread_by_time
> 
> But thread__comm_time() for me is to get the time of most recent comm
> for that thread. I.e. I agree with Frédéric, if what you want is to
> _find_ the comm for a specific key, i.e. time, then I think it should
> be:
> 
>         thread__find_comm_by_time(thread, tstamp)

I'm okay with it, will change!


> 
> Ah, how do you manage the thread rbtree? First a search by pid/tid and
> then another by time? Or put it all together and then
> machine__find_thread becomes just a:
> 
> struct thread *machine__find_thread(machine, pid, tid)
> {
> 	return machine__find_thread_by_time(machine, pid, tid, -1);
> }
> 
> And it returns the most recent thread for that pid/tid?

Yeah, but there's an issue with multi-thread access.  The find_thread
API is good but findnew_thread API is a problem since we need to
protect creation of new thread from multiple concurrent accesses.  So
basically it needs a lock but since it's in a hotpath it'd hurt the
performance IMHO.

So I separate the API - existing functions should be called in a
serial execution only.  The new APIs allow concurrent accesses in that
it searches (current) thread tree first and then searches dead thread
tree if not found.  Almost all of thread search should success at this
stage.  Since these operations don't change anything, they don't
require locking.

For occasional search failure, it introduced a new tree (called
'missing tree') protected with a lock.  Threads are added only if
they're not found in the normal (current + dead) tree.


> 
> I.e.  machine__remove_thread() becomes a noop for such use cases and we
> never delete a struct thread instance?

Right.  I just converted to keep removed threads in a rbtree as
they can be quite large in a long session.

Thanks,
Namhyung


> 
> - Arnaldo
> --
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/

^ permalink raw reply	[flat|nested] 221+ messages in thread

* [RFC/PATCHSET 00/40] perf tools: Speed-up perf report by using multi thread (v4)
@ 2015-05-18  0:30 Namhyung Kim
  2015-05-18  0:30 ` [PATCH 01/40] perf tools: Use a software dummy event to track task/mmap events Namhyung Kim
                   ` (39 more replies)
  0 siblings, 40 replies; 221+ messages in thread
From: Namhyung Kim @ 2015-05-18  0:30 UTC (permalink / raw)
  To: Arnaldo Carvalho de Melo
  Cc: Ingo Molnar, Peter Zijlstra, Jiri Olsa, LKML, David Ahern,
	Adrian Hunter, Andi Kleen, Frederic Weisbecker, Stephane Eranian

Hello,

This patchset converts perf report to use multiple threads in order to
speed up the processing on large data files.  I can see a minimum ~30%
of speedup with this change.  The code is still experimental and
contains many rough edges.  But I'd like to share and give some
feedbacks.

 * changes in v4)
  - rename to *_find(new)_by_time()
  - also sort struct map by time
  - use 'perf_has_index' global variable
  - fix an off-by-one bug in index generation
  - rebased onto the recent atomic thread refcounting
  - remove missing threads tree in favor of thread rwlock
  
 * changes in v3)
  - handle header (metadata) same as sample data (at index 0)
  - maintain libunwind address space in map_groups instead of thread
  - use *_time API only for indexed data file
  - resolve callchain with the *_time API
  - use dso__data_get/put_fd() to protect access to fd
  - synthesize COMM event for command line workload

 * changes in v2)
  - rework with single indexed data file rather than multiple files in
    a directory

The perf report processes (sample) events like below:

  1. preprocess sample to get matching thread/dso/symbol info
  2. insert it to hists rbtree (with callchain tree) based on the info
  3. optionally collapse hist entries that match given sort key(s)
  4. resort hist entries (by overhead) for output
  5. display the hist entries

The stage 1 is a preprocessing and mostly act like a read-only
operation in that it doesn't change a machine state during the sample
processing.  Meta events like fork, comm and mmap can change the
machine/thread state but symbols can be loaded during the processing
(stage 2).

The stage 2 consumes most of the time especially with callchains and
 --children option is enabled.  And this work can be easily patitioned
as each sample is independent to others.  But the resulting hists must
be combined/collapsed to a single global hists before going to further
steps.

The stage 3 is optional and only needed by certain sort keys - but
with stage 2 paralellized, it needs to be done always.

The stage 4 and 5 works on whole hists so must be done serially.

So my approach is like this:

Partially do stage 1 first - but only for meta events that changes
machine state.  To do this I add a dummy tracking event to perf record
and make it collect such meta events only.  They are saved as normal
data and processed before sample events at perf report time.

This also requires to handle multiple sample data concurrently and to
find a corresponding machine state when processing samples.  On a
large profiling session, many tasks were created and exited so pid
might be recycled (even more than once!).  To deal with it, I managed
to have thread, map_groups, map and comm in time sorted.  The only
remaining thing is symbol loading as it's done lazily when sample
requires it.

With that being done, the stage 2 can be done by multiple threads.  I
also save each sample data (per-cpu or per-thread) in separate files
during record and then merge them into a single data file with an
index table.  On perf report time, each region of sample data will be
processed by each thread.  And symbol loading is protected by a mutex
lock.

For DWARF post-unwinding, dso cache data also needs to be protected by
a lock and this caused a huge contention.  I made it to search the
rbtree speculatively first and then, if it didn't find one, search it
again under the dso lock.  Please take a look at it if it's acceptable.

The patch 1-8 are to support indexing for data file.  With --index
option, perf record will create a intermediate directory and then save
meta events and sample data to separate files.  And finally it'll
build an index table and concatenate the data files (and also remove
the intermediate direcotry).

The patch 9-24 are to manage machine and thread state using timestamp
so that it can be searched when processing samples.  The patch 25-38
are to implement parallel report.  And finally I implemented 'perf
data index' command to build an index table for a given data file.
The last patch is just to prevent segfault due to a bug in thread
refcounting and not inteneded to be merged.

This patchset didn't change perf record to use multi-thread.  But I
think it can be easily done later if needed.

Note that output has a slight difference to original version when
compared using indexed data file.  But they're mostly unresolved
symbols for callchains.

Here is the result:

This is just elapsed time measured by 'perf stat -r 5'.  Note that
overall performance in v4 slows down than v3 by ~30% possibly due to
atomic thread refcounting and rwlock acquisition.

The data file was recorded during kernel build with fp callchain and
size is 2.1GB.  The machine has 6 core with hyper-threading enabled
and I got a similar result on my laptop too.

 perf report          --children  --no-children  + --call-graph none
 		   -------------  -------------  -------------------
 current           368.290435503  122.786548856      51.269541556
 with index        339.618194440  101.383647394      35.880577271
 + --multi-thread  219.711277265   70.448753814      29.324704738

This result is with 7.7GB data file using libunwind for callchain.

 perf report          --children  --no-children  + --call-graph none
 		   -------------  -------------  -------------------
 current           213.799598463  155.297229724       6.235268506
 with index        216.429553998  156.337434110       4.771545122
 + --multi-thread   54.892308166   35.090129366       3.415513290

This result is with same file but using libdw for callchain unwind.

 perf report          --children  --no-children  + --call-graph none
 		   -------------  -------------  -------------------
 current           401.405331379  399.054998021       6.015400857
 with index        383.643272700  372.205715392       5.050819167
 + --multi-thread  176.419091057  139.776442715       3.356360850

On my archlinux system, callchain unwind using libdw is much slower
than libunwind.  I'm using elfutils version 0.161.  Also I don't know
why --children takes less time than --no-children.  Anyway we can see
the --multi-thread performance is much better for each case.

This patch is based on acme/perf/core - commit 70923bd26c73 ("perf
tools: Make flex/bison calls honour V=1").  I've just found that it's
updated to apply atomic refcounting to other structs.  But as it
requires another round of rebase, fix and testing cycles on my side,
I'd like to send out the current version before doing that.

You can get it from 'perf/threaded-v4' branch on my tree at:

  git://git.kernel.org/pub/scm/linux/kernel/git/namhyung/linux-perf.git

Please take a look and play with it.  Any comments are welcome! :)

Thanks,
Namhyung


Namhyung Kim (40):
  perf tools: Use a software dummy event to track task/mmap events
  perf tools: Add rm_rf() utility function
  perf tools: Introduce copyfile_offset() function
  perf tools: Create separate mmap for dummy tracking event
  perf tools: Introduce perf_evlist__mmap_track()
  perf tools: Add HEADER_DATA_INDEX feature
  perf tools: Handle indexed data file properly
  perf record: Add --index option for building index table
  perf report: Skip dummy tracking event
  perf tools: Introduce thread__comm(_str)_by_time() helpers
  perf tools: Add a test case for thread comm handling
  perf tools: Use thread__comm_by_time() when adding hist entries
  perf tools: Convert dead thread list into rbtree
  perf tools: Introduce machine__find*_thread_by_time()
  perf tools: Add a test case for timed thread handling
  perf tools: Reducing arguments of hist_entry_iter__add()
  perf tools: Maintain map groups list in a leader thread
  perf tools: Introduce thread__find_addr_location_by_time() and friends
  perf callchain: Use thread__find_addr_location_by_time() and friends
  perf tools: Add a test case for timed map groups handling
  perf tools: Save timestamp of a map creation
  perf tools: Introduce map_groups__{insert,find}_by_time()
  perf tools: Use map_groups__find_addr_by_time()
  perf tools: Add testcase for managing maps with time
  perf tools: Protect dso symbol loading using a mutex
  perf tools: Protect dso cache tree using dso->lock
  perf tools: Protect dso cache fd with a mutex
  perf callchain: Maintain libunwind's address space in map_groups
  perf tools: Add dso__data_get/put_fd()
  perf session: Pass struct events stats to event processing functions
  perf hists: Pass hists struct to hist_entry_iter struct
  perf tools: Move BUILD_ID_SIZE definition to perf.h
  perf session: Separate struct machines from session
  perf report: Parallelize perf report using multi-thread
  perf record: Synthesize COMM event for a command line workload
  perf tools: Fix progress ui to support multi thread
  perf report: Add --multi-thread option and config item
  perf session: Handle index files generally
  perf data: Implement 'index' subcommand
  perf tools: Disable thread refcount due to bug

 tools/perf/Documentation/perf-data.txt             |  25 +-
 tools/perf/Documentation/perf-record.txt           |   4 +
 tools/perf/Documentation/perf-report.txt           |   2 +
 tools/perf/builtin-annotate.c                      |   7 +-
 tools/perf/builtin-data.c                          | 351 +++++++++++++++++-
 tools/perf/builtin-diff.c                          |   8 +-
 tools/perf/builtin-kmem.c                          |  10 +-
 tools/perf/builtin-kvm.c                           |   2 +-
 tools/perf/builtin-record.c                        | 202 ++++++++++-
 tools/perf/builtin-report.c                        |  81 ++++-
 tools/perf/builtin-top.c                           |  16 +-
 tools/perf/builtin-trace.c                         |   2 +-
 tools/perf/perf.c                                  |   1 +
 tools/perf/perf.h                                  |   4 +
 tools/perf/tests/Build                             |   4 +
 tools/perf/tests/builtin-test.c                    |  16 +
 tools/perf/tests/dwarf-unwind.c                    |  12 +-
 tools/perf/tests/hists_common.c                    |   3 +-
 tools/perf/tests/hists_cumulate.c                  |   7 +-
 tools/perf/tests/hists_filter.c                    |   5 +-
 tools/perf/tests/hists_link.c                      |   6 +-
 tools/perf/tests/hists_output.c                    |   7 +-
 tools/perf/tests/perf-targz-src-pkg                |  21 --
 tools/perf/tests/tests.h                           |   4 +
 tools/perf/tests/thread-comm.c                     |  47 +++
 tools/perf/tests/thread-lookup-time.c              | 179 ++++++++++
 tools/perf/tests/thread-map-time.c                 |  90 +++++
 tools/perf/tests/thread-mg-share.c                 |   7 +-
 tools/perf/tests/thread-mg-time.c                  |  93 +++++
 tools/perf/ui/browsers/hists.c                     |  30 +-
 tools/perf/ui/gtk/hists.c                          |   3 +
 tools/perf/util/build-id.c                         |  16 +-
 tools/perf/util/build-id.h                         |   2 -
 tools/perf/util/dso.c                              | 174 ++++++---
 tools/perf/util/dso.h                              |  11 +-
 tools/perf/util/event.c                            | 142 +++++++-
 tools/perf/util/event.h                            |   6 +-
 tools/perf/util/evlist.c                           | 168 +++++++--
 tools/perf/util/evlist.h                           |  14 +-
 tools/perf/util/evsel.h                            |  15 +
 tools/perf/util/header.c                           |  64 ++++
 tools/perf/util/header.h                           |   3 +
 tools/perf/util/hist.c                             | 123 ++++---
 tools/perf/util/hist.h                             |   8 +-
 tools/perf/util/machine.c                          | 291 ++++++++++++---
 tools/perf/util/machine.h                          |  14 +-
 tools/perf/util/map.c                              |  72 +++-
 tools/perf/util/map.h                              |  37 +-
 tools/perf/util/probe-event.c                      |   2 +-
 .../util/scripting-engines/trace-event-python.c    |   4 +-
 tools/perf/util/session.c                          | 396 ++++++++++++++++++---
 tools/perf/util/session.h                          |   7 +-
 tools/perf/util/symbol-elf.c                       |   2 +-
 tools/perf/util/symbol.c                           |  38 +-
 tools/perf/util/thread.c                           | 207 ++++++++++-
 tools/perf/util/thread.h                           |  28 +-
 tools/perf/util/tool.h                             |  14 +
 tools/perf/util/unwind-libdw.c                     |  12 +-
 tools/perf/util/unwind-libunwind.c                 |  93 ++---
 tools/perf/util/unwind.h                           |  15 +-
 tools/perf/util/util.c                             |  81 ++++-
 tools/perf/util/util.h                             |   2 +
 62 files changed, 2856 insertions(+), 454 deletions(-)
 delete mode 100755 tools/perf/tests/perf-targz-src-pkg
 create mode 100644 tools/perf/tests/thread-comm.c
 create mode 100644 tools/perf/tests/thread-lookup-time.c
 create mode 100644 tools/perf/tests/thread-map-time.c
 create mode 100644 tools/perf/tests/thread-mg-time.c

-- 
2.4.0


^ permalink raw reply	[flat|nested] 221+ messages in thread

* [PATCH 01/40] perf tools: Use a software dummy event to track task/mmap events
  2015-05-18  0:30 [RFC/PATCHSET 00/40] perf tools: Speed-up perf report by using multi thread (v4) Namhyung Kim
@ 2015-05-18  0:30 ` Namhyung Kim
  2015-05-18  0:30 ` [PATCH 02/40] perf tools: Add rm_rf() utility function Namhyung Kim
                   ` (38 subsequent siblings)
  39 siblings, 0 replies; 221+ messages in thread
From: Namhyung Kim @ 2015-05-18  0:30 UTC (permalink / raw)
  To: Arnaldo Carvalho de Melo
  Cc: Ingo Molnar, Peter Zijlstra, Jiri Olsa, LKML, David Ahern,
	Adrian Hunter, Andi Kleen, Frederic Weisbecker, Stephane Eranian

Add APIs for software dummy event to track task/comm/mmap events
separately.  The perf record will use them to save such events in a
separate mmap buffer to make it easy to index.  This is a preparation of
multi-thread support which will come later.

Cc: Adrian Hunter <adrian.hunter@intel.com>
Signed-off-by: Namhyung Kim <namhyung@kernel.org>
---
 tools/perf/util/evlist.c | 30 ++++++++++++++++++++++++++++++
 tools/perf/util/evlist.h |  1 +
 tools/perf/util/evsel.h  | 15 +++++++++++++++
 3 files changed, 46 insertions(+)

diff --git a/tools/perf/util/evlist.c b/tools/perf/util/evlist.c
index 7ec1bf93ab28..5bbd0ea82fc4 100644
--- a/tools/perf/util/evlist.c
+++ b/tools/perf/util/evlist.c
@@ -194,6 +194,36 @@ int perf_evlist__add_default(struct perf_evlist *evlist)
 	return -ENOMEM;
 }
 
+int perf_evlist__add_dummy_tracking(struct perf_evlist *evlist)
+{
+	struct perf_event_attr attr = {
+		.type = PERF_TYPE_SOFTWARE,
+		.config = PERF_COUNT_SW_DUMMY,
+		.exclude_kernel = 1,
+	};
+	struct perf_evsel *evsel;
+
+	event_attr_init(&attr);
+
+	evsel = perf_evsel__new(&attr);
+	if (evsel == NULL)
+		goto error;
+
+	/* use strdup() because free(evsel) assumes name is allocated */
+	evsel->name = strdup("dummy");
+	if (!evsel->name)
+		goto error_free;
+
+	perf_evlist__add(evlist, evsel);
+	perf_evlist__set_tracking_event(evlist, evsel);
+
+	return 0;
+error_free:
+	perf_evsel__delete(evsel);
+error:
+	return -ENOMEM;
+}
+
 static int perf_evlist__add_attrs(struct perf_evlist *evlist,
 				  struct perf_event_attr *attrs, size_t nr_attrs)
 {
diff --git a/tools/perf/util/evlist.h b/tools/perf/util/evlist.h
index c07b1a94a724..6cc580e1a43b 100644
--- a/tools/perf/util/evlist.h
+++ b/tools/perf/util/evlist.h
@@ -70,6 +70,7 @@ void perf_evlist__delete(struct perf_evlist *evlist);
 
 void perf_evlist__add(struct perf_evlist *evlist, struct perf_evsel *entry);
 int perf_evlist__add_default(struct perf_evlist *evlist);
+int perf_evlist__add_dummy_tracking(struct perf_evlist *evlist);
 int __perf_evlist__add_default_attrs(struct perf_evlist *evlist,
 				     struct perf_event_attr *attrs, size_t nr_attrs);
 
diff --git a/tools/perf/util/evsel.h b/tools/perf/util/evsel.h
index e486151b0308..4c47ebdf3c10 100644
--- a/tools/perf/util/evsel.h
+++ b/tools/perf/util/evsel.h
@@ -331,6 +331,21 @@ static inline bool perf_evsel__is_function_event(struct perf_evsel *evsel)
 #undef FUNCTION_EVENT
 }
 
+/**
+ * perf_evsel__is_dummy_tracking - Return whether given evsel is a dummy
+ * event for tracking meta events only
+ *
+ * @evsel - evsel selector to be tested
+ *
+ * Return %true if event is a dummy tracking event
+ */
+static inline bool perf_evsel__is_dummy_tracking(struct perf_evsel *evsel)
+{
+	return evsel->attr.type == PERF_TYPE_SOFTWARE &&
+		evsel->attr.config == PERF_COUNT_SW_DUMMY &&
+		evsel->attr.task == 1 && evsel->attr.mmap == 1;
+}
+
 struct perf_attr_details {
 	bool freq;
 	bool verbose;
-- 
2.4.0


^ permalink raw reply related	[flat|nested] 221+ messages in thread

* [PATCH 02/40] perf tools: Add rm_rf() utility function
  2015-05-18  0:30 [RFC/PATCHSET 00/40] perf tools: Speed-up perf report by using multi thread (v4) Namhyung Kim
  2015-05-18  0:30 ` [PATCH 01/40] perf tools: Use a software dummy event to track task/mmap events Namhyung Kim
@ 2015-05-18  0:30 ` Namhyung Kim
  2015-05-18  0:30 ` [PATCH 03/40] perf tools: Introduce copyfile_offset() function Namhyung Kim
                   ` (37 subsequent siblings)
  39 siblings, 0 replies; 221+ messages in thread
From: Namhyung Kim @ 2015-05-18  0:30 UTC (permalink / raw)
  To: Arnaldo Carvalho de Melo
  Cc: Ingo Molnar, Peter Zijlstra, Jiri Olsa, LKML, David Ahern,
	Adrian Hunter, Andi Kleen, Frederic Weisbecker, Stephane Eranian

The rm_rf() function does same as the shell command 'rm -rf' which
removes all directory entries recursively.

Signed-off-by: Namhyung Kim <namhyung@kernel.org>
---
 tools/perf/util/util.c | 43 +++++++++++++++++++++++++++++++++++++++++++
 tools/perf/util/util.h |  1 +
 2 files changed, 44 insertions(+)

diff --git a/tools/perf/util/util.c b/tools/perf/util/util.c
index 4ee6d0d4c993..6104afb7e1ef 100644
--- a/tools/perf/util/util.c
+++ b/tools/perf/util/util.c
@@ -72,6 +72,49 @@ int mkdir_p(char *path, mode_t mode)
 	return (stat(path, &st) && mkdir(path, mode)) ? -1 : 0;
 }
 
+int rm_rf(char *path)
+{
+	DIR *dir;
+	int ret = 0;
+	struct dirent *d;
+	char namebuf[PATH_MAX];
+
+	dir = opendir(path);
+	if (dir == NULL)
+		return 0;
+
+	while ((d = readdir(dir)) != NULL && !ret) {
+		struct stat statbuf;
+
+		if (!strcmp(d->d_name, ".") || !strcmp(d->d_name, ".."))
+			continue;
+
+		scnprintf(namebuf, sizeof(namebuf), "%s/%s",
+			  path, d->d_name);
+
+		ret = stat(namebuf, &statbuf);
+		if (ret < 0) {
+			pr_debug("stat failed: %s\n", namebuf);
+			break;
+		}
+
+		if (S_ISREG(statbuf.st_mode))
+			ret = unlink(namebuf);
+		else if (S_ISDIR(statbuf.st_mode))
+			ret = rm_rf(namebuf);
+		else {
+			pr_debug("unknown file: %s\n", namebuf);
+			ret = -1;
+		}
+	}
+	closedir(dir);
+
+	if (ret < 0)
+		return ret;
+
+	return rmdir(path);
+}
+
 static int slow_copyfile(const char *from, const char *to, mode_t mode)
 {
 	int err = -1;
diff --git a/tools/perf/util/util.h b/tools/perf/util/util.h
index 3601ffd3d8b4..c4fe38ac8c00 100644
--- a/tools/perf/util/util.h
+++ b/tools/perf/util/util.h
@@ -249,6 +249,7 @@ static inline int sane_case(int x, int high)
 }
 
 int mkdir_p(char *path, mode_t mode);
+int rm_rf(char *path);
 int copyfile(const char *from, const char *to);
 int copyfile_mode(const char *from, const char *to, mode_t mode);
 
-- 
2.4.0


^ permalink raw reply related	[flat|nested] 221+ messages in thread

* [PATCH 03/40] perf tools: Introduce copyfile_offset() function
  2015-05-18  0:30 [RFC/PATCHSET 00/40] perf tools: Speed-up perf report by using multi thread (v4) Namhyung Kim
  2015-05-18  0:30 ` [PATCH 01/40] perf tools: Use a software dummy event to track task/mmap events Namhyung Kim
  2015-05-18  0:30 ` [PATCH 02/40] perf tools: Add rm_rf() utility function Namhyung Kim
@ 2015-05-18  0:30 ` Namhyung Kim
  2015-05-18 12:57   ` Arnaldo Carvalho de Melo
  2015-05-20 12:24   ` [tip:perf/core] " tip-bot for Namhyung Kim
  2015-05-18  0:30 ` [PATCH 04/40] perf tools: Create separate mmap for dummy tracking event Namhyung Kim
                   ` (36 subsequent siblings)
  39 siblings, 2 replies; 221+ messages in thread
From: Namhyung Kim @ 2015-05-18  0:30 UTC (permalink / raw)
  To: Arnaldo Carvalho de Melo
  Cc: Ingo Molnar, Peter Zijlstra, Jiri Olsa, LKML, David Ahern,
	Adrian Hunter, Andi Kleen, Frederic Weisbecker, Stephane Eranian

The copyfile_offset() function is to copy source data from given
offset to a destination file with an offset.  It'll be used to build
an indexed data file.

Signed-off-by: Namhyung Kim <namhyung@kernel.org>
---
 tools/perf/util/util.c | 38 +++++++++++++++++++++++++++++---------
 tools/perf/util/util.h |  1 +
 2 files changed, 30 insertions(+), 9 deletions(-)

diff --git a/tools/perf/util/util.c b/tools/perf/util/util.c
index 6104afb7e1ef..0c264bc685ac 100644
--- a/tools/perf/util/util.c
+++ b/tools/perf/util/util.c
@@ -145,11 +145,38 @@ static int slow_copyfile(const char *from, const char *to, mode_t mode)
 	return err;
 }
 
+int copyfile_offset(int ifd, loff_t off_in, int ofd, loff_t off_out, u64 size)
+{
+	void *ptr;
+	loff_t pgoff;
+
+	pgoff = off_in & ~(page_size - 1);
+	off_in -= pgoff;
+
+	ptr = mmap(NULL, off_in + size, PROT_READ, MAP_PRIVATE, ifd, pgoff);
+	if (ptr == MAP_FAILED)
+		return -1;
+
+	while (size) {
+		ssize_t ret = pwrite(ofd, ptr + off_in, size, off_out);
+		if (ret < 0 && errno == EINTR)
+			continue;
+		if (ret <= 0)
+			break;
+
+		size -= ret;
+		off_in += ret;
+		off_out -= ret;
+	}
+	munmap(ptr, off_in + size);
+
+	return size ? -1 : 0;
+}
+
 int copyfile_mode(const char *from, const char *to, mode_t mode)
 {
 	int fromfd, tofd;
 	struct stat st;
-	void *addr;
 	int err = -1;
 
 	if (stat(from, &st))
@@ -166,15 +193,8 @@ int copyfile_mode(const char *from, const char *to, mode_t mode)
 	if (tofd < 0)
 		goto out_close_from;
 
-	addr = mmap(NULL, st.st_size, PROT_READ, MAP_PRIVATE, fromfd, 0);
-	if (addr == MAP_FAILED)
-		goto out_close_to;
-
-	if (write(tofd, addr, st.st_size) == st.st_size)
-		err = 0;
+	err = copyfile_offset(fromfd, 0, tofd, 0, st.st_size);
 
-	munmap(addr, st.st_size);
-out_close_to:
 	close(tofd);
 	if (err)
 		unlink(to);
diff --git a/tools/perf/util/util.h b/tools/perf/util/util.h
index c4fe38ac8c00..8bce58b47a82 100644
--- a/tools/perf/util/util.h
+++ b/tools/perf/util/util.h
@@ -252,6 +252,7 @@ int mkdir_p(char *path, mode_t mode);
 int rm_rf(char *path);
 int copyfile(const char *from, const char *to);
 int copyfile_mode(const char *from, const char *to, mode_t mode);
+int copyfile_offset(int fromfd, loff_t from_ofs, int tofd, loff_t to_ofs, u64 size);
 
 s64 perf_atoll(const char *str);
 char **argv_split(const char *str, int *argcp);
-- 
2.4.0


^ permalink raw reply related	[flat|nested] 221+ messages in thread

* [PATCH 04/40] perf tools: Create separate mmap for dummy tracking event
  2015-05-18  0:30 [RFC/PATCHSET 00/40] perf tools: Speed-up perf report by using multi thread (v4) Namhyung Kim
                   ` (2 preceding siblings ...)
  2015-05-18  0:30 ` [PATCH 03/40] perf tools: Introduce copyfile_offset() function Namhyung Kim
@ 2015-05-18  0:30 ` Namhyung Kim
  2015-05-18 18:07   ` Jiri Olsa
  2015-05-18  0:30 ` [PATCH 05/40] perf tools: Introduce perf_evlist__mmap_track() Namhyung Kim
                   ` (35 subsequent siblings)
  39 siblings, 1 reply; 221+ messages in thread
From: Namhyung Kim @ 2015-05-18  0:30 UTC (permalink / raw)
  To: Arnaldo Carvalho de Melo
  Cc: Ingo Molnar, Peter Zijlstra, Jiri Olsa, LKML, David Ahern,
	Adrian Hunter, Andi Kleen, Frederic Weisbecker, Stephane Eranian

When indexed data file support is enabled, a dummy tracking event will
be used to track metadata (like task, comm and mmap events) for a
session and actual samples will be recorded in separate (intermediate)
files and then merged (with index table).

Provide separate mmap to the dummy tracking event.  The size is fixed
to 128KiB (+ 1 page) as the event rate will be lower than samples.  I
originally wanted to use a single mmap for this but cross-cpu sharing
is prohibited so it's per-cpu (or per-task) like normal mmaps.

Cc: Adrian Hunter <adrian.hunter@intel.com>
Signed-off-by: Namhyung Kim <namhyung@kernel.org>
---
 tools/perf/builtin-record.c |  10 +++-
 tools/perf/util/evlist.c    | 131 +++++++++++++++++++++++++++++++++++---------
 tools/perf/util/evlist.h    |  11 +++-
 3 files changed, 123 insertions(+), 29 deletions(-)

diff --git a/tools/perf/builtin-record.c b/tools/perf/builtin-record.c
index 5dfe91395617..7d7ef5a1b0a6 100644
--- a/tools/perf/builtin-record.c
+++ b/tools/perf/builtin-record.c
@@ -71,7 +71,7 @@ static int process_synthesized_event(struct perf_tool *tool,
 
 static int record__mmap_read(struct record *rec, int idx)
 {
-	struct perf_mmap *md = &rec->evlist->mmap[idx];
+	struct perf_mmap *md = perf_evlist__mmap_desc(rec->evlist, idx);
 	u64 head = perf_mmap__read_head(md);
 	u64 old = md->prev;
 	unsigned char *data = md->base + page_size;
@@ -107,6 +107,7 @@ static int record__mmap_read(struct record *rec, int idx)
 	}
 
 	md->prev = old;
+
 	perf_evlist__mmap_consume(rec->evlist, idx);
 out:
 	return rc;
@@ -414,6 +415,13 @@ static int record__mmap_read_all(struct record *rec)
 			}
 		}
 
+		if (rec->evlist->track_mmap[i].base) {
+			if (record__mmap_read(rec, track_mmap_idx(i)) != 0) {
+				rc = -1;
+				goto out;
+			}
+		}
+
 		if (mm->base && !rec->opts.auxtrace_snapshot_mode &&
 		    record__auxtrace_mmap_read(rec, mm) != 0) {
 			rc = -1;
diff --git a/tools/perf/util/evlist.c b/tools/perf/util/evlist.c
index 5bbd0ea82fc4..5db249eb4dcd 100644
--- a/tools/perf/util/evlist.c
+++ b/tools/perf/util/evlist.c
@@ -28,6 +28,7 @@
 
 static void perf_evlist__mmap_put(struct perf_evlist *evlist, int idx);
 static void __perf_evlist__munmap(struct perf_evlist *evlist, int idx);
+static void __perf_evlist__munmap_track(struct perf_evlist *evlist, int idx);
 
 #define FD(e, x, y) (*(int *)xyarray__entry(e->fd, x, y))
 #define SID(e, x, y) xyarray__entry(e->sample_id, x, y)
@@ -728,22 +729,39 @@ static bool perf_mmap__empty(struct perf_mmap *md)
 	return perf_mmap__read_head(md) == md->prev && !md->auxtrace_mmap.base;
 }
 
+struct perf_mmap *perf_evlist__mmap_desc(struct perf_evlist *evlist, int idx)
+{
+	if (idx >= 0)
+		return &evlist->mmap[idx];
+	else
+		return &evlist->track_mmap[track_mmap_idx(idx)];
+}
+
 static void perf_evlist__mmap_get(struct perf_evlist *evlist, int idx)
 {
-	++evlist->mmap[idx].refcnt;
+	struct perf_mmap *md = perf_evlist__mmap_desc(evlist, idx);
+
+	++md->refcnt;
 }
 
 static void perf_evlist__mmap_put(struct perf_evlist *evlist, int idx)
 {
-	BUG_ON(evlist->mmap[idx].refcnt == 0);
+	struct perf_mmap *md = perf_evlist__mmap_desc(evlist, idx);
 
-	if (--evlist->mmap[idx].refcnt == 0)
+	BUG_ON(md->refcnt == 0);
+
+	if (--md->refcnt != 0)
+		return;
+
+	if (idx >= 0)
 		__perf_evlist__munmap(evlist, idx);
+	else
+		__perf_evlist__munmap_track(evlist, track_mmap_idx(idx));
 }
 
 void perf_evlist__mmap_consume(struct perf_evlist *evlist, int idx)
 {
-	struct perf_mmap *md = &evlist->mmap[idx];
+	struct perf_mmap *md = perf_evlist__mmap_desc(evlist, idx);
 
 	if (!evlist->overwrite) {
 		u64 old = md->prev;
@@ -793,6 +811,15 @@ static void __perf_evlist__munmap(struct perf_evlist *evlist, int idx)
 	auxtrace_mmap__munmap(&evlist->mmap[idx].auxtrace_mmap);
 }
 
+static void __perf_evlist__munmap_track(struct perf_evlist *evlist, int idx)
+{
+	if (evlist->track_mmap[idx].base != NULL) {
+		munmap(evlist->track_mmap[idx].base, TRACK_MMAP_SIZE);
+		evlist->track_mmap[idx].base = NULL;
+		evlist->track_mmap[idx].refcnt = 0;
+	}
+}
+
 void perf_evlist__munmap(struct perf_evlist *evlist)
 {
 	int i;
@@ -804,24 +831,44 @@ void perf_evlist__munmap(struct perf_evlist *evlist)
 		__perf_evlist__munmap(evlist, i);
 
 	zfree(&evlist->mmap);
+
+	if (evlist->track_mmap == NULL)
+		return;
+
+	for (i = 0; i < evlist->nr_mmaps; i++)
+		__perf_evlist__munmap_track(evlist, i);
+
+	zfree(&evlist->track_mmap);
 }
 
-static int perf_evlist__alloc_mmap(struct perf_evlist *evlist)
+static int perf_evlist__alloc_mmap(struct perf_evlist *evlist, bool track_mmap)
 {
 	evlist->nr_mmaps = cpu_map__nr(evlist->cpus);
 	if (cpu_map__empty(evlist->cpus))
 		evlist->nr_mmaps = thread_map__nr(evlist->threads);
 	evlist->mmap = zalloc(evlist->nr_mmaps * sizeof(struct perf_mmap));
-	return evlist->mmap != NULL ? 0 : -ENOMEM;
+	if (evlist->mmap == NULL)
+		return -ENOMEM;
+
+	if (track_mmap) {
+		evlist->track_mmap = calloc(evlist->nr_mmaps,
+					    sizeof(struct perf_mmap));
+		if (evlist->track_mmap == NULL) {
+			zfree(&evlist->mmap);
+			return -ENOMEM;
+		}
+	}
+	return 0;
 }
 
 struct mmap_params {
-	int prot;
-	int mask;
+	int	prot;
+	size_t	len;
 	struct auxtrace_mmap_params auxtrace_mp;
 };
 
-static int __perf_evlist__mmap(struct perf_evlist *evlist, int idx,
+static int __perf_evlist__mmap(struct perf_evlist *evlist __maybe_unused,
+			       struct perf_mmap *pmmap,
 			       struct mmap_params *mp, int fd)
 {
 	/*
@@ -837,28 +884,24 @@ static int __perf_evlist__mmap(struct perf_evlist *evlist, int idx,
 	 * evlist layer can't just drop it when filtering events in
 	 * perf_evlist__filter_pollfd().
 	 */
-	evlist->mmap[idx].refcnt = 2;
-	evlist->mmap[idx].prev = 0;
-	evlist->mmap[idx].mask = mp->mask;
-	evlist->mmap[idx].base = mmap(NULL, evlist->mmap_len, mp->prot,
-				      MAP_SHARED, fd, 0);
-	if (evlist->mmap[idx].base == MAP_FAILED) {
+	pmmap->refcnt = 2;
+	pmmap->prev = 0;
+	pmmap->mask = mp->len - page_size - 1;
+	pmmap->base = mmap(NULL, mp->len, mp->prot, MAP_SHARED, fd, 0);
+	if (pmmap->base == MAP_FAILED) {
 		pr_debug2("failed to mmap perf event ring buffer, error %d\n",
 			  errno);
-		evlist->mmap[idx].base = NULL;
+		pmmap->base = NULL;
 		return -1;
 	}
 
-	if (auxtrace_mmap__mmap(&evlist->mmap[idx].auxtrace_mmap,
-				&mp->auxtrace_mp, evlist->mmap[idx].base, fd))
-		return -1;
-
 	return 0;
 }
 
 static int perf_evlist__mmap_per_evsel(struct perf_evlist *evlist, int idx,
 				       struct mmap_params *mp, int cpu,
-				       int thread, int *output)
+				       int thread, int *output,
+				       int *track_output)
 {
 	struct perf_evsel *evsel;
 
@@ -870,9 +913,35 @@ static int perf_evlist__mmap_per_evsel(struct perf_evlist *evlist, int idx,
 
 		fd = FD(evsel, cpu, thread);
 
-		if (*output == -1) {
+		if (perf_evsel__is_dummy_tracking(evsel)) {
+			struct mmap_params track_mp = {
+				.prot	= mp->prot,
+				.len	= TRACK_MMAP_SIZE,
+			};
+
+			if (*track_output == -1) {
+				*track_output = fd;
+				if (__perf_evlist__mmap(evlist,
+							&evlist->track_mmap[idx],
+							&track_mp, fd) < 0)
+					return -1;
+			} else {
+				if (ioctl(fd, PERF_EVENT_IOC_SET_OUTPUT,
+					  *track_output) != 0)
+					return -1;
+			}
+
+			/* mark idx as track mmap idx (negative) */
+			idx = track_mmap_idx(idx);
+		} else if (*output == -1) {
 			*output = fd;
-			if (__perf_evlist__mmap(evlist, idx, mp, *output) < 0)
+			if (__perf_evlist__mmap(evlist, &evlist->mmap[idx],
+						mp, *output) < 0)
+				return -1;
+
+			if (auxtrace_mmap__mmap(&evlist->mmap[idx].auxtrace_mmap,
+						&mp->auxtrace_mp,
+						evlist->mmap[idx].base, fd))
 				return -1;
 		} else {
 			if (ioctl(fd, PERF_EVENT_IOC_SET_OUTPUT, *output) != 0)
@@ -901,6 +970,11 @@ static int perf_evlist__mmap_per_evsel(struct perf_evlist *evlist, int idx,
 			perf_evlist__set_sid_idx(evlist, evsel, idx, cpu,
 						 thread);
 		}
+
+		if (perf_evsel__is_dummy_tracking(evsel)) {
+			/* restore idx as normal idx (positive) */
+			idx = track_mmap_idx(idx);
+		}
 	}
 
 	return 0;
@@ -916,13 +990,15 @@ static int perf_evlist__mmap_per_cpu(struct perf_evlist *evlist,
 	pr_debug2("perf event ring buffer mmapped per cpu\n");
 	for (cpu = 0; cpu < nr_cpus; cpu++) {
 		int output = -1;
+		int track_output = -1;
 
 		auxtrace_mmap_params__set_idx(&mp->auxtrace_mp, evlist, cpu,
 					      true);
 
 		for (thread = 0; thread < nr_threads; thread++) {
 			if (perf_evlist__mmap_per_evsel(evlist, cpu, mp, cpu,
-							thread, &output))
+							thread, &output,
+							&track_output))
 				goto out_unmap;
 		}
 	}
@@ -944,12 +1020,13 @@ static int perf_evlist__mmap_per_thread(struct perf_evlist *evlist,
 	pr_debug2("perf event ring buffer mmapped per thread\n");
 	for (thread = 0; thread < nr_threads; thread++) {
 		int output = -1;
+		int track_output = -1;
 
 		auxtrace_mmap_params__set_idx(&mp->auxtrace_mp, evlist, thread,
 					      false);
 
 		if (perf_evlist__mmap_per_evsel(evlist, thread, mp, 0, thread,
-						&output))
+						&output, &track_output))
 			goto out_unmap;
 	}
 
@@ -1082,7 +1159,7 @@ int perf_evlist__mmap_ex(struct perf_evlist *evlist, unsigned int pages,
 		.prot = PROT_READ | (overwrite ? 0 : PROT_WRITE),
 	};
 
-	if (evlist->mmap == NULL && perf_evlist__alloc_mmap(evlist) < 0)
+	if (evlist->mmap == NULL && perf_evlist__alloc_mmap(evlist, true) < 0)
 		return -ENOMEM;
 
 	if (evlist->pollfd.entries == NULL && perf_evlist__alloc_pollfd(evlist) < 0)
@@ -1091,7 +1168,7 @@ int perf_evlist__mmap_ex(struct perf_evlist *evlist, unsigned int pages,
 	evlist->overwrite = overwrite;
 	evlist->mmap_len = perf_evlist__mmap_size(pages);
 	pr_debug("mmap size %zuB\n", evlist->mmap_len);
-	mp.mask = evlist->mmap_len - page_size - 1;
+	mp.len = evlist->mmap_len;
 
 	auxtrace_mmap_params__init(&mp.auxtrace_mp, evlist->mmap_len,
 				   auxtrace_pages, auxtrace_overwrite);
diff --git a/tools/perf/util/evlist.h b/tools/perf/util/evlist.h
index 6cc580e1a43b..335988577c18 100644
--- a/tools/perf/util/evlist.h
+++ b/tools/perf/util/evlist.h
@@ -50,12 +50,15 @@ struct perf_evlist {
 	bool		 overwrite;
 	struct fdarray	 pollfd;
 	struct perf_mmap *mmap;
+	struct perf_mmap *track_mmap;
 	struct thread_map *threads;
 	struct cpu_map	  *cpus;
 	struct perf_evsel *selected;
 	struct events_stats stats;
 };
 
+#define TRACK_MMAP_SIZE  (((128 * 1024 / page_size) + 1) * page_size)
+
 struct perf_evsel_str_handler {
 	const char *name;
 	void	   *handler;
@@ -105,8 +108,8 @@ struct perf_evsel *perf_evlist__id2evsel(struct perf_evlist *evlist, u64 id);
 struct perf_sample_id *perf_evlist__id2sid(struct perf_evlist *evlist, u64 id);
 
 union perf_event *perf_evlist__mmap_read(struct perf_evlist *evlist, int idx);
-
 void perf_evlist__mmap_consume(struct perf_evlist *evlist, int idx);
+struct perf_mmap *perf_evlist__mmap_desc(struct perf_evlist *evlist, int idx);
 
 int perf_evlist__open(struct perf_evlist *evlist);
 void perf_evlist__close(struct perf_evlist *evlist);
@@ -219,6 +222,12 @@ bool perf_evlist__can_select_event(struct perf_evlist *evlist, const char *str);
 void perf_evlist__to_front(struct perf_evlist *evlist,
 			   struct perf_evsel *move_evsel);
 
+/* convert from/to negative idx for track mmaps */
+static inline int track_mmap_idx(int idx)
+{
+	return -idx - 1;
+}
+
 /**
  * __evlist__for_each - iterate thru all the evsels
  * @list: list_head instance to iterate
-- 
2.4.0


^ permalink raw reply related	[flat|nested] 221+ messages in thread

* [PATCH 05/40] perf tools: Introduce perf_evlist__mmap_track()
  2015-05-18  0:30 [RFC/PATCHSET 00/40] perf tools: Speed-up perf report by using multi thread (v4) Namhyung Kim
                   ` (3 preceding siblings ...)
  2015-05-18  0:30 ` [PATCH 04/40] perf tools: Create separate mmap for dummy tracking event Namhyung Kim
@ 2015-05-18  0:30 ` Namhyung Kim
  2015-05-18 18:09   ` Jiri Olsa
  2015-05-18  0:30 ` [PATCH 06/40] perf tools: Add HEADER_DATA_INDEX feature Namhyung Kim
                   ` (34 subsequent siblings)
  39 siblings, 1 reply; 221+ messages in thread
From: Namhyung Kim @ 2015-05-18  0:30 UTC (permalink / raw)
  To: Arnaldo Carvalho de Melo
  Cc: Ingo Molnar, Peter Zijlstra, Jiri Olsa, LKML, David Ahern,
	Adrian Hunter, Andi Kleen, Frederic Weisbecker, Stephane Eranian

The perf_evlist__mmap_track function creates data mmaps and optionally
tracking mmaps for events.  It'll be used for perf record to save events
in a separate files and build an index table.  Checking dummy tracking
event in perf_evlist__mmap() alone is not enough as users can specify a
dummy event (like in keep tracking testcase) without the index option.

Cc: Adrian Hunter <adrian.hunter@intel.com>
Signed-off-by: Namhyung Kim <namhyung@kernel.org>
---
 tools/perf/builtin-record.c |  2 +-
 tools/perf/util/evlist.c    | 13 ++++++++-----
 tools/perf/util/evlist.h    |  2 +-
 3 files changed, 10 insertions(+), 7 deletions(-)

diff --git a/tools/perf/builtin-record.c b/tools/perf/builtin-record.c
index 7d7ef5a1b0a6..21f7edb23370 100644
--- a/tools/perf/builtin-record.c
+++ b/tools/perf/builtin-record.c
@@ -304,7 +304,7 @@ static int record__open(struct record *rec)
 
 	if (perf_evlist__mmap_ex(evlist, opts->mmap_pages, false,
 				 opts->auxtrace_mmap_pages,
-				 opts->auxtrace_snapshot_mode) < 0) {
+				 opts->auxtrace_snapshot_mode, false) < 0) {
 		if (errno == EPERM) {
 			pr_err("Permission error mapping pages.\n"
 			       "Consider increasing "
diff --git a/tools/perf/util/evlist.c b/tools/perf/util/evlist.c
index 5db249eb4dcd..303249467672 100644
--- a/tools/perf/util/evlist.c
+++ b/tools/perf/util/evlist.c
@@ -863,6 +863,7 @@ static int perf_evlist__alloc_mmap(struct perf_evlist *evlist, bool track_mmap)
 
 struct mmap_params {
 	int	prot;
+	bool	track;
 	size_t	len;
 	struct auxtrace_mmap_params auxtrace_mp;
 };
@@ -913,7 +914,7 @@ static int perf_evlist__mmap_per_evsel(struct perf_evlist *evlist, int idx,
 
 		fd = FD(evsel, cpu, thread);
 
-		if (perf_evsel__is_dummy_tracking(evsel)) {
+		if (mp->track && perf_evsel__is_dummy_tracking(evsel)) {
 			struct mmap_params track_mp = {
 				.prot	= mp->prot,
 				.len	= TRACK_MMAP_SIZE,
@@ -971,7 +972,7 @@ static int perf_evlist__mmap_per_evsel(struct perf_evlist *evlist, int idx,
 						 thread);
 		}
 
-		if (perf_evsel__is_dummy_tracking(evsel)) {
+		if (mp->track && perf_evsel__is_dummy_tracking(evsel)) {
 			/* restore idx as normal idx (positive) */
 			idx = track_mmap_idx(idx);
 		}
@@ -1138,6 +1139,7 @@ int perf_evlist__parse_mmap_pages(const struct option *opt, const char *str,
  * @overwrite: overwrite older events?
  * @auxtrace_pages - auxtrace map length in pages
  * @auxtrace_overwrite - overwrite older auxtrace data?
+ * @use_track_mmap: use another mmaps to track meta events
  *
  * If @overwrite is %false the user needs to signal event consumption using
  * perf_mmap__write_tail().  Using perf_evlist__mmap_read() does this
@@ -1150,16 +1152,17 @@ int perf_evlist__parse_mmap_pages(const struct option *opt, const char *str,
  */
 int perf_evlist__mmap_ex(struct perf_evlist *evlist, unsigned int pages,
 			 bool overwrite, unsigned int auxtrace_pages,
-			 bool auxtrace_overwrite)
+			 bool auxtrace_overwrite, bool use_track_mmap)
 {
 	struct perf_evsel *evsel;
 	const struct cpu_map *cpus = evlist->cpus;
 	const struct thread_map *threads = evlist->threads;
 	struct mmap_params mp = {
 		.prot = PROT_READ | (overwrite ? 0 : PROT_WRITE),
+		.track = use_track_mmap,
 	};
 
-	if (evlist->mmap == NULL && perf_evlist__alloc_mmap(evlist, true) < 0)
+	if (evlist->mmap == NULL && perf_evlist__alloc_mmap(evlist, mp.track) < 0)
 		return -ENOMEM;
 
 	if (evlist->pollfd.entries == NULL && perf_evlist__alloc_pollfd(evlist) < 0)
@@ -1189,7 +1192,7 @@ int perf_evlist__mmap_ex(struct perf_evlist *evlist, unsigned int pages,
 int perf_evlist__mmap(struct perf_evlist *evlist, unsigned int pages,
 		      bool overwrite)
 {
-	return perf_evlist__mmap_ex(evlist, pages, overwrite, 0, false);
+	return perf_evlist__mmap_ex(evlist, pages, overwrite, 0, false, false);
 }
 
 int perf_evlist__create_maps(struct perf_evlist *evlist, struct target *target)
diff --git a/tools/perf/util/evlist.h b/tools/perf/util/evlist.h
index 335988577c18..27453338a8f5 100644
--- a/tools/perf/util/evlist.h
+++ b/tools/perf/util/evlist.h
@@ -135,7 +135,7 @@ int perf_evlist__parse_mmap_pages(const struct option *opt,
 
 int perf_evlist__mmap_ex(struct perf_evlist *evlist, unsigned int pages,
 			 bool overwrite, unsigned int auxtrace_pages,
-			 bool auxtrace_overwrite);
+			 bool auxtrace_overwrite, bool use_track_mmap);
 int perf_evlist__mmap(struct perf_evlist *evlist, unsigned int pages,
 		      bool overwrite);
 void perf_evlist__munmap(struct perf_evlist *evlist);
-- 
2.4.0


^ permalink raw reply related	[flat|nested] 221+ messages in thread

* [PATCH 06/40] perf tools: Add HEADER_DATA_INDEX feature
  2015-05-18  0:30 [RFC/PATCHSET 00/40] perf tools: Speed-up perf report by using multi thread (v4) Namhyung Kim
                   ` (4 preceding siblings ...)
  2015-05-18  0:30 ` [PATCH 05/40] perf tools: Introduce perf_evlist__mmap_track() Namhyung Kim
@ 2015-05-18  0:30 ` Namhyung Kim
  2015-05-18 18:17   ` Jiri Olsa
  2015-05-18  0:30 ` [PATCH 07/40] perf tools: Handle indexed data file properly Namhyung Kim
                   ` (33 subsequent siblings)
  39 siblings, 1 reply; 221+ messages in thread
From: Namhyung Kim @ 2015-05-18  0:30 UTC (permalink / raw)
  To: Arnaldo Carvalho de Melo
  Cc: Ingo Molnar, Peter Zijlstra, Jiri Olsa, LKML, David Ahern,
	Adrian Hunter, Andi Kleen, Frederic Weisbecker, Stephane Eranian

The HEADER_DATA_INDEX feature is to record index table for sample data
so that they can be processed by multiple thread concurrently.  Each
item is a struct perf_file_section which consists of an offset and size.

Signed-off-by: Namhyung Kim <namhyung@kernel.org>
---
 tools/perf/builtin-record.c |  2 ++
 tools/perf/util/header.c    | 62 +++++++++++++++++++++++++++++++++++++++++++++
 tools/perf/util/header.h    |  3 +++
 3 files changed, 67 insertions(+)

diff --git a/tools/perf/builtin-record.c b/tools/perf/builtin-record.c
index 21f7edb23370..303116c9a38a 100644
--- a/tools/perf/builtin-record.c
+++ b/tools/perf/builtin-record.c
@@ -459,6 +459,8 @@ static void record__init_features(struct record *rec)
 
 	if (!rec->opts.full_auxtrace)
 		perf_header__clear_feat(&session->header, HEADER_AUXTRACE);
+
+	perf_header__clear_feat(&session->header, HEADER_DATA_INDEX);
 }
 
 static volatile int workload_exec_errno;
diff --git a/tools/perf/util/header.c b/tools/perf/util/header.c
index 3f0d809d853a..1e10f4bc664d 100644
--- a/tools/perf/util/header.c
+++ b/tools/perf/util/header.c
@@ -883,6 +883,24 @@ static int write_auxtrace(int fd, struct perf_header *h,
 	return err;
 }
 
+static int write_data_index(int fd, struct perf_header *h,
+			    struct perf_evlist *evlist __maybe_unused)
+{
+	int ret;
+	unsigned i;
+
+	ret = do_write(fd, &h->nr_index, sizeof(h->nr_index));
+	if (ret < 0)
+		return ret;
+
+	for (i = 0; i < h->nr_index; i++) {
+		ret = do_write(fd, &h->index[i], sizeof(*h->index));
+		if (ret < 0)
+			return ret;
+	}
+	return 0;
+}
+
 static void print_hostname(struct perf_header *ph, int fd __maybe_unused,
 			   FILE *fp)
 {
@@ -1233,6 +1251,12 @@ static void print_group_desc(struct perf_header *ph, int fd __maybe_unused,
 	}
 }
 
+static void print_data_index(struct perf_header *ph __maybe_unused,
+			     int fd __maybe_unused, FILE *fp)
+{
+	fprintf(fp, "# contains data index for parallel processing\n");
+}
+
 static int __event_process_build_id(struct build_id_event *bev,
 				    char *filename,
 				    struct perf_session *session)
@@ -1841,6 +1865,7 @@ static int process_group_desc(struct perf_file_section *section __maybe_unused,
 	return ret;
 }
 
+
 static int process_auxtrace(struct perf_file_section *section,
 			    struct perf_header *ph, int fd,
 			    void *data __maybe_unused)
@@ -1857,6 +1882,42 @@ static int process_auxtrace(struct perf_file_section *section,
 	return err;
 }
 
+static int process_data_index(struct perf_file_section *section __maybe_unused,
+			      struct perf_header *ph, int fd,
+			      void *data __maybe_unused)
+{
+	ssize_t ret;
+	u64 nr_idx;
+	unsigned i;
+	struct perf_file_section *idx;
+
+	ret = readn(fd, &nr_idx, sizeof(nr_idx));
+	if (ret != sizeof(nr_idx))
+		return -1;
+
+	if (ph->needs_swap)
+		nr_idx = bswap_64(nr_idx);
+
+	idx = calloc(nr_idx, sizeof(*idx));
+	if (idx == NULL)
+		return -1;
+
+	for (i = 0; i < nr_idx; i++) {
+		ret = readn(fd, &idx[i], sizeof(*idx));
+		if (ret != sizeof(*idx))
+			return ret;
+
+		if (ph->needs_swap) {
+			idx[i].offset = bswap_64(idx[i].offset);
+			idx[i].size   = bswap_64(idx[i].size);
+		}
+	}
+
+	ph->index = idx;
+	ph->nr_index = nr_idx;
+	return 0;
+}
+
 struct feature_ops {
 	int (*write)(int fd, struct perf_header *h, struct perf_evlist *evlist);
 	void (*print)(struct perf_header *h, int fd, FILE *fp);
@@ -1898,6 +1959,7 @@ static const struct feature_ops feat_ops[HEADER_LAST_FEATURE] = {
 	FEAT_OPP(HEADER_PMU_MAPPINGS,	pmu_mappings),
 	FEAT_OPP(HEADER_GROUP_DESC,	group_desc),
 	FEAT_OPP(HEADER_AUXTRACE,	auxtrace),
+	FEAT_OPP(HEADER_DATA_INDEX,	data_index),
 };
 
 struct header_print_data {
diff --git a/tools/perf/util/header.h b/tools/perf/util/header.h
index d4d57962c591..746e27763fab 100644
--- a/tools/perf/util/header.h
+++ b/tools/perf/util/header.h
@@ -31,6 +31,7 @@ enum {
 	HEADER_PMU_MAPPINGS,
 	HEADER_GROUP_DESC,
 	HEADER_AUXTRACE,
+	HEADER_DATA_INDEX,
 	HEADER_LAST_FEATURE,
 	HEADER_FEAT_BITS	= 256,
 };
@@ -95,6 +96,8 @@ struct perf_header {
 	bool				needs_swap;
 	u64				data_offset;
 	u64				data_size;
+	struct perf_file_section	*index;
+	u64				nr_index;
 	u64				feat_offset;
 	DECLARE_BITMAP(adds_features, HEADER_FEAT_BITS);
 	struct perf_session_env 	env;
-- 
2.4.0


^ permalink raw reply related	[flat|nested] 221+ messages in thread

* [PATCH 07/40] perf tools: Handle indexed data file properly
  2015-05-18  0:30 [RFC/PATCHSET 00/40] perf tools: Speed-up perf report by using multi thread (v4) Namhyung Kim
                   ` (5 preceding siblings ...)
  2015-05-18  0:30 ` [PATCH 06/40] perf tools: Add HEADER_DATA_INDEX feature Namhyung Kim
@ 2015-05-18  0:30 ` Namhyung Kim
  2015-05-18 18:37   ` Jiri Olsa
  2015-05-18  0:30 ` [PATCH 08/40] perf record: Add --index option for building index table Namhyung Kim
                   ` (32 subsequent siblings)
  39 siblings, 1 reply; 221+ messages in thread
From: Namhyung Kim @ 2015-05-18  0:30 UTC (permalink / raw)
  To: Arnaldo Carvalho de Melo
  Cc: Ingo Molnar, Peter Zijlstra, Jiri Olsa, LKML, David Ahern,
	Adrian Hunter, Andi Kleen, Frederic Weisbecker, Stephane Eranian

When perf detects data file has index table, process header part first
and then rest data files in a row.  Note that the indexed sample data is
recorded for each cpu/thread separately, it's already ordered with
respect to themselves so no need to use the ordered event queue
interface.

Signed-off-by: Namhyung Kim <namhyung@kernel.org>
---
 tools/perf/perf.c         |  1 +
 tools/perf/perf.h         |  2 ++
 tools/perf/util/session.c | 55 +++++++++++++++++++++++++++++++++++++++--------
 3 files changed, 49 insertions(+), 9 deletions(-)

diff --git a/tools/perf/perf.c b/tools/perf/perf.c
index b857fcbd00cf..960942d9a0b7 100644
--- a/tools/perf/perf.c
+++ b/tools/perf/perf.c
@@ -27,6 +27,7 @@ const char perf_more_info_string[] =
 int use_browser = -1;
 static int use_pager = -1;
 const char *input_name;
+bool perf_has_index;
 
 struct cmd_struct {
 	const char *cmd;
diff --git a/tools/perf/perf.h b/tools/perf/perf.h
index aa79fb8a16d4..61ce68d5c59f 100644
--- a/tools/perf/perf.h
+++ b/tools/perf/perf.h
@@ -39,6 +39,8 @@ void pthread__unblock_sigwinch(void);
 
 #include "util/target.h"
 
+extern bool perf_has_index;
+
 struct record_opts {
 	struct target target;
 	bool	     group;
diff --git a/tools/perf/util/session.c b/tools/perf/util/session.c
index e722107f932a..edb6ca22bff2 100644
--- a/tools/perf/util/session.c
+++ b/tools/perf/util/session.c
@@ -1486,7 +1486,9 @@ static int __perf_session__process_events(struct perf_session *session,
 	mmap_size = MMAP_SIZE;
 	if (mmap_size > file_size) {
 		mmap_size = file_size;
-		session->one_mmap = true;
+
+		if (!perf_has_index)
+			session->one_mmap = true;
 	}
 
 	memset(mmaps, 0, sizeof(mmaps));
@@ -1561,28 +1563,63 @@ static int __perf_session__process_events(struct perf_session *session,
 	err = auxtrace__flush_events(session, tool);
 out_err:
 	ui_progress__finish();
-	perf_session__warn_about_errors(session);
 	ordered_events__free(&session->ordered_events);
 	auxtrace__free_events(session);
 	session->one_mmap = false;
 	return err;
 }
 
+static int __perf_session__process_indexed_events(struct perf_session *session)
+{
+	struct perf_data_file *file = session->file;
+	struct perf_tool *tool = session->tool;
+	u64 size = perf_data_file__size(file);
+	int err = 0, i;
+
+	for (i = 0; i < (int)session->header.nr_index; i++) {
+		struct perf_file_section *idx = &session->header.index[i];
+
+		if (!idx->size)
+			continue;
+
+		/*
+		 * For indexed data file, samples are processed for
+		 * each cpu/thread so it's already ordered.  However
+		 * meta-events at index 0 should be processed in order.
+		 */
+		if (i > 0)
+			tool->ordered_events = false;
+
+		err = __perf_session__process_events(session, idx->offset,
+						     idx->size, size);
+		if (err < 0)
+			break;
+	}
+
+	perf_session__warn_about_errors(session);
+	return err;
+}
+
 int perf_session__process_events(struct perf_session *session)
 {
-	u64 size = perf_data_file__size(session->file);
+	struct perf_data_file *file = session->file;
+	u64 size = perf_data_file__size(file);
 	int err;
 
 	if (perf_session__register_idle_thread(session) == NULL)
 		return -ENOMEM;
 
-	if (!perf_data_file__is_pipe(session->file))
-		err = __perf_session__process_events(session,
-						     session->header.data_offset,
-						     session->header.data_size, size);
-	else
-		err = __perf_session__process_pipe_events(session);
+	if (perf_data_file__is_pipe(file))
+		return __perf_session__process_pipe_events(session);
+	if (perf_has_index)
+		return __perf_session__process_indexed_events(session);
+
+	err = __perf_session__process_events(session,
+					     session->header.data_offset,
+					     session->header.data_size,
+					     size);
 
+	perf_session__warn_about_errors(session);
 	return err;
 }
 
-- 
2.4.0


^ permalink raw reply related	[flat|nested] 221+ messages in thread

* [PATCH 08/40] perf record: Add --index option for building index table
  2015-05-18  0:30 [RFC/PATCHSET 00/40] perf tools: Speed-up perf report by using multi thread (v4) Namhyung Kim
                   ` (6 preceding siblings ...)
  2015-05-18  0:30 ` [PATCH 07/40] perf tools: Handle indexed data file properly Namhyung Kim
@ 2015-05-18  0:30 ` Namhyung Kim
  2015-05-18  0:30 ` [PATCH 09/40] perf report: Skip dummy tracking event Namhyung Kim
                   ` (31 subsequent siblings)
  39 siblings, 0 replies; 221+ messages in thread
From: Namhyung Kim @ 2015-05-18  0:30 UTC (permalink / raw)
  To: Arnaldo Carvalho de Melo
  Cc: Ingo Molnar, Peter Zijlstra, Jiri Olsa, LKML, David Ahern,
	Adrian Hunter, Andi Kleen, Frederic Weisbecker, Stephane Eranian

The new --index option will create indexed data file which can be
processed by multiple threads parallelly.  It saves meta event and
sample data in separate files and merges them with an index table.

If there's an index table in the data file, the HEADER_DATA_INDEX
feature bit is set and session->header.index[0] will point to the meta
event area, and rest are sample data.  It'd look like below:

        +---------------------+
        |     file header     |
        |---------------------|
        |                     |
        |    meta events[0] <-+--+
        |                     |  |
        |---------------------|  |
        |                     |  |
        |    sample data[1] <-+--+
        |                     |  |
        |---------------------|  |
        |                     |  |
        |    sample data[2] <-|--+
        |                     |  |
        |---------------------|  |
        |         ...         | ...
        |---------------------|  |
        |     feature data    |  |
        |   (contains index) -+--+
        +---------------------+

Signed-off-by: Namhyung Kim <namhyung@kernel.org>
---
 tools/perf/Documentation/perf-record.txt |   4 +
 tools/perf/builtin-record.c              | 172 ++++++++++++++++++++++++++++---
 tools/perf/perf.h                        |   1 +
 tools/perf/util/header.c                 |   2 +
 tools/perf/util/session.c                |   1 +
 5 files changed, 166 insertions(+), 14 deletions(-)

diff --git a/tools/perf/Documentation/perf-record.txt b/tools/perf/Documentation/perf-record.txt
index 280533ebf9df..7eac31f02f8c 100644
--- a/tools/perf/Documentation/perf-record.txt
+++ b/tools/perf/Documentation/perf-record.txt
@@ -267,6 +267,10 @@ AUX area tracing event. Optionally the number of bytes to capture per
 snapshot can be specified. In Snapshot Mode, trace data is captured only when
 signal SIGUSR2 is received.
 
+--index::
+Build an index table for sample data.  This will speed up perf report by
+parallel processing.
+
 SEE ALSO
 --------
 linkperf:perf-stat[1], linkperf:perf-list[1]
diff --git a/tools/perf/builtin-record.c b/tools/perf/builtin-record.c
index 303116c9a38a..4ddf104f50ff 100644
--- a/tools/perf/builtin-record.c
+++ b/tools/perf/builtin-record.c
@@ -40,6 +40,7 @@ struct record {
 	u64			bytes_written;
 	struct perf_data_file	file;
 	struct auxtrace_record	*itr;
+	int			*fds;
 	struct perf_evlist	*evlist;
 	struct perf_session	*session;
 	const char		*progname;
@@ -49,9 +50,16 @@ struct record {
 	long			samples;
 };
 
-static int record__write(struct record *rec, void *bf, size_t size)
+static int record__write(struct record *rec, void *bf, size_t size, int idx)
 {
-	if (perf_data_file__write(rec->session->file, bf, size) < 0) {
+	int fd;
+
+	if (rec->fds && idx >= 0)
+		fd = rec->fds[idx];
+	else
+		fd = perf_data_file__fd(rec->session->file);
+
+	if (writen(fd, bf, size) < 0) {
 		pr_err("failed to write perf data, error: %m\n");
 		return -1;
 	}
@@ -66,7 +74,7 @@ static int process_synthesized_event(struct perf_tool *tool,
 				     struct machine *machine __maybe_unused)
 {
 	struct record *rec = container_of(tool, struct record, tool);
-	return record__write(rec, event, event->header.size);
+	return record__write(rec, event, event->header.size, -1);
 }
 
 static int record__mmap_read(struct record *rec, int idx)
@@ -91,7 +99,7 @@ static int record__mmap_read(struct record *rec, int idx)
 		size = md->mask + 1 - (old & md->mask);
 		old += size;
 
-		if (record__write(rec, buf, size) < 0) {
+		if (record__write(rec, buf, size, idx) < 0) {
 			rc = -1;
 			goto out;
 		}
@@ -101,7 +109,7 @@ static int record__mmap_read(struct record *rec, int idx)
 	size = head - old;
 	old += size;
 
-	if (record__write(rec, buf, size) < 0) {
+	if (record__write(rec, buf, size, idx) < 0) {
 		rc = -1;
 		goto out;
 	}
@@ -149,6 +157,7 @@ static int record__process_auxtrace(struct perf_tool *tool,
 	struct perf_data_file *file = &rec->file;
 	size_t padding;
 	u8 pad[8] = {0};
+	int idx = event->auxtrace.idx;
 
 	if (!perf_data_file__is_pipe(file)) {
 		off_t file_offset;
@@ -169,11 +178,11 @@ static int record__process_auxtrace(struct perf_tool *tool,
 	if (padding)
 		padding = 8 - padding;
 
-	record__write(rec, event, event->header.size);
-	record__write(rec, data1, len1);
+	record__write(rec, event, event->header.size, idx);
+	record__write(rec, data1, len1, idx);
 	if (len2)
-		record__write(rec, data2, len2);
-	record__write(rec, &pad, padding);
+		record__write(rec, data2, len2, idx);
+	record__write(rec, &pad, padding, idx);
 
 	return 0;
 }
@@ -266,6 +275,110 @@ int auxtrace_record__snapshot_start(struct auxtrace_record *itr __maybe_unused)
 
 #endif
 
+#define INDEX_FILE_FMT  "%s.dir/perf.data.%d"
+
+static int record__create_index_files(struct record *rec, int nr_index)
+{
+	int i = 0;
+	int ret = -1;
+	char path[PATH_MAX];
+	struct perf_data_file *file = &rec->file;
+
+	rec->fds = malloc(nr_index * sizeof(int));
+	if (rec->fds == NULL)
+		return -ENOMEM;
+
+	scnprintf(path, sizeof(path), "%s.dir", file->path);
+	if (rm_rf(path) < 0 || mkdir(path, S_IRWXU) < 0)
+		goto out_err;
+
+	for (i = 0; i < nr_index; i++) {
+		scnprintf(path, sizeof(path), INDEX_FILE_FMT, file->path, i);
+		ret = open(path, O_RDWR|O_CREAT|O_TRUNC, S_IRUSR|S_IWUSR);
+		if (ret < 0)
+			goto out_err;
+
+		rec->fds[i] = ret;
+	}
+	return 0;
+
+out_err:
+	while (--i >= 1)
+		close(rec->fds[i]);
+	zfree(&rec->fds);
+
+	scnprintf(path, sizeof(path), "%s.dir", file->path);
+	rm_rf(path);
+
+	return ret;
+}
+
+static int record__merge_index_files(struct record *rec, int nr_index)
+{
+	int i;
+	int ret = -ENOMEM;
+	u64 offset;
+	char path[PATH_MAX];
+	struct perf_file_section *idx;
+	struct perf_data_file *file = &rec->file;
+	struct perf_session *session = rec->session;
+	int output_fd = perf_data_file__fd(file);
+
+	/* +1 for header file itself */
+	nr_index++;
+
+	idx = calloc(nr_index, sizeof(*idx));
+	if (idx == NULL)
+		goto out_close;
+
+	offset = lseek(output_fd, 0, SEEK_END);
+
+	idx[0].offset = session->header.data_offset;
+	idx[0].size   = offset - idx[0].offset;
+
+	for (i = 1; i < nr_index; i++) {
+		struct stat stbuf;
+		int fd = rec->fds[i - 1];
+
+		ret = fstat(fd, &stbuf);
+		if (ret < 0)
+			goto out_close;
+
+		idx[i].offset = offset;
+		idx[i].size   = stbuf.st_size;
+
+		offset += stbuf.st_size;
+
+		if (idx[i].size == 0)
+			continue;
+
+		ret = copyfile_offset(fd, 0, output_fd, idx[i].offset,
+				      idx[i].size);
+		if (ret < 0)
+			goto out_close;
+	}
+
+	session->header.index = idx;
+	session->header.nr_index = nr_index;
+
+	perf_has_index = true;
+
+	ret = 0;
+
+out_close:
+	if (ret < 0)
+		pr_err("failed to merge index files: %d\n", ret);
+
+	for (i = 0; i < nr_index - 1; i++)
+		close(rec->fds[i]);
+
+	scnprintf(path, sizeof(path), "%s.dir", file->path);
+	rm_rf(path);
+
+	zfree(&rec->fds);
+	return ret;
+}
+
 static int record__open(struct record *rec)
 {
 	char msg[512];
@@ -304,7 +417,8 @@ static int record__open(struct record *rec)
 
 	if (perf_evlist__mmap_ex(evlist, opts->mmap_pages, false,
 				 opts->auxtrace_mmap_pages,
-				 opts->auxtrace_snapshot_mode, false) < 0) {
+				 opts->auxtrace_snapshot_mode,
+				 opts->index) < 0) {
 		if (errno == EPERM) {
 			pr_err("Permission error mapping pages.\n"
 			       "Consider increasing "
@@ -321,6 +435,14 @@ static int record__open(struct record *rec)
 		goto out;
 	}
 
+	if (opts->index) {
+		rc = record__create_index_files(rec, evlist->nr_mmaps);
+		if (rc < 0) {
+			pr_err("failed to create index file: %d\n", rc);
+			goto out;
+		}
+	}
+
 	session->evlist = evlist;
 	perf_session__set_id_hdr_size(session);
 out:
@@ -345,7 +467,8 @@ static int process_buildids(struct record *rec)
 	struct perf_data_file *file  = &rec->file;
 	struct perf_session *session = rec->session;
 
-	u64 size = lseek(perf_data_file__fd(file), 0, SEEK_CUR);
+	/* update file size after merging sample files with index */
+	u64 size = lseek(perf_data_file__fd(file), 0, SEEK_END);
 	if (size == 0)
 		return 0;
 
@@ -415,7 +538,7 @@ static int record__mmap_read_all(struct record *rec)
 			}
 		}
 
-		if (rec->evlist->track_mmap[i].base) {
+		if (rec->evlist->track_mmap && rec->evlist->track_mmap[i].base) {
 			if (record__mmap_read(rec, track_mmap_idx(i)) != 0) {
 				rc = -1;
 				goto out;
@@ -434,7 +557,8 @@ static int record__mmap_read_all(struct record *rec)
 	 * at least one event.
 	 */
 	if (bytes_written != rec->bytes_written)
-		rc = record__write(rec, &finished_round_event, sizeof(finished_round_event));
+		rc = record__write(rec, &finished_round_event,
+				   sizeof(finished_round_event), -1);
 
 out:
 	return rc;
@@ -460,7 +584,8 @@ static void record__init_features(struct record *rec)
 	if (!rec->opts.full_auxtrace)
 		perf_header__clear_feat(&session->header, HEADER_AUXTRACE);
 
-	perf_header__clear_feat(&session->header, HEADER_DATA_INDEX);
+	if (!rec->opts.index)
+		perf_header__clear_feat(&session->header, HEADER_DATA_INDEX);
 }
 
 static volatile int workload_exec_errno;
@@ -528,6 +653,11 @@ static int __cmd_record(struct record *rec, int argc, const char **argv)
 		}
 	}
 
+	if (file->is_pipe && opts->index) {
+		pr_warning("Indexing is disabled for pipe output\n");
+		opts->index = false;
+	}
+
 	if (record__open(rec) != 0) {
 		err = -1;
 		goto out_child;
@@ -729,6 +859,9 @@ static int __cmd_record(struct record *rec, int argc, const char **argv)
 	if (!err && !file->is_pipe) {
 		rec->session->header.data_size += rec->bytes_written;
 
+		if (rec->opts.index)
+			record__merge_index_files(rec, rec->evlist->nr_mmaps);
+
 		if (!rec->no_buildid) {
 			process_buildids(rec);
 			/*
@@ -1166,6 +1299,8 @@ struct option __record_options[] = {
 	parse_clockid),
 	OPT_STRING_OPTARG('S', "snapshot", &record.opts.auxtrace_snapshot_opts,
 			  "opts", "AUX area tracing Snapshot Mode", ""),
+	OPT_BOOLEAN(0, "index", &record.opts.index,
+		    "make index for sample data to speed-up processing"),
 	OPT_END()
 };
 
@@ -1228,6 +1363,15 @@ int cmd_record(int argc, const char **argv, const char *prefix __maybe_unused)
 		goto out_symbol_exit;
 	}
 
+	if (rec->opts.index) {
+		if (!rec->opts.sample_time) {
+			pr_err("Sample timestamp is required for indexing\n");
+			goto out_symbol_exit;
+		}
+
+		perf_evlist__add_dummy_tracking(rec->evlist);
+	}
+
 	if (rec->opts.target.tid && !rec->opts.no_inherit_set)
 		rec->opts.no_inherit = true;
 
diff --git a/tools/perf/perf.h b/tools/perf/perf.h
index 61ce68d5c59f..192d936020ea 100644
--- a/tools/perf/perf.h
+++ b/tools/perf/perf.h
@@ -58,6 +58,7 @@ struct record_opts {
 	bool	     running_time;
 	bool	     full_auxtrace;
 	bool	     auxtrace_snapshot_mode;
+	bool	     index;
 	unsigned int freq;
 	unsigned int mmap_pages;
 	unsigned int auxtrace_mmap_pages;
diff --git a/tools/perf/util/header.c b/tools/perf/util/header.c
index 1e10f4bc664d..796a958d951a 100644
--- a/tools/perf/util/header.c
+++ b/tools/perf/util/header.c
@@ -2652,6 +2652,8 @@ int perf_session__read_header(struct perf_session *session)
 						   session->tevent.pevent))
 		goto out_delete_evlist;
 
+	perf_has_index = perf_header__has_feat(&session->header, HEADER_DATA_INDEX);
+
 	return 0;
 out_errno:
 	return -errno;
diff --git a/tools/perf/util/session.c b/tools/perf/util/session.c
index edb6ca22bff2..bc738216de36 100644
--- a/tools/perf/util/session.c
+++ b/tools/perf/util/session.c
@@ -195,6 +195,7 @@ void perf_session__delete(struct perf_session *session)
 	machines__exit(&session->machines);
 	if (session->file)
 		perf_data_file__close(session->file);
+	free(session->header.index);
 	free(session);
 }
 
-- 
2.4.0


^ permalink raw reply related	[flat|nested] 221+ messages in thread

* [PATCH 09/40] perf report: Skip dummy tracking event
  2015-05-18  0:30 [RFC/PATCHSET 00/40] perf tools: Speed-up perf report by using multi thread (v4) Namhyung Kim
                   ` (7 preceding siblings ...)
  2015-05-18  0:30 ` [PATCH 08/40] perf record: Add --index option for building index table Namhyung Kim
@ 2015-05-18  0:30 ` Namhyung Kim
  2015-05-18  0:30 ` [PATCH 10/40] perf tools: Introduce thread__comm(_str)_by_time() helpers Namhyung Kim
                   ` (30 subsequent siblings)
  39 siblings, 0 replies; 221+ messages in thread
From: Namhyung Kim @ 2015-05-18  0:30 UTC (permalink / raw)
  To: Arnaldo Carvalho de Melo
  Cc: Ingo Molnar, Peter Zijlstra, Jiri Olsa, LKML, David Ahern,
	Adrian Hunter, Andi Kleen, Frederic Weisbecker, Stephane Eranian

The dummy tracking event is only for tracking task/comom/mmap events
and has no sample data for itself.  So no need to report, just skip it.

Signed-off-by: Namhyung Kim <namhyung@kernel.org>
---
 tools/perf/builtin-report.c    |  3 +++
 tools/perf/ui/browsers/hists.c | 30 ++++++++++++++++++++++++------
 tools/perf/ui/gtk/hists.c      |  3 +++
 3 files changed, 30 insertions(+), 6 deletions(-)

diff --git a/tools/perf/builtin-report.c b/tools/perf/builtin-report.c
index 92fca2157e5e..fee770935eab 100644
--- a/tools/perf/builtin-report.c
+++ b/tools/perf/builtin-report.c
@@ -323,6 +323,9 @@ static int perf_evlist__tty_browse_hists(struct perf_evlist *evlist,
 		struct hists *hists = evsel__hists(pos);
 		const char *evname = perf_evsel__name(pos);
 
+		if (perf_evsel__is_dummy_tracking(pos))
+			continue;
+
 		if (symbol_conf.event_group &&
 		    !perf_evsel__is_group_leader(pos))
 			continue;
diff --git a/tools/perf/ui/browsers/hists.c b/tools/perf/ui/browsers/hists.c
index f981cb8f0158..2cc18b693950 100644
--- a/tools/perf/ui/browsers/hists.c
+++ b/tools/perf/ui/browsers/hists.c
@@ -2128,14 +2128,17 @@ static int perf_evsel_menu__run(struct perf_evsel_menu *menu,
 	return key;
 }
 
-static bool filter_group_entries(struct ui_browser *browser __maybe_unused,
-				 void *entry)
+static bool filter_entries(struct ui_browser *browser __maybe_unused,
+			   void *entry)
 {
 	struct perf_evsel *evsel = list_entry(entry, struct perf_evsel, node);
 
 	if (symbol_conf.event_group && !perf_evsel__is_group_leader(evsel))
 		return true;
 
+	if (perf_evsel__is_dummy_tracking(evsel))
+		return true;
+
 	return false;
 }
 
@@ -2152,7 +2155,7 @@ static int __perf_evlist__tui_browse_hists(struct perf_evlist *evlist,
 			.refresh    = ui_browser__list_head_refresh,
 			.seek	    = ui_browser__list_head_seek,
 			.write	    = perf_evsel_menu__write,
-			.filter	    = filter_group_entries,
+			.filter	    = filter_entries,
 			.nr_entries = nr_entries,
 			.priv	    = evlist,
 		},
@@ -2179,21 +2182,22 @@ int perf_evlist__tui_browse_hists(struct perf_evlist *evlist, const char *help,
 				  struct perf_session_env *env)
 {
 	int nr_entries = evlist->nr_entries;
+	struct perf_evsel *first = perf_evlist__first(evlist);
+	struct perf_evsel *pos;
 
 single_entry:
 	if (nr_entries == 1) {
-		struct perf_evsel *first = perf_evlist__first(evlist);
-
 		return perf_evsel__hists_browse(first, nr_entries, help,
 						false, hbt, min_pcnt,
 						env);
 	}
 
 	if (symbol_conf.event_group) {
-		struct perf_evsel *pos;
 
 		nr_entries = 0;
 		evlist__for_each(evlist, pos) {
+			if (perf_evsel__is_dummy_tracking(pos))
+				continue;
 			if (perf_evsel__is_group_leader(pos))
 				nr_entries++;
 		}
@@ -2202,6 +2206,20 @@ int perf_evlist__tui_browse_hists(struct perf_evlist *evlist, const char *help,
 			goto single_entry;
 	}
 
+	evlist__for_each(evlist, pos) {
+		if (perf_evsel__is_dummy_tracking(pos))
+			nr_entries--;
+	}
+
+	if (nr_entries == 1) {
+		evlist__for_each(evlist, pos) {
+			if (!perf_evsel__is_dummy_tracking(pos)) {
+				first = pos;
+				goto single_entry;
+			}
+		}
+	}
+
 	return __perf_evlist__tui_browse_hists(evlist, nr_entries, help,
 					       hbt, min_pcnt, env);
 }
diff --git a/tools/perf/ui/gtk/hists.c b/tools/perf/ui/gtk/hists.c
index 4b3585eed1e8..83a7ecd5cda8 100644
--- a/tools/perf/ui/gtk/hists.c
+++ b/tools/perf/ui/gtk/hists.c
@@ -317,6 +317,9 @@ int perf_evlist__gtk_browse_hists(struct perf_evlist *evlist,
 		char buf[512];
 		size_t size = sizeof(buf);
 
+		if (perf_evsel__is_dummy_tracking(pos))
+			continue;
+
 		if (symbol_conf.event_group) {
 			if (!perf_evsel__is_group_leader(pos))
 				continue;
-- 
2.4.0


^ permalink raw reply related	[flat|nested] 221+ messages in thread

* [PATCH 10/40] perf tools: Introduce thread__comm(_str)_by_time() helpers
  2015-05-18  0:30 [RFC/PATCHSET 00/40] perf tools: Speed-up perf report by using multi thread (v4) Namhyung Kim
                   ` (8 preceding siblings ...)
  2015-05-18  0:30 ` [PATCH 09/40] perf report: Skip dummy tracking event Namhyung Kim
@ 2015-05-18  0:30 ` Namhyung Kim
  2015-05-18  0:30 ` [PATCH 11/40] perf tools: Add a test case for thread comm handling Namhyung Kim
                   ` (29 subsequent siblings)
  39 siblings, 0 replies; 221+ messages in thread
From: Namhyung Kim @ 2015-05-18  0:30 UTC (permalink / raw)
  To: Arnaldo Carvalho de Melo
  Cc: Ingo Molnar, Peter Zijlstra, Jiri Olsa, LKML, David Ahern,
	Adrian Hunter, Andi Kleen, Frederic Weisbecker, Stephane Eranian

When data file indexing is enabled, it processes all task, comm and mmap
events first and then goes to the sample events.  So all it sees is the
last comm of a thread although it has information at the time of sample.

Sort thread's comm by time so that it can find appropriate comm at the
sample time.  The thread__comm_by_time() will mostly work even if
PERF_SAMPLE_TIME bit is off since in that case, sample->time will be
-1 so it'll take the last comm anyway.

Cc: Frederic Weisbecker <fweisbec@gmail.com>
Signed-off-by: Namhyung Kim <namhyung@kernel.org>
---
 tools/perf/util/thread.c | 33 ++++++++++++++++++++++++++++++++-
 tools/perf/util/thread.h |  2 ++
 2 files changed, 34 insertions(+), 1 deletion(-)

diff --git a/tools/perf/util/thread.c b/tools/perf/util/thread.c
index 16c28a37a9e4..962558024415 100644
--- a/tools/perf/util/thread.c
+++ b/tools/perf/util/thread.c
@@ -123,6 +123,21 @@ struct comm *thread__exec_comm(const struct thread *thread)
 	return last;
 }
 
+struct comm *thread__comm_by_time(const struct thread *thread, u64 timestamp)
+{
+	struct comm *comm;
+
+	list_for_each_entry(comm, &thread->comm_list, list) {
+		if (timestamp >= comm->start)
+			return comm;
+	}
+
+	if (list_empty(&thread->comm_list))
+		return NULL;
+
+	return list_last_entry(&thread->comm_list, struct comm, list);
+}
+
 int __thread__set_comm(struct thread *thread, const char *str, u64 timestamp,
 		       bool exec)
 {
@@ -138,7 +153,13 @@ int __thread__set_comm(struct thread *thread, const char *str, u64 timestamp,
 		new = comm__new(str, timestamp, exec);
 		if (!new)
 			return -ENOMEM;
-		list_add(&new->list, &thread->comm_list);
+
+		/* sort by time */
+		list_for_each_entry(curr, &thread->comm_list, list) {
+			if (timestamp >= curr->start)
+				break;
+		}
+		list_add_tail(&new->list, &curr->list);
 
 		if (exec)
 			unwind__flush_access(thread);
@@ -159,6 +180,16 @@ const char *thread__comm_str(const struct thread *thread)
 	return comm__str(comm);
 }
 
+const char *thread__comm_str_by_time(const struct thread *thread, u64 timestamp)
+{
+	const struct comm *comm = thread__comm_by_time(thread, timestamp);
+
+	if (!comm)
+		return NULL;
+
+	return comm__str(comm);
+}
+
 /* CHECKME: it should probably better return the max comm len from its comm list */
 int thread__comm_len(struct thread *thread)
 {
diff --git a/tools/perf/util/thread.h b/tools/perf/util/thread.h
index f33c48cfdaa0..903cfaf2628d 100644
--- a/tools/perf/util/thread.h
+++ b/tools/perf/util/thread.h
@@ -68,7 +68,9 @@ static inline int thread__set_comm(struct thread *thread, const char *comm,
 int thread__comm_len(struct thread *thread);
 struct comm *thread__comm(const struct thread *thread);
 struct comm *thread__exec_comm(const struct thread *thread);
+struct comm *thread__comm_by_time(const struct thread *thread, u64 timestamp);
 const char *thread__comm_str(const struct thread *thread);
+const char *thread__comm_str_by_time(const struct thread *thread, u64 timestamp);
 void thread__insert_map(struct thread *thread, struct map *map);
 int thread__fork(struct thread *thread, struct thread *parent, u64 timestamp);
 size_t thread__fprintf(struct thread *thread, FILE *fp);
-- 
2.4.0


^ permalink raw reply related	[flat|nested] 221+ messages in thread

* [PATCH 11/40] perf tools: Add a test case for thread comm handling
  2015-05-18  0:30 [RFC/PATCHSET 00/40] perf tools: Speed-up perf report by using multi thread (v4) Namhyung Kim
                   ` (9 preceding siblings ...)
  2015-05-18  0:30 ` [PATCH 10/40] perf tools: Introduce thread__comm(_str)_by_time() helpers Namhyung Kim
@ 2015-05-18  0:30 ` Namhyung Kim
  2015-05-18 19:29   ` Jiri Olsa
  2015-05-18  0:30 ` [PATCH 12/40] perf tools: Use thread__comm_by_time() when adding hist entries Namhyung Kim
                   ` (28 subsequent siblings)
  39 siblings, 1 reply; 221+ messages in thread
From: Namhyung Kim @ 2015-05-18  0:30 UTC (permalink / raw)
  To: Arnaldo Carvalho de Melo
  Cc: Ingo Molnar, Peter Zijlstra, Jiri Olsa, LKML, David Ahern,
	Adrian Hunter, Andi Kleen, Frederic Weisbecker, Stephane Eranian

The new test case checks various thread comm handling like overridding
and time sorting.

Cc: Frederic Weisbecker <fweisbec@gmail.com>
Signed-off-by: Namhyung Kim <namhyung@kernel.org>
---
 tools/perf/tests/Build              |  1 +
 tools/perf/tests/builtin-test.c     |  4 ++++
 tools/perf/tests/perf-targz-src-pkg | 21 -----------------
 tools/perf/tests/tests.h            |  1 +
 tools/perf/tests/thread-comm.c      | 47 +++++++++++++++++++++++++++++++++++++
 5 files changed, 53 insertions(+), 21 deletions(-)
 delete mode 100755 tools/perf/tests/perf-targz-src-pkg
 create mode 100644 tools/perf/tests/thread-comm.c

diff --git a/tools/perf/tests/Build b/tools/perf/tests/Build
index 6a8801b32017..78d29a3a6a97 100644
--- a/tools/perf/tests/Build
+++ b/tools/perf/tests/Build
@@ -24,6 +24,7 @@ perf-y += bp_signal_overflow.o
 perf-y += task-exit.o
 perf-y += sw-clock.o
 perf-y += mmap-thread-lookup.o
+perf-y += thread-comm.o
 perf-y += thread-mg-share.o
 perf-y += switch-tracking.o
 perf-y += keep-tracking.o
diff --git a/tools/perf/tests/builtin-test.c b/tools/perf/tests/builtin-test.c
index f42af98a5c16..372b6395a448 100644
--- a/tools/perf/tests/builtin-test.c
+++ b/tools/perf/tests/builtin-test.c
@@ -171,6 +171,10 @@ static struct test {
 		.func = test__kmod_path__parse,
 	},
 	{
+		.desc = "Test thread comm handling",
+		.func = test__thread_comm,
+	},
+	{
 		.func = NULL,
 	},
 };
diff --git a/tools/perf/tests/perf-targz-src-pkg b/tools/perf/tests/perf-targz-src-pkg
deleted file mode 100755
index 238aa3927c71..000000000000
--- a/tools/perf/tests/perf-targz-src-pkg
+++ /dev/null
@@ -1,21 +0,0 @@
-#!/bin/sh
-# Test one of the main kernel Makefile targets to generate a perf sources tarball
-# suitable for build outside the full kernel sources.
-#
-# This is to test that the tools/perf/MANIFEST file lists all the files needed to
-# be in such tarball, which sometimes gets broken when we move files around,
-# like when we made some files that were in tools/perf/ available to other tools/
-# codebases by moving it to tools/include/, etc.
-
-PERF=$1
-cd ${PERF}/../..
-make perf-targz-src-pkg > /dev/null
-TARBALL=$(ls -rt perf-*.tar.gz)
-TMP_DEST=$(mktemp -d)
-tar xf ${TARBALL} -C $TMP_DEST
-rm -f ${TARBALL}
-cd - > /dev/null
-make -C $TMP_DEST/perf*/tools/perf > /dev/null 2>&1
-RC=$?
-rm -rf ${TMP_DEST}
-exit $RC
diff --git a/tools/perf/tests/tests.h b/tools/perf/tests/tests.h
index a10eaf5c4767..aa269eff798a 100644
--- a/tools/perf/tests/tests.h
+++ b/tools/perf/tests/tests.h
@@ -61,6 +61,7 @@ int test__switch_tracking(void);
 int test__fdarray__filter(void);
 int test__fdarray__add(void);
 int test__kmod_path__parse(void);
+int test__thread_comm(void);
 
 #if defined(__x86_64__) || defined(__i386__) || defined(__arm__)
 #ifdef HAVE_DWARF_UNWIND_SUPPORT
diff --git a/tools/perf/tests/thread-comm.c b/tools/perf/tests/thread-comm.c
new file mode 100644
index 000000000000..d146dedf63b4
--- /dev/null
+++ b/tools/perf/tests/thread-comm.c
@@ -0,0 +1,47 @@
+#include "tests.h"
+#include "machine.h"
+#include "thread.h"
+#include "debug.h"
+
+int test__thread_comm(void)
+{
+	struct machines machines;
+	struct machine *machine;
+	struct thread *t;
+
+	/*
+	 * This test is to check whether it can retrieve a correct
+	 * comm for a given time.  When multi-file data storage is
+	 * enabled, those task/comm events are processed first so the
+	 * later sample should find a matching comm properly.
+	 */
+	machines__init(&machines);
+	machine = &machines.host;
+
+	t = machine__findnew_thread(machine, 100, 100);
+	TEST_ASSERT_VAL("wrong init thread comm",
+			!strcmp(thread__comm_str(t), ":100"));
+
+	thread__set_comm(t, "perf-test1", 10000);
+	TEST_ASSERT_VAL("failed to override thread comm",
+			!strcmp(thread__comm_str(t), "perf-test1"));
+
+	thread__set_comm(t, "perf-test2", 20000);
+	thread__set_comm(t, "perf-test3", 30000);
+	thread__set_comm(t, "perf-test4", 40000);
+
+	TEST_ASSERT_VAL("failed to find timed comm",
+			!strcmp(thread__comm_str_by_time(t, 20000), "perf-test2"));
+	TEST_ASSERT_VAL("failed to find timed comm",
+			!strcmp(thread__comm_str_by_time(t, 35000), "perf-test3"));
+	TEST_ASSERT_VAL("failed to find timed comm",
+			!strcmp(thread__comm_str_by_time(t, 50000), "perf-test4"));
+
+	thread__set_comm(t, "perf-test1.5", 15000);
+	TEST_ASSERT_VAL("failed to sort timed comm",
+			!strcmp(thread__comm_str_by_time(t, 15000), "perf-test1.5"));
+
+	machine__delete_threads(machine);
+	machines__exit(&machines);
+	return 0;
+}
-- 
2.4.0


^ permalink raw reply related	[flat|nested] 221+ messages in thread

* [PATCH 12/40] perf tools: Use thread__comm_by_time() when adding hist entries
  2015-05-18  0:30 [RFC/PATCHSET 00/40] perf tools: Speed-up perf report by using multi thread (v4) Namhyung Kim
                   ` (10 preceding siblings ...)
  2015-05-18  0:30 ` [PATCH 11/40] perf tools: Add a test case for thread comm handling Namhyung Kim
@ 2015-05-18  0:30 ` Namhyung Kim
  2015-05-18  0:30 ` [PATCH 13/40] perf tools: Convert dead thread list into rbtree Namhyung Kim
                   ` (27 subsequent siblings)
  39 siblings, 0 replies; 221+ messages in thread
From: Namhyung Kim @ 2015-05-18  0:30 UTC (permalink / raw)
  To: Arnaldo Carvalho de Melo
  Cc: Ingo Molnar, Peter Zijlstra, Jiri Olsa, LKML, David Ahern,
	Adrian Hunter, Andi Kleen, Frederic Weisbecker, Stephane Eranian

Now thread->comm can be handled with time properly, use it to find
correct comm when adding hist entries.

Cc: Frederic Weisbecker <fweisbec@gmail.com>
Signed-off-by: Namhyung Kim <namhyung@kernel.org>
---
 tools/perf/builtin-annotate.c |  5 +++--
 tools/perf/builtin-diff.c     |  8 ++++----
 tools/perf/tests/hists_link.c |  4 ++--
 tools/perf/util/hist.c        | 19 ++++++++++---------
 tools/perf/util/hist.h        |  2 +-
 5 files changed, 20 insertions(+), 18 deletions(-)

diff --git a/tools/perf/builtin-annotate.c b/tools/perf/builtin-annotate.c
index b57a027fb200..761f902473b7 100644
--- a/tools/perf/builtin-annotate.c
+++ b/tools/perf/builtin-annotate.c
@@ -47,7 +47,7 @@ struct perf_annotate {
 };
 
 static int perf_evsel__add_sample(struct perf_evsel *evsel,
-				  struct perf_sample *sample __maybe_unused,
+				  struct perf_sample *sample,
 				  struct addr_location *al,
 				  struct perf_annotate *ann)
 {
@@ -67,7 +67,8 @@ static int perf_evsel__add_sample(struct perf_evsel *evsel,
 		return 0;
 	}
 
-	he = __hists__add_entry(hists, al, NULL, NULL, NULL, 1, 1, 0, true);
+	he = __hists__add_entry(hists, al, NULL, NULL, NULL, 1, 1, 0,
+				sample->time, true);
 	if (he == NULL)
 		return -ENOMEM;
 
diff --git a/tools/perf/builtin-diff.c b/tools/perf/builtin-diff.c
index daaa7dca9c3b..0fe54a633a5e 100644
--- a/tools/perf/builtin-diff.c
+++ b/tools/perf/builtin-diff.c
@@ -312,10 +312,10 @@ static int formula_fprintf(struct hist_entry *he, struct hist_entry *pair,
 
 static int hists__add_entry(struct hists *hists,
 			    struct addr_location *al, u64 period,
-			    u64 weight, u64 transaction)
+			    u64 weight, u64 transaction, u64 timestamp)
 {
 	if (__hists__add_entry(hists, al, NULL, NULL, NULL, period, weight,
-			       transaction, true) != NULL)
+			       transaction, timestamp, true) != NULL)
 		return 0;
 	return -ENOMEM;
 }
@@ -336,8 +336,8 @@ static int diff__process_sample_event(struct perf_tool *tool __maybe_unused,
 		return -1;
 	}
 
-	if (hists__add_entry(hists, &al, sample->period,
-			     sample->weight, sample->transaction)) {
+	if (hists__add_entry(hists, &al, sample->period, sample->weight,
+			     sample->transaction, sample->time)) {
 		pr_warning("problem incrementing symbol period, skipping event\n");
 		goto out_put;
 	}
diff --git a/tools/perf/tests/hists_link.c b/tools/perf/tests/hists_link.c
index 8c102b011424..27bae90c9a95 100644
--- a/tools/perf/tests/hists_link.c
+++ b/tools/perf/tests/hists_link.c
@@ -90,7 +90,7 @@ static int add_hist_entries(struct perf_evlist *evlist, struct machine *machine)
 				goto out;
 
 			he = __hists__add_entry(hists, &al, NULL,
-						NULL, NULL, 1, 1, 0, true);
+						NULL, NULL, 1, 1, 0, -1, true);
 			if (he == NULL) {
 				addr_location__put(&al);
 				goto out;
@@ -116,7 +116,7 @@ static int add_hist_entries(struct perf_evlist *evlist, struct machine *machine)
 				goto out;
 
 			he = __hists__add_entry(hists, &al, NULL,
-						NULL, NULL, 1, 1, 0, true);
+						NULL, NULL, 1, 1, 0, -1, true);
 			if (he == NULL) {
 				addr_location__put(&al);
 				goto out;
diff --git a/tools/perf/util/hist.c b/tools/perf/util/hist.c
index 338770679863..f13993e53e4e 100644
--- a/tools/perf/util/hist.c
+++ b/tools/perf/util/hist.c
@@ -442,11 +442,11 @@ struct hist_entry *__hists__add_entry(struct hists *hists,
 				      struct branch_info *bi,
 				      struct mem_info *mi,
 				      u64 period, u64 weight, u64 transaction,
-				      bool sample_self)
+				      u64 timestamp, bool sample_self)
 {
 	struct hist_entry entry = {
 		.thread	= al->thread,
-		.comm = thread__comm(al->thread),
+		.comm = thread__comm_by_time(al->thread, timestamp),
 		.ms = {
 			.map	= al->map,
 			.sym	= al->sym,
@@ -504,13 +504,14 @@ iter_add_single_mem_entry(struct hist_entry_iter *iter, struct addr_location *al
 {
 	u64 cost;
 	struct mem_info *mi = iter->priv;
+	struct perf_sample *sample = iter->sample;
 	struct hists *hists = evsel__hists(iter->evsel);
 	struct hist_entry *he;
 
 	if (mi == NULL)
 		return -EINVAL;
 
-	cost = iter->sample->weight;
+	cost = sample->weight;
 	if (!cost)
 		cost = 1;
 
@@ -522,7 +523,7 @@ iter_add_single_mem_entry(struct hist_entry_iter *iter, struct addr_location *al
 	 * and the he_stat__add_period() function.
 	 */
 	he = __hists__add_entry(hists, al, iter->parent, NULL, mi,
-				cost, cost, 0, true);
+				cost, cost, 0, sample->time, true);
 	if (!he)
 		return -ENOMEM;
 
@@ -623,7 +624,7 @@ iter_add_next_branch_entry(struct hist_entry_iter *iter, struct addr_location *a
 	 * and not events sampled. Thus we use a pseudo period of 1.
 	 */
 	he = __hists__add_entry(hists, al, iter->parent, &bi[i], NULL,
-				1, 1, 0, true);
+				1, 1, 0, iter->sample->time, true);
 	if (he == NULL)
 		return -ENOMEM;
 
@@ -661,7 +662,7 @@ iter_add_single_normal_entry(struct hist_entry_iter *iter, struct addr_location
 
 	he = __hists__add_entry(evsel__hists(evsel), al, iter->parent, NULL, NULL,
 				sample->period, sample->weight,
-				sample->transaction, true);
+				sample->transaction, sample->time, true);
 	if (he == NULL)
 		return -ENOMEM;
 
@@ -723,7 +724,7 @@ iter_add_single_cumulative_entry(struct hist_entry_iter *iter,
 
 	he = __hists__add_entry(hists, al, iter->parent, NULL, NULL,
 				sample->period, sample->weight,
-				sample->transaction, true);
+				sample->transaction, sample->time, true);
 	if (he == NULL)
 		return -ENOMEM;
 
@@ -767,7 +768,7 @@ iter_add_next_cumulative_entry(struct hist_entry_iter *iter,
 	struct hist_entry he_tmp = {
 		.cpu = al->cpu,
 		.thread = al->thread,
-		.comm = thread__comm(al->thread),
+		.comm = thread__comm_by_time(al->thread, sample->time),
 		.ip = al->addr,
 		.ms = {
 			.map = al->map,
@@ -796,7 +797,7 @@ iter_add_next_cumulative_entry(struct hist_entry_iter *iter,
 
 	he = __hists__add_entry(evsel__hists(evsel), al, iter->parent, NULL, NULL,
 				sample->period, sample->weight,
-				sample->transaction, false);
+				sample->transaction, sample->time, false);
 	if (he == NULL)
 		return -ENOMEM;
 
diff --git a/tools/perf/util/hist.h b/tools/perf/util/hist.h
index 9f31b89a527a..43ffe62e25d2 100644
--- a/tools/perf/util/hist.h
+++ b/tools/perf/util/hist.h
@@ -109,7 +109,7 @@ struct hist_entry *__hists__add_entry(struct hists *hists,
 				      struct branch_info *bi,
 				      struct mem_info *mi, u64 period,
 				      u64 weight, u64 transaction,
-				      bool sample_self);
+				      u64 timestamp, bool sample_self);
 int hist_entry_iter__add(struct hist_entry_iter *iter, struct addr_location *al,
 			 struct perf_evsel *evsel, struct perf_sample *sample,
 			 int max_stack_depth, void *arg);
-- 
2.4.0


^ permalink raw reply related	[flat|nested] 221+ messages in thread

* [PATCH 13/40] perf tools: Convert dead thread list into rbtree
  2015-05-18  0:30 [RFC/PATCHSET 00/40] perf tools: Speed-up perf report by using multi thread (v4) Namhyung Kim
                   ` (11 preceding siblings ...)
  2015-05-18  0:30 ` [PATCH 12/40] perf tools: Use thread__comm_by_time() when adding hist entries Namhyung Kim
@ 2015-05-18  0:30 ` Namhyung Kim
  2015-05-18 19:34   ` Jiri Olsa
  2015-05-18  0:30 ` [PATCH 14/40] perf tools: Introduce machine__find*_thread_by_time() Namhyung Kim
                   ` (26 subsequent siblings)
  39 siblings, 1 reply; 221+ messages in thread
From: Namhyung Kim @ 2015-05-18  0:30 UTC (permalink / raw)
  To: Arnaldo Carvalho de Melo
  Cc: Ingo Molnar, Peter Zijlstra, Jiri Olsa, LKML, David Ahern,
	Adrian Hunter, Andi Kleen, Frederic Weisbecker, Stephane Eranian

Currently perf maintains dead threads in a linked list but this can be
a problem if someone needs to search from it especially in a large
session which might have many dead threads.  Convert it to a rbtree
like normal threads and it'll be used later with multi-file changes.

The list node is now used for chaining dead threads of same tid since
it's easier to handle such threads in time order.

Cc: Frederic Weisbecker <fweisbec@gmail.com>
Signed-off-by: Namhyung Kim <namhyung@kernel.org>
---
 tools/perf/util/machine.c | 80 ++++++++++++++++++++++++++++++++++++++++++-----
 tools/perf/util/machine.h |  2 +-
 tools/perf/util/thread.c  | 21 +++++++++++--
 tools/perf/util/thread.h  | 11 +++----
 4 files changed, 96 insertions(+), 18 deletions(-)

diff --git a/tools/perf/util/machine.c b/tools/perf/util/machine.c
index 34bf89f7f4f3..ae07b84a40f5 100644
--- a/tools/perf/util/machine.c
+++ b/tools/perf/util/machine.c
@@ -30,8 +30,8 @@ int machine__init(struct machine *machine, const char *root_dir, pid_t pid)
 	dsos__init(&machine->kernel_dsos);
 
 	machine->threads = RB_ROOT;
+	machine->dead_threads = RB_ROOT;
 	pthread_rwlock_init(&machine->threads_lock, NULL);
-	INIT_LIST_HEAD(&machine->dead_threads);
 	machine->last_match = NULL;
 
 	machine->vdso_info = NULL;
@@ -93,6 +93,29 @@ static void dsos__delete(struct dsos *dsos)
 	}
 }
 
+static void machine__delete_dead_threads(struct machine *machine)
+{
+	struct rb_node *nd = rb_first(&machine->dead_threads);
+
+	while (nd) {
+		struct thread *t = rb_entry(nd, struct thread, rb_node);
+		struct thread *pos;
+
+		nd = rb_next(nd);
+		rb_erase(&t->rb_node, &machine->dead_threads);
+		RB_CLEAR_NODE(&t->rb_node);
+
+		while (!list_empty(&t->tid_node)) {
+			pos = list_first_entry(&t->tid_node,
+					       struct thread, tid_node);
+			list_del_init(&pos->tid_node);
+			thread__delete(pos);
+		}
+
+		thread__delete(t);
+	}
+}
+
 void machine__delete_threads(struct machine *machine)
 {
 	struct rb_node *nd;
@@ -106,6 +129,8 @@ void machine__delete_threads(struct machine *machine)
 		__machine__remove_thread(machine, t, false);
 	}
 	pthread_rwlock_unlock(&machine->threads_lock);
+
+	machine__delete_dead_threads(machine);
 }
 
 void machine__exit(struct machine *machine)
@@ -1308,6 +1333,10 @@ int machine__process_mmap_event(struct machine *machine, union perf_event *event
 
 static void __machine__remove_thread(struct machine *machine, struct thread *th, bool lock)
 {
+	struct rb_node **p = &machine->dead_threads.rb_node;
+	struct rb_node *parent = NULL;
+	struct thread *pos;
+
 	if (machine->last_match == th)
 		machine->last_match = NULL;
 
@@ -1316,15 +1345,43 @@ static void __machine__remove_thread(struct machine *machine, struct thread *th,
 		pthread_rwlock_wrlock(&machine->threads_lock);
 	rb_erase(&th->rb_node, &machine->threads);
 	RB_CLEAR_NODE(&th->rb_node);
+
+	th->dead = true;
+
+	/*
+	 * No need to have an additional reference for non-index file.
+	 */
+	if (!perf_has_index) {
+		thread__put(th);
+		goto out;
+	}
+
 	/*
-	 * Move it first to the dead_threads list, then drop the reference,
-	 * if this is the last reference, then the thread__delete destructor
-	 * will be called and we will remove it from the dead_threads list.
+	 * For indexed file, We may have references to this (dead)
+	 * thread, as samples are processed after fork/exit events.
+	 * Just move them to a separate rbtree.
 	 */
-	list_add_tail(&th->node, &machine->dead_threads);
+	while (*p != NULL) {
+		parent = *p;
+		pos = rb_entry(parent, struct thread, rb_node);
+
+		if (pos->tid == th->tid) {
+			list_add_tail(&th->tid_node, &pos->tid_node);
+			goto out;
+		}
+
+		if (th->tid < pos->tid)
+			p = &(*p)->rb_left;
+		else
+			p = &(*p)->rb_right;
+	}
+
+	rb_link_node(&th->rb_node, parent, p);
+	rb_insert_color(&th->rb_node, &machine->dead_threads);
+
+out:
 	if (lock)
 		pthread_rwlock_unlock(&machine->threads_lock);
-	thread__put(th);
 }
 
 void machine__remove_thread(struct machine *machine, struct thread *th)
@@ -1826,7 +1883,7 @@ int machine__for_each_thread(struct machine *machine,
 			     void *priv)
 {
 	struct rb_node *nd;
-	struct thread *thread;
+	struct thread *thread, *pos;
 	int rc = 0;
 
 	for (nd = rb_first(&machine->threads); nd; nd = rb_next(nd)) {
@@ -1836,10 +1893,17 @@ int machine__for_each_thread(struct machine *machine,
 			return rc;
 	}
 
-	list_for_each_entry(thread, &machine->dead_threads, node) {
+	for (nd = rb_first(&machine->dead_threads); nd; nd = rb_next(nd)) {
+		thread = rb_entry(nd, struct thread, rb_node);
 		rc = fn(thread, priv);
 		if (rc != 0)
 			return rc;
+
+		list_for_each_entry(pos, &thread->tid_node, tid_node) {
+			rc = fn(pos, priv);
+			if (rc != 0)
+				return rc;
+		}
 	}
 	return rc;
 }
diff --git a/tools/perf/util/machine.h b/tools/perf/util/machine.h
index c7963c63c474..53cdd1aad3ff 100644
--- a/tools/perf/util/machine.h
+++ b/tools/perf/util/machine.h
@@ -30,8 +30,8 @@ struct machine {
 	bool		  comm_exec;
 	char		  *root_dir;
 	struct rb_root	  threads;
+	struct rb_root	  dead_threads;
 	pthread_rwlock_t  threads_lock;
-	struct list_head  dead_threads;
 	struct thread	  *last_match;
 	struct vdso_info  *vdso_info;
 	struct dsos	  user_dsos;
diff --git a/tools/perf/util/thread.c b/tools/perf/util/thread.c
index 962558024415..ccf1808348fe 100644
--- a/tools/perf/util/thread.c
+++ b/tools/perf/util/thread.c
@@ -9,6 +9,7 @@
 #include "debug.h"
 #include "comm.h"
 #include "unwind.h"
+#include "machine.h"
 
 int thread__init_map_groups(struct thread *thread, struct machine *machine)
 {
@@ -38,6 +39,7 @@ struct thread *thread__new(pid_t pid, pid_t tid)
 		thread->ppid = -1;
 		thread->cpu = -1;
 		INIT_LIST_HEAD(&thread->comm_list);
+		INIT_LIST_HEAD(&thread->tid_node);
 
 		if (unwind__prepare_access(thread) < 0)
 			goto err_thread;
@@ -54,7 +56,7 @@ struct thread *thread__new(pid_t pid, pid_t tid)
 
 		list_add(&comm->list, &thread->comm_list);
 		atomic_set(&thread->refcnt, 0);
-		INIT_LIST_HEAD(&thread->node);
+		INIT_LIST_HEAD(&thread->tid_node);
 		RB_CLEAR_NODE(&thread->rb_node);
 	}
 
@@ -70,7 +72,7 @@ void thread__delete(struct thread *thread)
 	struct comm *comm, *tmp;
 
 	BUG_ON(!RB_EMPTY_NODE(&thread->rb_node));
-	BUG_ON(!list_empty(&thread->node));
+	BUG_ON(!list_empty(&thread->tid_node));
 
 	thread_stack__free(thread);
 
@@ -97,7 +99,20 @@ struct thread *thread__get(struct thread *thread)
 void thread__put(struct thread *thread)
 {
 	if (thread && atomic_dec_and_test(&thread->refcnt)) {
-		list_del_init(&thread->node);
+		if (!RB_EMPTY_NODE(&thread->rb_node)) {
+			struct machine *machine = thread->mg->machine;
+
+			if (thread->dead) {
+				rb_erase(&thread->rb_node,
+					 &machine->dead_threads);
+			} else {
+				rb_erase(&thread->rb_node,
+					 &machine->threads);
+			}
+			RB_CLEAR_NODE(&thread->rb_node);
+		}
+
+		list_del_init(&thread->tid_node);
 		thread__delete(thread);
 	}
 }
diff --git a/tools/perf/util/thread.h b/tools/perf/util/thread.h
index 903cfaf2628d..d6f6f150f3ff 100644
--- a/tools/perf/util/thread.h
+++ b/tools/perf/util/thread.h
@@ -13,10 +13,8 @@
 struct thread_stack;
 
 struct thread {
-	union {
-		struct rb_node	 rb_node;
-		struct list_head node;
-	};
+	struct rb_node	 	rb_node;
+	struct list_head 	tid_node;
 	struct map_groups	*mg;
 	pid_t			pid_; /* Not all tools update this */
 	pid_t			tid;
@@ -25,7 +23,8 @@ struct thread {
 	atomic_t		refcnt;
 	char			shortname[3];
 	bool			comm_set;
-	bool			dead; /* if set thread has exited */
+	bool			exited; /* if set thread has exited */
+	bool			dead; /* thread is in dead_threads list */
 	struct list_head	comm_list;
 	int			comm_len;
 	u64			db_id;
@@ -54,7 +53,7 @@ static inline void __thread__zput(struct thread **thread)
 
 static inline void thread__exited(struct thread *thread)
 {
-	thread->dead = true;
+	thread->exited = true;
 }
 
 int __thread__set_comm(struct thread *thread, const char *comm, u64 timestamp,
-- 
2.4.0


^ permalink raw reply related	[flat|nested] 221+ messages in thread

* [PATCH 14/40] perf tools: Introduce machine__find*_thread_by_time()
  2015-05-18  0:30 [RFC/PATCHSET 00/40] perf tools: Speed-up perf report by using multi thread (v4) Namhyung Kim
                   ` (12 preceding siblings ...)
  2015-05-18  0:30 ` [PATCH 13/40] perf tools: Convert dead thread list into rbtree Namhyung Kim
@ 2015-05-18  0:30 ` Namhyung Kim
  2015-05-18 19:50   ` Jiri Olsa
  2015-05-18  0:30 ` [PATCH 15/40] perf tools: Add a test case for timed thread handling Namhyung Kim
                   ` (25 subsequent siblings)
  39 siblings, 1 reply; 221+ messages in thread
From: Namhyung Kim @ 2015-05-18  0:30 UTC (permalink / raw)
  To: Arnaldo Carvalho de Melo
  Cc: Ingo Molnar, Peter Zijlstra, Jiri Olsa, LKML, David Ahern,
	Adrian Hunter, Andi Kleen, Frederic Weisbecker, Stephane Eranian

With data file indexing is enabled, it needs to search thread based on
sample time since sample processing is done after other (task, comm and
mmap) events are processed.  This can be a problem if a session is very
long and pid is recycled - in that case it'll only see the last one.

So keep thread start time in it, and search thread based on the time.
This patch introduces machine__find{,new}_thread_by_time() function
for this.  It'll first search current thread rbtree and then dead
thread tree and list.  If it couldn't find anyone, it'll create a new
thread.

The sample timestamp of 0 means that this is called from synthesized
event so just use current rbtree.  The timestamp will be -1 if sample
didn't record the timestamp so will see current threads automatically.

Cc: Frederic Weisbecker <fweisbec@gmail.com>
Signed-off-by: Namhyung Kim <namhyung@kernel.org>
---
 tools/perf/tests/dwarf-unwind.c |   8 +--
 tools/perf/tests/hists_common.c |   3 +-
 tools/perf/tests/hists_link.c   |   2 +-
 tools/perf/util/event.c         |   6 +-
 tools/perf/util/machine.c       | 126 +++++++++++++++++++++++++++++++++++++++-
 tools/perf/util/machine.h       |  10 +++-
 tools/perf/util/thread.c        |   4 ++
 tools/perf/util/thread.h        |   1 +
 8 files changed, 148 insertions(+), 12 deletions(-)

diff --git a/tools/perf/tests/dwarf-unwind.c b/tools/perf/tests/dwarf-unwind.c
index 9b748e1ad46e..1926799bfcdb 100644
--- a/tools/perf/tests/dwarf-unwind.c
+++ b/tools/perf/tests/dwarf-unwind.c
@@ -16,10 +16,10 @@
 
 static int mmap_handler(struct perf_tool *tool __maybe_unused,
 			union perf_event *event,
-			struct perf_sample *sample __maybe_unused,
+			struct perf_sample *sample,
 			struct machine *machine)
 {
-	return machine__process_mmap2_event(machine, event, NULL);
+	return machine__process_mmap2_event(machine, event, sample);
 }
 
 static int init_live_machine(struct machine *machine)
@@ -66,12 +66,10 @@ static int unwind_entry(struct unwind_entry *entry, void *arg)
 __attribute__ ((noinline))
 static int unwind_thread(struct thread *thread)
 {
-	struct perf_sample sample;
+	struct perf_sample sample = { .time = -1ULL, };
 	unsigned long cnt = 0;
 	int err = -1;
 
-	memset(&sample, 0, sizeof(sample));
-
 	if (test__arch_unwind_sample(&sample, thread)) {
 		pr_debug("failed to get unwind sample\n");
 		goto out;
diff --git a/tools/perf/tests/hists_common.c b/tools/perf/tests/hists_common.c
index 456f884eb27b..2fd9cb71b258 100644
--- a/tools/perf/tests/hists_common.c
+++ b/tools/perf/tests/hists_common.c
@@ -80,6 +80,7 @@ static struct {
 struct machine *setup_fake_machine(struct machines *machines)
 {
 	struct machine *machine = machines__find(machines, HOST_KERNEL_ID);
+	struct perf_sample sample = { .time = -1ULL, };
 	size_t i;
 
 	if (machine == NULL) {
@@ -114,7 +115,7 @@ struct machine *setup_fake_machine(struct machines *machines)
 		strcpy(fake_mmap_event.mmap.filename,
 		       fake_mmap_info[i].filename);
 
-		machine__process_mmap_event(machine, &fake_mmap_event, NULL);
+		machine__process_mmap_event(machine, &fake_mmap_event, &sample);
 	}
 
 	for (i = 0; i < ARRAY_SIZE(fake_symbols); i++) {
diff --git a/tools/perf/tests/hists_link.c b/tools/perf/tests/hists_link.c
index 27bae90c9a95..cacc8617bf02 100644
--- a/tools/perf/tests/hists_link.c
+++ b/tools/perf/tests/hists_link.c
@@ -64,7 +64,7 @@ static int add_hist_entries(struct perf_evlist *evlist, struct machine *machine)
 	struct perf_evsel *evsel;
 	struct addr_location al;
 	struct hist_entry *he;
-	struct perf_sample sample = { .period = 1, };
+	struct perf_sample sample = { .period = 1, .time = -1ULL, };
 	size_t i = 0, k;
 
 	/*
diff --git a/tools/perf/util/event.c b/tools/perf/util/event.c
index a513a51f7330..819a2d75411c 100644
--- a/tools/perf/util/event.c
+++ b/tools/perf/util/event.c
@@ -9,6 +9,7 @@
 #include "strlist.h"
 #include "thread.h"
 #include "thread_map.h"
+#include "session.h"
 #include "symbol/kallsyms.h"
 
 static const char *perf_event__names[] = {
@@ -929,9 +930,10 @@ int perf_event__preprocess_sample(const union perf_event *event,
 				  struct perf_sample *sample)
 {
 	u8 cpumode = event->header.misc & PERF_RECORD_MISC_CPUMODE_MASK;
-	struct thread *thread = machine__findnew_thread(machine, sample->pid,
-							sample->tid);
+	struct thread *thread;
 
+	thread = machine__findnew_thread_by_time(machine, sample->pid,
+						 sample->tid, sample->time);
 	if (thread == NULL)
 		return -1;
 
diff --git a/tools/perf/util/machine.c b/tools/perf/util/machine.c
index ae07b84a40f5..9e2f4e8663d5 100644
--- a/tools/perf/util/machine.c
+++ b/tools/perf/util/machine.c
@@ -466,6 +466,120 @@ struct thread *machine__find_thread(struct machine *machine, pid_t pid,
 	return th;
 }
 
+static struct thread *
+__machine__findnew_thread_by_time(struct machine *machine, pid_t pid, pid_t tid,
+				  u64 timestamp, bool create)
+{
+	struct thread *curr, *pos, *new;
+	struct thread *th = NULL;
+	struct rb_node **p;
+	struct rb_node *parent = NULL;
+
+	if (!perf_has_index)
+		return ____machine__findnew_thread(machine, pid, tid, create);
+
+	/* lookup current thread first */
+	curr = ____machine__findnew_thread(machine, pid, tid, false);
+	if (curr && timestamp >= curr->start_time)
+		return curr;
+
+	/* and then check dead threads tree & list */
+	p = &machine->dead_threads.rb_node;
+	while (*p != NULL) {
+		parent = *p;
+		th = rb_entry(parent, struct thread, rb_node);
+
+		if (th->tid == tid) {
+			list_for_each_entry(pos, &th->tid_node, tid_node) {
+				if (timestamp >= pos->start_time &&
+				    pos->start_time > th->start_time) {
+					th = pos;
+					break;
+				}
+			}
+
+			if (timestamp >= th->start_time) {
+				machine__update_thread_pid(machine, th, pid);
+				return th;
+			}
+			break;
+		}
+
+		if (tid < th->tid)
+			p = &(*p)->rb_left;
+		else
+			p = &(*p)->rb_right;
+	}
+
+	if (!create)
+		return NULL;
+
+	if (!curr && !*p) {
+		/* found no thread.  create one as current thread */
+		return __machine__findnew_thread(machine, pid, tid);
+	}
+
+	new = thread__new(pid, tid);
+	if (new == NULL)
+		return NULL;
+
+	new->dead = true;
+	new->start_time = timestamp;
+
+	if (*p) {
+		list_for_each_entry(pos, &th->tid_node, tid_node) {
+			/* sort by time */
+			if (timestamp >= pos->start_time) {
+				th = pos;
+				break;
+			}
+		}
+		list_add_tail(&new->tid_node, &th->tid_node);
+	} else {
+		rb_link_node(&new->rb_node, parent, p);
+		rb_insert_color(&new->rb_node, &machine->dead_threads);
+	}
+
+	thread__get(new);
+
+	/*
+	 * We have to initialize map_groups separately
+	 * after rb tree is updated.
+	 *
+	 * The reason is that we call machine__findnew_thread
+	 * within thread__init_map_groups to find the thread
+	 * leader and that would screwed the rb tree.
+	 */
+	if (thread__init_map_groups(new, machine))
+		thread__zput(new);
+
+	return new;
+}
+
+struct thread *machine__find_thread_by_time(struct machine *machine, pid_t pid,
+					    pid_t tid, u64 timestamp)
+{
+	struct thread *th;
+
+	pthread_rwlock_rdlock(&machine->threads_lock);
+	th = thread__get(__machine__findnew_thread_by_time(machine, pid, tid,
+							   timestamp, false));
+	pthread_rwlock_unlock(&machine->threads_lock);
+	return th;
+}
+
+struct thread *machine__findnew_thread_by_time(struct machine *machine, pid_t pid,
+					       pid_t tid, u64 timestamp)
+{
+	struct thread *th;
+
+	pthread_rwlock_wrlock(&machine->threads_lock);
+	th = thread__get(__machine__findnew_thread_by_time(machine, pid, tid,
+							   timestamp, true));
+	pthread_rwlock_unlock(&machine->threads_lock);
+	return th;
+}
+
 struct comm *machine__thread_exec_comm(struct machine *machine,
 				       struct thread *thread)
 {
@@ -1249,7 +1363,7 @@ int machine__process_mmap2_event(struct machine *machine,
 	}
 
 	thread = machine__findnew_thread(machine, event->mmap2.pid,
-					event->mmap2.tid);
+					 event->mmap2.tid);
 	if (thread == NULL)
 		goto out_problem;
 
@@ -1366,6 +1480,16 @@ static void __machine__remove_thread(struct machine *machine, struct thread *th,
 		pos = rb_entry(parent, struct thread, rb_node);
 
 		if (pos->tid == th->tid) {
+			struct thread *old;
+
+			/* sort by time */
+			list_for_each_entry(old, &pos->tid_node, tid_node) {
+				if (th->start_time >= old->start_time) {
+					pos = old;
+					break;
+				}
+			}
+
 			list_add_tail(&th->tid_node, &pos->tid_node);
 			goto out;
 		}
diff --git a/tools/perf/util/machine.h b/tools/perf/util/machine.h
index 53cdd1aad3ff..863c0e90ceb1 100644
--- a/tools/perf/util/machine.h
+++ b/tools/perf/util/machine.h
@@ -69,8 +69,6 @@ static inline bool machine__kernel_ip(struct machine *machine, u64 ip)
 	return ip >= kernel_start;
 }
 
-struct thread *machine__find_thread(struct machine *machine, pid_t pid,
-				    pid_t tid);
 struct comm *machine__thread_exec_comm(struct machine *machine,
 				       struct thread *thread);
 
@@ -154,6 +152,14 @@ static inline bool machine__is_host(struct machine *machine)
 
 struct thread *__machine__findnew_thread(struct machine *machine, pid_t pid, pid_t tid);
 struct thread *machine__findnew_thread(struct machine *machine, pid_t pid, pid_t tid);
+struct thread *machine__find_thread(struct machine *machine, pid_t pid,
+				    pid_t tid);
+struct thread *machine__findnew_thread_by_time(struct machine *machine,
+					       pid_t pid, pid_t tid,
+					       u64 timestamp);
+struct thread *machine__find_thread_by_time(struct machine *machine,
+					    pid_t pid, pid_t tid,
+					    u64 timestamp);
 
 size_t machine__fprintf(struct machine *machine, FILE *fp);
 
diff --git a/tools/perf/util/thread.c b/tools/perf/util/thread.c
index ccf1808348fe..c8c927488ea0 100644
--- a/tools/perf/util/thread.c
+++ b/tools/perf/util/thread.c
@@ -161,6 +161,9 @@ int __thread__set_comm(struct thread *thread, const char *str, u64 timestamp,
 
 	/* Override the default :tid entry */
 	if (!thread->comm_set) {
+		if (!thread->start_time)
+			thread->start_time = timestamp;
+
 		err = comm__override(curr, str, timestamp, exec);
 		if (err)
 			return err;
@@ -261,6 +264,7 @@ int thread__fork(struct thread *thread, struct thread *parent, u64 timestamp)
 	}
 
 	thread->ppid = parent->tid;
+	thread->start_time = timestamp;
 	return thread__clone_map_groups(thread, parent);
 }
 
diff --git a/tools/perf/util/thread.h b/tools/perf/util/thread.h
index d6f6f150f3ff..60250db7f77b 100644
--- a/tools/perf/util/thread.h
+++ b/tools/perf/util/thread.h
@@ -28,6 +28,7 @@ struct thread {
 	struct list_head	comm_list;
 	int			comm_len;
 	u64			db_id;
+	u64			start_time;
 
 	void			*priv;
 	struct thread_stack	*ts;
-- 
2.4.0


^ permalink raw reply related	[flat|nested] 221+ messages in thread

* [PATCH 15/40] perf tools: Add a test case for timed thread handling
  2015-05-18  0:30 [RFC/PATCHSET 00/40] perf tools: Speed-up perf report by using multi thread (v4) Namhyung Kim
                   ` (13 preceding siblings ...)
  2015-05-18  0:30 ` [PATCH 14/40] perf tools: Introduce machine__find*_thread_by_time() Namhyung Kim
@ 2015-05-18  0:30 ` Namhyung Kim
  2015-05-18  0:30 ` [PATCH 16/40] perf tools: Reducing arguments of hist_entry_iter__add() Namhyung Kim
                   ` (24 subsequent siblings)
  39 siblings, 0 replies; 221+ messages in thread
From: Namhyung Kim @ 2015-05-18  0:30 UTC (permalink / raw)
  To: Arnaldo Carvalho de Melo
  Cc: Ingo Molnar, Peter Zijlstra, Jiri Olsa, LKML, David Ahern,
	Adrian Hunter, Andi Kleen, Frederic Weisbecker, Stephane Eranian

A test case for verifying live and dead thread tree management during
time change and new machine__find{,new}_thread_time().

Cc: Frederic Weisbecker <fweisbec@gmail.com>
Signed-off-by: Namhyung Kim <namhyung@kernel.org>
---
 tools/perf/tests/Build                |   1 +
 tools/perf/tests/builtin-test.c       |   4 +
 tools/perf/tests/tests.h              |   1 +
 tools/perf/tests/thread-lookup-time.c | 179 ++++++++++++++++++++++++++++++++++
 4 files changed, 185 insertions(+)
 create mode 100644 tools/perf/tests/thread-lookup-time.c

diff --git a/tools/perf/tests/Build b/tools/perf/tests/Build
index 78d29a3a6a97..5ad495823b49 100644
--- a/tools/perf/tests/Build
+++ b/tools/perf/tests/Build
@@ -26,6 +26,7 @@ perf-y += sw-clock.o
 perf-y += mmap-thread-lookup.o
 perf-y += thread-comm.o
 perf-y += thread-mg-share.o
+perf-y += thread-lookup-time.o
 perf-y += switch-tracking.o
 perf-y += keep-tracking.o
 perf-y += code-reading.o
diff --git a/tools/perf/tests/builtin-test.c b/tools/perf/tests/builtin-test.c
index 372b6395a448..e83c7ce1b38a 100644
--- a/tools/perf/tests/builtin-test.c
+++ b/tools/perf/tests/builtin-test.c
@@ -175,6 +175,10 @@ static struct test {
 		.func = test__thread_comm,
 	},
 	{
+		.desc = "Test thread lookup with time",
+		.func = test__thread_lookup_time,
+	},
+	{
 		.func = NULL,
 	},
 };
diff --git a/tools/perf/tests/tests.h b/tools/perf/tests/tests.h
index aa269eff798a..e9aa78c3d8fc 100644
--- a/tools/perf/tests/tests.h
+++ b/tools/perf/tests/tests.h
@@ -62,6 +62,7 @@ int test__fdarray__filter(void);
 int test__fdarray__add(void);
 int test__kmod_path__parse(void);
 int test__thread_comm(void);
+int test__thread_lookup_time(void);
 
 #if defined(__x86_64__) || defined(__i386__) || defined(__arm__)
 #ifdef HAVE_DWARF_UNWIND_SUPPORT
diff --git a/tools/perf/tests/thread-lookup-time.c b/tools/perf/tests/thread-lookup-time.c
new file mode 100644
index 000000000000..0133a241b9fc
--- /dev/null
+++ b/tools/perf/tests/thread-lookup-time.c
@@ -0,0 +1,179 @@
+#include "tests.h"
+#include "machine.h"
+#include "thread.h"
+#include "map.h"
+#include "debug.h"
+
+static int thread__print_cb(struct thread *th, void *arg __maybe_unused)
+{
+	printf("thread: %d, start time: %"PRIu64" %s\n",
+	       th->tid, th->start_time,
+	       th->dead ? "(dead)" : th->exited ? "(exited)" : "");
+	return 0;
+}
+
+static int lookup_with_timestamp(struct machine *machine)
+{
+	struct thread *t1, *t2, *t3;
+	union perf_event fork_event = {
+		.fork = {
+			.pid = 0,
+			.tid = 0,
+			.ppid = 1,
+			.ptid = 1,
+		},
+	};
+	struct perf_sample sample = {
+		.time = 50000,
+	};
+
+	/* this is needed to keep dead threads in rbtree */
+	perf_has_index = true;
+
+	/* start_time is set to 0 */
+	t1 = machine__findnew_thread(machine, 0, 0);
+
+	if (verbose > 1) {
+		printf("========= after t1 created ==========\n");
+		machine__for_each_thread(machine, thread__print_cb, NULL);
+	}
+
+	TEST_ASSERT_VAL("wrong start time of old thread", t1->start_time == 0);
+
+	TEST_ASSERT_VAL("cannot find current thread",
+			machine__find_thread(machine, 0, 0) == t1);
+
+	TEST_ASSERT_VAL("cannot find current thread with time",
+			machine__findnew_thread_by_time(machine, 0, 0, 10000) == t1);
+
+	/* start_time is overwritten to new value */
+	thread__set_comm(t1, "/usr/bin/perf", 20000);
+
+	if (verbose > 1) {
+		printf("========= after t1 set comm ==========\n");
+		machine__for_each_thread(machine, thread__print_cb, NULL);
+	}
+
+	TEST_ASSERT_VAL("failed to update start time", t1->start_time == 20000);
+
+	TEST_ASSERT_VAL("should not find passed thread",
+			/* this will create yet another dead thread */
+			machine__findnew_thread_by_time(machine, 0, 0, 10000) != t1);
+
+	TEST_ASSERT_VAL("cannot find overwritten thread with time",
+			machine__find_thread_by_time(machine, 0, 0, 20000) == t1);
+
+	/* now t1 goes to dead thread tree, and create t2 */
+	machine__process_fork_event(machine, &fork_event, &sample);
+
+	if (verbose > 1) {
+		printf("========= after t2 forked ==========\n");
+		machine__for_each_thread(machine, thread__print_cb, NULL);
+	}
+
+	t2 = machine__find_thread(machine, 0, 0);
+
+	TEST_ASSERT_VAL("cannot find current thread", t2 != NULL);
+
+	TEST_ASSERT_VAL("wrong start time of new thread", t2->start_time == 50000);
+
+	TEST_ASSERT_VAL("dead thread cannot be found",
+			machine__find_thread_by_time(machine, 0, 0, 10000) != t1);
+
+	TEST_ASSERT_VAL("cannot find dead thread after new thread",
+			machine__find_thread_by_time(machine, 0, 0, 30000) == t1);
+
+	TEST_ASSERT_VAL("cannot find current thread after new thread",
+			machine__find_thread_by_time(machine, 0, 0, 50000) == t2);
+
+	/* now t2 goes to dead thread tree, and create t3 */
+	sample.time = 60000;
+	machine__process_fork_event(machine, &fork_event, &sample);
+
+	if (verbose > 1) {
+		printf("========= after t3 forked ==========\n");
+		machine__for_each_thread(machine, thread__print_cb, NULL);
+	}
+
+	t3 = machine__find_thread(machine, 0, 0);
+	TEST_ASSERT_VAL("cannot find current thread", t3 != NULL);
+
+	TEST_ASSERT_VAL("wrong start time of new thread", t3->start_time == 60000);
+
+	TEST_ASSERT_VAL("cannot find dead thread after new thread",
+			machine__findnew_thread_by_time(machine, 0, 0, 30000) == t1);
+
+	TEST_ASSERT_VAL("cannot find dead thread after new thread",
+			machine__findnew_thread_by_time(machine, 0, 0, 50000) == t2);
+
+	TEST_ASSERT_VAL("cannot find current thread after new thread",
+			machine__findnew_thread_by_time(machine, 0, 0, 70000) == t3);
+
+	machine__delete_threads(machine);
+	return 0;
+}
+
+static int lookup_without_timestamp(struct machine *machine)
+{
+	struct thread *t1, *t2, *t3;
+	union perf_event fork_event = {
+		.fork = {
+			.pid = 0,
+			.tid = 0,
+			.ppid = 1,
+			.ptid = 1,
+		},
+	};
+	struct perf_sample sample = {
+		.time = -1ULL,
+	};
+
+	t1 = machine__findnew_thread(machine, 0, 0);
+	TEST_ASSERT_VAL("cannot find current thread", t1 != NULL);
+
+	TEST_ASSERT_VAL("cannot find new thread with time",
+			machine__findnew_thread_by_time(machine, 0, 0, -1ULL) == t1);
+
+	machine__process_fork_event(machine, &fork_event, &sample);
+
+	t2 = machine__find_thread(machine, 0, 0);
+	TEST_ASSERT_VAL("cannot find current thread", t2 != NULL);
+
+	TEST_ASSERT_VAL("cannot find new thread with time",
+			machine__find_thread_by_time(machine, 0, 0, -1ULL) == t2);
+
+	machine__process_fork_event(machine, &fork_event, &sample);
+
+	t3 = machine__find_thread(machine, 0, 0);
+	TEST_ASSERT_VAL("cannot find current thread", t3 != NULL);
+
+	TEST_ASSERT_VAL("cannot find new thread with time",
+			machine__findnew_thread_by_time(machine, 0, 0, -1ULL) == t3);
+
+	machine__delete_threads(machine);
+	return 0;
+}
+
+int test__thread_lookup_time(void)
+{
+	struct machines machines;
+	struct machine *machine;
+
+	/*
+	 * This test is to check whether it can retrieve a correct
+	 * thread for a given time.  When multi-file data storage is
+	 * enabled, those task/comm/mmap events are processed first so
+	 * the later sample should find a matching thread properly.
+	 */
+	machines__init(&machines);
+	machine = &machines.host;
+
+	if (lookup_with_timestamp(machine) < 0)
+		return -1;
+
+	if (lookup_without_timestamp(machine) < 0)
+		return -1;
+
+	machines__exit(&machines);
+	return 0;
+}
-- 
2.4.0


^ permalink raw reply related	[flat|nested] 221+ messages in thread

* [PATCH 16/40] perf tools: Reducing arguments of hist_entry_iter__add()
  2015-05-18  0:30 [RFC/PATCHSET 00/40] perf tools: Speed-up perf report by using multi thread (v4) Namhyung Kim
                   ` (14 preceding siblings ...)
  2015-05-18  0:30 ` [PATCH 15/40] perf tools: Add a test case for timed thread handling Namhyung Kim
@ 2015-05-18  0:30 ` Namhyung Kim
  2015-05-18 12:55   ` Arnaldo Carvalho de Melo
  2015-05-18  0:30 ` [PATCH 17/40] perf tools: Maintain map groups list in a leader thread Namhyung Kim
                   ` (23 subsequent siblings)
  39 siblings, 1 reply; 221+ messages in thread
From: Namhyung Kim @ 2015-05-18  0:30 UTC (permalink / raw)
  To: Arnaldo Carvalho de Melo
  Cc: Ingo Molnar, Peter Zijlstra, Jiri Olsa, LKML, David Ahern,
	Adrian Hunter, Andi Kleen, Frederic Weisbecker, Stephane Eranian

The evsel and sample arguments are to set iter for later use.  As it
also receives an iter as another argument, just set them before
calling the function.

Signed-off-by: Namhyung Kim <namhyung@kernel.org>
---
 tools/perf/builtin-report.c       | 9 +++++----
 tools/perf/builtin-top.c          | 7 ++++---
 tools/perf/tests/hists_cumulate.c | 6 ++++--
 tools/perf/tests/hists_filter.c   | 4 +++-
 tools/perf/tests/hists_output.c   | 6 ++++--
 tools/perf/util/hist.c            | 8 ++------
 tools/perf/util/hist.h            | 1 -
 7 files changed, 22 insertions(+), 19 deletions(-)

diff --git a/tools/perf/builtin-report.c b/tools/perf/builtin-report.c
index fee770935eab..decd9e8584b5 100644
--- a/tools/perf/builtin-report.c
+++ b/tools/perf/builtin-report.c
@@ -139,8 +139,10 @@ static int process_sample_event(struct perf_tool *tool,
 	struct report *rep = container_of(tool, struct report, tool);
 	struct addr_location al;
 	struct hist_entry_iter iter = {
-		.hide_unresolved = rep->hide_unresolved,
-		.add_entry_cb = hist_iter__report_callback,
+		.evsel 			= evsel,
+		.sample 		= sample,
+		.hide_unresolved 	= rep->hide_unresolved,
+		.add_entry_cb 		= hist_iter__report_callback,
 	};
 	int ret = 0;
 
@@ -168,8 +170,7 @@ static int process_sample_event(struct perf_tool *tool,
 	if (al.map != NULL)
 		al.map->dso->hit = 1;
 
-	ret = hist_entry_iter__add(&iter, &al, evsel, sample, rep->max_stack,
-				   rep);
+	ret = hist_entry_iter__add(&iter, &al, rep->max_stack, rep);
 	if (ret < 0)
 		pr_debug("problem adding hist entry, skipping event\n");
 out_put:
diff --git a/tools/perf/builtin-top.c b/tools/perf/builtin-top.c
index a19351728f0f..6b987424d015 100644
--- a/tools/perf/builtin-top.c
+++ b/tools/perf/builtin-top.c
@@ -775,7 +775,9 @@ static void perf_event__process_sample(struct perf_tool *tool,
 	if (al.sym == NULL || !al.sym->ignore) {
 		struct hists *hists = evsel__hists(evsel);
 		struct hist_entry_iter iter = {
-			.add_entry_cb = hist_iter__top_callback,
+			.evsel		= evsel,
+			.sample 	= sample,
+			.add_entry_cb 	= hist_iter__top_callback,
 		};
 
 		if (symbol_conf.cumulate_callchain)
@@ -785,8 +787,7 @@ static void perf_event__process_sample(struct perf_tool *tool,
 
 		pthread_mutex_lock(&hists->lock);
 
-		err = hist_entry_iter__add(&iter, &al, evsel, sample,
-					   top->max_stack, top);
+		err = hist_entry_iter__add(&iter, &al, top->max_stack, top);
 		if (err < 0)
 			pr_err("Problem incrementing symbol period, skipping event\n");
 
diff --git a/tools/perf/tests/hists_cumulate.c b/tools/perf/tests/hists_cumulate.c
index 620f626e5b35..7d82c8be5e36 100644
--- a/tools/perf/tests/hists_cumulate.c
+++ b/tools/perf/tests/hists_cumulate.c
@@ -87,6 +87,8 @@ static int add_hist_entries(struct hists *hists, struct machine *machine)
 			},
 		};
 		struct hist_entry_iter iter = {
+			.evsel = evsel,
+			.sample	= &sample,
 			.hide_unresolved = false,
 		};
 
@@ -104,8 +106,8 @@ static int add_hist_entries(struct hists *hists, struct machine *machine)
 						  &sample) < 0)
 			goto out;
 
-		if (hist_entry_iter__add(&iter, &al, evsel, &sample,
-					 PERF_MAX_STACK_DEPTH, NULL) < 0) {
+		if (hist_entry_iter__add(&iter, &al, PERF_MAX_STACK_DEPTH,
+					 NULL) < 0) {
 			addr_location__put(&al);
 			goto out;
 		}
diff --git a/tools/perf/tests/hists_filter.c b/tools/perf/tests/hists_filter.c
index 82e1ee52e024..ce48775e6ada 100644
--- a/tools/perf/tests/hists_filter.c
+++ b/tools/perf/tests/hists_filter.c
@@ -63,6 +63,8 @@ static int add_hist_entries(struct perf_evlist *evlist,
 				},
 			};
 			struct hist_entry_iter iter = {
+				.evsel = evsel,
+				.sample = &sample,
 				.ops = &hist_iter_normal,
 				.hide_unresolved = false,
 			};
@@ -81,7 +83,7 @@ static int add_hist_entries(struct perf_evlist *evlist,
 							  &sample) < 0)
 				goto out;
 
-			if (hist_entry_iter__add(&iter, &al, evsel, &sample,
+			if (hist_entry_iter__add(&iter, &al,
 						 PERF_MAX_STACK_DEPTH, NULL) < 0) {
 				addr_location__put(&al);
 				goto out;
diff --git a/tools/perf/tests/hists_output.c b/tools/perf/tests/hists_output.c
index fd7ec4f9aeb4..adbebc852cc8 100644
--- a/tools/perf/tests/hists_output.c
+++ b/tools/perf/tests/hists_output.c
@@ -57,6 +57,8 @@ static int add_hist_entries(struct hists *hists, struct machine *machine)
 			},
 		};
 		struct hist_entry_iter iter = {
+			.evsel = evsel,
+			.sample = &sample,
 			.ops = &hist_iter_normal,
 			.hide_unresolved = false,
 		};
@@ -70,8 +72,8 @@ static int add_hist_entries(struct hists *hists, struct machine *machine)
 						  &sample) < 0)
 			goto out;
 
-		if (hist_entry_iter__add(&iter, &al, evsel, &sample,
-					 PERF_MAX_STACK_DEPTH, NULL) < 0) {
+		if (hist_entry_iter__add(&iter, &al, PERF_MAX_STACK_DEPTH,
+					 NULL) < 0) {
 			addr_location__put(&al);
 			goto out;
 		}
diff --git a/tools/perf/util/hist.c b/tools/perf/util/hist.c
index f13993e53e4e..b492968913e1 100644
--- a/tools/perf/util/hist.c
+++ b/tools/perf/util/hist.c
@@ -852,19 +852,15 @@ const struct hist_iter_ops hist_iter_cumulative = {
 };
 
 int hist_entry_iter__add(struct hist_entry_iter *iter, struct addr_location *al,
-			 struct perf_evsel *evsel, struct perf_sample *sample,
 			 int max_stack_depth, void *arg)
 {
 	int err, err2;
 
-	err = sample__resolve_callchain(sample, &iter->parent, evsel, al,
-					max_stack_depth);
+	err = sample__resolve_callchain(iter->sample, &iter->parent,
+					iter->evsel, al, max_stack_depth);
 	if (err)
 		return err;
 
-	iter->evsel = evsel;
-	iter->sample = sample;
-
 	err = iter->ops->prepare_entry(iter, al);
 	if (err)
 		goto out;
diff --git a/tools/perf/util/hist.h b/tools/perf/util/hist.h
index 43ffe62e25d2..d47f6730f6a4 100644
--- a/tools/perf/util/hist.h
+++ b/tools/perf/util/hist.h
@@ -111,7 +111,6 @@ struct hist_entry *__hists__add_entry(struct hists *hists,
 				      u64 weight, u64 transaction,
 				      u64 timestamp, bool sample_self);
 int hist_entry_iter__add(struct hist_entry_iter *iter, struct addr_location *al,
-			 struct perf_evsel *evsel, struct perf_sample *sample,
 			 int max_stack_depth, void *arg);
 
 int64_t hist_entry__cmp(struct hist_entry *left, struct hist_entry *right);
-- 
2.4.0


^ permalink raw reply related	[flat|nested] 221+ messages in thread

* [PATCH 17/40] perf tools: Maintain map groups list in a leader thread
  2015-05-18  0:30 [RFC/PATCHSET 00/40] perf tools: Speed-up perf report by using multi thread (v4) Namhyung Kim
                   ` (15 preceding siblings ...)
  2015-05-18  0:30 ` [PATCH 16/40] perf tools: Reducing arguments of hist_entry_iter__add() Namhyung Kim
@ 2015-05-18  0:30 ` Namhyung Kim
  2015-05-18  0:30 ` [PATCH 18/40] perf tools: Introduce thread__find_addr_location_by_time() and friends Namhyung Kim
                   ` (22 subsequent siblings)
  39 siblings, 0 replies; 221+ messages in thread
From: Namhyung Kim @ 2015-05-18  0:30 UTC (permalink / raw)
  To: Arnaldo Carvalho de Melo
  Cc: Ingo Molnar, Peter Zijlstra, Jiri Olsa, LKML, David Ahern,
	Adrian Hunter, Andi Kleen, Frederic Weisbecker, Stephane Eranian

To support multi-threaded perf report, we need to maintain time-sorted
map groups.  Add ->mg_list member to struct thread and sort the list
by time.  Now leader threads have one more refcnt for map groups in
the list so also update the thread-mg-share test case.

Currently only add a new map groups when an exec (comm) event is
received.

Cc: Frederic Weisbecker <fweisbec@gmail.com>
Signed-off-by: Namhyung Kim <namhyung@kernel.org>
---
 tools/perf/tests/thread-mg-share.c |   7 ++-
 tools/perf/util/event.c            |   2 +
 tools/perf/util/machine.c          |   4 +-
 tools/perf/util/map.c              |   3 ++
 tools/perf/util/map.h              |   2 +
 tools/perf/util/thread.c           | 108 ++++++++++++++++++++++++++++++++++++-
 tools/perf/util/thread.h           |   3 ++
 7 files changed, 124 insertions(+), 5 deletions(-)

diff --git a/tools/perf/tests/thread-mg-share.c b/tools/perf/tests/thread-mg-share.c
index c0ed56f7efc6..50a2c68d6379 100644
--- a/tools/perf/tests/thread-mg-share.c
+++ b/tools/perf/tests/thread-mg-share.c
@@ -23,6 +23,9 @@ int test__thread_mg_share(void)
 	 * with several threads and checks they properly share and
 	 * maintain map groups info (struct map_groups).
 	 *
+	 * Note that a leader thread has one more refcnt for its
+	 * (current) map groups.
+	 *
 	 * thread group (pid: 0, tids: 0, 1, 2, 3)
 	 * other  group (pid: 4, tids: 4, 5)
 	*/
@@ -43,7 +46,7 @@ int test__thread_mg_share(void)
 			leader && t1 && t2 && t3 && other);
 
 	mg = leader->mg;
-	TEST_ASSERT_EQUAL("wrong refcnt", mg->refcnt, 4);
+	TEST_ASSERT_EQUAL("wrong refcnt", mg->refcnt, 5);
 
 	/* test the map groups pointer is shared */
 	TEST_ASSERT_VAL("map groups don't match", mg == t1->mg);
@@ -71,7 +74,7 @@ int test__thread_mg_share(void)
 	machine__remove_thread(machine, other_leader);
 
 	other_mg = other->mg;
-	TEST_ASSERT_EQUAL("wrong refcnt", other_mg->refcnt, 2);
+	TEST_ASSERT_EQUAL("wrong refcnt", other_mg->refcnt, 3);
 
 	TEST_ASSERT_VAL("map groups don't match", other_mg == other_leader->mg);
 
diff --git a/tools/perf/util/event.c b/tools/perf/util/event.c
index 819a2d75411c..0ad76b06cd48 100644
--- a/tools/perf/util/event.c
+++ b/tools/perf/util/event.c
@@ -851,6 +851,8 @@ void thread__find_addr_map(struct thread *thread, u8 cpumode,
 		return;
 	}
 
+	BUG_ON(mg == NULL);
+
 	if (cpumode == PERF_RECORD_MISC_KERNEL && perf_host) {
 		al->level = 'k';
 		mg = &machine->kmaps;
diff --git a/tools/perf/util/machine.c b/tools/perf/util/machine.c
index 9e2f4e8663d5..99fb14926351 100644
--- a/tools/perf/util/machine.c
+++ b/tools/perf/util/machine.c
@@ -341,7 +341,7 @@ static void machine__update_thread_pid(struct machine *machine,
 		goto out_err;
 
 	if (!leader->mg)
-		leader->mg = map_groups__new(machine);
+		thread__set_map_groups(leader, map_groups__new(machine), 0);
 
 	if (!leader->mg)
 		goto out_err;
@@ -358,7 +358,7 @@ static void machine__update_thread_pid(struct machine *machine,
 		if (!map_groups__empty(th->mg))
 			pr_err("Discarding thread maps for %d:%d\n",
 			       th->pid_, th->tid);
-		map_groups__delete(th->mg);
+		map_groups__put(th->mg);
 	}
 
 	th->mg = map_groups__get(leader->mg);
diff --git a/tools/perf/util/map.c b/tools/perf/util/map.c
index cd0e335008b4..b794c3561995 100644
--- a/tools/perf/util/map.c
+++ b/tools/perf/util/map.c
@@ -427,6 +427,8 @@ void map_groups__init(struct map_groups *mg, struct machine *machine)
 	}
 	mg->machine = machine;
 	mg->refcnt = 1;
+	mg->timestamp = 0;
+	INIT_LIST_HEAD(&mg->list);
 }
 
 static void maps__delete(struct rb_root *maps)
@@ -489,6 +491,7 @@ struct map_groups *map_groups__new(struct machine *machine)
 void map_groups__delete(struct map_groups *mg)
 {
 	map_groups__exit(mg);
+	list_del(&mg->list);
 	free(mg);
 }
 
diff --git a/tools/perf/util/map.h b/tools/perf/util/map.h
index 4e0c729841ab..074453d332dd 100644
--- a/tools/perf/util/map.h
+++ b/tools/perf/util/map.h
@@ -61,7 +61,9 @@ struct map_groups {
 	struct rb_root	 maps[MAP__NR_TYPES];
 	struct list_head removed_maps[MAP__NR_TYPES];
 	struct machine	 *machine;
+	u64		 timestamp;
 	int		 refcnt;
+	struct list_head list;
 };
 
 struct map_groups *map_groups__new(struct machine *machine);
diff --git a/tools/perf/util/thread.c b/tools/perf/util/thread.c
index c8c927488ea0..fc4e51afaf18 100644
--- a/tools/perf/util/thread.c
+++ b/tools/perf/util/thread.c
@@ -11,13 +11,76 @@
 #include "unwind.h"
 #include "machine.h"
 
+struct map_groups *thread__get_map_groups(struct thread *thread, u64 timestamp)
+{
+	struct map_groups *mg;
+	struct thread *leader = thread;
+
+	BUG_ON(thread->mg == NULL);
+
+	if (thread->tid != thread->pid_) {
+		leader = machine__find_thread_by_time(thread->mg->machine,
+						      thread->pid_, thread->pid_,
+						      timestamp);
+		if (leader == NULL)
+			goto out;
+	}
+
+	list_for_each_entry(mg, &leader->mg_list, list)
+		if (timestamp >= mg->timestamp)
+			return mg;
+
+out:
+	return thread->mg;
+}
+
+int thread__set_map_groups(struct thread *thread, struct map_groups *mg,
+			   u64 timestamp)
+{
+	struct list_head *pos;
+	struct map_groups *old;
+
+	if (mg == NULL)
+		return -ENOMEM;
+
+	/*
+	 * Only a leader thread can have map groups list - others
+	 * reference it through map_groups__get.  This means the
+	 * leader thread will have one more refcnt than others.
+	 */
+	if (thread->tid != thread->pid_)
+		return -EINVAL;
+
+	if (thread->mg) {
+		BUG_ON(thread->mg->refcnt <= 1);
+		map_groups__put(thread->mg);
+	}
+
+	/* sort by time */
+	list_for_each(pos, &thread->mg_list) {
+		old = list_entry(pos, struct map_groups, list);
+		if (timestamp > old->timestamp)
+			break;
+	}
+
+	list_add_tail(&mg->list, pos);
+	mg->timestamp = timestamp;
+
+	/* set current ->mg to most recent one */
+	thread->mg = list_first_entry(&thread->mg_list, struct map_groups, list);
+	/* increase one more refcnt for current */
+	map_groups__get(thread->mg);
+
+	return 0;
+}
+
 int thread__init_map_groups(struct thread *thread, struct machine *machine)
 {
 	struct thread *leader;
 	pid_t pid = thread->pid_;
 
 	if (pid == thread->tid || pid == -1) {
-		thread->mg = map_groups__new(machine);
+		thread__set_map_groups(thread, map_groups__new(machine), 0);
 	} else {
 		leader = __machine__findnew_thread(machine, pid, pid);
 		if (leader)
@@ -40,6 +103,7 @@ struct thread *thread__new(pid_t pid, pid_t tid)
 		thread->cpu = -1;
 		INIT_LIST_HEAD(&thread->comm_list);
 		INIT_LIST_HEAD(&thread->tid_node);
+		INIT_LIST_HEAD(&thread->mg_list);
 
 		if (unwind__prepare_access(thread) < 0)
 			goto err_thread;
@@ -70,6 +134,7 @@ struct thread *thread__new(pid_t pid, pid_t tid)
 void thread__delete(struct thread *thread)
 {
 	struct comm *comm, *tmp;
+	struct map_groups *mg, *tmp_mg;
 
 	BUG_ON(!RB_EMPTY_NODE(&thread->rb_node));
 	BUG_ON(!list_empty(&thread->tid_node));
@@ -80,6 +145,10 @@ void thread__delete(struct thread *thread)
 		map_groups__put(thread->mg);
 		thread->mg = NULL;
 	}
+	/* only leader threads have mg list */
+	list_for_each_entry_safe(mg, tmp_mg, &thread->mg_list, list)
+		map_groups__put(mg);
+
 	list_for_each_entry_safe(comm, tmp, &thread->comm_list, list) {
 		list_del(&comm->list);
 		comm__free(comm);
@@ -153,6 +222,9 @@ struct comm *thread__comm_by_time(const struct thread *thread, u64 timestamp)
 	return list_last_entry(&thread->comm_list, struct comm, list);
 }
 
+static int thread__clone_map_groups(struct thread *thread,
+				    struct thread *parent);
+
 int __thread__set_comm(struct thread *thread, const char *str, u64 timestamp,
 		       bool exec)
 {
@@ -183,6 +255,40 @@ int __thread__set_comm(struct thread *thread, const char *str, u64 timestamp,
 			unwind__flush_access(thread);
 	}
 
+	if (exec) {
+		struct machine *machine;
+
+		BUG_ON(thread->mg == NULL || thread->mg->machine == NULL);
+
+		machine = thread->mg->machine;
+
+		if (thread->tid != thread->pid_) {
+			struct map_groups *old = thread->mg;
+			struct thread *leader;
+
+			leader = machine__findnew_thread(machine, thread->pid_,
+							 thread->pid_);
+
+			/* now it'll be a new leader */
+			thread->pid_ = thread->tid;
+
+			thread->mg = map_groups__new(old->machine);
+			if (thread->mg == NULL)
+				return -ENOMEM;
+
+			/* save current mg in the new leader */
+			thread__clone_map_groups(thread, leader);
+
+			/* current mg of leader thread needs one more refcnt */
+			map_groups__get(thread->mg);
+
+			thread__set_map_groups(thread, thread->mg, old->timestamp);
+		}
+
+		/* create a new mg for newly executed binary */
+		thread__set_map_groups(thread, map_groups__new(machine), timestamp);
+	}
+
 	thread->comm_set = true;
 
 	return 0;
diff --git a/tools/perf/util/thread.h b/tools/perf/util/thread.h
index 60250db7f77b..468bb785f061 100644
--- a/tools/perf/util/thread.h
+++ b/tools/perf/util/thread.h
@@ -16,6 +16,7 @@ struct thread {
 	struct rb_node	 	rb_node;
 	struct list_head 	tid_node;
 	struct map_groups	*mg;
+	struct list_head	mg_list;
 	pid_t			pid_; /* Not all tools update this */
 	pid_t			tid;
 	pid_t			ppid;
@@ -71,6 +72,8 @@ struct comm *thread__exec_comm(const struct thread *thread);
 struct comm *thread__comm_by_time(const struct thread *thread, u64 timestamp);
 const char *thread__comm_str(const struct thread *thread);
 const char *thread__comm_str_by_time(const struct thread *thread, u64 timestamp);
+struct map_groups *thread__get_map_groups(struct thread *thread, u64 timestamp);
+int thread__set_map_groups(struct thread *thread, struct map_groups *mg, u64 timestamp);
 void thread__insert_map(struct thread *thread, struct map *map);
 int thread__fork(struct thread *thread, struct thread *parent, u64 timestamp);
 size_t thread__fprintf(struct thread *thread, FILE *fp);
-- 
2.4.0


^ permalink raw reply related	[flat|nested] 221+ messages in thread

* [PATCH 18/40] perf tools: Introduce thread__find_addr_location_by_time() and friends
  2015-05-18  0:30 [RFC/PATCHSET 00/40] perf tools: Speed-up perf report by using multi thread (v4) Namhyung Kim
                   ` (16 preceding siblings ...)
  2015-05-18  0:30 ` [PATCH 17/40] perf tools: Maintain map groups list in a leader thread Namhyung Kim
@ 2015-05-18  0:30 ` Namhyung Kim
  2015-05-18  0:30 ` [PATCH 19/40] perf callchain: Use " Namhyung Kim
                   ` (21 subsequent siblings)
  39 siblings, 0 replies; 221+ messages in thread
From: Namhyung Kim @ 2015-05-18  0:30 UTC (permalink / raw)
  To: Arnaldo Carvalho de Melo
  Cc: Ingo Molnar, Peter Zijlstra, Jiri Olsa, LKML, David Ahern,
	Adrian Hunter, Andi Kleen, Frederic Weisbecker, Stephane Eranian

These new functions are for find appropriate map (and symbol) at the
given time when used with an indexed data file.  This is based on the
fact that map_groups list is sorted by time in the previous patch.

Cc: Frederic Weisbecker <fweisbec@gmail.com>
Signed-off-by: Namhyung Kim <namhyung@kernel.org>
---
 tools/perf/util/event.c   | 59 +++++++++++++++++++++++++++++++++++++++--------
 tools/perf/util/machine.c | 28 ++++++++++++++--------
 tools/perf/util/session.h |  1 +
 tools/perf/util/thread.c  | 26 +++++++++++++++++++++
 tools/perf/util/thread.h  | 11 +++++++++
 5 files changed, 106 insertions(+), 19 deletions(-)

diff --git a/tools/perf/util/event.c b/tools/perf/util/event.c
index 0ad76b06cd48..bb391c20920d 100644
--- a/tools/perf/util/event.c
+++ b/tools/perf/util/event.c
@@ -832,16 +832,14 @@ int perf_event__process(struct perf_tool *tool __maybe_unused,
 	return machine__process_event(machine, event, sample);
 }
 
-void thread__find_addr_map(struct thread *thread, u8 cpumode,
-			   enum map_type type, u64 addr,
-			   struct addr_location *al)
+static void map_groups__find_addr_map(struct map_groups *mg, u8 cpumode,
+				      enum map_type type, u64 addr,
+				      struct addr_location *al)
 {
-	struct map_groups *mg = thread->mg;
 	struct machine *machine = mg->machine;
 	bool load_map = false;
 
 	al->machine = machine;
-	al->thread = thread;
 	al->addr = addr;
 	al->cpumode = cpumode;
 	al->filtered = 0;
@@ -910,6 +908,29 @@ void thread__find_addr_map(struct thread *thread, u8 cpumode,
 	}
 }
 
+void thread__find_addr_map(struct thread *thread, u8 cpumode,
+			   enum map_type type, u64 addr,
+			   struct addr_location *al)
+{
+	al->thread = thread;
+	map_groups__find_addr_map(thread->mg, cpumode, type, addr, al);
+}
+
+void thread__find_addr_map_by_time(struct thread *thread, u8 cpumode,
+				   enum map_type type, u64 addr,
+				   struct addr_location *al, u64 timestamp)
+{
+	struct map_groups *mg;
+
+	if (perf_has_index)
+		mg = thread__get_map_groups(thread, timestamp);
+	else
+		mg = thread->mg;
+
+	al->thread = thread;
+	map_groups__find_addr_map(mg, cpumode, type, addr, al);
+}
+
 void thread__find_addr_location(struct thread *thread,
 				u8 cpumode, enum map_type type, u64 addr,
 				struct addr_location *al)
@@ -922,6 +943,23 @@ void thread__find_addr_location(struct thread *thread,
 		al->sym = NULL;
 }
 
+void thread__find_addr_location_by_time(struct thread *thread, u8 cpumode,
+					enum map_type type, u64 addr,
+					struct addr_location *al, u64 timestamp)
+{
+	if (perf_has_index)
+		thread__find_addr_map_by_time(thread, cpumode, type, addr, al,
+					      timestamp);
+	else
+		thread__find_addr_map(thread, cpumode, type, addr, al);
+
+	if (al->map != NULL)
+		al->sym = map__find_symbol(al->map, al->addr,
+					   thread->mg->machine->symbol_filter);
+	else
+		al->sym = NULL;
+}
+
 /*
  * Callers need to drop the reference to al->thread, obtained in
  * machine__findnew_thread()
@@ -951,7 +989,9 @@ int perf_event__preprocess_sample(const union perf_event *event,
 	    machine->vmlinux_maps[MAP__FUNCTION] == NULL)
 		machine__create_kernel_maps(machine);
 
-	thread__find_addr_map(thread, cpumode, MAP__FUNCTION, sample->ip, al);
+	thread__find_addr_map_by_time(thread, cpumode, MAP__FUNCTION,
+				      sample->ip, al, sample->time);
+
 	dump_printf(" ...... dso: %s\n",
 		    al->map ? al->map->dso->long_name :
 			al->level == 'H' ? "[hypervisor]" : "<not found>");
@@ -1026,10 +1066,11 @@ void perf_event__preprocess_sample_addr(union perf_event *event,
 {
 	u8 cpumode = event->header.misc & PERF_RECORD_MISC_CPUMODE_MASK;
 
-	thread__find_addr_map(thread, cpumode, MAP__FUNCTION, sample->addr, al);
+	thread__find_addr_map_by_time(thread, cpumode, MAP__FUNCTION,
+				      sample->addr, al, sample->time);
 	if (!al->map)
-		thread__find_addr_map(thread, cpumode, MAP__VARIABLE,
-				      sample->addr, al);
+		thread__find_addr_map_by_time(thread, cpumode, MAP__VARIABLE,
+					      sample->addr, al, sample->time);
 
 	al->cpu = sample->cpu;
 	al->sym = NULL;
diff --git a/tools/perf/util/machine.c b/tools/perf/util/machine.c
index 99fb14926351..64719692ec77 100644
--- a/tools/perf/util/machine.c
+++ b/tools/perf/util/machine.c
@@ -1604,7 +1604,7 @@ static bool symbol__match_regex(struct symbol *sym, regex_t *regex)
 
 static void ip__resolve_ams(struct thread *thread,
 			    struct addr_map_symbol *ams,
-			    u64 ip)
+			    u64 ip, u64 timestamp)
 {
 	struct addr_location al;
 
@@ -1616,7 +1616,8 @@ static void ip__resolve_ams(struct thread *thread,
 	 * Thus, we have to try consecutively until we find a match
 	 * or else, the symbol is unknown
 	 */
-	thread__find_cpumode_addr_location(thread, MAP__FUNCTION, ip, &al);
+	thread__find_cpumode_addr_location_by_time(thread, MAP__FUNCTION,
+						   ip, &al, timestamp);
 
 	ams->addr = ip;
 	ams->al_addr = al.addr;
@@ -1624,21 +1625,25 @@ static void ip__resolve_ams(struct thread *thread,
 	ams->map = al.map;
 }
 
-static void ip__resolve_data(struct thread *thread,
-			     u8 m, struct addr_map_symbol *ams, u64 addr)
+static void ip__resolve_data(struct thread *thread, u8 m,
+			     struct addr_map_symbol *ams,
+			     u64 addr, u64 timestamp)
 {
 	struct addr_location al;
 
 	memset(&al, 0, sizeof(al));
 
-	thread__find_addr_location(thread, m, MAP__VARIABLE, addr, &al);
+	thread__find_addr_location_by_time(thread, m, MAP__VARIABLE,
+					   addr, &al, timestamp);
+
 	if (al.map == NULL) {
 		/*
 		 * some shared data regions have execute bit set which puts
 		 * their mapping in the MAP__FUNCTION type array.
 		 * Check there as a fallback option before dropping the sample.
 		 */
-		thread__find_addr_location(thread, m, MAP__FUNCTION, addr, &al);
+		thread__find_addr_location_by_time(thread, m, MAP__FUNCTION,
+						   addr, &al, timestamp);
 	}
 
 	ams->addr = addr;
@@ -1655,8 +1660,9 @@ struct mem_info *sample__resolve_mem(struct perf_sample *sample,
 	if (!mi)
 		return NULL;
 
-	ip__resolve_ams(al->thread, &mi->iaddr, sample->ip);
-	ip__resolve_data(al->thread, al->cpumode, &mi->daddr, sample->addr);
+	ip__resolve_ams(al->thread, &mi->iaddr, sample->ip, sample->time);
+	ip__resolve_data(al->thread, al->cpumode, &mi->daddr, sample->addr,
+			 sample->time);
 	mi->data_src.val = sample->data_src;
 
 	return mi;
@@ -1730,8 +1736,10 @@ struct branch_info *sample__resolve_bstack(struct perf_sample *sample,
 		return NULL;
 
 	for (i = 0; i < bs->nr; i++) {
-		ip__resolve_ams(al->thread, &bi[i].to, bs->entries[i].to);
-		ip__resolve_ams(al->thread, &bi[i].from, bs->entries[i].from);
+		ip__resolve_ams(al->thread, &bi[i].to,
+				bs->entries[i].to, sample->time);
+		ip__resolve_ams(al->thread, &bi[i].from,
+				bs->entries[i].from, sample->time);
 		bi[i].flags = bs->entries[i].flags;
 	}
 	return bi;
diff --git a/tools/perf/util/session.h b/tools/perf/util/session.h
index b44afc75d1cc..2cf4ee24e40c 100644
--- a/tools/perf/util/session.h
+++ b/tools/perf/util/session.h
@@ -136,4 +136,5 @@ int perf_event__synthesize_id_index(struct perf_tool *tool,
 				    struct perf_evlist *evlist,
 				    struct machine *machine);
 
+
 #endif /* __PERF_SESSION_H */
diff --git a/tools/perf/util/thread.c b/tools/perf/util/thread.c
index fc4e51afaf18..e35cd18adc0e 100644
--- a/tools/perf/util/thread.c
+++ b/tools/perf/util/thread.c
@@ -392,3 +392,29 @@ void thread__find_cpumode_addr_location(struct thread *thread,
 			break;
 	}
 }
+
+void thread__find_cpumode_addr_location_by_time(struct thread *thread,
+						enum map_type type, u64 addr,
+						struct addr_location *al,
+						u64 timestamp)
+{
+	size_t i;
+	const u8 const cpumodes[] = {
+		PERF_RECORD_MISC_USER,
+		PERF_RECORD_MISC_KERNEL,
+		PERF_RECORD_MISC_GUEST_USER,
+		PERF_RECORD_MISC_GUEST_KERNEL
+	};
+
+	if (!perf_has_index) {
+		thread__find_cpumode_addr_location(thread, type, addr, al);
+		return;
+	}
+
+	for (i = 0; i < ARRAY_SIZE(cpumodes); i++) {
+		thread__find_addr_location_by_time(thread, cpumodes[i], type,
+						   addr, al, timestamp);
+		if (al->map)
+			break;
+	}
+}
diff --git a/tools/perf/util/thread.h b/tools/perf/util/thread.h
index 468bb785f061..eb73720f4009 100644
--- a/tools/perf/util/thread.h
+++ b/tools/perf/util/thread.h
@@ -81,14 +81,25 @@ size_t thread__fprintf(struct thread *thread, FILE *fp);
 void thread__find_addr_map(struct thread *thread,
 			   u8 cpumode, enum map_type type, u64 addr,
 			   struct addr_location *al);
+void thread__find_addr_map_by_time(struct thread *thread, u8 cpumode,
+				   enum map_type type, u64 addr,
+				   struct addr_location *al, u64 timestamp);
 
 void thread__find_addr_location(struct thread *thread,
 				u8 cpumode, enum map_type type, u64 addr,
 				struct addr_location *al);
+void thread__find_addr_location_by_time(struct thread *thread, u8 cpumode,
+					enum map_type type, u64 addr,
+					struct addr_location *al,
+					u64 timestamp);
 
 void thread__find_cpumode_addr_location(struct thread *thread,
 					enum map_type type, u64 addr,
 					struct addr_location *al);
+void thread__find_cpumode_addr_location_by_time(struct thread *thread,
+						enum map_type type, u64 addr,
+						struct addr_location *al,
+						u64 timestamp);
 
 static inline void *thread__priv(struct thread *thread)
 {
-- 
2.4.0


^ permalink raw reply related	[flat|nested] 221+ messages in thread

* [PATCH 19/40] perf callchain: Use thread__find_addr_location_by_time() and friends
  2015-05-18  0:30 [RFC/PATCHSET 00/40] perf tools: Speed-up perf report by using multi thread (v4) Namhyung Kim
                   ` (17 preceding siblings ...)
  2015-05-18  0:30 ` [PATCH 18/40] perf tools: Introduce thread__find_addr_location_by_time() and friends Namhyung Kim
@ 2015-05-18  0:30 ` Namhyung Kim
  2015-05-18  0:30 ` [PATCH 20/40] perf tools: Add a test case for timed map groups handling Namhyung Kim
                   ` (20 subsequent siblings)
  39 siblings, 0 replies; 221+ messages in thread
From: Namhyung Kim @ 2015-05-18  0:30 UTC (permalink / raw)
  To: Arnaldo Carvalho de Melo
  Cc: Ingo Molnar, Peter Zijlstra, Jiri Olsa, LKML, David Ahern,
	Adrian Hunter, Andi Kleen, Frederic Weisbecker, Stephane Eranian

Find correct thread/map/symbol using proper functions.

Cc: Frederic Weisbecker <fweisbec@gmail.com>
Signed-off-by: Namhyung Kim <namhyung@kernel.org>
---
 tools/perf/util/machine.c                          | 25 ++++++++++++--------
 .../util/scripting-engines/trace-event-python.c    |  4 ++--
 tools/perf/util/session.h                          |  1 -
 tools/perf/util/unwind-libdw.c                     | 12 ++++++----
 tools/perf/util/unwind-libunwind.c                 | 27 +++++++++++-----------
 5 files changed, 39 insertions(+), 30 deletions(-)

diff --git a/tools/perf/util/machine.c b/tools/perf/util/machine.c
index 64719692ec77..17e900a6a3c3 100644
--- a/tools/perf/util/machine.c
+++ b/tools/perf/util/machine.c
@@ -1672,15 +1672,17 @@ static int add_callchain_ip(struct thread *thread,
 			    struct symbol **parent,
 			    struct addr_location *root_al,
 			    u8 *cpumode,
-			    u64 ip)
+			    u64 ip,
+			    u64 timestamp)
 {
 	struct addr_location al;
 
 	al.filtered = 0;
 	al.sym = NULL;
 	if (!cpumode) {
-		thread__find_cpumode_addr_location(thread, MAP__FUNCTION,
-						   ip, &al);
+		thread__find_cpumode_addr_location_by_time(thread,
+							   MAP__FUNCTION, ip,
+							   &al, timestamp);
 	} else {
 		if (ip >= PERF_CONTEXT_MAX) {
 			switch (ip) {
@@ -1705,8 +1707,9 @@ static int add_callchain_ip(struct thread *thread,
 			}
 			return 0;
 		}
-		thread__find_addr_location(thread, *cpumode, MAP__FUNCTION,
-					   ip, &al);
+		thread__find_addr_location_by_time(thread, *cpumode,
+						   MAP__FUNCTION, ip,
+						   &al, timestamp);
 	}
 
 	if (al.sym != NULL) {
@@ -1848,7 +1851,8 @@ static int resolve_lbr_callchain_sample(struct thread *thread,
 					ip = lbr_stack->entries[0].to;
 			}
 
-			err = add_callchain_ip(thread, parent, root_al, &cpumode, ip);
+			err = add_callchain_ip(thread, parent, root_al, &cpumode, ip,
+					       sample->time);
 			if (err)
 				return (err < 0) ? err : 0;
 		}
@@ -1869,6 +1873,7 @@ static int thread__resolve_callchain_sample(struct thread *thread,
 	struct ip_callchain *chain = sample->callchain;
 	int chain_nr = min(max_stack, (int)chain->nr);
 	u8 cpumode = PERF_RECORD_MISC_USER;
+	u64 timestamp = sample->time;
 	int i, j, err;
 	int skip_idx = -1;
 	int first_call = 0;
@@ -1934,10 +1939,11 @@ static int thread__resolve_callchain_sample(struct thread *thread,
 
 		for (i = 0; i < nr; i++) {
 			err = add_callchain_ip(thread, parent, root_al,
-					       NULL, be[i].to);
+					       NULL, be[i].to, timestamp);
 			if (!err)
 				err = add_callchain_ip(thread, parent, root_al,
-						       NULL, be[i].from);
+						       NULL, be[i].from,
+						       timestamp);
 			if (err == -EINVAL)
 				break;
 			if (err)
@@ -1966,7 +1972,8 @@ static int thread__resolve_callchain_sample(struct thread *thread,
 #endif
 		ip = chain->ips[j];
 
-		err = add_callchain_ip(thread, parent, root_al, &cpumode, ip);
+		err = add_callchain_ip(thread, parent, root_al, &cpumode, ip,
+				       timestamp);
 
 		if (err)
 			return (err < 0) ? err : 0;
diff --git a/tools/perf/util/scripting-engines/trace-event-python.c b/tools/perf/util/scripting-engines/trace-event-python.c
index 5544b8cdd1ee..9f51b0dbb087 100644
--- a/tools/perf/util/scripting-engines/trace-event-python.c
+++ b/tools/perf/util/scripting-engines/trace-event-python.c
@@ -304,8 +304,8 @@ static PyObject *get_field_numeric_entry(struct event_format *event,
 
 
 static PyObject *python_process_callchain(struct perf_sample *sample,
-					 struct perf_evsel *evsel,
-					 struct addr_location *al)
+					  struct perf_evsel *evsel,
+					  struct addr_location *al)
 {
 	PyObject *pylist;
 
diff --git a/tools/perf/util/session.h b/tools/perf/util/session.h
index 2cf4ee24e40c..b44afc75d1cc 100644
--- a/tools/perf/util/session.h
+++ b/tools/perf/util/session.h
@@ -136,5 +136,4 @@ int perf_event__synthesize_id_index(struct perf_tool *tool,
 				    struct perf_evlist *evlist,
 				    struct machine *machine);
 
-
 #endif /* __PERF_SESSION_H */
diff --git a/tools/perf/util/unwind-libdw.c b/tools/perf/util/unwind-libdw.c
index 2dcfe9a7c8d0..0dd2b6ff4093 100644
--- a/tools/perf/util/unwind-libdw.c
+++ b/tools/perf/util/unwind-libdw.c
@@ -26,9 +26,9 @@ static int __report_module(struct addr_location *al, u64 ip,
 	Dwfl_Module *mod;
 	struct dso *dso = NULL;
 
-	thread__find_addr_location(ui->thread,
-				   PERF_RECORD_MISC_USER,
-				   MAP__FUNCTION, ip, al);
+	thread__find_addr_location_by_time(ui->thread, PERF_RECORD_MISC_USER,
+					   MAP__FUNCTION, ip, al,
+					   ui->sample->time);
 
 	if (al->map)
 		dso = al->map->dso;
@@ -89,8 +89,10 @@ static int access_dso_mem(struct unwind_info *ui, Dwarf_Addr addr,
 	struct addr_location al;
 	ssize_t size;
 
-	thread__find_addr_map(ui->thread, PERF_RECORD_MISC_USER,
-			      MAP__FUNCTION, addr, &al);
+	thread__find_addr_map_by_time(ui->thread, PERF_RECORD_MISC_USER,
+				      MAP__FUNCTION, addr, &al,
+				      ui->sample->time);
+
 	if (!al.map) {
 		pr_debug("unwind: no map for %lx\n", (unsigned long)addr);
 		return -1;
diff --git a/tools/perf/util/unwind-libunwind.c b/tools/perf/util/unwind-libunwind.c
index 7b09a443a280..0978697341c1 100644
--- a/tools/perf/util/unwind-libunwind.c
+++ b/tools/perf/util/unwind-libunwind.c
@@ -315,8 +315,10 @@ static struct map *find_map(unw_word_t ip, struct unwind_info *ui)
 {
 	struct addr_location al;
 
-	thread__find_addr_map(ui->thread, PERF_RECORD_MISC_USER,
-			      MAP__FUNCTION, ip, &al);
+	thread__find_addr_map_by_time(ui->thread, PERF_RECORD_MISC_USER,
+				      MAP__FUNCTION, ip, &al,
+				      ui->sample->time);
+
 	return al.map;
 }
 
@@ -406,20 +408,19 @@ get_proc_name(unw_addr_space_t __maybe_unused as,
 static int access_dso_mem(struct unwind_info *ui, unw_word_t addr,
 			  unw_word_t *data)
 {
-	struct addr_location al;
+	struct map *map;
 	ssize_t size;
 
-	thread__find_addr_map(ui->thread, PERF_RECORD_MISC_USER,
-			      MAP__FUNCTION, addr, &al);
-	if (!al.map) {
+	map = find_map(addr, ui);
+	if (!map) {
 		pr_debug("unwind: no map for %lx\n", (unsigned long)addr);
 		return -1;
 	}
 
-	if (!al.map->dso)
+	if (!map->dso)
 		return -1;
 
-	size = dso__data_read_addr(al.map->dso, al.map, ui->machine,
+	size = dso__data_read_addr(map->dso, map, ui->machine,
 				   addr, (u8 *) data, sizeof(*data));
 
 	return !(size == sizeof(*data));
@@ -511,14 +512,14 @@ static void put_unwind_info(unw_addr_space_t __maybe_unused as,
 	pr_debug("unwind: put_unwind_info called\n");
 }
 
-static int entry(u64 ip, struct thread *thread,
+static int entry(u64 ip, struct thread *thread, u64 timestamp,
 		 unwind_entry_cb_t cb, void *arg)
 {
 	struct unwind_entry e;
 	struct addr_location al;
 
-	thread__find_addr_location(thread, PERF_RECORD_MISC_USER,
-				   MAP__FUNCTION, ip, &al);
+	thread__find_addr_location_by_time(thread, PERF_RECORD_MISC_USER,
+					   MAP__FUNCTION, ip, &al, timestamp);
 
 	e.ip = ip;
 	e.map = al.map;
@@ -620,7 +621,7 @@ static int get_entries(struct unwind_info *ui, unwind_entry_cb_t cb,
 		unw_word_t ip;
 
 		unw_get_reg(&c, UNW_REG_IP, &ip);
-		ret = ip ? entry(ip, ui->thread, cb, arg) : 0;
+		ret = ip ? entry(ip, ui->thread, ui->sample->time, cb, arg) : 0;
 	}
 
 	return ret;
@@ -645,7 +646,7 @@ int unwind__get_entries(unwind_entry_cb_t cb, void *arg,
 	if (ret)
 		return ret;
 
-	ret = entry(ip, thread, cb, arg);
+	ret = entry(ip, thread, data->time, cb, arg);
 	if (ret)
 		return -ENOMEM;
 
-- 
2.4.0


^ permalink raw reply related	[flat|nested] 221+ messages in thread

* [PATCH 20/40] perf tools: Add a test case for timed map groups handling
  2015-05-18  0:30 [RFC/PATCHSET 00/40] perf tools: Speed-up perf report by using multi thread (v4) Namhyung Kim
                   ` (18 preceding siblings ...)
  2015-05-18  0:30 ` [PATCH 19/40] perf callchain: Use " Namhyung Kim
@ 2015-05-18  0:30 ` Namhyung Kim
  2015-05-18  0:30 ` [PATCH 21/40] perf tools: Save timestamp of a map creation Namhyung Kim
                   ` (19 subsequent siblings)
  39 siblings, 0 replies; 221+ messages in thread
From: Namhyung Kim @ 2015-05-18  0:30 UTC (permalink / raw)
  To: Arnaldo Carvalho de Melo
  Cc: Ingo Molnar, Peter Zijlstra, Jiri Olsa, LKML, David Ahern,
	Adrian Hunter, Andi Kleen, Frederic Weisbecker, Stephane Eranian

A test case for verifying thread->mg and ->mg_list handling during
time change and new thread__find_addr_map_by_time() and friends.

Cc: Frederic Weisbecker <fweisbec@gmail.com>
Signed-off-by: Namhyung Kim <namhyung@kernel.org>
---
 tools/perf/tests/Build            |  1 +
 tools/perf/tests/builtin-test.c   |  4 ++
 tools/perf/tests/tests.h          |  1 +
 tools/perf/tests/thread-mg-time.c | 93 +++++++++++++++++++++++++++++++++++++++
 4 files changed, 99 insertions(+)
 create mode 100644 tools/perf/tests/thread-mg-time.c

diff --git a/tools/perf/tests/Build b/tools/perf/tests/Build
index 5ad495823b49..cfd59c61bcd2 100644
--- a/tools/perf/tests/Build
+++ b/tools/perf/tests/Build
@@ -27,6 +27,7 @@ perf-y += mmap-thread-lookup.o
 perf-y += thread-comm.o
 perf-y += thread-mg-share.o
 perf-y += thread-lookup-time.o
+perf-y += thread-mg-time.o
 perf-y += switch-tracking.o
 perf-y += keep-tracking.o
 perf-y += code-reading.o
diff --git a/tools/perf/tests/builtin-test.c b/tools/perf/tests/builtin-test.c
index e83c7ce1b38a..c5dbeb3d75b1 100644
--- a/tools/perf/tests/builtin-test.c
+++ b/tools/perf/tests/builtin-test.c
@@ -179,6 +179,10 @@ static struct test {
 		.func = test__thread_lookup_time,
 	},
 	{
+		.desc = "Test thread map group handling with time",
+		.func = test__thread_mg_time,
+	},
+	{
 		.func = NULL,
 	},
 };
diff --git a/tools/perf/tests/tests.h b/tools/perf/tests/tests.h
index e9aa78c3d8fc..a2e1f729ae23 100644
--- a/tools/perf/tests/tests.h
+++ b/tools/perf/tests/tests.h
@@ -63,6 +63,7 @@ int test__fdarray__add(void);
 int test__kmod_path__parse(void);
 int test__thread_comm(void);
 int test__thread_lookup_time(void);
+int test__thread_mg_time(void);
 
 #if defined(__x86_64__) || defined(__i386__) || defined(__arm__)
 #ifdef HAVE_DWARF_UNWIND_SUPPORT
diff --git a/tools/perf/tests/thread-mg-time.c b/tools/perf/tests/thread-mg-time.c
new file mode 100644
index 000000000000..841777125a64
--- /dev/null
+++ b/tools/perf/tests/thread-mg-time.c
@@ -0,0 +1,93 @@
+#include "tests.h"
+#include "machine.h"
+#include "thread.h"
+#include "map.h"
+#include "debug.h"
+
+#define PERF_MAP_START  0x40000
+
+int test__thread_mg_time(void)
+{
+	struct machines machines;
+	struct machine *machine;
+	struct thread *t;
+	struct map_groups *mg;
+	struct map *map, *old_map;
+	struct addr_location al = { .map = NULL, };
+
+	/*
+	 * This test is to check whether it can retrieve a correct map
+	 * for a given time.  When multi-file data storage is enabled,
+	 * those task/comm/mmap events are processed first so the
+	 * later sample should find a matching comm properly.
+	 */
+	machines__init(&machines);
+	machine = &machines.host;
+
+	/* this is needed to add/find map by time */
+	perf_has_index = true;
+
+	t = machine__findnew_thread(machine, 0, 0);
+	mg = t->mg;
+
+	map = dso__new_map("/usr/bin/perf");
+	map->start = PERF_MAP_START;
+	map->end = PERF_MAP_START + 0x1000;
+
+	thread__insert_map(t, map);
+
+	if (verbose > 1)
+		map_groups__fprintf(t->mg, stderr);
+
+	thread__find_addr_map(t, PERF_RECORD_MISC_USER, MAP__FUNCTION,
+			      PERF_MAP_START, &al);
+
+	TEST_ASSERT_VAL("cannot find mapping for perf", al.map != NULL);
+	TEST_ASSERT_VAL("non matched mapping found", al.map == map);
+	TEST_ASSERT_VAL("incorrect map groups", al.map->groups == mg);
+	TEST_ASSERT_VAL("incorrect map groups", al.map->groups == t->mg);
+
+	thread__find_addr_map_by_time(t, PERF_RECORD_MISC_USER, MAP__FUNCTION,
+				      PERF_MAP_START, &al, -1ULL);
+
+	TEST_ASSERT_VAL("cannot find timed mapping for perf", al.map != NULL);
+	TEST_ASSERT_VAL("non matched timed mapping", al.map == map);
+	TEST_ASSERT_VAL("incorrect timed map groups", al.map->groups == mg);
+	TEST_ASSERT_VAL("incorrect map groups", al.map->groups == t->mg);
+
+
+	pr_debug("simulate EXEC event (generate new mg)\n");
+	__thread__set_comm(t, "perf-test", 10000, true);
+
+	old_map = map;
+
+	map = dso__new_map("/usr/bin/perf-test");
+	map->start = PERF_MAP_START;
+	map->end = PERF_MAP_START + 0x2000;
+
+	thread__insert_map(t, map);
+
+	if (verbose > 1)
+		map_groups__fprintf(t->mg, stderr);
+
+	thread__find_addr_map(t, PERF_RECORD_MISC_USER, MAP__FUNCTION,
+			      PERF_MAP_START + 4, &al);
+
+	TEST_ASSERT_VAL("cannot find mapping for perf-test", al.map != NULL);
+	TEST_ASSERT_VAL("invalid mapping found", al.map == map);
+	TEST_ASSERT_VAL("incorrect map groups", al.map->groups != mg);
+	TEST_ASSERT_VAL("incorrect map groups", al.map->groups == t->mg);
+
+	pr_debug("searching map in the old mag groups\n");
+	thread__find_addr_map_by_time(t, PERF_RECORD_MISC_USER, MAP__FUNCTION,
+				      PERF_MAP_START, &al, 5000);
+
+	TEST_ASSERT_VAL("cannot find timed mapping for perf-test", al.map != NULL);
+	TEST_ASSERT_VAL("non matched timed mapping", al.map == old_map);
+	TEST_ASSERT_VAL("incorrect timed map groups", al.map->groups == mg);
+	TEST_ASSERT_VAL("incorrect map groups", al.map->groups != t->mg);
+
+	machine__delete_threads(machine);
+	machines__exit(&machines);
+	return 0;
+}
-- 
2.4.0


^ permalink raw reply related	[flat|nested] 221+ messages in thread

* [PATCH 21/40] perf tools: Save timestamp of a map creation
  2015-05-18  0:30 [RFC/PATCHSET 00/40] perf tools: Speed-up perf report by using multi thread (v4) Namhyung Kim
                   ` (19 preceding siblings ...)
  2015-05-18  0:30 ` [PATCH 20/40] perf tools: Add a test case for timed map groups handling Namhyung Kim
@ 2015-05-18  0:30 ` Namhyung Kim
  2015-05-18  0:30 ` [PATCH 22/40] perf tools: Introduce map_groups__{insert,find}_by_time() Namhyung Kim
                   ` (18 subsequent siblings)
  39 siblings, 0 replies; 221+ messages in thread
From: Namhyung Kim @ 2015-05-18  0:30 UTC (permalink / raw)
  To: Arnaldo Carvalho de Melo
  Cc: Ingo Molnar, Peter Zijlstra, Jiri Olsa, LKML, David Ahern,
	Adrian Hunter, Andi Kleen, Frederic Weisbecker, Stephane Eranian

It'll be used to support multiple maps on a same address like dlopen()
and/or JIT compile cases.

Cc: Stephane Eranian <eranian@google.com>
Signed-off-by: Namhyung Kim <namhyung@kernel.org>
---
 tools/perf/util/dso.c         |  2 +-
 tools/perf/util/machine.c     | 28 ++++++++++++++++------------
 tools/perf/util/machine.h     |  2 +-
 tools/perf/util/map.c         | 12 +++++++-----
 tools/perf/util/map.h         |  9 ++++++---
 tools/perf/util/probe-event.c |  2 +-
 tools/perf/util/symbol-elf.c  |  2 +-
 tools/perf/util/symbol.c      |  4 ++--
 8 files changed, 35 insertions(+), 26 deletions(-)

diff --git a/tools/perf/util/dso.c b/tools/perf/util/dso.c
index 13d9ae0bd15c..7078700233fa 100644
--- a/tools/perf/util/dso.c
+++ b/tools/perf/util/dso.c
@@ -747,7 +747,7 @@ struct map *dso__new_map(const char *name)
 	struct dso *dso = dso__new(name);
 
 	if (dso)
-		map = map__new2(0, dso, MAP__FUNCTION);
+		map = map__new2(0, dso, MAP__FUNCTION, 0);
 
 	return map;
 }
diff --git a/tools/perf/util/machine.c b/tools/perf/util/machine.c
index 17e900a6a3c3..e3566365d320 100644
--- a/tools/perf/util/machine.c
+++ b/tools/perf/util/machine.c
@@ -668,7 +668,7 @@ int machine__process_itrace_start_event(struct machine *machine __maybe_unused,
 }
 
 struct map *machine__new_module(struct machine *machine, u64 start,
-				const char *filename)
+				const char *filename, u64 timestamp)
 {
 	struct map *map = NULL;
 	struct dso *dso;
@@ -686,7 +686,7 @@ struct map *machine__new_module(struct machine *machine, u64 start,
 	if (dso == NULL)
 		goto out;
 
-	map = map__new2(start, dso, MAP__FUNCTION);
+	map = map__new2(start, dso, MAP__FUNCTION, timestamp);
 	if (map == NULL)
 		goto out;
 
@@ -854,7 +854,7 @@ int __machine__create_kernel_maps(struct machine *machine, struct dso *kernel)
 	for (type = 0; type < MAP__NR_TYPES; ++type) {
 		struct kmap *kmap;
 
-		machine->vmlinux_maps[type] = map__new2(start, kernel, type);
+		machine->vmlinux_maps[type] = map__new2(start, kernel, type, 0);
 		if (machine->vmlinux_maps[type] == NULL)
 			return -1;
 
@@ -1155,7 +1155,7 @@ static int machine__create_module(void *arg, const char *name, u64 start)
 	struct machine *machine = arg;
 	struct map *map;
 
-	map = machine__new_module(machine, start, name);
+	map = machine__new_module(machine, start, name, 0);
 	if (map == NULL)
 		return -1;
 
@@ -1256,7 +1256,8 @@ static bool machine__uses_kcore(struct machine *machine)
 }
 
 static int machine__process_kernel_mmap_event(struct machine *machine,
-					      union perf_event *event)
+					      union perf_event *event,
+					      u64 timestamp)
 {
 	struct map *map;
 	char kmmap_prefix[PATH_MAX];
@@ -1279,7 +1280,7 @@ static int machine__process_kernel_mmap_event(struct machine *machine,
 	if (event->mmap.filename[0] == '/' ||
 	    (!is_kernel_mmap && event->mmap.filename[0] == '[')) {
 		map = machine__new_module(machine, event->mmap.start,
-					  event->mmap.filename);
+					  event->mmap.filename, timestamp);
 		if (map == NULL)
 			goto out_problem;
 
@@ -1343,7 +1344,7 @@ static int machine__process_kernel_mmap_event(struct machine *machine,
 
 int machine__process_mmap2_event(struct machine *machine,
 				 union perf_event *event,
-				 struct perf_sample *sample __maybe_unused)
+				 struct perf_sample *sample)
 {
 	u8 cpumode = event->header.misc & PERF_RECORD_MISC_CPUMODE_MASK;
 	struct thread *thread;
@@ -1356,7 +1357,8 @@ int machine__process_mmap2_event(struct machine *machine,
 
 	if (cpumode == PERF_RECORD_MISC_GUEST_KERNEL ||
 	    cpumode == PERF_RECORD_MISC_KERNEL) {
-		ret = machine__process_kernel_mmap_event(machine, event);
+		ret = machine__process_kernel_mmap_event(machine, event,
+							 sample->time);
 		if (ret < 0)
 			goto out_problem;
 		return 0;
@@ -1379,7 +1381,8 @@ int machine__process_mmap2_event(struct machine *machine,
 			event->mmap2.ino_generation,
 			event->mmap2.prot,
 			event->mmap2.flags,
-			event->mmap2.filename, type, thread);
+			event->mmap2.filename, type, thread,
+			sample->time);
 
 	if (map == NULL)
 		goto out_problem_map;
@@ -1396,7 +1399,7 @@ int machine__process_mmap2_event(struct machine *machine,
 }
 
 int machine__process_mmap_event(struct machine *machine, union perf_event *event,
-				struct perf_sample *sample __maybe_unused)
+				struct perf_sample *sample)
 {
 	u8 cpumode = event->header.misc & PERF_RECORD_MISC_CPUMODE_MASK;
 	struct thread *thread;
@@ -1409,7 +1412,8 @@ int machine__process_mmap_event(struct machine *machine, union perf_event *event
 
 	if (cpumode == PERF_RECORD_MISC_GUEST_KERNEL ||
 	    cpumode == PERF_RECORD_MISC_KERNEL) {
-		ret = machine__process_kernel_mmap_event(machine, event);
+		ret = machine__process_kernel_mmap_event(machine, event,
+							 sample->time);
 		if (ret < 0)
 			goto out_problem;
 		return 0;
@@ -1429,7 +1433,7 @@ int machine__process_mmap_event(struct machine *machine, union perf_event *event
 			event->mmap.len, event->mmap.pgoff,
 			event->mmap.pid, 0, 0, 0, 0, 0, 0,
 			event->mmap.filename,
-			type, thread);
+			type, thread, sample->time);
 
 	if (map == NULL)
 		goto out_problem_map;
diff --git a/tools/perf/util/machine.h b/tools/perf/util/machine.h
index 863c0e90ceb1..f0f6bd420237 100644
--- a/tools/perf/util/machine.h
+++ b/tools/perf/util/machine.h
@@ -193,7 +193,7 @@ struct symbol *machine__find_kernel_function_by_name(struct machine *machine,
 }
 
 struct map *machine__new_module(struct machine *machine, u64 start,
-				const char *filename);
+				const char *filename, u64 timestamp);
 
 int machine__load_kallsyms(struct machine *machine, const char *filename,
 			   enum map_type type, symbol_filter_t filter);
diff --git a/tools/perf/util/map.c b/tools/perf/util/map.c
index b794c3561995..fef4e38ccd93 100644
--- a/tools/perf/util/map.c
+++ b/tools/perf/util/map.c
@@ -123,7 +123,7 @@ static inline bool replace_android_lib(const char *filename, char *newfilename)
 }
 
 void map__init(struct map *map, enum map_type type,
-	       u64 start, u64 end, u64 pgoff, struct dso *dso)
+	       u64 start, u64 end, u64 pgoff, struct dso *dso, u64 timestamp)
 {
 	map->type     = type;
 	map->start    = start;
@@ -137,12 +137,13 @@ void map__init(struct map *map, enum map_type type,
 	map->groups   = NULL;
 	map->referenced = false;
 	map->erange_warned = false;
+	map->timestamp = timestamp;
 }
 
 struct map *map__new(struct machine *machine, u64 start, u64 len,
 		     u64 pgoff, u32 pid, u32 d_maj, u32 d_min, u64 ino,
 		     u64 ino_gen, u32 prot, u32 flags, char *filename,
-		     enum map_type type, struct thread *thread)
+		     enum map_type type, struct thread *thread, u64 timestamp)
 {
 	struct map *map = malloc(sizeof(*map));
 
@@ -182,7 +183,7 @@ struct map *map__new(struct machine *machine, u64 start, u64 len,
 		if (dso == NULL)
 			goto out_delete;
 
-		map__init(map, type, start, start + len, pgoff, dso);
+		map__init(map, type, start, start + len, pgoff, dso, timestamp);
 
 		if (anon || no_dso) {
 			map->map_ip = map->unmap_ip = identity__map_ip;
@@ -207,7 +208,8 @@ struct map *map__new(struct machine *machine, u64 start, u64 len,
  * they are loaded) and for vmlinux, where only after we load all the
  * symbols we'll know where it starts and ends.
  */
-struct map *map__new2(u64 start, struct dso *dso, enum map_type type)
+struct map *map__new2(u64 start, struct dso *dso, enum map_type type,
+		      u64 timestamp)
 {
 	struct map *map = calloc(1, (sizeof(*map) +
 				     (dso->kernel ? sizeof(struct kmap) : 0)));
@@ -215,7 +217,7 @@ struct map *map__new2(u64 start, struct dso *dso, enum map_type type)
 		/*
 		 * ->end will be filled after we load all the symbols
 		 */
-		map__init(map, type, start, 0, 0, dso);
+		map__init(map, type, start, 0, 0, dso, timestamp);
 	}
 
 	return map;
diff --git a/tools/perf/util/map.h b/tools/perf/util/map.h
index 074453d332dd..c1ed11a3e16c 100644
--- a/tools/perf/util/map.h
+++ b/tools/perf/util/map.h
@@ -42,6 +42,7 @@ struct map {
 	u32			maj, min; /* only valid for MMAP2 record */
 	u64			ino;      /* only valid for MMAP2 record */
 	u64			ino_generation;/* only valid for MMAP2 record */
+	u64			timestamp;
 
 	/* ip -> dso rip */
 	u64			(*map_ip)(struct map *, u64);
@@ -136,12 +137,14 @@ typedef int (*symbol_filter_t)(struct map *map, struct symbol *sym);
 
 int arch__compare_symbol_names(const char *namea, const char *nameb);
 void map__init(struct map *map, enum map_type type,
-	       u64 start, u64 end, u64 pgoff, struct dso *dso);
+	       u64 start, u64 end, u64 pgoff, struct dso *dso, u64 timestamp);
 struct map *map__new(struct machine *machine, u64 start, u64 len,
 		     u64 pgoff, u32 pid, u32 d_maj, u32 d_min, u64 ino,
 		     u64 ino_gen, u32 prot, u32 flags,
-		     char *filename, enum map_type type, struct thread *thread);
-struct map *map__new2(u64 start, struct dso *dso, enum map_type type);
+		     char *filename, enum map_type type, struct thread *thread,
+		     u64 timestamp);
+struct map *map__new2(u64 start, struct dso *dso, enum map_type type,
+		      u64 timestamp);
 void map__delete(struct map *map);
 struct map *map__clone(struct map *map);
 int map__overlap(struct map *l, struct map *r);
diff --git a/tools/perf/util/probe-event.c b/tools/perf/util/probe-event.c
index 2399dc4f6089..5a1d6bee4731 100644
--- a/tools/perf/util/probe-event.c
+++ b/tools/perf/util/probe-event.c
@@ -167,7 +167,7 @@ static struct map *kernel_get_module_map(const char *module)
 
 	/* A file path -- this is an offline module */
 	if (module && strchr(module, '/'))
-		return machine__new_module(host_machine, 0, module);
+		return machine__new_module(host_machine, 0, module, 0);
 
 	if (!module)
 		module = "kernel";
diff --git a/tools/perf/util/symbol-elf.c b/tools/perf/util/symbol-elf.c
index 9d526a5312b1..eb84dd5e86c9 100644
--- a/tools/perf/util/symbol-elf.c
+++ b/tools/perf/util/symbol-elf.c
@@ -1012,7 +1012,7 @@ int dso__load_sym(struct dso *dso, struct map *map,
 				curr_dso->long_name = dso->long_name;
 				curr_dso->long_name_len = dso->long_name_len;
 				curr_map = map__new2(start, curr_dso,
-						     map->type);
+						     map->type, map->timestamp);
 				if (curr_map == NULL) {
 					dso__delete(curr_dso);
 					goto out_elf_end;
diff --git a/tools/perf/util/symbol.c b/tools/perf/util/symbol.c
index 45ba48a7acb3..5d69d3c407e6 100644
--- a/tools/perf/util/symbol.c
+++ b/tools/perf/util/symbol.c
@@ -781,7 +781,7 @@ static int dso__split_kallsyms(struct dso *dso, struct map *map, u64 delta,
 
 			ndso->kernel = dso->kernel;
 
-			curr_map = map__new2(pos->start, ndso, map->type);
+			curr_map = map__new2(pos->start, ndso, map->type, 0);
 			if (curr_map == NULL) {
 				dso__delete(ndso);
 				return -1;
@@ -1083,7 +1083,7 @@ static int kcore_mapfn(u64 start, u64 len, u64 pgoff, void *data)
 	struct kcore_mapfn_data *md = data;
 	struct map *map;
 
-	map = map__new2(start, md->dso, md->type);
+	map = map__new2(start, md->dso, md->type, 0);
 	if (map == NULL)
 		return -ENOMEM;
 
-- 
2.4.0


^ permalink raw reply related	[flat|nested] 221+ messages in thread

* [PATCH 22/40] perf tools: Introduce map_groups__{insert,find}_by_time()
  2015-05-18  0:30 [RFC/PATCHSET 00/40] perf tools: Speed-up perf report by using multi thread (v4) Namhyung Kim
                   ` (20 preceding siblings ...)
  2015-05-18  0:30 ` [PATCH 21/40] perf tools: Save timestamp of a map creation Namhyung Kim
@ 2015-05-18  0:30 ` Namhyung Kim
  2015-05-18  0:30 ` [PATCH 23/40] perf tools: Use map_groups__find_addr_by_time() Namhyung Kim
                   ` (17 subsequent siblings)
  39 siblings, 0 replies; 221+ messages in thread
From: Namhyung Kim @ 2015-05-18  0:30 UTC (permalink / raw)
  To: Arnaldo Carvalho de Melo
  Cc: Ingo Molnar, Peter Zijlstra, Jiri Olsa, LKML, David Ahern,
	Adrian Hunter, Andi Kleen, Frederic Weisbecker, Stephane Eranian

It'll manage maps using timestamp so that it can find correct
map/symbol for sample at a certain time.  With this API, it can
maintain overlapping maps in a map_groups.

Cc: Stephane Eranian <eranian@google.com>
Signed-off-by: Namhyung Kim <namhyung@kernel.org>
---
 tools/perf/util/map.c | 52 +++++++++++++++++++++++++++++++++++++++++++++++++++
 tools/perf/util/map.h | 25 +++++++++++++++++++++++++
 2 files changed, 77 insertions(+)

diff --git a/tools/perf/util/map.c b/tools/perf/util/map.c
index fef4e38ccd93..8bc016648a34 100644
--- a/tools/perf/util/map.c
+++ b/tools/perf/util/map.c
@@ -746,6 +746,33 @@ void maps__insert(struct rb_root *maps, struct map *map)
 	rb_insert_color(&map->rb_node, maps);
 }
 
+void maps__insert_by_time(struct rb_root *maps, struct map *map)
+{
+	struct rb_node **p = &maps->rb_node;
+	struct rb_node *parent = NULL;
+	const u64 ip = map->start;
+	const u64 timestamp = map->timestamp;
+	struct map *m;
+
+	while (*p != NULL) {
+		parent = *p;
+		m = rb_entry(parent, struct map, rb_node);
+		if (ip < m->start)
+			p = &(*p)->rb_left;
+		else if (ip > m->start)
+			p = &(*p)->rb_right;
+		else if (timestamp > m->timestamp)
+			p = &(*p)->rb_left;
+		else if (timestamp < m->timestamp)
+			p = &(*p)->rb_right;
+		else
+			BUG_ON(1);
+	}
+
+	rb_link_node(&map->rb_node, parent, p);
+	rb_insert_color(&map->rb_node, maps);
+}
+
 void maps__remove(struct rb_root *maps, struct map *map)
 {
 	rb_erase(&map->rb_node, maps);
@@ -771,6 +798,31 @@ struct map *maps__find(struct rb_root *maps, u64 ip)
 	return NULL;
 }
 
+struct map *maps__find_by_time(struct rb_root *maps, u64 ip, u64 timestamp)
+{
+	struct rb_node **p = &maps->rb_node;
+	struct rb_node *parent = NULL;
+	struct map *m;
+	struct map *best = NULL;
+
+	while (*p != NULL) {
+		parent = *p;
+		m = rb_entry(parent, struct map, rb_node);
+		if (ip < m->start)
+			p = &(*p)->rb_left;
+		else if (ip >= m->end)
+			p = &(*p)->rb_right;
+		else if (timestamp >= m->timestamp) {
+			if (!best || best->timestamp < m->timestamp)
+				best = m;
+			p = &(*p)->rb_left;
+		} else
+			p = &(*p)->rb_right;
+	}
+
+	return best;
+}
+
 struct map *maps__first(struct rb_root *maps)
 {
 	struct rb_node *first = rb_first(maps);
diff --git a/tools/perf/util/map.h b/tools/perf/util/map.h
index c1ed11a3e16c..fc8cdb8853f5 100644
--- a/tools/perf/util/map.h
+++ b/tools/perf/util/map.h
@@ -8,6 +8,8 @@
 #include <stdbool.h>
 #include <linux/types.h>
 
+#include "perf.h"  /* for perf_has_index */
+
 enum map_type {
 	MAP__FUNCTION = 0,
 	MAP__VARIABLE,
@@ -166,8 +168,10 @@ void map__reloc_vmlinux(struct map *map);
 size_t __map_groups__fprintf_maps(struct map_groups *mg, enum map_type type,
 				  FILE *fp);
 void maps__insert(struct rb_root *maps, struct map *map);
+void maps__insert_by_time(struct rb_root *maps, struct map *map);
 void maps__remove(struct rb_root *maps, struct map *map);
 struct map *maps__find(struct rb_root *maps, u64 addr);
+struct map *maps__find_by_time(struct rb_root *maps, u64 addr, u64 timestamp);
 struct map *maps__first(struct rb_root *maps);
 struct map *maps__next(struct map *map);
 void map_groups__init(struct map_groups *mg, struct machine *machine);
@@ -185,6 +189,17 @@ static inline void map_groups__insert(struct map_groups *mg, struct map *map)
 	map->groups = mg;
 }
 
+static inline void map_groups__insert_by_time(struct map_groups *mg,
+					      struct map *map)
+{
+	if (perf_has_index)
+		maps__insert_by_time(&mg->maps[map->type], map);
+	else
+		maps__insert(&mg->maps[map->type], map);
+
+	map->groups = mg;
+}
+
 static inline void map_groups__remove(struct map_groups *mg, struct map *map)
 {
 	maps__remove(&mg->maps[map->type], map);
@@ -196,6 +211,16 @@ static inline struct map *map_groups__find(struct map_groups *mg,
 	return maps__find(&mg->maps[type], addr);
 }
 
+static inline struct map *map_groups__find_by_time(struct map_groups *mg,
+						   enum map_type type, u64 addr,
+						   u64 timestamp)
+{
+	if (!perf_has_index)
+		return maps__find(&mg->maps[type], addr);
+
+	return maps__find_by_time(&mg->maps[type], addr, timestamp);
+}
+
 static inline struct map *map_groups__first(struct map_groups *mg,
 					    enum map_type type)
 {
-- 
2.4.0


^ permalink raw reply related	[flat|nested] 221+ messages in thread

* [PATCH 23/40] perf tools: Use map_groups__find_addr_by_time()
  2015-05-18  0:30 [RFC/PATCHSET 00/40] perf tools: Speed-up perf report by using multi thread (v4) Namhyung Kim
                   ` (21 preceding siblings ...)
  2015-05-18  0:30 ` [PATCH 22/40] perf tools: Introduce map_groups__{insert,find}_by_time() Namhyung Kim
@ 2015-05-18  0:30 ` Namhyung Kim
  2015-05-18  0:30 ` [PATCH 24/40] perf tools: Add testcase for managing maps with time Namhyung Kim
                   ` (16 subsequent siblings)
  39 siblings, 0 replies; 221+ messages in thread
From: Namhyung Kim @ 2015-05-18  0:30 UTC (permalink / raw)
  To: Arnaldo Carvalho de Melo
  Cc: Ingo Molnar, Peter Zijlstra, Jiri Olsa, LKML, David Ahern,
	Adrian Hunter, Andi Kleen, Frederic Weisbecker, Stephane Eranian

Use timestamp to find a corresponding map so that it can find a match
symbol eventually.

Cc: Stephane Eranian <eranian@google.com>
Signed-off-by: Namhyung Kim <namhyung@kernel.org>
---
 tools/perf/util/event.c  | 81 ++++++++++++++++++++++++++++++++++++++++++------
 tools/perf/util/thread.c |  8 +++--
 2 files changed, 77 insertions(+), 12 deletions(-)

diff --git a/tools/perf/util/event.c b/tools/perf/util/event.c
index bb391c20920d..930d45d5a37a 100644
--- a/tools/perf/util/event.c
+++ b/tools/perf/util/event.c
@@ -832,12 +832,11 @@ int perf_event__process(struct perf_tool *tool __maybe_unused,
 	return machine__process_event(machine, event, sample);
 }
 
-static void map_groups__find_addr_map(struct map_groups *mg, u8 cpumode,
-				      enum map_type type, u64 addr,
-				      struct addr_location *al)
+static bool map_groups__set_addr_location(struct map_groups *mg,
+					  struct addr_location *al,
+					  u8 cpumode, u64 addr)
 {
 	struct machine *machine = mg->machine;
-	bool load_map = false;
 
 	al->machine = machine;
 	al->addr = addr;
@@ -846,21 +845,17 @@ static void map_groups__find_addr_map(struct map_groups *mg, u8 cpumode,
 
 	if (machine == NULL) {
 		al->map = NULL;
-		return;
+		return true;
 	}
 
 	BUG_ON(mg == NULL);
 
 	if (cpumode == PERF_RECORD_MISC_KERNEL && perf_host) {
 		al->level = 'k';
-		mg = &machine->kmaps;
-		load_map = true;
 	} else if (cpumode == PERF_RECORD_MISC_USER && perf_host) {
 		al->level = '.';
 	} else if (cpumode == PERF_RECORD_MISC_GUEST_KERNEL && perf_guest) {
 		al->level = 'g';
-		mg = &machine->kmaps;
-		load_map = true;
 	} else if (cpumode == PERF_RECORD_MISC_GUEST_USER && perf_guest) {
 		al->level = 'u';
 	} else {
@@ -876,8 +871,27 @@ static void map_groups__find_addr_map(struct map_groups *mg, u8 cpumode,
 			!perf_host)
 			al->filtered |= (1 << HIST_FILTER__HOST);
 
+		return true;
+	}
+	return false;
+}
+
+static void map_groups__find_addr_map(struct map_groups *mg, u8 cpumode,
+				      enum map_type type, u64 addr,
+				      struct addr_location *al)
+{
+	struct machine *machine = mg->machine;
+	bool load_map = false;
+
+	if (map_groups__set_addr_location(mg, al, cpumode, addr))
 		return;
+
+	if ((cpumode == PERF_RECORD_MISC_KERNEL && perf_host) ||
+	    (cpumode == PERF_RECORD_MISC_GUEST_KERNEL && perf_guest)) {
+		mg = &machine->kmaps;
+		load_map = true;
 	}
+
 try_again:
 	al->map = map_groups__find(mg, type, al->addr);
 	if (al->map == NULL) {
@@ -908,6 +922,53 @@ static void map_groups__find_addr_map(struct map_groups *mg, u8 cpumode,
 	}
 }
 
+static void map_groups__find_addr_map_by_time(struct map_groups *mg, u8 cpumode,
+					      enum map_type type, u64 addr,
+					      struct addr_location *al,
+					      u64 timestamp)
+{
+	struct machine *machine = mg->machine;
+	bool load_map = false;
+
+	if (map_groups__set_addr_location(mg, al, cpumode, addr))
+		return;
+
+	if ((cpumode == PERF_RECORD_MISC_KERNEL && perf_host) ||
+	    (cpumode == PERF_RECORD_MISC_GUEST_KERNEL && perf_guest)) {
+		mg = &machine->kmaps;
+		load_map = true;
+	}
+
+try_again:
+	al->map = map_groups__find_by_time(mg, type, al->addr, timestamp);
+	if (al->map == NULL) {
+		/*
+		 * If this is outside of all known maps, and is a negative
+		 * address, try to look it up in the kernel dso, as it might be
+		 * a vsyscall or vdso (which executes in user-mode).
+		 *
+		 * XXX This is nasty, we should have a symbol list in the
+		 * "[vdso]" dso, but for now lets use the old trick of looking
+		 * in the whole kernel symbol list.
+		 */
+		if (cpumode == PERF_RECORD_MISC_USER && machine &&
+		    mg != &machine->kmaps &&
+		    machine__kernel_ip(machine, al->addr)) {
+			mg = &machine->kmaps;
+			load_map = true;
+			goto try_again;
+		}
+	} else {
+		/*
+		 * Kernel maps might be changed when loading symbols so loading
+		 * must be done prior to using kernel maps.
+		 */
+		if (load_map)
+			map__load(al->map, machine->symbol_filter);
+		al->addr = al->map->map_ip(al->map, al->addr);
+	}
+}
+
 void thread__find_addr_map(struct thread *thread, u8 cpumode,
 			   enum map_type type, u64 addr,
 			   struct addr_location *al)
@@ -928,7 +989,7 @@ void thread__find_addr_map_by_time(struct thread *thread, u8 cpumode,
 		mg = thread->mg;
 
 	al->thread = thread;
-	map_groups__find_addr_map(mg, cpumode, type, addr, al);
+	map_groups__find_addr_map_by_time(mg, cpumode, type, addr, al, timestamp);
 }
 
 void thread__find_addr_location(struct thread *thread,
diff --git a/tools/perf/util/thread.c b/tools/perf/util/thread.c
index e35cd18adc0e..3fa3e558316a 100644
--- a/tools/perf/util/thread.c
+++ b/tools/perf/util/thread.c
@@ -335,8 +335,12 @@ size_t thread__fprintf(struct thread *thread, FILE *fp)
 
 void thread__insert_map(struct thread *thread, struct map *map)
 {
-	map_groups__fixup_overlappings(thread->mg, map, stderr);
-	map_groups__insert(thread->mg, map);
+	if (perf_has_index) {
+		map_groups__insert_by_time(thread->mg, map);
+	} else {
+		map_groups__fixup_overlappings(thread->mg, map, stderr);
+		map_groups__insert(thread->mg, map);
+	}
 }
 
 static int thread__clone_map_groups(struct thread *thread,
-- 
2.4.0


^ permalink raw reply related	[flat|nested] 221+ messages in thread

* [PATCH 24/40] perf tools: Add testcase for managing maps with time
  2015-05-18  0:30 [RFC/PATCHSET 00/40] perf tools: Speed-up perf report by using multi thread (v4) Namhyung Kim
                   ` (22 preceding siblings ...)
  2015-05-18  0:30 ` [PATCH 23/40] perf tools: Use map_groups__find_addr_by_time() Namhyung Kim
@ 2015-05-18  0:30 ` Namhyung Kim
  2015-05-18  0:30 ` [PATCH 25/40] perf tools: Protect dso symbol loading using a mutex Namhyung Kim
                   ` (15 subsequent siblings)
  39 siblings, 0 replies; 221+ messages in thread
From: Namhyung Kim @ 2015-05-18  0:30 UTC (permalink / raw)
  To: Arnaldo Carvalho de Melo
  Cc: Ingo Molnar, Peter Zijlstra, Jiri Olsa, LKML, David Ahern,
	Adrian Hunter, Andi Kleen, Frederic Weisbecker, Stephane Eranian

This tests new map_groups__{insert,find}_by_time() API working
correctly by using 3 * 100 maps.

Cc: Stephane Eranian <eranian@google.com>
Signed-off-by: Namhyung Kim <namhyung@kernel.org>
---
 tools/perf/tests/Build             |  1 +
 tools/perf/tests/builtin-test.c    |  4 ++
 tools/perf/tests/tests.h           |  1 +
 tools/perf/tests/thread-map-time.c | 90 ++++++++++++++++++++++++++++++++++++++
 4 files changed, 96 insertions(+)
 create mode 100644 tools/perf/tests/thread-map-time.c

diff --git a/tools/perf/tests/Build b/tools/perf/tests/Build
index cfd59c61bcd2..43bf2abf8c5a 100644
--- a/tools/perf/tests/Build
+++ b/tools/perf/tests/Build
@@ -28,6 +28,7 @@ perf-y += thread-comm.o
 perf-y += thread-mg-share.o
 perf-y += thread-lookup-time.o
 perf-y += thread-mg-time.o
+perf-y += thread-map-time.o
 perf-y += switch-tracking.o
 perf-y += keep-tracking.o
 perf-y += code-reading.o
diff --git a/tools/perf/tests/builtin-test.c b/tools/perf/tests/builtin-test.c
index c5dbeb3d75b1..ba3a7e5650e0 100644
--- a/tools/perf/tests/builtin-test.c
+++ b/tools/perf/tests/builtin-test.c
@@ -183,6 +183,10 @@ static struct test {
 		.func = test__thread_mg_time,
 	},
 	{
+		.desc = "Test thread map lookup with time",
+		.func = test__thread_map_lookup_time,
+	},
+	{
 		.func = NULL,
 	},
 };
diff --git a/tools/perf/tests/tests.h b/tools/perf/tests/tests.h
index a2e1f729ae23..0363a2c9526b 100644
--- a/tools/perf/tests/tests.h
+++ b/tools/perf/tests/tests.h
@@ -64,6 +64,7 @@ int test__kmod_path__parse(void);
 int test__thread_comm(void);
 int test__thread_lookup_time(void);
 int test__thread_mg_time(void);
+int test__thread_map_lookup_time(void);
 
 #if defined(__x86_64__) || defined(__i386__) || defined(__arm__)
 #ifdef HAVE_DWARF_UNWIND_SUPPORT
diff --git a/tools/perf/tests/thread-map-time.c b/tools/perf/tests/thread-map-time.c
new file mode 100644
index 000000000000..6f28975faeb5
--- /dev/null
+++ b/tools/perf/tests/thread-map-time.c
@@ -0,0 +1,90 @@
+#include "debug.h"
+#include "tests.h"
+#include "machine.h"
+#include "thread.h"
+#include "map.h"
+
+#define PERF_MAP_START  0x40000
+#define LIBC_MAP_START  0x80000
+#define VDSO_MAP_START  0x7F000
+
+#define NR_MAPS  100
+
+static int lookup_maps(struct map_groups *mg)
+{
+	struct map *map;
+	int i, ret = -1;
+	size_t n;
+	struct {
+		const char *path;
+		u64 start;
+	} maps[] = {
+		{ "/usr/bin/perf",	PERF_MAP_START },
+		{ "/usr/lib/libc.so",	LIBC_MAP_START },
+		{ "[vdso]",		VDSO_MAP_START },
+	};
+
+	/* this is needed to insert/find map by time */
+	perf_has_index = true;
+
+	for (n = 0; n < ARRAY_SIZE(maps); n++) {
+		for (i = 0; i < NR_MAPS; i++) {
+			map = map__new2(maps[n].start, dso__new(maps[n].path),
+					MAP__FUNCTION, i * 10000);
+			if (map == NULL) {
+				pr_debug("memory allocation failed\n");
+				goto out;
+			}
+
+			map->end = map->start + 0x1000;
+			map_groups__insert_by_time(mg, map);
+		}
+	}
+
+	if (verbose > 1)
+		map_groups__fprintf(mg, stderr);
+
+	for (n = 0; n < ARRAY_SIZE(maps); n++) {
+		for (i = 0; i < NR_MAPS; i++) {
+			u64 timestamp = i * 10000;
+
+			map = map_groups__find_by_time(mg, MAP__FUNCTION,
+						       maps[n].start,
+						       timestamp);
+
+			TEST_ASSERT_VAL("cannot find map", map);
+			TEST_ASSERT_VAL("addr not matched",
+					map->start == maps[n].start);
+			TEST_ASSERT_VAL("pathname not matched",
+					!strcmp(map->dso->name, maps[n].path));
+			TEST_ASSERT_VAL("timestamp not matched",
+					map->timestamp == timestamp);
+		}
+	}
+
+	ret = 0;
+out:
+	return ret;
+}
+
+/*
+ * This test creates large number of overlapping maps for increasing
+ * time and find a map based on timestamp.
+ */
+int test__thread_map_lookup_time(void)
+{
+	struct machines machines;
+	struct machine *machine;
+	struct thread *t;
+	int ret;
+
+	machines__init(&machines);
+	machine = &machines.host;
+
+	t = machine__findnew_thread(machine, 0, 0);
+
+	ret = lookup_maps(t->mg);
+
+	machine__delete_threads(machine);
+	return ret;
+}
-- 
2.4.0


^ permalink raw reply related	[flat|nested] 221+ messages in thread

* [PATCH 25/40] perf tools: Protect dso symbol loading using a mutex
  2015-05-18  0:30 [RFC/PATCHSET 00/40] perf tools: Speed-up perf report by using multi thread (v4) Namhyung Kim
                   ` (23 preceding siblings ...)
  2015-05-18  0:30 ` [PATCH 24/40] perf tools: Add testcase for managing maps with time Namhyung Kim
@ 2015-05-18  0:30 ` Namhyung Kim
  2015-05-20 12:25   ` [tip:perf/core] perf symbols: " tip-bot for Namhyung Kim
  2015-05-18  0:30 ` [PATCH 26/40] perf tools: Protect dso cache tree using dso->lock Namhyung Kim
                   ` (14 subsequent siblings)
  39 siblings, 1 reply; 221+ messages in thread
From: Namhyung Kim @ 2015-05-18  0:30 UTC (permalink / raw)
  To: Arnaldo Carvalho de Melo
  Cc: Ingo Molnar, Peter Zijlstra, Jiri Olsa, LKML, David Ahern,
	Adrian Hunter, Andi Kleen, Frederic Weisbecker, Stephane Eranian

When multi-thread support for perf report is enabled, it's possible to
access a dso concurrently.  Add a new pthread_mutex to protect it from
concurrent dso__load().

Signed-off-by: Namhyung Kim <namhyung@kernel.org>
---
 tools/perf/util/dso.c    |  2 ++
 tools/perf/util/dso.h    |  1 +
 tools/perf/util/symbol.c | 34 ++++++++++++++++++++++++----------
 3 files changed, 27 insertions(+), 10 deletions(-)

diff --git a/tools/perf/util/dso.c b/tools/perf/util/dso.c
index 7078700233fa..c14d981568fd 100644
--- a/tools/perf/util/dso.c
+++ b/tools/perf/util/dso.c
@@ -936,6 +936,7 @@ struct dso *dso__new(const char *name)
 		RB_CLEAR_NODE(&dso->rb_node);
 		INIT_LIST_HEAD(&dso->node);
 		INIT_LIST_HEAD(&dso->data.open_entry);
+		pthread_mutex_init(&dso->lock, NULL);
 	}
 
 	return dso;
@@ -966,6 +967,7 @@ void dso__delete(struct dso *dso)
 	dso_cache__free(&dso->data.cache);
 	dso__free_a2l(dso);
 	zfree(&dso->symsrc_filename);
+	pthread_mutex_destroy(&dso->lock);
 	free(dso);
 }
 
diff --git a/tools/perf/util/dso.h b/tools/perf/util/dso.h
index 3d79c749934c..b26ec3ab1336 100644
--- a/tools/perf/util/dso.h
+++ b/tools/perf/util/dso.h
@@ -129,6 +129,7 @@ struct dsos {
 struct auxtrace_cache;
 
 struct dso {
+	pthread_mutex_t	 lock;
 	struct list_head node;
 	struct rb_node	 rb_node;	/* rbtree node sorted by long name */
 	struct rb_root	 symbols[MAP__NR_TYPES];
diff --git a/tools/perf/util/symbol.c b/tools/perf/util/symbol.c
index 5d69d3c407e6..6c88c607930f 100644
--- a/tools/perf/util/symbol.c
+++ b/tools/perf/util/symbol.c
@@ -1383,12 +1383,22 @@ int dso__load(struct dso *dso, struct map *map, symbol_filter_t filter)
 	struct symsrc *syms_ss = NULL, *runtime_ss = NULL;
 	bool kmod;
 
-	dso__set_loaded(dso, map->type);
+	pthread_mutex_lock(&dso->lock);
+
+	/* check again under the dso->lock */
+	if (dso__loaded(dso, map->type)) {
+		ret = 1;
+		goto out;
+	}
+
+	if (dso->kernel) {
+		if (dso->kernel == DSO_TYPE_KERNEL)
+			ret = dso__load_kernel_sym(dso, map, filter);
+		else if (dso->kernel == DSO_TYPE_GUEST_KERNEL)
+			ret = dso__load_guest_kernel_sym(dso, map, filter);
 
-	if (dso->kernel == DSO_TYPE_KERNEL)
-		return dso__load_kernel_sym(dso, map, filter);
-	else if (dso->kernel == DSO_TYPE_GUEST_KERNEL)
-		return dso__load_guest_kernel_sym(dso, map, filter);
+		goto out;
+	}
 
 	if (map->groups && map->groups->machine)
 		machine = map->groups->machine;
@@ -1401,18 +1411,18 @@ int dso__load(struct dso *dso, struct map *map, symbol_filter_t filter)
 		struct stat st;
 
 		if (lstat(dso->name, &st) < 0)
-			return -1;
+			goto out;
 
 		if (st.st_uid && (st.st_uid != geteuid())) {
 			pr_warning("File %s not owned by current user or root, "
 				"ignoring it.\n", dso->name);
-			return -1;
+			goto out;
 		}
 
 		ret = dso__load_perf_map(dso, map, filter);
 		dso->symtab_type = ret > 0 ? DSO_BINARY_TYPE__JAVA_JIT :
 					     DSO_BINARY_TYPE__NOT_FOUND;
-		return ret;
+		goto out;
 	}
 
 	if (machine)
@@ -1420,7 +1430,7 @@ int dso__load(struct dso *dso, struct map *map, symbol_filter_t filter)
 
 	name = malloc(PATH_MAX);
 	if (!name)
-		return -1;
+		goto out;
 
 	kmod = dso->symtab_type == DSO_BINARY_TYPE__SYSTEM_PATH_KMODULE ||
 		dso->symtab_type == DSO_BINARY_TYPE__SYSTEM_PATH_KMODULE_COMP ||
@@ -1501,7 +1511,11 @@ int dso__load(struct dso *dso, struct map *map, symbol_filter_t filter)
 out_free:
 	free(name);
 	if (ret < 0 && strstr(dso->name, " (deleted)") != NULL)
-		return 0;
+		ret = 0;
+out:
+	dso__set_loaded(dso, map->type);
+	pthread_mutex_unlock(&dso->lock);
+
 	return ret;
 }
 
-- 
2.4.0


^ permalink raw reply related	[flat|nested] 221+ messages in thread

* [PATCH 26/40] perf tools: Protect dso cache tree using dso->lock
  2015-05-18  0:30 [RFC/PATCHSET 00/40] perf tools: Speed-up perf report by using multi thread (v4) Namhyung Kim
                   ` (24 preceding siblings ...)
  2015-05-18  0:30 ` [PATCH 25/40] perf tools: Protect dso symbol loading using a mutex Namhyung Kim
@ 2015-05-18  0:30 ` Namhyung Kim
  2015-05-20 12:25   ` [tip:perf/core] perf symbols: Protect dso cache tree using dso-> lock tip-bot for Namhyung Kim
  2015-05-18  0:30 ` [PATCH 27/40] perf tools: Protect dso cache fd with a mutex Namhyung Kim
                   ` (13 subsequent siblings)
  39 siblings, 1 reply; 221+ messages in thread
From: Namhyung Kim @ 2015-05-18  0:30 UTC (permalink / raw)
  To: Arnaldo Carvalho de Melo
  Cc: Ingo Molnar, Peter Zijlstra, Jiri Olsa, LKML, David Ahern,
	Adrian Hunter, Andi Kleen, Frederic Weisbecker, Stephane Eranian

The dso cache is accessed during dwarf callchain unwind and it might
be processed concurrently when multi-thread report is enabled.
Protect it under dso->lock.

Note that it doesn't protect dso_cache__find().  I think it's safe to
access to the cache tree without the lock since we don't delete nodes.
It it missed an existing node due to rotation, it'll find it during
dso_cache__insert() anyway.

Signed-off-by: Namhyung Kim <namhyung@kernel.org>
---
 tools/perf/util/dso.c | 34 +++++++++++++++++++++++++++-------
 1 file changed, 27 insertions(+), 7 deletions(-)

diff --git a/tools/perf/util/dso.c b/tools/perf/util/dso.c
index c14d981568fd..425989e85302 100644
--- a/tools/perf/util/dso.c
+++ b/tools/perf/util/dso.c
@@ -495,10 +495,12 @@ bool dso__data_status_seen(struct dso *dso, enum dso_data_status_seen by)
 }
 
 static void
-dso_cache__free(struct rb_root *root)
+dso_cache__free(struct dso *dso)
 {
+	struct rb_root *root = &dso->data.cache;
 	struct rb_node *next = rb_first(root);
 
+	pthread_mutex_lock(&dso->lock);
 	while (next) {
 		struct dso_cache *cache;
 
@@ -507,10 +509,12 @@ dso_cache__free(struct rb_root *root)
 		rb_erase(&cache->rb_node, root);
 		free(cache);
 	}
+	pthread_mutex_unlock(&dso->lock);
 }
 
-static struct dso_cache *dso_cache__find(const struct rb_root *root, u64 offset)
+static struct dso_cache *dso_cache__find(struct dso *dso, u64 offset)
 {
+	const struct rb_root *root = &dso->data.cache;
 	struct rb_node * const *p = &root->rb_node;
 	const struct rb_node *parent = NULL;
 	struct dso_cache *cache;
@@ -529,17 +533,20 @@ static struct dso_cache *dso_cache__find(const struct rb_root *root, u64 offset)
 		else
 			return cache;
 	}
+
 	return NULL;
 }
 
-static void
-dso_cache__insert(struct rb_root *root, struct dso_cache *new)
+static struct dso_cache *
+dso_cache__insert(struct dso *dso, struct dso_cache *new)
 {
+	struct rb_root *root = &dso->data.cache;
 	struct rb_node **p = &root->rb_node;
 	struct rb_node *parent = NULL;
 	struct dso_cache *cache;
 	u64 offset = new->offset;
 
+	pthread_mutex_lock(&dso->lock);
 	while (*p != NULL) {
 		u64 end;
 
@@ -551,10 +558,17 @@ dso_cache__insert(struct rb_root *root, struct dso_cache *new)
 			p = &(*p)->rb_left;
 		else if (offset >= end)
 			p = &(*p)->rb_right;
+		else
+			goto out;
 	}
 
 	rb_link_node(&new->rb_node, parent, p);
 	rb_insert_color(&new->rb_node, root);
+
+	cache = NULL;
+out:
+	pthread_mutex_unlock(&dso->lock);
+	return cache;
 }
 
 static ssize_t
@@ -572,6 +586,7 @@ static ssize_t
 dso_cache__read(struct dso *dso, u64 offset, u8 *data, ssize_t size)
 {
 	struct dso_cache *cache;
+	struct dso_cache *old;
 	ssize_t ret;
 
 	do {
@@ -591,7 +606,12 @@ dso_cache__read(struct dso *dso, u64 offset, u8 *data, ssize_t size)
 
 		cache->offset = cache_offset;
 		cache->size   = ret;
-		dso_cache__insert(&dso->data.cache, cache);
+		old = dso_cache__insert(dso, cache);
+		if (old) {
+			/* we lose the race */
+			free(cache);
+			cache = old;
+		}
 
 		ret = dso_cache__memcpy(cache, offset, data, size);
 
@@ -608,7 +628,7 @@ static ssize_t dso_cache_read(struct dso *dso, u64 offset,
 {
 	struct dso_cache *cache;
 
-	cache = dso_cache__find(&dso->data.cache, offset);
+	cache = dso_cache__find(dso, offset);
 	if (cache)
 		return dso_cache__memcpy(cache, offset, data, size);
 	else
@@ -964,7 +984,7 @@ void dso__delete(struct dso *dso)
 
 	dso__data_close(dso);
 	auxtrace_cache__free(dso->auxtrace_cache);
-	dso_cache__free(&dso->data.cache);
+	dso_cache__free(dso);
 	dso__free_a2l(dso);
 	zfree(&dso->symsrc_filename);
 	pthread_mutex_destroy(&dso->lock);
-- 
2.4.0


^ permalink raw reply related	[flat|nested] 221+ messages in thread

* [PATCH 27/40] perf tools: Protect dso cache fd with a mutex
  2015-05-18  0:30 [RFC/PATCHSET 00/40] perf tools: Speed-up perf report by using multi thread (v4) Namhyung Kim
                   ` (25 preceding siblings ...)
  2015-05-18  0:30 ` [PATCH 26/40] perf tools: Protect dso cache tree using dso->lock Namhyung Kim
@ 2015-05-18  0:30 ` Namhyung Kim
  2015-05-20 12:25   ` [tip:perf/core] " tip-bot for Namhyung Kim
  2015-05-18  0:30 ` [PATCH 28/40] perf callchain: Maintain libunwind's address space in map_groups Namhyung Kim
                   ` (12 subsequent siblings)
  39 siblings, 1 reply; 221+ messages in thread
From: Namhyung Kim @ 2015-05-18  0:30 UTC (permalink / raw)
  To: Arnaldo Carvalho de Melo
  Cc: Ingo Molnar, Peter Zijlstra, Jiri Olsa, LKML, David Ahern,
	Adrian Hunter, Andi Kleen, Frederic Weisbecker, Stephane Eranian

When dso cache is accessed in multi-thread environment, it's possible
to close other dso->data.fd during operation due to open file limit.
Protect the file descriptors using a separate mutex.

Signed-off-by: Namhyung Kim <namhyung@kernel.org>
---
 tools/perf/util/dso.c | 98 +++++++++++++++++++++++++++++++++++++--------------
 1 file changed, 72 insertions(+), 26 deletions(-)

diff --git a/tools/perf/util/dso.c b/tools/perf/util/dso.c
index 425989e85302..8857287afc14 100644
--- a/tools/perf/util/dso.c
+++ b/tools/perf/util/dso.c
@@ -265,6 +265,7 @@ int __kmod_path__parse(struct kmod_path *m, const char *path,
  */
 static LIST_HEAD(dso__data_open);
 static long dso__data_open_cnt;
+static pthread_mutex_t dso__data_open_lock = PTHREAD_MUTEX_INITIALIZER;
 
 static void dso__list_add(struct dso *dso)
 {
@@ -434,7 +435,9 @@ static void check_data_close(void)
  */
 void dso__data_close(struct dso *dso)
 {
+	pthread_mutex_lock(&dso__data_open_lock);
 	close_dso(dso);
+	pthread_mutex_unlock(&dso__data_open_lock);
 }
 
 /**
@@ -457,6 +460,8 @@ int dso__data_fd(struct dso *dso, struct machine *machine)
 	if (dso->data.status == DSO_DATA_STATUS_ERROR)
 		return -1;
 
+	pthread_mutex_lock(&dso__data_open_lock);
+
 	if (dso->data.fd >= 0)
 		goto out;
 
@@ -479,6 +484,7 @@ int dso__data_fd(struct dso *dso, struct machine *machine)
 	else
 		dso->data.status = DSO_DATA_STATUS_ERROR;
 
+	pthread_mutex_unlock(&dso__data_open_lock);
 	return dso->data.fd;
 }
 
@@ -583,7 +589,8 @@ dso_cache__memcpy(struct dso_cache *cache, u64 offset,
 }
 
 static ssize_t
-dso_cache__read(struct dso *dso, u64 offset, u8 *data, ssize_t size)
+dso_cache__read(struct dso *dso, struct machine *machine,
+		u64 offset, u8 *data, ssize_t size)
 {
 	struct dso_cache *cache;
 	struct dso_cache *old;
@@ -592,11 +599,24 @@ dso_cache__read(struct dso *dso, u64 offset, u8 *data, ssize_t size)
 	do {
 		u64 cache_offset;
 
-		ret = -ENOMEM;
-
 		cache = zalloc(sizeof(*cache) + DSO__DATA_CACHE_SIZE);
 		if (!cache)
-			break;
+			return -ENOMEM;
+
+		pthread_mutex_lock(&dso__data_open_lock);
+
+		/*
+		 * dso->data.fd might be closed if other thread opened another
+		 * file (dso) due to open file limit (RLIMIT_NOFILE).
+		 */
+		if (dso->data.fd < 0) {
+			dso->data.fd = open_dso(dso, machine);
+			if (dso->data.fd < 0) {
+				ret = -errno;
+				dso->data.status = DSO_DATA_STATUS_ERROR;
+				break;
+			}
+		}
 
 		cache_offset = offset & DSO__DATA_CACHE_MASK;
 
@@ -606,6 +626,11 @@ dso_cache__read(struct dso *dso, u64 offset, u8 *data, ssize_t size)
 
 		cache->offset = cache_offset;
 		cache->size   = ret;
+	} while (0);
+
+	pthread_mutex_unlock(&dso__data_open_lock);
+
+	if (ret > 0) {
 		old = dso_cache__insert(dso, cache);
 		if (old) {
 			/* we lose the race */
@@ -614,8 +639,7 @@ dso_cache__read(struct dso *dso, u64 offset, u8 *data, ssize_t size)
 		}
 
 		ret = dso_cache__memcpy(cache, offset, data, size);
-
-	} while (0);
+	}
 
 	if (ret <= 0)
 		free(cache);
@@ -623,8 +647,8 @@ dso_cache__read(struct dso *dso, u64 offset, u8 *data, ssize_t size)
 	return ret;
 }
 
-static ssize_t dso_cache_read(struct dso *dso, u64 offset,
-			      u8 *data, ssize_t size)
+static ssize_t dso_cache_read(struct dso *dso, struct machine *machine,
+			      u64 offset, u8 *data, ssize_t size)
 {
 	struct dso_cache *cache;
 
@@ -632,7 +656,7 @@ static ssize_t dso_cache_read(struct dso *dso, u64 offset,
 	if (cache)
 		return dso_cache__memcpy(cache, offset, data, size);
 	else
-		return dso_cache__read(dso, offset, data, size);
+		return dso_cache__read(dso, machine, offset, data, size);
 }
 
 /*
@@ -640,7 +664,8 @@ static ssize_t dso_cache_read(struct dso *dso, u64 offset,
  * in the rb_tree. Any read to already cached data is served
  * by cached data.
  */
-static ssize_t cached_read(struct dso *dso, u64 offset, u8 *data, ssize_t size)
+static ssize_t cached_read(struct dso *dso, struct machine *machine,
+			   u64 offset, u8 *data, ssize_t size)
 {
 	ssize_t r = 0;
 	u8 *p = data;
@@ -648,7 +673,7 @@ static ssize_t cached_read(struct dso *dso, u64 offset, u8 *data, ssize_t size)
 	do {
 		ssize_t ret;
 
-		ret = dso_cache_read(dso, offset, p, size);
+		ret = dso_cache_read(dso, machine, offset, p, size);
 		if (ret < 0)
 			return ret;
 
@@ -668,21 +693,42 @@ static ssize_t cached_read(struct dso *dso, u64 offset, u8 *data, ssize_t size)
 	return r;
 }
 
-static int data_file_size(struct dso *dso)
+static int data_file_size(struct dso *dso, struct machine *machine)
 {
+	int ret = 0;
 	struct stat st;
 	char sbuf[STRERR_BUFSIZE];
 
-	if (!dso->data.file_size) {
-		if (fstat(dso->data.fd, &st)) {
-			pr_err("dso mmap failed, fstat: %s\n",
-				strerror_r(errno, sbuf, sizeof(sbuf)));
-			return -1;
+	if (dso->data.file_size)
+		return 0;
+
+	pthread_mutex_lock(&dso__data_open_lock);
+
+	/*
+	 * dso->data.fd might be closed if other thread opened another
+	 * file (dso) due to open file limit (RLIMIT_NOFILE).
+	 */
+	if (dso->data.fd < 0) {
+		dso->data.fd = open_dso(dso, machine);
+		if (dso->data.fd < 0) {
+			ret = -errno;
+			dso->data.status = DSO_DATA_STATUS_ERROR;
+			goto out;
 		}
-		dso->data.file_size = st.st_size;
 	}
 
-	return 0;
+	if (fstat(dso->data.fd, &st) < 0) {
+		ret = -errno;
+		pr_err("dso cache fstat failed: %s\n",
+		       strerror_r(errno, sbuf, sizeof(sbuf)));
+		dso->data.status = DSO_DATA_STATUS_ERROR;
+		goto out;
+	}
+	dso->data.file_size = st.st_size;
+
+out:
+	pthread_mutex_unlock(&dso__data_open_lock);
+	return ret;
 }
 
 /**
@@ -700,17 +746,17 @@ off_t dso__data_size(struct dso *dso, struct machine *machine)
 	if (fd < 0)
 		return fd;
 
-	if (data_file_size(dso))
+	if (data_file_size(dso, machine))
 		return -1;
 
 	/* For now just estimate dso data size is close to file size */
 	return dso->data.file_size;
 }
 
-static ssize_t data_read_offset(struct dso *dso, u64 offset,
-				u8 *data, ssize_t size)
+static ssize_t data_read_offset(struct dso *dso, struct machine *machine,
+				u64 offset, u8 *data, ssize_t size)
 {
-	if (data_file_size(dso))
+	if (data_file_size(dso, machine))
 		return -1;
 
 	/* Check the offset sanity. */
@@ -720,7 +766,7 @@ static ssize_t data_read_offset(struct dso *dso, u64 offset,
 	if (offset + size < offset)
 		return -1;
 
-	return cached_read(dso, offset, data, size);
+	return cached_read(dso, machine, offset, data, size);
 }
 
 /**
@@ -737,10 +783,10 @@ static ssize_t data_read_offset(struct dso *dso, u64 offset,
 ssize_t dso__data_read_offset(struct dso *dso, struct machine *machine,
 			      u64 offset, u8 *data, ssize_t size)
 {
-	if (dso__data_fd(dso, machine) < 0)
+	if (dso->data.status == DSO_DATA_STATUS_ERROR)
 		return -1;
 
-	return data_read_offset(dso, offset, data, size);
+	return data_read_offset(dso, machine, offset, data, size);
 }
 
 /**
-- 
2.4.0


^ permalink raw reply related	[flat|nested] 221+ messages in thread

* [PATCH 28/40] perf callchain: Maintain libunwind's address space in map_groups
  2015-05-18  0:30 [RFC/PATCHSET 00/40] perf tools: Speed-up perf report by using multi thread (v4) Namhyung Kim
                   ` (26 preceding siblings ...)
  2015-05-18  0:30 ` [PATCH 27/40] perf tools: Protect dso cache fd with a mutex Namhyung Kim
@ 2015-05-18  0:30 ` Namhyung Kim
  2015-05-18  0:30 ` [PATCH 29/40] perf tools: Add dso__data_get/put_fd() Namhyung Kim
                   ` (11 subsequent siblings)
  39 siblings, 0 replies; 221+ messages in thread
From: Namhyung Kim @ 2015-05-18  0:30 UTC (permalink / raw)
  To: Arnaldo Carvalho de Melo
  Cc: Ingo Molnar, Peter Zijlstra, Jiri Olsa, LKML, David Ahern,
	Adrian Hunter, Andi Kleen, Frederic Weisbecker, Stephane Eranian

Currently the address_space was kept in thread struct but it's more
appropriate to keep it in map_groups as it's maintained with time.
Also we don't need to flush after exec since it still can be accessed
when used with an indexed data file.

Signed-off-by: Namhyung Kim <namhyung@kernel.org>
---
 tools/perf/tests/dwarf-unwind.c    |  4 ++--
 tools/perf/util/map.c              |  5 +++++
 tools/perf/util/map.h              |  1 +
 tools/perf/util/thread.c           |  7 -------
 tools/perf/util/unwind-libunwind.c | 28 +++++++++++++---------------
 tools/perf/util/unwind.h           | 15 ++++++---------
 6 files changed, 27 insertions(+), 33 deletions(-)

diff --git a/tools/perf/tests/dwarf-unwind.c b/tools/perf/tests/dwarf-unwind.c
index 1926799bfcdb..0e572eeabdb7 100644
--- a/tools/perf/tests/dwarf-unwind.c
+++ b/tools/perf/tests/dwarf-unwind.c
@@ -143,6 +143,8 @@ int test__dwarf_unwind(void)
 	struct thread *thread;
 	int err = -1;
 
+	callchain_param.record_mode = CALLCHAIN_DWARF;
+
 	machines__init(&machines);
 
 	machine = machines__find(&machines, HOST_KERNEL_ID);
@@ -151,8 +153,6 @@ int test__dwarf_unwind(void)
 		return -1;
 	}
 
-	callchain_param.record_mode = CALLCHAIN_DWARF;
-
 	if (init_live_machine(machine)) {
 		pr_err("Could not init machine\n");
 		goto out;
diff --git a/tools/perf/util/map.c b/tools/perf/util/map.c
index 8bc016648a34..a35fed9e5eba 100644
--- a/tools/perf/util/map.c
+++ b/tools/perf/util/map.c
@@ -14,6 +14,7 @@
 #include "util.h"
 #include "debug.h"
 #include "machine.h"
+#include "unwind.h"
 #include <linux/string.h>
 
 const char *map_type__name[MAP__NR_TYPES] = {
@@ -431,6 +432,8 @@ void map_groups__init(struct map_groups *mg, struct machine *machine)
 	mg->refcnt = 1;
 	mg->timestamp = 0;
 	INIT_LIST_HEAD(&mg->list);
+
+	unwind__prepare_access(mg);
 }
 
 static void maps__delete(struct rb_root *maps)
@@ -464,6 +467,8 @@ void map_groups__exit(struct map_groups *mg)
 		maps__delete(&mg->maps[i]);
 		maps__delete_removed(&mg->removed_maps[i]);
 	}
+
+	unwind__finish_access(mg);
 }
 
 bool map_groups__empty(struct map_groups *mg)
diff --git a/tools/perf/util/map.h b/tools/perf/util/map.h
index fc8cdb8853f5..a578771dd8f4 100644
--- a/tools/perf/util/map.h
+++ b/tools/perf/util/map.h
@@ -67,6 +67,7 @@ struct map_groups {
 	u64		 timestamp;
 	int		 refcnt;
 	struct list_head list;
+	void		 *priv;
 };
 
 struct map_groups *map_groups__new(struct machine *machine);
diff --git a/tools/perf/util/thread.c b/tools/perf/util/thread.c
index 3fa3e558316a..702f12dc5a90 100644
--- a/tools/perf/util/thread.c
+++ b/tools/perf/util/thread.c
@@ -105,9 +105,6 @@ struct thread *thread__new(pid_t pid, pid_t tid)
 		INIT_LIST_HEAD(&thread->tid_node);
 		INIT_LIST_HEAD(&thread->mg_list);
 
-		if (unwind__prepare_access(thread) < 0)
-			goto err_thread;
-
 		comm_str = malloc(32);
 		if (!comm_str)
 			goto err_thread;
@@ -153,7 +150,6 @@ void thread__delete(struct thread *thread)
 		list_del(&comm->list);
 		comm__free(comm);
 	}
-	unwind__finish_access(thread);
 
 	free(thread);
 }
@@ -250,9 +246,6 @@ int __thread__set_comm(struct thread *thread, const char *str, u64 timestamp,
 				break;
 		}
 		list_add_tail(&new->list, &curr->list);
-
-		if (exec)
-			unwind__flush_access(thread);
 	}
 
 	if (exec) {
diff --git a/tools/perf/util/unwind-libunwind.c b/tools/perf/util/unwind-libunwind.c
index 0978697341c1..b3214f76fd43 100644
--- a/tools/perf/util/unwind-libunwind.c
+++ b/tools/perf/util/unwind-libunwind.c
@@ -32,6 +32,7 @@
 #include "symbol.h"
 #include "util.h"
 #include "debug.h"
+#include "map.h"
 
 extern int
 UNW_OBJ(dwarf_search_unwind_table) (unw_addr_space_t as,
@@ -561,7 +562,7 @@ static unw_accessors_t accessors = {
 	.get_proc_name		= get_proc_name,
 };
 
-int unwind__prepare_access(struct thread *thread)
+int unwind__prepare_access(struct map_groups *mg)
 {
 	unw_addr_space_t addr_space;
 
@@ -575,41 +576,38 @@ int unwind__prepare_access(struct thread *thread)
 	}
 
 	unw_set_caching_policy(addr_space, UNW_CACHE_GLOBAL);
-	thread__set_priv(thread, addr_space);
+	mg->priv = addr_space;
 
 	return 0;
 }
 
-void unwind__flush_access(struct thread *thread)
+void unwind__finish_access(struct map_groups *mg)
 {
-	unw_addr_space_t addr_space;
+	unw_addr_space_t addr_space = mg->priv;
 
 	if (callchain_param.record_mode != CALLCHAIN_DWARF)
 		return;
 
-	addr_space = thread__priv(thread);
-	unw_flush_cache(addr_space, 0, 0);
-}
-
-void unwind__finish_access(struct thread *thread)
-{
-	unw_addr_space_t addr_space;
-
-	if (callchain_param.record_mode != CALLCHAIN_DWARF)
+	if (addr_space == NULL)
 		return;
 
-	addr_space = thread__priv(thread);
 	unw_destroy_addr_space(addr_space);
+	mg->priv = NULL;
 }
 
 static int get_entries(struct unwind_info *ui, unwind_entry_cb_t cb,
 		       void *arg, int max_stack)
 {
+	struct map_groups *mg;
 	unw_addr_space_t addr_space;
 	unw_cursor_t c;
 	int ret;
 
-	addr_space = thread__priv(ui->thread);
+	mg = thread__get_map_groups(ui->thread, ui->sample->time);
+	if (mg == NULL)
+		return -1;
+
+	addr_space = mg->priv;
 	if (addr_space == NULL)
 		return -1;
 
diff --git a/tools/perf/util/unwind.h b/tools/perf/util/unwind.h
index 12790cf94618..c6860b481d07 100644
--- a/tools/perf/util/unwind.h
+++ b/tools/perf/util/unwind.h
@@ -21,17 +21,15 @@ int unwind__get_entries(unwind_entry_cb_t cb, void *arg,
 /* libunwind specific */
 #ifdef HAVE_LIBUNWIND_SUPPORT
 int libunwind__arch_reg_id(int regnum);
-int unwind__prepare_access(struct thread *thread);
-void unwind__flush_access(struct thread *thread);
-void unwind__finish_access(struct thread *thread);
+int unwind__prepare_access(struct map_groups *mg);
+void unwind__finish_access(struct map_groups *mg);
 #else
-static inline int unwind__prepare_access(struct thread *thread __maybe_unused)
+static inline int unwind__prepare_access(struct map_groups *mg __maybe_unused)
 {
 	return 0;
 }
 
-static inline void unwind__flush_access(struct thread *thread __maybe_unused) {}
-static inline void unwind__finish_access(struct thread *thread __maybe_unused) {}
+static inline void unwind__finish_access(struct map_groups *mg __maybe_unused) {}
 #endif
 #else
 static inline int
@@ -44,12 +42,11 @@ unwind__get_entries(unwind_entry_cb_t cb __maybe_unused,
 	return 0;
 }
 
-static inline int unwind__prepare_access(struct thread *thread __maybe_unused)
+static inline int unwind__prepare_access(struct map_groups *mg __maybe_unused)
 {
 	return 0;
 }
 
-static inline void unwind__flush_access(struct thread *thread __maybe_unused) {}
-static inline void unwind__finish_access(struct thread *thread __maybe_unused) {}
+static inline void unwind__finish_access(struct map_groups *mg __maybe_unused) {}
 #endif /* HAVE_DWARF_UNWIND_SUPPORT */
 #endif /* __UNWIND_H */
-- 
2.4.0


^ permalink raw reply related	[flat|nested] 221+ messages in thread

* [PATCH 29/40] perf tools: Add dso__data_get/put_fd()
  2015-05-18  0:30 [RFC/PATCHSET 00/40] perf tools: Speed-up perf report by using multi thread (v4) Namhyung Kim
                   ` (27 preceding siblings ...)
  2015-05-18  0:30 ` [PATCH 28/40] perf callchain: Maintain libunwind's address space in map_groups Namhyung Kim
@ 2015-05-18  0:30 ` Namhyung Kim
  2015-05-18  0:30 ` [PATCH 30/40] perf session: Pass struct events stats to event processing functions Namhyung Kim
                   ` (10 subsequent siblings)
  39 siblings, 0 replies; 221+ messages in thread
From: Namhyung Kim @ 2015-05-18  0:30 UTC (permalink / raw)
  To: Arnaldo Carvalho de Melo
  Cc: Ingo Molnar, Peter Zijlstra, Jiri Olsa, LKML, David Ahern,
	Adrian Hunter, Andi Kleen, Frederic Weisbecker, Stephane Eranian

Using dso__data_fd() in multi-thread environment is not safe since
returned fd can be closed and/or reused anytime.  So convert it to the
dso__data_get/put_fd() pair to protect the access with lock.

The original dso__data_fd() is deprecated and kept only for testing.

Signed-off-by: Namhyung Kim <namhyung@kernel.org>
---
 tools/perf/util/dso.c              | 44 +++++++++++++++++++++++++++++---------
 tools/perf/util/dso.h              |  9 ++++++--
 tools/perf/util/unwind-libunwind.c | 38 +++++++++++++++++++-------------
 3 files changed, 64 insertions(+), 27 deletions(-)

diff --git a/tools/perf/util/dso.c b/tools/perf/util/dso.c
index 8857287afc14..272b207f4bef 100644
--- a/tools/perf/util/dso.c
+++ b/tools/perf/util/dso.c
@@ -441,14 +441,15 @@ void dso__data_close(struct dso *dso)
 }
 
 /**
- * dso__data_fd - Get dso's data file descriptor
+ * dso__data_get_fd - Get dso's data file descriptor
  * @dso: dso object
  * @machine: machine object
  *
  * External interface to find dso's file, open it and
- * returns file descriptor.
+ * returns file descriptor.  Should be paired with
+ * dso__data_put_fd().
  */
-int dso__data_fd(struct dso *dso, struct machine *machine)
+int dso__data_get_fd(struct dso *dso, struct machine *machine)
 {
 	enum dso_binary_type binary_type_data[] = {
 		DSO_BINARY_TYPE__BUILD_ID_CACHE,
@@ -457,11 +458,11 @@ int dso__data_fd(struct dso *dso, struct machine *machine)
 	};
 	int i = 0;
 
+	pthread_mutex_lock(&dso__data_open_lock);
+
 	if (dso->data.status == DSO_DATA_STATUS_ERROR)
 		return -1;
 
-	pthread_mutex_lock(&dso__data_open_lock);
-
 	if (dso->data.fd >= 0)
 		goto out;
 
@@ -484,10 +485,31 @@ int dso__data_fd(struct dso *dso, struct machine *machine)
 	else
 		dso->data.status = DSO_DATA_STATUS_ERROR;
 
-	pthread_mutex_unlock(&dso__data_open_lock);
 	return dso->data.fd;
 }
 
+void dso__data_put_fd(struct dso *dso __maybe_unused)
+{
+	pthread_mutex_unlock(&dso__data_open_lock);
+}
+
+/**
+ * dso__data_get_fd - Get dso's data file descriptor
+ * @dso: dso object
+ * @machine: machine object
+ *
+ * Obsolete interface to find dso's file, open it and
+ * returns file descriptor.  It's not thread-safe in that
+ * the returned fd may be reused for other file.
+ */
+int dso__data_fd(struct dso *dso, struct machine *machine)
+{
+	int fd = dso__data_get_fd(dso, machine);
+
+	dso__data_put_fd(dso);
+	return fd;
+}
+
 bool dso__data_status_seen(struct dso *dso, enum dso_data_status_seen by)
 {
 	u32 flag = 1 << by;
@@ -1200,12 +1222,14 @@ size_t dso__fprintf(struct dso *dso, enum map_type type, FILE *fp)
 enum dso_type dso__type(struct dso *dso, struct machine *machine)
 {
 	int fd;
+	enum dso_type type = DSO__TYPE_UNKNOWN;
 
-	fd = dso__data_fd(dso, machine);
-	if (fd < 0)
-		return DSO__TYPE_UNKNOWN;
+	fd = dso__data_get_fd(dso, machine);
+	if (fd >= 0)
+		type = dso__type_fd(fd);
+	dso__data_put_fd(dso);
 
-	return dso__type_fd(fd);
+	return type;
 }
 
 int dso__strerror_load(struct dso *dso, char *buf, size_t buflen)
diff --git a/tools/perf/util/dso.h b/tools/perf/util/dso.h
index b26ec3ab1336..de9d98c44ae2 100644
--- a/tools/perf/util/dso.h
+++ b/tools/perf/util/dso.h
@@ -240,7 +240,9 @@ int __kmod_path__parse(struct kmod_path *m, const char *path,
 
 /*
  * The dso__data_* external interface provides following functions:
- *   dso__data_fd
+ *   dso__data_fd (obsolete)
+ *   dso__data_get_fd
+ *   dso__data_put_fd
  *   dso__data_close
  *   dso__data_size
  *   dso__data_read_offset
@@ -257,8 +259,9 @@ int __kmod_path__parse(struct kmod_path *m, const char *path,
  * The current usage of the dso__data_* interface is as follows:
  *
  * Get DSO's fd:
- *   int fd = dso__data_fd(dso, machine);
+ *   int fd = dso__data_get_fd(dso, machine);
  *   USE 'fd' SOMEHOW
+ *   dso__data_put_fd(dso)
  *
  * Read DSO's data:
  *   n = dso__data_read_offset(dso_0, &machine, 0, buf, BUFSIZE);
@@ -278,6 +281,8 @@ int __kmod_path__parse(struct kmod_path *m, const char *path,
  * TODO
 */
 int dso__data_fd(struct dso *dso, struct machine *machine);
+int dso__data_get_fd(struct dso *dso, struct machine *machine);
+void dso__data_put_fd(struct dso *dso);
 void dso__data_close(struct dso *dso);
 
 off_t dso__data_size(struct dso *dso, struct machine *machine);
diff --git a/tools/perf/util/unwind-libunwind.c b/tools/perf/util/unwind-libunwind.c
index b3214f76fd43..58a9238b9b3e 100644
--- a/tools/perf/util/unwind-libunwind.c
+++ b/tools/perf/util/unwind-libunwind.c
@@ -270,13 +270,13 @@ static int read_unwind_spec_eh_frame(struct dso *dso, struct machine *machine,
 	u64 offset = dso->data.eh_frame_hdr_offset;
 
 	if (offset == 0) {
-		fd = dso__data_fd(dso, machine);
-		if (fd < 0)
-			return -EINVAL;
-
-		/* Check the .eh_frame section for unwinding info */
-		offset = elf_section_offset(fd, ".eh_frame_hdr");
-		dso->data.eh_frame_hdr_offset = offset;
+		fd = dso__data_get_fd(dso, machine);
+		if (fd >= 0) {
+			/* Check the .eh_frame section for unwinding info */
+			offset = elf_section_offset(fd, ".eh_frame_hdr");
+			dso->data.eh_frame_hdr_offset = offset;
+		}
+		dso__data_put_fd(dso);
 	}
 
 	if (offset)
@@ -295,13 +295,19 @@ static int read_unwind_spec_debug_frame(struct dso *dso,
 	u64 ofs = dso->data.debug_frame_offset;
 
 	if (ofs == 0) {
-		fd = dso__data_fd(dso, machine);
-		if (fd < 0)
-			return -EINVAL;
-
-		/* Check the .debug_frame section for unwinding info */
-		ofs = elf_section_offset(fd, ".debug_frame");
-		dso->data.debug_frame_offset = ofs;
+		int ret = 0;
+
+		fd = dso__data_get_fd(dso, machine);
+		if (fd >= 0) {
+			/* Check the .debug_frame section for unwinding info */
+			ofs = elf_section_offset(fd, ".debug_frame");
+			dso->data.debug_frame_offset = ofs;
+		} else
+			ret = -EINVAL;
+
+		dso__data_put_fd(dso);
+		if (ret)
+			return ret;
 	}
 
 	*offset = ofs;
@@ -356,10 +362,12 @@ find_proc_info(unw_addr_space_t as, unw_word_t ip, unw_proc_info_t *pi,
 #ifndef NO_LIBUNWIND_DEBUG_FRAME
 	/* Check the .debug_frame section for unwinding info */
 	if (!read_unwind_spec_debug_frame(map->dso, ui->machine, &segbase)) {
-		int fd = dso__data_fd(map->dso, ui->machine);
+		int fd = dso__data_get_fd(map->dso, ui->machine);
 		int is_exec = elf_is_exec(fd, map->dso->name);
 		unw_word_t base = is_exec ? 0 : map->start;
 
+		dso__data_put_fd(map->dso);
+
 		memset(&di, 0, sizeof(di));
 		if (dwarf_find_debug_frame(0, &di, ip, base, map->dso->name,
 					   map->start, map->end))
-- 
2.4.0


^ permalink raw reply related	[flat|nested] 221+ messages in thread

* [PATCH 30/40] perf session: Pass struct events stats to event processing functions
  2015-05-18  0:30 [RFC/PATCHSET 00/40] perf tools: Speed-up perf report by using multi thread (v4) Namhyung Kim
                   ` (28 preceding siblings ...)
  2015-05-18  0:30 ` [PATCH 29/40] perf tools: Add dso__data_get/put_fd() Namhyung Kim
@ 2015-05-18  0:30 ` Namhyung Kim
  2015-05-18  0:30 ` [PATCH 31/40] perf hists: Pass hists struct to hist_entry_iter struct Namhyung Kim
                   ` (9 subsequent siblings)
  39 siblings, 0 replies; 221+ messages in thread
From: Namhyung Kim @ 2015-05-18  0:30 UTC (permalink / raw)
  To: Arnaldo Carvalho de Melo
  Cc: Ingo Molnar, Peter Zijlstra, Jiri Olsa, LKML, David Ahern,
	Adrian Hunter, Andi Kleen, Frederic Weisbecker, Stephane Eranian

Pass stats structure so that it can point separate object when used in
multi-thread environment.

Signed-off-by: Namhyung Kim <namhyung@kernel.org>
---
 tools/perf/util/session.c | 71 ++++++++++++++++++++++++++++++-----------------
 1 file changed, 45 insertions(+), 26 deletions(-)

diff --git a/tools/perf/util/session.c b/tools/perf/util/session.c
index bc738216de36..0d080a95d2ff 100644
--- a/tools/perf/util/session.c
+++ b/tools/perf/util/session.c
@@ -18,6 +18,7 @@
 #include "auxtrace.h"
 
 static int perf_session__deliver_event(struct perf_session *session,
+				       struct events_stats *stats,
 				       union perf_event *event,
 				       struct perf_sample *sample,
 				       struct perf_tool *tool,
@@ -106,7 +107,8 @@ static int ordered_events__deliver_event(struct ordered_events *oe,
 		return ret;
 	}
 
-	return perf_session__deliver_event(session, event->event, &sample,
+	return perf_session__deliver_event(session, &session->evlist->stats,
+					   event->event, &sample,
 					   session->tool, event->file_offset);
 }
 
@@ -942,6 +944,7 @@ static struct machine *machines__find_for_cpumode(struct machines *machines,
 }
 
 static int deliver_sample_value(struct perf_evlist *evlist,
+				struct events_stats *stats,
 				struct perf_tool *tool,
 				union perf_event *event,
 				struct perf_sample *sample,
@@ -957,7 +960,7 @@ static int deliver_sample_value(struct perf_evlist *evlist,
 	}
 
 	if (!sid || sid->evsel == NULL) {
-		++evlist->stats.nr_unknown_id;
+		++stats->nr_unknown_id;
 		return 0;
 	}
 
@@ -965,6 +968,7 @@ static int deliver_sample_value(struct perf_evlist *evlist,
 }
 
 static int deliver_sample_group(struct perf_evlist *evlist,
+				struct events_stats *stats,
 				struct perf_tool *tool,
 				union  perf_event *event,
 				struct perf_sample *sample,
@@ -974,7 +978,7 @@ static int deliver_sample_group(struct perf_evlist *evlist,
 	u64 i;
 
 	for (i = 0; i < sample->read.group.nr; i++) {
-		ret = deliver_sample_value(evlist, tool, event, sample,
+		ret = deliver_sample_value(evlist, stats, tool, event, sample,
 					   &sample->read.group.values[i],
 					   machine);
 		if (ret)
@@ -986,6 +990,7 @@ static int deliver_sample_group(struct perf_evlist *evlist,
 
 static int
  perf_evlist__deliver_sample(struct perf_evlist *evlist,
+			     struct events_stats *stats,
 			     struct perf_tool *tool,
 			     union  perf_event *event,
 			     struct perf_sample *sample,
@@ -1002,14 +1007,15 @@ static int
 
 	/* For PERF_SAMPLE_READ we have either single or group mode. */
 	if (read_format & PERF_FORMAT_GROUP)
-		return deliver_sample_group(evlist, tool, event, sample,
+		return deliver_sample_group(evlist, stats, tool, event, sample,
 					    machine);
 	else
-		return deliver_sample_value(evlist, tool, event, sample,
+		return deliver_sample_value(evlist, stats, tool, event, sample,
 					    &sample->read.one, machine);
 }
 
 static int machines__deliver_event(struct machines *machines,
+				   struct events_stats *stats,
 				   struct perf_evlist *evlist,
 				   union perf_event *event,
 				   struct perf_sample *sample,
@@ -1028,14 +1034,15 @@ static int machines__deliver_event(struct machines *machines,
 	case PERF_RECORD_SAMPLE:
 		dump_sample(evsel, event, sample);
 		if (evsel == NULL) {
-			++evlist->stats.nr_unknown_id;
+			++stats->nr_unknown_id;
 			return 0;
 		}
 		if (machine == NULL) {
-			++evlist->stats.nr_unprocessable_samples;
+			++stats->nr_unprocessable_samples;
 			return 0;
 		}
-		return perf_evlist__deliver_sample(evlist, tool, event, sample, evsel, machine);
+		return perf_evlist__deliver_sample(evlist, stats, tool, event,
+						   sample, evsel, machine);
 	case PERF_RECORD_MMAP:
 		return tool->mmap(tool, event, sample, machine);
 	case PERF_RECORD_MMAP2:
@@ -1048,7 +1055,7 @@ static int machines__deliver_event(struct machines *machines,
 		return tool->exit(tool, event, sample, machine);
 	case PERF_RECORD_LOST:
 		if (tool->lost == perf_event__process_lost)
-			evlist->stats.total_lost += event->lost.lost;
+			stats->total_lost += event->lost.lost;
 		return tool->lost(tool, event, sample, machine);
 	case PERF_RECORD_READ:
 		return tool->read(tool, event, sample, evsel, machine);
@@ -1061,12 +1068,13 @@ static int machines__deliver_event(struct machines *machines,
 	case PERF_RECORD_ITRACE_START:
 		return tool->itrace_start(tool, event, sample, machine);
 	default:
-		++evlist->stats.nr_unknown_events;
+		++stats->nr_unknown_events;
 		return -1;
 	}
 }
 
 static int perf_session__deliver_event(struct perf_session *session,
+				       struct events_stats *stats,
 				       union perf_event *event,
 				       struct perf_sample *sample,
 				       struct perf_tool *tool,
@@ -1080,8 +1088,9 @@ static int perf_session__deliver_event(struct perf_session *session,
 	if (ret > 0)
 		return 0;
 
-	return machines__deliver_event(&session->machines, session->evlist,
-				       event, sample, tool, file_offset);
+	return machines__deliver_event(&session->machines, stats,
+				       session->evlist, event, sample,
+				       tool, file_offset);
 }
 
 static s64 perf_session__process_user_event(struct perf_session *session,
@@ -1146,7 +1155,8 @@ int perf_session__deliver_synth_event(struct perf_session *session,
 	if (event->header.type >= PERF_RECORD_USER_TYPE_START)
 		return perf_session__process_user_event(session, event, 0);
 
-	return machines__deliver_event(&session->machines, evlist, event, sample, tool, 0);
+	return machines__deliver_event(&session->machines, &evlist->stats,
+				       evlist, event, sample, tool, 0);
 }
 
 static void event_swap(union perf_event *event, bool sample_id_all)
@@ -1214,7 +1224,9 @@ int perf_session__peek_event(struct perf_session *session, off_t file_offset,
 }
 
 static s64 perf_session__process_event(struct perf_session *session,
-				       union perf_event *event, u64 file_offset)
+				       struct events_stats *stats,
+				       union perf_event *event,
+				       u64 file_offset)
 {
 	struct perf_evlist *evlist = session->evlist;
 	struct perf_tool *tool = session->tool;
@@ -1227,7 +1239,7 @@ static s64 perf_session__process_event(struct perf_session *session,
 	if (event->header.type >= PERF_RECORD_HEADER_MAX)
 		return -EINVAL;
 
-	events_stats__inc(&evlist->stats, event->header.type);
+	events_stats__inc(stats, event->header.type);
 
 	if (event->header.type >= PERF_RECORD_USER_TYPE_START)
 		return perf_session__process_user_event(session, event, file_offset);
@@ -1245,8 +1257,8 @@ static s64 perf_session__process_event(struct perf_session *session,
 			return ret;
 	}
 
-	return perf_session__deliver_event(session, event, &sample, tool,
-					   file_offset);
+	return perf_session__deliver_event(session, stats, event,
+					   &sample, tool, file_offset);
 }
 
 void perf_event_header__bswap(struct perf_event_header *hdr)
@@ -1274,9 +1286,9 @@ static struct thread *perf_session__register_idle_thread(struct perf_session *se
 	return thread;
 }
 
-static void perf_session__warn_about_errors(const struct perf_session *session)
+static void perf_session__warn_about_errors(const struct perf_session *session,
+					    const struct events_stats *stats)
 {
-	const struct events_stats *stats = &session->evlist->stats;
 	const struct ordered_events *oe = &session->ordered_events;
 
 	if (session->tool->lost == perf_event__process_lost &&
@@ -1326,6 +1338,7 @@ volatile int session_done;
 static int __perf_session__process_pipe_events(struct perf_session *session)
 {
 	struct ordered_events *oe = &session->ordered_events;
+	struct events_stats *stats = &session->evlist->stats;
 	struct perf_tool *tool = session->tool;
 	int fd = perf_data_file__fd(session->file);
 	union perf_event *event;
@@ -1390,7 +1403,8 @@ static int __perf_session__process_pipe_events(struct perf_session *session)
 		}
 	}
 
-	if ((skip = perf_session__process_event(session, event, head)) < 0) {
+	if ((skip = perf_session__process_event(session, stats, event,
+						head)) < 0) {
 		pr_err("%#" PRIx64 " [%#x]: failed to process type: %d\n",
 		       head, event->header.size, event->header.type);
 		err = -EINVAL;
@@ -1412,7 +1426,7 @@ static int __perf_session__process_pipe_events(struct perf_session *session)
 	err = auxtrace__flush_events(session, tool);
 out_err:
 	free(buf);
-	perf_session__warn_about_errors(session);
+	perf_session__warn_about_errors(session, stats);
 	ordered_events__free(&session->ordered_events);
 	auxtrace__free_events(session);
 	return err;
@@ -1459,6 +1473,7 @@ fetch_mmaped_event(struct perf_session *session,
 #endif
 
 static int __perf_session__process_events(struct perf_session *session,
+					  struct events_stats *stats,
 					  u64 data_offset, u64 data_size,
 					  u64 file_size)
 {
@@ -1534,7 +1549,8 @@ static int __perf_session__process_events(struct perf_session *session,
 	size = event->header.size;
 
 	if (size < sizeof(struct perf_event_header) ||
-	    (skip = perf_session__process_event(session, event, file_pos)) < 0) {
+	    (skip = perf_session__process_event(session, stats, event,
+						file_pos)) < 0) {
 		pr_err("%#" PRIx64 " [%#x]: failed to process type: %d\n",
 		       file_offset + head, event->header.size,
 		       event->header.type);
@@ -1575,6 +1591,7 @@ static int __perf_session__process_indexed_events(struct perf_session *session)
 	struct perf_data_file *file = session->file;
 	struct perf_tool *tool = session->tool;
 	u64 size = perf_data_file__size(file);
+	struct events_stats *stats = &session->evlist->stats;
 	int err = 0, i;
 
 	for (i = 0; i < (int)session->header.nr_index; i++) {
@@ -1591,13 +1608,14 @@ static int __perf_session__process_indexed_events(struct perf_session *session)
 		if (i > 0)
 			tool->ordered_events = false;
 
-		err = __perf_session__process_events(session, idx->offset,
+		err = __perf_session__process_events(session, stats,
+						     idx->offset,
 						     idx->size, size);
 		if (err < 0)
 			break;
 	}
 
-	perf_session__warn_about_errors(session);
+	perf_session__warn_about_errors(session, stats);
 	return err;
 }
 
@@ -1605,6 +1623,7 @@ int perf_session__process_events(struct perf_session *session)
 {
 	struct perf_data_file *file = session->file;
 	u64 size = perf_data_file__size(file);
+	struct events_stats *stats = &session->evlist->stats;
 	int err;
 
 	if (perf_session__register_idle_thread(session) == NULL)
@@ -1615,12 +1634,12 @@ int perf_session__process_events(struct perf_session *session)
 	if (perf_has_index)
 		return __perf_session__process_indexed_events(session);
 
-	err = __perf_session__process_events(session,
+	err = __perf_session__process_events(session, stats,
 					     session->header.data_offset,
 					     session->header.data_size,
 					     size);
 
-	perf_session__warn_about_errors(session);
+	perf_session__warn_about_errors(session, stats);
 	return err;
 }
 
-- 
2.4.0


^ permalink raw reply related	[flat|nested] 221+ messages in thread

* [PATCH 31/40] perf hists: Pass hists struct to hist_entry_iter struct
  2015-05-18  0:30 [RFC/PATCHSET 00/40] perf tools: Speed-up perf report by using multi thread (v4) Namhyung Kim
                   ` (29 preceding siblings ...)
  2015-05-18  0:30 ` [PATCH 30/40] perf session: Pass struct events stats to event processing functions Namhyung Kim
@ 2015-05-18  0:30 ` Namhyung Kim
  2015-05-18  0:30 ` [PATCH 32/40] perf tools: Move BUILD_ID_SIZE definition to perf.h Namhyung Kim
                   ` (8 subsequent siblings)
  39 siblings, 0 replies; 221+ messages in thread
From: Namhyung Kim @ 2015-05-18  0:30 UTC (permalink / raw)
  To: Arnaldo Carvalho de Melo
  Cc: Ingo Molnar, Peter Zijlstra, Jiri Olsa, LKML, David Ahern,
	Adrian Hunter, Andi Kleen, Frederic Weisbecker, Stephane Eranian

This is a preparation for perf report multi-thread support.  When
multi-thread is enable, each thread will have its own hists during the
sample processing.

Signed-off-by: Namhyung Kim <namhyung@kernel.org>
---
 tools/perf/builtin-report.c       |  1 +
 tools/perf/builtin-top.c          |  1 +
 tools/perf/tests/hists_cumulate.c |  1 +
 tools/perf/tests/hists_filter.c   |  1 +
 tools/perf/tests/hists_output.c   |  1 +
 tools/perf/util/hist.c            | 20 +++++++-------------
 tools/perf/util/hist.h            |  1 +
 7 files changed, 13 insertions(+), 13 deletions(-)

diff --git a/tools/perf/builtin-report.c b/tools/perf/builtin-report.c
index decd9e8584b5..5e53eee5a9a7 100644
--- a/tools/perf/builtin-report.c
+++ b/tools/perf/builtin-report.c
@@ -140,6 +140,7 @@ static int process_sample_event(struct perf_tool *tool,
 	struct addr_location al;
 	struct hist_entry_iter iter = {
 		.evsel 			= evsel,
+		.hists 			= evsel__hists(evsel),
 		.sample 		= sample,
 		.hide_unresolved 	= rep->hide_unresolved,
 		.add_entry_cb 		= hist_iter__report_callback,
diff --git a/tools/perf/builtin-top.c b/tools/perf/builtin-top.c
index 6b987424d015..ea6e7bd04f9a 100644
--- a/tools/perf/builtin-top.c
+++ b/tools/perf/builtin-top.c
@@ -776,6 +776,7 @@ static void perf_event__process_sample(struct perf_tool *tool,
 		struct hists *hists = evsel__hists(evsel);
 		struct hist_entry_iter iter = {
 			.evsel		= evsel,
+			.hists 		= evsel__hists(evsel),
 			.sample 	= sample,
 			.add_entry_cb 	= hist_iter__top_callback,
 		};
diff --git a/tools/perf/tests/hists_cumulate.c b/tools/perf/tests/hists_cumulate.c
index 7d82c8be5e36..36f31f839b96 100644
--- a/tools/perf/tests/hists_cumulate.c
+++ b/tools/perf/tests/hists_cumulate.c
@@ -88,6 +88,7 @@ static int add_hist_entries(struct hists *hists, struct machine *machine)
 		};
 		struct hist_entry_iter iter = {
 			.evsel = evsel,
+			.hists = evsel__hists(evsel),
 			.sample	= &sample,
 			.hide_unresolved = false,
 		};
diff --git a/tools/perf/tests/hists_filter.c b/tools/perf/tests/hists_filter.c
index ce48775e6ada..f8077b81b618 100644
--- a/tools/perf/tests/hists_filter.c
+++ b/tools/perf/tests/hists_filter.c
@@ -64,6 +64,7 @@ static int add_hist_entries(struct perf_evlist *evlist,
 			};
 			struct hist_entry_iter iter = {
 				.evsel = evsel,
+				.hists = evsel__hists(evsel),
 				.sample = &sample,
 				.ops = &hist_iter_normal,
 				.hide_unresolved = false,
diff --git a/tools/perf/tests/hists_output.c b/tools/perf/tests/hists_output.c
index adbebc852cc8..bf9efe145260 100644
--- a/tools/perf/tests/hists_output.c
+++ b/tools/perf/tests/hists_output.c
@@ -58,6 +58,7 @@ static int add_hist_entries(struct hists *hists, struct machine *machine)
 		};
 		struct hist_entry_iter iter = {
 			.evsel = evsel,
+			.hists = evsel__hists(evsel),
 			.sample = &sample,
 			.ops = &hist_iter_normal,
 			.hide_unresolved = false,
diff --git a/tools/perf/util/hist.c b/tools/perf/util/hist.c
index b492968913e1..c11f7fdc08fd 100644
--- a/tools/perf/util/hist.c
+++ b/tools/perf/util/hist.c
@@ -505,7 +505,7 @@ iter_add_single_mem_entry(struct hist_entry_iter *iter, struct addr_location *al
 	u64 cost;
 	struct mem_info *mi = iter->priv;
 	struct perf_sample *sample = iter->sample;
-	struct hists *hists = evsel__hists(iter->evsel);
+	struct hists *hists = iter->hists;
 	struct hist_entry *he;
 
 	if (mi == NULL)
@@ -535,8 +535,7 @@ static int
 iter_finish_mem_entry(struct hist_entry_iter *iter,
 		      struct addr_location *al __maybe_unused)
 {
-	struct perf_evsel *evsel = iter->evsel;
-	struct hists *hists = evsel__hists(evsel);
+	struct hists *hists = iter->hists;
 	struct hist_entry *he = iter->he;
 	int err = -EINVAL;
 
@@ -608,8 +607,7 @@ static int
 iter_add_next_branch_entry(struct hist_entry_iter *iter, struct addr_location *al)
 {
 	struct branch_info *bi;
-	struct perf_evsel *evsel = iter->evsel;
-	struct hists *hists = evsel__hists(evsel);
+	struct hists *hists = iter->hists;
 	struct hist_entry *he = NULL;
 	int i = iter->curr;
 	int err = 0;
@@ -656,11 +654,10 @@ iter_prepare_normal_entry(struct hist_entry_iter *iter __maybe_unused,
 static int
 iter_add_single_normal_entry(struct hist_entry_iter *iter, struct addr_location *al)
 {
-	struct perf_evsel *evsel = iter->evsel;
 	struct perf_sample *sample = iter->sample;
 	struct hist_entry *he;
 
-	he = __hists__add_entry(evsel__hists(evsel), al, iter->parent, NULL, NULL,
+	he = __hists__add_entry(iter->hists, al, iter->parent, NULL, NULL,
 				sample->period, sample->weight,
 				sample->transaction, sample->time, true);
 	if (he == NULL)
@@ -675,7 +672,6 @@ iter_finish_normal_entry(struct hist_entry_iter *iter,
 			 struct addr_location *al __maybe_unused)
 {
 	struct hist_entry *he = iter->he;
-	struct perf_evsel *evsel = iter->evsel;
 	struct perf_sample *sample = iter->sample;
 
 	if (he == NULL)
@@ -683,7 +679,7 @@ iter_finish_normal_entry(struct hist_entry_iter *iter,
 
 	iter->he = NULL;
 
-	hists__inc_nr_samples(evsel__hists(evsel), he->filtered);
+	hists__inc_nr_samples(iter->hists, he->filtered);
 
 	return hist_entry__append_callchain(he, sample);
 }
@@ -715,8 +711,7 @@ static int
 iter_add_single_cumulative_entry(struct hist_entry_iter *iter,
 				 struct addr_location *al)
 {
-	struct perf_evsel *evsel = iter->evsel;
-	struct hists *hists = evsel__hists(evsel);
+	struct hists *hists = iter->hists;
 	struct perf_sample *sample = iter->sample;
 	struct hist_entry **he_cache = iter->priv;
 	struct hist_entry *he;
@@ -761,7 +756,6 @@ static int
 iter_add_next_cumulative_entry(struct hist_entry_iter *iter,
 			       struct addr_location *al)
 {
-	struct perf_evsel *evsel = iter->evsel;
 	struct perf_sample *sample = iter->sample;
 	struct hist_entry **he_cache = iter->priv;
 	struct hist_entry *he;
@@ -795,7 +789,7 @@ iter_add_next_cumulative_entry(struct hist_entry_iter *iter,
 		}
 	}
 
-	he = __hists__add_entry(evsel__hists(evsel), al, iter->parent, NULL, NULL,
+	he = __hists__add_entry(iter->hists, al, iter->parent, NULL, NULL,
 				sample->period, sample->weight,
 				sample->transaction, sample->time, false);
 	if (he == NULL)
diff --git a/tools/perf/util/hist.h b/tools/perf/util/hist.h
index d47f6730f6a4..e5e7b7af6c56 100644
--- a/tools/perf/util/hist.h
+++ b/tools/perf/util/hist.h
@@ -86,6 +86,7 @@ struct hist_entry_iter {
 
 	bool hide_unresolved;
 
+	struct hists *hists;
 	struct perf_evsel *evsel;
 	struct perf_sample *sample;
 	struct hist_entry *he;
-- 
2.4.0


^ permalink raw reply related	[flat|nested] 221+ messages in thread

* [PATCH 32/40] perf tools: Move BUILD_ID_SIZE definition to perf.h
  2015-05-18  0:30 [RFC/PATCHSET 00/40] perf tools: Speed-up perf report by using multi thread (v4) Namhyung Kim
                   ` (30 preceding siblings ...)
  2015-05-18  0:30 ` [PATCH 31/40] perf hists: Pass hists struct to hist_entry_iter struct Namhyung Kim
@ 2015-05-18  0:30 ` Namhyung Kim
  2015-05-18  0:30 ` [PATCH 33/40] perf session: Separate struct machines from session Namhyung Kim
                   ` (7 subsequent siblings)
  39 siblings, 0 replies; 221+ messages in thread
From: Namhyung Kim @ 2015-05-18  0:30 UTC (permalink / raw)
  To: Arnaldo Carvalho de Melo
  Cc: Ingo Molnar, Peter Zijlstra, Jiri Olsa, LKML, David Ahern,
	Adrian Hunter, Andi Kleen, Frederic Weisbecker, Stephane Eranian

The util/event.h includes util/build-id.h only for BUILD_ID_SIZE.
This is a problem when I include util/event.h from util/tool.h which
is also included by util/build-id.h since it now makes a circular
dependency resulting in incomplete type error.

Signed-off-by: Namhyung Kim <namhyung@kernel.org>
---
 tools/perf/perf.h          | 1 +
 tools/perf/util/build-id.h | 2 --
 tools/perf/util/dso.h      | 1 +
 tools/perf/util/event.h    | 1 -
 4 files changed, 2 insertions(+), 3 deletions(-)

diff --git a/tools/perf/perf.h b/tools/perf/perf.h
index 192d936020ea..1f12336ca5a8 100644
--- a/tools/perf/perf.h
+++ b/tools/perf/perf.h
@@ -30,6 +30,7 @@ static inline unsigned long long rdclock(void)
 }
 
 #define MAX_NR_CPUS			1024
+#define BUILD_ID_SIZE			20
 
 extern const char *input_name;
 extern bool perf_host, perf_guest;
diff --git a/tools/perf/util/build-id.h b/tools/perf/util/build-id.h
index 85011222cc14..e71304c9c86f 100644
--- a/tools/perf/util/build-id.h
+++ b/tools/perf/util/build-id.h
@@ -1,8 +1,6 @@
 #ifndef PERF_BUILD_ID_H_
 #define PERF_BUILD_ID_H_ 1
 
-#define BUILD_ID_SIZE 20
-
 #include "tool.h"
 #include "strlist.h"
 #include <linux/types.h>
diff --git a/tools/perf/util/dso.h b/tools/perf/util/dso.h
index de9d98c44ae2..aa1f503c30bf 100644
--- a/tools/perf/util/dso.h
+++ b/tools/perf/util/dso.h
@@ -7,6 +7,7 @@
 #include <linux/types.h>
 #include <linux/bitops.h>
 #include "map.h"
+#include "perf.h"
 #include "build-id.h"
 
 enum dso_binary_type {
diff --git a/tools/perf/util/event.h b/tools/perf/util/event.h
index 97179abc80a1..40e02544f861 100644
--- a/tools/perf/util/event.h
+++ b/tools/perf/util/event.h
@@ -6,7 +6,6 @@
 
 #include "../perf.h"
 #include "map.h"
-#include "build-id.h"
 #include "perf_regs.h"
 
 struct mmap_event {
-- 
2.4.0


^ permalink raw reply related	[flat|nested] 221+ messages in thread

* [PATCH 33/40] perf session: Separate struct machines from session
  2015-05-18  0:30 [RFC/PATCHSET 00/40] perf tools: Speed-up perf report by using multi thread (v4) Namhyung Kim
                   ` (31 preceding siblings ...)
  2015-05-18  0:30 ` [PATCH 32/40] perf tools: Move BUILD_ID_SIZE definition to perf.h Namhyung Kim
@ 2015-05-18  0:30 ` Namhyung Kim
  2015-05-18 12:52   ` Arnaldo Carvalho de Melo
  2015-05-18  0:30 ` [PATCH 34/40] perf report: Parallelize perf report using multi-thread Namhyung Kim
                   ` (6 subsequent siblings)
  39 siblings, 1 reply; 221+ messages in thread
From: Namhyung Kim @ 2015-05-18  0:30 UTC (permalink / raw)
  To: Arnaldo Carvalho de Melo
  Cc: Ingo Molnar, Peter Zijlstra, Jiri Olsa, LKML, David Ahern,
	Adrian Hunter, Andi Kleen, Frederic Weisbecker, Stephane Eranian

With multi-thread report, separate sessions can be passed to each
thread, in this case we should keep a single machine state for all
struct sessions.  Separate machines and have a pointer in sessions.

Signed-off-by: Namhyung Kim <namhyung@kernel.org>
---
 tools/perf/builtin-annotate.c |  2 +-
 tools/perf/builtin-kmem.c     | 10 +++++-----
 tools/perf/builtin-kvm.c      |  2 +-
 tools/perf/builtin-record.c   |  4 ++--
 tools/perf/builtin-report.c   |  4 ++--
 tools/perf/builtin-top.c      |  8 ++++----
 tools/perf/builtin-trace.c    |  2 +-
 tools/perf/util/build-id.c    | 16 ++++++++--------
 tools/perf/util/session.c     | 36 +++++++++++++++++++++---------------
 tools/perf/util/session.h     |  6 +++---
 10 files changed, 48 insertions(+), 42 deletions(-)

diff --git a/tools/perf/builtin-annotate.c b/tools/perf/builtin-annotate.c
index 761f902473b7..d4ad323ddfe2 100644
--- a/tools/perf/builtin-annotate.c
+++ b/tools/perf/builtin-annotate.c
@@ -196,7 +196,7 @@ static int __cmd_annotate(struct perf_annotate *ann)
 	struct perf_evsel *pos;
 	u64 total_nr_samples;
 
-	machines__set_symbol_filter(&session->machines, symbol__annotate_init);
+	machines__set_symbol_filter(session->machines, symbol__annotate_init);
 
 	if (ann->cpu_list) {
 		ret = perf_session__cpu_bitmap(session, ann->cpu_list,
diff --git a/tools/perf/builtin-kmem.c b/tools/perf/builtin-kmem.c
index 254614b10c4a..4502954094e5 100644
--- a/tools/perf/builtin-kmem.c
+++ b/tools/perf/builtin-kmem.c
@@ -316,7 +316,7 @@ static int build_alloc_func_list(void)
 	struct symbol *sym;
 	struct rb_node *node;
 	struct alloc_func *func;
-	struct machine *machine = &kmem_session->machines.host;
+	struct machine *machine = &kmem_session->machines->host;
 	regex_t alloc_func_regex;
 	const char pattern[] = "^_?_?(alloc|get_free|get_zeroed)_pages?";
 
@@ -366,7 +366,7 @@ static int build_alloc_func_list(void)
 static u64 find_callsite(struct perf_evsel *evsel, struct perf_sample *sample)
 {
 	struct addr_location al;
-	struct machine *machine = &kmem_session->machines.host;
+	struct machine *machine = &kmem_session->machines->host;
 	struct callchain_cursor_node *node;
 
 	if (alloc_func_list == NULL) {
@@ -949,7 +949,7 @@ static void __print_slab_result(struct rb_root *root,
 				int n_lines, int is_caller)
 {
 	struct rb_node *next;
-	struct machine *machine = &session->machines.host;
+	struct machine *machine = &session->machines->host;
 
 	printf("%.105s\n", graph_dotted_line);
 	printf(" %-34s |",  is_caller ? "Callsite": "Alloc Ptr");
@@ -1010,7 +1010,7 @@ static const char * const migrate_type_str[] = {
 static void __print_page_alloc_result(struct perf_session *session, int n_lines)
 {
 	struct rb_node *next = rb_first(&page_alloc_sorted);
-	struct machine *machine = &session->machines.host;
+	struct machine *machine = &session->machines->host;
 	const char *format;
 	int gfp_len = max(strlen("GFP flags"), max_gfp_len);
 
@@ -1060,7 +1060,7 @@ static void __print_page_alloc_result(struct perf_session *session, int n_lines)
 static void __print_page_caller_result(struct perf_session *session, int n_lines)
 {
 	struct rb_node *next = rb_first(&page_caller_sorted);
-	struct machine *machine = &session->machines.host;
+	struct machine *machine = &session->machines->host;
 	int gfp_len = max(strlen("GFP flags"), max_gfp_len);
 
 	printf("\n%.105s\n", graph_dotted_line);
diff --git a/tools/perf/builtin-kvm.c b/tools/perf/builtin-kvm.c
index 15fecd3dc5d8..de3c7e1d0b80 100644
--- a/tools/perf/builtin-kvm.c
+++ b/tools/perf/builtin-kvm.c
@@ -1392,7 +1392,7 @@ static int kvm_events_live(struct perf_kvm_stat *kvm,
 	kvm->session->evlist = kvm->evlist;
 	perf_session__set_id_hdr_size(kvm->session);
 	ordered_events__set_copy_on_queue(&kvm->session->ordered_events, true);
-	machine__synthesize_threads(&kvm->session->machines.host, &kvm->opts.target,
+	machine__synthesize_threads(&kvm->session->machines->host, &kvm->opts.target,
 				    kvm->evlist->threads, false);
 	err = kvm_live_open_events(kvm);
 	if (err)
diff --git a/tools/perf/builtin-record.c b/tools/perf/builtin-record.c
index 4ddf104f50ff..978ebf648aab 100644
--- a/tools/perf/builtin-record.c
+++ b/tools/perf/builtin-record.c
@@ -684,7 +684,7 @@ static int __cmd_record(struct record *rec, int argc, const char **argv)
 		goto out_child;
 	}
 
-	machine = &session->machines.host;
+	machine = &session->machines->host;
 
 	if (file->is_pipe) {
 		err = perf_event__synthesize_attrs(tool, session,
@@ -735,7 +735,7 @@ static int __cmd_record(struct record *rec, int argc, const char **argv)
 		       "Check /proc/modules permission or run as root.\n");
 
 	if (perf_guest) {
-		machines__process_guests(&session->machines,
+		machines__process_guests(session->machines,
 					 perf_event__synthesize_guest_os, tool);
 	}
 
diff --git a/tools/perf/builtin-report.c b/tools/perf/builtin-report.c
index 5e53eee5a9a7..650d78ad3357 100644
--- a/tools/perf/builtin-report.c
+++ b/tools/perf/builtin-report.c
@@ -353,7 +353,7 @@ static int perf_evlist__tty_browse_hists(struct perf_evlist *evlist,
 
 static void report__warn_kptr_restrict(const struct report *rep)
 {
-	struct map *kernel_map = rep->session->machines.host.vmlinux_maps[MAP__FUNCTION];
+	struct map *kernel_map = rep->session->machines->host.vmlinux_maps[MAP__FUNCTION];
 	struct kmap *kernel_kmap = kernel_map ? map__kmap(kernel_map) : NULL;
 
 	if (kernel_map == NULL ||
@@ -845,7 +845,7 @@ int cmd_report(int argc, const char **argv, const char *prefix __maybe_unused)
 	 */
 	if (ui__has_annotation()) {
 		symbol_conf.priv_size = sizeof(struct annotation);
-		machines__set_symbol_filter(&session->machines,
+		machines__set_symbol_filter(session->machines,
 					    symbol__annotate_init);
 		/*
  		 * For searching by name on the "Browse map details".
diff --git a/tools/perf/builtin-top.c b/tools/perf/builtin-top.c
index ea6e7bd04f9a..0ab663e200ed 100644
--- a/tools/perf/builtin-top.c
+++ b/tools/perf/builtin-top.c
@@ -828,13 +828,13 @@ static void perf_top__mmap_read_idx(struct perf_top *top, int idx)
 			++top->us_samples;
 			if (top->hide_user_symbols)
 				goto next_event;
-			machine = &session->machines.host;
+			machine = &session->machines->host;
 			break;
 		case PERF_RECORD_MISC_KERNEL:
 			++top->kernel_samples;
 			if (top->hide_kernel_symbols)
 				goto next_event;
-			machine = &session->machines.host;
+			machine = &session->machines->host;
 			break;
 		case PERF_RECORD_MISC_GUEST_KERNEL:
 			++top->guest_kernel_samples;
@@ -939,7 +939,7 @@ static int __cmd_top(struct perf_top *top)
 	if (top->session == NULL)
 		return -1;
 
-	machines__set_symbol_filter(&top->session->machines, symbol_filter);
+	machines__set_symbol_filter(top->session->machines, symbol_filter);
 
 	if (!objdump_path) {
 		ret = perf_session_env__lookup_objdump(&top->session->header.env);
@@ -951,7 +951,7 @@ static int __cmd_top(struct perf_top *top)
 	if (ret)
 		goto out_delete;
 
-	machine__synthesize_threads(&top->session->machines.host, &opts->target,
+	machine__synthesize_threads(&top->session->machines->host, &opts->target,
 				    top->evlist->threads, false);
 	ret = perf_top__start_counters(top);
 	if (ret)
diff --git a/tools/perf/builtin-trace.c b/tools/perf/builtin-trace.c
index a05490d06374..f3df6fa596e9 100644
--- a/tools/perf/builtin-trace.c
+++ b/tools/perf/builtin-trace.c
@@ -2418,7 +2418,7 @@ static int trace__replay(struct trace *trace)
 	if (symbol__init(&session->header.env) < 0)
 		goto out;
 
-	trace->host = &session->machines.host;
+	trace->host = &session->machines->host;
 
 	err = perf_session__set_tracepoints_handlers(session, handlers);
 	if (err)
diff --git a/tools/perf/util/build-id.c b/tools/perf/util/build-id.c
index ad8cfcbaa25d..8720f71a96dd 100644
--- a/tools/perf/util/build-id.c
+++ b/tools/perf/util/build-id.c
@@ -221,12 +221,12 @@ static int machine__write_buildid_table(struct machine *machine, int fd)
 int perf_session__write_buildid_table(struct perf_session *session, int fd)
 {
 	struct rb_node *nd;
-	int err = machine__write_buildid_table(&session->machines.host, fd);
+	int err = machine__write_buildid_table(&session->machines->host, fd);
 
 	if (err)
 		return err;
 
-	for (nd = rb_first(&session->machines.guests); nd; nd = rb_next(nd)) {
+	for (nd = rb_first(&session->machines->guests); nd; nd = rb_next(nd)) {
 		struct machine *pos = rb_entry(nd, struct machine, rb_node);
 		err = machine__write_buildid_table(pos, fd);
 		if (err)
@@ -261,11 +261,11 @@ int dsos__hit_all(struct perf_session *session)
 	struct rb_node *nd;
 	int err;
 
-	err = machine__hit_all_dsos(&session->machines.host);
+	err = machine__hit_all_dsos(&session->machines->host);
 	if (err)
 		return err;
 
-	for (nd = rb_first(&session->machines.guests); nd; nd = rb_next(nd)) {
+	for (nd = rb_first(&session->machines->guests); nd; nd = rb_next(nd)) {
 		struct machine *pos = rb_entry(nd, struct machine, rb_node);
 
 		err = machine__hit_all_dsos(pos);
@@ -509,9 +509,9 @@ int perf_session__cache_build_ids(struct perf_session *session)
 	if (mkdir(buildid_dir, 0755) != 0 && errno != EEXIST)
 		return -1;
 
-	ret = machine__cache_build_ids(&session->machines.host);
+	ret = machine__cache_build_ids(&session->machines->host);
 
-	for (nd = rb_first(&session->machines.guests); nd; nd = rb_next(nd)) {
+	for (nd = rb_first(&session->machines->guests); nd; nd = rb_next(nd)) {
 		struct machine *pos = rb_entry(nd, struct machine, rb_node);
 		ret |= machine__cache_build_ids(pos);
 	}
@@ -530,9 +530,9 @@ static bool machine__read_build_ids(struct machine *machine, bool with_hits)
 bool perf_session__read_build_ids(struct perf_session *session, bool with_hits)
 {
 	struct rb_node *nd;
-	bool ret = machine__read_build_ids(&session->machines.host, with_hits);
+	bool ret = machine__read_build_ids(&session->machines->host, with_hits);
 
-	for (nd = rb_first(&session->machines.guests); nd; nd = rb_next(nd)) {
+	for (nd = rb_first(&session->machines->guests); nd; nd = rb_next(nd)) {
 		struct machine *pos = rb_entry(nd, struct machine, rb_node);
 		ret |= machine__read_build_ids(pos, with_hits);
 	}
diff --git a/tools/perf/util/session.c b/tools/perf/util/session.c
index 0d080a95d2ff..955bf5368751 100644
--- a/tools/perf/util/session.c
+++ b/tools/perf/util/session.c
@@ -58,21 +58,21 @@ void perf_session__set_id_hdr_size(struct perf_session *session)
 {
 	u16 id_hdr_size = perf_evlist__id_hdr_size(session->evlist);
 
-	machines__set_id_hdr_size(&session->machines, id_hdr_size);
+	machines__set_id_hdr_size(session->machines, id_hdr_size);
 }
 
 int perf_session__create_kernel_maps(struct perf_session *session)
 {
-	int ret = machine__create_kernel_maps(&session->machines.host);
+	int ret = machine__create_kernel_maps(&session->machines->host);
 
 	if (ret >= 0)
-		ret = machines__create_guest_kernel_maps(&session->machines);
+		ret = machines__create_guest_kernel_maps(session->machines);
 	return ret;
 }
 
 static void perf_session__destroy_kernel_maps(struct perf_session *session)
 {
-	machines__destroy_kernel_maps(&session->machines);
+	machines__destroy_kernel_maps(session->machines);
 }
 
 static bool perf_session__has_comm_exec(struct perf_session *session)
@@ -91,7 +91,7 @@ static void perf_session__set_comm_exec(struct perf_session *session)
 {
 	bool comm_exec = perf_session__has_comm_exec(session);
 
-	machines__set_comm_exec(&session->machines, comm_exec);
+	machines__set_comm_exec(session->machines, comm_exec);
 }
 
 static int ordered_events__deliver_event(struct ordered_events *oe,
@@ -123,7 +123,12 @@ struct perf_session *perf_session__new(struct perf_data_file *file,
 	session->repipe = repipe;
 	session->tool   = tool;
 	INIT_LIST_HEAD(&session->auxtrace_index);
-	machines__init(&session->machines);
+
+	session->machines = malloc(sizeof(*session->machines));
+	if (!session->machines)
+		goto out_delete;
+
+	machines__init(session->machines);
 	ordered_events__init(&session->ordered_events, ordered_events__deliver_event);
 
 	if (file) {
@@ -168,7 +173,7 @@ struct perf_session *perf_session__new(struct perf_data_file *file,
 
 static void perf_session__delete_threads(struct perf_session *session)
 {
-	machine__delete_threads(&session->machines.host);
+	machine__delete_threads(&session->machines->host);
 }
 
 static void perf_session_env__delete(struct perf_session_env *env)
@@ -194,7 +199,8 @@ void perf_session__delete(struct perf_session *session)
 	perf_session__destroy_kernel_maps(session);
 	perf_session__delete_threads(session);
 	perf_session_env__delete(&session->header.env);
-	machines__exit(&session->machines);
+	machines__exit(session->machines);
+	free(session->machines);
 	if (session->file)
 		perf_data_file__close(session->file);
 	free(session->header.index);
@@ -1088,7 +1094,7 @@ static int perf_session__deliver_event(struct perf_session *session,
 	if (ret > 0)
 		return 0;
 
-	return machines__deliver_event(&session->machines, stats,
+	return machines__deliver_event(session->machines, stats,
 				       session->evlist, event, sample,
 				       tool, file_offset);
 }
@@ -1155,7 +1161,7 @@ int perf_session__deliver_synth_event(struct perf_session *session,
 	if (event->header.type >= PERF_RECORD_USER_TYPE_START)
 		return perf_session__process_user_event(session, event, 0);
 
-	return machines__deliver_event(&session->machines, &evlist->stats,
+	return machines__deliver_event(session->machines, &evlist->stats,
 				       evlist, event, sample, tool, 0);
 }
 
@@ -1270,14 +1276,14 @@ void perf_event_header__bswap(struct perf_event_header *hdr)
 
 struct thread *perf_session__findnew(struct perf_session *session, pid_t pid)
 {
-	return machine__findnew_thread(&session->machines.host, -1, pid);
+	return machine__findnew_thread(&session->machines->host, -1, pid);
 }
 
 static struct thread *perf_session__register_idle_thread(struct perf_session *session)
 {
 	struct thread *thread;
 
-	thread = machine__findnew_thread(&session->machines.host, 0, 0);
+	thread = machine__findnew_thread(&session->machines->host, 0, 0);
 	if (thread == NULL || thread__set_comm(thread, "swapper", 0)) {
 		pr_err("problem inserting idle task.\n");
 		thread = NULL;
@@ -1692,13 +1698,13 @@ int maps__set_kallsyms_ref_reloc_sym(struct map **maps,
 
 size_t perf_session__fprintf_dsos(struct perf_session *session, FILE *fp)
 {
-	return machines__fprintf_dsos(&session->machines, fp);
+	return machines__fprintf_dsos(session->machines, fp);
 }
 
 size_t perf_session__fprintf_dsos_buildid(struct perf_session *session, FILE *fp,
 					  bool (skip)(struct dso *dso, int parm), int parm)
 {
-	return machines__fprintf_dsos_buildid(&session->machines, fp, skip, parm);
+	return machines__fprintf_dsos_buildid(session->machines, fp, skip, parm);
 }
 
 size_t perf_session__fprintf_nr_events(struct perf_session *session, FILE *fp)
@@ -1721,7 +1727,7 @@ size_t perf_session__fprintf(struct perf_session *session, FILE *fp)
 	 * FIXME: Here we have to actually print all the machines in this
 	 * session, not just the host...
 	 */
-	return machine__fprintf(&session->machines.host, fp);
+	return machine__fprintf(&session->machines->host, fp);
 }
 
 struct perf_evsel *perf_session__find_first_evtype(struct perf_session *session,
diff --git a/tools/perf/util/session.h b/tools/perf/util/session.h
index b44afc75d1cc..ac1796c7c799 100644
--- a/tools/perf/util/session.h
+++ b/tools/perf/util/session.h
@@ -20,7 +20,7 @@ struct itrace_synth_opts;
 
 struct perf_session {
 	struct perf_header	header;
-	struct machines		machines;
+	struct machines		*machines;
 	struct perf_evlist	*evlist;
 	struct auxtrace		*auxtrace;
 	struct itrace_synth_opts *itrace_synth_opts;
@@ -79,13 +79,13 @@ void perf_session__set_id_hdr_size(struct perf_session *session);
 static inline
 struct machine *perf_session__find_machine(struct perf_session *session, pid_t pid)
 {
-	return machines__find(&session->machines, pid);
+	return machines__find(session->machines, pid);
 }
 
 static inline
 struct machine *perf_session__findnew_machine(struct perf_session *session, pid_t pid)
 {
-	return machines__findnew(&session->machines, pid);
+	return machines__findnew(session->machines, pid);
 }
 
 struct thread *perf_session__findnew(struct perf_session *session, pid_t pid);
-- 
2.4.0


^ permalink raw reply related	[flat|nested] 221+ messages in thread

* [PATCH 34/40] perf report: Parallelize perf report using multi-thread
  2015-05-18  0:30 [RFC/PATCHSET 00/40] perf tools: Speed-up perf report by using multi thread (v4) Namhyung Kim
                   ` (32 preceding siblings ...)
  2015-05-18  0:30 ` [PATCH 33/40] perf session: Separate struct machines from session Namhyung Kim
@ 2015-05-18  0:30 ` Namhyung Kim
  2015-05-19 10:12   ` Jiri Olsa
  2015-05-18  0:30 ` [PATCH 35/40] perf record: Synthesize COMM event for a command line workload Namhyung Kim
                   ` (5 subsequent siblings)
  39 siblings, 1 reply; 221+ messages in thread
From: Namhyung Kim @ 2015-05-18  0:30 UTC (permalink / raw)
  To: Arnaldo Carvalho de Melo
  Cc: Ingo Molnar, Peter Zijlstra, Jiri Olsa, LKML, David Ahern,
	Adrian Hunter, Andi Kleen, Frederic Weisbecker, Stephane Eranian

Introduce perf_session__process_events_mt() to enable multi-thread
sample processing.  It allocates a struct perf_tool_mt and fills
needed info in it.

The session and hists event stats are counted for each thread and
summed after finishing the processing.  Similarly hist entries are
added to per-thread hists first and then move to the original hists
using hists__mt_resort().  This function reuses hists__collapse_
resort() code so makes sort__need_collapse force to true and skips
the collapsing function.

Note that most of preprocessing stage is already done by processing
meta events in dummy tracking evsel first.  We can find corresponding
thread and map based on the sample time and symbol loading and dso
cache access is protected by pthread mutex.

Signed-off-by: Namhyung Kim <namhyung@kernel.org>
---
 tools/perf/util/hist.c    |  75 +++++++++++++++++++----
 tools/perf/util/hist.h    |   3 +
 tools/perf/util/session.c | 153 ++++++++++++++++++++++++++++++++++++++++++++++
 tools/perf/util/session.h |   1 +
 tools/perf/util/tool.h    |  12 ++++
 5 files changed, 231 insertions(+), 13 deletions(-)

diff --git a/tools/perf/util/hist.c b/tools/perf/util/hist.c
index c11f7fdc08fd..1868116cdfb4 100644
--- a/tools/perf/util/hist.c
+++ b/tools/perf/util/hist.c
@@ -940,7 +940,7 @@ void hist_entry__delete(struct hist_entry *he)
  * collapse the histogram
  */
 
-static bool hists__collapse_insert_entry(struct hists *hists __maybe_unused,
+static bool hists__collapse_insert_entry(struct hists *hists,
 					 struct rb_root *root,
 					 struct hist_entry *he)
 {
@@ -977,6 +977,13 @@ static bool hists__collapse_insert_entry(struct hists *hists __maybe_unused,
 	}
 	hists->nr_entries++;
 
+	/*
+	 * For multi-threaded report, he->hists points to a dummy
+	 * hists in the struct perf_tool_mt.  Please see
+	 * perf_session__process_events_mt().
+	 */
+	he->hists = hists;
+
 	rb_link_node(&he->rb_node_in, parent, p);
 	rb_insert_color(&he->rb_node_in, root);
 	return true;
@@ -1004,19 +1011,12 @@ static void hists__apply_filters(struct hists *hists, struct hist_entry *he)
 	hists__filter_entry_by_symbol(hists, he);
 }
 
-void hists__collapse_resort(struct hists *hists, struct ui_progress *prog)
+static void __hists__collapse_resort(struct hists *hists, struct rb_root *root,
+				     struct ui_progress *prog)
 {
-	struct rb_root *root;
 	struct rb_node *next;
 	struct hist_entry *n;
 
-	if (!sort__need_collapse)
-		return;
-
-	hists->nr_entries = 0;
-
-	root = hists__get_rotate_entries_in(hists);
-
 	next = rb_first(root);
 
 	while (next) {
@@ -1039,6 +1039,27 @@ void hists__collapse_resort(struct hists *hists, struct ui_progress *prog)
 	}
 }
 
+void hists__collapse_resort(struct hists *hists, struct ui_progress *prog)
+{
+	struct rb_root *root;
+
+	if (!sort__need_collapse)
+		return;
+
+	hists->nr_entries = 0;
+
+	root = hists__get_rotate_entries_in(hists);
+	__hists__collapse_resort(hists, root, prog);
+}
+
+void hists__mt_resort(struct hists *dst, struct hists *src)
+{
+	struct rb_root *root = src->entries_in;
+
+	sort__need_collapse = 1;
+	__hists__collapse_resort(dst, root, NULL);
+}
+
 static int hist_entry__sort(struct hist_entry *a, struct hist_entry *b)
 {
 	struct perf_hpp_fmt *fmt;
@@ -1268,6 +1289,29 @@ void events_stats__inc(struct events_stats *stats, u32 type)
 	++stats->nr_events[type];
 }
 
+void events_stats__add(struct events_stats *dst, struct events_stats *src)
+{
+	int i;
+
+#define ADD(_field)  dst->_field += src->_field
+
+	ADD(total_period);
+	ADD(total_non_filtered_period);
+	ADD(total_lost);
+	ADD(total_invalid_chains);
+	ADD(nr_non_filtered_samples);
+	ADD(nr_lost_warned);
+	ADD(nr_unknown_events);
+	ADD(nr_invalid_chains);
+	ADD(nr_unknown_id);
+	ADD(nr_unprocessable_samples);
+
+	for (i = 0; i < PERF_RECORD_HEADER_MAX; i++)
+		ADD(nr_events[i]);
+
+#undef ADD
+}
+
 void hists__inc_nr_events(struct hists *hists, u32 type)
 {
 	events_stats__inc(&hists->stats, type);
@@ -1444,16 +1488,21 @@ int perf_hist_config(const char *var, const char *value)
 	return 0;
 }
 
-static int hists_evsel__init(struct perf_evsel *evsel)
+void __hists__init(struct hists *hists)
 {
-	struct hists *hists = evsel__hists(evsel);
-
 	memset(hists, 0, sizeof(*hists));
 	hists->entries_in_array[0] = hists->entries_in_array[1] = RB_ROOT;
 	hists->entries_in = &hists->entries_in_array[0];
 	hists->entries_collapsed = RB_ROOT;
 	hists->entries = RB_ROOT;
 	pthread_mutex_init(&hists->lock, NULL);
+}
+
+static int hists_evsel__init(struct perf_evsel *evsel)
+{
+	struct hists *hists = evsel__hists(evsel);
+
+	__hists__init(hists);
 	return 0;
 }
 
diff --git a/tools/perf/util/hist.h b/tools/perf/util/hist.h
index e5e7b7af6c56..79e96a1adee2 100644
--- a/tools/perf/util/hist.h
+++ b/tools/perf/util/hist.h
@@ -122,6 +122,7 @@ int hist_entry__sort_snprintf(struct hist_entry *he, char *bf, size_t size,
 void hist_entry__delete(struct hist_entry *he);
 
 void hists__output_resort(struct hists *hists, struct ui_progress *prog);
+void hists__mt_resort(struct hists *dst, struct hists *src);
 void hists__collapse_resort(struct hists *hists, struct ui_progress *prog);
 
 void hists__decay_entries(struct hists *hists, bool zap_user, bool zap_kernel);
@@ -134,6 +135,7 @@ void hists__inc_stats(struct hists *hists, struct hist_entry *h);
 void hists__inc_nr_events(struct hists *hists, u32 type);
 void hists__inc_nr_samples(struct hists *hists, bool filtered);
 void events_stats__inc(struct events_stats *stats, u32 type);
+void events_stats__add(struct events_stats *dst, struct events_stats *src);
 size_t events_stats__fprintf(struct events_stats *stats, FILE *fp);
 
 size_t hists__fprintf(struct hists *hists, bool show_header, int max_rows,
@@ -177,6 +179,7 @@ static inline struct hists *evsel__hists(struct perf_evsel *evsel)
 }
 
 int hists__init(void);
+void __hists__init(struct hists *hists);
 
 struct perf_hpp {
 	char *buf;
diff --git a/tools/perf/util/session.c b/tools/perf/util/session.c
index 955bf5368751..8ab65ac54258 100644
--- a/tools/perf/util/session.c
+++ b/tools/perf/util/session.c
@@ -1649,6 +1649,159 @@ int perf_session__process_events(struct perf_session *session)
 	return err;
 }
 
+static void *processing_thread_idx(void *arg)
+{
+	struct perf_tool_mt *mt_tool = arg;
+	struct perf_session *session = mt_tool->session;
+	u64 offset = session->header.index[mt_tool->idx].offset;
+	u64 size = session->header.index[mt_tool->idx].size;
+	u64 file_size = perf_data_file__size(session->file);
+
+	pr_debug("processing samples using thread [%d]\n", mt_tool->idx);
+	if (__perf_session__process_events(session, &mt_tool->stats,
+					   offset, size, file_size) < 0) {
+		pr_err("processing samples failed (thread [%d])\n", mt_tool->idx);
+		return NULL;
+	}
+
+	pr_debug("processing samples done for thread [%d]\n", mt_tool->idx);
+	return arg;
+}
+
+int perf_session__process_events_mt(struct perf_session *session, void *arg)
+{
+	struct perf_data_file *file = session->file;
+	struct perf_evlist *evlist = session->evlist;
+	struct perf_evsel *evsel;
+	u64 nr_entries = 0;
+	struct perf_tool *tool = session->tool;
+	struct perf_session *mt_sessions = NULL;
+	struct perf_tool_mt *mt_tools = NULL;
+	struct perf_session *ms;
+	struct perf_tool_mt *mt;
+	pthread_t *th_id;
+	int err, i, k;
+	int nr_index = session->header.nr_index;
+	u64 size = perf_data_file__size(file);
+
+	if (perf_data_file__is_pipe(file) || !session->header.index) {
+		pr_err("data file doesn't contain the index table\n");
+		return -EINVAL;
+	}
+
+	if (perf_session__register_idle_thread(session) == NULL)
+		return -ENOMEM;
+
+	err = __perf_session__process_events(session, &evlist->stats,
+					     session->header.index[0].offset,
+					     session->header.index[0].size,
+					     size);
+	if (err)
+		return err;
+
+	th_id = calloc(nr_index, sizeof(*th_id));
+	if (th_id == NULL)
+		goto out;
+
+	mt_sessions = calloc(nr_index, sizeof(*mt_sessions));
+	if (mt_sessions == NULL)
+		goto out;
+
+	mt_tools = calloc(nr_index, sizeof(*mt_tools));
+	if (mt_tools == NULL)
+		goto out;
+
+	tool->ordered_events = false;
+
+	for (i = 1; i < nr_index; i++) {
+		ms = &mt_sessions[i];
+		mt = &mt_tools[i];
+
+		ms->file = session->file;
+		ms->evlist = session->evlist;
+		ms->header = session->header;
+		ms->tevent = session->tevent;
+
+		memcpy(&mt->tool, tool, sizeof(*tool));
+
+		mt->hists = calloc(evlist->nr_entries, sizeof(*mt->hists));
+		if (mt->hists == NULL)
+			goto err;
+
+		for (k = 0; k < evlist->nr_entries; k++)
+			__hists__init(&mt->hists[k]);
+
+		mt->session = ms;
+		mt->idx = i;
+		mt->priv = arg;
+
+		pthread_create(&th_id[i], NULL, processing_thread_idx, mt);
+	}
+
+	for (i = 1; i < nr_index; i++) {
+		pthread_join(th_id[i], (void **)&mt);
+		if (mt == NULL) {
+			err = -EINVAL;
+			continue;
+		}
+
+		events_stats__add(&evlist->stats, &mt->stats);
+
+		evlist__for_each(evlist, evsel) {
+			struct hists *hists = evsel__hists(evsel);
+
+			events_stats__add(&hists->stats,
+					  &mt->hists[evsel->idx].stats);
+
+			nr_entries += mt->hists[evsel->idx].nr_entries;
+		}
+	}
+
+	for (i = 1; i < nr_index; i++) {
+		mt = &mt_tools[i];
+
+		evlist__for_each(evlist, evsel) {
+			struct hists *hists = evsel__hists(evsel);
+
+			if (perf_evsel__is_dummy_tracking(evsel))
+				continue;
+
+			hists__mt_resort(hists, &mt->hists[evsel->idx]);
+
+			/* Non-group events are considered as leader */
+			if (symbol_conf.event_group &&
+			    !perf_evsel__is_group_leader(evsel)) {
+				struct hists *leader_hists;
+
+				leader_hists = evsel__hists(evsel->leader);
+				hists__match(leader_hists, hists);
+				hists__link(leader_hists, hists);
+			}
+		}
+	}
+
+out:
+	perf_session__warn_about_errors(session, &evlist->stats);
+
+	if (mt_tools) {
+		for (i = 1; i < nr_index; i++)
+			free(mt_tools[i].hists);
+		free(mt_tools);
+	}
+
+	free(mt_sessions);
+	free(th_id);
+	return err;
+
+err:
+	while (i-- > 1) {
+		pthread_cancel(th_id[i]);
+		pthread_join(th_id[i], NULL);
+	}
+
+	goto out;
+}
+
 bool perf_session__has_traces(struct perf_session *session, const char *msg)
 {
 	struct perf_evsel *evsel;
diff --git a/tools/perf/util/session.h b/tools/perf/util/session.h
index ac1796c7c799..fde658d5d081 100644
--- a/tools/perf/util/session.h
+++ b/tools/perf/util/session.h
@@ -56,6 +56,7 @@ int perf_session__peek_event(struct perf_session *session, off_t file_offset,
 			     struct perf_sample *sample);
 
 int perf_session__process_events(struct perf_session *session);
+int perf_session__process_events_mt(struct perf_session *session, void *arg);
 
 int perf_session__queue_event(struct perf_session *s, union perf_event *event,
 			      struct perf_sample *sample, u64 file_offset);
diff --git a/tools/perf/util/tool.h b/tools/perf/util/tool.h
index 7f282ad1d2bd..acc75dcec71d 100644
--- a/tools/perf/util/tool.h
+++ b/tools/perf/util/tool.h
@@ -2,6 +2,7 @@
 #define __PERF_TOOL_H
 
 #include <stdbool.h>
+#include "util/event.h"
 
 #include <linux/types.h>
 
@@ -13,6 +14,7 @@ struct perf_sample;
 struct perf_tool;
 struct machine;
 struct ordered_events;
+struct hists;
 
 typedef int (*event_sample)(struct perf_tool *tool, union perf_event *event,
 			    struct perf_sample *sample,
@@ -59,4 +61,14 @@ struct perf_tool {
 	bool		ordering_requires_timestamps;
 };
 
+struct perf_tool_mt {
+	struct perf_tool	tool;
+	struct events_stats	stats;
+	struct hists		*hists;
+	struct perf_session	*session;
+	int			idx;
+
+	void			*priv;
+};
+
 #endif /* __PERF_TOOL_H */
-- 
2.4.0


^ permalink raw reply related	[flat|nested] 221+ messages in thread

* [PATCH 35/40] perf record: Synthesize COMM event for a command line workload
  2015-05-18  0:30 [RFC/PATCHSET 00/40] perf tools: Speed-up perf report by using multi thread (v4) Namhyung Kim
                   ` (33 preceding siblings ...)
  2015-05-18  0:30 ` [PATCH 34/40] perf report: Parallelize perf report using multi-thread Namhyung Kim
@ 2015-05-18  0:30 ` Namhyung Kim
  2015-05-18 12:45   ` Arnaldo Carvalho de Melo
  2015-05-18  0:30 ` [PATCH 36/40] perf tools: Fix progress ui to support multi thread Namhyung Kim
                   ` (4 subsequent siblings)
  39 siblings, 1 reply; 221+ messages in thread
From: Namhyung Kim @ 2015-05-18  0:30 UTC (permalink / raw)
  To: Arnaldo Carvalho de Melo
  Cc: Ingo Molnar, Peter Zijlstra, Jiri Olsa, LKML, David Ahern,
	Adrian Hunter, Andi Kleen, Frederic Weisbecker, Stephane Eranian

When perf creates a new child to profile, the events are enabled on
exec().  And in this case, it doesn't synthesize any event for the
child since they'll be generated during exec().  But there's an window
between the enabling and the event generation.

It used to be overcome since samples are only in kernel (so we always
have the map) and the comm is overridden by a later COMM event.
However it won't work anymore since those samples will go to a missing
thread now but the COMM event will create a (current) thread.  This
leads to those early samples (like native_write_msr_safe) not having a
comm but pid (like ':15328').

So it needs to synthesize COMM event for the child explicitly before
enabling so that it can have a correct comm.  But at this time, the
comm will be "perf" since it's not exec-ed yet.

Signed-off-by: Namhyung Kim <namhyung@kernel.org>
---
 tools/perf/builtin-record.c | 18 +++++++++++++++++-
 tools/perf/util/event.c     |  2 +-
 tools/perf/util/event.h     |  5 +++++
 3 files changed, 23 insertions(+), 2 deletions(-)

diff --git a/tools/perf/builtin-record.c b/tools/perf/builtin-record.c
index 978ebf648aab..153f38e35555 100644
--- a/tools/perf/builtin-record.c
+++ b/tools/perf/builtin-record.c
@@ -766,8 +766,24 @@ static int __cmd_record(struct record *rec, int argc, const char **argv)
 	/*
 	 * Let the child rip
 	 */
-	if (forks)
+	if (forks) {
+		union perf_event *comm_event;
+
+		comm_event = malloc(sizeof(*comm_event) + machine->id_hdr_size);
+		if (comm_event == NULL)
+			goto out_child;
+
+		err = perf_event__synthesize_comm(tool, comm_event,
+						  rec->evlist->workload.pid,
+						  process_synthesized_event,
+						  machine);
+		free(comm_event);
+
+		if (err < 0)
+			goto out_child;
+
 		perf_evlist__start_workload(rec->evlist);
+	}
 
 	if (opts->initial_delay) {
 		usleep(opts->initial_delay * 1000);
diff --git a/tools/perf/util/event.c b/tools/perf/util/event.c
index 930d45d5a37a..3adca1302150 100644
--- a/tools/perf/util/event.c
+++ b/tools/perf/util/event.c
@@ -165,7 +165,7 @@ static int perf_event__prepare_comm(union perf_event *event, pid_t pid,
 	return 0;
 }
 
-static pid_t perf_event__synthesize_comm(struct perf_tool *tool,
+pid_t perf_event__synthesize_comm(struct perf_tool *tool,
 					 union perf_event *event, pid_t pid,
 					 perf_event__handler_t process,
 					 struct machine *machine)
diff --git a/tools/perf/util/event.h b/tools/perf/util/event.h
index 40e02544f861..9aacd558ac0f 100644
--- a/tools/perf/util/event.h
+++ b/tools/perf/util/event.h
@@ -452,6 +452,11 @@ int perf_event__synthesize_mmap_events(struct perf_tool *tool,
 				       struct machine *machine,
 				       bool mmap_data);
 
+pid_t perf_event__synthesize_comm(struct perf_tool *tool,
+				  union perf_event *event, pid_t pid,
+				  perf_event__handler_t process,
+				  struct machine *machine);
+
 size_t perf_event__fprintf_comm(union perf_event *event, FILE *fp);
 size_t perf_event__fprintf_mmap(union perf_event *event, FILE *fp);
 size_t perf_event__fprintf_mmap2(union perf_event *event, FILE *fp);
-- 
2.4.0


^ permalink raw reply related	[flat|nested] 221+ messages in thread

* [PATCH 36/40] perf tools: Fix progress ui to support multi thread
  2015-05-18  0:30 [RFC/PATCHSET 00/40] perf tools: Speed-up perf report by using multi thread (v4) Namhyung Kim
                   ` (34 preceding siblings ...)
  2015-05-18  0:30 ` [PATCH 35/40] perf record: Synthesize COMM event for a command line workload Namhyung Kim
@ 2015-05-18  0:30 ` Namhyung Kim
  2015-05-18  0:30 ` [PATCH 37/40] perf report: Add --multi-thread option and config item Namhyung Kim
                   ` (3 subsequent siblings)
  39 siblings, 0 replies; 221+ messages in thread
From: Namhyung Kim @ 2015-05-18  0:30 UTC (permalink / raw)
  To: Arnaldo Carvalho de Melo
  Cc: Ingo Molnar, Peter Zijlstra, Jiri Olsa, LKML, David Ahern,
	Adrian Hunter, Andi Kleen, Frederic Weisbecker, Stephane Eranian

Split ui_progress struct into global and local one.  Each thread
updates local struct without lock and only updates global one if
meaningful progress is done (with lock).

To do that, pass struct ui_progress to __perf_session__process_event()
and set it for the total size of multi-file storage.

Signed-off-by: Namhyung Kim <namhyung@kernel.org>
---
 tools/perf/util/hist.c    |  5 ++--
 tools/perf/util/hist.h    |  3 +-
 tools/perf/util/session.c | 71 ++++++++++++++++++++++++++++++++++++++---------
 tools/perf/util/tool.h    |  3 ++
 4 files changed, 66 insertions(+), 16 deletions(-)

diff --git a/tools/perf/util/hist.c b/tools/perf/util/hist.c
index 1868116cdfb4..eafb09e8c487 100644
--- a/tools/perf/util/hist.c
+++ b/tools/perf/util/hist.c
@@ -1052,12 +1052,13 @@ void hists__collapse_resort(struct hists *hists, struct ui_progress *prog)
 	__hists__collapse_resort(hists, root, prog);
 }
 
-void hists__mt_resort(struct hists *dst, struct hists *src)
+void hists__mt_resort(struct hists *dst, struct hists *src,
+		      struct ui_progress *prog)
 {
 	struct rb_root *root = src->entries_in;
 
 	sort__need_collapse = 1;
-	__hists__collapse_resort(dst, root, NULL);
+	__hists__collapse_resort(dst, root, prog);
 }
 
 static int hist_entry__sort(struct hist_entry *a, struct hist_entry *b)
diff --git a/tools/perf/util/hist.h b/tools/perf/util/hist.h
index 79e96a1adee2..811bd5e69337 100644
--- a/tools/perf/util/hist.h
+++ b/tools/perf/util/hist.h
@@ -122,7 +122,8 @@ int hist_entry__sort_snprintf(struct hist_entry *he, char *bf, size_t size,
 void hist_entry__delete(struct hist_entry *he);
 
 void hists__output_resort(struct hists *hists, struct ui_progress *prog);
-void hists__mt_resort(struct hists *dst, struct hists *src);
+void hists__mt_resort(struct hists *dst, struct hists *src,
+		      struct ui_progress *prog);
 void hists__collapse_resort(struct hists *hists, struct ui_progress *prog);
 
 void hists__decay_entries(struct hists *hists, bool zap_user, bool zap_kernel);
diff --git a/tools/perf/util/session.c b/tools/perf/util/session.c
index 8ab65ac54258..dcb9747bbb49 100644
--- a/tools/perf/util/session.c
+++ b/tools/perf/util/session.c
@@ -1481,7 +1481,8 @@ fetch_mmaped_event(struct perf_session *session,
 static int __perf_session__process_events(struct perf_session *session,
 					  struct events_stats *stats,
 					  u64 data_offset, u64 data_size,
-					  u64 file_size)
+					  u64 file_size,
+					  struct ui_progress *prog)
 {
 	struct ordered_events *oe = &session->ordered_events;
 	struct perf_tool *tool = session->tool;
@@ -1491,7 +1492,6 @@ static int __perf_session__process_events(struct perf_session *session,
 	size_t	mmap_size;
 	char *buf, *mmaps[NUM_MMAPS];
 	union perf_event *event;
-	struct ui_progress prog;
 	s64 skip;
 
 	perf_tool__fill_defaults(tool);
@@ -1503,8 +1503,6 @@ static int __perf_session__process_events(struct perf_session *session,
 	if (data_size && (data_offset + data_size < file_size))
 		file_size = data_offset + data_size;
 
-	ui_progress__init(&prog, file_size, "Processing events...");
-
 	mmap_size = MMAP_SIZE;
 	if (mmap_size > file_size) {
 		mmap_size = file_size;
@@ -1570,7 +1568,7 @@ static int __perf_session__process_events(struct perf_session *session,
 	head += size;
 	file_pos += size;
 
-	ui_progress__update(&prog, size);
+	ui_progress__update(prog, size);
 
 	if (session_done())
 		goto out;
@@ -1585,7 +1583,6 @@ static int __perf_session__process_events(struct perf_session *session,
 		goto out_err;
 	err = auxtrace__flush_events(session, tool);
 out_err:
-	ui_progress__finish();
 	ordered_events__free(&session->ordered_events);
 	auxtrace__free_events(session);
 	session->one_mmap = false;
@@ -1594,12 +1591,15 @@ static int __perf_session__process_events(struct perf_session *session,
 
 static int __perf_session__process_indexed_events(struct perf_session *session)
 {
+	struct ui_progress prog;
 	struct perf_data_file *file = session->file;
 	struct perf_tool *tool = session->tool;
 	u64 size = perf_data_file__size(file);
 	struct events_stats *stats = &session->evlist->stats;
 	int err = 0, i;
 
+	ui_progress__init(&prog, size, "Processing events...");
+
 	for (i = 0; i < (int)session->header.nr_index; i++) {
 		struct perf_file_section *idx = &session->header.index[i];
 
@@ -1616,17 +1616,20 @@ static int __perf_session__process_indexed_events(struct perf_session *session)
 
 		err = __perf_session__process_events(session, stats,
 						     idx->offset,
-						     idx->size, size);
+						     idx->size, size, &prog);
 		if (err < 0)
 			break;
 	}
 
+	ui_progress__finish();
 	perf_session__warn_about_errors(session, stats);
+
 	return err;
 }
 
 int perf_session__process_events(struct perf_session *session)
 {
+	struct ui_progress prog;
 	struct perf_data_file *file = session->file;
 	u64 size = perf_data_file__size(file);
 	struct events_stats *stats = &session->evlist->stats;
@@ -1640,15 +1643,42 @@ int perf_session__process_events(struct perf_session *session)
 	if (perf_has_index)
 		return __perf_session__process_indexed_events(session);
 
+	ui_progress__init(&prog, size, "Processing events...");
+
 	err = __perf_session__process_events(session, stats,
 					     session->header.data_offset,
 					     session->header.data_size,
-					     size);
+					     size, &prog);
 
+	ui_progress__finish();
 	perf_session__warn_about_errors(session, stats);
+
 	return err;
 }
 
+struct ui_progress_ops *orig_progress__ops;
+
+static void mt_progress__update(struct ui_progress *p)
+{
+	struct perf_tool_mt *mt_tool = container_of(p, struct perf_tool_mt, prog);
+	struct ui_progress *gprog = mt_tool->global_prog;
+	static pthread_mutex_t prog_lock = PTHREAD_MUTEX_INITIALIZER;
+
+	pthread_mutex_lock(&prog_lock);
+
+	gprog->curr += p->step;
+	if (gprog->curr >= gprog->next) {
+		gprog->next += gprog->step;
+		orig_progress__ops->update(gprog);
+	}
+
+	pthread_mutex_unlock(&prog_lock);
+}
+
+static struct ui_progress_ops mt_progress__ops = {
+	.update = mt_progress__update,
+};
+
 static void *processing_thread_idx(void *arg)
 {
 	struct perf_tool_mt *mt_tool = arg;
@@ -1657,9 +1687,12 @@ static void *processing_thread_idx(void *arg)
 	u64 size = session->header.index[mt_tool->idx].size;
 	u64 file_size = perf_data_file__size(session->file);
 
+	ui_progress__init(&mt_tool->prog, size, "");
+
 	pr_debug("processing samples using thread [%d]\n", mt_tool->idx);
 	if (__perf_session__process_events(session, &mt_tool->stats,
-					   offset, size, file_size) < 0) {
+					   offset, size, file_size,
+					   &mt_tool->prog) < 0) {
 		pr_err("processing samples failed (thread [%d])\n", mt_tool->idx);
 		return NULL;
 	}
@@ -1679,7 +1712,8 @@ int perf_session__process_events_mt(struct perf_session *session, void *arg)
 	struct perf_tool_mt *mt_tools = NULL;
 	struct perf_session *ms;
 	struct perf_tool_mt *mt;
-	pthread_t *th_id;
+	struct ui_progress prog;
+	pthread_t *th_id = NULL;
 	int err, i, k;
 	int nr_index = session->header.nr_index;
 	u64 size = perf_data_file__size(file);
@@ -1692,12 +1726,18 @@ int perf_session__process_events_mt(struct perf_session *session, void *arg)
 	if (perf_session__register_idle_thread(session) == NULL)
 		return -ENOMEM;
 
+	ui_progress__init(&prog, size, "Processing events...");
+
 	err = __perf_session__process_events(session, &evlist->stats,
 					     session->header.index[0].offset,
 					     session->header.index[0].size,
-					     size);
+					     size, &prog);
 	if (err)
-		return err;
+		goto out;
+
+	orig_progress__ops = ui_progress__ops;
+	ui_progress__ops = &mt_progress__ops;
+	ui_progress__ops->finish = orig_progress__ops->finish;
 
 	th_id = calloc(nr_index, sizeof(*th_id));
 	if (th_id == NULL)
@@ -1734,6 +1774,7 @@ int perf_session__process_events_mt(struct perf_session *session, void *arg)
 		mt->session = ms;
 		mt->idx = i;
 		mt->priv = arg;
+		mt->global_prog = &prog;
 
 		pthread_create(&th_id[i], NULL, processing_thread_idx, mt);
 	}
@@ -1757,6 +1798,9 @@ int perf_session__process_events_mt(struct perf_session *session, void *arg)
 		}
 	}
 
+	ui_progress__ops = orig_progress__ops;
+	ui_progress__init(&prog, nr_entries, "Merging related events...");
+
 	for (i = 1; i < nr_index; i++) {
 		mt = &mt_tools[i];
 
@@ -1766,7 +1810,7 @@ int perf_session__process_events_mt(struct perf_session *session, void *arg)
 			if (perf_evsel__is_dummy_tracking(evsel))
 				continue;
 
-			hists__mt_resort(hists, &mt->hists[evsel->idx]);
+			hists__mt_resort(hists, &mt->hists[evsel->idx], &prog);
 
 			/* Non-group events are considered as leader */
 			if (symbol_conf.event_group &&
@@ -1781,6 +1825,7 @@ int perf_session__process_events_mt(struct perf_session *session, void *arg)
 	}
 
 out:
+	ui_progress__finish();
 	perf_session__warn_about_errors(session, &evlist->stats);
 
 	if (mt_tools) {
diff --git a/tools/perf/util/tool.h b/tools/perf/util/tool.h
index acc75dcec71d..144a414b1d5b 100644
--- a/tools/perf/util/tool.h
+++ b/tools/perf/util/tool.h
@@ -3,6 +3,7 @@
 
 #include <stdbool.h>
 #include "util/event.h"
+#include "ui/progress.h"
 
 #include <linux/types.h>
 
@@ -66,6 +67,8 @@ struct perf_tool_mt {
 	struct events_stats	stats;
 	struct hists		*hists;
 	struct perf_session	*session;
+	struct ui_progress	prog;
+	struct ui_progress	*global_prog;
 	int			idx;
 
 	void			*priv;
-- 
2.4.0


^ permalink raw reply related	[flat|nested] 221+ messages in thread

* [PATCH 37/40] perf report: Add --multi-thread option and config item
  2015-05-18  0:30 [RFC/PATCHSET 00/40] perf tools: Speed-up perf report by using multi thread (v4) Namhyung Kim
                   ` (35 preceding siblings ...)
  2015-05-18  0:30 ` [PATCH 36/40] perf tools: Fix progress ui to support multi thread Namhyung Kim
@ 2015-05-18  0:30 ` Namhyung Kim
  2015-05-18  0:30 ` [PATCH 38/40] perf session: Handle index files generally Namhyung Kim
                   ` (2 subsequent siblings)
  39 siblings, 0 replies; 221+ messages in thread
From: Namhyung Kim @ 2015-05-18  0:30 UTC (permalink / raw)
  To: Arnaldo Carvalho de Melo
  Cc: Ingo Molnar, Peter Zijlstra, Jiri Olsa, LKML, David Ahern,
	Adrian Hunter, Andi Kleen, Frederic Weisbecker, Stephane Eranian

The --multi-thread option is to enable parallel processing so user can
force serial processing even for indexed data file.  It default to false
for now but users also can changes this by setting "report.multi_thread"
config option in ~/.perfconfig file.

Signed-off-by: Namhyung Kim <namhyung@kernel.org>
---
 tools/perf/Documentation/perf-report.txt |  2 +
 tools/perf/builtin-report.c              | 66 +++++++++++++++++++++++++++-----
 2 files changed, 59 insertions(+), 9 deletions(-)

diff --git a/tools/perf/Documentation/perf-report.txt b/tools/perf/Documentation/perf-report.txt
index c33b69f3374f..3917710e2620 100644
--- a/tools/perf/Documentation/perf-report.txt
+++ b/tools/perf/Documentation/perf-report.txt
@@ -353,6 +353,8 @@ OPTIONS
 
 	To disable decoding entirely, use --no-itrace.
 
+--multi-thread::
+	Speed up report by parallelizing sample processing using multi-thread.
 
 include::callchain-overhead-calculation.txt[]
 
diff --git a/tools/perf/builtin-report.c b/tools/perf/builtin-report.c
index 650d78ad3357..4d08e5f0a7bb 100644
--- a/tools/perf/builtin-report.c
+++ b/tools/perf/builtin-report.c
@@ -53,6 +53,7 @@ struct report {
 	bool			mem_mode;
 	bool			header;
 	bool			header_only;
+	bool			multi_thread;
 	int			max_stack;
 	struct perf_read_values	show_threads_values;
 	const char		*pretty_printing_style;
@@ -84,6 +85,10 @@ static int report__config(const char *var, const char *value, void *cb)
 		rep->queue_size = perf_config_u64(var, value);
 		return 0;
 	}
+	if (!strcmp(var, "report.multi-thread")) {
+		rep->multi_thread = perf_config_bool(var, value);
+		return 0;
+	}
 
 	return perf_default_config(var, value, cb);
 }
@@ -130,17 +135,18 @@ static int hist_iter__report_callback(struct hist_entry_iter *iter,
 	return err;
 }
 
-static int process_sample_event(struct perf_tool *tool,
-				union perf_event *event,
-				struct perf_sample *sample,
-				struct perf_evsel *evsel,
-				struct machine *machine)
+static int __process_sample_event(struct perf_tool *tool __maybe_unused,
+				  union perf_event *event,
+				  struct perf_sample *sample,
+				  struct perf_evsel *evsel,
+				  struct machine *machine,
+				  struct hists *hists,
+				  struct report *rep)
 {
-	struct report *rep = container_of(tool, struct report, tool);
 	struct addr_location al;
 	struct hist_entry_iter iter = {
 		.evsel 			= evsel,
-		.hists 			= evsel__hists(evsel),
+		.hists 			= hists,
 		.sample 		= sample,
 		.hide_unresolved 	= rep->hide_unresolved,
 		.add_entry_cb 		= hist_iter__report_callback,
@@ -179,6 +185,31 @@ static int process_sample_event(struct perf_tool *tool,
 	return ret;
 }
 
+static int process_sample_event(struct perf_tool *tool,
+				union perf_event *event,
+				struct perf_sample *sample,
+				struct perf_evsel *evsel,
+				struct machine *machine)
+{
+	struct report *rep = container_of(tool, struct report, tool);
+
+	return __process_sample_event(tool, event, sample, evsel, machine,
+				      evsel__hists(evsel), rep);
+}
+
+static int process_sample_event_mt(struct perf_tool *tool,
+				   union perf_event *event,
+				   struct perf_sample *sample,
+				   struct perf_evsel *evsel,
+				   struct machine *machine)
+{
+	struct perf_tool_mt *mt = container_of(tool, struct perf_tool_mt, tool);
+	struct report *rep = mt->priv;
+
+	return __process_sample_event(tool, event, sample, evsel, machine,
+				      &mt->hists[evsel->idx], rep);
+}
+
 static int process_read_event(struct perf_tool *tool,
 			      union perf_event *event,
 			      struct perf_sample *sample __maybe_unused,
@@ -489,7 +520,12 @@ static int __cmd_report(struct report *rep)
 	if (ret)
 		return ret;
 
-	ret = perf_session__process_events(session);
+	if (rep->multi_thread) {
+		rep->tool.sample = process_sample_event_mt;
+		ret = perf_session__process_events_mt(session, rep);
+	} else {
+		ret = perf_session__process_events(session);
+	}
 	if (ret)
 		return ret;
 
@@ -512,7 +548,12 @@ static int __cmd_report(struct report *rep)
 		}
 	}
 
-	report__collapse_hists(rep);
+	/*
+	 * For multi-thread report, it already calls hists__mt_resort()
+	 * so no need to collapse here.
+	 */
+	if (!rep->multi_thread)
+		report__collapse_hists(rep);
 
 	if (session_done())
 		return 0;
@@ -731,6 +772,8 @@ int cmd_report(int argc, const char **argv, const char *prefix __maybe_unused)
 	OPT_CALLBACK_OPTARG(0, "itrace", &itrace_synth_opts, NULL, "opts",
 			    "Instruction Tracing options",
 			    itrace_parse_synth_opts),
+	OPT_BOOLEAN(0, "multi-thread", &report.multi_thread,
+		    "Speed up sample processing using multi-thead"),
 	OPT_END()
 	};
 	struct perf_data_file file = {
@@ -777,6 +820,11 @@ int cmd_report(int argc, const char **argv, const char *prefix __maybe_unused)
 
 	session->itrace_synth_opts = &itrace_synth_opts;
 
+	if (report.multi_thread && !perf_has_index) {
+		pr_debug("fallback to single thread for normal data file.\n");
+		report.multi_thread = false;
+	}
+
 	report.session = session;
 
 	has_br_stack = perf_header__has_feat(&session->header,
-- 
2.4.0


^ permalink raw reply related	[flat|nested] 221+ messages in thread

* [PATCH 38/40] perf session: Handle index files generally
  2015-05-18  0:30 [RFC/PATCHSET 00/40] perf tools: Speed-up perf report by using multi thread (v4) Namhyung Kim
                   ` (36 preceding siblings ...)
  2015-05-18  0:30 ` [PATCH 37/40] perf report: Add --multi-thread option and config item Namhyung Kim
@ 2015-05-18  0:30 ` Namhyung Kim
  2015-05-19 22:27   ` David Ahern
  2015-05-18  0:30 ` [PATCH 39/40] perf data: Implement 'index' subcommand Namhyung Kim
  2015-05-18  0:30 ` [PATCH Not-for-merge 40/40] perf tools: Disable thread refcount due to bug Namhyung Kim
  39 siblings, 1 reply; 221+ messages in thread
From: Namhyung Kim @ 2015-05-18  0:30 UTC (permalink / raw)
  To: Arnaldo Carvalho de Melo
  Cc: Ingo Molnar, Peter Zijlstra, Jiri Olsa, LKML, David Ahern,
	Adrian Hunter, Andi Kleen, Frederic Weisbecker, Stephane Eranian

The current code assumes that the number of index item and cpu are
matched so it creates that number of threads.  But it's not the case
of non-system-wide session or data came from different machine.

Just creates threads at most number of online cpus and process data.

Signed-off-by: Namhyung Kim <namhyung@kernel.org>
---
 tools/perf/util/session.c | 79 ++++++++++++++++++++++++++++++++++-------------
 tools/perf/util/tool.h    |  1 -
 2 files changed, 58 insertions(+), 22 deletions(-)

diff --git a/tools/perf/util/session.c b/tools/perf/util/session.c
index dcb9747bbb49..5f6c319bd236 100644
--- a/tools/perf/util/session.c
+++ b/tools/perf/util/session.c
@@ -1679,25 +1679,50 @@ static struct ui_progress_ops mt_progress__ops = {
 	.update = mt_progress__update,
 };
 
+static int perf_session__get_index(struct perf_session *session)
+{
+	int ret;
+	static unsigned idx = 1;
+	static pthread_mutex_t idx_lock = PTHREAD_MUTEX_INITIALIZER;
+
+	pthread_mutex_lock(&idx_lock);
+	if (idx < session->header.nr_index)
+		ret = idx++;
+	else
+		ret = -1;
+	pthread_mutex_unlock(&idx_lock);
+
+	return ret;
+}
+
 static void *processing_thread_idx(void *arg)
 {
 	struct perf_tool_mt *mt_tool = arg;
 	struct perf_session *session = mt_tool->session;
-	u64 offset = session->header.index[mt_tool->idx].offset;
-	u64 size = session->header.index[mt_tool->idx].size;
 	u64 file_size = perf_data_file__size(session->file);
+	int idx;
 
-	ui_progress__init(&mt_tool->prog, size, "");
+	while ((idx = perf_session__get_index(session)) >= 0) {
+		u64 offset = session->header.index[idx].offset;
+		u64 size = session->header.index[idx].size;
+		struct perf_tool_mt *mtt = &mt_tool[idx];
 
-	pr_debug("processing samples using thread [%d]\n", mt_tool->idx);
-	if (__perf_session__process_events(session, &mt_tool->stats,
-					   offset, size, file_size,
-					   &mt_tool->prog) < 0) {
-		pr_err("processing samples failed (thread [%d])\n", mt_tool->idx);
-		return NULL;
+		if (size == 0)
+			continue;
+
+		pr_debug("processing samples [index %d]\n", idx);
+
+		ui_progress__init(&mtt->prog, size, "");
+
+		if (__perf_session__process_events(mtt->session, &mtt->stats,
+						   offset, size, file_size,
+						   &mtt->prog) < 0) {
+			pr_err("processing samples failed [index %d]\n", idx);
+			return NULL;
+		}
+		pr_debug("processing samples done [index %d]\n", idx);
 	}
 
-	pr_debug("processing samples done for thread [%d]\n", mt_tool->idx);
 	return arg;
 }
 
@@ -1717,6 +1742,7 @@ int perf_session__process_events_mt(struct perf_session *session, void *arg)
 	int err, i, k;
 	int nr_index = session->header.nr_index;
 	u64 size = perf_data_file__size(file);
+	int nr_thread = sysconf(_SC_NPROCESSORS_ONLN);
 
 	if (perf_data_file__is_pipe(file) || !session->header.index) {
 		pr_err("data file doesn't contain the index table\n");
@@ -1753,15 +1779,18 @@ int perf_session__process_events_mt(struct perf_session *session, void *arg)
 
 	tool->ordered_events = false;
 
-	for (i = 1; i < nr_index; i++) {
+	for (i = 0; i < nr_index; i++) {
 		ms = &mt_sessions[i];
 		mt = &mt_tools[i];
 
+		ms->tool = &mt->tool;
 		ms->file = session->file;
 		ms->evlist = session->evlist;
 		ms->header = session->header;
 		ms->tevent = session->tevent;
+		ms->machines = session->machines;
 
+		ordered_events__init(&ms->ordered_events, NULL);
 		memcpy(&mt->tool, tool, sizeof(*tool));
 
 		mt->hists = calloc(evlist->nr_entries, sizeof(*mt->hists));
@@ -1772,20 +1801,28 @@ int perf_session__process_events_mt(struct perf_session *session, void *arg)
 			__hists__init(&mt->hists[k]);
 
 		mt->session = ms;
-		mt->idx = i;
 		mt->priv = arg;
 		mt->global_prog = &prog;
-
-		pthread_create(&th_id[i], NULL, processing_thread_idx, mt);
 	}
 
-	for (i = 1; i < nr_index; i++) {
+	if (nr_thread > nr_index - 1)
+		nr_thread = nr_index - 1;
+
+	th_id = calloc(nr_thread, sizeof(*th_id));
+	if (th_id == NULL)
+		goto out;
+
+	for (i = 0; i < nr_thread; i++)
+		pthread_create(&th_id[i], NULL, processing_thread_idx, mt_tools);
+
+	for (i = 0; i < nr_thread; i++) {
 		pthread_join(th_id[i], (void **)&mt);
-		if (mt == NULL) {
+		if (mt == NULL)
 			err = -EINVAL;
-			continue;
-		}
+	}
 
+	for (i = 0; i < nr_index; i++) {
+		mt = &mt_tools[i];
 		events_stats__add(&evlist->stats, &mt->stats);
 
 		evlist__for_each(evlist, evsel) {
@@ -1801,7 +1838,7 @@ int perf_session__process_events_mt(struct perf_session *session, void *arg)
 	ui_progress__ops = orig_progress__ops;
 	ui_progress__init(&prog, nr_entries, "Merging related events...");
 
-	for (i = 1; i < nr_index; i++) {
+	for (i = 0; i < nr_index; i++) {
 		mt = &mt_tools[i];
 
 		evlist__for_each(evlist, evsel) {
@@ -1829,7 +1866,7 @@ int perf_session__process_events_mt(struct perf_session *session, void *arg)
 	perf_session__warn_about_errors(session, &evlist->stats);
 
 	if (mt_tools) {
-		for (i = 1; i < nr_index; i++)
+		for (i = 0; i < nr_index; i++)
 			free(mt_tools[i].hists);
 		free(mt_tools);
 	}
@@ -1839,7 +1876,7 @@ int perf_session__process_events_mt(struct perf_session *session, void *arg)
 	return err;
 
 err:
-	while (i-- > 1) {
+	while (i-- > 0) {
 		pthread_cancel(th_id[i]);
 		pthread_join(th_id[i], NULL);
 	}
diff --git a/tools/perf/util/tool.h b/tools/perf/util/tool.h
index 144a414b1d5b..c96e805b9ff4 100644
--- a/tools/perf/util/tool.h
+++ b/tools/perf/util/tool.h
@@ -69,7 +69,6 @@ struct perf_tool_mt {
 	struct perf_session	*session;
 	struct ui_progress	prog;
 	struct ui_progress	*global_prog;
-	int			idx;
 
 	void			*priv;
 };
-- 
2.4.0


^ permalink raw reply related	[flat|nested] 221+ messages in thread

* [PATCH 39/40] perf data: Implement 'index' subcommand
  2015-05-18  0:30 [RFC/PATCHSET 00/40] perf tools: Speed-up perf report by using multi thread (v4) Namhyung Kim
                   ` (37 preceding siblings ...)
  2015-05-18  0:30 ` [PATCH 38/40] perf session: Handle index files generally Namhyung Kim
@ 2015-05-18  0:30 ` Namhyung Kim
  2015-05-18  0:30 ` [PATCH Not-for-merge 40/40] perf tools: Disable thread refcount due to bug Namhyung Kim
  39 siblings, 0 replies; 221+ messages in thread
From: Namhyung Kim @ 2015-05-18  0:30 UTC (permalink / raw)
  To: Arnaldo Carvalho de Melo
  Cc: Ingo Molnar, Peter Zijlstra, Jiri Olsa, LKML, David Ahern,
	Adrian Hunter, Andi Kleen, Frederic Weisbecker, Stephane Eranian

The index command first splits a given data file into intermediate
data files and merges them into a final data file with an index table
so that it can processed using multi threads.  The HEADER_DATA_INDEX
feature bit is added to distinguish data file that has an index table.

Signed-off-by: Namhyung Kim <namhyung@kernel.org>
---
 tools/perf/Documentation/perf-data.txt |  25 ++-
 tools/perf/builtin-data.c              | 351 ++++++++++++++++++++++++++++++++-
 2 files changed, 372 insertions(+), 4 deletions(-)

diff --git a/tools/perf/Documentation/perf-data.txt b/tools/perf/Documentation/perf-data.txt
index be8fa1a0a97e..fdac46ea6732 100644
--- a/tools/perf/Documentation/perf-data.txt
+++ b/tools/perf/Documentation/perf-data.txt
@@ -22,6 +22,11 @@ COMMANDS
 	like:
 	  perf --debug data-convert data convert ...
 
+index::
+	Build an index table for data file so that it can be processed
+	with multiple threads concurrently.
+
+
 OPTIONS for 'convert'
 ---------------------
 --to-ctf::
@@ -34,7 +39,25 @@ OPTIONS for 'convert'
 --verbose::
         Be more verbose (show counter open errors, etc).
 
+OPTIONS for 'index'
+-------------------
+-i::
+--input::
+	Specify input perf data file path.
+
+-o::
+--output::
+	Specify output perf data directory path.
+
+-v::
+--verbose::
+        Be more verbose (show counter open errors, etc).
+
+-f::
+--force::
+        Don't complain, do it.
+
 SEE ALSO
 --------
-linkperf:perf[1]
+linkperf:perf[1], linkperf:perf-report[1]
 [1] Common Trace Format - http://www.efficios.com/ctf
diff --git a/tools/perf/builtin-data.c b/tools/perf/builtin-data.c
index d6525bc54d13..71eb6db8b7ff 100644
--- a/tools/perf/builtin-data.c
+++ b/tools/perf/builtin-data.c
@@ -2,11 +2,16 @@
 #include "builtin.h"
 #include "perf.h"
 #include "debug.h"
+#include "session.h"
+#include "evlist.h"
 #include "parse-options.h"
 #include "data-convert-bt.h"
+#include <sys/mman.h>
 
 typedef int (*data_cmd_fn_t)(int argc, const char **argv, const char *prefix);
 
+static const char *output_name;
+
 struct data_cmd {
 	const char	*name;
 	const char	*summary;
@@ -44,6 +49,15 @@ static void print_usage(void)
 	printf("\n");
 }
 
+static int cmd_data_convert(int argc, const char **argv, const char *prefix);
+static int data_cmd_index(int argc, const char **argv, const char *prefix);
+
+static struct data_cmd data_cmds[] = {
+	{ "convert", "converts data file between formats", cmd_data_convert },
+	{ "index", "merge data file and add index", data_cmd_index },
+	{ .name = NULL, },
+};
+
 static const char * const data_convert_usage[] = {
 	"perf data convert [<options>]",
 	NULL
@@ -88,11 +102,342 @@ static int cmd_data_convert(int argc, const char **argv,
 	return 0;
 }
 
-static struct data_cmd data_cmds[] = {
-	{ "convert", "converts data file between formats", cmd_data_convert },
-	{ .name = NULL, },
+#define FD_HASH_BITS  7
+#define FD_HASH_SIZE  (1 << FD_HASH_BITS)
+#define FD_HASH_MASK  (FD_HASH_SIZE - 1)
+
+struct data_index {
+	struct perf_tool	tool;
+	struct perf_session	*session;
+	enum {
+		PER_CPU,
+		PER_THREAD,
+	} split_mode;
+	char			*tmpdir;
+	int 			header_fd;
+	u64			header_written;
+	struct hlist_head	fd_hash[FD_HASH_SIZE];
+	int			fd_hash_nr;
+	int			output_fd;
 };
 
+struct fdhash_node {
+	int			id;
+	int			fd;
+	struct hlist_node	list;
+};
+
+static struct hlist_head *get_hash(struct data_index *idx, int id)
+{
+	return &idx->fd_hash[id % FD_HASH_MASK];
+}
+
+static int perf_event__rewrite_header(struct perf_tool *tool,
+				      union perf_event *event)
+{
+	struct data_index *idx = container_of(tool, struct data_index, tool);
+	ssize_t size;
+
+	size = writen(idx->header_fd, event, event->header.size);
+	if (size < 0)
+		return -errno;
+
+	idx->header_written += size;
+	return 0;
+}
+
+static int split_other_events(struct perf_tool *tool,
+				union perf_event *event,
+				struct perf_sample *sample __maybe_unused,
+				struct machine *machine __maybe_unused)
+{
+	return perf_event__rewrite_header(tool, event);
+}
+
+static int split_sample_event(struct perf_tool *tool,
+				union perf_event *event,
+				struct perf_sample *sample,
+				struct perf_evsel *evsel __maybe_unused,
+				struct machine *machine __maybe_unused)
+{
+	struct data_index *idx = container_of(tool, struct data_index, tool);
+	int id = idx->split_mode == PER_CPU ? sample->cpu : sample->tid;
+	int fd = -1;
+	char buf[PATH_MAX];
+	struct hlist_head *head;
+	struct fdhash_node *node;
+
+	head = get_hash(idx, id);
+	hlist_for_each_entry(node, head, list) {
+		if (node->id == id) {
+			fd = node->fd;
+			break;
+		}
+	}
+
+	if (fd == -1) {
+		scnprintf(buf, sizeof(buf), "%s/perf.data.%d",
+			  idx->tmpdir, idx->fd_hash_nr++);
+
+		fd = open(buf, O_RDWR|O_CREAT|O_TRUNC, 0600);
+		if (fd < 0) {
+			pr_err("cannot open data file: %s: %m\n", buf);
+			return -1;
+		}
+
+		node = malloc(sizeof(*node));
+		if (node == NULL) {
+			pr_err("memory allocation failed\n");
+			return -1;
+		}
+
+		node->id = id;
+		node->fd = fd;
+
+		hlist_add_head(&node->list, head);
+	}
+
+	return writen(fd, event, event->header.size) > 0 ? 0 : -errno;
+}
+
+static int split_data_file(struct data_index *idx)
+{
+	struct perf_session *session = idx->session;
+	char buf[PATH_MAX];
+	u64 sample_type;
+	int header_fd;
+
+	if (asprintf(&idx->tmpdir, "%s.dir", output_name) < 0) {
+		pr_err("memory allocation failed\n");
+		return -1;
+	}
+
+	if (mkdir(idx->tmpdir, 0700) < 0) {
+		pr_err("cannot create intermediate directory\n");
+		return -1;
+	}
+
+	/*
+	 * This is necessary to write (copy) build-id table.  After
+	 * processing header, dsos list will only contain dso which
+	 * was on the original build-id table.
+	 */
+	dsos__hit_all(session);
+
+	scnprintf(buf, sizeof(buf), "%s/perf.header", idx->tmpdir);
+	header_fd = open(buf, O_RDWR|O_CREAT|O_TRUNC, 0600);
+	if (header_fd < 0) {
+		pr_err("cannot open header file: %s: %m\n", buf);
+		return -1;
+	}
+
+	lseek(header_fd, session->header.data_offset, SEEK_SET);
+
+	sample_type = perf_evlist__combined_sample_type(session->evlist);
+	if (sample_type & PERF_SAMPLE_CPU)
+		idx->split_mode = PER_CPU;
+	else
+		idx->split_mode = PER_THREAD;
+
+	pr_debug("splitting data file for %s\n",
+		 idx->split_mode == PER_CPU ? "CPUs" : "threads");
+
+	idx->header_fd = header_fd;
+	if (perf_session__process_events(session) < 0) {
+		pr_err("failed to process events\n");
+		return -1;
+	}
+
+	return 0;
+}
+
+static int build_index_table(struct data_index *idx)
+{
+	int i, n;
+	u64 offset;
+	u64 nr_idx = idx->fd_hash_nr + 1;
+	struct perf_file_section *sec;
+	struct perf_session *session = idx->session;
+
+	sec = calloc(nr_idx, sizeof(*sec));
+	if (sec == NULL)
+		return -1;
+
+	sec[0].offset = session->header.data_offset;
+	sec[0].size   = idx->header_written;
+
+	offset = sec[0].offset + sec[0].size;
+
+	for (i = 0, n = 1; i < FD_HASH_SIZE; i++) {
+		struct fdhash_node *node;
+
+		hlist_for_each_entry(node, &idx->fd_hash[i], list) {
+			struct stat stbuf;
+
+			if (fstat(node->fd, &stbuf) < 0)
+				goto out;
+
+			sec[n].offset = offset;
+			sec[n].size   = stbuf.st_size;
+			n++;
+
+			offset += stbuf.st_size;
+		}
+	}
+
+	BUG_ON(n != (int)nr_idx);
+
+	session->header.index = sec;
+	session->header.nr_index = nr_idx;
+
+	session->header.data_size = offset - sec[0].offset;
+	perf_header__set_feat(&session->header, HEADER_DATA_INDEX);
+
+	perf_session__write_header(session, session->evlist,
+				   idx->output_fd, true);
+	return 0;
+
+out:
+	free(sec);
+	return -1;
+}
+
+static int cleanup_temp_files(struct data_index *idx)
+{
+	int i;
+
+	for (i = 0; i < FD_HASH_SIZE; i++) {
+		struct fdhash_node *pos;
+		struct hlist_node *tmp;
+
+		hlist_for_each_entry_safe(pos, tmp, &idx->fd_hash[i], list) {
+			hlist_del(&pos->list);
+			close(pos->fd);
+			free(pos);
+		}
+	}
+	close(idx->header_fd);
+
+	rm_rf(idx->tmpdir);
+	zfree(&idx->tmpdir);
+	return 0;
+}
+
+static int __data_cmd_index(struct data_index *idx)
+{
+	struct perf_session *session = idx->session;
+	char *output = NULL;
+	int ret = -1;
+	int i, n;
+
+	if (!output_name) {
+		if (asprintf(&output, "%s.out", session->file->path) < 0) {
+			pr_err("memory allocation failed\n");
+			return -1;
+		}
+
+		output_name = output;
+	}
+
+	idx->output_fd = open(output_name, O_RDWR|O_CREAT|O_TRUNC, 0600);
+	if (idx->output_fd < 0) {
+		pr_err("cannot create output file: %s\n", output_name);
+		goto out;
+	}
+
+	/*
+	 * This is necessary to write (copy) build-id table.  After
+	 * processing header, dsos list will contain dso which was on
+	 * the original build-id table.
+	 */
+	dsos__hit_all(session);
+
+	if (split_data_file(idx) < 0)
+		goto out_clean;
+
+	if (build_index_table(idx) < 0)
+		goto out_clean;
+
+	/* copy meta-events */
+	if (copyfile_offset(idx->header_fd, session->header.data_offset,
+			   idx->output_fd, session->header.data_offset,
+			   idx->header_written) < 0)
+		goto out_clean;
+
+	/* copy sample events */
+	for (i = 0, n = 1; i < FD_HASH_SIZE; i++) {
+		struct fdhash_node *node;
+
+		hlist_for_each_entry(node, &idx->fd_hash[i], list) {
+			if (copyfile_offset(node->fd, 0, idx->output_fd,
+					    session->header.index[n].offset,
+					    session->header.index[n].size) < 0)
+				goto out_clean;
+			n++;
+		}
+	}
+	ret = 0;
+
+out_clean:
+	cleanup_temp_files(idx);
+	close(idx->output_fd);
+out:
+	free(output);
+	return ret;
+}
+
+int data_cmd_index(int argc, const char **argv, const char *prefix __maybe_unused)
+{
+	bool force = false;
+	struct perf_session *session;
+	struct perf_data_file file = {
+		.mode  = PERF_DATA_MODE_READ,
+	};
+	struct data_index idx = {
+		.tool = {
+			.sample		= split_sample_event,
+			.fork		= split_other_events,
+			.comm		= split_other_events,
+			.exit		= split_other_events,
+			.mmap		= split_other_events,
+			.mmap2		= split_other_events,
+			.lost		= split_other_events,
+			.throttle	= split_other_events,
+			.unthrottle	= split_other_events,
+			.ordered_events = false,
+		},
+	};
+	const char * const index_usage[] = {
+		"perf data index [<options>]",
+		NULL
+	};
+	const struct option index_options[] = {
+	OPT_STRING('i', "input", &input_name, "file", "input file name"),
+	OPT_STRING('o', "output", &output_name, "file", "output directory name"),
+	OPT_BOOLEAN('f', "force", &force, "don't complain, do it"),
+	OPT_INCR('v', "verbose", &verbose, "be more verbose"),
+	OPT_END()
+	};
+
+	argc = parse_options(argc, argv, index_options, index_usage, 0);
+	if (argc)
+		usage_with_options(index_usage, index_options);
+
+	file.path = input_name;
+	file.force = force;
+	session = perf_session__new(&file, false, &idx.tool);
+	if (session == NULL)
+		return -1;
+
+	idx.session = session;
+	symbol__init(&session->header.env);
+
+	__data_cmd_index(&idx);
+
+	perf_session__delete(session);
+	return 0;
+}
+
 int cmd_data(int argc, const char **argv, const char *prefix)
 {
 	struct data_cmd *cmd;
-- 
2.4.0


^ permalink raw reply related	[flat|nested] 221+ messages in thread

* [PATCH Not-for-merge 40/40] perf tools: Disable thread refcount due to bug
  2015-05-18  0:30 [RFC/PATCHSET 00/40] perf tools: Speed-up perf report by using multi thread (v4) Namhyung Kim
                   ` (38 preceding siblings ...)
  2015-05-18  0:30 ` [PATCH 39/40] perf data: Implement 'index' subcommand Namhyung Kim
@ 2015-05-18  0:30 ` Namhyung Kim
  2015-05-18  1:23   ` Arnaldo Carvalho de Melo
  39 siblings, 1 reply; 221+ messages in thread
From: Namhyung Kim @ 2015-05-18  0:30 UTC (permalink / raw)
  To: Arnaldo Carvalho de Melo
  Cc: Ingo Molnar, Peter Zijlstra, Jiri Olsa, LKML, David Ahern,
	Adrian Hunter, Andi Kleen, Frederic Weisbecker, Stephane Eranian

This makes thread mg sharing test failed due to not decrement
thread->refcnt on thread__put().

Not-signed-off-by: Namhyung Kim <namhyung@kernel.org>
---
 tools/perf/util/thread.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/tools/perf/util/thread.c b/tools/perf/util/thread.c
index 702f12dc5a90..dc5ec9a5cca1 100644
--- a/tools/perf/util/thread.c
+++ b/tools/perf/util/thread.c
@@ -163,7 +163,7 @@ struct thread *thread__get(struct thread *thread)
 
 void thread__put(struct thread *thread)
 {
-	if (thread && atomic_dec_and_test(&thread->refcnt)) {
+	if (thread && atomic_dec_and_test(&thread->refcnt) && 0) {
 		if (!RB_EMPTY_NODE(&thread->rb_node)) {
 			struct machine *machine = thread->mg->machine;
 
-- 
2.4.0


^ permalink raw reply related	[flat|nested] 221+ messages in thread

* Re: [PATCH Not-for-merge 40/40] perf tools: Disable thread refcount due to bug
  2015-05-18  0:30 ` [PATCH Not-for-merge 40/40] perf tools: Disable thread refcount due to bug Namhyung Kim
@ 2015-05-18  1:23   ` Arnaldo Carvalho de Melo
  2015-05-18 12:21     ` Namhyung Kim
  0 siblings, 1 reply; 221+ messages in thread
From: Arnaldo Carvalho de Melo @ 2015-05-18  1:23 UTC (permalink / raw)
  To: Namhyung Kim
  Cc: Ingo Molnar, Peter Zijlstra, Jiri Olsa, LKML, David Ahern,
	Adrian Hunter, Andi Kleen, Frederic Weisbecker, Stephane Eranian

Em Mon, May 18, 2015 at 09:30:55AM +0900, Namhyung Kim escreveu:
> This makes thread mg sharing test failed due to not decrement
> thread->refcnt on thread__put().

I fixed this one already:

https://git.kernel.org/cgit/linux/kernel/git/acme/linux.git/commit/?h=perf/core&id=8b00f46951bed1edd9c5cb9d9adb62d28bbe7623

No?

- Arnaldo

 
> Not-signed-off-by: Namhyung Kim <namhyung@kernel.org>
> ---
>  tools/perf/util/thread.c | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
> 
> diff --git a/tools/perf/util/thread.c b/tools/perf/util/thread.c
> index 702f12dc5a90..dc5ec9a5cca1 100644
> --- a/tools/perf/util/thread.c
> +++ b/tools/perf/util/thread.c
> @@ -163,7 +163,7 @@ struct thread *thread__get(struct thread *thread)
>  
>  void thread__put(struct thread *thread)
>  {
> -	if (thread && atomic_dec_and_test(&thread->refcnt)) {
> +	if (thread && atomic_dec_and_test(&thread->refcnt) && 0) {
>  		if (!RB_EMPTY_NODE(&thread->rb_node)) {
>  			struct machine *machine = thread->mg->machine;
>  
> -- 
> 2.4.0

^ permalink raw reply	[flat|nested] 221+ messages in thread

* Re: [PATCH Not-for-merge 40/40] perf tools: Disable thread refcount due to bug
  2015-05-18  1:23   ` Arnaldo Carvalho de Melo
@ 2015-05-18 12:21     ` Namhyung Kim
  2015-05-18 12:30       ` Arnaldo Carvalho de Melo
  0 siblings, 1 reply; 221+ messages in thread
From: Namhyung Kim @ 2015-05-18 12:21 UTC (permalink / raw)
  To: Arnaldo Carvalho de Melo
  Cc: Ingo Molnar, Peter Zijlstra, Jiri Olsa, LKML, David Ahern,
	Adrian Hunter, Andi Kleen, Frederic Weisbecker, Stephane Eranian

On Sun, May 17, 2015 at 10:23:36PM -0300, Arnaldo Carvalho de Melo wrote:
> Em Mon, May 18, 2015 at 09:30:55AM +0900, Namhyung Kim escreveu:
> > This makes thread mg sharing test failed due to not decrement
> > thread->refcnt on thread__put().
> 
> I fixed this one already:
> 
> https://git.kernel.org/cgit/linux/kernel/git/acme/linux.git/commit/?h=perf/core&id=8b00f46951bed1edd9c5cb9d9adb62d28bbe7623
> 
> No?

I meant that I added this patch not to segfault caused by the thread
refcount, but as a side-effect it makes thread_mg_sharing test failed
due to non-decremented refcount.

As of 70923bd26c73 ("perf tools: Make flex/bison calls honour V=1")
I can reproduce the segfault.

Thanks,
Namhyung


>  
> > Not-signed-off-by: Namhyung Kim <namhyung@kernel.org>
> > ---
> >  tools/perf/util/thread.c | 2 +-
> >  1 file changed, 1 insertion(+), 1 deletion(-)
> > 
> > diff --git a/tools/perf/util/thread.c b/tools/perf/util/thread.c
> > index 702f12dc5a90..dc5ec9a5cca1 100644
> > --- a/tools/perf/util/thread.c
> > +++ b/tools/perf/util/thread.c
> > @@ -163,7 +163,7 @@ struct thread *thread__get(struct thread *thread)
> >  
> >  void thread__put(struct thread *thread)
> >  {
> > -	if (thread && atomic_dec_and_test(&thread->refcnt)) {
> > +	if (thread && atomic_dec_and_test(&thread->refcnt) && 0) {
> >  		if (!RB_EMPTY_NODE(&thread->rb_node)) {
> >  			struct machine *machine = thread->mg->machine;
> >  
> > -- 
> > 2.4.0

^ permalink raw reply	[flat|nested] 221+ messages in thread

* Re: [PATCH Not-for-merge 40/40] perf tools: Disable thread refcount due to bug
  2015-05-18 12:21     ` Namhyung Kim
@ 2015-05-18 12:30       ` Arnaldo Carvalho de Melo
  0 siblings, 0 replies; 221+ messages in thread
From: Arnaldo Carvalho de Melo @ 2015-05-18 12:30 UTC (permalink / raw)
  To: Namhyung Kim
  Cc: Ingo Molnar, Peter Zijlstra, Jiri Olsa, LKML, David Ahern,
	Adrian Hunter, Andi Kleen, Frederic Weisbecker, Stephane Eranian

Em Mon, May 18, 2015 at 09:21:31PM +0900, Namhyung Kim escreveu:
> On Sun, May 17, 2015 at 10:23:36PM -0300, Arnaldo Carvalho de Melo wrote:
> > Em Mon, May 18, 2015 at 09:30:55AM +0900, Namhyung Kim escreveu:
> > > This makes thread mg sharing test failed due to not decrement
> > > thread->refcnt on thread__put().

> > I fixed this one already:

> > https://git.kernel.org/cgit/linux/kernel/git/acme/linux.git/commit/?h=perf/core&id=8b00f46951bed1edd9c5cb9d9adb62d28bbe7623

> > No?
 
> I meant that I added this patch not to segfault caused by the thread
> refcount, but as a side-effect it makes thread_mg_sharing test failed
> due to non-decremented refcount.
 
> As of 70923bd26c73 ("perf tools: Make flex/bison calls honour V=1")
> I can reproduce the segfault.

Ok, I'll investigate that report, looks like one more case where
longstanding bugs are exposed by work on multithreading.

- Arnaldo

^ permalink raw reply	[flat|nested] 221+ messages in thread

* Re: [PATCH 35/40] perf record: Synthesize COMM event for a command line workload
  2015-05-18  0:30 ` [PATCH 35/40] perf record: Synthesize COMM event for a command line workload Namhyung Kim
@ 2015-05-18 12:45   ` Arnaldo Carvalho de Melo
  2015-05-19  7:46     ` Namhyung Kim
  0 siblings, 1 reply; 221+ messages in thread
From: Arnaldo Carvalho de Melo @ 2015-05-18 12:45 UTC (permalink / raw)
  To: Namhyung Kim
  Cc: Ingo Molnar, Peter Zijlstra, Jiri Olsa, LKML, David Ahern,
	Adrian Hunter, Andi Kleen, Frederic Weisbecker, Stephane Eranian

Em Mon, May 18, 2015 at 09:30:50AM +0900, Namhyung Kim escreveu:
> When perf creates a new child to profile, the events are enabled on
> exec().  And in this case, it doesn't synthesize any event for the
> child since they'll be generated during exec().  But there's an window
> between the enabling and the event generation.
> 
> It used to be overcome since samples are only in kernel (so we always
> have the map) and the comm is overridden by a later COMM event.
> However it won't work anymore since those samples will go to a missing
> thread now but the COMM event will create a (current) thread.  This
> leads to those early samples (like native_write_msr_safe) not having a
> comm but pid (like ':15328').
 
> So it needs to synthesize COMM event for the child explicitly before
> enabling so that it can have a correct comm.  But at this time, the
> comm will be "perf" since it's not exec-ed yet.

This looks reasonable, but I think it probably needs to be done
somewhere in perf_evlist__prepare_workload() or
perf_evlist__start_workload(), as this affects other tools as well, like
'top', 'trace' and any other that may want to do this start-workload use
case.

I also wonder if we can't overcome this without using /proc, i.e.
actually moving the "start the workload" to just before the fork, so
that the kernel covers that as well.

Or, alternatively, the thread can be created without having to look at
/proc at all, but by directly creating the struct thread, with the
correct COMM, pid, etc, that we know, since we forked it, etc.

- Arnaldo
 
> Signed-off-by: Namhyung Kim <namhyung@kernel.org>
> ---
>  tools/perf/builtin-record.c | 18 +++++++++++++++++-
>  tools/perf/util/event.c     |  2 +-
>  tools/perf/util/event.h     |  5 +++++
>  3 files changed, 23 insertions(+), 2 deletions(-)
> 
> diff --git a/tools/perf/builtin-record.c b/tools/perf/builtin-record.c
> index 978ebf648aab..153f38e35555 100644
> --- a/tools/perf/builtin-record.c
> +++ b/tools/perf/builtin-record.c
> @@ -766,8 +766,24 @@ static int __cmd_record(struct record *rec, int argc, const char **argv)
>  	/*
>  	 * Let the child rip
>  	 */
> -	if (forks)
> +	if (forks) {
> +		union perf_event *comm_event;
> +
> +		comm_event = malloc(sizeof(*comm_event) + machine->id_hdr_size);
> +		if (comm_event == NULL)
> +			goto out_child;
> +
> +		err = perf_event__synthesize_comm(tool, comm_event,
> +						  rec->evlist->workload.pid,
> +						  process_synthesized_event,
> +						  machine);
> +		free(comm_event);
> +
> +		if (err < 0)
> +			goto out_child;
> +
>  		perf_evlist__start_workload(rec->evlist);
> +	}
>  
>  	if (opts->initial_delay) {
>  		usleep(opts->initial_delay * 1000);
> diff --git a/tools/perf/util/event.c b/tools/perf/util/event.c
> index 930d45d5a37a..3adca1302150 100644
> --- a/tools/perf/util/event.c
> +++ b/tools/perf/util/event.c
> @@ -165,7 +165,7 @@ static int perf_event__prepare_comm(union perf_event *event, pid_t pid,
>  	return 0;
>  }
>  
> -static pid_t perf_event__synthesize_comm(struct perf_tool *tool,
> +pid_t perf_event__synthesize_comm(struct perf_tool *tool,
>  					 union perf_event *event, pid_t pid,
>  					 perf_event__handler_t process,
>  					 struct machine *machine)
> diff --git a/tools/perf/util/event.h b/tools/perf/util/event.h
> index 40e02544f861..9aacd558ac0f 100644
> --- a/tools/perf/util/event.h
> +++ b/tools/perf/util/event.h
> @@ -452,6 +452,11 @@ int perf_event__synthesize_mmap_events(struct perf_tool *tool,
>  				       struct machine *machine,
>  				       bool mmap_data);
>  
> +pid_t perf_event__synthesize_comm(struct perf_tool *tool,
> +				  union perf_event *event, pid_t pid,
> +				  perf_event__handler_t process,
> +				  struct machine *machine);
> +
>  size_t perf_event__fprintf_comm(union perf_event *event, FILE *fp);
>  size_t perf_event__fprintf_mmap(union perf_event *event, FILE *fp);
>  size_t perf_event__fprintf_mmap2(union perf_event *event, FILE *fp);
> -- 
> 2.4.0

^ permalink raw reply	[flat|nested] 221+ messages in thread

* Re: [PATCH 33/40] perf session: Separate struct machines from session
  2015-05-18  0:30 ` [PATCH 33/40] perf session: Separate struct machines from session Namhyung Kim
@ 2015-05-18 12:52   ` Arnaldo Carvalho de Melo
  2015-05-19  7:28     ` Namhyung Kim
  0 siblings, 1 reply; 221+ messages in thread
From: Arnaldo Carvalho de Melo @ 2015-05-18 12:52 UTC (permalink / raw)
  To: Namhyung Kim
  Cc: Ingo Molnar, Peter Zijlstra, Jiri Olsa, LKML, David Ahern,
	Adrian Hunter, Andi Kleen, Frederic Weisbecker, Stephane Eranian

Em Mon, May 18, 2015 at 09:30:48AM +0900, Namhyung Kim escreveu:
> With multi-thread report, separate sessions can be passed to each
> thread, in this case we should keep a single machine state for all
> struct sessions.  Separate machines and have a pointer in sessions.

I had to look at all the patch to semi-figure this out, i.e. you said it
should be separated from 'perf_session', agreed.

But who will create it?  How will it be passed to the perf_session
instances?

Most of the patch is making session->machines be turned into a pointer,
but the meat, i.e. who creates it, is unclear, I see a malloc in
perf_session__new(), where I was kinda expecting that a higer layer,
perhaps in struct tool? Would create the list of all machines (struct
machines) and then pass it to multiple perf_session__new() calls.

But then perf_session__delete() calls 'free(session->machines)', huh?

- Arnaldo
 
> Signed-off-by: Namhyung Kim <namhyung@kernel.org>
> ---
>  tools/perf/builtin-annotate.c |  2 +-
>  tools/perf/builtin-kmem.c     | 10 +++++-----
>  tools/perf/builtin-kvm.c      |  2 +-
>  tools/perf/builtin-record.c   |  4 ++--
>  tools/perf/builtin-report.c   |  4 ++--
>  tools/perf/builtin-top.c      |  8 ++++----
>  tools/perf/builtin-trace.c    |  2 +-
>  tools/perf/util/build-id.c    | 16 ++++++++--------
>  tools/perf/util/session.c     | 36 +++++++++++++++++++++---------------
>  tools/perf/util/session.h     |  6 +++---
>  10 files changed, 48 insertions(+), 42 deletions(-)
> 
> diff --git a/tools/perf/builtin-annotate.c b/tools/perf/builtin-annotate.c
> index 761f902473b7..d4ad323ddfe2 100644
> --- a/tools/perf/builtin-annotate.c
> +++ b/tools/perf/builtin-annotate.c
> @@ -196,7 +196,7 @@ static int __cmd_annotate(struct perf_annotate *ann)
>  	struct perf_evsel *pos;
>  	u64 total_nr_samples;
>  
> -	machines__set_symbol_filter(&session->machines, symbol__annotate_init);
> +	machines__set_symbol_filter(session->machines, symbol__annotate_init);
>  
>  	if (ann->cpu_list) {
>  		ret = perf_session__cpu_bitmap(session, ann->cpu_list,
> diff --git a/tools/perf/builtin-kmem.c b/tools/perf/builtin-kmem.c
> index 254614b10c4a..4502954094e5 100644
> --- a/tools/perf/builtin-kmem.c
> +++ b/tools/perf/builtin-kmem.c
> @@ -316,7 +316,7 @@ static int build_alloc_func_list(void)
>  	struct symbol *sym;
>  	struct rb_node *node;
>  	struct alloc_func *func;
> -	struct machine *machine = &kmem_session->machines.host;
> +	struct machine *machine = &kmem_session->machines->host;
>  	regex_t alloc_func_regex;
>  	const char pattern[] = "^_?_?(alloc|get_free|get_zeroed)_pages?";
>  
> @@ -366,7 +366,7 @@ static int build_alloc_func_list(void)
>  static u64 find_callsite(struct perf_evsel *evsel, struct perf_sample *sample)
>  {
>  	struct addr_location al;
> -	struct machine *machine = &kmem_session->machines.host;
> +	struct machine *machine = &kmem_session->machines->host;
>  	struct callchain_cursor_node *node;
>  
>  	if (alloc_func_list == NULL) {
> @@ -949,7 +949,7 @@ static void __print_slab_result(struct rb_root *root,
>  				int n_lines, int is_caller)
>  {
>  	struct rb_node *next;
> -	struct machine *machine = &session->machines.host;
> +	struct machine *machine = &session->machines->host;
>  
>  	printf("%.105s\n", graph_dotted_line);
>  	printf(" %-34s |",  is_caller ? "Callsite": "Alloc Ptr");
> @@ -1010,7 +1010,7 @@ static const char * const migrate_type_str[] = {
>  static void __print_page_alloc_result(struct perf_session *session, int n_lines)
>  {
>  	struct rb_node *next = rb_first(&page_alloc_sorted);
> -	struct machine *machine = &session->machines.host;
> +	struct machine *machine = &session->machines->host;
>  	const char *format;
>  	int gfp_len = max(strlen("GFP flags"), max_gfp_len);
>  
> @@ -1060,7 +1060,7 @@ static void __print_page_alloc_result(struct perf_session *session, int n_lines)
>  static void __print_page_caller_result(struct perf_session *session, int n_lines)
>  {
>  	struct rb_node *next = rb_first(&page_caller_sorted);
> -	struct machine *machine = &session->machines.host;
> +	struct machine *machine = &session->machines->host;
>  	int gfp_len = max(strlen("GFP flags"), max_gfp_len);
>  
>  	printf("\n%.105s\n", graph_dotted_line);
> diff --git a/tools/perf/builtin-kvm.c b/tools/perf/builtin-kvm.c
> index 15fecd3dc5d8..de3c7e1d0b80 100644
> --- a/tools/perf/builtin-kvm.c
> +++ b/tools/perf/builtin-kvm.c
> @@ -1392,7 +1392,7 @@ static int kvm_events_live(struct perf_kvm_stat *kvm,
>  	kvm->session->evlist = kvm->evlist;
>  	perf_session__set_id_hdr_size(kvm->session);
>  	ordered_events__set_copy_on_queue(&kvm->session->ordered_events, true);
> -	machine__synthesize_threads(&kvm->session->machines.host, &kvm->opts.target,
> +	machine__synthesize_threads(&kvm->session->machines->host, &kvm->opts.target,
>  				    kvm->evlist->threads, false);
>  	err = kvm_live_open_events(kvm);
>  	if (err)
> diff --git a/tools/perf/builtin-record.c b/tools/perf/builtin-record.c
> index 4ddf104f50ff..978ebf648aab 100644
> --- a/tools/perf/builtin-record.c
> +++ b/tools/perf/builtin-record.c
> @@ -684,7 +684,7 @@ static int __cmd_record(struct record *rec, int argc, const char **argv)
>  		goto out_child;
>  	}
>  
> -	machine = &session->machines.host;
> +	machine = &session->machines->host;
>  
>  	if (file->is_pipe) {
>  		err = perf_event__synthesize_attrs(tool, session,
> @@ -735,7 +735,7 @@ static int __cmd_record(struct record *rec, int argc, const char **argv)
>  		       "Check /proc/modules permission or run as root.\n");
>  
>  	if (perf_guest) {
> -		machines__process_guests(&session->machines,
> +		machines__process_guests(session->machines,
>  					 perf_event__synthesize_guest_os, tool);
>  	}
>  
> diff --git a/tools/perf/builtin-report.c b/tools/perf/builtin-report.c
> index 5e53eee5a9a7..650d78ad3357 100644
> --- a/tools/perf/builtin-report.c
> +++ b/tools/perf/builtin-report.c
> @@ -353,7 +353,7 @@ static int perf_evlist__tty_browse_hists(struct perf_evlist *evlist,
>  
>  static void report__warn_kptr_restrict(const struct report *rep)
>  {
> -	struct map *kernel_map = rep->session->machines.host.vmlinux_maps[MAP__FUNCTION];
> +	struct map *kernel_map = rep->session->machines->host.vmlinux_maps[MAP__FUNCTION];
>  	struct kmap *kernel_kmap = kernel_map ? map__kmap(kernel_map) : NULL;
>  
>  	if (kernel_map == NULL ||
> @@ -845,7 +845,7 @@ int cmd_report(int argc, const char **argv, const char *prefix __maybe_unused)
>  	 */
>  	if (ui__has_annotation()) {
>  		symbol_conf.priv_size = sizeof(struct annotation);
> -		machines__set_symbol_filter(&session->machines,
> +		machines__set_symbol_filter(session->machines,
>  					    symbol__annotate_init);
>  		/*
>   		 * For searching by name on the "Browse map details".
> diff --git a/tools/perf/builtin-top.c b/tools/perf/builtin-top.c
> index ea6e7bd04f9a..0ab663e200ed 100644
> --- a/tools/perf/builtin-top.c
> +++ b/tools/perf/builtin-top.c
> @@ -828,13 +828,13 @@ static void perf_top__mmap_read_idx(struct perf_top *top, int idx)
>  			++top->us_samples;
>  			if (top->hide_user_symbols)
>  				goto next_event;
> -			machine = &session->machines.host;
> +			machine = &session->machines->host;
>  			break;
>  		case PERF_RECORD_MISC_KERNEL:
>  			++top->kernel_samples;
>  			if (top->hide_kernel_symbols)
>  				goto next_event;
> -			machine = &session->machines.host;
> +			machine = &session->machines->host;
>  			break;
>  		case PERF_RECORD_MISC_GUEST_KERNEL:
>  			++top->guest_kernel_samples;
> @@ -939,7 +939,7 @@ static int __cmd_top(struct perf_top *top)
>  	if (top->session == NULL)
>  		return -1;
>  
> -	machines__set_symbol_filter(&top->session->machines, symbol_filter);
> +	machines__set_symbol_filter(top->session->machines, symbol_filter);
>  
>  	if (!objdump_path) {
>  		ret = perf_session_env__lookup_objdump(&top->session->header.env);
> @@ -951,7 +951,7 @@ static int __cmd_top(struct perf_top *top)
>  	if (ret)
>  		goto out_delete;
>  
> -	machine__synthesize_threads(&top->session->machines.host, &opts->target,
> +	machine__synthesize_threads(&top->session->machines->host, &opts->target,
>  				    top->evlist->threads, false);
>  	ret = perf_top__start_counters(top);
>  	if (ret)
> diff --git a/tools/perf/builtin-trace.c b/tools/perf/builtin-trace.c
> index a05490d06374..f3df6fa596e9 100644
> --- a/tools/perf/builtin-trace.c
> +++ b/tools/perf/builtin-trace.c
> @@ -2418,7 +2418,7 @@ static int trace__replay(struct trace *trace)
>  	if (symbol__init(&session->header.env) < 0)
>  		goto out;
>  
> -	trace->host = &session->machines.host;
> +	trace->host = &session->machines->host;
>  
>  	err = perf_session__set_tracepoints_handlers(session, handlers);
>  	if (err)
> diff --git a/tools/perf/util/build-id.c b/tools/perf/util/build-id.c
> index ad8cfcbaa25d..8720f71a96dd 100644
> --- a/tools/perf/util/build-id.c
> +++ b/tools/perf/util/build-id.c
> @@ -221,12 +221,12 @@ static int machine__write_buildid_table(struct machine *machine, int fd)
>  int perf_session__write_buildid_table(struct perf_session *session, int fd)
>  {
>  	struct rb_node *nd;
> -	int err = machine__write_buildid_table(&session->machines.host, fd);
> +	int err = machine__write_buildid_table(&session->machines->host, fd);
>  
>  	if (err)
>  		return err;
>  
> -	for (nd = rb_first(&session->machines.guests); nd; nd = rb_next(nd)) {
> +	for (nd = rb_first(&session->machines->guests); nd; nd = rb_next(nd)) {
>  		struct machine *pos = rb_entry(nd, struct machine, rb_node);
>  		err = machine__write_buildid_table(pos, fd);
>  		if (err)
> @@ -261,11 +261,11 @@ int dsos__hit_all(struct perf_session *session)
>  	struct rb_node *nd;
>  	int err;
>  
> -	err = machine__hit_all_dsos(&session->machines.host);
> +	err = machine__hit_all_dsos(&session->machines->host);
>  	if (err)
>  		return err;
>  
> -	for (nd = rb_first(&session->machines.guests); nd; nd = rb_next(nd)) {
> +	for (nd = rb_first(&session->machines->guests); nd; nd = rb_next(nd)) {
>  		struct machine *pos = rb_entry(nd, struct machine, rb_node);
>  
>  		err = machine__hit_all_dsos(pos);
> @@ -509,9 +509,9 @@ int perf_session__cache_build_ids(struct perf_session *session)
>  	if (mkdir(buildid_dir, 0755) != 0 && errno != EEXIST)
>  		return -1;
>  
> -	ret = machine__cache_build_ids(&session->machines.host);
> +	ret = machine__cache_build_ids(&session->machines->host);
>  
> -	for (nd = rb_first(&session->machines.guests); nd; nd = rb_next(nd)) {
> +	for (nd = rb_first(&session->machines->guests); nd; nd = rb_next(nd)) {
>  		struct machine *pos = rb_entry(nd, struct machine, rb_node);
>  		ret |= machine__cache_build_ids(pos);
>  	}
> @@ -530,9 +530,9 @@ static bool machine__read_build_ids(struct machine *machine, bool with_hits)
>  bool perf_session__read_build_ids(struct perf_session *session, bool with_hits)
>  {
>  	struct rb_node *nd;
> -	bool ret = machine__read_build_ids(&session->machines.host, with_hits);
> +	bool ret = machine__read_build_ids(&session->machines->host, with_hits);
>  
> -	for (nd = rb_first(&session->machines.guests); nd; nd = rb_next(nd)) {
> +	for (nd = rb_first(&session->machines->guests); nd; nd = rb_next(nd)) {
>  		struct machine *pos = rb_entry(nd, struct machine, rb_node);
>  		ret |= machine__read_build_ids(pos, with_hits);
>  	}
> diff --git a/tools/perf/util/session.c b/tools/perf/util/session.c
> index 0d080a95d2ff..955bf5368751 100644
> --- a/tools/perf/util/session.c
> +++ b/tools/perf/util/session.c
> @@ -58,21 +58,21 @@ void perf_session__set_id_hdr_size(struct perf_session *session)
>  {
>  	u16 id_hdr_size = perf_evlist__id_hdr_size(session->evlist);
>  
> -	machines__set_id_hdr_size(&session->machines, id_hdr_size);
> +	machines__set_id_hdr_size(session->machines, id_hdr_size);
>  }
>  
>  int perf_session__create_kernel_maps(struct perf_session *session)
>  {
> -	int ret = machine__create_kernel_maps(&session->machines.host);
> +	int ret = machine__create_kernel_maps(&session->machines->host);
>  
>  	if (ret >= 0)
> -		ret = machines__create_guest_kernel_maps(&session->machines);
> +		ret = machines__create_guest_kernel_maps(session->machines);
>  	return ret;
>  }
>  
>  static void perf_session__destroy_kernel_maps(struct perf_session *session)
>  {
> -	machines__destroy_kernel_maps(&session->machines);
> +	machines__destroy_kernel_maps(session->machines);
>  }
>  
>  static bool perf_session__has_comm_exec(struct perf_session *session)
> @@ -91,7 +91,7 @@ static void perf_session__set_comm_exec(struct perf_session *session)
>  {
>  	bool comm_exec = perf_session__has_comm_exec(session);
>  
> -	machines__set_comm_exec(&session->machines, comm_exec);
> +	machines__set_comm_exec(session->machines, comm_exec);
>  }
>  
>  static int ordered_events__deliver_event(struct ordered_events *oe,
> @@ -123,7 +123,12 @@ struct perf_session *perf_session__new(struct perf_data_file *file,
>  	session->repipe = repipe;
>  	session->tool   = tool;
>  	INIT_LIST_HEAD(&session->auxtrace_index);
> -	machines__init(&session->machines);
> +
> +	session->machines = malloc(sizeof(*session->machines));
> +	if (!session->machines)
> +		goto out_delete;
> +
> +	machines__init(session->machines);
>  	ordered_events__init(&session->ordered_events, ordered_events__deliver_event);
>  
>  	if (file) {
> @@ -168,7 +173,7 @@ struct perf_session *perf_session__new(struct perf_data_file *file,
>  
>  static void perf_session__delete_threads(struct perf_session *session)
>  {
> -	machine__delete_threads(&session->machines.host);
> +	machine__delete_threads(&session->machines->host);
>  }
>  
>  static void perf_session_env__delete(struct perf_session_env *env)
> @@ -194,7 +199,8 @@ void perf_session__delete(struct perf_session *session)
>  	perf_session__destroy_kernel_maps(session);
>  	perf_session__delete_threads(session);
>  	perf_session_env__delete(&session->header.env);
> -	machines__exit(&session->machines);
> +	machines__exit(session->machines);
> +	free(session->machines);
>  	if (session->file)
>  		perf_data_file__close(session->file);
>  	free(session->header.index);
> @@ -1088,7 +1094,7 @@ static int perf_session__deliver_event(struct perf_session *session,
>  	if (ret > 0)
>  		return 0;
>  
> -	return machines__deliver_event(&session->machines, stats,
> +	return machines__deliver_event(session->machines, stats,
>  				       session->evlist, event, sample,
>  				       tool, file_offset);
>  }
> @@ -1155,7 +1161,7 @@ int perf_session__deliver_synth_event(struct perf_session *session,
>  	if (event->header.type >= PERF_RECORD_USER_TYPE_START)
>  		return perf_session__process_user_event(session, event, 0);
>  
> -	return machines__deliver_event(&session->machines, &evlist->stats,
> +	return machines__deliver_event(session->machines, &evlist->stats,
>  				       evlist, event, sample, tool, 0);
>  }
>  
> @@ -1270,14 +1276,14 @@ void perf_event_header__bswap(struct perf_event_header *hdr)
>  
>  struct thread *perf_session__findnew(struct perf_session *session, pid_t pid)
>  {
> -	return machine__findnew_thread(&session->machines.host, -1, pid);
> +	return machine__findnew_thread(&session->machines->host, -1, pid);
>  }
>  
>  static struct thread *perf_session__register_idle_thread(struct perf_session *session)
>  {
>  	struct thread *thread;
>  
> -	thread = machine__findnew_thread(&session->machines.host, 0, 0);
> +	thread = machine__findnew_thread(&session->machines->host, 0, 0);
>  	if (thread == NULL || thread__set_comm(thread, "swapper", 0)) {
>  		pr_err("problem inserting idle task.\n");
>  		thread = NULL;
> @@ -1692,13 +1698,13 @@ int maps__set_kallsyms_ref_reloc_sym(struct map **maps,
>  
>  size_t perf_session__fprintf_dsos(struct perf_session *session, FILE *fp)
>  {
> -	return machines__fprintf_dsos(&session->machines, fp);
> +	return machines__fprintf_dsos(session->machines, fp);
>  }
>  
>  size_t perf_session__fprintf_dsos_buildid(struct perf_session *session, FILE *fp,
>  					  bool (skip)(struct dso *dso, int parm), int parm)
>  {
> -	return machines__fprintf_dsos_buildid(&session->machines, fp, skip, parm);
> +	return machines__fprintf_dsos_buildid(session->machines, fp, skip, parm);
>  }
>  
>  size_t perf_session__fprintf_nr_events(struct perf_session *session, FILE *fp)
> @@ -1721,7 +1727,7 @@ size_t perf_session__fprintf(struct perf_session *session, FILE *fp)
>  	 * FIXME: Here we have to actually print all the machines in this
>  	 * session, not just the host...
>  	 */
> -	return machine__fprintf(&session->machines.host, fp);
> +	return machine__fprintf(&session->machines->host, fp);
>  }
>  
>  struct perf_evsel *perf_session__find_first_evtype(struct perf_session *session,
> diff --git a/tools/perf/util/session.h b/tools/perf/util/session.h
> index b44afc75d1cc..ac1796c7c799 100644
> --- a/tools/perf/util/session.h
> +++ b/tools/perf/util/session.h
> @@ -20,7 +20,7 @@ struct itrace_synth_opts;
>  
>  struct perf_session {
>  	struct perf_header	header;
> -	struct machines		machines;
> +	struct machines		*machines;
>  	struct perf_evlist	*evlist;
>  	struct auxtrace		*auxtrace;
>  	struct itrace_synth_opts *itrace_synth_opts;
> @@ -79,13 +79,13 @@ void perf_session__set_id_hdr_size(struct perf_session *session);
>  static inline
>  struct machine *perf_session__find_machine(struct perf_session *session, pid_t pid)
>  {
> -	return machines__find(&session->machines, pid);
> +	return machines__find(session->machines, pid);
>  }
>  
>  static inline
>  struct machine *perf_session__findnew_machine(struct perf_session *session, pid_t pid)
>  {
> -	return machines__findnew(&session->machines, pid);
> +	return machines__findnew(session->machines, pid);
>  }
>  
>  struct thread *perf_session__findnew(struct perf_session *session, pid_t pid);
> -- 
> 2.4.0

^ permalink raw reply	[flat|nested] 221+ messages in thread

* Re: [PATCH 16/40] perf tools: Reducing arguments of hist_entry_iter__add()
  2015-05-18  0:30 ` [PATCH 16/40] perf tools: Reducing arguments of hist_entry_iter__add() Namhyung Kim
@ 2015-05-18 12:55   ` Arnaldo Carvalho de Melo
  2015-05-19  7:01     ` Namhyung Kim
  0 siblings, 1 reply; 221+ messages in thread
From: Arnaldo Carvalho de Melo @ 2015-05-18 12:55 UTC (permalink / raw)
  To: Namhyung Kim
  Cc: Ingo Molnar, Peter Zijlstra, Jiri Olsa, LKML, David Ahern,
	Adrian Hunter, Andi Kleen, Frederic Weisbecker, Stephane Eranian

Em Mon, May 18, 2015 at 09:30:31AM +0900, Namhyung Kim escreveu:
> The evsel and sample arguments are to set iter for later use.  As it
> also receives an iter as another argument, just set them before
> calling the function.

This looks like can be applied sooner, tried but it didn't apply :-\

- Arnaldo
 
> Signed-off-by: Namhyung Kim <namhyung@kernel.org>
> ---
>  tools/perf/builtin-report.c       | 9 +++++----
>  tools/perf/builtin-top.c          | 7 ++++---
>  tools/perf/tests/hists_cumulate.c | 6 ++++--
>  tools/perf/tests/hists_filter.c   | 4 +++-
>  tools/perf/tests/hists_output.c   | 6 ++++--
>  tools/perf/util/hist.c            | 8 ++------
>  tools/perf/util/hist.h            | 1 -
>  7 files changed, 22 insertions(+), 19 deletions(-)
> 
> diff --git a/tools/perf/builtin-report.c b/tools/perf/builtin-report.c
> index fee770935eab..decd9e8584b5 100644
> --- a/tools/perf/builtin-report.c
> +++ b/tools/perf/builtin-report.c
> @@ -139,8 +139,10 @@ static int process_sample_event(struct perf_tool *tool,
>  	struct report *rep = container_of(tool, struct report, tool);
>  	struct addr_location al;
>  	struct hist_entry_iter iter = {
> -		.hide_unresolved = rep->hide_unresolved,
> -		.add_entry_cb = hist_iter__report_callback,
> +		.evsel 			= evsel,
> +		.sample 		= sample,
> +		.hide_unresolved 	= rep->hide_unresolved,
> +		.add_entry_cb 		= hist_iter__report_callback,
>  	};
>  	int ret = 0;
>  
> @@ -168,8 +170,7 @@ static int process_sample_event(struct perf_tool *tool,
>  	if (al.map != NULL)
>  		al.map->dso->hit = 1;
>  
> -	ret = hist_entry_iter__add(&iter, &al, evsel, sample, rep->max_stack,
> -				   rep);
> +	ret = hist_entry_iter__add(&iter, &al, rep->max_stack, rep);
>  	if (ret < 0)
>  		pr_debug("problem adding hist entry, skipping event\n");
>  out_put:
> diff --git a/tools/perf/builtin-top.c b/tools/perf/builtin-top.c
> index a19351728f0f..6b987424d015 100644
> --- a/tools/perf/builtin-top.c
> +++ b/tools/perf/builtin-top.c
> @@ -775,7 +775,9 @@ static void perf_event__process_sample(struct perf_tool *tool,
>  	if (al.sym == NULL || !al.sym->ignore) {
>  		struct hists *hists = evsel__hists(evsel);
>  		struct hist_entry_iter iter = {
> -			.add_entry_cb = hist_iter__top_callback,
> +			.evsel		= evsel,
> +			.sample 	= sample,
> +			.add_entry_cb 	= hist_iter__top_callback,
>  		};
>  
>  		if (symbol_conf.cumulate_callchain)
> @@ -785,8 +787,7 @@ static void perf_event__process_sample(struct perf_tool *tool,
>  
>  		pthread_mutex_lock(&hists->lock);
>  
> -		err = hist_entry_iter__add(&iter, &al, evsel, sample,
> -					   top->max_stack, top);
> +		err = hist_entry_iter__add(&iter, &al, top->max_stack, top);
>  		if (err < 0)
>  			pr_err("Problem incrementing symbol period, skipping event\n");
>  
> diff --git a/tools/perf/tests/hists_cumulate.c b/tools/perf/tests/hists_cumulate.c
> index 620f626e5b35..7d82c8be5e36 100644
> --- a/tools/perf/tests/hists_cumulate.c
> +++ b/tools/perf/tests/hists_cumulate.c
> @@ -87,6 +87,8 @@ static int add_hist_entries(struct hists *hists, struct machine *machine)
>  			},
>  		};
>  		struct hist_entry_iter iter = {
> +			.evsel = evsel,
> +			.sample	= &sample,
>  			.hide_unresolved = false,
>  		};
>  
> @@ -104,8 +106,8 @@ static int add_hist_entries(struct hists *hists, struct machine *machine)
>  						  &sample) < 0)
>  			goto out;
>  
> -		if (hist_entry_iter__add(&iter, &al, evsel, &sample,
> -					 PERF_MAX_STACK_DEPTH, NULL) < 0) {
> +		if (hist_entry_iter__add(&iter, &al, PERF_MAX_STACK_DEPTH,
> +					 NULL) < 0) {
>  			addr_location__put(&al);
>  			goto out;
>  		}
> diff --git a/tools/perf/tests/hists_filter.c b/tools/perf/tests/hists_filter.c
> index 82e1ee52e024..ce48775e6ada 100644
> --- a/tools/perf/tests/hists_filter.c
> +++ b/tools/perf/tests/hists_filter.c
> @@ -63,6 +63,8 @@ static int add_hist_entries(struct perf_evlist *evlist,
>  				},
>  			};
>  			struct hist_entry_iter iter = {
> +				.evsel = evsel,
> +				.sample = &sample,
>  				.ops = &hist_iter_normal,
>  				.hide_unresolved = false,
>  			};
> @@ -81,7 +83,7 @@ static int add_hist_entries(struct perf_evlist *evlist,
>  							  &sample) < 0)
>  				goto out;
>  
> -			if (hist_entry_iter__add(&iter, &al, evsel, &sample,
> +			if (hist_entry_iter__add(&iter, &al,
>  						 PERF_MAX_STACK_DEPTH, NULL) < 0) {
>  				addr_location__put(&al);
>  				goto out;
> diff --git a/tools/perf/tests/hists_output.c b/tools/perf/tests/hists_output.c
> index fd7ec4f9aeb4..adbebc852cc8 100644
> --- a/tools/perf/tests/hists_output.c
> +++ b/tools/perf/tests/hists_output.c
> @@ -57,6 +57,8 @@ static int add_hist_entries(struct hists *hists, struct machine *machine)
>  			},
>  		};
>  		struct hist_entry_iter iter = {
> +			.evsel = evsel,
> +			.sample = &sample,
>  			.ops = &hist_iter_normal,
>  			.hide_unresolved = false,
>  		};
> @@ -70,8 +72,8 @@ static int add_hist_entries(struct hists *hists, struct machine *machine)
>  						  &sample) < 0)
>  			goto out;
>  
> -		if (hist_entry_iter__add(&iter, &al, evsel, &sample,
> -					 PERF_MAX_STACK_DEPTH, NULL) < 0) {
> +		if (hist_entry_iter__add(&iter, &al, PERF_MAX_STACK_DEPTH,
> +					 NULL) < 0) {
>  			addr_location__put(&al);
>  			goto out;
>  		}
> diff --git a/tools/perf/util/hist.c b/tools/perf/util/hist.c
> index f13993e53e4e..b492968913e1 100644
> --- a/tools/perf/util/hist.c
> +++ b/tools/perf/util/hist.c
> @@ -852,19 +852,15 @@ const struct hist_iter_ops hist_iter_cumulative = {
>  };
>  
>  int hist_entry_iter__add(struct hist_entry_iter *iter, struct addr_location *al,
> -			 struct perf_evsel *evsel, struct perf_sample *sample,
>  			 int max_stack_depth, void *arg)
>  {
>  	int err, err2;
>  
> -	err = sample__resolve_callchain(sample, &iter->parent, evsel, al,
> -					max_stack_depth);
> +	err = sample__resolve_callchain(iter->sample, &iter->parent,
> +					iter->evsel, al, max_stack_depth);
>  	if (err)
>  		return err;
>  
> -	iter->evsel = evsel;
> -	iter->sample = sample;
> -
>  	err = iter->ops->prepare_entry(iter, al);
>  	if (err)
>  		goto out;
> diff --git a/tools/perf/util/hist.h b/tools/perf/util/hist.h
> index 43ffe62e25d2..d47f6730f6a4 100644
> --- a/tools/perf/util/hist.h
> +++ b/tools/perf/util/hist.h
> @@ -111,7 +111,6 @@ struct hist_entry *__hists__add_entry(struct hists *hists,
>  				      u64 weight, u64 transaction,
>  				      u64 timestamp, bool sample_self);
>  int hist_entry_iter__add(struct hist_entry_iter *iter, struct addr_location *al,
> -			 struct perf_evsel *evsel, struct perf_sample *sample,
>  			 int max_stack_depth, void *arg);
>  
>  int64_t hist_entry__cmp(struct hist_entry *left, struct hist_entry *right);
> -- 
> 2.4.0

^ permalink raw reply	[flat|nested] 221+ messages in thread

* Re: [PATCH 03/40] perf tools: Introduce copyfile_offset() function
  2015-05-18  0:30 ` [PATCH 03/40] perf tools: Introduce copyfile_offset() function Namhyung Kim
@ 2015-05-18 12:57   ` Arnaldo Carvalho de Melo
  2015-05-19  6:20     ` Namhyung Kim
  2015-05-20 12:24   ` [tip:perf/core] " tip-bot for Namhyung Kim
  1 sibling, 1 reply; 221+ messages in thread
From: Arnaldo Carvalho de Melo @ 2015-05-18 12:57 UTC (permalink / raw)
  To: Namhyung Kim
  Cc: Ingo Molnar, Peter Zijlstra, Jiri Olsa, LKML, David Ahern,
	Adrian Hunter, Andi Kleen, Frederic Weisbecker, Stephane Eranian

Em Mon, May 18, 2015 at 09:30:18AM +0900, Namhyung Kim escreveu:
> The copyfile_offset() function is to copy source data from given
> offset to a destination file with an offset.  It'll be used to build
> an indexed data file.
> 
> Signed-off-by: Namhyung Kim <namhyung@kernel.org>

>From looking at this patchkit before I recalled that some got Acks, from
Jiri, please try to collect those :-)

Doing it now, applying this patch.

- Arnaldo

> ---
>  tools/perf/util/util.c | 38 +++++++++++++++++++++++++++++---------
>  tools/perf/util/util.h |  1 +
>  2 files changed, 30 insertions(+), 9 deletions(-)
> 
> diff --git a/tools/perf/util/util.c b/tools/perf/util/util.c
> index 6104afb7e1ef..0c264bc685ac 100644
> --- a/tools/perf/util/util.c
> +++ b/tools/perf/util/util.c
> @@ -145,11 +145,38 @@ static int slow_copyfile(const char *from, const char *to, mode_t mode)
>  	return err;
>  }
>  
> +int copyfile_offset(int ifd, loff_t off_in, int ofd, loff_t off_out, u64 size)
> +{
> +	void *ptr;
> +	loff_t pgoff;
> +
> +	pgoff = off_in & ~(page_size - 1);
> +	off_in -= pgoff;
> +
> +	ptr = mmap(NULL, off_in + size, PROT_READ, MAP_PRIVATE, ifd, pgoff);
> +	if (ptr == MAP_FAILED)
> +		return -1;
> +
> +	while (size) {
> +		ssize_t ret = pwrite(ofd, ptr + off_in, size, off_out);
> +		if (ret < 0 && errno == EINTR)
> +			continue;
> +		if (ret <= 0)
> +			break;
> +
> +		size -= ret;
> +		off_in += ret;
> +		off_out -= ret;
> +	}
> +	munmap(ptr, off_in + size);
> +
> +	return size ? -1 : 0;
> +}
> +
>  int copyfile_mode(const char *from, const char *to, mode_t mode)
>  {
>  	int fromfd, tofd;
>  	struct stat st;
> -	void *addr;
>  	int err = -1;
>  
>  	if (stat(from, &st))
> @@ -166,15 +193,8 @@ int copyfile_mode(const char *from, const char *to, mode_t mode)
>  	if (tofd < 0)
>  		goto out_close_from;
>  
> -	addr = mmap(NULL, st.st_size, PROT_READ, MAP_PRIVATE, fromfd, 0);
> -	if (addr == MAP_FAILED)
> -		goto out_close_to;
> -
> -	if (write(tofd, addr, st.st_size) == st.st_size)
> -		err = 0;
> +	err = copyfile_offset(fromfd, 0, tofd, 0, st.st_size);
>  
> -	munmap(addr, st.st_size);
> -out_close_to:
>  	close(tofd);
>  	if (err)
>  		unlink(to);
> diff --git a/tools/perf/util/util.h b/tools/perf/util/util.h
> index c4fe38ac8c00..8bce58b47a82 100644
> --- a/tools/perf/util/util.h
> +++ b/tools/perf/util/util.h
> @@ -252,6 +252,7 @@ int mkdir_p(char *path, mode_t mode);
>  int rm_rf(char *path);
>  int copyfile(const char *from, const char *to);
>  int copyfile_mode(const char *from, const char *to, mode_t mode);
> +int copyfile_offset(int fromfd, loff_t from_ofs, int tofd, loff_t to_ofs, u64 size);
>  
>  s64 perf_atoll(const char *str);
>  char **argv_split(const char *str, int *argcp);
> -- 
> 2.4.0

^ permalink raw reply	[flat|nested] 221+ messages in thread

* Re: [PATCH 04/40] perf tools: Create separate mmap for dummy tracking event
  2015-05-18  0:30 ` [PATCH 04/40] perf tools: Create separate mmap for dummy tracking event Namhyung Kim
@ 2015-05-18 18:07   ` Jiri Olsa
  2015-05-19  6:24     ` Namhyung Kim
  0 siblings, 1 reply; 221+ messages in thread
From: Jiri Olsa @ 2015-05-18 18:07 UTC (permalink / raw)
  To: Namhyung Kim
  Cc: Arnaldo Carvalho de Melo, Ingo Molnar, Peter Zijlstra, LKML,
	David Ahern, Adrian Hunter, Andi Kleen, Frederic Weisbecker,
	Stephane Eranian

On Mon, May 18, 2015 at 09:30:19AM +0900, Namhyung Kim wrote:

SNIP

> -static int perf_evlist__alloc_mmap(struct perf_evlist *evlist)
> +static int perf_evlist__alloc_mmap(struct perf_evlist *evlist, bool track_mmap)
>  {
>  	evlist->nr_mmaps = cpu_map__nr(evlist->cpus);
>  	if (cpu_map__empty(evlist->cpus))
>  		evlist->nr_mmaps = thread_map__nr(evlist->threads);
>  	evlist->mmap = zalloc(evlist->nr_mmaps * sizeof(struct perf_mmap));
> -	return evlist->mmap != NULL ? 0 : -ENOMEM;
> +	if (evlist->mmap == NULL)
> +		return -ENOMEM;
> +
> +	if (track_mmap) {
> +		evlist->track_mmap = calloc(evlist->nr_mmaps,
> +					    sizeof(struct perf_mmap));
> +		if (evlist->track_mmap == NULL) {
> +			zfree(&evlist->mmap);
> +			return -ENOMEM;
> +		}
> +	}
> +	return 0;
>  }
>  
>  struct mmap_params {
> -	int prot;
> -	int mask;
> +	int	prot;
> +	size_t	len;
>  	struct auxtrace_mmap_params auxtrace_mp;
>  };
>  
> -static int __perf_evlist__mmap(struct perf_evlist *evlist, int idx,
> +static int __perf_evlist__mmap(struct perf_evlist *evlist __maybe_unused,
> +			       struct perf_mmap *pmmap,
>  			       struct mmap_params *mp, int fd)

hum, looks like this patch should be separated to:

 - one that makes __perf_evlist__mmap use perf_mmap directly
   and moves auxtrace_mmap__mmap..
   also __perf_evlist__mmap should be renamed (perf_mmap__mmap? ;-) )
   and evlist arg removed, because it's not used..

 - one that adds dummy tracing events mmapping

maybe also separate the code that alloc and free evlist->track_mmap
with perf_evlist__mmap_desc stuff, might help the readability..

jirka

^ permalink raw reply	[flat|nested] 221+ messages in thread

* Re: [PATCH 05/40] perf tools: Introduce perf_evlist__mmap_track()
  2015-05-18  0:30 ` [PATCH 05/40] perf tools: Introduce perf_evlist__mmap_track() Namhyung Kim
@ 2015-05-18 18:09   ` Jiri Olsa
  2015-05-19  6:28     ` Namhyung Kim
  0 siblings, 1 reply; 221+ messages in thread
From: Jiri Olsa @ 2015-05-18 18:09 UTC (permalink / raw)
  To: Namhyung Kim
  Cc: Arnaldo Carvalho de Melo, Ingo Molnar, Peter Zijlstra, LKML,
	David Ahern, Adrian Hunter, Andi Kleen, Frederic Weisbecker,
	Stephane Eranian

On Mon, May 18, 2015 at 09:30:20AM +0900, Namhyung Kim wrote:
> The perf_evlist__mmap_track function creates data mmaps and optionally

hum, there's no perf_evlist__mmap_track in the patch ;-)

you're now using perf_evlist__mmap_ex that was introduced by Adrian IIRC

jirka

^ permalink raw reply	[flat|nested] 221+ messages in thread

* Re: [PATCH 06/40] perf tools: Add HEADER_DATA_INDEX feature
  2015-05-18  0:30 ` [PATCH 06/40] perf tools: Add HEADER_DATA_INDEX feature Namhyung Kim
@ 2015-05-18 18:17   ` Jiri Olsa
  2015-05-19  6:34     ` Namhyung Kim
  0 siblings, 1 reply; 221+ messages in thread
From: Jiri Olsa @ 2015-05-18 18:17 UTC (permalink / raw)
  To: Namhyung Kim
  Cc: Arnaldo Carvalho de Melo, Ingo Molnar, Peter Zijlstra, LKML,
	David Ahern, Adrian Hunter, Andi Kleen, Frederic Weisbecker,
	Stephane Eranian

On Mon, May 18, 2015 at 09:30:21AM +0900, Namhyung Kim wrote:

SNIP

>  
> +static int process_data_index(struct perf_file_section *section __maybe_unused,
> +			      struct perf_header *ph, int fd,
> +			      void *data __maybe_unused)
> +{
> +	ssize_t ret;
> +	u64 nr_idx;
> +	unsigned i;
> +	struct perf_file_section *idx;
> +
> +	ret = readn(fd, &nr_idx, sizeof(nr_idx));
> +	if (ret != sizeof(nr_idx))
> +		return -1;
> +
> +	if (ph->needs_swap)
> +		nr_idx = bswap_64(nr_idx);
> +
> +	idx = calloc(nr_idx, sizeof(*idx));
> +	if (idx == NULL)
> +		return -1;
> +
> +	for (i = 0; i < nr_idx; i++) {
> +		ret = readn(fd, &idx[i], sizeof(*idx));
> +		if (ret != sizeof(*idx))

missing			free(idx);

jirka

> +			return ret;
> +
> +		if (ph->needs_swap) {
> +			idx[i].offset = bswap_64(idx[i].offset);
> +			idx[i].size   = bswap_64(idx[i].size);
> +		}
> +	}
> +
> +	ph->index = idx;
> +	ph->nr_index = nr_idx;
> +	return 0;

SNIP


^ permalink raw reply	[flat|nested] 221+ messages in thread

* Re: [PATCH 07/40] perf tools: Handle indexed data file properly
  2015-05-18  0:30 ` [PATCH 07/40] perf tools: Handle indexed data file properly Namhyung Kim
@ 2015-05-18 18:37   ` Jiri Olsa
  2015-05-19  6:40     ` Namhyung Kim
  0 siblings, 1 reply; 221+ messages in thread
From: Jiri Olsa @ 2015-05-18 18:37 UTC (permalink / raw)
  To: Namhyung Kim
  Cc: Arnaldo Carvalho de Melo, Ingo Molnar, Peter Zijlstra, LKML,
	David Ahern, Adrian Hunter, Andi Kleen, Frederic Weisbecker,
	Stephane Eranian

On Mon, May 18, 2015 at 09:30:22AM +0900, Namhyung Kim wrote:
> When perf detects data file has index table, process header part first
> and then rest data files in a row.  Note that the indexed sample data is
> recorded for each cpu/thread separately, it's already ordered with
> respect to themselves so no need to use the ordered event queue
> interface.
> 
> Signed-off-by: Namhyung Kim <namhyung@kernel.org>
> ---
>  tools/perf/perf.c         |  1 +
>  tools/perf/perf.h         |  2 ++
>  tools/perf/util/session.c | 55 +++++++++++++++++++++++++++++++++++++++--------
>  3 files changed, 49 insertions(+), 9 deletions(-)
> 
> diff --git a/tools/perf/perf.c b/tools/perf/perf.c
> index b857fcbd00cf..960942d9a0b7 100644
> --- a/tools/perf/perf.c
> +++ b/tools/perf/perf.c
> @@ -27,6 +27,7 @@ const char perf_more_info_string[] =
>  int use_browser = -1;
>  static int use_pager = -1;
>  const char *input_name;
> +bool perf_has_index;

I probably missed some discussion here, but why is this not
perf_session function? I saw some non session related functions
further in the patchset using this.. is that the reason?

it seems fairly perf_session specific.. might be worth to
keep it there? just asking ;-)

jirka

^ permalink raw reply	[flat|nested] 221+ messages in thread

* Re: [PATCH 11/40] perf tools: Add a test case for thread comm handling
  2015-05-18  0:30 ` [PATCH 11/40] perf tools: Add a test case for thread comm handling Namhyung Kim
@ 2015-05-18 19:29   ` Jiri Olsa
  2015-05-19  6:42     ` Namhyung Kim
  0 siblings, 1 reply; 221+ messages in thread
From: Jiri Olsa @ 2015-05-18 19:29 UTC (permalink / raw)
  To: Namhyung Kim
  Cc: Arnaldo Carvalho de Melo, Ingo Molnar, Peter Zijlstra, LKML,
	David Ahern, Adrian Hunter, Andi Kleen, Frederic Weisbecker,
	Stephane Eranian

On Mon, May 18, 2015 at 09:30:26AM +0900, Namhyung Kim wrote:

SNIP

> diff --git a/tools/perf/tests/builtin-test.c b/tools/perf/tests/builtin-test.c
> index f42af98a5c16..372b6395a448 100644
> --- a/tools/perf/tests/builtin-test.c
> +++ b/tools/perf/tests/builtin-test.c
> @@ -171,6 +171,10 @@ static struct test {
>  		.func = test__kmod_path__parse,
>  	},
>  	{
> +		.desc = "Test thread comm handling",
> +		.func = test__thread_comm,
> +	},
> +	{
>  		.func = NULL,
>  	},
>  };
> diff --git a/tools/perf/tests/perf-targz-src-pkg b/tools/perf/tests/perf-targz-src-pkg
> deleted file mode 100755
> index 238aa3927c71..000000000000
> --- a/tools/perf/tests/perf-targz-src-pkg
> +++ /dev/null

you dont like this one? ;-)

jirka

> @@ -1,21 +0,0 @@
> -#!/bin/sh
> -# Test one of the main kernel Makefile targets to generate a perf sources tarball
> -# suitable for build outside the full kernel sources.
> -#
> -# This is to test that the tools/perf/MANIFEST file lists all the files needed to
> -# be in such tarball, which sometimes gets broken when we move files around,
> -# like when we made some files that were in tools/perf/ available to other tools/
> -# codebases by moving it to tools/include/, etc.
> -
> -PERF=$1
> -cd ${PERF}/../..
> -make perf-targz-src-pkg > /dev/null
> -TARBALL=$(ls -rt perf-*.tar.gz)
> -TMP_DEST=$(mktemp -d)
> -tar xf ${TARBALL} -C $TMP_DEST
> -rm -f ${TARBALL}
> -cd - > /dev/null
> -make -C $TMP_DEST/perf*/tools/perf > /dev/null 2>&1
> -RC=$?
> -rm -rf ${TMP_DEST}
> -exit $RC

SNIP

^ permalink raw reply	[flat|nested] 221+ messages in thread

* Re: [PATCH 13/40] perf tools: Convert dead thread list into rbtree
  2015-05-18  0:30 ` [PATCH 13/40] perf tools: Convert dead thread list into rbtree Namhyung Kim
@ 2015-05-18 19:34   ` Jiri Olsa
  2015-05-19  6:45     ` Namhyung Kim
  0 siblings, 1 reply; 221+ messages in thread
From: Jiri Olsa @ 2015-05-18 19:34 UTC (permalink / raw)
  To: Namhyung Kim
  Cc: Arnaldo Carvalho de Melo, Ingo Molnar, Peter Zijlstra, LKML,
	David Ahern, Adrian Hunter, Andi Kleen, Frederic Weisbecker,
	Stephane Eranian

On Mon, May 18, 2015 at 09:30:28AM +0900, Namhyung Kim wrote:

SNIP

> @@ -38,6 +39,7 @@ struct thread *thread__new(pid_t pid, pid_t tid)
>  		thread->ppid = -1;
>  		thread->cpu = -1;
>  		INIT_LIST_HEAD(&thread->comm_list);
> +		INIT_LIST_HEAD(&thread->tid_node);
>  
>  		if (unwind__prepare_access(thread) < 0)
>  			goto err_thread;
> @@ -54,7 +56,7 @@ struct thread *thread__new(pid_t pid, pid_t tid)
>  
>  		list_add(&comm->list, &thread->comm_list);
>  		atomic_set(&thread->refcnt, 0);
> -		INIT_LIST_HEAD(&thread->node);
> +		INIT_LIST_HEAD(&thread->tid_node);

tid_node is initialized twice

jirka

^ permalink raw reply	[flat|nested] 221+ messages in thread

* Re: [PATCH 14/40] perf tools: Introduce machine__find*_thread_by_time()
  2015-05-18  0:30 ` [PATCH 14/40] perf tools: Introduce machine__find*_thread_by_time() Namhyung Kim
@ 2015-05-18 19:50   ` Jiri Olsa
  2015-05-18 19:56     ` Jiri Olsa
  2015-05-19  6:57     ` Namhyung Kim
  0 siblings, 2 replies; 221+ messages in thread
From: Jiri Olsa @ 2015-05-18 19:50 UTC (permalink / raw)
  To: Namhyung Kim
  Cc: Arnaldo Carvalho de Melo, Ingo Molnar, Peter Zijlstra, LKML,
	David Ahern, Adrian Hunter, Andi Kleen, Frederic Weisbecker,
	Stephane Eranian

On Mon, May 18, 2015 at 09:30:29AM +0900, Namhyung Kim wrote:

SNIP

> +static struct thread *
> +__machine__findnew_thread_by_time(struct machine *machine, pid_t pid, pid_t tid,
> +				  u64 timestamp, bool create)
> +{
> +	struct thread *curr, *pos, *new;
> +	struct thread *th = NULL;
> +	struct rb_node **p;
> +	struct rb_node *parent = NULL;
> +
> +	if (!perf_has_index)
> +		return ____machine__findnew_thread(machine, pid, tid, create);
> +
> +	/* lookup current thread first */
> +	curr = ____machine__findnew_thread(machine, pid, tid, false);
> +	if (curr && timestamp >= curr->start_time)
> +		return curr;
> +
> +	/* and then check dead threads tree & list */
> +	p = &machine->dead_threads.rb_node;
> +	while (*p != NULL) {
> +		parent = *p;
> +		th = rb_entry(parent, struct thread, rb_node);
> +
> +		if (th->tid == tid) {
> +			list_for_each_entry(pos, &th->tid_node, tid_node) {
> +				if (timestamp >= pos->start_time &&
> +				    pos->start_time > th->start_time) {
> +					th = pos;
> +					break;
> +				}

hum, how do we know, there's not another thread on the list
fitting the timestamp >= pos->start_time condition as well?

jirka

^ permalink raw reply	[flat|nested] 221+ messages in thread

* Re: [PATCH 14/40] perf tools: Introduce machine__find*_thread_by_time()
  2015-05-18 19:50   ` Jiri Olsa
@ 2015-05-18 19:56     ` Jiri Olsa
  2015-05-19  6:57     ` Namhyung Kim
  1 sibling, 0 replies; 221+ messages in thread
From: Jiri Olsa @ 2015-05-18 19:56 UTC (permalink / raw)
  To: Namhyung Kim
  Cc: Arnaldo Carvalho de Melo, Ingo Molnar, Peter Zijlstra, LKML,
	David Ahern, Adrian Hunter, Andi Kleen, Frederic Weisbecker,
	Stephane Eranian

On Mon, May 18, 2015 at 09:50:06PM +0200, Jiri Olsa wrote:
> On Mon, May 18, 2015 at 09:30:29AM +0900, Namhyung Kim wrote:
> 
> SNIP
> 
> > +static struct thread *
> > +__machine__findnew_thread_by_time(struct machine *machine, pid_t pid, pid_t tid,
> > +				  u64 timestamp, bool create)
> > +{
> > +	struct thread *curr, *pos, *new;
> > +	struct thread *th = NULL;
> > +	struct rb_node **p;
> > +	struct rb_node *parent = NULL;
> > +
> > +	if (!perf_has_index)
> > +		return ____machine__findnew_thread(machine, pid, tid, create);
> > +
> > +	/* lookup current thread first */
> > +	curr = ____machine__findnew_thread(machine, pid, tid, false);
> > +	if (curr && timestamp >= curr->start_time)
> > +		return curr;
> > +
> > +	/* and then check dead threads tree & list */
> > +	p = &machine->dead_threads.rb_node;
> > +	while (*p != NULL) {
> > +		parent = *p;
> > +		th = rb_entry(parent, struct thread, rb_node);
> > +
> > +		if (th->tid == tid) {
> > +			list_for_each_entry(pos, &th->tid_node, tid_node) {
> > +				if (timestamp >= pos->start_time &&
> > +				    pos->start_time > th->start_time) {
> > +					th = pos;
> > +					break;
> > +				}
> 
> hum, how do we know, there's not another thread on the list
> fitting the timestamp >= pos->start_time condition as well?

should we store exit_time for dead threads?

jirka

^ permalink raw reply	[flat|nested] 221+ messages in thread

* Re: [PATCH 03/40] perf tools: Introduce copyfile_offset() function
  2015-05-18 12:57   ` Arnaldo Carvalho de Melo
@ 2015-05-19  6:20     ` Namhyung Kim
  0 siblings, 0 replies; 221+ messages in thread
From: Namhyung Kim @ 2015-05-19  6:20 UTC (permalink / raw)
  To: Arnaldo Carvalho de Melo
  Cc: Ingo Molnar, Peter Zijlstra, Jiri Olsa, LKML, David Ahern,
	Adrian Hunter, Andi Kleen, Frederic Weisbecker, Stephane Eranian

On Mon, May 18, 2015 at 09:57:23AM -0300, Arnaldo Carvalho de Melo wrote:
> Em Mon, May 18, 2015 at 09:30:18AM +0900, Namhyung Kim escreveu:
> > The copyfile_offset() function is to copy source data from given
> > offset to a destination file with an offset.  It'll be used to build
> > an indexed data file.
> > 
> > Signed-off-by: Namhyung Kim <namhyung@kernel.org>
> 
> From looking at this patchkit before I recalled that some got Acks, from
> Jiri, please try to collect those :-)

Ah, right.  I completely forgot about it, sorry..

> 
> Doing it now, applying this patch.

Thank you!
Namhyung

^ permalink raw reply	[flat|nested] 221+ messages in thread

* Re: [PATCH 04/40] perf tools: Create separate mmap for dummy tracking event
  2015-05-18 18:07   ` Jiri Olsa
@ 2015-05-19  6:24     ` Namhyung Kim
  0 siblings, 0 replies; 221+ messages in thread
From: Namhyung Kim @ 2015-05-19  6:24 UTC (permalink / raw)
  To: Jiri Olsa
  Cc: Arnaldo Carvalho de Melo, Ingo Molnar, Peter Zijlstra, LKML,
	David Ahern, Adrian Hunter, Andi Kleen, Frederic Weisbecker,
	Stephane Eranian

Hi Jiri,

On Mon, May 18, 2015 at 08:07:13PM +0200, Jiri Olsa wrote:
> On Mon, May 18, 2015 at 09:30:19AM +0900, Namhyung Kim wrote:
> 
> SNIP
> 
> > -static int perf_evlist__alloc_mmap(struct perf_evlist *evlist)
> > +static int perf_evlist__alloc_mmap(struct perf_evlist *evlist, bool track_mmap)
> >  {
> >  	evlist->nr_mmaps = cpu_map__nr(evlist->cpus);
> >  	if (cpu_map__empty(evlist->cpus))
> >  		evlist->nr_mmaps = thread_map__nr(evlist->threads);
> >  	evlist->mmap = zalloc(evlist->nr_mmaps * sizeof(struct perf_mmap));
> > -	return evlist->mmap != NULL ? 0 : -ENOMEM;
> > +	if (evlist->mmap == NULL)
> > +		return -ENOMEM;
> > +
> > +	if (track_mmap) {
> > +		evlist->track_mmap = calloc(evlist->nr_mmaps,
> > +					    sizeof(struct perf_mmap));
> > +		if (evlist->track_mmap == NULL) {
> > +			zfree(&evlist->mmap);
> > +			return -ENOMEM;
> > +		}
> > +	}
> > +	return 0;
> >  }
> >  
> >  struct mmap_params {
> > -	int prot;
> > -	int mask;
> > +	int	prot;
> > +	size_t	len;
> >  	struct auxtrace_mmap_params auxtrace_mp;
> >  };
> >  
> > -static int __perf_evlist__mmap(struct perf_evlist *evlist, int idx,
> > +static int __perf_evlist__mmap(struct perf_evlist *evlist __maybe_unused,
> > +			       struct perf_mmap *pmmap,
> >  			       struct mmap_params *mp, int fd)
> 
> hum, looks like this patch should be separated to:
> 
>  - one that makes __perf_evlist__mmap use perf_mmap directly
>    and moves auxtrace_mmap__mmap..
>    also __perf_evlist__mmap should be renamed (perf_mmap__mmap? ;-) )
>    and evlist arg removed, because it's not used..
> 
>  - one that adds dummy tracing events mmapping
> 
> maybe also separate the code that alloc and free evlist->track_mmap
> with perf_evlist__mmap_desc stuff, might help the readability..

OK, will do.

Thanks,
Namhyung

^ permalink raw reply	[flat|nested] 221+ messages in thread

* Re: [PATCH 05/40] perf tools: Introduce perf_evlist__mmap_track()
  2015-05-18 18:09   ` Jiri Olsa
@ 2015-05-19  6:28     ` Namhyung Kim
  0 siblings, 0 replies; 221+ messages in thread
From: Namhyung Kim @ 2015-05-19  6:28 UTC (permalink / raw)
  To: Jiri Olsa
  Cc: Arnaldo Carvalho de Melo, Ingo Molnar, Peter Zijlstra, LKML,
	David Ahern, Adrian Hunter, Andi Kleen, Frederic Weisbecker,
	Stephane Eranian

On Mon, May 18, 2015 at 08:09:16PM +0200, Jiri Olsa wrote:
> On Mon, May 18, 2015 at 09:30:20AM +0900, Namhyung Kim wrote:
> > The perf_evlist__mmap_track function creates data mmaps and optionally
> 
> hum, there's no perf_evlist__mmap_track in the patch ;-)

Oops, I missed to update the changelog ;-P

> 
> you're now using perf_evlist__mmap_ex that was introduced by Adrian IIRC

Right, thanks for letting me know.  Will update.

Thanks
Namhyung

^ permalink raw reply	[flat|nested] 221+ messages in thread

* Re: [PATCH 06/40] perf tools: Add HEADER_DATA_INDEX feature
  2015-05-18 18:17   ` Jiri Olsa
@ 2015-05-19  6:34     ` Namhyung Kim
  0 siblings, 0 replies; 221+ messages in thread
From: Namhyung Kim @ 2015-05-19  6:34 UTC (permalink / raw)
  To: Jiri Olsa
  Cc: Arnaldo Carvalho de Melo, Ingo Molnar, Peter Zijlstra, LKML,
	David Ahern, Adrian Hunter, Andi Kleen, Frederic Weisbecker,
	Stephane Eranian

On Mon, May 18, 2015 at 08:17:23PM +0200, Jiri Olsa wrote:
> On Mon, May 18, 2015 at 09:30:21AM +0900, Namhyung Kim wrote:
> 
> SNIP
> 
> >  
> > +static int process_data_index(struct perf_file_section *section __maybe_unused,
> > +			      struct perf_header *ph, int fd,
> > +			      void *data __maybe_unused)
> > +{
> > +	ssize_t ret;
> > +	u64 nr_idx;
> > +	unsigned i;
> > +	struct perf_file_section *idx;
> > +
> > +	ret = readn(fd, &nr_idx, sizeof(nr_idx));
> > +	if (ret != sizeof(nr_idx))
> > +		return -1;
> > +
> > +	if (ph->needs_swap)
> > +		nr_idx = bswap_64(nr_idx);
> > +
> > +	idx = calloc(nr_idx, sizeof(*idx));
> > +	if (idx == NULL)
> > +		return -1;
> > +
> > +	for (i = 0; i < nr_idx; i++) {
> > +		ret = readn(fd, &idx[i], sizeof(*idx));
> > +		if (ret != sizeof(*idx))
> 
> missing			free(idx);

You're right.  Also it needs to return negative value for error, will update.

Thanks,
Namhyung


> 
> > +			return ret;
> > +
> > +		if (ph->needs_swap) {
> > +			idx[i].offset = bswap_64(idx[i].offset);
> > +			idx[i].size   = bswap_64(idx[i].size);
> > +		}
> > +	}
> > +
> > +	ph->index = idx;
> > +	ph->nr_index = nr_idx;
> > +	return 0;
> 
> SNIP
> 
> --
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/

^ permalink raw reply	[flat|nested] 221+ messages in thread

* Re: [PATCH 07/40] perf tools: Handle indexed data file properly
  2015-05-18 18:37   ` Jiri Olsa
@ 2015-05-19  6:40     ` Namhyung Kim
  0 siblings, 0 replies; 221+ messages in thread
From: Namhyung Kim @ 2015-05-19  6:40 UTC (permalink / raw)
  To: Jiri Olsa
  Cc: Arnaldo Carvalho de Melo, Ingo Molnar, Peter Zijlstra, LKML,
	David Ahern, Adrian Hunter, Andi Kleen, Frederic Weisbecker,
	Stephane Eranian

On Mon, May 18, 2015 at 08:37:28PM +0200, Jiri Olsa wrote:
> On Mon, May 18, 2015 at 09:30:22AM +0900, Namhyung Kim wrote:
> > When perf detects data file has index table, process header part first
> > and then rest data files in a row.  Note that the indexed sample data is
> > recorded for each cpu/thread separately, it's already ordered with
> > respect to themselves so no need to use the ordered event queue
> > interface.
> > 
> > Signed-off-by: Namhyung Kim <namhyung@kernel.org>
> > ---
> >  tools/perf/perf.c         |  1 +
> >  tools/perf/perf.h         |  2 ++
> >  tools/perf/util/session.c | 55 +++++++++++++++++++++++++++++++++++++++--------
> >  3 files changed, 49 insertions(+), 9 deletions(-)
> > 
> > diff --git a/tools/perf/perf.c b/tools/perf/perf.c
> > index b857fcbd00cf..960942d9a0b7 100644
> > --- a/tools/perf/perf.c
> > +++ b/tools/perf/perf.c
> > @@ -27,6 +27,7 @@ const char perf_more_info_string[] =
> >  int use_browser = -1;
> >  static int use_pager = -1;
> >  const char *input_name;
> > +bool perf_has_index;
> 
> I probably missed some discussion here, but why is this not
> perf_session function? I saw some non session related functions
> further in the patchset using this.. is that the reason?
> 
> it seems fairly perf_session specific.. might be worth to
> keep it there? just asking ;-)

I agree.  But it requires to pass a pointer to session wherever
processing samples.  Arnaldo said that he wanted to remove accesses to
a session in general.  So I added the global variable.

https://lkml.org/lkml/2015/3/3/353

Thanks,
Namhyung

^ permalink raw reply	[flat|nested] 221+ messages in thread

* Re: [PATCH 11/40] perf tools: Add a test case for thread comm handling
  2015-05-18 19:29   ` Jiri Olsa
@ 2015-05-19  6:42     ` Namhyung Kim
  0 siblings, 0 replies; 221+ messages in thread
From: Namhyung Kim @ 2015-05-19  6:42 UTC (permalink / raw)
  To: Jiri Olsa
  Cc: Arnaldo Carvalho de Melo, Ingo Molnar, Peter Zijlstra, LKML,
	David Ahern, Adrian Hunter, Andi Kleen, Frederic Weisbecker,
	Stephane Eranian

On Mon, May 18, 2015 at 09:29:03PM +0200, Jiri Olsa wrote:
> On Mon, May 18, 2015 at 09:30:26AM +0900, Namhyung Kim wrote:
> 
> SNIP
> 
> > diff --git a/tools/perf/tests/builtin-test.c b/tools/perf/tests/builtin-test.c
> > index f42af98a5c16..372b6395a448 100644
> > --- a/tools/perf/tests/builtin-test.c
> > +++ b/tools/perf/tests/builtin-test.c
> > @@ -171,6 +171,10 @@ static struct test {
> >  		.func = test__kmod_path__parse,
> >  	},
> >  	{
> > +		.desc = "Test thread comm handling",
> > +		.func = test__thread_comm,
> > +	},
> > +	{
> >  		.func = NULL,
> >  	},
> >  };
> > diff --git a/tools/perf/tests/perf-targz-src-pkg b/tools/perf/tests/perf-targz-src-pkg
> > deleted file mode 100755
> > index 238aa3927c71..000000000000
> > --- a/tools/perf/tests/perf-targz-src-pkg
> > +++ /dev/null
> 
> you dont like this one? ;-)

Argh, I don't know how it's included in this patch.  Will remove :-/

Thanks,
Namhyung


> 
> jirka
> 
> > @@ -1,21 +0,0 @@
> > -#!/bin/sh
> > -# Test one of the main kernel Makefile targets to generate a perf sources tarball
> > -# suitable for build outside the full kernel sources.
> > -#
> > -# This is to test that the tools/perf/MANIFEST file lists all the files needed to
> > -# be in such tarball, which sometimes gets broken when we move files around,
> > -# like when we made some files that were in tools/perf/ available to other tools/
> > -# codebases by moving it to tools/include/, etc.
> > -
> > -PERF=$1
> > -cd ${PERF}/../..
> > -make perf-targz-src-pkg > /dev/null
> > -TARBALL=$(ls -rt perf-*.tar.gz)
> > -TMP_DEST=$(mktemp -d)
> > -tar xf ${TARBALL} -C $TMP_DEST
> > -rm -f ${TARBALL}
> > -cd - > /dev/null
> > -make -C $TMP_DEST/perf*/tools/perf > /dev/null 2>&1
> > -RC=$?
> > -rm -rf ${TMP_DEST}
> > -exit $RC
> 
> SNIP
> --
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/

^ permalink raw reply	[flat|nested] 221+ messages in thread

* Re: [PATCH 13/40] perf tools: Convert dead thread list into rbtree
  2015-05-18 19:34   ` Jiri Olsa
@ 2015-05-19  6:45     ` Namhyung Kim
  0 siblings, 0 replies; 221+ messages in thread
From: Namhyung Kim @ 2015-05-19  6:45 UTC (permalink / raw)
  To: Jiri Olsa
  Cc: Arnaldo Carvalho de Melo, Ingo Molnar, Peter Zijlstra, LKML,
	David Ahern, Adrian Hunter, Andi Kleen, Frederic Weisbecker,
	Stephane Eranian

On Mon, May 18, 2015 at 09:34:25PM +0200, Jiri Olsa wrote:
> On Mon, May 18, 2015 at 09:30:28AM +0900, Namhyung Kim wrote:
> 
> SNIP
> 
> > @@ -38,6 +39,7 @@ struct thread *thread__new(pid_t pid, pid_t tid)
> >  		thread->ppid = -1;
> >  		thread->cpu = -1;
> >  		INIT_LIST_HEAD(&thread->comm_list);
> > +		INIT_LIST_HEAD(&thread->tid_node);
> >  
> >  		if (unwind__prepare_access(thread) < 0)
> >  			goto err_thread;
> > @@ -54,7 +56,7 @@ struct thread *thread__new(pid_t pid, pid_t tid)
> >  
> >  		list_add(&comm->list, &thread->comm_list);
> >  		atomic_set(&thread->refcnt, 0);
> > -		INIT_LIST_HEAD(&thread->node);
> > +		INIT_LIST_HEAD(&thread->tid_node);
> 
> tid_node is initialized twice

Okay, will fix.

Thanks,
Namhyung

^ permalink raw reply	[flat|nested] 221+ messages in thread

* Re: [PATCH 14/40] perf tools: Introduce machine__find*_thread_by_time()
  2015-05-18 19:50   ` Jiri Olsa
  2015-05-18 19:56     ` Jiri Olsa
@ 2015-05-19  6:57     ` Namhyung Kim
  1 sibling, 0 replies; 221+ messages in thread
From: Namhyung Kim @ 2015-05-19  6:57 UTC (permalink / raw)
  To: Jiri Olsa
  Cc: Arnaldo Carvalho de Melo, Ingo Molnar, Peter Zijlstra, LKML,
	David Ahern, Adrian Hunter, Andi Kleen, Frederic Weisbecker,
	Stephane Eranian

On Mon, May 18, 2015 at 09:50:06PM +0200, Jiri Olsa wrote:
> On Mon, May 18, 2015 at 09:30:29AM +0900, Namhyung Kim wrote:
> 
> SNIP
> 
> > +static struct thread *
> > +__machine__findnew_thread_by_time(struct machine *machine, pid_t pid, pid_t tid,
> > +				  u64 timestamp, bool create)
> > +{
> > +	struct thread *curr, *pos, *new;
> > +	struct thread *th = NULL;
> > +	struct rb_node **p;
> > +	struct rb_node *parent = NULL;
> > +
> > +	if (!perf_has_index)
> > +		return ____machine__findnew_thread(machine, pid, tid, create);
> > +
> > +	/* lookup current thread first */
> > +	curr = ____machine__findnew_thread(machine, pid, tid, false);
> > +	if (curr && timestamp >= curr->start_time)
> > +		return curr;
> > +
> > +	/* and then check dead threads tree & list */
> > +	p = &machine->dead_threads.rb_node;
> > +	while (*p != NULL) {
> > +		parent = *p;
> > +		th = rb_entry(parent, struct thread, rb_node);
> > +
> > +		if (th->tid == tid) {
> > +			list_for_each_entry(pos, &th->tid_node, tid_node) {
> > +				if (timestamp >= pos->start_time &&
> > +				    pos->start_time > th->start_time) {
> > +					th = pos;
> > +					break;
> > +				}
> 
> hum, how do we know, there's not another thread on the list
> fitting the timestamp >= pos->start_time condition as well?

Because the list is sorted on time - it'll see thread that has a
larger start_time first so it can ensure that a thread which satisfies
the expression first is the current thread at that time.  Later
threads in the list will satisfy the expression also but they're
already dead.

Thanks,
Namhyung

^ permalink raw reply	[flat|nested] 221+ messages in thread

* Re: [PATCH 16/40] perf tools: Reducing arguments of hist_entry_iter__add()
  2015-05-18 12:55   ` Arnaldo Carvalho de Melo
@ 2015-05-19  7:01     ` Namhyung Kim
  2015-05-19  8:04       ` [PATCH] " Namhyung Kim
  0 siblings, 1 reply; 221+ messages in thread
From: Namhyung Kim @ 2015-05-19  7:01 UTC (permalink / raw)
  To: Arnaldo Carvalho de Melo
  Cc: Ingo Molnar, Peter Zijlstra, Jiri Olsa, LKML, David Ahern,
	Adrian Hunter, Andi Kleen, Frederic Weisbecker, Stephane Eranian

On Mon, May 18, 2015 at 09:55:41AM -0300, Arnaldo Carvalho de Melo wrote:
> Em Mon, May 18, 2015 at 09:30:31AM +0900, Namhyung Kim escreveu:
> > The evsel and sample arguments are to set iter for later use.  As it
> > also receives an iter as another argument, just set them before
> > calling the function.
> 
> This looks like can be applied sooner, tried but it didn't apply :-\

I'll send it as a separate patch.

Thanks,
Namhyung


> 
> - Arnaldo
>  
> > Signed-off-by: Namhyung Kim <namhyung@kernel.org>
> > ---
> >  tools/perf/builtin-report.c       | 9 +++++----
> >  tools/perf/builtin-top.c          | 7 ++++---
> >  tools/perf/tests/hists_cumulate.c | 6 ++++--
> >  tools/perf/tests/hists_filter.c   | 4 +++-
> >  tools/perf/tests/hists_output.c   | 6 ++++--
> >  tools/perf/util/hist.c            | 8 ++------
> >  tools/perf/util/hist.h            | 1 -
> >  7 files changed, 22 insertions(+), 19 deletions(-)
> > 
> > diff --git a/tools/perf/builtin-report.c b/tools/perf/builtin-report.c
> > index fee770935eab..decd9e8584b5 100644
> > --- a/tools/perf/builtin-report.c
> > +++ b/tools/perf/builtin-report.c
> > @@ -139,8 +139,10 @@ static int process_sample_event(struct perf_tool *tool,
> >  	struct report *rep = container_of(tool, struct report, tool);
> >  	struct addr_location al;
> >  	struct hist_entry_iter iter = {
> > -		.hide_unresolved = rep->hide_unresolved,
> > -		.add_entry_cb = hist_iter__report_callback,
> > +		.evsel 			= evsel,
> > +		.sample 		= sample,
> > +		.hide_unresolved 	= rep->hide_unresolved,
> > +		.add_entry_cb 		= hist_iter__report_callback,
> >  	};
> >  	int ret = 0;
> >  
> > @@ -168,8 +170,7 @@ static int process_sample_event(struct perf_tool *tool,
> >  	if (al.map != NULL)
> >  		al.map->dso->hit = 1;
> >  
> > -	ret = hist_entry_iter__add(&iter, &al, evsel, sample, rep->max_stack,
> > -				   rep);
> > +	ret = hist_entry_iter__add(&iter, &al, rep->max_stack, rep);
> >  	if (ret < 0)
> >  		pr_debug("problem adding hist entry, skipping event\n");
> >  out_put:
> > diff --git a/tools/perf/builtin-top.c b/tools/perf/builtin-top.c
> > index a19351728f0f..6b987424d015 100644
> > --- a/tools/perf/builtin-top.c
> > +++ b/tools/perf/builtin-top.c
> > @@ -775,7 +775,9 @@ static void perf_event__process_sample(struct perf_tool *tool,
> >  	if (al.sym == NULL || !al.sym->ignore) {
> >  		struct hists *hists = evsel__hists(evsel);
> >  		struct hist_entry_iter iter = {
> > -			.add_entry_cb = hist_iter__top_callback,
> > +			.evsel		= evsel,
> > +			.sample 	= sample,
> > +			.add_entry_cb 	= hist_iter__top_callback,
> >  		};
> >  
> >  		if (symbol_conf.cumulate_callchain)
> > @@ -785,8 +787,7 @@ static void perf_event__process_sample(struct perf_tool *tool,
> >  
> >  		pthread_mutex_lock(&hists->lock);
> >  
> > -		err = hist_entry_iter__add(&iter, &al, evsel, sample,
> > -					   top->max_stack, top);
> > +		err = hist_entry_iter__add(&iter, &al, top->max_stack, top);
> >  		if (err < 0)
> >  			pr_err("Problem incrementing symbol period, skipping event\n");
> >  
> > diff --git a/tools/perf/tests/hists_cumulate.c b/tools/perf/tests/hists_cumulate.c
> > index 620f626e5b35..7d82c8be5e36 100644
> > --- a/tools/perf/tests/hists_cumulate.c
> > +++ b/tools/perf/tests/hists_cumulate.c
> > @@ -87,6 +87,8 @@ static int add_hist_entries(struct hists *hists, struct machine *machine)
> >  			},
> >  		};
> >  		struct hist_entry_iter iter = {
> > +			.evsel = evsel,
> > +			.sample	= &sample,
> >  			.hide_unresolved = false,
> >  		};
> >  
> > @@ -104,8 +106,8 @@ static int add_hist_entries(struct hists *hists, struct machine *machine)
> >  						  &sample) < 0)
> >  			goto out;
> >  
> > -		if (hist_entry_iter__add(&iter, &al, evsel, &sample,
> > -					 PERF_MAX_STACK_DEPTH, NULL) < 0) {
> > +		if (hist_entry_iter__add(&iter, &al, PERF_MAX_STACK_DEPTH,
> > +					 NULL) < 0) {
> >  			addr_location__put(&al);
> >  			goto out;
> >  		}
> > diff --git a/tools/perf/tests/hists_filter.c b/tools/perf/tests/hists_filter.c
> > index 82e1ee52e024..ce48775e6ada 100644
> > --- a/tools/perf/tests/hists_filter.c
> > +++ b/tools/perf/tests/hists_filter.c
> > @@ -63,6 +63,8 @@ static int add_hist_entries(struct perf_evlist *evlist,
> >  				},
> >  			};
> >  			struct hist_entry_iter iter = {
> > +				.evsel = evsel,
> > +				.sample = &sample,
> >  				.ops = &hist_iter_normal,
> >  				.hide_unresolved = false,
> >  			};
> > @@ -81,7 +83,7 @@ static int add_hist_entries(struct perf_evlist *evlist,
> >  							  &sample) < 0)
> >  				goto out;
> >  
> > -			if (hist_entry_iter__add(&iter, &al, evsel, &sample,
> > +			if (hist_entry_iter__add(&iter, &al,
> >  						 PERF_MAX_STACK_DEPTH, NULL) < 0) {
> >  				addr_location__put(&al);
> >  				goto out;
> > diff --git a/tools/perf/tests/hists_output.c b/tools/perf/tests/hists_output.c
> > index fd7ec4f9aeb4..adbebc852cc8 100644
> > --- a/tools/perf/tests/hists_output.c
> > +++ b/tools/perf/tests/hists_output.c
> > @@ -57,6 +57,8 @@ static int add_hist_entries(struct hists *hists, struct machine *machine)
> >  			},
> >  		};
> >  		struct hist_entry_iter iter = {
> > +			.evsel = evsel,
> > +			.sample = &sample,
> >  			.ops = &hist_iter_normal,
> >  			.hide_unresolved = false,
> >  		};
> > @@ -70,8 +72,8 @@ static int add_hist_entries(struct hists *hists, struct machine *machine)
> >  						  &sample) < 0)
> >  			goto out;
> >  
> > -		if (hist_entry_iter__add(&iter, &al, evsel, &sample,
> > -					 PERF_MAX_STACK_DEPTH, NULL) < 0) {
> > +		if (hist_entry_iter__add(&iter, &al, PERF_MAX_STACK_DEPTH,
> > +					 NULL) < 0) {
> >  			addr_location__put(&al);
> >  			goto out;
> >  		}
> > diff --git a/tools/perf/util/hist.c b/tools/perf/util/hist.c
> > index f13993e53e4e..b492968913e1 100644
> > --- a/tools/perf/util/hist.c
> > +++ b/tools/perf/util/hist.c
> > @@ -852,19 +852,15 @@ const struct hist_iter_ops hist_iter_cumulative = {
> >  };
> >  
> >  int hist_entry_iter__add(struct hist_entry_iter *iter, struct addr_location *al,
> > -			 struct perf_evsel *evsel, struct perf_sample *sample,
> >  			 int max_stack_depth, void *arg)
> >  {
> >  	int err, err2;
> >  
> > -	err = sample__resolve_callchain(sample, &iter->parent, evsel, al,
> > -					max_stack_depth);
> > +	err = sample__resolve_callchain(iter->sample, &iter->parent,
> > +					iter->evsel, al, max_stack_depth);
> >  	if (err)
> >  		return err;
> >  
> > -	iter->evsel = evsel;
> > -	iter->sample = sample;
> > -
> >  	err = iter->ops->prepare_entry(iter, al);
> >  	if (err)
> >  		goto out;
> > diff --git a/tools/perf/util/hist.h b/tools/perf/util/hist.h
> > index 43ffe62e25d2..d47f6730f6a4 100644
> > --- a/tools/perf/util/hist.h
> > +++ b/tools/perf/util/hist.h
> > @@ -111,7 +111,6 @@ struct hist_entry *__hists__add_entry(struct hists *hists,
> >  				      u64 weight, u64 transaction,
> >  				      u64 timestamp, bool sample_self);
> >  int hist_entry_iter__add(struct hist_entry_iter *iter, struct addr_location *al,
> > -			 struct perf_evsel *evsel, struct perf_sample *sample,
> >  			 int max_stack_depth, void *arg);
> >  
> >  int64_t hist_entry__cmp(struct hist_entry *left, struct hist_entry *right);
> > -- 
> > 2.4.0
> --
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/

^ permalink raw reply	[flat|nested] 221+ messages in thread

* Re: [PATCH 33/40] perf session: Separate struct machines from session
  2015-05-18 12:52   ` Arnaldo Carvalho de Melo
@ 2015-05-19  7:28     ` Namhyung Kim
  2015-05-19 22:46       ` Arnaldo Carvalho de Melo
  0 siblings, 1 reply; 221+ messages in thread
From: Namhyung Kim @ 2015-05-19  7:28 UTC (permalink / raw)
  To: Arnaldo Carvalho de Melo
  Cc: Ingo Molnar, Peter Zijlstra, Jiri Olsa, LKML, David Ahern,
	Adrian Hunter, Andi Kleen, Frederic Weisbecker, Stephane Eranian

On Mon, May 18, 2015 at 09:52:59AM -0300, Arnaldo Carvalho de Melo wrote:
> Em Mon, May 18, 2015 at 09:30:48AM +0900, Namhyung Kim escreveu:
> > With multi-thread report, separate sessions can be passed to each
> > thread, in this case we should keep a single machine state for all
> > struct sessions.  Separate machines and have a pointer in sessions.
> 
> I had to look at all the patch to semi-figure this out, i.e. you said it
> should be separated from 'perf_session', agreed.
> 
> But who will create it?  How will it be passed to the perf_session
> instances?
> 
> Most of the patch is making session->machines be turned into a pointer,
> but the meat, i.e. who creates it, is unclear, I see a malloc in
> perf_session__new(), where I was kinda expecting that a higer layer,
> perhaps in struct tool? Would create the list of all machines (struct
> machines) and then pass it to multiple perf_session__new() calls.
> 
> But then perf_session__delete() calls 'free(session->machines)', huh?

OK.  So, this is what I have in my head:

  perf_tool__create_machines(tool) {
    tool->machines = malloc();
    machines__init(tool->machines);
  }

  perf_session__new(file, repipe, tool) {
    session->machines = tool->machines;
    ...
  }

  perf_tool__delete_machines(tool) {
    /* call machines-related destructors */
    free(tool->machines);
  }

Are you OK with this?

Thanks,
Namhyung


> 
> - Arnaldo
>  
> > Signed-off-by: Namhyung Kim <namhyung@kernel.org>
> > ---
> >  tools/perf/builtin-annotate.c |  2 +-
> >  tools/perf/builtin-kmem.c     | 10 +++++-----
> >  tools/perf/builtin-kvm.c      |  2 +-
> >  tools/perf/builtin-record.c   |  4 ++--
> >  tools/perf/builtin-report.c   |  4 ++--
> >  tools/perf/builtin-top.c      |  8 ++++----
> >  tools/perf/builtin-trace.c    |  2 +-
> >  tools/perf/util/build-id.c    | 16 ++++++++--------
> >  tools/perf/util/session.c     | 36 +++++++++++++++++++++---------------
> >  tools/perf/util/session.h     |  6 +++---
> >  10 files changed, 48 insertions(+), 42 deletions(-)
> > 
> > diff --git a/tools/perf/builtin-annotate.c b/tools/perf/builtin-annotate.c
> > index 761f902473b7..d4ad323ddfe2 100644
> > --- a/tools/perf/builtin-annotate.c
> > +++ b/tools/perf/builtin-annotate.c
> > @@ -196,7 +196,7 @@ static int __cmd_annotate(struct perf_annotate *ann)
> >  	struct perf_evsel *pos;
> >  	u64 total_nr_samples;
> >  
> > -	machines__set_symbol_filter(&session->machines, symbol__annotate_init);
> > +	machines__set_symbol_filter(session->machines, symbol__annotate_init);
> >  
> >  	if (ann->cpu_list) {
> >  		ret = perf_session__cpu_bitmap(session, ann->cpu_list,
> > diff --git a/tools/perf/builtin-kmem.c b/tools/perf/builtin-kmem.c
> > index 254614b10c4a..4502954094e5 100644
> > --- a/tools/perf/builtin-kmem.c
> > +++ b/tools/perf/builtin-kmem.c
> > @@ -316,7 +316,7 @@ static int build_alloc_func_list(void)
> >  	struct symbol *sym;
> >  	struct rb_node *node;
> >  	struct alloc_func *func;
> > -	struct machine *machine = &kmem_session->machines.host;
> > +	struct machine *machine = &kmem_session->machines->host;
> >  	regex_t alloc_func_regex;
> >  	const char pattern[] = "^_?_?(alloc|get_free|get_zeroed)_pages?";
> >  
> > @@ -366,7 +366,7 @@ static int build_alloc_func_list(void)
> >  static u64 find_callsite(struct perf_evsel *evsel, struct perf_sample *sample)
> >  {
> >  	struct addr_location al;
> > -	struct machine *machine = &kmem_session->machines.host;
> > +	struct machine *machine = &kmem_session->machines->host;
> >  	struct callchain_cursor_node *node;
> >  
> >  	if (alloc_func_list == NULL) {
> > @@ -949,7 +949,7 @@ static void __print_slab_result(struct rb_root *root,
> >  				int n_lines, int is_caller)
> >  {
> >  	struct rb_node *next;
> > -	struct machine *machine = &session->machines.host;
> > +	struct machine *machine = &session->machines->host;
> >  
> >  	printf("%.105s\n", graph_dotted_line);
> >  	printf(" %-34s |",  is_caller ? "Callsite": "Alloc Ptr");
> > @@ -1010,7 +1010,7 @@ static const char * const migrate_type_str[] = {
> >  static void __print_page_alloc_result(struct perf_session *session, int n_lines)
> >  {
> >  	struct rb_node *next = rb_first(&page_alloc_sorted);
> > -	struct machine *machine = &session->machines.host;
> > +	struct machine *machine = &session->machines->host;
> >  	const char *format;
> >  	int gfp_len = max(strlen("GFP flags"), max_gfp_len);
> >  
> > @@ -1060,7 +1060,7 @@ static void __print_page_alloc_result(struct perf_session *session, int n_lines)
> >  static void __print_page_caller_result(struct perf_session *session, int n_lines)
> >  {
> >  	struct rb_node *next = rb_first(&page_caller_sorted);
> > -	struct machine *machine = &session->machines.host;
> > +	struct machine *machine = &session->machines->host;
> >  	int gfp_len = max(strlen("GFP flags"), max_gfp_len);
> >  
> >  	printf("\n%.105s\n", graph_dotted_line);
> > diff --git a/tools/perf/builtin-kvm.c b/tools/perf/builtin-kvm.c
> > index 15fecd3dc5d8..de3c7e1d0b80 100644
> > --- a/tools/perf/builtin-kvm.c
> > +++ b/tools/perf/builtin-kvm.c
> > @@ -1392,7 +1392,7 @@ static int kvm_events_live(struct perf_kvm_stat *kvm,
> >  	kvm->session->evlist = kvm->evlist;
> >  	perf_session__set_id_hdr_size(kvm->session);
> >  	ordered_events__set_copy_on_queue(&kvm->session->ordered_events, true);
> > -	machine__synthesize_threads(&kvm->session->machines.host, &kvm->opts.target,
> > +	machine__synthesize_threads(&kvm->session->machines->host, &kvm->opts.target,
> >  				    kvm->evlist->threads, false);
> >  	err = kvm_live_open_events(kvm);
> >  	if (err)
> > diff --git a/tools/perf/builtin-record.c b/tools/perf/builtin-record.c
> > index 4ddf104f50ff..978ebf648aab 100644
> > --- a/tools/perf/builtin-record.c
> > +++ b/tools/perf/builtin-record.c
> > @@ -684,7 +684,7 @@ static int __cmd_record(struct record *rec, int argc, const char **argv)
> >  		goto out_child;
> >  	}
> >  
> > -	machine = &session->machines.host;
> > +	machine = &session->machines->host;
> >  
> >  	if (file->is_pipe) {
> >  		err = perf_event__synthesize_attrs(tool, session,
> > @@ -735,7 +735,7 @@ static int __cmd_record(struct record *rec, int argc, const char **argv)
> >  		       "Check /proc/modules permission or run as root.\n");
> >  
> >  	if (perf_guest) {
> > -		machines__process_guests(&session->machines,
> > +		machines__process_guests(session->machines,
> >  					 perf_event__synthesize_guest_os, tool);
> >  	}
> >  
> > diff --git a/tools/perf/builtin-report.c b/tools/perf/builtin-report.c
> > index 5e53eee5a9a7..650d78ad3357 100644
> > --- a/tools/perf/builtin-report.c
> > +++ b/tools/perf/builtin-report.c
> > @@ -353,7 +353,7 @@ static int perf_evlist__tty_browse_hists(struct perf_evlist *evlist,
> >  
> >  static void report__warn_kptr_restrict(const struct report *rep)
> >  {
> > -	struct map *kernel_map = rep->session->machines.host.vmlinux_maps[MAP__FUNCTION];
> > +	struct map *kernel_map = rep->session->machines->host.vmlinux_maps[MAP__FUNCTION];
> >  	struct kmap *kernel_kmap = kernel_map ? map__kmap(kernel_map) : NULL;
> >  
> >  	if (kernel_map == NULL ||
> > @@ -845,7 +845,7 @@ int cmd_report(int argc, const char **argv, const char *prefix __maybe_unused)
> >  	 */
> >  	if (ui__has_annotation()) {
> >  		symbol_conf.priv_size = sizeof(struct annotation);
> > -		machines__set_symbol_filter(&session->machines,
> > +		machines__set_symbol_filter(session->machines,
> >  					    symbol__annotate_init);
> >  		/*
> >   		 * For searching by name on the "Browse map details".
> > diff --git a/tools/perf/builtin-top.c b/tools/perf/builtin-top.c
> > index ea6e7bd04f9a..0ab663e200ed 100644
> > --- a/tools/perf/builtin-top.c
> > +++ b/tools/perf/builtin-top.c
> > @@ -828,13 +828,13 @@ static void perf_top__mmap_read_idx(struct perf_top *top, int idx)
> >  			++top->us_samples;
> >  			if (top->hide_user_symbols)
> >  				goto next_event;
> > -			machine = &session->machines.host;
> > +			machine = &session->machines->host;
> >  			break;
> >  		case PERF_RECORD_MISC_KERNEL:
> >  			++top->kernel_samples;
> >  			if (top->hide_kernel_symbols)
> >  				goto next_event;
> > -			machine = &session->machines.host;
> > +			machine = &session->machines->host;
> >  			break;
> >  		case PERF_RECORD_MISC_GUEST_KERNEL:
> >  			++top->guest_kernel_samples;
> > @@ -939,7 +939,7 @@ static int __cmd_top(struct perf_top *top)
> >  	if (top->session == NULL)
> >  		return -1;
> >  
> > -	machines__set_symbol_filter(&top->session->machines, symbol_filter);
> > +	machines__set_symbol_filter(top->session->machines, symbol_filter);
> >  
> >  	if (!objdump_path) {
> >  		ret = perf_session_env__lookup_objdump(&top->session->header.env);
> > @@ -951,7 +951,7 @@ static int __cmd_top(struct perf_top *top)
> >  	if (ret)
> >  		goto out_delete;
> >  
> > -	machine__synthesize_threads(&top->session->machines.host, &opts->target,
> > +	machine__synthesize_threads(&top->session->machines->host, &opts->target,
> >  				    top->evlist->threads, false);
> >  	ret = perf_top__start_counters(top);
> >  	if (ret)
> > diff --git a/tools/perf/builtin-trace.c b/tools/perf/builtin-trace.c
> > index a05490d06374..f3df6fa596e9 100644
> > --- a/tools/perf/builtin-trace.c
> > +++ b/tools/perf/builtin-trace.c
> > @@ -2418,7 +2418,7 @@ static int trace__replay(struct trace *trace)
> >  	if (symbol__init(&session->header.env) < 0)
> >  		goto out;
> >  
> > -	trace->host = &session->machines.host;
> > +	trace->host = &session->machines->host;
> >  
> >  	err = perf_session__set_tracepoints_handlers(session, handlers);
> >  	if (err)
> > diff --git a/tools/perf/util/build-id.c b/tools/perf/util/build-id.c
> > index ad8cfcbaa25d..8720f71a96dd 100644
> > --- a/tools/perf/util/build-id.c
> > +++ b/tools/perf/util/build-id.c
> > @@ -221,12 +221,12 @@ static int machine__write_buildid_table(struct machine *machine, int fd)
> >  int perf_session__write_buildid_table(struct perf_session *session, int fd)
> >  {
> >  	struct rb_node *nd;
> > -	int err = machine__write_buildid_table(&session->machines.host, fd);
> > +	int err = machine__write_buildid_table(&session->machines->host, fd);
> >  
> >  	if (err)
> >  		return err;
> >  
> > -	for (nd = rb_first(&session->machines.guests); nd; nd = rb_next(nd)) {
> > +	for (nd = rb_first(&session->machines->guests); nd; nd = rb_next(nd)) {
> >  		struct machine *pos = rb_entry(nd, struct machine, rb_node);
> >  		err = machine__write_buildid_table(pos, fd);
> >  		if (err)
> > @@ -261,11 +261,11 @@ int dsos__hit_all(struct perf_session *session)
> >  	struct rb_node *nd;
> >  	int err;
> >  
> > -	err = machine__hit_all_dsos(&session->machines.host);
> > +	err = machine__hit_all_dsos(&session->machines->host);
> >  	if (err)
> >  		return err;
> >  
> > -	for (nd = rb_first(&session->machines.guests); nd; nd = rb_next(nd)) {
> > +	for (nd = rb_first(&session->machines->guests); nd; nd = rb_next(nd)) {
> >  		struct machine *pos = rb_entry(nd, struct machine, rb_node);
> >  
> >  		err = machine__hit_all_dsos(pos);
> > @@ -509,9 +509,9 @@ int perf_session__cache_build_ids(struct perf_session *session)
> >  	if (mkdir(buildid_dir, 0755) != 0 && errno != EEXIST)
> >  		return -1;
> >  
> > -	ret = machine__cache_build_ids(&session->machines.host);
> > +	ret = machine__cache_build_ids(&session->machines->host);
> >  
> > -	for (nd = rb_first(&session->machines.guests); nd; nd = rb_next(nd)) {
> > +	for (nd = rb_first(&session->machines->guests); nd; nd = rb_next(nd)) {
> >  		struct machine *pos = rb_entry(nd, struct machine, rb_node);
> >  		ret |= machine__cache_build_ids(pos);
> >  	}
> > @@ -530,9 +530,9 @@ static bool machine__read_build_ids(struct machine *machine, bool with_hits)
> >  bool perf_session__read_build_ids(struct perf_session *session, bool with_hits)
> >  {
> >  	struct rb_node *nd;
> > -	bool ret = machine__read_build_ids(&session->machines.host, with_hits);
> > +	bool ret = machine__read_build_ids(&session->machines->host, with_hits);
> >  
> > -	for (nd = rb_first(&session->machines.guests); nd; nd = rb_next(nd)) {
> > +	for (nd = rb_first(&session->machines->guests); nd; nd = rb_next(nd)) {
> >  		struct machine *pos = rb_entry(nd, struct machine, rb_node);
> >  		ret |= machine__read_build_ids(pos, with_hits);
> >  	}
> > diff --git a/tools/perf/util/session.c b/tools/perf/util/session.c
> > index 0d080a95d2ff..955bf5368751 100644
> > --- a/tools/perf/util/session.c
> > +++ b/tools/perf/util/session.c
> > @@ -58,21 +58,21 @@ void perf_session__set_id_hdr_size(struct perf_session *session)
> >  {
> >  	u16 id_hdr_size = perf_evlist__id_hdr_size(session->evlist);
> >  
> > -	machines__set_id_hdr_size(&session->machines, id_hdr_size);
> > +	machines__set_id_hdr_size(session->machines, id_hdr_size);
> >  }
> >  
> >  int perf_session__create_kernel_maps(struct perf_session *session)
> >  {
> > -	int ret = machine__create_kernel_maps(&session->machines.host);
> > +	int ret = machine__create_kernel_maps(&session->machines->host);
> >  
> >  	if (ret >= 0)
> > -		ret = machines__create_guest_kernel_maps(&session->machines);
> > +		ret = machines__create_guest_kernel_maps(session->machines);
> >  	return ret;
> >  }
> >  
> >  static void perf_session__destroy_kernel_maps(struct perf_session *session)
> >  {
> > -	machines__destroy_kernel_maps(&session->machines);
> > +	machines__destroy_kernel_maps(session->machines);
> >  }
> >  
> >  static bool perf_session__has_comm_exec(struct perf_session *session)
> > @@ -91,7 +91,7 @@ static void perf_session__set_comm_exec(struct perf_session *session)
> >  {
> >  	bool comm_exec = perf_session__has_comm_exec(session);
> >  
> > -	machines__set_comm_exec(&session->machines, comm_exec);
> > +	machines__set_comm_exec(session->machines, comm_exec);
> >  }
> >  
> >  static int ordered_events__deliver_event(struct ordered_events *oe,
> > @@ -123,7 +123,12 @@ struct perf_session *perf_session__new(struct perf_data_file *file,
> >  	session->repipe = repipe;
> >  	session->tool   = tool;
> >  	INIT_LIST_HEAD(&session->auxtrace_index);
> > -	machines__init(&session->machines);
> > +
> > +	session->machines = malloc(sizeof(*session->machines));
> > +	if (!session->machines)
> > +		goto out_delete;
> > +
> > +	machines__init(session->machines);
> >  	ordered_events__init(&session->ordered_events, ordered_events__deliver_event);
> >  
> >  	if (file) {
> > @@ -168,7 +173,7 @@ struct perf_session *perf_session__new(struct perf_data_file *file,
> >  
> >  static void perf_session__delete_threads(struct perf_session *session)
> >  {
> > -	machine__delete_threads(&session->machines.host);
> > +	machine__delete_threads(&session->machines->host);
> >  }
> >  
> >  static void perf_session_env__delete(struct perf_session_env *env)
> > @@ -194,7 +199,8 @@ void perf_session__delete(struct perf_session *session)
> >  	perf_session__destroy_kernel_maps(session);
> >  	perf_session__delete_threads(session);
> >  	perf_session_env__delete(&session->header.env);
> > -	machines__exit(&session->machines);
> > +	machines__exit(session->machines);
> > +	free(session->machines);
> >  	if (session->file)
> >  		perf_data_file__close(session->file);
> >  	free(session->header.index);
> > @@ -1088,7 +1094,7 @@ static int perf_session__deliver_event(struct perf_session *session,
> >  	if (ret > 0)
> >  		return 0;
> >  
> > -	return machines__deliver_event(&session->machines, stats,
> > +	return machines__deliver_event(session->machines, stats,
> >  				       session->evlist, event, sample,
> >  				       tool, file_offset);
> >  }
> > @@ -1155,7 +1161,7 @@ int perf_session__deliver_synth_event(struct perf_session *session,
> >  	if (event->header.type >= PERF_RECORD_USER_TYPE_START)
> >  		return perf_session__process_user_event(session, event, 0);
> >  
> > -	return machines__deliver_event(&session->machines, &evlist->stats,
> > +	return machines__deliver_event(session->machines, &evlist->stats,
> >  				       evlist, event, sample, tool, 0);
> >  }
> >  
> > @@ -1270,14 +1276,14 @@ void perf_event_header__bswap(struct perf_event_header *hdr)
> >  
> >  struct thread *perf_session__findnew(struct perf_session *session, pid_t pid)
> >  {
> > -	return machine__findnew_thread(&session->machines.host, -1, pid);
> > +	return machine__findnew_thread(&session->machines->host, -1, pid);
> >  }
> >  
> >  static struct thread *perf_session__register_idle_thread(struct perf_session *session)
> >  {
> >  	struct thread *thread;
> >  
> > -	thread = machine__findnew_thread(&session->machines.host, 0, 0);
> > +	thread = machine__findnew_thread(&session->machines->host, 0, 0);
> >  	if (thread == NULL || thread__set_comm(thread, "swapper", 0)) {
> >  		pr_err("problem inserting idle task.\n");
> >  		thread = NULL;
> > @@ -1692,13 +1698,13 @@ int maps__set_kallsyms_ref_reloc_sym(struct map **maps,
> >  
> >  size_t perf_session__fprintf_dsos(struct perf_session *session, FILE *fp)
> >  {
> > -	return machines__fprintf_dsos(&session->machines, fp);
> > +	return machines__fprintf_dsos(session->machines, fp);
> >  }
> >  
> >  size_t perf_session__fprintf_dsos_buildid(struct perf_session *session, FILE *fp,
> >  					  bool (skip)(struct dso *dso, int parm), int parm)
> >  {
> > -	return machines__fprintf_dsos_buildid(&session->machines, fp, skip, parm);
> > +	return machines__fprintf_dsos_buildid(session->machines, fp, skip, parm);
> >  }
> >  
> >  size_t perf_session__fprintf_nr_events(struct perf_session *session, FILE *fp)
> > @@ -1721,7 +1727,7 @@ size_t perf_session__fprintf(struct perf_session *session, FILE *fp)
> >  	 * FIXME: Here we have to actually print all the machines in this
> >  	 * session, not just the host...
> >  	 */
> > -	return machine__fprintf(&session->machines.host, fp);
> > +	return machine__fprintf(&session->machines->host, fp);
> >  }
> >  
> >  struct perf_evsel *perf_session__find_first_evtype(struct perf_session *session,
> > diff --git a/tools/perf/util/session.h b/tools/perf/util/session.h
> > index b44afc75d1cc..ac1796c7c799 100644
> > --- a/tools/perf/util/session.h
> > +++ b/tools/perf/util/session.h
> > @@ -20,7 +20,7 @@ struct itrace_synth_opts;
> >  
> >  struct perf_session {
> >  	struct perf_header	header;
> > -	struct machines		machines;
> > +	struct machines		*machines;
> >  	struct perf_evlist	*evlist;
> >  	struct auxtrace		*auxtrace;
> >  	struct itrace_synth_opts *itrace_synth_opts;
> > @@ -79,13 +79,13 @@ void perf_session__set_id_hdr_size(struct perf_session *session);
> >  static inline
> >  struct machine *perf_session__find_machine(struct perf_session *session, pid_t pid)
> >  {
> > -	return machines__find(&session->machines, pid);
> > +	return machines__find(session->machines, pid);
> >  }
> >  
> >  static inline
> >  struct machine *perf_session__findnew_machine(struct perf_session *session, pid_t pid)
> >  {
> > -	return machines__findnew(&session->machines, pid);
> > +	return machines__findnew(session->machines, pid);
> >  }
> >  
> >  struct thread *perf_session__findnew(struct perf_session *session, pid_t pid);
> > -- 
> > 2.4.0
> --
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/

^ permalink raw reply	[flat|nested] 221+ messages in thread

* Re: [PATCH 35/40] perf record: Synthesize COMM event for a command line workload
  2015-05-18 12:45   ` Arnaldo Carvalho de Melo
@ 2015-05-19  7:46     ` Namhyung Kim
  2015-05-19 14:02       ` Arnaldo Carvalho de Melo
  0 siblings, 1 reply; 221+ messages in thread
From: Namhyung Kim @ 2015-05-19  7:46 UTC (permalink / raw)
  To: Arnaldo Carvalho de Melo
  Cc: Ingo Molnar, Peter Zijlstra, Jiri Olsa, LKML, David Ahern,
	Adrian Hunter, Andi Kleen, Frederic Weisbecker, Stephane Eranian

On Mon, May 18, 2015 at 09:45:35AM -0300, Arnaldo Carvalho de Melo wrote:
> Em Mon, May 18, 2015 at 09:30:50AM +0900, Namhyung Kim escreveu:
> > When perf creates a new child to profile, the events are enabled on
> > exec().  And in this case, it doesn't synthesize any event for the
> > child since they'll be generated during exec().  But there's an window
> > between the enabling and the event generation.
> > 
> > It used to be overcome since samples are only in kernel (so we always
> > have the map) and the comm is overridden by a later COMM event.
> > However it won't work anymore since those samples will go to a missing
> > thread now but the COMM event will create a (current) thread.  This
> > leads to those early samples (like native_write_msr_safe) not having a
> > comm but pid (like ':15328').
>  
> > So it needs to synthesize COMM event for the child explicitly before
> > enabling so that it can have a correct comm.  But at this time, the
> > comm will be "perf" since it's not exec-ed yet.
> 
> This looks reasonable, but I think it probably needs to be done
> somewhere in perf_evlist__prepare_workload() or
> perf_evlist__start_workload(), as this affects other tools as well, like
> 'top', 'trace' and any other that may want to do this start-workload use
> case.

Hmm.. I need to look at this again as it only affects on processing
indexed data files which used to have a separate missing threads tree.
That's the reason why I didn't put it in a generic place like you
said.

However I changed not to use the separate tree - the purpose of the
tree was to reduce lock acquisition on thread searching but it already
grabs a rwlock with thread refcounting change.

Will check whether this is still needed..

Thanks,
Namhyung


> 
> I also wonder if we can't overcome this without using /proc, i.e.
> actually moving the "start the workload" to just before the fork, so
> that the kernel covers that as well.
> 
> Or, alternatively, the thread can be created without having to look at
> /proc at all, but by directly creating the struct thread, with the
> correct COMM, pid, etc, that we know, since we forked it, etc.
> 
> - Arnaldo
>  
> > Signed-off-by: Namhyung Kim <namhyung@kernel.org>
> > ---
> >  tools/perf/builtin-record.c | 18 +++++++++++++++++-
> >  tools/perf/util/event.c     |  2 +-
> >  tools/perf/util/event.h     |  5 +++++
> >  3 files changed, 23 insertions(+), 2 deletions(-)
> > 
> > diff --git a/tools/perf/builtin-record.c b/tools/perf/builtin-record.c
> > index 978ebf648aab..153f38e35555 100644
> > --- a/tools/perf/builtin-record.c
> > +++ b/tools/perf/builtin-record.c
> > @@ -766,8 +766,24 @@ static int __cmd_record(struct record *rec, int argc, const char **argv)
> >  	/*
> >  	 * Let the child rip
> >  	 */
> > -	if (forks)
> > +	if (forks) {
> > +		union perf_event *comm_event;
> > +
> > +		comm_event = malloc(sizeof(*comm_event) + machine->id_hdr_size);
> > +		if (comm_event == NULL)
> > +			goto out_child;
> > +
> > +		err = perf_event__synthesize_comm(tool, comm_event,
> > +						  rec->evlist->workload.pid,
> > +						  process_synthesized_event,
> > +						  machine);
> > +		free(comm_event);
> > +
> > +		if (err < 0)
> > +			goto out_child;
> > +
> >  		perf_evlist__start_workload(rec->evlist);
> > +	}
> >  
> >  	if (opts->initial_delay) {
> >  		usleep(opts->initial_delay * 1000);
> > diff --git a/tools/perf/util/event.c b/tools/perf/util/event.c
> > index 930d45d5a37a..3adca1302150 100644
> > --- a/tools/perf/util/event.c
> > +++ b/tools/perf/util/event.c
> > @@ -165,7 +165,7 @@ static int perf_event__prepare_comm(union perf_event *event, pid_t pid,
> >  	return 0;
> >  }
> >  
> > -static pid_t perf_event__synthesize_comm(struct perf_tool *tool,
> > +pid_t perf_event__synthesize_comm(struct perf_tool *tool,
> >  					 union perf_event *event, pid_t pid,
> >  					 perf_event__handler_t process,
> >  					 struct machine *machine)
> > diff --git a/tools/perf/util/event.h b/tools/perf/util/event.h
> > index 40e02544f861..9aacd558ac0f 100644
> > --- a/tools/perf/util/event.h
> > +++ b/tools/perf/util/event.h
> > @@ -452,6 +452,11 @@ int perf_event__synthesize_mmap_events(struct perf_tool *tool,
> >  				       struct machine *machine,
> >  				       bool mmap_data);
> >  
> > +pid_t perf_event__synthesize_comm(struct perf_tool *tool,
> > +				  union perf_event *event, pid_t pid,
> > +				  perf_event__handler_t process,
> > +				  struct machine *machine);
> > +
> >  size_t perf_event__fprintf_comm(union perf_event *event, FILE *fp);
> >  size_t perf_event__fprintf_mmap(union perf_event *event, FILE *fp);
> >  size_t perf_event__fprintf_mmap2(union perf_event *event, FILE *fp);
> > -- 
> > 2.4.0
> --
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/

^ permalink raw reply	[flat|nested] 221+ messages in thread

* [PATCH] perf tools: Reducing arguments of hist_entry_iter__add()
  2015-05-19  7:01     ` Namhyung Kim
@ 2015-05-19  8:04       ` Namhyung Kim
  2015-05-19 14:03         ` Arnaldo Carvalho de Melo
  2015-05-27 16:47         ` [tip:perf/core] perf hists: " tip-bot for Namhyung Kim
  0 siblings, 2 replies; 221+ messages in thread
From: Namhyung Kim @ 2015-05-19  8:04 UTC (permalink / raw)
  To: Arnaldo Carvalho de Melo
  Cc: Ingo Molnar, Peter Zijlstra, Jiri Olsa, LKML, David Ahern

The evsel and sample arguments are to set iter for later use.  As it
also receives an iter as another argument, just set them before
calling the function.

Signed-off-by: Namhyung Kim <namhyung@kernel.org>
---
Arnaldo, it seems that this patch can be applied to the current perf/core
2d8e405acd78 ("perf bench numa: Share sched_getcpu() __weak def with
cloexec.c") cleanly.  If you see any conflict, please let me know.

 tools/perf/builtin-report.c       | 9 +++++----
 tools/perf/builtin-top.c          | 7 ++++---
 tools/perf/tests/hists_cumulate.c | 6 ++++--
 tools/perf/tests/hists_filter.c   | 4 +++-
 tools/perf/tests/hists_output.c   | 6 ++++--
 tools/perf/util/hist.c            | 8 ++------
 tools/perf/util/hist.h            | 1 -
 7 files changed, 22 insertions(+), 19 deletions(-)

diff --git a/tools/perf/builtin-report.c b/tools/perf/builtin-report.c
index 92fca2157e5e..56025d90622f 100644
--- a/tools/perf/builtin-report.c
+++ b/tools/perf/builtin-report.c
@@ -139,8 +139,10 @@ static int process_sample_event(struct perf_tool *tool,
 	struct report *rep = container_of(tool, struct report, tool);
 	struct addr_location al;
 	struct hist_entry_iter iter = {
-		.hide_unresolved = rep->hide_unresolved,
-		.add_entry_cb = hist_iter__report_callback,
+		.evsel 			= evsel,
+		.sample 		= sample,
+		.hide_unresolved 	= rep->hide_unresolved,
+		.add_entry_cb 		= hist_iter__report_callback,
 	};
 	int ret = 0;
 
@@ -168,8 +170,7 @@ static int process_sample_event(struct perf_tool *tool,
 	if (al.map != NULL)
 		al.map->dso->hit = 1;
 
-	ret = hist_entry_iter__add(&iter, &al, evsel, sample, rep->max_stack,
-				   rep);
+	ret = hist_entry_iter__add(&iter, &al, rep->max_stack, rep);
 	if (ret < 0)
 		pr_debug("problem adding hist entry, skipping event\n");
 out_put:
diff --git a/tools/perf/builtin-top.c b/tools/perf/builtin-top.c
index a19351728f0f..6b987424d015 100644
--- a/tools/perf/builtin-top.c
+++ b/tools/perf/builtin-top.c
@@ -775,7 +775,9 @@ static void perf_event__process_sample(struct perf_tool *tool,
 	if (al.sym == NULL || !al.sym->ignore) {
 		struct hists *hists = evsel__hists(evsel);
 		struct hist_entry_iter iter = {
-			.add_entry_cb = hist_iter__top_callback,
+			.evsel		= evsel,
+			.sample 	= sample,
+			.add_entry_cb 	= hist_iter__top_callback,
 		};
 
 		if (symbol_conf.cumulate_callchain)
@@ -785,8 +787,7 @@ static void perf_event__process_sample(struct perf_tool *tool,
 
 		pthread_mutex_lock(&hists->lock);
 
-		err = hist_entry_iter__add(&iter, &al, evsel, sample,
-					   top->max_stack, top);
+		err = hist_entry_iter__add(&iter, &al, top->max_stack, top);
 		if (err < 0)
 			pr_err("Problem incrementing symbol period, skipping event\n");
 
diff --git a/tools/perf/tests/hists_cumulate.c b/tools/perf/tests/hists_cumulate.c
index 620f626e5b35..7d82c8be5e36 100644
--- a/tools/perf/tests/hists_cumulate.c
+++ b/tools/perf/tests/hists_cumulate.c
@@ -87,6 +87,8 @@ static int add_hist_entries(struct hists *hists, struct machine *machine)
 			},
 		};
 		struct hist_entry_iter iter = {
+			.evsel = evsel,
+			.sample	= &sample,
 			.hide_unresolved = false,
 		};
 
@@ -104,8 +106,8 @@ static int add_hist_entries(struct hists *hists, struct machine *machine)
 						  &sample) < 0)
 			goto out;
 
-		if (hist_entry_iter__add(&iter, &al, evsel, &sample,
-					 PERF_MAX_STACK_DEPTH, NULL) < 0) {
+		if (hist_entry_iter__add(&iter, &al, PERF_MAX_STACK_DEPTH,
+					 NULL) < 0) {
 			addr_location__put(&al);
 			goto out;
 		}
diff --git a/tools/perf/tests/hists_filter.c b/tools/perf/tests/hists_filter.c
index 82e1ee52e024..ce48775e6ada 100644
--- a/tools/perf/tests/hists_filter.c
+++ b/tools/perf/tests/hists_filter.c
@@ -63,6 +63,8 @@ static int add_hist_entries(struct perf_evlist *evlist,
 				},
 			};
 			struct hist_entry_iter iter = {
+				.evsel = evsel,
+				.sample = &sample,
 				.ops = &hist_iter_normal,
 				.hide_unresolved = false,
 			};
@@ -81,7 +83,7 @@ static int add_hist_entries(struct perf_evlist *evlist,
 							  &sample) < 0)
 				goto out;
 
-			if (hist_entry_iter__add(&iter, &al, evsel, &sample,
+			if (hist_entry_iter__add(&iter, &al,
 						 PERF_MAX_STACK_DEPTH, NULL) < 0) {
 				addr_location__put(&al);
 				goto out;
diff --git a/tools/perf/tests/hists_output.c b/tools/perf/tests/hists_output.c
index fd7ec4f9aeb4..adbebc852cc8 100644
--- a/tools/perf/tests/hists_output.c
+++ b/tools/perf/tests/hists_output.c
@@ -57,6 +57,8 @@ static int add_hist_entries(struct hists *hists, struct machine *machine)
 			},
 		};
 		struct hist_entry_iter iter = {
+			.evsel = evsel,
+			.sample = &sample,
 			.ops = &hist_iter_normal,
 			.hide_unresolved = false,
 		};
@@ -70,8 +72,8 @@ static int add_hist_entries(struct hists *hists, struct machine *machine)
 						  &sample) < 0)
 			goto out;
 
-		if (hist_entry_iter__add(&iter, &al, evsel, &sample,
-					 PERF_MAX_STACK_DEPTH, NULL) < 0) {
+		if (hist_entry_iter__add(&iter, &al, PERF_MAX_STACK_DEPTH,
+					 NULL) < 0) {
 			addr_location__put(&al);
 			goto out;
 		}
diff --git a/tools/perf/util/hist.c b/tools/perf/util/hist.c
index 338770679863..2504b5b1a308 100644
--- a/tools/perf/util/hist.c
+++ b/tools/perf/util/hist.c
@@ -851,19 +851,15 @@ const struct hist_iter_ops hist_iter_cumulative = {
 };
 
 int hist_entry_iter__add(struct hist_entry_iter *iter, struct addr_location *al,
-			 struct perf_evsel *evsel, struct perf_sample *sample,
 			 int max_stack_depth, void *arg)
 {
 	int err, err2;
 
-	err = sample__resolve_callchain(sample, &iter->parent, evsel, al,
-					max_stack_depth);
+	err = sample__resolve_callchain(iter->sample, &iter->parent,
+					iter->evsel, al, max_stack_depth);
 	if (err)
 		return err;
 
-	iter->evsel = evsel;
-	iter->sample = sample;
-
 	err = iter->ops->prepare_entry(iter, al);
 	if (err)
 		goto out;
diff --git a/tools/perf/util/hist.h b/tools/perf/util/hist.h
index 9f31b89a527a..5ed8d9c22981 100644
--- a/tools/perf/util/hist.h
+++ b/tools/perf/util/hist.h
@@ -111,7 +111,6 @@ struct hist_entry *__hists__add_entry(struct hists *hists,
 				      u64 weight, u64 transaction,
 				      bool sample_self);
 int hist_entry_iter__add(struct hist_entry_iter *iter, struct addr_location *al,
-			 struct perf_evsel *evsel, struct perf_sample *sample,
 			 int max_stack_depth, void *arg);
 
 int64_t hist_entry__cmp(struct hist_entry *left, struct hist_entry *right);
-- 
2.4.0


^ permalink raw reply related	[flat|nested] 221+ messages in thread

* Re: [PATCH 34/40] perf report: Parallelize perf report using multi-thread
  2015-05-18  0:30 ` [PATCH 34/40] perf report: Parallelize perf report using multi-thread Namhyung Kim
@ 2015-05-19 10:12   ` Jiri Olsa
  2015-05-19 14:58     ` Namhyung Kim
  0 siblings, 1 reply; 221+ messages in thread
From: Jiri Olsa @ 2015-05-19 10:12 UTC (permalink / raw)
  To: Namhyung Kim
  Cc: Arnaldo Carvalho de Melo, Ingo Molnar, Peter Zijlstra, LKML,
	David Ahern, Adrian Hunter, Andi Kleen, Frederic Weisbecker,
	Stephane Eranian

On Mon, May 18, 2015 at 09:30:49AM +0900, Namhyung Kim wrote:

SNIP

> +
> +int perf_session__process_events_mt(struct perf_session *session, void *arg)
> +{
> +	struct perf_data_file *file = session->file;
> +	struct perf_evlist *evlist = session->evlist;
> +	struct perf_evsel *evsel;
> +	u64 nr_entries = 0;
> +	struct perf_tool *tool = session->tool;
> +	struct perf_session *mt_sessions = NULL;
> +	struct perf_tool_mt *mt_tools = NULL;
> +	struct perf_session *ms;
> +	struct perf_tool_mt *mt;
> +	pthread_t *th_id;
> +	int err, i, k;
> +	int nr_index = session->header.nr_index;
> +	u64 size = perf_data_file__size(file);
> +

SNIP

> +
> +	for (i = 1; i < nr_index; i++) {
> +		mt = &mt_tools[i];
> +
> +		evlist__for_each(evlist, evsel) {
> +			struct hists *hists = evsel__hists(evsel);
> +
> +			if (perf_evsel__is_dummy_tracking(evsel))
> +				continue;
> +
> +			hists__mt_resort(hists, &mt->hists[evsel->idx]);
> +
> +			/* Non-group events are considered as leader */
> +			if (symbol_conf.event_group &&
> +			    !perf_evsel__is_group_leader(evsel)) {
> +				struct hists *leader_hists;
> +
> +				leader_hists = evsel__hists(evsel->leader);
> +				hists__match(leader_hists, hists);
> +				hists__link(leader_hists, hists);

hum, could you please describe/comment on why is this needed?

thanks,
jirka

^ permalink raw reply	[flat|nested] 221+ messages in thread

* Re: [PATCH 35/40] perf record: Synthesize COMM event for a command line workload
  2015-05-19  7:46     ` Namhyung Kim
@ 2015-05-19 14:02       ` Arnaldo Carvalho de Melo
  2015-05-19 15:12         ` Namhyung Kim
  0 siblings, 1 reply; 221+ messages in thread
From: Arnaldo Carvalho de Melo @ 2015-05-19 14:02 UTC (permalink / raw)
  To: Namhyung Kim
  Cc: Ingo Molnar, Peter Zijlstra, Jiri Olsa, LKML, David Ahern,
	Adrian Hunter, Andi Kleen, Frederic Weisbecker, Stephane Eranian

Em Tue, May 19, 2015 at 04:46:43PM +0900, Namhyung Kim escreveu:
> On Mon, May 18, 2015 at 09:45:35AM -0300, Arnaldo Carvalho de Melo wrote:
> > Em Mon, May 18, 2015 at 09:30:50AM +0900, Namhyung Kim escreveu:
> > > When perf creates a new child to profile, the events are enabled on
> > > exec().  And in this case, it doesn't synthesize any event for the
> > > child since they'll be generated during exec().  But there's an window
> > > between the enabling and the event generation.
> > > 
> > > It used to be overcome since samples are only in kernel (so we always
> > > have the map) and the comm is overridden by a later COMM event.
> > > However it won't work anymore since those samples will go to a missing
> > > thread now but the COMM event will create a (current) thread.  This
> > > leads to those early samples (like native_write_msr_safe) not having a
> > > comm but pid (like ':15328').
> >  
> > > So it needs to synthesize COMM event for the child explicitly before
> > > enabling so that it can have a correct comm.  But at this time, the
> > > comm will be "perf" since it's not exec-ed yet.
> > 
> > This looks reasonable, but I think it probably needs to be done
> > somewhere in perf_evlist__prepare_workload() or
> > perf_evlist__start_workload(), as this affects other tools as well, like
> > 'top', 'trace' and any other that may want to do this start-workload use
> > case.
> 
> Hmm.. I need to look at this again as it only affects on processing
> indexed data files which used to have a separate missing threads tree.

Humm, you're thinking about where you managed to reproduce the problem,
I am thinking outside indexing, etc, i.e. by definition we either enable
the event before we fork, so that we get the PERF_RECORD_FORK/COMM or we
synthesize it either from /proc or directly (preferred) if we decide to
do it after the fork/exec, right?

- Arnaldo

> That's the reason why I didn't put it in a generic place like you
> said.
> 
> However I changed not to use the separate tree - the purpose of the
> tree was to reduce lock acquisition on thread searching but it already
> grabs a rwlock with thread refcounting change.
> 
> Will check whether this is still needed..
> 
> Thanks,
> Namhyung
> 
> 
> > 
> > I also wonder if we can't overcome this without using /proc, i.e.
> > actually moving the "start the workload" to just before the fork, so
> > that the kernel covers that as well.
> > 
> > Or, alternatively, the thread can be created without having to look at
> > /proc at all, but by directly creating the struct thread, with the
> > correct COMM, pid, etc, that we know, since we forked it, etc.
> > 
> > - Arnaldo
> >  
> > > Signed-off-by: Namhyung Kim <namhyung@kernel.org>
> > > ---
> > >  tools/perf/builtin-record.c | 18 +++++++++++++++++-
> > >  tools/perf/util/event.c     |  2 +-
> > >  tools/perf/util/event.h     |  5 +++++
> > >  3 files changed, 23 insertions(+), 2 deletions(-)
> > > 
> > > diff --git a/tools/perf/builtin-record.c b/tools/perf/builtin-record.c
> > > index 978ebf648aab..153f38e35555 100644
> > > --- a/tools/perf/builtin-record.c
> > > +++ b/tools/perf/builtin-record.c
> > > @@ -766,8 +766,24 @@ static int __cmd_record(struct record *rec, int argc, const char **argv)
> > >  	/*
> > >  	 * Let the child rip
> > >  	 */
> > > -	if (forks)
> > > +	if (forks) {
> > > +		union perf_event *comm_event;
> > > +
> > > +		comm_event = malloc(sizeof(*comm_event) + machine->id_hdr_size);
> > > +		if (comm_event == NULL)
> > > +			goto out_child;
> > > +
> > > +		err = perf_event__synthesize_comm(tool, comm_event,
> > > +						  rec->evlist->workload.pid,
> > > +						  process_synthesized_event,
> > > +						  machine);
> > > +		free(comm_event);
> > > +
> > > +		if (err < 0)
> > > +			goto out_child;
> > > +
> > >  		perf_evlist__start_workload(rec->evlist);
> > > +	}
> > >  
> > >  	if (opts->initial_delay) {
> > >  		usleep(opts->initial_delay * 1000);
> > > diff --git a/tools/perf/util/event.c b/tools/perf/util/event.c
> > > index 930d45d5a37a..3adca1302150 100644
> > > --- a/tools/perf/util/event.c
> > > +++ b/tools/perf/util/event.c
> > > @@ -165,7 +165,7 @@ static int perf_event__prepare_comm(union perf_event *event, pid_t pid,
> > >  	return 0;
> > >  }
> > >  
> > > -static pid_t perf_event__synthesize_comm(struct perf_tool *tool,
> > > +pid_t perf_event__synthesize_comm(struct perf_tool *tool,
> > >  					 union perf_event *event, pid_t pid,
> > >  					 perf_event__handler_t process,
> > >  					 struct machine *machine)
> > > diff --git a/tools/perf/util/event.h b/tools/perf/util/event.h
> > > index 40e02544f861..9aacd558ac0f 100644
> > > --- a/tools/perf/util/event.h
> > > +++ b/tools/perf/util/event.h
> > > @@ -452,6 +452,11 @@ int perf_event__synthesize_mmap_events(struct perf_tool *tool,
> > >  				       struct machine *machine,
> > >  				       bool mmap_data);
> > >  
> > > +pid_t perf_event__synthesize_comm(struct perf_tool *tool,
> > > +				  union perf_event *event, pid_t pid,
> > > +				  perf_event__handler_t process,
> > > +				  struct machine *machine);
> > > +
> > >  size_t perf_event__fprintf_comm(union perf_event *event, FILE *fp);
> > >  size_t perf_event__fprintf_mmap(union perf_event *event, FILE *fp);
> > >  size_t perf_event__fprintf_mmap2(union perf_event *event, FILE *fp);
> > > -- 
> > > 2.4.0
> > --
> > To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> > the body of a message to majordomo@vger.kernel.org
> > More majordomo info at  http://vger.kernel.org/majordomo-info.html
> > Please read the FAQ at  http://www.tux.org/lkml/

^ permalink raw reply	[flat|nested] 221+ messages in thread

* Re: [PATCH] perf tools: Reducing arguments of hist_entry_iter__add()
  2015-05-19  8:04       ` [PATCH] " Namhyung Kim
@ 2015-05-19 14:03         ` Arnaldo Carvalho de Melo
  2015-05-27 16:47         ` [tip:perf/core] perf hists: " tip-bot for Namhyung Kim
  1 sibling, 0 replies; 221+ messages in thread
From: Arnaldo Carvalho de Melo @ 2015-05-19 14:03 UTC (permalink / raw)
  To: Namhyung Kim; +Cc: Ingo Molnar, Peter Zijlstra, Jiri Olsa, LKML, David Ahern

Em Tue, May 19, 2015 at 05:04:10PM +0900, Namhyung Kim escreveu:
> The evsel and sample arguments are to set iter for later use.  As it
> also receives an iter as another argument, just set them before
> calling the function.
> 
> Signed-off-by: Namhyung Kim <namhyung@kernel.org>
> ---
> Arnaldo, it seems that this patch can be applied to the current perf/core
> 2d8e405acd78 ("perf bench numa: Share sched_getcpu() __weak def with
> cloexec.c") cleanly.  If you see any conflict, please let me know.

Thanks, applied.

- Arnaldo

^ permalink raw reply	[flat|nested] 221+ messages in thread

* Re: [PATCH 34/40] perf report: Parallelize perf report using multi-thread
  2015-05-19 10:12   ` Jiri Olsa
@ 2015-05-19 14:58     ` Namhyung Kim
  0 siblings, 0 replies; 221+ messages in thread
From: Namhyung Kim @ 2015-05-19 14:58 UTC (permalink / raw)
  To: Jiri Olsa
  Cc: Arnaldo Carvalho de Melo, Ingo Molnar, Peter Zijlstra, LKML,
	David Ahern, Adrian Hunter, Andi Kleen, Frederic Weisbecker,
	Stephane Eranian

On Tue, May 19, 2015 at 12:12:30PM +0200, Jiri Olsa wrote:
> On Mon, May 18, 2015 at 09:30:49AM +0900, Namhyung Kim wrote:
> 
> SNIP
> 
> > +
> > +int perf_session__process_events_mt(struct perf_session *session, void *arg)
> > +{
> > +	struct perf_data_file *file = session->file;
> > +	struct perf_evlist *evlist = session->evlist;
> > +	struct perf_evsel *evsel;
> > +	u64 nr_entries = 0;
> > +	struct perf_tool *tool = session->tool;
> > +	struct perf_session *mt_sessions = NULL;
> > +	struct perf_tool_mt *mt_tools = NULL;
> > +	struct perf_session *ms;
> > +	struct perf_tool_mt *mt;
> > +	pthread_t *th_id;
> > +	int err, i, k;
> > +	int nr_index = session->header.nr_index;
> > +	u64 size = perf_data_file__size(file);
> > +
> 
> SNIP
> 
> > +
> > +	for (i = 1; i < nr_index; i++) {
> > +		mt = &mt_tools[i];
> > +
> > +		evlist__for_each(evlist, evsel) {
> > +			struct hists *hists = evsel__hists(evsel);
> > +
> > +			if (perf_evsel__is_dummy_tracking(evsel))
> > +				continue;
> > +
> > +			hists__mt_resort(hists, &mt->hists[evsel->idx]);
> > +
> > +			/* Non-group events are considered as leader */
> > +			if (symbol_conf.event_group &&
> > +			    !perf_evsel__is_group_leader(evsel)) {
> > +				struct hists *leader_hists;
> > +
> > +				leader_hists = evsel__hists(evsel->leader);
> > +				hists__match(leader_hists, hists);
> > +				hists__link(leader_hists, hists);
> 
> hum, could you please describe/comment on why is this needed?

This is because it skips report__collapse_hists() when multi-thread
processing is enabled.  For multi-thread, it needs to collect
per-thread hists into a global hists.  This step is very similar to
hists__collapse_resort() and called regardless of sort keys, so it
skips to call the report__collapse_hists().

But I think it should be called after collapsing all per-thread hists.
I'll move it out of the loop.

Thanks,
Namhyung

^ permalink raw reply	[flat|nested] 221+ messages in thread

* Re: [PATCH 35/40] perf record: Synthesize COMM event for a command line workload
  2015-05-19 14:02       ` Arnaldo Carvalho de Melo
@ 2015-05-19 15:12         ` Namhyung Kim
  2015-05-19 16:30           ` Arnaldo Carvalho de Melo
  2015-05-19 19:49           ` Jiri Olsa
  0 siblings, 2 replies; 221+ messages in thread
From: Namhyung Kim @ 2015-05-19 15:12 UTC (permalink / raw)
  To: Arnaldo Carvalho de Melo
  Cc: Ingo Molnar, Peter Zijlstra, Jiri Olsa, LKML, David Ahern,
	Adrian Hunter, Andi Kleen, Frederic Weisbecker, Stephane Eranian

On Tue, May 19, 2015 at 11:02:20AM -0300, Arnaldo Carvalho de Melo wrote:
> Em Tue, May 19, 2015 at 04:46:43PM +0900, Namhyung Kim escreveu:
> > On Mon, May 18, 2015 at 09:45:35AM -0300, Arnaldo Carvalho de Melo wrote:
> > > Em Mon, May 18, 2015 at 09:30:50AM +0900, Namhyung Kim escreveu:
> > > > When perf creates a new child to profile, the events are enabled on
> > > > exec().  And in this case, it doesn't synthesize any event for the
> > > > child since they'll be generated during exec().  But there's an window
> > > > between the enabling and the event generation.
> > > > 
> > > > It used to be overcome since samples are only in kernel (so we always
> > > > have the map) and the comm is overridden by a later COMM event.
> > > > However it won't work anymore since those samples will go to a missing
> > > > thread now but the COMM event will create a (current) thread.  This
> > > > leads to those early samples (like native_write_msr_safe) not having a
> > > > comm but pid (like ':15328').
> > >  
> > > > So it needs to synthesize COMM event for the child explicitly before
> > > > enabling so that it can have a correct comm.  But at this time, the
> > > > comm will be "perf" since it's not exec-ed yet.
> > > 
> > > This looks reasonable, but I think it probably needs to be done
> > > somewhere in perf_evlist__prepare_workload() or
> > > perf_evlist__start_workload(), as this affects other tools as well, like
> > > 'top', 'trace' and any other that may want to do this start-workload use
> > > case.
> > 
> > Hmm.. I need to look at this again as it only affects on processing
> > indexed data files which used to have a separate missing threads tree.
> 
> Humm, you're thinking about where you managed to reproduce the problem,
> I am thinking outside indexing, etc, i.e. by definition we either enable
> the event before we fork, so that we get the PERF_RECORD_FORK/COMM or we
> synthesize it either from /proc or directly (preferred) if we decide to
> do it after the fork/exec, right?

But as I said before, later COMM event will override thread->comm to a
proper string as long as it can find a matching thread.  So I think it
has no problem in the current code.

In the old version of this patchset (v3), indexing made it impossible
for COMM event to find a matching thread since it used to have a
separate tree for threads that have sampled before any FORK/COMM event
came.  I think it doesn't apply to the current version anymore, I
will check it tomorrow.

Thanks,
Namhyung


> 
> - Arnaldo
> 
> > That's the reason why I didn't put it in a generic place like you
> > said.
> > 
> > However I changed not to use the separate tree - the purpose of the
> > tree was to reduce lock acquisition on thread searching but it already
> > grabs a rwlock with thread refcounting change.
> > 
> > Will check whether this is still needed..
> > 
> > Thanks,
> > Namhyung
> > 
> > 
> > > 
> > > I also wonder if we can't overcome this without using /proc, i.e.
> > > actually moving the "start the workload" to just before the fork, so
> > > that the kernel covers that as well.
> > > 
> > > Or, alternatively, the thread can be created without having to look at
> > > /proc at all, but by directly creating the struct thread, with the
> > > correct COMM, pid, etc, that we know, since we forked it, etc.

^ permalink raw reply	[flat|nested] 221+ messages in thread

* Re: [PATCH 35/40] perf record: Synthesize COMM event for a command line workload
  2015-05-19 15:12         ` Namhyung Kim
@ 2015-05-19 16:30           ` Arnaldo Carvalho de Melo
  2015-05-19 19:49           ` Jiri Olsa
  1 sibling, 0 replies; 221+ messages in thread
From: Arnaldo Carvalho de Melo @ 2015-05-19 16:30 UTC (permalink / raw)
  To: Namhyung Kim
  Cc: Ingo Molnar, Peter Zijlstra, Jiri Olsa, LKML, David Ahern,
	Adrian Hunter, Andi Kleen, Frederic Weisbecker, Stephane Eranian

Em Wed, May 20, 2015 at 12:12:45AM +0900, Namhyung Kim escreveu:
> On Tue, May 19, 2015 at 11:02:20AM -0300, Arnaldo Carvalho de Melo wrote:
> > Em Tue, May 19, 2015 at 04:46:43PM +0900, Namhyung Kim escreveu:
> > > On Mon, May 18, 2015 at 09:45:35AM -0300, Arnaldo Carvalho de Melo wrote:
> > > > Em Mon, May 18, 2015 at 09:30:50AM +0900, Namhyung Kim escreveu:
> > > > > When perf creates a new child to profile, the events are enabled on
> > > > > exec().  And in this case, it doesn't synthesize any event for the
> > > > > child since they'll be generated during exec().  But there's an window
> > > > > between the enabling and the event generation.
> > > > > 
> > > > > It used to be overcome since samples are only in kernel (so we always
> > > > > have the map) and the comm is overridden by a later COMM event.
> > > > > However it won't work anymore since those samples will go to a missing
> > > > > thread now but the COMM event will create a (current) thread.  This
> > > > > leads to those early samples (like native_write_msr_safe) not having a
> > > > > comm but pid (like ':15328').
> > > >  
> > > > > So it needs to synthesize COMM event for the child explicitly before
> > > > > enabling so that it can have a correct comm.  But at this time, the
> > > > > comm will be "perf" since it's not exec-ed yet.
> > > > 
> > > > This looks reasonable, but I think it probably needs to be done
> > > > somewhere in perf_evlist__prepare_workload() or
> > > > perf_evlist__start_workload(), as this affects other tools as well, like
> > > > 'top', 'trace' and any other that may want to do this start-workload use
> > > > case.
> > > 
> > > Hmm.. I need to look at this again as it only affects on processing
> > > indexed data files which used to have a separate missing threads tree.
> > 
> > Humm, you're thinking about where you managed to reproduce the problem,
> > I am thinking outside indexing, etc, i.e. by definition we either enable
> > the event before we fork, so that we get the PERF_RECORD_FORK/COMM or we
> > synthesize it either from /proc or directly (preferred) if we decide to
> > do it after the fork/exec, right?
> 
> But as I said before, later COMM event will override thread->comm to a

But what happens for samples that take place before this "later COMM
event"? Even tho we know the COMM (we're starting the workload) we will
show just the pid, right?

Think about processing the samples with minimum delay (perf trace),
ordered by timestamp, but only ordering the head of the ring buffers,
not with a perf.data file.

- Arnaldo

> proper string as long as it can find a matching thread.  So I think it
> has no problem in the current code.
> 
> In the old version of this patchset (v3), indexing made it impossible
> for COMM event to find a matching thread since it used to have a
> separate tree for threads that have sampled before any FORK/COMM event
> came.  I think it doesn't apply to the current version anymore, I
> will check it tomorrow.
> 
> Thanks,
> Namhyung
> 
> 
> > 
> > - Arnaldo
> > 
> > > That's the reason why I didn't put it in a generic place like you
> > > said.
> > > 
> > > However I changed not to use the separate tree - the purpose of the
> > > tree was to reduce lock acquisition on thread searching but it already
> > > grabs a rwlock with thread refcounting change.
> > > 
> > > Will check whether this is still needed..
> > > 
> > > Thanks,
> > > Namhyung
> > > 
> > > 
> > > > 
> > > > I also wonder if we can't overcome this without using /proc, i.e.
> > > > actually moving the "start the workload" to just before the fork, so
> > > > that the kernel covers that as well.
> > > > 
> > > > Or, alternatively, the thread can be created without having to look at
> > > > /proc at all, but by directly creating the struct thread, with the
> > > > correct COMM, pid, etc, that we know, since we forked it, etc.

^ permalink raw reply	[flat|nested] 221+ messages in thread

* Re: [PATCH 35/40] perf record: Synthesize COMM event for a command line workload
  2015-05-19 15:12         ` Namhyung Kim
  2015-05-19 16:30           ` Arnaldo Carvalho de Melo
@ 2015-05-19 19:49           ` Jiri Olsa
  2015-05-19 20:18             ` Arnaldo Carvalho de Melo
  1 sibling, 1 reply; 221+ messages in thread
From: Jiri Olsa @ 2015-05-19 19:49 UTC (permalink / raw)
  To: Namhyung Kim
  Cc: Arnaldo Carvalho de Melo, Ingo Molnar, Peter Zijlstra, LKML,
	David Ahern, Adrian Hunter, Andi Kleen, Frederic Weisbecker,
	Stephane Eranian

On Wed, May 20, 2015 at 12:12:45AM +0900, Namhyung Kim wrote:

SNIP

> > > > 
> > > > This looks reasonable, but I think it probably needs to be done
> > > > somewhere in perf_evlist__prepare_workload() or
> > > > perf_evlist__start_workload(), as this affects other tools as well, like
> > > > 'top', 'trace' and any other that may want to do this start-workload use
> > > > case.
> > > 
> > > Hmm.. I need to look at this again as it only affects on processing
> > > indexed data files which used to have a separate missing threads tree.
> > 
> > Humm, you're thinking about where you managed to reproduce the problem,
> > I am thinking outside indexing, etc, i.e. by definition we either enable
> > the event before we fork, so that we get the PERF_RECORD_FORK/COMM or we
> > synthesize it either from /proc or directly (preferred) if we decide to
> > do it after the fork/exec, right?
> 
> But as I said before, later COMM event will override thread->comm to a
> proper string as long as it can find a matching thread.  So I think it
> has no problem in the current code.

I can see the issue in the current script code and the patch cured it ;-)

before:

[jolsa@krava perf]$ ./perf record -o - true | ./perf script 
[ perf record: Woken up 1 times to write data ]
[ perf record: Captured and wrote 0.000 MB - ]
          :29236 29236 350869.479255:          1 cycles:  ffffffff8105e8aa native_write_msr_safe ([kernel.kallsy
          :29236 29236 350869.479260:          1 cycles:  ffffffff8105e8aa native_write_msr_safe ([kernel.kallsy
          :29236 29236 350869.479261:          7 cycles:  ffffffff8105e8aa native_write_msr_safe ([kernel.kallsy
          :29236 29236 350869.479263:        122 cycles:  ffffffff8105e8aa native_write_msr_safe ([kernel.kallsy
          :29236 29236 350869.479264:       2303 cycles:  ffffffff8105e8aa native_write_msr_safe ([kernel.kallsy
          :29236 29236 350869.479266:      49582 cycles:  ffffffff8105e8aa native_write_msr_safe ([kernel.kallsy
            true 29236 350869.479283:    1041734 cycles:  ffffffff81190a4a perf_output_begin ([kernel.kallsyms])

after:

[jolsa@krava perf]$ ./perf record -o - true | ./perf script 
[ perf record: Woken up 1 times to write data ]
[ perf record: Captured and wrote 0.000 MB - ]
            perf 28041 350801.438466:          1 cycles:  ffffffff8105e8aa native_write_msr_safe ([kernel.kallsy
            perf 28041 350801.438471:          1 cycles:  ffffffff8105e8aa native_write_msr_safe ([kernel.kallsy
            perf 28041 350801.438472:          7 cycles:  ffffffff8105e8aa native_write_msr_safe ([kernel.kallsy
            perf 28041 350801.438473:        140 cycles:  ffffffff8105e8aa native_write_msr_safe ([kernel.kallsy
            perf 28041 350801.438475:       3721 cycles:  ffffffff8105e8aa native_write_msr_safe ([kernel.kallsy
            perf 28041 350801.438476:      94744 cycles:  ffffffff8118e270 perf_event_comm ([kernel.kallsyms])
            true 28041 350801.438508:    1604677 cycles:  ffffffff811d0781 do_mmap_pgoff ([kernel.kallsyms])

jirka

^ permalink raw reply	[flat|nested] 221+ messages in thread

* Re: [PATCH 35/40] perf record: Synthesize COMM event for a command line workload
  2015-05-19 19:49           ` Jiri Olsa
@ 2015-05-19 20:18             ` Arnaldo Carvalho de Melo
  2015-05-19 23:56               ` Namhyung Kim
  0 siblings, 1 reply; 221+ messages in thread
From: Arnaldo Carvalho de Melo @ 2015-05-19 20:18 UTC (permalink / raw)
  To: Jiri Olsa
  Cc: Namhyung Kim, Ingo Molnar, Peter Zijlstra, LKML, David Ahern,
	Adrian Hunter, Andi Kleen, Frederic Weisbecker, Stephane Eranian

Em Tue, May 19, 2015 at 09:49:03PM +0200, Jiri Olsa escreveu:
> On Wed, May 20, 2015 at 12:12:45AM +0900, Namhyung Kim wrote:
> 
> SNIP
> 
> > > > > 
> > > > > This looks reasonable, but I think it probably needs to be done
> > > > > somewhere in perf_evlist__prepare_workload() or
> > > > > perf_evlist__start_workload(), as this affects other tools as well, like
> > > > > 'top', 'trace' and any other that may want to do this start-workload use
> > > > > case.
> > > > 
> > > > Hmm.. I need to look at this again as it only affects on processing
> > > > indexed data files which used to have a separate missing threads tree.
> > > 
> > > Humm, you're thinking about where you managed to reproduce the problem,
> > > I am thinking outside indexing, etc, i.e. by definition we either enable
> > > the event before we fork, so that we get the PERF_RECORD_FORK/COMM or we
> > > synthesize it either from /proc or directly (preferred) if we decide to
> > > do it after the fork/exec, right?
> > 
> > But as I said before, later COMM event will override thread->comm to a
> > proper string as long as it can find a matching thread.  So I think it
> > has no problem in the current code.
 
> I can see the issue in the current script code and the patch cured it ;-)

Exactly, this is my point, this is not something new :-)

- Arnaldo
 
> before:
> 
> [jolsa@krava perf]$ ./perf record -o - true | ./perf script 
> [ perf record: Woken up 1 times to write data ]
> [ perf record: Captured and wrote 0.000 MB - ]
>           :29236 29236 350869.479255:          1 cycles:  ffffffff8105e8aa native_write_msr_safe ([kernel.kallsy
>           :29236 29236 350869.479260:          1 cycles:  ffffffff8105e8aa native_write_msr_safe ([kernel.kallsy
>           :29236 29236 350869.479261:          7 cycles:  ffffffff8105e8aa native_write_msr_safe ([kernel.kallsy
>           :29236 29236 350869.479263:        122 cycles:  ffffffff8105e8aa native_write_msr_safe ([kernel.kallsy
>           :29236 29236 350869.479264:       2303 cycles:  ffffffff8105e8aa native_write_msr_safe ([kernel.kallsy
>           :29236 29236 350869.479266:      49582 cycles:  ffffffff8105e8aa native_write_msr_safe ([kernel.kallsy
>             true 29236 350869.479283:    1041734 cycles:  ffffffff81190a4a perf_output_begin ([kernel.kallsyms])
> 
> after:
> 
> [jolsa@krava perf]$ ./perf record -o - true | ./perf script 
> [ perf record: Woken up 1 times to write data ]
> [ perf record: Captured and wrote 0.000 MB - ]
>             perf 28041 350801.438466:          1 cycles:  ffffffff8105e8aa native_write_msr_safe ([kernel.kallsy
>             perf 28041 350801.438471:          1 cycles:  ffffffff8105e8aa native_write_msr_safe ([kernel.kallsy
>             perf 28041 350801.438472:          7 cycles:  ffffffff8105e8aa native_write_msr_safe ([kernel.kallsy
>             perf 28041 350801.438473:        140 cycles:  ffffffff8105e8aa native_write_msr_safe ([kernel.kallsy
>             perf 28041 350801.438475:       3721 cycles:  ffffffff8105e8aa native_write_msr_safe ([kernel.kallsy
>             perf 28041 350801.438476:      94744 cycles:  ffffffff8118e270 perf_event_comm ([kernel.kallsyms])
>             true 28041 350801.438508:    1604677 cycles:  ffffffff811d0781 do_mmap_pgoff ([kernel.kallsyms])
> 
> jirka

^ permalink raw reply	[flat|nested] 221+ messages in thread

* Re: [PATCH 38/40] perf session: Handle index files generally
  2015-05-18  0:30 ` [PATCH 38/40] perf session: Handle index files generally Namhyung Kim
@ 2015-05-19 22:27   ` David Ahern
  2015-05-20  0:05     ` Namhyung Kim
  0 siblings, 1 reply; 221+ messages in thread
From: David Ahern @ 2015-05-19 22:27 UTC (permalink / raw)
  To: Namhyung Kim, Arnaldo Carvalho de Melo
  Cc: Ingo Molnar, Peter Zijlstra, Jiri Olsa, LKML, Adrian Hunter,
	Andi Kleen, Frederic Weisbecker, Stephane Eranian

On 5/17/15 6:30 PM, Namhyung Kim wrote:
> The current code assumes that the number of index item and cpu are
> matched so it creates that number of threads.  But it's not the case
> of non-system-wide session or data came from different machine.
>
> Just creates threads at most number of online cpus and process data.

-----8<-----

> @@ -1717,6 +1742,7 @@ int perf_session__process_events_mt(struct perf_session *session, void *arg)
>   	int err, i, k;
>   	int nr_index = session->header.nr_index;
>   	u64 size = perf_data_file__size(file);
> +	int nr_thread = sysconf(_SC_NPROCESSORS_ONLN);

It's not clear to me how this multi-threaded perf is going to work on 
large systems especially this patch if a system has holes in the active 
cpus. e.g,

# lscpu
Architecture:          sparc64
CPU op-mode(s):        32-bit, 64-bit
Byte Order:            Big Endian
CPU(s):                704
On-line CPU(s) list: 
32-63,128-223,256-351,384-479,576-831,864-927,960-1023
Thread(s) per core:    12
Core(s) per socket:    18
Socket(s):             3
NUMA node(s):          4
NUMA node0 CPU(s):     32-63,128-223
NUMA node1 CPU(s):     256-351,384-479
NUMA node2 CPU(s):     576-767
NUMA node3 CPU(s):     768-831,864-927,960-1023

So you are going to spawn 704 threads? Each thread handles a per-cpu buffer?

yes, I still need to find time to take if for a test drive; maybe by the 
end of the week.

David

^ permalink raw reply	[flat|nested] 221+ messages in thread

* Re: [PATCH 33/40] perf session: Separate struct machines from session
  2015-05-19  7:28     ` Namhyung Kim
@ 2015-05-19 22:46       ` Arnaldo Carvalho de Melo
  2015-05-19 23:58         ` Namhyung Kim
  0 siblings, 1 reply; 221+ messages in thread
From: Arnaldo Carvalho de Melo @ 2015-05-19 22:46 UTC (permalink / raw)
  To: Namhyung Kim
  Cc: Ingo Molnar, Peter Zijlstra, Jiri Olsa, LKML, David Ahern,
	Adrian Hunter, Andi Kleen, Frederic Weisbecker, Stephane Eranian

Em Tue, May 19, 2015 at 04:28:15PM +0900, Namhyung Kim escreveu:
> On Mon, May 18, 2015 at 09:52:59AM -0300, Arnaldo Carvalho de Melo wrote:
> > Em Mon, May 18, 2015 at 09:30:48AM +0900, Namhyung Kim escreveu:
> > > With multi-thread report, separate sessions can be passed to each
> > > thread, in this case we should keep a single machine state for all
> > > struct sessions.  Separate machines and have a pointer in sessions.
> > 
> > I had to look at all the patch to semi-figure this out, i.e. you said it
> > should be separated from 'perf_session', agreed.
> > 
> > But who will create it?  How will it be passed to the perf_session
> > instances?
> > 
> > Most of the patch is making session->machines be turned into a pointer,
> > but the meat, i.e. who creates it, is unclear, I see a malloc in
> > perf_session__new(), where I was kinda expecting that a higer layer,
> > perhaps in struct tool? Would create the list of all machines (struct
> > machines) and then pass it to multiple perf_session__new() calls.
> > 
> > But then perf_session__delete() calls 'free(session->machines)', huh?
> 
> OK.  So, this is what I have in my head:
> 
>   perf_tool__create_machines(tool) {
>     tool->machines = malloc();
>     machines__init(tool->machines);
>   }


Probably, but then in this case you would call machines__new(), that
does the malloc and init.

> 
>   perf_session__new(file, repipe, tool) {
>     session->machines = tool->machines;
>     ...
>   }

That could be ok.

> 
>   perf_tool__delete_machines(tool) {
>     /* call machines-related destructors */
>     free(tool->machines);
>   }

That would be machines__delete(tool->machines), that calls
machine__exit() and then does the free.

- Arnaldo

^ permalink raw reply	[flat|nested] 221+ messages in thread

* Re: [PATCH 35/40] perf record: Synthesize COMM event for a command line workload
  2015-05-19 20:18             ` Arnaldo Carvalho de Melo
@ 2015-05-19 23:56               ` Namhyung Kim
  2015-05-20  0:22                 ` Arnaldo Carvalho de Melo
  0 siblings, 1 reply; 221+ messages in thread
From: Namhyung Kim @ 2015-05-19 23:56 UTC (permalink / raw)
  To: Arnaldo Carvalho de Melo
  Cc: Jiri Olsa, Ingo Molnar, Peter Zijlstra, LKML, David Ahern,
	Adrian Hunter, Andi Kleen, Frederic Weisbecker, Stephane Eranian

On Tue, May 19, 2015 at 05:18:54PM -0300, Arnaldo Carvalho de Melo wrote:
> Em Tue, May 19, 2015 at 09:49:03PM +0200, Jiri Olsa escreveu:
> > On Wed, May 20, 2015 at 12:12:45AM +0900, Namhyung Kim wrote:
> > 
> > SNIP
> > 
> > > > > > 
> > > > > > This looks reasonable, but I think it probably needs to be done
> > > > > > somewhere in perf_evlist__prepare_workload() or
> > > > > > perf_evlist__start_workload(), as this affects other tools as well, like
> > > > > > 'top', 'trace' and any other that may want to do this start-workload use
> > > > > > case.
> > > > > 
> > > > > Hmm.. I need to look at this again as it only affects on processing
> > > > > indexed data files which used to have a separate missing threads tree.
> > > > 
> > > > Humm, you're thinking about where you managed to reproduce the problem,
> > > > I am thinking outside indexing, etc, i.e. by definition we either enable
> > > > the event before we fork, so that we get the PERF_RECORD_FORK/COMM or we
> > > > synthesize it either from /proc or directly (preferred) if we decide to
> > > > do it after the fork/exec, right?
> > > 
> > > But as I said before, later COMM event will override thread->comm to a
> > > proper string as long as it can find a matching thread.  So I think it
> > > has no problem in the current code.
>  
> > I can see the issue in the current script code and the patch cured it ;-)
> 
> Exactly, this is my point, this is not something new :-)

Ah, okay.  The perf script shows samples before processing comm events
while perf report shows after processing all events.

But to move it under generic place like perf_evlist__{prepare,start}_
workload(), it seems we need to pass an additional callback and data.

I'll try to write a patch.

Thanks,
Namhyung

> 
> - Arnaldo
>  
> > before:
> > 
> > [jolsa@krava perf]$ ./perf record -o - true | ./perf script 
> > [ perf record: Woken up 1 times to write data ]
> > [ perf record: Captured and wrote 0.000 MB - ]
> >           :29236 29236 350869.479255:          1 cycles:  ffffffff8105e8aa native_write_msr_safe ([kernel.kallsy
> >           :29236 29236 350869.479260:          1 cycles:  ffffffff8105e8aa native_write_msr_safe ([kernel.kallsy
> >           :29236 29236 350869.479261:          7 cycles:  ffffffff8105e8aa native_write_msr_safe ([kernel.kallsy
> >           :29236 29236 350869.479263:        122 cycles:  ffffffff8105e8aa native_write_msr_safe ([kernel.kallsy
> >           :29236 29236 350869.479264:       2303 cycles:  ffffffff8105e8aa native_write_msr_safe ([kernel.kallsy
> >           :29236 29236 350869.479266:      49582 cycles:  ffffffff8105e8aa native_write_msr_safe ([kernel.kallsy
> >             true 29236 350869.479283:    1041734 cycles:  ffffffff81190a4a perf_output_begin ([kernel.kallsyms])
> > 
> > after:
> > 
> > [jolsa@krava perf]$ ./perf record -o - true | ./perf script 
> > [ perf record: Woken up 1 times to write data ]
> > [ perf record: Captured and wrote 0.000 MB - ]
> >             perf 28041 350801.438466:          1 cycles:  ffffffff8105e8aa native_write_msr_safe ([kernel.kallsy
> >             perf 28041 350801.438471:          1 cycles:  ffffffff8105e8aa native_write_msr_safe ([kernel.kallsy
> >             perf 28041 350801.438472:          7 cycles:  ffffffff8105e8aa native_write_msr_safe ([kernel.kallsy
> >             perf 28041 350801.438473:        140 cycles:  ffffffff8105e8aa native_write_msr_safe ([kernel.kallsy
> >             perf 28041 350801.438475:       3721 cycles:  ffffffff8105e8aa native_write_msr_safe ([kernel.kallsy
> >             perf 28041 350801.438476:      94744 cycles:  ffffffff8118e270 perf_event_comm ([kernel.kallsyms])
> >             true 28041 350801.438508:    1604677 cycles:  ffffffff811d0781 do_mmap_pgoff ([kernel.kallsyms])
> > 
> > jirka
> --
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/

^ permalink raw reply	[flat|nested] 221+ messages in thread

* Re: [PATCH 33/40] perf session: Separate struct machines from session
  2015-05-19 22:46       ` Arnaldo Carvalho de Melo
@ 2015-05-19 23:58         ` Namhyung Kim
  2015-05-28  2:39           ` [RFC 0/2] " Namhyung Kim
  0 siblings, 1 reply; 221+ messages in thread
From: Namhyung Kim @ 2015-05-19 23:58 UTC (permalink / raw)
  To: Arnaldo Carvalho de Melo
  Cc: Ingo Molnar, Peter Zijlstra, Jiri Olsa, LKML, David Ahern,
	Adrian Hunter, Andi Kleen, Frederic Weisbecker, Stephane Eranian

On Tue, May 19, 2015 at 07:46:18PM -0300, Arnaldo Carvalho de Melo wrote:
> Em Tue, May 19, 2015 at 04:28:15PM +0900, Namhyung Kim escreveu:
> > On Mon, May 18, 2015 at 09:52:59AM -0300, Arnaldo Carvalho de Melo wrote:
> > > Em Mon, May 18, 2015 at 09:30:48AM +0900, Namhyung Kim escreveu:
> > > > With multi-thread report, separate sessions can be passed to each
> > > > thread, in this case we should keep a single machine state for all
> > > > struct sessions.  Separate machines and have a pointer in sessions.
> > > 
> > > I had to look at all the patch to semi-figure this out, i.e. you said it
> > > should be separated from 'perf_session', agreed.
> > > 
> > > But who will create it?  How will it be passed to the perf_session
> > > instances?
> > > 
> > > Most of the patch is making session->machines be turned into a pointer,
> > > but the meat, i.e. who creates it, is unclear, I see a malloc in
> > > perf_session__new(), where I was kinda expecting that a higer layer,
> > > perhaps in struct tool? Would create the list of all machines (struct
> > > machines) and then pass it to multiple perf_session__new() calls.
> > > 
> > > But then perf_session__delete() calls 'free(session->machines)', huh?
> > 
> > OK.  So, this is what I have in my head:
> > 
> >   perf_tool__create_machines(tool) {
> >     tool->machines = malloc();
> >     machines__init(tool->machines);
> >   }
> 
> 
> Probably, but then in this case you would call machines__new(), that
> does the malloc and init.
> 
> > 
> >   perf_session__new(file, repipe, tool) {
> >     session->machines = tool->machines;
> >     ...
> >   }
> 
> That could be ok.
> 
> > 
> >   perf_tool__delete_machines(tool) {
> >     /* call machines-related destructors */
> >     free(tool->machines);
> >   }
> 
> That would be machines__delete(tool->machines), that calls
> machine__exit() and then does the free.

Right.  I'll change it this way.

Thanks,
Namhyung

^ permalink raw reply	[flat|nested] 221+ messages in thread

* Re: [PATCH 38/40] perf session: Handle index files generally
  2015-05-19 22:27   ` David Ahern
@ 2015-05-20  0:05     ` Namhyung Kim
  2015-05-20  7:20       ` [PATCH 41/40] perf report: Add --num-thread option to control number of thread Namhyung Kim
  0 siblings, 1 reply; 221+ messages in thread
From: Namhyung Kim @ 2015-05-20  0:05 UTC (permalink / raw)
  To: David Ahern
  Cc: Arnaldo Carvalho de Melo, Ingo Molnar, Peter Zijlstra, Jiri Olsa,
	LKML, Adrian Hunter, Andi Kleen, Frederic Weisbecker,
	Stephane Eranian

Hi David,

On Tue, May 19, 2015 at 04:27:02PM -0600, David Ahern wrote:
> On 5/17/15 6:30 PM, Namhyung Kim wrote:
> >The current code assumes that the number of index item and cpu are
> >matched so it creates that number of threads.  But it's not the case
> >of non-system-wide session or data came from different machine.
> >
> >Just creates threads at most number of online cpus and process data.
> 
> -----8<-----
> 
> >@@ -1717,6 +1742,7 @@ int perf_session__process_events_mt(struct perf_session *session, void *arg)
> >  	int err, i, k;
> >  	int nr_index = session->header.nr_index;
> >  	u64 size = perf_data_file__size(file);
> >+	int nr_thread = sysconf(_SC_NPROCESSORS_ONLN);
> 
> It's not clear to me how this multi-threaded perf is going to work on large
> systems especially this patch if a system has holes in the active cpus. e.g,
> 
> # lscpu
> Architecture:          sparc64
> CPU op-mode(s):        32-bit, 64-bit
> Byte Order:            Big Endian
> CPU(s):                704
> On-line CPU(s) list: 32-63,128-223,256-351,384-479,576-831,864-927,960-1023
> Thread(s) per core:    12
> Core(s) per socket:    18
> Socket(s):             3
> NUMA node(s):          4
> NUMA node0 CPU(s):     32-63,128-223
> NUMA node1 CPU(s):     256-351,384-479
> NUMA node2 CPU(s):     576-767
> NUMA node3 CPU(s):     768-831,864-927,960-1023
> 
> So you are going to spawn 704 threads? Each thread handles a per-cpu buffer?

Oh, I need to add an option (and probably a config variable too) to
specify number of thread to run.

> 
> yes, I still need to find time to take if for a test drive; maybe by the end
> of the week.

Oh, it'd be very nice if you can test and share the numbers on such a
large machine.  I'll send the --num-thread option patch soon on top of
this series for you. :)

Thanks,
Namhyung


> 
> David
> --
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/

^ permalink raw reply	[flat|nested] 221+ messages in thread

* Re: [PATCH 35/40] perf record: Synthesize COMM event for a command line workload
  2015-05-19 23:56               ` Namhyung Kim
@ 2015-05-20  0:22                 ` Arnaldo Carvalho de Melo
  2015-05-20  0:56                   ` Namhyung Kim
  0 siblings, 1 reply; 221+ messages in thread
From: Arnaldo Carvalho de Melo @ 2015-05-20  0:22 UTC (permalink / raw)
  To: Namhyung Kim
  Cc: Jiri Olsa, Ingo Molnar, Peter Zijlstra, LKML, David Ahern,
	Adrian Hunter, Andi Kleen, Frederic Weisbecker, Stephane Eranian

Em Wed, May 20, 2015 at 08:56:58AM +0900, Namhyung Kim escreveu:
> On Tue, May 19, 2015 at 05:18:54PM -0300, Arnaldo Carvalho de Melo wrote:
> > Em Tue, May 19, 2015 at 09:49:03PM +0200, Jiri Olsa escreveu:
> > > > > Humm, you're thinking about where you managed to reproduce the problem,
> > > > > I am thinking outside indexing, etc, i.e. by definition we either enable
> > > > > the event before we fork, so that we get the PERF_RECORD_FORK/COMM or we
> > > > > synthesize it either from /proc or directly (preferred) if we decide to
> > > > > do it after the fork/exec, right?

> > > > But as I said before, later COMM event will override thread->comm to a
> > > > proper string as long as it can find a matching thread.  So I think it
> > > > has no problem in the current code.
 
> > > I can see the issue in the current script code and the patch cured it ;-)
 
> > Exactly, this is my point, this is not something new :-)
 
> Ah, okay.  The perf script shows samples before processing comm events
> while perf report shows after processing all events.

I.e. 'perf script' behaves like 'perf trace' and 'perf top'. 'perf
report' is the odd one out, and I think it should be not, i.e. you
should try to think more about the non 'report' use cases when thinking
about how to improve report :-)

But I digress, lets get back to the question at hand...
 
> But to move it under generic place like perf_evlist__{prepare,start}_
> workload(), it seems we need to pass an additional callback and data.

Only if you want to do it with perf_event__synthesize_comm(). I
suggested writing a new synthesize routine that doesn't parses /proc, as
we have all that we need, no?

I think that just doing something like:

	thread = machine__findnew_thread(evlist->workload.pid, evlist->workload.pid);
	if (thread)
		thread__set_comm(thread, argv[0], timestamp);

Should be enough, no? I.e. no need for setting up a PERF_RECORD_FORK and
a PERF_RECORD_COMM, read /proc, etc, just do it directly with the info
we used to do the fork in perf_evlist__prepare_workload(), etc.

> I'll try to write a patch.

Great!

- Arnaldo

^ permalink raw reply	[flat|nested] 221+ messages in thread

* Re: [PATCH 35/40] perf record: Synthesize COMM event for a command line workload
  2015-05-20  0:22                 ` Arnaldo Carvalho de Melo
@ 2015-05-20  0:56                   ` Namhyung Kim
  2015-05-20  1:16                     ` Arnaldo Carvalho de Melo
  0 siblings, 1 reply; 221+ messages in thread
From: Namhyung Kim @ 2015-05-20  0:56 UTC (permalink / raw)
  To: Arnaldo Carvalho de Melo
  Cc: Jiri Olsa, Ingo Molnar, Peter Zijlstra, LKML, David Ahern,
	Adrian Hunter, Andi Kleen, Frederic Weisbecker, Stephane Eranian

On Tue, May 19, 2015 at 09:22:10PM -0300, Arnaldo Carvalho de Melo wrote:
> Em Wed, May 20, 2015 at 08:56:58AM +0900, Namhyung Kim escreveu:
> > On Tue, May 19, 2015 at 05:18:54PM -0300, Arnaldo Carvalho de Melo wrote:
> > > Em Tue, May 19, 2015 at 09:49:03PM +0200, Jiri Olsa escreveu:
> > > > > > Humm, you're thinking about where you managed to reproduce the problem,
> > > > > > I am thinking outside indexing, etc, i.e. by definition we either enable
> > > > > > the event before we fork, so that we get the PERF_RECORD_FORK/COMM or we
> > > > > > synthesize it either from /proc or directly (preferred) if we decide to
> > > > > > do it after the fork/exec, right?
> 
> > > > > But as I said before, later COMM event will override thread->comm to a
> > > > > proper string as long as it can find a matching thread.  So I think it
> > > > > has no problem in the current code.
>  
> > > > I can see the issue in the current script code and the patch cured it ;-)
>  
> > > Exactly, this is my point, this is not something new :-)
>  
> > Ah, okay.  The perf script shows samples before processing comm events
> > while perf report shows after processing all events.
> 
> I.e. 'perf script' behaves like 'perf trace' and 'perf top'. 'perf
> report' is the odd one out, and I think it should be not, i.e. you
> should try to think more about the non 'report' use cases when thinking
> about how to improve report :-)

I'll keep it in mind.

> 
> But I digress, lets get back to the question at hand...
>  
> > But to move it under generic place like perf_evlist__{prepare,start}_
> > workload(), it seems we need to pass an additional callback and data.
> 
> Only if you want to do it with perf_event__synthesize_comm(). I
> suggested writing a new synthesize routine that doesn't parses /proc, as
> we have all that we need, no?

Agreed.

> 
> I think that just doing something like:
> 
> 	thread = machine__findnew_thread(evlist->workload.pid, evlist->workload.pid);
> 	if (thread)
> 		thread__set_comm(thread, argv[0], timestamp);
> 
> Should be enough, no? I.e. no need for setting up a PERF_RECORD_FORK and
> a PERF_RECORD_COMM, read /proc, etc, just do it directly with the info
> we used to do the fork in perf_evlist__prepare_workload(), etc.

For non-record use case it'd be enough.  But for record, it needs to
synthesize/write an event to data file so that perf report can
recognize it later.  That's why I think it needs callback.

Thanks,
Namhyung

^ permalink raw reply	[flat|nested] 221+ messages in thread

* Re: [PATCH 35/40] perf record: Synthesize COMM event for a command line workload
  2015-05-20  0:56                   ` Namhyung Kim
@ 2015-05-20  1:16                     ` Arnaldo Carvalho de Melo
  0 siblings, 0 replies; 221+ messages in thread
From: Arnaldo Carvalho de Melo @ 2015-05-20  1:16 UTC (permalink / raw)
  To: Namhyung Kim
  Cc: Jiri Olsa, Ingo Molnar, Peter Zijlstra, LKML, David Ahern,
	Adrian Hunter, Andi Kleen, Frederic Weisbecker, Stephane Eranian

Em Wed, May 20, 2015 at 09:56:25AM +0900, Namhyung Kim escreveu:
> On Tue, May 19, 2015 at 09:22:10PM -0300, Arnaldo Carvalho de Melo wrote:
> > Em Wed, May 20, 2015 at 08:56:58AM +0900, Namhyung Kim escreveu:
> > > On Tue, May 19, 2015 at 05:18:54PM -0300, Arnaldo Carvalho de Melo wrote:
> > > > Em Tue, May 19, 2015 at 09:49:03PM +0200, Jiri Olsa escreveu:
> > > > > > > Humm, you're thinking about where you managed to reproduce the problem,
> > > > > > > I am thinking outside indexing, etc, i.e. by definition we either enable
> > > > > > > the event before we fork, so that we get the PERF_RECORD_FORK/COMM or we
> > > > > > > synthesize it either from /proc or directly (preferred) if we decide to
> > > > > > > do it after the fork/exec, right?
> > 
> > > > > > But as I said before, later COMM event will override thread->comm to a
> > > > > > proper string as long as it can find a matching thread.  So I think it
> > > > > > has no problem in the current code.
> >  
> > > > > I can see the issue in the current script code and the patch cured it ;-)
> >  
> > > > Exactly, this is my point, this is not something new :-)
> >  
> > > Ah, okay.  The perf script shows samples before processing comm events
> > > while perf report shows after processing all events.
> > 
> > I.e. 'perf script' behaves like 'perf trace' and 'perf top'. 'perf
> > report' is the odd one out, and I think it should be not, i.e. you
> > should try to think more about the non 'report' use cases when thinking
> > about how to improve report :-)
> 
> I'll keep it in mind.
> 
> > 
> > But I digress, lets get back to the question at hand...
> >  
> > > But to move it under generic place like perf_evlist__{prepare,start}_
> > > workload(), it seems we need to pass an additional callback and data.
> > 
> > Only if you want to do it with perf_event__synthesize_comm(). I
> > suggested writing a new synthesize routine that doesn't parses /proc, as
> > we have all that we need, no?
> 
> Agreed.
> 
> > 
> > I think that just doing something like:
> > 
> > 	thread = machine__findnew_thread(evlist->workload.pid, evlist->workload.pid);
> > 	if (thread)
> > 		thread__set_comm(thread, argv[0], timestamp);
> > 
> > Should be enough, no? I.e. no need for setting up a PERF_RECORD_FORK and
> > a PERF_RECORD_COMM, read /proc, etc, just do it directly with the info
> > we used to do the fork in perf_evlist__prepare_workload(), etc.
> 
> For non-record use case it'd be enough.  But for record, it needs to
> synthesize/write an event to data file so that perf report canout
> recognize it later.  That's why I think it needs callback.

You have a point, then probably it is better to have it as a function to
be called by each kind of tool, as needed.

- Arnaldo

^ permalink raw reply	[flat|nested] 221+ messages in thread

* [PATCH 41/40] perf report: Add --num-thread option to control number of thread
  2015-05-20  0:05     ` Namhyung Kim
@ 2015-05-20  7:20       ` Namhyung Kim
  0 siblings, 0 replies; 221+ messages in thread
From: Namhyung Kim @ 2015-05-20  7:20 UTC (permalink / raw)
  To: Arnaldo Carvalho de Melo, David Ahern
  Cc: Ingo Molnar, Peter Zijlstra, Jiri Olsa, LKML

The --num-thread is to control number of thread to process events when
--multi-thread option is enabled.  Default value is the number of cpu
on the system that runs perf report.  The config variable
'report.num-thread' is also added to set a different default value.

Signed-off-by: Namhyung Kim <namhyung@kernel.org>
---
Also update the perf/threaded-v4 branch.

 tools/perf/Documentation/perf-report.txt |  5 +++++
 tools/perf/builtin-report.c              | 29 +++++++++++++++++++++++++----
 tools/perf/util/session.c                |  4 ++--
 tools/perf/util/session.h                |  3 ++-
 4 files changed, 34 insertions(+), 7 deletions(-)

diff --git a/tools/perf/Documentation/perf-report.txt b/tools/perf/Documentation/perf-report.txt
index 3917710e2620..c6050496d720 100644
--- a/tools/perf/Documentation/perf-report.txt
+++ b/tools/perf/Documentation/perf-report.txt
@@ -356,6 +356,11 @@ OPTIONS
 --multi-thread::
 	Speed up report by parallelizing sample processing using multi-thread.
 
+--num-thread::
+	Number of thread to process sample events.  Should be greater than or
+	equals to 2.  Default is number of cpus on the system unless
+	report.num-thread config variable is set.
+
 include::callchain-overhead-calculation.txt[]
 
 SEE ALSO
diff --git a/tools/perf/builtin-report.c b/tools/perf/builtin-report.c
index 4d08e5f0a7bb..71ab710f2a1e 100644
--- a/tools/perf/builtin-report.c
+++ b/tools/perf/builtin-report.c
@@ -54,6 +54,7 @@ struct report {
 	bool			header;
 	bool			header_only;
 	bool			multi_thread;
+	int			nr_thread;
 	int			max_stack;
 	struct perf_read_values	show_threads_values;
 	const char		*pretty_printing_style;
@@ -89,6 +90,10 @@ static int report__config(const char *var, const char *value, void *cb)
 		rep->multi_thread = perf_config_bool(var, value);
 		return 0;
 	}
+	if (!strcmp(var, "report.num-thread")) {
+		rep->nr_thread = perf_config_int(var, value);
+		return 0;
+	}
 
 	return perf_default_config(var, value, cb);
 }
@@ -522,7 +527,8 @@ static int __cmd_report(struct report *rep)
 
 	if (rep->multi_thread) {
 		rep->tool.sample = process_sample_event_mt;
-		ret = perf_session__process_events_mt(session, rep);
+		ret = perf_session__process_events_mt(session, rep->nr_thread,
+						      rep);
 	} else {
 		ret = perf_session__process_events(session);
 	}
@@ -664,6 +670,7 @@ int cmd_report(int argc, const char **argv, const char *prefix __maybe_unused)
 		},
 		.max_stack		 = PERF_MAX_STACK_DEPTH,
 		.pretty_printing_style	 = "normal",
+		.nr_thread		 = -1,
 	};
 	const struct option options[] = {
 	OPT_STRING('i', "input", &input_name, "file",
@@ -774,6 +781,8 @@ int cmd_report(int argc, const char **argv, const char *prefix __maybe_unused)
 			    itrace_parse_synth_opts),
 	OPT_BOOLEAN(0, "multi-thread", &report.multi_thread,
 		    "Speed up sample processing using multi-thead"),
+	OPT_INTEGER(0, "num-thread", &report.nr_thread,
+		    "Number of thread to process samples (>= 2)"),
 	OPT_END()
 	};
 	struct perf_data_file file = {
@@ -820,9 +829,21 @@ int cmd_report(int argc, const char **argv, const char *prefix __maybe_unused)
 
 	session->itrace_synth_opts = &itrace_synth_opts;
 
-	if (report.multi_thread && !perf_has_index) {
-		pr_debug("fallback to single thread for normal data file.\n");
-		report.multi_thread = false;
+	if (report.multi_thread) {
+		if (!perf_has_index) {
+			pr_debug("fallback to single thread for normal data file.\n");
+			report.multi_thread = false;
+		} else {
+			if (report.nr_thread == -1)
+				report.nr_thread = sysconf(_SC_NPROCESSORS_ONLN);
+			else if (report.nr_thread < 2) {
+				pr_err("invalid number of thread: %d\n",
+				       report.nr_thread);
+				parse_options_usage(report_usage, options,
+						    "num-thread", 0);
+				goto error;
+			}
+		}
 	}
 
 	report.session = session;
diff --git a/tools/perf/util/session.c b/tools/perf/util/session.c
index 5f6c319bd236..747d63843bab 100644
--- a/tools/perf/util/session.c
+++ b/tools/perf/util/session.c
@@ -1726,7 +1726,8 @@ static void *processing_thread_idx(void *arg)
 	return arg;
 }
 
-int perf_session__process_events_mt(struct perf_session *session, void *arg)
+int perf_session__process_events_mt(struct perf_session *session,
+				    int nr_thread, void *arg)
 {
 	struct perf_data_file *file = session->file;
 	struct perf_evlist *evlist = session->evlist;
@@ -1742,7 +1743,6 @@ int perf_session__process_events_mt(struct perf_session *session, void *arg)
 	int err, i, k;
 	int nr_index = session->header.nr_index;
 	u64 size = perf_data_file__size(file);
-	int nr_thread = sysconf(_SC_NPROCESSORS_ONLN);
 
 	if (perf_data_file__is_pipe(file) || !session->header.index) {
 		pr_err("data file doesn't contain the index table\n");
diff --git a/tools/perf/util/session.h b/tools/perf/util/session.h
index fde658d5d081..a033e104f124 100644
--- a/tools/perf/util/session.h
+++ b/tools/perf/util/session.h
@@ -56,7 +56,8 @@ int perf_session__peek_event(struct perf_session *session, off_t file_offset,
 			     struct perf_sample *sample);
 
 int perf_session__process_events(struct perf_session *session);
-int perf_session__process_events_mt(struct perf_session *session, void *arg);
+int perf_session__process_events_mt(struct perf_session *session,
+				    int nr_thread, void *arg);
 
 int perf_session__queue_event(struct perf_session *s, union perf_event *event,
 			      struct perf_sample *sample, u64 file_offset);
-- 
2.4.0


^ permalink raw reply related	[flat|nested] 221+ messages in thread

* [tip:perf/core] perf tools: Add rm_rf() utility function
  2015-01-30 15:02   ` Jiri Olsa
@ 2015-05-20 12:24     ` tip-bot for Namhyung Kim
  0 siblings, 0 replies; 221+ messages in thread
From: tip-bot for Namhyung Kim @ 2015-05-20 12:24 UTC (permalink / raw)
  To: linux-tip-commits
  Cc: adrian.hunter, andi, mingo, namhyung, acme, dsahern, hpa,
	eranian, tglx, linux-kernel, fweisbec, a.p.zijlstra, jolsa

Commit-ID:  0b1de0be1eac7b23e89cb43c17b02d38ead6b6c8
Gitweb:     http://git.kernel.org/tip/0b1de0be1eac7b23e89cb43c17b02d38ead6b6c8
Author:     Namhyung Kim <namhyung@kernel.org>
AuthorDate: Mon, 18 May 2015 09:30:17 +0900
Committer:  Arnaldo Carvalho de Melo <acme@redhat.com>
CommitDate: Mon, 18 May 2015 10:17:34 -0300

perf tools: Add rm_rf() utility function

The rm_rf() function does same as the shell command 'rm -rf' which
removes all directory entries recursively.

Signed-off-by: Namhyung Kim <namhyung@kernel.org>
Acked-by: Jiri Olsa <jolsa@kernel.org>
Cc: Adrian Hunter <adrian.hunter@intel.com>
Cc: Andi Kleen <andi@firstfloor.org>
Cc: David Ahern <dsahern@gmail.com>
Cc: Frederic Weisbecker <fweisbec@gmail.com>
Cc: Peter Zijlstra <a.p.zijlstra@chello.nl>
Cc: Stephane Eranian <eranian@google.com>
Link: http://lkml.kernel.org/r/1431909055-21442-3-git-send-email-namhyung@kernel.org
Link: http://lkml.kernel.org/r/20150130150256.GF6188@krava.brq.redhat.com
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
---
 tools/perf/util/util.c | 43 +++++++++++++++++++++++++++++++++++++++++++
 tools/perf/util/util.h |  1 +
 2 files changed, 44 insertions(+)

diff --git a/tools/perf/util/util.c b/tools/perf/util/util.c
index 4ee6d0d..6104afb 100644
--- a/tools/perf/util/util.c
+++ b/tools/perf/util/util.c
@@ -72,6 +72,49 @@ int mkdir_p(char *path, mode_t mode)
 	return (stat(path, &st) && mkdir(path, mode)) ? -1 : 0;
 }
 
+int rm_rf(char *path)
+{
+	DIR *dir;
+	int ret = 0;
+	struct dirent *d;
+	char namebuf[PATH_MAX];
+
+	dir = opendir(path);
+	if (dir == NULL)
+		return 0;
+
+	while ((d = readdir(dir)) != NULL && !ret) {
+		struct stat statbuf;
+
+		if (!strcmp(d->d_name, ".") || !strcmp(d->d_name, ".."))
+			continue;
+
+		scnprintf(namebuf, sizeof(namebuf), "%s/%s",
+			  path, d->d_name);
+
+		ret = stat(namebuf, &statbuf);
+		if (ret < 0) {
+			pr_debug("stat failed: %s\n", namebuf);
+			break;
+		}
+
+		if (S_ISREG(statbuf.st_mode))
+			ret = unlink(namebuf);
+		else if (S_ISDIR(statbuf.st_mode))
+			ret = rm_rf(namebuf);
+		else {
+			pr_debug("unknown file: %s\n", namebuf);
+			ret = -1;
+		}
+	}
+	closedir(dir);
+
+	if (ret < 0)
+		return ret;
+
+	return rmdir(path);
+}
+
 static int slow_copyfile(const char *from, const char *to, mode_t mode)
 {
 	int err = -1;
diff --git a/tools/perf/util/util.h b/tools/perf/util/util.h
index 3601ffd..c4fe38a 100644
--- a/tools/perf/util/util.h
+++ b/tools/perf/util/util.h
@@ -249,6 +249,7 @@ static inline int sane_case(int x, int high)
 }
 
 int mkdir_p(char *path, mode_t mode);
+int rm_rf(char *path);
 int copyfile(const char *from, const char *to);
 int copyfile_mode(const char *from, const char *to, mode_t mode);
 

^ permalink raw reply related	[flat|nested] 221+ messages in thread

* [tip:perf/core] perf tools: Introduce copyfile_offset() function
  2015-05-18  0:30 ` [PATCH 03/40] perf tools: Introduce copyfile_offset() function Namhyung Kim
  2015-05-18 12:57   ` Arnaldo Carvalho de Melo
@ 2015-05-20 12:24   ` tip-bot for Namhyung Kim
  1 sibling, 0 replies; 221+ messages in thread
From: tip-bot for Namhyung Kim @ 2015-05-20 12:24 UTC (permalink / raw)
  To: linux-tip-commits
  Cc: adrian.hunter, dsahern, andi, namhyung, tglx, a.p.zijlstra, hpa,
	mingo, eranian, acme, jolsa, linux-kernel, fweisbec

Commit-ID:  9c9f5a2f1944e8b6bf2b618d04b31e1c1760637e
Gitweb:     http://git.kernel.org/tip/9c9f5a2f1944e8b6bf2b618d04b31e1c1760637e
Author:     Namhyung Kim <namhyung@kernel.org>
AuthorDate: Mon, 18 May 2015 09:30:18 +0900
Committer:  Arnaldo Carvalho de Melo <acme@redhat.com>
CommitDate: Mon, 18 May 2015 10:17:35 -0300

perf tools: Introduce copyfile_offset() function

The copyfile_offset() function is to copy source data from given offset
to a destination file with an offset.  It'll be used to build an indexed
data file.

Signed-off-by: Namhyung Kim <namhyung@kernel.org>
Acked-by: Jiri Olsa <jolsa@kernel.org>
Cc: Adrian Hunter <adrian.hunter@intel.com>
Cc: Andi Kleen <andi@firstfloor.org>
Cc: David Ahern <dsahern@gmail.com>
Cc: Frederic Weisbecker <fweisbec@gmail.com>
Cc: Peter Zijlstra <a.p.zijlstra@chello.nl>
Cc: Stephane Eranian <eranian@google.com>
Link: http://lkml.kernel.org/r/20150304145824.GD7519@krava.brq.redhat.com
Link: http://lkml.kernel.org/r/1431909055-21442-4-git-send-email-namhyung@kernel.org
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
---
 tools/perf/util/util.c | 38 +++++++++++++++++++++++++++++---------
 tools/perf/util/util.h |  1 +
 2 files changed, 30 insertions(+), 9 deletions(-)

diff --git a/tools/perf/util/util.c b/tools/perf/util/util.c
index 6104afb..0c264bc 100644
--- a/tools/perf/util/util.c
+++ b/tools/perf/util/util.c
@@ -145,11 +145,38 @@ out:
 	return err;
 }
 
+int copyfile_offset(int ifd, loff_t off_in, int ofd, loff_t off_out, u64 size)
+{
+	void *ptr;
+	loff_t pgoff;
+
+	pgoff = off_in & ~(page_size - 1);
+	off_in -= pgoff;
+
+	ptr = mmap(NULL, off_in + size, PROT_READ, MAP_PRIVATE, ifd, pgoff);
+	if (ptr == MAP_FAILED)
+		return -1;
+
+	while (size) {
+		ssize_t ret = pwrite(ofd, ptr + off_in, size, off_out);
+		if (ret < 0 && errno == EINTR)
+			continue;
+		if (ret <= 0)
+			break;
+
+		size -= ret;
+		off_in += ret;
+		off_out -= ret;
+	}
+	munmap(ptr, off_in + size);
+
+	return size ? -1 : 0;
+}
+
 int copyfile_mode(const char *from, const char *to, mode_t mode)
 {
 	int fromfd, tofd;
 	struct stat st;
-	void *addr;
 	int err = -1;
 
 	if (stat(from, &st))
@@ -166,15 +193,8 @@ int copyfile_mode(const char *from, const char *to, mode_t mode)
 	if (tofd < 0)
 		goto out_close_from;
 
-	addr = mmap(NULL, st.st_size, PROT_READ, MAP_PRIVATE, fromfd, 0);
-	if (addr == MAP_FAILED)
-		goto out_close_to;
-
-	if (write(tofd, addr, st.st_size) == st.st_size)
-		err = 0;
+	err = copyfile_offset(fromfd, 0, tofd, 0, st.st_size);
 
-	munmap(addr, st.st_size);
-out_close_to:
 	close(tofd);
 	if (err)
 		unlink(to);
diff --git a/tools/perf/util/util.h b/tools/perf/util/util.h
index c4fe38a..8bce58b 100644
--- a/tools/perf/util/util.h
+++ b/tools/perf/util/util.h
@@ -252,6 +252,7 @@ int mkdir_p(char *path, mode_t mode);
 int rm_rf(char *path);
 int copyfile(const char *from, const char *to);
 int copyfile_mode(const char *from, const char *to, mode_t mode);
+int copyfile_offset(int fromfd, loff_t from_ofs, int tofd, loff_t to_ofs, u64 size);
 
 s64 perf_atoll(const char *str);
 char **argv_split(const char *str, int *argcp);

^ permalink raw reply related	[flat|nested] 221+ messages in thread

* [tip:perf/core] perf symbols: Protect dso symbol loading using a mutex
  2015-05-18  0:30 ` [PATCH 25/40] perf tools: Protect dso symbol loading using a mutex Namhyung Kim
@ 2015-05-20 12:25   ` tip-bot for Namhyung Kim
  0 siblings, 0 replies; 221+ messages in thread
From: tip-bot for Namhyung Kim @ 2015-05-20 12:25 UTC (permalink / raw)
  To: linux-tip-commits
  Cc: mingo, linux-kernel, a.p.zijlstra, jolsa, andi, tglx, eranian,
	hpa, adrian.hunter, namhyung, dsahern, fweisbec, acme

Commit-ID:  4a936edc317005e8cd2b501e7865721bec104544
Gitweb:     http://git.kernel.org/tip/4a936edc317005e8cd2b501e7865721bec104544
Author:     Namhyung Kim <namhyung@kernel.org>
AuthorDate: Mon, 18 May 2015 09:30:40 +0900
Committer:  Arnaldo Carvalho de Melo <acme@redhat.com>
CommitDate: Mon, 18 May 2015 10:17:36 -0300

perf symbols: Protect dso symbol loading using a mutex

Add mutex to protect it from concurrent dso__load().

Signed-off-by: Namhyung Kim <namhyung@kernel.org>
Cc: Adrian Hunter <adrian.hunter@intel.com>
Cc: Andi Kleen <andi@firstfloor.org>
Cc: David Ahern <dsahern@gmail.com>
Cc: Frederic Weisbecker <fweisbec@gmail.com>
Cc: Jiri Olsa <jolsa@redhat.com>
Cc: Peter Zijlstra <a.p.zijlstra@chello.nl>
Cc: Stephane Eranian <eranian@google.com>
Link: http://lkml.kernel.org/r/1431909055-21442-26-git-send-email-namhyung@kernel.org
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
---
 tools/perf/util/dso.c    |  2 ++
 tools/perf/util/dso.h    |  1 +
 tools/perf/util/symbol.c | 34 ++++++++++++++++++++++++----------
 3 files changed, 27 insertions(+), 10 deletions(-)

diff --git a/tools/perf/util/dso.c b/tools/perf/util/dso.c
index 13d9ae0..482d602 100644
--- a/tools/perf/util/dso.c
+++ b/tools/perf/util/dso.c
@@ -936,6 +936,7 @@ struct dso *dso__new(const char *name)
 		RB_CLEAR_NODE(&dso->rb_node);
 		INIT_LIST_HEAD(&dso->node);
 		INIT_LIST_HEAD(&dso->data.open_entry);
+		pthread_mutex_init(&dso->lock, NULL);
 	}
 
 	return dso;
@@ -966,6 +967,7 @@ void dso__delete(struct dso *dso)
 	dso_cache__free(&dso->data.cache);
 	dso__free_a2l(dso);
 	zfree(&dso->symsrc_filename);
+	pthread_mutex_destroy(&dso->lock);
 	free(dso);
 }
 
diff --git a/tools/perf/util/dso.h b/tools/perf/util/dso.h
index 3d79c74..b26ec3a 100644
--- a/tools/perf/util/dso.h
+++ b/tools/perf/util/dso.h
@@ -129,6 +129,7 @@ struct dsos {
 struct auxtrace_cache;
 
 struct dso {
+	pthread_mutex_t	 lock;
 	struct list_head node;
 	struct rb_node	 rb_node;	/* rbtree node sorted by long name */
 	struct rb_root	 symbols[MAP__NR_TYPES];
diff --git a/tools/perf/util/symbol.c b/tools/perf/util/symbol.c
index 45ba48a..9ef8b89 100644
--- a/tools/perf/util/symbol.c
+++ b/tools/perf/util/symbol.c
@@ -1383,12 +1383,22 @@ int dso__load(struct dso *dso, struct map *map, symbol_filter_t filter)
 	struct symsrc *syms_ss = NULL, *runtime_ss = NULL;
 	bool kmod;
 
-	dso__set_loaded(dso, map->type);
+	pthread_mutex_lock(&dso->lock);
+
+	/* check again under the dso->lock */
+	if (dso__loaded(dso, map->type)) {
+		ret = 1;
+		goto out;
+	}
+
+	if (dso->kernel) {
+		if (dso->kernel == DSO_TYPE_KERNEL)
+			ret = dso__load_kernel_sym(dso, map, filter);
+		else if (dso->kernel == DSO_TYPE_GUEST_KERNEL)
+			ret = dso__load_guest_kernel_sym(dso, map, filter);
 
-	if (dso->kernel == DSO_TYPE_KERNEL)
-		return dso__load_kernel_sym(dso, map, filter);
-	else if (dso->kernel == DSO_TYPE_GUEST_KERNEL)
-		return dso__load_guest_kernel_sym(dso, map, filter);
+		goto out;
+	}
 
 	if (map->groups && map->groups->machine)
 		machine = map->groups->machine;
@@ -1401,18 +1411,18 @@ int dso__load(struct dso *dso, struct map *map, symbol_filter_t filter)
 		struct stat st;
 
 		if (lstat(dso->name, &st) < 0)
-			return -1;
+			goto out;
 
 		if (st.st_uid && (st.st_uid != geteuid())) {
 			pr_warning("File %s not owned by current user or root, "
 				"ignoring it.\n", dso->name);
-			return -1;
+			goto out;
 		}
 
 		ret = dso__load_perf_map(dso, map, filter);
 		dso->symtab_type = ret > 0 ? DSO_BINARY_TYPE__JAVA_JIT :
 					     DSO_BINARY_TYPE__NOT_FOUND;
-		return ret;
+		goto out;
 	}
 
 	if (machine)
@@ -1420,7 +1430,7 @@ int dso__load(struct dso *dso, struct map *map, symbol_filter_t filter)
 
 	name = malloc(PATH_MAX);
 	if (!name)
-		return -1;
+		goto out;
 
 	kmod = dso->symtab_type == DSO_BINARY_TYPE__SYSTEM_PATH_KMODULE ||
 		dso->symtab_type == DSO_BINARY_TYPE__SYSTEM_PATH_KMODULE_COMP ||
@@ -1501,7 +1511,11 @@ int dso__load(struct dso *dso, struct map *map, symbol_filter_t filter)
 out_free:
 	free(name);
 	if (ret < 0 && strstr(dso->name, " (deleted)") != NULL)
-		return 0;
+		ret = 0;
+out:
+	dso__set_loaded(dso, map->type);
+	pthread_mutex_unlock(&dso->lock);
+
 	return ret;
 }
 

^ permalink raw reply related	[flat|nested] 221+ messages in thread

* [tip:perf/core] perf symbols: Protect dso cache tree using dso-> lock
  2015-05-18  0:30 ` [PATCH 26/40] perf tools: Protect dso cache tree using dso->lock Namhyung Kim
@ 2015-05-20 12:25   ` tip-bot for Namhyung Kim
  0 siblings, 0 replies; 221+ messages in thread
From: tip-bot for Namhyung Kim @ 2015-05-20 12:25 UTC (permalink / raw)
  To: linux-tip-commits
  Cc: tglx, jolsa, acme, andi, hpa, namhyung, dsahern, mingo,
	adrian.hunter, a.p.zijlstra, linux-kernel, eranian, fweisbec

Commit-ID:  8e67b7258e582998ab635bdc3c884d7a8077af5b
Gitweb:     http://git.kernel.org/tip/8e67b7258e582998ab635bdc3c884d7a8077af5b
Author:     Namhyung Kim <namhyung@kernel.org>
AuthorDate: Mon, 18 May 2015 09:30:41 +0900
Committer:  Arnaldo Carvalho de Melo <acme@redhat.com>
CommitDate: Mon, 18 May 2015 10:17:37 -0300

perf symbols: Protect dso cache tree using dso->lock

The dso cache is accessed during dwarf callchain unwind and it might be
processed concurrently.  Protect it under dso->lock.

Note that it doesn't protect dso_cache__find().  I think it's safe to
access to the cache tree without the lock since we don't delete nodes.

It it missed an existing node due to rotation, it'll find it during
dso_cache__insert() anyway.

Signed-off-by: Namhyung Kim <namhyung@kernel.org>
Cc: Adrian Hunter <adrian.hunter@intel.com>
Cc: Andi Kleen <andi@firstfloor.org>
Cc: David Ahern <dsahern@gmail.com>
Cc: Frederic Weisbecker <fweisbec@gmail.com>
Cc: Jiri Olsa <jolsa@redhat.com>
Cc: Peter Zijlstra <a.p.zijlstra@chello.nl>
Cc: Stephane Eranian <eranian@google.com>
Link: http://lkml.kernel.org/r/1431909055-21442-27-git-send-email-namhyung@kernel.org
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
---
 tools/perf/util/dso.c | 34 +++++++++++++++++++++++++++-------
 1 file changed, 27 insertions(+), 7 deletions(-)

diff --git a/tools/perf/util/dso.c b/tools/perf/util/dso.c
index 482d602..666e1db 100644
--- a/tools/perf/util/dso.c
+++ b/tools/perf/util/dso.c
@@ -495,10 +495,12 @@ bool dso__data_status_seen(struct dso *dso, enum dso_data_status_seen by)
 }
 
 static void
-dso_cache__free(struct rb_root *root)
+dso_cache__free(struct dso *dso)
 {
+	struct rb_root *root = &dso->data.cache;
 	struct rb_node *next = rb_first(root);
 
+	pthread_mutex_lock(&dso->lock);
 	while (next) {
 		struct dso_cache *cache;
 
@@ -507,10 +509,12 @@ dso_cache__free(struct rb_root *root)
 		rb_erase(&cache->rb_node, root);
 		free(cache);
 	}
+	pthread_mutex_unlock(&dso->lock);
 }
 
-static struct dso_cache *dso_cache__find(const struct rb_root *root, u64 offset)
+static struct dso_cache *dso_cache__find(struct dso *dso, u64 offset)
 {
+	const struct rb_root *root = &dso->data.cache;
 	struct rb_node * const *p = &root->rb_node;
 	const struct rb_node *parent = NULL;
 	struct dso_cache *cache;
@@ -529,17 +533,20 @@ static struct dso_cache *dso_cache__find(const struct rb_root *root, u64 offset)
 		else
 			return cache;
 	}
+
 	return NULL;
 }
 
-static void
-dso_cache__insert(struct rb_root *root, struct dso_cache *new)
+static struct dso_cache *
+dso_cache__insert(struct dso *dso, struct dso_cache *new)
 {
+	struct rb_root *root = &dso->data.cache;
 	struct rb_node **p = &root->rb_node;
 	struct rb_node *parent = NULL;
 	struct dso_cache *cache;
 	u64 offset = new->offset;
 
+	pthread_mutex_lock(&dso->lock);
 	while (*p != NULL) {
 		u64 end;
 
@@ -551,10 +558,17 @@ dso_cache__insert(struct rb_root *root, struct dso_cache *new)
 			p = &(*p)->rb_left;
 		else if (offset >= end)
 			p = &(*p)->rb_right;
+		else
+			goto out;
 	}
 
 	rb_link_node(&new->rb_node, parent, p);
 	rb_insert_color(&new->rb_node, root);
+
+	cache = NULL;
+out:
+	pthread_mutex_unlock(&dso->lock);
+	return cache;
 }
 
 static ssize_t
@@ -572,6 +586,7 @@ static ssize_t
 dso_cache__read(struct dso *dso, u64 offset, u8 *data, ssize_t size)
 {
 	struct dso_cache *cache;
+	struct dso_cache *old;
 	ssize_t ret;
 
 	do {
@@ -591,7 +606,12 @@ dso_cache__read(struct dso *dso, u64 offset, u8 *data, ssize_t size)
 
 		cache->offset = cache_offset;
 		cache->size   = ret;
-		dso_cache__insert(&dso->data.cache, cache);
+		old = dso_cache__insert(dso, cache);
+		if (old) {
+			/* we lose the race */
+			free(cache);
+			cache = old;
+		}
 
 		ret = dso_cache__memcpy(cache, offset, data, size);
 
@@ -608,7 +628,7 @@ static ssize_t dso_cache_read(struct dso *dso, u64 offset,
 {
 	struct dso_cache *cache;
 
-	cache = dso_cache__find(&dso->data.cache, offset);
+	cache = dso_cache__find(dso, offset);
 	if (cache)
 		return dso_cache__memcpy(cache, offset, data, size);
 	else
@@ -964,7 +984,7 @@ void dso__delete(struct dso *dso)
 
 	dso__data_close(dso);
 	auxtrace_cache__free(dso->auxtrace_cache);
-	dso_cache__free(&dso->data.cache);
+	dso_cache__free(dso);
 	dso__free_a2l(dso);
 	zfree(&dso->symsrc_filename);
 	pthread_mutex_destroy(&dso->lock);

^ permalink raw reply related	[flat|nested] 221+ messages in thread

* [tip:perf/core] perf tools: Protect dso cache fd with a mutex
  2015-05-18  0:30 ` [PATCH 27/40] perf tools: Protect dso cache fd with a mutex Namhyung Kim
@ 2015-05-20 12:25   ` tip-bot for Namhyung Kim
  0 siblings, 0 replies; 221+ messages in thread
From: tip-bot for Namhyung Kim @ 2015-05-20 12:25 UTC (permalink / raw)
  To: linux-tip-commits
  Cc: a.p.zijlstra, linux-kernel, fweisbec, acme, andi, adrian.hunter,
	dsahern, hpa, jolsa, eranian, tglx, namhyung, mingo

Commit-ID:  33bdedcea2d77231fe46b1204cf6fc3a7d7c96c9
Gitweb:     http://git.kernel.org/tip/33bdedcea2d77231fe46b1204cf6fc3a7d7c96c9
Author:     Namhyung Kim <namhyung@kernel.org>
AuthorDate: Mon, 18 May 2015 09:30:42 +0900
Committer:  Arnaldo Carvalho de Melo <acme@redhat.com>
CommitDate: Mon, 18 May 2015 10:17:38 -0300

perf tools: Protect dso cache fd with a mutex

When dso cache is accessed in multi-thread environment, it's possible to
close other dso->data.fd during operation due to open file limit.
Protect the file descriptors using a separate mutex.

Signed-off-by: Namhyung Kim <namhyung@kernel.org>
Cc: Adrian Hunter <adrian.hunter@intel.com>
Cc: Andi Kleen <andi@firstfloor.org>
Cc: David Ahern <dsahern@gmail.com>
Cc: Frederic Weisbecker <fweisbec@gmail.com>
Cc: Jiri Olsa <jolsa@redhat.com>
Cc: Peter Zijlstra <a.p.zijlstra@chello.nl>
Cc: Stephane Eranian <eranian@google.com>
Link: http://lkml.kernel.org/r/1431909055-21442-28-git-send-email-namhyung@kernel.org
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
---
 tools/perf/util/dso.c | 98 +++++++++++++++++++++++++++++++++++++--------------
 1 file changed, 72 insertions(+), 26 deletions(-)

diff --git a/tools/perf/util/dso.c b/tools/perf/util/dso.c
index 666e1db..1b96c8d 100644
--- a/tools/perf/util/dso.c
+++ b/tools/perf/util/dso.c
@@ -265,6 +265,7 @@ int __kmod_path__parse(struct kmod_path *m, const char *path,
  */
 static LIST_HEAD(dso__data_open);
 static long dso__data_open_cnt;
+static pthread_mutex_t dso__data_open_lock = PTHREAD_MUTEX_INITIALIZER;
 
 static void dso__list_add(struct dso *dso)
 {
@@ -434,7 +435,9 @@ static void check_data_close(void)
  */
 void dso__data_close(struct dso *dso)
 {
+	pthread_mutex_lock(&dso__data_open_lock);
 	close_dso(dso);
+	pthread_mutex_unlock(&dso__data_open_lock);
 }
 
 /**
@@ -457,6 +460,8 @@ int dso__data_fd(struct dso *dso, struct machine *machine)
 	if (dso->data.status == DSO_DATA_STATUS_ERROR)
 		return -1;
 
+	pthread_mutex_lock(&dso__data_open_lock);
+
 	if (dso->data.fd >= 0)
 		goto out;
 
@@ -479,6 +484,7 @@ out:
 	else
 		dso->data.status = DSO_DATA_STATUS_ERROR;
 
+	pthread_mutex_unlock(&dso__data_open_lock);
 	return dso->data.fd;
 }
 
@@ -583,7 +589,8 @@ dso_cache__memcpy(struct dso_cache *cache, u64 offset,
 }
 
 static ssize_t
-dso_cache__read(struct dso *dso, u64 offset, u8 *data, ssize_t size)
+dso_cache__read(struct dso *dso, struct machine *machine,
+		u64 offset, u8 *data, ssize_t size)
 {
 	struct dso_cache *cache;
 	struct dso_cache *old;
@@ -592,11 +599,24 @@ dso_cache__read(struct dso *dso, u64 offset, u8 *data, ssize_t size)
 	do {
 		u64 cache_offset;
 
-		ret = -ENOMEM;
-
 		cache = zalloc(sizeof(*cache) + DSO__DATA_CACHE_SIZE);
 		if (!cache)
-			break;
+			return -ENOMEM;
+
+		pthread_mutex_lock(&dso__data_open_lock);
+
+		/*
+		 * dso->data.fd might be closed if other thread opened another
+		 * file (dso) due to open file limit (RLIMIT_NOFILE).
+		 */
+		if (dso->data.fd < 0) {
+			dso->data.fd = open_dso(dso, machine);
+			if (dso->data.fd < 0) {
+				ret = -errno;
+				dso->data.status = DSO_DATA_STATUS_ERROR;
+				break;
+			}
+		}
 
 		cache_offset = offset & DSO__DATA_CACHE_MASK;
 
@@ -606,6 +626,11 @@ dso_cache__read(struct dso *dso, u64 offset, u8 *data, ssize_t size)
 
 		cache->offset = cache_offset;
 		cache->size   = ret;
+	} while (0);
+
+	pthread_mutex_unlock(&dso__data_open_lock);
+
+	if (ret > 0) {
 		old = dso_cache__insert(dso, cache);
 		if (old) {
 			/* we lose the race */
@@ -614,8 +639,7 @@ dso_cache__read(struct dso *dso, u64 offset, u8 *data, ssize_t size)
 		}
 
 		ret = dso_cache__memcpy(cache, offset, data, size);
-
-	} while (0);
+	}
 
 	if (ret <= 0)
 		free(cache);
@@ -623,8 +647,8 @@ dso_cache__read(struct dso *dso, u64 offset, u8 *data, ssize_t size)
 	return ret;
 }
 
-static ssize_t dso_cache_read(struct dso *dso, u64 offset,
-			      u8 *data, ssize_t size)
+static ssize_t dso_cache_read(struct dso *dso, struct machine *machine,
+			      u64 offset, u8 *data, ssize_t size)
 {
 	struct dso_cache *cache;
 
@@ -632,7 +656,7 @@ static ssize_t dso_cache_read(struct dso *dso, u64 offset,
 	if (cache)
 		return dso_cache__memcpy(cache, offset, data, size);
 	else
-		return dso_cache__read(dso, offset, data, size);
+		return dso_cache__read(dso, machine, offset, data, size);
 }
 
 /*
@@ -640,7 +664,8 @@ static ssize_t dso_cache_read(struct dso *dso, u64 offset,
  * in the rb_tree. Any read to already cached data is served
  * by cached data.
  */
-static ssize_t cached_read(struct dso *dso, u64 offset, u8 *data, ssize_t size)
+static ssize_t cached_read(struct dso *dso, struct machine *machine,
+			   u64 offset, u8 *data, ssize_t size)
 {
 	ssize_t r = 0;
 	u8 *p = data;
@@ -648,7 +673,7 @@ static ssize_t cached_read(struct dso *dso, u64 offset, u8 *data, ssize_t size)
 	do {
 		ssize_t ret;
 
-		ret = dso_cache_read(dso, offset, p, size);
+		ret = dso_cache_read(dso, machine, offset, p, size);
 		if (ret < 0)
 			return ret;
 
@@ -668,21 +693,42 @@ static ssize_t cached_read(struct dso *dso, u64 offset, u8 *data, ssize_t size)
 	return r;
 }
 
-static int data_file_size(struct dso *dso)
+static int data_file_size(struct dso *dso, struct machine *machine)
 {
+	int ret = 0;
 	struct stat st;
 	char sbuf[STRERR_BUFSIZE];
 
-	if (!dso->data.file_size) {
-		if (fstat(dso->data.fd, &st)) {
-			pr_err("dso mmap failed, fstat: %s\n",
-				strerror_r(errno, sbuf, sizeof(sbuf)));
-			return -1;
+	if (dso->data.file_size)
+		return 0;
+
+	pthread_mutex_lock(&dso__data_open_lock);
+
+	/*
+	 * dso->data.fd might be closed if other thread opened another
+	 * file (dso) due to open file limit (RLIMIT_NOFILE).
+	 */
+	if (dso->data.fd < 0) {
+		dso->data.fd = open_dso(dso, machine);
+		if (dso->data.fd < 0) {
+			ret = -errno;
+			dso->data.status = DSO_DATA_STATUS_ERROR;
+			goto out;
 		}
-		dso->data.file_size = st.st_size;
 	}
 
-	return 0;
+	if (fstat(dso->data.fd, &st) < 0) {
+		ret = -errno;
+		pr_err("dso cache fstat failed: %s\n",
+		       strerror_r(errno, sbuf, sizeof(sbuf)));
+		dso->data.status = DSO_DATA_STATUS_ERROR;
+		goto out;
+	}
+	dso->data.file_size = st.st_size;
+
+out:
+	pthread_mutex_unlock(&dso__data_open_lock);
+	return ret;
 }
 
 /**
@@ -700,17 +746,17 @@ off_t dso__data_size(struct dso *dso, struct machine *machine)
 	if (fd < 0)
 		return fd;
 
-	if (data_file_size(dso))
+	if (data_file_size(dso, machine))
 		return -1;
 
 	/* For now just estimate dso data size is close to file size */
 	return dso->data.file_size;
 }
 
-static ssize_t data_read_offset(struct dso *dso, u64 offset,
-				u8 *data, ssize_t size)
+static ssize_t data_read_offset(struct dso *dso, struct machine *machine,
+				u64 offset, u8 *data, ssize_t size)
 {
-	if (data_file_size(dso))
+	if (data_file_size(dso, machine))
 		return -1;
 
 	/* Check the offset sanity. */
@@ -720,7 +766,7 @@ static ssize_t data_read_offset(struct dso *dso, u64 offset,
 	if (offset + size < offset)
 		return -1;
 
-	return cached_read(dso, offset, data, size);
+	return cached_read(dso, machine, offset, data, size);
 }
 
 /**
@@ -737,10 +783,10 @@ static ssize_t data_read_offset(struct dso *dso, u64 offset,
 ssize_t dso__data_read_offset(struct dso *dso, struct machine *machine,
 			      u64 offset, u8 *data, ssize_t size)
 {
-	if (dso__data_fd(dso, machine) < 0)
+	if (dso->data.status == DSO_DATA_STATUS_ERROR)
 		return -1;
 
-	return data_read_offset(dso, offset, data, size);
+	return data_read_offset(dso, machine, offset, data, size);
 }
 
 /**

^ permalink raw reply related	[flat|nested] 221+ messages in thread

* [tip:perf/core] perf hists: Reducing arguments of hist_entry_iter__add()
  2015-05-19  8:04       ` [PATCH] " Namhyung Kim
  2015-05-19 14:03         ` Arnaldo Carvalho de Melo
@ 2015-05-27 16:47         ` tip-bot for Namhyung Kim
  1 sibling, 0 replies; 221+ messages in thread
From: tip-bot for Namhyung Kim @ 2015-05-27 16:47 UTC (permalink / raw)
  To: linux-tip-commits
  Cc: dsahern, acme, namhyung, hpa, a.p.zijlstra, tglx, mingo, jolsa,
	linux-kernel

Commit-ID:  063bd9363bb8979b2939bdc0412d98a8ac062e3b
Gitweb:     http://git.kernel.org/tip/063bd9363bb8979b2939bdc0412d98a8ac062e3b
Author:     Namhyung Kim <namhyung@kernel.org>
AuthorDate: Tue, 19 May 2015 17:04:10 +0900
Committer:  Arnaldo Carvalho de Melo <acme@redhat.com>
CommitDate: Wed, 27 May 2015 12:21:43 -0300

perf hists: Reducing arguments of hist_entry_iter__add()

The evsel and sample arguments are to set iter for later use.  As it
also receives an iter as another argument, just set them before calling
the function.

Signed-off-by: Namhyung Kim <namhyung@kernel.org>
Cc: David Ahern <dsahern@gmail.com>
Cc: Jiri Olsa <jolsa@redhat.com>
Cc: Peter Zijlstra <a.p.zijlstra@chello.nl>
Link: http://lkml.kernel.org/r/1432022650-18205-1-git-send-email-namhyung@kernel.org
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
---
 tools/perf/builtin-report.c       | 9 +++++----
 tools/perf/builtin-top.c          | 7 ++++---
 tools/perf/tests/hists_cumulate.c | 6 ++++--
 tools/perf/tests/hists_filter.c   | 4 +++-
 tools/perf/tests/hists_output.c   | 6 ++++--
 tools/perf/util/hist.c            | 8 ++------
 tools/perf/util/hist.h            | 1 -
 7 files changed, 22 insertions(+), 19 deletions(-)

diff --git a/tools/perf/builtin-report.c b/tools/perf/builtin-report.c
index 92fca21..56025d9 100644
--- a/tools/perf/builtin-report.c
+++ b/tools/perf/builtin-report.c
@@ -139,8 +139,10 @@ static int process_sample_event(struct perf_tool *tool,
 	struct report *rep = container_of(tool, struct report, tool);
 	struct addr_location al;
 	struct hist_entry_iter iter = {
-		.hide_unresolved = rep->hide_unresolved,
-		.add_entry_cb = hist_iter__report_callback,
+		.evsel 			= evsel,
+		.sample 		= sample,
+		.hide_unresolved 	= rep->hide_unresolved,
+		.add_entry_cb 		= hist_iter__report_callback,
 	};
 	int ret = 0;
 
@@ -168,8 +170,7 @@ static int process_sample_event(struct perf_tool *tool,
 	if (al.map != NULL)
 		al.map->dso->hit = 1;
 
-	ret = hist_entry_iter__add(&iter, &al, evsel, sample, rep->max_stack,
-				   rep);
+	ret = hist_entry_iter__add(&iter, &al, rep->max_stack, rep);
 	if (ret < 0)
 		pr_debug("problem adding hist entry, skipping event\n");
 out_put:
diff --git a/tools/perf/builtin-top.c b/tools/perf/builtin-top.c
index a193517..6b98742 100644
--- a/tools/perf/builtin-top.c
+++ b/tools/perf/builtin-top.c
@@ -775,7 +775,9 @@ static void perf_event__process_sample(struct perf_tool *tool,
 	if (al.sym == NULL || !al.sym->ignore) {
 		struct hists *hists = evsel__hists(evsel);
 		struct hist_entry_iter iter = {
-			.add_entry_cb = hist_iter__top_callback,
+			.evsel		= evsel,
+			.sample 	= sample,
+			.add_entry_cb 	= hist_iter__top_callback,
 		};
 
 		if (symbol_conf.cumulate_callchain)
@@ -785,8 +787,7 @@ static void perf_event__process_sample(struct perf_tool *tool,
 
 		pthread_mutex_lock(&hists->lock);
 
-		err = hist_entry_iter__add(&iter, &al, evsel, sample,
-					   top->max_stack, top);
+		err = hist_entry_iter__add(&iter, &al, top->max_stack, top);
 		if (err < 0)
 			pr_err("Problem incrementing symbol period, skipping event\n");
 
diff --git a/tools/perf/tests/hists_cumulate.c b/tools/perf/tests/hists_cumulate.c
index 620f626..7d82c8b 100644
--- a/tools/perf/tests/hists_cumulate.c
+++ b/tools/perf/tests/hists_cumulate.c
@@ -87,6 +87,8 @@ static int add_hist_entries(struct hists *hists, struct machine *machine)
 			},
 		};
 		struct hist_entry_iter iter = {
+			.evsel = evsel,
+			.sample	= &sample,
 			.hide_unresolved = false,
 		};
 
@@ -104,8 +106,8 @@ static int add_hist_entries(struct hists *hists, struct machine *machine)
 						  &sample) < 0)
 			goto out;
 
-		if (hist_entry_iter__add(&iter, &al, evsel, &sample,
-					 PERF_MAX_STACK_DEPTH, NULL) < 0) {
+		if (hist_entry_iter__add(&iter, &al, PERF_MAX_STACK_DEPTH,
+					 NULL) < 0) {
 			addr_location__put(&al);
 			goto out;
 		}
diff --git a/tools/perf/tests/hists_filter.c b/tools/perf/tests/hists_filter.c
index 82e1ee5..ce48775 100644
--- a/tools/perf/tests/hists_filter.c
+++ b/tools/perf/tests/hists_filter.c
@@ -63,6 +63,8 @@ static int add_hist_entries(struct perf_evlist *evlist,
 				},
 			};
 			struct hist_entry_iter iter = {
+				.evsel = evsel,
+				.sample = &sample,
 				.ops = &hist_iter_normal,
 				.hide_unresolved = false,
 			};
@@ -81,7 +83,7 @@ static int add_hist_entries(struct perf_evlist *evlist,
 							  &sample) < 0)
 				goto out;
 
-			if (hist_entry_iter__add(&iter, &al, evsel, &sample,
+			if (hist_entry_iter__add(&iter, &al,
 						 PERF_MAX_STACK_DEPTH, NULL) < 0) {
 				addr_location__put(&al);
 				goto out;
diff --git a/tools/perf/tests/hists_output.c b/tools/perf/tests/hists_output.c
index fd7ec4f..adbebc8 100644
--- a/tools/perf/tests/hists_output.c
+++ b/tools/perf/tests/hists_output.c
@@ -57,6 +57,8 @@ static int add_hist_entries(struct hists *hists, struct machine *machine)
 			},
 		};
 		struct hist_entry_iter iter = {
+			.evsel = evsel,
+			.sample = &sample,
 			.ops = &hist_iter_normal,
 			.hide_unresolved = false,
 		};
@@ -70,8 +72,8 @@ static int add_hist_entries(struct hists *hists, struct machine *machine)
 						  &sample) < 0)
 			goto out;
 
-		if (hist_entry_iter__add(&iter, &al, evsel, &sample,
-					 PERF_MAX_STACK_DEPTH, NULL) < 0) {
+		if (hist_entry_iter__add(&iter, &al, PERF_MAX_STACK_DEPTH,
+					 NULL) < 0) {
 			addr_location__put(&al);
 			goto out;
 		}
diff --git a/tools/perf/util/hist.c b/tools/perf/util/hist.c
index 3387706..2504b5b 100644
--- a/tools/perf/util/hist.c
+++ b/tools/perf/util/hist.c
@@ -851,19 +851,15 @@ const struct hist_iter_ops hist_iter_cumulative = {
 };
 
 int hist_entry_iter__add(struct hist_entry_iter *iter, struct addr_location *al,
-			 struct perf_evsel *evsel, struct perf_sample *sample,
 			 int max_stack_depth, void *arg)
 {
 	int err, err2;
 
-	err = sample__resolve_callchain(sample, &iter->parent, evsel, al,
-					max_stack_depth);
+	err = sample__resolve_callchain(iter->sample, &iter->parent,
+					iter->evsel, al, max_stack_depth);
 	if (err)
 		return err;
 
-	iter->evsel = evsel;
-	iter->sample = sample;
-
 	err = iter->ops->prepare_entry(iter, al);
 	if (err)
 		goto out;
diff --git a/tools/perf/util/hist.h b/tools/perf/util/hist.h
index 9f31b89..5ed8d9c 100644
--- a/tools/perf/util/hist.h
+++ b/tools/perf/util/hist.h
@@ -111,7 +111,6 @@ struct hist_entry *__hists__add_entry(struct hists *hists,
 				      u64 weight, u64 transaction,
 				      bool sample_self);
 int hist_entry_iter__add(struct hist_entry_iter *iter, struct addr_location *al,
-			 struct perf_evsel *evsel, struct perf_sample *sample,
 			 int max_stack_depth, void *arg);
 
 int64_t hist_entry__cmp(struct hist_entry *left, struct hist_entry *right);

^ permalink raw reply related	[flat|nested] 221+ messages in thread

* [RFC 0/2] perf session: Separate struct machines from session
  2015-05-19 23:58         ` Namhyung Kim
@ 2015-05-28  2:39           ` Namhyung Kim
  2015-05-28  2:39             ` [PATCH 1/2] perf tools: Introduce machines__new/delete() Namhyung Kim
  2015-05-28  2:39             ` [PATCH 2/2] perf session: Separate struct machines from session Namhyung Kim
  0 siblings, 2 replies; 221+ messages in thread
From: Namhyung Kim @ 2015-05-28  2:39 UTC (permalink / raw)
  To: Arnaldo Carvalho de Melo
  Cc: Ingo Molnar, Peter Zijlstra, Jiri Olsa, LKML, David Ahern

Hi Arnaldo,

This is what I have now for the earlier discussion [1].  I just used
machines__new/delete functions directly whenever a session is
created/deleted.  As it touches many callsites, and it needs bit more
caring about the error path, you may want to split the first patch for
each command.  In any case, pleaes let me know what do you think.

Note that this patchset bases on my (slightely modified) previous
multi-thread patchset so will not be applied to the current perf/core.
I'll rebase and send out after we conclude how to deal with it.

Thanks,
Namhyung


[1] https://lkml.org/lkml/2015/5/19/1062

Namhyung Kim (2):
  perf tools: Introduce machines__new/delete()
  perf session: Separate struct machines from session

 tools/perf/builtin-annotate.c      |  7 ++++++-
 tools/perf/builtin-buildid-cache.c | 14 ++++++++++++--
 tools/perf/builtin-buildid-list.c  | 16 +++++++++++++---
 tools/perf/builtin-diff.c          | 16 ++++++++++++++--
 tools/perf/builtin-evlist.c        | 18 +++++++++++++++---
 tools/perf/builtin-inject.c        |  5 +++++
 tools/perf/builtin-kmem.c          | 14 +++++++++-----
 tools/perf/builtin-kvm.c           | 14 +++++++++++++-
 tools/perf/builtin-lock.c          |  7 ++++++-
 tools/perf/builtin-mem.c           | 12 +++++++++---
 tools/perf/builtin-record.c        | 14 ++++++++++----
 tools/perf/builtin-report.c        | 13 ++++++++++---
 tools/perf/builtin-sched.c         |  8 +++++++-
 tools/perf/builtin-script.c        | 32 ++++++++++++++++++++++++--------
 tools/perf/builtin-timechart.c     | 12 +++++++++---
 tools/perf/builtin-top.c           | 23 ++++++++++++++++-------
 tools/perf/builtin-trace.c         | 10 ++++++++--
 tools/perf/util/build-id.c         | 16 ++++++++--------
 tools/perf/util/data-convert-bt.c  |  8 +++++++-
 tools/perf/util/intel-bts.c        |  2 +-
 tools/perf/util/intel-pt.c         |  2 +-
 tools/perf/util/machine.c          | 15 +++++++++++++++
 tools/perf/util/machine.h          |  3 +++
 tools/perf/util/session.c          | 30 +++++++++++++++---------------
 tools/perf/util/session.h          |  6 +++---
 tools/perf/util/tool.h             |  2 ++
 26 files changed, 241 insertions(+), 78 deletions(-)

-- 
2.4.1


^ permalink raw reply	[flat|nested] 221+ messages in thread

* [PATCH 1/2] perf tools: Introduce machines__new/delete()
  2015-05-28  2:39           ` [RFC 0/2] " Namhyung Kim
@ 2015-05-28  2:39             ` Namhyung Kim
  2015-05-28  2:39             ` [PATCH 2/2] perf session: Separate struct machines from session Namhyung Kim
  1 sibling, 0 replies; 221+ messages in thread
From: Namhyung Kim @ 2015-05-28  2:39 UTC (permalink / raw)
  To: Arnaldo Carvalho de Melo
  Cc: Ingo Molnar, Peter Zijlstra, Jiri Olsa, LKML, David Ahern

Create and maintain struct machines in struct perf_tool and pass it to
struct perf_session.  This is a preparation of sharing machines among
sessions in case of multi-thread report.

Suggested-by: Arnaldo Carvalho de Melo <acme@kernel.org>
Signed-off-by: Namhyung Kim <namhyung@kernel.org>
---
 tools/perf/builtin-annotate.c      |  5 +++++
 tools/perf/builtin-buildid-cache.c | 14 ++++++++++++--
 tools/perf/builtin-buildid-list.c  | 16 +++++++++++++---
 tools/perf/builtin-diff.c          | 16 ++++++++++++++--
 tools/perf/builtin-evlist.c        | 18 +++++++++++++++---
 tools/perf/builtin-inject.c        |  5 +++++
 tools/perf/builtin-kmem.c          |  4 ++++
 tools/perf/builtin-kvm.c           | 12 ++++++++++++
 tools/perf/builtin-lock.c          |  7 ++++++-
 tools/perf/builtin-mem.c           | 12 +++++++++---
 tools/perf/builtin-record.c        |  8 +++++++-
 tools/perf/builtin-report.c        |  9 ++++++++-
 tools/perf/builtin-sched.c         |  8 +++++++-
 tools/perf/builtin-script.c        | 32 ++++++++++++++++++++++++--------
 tools/perf/builtin-timechart.c     | 12 +++++++++---
 tools/perf/builtin-top.c           | 15 ++++++++++++---
 tools/perf/builtin-trace.c         |  8 +++++++-
 tools/perf/util/data-convert-bt.c  |  8 +++++++-
 tools/perf/util/machine.c          | 15 +++++++++++++++
 tools/perf/util/machine.h          |  3 +++
 tools/perf/util/tool.h             |  2 ++
 21 files changed, 196 insertions(+), 33 deletions(-)

diff --git a/tools/perf/builtin-annotate.c b/tools/perf/builtin-annotate.c
index 761f902473b7..e14a8a4d2f8d 100644
--- a/tools/perf/builtin-annotate.c
+++ b/tools/perf/builtin-annotate.c
@@ -345,6 +345,10 @@ int cmd_annotate(int argc, const char **argv, const char *prefix __maybe_unused)
 
 	setup_browser(true);
 
+	annotate.tool.machines = machines__new();
+	if (annotate.tool.machines == NULL)
+		return -1;
+
 	annotate.session = perf_session__new(&file, false, &annotate.tool);
 	if (annotate.session == NULL)
 		return -1;
@@ -384,6 +388,7 @@ int cmd_annotate(int argc, const char **argv, const char *prefix __maybe_unused)
 	 * up we'll get here.
 	 *
 	 * perf_session__delete(session);
+	 * machines__delete(annotate.tool.machines);
 	 */
 	return ret;
 }
diff --git a/tools/perf/builtin-buildid-cache.c b/tools/perf/builtin-buildid-cache.c
index d47a0cdc71c9..ff3eac1830fc 100644
--- a/tools/perf/builtin-buildid-cache.c
+++ b/tools/perf/builtin-buildid-cache.c
@@ -317,6 +317,9 @@ int cmd_buildid_cache(int argc, const char **argv,
 		.mode  = PERF_DATA_MODE_READ,
 	};
 	struct perf_session *session = NULL;
+	struct perf_tool tool = {
+		.machines = NULL,
+	};
 
 	const struct option buildid_cache_options[] = {
 	OPT_STRING('a', "add", &add_name_list_str,
@@ -352,9 +355,14 @@ int cmd_buildid_cache(int argc, const char **argv,
 		file.path = missing_filename;
 		file.force = force;
 
-		session = perf_session__new(&file, false, NULL);
-		if (session == NULL)
+		ret = -1;
+
+		tool.machines = machines__new();
+		if (tool.machines == NULL)
 			return -1;
+		session = perf_session__new(&file, false, &tool);
+		if (session == NULL)
+			goto out;
 	}
 
 	if (symbol__init(session ? &session->header.env : NULL) < 0)
@@ -443,6 +451,8 @@ int cmd_buildid_cache(int argc, const char **argv,
 out:
 	if (session)
 		perf_session__delete(session);
+	if (tool.machines)
+		machines__delete(tool.machines);
 
 	return ret;
 }
diff --git a/tools/perf/builtin-buildid-list.c b/tools/perf/builtin-buildid-list.c
index 9fe93c8d4fcf..2ab7d20a30a4 100644
--- a/tools/perf/builtin-buildid-list.c
+++ b/tools/perf/builtin-buildid-list.c
@@ -58,6 +58,8 @@ static int perf_session__list_build_ids(bool force, bool with_hits)
 		.mode  = PERF_DATA_MODE_READ,
 		.force = force,
 	};
+	struct perf_tool *tool = &build_id__mark_dso_hit_ops;
+	int ret = -1;
 
 	symbol__elf_init();
 	/*
@@ -66,10 +68,14 @@ static int perf_session__list_build_ids(bool force, bool with_hits)
 	if (filename__fprintf_build_id(input_name, stdout))
 		goto out;
 
-	session = perf_session__new(&file, false, &build_id__mark_dso_hit_ops);
-	if (session == NULL)
+	tool->machines = machines__new();
+	if (tool->machines == NULL)
 		return -1;
 
+	session = perf_session__new(&file, false, tool);
+	if (session == NULL)
+		goto out_delete;
+
 	/*
 	 * We take all buildids when the file contains AUX area tracing data
 	 * because we do not decode the trace because it would take too long.
@@ -86,9 +92,13 @@ static int perf_session__list_build_ids(bool force, bool with_hits)
 		perf_session__process_events(session);
 
 	perf_session__fprintf_dsos_buildid(session, stdout, dso__skip_buildid, with_hits);
+	ret = 0;
+
 	perf_session__delete(session);
+out_delete:
+	machines__delete(tool->machines);
 out:
-	return 0;
+	return ret;
 }
 
 int cmd_buildid_list(int argc, const char **argv,
diff --git a/tools/perf/builtin-diff.c b/tools/perf/builtin-diff.c
index 0fe54a633a5e..a4431f096697 100644
--- a/tools/perf/builtin-diff.c
+++ b/tools/perf/builtin-diff.c
@@ -743,10 +743,17 @@ static int __cmd_diff(void)
 	int ret = -EINVAL, i;
 
 	data__for_each_file(i, d) {
+		ret = -ENOMEM;
+
+		tool.machines = machines__new();
+		if (tool.machines == NULL) {
+			pr_err("Failed to allocate machines\n");
+			goto out_delete;
+		}
+
 		d->session = perf_session__new(&d->file, false, &tool);
 		if (!d->session) {
 			pr_err("Failed to open %s\n", d->file.path);
-			ret = -1;
 			goto out_delete;
 		}
 
@@ -763,8 +770,13 @@ static int __cmd_diff(void)
 
  out_delete:
 	data__for_each_file(i, d) {
-		if (d->session)
+		if (d->session) {
+			struct machines *machines = &d->session->machines;
+
 			perf_session__delete(d->session);
+			if (machines)
+				machines__delete(machines);
+		}
 
 		data__free(d);
 	}
diff --git a/tools/perf/builtin-evlist.c b/tools/perf/builtin-evlist.c
index 695ec5a50cf2..d742f987fde2 100644
--- a/tools/perf/builtin-evlist.c
+++ b/tools/perf/builtin-evlist.c
@@ -26,16 +26,28 @@ static int __cmd_evlist(const char *file_name, struct perf_attr_details *details
 		.mode = PERF_DATA_MODE_READ,
 		.force = details->force,
 	};
+	struct perf_tool tool = {
+		.machines = NULL,
+	};
+	int ret = -1;
 
-	session = perf_session__new(&file, 0, NULL);
-	if (session == NULL)
+	tool.machines = machines__new();
+	if (tool.machines == NULL)
 		return -1;
 
+	session = perf_session__new(&file, 0, &tool);
+	if (session == NULL)
+		goto out;
+
 	evlist__for_each(session->evlist, pos)
 		perf_evsel__fprintf(pos, details, stdout);
 
+	ret = 0;
+
 	perf_session__delete(session);
-	return 0;
+out:
+	machines__delete(tool.machines);
+	return ret;
 }
 
 int cmd_evlist(int argc, const char **argv, const char *prefix __maybe_unused)
diff --git a/tools/perf/builtin-inject.c b/tools/perf/builtin-inject.c
index 52ec66b23607..4dc8164b4927 100644
--- a/tools/perf/builtin-inject.c
+++ b/tools/perf/builtin-inject.c
@@ -625,6 +625,10 @@ int cmd_inject(int argc, const char **argv, const char *prefix __maybe_unused)
 
 	inject.tool.ordered_events = inject.sched_stat;
 
+	inject.tool.machines = machines__new();
+	if (inject.tool.machines == NULL)
+		return -1;
+
 	file.path = inject.input_name;
 	inject.session = perf_session__new(&file, true, &inject.tool);
 	if (inject.session == NULL)
@@ -636,6 +640,7 @@ int cmd_inject(int argc, const char **argv, const char *prefix __maybe_unused)
 	ret = __cmd_inject(&inject);
 
 	perf_session__delete(inject.session);
+	machines__delete(inject.tool.machines);
 
 	return ret;
 }
diff --git a/tools/perf/builtin-kmem.c b/tools/perf/builtin-kmem.c
index 254614b10c4a..979a032fe85f 100644
--- a/tools/perf/builtin-kmem.c
+++ b/tools/perf/builtin-kmem.c
@@ -1908,6 +1908,9 @@ int cmd_kmem(int argc, const char **argv, const char *prefix __maybe_unused)
 
 	file.path = input_name;
 
+	perf_kmem.machines = machines__new();
+	if (perf_kmem.machines == NULL)
+		return -1;
 	kmem_session = session = perf_session__new(&file, false, &perf_kmem);
 	if (session == NULL)
 		return -1;
@@ -1963,6 +1966,7 @@ int cmd_kmem(int argc, const char **argv, const char *prefix __maybe_unused)
 
 out_delete:
 	perf_session__delete(session);
+	machines__delete(perf_kmem.machines);
 
 	return ret;
 }
diff --git a/tools/perf/builtin-kvm.c b/tools/perf/builtin-kvm.c
index 15fecd3dc5d8..ca8836392af2 100644
--- a/tools/perf/builtin-kvm.c
+++ b/tools/perf/builtin-kvm.c
@@ -1053,6 +1053,10 @@ static int read_events(struct perf_kvm_stat *kvm)
 	};
 
 	kvm->tool = eops;
+	kvm->tool.machines = machines__new();
+	if (kvm->tool.machines == NULL)
+		return -1;
+
 	kvm->session = perf_session__new(&file, false, &kvm->tool);
 	if (!kvm->session) {
 		pr_err("Initializing perf session failed\n");
@@ -1381,6 +1385,12 @@ static int kvm_events_live(struct perf_kvm_stat *kvm,
 	if (perf_evlist__create_maps(kvm->evlist, &kvm->opts.target) < 0)
 		usage_with_options(live_usage, live_options);
 
+	kvm->tool.machines = machines__new();
+	if (kvm->tool.machines == NULL) {
+		err = -1;
+		goto out;
+	}
+
 	/*
 	 * perf session
 	 */
@@ -1406,6 +1416,8 @@ static int kvm_events_live(struct perf_kvm_stat *kvm,
 	if (kvm->session)
 		perf_session__delete(kvm->session);
 	kvm->session = NULL;
+	if (kvm->tool.machines)
+		machines__delete(kvm->tool.machines);
 	if (kvm->evlist)
 		perf_evlist__delete(kvm->evlist);
 
diff --git a/tools/perf/builtin-lock.c b/tools/perf/builtin-lock.c
index de16aaed516e..aa0011f5c0aa 100644
--- a/tools/perf/builtin-lock.c
+++ b/tools/perf/builtin-lock.c
@@ -866,10 +866,13 @@ static int __cmd_report(bool display_info)
 		.force = force,
 	};
 
+	eops.machines = machines__new();
+	if (eops.machines == NULL)
+		return -1;
 	session = perf_session__new(&file, false, &eops);
 	if (!session) {
 		pr_err("Initializing perf session failed\n");
-		return -1;
+		goto out_delete_machine;
 	}
 
 	symbol__init(&session->header.env);
@@ -899,6 +902,8 @@ static int __cmd_report(bool display_info)
 
 out_delete:
 	perf_session__delete(session);
+out_delete_machine:
+	machines__delete(eops.machines);
 	return err;
 }
 
diff --git a/tools/perf/builtin-mem.c b/tools/perf/builtin-mem.c
index da2ec06f0742..3399907a7d8d 100644
--- a/tools/perf/builtin-mem.c
+++ b/tools/perf/builtin-mem.c
@@ -126,12 +126,16 @@ static int report_raw_events(struct perf_mem *mem)
 	};
 	int err = -EINVAL;
 	int ret;
-	struct perf_session *session = perf_session__new(&file, false,
-							 &mem->tool);
+	struct perf_session *session;
 
-	if (session == NULL)
+	mem->tool.machines = machines__new();
+	if (mem->tool.machines == NULL)
 		return -1;
 
+	session = perf_session__new(&file, false, &mem->tool);
+	if (session == NULL)
+		goto out_delete_machine;
+
 	if (mem->cpu_list) {
 		ret = perf_session__cpu_bitmap(session, mem->cpu_list,
 					       mem->cpu_bitmap);
@@ -152,6 +156,8 @@ static int report_raw_events(struct perf_mem *mem)
 
 out_delete:
 	perf_session__delete(session);
+out_delete_machine:
+	machines__delete(mem->tool.machines);
 	return err;
 }
 
diff --git a/tools/perf/builtin-record.c b/tools/perf/builtin-record.c
index 25934b9d368f..7ab67139859b 100644
--- a/tools/perf/builtin-record.c
+++ b/tools/perf/builtin-record.c
@@ -667,10 +667,14 @@ static int __cmd_record(struct record *rec, int argc, const char **argv)
 	else
 		signal(SIGUSR2, SIG_IGN);
 
+	tool->machines = machines__new();
+	if (tool->machines == NULL)
+		return -1;
+
 	session = perf_session__new(file, false, tool);
 	if (session == NULL) {
 		pr_err("Perf session creation failed.\n");
-		return -1;
+		goto out_delete_machines;
 	}
 
 	fd = perf_data_file__fd(file);
@@ -929,6 +933,8 @@ static int __cmd_record(struct record *rec, int argc, const char **argv)
 
 out_delete_session:
 	perf_session__delete(session);
+out_delete_machines:
+	machines__delete(tool->machines);
 	return status;
 }
 
diff --git a/tools/perf/builtin-report.c b/tools/perf/builtin-report.c
index 5e53eee5a9a7..8baafb0a561e 100644
--- a/tools/perf/builtin-report.c
+++ b/tools/perf/builtin-report.c
@@ -766,9 +766,13 @@ int cmd_report(int argc, const char **argv, const char *prefix __maybe_unused)
 	file.force = report.force;
 
 repeat:
+	report.tool.machines = machines__new();
+	if (report.tool.machines == NULL)
+		return -1;
+
 	session = perf_session__new(&file, false, &report.tool);
 	if (session == NULL)
-		return -1;
+		goto error_delete;
 
 	if (report.queue_size) {
 		ordered_events__set_alloc_size(&session->ordered_events,
@@ -883,11 +887,14 @@ int cmd_report(int argc, const char **argv, const char *prefix __maybe_unused)
 	ret = __cmd_report(&report);
 	if (ret == K_SWITCH_INPUT_DATA) {
 		perf_session__delete(session);
+		machines__delete(report.tool.machines);
 		goto repeat;
 	} else
 		ret = 0;
 
 error:
 	perf_session__delete(session);
+error_delete:
+	machines__delete(report.tool.machines);
 	return ret;
 }
diff --git a/tools/perf/builtin-sched.c b/tools/perf/builtin-sched.c
index 33962612a5e9..f7cea0460656 100644
--- a/tools/perf/builtin-sched.c
+++ b/tools/perf/builtin-sched.c
@@ -1531,10 +1531,14 @@ static int perf_sched__read_events(struct perf_sched *sched)
 	};
 	int rc = -1;
 
+	sched->tool.machines = machines__new();
+	if (sched->tool.machines == NULL)
+		return -1;
+
 	session = perf_session__new(&file, false, &sched->tool);
 	if (session == NULL) {
 		pr_debug("No Memory for session\n");
-		return -1;
+		goto out_delete_machine;
 	}
 
 	symbol__init(&session->header.env);
@@ -1557,6 +1561,8 @@ static int perf_sched__read_events(struct perf_sched *sched)
 	rc = 0;
 out_delete:
 	perf_session__delete(session);
+out_delete_machine:
+	machines__delete(sched->tool.machines);
 	return rc;
 }
 
diff --git a/tools/perf/builtin-script.c b/tools/perf/builtin-script.c
index 24809787369f..80f7496e0742 100644
--- a/tools/perf/builtin-script.c
+++ b/tools/perf/builtin-script.c
@@ -1388,21 +1388,27 @@ int find_scripts(char **scripts_array, char **scripts_path_array)
 		.path = input_name,
 		.mode = PERF_DATA_MODE_READ,
 	};
+	struct perf_tool tool = {
+		.machines = NULL,
+	};
 	char *temp;
-	int i = 0;
+	int i = -1;
 
-	session = perf_session__new(&file, false, NULL);
-	if (!session)
+	tool.machines = machines__new();
+	if (tool.machines == NULL)
 		return -1;
 
+	session = perf_session__new(&file, false, &tool);
+	if (!session)
+		goto out;
+
 	snprintf(scripts_path, MAXPATHLEN, "%s/scripts", perf_exec_path());
 
 	scripts_dir = opendir(scripts_path);
-	if (!scripts_dir) {
-		perf_session__delete(session);
-		return -1;
-	}
+	if (!scripts_dir)
+		goto out_session;
 
+	i = 0;
 	for_each_lang(scripts_path, scripts_dir, lang_dirent, lang_next) {
 		snprintf(lang_path, MAXPATHLEN, "%s/%s", scripts_path,
 			 lang_dirent.d_name);
@@ -1440,7 +1446,11 @@ int find_scripts(char **scripts_array, char **scripts_path_array)
 	}
 
 	closedir(scripts_dir);
+
+out_session:
 	perf_session__delete(session);
+out:
+	machines__delete(tool.machines);
 	return i;
 }
 
@@ -1802,9 +1812,13 @@ int cmd_script(int argc, const char **argv, const char *prefix __maybe_unused)
 	if (!script_name)
 		setup_pager();
 
+	script.tool.machines = machines__new();
+	if (script.tool.machines == NULL)
+		return -1;
+
 	session = perf_session__new(&file, false, &script.tool);
 	if (session == NULL)
-		return -1;
+		goto out_delete_machine;
 
 	if (header || header_only) {
 		perf_session__fprintf_info(session, stdout, show_full_info);
@@ -1890,6 +1904,8 @@ int cmd_script(int argc, const char **argv, const char *prefix __maybe_unused)
 
 out_delete:
 	perf_session__delete(session);
+out_delete_machine:
+	machines__delete(script.tool.machines);
 
 	if (script_started)
 		cleanup_scripting();
diff --git a/tools/perf/builtin-timechart.c b/tools/perf/builtin-timechart.c
index 30e59620179d..52cac7796014 100644
--- a/tools/perf/builtin-timechart.c
+++ b/tools/perf/builtin-timechart.c
@@ -1603,13 +1603,17 @@ static int __cmd_timechart(struct timechart *tchart, const char *output_name)
 		.force = tchart->force,
 	};
 
-	struct perf_session *session = perf_session__new(&file, false,
-							 &tchart->tool);
+	struct perf_session *session;
 	int ret = -EINVAL;
 
-	if (session == NULL)
+	tchart->tool.machines = machines__new();
+	if (tchart->tool.machines == NULL)
 		return -1;
 
+	session = perf_session__new(&file, false, &tchart->tool);
+	if (session == NULL)
+		goto out_delete_machine;
+
 	symbol__init(&session->header.env);
 
 	(void)perf_header__process_sections(&session->header,
@@ -1640,6 +1644,8 @@ static int __cmd_timechart(struct timechart *tchart, const char *output_name)
 		(tchart->last_time - tchart->first_time) / 1000000000.0, output_name);
 out_delete:
 	perf_session__delete(session);
+out_delete_machine:
+	machines__delete(tchart->tool.machines);
 	return ret;
 }
 
diff --git a/tools/perf/builtin-top.c b/tools/perf/builtin-top.c
index ea6e7bd04f9a..9d99bebae955 100644
--- a/tools/perf/builtin-top.c
+++ b/tools/perf/builtin-top.c
@@ -933,12 +933,19 @@ static int __cmd_top(struct perf_top *top)
 {
 	struct record_opts *opts = &top->record_opts;
 	pthread_t thread;
-	int ret;
+	int ret = -1;
+	struct perf_tool tool = {
+		.machines = NULL,
+	};
 
-	top->session = perf_session__new(NULL, false, NULL);
-	if (top->session == NULL)
+	tool.machines = machines__new();
+	if (tool.machines == NULL)
 		return -1;
 
+	top->session = perf_session__new(NULL, false, &tool);
+	if (top->session == NULL)
+		goto out_delete_machine;
+
 	machines__set_symbol_filter(&top->session->machines, symbol_filter);
 
 	if (!objdump_path) {
@@ -1008,6 +1015,8 @@ static int __cmd_top(struct perf_top *top)
 out_delete:
 	perf_session__delete(top->session);
 	top->session = NULL;
+out_delete_machine:
+	machines__delete(tool.machines);
 
 	return ret;
 }
diff --git a/tools/perf/builtin-trace.c b/tools/perf/builtin-trace.c
index e30d0ed1e60f..0e2c60af7cb6 100644
--- a/tools/perf/builtin-trace.c
+++ b/tools/perf/builtin-trace.c
@@ -2413,9 +2413,13 @@ static int trace__replay(struct trace *trace)
 	/* add tid to output */
 	trace->multiple_threads = true;
 
+	trace->tool.machines = machines__new();
+	if (trace->tool.machines == NULL)
+		return -1;
+
 	session = perf_session__new(&file, false, &trace->tool);
 	if (session == NULL)
-		return -1;
+		goto out_delete;
 
 	if (symbol__init(&session->header.env) < 0)
 		goto out;
@@ -2475,6 +2479,8 @@ static int trace__replay(struct trace *trace)
 
 out:
 	perf_session__delete(session);
+out_delete:
+	machines__delete(trace->tool.machines);
 
 	return err;
 }
diff --git a/tools/perf/util/data-convert-bt.c b/tools/perf/util/data-convert-bt.c
index 5bfc1198ab46..270e0298c9f3 100644
--- a/tools/perf/util/data-convert-bt.c
+++ b/tools/perf/util/data-convert-bt.c
@@ -1133,10 +1133,14 @@ int bt_convert__perf2ctf(const char *input, const char *path, bool force)
 	if (ctf_writer__init(cw, path))
 		return -1;
 
+	c.tool.machines = machines__new();
+	if (c.tool.machines == NULL)
+		goto free_writer;
+
 	/* perf.data session */
 	session = perf_session__new(&file, 0, &c.tool);
 	if (!session)
-		goto free_writer;
+		goto free_machines;
 
 	if (c.queue_size) {
 		ordered_events__set_alloc_size(&session->ordered_events,
@@ -1176,6 +1180,8 @@ int bt_convert__perf2ctf(const char *input, const char *path, bool force)
 
 free_session:
 	perf_session__delete(session);
+free_machines:
+	machines__delete(c.tool.machines);
 free_writer:
 	ctf_writer__cleanup(cw);
 	pr_err("Error during conversion setup.\n");
diff --git a/tools/perf/util/machine.c b/tools/perf/util/machine.c
index 308656be8ddb..d770b705cb16 100644
--- a/tools/perf/util/machine.c
+++ b/tools/perf/util/machine.c
@@ -163,6 +163,21 @@ void machines__exit(struct machines *machines)
 	/* XXX exit guest */
 }
 
+struct machines *machines__new(void)
+{
+	struct machines *machines = zalloc(sizeof(*machines));
+
+	if (machines)
+		machines__init(machines);
+	return machines;
+}
+
+void machines__delete(struct machines *machines)
+{
+	machines__exit(machines);
+	free(machines);
+}
+
 struct machine *machines__add(struct machines *machines, pid_t pid,
 			      const char *root_dir)
 {
diff --git a/tools/perf/util/machine.h b/tools/perf/util/machine.h
index f0f6bd420237..81494371f459 100644
--- a/tools/perf/util/machine.h
+++ b/tools/perf/util/machine.h
@@ -102,6 +102,9 @@ struct machines {
 void machines__init(struct machines *machines);
 void machines__exit(struct machines *machines);
 
+struct machines *machines__new(void);
+void machines__delete(struct machines *machines);
+
 void machines__process_guests(struct machines *machines,
 			      machine__process_t process, void *data);
 
diff --git a/tools/perf/util/tool.h b/tools/perf/util/tool.h
index 7f282ad1d2bd..2d399eba236b 100644
--- a/tools/perf/util/tool.h
+++ b/tools/perf/util/tool.h
@@ -12,6 +12,7 @@ struct perf_evsel;
 struct perf_sample;
 struct perf_tool;
 struct machine;
+struct machines;
 struct ordered_events;
 
 typedef int (*event_sample)(struct perf_tool *tool, union perf_event *event,
@@ -35,6 +36,7 @@ typedef s64 (*event_op3)(struct perf_tool *tool, union perf_event *event,
 			 struct perf_session *session);
 
 struct perf_tool {
+	struct machines *machines;
 	event_sample	sample,
 			read;
 	event_op	mmap,
-- 
2.4.1


^ permalink raw reply related	[flat|nested] 221+ messages in thread

* [PATCH 2/2] perf session: Separate struct machines from session
  2015-05-28  2:39           ` [RFC 0/2] " Namhyung Kim
  2015-05-28  2:39             ` [PATCH 1/2] perf tools: Introduce machines__new/delete() Namhyung Kim
@ 2015-05-28  2:39             ` Namhyung Kim
  1 sibling, 0 replies; 221+ messages in thread
From: Namhyung Kim @ 2015-05-28  2:39 UTC (permalink / raw)
  To: Arnaldo Carvalho de Melo
  Cc: Ingo Molnar, Peter Zijlstra, Jiri Olsa, LKML, David Ahern

With multi-thread report, separate sessions can be passed to each
thread, in this case we should keep a single machine state for all
struct sessions.  Separate machines and have a pointer in sessions.

Signed-off-by: Namhyung Kim <namhyung@kernel.org>
---
 tools/perf/builtin-annotate.c |  2 +-
 tools/perf/builtin-diff.c     |  2 +-
 tools/perf/builtin-kmem.c     | 10 +++++-----
 tools/perf/builtin-kvm.c      |  2 +-
 tools/perf/builtin-record.c   |  6 +++---
 tools/perf/builtin-report.c   |  4 ++--
 tools/perf/builtin-top.c      | 10 +++++-----
 tools/perf/builtin-trace.c    |  2 +-
 tools/perf/util/build-id.c    | 16 ++++++++--------
 tools/perf/util/intel-bts.c   |  2 +-
 tools/perf/util/intel-pt.c    |  2 +-
 tools/perf/util/session.c     | 30 +++++++++++++++---------------
 tools/perf/util/session.h     |  6 +++---
 13 files changed, 47 insertions(+), 47 deletions(-)

diff --git a/tools/perf/builtin-annotate.c b/tools/perf/builtin-annotate.c
index e14a8a4d2f8d..91e365cf3154 100644
--- a/tools/perf/builtin-annotate.c
+++ b/tools/perf/builtin-annotate.c
@@ -196,7 +196,7 @@ static int __cmd_annotate(struct perf_annotate *ann)
 	struct perf_evsel *pos;
 	u64 total_nr_samples;
 
-	machines__set_symbol_filter(&session->machines, symbol__annotate_init);
+	machines__set_symbol_filter(session->machines, symbol__annotate_init);
 
 	if (ann->cpu_list) {
 		ret = perf_session__cpu_bitmap(session, ann->cpu_list,
diff --git a/tools/perf/builtin-diff.c b/tools/perf/builtin-diff.c
index a4431f096697..ff818cef0d94 100644
--- a/tools/perf/builtin-diff.c
+++ b/tools/perf/builtin-diff.c
@@ -771,7 +771,7 @@ static int __cmd_diff(void)
  out_delete:
 	data__for_each_file(i, d) {
 		if (d->session) {
-			struct machines *machines = &d->session->machines;
+			struct machines *machines = d->session->machines;
 
 			perf_session__delete(d->session);
 			if (machines)
diff --git a/tools/perf/builtin-kmem.c b/tools/perf/builtin-kmem.c
index 979a032fe85f..2ffed554ef7d 100644
--- a/tools/perf/builtin-kmem.c
+++ b/tools/perf/builtin-kmem.c
@@ -316,7 +316,7 @@ static int build_alloc_func_list(void)
 	struct symbol *sym;
 	struct rb_node *node;
 	struct alloc_func *func;
-	struct machine *machine = &kmem_session->machines.host;
+	struct machine *machine = &kmem_session->machines->host;
 	regex_t alloc_func_regex;
 	const char pattern[] = "^_?_?(alloc|get_free|get_zeroed)_pages?";
 
@@ -366,7 +366,7 @@ static int build_alloc_func_list(void)
 static u64 find_callsite(struct perf_evsel *evsel, struct perf_sample *sample)
 {
 	struct addr_location al;
-	struct machine *machine = &kmem_session->machines.host;
+	struct machine *machine = &kmem_session->machines->host;
 	struct callchain_cursor_node *node;
 
 	if (alloc_func_list == NULL) {
@@ -949,7 +949,7 @@ static void __print_slab_result(struct rb_root *root,
 				int n_lines, int is_caller)
 {
 	struct rb_node *next;
-	struct machine *machine = &session->machines.host;
+	struct machine *machine = &session->machines->host;
 
 	printf("%.105s\n", graph_dotted_line);
 	printf(" %-34s |",  is_caller ? "Callsite": "Alloc Ptr");
@@ -1010,7 +1010,7 @@ static const char * const migrate_type_str[] = {
 static void __print_page_alloc_result(struct perf_session *session, int n_lines)
 {
 	struct rb_node *next = rb_first(&page_alloc_sorted);
-	struct machine *machine = &session->machines.host;
+	struct machine *machine = &session->machines->host;
 	const char *format;
 	int gfp_len = max(strlen("GFP flags"), max_gfp_len);
 
@@ -1060,7 +1060,7 @@ static void __print_page_alloc_result(struct perf_session *session, int n_lines)
 static void __print_page_caller_result(struct perf_session *session, int n_lines)
 {
 	struct rb_node *next = rb_first(&page_caller_sorted);
-	struct machine *machine = &session->machines.host;
+	struct machine *machine = &session->machines->host;
 	int gfp_len = max(strlen("GFP flags"), max_gfp_len);
 
 	printf("\n%.105s\n", graph_dotted_line);
diff --git a/tools/perf/builtin-kvm.c b/tools/perf/builtin-kvm.c
index ca8836392af2..531df6036709 100644
--- a/tools/perf/builtin-kvm.c
+++ b/tools/perf/builtin-kvm.c
@@ -1402,7 +1402,7 @@ static int kvm_events_live(struct perf_kvm_stat *kvm,
 	kvm->session->evlist = kvm->evlist;
 	perf_session__set_id_hdr_size(kvm->session);
 	ordered_events__set_copy_on_queue(&kvm->session->ordered_events, true);
-	machine__synthesize_threads(&kvm->session->machines.host, &kvm->opts.target,
+	machine__synthesize_threads(&kvm->session->machines->host, &kvm->opts.target,
 				    kvm->evlist->threads, false);
 	err = kvm_live_open_events(kvm);
 	if (err)
diff --git a/tools/perf/builtin-record.c b/tools/perf/builtin-record.c
index 7ab67139859b..2063d1b11948 100644
--- a/tools/perf/builtin-record.c
+++ b/tools/perf/builtin-record.c
@@ -607,7 +607,7 @@ static int synthesize_workload_comm_event(struct perf_evlist *evlist, void *arg)
 {
 	union perf_event *event;
 	struct record *rec = arg;
-	struct machine *machine = &rec->session->machines.host;
+	struct machine *machine = &rec->session->machines->host;
 	int pid = evlist->workload.pid;
 	const char *comm_str = program_invocation_short_name;
 	size_t comm_size, total_size;
@@ -724,7 +724,7 @@ static int __cmd_record(struct record *rec, int argc, const char **argv)
 		goto out_child;
 	}
 
-	machine = &session->machines.host;
+	machine = &session->machines->host;
 
 	if (file->is_pipe) {
 		err = perf_event__synthesize_attrs(tool, session,
@@ -775,7 +775,7 @@ static int __cmd_record(struct record *rec, int argc, const char **argv)
 		       "Check /proc/modules permission or run as root.\n");
 
 	if (perf_guest) {
-		machines__process_guests(&session->machines,
+		machines__process_guests(session->machines,
 					 perf_event__synthesize_guest_os, tool);
 	}
 
diff --git a/tools/perf/builtin-report.c b/tools/perf/builtin-report.c
index 8baafb0a561e..ad855731bdce 100644
--- a/tools/perf/builtin-report.c
+++ b/tools/perf/builtin-report.c
@@ -353,7 +353,7 @@ static int perf_evlist__tty_browse_hists(struct perf_evlist *evlist,
 
 static void report__warn_kptr_restrict(const struct report *rep)
 {
-	struct map *kernel_map = rep->session->machines.host.vmlinux_maps[MAP__FUNCTION];
+	struct map *kernel_map = rep->session->machines->host.vmlinux_maps[MAP__FUNCTION];
 	struct kmap *kernel_kmap = kernel_map ? map__kmap(kernel_map) : NULL;
 
 	if (kernel_map == NULL ||
@@ -849,7 +849,7 @@ int cmd_report(int argc, const char **argv, const char *prefix __maybe_unused)
 	 */
 	if (ui__has_annotation()) {
 		symbol_conf.priv_size = sizeof(struct annotation);
-		machines__set_symbol_filter(&session->machines,
+		machines__set_symbol_filter(session->machines,
 					    symbol__annotate_init);
 		/*
  		 * For searching by name on the "Browse map details".
diff --git a/tools/perf/builtin-top.c b/tools/perf/builtin-top.c
index 9d99bebae955..2129ab6c3936 100644
--- a/tools/perf/builtin-top.c
+++ b/tools/perf/builtin-top.c
@@ -828,13 +828,13 @@ static void perf_top__mmap_read_idx(struct perf_top *top, int idx)
 			++top->us_samples;
 			if (top->hide_user_symbols)
 				goto next_event;
-			machine = &session->machines.host;
+			machine = &session->machines->host;
 			break;
 		case PERF_RECORD_MISC_KERNEL:
 			++top->kernel_samples;
 			if (top->hide_kernel_symbols)
 				goto next_event;
-			machine = &session->machines.host;
+			machine = &session->machines->host;
 			break;
 		case PERF_RECORD_MISC_GUEST_KERNEL:
 			++top->guest_kernel_samples;
@@ -942,12 +942,12 @@ static int __cmd_top(struct perf_top *top)
 	if (tool.machines == NULL)
 		return -1;
 
+	machines__set_symbol_filter(tool.machines, symbol_filter);
+
 	top->session = perf_session__new(NULL, false, &tool);
 	if (top->session == NULL)
 		goto out_delete_machine;
 
-	machines__set_symbol_filter(&top->session->machines, symbol_filter);
-
 	if (!objdump_path) {
 		ret = perf_session_env__lookup_objdump(&top->session->header.env);
 		if (ret)
@@ -958,7 +958,7 @@ static int __cmd_top(struct perf_top *top)
 	if (ret)
 		goto out_delete;
 
-	machine__synthesize_threads(&top->session->machines.host, &opts->target,
+	machine__synthesize_threads(&top->session->machines->host, &opts->target,
 				    top->evlist->threads, false);
 	ret = perf_top__start_counters(top);
 	if (ret)
diff --git a/tools/perf/builtin-trace.c b/tools/perf/builtin-trace.c
index 0e2c60af7cb6..0eff071be38e 100644
--- a/tools/perf/builtin-trace.c
+++ b/tools/perf/builtin-trace.c
@@ -2424,7 +2424,7 @@ static int trace__replay(struct trace *trace)
 	if (symbol__init(&session->header.env) < 0)
 		goto out;
 
-	trace->host = &session->machines.host;
+	trace->host = &session->machines->host;
 
 	err = perf_session__set_tracepoints_handlers(session, handlers);
 	if (err)
diff --git a/tools/perf/util/build-id.c b/tools/perf/util/build-id.c
index ad8cfcbaa25d..8720f71a96dd 100644
--- a/tools/perf/util/build-id.c
+++ b/tools/perf/util/build-id.c
@@ -221,12 +221,12 @@ static int machine__write_buildid_table(struct machine *machine, int fd)
 int perf_session__write_buildid_table(struct perf_session *session, int fd)
 {
 	struct rb_node *nd;
-	int err = machine__write_buildid_table(&session->machines.host, fd);
+	int err = machine__write_buildid_table(&session->machines->host, fd);
 
 	if (err)
 		return err;
 
-	for (nd = rb_first(&session->machines.guests); nd; nd = rb_next(nd)) {
+	for (nd = rb_first(&session->machines->guests); nd; nd = rb_next(nd)) {
 		struct machine *pos = rb_entry(nd, struct machine, rb_node);
 		err = machine__write_buildid_table(pos, fd);
 		if (err)
@@ -261,11 +261,11 @@ int dsos__hit_all(struct perf_session *session)
 	struct rb_node *nd;
 	int err;
 
-	err = machine__hit_all_dsos(&session->machines.host);
+	err = machine__hit_all_dsos(&session->machines->host);
 	if (err)
 		return err;
 
-	for (nd = rb_first(&session->machines.guests); nd; nd = rb_next(nd)) {
+	for (nd = rb_first(&session->machines->guests); nd; nd = rb_next(nd)) {
 		struct machine *pos = rb_entry(nd, struct machine, rb_node);
 
 		err = machine__hit_all_dsos(pos);
@@ -509,9 +509,9 @@ int perf_session__cache_build_ids(struct perf_session *session)
 	if (mkdir(buildid_dir, 0755) != 0 && errno != EEXIST)
 		return -1;
 
-	ret = machine__cache_build_ids(&session->machines.host);
+	ret = machine__cache_build_ids(&session->machines->host);
 
-	for (nd = rb_first(&session->machines.guests); nd; nd = rb_next(nd)) {
+	for (nd = rb_first(&session->machines->guests); nd; nd = rb_next(nd)) {
 		struct machine *pos = rb_entry(nd, struct machine, rb_node);
 		ret |= machine__cache_build_ids(pos);
 	}
@@ -530,9 +530,9 @@ static bool machine__read_build_ids(struct machine *machine, bool with_hits)
 bool perf_session__read_build_ids(struct perf_session *session, bool with_hits)
 {
 	struct rb_node *nd;
-	bool ret = machine__read_build_ids(&session->machines.host, with_hits);
+	bool ret = machine__read_build_ids(&session->machines->host, with_hits);
 
-	for (nd = rb_first(&session->machines.guests); nd; nd = rb_next(nd)) {
+	for (nd = rb_first(&session->machines->guests); nd; nd = rb_next(nd)) {
 		struct machine *pos = rb_entry(nd, struct machine, rb_node);
 		ret |= machine__read_build_ids(pos, with_hits);
 	}
diff --git a/tools/perf/util/intel-bts.c b/tools/perf/util/intel-bts.c
index cd7bde33b635..61985f94146d 100644
--- a/tools/perf/util/intel-bts.c
+++ b/tools/perf/util/intel-bts.c
@@ -862,7 +862,7 @@ int intel_bts_process_auxtrace_info(union perf_event *event,
 		goto err_free;
 
 	bts->session = session;
-	bts->machine = &session->machines.host; /* No kvm support */
+	bts->machine = &session->machines->host; /* No kvm support */
 	bts->auxtrace_type = auxtrace_info->type;
 	bts->pmu_type = auxtrace_info->priv[INTEL_BTS_PMU_TYPE];
 	bts->tc.time_shift = auxtrace_info->priv[INTEL_BTS_TIME_SHIFT];
diff --git a/tools/perf/util/intel-pt.c b/tools/perf/util/intel-pt.c
index 5a59fd8e79ae..fc0ba91bf6d1 100644
--- a/tools/perf/util/intel-pt.c
+++ b/tools/perf/util/intel-pt.c
@@ -1788,7 +1788,7 @@ int intel_pt_process_auxtrace_info(union perf_event *event,
 	intel_pt_log_set_name(INTEL_PT_PMU_NAME);
 
 	pt->session = session;
-	pt->machine = &session->machines.host; /* No kvm support */
+	pt->machine = &session->machines->host; /* No kvm support */
 	pt->auxtrace_type = auxtrace_info->type;
 	pt->pmu_type = auxtrace_info->priv[INTEL_PT_PMU_TYPE];
 	pt->tc.time_shift = auxtrace_info->priv[INTEL_PT_TIME_SHIFT];
diff --git a/tools/perf/util/session.c b/tools/perf/util/session.c
index 75e32fb31c1a..ae9f4e1a6079 100644
--- a/tools/perf/util/session.c
+++ b/tools/perf/util/session.c
@@ -58,21 +58,21 @@ void perf_session__set_id_hdr_size(struct perf_session *session)
 {
 	u16 id_hdr_size = perf_evlist__id_hdr_size(session->evlist);
 
-	machines__set_id_hdr_size(&session->machines, id_hdr_size);
+	machines__set_id_hdr_size(session->machines, id_hdr_size);
 }
 
 int perf_session__create_kernel_maps(struct perf_session *session)
 {
-	int ret = machine__create_kernel_maps(&session->machines.host);
+	int ret = machine__create_kernel_maps(&session->machines->host);
 
 	if (ret >= 0)
-		ret = machines__create_guest_kernel_maps(&session->machines);
+		ret = machines__create_guest_kernel_maps(session->machines);
 	return ret;
 }
 
 static void perf_session__destroy_kernel_maps(struct perf_session *session)
 {
-	machines__destroy_kernel_maps(&session->machines);
+	machines__destroy_kernel_maps(session->machines);
 }
 
 static bool perf_session__has_comm_exec(struct perf_session *session)
@@ -91,7 +91,7 @@ static void perf_session__set_comm_exec(struct perf_session *session)
 {
 	bool comm_exec = perf_session__has_comm_exec(session);
 
-	machines__set_comm_exec(&session->machines, comm_exec);
+	machines__set_comm_exec(session->machines, comm_exec);
 }
 
 static int ordered_events__deliver_event(struct ordered_events *oe,
@@ -123,7 +123,8 @@ struct perf_session *perf_session__new(struct perf_data_file *file,
 	session->repipe = repipe;
 	session->tool   = tool;
 	INIT_LIST_HEAD(&session->auxtrace_index);
-	machines__init(&session->machines);
+
+	session->machines = tool->machines;
 	ordered_events__init(&session->ordered_events, ordered_events__deliver_event);
 
 	if (file) {
@@ -168,7 +169,7 @@ struct perf_session *perf_session__new(struct perf_data_file *file,
 
 static void perf_session__delete_threads(struct perf_session *session)
 {
-	machine__delete_threads(&session->machines.host);
+	machine__delete_threads(&session->machines->host);
 }
 
 static void perf_session_env__delete(struct perf_session_env *env)
@@ -194,7 +195,6 @@ void perf_session__delete(struct perf_session *session)
 	perf_session__destroy_kernel_maps(session);
 	perf_session__delete_threads(session);
 	perf_session_env__delete(&session->header.env);
-	machines__exit(&session->machines);
 	if (session->file)
 		perf_data_file__close(session->file);
 	free(session->header.index);
@@ -1088,7 +1088,7 @@ static int perf_session__deliver_event(struct perf_session *session,
 	if (ret > 0)
 		return 0;
 
-	return machines__deliver_event(&session->machines, stats,
+	return machines__deliver_event(session->machines, stats,
 				       session->evlist, event, sample,
 				       tool, file_offset);
 }
@@ -1155,7 +1155,7 @@ int perf_session__deliver_synth_event(struct perf_session *session,
 	if (event->header.type >= PERF_RECORD_USER_TYPE_START)
 		return perf_session__process_user_event(session, event, 0);
 
-	return machines__deliver_event(&session->machines, &evlist->stats,
+	return machines__deliver_event(session->machines, &evlist->stats,
 				       evlist, event, sample, tool, 0);
 }
 
@@ -1270,14 +1270,14 @@ void perf_event_header__bswap(struct perf_event_header *hdr)
 
 struct thread *perf_session__findnew(struct perf_session *session, pid_t pid)
 {
-	return machine__findnew_thread(&session->machines.host, -1, pid);
+	return machine__findnew_thread(&session->machines->host, -1, pid);
 }
 
 static struct thread *perf_session__register_idle_thread(struct perf_session *session)
 {
 	struct thread *thread;
 
-	thread = machine__findnew_thread(&session->machines.host, 0, 0);
+	thread = machine__findnew_thread(&session->machines->host, 0, 0);
 	if (thread == NULL || thread__set_comm(thread, "swapper", 0)) {
 		pr_err("problem inserting idle task.\n");
 		thread = NULL;
@@ -1692,13 +1692,13 @@ int maps__set_kallsyms_ref_reloc_sym(struct map **maps,
 
 size_t perf_session__fprintf_dsos(struct perf_session *session, FILE *fp)
 {
-	return machines__fprintf_dsos(&session->machines, fp);
+	return machines__fprintf_dsos(session->machines, fp);
 }
 
 size_t perf_session__fprintf_dsos_buildid(struct perf_session *session, FILE *fp,
 					  bool (skip)(struct dso *dso, int parm), int parm)
 {
-	return machines__fprintf_dsos_buildid(&session->machines, fp, skip, parm);
+	return machines__fprintf_dsos_buildid(session->machines, fp, skip, parm);
 }
 
 size_t perf_session__fprintf_nr_events(struct perf_session *session, FILE *fp)
@@ -1721,7 +1721,7 @@ size_t perf_session__fprintf(struct perf_session *session, FILE *fp)
 	 * FIXME: Here we have to actually print all the machines in this
 	 * session, not just the host...
 	 */
-	return machine__fprintf(&session->machines.host, fp);
+	return machine__fprintf(&session->machines->host, fp);
 }
 
 struct perf_evsel *perf_session__find_first_evtype(struct perf_session *session,
diff --git a/tools/perf/util/session.h b/tools/perf/util/session.h
index b44afc75d1cc..ac1796c7c799 100644
--- a/tools/perf/util/session.h
+++ b/tools/perf/util/session.h
@@ -20,7 +20,7 @@ struct itrace_synth_opts;
 
 struct perf_session {
 	struct perf_header	header;
-	struct machines		machines;
+	struct machines		*machines;
 	struct perf_evlist	*evlist;
 	struct auxtrace		*auxtrace;
 	struct itrace_synth_opts *itrace_synth_opts;
@@ -79,13 +79,13 @@ void perf_session__set_id_hdr_size(struct perf_session *session);
 static inline
 struct machine *perf_session__find_machine(struct perf_session *session, pid_t pid)
 {
-	return machines__find(&session->machines, pid);
+	return machines__find(session->machines, pid);
 }
 
 static inline
 struct machine *perf_session__findnew_machine(struct perf_session *session, pid_t pid)
 {
-	return machines__findnew(&session->machines, pid);
+	return machines__findnew(session->machines, pid);
 }
 
 struct thread *perf_session__findnew(struct perf_session *session, pid_t pid);
-- 
2.4.1


^ permalink raw reply related	[flat|nested] 221+ messages in thread

end of thread, other threads:[~2015-05-28  2:50 UTC | newest]

Thread overview: 221+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2015-01-29  8:06 [RFC/PATCHSET 00/42] perf tools: Speed-up perf report by using multi thread (v2) Namhyung Kim
2015-01-29  8:06 ` [PATCH 01/42] perf tools: Support to read compressed module from build-id cache Namhyung Kim
2015-01-30 14:32   ` Jiri Olsa
2015-02-02 15:03     ` Namhyung Kim
2015-01-30 18:33   ` [tip:perf/core] perf symbols: " tip-bot for Namhyung Kim
2015-01-29  8:06 ` [PATCH 02/42] perf tools: Do not use __perf_session__process_events() directly Namhyung Kim
2015-01-30 18:32   ` [tip:perf/core] " tip-bot for Namhyung Kim
2015-01-29  8:06 ` [PATCH 03/42] perf record: Show precise number of samples Namhyung Kim
2015-01-30 18:32   ` [tip:perf/core] " tip-bot for Namhyung Kim
2015-01-29  8:06 ` [PATCH 04/42] perf header: Set header version correctly Namhyung Kim
2015-01-30 18:33   ` [tip:perf/core] " tip-bot for Namhyung Kim
2015-01-29  8:06 ` [PATCH 05/42] perf tools: Set attr.task bit for a tracking event Namhyung Kim
2015-01-30 18:33   ` [tip:perf/core] perf evsel: " tip-bot for Namhyung Kim
2015-01-29  8:06 ` [PATCH 06/42] perf tools: Use a software dummy event to track task/mmap events Namhyung Kim
2015-01-29  8:06 ` [PATCH 07/42] perf tools: Use perf_data_file__fd() consistently Namhyung Kim
2015-01-30 18:33   ` [tip:perf/core] " tip-bot for Namhyung Kim
2015-01-29  8:06 ` [PATCH 08/42] perf tools: Add rm_rf() utility function Namhyung Kim
2015-01-30 15:02   ` Jiri Olsa
2015-05-20 12:24     ` [tip:perf/core] " tip-bot for Namhyung Kim
2015-01-29  8:06 ` [PATCH 09/42] perf tools: Introduce copyfile_offset() function Namhyung Kim
2015-01-29  8:06 ` [PATCH 10/42] perf tools: Create separate mmap for dummy tracking event Namhyung Kim
2015-01-29  8:06 ` [PATCH 11/42] perf tools: Introduce perf_evlist__mmap_track() Namhyung Kim
2015-01-29  8:06 ` [PATCH 12/42] perf tools: Add HEADER_DATA_INDEX feature Namhyung Kim
2015-01-29  8:06 ` [PATCH 13/42] perf tools: Handle indexed data file properly Namhyung Kim
2015-01-29  8:06 ` [PATCH 14/42] perf record: Add --index option for building index table Namhyung Kim
2015-02-01 18:06   ` Jiri Olsa
2015-02-02  8:34     ` Adrian Hunter
2015-02-02  9:15       ` Jiri Olsa
2015-02-02  9:52         ` Adrian Hunter
2015-02-02 10:05           ` Jiri Olsa
2015-02-02 12:07             ` Adrian Hunter
2015-02-02 12:13               ` Jiri Olsa
2015-02-02 14:56                 ` Namhyung Kim
2015-02-02 17:30                   ` Jiri Olsa
2015-02-03  8:42                     ` Adrian Hunter
2015-01-29  8:06 ` [PATCH 15/42] perf report: Skip dummy tracking event Namhyung Kim
2015-01-29  8:06 ` [PATCH 16/42] perf tools: Pass session arg to perf_event__preprocess_sample() Namhyung Kim
2015-01-29  8:06 ` [PATCH 17/42] perf script: Pass session arg to ->process_event callback Namhyung Kim
2015-01-29  8:06 ` [PATCH 18/42] perf tools: Introduce thread__comm_time() helpers Namhyung Kim
2015-01-29  8:07 ` [PATCH 19/42] perf tools: Add a test case for thread comm handling Namhyung Kim
2015-01-29  8:07 ` [PATCH 20/42] perf tools: Use thread__comm_time() when adding hist entries Namhyung Kim
2015-01-29  8:07 ` [PATCH 21/42] perf tools: Convert dead thread list into rbtree Namhyung Kim
2015-01-29  8:07 ` [PATCH 22/42] perf tools: Introduce machine__find*_thread_time() Namhyung Kim
2015-01-29  8:07 ` [PATCH 23/42] perf tools: Add a test case for timed thread handling Namhyung Kim
2015-01-29  8:07 ` [PATCH 24/42] perf tools: Maintain map groups list in a leader thread Namhyung Kim
2015-01-29  8:07 ` [PATCH 25/42] perf tools: Introduce thread__find_addr_location_time() and friends Namhyung Kim
2015-01-29  8:07 ` [PATCH 26/42] perf tools: Add a test case for timed map groups handling Namhyung Kim
2015-01-29  8:07 ` [PATCH 27/42] perf tools: Protect dso symbol loading using a mutex Namhyung Kim
2015-01-29 12:34   ` Arnaldo Carvalho de Melo
2015-01-29 12:48     ` Namhyung Kim
2015-01-29  8:07 ` [PATCH 28/42] perf tools: Protect dso cache tree using dso->lock Namhyung Kim
2015-01-29  8:07 ` [PATCH 29/42] perf tools: Protect dso cache fd with a mutex Namhyung Kim
2015-01-29 12:31   ` Arnaldo Carvalho de Melo
2015-01-29 13:19     ` Namhyung Kim
2015-01-29 16:23       ` Arnaldo Carvalho de Melo
2015-01-30  0:51         ` Namhyung Kim
2015-01-29  8:07 ` [PATCH 30/42] perf session: Pass struct events stats to event processing functions Namhyung Kim
2015-01-29  8:07 ` [PATCH 31/42] perf hists: Pass hists struct to hist_entry_iter functions Namhyung Kim
2015-01-29  8:07 ` [PATCH 32/42] perf tools: Move BUILD_ID_SIZE definition to perf.h Namhyung Kim
2015-01-29  8:07 ` [PATCH 33/42] perf report: Parallelize perf report using multi-thread Namhyung Kim
2015-01-29  8:07 ` [PATCH 34/42] perf tools: Add missing_threads rb tree Namhyung Kim
2015-01-29  8:07 ` [PATCH 35/42] perf record: Synthesize COMM event for a command line workload Namhyung Kim
2015-01-29  8:07 ` [PATCH 36/42] perf tools: Fix progress ui to support multi thread Namhyung Kim
2015-01-29  8:07 ` [PATCH 37/42] perf report: Add --multi-thread option and config item Namhyung Kim
2015-01-29  8:07 ` [PATCH 38/42] perf session: Handle index files generally Namhyung Kim
2015-01-29  8:07 ` [PATCH 39/42] perf tools: Convert lseek + read to pread Namhyung Kim
2015-01-30 18:34   ` [tip:perf/core] perf symbols: " tip-bot for Namhyung Kim
2015-01-29  8:07 ` [PATCH 40/42] perf callchain: Save eh/debug frame offset for dwarf unwind Namhyung Kim
2015-01-29 12:38   ` Arnaldo Carvalho de Melo
2015-01-29 13:23     ` Namhyung Kim
2015-01-29 19:22   ` Arnaldo Carvalho de Melo
2015-01-30 18:32   ` [tip:perf/core] perf callchain: Cache eh/ debug " tip-bot for Namhyung Kim
2015-01-29  8:07 ` [PATCH 41/42] perf tools: Add new perf data command Namhyung Kim
2015-01-29  8:07 ` [PATCH 42/42] perf data: Implement 'index' subcommand Namhyung Kim
2015-01-29 19:56 ` [RFC/PATCHSET 00/42] perf tools: Speed-up perf report by using multi thread (v2) Arnaldo Carvalho de Melo
2015-03-03  3:07 [RFC/PATCHSET 00/38] perf tools: Speed-up perf report by using multi thread (v3) Namhyung Kim
2015-03-03  3:07 ` [PATCH 01/38] perf tools: Use a software dummy event to track task/mmap events Namhyung Kim
2015-03-03  3:07 ` [PATCH 02/38] perf tools: Add rm_rf() utility function Namhyung Kim
2015-03-03  3:07 ` [PATCH 03/38] perf tools: Introduce copyfile_offset() function Namhyung Kim
2015-03-04 14:58   ` Jiri Olsa
2015-03-03  3:07 ` [PATCH 04/38] perf tools: Create separate mmap for dummy tracking event Namhyung Kim
2015-03-03  3:07 ` [PATCH 05/38] perf tools: Introduce perf_evlist__mmap_track() Namhyung Kim
2015-03-03  3:07 ` [PATCH 06/38] perf tools: Add HEADER_DATA_INDEX feature Namhyung Kim
2015-03-03  3:07 ` [PATCH 07/38] perf tools: Handle indexed data file properly Namhyung Kim
2015-03-04 16:19   ` Jiri Olsa
2015-03-06  4:17     ` Namhyung Kim
2015-03-03  3:07 ` [PATCH 08/38] perf record: Add --index option for building index table Namhyung Kim
2015-03-05  7:56   ` Jiri Olsa
2015-03-06  4:19     ` Namhyung Kim
2015-03-03  3:07 ` [PATCH 09/38] perf report: Skip dummy tracking event Namhyung Kim
2015-03-03  3:07 ` [PATCH 10/38] perf tools: Pass session arg to perf_event__preprocess_sample() Namhyung Kim
2015-03-03 13:59   ` Arnaldo Carvalho de Melo
2015-03-03 14:18     ` Namhyung Kim
2015-03-03  3:07 ` [PATCH 11/38] perf script: Pass session arg to ->process_event callback Namhyung Kim
2015-03-03  3:07 ` [PATCH 12/38] perf tools: Introduce thread__comm_time() helpers Namhyung Kim
2015-03-03 16:28   ` Frederic Weisbecker
2015-03-04  0:02     ` Namhyung Kim
2015-03-05 16:08       ` Frederic Weisbecker
2015-03-06  4:38         ` Namhyung Kim
2015-03-06 12:48           ` Arnaldo Carvalho de Melo
2015-03-10  6:55             ` Namhyung Kim
2015-03-03  3:07 ` [PATCH 13/38] perf tools: Add a test case for thread comm handling Namhyung Kim
2015-03-03  3:07 ` [PATCH 14/38] perf tools: Use thread__comm_time() when adding hist entries Namhyung Kim
2015-03-03  3:07 ` [PATCH 15/38] perf tools: Convert dead thread list into rbtree Namhyung Kim
2015-03-03  3:07 ` [PATCH 16/38] perf tools: Introduce machine__find*_thread_time() Namhyung Kim
2015-03-03  3:07 ` [PATCH 17/38] perf tools: Add a test case for timed thread handling Namhyung Kim
2015-03-03  3:07 ` [PATCH 18/38] perf tools: Reducing arguments of hist_entry_iter__add() Namhyung Kim
2015-03-03  3:07 ` [PATCH 19/38] perf tools: Pass session to hist_entry_iter struct Namhyung Kim
2015-03-03  3:07 ` [PATCH 20/38] perf tools: Maintain map groups list in a leader thread Namhyung Kim
2015-03-03  3:07 ` [PATCH 21/38] perf tools: Introduce session__find_addr_location() and friends Namhyung Kim
2015-03-03  3:07 ` [PATCH 22/38] perf callchain: Use " Namhyung Kim
2015-03-03 14:01   ` Arnaldo Carvalho de Melo
2015-03-03  3:07 ` [PATCH 23/38] perf tools: Add a test case for timed map groups handling Namhyung Kim
2015-03-03  3:07 ` [PATCH 24/38] perf tools: Protect dso symbol loading using a mutex Namhyung Kim
2015-03-03  3:07 ` [PATCH 25/38] perf tools: Protect dso cache tree using dso->lock Namhyung Kim
2015-03-03  3:07 ` [PATCH 26/38] perf tools: Protect dso cache fd with a mutex Namhyung Kim
2015-03-03  3:07 ` [PATCH 27/38] perf callchain: Maintain libunwind's address space in map_groups Namhyung Kim
2015-03-03  3:07 ` [PATCH 28/38] perf tools: Add dso__data_get/put_fd() Namhyung Kim
2015-03-03  3:07 ` [PATCH 29/38] perf session: Pass struct events stats to event processing functions Namhyung Kim
2015-03-03  3:07 ` [PATCH 30/38] perf hists: Pass hists struct to hist_entry_iter struct Namhyung Kim
2015-03-03  3:07 ` [PATCH 31/38] perf tools: Move BUILD_ID_SIZE definition to perf.h Namhyung Kim
2015-03-03  3:07 ` [PATCH 32/38] perf report: Parallelize perf report using multi-thread Namhyung Kim
2015-03-03  3:07 ` [PATCH 33/38] perf tools: Add missing_threads rb tree Namhyung Kim
2015-03-03  3:07 ` [PATCH 34/38] perf record: Synthesize COMM event for a command line workload Namhyung Kim
2015-03-03  3:07 ` [PATCH 35/38] perf tools: Fix progress ui to support multi thread Namhyung Kim
2015-03-03  3:07 ` [PATCH 36/38] perf report: Add --multi-thread option and config item Namhyung Kim
2015-03-03  3:07 ` [PATCH 37/38] perf session: Handle index files generally Namhyung Kim
2015-03-03  3:07 ` [PATCH 38/38] perf data: Implement 'index' subcommand Namhyung Kim
2015-05-18  0:30 [RFC/PATCHSET 00/40] perf tools: Speed-up perf report by using multi thread (v4) Namhyung Kim
2015-05-18  0:30 ` [PATCH 01/40] perf tools: Use a software dummy event to track task/mmap events Namhyung Kim
2015-05-18  0:30 ` [PATCH 02/40] perf tools: Add rm_rf() utility function Namhyung Kim
2015-05-18  0:30 ` [PATCH 03/40] perf tools: Introduce copyfile_offset() function Namhyung Kim
2015-05-18 12:57   ` Arnaldo Carvalho de Melo
2015-05-19  6:20     ` Namhyung Kim
2015-05-20 12:24   ` [tip:perf/core] " tip-bot for Namhyung Kim
2015-05-18  0:30 ` [PATCH 04/40] perf tools: Create separate mmap for dummy tracking event Namhyung Kim
2015-05-18 18:07   ` Jiri Olsa
2015-05-19  6:24     ` Namhyung Kim
2015-05-18  0:30 ` [PATCH 05/40] perf tools: Introduce perf_evlist__mmap_track() Namhyung Kim
2015-05-18 18:09   ` Jiri Olsa
2015-05-19  6:28     ` Namhyung Kim
2015-05-18  0:30 ` [PATCH 06/40] perf tools: Add HEADER_DATA_INDEX feature Namhyung Kim
2015-05-18 18:17   ` Jiri Olsa
2015-05-19  6:34     ` Namhyung Kim
2015-05-18  0:30 ` [PATCH 07/40] perf tools: Handle indexed data file properly Namhyung Kim
2015-05-18 18:37   ` Jiri Olsa
2015-05-19  6:40     ` Namhyung Kim
2015-05-18  0:30 ` [PATCH 08/40] perf record: Add --index option for building index table Namhyung Kim
2015-05-18  0:30 ` [PATCH 09/40] perf report: Skip dummy tracking event Namhyung Kim
2015-05-18  0:30 ` [PATCH 10/40] perf tools: Introduce thread__comm(_str)_by_time() helpers Namhyung Kim
2015-05-18  0:30 ` [PATCH 11/40] perf tools: Add a test case for thread comm handling Namhyung Kim
2015-05-18 19:29   ` Jiri Olsa
2015-05-19  6:42     ` Namhyung Kim
2015-05-18  0:30 ` [PATCH 12/40] perf tools: Use thread__comm_by_time() when adding hist entries Namhyung Kim
2015-05-18  0:30 ` [PATCH 13/40] perf tools: Convert dead thread list into rbtree Namhyung Kim
2015-05-18 19:34   ` Jiri Olsa
2015-05-19  6:45     ` Namhyung Kim
2015-05-18  0:30 ` [PATCH 14/40] perf tools: Introduce machine__find*_thread_by_time() Namhyung Kim
2015-05-18 19:50   ` Jiri Olsa
2015-05-18 19:56     ` Jiri Olsa
2015-05-19  6:57     ` Namhyung Kim
2015-05-18  0:30 ` [PATCH 15/40] perf tools: Add a test case for timed thread handling Namhyung Kim
2015-05-18  0:30 ` [PATCH 16/40] perf tools: Reducing arguments of hist_entry_iter__add() Namhyung Kim
2015-05-18 12:55   ` Arnaldo Carvalho de Melo
2015-05-19  7:01     ` Namhyung Kim
2015-05-19  8:04       ` [PATCH] " Namhyung Kim
2015-05-19 14:03         ` Arnaldo Carvalho de Melo
2015-05-27 16:47         ` [tip:perf/core] perf hists: " tip-bot for Namhyung Kim
2015-05-18  0:30 ` [PATCH 17/40] perf tools: Maintain map groups list in a leader thread Namhyung Kim
2015-05-18  0:30 ` [PATCH 18/40] perf tools: Introduce thread__find_addr_location_by_time() and friends Namhyung Kim
2015-05-18  0:30 ` [PATCH 19/40] perf callchain: Use " Namhyung Kim
2015-05-18  0:30 ` [PATCH 20/40] perf tools: Add a test case for timed map groups handling Namhyung Kim
2015-05-18  0:30 ` [PATCH 21/40] perf tools: Save timestamp of a map creation Namhyung Kim
2015-05-18  0:30 ` [PATCH 22/40] perf tools: Introduce map_groups__{insert,find}_by_time() Namhyung Kim
2015-05-18  0:30 ` [PATCH 23/40] perf tools: Use map_groups__find_addr_by_time() Namhyung Kim
2015-05-18  0:30 ` [PATCH 24/40] perf tools: Add testcase for managing maps with time Namhyung Kim
2015-05-18  0:30 ` [PATCH 25/40] perf tools: Protect dso symbol loading using a mutex Namhyung Kim
2015-05-20 12:25   ` [tip:perf/core] perf symbols: " tip-bot for Namhyung Kim
2015-05-18  0:30 ` [PATCH 26/40] perf tools: Protect dso cache tree using dso->lock Namhyung Kim
2015-05-20 12:25   ` [tip:perf/core] perf symbols: Protect dso cache tree using dso-> lock tip-bot for Namhyung Kim
2015-05-18  0:30 ` [PATCH 27/40] perf tools: Protect dso cache fd with a mutex Namhyung Kim
2015-05-20 12:25   ` [tip:perf/core] " tip-bot for Namhyung Kim
2015-05-18  0:30 ` [PATCH 28/40] perf callchain: Maintain libunwind's address space in map_groups Namhyung Kim
2015-05-18  0:30 ` [PATCH 29/40] perf tools: Add dso__data_get/put_fd() Namhyung Kim
2015-05-18  0:30 ` [PATCH 30/40] perf session: Pass struct events stats to event processing functions Namhyung Kim
2015-05-18  0:30 ` [PATCH 31/40] perf hists: Pass hists struct to hist_entry_iter struct Namhyung Kim
2015-05-18  0:30 ` [PATCH 32/40] perf tools: Move BUILD_ID_SIZE definition to perf.h Namhyung Kim
2015-05-18  0:30 ` [PATCH 33/40] perf session: Separate struct machines from session Namhyung Kim
2015-05-18 12:52   ` Arnaldo Carvalho de Melo
2015-05-19  7:28     ` Namhyung Kim
2015-05-19 22:46       ` Arnaldo Carvalho de Melo
2015-05-19 23:58         ` Namhyung Kim
2015-05-28  2:39           ` [RFC 0/2] " Namhyung Kim
2015-05-28  2:39             ` [PATCH 1/2] perf tools: Introduce machines__new/delete() Namhyung Kim
2015-05-28  2:39             ` [PATCH 2/2] perf session: Separate struct machines from session Namhyung Kim
2015-05-18  0:30 ` [PATCH 34/40] perf report: Parallelize perf report using multi-thread Namhyung Kim
2015-05-19 10:12   ` Jiri Olsa
2015-05-19 14:58     ` Namhyung Kim
2015-05-18  0:30 ` [PATCH 35/40] perf record: Synthesize COMM event for a command line workload Namhyung Kim
2015-05-18 12:45   ` Arnaldo Carvalho de Melo
2015-05-19  7:46     ` Namhyung Kim
2015-05-19 14:02       ` Arnaldo Carvalho de Melo
2015-05-19 15:12         ` Namhyung Kim
2015-05-19 16:30           ` Arnaldo Carvalho de Melo
2015-05-19 19:49           ` Jiri Olsa
2015-05-19 20:18             ` Arnaldo Carvalho de Melo
2015-05-19 23:56               ` Namhyung Kim
2015-05-20  0:22                 ` Arnaldo Carvalho de Melo
2015-05-20  0:56                   ` Namhyung Kim
2015-05-20  1:16                     ` Arnaldo Carvalho de Melo
2015-05-18  0:30 ` [PATCH 36/40] perf tools: Fix progress ui to support multi thread Namhyung Kim
2015-05-18  0:30 ` [PATCH 37/40] perf report: Add --multi-thread option and config item Namhyung Kim
2015-05-18  0:30 ` [PATCH 38/40] perf session: Handle index files generally Namhyung Kim
2015-05-19 22:27   ` David Ahern
2015-05-20  0:05     ` Namhyung Kim
2015-05-20  7:20       ` [PATCH 41/40] perf report: Add --num-thread option to control number of thread Namhyung Kim
2015-05-18  0:30 ` [PATCH 39/40] perf data: Implement 'index' subcommand Namhyung Kim
2015-05-18  0:30 ` [PATCH Not-for-merge 40/40] perf tools: Disable thread refcount due to bug Namhyung Kim
2015-05-18  1:23   ` Arnaldo Carvalho de Melo
2015-05-18 12:21     ` Namhyung Kim
2015-05-18 12:30       ` Arnaldo Carvalho de Melo

This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.