LKML Archive on lore.kernel.org
 help / color / Atom feed
From: Jiri Olsa <jolsa@kernel.org>
To: Arnaldo Carvalho de Melo <acme@kernel.org>
Cc: lkml <linux-kernel@vger.kernel.org>,
	Ingo Molnar <mingo@kernel.org>,
	Namhyung Kim <namhyung@kernel.org>,
	Alexander Shishkin <alexander.shishkin@linux.intel.com>,
	Peter Zijlstra <a.p.zijlstra@chello.nl>,
	Andi Kleen <andi@firstfloor.org>,
	Alexey Budankov <alexey.budankov@linux.intel.com>
Subject: [RFCv2 00/48] perf tools: Add threads to record command
Date: Thu, 13 Sep 2018 14:54:02 +0200
Message-ID: <20180913125450.21342-1-jolsa@kernel.org> (raw)

hi,
sending *RFC* for threads support in perf record command.

In big picture this patchset adds perf record --threads
option that allows to create threads in following modes:

1) single thread mode (current)

  $ perf record ...
  $ perf record --threads=1 ...

  - all maps are read/stored under process thread

2) mode with specific (X) number of threads

  $ perf record --threads=X ...

  - maps are spread equaly among threads

3) mode that creates thread for every monitored memory map

  $ perf record --threads ...

  - which in perf record is equal to number of CPUs, and
    it pins each thread to its map's cpu:

4) TODO - NUMA aware threads/maps separation
   ...

The perf.data stays as a single file.

v2 changes:
  - rebased to current Arnaldo's perf/core
    (also based on few fixes from my perf/core, see the branch details below)

This patchset contains lot of preparation changes to make
threaded record possible:

  - Namhyung's changes to create multiple data streams in
    perf data file, which allows having each thread data
    being stored in separate files and merged into single
    perf data after

  - Namhyung's changes to create track mmaps for auxiliary
    events

  - Namhyung's changes to search for threads/mmaps/comms
    using the time. This is needed because we have now
    multiple data streams which are processed separately,
    but they all need access to complete auxiliary events
    data (threads/mmaps/comms). That's also a reason why
    the auxiliary events are stored into separate data
    stream, which is processed before real data.

  - the rest of the code that adds threads abstraction into
    record command allows to create them and distribute maps
    among them

  - other preparational changes

The threaded monitoring currently can't monitor backward maps
and there are probably more limitations which I haven't spotted
yet.

So far I tested on laptop:
  http://people.redhat.com/~jolsa/record_threads/test-4CPU.txt

and a one bigger server:
  http://people.redhat.com/~jolsa/record_threads/test-208CPU.txt

I can see decrease in recorded LOST events, but both the benchmark
and the monitoring must be carefully configured wrt:
  - number of events (frequency)
  - size of the memory maps
  - size of events (callchains)
  - final perf.data size

It's also available in:
  git://git.kernel.org/pub/scm/linux/kernel/git/jolsa/perf.git
  perf/record_threads

thoughts? ;-) thanks
jirka


---
Jiri Olsa (30):
      perf tools: Remove perf_tool from event_op2
      perf tools: Remove perf_tool from event_op3
      perf tools: Pass struct perf_mmap into auxtrace_mmap__read* functions
      perf tools: Add struct perf_mmap arg into record__write
      perf tools: Create separate mmap for dummy tracking event
      perf tools: Make copyfile_offset global
      perf tools: Add perf_data__create_index function
      perf record: Add --index option for building index table
      perf tools: Convert dead thread list into rbtree
      perf tools: Add thread::exited flag
      perf callchain: Maintain libunwind's address space in map_groups
      perf tools: Rename perf_evlist__munmap_filtered to perf_mmap__put_filtered
      tools lib fd array: Introduce fdarray__add_clone function
      tools lib subcmd: Add OPT_INTEGER_OPTARG|_SET options
      perf tools: Move __perf_session__process_events args into struct
      perf ui progress: Fix index progress display
      perf tools: Add threads debug variable
      perf tools: Add perf_mmap__read_tail function
      perf record: Introduce struct record_thread
      perf record: Read record thread's mmaps
      perf record: Move waking into struct record
      perf record: Move samples into struct record_thread
      perf record: Move bytes_written into struct record_thread
      perf record: Add record_thread start/stop/process functions
      perf record: Wait for all threads being started
      perf record: Add --threads option
      perf record: Add --thread-stats option support
      perf record: Add maps to --thread-stats output
      perf record: Spread maps for --threads option
      perf record: Spread maps for --threads=X option

Namhyung Kim (18):
      perf tools: Use a software dummy event to track task/mmap events
      perf tools: Extend perf_evlist__mmap_ex() to use track mmap
      perf report: Skip dummy tracking event
      perf tools: Add HEADER_DATA_INDEX feature
      perf tools: Handle indexed data file properly
      perf tools: Introduce thread__comm(_str)_by_time() helpers
      perf tools: Add a test case for thread comm handling
      perf tools: Use thread__comm_by_time() when adding hist entries
      perf tools: Introduce machine__find*_thread_by_time()
      perf tools: Add a test case for timed thread handling
      perf tools: Maintain map groups list in a leader thread
      perf tools: Introduce thread__find_symbol_by_time() and friends
      perf callchain: Use thread__find_addr_location_by_time() and friends
      perf tools: Add a test case for timed map groups handling
      perf tools: Save timestamp of a map creation
      perf tools: Introduce map_groups__{insert,find}_by_time()
      perf tools: Use map_groups__find_addr_by_time()
      perf tools: Add testcase for managing maps with time

 tools/lib/api/fd/array.c                 |  17 +
 tools/lib/api/fd/array.h                 |   1 +
 tools/lib/subcmd/parse-options.c         |   2 +
 tools/lib/subcmd/parse-options.h         |   9 +
 tools/perf/Documentation/perf-record.txt |   4 +
 tools/perf/Documentation/perf.txt        |   1 +
 tools/perf/builtin-annotate.c            |   7 +-
 tools/perf/builtin-inject.c              |  32 +-
 tools/perf/builtin-record.c              | 899 +++++++++++++++++++++++++++++--
 tools/perf/builtin-report.c              |  12 +-
 tools/perf/builtin-script.c              |  38 +-
 tools/perf/builtin-stat.c                |  23 +-
 tools/perf/perf.c                        |   1 +
 tools/perf/perf.h                        |   3 +
 tools/perf/tests/Build                   |   4 +
 tools/perf/tests/builtin-test.c          |  16 +
 tools/perf/tests/dwarf-unwind.c          |   4 +-
 tools/perf/tests/hists_common.c          |   2 +-
 tools/perf/tests/hists_link.c            |   2 +-
 tools/perf/tests/tests.h                 |   4 +
 tools/perf/tests/thread-comm.c           |  48 ++
 tools/perf/tests/thread-lookup-time.c    | 181 +++++++
 tools/perf/tests/thread-map-time.c       |  90 ++++
 tools/perf/tests/thread-mg-share.c       |   7 +-
 tools/perf/tests/thread-mg-time.c        |  94 ++++
 tools/perf/ui/browsers/hists.c           |  30 +-
 tools/perf/ui/gtk/hists.c                |   3 +
 tools/perf/util/auxtrace.c               |  30 +-
 tools/perf/util/auxtrace.h               |  21 +-
 tools/perf/util/data.c                   |  64 +++
 tools/perf/util/data.h                   |   5 +
 tools/perf/util/debug.c                  |   2 +
 tools/perf/util/debug.h                  |   1 +
 tools/perf/util/dso.c                    |   2 +-
 tools/perf/util/event.c                  | 135 ++++-
 tools/perf/util/evlist.c                 |  96 +++-
 tools/perf/util/evlist.h                 |   7 +-
 tools/perf/util/evsel.h                  |  15 +
 tools/perf/util/header.c                 |  93 +++-
 tools/perf/util/header.h                 |  18 +-
 tools/perf/util/hist.c                   |   4 +-
 tools/perf/util/intel-pt.c               |   2 +-
 tools/perf/util/machine.c                | 293 ++++++++--
 tools/perf/util/machine.h                |  22 +-
 tools/perf/util/map.c                    |  79 ++-
 tools/perf/util/map.h                    |  40 +-
 tools/perf/util/mmap.c                   |   6 +-
 tools/perf/util/mmap.h                   |  33 +-
 tools/perf/util/session.c                | 178 +++---
 tools/perf/util/session.h                |   5 +-
 tools/perf/util/stat.c                   |   5 +-
 tools/perf/util/stat.h                   |   5 +-
 tools/perf/util/symbol-elf.c             |   2 +-
 tools/perf/util/symbol.c                 |   4 +-
 tools/perf/util/thread.c                 | 200 ++++++-
 tools/perf/util/thread.h                 |  27 +-
 tools/perf/util/tool.h                   |   7 +-
 tools/perf/util/unwind-libdw.c           |   6 +-
 tools/perf/util/unwind-libunwind-local.c |  39 +-
 tools/perf/util/unwind-libunwind.c       |   9 +-
 tools/perf/util/unwind.h                 |   7 +-
 tools/perf/util/util.c                   |   2 +-
 tools/perf/util/util.h                   |   2 +
 63 files changed, 2608 insertions(+), 392 deletions(-)
 create mode 100644 tools/perf/tests/thread-comm.c
 create mode 100644 tools/perf/tests/thread-lookup-time.c
 create mode 100644 tools/perf/tests/thread-map-time.c
 create mode 100644 tools/perf/tests/thread-mg-time.c

             reply index

Thread overview: 101+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2018-09-13 12:54 Jiri Olsa [this message]
2018-09-13 12:54 ` [PATCH 01/48] perf tools: Remove perf_tool from event_op2 Jiri Olsa
2018-09-25  9:31   ` [tip:perf/core] " tip-bot for Jiri Olsa
2018-09-13 12:54 ` [PATCH 02/48] perf tools: Remove perf_tool from event_op3 Jiri Olsa
2018-09-18 20:56   ` Arnaldo Carvalho de Melo
2018-09-23 19:45     ` Jiri Olsa
2018-09-25  9:31   ` [tip:perf/core] " tip-bot for Jiri Olsa
2018-09-13 12:54 ` [PATCH 03/48] perf tools: Pass struct perf_mmap into auxtrace_mmap__read* functions Jiri Olsa
2018-09-25  9:32   ` [tip:perf/core] perf auxtrace: Pass struct perf_mmap into mmap__read* functions tip-bot for Jiri Olsa
2018-09-13 12:54 ` [PATCH 04/48] perf tools: Add struct perf_mmap arg into record__write Jiri Olsa
2018-09-25  9:32   ` [tip:perf/core] perf tools: Add 'struct perf_mmap' arg to record__write() tip-bot for Jiri Olsa
2018-09-13 12:54 ` [PATCH 05/48] perf tools: Use a software dummy event to track task/mmap events Jiri Olsa
2018-09-13 12:54 ` [PATCH 06/48] perf tools: Create separate mmap for dummy tracking event Jiri Olsa
2018-09-13 12:54 ` [PATCH 07/48] perf tools: Extend perf_evlist__mmap_ex() to use track mmap Jiri Olsa
2018-09-13 12:54 ` [PATCH 08/48] perf report: Skip dummy tracking event Jiri Olsa
2018-09-13 12:54 ` [PATCH 09/48] perf tools: Make copyfile_offset global Jiri Olsa
2018-09-18 20:54   ` Arnaldo Carvalho de Melo
2018-09-23 19:44     ` Jiri Olsa
2018-09-25  9:33   ` [tip:perf/core] perf util: Make copyfile_offset() global tip-bot for Jiri Olsa
2018-09-13 12:54 ` [PATCH 10/48] perf tools: Add HEADER_DATA_INDEX feature Jiri Olsa
2018-09-13 12:54 ` [PATCH 11/48] perf tools: Handle indexed data file properly Jiri Olsa
2018-09-13 12:54 ` [PATCH 12/48] perf tools: Add perf_data__create_index function Jiri Olsa
2018-09-13 12:54 ` [PATCH 13/48] perf record: Add --index option for building index table Jiri Olsa
2018-09-13 12:54 ` [PATCH 14/48] perf tools: Introduce thread__comm(_str)_by_time() helpers Jiri Olsa
2018-09-13 12:54 ` [PATCH 15/48] perf tools: Add a test case for thread comm handling Jiri Olsa
2018-09-13 12:54 ` [PATCH 16/48] perf tools: Use thread__comm_by_time() when adding hist entries Jiri Olsa
2018-09-13 12:54 ` [PATCH 17/48] perf tools: Convert dead thread list into rbtree Jiri Olsa
2018-09-13 12:54 ` [PATCH 18/48] perf tools: Introduce machine__find*_thread_by_time() Jiri Olsa
2018-09-13 12:54 ` [PATCH 19/48] perf tools: Add thread::exited flag Jiri Olsa
2018-09-13 12:54 ` [PATCH 20/48] perf tools: Add a test case for timed thread handling Jiri Olsa
2018-09-13 12:54 ` [PATCH 21/48] perf tools: Maintain map groups list in a leader thread Jiri Olsa
2018-09-13 12:54 ` [PATCH 22/48] perf tools: Introduce thread__find_symbol_by_time() and friends Jiri Olsa
2018-09-13 12:54 ` [PATCH 23/48] perf callchain: Use thread__find_addr_location_by_time() " Jiri Olsa
2018-09-13 12:54 ` [PATCH 24/48] perf tools: Add a test case for timed map groups handling Jiri Olsa
2018-09-13 12:54 ` [PATCH 25/48] perf tools: Save timestamp of a map creation Jiri Olsa
2018-09-13 12:54 ` [PATCH 26/48] perf tools: Introduce map_groups__{insert,find}_by_time() Jiri Olsa
2018-09-13 12:54 ` [PATCH 27/48] perf tools: Use map_groups__find_addr_by_time() Jiri Olsa
2018-09-13 12:54 ` [PATCH 28/48] perf tools: Add testcase for managing maps with time Jiri Olsa
2018-09-13 12:54 ` [PATCH 29/48] perf callchain: Maintain libunwind's address space in map_groups Jiri Olsa
2018-09-14 18:15   ` Arnaldo Carvalho de Melo
2018-09-14 19:00     ` Jiri Olsa
2018-09-13 12:54 ` [PATCH 30/48] perf tools: Rename perf_evlist__munmap_filtered to perf_mmap__put_filtered Jiri Olsa
2018-09-13 12:54 ` [PATCH 31/48] tools lib fd array: Introduce fdarray__add_clone function Jiri Olsa
2018-09-13 12:54 ` [PATCH 32/48] tools lib subcmd: Add OPT_INTEGER_OPTARG|_SET options Jiri Olsa
2018-09-13 12:54 ` [PATCH 33/48] perf tools: Move __perf_session__process_events args into struct Jiri Olsa
2018-09-13 12:54 ` [PATCH 34/48] perf ui progress: Fix index progress display Jiri Olsa
2018-09-13 12:54 ` [PATCH 35/48] perf tools: Add threads debug variable Jiri Olsa
2018-09-13 12:54 ` [PATCH 36/48] perf tools: Add perf_mmap__read_tail function Jiri Olsa
2018-09-13 12:54 ` [PATCH 37/48] perf record: Introduce struct record_thread Jiri Olsa
2018-09-17 11:26   ` Namhyung Kim
2018-09-23 19:31     ` Jiri Olsa
2018-09-13 12:54 ` [PATCH 38/48] perf record: Read record thread's mmaps Jiri Olsa
2018-09-17 11:28   ` Namhyung Kim
2018-09-23 19:35     ` Jiri Olsa
2018-09-13 12:54 ` [PATCH 39/48] perf record: Move waking into struct record Jiri Olsa
2018-09-17 11:31   ` Namhyung Kim
2018-09-23 19:36     ` Jiri Olsa
2018-09-13 12:54 ` [PATCH 40/48] perf record: Move samples into struct record_thread Jiri Olsa
2018-09-13 12:54 ` [PATCH 41/48] perf record: Move bytes_written " Jiri Olsa
2018-09-13 12:54 ` [PATCH 42/48] perf record: Add record_thread start/stop/process functions Jiri Olsa
2018-09-13 12:54 ` [PATCH 43/48] perf record: Wait for all threads being started Jiri Olsa
2018-09-13 12:54 ` [PATCH 44/48] perf record: Add --threads option Jiri Olsa
2018-09-17 11:37   ` Namhyung Kim
2018-09-13 12:54 ` [PATCH 45/48] perf record: Add --thread-stats option support Jiri Olsa
2018-09-13 12:54 ` [PATCH 46/48] perf record: Add maps to --thread-stats output Jiri Olsa
2018-09-13 12:54 ` [PATCH 47/48] perf record: Spread maps for --threads option Jiri Olsa
2018-09-17 11:40   ` Namhyung Kim
2018-09-23 19:44     ` Jiri Olsa
2018-09-24 14:22       ` Arnaldo Carvalho de Melo
2018-09-26  6:23         ` Jiri Olsa
2018-09-27 16:01           ` Jiri Olsa
2018-09-28  6:25             ` Namhyung Kim
2018-09-13 12:54 ` [PATCH 48/48] perf record: Spread maps for --threads=X option Jiri Olsa
2018-09-13 16:10 ` [RFCv2 00/48] perf tools: Add threads to record command Alexey Budankov
2018-09-14  2:29   ` Namhyung Kim
2018-09-14  7:15     ` Alexey Budankov
2018-09-14  8:23     ` Jiri Olsa
2018-09-14  9:40       ` Ingo Molnar
2018-09-14 11:15         ` Peter Zijlstra
2018-09-14 11:47           ` Jiri Olsa
2018-09-14 12:01             ` Peter Zijlstra
2018-09-14 12:13               ` Ingo Molnar
2018-09-14 12:19                 ` Jiri Olsa
2018-09-14 12:45                   ` Ingo Molnar
2018-09-14  9:33     ` Ingo Molnar
2018-09-14  8:26   ` Jiri Olsa
2018-09-14  8:28     ` Jiri Olsa
2018-09-14  9:37       ` Alexey Budankov
2018-09-21  6:13         ` Alexey Budankov
2018-09-21 12:15           ` Alexey Budankov
2018-09-24 19:23             ` Alexey Budankov
2018-10-02 21:41               ` Jiri Olsa
2018-10-03  7:01                 ` Alexey Budankov
2018-09-23 19:30           ` Jiri Olsa
2018-09-24  7:02             ` Alexey Budankov
2018-09-24 13:09               ` Alexey Budankov
2018-09-24 14:29                 ` Jiri Olsa
2018-09-24 18:32                   ` Alexey Budankov
2018-09-24 19:12                     ` Alexey Budankov
2018-10-05  6:14                     ` Namhyung Kim
2018-09-14 17:02 ` Andi Kleen

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20180913125450.21342-1-jolsa@kernel.org \
    --to=jolsa@kernel.org \
    --cc=a.p.zijlstra@chello.nl \
    --cc=acme@kernel.org \
    --cc=alexander.shishkin@linux.intel.com \
    --cc=alexey.budankov@linux.intel.com \
    --cc=andi@firstfloor.org \
    --cc=linux-kernel@vger.kernel.org \
    --cc=mingo@kernel.org \
    --cc=namhyung@kernel.org \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link

LKML Archive on lore.kernel.org

Archives are clonable:
	git clone --mirror https://lore.kernel.org/lkml/0 lkml/git/0.git
	git clone --mirror https://lore.kernel.org/lkml/1 lkml/git/1.git
	git clone --mirror https://lore.kernel.org/lkml/2 lkml/git/2.git
	git clone --mirror https://lore.kernel.org/lkml/3 lkml/git/3.git
	git clone --mirror https://lore.kernel.org/lkml/4 lkml/git/4.git
	git clone --mirror https://lore.kernel.org/lkml/5 lkml/git/5.git
	git clone --mirror https://lore.kernel.org/lkml/6 lkml/git/6.git
	git clone --mirror https://lore.kernel.org/lkml/7 lkml/git/7.git
	git clone --mirror https://lore.kernel.org/lkml/8 lkml/git/8.git
	git clone --mirror https://lore.kernel.org/lkml/9 lkml/git/9.git
	git clone --mirror https://lore.kernel.org/lkml/10 lkml/git/10.git

	# If you have public-inbox 1.1+ installed, you may
	# initialize and index your mirror using the following commands:
	public-inbox-init -V2 lkml lkml/ https://lore.kernel.org/lkml \
		linux-kernel@vger.kernel.org
	public-inbox-index lkml

Example config snippet for mirrors

Newsgroup available over NNTP:
	nntp://nntp.lore.kernel.org/org.kernel.vger.linux-kernel


AGPL code for this site: git clone https://public-inbox.org/public-inbox.git