From: Riccardo Mancini <rickyman7@gmail.com>
To: Arnaldo Carvalho de Melo <acme@kernel.org>
Cc: Ian Rogers <irogers@google.com>,
Namhyung Kim <namhyung@kernel.org>,
Peter Zijlstra <peterz@infradead.org>,
Ingo Molnar <mingo@redhat.com>,
Mark Rutland <mark.rutland@arm.com>, Jiri Olsa <jolsa@redhat.com>,
linux-kernel@vger.kernel.org, linux-perf-users@vger.kernel.org,
Riccardo Mancini <rickyman7@gmail.com>
Subject: [RFC PATCH 00/10] perf: add workqueue library and use it in synthetic-events
Date: Tue, 13 Jul 2021 14:11:11 +0200 [thread overview]
Message-ID: <cover.1626177381.git.rickyman7@gmail.com> (raw)
This patchset introduces a new utility library inside perf/util, which
provides a work queue abstraction, which loosely follows the Kernel
workqueue API.
The workqueue abstraction is made up by two components:
- threadpool: which takes care of managing a pool of threads. It is
inspired by the prototype for threaded trace in perf-record from Alexey:
https://lore.kernel.org/lkml/cover.1625227739.git.alexey.v.bayduraev@linux.intel.com/
- workqueue: manages a shared queue and provides the workers implementation.
On top of the workqueue, a simple parallel-for utility is implemented
which is then showcased in synthetic-events.c, replacing the previous
manual pthread-created threads.
Through some experiments with perf bench, I can see how the new
workqueue has a higher overhead compared to manual creation of threads,
but is able to more effectively partition work among threads, yielding
a better result with more threads.
Furthermore, the overhead could be configured by changing the
`work_size` (currently 1), aka the number of dirents that are
processed by a thread before grabbing a lock to get the new work item.
I experimented with different sizes but, while bigger sizes reduce overhead
as expected, they do not scale as well to more threads.
I tried to keep the patchset as simple as possible, deferring possible
improvements and features to future work.
Naming a few:
- in order to achieve a better performance, we could consider using
work-stealing instead of a common queue.
- affinities in the thread pool, as in Alexey prototype for
perf-record. Doing so would enable reusing the same threadpool for
different purposes (evlist open, threaded trace, synthetic threads),
avoiding having to spin up threads multiple times.
- resizable threadpool, e.g. for lazy spawining of threads.
@Arnaldo
Since I wanted the workqueue to provide a similar API to the Kernel's
workqueue, I followed the naming style I found there, instead of the
usual object__method style that is typically found in perf.
Let me know if you'd like me to follow perf style instead.
Thanks,
Riccardo
Riccardo Mancini (10):
perf workqueue: threadpool creation and destruction
perf tests: add test for workqueue
perf workqueue: add threadpool start and stop functions
perf workqueue: add threadpool execute and wait functions
perf workqueue: add sparse annotation header
perf workqueue: introduce workqueue struct
perf workqueue: implement worker thread and management
perf workqueue: add queue_work and flush_workqueue functions
perf workqueue: add utility to execute a for loop in parallel
perf synthetic-events: use workqueue parallel_for
tools/perf/tests/Build | 1 +
tools/perf/tests/builtin-test.c | 9 +
tools/perf/tests/tests.h | 3 +
tools/perf/tests/workqueue.c | 453 +++++++++++++++++
tools/perf/util/Build | 1 +
tools/perf/util/synthetic-events.c | 131 +++--
tools/perf/util/workqueue/Build | 2 +
tools/perf/util/workqueue/sparse.h | 21 +
tools/perf/util/workqueue/threadpool.c | 516 ++++++++++++++++++++
tools/perf/util/workqueue/threadpool.h | 29 ++
tools/perf/util/workqueue/workqueue.c | 642 +++++++++++++++++++++++++
tools/perf/util/workqueue/workqueue.h | 38 ++
12 files changed, 1771 insertions(+), 75 deletions(-)
create mode 100644 tools/perf/tests/workqueue.c
create mode 100644 tools/perf/util/workqueue/Build
create mode 100644 tools/perf/util/workqueue/sparse.h
create mode 100644 tools/perf/util/workqueue/threadpool.c
create mode 100644 tools/perf/util/workqueue/threadpool.h
create mode 100644 tools/perf/util/workqueue/workqueue.c
create mode 100644 tools/perf/util/workqueue/workqueue.h
--
2.31.1
next reply other threads:[~2021-07-13 12:11 UTC|newest]
Thread overview: 33+ messages / expand[flat|nested] mbox.gz Atom feed top
2021-07-13 12:11 Riccardo Mancini [this message]
2021-07-13 12:11 ` [RFC PATCH 01/10] perf workqueue: threadpool creation and destruction Riccardo Mancini
2021-07-14 14:16 ` Arnaldo Carvalho de Melo
2021-07-15 16:31 ` Riccardo Mancini
2021-07-15 20:48 ` Arnaldo Carvalho de Melo
2021-07-15 23:29 ` Namhyung Kim
2021-07-16 13:36 ` Riccardo Mancini
2021-07-19 19:39 ` Namhyung Kim
2021-07-13 12:11 ` [RFC PATCH 02/10] perf tests: add test for workqueue Riccardo Mancini
2021-07-14 15:10 ` Arnaldo Carvalho de Melo
2021-07-15 16:33 ` Riccardo Mancini
2021-07-13 12:11 ` [RFC PATCH 03/10] perf workqueue: add threadpool start and stop functions Riccardo Mancini
2021-07-14 15:15 ` Arnaldo Carvalho de Melo
2021-07-15 16:42 ` Riccardo Mancini
2021-07-15 20:43 ` Arnaldo Carvalho de Melo
2021-07-15 23:48 ` Namhyung Kim
2021-07-16 13:53 ` Riccardo Mancini
2021-07-16 16:29 ` Arnaldo Carvalho de Melo
2021-07-13 12:11 ` [RFC PATCH 04/10] perf workqueue: add threadpool execute and wait functions Riccardo Mancini
2021-07-15 23:56 ` Namhyung Kim
2021-07-16 13:55 ` Riccardo Mancini
2021-07-13 12:11 ` [RFC PATCH 05/10] perf workqueue: add sparse annotation header Riccardo Mancini
2021-07-13 12:11 ` [RFC PATCH 06/10] perf workqueue: introduce workqueue struct Riccardo Mancini
2021-07-14 15:22 ` Arnaldo Carvalho de Melo
2021-07-15 16:49 ` Riccardo Mancini
2021-07-15 20:47 ` Arnaldo Carvalho de Melo
2021-07-13 12:11 ` [RFC PATCH 07/10] perf workqueue: implement worker thread and management Riccardo Mancini
2021-07-13 12:11 ` [RFC PATCH 08/10] perf workqueue: add queue_work and flush_workqueue functions Riccardo Mancini
2021-07-13 12:11 ` [RFC PATCH 09/10] perf workqueue: add utility to execute a for loop in parallel Riccardo Mancini
2021-07-13 12:11 ` [RFC PATCH 10/10] perf synthetic-events: use workqueue parallel_for Riccardo Mancini
2021-07-13 19:14 ` [RFC PATCH 00/10] perf: add workqueue library and use it in synthetic-events Arnaldo Carvalho de Melo
2021-07-19 21:13 ` Jiri Olsa
2021-07-22 16:15 ` Riccardo Mancini
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=cover.1626177381.git.rickyman7@gmail.com \
--to=rickyman7@gmail.com \
--cc=acme@kernel.org \
--cc=irogers@google.com \
--cc=jolsa@redhat.com \
--cc=linux-kernel@vger.kernel.org \
--cc=linux-perf-users@vger.kernel.org \
--cc=mark.rutland@arm.com \
--cc=mingo@redhat.com \
--cc=namhyung@kernel.org \
--cc=peterz@infradead.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).