From: Arnaldo Carvalho de Melo <acme@kernel.org>
To: Ingo Molnar <mingo@kernel.org>, Thomas Gleixner <tglx@linutronix.de>
Cc: Jiri Olsa <jolsa@kernel.org>, Namhyung Kim <namhyung@kernel.org>,
Clark Williams <williams@redhat.com>,
linux-kernel@vger.kernel.org, linux-perf-users@vger.kernel.org,
Stephane Eranian <eranian@google.com>,
Ian Rogers <irogers@google.com>, Jiri Olsa <jolsa@redhat.com>,
Adrian Hunter <adrian.hunter@intel.com>,
Alexander Shishkin <alexander.shishkin@linux.intel.com>,
Alexey Budankov <alexey.budankov@linux.intel.com>,
Kan Liang <kan.liang@linux.intel.com>,
Mark Rutland <mark.rutland@arm.com>,
Peter Zijlstra <peterz@infradead.org>, Tony Jones <tonyj@suse.de>,
yuzhoujian <yuzhoujian@didichuxing.com>,
Arnaldo Carvalho de Melo <acme@redhat.com>
Subject: [PATCH 08/91] perf record: Add num-synthesize-threads option
Date: Wed, 6 May 2020 12:21:11 -0300 [thread overview]
Message-ID: <20200506152234.21977-9-acme@kernel.org> (raw)
In-Reply-To: <20200506152234.21977-1-acme@kernel.org>
From: Stephane Eranian <eranian@google.com>
To control degree of parallelism of the synthesize_mmap() code which
is scanning /proc/PID/task/PID/maps and can be time consuming.
Mimic perf top way of handling the option.
If not specified will default to 1 thread, i.e. default behavior before
this option.
On a desktop computer the processing of /proc/PID/task/PID/maps isn't
slow enough to warrant parallel processing and the thread creation has
some cost - hence the default of 1. On a loaded server with
>100 cores it is possible to see synthesis times in the order of
seconds and in this case having the option is desirable.
As the processing is a synchronization point, it is legitimate to worry if
Amdahl's law will apply to this patch. Profiling with this patch in
place:
https://lore.kernel.org/lkml/20200415054050.31645-4-irogers@google.com/
shows:
...
- 32.59% __perf_event__synthesize_threads
- 32.54% __event__synthesize_thread
+ 22.13% perf_event__synthesize_mmap_events
+ 6.68% perf_event__get_comm_ids.constprop.0
+ 1.49% process_synthesized_event
+ 1.29% __GI___readdir64
+ 0.60% __opendir
...
That is the processing is 1.49% of execution time and there is plenty to
make parallel. This is shown in the benchmark in this patch:
https://lore.kernel.org/lkml/20200415054050.31645-2-irogers@google.com/
Computing performance of multi threaded perf event synthesis by
synthesizing events on CPU 0:
Number of synthesis threads: 1
Average synthesis took: 127729.000 usec (+- 3372.880 usec)
Average num. events: 21548.600 (+- 0.306)
Average time per event 5.927 usec
Number of synthesis threads: 2
Average synthesis took: 88863.500 usec (+- 385.168 usec)
Average num. events: 21552.800 (+- 0.327)
Average time per event 4.123 usec
Number of synthesis threads: 3
Average synthesis took: 83257.400 usec (+- 348.617 usec)
Average num. events: 21553.200 (+- 0.327)
Average time per event 3.863 usec
Number of synthesis threads: 4
Average synthesis took: 75093.000 usec (+- 422.978 usec)
Average num. events: 21554.200 (+- 0.200)
Average time per event 3.484 usec
Number of synthesis threads: 5
Average synthesis took: 64896.600 usec (+- 353.348 usec)
Average num. events: 21558.000 (+- 0.000)
Average time per event 3.010 usec
Number of synthesis threads: 6
Average synthesis took: 59210.200 usec (+- 342.890 usec)
Average num. events: 21560.000 (+- 0.000)
Average time per event 2.746 usec
Number of synthesis threads: 7
Average synthesis took: 54093.900 usec (+- 306.247 usec)
Average num. events: 21562.000 (+- 0.000)
Average time per event 2.509 usec
Number of synthesis threads: 8
Average synthesis took: 48938.700 usec (+- 341.732 usec)
Average num. events: 21564.000 (+- 0.000)
Average time per event 2.269 usec
Where average time per synthesized event goes from 5.927 usec with 1
thread to 2.269 usec with 8. This isn't a linear speed up as not all of
synthesize code has been made parallel. If the synthesis time was about
10 seconds then using 8 threads may bring this down to less than 4.
Signed-off-by: Stephane Eranian <eranian@google.com>
Reviewed-by: Ian Rogers <irogers@google.com>
Acked-by: Jiri Olsa <jolsa@redhat.com>
Cc: Adrian Hunter <adrian.hunter@intel.com>
Cc: Alexander Shishkin <alexander.shishkin@linux.intel.com>
Cc: Alexey Budankov <alexey.budankov@linux.intel.com>
Cc: Kan Liang <kan.liang@linux.intel.com>
Cc: Mark Rutland <mark.rutland@arm.com>
Cc: Namhyung Kim <namhyung@kernel.org>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Tony Jones <tonyj@suse.de>
Cc: yuzhoujian <yuzhoujian@didichuxing.com>
Link: http://lore.kernel.org/lkml/20200422155038.9380-1-irogers@google.com
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
---
tools/perf/Documentation/perf-record.txt | 4 +++
tools/perf/builtin-record.c | 34 ++++++++++++++++++++++--
tools/perf/util/record.h | 1 +
3 files changed, 37 insertions(+), 2 deletions(-)
diff --git a/tools/perf/Documentation/perf-record.txt b/tools/perf/Documentation/perf-record.txt
index b3f3b3f1c161..6e8b4649307c 100644
--- a/tools/perf/Documentation/perf-record.txt
+++ b/tools/perf/Documentation/perf-record.txt
@@ -596,6 +596,10 @@ Make a copy of /proc/kcore and place it into a directory with the perf data file
Limit the sample data max size, <size> is expected to be a number with
appended unit character - B/K/M/G
+--num-thread-synthesize::
+ The number of threads to run when synthesizing events for existing processes.
+ By default, the number of threads equals 1.
+
SEE ALSO
--------
linkperf:perf-stat[1], linkperf:perf-list[1], linkperf:perf-intel-pt[1]
diff --git a/tools/perf/builtin-record.c b/tools/perf/builtin-record.c
index 1ab349abe904..2e8011f179f2 100644
--- a/tools/perf/builtin-record.c
+++ b/tools/perf/builtin-record.c
@@ -43,6 +43,7 @@
#include "util/time-utils.h"
#include "util/units.h"
#include "util/bpf-event.h"
+#include "util/util.h"
#include "asm/bug.h"
#include "perf.h"
@@ -50,6 +51,7 @@
#include <inttypes.h>
#include <locale.h>
#include <poll.h>
+#include <pthread.h>
#include <unistd.h>
#include <sched.h>
#include <signal.h>
@@ -503,6 +505,20 @@ static int process_synthesized_event(struct perf_tool *tool,
return record__write(rec, NULL, event, event->header.size);
}
+static int process_locked_synthesized_event(struct perf_tool *tool,
+ union perf_event *event,
+ struct perf_sample *sample __maybe_unused,
+ struct machine *machine __maybe_unused)
+{
+ static pthread_mutex_t synth_lock = PTHREAD_MUTEX_INITIALIZER;
+ int ret;
+
+ pthread_mutex_lock(&synth_lock);
+ ret = process_synthesized_event(tool, event, sample, machine);
+ pthread_mutex_unlock(&synth_lock);
+ return ret;
+}
+
static int record__pushfn(struct mmap *map, void *to, void *bf, size_t size)
{
struct record *rec = to;
@@ -1288,6 +1304,7 @@ static int record__synthesize(struct record *rec, bool tail)
struct perf_tool *tool = &rec->tool;
int fd = perf_data__fd(data);
int err = 0;
+ event_op f = process_synthesized_event;
if (rec->opts.tail_synthesize != tail)
return 0;
@@ -1402,9 +1419,18 @@ static int record__synthesize(struct record *rec, bool tail)
if (err < 0)
pr_warning("Couldn't synthesize cgroup events.\n");
+ if (rec->opts.nr_threads_synthesize > 1) {
+ perf_set_multithreaded();
+ f = process_locked_synthesized_event;
+ }
+
err = __machine__synthesize_threads(machine, tool, &opts->target, rec->evlist->core.threads,
- process_synthesized_event, opts->sample_address,
- 1);
+ f, opts->sample_address,
+ rec->opts.nr_threads_synthesize);
+
+ if (rec->opts.nr_threads_synthesize > 1)
+ perf_set_singlethreaded();
+
out:
return err;
}
@@ -2232,6 +2258,7 @@ static struct record record = {
.default_per_cpu = true,
},
.mmap_flush = MMAP_FLUSH_DEFAULT,
+ .nr_threads_synthesize = 1,
},
.tool = {
.sample = process_sample_event,
@@ -2421,6 +2448,9 @@ static struct option __record_options[] = {
#endif
OPT_CALLBACK(0, "max-size", &record.output_max_size,
"size", "Limit the maximum size of the output file", parse_output_max_size),
+ OPT_UINTEGER(0, "num-thread-synthesize",
+ &record.opts.nr_threads_synthesize,
+ "number of threads to run for event synthesis"),
OPT_END()
};
diff --git a/tools/perf/util/record.h b/tools/perf/util/record.h
index 24316458be20..923565c3b155 100644
--- a/tools/perf/util/record.h
+++ b/tools/perf/util/record.h
@@ -68,6 +68,7 @@ struct record_opts {
int affinity;
int mmap_flush;
unsigned int comp_level;
+ unsigned int nr_threads_synthesize;
};
extern const char * const *record_usage;
--
2.21.1
next prev parent reply other threads:[~2020-05-06 15:23 UTC|newest]
Thread overview: 92+ messages / expand[flat|nested] mbox.gz Atom feed top
2020-05-06 15:21 [GIT PULL] perf/core improvements and fixes Arnaldo Carvalho de Melo
2020-05-06 15:21 ` [PATCH 01/91] perf cgroup: Avoid needless closing of unopened fd Arnaldo Carvalho de Melo
2020-05-06 15:21 ` [PATCH 02/91] perf bench: Fix div-by-zero if runtime is zero Arnaldo Carvalho de Melo
2020-05-06 15:21 ` [PATCH 03/91] perf evlist: Remove duplicate headers Arnaldo Carvalho de Melo
2020-05-06 15:21 ` [PATCH 04/91] perf script: Avoid NULL dereference on symbol Arnaldo Carvalho de Melo
2020-05-06 15:21 ` [PATCH 05/91] perf stat: Zero all the 'ena' and 'run' array slot stats for interval mode Arnaldo Carvalho de Melo
2020-05-06 15:21 ` [PATCH 06/91] perf stat: Improve runtime stat " Arnaldo Carvalho de Melo
2020-05-06 15:21 ` [PATCH 07/91] perf test session topology: Fix data path Arnaldo Carvalho de Melo
2020-05-06 15:21 ` Arnaldo Carvalho de Melo [this message]
2020-05-06 15:21 ` [PATCH 09/91] perf bench: Add a multi-threaded synthesize benchmark Arnaldo Carvalho de Melo
2020-05-06 15:21 ` [PATCH 10/91] tools api: Add a lightweight buffered reading api Arnaldo Carvalho de Melo
2020-05-06 15:21 ` [PATCH 11/91] perf synthetic events: Remove use of sscanf from /proc reading Arnaldo Carvalho de Melo
2020-05-06 15:21 ` [PATCH 12/91] perf script: Remove extraneous newline in perf_sample__fprintf_regs() Arnaldo Carvalho de Melo
2020-05-06 15:21 ` [PATCH 13/91] libtraceevent: Remove unneeded semicolon Arnaldo Carvalho de Melo
2020-05-06 15:21 ` [PATCH 14/91] perf c2c: " Arnaldo Carvalho de Melo
2020-05-06 15:21 ` [PATCH 15/91] perf tools: Remove unneeded semicolons Arnaldo Carvalho de Melo
2020-05-06 15:21 ` [PATCH 16/91] perf report: Fix warning assignment of 0/1 to bool variable Arnaldo Carvalho de Melo
2020-05-06 15:21 ` [PATCH 17/91] perf pmu: Fix function name in comment, its get_cpuid_str(), not get_cpustr() Arnaldo Carvalho de Melo
2020-05-06 15:21 ` [PATCH 18/91] perf metricgroups: Enhance JSON/metric infrastructure to handle "?" Arnaldo Carvalho de Melo
2020-05-06 15:21 ` [PATCH 19/91] perf tests expr: Added test for runtime param in metric expression Arnaldo Carvalho de Melo
2020-05-06 15:21 ` [PATCH 20/91] perf tools: Enable Hz/hz prinitg for --metric-only option Arnaldo Carvalho de Melo
2020-05-06 15:21 ` [PATCH 21/91] perf vendor events power9: Add hv_24x7 socket/chip level metric events Arnaldo Carvalho de Melo
2020-05-06 15:21 ` [PATCH 22/91] perf tools: Move routines that probe for perf API features to separate file Arnaldo Carvalho de Melo
2020-05-06 15:21 ` [PATCH 23/91] perf record: Move sb_evlist to 'struct record' Arnaldo Carvalho de Melo
2020-05-06 15:21 ` [PATCH 24/91] perf top: Move sb_evlist to 'struct perf_top' Arnaldo Carvalho de Melo
2020-05-06 15:21 ` [PATCH 25/91] perf bpf: Decouple creating the evlist from adding the SB event Arnaldo Carvalho de Melo
2020-05-06 15:21 ` [PATCH 26/91] perf parse-events: Add parse_events_option() variant that creates evlist Arnaldo Carvalho de Melo
2020-05-06 15:21 ` [PATCH 27/91] perf evlist: Move the sideband thread routines to separate object Arnaldo Carvalho de Melo
2020-05-06 15:21 ` [PATCH 28/91] perf evlist: Allow reusing the side band thread for more purposes Arnaldo Carvalho de Melo
2020-05-06 15:21 ` [PATCH 29/91] libsubcmd: Introduce OPT_CALLBACK_SET() Arnaldo Carvalho de Melo
2020-05-06 15:21 ` [PATCH 30/91] perf record: Introduce --switch-output-event Arnaldo Carvalho de Melo
2020-05-06 15:21 ` [PATCH 31/91] perf record: Move side band evlist setup to separate routine Arnaldo Carvalho de Melo
2020-05-06 15:21 ` [PATCH 32/91] libperf: Add NULL pointer check for cpu_map iteration and NULL assignment for all_cpus Arnaldo Carvalho de Melo
2020-05-06 15:21 ` [PATCH 33/91] perf parse-events: Fix memory leaks found on parse_events Arnaldo Carvalho de Melo
2020-05-06 15:21 ` [PATCH 34/91] " Arnaldo Carvalho de Melo
2020-05-06 15:21 ` [PATCH 35/91] perf parse-events: Fix another memory leaks found on parse_events() Arnaldo Carvalho de Melo
2020-05-06 15:21 ` [PATCH 36/91] libperf evlist: Fix a refcount leak Arnaldo Carvalho de Melo
2020-05-06 15:21 ` [PATCH 37/91] perf tools: Fix reading new topology attribute "core_cpus" Arnaldo Carvalho de Melo
2020-05-06 15:21 ` [PATCH 38/91] perf tools: Simplify checking if SMT is active Arnaldo Carvalho de Melo
2020-05-06 15:21 ` [PATCH 39/91] perf thread-stack: Add branch stack support Arnaldo Carvalho de Melo
2020-05-06 15:21 ` [PATCH 40/91] perf intel-pt: Consolidate thread-stack use condition Arnaldo Carvalho de Melo
2020-05-06 15:21 ` [PATCH 41/91] perf intel-pt: Change branch stack support to use thread-stacks Arnaldo Carvalho de Melo
2020-05-06 15:21 ` [PATCH 42/91] perf auxtrace: Add option to synthesize branch stack for regular events Arnaldo Carvalho de Melo
2020-05-06 15:21 ` [PATCH 43/91] perf evsel: Add support for synthesized branch stack sample type Arnaldo Carvalho de Melo
2020-05-06 15:21 ` [PATCH 44/91] perf thread-stack: Add thread_stack__br_sample_late() Arnaldo Carvalho de Melo
2020-05-06 15:21 ` [PATCH 45/91] perf intel-pt: Add support for synthesizing branch stacks for regular events Arnaldo Carvalho de Melo
2020-05-06 15:21 ` [PATCH 46/91] perf intel-pt: Update documentation about itrace G and L options Arnaldo Carvalho de Melo
2020-05-06 15:21 ` [PATCH 47/91] perf intel-pt: Update documentation about using /proc/kcore Arnaldo Carvalho de Melo
2020-05-06 15:21 ` [PATCH 48/91] perf evsel: Rename 'struct perf_evsel__sb_cb_t' to 'struct evsel__sb_cb_t' Arnaldo Carvalho de Melo
2020-05-06 15:21 ` [PATCH 49/91] perf evsel: Rename perf_evsel__nr_cpus() to evsel__nr_cpus() Arnaldo Carvalho de Melo
2020-05-06 15:21 ` [PATCH 50/91] perf evsel: Rename perf_evsel__compute_deltas() to evsel__compute_deltas() Arnaldo Carvalho de Melo
2020-05-06 15:21 ` [PATCH 51/91] perf evsel: Rename perf_evsel__find_pmu() to evsel__find_pmu() Arnaldo Carvalho de Melo
2020-05-06 15:21 ` [PATCH 52/91] perf evsel: Rename perf_evsel__is_aux_event() to evsel__is_aux_event() Arnaldo Carvalho de Melo
2020-05-06 15:21 ` [PATCH 53/91] perf evsel: Rename perf_evsel__exit() to evsel__exit() Arnaldo Carvalho de Melo
2020-05-06 15:21 ` [PATCH 54/91] perf evsel: Rename perf_evsel__config*() to evsel__config*() Arnaldo Carvalho de Melo
2020-05-06 15:21 ` [PATCH 55/91] perf evsel: Rename perf_evsel__calc_id_pos() to evsel__calc_id_pos() Arnaldo Carvalho de Melo
2020-05-06 15:21 ` [PATCH 56/91] perf evsel: Rename __perf_evsel__sample_size() to __evsel__sample_size() Arnaldo Carvalho de Melo
2020-05-06 15:22 ` [PATCH 57/91] perf evsel: Rename *perf_evsel__*name() to *evsel__*name() Arnaldo Carvalho de Melo
2020-05-06 15:22 ` [PATCH 58/91] perf evsel: Rename perf_evsel__group_desc() to evsel__group_desc() Arnaldo Carvalho de Melo
2020-05-06 15:22 ` [PATCH 59/91] perf evsel: Rename *perf_evsel__*set_sample_*() to *evsel__*set_sample_*() Arnaldo Carvalho de Melo
2020-05-06 15:22 ` [PATCH 60/91] perf evsel: Rename perf_evsel__*filter*() to evsel__*filter*() Arnaldo Carvalho de Melo
2020-05-06 15:22 ` [PATCH 61/91] perf evsel: Rename perf_evsel__open_per_*() to evsel__open_per_*() Arnaldo Carvalho de Melo
2020-05-06 15:22 ` [PATCH 62/91] perf evsel: Rename perf_evsel__{str,int}val() and other tracepoint field metehods to to evsel__*() Arnaldo Carvalho de Melo
2020-05-06 15:22 ` [PATCH 63/91] perf mem2node: Avoid double free related to realloc Arnaldo Carvalho de Melo
2020-05-06 15:22 ` [PATCH 64/91] perf doc: Pass ASCIIDOC_EXTRA as an argument Arnaldo Carvalho de Melo
2020-05-06 15:22 ` [PATCH 65/91] tools feature: Add support for detecting libpfm4 Arnaldo Carvalho de Melo
2020-05-06 15:22 ` [PATCH 66/91] perf pmu: Add perf_pmu__find_by_type helper Arnaldo Carvalho de Melo
2020-05-06 15:22 ` [PATCH 67/91] perf evsel: Rename perf_evsel__is_*() to evsel__is*() Arnaldo Carvalho de Melo
2020-05-06 15:22 ` [PATCH 68/91] perf evsel: Ditch perf_evsel__cmp(), not used for quite a while Arnaldo Carvalho de Melo
2020-05-06 15:22 ` [PATCH 69/91] perf evsel: Rename *perf_evsel__read*() to *evsel__read() Arnaldo Carvalho de Melo
2020-05-06 15:22 ` [PATCH 70/91] perf evsel: Rename perf_evsel__parse_sample*() to evsel__parse_sample*() Arnaldo Carvalho de Melo
2020-05-06 15:22 ` [PATCH 71/91] perf evsel: Rename perf_evsel__{prev,next}() to evsel__{prev,next}() Arnaldo Carvalho de Melo
2020-05-06 15:22 ` [PATCH 72/91] perf evsel: Rename perf_evsel__has*() to evsel__has*() Arnaldo Carvalho de Melo
2020-05-06 15:22 ` [PATCH 73/91] perf evsel: Rename perf_evsel__fallback() to evsel__fallback() Arnaldo Carvalho de Melo
2020-05-06 15:22 ` [PATCH 74/91] perf evsel: Rename perf_evsel__group_idx() to evsel__group_idx() Arnaldo Carvalho de Melo
2020-05-06 15:22 ` [PATCH 75/91] perf evsel: Rename perf_evsel__env() to evsel__env() Arnaldo Carvalho de Melo
2020-05-06 15:22 ` [PATCH 76/91] perf evsel: Rename perf_evsel__store_ids() to evsel__store_id() Arnaldo Carvalho de Melo
2020-05-06 15:22 ` [PATCH 77/91] perf stat: Rename perf_evsel__*() operating on 'struct evsel *' to evsel__*() Arnaldo Carvalho de Melo
2020-05-06 15:22 ` [PATCH 78/91] perf kmem: " Arnaldo Carvalho de Melo
2020-05-06 15:22 ` [PATCH 79/91] perf lock: " Arnaldo Carvalho de Melo
2020-05-06 15:22 ` [PATCH 80/91] perf sched: " Arnaldo Carvalho de Melo
2020-05-06 15:22 ` [PATCH 81/91] perf script: " Arnaldo Carvalho de Melo
2020-05-06 15:22 ` [PATCH 82/91] perf trace: " Arnaldo Carvalho de Melo
2020-05-06 15:22 ` [PATCH 83/91] perf annotate: " Arnaldo Carvalho de Melo
2020-05-06 15:22 ` [PATCH 84/91] perf inject: " Arnaldo Carvalho de Melo
2020-05-06 15:22 ` [PATCH 85/91] perf symbol: Fix kernel symbol address display Arnaldo Carvalho de Melo
2020-05-06 15:22 ` [PATCH 86/91] perf: cs-etm: Update to build with latest opencsd version Arnaldo Carvalho de Melo
2020-05-06 15:22 ` [PATCH 87/91] perf bench: Add kallsyms parsing Arnaldo Carvalho de Melo
2020-05-06 15:22 ` [PATCH 88/91] libsymbols kallsyms: Parse using io api Arnaldo Carvalho de Melo
2020-05-06 15:22 ` [PATCH 89/91] libsymbols kallsyms: Move hex2u64 out of header Arnaldo Carvalho de Melo
2020-05-06 15:22 ` [PATCH 90/91] perf cs-etm: Move definition of 'traceid_list' global variable from header file Arnaldo Carvalho de Melo
2020-05-06 15:22 ` [PATCH 91/91] perf flamegraph: Use /bin/bash for report and record scripts Arnaldo Carvalho de Melo
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20200506152234.21977-9-acme@kernel.org \
--to=acme@kernel.org \
--cc=acme@redhat.com \
--cc=adrian.hunter@intel.com \
--cc=alexander.shishkin@linux.intel.com \
--cc=alexey.budankov@linux.intel.com \
--cc=eranian@google.com \
--cc=irogers@google.com \
--cc=jolsa@kernel.org \
--cc=jolsa@redhat.com \
--cc=kan.liang@linux.intel.com \
--cc=linux-kernel@vger.kernel.org \
--cc=linux-perf-users@vger.kernel.org \
--cc=mark.rutland@arm.com \
--cc=mingo@kernel.org \
--cc=namhyung@kernel.org \
--cc=peterz@infradead.org \
--cc=tglx@linutronix.de \
--cc=tonyj@suse.de \
--cc=williams@redhat.com \
--cc=yuzhoujian@didichuxing.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).