From: Arnaldo Carvalho de Melo <acme@kernel.org>
To: Ingo Molnar <mingo@kernel.org>, Thomas Gleixner <tglx@linutronix.de>
Cc: "Jiri Olsa" <jolsa@kernel.org>,
"Namhyung Kim" <namhyung@kernel.org>,
"Clark Williams" <williams@redhat.com>,
linux-kernel@vger.kernel.org, linux-perf-users@vger.kernel.org,
"Arnaldo Carvalho de Melo" <acme@redhat.com>,
"Adrian Hunter" <adrian.hunter@intel.com>,
"Brendan Gregg" <brendan.d.gregg@gmail.com>,
"Luis Cláudio Gonçalves" <lclaudio@redhat.com>,
"Marcelo Tosatti" <mtosatti@redhat.com>
Subject: [PATCH 35/69] perf trace: Introduce --filter for tracepoint events
Date: Fri, 11 Oct 2019 17:05:25 -0300 [thread overview]
Message-ID: <20191011200559.7156-36-acme@kernel.org> (raw)
In-Reply-To: <20191011200559.7156-1-acme@kernel.org>
From: Arnaldo Carvalho de Melo <acme@redhat.com>
Similar to what is in 'perf record', works just like there:
# perf trace -e msr:*
328.297 :0/0 msr:write_msr(msr: FS_BASE, val: 140240388381888)
328.302 :0/0 msr:write_msr(msr: FS_BASE, val: 140240388381888)
328.306 :0/0 msr:write_msr(msr: FS_BASE, val: 140240388381888)
328.317 :0/0 msr:write_msr(msr: FS_BASE, val: 140240388381888)
328.322 :0/0 msr:write_msr(msr: FS_BASE, val: 140240388381888)
328.327 :0/0 msr:write_msr(msr: FS_BASE, val: 140240388381888)
328.331 :0/0 msr:write_msr(msr: FS_BASE, val: 140240388381888)
328.336 :0/0 msr:write_msr(msr: FS_BASE, val: 140240388381888)
328.340 :0/0 ^Cmsr:write_msr(msr: FS_BASE, val: 140240388381888)
#
So, for a system wide trace session looking at the write_msr tracepoint
we see a flood of MSR_FS_BASE, we need to get the number for that:
# grep FS_BASE /tmp/build/perf/trace/beauty/generated/x86_arch_MSRs_array.c
[0xc0000100 - x86_64_specific_MSRs_offset] = "FS_BASE",
#
And then use it in a filter:
# perf trace -e msr:* --filter="msr!=0xc0000100"
<SNIP>
942.177 :0/0 msr:write_msr(msr: IA32_TSC_DEADLINE, val: 3056931068232)
942.199 :0/0 msr:write_msr(msr: IA32_TSC_DEADLINE, val: 3057135655252)
942.203 :0/0 msr:write_msr(msr: IA32_TSC_DEADLINE, val: 3056931068222)
942.231 :0/0 msr:write_msr(msr: IA32_TSC_DEADLINE, val: 3056998373022)
942.241 :0/0 msr:write_msr(msr: IA32_TSC_DEADLINE, val: 3056931068236)
<SNIP>
#
Ok, lets filter that too, too noisy:
# grep TSC_DEADLINE /tmp/build/perf/trace/beauty/generated/x86_arch_MSRs_array.c
[0x000006E0] = "IA32_TSC_DEADLINE",
#
# perf trace -e msr:* --filter="msr!=0xc0000100 && msr!=0x6e0" -a sleep 0.1
0.000 :0/0 msr:read_msr(msr: IA32_TSC_ADJUST)
0.066 CPU 0/KVM/4895 msr:write_msr(msr: IA32_SPEC_CTRL, val: 6)
0.070 CPU 0/KVM/4895 msr:write_msr(msr: 0x830, val: 34359740667)
0.099 CPU 0/KVM/4895 msr:read_msr(msr: IA32_SYSENTER_ESP, val: -2199021993472)
0.100 CPU 0/KVM/4895 msr:read_msr(msr: IA32_APICBASE, val: 4276096000)
0.101 CPU 0/KVM/4895 msr:read_msr(msr: IA32_DEBUGCTLMSR)
0.109 :0/0 msr:write_msr(msr: IA32_SPEC_CTRL)
1.000 :0/0 msr:write_msr(msr: 0x830, val: 17179871485)
18.893 :0/0 msr:write_msr(msr: 0x83f, val: 246)
28.810 :0/0 msr:write_msr(msr: 0x830, val: 68719479037)
40.117 CPU 0/KVM/4895 msr:write_msr(msr: IA32_SPEC_CTRL, val: 6)
40.127 CPU 0/KVM/4895 msr:read_msr(msr: IA32_DEBUGCTLMSR)
40.139 CPU 0/KVM/4895 msr:write_msr(msr: LSTAR, val: -2130661312)
40.141 CPU 0/KVM/4895 msr:write_msr(msr: SYSCALL_MASK, val: 14080)
40.142 CPU 0/KVM/4895 msr:write_msr(msr: TSC_AUX)
40.144 CPU 0/KVM/4895 msr:write_msr(msr: KERNEL_GS_BASE)
40.147 CPU 0/KVM/4895 msr:write_msr(msr: IA32_SPEC_CTRL)
40.148 CPU 0/KVM/4895 msr:write_msr(msr: IA32_FLUSH_CMD, val: 1)
40.151 CPU 0/KVM/4895 msr:write_msr(msr: IA32_SPEC_CTRL, val: 6)
^C
#
One can combine that with filtering pids as well:
# perf trace -e msr:* --filter="msr!=0xc0000100 && msr!=0x6e0" --filter-pids 4895 -a sleep 0.09
0.000 :0/0 msr:write_msr(msr: 0x830, val: 4294969597)
0.291 gnome-terminal/2790 msr:write_msr(msr: SYSCALL_MASK, val: 292608)
0.294 gnome-terminal/2790 msr:write_msr(msr: LSTAR, val: -1935671280)
0.295 gnome-terminal/2790 msr:write_msr(msr: TSC_AUX, val: 6)
10.940 gnome-terminal/2790 msr:write_msr(msr: 0x830, val: 4294969597)
15.943 gnome-shell/2096 msr:write_msr(msr: 0x830, val: 4294969597)
16.975 :0/0 msr:write_msr(msr: 0x830, val: 4294969597)
19.560 :0/0 msr:write_msr(msr: 0x83f, val: 246)
25.162 :0/0 msr:read_msr(msr: IA32_TSC_ADJUST)
25.807 JS Watchdog/3635 msr:write_msr(msr: IA32_SPEC_CTRL, val: 6)
25.820 :0/0 msr:write_msr(msr: IA32_SPEC_CTRL)
25.941 gnome-terminal/2790 msr:write_msr(msr: 0x830, val: 4294969597)
26.941 gnome-terminal/2790 msr:write_msr(msr: 0x830, val: 4294969597)
29.942 gnome-terminal/2790 msr:write_msr(msr: 0x830, val: 4294969597)
45.313 :0/0 msr:write_msr(msr: 0x83f, val: 246)
56.945 gnome-terminal/2790 msr:write_msr(msr: 0x830, val: 4294969597)
60.946 gnome-terminal/2790 msr:write_msr(msr: 0x830, val: 4294969597)
74.096 JS Watchdog/8971 msr:write_msr(msr: IA32_SPEC_CTRL, val: 6)
74.130 :0/0 msr:write_msr(msr: IA32_SPEC_CTRL)
79.673 :0/0 msr:write_msr(msr: 0x83f, val: 246)
79.947 gnome-terminal/2790 msr:write_msr(msr: 0x830, val: 17179871485)
#
Or for just a pid, with callchains:
# grep SYSCALL_MAS /tmp/build/perf/trace/beauty/generated/x86_arch_MSRs_array.c
[0xc0000084 - x86_64_specific_MSRs_offset] = "SYSCALL_MASK",
# perf trace -e msr:* --filter="msr==0xc0000084" --pid 2790 --call-graph=dwarf
0.000 gnome-terminal/2790 msr:write_msr(msr: SYSCALL_MASK, val: 292608)
do_trace_write_msr ([kernel.kallsyms])
do_trace_write_msr ([kernel.kallsyms])
kvm_on_user_return ([kvm])
fire_user_return_notifiers ([kernel.kallsyms])
exit_to_usermode_loop ([kernel.kallsyms])
do_syscall_64 ([kernel.kallsyms])
entry_SYSCALL_64 ([kernel.kallsyms])
__GI___poll (inlined)
9299.073 gnome-terminal/2790 msr:write_msr(msr: SYSCALL_MASK, val: 292608)
do_trace_write_msr ([kernel.kallsyms])
do_trace_write_msr ([kernel.kallsyms])
kvm_on_user_return ([kvm])
fire_user_return_notifiers ([kernel.kallsyms])
exit_to_usermode_loop ([kernel.kallsyms])
do_syscall_64 ([kernel.kallsyms])
entry_SYSCALL_64 ([kernel.kallsyms])
__GI___poll (inlined)
9348.374 gnome-terminal/2790 msr:write_msr(msr: SYSCALL_MASK, val: 292608)
do_trace_write_msr ([kernel.kallsyms])
do_trace_write_msr ([kernel.kallsyms])
kvm_on_user_return ([kvm])
fire_user_return_notifiers ([kernel.kallsyms])
exit_to_usermode_loop ([kernel.kallsyms])
do_syscall_64 ([kernel.kallsyms])
entry_SYSCALL_64 ([kernel.kallsyms])
__GI___poll (inlined)
<SNIP>
#
Ok, just another form of KVM to emit MSRs :-)
Next step: elliminate those greps by getting the filter expression,
looking for arg names, then for the arrays associated with it to do a
reverse lookup.
Also allow those filters to be associated with strace-like syscall
names.
After that: augment the 'val' arg for 'msr:write_msr' based on the first
arg, 'msr'.
Then, do that with eBPF too, not just with tracepoint filters.
Cc: Adrian Hunter <adrian.hunter@intel.com>
Cc: Brendan Gregg <brendan.d.gregg@gmail.com>
Cc: Jiri Olsa <jolsa@kernel.org>
Cc: Luis Cláudio Gonçalves <lclaudio@redhat.com>
Cc: Marcelo Tosatti <mtosatti@redhat.com>
Cc: Namhyung Kim <namhyung@kernel.org>
Link: https://lkml.kernel.org/n/tip-95bfe5d4tzy5f66bx49d05rj@git.kernel.org
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
---
tools/perf/Documentation/perf-trace.txt | 5 +++++
tools/perf/builtin-trace.c | 8 +++++---
2 files changed, 10 insertions(+), 3 deletions(-)
diff --git a/tools/perf/Documentation/perf-trace.txt b/tools/perf/Documentation/perf-trace.txt
index ba16cd5b680f..3bb89c2e9020 100644
--- a/tools/perf/Documentation/perf-trace.txt
+++ b/tools/perf/Documentation/perf-trace.txt
@@ -42,6 +42,11 @@ OPTIONS
Prefixing with ! shows all syscalls but the ones specified. You may
need to escape it.
+--filter=<filter>::
+ Event filter. This option should follow an event selector (-e) which
+ selects tracepoint event(s).
+
+
-D msecs::
--delay msecs::
After starting the program, wait msecs before measuring. This is useful to
diff --git a/tools/perf/builtin-trace.c b/tools/perf/builtin-trace.c
index e9f132aa5a09..2c1968061b4b 100644
--- a/tools/perf/builtin-trace.c
+++ b/tools/perf/builtin-trace.c
@@ -3362,7 +3362,7 @@ static int trace__set_filter_loop_pids(struct trace *trace)
thread = parent;
}
- err = perf_evlist__set_tp_filter_pids(trace->evlist, nr, pids);
+ err = perf_evlist__append_tp_filter_pids(trace->evlist, nr, pids);
if (!err && trace->filter_pids.map)
err = bpf_map__set_filter_pids(trace->filter_pids.map, nr, pids);
@@ -3379,8 +3379,8 @@ static int trace__set_filter_pids(struct trace *trace)
* we fork the workload in perf_evlist__prepare_workload.
*/
if (trace->filter_pids.nr > 0) {
- err = perf_evlist__set_tp_filter_pids(trace->evlist, trace->filter_pids.nr,
- trace->filter_pids.entries);
+ err = perf_evlist__append_tp_filter_pids(trace->evlist, trace->filter_pids.nr,
+ trace->filter_pids.entries);
if (!err && trace->filter_pids.map) {
err = bpf_map__set_filter_pids(trace->filter_pids.map, trace->filter_pids.nr,
trace->filter_pids.entries);
@@ -4294,6 +4294,8 @@ int cmd_trace(int argc, const char **argv)
OPT_CALLBACK('e', "event", &trace, "event",
"event/syscall selector. use 'perf list' to list available events",
trace__parse_events_option),
+ OPT_CALLBACK(0, "filter", &trace.evlist, "filter",
+ "event filter", parse_filter),
OPT_BOOLEAN(0, "comm", &trace.show_comm,
"show the thread COMM next to its id"),
OPT_BOOLEAN(0, "tool_stats", &trace.show_tool_stats, "show tool stats"),
--
2.21.0
next prev parent reply other threads:[~2019-10-11 20:09 UTC|newest]
Thread overview: 71+ messages / expand[flat|nested] mbox.gz Atom feed top
2019-10-11 20:04 [GIT PULL] perf/core improvements and fixes Arnaldo Carvalho de Melo
2019-10-11 20:04 ` [PATCH 01/69] perf env: Add routine to read the env->cpuid from the running machine Arnaldo Carvalho de Melo
2019-10-11 20:04 ` [PATCH 02/69] perf top: Initialize perf_env->cpuid, needed by the per arch annotation init routine Arnaldo Carvalho de Melo
2019-10-11 20:04 ` [PATCH 03/69] perf evlist: Adopt __set_tracepoint_handlers method from perf_session Arnaldo Carvalho de Melo
2019-10-11 20:04 ` [PATCH 04/69] perf trace: Make evlist__set_evsel_handler() affect just entries without a handler Arnaldo Carvalho de Melo
2019-10-11 20:04 ` [PATCH 05/69] perf trace: Separate 'struct syscall_fmt' definition from syscall_fmts variable Arnaldo Carvalho de Melo
2019-10-11 20:04 ` [PATCH 06/69] perf trace: Generalize the syscall_fmt find routines Arnaldo Carvalho de Melo
2019-10-11 20:04 ` [PATCH 07/69] perf trace: Postpone parsing .perfconfig trace.add_events to after --verbose is processed Arnaldo Carvalho de Melo
2019-10-11 20:04 ` [PATCH 08/69] perf trace augmented_syscalls: Do not show syscalls when none was asked for Arnaldo Carvalho de Melo
2019-10-11 20:04 ` [PATCH 09/69] perf scripts python: exported-sql-viewer.py: Add LookupModel() Arnaldo Carvalho de Melo
2019-10-11 20:05 ` [PATCH 10/69] perf scripts python: exported-sql-viewer.py: Add HBoxLayout and VBoxLayout Arnaldo Carvalho de Melo
2019-10-11 20:05 ` [PATCH 11/69] perf scripts python: exported-sql-viewer.py: Add global time range calculations Arnaldo Carvalho de Melo
2019-10-11 20:05 ` [PATCH 12/69] perf scripts python: exported-sql-viewer.py: Tidy up Call tree call_time Arnaldo Carvalho de Melo
2019-10-11 20:05 ` [PATCH 13/69] perf scripts python: exported-sql-viewer.py: Add ability for Call tree to open at a specified task and time Arnaldo Carvalho de Melo
2019-10-11 20:05 ` [PATCH 14/69] perf scripts python: exported-sql-viewer.py: Add Time chart by CPU Arnaldo Carvalho de Melo
2019-10-11 20:05 ` [PATCH 15/69] perf tools: Make usage of test_attr__* optional for perf-sys.h Arnaldo Carvalho de Melo
2019-10-11 20:05 ` [PATCH 16/69] samples/bpf: fix build by setting HAVE_ATTR_TEST to zero Arnaldo Carvalho de Melo
2019-10-11 20:05 ` [PATCH 17/69] perf script: Allow --time with --reltime Arnaldo Carvalho de Melo
2019-10-11 20:05 ` [PATCH 18/69] perf trace: Factor out the initialization of syscal_arg_fmt->scnprintf Arnaldo Carvalho de Melo
2019-10-11 20:05 ` [PATCH 19/69] perf trace: Allocate an array of beautifiers for tracepoint args Arnaldo Carvalho de Melo
2019-10-11 20:05 ` [PATCH 20/69] perf trace: Move some scnprintf methods from syscall to syscall_arg_fmt Arnaldo Carvalho de Melo
2019-10-11 20:05 ` [PATCH 21/69] perf trace: Add the syscall_arg_fmt pointer to syscall_arg Arnaldo Carvalho de Melo
2019-10-11 20:05 ` [PATCH 22/69] perf trace: Add array of chars scnprintf beautifier Arnaldo Carvalho de Melo
2019-10-11 20:05 ` [PATCH 23/69] perf trace: Enclose all events argument lists with () Arnaldo Carvalho de Melo
2019-10-11 20:05 ` [PATCH 24/69] perf trace: Allow choosing how to augment the tracepoint arguments Arnaldo Carvalho de Melo
2019-10-11 20:05 ` [PATCH 25/69] tools arch x86: Grab a copy of the file containing the MSR numbers Arnaldo Carvalho de Melo
2019-10-11 20:05 ` [PATCH 26/69] perf beauty: Make strarray's offset be u64 Arnaldo Carvalho de Melo
2019-10-11 20:05 ` [PATCH 27/69] perf trace beauty: Add a x86 MSR cmd id->str table generator Arnaldo Carvalho de Melo
2019-10-11 20:05 ` [PATCH 28/69] perf beauty: Hook up the x86 MSR " Arnaldo Carvalho de Melo
2019-10-11 20:05 ` [PATCH 29/69] perf trace: Allow associating scnprintf routines with well known arg names Arnaldo Carvalho de Melo
2019-10-11 20:05 ` [PATCH 30/69] perf trace beauty: Add the glue for the autogenerated MSR arrays Arnaldo Carvalho de Melo
2019-10-11 20:05 ` [PATCH 31/69] perf trace: Associate the "msr" tracepoint arg name with x86_MSR__scnprintf() Arnaldo Carvalho de Melo
2019-10-11 20:05 ` [PATCH 32/69] perf evlist: Factor out asprintf routine to build a tracepoint pid filter Arnaldo Carvalho de Melo
2019-10-11 20:05 ` [PATCH 33/69] perf evlist: Introduce append_tp_filter() method Arnaldo Carvalho de Melo
2019-10-11 20:05 ` [PATCH 34/69] perf evlist: Introduce append_tp_filter_pid() and append_tp_filter_pids() Arnaldo Carvalho de Melo
2019-10-11 20:05 ` Arnaldo Carvalho de Melo [this message]
2019-10-11 20:05 ` [PATCH 36/69] perf trace: Add a strtoul() method to 'struct syscall_arg_fmt' Arnaldo Carvalho de Melo
2019-10-11 20:05 ` [PATCH 37/69] perf trace: Introduce a strtoul() method for 'struct strarrays' Arnaldo Carvalho de Melo
2019-10-11 20:05 ` [PATCH 38/69] perf trace: Expand strings in filters to integers Arnaldo Carvalho de Melo
2019-10-11 20:05 ` [PATCH 39/69] perf beauty: Introduce strtoul() for x86 MSRs Arnaldo Carvalho de Melo
2019-10-11 20:05 ` [PATCH 40/69] perf tools: Avoid 'sample_reg_masks' being const + weak Arnaldo Carvalho de Melo
2019-10-11 20:05 ` [PATCH 41/69] MAINTAINERS: Add entry for perf tool arm64 pmu-events files Arnaldo Carvalho de Melo
2019-10-11 20:05 ` [PATCH 42/69] libperf: Add perf_mmap__init() function Arnaldo Carvalho de Melo
2019-10-11 20:05 ` [PATCH 43/69] libperf: Add 'struct perf_mmap_param' Arnaldo Carvalho de Melo
2019-10-11 20:05 ` [PATCH 44/69] libperf: Adopt perf_mmap__mmap_len() function from tools/perf Arnaldo Carvalho de Melo
2019-10-11 20:05 ` [PATCH 45/69] libperf: Adopt perf_mmap__mmap() " Arnaldo Carvalho de Melo
2019-10-11 20:05 ` [PATCH 46/69] libperf: Adopt perf_mmap__get() " Arnaldo Carvalho de Melo
2019-10-11 20:05 ` [PATCH 47/69] libperf: Adopt perf_mmap__unmap() " Arnaldo Carvalho de Melo
2019-10-11 20:05 ` [PATCH 48/69] libperf: Adopt perf_mmap__put() " Arnaldo Carvalho de Melo
2019-10-11 20:05 ` [PATCH 49/69] perf tools: Use perf_mmap way to detect aux mmap Arnaldo Carvalho de Melo
2019-10-11 20:05 ` [PATCH 50/69] libperf: Adopt perf_mmap__consume() function from tools/perf Arnaldo Carvalho de Melo
2019-10-11 20:05 ` [PATCH 51/69] libperf: Adopt perf_mmap__read_init() " Arnaldo Carvalho de Melo
2019-10-11 20:05 ` [PATCH 52/69] libperf: Adopt perf_mmap__read_done() " Arnaldo Carvalho de Melo
2019-10-11 20:05 ` [PATCH 53/69] libperf: Adopt perf_mmap__read_event() " Arnaldo Carvalho de Melo
2019-10-11 20:05 ` [PATCH 54/69] libperf: Adopt perf_evlist__mmap()/munmap() " Arnaldo Carvalho de Melo
2019-10-11 20:05 ` [PATCH 55/69] libperf: Introduce perf_evlist__mmap_ops() Arnaldo Carvalho de Melo
2019-10-11 20:05 ` [PATCH 56/69] libperf: Introduce perf_evlist_mmap_ops::idx callback Arnaldo Carvalho de Melo
2019-10-11 20:05 ` [PATCH 57/69] libperf: Add perf_evlist_mmap_ops::get callback Arnaldo Carvalho de Melo
2019-10-11 20:05 ` [PATCH 58/69] libperf: Introduce perf_evlist_mmap_ops::mmap callback Arnaldo Carvalho de Melo
2019-10-11 20:05 ` [PATCH 59/69] perf tools: Introduce perf_evlist__mmap_cb_idx() Arnaldo Carvalho de Melo
2019-10-11 20:05 ` [PATCH 60/69] perf evlist: Introduce perf_evlist__mmap_cb_get() Arnaldo Carvalho de Melo
2019-10-11 20:05 ` [PATCH 61/69] perf evlist: Introduce perf_evlist__mmap_cb_mmap() Arnaldo Carvalho de Melo
2019-10-11 20:05 ` [PATCH 62/69] perf evlist: Switch to libperf's mmap interface Arnaldo Carvalho de Melo
2019-10-11 20:05 ` [PATCH 63/69] libperf: Centralize map refcnt setting Arnaldo Carvalho de Melo
2019-10-11 20:05 ` [PATCH 64/69] libperf: Move the pollfd allocation from tools/perf to libperf Arnaldo Carvalho de Melo
2019-10-11 20:05 ` [PATCH 65/69] libperf: Introduce perf_evlist__exit() Arnaldo Carvalho de Melo
2019-10-11 20:05 ` [PATCH 66/69] libperf: Introduce perf_evlist__purge() Arnaldo Carvalho de Melo
2019-10-11 20:05 ` [PATCH 67/69] libperf: Adopt perf_evlist__filter_pollfd() from tools/perf Arnaldo Carvalho de Melo
2019-10-11 20:05 ` [PATCH 68/69] perf tools: Propagate CFLAGS to libperf Arnaldo Carvalho de Melo
2019-10-11 20:05 ` [PATCH 69/69] perf diff: Report noisy for cycles diff Arnaldo Carvalho de Melo
2019-10-15 5:25 ` [GIT PULL] perf/core improvements and fixes Ingo Molnar
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20191011200559.7156-36-acme@kernel.org \
--to=acme@kernel.org \
--cc=acme@redhat.com \
--cc=adrian.hunter@intel.com \
--cc=brendan.d.gregg@gmail.com \
--cc=jolsa@kernel.org \
--cc=lclaudio@redhat.com \
--cc=linux-kernel@vger.kernel.org \
--cc=linux-perf-users@vger.kernel.org \
--cc=mingo@kernel.org \
--cc=mtosatti@redhat.com \
--cc=namhyung@kernel.org \
--cc=tglx@linutronix.de \
--cc=williams@redhat.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).