From: Arnaldo Carvalho de Melo <arnaldo.melo@gmail.com>
To: Jiri Olsa <jolsa@kernel.org>
Cc: Jesper Dangaard Brouer <brouer@redhat.com>,
Alexei Starovoitov <ast@kernel.org>,
Daniel Borkmann <daniel@iogearbox.net>,
lkml <linux-kernel@vger.kernel.org>,
Ingo Molnar <mingo@kernel.org>,
Namhyung Kim <namhyung@kernel.org>,
Alexander Shishkin <alexander.shishkin@linux.intel.com>,
Peter Zijlstra <a.p.zijlstra@chello.nl>,
Stanislav Fomichev <sdf@fomichev.me>,
Song Liu <songliubraving@fb.com>,
Adrian Hunter <adrian.hunter@intel.com>,
Andi Kleen <ak@linux.intel.com>
Subject: Re: [PATCHv3 00/12] perf tools: Display eBPF code in intel_pt trace
Date: Thu, 23 May 2019 21:27:21 -0300 [thread overview]
Message-ID: <20190524002721.GA17479@kernel.org> (raw)
In-Reply-To: <20190508132010.14512-1-jolsa@kernel.org>
Em Wed, May 08, 2019 at 03:19:58PM +0200, Jiri Olsa escreveu:
> hi,
> this patchset adds dso support to read and display
> bpf code in intel_pt trace output. I had to change
> some of the kernel maps processing code, so hopefully
> I did not break too many things ;-)
One of these patches need some uintptr_t (IIRC) somewhere:
CC /tmp/build/perf/util/color_config.o
util/dso.c: In function 'bpf_read':
util/dso.c:725:8: error: cast to pointer from integer of different size [-Werror=int-to-pointer-cast]
buf = (u8 *) node->info_linear->info.jited_prog_insns;
^
CC /tmp/build/perf/util/metricgroup.o
Several 32-bit cross builds failed:
51 30.64 ubuntu:16.04-x-arm : FAIL arm-linux-gnueabihf-gcc (Ubuntu/Linaro 5.4.0-6ubuntu1~16.04.9) 5.4.0 20160609
52 74.89 ubuntu:16.04-x-arm64 : Ok aarch64-linux-gnu-gcc (Ubuntu/Linaro 5.4.0-6ubuntu1~16.04.9) 5.4.0 20160609
53 32.23 ubuntu:16.04-x-powerpc : FAIL powerpc-linux-gnu-gcc (Ubuntu 5.4.0-6ubuntu1~16.04.9) 5.4.0 20160609
54 75.33 ubuntu:16.04-x-powerpc64 : Ok powerpc64-linux-gnu-gcc (Ubuntu/IBM 5.4.0-6ubuntu1~16.04.9) 5.4.0 20160609
55 75.96 ubuntu:16.04-x-powerpc64el : Ok powerpc64le-linux-gnu-gcc (Ubuntu/IBM 5.4.0-6ubuntu1~16.04.9) 5.4.0 20160609
56 71.70 ubuntu:16.04-x-s390 : Ok s390x-linux-gnu-gcc (Ubuntu 5.4.0-6ubuntu1~16.04.9) 5.4.0 20160609
57 280.83 ubuntu:17.10 : Ok gcc (Ubuntu 7.2.0-8ubuntu3.2) 7.2.0
58 274.85 ubuntu:18.04 : Ok gcc (Ubuntu 7.4.0-1ubuntu1~18.04) 7.4.0
59 33.88 ubuntu:18.04-x-arm : FAIL arm-linux-gnueabihf-gcc (Ubuntu/Linaro 7.4.0-1ubuntu1~18.04) 7.4.0
60 84.57 ubuntu:18.04-x-arm64 : Ok aarch64-linux-gnu-gcc (Ubuntu/Linaro 7.4.0-1ubuntu1~18.04) 7.4.0
61 27.01 ubuntu:18.04-x-m68k : FAIL m68k-linux-gnu-gcc (Ubuntu 7.4.0-1ubuntu1~18.04) 7.4.0
62 34.56 ubuntu:18.04-x-powerpc : FAIL powerpc-linux-gnu-gcc (Ubuntu 7.4.0-1ubuntu1~18.04) 7.4.0
63 87.19 ubuntu:18.04-x-powerpc64 : Ok powerpc64-linux-gnu-gcc (Ubuntu 7.4.0-1ubuntu1~18.04) 7.4.0
64 88.42 ubuntu:18.04-x-powerpc64el : Ok powerpc64le-linux-gnu-gcc (Ubuntu 7.4.0-1ubuntu1~18.04) 7.4.0
65 150.80 ubuntu:18.04-x-riscv64 : Ok riscv64-linux-gnu-gcc (Ubuntu 7.4.0-1ubuntu1~18.04) 7.4.0
66 75.12 ubuntu:18.04-x-s390 : Ok s390x-linux-gnu-gcc (Ubuntu 7.4.0-1ubuntu1~18.04) 7.4.0
67 31.68 ubuntu:18.04-x-sh4 : FAIL sh4-linux-gnu-gcc (Ubuntu 7.4.0-1ubuntu1~18.04) 7.4.0
68 73.71 ubuntu:18.04-x-sparc64 : Ok sparc64-linux-gnu-gcc (Ubuntu 7.4.0-1ubuntu1~18.04) 7.4.0
69 263.82 ubuntu:18.10 : Ok gcc (Ubuntu 8.2.0-7ubuntu1) 8.2.0
70 261.61 ubuntu:19.04 : Ok gcc (Ubuntu 8.3.0-6ubuntu1) 8.3.0
71 71.94 ubuntu:19.04-x-alpha : Ok alpha-linux-gnu-gcc (Ubuntu 8.3.0-6ubuntu1) 8.3.0
72 87.70 ubuntu:19.04-x-arm64 : Ok aarch64-linux-gnu-gcc (Ubuntu/Linaro 8.3.0-6ubuntu1) 8.3.0
73 28.21 ubuntu:19.04-x-hppa : FAIL hppa-linux-gnu-gcc (Ubuntu 8.3.0-6ubuntu1) 8.3.0
> It's now possible to see bpf code flow via:
>
> # perf-with-kcore record pt -e intel_pt//ku -- sleep 1
> # perf-with-kcore script pt --insn-trace --xed
> ...
> sleep 36660 [016] 1057036.806464404: ffffffff811dfba5 trace_call_bpf+0x65 ([kernel.kallsyms]) nopl %eax, (%rax,%rax,1)
> sleep 36660 [016] 1057036.806464404: ffffffff811dfbaa trace_call_bpf+0x6a ([kernel.kallsyms]) movq 0x30(%rbx), %rax
> sleep 36660 [016] 1057036.806464404: ffffffff811dfbae trace_call_bpf+0x6e ([kernel.kallsyms]) leaq 0x38(%rbx), %rsi
> sleep 36660 [016] 1057036.806464404: ffffffff811dfbb2 trace_call_bpf+0x72 ([kernel.kallsyms]) mov %r13, %rdi
> sleep 36660 [016] 1057036.806464404: ffffffff811dfbb5 trace_call_bpf+0x75 ([kernel.kallsyms]) callq 0xffffffff81c00c30
> sleep 36660 [016] 1057036.806464404: ffffffff81c00c30 __x86_indirect_thunk_rax+0x0 ([kernel.kallsyms]) callq 0xffffffff81c00c3c
> sleep 36660 [016] 1057036.806464404: ffffffff81c00c3c __x86_indirect_thunk_rax+0xc ([kernel.kallsyms]) movq %rax, (%rsp)
> sleep 36660 [016] 1057036.806464404: ffffffff81c00c40 __x86_indirect_thunk_rax+0x10 ([kernel.kallsyms]) retq
> sleep 36660 [016] 1057036.806464725: ffffffffa05e89ef bpf_prog_da4fe6b3d2c29b25_trace_return+0x0 (bpf_prog_da4fe6b3d2c29b25_trace_return) pushq %rbp
> sleep 36660 [016] 1057036.806464725: ffffffffa05e89f0 bpf_prog_da4fe6b3d2c29b25_trace_return+0x1 (bpf_prog_da4fe6b3d2c29b25_trace_return) mov %rsp, %rbp
> sleep 36660 [016] 1057036.806464725: ffffffffa05e89f3 bpf_prog_da4fe6b3d2c29b25_trace_return+0x4 (bpf_prog_da4fe6b3d2c29b25_trace_return) sub $0x158, %rsp
> sleep 36660 [016] 1057036.806464725: ffffffffa05e89fa bpf_prog_da4fe6b3d2c29b25_trace_return+0xb (bpf_prog_da4fe6b3d2c29b25_trace_return) sub $0x28, %rbp
> sleep 36660 [016] 1057036.806464725: ffffffffa05e89fe bpf_prog_da4fe6b3d2c29b25_trace_return+0xf (bpf_prog_da4fe6b3d2c29b25_trace_return) movq %rbx, (%rbp)
> sleep 36660 [016] 1057036.806464725: ffffffffa05e8a02 bpf_prog_da4fe6b3d2c29b25_trace_return+0x13 (bpf_prog_da4fe6b3d2c29b25_trace_return) movq %r13, 0x8(%rbp)
> sleep 36660 [016] 1057036.806464725: ffffffffa05e8a06 bpf_prog_da4fe6b3d2c29b25_trace_return+0x17 (bpf_prog_da4fe6b3d2c29b25_trace_return) movq %r14, 0x10(%rbp)
> sleep 36660 [016] 1057036.806464725: ffffffffa05e8a0a bpf_prog_da4fe6b3d2c29b25_trace_return+0x1b (bpf_prog_da4fe6b3d2c29b25_trace_return) movq %r15, 0x18(%rbp)
> sleep 36660 [016] 1057036.806464725: ffffffffa05e8a0e bpf_prog_da4fe6b3d2c29b25_trace_return+0x1f (bpf_prog_da4fe6b3d2c29b25_trace_return) xor %eax, %eax
> sleep 36660 [016] 1057036.806464725: ffffffffa05e8a10 bpf_prog_da4fe6b3d2c29b25_trace_return+0x21 (bpf_prog_da4fe6b3d2c29b25_trace_return) movq %rax, 0x20(%rbp)
> sleep 36660 [016] 1057036.806464725: ffffffffa05e8a14 bpf_prog_da4fe6b3d2c29b25_trace_return+0x25 (bpf_prog_da4fe6b3d2c29b25_trace_return) mov %rdi, %rbx
> sleep 36660 [016] 1057036.806464725: ffffffffa05e8a17 bpf_prog_da4fe6b3d2c29b25_trace_return+0x28 (bpf_prog_da4fe6b3d2c29b25_trace_return) callq 0xffffffff811fed50
> sleep 36660 [016] 1057036.806464725: ffffffff811fed50 bpf_get_current_pid_tgid+0x0 ([kernel.kallsyms]) nopl %eax, (%rax,%rax,1)
> sleep 36660 [016] 1057036.806464725: ffffffff811fed55 bpf_get_current_pid_tgid+0x5 ([kernel.kallsyms]) movq %gs:0x15c00, %rdx
> sleep 36660 [016] 1057036.806464725: ffffffff811fed5e bpf_get_current_pid_tgid+0xe ([kernel.kallsyms]) test %rdx, %rdx
> sleep 36660 [016] 1057036.806464725: ffffffff811fed61 bpf_get_current_pid_tgid+0x11 ([kernel.kallsyms]) jz 0xffffffff811fed79
> sleep 36660 [016] 1057036.806464725: ffffffff811fed63 bpf_get_current_pid_tgid+0x13 ([kernel.kallsyms]) movsxdl 0x494(%rdx), %rax
> sleep 36660 [016] 1057036.806464725: ffffffff811fed6a bpf_get_current_pid_tgid+0x1a ([kernel.kallsyms]) movsxdl 0x490(%rdx), %rdx
> sleep 36660 [016] 1057036.806464725: ffffffff811fed71 bpf_get_current_pid_tgid+0x21 ([kernel.kallsyms]) shl $0x20, %rax
> sleep 36660 [016] 1057036.806464725: ffffffff811fed75 bpf_get_current_pid_tgid+0x25 ([kernel.kallsyms]) or %rdx, %rax
> sleep 36660 [016] 1057036.806464725: ffffffff811fed78 bpf_get_current_pid_tgid+0x28 ([kernel.kallsyms]) retq
> sleep 36660 [016] 1057036.806464725: ffffffffa05e8a1c bpf_prog_da4fe6b3d2c29b25_trace_return+0x2d (bpf_prog_da4fe6b3d2c29b25_trace_return) movq %rax, -0x8(%rbp)
> sleep 36660 [016] 1057036.806464725: ffffffffa05e8a20 bpf_prog_da4fe6b3d2c29b25_trace_return+0x31 (bpf_prog_da4fe6b3d2c29b25_trace_return) xor %edi, %edi
> sleep 36660 [016] 1057036.806464725: ffffffffa05e8a22 bpf_prog_da4fe6b3d2c29b25_trace_return+0x33 (bpf_prog_da4fe6b3d2c29b25_trace_return) movq %rdi, -0x10(%rbp)
> sleep 36660 [016] 1057036.806464725: ffffffffa05e8a26 bpf_prog_da4fe6b3d2c29b25_trace_return+0x37 (bpf_prog_da4fe6b3d2c29b25_trace_return) movq %rdi, -0x18(%rbp)
>
> # perf-core/perf-with-kcore script pt --call-trace
> ...
> sleep 36660 [016] 1057036.806464404: ([kernel.kallsyms] ) kretprobe_perf_func
> sleep 36660 [016] 1057036.806464404: ([kernel.kallsyms] ) trace_call_bpf
> sleep 36660 [016] 1057036.806464404: ([kernel.kallsyms] ) __x86_indirect_thunk_rax
> sleep 36660 [016] 1057036.806464404: ([kernel.kallsyms] ) __x86_indirect_thunk_rax
> sleep 36660 [016] 1057036.806464725: (bpf_prog_da4fe6b3d2c29b25_trace_return ) bpf_get_current_pid_tgid
> sleep 36660 [016] 1057036.806464725: (bpf_prog_da4fe6b3d2c29b25_trace_return ) bpf_ktime_get_ns
> sleep 36660 [016] 1057036.806464725: ([kernel.kallsyms] ) __x86_indirect_thunk_rax
> sleep 36660 [016] 1057036.806464725: ([kernel.kallsyms] ) __x86_indirect_thunk_rax
> sleep 36660 [016] 1057036.806465045: (bpf_prog_da4fe6b3d2c29b25_trace_return ) __htab_map_lookup_elem
> sleep 36660 [016] 1057036.806465366: ([kernel.kallsyms] ) memcmp
> sleep 36660 [016] 1057036.806465687: (bpf_prog_da4fe6b3d2c29b25_trace_return ) bpf_probe_read
> sleep 36660 [016] 1057036.806465687: ([kernel.kallsyms] ) probe_kernel_read
> sleep 36660 [016] 1057036.806465687: ([kernel.kallsyms] ) __check_object_size
> sleep 36660 [016] 1057036.806465687: ([kernel.kallsyms] ) check_stack_object
> sleep 36660 [016] 1057036.806465687: ([kernel.kallsyms] ) copy_user_enhanced_fast_string
> sleep 36660 [016] 1057036.806465687: (bpf_prog_da4fe6b3d2c29b25_trace_return ) bpf_probe_read
> sleep 36660 [016] 1057036.806465687: ([kernel.kallsyms] ) probe_kernel_read
> sleep 36660 [016] 1057036.806465687: ([kernel.kallsyms] ) __check_object_size
> sleep 36660 [016] 1057036.806465687: ([kernel.kallsyms] ) check_stack_object
> sleep 36660 [016] 1057036.806465687: ([kernel.kallsyms] ) copy_user_enhanced_fast_string
> sleep 36660 [016] 1057036.806466008: (bpf_prog_da4fe6b3d2c29b25_trace_return ) bpf_get_current_uid_gid
> sleep 36660 [016] 1057036.806466008: ([kernel.kallsyms] ) from_kgid
> sleep 36660 [016] 1057036.806466008: ([kernel.kallsyms] ) from_kuid
> sleep 36660 [016] 1057036.806466008: (bpf_prog_da4fe6b3d2c29b25_trace_return ) bpf_perf_event_output
> sleep 36660 [016] 1057036.806466328: ([kernel.kallsyms] ) perf_event_output
> sleep 36660 [016] 1057036.806466328: ([kernel.kallsyms] ) perf_prepare_sample
> sleep 36660 [016] 1057036.806466328: ([kernel.kallsyms] ) perf_misc_flags
> sleep 36660 [016] 1057036.806466328: ([kernel.kallsyms] ) __x86_indirect_thunk_rax
> sleep 36660 [016] 1057036.806466328: ([kernel.kallsyms] ) __x86_indirect_thunk_rax
>
>
> v3 changes:
> - fix the padding code in patch 7 (Song Liu)
> - rebased on current Arnaldo's perf/core with merged perf/urgent
>
> v2 changes:
> - fix missing pthread_mutex_unlock [Stanislav Fomichev]
> - removed eBPF dso reusing [Adrian Hunter]
> - merge eBPF and kcore maps [Adrian Hunter]
> - few patches already applied
> - added some perf script helper options
> - patches are rebased on top of Arnaldo's perf/core with perf/urgent merged in
>
> It's also available in here:
> git://git.kernel.org/pub/scm/linux/kernel/git/jolsa/perf.git
> perf/fixes
>
> thanks,
> jirka
>
>
> Cc: Jesper Dangaard Brouer <brouer@redhat.com>
> Cc: Alexei Starovoitov <ast@kernel.org>
> Cc: Daniel Borkmann <daniel@iogearbox.net>
> ---
> Jiri Olsa (12):
> perf tools: Separate generic code in dso__data_file_size
> perf tools: Separate generic code in dso_cache__read
> perf tools: Simplify dso_cache__read function
> perf tools: Add bpf dso read and size hooks
> perf tools: Read also the end of the kernel
> perf tools: Keep zero in pgoff bpf map
> perf script: Pad dso name for --call-trace
> perf tools: Preserve eBPF maps when loading kcore
> perf tests: Add map_groups__merge_in test
> perf script: Add --show-bpf-events to show eBPF related events
> perf script: Remove superfluous bpf event titles
> perf script: Add --show-all-events option
>
> tools/include/linux/kernel.h | 1 +
> tools/lib/vsprintf.c | 19 +++++++++++++++++++
> tools/perf/Documentation/perf-script.txt | 6 ++++++
> tools/perf/builtin-script.c | 56 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> tools/perf/tests/Build | 1 +
> tools/perf/tests/builtin-test.c | 4 ++++
> tools/perf/tests/map_groups.c | 120 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> tools/perf/tests/tests.h | 1 +
> tools/perf/util/dso.c | 125 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++------------------------------------
> tools/perf/util/event.c | 4 ++--
> tools/perf/util/machine.c | 31 ++++++++++++++++++++-----------
> tools/perf/util/map.c | 6 ++++++
> tools/perf/util/map_groups.h | 2 ++
> tools/perf/util/symbol.c | 97 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++----
> tools/perf/util/symbol_conf.h | 1 +
> 15 files changed, 421 insertions(+), 53 deletions(-)
> create mode 100644 tools/perf/tests/map_groups.c
--
- Arnaldo
next prev parent reply other threads:[~2019-05-24 0:27 UTC|newest]
Thread overview: 50+ messages / expand[flat|nested] mbox.gz Atom feed top
2019-05-08 13:19 [PATCHv3 00/12] perf tools: Display eBPF code in intel_pt trace Jiri Olsa
2019-05-08 13:19 ` [PATCH 01/12] perf tools: Separate generic code in dso__data_file_size Jiri Olsa
2019-05-13 19:47 ` Arnaldo Carvalho de Melo
2019-05-13 20:00 ` Jiri Olsa
2019-05-23 3:10 ` Namhyung Kim
2019-05-23 7:49 ` Jiri Olsa
2019-05-30 8:08 ` [tip:perf/core] perf dso: Separate generic code in dso__data_file_size() tip-bot for Jiri Olsa
2019-05-08 13:20 ` [PATCH 02/12] perf tools: Separate generic code in dso_cache__read Jiri Olsa
2019-05-30 8:09 ` [tip:perf/core] perf dso: " tip-bot for Jiri Olsa
2019-05-08 13:20 ` [PATCH 03/12] perf tools: Simplify dso_cache__read function Jiri Olsa
2019-05-30 8:09 ` [tip:perf/core] perf dso: " tip-bot for Jiri Olsa
2019-05-08 13:20 ` [PATCH 04/12] perf tools: Add bpf dso read and size hooks Jiri Olsa
2019-05-30 8:10 ` [tip:perf/core] perf dso: Add BPF DSO " tip-bot for Jiri Olsa
2019-05-08 13:20 ` [PATCH 05/12] perf tools: Read also the end of the kernel Jiri Olsa
2019-05-24 18:15 ` Arnaldo Carvalho de Melo
2019-05-24 18:17 ` Arnaldo Carvalho de Melo
2019-05-24 18:46 ` Arnaldo Carvalho de Melo
2019-05-26 13:09 ` Jiri Olsa
2019-05-24 23:21 ` Jiri Olsa
2019-05-28 21:29 ` [tip:perf/urgent] perf machine: " tip-bot for Jiri Olsa
2019-05-08 13:20 ` [PATCH 06/12] perf tools: Keep zero in pgoff bpf map Jiri Olsa
2019-05-30 7:55 ` [tip:perf/core] perf machine: Keep zero in pgoff BPF map tip-bot for Jiri Olsa
2019-05-08 13:20 ` [PATCH 07/12] perf script: Pad dso name for --call-trace Jiri Olsa
2019-05-30 8:11 ` [tip:perf/core] perf script: Pad DSO " tip-bot for Jiri Olsa
2019-05-08 13:20 ` [PATCH 08/12] perf tools: Preserve eBPF maps when loading kcore Jiri Olsa
2019-05-22 16:06 ` Arnaldo Carvalho de Melo
2019-05-22 21:22 ` Jiri Olsa
2019-05-30 7:56 ` [tip:perf/core] " tip-bot for Jiri Olsa
2019-05-08 13:20 ` [PATCH 09/12] perf tests: Add map_groups__merge_in test Jiri Olsa
2019-05-30 8:11 ` [tip:perf/core] " tip-bot for Jiri Olsa
2019-05-08 13:20 ` [PATCH 10/12] perf script: Add --show-bpf-events to show eBPF related events Jiri Olsa
2019-05-30 8:12 ` [tip:perf/core] " tip-bot for Jiri Olsa
2019-05-08 13:20 ` [PATCH 11/12] perf script: Remove superfluous bpf event titles Jiri Olsa
2019-05-30 8:13 ` [tip:perf/core] perf script: Remove superfluous BPF " tip-bot for Jiri Olsa
2019-05-08 13:20 ` [PATCH 12/12] perf script: Add --show-all-events option Jiri Olsa
2019-05-22 16:18 ` Arnaldo Carvalho de Melo
2019-05-22 21:30 ` Jiri Olsa
2019-05-08 21:49 ` [PATCHv3 00/12] perf tools: Display eBPF code in intel_pt trace Song Liu
2019-05-08 23:41 ` Arnaldo Carvalho de Melo
2019-05-24 0:27 ` Arnaldo Carvalho de Melo [this message]
2019-05-24 10:06 ` Jiri Olsa
2019-05-30 10:54 ` Leo Yan
2019-05-30 12:07 ` Jiri Olsa
2019-05-30 12:57 ` Leo Yan
2019-05-30 13:27 ` Jiri Olsa
2019-05-30 13:36 ` Arnaldo Carvalho de Melo
2019-05-30 14:05 ` Leo Yan
2019-05-31 9:19 ` Jiri Olsa
2019-06-02 6:42 ` Leo Yan
2019-05-31 9:09 ` Leo Yan
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20190524002721.GA17479@kernel.org \
--to=arnaldo.melo@gmail.com \
--cc=a.p.zijlstra@chello.nl \
--cc=adrian.hunter@intel.com \
--cc=ak@linux.intel.com \
--cc=alexander.shishkin@linux.intel.com \
--cc=ast@kernel.org \
--cc=brouer@redhat.com \
--cc=daniel@iogearbox.net \
--cc=jolsa@kernel.org \
--cc=linux-kernel@vger.kernel.org \
--cc=mingo@kernel.org \
--cc=namhyung@kernel.org \
--cc=sdf@fomichev.me \
--cc=songliubraving@fb.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.