From: Song Liu <songliubraving@fb.com>
To: Arnaldo Carvalho de Melo <acme@kernel.org>
Cc: open list <linux-kernel@vger.kernel.org>,
Peter Zijlstra <peterz@infradead.org>,
Ingo Molnar <mingo@redhat.com>,
Alexander Shishkin <alexander.shishkin@linux.intel.com>,
Namhyung Kim <namhyung@kernel.org>,
Mark Rutland <mark.rutland@arm.com>, Jiri Olsa <jolsa@redhat.com>,
"Kernel Team" <Kernel-team@fb.com>
Subject: Re: [PATCH v7 3/3] perf-stat: enable counting events for BPF programs
Date: Tue, 19 Jan 2021 00:48:19 +0000 [thread overview]
Message-ID: <379919CC-594F-40C5-A10E-97E048F73AE2@fb.com> (raw)
In-Reply-To: <20210118193817.GG12699@kernel.org>
Hi Arnaldo,
> On Jan 18, 2021, at 11:38 AM, Arnaldo Carvalho de Melo <acme@kernel.org> wrote:
>
> Em Tue, Dec 29, 2020 at 01:42:14PM -0800, Song Liu escreveu:
>> Introduce perf-stat -b option, which counts events for BPF programs, like:
>>
>> [root@localhost ~]# ~/perf stat -e ref-cycles,cycles -b 254 -I 1000
>> 1.487903822 115,200 ref-cycles
>> 1.487903822 86,012 cycles
>> 2.489147029 80,560 ref-cycles
>> 2.489147029 73,784 cycles
>> 3.490341825 60,720 ref-cycles
>> 3.490341825 37,797 cycles
>> 4.491540887 37,120 ref-cycles
>> 4.491540887 31,963 cycles
>>
>> The example above counts cycles and ref-cycles of BPF program of id 254.
>> This is similar to bpftool-prog-profile command, but more flexible.
>
> So trying to get back to this after vacation:
>
> [root@five ~]# perf stat -vvv -e cycles -b 103
> Control descriptor is not initialized
> libbpf: elf: skipping unrecognized data section(9) .eh_frame
> libbpf: elf: skipping relo section(15) .rel.eh_frame for section(9) .eh_frame
> libbpf: load bpf program failed: Invalid argument
> libbpf: -- BEGIN DUMP LOG ---
> libbpf:
> Cannot recursively attach
> processed 0 insns (limit 1000000) max_states_per_insn 0 total_states 0 peak_states 0 mark_read 0
>
> libbpf: -- END LOG --
> libbpf: failed to load program 'fentry_XXX'
> libbpf: failed to load object 'bpf_prog_profiler_bpf'
> libbpf: failed to load BPF skeleton 'bpf_prog_profiler_bpf': -4007
> bpf_prog_profiler_bpf__load failed
> [root@five ~]#
>
> [root@five perf]# tools/bpf/runqslower/.output/
> bpftool/ libbpf/ runqslower
> [root@five perf]# tools/bpf/runqslower/.output/runqslower
> libbpf: elf: skipping unrecognized data section(10) .eh_frame
> libbpf: elf: skipping relo section(17) .rel.eh_frame for section(10) .eh_frame
> Tracing run queue latency higher than 10000 us
> TIME COMM PID LAT(us)
>
> [root@five ~]# bpftool prog
> 6: tracing name dump_bpf_map tag e54c922dfa54f65f gpl
> loaded_at 2021-01-15T11:14:38-0300 uid 0
> xlated 256B jited 154B memlock 4096B map_ids 3
> btf_id 6
> 8: tracing name dump_bpf_prog tag 4cb5558b902ee92e gpl
> loaded_at 2021-01-15T11:14:38-0300 uid 0
> xlated 520B jited 301B memlock 4096B map_ids 3
> btf_id 6
> 80: cgroup_device tag 8b9c33f36f812014 gpl
> loaded_at 2021-01-17T18:35:22-0300 uid 0
> xlated 744B jited 447B memlock 4096B
> 81: cgroup_skb tag 6deef7357e7b4530 gpl
> loaded_at 2021-01-17T18:35:22-0300 uid 0
> xlated 64B jited 54B memlock 4096B
> 82: cgroup_skb tag 6deef7357e7b4530 gpl
> loaded_at 2021-01-17T18:35:22-0300 uid 0
> xlated 64B jited 54B memlock 4096B
> 83: cgroup_device tag ee0e253c78993a24 gpl
> loaded_at 2021-01-17T18:35:22-0300 uid 0
> xlated 416B jited 255B memlock 4096B
> 84: cgroup_device tag e3dbd137be8d6168 gpl
> loaded_at 2021-01-17T18:35:22-0300 uid 0
> xlated 504B jited 309B memlock 4096B
> 85: cgroup_skb tag 6deef7357e7b4530 gpl
> loaded_at 2021-01-17T18:35:22-0300 uid 0
> xlated 64B jited 54B memlock 4096B
> 86: cgroup_skb tag 6deef7357e7b4530 gpl
> loaded_at 2021-01-17T18:35:22-0300 uid 0
> xlated 64B jited 54B memlock 4096B
> 87: cgroup_device tag c8b47a902f1cc68b gpl
> loaded_at 2021-01-17T18:35:22-0300 uid 0
> xlated 464B jited 288B memlock 4096B
> 88: cgroup_device tag 0ecd07b7b633809f gpl
> loaded_at 2021-01-17T18:35:22-0300 uid 0
> xlated 496B jited 307B memlock 4096B
> 89: cgroup_skb tag 6deef7357e7b4530 gpl
> loaded_at 2021-01-17T18:35:22-0300 uid 0
> xlated 64B jited 54B memlock 4096B
> 90: cgroup_skb tag 6deef7357e7b4530 gpl
> loaded_at 2021-01-17T18:35:22-0300 uid 0
> xlated 64B jited 54B memlock 4096B
> 91: cgroup_skb tag 6deef7357e7b4530 gpl
> loaded_at 2021-01-17T18:35:22-0300 uid 0
> xlated 64B jited 54B memlock 4096B
> 92: cgroup_skb tag 6deef7357e7b4530 gpl
> loaded_at 2021-01-17T18:35:22-0300 uid 0
> xlated 64B jited 54B memlock 4096B
> 100: tracing name handle__sched_w tag 00ba1f1d332f5a57 gpl
> loaded_at 2021-01-18T16:34:00-0300 uid 0
> xlated 152B jited 93B memlock 4096B map_ids 11,9
> btf_id 248
> 102: tracing name handle__sched_w tag 00ba1f1d332f5a57 gpl
> loaded_at 2021-01-18T16:34:00-0300 uid 0
> xlated 152B jited 93B memlock 4096B map_ids 11,9
> btf_id 248
> 103: tracing name handle__sched_s tag a4ac42153fd49be2 gpl
> loaded_at 2021-01-18T16:34:00-0300 uid 0
> xlated 560B jited 332B memlock 4096B map_ids 11,9,10
> btf_id 248
> [root@five ~]# perf stat -vvv -e cycles -b 92
> Control descriptor is not initialized
> libbpf: elf: skipping unrecognized data section(9) .eh_frame
> libbpf: elf: skipping relo section(15) .rel.eh_frame for section(9) .eh_frame
> prog FD 3 doesn't have valid btf
> Failed to get program name for bpf prog 92. Does it have BTF?
> [root@five ~]# perf stat -vvv -e cycles -b 103
> Control descriptor is not initialized
> libbpf: elf: skipping unrecognized data section(9) .eh_frame
> libbpf: elf: skipping relo section(15) .rel.eh_frame for section(9) .eh_frame
> libbpf: load bpf program failed: Invalid argument
> libbpf: -- BEGIN DUMP LOG ---
> libbpf:
> Cannot recursively attach
> processed 0 insns (limit 1000000) max_states_per_insn 0 total_states 0 peak_states 0 mark_read 0
>
> libbpf: -- END LOG --
> libbpf: failed to load program 'fentry_XXX'
> libbpf: failed to load object 'bpf_prog_profiler_bpf'
> libbpf: failed to load BPF skeleton 'bpf_prog_profiler_bpf': -4007
> bpf_prog_profiler_bpf__load failed
> [root@five ~]# perf stat -vvv -e cycles -b 102
> Control descriptor is not initialized
> libbpf: elf: skipping unrecognized data section(9) .eh_frame
> libbpf: elf: skipping relo section(15) .rel.eh_frame for section(9) .eh_frame
> libbpf: load bpf program failed: Invalid argument
> libbpf: -- BEGIN DUMP LOG ---
> libbpf:
> Cannot recursively attach
> processed 0 insns (limit 1000000) max_states_per_insn 0 total_states 0 peak_states 0 mark_read 0
>
> libbpf: -- END LOG --
> libbpf: failed to load program 'fentry_XXX'
> libbpf: failed to load object 'bpf_prog_profiler_bpf'
> libbpf: failed to load BPF skeleton 'bpf_prog_profiler_bpf': -4007
> bpf_prog_profiler_bpf__load failed
> [root@five ~]#
> [root@five ~]# perf stat -vvv -e cycles -b 6
> Control descriptor is not initialized
> libbpf: elf: skipping unrecognized data section(9) .eh_frame
> libbpf: elf: skipping relo section(15) .rel.eh_frame for section(9) .eh_frame
> libbpf: load bpf program failed: Invalid argument
> libbpf: -- BEGIN DUMP LOG ---
> libbpf:
> Cannot recursively attach
> processed 0 insns (limit 1000000) max_states_per_insn 0 total_states 0 peak_states 0 mark_read 0
>
> libbpf: -- END LOG --
> libbpf: failed to load program 'fentry_XXX'
> libbpf: failed to load object 'bpf_prog_profiler_bpf'
> libbpf: failed to load BPF skeleton 'bpf_prog_profiler_bpf': -4007
> bpf_prog_profiler_bpf__load failed
> [root@five ~]#
>
> So no more segfaults, but still something to look at, I'm pushing my
> local perf/core to tmp.perf/core, please take a look.
>
> - Arnaldo
We are looking at two issues:
1. Cannot recursively attach;
2. prog FD 3 doesn't have valid btf.
#1 was caused by the verifier disallowing attaching fentry/fexit program
to program with type BPF_PROG_TYPE_TRACING (in bpf_check_attach_target).
This constraint was added when we only had fentry/fexit in the TRACING
type. We have extended the TRACING type to many other use cases, like
"tp_btf/", "fmod_ret" and "iter/". Therefore, it is good time to revisit
this constraint. I will work on this.
For #2, we require the target program to have BTF. I guess we won't remove
this requirement.
While I work on improving #1, could you please test with some kprobe
programs? For example, we can use fileslower.py from bcc.
Thanks,
Song
[...]
next prev parent reply other threads:[~2021-01-19 0:49 UTC|newest]
Thread overview: 19+ messages / expand[flat|nested] mbox.gz Atom feed top
2020-12-29 21:42 [PATCH v7 0/3] Introduce perf-stat -b for BPF programs Song Liu
2020-12-29 21:42 ` [PATCH v7 1/3] bpftool: add Makefile target bootstrap Song Liu
2020-12-29 21:42 ` [PATCH v7 2/3] perf: support build BPF skeletons with perf Song Liu
2020-12-29 21:42 ` [PATCH v7 3/3] perf-stat: enable counting events for BPF programs Song Liu
2021-01-12 7:35 ` Namhyung Kim
2021-01-15 18:53 ` Arnaldo Carvalho de Melo
2021-01-18 19:38 ` Arnaldo Carvalho de Melo
2021-01-19 0:48 ` Song Liu [this message]
2021-01-19 14:31 ` Arnaldo Carvalho de Melo
2021-01-19 14:42 ` Arnaldo Carvalho de Melo
2021-01-19 16:31 ` Arnaldo Carvalho de Melo
2021-01-19 21:54 ` Song Liu
2021-01-19 22:30 ` Arnaldo Carvalho de Melo
2021-01-20 12:37 ` Arnaldo Carvalho de Melo
2021-01-20 13:01 ` Arnaldo Carvalho de Melo
2021-01-20 13:50 ` Arnaldo Carvalho de Melo
2021-01-20 16:30 ` FIX " Arnaldo Carvalho de Melo
2021-01-20 16:40 ` Song Liu
2021-01-20 17:04 ` Arnaldo Carvalho de Melo
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=379919CC-594F-40C5-A10E-97E048F73AE2@fb.com \
--to=songliubraving@fb.com \
--cc=Kernel-team@fb.com \
--cc=acme@kernel.org \
--cc=alexander.shishkin@linux.intel.com \
--cc=jolsa@redhat.com \
--cc=linux-kernel@vger.kernel.org \
--cc=mark.rutland@arm.com \
--cc=mingo@redhat.com \
--cc=namhyung@kernel.org \
--cc=peterz@infradead.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).