* [PATCH 0/4] perf stat: Fix bperf cgroup counters (v2)
@ 2022-09-16 18:41 Namhyung Kim
2022-09-16 18:41 ` [PATCH 1/4] perf stat: Fix BPF program section name Namhyung Kim
` (4 more replies)
0 siblings, 5 replies; 6+ messages in thread
From: Namhyung Kim @ 2022-09-16 18:41 UTC (permalink / raw)
To: Arnaldo Carvalho de Melo, Jiri Olsa
Cc: Ingo Molnar, Peter Zijlstra, LKML, Ian Rogers, Adrian Hunter,
linux-perf-users, Song Liu, bpf
Hello,
This fixes random failures in the perf stat cgroup BPF counters (bperf).
I've also added a new test to ensure it working properly.
v2 changes)
* fix a segfault with uncore event
You can get it from 'perf/bperf-cgrp-fix-v2' branch in
git://git.kernel.org/pub/scm/linux/kernel/git/namhyung/linux-perf.git
Thanks,
Namhyung
Namhyung Kim (4):
perf stat: Fix BPF program section name
perf stat: Fix cpu map index in bperf cgroup code
perf stat: Use evsel->core.cpus to iterate cpus in BPF cgroup counters
perf test: Add a new test for perf stat cgroup BPF counter
.../tests/shell/stat_bpf_counters_cgrp.sh | 83 +++++++++++++++++++
tools/perf/util/bpf_counter_cgroup.c | 10 +--
tools/perf/util/bpf_skel/bperf_cgroup.bpf.c | 2 +-
3 files changed, 89 insertions(+), 6 deletions(-)
create mode 100755 tools/perf/tests/shell/stat_bpf_counters_cgrp.sh
base-commit: 62e64c9d2fd12839c02f1b3e8b873e7cb34e8720
--
2.37.3.968.ga6b4b080e4-goog
^ permalink raw reply [flat|nested] 6+ messages in thread
* [PATCH 1/4] perf stat: Fix BPF program section name
2022-09-16 18:41 [PATCH 0/4] perf stat: Fix bperf cgroup counters (v2) Namhyung Kim
@ 2022-09-16 18:41 ` Namhyung Kim
2022-09-16 18:41 ` [PATCH 2/4] perf stat: Fix cpu map index in bperf cgroup code Namhyung Kim
` (3 subsequent siblings)
4 siblings, 0 replies; 6+ messages in thread
From: Namhyung Kim @ 2022-09-16 18:41 UTC (permalink / raw)
To: Arnaldo Carvalho de Melo, Jiri Olsa
Cc: Ingo Molnar, Peter Zijlstra, LKML, Ian Rogers, Adrian Hunter,
linux-perf-users, Song Liu, bpf
It seems the recent libbpf got more strict about the section name.
I'm seeing a failure like this:
$ sudo ./perf stat -a --bpf-counters --for-each-cgroup ^. sleep 1
libbpf: prog 'on_cgrp_switch': missing BPF prog type, check ELF section name 'perf_events'
libbpf: prog 'on_cgrp_switch': failed to load: -22
libbpf: failed to load object 'bperf_cgroup_bpf'
libbpf: failed to load BPF skeleton 'bperf_cgroup_bpf': -22
Failed to load cgroup skeleton
The section name should be 'perf_event' (without the trailing 's').
Although it's related to the libbpf change, it'd be better fix the
section name in the first place.
Fixes: 944138f048f7 ("perf stat: Enable BPF counter with --for-each-cgroup")
Signed-off-by: Namhyung Kim <namhyung@kernel.org>
---
tools/perf/util/bpf_skel/bperf_cgroup.bpf.c | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/tools/perf/util/bpf_skel/bperf_cgroup.bpf.c b/tools/perf/util/bpf_skel/bperf_cgroup.bpf.c
index 292c430768b5..c72f8ad96f75 100644
--- a/tools/perf/util/bpf_skel/bperf_cgroup.bpf.c
+++ b/tools/perf/util/bpf_skel/bperf_cgroup.bpf.c
@@ -176,7 +176,7 @@ static int bperf_cgroup_count(void)
}
// This will be attached to cgroup-switches event for each cpu
-SEC("perf_events")
+SEC("perf_event")
int BPF_PROG(on_cgrp_switch)
{
return bperf_cgroup_count();
--
2.37.3.968.ga6b4b080e4-goog
^ permalink raw reply related [flat|nested] 6+ messages in thread
* [PATCH 2/4] perf stat: Fix cpu map index in bperf cgroup code
2022-09-16 18:41 [PATCH 0/4] perf stat: Fix bperf cgroup counters (v2) Namhyung Kim
2022-09-16 18:41 ` [PATCH 1/4] perf stat: Fix BPF program section name Namhyung Kim
@ 2022-09-16 18:41 ` Namhyung Kim
2022-09-16 18:41 ` [PATCH 3/4] perf stat: Use evsel->core.cpus to iterate cpus in BPF cgroup counters Namhyung Kim
` (2 subsequent siblings)
4 siblings, 0 replies; 6+ messages in thread
From: Namhyung Kim @ 2022-09-16 18:41 UTC (permalink / raw)
To: Arnaldo Carvalho de Melo, Jiri Olsa
Cc: Ingo Molnar, Peter Zijlstra, LKML, Ian Rogers, Adrian Hunter,
linux-perf-users, Song Liu, bpf
The previous cpu map introduced a bug in the bperf cgroup counter. This
results in a failure when user gives a partial cpu map starting from
non-zero.
$ sudo ./perf stat -C 1-2 --bpf-counters --for-each-cgroup ^. sleep 1
libbpf: prog 'on_cgrp_switch': failed to create BPF link for perf_event FD 0:
-9 (Bad file descriptor)
Failed to attach cgroup program
To get the FD of an evsel, it should use a map index not the CPU number.
Fixes: 0255571a1605 ("perf cpumap: Switch to using perf_cpu_map API")
Signed-off-by: Namhyung Kim <namhyung@kernel.org>
---
tools/perf/util/bpf_counter_cgroup.c | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git a/tools/perf/util/bpf_counter_cgroup.c b/tools/perf/util/bpf_counter_cgroup.c
index 63b9db657442..97c69a249c6e 100644
--- a/tools/perf/util/bpf_counter_cgroup.c
+++ b/tools/perf/util/bpf_counter_cgroup.c
@@ -95,7 +95,7 @@ static int bperf_load_program(struct evlist *evlist)
perf_cpu_map__for_each_cpu(cpu, i, evlist->core.all_cpus) {
link = bpf_program__attach_perf_event(skel->progs.on_cgrp_switch,
- FD(cgrp_switch, cpu.cpu));
+ FD(cgrp_switch, i));
if (IS_ERR(link)) {
pr_err("Failed to attach cgroup program\n");
err = PTR_ERR(link);
@@ -123,7 +123,7 @@ static int bperf_load_program(struct evlist *evlist)
map_fd = bpf_map__fd(skel->maps.events);
perf_cpu_map__for_each_cpu(cpu, j, evlist->core.all_cpus) {
- int fd = FD(evsel, cpu.cpu);
+ int fd = FD(evsel, j);
__u32 idx = evsel->core.idx * total_cpus + cpu.cpu;
err = bpf_map_update_elem(map_fd, &idx, &fd,
--
2.37.3.968.ga6b4b080e4-goog
^ permalink raw reply related [flat|nested] 6+ messages in thread
* [PATCH 3/4] perf stat: Use evsel->core.cpus to iterate cpus in BPF cgroup counters
2022-09-16 18:41 [PATCH 0/4] perf stat: Fix bperf cgroup counters (v2) Namhyung Kim
2022-09-16 18:41 ` [PATCH 1/4] perf stat: Fix BPF program section name Namhyung Kim
2022-09-16 18:41 ` [PATCH 2/4] perf stat: Fix cpu map index in bperf cgroup code Namhyung Kim
@ 2022-09-16 18:41 ` Namhyung Kim
2022-09-16 18:41 ` [PATCH 4/4] perf test: Add a new test for perf stat cgroup BPF counter Namhyung Kim
2022-09-20 19:53 ` [PATCH 0/4] perf stat: Fix bperf cgroup counters (v2) Arnaldo Carvalho de Melo
4 siblings, 0 replies; 6+ messages in thread
From: Namhyung Kim @ 2022-09-16 18:41 UTC (permalink / raw)
To: Arnaldo Carvalho de Melo, Jiri Olsa
Cc: Ingo Molnar, Peter Zijlstra, LKML, Ian Rogers, Adrian Hunter,
linux-perf-users, Song Liu, bpf
If it mixes core and uncore events, each evsel would have different cpu map.
But it assumed they are same with evlist's all_cpus and accessed by the same
index. This resulted in a crash like below.
$ perf stat -a --bpf-counters --for-each_cgroup ^. -e cycles,imc/cas_count_read/ sleep 1
Segmentation fault
While it's not recommended to use uncore events for cgroup aggregation, it
should not crash.
Signed-off-by: Namhyung Kim <namhyung@kernel.org>
---
tools/perf/util/bpf_counter_cgroup.c | 6 +++---
1 file changed, 3 insertions(+), 3 deletions(-)
diff --git a/tools/perf/util/bpf_counter_cgroup.c b/tools/perf/util/bpf_counter_cgroup.c
index 97c69a249c6e..3c2df7522f6f 100644
--- a/tools/perf/util/bpf_counter_cgroup.c
+++ b/tools/perf/util/bpf_counter_cgroup.c
@@ -115,14 +115,14 @@ static int bperf_load_program(struct evlist *evlist)
evsel->cgrp = NULL;
/* open single copy of the events w/o cgroup */
- err = evsel__open_per_cpu(evsel, evlist->core.all_cpus, -1);
+ err = evsel__open_per_cpu(evsel, evsel->core.cpus, -1);
if (err) {
pr_err("Failed to open first cgroup events\n");
goto out;
}
map_fd = bpf_map__fd(skel->maps.events);
- perf_cpu_map__for_each_cpu(cpu, j, evlist->core.all_cpus) {
+ perf_cpu_map__for_each_cpu(cpu, j, evsel->core.cpus) {
int fd = FD(evsel, j);
__u32 idx = evsel->core.idx * total_cpus + cpu.cpu;
@@ -269,7 +269,7 @@ static int bperf_cgrp__read(struct evsel *evsel)
goto out;
}
- perf_cpu_map__for_each_cpu(cpu, i, evlist->core.all_cpus) {
+ perf_cpu_map__for_each_cpu(cpu, i, evsel->core.cpus) {
counts = perf_counts(evsel->counts, i, 0);
counts->val = values[cpu.cpu].counter;
counts->ena = values[cpu.cpu].enabled;
--
2.37.3.968.ga6b4b080e4-goog
^ permalink raw reply related [flat|nested] 6+ messages in thread
* [PATCH 4/4] perf test: Add a new test for perf stat cgroup BPF counter
2022-09-16 18:41 [PATCH 0/4] perf stat: Fix bperf cgroup counters (v2) Namhyung Kim
` (2 preceding siblings ...)
2022-09-16 18:41 ` [PATCH 3/4] perf stat: Use evsel->core.cpus to iterate cpus in BPF cgroup counters Namhyung Kim
@ 2022-09-16 18:41 ` Namhyung Kim
2022-09-20 19:53 ` [PATCH 0/4] perf stat: Fix bperf cgroup counters (v2) Arnaldo Carvalho de Melo
4 siblings, 0 replies; 6+ messages in thread
From: Namhyung Kim @ 2022-09-16 18:41 UTC (permalink / raw)
To: Arnaldo Carvalho de Melo, Jiri Olsa
Cc: Ingo Molnar, Peter Zijlstra, LKML, Ian Rogers, Adrian Hunter,
linux-perf-users, Song Liu, bpf
$ sudo ./perf test -v each-cgroup
96: perf stat --bpf-counters --for-each-cgroup test :
--- start ---
test child forked, pid 79600
test child finished with 0
---- end ----
perf stat --bpf-counters --for-each-cgroup test: Ok
Signed-off-by: Namhyung Kim <namhyung@kernel.org>
---
.../tests/shell/stat_bpf_counters_cgrp.sh | 83 +++++++++++++++++++
1 file changed, 83 insertions(+)
create mode 100755 tools/perf/tests/shell/stat_bpf_counters_cgrp.sh
diff --git a/tools/perf/tests/shell/stat_bpf_counters_cgrp.sh b/tools/perf/tests/shell/stat_bpf_counters_cgrp.sh
new file mode 100755
index 000000000000..d724855d097c
--- /dev/null
+++ b/tools/perf/tests/shell/stat_bpf_counters_cgrp.sh
@@ -0,0 +1,83 @@
+#!/bin/sh
+# perf stat --bpf-counters --for-each-cgroup test
+# SPDX-License-Identifier: GPL-2.0
+
+set -e
+
+test_cgroups=
+if [ "$1" = "-v" ]; then
+ verbose="1"
+fi
+
+# skip if --bpf-counters --for-each-cgroup is not supported
+check_bpf_counter()
+{
+ if ! perf stat -a --bpf-counters --for-each-cgroup / true > /dev/null 2>&1; then
+ if [ "${verbose}" = "1" ]; then
+ echo "Skipping: --bpf-counters --for-each-cgroup not supported"
+ perf --no-pager stat -a --bpf-counters --for-each-cgroup / true || true
+ fi
+ exit 2
+ fi
+}
+
+# find two cgroups to measure
+find_cgroups()
+{
+ # try usual systemd slices first
+ if [ -d /sys/fs/cgroup/system.slice -a -d /sys/fs/cgroup/user.slice ]; then
+ test_cgroups="system.slice,user.slice"
+ return
+ fi
+
+ # try root and self cgroups
+ local self_cgrp=$(grep perf_event /proc/self/cgroup | cut -d: -f3)
+ if [ -z ${self_cgrp} ]; then
+ # cgroup v2 doesn't specify perf_event
+ self_cgrp=$(grep ^0: /proc/self/cgroup | cut -d: -f3)
+ fi
+
+ if [ -z ${self_cgrp} ]; then
+ test_cgroups="/"
+ else
+ test_cgroups="/,${self_cgrp}"
+ fi
+}
+
+# As cgroup events are cpu-wide, we cannot simply compare the result.
+# Just check if it runs without failure and has non-zero results.
+check_system_wide_counted()
+{
+ local output
+
+ output=$(perf stat -a --bpf-counters --for-each-cgroup ${test_cgroups} -e cpu-clock -x, sleep 1 2>&1)
+ if echo ${output} | grep -q -F "<not "; then
+ echo "Some system-wide events are not counted"
+ if [ "${verbose}" = "1" ]; then
+ echo ${output}
+ fi
+ exit 1
+ fi
+}
+
+check_cpu_list_counted()
+{
+ local output
+
+ output=$(perf stat -C 1 --bpf-counters --for-each-cgroup ${test_cgroups} -e cpu-clock -x, taskset -c 1 sleep 1 2>&1)
+ if echo ${output} | grep -q -F "<not "; then
+ echo "Some CPU events are not counted"
+ if [ "${verbose}" = "1" ]; then
+ echo ${output}
+ fi
+ exit 1
+ fi
+}
+
+check_bpf_counter
+find_cgroups
+
+check_system_wide_counted
+check_cpu_list_counted
+
+exit 0
--
2.37.3.968.ga6b4b080e4-goog
^ permalink raw reply related [flat|nested] 6+ messages in thread
* Re: [PATCH 0/4] perf stat: Fix bperf cgroup counters (v2)
2022-09-16 18:41 [PATCH 0/4] perf stat: Fix bperf cgroup counters (v2) Namhyung Kim
` (3 preceding siblings ...)
2022-09-16 18:41 ` [PATCH 4/4] perf test: Add a new test for perf stat cgroup BPF counter Namhyung Kim
@ 2022-09-20 19:53 ` Arnaldo Carvalho de Melo
4 siblings, 0 replies; 6+ messages in thread
From: Arnaldo Carvalho de Melo @ 2022-09-20 19:53 UTC (permalink / raw)
To: Namhyung Kim
Cc: Jiri Olsa, Ingo Molnar, Peter Zijlstra, LKML, Ian Rogers,
Adrian Hunter, linux-perf-users, Song Liu, bpf
Em Fri, Sep 16, 2022 at 11:41:28AM -0700, Namhyung Kim escreveu:
> Hello,
>
> This fixes random failures in the perf stat cgroup BPF counters (bperf).
> I've also added a new test to ensure it working properly.
>
> v2 changes)
> * fix a segfault with uncore event
Thanks, applied.
- Arnaldo
> You can get it from 'perf/bperf-cgrp-fix-v2' branch in
>
> git://git.kernel.org/pub/scm/linux/kernel/git/namhyung/linux-perf.git
>
> Thanks,
> Namhyung
>
>
> Namhyung Kim (4):
> perf stat: Fix BPF program section name
> perf stat: Fix cpu map index in bperf cgroup code
> perf stat: Use evsel->core.cpus to iterate cpus in BPF cgroup counters
> perf test: Add a new test for perf stat cgroup BPF counter
>
> .../tests/shell/stat_bpf_counters_cgrp.sh | 83 +++++++++++++++++++
> tools/perf/util/bpf_counter_cgroup.c | 10 +--
> tools/perf/util/bpf_skel/bperf_cgroup.bpf.c | 2 +-
> 3 files changed, 89 insertions(+), 6 deletions(-)
> create mode 100755 tools/perf/tests/shell/stat_bpf_counters_cgrp.sh
>
>
> base-commit: 62e64c9d2fd12839c02f1b3e8b873e7cb34e8720
> --
> 2.37.3.968.ga6b4b080e4-goog
--
- Arnaldo
^ permalink raw reply [flat|nested] 6+ messages in thread
end of thread, other threads:[~2022-09-20 19:53 UTC | newest]
Thread overview: 6+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2022-09-16 18:41 [PATCH 0/4] perf stat: Fix bperf cgroup counters (v2) Namhyung Kim
2022-09-16 18:41 ` [PATCH 1/4] perf stat: Fix BPF program section name Namhyung Kim
2022-09-16 18:41 ` [PATCH 2/4] perf stat: Fix cpu map index in bperf cgroup code Namhyung Kim
2022-09-16 18:41 ` [PATCH 3/4] perf stat: Use evsel->core.cpus to iterate cpus in BPF cgroup counters Namhyung Kim
2022-09-16 18:41 ` [PATCH 4/4] perf test: Add a new test for perf stat cgroup BPF counter Namhyung Kim
2022-09-20 19:53 ` [PATCH 0/4] perf stat: Fix bperf cgroup counters (v2) Arnaldo Carvalho de Melo
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).