[PATCH v2 0/3] perf-stat: share hardware PMCs with BPF

[PATCH v2 0/3] perf-stat: share hardware PMCs with BPF

[Song Liu]

perf uses performance monitoring counters (PMCs) to monitor system
performance. The PMCs are limited hardware resources. For example,
Intel CPUs have 3x fixed PMCs and 4x programmable PMCs per cpu.

Modern data center systems use these PMCs in many different ways:
system level monitoring, (maybe nested) container level monitoring, per
process monitoring, profiling (in sample mode), etc. In some cases,
there are more active perf_events than available hardware PMCs. To allow
all perf_events to have a chance to run, it is necessary to do expensive
time multiplexing of events.

On the other hand, many monitoring tools count the common metrics (cycles,
instructions). It is a waste to have multiple tools create multiple
perf_events of "cycles" and occupy multiple PMCs.

bperf tries to reduce such wastes by allowing multiple perf_events of
"cycles" or "instructions" (at different scopes) to share PMUs. Instead
of having each perf-stat session to read its own perf_events, bperf uses
BPF programs to read the perf_events and aggregate readings to BPF maps.
Then, the perf-stat session(s) reads the values from these BPF maps.

Changes v1 => v2:
  1. Add documentation.
  2. Add a shell test.
  3. Rename options, default path of the atto-map, and some variables.
  4. Add a separate patch that moves clock_gettime() in __run_perf_stat()
     to after enable_counters().
  5. Make perf_cpu_map for all cpus a global variable.
  6. Use sysfs__mountpoint() for default attr-map path.
  7. Use cpu__max_cpu() instead of libbpf_num_possible_cpus().
  8. Add flag "enabled" to the follower program. Then move follower attach
     to bperf__load() and simplify bperf__enable().

Song Liu (3):
  perf-stat: introduce bperf, share hardware PMCs with BPF
  perf-stat: measure t0 and ref_time after enable_counters()
  perf-test: add a test for perf-stat --bpf-counters option

 tools/perf/Documentation/perf-stat.txt        |  11 +
 tools/perf/Makefile.perf                      |   1 +
 tools/perf/builtin-stat.c                     |  20 +-
 tools/perf/tests/shell/stat_bpf_counters.sh   |  34 ++
 tools/perf/util/bpf_counter.c                 | 519 +++++++++++++++++-
 tools/perf/util/bpf_skel/bperf.h              |  14 +
 tools/perf/util/bpf_skel/bperf_follower.bpf.c |  69 +++
 tools/perf/util/bpf_skel/bperf_leader.bpf.c   |  46 ++
 tools/perf/util/bpf_skel/bperf_u.h            |  14 +
 tools/perf/util/evsel.h                       |  20 +-
 tools/perf/util/target.h                      |   4 +-
 11 files changed, 742 insertions(+), 10 deletions(-)
 create mode 100755 tools/perf/tests/shell/stat_bpf_counters.sh
 create mode 100644 tools/perf/util/bpf_skel/bperf.h
 create mode 100644 tools/perf/util/bpf_skel/bperf_follower.bpf.c
 create mode 100644 tools/perf/util/bpf_skel/bperf_leader.bpf.c
 create mode 100644 tools/perf/util/bpf_skel/bperf_u.h

--
2.30.2

[PATCH v2 1/3] perf-stat: introduce bperf, share hardware PMCs with BPF

[Song Liu]

perf uses performance monitoring counters (PMCs) to monitor system
performance. The PMCs are limited hardware resources. For example,
Intel CPUs have 3x fixed PMCs and 4x programmable PMCs per cpu.

Modern data center systems use these PMCs in many different ways:
system level monitoring, (maybe nested) container level monitoring, per
process monitoring, profiling (in sample mode), etc. In some cases,
there are more active perf_events than available hardware PMCs. To allow
all perf_events to have a chance to run, it is necessary to do expensive
time multiplexing of events.

On the other hand, many monitoring tools count the common metrics (cycles,
instructions). It is a waste to have multiple tools create multiple
perf_events of "cycles" and occupy multiple PMCs.

bperf tries to reduce such wastes by allowing multiple perf_events of
"cycles" or "instructions" (at different scopes) to share PMUs. Instead
of having each perf-stat session to read its own perf_events, bperf uses
BPF programs to read the perf_events and aggregate readings to BPF maps.
Then, the perf-stat session(s) reads the values from these BPF maps.

Please refer to the comment before the definition of bperf_ops for the
description of bperf architecture.

bperf is off by default. To enable it, pass --bpf-counters option to
perf-stat. bperf uses a BPF hashmap to share information about BPF
programs and maps used by bperf. This map is pinned to bpffs. The default
path is /sys/fs/bpf/perf_attr_map. The user could change the path with
option --bpf-attr-map.

Signed-off-by: Song Liu <songliubraving@fb.com>

---
Known limitations:
1. Do not support per cgroup events;
2. Do not support monitoring of BPF program (perf-stat -b);
3. Do not support event groups;
4. Do not support inherit events during fork().

The following commands have been tested:

   perf stat --bpf-counters -e cycles,ref-cycles -a
   perf stat --bpf-counters -e cycles,instructions -C 1,3,4
   perf stat --bpf-counters -e cycles -p 123
   perf stat --bpf-counters -e cycles -t 100,101
   perf stat --bpf-counters -e cycles,ref-cycles -- stressapptest ...
---
 tools/perf/Documentation/perf-stat.txt        |  11 +
 tools/perf/Makefile.perf                      |   1 +
 tools/perf/builtin-stat.c                     |  10 +
 tools/perf/util/bpf_counter.c                 | 519 +++++++++++++++++-
 tools/perf/util/bpf_skel/bperf.h              |  14 +
 tools/perf/util/bpf_skel/bperf_follower.bpf.c |  69 +++
 tools/perf/util/bpf_skel/bperf_leader.bpf.c   |  46 ++
 tools/perf/util/bpf_skel/bperf_u.h            |  14 +
 tools/perf/util/evsel.h                       |  20 +-
 tools/perf/util/target.h                      |   4 +-
 10 files changed, 701 insertions(+), 7 deletions(-)
 create mode 100644 tools/perf/util/bpf_skel/bperf.h
 create mode 100644 tools/perf/util/bpf_skel/bperf_follower.bpf.c
 create mode 100644 tools/perf/util/bpf_skel/bperf_leader.bpf.c
 create mode 100644 tools/perf/util/bpf_skel/bperf_u.h

diff --git a/tools/perf/Documentation/perf-stat.txt b/tools/perf/Documentation/perf-stat.txt
index 08a1714494f87..d2e7656b5ef81 100644
--- a/tools/perf/Documentation/perf-stat.txt
+++ b/tools/perf/Documentation/perf-stat.txt
@@ -93,6 +93,17 @@ report::
 
         1.102235068 seconds time elapsed
 
+--bpf-counters::
+	Use BPF programs to aggregate readings from perf_events.  This
+	allows multiple perf-stat sessions that are counting the same metric (cycles,
+	instructions, etc.) to share hardware counters.
+
+--bpf-attr-map::
+	With option "--bpf-counters", different perf-stat sessions share
+	information about shared BPF programs and maps via a pinned hashmap.
+	Use "--bpf-attr-map" to specify the path of this pinned hashmap.
+	The default path is /sys/fs/bpf/perf_attr_map.
+
 ifdef::HAVE_LIBPFM[]
 --pfm-events events::
 Select a PMU event using libpfm4 syntax (see http://perfmon2.sf.net)
diff --git a/tools/perf/Makefile.perf b/tools/perf/Makefile.perf
index f6e609673de2b..ca9aa08e85a1f 100644
--- a/tools/perf/Makefile.perf
+++ b/tools/perf/Makefile.perf
@@ -1007,6 +1007,7 @@ python-clean:
 SKEL_OUT := $(abspath $(OUTPUT)util/bpf_skel)
 SKEL_TMP_OUT := $(abspath $(SKEL_OUT)/.tmp)
 SKELETONS := $(SKEL_OUT)/bpf_prog_profiler.skel.h
+SKELETONS += $(SKEL_OUT)/bperf_leader.skel.h $(SKEL_OUT)/bperf_follower.skel.h
 
 ifdef BUILD_BPF_SKEL
 BPFTOOL := $(SKEL_TMP_OUT)/bootstrap/bpftool
diff --git a/tools/perf/builtin-stat.c b/tools/perf/builtin-stat.c
index 2e2e4a8345ea2..92696373da994 100644
--- a/tools/perf/builtin-stat.c
+++ b/tools/perf/builtin-stat.c
@@ -792,6 +792,12 @@ static int __run_perf_stat(int argc, const char **argv, int run_idx)
 	}
 
 	evlist__for_each_cpu (evsel_list, i, cpu) {
+		/*
+		 * bperf calls evsel__open_per_cpu() in bperf__load(), so
+		 * no need to call it again here.
+		 */
+		if (target.use_bpf)
+			break;
 		affinity__set(&affinity, cpu);
 
 		evlist__for_each_entry(evsel_list, counter) {
@@ -1146,6 +1152,10 @@ static struct option stat_options[] = {
 #ifdef HAVE_BPF_SKEL
 	OPT_STRING('b', "bpf-prog", &target.bpf_str, "bpf-prog-id",
 		   "stat events on existing bpf program id"),
+	OPT_BOOLEAN(0, "bpf-counters", &target.use_bpf,
+		    "use bpf program to count events"),
+	OPT_STRING(0, "bpf-attr-map", &target.attr_map, "attr-map-path",
+		   "path to perf_event_attr map"),
 #endif
 	OPT_BOOLEAN('a', "all-cpus", &target.system_wide,
 		    "system-wide collection from all CPUs"),
diff --git a/tools/perf/util/bpf_counter.c b/tools/perf/util/bpf_counter.c
index 04f89120b3232..81d1df3c4ec0e 100644
--- a/tools/perf/util/bpf_counter.c
+++ b/tools/perf/util/bpf_counter.c
@@ -5,6 +5,7 @@
 #include <assert.h>
 #include <limits.h>
 #include <unistd.h>
+#include <sys/file.h>
 #include <sys/time.h>
 #include <sys/resource.h>
 #include <linux/err.h>
@@ -12,14 +13,45 @@
 #include <bpf/bpf.h>
 #include <bpf/btf.h>
 #include <bpf/libbpf.h>
+#include <api/fs/fs.h>
 
 #include "bpf_counter.h"
 #include "counts.h"
 #include "debug.h"
 #include "evsel.h"
+#include "evlist.h"
 #include "target.h"
+#include "cpumap.h"
+#include "thread_map.h"
 
 #include "bpf_skel/bpf_prog_profiler.skel.h"
+#include "bpf_skel/bperf_u.h"
+#include "bpf_skel/bperf_leader.skel.h"
+#include "bpf_skel/bperf_follower.skel.h"
+
+/*
+ * bperf uses a hashmap, the attr_map, to track all the leader programs.
+ * The hashmap is pinned in bpffs. flock() on this file is used to ensure
+ * no concurrent access to the attr_map.  The key of attr_map is struct
+ * perf_event_attr, and the value is struct perf_event_attr_map_entry.
+ *
+ * struct perf_event_attr_map_entry contains two __u32 IDs, bpf_link of the
+ * leader prog, and the diff_map. Each perf-stat session holds a reference
+ * to the bpf_link to make sure the leader prog is attached to sched_switch
+ * tracepoint.
+ *
+ * Since the hashmap only contains IDs of the bpf_link and diff_map, it
+ * does not hold any references to the leader program. Once all perf-stat
+ * sessions of these events exit, the leader prog, its maps, and the
+ * perf_events will be freed.
+ */
+struct perf_event_attr_map_entry {
+	__u32 link_id;
+	__u32 diff_map_id;
+};
+
+#define DEFAULT_ATTR_MAP_PATH "fs/bpf/perf_attr_map"
+#define ATTR_MAP_SIZE 16
 
 static inline void *u64_to_ptr(__u64 ptr)
 {
@@ -274,17 +306,494 @@ struct bpf_counter_ops bpf_program_profiler_ops = {
 	.install_pe = bpf_program_profiler__install_pe,
 };
 
+static __u32 bpf_link_get_id(int fd)
+{
+	struct bpf_link_info link_info = {0};
+	__u32 link_info_len = sizeof(link_info);
+
+	bpf_obj_get_info_by_fd(fd, &link_info, &link_info_len);
+	return link_info.id;
+}
+
+static __u32 bpf_link_get_prog_id(int fd)
+{
+	struct bpf_link_info link_info = {0};
+	__u32 link_info_len = sizeof(link_info);
+
+	bpf_obj_get_info_by_fd(fd, &link_info, &link_info_len);
+	return link_info.prog_id;
+}
+
+static __u32 bpf_map_get_id(int fd)
+{
+	struct bpf_map_info map_info = {0};
+	__u32 map_info_len = sizeof(map_info);
+
+	bpf_obj_get_info_by_fd(fd, &map_info, &map_info_len);
+	return map_info.id;
+}
+
+static int bperf_lock_attr_map(struct target *target)
+{
+	char path[PATH_MAX];
+	int map_fd, err;
+
+	if (target->attr_map) {
+		scnprintf(path, PATH_MAX, "%s", target->attr_map);
+	} else {
+		scnprintf(path, PATH_MAX, "%s/%s", sysfs__mountpoint(),
+			  DEFAULT_ATTR_MAP_PATH);
+	}
+
+	if (access(path, F_OK)) {
+		map_fd = bpf_create_map(BPF_MAP_TYPE_HASH,
+					sizeof(struct perf_event_attr),
+					sizeof(struct perf_event_attr_map_entry),
+					ATTR_MAP_SIZE, 0);
+		if (map_fd < 0)
+			return -1;
+
+		err = bpf_obj_pin(map_fd, path);
+		if (err) {
+			/* someone pinned the map in parallel? */
+			close(map_fd);
+			map_fd = bpf_obj_get(path);
+			if (map_fd < 0)
+				return -1;
+		}
+	} else {
+		map_fd = bpf_obj_get(path);
+		if (map_fd < 0)
+			return -1;
+	}
+
+	err = flock(map_fd, LOCK_EX);
+	if (err) {
+		close(map_fd);
+		return -1;
+	}
+	return map_fd;
+}
+
+/* trigger the leader program on a cpu */
+static int bperf_trigger_reading(int prog_fd, int cpu)
+{
+	DECLARE_LIBBPF_OPTS(bpf_test_run_opts, opts,
+			    .ctx_in = NULL,
+			    .ctx_size_in = 0,
+			    .flags = BPF_F_TEST_RUN_ON_CPU,
+			    .cpu = cpu,
+			    .retval = 0,
+		);
+
+	return bpf_prog_test_run_opts(prog_fd, &opts);
+}
+
+static int bperf_check_target(struct evsel *evsel,
+			      struct target *target,
+			      enum bperf_filter_type *filter_type,
+			      __u32 *filter_entry_cnt)
+{
+	if (evsel->leader->core.nr_members > 1) {
+		pr_err("bpf managed perf events do not yet support groups.\n");
+		return -1;
+	}
+
+	/* determine filter type based on target */
+	if (target->system_wide) {
+		*filter_type = BPERF_FILTER_GLOBAL;
+		*filter_entry_cnt = 1;
+	} else if (target->cpu_list) {
+		*filter_type = BPERF_FILTER_CPU;
+		*filter_entry_cnt = perf_cpu_map__nr(evsel__cpus(evsel));
+	} else if (target->tid) {
+		*filter_type = BPERF_FILTER_PID;
+		*filter_entry_cnt = perf_thread_map__nr(evsel->core.threads);
+	} else if (target->pid || evsel->evlist->workload.pid != -1) {
+		*filter_type = BPERF_FILTER_TGID;
+		*filter_entry_cnt = perf_thread_map__nr(evsel->core.threads);
+	} else {
+		pr_err("bpf managed perf events do not yet support these targets.\n");
+		return -1;
+	}
+
+	return 0;
+}
+
+static	struct perf_cpu_map *all_cpu_map;
+
+static int bperf_reload_leader_program(struct evsel *evsel, int attr_map_fd,
+				       struct perf_event_attr_map_entry *entry)
+{
+	struct bperf_leader_bpf *skel = bperf_leader_bpf__open();
+	int link_fd, diff_map_fd, err;
+	struct bpf_link *link = NULL;
+
+	if (!skel) {
+		pr_err("Failed to open leader skeleton\n");
+		return -1;
+	}
+
+	bpf_map__resize(skel->maps.events, libbpf_num_possible_cpus());
+	err = bperf_leader_bpf__load(skel);
+	if (err) {
+		pr_err("Failed to load leader skeleton\n");
+		goto out;
+	}
+
+	err = -1;
+	link = bpf_program__attach(skel->progs.on_switch);
+	if (!link) {
+		pr_err("Failed to attach leader program\n");
+		goto out;
+	}
+
+	link_fd = bpf_link__fd(link);
+	diff_map_fd = bpf_map__fd(skel->maps.diff_readings);
+	entry->link_id = bpf_link_get_id(link_fd);
+	entry->diff_map_id = bpf_map_get_id(diff_map_fd);
+	err = bpf_map_update_elem(attr_map_fd, &evsel->core.attr, entry, BPF_ANY);
+	assert(err == 0);
+
+	evsel->bperf_leader_link_fd = bpf_link_get_fd_by_id(entry->link_id);
+	assert(evsel->bperf_leader_link_fd >= 0);
+
+	/*
+	 * save leader_skel for install_pe, which is called within
+	 * following evsel__open_per_cpu call
+	 */
+	evsel->leader_skel = skel;
+	evsel__open_per_cpu(evsel, all_cpu_map, -1);
+
+out:
+	bperf_leader_bpf__destroy(skel);
+	bpf_link__destroy(link);
+	return err;
+}
+
+static int bperf__load(struct evsel *evsel, struct target *target)
+{
+	struct perf_event_attr_map_entry entry = {0xffffffff, 0xffffffff};
+	int attr_map_fd, diff_map_fd = -1, err;
+	enum bperf_filter_type filter_type;
+	__u32 filter_entry_cnt, i;
+
+	if (bperf_check_target(evsel, target, &filter_type, &filter_entry_cnt))
+		return -1;
+
+	if (!all_cpu_map) {
+		all_cpu_map = perf_cpu_map__new(NULL);
+		if (!all_cpu_map)
+			return -1;
+	}
+
+	evsel->bperf_leader_prog_fd = -1;
+	evsel->bperf_leader_link_fd = -1;
+
+	/*
+	 * Step 1: hold a fd on the leader program and the bpf_link, if
+	 * the program is not already gone, reload the program.
+	 * Use flock() to ensure exclusive access to the perf_event_attr
+	 * map.
+	 */
+	attr_map_fd = bperf_lock_attr_map(target);
+	if (attr_map_fd < 0) {
+		pr_err("Failed to lock perf_event_attr map\n");
+		return -1;
+	}
+
+	err = bpf_map_lookup_elem(attr_map_fd, &evsel->core.attr, &entry);
+	if (err) {
+		err = bpf_map_update_elem(attr_map_fd, &evsel->core.attr, &entry, BPF_ANY);
+		if (err)
+			goto out;
+	}
+
+	evsel->bperf_leader_link_fd = bpf_link_get_fd_by_id(entry.link_id);
+	if (evsel->bperf_leader_link_fd < 0 &&
+	    bperf_reload_leader_program(evsel, attr_map_fd, &entry))
+		goto out;
+
+	/*
+	 * The bpf_link holds reference to the leader program, and the
+	 * leader program holds reference to the maps. Therefore, if
+	 * link_id is valid, diff_map_id should also be valid.
+	 */
+	evsel->bperf_leader_prog_fd = bpf_prog_get_fd_by_id(
+		bpf_link_get_prog_id(evsel->bperf_leader_link_fd));
+	assert(evsel->bperf_leader_prog_fd >= 0);
+
+	diff_map_fd = bpf_map_get_fd_by_id(entry.diff_map_id);
+	assert(diff_map_fd >= 0);
+
+	/*
+	 * bperf uses BPF_PROG_TEST_RUN to get accurate reading. Check
+	 * whether the kernel support it
+	 */
+	err = bperf_trigger_reading(evsel->bperf_leader_prog_fd, 0);
+	if (err) {
+		pr_err("The kernel does not support test_run for raw_tp BPF programs.\n"
+		       "Therefore, --use-bpf might show inaccurate readings\n");
+		goto out;
+	}
+
+	/* Step 2: load the follower skeleton */
+	evsel->follower_skel = bperf_follower_bpf__open();
+	if (!evsel->follower_skel) {
+		pr_err("Failed to open follower skeleton\n");
+		goto out;
+	}
+
+	/* attach fexit program to the leader program */
+	bpf_program__set_attach_target(evsel->follower_skel->progs.fexit_XXX,
+				       evsel->bperf_leader_prog_fd, "on_switch");
+
+	/* connect to leader diff_reading map */
+	bpf_map__reuse_fd(evsel->follower_skel->maps.diff_readings, diff_map_fd);
+
+	/* set up reading map */
+	bpf_map__set_max_entries(evsel->follower_skel->maps.accum_readings,
+				 filter_entry_cnt);
+	/* set up follower filter based on target */
+	bpf_map__set_max_entries(evsel->follower_skel->maps.filter,
+				 filter_entry_cnt);
+	err = bperf_follower_bpf__load(evsel->follower_skel);
+	if (err) {
+		pr_err("Failed to load follower skeleton\n");
+		bperf_follower_bpf__destroy(evsel->follower_skel);
+		evsel->follower_skel = NULL;
+		goto out;
+	}
+
+	for (i = 0; i < filter_entry_cnt; i++) {
+		int filter_map_fd;
+		__u32 key;
+
+		if (filter_type == BPERF_FILTER_PID ||
+		    filter_type == BPERF_FILTER_TGID)
+			key = evsel->core.threads->map[i].pid;
+		else if (filter_type == BPERF_FILTER_CPU)
+			key = evsel->core.cpus->map[i];
+		else
+			break;
+
+		filter_map_fd = bpf_map__fd(evsel->follower_skel->maps.filter);
+		bpf_map_update_elem(filter_map_fd, &key, &i, BPF_ANY);
+	}
+
+	evsel->follower_skel->bss->type = filter_type;
+
+	err = bperf_follower_bpf__attach(evsel->follower_skel);
+
+out:
+	if (err && evsel->bperf_leader_link_fd >= 0)
+		close(evsel->bperf_leader_link_fd);
+	if (err && evsel->bperf_leader_prog_fd >= 0)
+		close(evsel->bperf_leader_prog_fd);
+	if (diff_map_fd >= 0)
+		close(diff_map_fd);
+
+	flock(attr_map_fd, LOCK_UN);
+	close(attr_map_fd);
+
+	return err;
+}
+
+static int bperf__install_pe(struct evsel *evsel, int cpu, int fd)
+{
+	struct bperf_leader_bpf *skel = evsel->leader_skel;
+
+	return bpf_map_update_elem(bpf_map__fd(skel->maps.events),
+				   &cpu, &fd, BPF_ANY);
+}
+
+/*
+ * trigger the leader prog on each cpu, so the accum_reading map could get
+ * the latest readings.
+ */
+static int bperf_sync_counters(struct evsel *evsel)
+{
+	int num_cpu, i, cpu;
+
+	num_cpu = all_cpu_map->nr;
+	for (i = 0; i < num_cpu; i++) {
+		cpu = all_cpu_map->map[i];
+		bperf_trigger_reading(evsel->bperf_leader_prog_fd, cpu);
+	}
+	return 0;
+}
+
+static int bperf__enable(struct evsel *evsel)
+{
+	evsel->follower_skel->bss->enabled = 1;
+	return 0;
+}
+
+static int bperf__read(struct evsel *evsel)
+{
+	struct bperf_follower_bpf *skel = evsel->follower_skel;
+	__u32 num_cpu_bpf = cpu__max_cpu();
+	struct bpf_perf_event_value values[num_cpu_bpf];
+	int reading_map_fd, err = 0;
+	__u32 i, j, num_cpu;
+
+	bperf_sync_counters(evsel);
+	reading_map_fd = bpf_map__fd(skel->maps.accum_readings);
+
+	for (i = 0; i < bpf_map__max_entries(skel->maps.accum_readings); i++) {
+		__u32 cpu;
+
+		err = bpf_map_lookup_elem(reading_map_fd, &i, values);
+		if (err)
+			goto out;
+		switch (evsel->follower_skel->bss->type) {
+		case BPERF_FILTER_GLOBAL:
+			assert(i == 0);
+
+			num_cpu = all_cpu_map->nr;
+			for (j = 0; j < num_cpu; j++) {
+				cpu = all_cpu_map->map[j];
+				perf_counts(evsel->counts, cpu, 0)->val = values[cpu].counter;
+				perf_counts(evsel->counts, cpu, 0)->ena = values[cpu].enabled;
+				perf_counts(evsel->counts, cpu, 0)->run = values[cpu].running;
+			}
+			break;
+		case BPERF_FILTER_CPU:
+			cpu = evsel->core.cpus->map[i];
+			perf_counts(evsel->counts, i, 0)->val = values[cpu].counter;
+			perf_counts(evsel->counts, i, 0)->ena = values[cpu].enabled;
+			perf_counts(evsel->counts, i, 0)->run = values[cpu].running;
+			break;
+		case BPERF_FILTER_PID:
+		case BPERF_FILTER_TGID:
+			perf_counts(evsel->counts, 0, i)->val = 0;
+			perf_counts(evsel->counts, 0, i)->ena = 0;
+			perf_counts(evsel->counts, 0, i)->run = 0;
+
+			for (cpu = 0; cpu < num_cpu_bpf; cpu++) {
+				perf_counts(evsel->counts, 0, i)->val += values[cpu].counter;
+				perf_counts(evsel->counts, 0, i)->ena += values[cpu].enabled;
+				perf_counts(evsel->counts, 0, i)->run += values[cpu].running;
+			}
+			break;
+		default:
+			break;
+		}
+	}
+out:
+	return err;
+}
+
+static int bperf__destroy(struct evsel *evsel)
+{
+	bperf_follower_bpf__destroy(evsel->follower_skel);
+	close(evsel->bperf_leader_prog_fd);
+	close(evsel->bperf_leader_link_fd);
+	return 0;
+}
+
+/*
+ * bperf: share hardware PMCs with BPF
+ *
+ * perf uses performance monitoring counters (PMC) to monitor system
+ * performance. The PMCs are limited hardware resources. For example,
+ * Intel CPUs have 3x fixed PMCs and 4x programmable PMCs per cpu.
+ *
+ * Modern data center systems use these PMCs in many different ways:
+ * system level monitoring, (maybe nested) container level monitoring, per
+ * process monitoring, profiling (in sample mode), etc. In some cases,
+ * there are more active perf_events than available hardware PMCs. To allow
+ * all perf_events to have a chance to run, it is necessary to do expensive
+ * time multiplexing of events.
+ *
+ * On the other hand, many monitoring tools count the common metrics
+ * (cycles, instructions). It is a waste to have multiple tools create
+ * multiple perf_events of "cycles" and occupy multiple PMCs.
+ *
+ * bperf tries to reduce such wastes by allowing multiple perf_events of
+ * "cycles" or "instructions" (at different scopes) to share PMUs. Instead
+ * of having each perf-stat session to read its own perf_events, bperf uses
+ * BPF programs to read the perf_events and aggregate readings to BPF maps.
+ * Then, the perf-stat session(s) reads the values from these BPF maps.
+ *
+ *                                ||
+ *       shared progs and maps <- || -> per session progs and maps
+ *                                ||
+ *   ---------------              ||
+ *   | perf_events |              ||
+ *   ---------------       fexit  ||      -----------------
+ *          |             --------||----> | follower prog |
+ *       --------------- /        || ---  -----------------
+ * cs -> | leader prog |/         ||/        |         |
+ *   --> ---------------         /||  --------------  ------------------
+ *  /       |         |         / ||  | filter map |  | accum_readings |
+ * /  ------------  ------------  ||  --------------  ------------------
+ * |  | prev map |  | diff map |  ||                        |
+ * |  ------------  ------------  ||                        |
+ *  \                             ||                        |
+ * = \ ==================================================== | ============
+ *    \                                                    /   user space
+ *     \                                                  /
+ *      \                                                /
+ *    BPF_PROG_TEST_RUN                    BPF_MAP_LOOKUP_ELEM
+ *        \                                            /
+ *         \                                          /
+ *          \------  perf-stat ----------------------/
+ *
+ * The figure above shows the architecture of bperf. Note that the figure
+ * is divided into 3 regions: shared progs and maps (top left), per session
+ * progs and maps (top right), and user space (bottom).
+ *
+ * The leader prog is triggered on each context switch (cs). The leader
+ * prog reads perf_events and stores the difference (current_reading -
+ * previous_reading) to the diff map. For the same metric, e.g. "cycles",
+ * multiple perf-stat sessions share the same leader prog.
+ *
+ * Each perf-stat session creates a follower prog as fexit program to the
+ * leader prog. It is possible to attach up to BPF_MAX_TRAMP_PROGS (38)
+ * follower progs to the same leader prog. The follower prog checks current
+ * task and processor ID to decide whether to add the value from the diff
+ * map to its accumulated reading map (accum_readings).
+ *
+ * Finally, perf-stat user space reads the value from accum_reading map.
+ *
+ * Besides context switch, it is also necessary to trigger the leader prog
+ * before perf-stat reads the value. Otherwise, the accum_reading map may
+ * not have the latest reading from the perf_events. This is achieved by
+ * triggering the event via sys_bpf(BPF_PROG_TEST_RUN) to each CPU.
+ *
+ * Comment before the definition of struct perf_event_attr_map_entry
+ * describes how different sessions of perf-stat share information about
+ * the leader prog.
+ */
+
+struct bpf_counter_ops bperf_ops = {
+	.load       = bperf__load,
+	.enable     = bperf__enable,
+	.read       = bperf__read,
+	.install_pe = bperf__install_pe,
+	.destroy    = bperf__destroy,
+};
+
+static inline bool bpf_counter_skip(struct evsel *evsel)
+{
+	return list_empty(&evsel->bpf_counter_list) &&
+		evsel->follower_skel == NULL;
+}
+
 int bpf_counter__install_pe(struct evsel *evsel, int cpu, int fd)
 {
-	if (list_empty(&evsel->bpf_counter_list))
+	if (bpf_counter_skip(evsel))
 		return 0;
 	return evsel->bpf_counter_ops->install_pe(evsel, cpu, fd);
 }
 
 int bpf_counter__load(struct evsel *evsel, struct target *target)
 {
-	if (target__has_bpf(target))
+	if (target->bpf_str)
 		evsel->bpf_counter_ops = &bpf_program_profiler_ops;
+	else if (target->use_bpf)
+		evsel->bpf_counter_ops = &bperf_ops;
 
 	if (evsel->bpf_counter_ops)
 		return evsel->bpf_counter_ops->load(evsel, target);
@@ -293,21 +802,21 @@ int bpf_counter__load(struct evsel *evsel, struct target *target)
 
 int bpf_counter__enable(struct evsel *evsel)
 {
-	if (list_empty(&evsel->bpf_counter_list))
+	if (bpf_counter_skip(evsel))
 		return 0;
 	return evsel->bpf_counter_ops->enable(evsel);
 }
 
 int bpf_counter__read(struct evsel *evsel)
 {
-	if (list_empty(&evsel->bpf_counter_list))
+	if (bpf_counter_skip(evsel))
 		return -EAGAIN;
 	return evsel->bpf_counter_ops->read(evsel);
 }
 
 void bpf_counter__destroy(struct evsel *evsel)
 {
-	if (list_empty(&evsel->bpf_counter_list))
+	if (bpf_counter_skip(evsel))
 		return;
 	evsel->bpf_counter_ops->destroy(evsel);
 	evsel->bpf_counter_ops = NULL;
diff --git a/tools/perf/util/bpf_skel/bperf.h b/tools/perf/util/bpf_skel/bperf.h
new file mode 100644
index 0000000000000..186a5551ddb9d
--- /dev/null
+++ b/tools/perf/util/bpf_skel/bperf.h
@@ -0,0 +1,14 @@
+// SPDX-License-Identifier: (GPL-2.0-only OR BSD-2-Clause)
+// Copyright (c) 2021 Facebook
+
+#ifndef __BPERF_STAT_H
+#define __BPERF_STAT_H
+
+typedef struct {
+	__uint(type, BPF_MAP_TYPE_PERCPU_ARRAY);
+	__uint(key_size, sizeof(__u32));
+	__uint(value_size, sizeof(struct bpf_perf_event_value));
+	__uint(max_entries, 1);
+} reading_map;
+
+#endif /* __BPERF_STAT_H */
diff --git a/tools/perf/util/bpf_skel/bperf_follower.bpf.c b/tools/perf/util/bpf_skel/bperf_follower.bpf.c
new file mode 100644
index 0000000000000..b8fa3cb2da230
--- /dev/null
+++ b/tools/perf/util/bpf_skel/bperf_follower.bpf.c
@@ -0,0 +1,69 @@
+// SPDX-License-Identifier: (GPL-2.0-only OR BSD-2-Clause)
+// Copyright (c) 2021 Facebook
+#include <linux/bpf.h>
+#include <linux/perf_event.h>
+#include <bpf/bpf_helpers.h>
+#include <bpf/bpf_tracing.h>
+#include "bperf.h"
+#include "bperf_u.h"
+
+reading_map diff_readings SEC(".maps");
+reading_map accum_readings SEC(".maps");
+
+struct {
+	__uint(type, BPF_MAP_TYPE_HASH);
+	__uint(key_size, sizeof(__u32));
+	__uint(value_size, sizeof(__u32));
+} filter SEC(".maps");
+
+enum bperf_filter_type type = 0;
+int enabled = 0;
+
+SEC("fexit/XXX")
+int BPF_PROG(fexit_XXX)
+{
+	struct bpf_perf_event_value *diff_val, *accum_val;
+	__u32 filter_key, zero = 0;
+	__u32 *accum_key;
+
+	if (!enabled)
+		return 0;
+
+	switch (type) {
+	case BPERF_FILTER_GLOBAL:
+		accum_key = &zero;
+		goto do_add;
+	case BPERF_FILTER_CPU:
+		filter_key = bpf_get_smp_processor_id();
+		break;
+	case BPERF_FILTER_PID:
+		filter_key = bpf_get_current_pid_tgid() & 0xffffffff;
+		break;
+	case BPERF_FILTER_TGID:
+		filter_key = bpf_get_current_pid_tgid() >> 32;
+		break;
+	default:
+		return 0;
+	}
+
+	accum_key = bpf_map_lookup_elem(&filter, &filter_key);
+	if (!accum_key)
+		return 0;
+
+do_add:
+	diff_val = bpf_map_lookup_elem(&diff_readings, &zero);
+	if (!diff_val)
+		return 0;
+
+	accum_val = bpf_map_lookup_elem(&accum_readings, accum_key);
+	if (!accum_val)
+		return 0;
+
+	accum_val->counter += diff_val->counter;
+	accum_val->enabled += diff_val->enabled;
+	accum_val->running += diff_val->running;
+
+	return 0;
+}
+
+char LICENSE[] SEC("license") = "Dual BSD/GPL";
diff --git a/tools/perf/util/bpf_skel/bperf_leader.bpf.c b/tools/perf/util/bpf_skel/bperf_leader.bpf.c
new file mode 100644
index 0000000000000..4f70d1459e86c
--- /dev/null
+++ b/tools/perf/util/bpf_skel/bperf_leader.bpf.c
@@ -0,0 +1,46 @@
+// SPDX-License-Identifier: (GPL-2.0-only OR BSD-2-Clause)
+// Copyright (c) 2021 Facebook
+#include <linux/bpf.h>
+#include <linux/perf_event.h>
+#include <bpf/bpf_helpers.h>
+#include <bpf/bpf_tracing.h>
+#include "bperf.h"
+
+struct {
+	__uint(type, BPF_MAP_TYPE_PERF_EVENT_ARRAY);
+	__uint(key_size, sizeof(__u32));
+	__uint(value_size, sizeof(int));
+	__uint(map_flags, BPF_F_PRESERVE_ELEMS);
+} events SEC(".maps");
+
+reading_map prev_readings SEC(".maps");
+reading_map diff_readings SEC(".maps");
+
+SEC("raw_tp/sched_switch")
+int BPF_PROG(on_switch)
+{
+	struct bpf_perf_event_value val, *prev_val, *diff_val;
+	__u32 key = bpf_get_smp_processor_id();
+	__u32 zero = 0;
+	long err;
+
+	prev_val = bpf_map_lookup_elem(&prev_readings, &zero);
+	if (!prev_val)
+		return 0;
+
+	diff_val = bpf_map_lookup_elem(&diff_readings, &zero);
+	if (!diff_val)
+		return 0;
+
+	err = bpf_perf_event_read_value(&events, key, &val, sizeof(val));
+	if (err)
+		return 0;
+
+	diff_val->counter = val.counter - prev_val->counter;
+	diff_val->enabled = val.enabled - prev_val->enabled;
+	diff_val->running = val.running - prev_val->running;
+	*prev_val = val;
+	return 0;
+}
+
+char LICENSE[] SEC("license") = "Dual BSD/GPL";
diff --git a/tools/perf/util/bpf_skel/bperf_u.h b/tools/perf/util/bpf_skel/bperf_u.h
new file mode 100644
index 0000000000000..1ce0c2c905c11
--- /dev/null
+++ b/tools/perf/util/bpf_skel/bperf_u.h
@@ -0,0 +1,14 @@
+// SPDX-License-Identifier: (GPL-2.0-only OR BSD-2-Clause)
+// Copyright (c) 2021 Facebook
+
+#ifndef __BPERF_STAT_U_H
+#define __BPERF_STAT_U_H
+
+enum bperf_filter_type {
+	BPERF_FILTER_GLOBAL = 1,
+	BPERF_FILTER_CPU,
+	BPERF_FILTER_PID,
+	BPERF_FILTER_TGID,
+};
+
+#endif /* __BPERF_STAT_U_H */
diff --git a/tools/perf/util/evsel.h b/tools/perf/util/evsel.h
index 6026487353dd8..dd4f56f9cfdf5 100644
--- a/tools/perf/util/evsel.h
+++ b/tools/perf/util/evsel.h
@@ -20,6 +20,8 @@ union perf_event;
 struct bpf_counter_ops;
 struct target;
 struct hashmap;
+struct bperf_leader_bpf;
+struct bperf_follower_bpf;
 
 typedef int (evsel__sb_cb_t)(union perf_event *event, void *data);
 
@@ -130,8 +132,24 @@ struct evsel {
 	 * See also evsel__has_callchain().
 	 */
 	__u64			synth_sample_type;
-	struct list_head	bpf_counter_list;
+
+	/*
+	 * bpf_counter_ops serves two use cases:
+	 *   1. perf-stat -b          counting events used byBPF programs
+	 *   2. perf-stat --use-bpf   use BPF programs to aggregate counts
+	 */
 	struct bpf_counter_ops	*bpf_counter_ops;
+
+	/* for perf-stat -b */
+	struct list_head	bpf_counter_list;
+
+	/* for perf-stat --use-bpf */
+	int			bperf_leader_prog_fd;
+	int			bperf_leader_link_fd;
+	union {
+		struct bperf_leader_bpf *leader_skel;
+		struct bperf_follower_bpf *follower_skel;
+	};
 };
 
 struct perf_missing_features {
diff --git a/tools/perf/util/target.h b/tools/perf/util/target.h
index f132c6c2eef81..1bce3eb28ef25 100644
--- a/tools/perf/util/target.h
+++ b/tools/perf/util/target.h
@@ -16,6 +16,8 @@ struct target {
 	bool	     uses_mmap;
 	bool	     default_per_cpu;
 	bool	     per_thread;
+	bool	     use_bpf;
+	const char   *attr_map;
 };
 
 enum target_errno {
@@ -66,7 +68,7 @@ static inline bool target__has_cpu(struct target *target)
 
 static inline bool target__has_bpf(struct target *target)
 {
-	return target->bpf_str;
+	return target->bpf_str || target->use_bpf;
 }
 
 static inline bool target__none(struct target *target)
-- 
2.30.2

[PATCH v2 2/3] perf-stat: measure t0 and ref_time after enable_counters()

[Song Liu]

Take measurements of t0 and ref_time after enable_counters(), so that
they only measure the time consumed when the counters are enabled.

Signed-off-by: Song Liu <songliubraving@fb.com>
---
 tools/perf/builtin-stat.c | 10 +++++++---
 1 file changed, 7 insertions(+), 3 deletions(-)

diff --git a/tools/perf/builtin-stat.c b/tools/perf/builtin-stat.c
index 92696373da994..d030c3a49a8e4 100644
--- a/tools/perf/builtin-stat.c
+++ b/tools/perf/builtin-stat.c
@@ -931,15 +931,15 @@ static int __run_perf_stat(int argc, const char **argv, int run_idx)
 	/*
 	 * Enable counters and exec the command:
 	 */
-	t0 = rdclock();
-	clock_gettime(CLOCK_MONOTONIC, &ref_time);
-
 	if (forks) {
 		evlist__start_workload(evsel_list);
 		err = enable_counters();
 		if (err)
 			return -1;
 
+		t0 = rdclock();
+		clock_gettime(CLOCK_MONOTONIC, &ref_time);
+
 		if (interval || timeout || evlist__ctlfd_initialized(evsel_list))
 			status = dispatch_events(forks, timeout, interval, &times);
 		if (child_pid != -1) {
@@ -960,6 +960,10 @@ static int __run_perf_stat(int argc, const char **argv, int run_idx)
 		err = enable_counters();
 		if (err)
 			return -1;
+
+		t0 = rdclock();
+		clock_gettime(CLOCK_MONOTONIC, &ref_time);
+
 		status = dispatch_events(forks, timeout, interval, &times);
 	}
 
-- 
2.30.2

[PATCH v2 3/3] perf-test: add a test for perf-stat --bpf-counters option

[Song Liu]

Add a test to compare the output of perf-stat with and without option
--bpf-counters. If the difference is more than 10%, the test is considered
as failed.

For stable results between two runs (w/ and w/o --bpf-counters), the test
program should: 1) be long enough for better signal-noise-ratio; 2) not
depend on the behavior of IO subsystem (for less noise from caching). So
far, the best option we found is stressapptest.

Signed-off-by: Song Liu <songliubraving@fb.com>
---
 tools/perf/tests/shell/stat_bpf_counters.sh | 34 +++++++++++++++++++++
 1 file changed, 34 insertions(+)
 create mode 100755 tools/perf/tests/shell/stat_bpf_counters.sh

diff --git a/tools/perf/tests/shell/stat_bpf_counters.sh b/tools/perf/tests/shell/stat_bpf_counters.sh
new file mode 100755
index 0000000000000..c0bcb38d6b53c
--- /dev/null
+++ b/tools/perf/tests/shell/stat_bpf_counters.sh
@@ -0,0 +1,34 @@
+#!/bin/sh
+# perf stat --bpf-counters test
+# SPDX-License-Identifier: GPL-2.0
+
+set -e
+
+# check whether $2 is within +/- 10% of $1
+compare_number()
+{
+	first_num=$1
+	second_num=$2
+
+	# upper bound is first_num * 110%
+	upper=$(( $first_num + $first_num / 10 ))
+	# lower bound is first_num * 90%
+	lower=$(( $first_num - $first_num / 10 ))
+
+	if [ $second_num -gt $upper ] || [ $second_num -lt $lower ]; then
+		echo "The difference between $first_num and $second_num are greater than 10%."
+		exit 1
+	fi
+}
+
+# skip if --bpf-counters is not supported
+perf stat --bpf-counters true > /dev/null 2>&1 || exit 2
+
+# skip if stressapptest is not available
+stressapptest -s 1 -M 100 -m 1 > /dev/null 2>&1 || exit 2
+
+base_cycles=$(perf stat --no-big-num -e cycles -- stressapptest -s 3 -M 100 -m 1 2>&1 | grep -e cycles | awk '{print $1}')
+bpf_cycles=$(perf stat --no-big-num --bpf-counters -e cycles -- stressapptest -s 3 -M 100 -m 1 2>&1 | grep -e cycles | awk '{print $1}')
+
+compare_number $base_cycles $bpf_cycles
+exit 0
-- 
2.30.2

Re: [PATCH v2 0/3] perf-stat: share hardware PMCs with BPF

[Namhyung Kim]

Hi Song,

On Wed, Mar 17, 2021 at 6:18 AM Song Liu <songliubraving@fb.com> wrote:
>
> perf uses performance monitoring counters (PMCs) to monitor system
> performance. The PMCs are limited hardware resources. For example,
> Intel CPUs have 3x fixed PMCs and 4x programmable PMCs per cpu.
>
> Modern data center systems use these PMCs in many different ways:
> system level monitoring, (maybe nested) container level monitoring, per
> process monitoring, profiling (in sample mode), etc. In some cases,
> there are more active perf_events than available hardware PMCs. To allow
> all perf_events to have a chance to run, it is necessary to do expensive
> time multiplexing of events.
>
> On the other hand, many monitoring tools count the common metrics (cycles,
> instructions). It is a waste to have multiple tools create multiple
> perf_events of "cycles" and occupy multiple PMCs.

Right, it'd be really helpful when the PMCs are frequently or mostly shared.
But it'd also increase the overhead for uncontended cases as BPF programs
need to run on every context switch.  Depending on the workload, it may
cause a non-negligible performance impact.  So users should be aware of it.

Thanks,
Namhyung

>
> bperf tries to reduce such wastes by allowing multiple perf_events of
> "cycles" or "instructions" (at different scopes) to share PMUs. Instead
> of having each perf-stat session to read its own perf_events, bperf uses
> BPF programs to read the perf_events and aggregate readings to BPF maps.
> Then, the perf-stat session(s) reads the values from these BPF maps.
>
> Changes v1 => v2:
>   1. Add documentation.
>   2. Add a shell test.
>   3. Rename options, default path of the atto-map, and some variables.
>   4. Add a separate patch that moves clock_gettime() in __run_perf_stat()
>      to after enable_counters().
>   5. Make perf_cpu_map for all cpus a global variable.
>   6. Use sysfs__mountpoint() for default attr-map path.
>   7. Use cpu__max_cpu() instead of libbpf_num_possible_cpus().
>   8. Add flag "enabled" to the follower program. Then move follower attach
>      to bperf__load() and simplify bperf__enable().
>
> Song Liu (3):
>   perf-stat: introduce bperf, share hardware PMCs with BPF
>   perf-stat: measure t0 and ref_time after enable_counters()
>   perf-test: add a test for perf-stat --bpf-counters option
>
>  tools/perf/Documentation/perf-stat.txt        |  11 +
>  tools/perf/Makefile.perf                      |   1 +
>  tools/perf/builtin-stat.c                     |  20 +-
>  tools/perf/tests/shell/stat_bpf_counters.sh   |  34 ++
>  tools/perf/util/bpf_counter.c                 | 519 +++++++++++++++++-
>  tools/perf/util/bpf_skel/bperf.h              |  14 +
>  tools/perf/util/bpf_skel/bperf_follower.bpf.c |  69 +++
>  tools/perf/util/bpf_skel/bperf_leader.bpf.c   |  46 ++
>  tools/perf/util/bpf_skel/bperf_u.h            |  14 +
>  tools/perf/util/evsel.h                       |  20 +-
>  tools/perf/util/target.h                      |   4 +-
>  11 files changed, 742 insertions(+), 10 deletions(-)
>  create mode 100755 tools/perf/tests/shell/stat_bpf_counters.sh
>  create mode 100644 tools/perf/util/bpf_skel/bperf.h
>  create mode 100644 tools/perf/util/bpf_skel/bperf_follower.bpf.c
>  create mode 100644 tools/perf/util/bpf_skel/bperf_leader.bpf.c
>  create mode 100644 tools/perf/util/bpf_skel/bperf_u.h
>
> --
> 2.30.2

Re: [PATCH v2 0/3] perf-stat: share hardware PMCs with BPF

[Jiri Olsa]

On Wed, Mar 17, 2021 at 02:29:28PM +0900, Namhyung Kim wrote:
> Hi Song,
> 
> On Wed, Mar 17, 2021 at 6:18 AM Song Liu <songliubraving@fb.com> wrote:
> >
> > perf uses performance monitoring counters (PMCs) to monitor system
> > performance. The PMCs are limited hardware resources. For example,
> > Intel CPUs have 3x fixed PMCs and 4x programmable PMCs per cpu.
> >
> > Modern data center systems use these PMCs in many different ways:
> > system level monitoring, (maybe nested) container level monitoring, per
> > process monitoring, profiling (in sample mode), etc. In some cases,
> > there are more active perf_events than available hardware PMCs. To allow
> > all perf_events to have a chance to run, it is necessary to do expensive
> > time multiplexing of events.
> >
> > On the other hand, many monitoring tools count the common metrics (cycles,
> > instructions). It is a waste to have multiple tools create multiple
> > perf_events of "cycles" and occupy multiple PMCs.
> 
> Right, it'd be really helpful when the PMCs are frequently or mostly shared.
> But it'd also increase the overhead for uncontended cases as BPF programs
> need to run on every context switch.  Depending on the workload, it may
> cause a non-negligible performance impact.  So users should be aware of it.

right, let's get get some idea of how bad that actualy is

Song,
could you please get some numbers from runnning for example
'perf bench sched messaging ...' with both normal and bpf
mode perf stat? for all supported target options

thanks,
jirka

Re: [PATCH v2 0/3] perf-stat: share hardware PMCs with BPF

[Arnaldo Carvalho de Melo]

Em Wed, Mar 17, 2021 at 02:29:28PM +0900, Namhyung Kim escreveu:
> Hi Song,
> 
> On Wed, Mar 17, 2021 at 6:18 AM Song Liu <songliubraving@fb.com> wrote:
> >
> > perf uses performance monitoring counters (PMCs) to monitor system
> > performance. The PMCs are limited hardware resources. For example,
> > Intel CPUs have 3x fixed PMCs and 4x programmable PMCs per cpu.
> >
> > Modern data center systems use these PMCs in many different ways:
> > system level monitoring, (maybe nested) container level monitoring, per
> > process monitoring, profiling (in sample mode), etc. In some cases,
> > there are more active perf_events than available hardware PMCs. To allow
> > all perf_events to have a chance to run, it is necessary to do expensive
> > time multiplexing of events.
> >
> > On the other hand, many monitoring tools count the common metrics (cycles,
> > instructions). It is a waste to have multiple tools create multiple
> > perf_events of "cycles" and occupy multiple PMCs.
> 
> Right, it'd be really helpful when the PMCs are frequently or mostly shared.
> But it'd also increase the overhead for uncontended cases as BPF programs
> need to run on every context switch.  Depending on the workload, it may
> cause a non-negligible performance impact.  So users should be aware of it.

Would be interesting to, humm, measure both cases to have a firm number
of the impact, how many instructions are added when sharing using
--bpf-counters?

I.e. compare the "expensive time multiplexing of events" with its
avoidance by using --bpf-counters.

Song, have you perfmormed such measurements?

- Arnaldo
 
> Thanks,
> Namhyung
> 
> >
> > bperf tries to reduce such wastes by allowing multiple perf_events of
> > "cycles" or "instructions" (at different scopes) to share PMUs. Instead
> > of having each perf-stat session to read its own perf_events, bperf uses
> > BPF programs to read the perf_events and aggregate readings to BPF maps.
> > Then, the perf-stat session(s) reads the values from these BPF maps.
> >
> > Changes v1 => v2:
> >   1. Add documentation.
> >   2. Add a shell test.
> >   3. Rename options, default path of the atto-map, and some variables.
> >   4. Add a separate patch that moves clock_gettime() in __run_perf_stat()
> >      to after enable_counters().
> >   5. Make perf_cpu_map for all cpus a global variable.
> >   6. Use sysfs__mountpoint() for default attr-map path.
> >   7. Use cpu__max_cpu() instead of libbpf_num_possible_cpus().
> >   8. Add flag "enabled" to the follower program. Then move follower attach
> >      to bperf__load() and simplify bperf__enable().
> >
> > Song Liu (3):
> >   perf-stat: introduce bperf, share hardware PMCs with BPF
> >   perf-stat: measure t0 and ref_time after enable_counters()
> >   perf-test: add a test for perf-stat --bpf-counters option
> >
> >  tools/perf/Documentation/perf-stat.txt        |  11 +
> >  tools/perf/Makefile.perf                      |   1 +
> >  tools/perf/builtin-stat.c                     |  20 +-
> >  tools/perf/tests/shell/stat_bpf_counters.sh   |  34 ++
> >  tools/perf/util/bpf_counter.c                 | 519 +++++++++++++++++-
> >  tools/perf/util/bpf_skel/bperf.h              |  14 +
> >  tools/perf/util/bpf_skel/bperf_follower.bpf.c |  69 +++
> >  tools/perf/util/bpf_skel/bperf_leader.bpf.c   |  46 ++
> >  tools/perf/util/bpf_skel/bperf_u.h            |  14 +
> >  tools/perf/util/evsel.h                       |  20 +-
> >  tools/perf/util/target.h                      |   4 +-
> >  11 files changed, 742 insertions(+), 10 deletions(-)
> >  create mode 100755 tools/perf/tests/shell/stat_bpf_counters.sh
> >  create mode 100644 tools/perf/util/bpf_skel/bperf.h
> >  create mode 100644 tools/perf/util/bpf_skel/bperf_follower.bpf.c
> >  create mode 100644 tools/perf/util/bpf_skel/bperf_leader.bpf.c
> >  create mode 100644 tools/perf/util/bpf_skel/bperf_u.h
> >
> > --
> > 2.30.2

-- 

- Arnaldo

Re: [PATCH v2 0/3] perf-stat: share hardware PMCs with BPF

[Song Liu]

> On Mar 17, 2021, at 6:11 AM, Arnaldo Carvalho de Melo <acme@kernel.org> wrote:
> 
> Em Wed, Mar 17, 2021 at 02:29:28PM +0900, Namhyung Kim escreveu:
>> Hi Song,
>> 
>> On Wed, Mar 17, 2021 at 6:18 AM Song Liu <songliubraving@fb.com> wrote:
>>> 
>>> perf uses performance monitoring counters (PMCs) to monitor system
>>> performance. The PMCs are limited hardware resources. For example,
>>> Intel CPUs have 3x fixed PMCs and 4x programmable PMCs per cpu.
>>> 
>>> Modern data center systems use these PMCs in many different ways:
>>> system level monitoring, (maybe nested) container level monitoring, per
>>> process monitoring, profiling (in sample mode), etc. In some cases,
>>> there are more active perf_events than available hardware PMCs. To allow
>>> all perf_events to have a chance to run, it is necessary to do expensive
>>> time multiplexing of events.
>>> 
>>> On the other hand, many monitoring tools count the common metrics (cycles,
>>> instructions). It is a waste to have multiple tools create multiple
>>> perf_events of "cycles" and occupy multiple PMCs.
>> 
>> Right, it'd be really helpful when the PMCs are frequently or mostly shared.
>> But it'd also increase the overhead for uncontended cases as BPF programs
>> need to run on every context switch.  Depending on the workload, it may
>> cause a non-negligible performance impact.  So users should be aware of it.
> 
> Would be interesting to, humm, measure both cases to have a firm number
> of the impact, how many instructions are added when sharing using
> --bpf-counters?
> 
> I.e. compare the "expensive time multiplexing of events" with its
> avoidance by using --bpf-counters.
> 
> Song, have you perfmormed such measurements?

I have got some measurements with perf-bench-sched-messaging:

The system: x86_64 with 23 cores (46 HT)

The perf-stat command:
perf stat -e cycles,cycles,instructions,instructions,ref-cycles,ref-cycles <target, etc.>

The benchmark command and output:
./perf bench sched messaging -g 40 -l 50000 -t
# Running 'sched/messaging' benchmark:
# 20 sender and receiver threads per group
# 40 groups == 1600 threads run
     Total time: 10X.XXX [sec]


I use the "Total time" as measurement, so smaller number is better. 

For each condition, I run the command 5 times, and took the median of 
"Total time". 

Baseline (no perf-stat)			104.873 [sec]
# global
perf stat -a				107.887 [sec]
perf stat -a --bpf-counters		106.071 [sec]
# per task
perf stat 				106.314 [sec]
perf stat --bpf-counters 		105.965 [sec]
# per cpu
perf stat -C 1,3,5 			107.063 [sec]
perf stat -C 1,3,5 --bpf-counters 	106.406 [sec]

From the data, --bpf-counters is slightly better than the regular event
for all targets. I noticed that the results are not very stable. There 
are a couple 108.xx runs in some of the conditions (w/ and w/o 
--bpf-counters).


I also measured the average runtime of the BPF programs, with 

	sysctl kernel.bpf_stats_enabled=1

For each event, if we have one leader and two followers, the total run 
time is about 340ns. IOW, 340ns for two perf-stat reading instructions, 
340ns for two perf-stat reading cycles, etc. 

Thanks,
Song

Re: [PATCH v2 0/3] perf-stat: share hardware PMCs with BPF

[Namhyung Kim]

On Thu, Mar 18, 2021 at 12:52 PM Song Liu <songliubraving@fb.com> wrote:
>
>
>
> > On Mar 17, 2021, at 6:11 AM, Arnaldo Carvalho de Melo <acme@kernel.org> wrote:
> >
> > Em Wed, Mar 17, 2021 at 02:29:28PM +0900, Namhyung Kim escreveu:
> >> Hi Song,
> >>
> >> On Wed, Mar 17, 2021 at 6:18 AM Song Liu <songliubraving@fb.com> wrote:
> >>>
> >>> perf uses performance monitoring counters (PMCs) to monitor system
> >>> performance. The PMCs are limited hardware resources. For example,
> >>> Intel CPUs have 3x fixed PMCs and 4x programmable PMCs per cpu.
> >>>
> >>> Modern data center systems use these PMCs in many different ways:
> >>> system level monitoring, (maybe nested) container level monitoring, per
> >>> process monitoring, profiling (in sample mode), etc. In some cases,
> >>> there are more active perf_events than available hardware PMCs. To allow
> >>> all perf_events to have a chance to run, it is necessary to do expensive
> >>> time multiplexing of events.
> >>>
> >>> On the other hand, many monitoring tools count the common metrics (cycles,
> >>> instructions). It is a waste to have multiple tools create multiple
> >>> perf_events of "cycles" and occupy multiple PMCs.
> >>
> >> Right, it'd be really helpful when the PMCs are frequently or mostly shared.
> >> But it'd also increase the overhead for uncontended cases as BPF programs
> >> need to run on every context switch.  Depending on the workload, it may
> >> cause a non-negligible performance impact.  So users should be aware of it.
> >
> > Would be interesting to, humm, measure both cases to have a firm number
> > of the impact, how many instructions are added when sharing using
> > --bpf-counters?
> >
> > I.e. compare the "expensive time multiplexing of events" with its
> > avoidance by using --bpf-counters.
> >
> > Song, have you perfmormed such measurements?
>
> I have got some measurements with perf-bench-sched-messaging:
>
> The system: x86_64 with 23 cores (46 HT)
>
> The perf-stat command:
> perf stat -e cycles,cycles,instructions,instructions,ref-cycles,ref-cycles <target, etc.>
>
> The benchmark command and output:
> ./perf bench sched messaging -g 40 -l 50000 -t
> # Running 'sched/messaging' benchmark:
> # 20 sender and receiver threads per group
> # 40 groups == 1600 threads run
>      Total time: 10X.XXX [sec]
>
>
> I use the "Total time" as measurement, so smaller number is better.
>
> For each condition, I run the command 5 times, and took the median of
> "Total time".
>
> Baseline (no perf-stat)                 104.873 [sec]
> # global
> perf stat -a                            107.887 [sec]
> perf stat -a --bpf-counters             106.071 [sec]
> # per task
> perf stat                               106.314 [sec]
> perf stat --bpf-counters                105.965 [sec]
> # per cpu
> perf stat -C 1,3,5                      107.063 [sec]
> perf stat -C 1,3,5 --bpf-counters       106.406 [sec]
>
> From the data, --bpf-counters is slightly better than the regular event
> for all targets. I noticed that the results are not very stable. There
> are a couple 108.xx runs in some of the conditions (w/ and w/o
> --bpf-counters).

Hmm.. so this result is when multiplexing happened, right?
I wondered how/why the regular perf stat is slower..

Thanks,
Namhyung

>
>
> I also measured the average runtime of the BPF programs, with
>
>         sysctl kernel.bpf_stats_enabled=1
>
> For each event, if we have one leader and two followers, the total run
> time is about 340ns. IOW, 340ns for two perf-stat reading instructions,
> 340ns for two perf-stat reading cycles, etc.
>
> Thanks,
> Song

Re: [PATCH v2 1/3] perf-stat: introduce bperf, share hardware PMCs with BPF

[Namhyung Kim]

On Wed, Mar 17, 2021 at 6:18 AM Song Liu <songliubraving@fb.com> wrote:
> +static int bperf_check_target(struct evsel *evsel,
> +                             struct target *target,
> +                             enum bperf_filter_type *filter_type,
> +                             __u32 *filter_entry_cnt)
> +{
> +       if (evsel->leader->core.nr_members > 1) {
> +               pr_err("bpf managed perf events do not yet support groups.\n");
> +               return -1;
> +       }
> +
> +       /* determine filter type based on target */
> +       if (target->system_wide) {
> +               *filter_type = BPERF_FILTER_GLOBAL;
> +               *filter_entry_cnt = 1;
> +       } else if (target->cpu_list) {
> +               *filter_type = BPERF_FILTER_CPU;
> +               *filter_entry_cnt = perf_cpu_map__nr(evsel__cpus(evsel));
> +       } else if (target->tid) {
> +               *filter_type = BPERF_FILTER_PID;
> +               *filter_entry_cnt = perf_thread_map__nr(evsel->core.threads);
> +       } else if (target->pid || evsel->evlist->workload.pid != -1) {
> +               *filter_type = BPERF_FILTER_TGID;
> +               *filter_entry_cnt = perf_thread_map__nr(evsel->core.threads);
> +       } else {
> +               pr_err("bpf managed perf events do not yet support these targets.\n");
> +               return -1;
> +       }
> +
> +       return 0;
> +}
> +
> +static struct perf_cpu_map *all_cpu_map;
> +
> +static int bperf_reload_leader_program(struct evsel *evsel, int attr_map_fd,
> +                                      struct perf_event_attr_map_entry *entry)
> +{
> +       struct bperf_leader_bpf *skel = bperf_leader_bpf__open();
> +       int link_fd, diff_map_fd, err;
> +       struct bpf_link *link = NULL;
> +
> +       if (!skel) {
> +               pr_err("Failed to open leader skeleton\n");
> +               return -1;
> +       }
> +
> +       bpf_map__resize(skel->maps.events, libbpf_num_possible_cpus());
> +       err = bperf_leader_bpf__load(skel);
> +       if (err) {
> +               pr_err("Failed to load leader skeleton\n");
> +               goto out;
> +       }
> +
> +       err = -1;
> +       link = bpf_program__attach(skel->progs.on_switch);
> +       if (!link) {
> +               pr_err("Failed to attach leader program\n");
> +               goto out;
> +       }
> +
> +       link_fd = bpf_link__fd(link);
> +       diff_map_fd = bpf_map__fd(skel->maps.diff_readings);
> +       entry->link_id = bpf_link_get_id(link_fd);
> +       entry->diff_map_id = bpf_map_get_id(diff_map_fd);
> +       err = bpf_map_update_elem(attr_map_fd, &evsel->core.attr, entry, BPF_ANY);
> +       assert(err == 0);
> +
> +       evsel->bperf_leader_link_fd = bpf_link_get_fd_by_id(entry->link_id);
> +       assert(evsel->bperf_leader_link_fd >= 0);

Isn't it the same as link_fd?

> +
> +       /*
> +        * save leader_skel for install_pe, which is called within
> +        * following evsel__open_per_cpu call
> +        */
> +       evsel->leader_skel = skel;
> +       evsel__open_per_cpu(evsel, all_cpu_map, -1);
> +
> +out:
> +       bperf_leader_bpf__destroy(skel);
> +       bpf_link__destroy(link);

Why do we destroy it?  Is it because we get an another reference?

> +       return err;
> +}
> +
> +static int bperf__load(struct evsel *evsel, struct target *target)
> +{
> +       struct perf_event_attr_map_entry entry = {0xffffffff, 0xffffffff};
> +       int attr_map_fd, diff_map_fd = -1, err;
> +       enum bperf_filter_type filter_type;
> +       __u32 filter_entry_cnt, i;
> +
> +       if (bperf_check_target(evsel, target, &filter_type, &filter_entry_cnt))
> +               return -1;
> +
> +       if (!all_cpu_map) {
> +               all_cpu_map = perf_cpu_map__new(NULL);
> +               if (!all_cpu_map)
> +                       return -1;
> +       }
> +
> +       evsel->bperf_leader_prog_fd = -1;
> +       evsel->bperf_leader_link_fd = -1;
> +
> +       /*
> +        * Step 1: hold a fd on the leader program and the bpf_link, if
> +        * the program is not already gone, reload the program.
> +        * Use flock() to ensure exclusive access to the perf_event_attr
> +        * map.
> +        */
> +       attr_map_fd = bperf_lock_attr_map(target);
> +       if (attr_map_fd < 0) {
> +               pr_err("Failed to lock perf_event_attr map\n");
> +               return -1;
> +       }
> +
> +       err = bpf_map_lookup_elem(attr_map_fd, &evsel->core.attr, &entry);
> +       if (err) {
> +               err = bpf_map_update_elem(attr_map_fd, &evsel->core.attr, &entry, BPF_ANY);
> +               if (err)
> +                       goto out;
> +       }
> +
> +       evsel->bperf_leader_link_fd = bpf_link_get_fd_by_id(entry.link_id);
> +       if (evsel->bperf_leader_link_fd < 0 &&
> +           bperf_reload_leader_program(evsel, attr_map_fd, &entry))
> +               goto out;
> +
> +       /*
> +        * The bpf_link holds reference to the leader program, and the
> +        * leader program holds reference to the maps. Therefore, if
> +        * link_id is valid, diff_map_id should also be valid.
> +        */
> +       evsel->bperf_leader_prog_fd = bpf_prog_get_fd_by_id(
> +               bpf_link_get_prog_id(evsel->bperf_leader_link_fd));
> +       assert(evsel->bperf_leader_prog_fd >= 0);
> +
> +       diff_map_fd = bpf_map_get_fd_by_id(entry.diff_map_id);
> +       assert(diff_map_fd >= 0);
> +
> +       /*
> +        * bperf uses BPF_PROG_TEST_RUN to get accurate reading. Check
> +        * whether the kernel support it
> +        */
> +       err = bperf_trigger_reading(evsel->bperf_leader_prog_fd, 0);
> +       if (err) {
> +               pr_err("The kernel does not support test_run for raw_tp BPF programs.\n"
> +                      "Therefore, --use-bpf might show inaccurate readings\n");
> +               goto out;
> +       }
> +
> +       /* Step 2: load the follower skeleton */
> +       evsel->follower_skel = bperf_follower_bpf__open();
> +       if (!evsel->follower_skel) {
> +               pr_err("Failed to open follower skeleton\n");
> +               goto out;
> +       }
> +
> +       /* attach fexit program to the leader program */
> +       bpf_program__set_attach_target(evsel->follower_skel->progs.fexit_XXX,
> +                                      evsel->bperf_leader_prog_fd, "on_switch");
> +
> +       /* connect to leader diff_reading map */
> +       bpf_map__reuse_fd(evsel->follower_skel->maps.diff_readings, diff_map_fd);
> +
> +       /* set up reading map */
> +       bpf_map__set_max_entries(evsel->follower_skel->maps.accum_readings,
> +                                filter_entry_cnt);
> +       /* set up follower filter based on target */
> +       bpf_map__set_max_entries(evsel->follower_skel->maps.filter,
> +                                filter_entry_cnt);
> +       err = bperf_follower_bpf__load(evsel->follower_skel);
> +       if (err) {
> +               pr_err("Failed to load follower skeleton\n");
> +               bperf_follower_bpf__destroy(evsel->follower_skel);
> +               evsel->follower_skel = NULL;
> +               goto out;
> +       }
> +
> +       for (i = 0; i < filter_entry_cnt; i++) {
> +               int filter_map_fd;
> +               __u32 key;
> +
> +               if (filter_type == BPERF_FILTER_PID ||
> +                   filter_type == BPERF_FILTER_TGID)
> +                       key = evsel->core.threads->map[i].pid;
> +               else if (filter_type == BPERF_FILTER_CPU)
> +                       key = evsel->core.cpus->map[i];
> +               else
> +                       break;
> +
> +               filter_map_fd = bpf_map__fd(evsel->follower_skel->maps.filter);
> +               bpf_map_update_elem(filter_map_fd, &key, &i, BPF_ANY);
> +       }
> +
> +       evsel->follower_skel->bss->type = filter_type;
> +
> +       err = bperf_follower_bpf__attach(evsel->follower_skel);
> +
> +out:
> +       if (err && evsel->bperf_leader_link_fd >= 0)
> +               close(evsel->bperf_leader_link_fd);
> +       if (err && evsel->bperf_leader_prog_fd >= 0)
> +               close(evsel->bperf_leader_prog_fd);
> +       if (diff_map_fd >= 0)
> +               close(diff_map_fd);
> +
> +       flock(attr_map_fd, LOCK_UN);
> +       close(attr_map_fd);
> +
> +       return err;
> +}
> +
> +static int bperf__install_pe(struct evsel *evsel, int cpu, int fd)
> +{
> +       struct bperf_leader_bpf *skel = evsel->leader_skel;
> +
> +       return bpf_map_update_elem(bpf_map__fd(skel->maps.events),
> +                                  &cpu, &fd, BPF_ANY);
> +}
> +
> +/*
> + * trigger the leader prog on each cpu, so the accum_reading map could get
> + * the latest readings.
> + */
> +static int bperf_sync_counters(struct evsel *evsel)
> +{
> +       int num_cpu, i, cpu;
> +
> +       num_cpu = all_cpu_map->nr;
> +       for (i = 0; i < num_cpu; i++) {
> +               cpu = all_cpu_map->map[i];
> +               bperf_trigger_reading(evsel->bperf_leader_prog_fd, cpu);
> +       }
> +       return 0;
> +}
> +
> +static int bperf__enable(struct evsel *evsel)
> +{
> +       evsel->follower_skel->bss->enabled = 1;
> +       return 0;
> +}
> +
> +static int bperf__read(struct evsel *evsel)
> +{
> +       struct bperf_follower_bpf *skel = evsel->follower_skel;
> +       __u32 num_cpu_bpf = cpu__max_cpu();
> +       struct bpf_perf_event_value values[num_cpu_bpf];
> +       int reading_map_fd, err = 0;
> +       __u32 i, j, num_cpu;
> +
> +       bperf_sync_counters(evsel);
> +       reading_map_fd = bpf_map__fd(skel->maps.accum_readings);
> +
> +       for (i = 0; i < bpf_map__max_entries(skel->maps.accum_readings); i++) {
> +               __u32 cpu;
> +
> +               err = bpf_map_lookup_elem(reading_map_fd, &i, values);
> +               if (err)
> +                       goto out;
> +               switch (evsel->follower_skel->bss->type) {
> +               case BPERF_FILTER_GLOBAL:
> +                       assert(i == 0);
> +
> +                       num_cpu = all_cpu_map->nr;
> +                       for (j = 0; j < num_cpu; j++) {
> +                               cpu = all_cpu_map->map[j];
> +                               perf_counts(evsel->counts, cpu, 0)->val = values[cpu].counter;
> +                               perf_counts(evsel->counts, cpu, 0)->ena = values[cpu].enabled;
> +                               perf_counts(evsel->counts, cpu, 0)->run = values[cpu].running;

I'm confused with this.  Does the accum_readings map contain values
for all cpus?  IIUC it has only a single entry but you access it for each cpu.
What am I missing?

Thanks,
Namhyung


> +                       }
> +                       break;
> +               case BPERF_FILTER_CPU:
> +                       cpu = evsel->core.cpus->map[i];
> +                       perf_counts(evsel->counts, i, 0)->val = values[cpu].counter;
> +                       perf_counts(evsel->counts, i, 0)->ena = values[cpu].enabled;
> +                       perf_counts(evsel->counts, i, 0)->run = values[cpu].running;
> +                       break;
> +               case BPERF_FILTER_PID:
> +               case BPERF_FILTER_TGID:
> +                       perf_counts(evsel->counts, 0, i)->val = 0;
> +                       perf_counts(evsel->counts, 0, i)->ena = 0;
> +                       perf_counts(evsel->counts, 0, i)->run = 0;
> +
> +                       for (cpu = 0; cpu < num_cpu_bpf; cpu++) {
> +                               perf_counts(evsel->counts, 0, i)->val += values[cpu].counter;
> +                               perf_counts(evsel->counts, 0, i)->ena += values[cpu].enabled;
> +                               perf_counts(evsel->counts, 0, i)->run += values[cpu].running;
> +                       }
> +                       break;
> +               default:
> +                       break;
> +               }
> +       }
> +out:
> +       return err;
> +}
> +
> +static int bperf__destroy(struct evsel *evsel)
> +{
> +       bperf_follower_bpf__destroy(evsel->follower_skel);
> +       close(evsel->bperf_leader_prog_fd);
> +       close(evsel->bperf_leader_link_fd);
> +       return 0;
> +}

Re: [PATCH v2 3/3] perf-test: add a test for perf-stat --bpf-counters option

[Namhyung Kim]

On Wed, Mar 17, 2021 at 6:18 AM Song Liu <songliubraving@fb.com> wrote:
>
> Add a test to compare the output of perf-stat with and without option
> --bpf-counters. If the difference is more than 10%, the test is considered
> as failed.
>
> For stable results between two runs (w/ and w/o --bpf-counters), the test
> program should: 1) be long enough for better signal-noise-ratio; 2) not
> depend on the behavior of IO subsystem (for less noise from caching). So
> far, the best option we found is stressapptest.
>
> Signed-off-by: Song Liu <songliubraving@fb.com>
> ---
>  tools/perf/tests/shell/stat_bpf_counters.sh | 34 +++++++++++++++++++++
>  1 file changed, 34 insertions(+)
>  create mode 100755 tools/perf/tests/shell/stat_bpf_counters.sh
>
> diff --git a/tools/perf/tests/shell/stat_bpf_counters.sh b/tools/perf/tests/shell/stat_bpf_counters.sh
> new file mode 100755
> index 0000000000000..c0bcb38d6b53c
> --- /dev/null
> +++ b/tools/perf/tests/shell/stat_bpf_counters.sh
> @@ -0,0 +1,34 @@
> +#!/bin/sh
> +# perf stat --bpf-counters test
> +# SPDX-License-Identifier: GPL-2.0
> +
> +set -e
> +
> +# check whether $2 is within +/- 10% of $1
> +compare_number()
> +{
> +       first_num=$1
> +       second_num=$2
> +
> +       # upper bound is first_num * 110%
> +       upper=$(( $first_num + $first_num / 10 ))
> +       # lower bound is first_num * 90%
> +       lower=$(( $first_num - $first_num / 10 ))
> +
> +       if [ $second_num -gt $upper ] || [ $second_num -lt $lower ]; then
> +               echo "The difference between $first_num and $second_num are greater than 10%."
> +               exit 1
> +       fi
> +}
> +
> +# skip if --bpf-counters is not supported
> +perf stat --bpf-counters true > /dev/null 2>&1 || exit 2
> +
> +# skip if stressapptest is not available
> +stressapptest -s 1 -M 100 -m 1 > /dev/null 2>&1 || exit 2

I don't know how popular it is, but we can print some info
in case we miss it.

> +
> +base_cycles=$(perf stat --no-big-num -e cycles -- stressapptest -s 3 -M 100 -m 1 2>&1 | grep -e cycles | awk '{print $1}')
> +bpf_cycles=$(perf stat --no-big-num --bpf-counters -e cycles -- stressapptest -s 3 -M 100 -m 1 2>&1 | grep -e cycles | awk '{print $1}')

I think just awk '/cycles/ {print $1}' should work.

Thanks,
Namhyung


> +
> +compare_number $base_cycles $bpf_cycles
> +exit 0
> --
> 2.30.2
>

Re: [PATCH v2 0/3] perf-stat: share hardware PMCs with BPF

[Song Liu]

> On Mar 17, 2021, at 9:32 PM, Namhyung Kim <namhyung@kernel.org> wrote:
> 
> On Thu, Mar 18, 2021 at 12:52 PM Song Liu <songliubraving@fb.com> wrote:
>> 
>> 
>> 
>>> On Mar 17, 2021, at 6:11 AM, Arnaldo Carvalho de Melo <acme@kernel.org> wrote:
>>> 
>>> Em Wed, Mar 17, 2021 at 02:29:28PM +0900, Namhyung Kim escreveu:
>>>> Hi Song,
>>>> 
>>>> On Wed, Mar 17, 2021 at 6:18 AM Song Liu <songliubraving@fb.com> wrote:
>>>>> 
>>>>> perf uses performance monitoring counters (PMCs) to monitor system
>>>>> performance. The PMCs are limited hardware resources. For example,
>>>>> Intel CPUs have 3x fixed PMCs and 4x programmable PMCs per cpu.
>>>>> 
>>>>> Modern data center systems use these PMCs in many different ways:
>>>>> system level monitoring, (maybe nested) container level monitoring, per
>>>>> process monitoring, profiling (in sample mode), etc. In some cases,
>>>>> there are more active perf_events than available hardware PMCs. To allow
>>>>> all perf_events to have a chance to run, it is necessary to do expensive
>>>>> time multiplexing of events.
>>>>> 
>>>>> On the other hand, many monitoring tools count the common metrics (cycles,
>>>>> instructions). It is a waste to have multiple tools create multiple
>>>>> perf_events of "cycles" and occupy multiple PMCs.
>>>> 
>>>> Right, it'd be really helpful when the PMCs are frequently or mostly shared.
>>>> But it'd also increase the overhead for uncontended cases as BPF programs
>>>> need to run on every context switch.  Depending on the workload, it may
>>>> cause a non-negligible performance impact.  So users should be aware of it.
>>> 
>>> Would be interesting to, humm, measure both cases to have a firm number
>>> of the impact, how many instructions are added when sharing using
>>> --bpf-counters?
>>> 
>>> I.e. compare the "expensive time multiplexing of events" with its
>>> avoidance by using --bpf-counters.
>>> 
>>> Song, have you perfmormed such measurements?
>> 
>> I have got some measurements with perf-bench-sched-messaging:
>> 
>> The system: x86_64 with 23 cores (46 HT)
>> 
>> The perf-stat command:
>> perf stat -e cycles,cycles,instructions,instructions,ref-cycles,ref-cycles <target, etc.>
>> 
>> The benchmark command and output:
>> ./perf bench sched messaging -g 40 -l 50000 -t
>> # Running 'sched/messaging' benchmark:
>> # 20 sender and receiver threads per group
>> # 40 groups == 1600 threads run
>>     Total time: 10X.XXX [sec]
>> 
>> 
>> I use the "Total time" as measurement, so smaller number is better.
>> 
>> For each condition, I run the command 5 times, and took the median of
>> "Total time".
>> 
>> Baseline (no perf-stat)                 104.873 [sec]
>> # global
>> perf stat -a                            107.887 [sec]
>> perf stat -a --bpf-counters             106.071 [sec]
>> # per task
>> perf stat                               106.314 [sec]
>> perf stat --bpf-counters                105.965 [sec]
>> # per cpu
>> perf stat -C 1,3,5                      107.063 [sec]
>> perf stat -C 1,3,5 --bpf-counters       106.406 [sec]
>> 
>> From the data, --bpf-counters is slightly better than the regular event
>> for all targets. I noticed that the results are not very stable. There
>> are a couple 108.xx runs in some of the conditions (w/ and w/o
>> --bpf-counters).
> 
> Hmm.. so this result is when multiplexing happened, right?
> I wondered how/why the regular perf stat is slower..

I should have made this more clear. This is when regular perf-stat time 
multiplexing (2x ref-cycles on Intel). OTOH, bpf-counters does enables 
sharing, so there is no time multiplexing. IOW, this is overhead of BPF 
vs. overhead of time multiplexing. 

Thanks,
Song

Re: [PATCH v2 1/3] perf-stat: introduce bperf, share hardware PMCs with BPF

[Song Liu]

> On Mar 17, 2021, at 10:54 PM, Namhyung Kim <namhyung@kernel.org> wrote:
> 

[...]

>> +
>> +static int bperf_reload_leader_program(struct evsel *evsel, int attr_map_fd,
>> +                                      struct perf_event_attr_map_entry *entry)
>> +{
>> +       struct bperf_leader_bpf *skel = bperf_leader_bpf__open();
>> +       int link_fd, diff_map_fd, err;
>> +       struct bpf_link *link = NULL;
>> +
>> +       if (!skel) {
>> +               pr_err("Failed to open leader skeleton\n");
>> +               return -1;
>> +       }
>> +
>> +       bpf_map__resize(skel->maps.events, libbpf_num_possible_cpus());
>> +       err = bperf_leader_bpf__load(skel);
>> +       if (err) {
>> +               pr_err("Failed to load leader skeleton\n");
>> +               goto out;
>> +       }
>> +
>> +       err = -1;
>> +       link = bpf_program__attach(skel->progs.on_switch);
>> +       if (!link) {
>> +               pr_err("Failed to attach leader program\n");
>> +               goto out;
>> +       }
>> +
>> +       link_fd = bpf_link__fd(link);
>> +       diff_map_fd = bpf_map__fd(skel->maps.diff_readings);
>> +       entry->link_id = bpf_link_get_id(link_fd);
>> +       entry->diff_map_id = bpf_map_get_id(diff_map_fd);
>> +       err = bpf_map_update_elem(attr_map_fd, &evsel->core.attr, entry, BPF_ANY);
>> +       assert(err == 0);
>> +
>> +       evsel->bperf_leader_link_fd = bpf_link_get_fd_by_id(entry->link_id);
>> +       assert(evsel->bperf_leader_link_fd >= 0);
> 
> Isn't it the same as link_fd?

This is a different fd on the same link. 

> 
>> +
>> +       /*
>> +        * save leader_skel for install_pe, which is called within
>> +        * following evsel__open_per_cpu call
>> +        */
>> +       evsel->leader_skel = skel;
>> +       evsel__open_per_cpu(evsel, all_cpu_map, -1);
>> +
>> +out:
>> +       bperf_leader_bpf__destroy(skel);
>> +       bpf_link__destroy(link);
> 
> Why do we destroy it?  Is it because we get an another reference?

Yes. We only need evsel->bperf_leader_link_fd to keep the whole 
skeleton attached. 

When multiple perf-stat sessions are sharing the leader skeleton, 
only the first one loads the leader skeleton, by calling 
bperf_reload_leader_program(). Other sessions simply hold a fd to 
the bpf_link. More explanation in bperf__load() below.  


> 
>> +       return err;
>> +}
>> +
>> +static int bperf__load(struct evsel *evsel, struct target *target)
>> +{
>> +       struct perf_event_attr_map_entry entry = {0xffffffff, 0xffffffff};
>> +       int attr_map_fd, diff_map_fd = -1, err;
>> +       enum bperf_filter_type filter_type;
>> +       __u32 filter_entry_cnt, i;
>> +
>> +       if (bperf_check_target(evsel, target, &filter_type, &filter_entry_cnt))
>> +               return -1;
>> +
>> +       if (!all_cpu_map) {
>> +               all_cpu_map = perf_cpu_map__new(NULL);
>> +               if (!all_cpu_map)
>> +                       return -1;
>> +       }
>> +
>> +       evsel->bperf_leader_prog_fd = -1;
>> +       evsel->bperf_leader_link_fd = -1;
>> +
>> +       /*
>> +        * Step 1: hold a fd on the leader program and the bpf_link, if
>> +        * the program is not already gone, reload the program.
>> +        * Use flock() to ensure exclusive access to the perf_event_attr
>> +        * map.
>> +        */
>> +       attr_map_fd = bperf_lock_attr_map(target);
>> +       if (attr_map_fd < 0) {
>> +               pr_err("Failed to lock perf_event_attr map\n");
>> +               return -1;
>> +       }
>> +
>> +       err = bpf_map_lookup_elem(attr_map_fd, &evsel->core.attr, &entry);
>> +       if (err) {
>> +               err = bpf_map_update_elem(attr_map_fd, &evsel->core.attr, &entry, BPF_ANY);
>> +               if (err)
>> +                       goto out;
>> +       }
>> +
>> +       evsel->bperf_leader_link_fd = bpf_link_get_fd_by_id(entry.link_id);
>> +       if (evsel->bperf_leader_link_fd < 0 &&
>> +           bperf_reload_leader_program(evsel, attr_map_fd, &entry))
>> +               goto out;

Continue with previous explanation. In bperf_reload_leader_program(), 
we open another reference to the link, and destroy the skeleton. This 
brings the code to the same state as evsel->bperf_leader_link_fd >= 
condition above. 

>> +
>> +       /*
>> +        * The bpf_link holds reference to the leader program, and the
>> +        * leader program holds reference to the maps. Therefore, if
>> +        * link_id is valid, diff_map_id should also be valid.
>> +        */
>> +       evsel->bperf_leader_prog_fd = bpf_prog_get_fd_by_id(
>> +               bpf_link_get_prog_id(evsel->bperf_leader_link_fd));
>> +       assert(evsel->bperf_leader_prog_fd >= 0);
>> +
>> +       diff_map_fd = bpf_map_get_fd_by_id(entry.diff_map_id);
>> +       assert(diff_map_fd >= 0);
>> +

[...]

>> +static int bperf__read(struct evsel *evsel)
>> +{
>> +       struct bperf_follower_bpf *skel = evsel->follower_skel;
>> +       __u32 num_cpu_bpf = cpu__max_cpu();
>> +       struct bpf_perf_event_value values[num_cpu_bpf];
>> +       int reading_map_fd, err = 0;
>> +       __u32 i, j, num_cpu;
>> +
>> +       bperf_sync_counters(evsel);
>> +       reading_map_fd = bpf_map__fd(skel->maps.accum_readings);
>> +
>> +       for (i = 0; i < bpf_map__max_entries(skel->maps.accum_readings); i++) {
>> +               __u32 cpu;
>> +
>> +               err = bpf_map_lookup_elem(reading_map_fd, &i, values);
>> +               if (err)
>> +                       goto out;
>> +               switch (evsel->follower_skel->bss->type) {
>> +               case BPERF_FILTER_GLOBAL:
>> +                       assert(i == 0);
>> +
>> +                       num_cpu = all_cpu_map->nr;
>> +                       for (j = 0; j < num_cpu; j++) {
>> +                               cpu = all_cpu_map->map[j];
>> +                               perf_counts(evsel->counts, cpu, 0)->val = values[cpu].counter;
>> +                               perf_counts(evsel->counts, cpu, 0)->ena = values[cpu].enabled;
>> +                               perf_counts(evsel->counts, cpu, 0)->run = values[cpu].running;
> 
> I'm confused with this.  Does the accum_readings map contain values
> for all cpus?  IIUC it has only a single entry but you access it for each cpu.
> What am I missing?

accumulated_reading is a percpu array. In this case, each cpu has its own 
bpf_perf_event_value with index 0. The BPF program could only access the 
data on current cpu. When reading from use space, we get #-of-cpus entries 
for index 0.  

Does this make sense?

Thanks,
Song

Re: [PATCH v2 3/3] perf-test: add a test for perf-stat --bpf-counters option

[Song Liu]

> On Mar 17, 2021, at 11:07 PM, Namhyung Kim <namhyung@kernel.org> wrote:
> 
> On Wed, Mar 17, 2021 at 6:18 AM Song Liu <songliubraving@fb.com> wrote:
>> 
>> Add a test to compare the output of perf-stat with and without option
>> --bpf-counters. If the difference is more than 10%, the test is considered
>> as failed.
>> 
>> For stable results between two runs (w/ and w/o --bpf-counters), the test
>> program should: 1) be long enough for better signal-noise-ratio; 2) not
>> depend on the behavior of IO subsystem (for less noise from caching). So
>> far, the best option we found is stressapptest.
>> 
>> Signed-off-by: Song Liu <songliubraving@fb.com>
>> ---
>> tools/perf/tests/shell/stat_bpf_counters.sh | 34 +++++++++++++++++++++
>> 1 file changed, 34 insertions(+)
>> create mode 100755 tools/perf/tests/shell/stat_bpf_counters.sh
>> 
>> diff --git a/tools/perf/tests/shell/stat_bpf_counters.sh b/tools/perf/tests/shell/stat_bpf_counters.sh
>> new file mode 100755
>> index 0000000000000..c0bcb38d6b53c
>> --- /dev/null
>> +++ b/tools/perf/tests/shell/stat_bpf_counters.sh
>> @@ -0,0 +1,34 @@
>> +#!/bin/sh
>> +# perf stat --bpf-counters test
>> +# SPDX-License-Identifier: GPL-2.0
>> +
>> +set -e
>> +
>> +# check whether $2 is within +/- 10% of $1
>> +compare_number()
>> +{
>> +       first_num=$1
>> +       second_num=$2
>> +
>> +       # upper bound is first_num * 110%
>> +       upper=$(( $first_num + $first_num / 10 ))
>> +       # lower bound is first_num * 90%
>> +       lower=$(( $first_num - $first_num / 10 ))
>> +
>> +       if [ $second_num -gt $upper ] || [ $second_num -lt $lower ]; then
>> +               echo "The difference between $first_num and $second_num are greater than 10%."
>> +               exit 1
>> +       fi
>> +}
>> +
>> +# skip if --bpf-counters is not supported
>> +perf stat --bpf-counters true > /dev/null 2>&1 || exit 2
>> +
>> +# skip if stressapptest is not available
>> +stressapptest -s 1 -M 100 -m 1 > /dev/null 2>&1 || exit 2
> 
> I don't know how popular it is, but we can print some info
> in case we miss it.

I just realized that perf-bench-sched-messaging is a good test to use, 
so we don't need stressapptest. Attached the updated version below.

> 
>> +
>> +base_cycles=$(perf stat --no-big-num -e cycles -- stressapptest -s 3 -M 100 -m 1 2>&1 | grep -e cycles | awk '{print $1}')
>> +bpf_cycles=$(perf stat --no-big-num --bpf-counters -e cycles -- stressapptest -s 3 -M 100 -m 1 2>&1 | grep -e cycles | awk '{print $1}')
> 
> I think just awk '/cycles/ {print $1}' should work.

Thanks! Fixed in the new version. 

Song




cat tools/perf/tests/shell/stat_bpf_counters.sh
#!/bin/sh
# perf stat --bpf-counters test
# SPDX-License-Identifier: GPL-2.0

set -e

# check whether $2 is within +/- 10% of $1
compare_number()
{
        first_num=$1
        second_num=$2

        # upper bound is first_num * 110%
        upper=$(( $first_num + $first_num / 10 ))
        # lower bound is first_num * 90%
        lower=$(( $first_num - $first_num / 10 ))

        if [ $second_num -gt $upper ] || [ $second_num -lt $lower ]; then
                echo "The difference between $first_num and $second_num are greater than 10%."
                exit 1
        fi
}

# skip if --bpf-counters is not supported
perf stat --bpf-counters true > /dev/null 2>&1 || exit 2

base_cycles=$(perf stat --no-big-num -e cycles -- perf bench sched messaging -g 1 -l 100 -t 2>&1 | awk '/cycles/ {print $1}')
bpf_cycles=$(perf stat --no-big-num --bpf-counters -e cycles -- perf bench sched messaging -g 1 -l 100 -t 2>&1 | awk '/cycles/ {print $1}')

compare_number $base_cycles $bpf_cycles
exit 0

Re: [PATCH v2 1/3] perf-stat: introduce bperf, share hardware PMCs with BPF

[Namhyung Kim]

On Thu, Mar 18, 2021 at 4:22 PM Song Liu <songliubraving@fb.com> wrote:
>
>
>
> > On Mar 17, 2021, at 10:54 PM, Namhyung Kim <namhyung@kernel.org> wrote:
> >
>
> [...]
>
> >> +
> >> +static int bperf_reload_leader_program(struct evsel *evsel, int attr_map_fd,
> >> +                                      struct perf_event_attr_map_entry *entry)
> >> +{
> >> +       struct bperf_leader_bpf *skel = bperf_leader_bpf__open();
> >> +       int link_fd, diff_map_fd, err;
> >> +       struct bpf_link *link = NULL;
> >> +
> >> +       if (!skel) {
> >> +               pr_err("Failed to open leader skeleton\n");
> >> +               return -1;
> >> +       }
> >> +
> >> +       bpf_map__resize(skel->maps.events, libbpf_num_possible_cpus());
> >> +       err = bperf_leader_bpf__load(skel);
> >> +       if (err) {
> >> +               pr_err("Failed to load leader skeleton\n");
> >> +               goto out;
> >> +       }
> >> +
> >> +       err = -1;
> >> +       link = bpf_program__attach(skel->progs.on_switch);
> >> +       if (!link) {
> >> +               pr_err("Failed to attach leader program\n");
> >> +               goto out;
> >> +       }
> >> +
> >> +       link_fd = bpf_link__fd(link);
> >> +       diff_map_fd = bpf_map__fd(skel->maps.diff_readings);
> >> +       entry->link_id = bpf_link_get_id(link_fd);
> >> +       entry->diff_map_id = bpf_map_get_id(diff_map_fd);
> >> +       err = bpf_map_update_elem(attr_map_fd, &evsel->core.attr, entry, BPF_ANY);
> >> +       assert(err == 0);
> >> +
> >> +       evsel->bperf_leader_link_fd = bpf_link_get_fd_by_id(entry->link_id);
> >> +       assert(evsel->bperf_leader_link_fd >= 0);
> >
> > Isn't it the same as link_fd?
>
> This is a different fd on the same link.

Ok

>
> >
> >> +
> >> +       /*
> >> +        * save leader_skel for install_pe, which is called within
> >> +        * following evsel__open_per_cpu call
> >> +        */
> >> +       evsel->leader_skel = skel;
> >> +       evsel__open_per_cpu(evsel, all_cpu_map, -1);
> >> +
> >> +out:
> >> +       bperf_leader_bpf__destroy(skel);
> >> +       bpf_link__destroy(link);
> >
> > Why do we destroy it?  Is it because we get an another reference?
>
> Yes. We only need evsel->bperf_leader_link_fd to keep the whole
> skeleton attached.
>
> When multiple perf-stat sessions are sharing the leader skeleton,
> only the first one loads the leader skeleton, by calling
> bperf_reload_leader_program(). Other sessions simply hold a fd to
> the bpf_link. More explanation in bperf__load() below.

Ok.

>
>
> >
> >> +       return err;
> >> +}
> >> +
> >> +static int bperf__load(struct evsel *evsel, struct target *target)
> >> +{
> >> +       struct perf_event_attr_map_entry entry = {0xffffffff, 0xffffffff};
> >> +       int attr_map_fd, diff_map_fd = -1, err;
> >> +       enum bperf_filter_type filter_type;
> >> +       __u32 filter_entry_cnt, i;
> >> +
> >> +       if (bperf_check_target(evsel, target, &filter_type, &filter_entry_cnt))
> >> +               return -1;
> >> +
> >> +       if (!all_cpu_map) {
> >> +               all_cpu_map = perf_cpu_map__new(NULL);
> >> +               if (!all_cpu_map)
> >> +                       return -1;
> >> +       }
> >> +
> >> +       evsel->bperf_leader_prog_fd = -1;
> >> +       evsel->bperf_leader_link_fd = -1;
> >> +
> >> +       /*
> >> +        * Step 1: hold a fd on the leader program and the bpf_link, if
> >> +        * the program is not already gone, reload the program.
> >> +        * Use flock() to ensure exclusive access to the perf_event_attr
> >> +        * map.
> >> +        */
> >> +       attr_map_fd = bperf_lock_attr_map(target);
> >> +       if (attr_map_fd < 0) {
> >> +               pr_err("Failed to lock perf_event_attr map\n");
> >> +               return -1;
> >> +       }
> >> +
> >> +       err = bpf_map_lookup_elem(attr_map_fd, &evsel->core.attr, &entry);
> >> +       if (err) {
> >> +               err = bpf_map_update_elem(attr_map_fd, &evsel->core.attr, &entry, BPF_ANY);
> >> +               if (err)
> >> +                       goto out;
> >> +       }
> >> +
> >> +       evsel->bperf_leader_link_fd = bpf_link_get_fd_by_id(entry.link_id);
> >> +       if (evsel->bperf_leader_link_fd < 0 &&
> >> +           bperf_reload_leader_program(evsel, attr_map_fd, &entry))
> >> +               goto out;
>
> Continue with previous explanation. In bperf_reload_leader_program(),
> we open another reference to the link, and destroy the skeleton. This
> brings the code to the same state as evsel->bperf_leader_link_fd >=
> condition above.

Thanks for the explanation.

>
> >> +
> >> +       /*
> >> +        * The bpf_link holds reference to the leader program, and the
> >> +        * leader program holds reference to the maps. Therefore, if
> >> +        * link_id is valid, diff_map_id should also be valid.
> >> +        */
> >> +       evsel->bperf_leader_prog_fd = bpf_prog_get_fd_by_id(
> >> +               bpf_link_get_prog_id(evsel->bperf_leader_link_fd));
> >> +       assert(evsel->bperf_leader_prog_fd >= 0);
> >> +
> >> +       diff_map_fd = bpf_map_get_fd_by_id(entry.diff_map_id);
> >> +       assert(diff_map_fd >= 0);
> >> +
>
> [...]
>
> >> +static int bperf__read(struct evsel *evsel)
> >> +{
> >> +       struct bperf_follower_bpf *skel = evsel->follower_skel;
> >> +       __u32 num_cpu_bpf = cpu__max_cpu();
> >> +       struct bpf_perf_event_value values[num_cpu_bpf];
> >> +       int reading_map_fd, err = 0;
> >> +       __u32 i, j, num_cpu;
> >> +
> >> +       bperf_sync_counters(evsel);
> >> +       reading_map_fd = bpf_map__fd(skel->maps.accum_readings);
> >> +
> >> +       for (i = 0; i < bpf_map__max_entries(skel->maps.accum_readings); i++) {
> >> +               __u32 cpu;
> >> +
> >> +               err = bpf_map_lookup_elem(reading_map_fd, &i, values);
> >> +               if (err)
> >> +                       goto out;
> >> +               switch (evsel->follower_skel->bss->type) {
> >> +               case BPERF_FILTER_GLOBAL:
> >> +                       assert(i == 0);
> >> +
> >> +                       num_cpu = all_cpu_map->nr;
> >> +                       for (j = 0; j < num_cpu; j++) {
> >> +                               cpu = all_cpu_map->map[j];
> >> +                               perf_counts(evsel->counts, cpu, 0)->val = values[cpu].counter;
> >> +                               perf_counts(evsel->counts, cpu, 0)->ena = values[cpu].enabled;
> >> +                               perf_counts(evsel->counts, cpu, 0)->run = values[cpu].running;
> >
> > I'm confused with this.  Does the accum_readings map contain values
> > for all cpus?  IIUC it has only a single entry but you access it for each cpu.
> > What am I missing?
>
> accumulated_reading is a percpu array. In this case, each cpu has its own
> bpf_perf_event_value with index 0. The BPF program could only access the
> data on current cpu. When reading from use space, we get #-of-cpus entries
> for index 0.
>
> Does this make sense?

Yep, I didn't know it returns all values when reading from user space.  Then
I think per cpu event doesn't have many entries too.  Like the global case
it can simply put the value with key 0, no?

Thanks,
Namhyung

Re: [PATCH v2 1/3] perf-stat: introduce bperf, share hardware PMCs with BPF

[Song Liu]

> On Mar 18, 2021, at 6:49 AM, Namhyung Kim <namhyung@kernel.org> wrote:
> 
> On Thu, Mar 18, 2021 at 4:22 PM Song Liu <songliubraving@fb.com> wrote:
>> 
>> 
>> 
>>> On Mar 17, 2021, at 10:54 PM, Namhyung Kim <namhyung@kernel.org> wrote:
>>> 
>> 
>> [...]
>> 
>>>> +
>>>> +static int bperf_reload_leader_program(struct evsel *evsel, int attr_map_fd,
>>>> +                                      struct perf_event_attr_map_entry *entry)
>>>> +{
>>>> +       struct bperf_leader_bpf *skel = bperf_leader_bpf__open();
>>>> +       int link_fd, diff_map_fd, err;
>>>> +       struct bpf_link *link = NULL;
>>>> +
>>>> +       if (!skel) {
>>>> +               pr_err("Failed to open leader skeleton\n");
>>>> +               return -1;
>>>> +       }
>>>> +
>>>> +       bpf_map__resize(skel->maps.events, libbpf_num_possible_cpus());
>>>> +       err = bperf_leader_bpf__load(skel);
>>>> +       if (err) {
>>>> +               pr_err("Failed to load leader skeleton\n");
>>>> +               goto out;
>>>> +       }
>>>> +
>>>> +       err = -1;
>>>> +       link = bpf_program__attach(skel->progs.on_switch);
>>>> +       if (!link) {
>>>> +               pr_err("Failed to attach leader program\n");
>>>> +               goto out;
>>>> +       }
>>>> +
>>>> +       link_fd = bpf_link__fd(link);
>>>> +       diff_map_fd = bpf_map__fd(skel->maps.diff_readings);
>>>> +       entry->link_id = bpf_link_get_id(link_fd);
>>>> +       entry->diff_map_id = bpf_map_get_id(diff_map_fd);
>>>> +       err = bpf_map_update_elem(attr_map_fd, &evsel->core.attr, entry, BPF_ANY);
>>>> +       assert(err == 0);
>>>> +
>>>> +       evsel->bperf_leader_link_fd = bpf_link_get_fd_by_id(entry->link_id);
>>>> +       assert(evsel->bperf_leader_link_fd >= 0);
>>> 
>>> Isn't it the same as link_fd?
>> 
>> This is a different fd on the same link.
> 
> Ok
> 
>> 
>>> 
>>>> +
>>>> +       /*
>>>> +        * save leader_skel for install_pe, which is called within
>>>> +        * following evsel__open_per_cpu call
>>>> +        */
>>>> +       evsel->leader_skel = skel;
>>>> +       evsel__open_per_cpu(evsel, all_cpu_map, -1);
>>>> +
>>>> +out:
>>>> +       bperf_leader_bpf__destroy(skel);
>>>> +       bpf_link__destroy(link);
>>> 
>>> Why do we destroy it?  Is it because we get an another reference?
>> 
>> Yes. We only need evsel->bperf_leader_link_fd to keep the whole
>> skeleton attached.
>> 
>> When multiple perf-stat sessions are sharing the leader skeleton,
>> only the first one loads the leader skeleton, by calling
>> bperf_reload_leader_program(). Other sessions simply hold a fd to
>> the bpf_link. More explanation in bperf__load() below.
> 
> Ok.
> 
>> 
>> 
>>> 
>>>> +       return err;
>>>> +}
>>>> +
>>>> +static int bperf__load(struct evsel *evsel, struct target *target)
>>>> +{
>>>> +       struct perf_event_attr_map_entry entry = {0xffffffff, 0xffffffff};
>>>> +       int attr_map_fd, diff_map_fd = -1, err;
>>>> +       enum bperf_filter_type filter_type;
>>>> +       __u32 filter_entry_cnt, i;
>>>> +
>>>> +       if (bperf_check_target(evsel, target, &filter_type, &filter_entry_cnt))
>>>> +               return -1;
>>>> +
>>>> +       if (!all_cpu_map) {
>>>> +               all_cpu_map = perf_cpu_map__new(NULL);
>>>> +               if (!all_cpu_map)
>>>> +                       return -1;
>>>> +       }
>>>> +
>>>> +       evsel->bperf_leader_prog_fd = -1;
>>>> +       evsel->bperf_leader_link_fd = -1;
>>>> +
>>>> +       /*
>>>> +        * Step 1: hold a fd on the leader program and the bpf_link, if
>>>> +        * the program is not already gone, reload the program.
>>>> +        * Use flock() to ensure exclusive access to the perf_event_attr
>>>> +        * map.
>>>> +        */
>>>> +       attr_map_fd = bperf_lock_attr_map(target);
>>>> +       if (attr_map_fd < 0) {
>>>> +               pr_err("Failed to lock perf_event_attr map\n");
>>>> +               return -1;
>>>> +       }
>>>> +
>>>> +       err = bpf_map_lookup_elem(attr_map_fd, &evsel->core.attr, &entry);
>>>> +       if (err) {
>>>> +               err = bpf_map_update_elem(attr_map_fd, &evsel->core.attr, &entry, BPF_ANY);
>>>> +               if (err)
>>>> +                       goto out;
>>>> +       }
>>>> +
>>>> +       evsel->bperf_leader_link_fd = bpf_link_get_fd_by_id(entry.link_id);
>>>> +       if (evsel->bperf_leader_link_fd < 0 &&
>>>> +           bperf_reload_leader_program(evsel, attr_map_fd, &entry))
>>>> +               goto out;
>> 
>> Continue with previous explanation. In bperf_reload_leader_program(),
>> we open another reference to the link, and destroy the skeleton. This
>> brings the code to the same state as evsel->bperf_leader_link_fd >=
>> condition above.
> 
> Thanks for the explanation.
> 
>> 
>>>> +
>>>> +       /*
>>>> +        * The bpf_link holds reference to the leader program, and the
>>>> +        * leader program holds reference to the maps. Therefore, if
>>>> +        * link_id is valid, diff_map_id should also be valid.
>>>> +        */
>>>> +       evsel->bperf_leader_prog_fd = bpf_prog_get_fd_by_id(
>>>> +               bpf_link_get_prog_id(evsel->bperf_leader_link_fd));
>>>> +       assert(evsel->bperf_leader_prog_fd >= 0);
>>>> +
>>>> +       diff_map_fd = bpf_map_get_fd_by_id(entry.diff_map_id);
>>>> +       assert(diff_map_fd >= 0);
>>>> +
>> 
>> [...]
>> 
>>>> +static int bperf__read(struct evsel *evsel)
>>>> +{
>>>> +       struct bperf_follower_bpf *skel = evsel->follower_skel;
>>>> +       __u32 num_cpu_bpf = cpu__max_cpu();
>>>> +       struct bpf_perf_event_value values[num_cpu_bpf];
>>>> +       int reading_map_fd, err = 0;
>>>> +       __u32 i, j, num_cpu;
>>>> +
>>>> +       bperf_sync_counters(evsel);
>>>> +       reading_map_fd = bpf_map__fd(skel->maps.accum_readings);
>>>> +
>>>> +       for (i = 0; i < bpf_map__max_entries(skel->maps.accum_readings); i++) {
>>>> +               __u32 cpu;
>>>> +
>>>> +               err = bpf_map_lookup_elem(reading_map_fd, &i, values);
>>>> +               if (err)
>>>> +                       goto out;
>>>> +               switch (evsel->follower_skel->bss->type) {
>>>> +               case BPERF_FILTER_GLOBAL:
>>>> +                       assert(i == 0);
>>>> +
>>>> +                       num_cpu = all_cpu_map->nr;
>>>> +                       for (j = 0; j < num_cpu; j++) {
>>>> +                               cpu = all_cpu_map->map[j];
>>>> +                               perf_counts(evsel->counts, cpu, 0)->val = values[cpu].counter;
>>>> +                               perf_counts(evsel->counts, cpu, 0)->ena = values[cpu].enabled;
>>>> +                               perf_counts(evsel->counts, cpu, 0)->run = values[cpu].running;
>>> 
>>> I'm confused with this.  Does the accum_readings map contain values
>>> for all cpus?  IIUC it has only a single entry but you access it for each cpu.
>>> What am I missing?
>> 
>> accumulated_reading is a percpu array. In this case, each cpu has its own
>> bpf_perf_event_value with index 0. The BPF program could only access the
>> data on current cpu. When reading from use space, we get #-of-cpus entries
>> for index 0.
>> 
>> Does this make sense?
> 
> Yep, I didn't know it returns all values when reading from user space.  Then
> I think per cpu event doesn't have many entries too.  Like the global case
> it can simply put the value with key 0, no?

Current per cpu event use same logic as per task events, so we do have multiple
entries. I think it is possible to modify the logic to use one entry for per
cpu events. 

Thanks,
Song

Re: [PATCH v2 0/3] perf-stat: share hardware PMCs with BPF

[Jiri Olsa]

On Thu, Mar 18, 2021 at 03:52:51AM +0000, Song Liu wrote:
> 
> 
> > On Mar 17, 2021, at 6:11 AM, Arnaldo Carvalho de Melo <acme@kernel.org> wrote:
> > 
> > Em Wed, Mar 17, 2021 at 02:29:28PM +0900, Namhyung Kim escreveu:
> >> Hi Song,
> >> 
> >> On Wed, Mar 17, 2021 at 6:18 AM Song Liu <songliubraving@fb.com> wrote:
> >>> 
> >>> perf uses performance monitoring counters (PMCs) to monitor system
> >>> performance. The PMCs are limited hardware resources. For example,
> >>> Intel CPUs have 3x fixed PMCs and 4x programmable PMCs per cpu.
> >>> 
> >>> Modern data center systems use these PMCs in many different ways:
> >>> system level monitoring, (maybe nested) container level monitoring, per
> >>> process monitoring, profiling (in sample mode), etc. In some cases,
> >>> there are more active perf_events than available hardware PMCs. To allow
> >>> all perf_events to have a chance to run, it is necessary to do expensive
> >>> time multiplexing of events.
> >>> 
> >>> On the other hand, many monitoring tools count the common metrics (cycles,
> >>> instructions). It is a waste to have multiple tools create multiple
> >>> perf_events of "cycles" and occupy multiple PMCs.
> >> 
> >> Right, it'd be really helpful when the PMCs are frequently or mostly shared.
> >> But it'd also increase the overhead for uncontended cases as BPF programs
> >> need to run on every context switch.  Depending on the workload, it may
> >> cause a non-negligible performance impact.  So users should be aware of it.
> > 
> > Would be interesting to, humm, measure both cases to have a firm number
> > of the impact, how many instructions are added when sharing using
> > --bpf-counters?
> > 
> > I.e. compare the "expensive time multiplexing of events" with its
> > avoidance by using --bpf-counters.
> > 
> > Song, have you perfmormed such measurements?
> 
> I have got some measurements with perf-bench-sched-messaging:
> 
> The system: x86_64 with 23 cores (46 HT)
> 
> The perf-stat command:
> perf stat -e cycles,cycles,instructions,instructions,ref-cycles,ref-cycles <target, etc.>
> 
> The benchmark command and output:
> ./perf bench sched messaging -g 40 -l 50000 -t
> # Running 'sched/messaging' benchmark:
> # 20 sender and receiver threads per group
> # 40 groups == 1600 threads run
>      Total time: 10X.XXX [sec]
> 
> 
> I use the "Total time" as measurement, so smaller number is better. 
> 
> For each condition, I run the command 5 times, and took the median of 
> "Total time". 
> 
> Baseline (no perf-stat)			104.873 [sec]
> # global
> perf stat -a				107.887 [sec]
> perf stat -a --bpf-counters		106.071 [sec]
> # per task
> perf stat 				106.314 [sec]
> perf stat --bpf-counters 		105.965 [sec]
> # per cpu
> perf stat -C 1,3,5 			107.063 [sec]
> perf stat -C 1,3,5 --bpf-counters 	106.406 [sec]

I can't see why it's actualy faster than normal perf ;-)
would be worth to find out

jirka

> 
> From the data, --bpf-counters is slightly better than the regular event
> for all targets. I noticed that the results are not very stable. There 
> are a couple 108.xx runs in some of the conditions (w/ and w/o 
> --bpf-counters).
> 
> 
> I also measured the average runtime of the BPF programs, with 
> 
> 	sysctl kernel.bpf_stats_enabled=1
> 
> For each event, if we have one leader and two followers, the total run 
> time is about 340ns. IOW, 340ns for two perf-stat reading instructions, 
> 340ns for two perf-stat reading cycles, etc. 
> 
> Thanks,
> Song
> 

Re: [PATCH v2 1/3] perf-stat: introduce bperf, share hardware PMCs with BPF

[Jiri Olsa]

On Tue, Mar 16, 2021 at 02:18:35PM -0700, Song Liu wrote:
> perf uses performance monitoring counters (PMCs) to monitor system
> performance. The PMCs are limited hardware resources. For example,
> Intel CPUs have 3x fixed PMCs and 4x programmable PMCs per cpu.
> 
> Modern data center systems use these PMCs in many different ways:
> system level monitoring, (maybe nested) container level monitoring, per
> process monitoring, profiling (in sample mode), etc. In some cases,
> there are more active perf_events than available hardware PMCs. To allow
> all perf_events to have a chance to run, it is necessary to do expensive
> time multiplexing of events.
> 
> On the other hand, many monitoring tools count the common metrics (cycles,
> instructions). It is a waste to have multiple tools create multiple
> perf_events of "cycles" and occupy multiple PMCs.
> 
> bperf tries to reduce such wastes by allowing multiple perf_events of
> "cycles" or "instructions" (at different scopes) to share PMUs. Instead
> of having each perf-stat session to read its own perf_events, bperf uses
> BPF programs to read the perf_events and aggregate readings to BPF maps.
> Then, the perf-stat session(s) reads the values from these BPF maps.
> 
> Please refer to the comment before the definition of bperf_ops for the
> description of bperf architecture.
> 
> bperf is off by default. To enable it, pass --bpf-counters option to
> perf-stat. bperf uses a BPF hashmap to share information about BPF
> programs and maps used by bperf. This map is pinned to bpffs. The default
> path is /sys/fs/bpf/perf_attr_map. The user could change the path with
> option --bpf-attr-map.
> 
> Signed-off-by: Song Liu <songliubraving@fb.com>

Reviewed-by: Jiri Olsa <jolsa@redhat.com>

thanks,
jirka

> 
> ---
> Known limitations:
> 1. Do not support per cgroup events;
> 2. Do not support monitoring of BPF program (perf-stat -b);
> 3. Do not support event groups;
> 4. Do not support inherit events during fork().
> 
> The following commands have been tested:
> 
>    perf stat --bpf-counters -e cycles,ref-cycles -a
>    perf stat --bpf-counters -e cycles,instructions -C 1,3,4
>    perf stat --bpf-counters -e cycles -p 123
>    perf stat --bpf-counters -e cycles -t 100,101
>    perf stat --bpf-counters -e cycles,ref-cycles -- stressapptest ...
> ---
>  tools/perf/Documentation/perf-stat.txt        |  11 +
>  tools/perf/Makefile.perf                      |   1 +
>  tools/perf/builtin-stat.c                     |  10 +
>  tools/perf/util/bpf_counter.c                 | 519 +++++++++++++++++-
>  tools/perf/util/bpf_skel/bperf.h              |  14 +
>  tools/perf/util/bpf_skel/bperf_follower.bpf.c |  69 +++
>  tools/perf/util/bpf_skel/bperf_leader.bpf.c   |  46 ++
>  tools/perf/util/bpf_skel/bperf_u.h            |  14 +
>  tools/perf/util/evsel.h                       |  20 +-
>  tools/perf/util/target.h                      |   4 +-
>  10 files changed, 701 insertions(+), 7 deletions(-)
>  create mode 100644 tools/perf/util/bpf_skel/bperf.h
>  create mode 100644 tools/perf/util/bpf_skel/bperf_follower.bpf.c
>  create mode 100644 tools/perf/util/bpf_skel/bperf_leader.bpf.c
>  create mode 100644 tools/perf/util/bpf_skel/bperf_u.h
> 
> diff --git a/tools/perf/Documentation/perf-stat.txt b/tools/perf/Documentation/perf-stat.txt
> index 08a1714494f87..d2e7656b5ef81 100644
> --- a/tools/perf/Documentation/perf-stat.txt
> +++ b/tools/perf/Documentation/perf-stat.txt
> @@ -93,6 +93,17 @@ report::
>  
>          1.102235068 seconds time elapsed
>  
> +--bpf-counters::
> +	Use BPF programs to aggregate readings from perf_events.  This
> +	allows multiple perf-stat sessions that are counting the same metric (cycles,
> +	instructions, etc.) to share hardware counters.
> +
> +--bpf-attr-map::
> +	With option "--bpf-counters", different perf-stat sessions share
> +	information about shared BPF programs and maps via a pinned hashmap.
> +	Use "--bpf-attr-map" to specify the path of this pinned hashmap.
> +	The default path is /sys/fs/bpf/perf_attr_map.
> +
>  ifdef::HAVE_LIBPFM[]
>  --pfm-events events::
>  Select a PMU event using libpfm4 syntax (see http://perfmon2.sf.net)
> diff --git a/tools/perf/Makefile.perf b/tools/perf/Makefile.perf
> index f6e609673de2b..ca9aa08e85a1f 100644
> --- a/tools/perf/Makefile.perf
> +++ b/tools/perf/Makefile.perf
> @@ -1007,6 +1007,7 @@ python-clean:
>  SKEL_OUT := $(abspath $(OUTPUT)util/bpf_skel)
>  SKEL_TMP_OUT := $(abspath $(SKEL_OUT)/.tmp)
>  SKELETONS := $(SKEL_OUT)/bpf_prog_profiler.skel.h
> +SKELETONS += $(SKEL_OUT)/bperf_leader.skel.h $(SKEL_OUT)/bperf_follower.skel.h
>  
>  ifdef BUILD_BPF_SKEL
>  BPFTOOL := $(SKEL_TMP_OUT)/bootstrap/bpftool
> diff --git a/tools/perf/builtin-stat.c b/tools/perf/builtin-stat.c
> index 2e2e4a8345ea2..92696373da994 100644
> --- a/tools/perf/builtin-stat.c
> +++ b/tools/perf/builtin-stat.c
> @@ -792,6 +792,12 @@ static int __run_perf_stat(int argc, const char **argv, int run_idx)
>  	}
>  
>  	evlist__for_each_cpu (evsel_list, i, cpu) {
> +		/*
> +		 * bperf calls evsel__open_per_cpu() in bperf__load(), so
> +		 * no need to call it again here.
> +		 */
> +		if (target.use_bpf)
> +			break;
>  		affinity__set(&affinity, cpu);
>  
>  		evlist__for_each_entry(evsel_list, counter) {
> @@ -1146,6 +1152,10 @@ static struct option stat_options[] = {
>  #ifdef HAVE_BPF_SKEL
>  	OPT_STRING('b', "bpf-prog", &target.bpf_str, "bpf-prog-id",
>  		   "stat events on existing bpf program id"),
> +	OPT_BOOLEAN(0, "bpf-counters", &target.use_bpf,
> +		    "use bpf program to count events"),
> +	OPT_STRING(0, "bpf-attr-map", &target.attr_map, "attr-map-path",
> +		   "path to perf_event_attr map"),
>  #endif
>  	OPT_BOOLEAN('a', "all-cpus", &target.system_wide,
>  		    "system-wide collection from all CPUs"),
> diff --git a/tools/perf/util/bpf_counter.c b/tools/perf/util/bpf_counter.c
> index 04f89120b3232..81d1df3c4ec0e 100644
> --- a/tools/perf/util/bpf_counter.c
> +++ b/tools/perf/util/bpf_counter.c
> @@ -5,6 +5,7 @@
>  #include <assert.h>
>  #include <limits.h>
>  #include <unistd.h>
> +#include <sys/file.h>
>  #include <sys/time.h>
>  #include <sys/resource.h>
>  #include <linux/err.h>
> @@ -12,14 +13,45 @@
>  #include <bpf/bpf.h>
>  #include <bpf/btf.h>
>  #include <bpf/libbpf.h>
> +#include <api/fs/fs.h>
>  
>  #include "bpf_counter.h"
>  #include "counts.h"
>  #include "debug.h"
>  #include "evsel.h"
> +#include "evlist.h"
>  #include "target.h"
> +#include "cpumap.h"
> +#include "thread_map.h"
>  
>  #include "bpf_skel/bpf_prog_profiler.skel.h"
> +#include "bpf_skel/bperf_u.h"
> +#include "bpf_skel/bperf_leader.skel.h"
> +#include "bpf_skel/bperf_follower.skel.h"
> +
> +/*
> + * bperf uses a hashmap, the attr_map, to track all the leader programs.
> + * The hashmap is pinned in bpffs. flock() on this file is used to ensure
> + * no concurrent access to the attr_map.  The key of attr_map is struct
> + * perf_event_attr, and the value is struct perf_event_attr_map_entry.
> + *
> + * struct perf_event_attr_map_entry contains two __u32 IDs, bpf_link of the
> + * leader prog, and the diff_map. Each perf-stat session holds a reference
> + * to the bpf_link to make sure the leader prog is attached to sched_switch
> + * tracepoint.
> + *
> + * Since the hashmap only contains IDs of the bpf_link and diff_map, it
> + * does not hold any references to the leader program. Once all perf-stat
> + * sessions of these events exit, the leader prog, its maps, and the
> + * perf_events will be freed.
> + */
> +struct perf_event_attr_map_entry {
> +	__u32 link_id;
> +	__u32 diff_map_id;
> +};
> +
> +#define DEFAULT_ATTR_MAP_PATH "fs/bpf/perf_attr_map"
> +#define ATTR_MAP_SIZE 16
>  
>  static inline void *u64_to_ptr(__u64 ptr)
>  {
> @@ -274,17 +306,494 @@ struct bpf_counter_ops bpf_program_profiler_ops = {
>  	.install_pe = bpf_program_profiler__install_pe,
>  };
>  
> +static __u32 bpf_link_get_id(int fd)
> +{
> +	struct bpf_link_info link_info = {0};
> +	__u32 link_info_len = sizeof(link_info);
> +
> +	bpf_obj_get_info_by_fd(fd, &link_info, &link_info_len);
> +	return link_info.id;
> +}
> +
> +static __u32 bpf_link_get_prog_id(int fd)
> +{
> +	struct bpf_link_info link_info = {0};
> +	__u32 link_info_len = sizeof(link_info);
> +
> +	bpf_obj_get_info_by_fd(fd, &link_info, &link_info_len);
> +	return link_info.prog_id;
> +}
> +
> +static __u32 bpf_map_get_id(int fd)
> +{
> +	struct bpf_map_info map_info = {0};
> +	__u32 map_info_len = sizeof(map_info);
> +
> +	bpf_obj_get_info_by_fd(fd, &map_info, &map_info_len);
> +	return map_info.id;
> +}
> +
> +static int bperf_lock_attr_map(struct target *target)
> +{
> +	char path[PATH_MAX];
> +	int map_fd, err;
> +
> +	if (target->attr_map) {
> +		scnprintf(path, PATH_MAX, "%s", target->attr_map);
> +	} else {
> +		scnprintf(path, PATH_MAX, "%s/%s", sysfs__mountpoint(),
> +			  DEFAULT_ATTR_MAP_PATH);
> +	}
> +
> +	if (access(path, F_OK)) {
> +		map_fd = bpf_create_map(BPF_MAP_TYPE_HASH,
> +					sizeof(struct perf_event_attr),
> +					sizeof(struct perf_event_attr_map_entry),
> +					ATTR_MAP_SIZE, 0);
> +		if (map_fd < 0)
> +			return -1;
> +
> +		err = bpf_obj_pin(map_fd, path);
> +		if (err) {
> +			/* someone pinned the map in parallel? */
> +			close(map_fd);
> +			map_fd = bpf_obj_get(path);
> +			if (map_fd < 0)
> +				return -1;
> +		}
> +	} else {
> +		map_fd = bpf_obj_get(path);
> +		if (map_fd < 0)
> +			return -1;
> +	}
> +
> +	err = flock(map_fd, LOCK_EX);
> +	if (err) {
> +		close(map_fd);
> +		return -1;
> +	}
> +	return map_fd;
> +}
> +
> +/* trigger the leader program on a cpu */
> +static int bperf_trigger_reading(int prog_fd, int cpu)
> +{
> +	DECLARE_LIBBPF_OPTS(bpf_test_run_opts, opts,
> +			    .ctx_in = NULL,
> +			    .ctx_size_in = 0,
> +			    .flags = BPF_F_TEST_RUN_ON_CPU,
> +			    .cpu = cpu,
> +			    .retval = 0,
> +		);
> +
> +	return bpf_prog_test_run_opts(prog_fd, &opts);
> +}
> +
> +static int bperf_check_target(struct evsel *evsel,
> +			      struct target *target,
> +			      enum bperf_filter_type *filter_type,
> +			      __u32 *filter_entry_cnt)
> +{
> +	if (evsel->leader->core.nr_members > 1) {
> +		pr_err("bpf managed perf events do not yet support groups.\n");
> +		return -1;
> +	}
> +
> +	/* determine filter type based on target */
> +	if (target->system_wide) {
> +		*filter_type = BPERF_FILTER_GLOBAL;
> +		*filter_entry_cnt = 1;
> +	} else if (target->cpu_list) {
> +		*filter_type = BPERF_FILTER_CPU;
> +		*filter_entry_cnt = perf_cpu_map__nr(evsel__cpus(evsel));
> +	} else if (target->tid) {
> +		*filter_type = BPERF_FILTER_PID;
> +		*filter_entry_cnt = perf_thread_map__nr(evsel->core.threads);
> +	} else if (target->pid || evsel->evlist->workload.pid != -1) {
> +		*filter_type = BPERF_FILTER_TGID;
> +		*filter_entry_cnt = perf_thread_map__nr(evsel->core.threads);
> +	} else {
> +		pr_err("bpf managed perf events do not yet support these targets.\n");
> +		return -1;
> +	}
> +
> +	return 0;
> +}
> +
> +static	struct perf_cpu_map *all_cpu_map;
> +
> +static int bperf_reload_leader_program(struct evsel *evsel, int attr_map_fd,
> +				       struct perf_event_attr_map_entry *entry)
> +{
> +	struct bperf_leader_bpf *skel = bperf_leader_bpf__open();
> +	int link_fd, diff_map_fd, err;
> +	struct bpf_link *link = NULL;
> +
> +	if (!skel) {
> +		pr_err("Failed to open leader skeleton\n");
> +		return -1;
> +	}
> +
> +	bpf_map__resize(skel->maps.events, libbpf_num_possible_cpus());
> +	err = bperf_leader_bpf__load(skel);
> +	if (err) {
> +		pr_err("Failed to load leader skeleton\n");
> +		goto out;
> +	}
> +
> +	err = -1;
> +	link = bpf_program__attach(skel->progs.on_switch);
> +	if (!link) {
> +		pr_err("Failed to attach leader program\n");
> +		goto out;
> +	}
> +
> +	link_fd = bpf_link__fd(link);
> +	diff_map_fd = bpf_map__fd(skel->maps.diff_readings);
> +	entry->link_id = bpf_link_get_id(link_fd);
> +	entry->diff_map_id = bpf_map_get_id(diff_map_fd);
> +	err = bpf_map_update_elem(attr_map_fd, &evsel->core.attr, entry, BPF_ANY);
> +	assert(err == 0);
> +
> +	evsel->bperf_leader_link_fd = bpf_link_get_fd_by_id(entry->link_id);
> +	assert(evsel->bperf_leader_link_fd >= 0);
> +
> +	/*
> +	 * save leader_skel for install_pe, which is called within
> +	 * following evsel__open_per_cpu call
> +	 */
> +	evsel->leader_skel = skel;
> +	evsel__open_per_cpu(evsel, all_cpu_map, -1);
> +
> +out:
> +	bperf_leader_bpf__destroy(skel);
> +	bpf_link__destroy(link);
> +	return err;
> +}
> +
> +static int bperf__load(struct evsel *evsel, struct target *target)
> +{
> +	struct perf_event_attr_map_entry entry = {0xffffffff, 0xffffffff};
> +	int attr_map_fd, diff_map_fd = -1, err;
> +	enum bperf_filter_type filter_type;
> +	__u32 filter_entry_cnt, i;
> +
> +	if (bperf_check_target(evsel, target, &filter_type, &filter_entry_cnt))
> +		return -1;
> +
> +	if (!all_cpu_map) {
> +		all_cpu_map = perf_cpu_map__new(NULL);
> +		if (!all_cpu_map)
> +			return -1;
> +	}
> +
> +	evsel->bperf_leader_prog_fd = -1;
> +	evsel->bperf_leader_link_fd = -1;
> +
> +	/*
> +	 * Step 1: hold a fd on the leader program and the bpf_link, if
> +	 * the program is not already gone, reload the program.
> +	 * Use flock() to ensure exclusive access to the perf_event_attr
> +	 * map.
> +	 */
> +	attr_map_fd = bperf_lock_attr_map(target);
> +	if (attr_map_fd < 0) {
> +		pr_err("Failed to lock perf_event_attr map\n");
> +		return -1;
> +	}
> +
> +	err = bpf_map_lookup_elem(attr_map_fd, &evsel->core.attr, &entry);
> +	if (err) {
> +		err = bpf_map_update_elem(attr_map_fd, &evsel->core.attr, &entry, BPF_ANY);
> +		if (err)
> +			goto out;
> +	}
> +
> +	evsel->bperf_leader_link_fd = bpf_link_get_fd_by_id(entry.link_id);
> +	if (evsel->bperf_leader_link_fd < 0 &&
> +	    bperf_reload_leader_program(evsel, attr_map_fd, &entry))
> +		goto out;
> +
> +	/*
> +	 * The bpf_link holds reference to the leader program, and the
> +	 * leader program holds reference to the maps. Therefore, if
> +	 * link_id is valid, diff_map_id should also be valid.
> +	 */
> +	evsel->bperf_leader_prog_fd = bpf_prog_get_fd_by_id(
> +		bpf_link_get_prog_id(evsel->bperf_leader_link_fd));
> +	assert(evsel->bperf_leader_prog_fd >= 0);
> +
> +	diff_map_fd = bpf_map_get_fd_by_id(entry.diff_map_id);
> +	assert(diff_map_fd >= 0);
> +
> +	/*
> +	 * bperf uses BPF_PROG_TEST_RUN to get accurate reading. Check
> +	 * whether the kernel support it
> +	 */
> +	err = bperf_trigger_reading(evsel->bperf_leader_prog_fd, 0);
> +	if (err) {
> +		pr_err("The kernel does not support test_run for raw_tp BPF programs.\n"
> +		       "Therefore, --use-bpf might show inaccurate readings\n");
> +		goto out;
> +	}
> +
> +	/* Step 2: load the follower skeleton */
> +	evsel->follower_skel = bperf_follower_bpf__open();
> +	if (!evsel->follower_skel) {
> +		pr_err("Failed to open follower skeleton\n");
> +		goto out;
> +	}
> +
> +	/* attach fexit program to the leader program */
> +	bpf_program__set_attach_target(evsel->follower_skel->progs.fexit_XXX,
> +				       evsel->bperf_leader_prog_fd, "on_switch");
> +
> +	/* connect to leader diff_reading map */
> +	bpf_map__reuse_fd(evsel->follower_skel->maps.diff_readings, diff_map_fd);
> +
> +	/* set up reading map */
> +	bpf_map__set_max_entries(evsel->follower_skel->maps.accum_readings,
> +				 filter_entry_cnt);
> +	/* set up follower filter based on target */
> +	bpf_map__set_max_entries(evsel->follower_skel->maps.filter,
> +				 filter_entry_cnt);
> +	err = bperf_follower_bpf__load(evsel->follower_skel);
> +	if (err) {
> +		pr_err("Failed to load follower skeleton\n");
> +		bperf_follower_bpf__destroy(evsel->follower_skel);
> +		evsel->follower_skel = NULL;
> +		goto out;
> +	}
> +
> +	for (i = 0; i < filter_entry_cnt; i++) {
> +		int filter_map_fd;
> +		__u32 key;
> +
> +		if (filter_type == BPERF_FILTER_PID ||
> +		    filter_type == BPERF_FILTER_TGID)
> +			key = evsel->core.threads->map[i].pid;
> +		else if (filter_type == BPERF_FILTER_CPU)
> +			key = evsel->core.cpus->map[i];
> +		else
> +			break;
> +
> +		filter_map_fd = bpf_map__fd(evsel->follower_skel->maps.filter);
> +		bpf_map_update_elem(filter_map_fd, &key, &i, BPF_ANY);
> +	}
> +
> +	evsel->follower_skel->bss->type = filter_type;
> +
> +	err = bperf_follower_bpf__attach(evsel->follower_skel);
> +
> +out:
> +	if (err && evsel->bperf_leader_link_fd >= 0)
> +		close(evsel->bperf_leader_link_fd);
> +	if (err && evsel->bperf_leader_prog_fd >= 0)
> +		close(evsel->bperf_leader_prog_fd);
> +	if (diff_map_fd >= 0)
> +		close(diff_map_fd);
> +
> +	flock(attr_map_fd, LOCK_UN);
> +	close(attr_map_fd);
> +
> +	return err;
> +}
> +
> +static int bperf__install_pe(struct evsel *evsel, int cpu, int fd)
> +{
> +	struct bperf_leader_bpf *skel = evsel->leader_skel;
> +
> +	return bpf_map_update_elem(bpf_map__fd(skel->maps.events),
> +				   &cpu, &fd, BPF_ANY);
> +}
> +
> +/*
> + * trigger the leader prog on each cpu, so the accum_reading map could get
> + * the latest readings.
> + */
> +static int bperf_sync_counters(struct evsel *evsel)
> +{
> +	int num_cpu, i, cpu;
> +
> +	num_cpu = all_cpu_map->nr;
> +	for (i = 0; i < num_cpu; i++) {
> +		cpu = all_cpu_map->map[i];
> +		bperf_trigger_reading(evsel->bperf_leader_prog_fd, cpu);
> +	}
> +	return 0;
> +}
> +
> +static int bperf__enable(struct evsel *evsel)
> +{
> +	evsel->follower_skel->bss->enabled = 1;
> +	return 0;
> +}
> +
> +static int bperf__read(struct evsel *evsel)
> +{
> +	struct bperf_follower_bpf *skel = evsel->follower_skel;
> +	__u32 num_cpu_bpf = cpu__max_cpu();
> +	struct bpf_perf_event_value values[num_cpu_bpf];
> +	int reading_map_fd, err = 0;
> +	__u32 i, j, num_cpu;
> +
> +	bperf_sync_counters(evsel);
> +	reading_map_fd = bpf_map__fd(skel->maps.accum_readings);
> +
> +	for (i = 0; i < bpf_map__max_entries(skel->maps.accum_readings); i++) {
> +		__u32 cpu;
> +
> +		err = bpf_map_lookup_elem(reading_map_fd, &i, values);
> +		if (err)
> +			goto out;
> +		switch (evsel->follower_skel->bss->type) {
> +		case BPERF_FILTER_GLOBAL:
> +			assert(i == 0);
> +
> +			num_cpu = all_cpu_map->nr;
> +			for (j = 0; j < num_cpu; j++) {
> +				cpu = all_cpu_map->map[j];
> +				perf_counts(evsel->counts, cpu, 0)->val = values[cpu].counter;
> +				perf_counts(evsel->counts, cpu, 0)->ena = values[cpu].enabled;
> +				perf_counts(evsel->counts, cpu, 0)->run = values[cpu].running;
> +			}
> +			break;
> +		case BPERF_FILTER_CPU:
> +			cpu = evsel->core.cpus->map[i];
> +			perf_counts(evsel->counts, i, 0)->val = values[cpu].counter;
> +			perf_counts(evsel->counts, i, 0)->ena = values[cpu].enabled;
> +			perf_counts(evsel->counts, i, 0)->run = values[cpu].running;
> +			break;
> +		case BPERF_FILTER_PID:
> +		case BPERF_FILTER_TGID:
> +			perf_counts(evsel->counts, 0, i)->val = 0;
> +			perf_counts(evsel->counts, 0, i)->ena = 0;
> +			perf_counts(evsel->counts, 0, i)->run = 0;
> +
> +			for (cpu = 0; cpu < num_cpu_bpf; cpu++) {
> +				perf_counts(evsel->counts, 0, i)->val += values[cpu].counter;
> +				perf_counts(evsel->counts, 0, i)->ena += values[cpu].enabled;
> +				perf_counts(evsel->counts, 0, i)->run += values[cpu].running;
> +			}
> +			break;
> +		default:
> +			break;
> +		}
> +	}
> +out:
> +	return err;
> +}
> +
> +static int bperf__destroy(struct evsel *evsel)
> +{
> +	bperf_follower_bpf__destroy(evsel->follower_skel);
> +	close(evsel->bperf_leader_prog_fd);
> +	close(evsel->bperf_leader_link_fd);
> +	return 0;
> +}
> +
> +/*
> + * bperf: share hardware PMCs with BPF
> + *
> + * perf uses performance monitoring counters (PMC) to monitor system
> + * performance. The PMCs are limited hardware resources. For example,
> + * Intel CPUs have 3x fixed PMCs and 4x programmable PMCs per cpu.
> + *
> + * Modern data center systems use these PMCs in many different ways:
> + * system level monitoring, (maybe nested) container level monitoring, per
> + * process monitoring, profiling (in sample mode), etc. In some cases,
> + * there are more active perf_events than available hardware PMCs. To allow
> + * all perf_events to have a chance to run, it is necessary to do expensive
> + * time multiplexing of events.
> + *
> + * On the other hand, many monitoring tools count the common metrics
> + * (cycles, instructions). It is a waste to have multiple tools create
> + * multiple perf_events of "cycles" and occupy multiple PMCs.
> + *
> + * bperf tries to reduce such wastes by allowing multiple perf_events of
> + * "cycles" or "instructions" (at different scopes) to share PMUs. Instead
> + * of having each perf-stat session to read its own perf_events, bperf uses
> + * BPF programs to read the perf_events and aggregate readings to BPF maps.
> + * Then, the perf-stat session(s) reads the values from these BPF maps.
> + *
> + *                                ||
> + *       shared progs and maps <- || -> per session progs and maps
> + *                                ||
> + *   ---------------              ||
> + *   | perf_events |              ||
> + *   ---------------       fexit  ||      -----------------
> + *          |             --------||----> | follower prog |
> + *       --------------- /        || ---  -----------------
> + * cs -> | leader prog |/         ||/        |         |
> + *   --> ---------------         /||  --------------  ------------------
> + *  /       |         |         / ||  | filter map |  | accum_readings |
> + * /  ------------  ------------  ||  --------------  ------------------
> + * |  | prev map |  | diff map |  ||                        |
> + * |  ------------  ------------  ||                        |
> + *  \                             ||                        |
> + * = \ ==================================================== | ============
> + *    \                                                    /   user space
> + *     \                                                  /
> + *      \                                                /
> + *    BPF_PROG_TEST_RUN                    BPF_MAP_LOOKUP_ELEM
> + *        \                                            /
> + *         \                                          /
> + *          \------  perf-stat ----------------------/
> + *
> + * The figure above shows the architecture of bperf. Note that the figure
> + * is divided into 3 regions: shared progs and maps (top left), per session
> + * progs and maps (top right), and user space (bottom).
> + *
> + * The leader prog is triggered on each context switch (cs). The leader
> + * prog reads perf_events and stores the difference (current_reading -
> + * previous_reading) to the diff map. For the same metric, e.g. "cycles",
> + * multiple perf-stat sessions share the same leader prog.
> + *
> + * Each perf-stat session creates a follower prog as fexit program to the
> + * leader prog. It is possible to attach up to BPF_MAX_TRAMP_PROGS (38)
> + * follower progs to the same leader prog. The follower prog checks current
> + * task and processor ID to decide whether to add the value from the diff
> + * map to its accumulated reading map (accum_readings).
> + *
> + * Finally, perf-stat user space reads the value from accum_reading map.
> + *
> + * Besides context switch, it is also necessary to trigger the leader prog
> + * before perf-stat reads the value. Otherwise, the accum_reading map may
> + * not have the latest reading from the perf_events. This is achieved by
> + * triggering the event via sys_bpf(BPF_PROG_TEST_RUN) to each CPU.
> + *
> + * Comment before the definition of struct perf_event_attr_map_entry
> + * describes how different sessions of perf-stat share information about
> + * the leader prog.
> + */
> +
> +struct bpf_counter_ops bperf_ops = {
> +	.load       = bperf__load,
> +	.enable     = bperf__enable,
> +	.read       = bperf__read,
> +	.install_pe = bperf__install_pe,
> +	.destroy    = bperf__destroy,
> +};
> +
> +static inline bool bpf_counter_skip(struct evsel *evsel)
> +{
> +	return list_empty(&evsel->bpf_counter_list) &&
> +		evsel->follower_skel == NULL;
> +}
> +
>  int bpf_counter__install_pe(struct evsel *evsel, int cpu, int fd)
>  {
> -	if (list_empty(&evsel->bpf_counter_list))
> +	if (bpf_counter_skip(evsel))
>  		return 0;
>  	return evsel->bpf_counter_ops->install_pe(evsel, cpu, fd);
>  }
>  
>  int bpf_counter__load(struct evsel *evsel, struct target *target)
>  {
> -	if (target__has_bpf(target))
> +	if (target->bpf_str)
>  		evsel->bpf_counter_ops = &bpf_program_profiler_ops;
> +	else if (target->use_bpf)
> +		evsel->bpf_counter_ops = &bperf_ops;
>  
>  	if (evsel->bpf_counter_ops)
>  		return evsel->bpf_counter_ops->load(evsel, target);
> @@ -293,21 +802,21 @@ int bpf_counter__load(struct evsel *evsel, struct target *target)
>  
>  int bpf_counter__enable(struct evsel *evsel)
>  {
> -	if (list_empty(&evsel->bpf_counter_list))
> +	if (bpf_counter_skip(evsel))
>  		return 0;
>  	return evsel->bpf_counter_ops->enable(evsel);
>  }
>  
>  int bpf_counter__read(struct evsel *evsel)
>  {
> -	if (list_empty(&evsel->bpf_counter_list))
> +	if (bpf_counter_skip(evsel))
>  		return -EAGAIN;
>  	return evsel->bpf_counter_ops->read(evsel);
>  }
>  
>  void bpf_counter__destroy(struct evsel *evsel)
>  {
> -	if (list_empty(&evsel->bpf_counter_list))
> +	if (bpf_counter_skip(evsel))
>  		return;
>  	evsel->bpf_counter_ops->destroy(evsel);
>  	evsel->bpf_counter_ops = NULL;
> diff --git a/tools/perf/util/bpf_skel/bperf.h b/tools/perf/util/bpf_skel/bperf.h
> new file mode 100644
> index 0000000000000..186a5551ddb9d
> --- /dev/null
> +++ b/tools/perf/util/bpf_skel/bperf.h
> @@ -0,0 +1,14 @@
> +// SPDX-License-Identifier: (GPL-2.0-only OR BSD-2-Clause)
> +// Copyright (c) 2021 Facebook
> +
> +#ifndef __BPERF_STAT_H
> +#define __BPERF_STAT_H
> +
> +typedef struct {
> +	__uint(type, BPF_MAP_TYPE_PERCPU_ARRAY);
> +	__uint(key_size, sizeof(__u32));
> +	__uint(value_size, sizeof(struct bpf_perf_event_value));
> +	__uint(max_entries, 1);
> +} reading_map;
> +
> +#endif /* __BPERF_STAT_H */
> diff --git a/tools/perf/util/bpf_skel/bperf_follower.bpf.c b/tools/perf/util/bpf_skel/bperf_follower.bpf.c
> new file mode 100644
> index 0000000000000..b8fa3cb2da230
> --- /dev/null
> +++ b/tools/perf/util/bpf_skel/bperf_follower.bpf.c
> @@ -0,0 +1,69 @@
> +// SPDX-License-Identifier: (GPL-2.0-only OR BSD-2-Clause)
> +// Copyright (c) 2021 Facebook
> +#include <linux/bpf.h>
> +#include <linux/perf_event.h>
> +#include <bpf/bpf_helpers.h>
> +#include <bpf/bpf_tracing.h>
> +#include "bperf.h"
> +#include "bperf_u.h"
> +
> +reading_map diff_readings SEC(".maps");
> +reading_map accum_readings SEC(".maps");
> +
> +struct {
> +	__uint(type, BPF_MAP_TYPE_HASH);
> +	__uint(key_size, sizeof(__u32));
> +	__uint(value_size, sizeof(__u32));
> +} filter SEC(".maps");
> +
> +enum bperf_filter_type type = 0;
> +int enabled = 0;
> +
> +SEC("fexit/XXX")
> +int BPF_PROG(fexit_XXX)
> +{
> +	struct bpf_perf_event_value *diff_val, *accum_val;
> +	__u32 filter_key, zero = 0;
> +	__u32 *accum_key;
> +
> +	if (!enabled)
> +		return 0;
> +
> +	switch (type) {
> +	case BPERF_FILTER_GLOBAL:
> +		accum_key = &zero;
> +		goto do_add;
> +	case BPERF_FILTER_CPU:
> +		filter_key = bpf_get_smp_processor_id();
> +		break;
> +	case BPERF_FILTER_PID:
> +		filter_key = bpf_get_current_pid_tgid() & 0xffffffff;
> +		break;
> +	case BPERF_FILTER_TGID:
> +		filter_key = bpf_get_current_pid_tgid() >> 32;
> +		break;
> +	default:
> +		return 0;
> +	}
> +
> +	accum_key = bpf_map_lookup_elem(&filter, &filter_key);
> +	if (!accum_key)
> +		return 0;
> +
> +do_add:
> +	diff_val = bpf_map_lookup_elem(&diff_readings, &zero);
> +	if (!diff_val)
> +		return 0;
> +
> +	accum_val = bpf_map_lookup_elem(&accum_readings, accum_key);
> +	if (!accum_val)
> +		return 0;
> +
> +	accum_val->counter += diff_val->counter;
> +	accum_val->enabled += diff_val->enabled;
> +	accum_val->running += diff_val->running;
> +
> +	return 0;
> +}
> +
> +char LICENSE[] SEC("license") = "Dual BSD/GPL";
> diff --git a/tools/perf/util/bpf_skel/bperf_leader.bpf.c b/tools/perf/util/bpf_skel/bperf_leader.bpf.c
> new file mode 100644
> index 0000000000000..4f70d1459e86c
> --- /dev/null
> +++ b/tools/perf/util/bpf_skel/bperf_leader.bpf.c
> @@ -0,0 +1,46 @@
> +// SPDX-License-Identifier: (GPL-2.0-only OR BSD-2-Clause)
> +// Copyright (c) 2021 Facebook
> +#include <linux/bpf.h>
> +#include <linux/perf_event.h>
> +#include <bpf/bpf_helpers.h>
> +#include <bpf/bpf_tracing.h>
> +#include "bperf.h"
> +
> +struct {
> +	__uint(type, BPF_MAP_TYPE_PERF_EVENT_ARRAY);
> +	__uint(key_size, sizeof(__u32));
> +	__uint(value_size, sizeof(int));
> +	__uint(map_flags, BPF_F_PRESERVE_ELEMS);
> +} events SEC(".maps");
> +
> +reading_map prev_readings SEC(".maps");
> +reading_map diff_readings SEC(".maps");
> +
> +SEC("raw_tp/sched_switch")
> +int BPF_PROG(on_switch)
> +{
> +	struct bpf_perf_event_value val, *prev_val, *diff_val;
> +	__u32 key = bpf_get_smp_processor_id();
> +	__u32 zero = 0;
> +	long err;
> +
> +	prev_val = bpf_map_lookup_elem(&prev_readings, &zero);
> +	if (!prev_val)
> +		return 0;
> +
> +	diff_val = bpf_map_lookup_elem(&diff_readings, &zero);
> +	if (!diff_val)
> +		return 0;
> +
> +	err = bpf_perf_event_read_value(&events, key, &val, sizeof(val));
> +	if (err)
> +		return 0;
> +
> +	diff_val->counter = val.counter - prev_val->counter;
> +	diff_val->enabled = val.enabled - prev_val->enabled;
> +	diff_val->running = val.running - prev_val->running;
> +	*prev_val = val;
> +	return 0;
> +}
> +
> +char LICENSE[] SEC("license") = "Dual BSD/GPL";
> diff --git a/tools/perf/util/bpf_skel/bperf_u.h b/tools/perf/util/bpf_skel/bperf_u.h
> new file mode 100644
> index 0000000000000..1ce0c2c905c11
> --- /dev/null
> +++ b/tools/perf/util/bpf_skel/bperf_u.h
> @@ -0,0 +1,14 @@
> +// SPDX-License-Identifier: (GPL-2.0-only OR BSD-2-Clause)
> +// Copyright (c) 2021 Facebook
> +
> +#ifndef __BPERF_STAT_U_H
> +#define __BPERF_STAT_U_H
> +
> +enum bperf_filter_type {
> +	BPERF_FILTER_GLOBAL = 1,
> +	BPERF_FILTER_CPU,
> +	BPERF_FILTER_PID,
> +	BPERF_FILTER_TGID,
> +};
> +
> +#endif /* __BPERF_STAT_U_H */
> diff --git a/tools/perf/util/evsel.h b/tools/perf/util/evsel.h
> index 6026487353dd8..dd4f56f9cfdf5 100644
> --- a/tools/perf/util/evsel.h
> +++ b/tools/perf/util/evsel.h
> @@ -20,6 +20,8 @@ union perf_event;
>  struct bpf_counter_ops;
>  struct target;
>  struct hashmap;
> +struct bperf_leader_bpf;
> +struct bperf_follower_bpf;
>  
>  typedef int (evsel__sb_cb_t)(union perf_event *event, void *data);
>  
> @@ -130,8 +132,24 @@ struct evsel {
>  	 * See also evsel__has_callchain().
>  	 */
>  	__u64			synth_sample_type;
> -	struct list_head	bpf_counter_list;
> +
> +	/*
> +	 * bpf_counter_ops serves two use cases:
> +	 *   1. perf-stat -b          counting events used byBPF programs
> +	 *   2. perf-stat --use-bpf   use BPF programs to aggregate counts
> +	 */
>  	struct bpf_counter_ops	*bpf_counter_ops;
> +
> +	/* for perf-stat -b */
> +	struct list_head	bpf_counter_list;
> +
> +	/* for perf-stat --use-bpf */
> +	int			bperf_leader_prog_fd;
> +	int			bperf_leader_link_fd;
> +	union {
> +		struct bperf_leader_bpf *leader_skel;
> +		struct bperf_follower_bpf *follower_skel;
> +	};
>  };
>  
>  struct perf_missing_features {
> diff --git a/tools/perf/util/target.h b/tools/perf/util/target.h
> index f132c6c2eef81..1bce3eb28ef25 100644
> --- a/tools/perf/util/target.h
> +++ b/tools/perf/util/target.h
> @@ -16,6 +16,8 @@ struct target {
>  	bool	     uses_mmap;
>  	bool	     default_per_cpu;
>  	bool	     per_thread;
> +	bool	     use_bpf;
> +	const char   *attr_map;
>  };
>  
>  enum target_errno {
> @@ -66,7 +68,7 @@ static inline bool target__has_cpu(struct target *target)
>  
>  static inline bool target__has_bpf(struct target *target)
>  {
> -	return target->bpf_str;
> +	return target->bpf_str || target->use_bpf;
>  }
>  
>  static inline bool target__none(struct target *target)
> -- 
> 2.30.2
> 

Re: [PATCH v2 0/3] perf-stat: share hardware PMCs with BPF

[Arnaldo]

On March 18, 2021 6:14:34 PM GMT-03:00, Jiri Olsa <jolsa@redhat.com> wrote:
>On Thu, Mar 18, 2021 at 03:52:51AM +0000, Song Liu wrote:
>> 
>> 
>> > On Mar 17, 2021, at 6:11 AM, Arnaldo Carvalho de Melo
><acme@kernel.org> wrote:
>> > 
>> > Em Wed, Mar 17, 2021 at 02:29:28PM +0900, Namhyung Kim escreveu:
>> >> Hi Song,
>> >> 
>> >> On Wed, Mar 17, 2021 at 6:18 AM Song Liu <songliubraving@fb.com>
>wrote:
>> >>> 
>> >>> perf uses performance monitoring counters (PMCs) to monitor
>system
>> >>> performance. The PMCs are limited hardware resources. For
>example,
>> >>> Intel CPUs have 3x fixed PMCs and 4x programmable PMCs per cpu.
>> >>> 
>> >>> Modern data center systems use these PMCs in many different ways:
>> >>> system level monitoring, (maybe nested) container level
>monitoring, per
>> >>> process monitoring, profiling (in sample mode), etc. In some
>cases,
>> >>> there are more active perf_events than available hardware PMCs.
>To allow
>> >>> all perf_events to have a chance to run, it is necessary to do
>expensive
>> >>> time multiplexing of events.
>> >>> 
>> >>> On the other hand, many monitoring tools count the common metrics
>(cycles,
>> >>> instructions). It is a waste to have multiple tools create
>multiple
>> >>> perf_events of "cycles" and occupy multiple PMCs.
>> >> 
>> >> Right, it'd be really helpful when the PMCs are frequently or
>mostly shared.
>> >> But it'd also increase the overhead for uncontended cases as BPF
>programs
>> >> need to run on every context switch.  Depending on the workload,
>it may
>> >> cause a non-negligible performance impact.  So users should be
>aware of it.
>> > 
>> > Would be interesting to, humm, measure both cases to have a firm
>number
>> > of the impact, how many instructions are added when sharing using
>> > --bpf-counters?
>> > 
>> > I.e. compare the "expensive time multiplexing of events" with its
>> > avoidance by using --bpf-counters.
>> > 
>> > Song, have you perfmormed such measurements?
>> 
>> I have got some measurements with perf-bench-sched-messaging:
>> 
>> The system: x86_64 with 23 cores (46 HT)
>> 
>> The perf-stat command:
>> perf stat -e
>cycles,cycles,instructions,instructions,ref-cycles,ref-cycles <target,
>etc.>
>> 
>> The benchmark command and output:
>> ./perf bench sched messaging -g 40 -l 50000 -t
>> # Running 'sched/messaging' benchmark:
>> # 20 sender and receiver threads per group
>> # 40 groups == 1600 threads run
>>      Total time: 10X.XXX [sec]
>> 
>> 
>> I use the "Total time" as measurement, so smaller number is better. 
>> 
>> For each condition, I run the command 5 times, and took the median of
>
>> "Total time". 
>> 
>> Baseline (no perf-stat)			104.873 [sec]
>> # global
>> perf stat -a				107.887 [sec]
>> perf stat -a --bpf-counters		106.071 [sec]
>> # per task
>> perf stat 				106.314 [sec]
>> perf stat --bpf-counters 		105.965 [sec]
>> # per cpu
>> perf stat -C 1,3,5 			107.063 [sec]
>> perf stat -C 1,3,5 --bpf-counters 	106.406 [sec]
>
>I can't see why it's actualy faster than normal perf ;-)
>would be worth to find out

Isn't this all about contended cases?

>
>jirka
>
>> 
>> From the data, --bpf-counters is slightly better than the regular
>event
>> for all targets. I noticed that the results are not very stable.
>There 
>> are a couple 108.xx runs in some of the conditions (w/ and w/o 
>> --bpf-counters).
>> 
>> 
>> I also measured the average runtime of the BPF programs, with 
>> 
>> 	sysctl kernel.bpf_stats_enabled=1
>> 
>> For each event, if we have one leader and two followers, the total
>run 
>> time is about 340ns. IOW, 340ns for two perf-stat reading
>instructions, 
>> 340ns for two perf-stat reading cycles, etc. 
>> 
>> Thanks,
>> Song
>> 

-- 
Sent from my Android device with K-9 Mail. Please excuse my brevity.

Re: [PATCH v2 0/3] perf-stat: share hardware PMCs with BPF

[Song Liu]

> On Mar 18, 2021, at 5:09 PM, Arnaldo <arnaldo.melo@gmail.com> wrote:
> 
> 
> 
> On March 18, 2021 6:14:34 PM GMT-03:00, Jiri Olsa <jolsa@redhat.com> wrote:
>> On Thu, Mar 18, 2021 at 03:52:51AM +0000, Song Liu wrote:
>>> 
>>> 
>>>> On Mar 17, 2021, at 6:11 AM, Arnaldo Carvalho de Melo
>> <acme@kernel.org> wrote:
>>>> 
>>>> Em Wed, Mar 17, 2021 at 02:29:28PM +0900, Namhyung Kim escreveu:
>>>>> Hi Song,
>>>>> 
>>>>> On Wed, Mar 17, 2021 at 6:18 AM Song Liu <songliubraving@fb.com>
>> wrote:
>>>>>> 
>>>>>> perf uses performance monitoring counters (PMCs) to monitor
>> system
>>>>>> performance. The PMCs are limited hardware resources. For
>> example,
>>>>>> Intel CPUs have 3x fixed PMCs and 4x programmable PMCs per cpu.
>>>>>> 
>>>>>> Modern data center systems use these PMCs in many different ways:
>>>>>> system level monitoring, (maybe nested) container level
>> monitoring, per
>>>>>> process monitoring, profiling (in sample mode), etc. In some
>> cases,
>>>>>> there are more active perf_events than available hardware PMCs.
>> To allow
>>>>>> all perf_events to have a chance to run, it is necessary to do
>> expensive
>>>>>> time multiplexing of events.
>>>>>> 
>>>>>> On the other hand, many monitoring tools count the common metrics
>> (cycles,
>>>>>> instructions). It is a waste to have multiple tools create
>> multiple
>>>>>> perf_events of "cycles" and occupy multiple PMCs.
>>>>> 
>>>>> Right, it'd be really helpful when the PMCs are frequently or
>> mostly shared.
>>>>> But it'd also increase the overhead for uncontended cases as BPF
>> programs
>>>>> need to run on every context switch.  Depending on the workload,
>> it may
>>>>> cause a non-negligible performance impact.  So users should be
>> aware of it.
>>>> 
>>>> Would be interesting to, humm, measure both cases to have a firm
>> number
>>>> of the impact, how many instructions are added when sharing using
>>>> --bpf-counters?
>>>> 
>>>> I.e. compare the "expensive time multiplexing of events" with its
>>>> avoidance by using --bpf-counters.
>>>> 
>>>> Song, have you perfmormed such measurements?
>>> 
>>> I have got some measurements with perf-bench-sched-messaging:
>>> 
>>> The system: x86_64 with 23 cores (46 HT)
>>> 
>>> The perf-stat command:
>>> perf stat -e
>> cycles,cycles,instructions,instructions,ref-cycles,ref-cycles <target,
>> etc.>
>>> 
>>> The benchmark command and output:
>>> ./perf bench sched messaging -g 40 -l 50000 -t
>>> # Running 'sched/messaging' benchmark:
>>> # 20 sender and receiver threads per group
>>> # 40 groups == 1600 threads run
>>>     Total time: 10X.XXX [sec]
>>> 
>>> 
>>> I use the "Total time" as measurement, so smaller number is better. 
>>> 
>>> For each condition, I run the command 5 times, and took the median of
>> 
>>> "Total time". 
>>> 
>>> Baseline (no perf-stat)			104.873 [sec]
>>> # global
>>> perf stat -a				107.887 [sec]
>>> perf stat -a --bpf-counters		106.071 [sec]
>>> # per task
>>> perf stat 				106.314 [sec]
>>> perf stat --bpf-counters 		105.965 [sec]
>>> # per cpu
>>> perf stat -C 1,3,5 			107.063 [sec]
>>> perf stat -C 1,3,5 --bpf-counters 	106.406 [sec]
>> 
>> I can't see why it's actualy faster than normal perf ;-)
>> would be worth to find out
> 
> Isn't this all about contended cases?

Yeah, the normal perf is doing time multiplexing; while --bpf-counters 
doesn't need it. 

Thanks,
Song

Re: [PATCH v2 0/3] perf-stat: share hardware PMCs with BPF

[Namhyung Kim]

On Fri, Mar 19, 2021 at 9:22 AM Song Liu <songliubraving@fb.com> wrote:
>
>
>
> > On Mar 18, 2021, at 5:09 PM, Arnaldo <arnaldo.melo@gmail.com> wrote:
> >
> >
> >
> > On March 18, 2021 6:14:34 PM GMT-03:00, Jiri Olsa <jolsa@redhat.com> wrote:
> >> On Thu, Mar 18, 2021 at 03:52:51AM +0000, Song Liu wrote:
> >>>
> >>>
> >>>> On Mar 17, 2021, at 6:11 AM, Arnaldo Carvalho de Melo
> >> <acme@kernel.org> wrote:
> >>>>
> >>>> Em Wed, Mar 17, 2021 at 02:29:28PM +0900, Namhyung Kim escreveu:
> >>>>> Hi Song,
> >>>>>
> >>>>> On Wed, Mar 17, 2021 at 6:18 AM Song Liu <songliubraving@fb.com>
> >> wrote:
> >>>>>>
> >>>>>> perf uses performance monitoring counters (PMCs) to monitor
> >> system
> >>>>>> performance. The PMCs are limited hardware resources. For
> >> example,
> >>>>>> Intel CPUs have 3x fixed PMCs and 4x programmable PMCs per cpu.
> >>>>>>
> >>>>>> Modern data center systems use these PMCs in many different ways:
> >>>>>> system level monitoring, (maybe nested) container level
> >> monitoring, per
> >>>>>> process monitoring, profiling (in sample mode), etc. In some
> >> cases,
> >>>>>> there are more active perf_events than available hardware PMCs.
> >> To allow
> >>>>>> all perf_events to have a chance to run, it is necessary to do
> >> expensive
> >>>>>> time multiplexing of events.
> >>>>>>
> >>>>>> On the other hand, many monitoring tools count the common metrics
> >> (cycles,
> >>>>>> instructions). It is a waste to have multiple tools create
> >> multiple
> >>>>>> perf_events of "cycles" and occupy multiple PMCs.
> >>>>>
> >>>>> Right, it'd be really helpful when the PMCs are frequently or
> >> mostly shared.
> >>>>> But it'd also increase the overhead for uncontended cases as BPF
> >> programs
> >>>>> need to run on every context switch.  Depending on the workload,
> >> it may
> >>>>> cause a non-negligible performance impact.  So users should be
> >> aware of it.
> >>>>
> >>>> Would be interesting to, humm, measure both cases to have a firm
> >> number
> >>>> of the impact, how many instructions are added when sharing using
> >>>> --bpf-counters?
> >>>>
> >>>> I.e. compare the "expensive time multiplexing of events" with its
> >>>> avoidance by using --bpf-counters.
> >>>>
> >>>> Song, have you perfmormed such measurements?
> >>>
> >>> I have got some measurements with perf-bench-sched-messaging:
> >>>
> >>> The system: x86_64 with 23 cores (46 HT)
> >>>
> >>> The perf-stat command:
> >>> perf stat -e
> >> cycles,cycles,instructions,instructions,ref-cycles,ref-cycles <target,
> >> etc.>
> >>>
> >>> The benchmark command and output:
> >>> ./perf bench sched messaging -g 40 -l 50000 -t
> >>> # Running 'sched/messaging' benchmark:
> >>> # 20 sender and receiver threads per group
> >>> # 40 groups == 1600 threads run
> >>>     Total time: 10X.XXX [sec]
> >>>
> >>>
> >>> I use the "Total time" as measurement, so smaller number is better.
> >>>
> >>> For each condition, I run the command 5 times, and took the median of
> >>
> >>> "Total time".
> >>>
> >>> Baseline (no perf-stat)                     104.873 [sec]
> >>> # global
> >>> perf stat -a                                107.887 [sec]
> >>> perf stat -a --bpf-counters         106.071 [sec]
> >>> # per task
> >>> perf stat                           106.314 [sec]
> >>> perf stat --bpf-counters            105.965 [sec]
> >>> # per cpu
> >>> perf stat -C 1,3,5                  107.063 [sec]
> >>> perf stat -C 1,3,5 --bpf-counters   106.406 [sec]
> >>
> >> I can't see why it's actualy faster than normal perf ;-)
> >> would be worth to find out
> >
> > Isn't this all about contended cases?
>
> Yeah, the normal perf is doing time multiplexing; while --bpf-counters
> doesn't need it.

Yep, so for uncontended cases, normal perf should be the same as the
baseline (faster than the bperf).  But for contended cases, the bperf
works faster.

Thanks,
Namhyung

Re: [PATCH v2 0/3] perf-stat: share hardware PMCs with BPF

[Arnaldo Carvalho de Melo]

Em Fri, Mar 19, 2021 at 09:54:59AM +0900, Namhyung Kim escreveu:
> On Fri, Mar 19, 2021 at 9:22 AM Song Liu <songliubraving@fb.com> wrote:
> > > On Mar 18, 2021, at 5:09 PM, Arnaldo <arnaldo.melo@gmail.com> wrote:
> > > On March 18, 2021 6:14:34 PM GMT-03:00, Jiri Olsa <jolsa@redhat.com> wrote:
> > >> On Thu, Mar 18, 2021 at 03:52:51AM +0000, Song Liu wrote:
> > >>> perf stat -C 1,3,5                  107.063 [sec]
> > >>> perf stat -C 1,3,5 --bpf-counters   106.406 [sec]

> > >> I can't see why it's actualy faster than normal perf ;-)
> > >> would be worth to find out

> > > Isn't this all about contended cases?

> > Yeah, the normal perf is doing time multiplexing; while --bpf-counters
> > doesn't need it.

> Yep, so for uncontended cases, normal perf should be the same as the
> baseline (faster than the bperf).  But for contended cases, the bperf
> works faster.

The difference should be small enough that for people that use this in a
machine where contention happens most of the time, setting a
~/.perfconfig to use it by default should be advantageous, i.e. no need
to use --bpf-counters on the command line all the time.

So, Namhyung, can I take that as an Acked-by or a Reviewed-by? I'll take
a look again now but I want to have this merged on perf/core so that I
can work on a new BPF SKEL to use this:

https://git.kernel.org/pub/scm/linux/kernel/git/acme/linux.git/log/?h=tmp.bpf/bpf_perf_enable

:-)

- Arnaldo

Re: [PATCH v2 0/3] perf-stat: share hardware PMCs with BPF

[Namhyung Kim]

Hi Arnaldo,

On Sat, Mar 20, 2021 at 12:35 AM Arnaldo Carvalho de Melo
<acme@kernel.org> wrote:
>
> Em Fri, Mar 19, 2021 at 09:54:59AM +0900, Namhyung Kim escreveu:
> > On Fri, Mar 19, 2021 at 9:22 AM Song Liu <songliubraving@fb.com> wrote:
> > > > On Mar 18, 2021, at 5:09 PM, Arnaldo <arnaldo.melo@gmail.com> wrote:
> > > > On March 18, 2021 6:14:34 PM GMT-03:00, Jiri Olsa <jolsa@redhat.com> wrote:
> > > >> On Thu, Mar 18, 2021 at 03:52:51AM +0000, Song Liu wrote:
> > > >>> perf stat -C 1,3,5                  107.063 [sec]
> > > >>> perf stat -C 1,3,5 --bpf-counters   106.406 [sec]
>
> > > >> I can't see why it's actualy faster than normal perf ;-)
> > > >> would be worth to find out
>
> > > > Isn't this all about contended cases?
>
> > > Yeah, the normal perf is doing time multiplexing; while --bpf-counters
> > > doesn't need it.
>
> > Yep, so for uncontended cases, normal perf should be the same as the
> > baseline (faster than the bperf).  But for contended cases, the bperf
> > works faster.
>
> The difference should be small enough that for people that use this in a
> machine where contention happens most of the time, setting a
> ~/.perfconfig to use it by default should be advantageous, i.e. no need
> to use --bpf-counters on the command line all the time.
>
> So, Namhyung, can I take that as an Acked-by or a Reviewed-by? I'll take
> a look again now but I want to have this merged on perf/core so that I
> can work on a new BPF SKEL to use this:

I have a concern for the per cpu target, but it can be done later, so

Acked-by: Namhyung Kim <namhyung@kernel.org>

>
> https://git.kernel.org/pub/scm/linux/kernel/git/acme/linux.git/log/?h=tmp.bpf/bpf_perf_enable

Interesting!  Actually I was thinking about the similar too. :)

Thanks,
Namhyung

Re: [PATCH v2 0/3] perf-stat: share hardware PMCs with BPF

[Song Liu]

> On Mar 19, 2021, at 8:58 AM, Namhyung Kim <namhyung@kernel.org> wrote:
> 
> Hi Arnaldo,
> 
> On Sat, Mar 20, 2021 at 12:35 AM Arnaldo Carvalho de Melo
> <acme@kernel.org> wrote:
>> 
>> Em Fri, Mar 19, 2021 at 09:54:59AM +0900, Namhyung Kim escreveu:
>>> On Fri, Mar 19, 2021 at 9:22 AM Song Liu <songliubraving@fb.com> wrote:
>>>>> On Mar 18, 2021, at 5:09 PM, Arnaldo <arnaldo.melo@gmail.com> wrote:
>>>>> On March 18, 2021 6:14:34 PM GMT-03:00, Jiri Olsa <jolsa@redhat.com> wrote:
>>>>>> On Thu, Mar 18, 2021 at 03:52:51AM +0000, Song Liu wrote:
>>>>>>> perf stat -C 1,3,5                  107.063 [sec]
>>>>>>> perf stat -C 1,3,5 --bpf-counters   106.406 [sec]
>> 
>>>>>> I can't see why it's actualy faster than normal perf ;-)
>>>>>> would be worth to find out
>> 
>>>>> Isn't this all about contended cases?
>> 
>>>> Yeah, the normal perf is doing time multiplexing; while --bpf-counters
>>>> doesn't need it.
>> 
>>> Yep, so for uncontended cases, normal perf should be the same as the
>>> baseline (faster than the bperf).  But for contended cases, the bperf
>>> works faster.
>> 
>> The difference should be small enough that for people that use this in a
>> machine where contention happens most of the time, setting a
>> ~/.perfconfig to use it by default should be advantageous, i.e. no need
>> to use --bpf-counters on the command line all the time.
>> 
>> So, Namhyung, can I take that as an Acked-by or a Reviewed-by? I'll take
>> a look again now but I want to have this merged on perf/core so that I
>> can work on a new BPF SKEL to use this:
> 
> I have a concern for the per cpu target, but it can be done later, so
> 
> Acked-by: Namhyung Kim <namhyung@kernel.org>
> 
>> 
>> https://git.kernel.org/pub/scm/linux/kernel/git/acme/linux.git/log/?h=tmp.bpf/bpf_perf_enable
> 
> Interesting!  Actually I was thinking about the similar too. :)

Hi Namhyung, Jiri, and Arnaldo,

Thanks a lot for your kind review. 

Here is updated 3/3, where we use perf-bench instead of stressapptest.

Thanks,
Song


From cc79d161be9c9d24198f7e35b50058a6e15076fd Mon Sep 17 00:00:00 2001
From: Song Liu <songliubraving@fb.com>
Date: Tue, 16 Mar 2021 00:19:53 -0700
Subject: [PATCH v3 3/3] perf-test: add a test for perf-stat --bpf-counters
 option

Add a test to compare the output of perf-stat with and without option
--bpf-counters. If the difference is more than 10%, the test is considered
as failed.

Signed-off-by: Song Liu <songliubraving@fb.com>
---
 tools/perf/tests/shell/stat_bpf_counters.sh | 31 +++++++++++++++++++++
 1 file changed, 31 insertions(+)
 create mode 100755 tools/perf/tests/shell/stat_bpf_counters.sh

diff --git a/tools/perf/tests/shell/stat_bpf_counters.sh b/tools/perf/tests/shell/stat_bpf_counters.sh
new file mode 100755
index 0000000000000..7aabf177ce8d1
--- /dev/null
+++ b/tools/perf/tests/shell/stat_bpf_counters.sh
@@ -0,0 +1,31 @@
+#!/bin/sh
+# perf stat --bpf-counters test
+# SPDX-License-Identifier: GPL-2.0
+
+set -e
+
+# check whether $2 is within +/- 10% of $1
+compare_number()
+{
+       first_num=$1
+       second_num=$2
+
+       # upper bound is first_num * 110%
+       upper=$(( $first_num + $first_num / 10 ))
+       # lower bound is first_num * 90%
+       lower=$(( $first_num - $first_num / 10 ))
+
+       if [ $second_num -gt $upper ] || [ $second_num -lt $lower ]; then
+               echo "The difference between $first_num and $second_num are greater than 10%."
+               exit 1
+       fi
+}
+
+# skip if --bpf-counters is not supported
+perf stat --bpf-counters true > /dev/null 2>&1 || exit 2
+
+base_cycles=$(perf stat --no-big-num -e cycles -- perf bench sched messaging -g 1 -l 100 -t 2>&1 | awk '/cycles/ {print $1}')
+bpf_cycles=$(perf stat --no-big-num --bpf-counters -e cycles -- perf bench sched messaging -g 1 -l 100 -t 2>&1 | awk '/cycles/ {print $1}')
+
+compare_number $base_cycles $bpf_cycles
+exit 0
--
2.30.2

Re: [PATCH v2 1/3] perf-stat: introduce bperf, share hardware PMCs with BPF

[Arnaldo Carvalho de Melo]

Em Thu, Mar 18, 2021 at 10:15:13PM +0100, Jiri Olsa escreveu:
> On Tue, Mar 16, 2021 at 02:18:35PM -0700, Song Liu wrote:
> > bperf is off by default. To enable it, pass --bpf-counters option to
> > perf-stat. bperf uses a BPF hashmap to share information about BPF
> > programs and maps used by bperf. This map is pinned to bpffs. The default
> > path is /sys/fs/bpf/perf_attr_map. The user could change the path with
> > option --bpf-attr-map.
> > 
> > Signed-off-by: Song Liu <songliubraving@fb.com>
> 
> Reviewed-by: Jiri Olsa <jolsa@redhat.com>

After applying just this first patch in the series I'm getting this
after a 'make -C tools/ clean', now I'm checking if I need some new
clang, ideas?

- Arnaldo

[acme@quaco perf]$ make O=/tmp/build/perf -C tools/perf BUILD_BPF_SKEL=1 PYTHON=python3 install-bin
make: Entering directory '/home/acme/git/perf/tools/perf'
  BUILD:   Doing 'make -j8' parallel build
Warning: Kernel ABI header at 'tools/include/uapi/linux/kvm.h' differs from latest version at 'include/uapi/linux/kvm.h'
diff -u tools/include/uapi/linux/kvm.h include/uapi/linux/kvm.h
Warning: Kernel ABI header at 'tools/perf/arch/mips/entry/syscalls/syscall_n64.tbl' differs from latest version at 'arch/mips/kernel/syscalls/syscall_n64.tbl'
diff -u tools/perf/arch/mips/entry/syscalls/syscall_n64.tbl arch/mips/kernel/syscalls/syscall_n64.tbl

Auto-detecting system features:
...                         dwarf: [ on  ]
...            dwarf_getlocations: [ on  ]
...                         glibc: [ on  ]
...                        libbfd: [ on  ]
...                libbfd-buildid: [ on  ]
...                        libcap: [ on  ]
...                        libelf: [ on  ]
...                       libnuma: [ on  ]
...        numa_num_possible_cpus: [ on  ]
...                       libperl: [ on  ]
...                     libpython: [ on  ]
...                     libcrypto: [ on  ]
...                     libunwind: [ on  ]
...            libdw-dwarf-unwind: [ on  ]
...                          zlib: [ on  ]
...                          lzma: [ on  ]
...                     get_cpuid: [ on  ]
...                           bpf: [ on  ]
...                        libaio: [ on  ]
...                       libzstd: [ on  ]
...        disassembler-four-args: [ on  ]

  GEN      /tmp/build/perf/common-cmds.h
  CC       /tmp/build/perf/exec-cmd.o
  MKDIR    /tmp/build/perf/fd/
  MKDIR    /tmp/build/perf/fs/
  CC       /tmp/build/perf/fs/fs.o
  CC       /tmp/build/perf/event-parse.o
  CC       /tmp/build/perf/fd/array.o
  CC       /tmp/build/perf/core.o
  GEN      /tmp/build/perf/bpf_helper_defs.h
  CC       /tmp/build/perf/event-plugin.o
  MKDIR    /tmp/build/perf/staticobjs/
  PERF_VERSION = 5.12.rc2.g3df07f57f205
  CC       /tmp/build/perf/staticobjs/libbpf.o
  CC       /tmp/build/perf/cpu.o
  LD       /tmp/build/perf/fd/libapi-in.o
  CC       /tmp/build/perf/cpumap.o
  CC       /tmp/build/perf/help.o
  MKDIR    /tmp/build/perf/fs/
  CC       /tmp/build/perf/fs/tracing_path.o
  CC       /tmp/build/perf/fs/cgroup.o
  CC       /tmp/build/perf/trace-seq.o
  CC       /tmp/build/perf/pager.o
  CC       /tmp/build/perf/parse-options.o
  LD       /tmp/build/perf/fs/libapi-in.o
  CC       /tmp/build/perf/debug.o
  CC       /tmp/build/perf/str_error_r.o
  CC       /tmp/build/perf/run-command.o
  CC       /tmp/build/perf/sigchain.o
  LD       /tmp/build/perf/libapi-in.o
  AR       /tmp/build/perf/libapi.a
  CC       /tmp/build/perf/subcmd-config.o
  CC       /tmp/build/perf/threadmap.o
  CC       /tmp/build/perf/evsel.o
  CC       /tmp/build/perf/parse-filter.o
  MKDIR    /tmp/build/perf/staticobjs/
  CC       /tmp/build/perf/staticobjs/bpf.o
  CC       /tmp/build/perf/evlist.o
  CC       /tmp/build/perf/parse-utils.o
  CC       /tmp/build/perf/kbuffer-parse.o
  CC       /tmp/build/perf/tep_strerror.o
  CC       /tmp/build/perf/mmap.o
  CC       /tmp/build/perf/zalloc.o
  CC       /tmp/build/perf/event-parse-api.o
  LD       /tmp/build/perf/libsubcmd-in.o
  AR       /tmp/build/perf/libsubcmd.a
  CC       /tmp/build/perf/xyarray.o
  LD       /tmp/build/perf/libtraceevent-in.o
  LINK     /tmp/build/perf/libtraceevent.a
  CC       /tmp/build/perf/staticobjs/nlattr.o
  CC       /tmp/build/perf/staticobjs/btf.o
  CC       /tmp/build/perf/lib.o
  CC       /tmp/build/perf/staticobjs/libbpf_errno.o
  CC       /tmp/build/perf/staticobjs/str_error.o
  CC       /tmp/build/perf/staticobjs/netlink.o
  CC       /tmp/build/perf/staticobjs/bpf_prog_linfo.o
  CC       /tmp/build/perf/staticobjs/libbpf_probes.o
  LD       /tmp/build/perf/libperf-in.o
  AR       /tmp/build/perf/libperf.a
  MKDIR    /tmp/build/perf/pmu-events/
  HOSTCC   /tmp/build/perf/pmu-events/json.o
  CC       /tmp/build/perf/plugin_jbd2.o
  CC       /tmp/build/perf/staticobjs/xsk.o
  MKDIR    /tmp/build/perf/pmu-events/
  HOSTCC   /tmp/build/perf/pmu-events/jsmn.o
  CC       /tmp/build/perf/staticobjs/hashmap.o
  LD       /tmp/build/perf/plugin_jbd2-in.o
  CC       /tmp/build/perf/staticobjs/btf_dump.o
  CC       /tmp/build/perf/plugin_hrtimer.o
  HOSTCC   /tmp/build/perf/pmu-events/jevents.o
  LD       /tmp/build/perf/plugin_hrtimer-in.o
  CC       /tmp/build/perf/plugin_kmem.o
  CC       /tmp/build/perf/staticobjs/ringbuf.o
  LD       /tmp/build/perf/plugin_kmem-in.o
  CC       /tmp/build/perf/plugin_kvm.o
  HOSTLD   /tmp/build/perf/pmu-events/jevents-in.o
  CC       /tmp/build/perf/perf-read-vdso32
  CC       /tmp/build/perf/plugin_mac80211.o
  LD       /tmp/build/perf/plugin_kvm-in.o
  CC       /tmp/build/perf/plugin_sched_switch.o
  CC       /tmp/build/perf/plugin_function.o
  MKDIR    /tmp/build/perf/jvmti/
  CC       /tmp/build/perf/jvmti/libjvmti.o
  LD       /tmp/build/perf/plugin_mac80211-in.o
  GEN      perf-archive
  LD       /tmp/build/perf/plugin_sched_switch-in.o
  CC       /tmp/build/perf/plugin_futex.o
  CC       /tmp/build/perf/plugin_xen.o
  CC       /tmp/build/perf/plugin_scsi.o
  LD       /tmp/build/perf/plugin_function-in.o
  CC       /tmp/build/perf/plugin_cfg80211.o
  LD       /tmp/build/perf/plugin_futex-in.o
  LD       /tmp/build/perf/plugin_xen-in.o
  LINK     /tmp/build/perf/plugin_jbd2.so
  LINK     /tmp/build/perf/plugin_hrtimer.so
  CC       /tmp/build/perf/plugin_tlb.o
  LINK     /tmp/build/perf/plugin_kmem.so
  LINK     /tmp/build/perf/plugin_kvm.so
  LD       /tmp/build/perf/plugin_scsi-in.o
  LINK     /tmp/build/perf/plugin_mac80211.so
  LINK     /tmp/build/perf/plugin_sched_switch.so
  LINK     /tmp/build/perf/plugin_function.so
  LINK     /tmp/build/perf/plugin_futex.so
  LD       /tmp/build/perf/plugin_cfg80211-in.o
  LINK     /tmp/build/perf/plugin_xen.so
  LINK     /tmp/build/perf/plugin_scsi.so
  LD       /tmp/build/perf/plugin_tlb-in.o
  MKDIR    /tmp/build/perf/jvmti/
  LINK     /tmp/build/perf/plugin_cfg80211.so
  CC       /tmp/build/perf/jvmti/jvmti_agent.o
  LINK     /tmp/build/perf/plugin_tlb.so
  GEN      perf-with-kcore
  CC       /tmp/build/perf/jvmti/libstring.o
CFLAGS= make -C ../bpf/bpftool \
	OUTPUT=/tmp/build/perf/util/bpf_skel/.tmp/ bootstrap
  CC       /tmp/build/perf/jvmti/libctype.o
  GEN      /tmp/build/perf/libtraceevent-dynamic-list
  LINK     /tmp/build/perf/pmu-events/jevents
  DESCEND  plugins
  GEN      /tmp/build/perf/python/perf.so
  GEN      /tmp/build/perf/pmu-events/pmu-events.c
  CC       /tmp/build/perf/plugins/plugin_jbd2.o
  CC       /tmp/build/perf/plugins/plugin_hrtimer.o
  LD       /tmp/build/perf/plugins/plugin_jbd2-in.o
  LD       /tmp/build/perf/plugins/plugin_hrtimer-in.o
  CC       /tmp/build/perf/plugins/plugin_kmem.o
  CC       /tmp/build/perf/plugins/plugin_kvm.o
  LD       /tmp/build/perf/jvmti/jvmti-in.o
  LD       /tmp/build/perf/plugins/plugin_kmem-in.o
  LINK     /tmp/build/perf/libperf-jvmti.so
  CC       /tmp/build/perf/plugins/plugin_mac80211.o
  CC       /tmp/build/perf/plugins/plugin_sched_switch.o
  CC       /tmp/build/perf/pmu-events/pmu-events.o
  LD       /tmp/build/perf/plugins/plugin_kvm-in.o
  CC       /tmp/build/perf/plugins/plugin_function.o
  LD       /tmp/build/perf/plugins/plugin_mac80211-in.o
  CC       /tmp/build/perf/plugins/plugin_futex.o
  LD       /tmp/build/perf/plugins/plugin_sched_switch-in.o
  CC       /tmp/build/perf/plugins/plugin_xen.o
  LD       /tmp/build/perf/plugins/plugin_function-in.o
  CC       /tmp/build/perf/plugins/plugin_scsi.o
  LD       /tmp/build/perf/plugins/plugin_futex-in.o
  CC       /tmp/build/perf/plugins/plugin_cfg80211.o
  LD       /tmp/build/perf/plugins/plugin_xen-in.o
  CC       /tmp/build/perf/plugins/plugin_tlb.o
  LD       /tmp/build/perf/plugins/plugin_scsi-in.o
  LD       /tmp/build/perf/plugins/plugin_cfg80211-in.o
  LINK     /tmp/build/perf/plugins/plugin_jbd2.so
  LINK     /tmp/build/perf/plugins/plugin_hrtimer.so
  LINK     /tmp/build/perf/plugins/plugin_kmem.so
  LINK     /tmp/build/perf/plugins/plugin_kvm.so
  LINK     /tmp/build/perf/plugins/plugin_mac80211.so
  LD       /tmp/build/perf/plugins/plugin_tlb-in.o
  LINK     /tmp/build/perf/plugins/plugin_function.so
  LINK     /tmp/build/perf/plugins/plugin_sched_switch.so
  LINK     /tmp/build/perf/plugins/plugin_futex.so
  LINK     /tmp/build/perf/plugins/plugin_xen.so
  LINK     /tmp/build/perf/plugins/plugin_scsi.so
  LINK     /tmp/build/perf/plugins/plugin_cfg80211.so
  LINK     /tmp/build/perf/plugins/plugin_tlb.so
  INSTALL  trace_plugins
  LD       /tmp/build/perf/pmu-events/pmu-events-in.o

Auto-detecting system features:
...                        libbfd: [ on  ]
...        disassembler-four-args: [ on  ]
...                          zlib: [ on  ]
...                        libcap: [ on  ]
...               clang-bpf-co-re: [ on  ]

  MKDIR    /tmp/build/perf/util/bpf_skel/.tmp//bootstrap/
  MKDIR    /tmp/build/perf/util/bpf_skel/.tmp//bootstrap/libbpf/
  CC       /tmp/build/perf/util/bpf_skel/.tmp//bootstrap/main.o
  CC       /tmp/build/perf/util/bpf_skel/.tmp//bootstrap/common.o
  CC       /tmp/build/perf/util/bpf_skel/.tmp//bootstrap/json_writer.o
  CC       /tmp/build/perf/util/bpf_skel/.tmp//bootstrap/gen.o
  CC       /tmp/build/perf/util/bpf_skel/.tmp//bootstrap/btf.o
  GEN      /tmp/build/perf/util/bpf_skel/.tmp//bootstrap/libbpf/bpf_helper_defs.h
  MKDIR    /tmp/build/perf/util/bpf_skel/.tmp//bootstrap/libbpf/staticobjs/
  MKDIR    /tmp/build/perf/util/bpf_skel/.tmp//bootstrap/libbpf/staticobjs/
  CC       /tmp/build/perf/util/bpf_skel/.tmp//bootstrap/libbpf/staticobjs/libbpf.o
  CC       /tmp/build/perf/util/bpf_skel/.tmp//bootstrap/libbpf/staticobjs/bpf.o
  MKDIR    /tmp/build/perf/util/bpf_skel/.tmp//bootstrap/libbpf/staticobjs/
  CC       /tmp/build/perf/util/bpf_skel/.tmp//bootstrap/libbpf/staticobjs/nlattr.o
  CC       /tmp/build/perf/util/bpf_skel/.tmp//bootstrap/libbpf/staticobjs/btf.o
  CC       /tmp/build/perf/util/bpf_skel/.tmp//bootstrap/libbpf/staticobjs/libbpf_errno.o
  CC       /tmp/build/perf/util/bpf_skel/.tmp//bootstrap/libbpf/staticobjs/str_error.o
  CC       /tmp/build/perf/util/bpf_skel/.tmp//bootstrap/libbpf/staticobjs/netlink.o
  CC       /tmp/build/perf/util/bpf_skel/.tmp//bootstrap/libbpf/staticobjs/bpf_prog_linfo.o
  CC       /tmp/build/perf/util/bpf_skel/.tmp//bootstrap/libbpf/staticobjs/libbpf_probes.o
  CC       /tmp/build/perf/util/bpf_skel/.tmp//bootstrap/libbpf/staticobjs/xsk.o
  CC       /tmp/build/perf/util/bpf_skel/.tmp//bootstrap/libbpf/staticobjs/hashmap.o
  CC       /tmp/build/perf/util/bpf_skel/.tmp//bootstrap/libbpf/staticobjs/btf_dump.o
  CC       /tmp/build/perf/util/bpf_skel/.tmp//bootstrap/libbpf/staticobjs/ringbuf.o
  LD       /tmp/build/perf/staticobjs/libbpf-in.o
  LINK     /tmp/build/perf/libbpf.a
  CLANG    /tmp/build/perf/util/bpf_skel/.tmp/bpf_prog_profiler.bpf.o
  CLANG    /tmp/build/perf/util/bpf_skel/.tmp/bperf_leader.bpf.o
  CLANG    /tmp/build/perf/util/bpf_skel/.tmp/bperf_follower.bpf.o
  LD       /tmp/build/perf/util/bpf_skel/.tmp//bootstrap/libbpf/staticobjs/libbpf-in.o
  LINK     /tmp/build/perf/util/bpf_skel/.tmp//bootstrap/libbpf/libbpf.a
  LINK     /tmp/build/perf/util/bpf_skel/.tmp//bootstrap/bpftool
  GEN-SKEL /tmp/build/perf/util/bpf_skel/bpf_prog_profiler.skel.h
  GEN-SKEL /tmp/build/perf/util/bpf_skel/bperf_leader.skel.h
  GEN-SKEL /tmp/build/perf/util/bpf_skel/bperf_follower.skel.h
libbpf: map 'prev_readings': unexpected def kind var.
Error: failed to open BPF object file: Invalid argument
libbpf: map 'diff_readings': unexpected def kind var.
Error: failed to open BPF object file: Invalid argument
make[2]: *** [Makefile.perf:1029: /tmp/build/perf/util/bpf_skel/bperf_leader.skel.h] Error 255
make[2]: *** Waiting for unfinished jobs....
make[2]: *** [Makefile.perf:1029: /tmp/build/perf/util/bpf_skel/bperf_follower.skel.h] Error 255
make[1]: *** [Makefile.perf:236: sub-make] Error 2
make: *** [Makefile:110: install-bin] Error 2
make: Leaving directory '/home/acme/git/perf/tools/perf'
[acme@quaco perf]$ clang -v
clang version 11.0.0 (https://github.com/llvm/llvm-project 67420f1b0e9c673ee638f2680fa83f468019004f)
Target: x86_64-unknown-linux-gnu
Thread model: posix
InstalledDir: /usr/local/bin
Found candidate GCC installation: /usr/lib/gcc/x86_64-redhat-linux/10
Selected GCC installation: /usr/lib/gcc/x86_64-redhat-linux/10
Candidate multilib: .;@m64
Candidate multilib: 32;@m32
Selected multilib: .;@m64
[acme@quaco perf]$

Re: [PATCH v2 1/3] perf-stat: introduce bperf, share hardware PMCs with BPF

[Jiri Olsa]

On Fri, Mar 19, 2021 at 03:41:57PM -0300, Arnaldo Carvalho de Melo wrote:

SNIP

>   LD       /tmp/build/perf/util/bpf_skel/.tmp//bootstrap/libbpf/staticobjs/libbpf-in.o
>   LINK     /tmp/build/perf/util/bpf_skel/.tmp//bootstrap/libbpf/libbpf.a
>   LINK     /tmp/build/perf/util/bpf_skel/.tmp//bootstrap/bpftool
>   GEN-SKEL /tmp/build/perf/util/bpf_skel/bpf_prog_profiler.skel.h
>   GEN-SKEL /tmp/build/perf/util/bpf_skel/bperf_leader.skel.h
>   GEN-SKEL /tmp/build/perf/util/bpf_skel/bperf_follower.skel.h
> libbpf: map 'prev_readings': unexpected def kind var.
> Error: failed to open BPF object file: Invalid argument
> libbpf: map 'diff_readings': unexpected def kind var.
> Error: failed to open BPF object file: Invalid argument

I'm getting clean build for the same options,
could you please send the same output also with 'JOBS=1 V=1'


> make[2]: *** [Makefile.perf:1029: /tmp/build/perf/util/bpf_skel/bperf_leader.skel.h] Error 255
> make[2]: *** Waiting for unfinished jobs....
> make[2]: *** [Makefile.perf:1029: /tmp/build/perf/util/bpf_skel/bperf_follower.skel.h] Error 255
> make[1]: *** [Makefile.perf:236: sub-make] Error 2
> make: *** [Makefile:110: install-bin] Error 2
> make: Leaving directory '/home/acme/git/perf/tools/perf'
> [acme@quaco perf]$ clang -v
> clang version 11.0.0 (https://github.com/llvm/llvm-project 67420f1b0e9c673ee638f2680fa83f468019004f)
> Target: x86_64-unknown-linux-gnu
> Thread model: posix
> InstalledDir: /usr/local/bin
> Found candidate GCC installation: /usr/lib/gcc/x86_64-redhat-linux/10
> Selected GCC installation: /usr/lib/gcc/x86_64-redhat-linux/10
> Candidate multilib: .;@m64
> Candidate multilib: 32;@m32
> Selected multilib: .;@m64
> [acme@quaco perf]$
> 

I have:

[jolsa@dell-r440-01 linux-perf]$ clang --version
clang version 11.0.0 (Fedora 11.0.0-2.fc33)
Target: x86_64-unknown-linux-gnu
Thread model: posix
InstalledDir: /usr/bin


jirka

Re: [PATCH v2 1/3] perf-stat: introduce bperf, share hardware PMCs with BPF

[Song Liu]

> On Mar 19, 2021, at 11:55 AM, Jiri Olsa <jolsa@redhat.com> wrote:
> 
> On Fri, Mar 19, 2021 at 03:41:57PM -0300, Arnaldo Carvalho de Melo wrote:
> 
> SNIP
> 
>>  LD       /tmp/build/perf/util/bpf_skel/.tmp//bootstrap/libbpf/staticobjs/libbpf-in.o
>>  LINK     /tmp/build/perf/util/bpf_skel/.tmp//bootstrap/libbpf/libbpf.a
>>  LINK     /tmp/build/perf/util/bpf_skel/.tmp//bootstrap/bpftool
>>  GEN-SKEL /tmp/build/perf/util/bpf_skel/bpf_prog_profiler.skel.h
>>  GEN-SKEL /tmp/build/perf/util/bpf_skel/bperf_leader.skel.h
>>  GEN-SKEL /tmp/build/perf/util/bpf_skel/bperf_follower.skel.h
>> libbpf: map 'prev_readings': unexpected def kind var.
>> Error: failed to open BPF object file: Invalid argument
>> libbpf: map 'diff_readings': unexpected def kind var.
>> Error: failed to open BPF object file: Invalid argument
> 
> I'm getting clean build for the same options,
> could you please send the same output also with 'JOBS=1 V=1'
> 
> 
>> make[2]: *** [Makefile.perf:1029: /tmp/build/perf/util/bpf_skel/bperf_leader.skel.h] Error 255
>> make[2]: *** Waiting for unfinished jobs....
>> make[2]: *** [Makefile.perf:1029: /tmp/build/perf/util/bpf_skel/bperf_follower.skel.h] Error 255
>> make[1]: *** [Makefile.perf:236: sub-make] Error 2
>> make: *** [Makefile:110: install-bin] Error 2
>> make: Leaving directory '/home/acme/git/perf/tools/perf'
>> [acme@quaco perf]$ clang -v
>> clang version 11.0.0 (https://github.com/llvm/llvm-project 67420f1b0e9c673ee638f2680fa83f468019004f)
>> Target: x86_64-unknown-linux-gnu
>> Thread model: posix
>> InstalledDir: /usr/local/bin
>> Found candidate GCC installation: /usr/lib/gcc/x86_64-redhat-linux/10
>> Selected GCC installation: /usr/lib/gcc/x86_64-redhat-linux/10
>> Candidate multilib: .;@m64
>> Candidate multilib: 32;@m32
>> Selected multilib: .;@m64
>> [acme@quaco perf]$
>> 
> 
> I have:
> 
> [jolsa@dell-r440-01 linux-perf]$ clang --version
> clang version 11.0.0 (Fedora 11.0.0-2.fc33)
> Target: x86_64-unknown-linux-gnu
> Thread model: posix
> InstalledDir: /usr/bin

I am not able to repro this error either. I tried two versions of clang:

clang version 11.0.0 (Red Hat 11.0.0-0.2.rc2.module_el8.4.0+533+50191577)
Target: x86_64-unknown-linux-gnu
Thread model: posix
InstalledDir: /bin

clang version 12.0.0 (https://github.com/llvm/llvm-project.git 07f1e1f44c87d1ee84caf13d6e5aa64eb7e1b068)
Target: x86_64-unknown-linux-gnu
Thread model: posix
InstalledDir: /usr/local/bin

Thanks,
Song

Re: [PATCH v2 1/3] perf-stat: introduce bperf, share hardware PMCs with BPF

[Song Liu]

> On Mar 19, 2021, at 11:41 AM, Arnaldo Carvalho de Melo <acme@kernel.org> wrote:
> 
> Em Thu, Mar 18, 2021 at 10:15:13PM +0100, Jiri Olsa escreveu:
>> On Tue, Mar 16, 2021 at 02:18:35PM -0700, Song Liu wrote:
>>> bperf is off by default. To enable it, pass --bpf-counters option to
>>> perf-stat. bperf uses a BPF hashmap to share information about BPF
>>> programs and maps used by bperf. This map is pinned to bpffs. The default
>>> path is /sys/fs/bpf/perf_attr_map. The user could change the path with
>>> option --bpf-attr-map.
>>> 
>>> Signed-off-by: Song Liu <songliubraving@fb.com>
>> 
>> Reviewed-by: Jiri Olsa <jolsa@redhat.com>
> 
> After applying just this first patch in the series I'm getting this
> after a 'make -C tools/ clean', now I'm checking if I need some new
> clang, ideas?
> 
> - Arnaldo

Hi Arnaldo, 

Are you still getting this error? 

Thanks,
Song

> 
> [acme@quaco perf]$ make O=/tmp/build/perf -C tools/perf BUILD_BPF_SKEL=1 PYTHON=python3 install-bin
> make: Entering directory '/home/acme/git/perf/tools/perf'
>  BUILD:   Doing 'make -j8' parallel build
> Warning: Kernel ABI header at 'tools/include/uapi/linux/kvm.h' differs from latest version at 'include/uapi/linux/kvm.h'
> diff -u tools/include/uapi/linux/kvm.h include/uapi/linux/kvm.h
> Warning: Kernel ABI header at 'tools/perf/arch/mips/entry/syscalls/syscall_n64.tbl' differs from latest version at 'arch/mips/kernel/syscalls/syscall_n64.tbl'
> diff -u tools/perf/arch/mips/entry/syscalls/syscall_n64.tbl arch/mips/kernel/syscalls/syscall_n64.tbl
> 
> Auto-detecting system features:
> ...                         dwarf: [ on  ]
> ...            dwarf_getlocations: [ on  ]
> ...                         glibc: [ on  ]
> ...                        libbfd: [ on  ]
> ...                libbfd-buildid: [ on  ]
> ...                        libcap: [ on  ]
> ...                        libelf: [ on  ]
> ...                       libnuma: [ on  ]
> ...        numa_num_possible_cpus: [ on  ]
> ...                       libperl: [ on  ]
> ...                     libpython: [ on  ]
> ...                     libcrypto: [ on  ]
> ...                     libunwind: [ on  ]
> ...            libdw-dwarf-unwind: [ on  ]
> ...                          zlib: [ on  ]
> ...                          lzma: [ on  ]
> ...                     get_cpuid: [ on  ]
> ...                           bpf: [ on  ]
> ...                        libaio: [ on  ]
> ...                       libzstd: [ on  ]
> ...        disassembler-four-args: [ on  ]
> 
>  GEN      /tmp/build/perf/common-cmds.h
>  CC       /tmp/build/perf/exec-cmd.o
>  MKDIR    /tmp/build/perf/fd/
>  MKDIR    /tmp/build/perf/fs/
>  CC       /tmp/build/perf/fs/fs.o
>  CC       /tmp/build/perf/event-parse.o
>  CC       /tmp/build/perf/fd/array.o
>  CC       /tmp/build/perf/core.o
>  GEN      /tmp/build/perf/bpf_helper_defs.h
>  CC       /tmp/build/perf/event-plugin.o
>  MKDIR    /tmp/build/perf/staticobjs/
>  PERF_VERSION = 5.12.rc2.g3df07f57f205
>  CC       /tmp/build/perf/staticobjs/libbpf.o
>  CC       /tmp/build/perf/cpu.o
>  LD       /tmp/build/perf/fd/libapi-in.o
>  CC       /tmp/build/perf/cpumap.o
>  CC       /tmp/build/perf/help.o
>  MKDIR    /tmp/build/perf/fs/
>  CC       /tmp/build/perf/fs/tracing_path.o
>  CC       /tmp/build/perf/fs/cgroup.o
>  CC       /tmp/build/perf/trace-seq.o
>  CC       /tmp/build/perf/pager.o
>  CC       /tmp/build/perf/parse-options.o
>  LD       /tmp/build/perf/fs/libapi-in.o
>  CC       /tmp/build/perf/debug.o
>  CC       /tmp/build/perf/str_error_r.o
>  CC       /tmp/build/perf/run-command.o
>  CC       /tmp/build/perf/sigchain.o
>  LD       /tmp/build/perf/libapi-in.o
>  AR       /tmp/build/perf/libapi.a
>  CC       /tmp/build/perf/subcmd-config.o
>  CC       /tmp/build/perf/threadmap.o
>  CC       /tmp/build/perf/evsel.o
>  CC       /tmp/build/perf/parse-filter.o
>  MKDIR    /tmp/build/perf/staticobjs/
>  CC       /tmp/build/perf/staticobjs/bpf.o
>  CC       /tmp/build/perf/evlist.o
>  CC       /tmp/build/perf/parse-utils.o
>  CC       /tmp/build/perf/kbuffer-parse.o
>  CC       /tmp/build/perf/tep_strerror.o
>  CC       /tmp/build/perf/mmap.o
>  CC       /tmp/build/perf/zalloc.o
>  CC       /tmp/build/perf/event-parse-api.o
>  LD       /tmp/build/perf/libsubcmd-in.o
>  AR       /tmp/build/perf/libsubcmd.a
>  CC       /tmp/build/perf/xyarray.o
>  LD       /tmp/build/perf/libtraceevent-in.o
>  LINK     /tmp/build/perf/libtraceevent.a
>  CC       /tmp/build/perf/staticobjs/nlattr.o
>  CC       /tmp/build/perf/staticobjs/btf.o
>  CC       /tmp/build/perf/lib.o
>  CC       /tmp/build/perf/staticobjs/libbpf_errno.o
>  CC       /tmp/build/perf/staticobjs/str_error.o
>  CC       /tmp/build/perf/staticobjs/netlink.o
>  CC       /tmp/build/perf/staticobjs/bpf_prog_linfo.o
>  CC       /tmp/build/perf/staticobjs/libbpf_probes.o
>  LD       /tmp/build/perf/libperf-in.o
>  AR       /tmp/build/perf/libperf.a
>  MKDIR    /tmp/build/perf/pmu-events/
>  HOSTCC   /tmp/build/perf/pmu-events/json.o
>  CC       /tmp/build/perf/plugin_jbd2.o
>  CC       /tmp/build/perf/staticobjs/xsk.o
>  MKDIR    /tmp/build/perf/pmu-events/
>  HOSTCC   /tmp/build/perf/pmu-events/jsmn.o
>  CC       /tmp/build/perf/staticobjs/hashmap.o
>  LD       /tmp/build/perf/plugin_jbd2-in.o
>  CC       /tmp/build/perf/staticobjs/btf_dump.o
>  CC       /tmp/build/perf/plugin_hrtimer.o
>  HOSTCC   /tmp/build/perf/pmu-events/jevents.o
>  LD       /tmp/build/perf/plugin_hrtimer-in.o
>  CC       /tmp/build/perf/plugin_kmem.o
>  CC       /tmp/build/perf/staticobjs/ringbuf.o
>  LD       /tmp/build/perf/plugin_kmem-in.o
>  CC       /tmp/build/perf/plugin_kvm.o
>  HOSTLD   /tmp/build/perf/pmu-events/jevents-in.o
>  CC       /tmp/build/perf/perf-read-vdso32
>  CC       /tmp/build/perf/plugin_mac80211.o
>  LD       /tmp/build/perf/plugin_kvm-in.o
>  CC       /tmp/build/perf/plugin_sched_switch.o
>  CC       /tmp/build/perf/plugin_function.o
>  MKDIR    /tmp/build/perf/jvmti/
>  CC       /tmp/build/perf/jvmti/libjvmti.o
>  LD       /tmp/build/perf/plugin_mac80211-in.o
>  GEN      perf-archive
>  LD       /tmp/build/perf/plugin_sched_switch-in.o
>  CC       /tmp/build/perf/plugin_futex.o
>  CC       /tmp/build/perf/plugin_xen.o
>  CC       /tmp/build/perf/plugin_scsi.o
>  LD       /tmp/build/perf/plugin_function-in.o
>  CC       /tmp/build/perf/plugin_cfg80211.o
>  LD       /tmp/build/perf/plugin_futex-in.o
>  LD       /tmp/build/perf/plugin_xen-in.o
>  LINK     /tmp/build/perf/plugin_jbd2.so
>  LINK     /tmp/build/perf/plugin_hrtimer.so
>  CC       /tmp/build/perf/plugin_tlb.o
>  LINK     /tmp/build/perf/plugin_kmem.so
>  LINK     /tmp/build/perf/plugin_kvm.so
>  LD       /tmp/build/perf/plugin_scsi-in.o
>  LINK     /tmp/build/perf/plugin_mac80211.so
>  LINK     /tmp/build/perf/plugin_sched_switch.so
>  LINK     /tmp/build/perf/plugin_function.so
>  LINK     /tmp/build/perf/plugin_futex.so
>  LD       /tmp/build/perf/plugin_cfg80211-in.o
>  LINK     /tmp/build/perf/plugin_xen.so
>  LINK     /tmp/build/perf/plugin_scsi.so
>  LD       /tmp/build/perf/plugin_tlb-in.o
>  MKDIR    /tmp/build/perf/jvmti/
>  LINK     /tmp/build/perf/plugin_cfg80211.so
>  CC       /tmp/build/perf/jvmti/jvmti_agent.o
>  LINK     /tmp/build/perf/plugin_tlb.so
>  GEN      perf-with-kcore
>  CC       /tmp/build/perf/jvmti/libstring.o
> CFLAGS= make -C ../bpf/bpftool \
> 	OUTPUT=/tmp/build/perf/util/bpf_skel/.tmp/ bootstrap
>  CC       /tmp/build/perf/jvmti/libctype.o
>  GEN      /tmp/build/perf/libtraceevent-dynamic-list
>  LINK     /tmp/build/perf/pmu-events/jevents
>  DESCEND  plugins
>  GEN      /tmp/build/perf/python/perf.so
>  GEN      /tmp/build/perf/pmu-events/pmu-events.c
>  CC       /tmp/build/perf/plugins/plugin_jbd2.o
>  CC       /tmp/build/perf/plugins/plugin_hrtimer.o
>  LD       /tmp/build/perf/plugins/plugin_jbd2-in.o
>  LD       /tmp/build/perf/plugins/plugin_hrtimer-in.o
>  CC       /tmp/build/perf/plugins/plugin_kmem.o
>  CC       /tmp/build/perf/plugins/plugin_kvm.o
>  LD       /tmp/build/perf/jvmti/jvmti-in.o
>  LD       /tmp/build/perf/plugins/plugin_kmem-in.o
>  LINK     /tmp/build/perf/libperf-jvmti.so
>  CC       /tmp/build/perf/plugins/plugin_mac80211.o
>  CC       /tmp/build/perf/plugins/plugin_sched_switch.o
>  CC       /tmp/build/perf/pmu-events/pmu-events.o
>  LD       /tmp/build/perf/plugins/plugin_kvm-in.o
>  CC       /tmp/build/perf/plugins/plugin_function.o
>  LD       /tmp/build/perf/plugins/plugin_mac80211-in.o
>  CC       /tmp/build/perf/plugins/plugin_futex.o
>  LD       /tmp/build/perf/plugins/plugin_sched_switch-in.o
>  CC       /tmp/build/perf/plugins/plugin_xen.o
>  LD       /tmp/build/perf/plugins/plugin_function-in.o
>  CC       /tmp/build/perf/plugins/plugin_scsi.o
>  LD       /tmp/build/perf/plugins/plugin_futex-in.o
>  CC       /tmp/build/perf/plugins/plugin_cfg80211.o
>  LD       /tmp/build/perf/plugins/plugin_xen-in.o
>  CC       /tmp/build/perf/plugins/plugin_tlb.o
>  LD       /tmp/build/perf/plugins/plugin_scsi-in.o
>  LD       /tmp/build/perf/plugins/plugin_cfg80211-in.o
>  LINK     /tmp/build/perf/plugins/plugin_jbd2.so
>  LINK     /tmp/build/perf/plugins/plugin_hrtimer.so
>  LINK     /tmp/build/perf/plugins/plugin_kmem.so
>  LINK     /tmp/build/perf/plugins/plugin_kvm.so
>  LINK     /tmp/build/perf/plugins/plugin_mac80211.so
>  LD       /tmp/build/perf/plugins/plugin_tlb-in.o
>  LINK     /tmp/build/perf/plugins/plugin_function.so
>  LINK     /tmp/build/perf/plugins/plugin_sched_switch.so
>  LINK     /tmp/build/perf/plugins/plugin_futex.so
>  LINK     /tmp/build/perf/plugins/plugin_xen.so
>  LINK     /tmp/build/perf/plugins/plugin_scsi.so
>  LINK     /tmp/build/perf/plugins/plugin_cfg80211.so
>  LINK     /tmp/build/perf/plugins/plugin_tlb.so
>  INSTALL  trace_plugins
>  LD       /tmp/build/perf/pmu-events/pmu-events-in.o
> 
> Auto-detecting system features:
> ...                        libbfd: [ on  ]
> ...        disassembler-four-args: [ on  ]
> ...                          zlib: [ on  ]
> ...                        libcap: [ on  ]
> ...               clang-bpf-co-re: [ on  ]
> 
>  MKDIR    /tmp/build/perf/util/bpf_skel/.tmp//bootstrap/
>  MKDIR    /tmp/build/perf/util/bpf_skel/.tmp//bootstrap/libbpf/
>  CC       /tmp/build/perf/util/bpf_skel/.tmp//bootstrap/main.o
>  CC       /tmp/build/perf/util/bpf_skel/.tmp//bootstrap/common.o
>  CC       /tmp/build/perf/util/bpf_skel/.tmp//bootstrap/json_writer.o
>  CC       /tmp/build/perf/util/bpf_skel/.tmp//bootstrap/gen.o
>  CC       /tmp/build/perf/util/bpf_skel/.tmp//bootstrap/btf.o
>  GEN      /tmp/build/perf/util/bpf_skel/.tmp//bootstrap/libbpf/bpf_helper_defs.h
>  MKDIR    /tmp/build/perf/util/bpf_skel/.tmp//bootstrap/libbpf/staticobjs/
>  MKDIR    /tmp/build/perf/util/bpf_skel/.tmp//bootstrap/libbpf/staticobjs/
>  CC       /tmp/build/perf/util/bpf_skel/.tmp//bootstrap/libbpf/staticobjs/libbpf.o
>  CC       /tmp/build/perf/util/bpf_skel/.tmp//bootstrap/libbpf/staticobjs/bpf.o
>  MKDIR    /tmp/build/perf/util/bpf_skel/.tmp//bootstrap/libbpf/staticobjs/
>  CC       /tmp/build/perf/util/bpf_skel/.tmp//bootstrap/libbpf/staticobjs/nlattr.o
>  CC       /tmp/build/perf/util/bpf_skel/.tmp//bootstrap/libbpf/staticobjs/btf.o
>  CC       /tmp/build/perf/util/bpf_skel/.tmp//bootstrap/libbpf/staticobjs/libbpf_errno.o
>  CC       /tmp/build/perf/util/bpf_skel/.tmp//bootstrap/libbpf/staticobjs/str_error.o
>  CC       /tmp/build/perf/util/bpf_skel/.tmp//bootstrap/libbpf/staticobjs/netlink.o
>  CC       /tmp/build/perf/util/bpf_skel/.tmp//bootstrap/libbpf/staticobjs/bpf_prog_linfo.o
>  CC       /tmp/build/perf/util/bpf_skel/.tmp//bootstrap/libbpf/staticobjs/libbpf_probes.o
>  CC       /tmp/build/perf/util/bpf_skel/.tmp//bootstrap/libbpf/staticobjs/xsk.o
>  CC       /tmp/build/perf/util/bpf_skel/.tmp//bootstrap/libbpf/staticobjs/hashmap.o
>  CC       /tmp/build/perf/util/bpf_skel/.tmp//bootstrap/libbpf/staticobjs/btf_dump.o
>  CC       /tmp/build/perf/util/bpf_skel/.tmp//bootstrap/libbpf/staticobjs/ringbuf.o
>  LD       /tmp/build/perf/staticobjs/libbpf-in.o
>  LINK     /tmp/build/perf/libbpf.a
>  CLANG    /tmp/build/perf/util/bpf_skel/.tmp/bpf_prog_profiler.bpf.o
>  CLANG    /tmp/build/perf/util/bpf_skel/.tmp/bperf_leader.bpf.o
>  CLANG    /tmp/build/perf/util/bpf_skel/.tmp/bperf_follower.bpf.o
>  LD       /tmp/build/perf/util/bpf_skel/.tmp//bootstrap/libbpf/staticobjs/libbpf-in.o
>  LINK     /tmp/build/perf/util/bpf_skel/.tmp//bootstrap/libbpf/libbpf.a
>  LINK     /tmp/build/perf/util/bpf_skel/.tmp//bootstrap/bpftool
>  GEN-SKEL /tmp/build/perf/util/bpf_skel/bpf_prog_profiler.skel.h
>  GEN-SKEL /tmp/build/perf/util/bpf_skel/bperf_leader.skel.h
>  GEN-SKEL /tmp/build/perf/util/bpf_skel/bperf_follower.skel.h
> libbpf: map 'prev_readings': unexpected def kind var.
> Error: failed to open BPF object file: Invalid argument
> libbpf: map 'diff_readings': unexpected def kind var.
> Error: failed to open BPF object file: Invalid argument
> make[2]: *** [Makefile.perf:1029: /tmp/build/perf/util/bpf_skel/bperf_leader.skel.h] Error 255
> make[2]: *** Waiting for unfinished jobs....
> make[2]: *** [Makefile.perf:1029: /tmp/build/perf/util/bpf_skel/bperf_follower.skel.h] Error 255
> make[1]: *** [Makefile.perf:236: sub-make] Error 2
> make: *** [Makefile:110: install-bin] Error 2
> make: Leaving directory '/home/acme/git/perf/tools/perf'
> [acme@quaco perf]$ clang -v
> clang version 11.0.0 (https://github.com/llvm/llvm-project 67420f1b0e9c673ee638f2680fa83f468019004f)
> Target: x86_64-unknown-linux-gnu
> Thread model: posix
> InstalledDir: /usr/local/bin
> Found candidate GCC installation: /usr/lib/gcc/x86_64-redhat-linux/10
> Selected GCC installation: /usr/lib/gcc/x86_64-redhat-linux/10
> Candidate multilib: .;@m64
> Candidate multilib: 32;@m32
> Selected multilib: .;@m64
> [acme@quaco perf]$

Re: [PATCH v2 1/3] perf-stat: introduce bperf, share hardware PMCs with BPF

[Arnaldo Carvalho de Melo]

Em Fri, Mar 19, 2021 at 03:41:57PM -0300, Arnaldo Carvalho de Melo escreveu:
> Em Thu, Mar 18, 2021 at 10:15:13PM +0100, Jiri Olsa escreveu:
> > On Tue, Mar 16, 2021 at 02:18:35PM -0700, Song Liu wrote:
> > > bperf is off by default. To enable it, pass --bpf-counters option to
> > > perf-stat. bperf uses a BPF hashmap to share information about BPF
> > > programs and maps used by bperf. This map is pinned to bpffs. The default
> > > path is /sys/fs/bpf/perf_attr_map. The user could change the path with
> > > option --bpf-attr-map.
> > > 
> > > Signed-off-by: Song Liu <songliubraving@fb.com>
> > 
> > Reviewed-by: Jiri Olsa <jolsa@redhat.com>
> 
> After applying just this first patch in the series I'm getting this
> after a 'make -C tools/ clean', now I'm checking if I need some new
> clang, ideas?

Works now with clang from fedora 33, I was using a locally built, older,
now I get this when trying as non-root, expected, but we need to improve
the wording.

[acme@five perf]$ perf stat --bpf-counters sleep 1
Failed to lock perf_event_attr map
[acme@five perf]$
 
> - Arnaldo
> 
> [acme@quaco perf]$ make O=/tmp/build/perf -C tools/perf BUILD_BPF_SKEL=1 PYTHON=python3 install-bin
> make: Entering directory '/home/acme/git/perf/tools/perf'
>   BUILD:   Doing 'make -j8' parallel build
> Warning: Kernel ABI header at 'tools/include/uapi/linux/kvm.h' differs from latest version at 'include/uapi/linux/kvm.h'
> diff -u tools/include/uapi/linux/kvm.h include/uapi/linux/kvm.h
> Warning: Kernel ABI header at 'tools/perf/arch/mips/entry/syscalls/syscall_n64.tbl' differs from latest version at 'arch/mips/kernel/syscalls/syscall_n64.tbl'
> diff -u tools/perf/arch/mips/entry/syscalls/syscall_n64.tbl arch/mips/kernel/syscalls/syscall_n64.tbl
> 
> Auto-detecting system features:
> ...                         dwarf: [ on  ]
> ...            dwarf_getlocations: [ on  ]
> ...                         glibc: [ on  ]
> ...                        libbfd: [ on  ]
> ...                libbfd-buildid: [ on  ]
> ...                        libcap: [ on  ]
> ...                        libelf: [ on  ]
> ...                       libnuma: [ on  ]
> ...        numa_num_possible_cpus: [ on  ]
> ...                       libperl: [ on  ]
> ...                     libpython: [ on  ]
> ...                     libcrypto: [ on  ]
> ...                     libunwind: [ on  ]
> ...            libdw-dwarf-unwind: [ on  ]
> ...                          zlib: [ on  ]
> ...                          lzma: [ on  ]
> ...                     get_cpuid: [ on  ]
> ...                           bpf: [ on  ]
> ...                        libaio: [ on  ]
> ...                       libzstd: [ on  ]
> ...        disassembler-four-args: [ on  ]
> 
>   GEN      /tmp/build/perf/common-cmds.h
>   CC       /tmp/build/perf/exec-cmd.o
>   MKDIR    /tmp/build/perf/fd/
>   MKDIR    /tmp/build/perf/fs/
>   CC       /tmp/build/perf/fs/fs.o
>   CC       /tmp/build/perf/event-parse.o
>   CC       /tmp/build/perf/fd/array.o
>   CC       /tmp/build/perf/core.o
>   GEN      /tmp/build/perf/bpf_helper_defs.h
>   CC       /tmp/build/perf/event-plugin.o
>   MKDIR    /tmp/build/perf/staticobjs/
>   PERF_VERSION = 5.12.rc2.g3df07f57f205
>   CC       /tmp/build/perf/staticobjs/libbpf.o
>   CC       /tmp/build/perf/cpu.o
>   LD       /tmp/build/perf/fd/libapi-in.o
>   CC       /tmp/build/perf/cpumap.o
>   CC       /tmp/build/perf/help.o
>   MKDIR    /tmp/build/perf/fs/
>   CC       /tmp/build/perf/fs/tracing_path.o
>   CC       /tmp/build/perf/fs/cgroup.o
>   CC       /tmp/build/perf/trace-seq.o
>   CC       /tmp/build/perf/pager.o
>   CC       /tmp/build/perf/parse-options.o
>   LD       /tmp/build/perf/fs/libapi-in.o
>   CC       /tmp/build/perf/debug.o
>   CC       /tmp/build/perf/str_error_r.o
>   CC       /tmp/build/perf/run-command.o
>   CC       /tmp/build/perf/sigchain.o
>   LD       /tmp/build/perf/libapi-in.o
>   AR       /tmp/build/perf/libapi.a
>   CC       /tmp/build/perf/subcmd-config.o
>   CC       /tmp/build/perf/threadmap.o
>   CC       /tmp/build/perf/evsel.o
>   CC       /tmp/build/perf/parse-filter.o
>   MKDIR    /tmp/build/perf/staticobjs/
>   CC       /tmp/build/perf/staticobjs/bpf.o
>   CC       /tmp/build/perf/evlist.o
>   CC       /tmp/build/perf/parse-utils.o
>   CC       /tmp/build/perf/kbuffer-parse.o
>   CC       /tmp/build/perf/tep_strerror.o
>   CC       /tmp/build/perf/mmap.o
>   CC       /tmp/build/perf/zalloc.o
>   CC       /tmp/build/perf/event-parse-api.o
>   LD       /tmp/build/perf/libsubcmd-in.o
>   AR       /tmp/build/perf/libsubcmd.a
>   CC       /tmp/build/perf/xyarray.o
>   LD       /tmp/build/perf/libtraceevent-in.o
>   LINK     /tmp/build/perf/libtraceevent.a
>   CC       /tmp/build/perf/staticobjs/nlattr.o
>   CC       /tmp/build/perf/staticobjs/btf.o
>   CC       /tmp/build/perf/lib.o
>   CC       /tmp/build/perf/staticobjs/libbpf_errno.o
>   CC       /tmp/build/perf/staticobjs/str_error.o
>   CC       /tmp/build/perf/staticobjs/netlink.o
>   CC       /tmp/build/perf/staticobjs/bpf_prog_linfo.o
>   CC       /tmp/build/perf/staticobjs/libbpf_probes.o
>   LD       /tmp/build/perf/libperf-in.o
>   AR       /tmp/build/perf/libperf.a
>   MKDIR    /tmp/build/perf/pmu-events/
>   HOSTCC   /tmp/build/perf/pmu-events/json.o
>   CC       /tmp/build/perf/plugin_jbd2.o
>   CC       /tmp/build/perf/staticobjs/xsk.o
>   MKDIR    /tmp/build/perf/pmu-events/
>   HOSTCC   /tmp/build/perf/pmu-events/jsmn.o
>   CC       /tmp/build/perf/staticobjs/hashmap.o
>   LD       /tmp/build/perf/plugin_jbd2-in.o
>   CC       /tmp/build/perf/staticobjs/btf_dump.o
>   CC       /tmp/build/perf/plugin_hrtimer.o
>   HOSTCC   /tmp/build/perf/pmu-events/jevents.o
>   LD       /tmp/build/perf/plugin_hrtimer-in.o
>   CC       /tmp/build/perf/plugin_kmem.o
>   CC       /tmp/build/perf/staticobjs/ringbuf.o
>   LD       /tmp/build/perf/plugin_kmem-in.o
>   CC       /tmp/build/perf/plugin_kvm.o
>   HOSTLD   /tmp/build/perf/pmu-events/jevents-in.o
>   CC       /tmp/build/perf/perf-read-vdso32
>   CC       /tmp/build/perf/plugin_mac80211.o
>   LD       /tmp/build/perf/plugin_kvm-in.o
>   CC       /tmp/build/perf/plugin_sched_switch.o
>   CC       /tmp/build/perf/plugin_function.o
>   MKDIR    /tmp/build/perf/jvmti/
>   CC       /tmp/build/perf/jvmti/libjvmti.o
>   LD       /tmp/build/perf/plugin_mac80211-in.o
>   GEN      perf-archive
>   LD       /tmp/build/perf/plugin_sched_switch-in.o
>   CC       /tmp/build/perf/plugin_futex.o
>   CC       /tmp/build/perf/plugin_xen.o
>   CC       /tmp/build/perf/plugin_scsi.o
>   LD       /tmp/build/perf/plugin_function-in.o
>   CC       /tmp/build/perf/plugin_cfg80211.o
>   LD       /tmp/build/perf/plugin_futex-in.o
>   LD       /tmp/build/perf/plugin_xen-in.o
>   LINK     /tmp/build/perf/plugin_jbd2.so
>   LINK     /tmp/build/perf/plugin_hrtimer.so
>   CC       /tmp/build/perf/plugin_tlb.o
>   LINK     /tmp/build/perf/plugin_kmem.so
>   LINK     /tmp/build/perf/plugin_kvm.so
>   LD       /tmp/build/perf/plugin_scsi-in.o
>   LINK     /tmp/build/perf/plugin_mac80211.so
>   LINK     /tmp/build/perf/plugin_sched_switch.so
>   LINK     /tmp/build/perf/plugin_function.so
>   LINK     /tmp/build/perf/plugin_futex.so
>   LD       /tmp/build/perf/plugin_cfg80211-in.o
>   LINK     /tmp/build/perf/plugin_xen.so
>   LINK     /tmp/build/perf/plugin_scsi.so
>   LD       /tmp/build/perf/plugin_tlb-in.o
>   MKDIR    /tmp/build/perf/jvmti/
>   LINK     /tmp/build/perf/plugin_cfg80211.so
>   CC       /tmp/build/perf/jvmti/jvmti_agent.o
>   LINK     /tmp/build/perf/plugin_tlb.so
>   GEN      perf-with-kcore
>   CC       /tmp/build/perf/jvmti/libstring.o
> CFLAGS= make -C ../bpf/bpftool \
> 	OUTPUT=/tmp/build/perf/util/bpf_skel/.tmp/ bootstrap
>   CC       /tmp/build/perf/jvmti/libctype.o
>   GEN      /tmp/build/perf/libtraceevent-dynamic-list
>   LINK     /tmp/build/perf/pmu-events/jevents
>   DESCEND  plugins
>   GEN      /tmp/build/perf/python/perf.so
>   GEN      /tmp/build/perf/pmu-events/pmu-events.c
>   CC       /tmp/build/perf/plugins/plugin_jbd2.o
>   CC       /tmp/build/perf/plugins/plugin_hrtimer.o
>   LD       /tmp/build/perf/plugins/plugin_jbd2-in.o
>   LD       /tmp/build/perf/plugins/plugin_hrtimer-in.o
>   CC       /tmp/build/perf/plugins/plugin_kmem.o
>   CC       /tmp/build/perf/plugins/plugin_kvm.o
>   LD       /tmp/build/perf/jvmti/jvmti-in.o
>   LD       /tmp/build/perf/plugins/plugin_kmem-in.o
>   LINK     /tmp/build/perf/libperf-jvmti.so
>   CC       /tmp/build/perf/plugins/plugin_mac80211.o
>   CC       /tmp/build/perf/plugins/plugin_sched_switch.o
>   CC       /tmp/build/perf/pmu-events/pmu-events.o
>   LD       /tmp/build/perf/plugins/plugin_kvm-in.o
>   CC       /tmp/build/perf/plugins/plugin_function.o
>   LD       /tmp/build/perf/plugins/plugin_mac80211-in.o
>   CC       /tmp/build/perf/plugins/plugin_futex.o
>   LD       /tmp/build/perf/plugins/plugin_sched_switch-in.o
>   CC       /tmp/build/perf/plugins/plugin_xen.o
>   LD       /tmp/build/perf/plugins/plugin_function-in.o
>   CC       /tmp/build/perf/plugins/plugin_scsi.o
>   LD       /tmp/build/perf/plugins/plugin_futex-in.o
>   CC       /tmp/build/perf/plugins/plugin_cfg80211.o
>   LD       /tmp/build/perf/plugins/plugin_xen-in.o
>   CC       /tmp/build/perf/plugins/plugin_tlb.o
>   LD       /tmp/build/perf/plugins/plugin_scsi-in.o
>   LD       /tmp/build/perf/plugins/plugin_cfg80211-in.o
>   LINK     /tmp/build/perf/plugins/plugin_jbd2.so
>   LINK     /tmp/build/perf/plugins/plugin_hrtimer.so
>   LINK     /tmp/build/perf/plugins/plugin_kmem.so
>   LINK     /tmp/build/perf/plugins/plugin_kvm.so
>   LINK     /tmp/build/perf/plugins/plugin_mac80211.so
>   LD       /tmp/build/perf/plugins/plugin_tlb-in.o
>   LINK     /tmp/build/perf/plugins/plugin_function.so
>   LINK     /tmp/build/perf/plugins/plugin_sched_switch.so
>   LINK     /tmp/build/perf/plugins/plugin_futex.so
>   LINK     /tmp/build/perf/plugins/plugin_xen.so
>   LINK     /tmp/build/perf/plugins/plugin_scsi.so
>   LINK     /tmp/build/perf/plugins/plugin_cfg80211.so
>   LINK     /tmp/build/perf/plugins/plugin_tlb.so
>   INSTALL  trace_plugins
>   LD       /tmp/build/perf/pmu-events/pmu-events-in.o
> 
> Auto-detecting system features:
> ...                        libbfd: [ on  ]
> ...        disassembler-four-args: [ on  ]
> ...                          zlib: [ on  ]
> ...                        libcap: [ on  ]
> ...               clang-bpf-co-re: [ on  ]
> 
>   MKDIR    /tmp/build/perf/util/bpf_skel/.tmp//bootstrap/
>   MKDIR    /tmp/build/perf/util/bpf_skel/.tmp//bootstrap/libbpf/
>   CC       /tmp/build/perf/util/bpf_skel/.tmp//bootstrap/main.o
>   CC       /tmp/build/perf/util/bpf_skel/.tmp//bootstrap/common.o
>   CC       /tmp/build/perf/util/bpf_skel/.tmp//bootstrap/json_writer.o
>   CC       /tmp/build/perf/util/bpf_skel/.tmp//bootstrap/gen.o
>   CC       /tmp/build/perf/util/bpf_skel/.tmp//bootstrap/btf.o
>   GEN      /tmp/build/perf/util/bpf_skel/.tmp//bootstrap/libbpf/bpf_helper_defs.h
>   MKDIR    /tmp/build/perf/util/bpf_skel/.tmp//bootstrap/libbpf/staticobjs/
>   MKDIR    /tmp/build/perf/util/bpf_skel/.tmp//bootstrap/libbpf/staticobjs/
>   CC       /tmp/build/perf/util/bpf_skel/.tmp//bootstrap/libbpf/staticobjs/libbpf.o
>   CC       /tmp/build/perf/util/bpf_skel/.tmp//bootstrap/libbpf/staticobjs/bpf.o
>   MKDIR    /tmp/build/perf/util/bpf_skel/.tmp//bootstrap/libbpf/staticobjs/
>   CC       /tmp/build/perf/util/bpf_skel/.tmp//bootstrap/libbpf/staticobjs/nlattr.o
>   CC       /tmp/build/perf/util/bpf_skel/.tmp//bootstrap/libbpf/staticobjs/btf.o
>   CC       /tmp/build/perf/util/bpf_skel/.tmp//bootstrap/libbpf/staticobjs/libbpf_errno.o
>   CC       /tmp/build/perf/util/bpf_skel/.tmp//bootstrap/libbpf/staticobjs/str_error.o
>   CC       /tmp/build/perf/util/bpf_skel/.tmp//bootstrap/libbpf/staticobjs/netlink.o
>   CC       /tmp/build/perf/util/bpf_skel/.tmp//bootstrap/libbpf/staticobjs/bpf_prog_linfo.o
>   CC       /tmp/build/perf/util/bpf_skel/.tmp//bootstrap/libbpf/staticobjs/libbpf_probes.o
>   CC       /tmp/build/perf/util/bpf_skel/.tmp//bootstrap/libbpf/staticobjs/xsk.o
>   CC       /tmp/build/perf/util/bpf_skel/.tmp//bootstrap/libbpf/staticobjs/hashmap.o
>   CC       /tmp/build/perf/util/bpf_skel/.tmp//bootstrap/libbpf/staticobjs/btf_dump.o
>   CC       /tmp/build/perf/util/bpf_skel/.tmp//bootstrap/libbpf/staticobjs/ringbuf.o
>   LD       /tmp/build/perf/staticobjs/libbpf-in.o
>   LINK     /tmp/build/perf/libbpf.a
>   CLANG    /tmp/build/perf/util/bpf_skel/.tmp/bpf_prog_profiler.bpf.o
>   CLANG    /tmp/build/perf/util/bpf_skel/.tmp/bperf_leader.bpf.o
>   CLANG    /tmp/build/perf/util/bpf_skel/.tmp/bperf_follower.bpf.o
>   LD       /tmp/build/perf/util/bpf_skel/.tmp//bootstrap/libbpf/staticobjs/libbpf-in.o
>   LINK     /tmp/build/perf/util/bpf_skel/.tmp//bootstrap/libbpf/libbpf.a
>   LINK     /tmp/build/perf/util/bpf_skel/.tmp//bootstrap/bpftool
>   GEN-SKEL /tmp/build/perf/util/bpf_skel/bpf_prog_profiler.skel.h
>   GEN-SKEL /tmp/build/perf/util/bpf_skel/bperf_leader.skel.h
>   GEN-SKEL /tmp/build/perf/util/bpf_skel/bperf_follower.skel.h
> libbpf: map 'prev_readings': unexpected def kind var.
> Error: failed to open BPF object file: Invalid argument
> libbpf: map 'diff_readings': unexpected def kind var.
> Error: failed to open BPF object file: Invalid argument
> make[2]: *** [Makefile.perf:1029: /tmp/build/perf/util/bpf_skel/bperf_leader.skel.h] Error 255
> make[2]: *** Waiting for unfinished jobs....
> make[2]: *** [Makefile.perf:1029: /tmp/build/perf/util/bpf_skel/bperf_follower.skel.h] Error 255
> make[1]: *** [Makefile.perf:236: sub-make] Error 2
> make: *** [Makefile:110: install-bin] Error 2
> make: Leaving directory '/home/acme/git/perf/tools/perf'
> [acme@quaco perf]$ clang -v
> clang version 11.0.0 (https://github.com/llvm/llvm-project 67420f1b0e9c673ee638f2680fa83f468019004f)
> Target: x86_64-unknown-linux-gnu
> Thread model: posix
> InstalledDir: /usr/local/bin
> Found candidate GCC installation: /usr/lib/gcc/x86_64-redhat-linux/10
> Selected GCC installation: /usr/lib/gcc/x86_64-redhat-linux/10
> Candidate multilib: .;@m64
> Candidate multilib: 32;@m32
> Selected multilib: .;@m64
> [acme@quaco perf]$

-- 

- Arnaldo

Re: [PATCH v2 1/3] perf-stat: introduce bperf, share hardware PMCs with BPF

[Arnaldo Carvalho de Melo]

Em Tue, Mar 23, 2021 at 09:25:52AM -0300, Arnaldo Carvalho de Melo escreveu:
> Em Fri, Mar 19, 2021 at 03:41:57PM -0300, Arnaldo Carvalho de Melo escreveu:
> > Em Thu, Mar 18, 2021 at 10:15:13PM +0100, Jiri Olsa escreveu:
> > > On Tue, Mar 16, 2021 at 02:18:35PM -0700, Song Liu wrote:
> > > > bperf is off by default. To enable it, pass --bpf-counters option to
> > > > perf-stat. bperf uses a BPF hashmap to share information about BPF
> > > > programs and maps used by bperf. This map is pinned to bpffs. The default
> > > > path is /sys/fs/bpf/perf_attr_map. The user could change the path with
> > > > option --bpf-attr-map.
> > > > 
> > > > Signed-off-by: Song Liu <songliubraving@fb.com>
> > > 
> > > Reviewed-by: Jiri Olsa <jolsa@redhat.com>
> > 
> > After applying just this first patch in the series I'm getting this
> > after a 'make -C tools/ clean', now I'm checking if I need some new
> > clang, ideas?
> 
> Works now with clang from fedora 33, I was using a locally built, older,
> now I get this when trying as non-root, expected, but we need to improve
> the wording.

Fails as root as well, investigating:

[root@five ~]# ls -lad /sys/fs/bpf/
drwx-----T. 2 root root 0 Mar 23 06:03 /sys/fs/bpf/
[root@five ~]# strace -e bpf perf stat --bpf-counters sleep 1
bpf(BPF_MAP_CREATE, {map_type=BPF_MAP_TYPE_HASH, key_size=120, value_size=8, max_entries=16, map_flags=0, inner_map_fd=0, map_name="", map_ifindex=0, btf_fd=0, btf_key_type_id=0, btf_value_type_id=0, btf_vmlinux_value_type_id=0}, 120) = -1 EPERM (Operation not permitted)
Failed to lock perf_event_attr map
--- SIGCHLD {si_signo=SIGCHLD, si_code=CLD_KILLED, si_pid=13916, si_uid=0, si_status=SIGTERM, si_utime=0, si_stime=0} ---
+++ exited with 255 +++
[root@five ~]#
 
> [acme@five perf]$ perf stat --bpf-counters sleep 1
> Failed to lock perf_event_attr map
> [acme@five perf]$
>  
> > - Arnaldo
> > 
> > [acme@quaco perf]$ make O=/tmp/build/perf -C tools/perf BUILD_BPF_SKEL=1 PYTHON=python3 install-bin
> > make: Entering directory '/home/acme/git/perf/tools/perf'
> >   BUILD:   Doing 'make -j8' parallel build
> > Warning: Kernel ABI header at 'tools/include/uapi/linux/kvm.h' differs from latest version at 'include/uapi/linux/kvm.h'
> > diff -u tools/include/uapi/linux/kvm.h include/uapi/linux/kvm.h
> > Warning: Kernel ABI header at 'tools/perf/arch/mips/entry/syscalls/syscall_n64.tbl' differs from latest version at 'arch/mips/kernel/syscalls/syscall_n64.tbl'
> > diff -u tools/perf/arch/mips/entry/syscalls/syscall_n64.tbl arch/mips/kernel/syscalls/syscall_n64.tbl
> > 
> > Auto-detecting system features:
> > ...                         dwarf: [ on  ]
> > ...            dwarf_getlocations: [ on  ]
> > ...                         glibc: [ on  ]
> > ...                        libbfd: [ on  ]
> > ...                libbfd-buildid: [ on  ]
> > ...                        libcap: [ on  ]
> > ...                        libelf: [ on  ]
> > ...                       libnuma: [ on  ]
> > ...        numa_num_possible_cpus: [ on  ]
> > ...                       libperl: [ on  ]
> > ...                     libpython: [ on  ]
> > ...                     libcrypto: [ on  ]
> > ...                     libunwind: [ on  ]
> > ...            libdw-dwarf-unwind: [ on  ]
> > ...                          zlib: [ on  ]
> > ...                          lzma: [ on  ]
> > ...                     get_cpuid: [ on  ]
> > ...                           bpf: [ on  ]
> > ...                        libaio: [ on  ]
> > ...                       libzstd: [ on  ]
> > ...        disassembler-four-args: [ on  ]
> > 
> >   GEN      /tmp/build/perf/common-cmds.h
> >   CC       /tmp/build/perf/exec-cmd.o
> >   MKDIR    /tmp/build/perf/fd/
> >   MKDIR    /tmp/build/perf/fs/
> >   CC       /tmp/build/perf/fs/fs.o
> >   CC       /tmp/build/perf/event-parse.o
> >   CC       /tmp/build/perf/fd/array.o
> >   CC       /tmp/build/perf/core.o
> >   GEN      /tmp/build/perf/bpf_helper_defs.h
> >   CC       /tmp/build/perf/event-plugin.o
> >   MKDIR    /tmp/build/perf/staticobjs/
> >   PERF_VERSION = 5.12.rc2.g3df07f57f205
> >   CC       /tmp/build/perf/staticobjs/libbpf.o
> >   CC       /tmp/build/perf/cpu.o
> >   LD       /tmp/build/perf/fd/libapi-in.o
> >   CC       /tmp/build/perf/cpumap.o
> >   CC       /tmp/build/perf/help.o
> >   MKDIR    /tmp/build/perf/fs/
> >   CC       /tmp/build/perf/fs/tracing_path.o
> >   CC       /tmp/build/perf/fs/cgroup.o
> >   CC       /tmp/build/perf/trace-seq.o
> >   CC       /tmp/build/perf/pager.o
> >   CC       /tmp/build/perf/parse-options.o
> >   LD       /tmp/build/perf/fs/libapi-in.o
> >   CC       /tmp/build/perf/debug.o
> >   CC       /tmp/build/perf/str_error_r.o
> >   CC       /tmp/build/perf/run-command.o
> >   CC       /tmp/build/perf/sigchain.o
> >   LD       /tmp/build/perf/libapi-in.o
> >   AR       /tmp/build/perf/libapi.a
> >   CC       /tmp/build/perf/subcmd-config.o
> >   CC       /tmp/build/perf/threadmap.o
> >   CC       /tmp/build/perf/evsel.o
> >   CC       /tmp/build/perf/parse-filter.o
> >   MKDIR    /tmp/build/perf/staticobjs/
> >   CC       /tmp/build/perf/staticobjs/bpf.o
> >   CC       /tmp/build/perf/evlist.o
> >   CC       /tmp/build/perf/parse-utils.o
> >   CC       /tmp/build/perf/kbuffer-parse.o
> >   CC       /tmp/build/perf/tep_strerror.o
> >   CC       /tmp/build/perf/mmap.o
> >   CC       /tmp/build/perf/zalloc.o
> >   CC       /tmp/build/perf/event-parse-api.o
> >   LD       /tmp/build/perf/libsubcmd-in.o
> >   AR       /tmp/build/perf/libsubcmd.a
> >   CC       /tmp/build/perf/xyarray.o
> >   LD       /tmp/build/perf/libtraceevent-in.o
> >   LINK     /tmp/build/perf/libtraceevent.a
> >   CC       /tmp/build/perf/staticobjs/nlattr.o
> >   CC       /tmp/build/perf/staticobjs/btf.o
> >   CC       /tmp/build/perf/lib.o
> >   CC       /tmp/build/perf/staticobjs/libbpf_errno.o
> >   CC       /tmp/build/perf/staticobjs/str_error.o
> >   CC       /tmp/build/perf/staticobjs/netlink.o
> >   CC       /tmp/build/perf/staticobjs/bpf_prog_linfo.o
> >   CC       /tmp/build/perf/staticobjs/libbpf_probes.o
> >   LD       /tmp/build/perf/libperf-in.o
> >   AR       /tmp/build/perf/libperf.a
> >   MKDIR    /tmp/build/perf/pmu-events/
> >   HOSTCC   /tmp/build/perf/pmu-events/json.o
> >   CC       /tmp/build/perf/plugin_jbd2.o
> >   CC       /tmp/build/perf/staticobjs/xsk.o
> >   MKDIR    /tmp/build/perf/pmu-events/
> >   HOSTCC   /tmp/build/perf/pmu-events/jsmn.o
> >   CC       /tmp/build/perf/staticobjs/hashmap.o
> >   LD       /tmp/build/perf/plugin_jbd2-in.o
> >   CC       /tmp/build/perf/staticobjs/btf_dump.o
> >   CC       /tmp/build/perf/plugin_hrtimer.o
> >   HOSTCC   /tmp/build/perf/pmu-events/jevents.o
> >   LD       /tmp/build/perf/plugin_hrtimer-in.o
> >   CC       /tmp/build/perf/plugin_kmem.o
> >   CC       /tmp/build/perf/staticobjs/ringbuf.o
> >   LD       /tmp/build/perf/plugin_kmem-in.o
> >   CC       /tmp/build/perf/plugin_kvm.o
> >   HOSTLD   /tmp/build/perf/pmu-events/jevents-in.o
> >   CC       /tmp/build/perf/perf-read-vdso32
> >   CC       /tmp/build/perf/plugin_mac80211.o
> >   LD       /tmp/build/perf/plugin_kvm-in.o
> >   CC       /tmp/build/perf/plugin_sched_switch.o
> >   CC       /tmp/build/perf/plugin_function.o
> >   MKDIR    /tmp/build/perf/jvmti/
> >   CC       /tmp/build/perf/jvmti/libjvmti.o
> >   LD       /tmp/build/perf/plugin_mac80211-in.o
> >   GEN      perf-archive
> >   LD       /tmp/build/perf/plugin_sched_switch-in.o
> >   CC       /tmp/build/perf/plugin_futex.o
> >   CC       /tmp/build/perf/plugin_xen.o
> >   CC       /tmp/build/perf/plugin_scsi.o
> >   LD       /tmp/build/perf/plugin_function-in.o
> >   CC       /tmp/build/perf/plugin_cfg80211.o
> >   LD       /tmp/build/perf/plugin_futex-in.o
> >   LD       /tmp/build/perf/plugin_xen-in.o
> >   LINK     /tmp/build/perf/plugin_jbd2.so
> >   LINK     /tmp/build/perf/plugin_hrtimer.so
> >   CC       /tmp/build/perf/plugin_tlb.o
> >   LINK     /tmp/build/perf/plugin_kmem.so
> >   LINK     /tmp/build/perf/plugin_kvm.so
> >   LD       /tmp/build/perf/plugin_scsi-in.o
> >   LINK     /tmp/build/perf/plugin_mac80211.so
> >   LINK     /tmp/build/perf/plugin_sched_switch.so
> >   LINK     /tmp/build/perf/plugin_function.so
> >   LINK     /tmp/build/perf/plugin_futex.so
> >   LD       /tmp/build/perf/plugin_cfg80211-in.o
> >   LINK     /tmp/build/perf/plugin_xen.so
> >   LINK     /tmp/build/perf/plugin_scsi.so
> >   LD       /tmp/build/perf/plugin_tlb-in.o
> >   MKDIR    /tmp/build/perf/jvmti/
> >   LINK     /tmp/build/perf/plugin_cfg80211.so
> >   CC       /tmp/build/perf/jvmti/jvmti_agent.o
> >   LINK     /tmp/build/perf/plugin_tlb.so
> >   GEN      perf-with-kcore
> >   CC       /tmp/build/perf/jvmti/libstring.o
> > CFLAGS= make -C ../bpf/bpftool \
> > 	OUTPUT=/tmp/build/perf/util/bpf_skel/.tmp/ bootstrap
> >   CC       /tmp/build/perf/jvmti/libctype.o
> >   GEN      /tmp/build/perf/libtraceevent-dynamic-list
> >   LINK     /tmp/build/perf/pmu-events/jevents
> >   DESCEND  plugins
> >   GEN      /tmp/build/perf/python/perf.so
> >   GEN      /tmp/build/perf/pmu-events/pmu-events.c
> >   CC       /tmp/build/perf/plugins/plugin_jbd2.o
> >   CC       /tmp/build/perf/plugins/plugin_hrtimer.o
> >   LD       /tmp/build/perf/plugins/plugin_jbd2-in.o
> >   LD       /tmp/build/perf/plugins/plugin_hrtimer-in.o
> >   CC       /tmp/build/perf/plugins/plugin_kmem.o
> >   CC       /tmp/build/perf/plugins/plugin_kvm.o
> >   LD       /tmp/build/perf/jvmti/jvmti-in.o
> >   LD       /tmp/build/perf/plugins/plugin_kmem-in.o
> >   LINK     /tmp/build/perf/libperf-jvmti.so
> >   CC       /tmp/build/perf/plugins/plugin_mac80211.o
> >   CC       /tmp/build/perf/plugins/plugin_sched_switch.o
> >   CC       /tmp/build/perf/pmu-events/pmu-events.o
> >   LD       /tmp/build/perf/plugins/plugin_kvm-in.o
> >   CC       /tmp/build/perf/plugins/plugin_function.o
> >   LD       /tmp/build/perf/plugins/plugin_mac80211-in.o
> >   CC       /tmp/build/perf/plugins/plugin_futex.o
> >   LD       /tmp/build/perf/plugins/plugin_sched_switch-in.o
> >   CC       /tmp/build/perf/plugins/plugin_xen.o
> >   LD       /tmp/build/perf/plugins/plugin_function-in.o
> >   CC       /tmp/build/perf/plugins/plugin_scsi.o
> >   LD       /tmp/build/perf/plugins/plugin_futex-in.o
> >   CC       /tmp/build/perf/plugins/plugin_cfg80211.o
> >   LD       /tmp/build/perf/plugins/plugin_xen-in.o
> >   CC       /tmp/build/perf/plugins/plugin_tlb.o
> >   LD       /tmp/build/perf/plugins/plugin_scsi-in.o
> >   LD       /tmp/build/perf/plugins/plugin_cfg80211-in.o
> >   LINK     /tmp/build/perf/plugins/plugin_jbd2.so
> >   LINK     /tmp/build/perf/plugins/plugin_hrtimer.so
> >   LINK     /tmp/build/perf/plugins/plugin_kmem.so
> >   LINK     /tmp/build/perf/plugins/plugin_kvm.so
> >   LINK     /tmp/build/perf/plugins/plugin_mac80211.so
> >   LD       /tmp/build/perf/plugins/plugin_tlb-in.o
> >   LINK     /tmp/build/perf/plugins/plugin_function.so
> >   LINK     /tmp/build/perf/plugins/plugin_sched_switch.so
> >   LINK     /tmp/build/perf/plugins/plugin_futex.so
> >   LINK     /tmp/build/perf/plugins/plugin_xen.so
> >   LINK     /tmp/build/perf/plugins/plugin_scsi.so
> >   LINK     /tmp/build/perf/plugins/plugin_cfg80211.so
> >   LINK     /tmp/build/perf/plugins/plugin_tlb.so
> >   INSTALL  trace_plugins
> >   LD       /tmp/build/perf/pmu-events/pmu-events-in.o
> > 
> > Auto-detecting system features:
> > ...                        libbfd: [ on  ]
> > ...        disassembler-four-args: [ on  ]
> > ...                          zlib: [ on  ]
> > ...                        libcap: [ on  ]
> > ...               clang-bpf-co-re: [ on  ]
> > 
> >   MKDIR    /tmp/build/perf/util/bpf_skel/.tmp//bootstrap/
> >   MKDIR    /tmp/build/perf/util/bpf_skel/.tmp//bootstrap/libbpf/
> >   CC       /tmp/build/perf/util/bpf_skel/.tmp//bootstrap/main.o
> >   CC       /tmp/build/perf/util/bpf_skel/.tmp//bootstrap/common.o
> >   CC       /tmp/build/perf/util/bpf_skel/.tmp//bootstrap/json_writer.o
> >   CC       /tmp/build/perf/util/bpf_skel/.tmp//bootstrap/gen.o
> >   CC       /tmp/build/perf/util/bpf_skel/.tmp//bootstrap/btf.o
> >   GEN      /tmp/build/perf/util/bpf_skel/.tmp//bootstrap/libbpf/bpf_helper_defs.h
> >   MKDIR    /tmp/build/perf/util/bpf_skel/.tmp//bootstrap/libbpf/staticobjs/
> >   MKDIR    /tmp/build/perf/util/bpf_skel/.tmp//bootstrap/libbpf/staticobjs/
> >   CC       /tmp/build/perf/util/bpf_skel/.tmp//bootstrap/libbpf/staticobjs/libbpf.o
> >   CC       /tmp/build/perf/util/bpf_skel/.tmp//bootstrap/libbpf/staticobjs/bpf.o
> >   MKDIR    /tmp/build/perf/util/bpf_skel/.tmp//bootstrap/libbpf/staticobjs/
> >   CC       /tmp/build/perf/util/bpf_skel/.tmp//bootstrap/libbpf/staticobjs/nlattr.o
> >   CC       /tmp/build/perf/util/bpf_skel/.tmp//bootstrap/libbpf/staticobjs/btf.o
> >   CC       /tmp/build/perf/util/bpf_skel/.tmp//bootstrap/libbpf/staticobjs/libbpf_errno.o
> >   CC       /tmp/build/perf/util/bpf_skel/.tmp//bootstrap/libbpf/staticobjs/str_error.o
> >   CC       /tmp/build/perf/util/bpf_skel/.tmp//bootstrap/libbpf/staticobjs/netlink.o
> >   CC       /tmp/build/perf/util/bpf_skel/.tmp//bootstrap/libbpf/staticobjs/bpf_prog_linfo.o
> >   CC       /tmp/build/perf/util/bpf_skel/.tmp//bootstrap/libbpf/staticobjs/libbpf_probes.o
> >   CC       /tmp/build/perf/util/bpf_skel/.tmp//bootstrap/libbpf/staticobjs/xsk.o
> >   CC       /tmp/build/perf/util/bpf_skel/.tmp//bootstrap/libbpf/staticobjs/hashmap.o
> >   CC       /tmp/build/perf/util/bpf_skel/.tmp//bootstrap/libbpf/staticobjs/btf_dump.o
> >   CC       /tmp/build/perf/util/bpf_skel/.tmp//bootstrap/libbpf/staticobjs/ringbuf.o
> >   LD       /tmp/build/perf/staticobjs/libbpf-in.o
> >   LINK     /tmp/build/perf/libbpf.a
> >   CLANG    /tmp/build/perf/util/bpf_skel/.tmp/bpf_prog_profiler.bpf.o
> >   CLANG    /tmp/build/perf/util/bpf_skel/.tmp/bperf_leader.bpf.o
> >   CLANG    /tmp/build/perf/util/bpf_skel/.tmp/bperf_follower.bpf.o
> >   LD       /tmp/build/perf/util/bpf_skel/.tmp//bootstrap/libbpf/staticobjs/libbpf-in.o
> >   LINK     /tmp/build/perf/util/bpf_skel/.tmp//bootstrap/libbpf/libbpf.a
> >   LINK     /tmp/build/perf/util/bpf_skel/.tmp//bootstrap/bpftool
> >   GEN-SKEL /tmp/build/perf/util/bpf_skel/bpf_prog_profiler.skel.h
> >   GEN-SKEL /tmp/build/perf/util/bpf_skel/bperf_leader.skel.h
> >   GEN-SKEL /tmp/build/perf/util/bpf_skel/bperf_follower.skel.h
> > libbpf: map 'prev_readings': unexpected def kind var.
> > Error: failed to open BPF object file: Invalid argument
> > libbpf: map 'diff_readings': unexpected def kind var.
> > Error: failed to open BPF object file: Invalid argument
> > make[2]: *** [Makefile.perf:1029: /tmp/build/perf/util/bpf_skel/bperf_leader.skel.h] Error 255
> > make[2]: *** Waiting for unfinished jobs....
> > make[2]: *** [Makefile.perf:1029: /tmp/build/perf/util/bpf_skel/bperf_follower.skel.h] Error 255
> > make[1]: *** [Makefile.perf:236: sub-make] Error 2
> > make: *** [Makefile:110: install-bin] Error 2
> > make: Leaving directory '/home/acme/git/perf/tools/perf'
> > [acme@quaco perf]$ clang -v
> > clang version 11.0.0 (https://github.com/llvm/llvm-project 67420f1b0e9c673ee638f2680fa83f468019004f)
> > Target: x86_64-unknown-linux-gnu
> > Thread model: posix
> > InstalledDir: /usr/local/bin
> > Found candidate GCC installation: /usr/lib/gcc/x86_64-redhat-linux/10
> > Selected GCC installation: /usr/lib/gcc/x86_64-redhat-linux/10
> > Candidate multilib: .;@m64
> > Candidate multilib: 32;@m32
> > Selected multilib: .;@m64
> > [acme@quaco perf]$
> 
> -- 
> 
> - Arnaldo

-- 

- Arnaldo

Re: [PATCH v2 1/3] perf-stat: introduce bperf, share hardware PMCs with BPF

[Arnaldo Carvalho de Melo]

Em Tue, Mar 23, 2021 at 09:37:42AM -0300, Arnaldo Carvalho de Melo escreveu:
> Em Tue, Mar 23, 2021 at 09:25:52AM -0300, Arnaldo Carvalho de Melo escreveu:
> > Em Fri, Mar 19, 2021 at 03:41:57PM -0300, Arnaldo Carvalho de Melo escreveu:
> > > Em Thu, Mar 18, 2021 at 10:15:13PM +0100, Jiri Olsa escreveu:
> > > > On Tue, Mar 16, 2021 at 02:18:35PM -0700, Song Liu wrote:
> > > > > bperf is off by default. To enable it, pass --bpf-counters option to
> > > > > perf-stat. bperf uses a BPF hashmap to share information about BPF
> > > > > programs and maps used by bperf. This map is pinned to bpffs. The default
> > > > > path is /sys/fs/bpf/perf_attr_map. The user could change the path with
> > > > > option --bpf-attr-map.
> > > > > 
> > > > > Signed-off-by: Song Liu <songliubraving@fb.com>
> > > > 
> > > > Reviewed-by: Jiri Olsa <jolsa@redhat.com>
> > > 
> > > After applying just this first patch in the series I'm getting this
> > > after a 'make -C tools/ clean', now I'm checking if I need some new
> > > clang, ideas?
> > 
> > Works now with clang from fedora 33, I was using a locally built, older,
> > now I get this when trying as non-root, expected, but we need to improve
> > the wording.
> 
> Fails as root as well, investigating:
> 
> [root@five ~]# ls -lad /sys/fs/bpf/
> drwx-----T. 2 root root 0 Mar 23 06:03 /sys/fs/bpf/
> [root@five ~]# strace -e bpf perf stat --bpf-counters sleep 1
> bpf(BPF_MAP_CREATE, {map_type=BPF_MAP_TYPE_HASH, key_size=120, value_size=8, max_entries=16, map_flags=0, inner_map_fd=0, map_name="", map_ifindex=0, btf_fd=0, btf_key_type_id=0, btf_value_type_id=0, btf_vmlinux_value_type_id=0}, 120) = -1 EPERM (Operation not permitted)
> Failed to lock perf_event_attr map
> --- SIGCHLD {si_signo=SIGCHLD, si_code=CLD_KILLED, si_pid=13916, si_uid=0, si_status=SIGTERM, si_utime=0, si_stime=0} ---
> +++ exited with 255 +++
> [root@five ~]#
>  
> > [acme@five perf]$ perf stat --bpf-counters sleep 1
> > Failed to lock perf_event_attr map
> > [acme@five perf]$

Now it works, on 5.12-rc2+

[root@five pahole]# perf stat --bpf-counters sleep 1
libbpf: elf: skipping unrecognized data section(7) .eh_frame
libbpf: elf: skipping relo section(12) .rel.eh_frame for section(7) .eh_frame
libbpf: elf: skipping unrecognized data section(8) .eh_frame
libbpf: elf: skipping relo section(13) .rel.eh_frame for section(8) .eh_frame
libbpf: elf: skipping unrecognized data section(7) .eh_frame
libbpf: elf: skipping relo section(12) .rel.eh_frame for section(7) .eh_frame
libbpf: elf: skipping unrecognized data section(8) .eh_frame
libbpf: elf: skipping relo section(13) .rel.eh_frame for section(8) .eh_frame
libbpf: elf: skipping unrecognized data section(7) .eh_frame
libbpf: elf: skipping relo section(12) .rel.eh_frame for section(7) .eh_frame
libbpf: elf: skipping unrecognized data section(8) .eh_frame
libbpf: elf: skipping relo section(13) .rel.eh_frame for section(8) .eh_frame
libbpf: elf: skipping unrecognized data section(7) .eh_frame
libbpf: elf: skipping relo section(12) .rel.eh_frame for section(7) .eh_frame
libbpf: elf: skipping unrecognized data section(8) .eh_frame
libbpf: elf: skipping relo section(13) .rel.eh_frame for section(8) .eh_frame
libbpf: elf: skipping unrecognized data section(7) .eh_frame
libbpf: elf: skipping relo section(12) .rel.eh_frame for section(7) .eh_frame
libbpf: elf: skipping unrecognized data section(8) .eh_frame
libbpf: elf: skipping relo section(13) .rel.eh_frame for section(8) .eh_frame
libbpf: elf: skipping unrecognized data section(7) .eh_frame
libbpf: elf: skipping relo section(12) .rel.eh_frame for section(7) .eh_frame
libbpf: elf: skipping unrecognized data section(8) .eh_frame
libbpf: elf: skipping relo section(13) .rel.eh_frame for section(8) .eh_frame
libbpf: elf: skipping unrecognized data section(7) .eh_frame
libbpf: elf: skipping relo section(12) .rel.eh_frame for section(7) .eh_frame
libbpf: elf: skipping unrecognized data section(8) .eh_frame
libbpf: elf: skipping relo section(13) .rel.eh_frame for section(8) .eh_frame
libbpf: elf: skipping unrecognized data section(7) .eh_frame
libbpf: elf: skipping relo section(12) .rel.eh_frame for section(7) .eh_frame
libbpf: elf: skipping unrecognized data section(8) .eh_frame
libbpf: elf: skipping relo section(13) .rel.eh_frame for section(8) .eh_frame
libbpf: elf: skipping unrecognized data section(7) .eh_frame
libbpf: elf: skipping relo section(12) .rel.eh_frame for section(7) .eh_frame
libbpf: elf: skipping unrecognized data section(8) .eh_frame
libbpf: elf: skipping relo section(13) .rel.eh_frame for section(8) .eh_frame
libbpf: elf: skipping unrecognized data section(7) .eh_frame
libbpf: elf: skipping relo section(12) .rel.eh_frame for section(7) .eh_frame
libbpf: elf: skipping unrecognized data section(8) .eh_frame
libbpf: elf: skipping relo section(13) .rel.eh_frame for section(8) .eh_frame

 Performance counter stats for 'sleep 1':

              0.84 msec task-clock                #    0.001 CPUs utilized
                 3      context-switches          #    3.589 K/sec
                 2      cpu-migrations            #    2.393 K/sec
                67      page-faults               #   80.164 K/sec
         2,760,235      cycles                    #    3.303 GHz                      (88.36%)
           241,055      stalled-cycles-frontend   #    8.73% frontend cycles idle     (45.93%)
         1,251,751      stalled-cycles-backend    #   45.35% backend cycles idle      (65.67%)
         2,340,813      instructions              #    0.85  insn per cycle
                                                  #    0.53  stalled cycles per insn
           459,088      branches                  #  549.289 M/sec
            12,243      branch-misses             #    2.67% of all branches

       1.000928326 seconds time elapsed

       0.000881000 seconds user
       0.000000000 seconds sys


[root@five pahole]# clang --version
clang version 11.0.0 (Fedora 11.0.0-2.fc33)
Target: x86_64-unknown-linux-gnu
Thread model: posix
InstalledDir: /usr/bin
[root@five pahole]# uname -a
Linux five 5.12.0-rc2+ #2 SMP Tue Mar 23 12:51:43 -03 2021 x86_64 x86_64 x86_64 GNU/Linux
[root@five pahole]#


Full BPF setup, lotsa syscalls:

[root@five pahole]# strace -e bpf perf stat --bpf-counters sleep 1
bpf(BPF_OBJ_GET, {pathname="/sys/fs/bpf/perf_attr_map", bpf_fd=0, file_flags=0}, 120) = 3
bpf(BPF_MAP_LOOKUP_ELEM, {map_fd=3, key=0x1b39760, value=0x7ffda32551a8, flags=BPF_ANY}, 120) = 0
bpf(BPF_LINK_GET_FD_BY_ID, 0x7ffda32550e0, 120) = -1 ENOENT (No such file or directory)
libbpf: elf: skipping unrecognized data section(7) .eh_frame
libbpf: elf: skipping relo section(12) .rel.eh_frame for section(7) .eh_frame
bpf(BPF_PROG_LOAD, {prog_type=BPF_PROG_TYPE_SOCKET_FILTER, insn_cnt=2, insns=0x7ffda3254ff0, license="GPL", log_level=0, log_size=0, log_buf=NULL, kern_version=KERNEL_VERSION(0, 0, 0), prog_flags=0, prog_name="", prog_ifindex=0, expected_attach_type=BPF_CGROUP_INET_INGRESS, prog_btf_fd=0, func_info_rec_size=0, func_info=NULL, func_info_cnt=0, line_info_rec_size=0, line_info=NULL, line_info_cnt=0, attach_btf_id=0, attach_prog_fd=0}, 120) = 4
bpf(BPF_BTF_LOAD, {btf="\237\353\1\0\30\0\0\0\0\0\0\0\20\0\0\0\20\0\0\0\5\0\0\0\1\0\0\0\0\0\0\1"..., btf_log_buf=NULL, btf_size=45, btf_log_size=0, btf_log_level=0}, 120) = 4
bpf(BPF_BTF_LOAD, {btf="\237\353\1\0\30\0\0\0\0\0\0\0000\0\0\0000\0\0\0\t\0\0\0\1\0\0\0\0\0\0\1"..., btf_log_buf=NULL, btf_size=81, btf_log_size=0, btf_log_level=0}, 120) = 4
bpf(BPF_BTF_LOAD, {btf="\237\353\1\0\30\0\0\0\0\0\0\08\0\0\08\0\0\0\t\0\0\0\0\0\0\0\0\0\0\1"..., btf_log_buf=NULL, btf_size=89, btf_log_size=0, btf_log_level=0}, 120) = 4
bpf(BPF_BTF_LOAD, {btf="\237\353\1\0\30\0\0\0\0\0\0\0000\0\0\0000\0\0\0\t\0\0\0\1\0\0\0\0\0\0\1"..., btf_log_buf=NULL, btf_size=81, btf_log_size=0, btf_log_level=0}, 120) = 4
bpf(BPF_BTF_LOAD, {btf="\237\353\1\0\30\0\0\0\0\0\0\0D\2\0\0D\2\0\0\356\2\0\0\0\0\0\0\0\0\0\2"..., btf_log_buf=NULL, btf_size=1354, btf_log_size=0, btf_log_level=0}, 120) = 4
bpf(BPF_PROG_LOAD, {prog_type=BPF_PROG_TYPE_SOCKET_FILTER, insn_cnt=2, insns=0x7ffda3254c90, license="GPL", log_level=0, log_size=0, log_buf=NULL, kern_version=KERNEL_VERSION(0, 0, 0), prog_flags=0, prog_name="test", prog_ifindex=0, expected_attach_type=BPF_CGROUP_INET_INGRESS, prog_btf_fd=0, func_info_rec_size=0, func_info=NULL, func_info_cnt=0, line_info_rec_size=0, line_info=NULL, line_info_cnt=0, attach_btf_id=0, attach_prog_fd=0}, 120) = 5
bpf(BPF_MAP_CREATE, {map_type=BPF_MAP_TYPE_PERF_EVENT_ARRAY, key_size=4, value_size=4, max_entries=24, map_flags=BPF_F_PRESERVE_ELEMS, inner_map_fd=0, map_name="events", map_ifindex=0, btf_fd=4, btf_key_type_id=0, btf_value_type_id=0, btf_vmlinux_value_type_id=0}, 120) = 5
bpf(BPF_MAP_CREATE, {map_type=BPF_MAP_TYPE_PERCPU_ARRAY, key_size=4, value_size=24, max_entries=1, map_flags=0, inner_map_fd=0, map_name="prev_readings", map_ifindex=0, btf_fd=4, btf_key_type_id=0, btf_value_type_id=0, btf_vmlinux_value_type_id=0}, 120) = 7
bpf(BPF_MAP_CREATE, {map_type=BPF_MAP_TYPE_PERCPU_ARRAY, key_size=4, value_size=24, max_entries=1, map_flags=0, inner_map_fd=0, map_name="diff_readings", map_ifindex=0, btf_fd=4, btf_key_type_id=0, btf_value_type_id=0, btf_vmlinux_value_type_id=0}, 120) = 8
bpf(BPF_PROG_LOAD, {prog_type=BPF_PROG_TYPE_CGROUP_SOCK, insn_cnt=2, insns=0x7ffda3254c90, license="GPL", log_level=0, log_size=0, log_buf=NULL, kern_version=KERNEL_VERSION(0, 0, 0), prog_flags=0, prog_name="", prog_ifindex=0, expected_attach_type=BPF_CGROUP_INET_SOCK_CREATE, prog_btf_fd=0, func_info_rec_size=0, func_info=NULL, func_info_cnt=0, line_info_rec_size=0, line_info=NULL, line_info_cnt=0, attach_btf_id=0, attach_prog_fd=0}, 120) = 9
bpf(BPF_PROG_LOAD, {prog_type=BPF_PROG_TYPE_RAW_TRACEPOINT, insn_cnt=48, insns=0x1b7fd00, license="Dual BSD/GPL", log_level=0, log_size=0, log_buf=NULL, kern_version=KERNEL_VERSION(5, 12, 0), prog_flags=0, prog_name="on_switch", prog_ifindex=0, expected_attach_type=BPF_CGROUP_INET_INGRESS, prog_btf_fd=4, func_info_rec_size=8, func_info=0x1b7e6c0, func_info_cnt=1, line_info_rec_size=16, line_info=0x1b761b0, line_info_cnt=25, attach_btf_id=0, attach_prog_fd=0}, 120) = 9
bpf(BPF_RAW_TRACEPOINT_OPEN, {raw_tracepoint={name="sched_switch", prog_fd=9}}, 120) = 10
bpf(BPF_OBJ_GET_INFO_BY_FD, {info={bpf_fd=10, info_len=32, info=0x7ffda32551b0}}, 120) = 0
bpf(BPF_OBJ_GET_INFO_BY_FD, {info={bpf_fd=8, info_len=80, info=0x7ffda3255200}}, 120) = 0
bpf(BPF_MAP_UPDATE_ELEM, {map_fd=3, key=0x1b39760, value=0x7ffda32551a8, flags=BPF_ANY}, 120) = 0
bpf(BPF_LINK_GET_FD_BY_ID, 0x7ffda32550e0, 120) = 11
bpf(BPF_MAP_UPDATE_ELEM, {map_fd=5, key=0x7ffda325505c, value=0x7ffda3255058, flags=BPF_ANY}, 120) = 0
bpf(BPF_MAP_UPDATE_ELEM, {map_fd=5, key=0x7ffda325505c, value=0x7ffda3255058, flags=BPF_ANY}, 120) = 0
bpf(BPF_MAP_UPDATE_ELEM, {map_fd=5, key=0x7ffda325505c, value=0x7ffda3255058, flags=BPF_ANY}, 120) = 0
bpf(BPF_MAP_UPDATE_ELEM, {map_fd=5, key=0x7ffda325505c, value=0x7ffda3255058, flags=BPF_ANY}, 120) = 0
bpf(BPF_MAP_UPDATE_ELEM, {map_fd=5, key=0x7ffda325505c, value=0x7ffda3255058, flags=BPF_ANY}, 120) = 0
bpf(BPF_MAP_UPDATE_ELEM, {map_fd=5, key=0x7ffda325505c, value=0x7ffda3255058, flags=BPF_ANY}, 120) = 0
bpf(BPF_MAP_UPDATE_ELEM, {map_fd=5, key=0x7ffda325505c, value=0x7ffda3255058, flags=BPF_ANY}, 120) = 0
bpf(BPF_MAP_UPDATE_ELEM, {map_fd=5, key=0x7ffda325505c, value=0x7ffda3255058, flags=BPF_ANY}, 120) = 0
bpf(BPF_MAP_UPDATE_ELEM, {map_fd=5, key=0x7ffda325505c, value=0x7ffda3255058, flags=BPF_ANY}, 120) = 0
bpf(BPF_MAP_UPDATE_ELEM, {map_fd=5, key=0x7ffda325505c, value=0x7ffda3255058, flags=BPF_ANY}, 120) = 0
bpf(BPF_MAP_UPDATE_ELEM, {map_fd=5, key=0x7ffda325505c, value=0x7ffda3255058, flags=BPF_ANY}, 120) = 0
bpf(BPF_MAP_UPDATE_ELEM, {map_fd=5, key=0x7ffda325505c, value=0x7ffda3255058, flags=BPF_ANY}, 120) = 0
bpf(BPF_MAP_UPDATE_ELEM, {map_fd=5, key=0x7ffda325505c, value=0x7ffda3255058, flags=BPF_ANY}, 120) = 0
bpf(BPF_MAP_UPDATE_ELEM, {map_fd=5, key=0x7ffda325505c, value=0x7ffda3255058, flags=BPF_ANY}, 120) = 0
bpf(BPF_MAP_UPDATE_ELEM, {map_fd=5, key=0x7ffda325505c, value=0x7ffda3255058, flags=BPF_ANY}, 120) = 0
bpf(BPF_MAP_UPDATE_ELEM, {map_fd=5, key=0x7ffda325505c, value=0x7ffda3255058, flags=BPF_ANY}, 120) = 0
bpf(BPF_MAP_UPDATE_ELEM, {map_fd=5, key=0x7ffda325505c, value=0x7ffda3255058, flags=BPF_ANY}, 120) = 0
bpf(BPF_MAP_UPDATE_ELEM, {map_fd=5, key=0x7ffda325505c, value=0x7ffda3255058, flags=BPF_ANY}, 120) = 0
bpf(BPF_MAP_UPDATE_ELEM, {map_fd=5, key=0x7ffda325505c, value=0x7ffda3255058, flags=BPF_ANY}, 120) = 0
bpf(BPF_MAP_UPDATE_ELEM, {map_fd=5, key=0x7ffda325505c, value=0x7ffda3255058, flags=BPF_ANY}, 120) = 0
bpf(BPF_MAP_UPDATE_ELEM, {map_fd=5, key=0x7ffda325505c, value=0x7ffda3255058, flags=BPF_ANY}, 120) = 0
bpf(BPF_MAP_UPDATE_ELEM, {map_fd=5, key=0x7ffda325505c, value=0x7ffda3255058, flags=BPF_ANY}, 120) = 0
bpf(BPF_MAP_UPDATE_ELEM, {map_fd=5, key=0x7ffda325505c, value=0x7ffda3255058, flags=BPF_ANY}, 120) = 0
bpf(BPF_MAP_UPDATE_ELEM, {map_fd=5, key=0x7ffda325505c, value=0x7ffda3255058, flags=BPF_ANY}, 120) = 0
bpf(BPF_OBJ_GET_INFO_BY_FD, {info={bpf_fd=11, info_len=32, info=0x7ffda32551b0}}, 120) = 0
bpf(BPF_PROG_GET_FD_BY_ID, {prog_id=192, next_id=0, open_flags=0}, 120) = 4
bpf(BPF_MAP_GET_FD_BY_ID, {map_id=195, next_id=0, open_flags=0}, 120) = 5
bpf(BPF_PROG_TEST_RUN, {test={prog_fd=4, retval=0, data_size_in=0, data_size_out=0, data_in=NULL, data_out=NULL, repeat=0, duration=0, ctx_size_in=0, ctx_size_out=0, ctx_in=NULL, ctx_out=NULL}, ...}, 120) = 0
libbpf: elf: skipping unrecognized data section(8) .eh_frame
libbpf: elf: skipping relo section(13) .rel.eh_frame for section(8) .eh_frame
bpf(BPF_OBJ_GET_INFO_BY_FD, {info={bpf_fd=4, info_len=216, info=0x7ffda3254f50}}, 120) = 0
bpf(BPF_OBJ_GET_INFO_BY_FD, {info={bpf_fd=4, info_len=216, info=0x1b72d20}}, 120) = 0
bpf(BPF_BTF_GET_FD_BY_ID, {btf_id=305}, 120) = 7
bpf(BPF_OBJ_GET_INFO_BY_FD, {info={bpf_fd=7, info_len=32, info=0x7ffda3255000}}, 120) = 0
bpf(BPF_OBJ_GET_INFO_BY_FD, {info={bpf_fd=5, info_len=80, info=0x7ffda32550f0}}, 120) = 0
bpf(BPF_PROG_LOAD, {prog_type=BPF_PROG_TYPE_SOCKET_FILTER, insn_cnt=2, insns=0x7ffda3254ff0, license="GPL", log_level=0, log_size=0, log_buf=NULL, kern_version=KERNEL_VERSION(0, 0, 0), prog_flags=0, prog_name="", prog_ifindex=0, expected_attach_type=BPF_CGROUP_INET_INGRESS, prog_btf_fd=0, func_info_rec_size=0, func_info=NULL, func_info_cnt=0, line_info_rec_size=0, line_info=NULL, line_info_cnt=0, attach_btf_id=0, attach_prog_fd=0}, 120) = 8
bpf(BPF_BTF_LOAD, {btf="\237\353\1\0\30\0\0\0\0\0\0\0\204\2\0\0\204\2\0\0\250\3\0\0\0\0\0\0\0\0\0\2"..., btf_log_buf=NULL, btf_size=1604, btf_log_size=0, btf_log_level=0}, 120) = 8
bpf(BPF_MAP_CREATE, {map_type=BPF_MAP_TYPE_ARRAY, key_size=4, value_size=32, max_entries=1, map_flags=0, inner_map_fd=0, map_name="", map_ifindex=0, btf_fd=0, btf_key_type_id=0, btf_value_type_id=0, btf_vmlinux_value_type_id=0}, 120) = 9
bpf(BPF_PROG_LOAD, {prog_type=BPF_PROG_TYPE_SOCKET_FILTER, insn_cnt=5, insns=0x7ffda3254e30, license="GPL", log_level=0, log_size=0, log_buf=NULL, kern_version=KERNEL_VERSION(0, 0, 0), prog_flags=0, prog_name="", prog_ifindex=0, expected_attach_type=BPF_CGROUP_INET_INGRESS, prog_btf_fd=0, func_info_rec_size=0, func_info=NULL, func_info_cnt=0, line_info_rec_size=0, line_info=NULL, line_info_cnt=0, attach_btf_id=0, attach_prog_fd=0}, 120) = 10
bpf(BPF_MAP_CREATE, {map_type=BPF_MAP_TYPE_ARRAY, key_size=4, value_size=4, max_entries=1, map_flags=BPF_F_MMAPABLE, inner_map_fd=0, map_name="", map_ifindex=0, btf_fd=0, btf_key_type_id=0, btf_value_type_id=0, btf_vmlinux_value_type_id=0}, 120) = 9
bpf(BPF_MAP_CREATE, {map_type=BPF_MAP_TYPE_PERCPU_ARRAY, key_size=4, value_size=24, max_entries=1, map_flags=0, inner_map_fd=0, map_name="accum_readings", map_ifindex=0, btf_fd=8, btf_key_type_id=0, btf_value_type_id=0, btf_vmlinux_value_type_id=0}, 120) = 9
bpf(BPF_MAP_CREATE, {map_type=BPF_MAP_TYPE_HASH, key_size=4, value_size=4, max_entries=1, map_flags=0, inner_map_fd=0, map_name="filter", map_ifindex=0, btf_fd=8, btf_key_type_id=0, btf_value_type_id=0, btf_vmlinux_value_type_id=0}, 120) = 10
bpf(BPF_MAP_CREATE, {map_type=BPF_MAP_TYPE_ARRAY, key_size=4, value_size=8, max_entries=1, map_flags=BPF_F_MMAPABLE, inner_map_fd=0, map_name="bperf_fo.bss", map_ifindex=0, btf_fd=8, btf_key_type_id=0, btf_value_type_id=27, btf_vmlinux_value_type_id=0}, 120) = 36
bpf(BPF_MAP_UPDATE_ELEM, {map_fd=36, key=0x7ffda3254e50, value=0x7f0a61b70000, flags=BPF_ANY}, 120) = 0
bpf(BPF_PROG_LOAD, {prog_type=BPF_PROG_TYPE_TRACING, insn_cnt=58, insns=0x1b35650, license="Dual BSD/GPL", log_level=0, log_size=0, log_buf=NULL, kern_version=KERNEL_VERSION(5, 12, 0), prog_flags=0, prog_name="fexit_XXX", prog_ifindex=0, expected_attach_type=BPF_TRACE_FEXIT, prog_btf_fd=8, func_info_rec_size=8, func_info=0x1b80000, func_info_cnt=1, line_info_rec_size=16, line_info=0x1b75420, line_info_cnt=28, attach_btf_id=22, attach_prog_fd=4}, 120) = 37
bpf(BPF_MAP_UPDATE_ELEM, {map_fd=10, key=0x7ffda32551b0, value=0x7ffda32551a4, flags=BPF_ANY}, 120) = 0
bpf(BPF_RAW_TRACEPOINT_OPEN, {raw_tracepoint={name=NULL, prog_fd=37}}, 120) = 38
bpf(BPF_OBJ_GET, {pathname="/sys/fs/bpf/perf_attr_map", bpf_fd=0, file_flags=0}, 120) = 3
bpf(BPF_MAP_LOOKUP_ELEM, {map_fd=3, key=0x1b392b0, value=0x7ffda32551a8, flags=BPF_ANY}, 120) = 0
bpf(BPF_LINK_GET_FD_BY_ID, 0x7ffda32550e0, 120) = -1 ENOENT (No such file or directory)
libbpf: elf: skipping unrecognized data section(7) .eh_frame
libbpf: elf: skipping relo section(12) .rel.eh_frame for section(7) .eh_frame
bpf(BPF_PROG_LOAD, {prog_type=BPF_PROG_TYPE_SOCKET_FILTER, insn_cnt=2, insns=0x7ffda3254ff0, license="GPL", log_level=0, log_size=0, log_buf=NULL, kern_version=KERNEL_VERSION(0, 0, 0), prog_flags=0, prog_name="", prog_ifindex=0, expected_attach_type=BPF_CGROUP_INET_INGRESS, prog_btf_fd=0, func_info_rec_size=0, func_info=NULL, func_info_cnt=0, line_info_rec_size=0, line_info=NULL, line_info_cnt=0, attach_btf_id=0, attach_prog_fd=0}, 120) = 5
bpf(BPF_BTF_LOAD, {btf="\237\353\1\0\30\0\0\0\0\0\0\0D\2\0\0D\2\0\0\356\2\0\0\0\0\0\0\0\0\0\2"..., btf_log_buf=NULL, btf_size=1354, btf_log_size=0, btf_log_level=0}, 120) = 5
bpf(BPF_MAP_CREATE, {map_type=BPF_MAP_TYPE_PERF_EVENT_ARRAY, key_size=4, value_size=4, max_entries=24, map_flags=BPF_F_PRESERVE_ELEMS, inner_map_fd=0, map_name="events", map_ifindex=0, btf_fd=5, btf_key_type_id=0, btf_value_type_id=0, btf_vmlinux_value_type_id=0}, 120) = 39
bpf(BPF_MAP_CREATE, {map_type=BPF_MAP_TYPE_PERCPU_ARRAY, key_size=4, value_size=24, max_entries=1, map_flags=0, inner_map_fd=0, map_name="prev_readings", map_ifindex=0, btf_fd=5, btf_key_type_id=0, btf_value_type_id=0, btf_vmlinux_value_type_id=0}, 120) = 40
bpf(BPF_MAP_CREATE, {map_type=BPF_MAP_TYPE_PERCPU_ARRAY, key_size=4, value_size=24, max_entries=1, map_flags=0, inner_map_fd=0, map_name="diff_readings", map_ifindex=0, btf_fd=5, btf_key_type_id=0, btf_value_type_id=0, btf_vmlinux_value_type_id=0}, 120) = 41
bpf(BPF_PROG_LOAD, {prog_type=BPF_PROG_TYPE_RAW_TRACEPOINT, insn_cnt=48, insns=0x1b7fd00, license="Dual BSD/GPL", log_level=0, log_size=0, log_buf=NULL, kern_version=KERNEL_VERSION(5, 12, 0), prog_flags=0, prog_name="on_switch", prog_ifindex=0, expected_attach_type=BPF_CGROUP_INET_INGRESS, prog_btf_fd=5, func_info_rec_size=8, func_info=0x1b78bb0, func_info_cnt=1, line_info_rec_size=16, line_info=0x1b761b0, line_info_cnt=25, attach_btf_id=0, attach_prog_fd=0}, 120) = 42
bpf(BPF_RAW_TRACEPOINT_OPEN, {raw_tracepoint={name="sched_switch", prog_fd=42}}, 120) = 43
bpf(BPF_OBJ_GET_INFO_BY_FD, {info={bpf_fd=43, info_len=32, info=0x7ffda32551b0}}, 120) = 0
bpf(BPF_OBJ_GET_INFO_BY_FD, {info={bpf_fd=41, info_len=80, info=0x7ffda3255200}}, 120) = 0
bpf(BPF_MAP_UPDATE_ELEM, {map_fd=3, key=0x1b392b0, value=0x7ffda32551a8, flags=BPF_ANY}, 120) = 0
bpf(BPF_LINK_GET_FD_BY_ID, 0x7ffda32550e0, 120) = 44
bpf(BPF_MAP_UPDATE_ELEM, {map_fd=39, key=0x7ffda325505c, value=0x7ffda3255058, flags=BPF_ANY}, 120) = 0
bpf(BPF_MAP_UPDATE_ELEM, {map_fd=39, key=0x7ffda325505c, value=0x7ffda3255058, flags=BPF_ANY}, 120) = 0
bpf(BPF_MAP_UPDATE_ELEM, {map_fd=39, key=0x7ffda325505c, value=0x7ffda3255058, flags=BPF_ANY}, 120) = 0
bpf(BPF_MAP_UPDATE_ELEM, {map_fd=39, key=0x7ffda325505c, value=0x7ffda3255058, flags=BPF_ANY}, 120) = 0
bpf(BPF_MAP_UPDATE_ELEM, {map_fd=39, key=0x7ffda325505c, value=0x7ffda3255058, flags=BPF_ANY}, 120) = 0
bpf(BPF_MAP_UPDATE_ELEM, {map_fd=39, key=0x7ffda325505c, value=0x7ffda3255058, flags=BPF_ANY}, 120) = 0
bpf(BPF_MAP_UPDATE_ELEM, {map_fd=39, key=0x7ffda325505c, value=0x7ffda3255058, flags=BPF_ANY}, 120) = 0
bpf(BPF_MAP_UPDATE_ELEM, {map_fd=39, key=0x7ffda325505c, value=0x7ffda3255058, flags=BPF_ANY}, 120) = 0
bpf(BPF_MAP_UPDATE_ELEM, {map_fd=39, key=0x7ffda325505c, value=0x7ffda3255058, flags=BPF_ANY}, 120) = 0
bpf(BPF_MAP_UPDATE_ELEM, {map_fd=39, key=0x7ffda325505c, value=0x7ffda3255058, flags=BPF_ANY}, 120) = 0
bpf(BPF_MAP_UPDATE_ELEM, {map_fd=39, key=0x7ffda325505c, value=0x7ffda3255058, flags=BPF_ANY}, 120) = 0
bpf(BPF_MAP_UPDATE_ELEM, {map_fd=39, key=0x7ffda325505c, value=0x7ffda3255058, flags=BPF_ANY}, 120) = 0
bpf(BPF_MAP_UPDATE_ELEM, {map_fd=39, key=0x7ffda325505c, value=0x7ffda3255058, flags=BPF_ANY}, 120) = 0
bpf(BPF_MAP_UPDATE_ELEM, {map_fd=39, key=0x7ffda325505c, value=0x7ffda3255058, flags=BPF_ANY}, 120) = 0
bpf(BPF_MAP_UPDATE_ELEM, {map_fd=39, key=0x7ffda325505c, value=0x7ffda3255058, flags=BPF_ANY}, 120) = 0
bpf(BPF_MAP_UPDATE_ELEM, {map_fd=39, key=0x7ffda325505c, value=0x7ffda3255058, flags=BPF_ANY}, 120) = 0
bpf(BPF_MAP_UPDATE_ELEM, {map_fd=39, key=0x7ffda325505c, value=0x7ffda3255058, flags=BPF_ANY}, 120) = 0
bpf(BPF_MAP_UPDATE_ELEM, {map_fd=39, key=0x7ffda325505c, value=0x7ffda3255058, flags=BPF_ANY}, 120) = 0
bpf(BPF_MAP_UPDATE_ELEM, {map_fd=39, key=0x7ffda325505c, value=0x7ffda3255058, flags=BPF_ANY}, 120) = 0
bpf(BPF_MAP_UPDATE_ELEM, {map_fd=39, key=0x7ffda325505c, value=0x7ffda3255058, flags=BPF_ANY}, 120) = 0
bpf(BPF_MAP_UPDATE_ELEM, {map_fd=39, key=0x7ffda325505c, value=0x7ffda3255058, flags=BPF_ANY}, 120) = 0
bpf(BPF_MAP_UPDATE_ELEM, {map_fd=39, key=0x7ffda325505c, value=0x7ffda3255058, flags=BPF_ANY}, 120) = 0
bpf(BPF_MAP_UPDATE_ELEM, {map_fd=39, key=0x7ffda325505c, value=0x7ffda3255058, flags=BPF_ANY}, 120) = 0
bpf(BPF_MAP_UPDATE_ELEM, {map_fd=39, key=0x7ffda325505c, value=0x7ffda3255058, flags=BPF_ANY}, 120) = 0
bpf(BPF_OBJ_GET_INFO_BY_FD, {info={bpf_fd=44, info_len=32, info=0x7ffda32551b0}}, 120) = 0
bpf(BPF_PROG_GET_FD_BY_ID, {prog_id=197, next_id=0, open_flags=0}, 120) = 5
bpf(BPF_MAP_GET_FD_BY_ID, {map_id=203, next_id=0, open_flags=0}, 120) = 39
bpf(BPF_PROG_TEST_RUN, {test={prog_fd=5, retval=0, data_size_in=0, data_size_out=0, data_in=NULL, data_out=NULL, repeat=0, duration=0, ctx_size_in=0, ctx_size_out=0, ctx_in=NULL, ctx_out=NULL}, ...}, 120) = 0
libbpf: elf: skipping unrecognized data section(8) .eh_frame
libbpf: elf: skipping relo section(13) .rel.eh_frame for section(8) .eh_frame
bpf(BPF_OBJ_GET_INFO_BY_FD, {info={bpf_fd=5, info_len=216, info=0x7ffda3254f50}}, 120) = 0
bpf(BPF_OBJ_GET_INFO_BY_FD, {info={bpf_fd=5, info_len=216, info=0x1b72d20}}, 120) = 0
bpf(BPF_BTF_GET_FD_BY_ID, {btf_id=307}, 120) = 40
bpf(BPF_OBJ_GET_INFO_BY_FD, {info={bpf_fd=40, info_len=32, info=0x7ffda3255000}}, 120) = 0
bpf(BPF_OBJ_GET_INFO_BY_FD, {info={bpf_fd=39, info_len=80, info=0x7ffda32550f0}}, 120) = 0
bpf(BPF_PROG_LOAD, {prog_type=BPF_PROG_TYPE_SOCKET_FILTER, insn_cnt=2, insns=0x7ffda3254ff0, license="GPL", log_level=0, log_size=0, log_buf=NULL, kern_version=KERNEL_VERSION(0, 0, 0), prog_flags=0, prog_name="", prog_ifindex=0, expected_attach_type=BPF_CGROUP_INET_INGRESS, prog_btf_fd=0, func_info_rec_size=0, func_info=NULL, func_info_cnt=0, line_info_rec_size=0, line_info=NULL, line_info_cnt=0, attach_btf_id=0, attach_prog_fd=0}, 120) = 41
bpf(BPF_BTF_LOAD, {btf="\237\353\1\0\30\0\0\0\0\0\0\0\204\2\0\0\204\2\0\0\250\3\0\0\0\0\0\0\0\0\0\2"..., btf_log_buf=NULL, btf_size=1604, btf_log_size=0, btf_log_level=0}, 120) = 41
bpf(BPF_MAP_CREATE, {map_type=BPF_MAP_TYPE_PERCPU_ARRAY, key_size=4, value_size=24, max_entries=1, map_flags=0, inner_map_fd=0, map_name="accum_readings", map_ifindex=0, btf_fd=41, btf_key_type_id=0, btf_value_type_id=0, btf_vmlinux_value_type_id=0}, 120) = 42
bpf(BPF_MAP_CREATE, {map_type=BPF_MAP_TYPE_HASH, key_size=4, value_size=4, max_entries=1, map_flags=0, inner_map_fd=0, map_name="filter", map_ifindex=0, btf_fd=41, btf_key_type_id=0, btf_value_type_id=0, btf_vmlinux_value_type_id=0}, 120) = 43
bpf(BPF_MAP_CREATE, {map_type=BPF_MAP_TYPE_ARRAY, key_size=4, value_size=8, max_entries=1, map_flags=BPF_F_MMAPABLE, inner_map_fd=0, map_name="bperf_fo.bss", map_ifindex=0, btf_fd=41, btf_key_type_id=0, btf_value_type_id=27, btf_vmlinux_value_type_id=0}, 120) = 69
bpf(BPF_MAP_UPDATE_ELEM, {map_fd=69, key=0x7ffda3254e50, value=0x7f0a61b6f000, flags=BPF_ANY}, 120) = 0
bpf(BPF_PROG_LOAD, {prog_type=BPF_PROG_TYPE_TRACING, insn_cnt=58, insns=0x1b35650, license="Dual BSD/GPL", log_level=0, log_size=0, log_buf=NULL, kern_version=KERNEL_VERSION(5, 12, 0), prog_flags=0, prog_name="fexit_XXX", prog_ifindex=0, expected_attach_type=BPF_TRACE_FEXIT, prog_btf_fd=41, func_info_rec_size=8, func_info=0x1b805c0, func_info_cnt=1, line_info_rec_size=16, line_info=0x1b78100, line_info_cnt=28, attach_btf_id=22, attach_prog_fd=5}, 120) = 70
bpf(BPF_MAP_UPDATE_ELEM, {map_fd=43, key=0x7ffda32551b0, value=0x7ffda32551a4, flags=BPF_ANY}, 120) = 0
bpf(BPF_RAW_TRACEPOINT_OPEN, {raw_tracepoint={name=NULL, prog_fd=70}}, 120) = 71
bpf(BPF_OBJ_GET, {pathname="/sys/fs/bpf/perf_attr_map", bpf_fd=0, file_flags=0}, 120) = 3
bpf(BPF_MAP_LOOKUP_ELEM, {map_fd=3, key=0x1b36840, value=0x7ffda32551a8, flags=BPF_ANY}, 120) = 0
bpf(BPF_LINK_GET_FD_BY_ID, 0x7ffda32550e0, 120) = -1 ENOENT (No such file or directory)
libbpf: elf: skipping unrecognized data section(7) .eh_frame
libbpf: elf: skipping relo section(12) .rel.eh_frame for section(7) .eh_frame
bpf(BPF_PROG_LOAD, {prog_type=BPF_PROG_TYPE_SOCKET_FILTER, insn_cnt=2, insns=0x7ffda3254ff0, license="GPL", log_level=0, log_size=0, log_buf=NULL, kern_version=KERNEL_VERSION(0, 0, 0), prog_flags=0, prog_name="", prog_ifindex=0, expected_attach_type=BPF_CGROUP_INET_INGRESS, prog_btf_fd=0, func_info_rec_size=0, func_info=NULL, func_info_cnt=0, line_info_rec_size=0, line_info=NULL, line_info_cnt=0, attach_btf_id=0, attach_prog_fd=0}, 120) = 39
bpf(BPF_BTF_LOAD, {btf="\237\353\1\0\30\0\0\0\0\0\0\0D\2\0\0D\2\0\0\356\2\0\0\0\0\0\0\0\0\0\2"..., btf_log_buf=NULL, btf_size=1354, btf_log_size=0, btf_log_level=0}, 120) = 39
bpf(BPF_MAP_CREATE, {map_type=BPF_MAP_TYPE_PERF_EVENT_ARRAY, key_size=4, value_size=4, max_entries=24, map_flags=BPF_F_PRESERVE_ELEMS, inner_map_fd=0, map_name="events", map_ifindex=0, btf_fd=39, btf_key_type_id=0, btf_value_type_id=0, btf_vmlinux_value_type_id=0}, 120) = 72
bpf(BPF_MAP_CREATE, {map_type=BPF_MAP_TYPE_PERCPU_ARRAY, key_size=4, value_size=24, max_entries=1, map_flags=0, inner_map_fd=0, map_name="prev_readings", map_ifindex=0, btf_fd=39, btf_key_type_id=0, btf_value_type_id=0, btf_vmlinux_value_type_id=0}, 120) = 73
bpf(BPF_MAP_CREATE, {map_type=BPF_MAP_TYPE_PERCPU_ARRAY, key_size=4, value_size=24, max_entries=1, map_flags=0, inner_map_fd=0, map_name="diff_readings", map_ifindex=0, btf_fd=39, btf_key_type_id=0, btf_value_type_id=0, btf_vmlinux_value_type_id=0}, 120) = 74
bpf(BPF_PROG_LOAD, {prog_type=BPF_PROG_TYPE_RAW_TRACEPOINT, insn_cnt=48, insns=0x1b7fd00, license="Dual BSD/GPL", log_level=0, log_size=0, log_buf=NULL, kern_version=KERNEL_VERSION(5, 12, 0), prog_flags=0, prog_name="on_switch", prog_ifindex=0, expected_attach_type=BPF_CGROUP_INET_INGRESS, prog_btf_fd=39, func_info_rec_size=8, func_info=0x1b813d0, func_info_cnt=1, line_info_rec_size=16, line_info=0x1b761b0, line_info_cnt=25, attach_btf_id=0, attach_prog_fd=0}, 120) = 75
bpf(BPF_RAW_TRACEPOINT_OPEN, {raw_tracepoint={name="sched_switch", prog_fd=75}}, 120) = 76
bpf(BPF_OBJ_GET_INFO_BY_FD, {info={bpf_fd=76, info_len=32, info=0x7ffda32551b0}}, 120) = 0
bpf(BPF_OBJ_GET_INFO_BY_FD, {info={bpf_fd=74, info_len=80, info=0x7ffda3255200}}, 120) = 0
bpf(BPF_MAP_UPDATE_ELEM, {map_fd=3, key=0x1b36840, value=0x7ffda32551a8, flags=BPF_ANY}, 120) = 0
bpf(BPF_LINK_GET_FD_BY_ID, 0x7ffda32550e0, 120) = 77
bpf(BPF_MAP_UPDATE_ELEM, {map_fd=72, key=0x7ffda325505c, value=0x7ffda3255058, flags=BPF_ANY}, 120) = 0
bpf(BPF_MAP_UPDATE_ELEM, {map_fd=72, key=0x7ffda325505c, value=0x7ffda3255058, flags=BPF_ANY}, 120) = 0
bpf(BPF_MAP_UPDATE_ELEM, {map_fd=72, key=0x7ffda325505c, value=0x7ffda3255058, flags=BPF_ANY}, 120) = 0
bpf(BPF_MAP_UPDATE_ELEM, {map_fd=72, key=0x7ffda325505c, value=0x7ffda3255058, flags=BPF_ANY}, 120) = 0
bpf(BPF_MAP_UPDATE_ELEM, {map_fd=72, key=0x7ffda325505c, value=0x7ffda3255058, flags=BPF_ANY}, 120) = 0
bpf(BPF_MAP_UPDATE_ELEM, {map_fd=72, key=0x7ffda325505c, value=0x7ffda3255058, flags=BPF_ANY}, 120) = 0
bpf(BPF_MAP_UPDATE_ELEM, {map_fd=72, key=0x7ffda325505c, value=0x7ffda3255058, flags=BPF_ANY}, 120) = 0
bpf(BPF_MAP_UPDATE_ELEM, {map_fd=72, key=0x7ffda325505c, value=0x7ffda3255058, flags=BPF_ANY}, 120) = 0
bpf(BPF_MAP_UPDATE_ELEM, {map_fd=72, key=0x7ffda325505c, value=0x7ffda3255058, flags=BPF_ANY}, 120) = 0
bpf(BPF_MAP_UPDATE_ELEM, {map_fd=72, key=0x7ffda325505c, value=0x7ffda3255058, flags=BPF_ANY}, 120) = 0
bpf(BPF_MAP_UPDATE_ELEM, {map_fd=72, key=0x7ffda325505c, value=0x7ffda3255058, flags=BPF_ANY}, 120) = 0
bpf(BPF_MAP_UPDATE_ELEM, {map_fd=72, key=0x7ffda325505c, value=0x7ffda3255058, flags=BPF_ANY}, 120) = 0
bpf(BPF_MAP_UPDATE_ELEM, {map_fd=72, key=0x7ffda325505c, value=0x7ffda3255058, flags=BPF_ANY}, 120) = 0
bpf(BPF_MAP_UPDATE_ELEM, {map_fd=72, key=0x7ffda325505c, value=0x7ffda3255058, flags=BPF_ANY}, 120) = 0
bpf(BPF_MAP_UPDATE_ELEM, {map_fd=72, key=0x7ffda325505c, value=0x7ffda3255058, flags=BPF_ANY}, 120) = 0
bpf(BPF_MAP_UPDATE_ELEM, {map_fd=72, key=0x7ffda325505c, value=0x7ffda3255058, flags=BPF_ANY}, 120) = 0
bpf(BPF_MAP_UPDATE_ELEM, {map_fd=72, key=0x7ffda325505c, value=0x7ffda3255058, flags=BPF_ANY}, 120) = 0
bpf(BPF_MAP_UPDATE_ELEM, {map_fd=72, key=0x7ffda325505c, value=0x7ffda3255058, flags=BPF_ANY}, 120) = 0
bpf(BPF_MAP_UPDATE_ELEM, {map_fd=72, key=0x7ffda325505c, value=0x7ffda3255058, flags=BPF_ANY}, 120) = 0
bpf(BPF_MAP_UPDATE_ELEM, {map_fd=72, key=0x7ffda325505c, value=0x7ffda3255058, flags=BPF_ANY}, 120) = 0
bpf(BPF_MAP_UPDATE_ELEM, {map_fd=72, key=0x7ffda325505c, value=0x7ffda3255058, flags=BPF_ANY}, 120) = 0
bpf(BPF_MAP_UPDATE_ELEM, {map_fd=72, key=0x7ffda325505c, value=0x7ffda3255058, flags=BPF_ANY}, 120) = 0
bpf(BPF_MAP_UPDATE_ELEM, {map_fd=72, key=0x7ffda325505c, value=0x7ffda3255058, flags=BPF_ANY}, 120) = 0
bpf(BPF_MAP_UPDATE_ELEM, {map_fd=72, key=0x7ffda325505c, value=0x7ffda3255058, flags=BPF_ANY}, 120) = 0
bpf(BPF_OBJ_GET_INFO_BY_FD, {info={bpf_fd=77, info_len=32, info=0x7ffda32551b0}}, 120) = 0
bpf(BPF_PROG_GET_FD_BY_ID, {prog_id=201, next_id=0, open_flags=0}, 120) = 39
bpf(BPF_MAP_GET_FD_BY_ID, {map_id=209, next_id=0, open_flags=0}, 120) = 72
bpf(BPF_PROG_TEST_RUN, {test={prog_fd=39, retval=0, data_size_in=0, data_size_out=0, data_in=NULL, data_out=NULL, repeat=0, duration=0, ctx_size_in=0, ctx_size_out=0, ctx_in=NULL, ctx_out=NULL}, ...}, 120) = 0
libbpf: elf: skipping unrecognized data section(8) .eh_frame
libbpf: elf: skipping relo section(13) .rel.eh_frame for section(8) .eh_frame
bpf(BPF_OBJ_GET_INFO_BY_FD, {info={bpf_fd=39, info_len=216, info=0x7ffda3254f50}}, 120) = 0
bpf(BPF_OBJ_GET_INFO_BY_FD, {info={bpf_fd=39, info_len=216, info=0x1b72d20}}, 120) = 0
bpf(BPF_BTF_GET_FD_BY_ID, {btf_id=309}, 120) = 73
bpf(BPF_OBJ_GET_INFO_BY_FD, {info={bpf_fd=73, info_len=32, info=0x7ffda3255000}}, 120) = 0
bpf(BPF_OBJ_GET_INFO_BY_FD, {info={bpf_fd=72, info_len=80, info=0x7ffda32550f0}}, 120) = 0
bpf(BPF_PROG_LOAD, {prog_type=BPF_PROG_TYPE_SOCKET_FILTER, insn_cnt=2, insns=0x7ffda3254ff0, license="GPL", log_level=0, log_size=0, log_buf=NULL, kern_version=KERNEL_VERSION(0, 0, 0), prog_flags=0, prog_name="", prog_ifindex=0, expected_attach_type=BPF_CGROUP_INET_INGRESS, prog_btf_fd=0, func_info_rec_size=0, func_info=NULL, func_info_cnt=0, line_info_rec_size=0, line_info=NULL, line_info_cnt=0, attach_btf_id=0, attach_prog_fd=0}, 120) = 74
bpf(BPF_BTF_LOAD, {btf="\237\353\1\0\30\0\0\0\0\0\0\0\204\2\0\0\204\2\0\0\250\3\0\0\0\0\0\0\0\0\0\2"..., btf_log_buf=NULL, btf_size=1604, btf_log_size=0, btf_log_level=0}, 120) = 74
bpf(BPF_MAP_CREATE, {map_type=BPF_MAP_TYPE_PERCPU_ARRAY, key_size=4, value_size=24, max_entries=1, map_flags=0, inner_map_fd=0, map_name="accum_readings", map_ifindex=0, btf_fd=74, btf_key_type_id=0, btf_value_type_id=0, btf_vmlinux_value_type_id=0}, 120) = 75
bpf(BPF_MAP_CREATE, {map_type=BPF_MAP_TYPE_HASH, key_size=4, value_size=4, max_entries=1, map_flags=0, inner_map_fd=0, map_name="filter", map_ifindex=0, btf_fd=74, btf_key_type_id=0, btf_value_type_id=0, btf_vmlinux_value_type_id=0}, 120) = 76
bpf(BPF_MAP_CREATE, {map_type=BPF_MAP_TYPE_ARRAY, key_size=4, value_size=8, max_entries=1, map_flags=BPF_F_MMAPABLE, inner_map_fd=0, map_name="bperf_fo.bss", map_ifindex=0, btf_fd=74, btf_key_type_id=0, btf_value_type_id=27, btf_vmlinux_value_type_id=0}, 120) = 102
bpf(BPF_MAP_UPDATE_ELEM, {map_fd=102, key=0x7ffda3254e50, value=0x7f0a61b6e000, flags=BPF_ANY}, 120) = 0
bpf(BPF_PROG_LOAD, {prog_type=BPF_PROG_TYPE_TRACING, insn_cnt=58, insns=0x1b35650, license="Dual BSD/GPL", log_level=0, log_size=0, log_buf=NULL, kern_version=KERNEL_VERSION(5, 12, 0), prog_flags=0, prog_name="fexit_XXX", prog_ifindex=0, expected_attach_type=BPF_TRACE_FEXIT, prog_btf_fd=74, func_info_rec_size=8, func_info=0x1b79b40, func_info_cnt=1, line_info_rec_size=16, line_info=0x1b786a0, line_info_cnt=28, attach_btf_id=22, attach_prog_fd=39}, 120) = 103
bpf(BPF_MAP_UPDATE_ELEM, {map_fd=76, key=0x7ffda32551b0, value=0x7ffda32551a4, flags=BPF_ANY}, 120) = 0
bpf(BPF_RAW_TRACEPOINT_OPEN, {raw_tracepoint={name=NULL, prog_fd=103}}, 120) = 104
bpf(BPF_OBJ_GET, {pathname="/sys/fs/bpf/perf_attr_map", bpf_fd=0, file_flags=0}, 120) = 3
bpf(BPF_MAP_LOOKUP_ELEM, {map_fd=3, key=0x1b36a70, value=0x7ffda32551a8, flags=BPF_ANY}, 120) = 0
bpf(BPF_LINK_GET_FD_BY_ID, 0x7ffda32550e0, 120) = -1 ENOENT (No such file or directory)
libbpf: elf: skipping unrecognized data section(7) .eh_frame
libbpf: elf: skipping relo section(12) .rel.eh_frame for section(7) .eh_frame
bpf(BPF_PROG_LOAD, {prog_type=BPF_PROG_TYPE_SOCKET_FILTER, insn_cnt=2, insns=0x7ffda3254ff0, license="GPL", log_level=0, log_size=0, log_buf=NULL, kern_version=KERNEL_VERSION(0, 0, 0), prog_flags=0, prog_name="", prog_ifindex=0, expected_attach_type=BPF_CGROUP_INET_INGRESS, prog_btf_fd=0, func_info_rec_size=0, func_info=NULL, func_info_cnt=0, line_info_rec_size=0, line_info=NULL, line_info_cnt=0, attach_btf_id=0, attach_prog_fd=0}, 120) = 72
bpf(BPF_BTF_LOAD, {btf="\237\353\1\0\30\0\0\0\0\0\0\0D\2\0\0D\2\0\0\356\2\0\0\0\0\0\0\0\0\0\2"..., btf_log_buf=NULL, btf_size=1354, btf_log_size=0, btf_log_level=0}, 120) = 72
bpf(BPF_MAP_CREATE, {map_type=BPF_MAP_TYPE_PERF_EVENT_ARRAY, key_size=4, value_size=4, max_entries=24, map_flags=BPF_F_PRESERVE_ELEMS, inner_map_fd=0, map_name="events", map_ifindex=0, btf_fd=72, btf_key_type_id=0, btf_value_type_id=0, btf_vmlinux_value_type_id=0}, 120) = 105
bpf(BPF_MAP_CREATE, {map_type=BPF_MAP_TYPE_PERCPU_ARRAY, key_size=4, value_size=24, max_entries=1, map_flags=0, inner_map_fd=0, map_name="prev_readings", map_ifindex=0, btf_fd=72, btf_key_type_id=0, btf_value_type_id=0, btf_vmlinux_value_type_id=0}, 120) = 106
bpf(BPF_MAP_CREATE, {map_type=BPF_MAP_TYPE_PERCPU_ARRAY, key_size=4, value_size=24, max_entries=1, map_flags=0, inner_map_fd=0, map_name="diff_readings", map_ifindex=0, btf_fd=72, btf_key_type_id=0, btf_value_type_id=0, btf_vmlinux_value_type_id=0}, 120) = 107
bpf(BPF_PROG_LOAD, {prog_type=BPF_PROG_TYPE_RAW_TRACEPOINT, insn_cnt=48, insns=0x1b7fd00, license="Dual BSD/GPL", log_level=0, log_size=0, log_buf=NULL, kern_version=KERNEL_VERSION(5, 12, 0), prog_flags=0, prog_name="on_switch", prog_ifindex=0, expected_attach_type=BPF_CGROUP_INET_INGRESS, prog_btf_fd=72, func_info_rec_size=8, func_info=0x1b75a70, func_info_cnt=1, line_info_rec_size=16, line_info=0x1b761b0, line_info_cnt=25, attach_btf_id=0, attach_prog_fd=0}, 120) = 108
bpf(BPF_RAW_TRACEPOINT_OPEN, {raw_tracepoint={name="sched_switch", prog_fd=108}}, 120) = 109
bpf(BPF_OBJ_GET_INFO_BY_FD, {info={bpf_fd=109, info_len=32, info=0x7ffda32551b0}}, 120) = 0
bpf(BPF_OBJ_GET_INFO_BY_FD, {info={bpf_fd=107, info_len=80, info=0x7ffda3255200}}, 120) = 0
bpf(BPF_MAP_UPDATE_ELEM, {map_fd=3, key=0x1b36a70, value=0x7ffda32551a8, flags=BPF_ANY}, 120) = 0
bpf(BPF_LINK_GET_FD_BY_ID, 0x7ffda32550e0, 120) = 110
bpf(BPF_MAP_UPDATE_ELEM, {map_fd=105, key=0x7ffda325505c, value=0x7ffda3255058, flags=BPF_ANY}, 120) = 0
bpf(BPF_MAP_UPDATE_ELEM, {map_fd=105, key=0x7ffda325505c, value=0x7ffda3255058, flags=BPF_ANY}, 120) = 0
bpf(BPF_MAP_UPDATE_ELEM, {map_fd=105, key=0x7ffda325505c, value=0x7ffda3255058, flags=BPF_ANY}, 120) = 0
bpf(BPF_MAP_UPDATE_ELEM, {map_fd=105, key=0x7ffda325505c, value=0x7ffda3255058, flags=BPF_ANY}, 120) = 0
bpf(BPF_MAP_UPDATE_ELEM, {map_fd=105, key=0x7ffda325505c, value=0x7ffda3255058, flags=BPF_ANY}, 120) = 0
bpf(BPF_MAP_UPDATE_ELEM, {map_fd=105, key=0x7ffda325505c, value=0x7ffda3255058, flags=BPF_ANY}, 120) = 0
bpf(BPF_MAP_UPDATE_ELEM, {map_fd=105, key=0x7ffda325505c, value=0x7ffda3255058, flags=BPF_ANY}, 120) = 0
bpf(BPF_MAP_UPDATE_ELEM, {map_fd=105, key=0x7ffda325505c, value=0x7ffda3255058, flags=BPF_ANY}, 120) = 0
bpf(BPF_MAP_UPDATE_ELEM, {map_fd=105, key=0x7ffda325505c, value=0x7ffda3255058, flags=BPF_ANY}, 120) = 0
bpf(BPF_MAP_UPDATE_ELEM, {map_fd=105, key=0x7ffda325505c, value=0x7ffda3255058, flags=BPF_ANY}, 120) = 0
bpf(BPF_MAP_UPDATE_ELEM, {map_fd=105, key=0x7ffda325505c, value=0x7ffda3255058, flags=BPF_ANY}, 120) = 0
bpf(BPF_MAP_UPDATE_ELEM, {map_fd=105, key=0x7ffda325505c, value=0x7ffda3255058, flags=BPF_ANY}, 120) = 0
bpf(BPF_MAP_UPDATE_ELEM, {map_fd=105, key=0x7ffda325505c, value=0x7ffda3255058, flags=BPF_ANY}, 120) = 0
bpf(BPF_MAP_UPDATE_ELEM, {map_fd=105, key=0x7ffda325505c, value=0x7ffda3255058, flags=BPF_ANY}, 120) = 0
bpf(BPF_MAP_UPDATE_ELEM, {map_fd=105, key=0x7ffda325505c, value=0x7ffda3255058, flags=BPF_ANY}, 120) = 0
bpf(BPF_MAP_UPDATE_ELEM, {map_fd=105, key=0x7ffda325505c, value=0x7ffda3255058, flags=BPF_ANY}, 120) = 0
bpf(BPF_MAP_UPDATE_ELEM, {map_fd=105, key=0x7ffda325505c, value=0x7ffda3255058, flags=BPF_ANY}, 120) = 0
bpf(BPF_MAP_UPDATE_ELEM, {map_fd=105, key=0x7ffda325505c, value=0x7ffda3255058, flags=BPF_ANY}, 120) = 0
bpf(BPF_MAP_UPDATE_ELEM, {map_fd=105, key=0x7ffda325505c, value=0x7ffda3255058, flags=BPF_ANY}, 120) = 0
bpf(BPF_MAP_UPDATE_ELEM, {map_fd=105, key=0x7ffda325505c, value=0x7ffda3255058, flags=BPF_ANY}, 120) = 0
bpf(BPF_MAP_UPDATE_ELEM, {map_fd=105, key=0x7ffda325505c, value=0x7ffda3255058, flags=BPF_ANY}, 120) = 0
bpf(BPF_MAP_UPDATE_ELEM, {map_fd=105, key=0x7ffda325505c, value=0x7ffda3255058, flags=BPF_ANY}, 120) = 0
bpf(BPF_MAP_UPDATE_ELEM, {map_fd=105, key=0x7ffda325505c, value=0x7ffda3255058, flags=BPF_ANY}, 120) = 0
bpf(BPF_MAP_UPDATE_ELEM, {map_fd=105, key=0x7ffda325505c, value=0x7ffda3255058, flags=BPF_ANY}, 120) = 0
bpf(BPF_OBJ_GET_INFO_BY_FD, {info={bpf_fd=110, info_len=32, info=0x7ffda32551b0}}, 120) = 0
bpf(BPF_PROG_GET_FD_BY_ID, {prog_id=205, next_id=0, open_flags=0}, 120) = 72
bpf(BPF_MAP_GET_FD_BY_ID, {map_id=215, next_id=0, open_flags=0}, 120) = 105
bpf(BPF_PROG_TEST_RUN, {test={prog_fd=72, retval=0, data_size_in=0, data_size_out=0, data_in=NULL, data_out=NULL, repeat=0, duration=0, ctx_size_in=0, ctx_size_out=0, ctx_in=NULL, ctx_out=NULL}, ...}, 120) = 0
libbpf: elf: skipping unrecognized data section(8) .eh_frame
libbpf: elf: skipping relo section(13) .rel.eh_frame for section(8) .eh_frame
bpf(BPF_OBJ_GET_INFO_BY_FD, {info={bpf_fd=72, info_len=216, info=0x7ffda3254f50}}, 120) = 0
bpf(BPF_OBJ_GET_INFO_BY_FD, {info={bpf_fd=72, info_len=216, info=0x1b72d20}}, 120) = 0
bpf(BPF_BTF_GET_FD_BY_ID, {btf_id=311}, 120) = 106
bpf(BPF_OBJ_GET_INFO_BY_FD, {info={bpf_fd=106, info_len=32, info=0x7ffda3255000}}, 120) = 0
bpf(BPF_OBJ_GET_INFO_BY_FD, {info={bpf_fd=105, info_len=80, info=0x7ffda32550f0}}, 120) = 0
bpf(BPF_PROG_LOAD, {prog_type=BPF_PROG_TYPE_SOCKET_FILTER, insn_cnt=2, insns=0x7ffda3254ff0, license="GPL", log_level=0, log_size=0, log_buf=NULL, kern_version=KERNEL_VERSION(0, 0, 0), prog_flags=0, prog_name="", prog_ifindex=0, expected_attach_type=BPF_CGROUP_INET_INGRESS, prog_btf_fd=0, func_info_rec_size=0, func_info=NULL, func_info_cnt=0, line_info_rec_size=0, line_info=NULL, line_info_cnt=0, attach_btf_id=0, attach_prog_fd=0}, 120) = 107
bpf(BPF_BTF_LOAD, {btf="\237\353\1\0\30\0\0\0\0\0\0\0\204\2\0\0\204\2\0\0\250\3\0\0\0\0\0\0\0\0\0\2"..., btf_log_buf=NULL, btf_size=1604, btf_log_size=0, btf_log_level=0}, 120) = 107
bpf(BPF_MAP_CREATE, {map_type=BPF_MAP_TYPE_PERCPU_ARRAY, key_size=4, value_size=24, max_entries=1, map_flags=0, inner_map_fd=0, map_name="accum_readings", map_ifindex=0, btf_fd=107, btf_key_type_id=0, btf_value_type_id=0, btf_vmlinux_value_type_id=0}, 120) = 108
bpf(BPF_MAP_CREATE, {map_type=BPF_MAP_TYPE_HASH, key_size=4, value_size=4, max_entries=1, map_flags=0, inner_map_fd=0, map_name="filter", map_ifindex=0, btf_fd=107, btf_key_type_id=0, btf_value_type_id=0, btf_vmlinux_value_type_id=0}, 120) = 109
bpf(BPF_MAP_CREATE, {map_type=BPF_MAP_TYPE_ARRAY, key_size=4, value_size=8, max_entries=1, map_flags=BPF_F_MMAPABLE, inner_map_fd=0, map_name="bperf_fo.bss", map_ifindex=0, btf_fd=107, btf_key_type_id=0, btf_value_type_id=27, btf_vmlinux_value_type_id=0}, 120) = 135
bpf(BPF_MAP_UPDATE_ELEM, {map_fd=135, key=0x7ffda3254e50, value=0x7f0a61b6d000, flags=BPF_ANY}, 120) = 0
bpf(BPF_PROG_LOAD, {prog_type=BPF_PROG_TYPE_TRACING, insn_cnt=58, insns=0x1b35650, license="Dual BSD/GPL", log_level=0, log_size=0, log_buf=NULL, kern_version=KERNEL_VERSION(5, 12, 0), prog_flags=0, prog_name="fexit_XXX", prog_ifindex=0, expected_attach_type=BPF_TRACE_FEXIT, prog_btf_fd=107, func_info_rec_size=8, func_info=0x1b7b310, func_info_cnt=1, line_info_rec_size=16, line_info=0x1b7b740, line_info_cnt=28, attach_btf_id=22, attach_prog_fd=72}, 120) = 136
bpf(BPF_MAP_UPDATE_ELEM, {map_fd=109, key=0x7ffda32551b0, value=0x7ffda32551a4, flags=BPF_ANY}, 120) = 0
bpf(BPF_RAW_TRACEPOINT_OPEN, {raw_tracepoint={name=NULL, prog_fd=136}}, 120) = 137
bpf(BPF_OBJ_GET, {pathname="/sys/fs/bpf/perf_attr_map", bpf_fd=0, file_flags=0}, 120) = 3
bpf(BPF_MAP_LOOKUP_ELEM, {map_fd=3, key=0x1b3da70, value=0x7ffda32551a8, flags=BPF_ANY}, 120) = 0
bpf(BPF_LINK_GET_FD_BY_ID, 0x7ffda32550e0, 120) = -1 ENOENT (No such file or directory)
libbpf: elf: skipping unrecognized data section(7) .eh_frame
libbpf: elf: skipping relo section(12) .rel.eh_frame for section(7) .eh_frame
bpf(BPF_PROG_LOAD, {prog_type=BPF_PROG_TYPE_SOCKET_FILTER, insn_cnt=2, insns=0x7ffda3254ff0, license="GPL", log_level=0, log_size=0, log_buf=NULL, kern_version=KERNEL_VERSION(0, 0, 0), prog_flags=0, prog_name="", prog_ifindex=0, expected_attach_type=BPF_CGROUP_INET_INGRESS, prog_btf_fd=0, func_info_rec_size=0, func_info=NULL, func_info_cnt=0, line_info_rec_size=0, line_info=NULL, line_info_cnt=0, attach_btf_id=0, attach_prog_fd=0}, 120) = 105
bpf(BPF_BTF_LOAD, {btf="\237\353\1\0\30\0\0\0\0\0\0\0D\2\0\0D\2\0\0\356\2\0\0\0\0\0\0\0\0\0\2"..., btf_log_buf=NULL, btf_size=1354, btf_log_size=0, btf_log_level=0}, 120) = 105
bpf(BPF_MAP_CREATE, {map_type=BPF_MAP_TYPE_PERF_EVENT_ARRAY, key_size=4, value_size=4, max_entries=24, map_flags=BPF_F_PRESERVE_ELEMS, inner_map_fd=0, map_name="events", map_ifindex=0, btf_fd=105, btf_key_type_id=0, btf_value_type_id=0, btf_vmlinux_value_type_id=0}, 120) = 138
bpf(BPF_MAP_CREATE, {map_type=BPF_MAP_TYPE_PERCPU_ARRAY, key_size=4, value_size=24, max_entries=1, map_flags=0, inner_map_fd=0, map_name="prev_readings", map_ifindex=0, btf_fd=105, btf_key_type_id=0, btf_value_type_id=0, btf_vmlinux_value_type_id=0}, 120) = 139
bpf(BPF_MAP_CREATE, {map_type=BPF_MAP_TYPE_PERCPU_ARRAY, key_size=4, value_size=24, max_entries=1, map_flags=0, inner_map_fd=0, map_name="diff_readings", map_ifindex=0, btf_fd=105, btf_key_type_id=0, btf_value_type_id=0, btf_vmlinux_value_type_id=0}, 120) = 140
bpf(BPF_PROG_LOAD, {prog_type=BPF_PROG_TYPE_RAW_TRACEPOINT, insn_cnt=48, insns=0x1b7fd00, license="Dual BSD/GPL", log_level=0, log_size=0, log_buf=NULL, kern_version=KERNEL_VERSION(5, 12, 0), prog_flags=0, prog_name="on_switch", prog_ifindex=0, expected_attach_type=BPF_CGROUP_INET_INGRESS, prog_btf_fd=105, func_info_rec_size=8, func_info=0x1b7a670, func_info_cnt=1, line_info_rec_size=16, line_info=0x1b761b0, line_info_cnt=25, attach_btf_id=0, attach_prog_fd=0}, 120) = 141
bpf(BPF_RAW_TRACEPOINT_OPEN, {raw_tracepoint={name="sched_switch", prog_fd=141}}, 120) = 142
bpf(BPF_OBJ_GET_INFO_BY_FD, {info={bpf_fd=142, info_len=32, info=0x7ffda32551b0}}, 120) = 0
bpf(BPF_OBJ_GET_INFO_BY_FD, {info={bpf_fd=140, info_len=80, info=0x7ffda3255200}}, 120) = 0
bpf(BPF_MAP_UPDATE_ELEM, {map_fd=3, key=0x1b3da70, value=0x7ffda32551a8, flags=BPF_ANY}, 120) = 0
bpf(BPF_LINK_GET_FD_BY_ID, 0x7ffda32550e0, 120) = 143
bpf(BPF_MAP_UPDATE_ELEM, {map_fd=138, key=0x7ffda325505c, value=0x7ffda3255058, flags=BPF_ANY}, 120) = 0
bpf(BPF_MAP_UPDATE_ELEM, {map_fd=138, key=0x7ffda325505c, value=0x7ffda3255058, flags=BPF_ANY}, 120) = 0
bpf(BPF_MAP_UPDATE_ELEM, {map_fd=138, key=0x7ffda325505c, value=0x7ffda3255058, flags=BPF_ANY}, 120) = 0
bpf(BPF_MAP_UPDATE_ELEM, {map_fd=138, key=0x7ffda325505c, value=0x7ffda3255058, flags=BPF_ANY}, 120) = 0
bpf(BPF_MAP_UPDATE_ELEM, {map_fd=138, key=0x7ffda325505c, value=0x7ffda3255058, flags=BPF_ANY}, 120) = 0
bpf(BPF_MAP_UPDATE_ELEM, {map_fd=138, key=0x7ffda325505c, value=0x7ffda3255058, flags=BPF_ANY}, 120) = 0
bpf(BPF_MAP_UPDATE_ELEM, {map_fd=138, key=0x7ffda325505c, value=0x7ffda3255058, flags=BPF_ANY}, 120) = 0
bpf(BPF_MAP_UPDATE_ELEM, {map_fd=138, key=0x7ffda325505c, value=0x7ffda3255058, flags=BPF_ANY}, 120) = 0
bpf(BPF_MAP_UPDATE_ELEM, {map_fd=138, key=0x7ffda325505c, value=0x7ffda3255058, flags=BPF_ANY}, 120) = 0
bpf(BPF_MAP_UPDATE_ELEM, {map_fd=138, key=0x7ffda325505c, value=0x7ffda3255058, flags=BPF_ANY}, 120) = 0
bpf(BPF_MAP_UPDATE_ELEM, {map_fd=138, key=0x7ffda325505c, value=0x7ffda3255058, flags=BPF_ANY}, 120) = 0
bpf(BPF_MAP_UPDATE_ELEM, {map_fd=138, key=0x7ffda325505c, value=0x7ffda3255058, flags=BPF_ANY}, 120) = 0
bpf(BPF_MAP_UPDATE_ELEM, {map_fd=138, key=0x7ffda325505c, value=0x7ffda3255058, flags=BPF_ANY}, 120) = 0
bpf(BPF_MAP_UPDATE_ELEM, {map_fd=138, key=0x7ffda325505c, value=0x7ffda3255058, flags=BPF_ANY}, 120) = 0
bpf(BPF_MAP_UPDATE_ELEM, {map_fd=138, key=0x7ffda325505c, value=0x7ffda3255058, flags=BPF_ANY}, 120) = 0
bpf(BPF_MAP_UPDATE_ELEM, {map_fd=138, key=0x7ffda325505c, value=0x7ffda3255058, flags=BPF_ANY}, 120) = 0
bpf(BPF_MAP_UPDATE_ELEM, {map_fd=138, key=0x7ffda325505c, value=0x7ffda3255058, flags=BPF_ANY}, 120) = 0
bpf(BPF_MAP_UPDATE_ELEM, {map_fd=138, key=0x7ffda325505c, value=0x7ffda3255058, flags=BPF_ANY}, 120) = 0
bpf(BPF_MAP_UPDATE_ELEM, {map_fd=138, key=0x7ffda325505c, value=0x7ffda3255058, flags=BPF_ANY}, 120) = 0
bpf(BPF_MAP_UPDATE_ELEM, {map_fd=138, key=0x7ffda325505c, value=0x7ffda3255058, flags=BPF_ANY}, 120) = 0
bpf(BPF_MAP_UPDATE_ELEM, {map_fd=138, key=0x7ffda325505c, value=0x7ffda3255058, flags=BPF_ANY}, 120) = 0
bpf(BPF_MAP_UPDATE_ELEM, {map_fd=138, key=0x7ffda325505c, value=0x7ffda3255058, flags=BPF_ANY}, 120) = 0
bpf(BPF_MAP_UPDATE_ELEM, {map_fd=138, key=0x7ffda325505c, value=0x7ffda3255058, flags=BPF_ANY}, 120) = 0
bpf(BPF_MAP_UPDATE_ELEM, {map_fd=138, key=0x7ffda325505c, value=0x7ffda3255058, flags=BPF_ANY}, 120) = 0
bpf(BPF_OBJ_GET_INFO_BY_FD, {info={bpf_fd=143, info_len=32, info=0x7ffda32551b0}}, 120) = 0
bpf(BPF_PROG_GET_FD_BY_ID, {prog_id=209, next_id=0, open_flags=0}, 120) = 105
bpf(BPF_MAP_GET_FD_BY_ID, {map_id=221, next_id=0, open_flags=0}, 120) = 138
bpf(BPF_PROG_TEST_RUN, {test={prog_fd=105, retval=0, data_size_in=0, data_size_out=0, data_in=NULL, data_out=NULL, repeat=0, duration=0, ctx_size_in=0, ctx_size_out=0, ctx_in=NULL, ctx_out=NULL}, ...}, 120) = 0
libbpf: elf: skipping unrecognized data section(8) .eh_frame
libbpf: elf: skipping relo section(13) .rel.eh_frame for section(8) .eh_frame
bpf(BPF_OBJ_GET_INFO_BY_FD, {info={bpf_fd=105, info_len=216, info=0x7ffda3254f50}}, 120) = 0
bpf(BPF_OBJ_GET_INFO_BY_FD, {info={bpf_fd=105, info_len=216, info=0x1b72d20}}, 120) = 0
bpf(BPF_BTF_GET_FD_BY_ID, {btf_id=313}, 120) = 139
bpf(BPF_OBJ_GET_INFO_BY_FD, {info={bpf_fd=139, info_len=32, info=0x7ffda3255000}}, 120) = 0
bpf(BPF_OBJ_GET_INFO_BY_FD, {info={bpf_fd=138, info_len=80, info=0x7ffda32550f0}}, 120) = 0
bpf(BPF_PROG_LOAD, {prog_type=BPF_PROG_TYPE_SOCKET_FILTER, insn_cnt=2, insns=0x7ffda3254ff0, license="GPL", log_level=0, log_size=0, log_buf=NULL, kern_version=KERNEL_VERSION(0, 0, 0), prog_flags=0, prog_name="", prog_ifindex=0, expected_attach_type=BPF_CGROUP_INET_INGRESS, prog_btf_fd=0, func_info_rec_size=0, func_info=NULL, func_info_cnt=0, line_info_rec_size=0, line_info=NULL, line_info_cnt=0, attach_btf_id=0, attach_prog_fd=0}, 120) = 140
bpf(BPF_BTF_LOAD, {btf="\237\353\1\0\30\0\0\0\0\0\0\0\204\2\0\0\204\2\0\0\250\3\0\0\0\0\0\0\0\0\0\2"..., btf_log_buf=NULL, btf_size=1604, btf_log_size=0, btf_log_level=0}, 120) = 140
bpf(BPF_MAP_CREATE, {map_type=BPF_MAP_TYPE_PERCPU_ARRAY, key_size=4, value_size=24, max_entries=1, map_flags=0, inner_map_fd=0, map_name="accum_readings", map_ifindex=0, btf_fd=140, btf_key_type_id=0, btf_value_type_id=0, btf_vmlinux_value_type_id=0}, 120) = 141
bpf(BPF_MAP_CREATE, {map_type=BPF_MAP_TYPE_HASH, key_size=4, value_size=4, max_entries=1, map_flags=0, inner_map_fd=0, map_name="filter", map_ifindex=0, btf_fd=140, btf_key_type_id=0, btf_value_type_id=0, btf_vmlinux_value_type_id=0}, 120) = 142
bpf(BPF_MAP_CREATE, {map_type=BPF_MAP_TYPE_ARRAY, key_size=4, value_size=8, max_entries=1, map_flags=BPF_F_MMAPABLE, inner_map_fd=0, map_name="bperf_fo.bss", map_ifindex=0, btf_fd=140, btf_key_type_id=0, btf_value_type_id=27, btf_vmlinux_value_type_id=0}, 120) = 168
bpf(BPF_MAP_UPDATE_ELEM, {map_fd=168, key=0x7ffda3254e50, value=0x7f0a61b6c000, flags=BPF_ANY}, 120) = 0
bpf(BPF_PROG_LOAD, {prog_type=BPF_PROG_TYPE_TRACING, insn_cnt=58, insns=0x1b35650, license="Dual BSD/GPL", log_level=0, log_size=0, log_buf=NULL, kern_version=KERNEL_VERSION(5, 12, 0), prog_flags=0, prog_name="fexit_XXX", prog_ifindex=0, expected_attach_type=BPF_TRACE_FEXIT, prog_btf_fd=140, func_info_rec_size=8, func_info=0x1b3ec60, func_info_cnt=1, line_info_rec_size=16, line_info=0x1b3dd00, line_info_cnt=28, attach_btf_id=22, attach_prog_fd=105}, 120) = 169
bpf(BPF_MAP_UPDATE_ELEM, {map_fd=142, key=0x7ffda32551b0, value=0x7ffda32551a4, flags=BPF_ANY}, 120) = 0
bpf(BPF_RAW_TRACEPOINT_OPEN, {raw_tracepoint={name=NULL, prog_fd=169}}, 120) = 170
bpf(BPF_OBJ_GET, {pathname="/sys/fs/bpf/perf_attr_map", bpf_fd=0, file_flags=0}, 120) = 3
bpf(BPF_MAP_LOOKUP_ELEM, {map_fd=3, key=0x1b7ee10, value=0x7ffda32551a8, flags=BPF_ANY}, 120) = 0
bpf(BPF_LINK_GET_FD_BY_ID, 0x7ffda32550e0, 120) = -1 ENOENT (No such file or directory)
libbpf: elf: skipping unrecognized data section(7) .eh_frame
libbpf: elf: skipping relo section(12) .rel.eh_frame for section(7) .eh_frame
bpf(BPF_PROG_LOAD, {prog_type=BPF_PROG_TYPE_SOCKET_FILTER, insn_cnt=2, insns=0x7ffda3254ff0, license="GPL", log_level=0, log_size=0, log_buf=NULL, kern_version=KERNEL_VERSION(0, 0, 0), prog_flags=0, prog_name="", prog_ifindex=0, expected_attach_type=BPF_CGROUP_INET_INGRESS, prog_btf_fd=0, func_info_rec_size=0, func_info=NULL, func_info_cnt=0, line_info_rec_size=0, line_info=NULL, line_info_cnt=0, attach_btf_id=0, attach_prog_fd=0}, 120) = 138
bpf(BPF_BTF_LOAD, {btf="\237\353\1\0\30\0\0\0\0\0\0\0D\2\0\0D\2\0\0\356\2\0\0\0\0\0\0\0\0\0\2"..., btf_log_buf=NULL, btf_size=1354, btf_log_size=0, btf_log_level=0}, 120) = 138
bpf(BPF_MAP_CREATE, {map_type=BPF_MAP_TYPE_PERF_EVENT_ARRAY, key_size=4, value_size=4, max_entries=24, map_flags=BPF_F_PRESERVE_ELEMS, inner_map_fd=0, map_name="events", map_ifindex=0, btf_fd=138, btf_key_type_id=0, btf_value_type_id=0, btf_vmlinux_value_type_id=0}, 120) = 171
bpf(BPF_MAP_CREATE, {map_type=BPF_MAP_TYPE_PERCPU_ARRAY, key_size=4, value_size=24, max_entries=1, map_flags=0, inner_map_fd=0, map_name="prev_readings", map_ifindex=0, btf_fd=138, btf_key_type_id=0, btf_value_type_id=0, btf_vmlinux_value_type_id=0}, 120) = 172
bpf(BPF_MAP_CREATE, {map_type=BPF_MAP_TYPE_PERCPU_ARRAY, key_size=4, value_size=24, max_entries=1, map_flags=0, inner_map_fd=0, map_name="diff_readings", map_ifindex=0, btf_fd=138, btf_key_type_id=0, btf_value_type_id=0, btf_vmlinux_value_type_id=0}, 120) = 173
bpf(BPF_PROG_LOAD, {prog_type=BPF_PROG_TYPE_RAW_TRACEPOINT, insn_cnt=48, insns=0x1b7fd00, license="Dual BSD/GPL", log_level=0, log_size=0, log_buf=NULL, kern_version=KERNEL_VERSION(5, 12, 0), prog_flags=0, prog_name="on_switch", prog_ifindex=0, expected_attach_type=BPF_CGROUP_INET_INGRESS, prog_btf_fd=138, func_info_rec_size=8, func_info=0x1b7a850, func_info_cnt=1, line_info_rec_size=16, line_info=0x1b761b0, line_info_cnt=25, attach_btf_id=0, attach_prog_fd=0}, 120) = 174
bpf(BPF_RAW_TRACEPOINT_OPEN, {raw_tracepoint={name="sched_switch", prog_fd=174}}, 120) = 175
bpf(BPF_OBJ_GET_INFO_BY_FD, {info={bpf_fd=175, info_len=32, info=0x7ffda32551b0}}, 120) = 0
bpf(BPF_OBJ_GET_INFO_BY_FD, {info={bpf_fd=173, info_len=80, info=0x7ffda3255200}}, 120) = 0
bpf(BPF_MAP_UPDATE_ELEM, {map_fd=3, key=0x1b7ee10, value=0x7ffda32551a8, flags=BPF_ANY}, 120) = 0
bpf(BPF_LINK_GET_FD_BY_ID, 0x7ffda32550e0, 120) = 176
bpf(BPF_MAP_UPDATE_ELEM, {map_fd=171, key=0x7ffda325505c, value=0x7ffda3255058, flags=BPF_ANY}, 120) = 0
bpf(BPF_MAP_UPDATE_ELEM, {map_fd=171, key=0x7ffda325505c, value=0x7ffda3255058, flags=BPF_ANY}, 120) = 0
bpf(BPF_MAP_UPDATE_ELEM, {map_fd=171, key=0x7ffda325505c, value=0x7ffda3255058, flags=BPF_ANY}, 120) = 0
bpf(BPF_MAP_UPDATE_ELEM, {map_fd=171, key=0x7ffda325505c, value=0x7ffda3255058, flags=BPF_ANY}, 120) = 0
bpf(BPF_MAP_UPDATE_ELEM, {map_fd=171, key=0x7ffda325505c, value=0x7ffda3255058, flags=BPF_ANY}, 120) = 0
bpf(BPF_MAP_UPDATE_ELEM, {map_fd=171, key=0x7ffda325505c, value=0x7ffda3255058, flags=BPF_ANY}, 120) = 0
bpf(BPF_MAP_UPDATE_ELEM, {map_fd=171, key=0x7ffda325505c, value=0x7ffda3255058, flags=BPF_ANY}, 120) = 0
bpf(BPF_MAP_UPDATE_ELEM, {map_fd=171, key=0x7ffda325505c, value=0x7ffda3255058, flags=BPF_ANY}, 120) = 0
bpf(BPF_MAP_UPDATE_ELEM, {map_fd=171, key=0x7ffda325505c, value=0x7ffda3255058, flags=BPF_ANY}, 120) = 0
bpf(BPF_MAP_UPDATE_ELEM, {map_fd=171, key=0x7ffda325505c, value=0x7ffda3255058, flags=BPF_ANY}, 120) = 0
bpf(BPF_MAP_UPDATE_ELEM, {map_fd=171, key=0x7ffda325505c, value=0x7ffda3255058, flags=BPF_ANY}, 120) = 0
bpf(BPF_MAP_UPDATE_ELEM, {map_fd=171, key=0x7ffda325505c, value=0x7ffda3255058, flags=BPF_ANY}, 120) = 0
bpf(BPF_MAP_UPDATE_ELEM, {map_fd=171, key=0x7ffda325505c, value=0x7ffda3255058, flags=BPF_ANY}, 120) = 0
bpf(BPF_MAP_UPDATE_ELEM, {map_fd=171, key=0x7ffda325505c, value=0x7ffda3255058, flags=BPF_ANY}, 120) = 0
bpf(BPF_MAP_UPDATE_ELEM, {map_fd=171, key=0x7ffda325505c, value=0x7ffda3255058, flags=BPF_ANY}, 120) = 0
bpf(BPF_MAP_UPDATE_ELEM, {map_fd=171, key=0x7ffda325505c, value=0x7ffda3255058, flags=BPF_ANY}, 120) = 0
bpf(BPF_MAP_UPDATE_ELEM, {map_fd=171, key=0x7ffda325505c, value=0x7ffda3255058, flags=BPF_ANY}, 120) = 0
bpf(BPF_MAP_UPDATE_ELEM, {map_fd=171, key=0x7ffda325505c, value=0x7ffda3255058, flags=BPF_ANY}, 120) = 0
bpf(BPF_MAP_UPDATE_ELEM, {map_fd=171, key=0x7ffda325505c, value=0x7ffda3255058, flags=BPF_ANY}, 120) = 0
bpf(BPF_MAP_UPDATE_ELEM, {map_fd=171, key=0x7ffda325505c, value=0x7ffda3255058, flags=BPF_ANY}, 120) = 0
bpf(BPF_MAP_UPDATE_ELEM, {map_fd=171, key=0x7ffda325505c, value=0x7ffda3255058, flags=BPF_ANY}, 120) = 0
bpf(BPF_MAP_UPDATE_ELEM, {map_fd=171, key=0x7ffda325505c, value=0x7ffda3255058, flags=BPF_ANY}, 120) = 0
bpf(BPF_MAP_UPDATE_ELEM, {map_fd=171, key=0x7ffda325505c, value=0x7ffda3255058, flags=BPF_ANY}, 120) = 0
bpf(BPF_MAP_UPDATE_ELEM, {map_fd=171, key=0x7ffda325505c, value=0x7ffda3255058, flags=BPF_ANY}, 120) = 0
bpf(BPF_OBJ_GET_INFO_BY_FD, {info={bpf_fd=176, info_len=32, info=0x7ffda32551b0}}, 120) = 0
bpf(BPF_PROG_GET_FD_BY_ID, {prog_id=213, next_id=0, open_flags=0}, 120) = 138
bpf(BPF_MAP_GET_FD_BY_ID, {map_id=227, next_id=0, open_flags=0}, 120) = 171
bpf(BPF_PROG_TEST_RUN, {test={prog_fd=138, retval=0, data_size_in=0, data_size_out=0, data_in=NULL, data_out=NULL, repeat=0, duration=0, ctx_size_in=0, ctx_size_out=0, ctx_in=NULL, ctx_out=NULL}, ...}, 120) = 0
libbpf: elf: skipping unrecognized data section(8) .eh_frame
libbpf: elf: skipping relo section(13) .rel.eh_frame for section(8) .eh_frame
bpf(BPF_OBJ_GET_INFO_BY_FD, {info={bpf_fd=138, info_len=216, info=0x7ffda3254f50}}, 120) = 0
bpf(BPF_OBJ_GET_INFO_BY_FD, {info={bpf_fd=138, info_len=216, info=0x1b72d20}}, 120) = 0
bpf(BPF_BTF_GET_FD_BY_ID, {btf_id=315}, 120) = 172
bpf(BPF_OBJ_GET_INFO_BY_FD, {info={bpf_fd=172, info_len=32, info=0x7ffda3255000}}, 120) = 0
bpf(BPF_OBJ_GET_INFO_BY_FD, {info={bpf_fd=171, info_len=80, info=0x7ffda32550f0}}, 120) = 0
bpf(BPF_PROG_LOAD, {prog_type=BPF_PROG_TYPE_SOCKET_FILTER, insn_cnt=2, insns=0x7ffda3254ff0, license="GPL", log_level=0, log_size=0, log_buf=NULL, kern_version=KERNEL_VERSION(0, 0, 0), prog_flags=0, prog_name="", prog_ifindex=0, expected_attach_type=BPF_CGROUP_INET_INGRESS, prog_btf_fd=0, func_info_rec_size=0, func_info=NULL, func_info_cnt=0, line_info_rec_size=0, line_info=NULL, line_info_cnt=0, attach_btf_id=0, attach_prog_fd=0}, 120) = 173
bpf(BPF_BTF_LOAD, {btf="\237\353\1\0\30\0\0\0\0\0\0\0\204\2\0\0\204\2\0\0\250\3\0\0\0\0\0\0\0\0\0\2"..., btf_log_buf=NULL, btf_size=1604, btf_log_size=0, btf_log_level=0}, 120) = 173
bpf(BPF_MAP_CREATE, {map_type=BPF_MAP_TYPE_PERCPU_ARRAY, key_size=4, value_size=24, max_entries=1, map_flags=0, inner_map_fd=0, map_name="accum_readings", map_ifindex=0, btf_fd=173, btf_key_type_id=0, btf_value_type_id=0, btf_vmlinux_value_type_id=0}, 120) = 174
bpf(BPF_MAP_CREATE, {map_type=BPF_MAP_TYPE_HASH, key_size=4, value_size=4, max_entries=1, map_flags=0, inner_map_fd=0, map_name="filter", map_ifindex=0, btf_fd=173, btf_key_type_id=0, btf_value_type_id=0, btf_vmlinux_value_type_id=0}, 120) = 175
bpf(BPF_MAP_CREATE, {map_type=BPF_MAP_TYPE_ARRAY, key_size=4, value_size=8, max_entries=1, map_flags=BPF_F_MMAPABLE, inner_map_fd=0, map_name="bperf_fo.bss", map_ifindex=0, btf_fd=173, btf_key_type_id=0, btf_value_type_id=27, btf_vmlinux_value_type_id=0}, 120) = 201
bpf(BPF_MAP_UPDATE_ELEM, {map_fd=201, key=0x7ffda3254e50, value=0x7f0a61b6b000, flags=BPF_ANY}, 120) = 0
bpf(BPF_PROG_LOAD, {prog_type=BPF_PROG_TYPE_TRACING, insn_cnt=58, insns=0x1b35650, license="Dual BSD/GPL", log_level=0, log_size=0, log_buf=NULL, kern_version=KERNEL_VERSION(5, 12, 0), prog_flags=0, prog_name="fexit_XXX", prog_ifindex=0, expected_attach_type=BPF_TRACE_FEXIT, prog_btf_fd=173, func_info_rec_size=8, func_info=0x1b3fc70, func_info_cnt=1, line_info_rec_size=16, line_info=0x1b3e4c0, line_info_cnt=28, attach_btf_id=22, attach_prog_fd=138}, 120) = 202
bpf(BPF_MAP_UPDATE_ELEM, {map_fd=175, key=0x7ffda32551b0, value=0x7ffda32551a4, flags=BPF_ANY}, 120) = 0
bpf(BPF_RAW_TRACEPOINT_OPEN, {raw_tracepoint={name=NULL, prog_fd=202}}, 120) = 203
bpf(BPF_OBJ_GET, {pathname="/sys/fs/bpf/perf_attr_map", bpf_fd=0, file_flags=0}, 120) = 3
bpf(BPF_MAP_LOOKUP_ELEM, {map_fd=3, key=0x1b7f0f0, value=0x7ffda32551a8, flags=BPF_ANY}, 120) = 0
bpf(BPF_LINK_GET_FD_BY_ID, 0x7ffda32550e0, 120) = -1 ENOENT (No such file or directory)
libbpf: elf: skipping unrecognized data section(7) .eh_frame
libbpf: elf: skipping relo section(12) .rel.eh_frame for section(7) .eh_frame
bpf(BPF_PROG_LOAD, {prog_type=BPF_PROG_TYPE_SOCKET_FILTER, insn_cnt=2, insns=0x7ffda3254ff0, license="GPL", log_level=0, log_size=0, log_buf=NULL, kern_version=KERNEL_VERSION(0, 0, 0), prog_flags=0, prog_name="", prog_ifindex=0, expected_attach_type=BPF_CGROUP_INET_INGRESS, prog_btf_fd=0, func_info_rec_size=0, func_info=NULL, func_info_cnt=0, line_info_rec_size=0, line_info=NULL, line_info_cnt=0, attach_btf_id=0, attach_prog_fd=0}, 120) = 171
bpf(BPF_BTF_LOAD, {btf="\237\353\1\0\30\0\0\0\0\0\0\0D\2\0\0D\2\0\0\356\2\0\0\0\0\0\0\0\0\0\2"..., btf_log_buf=NULL, btf_size=1354, btf_log_size=0, btf_log_level=0}, 120) = 171
bpf(BPF_MAP_CREATE, {map_type=BPF_MAP_TYPE_PERF_EVENT_ARRAY, key_size=4, value_size=4, max_entries=24, map_flags=BPF_F_PRESERVE_ELEMS, inner_map_fd=0, map_name="events", map_ifindex=0, btf_fd=171, btf_key_type_id=0, btf_value_type_id=0, btf_vmlinux_value_type_id=0}, 120) = 204
bpf(BPF_MAP_CREATE, {map_type=BPF_MAP_TYPE_PERCPU_ARRAY, key_size=4, value_size=24, max_entries=1, map_flags=0, inner_map_fd=0, map_name="prev_readings", map_ifindex=0, btf_fd=171, btf_key_type_id=0, btf_value_type_id=0, btf_vmlinux_value_type_id=0}, 120) = 205
bpf(BPF_MAP_CREATE, {map_type=BPF_MAP_TYPE_PERCPU_ARRAY, key_size=4, value_size=24, max_entries=1, map_flags=0, inner_map_fd=0, map_name="diff_readings", map_ifindex=0, btf_fd=171, btf_key_type_id=0, btf_value_type_id=0, btf_vmlinux_value_type_id=0}, 120) = 206
bpf(BPF_PROG_LOAD, {prog_type=BPF_PROG_TYPE_RAW_TRACEPOINT, insn_cnt=48, insns=0x1b7fd00, license="Dual BSD/GPL", log_level=0, log_size=0, log_buf=NULL, kern_version=KERNEL_VERSION(5, 12, 0), prog_flags=0, prog_name="on_switch", prog_ifindex=0, expected_attach_type=BPF_CGROUP_INET_INGRESS, prog_btf_fd=171, func_info_rec_size=8, func_info=0x1b3e990, func_info_cnt=1, line_info_rec_size=16, line_info=0x1b761b0, line_info_cnt=25, attach_btf_id=0, attach_prog_fd=0}, 120) = 207
bpf(BPF_RAW_TRACEPOINT_OPEN, {raw_tracepoint={name="sched_switch", prog_fd=207}}, 120) = 208
bpf(BPF_OBJ_GET_INFO_BY_FD, {info={bpf_fd=208, info_len=32, info=0x7ffda32551b0}}, 120) = 0
bpf(BPF_OBJ_GET_INFO_BY_FD, {info={bpf_fd=206, info_len=80, info=0x7ffda3255200}}, 120) = 0
bpf(BPF_MAP_UPDATE_ELEM, {map_fd=3, key=0x1b7f0f0, value=0x7ffda32551a8, flags=BPF_ANY}, 120) = 0
bpf(BPF_LINK_GET_FD_BY_ID, 0x7ffda32550e0, 120) = 209
bpf(BPF_MAP_UPDATE_ELEM, {map_fd=204, key=0x7ffda325505c, value=0x7ffda3255058, flags=BPF_ANY}, 120) = 0
bpf(BPF_MAP_UPDATE_ELEM, {map_fd=204, key=0x7ffda325505c, value=0x7ffda3255058, flags=BPF_ANY}, 120) = 0
bpf(BPF_MAP_UPDATE_ELEM, {map_fd=204, key=0x7ffda325505c, value=0x7ffda3255058, flags=BPF_ANY}, 120) = 0
bpf(BPF_MAP_UPDATE_ELEM, {map_fd=204, key=0x7ffda325505c, value=0x7ffda3255058, flags=BPF_ANY}, 120) = 0
bpf(BPF_MAP_UPDATE_ELEM, {map_fd=204, key=0x7ffda325505c, value=0x7ffda3255058, flags=BPF_ANY}, 120) = 0
bpf(BPF_MAP_UPDATE_ELEM, {map_fd=204, key=0x7ffda325505c, value=0x7ffda3255058, flags=BPF_ANY}, 120) = 0
bpf(BPF_MAP_UPDATE_ELEM, {map_fd=204, key=0x7ffda325505c, value=0x7ffda3255058, flags=BPF_ANY}, 120) = 0
bpf(BPF_MAP_UPDATE_ELEM, {map_fd=204, key=0x7ffda325505c, value=0x7ffda3255058, flags=BPF_ANY}, 120) = 0
bpf(BPF_MAP_UPDATE_ELEM, {map_fd=204, key=0x7ffda325505c, value=0x7ffda3255058, flags=BPF_ANY}, 120) = 0
bpf(BPF_MAP_UPDATE_ELEM, {map_fd=204, key=0x7ffda325505c, value=0x7ffda3255058, flags=BPF_ANY}, 120) = 0
bpf(BPF_MAP_UPDATE_ELEM, {map_fd=204, key=0x7ffda325505c, value=0x7ffda3255058, flags=BPF_ANY}, 120) = 0
bpf(BPF_MAP_UPDATE_ELEM, {map_fd=204, key=0x7ffda325505c, value=0x7ffda3255058, flags=BPF_ANY}, 120) = 0
bpf(BPF_MAP_UPDATE_ELEM, {map_fd=204, key=0x7ffda325505c, value=0x7ffda3255058, flags=BPF_ANY}, 120) = 0
bpf(BPF_MAP_UPDATE_ELEM, {map_fd=204, key=0x7ffda325505c, value=0x7ffda3255058, flags=BPF_ANY}, 120) = 0
bpf(BPF_MAP_UPDATE_ELEM, {map_fd=204, key=0x7ffda325505c, value=0x7ffda3255058, flags=BPF_ANY}, 120) = 0
bpf(BPF_MAP_UPDATE_ELEM, {map_fd=204, key=0x7ffda325505c, value=0x7ffda3255058, flags=BPF_ANY}, 120) = 0
bpf(BPF_MAP_UPDATE_ELEM, {map_fd=204, key=0x7ffda325505c, value=0x7ffda3255058, flags=BPF_ANY}, 120) = 0
bpf(BPF_MAP_UPDATE_ELEM, {map_fd=204, key=0x7ffda325505c, value=0x7ffda3255058, flags=BPF_ANY}, 120) = 0
bpf(BPF_MAP_UPDATE_ELEM, {map_fd=204, key=0x7ffda325505c, value=0x7ffda3255058, flags=BPF_ANY}, 120) = 0
bpf(BPF_MAP_UPDATE_ELEM, {map_fd=204, key=0x7ffda325505c, value=0x7ffda3255058, flags=BPF_ANY}, 120) = 0
bpf(BPF_MAP_UPDATE_ELEM, {map_fd=204, key=0x7ffda325505c, value=0x7ffda3255058, flags=BPF_ANY}, 120) = 0
bpf(BPF_MAP_UPDATE_ELEM, {map_fd=204, key=0x7ffda325505c, value=0x7ffda3255058, flags=BPF_ANY}, 120) = 0
bpf(BPF_MAP_UPDATE_ELEM, {map_fd=204, key=0x7ffda325505c, value=0x7ffda3255058, flags=BPF_ANY}, 120) = 0
bpf(BPF_MAP_UPDATE_ELEM, {map_fd=204, key=0x7ffda325505c, value=0x7ffda3255058, flags=BPF_ANY}, 120) = 0
bpf(BPF_OBJ_GET_INFO_BY_FD, {info={bpf_fd=209, info_len=32, info=0x7ffda32551b0}}, 120) = 0
bpf(BPF_PROG_GET_FD_BY_ID, {prog_id=217, next_id=0, open_flags=0}, 120) = 171
bpf(BPF_MAP_GET_FD_BY_ID, {map_id=233, next_id=0, open_flags=0}, 120) = 204
bpf(BPF_PROG_TEST_RUN, {test={prog_fd=171, retval=0, data_size_in=0, data_size_out=0, data_in=NULL, data_out=NULL, repeat=0, duration=0, ctx_size_in=0, ctx_size_out=0, ctx_in=NULL, ctx_out=NULL}, ...}, 120) = 0
libbpf: elf: skipping unrecognized data section(8) .eh_frame
libbpf: elf: skipping relo section(13) .rel.eh_frame for section(8) .eh_frame
bpf(BPF_OBJ_GET_INFO_BY_FD, {info={bpf_fd=171, info_len=216, info=0x7ffda3254f50}}, 120) = 0
bpf(BPF_OBJ_GET_INFO_BY_FD, {info={bpf_fd=171, info_len=216, info=0x1b72d20}}, 120) = 0
bpf(BPF_BTF_GET_FD_BY_ID, {btf_id=317}, 120) = 205
bpf(BPF_OBJ_GET_INFO_BY_FD, {info={bpf_fd=205, info_len=32, info=0x7ffda3255000}}, 120) = 0
bpf(BPF_OBJ_GET_INFO_BY_FD, {info={bpf_fd=204, info_len=80, info=0x7ffda32550f0}}, 120) = 0
bpf(BPF_PROG_LOAD, {prog_type=BPF_PROG_TYPE_SOCKET_FILTER, insn_cnt=2, insns=0x7ffda3254ff0, license="GPL", log_level=0, log_size=0, log_buf=NULL, kern_version=KERNEL_VERSION(0, 0, 0), prog_flags=0, prog_name="", prog_ifindex=0, expected_attach_type=BPF_CGROUP_INET_INGRESS, prog_btf_fd=0, func_info_rec_size=0, func_info=NULL, func_info_cnt=0, line_info_rec_size=0, line_info=NULL, line_info_cnt=0, attach_btf_id=0, attach_prog_fd=0}, 120) = 206
bpf(BPF_BTF_LOAD, {btf="\237\353\1\0\30\0\0\0\0\0\0\0\204\2\0\0\204\2\0\0\250\3\0\0\0\0\0\0\0\0\0\2"..., btf_log_buf=NULL, btf_size=1604, btf_log_size=0, btf_log_level=0}, 120) = 206
bpf(BPF_MAP_CREATE, {map_type=BPF_MAP_TYPE_PERCPU_ARRAY, key_size=4, value_size=24, max_entries=1, map_flags=0, inner_map_fd=0, map_name="accum_readings", map_ifindex=0, btf_fd=206, btf_key_type_id=0, btf_value_type_id=0, btf_vmlinux_value_type_id=0}, 120) = 207
bpf(BPF_MAP_CREATE, {map_type=BPF_MAP_TYPE_HASH, key_size=4, value_size=4, max_entries=1, map_flags=0, inner_map_fd=0, map_name="filter", map_ifindex=0, btf_fd=206, btf_key_type_id=0, btf_value_type_id=0, btf_vmlinux_value_type_id=0}, 120) = 208
bpf(BPF_MAP_CREATE, {map_type=BPF_MAP_TYPE_ARRAY, key_size=4, value_size=8, max_entries=1, map_flags=BPF_F_MMAPABLE, inner_map_fd=0, map_name="bperf_fo.bss", map_ifindex=0, btf_fd=206, btf_key_type_id=0, btf_value_type_id=27, btf_vmlinux_value_type_id=0}, 120) = 234
bpf(BPF_MAP_UPDATE_ELEM, {map_fd=234, key=0x7ffda3254e50, value=0x7f0a61b6a000, flags=BPF_ANY}, 120) = 0
bpf(BPF_PROG_LOAD, {prog_type=BPF_PROG_TYPE_TRACING, insn_cnt=58, insns=0x1b35650, license="Dual BSD/GPL", log_level=0, log_size=0, log_buf=NULL, kern_version=KERNEL_VERSION(5, 12, 0), prog_flags=0, prog_name="fexit_XXX", prog_ifindex=0, expected_attach_type=BPF_TRACE_FEXIT, prog_btf_fd=206, func_info_rec_size=8, func_info=0x1b3f8c0, func_info_cnt=1, line_info_rec_size=16, line_info=0x1b3fd40, line_info_cnt=28, attach_btf_id=22, attach_prog_fd=171}, 120) = 235
bpf(BPF_MAP_UPDATE_ELEM, {map_fd=208, key=0x7ffda32551b0, value=0x7ffda32551a4, flags=BPF_ANY}, 120) = 0
bpf(BPF_RAW_TRACEPOINT_OPEN, {raw_tracepoint={name=NULL, prog_fd=235}}, 120) = 236
bpf(BPF_OBJ_GET, {pathname="/sys/fs/bpf/perf_attr_map", bpf_fd=0, file_flags=0}, 120) = 3
bpf(BPF_MAP_LOOKUP_ELEM, {map_fd=3, key=0x1b7f3b0, value=0x7ffda32551a8, flags=BPF_ANY}, 120) = 0
bpf(BPF_LINK_GET_FD_BY_ID, 0x7ffda32550e0, 120) = -1 ENOENT (No such file or directory)
libbpf: elf: skipping unrecognized data section(7) .eh_frame
libbpf: elf: skipping relo section(12) .rel.eh_frame for section(7) .eh_frame
bpf(BPF_PROG_LOAD, {prog_type=BPF_PROG_TYPE_SOCKET_FILTER, insn_cnt=2, insns=0x7ffda3254ff0, license="GPL", log_level=0, log_size=0, log_buf=NULL, kern_version=KERNEL_VERSION(0, 0, 0), prog_flags=0, prog_name="", prog_ifindex=0, expected_attach_type=BPF_CGROUP_INET_INGRESS, prog_btf_fd=0, func_info_rec_size=0, func_info=NULL, func_info_cnt=0, line_info_rec_size=0, line_info=NULL, line_info_cnt=0, attach_btf_id=0, attach_prog_fd=0}, 120) = 204
bpf(BPF_BTF_LOAD, {btf="\237\353\1\0\30\0\0\0\0\0\0\0D\2\0\0D\2\0\0\356\2\0\0\0\0\0\0\0\0\0\2"..., btf_log_buf=NULL, btf_size=1354, btf_log_size=0, btf_log_level=0}, 120) = 204
bpf(BPF_MAP_CREATE, {map_type=BPF_MAP_TYPE_PERF_EVENT_ARRAY, key_size=4, value_size=4, max_entries=24, map_flags=BPF_F_PRESERVE_ELEMS, inner_map_fd=0, map_name="events", map_ifindex=0, btf_fd=204, btf_key_type_id=0, btf_value_type_id=0, btf_vmlinux_value_type_id=0}, 120) = 237
bpf(BPF_MAP_CREATE, {map_type=BPF_MAP_TYPE_PERCPU_ARRAY, key_size=4, value_size=24, max_entries=1, map_flags=0, inner_map_fd=0, map_name="prev_readings", map_ifindex=0, btf_fd=204, btf_key_type_id=0, btf_value_type_id=0, btf_vmlinux_value_type_id=0}, 120) = 238
bpf(BPF_MAP_CREATE, {map_type=BPF_MAP_TYPE_PERCPU_ARRAY, key_size=4, value_size=24, max_entries=1, map_flags=0, inner_map_fd=0, map_name="diff_readings", map_ifindex=0, btf_fd=204, btf_key_type_id=0, btf_value_type_id=0, btf_vmlinux_value_type_id=0}, 120) = 239
bpf(BPF_PROG_LOAD, {prog_type=BPF_PROG_TYPE_RAW_TRACEPOINT, insn_cnt=48, insns=0x1b7fd00, license="Dual BSD/GPL", log_level=0, log_size=0, log_buf=NULL, kern_version=KERNEL_VERSION(5, 12, 0), prog_flags=0, prog_name="on_switch", prog_ifindex=0, expected_attach_type=BPF_CGROUP_INET_INGRESS, prog_btf_fd=204, func_info_rec_size=8, func_info=0x1b40a20, func_info_cnt=1, line_info_rec_size=16, line_info=0x1b761b0, line_info_cnt=25, attach_btf_id=0, attach_prog_fd=0}, 120) = 240
bpf(BPF_RAW_TRACEPOINT_OPEN, {raw_tracepoint={name="sched_switch", prog_fd=240}}, 120) = 241
bpf(BPF_OBJ_GET_INFO_BY_FD, {info={bpf_fd=241, info_len=32, info=0x7ffda32551b0}}, 120) = 0
bpf(BPF_OBJ_GET_INFO_BY_FD, {info={bpf_fd=239, info_len=80, info=0x7ffda3255200}}, 120) = 0
bpf(BPF_MAP_UPDATE_ELEM, {map_fd=3, key=0x1b7f3b0, value=0x7ffda32551a8, flags=BPF_ANY}, 120) = 0
bpf(BPF_LINK_GET_FD_BY_ID, 0x7ffda32550e0, 120) = 242
bpf(BPF_MAP_UPDATE_ELEM, {map_fd=237, key=0x7ffda325505c, value=0x7ffda3255058, flags=BPF_ANY}, 120) = 0
bpf(BPF_MAP_UPDATE_ELEM, {map_fd=237, key=0x7ffda325505c, value=0x7ffda3255058, flags=BPF_ANY}, 120) = 0
bpf(BPF_MAP_UPDATE_ELEM, {map_fd=237, key=0x7ffda325505c, value=0x7ffda3255058, flags=BPF_ANY}, 120) = 0
bpf(BPF_MAP_UPDATE_ELEM, {map_fd=237, key=0x7ffda325505c, value=0x7ffda3255058, flags=BPF_ANY}, 120) = 0
bpf(BPF_MAP_UPDATE_ELEM, {map_fd=237, key=0x7ffda325505c, value=0x7ffda3255058, flags=BPF_ANY}, 120) = 0
bpf(BPF_MAP_UPDATE_ELEM, {map_fd=237, key=0x7ffda325505c, value=0x7ffda3255058, flags=BPF_ANY}, 120) = 0
bpf(BPF_MAP_UPDATE_ELEM, {map_fd=237, key=0x7ffda325505c, value=0x7ffda3255058, flags=BPF_ANY}, 120) = 0
bpf(BPF_MAP_UPDATE_ELEM, {map_fd=237, key=0x7ffda325505c, value=0x7ffda3255058, flags=BPF_ANY}, 120) = 0
bpf(BPF_MAP_UPDATE_ELEM, {map_fd=237, key=0x7ffda325505c, value=0x7ffda3255058, flags=BPF_ANY}, 120) = 0
bpf(BPF_MAP_UPDATE_ELEM, {map_fd=237, key=0x7ffda325505c, value=0x7ffda3255058, flags=BPF_ANY}, 120) = 0
bpf(BPF_MAP_UPDATE_ELEM, {map_fd=237, key=0x7ffda325505c, value=0x7ffda3255058, flags=BPF_ANY}, 120) = 0
bpf(BPF_MAP_UPDATE_ELEM, {map_fd=237, key=0x7ffda325505c, value=0x7ffda3255058, flags=BPF_ANY}, 120) = 0
bpf(BPF_MAP_UPDATE_ELEM, {map_fd=237, key=0x7ffda325505c, value=0x7ffda3255058, flags=BPF_ANY}, 120) = 0
bpf(BPF_MAP_UPDATE_ELEM, {map_fd=237, key=0x7ffda325505c, value=0x7ffda3255058, flags=BPF_ANY}, 120) = 0
bpf(BPF_MAP_UPDATE_ELEM, {map_fd=237, key=0x7ffda325505c, value=0x7ffda3255058, flags=BPF_ANY}, 120) = 0
bpf(BPF_MAP_UPDATE_ELEM, {map_fd=237, key=0x7ffda325505c, value=0x7ffda3255058, flags=BPF_ANY}, 120) = 0
bpf(BPF_MAP_UPDATE_ELEM, {map_fd=237, key=0x7ffda325505c, value=0x7ffda3255058, flags=BPF_ANY}, 120) = 0
bpf(BPF_MAP_UPDATE_ELEM, {map_fd=237, key=0x7ffda325505c, value=0x7ffda3255058, flags=BPF_ANY}, 120) = 0
bpf(BPF_MAP_UPDATE_ELEM, {map_fd=237, key=0x7ffda325505c, value=0x7ffda3255058, flags=BPF_ANY}, 120) = 0
bpf(BPF_MAP_UPDATE_ELEM, {map_fd=237, key=0x7ffda325505c, value=0x7ffda3255058, flags=BPF_ANY}, 120) = 0
bpf(BPF_MAP_UPDATE_ELEM, {map_fd=237, key=0x7ffda325505c, value=0x7ffda3255058, flags=BPF_ANY}, 120) = 0
bpf(BPF_MAP_UPDATE_ELEM, {map_fd=237, key=0x7ffda325505c, value=0x7ffda3255058, flags=BPF_ANY}, 120) = 0
bpf(BPF_MAP_UPDATE_ELEM, {map_fd=237, key=0x7ffda325505c, value=0x7ffda3255058, flags=BPF_ANY}, 120) = 0
bpf(BPF_MAP_UPDATE_ELEM, {map_fd=237, key=0x7ffda325505c, value=0x7ffda3255058, flags=BPF_ANY}, 120) = 0
bpf(BPF_OBJ_GET_INFO_BY_FD, {info={bpf_fd=242, info_len=32, info=0x7ffda32551b0}}, 120) = 0
bpf(BPF_PROG_GET_FD_BY_ID, {prog_id=221, next_id=0, open_flags=0}, 120) = 204
bpf(BPF_MAP_GET_FD_BY_ID, {map_id=239, next_id=0, open_flags=0}, 120) = 237
bpf(BPF_PROG_TEST_RUN, {test={prog_fd=204, retval=0, data_size_in=0, data_size_out=0, data_in=NULL, data_out=NULL, repeat=0, duration=0, ctx_size_in=0, ctx_size_out=0, ctx_in=NULL, ctx_out=NULL}, ...}, 120) = 0
libbpf: elf: skipping unrecognized data section(8) .eh_frame
libbpf: elf: skipping relo section(13) .rel.eh_frame for section(8) .eh_frame
bpf(BPF_OBJ_GET_INFO_BY_FD, {info={bpf_fd=204, info_len=216, info=0x7ffda3254f50}}, 120) = 0
bpf(BPF_OBJ_GET_INFO_BY_FD, {info={bpf_fd=204, info_len=216, info=0x1b72d20}}, 120) = 0
bpf(BPF_BTF_GET_FD_BY_ID, {btf_id=319}, 120) = 238
bpf(BPF_OBJ_GET_INFO_BY_FD, {info={bpf_fd=238, info_len=32, info=0x7ffda3255000}}, 120) = 0
bpf(BPF_OBJ_GET_INFO_BY_FD, {info={bpf_fd=237, info_len=80, info=0x7ffda32550f0}}, 120) = 0
bpf(BPF_PROG_LOAD, {prog_type=BPF_PROG_TYPE_SOCKET_FILTER, insn_cnt=2, insns=0x7ffda3254ff0, license="GPL", log_level=0, log_size=0, log_buf=NULL, kern_version=KERNEL_VERSION(0, 0, 0), prog_flags=0, prog_name="", prog_ifindex=0, expected_attach_type=BPF_CGROUP_INET_INGRESS, prog_btf_fd=0, func_info_rec_size=0, func_info=NULL, func_info_cnt=0, line_info_rec_size=0, line_info=NULL, line_info_cnt=0, attach_btf_id=0, attach_prog_fd=0}, 120) = 239
bpf(BPF_BTF_LOAD, {btf="\237\353\1\0\30\0\0\0\0\0\0\0\204\2\0\0\204\2\0\0\250\3\0\0\0\0\0\0\0\0\0\2"..., btf_log_buf=NULL, btf_size=1604, btf_log_size=0, btf_log_level=0}, 120) = 239
bpf(BPF_MAP_CREATE, {map_type=BPF_MAP_TYPE_PERCPU_ARRAY, key_size=4, value_size=24, max_entries=1, map_flags=0, inner_map_fd=0, map_name="accum_readings", map_ifindex=0, btf_fd=239, btf_key_type_id=0, btf_value_type_id=0, btf_vmlinux_value_type_id=0}, 120) = 240
bpf(BPF_MAP_CREATE, {map_type=BPF_MAP_TYPE_HASH, key_size=4, value_size=4, max_entries=1, map_flags=0, inner_map_fd=0, map_name="filter", map_ifindex=0, btf_fd=239, btf_key_type_id=0, btf_value_type_id=0, btf_vmlinux_value_type_id=0}, 120) = 241
bpf(BPF_MAP_CREATE, {map_type=BPF_MAP_TYPE_ARRAY, key_size=4, value_size=8, max_entries=1, map_flags=BPF_F_MMAPABLE, inner_map_fd=0, map_name="bperf_fo.bss", map_ifindex=0, btf_fd=239, btf_key_type_id=0, btf_value_type_id=27, btf_vmlinux_value_type_id=0}, 120) = 267
bpf(BPF_MAP_UPDATE_ELEM, {map_fd=267, key=0x7ffda3254e50, value=0x7f0a61b69000, flags=BPF_ANY}, 120) = 0
bpf(BPF_PROG_LOAD, {prog_type=BPF_PROG_TYPE_TRACING, insn_cnt=58, insns=0x1b35650, license="Dual BSD/GPL", log_level=0, log_size=0, log_buf=NULL, kern_version=KERNEL_VERSION(5, 12, 0), prog_flags=0, prog_name="fexit_XXX", prog_ifindex=0, expected_attach_type=BPF_TRACE_FEXIT, prog_btf_fd=239, func_info_rec_size=8, func_info=0x1b421c0, func_info_cnt=1, line_info_rec_size=16, line_info=0x1b41550, line_info_cnt=28, attach_btf_id=22, attach_prog_fd=204}, 120) = 268
bpf(BPF_MAP_UPDATE_ELEM, {map_fd=241, key=0x7ffda32551b0, value=0x7ffda32551a4, flags=BPF_ANY}, 120) = 0
bpf(BPF_RAW_TRACEPOINT_OPEN, {raw_tracepoint={name=NULL, prog_fd=268}}, 120) = 269
bpf(BPF_OBJ_GET, {pathname="/sys/fs/bpf/perf_attr_map", bpf_fd=0, file_flags=0}, 120) = 3
bpf(BPF_MAP_LOOKUP_ELEM, {map_fd=3, key=0x1b7e8c0, value=0x7ffda32551a8, flags=BPF_ANY}, 120) = 0
bpf(BPF_LINK_GET_FD_BY_ID, 0x7ffda32550e0, 120) = -1 ENOENT (No such file or directory)
libbpf: elf: skipping unrecognized data section(7) .eh_frame
libbpf: elf: skipping relo section(12) .rel.eh_frame for section(7) .eh_frame
bpf(BPF_PROG_LOAD, {prog_type=BPF_PROG_TYPE_SOCKET_FILTER, insn_cnt=2, insns=0x7ffda3254ff0, license="GPL", log_level=0, log_size=0, log_buf=NULL, kern_version=KERNEL_VERSION(0, 0, 0), prog_flags=0, prog_name="", prog_ifindex=0, expected_attach_type=BPF_CGROUP_INET_INGRESS, prog_btf_fd=0, func_info_rec_size=0, func_info=NULL, func_info_cnt=0, line_info_rec_size=0, line_info=NULL, line_info_cnt=0, attach_btf_id=0, attach_prog_fd=0}, 120) = 237
bpf(BPF_BTF_LOAD, {btf="\237\353\1\0\30\0\0\0\0\0\0\0D\2\0\0D\2\0\0\356\2\0\0\0\0\0\0\0\0\0\2"..., btf_log_buf=NULL, btf_size=1354, btf_log_size=0, btf_log_level=0}, 120) = 237
bpf(BPF_MAP_CREATE, {map_type=BPF_MAP_TYPE_PERF_EVENT_ARRAY, key_size=4, value_size=4, max_entries=24, map_flags=BPF_F_PRESERVE_ELEMS, inner_map_fd=0, map_name="events", map_ifindex=0, btf_fd=237, btf_key_type_id=0, btf_value_type_id=0, btf_vmlinux_value_type_id=0}, 120) = 270
bpf(BPF_MAP_CREATE, {map_type=BPF_MAP_TYPE_PERCPU_ARRAY, key_size=4, value_size=24, max_entries=1, map_flags=0, inner_map_fd=0, map_name="prev_readings", map_ifindex=0, btf_fd=237, btf_key_type_id=0, btf_value_type_id=0, btf_vmlinux_value_type_id=0}, 120) = 271
bpf(BPF_MAP_CREATE, {map_type=BPF_MAP_TYPE_PERCPU_ARRAY, key_size=4, value_size=24, max_entries=1, map_flags=0, inner_map_fd=0, map_name="diff_readings", map_ifindex=0, btf_fd=237, btf_key_type_id=0, btf_value_type_id=0, btf_vmlinux_value_type_id=0}, 120) = 272
bpf(BPF_PROG_LOAD, {prog_type=BPF_PROG_TYPE_RAW_TRACEPOINT, insn_cnt=48, insns=0x1b7fd00, license="Dual BSD/GPL", log_level=0, log_size=0, log_buf=NULL, kern_version=KERNEL_VERSION(5, 12, 0), prog_flags=0, prog_name="on_switch", prog_ifindex=0, expected_attach_type=BPF_CGROUP_INET_INGRESS, prog_btf_fd=237, func_info_rec_size=8, func_info=0x1b42200, func_info_cnt=1, line_info_rec_size=16, line_info=0x1b761b0, line_info_cnt=25, attach_btf_id=0, attach_prog_fd=0}, 120) = 273
bpf(BPF_RAW_TRACEPOINT_OPEN, {raw_tracepoint={name="sched_switch", prog_fd=273}}, 120) = 274
bpf(BPF_OBJ_GET_INFO_BY_FD, {info={bpf_fd=274, info_len=32, info=0x7ffda32551b0}}, 120) = 0
bpf(BPF_OBJ_GET_INFO_BY_FD, {info={bpf_fd=272, info_len=80, info=0x7ffda3255200}}, 120) = 0
bpf(BPF_MAP_UPDATE_ELEM, {map_fd=3, key=0x1b7e8c0, value=0x7ffda32551a8, flags=BPF_ANY}, 120) = 0
bpf(BPF_LINK_GET_FD_BY_ID, 0x7ffda32550e0, 120) = 275
bpf(BPF_MAP_UPDATE_ELEM, {map_fd=270, key=0x7ffda325505c, value=0x7ffda3255058, flags=BPF_ANY}, 120) = 0
bpf(BPF_MAP_UPDATE_ELEM, {map_fd=270, key=0x7ffda325505c, value=0x7ffda3255058, flags=BPF_ANY}, 120) = 0
bpf(BPF_MAP_UPDATE_ELEM, {map_fd=270, key=0x7ffda325505c, value=0x7ffda3255058, flags=BPF_ANY}, 120) = 0
bpf(BPF_MAP_UPDATE_ELEM, {map_fd=270, key=0x7ffda325505c, value=0x7ffda3255058, flags=BPF_ANY}, 120) = 0
bpf(BPF_MAP_UPDATE_ELEM, {map_fd=270, key=0x7ffda325505c, value=0x7ffda3255058, flags=BPF_ANY}, 120) = 0
bpf(BPF_MAP_UPDATE_ELEM, {map_fd=270, key=0x7ffda325505c, value=0x7ffda3255058, flags=BPF_ANY}, 120) = 0
bpf(BPF_MAP_UPDATE_ELEM, {map_fd=270, key=0x7ffda325505c, value=0x7ffda3255058, flags=BPF_ANY}, 120) = 0
bpf(BPF_MAP_UPDATE_ELEM, {map_fd=270, key=0x7ffda325505c, value=0x7ffda3255058, flags=BPF_ANY}, 120) = 0
bpf(BPF_MAP_UPDATE_ELEM, {map_fd=270, key=0x7ffda325505c, value=0x7ffda3255058, flags=BPF_ANY}, 120) = 0
bpf(BPF_MAP_UPDATE_ELEM, {map_fd=270, key=0x7ffda325505c, value=0x7ffda3255058, flags=BPF_ANY}, 120) = 0
bpf(BPF_MAP_UPDATE_ELEM, {map_fd=270, key=0x7ffda325505c, value=0x7ffda3255058, flags=BPF_ANY}, 120) = 0
bpf(BPF_MAP_UPDATE_ELEM, {map_fd=270, key=0x7ffda325505c, value=0x7ffda3255058, flags=BPF_ANY}, 120) = 0
bpf(BPF_MAP_UPDATE_ELEM, {map_fd=270, key=0x7ffda325505c, value=0x7ffda3255058, flags=BPF_ANY}, 120) = 0
bpf(BPF_MAP_UPDATE_ELEM, {map_fd=270, key=0x7ffda325505c, value=0x7ffda3255058, flags=BPF_ANY}, 120) = 0
bpf(BPF_MAP_UPDATE_ELEM, {map_fd=270, key=0x7ffda325505c, value=0x7ffda3255058, flags=BPF_ANY}, 120) = 0
bpf(BPF_MAP_UPDATE_ELEM, {map_fd=270, key=0x7ffda325505c, value=0x7ffda3255058, flags=BPF_ANY}, 120) = 0
bpf(BPF_MAP_UPDATE_ELEM, {map_fd=270, key=0x7ffda325505c, value=0x7ffda3255058, flags=BPF_ANY}, 120) = 0
bpf(BPF_MAP_UPDATE_ELEM, {map_fd=270, key=0x7ffda325505c, value=0x7ffda3255058, flags=BPF_ANY}, 120) = 0
bpf(BPF_MAP_UPDATE_ELEM, {map_fd=270, key=0x7ffda325505c, value=0x7ffda3255058, flags=BPF_ANY}, 120) = 0
bpf(BPF_MAP_UPDATE_ELEM, {map_fd=270, key=0x7ffda325505c, value=0x7ffda3255058, flags=BPF_ANY}, 120) = 0
bpf(BPF_MAP_UPDATE_ELEM, {map_fd=270, key=0x7ffda325505c, value=0x7ffda3255058, flags=BPF_ANY}, 120) = 0
bpf(BPF_MAP_UPDATE_ELEM, {map_fd=270, key=0x7ffda325505c, value=0x7ffda3255058, flags=BPF_ANY}, 120) = 0
bpf(BPF_MAP_UPDATE_ELEM, {map_fd=270, key=0x7ffda325505c, value=0x7ffda3255058, flags=BPF_ANY}, 120) = 0
bpf(BPF_MAP_UPDATE_ELEM, {map_fd=270, key=0x7ffda325505c, value=0x7ffda3255058, flags=BPF_ANY}, 120) = 0
bpf(BPF_OBJ_GET_INFO_BY_FD, {info={bpf_fd=275, info_len=32, info=0x7ffda32551b0}}, 120) = 0
bpf(BPF_PROG_GET_FD_BY_ID, {prog_id=225, next_id=0, open_flags=0}, 120) = 237
bpf(BPF_MAP_GET_FD_BY_ID, {map_id=245, next_id=0, open_flags=0}, 120) = 270
bpf(BPF_PROG_TEST_RUN, {test={prog_fd=237, retval=0, data_size_in=0, data_size_out=0, data_in=NULL, data_out=NULL, repeat=0, duration=0, ctx_size_in=0, ctx_size_out=0, ctx_in=NULL, ctx_out=NULL}, ...}, 120) = 0
libbpf: elf: skipping unrecognized data section(8) .eh_frame
libbpf: elf: skipping relo section(13) .rel.eh_frame for section(8) .eh_frame
bpf(BPF_OBJ_GET_INFO_BY_FD, {info={bpf_fd=237, info_len=216, info=0x7ffda3254f50}}, 120) = 0
bpf(BPF_OBJ_GET_INFO_BY_FD, {info={bpf_fd=237, info_len=216, info=0x1b72d20}}, 120) = 0
bpf(BPF_BTF_GET_FD_BY_ID, {btf_id=321}, 120) = 271
bpf(BPF_OBJ_GET_INFO_BY_FD, {info={bpf_fd=271, info_len=32, info=0x7ffda3255000}}, 120) = 0
bpf(BPF_OBJ_GET_INFO_BY_FD, {info={bpf_fd=270, info_len=80, info=0x7ffda32550f0}}, 120) = 0
bpf(BPF_PROG_LOAD, {prog_type=BPF_PROG_TYPE_SOCKET_FILTER, insn_cnt=2, insns=0x7ffda3254ff0, license="GPL", log_level=0, log_size=0, log_buf=NULL, kern_version=KERNEL_VERSION(0, 0, 0), prog_flags=0, prog_name="", prog_ifindex=0, expected_attach_type=BPF_CGROUP_INET_INGRESS, prog_btf_fd=0, func_info_rec_size=0, func_info=NULL, func_info_cnt=0, line_info_rec_size=0, line_info=NULL, line_info_cnt=0, attach_btf_id=0, attach_prog_fd=0}, 120) = 272
bpf(BPF_BTF_LOAD, {btf="\237\353\1\0\30\0\0\0\0\0\0\0\204\2\0\0\204\2\0\0\250\3\0\0\0\0\0\0\0\0\0\2"..., btf_log_buf=NULL, btf_size=1604, btf_log_size=0, btf_log_level=0}, 120) = 272
bpf(BPF_MAP_CREATE, {map_type=BPF_MAP_TYPE_PERCPU_ARRAY, key_size=4, value_size=24, max_entries=1, map_flags=0, inner_map_fd=0, map_name="accum_readings", map_ifindex=0, btf_fd=272, btf_key_type_id=0, btf_value_type_id=0, btf_vmlinux_value_type_id=0}, 120) = 273
bpf(BPF_MAP_CREATE, {map_type=BPF_MAP_TYPE_HASH, key_size=4, value_size=4, max_entries=1, map_flags=0, inner_map_fd=0, map_name="filter", map_ifindex=0, btf_fd=272, btf_key_type_id=0, btf_value_type_id=0, btf_vmlinux_value_type_id=0}, 120) = 274
bpf(BPF_MAP_CREATE, {map_type=BPF_MAP_TYPE_ARRAY, key_size=4, value_size=8, max_entries=1, map_flags=BPF_F_MMAPABLE, inner_map_fd=0, map_name="bperf_fo.bss", map_ifindex=0, btf_fd=272, btf_key_type_id=0, btf_value_type_id=27, btf_vmlinux_value_type_id=0}, 120) = 300
bpf(BPF_MAP_UPDATE_ELEM, {map_fd=300, key=0x7ffda3254e50, value=0x7f0a61b68000, flags=BPF_ANY}, 120) = 0
bpf(BPF_PROG_LOAD, {prog_type=BPF_PROG_TYPE_TRACING, insn_cnt=58, insns=0x1b35650, license="Dual BSD/GPL", log_level=0, log_size=0, log_buf=NULL, kern_version=KERNEL_VERSION(5, 12, 0), prog_flags=0, prog_name="fexit_XXX", prog_ifindex=0, expected_attach_type=BPF_TRACE_FEXIT, prog_btf_fd=272, func_info_rec_size=8, func_info=0x1b43810, func_info_cnt=1, line_info_rec_size=16, line_info=0x1b42300, line_info_cnt=28, attach_btf_id=22, attach_prog_fd=237}, 120) = 301
bpf(BPF_MAP_UPDATE_ELEM, {map_fd=274, key=0x7ffda32551b0, value=0x7ffda32551a4, flags=BPF_ANY}, 120) = 0
bpf(BPF_RAW_TRACEPOINT_OPEN, {raw_tracepoint={name=NULL, prog_fd=301}}, 120) = 302
bpf(BPF_OBJ_GET, {pathname="/sys/fs/bpf/perf_attr_map", bpf_fd=0, file_flags=0}, 120) = 3
bpf(BPF_MAP_LOOKUP_ELEM, {map_fd=3, key=0x1b76d00, value=0x7ffda32551a8, flags=BPF_ANY}, 120) = 0
bpf(BPF_LINK_GET_FD_BY_ID, 0x7ffda32550e0, 120) = -1 ENOENT (No such file or directory)
libbpf: elf: skipping unrecognized data section(7) .eh_frame
libbpf: elf: skipping relo section(12) .rel.eh_frame for section(7) .eh_frame
bpf(BPF_PROG_LOAD, {prog_type=BPF_PROG_TYPE_SOCKET_FILTER, insn_cnt=2, insns=0x7ffda3254ff0, license="GPL", log_level=0, log_size=0, log_buf=NULL, kern_version=KERNEL_VERSION(0, 0, 0), prog_flags=0, prog_name="", prog_ifindex=0, expected_attach_type=BPF_CGROUP_INET_INGRESS, prog_btf_fd=0, func_info_rec_size=0, func_info=NULL, func_info_cnt=0, line_info_rec_size=0, line_info=NULL, line_info_cnt=0, attach_btf_id=0, attach_prog_fd=0}, 120) = 270
bpf(BPF_BTF_LOAD, {btf="\237\353\1\0\30\0\0\0\0\0\0\0D\2\0\0D\2\0\0\356\2\0\0\0\0\0\0\0\0\0\2"..., btf_log_buf=NULL, btf_size=1354, btf_log_size=0, btf_log_level=0}, 120) = 270
bpf(BPF_MAP_CREATE, {map_type=BPF_MAP_TYPE_PERF_EVENT_ARRAY, key_size=4, value_size=4, max_entries=24, map_flags=BPF_F_PRESERVE_ELEMS, inner_map_fd=0, map_name="events", map_ifindex=0, btf_fd=270, btf_key_type_id=0, btf_value_type_id=0, btf_vmlinux_value_type_id=0}, 120) = 303
bpf(BPF_MAP_CREATE, {map_type=BPF_MAP_TYPE_PERCPU_ARRAY, key_size=4, value_size=24, max_entries=1, map_flags=0, inner_map_fd=0, map_name="prev_readings", map_ifindex=0, btf_fd=270, btf_key_type_id=0, btf_value_type_id=0, btf_vmlinux_value_type_id=0}, 120) = 304
bpf(BPF_MAP_CREATE, {map_type=BPF_MAP_TYPE_PERCPU_ARRAY, key_size=4, value_size=24, max_entries=1, map_flags=0, inner_map_fd=0, map_name="diff_readings", map_ifindex=0, btf_fd=270, btf_key_type_id=0, btf_value_type_id=0, btf_vmlinux_value_type_id=0}, 120) = 305
bpf(BPF_PROG_LOAD, {prog_type=BPF_PROG_TYPE_RAW_TRACEPOINT, insn_cnt=48, insns=0x1b7fd00, license="Dual BSD/GPL", log_level=0, log_size=0, log_buf=NULL, kern_version=KERNEL_VERSION(5, 12, 0), prog_flags=0, prog_name="on_switch", prog_ifindex=0, expected_attach_type=BPF_CGROUP_INET_INGRESS, prog_btf_fd=270, func_info_rec_size=8, func_info=0x1b451b0, func_info_cnt=1, line_info_rec_size=16, line_info=0x1b761b0, line_info_cnt=25, attach_btf_id=0, attach_prog_fd=0}, 120) = 306
bpf(BPF_RAW_TRACEPOINT_OPEN, {raw_tracepoint={name="sched_switch", prog_fd=306}}, 120) = 307
bpf(BPF_OBJ_GET_INFO_BY_FD, {info={bpf_fd=307, info_len=32, info=0x7ffda32551b0}}, 120) = 0
bpf(BPF_OBJ_GET_INFO_BY_FD, {info={bpf_fd=305, info_len=80, info=0x7ffda3255200}}, 120) = 0
bpf(BPF_MAP_UPDATE_ELEM, {map_fd=3, key=0x1b76d00, value=0x7ffda32551a8, flags=BPF_ANY}, 120) = 0
bpf(BPF_LINK_GET_FD_BY_ID, 0x7ffda32550e0, 120) = 308
bpf(BPF_MAP_UPDATE_ELEM, {map_fd=303, key=0x7ffda325505c, value=0x7ffda3255058, flags=BPF_ANY}, 120) = 0
bpf(BPF_MAP_UPDATE_ELEM, {map_fd=303, key=0x7ffda325505c, value=0x7ffda3255058, flags=BPF_ANY}, 120) = 0
bpf(BPF_MAP_UPDATE_ELEM, {map_fd=303, key=0x7ffda325505c, value=0x7ffda3255058, flags=BPF_ANY}, 120) = 0
bpf(BPF_MAP_UPDATE_ELEM, {map_fd=303, key=0x7ffda325505c, value=0x7ffda3255058, flags=BPF_ANY}, 120) = 0
bpf(BPF_MAP_UPDATE_ELEM, {map_fd=303, key=0x7ffda325505c, value=0x7ffda3255058, flags=BPF_ANY}, 120) = 0
bpf(BPF_MAP_UPDATE_ELEM, {map_fd=303, key=0x7ffda325505c, value=0x7ffda3255058, flags=BPF_ANY}, 120) = 0
bpf(BPF_MAP_UPDATE_ELEM, {map_fd=303, key=0x7ffda325505c, value=0x7ffda3255058, flags=BPF_ANY}, 120) = 0
bpf(BPF_MAP_UPDATE_ELEM, {map_fd=303, key=0x7ffda325505c, value=0x7ffda3255058, flags=BPF_ANY}, 120) = 0
bpf(BPF_MAP_UPDATE_ELEM, {map_fd=303, key=0x7ffda325505c, value=0x7ffda3255058, flags=BPF_ANY}, 120) = 0
bpf(BPF_MAP_UPDATE_ELEM, {map_fd=303, key=0x7ffda325505c, value=0x7ffda3255058, flags=BPF_ANY}, 120) = 0
bpf(BPF_MAP_UPDATE_ELEM, {map_fd=303, key=0x7ffda325505c, value=0x7ffda3255058, flags=BPF_ANY}, 120) = 0
bpf(BPF_MAP_UPDATE_ELEM, {map_fd=303, key=0x7ffda325505c, value=0x7ffda3255058, flags=BPF_ANY}, 120) = 0
bpf(BPF_MAP_UPDATE_ELEM, {map_fd=303, key=0x7ffda325505c, value=0x7ffda3255058, flags=BPF_ANY}, 120) = 0
bpf(BPF_MAP_UPDATE_ELEM, {map_fd=303, key=0x7ffda325505c, value=0x7ffda3255058, flags=BPF_ANY}, 120) = 0
bpf(BPF_MAP_UPDATE_ELEM, {map_fd=303, key=0x7ffda325505c, value=0x7ffda3255058, flags=BPF_ANY}, 120) = 0
bpf(BPF_MAP_UPDATE_ELEM, {map_fd=303, key=0x7ffda325505c, value=0x7ffda3255058, flags=BPF_ANY}, 120) = 0
bpf(BPF_MAP_UPDATE_ELEM, {map_fd=303, key=0x7ffda325505c, value=0x7ffda3255058, flags=BPF_ANY}, 120) = 0
bpf(BPF_MAP_UPDATE_ELEM, {map_fd=303, key=0x7ffda325505c, value=0x7ffda3255058, flags=BPF_ANY}, 120) = 0
bpf(BPF_MAP_UPDATE_ELEM, {map_fd=303, key=0x7ffda325505c, value=0x7ffda3255058, flags=BPF_ANY}, 120) = 0
bpf(BPF_MAP_UPDATE_ELEM, {map_fd=303, key=0x7ffda325505c, value=0x7ffda3255058, flags=BPF_ANY}, 120) = 0
bpf(BPF_MAP_UPDATE_ELEM, {map_fd=303, key=0x7ffda325505c, value=0x7ffda3255058, flags=BPF_ANY}, 120) = 0
bpf(BPF_MAP_UPDATE_ELEM, {map_fd=303, key=0x7ffda325505c, value=0x7ffda3255058, flags=BPF_ANY}, 120) = 0
bpf(BPF_MAP_UPDATE_ELEM, {map_fd=303, key=0x7ffda325505c, value=0x7ffda3255058, flags=BPF_ANY}, 120) = 0
bpf(BPF_MAP_UPDATE_ELEM, {map_fd=303, key=0x7ffda325505c, value=0x7ffda3255058, flags=BPF_ANY}, 120) = 0
bpf(BPF_OBJ_GET_INFO_BY_FD, {info={bpf_fd=308, info_len=32, info=0x7ffda32551b0}}, 120) = 0
bpf(BPF_PROG_GET_FD_BY_ID, {prog_id=229, next_id=0, open_flags=0}, 120) = 270
bpf(BPF_MAP_GET_FD_BY_ID, {map_id=251, next_id=0, open_flags=0}, 120) = 303
bpf(BPF_PROG_TEST_RUN, {test={prog_fd=270, retval=0, data_size_in=0, data_size_out=0, data_in=NULL, data_out=NULL, repeat=0, duration=0, ctx_size_in=0, ctx_size_out=0, ctx_in=NULL, ctx_out=NULL}, ...}, 120) = 0
libbpf: elf: skipping unrecognized data section(8) .eh_frame
libbpf: elf: skipping relo section(13) .rel.eh_frame for section(8) .eh_frame
bpf(BPF_OBJ_GET_INFO_BY_FD, {info={bpf_fd=270, info_len=216, info=0x7ffda3254f50}}, 120) = 0
bpf(BPF_OBJ_GET_INFO_BY_FD, {info={bpf_fd=270, info_len=216, info=0x1b72d20}}, 120) = 0
bpf(BPF_BTF_GET_FD_BY_ID, {btf_id=323}, 120) = 304
bpf(BPF_OBJ_GET_INFO_BY_FD, {info={bpf_fd=304, info_len=32, info=0x7ffda3255000}}, 120) = 0
bpf(BPF_OBJ_GET_INFO_BY_FD, {info={bpf_fd=303, info_len=80, info=0x7ffda32550f0}}, 120) = 0
bpf(BPF_PROG_LOAD, {prog_type=BPF_PROG_TYPE_SOCKET_FILTER, insn_cnt=2, insns=0x7ffda3254ff0, license="GPL", log_level=0, log_size=0, log_buf=NULL, kern_version=KERNEL_VERSION(0, 0, 0), prog_flags=0, prog_name="", prog_ifindex=0, expected_attach_type=BPF_CGROUP_INET_INGRESS, prog_btf_fd=0, func_info_rec_size=0, func_info=NULL, func_info_cnt=0, line_info_rec_size=0, line_info=NULL, line_info_cnt=0, attach_btf_id=0, attach_prog_fd=0}, 120) = 305
bpf(BPF_BTF_LOAD, {btf="\237\353\1\0\30\0\0\0\0\0\0\0\204\2\0\0\204\2\0\0\250\3\0\0\0\0\0\0\0\0\0\2"..., btf_log_buf=NULL, btf_size=1604, btf_log_size=0, btf_log_level=0}, 120) = 305
bpf(BPF_MAP_CREATE, {map_type=BPF_MAP_TYPE_PERCPU_ARRAY, key_size=4, value_size=24, max_entries=1, map_flags=0, inner_map_fd=0, map_name="accum_readings", map_ifindex=0, btf_fd=305, btf_key_type_id=0, btf_value_type_id=0, btf_vmlinux_value_type_id=0}, 120) = 306
bpf(BPF_MAP_CREATE, {map_type=BPF_MAP_TYPE_HASH, key_size=4, value_size=4, max_entries=1, map_flags=0, inner_map_fd=0, map_name="filter", map_ifindex=0, btf_fd=305, btf_key_type_id=0, btf_value_type_id=0, btf_vmlinux_value_type_id=0}, 120) = 307
bpf(BPF_MAP_CREATE, {map_type=BPF_MAP_TYPE_ARRAY, key_size=4, value_size=8, max_entries=1, map_flags=BPF_F_MMAPABLE, inner_map_fd=0, map_name="bperf_fo.bss", map_ifindex=0, btf_fd=305, btf_key_type_id=0, btf_value_type_id=27, btf_vmlinux_value_type_id=0}, 120) = 333
bpf(BPF_MAP_UPDATE_ELEM, {map_fd=333, key=0x7ffda3254e50, value=0x7f0a61b67000, flags=BPF_ANY}, 120) = 0
bpf(BPF_PROG_LOAD, {prog_type=BPF_PROG_TYPE_TRACING, insn_cnt=58, insns=0x1b35650, license="Dual BSD/GPL", log_level=0, log_size=0, log_buf=NULL, kern_version=KERNEL_VERSION(5, 12, 0), prog_flags=0, prog_name="fexit_XXX", prog_ifindex=0, expected_attach_type=BPF_TRACE_FEXIT, prog_btf_fd=305, func_info_rec_size=8, func_info=0x1b45120, func_info_cnt=1, line_info_rec_size=16, line_info=0x1b44100, line_info_cnt=28, attach_btf_id=22, attach_prog_fd=270}, 120) = 334
bpf(BPF_MAP_UPDATE_ELEM, {map_fd=307, key=0x7ffda32551b0, value=0x7ffda32551a4, flags=BPF_ANY}, 120) = 0
bpf(BPF_RAW_TRACEPOINT_OPEN, {raw_tracepoint={name=NULL, prog_fd=334}}, 120) = 335
--- SIGCHLD {si_signo=SIGCHLD, si_code=CLD_EXITED, si_pid=75357, si_uid=0, si_status=0, si_utime=0, si_stime=0} ---
bpf(BPF_PROG_TEST_RUN, {test={prog_fd=4, retval=0, data_size_in=0, data_size_out=0, data_in=NULL, data_out=NULL, repeat=0, duration=0, ctx_size_in=0, ctx_size_out=0, ctx_in=NULL, ctx_out=NULL}, ...}, 120) = 0
bpf(BPF_PROG_TEST_RUN, {test={prog_fd=4, retval=0, data_size_in=0, data_size_out=0, data_in=NULL, data_out=NULL, repeat=0, duration=0, ctx_size_in=0, ctx_size_out=0, ctx_in=NULL, ctx_out=NULL}, ...}, 120) = 0
bpf(BPF_PROG_TEST_RUN, {test={prog_fd=4, retval=0, data_size_in=0, data_size_out=0, data_in=NULL, data_out=NULL, repeat=0, duration=0, ctx_size_in=0, ctx_size_out=0, ctx_in=NULL, ctx_out=NULL}, ...}, 120) = 0
bpf(BPF_PROG_TEST_RUN, {test={prog_fd=4, retval=0, data_size_in=0, data_size_out=0, data_in=NULL, data_out=NULL, repeat=0, duration=0, ctx_size_in=0, ctx_size_out=0, ctx_in=NULL, ctx_out=NULL}, ...}, 120) = 0
bpf(BPF_PROG_TEST_RUN, {test={prog_fd=4, retval=0, data_size_in=0, data_size_out=0, data_in=NULL, data_out=NULL, repeat=0, duration=0, ctx_size_in=0, ctx_size_out=0, ctx_in=NULL, ctx_out=NULL}, ...}, 120) = 0
bpf(BPF_PROG_TEST_RUN, {test={prog_fd=4, retval=0, data_size_in=0, data_size_out=0, data_in=NULL, data_out=NULL, repeat=0, duration=0, ctx_size_in=0, ctx_size_out=0, ctx_in=NULL, ctx_out=NULL}, ...}, 120) = 0
bpf(BPF_PROG_TEST_RUN, {test={prog_fd=4, retval=0, data_size_in=0, data_size_out=0, data_in=NULL, data_out=NULL, repeat=0, duration=0, ctx_size_in=0, ctx_size_out=0, ctx_in=NULL, ctx_out=NULL}, ...}, 120) = 0
bpf(BPF_PROG_TEST_RUN, {test={prog_fd=4, retval=0, data_size_in=0, data_size_out=0, data_in=NULL, data_out=NULL, repeat=0, duration=0, ctx_size_in=0, ctx_size_out=0, ctx_in=NULL, ctx_out=NULL}, ...}, 120) = 0
bpf(BPF_PROG_TEST_RUN, {test={prog_fd=4, retval=0, data_size_in=0, data_size_out=0, data_in=NULL, data_out=NULL, repeat=0, duration=0, ctx_size_in=0, ctx_size_out=0, ctx_in=NULL, ctx_out=NULL}, ...}, 120) = 0
bpf(BPF_PROG_TEST_RUN, {test={prog_fd=4, retval=0, data_size_in=0, data_size_out=0, data_in=NULL, data_out=NULL, repeat=0, duration=0, ctx_size_in=0, ctx_size_out=0, ctx_in=NULL, ctx_out=NULL}, ...}, 120) = 0
bpf(BPF_PROG_TEST_RUN, {test={prog_fd=4, retval=0, data_size_in=0, data_size_out=0, data_in=NULL, data_out=NULL, repeat=0, duration=0, ctx_size_in=0, ctx_size_out=0, ctx_in=NULL, ctx_out=NULL}, ...}, 120) = 0
bpf(BPF_PROG_TEST_RUN, {test={prog_fd=4, retval=0, data_size_in=0, data_size_out=0, data_in=NULL, data_out=NULL, repeat=0, duration=0, ctx_size_in=0, ctx_size_out=0, ctx_in=NULL, ctx_out=NULL}, ...}, 120) = 0
bpf(BPF_PROG_TEST_RUN, {test={prog_fd=4, retval=0, data_size_in=0, data_size_out=0, data_in=NULL, data_out=NULL, repeat=0, duration=0, ctx_size_in=0, ctx_size_out=0, ctx_in=NULL, ctx_out=NULL}, ...}, 120) = 0
bpf(BPF_PROG_TEST_RUN, {test={prog_fd=4, retval=0, data_size_in=0, data_size_out=0, data_in=NULL, data_out=NULL, repeat=0, duration=0, ctx_size_in=0, ctx_size_out=0, ctx_in=NULL, ctx_out=NULL}, ...}, 120) = 0
bpf(BPF_PROG_TEST_RUN, {test={prog_fd=4, retval=0, data_size_in=0, data_size_out=0, data_in=NULL, data_out=NULL, repeat=0, duration=0, ctx_size_in=0, ctx_size_out=0, ctx_in=NULL, ctx_out=NULL}, ...}, 120) = 0
bpf(BPF_PROG_TEST_RUN, {test={prog_fd=4, retval=0, data_size_in=0, data_size_out=0, data_in=NULL, data_out=NULL, repeat=0, duration=0, ctx_size_in=0, ctx_size_out=0, ctx_in=NULL, ctx_out=NULL}, ...}, 120) = 0
bpf(BPF_PROG_TEST_RUN, {test={prog_fd=4, retval=0, data_size_in=0, data_size_out=0, data_in=NULL, data_out=NULL, repeat=0, duration=0, ctx_size_in=0, ctx_size_out=0, ctx_in=NULL, ctx_out=NULL}, ...}, 120) = 0
bpf(BPF_PROG_TEST_RUN, {test={prog_fd=4, retval=0, data_size_in=0, data_size_out=0, data_in=NULL, data_out=NULL, repeat=0, duration=0, ctx_size_in=0, ctx_size_out=0, ctx_in=NULL, ctx_out=NULL}, ...}, 120) = 0
bpf(BPF_PROG_TEST_RUN, {test={prog_fd=4, retval=0, data_size_in=0, data_size_out=0, data_in=NULL, data_out=NULL, repeat=0, duration=0, ctx_size_in=0, ctx_size_out=0, ctx_in=NULL, ctx_out=NULL}, ...}, 120) = 0
bpf(BPF_PROG_TEST_RUN, {test={prog_fd=4, retval=0, data_size_in=0, data_size_out=0, data_in=NULL, data_out=NULL, repeat=0, duration=0, ctx_size_in=0, ctx_size_out=0, ctx_in=NULL, ctx_out=NULL}, ...}, 120) = 0
bpf(BPF_PROG_TEST_RUN, {test={prog_fd=4, retval=0, data_size_in=0, data_size_out=0, data_in=NULL, data_out=NULL, repeat=0, duration=0, ctx_size_in=0, ctx_size_out=0, ctx_in=NULL, ctx_out=NULL}, ...}, 120) = 0
bpf(BPF_PROG_TEST_RUN, {test={prog_fd=4, retval=0, data_size_in=0, data_size_out=0, data_in=NULL, data_out=NULL, repeat=0, duration=0, ctx_size_in=0, ctx_size_out=0, ctx_in=NULL, ctx_out=NULL}, ...}, 120) = 0
bpf(BPF_PROG_TEST_RUN, {test={prog_fd=4, retval=0, data_size_in=0, data_size_out=0, data_in=NULL, data_out=NULL, repeat=0, duration=0, ctx_size_in=0, ctx_size_out=0, ctx_in=NULL, ctx_out=NULL}, ...}, 120) = 0
bpf(BPF_PROG_TEST_RUN, {test={prog_fd=4, retval=0, data_size_in=0, data_size_out=0, data_in=NULL, data_out=NULL, repeat=0, duration=0, ctx_size_in=0, ctx_size_out=0, ctx_in=NULL, ctx_out=NULL}, ...}, 120) = 0
bpf(BPF_MAP_LOOKUP_ELEM, {map_fd=9, key=0x7ffda3255160, value=0x7ffda3254f10, flags=BPF_ANY}, 120) = 0
bpf(BPF_PROG_TEST_RUN, {test={prog_fd=5, retval=0, data_size_in=0, data_size_out=0, data_in=NULL, data_out=NULL, repeat=0, duration=0, ctx_size_in=0, ctx_size_out=0, ctx_in=NULL, ctx_out=NULL}, ...}, 120) = 0
bpf(BPF_PROG_TEST_RUN, {test={prog_fd=5, retval=0, data_size_in=0, data_size_out=0, data_in=NULL, data_out=NULL, repeat=0, duration=0, ctx_size_in=0, ctx_size_out=0, ctx_in=NULL, ctx_out=NULL}, ...}, 120) = 0
bpf(BPF_PROG_TEST_RUN, {test={prog_fd=5, retval=0, data_size_in=0, data_size_out=0, data_in=NULL, data_out=NULL, repeat=0, duration=0, ctx_size_in=0, ctx_size_out=0, ctx_in=NULL, ctx_out=NULL}, ...}, 120) = 0
bpf(BPF_PROG_TEST_RUN, {test={prog_fd=5, retval=0, data_size_in=0, data_size_out=0, data_in=NULL, data_out=NULL, repeat=0, duration=0, ctx_size_in=0, ctx_size_out=0, ctx_in=NULL, ctx_out=NULL}, ...}, 120) = 0
bpf(BPF_PROG_TEST_RUN, {test={prog_fd=5, retval=0, data_size_in=0, data_size_out=0, data_in=NULL, data_out=NULL, repeat=0, duration=0, ctx_size_in=0, ctx_size_out=0, ctx_in=NULL, ctx_out=NULL}, ...}, 120) = 0
bpf(BPF_PROG_TEST_RUN, {test={prog_fd=5, retval=0, data_size_in=0, data_size_out=0, data_in=NULL, data_out=NULL, repeat=0, duration=0, ctx_size_in=0, ctx_size_out=0, ctx_in=NULL, ctx_out=NULL}, ...}, 120) = 0
bpf(BPF_PROG_TEST_RUN, {test={prog_fd=5, retval=0, data_size_in=0, data_size_out=0, data_in=NULL, data_out=NULL, repeat=0, duration=0, ctx_size_in=0, ctx_size_out=0, ctx_in=NULL, ctx_out=NULL}, ...}, 120) = 0
bpf(BPF_PROG_TEST_RUN, {test={prog_fd=5, retval=0, data_size_in=0, data_size_out=0, data_in=NULL, data_out=NULL, repeat=0, duration=0, ctx_size_in=0, ctx_size_out=0, ctx_in=NULL, ctx_out=NULL}, ...}, 120) = 0
bpf(BPF_PROG_TEST_RUN, {test={prog_fd=5, retval=0, data_size_in=0, data_size_out=0, data_in=NULL, data_out=NULL, repeat=0, duration=0, ctx_size_in=0, ctx_size_out=0, ctx_in=NULL, ctx_out=NULL}, ...}, 120) = 0
bpf(BPF_PROG_TEST_RUN, {test={prog_fd=5, retval=0, data_size_in=0, data_size_out=0, data_in=NULL, data_out=NULL, repeat=0, duration=0, ctx_size_in=0, ctx_size_out=0, ctx_in=NULL, ctx_out=NULL}, ...}, 120) = 0
bpf(BPF_PROG_TEST_RUN, {test={prog_fd=5, retval=0, data_size_in=0, data_size_out=0, data_in=NULL, data_out=NULL, repeat=0, duration=0, ctx_size_in=0, ctx_size_out=0, ctx_in=NULL, ctx_out=NULL}, ...}, 120) = 0
bpf(BPF_PROG_TEST_RUN, {test={prog_fd=5, retval=0, data_size_in=0, data_size_out=0, data_in=NULL, data_out=NULL, repeat=0, duration=0, ctx_size_in=0, ctx_size_out=0, ctx_in=NULL, ctx_out=NULL}, ...}, 120) = 0
bpf(BPF_PROG_TEST_RUN, {test={prog_fd=5, retval=0, data_size_in=0, data_size_out=0, data_in=NULL, data_out=NULL, repeat=0, duration=0, ctx_size_in=0, ctx_size_out=0, ctx_in=NULL, ctx_out=NULL}, ...}, 120) = 0
bpf(BPF_PROG_TEST_RUN, {test={prog_fd=5, retval=0, data_size_in=0, data_size_out=0, data_in=NULL, data_out=NULL, repeat=0, duration=0, ctx_size_in=0, ctx_size_out=0, ctx_in=NULL, ctx_out=NULL}, ...}, 120) = 0
bpf(BPF_PROG_TEST_RUN, {test={prog_fd=5, retval=0, data_size_in=0, data_size_out=0, data_in=NULL, data_out=NULL, repeat=0, duration=0, ctx_size_in=0, ctx_size_out=0, ctx_in=NULL, ctx_out=NULL}, ...}, 120) = 0
bpf(BPF_PROG_TEST_RUN, {test={prog_fd=5, retval=0, data_size_in=0, data_size_out=0, data_in=NULL, data_out=NULL, repeat=0, duration=0, ctx_size_in=0, ctx_size_out=0, ctx_in=NULL, ctx_out=NULL}, ...}, 120) = 0
bpf(BPF_PROG_TEST_RUN, {test={prog_fd=5, retval=0, data_size_in=0, data_size_out=0, data_in=NULL, data_out=NULL, repeat=0, duration=0, ctx_size_in=0, ctx_size_out=0, ctx_in=NULL, ctx_out=NULL}, ...}, 120) = 0
bpf(BPF_PROG_TEST_RUN, {test={prog_fd=5, retval=0, data_size_in=0, data_size_out=0, data_in=NULL, data_out=NULL, repeat=0, duration=0, ctx_size_in=0, ctx_size_out=0, ctx_in=NULL, ctx_out=NULL}, ...}, 120) = 0
bpf(BPF_PROG_TEST_RUN, {test={prog_fd=5, retval=0, data_size_in=0, data_size_out=0, data_in=NULL, data_out=NULL, repeat=0, duration=0, ctx_size_in=0, ctx_size_out=0, ctx_in=NULL, ctx_out=NULL}, ...}, 120) = 0
bpf(BPF_PROG_TEST_RUN, {test={prog_fd=5, retval=0, data_size_in=0, data_size_out=0, data_in=NULL, data_out=NULL, repeat=0, duration=0, ctx_size_in=0, ctx_size_out=0, ctx_in=NULL, ctx_out=NULL}, ...}, 120) = 0
bpf(BPF_PROG_TEST_RUN, {test={prog_fd=5, retval=0, data_size_in=0, data_size_out=0, data_in=NULL, data_out=NULL, repeat=0, duration=0, ctx_size_in=0, ctx_size_out=0, ctx_in=NULL, ctx_out=NULL}, ...}, 120) = 0
bpf(BPF_PROG_TEST_RUN, {test={prog_fd=5, retval=0, data_size_in=0, data_size_out=0, data_in=NULL, data_out=NULL, repeat=0, duration=0, ctx_size_in=0, ctx_size_out=0, ctx_in=NULL, ctx_out=NULL}, ...}, 120) = 0
bpf(BPF_PROG_TEST_RUN, {test={prog_fd=5, retval=0, data_size_in=0, data_size_out=0, data_in=NULL, data_out=NULL, repeat=0, duration=0, ctx_size_in=0, ctx_size_out=0, ctx_in=NULL, ctx_out=NULL}, ...}, 120) = 0
bpf(BPF_PROG_TEST_RUN, {test={prog_fd=5, retval=0, data_size_in=0, data_size_out=0, data_in=NULL, data_out=NULL, repeat=0, duration=0, ctx_size_in=0, ctx_size_out=0, ctx_in=NULL, ctx_out=NULL}, ...}, 120) = 0
bpf(BPF_MAP_LOOKUP_ELEM, {map_fd=42, key=0x7ffda3255160, value=0x7ffda3254f10, flags=BPF_ANY}, 120) = 0
bpf(BPF_PROG_TEST_RUN, {test={prog_fd=39, retval=0, data_size_in=0, data_size_out=0, data_in=NULL, data_out=NULL, repeat=0, duration=0, ctx_size_in=0, ctx_size_out=0, ctx_in=NULL, ctx_out=NULL}, ...}, 120) = 0
bpf(BPF_PROG_TEST_RUN, {test={prog_fd=39, retval=0, data_size_in=0, data_size_out=0, data_in=NULL, data_out=NULL, repeat=0, duration=0, ctx_size_in=0, ctx_size_out=0, ctx_in=NULL, ctx_out=NULL}, ...}, 120) = 0
bpf(BPF_PROG_TEST_RUN, {test={prog_fd=39, retval=0, data_size_in=0, data_size_out=0, data_in=NULL, data_out=NULL, repeat=0, duration=0, ctx_size_in=0, ctx_size_out=0, ctx_in=NULL, ctx_out=NULL}, ...}, 120) = 0
bpf(BPF_PROG_TEST_RUN, {test={prog_fd=39, retval=0, data_size_in=0, data_size_out=0, data_in=NULL, data_out=NULL, repeat=0, duration=0, ctx_size_in=0, ctx_size_out=0, ctx_in=NULL, ctx_out=NULL}, ...}, 120) = 0
bpf(BPF_PROG_TEST_RUN, {test={prog_fd=39, retval=0, data_size_in=0, data_size_out=0, data_in=NULL, data_out=NULL, repeat=0, duration=0, ctx_size_in=0, ctx_size_out=0, ctx_in=NULL, ctx_out=NULL}, ...}, 120) = 0
bpf(BPF_PROG_TEST_RUN, {test={prog_fd=39, retval=0, data_size_in=0, data_size_out=0, data_in=NULL, data_out=NULL, repeat=0, duration=0, ctx_size_in=0, ctx_size_out=0, ctx_in=NULL, ctx_out=NULL}, ...}, 120) = 0
bpf(BPF_PROG_TEST_RUN, {test={prog_fd=39, retval=0, data_size_in=0, data_size_out=0, data_in=NULL, data_out=NULL, repeat=0, duration=0, ctx_size_in=0, ctx_size_out=0, ctx_in=NULL, ctx_out=NULL}, ...}, 120) = 0
bpf(BPF_PROG_TEST_RUN, {test={prog_fd=39, retval=0, data_size_in=0, data_size_out=0, data_in=NULL, data_out=NULL, repeat=0, duration=0, ctx_size_in=0, ctx_size_out=0, ctx_in=NULL, ctx_out=NULL}, ...}, 120) = 0
bpf(BPF_PROG_TEST_RUN, {test={prog_fd=39, retval=0, data_size_in=0, data_size_out=0, data_in=NULL, data_out=NULL, repeat=0, duration=0, ctx_size_in=0, ctx_size_out=0, ctx_in=NULL, ctx_out=NULL}, ...}, 120) = 0
bpf(BPF_PROG_TEST_RUN, {test={prog_fd=39, retval=0, data_size_in=0, data_size_out=0, data_in=NULL, data_out=NULL, repeat=0, duration=0, ctx_size_in=0, ctx_size_out=0, ctx_in=NULL, ctx_out=NULL}, ...}, 120) = 0
bpf(BPF_PROG_TEST_RUN, {test={prog_fd=39, retval=0, data_size_in=0, data_size_out=0, data_in=NULL, data_out=NULL, repeat=0, duration=0, ctx_size_in=0, ctx_size_out=0, ctx_in=NULL, ctx_out=NULL}, ...}, 120) = 0
bpf(BPF_PROG_TEST_RUN, {test={prog_fd=39, retval=0, data_size_in=0, data_size_out=0, data_in=NULL, data_out=NULL, repeat=0, duration=0, ctx_size_in=0, ctx_size_out=0, ctx_in=NULL, ctx_out=NULL}, ...}, 120) = 0
bpf(BPF_PROG_TEST_RUN, {test={prog_fd=39, retval=0, data_size_in=0, data_size_out=0, data_in=NULL, data_out=NULL, repeat=0, duration=0, ctx_size_in=0, ctx_size_out=0, ctx_in=NULL, ctx_out=NULL}, ...}, 120) = 0
bpf(BPF_PROG_TEST_RUN, {test={prog_fd=39, retval=0, data_size_in=0, data_size_out=0, data_in=NULL, data_out=NULL, repeat=0, duration=0, ctx_size_in=0, ctx_size_out=0, ctx_in=NULL, ctx_out=NULL}, ...}, 120) = 0
bpf(BPF_PROG_TEST_RUN, {test={prog_fd=39, retval=0, data_size_in=0, data_size_out=0, data_in=NULL, data_out=NULL, repeat=0, duration=0, ctx_size_in=0, ctx_size_out=0, ctx_in=NULL, ctx_out=NULL}, ...}, 120) = 0
bpf(BPF_PROG_TEST_RUN, {test={prog_fd=39, retval=0, data_size_in=0, data_size_out=0, data_in=NULL, data_out=NULL, repeat=0, duration=0, ctx_size_in=0, ctx_size_out=0, ctx_in=NULL, ctx_out=NULL}, ...}, 120) = 0
bpf(BPF_PROG_TEST_RUN, {test={prog_fd=39, retval=0, data_size_in=0, data_size_out=0, data_in=NULL, data_out=NULL, repeat=0, duration=0, ctx_size_in=0, ctx_size_out=0, ctx_in=NULL, ctx_out=NULL}, ...}, 120) = 0
bpf(BPF_PROG_TEST_RUN, {test={prog_fd=39, retval=0, data_size_in=0, data_size_out=0, data_in=NULL, data_out=NULL, repeat=0, duration=0, ctx_size_in=0, ctx_size_out=0, ctx_in=NULL, ctx_out=NULL}, ...}, 120) = 0
bpf(BPF_PROG_TEST_RUN, {test={prog_fd=39, retval=0, data_size_in=0, data_size_out=0, data_in=NULL, data_out=NULL, repeat=0, duration=0, ctx_size_in=0, ctx_size_out=0, ctx_in=NULL, ctx_out=NULL}, ...}, 120) = 0
bpf(BPF_PROG_TEST_RUN, {test={prog_fd=39, retval=0, data_size_in=0, data_size_out=0, data_in=NULL, data_out=NULL, repeat=0, duration=0, ctx_size_in=0, ctx_size_out=0, ctx_in=NULL, ctx_out=NULL}, ...}, 120) = 0
bpf(BPF_PROG_TEST_RUN, {test={prog_fd=39, retval=0, data_size_in=0, data_size_out=0, data_in=NULL, data_out=NULL, repeat=0, duration=0, ctx_size_in=0, ctx_size_out=0, ctx_in=NULL, ctx_out=NULL}, ...}, 120) = 0
bpf(BPF_PROG_TEST_RUN, {test={prog_fd=39, retval=0, data_size_in=0, data_size_out=0, data_in=NULL, data_out=NULL, repeat=0, duration=0, ctx_size_in=0, ctx_size_out=0, ctx_in=NULL, ctx_out=NULL}, ...}, 120) = 0
bpf(BPF_PROG_TEST_RUN, {test={prog_fd=39, retval=0, data_size_in=0, data_size_out=0, data_in=NULL, data_out=NULL, repeat=0, duration=0, ctx_size_in=0, ctx_size_out=0, ctx_in=NULL, ctx_out=NULL}, ...}, 120) = 0
bpf(BPF_PROG_TEST_RUN, {test={prog_fd=39, retval=0, data_size_in=0, data_size_out=0, data_in=NULL, data_out=NULL, repeat=0, duration=0, ctx_size_in=0, ctx_size_out=0, ctx_in=NULL, ctx_out=NULL}, ...}, 120) = 0
bpf(BPF_MAP_LOOKUP_ELEM, {map_fd=75, key=0x7ffda3255160, value=0x7ffda3254f10, flags=BPF_ANY}, 120) = 0
bpf(BPF_PROG_TEST_RUN, {test={prog_fd=72, retval=0, data_size_in=0, data_size_out=0, data_in=NULL, data_out=NULL, repeat=0, duration=0, ctx_size_in=0, ctx_size_out=0, ctx_in=NULL, ctx_out=NULL}, ...}, 120) = 0
bpf(BPF_PROG_TEST_RUN, {test={prog_fd=72, retval=0, data_size_in=0, data_size_out=0, data_in=NULL, data_out=NULL, repeat=0, duration=0, ctx_size_in=0, ctx_size_out=0, ctx_in=NULL, ctx_out=NULL}, ...}, 120) = 0
bpf(BPF_PROG_TEST_RUN, {test={prog_fd=72, retval=0, data_size_in=0, data_size_out=0, data_in=NULL, data_out=NULL, repeat=0, duration=0, ctx_size_in=0, ctx_size_out=0, ctx_in=NULL, ctx_out=NULL}, ...}, 120) = 0
bpf(BPF_PROG_TEST_RUN, {test={prog_fd=72, retval=0, data_size_in=0, data_size_out=0, data_in=NULL, data_out=NULL, repeat=0, duration=0, ctx_size_in=0, ctx_size_out=0, ctx_in=NULL, ctx_out=NULL}, ...}, 120) = 0
bpf(BPF_PROG_TEST_RUN, {test={prog_fd=72, retval=0, data_size_in=0, data_size_out=0, data_in=NULL, data_out=NULL, repeat=0, duration=0, ctx_size_in=0, ctx_size_out=0, ctx_in=NULL, ctx_out=NULL}, ...}, 120) = 0
bpf(BPF_PROG_TEST_RUN, {test={prog_fd=72, retval=0, data_size_in=0, data_size_out=0, data_in=NULL, data_out=NULL, repeat=0, duration=0, ctx_size_in=0, ctx_size_out=0, ctx_in=NULL, ctx_out=NULL}, ...}, 120) = 0
bpf(BPF_PROG_TEST_RUN, {test={prog_fd=72, retval=0, data_size_in=0, data_size_out=0, data_in=NULL, data_out=NULL, repeat=0, duration=0, ctx_size_in=0, ctx_size_out=0, ctx_in=NULL, ctx_out=NULL}, ...}, 120) = 0
bpf(BPF_PROG_TEST_RUN, {test={prog_fd=72, retval=0, data_size_in=0, data_size_out=0, data_in=NULL, data_out=NULL, repeat=0, duration=0, ctx_size_in=0, ctx_size_out=0, ctx_in=NULL, ctx_out=NULL}, ...}, 120) = 0
bpf(BPF_PROG_TEST_RUN, {test={prog_fd=72, retval=0, data_size_in=0, data_size_out=0, data_in=NULL, data_out=NULL, repeat=0, duration=0, ctx_size_in=0, ctx_size_out=0, ctx_in=NULL, ctx_out=NULL}, ...}, 120) = 0
bpf(BPF_PROG_TEST_RUN, {test={prog_fd=72, retval=0, data_size_in=0, data_size_out=0, data_in=NULL, data_out=NULL, repeat=0, duration=0, ctx_size_in=0, ctx_size_out=0, ctx_in=NULL, ctx_out=NULL}, ...}, 120) = 0
bpf(BPF_PROG_TEST_RUN, {test={prog_fd=72, retval=0, data_size_in=0, data_size_out=0, data_in=NULL, data_out=NULL, repeat=0, duration=0, ctx_size_in=0, ctx_size_out=0, ctx_in=NULL, ctx_out=NULL}, ...}, 120) = 0
bpf(BPF_PROG_TEST_RUN, {test={prog_fd=72, retval=0, data_size_in=0, data_size_out=0, data_in=NULL, data_out=NULL, repeat=0, duration=0, ctx_size_in=0, ctx_size_out=0, ctx_in=NULL, ctx_out=NULL}, ...}, 120) = 0
bpf(BPF_PROG_TEST_RUN, {test={prog_fd=72, retval=0, data_size_in=0, data_size_out=0, data_in=NULL, data_out=NULL, repeat=0, duration=0, ctx_size_in=0, ctx_size_out=0, ctx_in=NULL, ctx_out=NULL}, ...}, 120) = 0
bpf(BPF_PROG_TEST_RUN, {test={prog_fd=72, retval=0, data_size_in=0, data_size_out=0, data_in=NULL, data_out=NULL, repeat=0, duration=0, ctx_size_in=0, ctx_size_out=0, ctx_in=NULL, ctx_out=NULL}, ...}, 120) = 0
bpf(BPF_PROG_TEST_RUN, {test={prog_fd=72, retval=0, data_size_in=0, data_size_out=0, data_in=NULL, data_out=NULL, repeat=0, duration=0, ctx_size_in=0, ctx_size_out=0, ctx_in=NULL, ctx_out=NULL}, ...}, 120) = 0
bpf(BPF_PROG_TEST_RUN, {test={prog_fd=72, retval=0, data_size_in=0, data_size_out=0, data_in=NULL, data_out=NULL, repeat=0, duration=0, ctx_size_in=0, ctx_size_out=0, ctx_in=NULL, ctx_out=NULL}, ...}, 120) = 0
bpf(BPF_PROG_TEST_RUN, {test={prog_fd=72, retval=0, data_size_in=0, data_size_out=0, data_in=NULL, data_out=NULL, repeat=0, duration=0, ctx_size_in=0, ctx_size_out=0, ctx_in=NULL, ctx_out=NULL}, ...}, 120) = 0
bpf(BPF_PROG_TEST_RUN, {test={prog_fd=72, retval=0, data_size_in=0, data_size_out=0, data_in=NULL, data_out=NULL, repeat=0, duration=0, ctx_size_in=0, ctx_size_out=0, ctx_in=NULL, ctx_out=NULL}, ...}, 120) = 0
bpf(BPF_PROG_TEST_RUN, {test={prog_fd=72, retval=0, data_size_in=0, data_size_out=0, data_in=NULL, data_out=NULL, repeat=0, duration=0, ctx_size_in=0, ctx_size_out=0, ctx_in=NULL, ctx_out=NULL}, ...}, 120) = 0
bpf(BPF_PROG_TEST_RUN, {test={prog_fd=72, retval=0, data_size_in=0, data_size_out=0, data_in=NULL, data_out=NULL, repeat=0, duration=0, ctx_size_in=0, ctx_size_out=0, ctx_in=NULL, ctx_out=NULL}, ...}, 120) = 0
bpf(BPF_PROG_TEST_RUN, {test={prog_fd=72, retval=0, data_size_in=0, data_size_out=0, data_in=NULL, data_out=NULL, repeat=0, duration=0, ctx_size_in=0, ctx_size_out=0, ctx_in=NULL, ctx_out=NULL}, ...}, 120) = 0
bpf(BPF_PROG_TEST_RUN, {test={prog_fd=72, retval=0, data_size_in=0, data_size_out=0, data_in=NULL, data_out=NULL, repeat=0, duration=0, ctx_size_in=0, ctx_size_out=0, ctx_in=NULL, ctx_out=NULL}, ...}, 120) = 0
bpf(BPF_PROG_TEST_RUN, {test={prog_fd=72, retval=0, data_size_in=0, data_size_out=0, data_in=NULL, data_out=NULL, repeat=0, duration=0, ctx_size_in=0, ctx_size_out=0, ctx_in=NULL, ctx_out=NULL}, ...}, 120) = 0
bpf(BPF_PROG_TEST_RUN, {test={prog_fd=72, retval=0, data_size_in=0, data_size_out=0, data_in=NULL, data_out=NULL, repeat=0, duration=0, ctx_size_in=0, ctx_size_out=0, ctx_in=NULL, ctx_out=NULL}, ...}, 120) = 0
bpf(BPF_MAP_LOOKUP_ELEM, {map_fd=108, key=0x7ffda3255160, value=0x7ffda3254f10, flags=BPF_ANY}, 120) = 0
bpf(BPF_PROG_TEST_RUN, {test={prog_fd=105, retval=0, data_size_in=0, data_size_out=0, data_in=NULL, data_out=NULL, repeat=0, duration=0, ctx_size_in=0, ctx_size_out=0, ctx_in=NULL, ctx_out=NULL}, ...}, 120) = 0
bpf(BPF_PROG_TEST_RUN, {test={prog_fd=105, retval=0, data_size_in=0, data_size_out=0, data_in=NULL, data_out=NULL, repeat=0, duration=0, ctx_size_in=0, ctx_size_out=0, ctx_in=NULL, ctx_out=NULL}, ...}, 120) = 0
bpf(BPF_PROG_TEST_RUN, {test={prog_fd=105, retval=0, data_size_in=0, data_size_out=0, data_in=NULL, data_out=NULL, repeat=0, duration=0, ctx_size_in=0, ctx_size_out=0, ctx_in=NULL, ctx_out=NULL}, ...}, 120) = 0
bpf(BPF_PROG_TEST_RUN, {test={prog_fd=105, retval=0, data_size_in=0, data_size_out=0, data_in=NULL, data_out=NULL, repeat=0, duration=0, ctx_size_in=0, ctx_size_out=0, ctx_in=NULL, ctx_out=NULL}, ...}, 120) = 0
bpf(BPF_PROG_TEST_RUN, {test={prog_fd=105, retval=0, data_size_in=0, data_size_out=0, data_in=NULL, data_out=NULL, repeat=0, duration=0, ctx_size_in=0, ctx_size_out=0, ctx_in=NULL, ctx_out=NULL}, ...}, 120) = 0
bpf(BPF_PROG_TEST_RUN, {test={prog_fd=105, retval=0, data_size_in=0, data_size_out=0, data_in=NULL, data_out=NULL, repeat=0, duration=0, ctx_size_in=0, ctx_size_out=0, ctx_in=NULL, ctx_out=NULL}, ...}, 120) = 0
bpf(BPF_PROG_TEST_RUN, {test={prog_fd=105, retval=0, data_size_in=0, data_size_out=0, data_in=NULL, data_out=NULL, repeat=0, duration=0, ctx_size_in=0, ctx_size_out=0, ctx_in=NULL, ctx_out=NULL}, ...}, 120) = 0
bpf(BPF_PROG_TEST_RUN, {test={prog_fd=105, retval=0, data_size_in=0, data_size_out=0, data_in=NULL, data_out=NULL, repeat=0, duration=0, ctx_size_in=0, ctx_size_out=0, ctx_in=NULL, ctx_out=NULL}, ...}, 120) = 0
bpf(BPF_PROG_TEST_RUN, {test={prog_fd=105, retval=0, data_size_in=0, data_size_out=0, data_in=NULL, data_out=NULL, repeat=0, duration=0, ctx_size_in=0, ctx_size_out=0, ctx_in=NULL, ctx_out=NULL}, ...}, 120) = 0
bpf(BPF_PROG_TEST_RUN, {test={prog_fd=105, retval=0, data_size_in=0, data_size_out=0, data_in=NULL, data_out=NULL, repeat=0, duration=0, ctx_size_in=0, ctx_size_out=0, ctx_in=NULL, ctx_out=NULL}, ...}, 120) = 0
bpf(BPF_PROG_TEST_RUN, {test={prog_fd=105, retval=0, data_size_in=0, data_size_out=0, data_in=NULL, data_out=NULL, repeat=0, duration=0, ctx_size_in=0, ctx_size_out=0, ctx_in=NULL, ctx_out=NULL}, ...}, 120) = 0
bpf(BPF_PROG_TEST_RUN, {test={prog_fd=105, retval=0, data_size_in=0, data_size_out=0, data_in=NULL, data_out=NULL, repeat=0, duration=0, ctx_size_in=0, ctx_size_out=0, ctx_in=NULL, ctx_out=NULL}, ...}, 120) = 0
bpf(BPF_PROG_TEST_RUN, {test={prog_fd=105, retval=0, data_size_in=0, data_size_out=0, data_in=NULL, data_out=NULL, repeat=0, duration=0, ctx_size_in=0, ctx_size_out=0, ctx_in=NULL, ctx_out=NULL}, ...}, 120) = 0
bpf(BPF_PROG_TEST_RUN, {test={prog_fd=105, retval=0, data_size_in=0, data_size_out=0, data_in=NULL, data_out=NULL, repeat=0, duration=0, ctx_size_in=0, ctx_size_out=0, ctx_in=NULL, ctx_out=NULL}, ...}, 120) = 0
bpf(BPF_PROG_TEST_RUN, {test={prog_fd=105, retval=0, data_size_in=0, data_size_out=0, data_in=NULL, data_out=NULL, repeat=0, duration=0, ctx_size_in=0, ctx_size_out=0, ctx_in=NULL, ctx_out=NULL}, ...}, 120) = 0
bpf(BPF_PROG_TEST_RUN, {test={prog_fd=105, retval=0, data_size_in=0, data_size_out=0, data_in=NULL, data_out=NULL, repeat=0, duration=0, ctx_size_in=0, ctx_size_out=0, ctx_in=NULL, ctx_out=NULL}, ...}, 120) = 0
bpf(BPF_PROG_TEST_RUN, {test={prog_fd=105, retval=0, data_size_in=0, data_size_out=0, data_in=NULL, data_out=NULL, repeat=0, duration=0, ctx_size_in=0, ctx_size_out=0, ctx_in=NULL, ctx_out=NULL}, ...}, 120) = 0
bpf(BPF_PROG_TEST_RUN, {test={prog_fd=105, retval=0, data_size_in=0, data_size_out=0, data_in=NULL, data_out=NULL, repeat=0, duration=0, ctx_size_in=0, ctx_size_out=0, ctx_in=NULL, ctx_out=NULL}, ...}, 120) = 0
bpf(BPF_PROG_TEST_RUN, {test={prog_fd=105, retval=0, data_size_in=0, data_size_out=0, data_in=NULL, data_out=NULL, repeat=0, duration=0, ctx_size_in=0, ctx_size_out=0, ctx_in=NULL, ctx_out=NULL}, ...}, 120) = 0
bpf(BPF_PROG_TEST_RUN, {test={prog_fd=105, retval=0, data_size_in=0, data_size_out=0, data_in=NULL, data_out=NULL, repeat=0, duration=0, ctx_size_in=0, ctx_size_out=0, ctx_in=NULL, ctx_out=NULL}, ...}, 120) = 0
bpf(BPF_PROG_TEST_RUN, {test={prog_fd=105, retval=0, data_size_in=0, data_size_out=0, data_in=NULL, data_out=NULL, repeat=0, duration=0, ctx_size_in=0, ctx_size_out=0, ctx_in=NULL, ctx_out=NULL}, ...}, 120) = 0
bpf(BPF_PROG_TEST_RUN, {test={prog_fd=105, retval=0, data_size_in=0, data_size_out=0, data_in=NULL, data_out=NULL, repeat=0, duration=0, ctx_size_in=0, ctx_size_out=0, ctx_in=NULL, ctx_out=NULL}, ...}, 120) = 0
bpf(BPF_PROG_TEST_RUN, {test={prog_fd=105, retval=0, data_size_in=0, data_size_out=0, data_in=NULL, data_out=NULL, repeat=0, duration=0, ctx_size_in=0, ctx_size_out=0, ctx_in=NULL, ctx_out=NULL}, ...}, 120) = 0
bpf(BPF_PROG_TEST_RUN, {test={prog_fd=105, retval=0, data_size_in=0, data_size_out=0, data_in=NULL, data_out=NULL, repeat=0, duration=0, ctx_size_in=0, ctx_size_out=0, ctx_in=NULL, ctx_out=NULL}, ...}, 120) = 0
bpf(BPF_MAP_LOOKUP_ELEM, {map_fd=141, key=0x7ffda3255160, value=0x7ffda3254f10, flags=BPF_ANY}, 120) = 0
bpf(BPF_PROG_TEST_RUN, {test={prog_fd=138, retval=0, data_size_in=0, data_size_out=0, data_in=NULL, data_out=NULL, repeat=0, duration=0, ctx_size_in=0, ctx_size_out=0, ctx_in=NULL, ctx_out=NULL}, ...}, 120) = 0
bpf(BPF_PROG_TEST_RUN, {test={prog_fd=138, retval=0, data_size_in=0, data_size_out=0, data_in=NULL, data_out=NULL, repeat=0, duration=0, ctx_size_in=0, ctx_size_out=0, ctx_in=NULL, ctx_out=NULL}, ...}, 120) = 0
bpf(BPF_PROG_TEST_RUN, {test={prog_fd=138, retval=0, data_size_in=0, data_size_out=0, data_in=NULL, data_out=NULL, repeat=0, duration=0, ctx_size_in=0, ctx_size_out=0, ctx_in=NULL, ctx_out=NULL}, ...}, 120) = 0
bpf(BPF_PROG_TEST_RUN, {test={prog_fd=138, retval=0, data_size_in=0, data_size_out=0, data_in=NULL, data_out=NULL, repeat=0, duration=0, ctx_size_in=0, ctx_size_out=0, ctx_in=NULL, ctx_out=NULL}, ...}, 120) = 0
bpf(BPF_PROG_TEST_RUN, {test={prog_fd=138, retval=0, data_size_in=0, data_size_out=0, data_in=NULL, data_out=NULL, repeat=0, duration=0, ctx_size_in=0, ctx_size_out=0, ctx_in=NULL, ctx_out=NULL}, ...}, 120) = 0
bpf(BPF_PROG_TEST_RUN, {test={prog_fd=138, retval=0, data_size_in=0, data_size_out=0, data_in=NULL, data_out=NULL, repeat=0, duration=0, ctx_size_in=0, ctx_size_out=0, ctx_in=NULL, ctx_out=NULL}, ...}, 120) = 0
bpf(BPF_PROG_TEST_RUN, {test={prog_fd=138, retval=0, data_size_in=0, data_size_out=0, data_in=NULL, data_out=NULL, repeat=0, duration=0, ctx_size_in=0, ctx_size_out=0, ctx_in=NULL, ctx_out=NULL}, ...}, 120) = 0
bpf(BPF_PROG_TEST_RUN, {test={prog_fd=138, retval=0, data_size_in=0, data_size_out=0, data_in=NULL, data_out=NULL, repeat=0, duration=0, ctx_size_in=0, ctx_size_out=0, ctx_in=NULL, ctx_out=NULL}, ...}, 120) = 0
bpf(BPF_PROG_TEST_RUN, {test={prog_fd=138, retval=0, data_size_in=0, data_size_out=0, data_in=NULL, data_out=NULL, repeat=0, duration=0, ctx_size_in=0, ctx_size_out=0, ctx_in=NULL, ctx_out=NULL}, ...}, 120) = 0
bpf(BPF_PROG_TEST_RUN, {test={prog_fd=138, retval=0, data_size_in=0, data_size_out=0, data_in=NULL, data_out=NULL, repeat=0, duration=0, ctx_size_in=0, ctx_size_out=0, ctx_in=NULL, ctx_out=NULL}, ...}, 120) = 0
bpf(BPF_PROG_TEST_RUN, {test={prog_fd=138, retval=0, data_size_in=0, data_size_out=0, data_in=NULL, data_out=NULL, repeat=0, duration=0, ctx_size_in=0, ctx_size_out=0, ctx_in=NULL, ctx_out=NULL}, ...}, 120) = 0
bpf(BPF_PROG_TEST_RUN, {test={prog_fd=138, retval=0, data_size_in=0, data_size_out=0, data_in=NULL, data_out=NULL, repeat=0, duration=0, ctx_size_in=0, ctx_size_out=0, ctx_in=NULL, ctx_out=NULL}, ...}, 120) = 0
bpf(BPF_PROG_TEST_RUN, {test={prog_fd=138, retval=0, data_size_in=0, data_size_out=0, data_in=NULL, data_out=NULL, repeat=0, duration=0, ctx_size_in=0, ctx_size_out=0, ctx_in=NULL, ctx_out=NULL}, ...}, 120) = 0
bpf(BPF_PROG_TEST_RUN, {test={prog_fd=138, retval=0, data_size_in=0, data_size_out=0, data_in=NULL, data_out=NULL, repeat=0, duration=0, ctx_size_in=0, ctx_size_out=0, ctx_in=NULL, ctx_out=NULL}, ...}, 120) = 0
bpf(BPF_PROG_TEST_RUN, {test={prog_fd=138, retval=0, data_size_in=0, data_size_out=0, data_in=NULL, data_out=NULL, repeat=0, duration=0, ctx_size_in=0, ctx_size_out=0, ctx_in=NULL, ctx_out=NULL}, ...}, 120) = 0
bpf(BPF_PROG_TEST_RUN, {test={prog_fd=138, retval=0, data_size_in=0, data_size_out=0, data_in=NULL, data_out=NULL, repeat=0, duration=0, ctx_size_in=0, ctx_size_out=0, ctx_in=NULL, ctx_out=NULL}, ...}, 120) = 0
bpf(BPF_PROG_TEST_RUN, {test={prog_fd=138, retval=0, data_size_in=0, data_size_out=0, data_in=NULL, data_out=NULL, repeat=0, duration=0, ctx_size_in=0, ctx_size_out=0, ctx_in=NULL, ctx_out=NULL}, ...}, 120) = 0
bpf(BPF_PROG_TEST_RUN, {test={prog_fd=138, retval=0, data_size_in=0, data_size_out=0, data_in=NULL, data_out=NULL, repeat=0, duration=0, ctx_size_in=0, ctx_size_out=0, ctx_in=NULL, ctx_out=NULL}, ...}, 120) = 0
bpf(BPF_PROG_TEST_RUN, {test={prog_fd=138, retval=0, data_size_in=0, data_size_out=0, data_in=NULL, data_out=NULL, repeat=0, duration=0, ctx_size_in=0, ctx_size_out=0, ctx_in=NULL, ctx_out=NULL}, ...}, 120) = 0
bpf(BPF_PROG_TEST_RUN, {test={prog_fd=138, retval=0, data_size_in=0, data_size_out=0, data_in=NULL, data_out=NULL, repeat=0, duration=0, ctx_size_in=0, ctx_size_out=0, ctx_in=NULL, ctx_out=NULL}, ...}, 120) = 0
bpf(BPF_PROG_TEST_RUN, {test={prog_fd=138, retval=0, data_size_in=0, data_size_out=0, data_in=NULL, data_out=NULL, repeat=0, duration=0, ctx_size_in=0, ctx_size_out=0, ctx_in=NULL, ctx_out=NULL}, ...}, 120) = 0
bpf(BPF_PROG_TEST_RUN, {test={prog_fd=138, retval=0, data_size_in=0, data_size_out=0, data_in=NULL, data_out=NULL, repeat=0, duration=0, ctx_size_in=0, ctx_size_out=0, ctx_in=NULL, ctx_out=NULL}, ...}, 120) = 0
bpf(BPF_PROG_TEST_RUN, {test={prog_fd=138, retval=0, data_size_in=0, data_size_out=0, data_in=NULL, data_out=NULL, repeat=0, duration=0, ctx_size_in=0, ctx_size_out=0, ctx_in=NULL, ctx_out=NULL}, ...}, 120) = 0
bpf(BPF_PROG_TEST_RUN, {test={prog_fd=138, retval=0, data_size_in=0, data_size_out=0, data_in=NULL, data_out=NULL, repeat=0, duration=0, ctx_size_in=0, ctx_size_out=0, ctx_in=NULL, ctx_out=NULL}, ...}, 120) = 0
bpf(BPF_MAP_LOOKUP_ELEM, {map_fd=174, key=0x7ffda3255160, value=0x7ffda3254f10, flags=BPF_ANY}, 120) = 0
bpf(BPF_PROG_TEST_RUN, {test={prog_fd=171, retval=0, data_size_in=0, data_size_out=0, data_in=NULL, data_out=NULL, repeat=0, duration=0, ctx_size_in=0, ctx_size_out=0, ctx_in=NULL, ctx_out=NULL}, ...}, 120) = 0
bpf(BPF_PROG_TEST_RUN, {test={prog_fd=171, retval=0, data_size_in=0, data_size_out=0, data_in=NULL, data_out=NULL, repeat=0, duration=0, ctx_size_in=0, ctx_size_out=0, ctx_in=NULL, ctx_out=NULL}, ...}, 120) = 0
bpf(BPF_PROG_TEST_RUN, {test={prog_fd=171, retval=0, data_size_in=0, data_size_out=0, data_in=NULL, data_out=NULL, repeat=0, duration=0, ctx_size_in=0, ctx_size_out=0, ctx_in=NULL, ctx_out=NULL}, ...}, 120) = 0
bpf(BPF_PROG_TEST_RUN, {test={prog_fd=171, retval=0, data_size_in=0, data_size_out=0, data_in=NULL, data_out=NULL, repeat=0, duration=0, ctx_size_in=0, ctx_size_out=0, ctx_in=NULL, ctx_out=NULL}, ...}, 120) = 0
bpf(BPF_PROG_TEST_RUN, {test={prog_fd=171, retval=0, data_size_in=0, data_size_out=0, data_in=NULL, data_out=NULL, repeat=0, duration=0, ctx_size_in=0, ctx_size_out=0, ctx_in=NULL, ctx_out=NULL}, ...}, 120) = 0
bpf(BPF_PROG_TEST_RUN, {test={prog_fd=171, retval=0, data_size_in=0, data_size_out=0, data_in=NULL, data_out=NULL, repeat=0, duration=0, ctx_size_in=0, ctx_size_out=0, ctx_in=NULL, ctx_out=NULL}, ...}, 120) = 0
bpf(BPF_PROG_TEST_RUN, {test={prog_fd=171, retval=0, data_size_in=0, data_size_out=0, data_in=NULL, data_out=NULL, repeat=0, duration=0, ctx_size_in=0, ctx_size_out=0, ctx_in=NULL, ctx_out=NULL}, ...}, 120) = 0
bpf(BPF_PROG_TEST_RUN, {test={prog_fd=171, retval=0, data_size_in=0, data_size_out=0, data_in=NULL, data_out=NULL, repeat=0, duration=0, ctx_size_in=0, ctx_size_out=0, ctx_in=NULL, ctx_out=NULL}, ...}, 120) = 0
bpf(BPF_PROG_TEST_RUN, {test={prog_fd=171, retval=0, data_size_in=0, data_size_out=0, data_in=NULL, data_out=NULL, repeat=0, duration=0, ctx_size_in=0, ctx_size_out=0, ctx_in=NULL, ctx_out=NULL}, ...}, 120) = 0
bpf(BPF_PROG_TEST_RUN, {test={prog_fd=171, retval=0, data_size_in=0, data_size_out=0, data_in=NULL, data_out=NULL, repeat=0, duration=0, ctx_size_in=0, ctx_size_out=0, ctx_in=NULL, ctx_out=NULL}, ...}, 120) = 0
bpf(BPF_PROG_TEST_RUN, {test={prog_fd=171, retval=0, data_size_in=0, data_size_out=0, data_in=NULL, data_out=NULL, repeat=0, duration=0, ctx_size_in=0, ctx_size_out=0, ctx_in=NULL, ctx_out=NULL}, ...}, 120) = 0
bpf(BPF_PROG_TEST_RUN, {test={prog_fd=171, retval=0, data_size_in=0, data_size_out=0, data_in=NULL, data_out=NULL, repeat=0, duration=0, ctx_size_in=0, ctx_size_out=0, ctx_in=NULL, ctx_out=NULL}, ...}, 120) = 0
bpf(BPF_PROG_TEST_RUN, {test={prog_fd=171, retval=0, data_size_in=0, data_size_out=0, data_in=NULL, data_out=NULL, repeat=0, duration=0, ctx_size_in=0, ctx_size_out=0, ctx_in=NULL, ctx_out=NULL}, ...}, 120) = 0
bpf(BPF_PROG_TEST_RUN, {test={prog_fd=171, retval=0, data_size_in=0, data_size_out=0, data_in=NULL, data_out=NULL, repeat=0, duration=0, ctx_size_in=0, ctx_size_out=0, ctx_in=NULL, ctx_out=NULL}, ...}, 120) = 0
bpf(BPF_PROG_TEST_RUN, {test={prog_fd=171, retval=0, data_size_in=0, data_size_out=0, data_in=NULL, data_out=NULL, repeat=0, duration=0, ctx_size_in=0, ctx_size_out=0, ctx_in=NULL, ctx_out=NULL}, ...}, 120) = 0
bpf(BPF_PROG_TEST_RUN, {test={prog_fd=171, retval=0, data_size_in=0, data_size_out=0, data_in=NULL, data_out=NULL, repeat=0, duration=0, ctx_size_in=0, ctx_size_out=0, ctx_in=NULL, ctx_out=NULL}, ...}, 120) = 0
bpf(BPF_PROG_TEST_RUN, {test={prog_fd=171, retval=0, data_size_in=0, data_size_out=0, data_in=NULL, data_out=NULL, repeat=0, duration=0, ctx_size_in=0, ctx_size_out=0, ctx_in=NULL, ctx_out=NULL}, ...}, 120) = 0
bpf(BPF_PROG_TEST_RUN, {test={prog_fd=171, retval=0, data_size_in=0, data_size_out=0, data_in=NULL, data_out=NULL, repeat=0, duration=0, ctx_size_in=0, ctx_size_out=0, ctx_in=NULL, ctx_out=NULL}, ...}, 120) = 0
bpf(BPF_PROG_TEST_RUN, {test={prog_fd=171, retval=0, data_size_in=0, data_size_out=0, data_in=NULL, data_out=NULL, repeat=0, duration=0, ctx_size_in=0, ctx_size_out=0, ctx_in=NULL, ctx_out=NULL}, ...}, 120) = 0
bpf(BPF_PROG_TEST_RUN, {test={prog_fd=171, retval=0, data_size_in=0, data_size_out=0, data_in=NULL, data_out=NULL, repeat=0, duration=0, ctx_size_in=0, ctx_size_out=0, ctx_in=NULL, ctx_out=NULL}, ...}, 120) = 0
bpf(BPF_PROG_TEST_RUN, {test={prog_fd=171, retval=0, data_size_in=0, data_size_out=0, data_in=NULL, data_out=NULL, repeat=0, duration=0, ctx_size_in=0, ctx_size_out=0, ctx_in=NULL, ctx_out=NULL}, ...}, 120) = 0
bpf(BPF_PROG_TEST_RUN, {test={prog_fd=171, retval=0, data_size_in=0, data_size_out=0, data_in=NULL, data_out=NULL, repeat=0, duration=0, ctx_size_in=0, ctx_size_out=0, ctx_in=NULL, ctx_out=NULL}, ...}, 120) = 0
bpf(BPF_PROG_TEST_RUN, {test={prog_fd=171, retval=0, data_size_in=0, data_size_out=0, data_in=NULL, data_out=NULL, repeat=0, duration=0, ctx_size_in=0, ctx_size_out=0, ctx_in=NULL, ctx_out=NULL}, ...}, 120) = 0
bpf(BPF_PROG_TEST_RUN, {test={prog_fd=171, retval=0, data_size_in=0, data_size_out=0, data_in=NULL, data_out=NULL, repeat=0, duration=0, ctx_size_in=0, ctx_size_out=0, ctx_in=NULL, ctx_out=NULL}, ...}, 120) = 0
bpf(BPF_MAP_LOOKUP_ELEM, {map_fd=207, key=0x7ffda3255160, value=0x7ffda3254f10, flags=BPF_ANY}, 120) = 0
bpf(BPF_PROG_TEST_RUN, {test={prog_fd=204, retval=0, data_size_in=0, data_size_out=0, data_in=NULL, data_out=NULL, repeat=0, duration=0, ctx_size_in=0, ctx_size_out=0, ctx_in=NULL, ctx_out=NULL}, ...}, 120) = 0
bpf(BPF_PROG_TEST_RUN, {test={prog_fd=204, retval=0, data_size_in=0, data_size_out=0, data_in=NULL, data_out=NULL, repeat=0, duration=0, ctx_size_in=0, ctx_size_out=0, ctx_in=NULL, ctx_out=NULL}, ...}, 120) = 0
bpf(BPF_PROG_TEST_RUN, {test={prog_fd=204, retval=0, data_size_in=0, data_size_out=0, data_in=NULL, data_out=NULL, repeat=0, duration=0, ctx_size_in=0, ctx_size_out=0, ctx_in=NULL, ctx_out=NULL}, ...}, 120) = 0
bpf(BPF_PROG_TEST_RUN, {test={prog_fd=204, retval=0, data_size_in=0, data_size_out=0, data_in=NULL, data_out=NULL, repeat=0, duration=0, ctx_size_in=0, ctx_size_out=0, ctx_in=NULL, ctx_out=NULL}, ...}, 120) = 0
bpf(BPF_PROG_TEST_RUN, {test={prog_fd=204, retval=0, data_size_in=0, data_size_out=0, data_in=NULL, data_out=NULL, repeat=0, duration=0, ctx_size_in=0, ctx_size_out=0, ctx_in=NULL, ctx_out=NULL}, ...}, 120) = 0
bpf(BPF_PROG_TEST_RUN, {test={prog_fd=204, retval=0, data_size_in=0, data_size_out=0, data_in=NULL, data_out=NULL, repeat=0, duration=0, ctx_size_in=0, ctx_size_out=0, ctx_in=NULL, ctx_out=NULL}, ...}, 120) = 0
bpf(BPF_PROG_TEST_RUN, {test={prog_fd=204, retval=0, data_size_in=0, data_size_out=0, data_in=NULL, data_out=NULL, repeat=0, duration=0, ctx_size_in=0, ctx_size_out=0, ctx_in=NULL, ctx_out=NULL}, ...}, 120) = 0
bpf(BPF_PROG_TEST_RUN, {test={prog_fd=204, retval=0, data_size_in=0, data_size_out=0, data_in=NULL, data_out=NULL, repeat=0, duration=0, ctx_size_in=0, ctx_size_out=0, ctx_in=NULL, ctx_out=NULL}, ...}, 120) = 0
bpf(BPF_PROG_TEST_RUN, {test={prog_fd=204, retval=0, data_size_in=0, data_size_out=0, data_in=NULL, data_out=NULL, repeat=0, duration=0, ctx_size_in=0, ctx_size_out=0, ctx_in=NULL, ctx_out=NULL}, ...}, 120) = 0
bpf(BPF_PROG_TEST_RUN, {test={prog_fd=204, retval=0, data_size_in=0, data_size_out=0, data_in=NULL, data_out=NULL, repeat=0, duration=0, ctx_size_in=0, ctx_size_out=0, ctx_in=NULL, ctx_out=NULL}, ...}, 120) = 0
bpf(BPF_PROG_TEST_RUN, {test={prog_fd=204, retval=0, data_size_in=0, data_size_out=0, data_in=NULL, data_out=NULL, repeat=0, duration=0, ctx_size_in=0, ctx_size_out=0, ctx_in=NULL, ctx_out=NULL}, ...}, 120) = 0
bpf(BPF_PROG_TEST_RUN, {test={prog_fd=204, retval=0, data_size_in=0, data_size_out=0, data_in=NULL, data_out=NULL, repeat=0, duration=0, ctx_size_in=0, ctx_size_out=0, ctx_in=NULL, ctx_out=NULL}, ...}, 120) = 0
bpf(BPF_PROG_TEST_RUN, {test={prog_fd=204, retval=0, data_size_in=0, data_size_out=0, data_in=NULL, data_out=NULL, repeat=0, duration=0, ctx_size_in=0, ctx_size_out=0, ctx_in=NULL, ctx_out=NULL}, ...}, 120) = 0
bpf(BPF_PROG_TEST_RUN, {test={prog_fd=204, retval=0, data_size_in=0, data_size_out=0, data_in=NULL, data_out=NULL, repeat=0, duration=0, ctx_size_in=0, ctx_size_out=0, ctx_in=NULL, ctx_out=NULL}, ...}, 120) = 0
bpf(BPF_PROG_TEST_RUN, {test={prog_fd=204, retval=0, data_size_in=0, data_size_out=0, data_in=NULL, data_out=NULL, repeat=0, duration=0, ctx_size_in=0, ctx_size_out=0, ctx_in=NULL, ctx_out=NULL}, ...}, 120) = 0
bpf(BPF_PROG_TEST_RUN, {test={prog_fd=204, retval=0, data_size_in=0, data_size_out=0, data_in=NULL, data_out=NULL, repeat=0, duration=0, ctx_size_in=0, ctx_size_out=0, ctx_in=NULL, ctx_out=NULL}, ...}, 120) = 0
bpf(BPF_PROG_TEST_RUN, {test={prog_fd=204, retval=0, data_size_in=0, data_size_out=0, data_in=NULL, data_out=NULL, repeat=0, duration=0, ctx_size_in=0, ctx_size_out=0, ctx_in=NULL, ctx_out=NULL}, ...}, 120) = 0
bpf(BPF_PROG_TEST_RUN, {test={prog_fd=204, retval=0, data_size_in=0, data_size_out=0, data_in=NULL, data_out=NULL, repeat=0, duration=0, ctx_size_in=0, ctx_size_out=0, ctx_in=NULL, ctx_out=NULL}, ...}, 120) = 0
bpf(BPF_PROG_TEST_RUN, {test={prog_fd=204, retval=0, data_size_in=0, data_size_out=0, data_in=NULL, data_out=NULL, repeat=0, duration=0, ctx_size_in=0, ctx_size_out=0, ctx_in=NULL, ctx_out=NULL}, ...}, 120) = 0
bpf(BPF_PROG_TEST_RUN, {test={prog_fd=204, retval=0, data_size_in=0, data_size_out=0, data_in=NULL, data_out=NULL, repeat=0, duration=0, ctx_size_in=0, ctx_size_out=0, ctx_in=NULL, ctx_out=NULL}, ...}, 120) = 0
bpf(BPF_PROG_TEST_RUN, {test={prog_fd=204, retval=0, data_size_in=0, data_size_out=0, data_in=NULL, data_out=NULL, repeat=0, duration=0, ctx_size_in=0, ctx_size_out=0, ctx_in=NULL, ctx_out=NULL}, ...}, 120) = 0
bpf(BPF_PROG_TEST_RUN, {test={prog_fd=204, retval=0, data_size_in=0, data_size_out=0, data_in=NULL, data_out=NULL, repeat=0, duration=0, ctx_size_in=0, ctx_size_out=0, ctx_in=NULL, ctx_out=NULL}, ...}, 120) = 0
bpf(BPF_PROG_TEST_RUN, {test={prog_fd=204, retval=0, data_size_in=0, data_size_out=0, data_in=NULL, data_out=NULL, repeat=0, duration=0, ctx_size_in=0, ctx_size_out=0, ctx_in=NULL, ctx_out=NULL}, ...}, 120) = 0
bpf(BPF_PROG_TEST_RUN, {test={prog_fd=204, retval=0, data_size_in=0, data_size_out=0, data_in=NULL, data_out=NULL, repeat=0, duration=0, ctx_size_in=0, ctx_size_out=0, ctx_in=NULL, ctx_out=NULL}, ...}, 120) = 0
bpf(BPF_MAP_LOOKUP_ELEM, {map_fd=240, key=0x7ffda3255160, value=0x7ffda3254f10, flags=BPF_ANY}, 120) = 0
bpf(BPF_PROG_TEST_RUN, {test={prog_fd=237, retval=0, data_size_in=0, data_size_out=0, data_in=NULL, data_out=NULL, repeat=0, duration=0, ctx_size_in=0, ctx_size_out=0, ctx_in=NULL, ctx_out=NULL}, ...}, 120) = 0
bpf(BPF_PROG_TEST_RUN, {test={prog_fd=237, retval=0, data_size_in=0, data_size_out=0, data_in=NULL, data_out=NULL, repeat=0, duration=0, ctx_size_in=0, ctx_size_out=0, ctx_in=NULL, ctx_out=NULL}, ...}, 120) = 0
bpf(BPF_PROG_TEST_RUN, {test={prog_fd=237, retval=0, data_size_in=0, data_size_out=0, data_in=NULL, data_out=NULL, repeat=0, duration=0, ctx_size_in=0, ctx_size_out=0, ctx_in=NULL, ctx_out=NULL}, ...}, 120) = 0
bpf(BPF_PROG_TEST_RUN, {test={prog_fd=237, retval=0, data_size_in=0, data_size_out=0, data_in=NULL, data_out=NULL, repeat=0, duration=0, ctx_size_in=0, ctx_size_out=0, ctx_in=NULL, ctx_out=NULL}, ...}, 120) = 0
bpf(BPF_PROG_TEST_RUN, {test={prog_fd=237, retval=0, data_size_in=0, data_size_out=0, data_in=NULL, data_out=NULL, repeat=0, duration=0, ctx_size_in=0, ctx_size_out=0, ctx_in=NULL, ctx_out=NULL}, ...}, 120) = 0
bpf(BPF_PROG_TEST_RUN, {test={prog_fd=237, retval=0, data_size_in=0, data_size_out=0, data_in=NULL, data_out=NULL, repeat=0, duration=0, ctx_size_in=0, ctx_size_out=0, ctx_in=NULL, ctx_out=NULL}, ...}, 120) = 0
bpf(BPF_PROG_TEST_RUN, {test={prog_fd=237, retval=0, data_size_in=0, data_size_out=0, data_in=NULL, data_out=NULL, repeat=0, duration=0, ctx_size_in=0, ctx_size_out=0, ctx_in=NULL, ctx_out=NULL}, ...}, 120) = 0
bpf(BPF_PROG_TEST_RUN, {test={prog_fd=237, retval=0, data_size_in=0, data_size_out=0, data_in=NULL, data_out=NULL, repeat=0, duration=0, ctx_size_in=0, ctx_size_out=0, ctx_in=NULL, ctx_out=NULL}, ...}, 120) = 0
bpf(BPF_PROG_TEST_RUN, {test={prog_fd=237, retval=0, data_size_in=0, data_size_out=0, data_in=NULL, data_out=NULL, repeat=0, duration=0, ctx_size_in=0, ctx_size_out=0, ctx_in=NULL, ctx_out=NULL}, ...}, 120) = 0
bpf(BPF_PROG_TEST_RUN, {test={prog_fd=237, retval=0, data_size_in=0, data_size_out=0, data_in=NULL, data_out=NULL, repeat=0, duration=0, ctx_size_in=0, ctx_size_out=0, ctx_in=NULL, ctx_out=NULL}, ...}, 120) = 0
bpf(BPF_PROG_TEST_RUN, {test={prog_fd=237, retval=0, data_size_in=0, data_size_out=0, data_in=NULL, data_out=NULL, repeat=0, duration=0, ctx_size_in=0, ctx_size_out=0, ctx_in=NULL, ctx_out=NULL}, ...}, 120) = 0
bpf(BPF_PROG_TEST_RUN, {test={prog_fd=237, retval=0, data_size_in=0, data_size_out=0, data_in=NULL, data_out=NULL, repeat=0, duration=0, ctx_size_in=0, ctx_size_out=0, ctx_in=NULL, ctx_out=NULL}, ...}, 120) = 0
bpf(BPF_PROG_TEST_RUN, {test={prog_fd=237, retval=0, data_size_in=0, data_size_out=0, data_in=NULL, data_out=NULL, repeat=0, duration=0, ctx_size_in=0, ctx_size_out=0, ctx_in=NULL, ctx_out=NULL}, ...}, 120) = 0
bpf(BPF_PROG_TEST_RUN, {test={prog_fd=237, retval=0, data_size_in=0, data_size_out=0, data_in=NULL, data_out=NULL, repeat=0, duration=0, ctx_size_in=0, ctx_size_out=0, ctx_in=NULL, ctx_out=NULL}, ...}, 120) = 0
bpf(BPF_PROG_TEST_RUN, {test={prog_fd=237, retval=0, data_size_in=0, data_size_out=0, data_in=NULL, data_out=NULL, repeat=0, duration=0, ctx_size_in=0, ctx_size_out=0, ctx_in=NULL, ctx_out=NULL}, ...}, 120) = 0
bpf(BPF_PROG_TEST_RUN, {test={prog_fd=237, retval=0, data_size_in=0, data_size_out=0, data_in=NULL, data_out=NULL, repeat=0, duration=0, ctx_size_in=0, ctx_size_out=0, ctx_in=NULL, ctx_out=NULL}, ...}, 120) = 0
bpf(BPF_PROG_TEST_RUN, {test={prog_fd=237, retval=0, data_size_in=0, data_size_out=0, data_in=NULL, data_out=NULL, repeat=0, duration=0, ctx_size_in=0, ctx_size_out=0, ctx_in=NULL, ctx_out=NULL}, ...}, 120) = 0
bpf(BPF_PROG_TEST_RUN, {test={prog_fd=237, retval=0, data_size_in=0, data_size_out=0, data_in=NULL, data_out=NULL, repeat=0, duration=0, ctx_size_in=0, ctx_size_out=0, ctx_in=NULL, ctx_out=NULL}, ...}, 120) = 0
bpf(BPF_PROG_TEST_RUN, {test={prog_fd=237, retval=0, data_size_in=0, data_size_out=0, data_in=NULL, data_out=NULL, repeat=0, duration=0, ctx_size_in=0, ctx_size_out=0, ctx_in=NULL, ctx_out=NULL}, ...}, 120) = 0
bpf(BPF_PROG_TEST_RUN, {test={prog_fd=237, retval=0, data_size_in=0, data_size_out=0, data_in=NULL, data_out=NULL, repeat=0, duration=0, ctx_size_in=0, ctx_size_out=0, ctx_in=NULL, ctx_out=NULL}, ...}, 120) = 0
bpf(BPF_PROG_TEST_RUN, {test={prog_fd=237, retval=0, data_size_in=0, data_size_out=0, data_in=NULL, data_out=NULL, repeat=0, duration=0, ctx_size_in=0, ctx_size_out=0, ctx_in=NULL, ctx_out=NULL}, ...}, 120) = 0
bpf(BPF_PROG_TEST_RUN, {test={prog_fd=237, retval=0, data_size_in=0, data_size_out=0, data_in=NULL, data_out=NULL, repeat=0, duration=0, ctx_size_in=0, ctx_size_out=0, ctx_in=NULL, ctx_out=NULL}, ...}, 120) = 0
bpf(BPF_PROG_TEST_RUN, {test={prog_fd=237, retval=0, data_size_in=0, data_size_out=0, data_in=NULL, data_out=NULL, repeat=0, duration=0, ctx_size_in=0, ctx_size_out=0, ctx_in=NULL, ctx_out=NULL}, ...}, 120) = 0
bpf(BPF_PROG_TEST_RUN, {test={prog_fd=237, retval=0, data_size_in=0, data_size_out=0, data_in=NULL, data_out=NULL, repeat=0, duration=0, ctx_size_in=0, ctx_size_out=0, ctx_in=NULL, ctx_out=NULL}, ...}, 120) = 0
bpf(BPF_MAP_LOOKUP_ELEM, {map_fd=273, key=0x7ffda3255160, value=0x7ffda3254f10, flags=BPF_ANY}, 120) = 0
bpf(BPF_PROG_TEST_RUN, {test={prog_fd=270, retval=0, data_size_in=0, data_size_out=0, data_in=NULL, data_out=NULL, repeat=0, duration=0, ctx_size_in=0, ctx_size_out=0, ctx_in=NULL, ctx_out=NULL}, ...}, 120) = 0
bpf(BPF_PROG_TEST_RUN, {test={prog_fd=270, retval=0, data_size_in=0, data_size_out=0, data_in=NULL, data_out=NULL, repeat=0, duration=0, ctx_size_in=0, ctx_size_out=0, ctx_in=NULL, ctx_out=NULL}, ...}, 120) = 0
bpf(BPF_PROG_TEST_RUN, {test={prog_fd=270, retval=0, data_size_in=0, data_size_out=0, data_in=NULL, data_out=NULL, repeat=0, duration=0, ctx_size_in=0, ctx_size_out=0, ctx_in=NULL, ctx_out=NULL}, ...}, 120) = 0
bpf(BPF_PROG_TEST_RUN, {test={prog_fd=270, retval=0, data_size_in=0, data_size_out=0, data_in=NULL, data_out=NULL, repeat=0, duration=0, ctx_size_in=0, ctx_size_out=0, ctx_in=NULL, ctx_out=NULL}, ...}, 120) = 0
bpf(BPF_PROG_TEST_RUN, {test={prog_fd=270, retval=0, data_size_in=0, data_size_out=0, data_in=NULL, data_out=NULL, repeat=0, duration=0, ctx_size_in=0, ctx_size_out=0, ctx_in=NULL, ctx_out=NULL}, ...}, 120) = 0
bpf(BPF_PROG_TEST_RUN, {test={prog_fd=270, retval=0, data_size_in=0, data_size_out=0, data_in=NULL, data_out=NULL, repeat=0, duration=0, ctx_size_in=0, ctx_size_out=0, ctx_in=NULL, ctx_out=NULL}, ...}, 120) = 0
bpf(BPF_PROG_TEST_RUN, {test={prog_fd=270, retval=0, data_size_in=0, data_size_out=0, data_in=NULL, data_out=NULL, repeat=0, duration=0, ctx_size_in=0, ctx_size_out=0, ctx_in=NULL, ctx_out=NULL}, ...}, 120) = 0
bpf(BPF_PROG_TEST_RUN, {test={prog_fd=270, retval=0, data_size_in=0, data_size_out=0, data_in=NULL, data_out=NULL, repeat=0, duration=0, ctx_size_in=0, ctx_size_out=0, ctx_in=NULL, ctx_out=NULL}, ...}, 120) = 0
bpf(BPF_PROG_TEST_RUN, {test={prog_fd=270, retval=0, data_size_in=0, data_size_out=0, data_in=NULL, data_out=NULL, repeat=0, duration=0, ctx_size_in=0, ctx_size_out=0, ctx_in=NULL, ctx_out=NULL}, ...}, 120) = 0
bpf(BPF_PROG_TEST_RUN, {test={prog_fd=270, retval=0, data_size_in=0, data_size_out=0, data_in=NULL, data_out=NULL, repeat=0, duration=0, ctx_size_in=0, ctx_size_out=0, ctx_in=NULL, ctx_out=NULL}, ...}, 120) = 0
bpf(BPF_PROG_TEST_RUN, {test={prog_fd=270, retval=0, data_size_in=0, data_size_out=0, data_in=NULL, data_out=NULL, repeat=0, duration=0, ctx_size_in=0, ctx_size_out=0, ctx_in=NULL, ctx_out=NULL}, ...}, 120) = 0
bpf(BPF_PROG_TEST_RUN, {test={prog_fd=270, retval=0, data_size_in=0, data_size_out=0, data_in=NULL, data_out=NULL, repeat=0, duration=0, ctx_size_in=0, ctx_size_out=0, ctx_in=NULL, ctx_out=NULL}, ...}, 120) = 0
bpf(BPF_PROG_TEST_RUN, {test={prog_fd=270, retval=0, data_size_in=0, data_size_out=0, data_in=NULL, data_out=NULL, repeat=0, duration=0, ctx_size_in=0, ctx_size_out=0, ctx_in=NULL, ctx_out=NULL}, ...}, 120) = 0
bpf(BPF_PROG_TEST_RUN, {test={prog_fd=270, retval=0, data_size_in=0, data_size_out=0, data_in=NULL, data_out=NULL, repeat=0, duration=0, ctx_size_in=0, ctx_size_out=0, ctx_in=NULL, ctx_out=NULL}, ...}, 120) = 0
bpf(BPF_PROG_TEST_RUN, {test={prog_fd=270, retval=0, data_size_in=0, data_size_out=0, data_in=NULL, data_out=NULL, repeat=0, duration=0, ctx_size_in=0, ctx_size_out=0, ctx_in=NULL, ctx_out=NULL}, ...}, 120) = 0
bpf(BPF_PROG_TEST_RUN, {test={prog_fd=270, retval=0, data_size_in=0, data_size_out=0, data_in=NULL, data_out=NULL, repeat=0, duration=0, ctx_size_in=0, ctx_size_out=0, ctx_in=NULL, ctx_out=NULL}, ...}, 120) = 0
bpf(BPF_PROG_TEST_RUN, {test={prog_fd=270, retval=0, data_size_in=0, data_size_out=0, data_in=NULL, data_out=NULL, repeat=0, duration=0, ctx_size_in=0, ctx_size_out=0, ctx_in=NULL, ctx_out=NULL}, ...}, 120) = 0
bpf(BPF_PROG_TEST_RUN, {test={prog_fd=270, retval=0, data_size_in=0, data_size_out=0, data_in=NULL, data_out=NULL, repeat=0, duration=0, ctx_size_in=0, ctx_size_out=0, ctx_in=NULL, ctx_out=NULL}, ...}, 120) = 0
bpf(BPF_PROG_TEST_RUN, {test={prog_fd=270, retval=0, data_size_in=0, data_size_out=0, data_in=NULL, data_out=NULL, repeat=0, duration=0, ctx_size_in=0, ctx_size_out=0, ctx_in=NULL, ctx_out=NULL}, ...}, 120) = 0
bpf(BPF_PROG_TEST_RUN, {test={prog_fd=270, retval=0, data_size_in=0, data_size_out=0, data_in=NULL, data_out=NULL, repeat=0, duration=0, ctx_size_in=0, ctx_size_out=0, ctx_in=NULL, ctx_out=NULL}, ...}, 120) = 0
bpf(BPF_PROG_TEST_RUN, {test={prog_fd=270, retval=0, data_size_in=0, data_size_out=0, data_in=NULL, data_out=NULL, repeat=0, duration=0, ctx_size_in=0, ctx_size_out=0, ctx_in=NULL, ctx_out=NULL}, ...}, 120) = 0
bpf(BPF_PROG_TEST_RUN, {test={prog_fd=270, retval=0, data_size_in=0, data_size_out=0, data_in=NULL, data_out=NULL, repeat=0, duration=0, ctx_size_in=0, ctx_size_out=0, ctx_in=NULL, ctx_out=NULL}, ...}, 120) = 0
bpf(BPF_PROG_TEST_RUN, {test={prog_fd=270, retval=0, data_size_in=0, data_size_out=0, data_in=NULL, data_out=NULL, repeat=0, duration=0, ctx_size_in=0, ctx_size_out=0, ctx_in=NULL, ctx_out=NULL}, ...}, 120) = 0
bpf(BPF_PROG_TEST_RUN, {test={prog_fd=270, retval=0, data_size_in=0, data_size_out=0, data_in=NULL, data_out=NULL, repeat=0, duration=0, ctx_size_in=0, ctx_size_out=0, ctx_in=NULL, ctx_out=NULL}, ...}, 120) = 0
bpf(BPF_MAP_LOOKUP_ELEM, {map_fd=306, key=0x7ffda3255160, value=0x7ffda3254f10, flags=BPF_ANY}, 120) = 0

 Performance counter stats for 'sleep 1':

              1.64 msec task-clock                #    0.002 CPUs utilized          
                 3      context-switches          #    1.827 K/sec                  
                 1      cpu-migrations            #  609.091 /sec                   
                69      page-faults               #   42.027 K/sec                  
         6,661,659      cycles                    #    4.058 GHz                    
         2,098,928      stalled-cycles-frontend   #   31.51% frontend cycles idle     (88.25%)
           354,763      stalled-cycles-backend    #    5.33% backend cycles idle      (87.59%)
         2,741,209      instructions              #    0.41  insn per cycle         
                                                  #    0.77  stalled cycles per insn  (43.20%)
           463,473      branches                  #  282.297 M/sec                    (80.93%)
            13,458      branch-misses             #    2.90% of all branches        

       1.001882586 seconds time elapsed

       0.000000000 seconds user
       0.001667000 seconds sys


--- SIGCHLD {si_signo=SIGCHLD, si_code=SI_USER, si_pid=75348, si_uid=0} ---
+++ exited with 0 +++
[root@five pahole]# 

Re: [PATCH v2 0/3] perf-stat: share hardware PMCs with BPF

[Arnaldo Carvalho de Melo]

Em Fri, Mar 19, 2021 at 04:14:42PM +0000, Song Liu escreveu:
> > On Mar 19, 2021, at 8:58 AM, Namhyung Kim <namhyung@kernel.org> wrote:
> > On Sat, Mar 20, 2021 at 12:35 AM Arnaldo Carvalho de Melo <acme@kernel.org> wrote:
> >> Em Fri, Mar 19, 2021 at 09:54:59AM +0900, Namhyung Kim escreveu:
> >>> On Fri, Mar 19, 2021 at 9:22 AM Song Liu <songliubraving@fb.com> wrote:
> >>>>> On Mar 18, 2021, at 5:09 PM, Arnaldo <arnaldo.melo@gmail.com> wrote:
> >>>>> On March 18, 2021 6:14:34 PM GMT-03:00, Jiri Olsa <jolsa@redhat.com> wrote:
> >>>>>> On Thu, Mar 18, 2021 at 03:52:51AM +0000, Song Liu wrote:
> >>>>>>> perf stat -C 1,3,5                  107.063 [sec]
> >>>>>>> perf stat -C 1,3,5 --bpf-counters   106.406 [sec]

> >>>>>> I can't see why it's actualy faster than normal perf ;-)
> >>>>>> would be worth to find out

> >>>>> Isn't this all about contended cases?

> >>>> Yeah, the normal perf is doing time multiplexing; while --bpf-counters
> >>>> doesn't need it.

> >>> Yep, so for uncontended cases, normal perf should be the same as the
> >>> baseline (faster than the bperf).  But for contended cases, the bperf
> >>> works faster.

> >> The difference should be small enough that for people that use this in a
> >> machine where contention happens most of the time, setting a
> >> ~/.perfconfig to use it by default should be advantageous, i.e. no need
> >> to use --bpf-counters on the command line all the time.

> >> So, Namhyung, can I take that as an Acked-by or a Reviewed-by? I'll take
> >> a look again now but I want to have this merged on perf/core so that I
> >> can work on a new BPF SKEL to use this:

> > I have a concern for the per cpu target, but it can be done later, so

> > Acked-by: Namhyung Kim <namhyung@kernel.org>

> >> https://git.kernel.org/pub/scm/linux/kernel/git/acme/linux.git/log/?h=tmp.bpf/bpf_perf_enable

> > Interesting!  Actually I was thinking about the similar too. :)
> 
> Hi Namhyung, Jiri, and Arnaldo,
> 
> Thanks a lot for your kind review. 
> 
> Here is updated 3/3, where we use perf-bench instead of stressapptest.

I had to apply this updated 3/3 manually, as there was some munging, its
all now at:

https://git.kernel.org/pub/scm/linux/kernel/git/acme/linux.git/log/?h=tmp.perf/core

Please take a look at the "Committer testing" section I added to the
main patch, introducing bperf:

https://git.kernel.org/pub/scm/linux/kernel/git/acme/linux.git/commit/?h=tmp.perf/core&id=7fac83aaf2eecc9e7e7b72da694c49bb4ce7fdfc

And check if I made any mistake or if something else could be added.

It'll move to perf/core after my set of automated tests finishes.

- Arnaldo

Re: [PATCH v2 0/3] perf-stat: share hardware PMCs with BPF

[Song Liu]

> On Mar 23, 2021, at 2:10 PM, Arnaldo Carvalho de Melo <acme@kernel.org> wrote:
> 
> Em Fri, Mar 19, 2021 at 04:14:42PM +0000, Song Liu escreveu:
>>> On Mar 19, 2021, at 8:58 AM, Namhyung Kim <namhyung@kernel.org> wrote:
>>> On Sat, Mar 20, 2021 at 12:35 AM Arnaldo Carvalho de Melo <acme@kernel.org> wrote:
>>>> Em Fri, Mar 19, 2021 at 09:54:59AM +0900, Namhyung Kim escreveu:
>>>>> On Fri, Mar 19, 2021 at 9:22 AM Song Liu <songliubraving@fb.com> wrote:
>>>>>>> On Mar 18, 2021, at 5:09 PM, Arnaldo <arnaldo.melo@gmail.com> wrote:
>>>>>>> On March 18, 2021 6:14:34 PM GMT-03:00, Jiri Olsa <jolsa@redhat.com> wrote:
>>>>>>>> On Thu, Mar 18, 2021 at 03:52:51AM +0000, Song Liu wrote:
>>>>>>>>> perf stat -C 1,3,5                  107.063 [sec]
>>>>>>>>> perf stat -C 1,3,5 --bpf-counters   106.406 [sec]
> 
>>>>>>>> I can't see why it's actualy faster than normal perf ;-)
>>>>>>>> would be worth to find out
> 
>>>>>>> Isn't this all about contended cases?
> 
>>>>>> Yeah, the normal perf is doing time multiplexing; while --bpf-counters
>>>>>> doesn't need it.
> 
>>>>> Yep, so for uncontended cases, normal perf should be the same as the
>>>>> baseline (faster than the bperf).  But for contended cases, the bperf
>>>>> works faster.
> 
>>>> The difference should be small enough that for people that use this in a
>>>> machine where contention happens most of the time, setting a
>>>> ~/.perfconfig to use it by default should be advantageous, i.e. no need
>>>> to use --bpf-counters on the command line all the time.
> 
>>>> So, Namhyung, can I take that as an Acked-by or a Reviewed-by? I'll take
>>>> a look again now but I want to have this merged on perf/core so that I
>>>> can work on a new BPF SKEL to use this:
> 
>>> I have a concern for the per cpu target, but it can be done later, so
> 
>>> Acked-by: Namhyung Kim <namhyung@kernel.org>
> 
>>>> https://git.kernel.org/pub/scm/linux/kernel/git/acme/linux.git/log/?h=tmp.bpf/bpf_perf_enable
> 
>>> Interesting!  Actually I was thinking about the similar too. :)
>> 
>> Hi Namhyung, Jiri, and Arnaldo,
>> 
>> Thanks a lot for your kind review. 
>> 
>> Here is updated 3/3, where we use perf-bench instead of stressapptest.
> 
> I had to apply this updated 3/3 manually, as there was some munging, its
> all now at:
> 
> https://git.kernel.org/pub/scm/linux/kernel/git/acme/linux.git/log/?h=tmp.perf/core
> 
> Please take a look at the "Committer testing" section I added to the
> main patch, introducing bperf:
> 
> https://git.kernel.org/pub/scm/linux/kernel/git/acme/linux.git/commit/?h=tmp.perf/core&id=7fac83aaf2eecc9e7e7b72da694c49bb4ce7fdfc
> 
> And check if I made any mistake or if something else could be added.
> 
> It'll move to perf/core after my set of automated tests finishes.

Thanks Arnaldo! Looks great!

Song