perf software events broken in containers

[Brendan Gregg <brendan.d.gregg@gmail.com>]

G'Day,

I think something broke recently with using perf from within a docker
container. We used to be able to run "docker exec -it --privileged
UUID bash", and then run perf from it for CPU sampling. But we just
noticed on recent kernels (4.10 and updated 4.4) that it no longer
works. Anyone else see this?

perf's error message is contradictory:

# /perf record -F 99 -a -- sleep 1
perf_event_open(..., PERF_FLAG_FD_CLOEXEC) failed with unexpected
error 1 (Operation not permitted)
perf_event_open(..., 0) failed unexpectedly with error 1 (Operation
not permitted)
Error:
You may not have permission to collect system-wide stats.

Consider tweaking /proc/sys/kernel/perf_event_paranoid,
which controls use of the performance events system by
unprivileged users (without CAP_SYS_ADMIN).

The current value is -1:

  -1: Allow use of (almost) all events by all users
>= 0: Disallow raw tracepoint access by users without CAP_IOC_LOCK
>= 1: Disallow CPU event access by users without CAP_SYS_ADMIN
>= 2: Disallow kernel profiling by users without CAP_SYS_ADMIN

With -v, I can see that it tries the PMC based cycles, then gives up.
What normally happens is it then switches to the cpu-clock software
event, but it's not trying that anymore. Those software events are
also no longer visible:

# /perf list sw

List of pre-defined events (to be used in -e):

(no output)

The kernel is returning errno 1 to the sys_perf_event_open() call in
__perf_evsel__open(). I'm trying to find out which kernel function
throws the EPERM, but almost nothing is tracable via ftrace/kprobes.
It's pretty annoying... Thanks for any ideas,

Brendan