Re: perf software events broken in containers

[William Cohen <wcohen@redhat.com>]

On 03/22/2017 02:24 PM, Brendan Gregg wrote:
> G'Day,
> 
> I think something broke recently with using perf from within a docker
> container. We used to be able to run "docker exec -it --privileged
> UUID bash", and then run perf from it for CPU sampling. But we just
> noticed on recent kernels (4.10 and updated 4.4) that it no longer
> works. Anyone else see this?
> 
> perf's error message is contradictory:
> 
> # /perf record -F 99 -a -- sleep 1
> perf_event_open(..., PERF_FLAG_FD_CLOEXEC) failed with unexpected
> error 1 (Operation not permitted)
> perf_event_open(..., 0) failed unexpectedly with error 1 (Operation
> not permitted)
> Error:
> You may not have permission to collect system-wide stats.
> 
> Consider tweaking /proc/sys/kernel/perf_event_paranoid,
> which controls use of the performance events system by
> unprivileged users (without CAP_SYS_ADMIN).
> 
> The current value is -1:
> 
>   -1: Allow use of (almost) all events by all users
>> = 0: Disallow raw tracepoint access by users without CAP_IOC_LOCK
>> = 1: Disallow CPU event access by users without CAP_SYS_ADMIN
>> = 2: Disallow kernel profiling by users without CAP_SYS_ADMIN
> 
> With -v, I can see that it tries the PMC based cycles, then gives up.
> What normally happens is it then switches to the cpu-clock software
> event, but it's not trying that anymore. Those software events are
> also no longer visible:
> 
> # /perf list sw
> 
> List of pre-defined events (to be used in -e):
> 
> (no output)
> 
> The kernel is returning errno 1 to the sys_perf_event_open() call in
> __perf_evsel__open(). I'm trying to find out which kernel function
> throws the EPERM, but almost nothing is tracable via ftrace/kprobes.
> It's pretty annoying... Thanks for any ideas,

Hi Brendan,

Several years ago I had a problem with the perf_event_open not working on arm machines.  I looked at the kernel source code and just kept putting systemtap one-liners like the following on the various functions used by the perf_event_open to see which function was the one returning the error code:

stap -v -e 'probe kernel.function("idr_find").return {printf("%s %s 0x%x\n", pn(), $$parms$, $return)}'


Could the docker container not have the perf_event_open syscall on the whitelist of allowed syscalls?

Would bcc's capable.py tool give some insight into whether there are some other capabilities that the perf_event_open might be needing?

-Will
> 
> Brendan
> --
> To unsubscribe from this list: send the line "unsubscribe linux-perf-users" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>