Re: perf segfault in docker container

[Aravinda Prasad <aravinda@linux.vnet.ibm.com>]

Hi Brendan,

On Wednesday 22 June 2016 04:02 AM, Brendan Gregg wrote:
> G'Day Aravinda,
> 
> Sorry for the delay; answers inline:
> 
> On Fri, Jun 10, 2016 at 3:28 AM, Aravinda Prasad
> <aravinda@linux.vnet.ibm.com> wrote:
>>
>> Hi Brendan,
>>
>> I though of replying to your mail as I saw you running perf inside a
>> docker container. I believe you would be interested in events specific
>> to the container context as you are using "perf record -a".
>>
>> We are working on supporting "container-aware tracing" i.e., whenever
>> you run "perf record -a" inside a container it should report
>> container-wide events rather than system-wide events. Towards that goal,
>> we posted an RFC patch in LKML [1] last year and also discussed possible
>> ways to restrict events within a container in Plumbers (Container
>> Microconf) [2].
> 
> Sounds great.
> 
>>
>>
>> Based on the discussion in Container Microconf, we are coming up with a
>> new prototype which should be ready for review by next week. The new
>> prototype introduces a new namespace "perf-namespace" (namespace name is
>> just a placeholder. Suggestions welcome). If the container is created
>> with perf-namespace, then "perf record -a" inside the container reports
>> only those events that are triggered within the container.
> 
> I'd think that this restriction should be the default, rather than
> needing to create a container with a perf-namespace. Why wouldn't it
> make use of the existing pid namespace?

Our initial prototype (lkml.org/lkml/2015/7/15/192) was based on
pid-namespace. However, during the discussion in Plumbers, it was
mentioned that the requirement of PID namespace is insufficient for
containers that need access to the host PID namespace as these
containers are created without a PID namespace. Hence, we thought of
introducing perf-namespace.

We have posted the RFC patches for perf-namespace prototype:
https://lkml.org/lkml/2016/6/14/547

> 
>>
>> We would like to know if you are looking for "container-aware tracing"
>> and also like to know the scenarios/problems you are trying to debug by
>> running perf inside a container.
> 
> Yes, perf needs to be container-aware.
> 
> To start with, we'd like to profile apps running inside Docker
> containers, either by running perf in the container, or by running
> perf from the host. As in, "perf record -F49 -a -g -- sleep 30". I've
> tried both and had both approaches work, with some wrestling of
> /tmp/perf-PID.map files and things.

We are also working on enabling running perf from host with a container
ID as an argument. This is in addition to enabling perf inside a container.

> 
> If perf was container-aware, then running it in the container should
> be the easiest way to profile an app, if it's only sampling that
> container.
> 
> Also, from within a container, I'd expect to be able to sample kernel
> stacks that are running for the container processes (eg, syscalls),
> but not asynchronous kernel threads that are running host-wide (eg,
> background fsflush).

Our current and previous prototypes sample kernel events which are
triggered from the container context. And yes, they do not include
events from asynchronous kernel threads.

> 
> More advanced things would involve tracing syscall latency and using
> BPF for latency histograms, from within a container. That should be
> allowed.

Sure. Noted.

> 
> What about tracepoints? Should a container be able to use the block
> I/O tracepoints and see disk I/O latency histograms? Filtering this to
> be just the container's block I/O would be tricky. Doing it
> system-wide may be allowable, depending on a setting in
> perf_event_paranoid. I think in some environments, having a container
> trace all tracepoints (disk, tcp, etc) is ok, provided to data is
> leaked from another container; whereas in other environments tracing
> non-container events would not be ok. Hence setting this in
> perf_event_paranoid.

Yes, filtering such tracepoints to just the container's instance is
tricky and we have not yet figured out any solution for that.

Regards,
Aravinda

> 
> Brendan
> 

-- 
Regards,
Aravinda