Re: [Ksummit-discuss] [TECH TOPIC] Kernel tracing and end-to-end performance breakdown

[Chris Mason <clm@fb.com>]

On 07/21/2016 06:00 AM, Jan Kara wrote:
>
> So I think improvements in performance analysis are always welcome but
> current proposal seems to be somewhat handwavy so I'm not sure what outcome
> you'd like to get from the discussion... If you have a more concrete
> proposal how you'd like to achieve what you need, then it may be worth
> discussion.
>
> As a side note I know that Google (and maybe Facebook, not sure here) have
> out-of-tree patches which provide really neat performance analysis
> capabilities. I have heard they are not really upstreamable because they
> are horrible hacks but maybe they can be a good inspiration for this work.
> If we could get someone from these companies to explain what capabilities
> they have and how they achieve this (regardless how hacky the
> implementation may be), that may be an interesting topic.

At least for facebook, we're moving most things to bpf.  The most 
interesting part of our analysis isn't so much from the tool used to 
record it, it's from being able to aggregate over the fleet and making 
comparisons at scale.

For example, Josef setup the off-cpu flame graphs such that we can 
record stack traces for a latency higher than N, and then sum up the 
most expensive stack traces over a large number of machines.  It makes 
it much easier to find those happens-once-a-day problems.

-chris