Re: [PATCH 13/16] perf callchain: Switch default to 'graph,0.5,caller'

From: Brendan Gregg <brendan.d.gregg@gmail.com>
To: Chandler Carruth <chandlerc@gmail.com>
Cc: Arnaldo Carvalho de Melo <arnaldo.melo@gmail.com>,
	Frederic Weisbecker <fweisbec@gmail.com>,
	Ingo Molnar <mingo@kernel.org>,
	LKML <linux-kernel@vger.kernel.org>,
	Adrian Hunter <adrian.hunter@intel.com>,
	Borislav Petkov <bp@suse.de>, David Ahern <dsahern@gmail.com>,
	Jiri Olsa <jolsa@redhat.com>, Namhyung Kim <namhyung@kernel.org>,
	Stephane Eranian <eranian@google.com>,
	Wang Nan <wangnan0@huawei.com>
Subject: Re: [PATCH 13/16] perf callchain: Switch default to 'graph,0.5,caller'
Date: Wed, 21 Oct 2015 11:28:54 -0700	[thread overview]
Message-ID: <CAE40pdfUoe-f=dpM_d6RpA64VQFNnYe30pMutTqzOKC6FinUYw@mail.gmail.com> (raw)
In-Reply-To: <CAAwGriEtYeBytGt9x24=uUqSEy5oJ2HigfA2KXnKyrAioKrtNg@mail.gmail.com>

On Tue, Oct 20, 2015 at 7:21 PM, Chandler Carruth <chandlerc@gmail.com> wrote:
> Since Arnaldo asked, I thought I should actually try to respond specifically
> to the question of why I favor the 'caller' view as the default.
>
> On Tue, Oct 20, 2015 at 3:06 AM Arnaldo Carvalho de Melo
> <arnaldo.melo@gmail.com> wrote:
>>
>> > IMHO changing that order is not a good idea. Unless many users
>> > complained
>> > about it.
>>
>> Perhaps there are not that many users of callchains because the default
>> is not what they're used to see?
>>
>> Motivation for the change came from a video from Chandler, that
>> resurfaced the callchain default issue, Chandler?
>
>
> So, first and foremost, thanks for fixing some of my gripes about the
> usability of the perf tool, I'm super excited about the changes you're
> making, even if this one isn't among them.
>
> I think the default of caller vs. callee is probably the hardest judgement
> call to make about the right defaults. I can see it going both ways.
>
> When profiling my *system*, or a diverse group of programs or tasks, I often
> find callee useful. Were I a kernel developer, I suspect callee would be
> *dramatically* more common than caller.
>
> For me, what makes the caller view much more frequently desired is that I'm
> usually profiling a fairly isolated application, or benchmark for an
> isolated library. While I always start off with some more system-level
> performance problem, I rarely need a detailed profile to get a reasonable
> idea of what subsystem to stare at, and then I spend days looking at a
> relatively isolated reproduction.
>
> Anyways, for profiling user-land applications, I suspect from my
> conversations with users that "caller" is the more common expectation.
[...]

I would have said callee is the more common expectation, certainly for
system profilers. I quickly checked various tools to see what their
defaults are:

callee:

gdb
lldb
pstack
jstack
perf
stap
ktap
dtrace
kernel oops message
jvm crash
node.js/v8 crash

caller:

python traceback

Python was the only one I knew off-hand that is caller by default (and
it includes the text "most recent call last", suggesting the
developers thought it was necessary to point out that it was
different). Do you know what tools/profilers/debuggers these users are
using?

I really think people will be surprised if by default perf prints
stacks differently to gdb, lldb, oops messages, etc, etc. That may be
true for a specific developer community (eg, Python), but not for
system profilers.

Brendan