Re: [PATCHSET 0/4] perf stat: Add --multiply-cgroup option

From: Ian Rogers <irogers@google.com>
To: Andi Kleen <andi@firstfloor.org>
Cc: Namhyung Kim <namhyung@kernel.org>,
	Arnaldo Carvalho de Melo <acme@kernel.org>,
	Jiri Olsa <jolsa@redhat.com>, Ingo Molnar <mingo@kernel.org>,
	Peter Zijlstra <a.p.zijlstra@chello.nl>,
	Mark Rutland <mark.rutland@arm.com>,
	Alexander Shishkin <alexander.shishkin@linux.intel.com>,
	Stephane Eranian <eranian@google.com>,
	LKML <linux-kernel@vger.kernel.org>
Subject: Re: [PATCHSET 0/4] perf stat: Add --multiply-cgroup option
Date: Thu, 10 Sep 2020 10:11:21 -0700	[thread overview]
Message-ID: <CAP-5=fUp90QrY+HcetpgZJeUn2fSqndRCZ8f1m=rFcEwO7ZZew@mail.gmail.com> (raw)
In-Reply-To: <20200910155736.jadhmqvnqquammpn@two.firstfloor.org>

On Thu, Sep 10, 2020 at 8:57 AM Andi Kleen <andi@firstfloor.org> wrote:
>
> On Tue, Sep 08, 2020 at 01:42:24PM +0900, Namhyung Kim wrote:
> > When we profile cgroup events with perf stat, it's very annoying to
> > specify events and cgroups on the command line as it requires the
> > mapping between events and cgroups.  (Note that perf record can use
> > cgroup sampling but it's not usable for perf stat).
>
> The problem is real, but I don't really like your solution.
> The option is ugly. Should rather be solved with some suitable
> syntax in the expression parser to express: apply to all,
> instead of adding adhoc options like this.
>
> There are some additional problems that really need to be eventually
> solved too:
>
> - If you use the old syntax and some cgroups are not covered you don't
> get any warning. At least that should be fixed too.
>
> - And of course if everything works it is still very slow for the kernel
> because there are so many perf events to handle. Long term we probably
> need some more flexible way to just specify for given perf events which set of
> cgroups they should apply, so that sharing and low overhead monitoring
> of many cgroups is possible I hate to say it, but maybe some eBPF filter
> is the solution here.
>
> -Andi

Just to throw in some 2c worth. A nice thing about Namhyung's approach
is that it can apply for metrics, events and pfm-events. It would be
nice if there were a single option for specifying these, perhaps we
can figure one out.

One thought that came to mind based on Namhyung's multiply name was
and assuming a cgroup could be a modifier:

perf record -e cycles:G1

there could be equivalent to a new syntax of lists and multiplies of:

perf record -e [cycles]*[:G1]

This would allow expressions like:

perf record -e [{instructions,cycles},branches]*[:u,:k]

which would be equivalent to (showing the effect on sibling groups):

perf record -e {instructions:u,cycles:u},branches:u,{instructions:k,cycles:k},branches:k

Adding in cgroups makes a longer list of events:

perf record -e [{instructions,cycles},branches]*[:u,:k]*[:G1,:G2]

becomes:

perf record -e {instructions:u:G1,cycles:u:G1},branches:u:G1,{instructions:k:G1,cycles:k:G1},branches:k:G1,{instructions:u:G2,cycles:u:G2},branches:u:G2,{instructions:k:G2,cycles:k:G2},branches:k:G2

This is somewhat similar to Arnaldo's proposal but trying to make
things a bit more generic, avoiding overloading the use of sibling
groups, .. Perhaps there is a syntax that others prefer or could be
borrowed from a familiar source like a programming language.

I think for Namhyung's sake it is important not to get too distracted
by desires for better syntax, as this change makes a benefit in a way
that works with the existing flags. If it is accepted, the man pages
need to be updated.

Thanks,
Ian