'perf top' broken on intel hybrid systems

'perf top' broken on intel hybrid systems

[Arnaldo Carvalho de Melo]

Hi,

	So I finally got a recent Intel hybrid system:

root@fedora:~# grep -m1 "model name" /proc/cpuinfo
model name	: Intel(R) Core(TM) i7-14700K
root@fedora:~#

	Most things work, but:

root@fedora:~# perf top

Error:
The cycles:P event is not supported.
root@fedora:~#

root@fedora:~# perf top -e cycles:p
Error:
The cycles:p event is not supported.
root@fedora:~# perf top -e cycles:pp
Error:
The cycles:pp event is not supported.
^[[Aroot@fedora:~# perf top -e cycles:ppp
Error:
The cycles:ppp event is not supported.
root@fedora:~#
root@fedora:~# perf top -e cycles
Error:
The cycles event is not supported.
root@fedora:~#

root@fedora:~# perf top -e instructions
Error:
The instructions event is not supported.
root@fedora:~#root@fedora:~# perf top -e cache-misses
Error:
The cache-misses event is not supported.
root@fedora:~#

------------------------------------------------------------
perf_event_attr:
  type                             0 (PERF_TYPE_HARDWARE)
  size                             136
  config                           0xa00000000
  { sample_period, sample_freq }   4000
  sample_type                      IP|TID|TIME|ID|CPU|PERIOD
  read_format                      ID|LOST
  disabled                         1
  inherit                          1
  mmap                             1
  comm                             1
  freq                             1
  task                             1
  sample_id_all                    1
  exclude_guest                    1
  mmap2                            1
  comm_exec                        1
  ksymbol                          1
  bpf_event                        1
------------------------------------------------------------
sys_perf_event_open: pid -1  cpu 0  group_fd -1  flags 0x8
sys_perf_event_open failed, error -2
Error:
The cycles event is not supported.
root@fedora:~#

But...

root@fedora:~# perf record
^C[ perf record: Woken up 1 times to write data ]
[ perf record: Captured and wrote 2.631 MB perf.data (24079 samples) ]

root@fedora:~# perf evlist -v
cpu_atom/cycles:P/: type: 0 (PERF_TYPE_HARDWARE), size: 136, config: 0xa00000000, { sample_period, sample_freq }: 4000, sample_type: IP|TID|TIME|CPU|PERIOD|IDENTIFIER, read_format: ID|LOST, disabled: 1, inherit: 1, freq: 1, precise_ip: 3, sample_id_all: 1
cpu_core/cycles:P/: type: 0 (PERF_TYPE_HARDWARE), size: 136, config: 0x400000000, { sample_period, sample_freq }: 4000, sample_type: IP|TID|TIME|CPU|PERIOD|IDENTIFIER, read_format: ID|LOST, disabled: 1, inherit: 1, freq: 1, precise_ip: 3, sample_id_all: 1
dummy:u: type: 1 (PERF_TYPE_SOFTWARE), size: 136, config: 0x9 (PERF_COUNT_SW_DUMMY), { sample_period, sample_freq }: 1, sample_type: IP|TID|TIME|CPU|IDENTIFIER, read_format: ID|LOST, inherit: 1, exclude_kernel: 1, exclude_hv: 1, mmap: 1, comm: 1, task: 1, sample_id_all: 1, exclude_guest: 1, mmap2: 1, comm_exec: 1, ksymbol: 1, bpf_event: 1
root@fedora:~#


root@fedora:~# perf top -e cpu_core/cpu-cycles:P/
event syntax error: 'cpu_core/cpu-cycles:P/'
                              \___ Bad event or PMU

Unable to find PMU or event on a PMU of 'cpu_core'

Initial error:
event syntax error: 'cpu_core/cpu-cycles:P/'
                              \___ unknown term 'cpu-cycles:P' for pmu 'cpu_core'

valid terms: event,pc,edge,offcore_rsp,ldlat,inv,umask,frontend,cmask,config,config1,config2,config3,name,period,freq,branch_type,time,call-graph,stack-size,no-inherit,inherit,max-stack,nr,no-overwrite,overwrite,driver-config,percore,aux-output,aux-sample-size,metric-id,raw,legacy-cache,hardware
Run 'perf list' for a list of valid events

 Usage: perf top [<options>]

    -e, --event <event>   event selector. use 'perf list' to list available events
root@fedora:~#

So it seems its just 'perf top' breakage:

root@fedora:~# perf top -e cpu_core/cpu-cycles/
Error:
The cpu_core/cpu-cycles/ event is not supported.
root@fedora:~# perf top -e cpu_atom/cpu-cycles/
Error:
The cpu_atom/cpu-cycles/ event is not supported.
root@fedora:~#

I bisected this to:

⬢[acme@toolbox perf-tools-next]$ git bisect good
251aa040244a3b17068e4e6ec61f138d7e50681a is the first bad commit
commit 251aa040244a3b17068e4e6ec61f138d7e50681a
Author: Ian Rogers <irogers@google.com>
Date:   Thu Jun 1 01:29:53 2023 -0700

    perf parse-events: Wildcard most "numeric" events

    Numeric events are either raw events or those with ABI defined numbers
    matched by the lexer. PERF_TYPE_HARDWARE and PERF_TYPE_HW_CACHE events
    should wildcard match on hybrid systems. So "cycles" should match each
    PMU type with an extended type, not just PERF_TYPE_HARDWARE.

    Change wildcard matching to add the event even if wildcard PMU
    scanning fails, there will be no extended type but this best matches
    previous behavior.

    Only set the extended type when the event type supports it and when
    perf_pmus__supports_extended_type is true. This new function returns
    true if >1 core PMU and avoids potential errors on older kernels.

    Modify evsel__compute_group_pmu_name using a helper
    perf_pmu__is_software to determine when grouping should occur. Try to
    use PMUs, and evsel__find_pmu, as being more dependable than
    evsel->pmu_name.

    Set a parse events error if a hardware term's PMU lookup fails, to
    provide extra diagnostics.

    Fixes: 8bc75f699c141420 ("perf parse-events: Support wildcards on raw events")
    Reported-by: Kan Liang <kan.liang@linux.intel.com>
    Signed-off-by: Ian Rogers <irogers@google.com>
    Tested-by: Kan Liang <kan.liang@linux.intel.com>
    Cc: Adrian Hunter <adrian.hunter@intel.com>
    Cc: Alexander Shishkin <alexander.shishkin@linux.intel.com>
    Cc: Ingo Molnar <mingo@redhat.com>
    Cc: Jiri Olsa <jolsa@kernel.org>
    Cc: Mark Rutland <mark.rutland@arm.com>
    Cc: Namhyung Kim <namhyung@kernel.org>
    Cc: Peter Zijlstra <peterz@infradead.org>
    Cc: Ravi Bangoria <ravi.bangoria@amd.com>
    Cc: Rob Herring <robh@kernel.org>
    Cc: Thomas Richter <tmricht@linux.ibm.com>
    Cc: Xing Zhengjun <zhengjun.xing@linux.intel.com>
    Link: https://lore.kernel.org/r/20230601082954.754318-4-irogers@google.com
    Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>

 tools/perf/util/parse-events.c | 104 +++++++++++++++++++++++++++++------------
 tools/perf/util/parse-events.y |  10 ++--
 tools/perf/util/pmu.c          |  16 +++++++
 tools/perf/util/pmu.h          |   5 ++
 tools/perf/util/pmus.c         |   5 ++
 tools/perf/util/pmus.h         |   1 +
 6 files changed, 106 insertions(+), 35 deletions(-)
⬢[acme@toolbox perf-tools-next]$

- Arnaldo

Re: 'perf top' broken on intel hybrid systems

[Liang, Kan]

On 2023-12-08 1:57 p.m., Arnaldo Carvalho de Melo wrote:
> Hi,
> 
> 	So I finally got a recent Intel hybrid system:
> 
> root@fedora:~# grep -m1 "model name" /proc/cpuinfo
> model name	: Intel(R) Core(TM) i7-14700K
> root@fedora:~#
> 
> 	Most things work, but:
> 
> root@fedora:~# perf top
> 
> Error:
> The cycles:P event is not supported.
> root@fedora:~#
> 
> root@fedora:~# perf top -e cycles:p
> Error:
> The cycles:p event is not supported.
> root@fedora:~# perf top -e cycles:pp
> Error:
> The cycles:pp event is not supported.
> ^[[Aroot@fedora:~# perf top -e cycles:ppp
> Error:
> The cycles:ppp event is not supported.
> root@fedora:~#
> root@fedora:~# perf top -e cycles
> Error:
> The cycles event is not supported.
> root@fedora:~#
> 
> root@fedora:~# perf top -e instructions
> Error:
> The instructions event is not supported.
> root@fedora:~#root@fedora:~# perf top -e cache-misses
> Error:
> The cache-misses event is not supported.
> root@fedora:~#
> 
> ------------------------------------------------------------
> perf_event_attr:
>   type                             0 (PERF_TYPE_HARDWARE)
>   size                             136
>   config                           0xa00000000
>   { sample_period, sample_freq }   4000
>   sample_type                      IP|TID|TIME|ID|CPU|PERIOD
>   read_format                      ID|LOST
>   disabled                         1
>   inherit                          1
>   mmap                             1
>   comm                             1
>   freq                             1
>   task                             1
>   sample_id_all                    1
>   exclude_guest                    1
>   mmap2                            1
>   comm_exec                        1
>   ksymbol                          1
>   bpf_event                        1
> ------------------------------------------------------------
> sys_perf_event_open: pid -1  cpu 0  group_fd -1  flags 0x8
> sys_perf_event_open failed, error -2


The error is because the perf top always tries to open an event on the
user_requested_cpus, which are all CPUs by default.

Something as below should fix it. For hybrid, open a PMU event on an
unsupported CPU should be error out.

diff --git a/tools/perf/builtin-top.c b/tools/perf/builtin-top.c
index 1e42bd1c7d5a..7aa29e6264e4 100644
--- a/tools/perf/builtin-top.c
+++ b/tools/perf/builtin-top.c
@@ -1027,8 +1027,8 @@ static int perf_top__start_counters(struct
perf_top *top)

        evlist__for_each_entry(evlist, counter) {
 try_again:
-               if (evsel__open(counter,
top->evlist->core.user_requested_cpus,
-                                    top->evlist->core.threads) < 0) {
+               if (evsel__open(counter, counter->core.cpus,
+                               counter->core.threads) < 0) {

                        /*
                         * Specially handle overwrite fall back.


> Error:
> The cycles event is not supported.
> root@fedora:~#
> 
> But...
> 
> root@fedora:~# perf record
> ^C[ perf record: Woken up 1 times to write data ]
> [ perf record: Captured and wrote 2.631 MB perf.data (24079 samples) ]
> 
> root@fedora:~# perf evlist -v
> cpu_atom/cycles:P/: type: 0 (PERF_TYPE_HARDWARE), size: 136, config: 0xa00000000, { sample_period, sample_freq }: 4000, sample_type: IP|TID|TIME|CPU|PERIOD|IDENTIFIER, read_format: ID|LOST, disabled: 1, inherit: 1, freq: 1, precise_ip: 3, sample_id_all: 1
> cpu_core/cycles:P/: type: 0 (PERF_TYPE_HARDWARE), size: 136, config: 0x400000000, { sample_period, sample_freq }: 4000, sample_type: IP|TID|TIME|CPU|PERIOD|IDENTIFIER, read_format: ID|LOST, disabled: 1, inherit: 1, freq: 1, precise_ip: 3, sample_id_all: 1
> dummy:u: type: 1 (PERF_TYPE_SOFTWARE), size: 136, config: 0x9 (PERF_COUNT_SW_DUMMY), { sample_period, sample_freq }: 1, sample_type: IP|TID|TIME|CPU|IDENTIFIER, read_format: ID|LOST, inherit: 1, exclude_kernel: 1, exclude_hv: 1, mmap: 1, comm: 1, task: 1, sample_id_all: 1, exclude_guest: 1, mmap2: 1, comm_exec: 1, ksymbol: 1, bpf_event: 1
> root@fedora:~#
> 
> 
> root@fedora:~# perf top -e cpu_core/cpu-cycles:P/
> event syntax error: 'cpu_core/cpu-cycles:P/'
>                               \___ Bad event or PMU
> 
> Unable to find PMU or event on a PMU of 'cpu_core'
> 
> Initial error:
> event syntax error: 'cpu_core/cpu-cycles:P/'
>                               \___ unknown term 'cpu-cycles:P' for pmu 'cpu_core'

The P or other event modifier should be after the last '/'.
Something as below:
perf top -e cpu_core/cpu-cycles/P

Even for the non-hybrid or perf record, the /cpu-cycles:P/ format is
incorrect.

It looks like by design, not a bug.

$ perf record -e cpu/cycles:P/
event syntax error: 'cpu/cycles:P/'
                         \___ unknown term 'cycles:P' for pmu 'cpu'

valid terms:
event,pc,edge,any,offcore_rsp,ldlat,inv,umask,frontend,cmask,config,config1,config2,config3,name,period,freq,branch_type,time,call-graph,stack-size,no-inherit,inherit,max-stack,nr,no-overwrite,overwrite,driver-config,percore,aux-output,aux-sample-size,metric-id,raw,legacy-cache,hardware

Initial error:
event syntax error: 'cpu/cycles:P/'
                         \___ unknown term 'cycles:P' for pmu 'cpu'

$ perf record -e cpu/cycles/P
^C[ perf record: Woken up 1 times to write data ]
[ perf record: Captured and wrote 0.695 MB perf.data (254 samples) ]


Thanks,
Kan
> 
> valid terms: event,pc,edge,offcore_rsp,ldlat,inv,umask,frontend,cmask,config,config1,config2,config3,name,period,freq,branch_type,time,call-graph,stack-size,no-inherit,inherit,max-stack,nr,no-overwrite,overwrite,driver-config,percore,aux-output,aux-sample-size,metric-id,raw,legacy-cache,hardware
> Run 'perf list' for a list of valid events
> 
>  Usage: perf top [<options>]
> 
>     -e, --event <event>   event selector. use 'perf list' to list available events
> root@fedora:~#
> 
> So it seems its just 'perf top' breakage:
> 
> root@fedora:~# perf top -e cpu_core/cpu-cycles/
> Error:
> The cpu_core/cpu-cycles/ event is not supported.
> root@fedora:~# perf top -e cpu_atom/cpu-cycles/
> Error:
> The cpu_atom/cpu-cycles/ event is not supported.
> root@fedora:~#
> 
> I bisected this to:
> 
> ⬢[acme@toolbox perf-tools-next]$ git bisect good
> 251aa040244a3b17068e4e6ec61f138d7e50681a is the first bad commit
> commit 251aa040244a3b17068e4e6ec61f138d7e50681a
> Author: Ian Rogers <irogers@google.com>
> Date:   Thu Jun 1 01:29:53 2023 -0700
> 
>     perf parse-events: Wildcard most "numeric" events
> 
>     Numeric events are either raw events or those with ABI defined numbers
>     matched by the lexer. PERF_TYPE_HARDWARE and PERF_TYPE_HW_CACHE events
>     should wildcard match on hybrid systems. So "cycles" should match each
>     PMU type with an extended type, not just PERF_TYPE_HARDWARE.
> 
>     Change wildcard matching to add the event even if wildcard PMU
>     scanning fails, there will be no extended type but this best matches
>     previous behavior.
> 
>     Only set the extended type when the event type supports it and when
>     perf_pmus__supports_extended_type is true. This new function returns
>     true if >1 core PMU and avoids potential errors on older kernels.
> 
>     Modify evsel__compute_group_pmu_name using a helper
>     perf_pmu__is_software to determine when grouping should occur. Try to
>     use PMUs, and evsel__find_pmu, as being more dependable than
>     evsel->pmu_name.
> 
>     Set a parse events error if a hardware term's PMU lookup fails, to
>     provide extra diagnostics.
> 
>     Fixes: 8bc75f699c141420 ("perf parse-events: Support wildcards on raw events")
>     Reported-by: Kan Liang <kan.liang@linux.intel.com>
>     Signed-off-by: Ian Rogers <irogers@google.com>
>     Tested-by: Kan Liang <kan.liang@linux.intel.com>
>     Cc: Adrian Hunter <adrian.hunter@intel.com>
>     Cc: Alexander Shishkin <alexander.shishkin@linux.intel.com>
>     Cc: Ingo Molnar <mingo@redhat.com>
>     Cc: Jiri Olsa <jolsa@kernel.org>
>     Cc: Mark Rutland <mark.rutland@arm.com>
>     Cc: Namhyung Kim <namhyung@kernel.org>
>     Cc: Peter Zijlstra <peterz@infradead.org>
>     Cc: Ravi Bangoria <ravi.bangoria@amd.com>
>     Cc: Rob Herring <robh@kernel.org>
>     Cc: Thomas Richter <tmricht@linux.ibm.com>
>     Cc: Xing Zhengjun <zhengjun.xing@linux.intel.com>
>     Link: https://lore.kernel.org/r/20230601082954.754318-4-irogers@google.com
>     Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
> 
>  tools/perf/util/parse-events.c | 104 +++++++++++++++++++++++++++++------------
>  tools/perf/util/parse-events.y |  10 ++--
>  tools/perf/util/pmu.c          |  16 +++++++
>  tools/perf/util/pmu.h          |   5 ++
>  tools/perf/util/pmus.c         |   5 ++
>  tools/perf/util/pmus.h         |   1 +
>  6 files changed, 106 insertions(+), 35 deletions(-)
> ⬢[acme@toolbox perf-tools-next]$
> 
> - Arnaldo

Re: 'perf top' broken on intel hybrid systems

[Leah Neukirchen]

> The error is because the perf top always tries to open an event on the
> user_requested_cpus, which are all CPUs by default.
> 
> Something as below should fix it. For hybrid, open a PMU event on an
> unsupported CPU should be error out.
> 
> diff --git a/tools/perf/builtin-top.c b/tools/perf/builtin-top.c

This fixes "perf top" on my i7-1355U.  It would be great if you could
get this patch into upstream and stable branches.

Thanks,
-- 
Leah Neukirchen  <leah@vuxu.org>  https://leahneukirchen.org/

Re: 'perf top' broken on intel hybrid systems

[Ian Rogers]

On Sat, Dec 9, 2023 at 9:26 PM Ian Rogers <irogers@google.com> wrote:
>
> On Sat, Dec 9, 2023, 12:14 PM Leah Neukirchen <leah@vuxu.org> wrote:
>>
>>
>> > The error is because the perf top always tries to open an event on the
>> > user_requested_cpus, which are all CPUs by default.
>> >
>> > Something as below should fix it. For hybrid, open a PMU event on an
>> > unsupported CPU should be error out.
>> >
>> > diff --git a/tools/perf/builtin-top.c b/tools/perf/builtin-top.c
>>
>> This fixes "perf top" on my i7-1355U.  It would be great if you could
>> get this patch into upstream and stable branches.
>
>
>
> I'll try to take a look ASAP. As we don't need this for record there is some existing logic that perf top is clearly missing. We should also have a test on perf top.

So for stat and record there is a call to evlist__create_maps:
https://git.kernel.org/pub/scm/linux/kernel/git/perf/perf-tools-next.git/tree/tools/perf/builtin-stat.c?h=perf-tools-next#n2730
https://git.kernel.org/pub/scm/linux/kernel/git/perf/perf-tools-next.git/tree/tools/perf/builtin-record.c?h=perf-tools-next#n4210

In evlist__create_maps we do perf_evlist__propagate_maps and that will:
https://git.kernel.org/pub/scm/linux/kernel/git/perf/perf-tools-next.git/tree/tools/lib/perf/evlist.c?h=perf-tools-next#n43
```
} else if (evlist->has_user_cpus && evsel->is_pmu_core) {
/*
* User requested CPUs on a core PMU, ensure the requested CPUs
* are valid by intersecting with those of the PMU.
*/
perf_cpu_map__put(evsel->cpus);
evsel->cpus = perf_cpu_map__intersect(evlist->user_requested_cpus,
evsel->own_cpus);
```

I think the fix should be looking to do the same map set up in top as
stat and record, presumably there was a refactor and top was
overlooked. I'll try to address this and add a basic sanity test of
doing a few iterations of perf top on stdio, which would have been
sufficient to catch this.

Thanks,
Ian

Re: 'perf top' broken on intel hybrid systems

[Arnaldo Carvalho de Melo]

Em Fri, Dec 08, 2023 at 02:39:37PM -0500, Liang, Kan escreveu:
> On 2023-12-08 1:57 p.m., Arnaldo Carvalho de Melo wrote:
> > 	So I finally got a recent Intel hybrid system:
> > root@fedora:~# grep -m1 "model name" /proc/cpuinfo
> > model name	: Intel(R) Core(TM) i7-14700K
> > root@fedora:~#
> > 	Most things work, but:

> > root@fedora:~# perf top

> > Error:
> > The cycles:P event is not supported.
> > root@fedora:~#
> > 
> > root@fedora:~# perf top -e cycles:p
> > Error:
> > The cycles:p event is not supported.
> > root@fedora:~# perf top -e cycles:pp
> > Error:
> > The cycles:pp event is not supported.
> > ^[[Aroot@fedora:~# perf top -e cycles:ppp
> > Error:
> > The cycles:ppp event is not supported.
> > root@fedora:~#
> > root@fedora:~# perf top -e cycles
> > Error:
> > The cycles event is not supported.
> > root@fedora:~#
 
> The error is because the perf top always tries to open an event on the
> user_requested_cpus, which are all CPUs by default.

But what is wrong with that for the default event, CPU cycles?

It should work for all CPUs, its the most basic event, right?

We should have a rough idea where CPU (no matter which CPUs) cycles are
being used.

- Arnaldo

Re: 'perf top' broken on intel hybrid systems

[Ian Rogers]

On Mon, Dec 11, 2023 at 1:11 PM Arnaldo Carvalho de Melo
<acme@kernel.org> wrote:
>
> Em Fri, Dec 08, 2023 at 02:39:37PM -0500, Liang, Kan escreveu:
> > On 2023-12-08 1:57 p.m., Arnaldo Carvalho de Melo wrote:
> > >     So I finally got a recent Intel hybrid system:
> > > root@fedora:~# grep -m1 "model name" /proc/cpuinfo
> > > model name  : Intel(R) Core(TM) i7-14700K
> > > root@fedora:~#
> > >     Most things work, but:
>
> > > root@fedora:~# perf top
>
> > > Error:
> > > The cycles:P event is not supported.
> > > root@fedora:~#
> > >
> > > root@fedora:~# perf top -e cycles:p
> > > Error:
> > > The cycles:p event is not supported.
> > > root@fedora:~# perf top -e cycles:pp
> > > Error:
> > > The cycles:pp event is not supported.
> > > ^[[Aroot@fedora:~# perf top -e cycles:ppp
> > > Error:
> > > The cycles:ppp event is not supported.
> > > root@fedora:~#
> > > root@fedora:~# perf top -e cycles
> > > Error:
> > > The cycles event is not supported.
> > > root@fedora:~#
>
> > The error is because the perf top always tries to open an event on the
> > user_requested_cpus, which are all CPUs by default.
>
> But what is wrong with that for the default event, CPU cycles?
>
> It should work for all CPUs, its the most basic event, right?
>
> We should have a rough idea where CPU (no matter which CPUs) cycles are
> being used.

Generally we wouldn't want to aggregate cycles events on two differing
PMUs as the CPUs for them likely behave quite differently. Most of the
cycle event opening is now done by parsing "cycles:P" which will give
two evsels, one for each core PMU. You're right that the correct PMU
can be determined for the legacy events without an extended type if
the CPU isn't the -1 any CPU value. We'd need special parsing to make
this work in the context of perf top, which doesn't seem desirable. I
think we should support wild card PMUs and fix the CPU maps like we do
in perf stat and perf record.

Thanks,
Ian